Author SHA1 Message Date
vika 4696f89de2 Проверять членство в чате раньше тела повторного ключа.
Исключённый участник получает 404 и на другой текст. Отдельная серия показывает, что при потолке CPU Postgres лишние реплики перестают поднимать RPS.
2026-10-30 11:01:02 +03:00
vika b97fc3304f Указать в отчёте ссылку на запрос на слияние. 2026-10-30 01:29:10 +03:00
vika 59f8c55f93 Поставить несколько реплик API за общий балансировщик.
Сессия и ключ идемпотентности общие для реплик, миграция выполняется один раз, остановка одной реплики не теряет подтверждённые сообщения.
2026-10-30 01:15:28 +03:00
52 changed files with 3792 additions and 36 deletions
+18 -8
View File
@@ -1,13 +1,23 @@
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
# These are logical interface suggestions; map actual framework names in REPORT.md.
# Copy to .env for the local stand. .env is gitignored. Do not reuse these placeholders anywhere else.
APP_HOST=0.0.0.0
APP_PORT=8080
APP_ENV=lab
SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
POSTGRES_USER=messenger
POSTGRES_PASSWORD=replace-with-a-local-password
POSTGRES_DB=messenger
REDIS_PASSWORD=replace-with-a-local-password
API_REPLICAS=2
API_CPUS=0.50
API_MEMORY=512M
POSTGRES_CPUS=1.0
LB_CPUS=0.50
POOL_MAX=4
CAP_MODE=off
DRAIN_SECONDS=2
MAX_MESSAGES=100000
MAX_PER_SECOND=10000
DATABASE_URL_FILE=/run/secrets/database_url
SESSION_STORE=redis
CACHE_URL_FILE=/run/secrets/cache_url
+2
View File
@@ -0,0 +1,2 @@
*.sh text eol=lf
*.cmd text eol=crlf
+14 -14
View File
@@ -1,18 +1,18 @@
## Лабораторная 3
Автор / группа: TODO
Целевая оценка (3/4/5): TODO
Трек 5, если заявлен: TODO
Автор / группа: Казанцева Виктория / БПИ2301
Целевая оценка (3/4/5): 5
Трек 5, если заявлен: 1, цена масштабирования
Что реализовано и что перенесено: TODO
Команды запуска из чистого clone: TODO
Результаты проверок и ссылка на REPORT.md: TODO
Ограничения / известные ошибки: TODO
Источники и использование ИИ: TODO
Что реализовано и что перенесено: API лабораторной 2 за HAProxy, две и больше реплик без локальных сессий, общий Postgres и Redis, одноразовая миграция, общий Idempotency-Key, X-Instance-Id, graceful shutdown. Контракт и публичные тесты не менялись.
Команды запуска из чистого clone: см. REPORT.md. Кратко: скопировать `.env.example` в `.env`, задать локальные пароли, `docker compose --env-file .env up -d --build --scale api=2`, дождаться `/health/ready`.
Результаты проверок и ссылка на REPORT.md: smoke, persistence и failover прошли; grade 4 и трек 1 лежат в `evidence/`. Подробности и числа — в REPORT.md.
Ограничения / известные ошибки: LB, Postgres и Redis не продублированы. На фиксированном CPU четыре реплики медленнее одной. При потолке Postgres 0.20 CPU лишние реплики RPS не поднимают; снятие потолка на тех же четырёх репликах поднимает RPS примерно в 3.3 раза. Короткие ошибки при SIGTERM измеряются и допустимы. На Windows hook для публичного теста — `scripts/*.cmd`, на Linux — `scripts/*.sh`.
Источники: контракт skeleton, код лабораторной 2, документация HAProxy и Compose. Числа сняты локально.
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [ ] Контракт и публичные тесты не ослаблены.
- [ ] make check и make test выполнены; приложен реальный вывод.
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [x] Контракт и публичные тесты не ослаблены.
- [x] Проверки из make check и make test выполнены напрямую: на машине нет make. Реальный вывод — в REPORT.md.
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
+25
View File
@@ -0,0 +1,25 @@
FROM python:3.12.7-slim-bookworm
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
APP_HOST=0.0.0.0 \
APP_PORT=8080
WORKDIR /app
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin app
COPY requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt
COPY migrations /app/migrations
COPY src /app/src
USER app
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=3s --start-period=20s --retries=12 \
CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"]
CMD ["python", "/app/src/app.py"]
+133 -14
View File
@@ -1,37 +1,156 @@
# Отчёт — лабораторная 3
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Казанцева Виктория / БПИ2301
- Целевая оценка; выбранный трек 5: 5, трек 1 (цена масштабирования)
- Commit/PR: https://gitea.privcloud.tech/mtusi2026/mtusi2026lab3/pulls/71
- Использованные источники и библиотеки: контракт и публичные тесты skeleton, код лабораторной 2, документация HAProxy про DNS-резолвер и health-check, документация Docker Compose про `deploy.replicas` и лимиты. Библиотеки: argon2-cffi 23.1.0, psycopg 3.2.9 с пулом, redis 5.2.1. Образы: python:3.12.7-slim-bookworm, postgres:16.8-bookworm, redis:7.4.2-alpine, haproxy:2.8.14-alpine. Сборка приложения — локальный образ `mtusi-messenger:lab3`. Команды, smoke и замеры прогнаны локально 30.10.2026 около 00:20 (UTC+3), числа ниже с этого запуска.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Нужны Docker Compose v2 и Python 3.9+. У меня проверки шли интерпретатором `python` 3.11.5, рядом есть `python3` 3.11.9. Docker Engine 29.2.1, Windows. Программы `make` на машине нет, поэтому ниже те же команды, что в Makefile, вызваны напрямую. На Windows публичные тесты читают UTF-8 контракт системной кодировкой, поэтому перед ними нужен `PYTHONUTF8=1`. На Linux и macOS этого не требуется, и там же hook запускается как `scripts/restart.sh` и `scripts/stop-one.sh` после `chmod +x`. На Windows Python не исполняет shebang, поэтому те же действия лежат в `scripts/restart.cmd` и `scripts/stop-one.cmd`.
Пароли только в `.env`. Файл в gitignore. В репозиторий попадает `.env.example` с плейсхолдерами.
```text
copy .env.example .env
```
Дальше вписать локальные `POSTGRES_PASSWORD` и `REDIS_PASSWORD`. Затем одна последовательность:
```text
docker compose --env-file .env config --quiet
docker compose --env-file .env up -d --build --scale api=2
```
`deploy.replicas` тоже равен `API_REPLICAS` (по умолчанию 2). `--scale` явно задаёт то же число и им я меняла число реплик в эксперименте. Миграции выполняет одноразовый сервис `migrate` до старта API. Реплики сами SQL не применяют. Подождать, пока `http://127.0.0.1:8080/health/ready` ответит 200 и в заголовке будет `X-Instance-Id`.
Проверки, как я их запускала:
```text
set PYTHONUTF8=1
python scripts/check_contract.py
python tests/smoke.py --base-url http://127.0.0.1:8080
python tests/resilience.py --mode persistence --action-script scripts/restart.cmd --base-url http://127.0.0.1:8080
python tests/resilience.py --mode failover --action-script scripts/stop-one.cmd --base-url http://127.0.0.1:8080
python tests/load.py --base-url http://127.0.0.1:8080 --duration 20 --concurrency 4 --output evidence/load.json
python student-tests/experiment_grade4.py
python student-tests/experiment_track1.py
python student-tests/experiment_saturation.py
python student-tests/check_idem_access.py
```
На Linux то же самое через `make test`, `make test-persistence ACTION_SCRIPT=scripts/restart.sh` и `make test-failover ACTION_SCRIPT=scripts/stop-one.sh`.
После failover реплика остаётся остановленной. Вернуть её: `docker compose --env-file .env start api`. Остановка без удаления данных: `docker compose --env-file .env stop`. Данные остаются в томах `pg-data` и `redis-data`. Удаление учебных данных, если оно действительно нужно: `docker compose --env-file .env down -v`. Эту команду тесты и hook не вызывают.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab2, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Клиент ходит только на HAProxy (`lb`, `127.0.0.1:8080`). За ним сервис `api` без `container_name` и без проброшенного наружу порта. Снаружи виден только балансировщик. У API нет локальных сессий и нет своего тома. Пользователи, чаты, сообщения и ключи идемпотентности лежат в одном PostgreSQL. Сессии лежат в одном Redis с AOF и `appendfsync always`. Пароль Postgres есть у `postgres`, `migrate` и `api`. Пароль Redis есть у `redis` и `api`. У балансировщика учётных данных баз нет.
`X-Instance-Id` — hostname контейнера, первые 12 символов его id. Он не меняется, пока живёт этот контейнер, и это не cookie клиента. Балансировка `roundrobin`, sticky session нет.
Отправка сообщения требует `Idempotency-Key` (`[A-Za-z0-9._:-]{1,128}`). Область ключа: пользователь, метод POST, путь `/api/v1/chats/{id}/messages`, сам ключ. Отпечаток тела — SHA-256 канонического JSON. Пустой список вложений и отсутствующее поле для этого отпечатка совпадают. Успешная вставка сообщения и строка ключа коммитятся одной транзакцией. На время транзакции берётся advisory lock именно этой области, поэтому два параллельных повтора через разные реплики не создают два сообщения. Повтор того же тела возвращает тот же 201 и то же сообщение. Другое тело — 409. Ошибка 5xx транзакцию откатывает, ключ не занимается, клиент может повторить тот же ключ. Перед повтором снова проверяются сессия и членство в чате. Членство проверяется раньше сравнения отпечатка тела: если участника уже исключили, повтор с тем же ключом и другим текстом даёт 404, а не 409. Проверка: сообщение создано, тот же ключ с другим текстом дал 409, затем строка участника удалена прямо в Postgres, и оба повтора (старый и новый текст) дали 404. История этого чата для него тоже 404. Срок хранения ключа — `IDEMPOTENCY_TTL_SECONDS`, по умолчанию 86400.
`GET /health/live` отвечает 200, пока процесс жив, в том числе после SIGTERM. `GET /health/ready` отвечает 200 только если процесс принимает запросы, схема уже накатана, Postgres и Redis отвечают. На SIGTERM процесс сразу перестаёт принимать работу, ready становится 503, ждёт `DRAIN_SECONDS` (2) и только потом закрывает сокет. `stop_grace_period` у API — 8 с, `docker stop -t 6` укладывается в лимит hook 10 с.
HAProxy сам обновляет адреса через DNS Docker `127.0.0.11`. `hold valid 1s`. Проверка backend: `GET /health/ready` каждые 1 с, `fall 2`, `rise 1`. Недоступный backend выводится примерно за 2 с по health-check и не позже чем за 1 с по DNS, когда контейнер уже исчез из DNS. Верхняя оценка обнаружения отказа — 3 с. Таймауты: connect 500 мс, server 10 с, client 15 с, check 1 с, queue 1 с. Для обычных запросов повтор на другой backend есть только при обрыве соединения до отправки и при 503, который API отдаёт до commit. Потерянный ответ 201 у создания чата балансировщик не повторяет. Повтор пустого ответа, 502 и 503 включён только у `POST /chats/{id}/messages`: там общий ключ идемпотентности. `depends_on` только упорядочивает первый старт и балансировщиком не является.
Соответствие `.env.example` и стенда: `APP_PORT` публикует только `lb`. `SESSION_TTL_SECONDS` и `IDEMPOTENCY_TTL_SECONDS` читает API. `DATABASE_URL` и `REDIS_URL` собирает Compose из паролей `.env`; файлы `DATABASE_URL_FILE` и `CACHE_URL_FILE` код умеет читать, но этот стенд их не монтирует. `API_REPLICAS`, `API_CPUS`, `API_MEMORY`, `POOL_MAX`, `CAP_MODE`, `DRAIN_SECONDS` задают число реплик, лимиты API, пул и режим предела сообщений. `POSTGRES_CPUS` и `LB_CPUS` задают потолок CPU базы и балансировщика. В обычном стенде это 1.0 и 0.50. В серии насыщения Postgres временно ставился на 0.20, затем возвращался на 1.0.
Лимиты по умолчанию: API 2 × 0.50 CPU и 512 МиБ, Postgres 1.0 CPU и 512 МиБ, Redis 0.25 CPU и 128 МиБ, HAProxy 0.50 CPU и 128 МиБ. `max_connections` Postgres — 100, `shared_buffers` — 128 МиБ, `fsync` и `synchronous_commit` включены. Пул API по умолчанию 4 соединения на процесс.
`CAP_MODE=strict` оставляет глобальный lock и `count(*)` перед каждой вставкой, как в лабораторной 2. В обычном стенде `CAP_MODE=off`: контракт предел числа сообщений не требует, а общий lock сериализует все вставки. Это не источник выигрыша в треке 1, см. ниже.
Миграция одна, `migrations/001_init.sql`. Её пишет сервис `migrate` и отмечает в `schema_migrations`. Повторный запуск контейнера строки не вставляет. Реплики в ready только проверяют, что версия уже есть. Несовместимую миграцию при живых старых репликах этот стенд не умеет: схема одна и накатывается до приёма трафика.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
| Публичный smoke | `python tests/smoke.py --base-url http://127.0.0.1:8080` | PASS. Повторён после экспериментов, уже на восстановленных репликах | вывод в этом отчёте |
| Две реплики, одна точка входа, общая БД и Redis | `docker compose ps` | `api` ×2 без порта на хосте, `lb` только на `127.0.0.1:8080`, Postgres и Redis без публикации на хост | этот раздел |
| Разные реплики одной сессии | 30 запросов `GET /health/ready` через LB | два `X-Instance-Id`: `5331f4cbcae1` и `dea5f28995dd` | финальный стенд после трека |
| Отказ одной реплики | `tests/resilience.py --mode failover --action-script scripts/stop-one.cmd` | до отказа видно 2 реплики; 123 успешных записи, 0 ошибок; первая успешная запись через 3.328 с; остановлен именно `mtusi-lab3-api-2`, LB/Postgres/Redis живы | `evidence/failover-public.json` |
| Идемпотентность, в том числе параллельно и с двух реплик | `student-tests/experiment_grade4.py` | 16 параллельных POST с одним ключом и телом: 16×201, один id, в истории одна копия, оба instance id. 8 POST с одним ключом и разными телами: один 201 и семь 409 | `evidence/grade4.json` |
| Сохранность тома и сессии | `tests/resilience.py --mode persistence --action-script scripts/restart.cmd` | PASS, тот же токен. StartedAt изменился у api-1, api-2, postgres, redis и lb. `migrate` не перезапускался | `evidence/persistence-before.txt`, `evidence/persistence-after.txt` |
| Health, SIGTERM, время обнаружения | конфиг HAProxy и `DRAIN_SECONDS=2` | проверка раз в 1 с, fall 2, DNS hold 1 с; верхняя граница 3 с. Публичный failover уложился в 3.328 с при бюджете 10 с | `infra/haproxy.cfg` |
| Поочерёдный отказ каждой реплики | тот же grade4-скрипт | `836017c62a61`: stop 2.422 с, 27 ошибок в окне, реплика вернулась. `a0f9a69d71e5`: stop 2.390 с, 33 ошибки, реплика вернулась. Вторая реплика в это время продолжала писать | `evidence/grade4.json` |
| Лимиты, нет обхода LB, нет container_name | `compose.yaml`, `docker compose ps` | лимиты CPU/RAM у всех сервисов; у `api` нет `container_name` и нет host-порта | `compose.yaml` |
| Поочерёдная замена под нагрузкой | `scripts/rolling-replace.ps1` из grade4-скрипта | 7.219 с, 1341 успешное чтение, 401 ошибка на закрываемых сокетах, после замены два новых id | `evidence/grade4.json` |
| Миграция один раз | `psql` по `schema_migrations` | одна строка `001_init.sql` | команда в этом разделе |
| Трек 1 | `student-tests/experiment_track1.py` | см. таблицу ниже. Все 10 прогонов: 0 ошибок клиента, видно ровно заявленное число реплик | `evidence/track1.json`, `evidence/track1.svg` |
| Насыщение Postgres | `student-tests/experiment_saturation.py` | при потолке Postgres 0.20 CPU ряд 1/2/4 реплики по 1.0 CPU даёт 135.440 / 141.407 / 107.017 RPS. Те же 4 реплики со снятым потолком (1.0 CPU у Postgres) дают 352.885 RPS. Ошибок 0 | `evidence/saturation.json`, `evidence/load-sat-*.json` |
| Нагрузка рабочего стенда | `tests/load.py --duration 20 --concurrency 4` | 3630 успехов, 0 ошибок, 180.651 RPS, p50/p95/p99 16/47/94 мс | `evidence/load.json` |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
`check_contract.py` тоже PASS. Контракт и публичные тесты я не меняла.
Рабочая нагрузка после восстановления стенда идёт на две реплики по 0.50 CPU и 512 МиБ. Профиль трека другой: 12 с, concurrency 8, поэтому RPS там нельзя складывать с 180.651.
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Машина: Windows 10.0.26100, 16 CPU, около 7.6 ГиБ RAM у Docker. В `docker stats` 100% — это одно ядро: реплика с лимитом 1.0 CPU показывала около 74–80%, а не сотни процентов.
Гипотеза: при фиксированных 1.0 CPU и 1024 МиБ на все API рост числа процессов не поднимет RPS, потому что каждому достанется меньший кусок CPU. Рост числа реплик поможет, только если у каждой останется свой CPU. Отдельно проверила, не упрётся ли стенд в пул соединений.
Профиль трека: `tests/load.py`, closed-loop POST сообщений, 12 с, concurrency 8, без автоматических retry. Прогрев отдельно не выделяла. Каждая точка — один прогон. Клиент на той же машине, поэтому абсолютный RPS включает стоимость клиента. В `docker stats` среднее CPU — среднее по образцам контейнеров этой роли, не сумма. Для четырёх реплик суммарный CPU API примерно в четыре раза больше указанного среднего.
| Серия | Реплики | CPU на реплику | RPS | p50/p95/p99, мс | CPU API, среднее | CPU Postgres | CPU LB |
| --- | --- | --- | --- | --- | --- | --- | --- |
| фиксированный бюджет, strict | 1 | 1.00 | 158.286 | 47/94/125 | 74.21 | 23.25 | 10.45 |
| фиксированный бюджет, strict | 2 | 0.50 | 174.844 | 32/109/187 | 30.32 | 26.05 | 6.40 |
| фиксированный бюджет, strict | 4 | 0.25 | 136.692 | 46/188/281 | 18.74 | 15.16 | 8.94 |
| фиксированный бюджет, cap off | 1 | 1.00 | 184.701 | 32/93/109 | 80.13 | 15.36 | 8.82 |
| фиксированный бюджет, cap off | 2 | 0.50 | 165.162 | 31/125/203 | 36.09 | 15.01 | 10.63 |
| фиксированный бюджет, cap off | 4 | 0.25 | 109.204 | 31/281/329 | 16.83 | 11.07 | 7.92 |
| CPU растёт вместе с числом | 1 | 0.50 | 84.703 | 94/203/219 | 42.47 | 7.93 | 4.44 |
| CPU растёт вместе с числом | 2 | 0.50 | 147.996 | 32/141/203 | 31.20 | 15.35 | 9.64 |
| CPU растёт вместе с числом | 4 | 0.50 | 329.198 | 16/78/125 | 34.41 | 34.01 | 30.12 |
| тот же 4×0.50, пул 16 | 4 | 0.50 | 319.011 | 16/78/171 | 30.75 | 31.35 | 21.05 |
График: `evidence/track1.svg`. Сырые ответы нагрузки: `evidence/load-*.json`.
При фиксированном 1.0 CPU четыре реплики хуже одной: 109 против 185 RPS и p99 329 мс против 109 мс. Каждая реплика с лимитом 0.25 CPU сидит около этого потолка (среднее 17–19%). Postgres при этом занят на 0.11–0.26 ядра из разрешённого 1.0. Узкое место здесь — нарезанный CPU процессов API, а не база и не балансировщик.
Когда у каждой реплики остаётся 0.50 CPU, ряд 1/2/4 даёт 84.7 → 148.0 → 329.2 RPS. Четыре реплики почти в 3.9 раза быстрее одной. Postgres доходит до 0.34 ядра из разрешённого 1.0, LB до 0.30 при своём лимите 0.50. На этом профиле база и балансировщик ещё не упёрлись: предел здесь нарезанный CPU API. Увеличение пула с 4 до 16 на этой точке RPS не подняло: 329.2 против 319.0. Пул на этом профиле не насыщен.
Отдельная серия ищет предел уже не в CPU реплик. Каждой реплике отдан целый 1.0 CPU, пул оставлен 4, CAP выключен, нагрузка та же `tests/load.py`, но 12 с и concurrency 16. Postgres ограничен 0.20 CPU. В `docker stats` 100% — одно ядро, поэтому этот потолок виден около 20%.
| Серия | Реплики | CPU Postgres | RPS | p50/p95/p99, мс | CPU API, среднее | CPU Postgres | CPU LB |
| --- | --- | --- | --- | --- | --- | --- | --- |
| потолок базы 0.20 | 1 | 0.20 | 135.440 | 94/203/375 | 63.85 | 13.31 | 6.64 |
| потолок базы 0.20 | 2 | 0.20 | 141.407 | 109/219/328 | 43.39 | 15.00 | 9.32 |
| потолок базы 0.20 | 4 | 0.20 | 107.017 | 110/312/453 | 23.35 | 16.89 | 8.50 |
| тот же 4×1.0, потолок снят | 4 | 1.0 | 352.885 | 31/156/203 | 49.66 | 44.90 | 28.18 |
Сырые ответы: `evidence/load-sat-*.json`, сводка: `evidence/saturation.json`. Ошибок клиента не было.
При потолке 0.20 рост реплик останавливается: 135.4 → 141.4 → 107.0 RPS. Четыре реплики медленнее одной, p99 вырастает до 453 мс. Средний CPU Postgres подходит к потолку (13.3, затем 15.0, затем 16.9 при видимом потолке около 20). CPU одной реплики при этом падает (64 → 43 → 23), значит ядра API простаивают и не они держат RPS. Те же четыре реплики по 1.0 CPU, но уже с Postgres 1.0 CPU, дают 352.9 RPS, примерно в 3.3 раза больше. База тогда занята на 0.45 ядра из разрешённого 1.0, балансировщик на 0.28 при лимите 0.50, среднее API около половины ядра при лимите в целое ядро. Пул в этой серии не менялся. Общий предел, из-за которого лишние реплики перестают помогать, — CPU Postgres. Пока этот потолок снят, ни база, ни балансировщик, ни пул на этой точке ещё не заполнены.
Подтверждённое улучшение: не резать 1.0 CPU на четыре процесса, а давать реплике отдельные 0.50 CPU и добавлять такие реплики. Переход 1×0.50 → 2×0.50 поднимает RPS с 84.7 до 148.0, переход к 4×0.50 — до 329.2. Обратная нарезка того же суммарного CPU на четыре процесса снижает RPS со 184.7 до 109.2. Глобальный lock предела сообщений на одном ядре дал 158 против 185 RPS, но на двух и четырёх репликах картина уже шумная, поэтому улучшением я его не объявляю. В рабочем стенде он выключен.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
| Перезапуск API, Postgres, Redis и LB с сохранением томов | тот же токен и те же сообщения | PASS публичного persistence; StartedAt у хранилищ и API сменился, том не удалялся | данные и сессия на месте |
| `stop-one` одной реплики | не позже 10 с идут новые записи, остановленная реплика не отвечает | 3.328 с, 0 ошибок у публичного клиента, 123 успешных записи, контейнер api-2 в Exited | подтверждённые сообщения целы |
| Поочерёдная остановка каждой реплики | вторая продолжает писать, первая потом возвращается | stop около 2.4 с; 27 и 33 ошибки своего клиента без retry; обе реплики снова видны в LB | потери подтверждённых записей нет |
| Поочерёдная замена контейнеров | остаётся две живые реплики | 7.2 с, 1341 успешное чтение, 401 ошибка на закрытии сокета, два новых id | сессии в Redis не трогала |
Ошибки своего клиента в grade4 больше, чем в публичном failover: публичный тест идёт через HAProxy, а тот повторяет обрыв и 503 на другой реплике. Свой генератор считает каждый неуспешный ответ ошибкой и сам не повторяет его. Оба наблюдения совместимы.
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
Стенд переживает остановку одного процесса API и поочерёдную замену контейнеров. Он не переживает потерю узла, на котором лежат LB, Postgres или Redis: все три в одном экземпляре. Это не трек 2, и я не утверждаю отказоустойчивость базы.
Точки трека и серия насыщения сняты по одному разу, по 12 секунд, closed-loop, с клиента на той же машине. Это не интервал доверия и не открытый поток запросов. На профиле с Postgres 1.0 CPU база не упиралась в свой потолок. Она упирается, когда потолок задан в 0.20 CPU: это и есть измеренная точка, после которой новые реплики не поднимают RPS. Пул 16 против 4 на серии 4×0.50 ничего не ускорил. При снятом потолке базы ближайший общий кандидат — лимит 0.50 CPU у HAProxy: на четырёх репликах по 1.0 CPU он был на 0.28 ядра и ещё не заполнился.
Короткие ошибки в момент SIGTERM и удаления контейнера есть. Подтверждённая запись не пропадает: ключ и сообщение коммитятся вместе, а неподтверждённый POST можно повторить с тем же ключом. POST регистрации и создания чата так повторять нельзя, у них ключа нет.
## Вопросы на защите
Отказ процесса — это остановленный контейнер API. LB, база и Redis остаются, вторая реплика продолжает обслуживать сессию из Redis. Отказ узла забирает всё, что было только на этом узле. Здесь LB, Postgres и Redis не продублированы, поэтому потерю их узла стенд не переживёт.
`depends_on` и healthcheck контейнера только решают, можно ли стартовать зависимому сервису и здоров ли он для Docker. Они не принимают клиентские соединения и не выбирают живую реплику. Это делает HAProxy: round-robin, DNS и проверка `/health/ready`.
После таймаута можно повторять POST сообщения с тем же телом и тем же `Idempotency-Key`. Защищает общая строка ключа в Postgres, записанная в одной транзакции с сообщением. Повтор создания пользователя или чата таким ключом не защищён и может создать второй объект.
Если сессии лежат в памяти процесса, новая реплика не знает токен. Запрос уйдёт на неё и получит 401, хотя пользователь только что вошёл. Общий Redis это убирает: любая реплика читает одну и ту же сессию.
+169
View File
@@ -0,0 +1,169 @@
# Local stand for lab3. One compose service "api" is scaled; it has no container_name
# and publishes no host port. Clients enter only through lb.
# Volumes survive scripts/restart.sh. scripts/stop-one.sh stops a single api container.
name: mtusi-lab3
x-logging: &logging
driver: json-file
options:
max-size: "10m"
max-file: "2"
x-api-image: &api-image mtusi-messenger:lab3
services:
postgres:
image: postgres:16.8-bookworm
environment:
POSTGRES_USER: ${POSTGRES_USER:-messenger}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
POSTGRES_DB: ${POSTGRES_DB:-messenger}
command:
- postgres
- -c
- synchronous_commit=on
- -c
- fsync=on
- -c
- full_page_writes=on
- -c
- max_connections=100
- -c
- shared_buffers=128MB
volumes:
- pg-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
interval: 5s
timeout: 3s
retries: 20
start_period: 10s
deploy:
resources:
limits:
cpus: "${POSTGRES_CPUS:-1.0}"
memory: 512M
logging: *logging
networks: [lab]
restart: unless-stopped
redis:
image: redis:7.4.2-alpine
environment:
REDIS_PASSWORD: ${REDIS_PASSWORD:?set REDIS_PASSWORD in .env}
command:
- redis-server
- --appendonly
- "yes"
- --appendfsync
- always
- --dir
- /data
- --requirepass
- ${REDIS_PASSWORD}
volumes:
- redis-data:/data
healthcheck:
test: ["CMD-SHELL", "redis-cli -a \"$$REDIS_PASSWORD\" --no-auth-warning ping | grep -q PONG"]
interval: 5s
timeout: 3s
retries: 20
start_period: 5s
deploy:
resources:
limits:
cpus: "0.25"
memory: 128M
logging: *logging
networks: [lab]
restart: unless-stopped
migrate:
image: *api-image
build: .
command: ["python", "/app/src/migrate.py"]
environment:
DATABASE_URL: postgresql://messenger:${POSTGRES_PASSWORD}@postgres:5432/messenger
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "0.50"
memory: 256M
logging: *logging
networks: [lab]
restart: "no"
api:
image: *api-image
build: .
environment:
APP_HOST: 0.0.0.0
APP_PORT: "8080"
DATABASE_URL: postgresql://messenger:${POSTGRES_PASSWORD}@postgres:5432/messenger
REDIS_URL: redis://:${REDIS_PASSWORD}@redis:6379/0
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
MAX_MESSAGES: ${MAX_MESSAGES:-100000}
MAX_PER_SECOND: ${MAX_PER_SECOND:-10000}
MAX_INFLIGHT_BODIES: ${MAX_INFLIGHT_BODIES:-32}
POOL_MAX: ${POOL_MAX:-4}
CAP_MODE: ${CAP_MODE:-off}
DRAIN_SECONDS: ${DRAIN_SECONDS:-2}
depends_on:
migrate:
condition: service_completed_successfully
postgres:
condition: service_healthy
redis:
condition: service_healthy
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"]
interval: 2s
timeout: 2s
retries: 10
start_period: 20s
stop_grace_period: 8s
deploy:
replicas: ${API_REPLICAS:-2}
resources:
limits:
cpus: "${API_CPUS:-0.50}"
memory: ${API_MEMORY:-512M}
logging: *logging
networks: [lab]
restart: unless-stopped
lb:
image: haproxy:2.8.14-alpine
volumes:
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
ports:
- "127.0.0.1:${APP_PORT:-8080}:8080"
depends_on:
api:
condition: service_healthy
healthcheck:
test: ["CMD", "haproxy", "-c", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]
interval: 10s
timeout: 3s
retries: 3
start_period: 5s
deploy:
resources:
limits:
cpus: "${LB_CPUS:-0.50}"
memory: 128M
logging: *logging
networks: [lab]
restart: unless-stopped
networks:
lab:
name: mtusi-lab3-net
volumes:
pg-data:
redis-data:
+11
View File
@@ -0,0 +1,11 @@
{
"mode": "failover",
"action_script": "scripts/stop-one.cmd",
"observed_instances_before": 2,
"successful_writes": 123,
"failed_attempts": 0,
"first_recovery_seconds": 3.328,
"stopped_container": "mtusi-lab3-api-2",
"left_running": ["mtusi-lab3-api-1", "mtusi-lab3-lb-1", "mtusi-lab3-postgres-1", "mtusi-lab3-redis-1"],
"result": "PASS"
}
+55
View File
@@ -0,0 +1,55 @@
{
"instances_before": [
"836017c62a61",
"a0f9a69d71e5"
],
"parallel_same_key": {
"attempts": 16,
"successes": 16,
"errors": 0,
"distinct_message_ids": 1,
"history_copies": 1,
"instances": [
"836017c62a61",
"a0f9a69d71e5"
]
},
"parallel_different_body": {
"created": 1,
"conflicts": 7,
"other": []
},
"stop_each_replica": [
{
"stopped_instance": "836017c62a61",
"container": "mtusi-lab3-api-1",
"successful_writes_before_stop": 324,
"successful_writes_after_stop_call": 615,
"failed_attempts_after_stop_call": 27,
"docker_stop_seconds": 2.422,
"first_other_instance_seconds": 0.015,
"target_seen_again_after_start": true
},
{
"stopped_instance": "a0f9a69d71e5",
"container": "mtusi-lab3-api-2",
"successful_writes_before_stop": 372,
"successful_writes_after_stop_call": 586,
"failed_attempts_after_stop_call": 33,
"docker_stop_seconds": 2.39,
"first_other_instance_seconds": 0.0,
"target_seen_again_after_start": true
}
],
"rolling_replace": {
"script_exit": 0,
"elapsed_seconds": 7.219,
"successful_reads": 1341,
"failed_reads": 401,
"instances_after": [
"53e109b08cae",
"b0e0a9d3ce35"
],
"stderr_tail": " Container mtusi-lab3-api-2 Running \n Container mtusi-lab3-api-3 Creating \n Container mtusi-lab3-api-3 Created \n Container mtusi-lab3-api-3 Starting \n Container mtusi-lab3-api-3 Started \n Container mtusi-lab3-api-3 Running \n Container mtusi-lab3-api-4 Creating \n Container mtusi-lab3-api-4 Created \n Container mtusi-lab3-api-4 Starting \n Container mtusi-lab3-api-4 Started \n"
}
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.328,
"concurrency": 8,
"successful_requests": 2277,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 184.701,
"successful_latency_ms": {
"p50": 32.0,
"p95": 93.0,
"p99": 109.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.188,
"concurrency": 8,
"successful_requests": 2013,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 165.162,
"successful_latency_ms": {
"p50": 31.0,
"p95": 125.0,
"p99": 203.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.234,
"concurrency": 8,
"successful_requests": 1336,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 109.204,
"successful_latency_ms": {
"p50": 31.0,
"p95": 281.0,
"p99": 329.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.25,
"concurrency": 8,
"successful_requests": 1939,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 158.286,
"successful_latency_ms": {
"p50": 47.0,
"p95": 94.0,
"p99": 125.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.188,
"concurrency": 8,
"successful_requests": 2131,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 174.844,
"successful_latency_ms": {
"p50": 32.0,
"p95": 109.0,
"p99": 187.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.188,
"concurrency": 8,
"successful_requests": 1666,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 136.692,
"successful_latency_ms": {
"p50": 46.0,
"p95": 188.0,
"p99": 281.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.172,
"concurrency": 8,
"successful_requests": 1031,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 84.703,
"successful_latency_ms": {
"p50": 94.0,
"p95": 203.0,
"p99": 219.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.203,
"concurrency": 8,
"successful_requests": 1806,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 147.996,
"successful_latency_ms": {
"p50": 32.0,
"p95": 141.0,
"p99": 203.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.172,
"concurrency": 8,
"successful_requests": 4007,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 329.198,
"successful_latency_ms": {
"p50": 16.0,
"p95": 78.0,
"p99": 125.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.172,
"concurrency": 8,
"successful_requests": 3883,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 319.011,
"successful_latency_ms": {
"p50": 16.0,
"p95": 78.0,
"p99": 171.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.5,
"concurrency": 16,
"successful_requests": 1693,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 135.44,
"successful_latency_ms": {
"p50": 94.0,
"p95": 203.0,
"p99": 375.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.609,
"concurrency": 16,
"successful_requests": 1783,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 141.407,
"successful_latency_ms": {
"p50": 109.0,
"p95": 219.0,
"p99": 328.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.484,
"concurrency": 16,
"successful_requests": 1336,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 107.017,
"successful_latency_ms": {
"p50": 110.0,
"p95": 312.0,
"p99": 453.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 12.0,
"elapsed_seconds": 12.531,
"concurrency": 16,
"successful_requests": 4422,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 352.885,
"successful_latency_ms": {
"p50": 31.0,
"p95": 156.0,
"p99": 203.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 3,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 20.0,
"elapsed_seconds": 20.094,
"concurrency": 4,
"successful_requests": 3630,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 180.651,
"successful_latency_ms": {
"p50": 16.0,
"p95": 47.0,
"p99": 94.0
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+6
View File
@@ -0,0 +1,6 @@
/mtusi-lab3-lb-1 2026-10-29T21:19:32.893655112Z
/mtusi-lab3-api-2 2026-10-29T21:19:32.261827388Z
/mtusi-lab3-api-1 2026-10-29T21:19:32.001177584Z
/mtusi-lab3-migrate-1 2026-10-29T21:17:20.663556003Z
/mtusi-lab3-postgres-1 2026-10-29T21:19:23.048047616Z
/mtusi-lab3-redis-1 2026-10-29T21:19:23.118775663Z
+6
View File
@@ -0,0 +1,6 @@
/mtusi-lab3-lb-1 2026-10-29T21:17:28.533048092Z
/mtusi-lab3-api-2 2026-10-29T21:17:22.46958774Z
/mtusi-lab3-api-1 2026-10-29T21:17:22.668353102Z
/mtusi-lab3-migrate-1 2026-10-29T21:17:20.663556003Z
/mtusi-lab3-postgres-1 2026-10-29T21:17:14.869629237Z
/mtusi-lab3-redis-1 2026-10-29T21:17:14.875217527Z
+101
View File
@@ -0,0 +1,101 @@
{
"question": "With a whole core per API replica, extra replicas stop helping when Postgres CPU is the shared cap.",
"conclusion": "At a 0.20 CPU Postgres cap, 1/2/4 replicas gave 135.440 / 141.407 / 107.017 RPS. Postgres mean CPU rose toward the cap (13.31, 15.00, 16.89; the cap is about 20 in these units) while API CPU per replica fell, so the API cores were idle. The same 4 replicas with the Postgres cap released to 1.0 CPU gave 352.885 RPS and Postgres used 44.90, under its new cap. LB stayed at 28.18 against a 0.50 cap. The shared limit that stopped replica growth is Postgres CPU.",
"method": "tests/load.py closed-loop POST, 12s, concurrency 16, one run each. Postgres limit 0.20 CPU for 1/2/4 replicas, then the same 4 replicas with Postgres limit 1.0.",
"host": {
"platform": "Windows-10-10.0.26100-SP0",
"python": "3.11.5",
"docker_info": "16 8173215744"
},
"note": "docker stats 100 percent is one core. A 0.20 CPU limit is saturated near 20 percent.",
"runs": [
{
"name": "pg020-r1",
"replicas": 1,
"api_cpus_each": "1.0",
"postgres_cpus": "0.20",
"lb_cpus": "0.50",
"pool_max": 4,
"concurrency": 16,
"duration_seconds": 12,
"successful_rps": 135.44,
"failed_requests": 0,
"successful_requests": 1693,
"latency_ms": {
"p50": 94.0,
"p95": 203.0,
"p99": 375.0
},
"cpu_api_mean": 63.85,
"cpu_postgres_mean": 13.31,
"cpu_lb_mean": 6.64,
"load_exit": 0
},
{
"name": "pg020-r2",
"replicas": 2,
"api_cpus_each": "1.0",
"postgres_cpus": "0.20",
"lb_cpus": "0.50",
"pool_max": 4,
"concurrency": 16,
"duration_seconds": 12,
"successful_rps": 141.407,
"failed_requests": 0,
"successful_requests": 1783,
"latency_ms": {
"p50": 109.0,
"p95": 219.0,
"p99": 328.0
},
"cpu_api_mean": 43.39,
"cpu_postgres_mean": 15.0,
"cpu_lb_mean": 9.32,
"load_exit": 0
},
{
"name": "pg020-r4",
"replicas": 4,
"api_cpus_each": "1.0",
"postgres_cpus": "0.20",
"lb_cpus": "0.50",
"pool_max": 4,
"concurrency": 16,
"duration_seconds": 12,
"successful_rps": 107.017,
"failed_requests": 0,
"successful_requests": 1336,
"latency_ms": {
"p50": 110.0,
"p95": 312.0,
"p99": 453.0
},
"cpu_api_mean": 23.35,
"cpu_postgres_mean": 16.89,
"cpu_lb_mean": 8.5,
"load_exit": 0
},
{
"name": "pg100-r4",
"replicas": 4,
"api_cpus_each": "1.0",
"postgres_cpus": "1.0",
"lb_cpus": "0.50",
"pool_max": 4,
"concurrency": 16,
"duration_seconds": 12,
"successful_rps": 352.885,
"failed_requests": 0,
"successful_requests": 4422,
"latency_ms": {
"p50": 31.0,
"p95": 156.0,
"p99": 203.0
},
"cpu_api_mean": 49.66,
"cpu_postgres_mean": 44.9,
"cpu_lb_mean": 28.18,
"load_exit": 0
}
]
}
+229
View File
@@ -0,0 +1,229 @@
{
"hypothesis": "With a fixed total of 1.0 CPU and 1024 MiB, more API replicas will not raise RPS, because each process gets a smaller CPU slice. Extra replicas should help only when each keeps its own CPU. A larger DB pool or a global cap lock was checked as a possible next limit.",
"conclusion": "Fixed budget: cap off, 1 replica 184.7 RPS, 2 replicas 165.2, 4 replicas 109.2. Grow budget at 0.50 CPU each: 84.7, 148.0, 329.2 RPS. Pool 4 to 16 at 4x0.50 CPU did not raise RPS (329.2 to 319.0). Postgres stayed under 0.35 core. The confirmed improvement is giving each replica its own 0.50 CPU instead of slicing 1.0 CPU into four processes.",
"profile": {
"tool": "tests/load.py closed-loop POST messages",
"duration_seconds": 12,
"concurrency": 8,
"fixed_total": "1.0 CPU and 1024 MiB split across 1/2/4 replicas",
"grow": "0.50 CPU and 256 MiB per replica, count 1/2/4"
},
"host": {
"platform": "Windows-10-10.0.26100-SP0",
"python": "3.11.5",
"cpu_count": 16,
"docker_info": "16 8173215744"
},
"runs": [
{
"series": "fixed-strict",
"replicas": 1,
"cpus_each": "1.0",
"memory_each": "1024M",
"cap": "strict",
"pool_max": 4,
"instances_seen": 1,
"successful_rps": 158.286,
"failed_requests": 0,
"successful_requests": 1939,
"latency_ms": {
"p50": 47.0,
"p95": 94.0,
"p99": 125.0
},
"cpu_api_mean": 74.21,
"cpu_postgres_mean": 23.25,
"cpu_lb_mean": 10.45,
"load_exit": 0
},
{
"series": "fixed-strict",
"replicas": 2,
"cpus_each": "0.50",
"memory_each": "512M",
"cap": "strict",
"pool_max": 4,
"instances_seen": 2,
"successful_rps": 174.844,
"failed_requests": 0,
"successful_requests": 2131,
"latency_ms": {
"p50": 32.0,
"p95": 109.0,
"p99": 187.0
},
"cpu_api_mean": 30.32,
"cpu_postgres_mean": 26.05,
"cpu_lb_mean": 6.4,
"load_exit": 0
},
{
"series": "fixed-strict",
"replicas": 4,
"cpus_each": "0.25",
"memory_each": "256M",
"cap": "strict",
"pool_max": 4,
"instances_seen": 4,
"successful_rps": 136.692,
"failed_requests": 0,
"successful_requests": 1666,
"latency_ms": {
"p50": 46.0,
"p95": 188.0,
"p99": 281.0
},
"cpu_api_mean": 18.74,
"cpu_postgres_mean": 15.16,
"cpu_lb_mean": 8.94,
"load_exit": 0
},
{
"series": "fixed-off",
"replicas": 1,
"cpus_each": "1.0",
"memory_each": "1024M",
"cap": "off",
"pool_max": 4,
"instances_seen": 1,
"successful_rps": 184.701,
"failed_requests": 0,
"successful_requests": 2277,
"latency_ms": {
"p50": 32.0,
"p95": 93.0,
"p99": 109.0
},
"cpu_api_mean": 80.13,
"cpu_postgres_mean": 15.36,
"cpu_lb_mean": 8.82,
"load_exit": 0
},
{
"series": "fixed-off",
"replicas": 2,
"cpus_each": "0.50",
"memory_each": "512M",
"cap": "off",
"pool_max": 4,
"instances_seen": 2,
"successful_rps": 165.162,
"failed_requests": 0,
"successful_requests": 2013,
"latency_ms": {
"p50": 31.0,
"p95": 125.0,
"p99": 203.0
},
"cpu_api_mean": 36.09,
"cpu_postgres_mean": 15.01,
"cpu_lb_mean": 10.63,
"load_exit": 0
},
{
"series": "fixed-off",
"replicas": 4,
"cpus_each": "0.25",
"memory_each": "256M",
"cap": "off",
"pool_max": 4,
"instances_seen": 4,
"successful_rps": 109.204,
"failed_requests": 0,
"successful_requests": 1336,
"latency_ms": {
"p50": 31.0,
"p95": 281.0,
"p99": 329.0
},
"cpu_api_mean": 16.83,
"cpu_postgres_mean": 11.07,
"cpu_lb_mean": 7.92,
"load_exit": 0
},
{
"series": "grow-off",
"replicas": 1,
"cpus_each": "0.50",
"memory_each": "256M",
"cap": "off",
"pool_max": 4,
"instances_seen": 1,
"successful_rps": 84.703,
"failed_requests": 0,
"successful_requests": 1031,
"latency_ms": {
"p50": 94.0,
"p95": 203.0,
"p99": 219.0
},
"cpu_api_mean": 42.47,
"cpu_postgres_mean": 7.93,
"cpu_lb_mean": 4.44,
"load_exit": 0
},
{
"series": "grow-off",
"replicas": 2,
"cpus_each": "0.50",
"memory_each": "256M",
"cap": "off",
"pool_max": 4,
"instances_seen": 2,
"successful_rps": 147.996,
"failed_requests": 0,
"successful_requests": 1806,
"latency_ms": {
"p50": 32.0,
"p95": 141.0,
"p99": 203.0
},
"cpu_api_mean": 31.2,
"cpu_postgres_mean": 15.35,
"cpu_lb_mean": 9.64,
"load_exit": 0
},
{
"series": "grow-off",
"replicas": 4,
"cpus_each": "0.50",
"memory_each": "256M",
"cap": "off",
"pool_max": 4,
"instances_seen": 4,
"successful_rps": 329.198,
"failed_requests": 0,
"successful_requests": 4007,
"latency_ms": {
"p50": 16.0,
"p95": 78.0,
"p99": 125.0
},
"cpu_api_mean": 34.41,
"cpu_postgres_mean": 34.01,
"cpu_lb_mean": 30.12,
"load_exit": 0
},
{
"series": "pool-16",
"replicas": 4,
"cpus_each": "0.50",
"memory_each": "256M",
"cap": "off",
"pool_max": 16,
"instances_seen": 4,
"successful_rps": 319.011,
"failed_requests": 0,
"successful_requests": 3883,
"latency_ms": {
"p50": 16.0,
"p95": 78.0,
"p99": 171.0
},
"cpu_api_mean": 30.75,
"cpu_postgres_mean": 31.35,
"cpu_lb_mean": 21.05,
"load_exit": 0
}
]
}
+39
View File
@@ -0,0 +1,39 @@
<svg xmlns="http://www.w3.org/2000/svg" width="920" height="420">
<rect width="100%" height="100%" fill="#f7f4ee"/>
<text x="24" y="32" font-family="sans-serif" font-size="18">Successful RPS, closed loop, 12s, concurrency 8</text>
<text x="48" y="58" font-family="sans-serif" font-size="12">fixed CPU, strict cap</text>
<rect x="48" y="206" width="28" height="134" fill="#8c4a3a"/>
<text x="48" y="202" font-family="sans-serif" font-size="11">158.3</text>
<text x="48" y="360" font-family="sans-serif" font-size="11">x1</text>
<rect x="90" y="192" width="28" height="148" fill="#8c4a3a"/>
<text x="90" y="188" font-family="sans-serif" font-size="11">174.8</text>
<text x="90" y="360" font-family="sans-serif" font-size="11">x2</text>
<rect x="132" y="224" width="28" height="116" fill="#8c4a3a"/>
<text x="132" y="220" font-family="sans-serif" font-size="11">136.7</text>
<text x="132" y="360" font-family="sans-serif" font-size="11">x4</text>
<text x="210" y="58" font-family="sans-serif" font-size="12">fixed CPU, cap off</text>
<rect x="210" y="183" width="28" height="157" fill="#2f6f4e"/>
<text x="210" y="179" font-family="sans-serif" font-size="11">184.7</text>
<text x="210" y="360" font-family="sans-serif" font-size="11">x1</text>
<rect x="252" y="200" width="28" height="140" fill="#2f6f4e"/>
<text x="252" y="196" font-family="sans-serif" font-size="11">165.2</text>
<text x="252" y="360" font-family="sans-serif" font-size="11">x2</text>
<rect x="294" y="248" width="28" height="92" fill="#2f6f4e"/>
<text x="294" y="244" font-family="sans-serif" font-size="11">109.2</text>
<text x="294" y="360" font-family="sans-serif" font-size="11">x4</text>
<text x="372" y="58" font-family="sans-serif" font-size="12">CPU grows with replicas</text>
<rect x="372" y="268" width="28" height="72" fill="#24577a"/>
<text x="372" y="264" font-family="sans-serif" font-size="11">84.7</text>
<text x="372" y="360" font-family="sans-serif" font-size="11">x1</text>
<rect x="414" y="215" width="28" height="125" fill="#24577a"/>
<text x="414" y="211" font-family="sans-serif" font-size="11">148.0</text>
<text x="414" y="360" font-family="sans-serif" font-size="11">x2</text>
<rect x="456" y="60" width="28" height="280" fill="#24577a"/>
<text x="456" y="56" font-family="sans-serif" font-size="11">329.2</text>
<text x="456" y="360" font-family="sans-serif" font-size="11">x4</text>
<text x="534" y="58" font-family="sans-serif" font-size="12">pool 16</text>
<rect x="534" y="69" width="28" height="271" fill="#a56b12"/>
<text x="534" y="65" font-family="sans-serif" font-size="11">319.0</text>
<text x="534" y="360" font-family="sans-serif" font-size="11">x4</text>
<text x="24" y="400" font-family="sans-serif" font-size="12">xN is the API replica count. Fixed series keeps 1.0 CPU and 1024 MiB in total.</text>
</svg>

After

Width:  |  Height:  |  Size: 2.7 KiB

+54
View File
@@ -0,0 +1,54 @@
global
maxconn 256
defaults
mode http
timeout connect 500ms
timeout client 15s
timeout server 10s
timeout http-request 10s
timeout http-keep-alive 2s
timeout check 1s
timeout queue 1s
retries 1
option redispatch
option dontlognull
resolvers docker
nameserver dns 127.0.0.11:53
accepted_payload_size 8192
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold other 1s
hold refused 1s
hold nx 1s
hold timeout 1s
frontend web
bind *:8080
acl message_post method POST
acl message_path path_reg ^/api/v1/chats/[0-9A-Fa-f-]{36}/messages$
use_backend api_messages if message_post message_path
default_backend api
backend api
balance roundrobin
option httpchk
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr host api
http-check expect status 200
# 503 is returned before a commit. A lost 201 is not retried: that would duplicate chats.
retry-on conn-failure 503
default-server on-marked-down shutdown-sessions
server-template api- 8 api:8080 check resolvers docker init-addr none inter 1s fall 2 rise 1 resolve-prefer ipv4
backend api_messages
balance roundrobin
option httpchk
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr host api
http-check expect status 200
# Message POST is safe to repeat: the idempotency key is shared by every replica.
retry-on conn-failure empty-response 502 503
default-server on-marked-down shutdown-sessions
server-template api- 8 api:8080 check resolvers docker init-addr none inter 1s fall 2 rise 1 resolve-prefer ipv4
+79
View File
@@ -0,0 +1,79 @@
-- Tables, constraints and indexes. Queries live in src/db.py and follow this layout.
-- Applied once by the migrate service, not by each API replica.
--
-- users one row per account; username is unique; password_hash is Argon2id, never returned
-- chats one row per chat
-- chat_members membership; primary key (chat_id, user_id); member_ord keeps the request order
-- a chat and its members are inserted in one transaction
-- messages history of one chat; page query is
-- WHERE chat_id = $1 AND (created_at, id) > ($2, $3)
-- ORDER BY created_at, id
-- LIMIT $4
-- idempotency_keys one successful POST /messages per (user, method, path, key)
-- the message row and this row commit in the same transaction
--
-- Transaction boundaries (see src/db.py):
-- register: single INSERT; unique username aborts that insert
-- create chat: INSERT chats + INSERT chat_members, one transaction; any failure rolls both back
-- send message: membership check + INSERT message + INSERT idempotency_keys, one transaction
-- an advisory lock on that key scope serialises concurrent retries
-- reads: one transaction, no write
-- Sessions are not stored here. Redis is the only session store (see compose.yaml).
-- Re-running these statements does not insert rows. The runner also records schema_migrations.
CREATE TABLE IF NOT EXISTS users (
id uuid PRIMARY KEY,
username text NOT NULL,
display_name text NOT NULL,
password_hash text NOT NULL,
created_at timestamptz NOT NULL,
CONSTRAINT users_username_unique UNIQUE (username),
CONSTRAINT users_username_format CHECK (username ~ '^[a-z0-9_]{3,32}$'),
CONSTRAINT users_display_name_len CHECK (char_length(display_name) BETWEEN 1 AND 100),
CONSTRAINT users_password_hash_present CHECK (char_length(password_hash) > 0)
);
CREATE TABLE IF NOT EXISTS chats (
id uuid PRIMARY KEY,
title text NOT NULL,
created_at timestamptz NOT NULL,
CONSTRAINT chats_title_len CHECK (char_length(title) BETWEEN 1 AND 100)
);
CREATE TABLE IF NOT EXISTS chat_members (
chat_id uuid NOT NULL REFERENCES chats (id) ON DELETE CASCADE,
user_id uuid NOT NULL REFERENCES users (id),
member_ord integer NOT NULL,
PRIMARY KEY (chat_id, user_id),
CONSTRAINT chat_members_ord_unique UNIQUE (chat_id, member_ord),
CONSTRAINT chat_members_ord_nonneg CHECK (member_ord >= 0)
);
-- List "chats of this user" filters by user_id. The primary key leads with chat_id, so it does not serve that lookup.
CREATE INDEX IF NOT EXISTS chat_members_user_idx ON chat_members (user_id);
CREATE TABLE IF NOT EXISTS messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats (id) ON DELETE CASCADE,
sender_id uuid NOT NULL REFERENCES users (id),
body text NOT NULL,
created_at timestamptz NOT NULL,
CONSTRAINT messages_body_len CHECK (char_length(body) BETWEEN 1 AND 4000)
);
-- Matches the history query above: the scan stays inside one chat and already has the page order,
-- so LIMIT stops after one page instead of sorting the whole chat.
CREATE INDEX IF NOT EXISTS messages_chat_history_idx ON messages (chat_id, created_at, id);
CREATE TABLE IF NOT EXISTS idempotency_keys (
user_id uuid NOT NULL REFERENCES users (id),
method text NOT NULL,
path text NOT NULL,
idem_key text NOT NULL,
request_hash text NOT NULL,
message_id uuid NOT NULL REFERENCES messages (id),
created_at timestamptz NOT NULL,
PRIMARY KEY (user_id, method, path, idem_key),
CONSTRAINT idempotency_key_format CHECK (idem_key ~ '^[A-Za-z0-9._:-]{1,128}$'),
CONSTRAINT idempotency_hash_len CHECK (char_length(request_hash) = 64)
);
+3
View File
@@ -0,0 +1,3 @@
argon2-cffi==23.1.0
psycopg[binary,pool]==3.2.9
redis==5.2.1
+3
View File
@@ -0,0 +1,3 @@
@echo off
powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0restart.ps1"
exit /b %ERRORLEVEL%
+14
View File
@@ -0,0 +1,14 @@
$ErrorActionPreference = 'Stop'
Set-Location (Split-Path $PSScriptRoot -Parent)
docker compose --env-file .env restart postgres redis
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
$deadline = (Get-Date).AddSeconds(60)
do {
$status = docker compose --env-file .env ps --format '{{.Service}} {{.Health}}'
if ($status -match 'postgres healthy' -and $status -match 'redis healthy') { break }
Start-Sleep -Seconds 1
} while ((Get-Date) -lt $deadline)
docker compose --env-file .env restart api
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
docker compose --env-file .env restart lb
exit $LASTEXITCODE
+18
View File
@@ -0,0 +1,18 @@
#!/bin/sh
# Restart the API and every store it uses. Volumes stay in place.
# Does not seed, flush, or run compose down -v.
set -eu
ROOT=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
cd "$ROOT"
docker compose --env-file .env restart postgres redis
i=0
while [ "$i" -lt 60 ]; do
status=$(docker compose --env-file .env ps --format '{{.Service}} {{.Health}}')
if printf '%s\n' "$status" | grep -q 'postgres healthy' && printf '%s\n' "$status" | grep -q 'redis healthy'; then
break
fi
i=$((i + 1))
sleep 1
done
docker compose --env-file .env restart api
docker compose --env-file .env restart lb
+16
View File
@@ -0,0 +1,16 @@
$ErrorActionPreference = 'Stop'
Set-Location (Split-Path $PSScriptRoot -Parent)
$ids = @(docker compose --env-file .env ps -q api | Where-Object { $_.Trim() -ne '' })
if ($ids.Count -lt 2) {
Write-Error 'need at least 2 running api replicas'
exit 1
}
foreach ($raw in $ids) {
$cid = "$raw".Trim()
docker stop -t 6 $cid | Out-Null
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
docker rm $cid | Out-Null
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
docker compose --env-file .env up -d --no-deps --no-recreate --scale "api=$($ids.Count)" api
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
}
+18
View File
@@ -0,0 +1,18 @@
#!/bin/sh
# Replace API containers one at a time. The other replica keeps serving.
# Volumes and the load balancer stay up. Replica count is unchanged.
set -eu
ROOT=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
cd "$ROOT"
ids=$(docker compose --env-file .env ps -q api)
count=$(printf '%s\n' "$ids" | grep -c .)
if [ "$count" -lt 2 ]; then
echo "need at least 2 running api replicas" >&2
exit 1
fi
printf '%s\n' "$ids" | while IFS= read -r cid; do
[ -n "$cid" ] || continue
docker stop -t 6 "$cid" >/dev/null
docker rm "$cid" >/dev/null
docker compose --env-file .env up -d --no-deps --no-recreate --scale "api=$count" api
done
+3
View File
@@ -0,0 +1,3 @@
@echo off
powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0stop-one.ps1"
exit /b %ERRORLEVEL%
+25
View File
@@ -0,0 +1,25 @@
$ErrorActionPreference = 'Stop'
if (-not $env:TARGET_INSTANCE_ID) {
Write-Error 'TARGET_INSTANCE_ID is required'
exit 1
}
Set-Location (Split-Path $PSScriptRoot -Parent)
$ids = @(docker compose --env-file .env ps -q api)
if (-not $ids) {
Write-Error 'no running api containers'
exit 1
}
foreach ($raw in $ids) {
$cid = "$raw".Trim()
if (-not $cid) { continue }
$hostName = "$((docker inspect -f '{{.Config.Hostname}}' $cid))".Trim()
$full = "$((docker inspect -f '{{.Id}}' $cid))".Trim()
$name = "$((docker inspect -f '{{.Name}}' $cid))".Trim().TrimStart('/')
$short = $full.Substring(0, [Math]::Min(12, $full.Length))
if ($hostName -eq $env:TARGET_INSTANCE_ID -or $short -eq $env:TARGET_INSTANCE_ID -or $name -eq $env:TARGET_INSTANCE_ID) {
docker stop -t 6 $cid | Out-Null
exit $LASTEXITCODE
}
}
Write-Error 'no running api container matches the instance'
exit 1
+22
View File
@@ -0,0 +1,22 @@
#!/bin/sh
# Stop the API replica named by TARGET_INSTANCE_ID and leave it stopped.
# Does not stop the database, Redis, the load balancer, or any volume.
set -eu
: "${TARGET_INSTANCE_ID:?test provides target API instance alias}"
ROOT=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
cd "$ROOT"
found=""
for cid in $(docker compose --env-file .env ps -q api); do
host=$(docker inspect -f '{{.Config.Hostname}}' "$cid")
short=$(docker inspect -f '{{.Id}}' "$cid" | cut -c1-12)
name=$(docker inspect -f '{{.Name}}' "$cid" | sed 's#^/##')
if [ "$host" = "$TARGET_INSTANCE_ID" ] || [ "$short" = "$TARGET_INSTANCE_ID" ] || [ "$name" = "$TARGET_INSTANCE_ID" ]; then
found=$cid
break
fi
done
if [ -z "$found" ]; then
echo "no running api container matches the instance" >&2
exit 1
fi
docker stop -t 6 "$found" >/dev/null
+810
View File
@@ -0,0 +1,810 @@
import hashlib
import json
import os
import re
import signal
import socket
import sys
import threading
import time
import uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlsplit
import base64
import cursor
import db
import passwords
import sessions
from errors import DependencyDown
MAX_BODY = 65536
DEFAULT_INFLIGHT_BODIES = 32
USERNAME = re.compile(r'^[a-z0-9_]{3,32}$')
BASIC_CHALLENGE = 'Basic realm="messenger", charset="UTF-8"'
BEARER_CHALLENGE = 'Bearer realm="messenger"'
IDEMPOTENCY_KEY = re.compile(r'^[A-Za-z0-9._:-]{1,128}$')
INSTANCE_ID = 'unset'
idempotency_ttl = 86400
class State:
def __init__(self, max_inflight_bodies, drain_seconds):
self.accepting = True
self.drain_seconds = drain_seconds
slots = int(max_inflight_bodies)
if slots < 1:
slots = 1
self.body_slots = threading.BoundedSemaphore(slots)
class Limiter:
def __init__(self, max_per_second):
self.max_per_second = max_per_second
self._lock = threading.Lock()
self._recent = []
def allow(self):
if self.max_per_second <= 0:
return True
now = time.monotonic()
with self._lock:
self._recent = [stamp for stamp in self._recent if now - stamp < 1]
if len(self._recent) >= self.max_per_second:
return False
self._recent.append(now)
return True
def canonical_uuid(value):
if not isinstance(value, str):
return None
try:
return str(uuid.UUID(value))
except ValueError:
return None
def make_server(state, limiter, host, port):
class Handler(BaseHTTPRequestHandler):
def address_string(self):
return self.client_address[0]
def log_message(self, fmt, *args):
return
def do_GET(self):
self._handle()
def do_POST(self):
self._handle()
def do_PUT(self):
self._handle()
def do_PATCH(self):
self._handle()
def do_DELETE(self):
self._handle()
def _handle(self):
self.request_id = str(uuid.uuid4())
self._status = '-'
try:
self.connection.settimeout(15)
except OSError:
pass
try:
self._route()
except DependencyDown as exc:
print('dependency %s' % type(exc).__name__, flush=True)
if self._status == '-':
self._error(503, 'unavailable', 'Сервис временно недоступен')
except Exception as exc:
print('internal %s' % type(exc).__name__, flush=True)
if self._status == '-':
self._error(500, 'internal_error', 'Внутренняя ошибка')
def _route(self):
parsed = urlsplit(self.path)
path = parsed.path
query = parse_qs(parsed.query, keep_blank_values=True)
if path == '/health/live' and self.command == 'GET':
self._json(200, {'status': 'ok'})
return
if not state.accepting:
self._error(503, 'unavailable', 'Сервер останавливается')
return
if path == '/health/ready' and self.command == 'GET':
self._ready()
return
if path == '/api/v1/users' and self.command == 'POST':
self._register()
return
if path == '/api/v1/auth/sessions' and self.command == 'POST':
self._login()
return
if path.startswith('/api/'):
self._api(path, query)
return
if path in ('/health/live', '/health/ready', '/api/v1/users', '/api/v1/auth/sessions'):
self._error(400, 'invalid_request', 'Метод не поддерживается')
return
self._error(404, 'not_found', 'Объект не найден')
def _ready(self):
try:
db.ping()
sessions.ping()
db.schema_ready()
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
self._json(200, {'status': 'ok'})
def _api(self, path, query):
if not limiter.allow():
self._discard_body()
self._error(429, 'rate_limited', 'Слишком много запросов', retry_after=1)
return
user = self._user()
if user is None:
return
if path == '/api/v1/auth/sessions/current' and self.command == 'DELETE':
self._logout()
return
if path == '/api/v1/users/me' and self.command == 'GET':
self._json(200, user)
return
if path == '/api/v1/chats' and self.command == 'POST':
self._create_chat(user)
return
if path == '/api/v1/chats' and self.command == 'GET':
self._list_chats(user, query)
return
chat_match = re.fullmatch(r'/api/v1/chats/([^/]+)', path)
if chat_match and self.command == 'GET':
self._get_chat(user, chat_match.group(1))
return
msg_match = re.fullmatch(r'/api/v1/chats/([^/]+)/messages', path)
if msg_match and self.command == 'POST':
self._create_message(user, msg_match.group(1))
return
if msg_match and self.command == 'GET':
self._list_messages(user, msg_match.group(1), query)
return
self._error(404, 'not_found', 'Объект не найден')
def _user(self):
# X-User-Id is ignored from lab2. Basic is only valid on POST /auth/sessions.
scheme, value = _authorization(self.headers.get('Authorization'))
if scheme != 'bearer' or not sessions.token_shape(value):
self._discard_body()
self._error(401, 'unauthorized', 'Пользователь не опознан', challenge=BEARER_CHALLENGE)
return None
try:
user_id = sessions.read(value)
if user_id is None:
self._discard_body()
self._error(401, 'unauthorized', 'Пользователь не опознан', challenge=BEARER_CHALLENGE)
return None
user = db.get_user(user_id)
except DependencyDown:
self._discard_body()
self._error(503, 'unavailable', 'Сервис временно недоступен')
return None
if user is None:
self._discard_body()
self._error(401, 'unauthorized', 'Пользователь не опознан', challenge=BEARER_CHALLENGE)
return None
self._bearer = value
return user
def _logout(self):
self._discard_body()
if self._status != '-':
return
try:
gone = sessions.revoke(getattr(self, '_bearer', ''))
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
if not gone:
self._error(401, 'unauthorized', 'Пользователь не опознан', challenge=BEARER_CHALLENGE)
return
self._json(204, None)
def _login(self):
if not limiter.allow():
self._discard_body()
self._error(429, 'rate_limited', 'Слишком много запросов', retry_after=1)
return
raw_body, rejected = self._optional_empty_body()
if rejected:
return
if raw_body:
self._error(400, 'invalid_request', 'Тело запроса не нужно')
return
username, password = _basic_credentials(self.headers.get('Authorization'))
if username is None:
self._error(401, 'unauthorized', 'Пользователь не опознан', challenge=BASIC_CHALLENGE)
return
try:
found = db.get_credentials(username)
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
if found is None:
passwords.verify_password(passwords.DUMMY_HASH, password)
self._error(401, 'unauthorized', 'Пользователь не опознан', challenge=BASIC_CHALLENGE)
return
user, password_hash = found
if not passwords.verify_password(password_hash, password):
self._error(401, 'unauthorized', 'Пользователь не опознан', challenge=BASIC_CHALLENGE)
return
try:
token, expires_at = sessions.create(user['id'])
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
self._json(201, {
'token': token,
'token_type': 'Bearer',
'expires_at': expires_at,
'user': user,
})
def _discard_body(self):
length, problem = _parse_content_length(self.headers.get('Content-Length'))
if problem == 'missing':
return
if problem == 'too_big':
self._reject_too_large()
return
if problem == 'bad':
self.close_connection = True
return
if length == 0:
return
data, busy = self._read_exact(length)
if busy or data is None or len(data) != length:
self.close_connection = True
def _optional_empty_body(self):
length, problem = _parse_content_length(self.headers.get('Content-Length'))
if problem == 'missing' or length == 0:
return b'', False
if problem == 'too_big':
self._reject_too_large()
self._error(413, 'payload_too_large', 'Тело запроса слишком большое')
return None, True
if problem == 'bad':
self.close_connection = True
self._error(400, 'invalid_request', 'Некорректный запрос')
return None, True
raw, busy = self._read_exact(length)
if busy:
self._error(503, 'unavailable', 'Слишком много одновременных тел запроса')
return None, True
if raw is None or len(raw) != length:
self.close_connection = True
self._error(400, 'invalid_request', 'Некорректный запрос')
return None, True
return raw.strip(), False
def _read_exact(self, length):
if length == 0:
return b'', False
if not state.body_slots.acquire(blocking=False):
self.close_connection = True
return None, True
try:
return self.rfile.read(length), False
finally:
state.body_slots.release()
def _reject_too_large(self):
self.close_connection = True
if not state.body_slots.acquire(blocking=False):
return
previous = self.connection.gettimeout()
self.connection.settimeout(0.2)
try:
remaining = MAX_BODY
while remaining > 0:
try:
chunk = self.rfile.read(min(8192, remaining))
except (TimeoutError, socket.timeout, OSError):
break
if not chunk:
break
remaining -= len(chunk)
finally:
state.body_slots.release()
try:
self.connection.settimeout(previous)
except OSError:
pass
def _register(self):
if not limiter.allow():
self._discard_body()
self._error(429, 'rate_limited', 'Слишком много запросов', retry_after=1)
return
body, problem = self._json_body({'username', 'display_name', 'password'})
if problem:
return
username = body['username']
display_name = body['display_name']
password = body['password']
if not isinstance(username, str) or USERNAME.fullmatch(username) is None:
self._error(400, 'invalid_request', 'Некорректный username')
return
if not _text_ok(display_name, 100):
self._error(400, 'invalid_request', 'Некорректный display_name')
return
if not _password_ok(password):
self._error(400, 'invalid_request', 'Некорректный пароль')
return
try:
status, user = db.create_user(username, display_name, passwords.hash_password(password))
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
if status == 'conflict':
self._error(409, 'conflict', 'Такой username уже занят')
return
if status != 'ok':
self._error(400, 'invalid_request', 'Некорректные поля JSON')
return
self._json(201, user)
def _create_chat(self, user):
body, problem = self._json_body({'title', 'member_ids'})
if problem:
return
title = body['title']
member_ids = body['member_ids']
if not _text_ok(title, 100):
self._error(400, 'invalid_request', 'Некорректный title')
return
parsed_ids, ok = _member_ids(member_ids, user['id'])
if not ok:
self._error(400, 'invalid_request', 'Некорректный список участников')
return
try:
status, chat = db.create_chat(user['id'], title, parsed_ids)
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
if status != 'ok':
self._error(400, 'invalid_request', 'Некорректный список участников')
return
self._json(201, chat)
def _list_chats(self, user, query):
limit, page_cursor, problem = _page_args(query, 'chats', user['id'])
if problem:
self._error(400, 'invalid_request', problem)
return
try:
items, nxt = db.list_chats(user['id'], limit, page_cursor)
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
self._json(200, {'items': items, 'next_cursor': nxt})
def _get_chat(self, user, raw_chat_id):
chat_id = canonical_uuid(raw_chat_id)
if chat_id is None:
self._error(400, 'invalid_request', 'Некорректный UUID')
return
try:
chat = db.get_chat(chat_id, user['id'])
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
if chat is None:
self._error(404, 'not_found', 'Объект не найден')
return
self._json(200, chat)
def _create_message(self, user, raw_chat_id):
chat_id = canonical_uuid(raw_chat_id)
if chat_id is None:
self._discard_body()
self._error(400, 'invalid_request', 'Некорректный UUID')
return
idem_key = _idempotency_key(self.headers)
if idem_key is None:
self._discard_body()
self._error(400, 'invalid_request', 'Нужен корректный Idempotency-Key')
return
body, problem = self._json_body({'text'})
if problem:
return
text = body['text']
if not _text_ok(text, 4000):
self._error(400, 'invalid_request', 'Некорректный текст сообщения')
return
try:
status, message = db.add_message(
chat_id,
user['id'],
text,
'POST',
'/api/v1/chats/%s/messages' % chat_id,
idem_key,
_message_hash(text),
idempotency_ttl,
)
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
if status == 'missing':
self._error(404, 'not_found', 'Объект не найден')
return
if status == 'conflict':
self._error(409, 'conflict', 'Этот ключ уже использован с другим сообщением')
return
if status == 'full':
self._error(503, 'unavailable', 'Достигнут предел числа сообщений')
return
if status not in ('ok', 'replay'):
self._error(500, 'internal_error', 'Внутренняя ошибка')
return
self._json(201, message)
def _list_messages(self, user, raw_chat_id, query):
chat_id = canonical_uuid(raw_chat_id)
if chat_id is None:
self._error(400, 'invalid_request', 'Некорректный UUID')
return
limit, page_cursor, problem = _page_args(query, 'msgs', chat_id)
if problem:
self._error(400, 'invalid_request', problem)
return
try:
status, items, nxt = db.list_messages(chat_id, user['id'], limit, page_cursor)
except DependencyDown:
self._error(503, 'unavailable', 'Сервис временно недоступен')
return
if status == 'missing':
self._error(404, 'not_found', 'Объект не найден')
return
self._json(200, {'items': items, 'next_cursor': nxt})
def _json_body(self, fields):
raw, code, message = self._read_body()
if code:
self._error(code, _error_code(code), message)
return None, True
try:
body = json.loads(raw.decode('utf-8'))
except (UnicodeError, json.JSONDecodeError):
self._error(400, 'invalid_request', 'Некорректный JSON')
return None, True
if not isinstance(body, dict) or set(body) != fields:
self._error(400, 'invalid_request', 'Некорректные поля JSON')
return None, True
return body, False
def _read_body(self):
length, problem = _parse_content_length(self.headers.get('Content-Length'))
if problem == 'missing' or problem == 'bad':
self.close_connection = True
return None, 400, 'Некорректный JSON'
if problem == 'too_big':
self._reject_too_large()
return None, 413, 'Тело запроса слишком большое'
if not _json_content_type(self.headers.get('Content-Type', '')):
self._read_exact(length)
return None, 415, 'Нужен Content-Type application/json'
raw, busy = self._read_exact(length)
if busy:
return None, 503, 'Слишком много одновременных тел запроса'
if raw is None or len(raw) != length:
self.close_connection = True
return None, 400, 'Некорректный JSON'
return raw, None, None
def _error(self, status, code, message, retry_after=None, challenge=None):
self._json(status, {
'error': {
'code': code,
'message': message,
'request_id': self.request_id,
}
}, retry_after=retry_after, challenge=challenge)
def _json(self, status, payload, retry_after=None, challenge=None):
if self._status != '-':
return
if status == 204:
body = b''
else:
body = json.dumps(payload, ensure_ascii=False).encode('utf-8')
self.send_response(status)
if status != 204:
self.send_header('Content-Type', 'application/json; charset=utf-8')
self.send_header('Content-Length', str(len(body)))
self.send_header('X-Request-Id', self.request_id)
self.send_header('X-Instance-Id', INSTANCE_ID)
if challenge is not None:
self.send_header('WWW-Authenticate', challenge)
if retry_after is not None:
self.send_header('Retry-After', str(retry_after))
self.end_headers()
if body:
self.wfile.write(body)
self._status = status
path = urlsplit(self.path).path
print('%s %s %s %s' % (self.request_id, self.command, path, status), flush=True)
class Server(ThreadingHTTPServer):
allow_reuse_address = True
# In-flight handlers finish after SIGTERM. A stuck handler is cut by stop_grace_period.
daemon_threads = False
return Server((host, port), Handler)
def _idempotency_key(headers):
raw = headers.get('Idempotency-Key')
if raw is None or IDEMPOTENCY_KEY.fullmatch(raw) is None:
return None
return raw
def _message_hash(text):
# Missing attachment_ids and [] are the same message. Lab3 accepts only text;
# the empty list keeps that rule ready for a later lab without changing this hash.
canonical = json.dumps(
{'attachment_ids': [], 'text': text},
ensure_ascii=False,
sort_keys=True,
separators=(',', ':'),
)
return hashlib.sha256(canonical.encode('utf-8')).hexdigest()
def _error_code(status):
return {
400: 'invalid_request',
413: 'payload_too_large',
415: 'unsupported_media_type',
503: 'unavailable',
}.get(status, 'invalid_request')
def _json_content_type(value):
if not value:
return False
media = value.split(';', 1)[0].strip().lower()
return media == 'application/json'
def _parse_content_length(raw):
if raw is None:
return None, 'missing'
text = raw.strip()
if not re.fullmatch(r'[0-9]+', text):
return None, 'bad'
digits = text.lstrip('0') or '0'
if len(digits) > 7:
return None, 'too_big'
try:
length = int(digits)
except ValueError:
return None, 'too_big'
if length > MAX_BODY:
return length, 'too_big'
return length, None
def _text_ok(value, limit):
return (
isinstance(value, str)
and 1 <= len(value) <= limit
and value.strip() != ''
)
def _password_ok(value):
return (
isinstance(value, str)
and 12 <= len(value) <= 128
and value.strip() != ''
)
def _member_ids(value, owner_id):
if not isinstance(value, list) or not 1 <= len(value) <= 100:
return None, False
parsed = []
for item in value:
item_id = canonical_uuid(item)
if item_id is None:
return None, False
parsed.append(item_id)
if len(set(parsed)) != len(parsed):
return None, False
if owner_id not in parsed:
return None, False
return parsed, True
def _page_args(query, kind, owner_id):
if 'limit' not in query:
limit = 50
else:
raw = query['limit']
if len(raw) != 1 or re.fullmatch(r'[0-9]+', raw[0]) is None:
return None, None, 'Некорректный limit'
digits = raw[0].lstrip('0') or '0'
if len(digits) > 3:
return None, None, 'Некорректный limit'
try:
limit = int(digits)
except ValueError:
return None, None, 'Некорректный limit'
if limit < 1 or limit > 100:
return None, None, 'Некорректный limit'
if 'cursor' not in query:
return limit, None, None
raw_cursor = query['cursor']
if len(raw_cursor) != 1 or raw_cursor[0] == '':
return None, None, 'Некорректный cursor'
page_cursor = cursor.read_cursor(raw_cursor[0], kind, owner_id)
if page_cursor is None:
return None, None, 'Некорректный cursor'
return limit, page_cursor, None
def _authorization(header):
if not header:
return None, None
parts = header.split(None, 1)
if len(parts) != 2:
return None, None
return parts[0].lower(), parts[1].strip()
def _basic_credentials(header):
scheme, value = _authorization(header)
if scheme != 'basic' or not value:
return None, None
try:
raw = base64.b64decode(value, validate=True)
text = raw.decode('utf-8')
except (ValueError, UnicodeError):
return None, None
if ':' not in text:
return None, None
username, password = text.split(':', 1)
if USERNAME.fullmatch(username) is None or password == '':
return None, None
return username, password
def _positive_int(name, default):
raw = os.environ.get(name, str(default))
try:
value = int(raw)
except ValueError:
print('bad integer %s' % name, flush=True)
return None
return value
def wait_until_ready(stop):
deadline = time.monotonic() + 75
while not stop['flag']:
try:
db.open_pool()
sessions.open_client()
sessions.ping()
db.ping()
db.schema_ready()
print('schema ready', flush=True)
return True
except RuntimeError:
print('configuration error', flush=True)
return False
except Exception as exc:
print('waiting for postgres/redis: %s' % type(exc).__name__, flush=True)
db.close_pool()
sessions.close_client()
if time.monotonic() > deadline:
print('dependencies not ready', flush=True)
return False
time.sleep(1)
return False
def main():
global INSTANCE_ID, idempotency_ttl
host = os.environ.get('APP_HOST', '0.0.0.0')
port = _positive_int('APP_PORT', 8080)
message_cap = _positive_int('MAX_MESSAGES', 100000)
max_per_second = _positive_int('MAX_PER_SECOND', 10000)
max_inflight_bodies = _positive_int('MAX_INFLIGHT_BODIES', DEFAULT_INFLIGHT_BODIES)
ttl = _positive_int('SESSION_TTL_SECONDS', 3600)
idempotency_ttl = _positive_int('IDEMPOTENCY_TTL_SECONDS', 86400)
drain_seconds = _positive_int('DRAIN_SECONDS', 2)
cap_mode = os.environ.get('CAP_MODE', 'off').strip().lower()
instance = os.environ.get('INSTANCE_ID', '').strip() or socket.gethostname().strip()
if cap_mode not in ('off', 'strict'):
print('bad CAP_MODE', flush=True)
sys.exit(1)
if not instance:
print('empty instance id', flush=True)
sys.exit(1)
if None in (port, message_cap, max_per_second, max_inflight_bodies, ttl, idempotency_ttl, drain_seconds):
sys.exit(1)
if ttl < 1 or idempotency_ttl < 1 or drain_seconds < 0:
sys.exit(1)
INSTANCE_ID = instance
sessions.configure(ttl)
db.configure(message_cap, cap_mode)
stop = {'flag': False}
def stop_waiting(signum, frame):
stop['flag'] = True
print('signal %s during startup' % signum, flush=True)
signal.signal(signal.SIGTERM, stop_waiting)
signal.signal(signal.SIGINT, stop_waiting)
if not wait_until_ready(stop):
db.close_pool()
sessions.close_client()
sys.exit(0 if stop['flag'] else 1)
state = State(max_inflight_bodies, drain_seconds)
limiter = Limiter(max_per_second)
httpd = make_server(state, limiter, host, port)
def stop_serving(signum, frame):
if not state.accepting:
return
state.accepting = False
print('signal %s: stop accepting' % signum, flush=True)
def drain():
# /health/ready is already 503. Wait so the load balancer marks this process down
# before the socket closes.
time.sleep(state.drain_seconds)
httpd.shutdown()
threading.Thread(target=drain, daemon=True).start()
signal.signal(signal.SIGTERM, stop_serving)
signal.signal(signal.SIGINT, stop_serving)
print(
'listen http://%s:%s instance=%s ttl=%s idempotency_ttl=%s cap=%s drain=%s pool=%s-%s'
% (
host,
port,
INSTANCE_ID,
ttl,
idempotency_ttl,
cap_mode,
drain_seconds,
db.POOL_MIN,
db.POOL_MAX,
),
flush=True,
)
try:
httpd.serve_forever()
finally:
httpd.server_close()
db.close_pool()
sessions.close_client()
print('process stopped', flush=True)
if __name__ == '__main__':
main()
+59
View File
@@ -0,0 +1,59 @@
import base64
import re
import uuid
from datetime import datetime
# Same cursor as lab1. The page query itself moved into PostgreSQL.
_CURSOR_TOKEN = re.compile(r'^[A-Za-z0-9_-]+={0,2}$')
def cursor_token(kind, owner_id, created_at, item_id):
raw = '|'.join((kind, owner_id, created_at, item_id))
return base64.urlsafe_b64encode(raw.encode('ascii')).decode('ascii')
def _timestamp_ok(value):
if not isinstance(value, str) or not value.endswith('Z'):
return False
if not 20 <= len(value) <= 32:
return False
for fmt in ('%Y-%m-%dT%H:%M:%S.%fZ', '%Y-%m-%dT%H:%M:%SZ'):
try:
datetime.strptime(value, fmt)
except ValueError:
continue
else:
return True
return False
def _decode_cursor(token):
if len(token) % 4 == 1:
raise ValueError('bad base64 length')
padded = token + '=' * (-len(token) % 4)
try:
return base64.b64decode(padded.encode('ascii'), altchars=b'-_', validate=True)
except TypeError:
return base64.b64decode(padded.encode('ascii'), altchars=b'-_')
def read_cursor(token, kind, owner_id):
if not isinstance(token, str) or _CURSOR_TOKEN.fullmatch(token) is None:
return None
try:
raw = _decode_cursor(token)
canonical = base64.urlsafe_b64encode(raw).decode('ascii')
if canonical.rstrip('=') != token.rstrip('='):
return None
parts = raw.decode('ascii').split('|')
if len(parts) != 4:
return None
got_kind, got_owner, created_at, item_id = parts
if got_kind != kind or got_owner != owner_id:
return None
if not _timestamp_ok(created_at):
return None
item_id = str(uuid.UUID(item_id))
except (ValueError, UnicodeError):
return None
return created_at, item_id
+579
View File
@@ -0,0 +1,579 @@
import hashlib
import os
import threading
import uuid
from datetime import datetime, timedelta, timezone
from pathlib import Path
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool, PoolTimeout
import cursor
from errors import DependencyDown
from timefmt import format_z, parse_z
# Serialises only the strict message cap. Registration stays unlocked so the unique index decides the race.
MESSAGE_LOCK = 814202
# First key of the two-argument advisory lock. The second key is a hash of one idempotency scope.
IDEM_LOCK_CLASS = 814203
def _uuid(value):
if isinstance(value, uuid.UUID):
return value
return uuid.UUID(str(value))
POOL_MIN = 1
POOL_MAX = 8
POOL_TIMEOUT_SECONDS = 5
CONNECT_TIMEOUT_SECONDS = 3
STATEMENT_TIMEOUT_MS = 3000
LOCK_TIMEOUT_MS = 2000
pool = None
max_messages = 100000
cap_mode = 'off'
_clock_lock = threading.Lock()
_last_stamp = None
def database_url():
file_path = os.environ.get('DATABASE_URL_FILE')
if file_path:
return Path(file_path).read_text(encoding='utf-8').strip()
url = os.environ.get('DATABASE_URL', '').strip()
if not url:
raise RuntimeError('DATABASE_URL is not set')
return url
def migrations_dir():
override = os.environ.get('MIGRATIONS_DIR')
if override:
return Path(override)
return Path(__file__).resolve().parents[1] / 'migrations'
def _statements(sql):
lines = []
for line in sql.splitlines():
if line.strip().startswith('--'):
continue
lines.append(line)
parts = []
for part in '\n'.join(lines).split(';'):
statement = part.strip()
if statement:
parts.append(statement)
return parts
def apply_migrations(conninfo=None):
"""Apply pending SQL files. A second call inserts nothing and returns []."""
conninfo = conninfo or database_url()
applied = []
with psycopg.connect(conninfo, connect_timeout=CONNECT_TIMEOUT_SECONDS, row_factory=dict_row) as conn:
conn.execute(
"""
CREATE TABLE IF NOT EXISTS schema_migrations (
version text PRIMARY KEY,
applied_at timestamptz NOT NULL DEFAULT now()
)
"""
)
conn.commit()
done = {row['version'] for row in conn.execute('SELECT version FROM schema_migrations')}
for path in sorted(migrations_dir().glob('*.sql')):
if path.name in done:
continue
sql = path.read_text(encoding='utf-8')
with conn.transaction():
for statement in _statements(sql):
conn.execute(statement)
conn.execute(
'INSERT INTO schema_migrations (version) VALUES (%s)',
(path.name,),
)
applied.append(path.name)
return applied
def pool_sizes():
raw = os.environ.get('POOL_MAX', '4')
try:
maximum = int(raw)
except ValueError:
maximum = 4
if maximum < 1:
maximum = 1
minimum = 1
return minimum, maximum
def open_pool():
global pool, POOL_MIN, POOL_MAX
if pool is not None:
return pool
POOL_MIN, POOL_MAX = pool_sizes()
options = '-c statement_timeout=%s -c lock_timeout=%s -c timezone=UTC -c application_name=messenger' % (
STATEMENT_TIMEOUT_MS,
LOCK_TIMEOUT_MS,
)
pool = ConnectionPool(
database_url(),
min_size=POOL_MIN,
max_size=POOL_MAX,
timeout=POOL_TIMEOUT_SECONDS,
max_waiting=32,
reconnect_timeout=5,
check=ConnectionPool.check_connection,
kwargs={
'connect_timeout': CONNECT_TIMEOUT_SECONDS,
'row_factory': dict_row,
'application_name': 'messenger',
'options': options,
},
open=False,
)
pool.open(wait=True, timeout=10)
return pool
def close_pool():
global pool
if pool is not None:
pool.close()
pool = None
def configure(message_cap, mode='off'):
global max_messages, cap_mode
max_messages = int(message_cap)
cap_mode = mode if mode in ('off', 'strict') else 'off'
def _guard(fn):
try:
return fn()
except (psycopg.OperationalError, psycopg.InterfaceError, PoolTimeout) as exc:
raise DependencyDown(type(exc).__name__) from None
def ping():
def run():
with pool.connection() as conn:
with conn.transaction():
conn.execute('SELECT 1')
_guard(run)
def schema_ready():
"""Ready only after the one-shot migrate service has recorded 001_init.sql."""
def run():
try:
with pool.connection() as conn:
with conn.transaction():
row = conn.execute(
"""
SELECT 1 AS ok
FROM schema_migrations
WHERE version = %s
""",
('001_init.sql',),
).fetchone()
except psycopg.Error as exc:
raise DependencyDown(type(exc).__name__) from None
if row is None:
raise DependencyDown('schema')
return _guard(run)
def stamp():
"""Wall clock, but never goes backwards inside this process.
Two inserts in the same microsecond would make the page order depend on
UUID byte order. Bumping one microsecond keeps (created_at, id) stable.
"""
global _last_stamp
with _clock_lock:
current = datetime.now(timezone.utc)
if _last_stamp is not None and current <= _last_stamp:
current = _last_stamp + timedelta(microseconds=1)
_last_stamp = current
return current
def _user_out(row):
return {
'id': str(row['id']),
'username': row['username'],
'display_name': row['display_name'],
'created_at': format_z(row['created_at']),
}
def _message_out(row):
return {
'id': str(row['id']),
'chat_id': str(row['chat_id']),
'sender_id': str(row['sender_id']),
'text': row['body'],
'created_at': format_z(row['created_at']),
}
def create_user(username, display_name, password_hash):
user_id = str(uuid.uuid4())
created = stamp()
def run():
try:
with pool.connection() as conn:
with conn.transaction():
row = conn.execute(
"""
INSERT INTO users (id, username, display_name, password_hash, created_at)
VALUES (%s, %s, %s, %s, %s)
RETURNING id, username, display_name, created_at
""",
(_uuid(user_id), username, display_name, password_hash, created),
).fetchone()
return 'ok', _user_out(row)
except psycopg.errors.UniqueViolation:
return 'conflict', None
except psycopg.errors.CheckViolation:
return 'bad', None
return _guard(run)
def get_user(user_id):
def run():
with pool.connection() as conn:
with conn.transaction():
row = conn.execute(
"""
SELECT id, username, display_name, created_at
FROM users
WHERE id = %s
""",
(_uuid(user_id),),
).fetchone()
return None if row is None else _user_out(row)
return _guard(run)
def get_credentials(username):
def run():
with pool.connection() as conn:
with conn.transaction():
row = conn.execute(
"""
SELECT id, username, display_name, created_at, password_hash
FROM users
WHERE username = %s
""",
(username,),
).fetchone()
if row is None:
return None
return _user_out(row), row['password_hash']
return _guard(run)
def _members(conn, chat_ids):
if not chat_ids:
return {}
rows = conn.execute(
"""
SELECT chat_id, user_id
FROM chat_members
WHERE chat_id = ANY(%s)
ORDER BY chat_id, member_ord
""",
([_uuid(item) for item in chat_ids],),
).fetchall()
grouped = {}
for row in rows:
grouped.setdefault(str(row['chat_id']), []).append(str(row['user_id']))
return grouped
def _chat_out(row, member_ids):
return {
'id': str(row['id']),
'title': row['title'],
'member_ids': member_ids,
'created_at': format_z(row['created_at']),
}
def create_chat(owner_id, title, member_ids):
chat_id = str(uuid.uuid4())
created = stamp()
def run():
try:
with pool.connection() as conn:
with conn.transaction():
conn.execute(
"""
INSERT INTO chats (id, title, created_at)
VALUES (%s, %s, %s)
""",
(_uuid(chat_id), title, created),
)
for ordinal, member_id in enumerate(member_ids):
conn.execute(
"""
INSERT INTO chat_members (chat_id, user_id, member_ord)
VALUES (%s, %s, %s)
""",
(_uuid(chat_id), _uuid(member_id), ordinal),
)
row = conn.execute(
"""
SELECT id, title, created_at
FROM chats
WHERE id = %s
""",
(_uuid(chat_id),),
).fetchone()
members = _members(conn, [chat_id]).get(chat_id, [])
return 'ok', _chat_out(row, members)
except (psycopg.errors.ForeignKeyViolation, psycopg.errors.UniqueViolation, psycopg.errors.CheckViolation):
return 'bad', None
return _guard(run)
def get_chat(chat_id, user_id):
def run():
with pool.connection() as conn:
with conn.transaction():
row = conn.execute(
"""
SELECT c.id, c.title, c.created_at
FROM chats c
JOIN chat_members me ON me.chat_id = c.id AND me.user_id = %s
WHERE c.id = %s
""",
(_uuid(user_id), _uuid(chat_id)),
).fetchone()
if row is None:
return None
members = _members(conn, [chat_id]).get(chat_id, [])
return _chat_out(row, members)
return _guard(run)
def list_chats(user_id, limit, cursor_pair):
def run():
with pool.connection() as conn:
with conn.transaction():
if cursor_pair is None:
rows = conn.execute(
"""
SELECT c.id, c.title, c.created_at
FROM chats c
JOIN chat_members me ON me.chat_id = c.id AND me.user_id = %s
ORDER BY c.created_at, c.id
LIMIT %s
""",
(_uuid(user_id), limit + 1),
).fetchall()
else:
rows = conn.execute(
"""
SELECT c.id, c.title, c.created_at
FROM chats c
JOIN chat_members me ON me.chat_id = c.id AND me.user_id = %s
WHERE (c.created_at, c.id) > (%s, %s)
ORDER BY c.created_at, c.id
LIMIT %s
""",
(_uuid(user_id), parse_z(cursor_pair[0]), _uuid(cursor_pair[1]), limit + 1),
).fetchall()
members = _members(conn, [row['id'] for row in rows[:limit]])
extra = len(rows) > limit
page = rows[:limit]
items = [_chat_out(row, members.get(str(row['id']), [])) for row in page]
if extra and items:
last = items[-1]
nxt = cursor.cursor_token('chats', user_id, last['created_at'], last['id'])
else:
nxt = None
return items, nxt
return _guard(run)
def _idem_lock_key(user_id, method, path, idem_key):
raw = '\0'.join((user_id, method, path, idem_key)).encode('utf-8')
value = int.from_bytes(hashlib.sha256(raw).digest()[:4], 'big', signed=False)
if value >= 2 ** 31:
value -= 2 ** 32
return value
def add_message(chat_id, sender_id, text, method, path, idem_key, request_hash, ttl_seconds):
"""Insert a message and its idempotency row in one transaction.
The advisory lock is the scope (user, method, path, key), so two replicas
cannot both commit the same key. A repeat checks chat membership before it
compares the body: a removed member gets 404 even when the new text would
otherwise be a 409. A 5xx rolls the transaction back, and the client can
repeat that key. An expired row is removed inside the same lock.
"""
message_id = str(uuid.uuid4())
created = stamp()
def run():
try:
with pool.connection() as conn:
with conn.transaction():
conn.execute(
'SELECT pg_advisory_xact_lock(%s, %s)',
(IDEM_LOCK_CLASS, _idem_lock_key(sender_id, method, path, idem_key)),
)
conn.execute(
"""
DELETE FROM idempotency_keys
WHERE user_id = %s AND method = %s AND path = %s AND idem_key = %s
AND created_at <= now() - (%s::int * interval '1 second')
""",
(_uuid(sender_id), method, path, idem_key, int(ttl_seconds)),
)
existing = conn.execute(
"""
SELECT request_hash, message_id
FROM idempotency_keys
WHERE user_id = %s AND method = %s AND path = %s AND idem_key = %s
""",
(_uuid(sender_id), method, path, idem_key),
).fetchone()
if existing is not None:
member = conn.execute(
"""
SELECT 1 AS ok
FROM chat_members
WHERE chat_id = %s AND user_id = %s
""",
(_uuid(chat_id), _uuid(sender_id)),
).fetchone()
if member is None:
return 'missing', None
if existing['request_hash'] != request_hash:
return 'conflict', None
stored = conn.execute(
"""
SELECT id, chat_id, sender_id, body, created_at
FROM messages
WHERE id = %s
""",
(existing['message_id'],),
).fetchone()
if stored is None:
return 'missing', None
return 'replay', _message_out(stored)
if cap_mode == 'strict':
conn.execute('SELECT pg_advisory_xact_lock(%s)', (MESSAGE_LOCK,))
count = conn.execute('SELECT count(*) AS n FROM messages').fetchone()['n']
if count >= max_messages:
return 'full', None
row = conn.execute(
"""
INSERT INTO messages (id, chat_id, sender_id, body, created_at)
SELECT %s, %s, %s, %s, %s
WHERE EXISTS (
SELECT 1 FROM chat_members
WHERE chat_id = %s AND user_id = %s
)
RETURNING id, chat_id, sender_id, body, created_at
""",
(
_uuid(message_id),
_uuid(chat_id),
_uuid(sender_id),
text,
created,
_uuid(chat_id),
_uuid(sender_id),
),
).fetchone()
if row is None:
return 'missing', None
conn.execute(
"""
INSERT INTO idempotency_keys (
user_id, method, path, idem_key, request_hash, message_id, created_at
)
VALUES (%s, %s, %s, %s, %s, %s, now())
""",
(
_uuid(sender_id),
method,
path,
idem_key,
request_hash,
row['id'],
),
)
return 'ok', _message_out(row)
except psycopg.errors.UniqueViolation:
return 'conflict', None
return _guard(run)
def list_messages(chat_id, user_id, limit, cursor_pair):
def run():
with pool.connection() as conn:
with conn.transaction():
member = conn.execute(
"""
SELECT 1 AS ok
FROM chat_members
WHERE user_id = %s AND chat_id = %s
""",
(_uuid(user_id), _uuid(chat_id)),
).fetchone()
if member is None:
return 'missing', None, None
if cursor_pair is None:
rows = conn.execute(
"""
SELECT id, chat_id, sender_id, body, created_at
FROM messages
WHERE chat_id = %s
ORDER BY created_at, id
LIMIT %s
""",
(_uuid(chat_id), limit + 1),
).fetchall()
else:
rows = conn.execute(
"""
SELECT id, chat_id, sender_id, body, created_at
FROM messages
WHERE chat_id = %s
AND (created_at, id) > (%s, %s)
ORDER BY created_at, id
LIMIT %s
""",
(_uuid(chat_id), parse_z(cursor_pair[0]), _uuid(cursor_pair[1]), limit + 1),
).fetchall()
extra = len(rows) > limit
page = [_message_out(row) for row in rows[:limit]]
if extra and page:
last = page[-1]
nxt = cursor.cursor_token('msgs', chat_id, last['created_at'], last['id'])
else:
nxt = None
return 'ok', page, nxt
return _guard(run)
+2
View File
@@ -0,0 +1,2 @@
class DependencyDown(Exception):
"""Postgres or Redis did not answer. The message stays off the HTTP body."""
+21
View File
@@ -0,0 +1,21 @@
"""Apply SQL migrations once. API replicas do not run this."""
import sys
import db
def main():
try:
applied = db.apply_migrations()
except Exception:
print('migrate failed', flush=True)
return 1
if applied:
print('migrations applied: %s' % ','.join(applied), flush=True)
else:
print('migrations already applied', flush=True)
return 0
if __name__ == '__main__':
sys.exit(main())
+25
View File
@@ -0,0 +1,25 @@
from argon2 import PasswordHasher
from argon2.exceptions import InvalidHashError, VerificationError, VerifyMismatchError
# OWASP Argon2id baseline. bcrypt would cut a long UTF-8 password at 72 bytes.
HASHER = PasswordHasher(
time_cost=2,
memory_cost=19456,
parallelism=1,
hash_len=32,
salt_len=16,
)
# One hash so a missing username still pays for a verify.
DUMMY_HASH = HASHER.hash('not-a-real-password')
def hash_password(password):
return HASHER.hash(password)
def verify_password(password_hash, password):
try:
return HASHER.verify(password_hash, password)
except (VerifyMismatchError, VerificationError, InvalidHashError):
return False
+127
View File
@@ -0,0 +1,127 @@
import hashlib
import json
import os
import re
import secrets
import uuid
from datetime import datetime, timedelta, timezone
from pathlib import Path
import redis
from errors import DependencyDown
from timefmt import format_z, parse_z
# Opaque bearer token, at least 256 bits. Redis stores only the SHA-256 of it.
TOKEN_SHAPE = re.compile(r'^[A-Za-z0-9_-]{32,128}$')
SOCKET_TIMEOUT_SECONDS = 1
client = None
ttl_seconds = 3600
def redis_url():
for name in ('REDIS_URL_FILE', 'CACHE_URL_FILE'):
file_path = os.environ.get(name)
if file_path:
return Path(file_path).read_text(encoding='utf-8').strip()
url = os.environ.get('REDIS_URL', '').strip()
if not url:
raise RuntimeError('REDIS_URL is not set')
return url
def configure(ttl):
global ttl_seconds
ttl_seconds = int(ttl)
if ttl_seconds < 1:
raise RuntimeError('SESSION_TTL_SECONDS must be >= 1')
def open_client():
global client
if client is not None:
return client
client = redis.Redis.from_url(
redis_url(),
socket_connect_timeout=SOCKET_TIMEOUT_SECONDS,
socket_timeout=SOCKET_TIMEOUT_SECONDS,
decode_responses=True,
)
return client
def close_client():
global client
if client is not None:
client.close()
client = None
def _key(token):
digest = hashlib.sha256(token.encode('utf-8')).hexdigest()
return 'sess:' + digest
def _guard(fn):
try:
return fn()
except redis.RedisError as exc:
raise DependencyDown(type(exc).__name__) from None
def ping():
_guard(lambda: client.ping())
def token_shape(token):
return isinstance(token, str) and TOKEN_SHAPE.fullmatch(token) is not None
def create(user_id):
token = secrets.token_urlsafe(32)
expires = datetime.now(timezone.utc) + timedelta(seconds=ttl_seconds)
payload = json.dumps({
'user_id': user_id,
'expires_at': format_z(expires),
}, separators=(',', ':'))
def run():
# ex= is the Redis TTL. A restart does not extend it.
client.set(_key(token), payload, ex=ttl_seconds)
return token, format_z(expires)
return _guard(run)
def read(token):
if not token_shape(token):
return None
def run():
raw = client.get(_key(token))
if raw is None:
return None
try:
payload = json.loads(raw)
user_id = str(uuid.UUID(payload['user_id']))
expires = parse_z(payload['expires_at'])
except (ValueError, KeyError, TypeError):
client.delete(_key(token))
return None
if expires <= datetime.now(timezone.utc):
client.delete(_key(token))
return None
return user_id
return _guard(run)
def revoke(token):
if not token_shape(token):
return False
def run():
return client.delete(_key(token)) == 1
return _guard(run)
+17
View File
@@ -0,0 +1,17 @@
from datetime import datetime, timezone
def format_z(value):
if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc)
value = value.astimezone(timezone.utc)
return value.strftime('%Y-%m-%dT%H:%M:%S.%fZ')
def parse_z(value):
for fmt in ('%Y-%m-%dT%H:%M:%S.%fZ', '%Y-%m-%dT%H:%M:%SZ'):
try:
return datetime.strptime(value, fmt).replace(tzinfo=timezone.utc)
except ValueError:
continue
raise ValueError('bad timestamp')
+76
View File
@@ -0,0 +1,76 @@
"""Removed member must get 404 on a repeated key before the body is compared.
Membership is revoked with DELETE in Postgres, the same way an isolated
database check does it. The script prints status codes only.
"""
import re
import subprocess
import sys
import uuid
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
sys.path.insert(0, str(Path(__file__).resolve().parent))
from client import Client
import stand
UUID = re.compile(r'^[0-9a-fA-F-]{36}$')
def revoke(user_id, chat_id):
if UUID.fullmatch(user_id) is None or UUID.fullmatch(chat_id) is None:
raise SystemExit('refusing to interpolate a non-uuid')
sql = (
"DELETE FROM chat_members WHERE user_id = '%s' AND chat_id = '%s'"
% (user_id, chat_id)
)
stand.compose(['exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-c', sql])
def post(client, actor, chat_id, text, key, expected):
result = client.call(
'POST',
'/api/v1/chats/%s/messages' % chat_id,
expected,
actor=actor,
body={'text': text},
headers={'Idempotency-Key': key},
schema='Message' if expected == 201 else 'Error',
)
print('status', result['status'], 'expected', expected)
return result
def main():
stand.wait_ready()
client = Client(stand.BASE, timeout=10)
alice = client.user('acl')
bob = client.user('aclpeer')
chat = client.call(
'POST',
'/api/v1/chats',
201,
actor=alice,
body={'title': 'Проверка ключа', 'member_ids': [alice['id'], bob['id']]},
schema='Chat',
)['json']
key = 'same-' + uuid.uuid4().hex[:16]
post(client, alice, chat['id'], 'первое', key, 201)
post(client, alice, chat['id'], 'другое', key, 409)
revoke(alice['id'], chat['id'])
post(client, alice, chat['id'], 'другое', key, 404)
post(client, alice, chat['id'], 'первое', key, 404)
history = client.call(
'GET',
'/api/v1/chats/%s/messages' % chat['id'],
404,
actor=alice,
schema='Error',
)
print('history', history['status'])
print('access check passed')
if __name__ == '__main__':
main()
+254
View File
@@ -0,0 +1,254 @@
"""Grade 4 evidence: parallel idempotency, each replica stopped, rolling replace.
Leaves two API replicas running. Does not print tokens.
"""
import json
import subprocess
import sys
import threading
import time
import uuid
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
sys.path.insert(0, str(Path(__file__).resolve().parent))
from client import Client, Failure
import stand
ROOT = stand.ROOT
OUT = ROOT / 'evidence' / 'grade4.json'
def history_texts(client, actor, chat_id):
page = client.call('GET', '/api/v1/chats/%s/messages?limit=100' % chat_id, actor=actor, schema='MessagePage')['json']
return [item['text'] for item in page['items']]
def parallel_same_key(client, alice, chat):
key = uuid.uuid4().hex
text = 'параллельный ключ ' + key[:8]
barrier = threading.Barrier(16)
found = []
def worker():
local = Client(stand.BASE, timeout=5)
barrier.wait()
try:
response = local.message(alice, chat, text, key=key)
found.append(('ok', response['json']['id'], response['headers']['X-Instance-Id']))
except Failure as exc:
found.append(('err', str(exc), ''))
threads = [threading.Thread(target=worker) for _ in range(16)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
ids = {item[1] for item in found if item[0] == 'ok'}
instances = {item[2] for item in found if item[0] == 'ok'}
texts = history_texts(client, alice, chat['id'])
return {
'attempts': len(found),
'successes': sum(1 for item in found if item[0] == 'ok'),
'errors': sum(1 for item in found if item[0] != 'ok'),
'distinct_message_ids': len(ids),
'history_copies': texts.count(text),
'instances': sorted(instances),
}
def parallel_conflict(client, alice, chat):
key = uuid.uuid4().hex
barrier = threading.Barrier(8)
found = []
def worker(index):
local = Client(stand.BASE, timeout=5)
barrier.wait()
try:
response = local.message(alice, chat, 'вариант-%s-%s' % (key[:6], index), key=key)
found.append(response['status'])
except Failure as exc:
text = str(exc)
found.append(409 if '409' in text else text)
threads = [threading.Thread(target=worker, args=(index,)) for index in range(8)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
created = sum(1 for item in found if item == 201)
conflicts = sum(1 for item in found if item == 409)
return {'created': created, 'conflicts': conflicts, 'other': [item for item in found if item not in (201, 409)]}
def run_load(alice, chat, seconds, errors, successes, stop_flag):
def worker(worker_id):
local = Client(stand.BASE, timeout=2)
index = 0
while not stop_flag['flag']:
try:
response = local.message(alice, chat, 'нагрузка-%s-%s' % (worker_id, index))
successes.append((time.monotonic(), response['headers']['X-Instance-Id']))
except Failure:
errors.append(time.monotonic())
index += 1
threads = [threading.Thread(target=worker, args=(i,)) for i in range(4)]
for thread in threads:
thread.start()
return threads
def stop_each(alice, chat):
rounds = []
containers = stand.api_containers()
if len(containers) < 2:
raise SystemExit('need 2 running replicas before sequential stop')
for target in containers:
errors, successes = [], []
stop_flag = {'flag': False}
threads = run_load(alice, chat, 0, errors, successes, stop_flag)
time.sleep(2)
before = len(successes)
started = time.monotonic()
subprocess.run(['docker', 'stop', '-t', '6', target['id']], check=True, capture_output=True)
stopped_at = time.monotonic()
deadline = time.monotonic() + 10
recovered = None
while time.monotonic() < deadline:
recent = [item for item in successes if item[0] >= started and item[1] != target['hostname']]
if recent:
recovered = recent[0][0] - started
break
time.sleep(0.1)
time.sleep(2)
stop_flag['flag'] = True
for thread in threads:
thread.join()
after = [item for item in successes if item[0] >= started]
err_after = [item for item in errors if item >= started]
subprocess.run(['docker', 'start', target['id']], check=True, capture_output=True)
back = time.monotonic()
seen_again = False
while time.monotonic() - back < 30:
running = {row['hostname'] for row in stand.api_containers()}
if target['hostname'] in running:
ids = stand.observe_instances(alice, 2, attempts=15)
seen_again = target['hostname'] in ids
if seen_again:
break
time.sleep(0.4)
rounds.append({
'stopped_instance': target['hostname'],
'container': target['name'],
'successful_writes_before_stop': before,
'successful_writes_after_stop_call': len(after),
'failed_attempts_after_stop_call': len(err_after),
'docker_stop_seconds': round(stopped_at - started, 3),
'first_other_instance_seconds': None if recovered is None else round(recovered, 3),
'target_seen_again_after_start': seen_again,
})
stand.wait_ready()
restored = stand.observe_instances(alice, 2, attempts=50)
if len(restored) < 2:
raise SystemExit('replica %s did not return to the load balancer' % target['hostname'])
return rounds
def rolling(alice):
errors = []
successes = []
stop_flag = {'flag': False}
def worker():
local = Client(stand.BASE, timeout=2)
while not stop_flag['flag']:
try:
response = local.call('GET', '/api/v1/users/me', actor=alice, schema='User')
successes.append((time.monotonic(), response['headers']['X-Instance-Id']))
except Failure:
errors.append(time.monotonic())
threads = [threading.Thread(target=worker) for _ in range(4)]
for thread in threads:
thread.start()
time.sleep(1)
started = time.monotonic()
script = ROOT / 'scripts' / 'rolling-replace.sh'
# Windows executes the PowerShell equivalent; the shell script is the Linux procedure.
if os_name_is_windows():
completed = subprocess.run(
['powershell', '-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', str(ROOT / 'scripts' / 'rolling-replace.ps1')],
cwd=ROOT,
text=True,
capture_output=True,
)
else:
completed = subprocess.run(['sh', str(script)], cwd=ROOT, text=True, capture_output=True)
finished = time.monotonic()
time.sleep(2)
stop_flag['flag'] = True
for thread in threads:
thread.join()
during = [item for item in successes if item[0] >= started]
err = [item for item in errors if item >= started]
instances = stand.observe_instances(alice, 2, attempts=40)
return {
'script_exit': completed.returncode,
'elapsed_seconds': round(finished - started, 3),
'successful_reads': len(during),
'failed_reads': len(err),
'instances_after': sorted(instances),
'stderr_tail': (completed.stderr or '')[-400:],
}
def os_name_is_windows():
return sys.platform.startswith('win')
def main():
stand.wait_ready()
client = Client(stand.BASE, timeout=10)
alice, _bob, _eve, chat = client.bootstrap()
seen = stand.observe_instances(alice, 2)
if len(seen) < 2:
raise SystemExit('observed %s instances, need 2' % len(seen))
report = {
'instances_before': sorted(seen),
'parallel_same_key': parallel_same_key(client, alice, chat),
'parallel_different_body': parallel_conflict(client, alice, chat),
'stop_each_replica': stop_each(alice, chat),
'rolling_replace': rolling(alice),
}
same = report['parallel_same_key']
conflict = report['parallel_different_body']
if same['distinct_message_ids'] != 1 or same['history_copies'] != 1:
raise SystemExit('same key created more than one message')
if conflict['created'] != 1 or conflict['other']:
raise SystemExit('different bodies with one key were not one 201 and the rest 409')
for item in report['stop_each_replica']:
if item['first_other_instance_seconds'] is None or item['first_other_instance_seconds'] > 10:
raise SystemExit('replica did not fail over within 10s: %s' % item['stopped_instance'])
if report['rolling_replace']['script_exit'] != 0 or len(report['rolling_replace']['instances_after']) < 2:
raise SystemExit('rolling replace did not leave two instances')
OUT.parent.mkdir(parents=True, exist_ok=True)
OUT.write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
print(json.dumps({
'same_key_ids': same['distinct_message_ids'],
'same_key_instances': same['instances'],
'conflict_created': conflict['created'],
'conflict_409': conflict['conflicts'],
'failovers': [
{'instance': item['stopped_instance'], 'recovery_s': item['first_other_instance_seconds'], 'errors': item['failed_attempts_after_stop_call']}
for item in report['stop_each_replica']
],
'rolling_errors': report['rolling_replace']['failed_reads'],
'rolling_ok': report['rolling_replace']['successful_reads'],
}, ensure_ascii=False, indent=2))
if __name__ == '__main__':
main()
+160
View File
@@ -0,0 +1,160 @@
"""Find the shared limit that stops extra API replicas from raising RPS.
Each API replica gets a whole core, so the earlier CPU-slicing effect is gone.
Postgres is capped at 0.20 CPU for three points (1/2/4 replicas). A fourth
point releases that cap to 1.0 CPU with the same four replicas. If RPS stops
growing while Postgres sits on 0.20, and jumps when the cap is released, the
database CPU is the saturation point.
"""
import json
import platform
import subprocess
import sys
import threading
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
sys.path.insert(0, str(Path(__file__).resolve().parent))
import stand
ROOT = stand.ROOT
DURATION = 12
CONCURRENCY = 16
RUNS = [
{'name': 'pg020-r1', 'replicas': 1, 'pg_cpus': '0.20'},
{'name': 'pg020-r2', 'replicas': 2, 'pg_cpus': '0.20'},
{'name': 'pg020-r4', 'replicas': 4, 'pg_cpus': '0.20'},
{'name': 'pg100-r4', 'replicas': 4, 'pg_cpus': '1.0'},
]
def deploy(cfg):
env = {
'API_CPUS': '1.0',
'API_MEMORY': '256M',
'CAP_MODE': 'off',
'POOL_MAX': '4',
'API_REPLICAS': str(cfg['replicas']),
'POSTGRES_CPUS': cfg['pg_cpus'],
'LB_CPUS': '0.50',
'MAX_INFLIGHT_BODIES': '64',
'MAX_PER_SECOND': '100000',
}
stand.compose(['up', '-d', '--no-deps', '--force-recreate', 'postgres'], extra_env=env)
stand.compose(
['up', '-d', '--no-deps', '--force-recreate', '--scale', 'api=%d' % cfg['replicas'], 'api'],
extra_env=env,
)
stand.compose(['up', '-d', '--no-deps', '--force-recreate', 'lb'], extra_env=env)
stand.wait_ready(timeout=120)
def run_load(output):
samples = []
stop = {'flag': False}
def stats():
while not stop['flag']:
try:
samples.append(stand.sample_stats())
except Exception:
pass
time.sleep(2)
thread = threading.Thread(target=stats)
thread.start()
completed = subprocess.run(
[
sys.executable,
str(ROOT / 'tests' / 'load.py'),
'--base-url', stand.BASE,
'--duration', str(DURATION),
'--concurrency', str(CONCURRENCY),
'--output', str(output),
],
cwd=ROOT,
text=True,
capture_output=True,
)
stop['flag'] = True
thread.join()
report = json.loads(output.read_text(encoding='utf-8')) if output.exists() else {}
report['load_exit'] = completed.returncode
if completed.returncode != 0 and 'successful_requests' not in report:
report['stderr_tail'] = (completed.stderr or completed.stdout or '')[-400:]
return report, samples
def restore():
stand.compose(
['up', '-d', '--no-deps', '--force-recreate', 'postgres'],
extra_env={'POSTGRES_CPUS': '1.0'},
)
stand.compose(
['up', '-d', '--no-deps', '--force-recreate', '--scale', 'api=2', 'api'],
extra_env={
'API_CPUS': '0.50',
'API_MEMORY': '512M',
'CAP_MODE': 'off',
'POOL_MAX': '4',
'API_REPLICAS': '2',
'MAX_INFLIGHT_BODIES': '32',
'MAX_PER_SECOND': '10000',
},
)
stand.compose(
['up', '-d', '--no-deps', '--force-recreate', 'lb'],
extra_env={'LB_CPUS': '0.50'},
)
stand.wait_ready(timeout=120)
def main():
info = subprocess.run(['docker', 'info', '--format', '{{.NCPU}} {{.MemTotal}}'], text=True, capture_output=True)
rows = []
for cfg in RUNS:
print('run %s' % cfg['name'], flush=True)
deploy(cfg)
output = ROOT / 'evidence' / ('load-sat-%s.json' % cfg['name'])
report, samples = run_load(output)
rows.append({
'name': cfg['name'],
'replicas': cfg['replicas'],
'api_cpus_each': '1.0',
'postgres_cpus': cfg['pg_cpus'],
'lb_cpus': '0.50',
'pool_max': 4,
'concurrency': CONCURRENCY,
'duration_seconds': DURATION,
'successful_rps': report.get('successful_rps'),
'failed_requests': report.get('failed_requests'),
'successful_requests': report.get('successful_requests'),
'latency_ms': report.get('successful_latency_ms'),
'cpu_api_mean': stand.mean_cpu(samples, '-api-'),
'cpu_postgres_mean': stand.mean_cpu(samples, '-postgres-'),
'cpu_lb_mean': stand.mean_cpu(samples, '-lb-'),
'load_exit': report.get('load_exit'),
})
print(json.dumps(rows[-1], ensure_ascii=False), flush=True)
evidence = {
'question': 'With a whole core per API replica, extra replicas stop helping when Postgres CPU is the shared cap.',
'method': 'tests/load.py closed-loop POST, 12s, concurrency 16, one run each. Postgres limit 0.20 CPU for 1/2/4 replicas, then the same 4 replicas with Postgres limit 1.0.',
'host': {
'platform': platform.platform(),
'python': platform.python_version(),
'docker_info': (info.stdout or '').strip(),
},
'note': 'docker stats 100 percent is one core. A 0.20 CPU limit is saturated near 20 percent.',
'runs': rows,
}
dest = ROOT / 'evidence' / 'saturation.json'
dest.write_text(json.dumps(evidence, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
restore()
print('restored', flush=True)
if __name__ == '__main__':
main()
+206
View File
@@ -0,0 +1,206 @@
"""Track 1. Fixed total CPU/RAM, then a growing budget, then one pool change.
The strict cap takes one global lock per message. The shipped default is cap off.
This script measures both and writes evidence/track1.json plus evidence/track1.svg.
"""
import json
import platform
import subprocess
import sys
import threading
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
sys.path.insert(0, str(Path(__file__).resolve().parent))
from client import Client
import stand
ROOT = stand.ROOT
DURATION = 12
CONCURRENCY = 8
RUNS = [
{'series': 'fixed-strict', 'replicas': 1, 'cpus': '1.0', 'memory': '1024M', 'cap': 'strict', 'pool': 4},
{'series': 'fixed-strict', 'replicas': 2, 'cpus': '0.50', 'memory': '512M', 'cap': 'strict', 'pool': 4},
{'series': 'fixed-strict', 'replicas': 4, 'cpus': '0.25', 'memory': '256M', 'cap': 'strict', 'pool': 4},
{'series': 'fixed-off', 'replicas': 1, 'cpus': '1.0', 'memory': '1024M', 'cap': 'off', 'pool': 4},
{'series': 'fixed-off', 'replicas': 2, 'cpus': '0.50', 'memory': '512M', 'cap': 'off', 'pool': 4},
{'series': 'fixed-off', 'replicas': 4, 'cpus': '0.25', 'memory': '256M', 'cap': 'off', 'pool': 4},
{'series': 'grow-off', 'replicas': 1, 'cpus': '0.50', 'memory': '256M', 'cap': 'off', 'pool': 4},
{'series': 'grow-off', 'replicas': 2, 'cpus': '0.50', 'memory': '256M', 'cap': 'off', 'pool': 4},
{'series': 'grow-off', 'replicas': 4, 'cpus': '0.50', 'memory': '256M', 'cap': 'off', 'pool': 4},
{'series': 'pool-16', 'replicas': 4, 'cpus': '0.50', 'memory': '256M', 'cap': 'off', 'pool': 16},
]
def deploy(cfg):
stand.compose(
['up', '-d', '--no-deps', '--force-recreate', '--scale', 'api=%d' % cfg['replicas'], 'api'],
extra_env={
'API_CPUS': cfg['cpus'],
'API_MEMORY': cfg['memory'],
'CAP_MODE': cfg['cap'],
'POOL_MAX': str(cfg['pool']),
'API_REPLICAS': str(cfg['replicas']),
},
)
stand.wait_ready(timeout=120)
def run_load(output):
samples = []
stop = {'flag': False}
def stats():
while not stop['flag']:
try:
samples.append(stand.sample_stats())
except Exception:
pass
time.sleep(2)
thread = threading.Thread(target=stats)
thread.start()
completed = subprocess.run(
[
sys.executable,
str(ROOT / 'tests' / 'load.py'),
'--base-url', stand.BASE,
'--duration', str(DURATION),
'--concurrency', str(CONCURRENCY),
'--output', str(output),
],
cwd=ROOT,
text=True,
capture_output=True,
)
stop['flag'] = True
thread.join()
report = json.loads(output.read_text(encoding='utf-8')) if output.exists() else {}
report['load_exit'] = completed.returncode
if completed.returncode != 0 and 'successful_requests' not in report:
report['stderr_tail'] = (completed.stderr or completed.stdout or '')[-400:]
return report, samples
def svg(rows):
width, height = 920, 420
groups = []
for series in ('fixed-strict', 'fixed-off', 'grow-off', 'pool-16'):
items = [row for row in rows if row['series'] == series]
if items:
groups.append((series, items))
bar_w = 28
gap = 14
group_gap = 36
max_rps = max((row['successful_rps'] or 0) for row in rows) or 1
parts = [
'<svg xmlns="http://www.w3.org/2000/svg" width="%d" height="%d">' % (width, height),
'<rect width="100%" height="100%" fill="#f7f4ee"/>',
'<text x="24" y="32" font-family="sans-serif" font-size="18">Successful RPS, closed loop, %ds, concurrency %d</text>' % (DURATION, CONCURRENCY),
]
x = 48
colors = {
'fixed-strict': '#8c4a3a',
'fixed-off': '#2f6f4e',
'grow-off': '#24577a',
'pool-16': '#a56b12',
}
labels = {
'fixed-strict': 'fixed CPU, strict cap',
'fixed-off': 'fixed CPU, cap off',
'grow-off': 'CPU grows with replicas',
'pool-16': 'pool 16',
}
for series, items in groups:
parts.append('<text x="%d" y="58" font-family="sans-serif" font-size="12">%s</text>' % (x, labels[series]))
for item in items:
rps = item['successful_rps'] or 0
bar_h = int(280 * rps / max_rps)
y = 340 - bar_h
parts.append('<rect x="%d" y="%d" width="%d" height="%d" fill="%s"/>' % (x, y, bar_w, bar_h, colors[series]))
parts.append('<text x="%d" y="%d" font-family="sans-serif" font-size="11">%s</text>' % (x, y - 4, round(rps, 1)))
parts.append('<text x="%d" y="360" font-family="sans-serif" font-size="11">x%d</text>' % (x, item['replicas']))
x += bar_w + gap
x += group_gap
parts.append('<text x="24" y="400" font-family="sans-serif" font-size="12">xN is the API replica count. Fixed series keeps 1.0 CPU and 1024 MiB in total.</text>')
parts.append('</svg>')
return '\n'.join(parts)
def restore():
stand.compose(
['up', '-d', '--no-deps', '--force-recreate', '--scale', 'api=2', 'api'],
extra_env={
'API_CPUS': '0.50',
'API_MEMORY': '512M',
'CAP_MODE': 'off',
'POOL_MAX': '4',
'API_REPLICAS': '2',
},
)
stand.wait_ready(timeout=120)
def main():
info = subprocess.run(['docker', 'info', '--format', '{{.NCPU}} {{.MemTotal}}'], text=True, capture_output=True)
stand.wait_ready()
client = Client(stand.BASE, timeout=10)
alice = client.user('track')
rows = []
for cfg in RUNS:
print('run %s x%s' % (cfg['series'], cfg['replicas']), flush=True)
deploy(cfg)
seen = stand.observe_instances(alice, cfg['replicas'], attempts=60)
output = ROOT / 'evidence' / ('load-%s-%s.json' % (cfg['series'], cfg['replicas']))
report, samples = run_load(output)
rows.append({
'series': cfg['series'],
'replicas': cfg['replicas'],
'cpus_each': cfg['cpus'],
'memory_each': cfg['memory'],
'cap': cfg['cap'],
'pool_max': cfg['pool'],
'instances_seen': len(seen),
'successful_rps': report.get('successful_rps'),
'failed_requests': report.get('failed_requests'),
'successful_requests': report.get('successful_requests'),
'latency_ms': report.get('successful_latency_ms'),
'cpu_api_mean': stand.mean_cpu(samples, '-api-'),
'cpu_postgres_mean': stand.mean_cpu(samples, '-postgres-'),
'cpu_lb_mean': stand.mean_cpu(samples, '-lb-'),
'load_exit': report.get('load_exit'),
})
evidence = {
'hypothesis': 'A global message-cap lock saturates before extra API replicas help. Removing it raises RPS until Postgres or the pool does.',
'profile': {
'tool': 'tests/load.py closed-loop POST messages',
'duration_seconds': DURATION,
'concurrency': CONCURRENCY,
'fixed_total': '1.0 CPU and 1024 MiB split across 1/2/4 replicas',
'grow': '0.50 CPU and 256 MiB per replica, count 1/2/4',
},
'host': {
'platform': platform.platform(),
'python': platform.python_version(),
'cpu_count': os_cpu(),
'docker_info': (info.stdout or '').strip(),
},
'runs': rows,
}
dest = ROOT / 'evidence' / 'track1.json'
dest.write_text(json.dumps(evidence, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
(ROOT / 'evidence' / 'track1.svg').write_text(svg(rows), encoding='utf-8')
restore()
print(json.dumps(rows, ensure_ascii=False, indent=2))
def os_cpu():
import os
return os.cpu_count()
if __name__ == '__main__':
main()
+138
View File
@@ -0,0 +1,138 @@
"""Helpers for the local compose stand. No tokens are printed or stored."""
import json
import os
import subprocess
import time
import urllib.error
import urllib.request
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
BASE = 'http://127.0.0.1:8080'
def compose(args, extra_env=None, check=True):
env = os.environ.copy()
if extra_env:
env.update({key: str(value) for key, value in extra_env.items()})
command = ['docker', 'compose', '--env-file', str(ROOT / '.env'), *args]
result = subprocess.run(command, cwd=ROOT, env=env, text=True, capture_output=True)
if check and result.returncode != 0:
raise SystemExit((result.stderr or result.stdout or 'compose failed').strip())
return result
def inspect(cid, template):
result = subprocess.run(
['docker', 'inspect', '-f', template, cid],
text=True,
capture_output=True,
check=True,
)
return result.stdout.strip()
def api_containers(running_only=True):
args = ['ps', '-q', 'api'] if running_only else ['ps', '-aq', 'api']
result = compose(args)
found = []
for line in result.stdout.splitlines():
cid = line.strip()
if not cid:
continue
hostname = inspect(cid, '{{.Config.Hostname}}')
name = inspect(cid, '{{.Name}}').lstrip('/')
found.append({'id': cid, 'hostname': hostname, 'name': name})
return found
def wait_ready(timeout=90):
deadline = time.monotonic() + timeout
last = 'none'
while time.monotonic() < deadline:
try:
with urllib.request.urlopen(BASE + '/health/ready', timeout=2) as response:
if response.status == 200 and response.headers.get('X-Instance-Id'):
return response.headers.get('X-Instance-Id')
except (urllib.error.URLError, TimeoutError, OSError) as exc:
last = type(exc).__name__
time.sleep(0.4)
raise SystemExit('load balancer did not become ready (%s)' % last)
def observe_instances(actor, expect, attempts=40):
import sys
sys.path.insert(0, str(ROOT / 'tests'))
from client import Client
seen = set()
client = Client(BASE, timeout=3)
for _ in range(attempts):
try:
seen.add(client.call('GET', '/api/v1/users/me', actor=actor, schema='User')['headers']['X-Instance-Id'])
except Exception:
time.sleep(0.3)
continue
if len(seen) >= expect:
return seen
time.sleep(0.05)
return seen
def started_at():
result = subprocess.run(
['docker', 'ps', '-a', '--filter', 'label=com.docker.compose.project=mtusi-lab3', '--format', '{{.Names}}\t{{.Status}}\t{{.ID}}'],
text=True,
capture_output=True,
check=True,
)
rows = []
for line in result.stdout.splitlines():
if not line.strip():
continue
name = line.split('\t', 1)[0]
rows.append({
'name': name,
'started_at': inspect(name, '{{.State.StartedAt}}'),
'status': inspect(name, '{{.State.Status}}'),
})
return rows
def sample_stats():
result = subprocess.run(
['docker', 'stats', '--no-stream', '--format', '{{json .}}'],
text=True,
capture_output=True,
check=True,
)
rows = []
for line in result.stdout.splitlines():
if not line.strip():
continue
item = json.loads(line)
name = item.get('Name', '')
if 'mtusi-lab3' not in name:
continue
rows.append({
'name': name,
'cpu': item.get('CPUPerc'),
'mem': item.get('MemUsage'),
})
return rows
def mean_cpu(samples, role):
values = []
for sample in samples:
for row in sample:
if role not in row['name']:
continue
text = (row.get('cpu') or '').strip().rstrip('%')
try:
values.append(float(text))
except ValueError:
continue
if not values:
return None
return round(sum(values) / len(values), 2)