ЛР 4 - БВТ2304 - Соколова - на 4 #63

Open
sokolova-bvt2304 wants to merge 2 commits from sokolova-bvt2304/mtusi2026lab4:solution/bvt2304-sokolova into main
53 changed files with 6305 additions and 23 deletions
+4
View File
@@ -18,3 +18,7 @@ __pycache__
node_modules
evidence
.DS_Store
/secrets/
*.bak
__pycache__/
*.py[cod]
+2
View File
@@ -30,3 +30,5 @@ coverage/
.vscode/
# Keep small, sanitized evidence summaries in evidence/ under version control.
# Inspect git diff --cached: ignore rules are not a secret scanner.
/secrets/
*.bak
+26
View File
@@ -0,0 +1,26 @@
FROM python:3.12.10-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
RUN addgroup --system appgroup \
&& adduser --system --ingroup appgroup --no-create-home appuser \
&& chown appuser:appgroup /app
COPY --chown=appuser:appgroup app ./app
COPY --chown=appuser:appgroup migrations ./migrations
USER appuser
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()" || exit 1
CMD ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--no-access-log"]
+483 -23
View File
@@ -1,37 +1,497 @@
# Отчёт — лабораторная 4
# Отчёт — лабораторная 4. S3-хранилище и вложения
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Соколова Полина.
- Целевая оценка: 4. Трек уровня 5 не заявляется.
- Дата проверок: 05.11.2026.
## Запуск из чистого clone
## 1. Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
### Требования и версии
## Архитектура и решения
Нужны Linux/WSL, Docker Engine, Docker Compose, Python 3.9+ и make.
Команды выполняются из корня клонированного репозитория.
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Образы закреплены:
## Доказательства критериев
- приложение: python:3.12.10-slim-bookworm;
- PostgreSQL: postgres:16.4-bookworm;
- Redis: redis:7.2.5-bookworm;
- балансировщик: haproxy:3.0.5-bookworm;
- S3-хранилище: chrislusf/seaweedfs:4.44.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
Python-зависимости закреплены в requirements.txt, в том числе
boto3==1.43.124 и Pillow==13.0.0.
Используются оба файла Compose:
```sh
docker compose -f compose.yaml -f compose.s3.yaml --env-file .env ...
```
В compose.yaml сохранено историческое имя проекта mtusi2026lab3.
При переносе реализации это позволило использовать прежние тома.
Имя образа приложения для этой работы — mtusi-messenger:lab4.
Переименование проекта без явного переноса/подключения томов не выполнялось.
### Локальная конфигурация PostgreSQL
Если используется существующая БД, нужен прежний .env с её паролем.
Новый пароль в .env не меняет пароль пользователя уже инициализированной БД.
Для нового стенда без существующего тома PostgreSQL .env можно создать так:
```sh
python3 - <<'PY'
import os
from pathlib import Path
import secrets
import subprocess
path = Path(".env")
if path.exists():
raise SystemExit(".env уже существует; оставлен без изменений")
project = os.environ.get("COMPOSE_PROJECT_NAME", "mtusi2026lab3")
volumes = subprocess.check_output(
[
"docker", "volume", "ls", "-q",
"--filter", f"label=com.docker.compose.project={project}",
"--filter", "label=com.docker.compose.volume=postgres_data",
],
text=True,
).strip()
if volumes:
raise SystemExit(
"Найден существующий том PostgreSQL. "
"Восстановите прежний .env; новый пароль не генерируется."
)
with path.open("x") as stream:
stream.write("POSTGRES_PASSWORD=" + secrets.token_urlsafe(32) + "\n")
stream.write("SESSION_TTL_SECONDS=3600\n")
stream.write("USER_CACHE_TTL_SECONDS=60\n")
path.chmod(0o600)
print(".env создан; пароль не выводится")
PY
```
Файл .env локальный, в Git не включается.
### Создание S3-credentials и запуск
```sh
python3 scripts/setup_s3_secrets.py
python3 scripts/setup_s3_secrets.py
docker compose -f compose.yaml -f compose.s3.yaml \
--env-file .env config --quiet
docker compose -f compose.yaml -f compose.s3.yaml \
--env-file .env up -d --build
docker compose -f compose.yaml -f compose.s3.yaml \
--env-file .env ps -a
curl -i --max-time 10 http://127.0.0.1:18082/health/ready
```
Повторный запуск setup_s3_secrets.py проверяет существующий набор ключей,
не ротирует его. Неполный или противоречивый набор не перегенерируется
автоматически.
SeaweedFS запускается с постоянным каталогом /data.
Параметр -bucket=messenger-attachments создаёт bucket при его отсутствии.
API получает отдельную учётную запись с правами Read/Write/List только
на messenger-attachments. Административная конфигурация в API не монтируется.
Сервис migrate применяет SQL-миграции до запуска API.
Он проверяет контрольные суммы ранее применённых миграций.
Изменения ЛР 4 находятся в migrations/003_attachments.sql.
Сервис s3-check проверяет доступ к bucket с ключами API.
migrate и s3-check — одноразовые сервисы; Exited (0) для них означает успех.
Дождаться healthy у обеих API-реплик и HTTP 200 от /health/ready.
Единственная опубликованная точка входа API:
http://127.0.0.1:18082.
### Остановка с сохранением данных
```sh
docker compose -f compose.yaml -f compose.s3.yaml \
--env-file .env stop
```
### Намеренное удаление учебных данных
Следующая команда разрушительная: выполнять только при осознанном
сбросе стенда. Она не используется в проверках restart/failover.
```sh
docker compose -f compose.yaml -f compose.s3.yaml \
--env-file .env down -v
```
## 2. Архитектура и решения
```text
Клиент
→ HAProxy
→ API × 2
→ PostgreSQL: пользователи, чаты, членство, сессии,
сообщения, идемпотентность, метаданные вложений
→ Redis: кэш пользователей
→ SeaweedFS S3: оригинальные байты изображений
Периодический cleanup
→ PostgreSQL + S3: проверка и удаление старых объектов без метаданных
```
Подтверждённые бизнес-данные и сессии хранятся в PostgreSQL.
Redis не является единственным хранилищем сессий.
Оригиналы не сохраняются в БД или постоянном локальном каталоге API.
Сохранены возможности ЛР 3: Bearer-сессии, членство в чатах,
пагинация, две API-реплики, X-Instance-Id и идемпотентная отправка сообщений.
### Вложения
POST /api/v1/chats/{chat_id}/attachments принимает сырые байты PNG/JPEG.
Ответ 201 ready возвращается после записи оригинала в S3 и успешного
выхода из транзакции записи метаданных PostgreSQL.
Ограничения:
- максимум 10 485 760 байт;
- максимум 40 000 000 пикселей;
- проверка реального формата и соответствия Content-Type;
- проверка структуры и декодирования изображения через Pillow;
- отклонение опасных filename;
- filename не используется как путь или S3-ключ;
- ключ originals/<UUID> генерирует сервер;
- оригинал не перекодируется.
Тело читается частями с подсчётом размера; весь допустимый оригинал
затем удерживается в ограниченном буфере памяти.
Это не zero-copy и не сквозная передача клиента напрямую в S3.
На API-реплике одновременно выполняется одна бинарная операция.
Ожидание слота ограничено 3 секундами, чтение тела — 60 секундами.
Перегрузка может приводить к 503.
Синхронное декодирование выполняется вне основного event loop.
У download проверяются доступ, размер и SHA-256.
Байты возвращает API без redirect.
Публичные метаданные не содержат object_key и credentials.
### Сообщения и доступ
attachment_ids возвращается и для текстовых сообщений как [].
Допускается пустой/пробельный текст только при наличии вложений.
До 10 уникальных вложений, порядок сохраняется.
Любой участник может прикрепить доступное вложение своего чата,
даже если его загрузил другой участник.
Неизвестное или вложение другого чата даёт 404.
При повторе Idempotency-Key сравниваются текст и упорядоченный список
attachment_ids. Отсутствующее поле и [] эквивалентны.
Повтор возвращает то же Message, другой запрос с тем же ключом — 409.
### Конфигурация и secrets
Реальная конфигурация соответствует логическим ролям .env.example:
- PostgreSQL: PGHOST, PGPORT, PGDATABASE, PGUSER;
PGPASSWORD в контейнерах берётся из POSTGRES_PASSWORD локального .env;
- сессии: SESSION_TTL_SECONDS;
- Redis: REDIS_HOST, REDIS_PORT, USER_CACHE_TTL_SECONDS;
- S3: S3_ENDPOINT, S3_REGION, S3_BUCKET;
- ключи S3: S3_ACCESS_KEY_ID_FILE и S3_SECRET_ACCESS_KEY_FILE;
- лимиты 10 MiB и 40 млн пикселей заданы константами в app/images.py;
- две API-реплики заданы в Compose.
DATABASE_URL_FILE и CACHE_URL_FILE из шаблона не используются:
выбрана конфигурация через отдельные параметры подключения.
MAX_UPLOAD_BYTES и MAX_IMAGE_PIXELS не являются изменяемыми настройками
текущей реализации: контрактные пределы закреплены в коде.
secrets/ исключён из Git и Docker build context.
API видит только свои ключи; полный s3_config.json доступен S3-сервису.
S3 и внутренние API-порты не опубликованы на хост.
Admin UI SeaweedFS отключён.
HTTP используется только для локального учебного стенда.
## 3. Согласованность S3 и PostgreSQL
Между S3 и PostgreSQL нет общей распределённой транзакции.
Перед PUT загрузчик получает PostgreSQL advisory lock для пары
bucket/object_key. Блокировка удерживается до завершения транзакции
метаданных. Для этой транзакции idle timeout увеличен до 60 секунд.
Если PUT прошёл, а метаданные не зафиксированы, объект может временно
остаться без строки в attachments.
Очистка app.cleanup_orphans:
1. Постранично обходит originals/.
2. Рассматривает только управляемые ключи originals/<UUID>.
3. Пропускает объекты моложе grace period.
4. Пробует получить ту же транзакционную блокировку.
5. Пропускает занятый ключ.
6. Проверяет отсутствие метаданных в PostgreSQL.
7. Повторно проверяет LastModified через HEAD.
8. Удаляет только старый незанятый объект без метаданных.
Объект с метаданными сохраняется, даже если ещё не прикреплён к сообщению.
При ошибке проверки БД или S3 скрипт не принимает отсутствие данных
за разрешение на удаление.
По умолчанию команда работает в dry-run:
```sh
docker compose -f compose.yaml -f compose.s3.yaml \
--env-file .env run --rm --no-deps api \
python -m app.cleanup_orphans
```
Сервис cleanup запускает режим apply с grace period 86 400 секунд
и интервалом между обходами 3 600 секунд.
При ошибке процесс завершается; используется restart: unless-stopped.
Следовательно, при исправных зависимостях orphan становится кандидатом
после grace period и удаляется при последующем успешном обходе.
Укороченный grace period 60 секунд использовался только в контролируемом
тесте, ограниченном собственными тестовыми ключами.
## 4. Доказательства критериев
| Критерий | Команда/сценарий | Наблюдение | Доказательство |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
| Структура и синтаксис | make check | PASS | Результат ручного запуска, этот отчёт |
| Публичный сценарий ЛР 4 | make test BASE_URL=http://127.0.0.1:18082 | PASS | tests/smoke.py, результат запуска |
| Две API-реплики | docker compose ... ps -a | Обе healthy | Ручное наблюдение |
| Сохранность после restart | make test-persistence BASE_URL=http://127.0.0.1:18082 ACTION_SCRIPT=scripts/restart.sh | PASS, сохранены данные и прежняя сессия | scripts/restart.sh, времена StartedAt до/после |
| Failover при записях | make test-failover BASE_URL=http://127.0.0.1:18082 ACTION_SCRIPT=scripts/stop-one.sh | 2 реплики, 116 успешных записей, 3 неудачные попытки, первое восстановление 2,72 с, PASS | scripts/stop-one.sh, результат запуска |
| Оригинал после отказа загрузившей реплики | Загрузка, остановка точного контейнера по X-Instance-Id, download через другую реплику | Байты и SHA-256 совпали | Ручной сценарий, раздел отказов |
| Приватность bucket | Подписанный GET существующего оригинала, затем неподписанные GET объекта и списка | С ключами читается; анонимно оба запроса дают 403 | Ручной сценарий |
| Отрицательные проверки | python3 tests/attachments_negative.py | 22 проверки с подготовкой данных, PASS | tests/attachments_negative.py |
| Отказ S3 | python3 tests/s3_outage.py | Метаданные 200, upload/download 503, оригинал после восстановления цел | tests/s3_outage.py |
| Компенсация после PUT/rollback БД | tests/cleanup_orphans_check.py в контейнере | Старый orphan удалён, свежий и занятый сохранены, dry-run и повтор безопасны | tests/cleanup_orphans_check.py |
| Прерывание загрузки | tests/upload_abort_check.py в контейнере | Три обрыва, новых объектов и метаданных нет, следующая загрузка успешна | tests/upload_abort_check.py |
| Конкурентные загрузки и RAM | python3 tests/upload_memory.py | Два запуска по 10 клиентов, все ответы 201, новых OOM нет | evidence/upload_memory.json, evidence/upload_memory.csv |
| Идемпотентность настройки ключей | Дважды python3 scripts/setup_s3_secrets.py | Existing credentials verified; keys unchanged | scripts/setup_s3_secrets.py |
| Периодическая очистка | docker compose ... logs cleanup | mode=apply, grace_seconds=86400 | compose.s3.yaml, app/cleanup_orphans.py |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
Текстовые выводы большинства проверок были просмотрены вручную.
Отдельные raw-логи для каждой строки не сохранялись.
Машинные результаты эксперимента памяти сохранены в evidence/.
## Эксперименты и отказы
### Команды собственных контейнерных тестов
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
```sh
docker compose -f compose.yaml -f compose.s3.yaml --env-file .env \
run --rm --no-deps -e PYTHONPATH=/app \
-v "$PWD/tests/cleanup_orphans_check.py:/tmp/cleanup_orphans_check.py:ro" \
api python /tmp/cleanup_orphans_check.py
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
docker compose -f compose.yaml -f compose.s3.yaml --env-file .env \
run --rm --no-deps -e PYTHONPATH=/app \
-v "$PWD/tests/upload_abort_check.py:/tmp/upload_abort_check.py:ro" \
api python /tmp/upload_abort_check.py
```
## Ограничения и следующие шаги
Для cleanup-теста нужно существующее корректное вложение;
его создаёт предшествующий smoke.
Во время теста прерывания нельзя параллельно загружать другие файлы:
он сравнивает набор S3-ключей до и после обрывов.
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
## 5. Перезапуски и отказы
### Persistence
Штатная проверка завершилась:
```text
PASS: persistence preserved confirmed data and existing session.
Hook scope needs operator evidence.
```
StartedAt до → после:
- API-4: 2026-11-05T14:38:46Z → 2026-11-05T14:45:31Z;
- API-5: 2026-11-05T14:33:11Z → 2026-11-05T14:45:32Z;
- LB: 2026-11-05T14:33:12Z → 2026-11-05T14:45:33Z;
- PostgreSQL: 2026-10-29T17:28:22Z → 2026-11-05T14:45:26Z;
- Redis: 2026-10-29T17:28:22Z → 2026-11-05T14:45:26Z;
- S3: 2026-11-05T14:31:54Z → 2026-11-05T14:45:27Z.
Времена UTC, дробные секунды в отчёте опущены.
Изменение времён подтверждает перезапуск всех перечисленных компонентов,
а не только API. Тома не удалялись.
### Ручной failover оригинала
- Загрузившая реплика: dbaf4bc268d2:2b8507b1cf45.
- После её остановки скачала: 1e244991e550:6f695457e5ac.
- SHA-256: 4258699ba0181027db1c32e7d9360339ccf87658015a24cd1424a0a5bd1b329c.
- Сообщение с вложением сохранилось.
- Остановленная реплика затем восстановлена.
| Воздействие | Наблюдение | Ограничение вывода |
| --- | --- | --- |
| Остановка одной API-реплики | Штатный failover PASS; восстановление записи 2,72 с | Не доказывает HA PostgreSQL/LB/хоста |
| Перезапуск API и всех хранилищ | Persistence PASS с прежней сессией | Проверен restart с сохранёнными томами, не потеря диска |
| Остановка S3 | На обеих API live=200, ready=503 | LB может отдавать собственную ошибку без API-заголовков |
| Upload/download при остановленном S3 | Оба 503 unavailable | Проверены прямым обращением к API в Docker-сети |
| Возврат S3 | Readiness восстановился; оригинал скачался побайтно | Точное время восстановления не измерялось |
| Разрыв тела upload | Метаданные и S3-объекты не появились | Все три проверенных обрыва пришлись на одну реплику |
| S3 PUT и откат БД | Остался orphan, затем безопасно очищен | Контролируемый rollback, не физический отказ диска |
## 6. Эксперимент с конкурентными загрузками
### Цель и методика
Гипотеза: при ограничении бинарной обработки одним слотом на реплику
10 конкурентных клиентов не вызывают одновременное декодирование
10 больших изображений и не приводят к OOM в проверенном сценарии.
Стенд: Linux amd64 в локальном Docker/WSL.
Лимиты каждой API-реплики: 0,75 CPU и 512 MiB RAM.
Точная модель CPU и общий объём RAM хоста в данном опыте не фиксировались.
PNG: 6000×6000, 36 000 000 пикселей, 8 540 172 байта.
Изображение воспроизводимо генерируется тестом.
Часть строк содержит псевдослучайные данные, остальные хорошо сжимаются.
Профиль:
1. одиночная загрузка;
2. 10 клиентов, синхронизированных barrier;
3. повторный запуск 10 клиентов.
Между запусками — паузы 2 секунды.
Отдельного длительного прогрева не было; первая одиночная загрузка
является базовым сравнением.
Для каждого запроса сохранены статус, длительность и X-Instance-Id.
Прозрачные повторы upload отсутствуют.
### Результаты
| Сценарий | Клиентов | Максимум активных запросов клиента | Ответы 201 | Время всего запуска |
| --- | ---: | ---: | ---: | ---: |
| single | 1 | 1 | 1/1 | 0,7324 с |
| concurrent_1 | 10 | 10 | 10/10 | 2,3296 с |
| concurrent_2 | 10 | 10 | 10/10 | 2,2903 с |
Для всех 21 ответа 201 дополнительно проверены SHA-256, size_bytes
и status=ready; все проверки метаданных успешны.
Это не заменяет отдельную побайтную проверку каждого из 21 оригинала.
### Память
На каждую реплику собрано 130 измерений, ориентировочно раз в 100 мс.
| Реплика | Начальный memory.peak | Итоговый memory.peak | Максимум выборок memory.current | Итоговый VmHWM |
| --- | ---: | ---: | ---: | ---: |
| 5d51a6717cdb | 153268224 байт | 273399808 байт | 270393344 байт | 255696 KiB |
| eab2fd1a7047 | 152104960 байт | 265211904 байт | 261709824 байт | 253240 KiB |
Итоговый пик cgroup: примерно 261 и 253 MiB при лимите 512 MiB.
На обеих репликах прирост oom=0 и oom_kill=0.
Состояния, времена запуска и счётчики рестартов контейнеров не изменились.
Ошибок измерителя нет.
memory.peak и VmHWM — накопленные максимумы, не обнулялись.
Исходные значения сохранены, итоговые выше исходных.
VmRSS/VmHWM — оценочные показатели Linux.
Измеритель работает внутри контейнера: его память входит в cgroup,
но не в показатели процесса PID 1.
Вывод: для проверенного изображения и двух повторов нагрузки
все запросы успешны, память ниже лимита, новых OOM и рестартов нет.
Это не доказательство универсальной границы памяти при любой нагрузке.
Raw-данные:
- evidence/upload_memory.json — параметры, запросы, состояния, сводка;
- evidence/upload_memory.csv — временные выборки памяти.
Эксперимент ориентирован на RAM, а не максимальный throughput.
p50/p95/p99 и CPU не использовались как критерий успеха;
длительности отдельных запросов сохранены в JSON.
## 7. Ограничения и следующие шаги
- Нет общей транзакции S3/PostgreSQL; окно orphan компенсирует очистка.
- Очистка требует доступности БД и S3; при длительном отказе удаление
откладывается до успешного обхода.
- Не реализовано восстановление метаданных, для которых потерян S3-объект.
- S3, PostgreSQL, Redis и LB — одиночные экземпляры, не кластер HA.
- Нет утверждений о сохранности при потере диска/всего Docker-хоста.
- В памяти удерживается целый допустимый оригинал и декодированное
изображение; это ограниченная буферизация, не zero-copy.
- При более тяжёлой нагрузке возможны документированные 503 по ожиданию слота.
- HTTP и локальные файлы secrets — учебное упрощение для изолированного стенда.
- Механизмы TTL идемпотентности не сокращают существующие записи автоматически:
записи сохраняются дольше обязательного окна повторов.
- Полная матрица форматов и fuzz-тестирование декодеров не выполнялись.
- Треки уровня 5, presigned upload, фоновые варианты изображений не заявляются.
## 8. Источники и библиотеки
- Контракт и задание этого репозитория:
README.md, contracts/README.md, contracts/openapi.json,
contracts/infrastructure.json, tests/README.md.
- Docker Documentation: Compose, secrets, зависимости сервисов,
stop/start/restart, работа с несколькими Compose-файлами.
- SeaweedFS: справка weed mini версии 4.44, S3 Credentials.
- PostgreSQL Documentation: Explicit Locking, advisory locks.
- Psycopg 3 Documentation: управление транзакциями и пул соединений.
- Boto3/Botocore Documentation: S3 PUT/GET/HEAD/LIST/DELETE, ошибки, таймауты.
- Pillow Documentation: Image.open, verify, load, защита от больших изображений.
- Python Documentation: urllib.request, http.client, concurrent.futures.
- Linux Kernel Documentation: /proc, cgroup v2 memory counters.
- Python-библиотеки: FastAPI, Uvicorn, psycopg, psycopg-pool,
Redis client, argon2-cffi, boto3, Pillow.
## Исправление по замечанию: приостановка PostgreSQL
Проблема: при приостановке PostgreSQL запрос истории мог ждать
до клиентского таймаута. Серверный statement_timeout не обеспечивал
ограничение ожидания при замороженных процессах БД.
В app/db.py добавлен DeadlineConnection, используемый пулом соединений.
Ожидание отдельной операции соединения ограничено на стороне приложения
значением DB_IO_TIMEOUT_SECONDS, по умолчанию 3 секунды.
При превышении срока соединение закрывается и поднимается OperationalError.
Существующий обработчик app/main.py возвращает 503 unavailable.
Реализация использует низкоуровневое ожидание Psycopg;
проверена с закреплёнными psycopg[binary]==3.2.6
и psycopg-pool==3.2.6. При обновлении драйвера тест необходимо повторить.
Команда проверки:
```sh
python3 student-tests/postgres_pause.py
```
Фактический результат 06.11.2026:
- До отказа история доступна через обе API-реплики.
- PostgreSQL приостановлен через docker pause.
- Четыре параллельных запроса истории получили 503 unavailable.
- Ответы пришли от обеих API-реплик с X-Instance-Id.
- Время ответа: 3,0871–3,0881 секунды.
- PostgreSQL возобновлён в finally.
- История восстановилась, подтверждённое сообщение сохранилось.
- Итог: PASS: bounded API 503 on both replicas; history recovered.
Машинное доказательство: evidence/postgres_pause.json.
Тест проверяет HTTP-ответ при приостановленной БД с порогом 6 секунд.
Трёхсекундное ограничение отдельного ожидания не является универсальной
гарантией трёхсекундного времени любого составного HTTP-запроса.
View File
+108
View File
@@ -0,0 +1,108 @@
from __future__ import annotations
import base64
import binascii
import hashlib
import re
import secrets
from argon2 import PasswordHasher, Type
from argon2.exceptions import VerificationError
from starlette.concurrency import run_in_threadpool
USERNAME_RE = re.compile(r"[a-z0-9_]{3,32}", re.ASCII)
password_hasher = PasswordHasher(
time_cost=3,
memory_cost=65536,
parallelism=4,
hash_len=32,
salt_len=16,
type=Type.ID,
)
DUMMY_PASSWORD_HASH = password_hasher.hash(
secrets.token_urlsafe(32)
)
async def hash_password(password: str) -> str:
return await run_in_threadpool(
password_hasher.hash,
password,
)
async def verify_password(
password_hash: str,
password: str,
) -> bool:
try:
return await run_in_threadpool(
password_hasher.verify,
password_hash,
password,
)
except VerificationError:
return False
def parse_basic_credentials(
authorization: str | None,
) -> tuple[str, str] | None:
if authorization is None or len(authorization) > 2048:
return None
parts = authorization.split()
if len(parts) != 2 or parts[0].lower() != "basic":
return None
try:
decoded = base64.b64decode(
parts[1],
validate=True,
).decode("utf-8")
except (binascii.Error, UnicodeError, ValueError):
return None
username, separator, password = decoded.partition(":")
if not separator:
return None
if USERNAME_RE.fullmatch(username) is None:
return None
if not 12 <= len(password) <= 128:
return None
return username, password
def parse_bearer_token(
authorization: str | None,
) -> str | None:
if authorization is None or len(authorization) > 2048:
return None
parts = authorization.split()
if len(parts) != 2 or parts[0].lower() != "bearer":
return None
token = parts[1]
if re.fullmatch(r"[A-Za-z0-9_-]{43}", token) is None:
return None
return token
def generate_token() -> str:
return secrets.token_urlsafe(32)
def hash_token(token: str) -> str:
return hashlib.sha256(token.encode("ascii")).hexdigest()
+129
View File
@@ -0,0 +1,129 @@
from __future__ import annotations
import json
import logging
import os
from datetime import datetime, timezone
from uuid import UUID
from redis.asyncio import Redis
from redis.exceptions import RedisError
from app.store import store
logger = logging.getLogger("messenger.cache")
CACHE_TTL_SECONDS = int(
os.getenv("USER_CACHE_TTL_SECONDS", "60")
)
if not 1 <= CACHE_TTL_SECONDS <= 3600:
raise ValueError(
"USER_CACHE_TTL_SECONDS must be between 1 and 3600"
)
redis_client = Redis(
host=os.getenv("REDIS_HOST", "redis"),
port=int(os.getenv("REDIS_PORT", "6379")),
decode_responses=True,
socket_connect_timeout=0.2,
socket_timeout=0.2,
retry_on_timeout=False,
max_connections=10,
)
def serialize_user(user: dict) -> str:
created_at = user["created_at"]
return json.dumps(
{
"id": str(user["id"]),
"username": user["username"],
"display_name": user["display_name"],
"created_at": created_at.astimezone(
timezone.utc
).isoformat().replace("+00:00", "Z"),
},
ensure_ascii=False,
separators=(",", ":"),
)
def deserialize_user(
raw: str,
expected_id: UUID,
) -> dict:
data = json.loads(raw)
if not isinstance(data, dict):
raise ValueError("Invalid cached user")
expected_fields = {
"id",
"username",
"display_name",
"created_at",
}
if set(data) != expected_fields:
raise ValueError("Invalid cached user fields")
if not all(isinstance(value, str) for value in data.values()):
raise ValueError("Invalid cached user types")
user_id = UUID(data["id"])
if user_id != expected_id:
raise ValueError("Cached user ID mismatch")
created_at = datetime.fromisoformat(
data["created_at"].replace("Z", "+00:00")
)
if created_at.tzinfo is None:
raise ValueError("Cached timestamp has no timezone")
return {
"id": user_id,
"username": data["username"],
"display_name": data["display_name"],
"created_at": created_at,
}
async def get_cached_user(user_id: UUID) -> dict | None:
key = f"user:v1:{user_id}"
try:
raw = await redis_client.get(key)
if raw is not None:
return deserialize_user(raw, user_id)
except RedisError:
logger.warning("User cache read unavailable")
except (ValueError, TypeError, OverflowError):
logger.warning("Invalid user cache entry ignored")
user = await store.get_user(user_id)
if user is None:
return None
try:
await redis_client.set(
key,
serialize_user(user),
ex=CACHE_TTL_SECONDS,
)
except RedisError:
logger.warning("User cache write unavailable")
return user
async def close_cache() -> None:
await redis_client.aclose()
+208
View File
@@ -0,0 +1,208 @@
from __future__ import annotations
import argparse
from collections import Counter
from datetime import datetime, timedelta, timezone
import json
import os
import sys
import time
from uuid import UUID
import psycopg
from psycopg.rows import dict_row
from botocore.exceptions import ClientError
from app.object_locks import object_lock_id
from app.storage import bucket_name, get_client
def managed_key(key: str) -> bool:
prefix = "originals/"
if not key.startswith(prefix):
return False
suffix = key[len(prefix):]
try:
return str(UUID(suffix)) == suffix
except ValueError:
return False
def database_connection():
return psycopg.connect(
host=os.getenv("PGHOST", "postgres"),
port=int(os.getenv("PGPORT", "5432")),
dbname=os.getenv("PGDATABASE", "messenger"),
user=os.getenv("PGUSER", "messenger"),
password=os.environ["PGPASSWORD"],
connect_timeout=3,
options=(
"-c statement_timeout=3000 "
"-c lock_timeout=2000 "
"-c idle_in_transaction_session_timeout=10000"
),
autocommit=True,
row_factory=dict_row,
)
def sweep(*, apply: bool, grace_seconds: int) -> dict:
cutoff = datetime.now(timezone.utc) - timedelta(
seconds=grace_seconds
)
counts = Counter({
"scanned": 0,
"kept": 0,
"fresh": 0,
"busy": 0,
"ignored": 0,
"gone": 0,
"candidates": 0,
"deleted": 0,
})
with database_connection() as conn:
client = get_client()
bucket = bucket_name()
pages = client.get_paginator("list_objects_v2")
for page in pages.paginate(
Bucket=bucket,
Prefix="originals/",
PaginationConfig={"PageSize": 200},
):
for item in page.get("Contents", []):
counts["scanned"] += 1
key = item["Key"]
if not managed_key(key):
counts["ignored"] += 1
continue
if item["LastModified"] >= cutoff:
counts["fresh"] += 1
continue
with conn.transaction():
conn.execute(
"SET LOCAL "
"idle_in_transaction_session_timeout = '60s'"
)
locked = conn.execute(
"SELECT pg_try_advisory_xact_lock(%s) AS locked",
(object_lock_id(bucket, key),),
).fetchone()["locked"]
if not locked:
counts["busy"] += 1
continue
metadata = conn.execute(
"""
SELECT 1
FROM attachments
WHERE object_key = %s
""",
(key,),
).fetchone()
if metadata is not None:
counts["kept"] += 1
continue
try:
current = client.head_object(
Bucket=bucket,
Key=key,
)
except ClientError as exc:
status = exc.response.get(
"ResponseMetadata", {}
).get("HTTPStatusCode")
if status == 404:
counts["gone"] += 1
continue
raise
if current["LastModified"] >= cutoff:
counts["fresh"] += 1
continue
counts["candidates"] += 1
print(
json.dumps({
"action": "delete" if apply else "would_delete",
"key": key,
}),
flush=True,
)
if apply:
client.delete_object(
Bucket=bucket,
Key=key,
)
counts["deleted"] += 1
return {
"mode": "apply" if apply else "dry-run",
"grace_seconds": grace_seconds,
**dict(counts),
}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--apply", action="store_true")
parser.add_argument("--grace-seconds", type=int, default=86400)
parser.add_argument("--allow-short-grace", action="store_true")
parser.add_argument("--interval-seconds", type=int, default=0)
args = parser.parse_args()
if args.grace_seconds < 60:
parser.error("grace must be at least 60 seconds")
if args.grace_seconds < 3600 and not args.allow_short_grace:
parser.error(
"short test grace requires --allow-short-grace"
)
if args.interval_seconds != 0 and args.interval_seconds < 60:
parser.error("interval must be zero or at least 60 seconds")
if args.grace_seconds < 3600:
print(
"WARNING: short grace is for controlled local tests",
file=sys.stderr,
flush=True,
)
while True:
try:
result = sweep(
apply=args.apply,
grace_seconds=args.grace_seconds,
)
except Exception as exc:
print(
json.dumps({
"status": "failed",
"error_type": type(exc).__name__,
}),
file=sys.stderr,
flush=True,
)
return 1
print(json.dumps(result, sort_keys=True), flush=True)
if args.interval_seconds == 0:
return 0
time.sleep(args.interval_seconds)
if __name__ == "__main__":
raise SystemExit(main())
+12
View File
@@ -0,0 +1,12 @@
from __future__ import annotations
import os
SESSION_TTL_SECONDS = int(
os.getenv("SESSION_TTL_SECONDS", "3600")
)
if SESSION_TTL_SECONDS < 1:
raise ValueError("SESSION_TTL_SECONDS must be positive")
+76
View File
@@ -0,0 +1,76 @@
from __future__ import annotations
import asyncio
import os
from psycopg import AsyncConnection, OperationalError, waiting
from psycopg.rows import dict_row
from psycopg_pool import AsyncConnectionPool
DB_IO_TIMEOUT_SECONDS = float(
os.getenv("DB_IO_TIMEOUT_SECONDS", "3")
)
if DB_IO_TIMEOUT_SECONDS <= 0:
raise ValueError("DB_IO_TIMEOUT_SECONDS must be positive")
class DeadlineConnection(AsyncConnection):
async def wait(self, gen, interval=0.1):
try:
async with asyncio.timeout(DB_IO_TIMEOUT_SECONDS):
return await waiting.wait_async(
gen,
self.pgconn.socket,
interval=interval,
)
except TimeoutError:
await self.close()
raise OperationalError(
"PostgreSQL response deadline exceeded"
) from None
except asyncio.CancelledError:
await self.close()
raise
pool = AsyncConnectionPool(
conninfo="",
connection_class=DeadlineConnection,
kwargs={
"host": os.getenv("PGHOST", "postgres"),
"port": int(os.getenv("PGPORT", "5432")),
"dbname": os.getenv("PGDATABASE", "messenger"),
"user": os.getenv("PGUSER", "messenger"),
"password": os.environ.get("PGPASSWORD", ""),
"connect_timeout": 3,
"options": (
"-c statement_timeout=3000 "
"-c lock_timeout=2000 "
"-c idle_in_transaction_session_timeout=10000"
),
"row_factory": dict_row,
},
min_size=1,
max_size=5,
timeout=3,
max_waiting=20,
open=False,
check=AsyncConnectionPool.check_connection,
)
async def open_db() -> None:
await pool.open(wait=True, timeout=10)
async def close_db() -> None:
await pool.close()
async def check_db() -> None:
async with pool.connection() as conn:
await conn.execute("SELECT 1")
+48
View File
@@ -0,0 +1,48 @@
from __future__ import annotations
from uuid import UUID
from fastapi import Request
from app.auth import hash_token, parse_bearer_token
from app.cache import get_cached_user
from app.errors import not_found, unauthorized
from app.store import store
from app.validation import parse_uuid
async def current_user(request: Request) -> tuple[dict, str]:
token = parse_bearer_token(
request.headers.get("authorization")
)
if token is None:
raise unauthorized()
token_digest = hash_token(token)
user_id = await store.get_session_user_id(token_digest)
if user_id is None:
raise unauthorized()
user = await get_cached_user(user_id)
if user is None:
raise unauthorized()
return user, token_digest
async def accessible_chat(
raw_chat_id: str,
user_id: UUID,
) -> tuple[UUID, dict]:
chat_id = parse_uuid(raw_chat_id)
chat = await store.get_accessible_chat(chat_id, user_id)
if chat is None:
raise not_found()
return chat_id, chat
+71
View File
@@ -0,0 +1,71 @@
from __future__ import annotations
from uuid import uuid4
from fastapi import Request
from fastapi.responses import JSONResponse
class ApiError(Exception):
def __init__(
self,
status: int,
code: str,
message: str,
challenge: str | None = None,
):
super().__init__(message)
self.status = status
self.code = code
self.message = message
self.challenge = challenge
def invalid(message: str = "Некорректный запрос") -> ApiError:
return ApiError(400, "invalid_request", message)
def unauthorized(scheme: str = "Bearer") -> ApiError:
if scheme == "Basic":
challenge = 'Basic realm="messenger", charset="UTF-8"'
else:
challenge = 'Bearer realm="messenger"'
return ApiError(
401,
"unauthorized",
"Недействительная аутентификация",
challenge,
)
def not_found() -> ApiError:
return ApiError(404, "not_found", "Объект не найден")
def error_response(
request: Request,
error: ApiError,
) -> JSONResponse:
request_id = getattr(
request.state,
"request_id",
str(uuid4()),
)
headers = {"X-Request-Id": request_id}
if error.challenge is not None:
headers["WWW-Authenticate"] = error.challenge
return JSONResponse(
status_code=error.status,
content={
"error": {
"code": error.code,
"message": error.message,
"request_id": request_id,
}
},
headers=headers,
)
+174
View File
@@ -0,0 +1,174 @@
from __future__ import annotations
import hashlib
import json
from uuid import UUID, uuid4
from app.db import pool
from app.errors import ApiError, invalid, not_found
async def create_message_once(
chat_id: UUID,
sender_id: UUID,
text: str,
key: str,
attachment_ids: list[UUID] | None = None,
) -> dict:
attachment_ids = list(
attachment_ids if attachment_ids is not None else []
)
if (
len(attachment_ids) > 10
or len(set(attachment_ids)) != len(attachment_ids)
):
raise invalid("Некорректный список вложений")
if not isinstance(text, str) or len(text) > 4000:
raise invalid("Некорректный текст сообщения")
if not text.strip() and not attachment_ids:
raise invalid("Сообщение должно содержать текст или вложение")
scope = json.dumps(
[
str(sender_id),
"POST",
f"/api/v1/chats/{chat_id}/messages",
key,
],
separators=(",", ":"),
)
lock_id = int.from_bytes(
hashlib.sha256(scope.encode("utf-8")).digest()[:8],
byteorder="big",
signed=True,
)
async with pool.connection() as conn:
await conn.execute(
"SELECT pg_advisory_xact_lock(%s)",
(lock_id,),
)
cursor = await conn.execute(
"""
SELECT i.request_text,
i.request_attachment_ids,
m.id, m.chat_id, m.sender_id,
m.text, m.created_at,
ARRAY(
SELECT ma.attachment_id
FROM message_attachments AS ma
WHERE ma.message_id = m.id
ORDER BY ma.position
) AS attachment_ids
FROM message_idempotency AS i
JOIN messages AS m ON m.id = i.message_id
WHERE i.user_id = %s
AND i.chat_id = %s
AND i.http_method = 'POST'
AND i.idempotency_key = %s
""",
(sender_id, chat_id, key),
)
previous = await cursor.fetchone()
if previous is not None:
if (
previous["request_text"] != text
or previous["request_attachment_ids"] != attachment_ids
):
raise ApiError(
409,
"conflict",
"Ключ уже использован с другим сообщением",
)
message = {
field: previous[field]
for field in (
"id",
"chat_id",
"sender_id",
"text",
"created_at",
"attachment_ids",
)
}
else:
if attachment_ids:
cursor = await conn.execute(
"""
SELECT id
FROM attachments
WHERE id = ANY(%s::uuid[])
AND chat_id = %s
AND status = 'ready'
ORDER BY id
FOR SHARE
""",
(attachment_ids, chat_id),
)
available = await cursor.fetchall()
if len(available) != len(attachment_ids):
raise not_found()
cursor = await conn.execute(
"""
INSERT INTO messages (
id, chat_id, sender_id, text
)
VALUES (%s, %s, %s, %s)
RETURNING id, chat_id, sender_id,
text, created_at
""",
(uuid4(), chat_id, sender_id, text),
)
message = await cursor.fetchone()
if attachment_ids:
async with conn.cursor() as cursor:
await cursor.executemany(
"""
INSERT INTO message_attachments (
message_id, attachment_id, position
)
VALUES (%s, %s, %s)
""",
[
(message["id"], attachment_id, position)
for position, attachment_id
in enumerate(attachment_ids)
],
)
await conn.execute(
"""
INSERT INTO message_idempotency (
user_id,
chat_id,
http_method,
idempotency_key,
request_text,
request_attachment_ids,
message_id
)
VALUES (%s, %s, 'POST', %s, %s, %s::uuid[], %s)
""",
(
sender_id,
chat_id,
key,
text,
attachment_ids,
message["id"],
),
)
message["attachment_ids"] = attachment_ids
return message
+95
View File
@@ -0,0 +1,95 @@
from __future__ import annotations
import io
import unicodedata
from PIL import Image, UnidentifiedImageError
from app.errors import ApiError, invalid
MAX_BYTES = 10 * 1024 * 1024
MAX_PIXELS = 40_000_000
FORMAT_TO_MIME = {
"PNG": "image/png",
"JPEG": "image/jpeg",
}
def validate_filename(value: str | None) -> str:
if (
value is None
or not 1 <= len(value) <= 255
or not value.strip()
):
raise invalid("Некорректное имя файла")
for char in value:
if char in "/\\" or unicodedata.category(char) == "Cc":
raise invalid("Некорректное имя файла")
return value
def inspect_image(data: bytes, declared_type: str) -> str:
if len(data) > MAX_BYTES:
raise ApiError(
413,
"payload_too_large",
"Файл превышает 10 MiB",
)
if declared_type not in FORMAT_TO_MIME.values():
raise ApiError(
415,
"unsupported_media_type",
"Допустимы только image/png и image/jpeg",
)
try:
with Image.open(io.BytesIO(data)) as image:
detected = FORMAT_TO_MIME.get(image.format or "")
width, height = image.size
pixels = width * height
except Image.DecompressionBombError:
raise ApiError(
413,
"payload_too_large",
"Слишком много пикселей",
) from None
except (UnidentifiedImageError, OSError, SyntaxError, ValueError):
raise ApiError(
415,
"unsupported_media_type",
"Файл не является корректным PNG/JPEG",
) from None
if detected is None or detected != declared_type:
raise ApiError(
415,
"unsupported_media_type",
"Содержимое не соответствует Content-Type",
)
if width < 1 or height < 1 or pixels > MAX_PIXELS:
raise ApiError(
413,
"payload_too_large",
"Слишком много пикселей",
)
try:
with Image.open(io.BytesIO(data)) as image:
image.verify()
with Image.open(io.BytesIO(data)) as image:
image.load()
except (Image.DecompressionBombError, OSError, SyntaxError, ValueError):
raise ApiError(
415,
"unsupported_media_type",
"Файл повреждён",
) from None
return detected
+7
View File
@@ -0,0 +1,7 @@
from __future__ import annotations
import socket
from uuid import uuid4
INSTANCE_ID = f"{socket.gethostname()}:{uuid4().hex[:12]}"
+116
View File
@@ -0,0 +1,116 @@
from __future__ import annotations
import logging
from contextlib import asynccontextmanager
from uuid import uuid4
from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError
from psycopg import InterfaceError, OperationalError
from psycopg.errors import LockNotAvailable, QueryCanceled
from psycopg_pool import PoolTimeout, TooManyRequests
from starlette.exceptions import HTTPException
from app.cache import close_cache
from app.instance import INSTANCE_ID
from app.db import close_db, open_db
from app.errors import ApiError, error_response, invalid, not_found
from app.routers import attachments, chats, health, messages, sessions, users
logger = logging.getLogger("messenger.api")
@asynccontextmanager
async def lifespan(app: FastAPI):
try:
await open_db()
yield
finally:
await close_cache()
await close_db()
app = FastAPI(
title="Replicated PostgreSQL messenger — Lab 3",
docs_url=None,
redoc_url=None,
openapi_url=None,
lifespan=lifespan,
)
@app.middleware("http")
async def request_middleware(request: Request, call_next):
request.state.request_id = str(uuid4())
try:
response = await call_next(request)
except ApiError as exc:
response = error_response(request, exc)
except (
OperationalError,
InterfaceError,
PoolTimeout,
TooManyRequests,
QueryCanceled,
LockNotAvailable,
) as exc:
logger.warning(
"Database unavailable request_id=%s error_type=%s",
request.state.request_id,
type(exc).__name__,
)
response = error_response(
request,
ApiError(
503,
"unavailable",
"Сервис временно недоступен",
),
)
except Exception as exc:
logger.error(
"Unhandled error request_id=%s error_type=%s",
request.state.request_id,
type(exc).__name__,
)
response = error_response(
request,
ApiError(
500,
"internal_error",
"Внутренняя ошибка сервера",
),
)
response.headers["X-Request-Id"] = request.state.request_id
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
@app.exception_handler(RequestValidationError)
async def validation_error_handler(request: Request, exc):
return error_response(request, invalid())
@app.exception_handler(HTTPException)
async def http_error_handler(request: Request, exc: HTTPException):
if exc.status_code == 404:
error = not_found()
else:
error = invalid("Недопустимый HTTP-запрос")
return error_response(request, error)
app.include_router(health.router)
app.include_router(users.router)
app.include_router(sessions.router)
app.include_router(chats.router)
app.include_router(messages.router)
app.include_router(attachments.router)
+84
View File
@@ -0,0 +1,84 @@
from __future__ import annotations
import hashlib
import os
from pathlib import Path
import psycopg
MIGRATIONS_DIR = Path(__file__).resolve().parent.parent / "migrations"
MIGRATION_LOCK_ID = 202602001
def apply_migrations() -> None:
files = sorted(MIGRATIONS_DIR.glob("[0-9]*_*.sql"))
if not files:
raise RuntimeError("Migration files not found")
with psycopg.connect(
host=os.getenv("PGHOST", "postgres"),
port=int(os.getenv("PGPORT", "5432")),
dbname=os.getenv("PGDATABASE", "messenger"),
user=os.getenv("PGUSER", "messenger"),
password=os.environ["PGPASSWORD"],
connect_timeout=5,
options="-c statement_timeout=30000 -c lock_timeout=10000",
) as conn:
conn.execute(
"SELECT pg_advisory_xact_lock(%s)",
(MIGRATION_LOCK_ID,),
)
conn.execute(
"""
CREATE TABLE IF NOT EXISTS schema_migrations (
version TEXT PRIMARY KEY,
checksum TEXT NOT NULL,
applied_at TIMESTAMPTZ NOT NULL
DEFAULT clock_timestamp()
)
"""
)
for path in files:
version = path.name
content = path.read_bytes()
checksum = hashlib.sha256(content).hexdigest()
row = conn.execute(
"""
SELECT checksum
FROM schema_migrations
WHERE version = %s
""",
(version,),
).fetchone()
if row is not None:
if row[0] != checksum:
raise RuntimeError(
f"Applied migration was modified: {version}"
)
print(f"Already applied: {version}", flush=True)
continue
conn.execute(content.decode("utf-8"), prepare=False)
conn.execute(
"""
INSERT INTO schema_migrations (version, checksum)
VALUES (%s, %s)
""",
(version, checksum),
)
print(f"Executed: {version} (pending commit)", flush=True)
print("Migrations committed successfully", flush=True)
if __name__ == "__main__":
apply_migrations()
+12
View File
@@ -0,0 +1,12 @@
from __future__ import annotations
import hashlib
def object_lock_id(bucket: str, object_key: str) -> int:
value = f"s3-object\0{bucket}\0{object_key}".encode("utf-8")
return int.from_bytes(
hashlib.sha256(value).digest()[:8],
byteorder="big",
signed=True,
)
+128
View File
@@ -0,0 +1,128 @@
from __future__ import annotations
import base64
import binascii
import json
import re
from datetime import datetime
from uuid import UUID
from fastapi import Request
from fastapi.responses import JSONResponse
from app.errors import invalid
from app.responses import json_response, utc_text
def encode_cursor(scope: str, item: dict) -> str:
raw = json.dumps(
{
"scope": scope,
"created_at": utc_text(item["created_at"]),
"id": str(item["id"]),
},
separators=(",", ":"),
).encode("utf-8")
return (
base64.urlsafe_b64encode(raw)
.decode("ascii")
.rstrip("=")
)
def decode_cursor(
cursor: str,
scope: str,
) -> tuple[datetime, UUID]:
if not cursor or len(cursor) > 2048:
raise invalid("Некорректный cursor")
try:
padded = cursor + "=" * (-len(cursor) % 4)
raw = base64.b64decode(
padded,
altchars=b"-_",
validate=True,
)
data = json.loads(raw.decode("utf-8"))
if (
not isinstance(data, dict)
or set(data) != {"scope", "created_at", "id"}
or data["scope"] != scope
):
raise ValueError
timestamp = data["created_at"]
if (
not isinstance(timestamp, str)
or not timestamp.endswith("Z")
or not isinstance(data["id"], str)
):
raise ValueError
created_at = datetime.fromisoformat(
timestamp[:-1] + "+00:00"
)
item_id = UUID(data["id"])
return created_at, item_id
except (
ValueError,
TypeError,
UnicodeError,
binascii.Error,
OverflowError,
):
raise invalid("Некорректный cursor") from None
def pagination(
request: Request,
scope: str,
) -> tuple[int, tuple[datetime, UUID] | None]:
raw_limit = request.query_params.get("limit", "50")
if (
len(raw_limit) > 10
or re.fullmatch(r"[0-9]+", raw_limit) is None
):
raise invalid("Некорректный limit")
limit = int(raw_limit)
if not 1 <= limit <= 100:
raise invalid("Некорректный limit")
raw_cursor = request.query_params.get("cursor")
after = None
if raw_cursor is not None:
after = decode_cursor(raw_cursor, scope)
return limit, after
def page_response(
rows: list[dict],
limit: int,
scope: str,
) -> JSONResponse:
items = rows[:limit]
next_cursor = None
if len(rows) > limit:
next_cursor = encode_cursor(scope, items[-1])
return json_response(
{
"items": items,
"next_cursor": next_cursor,
}
)
+40
View File
@@ -0,0 +1,40 @@
from __future__ import annotations
from datetime import datetime, timezone
from uuid import UUID
from fastapi.responses import JSONResponse
def utc_text(value: datetime) -> str:
return (
value.astimezone(timezone.utc)
.isoformat(timespec="microseconds")
.replace("+00:00", "Z")
)
def to_json(value):
if isinstance(value, UUID):
return str(value)
if isinstance(value, datetime):
return utc_text(value)
if isinstance(value, dict):
return {
key: to_json(item)
for key, item in value.items()
}
if isinstance(value, (list, tuple)):
return [to_json(item) for item in value]
return value
def json_response(value, status: int = 200) -> JSONResponse:
return JSONResponse(
status_code=status,
content=to_json(value),
)
View File
+272
View File
@@ -0,0 +1,272 @@
from __future__ import annotations
import asyncio
import hashlib
from contextlib import asynccontextmanager
from uuid import UUID, uuid4
from fastapi import APIRouter, Request
from fastapi.responses import Response
from starlette.concurrency import run_in_threadpool
from starlette.requests import ClientDisconnect
from app.db import pool
from app.dependencies import accessible_chat, current_user
from app.errors import ApiError, invalid, not_found
from app.images import MAX_BYTES, inspect_image, validate_filename
from app.object_locks import object_lock_id
from app.responses import json_response
from app.storage import bucket_name, get_object, put_object
from app.validation import parse_uuid
router = APIRouter()
binary_slot = asyncio.Semaphore(1)
PUBLIC_FIELDS = (
"id",
"chat_id",
"owner_id",
"filename",
"content_type",
"size_bytes",
"sha256",
"status",
"created_at",
"updated_at",
)
@asynccontextmanager
async def binary_operation():
try:
await asyncio.wait_for(binary_slot.acquire(), timeout=3)
except TimeoutError:
raise ApiError(
503,
"unavailable",
"Обработка файлов временно занята",
) from None
try:
yield
finally:
binary_slot.release()
def public_attachment(row: dict) -> dict:
result = {field: row[field] for field in PUBLIC_FIELDS}
result["variants"] = []
result["error_code"] = None
return result
async def read_image_body(request: Request) -> bytes:
lengths = request.headers.getlist("content-length")
if len(lengths) > 1:
raise invalid("Некорректный Content-Length")
expected = None
if lengths:
value = lengths[0]
if not value.isascii() or not value.isdecimal():
raise invalid("Некорректный Content-Length")
if len(value) > 20:
raise ApiError(
413, "payload_too_large", "Файл превышает 10 MiB"
)
expected = int(value)
if expected > MAX_BYTES:
raise ApiError(
413, "payload_too_large", "Файл превышает 10 MiB"
)
data = bytearray()
async for chunk in request.stream():
if len(data) + len(chunk) > MAX_BYTES:
raise ApiError(
413, "payload_too_large", "Файл превышает 10 MiB"
)
data.extend(chunk)
if expected is not None and len(data) != expected:
raise invalid("Размер тела не соответствует Content-Length")
return bytes(data)
async def accessible_attachment(
attachment_id: UUID,
user_id: UUID,
) -> dict:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
SELECT a.id, a.chat_id, a.owner_id,
a.filename, a.content_type, a.size_bytes,
a.sha256, a.object_key, a.status,
a.created_at, a.updated_at
FROM attachments AS a
WHERE a.id = %s
AND EXISTS (
SELECT 1
FROM chat_members AS cm
WHERE cm.chat_id = a.chat_id
AND cm.user_id = %s
)
""",
(attachment_id, user_id),
)
row = await cursor.fetchone()
if row is None:
raise not_found()
return row
@router.post("/api/v1/chats/{chat_id}/attachments")
async def upload_attachment(chat_id: str, request: Request):
user, _ = await current_user(request)
chat_uuid, _ = await accessible_chat(chat_id, user["id"])
filenames = request.query_params.getlist("filename")
if len(filenames) != 1:
raise invalid("Требуется одно имя файла")
filename = validate_filename(filenames[0])
content_types = request.headers.getlist("content-type")
if len(content_types) != 1:
raise ApiError(
415,
"unsupported_media_type",
"Требуется Content-Type изображения",
)
declared_type = content_types[0].split(";", 1)[0].strip().lower()
if declared_type not in ("image/png", "image/jpeg"):
raise ApiError(
415,
"unsupported_media_type",
"Допустимы только image/png и image/jpeg",
)
async with binary_operation():
try:
data = await asyncio.wait_for(
read_image_body(request),
timeout=60,
)
except ClientDisconnect:
raise invalid("Загрузка файла прервана") from None
except TimeoutError:
raise invalid("Превышено время загрузки файла") from None
content_type = await run_in_threadpool(
inspect_image,
data,
declared_type,
)
attachment_id = uuid4()
object_key = f"originals/{attachment_id}"
digest = hashlib.sha256(data).hexdigest()
async with pool.connection() as conn:
await conn.execute(
"SET LOCAL idle_in_transaction_session_timeout = '60s'"
)
await conn.execute(
"SELECT pg_advisory_xact_lock(%s)",
(object_lock_id(bucket_name(), object_key),),
)
await put_object(object_key, data, content_type)
cursor = await conn.execute(
"""
INSERT INTO attachments (
id, chat_id, owner_id, filename,
content_type, size_bytes, sha256, object_key
)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
RETURNING id, chat_id, owner_id, filename,
content_type, size_bytes, sha256,
status, created_at, updated_at
""",
(
attachment_id,
chat_uuid,
user["id"],
filename,
content_type,
len(data),
digest,
object_key,
),
)
row = await cursor.fetchone()
return json_response(public_attachment(row), 201)
@router.get("/api/v1/attachments/{attachment_id}")
async def get_attachment(attachment_id: str, request: Request):
user, _ = await current_user(request)
row = await accessible_attachment(
parse_uuid(attachment_id),
user["id"],
)
return json_response(public_attachment(row))
@router.get("/api/v1/attachments/{attachment_id}/content")
async def download_attachment(attachment_id: str, request: Request):
user, _ = await current_user(request)
row = await accessible_attachment(
parse_uuid(attachment_id),
user["id"],
)
variants = request.query_params.getlist("variant")
if len(variants) > 1:
raise invalid("Допускается один вариант изображения")
if variants and variants[0] != "original":
raise not_found()
if row["status"] != "ready":
raise ApiError(
409,
"conflict",
"Вложение ещё не готово",
)
async with binary_operation():
data = await get_object(row["object_key"])
if (
len(data) != row["size_bytes"]
or hashlib.sha256(data).hexdigest() != row["sha256"]
):
raise ApiError(
503,
"unavailable",
"Оригинал временно недоступен",
)
return Response(
content=data,
media_type=row["content_type"],
headers={
"X-Content-Type-Options": "nosniff",
"Cache-Control": "private, no-store",
},
)
+74
View File
@@ -0,0 +1,74 @@
from __future__ import annotations
from fastapi import APIRouter, Request
from app.dependencies import accessible_chat, current_user
from app.errors import invalid
from app.pagination import page_response, pagination
from app.responses import json_response
from app.store import InvalidMembersError, store
from app.validation import parse_uuid, read_payload, valid_text
router = APIRouter()
@router.post("/api/v1/chats")
async def create_chat(request: Request):
user, _ = await current_user(request)
payload = await read_payload(
request,
{"title", "member_ids"},
)
if not valid_text(payload["title"], 100):
raise invalid("Некорректный title")
raw_members = payload["member_ids"]
if (
not isinstance(raw_members, list)
or not 1 <= len(raw_members) <= 100
):
raise invalid("Некорректный member_ids")
member_ids = [parse_uuid(item) for item in raw_members]
if len(set(member_ids)) != len(member_ids):
raise invalid("В member_ids есть дубликаты")
if user["id"] not in member_ids:
raise invalid("Создатель должен входить в member_ids")
try:
chat = await store.create_chat(
payload["title"],
member_ids,
)
except InvalidMembersError:
raise invalid(
"В member_ids есть неизвестный пользователь"
) from None
return json_response(chat, 201)
@router.get("/api/v1/chats")
async def list_chats(request: Request):
user, _ = await current_user(request)
scope = f"chats:{user['id']}"
limit, after = pagination(request, scope)
rows = await store.list_chats(user["id"], limit, after)
return page_response(rows, limit, scope)
@router.get("/api/v1/chats/{chat_id}")
async def get_chat(chat_id: str, request: Request):
user, _ = await current_user(request)
_, chat = await accessible_chat(chat_id, user["id"])
return json_response(chat)
+20
View File
@@ -0,0 +1,20 @@
from __future__ import annotations
from fastapi import APIRouter
from app.db import check_db
from app.storage import check_storage
router = APIRouter()
@router.get("/health/live")
async def health_live():
return {"status": "ok"}
@router.get("/health/ready")
async def health_ready():
await check_db()
await check_storage()
return {"status": "ok"}
+90
View File
@@ -0,0 +1,90 @@
from __future__ import annotations
import re
from fastapi import APIRouter, Request
from app.dependencies import accessible_chat, current_user
from app.errors import invalid
from app.idempotency import create_message_once
from app.pagination import page_response, pagination
from app.responses import json_response
from app.store import store
from app.validation import parse_uuid, read_payload
router = APIRouter()
KEY_RE = re.compile(r"[A-Za-z0-9._:-]{1,128}", re.ASCII)
@router.post("/api/v1/chats/{chat_id}/messages")
async def create_message(chat_id: str, request: Request):
user, _ = await current_user(request)
chat_uuid, _ = await accessible_chat(
chat_id,
user["id"],
)
keys = request.headers.getlist("idempotency-key")
if len(keys) != 1 or KEY_RE.fullmatch(keys[0]) is None:
raise invalid("Требуется корректный Idempotency-Key")
payload = await read_payload(
request,
{"text"},
optional_fields={"attachment_ids"},
)
text = payload["text"]
if not isinstance(text, str) or len(text) > 4000:
raise invalid("Некорректный текст сообщения")
raw_attachment_ids = payload.get("attachment_ids", [])
if (
not isinstance(raw_attachment_ids, list)
or len(raw_attachment_ids) > 10
):
raise invalid("Допускается список максимум из 10 вложений")
attachment_ids = [
parse_uuid(value)
for value in raw_attachment_ids
]
if len(set(attachment_ids)) != len(attachment_ids):
raise invalid("Вложения не должны повторяться")
if not text.strip() and not attachment_ids:
raise invalid("Сообщение должно содержать текст или вложение")
message = await create_message_once(
chat_uuid,
user["id"],
text,
keys[0],
attachment_ids=attachment_ids,
)
return json_response(message, 201)
@router.get("/api/v1/chats/{chat_id}/messages")
async def list_messages(chat_id: str, request: Request):
user, _ = await current_user(request)
chat_uuid, _ = await accessible_chat(
chat_id,
user["id"],
)
scope = f"messages:{chat_uuid}"
limit, after = pagination(request, scope)
rows = await store.list_messages(
chat_uuid,
limit,
after,
)
return page_response(rows, limit, scope)
+82
View File
@@ -0,0 +1,82 @@
from __future__ import annotations
from fastapi import APIRouter, Request
from fastapi.responses import Response
from app.auth import (
DUMMY_PASSWORD_HASH,
generate_token,
hash_token,
parse_basic_credentials,
verify_password,
)
from app.config import SESSION_TTL_SECONDS
from app.dependencies import current_user
from app.errors import invalid, unauthorized
from app.responses import json_response
from app.store import store
router = APIRouter()
@router.post("/api/v1/auth/sessions")
async def create_session(request: Request):
credentials = parse_basic_credentials(
request.headers.get("authorization")
)
if credentials is None:
raise unauthorized("Basic")
username, password = credentials
user = await store.get_user_for_login(username)
stored_hash = (
DUMMY_PASSWORD_HASH
if user is None
else user["password_hash"]
)
password_ok = await verify_password(stored_hash, password)
if user is None or not password_ok:
raise unauthorized("Basic")
if await request.body():
raise invalid("Тело запроса при входе не допускается")
token = generate_token()
session = await store.create_session(
user["id"],
hash_token(token),
SESSION_TTL_SECONDS,
)
public_user = {
field: user[field]
for field in ("id", "username", "display_name", "created_at")
}
return json_response(
{
"token": token,
"token_type": "Bearer",
"expires_at": session["expires_at"],
"user": public_user,
},
201,
)
@router.delete("/api/v1/auth/sessions/current")
async def revoke_session(request: Request):
_, token_digest = await current_user(request)
revoked = await store.revoke_session(token_digest)
if not revoked:
raise unauthorized()
return Response(status_code=204)
+59
View File
@@ -0,0 +1,59 @@
from __future__ import annotations
from fastapi import APIRouter, Request
from app.auth import USERNAME_RE, hash_password
from app.dependencies import current_user
from app.errors import ApiError, invalid
from app.responses import json_response
from app.store import store
from app.validation import read_payload, valid_text
router = APIRouter()
@router.post("/api/v1/users")
async def create_user(request: Request):
payload = await read_payload(
request,
{"username", "display_name", "password"},
)
username = payload["username"]
display_name = payload["display_name"]
password = payload["password"]
if (
not isinstance(username, str)
or USERNAME_RE.fullmatch(username) is None
):
raise invalid("Некорректный username")
if not valid_text(display_name, 100):
raise invalid("Некорректное display_name")
if not valid_text(password, 128) or len(password) < 12:
raise invalid("Пароль должен содержать от 12 до 128 символов")
password_digest = await hash_password(password)
user = await store.create_user(
username,
display_name,
password_digest,
)
if user is None:
raise ApiError(
409,
"conflict",
"Username уже занят",
)
return json_response(user, 201)
@router.get("/api/v1/users/me")
async def get_current_user(request: Request):
user, _ = await current_user(request)
return json_response(user)
+34
View File
@@ -0,0 +1,34 @@
from __future__ import annotations
import asyncio
import time
from app.errors import ApiError
from app.storage import check_storage
async def main() -> None:
deadline = time.monotonic() + 90
while True:
try:
await check_storage()
except ApiError:
if time.monotonic() >= deadline:
raise SystemExit(
"S3 readiness failed: bucket unavailable "
"or API credentials rejected"
) from None
print("Waiting for S3 bucket...", flush=True)
await asyncio.sleep(2)
else:
print(
"S3 bucket is accessible with API credentials",
flush=True,
)
return
if __name__ == "__main__":
asyncio.run(main())
+123
View File
@@ -0,0 +1,123 @@
from __future__ import annotations
import asyncio
import os
from pathlib import Path
import boto3
from botocore.config import Config
from botocore.exceptions import (
BotoCoreError,
ClientError,
)
from app.errors import ApiError
_client = None
def _unavailable() -> ApiError:
return ApiError(
503,
"unavailable",
"Сервис временно недоступен",
)
def _credential(file_variable: str) -> str:
file_path = os.environ[file_variable]
value = Path(file_path).read_text().rstrip("\r\n")
if not value:
raise ValueError("Empty S3 credential file")
return value
def bucket_name() -> str:
return os.environ["S3_BUCKET"]
def get_client():
global _client
if _client is None:
_client = boto3.client(
"s3",
endpoint_url=os.environ["S3_ENDPOINT"],
aws_access_key_id=_credential("S3_ACCESS_KEY_ID_FILE"),
aws_secret_access_key=_credential("S3_SECRET_ACCESS_KEY_FILE"),
region_name=os.getenv("S3_REGION", "us-east-1"),
config=Config(
signature_version="s3v4",
s3={"addressing_style": "path"},
connect_timeout=3,
read_timeout=10,
retries={
"mode": "standard",
"total_max_attempts": 2,
},
),
)
return _client
def _put(key: str, data: bytes, content_type: str) -> None:
get_client().put_object(
Bucket=bucket_name(),
Key=key,
Body=data,
ContentType=content_type,
)
def _get(key: str) -> bytes:
maximum = 10 * 1024 * 1024
response = get_client().get_object(
Bucket=bucket_name(),
Key=key,
)
body = response["Body"]
try:
expected_size = response["ContentLength"]
if not 1 <= expected_size <= maximum:
raise _unavailable()
data = body.read(maximum + 1)
if len(data) != expected_size or len(data) > maximum:
raise _unavailable()
return data
finally:
body.close()
async def put_object(
key: str,
data: bytes,
content_type: str,
) -> None:
try:
await asyncio.to_thread(_put, key, data, content_type)
except (BotoCoreError, ClientError, OSError, KeyError, ValueError):
raise _unavailable() from None
async def get_object(key: str) -> bytes:
try:
return await asyncio.to_thread(_get, key)
except (BotoCoreError, ClientError, OSError, KeyError, ValueError):
raise _unavailable() from None
def _check() -> None:
get_client().head_bucket(Bucket=bucket_name())
async def check_storage() -> None:
try:
await asyncio.to_thread(_check)
except (BotoCoreError, ClientError, OSError, KeyError, ValueError):
raise _unavailable() from None
+280
View File
@@ -0,0 +1,280 @@
from __future__ import annotations
from datetime import datetime
from uuid import UUID, uuid4
from psycopg.errors import ForeignKeyViolation, UniqueViolation
from app.db import pool
class InvalidMembersError(Exception):
pass
class PostgresStore:
async def create_user(
self,
username: str,
display_name: str,
password_hash: str,
) -> dict | None:
try:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
INSERT INTO users (
id, username, display_name, password_hash
)
VALUES (%s, %s, %s, %s)
RETURNING id, username, display_name, created_at
""",
(uuid4(), username, display_name, password_hash),
)
user = await cursor.fetchone()
return user
except UniqueViolation as exc:
if exc.diag.constraint_name == "users_username_unique":
return None
raise
async def get_user(self, user_id: UUID) -> dict | None:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
SELECT id, username, display_name, created_at
FROM users
WHERE id = %s
""",
(user_id,),
)
return await cursor.fetchone()
async def get_user_for_login(self, username: str) -> dict | None:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
SELECT id, username, display_name,
password_hash, created_at
FROM users
WHERE username = %s
""",
(username,),
)
return await cursor.fetchone()
async def create_chat(
self,
title: str,
member_ids: list[UUID],
) -> dict:
chat_id = uuid4()
try:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
INSERT INTO chats (id, title)
VALUES (%s, %s)
RETURNING id, title, created_at
""",
(chat_id, title),
)
chat = await cursor.fetchone()
async with conn.cursor() as cursor:
await cursor.executemany(
"""
INSERT INTO chat_members (chat_id, user_id)
VALUES (%s, %s)
""",
[(chat_id, user_id) for user_id in member_ids],
)
chat["member_ids"] = member_ids
return chat
except ForeignKeyViolation as exc:
raise InvalidMembersError from exc
async def get_accessible_chat(
self,
chat_id: UUID,
user_id: UUID,
) -> dict | None:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
SELECT c.id, c.title, c.created_at,
ARRAY(
SELECT cm.user_id
FROM chat_members AS cm
WHERE cm.chat_id = c.id
ORDER BY cm.user_id
) AS member_ids
FROM chats AS c
WHERE c.id = %s
AND EXISTS (
SELECT 1
FROM chat_members AS access
WHERE access.chat_id = c.id
AND access.user_id = %s
)
""",
(chat_id, user_id),
)
return await cursor.fetchone()
async def list_chats(
self,
user_id: UUID,
limit: int,
after: tuple[datetime, UUID] | None = None,
) -> list[dict]:
params: list[object] = [user_id]
after_sql = ""
if after is not None:
after_sql = "AND (c.created_at, c.id) > (%s, %s)"
params.extend(after)
params.append(limit + 1)
async with pool.connection() as conn:
cursor = await conn.execute(
f"""
SELECT c.id, c.title, c.created_at,
ARRAY(
SELECT cm.user_id
FROM chat_members AS cm
WHERE cm.chat_id = c.id
ORDER BY cm.user_id
) AS member_ids
FROM chats AS c
JOIN chat_members AS access
ON access.chat_id = c.id
WHERE access.user_id = %s
{after_sql}
ORDER BY c.created_at, c.id
LIMIT %s
""",
params,
)
return await cursor.fetchall()
async def create_message(
self,
chat_id: UUID,
sender_id: UUID,
text: str,
) -> dict:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
INSERT INTO messages (id, chat_id, sender_id, text)
VALUES (%s, %s, %s, %s)
RETURNING id, chat_id, sender_id, text, created_at
""",
(uuid4(), chat_id, sender_id, text),
)
message = await cursor.fetchone()
return message
async def list_messages(
self,
chat_id: UUID,
limit: int,
after: tuple[datetime, UUID] | None = None,
) -> list[dict]:
params: list[object] = [chat_id]
after_sql = ""
if after is not None:
after_sql = "AND (created_at, id) > (%s, %s)"
params.extend(after)
params.append(limit + 1)
async with pool.connection() as conn:
cursor = await conn.execute(
f"""
SELECT m.id, m.chat_id, m.sender_id,
m.text, m.created_at,
ARRAY(
SELECT ma.attachment_id
FROM message_attachments AS ma
WHERE ma.message_id = m.id
ORDER BY ma.position
) AS attachment_ids
FROM messages AS m
WHERE m.chat_id = %s
{after_sql}
ORDER BY created_at, id
LIMIT %s
""",
params,
)
return await cursor.fetchall()
async def create_session(
self,
user_id: UUID,
token_hash: str,
ttl_seconds: int,
) -> dict:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
INSERT INTO sessions (
token_hash, user_id, created_at, expires_at
)
VALUES (
%s, %s, statement_timestamp(),
statement_timestamp()
+ (%s * INTERVAL '1 second')
)
RETURNING expires_at
""",
(token_hash, user_id, ttl_seconds),
)
session = await cursor.fetchone()
return session
async def get_session_user_id(
self,
token_hash: str,
) -> UUID | None:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
SELECT user_id
FROM sessions
WHERE token_hash = %s
AND expires_at > statement_timestamp()
""",
(token_hash,),
)
row = await cursor.fetchone()
return None if row is None else row["user_id"]
async def revoke_session(self, token_hash: str) -> bool:
async with pool.connection() as conn:
cursor = await conn.execute(
"""
DELETE FROM sessions
WHERE token_hash = %s
AND expires_at > statement_timestamp()
RETURNING token_hash
""",
(token_hash,),
)
row = await cursor.fetchone()
return row is not None
store = PostgresStore()
+61
View File
@@ -0,0 +1,61 @@
from __future__ import annotations
from uuid import UUID
from fastapi import Request
from app.errors import ApiError, invalid
def parse_uuid(value: object) -> UUID:
if not isinstance(value, str):
raise invalid("Некорректный UUID")
try:
return UUID(value)
except ValueError:
raise invalid("Некорректный UUID") from None
def valid_text(value: object, maximum: int) -> bool:
return (
isinstance(value, str)
and 1 <= len(value) <= maximum
and bool(value.strip())
)
async def read_payload(
request: Request,
expected_fields: set[str],
optional_fields: set[str] | None = None,
) -> dict:
content_type = (
request.headers.get("content-type", "")
.split(";", 1)[0]
.strip()
.lower()
)
if content_type != "application/json":
raise ApiError(
415,
"unsupported_media_type",
"Требуется Content-Type: application/json",
)
try:
payload = await request.json()
except (ValueError, UnicodeError):
raise invalid("Некорректный JSON") from None
allowed_fields = expected_fields | (optional_fields or set())
if (
not isinstance(payload, dict)
or not expected_fields.issubset(payload)
or not set(payload).issubset(allowed_fields)
):
raise invalid("Некорректные поля запроса")
return payload
+120
View File
@@ -0,0 +1,120 @@
services:
s3:
image: chrislusf/seaweedfs:4.44
entrypoint: ["weed"]
command:
- mini
- -dir=/data
- -ip=s3
- -ip.bind=0.0.0.0
- -admin.ui=false
- -bucket=messenger-attachments
- -s3.port=9000
- -s3.config=/run/secrets/s3_config
- -s3.iam=false
- -s3.port.iceberg=0
- -s3.port.lance=0
volumes:
- s3_data:/data
secrets:
- s3_config
cpus: "1.0"
mem_limit: 768m
stop_grace_period: 20s
restart: unless-stopped
s3-check:
build:
context: .
image: mtusi-messenger:lab4
command: ["python", "-m", "app.s3_check"]
environment:
S3_ENDPOINT: http://s3:9000
S3_REGION: us-east-1
S3_BUCKET: messenger-attachments
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
secrets:
- s3_access_key_id
- s3_secret_access_key
depends_on:
s3:
condition: service_started
healthcheck:
disable: true
cpus: "0.5"
mem_limit: 256m
restart: "no"
migrate:
image: mtusi-messenger:lab4
api:
image: mtusi-messenger:lab4
environment:
S3_ENDPOINT: http://s3:9000
S3_REGION: us-east-1
S3_BUCKET: messenger-attachments
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
secrets:
- s3_access_key_id
- s3_secret_access_key
depends_on:
s3-check:
condition: service_completed_successfully
cleanup:
build:
context: .
image: mtusi-messenger:lab4
command:
- python
- -m
- app.cleanup_orphans
- --apply
- --grace-seconds
- "86400"
- --interval-seconds
- "3600"
environment:
PGHOST: postgres
PGPORT: "5432"
PGDATABASE: messenger
PGUSER: messenger
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
S3_ENDPOINT: http://s3:9000
S3_REGION: us-east-1
S3_BUCKET: messenger-attachments
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
secrets:
- s3_access_key_id
- s3_secret_access_key
depends_on:
postgres:
condition: service_healthy
migrate:
condition: service_completed_successfully
s3-check:
condition: service_completed_successfully
api:
condition: service_started
healthcheck:
disable: true
cpus: "0.25"
mem_limit: 256m
stop_grace_period: 30s
restart: unless-stopped
volumes:
s3_data:
secrets:
s3_config:
file: ./secrets/s3_config.json
s3_access_key_id:
file: ./secrets/s3_access_key_id
s3_secret_access_key:
file: ./secrets/s3_secret_access_key
+132
View File
@@ -0,0 +1,132 @@
name: mtusi2026lab3
x-db-environment: &db-environment
PGHOST: postgres
PGPORT: "5432"
PGDATABASE: messenger
PGUSER: messenger
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
services:
postgres:
image: postgres:16.4-bookworm
environment:
POSTGRES_DB: messenger
POSTGRES_USER: messenger
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test:
- CMD-SHELL
- pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
interval: 2s
timeout: 2s
retries: 15
start_period: 20s
cpus: "1.0"
mem_limit: 512m
restart: unless-stopped
redis:
image: redis:7.2.5-bookworm
command:
- redis-server
- --appendonly
- "yes"
- --appendfsync
- everysec
- --maxmemory
- 64mb
- --maxmemory-policy
- allkeys-lru
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 2s
timeout: 2s
retries: 10
start_period: 5s
cpus: "0.25"
mem_limit: 128m
restart: unless-stopped
migrate:
build:
context: .
image: mtusi-messenger:lab3
command: ["python", "-m", "app.migrate"]
environment: *db-environment
depends_on:
postgres:
condition: service_healthy
healthcheck:
disable: true
cpus: "0.5"
mem_limit: 256m
restart: "no"
api:
build:
context: .
image: mtusi-messenger:lab3
command:
- python
- -m
- uvicorn
- app.main:app
- --host
- 0.0.0.0
- --port
- "8080"
- --workers
- "1"
- --timeout-graceful-shutdown
- "5"
- --no-access-log
environment:
<<: *db-environment
REDIS_HOST: redis
REDIS_PORT: "6379"
USER_CACHE_TTL_SECONDS: "${USER_CACHE_TTL_SECONDS:-60}"
SESSION_TTL_SECONDS: "${SESSION_TTL_SECONDS:-3600}"
depends_on:
postgres:
condition: service_healthy
migrate:
condition: service_completed_successfully
healthcheck:
test:
- CMD
- python
- -c
- "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"
interval: 2s
timeout: 3s
retries: 3
start_period: 10s
deploy:
replicas: 2
cpus: "0.75"
mem_limit: 512m
stop_grace_period: 10s
restart: unless-stopped
lb:
image: haproxy:3.0.5-bookworm
ports:
- "127.0.0.1:18082:8080"
volumes:
- ./infra/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
depends_on:
api:
condition: service_started
cpus: "0.25"
mem_limit: 128m
stop_grace_period: 10s
restart: unless-stopped
volumes:
postgres_data:
redis_data:
+41
View File
@@ -0,0 +1,41 @@
{
"scenario": "postgres_pause_history",
"passed": true,
"max_response_seconds": 6.0,
"baseline_instances": [
"3dbd75387914:92d7529650bf",
"b9966df7dc8d:a4e095d4b95b"
],
"paused_requests": [
{
"ok": true,
"status": 503,
"code": "unavailable",
"instance": "b9966df7dc8d:a4e095d4b95b",
"elapsed_s": 3.0877
},
{
"ok": true,
"status": 503,
"code": "unavailable",
"instance": "3dbd75387914:92d7529650bf",
"elapsed_s": 3.0881
},
{
"ok": true,
"status": 503,
"code": "unavailable",
"instance": "3dbd75387914:92d7529650bf",
"elapsed_s": 3.088
},
{
"ok": true,
"status": 503,
"code": "unavailable",
"instance": "b9966df7dc8d:a4e095d4b95b",
"elapsed_s": 3.0871
}
],
"recovery_observed_after_s": 1.0124,
"confirmed_message_preserved": true
}
+261
View File
@@ -0,0 +1,261 @@
VmHWM_kib,VmRSS_kib,VmSwap_kib,container,events.high,events.low,events.max,events.oom,events.oom_group_kill,events.oom_kill,memory.current,memory.max,memory.peak,phase,time_s
158960,93500,0,5d51a6717cdb,0,0,0,0,0,0,82649088,536870912,153268224,baseline,0.2429
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,84201472,536870912,152104960,baseline,0.2763
158960,93500,0,5d51a6717cdb,0,0,0,0,0,0,82780160,536870912,153268224,baseline,0.3433
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,92549120,536870912,152104960,baseline,0.3768
158960,93500,0,5d51a6717cdb,0,0,0,0,0,0,82780160,536870912,153268224,baseline,0.444
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,94384128,536870912,152104960,baseline,0.4775
158960,93500,0,5d51a6717cdb,0,0,0,0,0,0,82780160,536870912,153268224,baseline,0.5447
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,95694848,536870912,152104960,baseline,0.5782
158960,93500,0,5d51a6717cdb,0,0,0,0,0,0,82780160,536870912,153268224,baseline,0.6457
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82395136,536870912,152104960,baseline,0.6792
158960,93500,0,5d51a6717cdb,0,0,0,0,0,0,82780160,536870912,153268224,baseline,0.7468
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82382848,536870912,152104960,baseline,0.7807
158960,93500,0,5d51a6717cdb,0,0,0,0,0,0,82780160,536870912,153268224,baseline,0.8482
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82382848,536870912,152104960,baseline,0.8821
158960,93500,0,5d51a6717cdb,0,0,0,0,0,0,82784256,536870912,153268224,baseline,0.9483
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82382848,536870912,152104960,baseline,0.9833
158960,93500,0,5d51a6717cdb,0,0,0,0,0,0,84430848,536870912,153268224,single,1.0494
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82382848,536870912,152104960,single,1.0834
158960,130852,0,5d51a6717cdb,0,0,0,0,0,0,131358720,536870912,153268224,single,1.1497
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82382848,536870912,152104960,single,1.184
161700,161700,0,5d51a6717cdb,0,0,0,0,0,0,163098624,536870912,163098624,single,1.2802
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single,1.2844
177956,177956,0,5d51a6717cdb,0,0,0,0,0,0,180326400,536870912,180326400,single,1.3779
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single,1.385
217380,217380,0,5d51a6717cdb,0,0,0,0,0,0,221511680,536870912,221511680,single,1.4793
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single,1.4858
241444,241444,0,5d51a6717cdb,0,0,0,0,0,0,247201792,536870912,247201792,single,1.5802
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single,1.5862
247204,123364,0,5d51a6717cdb,0,0,0,0,0,0,129179648,536870912,252706816,single,1.6828
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single,1.6867
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,105279488,536870912,252706816,single_cooldown,1.7829
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single_cooldown,1.7875
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,105000960,536870912,252706816,single_cooldown,1.8834
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single_cooldown,1.8881
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,105005056,536870912,252706816,single_cooldown,1.9845
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single_cooldown,1.989
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,105005056,536870912,252706816,single_cooldown,2.0858
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single_cooldown,2.0904
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,2.1889
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82378752,536870912,152104960,single_cooldown,2.1921
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,2.2914
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82382848,536870912,152104960,single_cooldown,2.292
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,2.3915
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82382848,536870912,152104960,single_cooldown,2.3921
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,2.4929
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82382848,536870912,152104960,single_cooldown,2.4932
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,2.5932
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82640896,536870912,152104960,single_cooldown,2.5937
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,2.6936
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,84459520,536870912,152104960,single_cooldown,2.694
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,2.7943
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,93184000,536870912,152104960,single_cooldown,2.7945
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,2.8953
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,94494720,536870912,152104960,single_cooldown,2.8955
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,2.9953
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82804736,536870912,152104960,single_cooldown,2.9962
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82685952,536870912,152104960,single_cooldown,3.0972
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,3.0974
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82685952,536870912,152104960,single_cooldown,3.1984
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,3.1986
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,3.2998
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82673664,536870912,152104960,single_cooldown,3.3001
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82673664,536870912,152104960,single_cooldown,3.4009
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,3.4011
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82673664,536870912,152104960,single_cooldown,3.5024
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,3.5026
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,3.6028
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82673664,536870912,152104960,single_cooldown,3.6029
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,104988672,536870912,252706816,single_cooldown,3.7037
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82673664,536870912,152104960,single_cooldown,3.7038
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,107089920,536870912,252706816,concurrent_1,3.8074
158768,93308,0,eab2fd1a7047,0,0,0,0,0,0,82530304,536870912,152104960,concurrent_1,3.8075
247204,115024,0,5d51a6717cdb,0,0,0,0,0,0,115400704,536870912,252706816,concurrent_1,3.9051
158768,95740,0,eab2fd1a7047,0,0,0,0,0,0,86138880,536870912,152104960,concurrent_1,3.9083
247204,115152,0,5d51a6717cdb,0,0,0,0,0,0,116346880,536870912,252706816,concurrent_1,4.0056
158768,96508,0,eab2fd1a7047,0,0,0,0,0,0,87777280,536870912,152104960,concurrent_1,4.0088
247204,115920,0,5d51a6717cdb,0,0,0,0,0,0,118923264,536870912,252706816,concurrent_1,4.1063
158768,96508,0,eab2fd1a7047,0,0,0,0,0,0,88481792,536870912,152104960,concurrent_1,4.1093
158768,102652,0,eab2fd1a7047,0,0,0,0,0,0,98099200,536870912,152104960,concurrent_1,4.2202
247204,117584,0,5d51a6717cdb,0,0,0,0,0,0,122830848,536870912,252706816,concurrent_1,4.2242
247204,157520,0,5d51a6717cdb,0,0,0,0,0,0,150671360,536870912,252706816,concurrent_1,4.3071
158768,122656,0,eab2fd1a7047,0,0,0,0,0,0,120557568,536870912,152104960,concurrent_1,4.3103
247204,205648,0,5d51a6717cdb,0,0,0,0,0,0,201543680,536870912,252706816,concurrent_1,4.4077
179232,179232,0,eab2fd1a7047,0,0,0,0,0,0,179081216,536870912,179081216,concurrent_1,4.4105
253264,253264,0,5d51a6717cdb,0,0,0,0,0,0,250019840,536870912,252706816,concurrent_1,4.5082
234912,234912,0,eab2fd1a7047,0,0,0,0,0,0,236494848,536870912,236494848,concurrent_1,4.5112
254032,254032,0,5d51a6717cdb,0,0,0,0,0,0,248127488,536870912,252891136,concurrent_1,4.6088
246944,123084,0,eab2fd1a7047,0,0,0,0,0,0,121741312,536870912,248553472,concurrent_1,4.6142
254032,254032,0,5d51a6717cdb,0,0,0,0,0,0,248049664,536870912,252891136,concurrent_1,4.7093
246944,140088,0,eab2fd1a7047,0,0,0,0,0,0,135274496,536870912,248553472,concurrent_1,4.7137
254032,254032,0,5d51a6717cdb,0,0,0,0,0,0,247922688,536870912,252891136,concurrent_1,4.81
246944,210360,0,eab2fd1a7047,0,0,0,0,0,0,207364096,536870912,248553472,concurrent_1,4.8141
254032,254032,0,5d51a6717cdb,0,0,0,0,0,0,248074240,536870912,252891136,concurrent_1,4.9105
252088,252088,0,eab2fd1a7047,0,0,0,0,0,0,250093568,536870912,250093568,concurrent_1,4.9147
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,248000512,536870912,252891136,concurrent_1,5.011
252088,252088,0,eab2fd1a7047,0,0,0,0,0,0,249987072,536870912,253325312,concurrent_1,5.0152
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247992320,536870912,252891136,concurrent_1,5.1116
252088,252088,0,eab2fd1a7047,0,0,0,0,0,0,259850240,536870912,259850240,concurrent_1,5.1158
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,248004608,536870912,252891136,concurrent_1,5.212
252088,252088,0,eab2fd1a7047,0,0,0,0,0,0,261709824,536870912,264503296,concurrent_1,5.2164
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,252211200,536870912,252891136,concurrent_1,5.3177
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,260464640,536870912,264503296,concurrent_1,5.3723
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247869440,536870912,252891136,concurrent_1,5.417
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,260984832,536870912,264503296,concurrent_1,5.4786
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,252891136,concurrent_1,5.5182
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,261287936,536870912,264503296,concurrent_1,5.5774
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,252891136,concurrent_1,5.619
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,258494464,536870912,265211904,concurrent_1,5.6777
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,252891136,concurrent_1,5.7194
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,259059712,536870912,265211904,concurrent_1,5.7781
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,252891136,concurrent_1,5.8203
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,259596288,536870912,265211904,concurrent_1,5.8788
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,252891136,concurrent_1,5.9206
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245735424,536870912,265211904,concurrent_1,5.9792
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,248131584,536870912,252891136,concurrent_1,6.0213
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1,6.0798
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,252891136,concurrent_1_cooldown,6.122
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,6.1804
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,252891136,concurrent_1_cooldown,6.2233
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,6.2809
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,257802240,536870912,257802240,concurrent_1_cooldown,6.3235
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245727232,536870912,265211904,concurrent_1_cooldown,6.3873
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,257896448,536870912,257949696,concurrent_1_cooldown,6.424
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,6.4862
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,258682880,536870912,258682880,concurrent_1_cooldown,6.5288
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,6.5868
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,259731456,536870912,259731456,concurrent_1_cooldown,6.6279
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,6.6873
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,255758336,536870912,261570560,concurrent_1_cooldown,6.7285
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,6.7882
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,261570560,concurrent_1_cooldown,6.8292
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,6.8892
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,261570560,concurrent_1_cooldown,6.9307
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,6.9902
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,261570560,concurrent_1_cooldown,7.0312
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,7.0916
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,261570560,concurrent_1_cooldown,7.1323
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,7.1923
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,261570560,concurrent_1_cooldown,7.2335
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,7.2927
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247869440,536870912,261570560,concurrent_1_cooldown,7.3336
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,7.3936
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,261570560,concurrent_1_cooldown,7.4354
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245735424,536870912,265211904,concurrent_1_cooldown,7.4949
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,261570560,concurrent_1_cooldown,7.5363
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245735424,536870912,265211904,concurrent_1_cooldown,7.5964
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247865344,536870912,261570560,concurrent_1_cooldown,7.638
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245727232,536870912,265211904,concurrent_1_cooldown,7.6967
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247861248,536870912,261570560,concurrent_1_cooldown,7.7387
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,7.798
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247861248,536870912,261570560,concurrent_1_cooldown,7.8395
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,245723136,536870912,265211904,concurrent_1_cooldown,7.8994
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247861248,536870912,261570560,concurrent_1_cooldown,7.9396
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,255762432,536870912,265211904,concurrent_1_cooldown,7.9995
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247861248,536870912,261570560,concurrent_1_cooldown,8.0402
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,256831488,536870912,265211904,concurrent_1_cooldown,8.1
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,247984128,536870912,261570560,concurrent_2,8.1406
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,258723840,536870912,265211904,concurrent_2,8.2044
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,249352192,536870912,261570560,concurrent_2,8.2411
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,260366336,536870912,265211904,concurrent_2,8.3047
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,249823232,536870912,261570560,concurrent_2,8.3417
252344,252344,0,eab2fd1a7047,0,0,0,0,0,0,248934400,536870912,265211904,concurrent_2,8.4062
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,251101184,536870912,261570560,concurrent_2,8.4423
252472,252472,0,eab2fd1a7047,0,0,0,0,0,0,251555840,536870912,265211904,concurrent_2,8.5069
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,252878848,536870912,261570560,concurrent_2,8.5432
252472,252472,0,eab2fd1a7047,0,0,0,0,0,0,253673472,536870912,265211904,concurrent_2,8.6083
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,253616128,536870912,261570560,concurrent_2,8.6448
252472,252472,0,eab2fd1a7047,0,0,0,0,0,0,255766528,536870912,265211904,concurrent_2,8.7092
254288,254288,0,5d51a6717cdb,0,0,0,0,0,0,254156800,536870912,261570560,concurrent_2,8.745
253112,253112,0,eab2fd1a7047,0,0,0,0,0,0,256135168,536870912,265211904,concurrent_2,8.8113
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,265682944,536870912,265682944,concurrent_2,8.8457
253112,253112,0,eab2fd1a7047,0,0,0,0,0,0,256057344,536870912,265211904,concurrent_2,8.9119
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,266375168,536870912,266379264,concurrent_2,8.9462
253112,253112,0,eab2fd1a7047,0,0,0,0,0,0,255188992,536870912,265211904,concurrent_2,9.0125
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,267030528,536870912,270118912,concurrent_2,9.0506
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,257814528,536870912,265211904,concurrent_2,9.1193
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,270254080,536870912,270274560,concurrent_2,9.1802
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,255205376,536870912,265211904,concurrent_2,9.2179
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,269029376,536870912,272986112,concurrent_2,9.2873
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,255209472,536870912,265211904,concurrent_2,9.3185
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,269139968,536870912,272986112,concurrent_2,9.3857
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,251457536,536870912,265211904,concurrent_2,9.4193
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,270393344,536870912,272986112,concurrent_2,9.4864
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,252125184,536870912,265211904,concurrent_2,9.52
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,270344192,536870912,272986112,concurrent_2,9.5918
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,250912768,536870912,265211904,concurrent_2,9.6205
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,270364672,536870912,272986112,concurrent_2,9.6918
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,250904576,536870912,265211904,concurrent_2,9.7211
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,260550656,536870912,273399808,concurrent_2,9.7982
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,250912768,536870912,265211904,concurrent_2,9.8216
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,256495616,536870912,273399808,concurrent_2,9.8968
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246960128,536870912,265211904,concurrent_2,9.9223
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,252379136,536870912,273399808,concurrent_2,9.9973
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246480896,536870912,265211904,concurrent_2,10.0226
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,252583936,536870912,273399808,concurrent_2,10.0978
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246480896,536870912,265211904,concurrent_2,10.1233
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249761792,536870912,273399808,concurrent_2,10.1985
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246480896,536870912,265211904,concurrent_2,10.2239
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,252166144,536870912,273399808,concurrent_2,10.2996
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2,10.3284
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249610240,536870912,273399808,concurrent_2,10.4047
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,247140352,536870912,265211904,concurrent_2_cooldown,10.4412
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,10.5099
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,256045056,536870912,265211904,concurrent_2_cooldown,10.5714
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,10.6168
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,256565248,536870912,265211904,concurrent_2_cooldown,10.6725
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249356288,536870912,273399808,concurrent_2_cooldown,10.7162
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,257880064,536870912,265211904,concurrent_2_cooldown,10.7728
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249352192,536870912,273399808,concurrent_2_cooldown,10.8167
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,259715072,536870912,265211904,concurrent_2_cooldown,10.8735
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,10.9174
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246505472,536870912,265211904,concurrent_2_cooldown,10.9743
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,11.0193
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246505472,536870912,265211904,concurrent_2_cooldown,11.0762
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,11.1203
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246501376,536870912,265211904,concurrent_2_cooldown,11.177
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,11.2203
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246501376,536870912,265211904,concurrent_2_cooldown,11.2775
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,11.3221
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246501376,536870912,265211904,concurrent_2_cooldown,11.379
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,11.4232
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2_cooldown,11.4801
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,11.5237
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2_cooldown,11.5817
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,concurrent_2_cooldown,11.6254
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246484992,536870912,265211904,concurrent_2_cooldown,11.683
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249356288,536870912,273399808,concurrent_2_cooldown,11.7271
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2_cooldown,11.7827
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,259432448,536870912,273399808,concurrent_2_cooldown,11.8261
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2_cooldown,11.8836
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,260481024,536870912,273399808,concurrent_2_cooldown,11.9266
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2_cooldown,11.9842
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,262840320,536870912,273399808,concurrent_2_cooldown,12.0271
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2_cooldown,12.0846
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249352192,536870912,273399808,concurrent_2_cooldown,12.1283
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2_cooldown,12.1863
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249352192,536870912,273399808,concurrent_2_cooldown,12.2288
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2_cooldown,12.2867
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249352192,536870912,273399808,concurrent_2_cooldown,12.3293
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,concurrent_2_cooldown,12.3877
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249352192,536870912,273399808,final,12.4308
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,final,12.4893
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249352192,536870912,273399808,final,12.5316
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,final,12.5894
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,final,12.6315
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,final,12.6903
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249352192,536870912,273399808,final,12.7328
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,final,12.791
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249344000,536870912,273399808,final,12.8329
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246476800,536870912,265211904,final,12.8919
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,final,12.9336
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,255774720,536870912,265211904,final,12.9924
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,final,13.0343
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,257429504,536870912,265211904,final,13.0928
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,final,13.1346
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,259530752,536870912,265211904,final,13.1934
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,final,13.2352
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246800384,536870912,265211904,final,13.2941
255696,255696,0,5d51a6717cdb,0,0,0,0,0,0,249348096,536870912,273399808,final,13.3368
253240,253240,0,eab2fd1a7047,0,0,0,0,0,0,246800384,536870912,265211904,final,13.3954
1 VmHWM_kib VmRSS_kib VmSwap_kib container events.high events.low events.max events.oom events.oom_group_kill events.oom_kill memory.current memory.max memory.peak phase time_s
2 158960 93500 0 5d51a6717cdb 0 0 0 0 0 0 82649088 536870912 153268224 baseline 0.2429
3 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 84201472 536870912 152104960 baseline 0.2763
4 158960 93500 0 5d51a6717cdb 0 0 0 0 0 0 82780160 536870912 153268224 baseline 0.3433
5 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 92549120 536870912 152104960 baseline 0.3768
6 158960 93500 0 5d51a6717cdb 0 0 0 0 0 0 82780160 536870912 153268224 baseline 0.444
7 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 94384128 536870912 152104960 baseline 0.4775
8 158960 93500 0 5d51a6717cdb 0 0 0 0 0 0 82780160 536870912 153268224 baseline 0.5447
9 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 95694848 536870912 152104960 baseline 0.5782
10 158960 93500 0 5d51a6717cdb 0 0 0 0 0 0 82780160 536870912 153268224 baseline 0.6457
11 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82395136 536870912 152104960 baseline 0.6792
12 158960 93500 0 5d51a6717cdb 0 0 0 0 0 0 82780160 536870912 153268224 baseline 0.7468
13 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82382848 536870912 152104960 baseline 0.7807
14 158960 93500 0 5d51a6717cdb 0 0 0 0 0 0 82780160 536870912 153268224 baseline 0.8482
15 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82382848 536870912 152104960 baseline 0.8821
16 158960 93500 0 5d51a6717cdb 0 0 0 0 0 0 82784256 536870912 153268224 baseline 0.9483
17 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82382848 536870912 152104960 baseline 0.9833
18 158960 93500 0 5d51a6717cdb 0 0 0 0 0 0 84430848 536870912 153268224 single 1.0494
19 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82382848 536870912 152104960 single 1.0834
20 158960 130852 0 5d51a6717cdb 0 0 0 0 0 0 131358720 536870912 153268224 single 1.1497
21 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82382848 536870912 152104960 single 1.184
22 161700 161700 0 5d51a6717cdb 0 0 0 0 0 0 163098624 536870912 163098624 single 1.2802
23 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single 1.2844
24 177956 177956 0 5d51a6717cdb 0 0 0 0 0 0 180326400 536870912 180326400 single 1.3779
25 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single 1.385
26 217380 217380 0 5d51a6717cdb 0 0 0 0 0 0 221511680 536870912 221511680 single 1.4793
27 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single 1.4858
28 241444 241444 0 5d51a6717cdb 0 0 0 0 0 0 247201792 536870912 247201792 single 1.5802
29 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single 1.5862
30 247204 123364 0 5d51a6717cdb 0 0 0 0 0 0 129179648 536870912 252706816 single 1.6828
31 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single 1.6867
32 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 105279488 536870912 252706816 single_cooldown 1.7829
33 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single_cooldown 1.7875
34 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 105000960 536870912 252706816 single_cooldown 1.8834
35 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single_cooldown 1.8881
36 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 105005056 536870912 252706816 single_cooldown 1.9845
37 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single_cooldown 1.989
38 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 105005056 536870912 252706816 single_cooldown 2.0858
39 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single_cooldown 2.0904
40 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 2.1889
41 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82378752 536870912 152104960 single_cooldown 2.1921
42 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 2.2914
43 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82382848 536870912 152104960 single_cooldown 2.292
44 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 2.3915
45 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82382848 536870912 152104960 single_cooldown 2.3921
46 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 2.4929
47 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82382848 536870912 152104960 single_cooldown 2.4932
48 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 2.5932
49 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82640896 536870912 152104960 single_cooldown 2.5937
50 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 2.6936
51 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 84459520 536870912 152104960 single_cooldown 2.694
52 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 2.7943
53 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 93184000 536870912 152104960 single_cooldown 2.7945
54 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 2.8953
55 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 94494720 536870912 152104960 single_cooldown 2.8955
56 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 2.9953
57 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82804736 536870912 152104960 single_cooldown 2.9962
58 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82685952 536870912 152104960 single_cooldown 3.0972
59 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 3.0974
60 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82685952 536870912 152104960 single_cooldown 3.1984
61 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 3.1986
62 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 3.2998
63 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82673664 536870912 152104960 single_cooldown 3.3001
64 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82673664 536870912 152104960 single_cooldown 3.4009
65 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 3.4011
66 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82673664 536870912 152104960 single_cooldown 3.5024
67 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 3.5026
68 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 3.6028
69 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82673664 536870912 152104960 single_cooldown 3.6029
70 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 104988672 536870912 252706816 single_cooldown 3.7037
71 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82673664 536870912 152104960 single_cooldown 3.7038
72 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 107089920 536870912 252706816 concurrent_1 3.8074
73 158768 93308 0 eab2fd1a7047 0 0 0 0 0 0 82530304 536870912 152104960 concurrent_1 3.8075
74 247204 115024 0 5d51a6717cdb 0 0 0 0 0 0 115400704 536870912 252706816 concurrent_1 3.9051
75 158768 95740 0 eab2fd1a7047 0 0 0 0 0 0 86138880 536870912 152104960 concurrent_1 3.9083
76 247204 115152 0 5d51a6717cdb 0 0 0 0 0 0 116346880 536870912 252706816 concurrent_1 4.0056
77 158768 96508 0 eab2fd1a7047 0 0 0 0 0 0 87777280 536870912 152104960 concurrent_1 4.0088
78 247204 115920 0 5d51a6717cdb 0 0 0 0 0 0 118923264 536870912 252706816 concurrent_1 4.1063
79 158768 96508 0 eab2fd1a7047 0 0 0 0 0 0 88481792 536870912 152104960 concurrent_1 4.1093
80 158768 102652 0 eab2fd1a7047 0 0 0 0 0 0 98099200 536870912 152104960 concurrent_1 4.2202
81 247204 117584 0 5d51a6717cdb 0 0 0 0 0 0 122830848 536870912 252706816 concurrent_1 4.2242
82 247204 157520 0 5d51a6717cdb 0 0 0 0 0 0 150671360 536870912 252706816 concurrent_1 4.3071
83 158768 122656 0 eab2fd1a7047 0 0 0 0 0 0 120557568 536870912 152104960 concurrent_1 4.3103
84 247204 205648 0 5d51a6717cdb 0 0 0 0 0 0 201543680 536870912 252706816 concurrent_1 4.4077
85 179232 179232 0 eab2fd1a7047 0 0 0 0 0 0 179081216 536870912 179081216 concurrent_1 4.4105
86 253264 253264 0 5d51a6717cdb 0 0 0 0 0 0 250019840 536870912 252706816 concurrent_1 4.5082
87 234912 234912 0 eab2fd1a7047 0 0 0 0 0 0 236494848 536870912 236494848 concurrent_1 4.5112
88 254032 254032 0 5d51a6717cdb 0 0 0 0 0 0 248127488 536870912 252891136 concurrent_1 4.6088
89 246944 123084 0 eab2fd1a7047 0 0 0 0 0 0 121741312 536870912 248553472 concurrent_1 4.6142
90 254032 254032 0 5d51a6717cdb 0 0 0 0 0 0 248049664 536870912 252891136 concurrent_1 4.7093
91 246944 140088 0 eab2fd1a7047 0 0 0 0 0 0 135274496 536870912 248553472 concurrent_1 4.7137
92 254032 254032 0 5d51a6717cdb 0 0 0 0 0 0 247922688 536870912 252891136 concurrent_1 4.81
93 246944 210360 0 eab2fd1a7047 0 0 0 0 0 0 207364096 536870912 248553472 concurrent_1 4.8141
94 254032 254032 0 5d51a6717cdb 0 0 0 0 0 0 248074240 536870912 252891136 concurrent_1 4.9105
95 252088 252088 0 eab2fd1a7047 0 0 0 0 0 0 250093568 536870912 250093568 concurrent_1 4.9147
96 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 248000512 536870912 252891136 concurrent_1 5.011
97 252088 252088 0 eab2fd1a7047 0 0 0 0 0 0 249987072 536870912 253325312 concurrent_1 5.0152
98 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247992320 536870912 252891136 concurrent_1 5.1116
99 252088 252088 0 eab2fd1a7047 0 0 0 0 0 0 259850240 536870912 259850240 concurrent_1 5.1158
100 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 248004608 536870912 252891136 concurrent_1 5.212
101 252088 252088 0 eab2fd1a7047 0 0 0 0 0 0 261709824 536870912 264503296 concurrent_1 5.2164
102 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 252211200 536870912 252891136 concurrent_1 5.3177
103 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 260464640 536870912 264503296 concurrent_1 5.3723
104 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247869440 536870912 252891136 concurrent_1 5.417
105 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 260984832 536870912 264503296 concurrent_1 5.4786
106 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 252891136 concurrent_1 5.5182
107 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 261287936 536870912 264503296 concurrent_1 5.5774
108 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 252891136 concurrent_1 5.619
109 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 258494464 536870912 265211904 concurrent_1 5.6777
110 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 252891136 concurrent_1 5.7194
111 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 259059712 536870912 265211904 concurrent_1 5.7781
112 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 252891136 concurrent_1 5.8203
113 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 259596288 536870912 265211904 concurrent_1 5.8788
114 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 252891136 concurrent_1 5.9206
115 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245735424 536870912 265211904 concurrent_1 5.9792
116 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 248131584 536870912 252891136 concurrent_1 6.0213
117 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1 6.0798
118 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 252891136 concurrent_1_cooldown 6.122
119 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 6.1804
120 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 252891136 concurrent_1_cooldown 6.2233
121 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 6.2809
122 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 257802240 536870912 257802240 concurrent_1_cooldown 6.3235
123 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245727232 536870912 265211904 concurrent_1_cooldown 6.3873
124 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 257896448 536870912 257949696 concurrent_1_cooldown 6.424
125 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 6.4862
126 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 258682880 536870912 258682880 concurrent_1_cooldown 6.5288
127 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 6.5868
128 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 259731456 536870912 259731456 concurrent_1_cooldown 6.6279
129 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 6.6873
130 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 255758336 536870912 261570560 concurrent_1_cooldown 6.7285
131 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 6.7882
132 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 261570560 concurrent_1_cooldown 6.8292
133 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 6.8892
134 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 261570560 concurrent_1_cooldown 6.9307
135 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 6.9902
136 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 261570560 concurrent_1_cooldown 7.0312
137 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 7.0916
138 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 261570560 concurrent_1_cooldown 7.1323
139 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 7.1923
140 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 261570560 concurrent_1_cooldown 7.2335
141 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 7.2927
142 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247869440 536870912 261570560 concurrent_1_cooldown 7.3336
143 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 7.3936
144 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 261570560 concurrent_1_cooldown 7.4354
145 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245735424 536870912 265211904 concurrent_1_cooldown 7.4949
146 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 261570560 concurrent_1_cooldown 7.5363
147 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245735424 536870912 265211904 concurrent_1_cooldown 7.5964
148 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247865344 536870912 261570560 concurrent_1_cooldown 7.638
149 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245727232 536870912 265211904 concurrent_1_cooldown 7.6967
150 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247861248 536870912 261570560 concurrent_1_cooldown 7.7387
151 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 7.798
152 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247861248 536870912 261570560 concurrent_1_cooldown 7.8395
153 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 245723136 536870912 265211904 concurrent_1_cooldown 7.8994
154 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247861248 536870912 261570560 concurrent_1_cooldown 7.9396
155 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 255762432 536870912 265211904 concurrent_1_cooldown 7.9995
156 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247861248 536870912 261570560 concurrent_1_cooldown 8.0402
157 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 256831488 536870912 265211904 concurrent_1_cooldown 8.1
158 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 247984128 536870912 261570560 concurrent_2 8.1406
159 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 258723840 536870912 265211904 concurrent_2 8.2044
160 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 249352192 536870912 261570560 concurrent_2 8.2411
161 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 260366336 536870912 265211904 concurrent_2 8.3047
162 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 249823232 536870912 261570560 concurrent_2 8.3417
163 252344 252344 0 eab2fd1a7047 0 0 0 0 0 0 248934400 536870912 265211904 concurrent_2 8.4062
164 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 251101184 536870912 261570560 concurrent_2 8.4423
165 252472 252472 0 eab2fd1a7047 0 0 0 0 0 0 251555840 536870912 265211904 concurrent_2 8.5069
166 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 252878848 536870912 261570560 concurrent_2 8.5432
167 252472 252472 0 eab2fd1a7047 0 0 0 0 0 0 253673472 536870912 265211904 concurrent_2 8.6083
168 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 253616128 536870912 261570560 concurrent_2 8.6448
169 252472 252472 0 eab2fd1a7047 0 0 0 0 0 0 255766528 536870912 265211904 concurrent_2 8.7092
170 254288 254288 0 5d51a6717cdb 0 0 0 0 0 0 254156800 536870912 261570560 concurrent_2 8.745
171 253112 253112 0 eab2fd1a7047 0 0 0 0 0 0 256135168 536870912 265211904 concurrent_2 8.8113
172 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 265682944 536870912 265682944 concurrent_2 8.8457
173 253112 253112 0 eab2fd1a7047 0 0 0 0 0 0 256057344 536870912 265211904 concurrent_2 8.9119
174 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 266375168 536870912 266379264 concurrent_2 8.9462
175 253112 253112 0 eab2fd1a7047 0 0 0 0 0 0 255188992 536870912 265211904 concurrent_2 9.0125
176 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 267030528 536870912 270118912 concurrent_2 9.0506
177 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 257814528 536870912 265211904 concurrent_2 9.1193
178 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 270254080 536870912 270274560 concurrent_2 9.1802
179 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 255205376 536870912 265211904 concurrent_2 9.2179
180 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 269029376 536870912 272986112 concurrent_2 9.2873
181 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 255209472 536870912 265211904 concurrent_2 9.3185
182 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 269139968 536870912 272986112 concurrent_2 9.3857
183 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 251457536 536870912 265211904 concurrent_2 9.4193
184 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 270393344 536870912 272986112 concurrent_2 9.4864
185 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 252125184 536870912 265211904 concurrent_2 9.52
186 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 270344192 536870912 272986112 concurrent_2 9.5918
187 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 250912768 536870912 265211904 concurrent_2 9.6205
188 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 270364672 536870912 272986112 concurrent_2 9.6918
189 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 250904576 536870912 265211904 concurrent_2 9.7211
190 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 260550656 536870912 273399808 concurrent_2 9.7982
191 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 250912768 536870912 265211904 concurrent_2 9.8216
192 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 256495616 536870912 273399808 concurrent_2 9.8968
193 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246960128 536870912 265211904 concurrent_2 9.9223
194 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 252379136 536870912 273399808 concurrent_2 9.9973
195 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246480896 536870912 265211904 concurrent_2 10.0226
196 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 252583936 536870912 273399808 concurrent_2 10.0978
197 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246480896 536870912 265211904 concurrent_2 10.1233
198 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249761792 536870912 273399808 concurrent_2 10.1985
199 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246480896 536870912 265211904 concurrent_2 10.2239
200 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 252166144 536870912 273399808 concurrent_2 10.2996
201 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2 10.3284
202 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249610240 536870912 273399808 concurrent_2 10.4047
203 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 247140352 536870912 265211904 concurrent_2_cooldown 10.4412
204 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 10.5099
205 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 256045056 536870912 265211904 concurrent_2_cooldown 10.5714
206 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 10.6168
207 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 256565248 536870912 265211904 concurrent_2_cooldown 10.6725
208 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249356288 536870912 273399808 concurrent_2_cooldown 10.7162
209 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 257880064 536870912 265211904 concurrent_2_cooldown 10.7728
210 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249352192 536870912 273399808 concurrent_2_cooldown 10.8167
211 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 259715072 536870912 265211904 concurrent_2_cooldown 10.8735
212 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 10.9174
213 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246505472 536870912 265211904 concurrent_2_cooldown 10.9743
214 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 11.0193
215 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246505472 536870912 265211904 concurrent_2_cooldown 11.0762
216 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 11.1203
217 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246501376 536870912 265211904 concurrent_2_cooldown 11.177
218 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 11.2203
219 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246501376 536870912 265211904 concurrent_2_cooldown 11.2775
220 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 11.3221
221 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246501376 536870912 265211904 concurrent_2_cooldown 11.379
222 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 11.4232
223 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2_cooldown 11.4801
224 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 11.5237
225 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2_cooldown 11.5817
226 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 concurrent_2_cooldown 11.6254
227 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246484992 536870912 265211904 concurrent_2_cooldown 11.683
228 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249356288 536870912 273399808 concurrent_2_cooldown 11.7271
229 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2_cooldown 11.7827
230 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 259432448 536870912 273399808 concurrent_2_cooldown 11.8261
231 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2_cooldown 11.8836
232 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 260481024 536870912 273399808 concurrent_2_cooldown 11.9266
233 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2_cooldown 11.9842
234 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 262840320 536870912 273399808 concurrent_2_cooldown 12.0271
235 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2_cooldown 12.0846
236 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249352192 536870912 273399808 concurrent_2_cooldown 12.1283
237 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2_cooldown 12.1863
238 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249352192 536870912 273399808 concurrent_2_cooldown 12.2288
239 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2_cooldown 12.2867
240 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249352192 536870912 273399808 concurrent_2_cooldown 12.3293
241 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 concurrent_2_cooldown 12.3877
242 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249352192 536870912 273399808 final 12.4308
243 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 final 12.4893
244 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249352192 536870912 273399808 final 12.5316
245 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 final 12.5894
246 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 final 12.6315
247 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 final 12.6903
248 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249352192 536870912 273399808 final 12.7328
249 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 final 12.791
250 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249344000 536870912 273399808 final 12.8329
251 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246476800 536870912 265211904 final 12.8919
252 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 final 12.9336
253 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 255774720 536870912 265211904 final 12.9924
254 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 final 13.0343
255 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 257429504 536870912 265211904 final 13.0928
256 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 final 13.1346
257 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 259530752 536870912 265211904 final 13.1934
258 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 final 13.2352
259 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246800384 536870912 265211904 final 13.2941
260 255696 255696 0 5d51a6717cdb 0 0 0 0 0 0 249348096 536870912 273399808 final 13.3368
261 253240 253240 0 eab2fd1a7047 0 0 0 0 0 0 246800384 536870912 265211904 final 13.3954
+241
View File
@@ -0,0 +1,241 @@
{
"fixture_bytes": 8540172,
"width": 6000,
"height": 6000,
"fixture_sha256": "372c13b4460b8236bf1051531bfd79e5d167145631c11a169b2c87e2aaabffbe",
"before": {
"5d51a6717cdb": {
"started_at": "2026-11-05T15:11:04.828036051Z",
"oom_killed": false,
"status": "running",
"restart_count": 0
},
"eab2fd1a7047": {
"started_at": "2026-11-05T15:11:04.447199657Z",
"oom_killed": false,
"status": "running",
"restart_count": 0
}
},
"after": {
"5d51a6717cdb": {
"started_at": "2026-11-05T15:11:04.828036051Z",
"oom_killed": false,
"status": "running",
"restart_count": 0
},
"eab2fd1a7047": {
"started_at": "2026-11-05T15:11:04.447199657Z",
"oom_killed": false,
"status": "running",
"restart_count": 0
}
},
"rounds": [
{
"label": "single",
"clients": 1,
"peak_client_in_flight": 1,
"elapsed_seconds": 0.7324,
"statuses": {
"201": 1
},
"requests": [
{
"number": 0,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 0.732
}
]
},
{
"label": "concurrent_1",
"clients": 10,
"peak_client_in_flight": 10,
"elapsed_seconds": 2.3296,
"statuses": {
"201": 10
},
"requests": [
{
"number": 0,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 1.8853
},
{
"number": 1,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 1.0085
},
{
"number": 2,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 0.897
},
{
"number": 3,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 1.1974
},
{
"number": 4,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 1.5063
},
{
"number": 5,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 1.3843
},
{
"number": 6,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 1.7244
},
{
"number": 7,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 2.1654
},
{
"number": 8,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 0.7962
},
{
"number": 9,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 2.3266
}
]
},
{
"label": "concurrent_2",
"clients": 10,
"peak_client_in_flight": 10,
"elapsed_seconds": 2.2903,
"statuses": {
"201": 10
},
"requests": [
{
"number": 0,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 1.4478
},
{
"number": 1,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 2.0626
},
{
"number": 2,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 1.045
},
{
"number": 3,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 2.2898
},
{
"number": 4,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 1.8708
},
{
"number": 5,
"status": 201,
"instance": "5d51a6717cdb:d87adb179b67",
"valid_metadata": true,
"seconds": 1.6656
},
{
"number": 6,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 1.2713
},
{
"number": 7,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 1.9693
},
{
"number": 8,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 1.7763
},
{
"number": 9,
"status": 201,
"instance": "eab2fd1a7047:65c91f8f9b16",
"valid_metadata": true,
"seconds": 0.8392
}
]
}
],
"monitor_errors": [],
"memory_summary": {
"5d51a6717cdb": {
"samples": 130,
"initial_cgroup_peak_bytes": 153268224,
"final_cgroup_peak_bytes": 273399808,
"sampled_max_cgroup_bytes": 270393344,
"sampled_max_rss_kib": 255696,
"initial_hwm_kib": 158960,
"final_hwm_kib": 255696,
"oom_delta": 0,
"oom_kill_delta": 0
},
"eab2fd1a7047": {
"samples": 130,
"initial_cgroup_peak_bytes": 152104960,
"final_cgroup_peak_bytes": 265211904,
"sampled_max_cgroup_bytes": 261709824,
"sampled_max_rss_kib": 253240,
"initial_hwm_kib": 158768,
"final_hwm_kib": 253240,
"oom_delta": 0,
"oom_kill_delta": 0
}
}
}
+42
View File
@@ -0,0 +1,42 @@
global
log stdout format raw local0
maxconn 512
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 1s
timeout client 30s
timeout server 30s
timeout http-request 10s
timeout http-keep-alive 5s
timeout queue 3s
timeout check 1s
retries 0
resolvers docker
nameserver dns 127.0.0.11:53
accepted_payload_size 8192
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold obsolete 1s
hold nx 1s
hold timeout 1s
hold refused 1s
hold other 1s
frontend public_api
bind :8080
default_backend api_replicas
backend api_replicas
balance roundrobin
option httpchk
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host localhost
http-check expect status 200
default-server inter 1s fastinter 500ms downinter 1s fall 2 rise 1
server-template api 1-4 api:8080 check resolvers docker resolve-prefer ipv4 init-addr none
+71
View File
@@ -0,0 +1,71 @@
CREATE TABLE users (
id UUID PRIMARY KEY,
username TEXT NOT NULL,
display_name TEXT NOT NULL,
password_hash TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
CONSTRAINT users_username_unique UNIQUE (username),
CONSTRAINT users_username_format
CHECK (username ~ '^[a-z0-9_]{3,32}$'),
CONSTRAINT users_display_name_length
CHECK (char_length(display_name) BETWEEN 1 AND 100)
);
CREATE TABLE chats (
id UUID PRIMARY KEY,
title TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
CONSTRAINT chats_title_length
CHECK (char_length(title) BETWEEN 1 AND 100)
);
CREATE TABLE chat_members (
chat_id UUID NOT NULL REFERENCES chats(id),
user_id UUID NOT NULL REFERENCES users(id),
PRIMARY KEY (chat_id, user_id)
);
CREATE INDEX chat_members_user_chat_idx
ON chat_members (user_id, chat_id);
CREATE INDEX chats_created_id_idx
ON chats (created_at, id);
CREATE TABLE messages (
id UUID PRIMARY KEY,
chat_id UUID NOT NULL,
sender_id UUID NOT NULL,
text TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
CONSTRAINT messages_sender_membership_fk
FOREIGN KEY (chat_id, sender_id)
REFERENCES chat_members (chat_id, user_id),
CONSTRAINT messages_text_length
CHECK (char_length(text) BETWEEN 1 AND 4000)
);
CREATE INDEX messages_chat_created_id_idx
ON messages (chat_id, created_at, id);
CREATE TABLE sessions (
token_hash TEXT PRIMARY KEY,
user_id UUID NOT NULL REFERENCES users(id),
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
expires_at TIMESTAMPTZ NOT NULL,
CONSTRAINT sessions_token_hash_format
CHECK (token_hash ~ '^[0-9a-f]{64}$'),
CONSTRAINT sessions_expiry_order
CHECK (expires_at > created_at)
);
CREATE INDEX sessions_expires_at_idx
ON sessions (expires_at);
CREATE INDEX sessions_user_id_idx
ON sessions (user_id);
+22
View File
@@ -0,0 +1,22 @@
CREATE TABLE message_idempotency (
user_id UUID NOT NULL REFERENCES users(id),
chat_id UUID NOT NULL REFERENCES chats(id),
http_method TEXT NOT NULL DEFAULT 'POST',
idempotency_key TEXT NOT NULL,
request_text TEXT NOT NULL,
message_id UUID NOT NULL REFERENCES messages(id),
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
PRIMARY KEY (
user_id, chat_id, http_method, idempotency_key
),
CONSTRAINT message_idempotency_method
CHECK (http_method = 'POST'),
CONSTRAINT message_idempotency_key_format
CHECK (
char_length(idempotency_key) BETWEEN 1 AND 128
AND idempotency_key ~ '^[A-Za-z0-9._:-]+$'
)
);
+67
View File
@@ -0,0 +1,67 @@
CREATE TABLE attachments (
id UUID PRIMARY KEY,
chat_id UUID NOT NULL REFERENCES chats(id),
owner_id UUID NOT NULL REFERENCES users(id),
filename TEXT NOT NULL,
content_type TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
sha256 TEXT NOT NULL,
object_key TEXT NOT NULL UNIQUE,
status TEXT NOT NULL DEFAULT 'ready',
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
CONSTRAINT attachments_filename_length
CHECK (char_length(filename) BETWEEN 1 AND 255),
CONSTRAINT attachments_content_type
CHECK (content_type IN ('image/png', 'image/jpeg')),
CONSTRAINT attachments_size_bytes
CHECK (size_bytes BETWEEN 1 AND 10485760),
CONSTRAINT attachments_sha256_format
CHECK (sha256 ~ '^[a-f0-9]{64}$'),
CONSTRAINT attachments_status
CHECK (status = 'ready'),
CONSTRAINT attachments_owner_membership_fk
FOREIGN KEY (chat_id, owner_id)
REFERENCES chat_members (chat_id, user_id)
);
CREATE INDEX attachments_chat_created_id_idx
ON attachments (chat_id, created_at, id);
CREATE TABLE message_attachments (
message_id UUID NOT NULL REFERENCES messages(id),
attachment_id UUID NOT NULL REFERENCES attachments(id),
position SMALLINT NOT NULL,
PRIMARY KEY (message_id, attachment_id),
CONSTRAINT message_attachments_position_unique
UNIQUE (message_id, position),
CONSTRAINT message_attachments_position_range
CHECK (position BETWEEN 0 AND 9)
);
CREATE INDEX message_attachments_attachment_idx
ON message_attachments (attachment_id);
ALTER TABLE messages
DROP CONSTRAINT messages_text_length;
ALTER TABLE messages
ADD CONSTRAINT messages_text_length
CHECK (char_length(text) BETWEEN 0 AND 4000);
ALTER TABLE message_idempotency
ADD COLUMN request_attachment_ids UUID[] NOT NULL
DEFAULT ARRAY[]::UUID[];
ALTER TABLE message_idempotency
ADD CONSTRAINT message_idempotency_attachment_count
CHECK (cardinality(request_attachment_ids) <= 10);
+8
View File
@@ -0,0 +1,8 @@
fastapi==0.115.12
uvicorn[standard]==0.34.2
psycopg[binary]==3.2.6
psycopg-pool==3.2.6
redis==5.2.1
argon2-cffi==23.1.0
boto3==1.43.124
Pillow==13.0.0
+36
View File
@@ -0,0 +1,36 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
compose() {
docker compose \
-f compose.yaml \
-f compose.s3.yaml \
--env-file .env \
"$@"
}
echo "Stopping LB and API..."
compose stop --timeout 5 lb api
echo "Restarting PostgreSQL, Redis and S3..."
compose restart --timeout 10 postgres redis s3
echo "Waiting for PostgreSQL..."
attempt=0
until compose exec -T postgres \
pg_isready -U messenger -d messenger >/dev/null 2>&1
do
attempt=$((attempt + 1))
if [ "$attempt" -ge 20 ]; then
echo "PostgreSQL did not become ready" >&2
exit 1
fi
sleep 1
done
echo "Starting API and LB..."
compose start api lb
echo "Restart hook completed."
+126
View File
@@ -0,0 +1,126 @@
from __future__ import annotations
import json
from pathlib import Path
import secrets
ROOT = Path(__file__).resolve().parent.parent
DIRECTORY = ROOT / "secrets"
BUCKET = "messenger-attachments"
KEY_FILES = (
"s3_admin_access_key",
"s3_admin_secret_key",
"s3_access_key_id",
"s3_secret_access_key",
)
def configuration(values):
return {
"identities": [
{
"name": "bootstrap-admin",
"credentials": [{
"accessKey": values["s3_admin_access_key"],
"secretKey": values["s3_admin_secret_key"],
}],
"actions": ["Admin", "Read", "Write", "List", "Tagging"],
},
{
"name": "messenger-api",
"credentials": [{
"accessKey": values["s3_access_key_id"],
"secretKey": values["s3_secret_access_key"],
}],
"actions": [
f"Read:{BUCKET}",
f"Write:{BUCKET}",
f"List:{BUCKET}",
],
},
]
}
def ensure_ignores():
patterns = (
".env",
"/secrets/",
"*.bak",
"__pycache__/",
"*.py[cod]",
)
for filename in (".gitignore", ".dockerignore"):
path = ROOT / filename
text = path.read_text() if path.exists() else ""
existing = set(text.splitlines())
missing = [item for item in patterns if item not in existing]
if missing:
path.write_text(
text.rstrip("\n") + "\n" + "\n".join(missing) + "\n"
)
def main():
if DIRECTORY.is_symlink():
raise SystemExit("Refusing a symlink at secrets/")
paths = [DIRECTORY / name for name in KEY_FILES]
paths.append(DIRECTORY / "s3_config.json")
present = [path.exists() for path in paths]
if any(present):
if not all(path.is_file() and not path.is_symlink() for path in paths):
raise SystemExit(
"Incomplete or unsafe secret files; nothing regenerated"
)
values = {
name: (DIRECTORY / name).read_text().rstrip("\r\n")
for name in KEY_FILES
}
if not all(values.values()):
raise SystemExit("Empty credential file; nothing regenerated")
actual = json.loads(
(DIRECTORY / "s3_config.json").read_text()
)
if actual != configuration(values):
raise SystemExit(
"Credentials and S3 configuration differ; nothing regenerated"
)
message = "Existing credentials verified; keys unchanged."
else:
DIRECTORY.mkdir(mode=0o700, exist_ok=True)
DIRECTORY.chmod(0o700)
values = {
"s3_admin_access_key": "admin-" + secrets.token_hex(12),
"s3_admin_secret_key": secrets.token_hex(32),
"s3_access_key_id": "api-" + secrets.token_hex(12),
"s3_secret_access_key": secrets.token_hex(32),
}
for name, value in values.items():
with (DIRECTORY / name).open("x") as stream:
stream.write(value + "\n")
with (DIRECTORY / "s3_config.json").open("x") as stream:
json.dump(configuration(values), stream, indent=2)
stream.write("\n")
message = "New credentials created; values are not printed."
DIRECTORY.chmod(0o700)
for path in paths:
path.chmod(0o644)
ensure_ignores()
print(message)
print("Secrets, .env, Python cache and .bak files are excluded.")
if __name__ == "__main__":
main()
+71
View File
@@ -0,0 +1,71 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec python3 - <<'PY'
import os
import subprocess
target = os.environ.get("TARGET_INSTANCE_ID")
if not target:
raise SystemExit("TARGET_INSTANCE_ID is required")
compose = [
"docker", "compose",
"-f", "compose.yaml",
"-f", "compose.s3.yaml",
"--env-file", ".env",
]
containers = subprocess.check_output(
compose + ["ps", "-q", "api"],
text=True,
timeout=10,
).split()
if len(containers) < 2:
raise SystemExit("At least two running API containers are required")
probe = (
"import urllib.request; "
"r=urllib.request.urlopen("
"'http://127.0.0.1:8080/health/ready', timeout=2); "
"print(r.headers['X-Instance-Id'])"
)
matches = []
for container in containers:
instance = subprocess.check_output(
["docker", "exec", container, "python", "-c", probe],
text=True,
timeout=4,
).strip()
if instance == target:
matches.append(container)
if len(matches) != 1:
raise SystemExit("Target instance was not resolved uniquely")
container = matches[0]
print("Stopping target API container:", container[:12], flush=True)
subprocess.run(
["docker", "stop", "--time", "5", container],
check=True,
stdout=subprocess.DEVNULL,
timeout=15,
)
running = subprocess.check_output(
["docker", "inspect", "-f", "{{.State.Running}}", container],
text=True,
timeout=5,
).strip()
if running != "false":
raise SystemExit("Target API container is still running")
print("Target API container is stopped", flush=True)
PY
+308
View File
@@ -0,0 +1,308 @@
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
import json
from pathlib import Path
import sys
import threading
import time
import uuid
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, arguments, check, run
from resilience import history
WORKERS = 12
ROUNDS = 3
def main():
args = arguments("Concurrent cross-replica idempotency")
control = Client(
args.base_url,
args.ca_file,
timeout=10,
)
alice, bob, eve, chat = control.bootstrap()
path = f"/api/v1/chats/{chat['id']}/messages"
check(
history(control, alice, chat) == [],
"fresh test chat is not empty",
)
clients = [
Client(args.base_url, args.ca_file, timeout=10)
for _ in range(WORKERS)
]
saved_messages = []
round_results = []
first_key = None
for round_number in range(1, ROUNDS + 1):
key = uuid.uuid4().hex
text = f"Один ключ — одно сообщение: {round_number} {key}"
if first_key is None:
first_key = key
barrier = threading.Barrier(WORKERS)
def send(index):
if index % 3 == 0:
body = {"text": text}
elif index % 3 == 1:
body = {
"attachment_ids": [],
"text": text,
}
else:
body = {
"text": text,
"attachment_ids": [],
}
raw = json.dumps(
body,
ensure_ascii=False,
indent=2 if index % 2 else None,
).encode("utf-8")
barrier.wait(timeout=15)
return clients[index].call(
"POST",
path,
expected=201,
actor=alice,
raw=raw,
headers={
"Content-Type": "application/json",
"Idempotency-Key": key,
},
schema="Message",
)
started = time.monotonic()
with ThreadPoolExecutor(max_workers=WORKERS) as executor:
responses = list(executor.map(send, range(WORKERS)))
elapsed = time.monotonic() - started
original = responses[0]["json"]
check(
all(
response["json"] == original
for response in responses
),
"same key returned different Message objects",
)
instances = Counter(
response["headers"]["X-Instance-Id"]
for response in responses
)
check(
len(instances) >= 2,
"concurrent requests did not reach two replicas",
)
messages = history(control, bob, chat)
same_text = [
message
for message in messages
if message["text"] == text
]
check(
same_text == [original],
"concurrent repeats created zero or multiple messages",
)
check(
len(messages) == round_number,
"unexpected message count after concurrent round",
)
saved_messages.append(original)
result = {
"round": round_number,
"requests": WORKERS,
"responses_201": len(responses),
"observed_instances": len(instances),
"responses_by_instance": dict(instances),
"unique_message_ids": len(
{
response["json"]["id"]
for response in responses
}
),
"matching_history_messages": len(same_text),
"duration_seconds": round(elapsed, 3),
}
round_results.append(result)
print(json.dumps(result, ensure_ascii=False))
conflict = control.call(
"POST",
path,
expected=409,
actor=alice,
body={"text": "Другой текст с прежним ключом"},
headers={"Idempotency-Key": first_key},
)
check(
conflict["json"]["error"]["code"] == "conflict",
"wrong error code for body conflict",
)
control.call(
"POST",
path,
expected=400,
actor=alice,
body={"text": "Без ключа"},
)
for invalid_key in ("bad key", "x" * 129):
control.call(
"POST",
path,
expected=400,
actor=alice,
body={"text": "Некорректный ключ"},
headers={"Idempotency-Key": invalid_key},
)
control.call(
"POST",
path,
expected=404,
actor=eve,
body={"text": saved_messages[0]["text"]},
headers={"Idempotency-Key": first_key},
)
bob_message = control.message(
bob,
chat,
"Тот же ключ другого пользователя",
key=first_key,
)["json"]
check(
bob_message["sender_id"] == bob["id"],
"wrong sender for another user's scope",
)
check(
bob_message["id"] != saved_messages[0]["id"],
"key incorrectly shared between users",
)
second_chat = control.call(
"POST",
"/api/v1/chats",
expected=201,
actor=alice,
body={
"title": "Другой чат для области ключа",
"member_ids": [alice["id"], bob["id"]],
},
schema="Chat",
)["json"]
other_chat_message = control.message(
alice,
second_chat,
"Тот же ключ в другом чате",
key=first_key,
)["json"]
check(
other_chat_message["chat_id"] == second_chat["id"],
"wrong chat in scoped result",
)
check(
other_chat_message["id"] != saved_messages[0]["id"],
"key incorrectly shared between chats",
)
replay = control.message(
alice,
chat,
saved_messages[0]["text"],
key=first_key,
)["json"]
check(
replay == saved_messages[0],
"original result changed after scope tests",
)
messages = history(control, bob, chat)
expected = saved_messages + [bob_message]
check(
len(messages) == len(expected),
"unexpected final message count",
)
check(
{message["id"] for message in messages}
== {message["id"] for message in expected},
"missing or extra messages in first chat",
)
check(
history(control, bob, second_chat) == [other_chat_message],
"unexpected history in second chat",
)
control.call(
"DELETE",
"/api/v1/auth/sessions/current",
expected=204,
actor=alice,
)
control.call(
"POST",
path,
expected=401,
actor=alice,
body={"text": saved_messages[0]["text"]},
headers={"Idempotency-Key": first_key},
)
print(
json.dumps(
{
"rounds": ROUNDS,
"concurrent_requests_per_round": WORKERS,
"concurrent_requests_total": WORKERS * ROUNDS,
"body_conflict_409": True,
"missing_and_invalid_key_400": True,
"foreign_user_404": True,
"user_scope_independent": True,
"chat_scope_independent": True,
"replay_preserved": True,
"revoked_session_401": True,
"unexpected_messages": 0,
},
ensure_ascii=False,
)
)
print(
"PASS: concurrent cross-replica idempotency, "
"scope, validation and authorization checks."
)
if __name__ == "__main__":
run(main)
+176
View File
@@ -0,0 +1,176 @@
#!/usr/bin/env python3
from __future__ import annotations
import concurrent.futures
import json
from pathlib import Path
import subprocess
import sys
import time
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / "tests"))
from client import Client
BASE_URL = "http://127.0.0.1:18082"
MAX_RESPONSE_SECONDS = 6.0
WATCHDOG_SECONDS = 8.0
COMPOSE = [
"docker", "compose",
"-f", "compose.yaml",
"-f", "compose.s3.yaml",
"--env-file", ".env",
]
def docker(*args):
return subprocess.check_output(
["docker", *args],
cwd=ROOT,
text=True,
timeout=20,
stderr=subprocess.PIPE,
).strip()
def main():
pg_id = subprocess.check_output(
[*COMPOSE, "ps", "-q", "postgres"],
cwd=ROOT,
text=True,
timeout=20,
).strip()
assert pg_id, "PostgreSQL container not found"
assert docker(
"inspect", "-f", "{{.State.Paused}}", pg_id
) == "false", "PostgreSQL is already paused"
c = Client(BASE_URL, None)
alice, _, _, chat = c.bootstrap()
message = c.message(alice, chat, "postgres-pause regression")["json"]
path = "/api/v1/chats/" + chat["id"] + "/messages?limit=100"
baseline_instances = set()
for _ in range(6):
response = c.call(
"GET", path, actor=alice, schema="MessagePage"
)
assert any(
item["id"] == message["id"]
for item in response["json"]["items"]
), "Baseline message is missing"
instance = response["headers"].get("X-Instance-Id")
assert instance, "Baseline response has no X-Instance-Id"
baseline_instances.add(instance)
assert len(baseline_instances) >= 2, (
"Baseline did not reach both API replicas"
)
print("Baseline: history is available through both replicas", flush=True)
def probe():
started = time.monotonic()
try:
client = Client(BASE_URL, None)
response = client.call("GET", path, 503, actor=alice)
elapsed = time.monotonic() - started
instance = response["headers"].get("X-Instance-Id")
code = response["json"]["error"]["code"]
return {
"ok": bool(instance) and code == "unavailable",
"status": 503,
"code": code,
"instance": instance,
"elapsed_s": round(elapsed, 4),
}
except Exception as exc:
return {
"ok": False,
"error_type": type(exc).__name__,
"elapsed_s": round(time.monotonic() - started, 4),
}
executor = concurrent.futures.ThreadPoolExecutor(max_workers=4)
paused_by_test = False
try:
paused_by_test = True
docker("pause", pg_id)
assert docker(
"inspect", "-f", "{{.State.Paused}}", pg_id
) == "true"
print("PostgreSQL paused; requesting history", flush=True)
futures = [executor.submit(probe) for _ in range(4)]
_, pending = concurrent.futures.wait(
futures, timeout=WATCHDOG_SECONDS
)
assert not pending, "History requests exceeded the test watchdog"
results = [future.result() for future in futures]
print(json.dumps(results, indent=2), flush=True)
assert all(item["ok"] for item in results), (
"Expected API JSON 503 unavailable with X-Instance-Id"
)
assert all(
item["elapsed_s"] < MAX_RESPONSE_SECONDS
for item in results
), "API response exceeded the declared deadline"
failed_instances = {item["instance"] for item in results}
assert baseline_instances.issubset(failed_instances), (
"Paused-DB requests did not exercise both baseline replicas"
)
finally:
try:
if paused_by_test:
state = docker(
"inspect", "-f", "{{.State.Paused}}", pg_id
)
if state == "true":
docker("unpause", pg_id)
print("PostgreSQL unpaused", flush=True)
finally:
executor.shutdown(wait=False, cancel_futures=True)
recovery_started = time.monotonic()
recovery_deadline = recovery_started + 60
while True:
try:
response = c.call(
"GET", path, actor=alice, schema="MessagePage"
)
assert any(
item["id"] == message["id"]
for item in response["json"]["items"]
), "Confirmed message disappeared"
break
except Exception:
if time.monotonic() >= recovery_deadline:
raise AssertionError("History did not recover") from None
time.sleep(1)
report = {
"scenario": "postgres_pause_history",
"passed": True,
"max_response_seconds": MAX_RESPONSE_SECONDS,
"baseline_instances": sorted(baseline_instances),
"paused_requests": results,
"recovery_observed_after_s": round(
time.monotonic() - recovery_started, 4
),
"confirmed_message_preserved": True,
}
target = ROOT / "evidence" / "postgres_pause.json"
target.parent.mkdir(exist_ok=True)
target.write_text(json.dumps(report, indent=2) + "\n")
print("PASS: bounded API 503 on both replicas; history recovered", flush=True)
if __name__ == "__main__":
main()
+284
View File
@@ -0,0 +1,284 @@
import json
from pathlib import Path
import subprocess
import sys
import time
from datetime import datetime, timezone
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, Failure, arguments, check, run
PROJECT = "mtusi2026lab3"
def timestamp():
return datetime.now(timezone.utc).isoformat()
def command(*args):
return subprocess.check_output(
list(args),
cwd=ROOT,
text=True,
timeout=45,
).strip()
def compose(*args):
return command(
"docker",
"compose",
"-p",
PROJECT,
*args,
)
def inspect(container, template):
return command(
"docker",
"inspect",
"--format",
template,
container,
)
def api_ids():
ids = compose("ps", "-q", "api").split()
return {
inspect(container, "{{.Id}}")
for container in ids
}
def healthy(container):
return inspect(
container,
"{{.State.Running}} {{.State.Health.Status}}",
) == "true healthy"
def instance_id(container):
code = (
"import urllib.request; "
"r=urllib.request.urlopen("
"'http://127.0.0.1:8080/health/live', timeout=2); "
"print(r.headers.get('X-Instance-Id', '')); "
"r.close()"
)
return command(
"docker",
"exec",
container,
"python",
"-c",
code,
)
def main():
args = arguments("Sequential API container replacement")
client = Client(
args.base_url,
args.ca_file,
timeout=2,
)
alice, bob, eve, chat = client.bootstrap()
import uuid
key = uuid.uuid4().hex
text = "Сохранить при замене " + key
original_message = client.message(
alice,
chat,
text,
key=key,
)["json"]
original_ids = sorted(api_ids())
check(len(original_ids) == 2, "expected exactly two API containers")
for container in original_ids:
check(healthy(container), "both API containers must be healthy")
print(
json.dumps(
{
"event": "replacement_started",
"utc": timestamp(),
"original_container_ids": original_ids,
},
ensure_ascii=False,
),
flush=True,
)
replacements = []
for old in original_ids:
current = api_ids()
check(len(current) == 2, "expected two APIs before replacement")
check(old in current, "original target container is missing")
other = next(container for container in current if container != old)
check(healthy(other), "remaining API is not healthy")
other_started = inspect(other, "{{.State.StartedAt}}")
old_instance = instance_id(old)
step_started = time.monotonic()
print(
json.dumps(
{
"event": "replacing",
"utc": timestamp(),
"old_container_id": old,
"old_instance": old_instance,
"remaining_container_id": other,
},
ensure_ascii=False,
),
flush=True,
)
command("docker", "stop", "--time", "10", old)
command("docker", "rm", old)
compose(
"up",
"-d",
"--no-deps",
"--no-recreate",
"--no-build",
"--scale",
"api=2",
"api",
)
deadline = time.monotonic() + 30
replacement = None
while time.monotonic() < deadline:
current = api_ids()
candidates = current - {other}
if (
len(current) == 2
and len(candidates) == 1
and other in current
):
candidate = next(iter(candidates))
if candidate != old and healthy(candidate):
replacement = candidate
break
time.sleep(0.5)
check(replacement is not None, "replacement did not become healthy")
new_instance = instance_id(replacement)
check(new_instance != old_instance, "instance did not change")
seen_through_lb = False
deadline = time.monotonic() + 20
while time.monotonic() < deadline:
try:
response = client.call(
"GET",
"/api/v1/users/me",
actor=alice,
schema="User",
)
except Failure:
time.sleep(0.2)
continue
if response["headers"]["X-Instance-Id"] == new_instance:
seen_through_lb = True
break
time.sleep(0.2)
check(
seen_through_lb,
"new API was not observed through LB with existing session",
)
check(healthy(other), "remaining API became unhealthy")
check(
inspect(other, "{{.State.StartedAt}}") == other_started,
"the other API was restarted unexpectedly",
)
replay = client.message(
alice,
chat,
text,
key=key,
)["json"]
check(
replay == original_message,
"confirmed message or idempotency result changed",
)
result = {
"event": "replacement_ready",
"utc": timestamp(),
"old_container_id": old,
"new_container_id": replacement,
"old_instance": old_instance,
"new_instance": new_instance,
"step_seconds": round(
time.monotonic() - step_started, 3
),
"other_api_not_restarted": True,
"existing_session_works": True,
"replay_preserved": True,
}
replacements.append(result)
print(json.dumps(result, ensure_ascii=False), flush=True)
time.sleep(3)
final_ids = api_ids()
check(len(final_ids) == 2, "expected two APIs after replacement")
check(
not final_ids.intersection(original_ids),
"some original API containers were not replaced",
)
for container in final_ids:
check(healthy(container), "final API is not healthy")
print(
json.dumps(
{
"event": "replacement_finished",
"utc": timestamp(),
"replacements": len(replacements),
"final_container_ids": sorted(final_ids),
},
ensure_ascii=False,
),
flush=True,
)
print(
"PASS: both API containers replaced sequentially; "
"existing session and confirmed message preserved.",
flush=True,
)
if __name__ == "__main__":
run(main)
+318
View File
@@ -0,0 +1,318 @@
#!/usr/bin/env python3
"""Targeted API failure test. Uses synthetic data; leaves target stopped."""
import json
import os
from pathlib import Path
import subprocess
import sys
import time
import uuid
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, Failure, arguments, check, run
from resilience import history
PROJECT = "mtusi2026lab3"
def docker_output(*args):
return subprocess.check_output(
["docker", *args],
cwd=ROOT,
text=True,
).strip()
def target_instance(container):
labels = json.loads(
docker_output(
"inspect",
"--format",
"{{json .Config.Labels}}",
container,
)
)
check(
labels.get("com.docker.compose.project") == PROJECT,
"target belongs to another Compose project",
)
check(
labels.get("com.docker.compose.service") == "api",
"target is not an API container",
)
check(
docker_output(
"inspect",
"--format",
"{{.State.Running}}",
container,
) == "true",
"target container is not running",
)
code = (
"import urllib.request; "
"r=urllib.request.urlopen("
"'http://127.0.0.1:8080/health/live', timeout=2); "
"print(r.headers.get('X-Instance-Id', '')); "
"r.close()"
)
instance = docker_output(
"exec", container, "python", "-c", code
)
check(bool(instance), "target has no X-Instance-Id")
return instance
def main():
def configure(parser):
parser.add_argument(
"--target-container",
required=True,
)
args = arguments(
"Targeted API failover through the load balancer",
configure,
)
script = ROOT / "scripts" / "stop-one.sh"
check(
script.is_file() and os.access(script, os.X_OK),
"stop-one.sh must exist and be executable",
)
target = target_instance(args.target_container)
client = Client(args.base_url, args.ca_file, timeout=2)
alice, bob, eve, chat = client.bootstrap()
original_key = uuid.uuid4().hex
original_text = "Подтверждено до отказа " + original_key
original = client.message(
alice,
chat,
original_text,
key=original_key,
)["json"]
observed = set()
for _ in range(40):
response = client.call(
"GET",
"/api/v1/users/me",
actor=alice,
schema="User",
)
observed.add(response["headers"]["X-Instance-Id"])
check(len(observed) >= 2, "less than two API instances")
check(target in observed, "target is not observed through LB")
env = dict(os.environ)
env["TARGET_INSTANCE_ID"] = target
print(
"Target container:",
args.target_container,
"| instance:",
target,
flush=True,
)
started = time.monotonic()
proc = subprocess.Popen(
[str(script)],
cwd=ROOT,
env=env,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
successes = []
write_failures = 0
read_successes = 0
read_failures = 0
after_instances = []
first_recovery = None
attempts = 0
key = uuid.uuid4().hex
text = "Во время отказа " + key
try:
while time.monotonic() - started < 15:
code = proc.poll()
elapsed = time.monotonic() - started
check(
code is None or code == 0,
"stop-one hook failed",
)
check(
code is not None or elapsed < 10,
"stop-one hook exceeded 10 seconds",
)
attempts += 1
try:
response = client.message(
alice,
chat,
text,
key=key,
)
except Failure:
write_failures += 1
# Keep the same key and text after uncertain POST.
else:
successes.append(response["json"])
if proc.poll() == 0:
if first_recovery is None:
first_recovery = time.monotonic() - started
after_instances.append(
response["headers"]["X-Instance-Id"]
)
key = uuid.uuid4().hex
text = "Во время отказа " + key
if attempts % 5 == 0:
try:
page = client.call(
"GET",
(
"/api/v1/chats/"
+ chat["id"]
+ "/messages?limit=100"
),
actor=bob,
schema="MessagePage",
)["json"]
except Failure:
read_failures += 1
else:
check(
any(
message == original
for message in page["items"]
),
"confirmed message absent during read",
)
read_successes += 1
time.sleep(0.1)
elapsed = time.monotonic() - started
check(proc.poll() == 0, "stop-one hook did not finish")
check(
first_recovery is not None and first_recovery <= 10,
"writes did not recover within 10 seconds",
)
check(
len(after_instances) >= 5,
"too few writes after stop",
)
check(
target not in after_instances[-5:],
"stopped instance still serves requests",
)
check(read_successes > 0, "no successful history reads")
check(
docker_output(
"inspect",
"--format",
"{{.State.Running}}",
args.target_container,
) == "false",
"target container is still running",
)
replay = client.message(
alice,
chat,
original_text,
key=original_key,
)["json"]
check(
replay == original,
"replay after failure returned another message",
)
messages = history(client, bob, chat)
by_id = {message["id"]: message for message in messages}
check(
len(by_id) == len(messages),
"duplicate message IDs",
)
check(
by_id.get(original["id"]) == original,
"pre-failure message lost or altered",
)
for message in successes:
check(
by_id.get(message["id"]) == message,
"confirmed in-flight message lost or altered",
)
texts = [message["text"] for message in messages]
check(
len(texts) == len(set(texts)),
"retry created duplicate message text",
)
print(
json.dumps(
{
"target_container": args.target_container,
"target_instance": target,
"observed_instances_before": len(observed),
"duration_seconds": round(elapsed, 3),
"successful_writes": len(successes),
"failed_write_attempts": write_failures,
"successful_reads": read_successes,
"failed_read_attempts": read_failures,
"successful_write_rps": round(
len(successes) / elapsed, 3
),
"first_recovery_seconds": round(
first_recovery, 3
),
"replay_preserved": True,
"target_stopped": True,
},
ensure_ascii=False,
)
)
print(
"PASS: targeted failover; session, messages "
"and replay preserved. Target remains stopped."
)
finally:
if proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait()
if __name__ == "__main__":
run(main)
+225
View File
@@ -0,0 +1,225 @@
from __future__ import annotations
import base64
import http.client
import json
import secrets
import struct
import urllib.error
import urllib.parse
import urllib.request
from uuid import uuid4
import zlib
BASE = "http://127.0.0.1:18082/api/v1"
token = None
passed = 0
def check(label, status, headers, body, expected):
global passed
if status != expected:
raise SystemExit(
f"FAIL {label}: expected={expected}, actual={status}\n"
+ body[:1000].decode("utf-8", errors="replace")
)
assert headers.get("X-Request-Id"), "Missing X-Request-Id"
assert headers.get("X-Instance-Id"), "Missing X-Instance-Id"
result = json.loads(body)
if expected >= 400:
codes = {
400: "invalid_request",
404: "not_found",
413: "payload_too_large",
415: "unsupported_media_type",
}
assert result["error"]["code"] == codes[expected]
assert (
result["error"]["request_id"]
== headers["X-Request-Id"]
)
passed += 1
print(f"OK {status}: {label}", flush=True)
return result
def call(label, method, path, expected, *,
payload=None, data=None, headers=None):
headers = dict(headers or {})
if token and "Authorization" not in headers:
headers["Authorization"] = "Bearer " + token
if payload is not None:
data = json.dumps(payload).encode()
headers["Content-Type"] = "application/json"
request = urllib.request.Request(
BASE + path, data=data, headers=headers, method=method
)
try:
response = urllib.request.urlopen(request, timeout=15)
except urllib.error.HTTPError as error:
response = error
with response:
body = response.read()
return check(
label, response.code, response.headers, body, expected
)
def chunk(kind, content):
return (
struct.pack(">I", len(content))
+ kind + content
+ struct.pack(">I", zlib.crc32(kind + content) & 0xffffffff)
)
def png(width, height):
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(
b"IHDR",
struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0),
)
+ chunk(b"IDAT", zlib.compress(b"\x00\x12\x34\x56"))
+ chunk(b"IEND", b"")
)
username = "negative_" + secrets.token_hex(6)
password = "Lab4-" + secrets.token_hex(16)
user = call(
"create user", "POST", "/users", 201,
payload={
"username": username,
"display_name": "Lab4 negative tests",
"password": password,
},
)
basic = base64.b64encode(
f"{username}:{password}".encode()
).decode()
session = call(
"login", "POST", "/auth/sessions", 201,
headers={"Authorization": "Basic " + basic},
)
token = session["token"]
chat_ids = []
for number in (1, 2):
chat = call(
f"create chat {number}", "POST", "/chats", 201,
payload={
"title": f"Negative check {number}",
"member_ids": [user["id"]],
},
)
chat_ids.append(chat["id"])
upload = f"/chats/{chat_ids[0]}/attachments"
image = png(1, 1)
attachment = call(
"valid upload", "POST", upload + "?filename=valid.png", 201,
data=image,
headers={"Content-Type": "image/png"},
)
for filename in (
"../bad.png",
"bad\\name.png",
"bad\x00.png",
"bad\n.png",
" ",
"a" * 256,
):
call(
"unsafe filename " + repr(filename[:30]),
"POST",
upload + "?filename=" + urllib.parse.quote(filename, safe=""),
400,
data=image,
headers={"Content-Type": "image/png"},
)
for label, data, mime in (
("PNG declared as JPEG", image, "image/jpeg"),
("non-image", b"this is not an image", "image/png"),
("truncated PNG", image[:-12], "image/png"),
("unsupported MIME", image, "application/octet-stream"),
("empty image", b"", "image/png"),
):
call(
label, "POST", upload + "?filename=bad.png", 415,
data=data,
headers={"Content-Type": mime},
)
call(
"over 40 million pixels",
"POST", upload + "?filename=huge.png", 413,
data=png(8001, 5000),
headers={"Content-Type": "image/png"},
)
connection = http.client.HTTPConnection(
"127.0.0.1", 18082, timeout=15
)
try:
connection.putrequest(
"POST", "/api/v1" + upload + "?filename=oversize.png"
)
connection.putheader("Authorization", "Bearer " + token)
connection.putheader("Content-Type", "image/png")
connection.putheader("Content-Length", str(10 * 1024 * 1024 + 1))
connection.putheader("Connection", "close")
connection.endheaders()
response = connection.getresponse()
check(
"oversized Content-Length rejected before body",
response.status,
response.headers,
response.read(),
413,
)
finally:
connection.close()
for label, chat_id, ids, expected in (
(
"attachment from another chat",
chat_ids[1], [attachment["id"]], 404,
),
(
"unknown attachment",
chat_ids[0], [str(uuid4())], 404,
),
(
"duplicate attachment",
chat_ids[0], [attachment["id"], attachment["id"]], 400,
),
(
"more than 10 attachments",
chat_ids[0], [str(uuid4()) for _ in range(11)], 400,
),
):
call(
label, "POST", f"/chats/{chat_id}/messages", expected,
payload={"text": "", "attachment_ids": ids},
headers={"Idempotency-Key": secrets.token_hex(16)},
)
print(f"\nPASS: {passed} checks including setup")
+190
View File
@@ -0,0 +1,190 @@
from __future__ import annotations
import asyncio
import hashlib
import time
from uuid import uuid4
from botocore.exceptions import ClientError
import psycopg
import app.cleanup_orphans as cleanup
from app.object_locks import object_lock_id
from app.storage import bucket_name, get_client, get_object
client = get_client()
bucket = bucket_name()
with cleanup.database_connection() as conn:
original = conn.execute(
"""
SELECT object_key, chat_id, owner_id, sha256
FROM attachments
ORDER BY created_at, id
LIMIT 1
"""
).fetchone()
if original is None:
raise SystemExit("Нужно существующее корректное вложение")
orphan_id = uuid4()
orphan_key = f"originals/{orphan_id}"
busy_key = f"originals/{uuid4()}"
fresh_key = f"originals/{uuid4()}"
test_keys = {orphan_key, busy_key, fresh_key}
scope = test_keys | {original["object_key"]}
body = b"controlled-orphan-test"
class ScopedPages:
def paginate(self, **kwargs):
paginator = client.get_paginator("list_objects_v2")
for page in paginator.paginate(**kwargs):
yield {
**page,
"Contents": [
item for item in page.get("Contents", [])
if item["Key"] in scope
],
}
class ScopedClient:
def get_paginator(self, name):
assert name == "list_objects_v2"
return ScopedPages()
def __getattr__(self, name):
return getattr(client, name)
cleanup.get_client = lambda: ScopedClient()
def sweep(apply):
result = cleanup.sweep(apply=apply, grace_seconds=60)
print(result, flush=True)
return result
def exists(key):
try:
client.head_object(Bucket=bucket, Key=key)
return True
except ClientError as exc:
status = exc.response.get(
"ResponseMetadata", {}
).get("HTTPStatusCode")
if status == 404:
return False
raise
try:
client.put_object(
Bucket=bucket, Key=orphan_key, Body=body,
ContentType="application/octet-stream",
)
with cleanup.database_connection() as conn:
with conn.transaction():
conn.execute(
"""
INSERT INTO attachments (
id, chat_id, owner_id, filename,
content_type, size_bytes, sha256, object_key
)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
""",
(
orphan_id,
original["chat_id"],
original["owner_id"],
"rollback-test.png",
"image/png",
len(body),
hashlib.sha256(body).hexdigest(),
orphan_key,
),
)
raise psycopg.Rollback()
row = conn.execute(
"SELECT 1 FROM attachments WHERE object_key = %s",
(orphan_key,),
).fetchone()
assert row is None
assert exists(orphan_key)
print("OK: S3 PUT сохранён, метаданные откатились", flush=True)
client.put_object(
Bucket=bucket, Key=busy_key, Body=body,
)
result = sweep(False)
assert result["candidates"] == 0
assert result["deleted"] == 0
assert result["fresh"] >= 2
print("OK: свежие orphan-объекты сохранены", flush=True)
print("Ждём 65 секунд для тестового grace period...", flush=True)
time.sleep(65)
client.put_object(
Bucket=bucket, Key=fresh_key, Body=body,
)
with cleanup.database_connection() as lock_conn:
with lock_conn.transaction():
lock_conn.execute(
"SET LOCAL "
"idle_in_transaction_session_timeout = '180s'"
)
lock_conn.execute(
"SELECT pg_advisory_xact_lock(%s)",
(object_lock_id(bucket, busy_key),),
)
result = sweep(False)
assert result["candidates"] == 1
assert result["deleted"] == 0
assert result["busy"] == 1
assert result["fresh"] == 1
assert result["kept"] == 1
assert exists(orphan_key)
print("OK: dry-run ничего не удалил", flush=True)
result = sweep(True)
assert result["deleted"] == 1
assert result["busy"] == 1
assert result["fresh"] == 1
assert result["kept"] == 1
assert not exists(orphan_key)
assert exists(busy_key)
assert exists(fresh_key)
print("OK: удалён только старый незанятый orphan", flush=True)
result = sweep(True)
assert result["deleted"] == 0
assert result["candidates"] == 0
print("OK: повторный запуск безопасен", flush=True)
result = sweep(True)
assert result["deleted"] == 1
assert not exists(busy_key)
assert exists(fresh_key)
print("OK: после освобождения блокировки orphan очищен", flush=True)
downloaded = asyncio.run(get_object(original["object_key"]))
assert hashlib.sha256(downloaded).hexdigest() == original["sha256"]
print("OK: существующее вложение не повреждено", flush=True)
print("PASS: orphan cleanup safety and rollback recovery", flush=True)
finally:
for key in test_keys:
client.delete_object(Bucket=bucket, Key=key)
print("Только собственные тестовые объекты убраны", flush=True)
+154
View File
@@ -0,0 +1,154 @@
from __future__ import annotations
import base64
import json
from pathlib import Path
import runpy
import subprocess
import time
import urllib.error
import urllib.request
ROOT = Path(__file__).resolve().parent.parent
COMPOSE = [
"docker", "compose",
"-f", "compose.yaml",
"-f", "compose.s3.yaml",
"--env-file", ".env",
]
def compose(*args):
return subprocess.run(
COMPOSE + list(args),
cwd=ROOT,
check=True,
)
state = runpy.run_path(str(ROOT / "tests/attachments_negative.py"))
containers = subprocess.check_output(
COMPOSE + ["ps", "-q", "api"],
cwd=ROOT,
text=True,
).split()
if not containers:
raise SystemExit("No running API container")
payload = {
"token": state["token"],
"chat_id": state["chat_ids"][0],
"attachment_id": state["attachment"]["id"],
"image": base64.b64encode(state["image"]).decode(),
}
probe = r'''
import base64
import json
import sys
import urllib.error
import urllib.request
state = json.load(sys.stdin)
base = "http://127.0.0.1:8080/api/v1"
attachment = state["attachment_id"]
cases = [
(
"metadata remains readable",
"GET",
f"/attachments/{attachment}",
None,
200,
),
(
"download fails safely",
"GET",
f"/attachments/{attachment}/content",
None,
503,
),
(
"upload fails safely",
"POST",
f"/chats/{state['chat_id']}/attachments?filename=outage.png",
base64.b64decode(state["image"]),
503,
),
]
for label, method, path, data, expected in cases:
headers = {"Authorization": "Bearer " + state["token"]}
if data is not None:
headers["Content-Type"] = "image/png"
request = urllib.request.Request(
base + path, method=method, headers=headers, data=data
)
try:
response = urllib.request.urlopen(request, timeout=40)
except urllib.error.HTTPError as error:
response = error
with response:
status = response.code
headers = response.headers
body = json.loads(response.read())
assert status == expected, (label, status, body)
assert headers.get("X-Request-Id")
assert headers.get("X-Instance-Id")
if expected == 503:
assert body["error"]["code"] == "unavailable"
assert (
body["error"]["request_id"]
== headers["X-Request-Id"]
)
else:
assert body["id"] == attachment
print(f"OK {status}: {label}", flush=True)
'''
try:
compose("stop", "--timeout", "10", "s3")
subprocess.run(
["docker", "exec", "-i", containers[0], "python", "-c", probe],
input=json.dumps(payload),
text=True,
check=True,
timeout=150,
)
finally:
compose("start", "s3")
deadline = time.monotonic() + 90
while True:
try:
with urllib.request.urlopen(
"http://127.0.0.1:18082/health/ready",
timeout=5,
) as response:
assert response.status == 200
break
except (urllib.error.URLError, TimeoutError, ConnectionError):
if time.monotonic() >= deadline:
raise SystemExit("Readiness did not recover")
time.sleep(2)
request = urllib.request.Request(
state["BASE"]
+ f"/attachments/{state['attachment']['id']}/content",
headers={"Authorization": "Bearer " + state["token"]},
)
with urllib.request.urlopen(request, timeout=15) as response:
downloaded = response.read()
assert downloaded == state["image"]
print("OK: original remains intact after S3 restart")
print("PASS: S3 outage operations and recovery")
+158
View File
@@ -0,0 +1,158 @@
from __future__ import annotations
import base64
import hashlib
import http.client
import json
import secrets
import struct
import time
import urllib.request
import zlib
from app.cleanup_orphans import database_connection
from app.storage import bucket_name, get_client
BASE = "http://lb:8080/api/v1"
token = None
def api(method, path, *, payload=None, data=None, headers=None):
headers = dict(headers or {})
if token and "Authorization" not in headers:
headers["Authorization"] = "Bearer " + token
if payload is not None:
data = json.dumps(payload).encode()
headers["Content-Type"] = "application/json"
request = urllib.request.Request(
BASE + path, method=method, headers=headers, data=data
)
with urllib.request.urlopen(request, timeout=15) as response:
assert response.code == 201
return json.loads(response.read())
def chunk(kind, content):
return (
struct.pack(">I", len(content))
+ kind + content
+ struct.pack(">I", zlib.crc32(kind + content) & 0xffffffff)
)
image = (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", struct.pack(">IIBBBBB", 1, 1, 8, 2, 0, 0, 0))
+ chunk(b"IDAT", zlib.compress(b"\x00\x12\x34\x56"))
+ chunk(b"IEND", b"")
)
username = "abort_" + secrets.token_hex(6)
password = "Lab4-" + secrets.token_hex(16)
user = api(
"POST", "/users",
payload={
"username": username,
"display_name": "Upload abort check",
"password": password,
},
)
basic = base64.b64encode(
f"{username}:{password}".encode()
).decode()
session = api(
"POST", "/auth/sessions",
headers={"Authorization": "Basic " + basic},
)
token = session["token"]
chat = api(
"POST", "/chats",
payload={
"title": "Upload abort check",
"member_ids": [user["id"]],
},
)
chat_id = chat["id"]
client = get_client()
bucket = bucket_name()
def object_keys():
return {
item["Key"]
for page in client.get_paginator("list_objects_v2").paginate(
Bucket=bucket, Prefix="originals/"
)
for item in page.get("Contents", [])
}
before = object_keys()
for number in range(1, 4):
connection = http.client.HTTPConnection("api", 8080, timeout=10)
try:
connection.request("GET", "/health/live")
response = connection.getresponse()
response.read()
assert response.status == 200
instance = response.getheader("X-Instance-Id")
connection.putrequest(
"POST",
f"/api/v1/chats/{chat_id}/attachments"
f"?filename=aborted-{number}.png",
)
connection.putheader("Authorization", "Bearer " + token)
connection.putheader("Content-Type", "image/png")
connection.putheader("Content-Length", str(len(image)))
connection.endheaders()
connection.send(image[:len(image) // 2])
time.sleep(0.3)
finally:
connection.close()
print(
f"Соединение {number} оборвано после половины тела; "
f"instance={instance}",
flush=True,
)
time.sleep(2)
with database_connection() as conn:
count = conn.execute(
"SELECT count(*) AS n FROM attachments WHERE chat_id = %s",
(chat_id,),
).fetchone()["n"]
assert count == 0, f"Появились метаданные: {count}"
print("OK: метаданные незавершённых загрузок не созданы", flush=True)
new_keys = object_keys() - before
assert not new_keys, f"Появились новые S3-ключи: {new_keys}"
print("OK: новые S3-объекты не появились", flush=True)
attachment = api(
"POST",
f"/chats/{chat_id}/attachments?filename=complete.png",
data=image,
headers={"Content-Type": "image/png"},
)
assert attachment["sha256"] == hashlib.sha256(image).hexdigest()
request = urllib.request.Request(
BASE + f"/attachments/{attachment['id']}/content",
headers={"Authorization": "Bearer " + token},
)
with urllib.request.urlopen(request, timeout=15) as response:
assert response.read() == image
print("OK: последующая загрузка и скачивание успешны", flush=True)
print("PASS: interrupted uploads leave no metadata or objects", flush=True)
+336
View File
@@ -0,0 +1,336 @@
from __future__ import annotations
import base64
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
import csv
import hashlib
import json
from pathlib import Path
import random
import secrets
import struct
import subprocess
import threading
import time
import urllib.error
import urllib.request
import zlib
BASE = "http://127.0.0.1:18082/api/v1"
COMPOSE = [
"docker", "compose",
"-f", "compose.yaml",
"-f", "compose.s3.yaml",
"--env-file", ".env",
]
token = None
def api(method, path, *, payload=None, headers=None):
headers = dict(headers or {})
if token and "Authorization" not in headers:
headers["Authorization"] = "Bearer " + token
data = None
if payload is not None:
data = json.dumps(payload).encode()
headers["Content-Type"] = "application/json"
request = urllib.request.Request(
BASE + path, method=method, headers=headers, data=data
)
with urllib.request.urlopen(request, timeout=30) as response:
return json.loads(response.read())
def chunk(kind, data):
return (
struct.pack(">I", len(data))
+ kind + data
+ struct.pack(">I", zlib.crc32(kind + data) & 0xffffffff)
)
print("Generating PNG 6000x6000...", flush=True)
rng = random.Random(44)
compressor = zlib.compressobj(level=1)
compressed = bytearray()
for row in range(6000):
pixels = rng.randbytes(18000) if row < 450 else bytes(18000)
compressed.extend(compressor.compress(b"\x00" + pixels))
compressed.extend(compressor.flush())
image = (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", struct.pack(">IIBBBBB", 6000, 6000, 8, 2, 0, 0, 0))
+ chunk(b"IDAT", bytes(compressed))
+ chunk(b"IEND", b"")
)
del compressed
assert len(image) <= 10485760
digest = hashlib.sha256(image).hexdigest()
print(f"Fixture: {len(image)} bytes, 36000000 pixels", flush=True)
username = "memory_" + secrets.token_hex(6)
password = "Lab4-" + secrets.token_hex(16)
user = api(
"POST", "/users",
payload={
"username": username,
"display_name": "Upload memory experiment",
"password": password,
},
)
basic = base64.b64encode(f"{username}:{password}".encode()).decode()
session = api(
"POST", "/auth/sessions",
headers={"Authorization": "Basic " + basic},
)
token = session["token"]
chat = api(
"POST", "/chats",
payload={
"title": "Upload memory experiment",
"member_ids": [user["id"]],
},
)
containers = subprocess.check_output(
COMPOSE + ["ps", "-q", "api"], text=True
).split()
assert len(containers) >= 2, "Need two running API replicas"
def container_states():
result = {}
for container in containers:
text = subprocess.check_output(
[
"docker", "inspect", "--format",
"{{json .State}}|{{.RestartCount}}", container,
],
text=True,
)
raw, restart_count = text.strip().rsplit("|", 1)
state = json.loads(raw)
result[container[:12]] = {
"started_at": state["StartedAt"],
"oom_killed": state["OOMKilled"],
"status": state["Status"],
"restart_count": int(restart_count),
}
return result
SAMPLER = r'''
import json
from pathlib import Path
import select
import sys
while True:
row = {}
for line in Path("/proc/1/status").read_text().splitlines():
name, _, value = line.partition(":")
if name in ("VmRSS", "VmHWM", "VmSwap"):
row[name + "_kib"] = int(value.split()[0])
root = Path("/sys/fs/cgroup")
for name in ("memory.current", "memory.peak", "memory.max"):
row[name] = int((root / name).read_text().strip())
for line in (root / "memory.events").read_text().splitlines():
name, value = line.split()
row["events." + name] = int(value)
print(json.dumps(row), flush=True)
ready, _, _ = select.select([sys.stdin], [], [], 0.1)
if ready and not sys.stdin.readline():
break
'''
samples = []
monitor_errors = []
monitors = []
readers = []
phase = {"name": "baseline"}
origin = time.monotonic()
before = container_states()
def read_samples(process, container):
try:
for line in process.stdout:
row = json.loads(line)
row["container"] = container[:12]
row["time_s"] = round(time.monotonic() - origin, 4)
row["phase"] = phase["name"]
samples.append(row)
except Exception as error:
monitor_errors.append(type(error).__name__)
for container in containers:
process = subprocess.Popen(
["docker", "exec", "-i", container, "python", "-u", "-c", SAMPLER],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
monitors.append(process)
reader = threading.Thread(
target=read_samples, args=(process, container), daemon=True
)
reader.start()
readers.append(reader)
rounds = []
def run_round(label, concurrency):
barrier = threading.Barrier(concurrency + 1)
lock = threading.Lock()
activity = {"current": 0, "peak": 0}
def worker(number):
barrier.wait()
with lock:
activity["current"] += 1
activity["peak"] = max(
activity["peak"], activity["current"]
)
started = time.monotonic()
result = {"number": number}
try:
request = urllib.request.Request(
BASE + f"/chats/{chat['id']}/attachments"
+ f"?filename={label}-{number}.png",
method="POST",
data=image,
headers={
"Authorization": "Bearer " + token,
"Content-Type": "image/png",
},
)
try:
response = urllib.request.urlopen(request, timeout=60)
except urllib.error.HTTPError as error:
response = error
with response:
body = response.read()
result["status"] = response.code
result["instance"] = response.headers.get("X-Instance-Id")
if response.code == 201:
metadata = json.loads(body)
result["valid_metadata"] = (
metadata["sha256"] == digest
and metadata["size_bytes"] == len(image)
and metadata["status"] == "ready"
)
except Exception as error:
result["status"] = "transport_error"
result["error_type"] = type(error).__name__
finally:
result["seconds"] = round(time.monotonic() - started, 4)
with lock:
activity["current"] -= 1
return result
phase["name"] = label
with ThreadPoolExecutor(max_workers=concurrency) as executor:
futures = [executor.submit(worker, n) for n in range(concurrency)]
started = time.monotonic()
barrier.wait()
results = [future.result() for future in futures]
elapsed = time.monotonic() - started
summary = {
"label": label,
"clients": concurrency,
"peak_client_in_flight": activity["peak"],
"elapsed_seconds": round(elapsed, 4),
"statuses": dict(Counter(str(row["status"]) for row in results)),
"requests": results,
}
rounds.append(summary)
print(json.dumps({
key: value for key, value in summary.items()
if key != "requests"
}), flush=True)
phase["name"] = label + "_cooldown"
time.sleep(2)
try:
time.sleep(1)
run_round("single", 1)
run_round("concurrent_1", 10)
run_round("concurrent_2", 10)
finally:
phase["name"] = "final"
time.sleep(1)
for process in monitors:
process.stdin.close()
for process in monitors:
try:
process.wait(timeout=10)
except subprocess.TimeoutExpired:
process.terminate()
monitor_errors.append("sampler_stop_timeout")
if process.returncode != 0:
monitor_errors.append("sampler_nonzero_exit")
for reader in readers:
reader.join(timeout=5)
after = container_states()
report = {
"fixture_bytes": len(image),
"width": 6000,
"height": 6000,
"fixture_sha256": digest,
"before": before,
"after": after,
"rounds": rounds,
"monitor_errors": monitor_errors,
"memory_summary": {},
}
for container in containers:
key = container[:12]
rows = [row for row in samples if row["container"] == key]
assert rows, f"No memory samples for {key}"
report["memory_summary"][key] = {
"samples": len(rows),
"initial_cgroup_peak_bytes": rows[0]["memory.peak"],
"final_cgroup_peak_bytes": rows[-1]["memory.peak"],
"sampled_max_cgroup_bytes": max(r["memory.current"] for r in rows),
"sampled_max_rss_kib": max(r["VmRSS_kib"] for r in rows),
"initial_hwm_kib": rows[0]["VmHWM_kib"],
"final_hwm_kib": rows[-1]["VmHWM_kib"],
"oom_delta": rows[-1]["events.oom"] - rows[0]["events.oom"],
"oom_kill_delta": (
rows[-1]["events.oom_kill"] - rows[0]["events.oom_kill"]
),
}
directory = Path("evidence")
directory.mkdir(exist_ok=True)
(directory / "upload_memory.json").write_text(
json.dumps(report, indent=2) + "\n"
)
fields = sorted({key for row in samples for key in row})
with (directory / "upload_memory.csv").open("w", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=fields, lineterminator="\n")
writer.writeheader()
writer.writerows(samples)
print("\nMEMORY SUMMARY", flush=True)
print(json.dumps(report["memory_summary"], indent=2), flush=True)
print("Container states unchanged:", before == after, flush=True)
print("Monitor errors:", monitor_errors, flush=True)
print("Saved: evidence/upload_memory.json and evidence/upload_memory.csv")
print("EXPERIMENT COMPLETE: inspect statuses and memory before conclusions")