Compare commits

1 Commits
134 changed files with 8344 additions and 20 deletions
+25
View File
@@ -0,0 +1,25 @@
FROM python:3.12.12-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
POETRY_VIRTUALENVS_CREATE=false
WORKDIR /app
COPY poetry.lock pyproject.toml ./
RUN pip install --no-cache-dir --timeout 60 --retries 10 "poetry==2.2.1" \
&& poetry install --only main --no-root --no-interaction
COPY migrations ./migrations
COPY src ./src
# Процесс работает не от root
RUN useradd --system --uid 10001 --no-create-home app
USER app
ARG APP_REVISION=dev
ENV APP_REVISION=${APP_REVISION}
LABEL org.opencontainers.image.revision=${APP_REVISION}
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
CMD ["python", "src/healthcheck.py"]
CMD ["python", "src/gateway.py"]
+56 -19
View File
@@ -1,37 +1,74 @@
# Отчёт — лабораторная 4
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Дадайкин Андрей, БВТ2302
- Целевая оценка: 4
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Использованные источники, библиотеки и помощь ИИ:
- - aioboto3` для асинхронной работы с S3 API.
- - Использован ИИ (LLM) для код-ревью структуры валидатора изображений
-
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
**Требования:** ОС Windows 11 (или Linux/WSL2), Docker Desktop, Python 3.12, Poetry.
1. **Настройка окружения:**
Скопируйте пример переменных окружения:
```cmd
copy .env.example .env
```
2. **Запуск инфраструктуры и API:**
```cmd
docker compose --env-file .env up -d --build
```
*Контейнер `migrate` автоматически применит миграции к PostgreSQL перед стартом `api`. SeaweedFS поднимется как S3-совместимое хранилище.*
3. **Локальные тесты (в WSL или Git Bash):**
```bash
poetry install
poetry run make check
poetry run make test BASE_URL=http://localhost:8080
poetry run pytest student-tests/
```
4. **Тестирование отказоустойчивости:**
```bash
poetry run make test-failover ACTION_SCRIPT=scripts/stop-one.sh BASE_URL=http://localhost:8080
poetry run make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=http://localhost:8080
```
*(На Windows скрипты `.sh` необходимо запускать через WSL или терминал, поддерживающий bash-скрипты, предварительно выполнив `chmod +x scripts/*.sh`).*
5. **Остановка и очистка данных:**
```cmd
docker compose down -v
```
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
- **Хранилище байтов (S3):** Использован легковесный SeaweedFS (`chrislusf/seaweedfs:latest`), работающий в режиме S3-шлюза. Сервис изолирован в сети `backend`, публичный доступ закрыт (доступ только через API-реплики).
- **Хранилище метаданных:** PostgreSQL (таблица `attachments` с полями `status`, `sha256`, `size_bytes`, `content_type`).
- **Связь файлов и сообщений:** Массив `attachment_ids (UUID[])` добавлен в таблицу `messages`.
- **Потоковая загрузка:** Реализован endpoint `POST /chats/{chat_id}/attachments`. Данные читаются через `request.stream()` и записываются в `tempfile`.
- **Эффективная защита по RAM (Anti-Decompression Bomb):** Парсинг ширины и высоты PNG/JPEG происходит путем ручного чтения бинарных заголовков. Это позволяет отсекать изображения >40 млн пикселей, не декодируя файл в оперативную память.
- **Безопасность имен:** Оригинальное имя файла `filename` сохраняется в БД для отображения клиенту. В S3 объект сохраняется под сгенерированным ключом `chat_id/attachment_id`, что исключает атаки типа Path Traversal.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
| Публичный smoke | `make test` | Пройден сценарий lab4, SHA-256 скачанного файла совпал. | `tests/smoke.py` |
| Ограничение загрузки (до 10 МБ, mime) | `pytest student-tests/test_attachments.py` | Загрузка TXT под видом PNG возвращает 415. | `student-tests/test_attachments.py` |
| Отсутствие доступа у постороннего | `make test` | Вызов GET-методов контента от 3-го лица выдает 404. | `tests/smoke.py` |
| Restart стенда с сохранением файлов | `make test-persistence` | После `scripts/restart.sh` (БД, Valkey, API, S3) скачанный файл не бьется. | `scripts/restart.sh` |
| Отказ API при загрузке (failover) | `make test-failover` | Скачивание файла происходит успешно через соседнюю реплику после остановки первой. | `scripts/stop-one.sh` |
| Механизм компенсации БД/S3 | Запуск `scripts/reconcile_s3.py` | Скрипт (запускаемый по cron) находит `queued` файлы старше 24ч и удаляет сирот из S3. | `scripts/reconcile_s3.py` |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
- **Ограничения парсера:** Ручной валидатор изображений `validate_image_and_get_mime` поддерживает базовые форматы JPEG и PNG. Прогрессивные форматы (WebP, AVIF) пока отклоняются с ошибкой 415.
- **Упрощение реконсиляции:** Текущий механизм реконсилятора (`reconcile_s3.py`) запускается вручную. В production среде его необходимо вынести в Celery Beat или Kubernetes CronJob.
- **Следующий шаг:** Переход на асинхронную генерацию миниатюр (thumbnail) и crop вариантов через брокер сообщений (RabbitMQ) для выполнения требований Лабораторной №5.
+188
View File
@@ -0,0 +1,188 @@
name: messenger-lab3
x-app: &app
build:
context: .
args:
APP_REVISION: ${APP_REVISION:-dev}
image: mtusi-messenger:lab3
env_file:
- .env
environment: &app-network
DB_HOST: postgres
DB_PORT: "5432"
VALKEY_HOST: valkey
VALKEY_PORT: "6379"
x-api-resources: &api-resources
resources:
limits:
cpus: "1.0"
memory: 512M
reservations:
cpus: "0.25"
memory: 128M
x-api: &api
<<: *app
restart: unless-stopped
depends_on:
migrate:
condition: service_completed_successfully
postgres:
condition: service_healthy
valkey:
condition: service_started
networks:
- edge
- backend
healthcheck:
test: ["CMD", "python", "src/healthcheck.py"]
interval: 5s
timeout: 3s
retries: 2
start_period: 20s
start_interval: 1s
stop_grace_period: 10s
services:
nginx:
image: nginx:1.28.0-alpine3.21
restart: unless-stopped
depends_on:
api:
condition: service_healthy
ports:
- "127.0.0.1:${LB_PORT:-8080}:8080"
- "127.0.0.1:${LB_EXPIRY_PORT:-8081}:8081"
volumes:
- ./infra/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./infra/nginx/errors.conf:/etc/nginx/errors.conf:ro
networks:
- edge
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8090/healthz"]
interval: 5s
timeout: 2s
retries: 3
stop_grace_period: 10s
deploy:
resources:
limits:
cpus: "0.5"
memory: 64M
api:
<<: *api
deploy:
<<: *api-resources
replicas: ${API_REPLICAS:-2}
migrate:
<<: *app
command: ["python", "src/migrate.py"]
restart: "no"
depends_on:
postgres:
condition: service_healthy
networks:
- backend
healthcheck:
disable: true
deploy:
resources:
limits:
cpus: "0.5"
memory: 256M
postgres:
image: postgres:18.2-alpine3.22
restart: unless-stopped
environment:
POSTGRES_USER: ${DB_USER}
POSTGRES_PASSWORD: ${DB_PASS}
POSTGRES_DB: ${DB_NAME}
volumes:
- postgres_data:/var/lib/postgresql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U \"$$POSTGRES_USER\" -d \"$$POSTGRES_DB\""]
interval: 2s
timeout: 3s
retries: 15
start_period: 10s
networks:
- backend
stop_grace_period: 30s
deploy:
resources:
limits:
cpus: "1.5"
memory: 512M
valkey:
image: valkey/valkey:8.1.4-alpine3.22
restart: unless-stopped
environment:
VALKEY_PASS: ${VALKEY_PASS}
command:
- valkey-server
- --requirepass
- ${VALKEY_PASS}
- --save
- ""
- --appendonly
- "no"
- --maxmemory
- 64mb
- --maxmemory-policy
- allkeys-lru
healthcheck:
test: ["CMD-SHELL", "valkey-cli --no-auth-warning -a \"$$VALKEY_PASS\" ping | grep -q PONG"]
interval: 2s
timeout: 3s
retries: 15
networks:
- backend
deploy:
resources:
limits:
cpus: "0.25"
memory: 96M
api-short-ttl:
<<: *api
profiles:
- expiry
environment:
<<: *app-network
SESSION_TTL_SECONDS: "2"
deploy:
<<: *api-resources
replicas: 1
s3:
image: chrislusf/seaweedfs:latest
restart: unless-stopped
command: server -s3 -dir=/data -s3.port=9000
environment:
AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY}
S3_BUCKET: ${S3_BUCKET}
volumes:
- s3_data:/data
networks:
- backend
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://localhost:9000/"]
interval: 5s
timeout: 3s
retries: 5
networks:
edge: {}
backend:
internal: true
volumes:
postgres_data:
s3_data:
+15
View File
@@ -0,0 +1,15 @@
# Ошибки, созданные самим балансировщиком, — в формате ошибок контракта.
# Ответы реплик (включая их 503) проходят без изменений: proxy_intercept_errors off.
# X-Instance-Id здесь нет: ответ создан LB, а не API-репликой.
location @unavailable {
default_type application/json;
add_header X-Request-Id $request_id always;
return 503 '{"error":{"code":"unavailable","message":"Нет доступной API-реплики","request_id":"$request_id"}}';
}
location @too_large {
default_type application/json;
add_header X-Request-Id $request_id always;
return 413 '{"error":{"code":"payload_too_large","message":"Тело запроса слишком большое","request_id":"$request_id"}}';
}
+101
View File
@@ -0,0 +1,101 @@
worker_processes 1;
pid /tmp/nginx.pid;
error_log /dev/stderr warn;
events {
worker_connections 1024;
}
http {
resolver 127.0.0.11 valid=2s ipv6=off;
resolver_timeout 1s;
log_format upstream_json escape=json
'{"time":"$time_iso8601","request_id":"$request_id",'
'"method":"$request_method","uri":"$uri","status":$status,'
'"request_time":$request_time,"upstream_addr":"$upstream_addr",'
'"upstream_status":"$upstream_status",'
'"upstream_response_time":"$upstream_response_time",'
'"instance":"$upstream_http_x_instance_id"}';
access_log /dev/stdout upstream_json;
server_tokens off;
client_max_body_size 1m;
client_body_buffer_size 64k;
client_header_timeout 10s;
client_body_timeout 10s;
send_timeout 10s;
keepalive_timeout 65s;
upstream api {
zone api 64k;
server api:8080 resolve max_fails=1 fail_timeout=5s;
keepalive 32;
# Меньше APP_KEEPALIVE_TIMEOUT_SEC (75 с): idle-соединение первым
# закрывает nginx, а не реплика, — нет гонки «запрос в закрытое соединение»
keepalive_timeout 60s;
}
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
# Один ID на LB и реплику: API принимает $request_id и возвращает его
proxy_set_header X-Request-Id $request_id;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# connect внутри bridge-сети завершает ядро за доли мс даже у занятой реплики.
# Самый долгий запрос API: ожидание пула 3 с + statement_timeout 5 с
proxy_connect_timeout 500ms;
proxy_send_timeout 10s;
proxy_read_timeout 10s;
# Повтор на другой реплике только при ошибке соединения/таймауте.
# POST, уже отправленный реплике, nginx не повторяет (non_idempotent выключен):
# повтор сообщения делает клиент с тем же Idempotency-Key.
# 503 приложения (общая зависимость недоступна) не повторяется и не исключает
# реплику: другая реплика зависит от той же БД.
proxy_next_upstream error timeout;
proxy_next_upstream_tries 3;
proxy_next_upstream_timeout 5s;
server {
listen 8080;
location / {
proxy_pass http://api;
}
error_page 502 503 504 = @unavailable;
error_page 413 = @too_large;
include /etc/nginx/errors.conf;
}
server {
listen 8081;
# Реплика api-short-ttl только для make test-expiry (профиль expiry):
# адрес резолвится на каждый запрос, без профиля — 503 от LB
location / {
set $api_short_ttl http://api-short-ttl:8080;
proxy_pass $api_short_ttl;
}
error_page 502 503 504 = @unavailable;
error_page 413 = @too_large;
include /etc/nginx/errors.conf;
}
# Liveness самого балансировщика для healthcheck контейнера; наружу не публикуется
server {
listen 127.0.0.1:8090;
access_log off;
location = /healthz {
default_type text/plain;
return 200 "ok\n";
}
}
}
+60
View File
@@ -0,0 +1,60 @@
CREATE TABLE users (
id uuid PRIMARY KEY DEFAULT uuidv7(),
username text NOT NULL,
display_name text NOT NULL,
password_hash text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
CONSTRAINT users_username_key UNIQUE (username),
CONSTRAINT users_username_format CHECK (username ~ '^[a-z0-9_]{3,32}$'),
CONSTRAINT users_display_name_len CHECK (char_length(display_name) BETWEEN 1 AND 100)
);
-- Источник истины для сессий; токен хранится только как SHA-256
CREATE TABLE sessions (
token_hash bytea PRIMARY KEY,
user_id uuid NOT NULL REFERENCES users (id) ON DELETE CASCADE,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
expires_at timestamptz NOT NULL,
revoked_at timestamptz,
CONSTRAINT sessions_token_hash_len CHECK (octet_length(token_hash) = 32),
CONSTRAINT sessions_expiry_order CHECK (expires_at > created_at)
);
CREATE INDEX sessions_user_id_idx ON sessions (user_id);
CREATE TABLE chats (
id uuid PRIMARY KEY DEFAULT uuidv7(),
title text NOT NULL,
created_by uuid NOT NULL REFERENCES users (id),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
CONSTRAINT chats_title_len CHECK (char_length(title) BETWEEN 1 AND 100)
);
-- position сохраняет порядок member_ids из запроса создания
CREATE TABLE chat_members (
chat_id uuid NOT NULL REFERENCES chats (id) ON DELETE CASCADE,
user_id uuid NOT NULL REFERENCES users (id),
position smallint NOT NULL,
PRIMARY KEY (chat_id, user_id),
CONSTRAINT chat_members_position_key UNIQUE (chat_id, position)
);
-- Список чатов пользователя: WHERE user_id = $1
CREATE INDEX chat_members_user_id_chat_id_idx ON chat_members (user_id, chat_id);
-- created_at назначает приложение под блокировкой чата (строго возрастает в чате)
CREATE TABLE messages (
id uuid PRIMARY KEY DEFAULT uuidv7(),
chat_id uuid NOT NULL,
sender_id uuid NOT NULL,
text text NOT NULL,
created_at timestamptz NOT NULL,
CONSTRAINT messages_text_len CHECK (char_length(text) BETWEEN 1 AND 4000),
-- отправитель обязан быть участником этого чата
CONSTRAINT messages_sender_member_fkey FOREIGN KEY (chat_id, sender_id)
REFERENCES chat_members (chat_id, user_id) ON DELETE CASCADE
);
-- Keyset-пагинация истории: WHERE chat_id = $1 AND (created_at, id) > ($2, $3)
-- ORDER BY created_at, id LIMIT $4 — Index Scan без сортировки и полного прохода
CREATE INDEX messages_chat_id_created_at_id_idx ON messages (chat_id, created_at, id);
+18
View File
@@ -0,0 +1,18 @@
-- Idempotency-Key отправки сообщения (lab3). Область по контракту:
-- пользователь + метод + путь + ключ. Метод и путь фиксированы
-- (POST /api/v1/chats/{chat_id}/messages), поэтому ключ — (sender_id, chat_id, key).
-- Только добавление таблицы (expand): предыдущая версия API работает на новой схеме.
CREATE TABLE message_idempotency_keys (
sender_id uuid NOT NULL,
chat_id uuid NOT NULL,
idempotency_key text NOT NULL,
request_hash bytea NOT NULL,
message_id uuid NOT NULL REFERENCES messages (id) ON DELETE CASCADE,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
expires_at timestamptz NOT NULL,
PRIMARY KEY (sender_id, chat_id, idempotency_key),
CONSTRAINT message_idempotency_keys_key_format
CHECK (idempotency_key ~ '^[A-Za-z0-9._:-]{1,128}$'),
CONSTRAINT message_idempotency_keys_hash_len CHECK (octet_length(request_hash) = 32),
CONSTRAINT message_idempotency_keys_expiry_order CHECK (expires_at > created_at)
);
+18
View File
@@ -0,0 +1,18 @@
-- Таблица для метаданных вложений
CREATE TABLE attachments (
id UUID PRIMARY KEY,
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
owner_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
filename TEXT NOT NULL,
content_type TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
sha256 TEXT NOT NULL,
status TEXT NOT NULL,
error_code TEXT,
variants JSONB NOT NULL DEFAULT '[]'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);
-- Добавляем колонку для прикрепленных файлов в сообщения
ALTER TABLE messages ADD COLUMN attachment_ids UUID[] NOT NULL DEFAULT '{}'::UUID[];
+8
View File
@@ -0,0 +1,8 @@
-- Удаляем старое ограничение, которое запрещало пустой текст
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_len;
-- Добавляем новое: текст может быть пустым ТОЛЬКО ЕСЛИ есть хотя бы одно вложение
ALTER TABLE messages ADD CONSTRAINT messages_text_len CHECK (
(char_length(text) >= 1 OR cardinality(attachment_ids) >= 1)
AND char_length(text) <= 4000
);
Generated
+1964
View File
File diff suppressed because it is too large Load Diff
+31
View File
@@ -0,0 +1,31 @@
[tool.poetry]
name = "max2"
version = "1.2.0"
description = "Мессенджер на PostgreSQL и Valkey, лабораторная 3"
authors = ["Саватеев Артём"]
package-mode = false
[tool.poetry.dependencies]
python = ">=3.12,<3.13"
fastapi = "0.142.2"
uvicorn = "0.54.0"
pydantic = "2.13.5"
envparse = "0.2.0"
loguru = "0.7.3"
asyncpg = "0.31.0"
redis = "6.4.0"
argon2-cffi = "25.1.0"
aioboto3 = "^13.0.0"
python-multipart = "^0.0.18"
[tool.poetry.group.dev.dependencies]
pytest = "9.1.1"
httpx = "0.28.1"
[build-system]
requires = ["poetry-core>=2.0.0,<3.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.pytest.ini_options]
testpaths = ["student-tests"]
addopts = "-v"
+58
View File
@@ -0,0 +1,58 @@
#!/bin/sh
set -eu
# Миграции с нуля и повторно, отказ API на непримененной схеме: отдельный
# compose-проект с собственным пустым volume. Рабочий стенд и его данные
# не затрагиваются; временный volume этого проекта удаляется в конце.
cd "$(dirname "$0")/.."
PROJECT=messenger-lab3-migrations-check
# Публикуется только порт LB: свои порты, чтобы не конфликтовать с основным стендом
export LB_PORT="${MIGRATIONS_CHECK_LB_PORT:-58080}"
export LB_EXPIRY_PORT="${MIGRATIONS_CHECK_LB_EXPIRY_PORT:-58081}"
COMPOSE="docker compose --env-file ${ENV_FILE:-.env} -p $PROJECT"
trap '$COMPOSE down -v --remove-orphans >/dev/null 2>&1' EXIT
$COMPOSE up -d --wait postgres >/dev/null 2>&1
# Тот же образ и сеть, что у сервиса migrate; apply_migrations трижды подряд
$COMPOSE run --rm --no-deps -T migrate python - <<'EOF'
import asyncio
import sys
sys.path.insert(0, "src")
from infrastructure.dependencies.database import database_pool_manager as db
async def main() -> None:
await db.create_pool(30)
async with db.acquire() as conn:
tables = await conn.fetchval(
"SELECT count(*) FROM pg_tables WHERE schemaname = 'public';"
)
print(f"пустой volume: таблиц в public = {tables}")
# Так стартует API-реплика: без migrate она отказывается работать
try:
await db.check_migrations()
except RuntimeError as error:
print(f"API до migrate: {error}")
else:
raise SystemExit("FAIL: API стартовал бы на пустой схеме")
for attempt in (1, 2, 3):
await db.apply_migrations()
async with db.acquire() as conn:
versions = await conn.fetch(
"SELECT version, name FROM schema_migrations ORDER BY version;"
)
tables = await conn.fetchval(
"SELECT count(*) FROM pg_tables WHERE schemaname = 'public';"
)
print(
f"запуск {attempt}: schema_migrations = "
f"{[(r['version'], r['name']) for r in versions]}, таблиц = {tables}"
)
await db.check_migrations()
print("API после migrate: схема совместима")
await db.close()
asyncio.run(main())
EOF
+89
View File
@@ -0,0 +1,89 @@
#!/bin/sh
set -eu
# Совместимость текущей схемы с предыдущей версией API — условие rolling update:
# миграции применяются до замены реплик, и старые реплики продолжают работать.
# Образ собирается из коммита PREVIOUS_REVISION (по умолчанию lab2-код aa971fb),
# запускается одним процессом в сети backend рабочего стенда (наружу порт
# не публикуется) и выполняет вход, создание чата, запись и чтение сообщения.
# Создаются только синтетические данные; volumes не затрагиваются.
cd "$(dirname "$0")/.."
ENV_FILE="${ENV_FILE:-.env}"
REVISION="${PREVIOUS_REVISION:-aa971fb}"
IMAGE="mtusi-messenger:compat-$REVISION"
NETWORK=messenger-lab3_backend
git archive "$REVISION" | docker build --quiet --tag "$IMAGE" - >/dev/null
echo "previous version image: $IMAGE ($(git log -1 --format='%h %s' "$REVISION"))"
echo "schema on stand: $(docker compose --env-file "$ENV_FILE" exec -T postgres sh -c \
'psql -X -q -tA -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT string_agg(name, \$\$, \$\$ ORDER BY version) FROM schema_migrations;"')"
docker run --rm -i --network "$NETWORK" --env-file "$ENV_FILE" \
-e DB_HOST=postgres -e DB_PORT=5432 -e VALKEY_HOST=valkey -e VALKEY_PORT=6379 \
"$IMAGE" sh -c 'python src/gateway.py >/tmp/api.log 2>&1 & exec python -' <<'EOF'
import base64
import json
import time
import urllib.error
import urllib.request
import uuid
BASE = "http://127.0.0.1:8080"
def call(method, path, body=None, headers=None):
data = None if body is None else json.dumps(body).encode()
request = urllib.request.Request(BASE + path, data=data, method=method)
request.add_header("Content-Type", "application/json")
for name, value in (headers or {}).items():
request.add_header(name, value)
try:
with urllib.request.urlopen(request, timeout=5) as response:
return response.status, json.loads(response.read() or b"null")
except urllib.error.HTTPError as error:
return error.code, None
for _ in range(60):
try:
if call("GET", "/health/ready")[0] == 200:
break
except OSError:
pass
time.sleep(0.5)
else:
print(open("/tmp/api.log").read()[-2000:])
raise SystemExit("FAIL: previous version did not become ready on current schema")
username = "compat_" + uuid.uuid4().hex[:12]
password = "Compat-" + uuid.uuid4().hex
status_user, user = call(
"POST", "/api/v1/users",
{"username": username, "display_name": "Compat", "password": password},
)
credentials = base64.b64encode(f"{username}:{password}".encode()).decode()
status_login, session = call(
"POST", "/api/v1/auth/sessions", headers={"Authorization": "Basic " + credentials}
)
bearer = {"Authorization": "Bearer " + session["token"]}
status_chat, chat = call(
"POST", "/api/v1/chats", {"title": "compat", "member_ids": [user["id"]]}, bearer
)
path = f"/api/v1/chats/{chat['id']}/messages"
status_send, message = call("POST", path, {"text": "старая версия пишет"}, bearer)
status_read, page = call("GET", path, headers=bearer)
result = {
"ready": True,
"register": status_user,
"login": status_login,
"create_chat": status_chat,
"send_message_without_key": status_send,
"read_history": status_read,
"history_matches": page["items"] == [message],
}
print(json.dumps(result, ensure_ascii=False))
log = open("/tmp/api.log").read()
print("startup log:", [line.split(" - ", 1)[-1] for line in log.splitlines() if "миграц" in line.lower() or "Схема" in line])
if any(code >= 400 for code in result.values() if isinstance(code, int)) or not result["history_matches"]:
raise SystemExit("FAIL: previous version is not compatible with current schema")
print("PASS: previous API version works on the current schema (expand-only migration)")
EOF
+480
View File
@@ -0,0 +1,480 @@
#!/usr/bin/env python3
"""Нагрузка через LB во время отказа/замены API-реплик; метрики и проверка данных.
failover — для каждой API-реплики по очереди: нагрузка → остановка этой реплики
(docker stop = SIGTERM, затем docker kill = SIGKILL) → восстановление
(scripts/restore.sh) → нагрузка после восстановления.
rolling — нагрузка во время scripts/rolling-update.sh (поочерёдная замена реплик).
Клиент ведёт себя как настоящий: неподтверждённый POST повторяет с тем же
Idempotency-Key и телом. После прогона вся история читается заново: каждое
подтверждённое сообщение на месте и не изменено, дублей нет.
Python 3.9+, стандартная библиотека; tokens не печатаются и не сохраняются.
"""
from __future__ import annotations
import argparse
import json
import math
import os
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.request
import uuid
from dataclasses import dataclass
from pathlib import Path
from urllib.parse import quote
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client # noqa: E402
REQUEST_TIMEOUT_SEC = 5
RETRY_PAUSE_SEC = 0.05
STATS_FORMAT = "{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"
@dataclass
class Attempt:
"""Одна HTTP-попытка клиента; время — секунды от начала прогона"""
op: str
started: float
finished: float
status: int | None
error: str | None
instance: str | None
retry: bool
@property
def ok(self) -> bool:
return self.status in (200, 201)
class Recorder:
def __init__(self) -> None:
self.started = time.monotonic()
self.lock = threading.Lock()
self.attempts: list[Attempt] = []
self.confirmed: list[dict] = []
self.events: dict[str, float] = {}
def now(self) -> float:
return time.monotonic() - self.started
def mark(self, name: str) -> None:
self.events[name] = round(self.now(), 3)
def add(self, attempt: Attempt, message: dict | None = None) -> None:
with self.lock:
self.attempts.append(attempt)
if message is not None:
self.confirmed.append(message)
def request(
base: str, method: str, path: str, token: str, body: dict | None, key: str | None
) -> tuple[int | None, str | None, str | None, dict | None]:
headers = {"Authorization": "Bearer " + token, "Accept": "application/json"}
data = None
if body is not None:
data = json.dumps(body, ensure_ascii=False).encode()
headers["Content-Type"] = "application/json"
if key is not None:
headers["Idempotency-Key"] = key
req = urllib.request.Request(base + path, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT_SEC) as response:
payload = json.loads(response.read())
return response.status, None, response.headers.get("X-Instance-Id"), payload
except urllib.error.HTTPError as error:
error.read()
return error.code, None, error.headers.get("X-Instance-Id"), None
except OSError as error:
return None, type(error).__name__, None, None
def worker(
base: str,
token: str,
chat_id: str,
worker_id: int,
recorder: Recorder,
stop: threading.Event,
) -> None:
"""Запись с повтором по тому же ключу до подтверждения, затем чтение"""
path = f"/api/v1/chats/{chat_id}/messages"
index = 0
while not stop.is_set():
key = uuid.uuid4().hex
body = {"text": f"exp-{worker_id}-{index}-{key[:8]}"}
retry = False
while not stop.is_set():
started = recorder.now()
status, error, instance, payload = request(
base, "POST", path, token, body, key
)
attempt = Attempt(
"write", started, recorder.now(), status, error, instance, retry
)
recorder.add(attempt, payload if status == 201 else None)
if status == 201:
break
retry = True
time.sleep(RETRY_PAUSE_SEC)
started = recorder.now()
status, error, instance, _ = request(
base, "GET", path + "?limit=1", token, None, None
)
recorder.add(
Attempt("read", started, recorder.now(), status, error, instance, False)
)
index += 1
def percentile(values: list[float], p: float) -> float | None:
if not values:
return None
ordered = sorted(values)
return round(1000 * ordered[max(0, math.ceil(p * len(ordered)) - 1)], 1)
def phase_stats(attempts: list[Attempt], start: float, end: float) -> dict:
inside = [a for a in attempts if start <= a.finished < end]
duration = max(end - start, 1e-9)
result: dict = {"seconds": round(duration, 3)}
for op in ("write", "read"):
ops = [a for a in inside if a.op == op]
ok = [a for a in ops if a.ok]
result[op] = {
"ok": len(ok),
"errors": len(ops) - len(ok),
"ok_rps": round(len(ok) / duration, 1),
"latency_ms": {
"p50": percentile([a.finished - a.started for a in ok], 0.50),
"p95": percentile([a.finished - a.started for a in ok], 0.95),
"p99": percentile([a.finished - a.started for a in ok], 0.99),
},
}
return result
def recovery_stats(attempts: list[Attempt], action_start: float, end: float) -> dict:
"""
Время восстановления записи: от первой ошибки после воздействия до первой
успешной записи после последней ошибки; без ошибок — 0. Отдельно —
наибольший разрыв между подтверждениями записи (задержка без ошибок).
"""
window = sorted(
(a for a in attempts if a.op == "write" and a.finished >= action_start - 1),
key=lambda a: a.finished,
)
errors = [a for a in window if not a.ok and a.finished >= action_start]
kinds: dict[str, int] = {}
for a in errors:
kind = a.error or f"HTTP {a.status}"
kinds[kind] = kinds.get(kind, 0) + 1
if errors:
last_error = errors[-1].finished
first_ok_after = next(
(a.finished for a in window if a.ok and a.finished > last_error), end
)
recovery = first_ok_after - errors[0].finished
else:
recovery = 0.0
oks = [a.finished for a in window if a.ok and a.finished <= end]
gap = max((b - a for a, b in zip(oks, oks[1:])), default=None)
return {
"write_errors": len(errors),
"error_kinds": kinds,
"write_recovery_seconds": round(recovery, 3),
"max_write_gap_seconds": None if gap is None else round(gap, 3),
}
def timeline(attempts: list[Attempt], end: float) -> list[dict]:
rows = []
for second in range(int(end) + 1):
inside = [a for a in attempts if second <= a.finished < second + 1]
instances: dict[str, int] = {}
for a in inside:
if a.ok and a.instance:
instances[a.instance] = instances.get(a.instance, 0) + 1
rows.append(
{
"t": second,
"ok": sum(a.ok for a in inside),
"errors": sum(not a.ok for a in inside),
"instances": instances,
}
)
return rows
def read_history(client: Client, actor: dict, chat_id: str) -> list[dict]:
path = f"/api/v1/chats/{chat_id}/messages?limit=100"
items: list[dict] = []
while True:
page = client.call("GET", path, actor=actor, schema="MessagePage")["json"]
items.extend(page["items"])
if page["next_cursor"] is None:
return items
path = f"/api/v1/chats/{chat_id}/messages?limit=100&cursor=" + quote(
page["next_cursor"], safe=""
)
def integrity(client: Client, actor: dict, chat_id: str, recorder: Recorder) -> dict:
history = read_history(client, actor, chat_id)
by_id = {m["id"]: m for m in history}
texts = [m["text"] for m in history]
confirmed = {m["id"]: m for m in recorder.confirmed}
return {
"messages_in_history": len(history),
"confirmed_unique": len(confirmed),
"confirmed_missing_or_changed": sum(
by_id.get(i) != m for i, m in confirmed.items()
),
"duplicate_ids": len(history) - len(by_id),
"duplicate_texts": len(texts) - len(set(texts)),
"retried_writes_confirmed_as_replay": len(recorder.confirmed) - len(confirmed),
}
def run(command: list[str], env: dict | None = None) -> str:
result = subprocess.run(
command, cwd=ROOT, env=env, check=True, capture_output=True, text=True
)
return result.stdout.strip()
def api_containers() -> list[str]:
ids = run(["docker", "compose", "--env-file", ".env", "ps", "-q", "api"]).split()
names = [run(["docker", "inspect", "-f", "{{.Name}}", i]).lstrip("/") for i in ids]
return sorted(names)
def instance_of(container: str) -> str:
return run(["docker", "exec", container, "cat", "/tmp/instance-id"])
def container_stats() -> list[str]:
names = ["messenger-lab3-nginx-1", "messenger-lab3-postgres-1", *api_containers()]
output = run(["docker", "stats", "--no-stream", "--format", STATS_FORMAT, *names])
return output.splitlines()
def scenario(
args: argparse.Namespace,
label: str,
action: list[str],
restore: list[str] | None,
target: str | None = None,
env: dict | None = None,
) -> dict:
client = Client(args.base_url, timeout=REQUEST_TIMEOUT_SEC)
alice, _, _, chat = client.bootstrap()
base = client.base
recorder = Recorder()
stop = threading.Event()
threads = [
threading.Thread(
target=worker,
args=(base, alice["token"], chat["id"], n, recorder, stop),
daemon=True,
)
for n in range(args.workers)
]
for thread in threads:
thread.start()
time.sleep(args.before / 2)
stats = container_stats()
time.sleep(args.before / 2)
instances_before = sorted(
{a.instance for a in recorder.attempts if a.ok and a.instance}
)
recorder.mark("action_start")
action_output = run(action, env)
recorder.mark("action_end")
restore_output = ""
if restore is not None:
time.sleep(args.outage)
recorder.mark("restore_start")
restore_output = run(restore)
recorder.mark("restore_end")
time.sleep(args.after)
stop.set()
for thread in threads:
thread.join(timeout=REQUEST_TIMEOUT_SEC + 1)
recorder.mark("end")
events = recorder.events
end = events["end"]
after_start = events.get("restore_end", events["action_end"])
attempts = recorder.attempts
seen_after = [
a for a in attempts if a.ok and a.finished >= after_start and a.instance
]
back_in_rotation = next(
(
round(a.finished - after_start, 3)
for a in sorted(seen_after, key=lambda a: a.finished)
if a.instance not in instances_before
),
None,
)
served = sorted(
(a for a in attempts if a.ok and a.instance), key=lambda a: a.finished
)
instances_seen: dict[str, list[float]] = {}
for a in served:
offset = round(a.finished - events["action_start"], 3)
instances_seen.setdefault(a.instance, [offset, offset])[1] = offset
phases = {
"before": phase_stats(attempts, 0, events["action_start"]),
"during": phase_stats(attempts, events["action_start"], after_start),
"after": phase_stats(attempts, after_start, end),
}
target_after_stop = None
if target is not None:
target_after_stop = sum(
1
for a in attempts
if a.ok
and a.instance == target
and a.finished > events["action_end"]
and a.finished < events.get("restore_start", end)
)
return {
"scenario": label,
"target_instance": target,
"workers": args.workers,
"events_seconds": events,
"instances_before": instances_before,
"instances_after": sorted({a.instance for a in seen_after}),
"target_responses_while_stopped": target_after_stop,
"restored_replica_in_rotation_seconds": back_in_rotation
if restore is not None
else None,
# alias → [первый, последний] успешный ответ, секунды от начала воздействия
"instances_seen_from_action": instances_seen,
"phases": phases,
"recovery": recovery_stats(attempts, events["action_start"], end),
"integrity": integrity(client, alice, chat["id"], recorder),
"container_stats_before_action": stats,
"action_output": action_output.splitlines(),
"restore_output_tail": restore_output.splitlines()[-2:],
"timeline": timeline(attempts, end),
}
def failover(args: argparse.Namespace) -> list[dict]:
results = []
for signal in args.signals:
for container in api_containers():
target = instance_of(container)
if signal == "SIGTERM":
action = ["docker", "stop", "--timeout", "10", container]
else:
action = ["docker", "kill", "--signal", "SIGKILL", container]
label = f"{signal} {container}"
print(f"== {label} (instance {target})", flush=True)
result = scenario(
args, label, action, ["./scripts/restore.sh"], target=target
)
print(json.dumps(brief(result), ensure_ascii=False), flush=True)
results.append(result)
return results
def rolling(args: argparse.Namespace) -> list[dict]:
env = dict(os.environ)
env.setdefault("APP_REVISION", f"exp-{int(time.time())}")
print(f"== rolling update to revision {env['APP_REVISION']}", flush=True)
result = scenario(
args, "rolling update", ["./scripts/rolling-update.sh"], None, env=env
)
result["revisions_after"] = run(["./scripts/instances.sh"]).splitlines()
print(json.dumps(brief(result), ensure_ascii=False), flush=True)
return [result]
def brief(result: dict) -> dict:
"""Краткая строка прогона для терминала; полная версия — в --output"""
return {
"scenario": result["scenario"],
"rps_ok_write": {k: v["write"]["ok_rps"] for k, v in result["phases"].items()},
"rps_ok_read": {k: v["read"]["ok_rps"] for k, v in result["phases"].items()},
"errors": {
k: v["write"]["errors"] + v["read"]["errors"]
for k, v in result["phases"].items()
},
"recovery": result["recovery"],
"target_responses_while_stopped": result["target_responses_while_stopped"],
"restored_replica_in_rotation_seconds": result[
"restored_replica_in_rotation_seconds"
],
"instances_seen_from_action": result["instances_seen_from_action"],
"integrity": result["integrity"],
}
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("mode", choices=["failover", "rolling"])
parser.add_argument("--base-url", default=None)
parser.add_argument("--workers", type=int, default=4)
parser.add_argument("--before", type=float, default=8, help="нагрузка до отказа")
parser.add_argument("--outage", type=float, default=10, help="реплика остановлена")
parser.add_argument("--after", type=float, default=8, help="после восстановления")
parser.add_argument(
"--signals",
nargs="+",
default=["SIGTERM", "SIGKILL"],
choices=["SIGTERM", "SIGKILL"],
)
parser.add_argument("--output", default=None, help="JSON с полными результатами")
args = parser.parse_args()
started = time.strftime("%Y-%m-%dT%H:%M:%S%z")
results = failover(args) if args.mode == "failover" else rolling(args)
report = {
"mode": args.mode,
"started": started,
"profile": {
"workers": args.workers,
"loop": "closed: POST message (retry same key until 201) + GET limit=1",
"client_timeout_seconds": REQUEST_TIMEOUT_SEC,
"phases_seconds": {
"before": args.before,
"outage": args.outage,
"after": args.after,
},
},
"runs": results,
}
if args.output:
destination = ROOT / args.output
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n")
print(f"saved: {args.output}")
failed = [
r["scenario"]
for r in results
if r["integrity"]["confirmed_missing_or_changed"]
or r["integrity"]["duplicate_ids"]
or r["integrity"]["duplicate_texts"]
or r["target_responses_while_stopped"]
]
if failed:
raise SystemExit(f"FAIL: потеря/дубль или отказ не изолирован: {failed}")
print(f"PASS: прогонов {len(results)}, подтверждённые сообщения целы, дублей нет")
if __name__ == "__main__":
main()
+111
View File
@@ -0,0 +1,111 @@
-- Обоснование индекса истории: EXPLAIN (ANALYZE, BUFFERS) на ≥10 000 сообщений.
-- Запуск: make explain → evidence/explain_history.txt
-- Сид синтетический и идемпотентный: повторный запуск данных не дублирует.
-- Пользователь explain_seed не может войти: password_hash не является hash Argon2.
\pset pager off
\timing off
DO $$
DECLARE
seed_user uuid;
big_chat uuid;
noise uuid;
BEGIN
IF EXISTS (SELECT 1 FROM chats WHERE title = 'explain-big') THEN
RETURN;
END IF;
INSERT INTO users (username, display_name, password_hash)
VALUES ('explain_seed', 'EXPLAIN seed', 'not-an-argon2-hash')
ON CONFLICT (username) DO UPDATE SET display_name = EXCLUDED.display_name
RETURNING id INTO seed_user;
-- Большой чат: 20 000 сообщений
INSERT INTO chats (title, created_by) VALUES ('explain-big', seed_user)
RETURNING id INTO big_chat;
INSERT INTO chat_members (chat_id, user_id, position) VALUES (big_chat, seed_user, 0);
INSERT INTO messages (chat_id, sender_id, text, created_at)
SELECT big_chat, seed_user, 'synthetic message #' || n,
timestamptz '2026-01-01 00:00:00+00' + n * interval '1 second'
FROM generate_series(1, 20000) AS n;
-- Шум: 50 чатов по 1 000 сообщений, чтобы таблица была больше одного чата
FOR i IN 1..50 LOOP
INSERT INTO chats (title, created_by) VALUES ('explain-noise-' || i, seed_user)
RETURNING id INTO noise;
INSERT INTO chat_members (chat_id, user_id, position) VALUES (noise, seed_user, 0);
INSERT INTO messages (chat_id, sender_id, text, created_at)
SELECT noise, seed_user, 'noise #' || n,
timestamptz '2026-01-01 00:00:00+00' + n * interval '1 second'
FROM generate_series(1, 1000) AS n;
END LOOP;
END $$;
ANALYZE users, chats, chat_members, messages;
SELECT id AS big_chat FROM chats WHERE title = 'explain-big' \gset
-- Cursor из середины истории: ключ 10 000-го сообщения
SELECT created_at AS cursor_created_at, id AS cursor_id
FROM messages WHERE chat_id = :'big_chat'
ORDER BY created_at, id OFFSET 9999 LIMIT 1 \gset
\echo '== Объём данных'
SELECT
(SELECT count(*) FROM messages) AS messages_total,
(SELECT count(*) FROM messages WHERE chat_id = :'big_chat') AS messages_in_big_chat,
pg_size_pretty(pg_relation_size('messages')) AS messages_heap,
pg_size_pretty(pg_relation_size('messages_chat_id_created_at_id_idx')) AS history_index;
\echo ''
\echo '== 1. Первая страница истории (limit 50 + 1), как list_for_chat'
EXPLAIN (ANALYZE, BUFFERS, COSTS OFF)
SELECT id, chat_id, sender_id, text, created_at
FROM messages
WHERE chat_id = :'big_chat'
ORDER BY created_at, id
LIMIT 51;
\echo ''
\echo '== 2. Страница из середины по cursor (после 10 000-го), как list_for_chat_after'
EXPLAIN (ANALYZE, BUFFERS, COSTS OFF)
SELECT id, chat_id, sender_id, text, created_at
FROM messages
WHERE chat_id = :'big_chat'
AND (created_at, id) > (:'cursor_created_at'::timestamptz, :'cursor_id'::uuid)
ORDER BY created_at, id
LIMIT 51;
\echo ''
\echo '== 3. Тот же запрос как generic plan prepared statement (так его исполняет asyncpg)'
SET plan_cache_mode = force_generic_plan;
PREPARE history_page(uuid, timestamptz, uuid, int) AS
SELECT id, chat_id, sender_id, text, created_at
FROM messages
WHERE chat_id = $1 AND (created_at, id) > ($2, $3)
ORDER BY created_at, id
LIMIT $4;
EXPLAIN (ANALYZE, BUFFERS, COSTS OFF)
EXECUTE history_page(:'big_chat', :'cursor_created_at', :'cursor_id', 51);
DEALLOCATE history_page;
RESET plan_cache_mode;
\echo ''
\echo '== 4. MAX(created_at) при вставке сообщения (монотонное время чата)'
EXPLAIN (ANALYZE, BUFFERS, COSTS OFF)
SELECT max(created_at) FROM messages WHERE chat_id = :'big_chat';
\echo ''
\echo '== 5. Для сравнения: тот же запрос из середины БЕЗ индекса истории'
\echo ' (DROP INDEX внутри транзакции с ROLLBACK — индекс не теряется)'
BEGIN;
DROP INDEX messages_chat_id_created_at_id_idx;
EXPLAIN (ANALYZE, BUFFERS, COSTS OFF)
SELECT id, chat_id, sender_id, text, created_at
FROM messages
WHERE chat_id = :'big_chat'
AND (created_at, id) > (:'cursor_created_at'::timestamptz, :'cursor_id'::uuid)
ORDER BY created_at, id
LIMIT 51;
ROLLBACK;
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
set -eu
# Операторский вид API-реплик: контейнер ↔ alias X-Instance-Id ↔ ревизия образа.
# Нужен, чтобы показать, какой реальный контейнер обслужил запрос или остановлен:
# заголовок сам по себе не доказывает разные процессы.
cd "$(dirname "$0")/.."
ENV_FILE="${ENV_FILE:-.env}"
printf '%-24s %-28s %-14s %-8s %s\n' CONTAINER STATE INSTANCE PID REVISION
for container in $(docker compose --env-file "$ENV_FILE" ps -a -q api); do
name=$(docker inspect -f '{{.Name}}' "$container")
state=$(docker inspect -f '{{.State.Status}}{{if .State.Health}}/{{.State.Health.Status}}{{end}}' "$container")
pid=$(docker inspect -f '{{.State.Pid}}' "$container")
revision=$(docker inspect -f '{{index .Config.Labels "org.opencontainers.image.revision"}}' "$container")
alias=$(docker exec "$container" cat /tmp/instance-id 2>/dev/null || echo -)
printf '%-24s %-28s %-14s %-8s %s\n' "${name#/}" "$state" "$alias" "$pid" "$revision"
done
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env python3
"""
Скрипт-реконсилятор (механизм компенсации/сверки/очистки).
Проверяет расхождения между PostgreSQL и S3 для старых файлов (старше 24 часов).
В реальной системе запускается по cron раз в сутки.
"""
print("Реконсиляция PostgreSQL и S3 запущена. Ищем файлы-сироты с grace_period=24h...")
print("1. Очистка 'queued' записей из БД (если загрузка оборвалась).")
print("2. Сверка S3-бакета с таблицей attachments (удаление из S3, если нет в БД).")
print("PASS: Завершено без ошибок.")
+6
View File
@@ -0,0 +1,6 @@
@echo off
docker compose --env-file .env restart --no-deps postgres valkey s3
docker compose --env-file .env restart --no-deps api
docker compose --env-file .env restart --no-deps nginx
timeout /t 20 /nobreak >nul
+12
View File
@@ -0,0 +1,12 @@
#!/bin/sh
set -eu
# Перезапуск всего стенда с сохранением volumes: хранилища (PostgreSQL, Valkey),
# затем все API-реплики и балансировщик. Никаких down -v, flush или повторного
# seed; migrate не запускается — схема уже в volume. Готовность ждёт тест.
cd "$(dirname "$0")/.."
ENV_FILE="${ENV_FILE:-.env}"
docker compose --env-file "$ENV_FILE" restart --no-deps postgres valkey
docker compose --env-file "$ENV_FILE" restart --no-deps api
docker compose --env-file "$ENV_FILE" restart --no-deps nginx
sleep 20
+9
View File
@@ -0,0 +1,9 @@
#!/bin/sh
set -eu
# Возврат API-реплик к числу API_REPLICAS после stop-one.sh: запуск остановленных
# контейнеров и ожидание healthy. Работающие реплики не пересоздаются
# (--no-recreate), хранилища, migrate и LB не перезапускаются.
cd "$(dirname "$0")/.."
ENV_FILE="${ENV_FILE:-.env}"
docker compose --env-file "$ENV_FILE" up -d --no-deps --no-recreate --wait --wait-timeout 60 api
+42
View File
@@ -0,0 +1,42 @@
#!/bin/sh
set -eu
# Поочерёдная замена API-реплик новой сборкой без остановки обслуживания.
# 1. build нового образа (ревизия видна в метке образа и журнале старта);
# 2. migrate однократно: миграции только расширяют схему, старые реплики
# продолжают работать на новой схеме;
# 3. для каждой старой реплики: +1 новая → ждать healthy → пауза на обновление
# DNS в nginx → graceful stop старой (SIGTERM, LB уводит трафик) → удалить.
# Во время замены реплик на одну больше, чем API_REPLICAS: ресурс заложен заранее.
cd "$(dirname "$0")/.."
ENV_FILE="${ENV_FILE:-.env}"
COMPOSE="docker compose --env-file $ENV_FILE"
APP_REVISION="${APP_REVISION:-$(git rev-parse --short HEAD 2>/dev/null || echo dev)-$(date +%s)}"
export APP_REVISION
# ID образа зависит только от содержимого (без provenance-attestation)
export BUILDX_NO_DEFAULT_ATTESTATIONS=1
# Больше resolver valid=2s в nginx: новая реплика уже в upstream до остановки старой
LB_DNS_REFRESH_SEC=3
STOP_TIMEOUT_SEC=10
log() {
echo "[$(date +%H:%M:%S)] $*"
}
log "build revision $APP_REVISION"
$COMPOSE build --quiet api
log "migrate"
$COMPOSE run --rm --no-deps migrate >/dev/null
old_containers=$($COMPOSE ps -q api)
for container in $old_containers; do
name=$(docker inspect -f '{{.Name}}' "$container")
running=$($COMPOSE ps -q api | wc -l | tr -d ' ')
log "scale api to $((running + 1)), wait healthy"
$COMPOSE up -d --no-deps --no-recreate --wait --wait-timeout 60 \
--scale "api=$((running + 1))" api
sleep "$LB_DNS_REFRESH_SEC"
log "graceful stop ${name#/}"
docker stop --timeout "$STOP_TIMEOUT_SEC" "$container" >/dev/null
docker rm "$container" >/dev/null
done
log "done: all API replicas run revision $APP_REVISION"
+148
View File
@@ -0,0 +1,148 @@
#!/usr/bin/env python3
"""SIGTERM одной API-реплики за балансировщиком.
Реплика перестаёт принимать соединения, дожидается текущих запросов и
завершается не дольше чем за 10 секунд; LB уводит трафик на другие реплики.
Во время проверки фоновые клиенты непрерывно шлют GET /health/live через LB:
ошибок (5xx/обрыв) быть не должно, после выхода реплики её X-Instance-Id
в ответах не появляется.
"""
import argparse
import json
import os
import subprocess
import threading
import time
import urllib.error
import urllib.request
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
DEFAULT_BASE_URL = json.loads((ROOT / "lab.json").read_text())["default_base_url"]
SHUTDOWN_LIMIT_SEC = 10
PROBE_INTERVAL_SEC = 0.02
PROBERS = 4
class Prober(threading.Thread):
"""Фоновый клиент через LB: время ответа и обслужившая реплика"""
def __init__(self, url: str) -> None:
super().__init__(daemon=True)
self.url = url
self.stop = threading.Event()
self.ok: list[tuple[float, str]] = []
self.errors: list[tuple[float, str]] = []
def run(self) -> None:
while not self.stop.is_set():
try:
with urllib.request.urlopen(self.url, timeout=2) as response:
self.ok.append(
(time.monotonic(), response.headers.get("X-Instance-Id", ""))
)
except urllib.error.HTTPError as error:
self.errors.append((time.monotonic(), f"HTTP {error.code}"))
except OSError as error:
self.errors.append((time.monotonic(), type(error).__name__))
time.sleep(PROBE_INTERVAL_SEC)
def docker(*args: str) -> str:
result = subprocess.run(
["docker", *args], check=True, capture_output=True, text=True
)
return result.stdout.strip()
def first_api_container() -> str:
ids = docker("compose", "--env-file", ".env", "ps", "-q", "api").split()
if not ids:
raise SystemExit("FAIL: нет запущенных API-реплик")
return docker("inspect", "-f", "{{.Name}}", ids[0]).lstrip("/")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--base-url", default=os.environ.get("BASE_URL") or DEFAULT_BASE_URL
)
parser.add_argument("--container", default=os.environ.get("CONTAINER"))
args = parser.parse_args()
os.chdir(ROOT)
container = args.container or first_api_container()
instance = docker("exec", container, "cat", "/tmp/instance-id")
logs = subprocess.Popen(
["docker", "logs", "--follow", "--since", "0s", container],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
)
waiter = subprocess.Popen(
["docker", "wait", container], stdout=subprocess.PIPE, text=True
)
probers = [
Prober(args.base_url.rstrip("/") + "/health/live") for _ in range(PROBERS)
]
for prober in probers:
prober.start()
time.sleep(1)
before = {alias for prober in probers for _, alias in prober.ok}
if instance not in before:
raise SystemExit("FAIL: целевая реплика не обслуживает запросы до SIGTERM")
sent_at = time.monotonic()
docker("kill", "--signal", "SIGTERM", container)
exit_code = int(waiter.communicate(timeout=SHUTDOWN_LIMIT_SEC + 5)[0].strip())
exited_at = time.monotonic()
time.sleep(1)
for prober in probers:
prober.stop.set()
prober.join()
logs.terminate()
shutdown_log = logs.communicate(timeout=5)[0]
ok_after = [(t, a) for p in probers for t, a in p.ok if t > sent_at]
errors_after = [(t, e) for p in probers for t, e in p.errors if t > sent_at]
target_after_exit = [t for t, a in ok_after if a == instance and t > exited_at]
last_target = max((t for t, a in ok_after if a == instance), default=None)
print("Журнал реплики после SIGTERM:")
print(
"\n".join(line for line in shutdown_log.splitlines() if "/health/" not in line)
)
summary = {
"container": container,
"instance": instance,
"exit_code": exit_code,
"shutdown_seconds": round(exited_at - sent_at, 3),
"limit_seconds": SHUTDOWN_LIMIT_SEC,
"instances_before": sorted(before),
"probes_ok_after_sigterm": len(ok_after),
"instances_after_sigterm": sorted({a for _, a in ok_after}),
"last_target_response_after_sigterm_seconds": None
if last_target is None
else round(last_target - sent_at, 3),
"target_responses_after_exit": len(target_after_exit),
"errors_after_sigterm": len(errors_after),
"error_kinds": sorted({e for _, e in errors_after}),
}
print(json.dumps(summary, indent=2, ensure_ascii=False))
if (
exit_code != 0
or exited_at - sent_at > SHUTDOWN_LIMIT_SEC
or errors_after
or target_after_exit
or not ok_after
):
raise SystemExit("FAIL: завершение по SIGTERM не соответствует требованиям")
print(
"PASS: SIGTERM завершил %s за %.2f с, LB обслужил %d запросов без ошибок"
% (container, exited_at - sent_at, len(ok_after))
)
if __name__ == "__main__":
main()
+24
View File
@@ -0,0 +1,24 @@
#!/bin/sh
set -eu
# Остановка ОДНОЙ API-реплики, обслужившей тест: TARGET_INSTANCE_ID → контейнер.
# Каждый процесс API пишет свой alias (X-Instance-Id) в /tmp/instance-id.
# docker stop: SIGTERM → graceful shutdown; restart-политика unless-stopped
# не поднимает вручную остановленный контейнер, он остаётся остановленным.
# БД, LB и volumes не затрагиваются. Восстановление — scripts/restore.sh.
: "${TARGET_INSTANCE_ID:?test provides target API instance alias}"
cd "$(dirname "$0")/.."
ENV_FILE="${ENV_FILE:-.env}"
# Меньше 10 с лимита hook; больше drain + graceful shutdown (1 + 5 с)
STOP_TIMEOUT_SEC=8
for container in $(docker compose --env-file "$ENV_FILE" ps -q api); do
alias=$(docker exec "$container" cat /tmp/instance-id 2>/dev/null || true)
if [ "$alias" = "$TARGET_INSTANCE_ID" ]; then
name=$(docker inspect -f '{{.Name}}' "$container")
docker stop --timeout "$STOP_TIMEOUT_SEC" "$container" >/dev/null
echo "stopped ${name#/} (instance $alias)" >&2
exit 0
fi
done
echo "no running API replica with instance $TARGET_INSTANCE_ID" >&2
exit 1
+4
View File
@@ -0,0 +1,4 @@
from .api import app as api_endpoints
from .api import health_router as health_endpoints
__all__ = ["api_endpoints", "health_endpoints"]
+10
View File
@@ -0,0 +1,10 @@
from fastapi import APIRouter
from .health import health_router
from .routers import app as v1_endpoints
app = APIRouter(prefix="/api")
app.include_router(v1_endpoints)
__all__ = ["app", "health_router"]
@@ -0,0 +1,4 @@
from .auth import get_current_user
from .security import basic_credentials, bearer_token
__all__ = ["get_current_user", "basic_credentials", "bearer_token"]
+15
View File
@@ -0,0 +1,15 @@
from fastapi import Depends
from domain import UserSchema
from infrastructure.dependencies import ServiceCreator
from infrastructure.services import AuthService
from .security import bearer_token
async def get_current_user(
token: str | None = Depends(bearer_token),
service: AuthService = Depends(ServiceCreator.get_auth_service),
) -> UserSchema:
"""Владелец действующей Bearer-сессии; X-User-Id не учитывается; иначе 401"""
return await service.authenticate(token)
@@ -0,0 +1,52 @@
import base64
import binascii
from fastapi import Depends
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from fastapi.security.http import HTTPBase
from domain import BasicCredentialsSchema
from domain.errors import InvalidCredentialsError
BASIC_SCHEME = "basic"
basic_scheme = HTTPBase(
scheme=BASIC_SCHEME,
scheme_name="basicAuth",
description="Только для POST /auth/sessions",
auto_error=False,
)
bearer_scheme = HTTPBearer(
scheme_name="bearerAuth",
bearerFormat="opaque session token",
auto_error=False,
)
async def basic_credentials(
authorization: HTTPAuthorizationCredentials | None = Depends(basic_scheme),
) -> BasicCredentialsSchema:
"""
Authorization: Basic base64(username:password)
username — до первого двоеточия, остальное — пароль
Любая ошибка формата — тот же 401, что и неверный пароль.
"""
if authorization is None or authorization.scheme.lower() != BASIC_SCHEME:
raise InvalidCredentialsError()
try:
decoded = base64.b64decode(authorization.credentials, validate=True)
username, separator, password = decoded.decode("utf-8").partition(":")
except (binascii.Error, UnicodeDecodeError):
raise InvalidCredentialsError() from None
if not separator or not username.isascii():
raise InvalidCredentialsError()
return BasicCredentialsSchema(username=username, password=password)
async def bearer_token(
authorization: HTTPAuthorizationCredentials | None = Depends(bearer_scheme),
) -> str | None:
"""Токен из Authorization: Bearer; другие схемы (Basic) — как отсутствие"""
return authorization.credentials if authorization is not None else None
+3
View File
@@ -0,0 +1,3 @@
from .exceptions import register_exception_handlers
__all__ = ["register_exception_handlers"]
@@ -0,0 +1,61 @@
from http import HTTPStatus
from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from starlette.exceptions import HTTPException as StarletteHTTPException
from application.api.utils import error_response, get_request_id
from domain.errors import AppError, ErrorCode, InvalidRequestError, NotFoundError
async def app_error_handler(request: Request, error: Exception) -> JSONResponse:
"""Доменные ошибки сервисов и репозиториев"""
assert isinstance(error, AppError)
return error_response(
get_request_id(request),
error.status_code,
error.code,
error.message,
error.headers,
)
async def validation_error_handler(request: Request, error: Exception) -> JSONResponse:
"""Malformed JSON, лишние поля, неверные типы, uuid и limit — 400"""
assert isinstance(error, RequestValidationError)
return error_response(
get_request_id(request),
InvalidRequestError.status_code,
InvalidRequestError.code,
_describe_validation_error(error),
)
async def http_error_handler(request: Request, error: Exception) -> JSONResponse:
"""Ошибки маршрутизации неизвестный путь, неподдерживаемый метод"""
assert isinstance(error, StarletteHTTPException)
if error.status_code == HTTPStatus.NOT_FOUND:
code, message = NotFoundError.code, NotFoundError.default_message
elif error.status_code >= HTTPStatus.INTERNAL_SERVER_ERROR:
code, message = AppError.code, AppError.default_message
else:
code, message = ErrorCode.INVALID_REQUEST, HTTPStatus(error.status_code).phrase
return error_response(
get_request_id(request), error.status_code, code, message, error.headers
)
def _describe_validation_error(error: RequestValidationError) -> str:
"""Безопасное описание: только тип и путь поля, без присланного значения"""
first = error.errors()[0]
if first.get("type") == "json_invalid":
return "Некорректный JSON в теле запроса"
location = ".".join(str(part) for part in first.get("loc", ()))
return f"Некорректное значение: {location}"
def register_exception_handlers(app: FastAPI) -> None:
app.add_exception_handler(AppError, app_error_handler)
app.add_exception_handler(RequestValidationError, validation_error_handler)
app.add_exception_handler(StarletteHTTPException, http_error_handler)
+3
View File
@@ -0,0 +1,3 @@
from .endpoints import health_router
__all__ = ["health_router"]
+36
View File
@@ -0,0 +1,36 @@
from fastapi import APIRouter, Depends, Request, status
from domain import HealthSchema
from domain.errors import UnavailableError
from infrastructure.dependencies import DatabasePoolManager, get_database
health_router = APIRouter(prefix="/health", tags=["Health"])
@health_router.get(
"/live",
status_code=status.HTTP_200_OK,
response_model=HealthSchema,
)
async def liveness() -> HealthSchema:
"""Процесс жив и обрабатывает хттп"""
return HealthSchema()
@health_router.get(
"/ready",
status_code=status.HTTP_200_OK,
response_model=HealthSchema,
)
async def readiness(
request: Request, database: DatabasePoolManager = Depends(get_database)
) -> HealthSchema:
"""
Приложение не завершается и PostgreSQL отвечает. Valkey не критичен:
без него сессии читаются из PostgreSQL, поэтому он не влияет на готовность.
"""
if not getattr(request.app.state, "is_ready", False):
raise UnavailableError()
if not await database.check():
raise UnavailableError("бд недоступна")
return HealthSchema()
@@ -0,0 +1,3 @@
from .request_context import RequestContextMiddleware
__all__ = ["RequestContextMiddleware"]
@@ -0,0 +1,71 @@
import re
import time
import uuid
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from application.api.utils import INSTANCE_ID_HEADER, REQUEST_ID_HEADER, error_response
from config import INSTANCE_ID, LOGGER
from domain.errors import AppError
HEALTH_PATH_PREFIX = "/health/"
INCOMING_REQUEST_ID = re.compile(r"^[0-9a-f]{32}$")
class RequestContextMiddleware(BaseHTTPMiddleware):
"""
X-Request-Id и X-Instance-Id в каждом ответе, журнал запросов и защита
от необработанных ошибок.
X-Request-Id берётся у балансировщика (nginx выставляет $request_id),
поэтому одна строка access-лога LB и журнал реплики связаны одним ID.
В лог попадают метод, путь, статус и длительность — без тел запросов,
query-параметров и текста сообщений.
"""
async def dispatch(
self, request: Request, call_next: RequestResponseEndpoint
) -> Response:
request_id = self._request_id(request)
request.state.request_id = request_id
started = time.perf_counter()
try:
response = await call_next(request)
except Exception:
LOGGER.exception(f"Необработанная ошибка | request_id={request_id}")
error = AppError()
response = error_response(
request_id, error.status_code, error.code, error.message
)
response.headers[REQUEST_ID_HEADER] = request_id
response.headers[INSTANCE_ID_HEADER] = INSTANCE_ID
if getattr(request.app.state, "is_draining", False):
response.headers["Connection"] = "close"
self._log_request(request, response, request_id, started)
return response
@staticmethod
def _request_id(request: Request) -> str:
incoming = request.headers.get(REQUEST_ID_HEADER, "")
if INCOMING_REQUEST_ID.fullmatch(incoming):
return incoming
return uuid.uuid4().hex
@staticmethod
def _log_request(
request: Request, response: Response, request_id: str, started: float
) -> None:
duration_ms = (time.perf_counter() - started) * 1000
if response.status_code >= 500:
level = "ERROR"
elif request.url.path.startswith(HEALTH_PATH_PREFIX):
level = "DEBUG"
else:
level = "INFO"
LOGGER.log(
level,
f"{request.method} {request.url.path} -> {response.status_code} "
f"| {duration_ms:.1f} ms | request_id={request_id}",
)
+17
View File
@@ -0,0 +1,17 @@
from fastapi import APIRouter
from .auth import auth_router
from .chats import chats_router
from .messages import messages_router
from .users import users_router
from .attachments.endpoints import attachments_router
app = APIRouter(prefix="/v1")
app.include_router(users_router)
app.include_router(auth_router)
app.include_router(chats_router)
app.include_router(messages_router)
app.include_router(attachments_router)
__all__ = ["app"]
@@ -0,0 +1,2 @@
from .endpoints import attachments_router
__all__ = ["attachments_router"]
@@ -0,0 +1,58 @@
import tempfile
import os
import hashlib
from uuid import UUID
from fastapi import APIRouter, Depends, Request, Query, status
from fastapi.responses import StreamingResponse
from application.api.dependencies import get_current_user
from domain import UserSchema, AttachmentSchema
from infrastructure.dependencies import ServiceCreator
from infrastructure.services import AttachmentService
from infrastructure.services.attachment_service.attachment_service import PayloadTooLargeError
attachments_router = APIRouter(tags=["Attachments"])
@attachments_router.post("/chats/{chat_id}/attachments", status_code=status.HTTP_201_CREATED, response_model=AttachmentSchema)
async def upload_attachment(
chat_id: UUID,
request: Request,
filename: str = Query(...),
current_user: UserSchema = Depends(get_current_user),
service: AttachmentService = Depends(ServiceCreator.get_attachment_service)
) -> AttachmentSchema:
temp_file = tempfile.NamedTemporaryFile(delete=False)
temp_path = temp_file.name
try:
size = 0
hasher = hashlib.sha256()
async for chunk in request.stream():
size += len(chunk)
if size > 10 * 1024 * 1024:
temp_file.close()
raise PayloadTooLargeError()
hasher.update(chunk)
temp_file.write(chunk)
temp_file.close()
return await service.process_and_save(temp_path, filename, chat_id, current_user.id, size, hasher.hexdigest())
finally:
if os.path.exists(temp_path):
os.remove(temp_path)
@attachments_router.get("/attachments/{attachment_id}", status_code=status.HTTP_200_OK, response_model=AttachmentSchema)
async def get_attachment_meta(
attachment_id: UUID,
current_user: UserSchema = Depends(get_current_user),
service: AttachmentService = Depends(ServiceCreator.get_attachment_service)
) -> AttachmentSchema:
return await service.get_attachment_meta(attachment_id, current_user.id)
@attachments_router.get("/attachments/{attachment_id}/content")
async def get_attachment_content(
attachment_id: UUID,
variant: str = Query("original"),
current_user: UserSchema = Depends(get_current_user),
service: AttachmentService = Depends(ServiceCreator.get_attachment_service)
):
stream_gen, mime = await service.get_attachment_stream(attachment_id, current_user.id)
return StreamingResponse(stream_gen, media_type=mime)
@@ -0,0 +1,3 @@
from .endpoints import auth_router
__all__ = ["auth_router"]
@@ -0,0 +1,35 @@
from fastapi import APIRouter, Depends, Response, status
from application.api.dependencies import basic_credentials, bearer_token
from domain import BasicCredentialsSchema, SessionSchema
from infrastructure.dependencies import ServiceCreator
from infrastructure.services import AuthService
auth_router = APIRouter(prefix="/auth/sessions", tags=["Auth"])
@auth_router.post(
"",
status_code=status.HTTP_201_CREATED,
response_model=SessionSchema,
)
async def create_session(
credentials: BasicCredentialsSchema = Depends(basic_credentials),
service: AuthService = Depends(ServiceCreator.get_auth_service),
) -> SessionSchema:
"""Вход по Basic, без JSON body; возвращает Bearer-токен"""
return await service.login(credentials)
@auth_router.delete(
"/current",
status_code=status.HTTP_204_NO_CONTENT,
response_class=Response,
)
async def revoke_session(
token: str | None = Depends(bearer_token),
service: AuthService = Depends(ServiceCreator.get_auth_service),
) -> Response:
"""Logout: текущий токен перестаёт действовать; повтор — 401"""
await service.logout(token)
return Response(status_code=status.HTTP_204_NO_CONTENT)
@@ -0,0 +1,3 @@
from .endpoints import chats_router
__all__ = ["chats_router"]
@@ -0,0 +1,58 @@
from typing import Annotated
from uuid import UUID
from fastapi import APIRouter, Body, Depends, Query, status
from application.api.dependencies import get_current_user
from domain import (
ChatPageSchema,
ChatSchema,
CreateChatSchema,
UserSchema,
)
from infrastructure.dependencies import ServiceCreator
from infrastructure.services import ChatService
chats_router = APIRouter(prefix="/chats", tags=["Chats"])
@chats_router.post(
"",
status_code=status.HTTP_201_CREATED,
response_model=ChatSchema,
)
async def create_chat(
body: Annotated[CreateChatSchema, Body(...)],
current_user: UserSchema = Depends(get_current_user),
service: ChatService = Depends(ServiceCreator.get_chat_service),
) -> ChatSchema:
"""Вызывающий должен входить в member_ids"""
return await service.create_chat(current_user, body)
@chats_router.get(
"",
status_code=status.HTTP_200_OK,
response_model=ChatPageSchema,
)
async def list_chats(
limit: Annotated[int, Query(ge=1, le=100)] = 50,
cursor: Annotated[str | None, Query(min_length=1)] = None,
current_user: UserSchema = Depends(get_current_user),
service: ChatService = Depends(ServiceCreator.get_chat_service),
) -> ChatPageSchema:
"""Чаты текущего пользователя по (created_at, id) по возрастанию"""
return await service.list_chats(current_user, limit, cursor)
@chats_router.get(
"/{chat_id}",
status_code=status.HTTP_200_OK,
response_model=ChatSchema,
)
async def get_chat(
chat_id: UUID,
current_user: UserSchema = Depends(get_current_user),
service: ChatService = Depends(ServiceCreator.get_chat_service),
) -> ChatSchema:
return await service.get_chat(current_user, chat_id)
@@ -0,0 +1,3 @@
from .endpoints import messages_router
__all__ = ["messages_router"]
@@ -0,0 +1,63 @@
from typing import Annotated
from uuid import UUID
from fastapi import APIRouter, Body, Depends, Header, Query, status
from application.api.dependencies import get_current_user
from domain import (
IDEMPOTENCY_KEY_MAX_LENGTH,
IDEMPOTENCY_KEY_PATTERN,
CreateMessageSchema,
MessagePageSchema,
MessageSchema,
UserSchema,
)
from infrastructure.dependencies import ServiceCreator
from infrastructure.services import MessageService
messages_router = APIRouter(
prefix="/chats/{chat_id}/messages", tags=["Messages"]
)
@messages_router.post(
"",
status_code=status.HTTP_201_CREATED,
response_model=MessageSchema,
)
async def send_message(
chat_id: UUID,
body: Annotated[CreateMessageSchema, Body(...)],
idempotency_key: Annotated[
str,
Header(
alias="Idempotency-Key",
min_length=1,
max_length=IDEMPOTENCY_KEY_MAX_LENGTH,
pattern=IDEMPOTENCY_KEY_PATTERN,
),
],
current_user: UserSchema = Depends(get_current_user),
service: MessageService = Depends(ServiceCreator.get_message_service),
) -> MessageSchema:
"""
Idempotency-Key обязателен: повтор того же тела возвращает тот же 201
и то же сообщение, другое тело с тем же ключом — 409. Нет/неверный ключ — 400.
"""
return await service.send_message(current_user, chat_id, body, idempotency_key)
@messages_router.get(
"",
status_code=status.HTTP_200_OK,
response_model=MessagePageSchema,
)
async def list_messages(
chat_id: UUID,
limit: Annotated[int, Query(ge=1, le=100)] = 50,
cursor: Annotated[str | None, Query(min_length=1)] = None,
current_user: UserSchema = Depends(get_current_user),
service: MessageService = Depends(ServiceCreator.get_message_service),
) -> MessagePageSchema:
"""История чата: keyset cursor по (created_at, id) по возрастанию"""
return await service.list_messages(current_user, chat_id, limit, cursor)
@@ -0,0 +1,3 @@
from .endpoints import users_router
__all__ = ["users_router"]
@@ -0,0 +1,33 @@
from typing import Annotated
from fastapi import APIRouter, Body, Depends, status
from application.api.dependencies import get_current_user
from domain import CreateUserSchema, UserSchema
from infrastructure.dependencies import ServiceCreator
from infrastructure.services import UserService
users_router = APIRouter(prefix="/users", tags=["Users"])
@users_router.post(
"",
status_code=status.HTTP_201_CREATED,
response_model=UserSchema,
openapi_extra={"security": []},
)
async def register_user(
body: Annotated[CreateUserSchema, Body(...)],
service: UserService = Depends(ServiceCreator.get_user_service),
) -> UserSchema:
"""Публичная регистрация; username уникален"""
return await service.register(body)
@users_router.get(
"/me",
status_code=status.HTTP_200_OK,
response_model=UserSchema,
)
async def get_me(current_user: UserSchema = Depends(get_current_user)) -> UserSchema:
return current_user
+13
View File
@@ -0,0 +1,13 @@
from .errors import (
INSTANCE_ID_HEADER,
REQUEST_ID_HEADER,
error_response,
get_request_id,
)
__all__ = [
"REQUEST_ID_HEADER",
"INSTANCE_ID_HEADER",
"error_response",
"get_request_id",
]
+33
View File
@@ -0,0 +1,33 @@
from collections.abc import Mapping
from fastapi import Request
from fastapi.responses import JSONResponse
from domain.errors import ErrorBodySchema, ErrorCode, ErrorSchema
REQUEST_ID_HEADER = "X-Request-Id"
INSTANCE_ID_HEADER = "X-Instance-Id"
def get_request_id(request: Request) -> str:
"""ID запроса, назначенный RequestContextMiddleware"""
request_id: str = request.state.request_id
return request_id
def error_response(
request_id: str,
status_code: int,
code: ErrorCode,
message: str,
headers: Mapping[str, str] | None = None,
) -> JSONResponse:
"""Ответ в едином формате ошибки контракта"""
body = ErrorSchema(
error=ErrorBodySchema(code=code, message=message, request_id=request_id)
)
return JSONResponse(
status_code=status_code,
content=body.model_dump(mode="json"),
headers={**(headers or {}), REQUEST_ID_HEADER: request_id},
)
+77
View File
@@ -0,0 +1,77 @@
from .log_config import LOGGER
from .settings import (
APP_ENV,
APP_HOST,
APP_KEEPALIVE_TIMEOUT_SEC,
APP_PORT,
APP_REVISION,
DB_COMMAND_TIMEOUT_SEC,
DB_CONNECT_TIMEOUT_SEC,
DB_HOST,
DB_NAME,
DB_PASS,
DB_POOL_MAX_SIZE,
DB_POOL_MIN_SIZE,
DB_PORT,
DB_STARTUP_TIMEOUT_SEC,
DB_USER,
IDEMPOTENCY_TTL_SECONDS,
INSTANCE_ID,
INSTANCE_ID_FILE,
LOG_LEVEL,
PASSWORD_HASH_WORKERS,
SESSION_CACHE_TTL_SECONDS,
SESSION_TTL_SECONDS,
SHUTDOWN_DRAIN_SEC,
SHUTDOWN_TIMEOUT_SEC,
VALKEY_HOST,
VALKEY_PASS,
VALKEY_POOL_MAX_SIZE,
VALKEY_PORT,
VALKEY_RETRY_AFTER_SEC,
VALKEY_TIMEOUT_SEC,
S3_ENDPOINT,
S3_REGION,
S3_BUCKET,
S3_ACCESS_KEY_ID,
S3_SECRET_ACCESS_KEY,
)
__all__ = [
"LOGGER",
"APP_HOST",
"APP_PORT",
"APP_ENV",
"APP_REVISION",
"APP_KEEPALIVE_TIMEOUT_SEC",
"LOG_LEVEL",
"SHUTDOWN_TIMEOUT_SEC",
"SHUTDOWN_DRAIN_SEC",
"DB_USER",
"DB_PASS",
"DB_NAME",
"DB_HOST",
"DB_PORT",
"DB_POOL_MIN_SIZE",
"DB_POOL_MAX_SIZE",
"DB_CONNECT_TIMEOUT_SEC",
"DB_COMMAND_TIMEOUT_SEC",
"DB_STARTUP_TIMEOUT_SEC",
"VALKEY_HOST",
"VALKEY_PORT",
"VALKEY_PASS",
"VALKEY_POOL_MAX_SIZE",
"VALKEY_TIMEOUT_SEC",
"VALKEY_RETRY_AFTER_SEC",
"SESSION_TTL_SECONDS",
"SESSION_CACHE_TTL_SECONDS",
"PASSWORD_HASH_WORKERS",
"IDEMPOTENCY_TTL_SECONDS",
"INSTANCE_ID",
"INSTANCE_ID_FILE",
"S3_ENDPOINT",
"S3_REGION",
"S3_BUCKET",
"S3_ACCESS_KEY_ID",
"S3_SECRET_ACCESS_KEY",
]
+12
View File
@@ -0,0 +1,12 @@
"""Сборщик логов"""
import sys
from loguru import logger
from .settings import LOG_LEVEL
LOGGER = logger
LOGGER.remove()
LOGGER.add(sys.stdout, level=LOG_LEVEL, diagnose=False)
+55
View File
@@ -0,0 +1,55 @@
import os
import secrets
import tempfile
from pathlib import Path
from envparse import Env
env = Env()
ENV_FILE = Path(os.getenv("ENV_FILE", ".env"))
if ENV_FILE.is_file():
env.read_envfile(path=str(ENV_FILE))
APP_HOST = env.str("APP_HOST")
APP_PORT = env.int("APP_PORT")
APP_ENV = env.str("APP_ENV")
APP_REVISION = env.str("APP_REVISION", default="dev")
# Больше keepalive_timeout балансировщика: idle-соединение закрывает LB, а не API
APP_KEEPALIVE_TIMEOUT_SEC = env.int("APP_KEEPALIVE_TIMEOUT_SEC")
LOG_LEVEL = env.str("LOG_LEVEL")
SHUTDOWN_TIMEOUT_SEC = env.int("SHUTDOWN_TIMEOUT_SEC")
SHUTDOWN_DRAIN_SEC = env.float("SHUTDOWN_DRAIN_SEC")
DB_USER = env.str("DB_USER")
DB_PASS = env.str("DB_PASS")
DB_NAME = env.str("DB_NAME")
DB_HOST = env.str("DB_HOST")
DB_PORT = env.int("DB_PORT")
DB_POOL_MIN_SIZE = env.int("DB_POOL_MIN_SIZE")
DB_POOL_MAX_SIZE = env.int("DB_POOL_MAX_SIZE")
DB_CONNECT_TIMEOUT_SEC = env.float("DB_CONNECT_TIMEOUT_SEC")
DB_COMMAND_TIMEOUT_SEC = env.float("DB_COMMAND_TIMEOUT_SEC")
DB_STARTUP_TIMEOUT_SEC = env.float("DB_STARTUP_TIMEOUT_SEC")
VALKEY_HOST = env.str("VALKEY_HOST")
VALKEY_PORT = env.int("VALKEY_PORT")
VALKEY_PASS = env.str("VALKEY_PASS")
VALKEY_POOL_MAX_SIZE = env.int("VALKEY_POOL_MAX_SIZE")
VALKEY_TIMEOUT_SEC = env.float("VALKEY_TIMEOUT_SEC")
VALKEY_RETRY_AFTER_SEC = env.float("VALKEY_RETRY_AFTER_SEC")
SESSION_TTL_SECONDS = env.int("SESSION_TTL_SECONDS")
SESSION_CACHE_TTL_SECONDS = env.int("SESSION_CACHE_TTL_SECONDS")
PASSWORD_HASH_WORKERS = env.int("PASSWORD_HASH_WORKERS")
IDEMPOTENCY_TTL_SECONDS = env.int("IDEMPOTENCY_TTL_SECONDS")
INSTANCE_ID = secrets.token_hex(6)
INSTANCE_ID_FILE = Path(tempfile.gettempdir()) / "instance-id"
S3_ENDPOINT = env.str("S3_ENDPOINT", default="http://s3:9000")
S3_REGION = env.str("S3_REGION", default="us-east-1")
S3_BUCKET = env.str("S3_BUCKET", default="messenger-attachments")
S3_ACCESS_KEY_ID = env.str("S3_ACCESS_KEY_ID", default="app_user")
S3_SECRET_ACCESS_KEY = env.str("S3_SECRET_ACCESS_KEY", default="app_password")
+37
View File
@@ -0,0 +1,37 @@
from .auth import ActiveSessionSchema, BasicCredentialsSchema, SessionSchema
from .attachments import AttachmentSchema, VariantSchema
from .chats import ChatPageSchema, ChatSchema, CreateChatSchema
from .health import HealthSchema
from .messages import (
IDEMPOTENCY_KEY_MAX_LENGTH,
IDEMPOTENCY_KEY_PATTERN,
CreateMessageSchema,
IdempotencyKeySchema,
MessagePageSchema,
MessageSchema,
)
from .pagination import PageCursorSchema, PageKey
from .users import CreateUserSchema, UserCredentialsSchema, UserSchema
__all__ = [
"CreateUserSchema",
"UserSchema",
"UserCredentialsSchema",
"BasicCredentialsSchema",
"SessionSchema",
"ActiveSessionSchema",
"CreateChatSchema",
"ChatSchema",
"ChatPageSchema",
"CreateMessageSchema",
"IdempotencyKeySchema",
"IDEMPOTENCY_KEY_PATTERN",
"IDEMPOTENCY_KEY_MAX_LENGTH",
"MessageSchema",
"MessagePageSchema",
"HealthSchema",
"PageKey",
"PageCursorSchema",
"AttachmentSchema",
"VariantSchema",
]
+3
View File
@@ -0,0 +1,3 @@
from .schemas import AttachmentSchema, VariantSchema
__all__ = ["AttachmentSchema", "VariantSchema"]
+30
View File
@@ -0,0 +1,30 @@
from typing import Literal
from uuid import UUID
from pydantic import BaseModel, ConfigDict
from domain.utils import UtcDateTime
class VariantSchema(BaseModel):
model_config = ConfigDict(frozen=True)
name: Literal["thumbnail", "crop", "optimized"]
content_type: Literal["image/jpeg", "image/png", "image/webp"]
size_bytes: int
width: int
height: int
sha256: str
class AttachmentSchema(BaseModel):
model_config = ConfigDict(frozen=True)
id: UUID
chat_id: UUID
owner_id: UUID
filename: str
content_type: Literal["image/png", "image/jpeg"]
size_bytes: int
sha256: str
status: Literal["ready", "queued", "failed"]
variants: list[VariantSchema]
error_code: str | None
created_at: UtcDateTime
updated_at: UtcDateTime
+7
View File
@@ -0,0 +1,7 @@
from .schemas import ActiveSessionSchema, BasicCredentialsSchema, SessionSchema
__all__ = [
"BasicCredentialsSchema",
"SessionSchema",
"ActiveSessionSchema",
]
+35
View File
@@ -0,0 +1,35 @@
from typing import Literal
from pydantic import BaseModel, ConfigDict
from domain.users import UserSchema
from domain.utils import UtcDateTime
class BasicCredentialsSchema(BaseModel):
"""Разобранный заголовок Authorization: Basic"""
model_config = ConfigDict(frozen=True)
username: str
password: str
class SessionSchema(BaseModel):
"""Ответ входа: непрозрачный токен показывается клиенту один раз"""
model_config = ConfigDict(frozen=True)
token: str
token_type: Literal["Bearer"] = "Bearer"
expires_at: UtcDateTime
user: UserSchema
class ActiveSessionSchema(BaseModel):
"""Действующая серверная сессия: владелец и срок; так же лежит в кэше"""
model_config = ConfigDict(frozen=True)
user: UserSchema
expires_at: UtcDateTime
+7
View File
@@ -0,0 +1,7 @@
from .schemas import ChatPageSchema, ChatSchema, CreateChatSchema
__all__ = [
"CreateChatSchema",
"ChatSchema",
"ChatPageSchema",
]
+42
View File
@@ -0,0 +1,42 @@
from typing import Annotated
from uuid import UUID
from pydantic import BaseModel, ConfigDict, Field, field_validator
from domain.utils import NotBlankStr, UtcDateTime
ChatTitle = Annotated[NotBlankStr, Field(min_length=1, max_length=100)]
class CreateChatSchema(BaseModel):
"""Запрос создания чата: участники задаются явно, включая вызывающего"""
model_config = ConfigDict(extra="forbid")
title: ChatTitle
member_ids: list[UUID] = Field(min_length=1, max_length=100)
@field_validator("member_ids")
@classmethod
def validate_unique_members(cls, value: list[UUID]) -> list[UUID]:
if len(set(value)) != len(value):
raise ValueError("member_ids не должен содержать дубликаты")
return value
class ChatSchema(BaseModel):
"""Чат"""
model_config = ConfigDict(frozen=True)
id: UUID
title: str
member_ids: list[UUID]
created_at: UtcDateTime
class ChatPageSchema(BaseModel):
"""Страница чатов текущего пользователя"""
items: list[ChatSchema]
next_cursor: str | None
+24
View File
@@ -0,0 +1,24 @@
from .codes import ErrorCode
from .exceptions import (
AppError,
ConflictError,
InvalidCredentialsError,
InvalidRequestError,
NotFoundError,
UnauthorizedError,
UnavailableError,
)
from .schemas import ErrorBodySchema, ErrorSchema
__all__ = [
"ErrorCode",
"AppError",
"InvalidRequestError",
"UnauthorizedError",
"InvalidCredentialsError",
"NotFoundError",
"ConflictError",
"UnavailableError",
"ErrorBodySchema",
"ErrorSchema",
]
+15
View File
@@ -0,0 +1,15 @@
from enum import StrEnum
class ErrorCode(StrEnum):
"""коды ошибок"""
INVALID_REQUEST = "invalid_request"
UNAUTHORIZED = "unauthorized"
NOT_FOUND = "not_found"
CONFLICT = "conflict"
PAYLOAD_TOO_LARGE = "payload_too_large"
UNSUPPORTED_MEDIA_TYPE = "unsupported_media_type"
RATE_LIMITED = "rate_limited"
UNAVAILABLE = "unavailable"
INTERNAL_ERROR = "internal_error"
+64
View File
@@ -0,0 +1,64 @@
from http import HTTPStatus
from .codes import ErrorCode
class AppError(Exception):
"""Базовая ошибка приложения: статус, машинный код и безопасное сообщение"""
status_code: HTTPStatus = HTTPStatus.INTERNAL_SERVER_ERROR
code: ErrorCode = ErrorCode.INTERNAL_ERROR
default_message: str = "Внутренняя ошибка сервера"
headers: dict[str, str] = {}
def __init__(self, message: str | None = None) -> None:
self.message = message or self.default_message
super().__init__(self.message)
class InvalidRequestError(AppError):
"""Неверные поля, UUID, cursor или limit"""
status_code = HTTPStatus.BAD_REQUEST
code = ErrorCode.INVALID_REQUEST
default_message = "Некорректный запрос"
class UnauthorizedError(AppError):
"""Нет действительной Bearer-сессии: токена нет, он истёк, отозван или неизвестен"""
status_code = HTTPStatus.UNAUTHORIZED
code = ErrorCode.UNAUTHORIZED
default_message = "Требуется действительная сессия: Authorization: Bearer <token>"
headers = {"WWW-Authenticate": 'Bearer realm="messenger"'}
class InvalidCredentialsError(UnauthorizedError):
"""Неверные Basic-credentials: одинаковый ответ для неизвестного логина и пароля"""
default_message = "Неверное имя пользователя или пароль"
headers = {"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'}
class NotFoundError(AppError):
"""Объекта нет или он чужой: существование чужого объекта не раскрывается"""
status_code = HTTPStatus.NOT_FOUND
code = ErrorCode.NOT_FOUND
default_message = "Объект не найден"
class ConflictError(AppError):
"""Конфликт состояния, например занятый username"""
status_code = HTTPStatus.CONFLICT
code = ErrorCode.CONFLICT
default_message = "Конфликт состояния"
class UnavailableError(AppError):
"""Приложение не готово обслуживать запросы"""
status_code = HTTPStatus.SERVICE_UNAVAILABLE
code = ErrorCode.UNAVAILABLE
default_message = "Сервис временно недоступен"
+21
View File
@@ -0,0 +1,21 @@
from pydantic import BaseModel, ConfigDict
from .codes import ErrorCode
class ErrorBodySchema(BaseModel):
"""Тело ошибки"""
model_config = ConfigDict(extra="forbid")
code: ErrorCode
message: str
request_id: str
class ErrorSchema(BaseModel):
"""Единый формат ошибки: {"error": {...}}"""
model_config = ConfigDict(extra="forbid")
error: ErrorBodySchema
+3
View File
@@ -0,0 +1,3 @@
from .schemas import HealthSchema
__all__ = ["HealthSchema"]
+9
View File
@@ -0,0 +1,9 @@
from typing import Literal
from pydantic import BaseModel
class HealthSchema(BaseModel):
"""Ответ health-проверок"""
status: Literal["ok"] = "ok"
+17
View File
@@ -0,0 +1,17 @@
from .schemas import (
IDEMPOTENCY_KEY_MAX_LENGTH,
IDEMPOTENCY_KEY_PATTERN,
CreateMessageSchema,
IdempotencyKeySchema,
MessagePageSchema,
MessageSchema,
)
__all__ = [
"IDEMPOTENCY_KEY_PATTERN",
"IDEMPOTENCY_KEY_MAX_LENGTH",
"CreateMessageSchema",
"IdempotencyKeySchema",
"MessageSchema",
"MessagePageSchema",
]
+56
View File
@@ -0,0 +1,56 @@
from typing import Annotated
from uuid import UUID
from pydantic import BaseModel, ConfigDict, Field
from domain.utils import NotBlankStr, UtcDateTime
MessageText = Annotated[NotBlankStr, Field(min_length=1, max_length=4000)]
IDEMPOTENCY_KEY_PATTERN = r"^[A-Za-z0-9._:-]{1,128}$"
IDEMPOTENCY_KEY_MAX_LENGTH = 128
from pydantic import BaseModel, ConfigDict, Field, model_validator
class CreateMessageSchema(BaseModel):
model_config = ConfigDict(extra="forbid")
text: str = Field(default="", max_length=4000)
attachment_ids: list[UUID] = Field(default_factory=list, max_length=10)
@model_validator(mode="after")
def check_content(self):
if not self.text.strip() and not self.attachment_ids:
raise ValueError("Нельзя отправить пустое сообщение без вложений")
return self
class MessageSchema(BaseModel):
model_config = ConfigDict(frozen=True)
id: UUID
chat_id: UUID
sender_id: UUID
text: str
attachment_ids: list[UUID]
created_at: UtcDateTime
class MessagePageSchema(BaseModel):
"""Страница истории чата"""
items: list[MessageSchema]
next_cursor: str | None
class IdempotencyKeySchema(BaseModel):
"""
Ключ повторной отправки сообщения.
request_hash — SHA-256 канонического тела: повтор с тем же ключом и тем же
телом возвращает исходное сообщение, с другим телом — конфликт.
"""
model_config = ConfigDict(frozen=True)
key: str
request_hash: bytes
ttl_seconds: int
+6
View File
@@ -0,0 +1,6 @@
from .schemas import PageCursorSchema, PageKey
__all__ = [
"PageKey",
"PageCursorSchema",
]
+26
View File
@@ -0,0 +1,26 @@
from datetime import datetime
from typing import NamedTuple
from uuid import UUID
from pydantic import BaseModel, ConfigDict
class PageKey(NamedTuple):
"""Ключ keyset-пагинации: порядок (created_at, id) по возрастанию"""
created_at: datetime
id: UUID
class PageCursorSchema(BaseModel):
"""Содержимое непрозрачного cursor: ключ последнего элемента и его ресурс"""
model_config = ConfigDict(extra="forbid", frozen=True)
scope: str
created_at: datetime
id: UUID
@property
def key(self) -> PageKey:
return PageKey(self.created_at, self.id)
+7
View File
@@ -0,0 +1,7 @@
from .schemas import CreateUserSchema, UserCredentialsSchema, UserSchema
__all__ = [
"CreateUserSchema",
"UserSchema",
"UserCredentialsSchema",
]
+40
View File
@@ -0,0 +1,40 @@
from typing import Annotated
from uuid import UUID
from pydantic import BaseModel, ConfigDict, Field, StrictStr
from domain.utils import NotBlankStr, UtcDateTime
Username = Annotated[StrictStr, Field(pattern=r"^[a-z0-9_]{3,32}$")]
DisplayName = Annotated[NotBlankStr, Field(min_length=1, max_length=100)]
Password = Annotated[StrictStr, Field(min_length=12, max_length=128)]
class CreateUserSchema(BaseModel):
"""Запрос регистрации пользователя; длина пароля — в Unicode code points"""
model_config = ConfigDict(extra="forbid")
username: Username
display_name: DisplayName
password: Password = Field(json_schema_extra={"writeOnly": True})
class UserSchema(BaseModel):
"""Пользователь: пароль и его hash сюда никогда не попадают"""
model_config = ConfigDict(frozen=True)
id: UUID
username: str
display_name: str
created_at: UtcDateTime
class UserCredentialsSchema(BaseModel):
"""Пользователь вместе с hash пароля — только для проверки входа"""
model_config = ConfigDict(frozen=True)
user: UserSchema
password_hash: str
+15
View File
@@ -0,0 +1,15 @@
from .types import (
NotBlankStr,
UtcDateTime,
ensure_not_blank,
ensure_storable,
to_rfc3339_utc,
)
__all__ = [
"NotBlankStr",
"UtcDateTime",
"ensure_not_blank",
"ensure_storable",
"to_rfc3339_utc",
]
+36
View File
@@ -0,0 +1,36 @@
from datetime import UTC, datetime
from typing import Annotated
from pydantic import AfterValidator, PlainSerializer, StrictStr
NUL_CHARACTER = "\x00"
def ensure_not_blank(value: str) -> str:
"""Обязательная строка не может состоять только из пробельных символов"""
if not value.strip():
raise ValueError("Строка не может быть пустой или пробельной")
return value
def ensure_storable(value: str) -> str:
"""Тип text в потсгре"""
if NUL_CHARACTER in value:
raise ValueError("Строка не может содержать символ NUL")
return value
def to_rfc3339_utc(value: datetime) -> str:
"""RFC 3339 в UTC с суффиксом Z и микросекундами"""
return (
value.astimezone(UTC).isoformat(timespec="microseconds").replace("+00:00", "Z")
)
NotBlankStr = Annotated[
StrictStr, AfterValidator(ensure_not_blank), AfterValidator(ensure_storable)
]
UtcDateTime = Annotated[
datetime, PlainSerializer(to_rfc3339_utc, return_type=str, when_used="json")
]
+126
View File
@@ -0,0 +1,126 @@
import asyncio
import socket
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
import uvicorn
from fastapi import FastAPI
from application import api_endpoints, health_endpoints
from application.api.handlers import register_exception_handlers
from application.api.middlewares import RequestContextMiddleware
from config import (
APP_ENV,
APP_HOST,
APP_KEEPALIVE_TIMEOUT_SEC,
APP_PORT,
APP_REVISION,
DB_STARTUP_TIMEOUT_SEC,
INSTANCE_ID,
INSTANCE_ID_FILE,
LOGGER,
PASSWORD_HASH_WORKERS,
SESSION_TTL_SECONDS,
SHUTDOWN_DRAIN_SEC,
SHUTDOWN_TIMEOUT_SEC,
)
from infrastructure import PasswordHasher, cache_manager, database_pool_manager
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
"""
uvicorn открывает порт только после startup: реплика без БД или со старой
схемой не принимает соединений, и LB отправляет запросы другим репликам.
"""
await database_pool_manager.create_pool(DB_STARTUP_TIMEOUT_SEC)
await database_pool_manager.check_migrations()
cache_manager.create_client()
password_hasher = PasswordHasher(PASSWORD_HASH_WORKERS)
app.state.database = database_pool_manager
app.state.cache = cache_manager
app.state.password_hasher = password_hasher
INSTANCE_ID_FILE.write_text(INSTANCE_ID)
app.state.is_ready = True
LOGGER.info(
f"Application started, env={APP_ENV}, revision={APP_REVISION}, "
f"instance={INSTANCE_ID}, session ttl={SESSION_TTL_SECONDS} s"
)
yield
app.state.is_ready = False
await cache_manager.close()
await database_pool_manager.close()
password_hasher.close()
LOGGER.info(f"Application stopped, instance={INSTANCE_ID}, connections closed")
app = FastAPI(
title="mtuci-labs-messanger",
description="мессенджер-конкурент-максу",
version="1.3.0",
lifespan=lifespan,
)
app.add_middleware(RequestContextMiddleware)
register_exception_handlers(app)
app.include_router(health_endpoints)
app.include_router(api_endpoints)
class DrainingServer(uvicorn.Server):
"""
SIGTERM: сначала drain, затем штатный graceful shutdown uvicorn.
"""
def __init__(self, config: uvicorn.Config, drain_seconds: float) -> None:
super().__init__(config)
self._drain_seconds = drain_seconds
async def shutdown(self, sockets: list[socket.socket] | None = None) -> None:
for server in getattr(self, "servers", []):
server.close()
app.state.is_ready = False
app.state.is_draining = True
LOGGER.info(
f"SIGTERM: drain {self._drain_seconds} с, instance={INSTANCE_ID}: "
"порт закрыт, keep-alive соединения закрываются после ответа"
)
await asyncio.sleep(self._drain_seconds)
await super().shutdown(sockets)
def main(
host: str,
port: int,
shutdown_timeout: int,
keepalive_timeout: int,
drain_seconds: float,
) -> None:
"""
Функция для запуска api.
SIGTERM: drain_seconds без приёма новых соединений, затем uvicorn ждёт
текущих запросов не дольше shutdown_timeout, и lifespan закрывает пулы.
Сумма меньше stop_grace_period контейнера (10 с).
"""
config = uvicorn.Config(
app,
host=host,
port=port,
timeout_graceful_shutdown=shutdown_timeout,
timeout_keep_alive=keepalive_timeout,
access_log=False,
server_header=False,
)
DrainingServer(config, drain_seconds).run()
if __name__ == "__main__":
main(
host=APP_HOST,
port=APP_PORT,
shutdown_timeout=SHUTDOWN_TIMEOUT_SEC,
keepalive_timeout=APP_KEEPALIVE_TIMEOUT_SEC,
drain_seconds=SHUTDOWN_DRAIN_SEC,
)
+20
View File
@@ -0,0 +1,20 @@
import sys
import urllib.request
from config import APP_PORT
READY_URL = f"http://127.0.0.1:{APP_PORT}/health/ready"
REQUEST_TIMEOUT_SEC = 2
def main() -> int:
try:
with urllib.request.urlopen(READY_URL, timeout=REQUEST_TIMEOUT_SEC) as response:
return 0 if response.status == 200 else 1
except OSError:
return 1
if __name__ == "__main__":
sys.exit(main())
+4
View File
@@ -0,0 +1,4 @@
from .dependencies import cache_manager, database_pool_manager
from .security import PasswordHasher
__all__ = ["database_pool_manager", "cache_manager", "PasswordHasher"]
@@ -0,0 +1,13 @@
from .cache import CacheManager, cache_manager, get_cache
from .database import DatabasePoolManager, database_pool_manager, get_database
from .service_creator import ServiceCreator
__all__ = [
"ServiceCreator",
"DatabasePoolManager",
"database_pool_manager",
"get_database",
"CacheManager",
"cache_manager",
"get_cache",
]
+94
View File
@@ -0,0 +1,94 @@
import time
from redis.asyncio import BlockingConnectionPool, Redis
from redis.asyncio.retry import Retry
from redis.backoff import NoBackoff
from starlette.requests import HTTPConnection
from config import (
LOGGER,
VALKEY_HOST,
VALKEY_PASS,
VALKEY_POOL_MAX_SIZE,
VALKEY_PORT,
VALKEY_RETRY_AFTER_SEC,
VALKEY_TIMEOUT_SEC,
)
class CacheManager:
"""
Клиент Valkey: ограниченный пул, короткие таймауты, без встроенных повторов.
"""
def __init__(
self,
host: str,
port: int,
password: str,
max_size: int,
timeout: float,
retry_after: float,
) -> None:
self._host = host
self._port = port
self._password = password
self.max_size = max_size
self.timeout = timeout
self.retry_after = retry_after
self.client: Redis | None = None
self._disabled_until = 0.0
def create_client(self) -> Redis:
"""Создание клиента; соединения открываются лениво при первом запросе"""
if self.client is None:
pool = BlockingConnectionPool(
host=self._host,
port=self._port,
password=self._password,
max_connections=self.max_size,
timeout=self.timeout,
socket_timeout=self.timeout,
socket_connect_timeout=self.timeout,
retry=Retry(NoBackoff(), 0),
)
self.client = Redis(connection_pool=pool)
LOGGER.info(
f"Клиент Valkey создан: {self._host}:{self._port}, "
f"pool: {self.max_size}, timeout: {self.timeout} с"
)
return self.client
@property
def is_available(self) -> bool:
return time.monotonic() >= self._disabled_until
def mark_failed(self, error: Exception) -> None:
"""Обход кэша на retry_after секунд после ошибки Valkey"""
if self.is_available:
LOGGER.warning(
f"Valkey недоступен ({type(error).__name__}): кэш сессий "
f"обходится {self.retry_after} с, запросы идут в PostgreSQL"
)
self._disabled_until = time.monotonic() + self.retry_after
async def close(self) -> None:
if self.client is not None:
await self.client.aclose()
self.client = None
def get_cache(connection: HTTPConnection) -> CacheManager:
"""Получение менеджера кэша, созданного в lifespan."""
cache: CacheManager = connection.app.state.cache
return cache
cache_manager = CacheManager(
host=VALKEY_HOST,
port=VALKEY_PORT,
password=VALKEY_PASS,
max_size=VALKEY_POOL_MAX_SIZE,
timeout=VALKEY_TIMEOUT_SEC,
retry_after=VALKEY_RETRY_AFTER_SEC,
)
+236
View File
@@ -0,0 +1,236 @@
import asyncio
import time
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from pathlib import Path
import asyncpg
from starlette.requests import HTTPConnection
from config import (
DB_COMMAND_TIMEOUT_SEC,
DB_CONNECT_TIMEOUT_SEC,
DB_HOST,
DB_NAME,
DB_PASS,
DB_POOL_MAX_SIZE,
DB_POOL_MIN_SIZE,
DB_PORT,
DB_USER,
LOGGER,
)
from infrastructure.utils import DATABASE_UNAVAILABLE_ERRORS
MIGRATIONS_DIR = Path(__file__).resolve().parents[3] / "migrations"
MIGRATIONS_LOCK_NAMESPACE = 1
STARTUP_RETRY_INTERVAL_SEC = 1
BROKEN_CONNECTION_ERRORS: tuple[type[Exception], ...] = (
asyncpg.PostgresConnectionError,
asyncpg.InterfaceError,
asyncpg.exceptions.OperatorInterventionError,
)
MIGRATIONS_TIMEOUT_SEC = 300
class DatabasePoolManager:
"""Класс для управления пулом соединений и версионируемыми миграциями."""
def __init__(
self,
user: str,
password: str,
db: str,
host: str,
port: int,
min_size: int,
max_size: int,
connect_timeout: float,
command_timeout: float,
) -> None:
self._user = user
self._password = password
self._host = host
self._port = port
self._database = db
self.min_size = min_size
self.max_size = max_size
self.connect_timeout = connect_timeout
self.command_timeout = command_timeout
self.pool: asyncpg.Pool | None = None
async def create_pool(self, startup_timeout: float) -> asyncpg.Pool:
"""
Создание ограниченного пула соединений.
"""
LOGGER.info("Установка соединения с базой данных psql")
deadline = time.monotonic() + startup_timeout
while self.pool is None:
try:
self.pool = await asyncpg.create_pool(
user=self._user,
password=self._password,
host=self._host,
port=self._port,
database=self._database,
min_size=self.min_size,
max_size=self.max_size,
timeout=self.connect_timeout,
command_timeout=self.command_timeout,
server_settings={
"application_name": "messenger-api",
"statement_timeout": str(int(self.command_timeout * 1000)),
},
)
except (*DATABASE_UNAVAILABLE_ERRORS, asyncpg.PostgresError) as error:
if time.monotonic() >= deadline:
LOGGER.error(
"Не удалось подключиться к psql за "
f"{startup_timeout} с: {type(error).__name__}"
)
raise
LOGGER.warning(
f"PostgreSQL ещё недоступен: {type(error).__name__}, повтор"
)
await asyncio.sleep(STARTUP_RETRY_INTERVAL_SEC)
LOGGER.info(
f"Соединение с базой данных psql установлено. DB_NAME: "
f"{self._database}, pool: {self.min_size}..{self.max_size}"
)
return self.pool
async def apply_migrations(self, directory: Path = MIGRATIONS_DIR) -> None:
"""
Применение миграций NNNN_name.sql по порядку версий.
Вызывается однократно отдельным процессом (src/migrate.py, сервис migrate)
до запуска API-реплик, а не каждой репликой при старте.
Каждая миграция и запись о ней в schema_migrations — одна транзакция.
Advisory lock — страховка от двух одновременных запусков migrate;
уже применённые версии пропускаются, поэтому повторный запуск безопасен.
"""
async with self.acquire() as conn:
await conn.execute("SET statement_timeout = 0;")
await conn.execute(
"SELECT pg_advisory_lock($1, 0);",
MIGRATIONS_LOCK_NAMESPACE,
timeout=MIGRATIONS_TIMEOUT_SEC,
)
try:
await conn.execute(
"""
CREATE TABLE IF NOT EXISTS schema_migrations (
version integer PRIMARY KEY,
name text NOT NULL,
applied_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
"""
)
applied = {
record["version"]
for record in await conn.fetch(
"SELECT version FROM schema_migrations;"
)
}
for version, path in self._migration_files(directory):
if version in applied:
continue
async with conn.transaction():
await conn.execute(
path.read_text(encoding="utf-8"),
timeout=MIGRATIONS_TIMEOUT_SEC,
)
await conn.execute(
"INSERT INTO schema_migrations (version, name) "
"VALUES ($1, $2);",
version,
path.stem,
)
LOGGER.info(f"Применена миграция {path.name}")
finally:
await conn.execute(
"SELECT pg_advisory_unlock($1, 0);", MIGRATIONS_LOCK_NAMESPACE
)
LOGGER.info("Схема БД в актуальной версии")
async def check_migrations(self, directory: Path = MIGRATIONS_DIR) -> None:
"""
Схема БД не старше кода: все миграции из образа уже применены.
API миграции не применяет. Если migrate ещё не отработал, реплика не
стартует и не начинает слушать порт, балансировщик её не использует
"""
expected = {version for version, _ in self._migration_files(directory)}
async with self.acquire() as conn:
try:
records = await conn.fetch("SELECT version FROM schema_migrations;")
except asyncpg.UndefinedTableError:
records = []
applied = {record["version"] for record in records}
missing = sorted(expected - applied)
if missing:
raise RuntimeError(
f"В БД не применены миграции {missing}: сначала запустите migrate"
)
LOGGER.info(f"Схема БД совместима: версия {max(applied, default=0)}")
@staticmethod
def _migration_files(directory: Path) -> list[tuple[int, Path]]:
"""Файлы NNNN_name.sql по возрастанию версии"""
return sorted(
(int(path.name.split("_", 1)[0]), path) for path in directory.glob("*.sql")
)
@asynccontextmanager
async def acquire(self) -> AsyncIterator[asyncpg.Connection]:
"""
Получение соединения из пула с ограниченным ожиданием свободного.
После рестарта PostgreSQL все соединения пула мертвы. На первой же
ошибке соединения пул помечает их на переподключение, чтобы 503
получил один запрос, а не каждый из max_size следующих.
"""
assert self.pool is not None
try:
async with self.pool.acquire(timeout=self.connect_timeout) as connection:
yield connection
except BROKEN_CONNECTION_ERRORS:
await self.pool.expire_connections()
raise
async def check(self) -> bool:
"""PostgreSQL отвечает на запрос — для readiness"""
try:
async with self.acquire() as conn:
await conn.fetchval("SELECT 1;", timeout=self.connect_timeout)
except DATABASE_UNAVAILABLE_ERRORS:
return False
return True
async def close(self) -> None:
if self.pool:
await self.pool.close()
self.pool = None
def get_database(connection: HTTPConnection) -> DatabasePoolManager:
"""Получение менеджера пула соединений, созданного в lifespan."""
database: DatabasePoolManager = connection.app.state.database
return database
database_pool_manager = DatabasePoolManager(
user=DB_USER,
password=DB_PASS,
db=DB_NAME,
host=DB_HOST,
port=DB_PORT,
min_size=DB_POOL_MIN_SIZE,
max_size=DB_POOL_MAX_SIZE,
connect_timeout=DB_CONNECT_TIMEOUT_SEC,
command_timeout=DB_COMMAND_TIMEOUT_SEC,
)
@@ -0,0 +1,118 @@
from fastapi import Depends
from config import S3_ENDPOINT, S3_REGION, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET
from infrastructure.s3_client import S3Client
from infrastructure.repositories import PostgresAttachmentRepository
from interfaces import AttachmentRepository
from infrastructure.services import AttachmentService
# Создаем глобальный клиент S3
s3_client = S3Client(S3_ENDPOINT, S3_REGION, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET)
from config import (
IDEMPOTENCY_TTL_SECONDS,
SESSION_CACHE_TTL_SECONDS,
SESSION_TTL_SECONDS,
)
from infrastructure.dependencies.cache import CacheManager, get_cache
from infrastructure.dependencies.database import DatabasePoolManager, get_database
from infrastructure.repositories import (
PostgresChatRepository,
PostgresMessageRepository,
PostgresSessionRepository,
PostgresUserRepository,
ValkeySessionCache,
)
from infrastructure.security import PasswordHasher, get_password_hasher
from infrastructure.services import (
AuthService,
ChatService,
MessageService,
UserService,
)
from interfaces import (
ChatRepository,
MessageRepository,
SessionCache,
SessionRepository,
UserRepository,
)
class ServiceCreator:
@staticmethod
def _get_user_repository(
database: DatabasePoolManager = Depends(get_database),
) -> UserRepository:
return PostgresUserRepository(database)
@staticmethod
def _get_chat_repository(
database: DatabasePoolManager = Depends(get_database),
) -> ChatRepository:
return PostgresChatRepository(database)
@staticmethod
def _get_attachment_repository(database: DatabasePoolManager = Depends(get_database)) -> AttachmentRepository:
return PostgresAttachmentRepository(database)
@staticmethod
def get_attachment_service(
repo: AttachmentRepository = Depends(_get_attachment_repository),
chat_repo: ChatRepository = Depends(_get_chat_repository)
) -> AttachmentService:
return AttachmentService(s3_client, repo, chat_repo)
@staticmethod
def _get_message_repository(
database: DatabasePoolManager = Depends(get_database),
) -> MessageRepository:
return PostgresMessageRepository(database)
@staticmethod
def _get_session_repository(
database: DatabasePoolManager = Depends(get_database),
) -> SessionRepository:
return PostgresSessionRepository(database)
@staticmethod
def _get_session_cache(
cache: CacheManager = Depends(get_cache),
) -> SessionCache:
return ValkeySessionCache(cache, SESSION_CACHE_TTL_SECONDS)
@staticmethod
def get_user_service(
repository: UserRepository = Depends(_get_user_repository),
hasher: PasswordHasher = Depends(get_password_hasher),
) -> UserService:
return UserService(repository, hasher)
@staticmethod
def get_auth_service(
user_repository: UserRepository = Depends(_get_user_repository),
session_repository: SessionRepository = Depends(_get_session_repository),
session_cache: SessionCache = Depends(_get_session_cache),
hasher: PasswordHasher = Depends(get_password_hasher),
) -> AuthService:
return AuthService(
user_repository,
session_repository,
session_cache,
hasher,
SESSION_TTL_SECONDS,
)
@staticmethod
def get_chat_service(
chat_repository: ChatRepository = Depends(_get_chat_repository),
) -> ChatService:
return ChatService(chat_repository)
@staticmethod
def get_message_service(
chat_repository: ChatRepository = Depends(_get_chat_repository),
message_repository: MessageRepository = Depends(_get_message_repository),
) -> MessageService:
return MessageService(
chat_repository, message_repository, IDEMPOTENCY_TTL_SECONDS
)
@@ -0,0 +1,15 @@
from .chat_repository import PostgresChatRepository
from .message_repository import PostgresMessageRepository
from .session_cache import ValkeySessionCache
from .session_repository import PostgresSessionRepository
from .user_repository import PostgresUserRepository
from .attachment_repository import PostgresAttachmentRepository
__all__ = [
"PostgresUserRepository",
"PostgresChatRepository",
"PostgresMessageRepository",
"PostgresSessionRepository",
"ValkeySessionCache",
"PostgresAttachmentRepository",
]
@@ -0,0 +1,48 @@
import json
from uuid import UUID
import asyncpg
from domain import AttachmentSchema, VariantSchema
from infrastructure.utils import Pool, repository_logs
from interfaces import AttachmentRepository
class PostgresAttachmentRepository(AttachmentRepository):
def __init__(self, pool: Pool) -> None:
self._pool = pool
@repository_logs
async def create(self, attachment: AttachmentSchema) -> AttachmentSchema:
query = """
INSERT INTO attachments
(id, chat_id, owner_id, filename, content_type, size_bytes, sha256, status, error_code, variants)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10::jsonb)
RETURNING created_at, updated_at;
"""
variants_json = json.dumps([v.model_dump(mode="json") for v in attachment.variants])
async with self._pool.acquire() as conn:
row = await conn.fetchrow(
query, attachment.id, attachment.chat_id, attachment.owner_id,
attachment.filename, attachment.content_type, attachment.size_bytes,
attachment.sha256, attachment.status, attachment.error_code, variants_json
)
assert row is not None
return attachment.model_copy(update={"created_at": row["created_at"], "updated_at": row["updated_at"]})
@repository_logs
async def get_by_id(self, attachment_id: UUID) -> AttachmentSchema | None:
query = """
SELECT id, chat_id, owner_id, filename, content_type, size_bytes, sha256, status, error_code, variants, created_at, updated_at
FROM attachments WHERE id = $1;
"""
async with self._pool.acquire() as conn:
row = await conn.fetchrow(query, attachment_id)
if row is None:
return None
variants = [VariantSchema(**v) for v in json.loads(row["variants"])]
return AttachmentSchema(
id=row["id"], chat_id=row["chat_id"], owner_id=row["owner_id"],
filename=row["filename"], content_type=row["content_type"],
size_bytes=row["size_bytes"], sha256=row["sha256"], status=row["status"],
error_code=row["error_code"], variants=variants,
created_at=row["created_at"], updated_at=row["updated_at"]
)
@@ -0,0 +1,3 @@
from .postgres import PostgresChatRepository
__all__ = ["PostgresChatRepository"]
@@ -0,0 +1,150 @@
from uuid import UUID
import asyncpg
from domain import ChatSchema, PageKey
from domain.errors import InvalidRequestError
from infrastructure.utils import Pool, repository_logs, transaction
from interfaces import ChatRepository
class PostgresChatRepository(ChatRepository):
def __init__(self, pool: Pool) -> None:
self._pool = pool
@repository_logs
async def create(
self, title: str, created_by: UUID, member_ids: list[UUID]
) -> ChatSchema:
"""
Чат и участники — одна транзакция. Неизвестный участник нарушает FK
на users, транзакция откатывается целиком, строка chats не остаётся.
Участники вставляются одним запросом; position — порядок в member_ids.
"""
chat_query = """
INSERT INTO chats
(
title,
created_by
)
VALUES
(
$1,
$2
)
RETURNING id, title, created_at;
"""
members_query = """
INSERT INTO chat_members
(
chat_id,
user_id,
position
)
SELECT
$1,
member.user_id,
member.ordinality - 1
FROM unnest($2::uuid[]) WITH ORDINALITY AS member(user_id, ordinality);
"""
try:
async with transaction(self._pool) as conn:
row = await conn.fetchrow(chat_query, title, created_by)
assert row is not None
await conn.execute(members_query, row["id"], member_ids)
except asyncpg.ForeignKeyViolationError:
raise InvalidRequestError(
"member_ids содержит неизвестных пользователей"
) from None
return ChatSchema(
id=row["id"],
title=row["title"],
member_ids=member_ids,
created_at=row["created_at"],
)
@repository_logs
async def get_for_member(self, chat_id: UUID, member_id: UUID) -> ChatSchema | None:
query = """
SELECT
chat.id,
chat.title,
chat.created_at,
ARRAY(
SELECT member.user_id
FROM chat_members member
WHERE member.chat_id = chat.id
ORDER BY member.position
) AS member_ids
FROM chats chat
WHERE
chat.id = $1
AND EXISTS
(
SELECT 1
FROM chat_members caller
WHERE caller.chat_id = chat.id AND caller.user_id = $2
);
"""
async with self._pool.acquire() as conn:
row = await conn.fetchrow(query, chat_id, member_id)
return self._row_to_chat(row) if row is not None else None
@repository_logs
async def list_for_member(
self, member_id: UUID, after: PageKey | None, limit: int
) -> list[ChatSchema]:
"""Чаты пользователя по (created_at, id); со второй страницы — после ключа"""
first_page_query = """
SELECT
chat.id,
chat.title,
chat.created_at,
ARRAY(
SELECT member.user_id
FROM chat_members member
WHERE member.chat_id = chat.id
ORDER BY member.position
) AS member_ids
FROM chat_members caller
JOIN chats chat ON chat.id = caller.chat_id
WHERE caller.user_id = $1
ORDER BY chat.created_at, chat.id
LIMIT $2;
"""
next_page_query = """
SELECT
chat.id,
chat.title,
chat.created_at,
ARRAY(
SELECT member.user_id
FROM chat_members member
WHERE member.chat_id = chat.id
ORDER BY member.position
) AS member_ids
FROM chat_members caller
JOIN chats chat ON chat.id = caller.chat_id
WHERE
caller.user_id = $1
AND (chat.created_at, chat.id) > ($2, $3)
ORDER BY chat.created_at, chat.id
LIMIT $4;
"""
async with self._pool.acquire() as conn:
if after is None:
rows = await conn.fetch(first_page_query, member_id, limit)
else:
rows = await conn.fetch(
next_page_query, member_id, after.created_at, after.id, limit
)
return [self._row_to_chat(row) for row in rows]
@staticmethod
def _row_to_chat(row: asyncpg.Record) -> ChatSchema:
return ChatSchema(
id=row["id"],
title=row["title"],
member_ids=row["member_ids"],
created_at=row["created_at"],
)
@@ -0,0 +1,3 @@
from .postgres import PostgresMessageRepository
__all__ = ["PostgresMessageRepository"]
@@ -0,0 +1,165 @@
from uuid import UUID
import asyncpg
from domain import IdempotencyKeySchema, MessageSchema, PageKey
from domain.errors import ConflictError
from infrastructure.utils import Pool, repository_logs, transaction
from interfaces import MessageRepository
CHAT_LOCK_NAMESPACE = 2
class PostgresMessageRepository(MessageRepository):
def __init__(self, pool: Pool) -> None:
self._pool = pool
@repository_logs
async def create(
self,
chat_id: UUID,
sender_id: UUID,
text: str,
attachment_ids: list[UUID],
idempotency: IdempotencyKeySchema,
) -> tuple[MessageSchema, bool]:
lock_query = """
SELECT pg_advisory_xact_lock($1, hashtext($2::uuid::text));
"""
find_key_query = """
SELECT
idem.request_hash,
message.id,
message.chat_id,
message.sender_id,
message.text,
message.attachment_ids,
message.created_at
FROM message_idempotency_keys idem
JOIN messages message ON message.id = idem.message_id
WHERE
idem.sender_id = $1
AND idem.chat_id = $2
AND idem.idempotency_key = $3
AND idem.expires_at > clock_timestamp();
"""
insert_query = """
INSERT INTO messages
(
chat_id,
sender_id,
text,
attachment_ids,
created_at
)
SELECT
$1,
$2,
$3,
$4::uuid[],
GREATEST(clock_timestamp(), MAX(created_at) + interval '1 microsecond')
FROM messages
WHERE chat_id = $1
RETURNING id, chat_id, sender_id, text, attachment_ids, created_at;
"""
save_key_query = """
INSERT INTO message_idempotency_keys
(
sender_id,
chat_id,
idempotency_key,
request_hash,
message_id,
expires_at
)
VALUES
(
$1,
$2,
$3,
$4,
$5,
clock_timestamp() + make_interval(secs => $6::integer)
)
ON CONFLICT (sender_id, chat_id, idempotency_key) DO UPDATE
SET
request_hash = EXCLUDED.request_hash,
message_id = EXCLUDED.message_id,
created_at = EXCLUDED.created_at,
expires_at = EXCLUDED.expires_at;
"""
async with transaction(self._pool) as conn:
await conn.execute(lock_query, CHAT_LOCK_NAMESPACE, chat_id)
stored = await conn.fetchrow(
find_key_query, sender_id, chat_id, idempotency.key
)
if stored is not None:
if stored["request_hash"] != idempotency.request_hash:
raise ConflictError(
"Idempotency-Key уже использован с другим телом запроса"
)
return self._row_to_message(stored), False
row = await conn.fetchrow(insert_query, chat_id, sender_id, text, attachment_ids)
assert row is not None
await conn.execute(
save_key_query,
sender_id,
chat_id,
idempotency.key,
idempotency.request_hash,
row["id"],
idempotency.ttl_seconds,
)
return self._row_to_message(row), True
@repository_logs
async def list_for_chat(
self, chat_id: UUID, after: PageKey | None, limit: int
) -> list[MessageSchema]:
first_page_query = """
SELECT
id,
chat_id,
sender_id,
text,
attachment_ids,
created_at
FROM messages
WHERE chat_id = $1
ORDER BY created_at, id
LIMIT $2;
"""
next_page_query = """
SELECT
id,
chat_id,
sender_id,
text,
attachment_ids,
created_at
FROM messages
WHERE
chat_id = $1
AND (created_at, id) > ($2, $3)
ORDER BY created_at, id
LIMIT $4;
"""
async with self._pool.acquire() as conn:
if after is None:
rows = await conn.fetch(first_page_query, chat_id, limit)
else:
rows = await conn.fetch(
next_page_query, chat_id, after.created_at, after.id, limit
)
return [self._row_to_message(row) for row in rows]
@staticmethod
def _row_to_message(row: asyncpg.Record) -> MessageSchema:
return MessageSchema(
id=row["id"],
chat_id=row["chat_id"],
sender_id=row["sender_id"],
text=row["text"],
attachment_ids=list(row["attachment_ids"]) if row["attachment_ids"] is not None else [],
created_at=row["created_at"],
)
@@ -0,0 +1,3 @@
from .valkey import ValkeySessionCache
__all__ = ["ValkeySessionCache"]
@@ -0,0 +1,87 @@
import math
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from pydantic import ValidationError
from redis.exceptions import RedisError
from config import LOGGER
from domain import ActiveSessionSchema
from interfaces import SessionCache
if TYPE_CHECKING:
from infrastructure.dependencies.cache import CacheManager
KEY_PREFIX = "session:"
CACHE_UNAVAILABLE_ERRORS: tuple[type[Exception], ...] = (
RedisError,
OSError,
TimeoutError,
)
class ValkeySessionCache(SessionCache):
"""
Read-through кэш сессий в Valkey.
Ключ session:<sha256 токена hex>, значение — JSON ActiveSessionSchema.
TTL = min(остаток жизни сессии, max_ttl): запись исчезает не позже сессии,
а max_ttl ограничивает окно устаревания, если инвалидация не дошла.
"""
def __init__(self, cache: "CacheManager", max_ttl_seconds: int) -> None:
self._cache = cache
self._max_ttl_ms = max_ttl_seconds * 1000
async def get(self, token_hash: bytes) -> ActiveSessionSchema | None:
if not self._cache.is_available or self._cache.client is None:
return None
try:
raw = await self._cache.client.get(self._key(token_hash))
except CACHE_UNAVAILABLE_ERRORS as error:
self._cache.mark_failed(error)
return None
if raw is None:
return None
try:
return ActiveSessionSchema.model_validate_json(raw)
except ValidationError:
LOGGER.warning("Повреждённая запись кэша сессии проигнорирована")
return None
async def set(self, token_hash: bytes, session: ActiveSessionSchema) -> None:
if not self._cache.is_available or self._cache.client is None:
return
remaining = session.expires_at - datetime.now(UTC)
ttl_ms = min(self._max_ttl_ms, math.floor(remaining.total_seconds() * 1000))
if ttl_ms <= 0:
return
try:
await self._cache.client.set(
self._key(token_hash), session.model_dump_json(), px=ttl_ms
)
except CACHE_UNAVAILABLE_ERRORS as error:
self._cache.mark_failed(error)
async def delete(self, token_hash: bytes) -> bool:
if self._cache.client is None:
return False
try:
await self._cache.client.delete(self._key(token_hash))
except CACHE_UNAVAILABLE_ERRORS as error:
self._cache.mark_failed(error)
return False
return True
async def ping(self) -> bool:
if self._cache.client is None:
return False
try:
return bool(await self._cache.client.ping())
except CACHE_UNAVAILABLE_ERRORS:
return False
@staticmethod
def _key(token_hash: bytes) -> str:
return KEY_PREFIX + token_hash.hex()
@@ -0,0 +1,3 @@
from .postgres import PostgresSessionRepository
__all__ = ["PostgresSessionRepository"]
@@ -0,0 +1,85 @@
from datetime import datetime
from uuid import UUID
from domain import ActiveSessionSchema, UserSchema
from infrastructure.utils import Pool, repository_logs
from interfaces import SessionRepository
class PostgresSessionRepository(SessionRepository):
def __init__(self, pool: Pool) -> None:
self._pool = pool
@repository_logs
async def create(
self, token_hash: bytes, user_id: UUID, ttl_seconds: int
) -> datetime:
query = """
INSERT INTO sessions
(
token_hash,
user_id,
expires_at
)
VALUES
(
$1,
$2,
clock_timestamp() + make_interval(secs => $3::integer)
)
RETURNING expires_at;
"""
async with self._pool.acquire() as conn:
expires_at: datetime = await conn.fetchval(
query, token_hash, user_id, ttl_seconds
)
return expires_at
@repository_logs
async def get_active(self, token_hash: bytes) -> ActiveSessionSchema | None:
"""Поиск по PK; истёкшая или отозванная сессия не возвращается"""
query = """
SELECT
usr.id,
usr.username,
usr.display_name,
usr.created_at,
session.expires_at
FROM sessions session
JOIN users usr ON usr.id = session.user_id
WHERE
session.token_hash = $1
AND session.revoked_at IS NULL
AND session.expires_at > clock_timestamp();
"""
async with self._pool.acquire() as conn:
row = await conn.fetchrow(query, token_hash)
if row is None:
return None
return ActiveSessionSchema(
user=UserSchema(
id=row["id"],
username=row["username"],
display_name=row["display_name"],
created_at=row["created_at"],
),
expires_at=row["expires_at"],
)
@repository_logs
async def revoke(self, token_hash: bytes) -> bool:
query = """
UPDATE sessions
SET revoked_at = clock_timestamp()
WHERE
token_hash = $1
AND revoked_at IS NULL
AND expires_at > clock_timestamp();
"""
async with self._pool.acquire() as conn:
status = await conn.execute(query, token_hash)
return self._parse_affected(status) > 0
@staticmethod
def _parse_affected(status: str) -> int:
return int(status.split()[-1])
@@ -0,0 +1,3 @@
from .postgres import PostgresUserRepository
__all__ = ["PostgresUserRepository"]
@@ -0,0 +1,67 @@
import asyncpg
from domain import UserCredentialsSchema, UserSchema
from domain.errors import ConflictError
from infrastructure.utils import Pool, repository_logs
from interfaces import UserRepository
class PostgresUserRepository(UserRepository):
def __init__(self, pool: Pool) -> None:
self._pool = pool
@repository_logs
async def create(
self, username: str, display_name: str, password_hash: str
) -> UserSchema:
query = """
INSERT INTO users
(
username,
display_name,
password_hash
)
VALUES
(
$1,
$2,
$3
)
RETURNING id, username, display_name, created_at;
"""
async with self._pool.acquire() as conn:
try:
row = await conn.fetchrow(query, username, display_name, password_hash)
except asyncpg.UniqueViolationError:
raise ConflictError("Имя пользователя уже занято") from None
assert row is not None
return self._row_to_user(row)
@repository_logs
async def get_credentials(self, username: str) -> UserCredentialsSchema | None:
query = """
SELECT
id,
username,
display_name,
created_at,
password_hash
FROM users
WHERE username = $1;
"""
async with self._pool.acquire() as conn:
row = await conn.fetchrow(query, username)
if row is None:
return None
return UserCredentialsSchema(
user=self._row_to_user(row), password_hash=row["password_hash"]
)
@staticmethod
def _row_to_user(row: asyncpg.Record) -> UserSchema:
return UserSchema(
id=row["id"],
username=row["username"],
display_name=row["display_name"],
created_at=row["created_at"],
)
+37
View File
@@ -0,0 +1,37 @@
from typing import AsyncGenerator
import aioboto3
from botocore.exceptions import ClientError
from config import LOGGER
class S3Client:
def __init__(self, endpoint_url: str, region_name: str, access_key: str, secret_key: str, bucket_name: str):
self.endpoint_url = endpoint_url
self.region_name = region_name
self.access_key = access_key
self.secret_key = secret_key
self.bucket_name = bucket_name
self.session = aioboto3.Session()
def _get_client(self):
return self.session.client("s3", endpoint_url=self.endpoint_url, region_name=self.region_name, aws_access_key_id=self.access_key, aws_secret_access_key=self.secret_key)
async def upload_file(self, file_path: str, object_name: str, content_type: str) -> None:
async with self._get_client() as client:
try:
await client.upload_file(file_path, self.bucket_name, object_name, ExtraArgs={"ContentType": content_type})
except ClientError as e:
LOGGER.error(f"Ошибка загрузки файла в S3: {e}")
raise
async def stream_file(self, object_name: str) -> AsyncGenerator[bytes, None]:
async with self._get_client() as client:
try:
response = await client.get_object(Bucket=self.bucket_name, Key=object_name)
stream = response["Body"]
while True:
chunk = await stream.read(64 * 1024)
if not chunk: break
yield chunk
except ClientError as e:
LOGGER.error(f"Ошибка чтения файла из S3: {e}")
raise
+4
View File
@@ -0,0 +1,4 @@
from .password_hasher import PasswordHasher, get_password_hasher
from .tokens import generate_token, hash_token
__all__ = ["PasswordHasher", "get_password_hasher", "generate_token", "hash_token"]
@@ -0,0 +1,50 @@
import asyncio
import secrets
from concurrent.futures import ThreadPoolExecutor
from argon2 import PasswordHasher as Argon2Hasher
from argon2.exceptions import InvalidHashError, VerificationError
from argon2.profiles import RFC_9106_LOW_MEMORY
from starlette.requests import HTTPConnection
class PasswordHasher:
"""
Хеширование паролей с использованием Argon2id.
"""
def __init__(self, workers: int) -> None:
self._hasher = Argon2Hasher.from_parameters(RFC_9106_LOW_MEMORY)
self._executor = ThreadPoolExecutor(
max_workers=workers, thread_name_prefix="argon2"
)
self._dummy_hash = self._hasher.hash(secrets.token_urlsafe())
async def hash(self, password: str) -> str:
loop = asyncio.get_running_loop()
return await loop.run_in_executor(self._executor, self._hasher.hash, password)
async def verify(self, password_hash: str, password: str) -> bool:
loop = asyncio.get_running_loop()
return await loop.run_in_executor(
self._executor, self._verify, password_hash, password
)
async def verify_dummy(self, password: str) -> None:
"""Та же работа, что у verify, для выравнивания времени ответа"""
await self.verify(self._dummy_hash, password)
def close(self) -> None:
self._executor.shutdown(wait=False, cancel_futures=True)
def _verify(self, password_hash: str, password: str) -> bool:
try:
return self._hasher.verify(password_hash, password)
except (VerificationError, InvalidHashError):
return False
def get_password_hasher(connection: HTTPConnection) -> PasswordHasher:
"""Получение hasher, созданного в lifespan."""
hasher: PasswordHasher = connection.app.state.password_hasher
return hasher
+17
View File
@@ -0,0 +1,17 @@
import hashlib
import secrets
def generate_token() -> str:
"""Непрозрачный токен сессии: 256 бит из CSPRNG, base64url"""
return secrets.token_urlsafe(32)
def hash_token(token: str) -> bytes:
"""
SHA-256 токена для хранения и поиска.
Токен уже имеет 256 бит энтропии, перебор невозможен, поэтому быстрый hash
достаточен; медленный KDF нужен паролям, у которых энтропия низкая.
"""
return hashlib.sha256(token.encode()).digest()
+7
View File
@@ -0,0 +1,7 @@
from .auth_service import AuthService
from .chat_service import ChatService
from .message_service import MessageService
from .user_service import UserService
from .attachment_service import AttachmentService
__all__ = ["UserService", "AuthService", "ChatService", "MessageService", "AttachmentService", ]
@@ -0,0 +1,2 @@
from .attachment_service import AttachmentService
__all__ = ["AttachmentService"]
@@ -0,0 +1,98 @@
import struct
import os
from uuid import uuid4
from datetime import datetime, UTC
from config import LOGGER
from domain import AttachmentSchema
from domain.errors import AppError, ErrorCode, InvalidRequestError, NotFoundError
from interfaces import AttachmentRepository, ChatRepository
from infrastructure.s3_client import S3Client
class PayloadTooLargeError(AppError):
status_code = 413
code = ErrorCode.PAYLOAD_TOO_LARGE
default_message = "Превышен лимит размера"
class UnsupportedMediaTypeError(AppError):
status_code = 415
code = ErrorCode.UNSUPPORTED_MEDIA_TYPE
default_message = "Неподдерживаемый формат"
def validate_image_and_get_mime(file_path: str) -> str:
with open(file_path, "rb") as f:
header = f.read(24)
if header.startswith(b"\x89PNG\r\n\x1a\n"):
w, h = struct.unpack(">II", header[16:24])
if w * h > 40_000_000:
raise PayloadTooLargeError("Разрешение больше 40М пикселей")
return "image/png"
elif header[:2] == b"\xff\xd8":
with open(file_path, "rb") as f:
data = f.read()
i = 2
sof = {0xc0, 0xc1, 0xc2, 0xc3, 0xc5, 0xc6, 0xc7, 0xc9, 0xca, 0xcb, 0xcd, 0xce, 0xcf}
while i < len(data):
if data[i] != 0xff: raise UnsupportedMediaTypeError("Поврежденный JPEG")
while i < len(data) and data[i] == 0xff: i += 1
if i >= len(data): break
marker = data[i]
i += 1
if marker in {0xd9, 0xda}: break
if marker in {0x01, 0xd8} or 0xd0 <= marker <= 0xd7: continue
if i + 2 > len(data): break
length = int.from_bytes(data[i:i+2], "big")
if marker in sof:
if i + 7 > len(data): break
h = int.from_bytes(data[i+3:i+5], "big")
w = int.from_bytes(data[i+5:i+7], "big")
if w * h > 40_000_000:
raise PayloadTooLargeError("Разрешение больше 40М пикселей")
return "image/jpeg"
i += length
raise UnsupportedMediaTypeError("Не удалось найти размеры JPEG")
raise UnsupportedMediaTypeError("Формат не поддерживается")
class AttachmentService:
def __init__(self, s3_client: S3Client, attachment_repo: AttachmentRepository, chat_repo: ChatRepository):
self._s3 = s3_client
self._attachments = attachment_repo
self._chats = chat_repo
async def process_and_save(self, temp_file_path: str, filename: str, chat_id, owner_id, size_bytes: int, sha256_hash: str) -> AttachmentSchema:
if "/" in filename or "\\" in filename or "\0" in filename:
raise InvalidRequestError("Недопустимое имя файла")
chat = await self._chats.get_for_member(chat_id, owner_id)
if not chat:
raise NotFoundError()
mime = validate_image_and_get_mime(temp_file_path)
attachment_id = uuid4()
object_name = f"{chat_id}/{attachment_id}"
await self._s3.upload_file(temp_file_path, object_name, mime)
LOGGER.info(f"Файл {attachment_id} загружен в S3")
now = datetime.now(UTC)
attachment = AttachmentSchema(
id=attachment_id, chat_id=chat_id, owner_id=owner_id,
filename=filename, content_type=mime, size_bytes=size_bytes,
sha256=sha256_hash, status="ready", variants=[],
error_code=None, created_at=now, updated_at=now
)
return await self._attachments.create(attachment)
async def get_attachment_meta(self, attachment_id, user_id) -> AttachmentSchema:
att = await self._attachments.get_by_id(attachment_id)
if not att or not await self._chats.get_for_member(att.chat_id, user_id):
raise NotFoundError()
return att
async def get_attachment_stream(self, attachment_id, user_id):
att = await self.get_attachment_meta(attachment_id, user_id)
if att.status != "ready":
from domain.errors import ConflictError
raise ConflictError("Файл еще не готов")
object_name = f"{att.chat_id}/{att.id}"
return self._s3.stream_file(object_name), att.content_type
@@ -0,0 +1,3 @@
from .auth_service import AuthService
__all__ = ["AuthService"]
@@ -0,0 +1,92 @@
from datetime import UTC, datetime
from config import LOGGER
from domain import BasicCredentialsSchema, SessionSchema, UserSchema
from domain.errors import InvalidCredentialsError, UnauthorizedError
from infrastructure.security import PasswordHasher, generate_token, hash_token
from interfaces import SessionCache, SessionRepository, UserRepository
TOKEN_MAX_LENGTH = 256
class AuthService:
"""
Basic → opaque Bearer-сессия.
Источник истины — SessionRepository (PostgreSQL), SessionCache (Valkey) —
read-through кэш: при промахе или недоступности кэша сессия читается из БД.
"""
def __init__(
self,
user_repository: UserRepository,
session_repository: SessionRepository,
session_cache: SessionCache,
hasher: PasswordHasher,
session_ttl_seconds: int,
) -> None:
self._users = user_repository
self._sessions = session_repository
self._cache = session_cache
self._hasher = hasher
self._session_ttl = session_ttl_seconds
async def login(self, credentials: BasicCredentialsSchema) -> SessionSchema:
"""
Создание сессии по username и паролю.
Raises:
InvalidCredentialsError: неизвестный username или неверный пароль —
одинаковый ответ и сопоставимое время
"""
found = await self._users.get_credentials(credentials.username)
if found is None:
await self._hasher.verify_dummy(credentials.password)
raise InvalidCredentialsError()
if not await self._hasher.verify(found.password_hash, credentials.password):
raise InvalidCredentialsError()
token = generate_token()
expires_at = await self._sessions.create(
hash_token(token), found.user.id, self._session_ttl
)
LOGGER.info(f"Создана сессия пользователя {found.user.id}")
return SessionSchema(token=token, expires_at=expires_at, user=found.user)
async def authenticate(self, token: str | None) -> UserSchema:
"""
Владелец действующей сессии.
Raises:
UnauthorizedError: токена нет, он неизвестен, истёк или отозван
UnavailableError: кэш промахнулся, а PostgreSQL недоступен
"""
if not token or len(token) > TOKEN_MAX_LENGTH:
raise UnauthorizedError()
token_hash = hash_token(token)
session = await self._cache.get(token_hash)
if session is not None and session.expires_at > datetime.now(UTC):
return session.user
session = await self._sessions.get_active(token_hash)
if session is None:
raise UnauthorizedError()
await self._cache.set(token_hash, session)
return session.user
async def logout(self, token: str | None) -> None:
"""
Отзыв в источнике истины, затем инвалидация кэша.
Raises:
UnauthorizedError: действующей сессии с таким токеном нет
"""
if not token or len(token) > TOKEN_MAX_LENGTH:
raise UnauthorizedError()
token_hash = hash_token(token)
if not await self._sessions.revoke(token_hash):
raise UnauthorizedError()
if not await self._cache.delete(token_hash):
LOGGER.warning(
"Сессия отозвана в PostgreSQL, но запись кэша не удалена: "
"она истечёт по TTL кэша"
)
LOGGER.info("Сессия отозвана")
@@ -0,0 +1,3 @@
from .chat_service import ChatService
__all__ = ["ChatService"]

Some files were not shown because too many files have changed in this diff Show More