This commit is contained in:
@@ -5,6 +5,7 @@ APP_PORT=8080
|
||||
APP_ENV=lab
|
||||
SESSION_TTL_SECONDS=3600
|
||||
CACHE_TTL_SECONDS=60
|
||||
CACHE_STAMPEDE_PROTECTION=true
|
||||
# Choose credentials locally. File paths contain no secret values.
|
||||
DATABASE_URL_FILE=/run/secrets/database_url
|
||||
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Отчёт — лабораторная 2
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка: 3
|
||||
- Автор, группа: Козырев Матвей БВТ2301
|
||||
- Целевая оценка: 5; трек 1 — согласованность кэша
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные библиотеки: FastAPI 0.141.1, Uvicorn 0.40.0, asyncpg 0.31.0, argon2-cffi 25.1.0; зафиксированы в `pyproject.toml` и `uv.lock`.
|
||||
- Использованные библиотеки: FastAPI 0.141.1, Uvicorn 0.40.0, asyncpg 0.31.0, argon2-cffi 25.1.0, redis 5.0.8; зафиксированы в `pyproject.toml` и `uv.lock`.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
@@ -62,6 +62,7 @@ docker compose down
|
||||
```text
|
||||
HTTP-клиент → FastAPI API → asyncpg pool (1..10 соединений) → PostgreSQL
|
||||
└→ миграции SQL при старте
|
||||
└→ Redis cache истории (pool до 50 соединений)
|
||||
```
|
||||
|
||||
PostgreSQL — источник истины для `users`, `chats`, `chat_members`,
|
||||
@@ -81,6 +82,13 @@ Logout удаляет текущую серверную сессию. Срок
|
||||
История выбирается keyset-запросом по `(created_at, id)`; индекс
|
||||
`messages_chat_created_id_idx` соответствует этому запросу.
|
||||
|
||||
Redis кэширует только первую страницу истории. Ключ включает `chat_id`,
|
||||
`limit` и `history_version` из PostgreSQL. В той же транзакции, что и
|
||||
вставка сообщения, PostgreSQL увеличивает `history_version`; старый ключ
|
||||
поэтому больше не читается. Redis не источник истины: при его отказе кэш
|
||||
отключается в процессе API, а чтение продолжает PostgreSQL. Redis использует
|
||||
AOF и named volume `redis_data`.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий | Команда | Наблюдаемый результат | Статус |
|
||||
@@ -94,10 +102,45 @@ Logout удаляет текущую серверную сессию. Срок
|
||||
| Гонка и rollback | `make test-grade4` | Ровно одна регистрация успешна; неуспешный чат не виден | `evidence/grade4-checks-2026-10-08.txt` |
|
||||
| TTL и зависимости | `make test-grade4`; TTL=2 | Expired session → 401; Redis fallback проходит smoke; DB readiness → 503 | `evidence/grade4-checks-2026-10-08.txt` |
|
||||
| Индекс истории | `./scripts/explain_history.sh` | Index Scan, 50 строк, shared hit=3 | `evidence/explain-history-2026-10-08.txt` |
|
||||
| Трек 5: stampede | `student-tests/cache_stampede.py` | 20 SQL-загрузок → 1 при 20 читателях | `evidence/track1-cache-stampede-2026-10-08.txt` |
|
||||
| Трек 5: restart cache | `student-tests/cache_recovery.py` | История не изменилась после restart Redis | `evidence/track1-cache-stampede-2026-10-08.txt` |
|
||||
|
||||
## Трек 5 — согласованность кэша
|
||||
|
||||
**Гипотеза.** При холодном ключе 20 одновременных запросов первой страницы
|
||||
истории создают 20 SQL-загрузок без координации. Redis-lock `SET NX EX 2`
|
||||
оставит одну загрузку; остальные запросы ждут заполнения ключа до 1 секунды.
|
||||
|
||||
**Методика.** Для каждой точки API запускался с одним и тем же Compose-стендом:
|
||||
PostgreSQL 17.6, Redis 7.4.2, Python 3.12.12, Docker Engine 29.8.0, 10 CPU и
|
||||
16 GiB RAM. `student-tests/cache_stampede.py` создаёт новый чат с одним
|
||||
сообщением и одновременно запускает 20 GET первой страницы. SQL-загрузка
|
||||
считается по записи `history_sql_page_load` в журнале API. Baseline запускает
|
||||
API с `CACHE_STAMPEDE_PROTECTION=false`; затем тот же сценарий повторяется с
|
||||
`true`. Данные, чат и ключ каждый раз новые.
|
||||
|
||||
| Точка | Успешных чтений | SQL-загрузок | p50 / p95 / max latency |
|
||||
| --- | ---: | ---: | --- |
|
||||
| Без координации | 20 | 20 | 111.620 / 113.364 / 113.603 ms |
|
||||
| Redis-lock | 20 | 1 | 157.831 / 159.477 / 159.549 ms |
|
||||
|
||||
Число SQL-загрузок уменьшилось на `(20 − 1) / 20 × 100 = 95%`. В этой
|
||||
единственной точке p50 вырос на `157.831 − 111.620 = 46.211 ms`, поскольку
|
||||
ожидающие запросы опрашивают кэш с интервалом 50 ms. Это не оценка
|
||||
производительности системы в целом: измерение одноразовое, closed-loop,
|
||||
включает HTTP-клиент и не отделяет влияние Docker Desktop.
|
||||
|
||||
read-after-write проверяет `student-tests/lab4_checks.py`: страница сначала
|
||||
кэшируется, затем создаётся сообщение, и следующий GET содержит оба сообщения.
|
||||
`student-tests/cache_recovery.py` заполняет кэш, перезапускает Redis и
|
||||
проверяет неизменность ответа. Raw results и команды сохранены в evidence.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
Redis/Valkey не используется: он относится к уровню 4 этой лабораторной.
|
||||
Результат трека относится к одному локальному прогону с 20 читателями; для
|
||||
обобщения нужны повторы, нагрузка других размеров и измерение CPU/RAM
|
||||
контейнеров. Redis-lock не заменяет PostgreSQL: при недоступности Redis API
|
||||
безопасно обходит кэш, а при недоступности PostgreSQL readiness возвращает 503.
|
||||
Публичные smoke и persistence выполнены на локальном Compose-стенде 2026-10-08.
|
||||
При фиксации работы в PR сохраните их stdout в `evidence/` без токенов и
|
||||
паролей.
|
||||
|
||||
@@ -26,6 +26,7 @@ services:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
CACHE_TTL_SECONDS: ${CACHE_TTL_SECONDS:-60}
|
||||
CACHE_STAMPEDE_PROTECTION: ${CACHE_STAMPEDE_PROTECTION:-true}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
secrets:
|
||||
- database_url
|
||||
@@ -33,6 +34,8 @@ services:
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "8080:8080"
|
||||
healthcheck:
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
Date: 2026-10-08
|
||||
Track: 1 — cache consistency
|
||||
Environment: Docker Engine 29.8.0; PostgreSQL 17.6; Redis 7.4.2;
|
||||
Python 3.12.12; 10 CPU; 16 GiB RAM.
|
||||
|
||||
Method:
|
||||
student-tests/cache_stampede.py creates a new chat with one message and
|
||||
launches 20 concurrent GET requests for its cold first history page.
|
||||
The API logs one history_sql_page_load per PostgreSQL page query.
|
||||
|
||||
Baseline: CACHE_STAMPEDE_PROTECTION=false
|
||||
{"chat_id":"5aa10886-61e0-47f2-82d2-c09a05bbd45f","concurrency":20,"successful_reads":20,"latency_ms":{"p50":111.620,"p95":113.364,"max":113.603},"sql_page_loads":20}
|
||||
|
||||
After: CACHE_STAMPEDE_PROTECTION=true
|
||||
{"chat_id":"96fd0c67-907f-4681-8ec5-6fcbfc4d8fc7","concurrency":20,"successful_reads":20,"latency_ms":{"p50":157.831,"p95":159.477,"max":159.549},"sql_page_loads":1}
|
||||
|
||||
Reduction: (20 - 1) / 20 * 100 = 95%.
|
||||
|
||||
Cache restart:
|
||||
Command: python3 student-tests/cache_recovery.py
|
||||
PASS: history remains correct after Redis restart
|
||||
|
||||
Read-after-write:
|
||||
Command: python3 student-tests/lab4_checks.py
|
||||
PASS: registration race, chat rollback and cache read-after-write
|
||||
+41
-38
@@ -13,6 +13,7 @@ _CONFIGURATION_KEYS = {
|
||||
"CACHE_URL",
|
||||
"CACHE_URL_FILE",
|
||||
"CACHE_TTL_SECONDS",
|
||||
"CACHE_STAMPEDE_PROTECTION",
|
||||
"SESSION_TTL_SECONDS",
|
||||
}
|
||||
|
||||
@@ -28,6 +29,7 @@ class Settings:
|
||||
session_ttl_seconds: int
|
||||
cache_url: str | None
|
||||
cache_ttl_seconds: int
|
||||
cache_stampede_protection: bool
|
||||
|
||||
|
||||
def load_settings(env_file: Path = ROOT / ".env") -> Settings:
|
||||
@@ -46,54 +48,55 @@ def load_settings(env_file: Path = ROOT / ".env") -> Settings:
|
||||
value = value[1:-1]
|
||||
os.environ.setdefault(key, value)
|
||||
|
||||
host = os.environ.get("APP_HOST", "0.0.0.0").strip()
|
||||
environment = os.environ.get("APP_ENV", "lab").strip()
|
||||
def value(name: str, default: str = "") -> str:
|
||||
return os.environ.get(name, default).strip()
|
||||
|
||||
def number(name: str, default: int) -> int:
|
||||
try:
|
||||
result = int(value(name, str(default)))
|
||||
except ValueError as error:
|
||||
raise ValueError(f"{name} должен быть целым числом") from error
|
||||
if result < 1:
|
||||
raise ValueError(f"{name} должен быть положительным")
|
||||
return result
|
||||
|
||||
def url(name: str, file_name: str, prefixes: tuple[str, ...]) -> str:
|
||||
result = value(name)
|
||||
path = value(file_name)
|
||||
if path:
|
||||
try:
|
||||
result = Path(path).read_text(encoding="utf-8").strip()
|
||||
except OSError as error:
|
||||
raise ValueError(f"{file_name} недоступен") from error
|
||||
if result and not result.startswith(prefixes):
|
||||
raise ValueError(f"{name} содержит неверный URL")
|
||||
return result
|
||||
|
||||
host = value("APP_HOST", "0.0.0.0")
|
||||
environment = value("APP_ENV", "lab")
|
||||
if not host:
|
||||
raise ValueError("APP_HOST не может быть пустым")
|
||||
if not environment:
|
||||
raise ValueError("APP_ENV не может быть пустым")
|
||||
try:
|
||||
port = int(os.environ.get("APP_PORT", "8080"))
|
||||
except ValueError as error:
|
||||
raise ValueError("APP_PORT должен быть целым числом") from error
|
||||
if not 1 <= port <= 65535:
|
||||
port = number("APP_PORT", 8080)
|
||||
if port > 65535:
|
||||
raise ValueError("APP_PORT должен находиться в диапазоне 1..65535")
|
||||
database_url_file = os.environ.get("DATABASE_URL_FILE", "").strip()
|
||||
database_url = os.environ.get("DATABASE_URL", "").strip()
|
||||
if database_url_file:
|
||||
try:
|
||||
database_url = Path(database_url_file).read_text(encoding="utf-8").strip()
|
||||
except OSError as error:
|
||||
raise ValueError("DATABASE_URL_FILE недоступен") from error
|
||||
if not database_url.startswith(("postgresql://", "postgres://")):
|
||||
database_url = url(
|
||||
"DATABASE_URL", "DATABASE_URL_FILE", ("postgresql://", "postgres://")
|
||||
)
|
||||
if not database_url:
|
||||
raise ValueError("Нужен PostgreSQL DATABASE_URL или DATABASE_URL_FILE")
|
||||
try:
|
||||
session_ttl_seconds = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
|
||||
except ValueError as error:
|
||||
raise ValueError("SESSION_TTL_SECONDS должен быть целым числом") from error
|
||||
if session_ttl_seconds < 1:
|
||||
raise ValueError("SESSION_TTL_SECONDS должен быть положительным")
|
||||
cache_url_file = os.environ.get("CACHE_URL_FILE", "").strip()
|
||||
cache_url = os.environ.get("CACHE_URL", "").strip()
|
||||
if cache_url_file:
|
||||
try:
|
||||
cache_url = Path(cache_url_file).read_text(encoding="utf-8").strip()
|
||||
except OSError as error:
|
||||
raise ValueError("CACHE_URL_FILE недоступен") from error
|
||||
if cache_url and not cache_url.startswith(("redis://", "rediss://")):
|
||||
raise ValueError("CACHE_URL должен содержать Redis URL")
|
||||
try:
|
||||
cache_ttl_seconds = int(os.environ.get("CACHE_TTL_SECONDS", "60"))
|
||||
except ValueError as error:
|
||||
raise ValueError("CACHE_TTL_SECONDS должен быть целым числом") from error
|
||||
if cache_ttl_seconds < 1:
|
||||
raise ValueError("CACHE_TTL_SECONDS должен быть положительным")
|
||||
cache_url = url("CACHE_URL", "CACHE_URL_FILE", ("redis://", "rediss://"))
|
||||
raw_protection = value("CACHE_STAMPEDE_PROTECTION", "true").lower()
|
||||
if raw_protection not in {"true", "false"}:
|
||||
raise ValueError("CACHE_STAMPEDE_PROTECTION должен быть true или false")
|
||||
return Settings(
|
||||
host=host,
|
||||
port=port,
|
||||
environment=environment,
|
||||
database_url=database_url,
|
||||
session_ttl_seconds=session_ttl_seconds,
|
||||
session_ttl_seconds=number("SESSION_TTL_SECONDS", 3600),
|
||||
cache_url=cache_url or None,
|
||||
cache_ttl_seconds=cache_ttl_seconds,
|
||||
cache_ttl_seconds=number("CACHE_TTL_SECONDS", 60),
|
||||
cache_stampede_protection=raw_protection == "true",
|
||||
)
|
||||
|
||||
+18
-9
@@ -17,15 +17,20 @@ from .storage.history_cache import HistoryCache
|
||||
from .storage.postgres import PostgresStore
|
||||
|
||||
|
||||
access_logger = logging.getLogger("messenger.access")
|
||||
if not access_logger.handlers:
|
||||
def logger(name: str) -> logging.Logger:
|
||||
result = logging.getLogger(name)
|
||||
if result.handlers:
|
||||
return result
|
||||
handler = logging.StreamHandler(sys.stdout)
|
||||
handler.setFormatter(
|
||||
logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
)
|
||||
access_logger.addHandler(handler)
|
||||
access_logger.setLevel(logging.INFO)
|
||||
access_logger.propagate = False
|
||||
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s"))
|
||||
result.addHandler(handler)
|
||||
result.setLevel(logging.INFO)
|
||||
result.propagate = False
|
||||
return result
|
||||
|
||||
|
||||
access_logger = logger("messenger.access")
|
||||
logger("messenger.cache")
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
@@ -36,7 +41,11 @@ async def lifespan(application: FastAPI):
|
||||
await store.migrate()
|
||||
application.state.store = store
|
||||
application.state.settings = settings
|
||||
cache = HistoryCache(settings.cache_url, settings.cache_ttl_seconds)
|
||||
cache = HistoryCache(
|
||||
settings.cache_url,
|
||||
settings.cache_ttl_seconds,
|
||||
settings.cache_stampede_protection,
|
||||
)
|
||||
await cache.connect()
|
||||
application.state.history_cache = cache
|
||||
try:
|
||||
|
||||
+102
-36
@@ -1,4 +1,5 @@
|
||||
import base64
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from uuid import UUID
|
||||
|
||||
@@ -6,13 +7,24 @@ from fastapi import APIRouter, Depends, Header, Query, Request, Response
|
||||
|
||||
from .cursor import CursorError, decode_cursor, encode_cursor
|
||||
from .errors import ApiError
|
||||
from .models import Chat, ChatPageResponse, CreateChatRequest, CreateMessageRequest, CreateUserRequest, Message, MessagePageResponse, Session, User
|
||||
from .storage.postgres import PostgresStore
|
||||
from .storage.history_cache import HistoryCache
|
||||
from .storage.postgres import PostgresStore
|
||||
from .models import (
|
||||
Chat,
|
||||
ChatPageResponse,
|
||||
CreateChatRequest,
|
||||
CreateMessageRequest,
|
||||
CreateUserRequest,
|
||||
Message,
|
||||
MessagePageResponse,
|
||||
Session,
|
||||
User,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
BEARER_CHALLENGE = {'WWW-Authenticate': 'Bearer realm="messenger"'}
|
||||
BASIC_CHALLENGE = {'WWW-Authenticate': 'Basic realm="messenger", charset="UTF-8"'}
|
||||
cache_logger = logging.getLogger("messenger.cache")
|
||||
BEARER_CHALLENGE = {"WWW-Authenticate": 'Bearer realm="messenger"'}
|
||||
BASIC_CHALLENGE = {"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'}
|
||||
|
||||
def get_store(request: Request) -> PostgresStore:
|
||||
return request.app.state.store
|
||||
@@ -20,7 +32,11 @@ def get_store(request: Request) -> PostgresStore:
|
||||
def get_history_cache(request: Request) -> HistoryCache:
|
||||
return request.app.state.history_cache
|
||||
|
||||
async def get_current_user(authorization: str | None = Header(default=None), store: PostgresStore = Depends(get_store)) -> dict:
|
||||
|
||||
async def get_current_user(
|
||||
authorization: str | None = Header(default=None),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
if authorization is None or not authorization.startswith("Bearer "):
|
||||
raise ApiError(401, "unauthorized", "Требуется Bearer-сессия", BEARER_CHALLENGE)
|
||||
token = authorization[7:]
|
||||
@@ -45,6 +61,23 @@ def parse_basic(authorization: str | None) -> tuple[str, str]:
|
||||
raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE)
|
||||
return username, password
|
||||
|
||||
|
||||
def cursor_position(cursor: str | None, scope: str) -> tuple[datetime, UUID] | None:
|
||||
if cursor is None:
|
||||
return None
|
||||
try:
|
||||
created_at, item_id = decode_cursor(cursor, scope)
|
||||
return created_at, UUID(item_id)
|
||||
except (CursorError, ValueError) as error:
|
||||
raise ApiError(400, "invalid_request", "Некорректный cursor") from error
|
||||
|
||||
|
||||
def next_page_cursor(rows: list[dict], limit: int, scope: str) -> str | None:
|
||||
if len(rows) <= limit:
|
||||
return None
|
||||
last = rows[limit - 1]
|
||||
return encode_cursor(scope, last["created_at"], str(last["id"]))
|
||||
|
||||
@router.get("/health/live")
|
||||
async def liveness() -> dict[str, str]:
|
||||
return {"status": "ok"}
|
||||
@@ -55,11 +88,17 @@ async def readiness(store: PostgresStore = Depends(get_store)) -> dict[str, str]
|
||||
return {"status": "ok"}
|
||||
|
||||
@router.post("/api/v1/users", response_model=User, status_code=201)
|
||||
async def register_user(payload: CreateUserRequest, store: PostgresStore = Depends(get_store)) -> dict:
|
||||
async def register_user(
|
||||
payload: CreateUserRequest, store: PostgresStore = Depends(get_store)
|
||||
) -> dict:
|
||||
return await store.create_user(payload.username, payload.display_name, payload.password)
|
||||
|
||||
@router.post("/api/v1/auth/sessions", response_model=Session, status_code=201)
|
||||
async def create_session(request: Request, authorization: str | None = Header(default=None), store: PostgresStore = Depends(get_store)) -> dict:
|
||||
async def create_session(
|
||||
request: Request,
|
||||
authorization: str | None = Header(default=None),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
if await request.body():
|
||||
raise ApiError(400, "invalid_request", "Тело запроса не требуется")
|
||||
username, password = parse_basic(authorization)
|
||||
@@ -69,7 +108,11 @@ async def create_session(request: Request, authorization: str | None = Header(de
|
||||
return await store.create_session(user, request.app.state.settings.session_ttl_seconds)
|
||||
|
||||
@router.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||
async def revoke_session(authorization: str | None = Header(default=None), store: PostgresStore = Depends(get_store), user: dict = Depends(get_current_user)) -> Response:
|
||||
async def revoke_session(
|
||||
authorization: str | None = Header(default=None),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
user: dict = Depends(get_current_user),
|
||||
) -> Response:
|
||||
del user
|
||||
await store.revoke_session(authorization[7:])
|
||||
return Response(status_code=204)
|
||||
@@ -79,51 +122,74 @@ async def current_user(user: dict = Depends(get_current_user)) -> dict:
|
||||
return user
|
||||
|
||||
@router.post("/api/v1/chats", response_model=Chat, status_code=201)
|
||||
async def create_chat(payload: CreateChatRequest, user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)) -> dict:
|
||||
async def create_chat(
|
||||
payload: CreateChatRequest,
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
return await store.create_chat(user["id"], payload.title, payload.member_ids)
|
||||
|
||||
@router.get("/api/v1/chats", response_model=ChatPageResponse)
|
||||
async def list_chats(user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store), limit: int = Query(default=50, ge=1, le=100), cursor: str | None = Query(default=None, min_length=1)) -> ChatPageResponse:
|
||||
after: tuple[datetime, UUID] | None = None
|
||||
async def list_chats(
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
limit: int = Query(default=50, ge=1, le=100),
|
||||
cursor: str | None = Query(default=None, min_length=1),
|
||||
) -> ChatPageResponse:
|
||||
scope = "chats:" + str(user["id"])
|
||||
if cursor is not None:
|
||||
try:
|
||||
created_at, item_id = decode_cursor(cursor, scope)
|
||||
after = created_at, UUID(item_id)
|
||||
except (CursorError, ValueError) as error:
|
||||
raise ApiError(400, "invalid_request", "Некорректный cursor") from error
|
||||
rows = await store.list_chats_for_member(user["id"], after, limit)
|
||||
rows = await store.list_chats_for_member(user["id"], cursor_position(cursor, scope), limit)
|
||||
page = rows[:limit]
|
||||
next_cursor = encode_cursor(scope, page[-1]["created_at"], str(page[-1]["id"])) if len(rows) > limit else None
|
||||
return ChatPageResponse(items=page, next_cursor=next_cursor)
|
||||
return ChatPageResponse(items=page, next_cursor=next_page_cursor(rows, limit, scope))
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}", response_model=Chat)
|
||||
async def get_chat(chat_id: UUID, user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)) -> dict:
|
||||
async def get_chat(
|
||||
chat_id: UUID,
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
return await store.get_chat_for_member(chat_id, user["id"])
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/messages", response_model=Message, status_code=201)
|
||||
async def create_message(chat_id: UUID, payload: CreateMessageRequest, user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)) -> dict:
|
||||
message, _version = await store.create_message(chat_id, user["id"], payload.text)
|
||||
return message
|
||||
async def create_message(
|
||||
chat_id: UUID,
|
||||
payload: CreateMessageRequest,
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
return await store.create_message(chat_id, user["id"], payload.text)
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePageResponse)
|
||||
async def list_messages(chat_id: UUID, user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store), cache: HistoryCache = Depends(get_history_cache), limit: int = Query(default=50, ge=1, le=100), cursor: str | None = Query(default=None, min_length=1)) -> MessagePageResponse:
|
||||
after: tuple[datetime, UUID] | None = None
|
||||
async def list_messages(
|
||||
chat_id: UUID,
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
cache: HistoryCache = Depends(get_history_cache),
|
||||
limit: int = Query(default=50, ge=1, le=100),
|
||||
cursor: str | None = Query(default=None, min_length=1),
|
||||
) -> MessagePageResponse:
|
||||
scope = "messages:" + str(chat_id)
|
||||
if cursor is not None:
|
||||
try:
|
||||
created_at, item_id = decode_cursor(cursor, scope)
|
||||
after = created_at, UUID(item_id)
|
||||
except (CursorError, ValueError) as error:
|
||||
raise ApiError(400, "invalid_request", "Некорректный cursor") from error
|
||||
after = cursor_position(cursor, scope)
|
||||
if cursor is None:
|
||||
version = await store.history_version_for_member(chat_id, user["id"])
|
||||
rows = await cache.get(chat_id, version, limit)
|
||||
if rows is None:
|
||||
rows = await store.list_messages_for_member(chat_id, user["id"], after, limit)
|
||||
await cache.set(chat_id, version, limit, rows)
|
||||
lock_token = await cache.acquire_load_lock(chat_id, version, limit)
|
||||
if lock_token == "":
|
||||
rows = await cache.wait_for_fill(chat_id, version, limit)
|
||||
if rows is None:
|
||||
cache_logger.info(
|
||||
"history_sql_page_load chat_id=%s version=%s limit=%s",
|
||||
chat_id,
|
||||
version,
|
||||
limit,
|
||||
)
|
||||
try:
|
||||
rows = await store.list_messages_for_member(chat_id, user["id"], None, limit)
|
||||
await cache.set(chat_id, version, limit, rows)
|
||||
finally:
|
||||
if lock_token:
|
||||
await cache.release_load_lock(chat_id, version, limit, lock_token)
|
||||
else:
|
||||
rows = await store.list_messages_for_member(chat_id, user["id"], after, limit)
|
||||
page = rows[:limit]
|
||||
next_cursor = encode_cursor(scope, page[-1]["created_at"], str(page[-1]["id"])) if len(rows) > limit else None
|
||||
return MessagePageResponse(items=page, next_cursor=next_cursor)
|
||||
return MessagePageResponse(items=page, next_cursor=next_page_cursor(rows, limit, scope))
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import json
|
||||
from asyncio import sleep
|
||||
from secrets import token_urlsafe
|
||||
from uuid import UUID
|
||||
|
||||
from redis.asyncio import Redis
|
||||
@@ -8,11 +10,12 @@ from redis.exceptions import RedisError
|
||||
class HistoryCache:
|
||||
"""Кэш только первой страницы истории; PostgreSQL остаётся источником истины."""
|
||||
|
||||
def __init__(self, url: str | None, ttl_seconds: int) -> None:
|
||||
def __init__(self, url: str | None, ttl_seconds: int, stampede_protection: bool) -> None:
|
||||
self._url = url
|
||||
self._ttl_seconds = ttl_seconds
|
||||
self._client: Redis | None = None
|
||||
self._disabled = not bool(url)
|
||||
self._stampede_protection = stampede_protection
|
||||
|
||||
async def connect(self) -> None:
|
||||
if self._disabled:
|
||||
@@ -23,7 +26,7 @@ class HistoryCache:
|
||||
decode_responses=True,
|
||||
socket_connect_timeout=1,
|
||||
socket_timeout=1,
|
||||
max_connections=10,
|
||||
max_connections=50,
|
||||
)
|
||||
await client.ping()
|
||||
self._client = client
|
||||
@@ -38,6 +41,10 @@ class HistoryCache:
|
||||
def _key(chat_id: UUID, version: int, limit: int) -> str:
|
||||
return f"history:first:{chat_id}:v{version}:l{limit}"
|
||||
|
||||
@classmethod
|
||||
def _lock_key(cls, chat_id: UUID, version: int, limit: int) -> str:
|
||||
return cls._key(chat_id, version, limit) + ":lock"
|
||||
|
||||
async def get(self, chat_id: UUID, version: int, limit: int) -> list[dict] | None:
|
||||
if self._disabled or self._client is None:
|
||||
return None
|
||||
@@ -58,3 +65,43 @@ class HistoryCache:
|
||||
)
|
||||
except RedisError:
|
||||
self._disabled = True
|
||||
|
||||
async def acquire_load_lock(
|
||||
self, chat_id: UUID, version: int, limit: int
|
||||
) -> str | None:
|
||||
if self._disabled or self._client is None or not self._stampede_protection:
|
||||
return None
|
||||
token = token_urlsafe(16)
|
||||
try:
|
||||
locked = await self._client.set(
|
||||
self._lock_key(chat_id, version, limit), token, nx=True, ex=2
|
||||
)
|
||||
return token if locked else ""
|
||||
except RedisError:
|
||||
self._disabled = True
|
||||
return None
|
||||
|
||||
async def wait_for_fill(
|
||||
self, chat_id: UUID, version: int, limit: int
|
||||
) -> list[dict] | None:
|
||||
for _ in range(20):
|
||||
await sleep(0.05)
|
||||
cached = await self.get(chat_id, version, limit)
|
||||
if cached is not None or self._disabled:
|
||||
return cached
|
||||
return None
|
||||
|
||||
async def release_load_lock(
|
||||
self, chat_id: UUID, version: int, limit: int, token: str
|
||||
) -> None:
|
||||
if self._disabled or self._client is None:
|
||||
return
|
||||
try:
|
||||
await self._client.eval(
|
||||
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) end return 0",
|
||||
1,
|
||||
self._lock_key(chat_id, version, limit),
|
||||
token,
|
||||
)
|
||||
except RedisError:
|
||||
self._disabled = True
|
||||
|
||||
+17
-25
@@ -14,6 +14,7 @@ from ..errors import ApiError
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
MIGRATIONS = ROOT / "migrations"
|
||||
PASSWORD_HASHER = PasswordHasher()
|
||||
DB_ERRORS = (asyncpg.PostgresError, asyncpg.InterfaceError, OSError)
|
||||
|
||||
|
||||
class PostgresStore:
|
||||
@@ -26,7 +27,7 @@ class PostgresStore:
|
||||
self._pool = await asyncpg.create_pool(
|
||||
self._database_url, min_size=1, max_size=10, command_timeout=5
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def close(self) -> None:
|
||||
@@ -87,7 +88,7 @@ class PostgresStore:
|
||||
async def ready(self) -> None:
|
||||
try:
|
||||
await self.pool.fetchval("SELECT 1")
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
|
||||
async def create_user(self, username: str, display_name: str, password: str) -> dict:
|
||||
@@ -101,26 +102,17 @@ class PostgresStore:
|
||||
)
|
||||
except asyncpg.UniqueViolationError as error:
|
||||
raise ApiError(409, "conflict", "Username уже занят") from error
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return self._user(row)
|
||||
|
||||
async def get_user(self, user_id: UUID) -> dict | None:
|
||||
try:
|
||||
row = await self.pool.fetchrow(
|
||||
"SELECT id, username, display_name, created_at FROM users WHERE id = $1", user_id
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return self._user(row) if row else None
|
||||
|
||||
async def authenticate(self, username: str, password: str) -> dict | None:
|
||||
try:
|
||||
row = await self.pool.fetchrow(
|
||||
"SELECT id, username, display_name, created_at, password_hash FROM users WHERE username = $1",
|
||||
username,
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
if row is None:
|
||||
return None
|
||||
@@ -140,7 +132,7 @@ class PostgresStore:
|
||||
user["id"],
|
||||
ttl_seconds,
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return {"token": token, "token_type": "Bearer", "expires_at": row["expires_at"], "user": user}
|
||||
|
||||
@@ -151,7 +143,7 @@ class PostgresStore:
|
||||
"SELECT u.id, u.username, u.display_name, u.created_at FROM sessions s JOIN users u ON u.id = s.user_id WHERE s.token_hash = $1 AND s.expires_at > now()",
|
||||
token_hash,
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return self._user(row) if row else None
|
||||
|
||||
@@ -160,7 +152,7 @@ class PostgresStore:
|
||||
await self.pool.execute(
|
||||
"DELETE FROM sessions WHERE token_hash = $1", sha256(token.encode("ascii")).hexdigest()
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
|
||||
async def create_chat(self, creator_id: UUID, title: str, member_ids: list[UUID]) -> dict:
|
||||
@@ -182,7 +174,7 @@ class PostgresStore:
|
||||
"INSERT INTO chat_members (chat_id, user_id) VALUES ($1, $2)",
|
||||
[(row["id"], member_id) for member_id in member_ids],
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return {
|
||||
"id": row["id"],
|
||||
@@ -198,7 +190,7 @@ class PostgresStore:
|
||||
chat_id,
|
||||
user_id,
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
if row is None:
|
||||
raise ApiError(404, "not_found", "Чат не найден")
|
||||
@@ -216,7 +208,7 @@ class PostgresStore:
|
||||
args.append(limit + 1)
|
||||
try:
|
||||
rows = await self.pool.fetch(query, *args)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return [self._chat(row) for row in rows]
|
||||
|
||||
@@ -237,13 +229,13 @@ class PostgresStore:
|
||||
sender_id,
|
||||
text,
|
||||
)
|
||||
version = await connection.fetchval(
|
||||
"UPDATE chats SET history_version = history_version + 1 WHERE id = $1 RETURNING history_version",
|
||||
await connection.execute(
|
||||
"UPDATE chats SET history_version = history_version + 1 WHERE id = $1",
|
||||
chat_id,
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return self._message(row), version
|
||||
return self._message(row)
|
||||
|
||||
async def history_version_for_member(self, chat_id: UUID, user_id: UUID) -> int:
|
||||
try:
|
||||
@@ -252,7 +244,7 @@ class PostgresStore:
|
||||
chat_id,
|
||||
user_id,
|
||||
)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
if version is None:
|
||||
raise ApiError(404, "not_found", "Чат не найден")
|
||||
@@ -271,6 +263,6 @@ class PostgresStore:
|
||||
args.append(limit + 1)
|
||||
try:
|
||||
rows = await self.pool.fetch(query, *args)
|
||||
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return [self._message(row) for row in rows]
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Доказывает корректное чтение после restart Redis."""
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
|
||||
from client import Client, check, run # noqa: E402
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
client = Client()
|
||||
alice, _bob, _eve, chat = client.bootstrap()
|
||||
message = client.message(alice, chat, "survives Redis restart")["json"]
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages?limit=100"
|
||||
before = client.call("GET", path, actor=alice, schema="MessagePage")["json"]
|
||||
check(before["items"] == [message], "initial cache fill failed")
|
||||
subprocess.run(["docker", "compose", "restart", "redis"], cwd=ROOT, check=True)
|
||||
time.sleep(1)
|
||||
after = client.call("GET", path, actor=alice, schema="MessagePage")["json"]
|
||||
check(after["items"] == [message], "history changed after Redis restart")
|
||||
print("PASS: history remains correct after Redis restart")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Одновременное чтение холодной первой страницы истории."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
import json
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
|
||||
from client import Client, check, run # noqa: E402
|
||||
|
||||
|
||||
def main() -> None:
|
||||
concurrency = 20
|
||||
client = Client()
|
||||
alice, _bob, _eve, chat = client.bootstrap()
|
||||
created = client.message(alice, chat, "cold-cache message")["json"]
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages?limit=1"
|
||||
barrier = threading.Barrier(concurrency)
|
||||
|
||||
def read_page(_worker: int) -> float:
|
||||
local = Client()
|
||||
barrier.wait()
|
||||
started = time.monotonic()
|
||||
response = local.call("GET", path, actor=alice, schema="MessagePage")["json"]
|
||||
check(response["items"] == [created], "cold-page response differs")
|
||||
return time.monotonic() - started
|
||||
|
||||
with ThreadPoolExecutor(max_workers=concurrency) as pool:
|
||||
latencies = list(pool.map(read_page, range(concurrency)))
|
||||
ordered = sorted(latencies)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"chat_id": chat["id"],
|
||||
"concurrency": concurrency,
|
||||
"successful_reads": len(latencies),
|
||||
"latency_ms": {
|
||||
"p50": round(1000 * ordered[(concurrency - 1) // 2], 3),
|
||||
"p95": round(1000 * ordered[18], 3),
|
||||
"max": round(1000 * ordered[-1], 3),
|
||||
},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
Reference in New Issue
Block a user