add code for level 5
Skeleton contracts / contracts (pull_request) Canceled after 0s

This commit is contained in:
Матвей
2026-10-08 19:48:18 +03:00
parent 13fa322ca2
commit 744e941d55
11 changed files with 384 additions and 114 deletions
+1
View File
@@ -5,6 +5,7 @@ APP_PORT=8080
APP_ENV=lab
SESSION_TTL_SECONDS=3600
CACHE_TTL_SECONDS=60
CACHE_STAMPEDE_PROTECTION=true
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
+47 -4
View File
@@ -1,9 +1,9 @@
# Отчёт — лабораторная 2
- Автор, группа: TODO
- Целевая оценка: 3
- Автор, группа: Козырев Матвей БВТ2301
- Целевая оценка: 5; трек 1 — согласованность кэша
- Commit/PR: TODO после публикации
- Использованные библиотеки: FastAPI 0.141.1, Uvicorn 0.40.0, asyncpg 0.31.0, argon2-cffi 25.1.0; зафиксированы в `pyproject.toml` и `uv.lock`.
- Использованные библиотеки: FastAPI 0.141.1, Uvicorn 0.40.0, asyncpg 0.31.0, argon2-cffi 25.1.0, redis 5.0.8; зафиксированы в `pyproject.toml` и `uv.lock`.
## Запуск из чистого clone
@@ -62,6 +62,7 @@ docker compose down
```text
HTTP-клиент → FastAPI API → asyncpg pool (1..10 соединений) → PostgreSQL
└→ миграции SQL при старте
└→ Redis cache истории (pool до 50 соединений)
```
PostgreSQL — источник истины для `users`, `chats`, `chat_members`,
@@ -81,6 +82,13 @@ Logout удаляет текущую серверную сессию. Срок
История выбирается keyset-запросом по `(created_at, id)`; индекс
`messages_chat_created_id_idx` соответствует этому запросу.
Redis кэширует только первую страницу истории. Ключ включает `chat_id`,
`limit` и `history_version` из PostgreSQL. В той же транзакции, что и
вставка сообщения, PostgreSQL увеличивает `history_version`; старый ключ
поэтому больше не читается. Redis не источник истины: при его отказе кэш
отключается в процессе API, а чтение продолжает PostgreSQL. Redis использует
AOF и named volume `redis_data`.
## Доказательства критериев
| Критерий | Команда | Наблюдаемый результат | Статус |
@@ -94,10 +102,45 @@ Logout удаляет текущую серверную сессию. Срок
| Гонка и rollback | `make test-grade4` | Ровно одна регистрация успешна; неуспешный чат не виден | `evidence/grade4-checks-2026-10-08.txt` |
| TTL и зависимости | `make test-grade4`; TTL=2 | Expired session → 401; Redis fallback проходит smoke; DB readiness → 503 | `evidence/grade4-checks-2026-10-08.txt` |
| Индекс истории | `./scripts/explain_history.sh` | Index Scan, 50 строк, shared hit=3 | `evidence/explain-history-2026-10-08.txt` |
| Трек 5: stampede | `student-tests/cache_stampede.py` | 20 SQL-загрузок → 1 при 20 читателях | `evidence/track1-cache-stampede-2026-10-08.txt` |
| Трек 5: restart cache | `student-tests/cache_recovery.py` | История не изменилась после restart Redis | `evidence/track1-cache-stampede-2026-10-08.txt` |
## Трек 5 — согласованность кэша
**Гипотеза.** При холодном ключе 20 одновременных запросов первой страницы
истории создают 20 SQL-загрузок без координации. Redis-lock `SET NX EX 2`
оставит одну загрузку; остальные запросы ждут заполнения ключа до 1 секунды.
**Методика.** Для каждой точки API запускался с одним и тем же Compose-стендом:
PostgreSQL 17.6, Redis 7.4.2, Python 3.12.12, Docker Engine 29.8.0, 10 CPU и
16 GiB RAM. `student-tests/cache_stampede.py` создаёт новый чат с одним
сообщением и одновременно запускает 20 GET первой страницы. SQL-загрузка
считается по записи `history_sql_page_load` в журнале API. Baseline запускает
API с `CACHE_STAMPEDE_PROTECTION=false`; затем тот же сценарий повторяется с
`true`. Данные, чат и ключ каждый раз новые.
| Точка | Успешных чтений | SQL-загрузок | p50 / p95 / max latency |
| --- | ---: | ---: | --- |
| Без координации | 20 | 20 | 111.620 / 113.364 / 113.603 ms |
| Redis-lock | 20 | 1 | 157.831 / 159.477 / 159.549 ms |
Число SQL-загрузок уменьшилось на `(20 − 1) / 20 × 100 = 95%`. В этой
единственной точке p50 вырос на `157.831 − 111.620 = 46.211 ms`, поскольку
ожидающие запросы опрашивают кэш с интервалом 50 ms. Это не оценка
производительности системы в целом: измерение одноразовое, closed-loop,
включает HTTP-клиент и не отделяет влияние Docker Desktop.
read-after-write проверяет `student-tests/lab4_checks.py`: страница сначала
кэшируется, затем создаётся сообщение, и следующий GET содержит оба сообщения.
`student-tests/cache_recovery.py` заполняет кэш, перезапускает Redis и
проверяет неизменность ответа. Raw results и команды сохранены в evidence.
## Ограничения и следующие шаги
Redis/Valkey не используется: он относится к уровню 4 этой лабораторной.
Результат трека относится к одному локальному прогону с 20 читателями; для
обобщения нужны повторы, нагрузка других размеров и измерение CPU/RAM
контейнеров. Redis-lock не заменяет PostgreSQL: при недоступности Redis API
безопасно обходит кэш, а при недоступности PostgreSQL readiness возвращает 503.
Публичные smoke и persistence выполнены на локальном Compose-стенде 2026-10-08.
При фиксации работы в PR сохраните их stdout в `evidence/` без токенов и
паролей.
+3
View File
@@ -26,6 +26,7 @@ services:
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
CACHE_TTL_SECONDS: ${CACHE_TTL_SECONDS:-60}
CACHE_STAMPEDE_PROTECTION: ${CACHE_STAMPEDE_PROTECTION:-true}
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
secrets:
- database_url
@@ -33,6 +34,8 @@ services:
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
ports:
- "8080:8080"
healthcheck:
@@ -0,0 +1,25 @@
Date: 2026-10-08
Track: 1 — cache consistency
Environment: Docker Engine 29.8.0; PostgreSQL 17.6; Redis 7.4.2;
Python 3.12.12; 10 CPU; 16 GiB RAM.
Method:
student-tests/cache_stampede.py creates a new chat with one message and
launches 20 concurrent GET requests for its cold first history page.
The API logs one history_sql_page_load per PostgreSQL page query.
Baseline: CACHE_STAMPEDE_PROTECTION=false
{"chat_id":"5aa10886-61e0-47f2-82d2-c09a05bbd45f","concurrency":20,"successful_reads":20,"latency_ms":{"p50":111.620,"p95":113.364,"max":113.603},"sql_page_loads":20}
After: CACHE_STAMPEDE_PROTECTION=true
{"chat_id":"96fd0c67-907f-4681-8ec5-6fcbfc4d8fc7","concurrency":20,"successful_reads":20,"latency_ms":{"p50":157.831,"p95":159.477,"max":159.549},"sql_page_loads":1}
Reduction: (20 - 1) / 20 * 100 = 95%.
Cache restart:
Command: python3 student-tests/cache_recovery.py
PASS: history remains correct after Redis restart
Read-after-write:
Command: python3 student-tests/lab4_checks.py
PASS: registration race, chat rollback and cache read-after-write
+41 -38
View File
@@ -13,6 +13,7 @@ _CONFIGURATION_KEYS = {
"CACHE_URL",
"CACHE_URL_FILE",
"CACHE_TTL_SECONDS",
"CACHE_STAMPEDE_PROTECTION",
"SESSION_TTL_SECONDS",
}
@@ -28,6 +29,7 @@ class Settings:
session_ttl_seconds: int
cache_url: str | None
cache_ttl_seconds: int
cache_stampede_protection: bool
def load_settings(env_file: Path = ROOT / ".env") -> Settings:
@@ -46,54 +48,55 @@ def load_settings(env_file: Path = ROOT / ".env") -> Settings:
value = value[1:-1]
os.environ.setdefault(key, value)
host = os.environ.get("APP_HOST", "0.0.0.0").strip()
environment = os.environ.get("APP_ENV", "lab").strip()
def value(name: str, default: str = "") -> str:
return os.environ.get(name, default).strip()
def number(name: str, default: int) -> int:
try:
result = int(value(name, str(default)))
except ValueError as error:
raise ValueError(f"{name} должен быть целым числом") from error
if result < 1:
raise ValueError(f"{name} должен быть положительным")
return result
def url(name: str, file_name: str, prefixes: tuple[str, ...]) -> str:
result = value(name)
path = value(file_name)
if path:
try:
result = Path(path).read_text(encoding="utf-8").strip()
except OSError as error:
raise ValueError(f"{file_name} недоступен") from error
if result and not result.startswith(prefixes):
raise ValueError(f"{name} содержит неверный URL")
return result
host = value("APP_HOST", "0.0.0.0")
environment = value("APP_ENV", "lab")
if not host:
raise ValueError("APP_HOST не может быть пустым")
if not environment:
raise ValueError("APP_ENV не может быть пустым")
try:
port = int(os.environ.get("APP_PORT", "8080"))
except ValueError as error:
raise ValueError("APP_PORT должен быть целым числом") from error
if not 1 <= port <= 65535:
port = number("APP_PORT", 8080)
if port > 65535:
raise ValueError("APP_PORT должен находиться в диапазоне 1..65535")
database_url_file = os.environ.get("DATABASE_URL_FILE", "").strip()
database_url = os.environ.get("DATABASE_URL", "").strip()
if database_url_file:
try:
database_url = Path(database_url_file).read_text(encoding="utf-8").strip()
except OSError as error:
raise ValueError("DATABASE_URL_FILE недоступен") from error
if not database_url.startswith(("postgresql://", "postgres://")):
database_url = url(
"DATABASE_URL", "DATABASE_URL_FILE", ("postgresql://", "postgres://")
)
if not database_url:
raise ValueError("Нужен PostgreSQL DATABASE_URL или DATABASE_URL_FILE")
try:
session_ttl_seconds = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
except ValueError as error:
raise ValueError("SESSION_TTL_SECONDS должен быть целым числом") from error
if session_ttl_seconds < 1:
raise ValueError("SESSION_TTL_SECONDS должен быть положительным")
cache_url_file = os.environ.get("CACHE_URL_FILE", "").strip()
cache_url = os.environ.get("CACHE_URL", "").strip()
if cache_url_file:
try:
cache_url = Path(cache_url_file).read_text(encoding="utf-8").strip()
except OSError as error:
raise ValueError("CACHE_URL_FILE недоступен") from error
if cache_url and not cache_url.startswith(("redis://", "rediss://")):
raise ValueError("CACHE_URL должен содержать Redis URL")
try:
cache_ttl_seconds = int(os.environ.get("CACHE_TTL_SECONDS", "60"))
except ValueError as error:
raise ValueError("CACHE_TTL_SECONDS должен быть целым числом") from error
if cache_ttl_seconds < 1:
raise ValueError("CACHE_TTL_SECONDS должен быть положительным")
cache_url = url("CACHE_URL", "CACHE_URL_FILE", ("redis://", "rediss://"))
raw_protection = value("CACHE_STAMPEDE_PROTECTION", "true").lower()
if raw_protection not in {"true", "false"}:
raise ValueError("CACHE_STAMPEDE_PROTECTION должен быть true или false")
return Settings(
host=host,
port=port,
environment=environment,
database_url=database_url,
session_ttl_seconds=session_ttl_seconds,
session_ttl_seconds=number("SESSION_TTL_SECONDS", 3600),
cache_url=cache_url or None,
cache_ttl_seconds=cache_ttl_seconds,
cache_ttl_seconds=number("CACHE_TTL_SECONDS", 60),
cache_stampede_protection=raw_protection == "true",
)
+18 -9
View File
@@ -17,15 +17,20 @@ from .storage.history_cache import HistoryCache
from .storage.postgres import PostgresStore
access_logger = logging.getLogger("messenger.access")
if not access_logger.handlers:
def logger(name: str) -> logging.Logger:
result = logging.getLogger(name)
if result.handlers:
return result
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(
logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")
)
access_logger.addHandler(handler)
access_logger.setLevel(logging.INFO)
access_logger.propagate = False
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s"))
result.addHandler(handler)
result.setLevel(logging.INFO)
result.propagate = False
return result
access_logger = logger("messenger.access")
logger("messenger.cache")
@asynccontextmanager
@@ -36,7 +41,11 @@ async def lifespan(application: FastAPI):
await store.migrate()
application.state.store = store
application.state.settings = settings
cache = HistoryCache(settings.cache_url, settings.cache_ttl_seconds)
cache = HistoryCache(
settings.cache_url,
settings.cache_ttl_seconds,
settings.cache_stampede_protection,
)
await cache.connect()
application.state.history_cache = cache
try:
+102 -36
View File
@@ -1,4 +1,5 @@
import base64
import logging
from datetime import datetime
from uuid import UUID
@@ -6,13 +7,24 @@ from fastapi import APIRouter, Depends, Header, Query, Request, Response
from .cursor import CursorError, decode_cursor, encode_cursor
from .errors import ApiError
from .models import Chat, ChatPageResponse, CreateChatRequest, CreateMessageRequest, CreateUserRequest, Message, MessagePageResponse, Session, User
from .storage.postgres import PostgresStore
from .storage.history_cache import HistoryCache
from .storage.postgres import PostgresStore
from .models import (
Chat,
ChatPageResponse,
CreateChatRequest,
CreateMessageRequest,
CreateUserRequest,
Message,
MessagePageResponse,
Session,
User,
)
router = APIRouter()
BEARER_CHALLENGE = {'WWW-Authenticate': 'Bearer realm="messenger"'}
BASIC_CHALLENGE = {'WWW-Authenticate': 'Basic realm="messenger", charset="UTF-8"'}
cache_logger = logging.getLogger("messenger.cache")
BEARER_CHALLENGE = {"WWW-Authenticate": 'Bearer realm="messenger"'}
BASIC_CHALLENGE = {"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'}
def get_store(request: Request) -> PostgresStore:
return request.app.state.store
@@ -20,7 +32,11 @@ def get_store(request: Request) -> PostgresStore:
def get_history_cache(request: Request) -> HistoryCache:
return request.app.state.history_cache
async def get_current_user(authorization: str | None = Header(default=None), store: PostgresStore = Depends(get_store)) -> dict:
async def get_current_user(
authorization: str | None = Header(default=None),
store: PostgresStore = Depends(get_store),
) -> dict:
if authorization is None or not authorization.startswith("Bearer "):
raise ApiError(401, "unauthorized", "Требуется Bearer-сессия", BEARER_CHALLENGE)
token = authorization[7:]
@@ -45,6 +61,23 @@ def parse_basic(authorization: str | None) -> tuple[str, str]:
raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE)
return username, password
def cursor_position(cursor: str | None, scope: str) -> tuple[datetime, UUID] | None:
if cursor is None:
return None
try:
created_at, item_id = decode_cursor(cursor, scope)
return created_at, UUID(item_id)
except (CursorError, ValueError) as error:
raise ApiError(400, "invalid_request", "Некорректный cursor") from error
def next_page_cursor(rows: list[dict], limit: int, scope: str) -> str | None:
if len(rows) <= limit:
return None
last = rows[limit - 1]
return encode_cursor(scope, last["created_at"], str(last["id"]))
@router.get("/health/live")
async def liveness() -> dict[str, str]:
return {"status": "ok"}
@@ -55,11 +88,17 @@ async def readiness(store: PostgresStore = Depends(get_store)) -> dict[str, str]
return {"status": "ok"}
@router.post("/api/v1/users", response_model=User, status_code=201)
async def register_user(payload: CreateUserRequest, store: PostgresStore = Depends(get_store)) -> dict:
async def register_user(
payload: CreateUserRequest, store: PostgresStore = Depends(get_store)
) -> dict:
return await store.create_user(payload.username, payload.display_name, payload.password)
@router.post("/api/v1/auth/sessions", response_model=Session, status_code=201)
async def create_session(request: Request, authorization: str | None = Header(default=None), store: PostgresStore = Depends(get_store)) -> dict:
async def create_session(
request: Request,
authorization: str | None = Header(default=None),
store: PostgresStore = Depends(get_store),
) -> dict:
if await request.body():
raise ApiError(400, "invalid_request", "Тело запроса не требуется")
username, password = parse_basic(authorization)
@@ -69,7 +108,11 @@ async def create_session(request: Request, authorization: str | None = Header(de
return await store.create_session(user, request.app.state.settings.session_ttl_seconds)
@router.delete("/api/v1/auth/sessions/current", status_code=204)
async def revoke_session(authorization: str | None = Header(default=None), store: PostgresStore = Depends(get_store), user: dict = Depends(get_current_user)) -> Response:
async def revoke_session(
authorization: str | None = Header(default=None),
store: PostgresStore = Depends(get_store),
user: dict = Depends(get_current_user),
) -> Response:
del user
await store.revoke_session(authorization[7:])
return Response(status_code=204)
@@ -79,51 +122,74 @@ async def current_user(user: dict = Depends(get_current_user)) -> dict:
return user
@router.post("/api/v1/chats", response_model=Chat, status_code=201)
async def create_chat(payload: CreateChatRequest, user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)) -> dict:
async def create_chat(
payload: CreateChatRequest,
user: dict = Depends(get_current_user),
store: PostgresStore = Depends(get_store),
) -> dict:
return await store.create_chat(user["id"], payload.title, payload.member_ids)
@router.get("/api/v1/chats", response_model=ChatPageResponse)
async def list_chats(user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store), limit: int = Query(default=50, ge=1, le=100), cursor: str | None = Query(default=None, min_length=1)) -> ChatPageResponse:
after: tuple[datetime, UUID] | None = None
async def list_chats(
user: dict = Depends(get_current_user),
store: PostgresStore = Depends(get_store),
limit: int = Query(default=50, ge=1, le=100),
cursor: str | None = Query(default=None, min_length=1),
) -> ChatPageResponse:
scope = "chats:" + str(user["id"])
if cursor is not None:
try:
created_at, item_id = decode_cursor(cursor, scope)
after = created_at, UUID(item_id)
except (CursorError, ValueError) as error:
raise ApiError(400, "invalid_request", "Некорректный cursor") from error
rows = await store.list_chats_for_member(user["id"], after, limit)
rows = await store.list_chats_for_member(user["id"], cursor_position(cursor, scope), limit)
page = rows[:limit]
next_cursor = encode_cursor(scope, page[-1]["created_at"], str(page[-1]["id"])) if len(rows) > limit else None
return ChatPageResponse(items=page, next_cursor=next_cursor)
return ChatPageResponse(items=page, next_cursor=next_page_cursor(rows, limit, scope))
@router.get("/api/v1/chats/{chat_id}", response_model=Chat)
async def get_chat(chat_id: UUID, user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)) -> dict:
async def get_chat(
chat_id: UUID,
user: dict = Depends(get_current_user),
store: PostgresStore = Depends(get_store),
) -> dict:
return await store.get_chat_for_member(chat_id, user["id"])
@router.post("/api/v1/chats/{chat_id}/messages", response_model=Message, status_code=201)
async def create_message(chat_id: UUID, payload: CreateMessageRequest, user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)) -> dict:
message, _version = await store.create_message(chat_id, user["id"], payload.text)
return message
async def create_message(
chat_id: UUID,
payload: CreateMessageRequest,
user: dict = Depends(get_current_user),
store: PostgresStore = Depends(get_store),
) -> dict:
return await store.create_message(chat_id, user["id"], payload.text)
@router.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePageResponse)
async def list_messages(chat_id: UUID, user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store), cache: HistoryCache = Depends(get_history_cache), limit: int = Query(default=50, ge=1, le=100), cursor: str | None = Query(default=None, min_length=1)) -> MessagePageResponse:
after: tuple[datetime, UUID] | None = None
async def list_messages(
chat_id: UUID,
user: dict = Depends(get_current_user),
store: PostgresStore = Depends(get_store),
cache: HistoryCache = Depends(get_history_cache),
limit: int = Query(default=50, ge=1, le=100),
cursor: str | None = Query(default=None, min_length=1),
) -> MessagePageResponse:
scope = "messages:" + str(chat_id)
if cursor is not None:
try:
created_at, item_id = decode_cursor(cursor, scope)
after = created_at, UUID(item_id)
except (CursorError, ValueError) as error:
raise ApiError(400, "invalid_request", "Некорректный cursor") from error
after = cursor_position(cursor, scope)
if cursor is None:
version = await store.history_version_for_member(chat_id, user["id"])
rows = await cache.get(chat_id, version, limit)
if rows is None:
rows = await store.list_messages_for_member(chat_id, user["id"], after, limit)
await cache.set(chat_id, version, limit, rows)
lock_token = await cache.acquire_load_lock(chat_id, version, limit)
if lock_token == "":
rows = await cache.wait_for_fill(chat_id, version, limit)
if rows is None:
cache_logger.info(
"history_sql_page_load chat_id=%s version=%s limit=%s",
chat_id,
version,
limit,
)
try:
rows = await store.list_messages_for_member(chat_id, user["id"], None, limit)
await cache.set(chat_id, version, limit, rows)
finally:
if lock_token:
await cache.release_load_lock(chat_id, version, limit, lock_token)
else:
rows = await store.list_messages_for_member(chat_id, user["id"], after, limit)
page = rows[:limit]
next_cursor = encode_cursor(scope, page[-1]["created_at"], str(page[-1]["id"])) if len(rows) > limit else None
return MessagePageResponse(items=page, next_cursor=next_cursor)
return MessagePageResponse(items=page, next_cursor=next_page_cursor(rows, limit, scope))
+49 -2
View File
@@ -1,4 +1,6 @@
import json
from asyncio import sleep
from secrets import token_urlsafe
from uuid import UUID
from redis.asyncio import Redis
@@ -8,11 +10,12 @@ from redis.exceptions import RedisError
class HistoryCache:
"""Кэш только первой страницы истории; PostgreSQL остаётся источником истины."""
def __init__(self, url: str | None, ttl_seconds: int) -> None:
def __init__(self, url: str | None, ttl_seconds: int, stampede_protection: bool) -> None:
self._url = url
self._ttl_seconds = ttl_seconds
self._client: Redis | None = None
self._disabled = not bool(url)
self._stampede_protection = stampede_protection
async def connect(self) -> None:
if self._disabled:
@@ -23,7 +26,7 @@ class HistoryCache:
decode_responses=True,
socket_connect_timeout=1,
socket_timeout=1,
max_connections=10,
max_connections=50,
)
await client.ping()
self._client = client
@@ -38,6 +41,10 @@ class HistoryCache:
def _key(chat_id: UUID, version: int, limit: int) -> str:
return f"history:first:{chat_id}:v{version}:l{limit}"
@classmethod
def _lock_key(cls, chat_id: UUID, version: int, limit: int) -> str:
return cls._key(chat_id, version, limit) + ":lock"
async def get(self, chat_id: UUID, version: int, limit: int) -> list[dict] | None:
if self._disabled or self._client is None:
return None
@@ -58,3 +65,43 @@ class HistoryCache:
)
except RedisError:
self._disabled = True
async def acquire_load_lock(
self, chat_id: UUID, version: int, limit: int
) -> str | None:
if self._disabled or self._client is None or not self._stampede_protection:
return None
token = token_urlsafe(16)
try:
locked = await self._client.set(
self._lock_key(chat_id, version, limit), token, nx=True, ex=2
)
return token if locked else ""
except RedisError:
self._disabled = True
return None
async def wait_for_fill(
self, chat_id: UUID, version: int, limit: int
) -> list[dict] | None:
for _ in range(20):
await sleep(0.05)
cached = await self.get(chat_id, version, limit)
if cached is not None or self._disabled:
return cached
return None
async def release_load_lock(
self, chat_id: UUID, version: int, limit: int, token: str
) -> None:
if self._disabled or self._client is None:
return
try:
await self._client.eval(
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) end return 0",
1,
self._lock_key(chat_id, version, limit),
token,
)
except RedisError:
self._disabled = True
+17 -25
View File
@@ -14,6 +14,7 @@ from ..errors import ApiError
ROOT = Path(__file__).resolve().parents[2]
MIGRATIONS = ROOT / "migrations"
PASSWORD_HASHER = PasswordHasher()
DB_ERRORS = (asyncpg.PostgresError, asyncpg.InterfaceError, OSError)
class PostgresStore:
@@ -26,7 +27,7 @@ class PostgresStore:
self._pool = await asyncpg.create_pool(
self._database_url, min_size=1, max_size=10, command_timeout=5
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise RuntimeError("PostgreSQL is unavailable") from error
async def close(self) -> None:
@@ -87,7 +88,7 @@ class PostgresStore:
async def ready(self) -> None:
try:
await self.pool.fetchval("SELECT 1")
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
async def create_user(self, username: str, display_name: str, password: str) -> dict:
@@ -101,26 +102,17 @@ class PostgresStore:
)
except asyncpg.UniqueViolationError as error:
raise ApiError(409, "conflict", "Username уже занят") from error
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
return self._user(row)
async def get_user(self, user_id: UUID) -> dict | None:
try:
row = await self.pool.fetchrow(
"SELECT id, username, display_name, created_at FROM users WHERE id = $1", user_id
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
return self._user(row) if row else None
async def authenticate(self, username: str, password: str) -> dict | None:
try:
row = await self.pool.fetchrow(
"SELECT id, username, display_name, created_at, password_hash FROM users WHERE username = $1",
username,
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
if row is None:
return None
@@ -140,7 +132,7 @@ class PostgresStore:
user["id"],
ttl_seconds,
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
return {"token": token, "token_type": "Bearer", "expires_at": row["expires_at"], "user": user}
@@ -151,7 +143,7 @@ class PostgresStore:
"SELECT u.id, u.username, u.display_name, u.created_at FROM sessions s JOIN users u ON u.id = s.user_id WHERE s.token_hash = $1 AND s.expires_at > now()",
token_hash,
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
return self._user(row) if row else None
@@ -160,7 +152,7 @@ class PostgresStore:
await self.pool.execute(
"DELETE FROM sessions WHERE token_hash = $1", sha256(token.encode("ascii")).hexdigest()
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
async def create_chat(self, creator_id: UUID, title: str, member_ids: list[UUID]) -> dict:
@@ -182,7 +174,7 @@ class PostgresStore:
"INSERT INTO chat_members (chat_id, user_id) VALUES ($1, $2)",
[(row["id"], member_id) for member_id in member_ids],
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
return {
"id": row["id"],
@@ -198,7 +190,7 @@ class PostgresStore:
chat_id,
user_id,
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
if row is None:
raise ApiError(404, "not_found", "Чат не найден")
@@ -216,7 +208,7 @@ class PostgresStore:
args.append(limit + 1)
try:
rows = await self.pool.fetch(query, *args)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
return [self._chat(row) for row in rows]
@@ -237,13 +229,13 @@ class PostgresStore:
sender_id,
text,
)
version = await connection.fetchval(
"UPDATE chats SET history_version = history_version + 1 WHERE id = $1 RETURNING history_version",
await connection.execute(
"UPDATE chats SET history_version = history_version + 1 WHERE id = $1",
chat_id,
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
return self._message(row), version
return self._message(row)
async def history_version_for_member(self, chat_id: UUID, user_id: UUID) -> int:
try:
@@ -252,7 +244,7 @@ class PostgresStore:
chat_id,
user_id,
)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
if version is None:
raise ApiError(404, "not_found", "Чат не найден")
@@ -271,6 +263,6 @@ class PostgresStore:
args.append(limit + 1)
try:
rows = await self.pool.fetch(query, *args)
except (asyncpg.PostgresError, asyncpg.InterfaceError, OSError) as error:
except DB_ERRORS as error:
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
return [self._message(row) for row in rows]
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env python3
"""Доказывает корректное чтение после restart Redis."""
from pathlib import Path
import subprocess
import sys
import time
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
from client import Client, check, run # noqa: E402
ROOT = Path(__file__).resolve().parents[1]
def main() -> None:
client = Client()
alice, _bob, _eve, chat = client.bootstrap()
message = client.message(alice, chat, "survives Redis restart")["json"]
path = "/api/v1/chats/" + chat["id"] + "/messages?limit=100"
before = client.call("GET", path, actor=alice, schema="MessagePage")["json"]
check(before["items"] == [message], "initial cache fill failed")
subprocess.run(["docker", "compose", "restart", "redis"], cwd=ROOT, check=True)
time.sleep(1)
after = client.call("GET", path, actor=alice, schema="MessagePage")["json"]
check(after["items"] == [message], "history changed after Redis restart")
print("PASS: history remains correct after Redis restart")
if __name__ == "__main__":
run(main)
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env python3
"""Одновременное чтение холодной первой страницы истории."""
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import json
import sys
import threading
import time
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
from client import Client, check, run # noqa: E402
def main() -> None:
concurrency = 20
client = Client()
alice, _bob, _eve, chat = client.bootstrap()
created = client.message(alice, chat, "cold-cache message")["json"]
path = "/api/v1/chats/" + chat["id"] + "/messages?limit=1"
barrier = threading.Barrier(concurrency)
def read_page(_worker: int) -> float:
local = Client()
barrier.wait()
started = time.monotonic()
response = local.call("GET", path, actor=alice, schema="MessagePage")["json"]
check(response["items"] == [created], "cold-page response differs")
return time.monotonic() - started
with ThreadPoolExecutor(max_workers=concurrency) as pool:
latencies = list(pool.map(read_page, range(concurrency)))
ordered = sorted(latencies)
print(
json.dumps(
{
"chat_id": chat["id"],
"concurrency": concurrency,
"successful_reads": len(latencies),
"latency_ms": {
"p50": round(1000 * ordered[(concurrency - 1) // 2], 3),
"p95": round(1000 * ordered[18], 3),
"max": round(1000 * ordered[-1], 3),
},
},
ensure_ascii=False,
)
)
if __name__ == "__main__":
run(main)