forked from mtusi2026/mtusi2026lab2
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b538b2db39 | ||
|
|
7cda433085 |
@@ -2,7 +2,7 @@
|
||||
|
||||
- Автор, группа: Козырев Матвей БВТ2301
|
||||
- Целевая оценка: 5; трек 1 — согласованность кэша
|
||||
- Commit/PR: TODO после публикации
|
||||
- PR: https://gitea.privcloud.tech/mtusi2026/mtusi2026lab2/pulls/22
|
||||
- Использованные библиотеки: FastAPI 0.141.1, Uvicorn 0.40.0, asyncpg 0.31.0, argon2-cffi 25.1.0, redis 5.0.8; зафиксированы в `pyproject.toml` и `uv.lock`.
|
||||
|
||||
## Запуск из чистого clone
|
||||
@@ -13,10 +13,11 @@
|
||||
```sh
|
||||
mkdir -p secrets
|
||||
openssl rand -hex 32 -out secrets/postgres_password
|
||||
awk '{printf "postgresql://messenger:%s@postgres:5432/messenger\\n", $0}' \
|
||||
awk '{printf "postgresql://messenger:%s@postgres:5432/messenger\n", $0}' \
|
||||
secrets/postgres_password > secrets/database_url
|
||||
printf '%s\n' 'redis://redis:6379/0' > secrets/cache_url
|
||||
chmod 700 secrets
|
||||
chmod 600 secrets/postgres_password secrets/database_url
|
||||
chmod 600 secrets/postgres_password secrets/database_url secrets/cache_url
|
||||
docker compose --env-file .env.example config --quiet
|
||||
docker compose --env-file .env.example up -d --build
|
||||
```
|
||||
@@ -86,7 +87,9 @@ Redis кэширует только первую страницу истории
|
||||
`limit` и `history_version` из PostgreSQL. В той же транзакции, что и
|
||||
вставка сообщения, PostgreSQL увеличивает `history_version`; старый ключ
|
||||
поэтому больше не читается. Redis не источник истины: при его отказе кэш
|
||||
отключается в процессе API, а чтение продолжает PostgreSQL. Redis использует
|
||||
временно обходится, а чтение продолжает PostgreSQL. API повторно проверяет
|
||||
Redis не чаще одного раза в секунду; после успешного `PING` заново создаёт
|
||||
клиент, и кэш с Redis-lock снова используются без restart API. Redis использует
|
||||
AOF и named volume `redis_data`.
|
||||
|
||||
## Доказательства критериев
|
||||
@@ -98,12 +101,12 @@ AOF и named volume `redis_data`.
|
||||
| Согласованность lock-файла | `uv lock --check` | PASS | выполнено |
|
||||
| Публичный smoke | `make test` | `PASS: lab2 public API scenario` | `evidence/smoke-2026-10-08.txt` |
|
||||
| Persistence | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS: persistence preserved confirmed data and existing session` | `evidence/persistence-2026-10-08.txt` |
|
||||
| Redis cache | `make test-grade4`; `redis-cli --scan` | read-after-write без stale read; создан ключ `history:first:…` | `evidence/redis-cache-2026-10-08.txt` |
|
||||
| Redis cache и курсор | `make test-grade4`; `redis-cli --scan` | read-after-write без stale read; повторный GET кэшированной страницы и её continuation возвращают 200 | `evidence/cache-pagination-2026-10-09.txt` |
|
||||
| Гонка и rollback | `make test-grade4` | Ровно одна регистрация успешна; неуспешный чат не виден | `evidence/grade4-checks-2026-10-08.txt` |
|
||||
| TTL и зависимости | `make test-grade4`; TTL=2 | Expired session → 401; Redis fallback проходит smoke; DB readiness → 503 | `evidence/grade4-checks-2026-10-08.txt` |
|
||||
| Индекс истории | `./scripts/explain_history.sh` | Index Scan, 50 строк, shared hit=3 | `evidence/explain-history-2026-10-08.txt` |
|
||||
| Трек 5: stampede | `student-tests/cache_stampede.py` | 20 SQL-загрузок → 1 при 20 читателях | `evidence/track1-cache-stampede-2026-10-08.txt` |
|
||||
| Трек 5: restart cache | `student-tests/cache_recovery.py` | История не изменилась после restart Redis | `evidence/track1-cache-stampede-2026-10-08.txt` |
|
||||
| Трек 5: отказ и восстановление cache | `student-tests/cache_recovery.py` | PostgreSQL fallback, затем кэш и lock восстановлены без restart API | `evidence/redis-recovery-2026-10-09.txt` |
|
||||
|
||||
## Трек 5 — согласованность кэша
|
||||
|
||||
@@ -132,8 +135,11 @@ API с `CACHE_STAMPEDE_PROTECTION=false`; затем тот же сценари
|
||||
|
||||
read-after-write проверяет `student-tests/lab4_checks.py`: страница сначала
|
||||
кэшируется, затем создаётся сообщение, и следующий GET содержит оба сообщения.
|
||||
`student-tests/cache_recovery.py` заполняет кэш, перезапускает Redis и
|
||||
проверяет неизменность ответа. Raw results и команды сохранены в evidence.
|
||||
`student-tests/cache_recovery.py` сначала останавливает Redis и подтверждает
|
||||
корректный PostgreSQL fallback. Затем Redis запускается обратно без restart API:
|
||||
тест проверяет создание нового ключа кэша и ровно одну SQL-загрузку для 20
|
||||
одновременных запросов новой версии истории. Raw results и команды сохранены в
|
||||
evidence.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
PASS: registration race, chat rollback and cache read-after-write
|
||||
@@ -0,0 +1 @@
|
||||
PASS: Redis cache and stampede lock recover without API restart
|
||||
@@ -1,5 +1,7 @@
|
||||
import json
|
||||
from asyncio import sleep
|
||||
from asyncio import Lock, sleep
|
||||
from datetime import UTC, datetime
|
||||
from time import monotonic
|
||||
from secrets import token_urlsafe
|
||||
from uuid import UUID
|
||||
|
||||
@@ -14,28 +16,55 @@ class HistoryCache:
|
||||
self._url = url
|
||||
self._ttl_seconds = ttl_seconds
|
||||
self._client: Redis | None = None
|
||||
self._disabled = not bool(url)
|
||||
self._reconnect_lock = Lock()
|
||||
self._next_reconnect_at = 0.0
|
||||
self._stampede_protection = stampede_protection
|
||||
|
||||
async def connect(self) -> None:
|
||||
if self._disabled:
|
||||
return
|
||||
try:
|
||||
client = Redis.from_url(
|
||||
self._url,
|
||||
decode_responses=True,
|
||||
socket_connect_timeout=1,
|
||||
socket_timeout=1,
|
||||
max_connections=50,
|
||||
)
|
||||
await client.ping()
|
||||
self._client = client
|
||||
except RedisError:
|
||||
self._disabled = True
|
||||
await self._client_or_none()
|
||||
|
||||
async def _client_or_none(self) -> Redis | None:
|
||||
"""Возвращает рабочий клиент или временно обходит кэш при отказе Redis."""
|
||||
if not self._url:
|
||||
return None
|
||||
if self._client is not None:
|
||||
return self._client
|
||||
if monotonic() < self._next_reconnect_at:
|
||||
return None
|
||||
|
||||
async with self._reconnect_lock:
|
||||
if self._client is not None:
|
||||
return self._client
|
||||
if monotonic() < self._next_reconnect_at:
|
||||
return None
|
||||
try:
|
||||
client = Redis.from_url(
|
||||
self._url,
|
||||
decode_responses=True,
|
||||
socket_connect_timeout=1,
|
||||
socket_timeout=1,
|
||||
max_connections=50,
|
||||
)
|
||||
await client.ping()
|
||||
self._client = client
|
||||
return client
|
||||
except RedisError:
|
||||
self._next_reconnect_at = monotonic() + 1
|
||||
return None
|
||||
|
||||
async def _mark_unavailable(self, client: Redis) -> None:
|
||||
"""Отбрасывает сломанное соединение и разрешает повторное подключение."""
|
||||
async with self._reconnect_lock:
|
||||
if self._client is client:
|
||||
self._client = None
|
||||
self._next_reconnect_at = monotonic() + 1
|
||||
await client.aclose()
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._client is not None:
|
||||
await self._client.aclose()
|
||||
if self._client is None:
|
||||
return
|
||||
await self._client.aclose()
|
||||
self._client = None
|
||||
|
||||
@staticmethod
|
||||
def _key(chat_id: UUID, version: int, limit: int) -> str:
|
||||
@@ -45,40 +74,54 @@ class HistoryCache:
|
||||
def _lock_key(cls, chat_id: UUID, version: int, limit: int) -> str:
|
||||
return cls._key(chat_id, version, limit) + ":lock"
|
||||
|
||||
@staticmethod
|
||||
def _decode_rows(value: str) -> list[dict]:
|
||||
rows = json.loads(value)
|
||||
for row in rows:
|
||||
created_at = row.get("created_at")
|
||||
if isinstance(created_at, str):
|
||||
row["created_at"] = datetime.fromisoformat(created_at).astimezone(UTC)
|
||||
return rows
|
||||
|
||||
async def get(self, chat_id: UUID, version: int, limit: int) -> list[dict] | None:
|
||||
if self._disabled or self._client is None:
|
||||
client = await self._client_or_none()
|
||||
if client is None:
|
||||
return None
|
||||
try:
|
||||
value = await self._client.get(self._key(chat_id, version, limit))
|
||||
return json.loads(value) if value else None
|
||||
except (RedisError, ValueError, TypeError):
|
||||
self._disabled = True
|
||||
value = await client.get(self._key(chat_id, version, limit))
|
||||
return self._decode_rows(value) if value else None
|
||||
except RedisError:
|
||||
await self._mark_unavailable(client)
|
||||
return None
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
async def set(self, chat_id: UUID, version: int, limit: int, items: list[dict]) -> None:
|
||||
if self._disabled or self._client is None:
|
||||
client = await self._client_or_none()
|
||||
if client is None:
|
||||
return
|
||||
try:
|
||||
encoded = json.dumps(items, default=str, separators=(",", ":"))
|
||||
await self._client.set(
|
||||
await client.set(
|
||||
self._key(chat_id, version, limit), encoded, ex=self._ttl_seconds
|
||||
)
|
||||
except RedisError:
|
||||
self._disabled = True
|
||||
await self._mark_unavailable(client)
|
||||
|
||||
async def acquire_load_lock(
|
||||
self, chat_id: UUID, version: int, limit: int
|
||||
) -> str | None:
|
||||
if self._disabled or self._client is None or not self._stampede_protection:
|
||||
client = await self._client_or_none()
|
||||
if client is None or not self._stampede_protection:
|
||||
return None
|
||||
token = token_urlsafe(16)
|
||||
try:
|
||||
locked = await self._client.set(
|
||||
locked = await client.set(
|
||||
self._lock_key(chat_id, version, limit), token, nx=True, ex=2
|
||||
)
|
||||
return token if locked else ""
|
||||
except RedisError:
|
||||
self._disabled = True
|
||||
await self._mark_unavailable(client)
|
||||
return None
|
||||
|
||||
async def wait_for_fill(
|
||||
@@ -87,21 +130,22 @@ class HistoryCache:
|
||||
for _ in range(20):
|
||||
await sleep(0.05)
|
||||
cached = await self.get(chat_id, version, limit)
|
||||
if cached is not None or self._disabled:
|
||||
if cached is not None or self._client is None:
|
||||
return cached
|
||||
return None
|
||||
|
||||
async def release_load_lock(
|
||||
self, chat_id: UUID, version: int, limit: int, token: str
|
||||
) -> None:
|
||||
if self._disabled or self._client is None:
|
||||
client = await self._client_or_none()
|
||||
if client is None:
|
||||
return
|
||||
try:
|
||||
await self._client.eval(
|
||||
await client.eval(
|
||||
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) end return 0",
|
||||
1,
|
||||
self._lock_key(chat_id, version, limit),
|
||||
token,
|
||||
)
|
||||
except RedisError:
|
||||
self._disabled = True
|
||||
await self._mark_unavailable(client)
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Доказывает корректное чтение после restart Redis."""
|
||||
"""Проверяет fallback и восстановление Redis-кэша без restart API."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
|
||||
@@ -19,11 +22,47 @@ def main() -> None:
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages?limit=100"
|
||||
before = client.call("GET", path, actor=alice, schema="MessagePage")["json"]
|
||||
check(before["items"] == [message], "initial cache fill failed")
|
||||
subprocess.run(["docker", "compose", "restart", "redis"], cwd=ROOT, check=True)
|
||||
time.sleep(1)
|
||||
|
||||
subprocess.run(["docker", "compose", "stop", "redis"], cwd=ROOT, check=True)
|
||||
fallback = client.call("GET", path, actor=alice, schema="MessagePage")["json"]
|
||||
check(fallback["items"] == [message], "PostgreSQL fallback returned a different history")
|
||||
|
||||
subprocess.run(["docker", "compose", "start", "redis"], cwd=ROOT, check=True)
|
||||
time.sleep(1.1) # Интервал повторного подключения к Redis в приложении.
|
||||
after = client.call("GET", path, actor=alice, schema="MessagePage")["json"]
|
||||
check(after["items"] == [message], "history changed after Redis restart")
|
||||
print("PASS: history remains correct after Redis restart")
|
||||
|
||||
keys = subprocess.check_output(
|
||||
[
|
||||
"docker", "compose", "exec", "-T", "redis", "redis-cli", "--scan",
|
||||
"--pattern", "history:first:" + chat["id"] + ":*",
|
||||
],
|
||||
cwd=ROOT,
|
||||
text=True,
|
||||
)
|
||||
check("history:first:" + chat["id"] in keys, "cache was not restored after Redis restart")
|
||||
|
||||
second = client.message(alice, chat, "cold page after Redis recovery")["json"]
|
||||
started_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
workers = 20
|
||||
barrier = threading.Barrier(workers)
|
||||
|
||||
def read_cold_page(_worker: int) -> None:
|
||||
barrier.wait()
|
||||
response = Client().call("GET", path, actor=alice, schema="MessagePage")["json"]
|
||||
check(response["items"] == [message, second], "concurrent history differs")
|
||||
|
||||
with ThreadPoolExecutor(max_workers=workers) as pool:
|
||||
list(pool.map(read_cold_page, range(workers)))
|
||||
|
||||
logs = subprocess.check_output(
|
||||
["docker", "compose", "logs", "--since", started_at, "api"],
|
||||
cwd=ROOT,
|
||||
text=True,
|
||||
)
|
||||
marker = "history_sql_page_load chat_id=" + chat["id"] + " version=2 limit=100"
|
||||
check(logs.count(marker) == 1, "Redis lock was not restored after Redis restart")
|
||||
print("PASS: Redis cache and stampede lock recover without API restart")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -59,6 +59,22 @@ def main() -> None:
|
||||
{message["id"] for message in history} == {first["id"], second["id"]},
|
||||
"history cache returned stale data after write",
|
||||
)
|
||||
|
||||
first_page_path = "/api/v1/chats/" + chat["id"] + "/messages?limit=1"
|
||||
first_page = client.call(
|
||||
"GET", first_page_path, actor=alice, schema="MessagePage"
|
||||
)["json"]
|
||||
cached_page = client.call(
|
||||
"GET", first_page_path, actor=alice, schema="MessagePage"
|
||||
)["json"]
|
||||
check(first_page == cached_page, "cached page differs from the first response")
|
||||
check(first_page["next_cursor"], "first page has no continuation cursor")
|
||||
client.call(
|
||||
"GET",
|
||||
first_page_path + "&cursor=" + first_page["next_cursor"],
|
||||
actor=alice,
|
||||
schema="MessagePage",
|
||||
)
|
||||
print("PASS: registration race, chat rollback and cache read-after-write")
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user