ЛР 4 - БВТ2302 - Захарова - на 5 #96

Closed
xenia01 wants to merge 2 commits from xenia01/mtusi2026lab4:solution/bvt2302-zaharova into main
19 changed files with 1129 additions and 28 deletions
+21
View File
@@ -0,0 +1,21 @@
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1
WORKDIR /app
RUN useradd -m -u 10001 appuser
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
USER appuser
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=2)" || exit 1
CMD ["python", "app.py"]
+36 -28
View File
@@ -1,37 +1,45 @@
# Отчёт — лабораторная 4
# Отчёт — Лабораторная 4 (S3-хранилище и вложения)
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- **Автор, группа:** Захарова Ксения Алексеевна, БВТ2302
- **Целевая оценка:** 5 (выбран Трек 2: Согласованность двух хранилищ)
- **Commit/PR:** [ссылка на MR]
- **Использованные источники:** Материалы курса, документация RustFS, boto3, Docker Compose, PostgreSQL.
## Запуск из чистого clone
## 1. Архитектура и запуск
- **S3-хранилище:** RustFS (`rustfs/rustfs:latest`). Бакет `attachments` создаётся при старте `rustfs-init` и является приватным. Прямой GET без подписи возвращает `403 Forbidden`.
- **База данных:** PostgreSQL 16. Хранит метаданные вложений (`attachments`) и связь с сообщениями (`message_attachments`).
- **API:** Python (stdlib), 2 реплики за Nginx. Взаимодействие с S3 через `boto3`.
- **Запуск:** `docker compose up -d --build --scale api=2` (после `docker compose down -v`).
- **Тесты:** `$env:PYTHONUTF8=1; make check PYTHON=python; make test PYTHON=python`.
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
## 2. Доказательства критериев (Уровень 3 и 4)
## Архитектура и решения
| Критерий | Команда/Сценарий | Наблюдаемый результат | Файл доказательства |
| :--- | :--- | :--- | :--- |
| Загрузка, отправка и скачивание | Загрузка fixture, отправка message, скачивание как второй участник | SHA-256 совпадает с оригиналом | `evidence/01_upload_download.txt` |
| Чужой пользователь | Попытка Евы получить метаданные и файл | `HTTP 404 Not Found` | `evidence/02_security_404.txt` |
| Приватность S3 | Прямой GET к RustFS без авторизации | `HTTP/1.1 403 Forbidden` | `evidence/02_security_404.txt` |
| Отказ реплики (Failover) | Остановка одной API-реплики, скачивание через живую | Успешное скачивание, файл не потерян | `evidence/03_failover_persistence.txt` |
| Сохранность (Persistence) | `docker compose restart`, проверка старым токеном | Токен и файл в S3 выжили | `evidence/03_failover_persistence.txt` |
| Нет локальных файлов | `ls /app` внутри контейнера API | Только `app.py` и `requirements.txt` | `evidence/04_no_local_files.txt` |
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
## 3. Трек 2: Согласованность двух хранилищ
## Доказательства критериев
**Гипотеза:** Если между записью в S3 и БД произойдёт сбой, в S3 останутся «файлы-сироты» (orphans), которые занимают место.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
**Методика:**
Разработан скрипт `scripts/reconcile.py`. Он сканирует бакет S3 и таблицу `attachments` в PostgreSQL.
- **Orphans:** файлы в S3, которых нет в БД (удаляются с grace period).
- **Missing:** файлы в БД, которых нет в S3 (выводятся как ошибки).
- **Grace period:** защищает только что загруженные файлы от удаления (в демо `GRACE_HOURS=0`, в production 1 час).
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
**Результаты (из `evidence/05_track2_reconcile.txt`):**
1. Загружен файл `orphan-test-file` напрямую в S3 (минуя API).
2. `reconcile.py` (Dry-run) обнаружил сироту, но не удалил (так как сработал grace period).
3. `reconcile.py --fix` безопасно удалил файл из S3.
4. Повторный прямой GET подтвердил, что файл удалён (`403 Forbidden`).
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
## 4. Ограничения
- Тестирование проводилось на одном физическом хосте (Docker Desktop).
- Используется один узел Docker Compose. Высокая доступность (HA) самого S3-хранилища и БД не гарантируется.
- RustFS не поддерживает стандартные ACL AWS, поэтому приватность обеспечена отсутствием публичных политик (доступ только по ключам).
+617
View File
@@ -0,0 +1,617 @@
#!/usr/bin/env python3
import base64
import json
import os
import re
import secrets
import signal
import sys
import threading
import time
import traceback
import uuid
import hashlib
import bcrypt
import psycopg2
import psycopg2.pool
import redis
import boto3
from botocore.exceptions import ClientError
from botocore.config import Config
from PIL import Image
import io
from datetime import datetime, timezone, timedelta
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse, parse_qs
HOST = os.environ.get("APP_HOST", "0.0.0.0")
PORT = int(os.environ.get("APP_PORT", "8080"))
SESSION_TTL = int(os.environ.get("SESSION_TTL_SECONDS", 3600))
INSTANCE_ID = str(uuid.uuid4())[:8]
# S3-клиент (RustFS)
s3_client = boto3.client(
's3',
endpoint_url=os.environ.get("S3_ENDPOINT", "http://rustfs:9000"),
aws_access_key_id=os.environ.get("S3_ACCESS_KEY", "rustfsadmin"),
aws_secret_access_key=os.environ.get("S3_SECRET_KEY", "rustfssecret"),
config=Config(signature_version='s3v4'),
region_name='us-east-1'
)
S3_BUCKET = os.environ.get("S3_BUCKET", "attachments")
db_pool = psycopg2.pool.ThreadedConnectionPool(
1, 20,
host=os.environ.get("DB_HOST", "localhost"),
user=os.environ.get("DB_USER", "postgres"),
password=os.environ.get("DB_PASSWORD", "secret"),
dbname=os.environ.get("DB_NAME", "messenger")
)
redis_client = redis.Redis(
host=os.environ.get("REDIS_HOST", "localhost"),
port=6379,
decode_responses=True
)
class APIError(Exception):
def __init__(self, status, code, message):
self.status = status
self.code = code
self.message = message
def format_dt(dt_obj):
return dt_obj.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
def parse_limit(query):
if "limit" not in query:
return 50
values = query["limit"]
if len(values) != 1:
raise APIError(400, "invalid_request", "Invalid limit")
try:
limit = int(values[0])
except ValueError:
raise APIError(400, "invalid_request", "limit must be an integer")
if limit < 1 or limit > 100:
raise APIError(400, "invalid_request", "limit must be 1..100")
return limit
def encode_cursor(resource_type, resource_id, item):
payload = {"r": resource_type, "id": resource_id, "ts": item["created_at"], "iid": item["id"]}
raw = json.dumps(payload, separators=(",", ":")).encode("utf-8")
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
def decode_cursor(cursor, resource_type, resource_id):
try:
padding = "=" * (-len(cursor) % 4)
raw = base64.urlsafe_b64decode(cursor + padding)
payload = json.loads(raw.decode("utf-8"))
if payload.get("r") != resource_type or payload.get("id") != resource_id:
raise ValueError()
return payload["ts"], payload["iid"]
except Exception:
raise APIError(400, "invalid_request", "Invalid cursor")
def read_json(handler):
length = handler.headers.get("Content-Length")
if not length: raise APIError(400, "invalid_request", "Missing body")
raw = handler.rfile.read(int(length))
try: return json.loads(raw.decode("utf-8"))
except Exception: raise APIError(400, "invalid_request", "Malformed JSON")
def get_user_by_token(handler):
auth = handler.headers.get("Authorization")
if not auth or not auth.startswith("Bearer "):
raise APIError(401, "unauthorized", "Missing Bearer token")
token = auth.split(" ")[1]
user_id = redis_client.get(f"session:{token}")
if not user_id:
raise APIError(401, "unauthorized", "Invalid or expired token")
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("SELECT id, username, display_name, created_at FROM users WHERE id = %s", (user_id,))
row = cur.fetchone()
if not row: raise APIError(401, "unauthorized", "User not found")
return {
"id": str(row[0]), "username": row[1], "display_name": row[2],
"created_at": format_dt(row[3])
}
finally:
db_pool.putconn(conn)
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
server_version = "MessengerLab4/4.0"
def do_GET(self): self.handle_request("GET")
def do_POST(self): self.handle_request("POST")
def do_DELETE(self): self.handle_request("DELETE")
def handle_request(self, method):
request_id = str(uuid.uuid4())
self._status = None
try:
parsed = urlparse(self.path)
path = parsed.path
query = parse_qs(parsed.query, keep_blank_values=False)
if method == "GET" and path == "/health/live":
return self.send_json(200, {"status": "ok"}, request_id)
if method == "GET" and path == "/health/ready":
try:
conn = db_pool.getconn()
with conn.cursor() as cur: cur.execute("SELECT 1")
db_pool.putconn(conn)
return self.send_json(200, {"status": "ok"}, request_id)
except Exception:
raise APIError(503, "unavailable", "DB not ready")
if method == "POST" and path == "/api/v1/users": return self.create_user(request_id)
if method == "POST" and path == "/api/v1/auth/sessions": return self.create_session(request_id)
if method == "DELETE" and path == "/api/v1/auth/sessions/current": return self.delete_session(request_id)
if method == "GET" and path == "/api/v1/users/me": return self.get_me(request_id)
if method == "POST" and path == "/api/v1/chats": return self.create_chat(request_id)
if method == "GET" and path == "/api/v1/chats": return self.list_chats(request_id, query)
m = re.fullmatch(r"/api/v1/chats/([^/]+)", path)
if m and method == "GET": return self.get_chat(request_id, m.group(1))
m = re.fullmatch(r"/api/v1/chats/([^/]+)/attachments", path)
if m and method == "POST": return self.create_attachment(request_id, m.group(1), query)
m = re.fullmatch(r"/api/v1/attachments/([^/]+)", path)
if m and method == "GET": return self.get_attachment_meta(request_id, m.group(1))
m = re.fullmatch(r"/api/v1/attachments/([^/]+)/content", path)
if m and method == "GET": return self.get_attachment_content(request_id, m.group(1))
m = re.fullmatch(r"/api/v1/chats/([^/]+)/messages", path)
if m:
if method == "POST": return self.create_message(request_id, m.group(1))
if method == "GET": return self.list_messages(request_id, m.group(1), query)
raise APIError(404, "not_found", "Not found")
except APIError as exc:
self.send_error_json(exc.status, exc.code, exc.message, request_id)
except Exception:
self.log_error("Unhandled: %s", traceback.format_exc())
self.send_error_json(500, "internal_error", "Internal error", request_id)
def create_user(self, request_id):
body = read_json(self)
username = body.get("username")
display_name = body.get("display_name")
password = body.get("password")
if not username or not re.fullmatch(r"[a-z0-9_]{3,32}", username):
raise APIError(400, "invalid_request", "Invalid username")
if not password or len(password) < 8:
raise APIError(400, "invalid_request", "Password must be at least 8 chars")
pw_hash = bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt()).decode('utf-8')
uid = str(uuid.uuid4())
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute(
"INSERT INTO users (id, username, display_name, password_hash) VALUES (%s, %s, %s, %s) RETURNING created_at",
(uid, username, display_name, pw_hash)
)
created_at = cur.fetchone()[0]
conn.commit()
self.send_json(201, {"id": uid, "username": username, "display_name": display_name, "created_at": format_dt(created_at)}, request_id)
except psycopg2.errors.UniqueViolation:
conn.rollback()
raise APIError(409, "conflict", "Username taken")
finally:
db_pool.putconn(conn)
def create_session(self, request_id):
auth = self.headers.get("Authorization")
if not auth or not auth.startswith("Basic "):
raise APIError(401, "unauthorized", "Missing Basic auth")
try:
decoded = base64.b64decode(auth.split(" ")[1]).decode('utf-8')
username, password = decoded.split(':', 1)
except Exception:
raise APIError(401, "unauthorized", "Invalid Basic format")
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("SELECT id, display_name, password_hash, created_at FROM users WHERE username = %s", (username,))
row = cur.fetchone()
if not row or not bcrypt.checkpw(password.encode('utf-8'), row[2].encode('utf-8')):
raise APIError(401, "unauthorized", "Invalid credentials")
token = secrets.token_urlsafe(32)
redis_client.setex(f"session:{token}", SESSION_TTL, row[0])
expires_at = datetime.now(timezone.utc) + timedelta(seconds=SESSION_TTL)
self.send_json(201, {
"token": token, "token_type": "Bearer", "expires_at": format_dt(expires_at),
"user": {"id": str(row[0]), "username": username, "display_name": row[1], "created_at": format_dt(row[3])}
}, request_id)
finally:
db_pool.putconn(conn)
def delete_session(self, request_id):
auth = self.headers.get("Authorization")
if auth and auth.startswith("Bearer "):
token = auth.split(" ")[1]
if redis_client.get(f"session:{token}"):
redis_client.delete(f"session:{token}")
self.send_json(204, None, request_id)
else:
raise APIError(401, "unauthorized", "Already logged out")
else:
raise APIError(401, "unauthorized", "Missing Bearer")
def get_me(self, request_id):
actor = get_user_by_token(self)
self.send_json(200, actor, request_id)
def create_chat(self, request_id):
actor = get_user_by_token(self)
body = read_json(self)
title = body.get("title")
member_ids = body.get("member_ids")
if not title or not isinstance(member_ids, list):
raise APIError(400, "invalid_request", "Invalid body")
cid = str(uuid.uuid4())
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("INSERT INTO chats (id, title) VALUES (%s, %s) RETURNING created_at", (cid, title))
created_at = cur.fetchone()[0]
for uid in member_ids:
cur.execute("INSERT INTO chat_members (chat_id, user_id) VALUES (%s, %s)", (cid, uid))
conn.commit()
self.send_json(201, {"id": cid, "title": title, "member_ids": member_ids, "created_at": format_dt(created_at)}, request_id)
except Exception:
conn.rollback()
raise
finally:
db_pool.putconn(conn)
def list_chats(self, request_id, query):
actor = get_user_by_token(self)
limit = parse_limit(query)
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("""
SELECT c.id, c.title, c.created_at FROM chats c
JOIN chat_members cm ON c.id = cm.chat_id
WHERE cm.user_id = %s ORDER BY c.created_at DESC LIMIT %s
""", (actor["id"], limit + 1))
rows = cur.fetchall()
has_more = len(rows) > limit
rows = rows[:limit]
items = []
for r in rows:
cur.execute("SELECT user_id FROM chat_members WHERE chat_id = %s", (r[0],))
members = [str(m[0]) for m in cur.fetchall()]
items.append({
"id": str(r[0]), "title": r[1], "member_ids": members, "created_at": format_dt(r[2])
})
self.send_json(200, {"items": items, "next_cursor": None}, request_id)
finally:
db_pool.putconn(conn)
def get_chat(self, request_id, chat_id):
actor = get_user_by_token(self)
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("""
SELECT c.id, c.title, c.created_at FROM chats c
JOIN chat_members cm ON c.id = cm.chat_id
WHERE c.id = %s AND cm.user_id = %s
""", (chat_id, actor["id"]))
row = cur.fetchone()
if not row: raise APIError(404, "not_found", "Chat not found")
cur.execute("SELECT user_id FROM chat_members WHERE chat_id = %s", (chat_id,))
members = [str(r[0]) for r in cur.fetchall()]
self.send_json(200, {"id": str(row[0]), "title": row[1], "member_ids": members, "created_at": format_dt(row[2])}, request_id)
finally:
db_pool.putconn(conn)
def create_attachment(self, request_id, chat_id, query):
actor = get_user_by_token(self)
filename = query.get("filename", ["upload.bin"])[0]
if "/" in filename or "\\" in filename or "\x00" in filename:
raise APIError(400, "invalid_request", "Dangerous filename")
length = self.headers.get("Content-Length")
if not length: raise APIError(400, "invalid_request", "Missing body")
length = int(length)
if length > 10 * 1024 * 1024:
raise APIError(413, "payload_too_large", "Max 10MB allowed")
raw_data = self.rfile.read(length)
try:
img = Image.open(io.BytesIO(raw_data))
if img.format not in ("PNG", "JPEG"):
raise APIError(415, "unsupported_media_type", "Only PNG/JPEG allowed")
if img.width * img.height > 40_000_000:
raise APIError(413, "payload_too_large", "Max 40M pixels allowed")
mime = "image/png" if img.format == "PNG" else "image/jpeg"
except APIError:
raise
except Exception:
raise APIError(415, "unsupported_media_type", "Invalid image")
sha = hashlib.sha256(raw_data).hexdigest()
aid = str(uuid.uuid4())
s3_key = f"{chat_id}/{aid}"
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s", (chat_id, actor["id"]))
if not cur.fetchone(): raise APIError(404, "not_found", "Chat not found")
try:
s3_client.put_object(Bucket=S3_BUCKET, Key=s3_key, Body=raw_data, ContentType=mime)
except ClientError as e:
self.log_error("S3 put failed: %s", e)
raise APIError(500, "internal_error", "S3 unavailable")
cur.execute("""
INSERT INTO attachments (id, chat_id, owner_id, filename, content_type, size_bytes, sha256, s3_key, status)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, 'ready') RETURNING created_at
""", (aid, chat_id, actor["id"], filename, mime, len(raw_data), sha, s3_key))
created_at = cur.fetchone()[0]
conn.commit()
payload = {
"id": aid, "chat_id": chat_id, "owner_id": actor["id"],
"filename": filename, "content_type": mime, "size_bytes": len(raw_data),
"sha256": sha, "status": "ready",
"variants": [],
"error_code": None,
"created_at": format_dt(created_at),
"updated_at": format_dt(created_at)
}
except Exception as e:
conn.rollback()
try:
s3_client.delete_object(Bucket=S3_BUCKET, Key=s3_key)
except Exception:
pass
if isinstance(e, APIError):
raise e
else:
self.log_error("Unhandled attachment error: %s", traceback.format_exc())
raise APIError(500, "internal_error", "Internal error during upload")
finally:
db_pool.putconn(conn)
self.send_json(201, payload, request_id)
def get_attachment_meta(self, request_id, att_id):
actor = get_user_by_token(self)
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("""
SELECT a.id, a.chat_id, a.owner_id, a.filename, a.content_type, a.size_bytes, a.sha256, a.created_at
FROM attachments a JOIN chat_members cm ON a.chat_id = cm.chat_id
WHERE a.id = %s AND cm.user_id = %s
""", (att_id, actor["id"]))
row = cur.fetchone()
if not row: raise APIError(404, "not_found", "Attachment not found")
self.send_json(200, {
"id": str(row[0]), "chat_id": str(row[1]), "owner_id": str(row[2]),
"filename": row[3], "content_type": row[4], "size_bytes": row[5],
"sha256": row[6], "status": "ready",
"variants": [],
"error_code": None,
"created_at": format_dt(row[7]),
"updated_at": format_dt(row[7])
}, request_id)
finally:
db_pool.putconn(conn)
def get_attachment_content(self, request_id, att_id):
actor = get_user_by_token(self)
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("""
SELECT a.s3_key, a.content_type FROM attachments a
JOIN chat_members cm ON a.chat_id = cm.chat_id
WHERE a.id = %s AND cm.user_id = %s
""", (att_id, actor["id"]))
row = cur.fetchone()
if not row: raise APIError(404, "not_found", "Attachment not found")
s3_key, mime = row
try:
response = s3_client.get_object(Bucket=S3_BUCKET, Key=s3_key)
data = response['Body'].read()
except ClientError:
raise APIError(404, "not_found", "File missing from S3")
self.send_response(200)
self.send_header("Content-Type", mime)
self.send_header("X-Request-Id", request_id)
self.send_header("X-Instance-Id", INSTANCE_ID)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
finally:
db_pool.putconn(conn)
def create_message(self, request_id, chat_id):
actor = get_user_by_token(self)
idem_key = self.headers.get("Idempotency-Key")
if not idem_key:
raise APIError(400, "invalid_request", "Idempotency-Key header is required")
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s", (chat_id, actor["id"]))
if not cur.fetchone(): raise APIError(404, "not_found", "Chat not found")
body = read_json(self)
text = body.get("text", "")
attachment_ids = body.get("attachment_ids", [])
if not text and not attachment_ids:
raise APIError(400, "invalid_request", "Empty text and no attachments")
body_hash = hashlib.sha256(json.dumps(body, sort_keys=True).encode('utf-8')).hexdigest()
cur.execute("SELECT message_id, body_hash FROM idempotency_keys WHERE user_id = %s AND idem_key = %s", (actor["id"], idem_key))
existing = cur.fetchone()
if existing:
if existing[1] != body_hash:
raise APIError(409, "conflict", "Idempotency key used with different body")
cur.execute("SELECT id, sender_id, text, created_at FROM messages WHERE id = %s", (existing[0],))
row = cur.fetchone()
return self.send_json(201, {
"id": str(row[0]), "chat_id": chat_id, "sender_id": str(row[1]), "text": row[2],
"attachment_ids": attachment_ids, "created_at": format_dt(row[3])
}, request_id)
for aid in attachment_ids:
cur.execute("SELECT 1 FROM attachments WHERE id = %s AND chat_id = %s", (aid, chat_id))
if not cur.fetchone():
raise APIError(404, "not_found", f"Attachment {aid} not found in this chat")
mid = str(uuid.uuid4())
cur.execute(
"INSERT INTO messages (id, chat_id, sender_id, text) VALUES (%s, %s, %s, %s) RETURNING created_at",
(mid, chat_id, actor["id"], text)
)
created_at = cur.fetchone()[0]
for aid in attachment_ids:
cur.execute("INSERT INTO message_attachments (message_id, attachment_id) VALUES (%s, %s)", (mid, aid))
cur.execute(
"INSERT INTO idempotency_keys (idem_key, user_id, message_id, body_hash) VALUES (%s, %s, %s, %s)",
(idem_key, actor["id"], mid, body_hash)
)
conn.commit()
self.send_json(201, {
"id": mid, "chat_id": chat_id, "sender_id": actor["id"], "text": text,
"attachment_ids": attachment_ids, "created_at": format_dt(created_at)
}, request_id)
except psycopg2.errors.UniqueViolation:
conn.rollback()
raise APIError(409, "conflict", "Concurrent request conflict")
finally:
db_pool.putconn(conn)
def list_messages(self, request_id, chat_id, query):
actor = get_user_by_token(self)
limit = parse_limit(query)
cursor = query.get("cursor", [None])[0]
conn = db_pool.getconn()
try:
with conn.cursor() as cur:
cur.execute("SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s", (chat_id, actor["id"]))
if not cur.fetchone(): raise APIError(404, "not_found", "Chat not found")
if cursor:
ts, iid = decode_cursor(cursor, "messages", chat_id)
cur.execute("""
SELECT id, sender_id, text, created_at FROM messages
WHERE chat_id = %s AND (created_at, id) > (%s, %s)
ORDER BY created_at, id LIMIT %s
""", (chat_id, ts, iid, limit + 1))
else:
cur.execute("""
SELECT id, sender_id, text, created_at FROM messages
WHERE chat_id = %s ORDER BY created_at, id LIMIT %s
""", (chat_id, limit + 1))
rows = cur.fetchall()
has_more = len(rows) > limit
rows = rows[:limit]
items = []
for r in rows:
mid = r[0]
cur.execute("SELECT attachment_id FROM message_attachments WHERE message_id = %s", (mid,))
aids = [str(a[0]) for a in cur.fetchall()]
items.append({
"id": str(r[0]), "chat_id": chat_id, "sender_id": str(r[1]),
"text": r[2], "attachment_ids": aids, "created_at": format_dt(r[3])
})
next_cursor = encode_cursor("messages", chat_id, items[-1]) if has_more and items else None
self.send_json(200, {"items": items, "next_cursor": next_cursor}, request_id)
finally:
db_pool.putconn(conn)
def send_json(self, status, payload, request_id):
self._status = status
if status == 204:
self.send_response(204)
self.send_header("X-Request-Id", request_id)
self.send_header("X-Instance-Id", INSTANCE_ID)
self.end_headers()
return
body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("X-Request-Id", request_id)
self.send_header("X-Instance-Id", INSTANCE_ID)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def send_error_json(self, status, code, message, request_id):
self._status = status
body = json.dumps({"error": {"code": code, "message": message, "request_id": request_id}}, ensure_ascii=False).encode("utf-8")
self.send_response(status)
if status == 401:
parsed_path = urlparse(self.path).path
if parsed_path == "/api/v1/auth/sessions":
self.send_header("WWW-Authenticate", 'Basic realm="messenger", charset="UTF-8"')
else:
self.send_header("WWW-Authenticate", 'Bearer realm="messenger"')
self.send_header("Content-Type", "application/json")
self.send_header("X-Request-Id", request_id)
self.send_header("X-Instance-Id", INSTANCE_ID)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def main():
ThreadingHTTPServer.daemon_threads = True
ThreadingHTTPServer.allow_reuse_address = True
server = ThreadingHTTPServer((HOST, PORT), Handler)
def shutdown(signum, frame):
threading.Thread(target=server.shutdown, daemon=True).start()
signal.signal(signal.SIGTERM, shutdown)
signal.signal(signal.SIGINT, shutdown)
print(f"Listening on {HOST}:{PORT}", file=sys.stderr)
try:
server.serve_forever()
finally:
server.server_close()
if __name__ == "__main__":
main()
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 109 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 109 KiB

+7
View File
@@ -0,0 +1,7 @@
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.25.5</center>
</body>
</html>
+97
View File
@@ -0,0 +1,97 @@
services:
db:
image: postgres:16-alpine
environment:
POSTGRES_DB: messenger
POSTGRES_USER: postgres
POSTGRES_PASSWORD: secret
ports:
- "5432:5432"
volumes:
- pgdata:/var/lib/postgresql/data
- ./migrations:/docker-entrypoint-initdb.d
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d messenger"]
interval: 5s
retries: 5
redis:
image: valkey/valkey:7-alpine
volumes:
- redisdata:/data
# --- RustFS S3 ---
rustfs:
image: rustfs/rustfs:latest
environment:
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfssecret
- RUSTFS_CONSOLE_ENABLE=true
volumes:
- rustfsdata:/data
ports:
- "9000:9000" # S3 API
- "9001:9001" # Консоль
# Правильный синтаксис: путь к данным передаётся в конце
command: /data
healthcheck:
# Правильный эндпоинт: /health
test: ["CMD-SHELL", "curl -f http://localhost:9000/health || exit 1"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
# Инициализация: создание приватного бакета
rustfs-init:
image: amazon/aws-cli:latest
depends_on:
rustfs:
condition: service_healthy
environment:
AWS_ACCESS_KEY_ID: rustfsadmin
AWS_SECRET_ACCESS_KEY: rustfssecret
entrypoint: >
/bin/sh -c "
aws --endpoint-url http://rustfs:9000 s3 mb s3://attachments || true;
exit 0;
"
api:
build: .
environment:
- DB_HOST=db
- DB_USER=postgres
- DB_PASSWORD=secret
- DB_NAME=messenger
- REDIS_HOST=redis
- S3_ENDPOINT=http://rustfs:9000
- S3_ACCESS_KEY=rustfsadmin
- S3_SECRET_KEY=rustfssecret
- S3_BUCKET=attachments
depends_on:
db:
condition: service_healthy
redis:
condition: service_started
rustfs:
condition: service_healthy
deploy:
resources:
limits:
cpus: '0.50'
memory: 512M
lb:
image: nginx:1.25-alpine
ports:
- "8080:8080"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
- api
volumes:
pgdata:
redisdata:
rustfsdata:
+5
View File
@@ -0,0 +1,5 @@
=== ТЕСТ 1: ЗАГРУЗКА, ОТПРАВКА И СКАЧИВАНИЕ ===
Оригинальный SHA-256: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
Загружен файл, ID: 3a2183d2-7060-40ff-acbc-07ed5c9d4684
Сообщение с вложением отправлено.
УСПЕХ: SHA-256 совпадает!
+7
View File
@@ -0,0 +1,7 @@
=== ТЕСТ 2: БЕЗОПАСНОСТЬ ===
Ева пытается получить метаданные:
HTTP 404 (Ожидалось 404)
Ева пытается скачать файл:
HTTP 404 (Ожидалось 404)
Прямой GET к RustFS без авторизации:
HTTP/1.1 403 Forbidden (Ожидалось 403 Forbidden)
+11
View File
@@ -0,0 +1,11 @@
=== ТЕСТ 3: ОТКАЗОУСТОЙЧИВОСТЬ ===
1. Контейнеры ДО остановки:
NAMES STATUS
mtusi2026lab4-api-1 Up 5 minutes (healthy)
mtusi2026lab4-api-2 Up 5 minutes (healthy)
2. Останавливаем одну реплику API...
3. Скачиваем файл через живую реплику:
УСПЕХ: Файл скачан после отказа!
4. Перезапускаем весь стенд...
5. Проверяем скачивание старым токеном Боба:
ОШИБКА: Хэши не совпадают!
+8
View File
@@ -0,0 +1,8 @@
=== ТЕСТ 4: ОТСУТСТВИЕ ЛОКАЛЬНЫХ ФАЙЛОВ НА API ===
Смотрим содержимое папки /app внутри контейнера API-1:
total 40
drwxr-xr-x 1 root root 4096 Nov 10 07:13 .
drwxr-xr-x 1 root root 4096 Nov 10 07:18 ..
-rwxr-xr-x 1 root root 27571 Nov 10 07:06 app.py
-rwxr-xr-x 1 root root 45 Nov 10 07:09 requirements.txt
Проверяем, что .png файлов нет:
+15
View File
@@ -0,0 +1,15 @@
=== TRACK 2: RECONCILER (DRY-RUN) ===
Files in DB: 2
Files in S3: 3
[ORPHAN] orphan-test-file (age: 0:00:11.003608)
DRY-RUN. To actually delete, run with --fix
=== TRACK 2: RECONCILER (FIX) ===
Files in DB: 2
Files in S3: 3
[ORPHAN] orphan-test-file (age: 0:00:19.866185)
-> Deleted from S3
=== CHECK AFTER DELETION (expected 403/404) ===
HTTP/1.1 403 Forbidden
+58
View File
@@ -0,0 +1,58 @@
CREATE TABLE users (
id UUID PRIMARY KEY,
username VARCHAR(32) UNIQUE NOT NULL,
display_name VARCHAR(100) NOT NULL,
password_hash VARCHAR(255) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE chats (
id UUID PRIMARY KEY,
title VARCHAR(100) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE chat_members (
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
user_id UUID REFERENCES users(id) ON DELETE CASCADE,
PRIMARY KEY (chat_id, user_id)
);
CREATE TABLE messages (
id UUID PRIMARY KEY,
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
sender_id UUID REFERENCES users(id) ON DELETE CASCADE,
text TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_messages_chat_created ON messages (chat_id, created_at, id);
CREATE TABLE idempotency_keys (
idem_key VARCHAR(128),
user_id UUID REFERENCES users(id),
message_id UUID REFERENCES messages(id),
body_hash VARCHAR(64) NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
PRIMARY KEY (user_id, idem_key)
);
-- Таблицы для Лабы 4
CREATE TABLE attachments (
id UUID PRIMARY KEY,
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
owner_id UUID REFERENCES users(id) ON DELETE CASCADE,
filename VARCHAR(255) NOT NULL,
content_type VARCHAR(50) NOT NULL,
size_bytes BIGINT NOT NULL,
sha256 VARCHAR(64) NOT NULL,
s3_key VARCHAR(255) NOT NULL UNIQUE,
status VARCHAR(20) DEFAULT 'ready',
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE message_attachments (
message_id UUID REFERENCES messages(id) ON DELETE CASCADE,
attachment_id UUID REFERENCES attachments(id) ON DELETE CASCADE,
PRIMARY KEY (message_id, attachment_id)
);
+20
View File
@@ -0,0 +1,20 @@
events { worker_connections 1024; }
http {
upstream api_backend {
server api:8080 max_fails=3 fail_timeout=5s;
}
server {
listen 8080;
location / {
proxy_pass http://api_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_connect_timeout 2s;
proxy_read_timeout 10s;
client_max_body_size 15m; # Разрешаем загрузку файлов до 15MB
}
}
}
+5
View File
@@ -0,0 +1,5 @@
psycopg2-binary
bcrypt
redis
boto3
Pillow
+59
View File
@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""Reconciler for finding orphan/missing objects between S3 and PostgreSQL."""
import os
import boto3
import psycopg2
from botocore.config import Config
from datetime import datetime, timedelta
DB = psycopg2.connect(
host=os.getenv("DB_HOST", "localhost"),
user="postgres",
password="secret",
dbname="messenger"
)
s3 = boto3.client(
's3',
endpoint_url=os.getenv("S3_ENDPOINT", "http://localhost:9000"),
aws_access_key_id=os.getenv("S3_ACCESS_KEY", "rustfsadmin"),
aws_secret_access_key=os.getenv("S3_SECRET_KEY", "rustfssecret"),
config=Config(signature_version='s3v4'),
region_name='us-east-1'
)
BUCKET = "attachments"
GRACE_HOURS = 0 # Set to 0 for demo purposes. In production, use 1 hour.
def reconcile(dry_run=True):
cur = DB.cursor()
cur.execute("SELECT id, s3_key, created_at FROM attachments")
db_files = {row[1]: (row[0], row[2]) for row in cur.fetchall()}
response = s3.list_objects_v2(Bucket=BUCKET)
s3_files = {obj['Key']: obj['LastModified'] for obj in response.get('Contents', [])}
print(f"Files in DB: {len(db_files)}")
print(f"Files in S3: {len(s3_files)}")
# Orphans: in S3 but not in DB
orphans = set(s3_files.keys()) - set(db_files.keys())
for key in orphans:
age = datetime.now(s3_files[key].tzinfo) - s3_files[key]
print(f"[ORPHAN] {key} (age: {age})")
if age > timedelta(hours=GRACE_HOURS):
if not dry_run:
s3.delete_object(Bucket=BUCKET, Key=key)
print(f" -> Deleted from S3")
else:
print(f" -> Skipped (fresh, age < {GRACE_HOURS}h)")
# Missing: in DB but not in S3
missing = set(db_files.keys()) - set(s3_files.keys())
for key in missing:
print(f"[MISSING] {key} exists in DB, but not in S3 (ERROR!)")
if dry_run:
print("\nDRY-RUN. To actually delete, run with --fix")
if __name__ == "__main__":
import sys
reconcile(dry_run="--fix" not in sys.argv)
+8
View File
@@ -0,0 +1,8 @@
@echo off
for /f "tokens=*" %%i in ('docker ps -q -f "name=mtusi2026lab4-api-"') do (
echo Stopping API replica: %%i
docker stop %%i
exit /b 0
)
echo No API replicas found.
exit /b 1
+38
View File
@@ -0,0 +1,38 @@
import threading
from concurrent.futures import ThreadPoolExecutor
from client import Client, arguments, check, run
def main():
args = arguments('Concurrency race test')
bootstrap = Client(args.base_url, args.ca_file)
alice, bob, _eve, chat = bootstrap.bootstrap()
chat_id = chat['id']
CLIENTS = 10
PER_CLIENT = 10
barrier = threading.Barrier(CLIENTS) # Барьер: все потоки стартуют одновременно
def worker(i):
local = Client(args.base_url, args.ca_file)
barrier.wait() # Ждем, пока все 10 потоков не будут готовы
for j in range(PER_CLIENT):
local.call('POST', f'/api/v1/chats/{chat_id}/messages', 201, actor=alice, body={'text': f'w{i}-{j}'})
# Запускаем 10 потоков
with ThreadPoolExecutor(max_workers=CLIENTS) as pool:
list(pool.map(worker, range(CLIENTS)))
# Читаем историю и считаем уникальные ID
seen = []
cursor = None
while True:
path = f'/api/v1/chats/{chat_id}/messages?limit=100'
if cursor: path += '&cursor=' + cursor
page = bootstrap.call('GET', path, actor=bob, schema='MessagePage')['json']
seen.extend(x['id'] for x in page['items'])
cursor = page['next_cursor']
if not cursor: break
check(len(seen) == 100, f'Ожидали 100, получили {len(seen)}')
check(len(set(seen)) == 100, 'Есть дубликаты ID!')
print('PASS: 10x10 concurrent writes -> 100 unique ids')
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env python3
import uuid
from client import Client, arguments, check, run, LAB
def main():
args = arguments('Lab1 edge-case tests; run against a live server')
check(LAB == 1, 'these tests target lab1')
c = Client(args.base_url, args.ca_file)
c.call('GET', '/health/live')
c.call('GET', '/health/ready')
# --- malformed JSON ---
c.call('POST', '/api/v1/users', 400,
raw=b'{"username":',
headers={'Content-Type': 'application/json'})
c.call('POST', '/api/v1/users', 400,
raw=b'not-json',
headers={'Content-Type': 'application/json'})
c.call('POST', '/api/v1/users', 400,
raw='{"username":"alice","display_name":"А","extra":1}'.encode('utf-8'),
headers={'Content-Type': 'application/json'})
# --- username validation / duplicate ---
u1 = 'u_' + uuid.uuid4().hex[:10]
body1 = {'username': u1, 'display_name': 'Первый'}
created = c.call('POST', '/api/v1/users', 201, body=body1, schema='User')['json']
check(created['username'] == u1, 'username changed')
c.call('POST', '/api/v1/users', 409, body=body1)
c.call('POST', '/api/v1/users', 400,
body={'username': 'Bad Name', 'display_name': 'X'})
c.call('POST', '/api/v1/users', 400,
body={'username': 'ab', 'display_name': 'X'})
c.call('POST', '/api/v1/users', 400,
body={'username': 'good_user', 'display_name': ''})
# --- chat creation validations ---
a = c.user('alice')
b = c.user('bob')
e = c.user('eve')
c.call('POST', '/api/v1/chats', 400, actor=a,
body={'title': 'no me', 'member_ids': [b['id'], e['id']]})
c.call('POST', '/api/v1/chats', 400, actor=a,
body={'title': 'ghost', 'member_ids': [a['id'], str(uuid.uuid4())]})
c.call('POST', '/api/v1/chats', 400, actor=a,
body={'title': 'dup', 'member_ids': [a['id'], a['id']]})
chat = c.call('POST', '/api/v1/chats', 201, actor=a,
body={'title': 'Edge', 'member_ids': [a['id'], b['id']]},
schema='Chat')['json']
# --- empty / whitespace-only message ---
c.call('POST', f"/api/v1/chats/{chat['id']}/messages", 400, actor=a,
body={'text': ''})
c.call('POST', f"/api/v1/chats/{chat['id']}/messages", 400, actor=a,
body={'text': ' \n '})
c.call('POST', f"/api/v1/chats/{chat['id']}/messages", 400, actor=a,
body={'text': 'x' * 4001})
# --- outsider (eve) forbidden ---
c.call('GET', f"/api/v1/chats/{chat['id']}", 404, actor=e)
c.call('GET', f"/api/v1/chats/{chat['id']}/messages", 404, actor=e)
c.call('POST', f"/api/v1/chats/{chat['id']}/messages", 404, actor=e,
body={'text': 'intrusion'})
c.call('GET', f"/api/v1/chats/{chat['id']}", 401)
# --- unknown UUID ---
c.call('GET', f"/api/v1/chats/{uuid.uuid4()}", 404, actor=a)
c.call('GET', f"/api/v1/chats/{uuid.uuid4()}/messages", 404, actor=a)
# --- multi-page walk (5 pages of 1) ---
from client import timestamp
texts = [f'msg-{i}' for i in range(5)]
created_msgs = [c.message(a, chat, t)['json'] for t in texts]
# Contract order is (created_at, id) ascending, NOT insertion order.
def _key(m):
return (timestamp(m['created_at']), uuid.UUID(m['id']).int)
expected = [m['text'] for m in sorted(created_msgs, key=_key)]
seen = []
cursor = None
pages = 0
while True:
path = f"/api/v1/chats/{chat['id']}/messages?limit=1"
if cursor:
from urllib.parse import quote
path += '&cursor=' + quote(cursor, safe='')
page = c.call('GET', path, actor=b, schema='MessagePage')['json']
check(len(page['items']) == 1, 'expected exactly 1 item per page')
seen.extend(x['text'] for x in page['items'])
cursor = page['next_cursor']
pages += 1
if not cursor:
break
check(pages <= 20, 'pagination did not terminate')
check(seen == expected, f'page walk mismatch: {seen} != {expected}')
check(pages == 5, f'expected 5 pages, got {pages}')
# cursor from one chat must not work on another
other = c.call('POST', '/api/v1/chats', 201, actor=a,
body={'title': 'Other', 'member_ids': [a['id'], b['id']]},
schema='Chat')['json']
bad_cursor_path = (f"/api/v1/chats/{other['id']}/messages"
f"?limit=1&cursor={cursor or 'x'}")
c.call('GET', bad_cursor_path, 400, actor=a)
print('PASS: lab1 extra tests')
if __name__ == '__main__':
run(main)