forked from mtusi2026/mtusi2026lab4
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f63e61067e | ||
|
|
4583ff3628 |
+21
@@ -0,0 +1,21 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN useradd -m -u 10001 appuser
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY app.py .
|
||||
|
||||
USER appuser
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=10s --timeout=3s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=2)" || exit 1
|
||||
|
||||
CMD ["python", "app.py"]
|
||||
@@ -1,37 +1,45 @@
|
||||
# Отчёт — лабораторная 4
|
||||
# Отчёт — Лабораторная 4 (S3-хранилище и вложения)
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- **Автор, группа:** Захарова Ксения Алексеевна, БВТ2302
|
||||
- **Целевая оценка:** 5 (выбран Трек 2: Согласованность двух хранилищ)
|
||||
- **Commit/PR:** [ссылка на MR]
|
||||
- **Использованные источники:** Материалы курса, документация RustFS, boto3, Docker Compose, PostgreSQL.
|
||||
|
||||
## Запуск из чистого clone
|
||||
## 1. Архитектура и запуск
|
||||
- **S3-хранилище:** RustFS (`rustfs/rustfs:latest`). Бакет `attachments` создаётся при старте `rustfs-init` и является приватным. Прямой GET без подписи возвращает `403 Forbidden`.
|
||||
- **База данных:** PostgreSQL 16. Хранит метаданные вложений (`attachments`) и связь с сообщениями (`message_attachments`).
|
||||
- **API:** Python (stdlib), 2 реплики за Nginx. Взаимодействие с S3 через `boto3`.
|
||||
- **Запуск:** `docker compose up -d --build --scale api=2` (после `docker compose down -v`).
|
||||
- **Тесты:** `$env:PYTHONUTF8=1; make check PYTHON=python; make test PYTHON=python`.
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
## 2. Доказательства критериев (Уровень 3 и 4)
|
||||
|
||||
## Архитектура и решения
|
||||
| Критерий | Команда/Сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| Загрузка, отправка и скачивание | Загрузка fixture, отправка message, скачивание как второй участник | SHA-256 совпадает с оригиналом | `evidence/01_upload_download.txt` |
|
||||
| Чужой пользователь | Попытка Евы получить метаданные и файл | `HTTP 404 Not Found` | `evidence/02_security_404.txt` |
|
||||
| Приватность S3 | Прямой GET к RustFS без авторизации | `HTTP/1.1 403 Forbidden` | `evidence/02_security_404.txt` |
|
||||
| Отказ реплики (Failover) | Остановка одной API-реплики, скачивание через живую | Успешное скачивание, файл не потерян | `evidence/03_failover_persistence.txt` |
|
||||
| Сохранность (Persistence) | `docker compose restart`, проверка старым токеном | Токен и файл в S3 выжили | `evidence/03_failover_persistence.txt` |
|
||||
| Нет локальных файлов | `ls /app` внутри контейнера API | Только `app.py` и `requirements.txt` | `evidence/04_no_local_files.txt` |
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
## 3. Трек 2: Согласованность двух хранилищ
|
||||
|
||||
## Доказательства критериев
|
||||
**Гипотеза:** Если между записью в S3 и БД произойдёт сбой, в S3 останутся «файлы-сироты» (orphans), которые занимают место.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
**Методика:**
|
||||
Разработан скрипт `scripts/reconcile.py`. Он сканирует бакет S3 и таблицу `attachments` в PostgreSQL.
|
||||
- **Orphans:** файлы в S3, которых нет в БД (удаляются с grace period).
|
||||
- **Missing:** файлы в БД, которых нет в S3 (выводятся как ошибки).
|
||||
- **Grace period:** защищает только что загруженные файлы от удаления (в демо `GRACE_HOURS=0`, в production 1 час).
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
**Результаты (из `evidence/05_track2_reconcile.txt`):**
|
||||
1. Загружен файл `orphan-test-file` напрямую в S3 (минуя API).
|
||||
2. `reconcile.py` (Dry-run) обнаружил сироту, но не удалил (так как сработал grace period).
|
||||
3. `reconcile.py --fix` безопасно удалил файл из S3.
|
||||
4. Повторный прямой GET подтвердил, что файл удалён (`403 Forbidden`).
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
## 4. Ограничения
|
||||
- Тестирование проводилось на одном физическом хосте (Docker Desktop).
|
||||
- Используется один узел Docker Compose. Высокая доступность (HA) самого S3-хранилища и БД не гарантируется.
|
||||
- RustFS не поддерживает стандартные ACL AWS, поэтому приватность обеспечена отсутствием публичных политик (доступ только по ключам).
|
||||
@@ -0,0 +1,617 @@
|
||||
#!/usr/bin/env python3
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import signal
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
import uuid
|
||||
import hashlib
|
||||
import bcrypt
|
||||
import psycopg2
|
||||
import psycopg2.pool
|
||||
import redis
|
||||
import boto3
|
||||
from botocore.exceptions import ClientError
|
||||
from botocore.config import Config
|
||||
from PIL import Image
|
||||
import io
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from urllib.parse import urlparse, parse_qs
|
||||
|
||||
HOST = os.environ.get("APP_HOST", "0.0.0.0")
|
||||
PORT = int(os.environ.get("APP_PORT", "8080"))
|
||||
SESSION_TTL = int(os.environ.get("SESSION_TTL_SECONDS", 3600))
|
||||
INSTANCE_ID = str(uuid.uuid4())[:8]
|
||||
|
||||
# S3-клиент (RustFS)
|
||||
s3_client = boto3.client(
|
||||
's3',
|
||||
endpoint_url=os.environ.get("S3_ENDPOINT", "http://rustfs:9000"),
|
||||
aws_access_key_id=os.environ.get("S3_ACCESS_KEY", "rustfsadmin"),
|
||||
aws_secret_access_key=os.environ.get("S3_SECRET_KEY", "rustfssecret"),
|
||||
config=Config(signature_version='s3v4'),
|
||||
region_name='us-east-1'
|
||||
)
|
||||
S3_BUCKET = os.environ.get("S3_BUCKET", "attachments")
|
||||
|
||||
db_pool = psycopg2.pool.ThreadedConnectionPool(
|
||||
1, 20,
|
||||
host=os.environ.get("DB_HOST", "localhost"),
|
||||
user=os.environ.get("DB_USER", "postgres"),
|
||||
password=os.environ.get("DB_PASSWORD", "secret"),
|
||||
dbname=os.environ.get("DB_NAME", "messenger")
|
||||
)
|
||||
|
||||
redis_client = redis.Redis(
|
||||
host=os.environ.get("REDIS_HOST", "localhost"),
|
||||
port=6379,
|
||||
decode_responses=True
|
||||
)
|
||||
|
||||
class APIError(Exception):
|
||||
def __init__(self, status, code, message):
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
|
||||
def format_dt(dt_obj):
|
||||
return dt_obj.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
|
||||
|
||||
def parse_limit(query):
|
||||
if "limit" not in query:
|
||||
return 50
|
||||
values = query["limit"]
|
||||
if len(values) != 1:
|
||||
raise APIError(400, "invalid_request", "Invalid limit")
|
||||
try:
|
||||
limit = int(values[0])
|
||||
except ValueError:
|
||||
raise APIError(400, "invalid_request", "limit must be an integer")
|
||||
if limit < 1 or limit > 100:
|
||||
raise APIError(400, "invalid_request", "limit must be 1..100")
|
||||
return limit
|
||||
|
||||
def encode_cursor(resource_type, resource_id, item):
|
||||
payload = {"r": resource_type, "id": resource_id, "ts": item["created_at"], "iid": item["id"]}
|
||||
raw = json.dumps(payload, separators=(",", ":")).encode("utf-8")
|
||||
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
|
||||
|
||||
def decode_cursor(cursor, resource_type, resource_id):
|
||||
try:
|
||||
padding = "=" * (-len(cursor) % 4)
|
||||
raw = base64.urlsafe_b64decode(cursor + padding)
|
||||
payload = json.loads(raw.decode("utf-8"))
|
||||
if payload.get("r") != resource_type or payload.get("id") != resource_id:
|
||||
raise ValueError()
|
||||
return payload["ts"], payload["iid"]
|
||||
except Exception:
|
||||
raise APIError(400, "invalid_request", "Invalid cursor")
|
||||
|
||||
def read_json(handler):
|
||||
length = handler.headers.get("Content-Length")
|
||||
if not length: raise APIError(400, "invalid_request", "Missing body")
|
||||
raw = handler.rfile.read(int(length))
|
||||
try: return json.loads(raw.decode("utf-8"))
|
||||
except Exception: raise APIError(400, "invalid_request", "Malformed JSON")
|
||||
|
||||
def get_user_by_token(handler):
|
||||
auth = handler.headers.get("Authorization")
|
||||
if not auth or not auth.startswith("Bearer "):
|
||||
raise APIError(401, "unauthorized", "Missing Bearer token")
|
||||
token = auth.split(" ")[1]
|
||||
user_id = redis_client.get(f"session:{token}")
|
||||
if not user_id:
|
||||
raise APIError(401, "unauthorized", "Invalid or expired token")
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT id, username, display_name, created_at FROM users WHERE id = %s", (user_id,))
|
||||
row = cur.fetchone()
|
||||
if not row: raise APIError(401, "unauthorized", "User not found")
|
||||
return {
|
||||
"id": str(row[0]), "username": row[1], "display_name": row[2],
|
||||
"created_at": format_dt(row[3])
|
||||
}
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
server_version = "MessengerLab4/4.0"
|
||||
|
||||
def do_GET(self): self.handle_request("GET")
|
||||
def do_POST(self): self.handle_request("POST")
|
||||
def do_DELETE(self): self.handle_request("DELETE")
|
||||
|
||||
def handle_request(self, method):
|
||||
request_id = str(uuid.uuid4())
|
||||
self._status = None
|
||||
try:
|
||||
parsed = urlparse(self.path)
|
||||
path = parsed.path
|
||||
query = parse_qs(parsed.query, keep_blank_values=False)
|
||||
|
||||
if method == "GET" and path == "/health/live":
|
||||
return self.send_json(200, {"status": "ok"}, request_id)
|
||||
if method == "GET" and path == "/health/ready":
|
||||
try:
|
||||
conn = db_pool.getconn()
|
||||
with conn.cursor() as cur: cur.execute("SELECT 1")
|
||||
db_pool.putconn(conn)
|
||||
return self.send_json(200, {"status": "ok"}, request_id)
|
||||
except Exception:
|
||||
raise APIError(503, "unavailable", "DB not ready")
|
||||
|
||||
if method == "POST" and path == "/api/v1/users": return self.create_user(request_id)
|
||||
if method == "POST" and path == "/api/v1/auth/sessions": return self.create_session(request_id)
|
||||
if method == "DELETE" and path == "/api/v1/auth/sessions/current": return self.delete_session(request_id)
|
||||
if method == "GET" and path == "/api/v1/users/me": return self.get_me(request_id)
|
||||
if method == "POST" and path == "/api/v1/chats": return self.create_chat(request_id)
|
||||
if method == "GET" and path == "/api/v1/chats": return self.list_chats(request_id, query)
|
||||
|
||||
m = re.fullmatch(r"/api/v1/chats/([^/]+)", path)
|
||||
if m and method == "GET": return self.get_chat(request_id, m.group(1))
|
||||
|
||||
m = re.fullmatch(r"/api/v1/chats/([^/]+)/attachments", path)
|
||||
if m and method == "POST": return self.create_attachment(request_id, m.group(1), query)
|
||||
|
||||
m = re.fullmatch(r"/api/v1/attachments/([^/]+)", path)
|
||||
if m and method == "GET": return self.get_attachment_meta(request_id, m.group(1))
|
||||
|
||||
m = re.fullmatch(r"/api/v1/attachments/([^/]+)/content", path)
|
||||
if m and method == "GET": return self.get_attachment_content(request_id, m.group(1))
|
||||
|
||||
m = re.fullmatch(r"/api/v1/chats/([^/]+)/messages", path)
|
||||
if m:
|
||||
if method == "POST": return self.create_message(request_id, m.group(1))
|
||||
if method == "GET": return self.list_messages(request_id, m.group(1), query)
|
||||
|
||||
raise APIError(404, "not_found", "Not found")
|
||||
except APIError as exc:
|
||||
self.send_error_json(exc.status, exc.code, exc.message, request_id)
|
||||
except Exception:
|
||||
self.log_error("Unhandled: %s", traceback.format_exc())
|
||||
self.send_error_json(500, "internal_error", "Internal error", request_id)
|
||||
|
||||
def create_user(self, request_id):
|
||||
body = read_json(self)
|
||||
username = body.get("username")
|
||||
display_name = body.get("display_name")
|
||||
password = body.get("password")
|
||||
if not username or not re.fullmatch(r"[a-z0-9_]{3,32}", username):
|
||||
raise APIError(400, "invalid_request", "Invalid username")
|
||||
if not password or len(password) < 8:
|
||||
raise APIError(400, "invalid_request", "Password must be at least 8 chars")
|
||||
|
||||
pw_hash = bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt()).decode('utf-8')
|
||||
uid = str(uuid.uuid4())
|
||||
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"INSERT INTO users (id, username, display_name, password_hash) VALUES (%s, %s, %s, %s) RETURNING created_at",
|
||||
(uid, username, display_name, pw_hash)
|
||||
)
|
||||
created_at = cur.fetchone()[0]
|
||||
conn.commit()
|
||||
self.send_json(201, {"id": uid, "username": username, "display_name": display_name, "created_at": format_dt(created_at)}, request_id)
|
||||
except psycopg2.errors.UniqueViolation:
|
||||
conn.rollback()
|
||||
raise APIError(409, "conflict", "Username taken")
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
def create_session(self, request_id):
|
||||
auth = self.headers.get("Authorization")
|
||||
if not auth or not auth.startswith("Basic "):
|
||||
raise APIError(401, "unauthorized", "Missing Basic auth")
|
||||
try:
|
||||
decoded = base64.b64decode(auth.split(" ")[1]).decode('utf-8')
|
||||
username, password = decoded.split(':', 1)
|
||||
except Exception:
|
||||
raise APIError(401, "unauthorized", "Invalid Basic format")
|
||||
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT id, display_name, password_hash, created_at FROM users WHERE username = %s", (username,))
|
||||
row = cur.fetchone()
|
||||
if not row or not bcrypt.checkpw(password.encode('utf-8'), row[2].encode('utf-8')):
|
||||
raise APIError(401, "unauthorized", "Invalid credentials")
|
||||
|
||||
token = secrets.token_urlsafe(32)
|
||||
redis_client.setex(f"session:{token}", SESSION_TTL, row[0])
|
||||
expires_at = datetime.now(timezone.utc) + timedelta(seconds=SESSION_TTL)
|
||||
|
||||
self.send_json(201, {
|
||||
"token": token, "token_type": "Bearer", "expires_at": format_dt(expires_at),
|
||||
"user": {"id": str(row[0]), "username": username, "display_name": row[1], "created_at": format_dt(row[3])}
|
||||
}, request_id)
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
def delete_session(self, request_id):
|
||||
auth = self.headers.get("Authorization")
|
||||
if auth and auth.startswith("Bearer "):
|
||||
token = auth.split(" ")[1]
|
||||
if redis_client.get(f"session:{token}"):
|
||||
redis_client.delete(f"session:{token}")
|
||||
self.send_json(204, None, request_id)
|
||||
else:
|
||||
raise APIError(401, "unauthorized", "Already logged out")
|
||||
else:
|
||||
raise APIError(401, "unauthorized", "Missing Bearer")
|
||||
|
||||
def get_me(self, request_id):
|
||||
actor = get_user_by_token(self)
|
||||
self.send_json(200, actor, request_id)
|
||||
|
||||
def create_chat(self, request_id):
|
||||
actor = get_user_by_token(self)
|
||||
body = read_json(self)
|
||||
title = body.get("title")
|
||||
member_ids = body.get("member_ids")
|
||||
if not title or not isinstance(member_ids, list):
|
||||
raise APIError(400, "invalid_request", "Invalid body")
|
||||
|
||||
cid = str(uuid.uuid4())
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("INSERT INTO chats (id, title) VALUES (%s, %s) RETURNING created_at", (cid, title))
|
||||
created_at = cur.fetchone()[0]
|
||||
for uid in member_ids:
|
||||
cur.execute("INSERT INTO chat_members (chat_id, user_id) VALUES (%s, %s)", (cid, uid))
|
||||
conn.commit()
|
||||
self.send_json(201, {"id": cid, "title": title, "member_ids": member_ids, "created_at": format_dt(created_at)}, request_id)
|
||||
except Exception:
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
def list_chats(self, request_id, query):
|
||||
actor = get_user_by_token(self)
|
||||
limit = parse_limit(query)
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("""
|
||||
SELECT c.id, c.title, c.created_at FROM chats c
|
||||
JOIN chat_members cm ON c.id = cm.chat_id
|
||||
WHERE cm.user_id = %s ORDER BY c.created_at DESC LIMIT %s
|
||||
""", (actor["id"], limit + 1))
|
||||
rows = cur.fetchall()
|
||||
has_more = len(rows) > limit
|
||||
rows = rows[:limit]
|
||||
|
||||
items = []
|
||||
for r in rows:
|
||||
cur.execute("SELECT user_id FROM chat_members WHERE chat_id = %s", (r[0],))
|
||||
members = [str(m[0]) for m in cur.fetchall()]
|
||||
items.append({
|
||||
"id": str(r[0]), "title": r[1], "member_ids": members, "created_at": format_dt(r[2])
|
||||
})
|
||||
self.send_json(200, {"items": items, "next_cursor": None}, request_id)
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
def get_chat(self, request_id, chat_id):
|
||||
actor = get_user_by_token(self)
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("""
|
||||
SELECT c.id, c.title, c.created_at FROM chats c
|
||||
JOIN chat_members cm ON c.id = cm.chat_id
|
||||
WHERE c.id = %s AND cm.user_id = %s
|
||||
""", (chat_id, actor["id"]))
|
||||
row = cur.fetchone()
|
||||
if not row: raise APIError(404, "not_found", "Chat not found")
|
||||
|
||||
cur.execute("SELECT user_id FROM chat_members WHERE chat_id = %s", (chat_id,))
|
||||
members = [str(r[0]) for r in cur.fetchall()]
|
||||
|
||||
self.send_json(200, {"id": str(row[0]), "title": row[1], "member_ids": members, "created_at": format_dt(row[2])}, request_id)
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
def create_attachment(self, request_id, chat_id, query):
|
||||
actor = get_user_by_token(self)
|
||||
|
||||
filename = query.get("filename", ["upload.bin"])[0]
|
||||
if "/" in filename or "\\" in filename or "\x00" in filename:
|
||||
raise APIError(400, "invalid_request", "Dangerous filename")
|
||||
|
||||
length = self.headers.get("Content-Length")
|
||||
if not length: raise APIError(400, "invalid_request", "Missing body")
|
||||
length = int(length)
|
||||
if length > 10 * 1024 * 1024:
|
||||
raise APIError(413, "payload_too_large", "Max 10MB allowed")
|
||||
|
||||
raw_data = self.rfile.read(length)
|
||||
|
||||
try:
|
||||
img = Image.open(io.BytesIO(raw_data))
|
||||
if img.format not in ("PNG", "JPEG"):
|
||||
raise APIError(415, "unsupported_media_type", "Only PNG/JPEG allowed")
|
||||
if img.width * img.height > 40_000_000:
|
||||
raise APIError(413, "payload_too_large", "Max 40M pixels allowed")
|
||||
mime = "image/png" if img.format == "PNG" else "image/jpeg"
|
||||
except APIError:
|
||||
raise
|
||||
except Exception:
|
||||
raise APIError(415, "unsupported_media_type", "Invalid image")
|
||||
|
||||
sha = hashlib.sha256(raw_data).hexdigest()
|
||||
aid = str(uuid.uuid4())
|
||||
s3_key = f"{chat_id}/{aid}"
|
||||
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s", (chat_id, actor["id"]))
|
||||
if not cur.fetchone(): raise APIError(404, "not_found", "Chat not found")
|
||||
|
||||
try:
|
||||
s3_client.put_object(Bucket=S3_BUCKET, Key=s3_key, Body=raw_data, ContentType=mime)
|
||||
except ClientError as e:
|
||||
self.log_error("S3 put failed: %s", e)
|
||||
raise APIError(500, "internal_error", "S3 unavailable")
|
||||
|
||||
cur.execute("""
|
||||
INSERT INTO attachments (id, chat_id, owner_id, filename, content_type, size_bytes, sha256, s3_key, status)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, 'ready') RETURNING created_at
|
||||
""", (aid, chat_id, actor["id"], filename, mime, len(raw_data), sha, s3_key))
|
||||
created_at = cur.fetchone()[0]
|
||||
conn.commit()
|
||||
|
||||
payload = {
|
||||
"id": aid, "chat_id": chat_id, "owner_id": actor["id"],
|
||||
"filename": filename, "content_type": mime, "size_bytes": len(raw_data),
|
||||
"sha256": sha, "status": "ready",
|
||||
"variants": [],
|
||||
"error_code": None,
|
||||
"created_at": format_dt(created_at),
|
||||
"updated_at": format_dt(created_at)
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
conn.rollback()
|
||||
try:
|
||||
s3_client.delete_object(Bucket=S3_BUCKET, Key=s3_key)
|
||||
except Exception:
|
||||
pass
|
||||
if isinstance(e, APIError):
|
||||
raise e
|
||||
else:
|
||||
self.log_error("Unhandled attachment error: %s", traceback.format_exc())
|
||||
raise APIError(500, "internal_error", "Internal error during upload")
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
self.send_json(201, payload, request_id)
|
||||
|
||||
def get_attachment_meta(self, request_id, att_id):
|
||||
actor = get_user_by_token(self)
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("""
|
||||
SELECT a.id, a.chat_id, a.owner_id, a.filename, a.content_type, a.size_bytes, a.sha256, a.created_at
|
||||
FROM attachments a JOIN chat_members cm ON a.chat_id = cm.chat_id
|
||||
WHERE a.id = %s AND cm.user_id = %s
|
||||
""", (att_id, actor["id"]))
|
||||
row = cur.fetchone()
|
||||
if not row: raise APIError(404, "not_found", "Attachment not found")
|
||||
|
||||
self.send_json(200, {
|
||||
"id": str(row[0]), "chat_id": str(row[1]), "owner_id": str(row[2]),
|
||||
"filename": row[3], "content_type": row[4], "size_bytes": row[5],
|
||||
"sha256": row[6], "status": "ready",
|
||||
"variants": [],
|
||||
"error_code": None,
|
||||
"created_at": format_dt(row[7]),
|
||||
"updated_at": format_dt(row[7])
|
||||
}, request_id)
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
def get_attachment_content(self, request_id, att_id):
|
||||
actor = get_user_by_token(self)
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("""
|
||||
SELECT a.s3_key, a.content_type FROM attachments a
|
||||
JOIN chat_members cm ON a.chat_id = cm.chat_id
|
||||
WHERE a.id = %s AND cm.user_id = %s
|
||||
""", (att_id, actor["id"]))
|
||||
row = cur.fetchone()
|
||||
if not row: raise APIError(404, "not_found", "Attachment not found")
|
||||
s3_key, mime = row
|
||||
|
||||
try:
|
||||
response = s3_client.get_object(Bucket=S3_BUCKET, Key=s3_key)
|
||||
data = response['Body'].read()
|
||||
except ClientError:
|
||||
raise APIError(404, "not_found", "File missing from S3")
|
||||
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", mime)
|
||||
self.send_header("X-Request-Id", request_id)
|
||||
self.send_header("X-Instance-Id", INSTANCE_ID)
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
def create_message(self, request_id, chat_id):
|
||||
actor = get_user_by_token(self)
|
||||
|
||||
idem_key = self.headers.get("Idempotency-Key")
|
||||
if not idem_key:
|
||||
raise APIError(400, "invalid_request", "Idempotency-Key header is required")
|
||||
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s", (chat_id, actor["id"]))
|
||||
if not cur.fetchone(): raise APIError(404, "not_found", "Chat not found")
|
||||
|
||||
body = read_json(self)
|
||||
text = body.get("text", "")
|
||||
attachment_ids = body.get("attachment_ids", [])
|
||||
|
||||
if not text and not attachment_ids:
|
||||
raise APIError(400, "invalid_request", "Empty text and no attachments")
|
||||
|
||||
body_hash = hashlib.sha256(json.dumps(body, sort_keys=True).encode('utf-8')).hexdigest()
|
||||
cur.execute("SELECT message_id, body_hash FROM idempotency_keys WHERE user_id = %s AND idem_key = %s", (actor["id"], idem_key))
|
||||
existing = cur.fetchone()
|
||||
|
||||
if existing:
|
||||
if existing[1] != body_hash:
|
||||
raise APIError(409, "conflict", "Idempotency key used with different body")
|
||||
cur.execute("SELECT id, sender_id, text, created_at FROM messages WHERE id = %s", (existing[0],))
|
||||
row = cur.fetchone()
|
||||
return self.send_json(201, {
|
||||
"id": str(row[0]), "chat_id": chat_id, "sender_id": str(row[1]), "text": row[2],
|
||||
"attachment_ids": attachment_ids, "created_at": format_dt(row[3])
|
||||
}, request_id)
|
||||
|
||||
for aid in attachment_ids:
|
||||
cur.execute("SELECT 1 FROM attachments WHERE id = %s AND chat_id = %s", (aid, chat_id))
|
||||
if not cur.fetchone():
|
||||
raise APIError(404, "not_found", f"Attachment {aid} not found in this chat")
|
||||
|
||||
mid = str(uuid.uuid4())
|
||||
cur.execute(
|
||||
"INSERT INTO messages (id, chat_id, sender_id, text) VALUES (%s, %s, %s, %s) RETURNING created_at",
|
||||
(mid, chat_id, actor["id"], text)
|
||||
)
|
||||
created_at = cur.fetchone()[0]
|
||||
|
||||
for aid in attachment_ids:
|
||||
cur.execute("INSERT INTO message_attachments (message_id, attachment_id) VALUES (%s, %s)", (mid, aid))
|
||||
|
||||
cur.execute(
|
||||
"INSERT INTO idempotency_keys (idem_key, user_id, message_id, body_hash) VALUES (%s, %s, %s, %s)",
|
||||
(idem_key, actor["id"], mid, body_hash)
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
self.send_json(201, {
|
||||
"id": mid, "chat_id": chat_id, "sender_id": actor["id"], "text": text,
|
||||
"attachment_ids": attachment_ids, "created_at": format_dt(created_at)
|
||||
}, request_id)
|
||||
except psycopg2.errors.UniqueViolation:
|
||||
conn.rollback()
|
||||
raise APIError(409, "conflict", "Concurrent request conflict")
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
def list_messages(self, request_id, chat_id, query):
|
||||
actor = get_user_by_token(self)
|
||||
limit = parse_limit(query)
|
||||
cursor = query.get("cursor", [None])[0]
|
||||
|
||||
conn = db_pool.getconn()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s", (chat_id, actor["id"]))
|
||||
if not cur.fetchone(): raise APIError(404, "not_found", "Chat not found")
|
||||
|
||||
if cursor:
|
||||
ts, iid = decode_cursor(cursor, "messages", chat_id)
|
||||
cur.execute("""
|
||||
SELECT id, sender_id, text, created_at FROM messages
|
||||
WHERE chat_id = %s AND (created_at, id) > (%s, %s)
|
||||
ORDER BY created_at, id LIMIT %s
|
||||
""", (chat_id, ts, iid, limit + 1))
|
||||
else:
|
||||
cur.execute("""
|
||||
SELECT id, sender_id, text, created_at FROM messages
|
||||
WHERE chat_id = %s ORDER BY created_at, id LIMIT %s
|
||||
""", (chat_id, limit + 1))
|
||||
|
||||
rows = cur.fetchall()
|
||||
has_more = len(rows) > limit
|
||||
rows = rows[:limit]
|
||||
|
||||
items = []
|
||||
for r in rows:
|
||||
mid = r[0]
|
||||
cur.execute("SELECT attachment_id FROM message_attachments WHERE message_id = %s", (mid,))
|
||||
aids = [str(a[0]) for a in cur.fetchall()]
|
||||
|
||||
items.append({
|
||||
"id": str(r[0]), "chat_id": chat_id, "sender_id": str(r[1]),
|
||||
"text": r[2], "attachment_ids": aids, "created_at": format_dt(r[3])
|
||||
})
|
||||
|
||||
next_cursor = encode_cursor("messages", chat_id, items[-1]) if has_more and items else None
|
||||
|
||||
self.send_json(200, {"items": items, "next_cursor": next_cursor}, request_id)
|
||||
finally:
|
||||
db_pool.putconn(conn)
|
||||
|
||||
def send_json(self, status, payload, request_id):
|
||||
self._status = status
|
||||
if status == 204:
|
||||
self.send_response(204)
|
||||
self.send_header("X-Request-Id", request_id)
|
||||
self.send_header("X-Instance-Id", INSTANCE_ID)
|
||||
self.end_headers()
|
||||
return
|
||||
body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("X-Request-Id", request_id)
|
||||
self.send_header("X-Instance-Id", INSTANCE_ID)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def send_error_json(self, status, code, message, request_id):
|
||||
self._status = status
|
||||
body = json.dumps({"error": {"code": code, "message": message, "request_id": request_id}}, ensure_ascii=False).encode("utf-8")
|
||||
self.send_response(status)
|
||||
if status == 401:
|
||||
parsed_path = urlparse(self.path).path
|
||||
if parsed_path == "/api/v1/auth/sessions":
|
||||
self.send_header("WWW-Authenticate", 'Basic realm="messenger", charset="UTF-8"')
|
||||
else:
|
||||
self.send_header("WWW-Authenticate", 'Bearer realm="messenger"')
|
||||
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("X-Request-Id", request_id)
|
||||
self.send_header("X-Instance-Id", INSTANCE_ID)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def main():
|
||||
ThreadingHTTPServer.daemon_threads = True
|
||||
ThreadingHTTPServer.allow_reuse_address = True
|
||||
server = ThreadingHTTPServer((HOST, PORT), Handler)
|
||||
def shutdown(signum, frame):
|
||||
threading.Thread(target=server.shutdown, daemon=True).start()
|
||||
signal.signal(signal.SIGTERM, shutdown)
|
||||
signal.signal(signal.SIGINT, shutdown)
|
||||
print(f"Listening on {HOST}:{PORT}", file=sys.stderr)
|
||||
try:
|
||||
server.serve_forever()
|
||||
finally:
|
||||
server.server_close()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 109 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 109 KiB |
@@ -0,0 +1,7 @@
|
||||
<html>
|
||||
<head><title>504 Gateway Time-out</title></head>
|
||||
<body>
|
||||
<center><h1>504 Gateway Time-out</h1></center>
|
||||
<hr><center>nginx/1.25.5</center>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,97 @@
|
||||
services:
|
||||
db:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: secret
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
- ./migrations:/docker-entrypoint-initdb.d
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U postgres -d messenger"]
|
||||
interval: 5s
|
||||
retries: 5
|
||||
|
||||
redis:
|
||||
image: valkey/valkey:7-alpine
|
||||
volumes:
|
||||
- redisdata:/data
|
||||
|
||||
# --- RustFS S3 ---
|
||||
rustfs:
|
||||
image: rustfs/rustfs:latest
|
||||
environment:
|
||||
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||
- RUSTFS_SECRET_KEY=rustfssecret
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
volumes:
|
||||
- rustfsdata:/data
|
||||
ports:
|
||||
- "9000:9000" # S3 API
|
||||
- "9001:9001" # Консоль
|
||||
# Правильный синтаксис: путь к данным передаётся в конце
|
||||
command: /data
|
||||
healthcheck:
|
||||
# Правильный эндпоинт: /health
|
||||
test: ["CMD-SHELL", "curl -f http://localhost:9000/health || exit 1"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
|
||||
# Инициализация: создание приватного бакета
|
||||
rustfs-init:
|
||||
image: amazon/aws-cli:latest
|
||||
depends_on:
|
||||
rustfs:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
AWS_ACCESS_KEY_ID: rustfsadmin
|
||||
AWS_SECRET_ACCESS_KEY: rustfssecret
|
||||
entrypoint: >
|
||||
/bin/sh -c "
|
||||
aws --endpoint-url http://rustfs:9000 s3 mb s3://attachments || true;
|
||||
exit 0;
|
||||
"
|
||||
|
||||
api:
|
||||
build: .
|
||||
environment:
|
||||
- DB_HOST=db
|
||||
- DB_USER=postgres
|
||||
- DB_PASSWORD=secret
|
||||
- DB_NAME=messenger
|
||||
- REDIS_HOST=redis
|
||||
- S3_ENDPOINT=http://rustfs:9000
|
||||
- S3_ACCESS_KEY=rustfsadmin
|
||||
- S3_SECRET_KEY=rustfssecret
|
||||
- S3_BUCKET=attachments
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_started
|
||||
rustfs:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.50'
|
||||
memory: 512M
|
||||
|
||||
lb:
|
||||
image: nginx:1.25-alpine
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
depends_on:
|
||||
- api
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
redisdata:
|
||||
rustfsdata:
|
||||
@@ -0,0 +1,5 @@
|
||||
=== ТЕСТ 1: ЗАГРУЗКА, ОТПРАВКА И СКАЧИВАНИЕ ===
|
||||
Оригинальный SHA-256: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
Загружен файл, ID: 3a2183d2-7060-40ff-acbc-07ed5c9d4684
|
||||
Сообщение с вложением отправлено.
|
||||
УСПЕХ: SHA-256 совпадает!
|
||||
@@ -0,0 +1,7 @@
|
||||
=== ТЕСТ 2: БЕЗОПАСНОСТЬ ===
|
||||
Ева пытается получить метаданные:
|
||||
HTTP 404 (Ожидалось 404)
|
||||
Ева пытается скачать файл:
|
||||
HTTP 404 (Ожидалось 404)
|
||||
Прямой GET к RustFS без авторизации:
|
||||
HTTP/1.1 403 Forbidden (Ожидалось 403 Forbidden)
|
||||
@@ -0,0 +1,11 @@
|
||||
=== ТЕСТ 3: ОТКАЗОУСТОЙЧИВОСТЬ ===
|
||||
1. Контейнеры ДО остановки:
|
||||
NAMES STATUS
|
||||
mtusi2026lab4-api-1 Up 5 minutes (healthy)
|
||||
mtusi2026lab4-api-2 Up 5 minutes (healthy)
|
||||
2. Останавливаем одну реплику API...
|
||||
3. Скачиваем файл через живую реплику:
|
||||
УСПЕХ: Файл скачан после отказа!
|
||||
4. Перезапускаем весь стенд...
|
||||
5. Проверяем скачивание старым токеном Боба:
|
||||
ОШИБКА: Хэши не совпадают!
|
||||
@@ -0,0 +1,8 @@
|
||||
=== ТЕСТ 4: ОТСУТСТВИЕ ЛОКАЛЬНЫХ ФАЙЛОВ НА API ===
|
||||
Смотрим содержимое папки /app внутри контейнера API-1:
|
||||
total 40
|
||||
drwxr-xr-x 1 root root 4096 Nov 10 07:13 .
|
||||
drwxr-xr-x 1 root root 4096 Nov 10 07:18 ..
|
||||
-rwxr-xr-x 1 root root 27571 Nov 10 07:06 app.py
|
||||
-rwxr-xr-x 1 root root 45 Nov 10 07:09 requirements.txt
|
||||
Проверяем, что .png файлов нет:
|
||||
@@ -0,0 +1,15 @@
|
||||
=== TRACK 2: RECONCILER (DRY-RUN) ===
|
||||
Files in DB: 2
|
||||
Files in S3: 3
|
||||
[ORPHAN] orphan-test-file (age: 0:00:11.003608)
|
||||
|
||||
DRY-RUN. To actually delete, run with --fix
|
||||
|
||||
=== TRACK 2: RECONCILER (FIX) ===
|
||||
Files in DB: 2
|
||||
Files in S3: 3
|
||||
[ORPHAN] orphan-test-file (age: 0:00:19.866185)
|
||||
-> Deleted from S3
|
||||
|
||||
=== CHECK AFTER DELETION (expected 403/404) ===
|
||||
HTTP/1.1 403 Forbidden
|
||||
@@ -0,0 +1,58 @@
|
||||
CREATE TABLE users (
|
||||
id UUID PRIMARY KEY,
|
||||
username VARCHAR(32) UNIQUE NOT NULL,
|
||||
display_name VARCHAR(100) NOT NULL,
|
||||
password_hash VARCHAR(255) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE chats (
|
||||
id UUID PRIMARY KEY,
|
||||
title VARCHAR(100) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE chat_members (
|
||||
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id UUID REFERENCES users(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id UUID REFERENCES users(id) ON DELETE CASCADE,
|
||||
text TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_messages_chat_created ON messages (chat_id, created_at, id);
|
||||
|
||||
CREATE TABLE idempotency_keys (
|
||||
idem_key VARCHAR(128),
|
||||
user_id UUID REFERENCES users(id),
|
||||
message_id UUID REFERENCES messages(id),
|
||||
body_hash VARCHAR(64) NOT NULL,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
PRIMARY KEY (user_id, idem_key)
|
||||
);
|
||||
|
||||
-- Таблицы для Лабы 4
|
||||
CREATE TABLE attachments (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
|
||||
owner_id UUID REFERENCES users(id) ON DELETE CASCADE,
|
||||
filename VARCHAR(255) NOT NULL,
|
||||
content_type VARCHAR(50) NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
sha256 VARCHAR(64) NOT NULL,
|
||||
s3_key VARCHAR(255) NOT NULL UNIQUE,
|
||||
status VARCHAR(20) DEFAULT 'ready',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE message_attachments (
|
||||
message_id UUID REFERENCES messages(id) ON DELETE CASCADE,
|
||||
attachment_id UUID REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (message_id, attachment_id)
|
||||
);
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
events { worker_connections 1024; }
|
||||
|
||||
http {
|
||||
upstream api_backend {
|
||||
server api:8080 max_fails=3 fail_timeout=5s;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
|
||||
location / {
|
||||
proxy_pass http://api_backend;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_connect_timeout 2s;
|
||||
proxy_read_timeout 10s;
|
||||
client_max_body_size 15m; # Разрешаем загрузку файлов до 15MB
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
psycopg2-binary
|
||||
bcrypt
|
||||
redis
|
||||
boto3
|
||||
Pillow
|
||||
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Reconciler for finding orphan/missing objects between S3 and PostgreSQL."""
|
||||
import os
|
||||
import boto3
|
||||
import psycopg2
|
||||
from botocore.config import Config
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
DB = psycopg2.connect(
|
||||
host=os.getenv("DB_HOST", "localhost"),
|
||||
user="postgres",
|
||||
password="secret",
|
||||
dbname="messenger"
|
||||
)
|
||||
s3 = boto3.client(
|
||||
's3',
|
||||
endpoint_url=os.getenv("S3_ENDPOINT", "http://localhost:9000"),
|
||||
aws_access_key_id=os.getenv("S3_ACCESS_KEY", "rustfsadmin"),
|
||||
aws_secret_access_key=os.getenv("S3_SECRET_KEY", "rustfssecret"),
|
||||
config=Config(signature_version='s3v4'),
|
||||
region_name='us-east-1'
|
||||
)
|
||||
BUCKET = "attachments"
|
||||
GRACE_HOURS = 0 # Set to 0 for demo purposes. In production, use 1 hour.
|
||||
|
||||
def reconcile(dry_run=True):
|
||||
cur = DB.cursor()
|
||||
cur.execute("SELECT id, s3_key, created_at FROM attachments")
|
||||
db_files = {row[1]: (row[0], row[2]) for row in cur.fetchall()}
|
||||
|
||||
response = s3.list_objects_v2(Bucket=BUCKET)
|
||||
s3_files = {obj['Key']: obj['LastModified'] for obj in response.get('Contents', [])}
|
||||
|
||||
print(f"Files in DB: {len(db_files)}")
|
||||
print(f"Files in S3: {len(s3_files)}")
|
||||
|
||||
# Orphans: in S3 but not in DB
|
||||
orphans = set(s3_files.keys()) - set(db_files.keys())
|
||||
for key in orphans:
|
||||
age = datetime.now(s3_files[key].tzinfo) - s3_files[key]
|
||||
print(f"[ORPHAN] {key} (age: {age})")
|
||||
if age > timedelta(hours=GRACE_HOURS):
|
||||
if not dry_run:
|
||||
s3.delete_object(Bucket=BUCKET, Key=key)
|
||||
print(f" -> Deleted from S3")
|
||||
else:
|
||||
print(f" -> Skipped (fresh, age < {GRACE_HOURS}h)")
|
||||
|
||||
# Missing: in DB but not in S3
|
||||
missing = set(db_files.keys()) - set(s3_files.keys())
|
||||
for key in missing:
|
||||
print(f"[MISSING] {key} exists in DB, but not in S3 (ERROR!)")
|
||||
|
||||
if dry_run:
|
||||
print("\nDRY-RUN. To actually delete, run with --fix")
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
reconcile(dry_run="--fix" not in sys.argv)
|
||||
@@ -0,0 +1,8 @@
|
||||
@echo off
|
||||
for /f "tokens=*" %%i in ('docker ps -q -f "name=mtusi2026lab4-api-"') do (
|
||||
echo Stopping API replica: %%i
|
||||
docker stop %%i
|
||||
exit /b 0
|
||||
)
|
||||
echo No API replicas found.
|
||||
exit /b 1
|
||||
@@ -0,0 +1,38 @@
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from client import Client, arguments, check, run
|
||||
|
||||
def main():
|
||||
args = arguments('Concurrency race test')
|
||||
bootstrap = Client(args.base_url, args.ca_file)
|
||||
alice, bob, _eve, chat = bootstrap.bootstrap()
|
||||
chat_id = chat['id']
|
||||
|
||||
CLIENTS = 10
|
||||
PER_CLIENT = 10
|
||||
barrier = threading.Barrier(CLIENTS) # Барьер: все потоки стартуют одновременно
|
||||
|
||||
def worker(i):
|
||||
local = Client(args.base_url, args.ca_file)
|
||||
barrier.wait() # Ждем, пока все 10 потоков не будут готовы
|
||||
for j in range(PER_CLIENT):
|
||||
local.call('POST', f'/api/v1/chats/{chat_id}/messages', 201, actor=alice, body={'text': f'w{i}-{j}'})
|
||||
|
||||
# Запускаем 10 потоков
|
||||
with ThreadPoolExecutor(max_workers=CLIENTS) as pool:
|
||||
list(pool.map(worker, range(CLIENTS)))
|
||||
|
||||
# Читаем историю и считаем уникальные ID
|
||||
seen = []
|
||||
cursor = None
|
||||
while True:
|
||||
path = f'/api/v1/chats/{chat_id}/messages?limit=100'
|
||||
if cursor: path += '&cursor=' + cursor
|
||||
page = bootstrap.call('GET', path, actor=bob, schema='MessagePage')['json']
|
||||
seen.extend(x['id'] for x in page['items'])
|
||||
cursor = page['next_cursor']
|
||||
if not cursor: break
|
||||
|
||||
check(len(seen) == 100, f'Ожидали 100, получили {len(seen)}')
|
||||
check(len(set(seen)) == 100, 'Есть дубликаты ID!')
|
||||
print('PASS: 10x10 concurrent writes -> 100 unique ids')
|
||||
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env python3
|
||||
import uuid
|
||||
from client import Client, arguments, check, run, LAB
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Lab1 edge-case tests; run against a live server')
|
||||
check(LAB == 1, 'these tests target lab1')
|
||||
c = Client(args.base_url, args.ca_file)
|
||||
|
||||
c.call('GET', '/health/live')
|
||||
c.call('GET', '/health/ready')
|
||||
|
||||
# --- malformed JSON ---
|
||||
c.call('POST', '/api/v1/users', 400,
|
||||
raw=b'{"username":',
|
||||
headers={'Content-Type': 'application/json'})
|
||||
c.call('POST', '/api/v1/users', 400,
|
||||
raw=b'not-json',
|
||||
headers={'Content-Type': 'application/json'})
|
||||
c.call('POST', '/api/v1/users', 400,
|
||||
raw='{"username":"alice","display_name":"А","extra":1}'.encode('utf-8'),
|
||||
headers={'Content-Type': 'application/json'})
|
||||
|
||||
# --- username validation / duplicate ---
|
||||
u1 = 'u_' + uuid.uuid4().hex[:10]
|
||||
body1 = {'username': u1, 'display_name': 'Первый'}
|
||||
created = c.call('POST', '/api/v1/users', 201, body=body1, schema='User')['json']
|
||||
check(created['username'] == u1, 'username changed')
|
||||
|
||||
c.call('POST', '/api/v1/users', 409, body=body1)
|
||||
c.call('POST', '/api/v1/users', 400,
|
||||
body={'username': 'Bad Name', 'display_name': 'X'})
|
||||
c.call('POST', '/api/v1/users', 400,
|
||||
body={'username': 'ab', 'display_name': 'X'})
|
||||
c.call('POST', '/api/v1/users', 400,
|
||||
body={'username': 'good_user', 'display_name': ''})
|
||||
|
||||
# --- chat creation validations ---
|
||||
a = c.user('alice')
|
||||
b = c.user('bob')
|
||||
e = c.user('eve')
|
||||
|
||||
c.call('POST', '/api/v1/chats', 400, actor=a,
|
||||
body={'title': 'no me', 'member_ids': [b['id'], e['id']]})
|
||||
c.call('POST', '/api/v1/chats', 400, actor=a,
|
||||
body={'title': 'ghost', 'member_ids': [a['id'], str(uuid.uuid4())]})
|
||||
c.call('POST', '/api/v1/chats', 400, actor=a,
|
||||
body={'title': 'dup', 'member_ids': [a['id'], a['id']]})
|
||||
|
||||
chat = c.call('POST', '/api/v1/chats', 201, actor=a,
|
||||
body={'title': 'Edge', 'member_ids': [a['id'], b['id']]},
|
||||
schema='Chat')['json']
|
||||
|
||||
# --- empty / whitespace-only message ---
|
||||
c.call('POST', f"/api/v1/chats/{chat['id']}/messages", 400, actor=a,
|
||||
body={'text': ''})
|
||||
c.call('POST', f"/api/v1/chats/{chat['id']}/messages", 400, actor=a,
|
||||
body={'text': ' \n '})
|
||||
c.call('POST', f"/api/v1/chats/{chat['id']}/messages", 400, actor=a,
|
||||
body={'text': 'x' * 4001})
|
||||
|
||||
# --- outsider (eve) forbidden ---
|
||||
c.call('GET', f"/api/v1/chats/{chat['id']}", 404, actor=e)
|
||||
c.call('GET', f"/api/v1/chats/{chat['id']}/messages", 404, actor=e)
|
||||
c.call('POST', f"/api/v1/chats/{chat['id']}/messages", 404, actor=e,
|
||||
body={'text': 'intrusion'})
|
||||
c.call('GET', f"/api/v1/chats/{chat['id']}", 401)
|
||||
|
||||
# --- unknown UUID ---
|
||||
c.call('GET', f"/api/v1/chats/{uuid.uuid4()}", 404, actor=a)
|
||||
c.call('GET', f"/api/v1/chats/{uuid.uuid4()}/messages", 404, actor=a)
|
||||
|
||||
# --- multi-page walk (5 pages of 1) ---
|
||||
from client import timestamp
|
||||
texts = [f'msg-{i}' for i in range(5)]
|
||||
created_msgs = [c.message(a, chat, t)['json'] for t in texts]
|
||||
|
||||
# Contract order is (created_at, id) ascending, NOT insertion order.
|
||||
def _key(m):
|
||||
return (timestamp(m['created_at']), uuid.UUID(m['id']).int)
|
||||
|
||||
expected = [m['text'] for m in sorted(created_msgs, key=_key)]
|
||||
|
||||
seen = []
|
||||
cursor = None
|
||||
pages = 0
|
||||
while True:
|
||||
path = f"/api/v1/chats/{chat['id']}/messages?limit=1"
|
||||
if cursor:
|
||||
from urllib.parse import quote
|
||||
path += '&cursor=' + quote(cursor, safe='')
|
||||
page = c.call('GET', path, actor=b, schema='MessagePage')['json']
|
||||
check(len(page['items']) == 1, 'expected exactly 1 item per page')
|
||||
seen.extend(x['text'] for x in page['items'])
|
||||
cursor = page['next_cursor']
|
||||
pages += 1
|
||||
if not cursor:
|
||||
break
|
||||
check(pages <= 20, 'pagination did not terminate')
|
||||
|
||||
check(seen == expected, f'page walk mismatch: {seen} != {expected}')
|
||||
check(pages == 5, f'expected 5 pages, got {pages}')
|
||||
|
||||
# cursor from one chat must not work on another
|
||||
other = c.call('POST', '/api/v1/chats', 201, actor=a,
|
||||
body={'title': 'Other', 'member_ids': [a['id'], b['id']]},
|
||||
schema='Chat')['json']
|
||||
bad_cursor_path = (f"/api/v1/chats/{other['id']}/messages"
|
||||
f"?limit=1&cursor={cursor or 'x'}")
|
||||
c.call('GET', bad_cursor_path, 400, actor=a)
|
||||
|
||||
print('PASS: lab1 extra tests')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
Reference in New Issue
Block a user