ЛР 5 — БПИ2302 — Максимова — на 4 #49

Open
kleprer wants to merge 15 commits from kleprer/mtusi2026lab5:solution/БПИ2302-Максимова into main
39 changed files with 2172 additions and 37 deletions
+17 -21
View File
@@ -1,25 +1,21 @@
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
# These are logical interface suggestions; map actual framework names in REPORT.md.
APP_HOST=0.0.0.0
APP_PORT=8080
APP_ENV=lab
SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
S3_ENDPOINT=http://s3:9000
S3_REGION=us-east-1
S3_BUCKET=messenger-attachments
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
BROKER_URL_FILE=/run/secrets/broker_url
WORKER_CONCURRENCY=2
JOB_MAX_ATTEMPTS=5
JOB_TIMEOUT_SECONDS=60
STAGING_RETENTION_SECONDS=86400
POSTGRES_USER=messenger
POSTGRES_PASSWORD=strongpassword
POSTGRES_DB=messenger
DATABASE_URL=postgresql+asyncpg://messenger:strongpassword@postgres:5432/messenger
REDIS_URL=redis://redis:6379/0
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
S3_ENDPOINT=http://minio:9000
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin
S3_BUCKET=attachments
RABBITMQ_USER=guest
RABBITMQ_PASSWORD=guest
+27
View File
@@ -0,0 +1,27 @@
# syntax=docker/dockerfile:1.7
FROM python:3.11.9-slim AS builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt
FROM python:3.11.9-slim
RUN useradd -m -u 10001 appuser
WORKDIR /app
COPY --from=builder /root/.local /home/appuser/.local
COPY app ./app
COPY migrations ./migrations
COPY alembic.ini .
ENV PATH=/home/appuser/.local/bin:$PATH \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
RUN chown -R appuser:appuser /app
USER appuser
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=5 \
CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).status==200 else 1)"
CMD ["python", "-m", "app.main"]
+35 -16
View File
@@ -1,37 +1,56 @@
# Отчёт — лабораторная 5
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Максимова Снежана, БПИ2302
- Целевая оценка: 4
- Commit/PR: <ссылка на PR появится после пуша>
- Использованные источники, библиотеки и помощь ИИ: FastAPI, SQLAlchemy, boto3, Pillow, pika, RabbitMQ, MinIO. ИИ использовался для генерации черновиков кода, финальная отладка проводилась вручную.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Для запуска требуются Docker, Docker Compose и Python.
```sh
git clone https://gitea.privcloud.tech/kleprer/mtusi2026lab5.git
cd mtusi2026lab5
python3 -c "import shutil; shutil.copy('.env.example', '.env')"
docker compose up -d --build --scale api=2
```
Безопасная остановка: `docker compose down`. Удаление данных: `docker compose down -v`.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
**Схема процессов:**
Клиент → Nginx → API-реплики (FastAPI) → RabbitMQ (очередь задач).
Фоновый Worker ← RabbitMQ → S3 (MinIO) / PostgreSQL.
**Что изменено по сравнению с lab4:**
1. **Асинхронная обработка (202 Accepted):** API сохраняет оригинал в S3 и запись в БД со статусом `queued`. Задача отправляется в RabbitMQ, клиент опрашивает статус.
2. **Transactional Outbox:** Для гарантированной доставки задач используется паттерн Outbox. Задача на публикацию в RabbitMQ сохраняется в таблицу `outbox` в **одной транзакции** с самим вложением. Отдельный фоновый таск `outbox_publisher` каждые 5 секунд читает оттуда и отправляет в брокер. Это решает проблему потери задачи при падении RabbitMQ после коммита в БД.
3. **Генерация вариантов (уровень 4):** Воркер с помощью `Pillow` создает 3 файла: `thumbnail` (JPEG 256x256), `crop` (JPEG 128x128) и `optimized` (WebP до 1280x1280).
4. **Идемпотентность и отказоустойчивость:** В `worker.py` реализован таймаут (`asyncio.wait_for(30s)`). Если обработка падает или зависает, вызывается `basic_nack(requeue=False)`, и сообщение отправляется в **DLQ (Dead Letter Queue)**. Повторная доставка не создает дубликаты благодаря проверке `status == "ready"`.
5. **Лимиты ресурсов:** В `compose.yaml` для API и Worker заданы ограничения CPU и RAM.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Публичный smoke (уровень 3/4) | `python tests/smoke.py` | PASS. Тест проверяет загрузку (202), ожидание `ready`, генерацию всех 3 вариантов. | вывод консоли |
| Негативные тесты (уровень 4) | `pytest student-tests/ -v` | PASS (4 passed). Проверяются лимиты размера, битые файлы, подмена MIME и чужие вложения. | `student-tests/test_attachments.py` |
| Генерация 3 вариантов | `GET /api/v1/attachments/{id}` | `status: ready`, массив `variants` содержит 3 элемента (thumbnail, crop, optimized). | вывод консоли |
| Outbox (защита от потери) | Проверка таблицы `outbox` | В одной транзакции с `Attachment` создается `Outbox(pending)`. | код `app/main.py` |
| DLQ и Timeout воркера | `docker compose logs worker` | При ошибке или таймауте сообщение уходит в DLQ. | код `app/worker.py` |
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
| Остановка worker во время загрузки | Задача не теряется, ждет в очереди | Файл получает `202 queued`. При запуске worker забирает задачу. | Без потери данных |
| Перезапуск RabbitMQ | Очередь сохраняется (durable) | Очередь и сообщения выживают благодаря `delivery_mode=2`. | Без потери данных |
| Зависание обработки (Timeout) | Сообщение не блокирует очередь | Срабатывает `wait_for(30s)`, задача уходит в DLQ. | Без потери данных |
| Скачивание файла в статусе `queued` | Ошибка `409 Conflict` | API не отдает пустой файл, возвращает корректную ошибку. | — |
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
- **Единый экземпляр RabbitMQ и MinIO:** Используется один узел для брокера и хранилища (SPOF).
- **Синхронное чтение при загрузке:** В API файл читается в память (`await request.body()`) перед отправкой в S3. Допустимо для 10 МБ, но не для больших файлов.
+114
View File
@@ -0,0 +1,114 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = migrations
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library.
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =
# max length of characters to apply to the
# "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to migrations/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "version_path_separator" below.
# version_locations = %(here)s/bar:%(here)s/bat:migrations/versions
# version path separator; As mentioned above, this is the character used to split
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
# Valid values for version_path_separator are:
#
# version_path_separator = :
# version_path_separator = ;
# version_path_separator = space
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
sqlalchemy.url = driver://user:pass@localhost/dbname
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
# hooks = ruff
# ruff.type = exec
# ruff.executable = %(here)s/.venv/bin/ruff
# ruff.options = --fix REVISION_SCRIPT_FILENAME
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
View File
+85
View File
@@ -0,0 +1,85 @@
import hashlib
import os
import secrets
from datetime import datetime, timezone
from uuid import UUID
import bcrypt
from fastapi import Depends, Header, HTTPException
from sqlalchemy import delete, select
from sqlalchemy.ext.asyncio import AsyncSession
from .db.models import SessionRow, User
from .db.session import get_db
import redis.asyncio as redis
redis_client = redis.from_url(os.getenv("REDIS_URL", "redis://redis:6379/0"))
SESSION_TTL = int(os.getenv("SESSION_TTL_SECONDS", "3600"))
def _prehash(password: str) -> bytes:
# bcrypt работает с ≤72 байт, SHA-256 даёт 32 байта и снимает ограничение
return hashlib.sha256(password.encode("utf-8")).hexdigest().encode("ascii")
def hash_password(password: str) -> str:
return bcrypt.hashpw(_prehash(password), bcrypt.gensalt(rounds=12)).decode("ascii")
def verify_password(password: str, password_hash: str) -> bool:
try:
return bcrypt.checkpw(_prehash(password), password_hash.encode("ascii"))
except ValueError:
return False
def generate_token() -> str:
return secrets.token_urlsafe(32)
def hash_token(token: str) -> str:
return hashlib.sha256(token.encode("ascii")).hexdigest()
def _unauthorized_bearer() -> HTTPException:
return HTTPException(
status_code=401,
detail="Unauthorized",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
)
async def create_session(user_id: str) -> str:
token = generate_token()
await redis_client.setex(
f"session:{hash_token(token)}", SESSION_TTL, str(user_id)
)
return token
async def get_current_user(
authorization: str | None = Header(None, alias="Authorization"),
db: AsyncSession = Depends(get_db),
) -> User:
if not authorization or not authorization.startswith("Bearer "):
raise _unauthorized_bearer()
token = authorization[7:]
token_hash = hash_token(token)
user_id_raw = await redis_client.get(f"session:{token_hash}")
if user_id_raw is None:
raise _unauthorized_bearer()
user = await db.get(User, UUID(user_id_raw.decode()))
if user is None:
raise _unauthorized_bearer()
return user
async def revoke_current_session(authorization: str, db: AsyncSession) -> None:
if not authorization or not authorization.startswith("Bearer "):
raise _unauthorized_bearer()
token = authorization[7:]
deleted = await redis_client.delete(f"session:{hash_token(token)}")
if deleted == 0:
raise _unauthorized_bearer()
View File
+5
View File
@@ -0,0 +1,5 @@
from sqlalchemy.orm import DeclarativeBase
class Base(DeclarativeBase):
pass
+85
View File
@@ -0,0 +1,85 @@
from datetime import datetime, timezone
from uuid import UUID, uuid4
from sqlalchemy import DateTime, ForeignKey, Index, Integer, String, Text, UniqueConstraint
from sqlalchemy.dialects.postgresql import UUID as PG_UUID, JSONB
from sqlalchemy.orm import Mapped, mapped_column
from .base import Base
def _now() -> datetime:
return datetime.now(timezone.utc)
class User(Base):
__tablename__ = "users"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
username: Mapped[str] = mapped_column(String(32), unique=True, nullable=False, index=True)
display_name: Mapped[str] = mapped_column(String(100), nullable=False)
password_hash: Mapped[str] = mapped_column(Text, nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, nullable=False)
class Chat(Base):
__tablename__ = "chats"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
title: Mapped[str] = mapped_column(String(100), nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, nullable=False)
__table_args__ = (Index("idx_chats_created_id", "created_at", "id"),)
class ChatMember(Base):
__tablename__ = "chat_members"
chat_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), primary_key=True)
user_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), primary_key=True)
__table_args__ = (Index("idx_chat_members_user", "user_id", "chat_id"),)
class Message(Base):
__tablename__ = "messages"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
chat_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False)
sender_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
text: Mapped[str] = mapped_column(Text, nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, nullable=False)
__table_args__ = (Index("idx_messages_chat_created_id", "chat_id", "created_at", "id"),)
class SessionRow(Base):
__tablename__ = "sessions"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
user_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
token_hash: Mapped[str] = mapped_column(String(64), unique=True, nullable=False, index=True)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, index=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, nullable=False)
class IdempotencyRecord(Base):
__tablename__ = "idempotency_keys"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
user_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
key: Mapped[str] = mapped_column(String(128), nullable=False)
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
response_body: Mapped[str] = mapped_column(Text, nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, nullable=False)
__table_args__ = (UniqueConstraint("user_id", "key", name="uq_idempotency_user_key"),)
class Attachment(Base):
__tablename__ = "attachments"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
chat_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False)
owner_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
filename: Mapped[str] = mapped_column(String(255), nullable=False)
content_type: Mapped[str] = mapped_column(String(100), nullable=False)
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False)
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
status: Mapped[str] = mapped_column(String(20), default="queued", nullable=False)
error_code: Mapped[str] = mapped_column(String(50), nullable=True)
s3_key: Mapped[str] = mapped_column(String(500), nullable=False)
variants: Mapped[list] = mapped_column(JSONB, default=list, nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, onupdate=_now, nullable=False)
class MessageAttachment(Base):
__tablename__ = "message_attachments"
message_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), primary_key=True)
attachment_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("attachments.id", ondelete="CASCADE"), primary_key=True)
class Outbox(Base):
__tablename__ = "outbox"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
payload: Mapped[str] = mapped_column(Text, nullable=False)
status: Mapped[str] = mapped_column(String(20), default="pending", nullable=False, index=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=_now, nullable=False)
+31
View File
@@ -0,0 +1,31 @@
import os
from sqlalchemy.ext.asyncio import (
AsyncSession,
async_sessionmaker,
create_async_engine,
)
DATABASE_URL = os.getenv(
"DATABASE_URL",
"postgresql+asyncpg://messenger:strongpassword@localhost:5432/messenger",
)
engine = create_async_engine(
DATABASE_URL,
pool_size=10,
max_overflow=20,
pool_timeout=5,
pool_recycle=1800,
pool_pre_ping=True,
connect_args={"timeout": 5, "command_timeout": 10},
)
async_session = async_sessionmaker(
engine, class_=AsyncSession, expire_on_commit=False
)
async def get_db() -> AsyncSession:
async with async_session() as session:
yield session
+25
View File
@@ -0,0 +1,25 @@
import io
from PIL import Image
from fastapi import HTTPException
MAX_SIZE = 10 * 1024 * 1024 # 10 мбайт
MAX_PIXELS = 40_000_000
def validate_image(content: bytes) -> str:
if len(content) > MAX_SIZE:
raise HTTPException(413, "File too large")
try:
img = Image.open(io.BytesIO(content))
img.verify()
img = Image.open(io.BytesIO(content))
except Exception:
raise HTTPException(415, "Invalid image format")
if img.format not in ("PNG", "JPEG"):
raise HTTPException(415, "Unsupported media type")
width, height = img.size
if width * height > MAX_PIXELS:
raise HTTPException(413, "Image too large in pixels")
return "image/png" if img.format == "PNG" else "image/jpeg"
+349
View File
@@ -0,0 +1,349 @@
import os
import socket
import uuid
import hashlib
import json
import asyncio
from contextvars import ContextVar
from datetime import datetime, timedelta, timezone
from typing import Optional
from uuid import UUID
from fastapi import Depends, FastAPI, Header, HTTPException, Query, Request, Response
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from fastapi.security import HTTPBasic, HTTPBasicCredentials
from sqlalchemy import select, text
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.concurrency import run_in_threadpool
from .auth import (
SESSION_TTL, create_session, get_current_user, hash_password,
revoke_current_session, verify_password,
)
from .db.models import IdempotencyRecord, User, Attachment, MessageAttachment, Outbox
from .db.session import get_db, async_session
from .models import CreateChat, CreateMessage, CreateUser
from .repositories import chats as chats_repo
from .repositories import messages as messages_repo
from .repositories import users as users_repo
from .s3_client import s3, BUCKET, ensure_bucket
from .image_utils import validate_image
from .queue import publish_task
INSTANCE_ID = os.environ.get("HOSTNAME", f"api-{socket.gethostname()}")
request_id_var: ContextVar[str] = ContextVar("request_id", default="")
_HTTP_CODES = {
400: "invalid_request", 401: "unauthorized", 404: "not_found",
409: "conflict", 413: "payload_too_large", 415: "unsupported_media_type",
429: "rate_limited", 503: "unavailable", 500: "internal_error",
}
# --- ФОНОВЫЙ ПРОЦЕСС OUTBOX ---
# Каждые 5 секунд ищет в БД неотправленные задачи и кидает их в RabbitMQ
async def outbox_publisher():
while True:
await asyncio.sleep(5)
try:
async with async_session() as db:
jobs = (await db.execute(select(Outbox).where(Outbox.status == "pending").limit(10))).scalars().all()
for job in jobs:
try:
# публикуем в rabbitmq в отдельном потоке, чтобы не блокировать event loop
await run_in_threadpool(publish_task, json.loads(job.payload))
job.status = "sent"
await db.commit()
except Exception as e:
print(f"Outbox failed to publish {job.id}: {e}", flush=True)
except Exception:
pass
def err_json(status: int, code: str, message: str) -> JSONResponse:
resp = JSONResponse(status_code=status, content={"error": {"code": code, "message": message, "request_id": request_id_var.get()}})
resp.headers["X-Instance-Id"] = INSTANCE_ID
return resp
class RequestIdMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
rid = request.headers.get("X-Request-Id") or uuid.uuid4().hex
request_id_var.set(rid)
response = await call_next(request)
response.headers["X-Request-Id"] = rid
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
app = FastAPI(title="MTUSI Messenger — lab5")
app.add_middleware(RequestIdMiddleware)
security = HTTPBasic(auto_error=False)
@app.on_event("startup")
async def startup_event():
ensure_bucket()
# Запускаем фоновую задачу для отправки сообщений из БД в RabbitMQ
asyncio.create_task(outbox_publisher())
@app.exception_handler(RequestValidationError)
async def _validation(request: Request, exc: RequestValidationError):
return err_json(400, "invalid_request", "Validation error")
@app.exception_handler(HTTPException)
async def _http(request: Request, exc: HTTPException):
code = _HTTP_CODES.get(exc.status_code, "internal_error")
detail = exc.detail if isinstance(exc.detail, str) else "Error"
resp = err_json(exc.status_code, code, detail)
if exc.headers:
for k, v in exc.headers.items():
resp.headers[k] = v
return resp
@app.exception_handler(Exception)
async def _generic(request: Request, exc: Exception):
return err_json(500, "internal_error", "Internal server error")
def _iso_z(dt: datetime) -> str:
return dt.isoformat().replace("+00:00", "Z")
def _user_json(u: User) -> dict:
return {"id": str(u.id), "username": u.username, "display_name": u.display_name, "created_at": _iso_z(u.created_at)}
def _chat_json(chat, member_ids: list[str]) -> dict:
return {"id": str(chat.id), "title": chat.title, "member_ids": member_ids, "created_at": _iso_z(chat.created_at)}
def _msg_json(m, attachment_ids=None) -> dict:
return {
"id": str(m.id), "chat_id": str(m.chat_id), "sender_id": str(m.sender_id),
"text": m.text, "created_at": _iso_z(m.created_at),
"attachment_ids": [str(a) for a in (attachment_ids or [])]
}
def _attachment_json(att: Attachment) -> dict:
return {
"id": str(att.id), "chat_id": str(att.chat_id), "owner_id": str(att.owner_id),
"filename": att.filename, "content_type": att.content_type, "size_bytes": att.size_bytes,
"sha256": att.sha256, "status": att.status, "error_code": att.error_code,
"variants": att.variants or [],
"created_at": _iso_z(att.created_at),
"updated_at": _iso_z(att.updated_at),
}
@app.get("/health/live")
def live(): return {"status": "ok"}
@app.get("/health/ready")
async def ready(db: AsyncSession = Depends(get_db)):
try: await db.execute(text("SELECT 1"))
except Exception: return err_json(503, "unavailable", "Database unavailable")
return {"status": "ok"}
@app.post("/api/v1/users", status_code=201)
async def register(body: CreateUser, db: AsyncSession = Depends(get_db)):
existing = await users_repo.get_by_username(db, body.username)
if existing is not None: raise HTTPException(409, "Username already exists")
try:
user = await users_repo.create(db, body.username, body.display_name, hash_password(body.password))
except Exception: raise HTTPException(409, "Username already exists")
return _user_json(user)
@app.post("/api/v1/auth/sessions", status_code=201)
async def login(creds: Optional[HTTPBasicCredentials] = Depends(security), db: AsyncSession = Depends(get_db)):
basic_challenge = 'Basic realm="messenger", charset="UTF-8"'
if creds is None: raise HTTPException(401, "Unauthorized", headers={"WWW-Authenticate": basic_challenge})
user = await users_repo.get_by_username(db, creds.username)
if user is None or not verify_password(creds.password, user.password_hash):
raise HTTPException(401, "Unauthorized", headers={"WWW-Authenticate": basic_challenge})
expires = datetime.now(timezone.utc) + timedelta(seconds=SESSION_TTL)
token = await create_session(str(user.id))
return {"token": token, "token_type": "Bearer", "expires_at": _iso_z(expires), "user": _user_json(user)}
@app.delete("/api/v1/auth/sessions/current", status_code=204)
async def logout(authorization: str = Header(...), db: AsyncSession = Depends(get_db)):
await revoke_current_session(authorization, db)
@app.get("/api/v1/users/me")
async def me(user: User = Depends(get_current_user)):
return _user_json(user)
@app.post("/api/v1/chats", status_code=201)
async def create_chat(body: CreateChat, user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
if user.id not in body.member_ids: raise HTTPException(400, "Caller must be in member_ids")
try:
chat = await chats_repo.create_atomic(db, body.title, body.member_ids)
except ValueError: raise HTTPException(404, "Member not found")
return _chat_json(chat, [str(x) for x in body.member_ids])
@app.get("/api/v1/chats")
async def list_chats(user: User = Depends(get_current_user), limit: int = Query(50, ge=1, le=100), cursor: Optional[str] = Query(None, min_length=1), db: AsyncSession = Depends(get_db)):
parsed = None
if cursor is not None:
try:
c_created, c_id = cursor.split("|", 1)
parsed = (datetime.fromisoformat(c_created), UUID(c_id))
except Exception: raise HTTPException(400, "Invalid cursor")
rows, next_cursor = await chats_repo.list_for_user(db, user.id, limit, parsed)
items = []
for chat in rows:
members = await chats_repo.list_member_ids(db, chat.id)
items.append(_chat_json(chat, [str(m) for m in members]))
nxt = f"{next_cursor[0].isoformat()}|{next_cursor[1]}" if next_cursor else None
return {"items": items, "next_cursor": nxt}
@app.get("/api/v1/chats/{chat_id}")
async def get_chat(chat_id: UUID, user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
chat = await chats_repo.get_if_member(db, chat_id, user.id)
if chat is None: raise HTTPException(404, "Chat not found")
members = await chats_repo.list_member_ids(db, chat.id)
return _chat_json(chat, [str(m) for m in members])
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201)
async def post_message(chat_id: UUID, body: CreateMessage, idempotency_key: str = Header(..., alias="Idempotency-Key"), user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
chat = await chats_repo.get_if_member(db, chat_id, user.id)
if chat is None: raise HTTPException(404, "Chat not found")
if not body.text.strip() and not body.attachment_ids:
raise HTTPException(400, "Empty message without attachments")
req_hash = hashlib.sha256(body.model_dump_json().encode("utf-8")).hexdigest()
stmt = select(IdempotencyRecord).where(IdempotencyRecord.user_id == user.id, IdempotencyRecord.key == idempotency_key)
record = (await db.execute(stmt)).scalar_one_or_none()
if record:
if record.request_hash != req_hash: raise HTTPException(409, "Idempotency key conflict")
return JSONResponse(status_code=201, content=json.loads(record.response_body))
msg = await messages_repo.create_and_invalidate(db, chat_id, user.id, body.text)
if body.attachment_ids:
for att_id in body.attachment_ids:
att = await db.get(Attachment, att_id)
if not att or att.chat_id != chat_id:
raise HTTPException(404, f"Attachment {att_id} not found in this chat")
db.add(MessageAttachment(message_id=msg.id, attachment_id=att_id))
resp_dict = _msg_json(msg, body.attachment_ids)
resp_body = json.dumps(resp_dict)
db.add(IdempotencyRecord(user_id=user.id, key=idempotency_key, request_hash=req_hash, response_body=resp_body))
try:
await db.commit()
except IntegrityError:
await db.rollback()
record = (await db.execute(stmt)).scalar_one_or_none()
if record and record.request_hash == req_hash:
return JSONResponse(status_code=201, content=json.loads(record.response_body))
raise HTTPException(409, "Idempotency key conflict")
return JSONResponse(status_code=201, content=json.loads(resp_body))
@app.get("/api/v1/chats/{chat_id}/messages")
async def list_messages(chat_id: UUID, user: User = Depends(get_current_user), limit: int = Query(50, ge=1, le=100), cursor: Optional[str] = Query(None, min_length=1), db: AsyncSession = Depends(get_db)):
chat = await chats_repo.get_if_member(db, chat_id, user.id)
if chat is None: raise HTTPException(404, "Chat not found")
parsed = None
if cursor is not None:
try:
c_created, c_id = cursor.split("|", 1)
parsed = (datetime.fromisoformat(c_created), UUID(c_id))
except Exception: raise HTTPException(400, "Invalid cursor")
rows, next_cursor, _cached = await messages_repo.list_page_cached(db, chat_id, limit, parsed)
msg_ids = [m.id for m in rows]
att_map = {}
if msg_ids:
att_stmt = select(MessageAttachment.message_id, MessageAttachment.attachment_id).where(MessageAttachment.message_id.in_(msg_ids))
att_rows = (await db.execute(att_stmt)).all()
for mid, aid in att_rows: att_map.setdefault(mid, []).append(aid)
items = [_msg_json(m, att_map.get(m.id, [])) for m in rows]
nxt = f"{next_cursor[0].isoformat()}|{next_cursor[1]}" if next_cursor else None
return {"items": items, "next_cursor": nxt}
@app.post("/api/v1/chats/{chat_id}/attachments", status_code=202)
async def upload_attachment(chat_id: UUID, request: Request, filename: str = Query(..., min_length=1, max_length=255), user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
chat = await chats_repo.get_if_member(db, chat_id, user.id)
if not chat: raise HTTPException(404, "Chat not found")
if "/" in filename or "\\" in filename or "\x00" in filename:
raise HTTPException(400, "Invalid filename")
content = await request.body()
mime = validate_image(content)
sha256 = hashlib.sha256(content).hexdigest()
ext = ".png" if mime == "image/png" else ".jpg"
s3_key = f"{chat_id}/{uuid.uuid4()}{ext}"
# Загрузка в S3
try:
s3.put_object(Bucket=BUCKET, Key=s3_key, Body=content, ContentType=mime)
except Exception:
raise HTTPException(503, "S3 Storage unavailable")
att = Attachment(chat_id=chat_id, owner_id=user.id, filename=filename, content_type=mime, size_bytes=len(content), sha256=sha256, status="queued", s3_key=s3_key)
db.add(att)
try:
# Получаем ID вложения без коммита
await db.flush()
# Создаем задачу в Outbox
task = {"attachment_id": str(att.id), "chat_id": str(chat_id), "s3_key": s3_key, "mime": mime}
db.add(Outbox(payload=json.dumps(task), status="pending"))
# Коммитим ОБЕ записи атомарно
await db.commit()
await db.refresh(att)
except Exception:
await db.rollback()
try: s3.delete_object(Bucket=BUCKET, Key=s3_key)
except Exception: pass
raise HTTPException(500, "Database error while saving attachment")
return _attachment_json(att)
@app.get("/api/v1/attachments/{attachment_id}")
async def get_attachment(attachment_id: UUID, user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
att = await db.get(Attachment, attachment_id)
if not att: raise HTTPException(404, "Attachment not found")
chat = await chats_repo.get_if_member(db, att.chat_id, user.id)
if not chat: raise HTTPException(404, "Attachment not found")
return _attachment_json(att)
@app.get("/api/v1/attachments/{attachment_id}/content")
async def download_attachment(attachment_id: UUID, variant: str = Query("original"), user: User = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
att = await db.get(Attachment, attachment_id)
if not att: raise HTTPException(404, "Attachment not found")
chat = await chats_repo.get_if_member(db, att.chat_id, user.id)
if not chat: raise HTTPException(404, "Attachment not found")
if att.status != "ready":
raise HTTPException(409, "Attachment not ready")
key = att.s3_key
mime = att.content_type
if variant == "thumbnail":
key = f"{att.s3_key}_thumb.jpg"
mime = "image/jpeg"
elif variant == "crop":
key = f"{att.s3_key}_crop.jpg"
mime = "image/jpeg"
elif variant == "optimized":
key = f"{att.s3_key}_opt.webp"
mime = "image/webp"
try:
obj = s3.get_object(Bucket=BUCKET, Key=key)
content = obj["Body"].read()
except Exception:
raise HTTPException(503, "S3 Storage unavailable")
return Response(content=content, media_type=mime)
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("APP_PORT", "8080")))
+57
View File
@@ -0,0 +1,57 @@
from uuid import UUID
from pydantic import BaseModel, ConfigDict, Field, field_validator
class CreateUser(BaseModel):
model_config = ConfigDict(extra="forbid")
username: str = Field(pattern=r"^[a-z0-9_]{3,32}$")
display_name: str = Field(min_length=1, max_length=100)
password: str = Field(min_length=12, max_length=128)
@field_validator("display_name", "password")
@classmethod
def _not_blank(cls, v: str) -> str:
if not v.strip(): raise ValueError("must not be blank")
return v
class CreateChat(BaseModel):
model_config = ConfigDict(extra="forbid")
title: str = Field(min_length=1, max_length=100)
member_ids: list[UUID] = Field(min_length=1, max_length=100)
@field_validator("title")
@classmethod
def _title_not_blank(cls, v: str) -> str:
if not v.strip(): raise ValueError("title must not be blank")
return v
@field_validator("member_ids")
@classmethod
def _unique_members(cls, v: list[UUID]) -> list[UUID]:
if len(set(v)) != len(v): raise ValueError("duplicate member_ids")
return v
class CreateMessage(BaseModel):
model_config = ConfigDict(extra="forbid")
text: str = Field(min_length=0, max_length=4000)
attachment_ids: list[UUID] | None = None
class SessionResponse(BaseModel):
token: str
token_type: str = "Bearer"
expires_at: str
user: dict
class AttachmentResponse(BaseModel):
model_config = ConfigDict(extra="forbid")
id: UUID
chat_id: UUID
owner_id: UUID
filename: str
content_type: str
size_bytes: int
sha256: str
status: str
error_code: str | None = None
variants: list = []
created_at: str
updated_at: str
+35
View File
@@ -0,0 +1,35 @@
import pika
import os
import json
RABBITMQ_URL = os.getenv("RABBITMQ_URL", "amqp://guest:guest@rabbitmq:5672/")
QUEUE_NAME = "attachment_tasks"
DLQ_NAME = "attachment_tasks_dlq"
def setup_rabbitmq(channel):
# Создаем Dead Letter Exchange и Queue
channel.exchange_declare(exchange='dlx', exchange_type='direct', durable=True)
channel.queue_declare(queue=DLQ_NAME, durable=True)
channel.queue_bind(exchange='dlx', queue=DLQ_NAME, routing_key='dlq')
# Основная очередь, привязанная к DLX
channel.queue_declare(
queue=QUEUE_NAME,
durable=True,
arguments={
'x-dead-letter-exchange': 'dlx',
'x-dead-letter-routing-key': 'dlq'
}
)
def publish_task(task: dict):
connection = pika.BlockingConnection(pika.URLParameters(RABBITMQ_URL))
channel = connection.channel()
setup_rabbitmq(channel)
channel.basic_publish(
exchange="",
routing_key=QUEUE_NAME,
body=json.dumps(task),
properties=pika.BasicProperties(delivery_mode=2),
)
connection.close()
View File
+80
View File
@@ -0,0 +1,80 @@
from datetime import datetime
from uuid import UUID
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ..db.models import Chat, ChatMember, User
async def create_atomic(
db: AsyncSession, title: str, member_ids: list[UUID]
) -> Chat:
found = (
(await db.execute(select(User.id).where(User.id.in_(member_ids))))
.scalars()
.all()
)
if len(set(found)) != len(set(member_ids)):
await db.rollback()
raise ValueError("member_not_found")
chat = Chat(title=title)
db.add(chat)
await db.flush()
for uid in member_ids:
db.add(ChatMember(chat_id=chat.id, user_id=uid))
try:
await db.commit()
except Exception:
await db.rollback()
raise
await db.refresh(chat)
return chat
async def get_if_member(
db: AsyncSession, chat_id: UUID, user_id: UUID
) -> Chat | None:
stmt = (
select(Chat)
.join(ChatMember, ChatMember.chat_id == Chat.id)
.where(Chat.id == chat_id, ChatMember.user_id == user_id)
)
return (await db.execute(stmt)).scalar_one_or_none()
async def list_member_ids(db: AsyncSession, chat_id: UUID) -> list[UUID]:
stmt = select(ChatMember.user_id).where(ChatMember.chat_id == chat_id)
return list((await db.execute(stmt)).scalars().all())
async def list_for_user(
db: AsyncSession,
user_id: UUID,
limit: int,
cursor: tuple[datetime, UUID] | None,
) -> tuple[list[Chat], tuple[datetime, UUID] | None]:
stmt = (
select(Chat)
.join(ChatMember, ChatMember.chat_id == Chat.id)
.where(ChatMember.user_id == user_id)
.order_by(Chat.created_at.asc(), Chat.id.asc())
.limit(limit + 1)
)
if cursor is not None:
c_created, c_id = cursor
stmt = stmt.where(
(Chat.created_at > c_created)
| ((Chat.created_at == c_created) & (Chat.id > c_id))
)
rows = list((await db.execute(stmt)).scalars().all())
next_cursor = None
if len(rows) > limit:
rows = rows[:limit]
last = rows[-1]
next_cursor = (last.created_at, last.id)
return rows, next_cursor
+128
View File
@@ -0,0 +1,128 @@
import json
import os
from datetime import datetime
from types import SimpleNamespace
from uuid import UUID
import redis.asyncio as redis
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ..db.models import Message
redis_client = redis.from_url(os.getenv("REDIS_URL", "redis://redis:6379/0"))
CACHE_TTL = 30
def _serialize(rows: list[Message], next_cursor):
return {
"items": [
{
"id": str(m.id),
"chat_id": str(m.chat_id),
"sender_id": str(m.sender_id),
"text": m.text,
"created_at": m.created_at.isoformat(),
}
for m in rows
],
"next_cursor": (
[next_cursor[0].isoformat(), str(next_cursor[1])] if next_cursor else None
),
}
def _deserialize(data: dict):
items = [
SimpleNamespace(
id=UUID(row["id"]),
chat_id=UUID(row["chat_id"]),
sender_id=UUID(row["sender_id"]),
text=row["text"],
created_at=datetime.fromisoformat(row["created_at"]),
)
for row in data["items"]
]
nc = data.get("next_cursor")
next_cursor = (datetime.fromisoformat(nc[0]), UUID(nc[1])) if nc else None
return items, next_cursor
async def create(
db: AsyncSession, chat_id: UUID, sender_id: UUID, text: str
) -> Message:
msg = Message(chat_id=chat_id, sender_id=sender_id, text=text)
db.add(msg)
await db.commit()
await db.refresh(msg)
return msg
async def create_and_invalidate(
db: AsyncSession, chat_id: UUID, sender_id: UUID, text: str
) -> Message:
msg = await create(db, chat_id, sender_id, text)
keys = [f"chat:{chat_id}:first_page:{n}" for n in (10, 20, 50)]
await redis_client.delete(*keys)
return msg
async def list_page(
db: AsyncSession,
chat_id: UUID,
limit: int,
cursor: tuple[datetime, UUID] | None,
) -> tuple[list[Message], tuple[datetime, UUID] | None]:
stmt = (
select(Message)
.where(Message.chat_id == chat_id)
.order_by(Message.created_at.asc(), Message.id.asc())
.limit(limit + 1)
)
if cursor is not None:
c_created, c_id = cursor
stmt = stmt.where(
(Message.created_at > c_created)
| ((Message.created_at == c_created) & (Message.id > c_id))
)
rows = list((await db.execute(stmt)).scalars().all())
next_cursor = None
if len(rows) > limit:
rows = rows[:limit]
last = rows[-1]
next_cursor = (last.created_at, last.id)
return rows, next_cursor
async def list_page_cached(
db: AsyncSession,
chat_id: UUID,
limit: int,
cursor: tuple[datetime, UUID] | None,
) -> tuple[list, tuple[datetime, UUID] | None, bool]:
"""Возвращает (rows, next_cursor, cached)."""
use_cache = cursor is None and limit <= 50
key = f"chat:{chat_id}:first_page:{limit}"
if use_cache:
raw = await redis_client.get(key)
if raw is not None:
try:
rows, next_cursor = _deserialize(json.loads(raw))
return rows, next_cursor, True
except Exception:
pass # повреждённый кэш — игнорируем и идём в БД
rows, next_cursor = await list_page(db, chat_id, limit, cursor)
if use_cache:
try:
payload = json.dumps(_serialize(rows, next_cursor))
await redis_client.setex(key, CACHE_TTL, payload)
print(f"CACHE SET OK: {key}", flush=True)
except Exception as e:
import traceback
traceback.print_exc()
print(f"CACHE SET FAILED: {type(e).__name__}: {e}", flush=True)
return rows, next_cursor, False
+28
View File
@@ -0,0 +1,28 @@
from uuid import UUID
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ..db.models import User
async def get_by_username(db: AsyncSession, username: str) -> User | None:
return (
await db.execute(select(User).where(User.username == username))
).scalar_one_or_none()
async def get_by_id(db: AsyncSession, user_id: UUID) -> User | None:
return await db.get(User, user_id)
async def create(
db: AsyncSession, username: str, display_name: str, password_hash: str
) -> User:
user = User(
username=username, display_name=display_name, password_hash=password_hash
)
db.add(user)
await db.commit()
await db.refresh(user)
return user
+20
View File
@@ -0,0 +1,20 @@
import boto3
from botocore.client import Config
import os
s3 = boto3.client(
"s3",
endpoint_url=os.getenv("S3_ENDPOINT", "http://minio:9000"),
aws_access_key_id=os.getenv("S3_ACCESS_KEY", "minioadmin"),
aws_secret_access_key=os.getenv("S3_SECRET_KEY", "minioadmin"),
config=Config(signature_version="s3v4"),
region_name="us-east-1",
)
BUCKET = os.getenv("S3_BUCKET", "attachments")
def ensure_bucket():
try:
s3.head_bucket(Bucket=BUCKET)
except Exception:
s3.create_bucket(Bucket=BUCKET)
+108
View File
@@ -0,0 +1,108 @@
import asyncio
import json
import os
import io
import hashlib
from datetime import datetime, timezone
import pika
from PIL import Image, ImageOps
from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker
from sqlalchemy.pool import NullPool
from .db.models import Attachment
from .s3_client import s3, BUCKET
from .queue import RABBITMQ_URL, QUEUE_NAME, setup_rabbitmq
DATABASE_URL = os.getenv("DATABASE_URL")
async def process_attachment(task: dict):
att_id = task["attachment_id"]
s3_key = task["s3_key"]
engine = create_async_engine(DATABASE_URL, poolclass=NullPool)
AsyncSessionLocal = async_sessionmaker(engine, expire_on_commit=False)
try:
async with AsyncSessionLocal() as session:
att = await session.get(Attachment, att_id)
if not att: return
if att.status == "ready" and len(att.variants or []) == 3: return
obj = s3.get_object(Bucket=BUCKET, Key=s3_key)
content = obj["Body"].read()
img = Image.open(io.BytesIO(content))
img = ImageOps.exif_transpose(img)
if img.mode in ("RGBA", "LA"):
background = Image.new("RGB", img.size, (255, 255, 255))
background.paste(img, mask=img.split()[-1])
img = background
elif img.mode != "RGB":
img = img.convert("RGB")
variants_meta = []
thumb = img.copy()
thumb.thumbnail((256, 256), Image.Resampling.LANCZOS)
thumb_bytes = io.BytesIO()
thumb.save(thumb_bytes, format="JPEG")
thumb_data = thumb_bytes.getvalue()
s3.put_object(Bucket=BUCKET, Key=f"{s3_key}_thumb.jpg", Body=thumb_data, ContentType="image/jpeg")
variants_meta.append({"name": "thumbnail", "content_type": "image/jpeg", "width": thumb.width, "height": thumb.height, "size_bytes": len(thumb_data), "sha256": hashlib.sha256(thumb_data).hexdigest()})
crop_img = img.copy()
width, height = crop_img.size
min_dim = min(width, height)
crop_img = crop_img.crop(((width - min_dim) / 2, (height - min_dim) / 2, (width + min_dim) / 2, (height + min_dim) / 2))
crop_img = crop_img.resize((128, 128), Image.Resampling.LANCZOS)
crop_bytes = io.BytesIO()
crop_img.save(crop_bytes, format="JPEG")
crop_data = crop_bytes.getvalue()
s3.put_object(Bucket=BUCKET, Key=f"{s3_key}_crop.jpg", Body=crop_data, ContentType="image/jpeg")
variants_meta.append({"name": "crop", "content_type": "image/jpeg", "width": 128, "height": 128, "size_bytes": len(crop_data), "sha256": hashlib.sha256(crop_data).hexdigest()})
opt_img = img.copy()
opt_img.thumbnail((1280, 1280), Image.Resampling.LANCZOS)
opt_bytes = io.BytesIO()
opt_img.save(opt_bytes, format="WEBP", quality=85)
opt_data = opt_bytes.getvalue()
s3.put_object(Bucket=BUCKET, Key=f"{s3_key}_opt.webp", Body=opt_data, ContentType="image/webp")
variants_meta.append({"name": "optimized", "content_type": "image/webp", "width": opt_img.width, "height": opt_img.height, "size_bytes": len(opt_data), "sha256": hashlib.sha256(opt_data).hexdigest()})
att.status = "ready"
att.error_code = None
att.variants = variants_meta
att.updated_at = datetime.now(timezone.utc)
await session.commit()
except Exception as e:
async with AsyncSessionLocal() as session:
att = await session.get(Attachment, att_id)
if att:
att.status = "failed"
att.error_code = "processing_failed"
att.updated_at = datetime.now(timezone.utc)
await session.commit()
print(f"Error processing {att_id}: {e}", flush=True)
raise e
finally:
await engine.dispose()
def callback(ch, method, properties, body):
task = json.loads(body)
try:
# Таймаут 30 секунд
asyncio.run(asyncio.wait_for(process_attachment(task), timeout=30.0))
ch.basic_ack(delivery_tag=method.delivery_tag)
except Exception as e:
print(f"Task failed, routing to DLQ. Error: {e}", flush=True)
# requeue=False отправляет в DLQ
ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)
def main():
connection = pika.BlockingConnection(pika.URLParameters(RABBITMQ_URL))
channel = connection.channel()
setup_rabbitmq(channel)
channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue=QUEUE_NAME, on_message_callback=callback)
print("Worker started...", flush=True)
channel.start_consuming()
if __name__ == "__main__":
main()
+135
View File
@@ -0,0 +1,135 @@
services:
postgres:
image: postgres:16.2-alpine
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
volumes:
- pgdata:/var/lib/postgresql/data
ports:
- "127.0.0.1:5432:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
interval: 5s
timeout: 5s
retries: 10
redis:
image: redis:7.2-alpine
command: ["redis-server", "--appendonly", "yes"]
volumes:
- redisdata:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 5s
retries: 10
minio:
image: dappros/minio:RELEASE.2025-09-07T16-13-09Z
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
volumes:
- miniodata:/data
ports:
- "127.0.0.1:9000:9000"
- "127.0.0.1:9001:9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 5s
timeout: 5s
retries: 10
rabbitmq:
image: rabbitmq:3.13-management-alpine
hostname: rabbitmq
environment:
RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
ports:
- "127.0.0.1:5672:5672"
- "127.0.0.1:15672:15672"
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "check_port_connectivity"]
interval: 5s
timeout: 5s
retries: 10
migrator:
build: .
command: alembic upgrade head
environment:
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}
depends_on:
postgres:
condition: service_healthy
api:
build: .
environment:
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}
REDIS_URL: redis://redis:6379/0
SESSION_TTL_SECONDS: 3600
S3_ENDPOINT: ${S3_ENDPOINT}
S3_ACCESS_KEY: ${S3_ACCESS_KEY}
S3_SECRET_KEY: ${S3_SECRET_KEY}
S3_BUCKET: ${S3_BUCKET}
RABBITMQ_URL: amqp://${RABBITMQ_USER}:${RABBITMQ_PASSWORD}@rabbitmq:5672/
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
minio:
condition: service_healthy
rabbitmq:
condition: service_healthy
migrator:
condition: service_completed_successfully
deploy:
resources:
limits:
cpus: '0.5'
memory: 512M
worker:
build: .
command: python -m app.worker
environment:
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}
S3_ENDPOINT: ${S3_ENDPOINT}
S3_ACCESS_KEY: ${S3_ACCESS_KEY}
S3_SECRET_KEY: ${S3_SECRET_KEY}
S3_BUCKET: ${S3_BUCKET}
RABBITMQ_URL: amqp://${RABBITMQ_USER}:${RABBITMQ_PASSWORD}@rabbitmq:5672/
depends_on:
postgres:
condition: service_healthy
minio:
condition: service_healthy
rabbitmq:
condition: service_healthy
migrator:
condition: service_completed_successfully
deploy:
resources:
limits:
cpus: '1.0'
memory: 1G
nginx:
image: nginx:1.25-alpine
ports:
- "127.0.0.1:8080:80"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
- api
volumes:
pgdata:
redisdata:
miniodata:
+1
View File
@@ -0,0 +1 @@
Generic single-database configuration with an async dbapi.
+99
View File
@@ -0,0 +1,99 @@
import asyncio
from logging.config import fileConfig
from sqlalchemy import pool
from sqlalchemy.engine import Connection
from sqlalchemy.ext.asyncio import async_engine_from_config
from alembic import context
import os
from app.db import models # noqa: F401 — регистрирует таблицы
from app.db.base import Base
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = Base.metadata
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def run_migrations_offline() -> None:
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def do_run_migrations(connection: Connection) -> None:
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations() -> None:
"""In this scenario we need to create an Engine
and associate a connection with the context.
"""
config.set_main_option(
"sqlalchemy.url",
os.getenv(
"DATABASE_URL",
"postgresql+asyncpg://messenger:strongpassword@localhost:5432/messenger",
),
)
connectable = async_engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
"""Run migrations in 'online' mode."""
asyncio.run(run_async_migrations())
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
+26
View File
@@ -0,0 +1,26 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
@@ -0,0 +1,30 @@
"""add updated_at to attachments
Revision ID: 133990c80d19
Revises: abc40f94dd51
Create Date: 2026-11-11 21:06:40.354578
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '133990c80d19'
down_revision: Union[str, None] = 'abc40f94dd51'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.add_column('attachments', sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.text('now()'), nullable=False))
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_column('attachments', 'updated_at')
# ### end Alembic commands ###
@@ -0,0 +1,38 @@
"""add outbox table
Revision ID: 164e1973508a
Revises: d15635c49e1a
Create Date: 2026-11-12 21:18:12.919668
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '164e1973508a'
down_revision: Union[str, None] = 'd15635c49e1a'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('outbox',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('payload', sa.Text(), nullable=False),
sa.Column('status', sa.String(length=20), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_outbox_status'), 'outbox', ['status'], unique=False)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_outbox_status'), table_name='outbox')
op.drop_table('outbox')
# ### end Alembic commands ###
@@ -0,0 +1,85 @@
"""initial schema
Revision ID: 44ea9d3b68ac
Revises:
Create Date: 2026-10-01 02:42:35.498512
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '44ea9d3b68ac'
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('chats',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('title', sa.String(length=100), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_index('idx_chats_created_id', 'chats', ['created_at', 'id'], unique=False)
op.create_table('users',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('username', sa.String(length=32), nullable=False),
sa.Column('display_name', sa.String(length=100), nullable=False),
sa.Column('password_hash', sa.Text(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_users_username'), 'users', ['username'], unique=True)
op.create_table('chat_members',
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('chat_id', 'user_id')
)
op.create_index('idx_chat_members_user', 'chat_members', ['user_id', 'chat_id'], unique=False)
op.create_table('messages',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('sender_id', sa.UUID(), nullable=False),
sa.Column('text', sa.Text(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['sender_id'], ['users.id'], ),
sa.PrimaryKeyConstraint('id')
)
op.create_index('idx_messages_chat_created_id', 'messages', ['chat_id', 'created_at', 'id'], unique=False)
op.create_table('sessions',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.Column('token_hash', sa.String(length=64), nullable=False),
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_sessions_expires_at'), 'sessions', ['expires_at'], unique=False)
op.create_index(op.f('ix_sessions_token_hash'), 'sessions', ['token_hash'], unique=True)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_sessions_token_hash'), table_name='sessions')
op.drop_index(op.f('ix_sessions_expires_at'), table_name='sessions')
op.drop_table('sessions')
op.drop_index('idx_messages_chat_created_id', table_name='messages')
op.drop_table('messages')
op.drop_index('idx_chat_members_user', table_name='chat_members')
op.drop_table('chat_members')
op.drop_index(op.f('ix_users_username'), table_name='users')
op.drop_table('users')
op.drop_index('idx_chats_created_id', table_name='chats')
op.drop_table('chats')
# ### end Alembic commands ###
@@ -0,0 +1,53 @@
"""add attachments tables
Revision ID: 67c68635da78
Revises: b7965b0447f4
Create Date: 2026-11-11 20:18:53.130479
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '67c68635da78'
down_revision: Union[str, None] = 'b7965b0447f4'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('attachments',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('owner_id', sa.UUID(), nullable=False),
sa.Column('filename', sa.String(length=255), nullable=False),
sa.Column('content_type', sa.String(length=100), nullable=False),
sa.Column('size_bytes', sa.Integer(), nullable=False),
sa.Column('sha256', sa.String(length=64), nullable=False),
sa.Column('status', sa.String(length=20), nullable=False),
sa.Column('error_code', sa.String(length=50), nullable=True),
sa.Column('s3_key', sa.String(length=500), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['owner_id'], ['users.id'], ),
sa.PrimaryKeyConstraint('id')
)
op.create_table('message_attachments',
sa.Column('message_id', sa.UUID(), nullable=False),
sa.Column('attachment_id', sa.UUID(), nullable=False),
sa.ForeignKeyConstraint(['attachment_id'], ['attachments.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['message_id'], ['messages.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('message_id', 'attachment_id')
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('message_attachments')
op.drop_table('attachments')
# ### end Alembic commands ###
@@ -0,0 +1,30 @@
"""add updated_at to attachments
Revision ID: abc40f94dd51
Revises: 67c68635da78
Create Date: 2026-11-11 20:58:37.486897
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = 'abc40f94dd51'
down_revision: Union[str, None] = '67c68635da78'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
pass
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
pass
# ### end Alembic commands ###
@@ -0,0 +1,40 @@
"""add idempotency keys
Revision ID: b7965b0447f4
Revises: 44ea9d3b68ac
Create Date: 2026-10-28 23:34:43.564839
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = 'b7965b0447f4'
down_revision: Union[str, None] = '44ea9d3b68ac'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('idempotency_keys',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.Column('key', sa.String(length=128), nullable=False),
sa.Column('request_hash', sa.String(length=64), nullable=False),
sa.Column('response_body', sa.Text(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('user_id', 'key', name='uq_idempotency_user_key')
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('idempotency_keys')
# ### end Alembic commands ###
@@ -0,0 +1,30 @@
"""add variants to attachments
Revision ID: d15635c49e1a
Revises: 133990c80d19
Create Date: 2026-11-11 21:38:16.802941
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision: str = 'd15635c49e1a'
down_revision: Union[str, None] = '133990c80d19'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.add_column('attachments', sa.Column('variants', postgresql.JSONB(astext_type=sa.Text()), server_default='[]', nullable=False))
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_column('attachments', 'variants')
# ### end Alembic commands ###
+25
View File
@@ -0,0 +1,25 @@
worker_processes 1;
events { worker_connections 1024; }
http {
resolver 127.0.0.11 valid=5s;
upstream api_backend {
server api:8080 max_fails=1 fail_timeout=2s;
keepalive 32;
}
server {
listen 80;
location / {
proxy_pass http://api_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_connect_timeout 1s;
proxy_next_upstream error timeout http_502 http_503 non_idempotent;
}
}
}
+15
View File
@@ -0,0 +1,15 @@
fastapi==0.110.0
uvicorn[standard]==0.29.0
pydantic==2.6.4
sqlalchemy[asyncio]==2.0.29
asyncpg==0.29.0
alembic==1.13.1
bcrypt==4.1.2
python-dotenv==1.0.1
redis==5.0.3
httpx==0.27.0
pytest==8.1.1
pytest-asyncio==0.23.5
boto3==1.34.0
Pillow==10.2.0
pika==1.3.2
+6
View File
@@ -0,0 +1,6 @@
#!/bin/bash
set -e
docker compose restart postgres
docker compose restart redis
docker compose restart api
docker compose restart nginx
+6
View File
@@ -0,0 +1,6 @@
@echo off
if "%TARGET_INSTANCE_ID%"=="" (
echo TARGET_INSTANCE_ID is not set
exit /b 1
)
docker kill %TARGET_INSTANCE_ID%
+79
View File
@@ -0,0 +1,79 @@
import time
import sys
import uuid
import json
import base64
import urllib.request
import urllib.error
import threading
BASE = "http://127.0.0.1:8080"
PASSWORD = "Example-only-not-a-real-secret-123"
def call(method, path, body=None, token=None):
h = {"Accept": "application/json"}
if token:
h["Authorization"] = f"Bearer {token}"
if method == "POST" and "/messages" in path:
h["Idempotency-Key"] = uuid.uuid4().hex
data = None
if body is not None:
data = json.dumps(body).encode("utf-8")
h["Content-Type"] = "application/json"
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
with urllib.request.urlopen(req, timeout=30) as r:
payload = r.read()
return r.status, (json.loads(payload) if payload else None)
except urllib.error.HTTPError as e:
payload = e.read()
return e.code, (json.loads(payload) if payload else None)
def main(replicas, concurrency, duration):
name = f"load_{uuid.uuid4().hex[:6]}"
s, u = call("POST", "/api/v1/users", {"username": name, "display_name": name, "password": PASSWORD})
if s != 201: raise Exception(f"Create user failed: {s} {u}")
cred = base64.b64encode(f"{name}:{PASSWORD}".encode()).decode()
req = urllib.request.Request(
BASE + "/api/v1/auth/sessions",
headers={"Authorization": f"Basic {cred}"},
method="POST"
)
with urllib.request.urlopen(req) as r:
s = json.loads(r.read())
tok = s["token"]
s, chat = call("POST", "/api/v1/chats", {"title": "bench", "member_ids": [u["id"]]}, token=tok)
if s != 201: raise Exception(f"Create chat failed: {s} {chat}")
cid = chat["id"]
latencies = []
errors = 0
lock = threading.Lock()
def worker():
nonlocal errors
end = time.time() + duration
while time.time() < end:
t0 = time.time()
s, _ = call("POST", f"/api/v1/chats/{cid}/messages", {"text": "hello"}, token=tok)
with lock:
if s == 201:
latencies.append(time.time() - t0)
else:
errors += 1
threads = [threading.Thread(target=worker) for _ in range(concurrency)]
start = time.time()
for t in threads: t.start()
for t in threads: t.join()
elapsed = time.time() - start
latencies.sort()
p95 = latencies[int(len(latencies) * 0.95)] * 1000 if latencies else 0
rps = len(latencies) / elapsed if elapsed > 0 else 0
print(f"Replicas: {replicas} | RPS: {rps:.2f} | p95: {p95:.2f}ms | Errors: {errors}")
if __name__ == "__main__":
main(sys.argv[1], int(sys.argv[2]), int(sys.argv[3]))
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env python3
import asyncio
import base64
import json
import urllib.request
import urllib.error
import uuid
from concurrent.futures import ThreadPoolExecutor
BASE = "http://127.0.0.1:8080"
PASSWORD = "Example-only-not-a-real-secret-123"
def call(method, path, body=None, token=None):
h = {"Accept": "application/json"}
if token:
h["Authorization"] = f"Bearer {token}"
# ДОБАВЛЯЕМ IDEMPOTENCY-KEY ДЛЯ POST
if method == "POST" and "/messages" in path:
h["Idempotency-Key"] = uuid.uuid4().hex
data = None
if body is not None:
data = json.dumps(body).encode("utf-8")
h["Content-Type"] = "application/json"
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
with urllib.request.urlopen(req, timeout=30) as r:
payload = r.read()
return r.status, (json.loads(payload) if payload else None)
except urllib.error.HTTPError as e:
payload = e.read()
return e.code, (json.loads(payload) if payload else None)
def create_user_and_login(name):
s, u = call("POST", "/api/v1/users",
{"username": name, "display_name": name, "password": PASSWORD})
assert s == 201, f"create_user {name}: {s} {u}"
cred = base64.b64encode(f"{name}:{PASSWORD}".encode()).decode()
req = urllib.request.Request(
BASE + "/api/v1/auth/sessions",
headers={"Authorization": f"Basic {cred}"},
method="POST",
)
with urllib.request.urlopen(req, timeout=30) as r:
session = json.loads(r.read())
return u, session["token"]
def main():
users = [create_user_and_login(f"conc_{i}_{uuid.uuid4().hex[:6]}") for i in range(10)]
owner, owner_token = users[0]
chat = call("POST", "/api/v1/chats",
{"title": "concurrent", "member_ids": [u["id"] for u, _ in users]},
token=owner_token)[1]
cid = chat["id"]
def post_10(user_and_token):
_, token = user_and_token
results = []
for k in range(10):
s, m = call("POST", f"/api/v1/chats/{cid}/messages",
{"text": f"m{k}"}, token=token)
results.append((s, m))
return results
with ThreadPoolExecutor(max_workers=10) as ex:
results = []
for fut in [ex.submit(post_10, u) for u in users]:
results.extend(fut.result())
failures = [r for r in results if r[0] != 201]
assert not failures, f"{len(failures)} failed posts"
seen, cursor = set(), None
while True:
path = f"/api/v1/chats/{cid}/messages?limit=100"
if cursor:
path += f"&cursor={cursor}"
s, page = call("GET", path, token=owner_token)
for m in page["items"]:
seen.add(m["id"])
cursor = page["next_cursor"]
if not cursor:
break
print(f"unique messages: {len(seen)} / 100")
assert len(seen) == 100, f"expected 100, got {len(seen)}"
print("PASS: no lost or duplicated messages")
if __name__ == "__main__":
main()
+69
View File
@@ -0,0 +1,69 @@
import base64
import json
import urllib.request
import urllib.error
import uuid
BASE = "http://127.0.0.1:8080"
PASSWORD = "Example-only-not-a-real-secret-123"
def call(method, path, body=None, token=None, raw_data=None, content_type=None):
h = {"Accept": "application/json"}
if token: h["Authorization"] = f"Bearer {token}"
if method == "POST" and "/messages" in path:
h["Idempotency-Key"] = str(uuid.uuid4())
data = None
if raw_data is not None:
data = raw_data
if content_type: h["Content-Type"] = content_type
elif body is not None:
data = json.dumps(body).encode("utf-8")
h["Content-Type"] = "application/json"
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
with urllib.request.urlopen(req, timeout=10) as r:
payload = r.read()
return r.status, (json.loads(payload) if payload and r.headers.get_content_type()=="application/json" else payload)
except urllib.error.HTTPError as e:
payload = e.read()
try: return e.code, json.loads(payload)
except: return e.code, payload
def setup():
name = f"test_{uuid.uuid4().hex[:6]}"
s, u = call("POST", "/api/v1/users", {"username": name, "display_name": name, "password": PASSWORD})
cred = base64.b64encode(f"{name}:{PASSWORD}".encode()).decode()
req = urllib.request.Request(BASE + "/api/v1/auth/sessions", headers={"Authorization": f"Basic {cred}"}, method="POST")
with urllib.request.urlopen(req) as r:
tok = json.loads(r.read())["token"]
s, chat = call("POST", "/api/v1/chats", {"title": "test", "member_ids": [u["id"]]}, token=tok)
return u, tok, chat["id"]
def test_large_file():
u, tok, cid = setup()
big_data = b"\x89PNG\r\n\x1a\n" + b"0" * (11 * 1024 * 1024)
s, _ = call("POST", f"/api/v1/chats/{cid}/attachments?filename=big.png", token=tok, raw_data=big_data, content_type="image/png")
assert s == 413
def test_corrupted_image():
u, tok, cid = setup()
s, _ = call("POST", f"/api/v1/chats/{cid}/attachments?filename=bad.png", token=tok, raw_data=b"not an image", content_type="image/png")
assert s == 415
def test_mime_spoofing():
u, tok, cid = setup()
s, _ = call("POST", f"/api/v1/chats/{cid}/attachments?filename=fake.png", token=tok, raw_data=b"<html>hack</html>", content_type="image/png")
assert s == 415
def test_foreign_attachment():
u1, tok1, cid1 = setup()
u2, tok2, cid2 = setup()
s, att = call("POST", f"/api/v1/chats/{cid1}/attachments?filename=test.png", token=tok1, raw_data=b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\nIDATx\x9cc\x00\x01\x00\x00\x05\x00\x01\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82", content_type="image/png")
assert s == 202, f"Upload failed: {att}"
s, response_body = call("POST", f"/api/v1/chats/{cid2}/messages", {"text": "hi", "attachment_ids": [att["id"]]}, token=tok2)
assert s == 404, f"Expected 404, got {s}. Server response: {response_body}"
+86
View File
@@ -0,0 +1,86 @@
import asyncio
import base64
import json
import urllib.error
import urllib.request
import uuid
import pytest
BASE = "http://127.0.0.1:8080"
PASSWORD = "Example-only-not-a-real-secret-123"
def _post(path, body, token=None):
h = {"Content-Type": "application/json", "Accept": "application/json"}
if token:
h["Authorization"] = f"Bearer {token}"
data = json.dumps(body).encode("utf-8")
req = urllib.request.Request(BASE + path, data=data, method="POST", headers=h)
try:
with urllib.request.urlopen(req, timeout=10) as r:
return r.status, json.loads(r.read())
except urllib.error.HTTPError as e:
payload = e.read()
try:
return e.code, json.loads(payload) if payload else None
except ValueError:
return e.code, None
def _post_basic(path, username, password):
cred = base64.b64encode(f"{username}:{password}".encode()).decode()
req = urllib.request.Request(
BASE + path, method="POST",
headers={"Authorization": f"Basic {cred}", "Accept": "application/json"},
)
with urllib.request.urlopen(req, timeout=10) as r:
return r.status, json.loads(r.read())
def _create_user(name):
return _post("/api/v1/users", {
"username": name, "display_name": name, "password": PASSWORD,
})
def _login(name):
_, body = _post_basic("/api/v1/auth/sessions", name, PASSWORD)
return body["token"]
def test_duplicate_username_race():
name = f"dup_{uuid.uuid4().hex[:8]}"
import threading
results = [None, None]
def worker(i):
results[i] = _create_user(name)
t1 = threading.Thread(target=worker, args=(0,))
t2 = threading.Thread(target=worker, args=(1,))
t1.start(); t2.start()
t1.join(); t2.join()
codes = sorted([results[0][0], results[1][0]])
assert codes == [201, 409], codes
def test_chat_with_unknown_member_rolls_back():
name = f"cb_{uuid.uuid4().hex[:8]}"
s, me = _create_user(name)
assert s == 201, me
tok = _login(name)
s, _ = _post("/api/v1/chats", {
"title": "x", "member_ids": [me["id"], str(uuid.uuid4())],
}, token=tok)
assert s == 404
def test_message_to_missing_chat():
name = f"mc_{uuid.uuid4().hex[:8]}"
_create_user(name)
tok = _login(name)
s, _ = _post(f"/api/v1/chats/{uuid.uuid4()}/messages", {"text": "hi"}, token=tok)
assert s == 404