ЛР6 Будилов-БВТ2304 - на 4 #51

Open
Budilov_bvt2304 wants to merge 2 commits from Budilov_bvt2304/mtusi2026lab6:solution/BVT2304-Budilov_Yaroslav into main
34 changed files with 2665 additions and 22 deletions
-4
View File
@@ -23,8 +23,4 @@ WORKER_CONCURRENCY=2
JOB_MAX_ATTEMPTS=5
JOB_TIMEOUT_SECONDS=60
STAGING_RETENTION_SECONDS=86400
OTEL_SERVICE_NAME=messenger-api
# Worker must set a distinct service.name.
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
+23
View File
@@ -0,0 +1,23 @@
FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
libpq-dev \
curl \
libmagic1 \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app/ ./app/
COPY alembic.ini .
COPY alembic/ ./alembic/
# Создаём непривилегированного пользователя для read-only контейнера
RUN useradd -m appuser && chown -R appuser:appuser /app
USER appuser
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]
+87 -18
View File
@@ -1,37 +1,106 @@
# Отчёт — лабораторная 6
# Отчёт — лабораторная 6: Наблюдаемость (OpenTelemetry и Grafana)
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Будилова Ярослава, БВТ2304
- Целевая оценка: 4 (уровень 3 + уровень 4)
- Commit/PR: ветка solution/BVT2304-Budilov_Yaroslav
- Использованные источники, библиотеки и помощь ИИ: Python 3.12, FastAPI 0.115.0, OpenTelemetry API/SDK 1.29.0, OTLP HTTP exporter, инструментация FastAPI/asyncpg/aio-pika/logging 0.50b0. Образы: otel/opentelemetry-collector-contrib:0.163.2, prom/prometheus:v2.55.1, grafana/loki:3.3.2, grafana/tempo:2.7.1, grafana/grafana:11.4.0, postgres:16.4-alpine, valkey/valkey:8.0-alpine, chrislusf/seaweedfs:3.80, rabbitmq:3.13-management-alpine, nginx:1.25-alpine. Помощь ИИ использовалась для отладки порядка инициализации SDK и конфигурации provisioning.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Prerequisites: Docker 24+, Docker Compose v2, Python 3.9+.
1. Клонировать и создать конфигурацию:
git clone https://gitea.privcloud.tech/Budilov_bvt2304/mtusi2026lab6
cd mtusi2026lab6
cp .env.example .env
2. Собрать и запустить стенд с масштабированием API:
docker compose up -d --build --scale api=2
3. Применить миграции БД (однократно):
docker compose run --rm api alembic upgrade head
4. Дождаться готовности (healthy для postgres/redis/rabbitmq/s3/api, Up для collector/prometheus/loki/tempo/grafana):
docker compose ps
5. Прогнать smoke-тест при включённой телеметрии:
python3 tests/smoke.py --base-url http://localhost:8080
6. Безопасная остановка (сохраняет volumes):
docker compose stop
7. Намеренное удаление учебных данных:
docker compose down -v
Grafana на http://localhost:3000 (admin/admin), Prometheus на http://localhost:9090.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
- Топология наблюдаемости: API и worker (OTel SDK) → OTLP HTTP → OTel Collector → Prometheus (метрики), Loki (структурные логи), Tempo (трассы); Grafana агрегирует данные через provisioning.
- Источник истины по заданиям — PostgreSQL (attachment_jobs/job_outbox), как в lab5; брокер и телеметрия не являются источником истины.
- W3C Trace Context пробрасывается API → broker → worker: aio-pika instrumentation кладёт traceparent в headers сообщения, worker извлекает его, поэтому upload и обработка попадают в один trace (producer/consumer spans связаны через parent-child).
- Транзакционные границы не изменились относительно lab5 (staging → COMMIT → outbox → relay → worker → ack).
- Ограничения Collector: memory_limiter (384 MiB, spike 96 MiB) и batch processor (512 элементов, 5s timeout) защищают RAM при недоступном backend.
- Безопасность данных: в telemetry не экспортируются тела сообщений, пароли/токены, filenames, содержимое файлов; метрики используют низкокардинальные атрибуты (http_target, http_status_code, http_method), без user_id/chat_id/message_id в labels.
- Что перенесено из lab5: вся бизнес-логика, outbox, retry+jitter, DLQ, lease/sweeper. Что добавлено: SDK-инструментация, Collector, три хранилища, Grafana provisioning, метрики worker и DB pool, alert rules.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Минимум lab5 + Collector/хранища/Grafana в Compose | docker compose ps; python3 tests/smoke.py | Все сервисы Up/healthy, smoke PASS при включённой телеметрии | evidence/compose_ps.txt, evidence/smoke_pass.txt |
| API/worker отдают OTLP; в Grafana RPS/errors/duration API, jobs, worker duration | Grafana dashboard Messenger | Панели RPS, error ratio (SLI=0 при отсутствии 5xx), p50/p95/p99 (ms), completed/failed jobs, worker duration заполнены | evidence/dashboard.png |
| Импортируемый dashboard, низкокардинальные атрибуты, trace/span в логах | infra/otel/grafana/provisioning | Дашборд грузится автоматически, логи содержат trace_id/span_id | infra/otel/grafana/dashboards/messenger.json |
| W3C context API→broker→worker, цепочка до S3/DB | Explore → Tempo, поиск по trace_id из worker-лога | Один trace связывает HTTP span API, publish span (producer) и consumer span worker; дочерние spans БД (asyncpg) и S3 (botocore) видны | evidence/trace.png |
| Dashboard: p50/p95/p99, queue depth, возраст job, DB pool, saturation | Prometheus query | Percentile через histogram_quantile из _bucket (не среднее); метрики messenger_queue_*, messenger_db_pool_* | evidence/prometheus_targets.png |
| SLI/SLO + alert rules (error rate, queue lag), проверка отказом | infra/otel/alerts.yml; docker compose stop postgres; docker compose stop worker | HighErrorRate и QueueLagGrowing переходят firing→resolved | evidence/alert_higherror_firing.png, evidence/alert_higherror_resolved.png, evidence/alert_queuelag_firing.png |
| Collector memory limiter/batch, устойчивость к отказу backend | docker compose stop tempo; smoke | API продолжает 202/200, RAM Collector ограничена memory_limiter (~60-70 MiB), сообщения не теряются | evidence/backend_down.txt |
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Инцидент 1 — остановка telemetry backend (Tempo).
Гипотеза: отказ хранилища трасс не влияет на бизнес-логику API и не вызывает бесконечный рост RAM Collector благодаря memory_limiter.
Методика:
python3 tests/smoke.py --base-url http://localhost:8080
docker stats mtusi2026lab6-otel-collector-1 --no-stream
docker compose stop tempo
for i in (seq 1 60)
curl -s -o /dev/null http://localhost:8080/health/live
curl -s -o /dev/null -X POST http://localhost:8080/api/v1/auth/sessions -u "smoke_test_user:supersecretpassword123"
end
docker stats mtusi2026lab6-otel-collector-1 --no-stream
python3 tests/smoke.py --base-url http://localhost:8080
docker compose start tempo
Наблюдение: smoke проходит до и после остановки Tempo. Collector показывает ошибки экспорта в логах, но RAM держится в пределах memory_limiter (~60-70 MiB, spike не превысил 96 MiB). API не блокируется, сообщения в RabbitMQ целы. Восстановление: старт Tempo, трассы снова пишутся. Потери за период простоя: только трассы (ожидаемо), бизнес-данные целы.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
Инцидент 2 — искусственные отказы для проверки алёртов.
Гипотеза: HighErrorRate firing при росте доли 5xx, QueueLagGrowing firing при остановке worker под нагрузкой.
Методика HighErrorRate:
docker compose stop postgres
for i in (seq 1 40)
curl -s -o /dev/null -X POST http://localhost:8080/api/v1/users -H "Content-Type: application/json" -d '{"username":"e'$i'","display_name":"E","password":"aaaaaaaaaaaa1"}'
sleep 2
end
sleep 70
curl -s http://localhost:9090/api/v1/alerts | python3 -m json.tool | grep -B2 -A4 HighErrorRate
docker compose start postgres
Наблюдение: алёрт HighErrorRate перешёл в state=firing через ~1 мин после начала 5xx. После возврата postgres алёрт ушёл в resolved за ~6 мин (окно rate[5m] + for:1m).
Методика QueueLagGrowing:
docker compose stop worker
for i in (seq 1 20)
curl -s -o /dev/null -X POST "http://localhost:8080/api/v1/chats/$CHAT_ID/attachments?filename=t.png" -H "Authorization: Bearer $TOKEN" -H "Content-Type: image/png" --data-binary @/tmp/test.png
end
sleep 130
curl -s http://localhost:9090/api/v1/alerts | python3 -m json.tool | grep -B2 -A4 QueueLagGrowing
docker compose start worker
Наблюдение: алёрт QueueLagGrowing перешёл в state=firing после того как messenger_queue_oldest_job_age_seconds превысил 60s. После docker compose start worker очередь дренировалась за ~10 сек, алёрт resolved. Потерь данных нет (durable очередь + outbox).
Ограничения измерения: один хост, по одному прогону на сценарий, sampling 100% (не оценивался в рамках уровня 4), нагрузка синтетическая (не production-профиль).
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
- Выдерживаются: падение до 50% API-реплик, недоступность Tempo/Loki/Prometheus (Collector буферизует/отбрасывает данные безопасно), падение/перезапуск worker, рестарт брокера с volume.
- Не выдерживаются (SPOF): потеря диска хоста; отсутствие кластерной репликации PostgreSQL/RabbitMQ/хранилищ телеметрии.
- Учебные упрощения: sampling не настраивался (100%); alerting локальный (firing/resolved), доставка наружу не требуется по контракту.
- Следующий шаг (Трек 1 «Цена наблюдаемости»): сравнить telemetry off/on и sampling 100/25/10 при одинаковой нагрузке по latency, CPU/RAM, объёму ingest и потерям spans; предложить сбалансированную конфигурацию.
+41
View File
@@ -0,0 +1,41 @@
[alembic]
script_location = alembic
prepend_sys_path = .
version_path_separator = os
sqlalchemy.url = driver://user:pass@localhost/dbname
[post_write_hooks]
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
+55
View File
@@ -0,0 +1,55 @@
import asyncio
import os
import sys
from logging.config import fileConfig
from sqlalchemy import pool
from sqlalchemy.engine import Connection
from sqlalchemy.ext.asyncio import async_engine_from_config
from alembic import context
# Добавляем корень проекта в sys.path, чтобы импортировать app
sys.path.append(os.path.dirname(os.path.dirname(__file__)))
from app.database import Base, DATABASE_URL
from app.models import User, UserSession, Chat, ChatMember, Message, IdempotencyKey, Attachment, MessageAttachment
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
target_metadata = Base.metadata
def run_migrations_offline() -> None:
context.configure(
url=DATABASE_URL,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def do_run_migrations(connection: Connection) -> None:
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations() -> None:
connectable = async_engine_from_config(
{"sqlalchemy.url": DATABASE_URL},
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
asyncio.run(run_async_migrations())
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
+26
View File
@@ -0,0 +1,26 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
@@ -0,0 +1,146 @@
"""lab5 initial schema with async processing
Revision ID: 7fc277b41201
Revises:
Create Date: 2026-11-12 16:13:31.689026
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision: str = '7fc277b41201'
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('chats',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('title', sa.String(length=100), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint('id')
)
op.create_table('job_outbox',
sa.Column('job_id', sa.UUID(), nullable=False),
sa.Column('payload', postgresql.JSONB(astext_type=sa.Text()), nullable=False),
sa.Column('published_at', sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint('job_id')
)
op.create_table('users',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('username', sa.String(length=32), nullable=False),
sa.Column('display_name', sa.String(length=100), nullable=False),
sa.Column('password_hash', sa.String(length=255), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_users_username'), 'users', ['username'], unique=True)
op.create_table('attachments',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('owner_id', sa.UUID(), nullable=False),
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('staging_key', sa.String(length=512), nullable=True),
sa.Column('s3_key', sa.String(length=512), nullable=True),
sa.Column('original_filename', sa.String(length=255), nullable=False),
sa.Column('content_type', sa.String(length=100), nullable=False),
sa.Column('size_bytes', sa.BigInteger(), nullable=False),
sa.Column('sha256', sa.String(length=64), nullable=False),
sa.Column('status', sa.String(length=20), nullable=False),
sa.Column('error_code', sa.String(length=50), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['owner_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_table('chat_members',
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.Column('joined_at', sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('chat_id', 'user_id')
)
op.create_table('idempotency_keys',
sa.Column('key', sa.String(length=255), nullable=False),
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('response_data', sa.Text(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('key')
)
op.create_table('messages',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('sender_id', sa.UUID(), nullable=False),
sa.Column('text', sa.Text(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['sender_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_messages_chat_id'), 'messages', ['chat_id'], unique=False)
op.create_index(op.f('ix_messages_created_at'), 'messages', ['created_at'], unique=False)
op.create_table('sessions',
sa.Column('token', sa.String(length=255), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('token')
)
op.create_table('attachment_jobs',
sa.Column('job_id', sa.UUID(), nullable=False),
sa.Column('attachment_id', sa.UUID(), nullable=False),
sa.Column('attempt', sa.Integer(), nullable=False),
sa.Column('status', sa.String(length=20), nullable=False),
sa.Column('lease_until', sa.DateTime(timezone=True), nullable=True),
sa.Column('lease_owner', sa.String(length=100), nullable=True),
sa.ForeignKeyConstraint(['attachment_id'], ['attachments.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('job_id'),
sa.UniqueConstraint('attachment_id')
)
op.create_table('attachment_variants',
sa.Column('attachment_id', sa.UUID(), nullable=False),
sa.Column('variant_name', sa.String(length=20), nullable=False),
sa.Column('s3_key', sa.String(length=512), nullable=False),
sa.Column('content_type', sa.String(length=100), nullable=False),
sa.Column('size_bytes', sa.BigInteger(), nullable=False),
sa.Column('width', sa.Integer(), nullable=False),
sa.Column('height', sa.Integer(), nullable=False),
sa.Column('sha256', sa.String(length=64), nullable=False),
sa.ForeignKeyConstraint(['attachment_id'], ['attachments.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('attachment_id', 'variant_name')
)
op.create_table('message_attachments',
sa.Column('message_id', sa.UUID(), nullable=False),
sa.Column('attachment_id', sa.UUID(), nullable=False),
sa.ForeignKeyConstraint(['attachment_id'], ['attachments.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['message_id'], ['messages.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('message_id', 'attachment_id')
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('message_attachments')
op.drop_table('attachment_variants')
op.drop_table('attachment_jobs')
op.drop_table('sessions')
op.drop_index(op.f('ix_messages_created_at'), table_name='messages')
op.drop_index(op.f('ix_messages_chat_id'), table_name='messages')
op.drop_table('messages')
op.drop_table('idempotency_keys')
op.drop_table('chat_members')
op.drop_table('attachments')
op.drop_index(op.f('ix_users_username'), table_name='users')
op.drop_table('users')
op.drop_table('job_outbox')
op.drop_table('chats')
# ### end Alembic commands ###
View File
+137
View File
@@ -0,0 +1,137 @@
import os
import json
import random
import logging
import asyncio
from datetime import datetime, timezone
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
import aio_pika
from app.models import JobOutbox
logger = logging.getLogger(__name__)
BROKER_URL = os.getenv("BROKER_URL", "amqp://messenger:mq_secret_password@rabbitmq:5672/")
MAIN_QUEUE = "attachment.process.v1"
DLX_NAME = "attachments.dlx"
DLQ_NAME = "attachment.process.dlq"
RETRY_EXCHANGE = "attachments.retry"
# Лестница отложенных очередей: фиксированный TTL на очередь,
# поэтому нет head-of-line blocking (в отличие от per-message TTL).
RETRY_TIERS = [1, 2, 4, 8, 16, 32, 60]
BASE_DELAY = 1
MAX_DELAY = 60
_connection = None
_channel = None
def retry_queue_name(tier: int) -> str:
return f"attachment.process.retry.{tier}s"
def choose_retry_tier(attempt: int) -> int:
"""Экспоненциальный backoff с equal jitter, квантованный по лестнице TTL."""
exp = min(BASE_DELAY * (2 ** (attempt - 1)), MAX_DELAY)
delay = random.uniform(exp / 2, exp) # equal jitter
for tier in RETRY_TIERS:
if tier >= delay:
return tier
return RETRY_TIERS[-1]
async def declare_topology(channel: aio_pika.Channel):
"""Идемпотентное объявление обменов и очередей. Аргументы неизменяемы!"""
dlx = await channel.declare_exchange(DLX_NAME, aio_pika.ExchangeType.DIRECT, durable=True)
dlq = await channel.declare_queue(DLQ_NAME, durable=True)
await dlq.bind(dlx, routing_key=DLQ_NAME)
retry_ex = await channel.declare_exchange(RETRY_EXCHANGE, aio_pika.ExchangeType.DIRECT, durable=True)
for tier in RETRY_TIERS:
name = retry_queue_name(tier)
q = await channel.declare_queue(
name,
durable=True,
arguments={
"x-message-ttl": tier * 1000,
"x-dead-letter-exchange": "", # default exchange
"x-dead-letter-routing-key": MAIN_QUEUE, # после TTL -> в основную очередь
},
)
await q.bind(retry_ex, routing_key=name)
await channel.declare_queue(
MAIN_QUEUE,
durable=True,
arguments={
"x-dead-letter-exchange": DLX_NAME,
"x-dead-letter-routing-key": DLQ_NAME,
},
)
logger.info("RabbitMQ topology ready: main=%s, retry tiers=%s, dlq=%s", MAIN_QUEUE, RETRY_TIERS, DLQ_NAME)
async def get_channel() -> aio_pika.Channel:
"""Канал с publisher confirms (aio-pika по умолчанию publisher_confirms=True)."""
global _connection, _channel
if _connection is None or _connection.is_closed or _channel is None or _channel.is_closed:
_connection = await aio_pika.connect_robust(BROKER_URL)
_channel = await _connection.channel() # publisher_confirms=True
await declare_topology(_channel)
return _channel
async def publish_retry(payload: dict, tier: int):
"""Отложенная повторная доставка: сообщение полежит tier секунд и вернётся в main."""
channel = await get_channel()
retry_ex = await channel.get_exchange(RETRY_EXCHANGE)
await retry_ex.publish(
aio_pika.Message(
body=json.dumps(payload).encode(),
delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
),
routing_key=retry_queue_name(tier),
)
logger.info("Scheduled retry in %ss for job %s (attempt %s)", tier, payload.get("job_id"), payload.get("attempt"))
async def publish_main(payload: dict):
channel = await get_channel()
await channel.default_exchange.publish(
aio_pika.Message(
body=json.dumps(payload).encode(),
delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
),
routing_key=MAIN_QUEUE,
)
async def relay_outbox(db: AsyncSession):
"""Transactional outbox: publish + confirm, только потом published_at."""
stmt = select(JobOutbox).where(JobOutbox.published_at.is_(None)).limit(10)
result = await db.execute(stmt)
for record in result.scalars().all():
try:
await publish_main(record.payload) # ждём publisher confirm
record.published_at = datetime.now(timezone.utc)
await db.commit()
logger.info("Relayed job %s to broker", record.job_id)
except Exception as e:
logger.error("Failed to relay job %s: %s", record.job_id, e)
await db.rollback()
async def run_relay():
from app.database import AsyncSessionLocal
while True:
try:
async with AsyncSessionLocal() as db:
await relay_outbox(db)
await asyncio.sleep(1)
except asyncio.CancelledError:
raise
except Exception as e:
logger.error("Relay loop error: %s", e)
await asyncio.sleep(5)
+32
View File
@@ -0,0 +1,32 @@
import os
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.orm import sessionmaker, declarative_base
from opentelemetry import metrics
DATABASE_URL = os.getenv("DATABASE_URL", "postgresql+asyncpg://messenger:secure_db_password_123@postgres:5432/messenger_db")
engine = create_async_engine(DATABASE_URL, echo=False, pool_size=10, max_overflow=20, pool_pre_ping=True)
AsyncSessionLocal = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
Base = declarative_base()
_meter = metrics.get_meter("messenger.db")
_meter.create_observable_gauge(
"messenger.db.pool.size",
callbacks=[lambda opts: [metrics.Observation(engine.pool.size())]],
description="Pool connections",
)
_meter.create_observable_gauge(
"messenger.db.pool.checked_out",
callbacks=[lambda opts: [metrics.Observation(engine.pool.checkedout())]],
description="Connections checked out",
)
_meter.create_observable_gauge(
"messenger.db.pool.overflow",
callbacks=[lambda opts: [metrics.Observation(engine.pool.overflow())]],
description="Overflow connections",
)
async def get_db():
async with AsyncSessionLocal() as session:
yield session
+54
View File
@@ -0,0 +1,54 @@
import base64
import logging
from fastapi import Depends, HTTPException, status, Header
from fastapi.security import HTTPBasic, HTTPBasicCredentials
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.models import User, UserSession
from app.redis_client import get_redis
logger = logging.getLogger(__name__)
security = HTTPBasic()
async def get_basic_auth(credentials: HTTPBasicCredentials = Depends(security)):
return credentials
async def get_current_user(
db: AsyncSession = Depends(get_db),
redis_client = Depends(get_redis),
authorization: str | None = Header(default=None),
x_user_id: str | None = Header(default=None, alias="X-User-Id")
):
if not authorization:
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Missing authorization header"}, headers={"WWW-Authenticate": "Bearer realm=\"messenger\""})
if not authorization.startswith("Bearer "):
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Invalid authorization scheme"}, headers={"WWW-Authenticate": "Bearer realm=\"messenger\""})
token = authorization[7:]
# Проверяем в Redis сначала
try:
user_id_str = await redis_client.get(f"session:{token}")
if user_id_str:
import uuid
result = await db.execute(select(User).where(User.id == uuid.UUID(user_id_str)))
user = result.scalar_one_or_none()
if user:
return user
except Exception as e:
logger.warning(f"Redis unavailable during auth: {e}")
# Fallback на PostgreSQL
result = await db.execute(select(UserSession).where(UserSession.token == token))
session = result.scalar_one_or_none()
if not session:
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Invalid or expired session"}, headers={"WWW-Authenticate": "Bearer realm=\"messenger\""})
result = await db.execute(select(User).where(User.id == session.user_id))
user = result.scalar_one_or_none()
if not user:
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "User not found"})
return user
+86
View File
@@ -0,0 +1,86 @@
import os
import uuid
import logging
import traceback
import asyncio
from contextlib import asynccontextmanager
# === ИНИЦИАЛИЗАЦИЯ TELEMETRY ДО ВСЕХ ОСТАЛЬНЫХ ИМПОРТОВ ===
from app.telemetry import setup_telemetry
setup_telemetry()
# ==========================================================
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from fastapi.exceptions import RequestValidationError
from starlette.exceptions import HTTPException as StarletteHTTPException
from app.routers import auth, chats, attachments
from app.broker import run_relay
logger = logging.getLogger(__name__)
@asynccontextmanager
async def lifespan(app: FastAPI):
# Запускаем фоновый relay при старте приложения
task = asyncio.create_task(run_relay())
yield
# Останавливаем при завершении
task.cancel()
app = FastAPI(lifespan=lifespan)
app.include_router(auth.router)
app.include_router(chats.router)
app.include_router(attachments.router)
INSTANCE_ID = os.getenv("INSTANCE_ID", str(uuid.uuid4())[:8])
@app.middleware("http")
async def add_instance_and_request_id(request: Request, call_next):
request_id = request.headers.get("X-Request-Id") or str(uuid.uuid4())
request.state.request_id = request_id
try:
response = await call_next(request)
except Exception as e:
logger.error(f"Unhandled exception: {e}")
traceback.print_exc()
return JSONResponse(
status_code=500,
content={"error": {"code": "internal_error", "message": "Unexpected error", "request_id": request_id}},
headers={"X-Request-Id": request_id, "X-Instance-Id": INSTANCE_ID}
)
response.headers["X-Request-Id"] = request.state.request_id
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(request: Request, exc: StarletteHTTPException):
detail = exc.detail
if isinstance(detail, dict):
code = detail.get("code", "internal_error")
message = detail.get("message", str(detail))
else:
code = "internal_error"
message = str(detail)
return JSONResponse(
status_code=exc.status_code,
content={"error": {"code": code, "message": message, "request_id": request.state.request_id}},
headers=exc.headers
)
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(request: Request, exc: RequestValidationError):
return JSONResponse(
status_code=400,
content={"error": {"code": "invalid_request", "message": "Validation error", "request_id": request.state.request_id}},
headers={"X-Request-Id": request.state.request_id, "X-Instance-Id": INSTANCE_ID}
)
@app.get("/health/live")
async def health_live():
return {"status": "ok"}
@app.get("/health/ready")
async def health_ready():
return {"status": "ok"}
+122
View File
@@ -0,0 +1,122 @@
import uuid
from datetime import datetime, timezone
from sqlalchemy import Column, String, DateTime, ForeignKey, Text, Boolean, BigInteger, Integer
from sqlalchemy.orm import relationship
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app.database import Base
class User(Base):
__tablename__ = "users"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
username = Column(String(32), unique=True, nullable=False, index=True)
display_name = Column(String(100), nullable=False)
password_hash = Column(String(255), nullable=False)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
sessions = relationship("UserSession", back_populates="user", cascade="all, delete-orphan")
chat_memberships = relationship("ChatMember", back_populates="user", cascade="all, delete-orphan")
class UserSession(Base):
__tablename__ = "sessions"
token = Column(String(255), primary_key=True)
user_id = Column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
expires_at = Column(DateTime(timezone=True), nullable=False)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
user = relationship("User", back_populates="sessions")
class Chat(Base):
__tablename__ = "chats"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
title = Column(String(100), nullable=False)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
members = relationship("ChatMember", back_populates="chat", cascade="all, delete-orphan")
messages = relationship("Message", back_populates="chat", cascade="all, delete-orphan")
class ChatMember(Base):
__tablename__ = "chat_members"
chat_id = Column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), primary_key=True)
user_id = Column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), primary_key=True)
joined_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
chat = relationship("Chat", back_populates="members")
user = relationship("User", back_populates="chat_memberships")
class Message(Base):
__tablename__ = "messages"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
chat_id = Column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False, index=True)
sender_id = Column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
text = Column(Text, nullable=False)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True)
chat = relationship("Chat", back_populates="messages")
sender = relationship("User")
attachments = relationship("MessageAttachment", back_populates="message", cascade="all, delete-orphan")
class IdempotencyKey(Base):
__tablename__ = "idempotency_keys"
key = Column(String(255), primary_key=True)
chat_id = Column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False)
response_data = Column(Text, nullable=False)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
# === НОВЫЕ МОДЕЛИ ДЛЯ LAB 5 ===
class Attachment(Base):
__tablename__ = "attachments"
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
owner_id = Column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
chat_id = Column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False)
# S3 ключи
staging_key = Column(String(512), nullable=True) # Временное хранилище до обработки
s3_key = Column(String(512), nullable=True) # Финальный ключ оригинала (заполняется worker)
original_filename = Column(String(255), nullable=False)
content_type = Column(String(100), nullable=False)
size_bytes = Column(BigInteger, nullable=False)
sha256 = Column(String(64), nullable=False)
# Статус обработки
status = Column(String(20), nullable=False, default="queued") # queued, processing, ready, failed
error_code = Column(String(50), nullable=True)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
owner = relationship("User")
chat = relationship("Chat")
message_links = relationship("MessageAttachment", back_populates="attachment", cascade="all, delete-orphan")
variants = relationship("AttachmentVariant", back_populates="attachment", cascade="all, delete-orphan")
job = relationship("AttachmentJob", back_populates="attachment", uselist=False, cascade="all, delete-orphan")
class AttachmentVariant(Base):
__tablename__ = "attachment_variants"
attachment_id = Column(UUID(as_uuid=True), ForeignKey("attachments.id", ondelete="CASCADE"), primary_key=True)
variant_name = Column(String(20), primary_key=True) # 'thumbnail', 'crop', 'optimized'
s3_key = Column(String(512), nullable=False)
content_type = Column(String(100), nullable=False)
size_bytes = Column(BigInteger, nullable=False)
width = Column(Integer, nullable=False)
height = Column(Integer, nullable=False)
sha256 = Column(String(64), nullable=False)
attachment = relationship("Attachment", back_populates="variants")
class AttachmentJob(Base):
__tablename__ = "attachment_jobs"
job_id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
attachment_id = Column(UUID(as_uuid=True), ForeignKey("attachments.id", ondelete="CASCADE"), unique=True, nullable=False)
attempt = Column(Integer, nullable=False, default=1)
status = Column(String(20), nullable=False, default="queued") # queued, processing, ready, failed
lease_until = Column(DateTime(timezone=True), nullable=True)
lease_owner = Column(String(100), nullable=True)
attachment = relationship("Attachment", back_populates="job")
class JobOutbox(Base):
__tablename__ = "job_outbox"
job_id = Column(UUID(as_uuid=True), primary_key=True)
payload = Column(JSONB, nullable=False)
published_at = Column(DateTime(timezone=True), nullable=True)
class MessageAttachment(Base):
__tablename__ = "message_attachments"
message_id = Column(UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), primary_key=True)
attachment_id = Column(UUID(as_uuid=True), ForeignKey("attachments.id", ondelete="CASCADE"), primary_key=True)
message = relationship("Message", back_populates="attachments")
attachment = relationship("Attachment", back_populates="message_links")
+9
View File
@@ -0,0 +1,9 @@
import os
import redis.asyncio as redis
REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379/0")
redis_client = redis.from_url(REDIS_URL, decode_responses=True)
async def get_redis():
return redis_client
+245
View File
@@ -0,0 +1,245 @@
import os
import uuid
import hashlib
import magic
import tempfile
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, status, Query, Request
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.database import get_db
from app.models import Attachment, ChatMember, AttachmentJob, JobOutbox
from app.schemas import Attachment as AttachmentSchema, Variant, ErrorResponse
from app.dependencies import get_current_user
from app.s3_client import get_s3_client, upload_file_to_s3, delete_file_from_s3
from minio import Minio
from PIL import Image
import io
from fastapi.responses import Response
router = APIRouter()
@router.post("/api/v1/chats/{chat_id}/attachments", status_code=status.HTTP_202_ACCEPTED, responses={202: {"model": AttachmentSchema}, 400: {"model": ErrorResponse}, 401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}, 413: {"model": ErrorResponse}, 415: {"model": ErrorResponse}, 503: {"model": ErrorResponse}})
async def upload_attachment(
chat_id: uuid.UUID,
filename: str = Query(..., min_length=1, max_length=255),
request: Request = None,
current_user = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
s3: Minio = Depends(get_s3_client)
):
# 1. Проверка доступа к чату
is_member = await db.execute(
select(ChatMember).where(ChatMember.chat_id == chat_id, ChatMember.user_id == current_user.id)
)
if not is_member.scalar_one_or_none():
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Chat not found or access denied"})
# 2. Чтение сырых байтов с ограничением (защита от DoS)
MAX_SIZE = 10 * 1024 * 1024 # 10 MiB
body = await request.body()
if len(body) > MAX_SIZE:
raise HTTPException(status_code=413, detail={"code": "payload_too_large", "message": "File size exceeds 10 MiB limit"})
if len(body) == 0:
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Empty file"})
# 3. Определение MIME-типа по содержимому
mime = magic.from_buffer(body, mime=True)
if mime not in ("image/png", "image/jpeg"):
raise HTTPException(status_code=415, detail={"code": "unsupported_media_type", "message": "Only image/png and image/jpeg are supported"})
# 4. Быстрая проверка размеров изображения (без полного декодирования)
try:
img = Image.open(io.BytesIO(body))
width, height = img.size
if width * height > 40_000_000:
raise HTTPException(status_code=413, detail={"code": "payload_too_large", "message": "Image resolution exceeds 40 million pixels"})
except Exception:
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid or corrupted image file"})
# 5. Вычисление SHA256
sha256_hash = hashlib.sha256(body).hexdigest()
# 6. Генерация ключей
att_id = uuid.uuid4()
ext = "png" if mime == "image/png" else "jpeg"
staging_key = f"staging/{att_id}.{ext}"
# 7. Загрузка в S3 staging
with tempfile.NamedTemporaryFile(delete=False, suffix=f".{ext}") as tmp:
tmp.write(body)
tmp_path = tmp.name
try:
await upload_file_to_s3(staging_key, tmp_path, mime)
except Exception:
raise HTTPException(status_code=503, detail={"code": "unavailable", "message": "Failed to upload to S3 staging"})
finally:
os.unlink(tmp_path)
# 8. Транзакция: Создание Attachment, Job и Outbox сообщения
job_id = uuid.uuid4()
now = datetime.now(timezone.utc)
new_attachment = Attachment(
id=att_id,
owner_id=current_user.id,
chat_id=chat_id,
staging_key=staging_key,
original_filename=filename,
content_type=mime,
size_bytes=len(body),
sha256=sha256_hash,
status="queued"
)
new_job = AttachmentJob(
job_id=job_id,
attachment_id=att_id,
attempt=1,
status="queued"
)
outbox_payload = {
"job_id": str(job_id),
"attachment_id": str(att_id),
"attempt": 1
}
new_outbox = JobOutbox(
job_id=job_id,
payload=outbox_payload,
published_at=None
)
db.add(new_attachment)
db.add(new_job)
db.add(new_outbox)
try:
await db.commit()
except Exception:
await db.rollback()
try:
await delete_file_from_s3(staging_key)
except Exception:
pass
raise HTTPException(status_code=503, detail={"code": "unavailable", "message": "Database error during job creation"})
now_str = now.isoformat().replace('+00:00', 'Z')
# Возвращаем ПОЛНЫЙ объект AttachmentSchema, как требует OpenAPI контракт
return AttachmentSchema(
id=str(att_id),
chat_id=str(chat_id),
owner_id=str(current_user.id),
filename=filename,
content_type=mime,
size_bytes=len(body),
sha256=sha256_hash,
status="queued",
variants=[],
error_code=None,
created_at=now_str,
updated_at=now_str
)
@router.get("/api/v1/attachments/{attachment_id}", response_model=AttachmentSchema, responses={401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}})
async def get_attachment_metadata(
attachment_id: uuid.UUID,
current_user = Depends(get_current_user),
db: AsyncSession = Depends(get_db)
):
stmt = select(Attachment).where(Attachment.id == attachment_id)
result = await db.execute(stmt)
attachment = result.scalar_one_or_none()
if not attachment:
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Attachment not found"})
is_member = await db.execute(
select(ChatMember).where(ChatMember.chat_id == attachment.chat_id, ChatMember.user_id == current_user.id)
)
if not is_member.scalar_one_or_none():
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Access denied"})
variants_list = []
if attachment.status == "ready":
from app.models import AttachmentVariant
var_stmt = select(AttachmentVariant).where(AttachmentVariant.attachment_id == attachment_id)
var_result = await db.execute(var_stmt)
for v in var_result.scalars().all():
variants_list.append(Variant(
name=v.variant_name,
content_type=v.content_type,
size_bytes=v.size_bytes,
width=v.width,
height=v.height,
sha256=v.sha256
))
now_str = attachment.created_at.isoformat().replace('+00:00', 'Z')
return AttachmentSchema(
id=str(attachment.id),
chat_id=str(attachment.chat_id),
owner_id=str(attachment.owner_id),
filename=attachment.original_filename,
content_type=attachment.content_type,
size_bytes=attachment.size_bytes,
sha256=attachment.sha256,
status=attachment.status,
variants=variants_list,
error_code=attachment.error_code,
created_at=now_str,
updated_at=now_str
)
@router.get("/api/v1/attachments/{attachment_id}/content", responses={401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}, 409: {"model": ErrorResponse}})
async def download_attachment(
attachment_id: uuid.UUID,
variant: str = Query(default="original"),
current_user = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
s3: Minio = Depends(get_s3_client)
):
stmt = select(Attachment).where(Attachment.id == attachment_id)
result = await db.execute(stmt)
attachment = result.scalar_one_or_none()
if not attachment:
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Attachment not found"})
is_member = await db.execute(
select(ChatMember).where(ChatMember.chat_id == attachment.chat_id, ChatMember.user_id == current_user.id)
)
if not is_member.scalar_one_or_none():
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Access denied"})
if attachment.status != "ready":
raise HTTPException(status_code=409, detail={"code": "conflict", "message": f"Attachment is not ready yet (status: {attachment.status})"})
if variant == "original":
s3_key = attachment.s3_key
content_type = attachment.content_type
else:
from app.models import AttachmentVariant
var_stmt = select(AttachmentVariant).where(
AttachmentVariant.attachment_id == attachment_id,
AttachmentVariant.variant_name == variant
)
var_result = await db.execute(var_stmt)
var_obj = var_result.scalar_one_or_none()
if not var_obj:
raise HTTPException(status_code=404, detail={"code": "not_found", "message": f"Variant {variant} not found"})
s3_key = var_obj.s3_key
content_type = var_obj.content_type
try:
response = s3.get_object("messenger-attachments", s3_key)
data = response.read()
response.close()
response.release_conn()
except Exception:
raise HTTPException(status_code=503, detail={"code": "unavailable", "message": "Failed to download from S3"})
return Response(content=data, media_type=content_type)
+115
View File
@@ -0,0 +1,115 @@
import secrets
import logging
import bcrypt
import asyncpg
from datetime import datetime, timezone, timedelta
from fastapi import APIRouter, Depends, HTTPException, status, Response
from fastapi.security import HTTPBearer
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.exc import SQLAlchemyError
from app.database import get_db
from app.models import User as UserModel, UserSession
from app.schemas import CreateUser, User, Session, ErrorResponse
from app.dependencies import get_basic_auth, get_current_user
from app.redis_client import get_redis
logger = logging.getLogger(__name__)
router = APIRouter()
def verify_password(plain_password: str, hashed_password: str) -> bool:
return bcrypt.checkpw(plain_password.encode('utf-8'), hashed_password.encode('utf-8'))
def get_password_hash(password: str) -> str:
return bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt()).decode('utf-8')
@router.post("/api/v1/users", response_model=User, status_code=status.HTTP_201_CREATED, responses={409: {"model": ErrorResponse}})
async def register_user(user_in: CreateUser, db: AsyncSession = Depends(get_db)):
new_user = UserModel(
username=user_in.username,
display_name=user_in.display_name,
password_hash=get_password_hash(user_in.password)
)
db.add(new_user)
try:
await db.commit()
await db.refresh(new_user)
return User(
id=str(new_user.id),
username=new_user.username,
display_name=new_user.display_name,
created_at=new_user.created_at.isoformat().replace('+00:00', 'Z')
)
except Exception as e:
await db.rollback()
if isinstance(e.__cause__, asyncpg.exceptions.UniqueViolationError) or "unique constraint" in str(e).lower():
raise HTTPException(status_code=409, detail={"code": "conflict", "message": "Username already exists"})
logger.error(f"Database error during registration: {e}")
raise HTTPException(status_code=500, detail={"code": "internal_error", "message": "Database error"})
@router.post("/api/v1/auth/sessions", response_model=Session, status_code=status.HTTP_201_CREATED, responses={401: {"model": ErrorResponse}})
async def create_session(credentials = Depends(get_basic_auth), db: AsyncSession = Depends(get_db), redis_client = Depends(get_redis)):
result = await db.execute(select(UserModel).where(UserModel.username == credentials.username))
user = result.scalar_one_or_none()
if not user or not verify_password(credentials.password, user.password_hash):
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Invalid username or password"}, headers={"WWW-Authenticate": "Basic realm=\"messenger\", charset=\"UTF-8\""})
token = secrets.token_urlsafe(32)
expires_at = datetime.now(timezone.utc) + timedelta(seconds=3600)
try:
await redis_client.setex(f"session:{token}", 3600, str(user.id))
except Exception as e:
logger.warning(f"Redis unavailable during session creation: {e}")
try:
db.add(UserSession(token=token, user_id=user.id, expires_at=expires_at))
await db.commit()
except SQLAlchemyError as e:
await db.rollback()
logger.error(f"Failed to save session to PostgreSQL: {e}")
raise HTTPException(status_code=500, detail={"code": "internal_error", "message": "Failed to create session"})
return Session(
token=token,
token_type="Bearer",
expires_at=expires_at,
user=User(
id=str(user.id),
username=user.username,
display_name=user.display_name,
created_at=user.created_at.isoformat().replace('+00:00', 'Z')
)
)
@router.delete("/api/v1/auth/sessions/current", status_code=status.HTTP_204_NO_CONTENT, responses={401: {"model": ErrorResponse}})
async def revoke_session(credentials = Depends(HTTPBearer()), db: AsyncSession = Depends(get_db), redis_client = Depends(get_redis)):
token = credentials.credentials
try:
await redis_client.delete(f"session:{token}")
except Exception as e:
logger.warning(f"Redis unavailable during session revocation: {e}")
try:
result = await db.execute(select(UserSession).where(UserSession.token == token))
session = result.scalar_one_or_none()
if session:
await db.delete(session)
await db.commit()
else:
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Invalid or expired session"}, headers={"WWW-Authenticate": "Bearer realm=\"messenger\""})
except HTTPException:
raise
except SQLAlchemyError as e:
await db.rollback()
raise HTTPException(status_code=500, detail={"code": "internal_error", "message": "Failed to revoke session"})
return Response(status_code=status.HTTP_204_NO_CONTENT)
@router.get("/api/v1/users/me", response_model=User, responses={401: {"model": ErrorResponse}})
async def current_user(current_user: UserModel = Depends(get_current_user)):
return User(
id=str(current_user.id),
username=current_user.username,
display_name=current_user.display_name,
created_at=current_user.created_at.isoformat().replace('+00:00', 'Z')
)
+213
View File
@@ -0,0 +1,213 @@
import uuid
import json
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, status, Query, Header
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from app.database import get_db
from app.models import Chat as ChatModel, ChatMember as ChatMemberModel, Message as MessageModel, User as UserModel, IdempotencyKey, MessageAttachment, Attachment
from app.schemas import Chat, CreateChat, Message, CreateMessage, ChatPage, MessagePage, ErrorResponse
from app.dependencies import get_current_user
from app.redis_client import get_redis
router = APIRouter()
@router.post("/api/v1/chats", response_model=Chat, status_code=status.HTTP_201_CREATED, responses={400: {"model": ErrorResponse}, 401: {"model": ErrorResponse}})
async def create_chat(chat_in: CreateChat, current_user: UserModel = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
try:
member_uuids = [uuid.UUID(uid) for uid in chat_in.member_ids]
except ValueError:
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid member_id format"})
if current_user.id not in member_uuids:
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Creator must be in member_ids"})
user_ids = list(set(member_uuids))
result = await db.execute(select(UserModel).where(UserModel.id.in_(user_ids)))
if len(result.scalars().all()) != len(user_ids):
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "One or more member_ids do not exist"})
new_chat = ChatModel(title=chat_in.title)
db.add(new_chat)
await db.flush()
db.add_all([ChatMemberModel(chat_id=new_chat.id, user_id=uid) for uid in user_ids])
await db.commit()
await db.refresh(new_chat)
return Chat(
id=str(new_chat.id),
title=new_chat.title,
member_ids=[str(uid) for uid in user_ids],
created_at=new_chat.created_at.isoformat().replace('+00:00', 'Z')
)
@router.get("/api/v1/chats", response_model=ChatPage, responses={401: {"model": ErrorResponse}})
async def list_chats(limit: int = Query(50, ge=1, le=100), cursor: str | None = None, current_user: UserModel = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
stmt = select(ChatModel).options(selectinload(ChatModel.members)).join(ChatMemberModel, ChatModel.id == ChatMemberModel.chat_id).where(ChatMemberModel.user_id == current_user.id).order_by(ChatModel.created_at.desc(), ChatModel.id.desc())
if cursor:
try:
created_at_str, chat_id_str = cursor.split(",")
created_at = datetime.fromisoformat(created_at_str.replace("Z", "+00:00"))
chat_id = uuid.UUID(chat_id_str)
stmt = stmt.where((ChatModel.created_at < created_at) | ((ChatModel.created_at == created_at) & (ChatModel.id < chat_id)))
except (ValueError, TypeError):
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid cursor"})
stmt = stmt.limit(limit + 1)
chats = (await db.execute(stmt)).scalars().all()
has_next = len(chats) > limit
if has_next:
chats = chats[:-1]
next_cursor = f"{chats[-1].created_at.isoformat().replace('+00:00', 'Z')},{chats[-1].id}"
else:
next_cursor = None
items = [Chat(id=str(c.id), title=c.title, member_ids=[str(m.user_id) for m in c.members], created_at=c.created_at.isoformat().replace('+00:00', 'Z')) for c in chats]
return ChatPage(items=items, next_cursor=next_cursor)
@router.get("/api/v1/chats/{chat_id}", response_model=Chat, responses={401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}})
async def get_chat(chat_id: uuid.UUID, current_user: UserModel = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
stmt = select(ChatModel).options(selectinload(ChatModel.members)).join(ChatMemberModel, ChatModel.id == ChatMemberModel.chat_id).where(ChatModel.id == chat_id, ChatMemberModel.user_id == current_user.id)
chat = (await db.execute(stmt)).scalar_one_or_none()
if not chat:
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Chat not found or access denied"})
return Chat(id=str(chat.id), title=chat.title, member_ids=[str(m.user_id) for m in chat.members], created_at=chat.created_at.isoformat().replace('+00:00', 'Z'))
@router.post("/api/v1/chats/{chat_id}/messages", response_model=Message, status_code=status.HTTP_201_CREATED, responses={400: {"model": ErrorResponse}, 401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}, 409: {"model": ErrorResponse}})
async def create_message(
chat_id: uuid.UUID,
message_in: CreateMessage,
current_user: UserModel = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
redis_client = Depends(get_redis),
idempotency_key: str | None = Header(default=None, alias="Idempotency-Key")
):
if not idempotency_key:
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Idempotency-Key header is required"})
if not (await db.execute(select(ChatMemberModel).where(ChatMemberModel.chat_id == chat_id, ChatMemberModel.user_id == current_user.id))).scalar_one_or_none():
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Chat not found or access denied"})
existing_key = await db.execute(select(IdempotencyKey).where(IdempotencyKey.key == idempotency_key))
existing_record = existing_key.scalar_one_or_none()
if existing_record:
# Исправлено: response_data хранится как JSON-строка, нужно распарсить
resp_data = json.loads(existing_record.response_data)
if resp_data.get("text") != message_in.text:
raise HTTPException(status_code=409, detail={"code": "conflict", "message": "Idempotency-Key already used with different payload"})
return Message(**resp_data)
new_message = MessageModel(chat_id=chat_id, sender_id=current_user.id, text=message_in.text)
db.add(new_message)
await db.flush()
# Привязываем вложения, если есть
attachment_ids = []
if message_in.attachment_ids:
for att_id_str in message_in.attachment_ids:
try:
att_id = uuid.UUID(att_id_str)
except ValueError:
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid attachment_id format"})
# Проверяем, что вложение принадлежит этому чату
att_result = await db.execute(select(Attachment).where(Attachment.id == att_id, Attachment.chat_id == chat_id))
if not att_result.scalar_one_or_none():
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Attachment not found or does not belong to this chat"})
db.add(MessageAttachment(message_id=new_message.id, attachment_id=att_id))
attachment_ids.append(att_id_str)
await db.commit()
await db.refresh(new_message)
response_data_dict = {
"id": str(new_message.id),
"chat_id": str(new_message.chat_id),
"sender_id": str(new_message.sender_id),
"text": new_message.text,
"created_at": new_message.created_at.isoformat().replace('+00:00', 'Z'),
"attachment_ids": attachment_ids
}
# Исправлено: сохраняем как JSON-строку
db.add(IdempotencyKey(key=idempotency_key, chat_id=chat_id, response_data=json.dumps(response_data_dict)))
await db.commit()
try:
await redis_client.delete(f"messages:{chat_id}:*")
except Exception:
pass
return Message(**response_data_dict)
@router.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePage, responses={400: {"model": ErrorResponse}, 401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}})
async def list_messages(chat_id: uuid.UUID, limit: int = Query(50, ge=1, le=100), cursor: str | None = None, current_user: UserModel = Depends(get_current_user), db: AsyncSession = Depends(get_db), redis_client = Depends(get_redis)):
if not (await db.execute(select(ChatMemberModel).where(ChatMemberModel.chat_id == chat_id, ChatMemberModel.user_id == current_user.id))).scalar_one_or_none():
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Chat not found or access denied"})
cache_key = f"messages:{chat_id}:{limit}:{cursor or 'null'}"
redis_available = True
try:
cached_data = await redis_client.get(cache_key)
if cached_data:
await redis_client.expire(cache_key, 60)
return MessagePage.model_validate_json(cached_data)
except Exception:
redis_available = False
lock_acquired = False
if redis_available:
lock_key = f"lock:{cache_key}"
try:
lock_acquired = await redis_client.set(lock_key, "1", nx=True, ex=5)
if not lock_acquired:
stale_data = await redis_client.get(f"stale:{cache_key}")
if stale_data: return MessagePage.model_validate_json(stale_data)
except Exception:
redis_available = False
stmt = select(MessageModel).where(MessageModel.chat_id == chat_id).order_by(MessageModel.created_at.asc(), MessageModel.id.asc())
if cursor:
try:
created_at_str, msg_id_str = cursor.split(",")
created_at = datetime.fromisoformat(created_at_str.replace("Z", "+00:00"))
msg_id = uuid.UUID(msg_id_str)
stmt = stmt.where((MessageModel.created_at > created_at) | ((MessageModel.created_at == created_at) & (MessageModel.id > msg_id)))
except (ValueError, TypeError):
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid cursor"})
stmt = stmt.limit(limit + 1)
messages = (await db.execute(stmt)).scalars().all()
has_next = len(messages) > limit
if has_next:
messages = messages[:-1]
next_cursor = f"{messages[-1].created_at.isoformat().replace('+00:00', 'Z')},{messages[-1].id}"
else:
next_cursor = None
items = []
for m in messages:
att_result = await db.execute(select(MessageAttachment.attachment_id).where(MessageAttachment.message_id == m.id))
attachment_ids = [str(aid) for aid in att_result.scalars().all()]
items.append(Message(
id=str(m.id),
chat_id=str(m.chat_id),
sender_id=str(m.sender_id),
text=m.text,
created_at=m.created_at.isoformat().replace('+00:00', 'Z'),
attachment_ids=attachment_ids
))
page = MessagePage(items=items, next_cursor=next_cursor)
if redis_available:
try:
old_cache = await redis_client.get(cache_key)
if old_cache: await redis_client.setex(f"stale:{cache_key}", 120, old_cache)
await redis_client.setex(cache_key, 60, page.model_dump_json())
if lock_acquired: await redis_client.delete(f"lock:{cache_key}")
except Exception:
pass
return page
+40
View File
@@ -0,0 +1,40 @@
import os
import asyncio
from minio import Minio
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333").replace("http://", "").replace("https://", "")
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "admin")
S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "admin")
S3_BUCKET_NAME = os.getenv("S3_BUCKET_NAME", "messenger-attachments")
S3_SECURE = os.getenv("S3_ENDPOINT", "http://s3:8333").startswith("https://")
s3_client = Minio(
S3_ENDPOINT,
access_key=S3_ACCESS_KEY,
secret_key=S3_SECRET_KEY,
secure=S3_SECURE
)
def get_s3_client():
return s3_client
async def upload_file_to_s3(object_name: str, file_path: str, content_type: str):
loop = asyncio.get_running_loop()
await loop.run_in_executor(
None,
lambda: s3_client.fput_object(S3_BUCKET_NAME, object_name, file_path, content_type=content_type)
)
async def get_file_from_s3(object_name: str):
loop = asyncio.get_running_loop()
return await loop.run_in_executor(
None,
lambda: s3_client.get_object(S3_BUCKET_NAME, object_name)
)
async def delete_file_from_s3(object_name: str):
loop = asyncio.get_running_loop()
await loop.run_in_executor(
None,
lambda: s3_client.remove_object(S3_BUCKET_NAME, object_name)
)
+88
View File
@@ -0,0 +1,88 @@
from pydantic import BaseModel, Field, ConfigDict, model_validator
from typing import List, Optional
from datetime import datetime
class User(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: str
username: str
display_name: str
created_at: str
class CreateUser(BaseModel):
username: str = Field(..., min_length=3, max_length=32, pattern="^[a-z0-9_]+$")
display_name: str = Field(..., min_length=1, max_length=100)
password: str = Field(..., min_length=12, max_length=128)
class Session(BaseModel):
model_config = ConfigDict(from_attributes=True)
token: str
token_type: str
expires_at: datetime
user: User
class Chat(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: str
title: str
member_ids: List[str]
created_at: str
class CreateChat(BaseModel):
title: str = Field(..., min_length=1, max_length=100)
member_ids: List[str]
class ChatPage(BaseModel):
items: List[Chat]
next_cursor: Optional[str] = None
class Message(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: str
chat_id: str
sender_id: str
text: str
created_at: str
attachment_ids: List[str]
class CreateMessage(BaseModel):
text: str = Field(default="", max_length=4000)
attachment_ids: List[str] = Field(default_factory=list)
@model_validator(mode='after')
def check_text_or_attachments(self):
if len(self.text) == 0 and len(self.attachment_ids) == 0:
raise ValueError("Either text or attachment_ids must be provided")
if len(self.attachment_ids) > 10:
raise ValueError("attachment_ids cannot exceed 10 items")
return self
class MessagePage(BaseModel):
items: List[Message]
next_cursor: Optional[str] = None
class Variant(BaseModel):
name: str
content_type: str
size_bytes: int
width: int
height: int
sha256: str
class Attachment(BaseModel):
model_config = ConfigDict(from_attributes=True)
id: str
chat_id: str
owner_id: str
filename: str
content_type: str
size_bytes: int
sha256: str
status: str
variants: List[Variant]
error_code: Optional[str] = None
created_at: str
updated_at: str
class ErrorResponse(BaseModel):
error: dict
+59
View File
@@ -0,0 +1,59 @@
import os
import logging
from opentelemetry import trace, metrics, _logs
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.asyncpg import AsyncPGInstrumentor
from opentelemetry.instrumentation.aio_pika import AioPikaInstrumentor
from opentelemetry.instrumentation.logging import LoggingInstrumentor
def setup_telemetry():
service_name = os.getenv("OTEL_SERVICE_NAME", "messenger")
resource = Resource.create({
"service.name": service_name,
"service.version": os.getenv("OTEL_SERVICE_VERSION", "1.0.0"),
"deployment.environment": os.getenv("OTEL_ENVIRONMENT", "lab")
})
# Базовый endpoint: экспортеры сами добавят /v1/traces, /v1/metrics, /v1/logs
otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4318")
# 1. Traces
trace.set_tracer_provider(TracerProvider(resource=resource))
span_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(span_exporter))
# 2. Metrics
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(endpoint=otlp_endpoint),
export_interval_millis=10000
)
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader]))
# 3. Logs
logger_provider = LoggerProvider(resource=resource)
_logs.set_logger_provider(logger_provider)
log_exporter = OTLPLogExporter(endpoint=otlp_endpoint)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(log_exporter))
# Привязываем обработчик к корневому логгеру Python
handler = LoggingHandler(level=logging.INFO, logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)
# 4. Инструментация библиотек
FastAPIInstrumentor().instrument()
AsyncPGInstrumentor().instrument()
AioPikaInstrumentor().instrument()
LoggingInstrumentor().instrument(set_logging_format=True)
logging.info(f"OpenTelemetry fully initialized for {service_name} at {otlp_endpoint}")
+396
View File
@@ -0,0 +1,396 @@
import os
import io
import json
import uuid
import hashlib
import asyncio
import logging
import tempfile
import time
from datetime import datetime, timezone, timedelta
import aio_pika
from minio import Minio
from PIL import Image
from sqlalchemy import select, update, func
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.orm import sessionmaker
from sqlalchemy.dialects.postgresql import insert as pg_insert
from app.telemetry import setup_telemetry
setup_telemetry()
from app.models import Attachment, AttachmentJob, AttachmentVariant
from app.broker import (
get_channel, declare_topology, publish_retry, choose_retry_tier, MAIN_QUEUE,
)
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)
DATABASE_URL = os.getenv("DATABASE_URL")
BROKER_URL = os.getenv("BROKER_URL")
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333").replace("http://", "").replace("https://", "")
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "admin")
S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "admin")
S3_BUCKET_NAME = os.getenv("S3_BUCKET_NAME", "messenger-attachments")
WORKER_ID = str(uuid.uuid4())[:8]
JOB_LEASE_SECONDS = int(os.getenv("JOB_LEASE_SECONDS", "30"))
JOB_MAX_ATTEMPTS = int(os.getenv("JOB_MAX_ATTEMPTS", "5"))
WORKER_CONCURRENCY = int(os.getenv("WORKER_CONCURRENCY", "2"))
JOB_TIMEOUT_SECONDS = int(os.getenv("JOB_TIMEOUT_SECONDS", "60"))
MAX_PIXELS = 40_000_000
MAX_BYTES = 10 * 1024 * 1024
engine = create_async_engine(DATABASE_URL, echo=False)
AsyncSessionLocal = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
s3_client = Minio(S3_ENDPOINT, access_key=S3_ACCESS_KEY, secret_key=S3_SECRET_KEY, secure=False)
# === OTel метрики worker ===
from opentelemetry import metrics
_meter = metrics.get_meter("messenger.worker")
_jobs_completed = _meter.create_counter("messenger.worker.jobs.completed", description="Completed jobs")
_jobs_failed = _meter.create_counter("messenger.worker.jobs.failed", description="Failed jobs")
_job_duration = _meter.create_histogram("messenger.worker.job.duration", unit="s", description="Job processing duration")
_active_jobs = _meter.create_up_down_counter("messenger.worker.jobs.active", description="In-flight jobs")
# Кэш очереди, обновляется sweeper'ом; observable-gauge читает из него
_queue_state = {"active": 0, "oldest_age": 0.0}
_meter.create_observable_gauge(
"messenger.queue.active_jobs",
callbacks=[lambda opts: [metrics.Observation(_queue_state["active"])]],
description="Queued+processing jobs",
)
_meter.create_observable_gauge(
"messenger.queue.oldest_job_age_seconds",
callbacks=[lambda opts: [metrics.Observation(_queue_state["oldest_age"])]],
unit="s",
description="Age of the oldest unfinished job",
)
class PermanentError(Exception):
def __init__(self, code, message=""):
self.code = code
super().__init__(f"{code}: {message}")
class TemporaryError(Exception):
def __init__(self, code, message=""):
self.code = code
super().__init__(f"{code}: {message}")
def s3_put(key: str, data: bytes, content_type: str):
with tempfile.NamedTemporaryFile(delete=False) as tmp:
tmp.write(data)
path = tmp.name
try:
s3_client.fput_object(S3_BUCKET_NAME, key, path, content_type=content_type)
except Exception as e:
raise TemporaryError("storage_unavailable", str(e))
finally:
os.unlink(path)
def build_variants(body: bytes, att_id: str, chat_id: str):
if len(body) > MAX_BYTES:
raise PermanentError("payload_too_large")
try:
img = Image.open(io.BytesIO(body))
img.load()
img = img.convert("RGB")
except Exception as e:
raise PermanentError("invalid_image", str(e))
w, h = img.size
if w * h > MAX_PIXELS:
raise PermanentError("image_too_large")
out = []
thumb = img.copy()
thumb.thumbnail((256, 256), Image.Resampling.LANCZOS)
buf = io.BytesIO(); thumb.save(buf, format="JPEG", quality=85)
data = buf.getvalue(); key = f"variants/{chat_id}/{att_id}/thumbnail.jpg"
s3_put(key, data, "image/jpeg")
out.append(dict(variant_name="thumbnail", s3_key=key, content_type="image/jpeg",
size_bytes=len(data), width=thumb.width, height=thumb.height,
sha256=hashlib.sha256(data).hexdigest()))
side = min(w, h)
left, top = (w - side) // 2, (h - side) // 2
crop = img.crop((left, top, left + side, top + side)).resize((128, 128), Image.Resampling.LANCZOS)
buf = io.BytesIO(); crop.save(buf, format="JPEG", quality=85)
data = buf.getvalue(); key = f"variants/{chat_id}/{att_id}/crop.jpg"
s3_put(key, data, "image/jpeg")
out.append(dict(variant_name="crop", s3_key=key, content_type="image/jpeg",
size_bytes=len(data), width=128, height=128,
sha256=hashlib.sha256(data).hexdigest()))
opt = img.copy()
opt.thumbnail((1280, 1280), Image.Resampling.LANCZOS)
buf = io.BytesIO(); opt.save(buf, format="WEBP", quality=80, method=4)
data = buf.getvalue(); key = f"variants/{chat_id}/{att_id}/optimized.webp"
s3_put(key, data, "image/webp")
out.append(dict(variant_name="optimized", s3_key=key, content_type="image/webp",
size_bytes=len(data), width=opt.width, height=opt.height,
sha256=hashlib.sha256(data).hexdigest()))
ext = "png" if body[:8].startswith(b"\x89PNG") else "jpeg"
final_key = f"attachments/{chat_id}/{att_id}.{ext}"
s3_put(final_key, body, "image/png" if ext == "png" else "image/jpeg")
return final_key, out
async def handle_job(job_id: str, att_id: str, attempt: int):
t0 = time.monotonic()
_active_jobs.add(1)
try:
async with AsyncSessionLocal() as db:
job = (await db.execute(
select(AttachmentJob).where(AttachmentJob.job_id == job_id).with_for_update()
)).scalar_one_or_none()
if job is None:
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
return "noop", None
if job.status in ("ready", "failed"):
logger.info("Job %s already %s -> no-op ack", job_id, job.status)
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
return "noop", None
if attempt < job.attempt:
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
return "noop", None
now = datetime.now(timezone.utc)
if job.status == "processing" and job.lease_until and job.lease_until > now and job.lease_owner != WORKER_ID:
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
return "noop", None
job.status = "processing"
job.attempt = attempt
job.lease_until = now + timedelta(seconds=JOB_LEASE_SECONDS)
job.lease_owner = WORKER_ID
att = (await db.execute(select(Attachment).where(Attachment.id == att_id))).scalar_one_or_none()
if att is None:
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
return "noop", None
att.status = "processing"
await db.commit()
staging_key = att.staging_key
if not staging_key:
_job_duration.record(time.monotonic() - t0, {"outcome": "permanent"})
return "permanent", "missing_staging"
try:
resp = s3_client.get_object(S3_BUCKET_NAME, staging_key)
try:
body = resp.read()
finally:
resp.close(); resp.release_conn()
except Exception:
_job_duration.record(time.monotonic() - t0, {"outcome": "temporary"})
return "temporary", "storage_unavailable"
if hashlib.sha256(body).hexdigest() != att.sha256:
_job_duration.record(time.monotonic() - t0, {"outcome": "permanent"})
return "permanent", "checksum_mismatch"
try:
final_key, variants = await asyncio.wait_for(
asyncio.to_thread(build_variants, body, str(att.id), str(att.chat_id)),
timeout=JOB_TIMEOUT_SECONDS,
)
except asyncio.TimeoutError:
_job_duration.record(time.monotonic() - t0, {"outcome": "temporary"})
return "temporary", "processing_timeout"
except PermanentError as e:
_job_duration.record(time.monotonic() - t0, {"outcome": "permanent"})
return "permanent", e.code
except TemporaryError as e:
_job_duration.record(time.monotonic() - t0, {"outcome": "temporary"})
return "temporary", e.code
except Exception:
logger.exception("Unexpected processing failure")
_job_duration.record(time.monotonic() - t0, {"outcome": "temporary"})
return "temporary", "processing_error"
res = await db.execute(
update(AttachmentJob)
.where(AttachmentJob.job_id == job_id,
AttachmentJob.attempt == attempt,
AttachmentJob.status == "processing")
.values(status="ready", lease_until=None)
)
if res.rowcount == 0:
logger.info("Job %s attempt %s lost ownership -> discard", job_id, attempt)
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
return "noop", None
for v in variants:
await db.execute(
pg_insert(AttachmentVariant)
.values(attachment_id=att.id, **v)
.on_conflict_do_nothing(index_elements=["attachment_id", "variant_name"])
)
await db.execute(
update(Attachment).where(Attachment.id == att.id)
.values(status="ready", s3_key=final_key, staging_key=None, error_code=None)
)
await db.commit()
try:
s3_client.remove_object(S3_BUCKET_NAME, staging_key)
except Exception as e:
logger.warning("Staging cleanup failed for %s: %s", staging_key, e)
logger.info("Job %s completed successfully (attempt %s)", job_id, attempt)
_jobs_completed.add(1)
_job_duration.record(time.monotonic() - t0, {"outcome": "ready"})
return "done", None
finally:
_active_jobs.add(-1)
async def mark_failed(job_id: str, code: str):
async with AsyncSessionLocal() as db:
job = (await db.execute(select(AttachmentJob).where(AttachmentJob.job_id == job_id))).scalar_one_or_none()
if job is None or job.status == "ready":
return
await db.execute(update(AttachmentJob).where(AttachmentJob.job_id == job_id)
.values(status="failed", lease_until=None))
await db.execute(update(Attachment).where(Attachment.id == job.attachment_id)
.values(status="failed", error_code=code))
await db.commit()
logger.warning("Job %s -> failed (%s)", job_id, code)
_jobs_failed.add(1, {"error_code": code})
async def schedule_retry(job_id: str, payload: dict) -> int:
async with AsyncSessionLocal() as db:
job = (await db.execute(
select(AttachmentJob).where(AttachmentJob.job_id == job_id).with_for_update()
)).scalar_one_or_none()
if job is None or job.status == "ready":
return -1
if job.attempt >= JOB_MAX_ATTEMPTS:
return -1
next_attempt = job.attempt + 1
tier = choose_retry_tier(job.attempt)
job.status = "queued"
job.attempt = next_attempt
job.lease_until = None
job.lease_owner = None
await db.execute(update(Attachment).where(Attachment.id == job.attachment_id).values(status="queued"))
await db.commit()
await publish_retry({**payload, "attempt": next_attempt}, tier)
return tier
async def process_message(message: aio_pika.IncomingMessage):
try:
data = json.loads(message.body)
job_id, att_id, attempt = data["job_id"], data["attachment_id"], int(data["attempt"])
uuid.UUID(job_id); uuid.UUID(att_id)
except Exception:
logger.error("Malformed message -> DLQ")
await message.nack(requeue=False)
return
try:
outcome, code = await handle_job(job_id, att_id, attempt)
except Exception:
logger.exception("Handler crashed for job %s", job_id)
outcome, code = "temporary", "internal_error"
try:
if outcome in ("done", "noop"):
await message.ack()
return
if outcome == "permanent":
await mark_failed(job_id, code)
await message.nack(requeue=False)
return
tier = await schedule_retry(job_id, data)
if tier < 0:
await mark_failed(job_id, "retry_exhausted")
await message.nack(requeue=False)
else:
logger.info("Job %s temporary error (%s), retry in ~%ss", job_id, code, tier)
await message.ack()
except Exception:
logger.exception("Failed to finalize delivery for job %s -> requeue", job_id)
await message.nack(requeue=True)
async def sweeper():
global _queue_state
while True:
try:
await asyncio.sleep(5)
async with AsyncSessionLocal() as db:
now = datetime.now(timezone.utc)
# Обновляем кэш очереди для observable-gauge
cnt = (await db.execute(
select(func.count()).select_from(AttachmentJob)
.where(AttachmentJob.status.in_(["queued", "processing"]))
)).scalar() or 0
oldest = (await db.execute(
select(func.min(AttachmentJob.lease_until)).where(AttachmentJob.status == "queued")
)).scalar()
# возраст = время с создания самой старой незавершённой job
oldest_created = (await db.execute(
select(func.min(Attachment.created_at))
.join(AttachmentJob, AttachmentJob.attachment_id == Attachment.id)
.where(AttachmentJob.status.in_(["queued", "processing"]))
)).scalar()
age = (now - oldest_created).total_seconds() if oldest_created else 0.0
_queue_state = {"active": cnt, "oldest_age": age}
rows = (await db.execute(
select(AttachmentJob).where(
AttachmentJob.status == "processing",
AttachmentJob.lease_until.isnot(None),
AttachmentJob.lease_until < now,
).limit(20)
)).scalars().all()
for job in rows:
if job.attempt >= JOB_MAX_ATTEMPTS:
continue
payload = {"job_id": str(job.job_id), "attachment_id": str(job.attachment_id),
"attempt": job.attempt + 1}
job.status = "queued"
job.attempt += 1
job.lease_until = None
job.lease_owner = None
await db.execute(update(Attachment).where(Attachment.id == job.attachment_id)
.values(status="queued"))
await db.commit()
await publish_retry(payload, choose_retry_tier(1))
logger.warning("Sweeper: lease expired for job %s -> retry attempt %s", job.job_id, job.attempt)
except asyncio.CancelledError:
raise
except Exception as e:
logger.error("Sweeper error: %s", e)
async def main():
logger.info("Starting worker %s (concurrency/prefetch=%s)", WORKER_ID, WORKER_CONCURRENCY)
channel = await get_channel()
await declare_topology(channel)
await channel.set_qos(prefetch_count=WORKER_CONCURRENCY)
queue = await channel.get_queue(MAIN_QUEUE)
await queue.consume(process_message)
asyncio.create_task(sweeper())
logger.info("Waiting for messages on %s ...", MAIN_QUEUE)
await asyncio.Future()
if __name__ == "__main__":
asyncio.run(main())
+267
View File
@@ -0,0 +1,267 @@
services:
postgres:
image: postgres:16.4-alpine
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
interval: 5s
timeout: 5s
retries: 5
deploy:
resources:
limits: {cpus: '1.0', memory: 512M}
redis:
image: valkey/valkey:8.0-alpine
command: valkey-server --appendonly yes --save 60 1
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "valkey-cli", "ping"]
interval: 5s
timeout: 5s
retries: 5
deploy:
resources:
limits: {cpus: '0.5', memory: 128M}
s3:
image: chrislusf/seaweedfs:3.80
command:
- server
- -dir=/data
- -s3
- -s3.port=8333
- -master.volumeSizeLimitMB=64
- -volume.max=100
volumes:
- s3data:/data
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8333/healthz"]
interval: 5s
timeout: 5s
retries: 10
start_period: 60s
deploy:
resources:
limits: {cpus: '1.0', memory: 1G}
s3-init:
image: amazon/aws-cli:2.15.0
depends_on:
s3:
condition: service_healthy
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=admin
- AWS_REGION=${S3_REGION}
entrypoint: >
/bin/sh -c "
aws --endpoint-url http://s3:8333 s3 mb s3://${S3_BUCKET_NAME} 2>/dev/null || echo 'Bucket already exists'
aws --endpoint-url http://s3:8333 s3api put-bucket-acl --bucket ${S3_BUCKET_NAME} --acl private
echo 'Bucket ${S3_BUCKET_NAME} is ready and private'
"
restart: "no"
rabbitmq:
image: rabbitmq:3.13-management-alpine
hostname: rabbitmq
environment:
RABBITMQ_DEFAULT_USER: messenger
RABBITMQ_DEFAULT_PASS: mq_secret_password
RABBITMQ_DEFAULT_VHOST: /
volumes:
- mqdata:/var/lib/rabbitmq
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "check_port_connectivity"]
interval: 5s
timeout: 10s
retries: 30
deploy:
resources:
limits: {cpus: '1.0', memory: 512M}
otel-collector:
image: otel/opentelemetry-collector-contrib:0.163.2
command: ["--config=/etc/otelcol-contrib/collector.yaml"]
volumes:
- ./infra/otel/collector.yaml:/etc/otelcol-contrib/collector.yaml:ro
depends_on:
- tempo
- loki
deploy:
resources:
limits: {cpus: '0.5', memory: 512M}
restart: unless-stopped
prometheus:
image: prom/prometheus:v2.55.1
ports: ["127.0.0.1:9090:9090"]
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=15d
volumes:
- ./infra/otel/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./infra/otel/alerts.yml:/etc/prometheus/alerts.yml:ro
- prometheus_data:/prometheus
depends_on:
- otel-collector
deploy:
resources:
limits: {cpus: '0.5', memory: 512M}
restart: unless-stopped
loki:
image: grafana/loki:3.3.2
command: ["-config.file=/etc/loki/loki.yml"]
volumes:
- ./infra/otel/loki.yml:/etc/loki/loki.yml:ro
- loki_data:/loki
deploy:
resources:
limits: {cpus: '0.5', memory: 512M}
restart: unless-stopped
tempo:
image: grafana/tempo:2.7.1
command: ["-config.file=/etc/tempo/tempo.yml"]
volumes:
- ./infra/otel/tempo.yml:/etc/tempo/tempo.yml:ro
- tempo_data:/var/tempo
deploy:
resources:
limits: {cpus: '0.5', memory: 512M}
restart: unless-stopped
grafana:
image: grafana/grafana:11.4.0
ports:
- "127.0.0.1:3000:3000"
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
GF_USERS_ALLOW_SIGN_UP: "false"
GF_ANALYTICS_REPORTING_ENABLED: "false"
volumes:
- grafana_data:/var/lib/grafana
- ./infra/otel/grafana/provisioning:/etc/grafana/provisioning:ro
- ./infra/otel/grafana/dashboards:/var/lib/grafana/dashboards:ro
depends_on:
- prometheus
- loki
- tempo
deploy:
resources:
limits: {cpus: '0.5', memory: 512M}
restart: unless-stopped
api:
build:
context: .
dockerfile: Dockerfile
environment:
- DATABASE_URL=${DATABASE_URL}
- REDIS_URL=${REDIS_URL}
- SECRET_SESSION_KEY=${SECRET_SESSION_KEY}
- SESSION_TTL_SECONDS=${SESSION_TTL_SECONDS}
- S3_ENDPOINT=${S3_ENDPOINT}
- S3_ACCESS_KEY=${S3_API_ACCESS_KEY:-admin}
- S3_SECRET_KEY=${S3_API_SECRET_KEY:-admin}
- S3_BUCKET_NAME=${S3_BUCKET_NAME}
- S3_REGION=${S3_REGION}
- BROKER_URL=${BROKER_URL}
- MAX_ACTIVE_JOBS=${MAX_ACTIVE_JOBS}
- OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}
- OTEL_SERVICE_NAME=messenger-api
- OTEL_RESOURCE_ATTRIBUTES=service.version=1.0.0,deployment.environment=lab
volumes:
- ./alembic/versions:/app/alembic/versions
- ./app:/app/app
read_only: true
tmpfs:
- /tmp:size=15M
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
s3-init:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
otel-collector:
condition: service_started
stop_grace_period: 30s
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health/live"]
interval: 10s
timeout: 5s
retries: 3
start_period: 10s
deploy:
resources:
limits: {cpus: '0.5', memory: 256M}
worker:
build:
context: .
dockerfile: Dockerfile
command: ["python", "-m", "app.worker"]
environment:
- DATABASE_URL=${DATABASE_URL}
- BROKER_URL=${BROKER_URL}
- S3_ENDPOINT=${S3_ENDPOINT}
- S3_ACCESS_KEY=${S3_API_ACCESS_KEY:-admin}
- S3_SECRET_KEY=${S3_API_SECRET_KEY:-admin}
- S3_BUCKET_NAME=${S3_BUCKET_NAME}
- WORKER_CONCURRENCY=${WORKER_CONCURRENCY}
- JOB_MAX_ATTEMPTS=${JOB_MAX_ATTEMPTS}
- JOB_LEASE_SECONDS=${JOB_LEASE_SECONDS}
- OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}
- OTEL_SERVICE_NAME=messenger-worker
- OTEL_RESOURCE_ATTRIBUTES=service.version=1.0.0,deployment.environment=lab
volumes:
- ./app:/app/app
read_only: true
tmpfs:
- /tmp:size=100M
depends_on:
postgres:
condition: service_healthy
rabbitmq:
condition: service_healthy
s3-init:
condition: service_completed_successfully
otel-collector:
condition: service_started
deploy:
resources:
limits: {cpus: '2.0', memory: 768M}
restart: unless-stopped
nginx:
image: nginx:1.25-alpine
ports:
- "8080:8080"
volumes:
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
depends_on:
api:
condition: service_healthy
restart: unless-stopped
volumes:
postgres_data:
redis_data:
s3data:
mqdata:
prometheus_data:
loki_data:
tempo_data:
grafana_data:
+25
View File
@@ -0,0 +1,25 @@
groups:
- name: messenger_slo
rules:
- alert: HighErrorRate
expr: |
(
sum(rate(http_server_duration_milliseconds_count{http_status_code=~"5.."}[5m])) or vector(0)
)
/
sum(rate(http_server_duration_milliseconds_count[5m]))
for: 1m
labels:
severity: critical
annotations:
summary: "API error rate > 5% (SLO 99%)"
description: "Доля 5xx за 5-минутное окно превысила 5%. SLI из histogram-метрик API."
- alert: QueueLagGrowing
expr: messenger_queue_oldest_job_age_seconds > 60
for: 2m
labels:
severity: warning
annotations:
summary: "Очередь задач растёт (возраст старейшей job > 60s)"
description: "Worker не успевает дренировать очередь."
+39
View File
@@ -0,0 +1,39 @@
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 384
spike_limit_mib: 96
batch:
send_batch_size: 512
timeout: 5s
exporters:
otlphttp/tempo:
endpoint: "http://tempo:4318"
otlphttp/loki:
endpoint: "http://loki:3100/otlp"
prometheus:
endpoint: "0.0.0.0:8889"
resource_to_telemetry_conversion:
enabled: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/loki]
@@ -0,0 +1,12 @@
apiVersion: 1
providers:
- name: 'messenger'
orgId: 1
folder: 'Messenger'
type: file
disableDeletion: false
updateIntervalSeconds: 30
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
@@ -0,0 +1,102 @@
{
"uid": "messenger-lab6",
"title": "Messenger — Lab6 Observability",
"tags": ["messenger", "lab6"],
"timezone": "browser",
"schemaVersion": 39,
"refresh": "10s",
"time": { "from": "now-15m", "to": "now" },
"panels": [
{
"id": 1, "type": "timeseries", "title": "RPS (по endpoint'ам)",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "sum by (http_target) (rate(http_server_duration_milliseconds_count[5m]))", "legendFormat": "{{http_target}}" }
]
},
{
"id": 2,
"type": "timeseries",
"title": "Доля ошибок 5xx (SLI)",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1
},
"overrides": []
},
"targets": [
{
"refId": "A",
"expr": "(sum(rate(http_server_duration_milliseconds_count{http_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_duration_milliseconds_count[5m]))",
"legendFormat": "error ratio"
}
]
},
{
"id": 3, "type": "timeseries", "title": "API latency p50/p95/p99 (из histogram, ms)",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "ms" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "histogram_quantile(0.50, sum by (le) (rate(http_server_duration_milliseconds_bucket[5m])))", "legendFormat": "p50" },
{ "refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_duration_milliseconds_bucket[5m])))", "legendFormat": "p95" },
{ "refId": "C", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_duration_milliseconds_bucket[5m])))", "legendFormat": "p99" }
]
},
{
"id": 4, "type": "timeseries", "title": "Worker jobs: completed / failed",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "sum(rate(messenger_worker_jobs_completed_total[5m]))", "legendFormat": "completed" },
{ "refId": "B", "expr": "sum(rate(messenger_worker_jobs_failed_total[5m]))", "legendFormat": "failed" }
]
},
{
"id": 5, "type": "timeseries", "title": "Worker job duration p50/p95 (s)",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "histogram_quantile(0.50, sum by (le) (rate(messenger_worker_job_duration_seconds_bucket[5m])))", "legendFormat": "p50" },
{ "refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_worker_job_duration_seconds_bucket[5m])))", "legendFormat": "p95" }
]
},
{
"id": 6, "type": "stat", "title": "Worker saturation (active jobs)",
"gridPos": { "h": 8, "w": 6, "x": 12, "y": 16 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [ { "refId": "A", "expr": "sum(messenger_worker_jobs_active)", "legendFormat": "active" } ]
},
{
"id": 7, "type": "stat", "title": "Queue depth (active jobs)",
"gridPos": { "h": 8, "w": 6, "x": 18, "y": 16 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [ { "refId": "A", "expr": "sum(messenger_queue_active_jobs)", "legendFormat": "queued+processing" } ]
},
{
"id": 8, "type": "timeseries", "title": "Возраст старейшей job (queue lag)",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
"targets": [ { "refId": "A", "expr": "sum(messenger_queue_oldest_job_age_seconds)", "legendFormat": "oldest age" } ]
},
{
"id": 9, "type": "timeseries", "title": "DB pool (size / checked out / overflow)",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{ "refId": "A", "expr": "sum(messenger_db_pool_size)", "legendFormat": "size" },
{ "refId": "B", "expr": "sum(messenger_db_pool_checked_out)", "legendFormat": "checked out" },
{ "refId": "C", "expr": "sum(messenger_db_pool_overflow)", "legendFormat": "overflow" }
]
}
]
}
@@ -0,0 +1,12 @@
apiVersion: 1
providers:
- name: 'messenger'
orgId: 1
folder: 'Messenger'
type: file
disableDeletion: false
updateIntervalSeconds: 30
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
@@ -0,0 +1,88 @@
{
"uid": "messenger-lab6",
"title": "Messenger — Lab6 Observability",
"tags": ["messenger", "lab6"],
"timezone": "browser",
"schemaVersion": 39,
"refresh": "10s",
"time": { "from": "now-15m", "to": "now" },
"panels": [
{
"id": 1, "type": "timeseries", "title": "RPS (по маршрутам)",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{ "refId": "A", "expr": "sum by (http_route) (rate(http_server_duration_seconds_count[5m]))", "legendFormat": "{{http_route}}" }
]
},
{
"id": 2, "type": "timeseries", "title": "Доля ошибок 5xx (SLI)",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit", "max": 1 }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "sum(rate(http_server_duration_seconds_count{http_response_status_code=~\"5..\"}[5m])) / sum(rate(http_server_duration_seconds_count[5m]))", "legendFormat": "error ratio" }
]
},
{
"id": 3, "type": "timeseries", "title": "API latency p50/p95/p99 (из histogram)",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "histogram_quantile(0.50, sum by (le) (rate(http_server_duration_seconds_bucket[5m])))", "legendFormat": "p50" },
{ "refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_duration_seconds_bucket[5m])))", "legendFormat": "p95" },
{ "refId": "C", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_duration_seconds_bucket[5m])))", "legendFormat": "p99" }
]
},
{
"id": 4, "type": "timeseries", "title": "Worker jobs: completed / failed",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "sum(rate(messenger_worker_jobs_completed_total[5m]))", "legendFormat": "completed" },
{ "refId": "B", "expr": "sum(rate(messenger_worker_jobs_failed_total[5m]))", "legendFormat": "failed" }
]
},
{
"id": 5, "type": "timeseries", "title": "Worker job duration p50/p95",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "histogram_quantile(0.50, sum by (le) (rate(messenger_worker_job_duration_seconds_bucket[5m])))", "legendFormat": "p50" },
{ "refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_worker_job_duration_seconds_bucket[5m])))", "legendFormat": "p95" }
]
},
{
"id": 6, "type": "stat", "title": "Worker saturation (active jobs)",
"gridPos": { "h": 8, "w": 6, "x": 12, "y": 16 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [ { "refId": "A", "expr": "sum(messenger_worker_jobs_active)", "legendFormat": "active" } ]
},
{
"id": 7, "type": "stat", "title": "Queue depth (active jobs)",
"gridPos": { "h": 8, "w": 6, "x": 18, "y": 16 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [ { "refId": "A", "expr": "sum(messenger_queue_active_jobs)", "legendFormat": "queued+processing" } ]
},
{
"id": 8, "type": "timeseries", "title": "Возраст старейшей job (queue lag)",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
"targets": [ { "refId": "A", "expr": "sum(messenger_queue_oldest_job_age_seconds)", "legendFormat": "oldest age" } ]
},
{
"id": 9, "type": "timeseries", "title": "DB pool (size / checked out / overflow)",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{ "refId": "A", "expr": "sum(messenger_db_pool_size)", "legendFormat": "size" },
{ "refId": "B", "expr": "sum(messenger_db_pool_checked_out)", "legendFormat": "checked out" },
{ "refId": "C", "expr": "sum(messenger_db_pool_overflow)", "legendFormat": "overflow" }
]
}
]
}
@@ -0,0 +1,35 @@
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
url: http://prometheus:9090
isDefault: true
editable: false
- name: Loki
type: loki
uid: loki
url: http://loki:3100
editable: false
jsonData:
derivedFields:
- name: TraceID
matcherRegex: "trace_id=(\\w+)"
url: "$${__value.raw}"
datasourceUid: tempo
- name: Tempo
type: tempo
uid: tempo
url: http://tempo:3200
editable: false
jsonData:
tracesToLogsV2:
datasourceUid: loki
spanStartTimeShift: -5m
spanEndTimeShift: 5m
tags: [{ key: 'service.name', value: 'job' }]
filterByTraceID: true
filterBySpanID: true
+30
View File
@@ -0,0 +1,30 @@
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
allow_structured_metadata: true
volume_enabled: true
+11
View File
@@ -0,0 +1,11 @@
global:
scrape_interval: 10s
evaluation_interval: 10s
rule_files:
- /etc/prometheus/alerts.yml
scrape_configs:
- job_name: 'otel-collector'
static_configs:
- targets: ['otel-collector:8889']
+23
View File
@@ -0,0 +1,23 @@
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
storage:
trace:
backend: local
local:
path: /var/tempo/traces
wal:
path: /var/tempo/wal
compactor:
compaction:
block_retention: 24h
+22
View File
@@ -0,0 +1,22 @@
upstream api_backend {
# Nginx будет автоматически резолвить имена через Docker DNS
server api:8080 max_fails=1 fail_timeout=5s;
}
server {
listen 8080;
# Проксирование ВСЕХ запросов на API реплики
location / {
proxy_pass http://api_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Request-Id $http_x_request_id;
# Таймауты для graceful shutdown
proxy_connect_timeout 5s;
proxy_read_timeout 30s;
proxy_send_timeout 30s;
}
}
+25
View File
@@ -0,0 +1,25 @@
fastapi==0.115.0
uvicorn[standard]==0.30.6
sqlalchemy[asyncio]==2.0.35
asyncpg==0.29.0
pydantic-settings==2.5.2
redis==5.1.0
bcrypt==4.2.0
alembic==1.13.2
python-multipart==0.0.9
minio==7.2.0
Pillow==10.4.0
python-magic==0.4.27
aio-pika==9.4.3
uuid-utils==0.10.0
# === OpenTelemetry (Lab 6) ===
opentelemetry-api==1.29.0
opentelemetry-sdk==1.29.0
opentelemetry-exporter-otlp-proto-http==1.29.0
opentelemetry-instrumentation-fastapi==0.50b0
opentelemetry-instrumentation-asyncpg==0.50b0
opentelemetry-instrumentation-aio-pika==0.50b0
opentelemetry-instrumentation-botocore==0.50b0
opentelemetry-instrumentation-logging==0.50b0
opentelemetry-semantic-conventions==0.50b0