ЛР6 Будилов-БВТ2304 - на 4 #51
@@ -23,8 +23,4 @@ WORKER_CONCURRENCY=2
|
||||
JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
OTEL_SERVICE_NAME=messenger-api
|
||||
# Worker must set a distinct service.name.
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
|
||||
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
gcc \
|
||||
libpq-dev \
|
||||
curl \
|
||||
libmagic1 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY app/ ./app/
|
||||
COPY alembic.ini .
|
||||
COPY alembic/ ./alembic/
|
||||
|
||||
# Создаём непривилегированного пользователя для read-only контейнера
|
||||
RUN useradd -m appuser && chown -R appuser:appuser /app
|
||||
USER appuser
|
||||
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]
|
||||
@@ -1,37 +1,106 @@
|
||||
# Отчёт — лабораторная 6
|
||||
# Отчёт — лабораторная 6: Наблюдаемость (OpenTelemetry и Grafana)
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Будилова Ярослава, БВТ2304
|
||||
- Целевая оценка: 4 (уровень 3 + уровень 4)
|
||||
- Commit/PR: ветка solution/BVT2304-Budilov_Yaroslav
|
||||
- Использованные источники, библиотеки и помощь ИИ: Python 3.12, FastAPI 0.115.0, OpenTelemetry API/SDK 1.29.0, OTLP HTTP exporter, инструментация FastAPI/asyncpg/aio-pika/logging 0.50b0. Образы: otel/opentelemetry-collector-contrib:0.163.2, prom/prometheus:v2.55.1, grafana/loki:3.3.2, grafana/tempo:2.7.1, grafana/grafana:11.4.0, postgres:16.4-alpine, valkey/valkey:8.0-alpine, chrislusf/seaweedfs:3.80, rabbitmq:3.13-management-alpine, nginx:1.25-alpine. Помощь ИИ использовалась для отладки порядка инициализации SDK и конфигурации provisioning.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Prerequisites: Docker 24+, Docker Compose v2, Python 3.9+.
|
||||
|
||||
1. Клонировать и создать конфигурацию:
|
||||
git clone https://gitea.privcloud.tech/Budilov_bvt2304/mtusi2026lab6
|
||||
cd mtusi2026lab6
|
||||
cp .env.example .env
|
||||
|
||||
2. Собрать и запустить стенд с масштабированием API:
|
||||
docker compose up -d --build --scale api=2
|
||||
|
||||
3. Применить миграции БД (однократно):
|
||||
docker compose run --rm api alembic upgrade head
|
||||
|
||||
4. Дождаться готовности (healthy для postgres/redis/rabbitmq/s3/api, Up для collector/prometheus/loki/tempo/grafana):
|
||||
docker compose ps
|
||||
|
||||
5. Прогнать smoke-тест при включённой телеметрии:
|
||||
python3 tests/smoke.py --base-url http://localhost:8080
|
||||
|
||||
6. Безопасная остановка (сохраняет volumes):
|
||||
docker compose stop
|
||||
|
||||
7. Намеренное удаление учебных данных:
|
||||
docker compose down -v
|
||||
|
||||
Grafana на http://localhost:3000 (admin/admin), Prometheus на http://localhost:9090.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
- Топология наблюдаемости: API и worker (OTel SDK) → OTLP HTTP → OTel Collector → Prometheus (метрики), Loki (структурные логи), Tempo (трассы); Grafana агрегирует данные через provisioning.
|
||||
- Источник истины по заданиям — PostgreSQL (attachment_jobs/job_outbox), как в lab5; брокер и телеметрия не являются источником истины.
|
||||
- W3C Trace Context пробрасывается API → broker → worker: aio-pika instrumentation кладёт traceparent в headers сообщения, worker извлекает его, поэтому upload и обработка попадают в один trace (producer/consumer spans связаны через parent-child).
|
||||
- Транзакционные границы не изменились относительно lab5 (staging → COMMIT → outbox → relay → worker → ack).
|
||||
- Ограничения Collector: memory_limiter (384 MiB, spike 96 MiB) и batch processor (512 элементов, 5s timeout) защищают RAM при недоступном backend.
|
||||
- Безопасность данных: в telemetry не экспортируются тела сообщений, пароли/токены, filenames, содержимое файлов; метрики используют низкокардинальные атрибуты (http_target, http_status_code, http_method), без user_id/chat_id/message_id в labels.
|
||||
- Что перенесено из lab5: вся бизнес-логика, outbox, retry+jitter, DLQ, lease/sweeper. Что добавлено: SDK-инструментация, Collector, три хранилища, Grafana provisioning, метрики worker и DB pool, alert rules.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Минимум lab5 + Collector/хранища/Grafana в Compose | docker compose ps; python3 tests/smoke.py | Все сервисы Up/healthy, smoke PASS при включённой телеметрии | evidence/compose_ps.txt, evidence/smoke_pass.txt |
|
||||
| API/worker отдают OTLP; в Grafana RPS/errors/duration API, jobs, worker duration | Grafana dashboard Messenger | Панели RPS, error ratio (SLI=0 при отсутствии 5xx), p50/p95/p99 (ms), completed/failed jobs, worker duration заполнены | evidence/dashboard.png |
|
||||
| Импортируемый dashboard, низкокардинальные атрибуты, trace/span в логах | infra/otel/grafana/provisioning | Дашборд грузится автоматически, логи содержат trace_id/span_id | infra/otel/grafana/dashboards/messenger.json |
|
||||
| W3C context API→broker→worker, цепочка до S3/DB | Explore → Tempo, поиск по trace_id из worker-лога | Один trace связывает HTTP span API, publish span (producer) и consumer span worker; дочерние spans БД (asyncpg) и S3 (botocore) видны | evidence/trace.png |
|
||||
| Dashboard: p50/p95/p99, queue depth, возраст job, DB pool, saturation | Prometheus query | Percentile через histogram_quantile из _bucket (не среднее); метрики messenger_queue_*, messenger_db_pool_* | evidence/prometheus_targets.png |
|
||||
| SLI/SLO + alert rules (error rate, queue lag), проверка отказом | infra/otel/alerts.yml; docker compose stop postgres; docker compose stop worker | HighErrorRate и QueueLagGrowing переходят firing→resolved | evidence/alert_higherror_firing.png, evidence/alert_higherror_resolved.png, evidence/alert_queuelag_firing.png |
|
||||
| Collector memory limiter/batch, устойчивость к отказу backend | docker compose stop tempo; smoke | API продолжает 202/200, RAM Collector ограничена memory_limiter (~60-70 MiB), сообщения не теряются | evidence/backend_down.txt |
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Инцидент 1 — остановка telemetry backend (Tempo).
|
||||
Гипотеза: отказ хранилища трасс не влияет на бизнес-логику API и не вызывает бесконечный рост RAM Collector благодаря memory_limiter.
|
||||
Методика:
|
||||
python3 tests/smoke.py --base-url http://localhost:8080
|
||||
docker stats mtusi2026lab6-otel-collector-1 --no-stream
|
||||
docker compose stop tempo
|
||||
for i in (seq 1 60)
|
||||
curl -s -o /dev/null http://localhost:8080/health/live
|
||||
curl -s -o /dev/null -X POST http://localhost:8080/api/v1/auth/sessions -u "smoke_test_user:supersecretpassword123"
|
||||
end
|
||||
docker stats mtusi2026lab6-otel-collector-1 --no-stream
|
||||
python3 tests/smoke.py --base-url http://localhost:8080
|
||||
docker compose start tempo
|
||||
Наблюдение: smoke проходит до и после остановки Tempo. Collector показывает ошибки экспорта в логах, но RAM держится в пределах memory_limiter (~60-70 MiB, spike не превысил 96 MiB). API не блокируется, сообщения в RabbitMQ целы. Восстановление: старт Tempo, трассы снова пишутся. Потери за период простоя: только трассы (ожидаемо), бизнес-данные целы.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Инцидент 2 — искусственные отказы для проверки алёртов.
|
||||
Гипотеза: HighErrorRate firing при росте доли 5xx, QueueLagGrowing firing при остановке worker под нагрузкой.
|
||||
Методика HighErrorRate:
|
||||
docker compose stop postgres
|
||||
for i in (seq 1 40)
|
||||
curl -s -o /dev/null -X POST http://localhost:8080/api/v1/users -H "Content-Type: application/json" -d '{"username":"e'$i'","display_name":"E","password":"aaaaaaaaaaaa1"}'
|
||||
sleep 2
|
||||
end
|
||||
sleep 70
|
||||
curl -s http://localhost:9090/api/v1/alerts | python3 -m json.tool | grep -B2 -A4 HighErrorRate
|
||||
docker compose start postgres
|
||||
Наблюдение: алёрт HighErrorRate перешёл в state=firing через ~1 мин после начала 5xx. После возврата postgres алёрт ушёл в resolved за ~6 мин (окно rate[5m] + for:1m).
|
||||
|
||||
Методика QueueLagGrowing:
|
||||
docker compose stop worker
|
||||
for i in (seq 1 20)
|
||||
curl -s -o /dev/null -X POST "http://localhost:8080/api/v1/chats/$CHAT_ID/attachments?filename=t.png" -H "Authorization: Bearer $TOKEN" -H "Content-Type: image/png" --data-binary @/tmp/test.png
|
||||
end
|
||||
sleep 130
|
||||
curl -s http://localhost:9090/api/v1/alerts | python3 -m json.tool | grep -B2 -A4 QueueLagGrowing
|
||||
docker compose start worker
|
||||
Наблюдение: алёрт QueueLagGrowing перешёл в state=firing после того как messenger_queue_oldest_job_age_seconds превысил 60s. После docker compose start worker очередь дренировалась за ~10 сек, алёрт resolved. Потерь данных нет (durable очередь + outbox).
|
||||
|
||||
Ограничения измерения: один хост, по одному прогону на сценарий, sampling 100% (не оценивался в рамках уровня 4), нагрузка синтетическая (не production-профиль).
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
- Выдерживаются: падение до 50% API-реплик, недоступность Tempo/Loki/Prometheus (Collector буферизует/отбрасывает данные безопасно), падение/перезапуск worker, рестарт брокера с volume.
|
||||
- Не выдерживаются (SPOF): потеря диска хоста; отсутствие кластерной репликации PostgreSQL/RabbitMQ/хранилищ телеметрии.
|
||||
- Учебные упрощения: sampling не настраивался (100%); alerting локальный (firing/resolved), доставка наружу не требуется по контракту.
|
||||
- Следующий шаг (Трек 1 «Цена наблюдаемости»): сравнить telemetry off/on и sampling 100/25/10 при одинаковой нагрузке по latency, CPU/RAM, объёму ingest и потерям spans; предложить сбалансированную конфигурацию.
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
[alembic]
|
||||
script_location = alembic
|
||||
prepend_sys_path = .
|
||||
version_path_separator = os
|
||||
sqlalchemy.url = driver://user:pass@localhost/dbname
|
||||
|
||||
[post_write_hooks]
|
||||
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARN
|
||||
handlers = console
|
||||
qualname =
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARN
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
@@ -0,0 +1,55 @@
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from logging.config import fileConfig
|
||||
|
||||
from sqlalchemy import pool
|
||||
from sqlalchemy.engine import Connection
|
||||
from sqlalchemy.ext.asyncio import async_engine_from_config
|
||||
from alembic import context
|
||||
|
||||
# Добавляем корень проекта в sys.path, чтобы импортировать app
|
||||
sys.path.append(os.path.dirname(os.path.dirname(__file__)))
|
||||
|
||||
from app.database import Base, DATABASE_URL
|
||||
from app.models import User, UserSession, Chat, ChatMember, Message, IdempotencyKey, Attachment, MessageAttachment
|
||||
|
||||
config = context.config
|
||||
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
target_metadata = Base.metadata
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
context.configure(
|
||||
url=DATABASE_URL,
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
def do_run_migrations(connection: Connection) -> None:
|
||||
context.configure(connection=connection, target_metadata=target_metadata)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
async def run_async_migrations() -> None:
|
||||
connectable = async_engine_from_config(
|
||||
{"sqlalchemy.url": DATABASE_URL},
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
async with connectable.connect() as connection:
|
||||
await connection.run_sync(do_run_migrations)
|
||||
await connectable.dispose()
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
asyncio.run(run_async_migrations())
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
run_migrations_online()
|
||||
@@ -0,0 +1,26 @@
|
||||
"""${message}
|
||||
|
||||
Revision ID: ${up_revision}
|
||||
Revises: ${down_revision | comma,n}
|
||||
Create Date: ${create_date}
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = ${repr(up_revision)}
|
||||
down_revision: Union[str, None] = ${repr(down_revision)}
|
||||
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
|
||||
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -0,0 +1,146 @@
|
||||
"""lab5 initial schema with async processing
|
||||
|
||||
Revision ID: 7fc277b41201
|
||||
Revises:
|
||||
Create Date: 2026-11-12 16:13:31.689026
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '7fc277b41201'
|
||||
down_revision: Union[str, None] = None
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.create_table('chats',
|
||||
sa.Column('id', sa.UUID(), nullable=False),
|
||||
sa.Column('title', sa.String(length=100), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
op.create_table('job_outbox',
|
||||
sa.Column('job_id', sa.UUID(), nullable=False),
|
||||
sa.Column('payload', postgresql.JSONB(astext_type=sa.Text()), nullable=False),
|
||||
sa.Column('published_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.PrimaryKeyConstraint('job_id')
|
||||
)
|
||||
op.create_table('users',
|
||||
sa.Column('id', sa.UUID(), nullable=False),
|
||||
sa.Column('username', sa.String(length=32), nullable=False),
|
||||
sa.Column('display_name', sa.String(length=100), nullable=False),
|
||||
sa.Column('password_hash', sa.String(length=255), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
op.create_index(op.f('ix_users_username'), 'users', ['username'], unique=True)
|
||||
op.create_table('attachments',
|
||||
sa.Column('id', sa.UUID(), nullable=False),
|
||||
sa.Column('owner_id', sa.UUID(), nullable=False),
|
||||
sa.Column('chat_id', sa.UUID(), nullable=False),
|
||||
sa.Column('staging_key', sa.String(length=512), nullable=True),
|
||||
sa.Column('s3_key', sa.String(length=512), nullable=True),
|
||||
sa.Column('original_filename', sa.String(length=255), nullable=False),
|
||||
sa.Column('content_type', sa.String(length=100), nullable=False),
|
||||
sa.Column('size_bytes', sa.BigInteger(), nullable=False),
|
||||
sa.Column('sha256', sa.String(length=64), nullable=False),
|
||||
sa.Column('status', sa.String(length=20), nullable=False),
|
||||
sa.Column('error_code', sa.String(length=50), nullable=True),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['owner_id'], ['users.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
op.create_table('chat_members',
|
||||
sa.Column('chat_id', sa.UUID(), nullable=False),
|
||||
sa.Column('user_id', sa.UUID(), nullable=False),
|
||||
sa.Column('joined_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('chat_id', 'user_id')
|
||||
)
|
||||
op.create_table('idempotency_keys',
|
||||
sa.Column('key', sa.String(length=255), nullable=False),
|
||||
sa.Column('chat_id', sa.UUID(), nullable=False),
|
||||
sa.Column('response_data', sa.Text(), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('key')
|
||||
)
|
||||
op.create_table('messages',
|
||||
sa.Column('id', sa.UUID(), nullable=False),
|
||||
sa.Column('chat_id', sa.UUID(), nullable=False),
|
||||
sa.Column('sender_id', sa.UUID(), nullable=False),
|
||||
sa.Column('text', sa.Text(), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['sender_id'], ['users.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
op.create_index(op.f('ix_messages_chat_id'), 'messages', ['chat_id'], unique=False)
|
||||
op.create_index(op.f('ix_messages_created_at'), 'messages', ['created_at'], unique=False)
|
||||
op.create_table('sessions',
|
||||
sa.Column('token', sa.String(length=255), nullable=False),
|
||||
sa.Column('user_id', sa.UUID(), nullable=False),
|
||||
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('token')
|
||||
)
|
||||
op.create_table('attachment_jobs',
|
||||
sa.Column('job_id', sa.UUID(), nullable=False),
|
||||
sa.Column('attachment_id', sa.UUID(), nullable=False),
|
||||
sa.Column('attempt', sa.Integer(), nullable=False),
|
||||
sa.Column('status', sa.String(length=20), nullable=False),
|
||||
sa.Column('lease_until', sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column('lease_owner', sa.String(length=100), nullable=True),
|
||||
sa.ForeignKeyConstraint(['attachment_id'], ['attachments.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('job_id'),
|
||||
sa.UniqueConstraint('attachment_id')
|
||||
)
|
||||
op.create_table('attachment_variants',
|
||||
sa.Column('attachment_id', sa.UUID(), nullable=False),
|
||||
sa.Column('variant_name', sa.String(length=20), nullable=False),
|
||||
sa.Column('s3_key', sa.String(length=512), nullable=False),
|
||||
sa.Column('content_type', sa.String(length=100), nullable=False),
|
||||
sa.Column('size_bytes', sa.BigInteger(), nullable=False),
|
||||
sa.Column('width', sa.Integer(), nullable=False),
|
||||
sa.Column('height', sa.Integer(), nullable=False),
|
||||
sa.Column('sha256', sa.String(length=64), nullable=False),
|
||||
sa.ForeignKeyConstraint(['attachment_id'], ['attachments.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('attachment_id', 'variant_name')
|
||||
)
|
||||
op.create_table('message_attachments',
|
||||
sa.Column('message_id', sa.UUID(), nullable=False),
|
||||
sa.Column('attachment_id', sa.UUID(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['attachment_id'], ['attachments.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['message_id'], ['messages.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('message_id', 'attachment_id')
|
||||
)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.drop_table('message_attachments')
|
||||
op.drop_table('attachment_variants')
|
||||
op.drop_table('attachment_jobs')
|
||||
op.drop_table('sessions')
|
||||
op.drop_index(op.f('ix_messages_created_at'), table_name='messages')
|
||||
op.drop_index(op.f('ix_messages_chat_id'), table_name='messages')
|
||||
op.drop_table('messages')
|
||||
op.drop_table('idempotency_keys')
|
||||
op.drop_table('chat_members')
|
||||
op.drop_table('attachments')
|
||||
op.drop_index(op.f('ix_users_username'), table_name='users')
|
||||
op.drop_table('users')
|
||||
op.drop_table('job_outbox')
|
||||
op.drop_table('chats')
|
||||
# ### end Alembic commands ###
|
||||
+137
@@ -0,0 +1,137 @@
|
||||
import os
|
||||
import json
|
||||
import random
|
||||
import logging
|
||||
import asyncio
|
||||
from datetime import datetime, timezone
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
import aio_pika
|
||||
from app.models import JobOutbox
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BROKER_URL = os.getenv("BROKER_URL", "amqp://messenger:mq_secret_password@rabbitmq:5672/")
|
||||
|
||||
MAIN_QUEUE = "attachment.process.v1"
|
||||
DLX_NAME = "attachments.dlx"
|
||||
DLQ_NAME = "attachment.process.dlq"
|
||||
RETRY_EXCHANGE = "attachments.retry"
|
||||
|
||||
# Лестница отложенных очередей: фиксированный TTL на очередь,
|
||||
# поэтому нет head-of-line blocking (в отличие от per-message TTL).
|
||||
RETRY_TIERS = [1, 2, 4, 8, 16, 32, 60]
|
||||
BASE_DELAY = 1
|
||||
MAX_DELAY = 60
|
||||
|
||||
_connection = None
|
||||
_channel = None
|
||||
|
||||
|
||||
def retry_queue_name(tier: int) -> str:
|
||||
return f"attachment.process.retry.{tier}s"
|
||||
|
||||
|
||||
def choose_retry_tier(attempt: int) -> int:
|
||||
"""Экспоненциальный backoff с equal jitter, квантованный по лестнице TTL."""
|
||||
exp = min(BASE_DELAY * (2 ** (attempt - 1)), MAX_DELAY)
|
||||
delay = random.uniform(exp / 2, exp) # equal jitter
|
||||
for tier in RETRY_TIERS:
|
||||
if tier >= delay:
|
||||
return tier
|
||||
return RETRY_TIERS[-1]
|
||||
|
||||
|
||||
async def declare_topology(channel: aio_pika.Channel):
|
||||
"""Идемпотентное объявление обменов и очередей. Аргументы неизменяемы!"""
|
||||
dlx = await channel.declare_exchange(DLX_NAME, aio_pika.ExchangeType.DIRECT, durable=True)
|
||||
dlq = await channel.declare_queue(DLQ_NAME, durable=True)
|
||||
await dlq.bind(dlx, routing_key=DLQ_NAME)
|
||||
|
||||
retry_ex = await channel.declare_exchange(RETRY_EXCHANGE, aio_pika.ExchangeType.DIRECT, durable=True)
|
||||
for tier in RETRY_TIERS:
|
||||
name = retry_queue_name(tier)
|
||||
q = await channel.declare_queue(
|
||||
name,
|
||||
durable=True,
|
||||
arguments={
|
||||
"x-message-ttl": tier * 1000,
|
||||
"x-dead-letter-exchange": "", # default exchange
|
||||
"x-dead-letter-routing-key": MAIN_QUEUE, # после TTL -> в основную очередь
|
||||
},
|
||||
)
|
||||
await q.bind(retry_ex, routing_key=name)
|
||||
|
||||
await channel.declare_queue(
|
||||
MAIN_QUEUE,
|
||||
durable=True,
|
||||
arguments={
|
||||
"x-dead-letter-exchange": DLX_NAME,
|
||||
"x-dead-letter-routing-key": DLQ_NAME,
|
||||
},
|
||||
)
|
||||
logger.info("RabbitMQ topology ready: main=%s, retry tiers=%s, dlq=%s", MAIN_QUEUE, RETRY_TIERS, DLQ_NAME)
|
||||
|
||||
|
||||
async def get_channel() -> aio_pika.Channel:
|
||||
"""Канал с publisher confirms (aio-pika по умолчанию publisher_confirms=True)."""
|
||||
global _connection, _channel
|
||||
if _connection is None or _connection.is_closed or _channel is None or _channel.is_closed:
|
||||
_connection = await aio_pika.connect_robust(BROKER_URL)
|
||||
_channel = await _connection.channel() # publisher_confirms=True
|
||||
await declare_topology(_channel)
|
||||
return _channel
|
||||
|
||||
|
||||
async def publish_retry(payload: dict, tier: int):
|
||||
"""Отложенная повторная доставка: сообщение полежит tier секунд и вернётся в main."""
|
||||
channel = await get_channel()
|
||||
retry_ex = await channel.get_exchange(RETRY_EXCHANGE)
|
||||
await retry_ex.publish(
|
||||
aio_pika.Message(
|
||||
body=json.dumps(payload).encode(),
|
||||
delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
|
||||
),
|
||||
routing_key=retry_queue_name(tier),
|
||||
)
|
||||
logger.info("Scheduled retry in %ss for job %s (attempt %s)", tier, payload.get("job_id"), payload.get("attempt"))
|
||||
|
||||
|
||||
async def publish_main(payload: dict):
|
||||
channel = await get_channel()
|
||||
await channel.default_exchange.publish(
|
||||
aio_pika.Message(
|
||||
body=json.dumps(payload).encode(),
|
||||
delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
|
||||
),
|
||||
routing_key=MAIN_QUEUE,
|
||||
)
|
||||
|
||||
|
||||
async def relay_outbox(db: AsyncSession):
|
||||
"""Transactional outbox: publish + confirm, только потом published_at."""
|
||||
stmt = select(JobOutbox).where(JobOutbox.published_at.is_(None)).limit(10)
|
||||
result = await db.execute(stmt)
|
||||
for record in result.scalars().all():
|
||||
try:
|
||||
await publish_main(record.payload) # ждём publisher confirm
|
||||
record.published_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
logger.info("Relayed job %s to broker", record.job_id)
|
||||
except Exception as e:
|
||||
logger.error("Failed to relay job %s: %s", record.job_id, e)
|
||||
await db.rollback()
|
||||
|
||||
|
||||
async def run_relay():
|
||||
from app.database import AsyncSessionLocal
|
||||
while True:
|
||||
try:
|
||||
async with AsyncSessionLocal() as db:
|
||||
await relay_outbox(db)
|
||||
await asyncio.sleep(1)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Relay loop error: %s", e)
|
||||
await asyncio.sleep(5)
|
||||
@@ -0,0 +1,32 @@
|
||||
import os
|
||||
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
|
||||
from sqlalchemy.orm import sessionmaker, declarative_base
|
||||
from opentelemetry import metrics
|
||||
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", "postgresql+asyncpg://messenger:secure_db_password_123@postgres:5432/messenger_db")
|
||||
|
||||
engine = create_async_engine(DATABASE_URL, echo=False, pool_size=10, max_overflow=20, pool_pre_ping=True)
|
||||
AsyncSessionLocal = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
||||
Base = declarative_base()
|
||||
|
||||
_meter = metrics.get_meter("messenger.db")
|
||||
_meter.create_observable_gauge(
|
||||
"messenger.db.pool.size",
|
||||
callbacks=[lambda opts: [metrics.Observation(engine.pool.size())]],
|
||||
description="Pool connections",
|
||||
)
|
||||
_meter.create_observable_gauge(
|
||||
"messenger.db.pool.checked_out",
|
||||
callbacks=[lambda opts: [metrics.Observation(engine.pool.checkedout())]],
|
||||
description="Connections checked out",
|
||||
)
|
||||
_meter.create_observable_gauge(
|
||||
"messenger.db.pool.overflow",
|
||||
callbacks=[lambda opts: [metrics.Observation(engine.pool.overflow())]],
|
||||
description="Overflow connections",
|
||||
)
|
||||
|
||||
|
||||
async def get_db():
|
||||
async with AsyncSessionLocal() as session:
|
||||
yield session
|
||||
@@ -0,0 +1,54 @@
|
||||
import base64
|
||||
import logging
|
||||
from fastapi import Depends, HTTPException, status, Header
|
||||
from fastapi.security import HTTPBasic, HTTPBasicCredentials
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.database import get_db
|
||||
from app.models import User, UserSession
|
||||
from app.redis_client import get_redis
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
security = HTTPBasic()
|
||||
|
||||
async def get_basic_auth(credentials: HTTPBasicCredentials = Depends(security)):
|
||||
return credentials
|
||||
|
||||
async def get_current_user(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
redis_client = Depends(get_redis),
|
||||
authorization: str | None = Header(default=None),
|
||||
x_user_id: str | None = Header(default=None, alias="X-User-Id")
|
||||
):
|
||||
if not authorization:
|
||||
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Missing authorization header"}, headers={"WWW-Authenticate": "Bearer realm=\"messenger\""})
|
||||
|
||||
if not authorization.startswith("Bearer "):
|
||||
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Invalid authorization scheme"}, headers={"WWW-Authenticate": "Bearer realm=\"messenger\""})
|
||||
|
||||
token = authorization[7:]
|
||||
|
||||
# Проверяем в Redis сначала
|
||||
try:
|
||||
user_id_str = await redis_client.get(f"session:{token}")
|
||||
if user_id_str:
|
||||
import uuid
|
||||
result = await db.execute(select(User).where(User.id == uuid.UUID(user_id_str)))
|
||||
user = result.scalar_one_or_none()
|
||||
if user:
|
||||
return user
|
||||
except Exception as e:
|
||||
logger.warning(f"Redis unavailable during auth: {e}")
|
||||
|
||||
# Fallback на PostgreSQL
|
||||
result = await db.execute(select(UserSession).where(UserSession.token == token))
|
||||
session = result.scalar_one_or_none()
|
||||
if not session:
|
||||
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Invalid or expired session"}, headers={"WWW-Authenticate": "Bearer realm=\"messenger\""})
|
||||
|
||||
result = await db.execute(select(User).where(User.id == session.user_id))
|
||||
user = result.scalar_one_or_none()
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "User not found"})
|
||||
|
||||
return user
|
||||
+86
@@ -0,0 +1,86 @@
|
||||
import os
|
||||
import uuid
|
||||
import logging
|
||||
import traceback
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
# === ИНИЦИАЛИЗАЦИЯ TELEMETRY ДО ВСЕХ ОСТАЛЬНЫХ ИМПОРТОВ ===
|
||||
from app.telemetry import setup_telemetry
|
||||
setup_telemetry()
|
||||
# ==========================================================
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||
from app.routers import auth, chats, attachments
|
||||
from app.broker import run_relay
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Запускаем фоновый relay при старте приложения
|
||||
task = asyncio.create_task(run_relay())
|
||||
yield
|
||||
# Останавливаем при завершении
|
||||
task.cancel()
|
||||
|
||||
app = FastAPI(lifespan=lifespan)
|
||||
|
||||
app.include_router(auth.router)
|
||||
app.include_router(chats.router)
|
||||
app.include_router(attachments.router)
|
||||
|
||||
INSTANCE_ID = os.getenv("INSTANCE_ID", str(uuid.uuid4())[:8])
|
||||
|
||||
@app.middleware("http")
|
||||
async def add_instance_and_request_id(request: Request, call_next):
|
||||
request_id = request.headers.get("X-Request-Id") or str(uuid.uuid4())
|
||||
request.state.request_id = request_id
|
||||
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except Exception as e:
|
||||
logger.error(f"Unhandled exception: {e}")
|
||||
traceback.print_exc()
|
||||
return JSONResponse(
|
||||
status_code=500,
|
||||
content={"error": {"code": "internal_error", "message": "Unexpected error", "request_id": request_id}},
|
||||
headers={"X-Request-Id": request_id, "X-Instance-Id": INSTANCE_ID}
|
||||
)
|
||||
|
||||
response.headers["X-Request-Id"] = request.state.request_id
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
return response
|
||||
|
||||
@app.exception_handler(StarletteHTTPException)
|
||||
async def http_exception_handler(request: Request, exc: StarletteHTTPException):
|
||||
detail = exc.detail
|
||||
if isinstance(detail, dict):
|
||||
code = detail.get("code", "internal_error")
|
||||
message = detail.get("message", str(detail))
|
||||
else:
|
||||
code = "internal_error"
|
||||
message = str(detail)
|
||||
|
||||
return JSONResponse(
|
||||
status_code=exc.status_code,
|
||||
content={"error": {"code": code, "message": message, "request_id": request.state.request_id}},
|
||||
headers=exc.headers
|
||||
)
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_exception_handler(request: Request, exc: RequestValidationError):
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={"error": {"code": "invalid_request", "message": "Validation error", "request_id": request.state.request_id}},
|
||||
headers={"X-Request-Id": request.state.request_id, "X-Instance-Id": INSTANCE_ID}
|
||||
)
|
||||
|
||||
@app.get("/health/live")
|
||||
async def health_live():
|
||||
return {"status": "ok"}
|
||||
|
||||
@app.get("/health/ready")
|
||||
async def health_ready():
|
||||
return {"status": "ok"}
|
||||
+122
@@ -0,0 +1,122 @@
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from sqlalchemy import Column, String, DateTime, ForeignKey, Text, Boolean, BigInteger, Integer
|
||||
from sqlalchemy.orm import relationship
|
||||
from sqlalchemy.dialects.postgresql import UUID, JSONB
|
||||
from app.database import Base
|
||||
|
||||
class User(Base):
|
||||
__tablename__ = "users"
|
||||
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
username = Column(String(32), unique=True, nullable=False, index=True)
|
||||
display_name = Column(String(100), nullable=False)
|
||||
password_hash = Column(String(255), nullable=False)
|
||||
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
|
||||
sessions = relationship("UserSession", back_populates="user", cascade="all, delete-orphan")
|
||||
chat_memberships = relationship("ChatMember", back_populates="user", cascade="all, delete-orphan")
|
||||
|
||||
class UserSession(Base):
|
||||
__tablename__ = "sessions"
|
||||
token = Column(String(255), primary_key=True)
|
||||
user_id = Column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
|
||||
expires_at = Column(DateTime(timezone=True), nullable=False)
|
||||
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
|
||||
user = relationship("User", back_populates="sessions")
|
||||
|
||||
class Chat(Base):
|
||||
__tablename__ = "chats"
|
||||
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
title = Column(String(100), nullable=False)
|
||||
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
|
||||
members = relationship("ChatMember", back_populates="chat", cascade="all, delete-orphan")
|
||||
messages = relationship("Message", back_populates="chat", cascade="all, delete-orphan")
|
||||
|
||||
class ChatMember(Base):
|
||||
__tablename__ = "chat_members"
|
||||
chat_id = Column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), primary_key=True)
|
||||
user_id = Column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), primary_key=True)
|
||||
joined_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
|
||||
chat = relationship("Chat", back_populates="members")
|
||||
user = relationship("User", back_populates="chat_memberships")
|
||||
|
||||
class Message(Base):
|
||||
__tablename__ = "messages"
|
||||
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
chat_id = Column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
sender_id = Column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
|
||||
text = Column(Text, nullable=False)
|
||||
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), index=True)
|
||||
chat = relationship("Chat", back_populates="messages")
|
||||
sender = relationship("User")
|
||||
attachments = relationship("MessageAttachment", back_populates="message", cascade="all, delete-orphan")
|
||||
|
||||
class IdempotencyKey(Base):
|
||||
__tablename__ = "idempotency_keys"
|
||||
key = Column(String(255), primary_key=True)
|
||||
chat_id = Column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False)
|
||||
response_data = Column(Text, nullable=False)
|
||||
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
|
||||
|
||||
# === НОВЫЕ МОДЕЛИ ДЛЯ LAB 5 ===
|
||||
|
||||
class Attachment(Base):
|
||||
__tablename__ = "attachments"
|
||||
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
owner_id = Column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
|
||||
chat_id = Column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False)
|
||||
|
||||
# S3 ключи
|
||||
staging_key = Column(String(512), nullable=True) # Временное хранилище до обработки
|
||||
s3_key = Column(String(512), nullable=True) # Финальный ключ оригинала (заполняется worker)
|
||||
|
||||
original_filename = Column(String(255), nullable=False)
|
||||
content_type = Column(String(100), nullable=False)
|
||||
size_bytes = Column(BigInteger, nullable=False)
|
||||
sha256 = Column(String(64), nullable=False)
|
||||
|
||||
# Статус обработки
|
||||
status = Column(String(20), nullable=False, default="queued") # queued, processing, ready, failed
|
||||
error_code = Column(String(50), nullable=True)
|
||||
|
||||
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
|
||||
|
||||
owner = relationship("User")
|
||||
chat = relationship("Chat")
|
||||
message_links = relationship("MessageAttachment", back_populates="attachment", cascade="all, delete-orphan")
|
||||
variants = relationship("AttachmentVariant", back_populates="attachment", cascade="all, delete-orphan")
|
||||
job = relationship("AttachmentJob", back_populates="attachment", uselist=False, cascade="all, delete-orphan")
|
||||
|
||||
class AttachmentVariant(Base):
|
||||
__tablename__ = "attachment_variants"
|
||||
attachment_id = Column(UUID(as_uuid=True), ForeignKey("attachments.id", ondelete="CASCADE"), primary_key=True)
|
||||
variant_name = Column(String(20), primary_key=True) # 'thumbnail', 'crop', 'optimized'
|
||||
s3_key = Column(String(512), nullable=False)
|
||||
content_type = Column(String(100), nullable=False)
|
||||
size_bytes = Column(BigInteger, nullable=False)
|
||||
width = Column(Integer, nullable=False)
|
||||
height = Column(Integer, nullable=False)
|
||||
sha256 = Column(String(64), nullable=False)
|
||||
attachment = relationship("Attachment", back_populates="variants")
|
||||
|
||||
class AttachmentJob(Base):
|
||||
__tablename__ = "attachment_jobs"
|
||||
job_id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
attachment_id = Column(UUID(as_uuid=True), ForeignKey("attachments.id", ondelete="CASCADE"), unique=True, nullable=False)
|
||||
attempt = Column(Integer, nullable=False, default=1)
|
||||
status = Column(String(20), nullable=False, default="queued") # queued, processing, ready, failed
|
||||
lease_until = Column(DateTime(timezone=True), nullable=True)
|
||||
lease_owner = Column(String(100), nullable=True)
|
||||
attachment = relationship("Attachment", back_populates="job")
|
||||
|
||||
class JobOutbox(Base):
|
||||
__tablename__ = "job_outbox"
|
||||
job_id = Column(UUID(as_uuid=True), primary_key=True)
|
||||
payload = Column(JSONB, nullable=False)
|
||||
published_at = Column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
class MessageAttachment(Base):
|
||||
__tablename__ = "message_attachments"
|
||||
message_id = Column(UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), primary_key=True)
|
||||
attachment_id = Column(UUID(as_uuid=True), ForeignKey("attachments.id", ondelete="CASCADE"), primary_key=True)
|
||||
message = relationship("Message", back_populates="attachments")
|
||||
attachment = relationship("Attachment", back_populates="message_links")
|
||||
@@ -0,0 +1,9 @@
|
||||
import os
|
||||
import redis.asyncio as redis
|
||||
|
||||
REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379/0")
|
||||
|
||||
redis_client = redis.from_url(REDIS_URL, decode_responses=True)
|
||||
|
||||
async def get_redis():
|
||||
return redis_client
|
||||
@@ -0,0 +1,245 @@
|
||||
import os
|
||||
import uuid
|
||||
import hashlib
|
||||
import magic
|
||||
import tempfile
|
||||
from datetime import datetime, timezone
|
||||
from fastapi import APIRouter, Depends, HTTPException, status, Query, Request
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.database import get_db
|
||||
from app.models import Attachment, ChatMember, AttachmentJob, JobOutbox
|
||||
from app.schemas import Attachment as AttachmentSchema, Variant, ErrorResponse
|
||||
from app.dependencies import get_current_user
|
||||
from app.s3_client import get_s3_client, upload_file_to_s3, delete_file_from_s3
|
||||
from minio import Minio
|
||||
from PIL import Image
|
||||
import io
|
||||
from fastapi.responses import Response
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/attachments", status_code=status.HTTP_202_ACCEPTED, responses={202: {"model": AttachmentSchema}, 400: {"model": ErrorResponse}, 401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}, 413: {"model": ErrorResponse}, 415: {"model": ErrorResponse}, 503: {"model": ErrorResponse}})
|
||||
async def upload_attachment(
|
||||
chat_id: uuid.UUID,
|
||||
filename: str = Query(..., min_length=1, max_length=255),
|
||||
request: Request = None,
|
||||
current_user = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
s3: Minio = Depends(get_s3_client)
|
||||
):
|
||||
# 1. Проверка доступа к чату
|
||||
is_member = await db.execute(
|
||||
select(ChatMember).where(ChatMember.chat_id == chat_id, ChatMember.user_id == current_user.id)
|
||||
)
|
||||
if not is_member.scalar_one_or_none():
|
||||
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Chat not found or access denied"})
|
||||
|
||||
# 2. Чтение сырых байтов с ограничением (защита от DoS)
|
||||
MAX_SIZE = 10 * 1024 * 1024 # 10 MiB
|
||||
body = await request.body()
|
||||
if len(body) > MAX_SIZE:
|
||||
raise HTTPException(status_code=413, detail={"code": "payload_too_large", "message": "File size exceeds 10 MiB limit"})
|
||||
if len(body) == 0:
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Empty file"})
|
||||
|
||||
# 3. Определение MIME-типа по содержимому
|
||||
mime = magic.from_buffer(body, mime=True)
|
||||
if mime not in ("image/png", "image/jpeg"):
|
||||
raise HTTPException(status_code=415, detail={"code": "unsupported_media_type", "message": "Only image/png and image/jpeg are supported"})
|
||||
|
||||
# 4. Быстрая проверка размеров изображения (без полного декодирования)
|
||||
try:
|
||||
img = Image.open(io.BytesIO(body))
|
||||
width, height = img.size
|
||||
if width * height > 40_000_000:
|
||||
raise HTTPException(status_code=413, detail={"code": "payload_too_large", "message": "Image resolution exceeds 40 million pixels"})
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid or corrupted image file"})
|
||||
|
||||
# 5. Вычисление SHA256
|
||||
sha256_hash = hashlib.sha256(body).hexdigest()
|
||||
|
||||
# 6. Генерация ключей
|
||||
att_id = uuid.uuid4()
|
||||
ext = "png" if mime == "image/png" else "jpeg"
|
||||
staging_key = f"staging/{att_id}.{ext}"
|
||||
|
||||
# 7. Загрузка в S3 staging
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=f".{ext}") as tmp:
|
||||
tmp.write(body)
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
await upload_file_to_s3(staging_key, tmp_path, mime)
|
||||
except Exception:
|
||||
raise HTTPException(status_code=503, detail={"code": "unavailable", "message": "Failed to upload to S3 staging"})
|
||||
finally:
|
||||
os.unlink(tmp_path)
|
||||
|
||||
# 8. Транзакция: Создание Attachment, Job и Outbox сообщения
|
||||
job_id = uuid.uuid4()
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
new_attachment = Attachment(
|
||||
id=att_id,
|
||||
owner_id=current_user.id,
|
||||
chat_id=chat_id,
|
||||
staging_key=staging_key,
|
||||
original_filename=filename,
|
||||
content_type=mime,
|
||||
size_bytes=len(body),
|
||||
sha256=sha256_hash,
|
||||
status="queued"
|
||||
)
|
||||
|
||||
new_job = AttachmentJob(
|
||||
job_id=job_id,
|
||||
attachment_id=att_id,
|
||||
attempt=1,
|
||||
status="queued"
|
||||
)
|
||||
|
||||
outbox_payload = {
|
||||
"job_id": str(job_id),
|
||||
"attachment_id": str(att_id),
|
||||
"attempt": 1
|
||||
}
|
||||
|
||||
new_outbox = JobOutbox(
|
||||
job_id=job_id,
|
||||
payload=outbox_payload,
|
||||
published_at=None
|
||||
)
|
||||
|
||||
db.add(new_attachment)
|
||||
db.add(new_job)
|
||||
db.add(new_outbox)
|
||||
|
||||
try:
|
||||
await db.commit()
|
||||
except Exception:
|
||||
await db.rollback()
|
||||
try:
|
||||
await delete_file_from_s3(staging_key)
|
||||
except Exception:
|
||||
pass
|
||||
raise HTTPException(status_code=503, detail={"code": "unavailable", "message": "Database error during job creation"})
|
||||
|
||||
now_str = now.isoformat().replace('+00:00', 'Z')
|
||||
|
||||
# Возвращаем ПОЛНЫЙ объект AttachmentSchema, как требует OpenAPI контракт
|
||||
return AttachmentSchema(
|
||||
id=str(att_id),
|
||||
chat_id=str(chat_id),
|
||||
owner_id=str(current_user.id),
|
||||
filename=filename,
|
||||
content_type=mime,
|
||||
size_bytes=len(body),
|
||||
sha256=sha256_hash,
|
||||
status="queued",
|
||||
variants=[],
|
||||
error_code=None,
|
||||
created_at=now_str,
|
||||
updated_at=now_str
|
||||
)
|
||||
|
||||
@router.get("/api/v1/attachments/{attachment_id}", response_model=AttachmentSchema, responses={401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}})
|
||||
async def get_attachment_metadata(
|
||||
attachment_id: uuid.UUID,
|
||||
current_user = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
stmt = select(Attachment).where(Attachment.id == attachment_id)
|
||||
result = await db.execute(stmt)
|
||||
attachment = result.scalar_one_or_none()
|
||||
|
||||
if not attachment:
|
||||
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Attachment not found"})
|
||||
|
||||
is_member = await db.execute(
|
||||
select(ChatMember).where(ChatMember.chat_id == attachment.chat_id, ChatMember.user_id == current_user.id)
|
||||
)
|
||||
if not is_member.scalar_one_or_none():
|
||||
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Access denied"})
|
||||
|
||||
variants_list = []
|
||||
if attachment.status == "ready":
|
||||
from app.models import AttachmentVariant
|
||||
var_stmt = select(AttachmentVariant).where(AttachmentVariant.attachment_id == attachment_id)
|
||||
var_result = await db.execute(var_stmt)
|
||||
for v in var_result.scalars().all():
|
||||
variants_list.append(Variant(
|
||||
name=v.variant_name,
|
||||
content_type=v.content_type,
|
||||
size_bytes=v.size_bytes,
|
||||
width=v.width,
|
||||
height=v.height,
|
||||
sha256=v.sha256
|
||||
))
|
||||
|
||||
now_str = attachment.created_at.isoformat().replace('+00:00', 'Z')
|
||||
return AttachmentSchema(
|
||||
id=str(attachment.id),
|
||||
chat_id=str(attachment.chat_id),
|
||||
owner_id=str(attachment.owner_id),
|
||||
filename=attachment.original_filename,
|
||||
content_type=attachment.content_type,
|
||||
size_bytes=attachment.size_bytes,
|
||||
sha256=attachment.sha256,
|
||||
status=attachment.status,
|
||||
variants=variants_list,
|
||||
error_code=attachment.error_code,
|
||||
created_at=now_str,
|
||||
updated_at=now_str
|
||||
)
|
||||
|
||||
@router.get("/api/v1/attachments/{attachment_id}/content", responses={401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}, 409: {"model": ErrorResponse}})
|
||||
async def download_attachment(
|
||||
attachment_id: uuid.UUID,
|
||||
variant: str = Query(default="original"),
|
||||
current_user = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
s3: Minio = Depends(get_s3_client)
|
||||
):
|
||||
stmt = select(Attachment).where(Attachment.id == attachment_id)
|
||||
result = await db.execute(stmt)
|
||||
attachment = result.scalar_one_or_none()
|
||||
|
||||
if not attachment:
|
||||
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Attachment not found"})
|
||||
|
||||
is_member = await db.execute(
|
||||
select(ChatMember).where(ChatMember.chat_id == attachment.chat_id, ChatMember.user_id == current_user.id)
|
||||
)
|
||||
if not is_member.scalar_one_or_none():
|
||||
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Access denied"})
|
||||
|
||||
if attachment.status != "ready":
|
||||
raise HTTPException(status_code=409, detail={"code": "conflict", "message": f"Attachment is not ready yet (status: {attachment.status})"})
|
||||
|
||||
if variant == "original":
|
||||
s3_key = attachment.s3_key
|
||||
content_type = attachment.content_type
|
||||
else:
|
||||
from app.models import AttachmentVariant
|
||||
var_stmt = select(AttachmentVariant).where(
|
||||
AttachmentVariant.attachment_id == attachment_id,
|
||||
AttachmentVariant.variant_name == variant
|
||||
)
|
||||
var_result = await db.execute(var_stmt)
|
||||
var_obj = var_result.scalar_one_or_none()
|
||||
if not var_obj:
|
||||
raise HTTPException(status_code=404, detail={"code": "not_found", "message": f"Variant {variant} not found"})
|
||||
s3_key = var_obj.s3_key
|
||||
content_type = var_obj.content_type
|
||||
|
||||
try:
|
||||
response = s3.get_object("messenger-attachments", s3_key)
|
||||
data = response.read()
|
||||
response.close()
|
||||
response.release_conn()
|
||||
except Exception:
|
||||
raise HTTPException(status_code=503, detail={"code": "unavailable", "message": "Failed to download from S3"})
|
||||
|
||||
return Response(content=data, media_type=content_type)
|
||||
@@ -0,0 +1,115 @@
|
||||
import secrets
|
||||
import logging
|
||||
import bcrypt
|
||||
import asyncpg
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from fastapi import APIRouter, Depends, HTTPException, status, Response
|
||||
from fastapi.security import HTTPBearer
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
from app.database import get_db
|
||||
from app.models import User as UserModel, UserSession
|
||||
from app.schemas import CreateUser, User, Session, ErrorResponse
|
||||
from app.dependencies import get_basic_auth, get_current_user
|
||||
from app.redis_client import get_redis
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter()
|
||||
|
||||
def verify_password(plain_password: str, hashed_password: str) -> bool:
|
||||
return bcrypt.checkpw(plain_password.encode('utf-8'), hashed_password.encode('utf-8'))
|
||||
|
||||
def get_password_hash(password: str) -> str:
|
||||
return bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt()).decode('utf-8')
|
||||
|
||||
@router.post("/api/v1/users", response_model=User, status_code=status.HTTP_201_CREATED, responses={409: {"model": ErrorResponse}})
|
||||
async def register_user(user_in: CreateUser, db: AsyncSession = Depends(get_db)):
|
||||
new_user = UserModel(
|
||||
username=user_in.username,
|
||||
display_name=user_in.display_name,
|
||||
password_hash=get_password_hash(user_in.password)
|
||||
)
|
||||
db.add(new_user)
|
||||
try:
|
||||
await db.commit()
|
||||
await db.refresh(new_user)
|
||||
return User(
|
||||
id=str(new_user.id),
|
||||
username=new_user.username,
|
||||
display_name=new_user.display_name,
|
||||
created_at=new_user.created_at.isoformat().replace('+00:00', 'Z')
|
||||
)
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
if isinstance(e.__cause__, asyncpg.exceptions.UniqueViolationError) or "unique constraint" in str(e).lower():
|
||||
raise HTTPException(status_code=409, detail={"code": "conflict", "message": "Username already exists"})
|
||||
logger.error(f"Database error during registration: {e}")
|
||||
raise HTTPException(status_code=500, detail={"code": "internal_error", "message": "Database error"})
|
||||
|
||||
@router.post("/api/v1/auth/sessions", response_model=Session, status_code=status.HTTP_201_CREATED, responses={401: {"model": ErrorResponse}})
|
||||
async def create_session(credentials = Depends(get_basic_auth), db: AsyncSession = Depends(get_db), redis_client = Depends(get_redis)):
|
||||
result = await db.execute(select(UserModel).where(UserModel.username == credentials.username))
|
||||
user = result.scalar_one_or_none()
|
||||
if not user or not verify_password(credentials.password, user.password_hash):
|
||||
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Invalid username or password"}, headers={"WWW-Authenticate": "Basic realm=\"messenger\", charset=\"UTF-8\""})
|
||||
|
||||
token = secrets.token_urlsafe(32)
|
||||
expires_at = datetime.now(timezone.utc) + timedelta(seconds=3600)
|
||||
|
||||
try:
|
||||
await redis_client.setex(f"session:{token}", 3600, str(user.id))
|
||||
except Exception as e:
|
||||
logger.warning(f"Redis unavailable during session creation: {e}")
|
||||
|
||||
try:
|
||||
db.add(UserSession(token=token, user_id=user.id, expires_at=expires_at))
|
||||
await db.commit()
|
||||
except SQLAlchemyError as e:
|
||||
await db.rollback()
|
||||
logger.error(f"Failed to save session to PostgreSQL: {e}")
|
||||
raise HTTPException(status_code=500, detail={"code": "internal_error", "message": "Failed to create session"})
|
||||
|
||||
return Session(
|
||||
token=token,
|
||||
token_type="Bearer",
|
||||
expires_at=expires_at,
|
||||
user=User(
|
||||
id=str(user.id),
|
||||
username=user.username,
|
||||
display_name=user.display_name,
|
||||
created_at=user.created_at.isoformat().replace('+00:00', 'Z')
|
||||
)
|
||||
)
|
||||
|
||||
@router.delete("/api/v1/auth/sessions/current", status_code=status.HTTP_204_NO_CONTENT, responses={401: {"model": ErrorResponse}})
|
||||
async def revoke_session(credentials = Depends(HTTPBearer()), db: AsyncSession = Depends(get_db), redis_client = Depends(get_redis)):
|
||||
token = credentials.credentials
|
||||
try:
|
||||
await redis_client.delete(f"session:{token}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Redis unavailable during session revocation: {e}")
|
||||
|
||||
try:
|
||||
result = await db.execute(select(UserSession).where(UserSession.token == token))
|
||||
session = result.scalar_one_or_none()
|
||||
if session:
|
||||
await db.delete(session)
|
||||
await db.commit()
|
||||
else:
|
||||
raise HTTPException(status_code=401, detail={"code": "unauthorized", "message": "Invalid or expired session"}, headers={"WWW-Authenticate": "Bearer realm=\"messenger\""})
|
||||
except HTTPException:
|
||||
raise
|
||||
except SQLAlchemyError as e:
|
||||
await db.rollback()
|
||||
raise HTTPException(status_code=500, detail={"code": "internal_error", "message": "Failed to revoke session"})
|
||||
return Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
|
||||
@router.get("/api/v1/users/me", response_model=User, responses={401: {"model": ErrorResponse}})
|
||||
async def current_user(current_user: UserModel = Depends(get_current_user)):
|
||||
return User(
|
||||
id=str(current_user.id),
|
||||
username=current_user.username,
|
||||
display_name=current_user.display_name,
|
||||
created_at=current_user.created_at.isoformat().replace('+00:00', 'Z')
|
||||
)
|
||||
@@ -0,0 +1,213 @@
|
||||
import uuid
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
from fastapi import APIRouter, Depends, HTTPException, status, Query, Header
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
from app.database import get_db
|
||||
from app.models import Chat as ChatModel, ChatMember as ChatMemberModel, Message as MessageModel, User as UserModel, IdempotencyKey, MessageAttachment, Attachment
|
||||
from app.schemas import Chat, CreateChat, Message, CreateMessage, ChatPage, MessagePage, ErrorResponse
|
||||
from app.dependencies import get_current_user
|
||||
from app.redis_client import get_redis
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@router.post("/api/v1/chats", response_model=Chat, status_code=status.HTTP_201_CREATED, responses={400: {"model": ErrorResponse}, 401: {"model": ErrorResponse}})
|
||||
async def create_chat(chat_in: CreateChat, current_user: UserModel = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
|
||||
try:
|
||||
member_uuids = [uuid.UUID(uid) for uid in chat_in.member_ids]
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid member_id format"})
|
||||
|
||||
if current_user.id not in member_uuids:
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Creator must be in member_ids"})
|
||||
|
||||
user_ids = list(set(member_uuids))
|
||||
result = await db.execute(select(UserModel).where(UserModel.id.in_(user_ids)))
|
||||
if len(result.scalars().all()) != len(user_ids):
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "One or more member_ids do not exist"})
|
||||
|
||||
new_chat = ChatModel(title=chat_in.title)
|
||||
db.add(new_chat)
|
||||
await db.flush()
|
||||
|
||||
db.add_all([ChatMemberModel(chat_id=new_chat.id, user_id=uid) for uid in user_ids])
|
||||
await db.commit()
|
||||
await db.refresh(new_chat)
|
||||
|
||||
return Chat(
|
||||
id=str(new_chat.id),
|
||||
title=new_chat.title,
|
||||
member_ids=[str(uid) for uid in user_ids],
|
||||
created_at=new_chat.created_at.isoformat().replace('+00:00', 'Z')
|
||||
)
|
||||
|
||||
@router.get("/api/v1/chats", response_model=ChatPage, responses={401: {"model": ErrorResponse}})
|
||||
async def list_chats(limit: int = Query(50, ge=1, le=100), cursor: str | None = None, current_user: UserModel = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
|
||||
stmt = select(ChatModel).options(selectinload(ChatModel.members)).join(ChatMemberModel, ChatModel.id == ChatMemberModel.chat_id).where(ChatMemberModel.user_id == current_user.id).order_by(ChatModel.created_at.desc(), ChatModel.id.desc())
|
||||
if cursor:
|
||||
try:
|
||||
created_at_str, chat_id_str = cursor.split(",")
|
||||
created_at = datetime.fromisoformat(created_at_str.replace("Z", "+00:00"))
|
||||
chat_id = uuid.UUID(chat_id_str)
|
||||
stmt = stmt.where((ChatModel.created_at < created_at) | ((ChatModel.created_at == created_at) & (ChatModel.id < chat_id)))
|
||||
except (ValueError, TypeError):
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid cursor"})
|
||||
stmt = stmt.limit(limit + 1)
|
||||
chats = (await db.execute(stmt)).scalars().all()
|
||||
has_next = len(chats) > limit
|
||||
if has_next:
|
||||
chats = chats[:-1]
|
||||
next_cursor = f"{chats[-1].created_at.isoformat().replace('+00:00', 'Z')},{chats[-1].id}"
|
||||
else:
|
||||
next_cursor = None
|
||||
items = [Chat(id=str(c.id), title=c.title, member_ids=[str(m.user_id) for m in c.members], created_at=c.created_at.isoformat().replace('+00:00', 'Z')) for c in chats]
|
||||
return ChatPage(items=items, next_cursor=next_cursor)
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}", response_model=Chat, responses={401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}})
|
||||
async def get_chat(chat_id: uuid.UUID, current_user: UserModel = Depends(get_current_user), db: AsyncSession = Depends(get_db)):
|
||||
stmt = select(ChatModel).options(selectinload(ChatModel.members)).join(ChatMemberModel, ChatModel.id == ChatMemberModel.chat_id).where(ChatModel.id == chat_id, ChatMemberModel.user_id == current_user.id)
|
||||
chat = (await db.execute(stmt)).scalar_one_or_none()
|
||||
if not chat:
|
||||
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Chat not found or access denied"})
|
||||
return Chat(id=str(chat.id), title=chat.title, member_ids=[str(m.user_id) for m in chat.members], created_at=chat.created_at.isoformat().replace('+00:00', 'Z'))
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/messages", response_model=Message, status_code=status.HTTP_201_CREATED, responses={400: {"model": ErrorResponse}, 401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}, 409: {"model": ErrorResponse}})
|
||||
async def create_message(
|
||||
chat_id: uuid.UUID,
|
||||
message_in: CreateMessage,
|
||||
current_user: UserModel = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
redis_client = Depends(get_redis),
|
||||
idempotency_key: str | None = Header(default=None, alias="Idempotency-Key")
|
||||
):
|
||||
if not idempotency_key:
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Idempotency-Key header is required"})
|
||||
|
||||
if not (await db.execute(select(ChatMemberModel).where(ChatMemberModel.chat_id == chat_id, ChatMemberModel.user_id == current_user.id))).scalar_one_or_none():
|
||||
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Chat not found or access denied"})
|
||||
|
||||
existing_key = await db.execute(select(IdempotencyKey).where(IdempotencyKey.key == idempotency_key))
|
||||
existing_record = existing_key.scalar_one_or_none()
|
||||
|
||||
if existing_record:
|
||||
# Исправлено: response_data хранится как JSON-строка, нужно распарсить
|
||||
resp_data = json.loads(existing_record.response_data)
|
||||
if resp_data.get("text") != message_in.text:
|
||||
raise HTTPException(status_code=409, detail={"code": "conflict", "message": "Idempotency-Key already used with different payload"})
|
||||
return Message(**resp_data)
|
||||
|
||||
new_message = MessageModel(chat_id=chat_id, sender_id=current_user.id, text=message_in.text)
|
||||
db.add(new_message)
|
||||
await db.flush()
|
||||
|
||||
# Привязываем вложения, если есть
|
||||
attachment_ids = []
|
||||
if message_in.attachment_ids:
|
||||
for att_id_str in message_in.attachment_ids:
|
||||
try:
|
||||
att_id = uuid.UUID(att_id_str)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid attachment_id format"})
|
||||
|
||||
# Проверяем, что вложение принадлежит этому чату
|
||||
att_result = await db.execute(select(Attachment).where(Attachment.id == att_id, Attachment.chat_id == chat_id))
|
||||
if not att_result.scalar_one_or_none():
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Attachment not found or does not belong to this chat"})
|
||||
|
||||
db.add(MessageAttachment(message_id=new_message.id, attachment_id=att_id))
|
||||
attachment_ids.append(att_id_str)
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(new_message)
|
||||
|
||||
response_data_dict = {
|
||||
"id": str(new_message.id),
|
||||
"chat_id": str(new_message.chat_id),
|
||||
"sender_id": str(new_message.sender_id),
|
||||
"text": new_message.text,
|
||||
"created_at": new_message.created_at.isoformat().replace('+00:00', 'Z'),
|
||||
"attachment_ids": attachment_ids
|
||||
}
|
||||
|
||||
# Исправлено: сохраняем как JSON-строку
|
||||
db.add(IdempotencyKey(key=idempotency_key, chat_id=chat_id, response_data=json.dumps(response_data_dict)))
|
||||
await db.commit()
|
||||
|
||||
try:
|
||||
await redis_client.delete(f"messages:{chat_id}:*")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return Message(**response_data_dict)
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePage, responses={400: {"model": ErrorResponse}, 401: {"model": ErrorResponse}, 404: {"model": ErrorResponse}})
|
||||
async def list_messages(chat_id: uuid.UUID, limit: int = Query(50, ge=1, le=100), cursor: str | None = None, current_user: UserModel = Depends(get_current_user), db: AsyncSession = Depends(get_db), redis_client = Depends(get_redis)):
|
||||
if not (await db.execute(select(ChatMemberModel).where(ChatMemberModel.chat_id == chat_id, ChatMemberModel.user_id == current_user.id))).scalar_one_or_none():
|
||||
raise HTTPException(status_code=404, detail={"code": "not_found", "message": "Chat not found or access denied"})
|
||||
|
||||
cache_key = f"messages:{chat_id}:{limit}:{cursor or 'null'}"
|
||||
redis_available = True
|
||||
try:
|
||||
cached_data = await redis_client.get(cache_key)
|
||||
if cached_data:
|
||||
await redis_client.expire(cache_key, 60)
|
||||
return MessagePage.model_validate_json(cached_data)
|
||||
except Exception:
|
||||
redis_available = False
|
||||
|
||||
lock_acquired = False
|
||||
if redis_available:
|
||||
lock_key = f"lock:{cache_key}"
|
||||
try:
|
||||
lock_acquired = await redis_client.set(lock_key, "1", nx=True, ex=5)
|
||||
if not lock_acquired:
|
||||
stale_data = await redis_client.get(f"stale:{cache_key}")
|
||||
if stale_data: return MessagePage.model_validate_json(stale_data)
|
||||
except Exception:
|
||||
redis_available = False
|
||||
|
||||
stmt = select(MessageModel).where(MessageModel.chat_id == chat_id).order_by(MessageModel.created_at.asc(), MessageModel.id.asc())
|
||||
if cursor:
|
||||
try:
|
||||
created_at_str, msg_id_str = cursor.split(",")
|
||||
created_at = datetime.fromisoformat(created_at_str.replace("Z", "+00:00"))
|
||||
msg_id = uuid.UUID(msg_id_str)
|
||||
stmt = stmt.where((MessageModel.created_at > created_at) | ((MessageModel.created_at == created_at) & (MessageModel.id > msg_id)))
|
||||
except (ValueError, TypeError):
|
||||
raise HTTPException(status_code=400, detail={"code": "invalid_request", "message": "Invalid cursor"})
|
||||
|
||||
stmt = stmt.limit(limit + 1)
|
||||
messages = (await db.execute(stmt)).scalars().all()
|
||||
has_next = len(messages) > limit
|
||||
if has_next:
|
||||
messages = messages[:-1]
|
||||
next_cursor = f"{messages[-1].created_at.isoformat().replace('+00:00', 'Z')},{messages[-1].id}"
|
||||
else:
|
||||
next_cursor = None
|
||||
|
||||
items = []
|
||||
for m in messages:
|
||||
att_result = await db.execute(select(MessageAttachment.attachment_id).where(MessageAttachment.message_id == m.id))
|
||||
attachment_ids = [str(aid) for aid in att_result.scalars().all()]
|
||||
items.append(Message(
|
||||
id=str(m.id),
|
||||
chat_id=str(m.chat_id),
|
||||
sender_id=str(m.sender_id),
|
||||
text=m.text,
|
||||
created_at=m.created_at.isoformat().replace('+00:00', 'Z'),
|
||||
attachment_ids=attachment_ids
|
||||
))
|
||||
|
||||
page = MessagePage(items=items, next_cursor=next_cursor)
|
||||
|
||||
if redis_available:
|
||||
try:
|
||||
old_cache = await redis_client.get(cache_key)
|
||||
if old_cache: await redis_client.setex(f"stale:{cache_key}", 120, old_cache)
|
||||
await redis_client.setex(cache_key, 60, page.model_dump_json())
|
||||
if lock_acquired: await redis_client.delete(f"lock:{cache_key}")
|
||||
except Exception:
|
||||
pass
|
||||
return page
|
||||
@@ -0,0 +1,40 @@
|
||||
import os
|
||||
import asyncio
|
||||
from minio import Minio
|
||||
|
||||
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333").replace("http://", "").replace("https://", "")
|
||||
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "admin")
|
||||
S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "admin")
|
||||
S3_BUCKET_NAME = os.getenv("S3_BUCKET_NAME", "messenger-attachments")
|
||||
S3_SECURE = os.getenv("S3_ENDPOINT", "http://s3:8333").startswith("https://")
|
||||
|
||||
s3_client = Minio(
|
||||
S3_ENDPOINT,
|
||||
access_key=S3_ACCESS_KEY,
|
||||
secret_key=S3_SECRET_KEY,
|
||||
secure=S3_SECURE
|
||||
)
|
||||
|
||||
def get_s3_client():
|
||||
return s3_client
|
||||
|
||||
async def upload_file_to_s3(object_name: str, file_path: str, content_type: str):
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(
|
||||
None,
|
||||
lambda: s3_client.fput_object(S3_BUCKET_NAME, object_name, file_path, content_type=content_type)
|
||||
)
|
||||
|
||||
async def get_file_from_s3(object_name: str):
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: s3_client.get_object(S3_BUCKET_NAME, object_name)
|
||||
)
|
||||
|
||||
async def delete_file_from_s3(object_name: str):
|
||||
loop = asyncio.get_running_loop()
|
||||
await loop.run_in_executor(
|
||||
None,
|
||||
lambda: s3_client.remove_object(S3_BUCKET_NAME, object_name)
|
||||
)
|
||||
@@ -0,0 +1,88 @@
|
||||
from pydantic import BaseModel, Field, ConfigDict, model_validator
|
||||
from typing import List, Optional
|
||||
from datetime import datetime
|
||||
|
||||
class User(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: str
|
||||
username: str
|
||||
display_name: str
|
||||
created_at: str
|
||||
|
||||
class CreateUser(BaseModel):
|
||||
username: str = Field(..., min_length=3, max_length=32, pattern="^[a-z0-9_]+$")
|
||||
display_name: str = Field(..., min_length=1, max_length=100)
|
||||
password: str = Field(..., min_length=12, max_length=128)
|
||||
|
||||
class Session(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
token: str
|
||||
token_type: str
|
||||
expires_at: datetime
|
||||
user: User
|
||||
|
||||
class Chat(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: str
|
||||
title: str
|
||||
member_ids: List[str]
|
||||
created_at: str
|
||||
|
||||
class CreateChat(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=100)
|
||||
member_ids: List[str]
|
||||
|
||||
class ChatPage(BaseModel):
|
||||
items: List[Chat]
|
||||
next_cursor: Optional[str] = None
|
||||
|
||||
class Message(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: str
|
||||
chat_id: str
|
||||
sender_id: str
|
||||
text: str
|
||||
created_at: str
|
||||
attachment_ids: List[str]
|
||||
|
||||
class CreateMessage(BaseModel):
|
||||
text: str = Field(default="", max_length=4000)
|
||||
attachment_ids: List[str] = Field(default_factory=list)
|
||||
|
||||
@model_validator(mode='after')
|
||||
def check_text_or_attachments(self):
|
||||
if len(self.text) == 0 and len(self.attachment_ids) == 0:
|
||||
raise ValueError("Either text or attachment_ids must be provided")
|
||||
if len(self.attachment_ids) > 10:
|
||||
raise ValueError("attachment_ids cannot exceed 10 items")
|
||||
return self
|
||||
|
||||
class MessagePage(BaseModel):
|
||||
items: List[Message]
|
||||
next_cursor: Optional[str] = None
|
||||
|
||||
class Variant(BaseModel):
|
||||
name: str
|
||||
content_type: str
|
||||
size_bytes: int
|
||||
width: int
|
||||
height: int
|
||||
sha256: str
|
||||
|
||||
class Attachment(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
id: str
|
||||
chat_id: str
|
||||
owner_id: str
|
||||
filename: str
|
||||
content_type: str
|
||||
size_bytes: int
|
||||
sha256: str
|
||||
status: str
|
||||
variants: List[Variant]
|
||||
error_code: Optional[str] = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
class ErrorResponse(BaseModel):
|
||||
error: dict
|
||||
@@ -0,0 +1,59 @@
|
||||
import os
|
||||
import logging
|
||||
from opentelemetry import trace, metrics, _logs
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
|
||||
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
|
||||
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
|
||||
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
|
||||
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
|
||||
from opentelemetry.instrumentation.asyncpg import AsyncPGInstrumentor
|
||||
from opentelemetry.instrumentation.aio_pika import AioPikaInstrumentor
|
||||
from opentelemetry.instrumentation.logging import LoggingInstrumentor
|
||||
|
||||
def setup_telemetry():
|
||||
service_name = os.getenv("OTEL_SERVICE_NAME", "messenger")
|
||||
resource = Resource.create({
|
||||
"service.name": service_name,
|
||||
"service.version": os.getenv("OTEL_SERVICE_VERSION", "1.0.0"),
|
||||
"deployment.environment": os.getenv("OTEL_ENVIRONMENT", "lab")
|
||||
})
|
||||
|
||||
# Базовый endpoint: экспортеры сами добавят /v1/traces, /v1/metrics, /v1/logs
|
||||
otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4318")
|
||||
|
||||
# 1. Traces
|
||||
trace.set_tracer_provider(TracerProvider(resource=resource))
|
||||
span_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
|
||||
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(span_exporter))
|
||||
|
||||
# 2. Metrics
|
||||
metric_reader = PeriodicExportingMetricReader(
|
||||
OTLPMetricExporter(endpoint=otlp_endpoint),
|
||||
export_interval_millis=10000
|
||||
)
|
||||
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader]))
|
||||
|
||||
# 3. Logs
|
||||
logger_provider = LoggerProvider(resource=resource)
|
||||
_logs.set_logger_provider(logger_provider)
|
||||
log_exporter = OTLPLogExporter(endpoint=otlp_endpoint)
|
||||
logger_provider.add_log_record_processor(BatchLogRecordProcessor(log_exporter))
|
||||
|
||||
# Привязываем обработчик к корневому логгеру Python
|
||||
handler = LoggingHandler(level=logging.INFO, logger_provider=logger_provider)
|
||||
logging.getLogger().addHandler(handler)
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
|
||||
# 4. Инструментация библиотек
|
||||
FastAPIInstrumentor().instrument()
|
||||
AsyncPGInstrumentor().instrument()
|
||||
AioPikaInstrumentor().instrument()
|
||||
LoggingInstrumentor().instrument(set_logging_format=True)
|
||||
|
||||
logging.info(f"OpenTelemetry fully initialized for {service_name} at {otlp_endpoint}")
|
||||
+396
@@ -0,0 +1,396 @@
|
||||
import os
|
||||
import io
|
||||
import json
|
||||
import uuid
|
||||
import hashlib
|
||||
import asyncio
|
||||
import logging
|
||||
import tempfile
|
||||
import time
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
import aio_pika
|
||||
from minio import Minio
|
||||
from PIL import Image
|
||||
from sqlalchemy import select, update, func
|
||||
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
|
||||
from app.telemetry import setup_telemetry
|
||||
setup_telemetry()
|
||||
|
||||
from app.models import Attachment, AttachmentJob, AttachmentVariant
|
||||
from app.broker import (
|
||||
get_channel, declare_topology, publish_retry, choose_retry_tier, MAIN_QUEUE,
|
||||
)
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DATABASE_URL = os.getenv("DATABASE_URL")
|
||||
BROKER_URL = os.getenv("BROKER_URL")
|
||||
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333").replace("http://", "").replace("https://", "")
|
||||
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY", "admin")
|
||||
S3_SECRET_KEY = os.getenv("S3_SECRET_KEY", "admin")
|
||||
S3_BUCKET_NAME = os.getenv("S3_BUCKET_NAME", "messenger-attachments")
|
||||
|
||||
WORKER_ID = str(uuid.uuid4())[:8]
|
||||
JOB_LEASE_SECONDS = int(os.getenv("JOB_LEASE_SECONDS", "30"))
|
||||
JOB_MAX_ATTEMPTS = int(os.getenv("JOB_MAX_ATTEMPTS", "5"))
|
||||
WORKER_CONCURRENCY = int(os.getenv("WORKER_CONCURRENCY", "2"))
|
||||
JOB_TIMEOUT_SECONDS = int(os.getenv("JOB_TIMEOUT_SECONDS", "60"))
|
||||
MAX_PIXELS = 40_000_000
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
|
||||
engine = create_async_engine(DATABASE_URL, echo=False)
|
||||
AsyncSessionLocal = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
||||
s3_client = Minio(S3_ENDPOINT, access_key=S3_ACCESS_KEY, secret_key=S3_SECRET_KEY, secure=False)
|
||||
|
||||
# === OTel метрики worker ===
|
||||
from opentelemetry import metrics
|
||||
_meter = metrics.get_meter("messenger.worker")
|
||||
_jobs_completed = _meter.create_counter("messenger.worker.jobs.completed", description="Completed jobs")
|
||||
_jobs_failed = _meter.create_counter("messenger.worker.jobs.failed", description="Failed jobs")
|
||||
_job_duration = _meter.create_histogram("messenger.worker.job.duration", unit="s", description="Job processing duration")
|
||||
_active_jobs = _meter.create_up_down_counter("messenger.worker.jobs.active", description="In-flight jobs")
|
||||
|
||||
# Кэш очереди, обновляется sweeper'ом; observable-gauge читает из него
|
||||
_queue_state = {"active": 0, "oldest_age": 0.0}
|
||||
_meter.create_observable_gauge(
|
||||
"messenger.queue.active_jobs",
|
||||
callbacks=[lambda opts: [metrics.Observation(_queue_state["active"])]],
|
||||
description="Queued+processing jobs",
|
||||
)
|
||||
_meter.create_observable_gauge(
|
||||
"messenger.queue.oldest_job_age_seconds",
|
||||
callbacks=[lambda opts: [metrics.Observation(_queue_state["oldest_age"])]],
|
||||
unit="s",
|
||||
description="Age of the oldest unfinished job",
|
||||
)
|
||||
|
||||
|
||||
class PermanentError(Exception):
|
||||
def __init__(self, code, message=""):
|
||||
self.code = code
|
||||
super().__init__(f"{code}: {message}")
|
||||
|
||||
|
||||
class TemporaryError(Exception):
|
||||
def __init__(self, code, message=""):
|
||||
self.code = code
|
||||
super().__init__(f"{code}: {message}")
|
||||
|
||||
|
||||
def s3_put(key: str, data: bytes, content_type: str):
|
||||
with tempfile.NamedTemporaryFile(delete=False) as tmp:
|
||||
tmp.write(data)
|
||||
path = tmp.name
|
||||
try:
|
||||
s3_client.fput_object(S3_BUCKET_NAME, key, path, content_type=content_type)
|
||||
except Exception as e:
|
||||
raise TemporaryError("storage_unavailable", str(e))
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
|
||||
def build_variants(body: bytes, att_id: str, chat_id: str):
|
||||
if len(body) > MAX_BYTES:
|
||||
raise PermanentError("payload_too_large")
|
||||
try:
|
||||
img = Image.open(io.BytesIO(body))
|
||||
img.load()
|
||||
img = img.convert("RGB")
|
||||
except Exception as e:
|
||||
raise PermanentError("invalid_image", str(e))
|
||||
|
||||
w, h = img.size
|
||||
if w * h > MAX_PIXELS:
|
||||
raise PermanentError("image_too_large")
|
||||
|
||||
out = []
|
||||
|
||||
thumb = img.copy()
|
||||
thumb.thumbnail((256, 256), Image.Resampling.LANCZOS)
|
||||
buf = io.BytesIO(); thumb.save(buf, format="JPEG", quality=85)
|
||||
data = buf.getvalue(); key = f"variants/{chat_id}/{att_id}/thumbnail.jpg"
|
||||
s3_put(key, data, "image/jpeg")
|
||||
out.append(dict(variant_name="thumbnail", s3_key=key, content_type="image/jpeg",
|
||||
size_bytes=len(data), width=thumb.width, height=thumb.height,
|
||||
sha256=hashlib.sha256(data).hexdigest()))
|
||||
|
||||
side = min(w, h)
|
||||
left, top = (w - side) // 2, (h - side) // 2
|
||||
crop = img.crop((left, top, left + side, top + side)).resize((128, 128), Image.Resampling.LANCZOS)
|
||||
buf = io.BytesIO(); crop.save(buf, format="JPEG", quality=85)
|
||||
data = buf.getvalue(); key = f"variants/{chat_id}/{att_id}/crop.jpg"
|
||||
s3_put(key, data, "image/jpeg")
|
||||
out.append(dict(variant_name="crop", s3_key=key, content_type="image/jpeg",
|
||||
size_bytes=len(data), width=128, height=128,
|
||||
sha256=hashlib.sha256(data).hexdigest()))
|
||||
|
||||
opt = img.copy()
|
||||
opt.thumbnail((1280, 1280), Image.Resampling.LANCZOS)
|
||||
buf = io.BytesIO(); opt.save(buf, format="WEBP", quality=80, method=4)
|
||||
data = buf.getvalue(); key = f"variants/{chat_id}/{att_id}/optimized.webp"
|
||||
s3_put(key, data, "image/webp")
|
||||
out.append(dict(variant_name="optimized", s3_key=key, content_type="image/webp",
|
||||
size_bytes=len(data), width=opt.width, height=opt.height,
|
||||
sha256=hashlib.sha256(data).hexdigest()))
|
||||
|
||||
ext = "png" if body[:8].startswith(b"\x89PNG") else "jpeg"
|
||||
final_key = f"attachments/{chat_id}/{att_id}.{ext}"
|
||||
s3_put(final_key, body, "image/png" if ext == "png" else "image/jpeg")
|
||||
return final_key, out
|
||||
|
||||
|
||||
async def handle_job(job_id: str, att_id: str, attempt: int):
|
||||
t0 = time.monotonic()
|
||||
_active_jobs.add(1)
|
||||
try:
|
||||
async with AsyncSessionLocal() as db:
|
||||
job = (await db.execute(
|
||||
select(AttachmentJob).where(AttachmentJob.job_id == job_id).with_for_update()
|
||||
)).scalar_one_or_none()
|
||||
|
||||
if job is None:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
|
||||
return "noop", None
|
||||
if job.status in ("ready", "failed"):
|
||||
logger.info("Job %s already %s -> no-op ack", job_id, job.status)
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
|
||||
return "noop", None
|
||||
if attempt < job.attempt:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
|
||||
return "noop", None
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
if job.status == "processing" and job.lease_until and job.lease_until > now and job.lease_owner != WORKER_ID:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
|
||||
return "noop", None
|
||||
|
||||
job.status = "processing"
|
||||
job.attempt = attempt
|
||||
job.lease_until = now + timedelta(seconds=JOB_LEASE_SECONDS)
|
||||
job.lease_owner = WORKER_ID
|
||||
|
||||
att = (await db.execute(select(Attachment).where(Attachment.id == att_id))).scalar_one_or_none()
|
||||
if att is None:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
|
||||
return "noop", None
|
||||
att.status = "processing"
|
||||
await db.commit()
|
||||
|
||||
staging_key = att.staging_key
|
||||
if not staging_key:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "permanent"})
|
||||
return "permanent", "missing_staging"
|
||||
|
||||
try:
|
||||
resp = s3_client.get_object(S3_BUCKET_NAME, staging_key)
|
||||
try:
|
||||
body = resp.read()
|
||||
finally:
|
||||
resp.close(); resp.release_conn()
|
||||
except Exception:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "temporary"})
|
||||
return "temporary", "storage_unavailable"
|
||||
|
||||
if hashlib.sha256(body).hexdigest() != att.sha256:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "permanent"})
|
||||
return "permanent", "checksum_mismatch"
|
||||
|
||||
try:
|
||||
final_key, variants = await asyncio.wait_for(
|
||||
asyncio.to_thread(build_variants, body, str(att.id), str(att.chat_id)),
|
||||
timeout=JOB_TIMEOUT_SECONDS,
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "temporary"})
|
||||
return "temporary", "processing_timeout"
|
||||
except PermanentError as e:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "permanent"})
|
||||
return "permanent", e.code
|
||||
except TemporaryError as e:
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "temporary"})
|
||||
return "temporary", e.code
|
||||
except Exception:
|
||||
logger.exception("Unexpected processing failure")
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "temporary"})
|
||||
return "temporary", "processing_error"
|
||||
|
||||
res = await db.execute(
|
||||
update(AttachmentJob)
|
||||
.where(AttachmentJob.job_id == job_id,
|
||||
AttachmentJob.attempt == attempt,
|
||||
AttachmentJob.status == "processing")
|
||||
.values(status="ready", lease_until=None)
|
||||
)
|
||||
if res.rowcount == 0:
|
||||
logger.info("Job %s attempt %s lost ownership -> discard", job_id, attempt)
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "noop"})
|
||||
return "noop", None
|
||||
|
||||
for v in variants:
|
||||
await db.execute(
|
||||
pg_insert(AttachmentVariant)
|
||||
.values(attachment_id=att.id, **v)
|
||||
.on_conflict_do_nothing(index_elements=["attachment_id", "variant_name"])
|
||||
)
|
||||
|
||||
await db.execute(
|
||||
update(Attachment).where(Attachment.id == att.id)
|
||||
.values(status="ready", s3_key=final_key, staging_key=None, error_code=None)
|
||||
)
|
||||
await db.commit()
|
||||
|
||||
try:
|
||||
s3_client.remove_object(S3_BUCKET_NAME, staging_key)
|
||||
except Exception as e:
|
||||
logger.warning("Staging cleanup failed for %s: %s", staging_key, e)
|
||||
|
||||
logger.info("Job %s completed successfully (attempt %s)", job_id, attempt)
|
||||
_jobs_completed.add(1)
|
||||
_job_duration.record(time.monotonic() - t0, {"outcome": "ready"})
|
||||
return "done", None
|
||||
finally:
|
||||
_active_jobs.add(-1)
|
||||
|
||||
|
||||
async def mark_failed(job_id: str, code: str):
|
||||
async with AsyncSessionLocal() as db:
|
||||
job = (await db.execute(select(AttachmentJob).where(AttachmentJob.job_id == job_id))).scalar_one_or_none()
|
||||
if job is None or job.status == "ready":
|
||||
return
|
||||
await db.execute(update(AttachmentJob).where(AttachmentJob.job_id == job_id)
|
||||
.values(status="failed", lease_until=None))
|
||||
await db.execute(update(Attachment).where(Attachment.id == job.attachment_id)
|
||||
.values(status="failed", error_code=code))
|
||||
await db.commit()
|
||||
logger.warning("Job %s -> failed (%s)", job_id, code)
|
||||
_jobs_failed.add(1, {"error_code": code})
|
||||
|
||||
|
||||
async def schedule_retry(job_id: str, payload: dict) -> int:
|
||||
async with AsyncSessionLocal() as db:
|
||||
job = (await db.execute(
|
||||
select(AttachmentJob).where(AttachmentJob.job_id == job_id).with_for_update()
|
||||
)).scalar_one_or_none()
|
||||
if job is None or job.status == "ready":
|
||||
return -1
|
||||
if job.attempt >= JOB_MAX_ATTEMPTS:
|
||||
return -1
|
||||
next_attempt = job.attempt + 1
|
||||
tier = choose_retry_tier(job.attempt)
|
||||
job.status = "queued"
|
||||
job.attempt = next_attempt
|
||||
job.lease_until = None
|
||||
job.lease_owner = None
|
||||
await db.execute(update(Attachment).where(Attachment.id == job.attachment_id).values(status="queued"))
|
||||
await db.commit()
|
||||
|
||||
await publish_retry({**payload, "attempt": next_attempt}, tier)
|
||||
return tier
|
||||
|
||||
|
||||
async def process_message(message: aio_pika.IncomingMessage):
|
||||
try:
|
||||
data = json.loads(message.body)
|
||||
job_id, att_id, attempt = data["job_id"], data["attachment_id"], int(data["attempt"])
|
||||
uuid.UUID(job_id); uuid.UUID(att_id)
|
||||
except Exception:
|
||||
logger.error("Malformed message -> DLQ")
|
||||
await message.nack(requeue=False)
|
||||
return
|
||||
|
||||
try:
|
||||
outcome, code = await handle_job(job_id, att_id, attempt)
|
||||
except Exception:
|
||||
logger.exception("Handler crashed for job %s", job_id)
|
||||
outcome, code = "temporary", "internal_error"
|
||||
|
||||
try:
|
||||
if outcome in ("done", "noop"):
|
||||
await message.ack()
|
||||
return
|
||||
if outcome == "permanent":
|
||||
await mark_failed(job_id, code)
|
||||
await message.nack(requeue=False)
|
||||
return
|
||||
tier = await schedule_retry(job_id, data)
|
||||
if tier < 0:
|
||||
await mark_failed(job_id, "retry_exhausted")
|
||||
await message.nack(requeue=False)
|
||||
else:
|
||||
logger.info("Job %s temporary error (%s), retry in ~%ss", job_id, code, tier)
|
||||
await message.ack()
|
||||
except Exception:
|
||||
logger.exception("Failed to finalize delivery for job %s -> requeue", job_id)
|
||||
await message.nack(requeue=True)
|
||||
|
||||
|
||||
async def sweeper():
|
||||
global _queue_state
|
||||
while True:
|
||||
try:
|
||||
await asyncio.sleep(5)
|
||||
async with AsyncSessionLocal() as db:
|
||||
now = datetime.now(timezone.utc)
|
||||
# Обновляем кэш очереди для observable-gauge
|
||||
cnt = (await db.execute(
|
||||
select(func.count()).select_from(AttachmentJob)
|
||||
.where(AttachmentJob.status.in_(["queued", "processing"]))
|
||||
)).scalar() or 0
|
||||
oldest = (await db.execute(
|
||||
select(func.min(AttachmentJob.lease_until)).where(AttachmentJob.status == "queued")
|
||||
)).scalar()
|
||||
# возраст = время с создания самой старой незавершённой job
|
||||
oldest_created = (await db.execute(
|
||||
select(func.min(Attachment.created_at))
|
||||
.join(AttachmentJob, AttachmentJob.attachment_id == Attachment.id)
|
||||
.where(AttachmentJob.status.in_(["queued", "processing"]))
|
||||
)).scalar()
|
||||
age = (now - oldest_created).total_seconds() if oldest_created else 0.0
|
||||
_queue_state = {"active": cnt, "oldest_age": age}
|
||||
|
||||
rows = (await db.execute(
|
||||
select(AttachmentJob).where(
|
||||
AttachmentJob.status == "processing",
|
||||
AttachmentJob.lease_until.isnot(None),
|
||||
AttachmentJob.lease_until < now,
|
||||
).limit(20)
|
||||
)).scalars().all()
|
||||
for job in rows:
|
||||
if job.attempt >= JOB_MAX_ATTEMPTS:
|
||||
continue
|
||||
payload = {"job_id": str(job.job_id), "attachment_id": str(job.attachment_id),
|
||||
"attempt": job.attempt + 1}
|
||||
job.status = "queued"
|
||||
job.attempt += 1
|
||||
job.lease_until = None
|
||||
job.lease_owner = None
|
||||
await db.execute(update(Attachment).where(Attachment.id == job.attachment_id)
|
||||
.values(status="queued"))
|
||||
await db.commit()
|
||||
await publish_retry(payload, choose_retry_tier(1))
|
||||
logger.warning("Sweeper: lease expired for job %s -> retry attempt %s", job.job_id, job.attempt)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Sweeper error: %s", e)
|
||||
|
||||
|
||||
async def main():
|
||||
logger.info("Starting worker %s (concurrency/prefetch=%s)", WORKER_ID, WORKER_CONCURRENCY)
|
||||
channel = await get_channel()
|
||||
await declare_topology(channel)
|
||||
await channel.set_qos(prefetch_count=WORKER_CONCURRENCY)
|
||||
queue = await channel.get_queue(MAIN_QUEUE)
|
||||
await queue.consume(process_message)
|
||||
asyncio.create_task(sweeper())
|
||||
logger.info("Waiting for messages on %s ...", MAIN_QUEUE)
|
||||
await asyncio.Future()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
+267
@@ -0,0 +1,267 @@
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16.4-alpine
|
||||
environment:
|
||||
POSTGRES_USER: ${POSTGRES_USER}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
POSTGRES_DB: ${POSTGRES_DB}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '1.0', memory: 512M}
|
||||
|
||||
redis:
|
||||
image: valkey/valkey:8.0-alpine
|
||||
command: valkey-server --appendonly yes --save 60 1
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "valkey-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '0.5', memory: 128M}
|
||||
|
||||
s3:
|
||||
image: chrislusf/seaweedfs:3.80
|
||||
command:
|
||||
- server
|
||||
- -dir=/data
|
||||
- -s3
|
||||
- -s3.port=8333
|
||||
- -master.volumeSizeLimitMB=64
|
||||
- -volume.max=100
|
||||
volumes:
|
||||
- s3data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8333/healthz"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
start_period: 60s
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '1.0', memory: 1G}
|
||||
|
||||
s3-init:
|
||||
image: amazon/aws-cli:2.15.0
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- AWS_ACCESS_KEY_ID=admin
|
||||
- AWS_SECRET_ACCESS_KEY=admin
|
||||
- AWS_REGION=${S3_REGION}
|
||||
entrypoint: >
|
||||
/bin/sh -c "
|
||||
aws --endpoint-url http://s3:8333 s3 mb s3://${S3_BUCKET_NAME} 2>/dev/null || echo 'Bucket already exists'
|
||||
aws --endpoint-url http://s3:8333 s3api put-bucket-acl --bucket ${S3_BUCKET_NAME} --acl private
|
||||
echo 'Bucket ${S3_BUCKET_NAME} is ready and private'
|
||||
"
|
||||
restart: "no"
|
||||
|
||||
rabbitmq:
|
||||
image: rabbitmq:3.13-management-alpine
|
||||
hostname: rabbitmq
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: messenger
|
||||
RABBITMQ_DEFAULT_PASS: mq_secret_password
|
||||
RABBITMQ_DEFAULT_VHOST: /
|
||||
volumes:
|
||||
- mqdata:/var/lib/rabbitmq
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "check_port_connectivity"]
|
||||
interval: 5s
|
||||
timeout: 10s
|
||||
retries: 30
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '1.0', memory: 512M}
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:0.163.2
|
||||
command: ["--config=/etc/otelcol-contrib/collector.yaml"]
|
||||
volumes:
|
||||
- ./infra/otel/collector.yaml:/etc/otelcol-contrib/collector.yaml:ro
|
||||
depends_on:
|
||||
- tempo
|
||||
- loki
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '0.5', memory: 512M}
|
||||
restart: unless-stopped
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:v2.55.1
|
||||
ports: ["127.0.0.1:9090:9090"]
|
||||
command:
|
||||
- --config.file=/etc/prometheus/prometheus.yml
|
||||
- --storage.tsdb.path=/prometheus
|
||||
- --storage.tsdb.retention.time=15d
|
||||
volumes:
|
||||
- ./infra/otel/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- ./infra/otel/alerts.yml:/etc/prometheus/alerts.yml:ro
|
||||
- prometheus_data:/prometheus
|
||||
depends_on:
|
||||
- otel-collector
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '0.5', memory: 512M}
|
||||
restart: unless-stopped
|
||||
|
||||
loki:
|
||||
image: grafana/loki:3.3.2
|
||||
command: ["-config.file=/etc/loki/loki.yml"]
|
||||
volumes:
|
||||
- ./infra/otel/loki.yml:/etc/loki/loki.yml:ro
|
||||
- loki_data:/loki
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '0.5', memory: 512M}
|
||||
restart: unless-stopped
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.7.1
|
||||
command: ["-config.file=/etc/tempo/tempo.yml"]
|
||||
volumes:
|
||||
- ./infra/otel/tempo.yml:/etc/tempo/tempo.yml:ro
|
||||
- tempo_data:/var/tempo
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '0.5', memory: 512M}
|
||||
restart: unless-stopped
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:11.4.0
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: admin
|
||||
GF_SECURITY_ADMIN_PASSWORD: admin
|
||||
GF_USERS_ALLOW_SIGN_UP: "false"
|
||||
GF_ANALYTICS_REPORTING_ENABLED: "false"
|
||||
volumes:
|
||||
- grafana_data:/var/lib/grafana
|
||||
- ./infra/otel/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./infra/otel/grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
depends_on:
|
||||
- prometheus
|
||||
- loki
|
||||
- tempo
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '0.5', memory: 512M}
|
||||
restart: unless-stopped
|
||||
|
||||
api:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
environment:
|
||||
- DATABASE_URL=${DATABASE_URL}
|
||||
- REDIS_URL=${REDIS_URL}
|
||||
- SECRET_SESSION_KEY=${SECRET_SESSION_KEY}
|
||||
- SESSION_TTL_SECONDS=${SESSION_TTL_SECONDS}
|
||||
- S3_ENDPOINT=${S3_ENDPOINT}
|
||||
- S3_ACCESS_KEY=${S3_API_ACCESS_KEY:-admin}
|
||||
- S3_SECRET_KEY=${S3_API_SECRET_KEY:-admin}
|
||||
- S3_BUCKET_NAME=${S3_BUCKET_NAME}
|
||||
- S3_REGION=${S3_REGION}
|
||||
- BROKER_URL=${BROKER_URL}
|
||||
- MAX_ACTIVE_JOBS=${MAX_ACTIVE_JOBS}
|
||||
- OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}
|
||||
- OTEL_SERVICE_NAME=messenger-api
|
||||
- OTEL_RESOURCE_ATTRIBUTES=service.version=1.0.0,deployment.environment=lab
|
||||
volumes:
|
||||
- ./alembic/versions:/app/alembic/versions
|
||||
- ./app:/app/app
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp:size=15M
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
stop_grace_period: 30s
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8080/health/live"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '0.5', memory: 256M}
|
||||
|
||||
worker:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
command: ["python", "-m", "app.worker"]
|
||||
environment:
|
||||
- DATABASE_URL=${DATABASE_URL}
|
||||
- BROKER_URL=${BROKER_URL}
|
||||
- S3_ENDPOINT=${S3_ENDPOINT}
|
||||
- S3_ACCESS_KEY=${S3_API_ACCESS_KEY:-admin}
|
||||
- S3_SECRET_KEY=${S3_API_SECRET_KEY:-admin}
|
||||
- S3_BUCKET_NAME=${S3_BUCKET_NAME}
|
||||
- WORKER_CONCURRENCY=${WORKER_CONCURRENCY}
|
||||
- JOB_MAX_ATTEMPTS=${JOB_MAX_ATTEMPTS}
|
||||
- JOB_LEASE_SECONDS=${JOB_LEASE_SECONDS}
|
||||
- OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}
|
||||
- OTEL_SERVICE_NAME=messenger-worker
|
||||
- OTEL_RESOURCE_ATTRIBUTES=service.version=1.0.0,deployment.environment=lab
|
||||
volumes:
|
||||
- ./app:/app/app
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp:size=100M
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '2.0', memory: 768M}
|
||||
restart: unless-stopped
|
||||
|
||||
nginx:
|
||||
image: nginx:1.25-alpine
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
s3data:
|
||||
mqdata:
|
||||
prometheus_data:
|
||||
loki_data:
|
||||
tempo_data:
|
||||
grafana_data:
|
||||
@@ -0,0 +1,25 @@
|
||||
groups:
|
||||
- name: messenger_slo
|
||||
rules:
|
||||
- alert: HighErrorRate
|
||||
expr: |
|
||||
(
|
||||
sum(rate(http_server_duration_milliseconds_count{http_status_code=~"5.."}[5m])) or vector(0)
|
||||
)
|
||||
/
|
||||
sum(rate(http_server_duration_milliseconds_count[5m]))
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "API error rate > 5% (SLO 99%)"
|
||||
description: "Доля 5xx за 5-минутное окно превысила 5%. SLI из histogram-метрик API."
|
||||
|
||||
- alert: QueueLagGrowing
|
||||
expr: messenger_queue_oldest_job_age_seconds > 60
|
||||
for: 2m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Очередь задач растёт (возраст старейшей job > 60s)"
|
||||
description: "Worker не успевает дренировать очередь."
|
||||
@@ -0,0 +1,39 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 384
|
||||
spike_limit_mib: 96
|
||||
batch:
|
||||
send_batch_size: 512
|
||||
timeout: 5s
|
||||
|
||||
exporters:
|
||||
otlphttp/tempo:
|
||||
endpoint: "http://tempo:4318"
|
||||
otlphttp/loki:
|
||||
endpoint: "http://loki:3100/otlp"
|
||||
prometheus:
|
||||
endpoint: "0.0.0.0:8889"
|
||||
resource_to_telemetry_conversion:
|
||||
enabled: true
|
||||
|
||||
service:
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/tempo]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [prometheus]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/loki]
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: 'messenger'
|
||||
orgId: 1
|
||||
folder: 'Messenger'
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 30
|
||||
allowUiUpdates: true
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"uid": "messenger-lab6",
|
||||
"title": "Messenger — Lab6 Observability",
|
||||
"tags": ["messenger", "lab6"],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"refresh": "10s",
|
||||
"time": { "from": "now-15m", "to": "now" },
|
||||
"panels": [
|
||||
{
|
||||
"id": 1, "type": "timeseries", "title": "RPS (по endpoint'ам)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "sum by (http_target) (rate(http_server_duration_milliseconds_count[5m]))", "legendFormat": "{{http_target}}" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "timeseries",
|
||||
"title": "Доля ошибок 5xx (SLI)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(sum(rate(http_server_duration_milliseconds_count{http_status_code=~\"5..\"}[5m])) or vector(0)) / sum(rate(http_server_duration_milliseconds_count[5m]))",
|
||||
"legendFormat": "error ratio"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3, "type": "timeseries", "title": "API latency p50/p95/p99 (из histogram, ms)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "ms" }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "histogram_quantile(0.50, sum by (le) (rate(http_server_duration_milliseconds_bucket[5m])))", "legendFormat": "p50" },
|
||||
{ "refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_duration_milliseconds_bucket[5m])))", "legendFormat": "p95" },
|
||||
{ "refId": "C", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_duration_milliseconds_bucket[5m])))", "legendFormat": "p99" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4, "type": "timeseries", "title": "Worker jobs: completed / failed",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "sum(rate(messenger_worker_jobs_completed_total[5m]))", "legendFormat": "completed" },
|
||||
{ "refId": "B", "expr": "sum(rate(messenger_worker_jobs_failed_total[5m]))", "legendFormat": "failed" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5, "type": "timeseries", "title": "Worker job duration p50/p95 (s)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "histogram_quantile(0.50, sum by (le) (rate(messenger_worker_job_duration_seconds_bucket[5m])))", "legendFormat": "p50" },
|
||||
{ "refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_worker_job_duration_seconds_bucket[5m])))", "legendFormat": "p95" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6, "type": "stat", "title": "Worker saturation (active jobs)",
|
||||
"gridPos": { "h": 8, "w": 6, "x": 12, "y": 16 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [ { "refId": "A", "expr": "sum(messenger_worker_jobs_active)", "legendFormat": "active" } ]
|
||||
},
|
||||
{
|
||||
"id": 7, "type": "stat", "title": "Queue depth (active jobs)",
|
||||
"gridPos": { "h": 8, "w": 6, "x": 18, "y": 16 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [ { "refId": "A", "expr": "sum(messenger_queue_active_jobs)", "legendFormat": "queued+processing" } ]
|
||||
},
|
||||
{
|
||||
"id": 8, "type": "timeseries", "title": "Возраст старейшей job (queue lag)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
|
||||
"targets": [ { "refId": "A", "expr": "sum(messenger_queue_oldest_job_age_seconds)", "legendFormat": "oldest age" } ]
|
||||
},
|
||||
{
|
||||
"id": 9, "type": "timeseries", "title": "DB pool (size / checked out / overflow)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "sum(messenger_db_pool_size)", "legendFormat": "size" },
|
||||
{ "refId": "B", "expr": "sum(messenger_db_pool_checked_out)", "legendFormat": "checked out" },
|
||||
{ "refId": "C", "expr": "sum(messenger_db_pool_overflow)", "legendFormat": "overflow" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: 'messenger'
|
||||
orgId: 1
|
||||
folder: 'Messenger'
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 30
|
||||
allowUiUpdates: true
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,88 @@
|
||||
{
|
||||
"uid": "messenger-lab6",
|
||||
"title": "Messenger — Lab6 Observability",
|
||||
"tags": ["messenger", "lab6"],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"refresh": "10s",
|
||||
"time": { "from": "now-15m", "to": "now" },
|
||||
"panels": [
|
||||
{
|
||||
"id": 1, "type": "timeseries", "title": "RPS (по маршрутам)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "sum by (http_route) (rate(http_server_duration_seconds_count[5m]))", "legendFormat": "{{http_route}}" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2, "type": "timeseries", "title": "Доля ошибок 5xx (SLI)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "percentunit", "max": 1 }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "sum(rate(http_server_duration_seconds_count{http_response_status_code=~\"5..\"}[5m])) / sum(rate(http_server_duration_seconds_count[5m]))", "legendFormat": "error ratio" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3, "type": "timeseries", "title": "API latency p50/p95/p99 (из histogram)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "histogram_quantile(0.50, sum by (le) (rate(http_server_duration_seconds_bucket[5m])))", "legendFormat": "p50" },
|
||||
{ "refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_duration_seconds_bucket[5m])))", "legendFormat": "p95" },
|
||||
{ "refId": "C", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_duration_seconds_bucket[5m])))", "legendFormat": "p99" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4, "type": "timeseries", "title": "Worker jobs: completed / failed",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "sum(rate(messenger_worker_jobs_completed_total[5m]))", "legendFormat": "completed" },
|
||||
{ "refId": "B", "expr": "sum(rate(messenger_worker_jobs_failed_total[5m]))", "legendFormat": "failed" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5, "type": "timeseries", "title": "Worker job duration p50/p95",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "histogram_quantile(0.50, sum by (le) (rate(messenger_worker_job_duration_seconds_bucket[5m])))", "legendFormat": "p50" },
|
||||
{ "refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_worker_job_duration_seconds_bucket[5m])))", "legendFormat": "p95" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6, "type": "stat", "title": "Worker saturation (active jobs)",
|
||||
"gridPos": { "h": 8, "w": 6, "x": 12, "y": 16 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [ { "refId": "A", "expr": "sum(messenger_worker_jobs_active)", "legendFormat": "active" } ]
|
||||
},
|
||||
{
|
||||
"id": 7, "type": "stat", "title": "Queue depth (active jobs)",
|
||||
"gridPos": { "h": 8, "w": 6, "x": 18, "y": 16 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [ { "refId": "A", "expr": "sum(messenger_queue_active_jobs)", "legendFormat": "queued+processing" } ]
|
||||
},
|
||||
{
|
||||
"id": 8, "type": "timeseries", "title": "Возраст старейшей job (queue lag)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
|
||||
"targets": [ { "refId": "A", "expr": "sum(messenger_queue_oldest_job_age_seconds)", "legendFormat": "oldest age" } ]
|
||||
},
|
||||
{
|
||||
"id": 9, "type": "timeseries", "title": "DB pool (size / checked out / overflow)",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{ "refId": "A", "expr": "sum(messenger_db_pool_size)", "legendFormat": "size" },
|
||||
{ "refId": "B", "expr": "sum(messenger_db_pool_checked_out)", "legendFormat": "checked out" },
|
||||
{ "refId": "C", "expr": "sum(messenger_db_pool_overflow)", "legendFormat": "overflow" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
uid: prometheus
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
editable: false
|
||||
|
||||
- name: Loki
|
||||
type: loki
|
||||
uid: loki
|
||||
url: http://loki:3100
|
||||
editable: false
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: TraceID
|
||||
matcherRegex: "trace_id=(\\w+)"
|
||||
url: "$${__value.raw}"
|
||||
datasourceUid: tempo
|
||||
|
||||
- name: Tempo
|
||||
type: tempo
|
||||
uid: tempo
|
||||
url: http://tempo:3200
|
||||
editable: false
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: -5m
|
||||
spanEndTimeShift: 5m
|
||||
tags: [{ key: 'service.name', value: 'job' }]
|
||||
filterByTraceID: true
|
||||
filterBySpanID: true
|
||||
@@ -0,0 +1,30 @@
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
grpc_listen_port: 9096
|
||||
|
||||
common:
|
||||
path_prefix: /loki
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2020-10-24
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
|
||||
limits_config:
|
||||
allow_structured_metadata: true
|
||||
volume_enabled: true
|
||||
@@ -0,0 +1,11 @@
|
||||
global:
|
||||
scrape_interval: 10s
|
||||
evaluation_interval: 10s
|
||||
|
||||
rule_files:
|
||||
- /etc/prometheus/alerts.yml
|
||||
|
||||
scrape_configs:
|
||||
- job_name: 'otel-collector'
|
||||
static_configs:
|
||||
- targets: ['otel-collector:8889']
|
||||
@@ -0,0 +1,23 @@
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: "0.0.0.0:4317"
|
||||
http:
|
||||
endpoint: "0.0.0.0:4318"
|
||||
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
local:
|
||||
path: /var/tempo/traces
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 24h
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
upstream api_backend {
|
||||
# Nginx будет автоматически резолвить имена через Docker DNS
|
||||
server api:8080 max_fails=1 fail_timeout=5s;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
|
||||
# Проксирование ВСЕХ запросов на API реплики
|
||||
location / {
|
||||
proxy_pass http://api_backend;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Request-Id $http_x_request_id;
|
||||
|
||||
# Таймауты для graceful shutdown
|
||||
proxy_connect_timeout 5s;
|
||||
proxy_read_timeout 30s;
|
||||
proxy_send_timeout 30s;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
fastapi==0.115.0
|
||||
uvicorn[standard]==0.30.6
|
||||
sqlalchemy[asyncio]==2.0.35
|
||||
asyncpg==0.29.0
|
||||
pydantic-settings==2.5.2
|
||||
redis==5.1.0
|
||||
bcrypt==4.2.0
|
||||
alembic==1.13.2
|
||||
python-multipart==0.0.9
|
||||
minio==7.2.0
|
||||
Pillow==10.4.0
|
||||
python-magic==0.4.27
|
||||
aio-pika==9.4.3
|
||||
uuid-utils==0.10.0
|
||||
|
||||
# === OpenTelemetry (Lab 6) ===
|
||||
opentelemetry-api==1.29.0
|
||||
opentelemetry-sdk==1.29.0
|
||||
opentelemetry-exporter-otlp-proto-http==1.29.0
|
||||
opentelemetry-instrumentation-fastapi==0.50b0
|
||||
opentelemetry-instrumentation-asyncpg==0.50b0
|
||||
opentelemetry-instrumentation-aio-pika==0.50b0
|
||||
opentelemetry-instrumentation-botocore==0.50b0
|
||||
opentelemetry-instrumentation-logging==0.50b0
|
||||
opentelemetry-semantic-conventions==0.50b0
|
||||
Reference in New Issue
Block a user