ЛР6 — БПИ2302 — Перевалова — на 3 #30

Closed
perevalova wants to merge 1 commits from perevalova/mtusi2026lab6:lab6-observability into main
58 changed files with 6886 additions and 20 deletions
+23
View File
@@ -0,0 +1,23 @@
FROM python:3.11.9-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1
WORKDIR /app
RUN groupadd --system app && useradd --system --gid app --home-dir /app app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app ./app
COPY migrations ./migrations
COPY scripts ./scripts
COPY tests ./tests
COPY contracts ./contracts
COPY lab.json .
COPY alembic.ini .
RUN chown -R app:app /app
USER app
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=3s --start-period=20s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-keep-alive", "5", "--timeout-graceful-shutdown", "20"]
+184 -20
View File
@@ -1,37 +1,201 @@
# Отчёт — лабораторная 6
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- **Автор, группа:** Перевалова Виктория, БПИ2302
- **Целевая оценка:** 3
- **Выбранный трек 5:** не заявляется
- **Commit/PR:** будет указан после публикации
- **Использованные источники, библиотеки и помощь ИИ:** OpenTelemetry SDK, OpenTelemetry Collector, Prometheus, Loki, Tempo, Grafana, FastAPI Instrumentation, SQLAlchemy Instrumentation. При разработке и отладке использовалась помощь ChatGPT. Документация: OpenTelemetry, Prometheus, Grafana.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
### Требования
- Python 3.11;
- Docker;
- Docker Compose;
- Git.
### Подготовка
Клонировать репозиторий, перейти в каталог проекта и создать виртуальное окружение:
```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
```
Установить зависимости:
```powershell
python -m pip install -r requirements.txt
```
Создать локальный `.env` на основе `.env.example` и заполнить необходимые настройки. Секреты должны храниться локально и не добавляться в Git.
### Запуск инфраструктуры
Проверить конфигурацию:
```powershell
docker compose config --quiet
```
Собрать и запустить стенд:
```powershell
docker compose up -d --build
```
Проверить состояние контейнеров:
```powershell
docker compose ps
```
### Проверки
Проверка структуры и синтаксиса:
```powershell
python scripts/check_contract.py
```
Публичный smoke-тест:
```powershell
python tests/smoke.py --base-url "http://localhost:18082"
```
Нагрузочный тест:
```powershell
python tests/load.py --base-url "http://localhost:18082"
```
### Адреса сервисов
- API через Nginx: `http://localhost:18082`
- Grafana: `http://localhost:13000`
- Prometheus: `http://localhost:19090`
### Остановка
```powershell
docker compose down
```
Для полного удаления учебных данных вместе с постоянными томами:
```powershell
docker compose down -v
```
Последняя команда удаляет сохранённые данные и применяется только при намеренном сбросе стенда.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
В лабораторной реализована система наблюдаемости приложения обработки сообщений и вложений.
### Основные компоненты
| Компонент | Назначение |
|---|---|
| API | Обработка HTTP-запросов |
| PostgreSQL | Хранение данных приложения |
| RabbitMQ | Передача фоновых заданий |
| SeaweedFS | Хранение файлов и вложений |
| Worker | Фоновая обработка изображений |
| Outbox Publisher | Публикация фоновых заданий |
| Nginx | Точка входа к API |
| OpenTelemetry Collector | Приём и маршрутизация телеметрии |
| Prometheus | Хранение метрик |
| Loki | Хранение логов |
| Tempo | Хранение распределённых трасс |
| Grafana | Визуализация и анализ телеметрии |
### Архитектура наблюдаемости
API и Worker отправляют телеметрию по протоколу OTLP в OpenTelemetry Collector.
Collector передаёт данные в соответствующие хранилища:
- метрики — Prometheus;
- логи — Loki;
- трассы — Tempo.
Grafana использует эти хранилища как источники данных и предоставляет единый интерфейс для анализа состояния приложения.
### Конфигурационные файлы
- `observability/otel-collector-config.yaml` — конфигурация OpenTelemetry Collector.
- `observability/prometheus.yml` — конфигурация Prometheus.
- `observability/loki-config.yaml` — конфигурация Loki.
- `observability/tempo-config.yaml` — конфигурация Tempo.
- `grafana/provisioning/datasources/datasources.yaml` — автоматическая настройка источников данных Grafana.
- `grafana/provisioning/dashboards/dashboards.yaml` — автоматическая загрузка дашбордов.
- `grafana/dashboards/messenger-overview.json` — импортируемый дашборд приложения.
### Инструментация
Для API настроена OpenTelemetry-инструментация с использованием SDK и инструментов FastAPI и SQLAlchemy.
Для Worker реализована отдельная инициализация OpenTelemetry. Добавлены метрики обработки фоновых заданий, включая количество успешных и неуспешных операций и длительность обработки.
Для логов предусмотрена передача идентификаторов trace и span, позволяющих связывать записи логов с трассами.
В телеметрию не должны передаваться пароли, токены, содержимое сообщений и изображения. Метки метрик не должны содержать высококардинальные идентификаторы пользователей, чатов и сообщений.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Критерий | Проверка или сценарий | Результат |
|---|---|---|
| Проверка структуры проекта | `python scripts/check_contract.py` | PASS: offline skeleton structure, local refs and Python syntax |
| Публичный API smoke | `python tests/smoke.py --base-url "http://localhost:18082"` | PASS: lab6 public API scenario |
| Конфигурация Compose | `docker compose config --quiet` | PASS |
| Инфраструктура | `docker compose ps` | Сервисы API, Worker, PostgreSQL, RabbitMQ, SeaweedFS, Collector, Prometheus, Loki, Tempo и Grafana запущены |
| Импортируемый дашборд | `grafana/dashboards/messenger-overview.json` | Файл дашборда присутствует в проекте |
| Нагрузочный тест | `python tests/load.py --base-url "http://localhost:18082"` | 60 успешных запросов, 0 ошибок |
| Отказ Worker | Остановка Worker во время поступления задания и последующий запуск | Задание обработано после восстановления Worker, статус `ready` |
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
### Нагрузочное тестирование
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
Нагрузка создавалась с помощью `tests/load.py`.
## Ограничения и следующие шаги
Результат:
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
| Показатель | Значение |
|---|---:|
| Запрошенная длительность | 30 секунд |
| Фактическая длительность | 32,25 секунды |
| Параллельность | 4 |
| Успешные запросы | 60 |
| Неуспешные запросы | 0 |
| Доля ошибок | 0 |
| Успешный RPS | 1,86 |
| p50 | 2078 мс |
| p95 | 2125 мс |
| p99 | 2141 мс |
Нагрузка выполнялась по модели closed-loop: новые запросы отправлялись с учётом завершения предыдущих. Поэтому показатели характеризуют именно проведённый тестовый сценарий и не являются оценкой предельной пропускной способности системы.
### Отказ Worker
**Гипотеза:** если Worker временно недоступен, API продолжит принимать задания, а сообщение останется в очереди до восстановления обработчика.
Ход эксперимента:
1. Worker был остановлен командой `docker compose stop worker`.
2. Через API было отправлено изображение.
3. Задание осталось в очереди RabbitMQ.
4. Worker был запущен повторно командой `docker compose start worker`.
5. После восстановления обработчик получил задание и выполнил обработку изображения.
Результат: вложение перешло в состояние `ready`, ошибка обработки отсутствовала. После восстановления Worker очередь вернулась к нулевому количеству ожидающих сообщений.
### Отказ хранилищ телеметрии
Выполнялась проверка недоступности компонентов хранения телеметрии: Prometheus, Loki и Tempo.
Контейнеры останавливались и затем запускались повторно. После восстановления проверено состояние сервисов через `docker compose ps`.
+117
View File
@@ -0,0 +1,117 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts
# Use forward slashes (/) also on windows to provide an os agnostic path
script_location = migrations
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
# for all available tokens
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library.
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =
# max length of characters to apply to the "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to migrations/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "version_path_separator" below.
# version_locations = %(here)s/bar:%(here)s/bat:migrations/versions
# version path separator; As mentioned above, this is the character used to split
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
# Valid values for version_path_separator are:
#
# version_path_separator = :
# version_path_separator = ;
# version_path_separator = space
# version_path_separator = newline
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
sqlalchemy.url = driver://user:pass@localhost/dbname
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
# hooks = ruff
# ruff.type = exec
# ruff.executable = %(here)s/.venv/bin/ruff
# ruff.options = --fix REVISION_SCRIPT_FILENAME
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARNING
handlers = console
qualname =
[logger_sqlalchemy]
level = WARNING
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
View File
View File
+645
View File
@@ -0,0 +1,645 @@
import hashlib
import tempfile
import uuid
from datetime import datetime, timezone, timedelta
from urllib.parse import quote
from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, Response
from fastapi.responses import StreamingResponse
from sqlalchemy import select, func, text
from sqlalchemy.orm import Session
from botocore.exceptions import BotoCoreError, ClientError
from opentelemetry.propagate import inject
from app.api.deps import get_db, get_current_user
from app.core.config import settings
from app.models import CreateAttachmentUpload
from app.db.models import (
Attachment,
AttachmentUpload,
AttachmentVariant,
ProcessingJob,
OutboxEvent,
User,
)
from app.services.object_storage import client, create_presigned_upload_url
from app.services.image_validation import MAX_BYTES, validate_image_file
from app.repositories.chat_repository import ChatRepository
router = APIRouter(prefix="/api/v1", tags=["Attachments"])
def iso_utc(value):
return value.astimezone(timezone.utc).isoformat(timespec="microseconds").replace("+00:00", "Z")
def attachment_json(item, db):
variants = db.scalars(
select(AttachmentVariant)
.where(AttachmentVariant.attachment_id == item.id)
.order_by(AttachmentVariant.variant)
).all()
return {
"id": str(item.id),
"chat_id": str(item.chat_id),
"owner_id": str(item.owner_id),
"filename": item.filename,
"content_type": item.content_type,
"size_bytes": item.size_bytes,
"sha256": item.sha256,
"status": item.status,
"variants": [
{
"name": variant.variant,
"content_type": variant.content_type,
"size_bytes": variant.size_bytes,
"width": variant.width,
"height": variant.height,
"sha256": variant.sha256,
}
for variant in variants
],
"error_code": item.error_code,
"created_at": iso_utc(item.created_at),
"updated_at": iso_utc(item.updated_at),
}
def member_attachment(db, attachment_id, user_id):
item = db.get(Attachment, attachment_id)
if item is None or not ChatRepository(db).is_member(item.chat_id, user_id):
raise HTTPException(404, "Attachment not found")
return item
async def read_limited(request: Request):
declared = request.headers.get("content-length")
if declared and declared.isdigit() and int(declared) > MAX_BYTES:
raise HTTPException(413, "Image exceeds 10 MiB")
size = 0
# Keep at most 1 MiB in RAM; larger bodies spill to a temporary file.
spool = tempfile.SpooledTemporaryFile(max_size=1024 * 1024, mode="w+b")
try:
async for chunk in request.stream():
size += len(chunk)
if size > MAX_BYTES:
raise HTTPException(413, "Image exceeds 10 MiB")
spool.write(chunk)
spool.seek(0)
return spool, size
except Exception:
spool.close()
raise
def sha256_file(fileobj):
digest = hashlib.sha256()
fileobj.seek(0)
while True:
chunk = fileobj.read(1024 * 1024)
if not chunk:
break
digest.update(chunk)
fileobj.seek(0)
return digest.hexdigest()
@router.post("/chats/{chat_id}/attachments/presigned", status_code=201)
def create_attachment_upload(
chat_id: str,
payload: CreateAttachmentUpload,
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
chat_uuid = uuid.UUID(chat_id)
except ValueError:
raise HTTPException(400, "Invalid chat UUID") from None
if not ChatRepository(db).is_member(chat_uuid, user.id):
raise HTTPException(404, "Chat not found")
if payload.size_bytes > settings.max_upload_bytes:
raise HTTPException(413, "Image exceeds configured upload limit")
upload_id = uuid.uuid4()
object_key = f"uploads/{chat_uuid}/{upload_id}"
expires_at = datetime.now(timezone.utc) + timedelta(
seconds=settings.presigned_upload_ttl_seconds
)
try:
upload_url = create_presigned_upload_url(
object_key=object_key,
content_type=payload.content_type,
)
except (BotoCoreError, ClientError):
raise HTTPException(503, "Object storage unavailable") from None
upload = AttachmentUpload(
id=upload_id,
chat_id=chat_uuid,
owner_id=user.id,
filename=payload.filename,
content_type=payload.content_type,
expected_size_bytes=payload.size_bytes,
expected_sha256=payload.sha256.lower(),
object_key=object_key,
status="pending",
expires_at=expires_at,
)
try:
db.add(upload)
db.commit()
except Exception:
db.rollback()
raise HTTPException(503, "Could not save upload metadata") from None
return {
"upload_id": str(upload_id),
"upload_url": upload_url,
"method": "PUT",
"headers": {
"Content-Type": payload.content_type,
},
"expires_in": settings.presigned_upload_ttl_seconds,
"expires_at": iso_utc(expires_at),
}
@router.post("/attachments/uploads/{upload_id}/complete", status_code=200)
def complete_attachment_upload(
upload_id: str,
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
parsed_id = uuid.UUID(upload_id)
except ValueError:
raise HTTPException(400, "Invalid upload UUID") from None
# Lock the upload row to prevent concurrent completion.
upload = db.scalar(
select(AttachmentUpload)
.where(AttachmentUpload.id == parsed_id)
.with_for_update()
)
if upload is None:
raise HTTPException(404, "Upload not found")
if (
upload.owner_id != user.id
or not ChatRepository(db).is_member(upload.chat_id, user.id)
):
raise HTTPException(404, "Upload not found")
# Idempotent completion: return the existing attachment.
if upload.status == "completed":
item = db.scalar(
select(Attachment).where(
Attachment.object_key == upload.object_key
)
)
if item is None:
raise HTTPException(409, "Completed attachment not found")
return attachment_json(item, db)
if upload.status != "pending":
raise HTTPException(409, "Upload is not pending")
now = datetime.now(timezone.utc)
if upload.expires_at <= now:
upload.status = "expired"
db.commit()
try:
client().delete_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
except Exception:
pass
raise HTTPException(409, "Upload has expired")
s3 = client()
# Read the object from S3.
try:
result = s3.get_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
except ClientError as exc:
code = exc.response.get("Error", {}).get("Code", "")
if code in {"NoSuchKey", "404", "NotFound"}:
raise HTTPException(409, "Uploaded object not found") from None
raise HTTPException(503, "Object storage unavailable") from None
except BotoCoreError:
raise HTTPException(503, "Object storage unavailable") from None
body = result["Body"]
spool = tempfile.SpooledTemporaryFile(
max_size=1024 * 1024,
mode="w+b",
)
def reject_upload(status_code: int, detail: str):
upload.status = "failed"
db.commit()
try:
s3.delete_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
except Exception:
pass
raise HTTPException(status_code=status_code, detail=detail)
try:
actual_size = result.get("ContentLength")
if actual_size != upload.expected_size_bytes:
reject_upload(409, "Uploaded file size mismatch")
if actual_size > settings.max_upload_bytes:
reject_upload(413, "Image exceeds configured upload limit")
total = 0
digest = hashlib.sha256()
try:
while True:
chunk = body.read(64 * 1024)
if not chunk:
break
total += len(chunk)
if total > settings.max_upload_bytes:
reject_upload(413, "Image exceeds configured upload limit")
if total > upload.expected_size_bytes:
reject_upload(409, "Uploaded file size mismatch")
digest.update(chunk)
spool.write(chunk)
finally:
body.close()
if total != upload.expected_size_bytes:
reject_upload(409, "Uploaded file size mismatch")
if digest.hexdigest() != upload.expected_sha256:
reject_upload(409, "Uploaded file checksum mismatch")
# Validate the actual image content, not only S3 metadata.
try:
actual_type, _, _ = validate_image_file(
spool,
upload.content_type,
)
except HTTPException as exc:
reject_upload(exc.status_code, str(exc.detail))
attachment = Attachment(
id=uuid.uuid4(),
chat_id=upload.chat_id,
owner_id=upload.owner_id,
filename=upload.filename,
content_type=actual_type,
size_bytes=total,
sha256=digest.hexdigest(),
object_key=upload.object_key,
status="ready",
error_code=None,
)
upload.status = "completed"
upload.completed_at = now
db.add(attachment)
db.commit()
db.refresh(attachment)
return attachment_json(attachment, db)
except HTTPException:
raise
except (BotoCoreError, ClientError):
db.rollback()
raise HTTPException(503, "Object storage unavailable") from None
except Exception:
db.rollback()
raise
finally:
spool.close()
@router.post("/chats/{chat_id}/attachments", status_code=202)
async def upload_attachment(
chat_id: str,
request: Request,
filename: str = Query(..., min_length=1, max_length=255),
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
chat_uuid = uuid.UUID(chat_id)
except ValueError:
raise HTTPException(400, "Invalid chat UUID") from None
if not ChatRepository(db).is_member(chat_uuid, user.id):
raise HTTPException(404, "Chat not found")
if (
not filename.strip()
or any(c in filename for c in ("/", "\\", "\x00"))
or any(ord(c) < 32 or ord(c) == 127 for c in filename)
):
raise HTTPException(400, "Unsafe filename")
content_type = (
request.headers.get("content-type", "")
.split(";", 1)[0]
.strip()
.lower()
)
if content_type not in {"image/png", "image/jpeg"}:
raise HTTPException(415, "Only PNG and JPEG are supported")
spool, size = await read_limited(request)
attachment_id = uuid.uuid4()
job_id = uuid.uuid4()
staging_key = f"staging/{chat_uuid}/{attachment_id}"
object_key = f"original/{chat_uuid}/{attachment_id}"
staged = False
try:
actual_type, _, _ = validate_image_file(spool, content_type)
digest = sha256_file(spool)
# Serialize admission checks across all API instances.
db.execute(
text("SELECT pg_advisory_xact_lock(:lock_id)"),
{"lock_id": 72420261110},
)
active_jobs = db.scalar(
select(func.count())
.select_from(ProcessingJob)
.where(ProcessingJob.status.in_(("queued", "processing")))
) or 0
if active_jobs >= settings.max_active_processing_jobs:
raise HTTPException(
status_code=429,
detail="Processing queue is full; retry later",
headers={"Retry-After": "5"},
)
s3 = client()
try:
spool.seek(0)
s3.put_object(
Bucket=settings.s3_bucket,
Key=staging_key,
Body=spool,
ContentType=actual_type,
)
staged = True
except (BotoCoreError, ClientError):
raise HTTPException(
503, "Object storage unavailable"
) from None
now = datetime.now(timezone.utc)
attachment = Attachment(
id=attachment_id,
chat_id=chat_uuid,
owner_id=user.id,
filename=filename,
content_type=actual_type,
size_bytes=size,
sha256=digest,
object_key=object_key,
staging_key=staging_key,
status="queued",
error_code=None,
pipeline_version=1,
)
job = ProcessingJob(
id=job_id,
attachment_id=attachment_id,
pipeline_version=1,
attempt=0,
status="queued",
available_at=now,
)
# Preserve the current HTTP trace context in the durable outbox event.
trace_context = {}
inject(trace_context)
event = OutboxEvent(
event_type="attachment.process.v1",
aggregate_id=attachment_id,
payload={
"schema_version": 1,
"job_id": str(job_id),
"type": "attachment.process.v1",
"attachment_id": str(attachment_id),
"pipeline_version": 1,
"attempt": 1,
"created_at": now.isoformat(),
"trace_context": trace_context,
},
status="pending",
)
db.add_all([attachment, job, event])
db.commit()
db.refresh(attachment)
return attachment_json(attachment, db)
except HTTPException:
db.rollback()
if staged:
try:
client().delete_object(
Bucket=settings.s3_bucket,
Key=staging_key,
)
except Exception:
pass
raise
except Exception:
db.rollback()
if staged:
try:
client().delete_object(
Bucket=settings.s3_bucket,
Key=staging_key,
)
except Exception:
pass
raise HTTPException(
503, "Could not save attachment processing job"
) from None
finally:
spool.close()
@router.get("/attachments/{attachment_id}")
def get_attachment(
attachment_id: str,
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
parsed = uuid.UUID(attachment_id)
except ValueError:
raise HTTPException(400, "Invalid attachment UUID") from None
item = member_attachment(db, parsed, user.id)
return attachment_json(item, db)
@router.get("/attachments/{attachment_id}/content")
def download_attachment(
attachment_id: str,
variant: str = "original",
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
parsed = uuid.UUID(attachment_id)
except ValueError:
raise HTTPException(400, "Invalid attachment UUID") from None
item = member_attachment(db, parsed, user.id)
if item.status != "ready":
raise HTTPException(409, "Attachment is not ready")
# Select the original file or a generated variant.
if variant == "original":
object_key = item.object_key
expected_size = item.size_bytes
expected_sha256 = item.sha256
content_type = item.content_type
else:
variant_item = db.scalar(
select(AttachmentVariant).where(
AttachmentVariant.attachment_id == item.id,
AttachmentVariant.variant == variant,
)
)
if variant_item is None:
raise HTTPException(404, "Variant not found")
object_key = variant_item.object_key
expected_size = variant_item.size_bytes
expected_sha256 = variant_item.sha256
content_type = variant_item.content_type
spool = tempfile.SpooledTemporaryFile(
max_size=1024 * 1024,
mode="w+b",
)
try:
result = client().get_object(
Bucket=settings.s3_bucket,
Key=object_key,
)
body = result["Body"]
total = 0
digest = hashlib.sha256()
try:
while True:
chunk = body.read(64 * 1024)
if not chunk:
break
total += len(chunk)
if total > MAX_BYTES:
raise HTTPException(
503,
"Stored object exceeds configured limit",
)
digest.update(chunk)
spool.write(chunk)
finally:
body.close()
# Verify stored object integrity.
if (
total != expected_size
or digest.hexdigest() != expected_sha256
):
raise HTTPException(
503,
"Stored object checksum mismatch",
)
spool.seek(0)
except (BotoCoreError, ClientError):
spool.close()
raise HTTPException(
503,
"Object storage unavailable",
) from None
except Exception:
spool.close()
raise
def stream_verified_file():
try:
while True:
chunk = spool.read(64 * 1024)
if not chunk:
break
yield chunk
finally:
spool.close()
return StreamingResponse(
stream_verified_file(),
media_type=content_type,
headers={
"Content-Disposition": (
f"inline; filename*=UTF-8''{quote(item.filename)}"
)
},
)
+96
View File
@@ -0,0 +1,96 @@
from datetime import timezone
from fastapi import APIRouter, Depends, Header, HTTPException, status
from fastapi.security import HTTPBasic, HTTPBasicCredentials
from sqlalchemy.orm import Session
from app.api.deps import get_db
from app.api.users import user_to_response
from app.services.auth_service import (
AuthService,
InvalidCredentialsError,
InvalidSessionError,
)
router = APIRouter(prefix="/api/v1/auth", tags=["Authentication"])
basic_security = HTTPBasic(auto_error=False)
@router.post("/sessions", status_code=status.HTTP_201_CREATED)
def create_session(
credentials: HTTPBasicCredentials = Depends(basic_security),
db: Session = Depends(get_db),
):
if credentials is None:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Basic authentication required",
headers={
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'
},
)
try:
result = AuthService(db).create_session(
username=credentials.username,
password=credentials.password,
)
expires_at = result["expires_at"]
if expires_at.tzinfo is None:
expires_at = expires_at.replace(tzinfo=timezone.utc)
else:
expires_at = expires_at.astimezone(timezone.utc)
return {
"token": result["token"],
"token_type": result["token_type"],
"expires_at": expires_at.isoformat(
timespec="microseconds"
).replace("+00:00", "Z"),
"user": user_to_response(result["user"]),
}
except InvalidCredentialsError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid username or password",
headers={
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'
},
) from None
@router.delete("/sessions/current", status_code=status.HTTP_204_NO_CONTENT)
def delete_current_session(
authorization: str | None = Header(default=None),
db: Session = Depends(get_db),
):
if not authorization or not authorization.startswith("Bearer "):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Bearer token required",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
)
token = authorization[7:].strip()
if not token:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Bearer token required",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
)
try:
AuthService(db).delete_session(token)
except InvalidSessionError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or expired session",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
) from None
+90
View File
@@ -0,0 +1,90 @@
import uuid
from fastapi import APIRouter, Depends, HTTPException, Query, status
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_user
from app.db.models import User as UserModel
from app.models import Chat as ChatResponse, CreateChat
from app.services.chat_service import (
ChatService,
ChatNotFoundError,
InvalidChatDataError,
)
router = APIRouter(prefix="/api/v1/chats", tags=["Chats"])
def chat_to_response(chat) -> ChatResponse:
return ChatResponse(
id=str(chat.id),
title=chat.title,
member_ids=[str(member.user_id) for member in chat.members],
created_at=chat.created_at,
)
@router.post("", response_model=ChatResponse, status_code=status.HTTP_201_CREATED)
def create_chat(
body: CreateChat,
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
try:
chat = ChatService(db).create_chat(
title=body.title,
member_ids=body.member_ids,
creator_id=user.id,
)
return chat_to_response(chat)
except InvalidChatDataError as exc:
raise HTTPException(
status_code=400,
detail=str(exc),
) from None
@router.get("")
def list_chats(
limit: int = Query(default=50, ge=1, le=100),
cursor: str | None = None,
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
if cursor is not None:
raise HTTPException(
status_code=400,
detail="Chat cursor pagination is not implemented yet",
)
chats = ChatService(db).list_chats(user.id)
chats = chats[:limit]
return {
"items": [chat_to_response(chat) for chat in chats],
"next_cursor": None,
}
@router.get("/{chat_id}", response_model=ChatResponse)
def get_chat(
chat_id: str,
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
try:
parsed_id = uuid.UUID(chat_id)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid chat UUID") from None
try:
chat = ChatService(db).get_chat(parsed_id, user.id)
return chat_to_response(chat)
except ChatNotFoundError:
raise HTTPException(
status_code=404,
detail="Chat not found",
) from None
+40
View File
@@ -0,0 +1,40 @@
from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from sqlalchemy.orm import Session
from app.db.session import SessionLocal
from app.services.auth_service import AuthService, InvalidSessionError
bearer_scheme = HTTPBearer(auto_error=False)
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
def get_current_user(
credentials: HTTPAuthorizationCredentials | None = Depends(bearer_scheme),
db: Session = Depends(get_db),
):
if credentials is None or credentials.scheme.lower() != "bearer":
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Bearer token required",
headers={"WWW-Authenticate": "Bearer"},
)
try:
user = AuthService(db).authenticate(credentials.credentials)
return user
except InvalidSessionError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or expired token",
headers={"WWW-Authenticate": "Bearer"},
) from None
+73
View File
@@ -0,0 +1,73 @@
import re
import uuid
from fastapi import APIRouter, Depends, Header, HTTPException, Query, status
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_user
from app.db.models import User as UserModel
from app.models import CreateMessage, Message as MessageResponse
from app.services.message_service import (
MessageService, MessageNotFoundError, InvalidMessageDataError,
IdempotencyConflictError,
)
router = APIRouter(prefix="/api/v1/chats/{chat_id}/messages", tags=["Messages"])
IDEMPOTENCY_KEY_RE = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
def message_to_response(message) -> MessageResponse:
return MessageResponse(
id=str(message.id), chat_id=str(message.chat_id),
sender_id=str(message.sender_id), text=message.text,
created_at=message.created_at, attachment_ids=message.attachment_ids or [],
)
def parse_chat_id(chat_id: str) -> uuid.UUID:
try:
return uuid.UUID(chat_id)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid chat UUID") from None
@router.post("", response_model=MessageResponse, status_code=status.HTTP_201_CREATED)
def create_message(
chat_id: str, body: CreateMessage,
idempotency_key: str = Header(..., alias="Idempotency-Key", min_length=1, max_length=128),
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
if not IDEMPOTENCY_KEY_RE.fullmatch(idempotency_key):
raise HTTPException(status_code=400, detail="Invalid Idempotency-Key")
parsed_id = parse_chat_id(chat_id)
try:
message = MessageService(db).create_message(
chat_id=parsed_id, sender_id=user.id, text=body.text,
idempotency_key=idempotency_key, attachment_ids=body.attachment_ids,
)
return message_to_response(message)
except MessageNotFoundError:
raise HTTPException(status_code=404, detail="Chat not found") from None
except InvalidMessageDataError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from None
except IdempotencyConflictError:
raise HTTPException(status_code=409, detail="Idempotency-Key was already used with a different request") from None
@router.get("")
def list_messages(
chat_id: str, limit: int = Query(default=50, ge=1, le=100),
cursor: str | None = None, db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
parsed_id = parse_chat_id(chat_id)
try:
messages, next_cursor = MessageService(db).get_history(
chat_id=parsed_id, user_id=user.id, limit=limit, cursor=cursor
)
return {"items": [message_to_response(item) for item in messages], "next_cursor": next_cursor}
except MessageNotFoundError:
raise HTTPException(status_code=404, detail="Chat not found") from None
except InvalidMessageDataError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from None
+55
View File
@@ -0,0 +1,55 @@
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_user
from app.db.models import User as UserModel
from app.models import CreateUser, User
from app.services.user_service import (
UserService,
UsernameAlreadyExistsError,
)
router = APIRouter(prefix="/api/v1/users", tags=["Users"])
def user_to_response(user: UserModel) -> User:
return User(
id=str(user.id),
username=user.username,
display_name=user.display_name,
created_at=user.created_at,
)
@router.post("", response_model=User, status_code=status.HTTP_201_CREATED)
def register_user(
body: CreateUser,
db: Session = Depends(get_db),
):
try:
user = UserService(db).register(
username=body.username,
display_name=body.display_name,
password=body.password,
)
return user_to_response(user)
except UsernameAlreadyExistsError:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail="Username already exists",
) from None
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=str(exc),
) from None
@router.get("/me", response_model=User)
def get_me(
current_user: UserModel = Depends(get_current_user),
):
return user_to_response(current_user)
View File
+73
View File
@@ -0,0 +1,73 @@
import os
from pydantic import AliasChoices, Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
app_name: str = "Messenger API"
app_env: str = "lab"
database_url: str = "postgresql+psycopg://messenger:messenger@postgres:5432/messenger"
db_pool_size: int = 5
db_max_overflow: int = 5
db_pool_timeout: int = 10
session_ttl_seconds: int = 3600
cache_url: str = "redis://redis:6379/0"
message_cache_ttl_seconds: int = 60
idempotency_ttl_seconds: int = 86400
instance_id: str = os.getenv("INSTANCE_ID") or os.getenv("HOSTNAME") or "api-local"
shutdown_grace_seconds: int = 20
max_active_processing_jobs: int = Field(default=20, ge=1, le=10000)
s3_endpoint_url: str = Field(
default="http://minio:9000",
validation_alias=AliasChoices("S3_ENDPOINT_URL", "S3_ENDPOINT"),
)
s3_public_endpoint_url: str = Field(
default="http://localhost:9000",
validation_alias=AliasChoices("S3_PUBLIC_ENDPOINT_URL"),
)
presigned_upload_ttl_seconds: int = 300
attachment_cleanup_interval_seconds: int = Field(default=60, ge=5, le=3600)
attachment_cleanup_grace_seconds: int = Field(default=600, ge=0, le=86400)
attachment_cleanup_batch_size: int = Field(default=100, ge=1, le=1000)
s3_region: str = Field(default="us-east-1", validation_alias=AliasChoices("S3_REGION"))
s3_bucket: str = Field(default="messenger-attachments", validation_alias=AliasChoices("S3_BUCKET"))
s3_access_key: str = Field(
default="messenger-api",
validation_alias=AliasChoices("S3_ACCESS_KEY", "S3_ACCESS_KEY_ID"),
)
s3_secret_key: str = Field(
default="change-me-api-secret",
validation_alias=AliasChoices("S3_SECRET_KEY", "S3_SECRET_ACCESS_KEY"),
)
max_upload_bytes: int = 10 * 1024 * 1024
max_image_pixels: int = 40_000_000
# Durable asynchronous image processing.
rabbitmq_url: str = Field(
default="amqp://guest:guest@rabbitmq:5672/",
validation_alias=AliasChoices("RABBITMQ_URL"),
)
rabbitmq_exchange: str = Field(default="attachments", validation_alias=AliasChoices("RABBITMQ_EXCHANGE"))
rabbitmq_queue: str = Field(default="attachment.process.v1", validation_alias=AliasChoices("RABBITMQ_QUEUE"))
rabbitmq_prefetch: int = Field(default=4, ge=1, le=64)
outbox_poll_interval_seconds: float = Field(default=1.0, ge=0.1, le=30)
outbox_batch_size: int = Field(default=50, ge=1, le=500)
worker_concurrency: int = Field(default=2, ge=1, le=16)
worker_max_attempts: int = Field(default=5, ge=1, le=20)
worker_lease_seconds: int = Field(default=180, ge=30, le=3600)
worker_retry_base_seconds: float = Field(default=2.0, ge=0.1, le=300)
worker_retry_max_seconds: float = Field(default=120.0, ge=1, le=3600)
worker_staging_retention_seconds: int = Field(default=86400, ge=3600)
image_processing_timeout_seconds: int = Field(default=60, ge=1, le=600)
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)
settings = Settings()
+118
View File
@@ -0,0 +1,118 @@
import logging
import os
import socket
from opentelemetry import metrics, trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import (
OTLPMetricExporter,
)
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
OTLPSpanExporter,
)
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.logging import LoggingInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from app.core.config import settings
logger = logging.getLogger(__name__)
_initialized = False
def setup_telemetry(app, engine) -> None:
"""Initialize OpenTelemetry instrumentation for the API service."""
global _initialized
if _initialized:
return
service_name = os.getenv("OTEL_SERVICE_NAME", "messenger-api")
endpoint = os.getenv(
"OTEL_EXPORTER_OTLP_ENDPOINT",
"http://otel-collector:4318",
).rstrip("/")
resource = Resource.create(
{
"service.name": service_name,
"service.version": "1.4.0",
"service.instance.id": settings.instance_id or socket.gethostname(),
"deployment.environment.name": settings.app_env,
}
)
# Distributed tracing.
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint=f"{endpoint}/v1/traces",
)
)
)
trace.set_tracer_provider(tracer_provider)
# Metrics.
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(
endpoint=f"{endpoint}/v1/metrics",
),
export_interval_millis=10000,
)
meter_provider = MeterProvider(
resource=resource,
metric_readers=[metric_reader],
)
metrics.set_meter_provider(meter_provider)
# Logs.
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
BatchLogRecordProcessor(
OTLPLogExporter(
endpoint=f"{endpoint}/v1/logs",
)
)
)
set_logger_provider(logger_provider)
# Add trace_id and span_id to Python log records.
LoggingInstrumentor().instrument(
tracer_provider=tracer_provider,
set_logging_format=False,
)
# Export Python logging records through OTLP.
logging_handler = LoggingHandler(
level=logging.INFO,
logger_provider=logger_provider,
)
logging.getLogger().addHandler(logging_handler)
# FastAPI HTTP server spans. Health endpoints are excluded.
FastAPIInstrumentor.instrument_app(
app,
tracer_provider=tracer_provider,
meter_provider=meter_provider,
excluded_urls=r".*/health/(live|ready)$",
)
# SQLAlchemy database spans.
SQLAlchemyInstrumentor().instrument(
engine=engine,
tracer_provider=tracer_provider,
)
_initialized = True
logger.info("OpenTelemetry initialized for service %s", service_name)
+239
View File
@@ -0,0 +1,239 @@
import logging
import os
import socket
import time
from opentelemetry import metrics, trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import (
OTLPMetricExporter,
)
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
OTLPSpanExporter,
)
from opentelemetry.instrumentation.logging import LoggingInstrumentor
from opentelemetry.propagate import extract
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import SpanKind, Status, StatusCode
logger = logging.getLogger(__name__)
_initialized = False
tracer = None
jobs_processed = None
jobs_failed = None
job_duration = None
def setup_worker_telemetry():
"""Initialize OpenTelemetry for the background worker."""
global _initialized
global tracer, jobs_processed, jobs_failed, job_duration
if _initialized:
return
endpoint = os.getenv(
"OTEL_EXPORTER_OTLP_ENDPOINT",
"http://otel-collector:4318",
).rstrip("/")
service_name = os.getenv(
"OTEL_SERVICE_NAME",
"messenger-worker",
)
resource = Resource.create(
{
"service.name": service_name,
"service.version": "1.4.0",
"service.instance.id": (
os.getenv("INSTANCE_ID")
or socket.gethostname()
),
"deployment.environment.name": os.getenv(
"APP_ENV",
"lab",
),
}
)
# Distributed tracing.
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint=f"{endpoint}/v1/traces",
)
)
)
trace.set_tracer_provider(tracer_provider)
# Metrics.
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(
endpoint=f"{endpoint}/v1/metrics",
),
export_interval_millis=10000,
)
meter_provider = MeterProvider(
resource=resource,
metric_readers=[metric_reader],
)
metrics.set_meter_provider(meter_provider)
tracer = trace.get_tracer("messenger.worker")
meter = metrics.get_meter("messenger.worker")
jobs_processed = meter.create_counter(
name="jobs_processed",
description="Number of successfully processed background jobs",
unit="1",
)
jobs_failed = meter.create_counter(
name="jobs_failed",
description="Number of failed background jobs",
unit="1",
)
job_duration = meter.create_histogram(
name="job_processing_duration",
description="Background job processing duration",
unit="s",
)
# Logs.
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
BatchLogRecordProcessor(
OTLPLogExporter(
endpoint=f"{endpoint}/v1/logs",
)
)
)
set_logger_provider(logger_provider)
# Add trace_id and span_id to Python log records.
LoggingInstrumentor().instrument(
tracer_provider=tracer_provider,
set_logging_format=False,
)
# Export Python logging records through OTLP.
logging_handler = LoggingHandler(
level=logging.INFO,
logger_provider=logger_provider,
)
logging.getLogger().addHandler(logging_handler)
_initialized = True
logger.info(
"OpenTelemetry initialized for worker service %s",
service_name,
)
def process_job_with_telemetry(process_function, payload):
"""Run one job inside a consumer span and record its metrics."""
if not _initialized:
setup_worker_telemetry()
started = time.perf_counter()
outcome = "other"
# Continue the trace started by the API request.
context = extract(payload.get("trace_context", {}))
with tracer.start_as_current_span(
"worker.process_job",
context=context,
kind=SpanKind.CONSUMER,
attributes={
"messaging.system": "rabbitmq",
"messaging.operation": "process",
"job.type": "image",
},
) as span:
try:
result = process_function(payload)
state = result.get("state", "unknown")
span.set_attribute("job.outcome", state)
if state == "ready":
outcome = "success"
jobs_processed.add(
1,
{
"job_type": "image",
"outcome": "success",
},
)
elif state == "failed":
outcome = "failed"
span.set_status(
Status(
StatusCode.ERROR,
"Background job processing failed",
)
)
jobs_failed.add(
1,
{
"job_type": "image",
"outcome": "failed",
},
)
elif state in {"retry", "busy", "deferred"}:
outcome = "retry"
else:
outcome = "other"
return result
except Exception as exc:
outcome = "failed"
span.set_attribute("job.outcome", "failed")
span.record_exception(exc)
span.set_status(
Status(
StatusCode.ERROR,
"Background job processing failed",
)
)
jobs_failed.add(
1,
{
"job_type": "image",
"outcome": "failed",
},
)
raise
finally:
elapsed = time.perf_counter() - started
job_duration.record(
elapsed,
{
"job_type": "image",
"outcome": outcome,
},
)
View File
+5
View File
@@ -0,0 +1,5 @@
from sqlalchemy.orm import DeclarativeBase
class Base(DeclarativeBase):
pass
+488
View File
@@ -0,0 +1,488 @@
import uuid
from datetime import datetime, timezone
from sqlalchemy import (
DateTime,
JSON,
ForeignKey,
Index,
String,
Text,
UniqueConstraint,
)
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db.base import Base
def utc_now() -> datetime:
return datetime.now(timezone.utc)
class User(Base):
__tablename__ = "users"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
username: Mapped[str] = mapped_column(
String(32),
unique=True,
nullable=False,
index=True,
)
display_name: Mapped[str] = mapped_column(
String(100),
nullable=False,
)
password_hash: Mapped[str] = mapped_column(
String(255),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
memberships: Mapped[list["ChatMember"]] = relationship(
back_populates="user",
cascade="all, delete-orphan",
)
messages: Mapped[list["Message"]] = relationship(
back_populates="sender",
)
class Chat(Base):
__tablename__ = "chats"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
title: Mapped[str] = mapped_column(
String(100),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
members: Mapped[list["ChatMember"]] = relationship(
back_populates="chat",
cascade="all, delete-orphan",
)
messages: Mapped[list["Message"]] = relationship(
back_populates="chat",
cascade="all, delete-orphan",
)
class ChatMember(Base):
__tablename__ = "chat_members"
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
primary_key=True,
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
primary_key=True,
)
chat: Mapped["Chat"] = relationship(
back_populates="members",
)
user: Mapped["User"] = relationship(
back_populates="memberships",
)
class Message(Base):
__tablename__ = "messages"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
nullable=False,
)
sender_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id"),
nullable=False,
)
text: Mapped[str] = mapped_column(
Text,
nullable=False,
)
attachment_ids: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
chat: Mapped["Chat"] = relationship(
back_populates="messages",
)
sender: Mapped["User"] = relationship(
back_populates="messages",
)
__table_args__ = (
Index(
"ix_messages_chat_created_id",
"chat_id",
"created_at",
"id",
),
)
class Session(Base):
__tablename__ = "sessions"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
token_hash: Mapped[str] = mapped_column(
String(64),
unique=True,
nullable=False,
index=True,
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
expires_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
user: Mapped["User"] = relationship()
class IdempotencyRecord(Base):
__tablename__ = "idempotency_records"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False
)
method: Mapped[str] = mapped_column(String(10), nullable=False)
path: Mapped[str] = mapped_column(String(255), nullable=False)
idempotency_key: Mapped[str] = mapped_column(String(128), nullable=False)
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
# NULL is visible only inside the creating transaction; the message and
# idempotency row are committed atomically.
message_id: Mapped[uuid.UUID | None] = mapped_column(
UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), nullable=True
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=utc_now, nullable=False
)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
__table_args__ = (
UniqueConstraint(
"user_id", "method", "path", "idempotency_key",
name="uq_idempotency_scope_key",
),
Index("ix_idempotency_records_expires_at", "expires_at"),
)
class Attachment(Base):
__tablename__ = "attachments"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
owner_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
filename: Mapped[str] = mapped_column(String(255), nullable=False)
content_type: Mapped[str] = mapped_column(String(32), nullable=False)
size_bytes: Mapped[int] = mapped_column(nullable=False)
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
# Ключ оригинала в конечном S3-prefix.
object_key: Mapped[str] = mapped_column(
String(512),
unique=True,
nullable=False,
)
# Ключ исходного файла в durable staging.
staging_key: Mapped[str | None] = mapped_column(
String(512),
nullable=True,
)
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="ready",
)
error_code: Mapped[str | None] = mapped_column(
String(64),
nullable=True,
)
pipeline_version: Mapped[int] = mapped_column(
nullable=False,
default=1,
server_default="1", )
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
onupdate=utc_now,
nullable=False,
)
__table_args__ = (
UniqueConstraint(
"staging_key",
name="uq_attachments_staging_key",
),
)
class AttachmentUpload(Base):
__tablename__ = "attachment_uploads"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
nullable=False,
)
owner_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
filename: Mapped[str] = mapped_column(
String(255),
nullable=False,
)
content_type: Mapped[str] = mapped_column(
String(32),
nullable=False,
)
expected_size_bytes: Mapped[int] = mapped_column(
nullable=False,
)
expected_sha256: Mapped[str] = mapped_column(
String(64),
nullable=False,
)
object_key: Mapped[str] = mapped_column(
String(512),
unique=True,
nullable=False,
)
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="pending",
)
expires_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
completed_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
)
__table_args__ = (
Index("ix_attachment_uploads_chat_id", "chat_id"),
Index("ix_attachment_uploads_owner_id", "owner_id"),
Index("ix_attachment_uploads_expires_at", "expires_at"),
)
class ProcessingJob(Base):
__tablename__ = "processing_jobs"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
attachment_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("attachments.id", ondelete="CASCADE"),
nullable=False,
)
pipeline_version: Mapped[int] = mapped_column(
nullable=False,
default=1,
server_default="1", )
attempt: Mapped[int] = mapped_column(
nullable=False,
default=1,
server_default="1", )
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="queued",
server_default="queued", )
available_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
lease_until: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
)
worker_id: Mapped[str | None] = mapped_column(
String(128),
nullable=True,
)
last_error_code: Mapped[str | None] = mapped_column(
String(64),
nullable=True,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
onupdate=utc_now,
nullable=False,
)
__table_args__ = (
UniqueConstraint(
"attachment_id",
"pipeline_version",
name="uq_processing_job_attachment_pipeline",
),
Index(
"ix_processing_jobs_status_available",
"status",
"available_at",
),
Index(
"ix_processing_jobs_lease_until",
"lease_until",
),
)
class AttachmentVariant(Base):
__tablename__ = "attachment_variants"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
attachment_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("attachments.id", ondelete="CASCADE"),
nullable=False,
)
variant: Mapped[str] = mapped_column(
String(32),
nullable=False,
)
object_key: Mapped[str] = mapped_column(
String(512),
unique=True,
nullable=False,
)
content_type: Mapped[str] = mapped_column(
String(32),
nullable=False,
)
width: Mapped[int] = mapped_column(nullable=False)
height: Mapped[int] = mapped_column(nullable=False)
size_bytes: Mapped[int] = mapped_column(nullable=False)
sha256: Mapped[str] = mapped_column(
String(64),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
__table_args__ = (
UniqueConstraint(
"attachment_id",
"variant",
name="uq_attachment_variant",
),
Index(
"ix_attachment_variants_attachment_id",
"attachment_id",
),
)
class OutboxEvent(Base):
__tablename__ = "outbox_events"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
event_type: Mapped[str] = mapped_column(
String(128),
nullable=False,
)
aggregate_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
nullable=False,
)
payload: Mapped[dict] = mapped_column(
JSON,
nullable=False,
)
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="pending",
server_default="pending", )
attempts: Mapped[int] = mapped_column(
nullable=False,
default=0,
server_default="0", )
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
published_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
)
last_error: Mapped[str | None] = mapped_column(
String(256),
nullable=True,
)
__table_args__ = (
Index(
"ix_outbox_events_status_created",
"status",
"created_at",
),
)
+15
View File
@@ -0,0 +1,15 @@
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.core.config import settings
engine = create_engine(
settings.database_url,
pool_pre_ping=True,
pool_size=settings.db_pool_size,
max_overflow=settings.db_max_overflow,
pool_timeout=settings.db_pool_timeout,
pool_recycle=1800,
)
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
+232
View File
@@ -0,0 +1,232 @@
import asyncio
import logging
import time
import uuid
from contextlib import asynccontextmanager, suppress
from pathlib import Path
from fastapi import FastAPI, HTTPException, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from sqlalchemy import text
from app.api.auth import router as auth_router
from app.api.users import router as users_router
from app.api.chats import router as chats_router
from app.api.messages import router as messages_router
from app.api.attachments import router as attachments_router
from app.core.config import settings
from app.core.telemetry import setup_telemetry
from app.db.session import engine
from app.services.object_storage import client as s3_client
from app.services.attachment_cleanup import cleanup_expired_uploads
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
logger = logging.getLogger(__name__)
async def attachment_cleanup_loop():
while True:
try:
# Synchronous boto3/SQLAlchemy I/O must not block the event loop.
await asyncio.to_thread(cleanup_expired_uploads)
except Exception:
logger.exception("Attachment cleanup worker iteration failed")
await asyncio.sleep(settings.attachment_cleanup_interval_seconds)
@asynccontextmanager
async def lifespan(app: FastAPI):
logger.info("Application startup instance_id=%s", settings.instance_id)
cleanup_task = asyncio.create_task(attachment_cleanup_loop())
try:
yield
finally:
cleanup_task.cancel()
with suppress(asyncio.CancelledError):
await cleanup_task
engine.dispose()
logger.info("Application shutdown instance_id=%s", settings.instance_id)
app = FastAPI(
title=settings.app_name,
version="1.4.0",
lifespan=lifespan,
)
app.include_router(users_router)
app.include_router(auth_router)
app.include_router(chats_router)
app.include_router(messages_router)
app.include_router(attachments_router)
@app.middleware("http")
async def response_headers(request: Request, call_next):
request_id = str(uuid.uuid4())
request.state.request_id = request_id
started_at = time.perf_counter()
try:
response = await call_next(request)
except Exception:
logger.exception(
"Unhandled request error; request_id=%s",
request_id,
)
response = JSONResponse(
status_code=500,
content={
"error": {
"code": "internal_error",
"message": "Unexpected error",
"request_id": request_id,
}
},
)
duration_ms = (time.perf_counter() - started_at) * 1000
route = request.scope.get("route")
route_template = getattr(route, "path", "unmatched")
if route_template not in ("/health/live", "/health/ready"):
logger.info(
"HTTP request completed",
extra={
"event": "http_request_completed",
"http_method": request.method,
"http_route": route_template,
"http_status_code": response.status_code,
"http_duration_ms": round(duration_ms, 3),
},
)
response.headers["X-Request-Id"] = request_id
response.headers["X-Instance-Id"] = settings.instance_id
response.headers["X-Content-Type-Options"] = "nosniff"
return response
# Initialize OpenTelemetry AFTER registering the application middleware.
# This makes OpenTelemetry middleware outermost, keeping the server span
# active while response_headers writes the completed-request log.
setup_telemetry(app, engine)
@app.exception_handler(HTTPException)
async def http_exception_handler(request: Request, exc: HTTPException):
request_id = getattr(
request.state,
"request_id",
str(uuid.uuid4()),
)
code = {
400: "invalid_request",
401: "unauthorized",
404: "not_found",
409: "conflict",
413: "payload_too_large",
415: "unsupported_media_type",
429: "rate_limited",
500: "internal_error",
503: "unavailable",
}.get(exc.status_code, "internal_error")
headers = dict(exc.headers or {})
headers["X-Request-Id"] = request_id
headers["X-Instance-Id"] = settings.instance_id
return JSONResponse(
status_code=exc.status_code,
content={
"error": {
"code": code,
"message": (
exc.detail
if isinstance(exc.detail, str)
else "Request failed"
),
"request_id": request_id,
}
},
headers=headers,
)
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(
request: Request,
exc: RequestValidationError,
):
request_id = getattr(
request.state,
"request_id",
str(uuid.uuid4()),
)
return JSONResponse(
status_code=400,
content={
"error": {
"code": "invalid_request",
"message": "Validation error",
"request_id": request_id,
}
},
headers={
"X-Request-Id": request_id,
"X-Instance-Id": settings.instance_id,
},
)
@app.get("/health/live")
def liveness():
return {"status": "ok"}
DRAIN_FILE = Path("/tmp/api-draining")
@app.get("/health/ready")
def readiness():
if DRAIN_FILE.exists():
raise HTTPException(
status_code=503,
detail="Instance is draining",
)
try:
with engine.connect() as connection:
connection.execute(text("SELECT 1"))
s3_client().head_bucket(Bucket=settings.s3_bucket)
return {"status": "ok"}
except Exception:
logger.exception(
"PostgreSQL or S3 readiness check failed"
)
raise HTTPException(
status_code=503,
detail="Required storage unavailable",
) from None
+138
View File
@@ -0,0 +1,138 @@
from datetime import datetime, timezone
from typing import List, Optional, Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator
class BaseConfig(BaseModel):
model_config = ConfigDict(
from_attributes=True,
json_encoders={
datetime: lambda value: (
value.astimezone(timezone.utc)
.isoformat(timespec="microseconds")
.replace("+00:00", "Z")
)
},
)
class CreateUser(BaseConfig):
username: str = Field(..., pattern=r"^[a-z0-9_]{3,32}$")
display_name: str = Field(..., min_length=1, max_length=100)
password: str = Field(..., min_length=12, max_length=128)
@field_validator("display_name")
@classmethod
def validate_display_name(cls, value: str) -> str:
value = value.strip()
if not value:
raise ValueError("display_name must not be blank")
return value
class User(BaseConfig):
id: str
username: str
display_name: str
created_at: datetime
class CreateChat(BaseConfig):
title: str = Field(..., min_length=1, max_length=100)
member_ids: List[str] = Field(..., min_length=1, max_length=100)
@field_validator("title")
@classmethod
def validate_title(cls, value: str) -> str:
value = value.strip()
if not value:
raise ValueError("title must not be blank")
return value
@field_validator("member_ids")
@classmethod
def validate_unique_members(cls, value: List[str]) -> List[str]:
if len(value) != len(set(value)):
raise ValueError("member_ids must be unique")
return value
class Chat(BaseConfig):
id: str
title: str
member_ids: List[str]
created_at: datetime
class CreateMessage(BaseConfig):
text: str = Field(..., max_length=4000)
attachment_ids: List[str] = Field(default_factory=list, max_length=10)
@field_validator("attachment_ids")
@classmethod
def validate_attachment_ids(cls, value):
if len(value) != len(set(value)):
raise ValueError("attachment_ids must be unique")
return value
@field_validator("text")
@classmethod
def validate_text(cls, value: str) -> str:
value = value.strip()
return value
class Message(BaseConfig):
id: str
chat_id: str
sender_id: str
text: str
created_at: datetime
attachment_ids: List[str] = Field(default_factory=list)
class Health(BaseConfig):
status: str = "ok"
class ErrorDetail(BaseConfig):
code: str
message: str
request_id: str
class ErrorResponse(BaseConfig):
error: ErrorDetail
class ChatPage(BaseConfig):
items: List[Chat]
next_cursor: Optional[str] = None
class MessagePage(BaseConfig):
items: List[Message]
next_cursor: Optional[str] = None
class CreateAttachmentUpload(BaseConfig):
filename: str = Field(..., min_length=1, max_length=255)
content_type: Literal["image/png", "image/jpeg"]
size_bytes: int = Field(..., gt=0, le=10 * 1024 * 1024)
sha256: str = Field(..., pattern=r"^[a-fA-F0-9]{64}$")
@field_validator("filename")
@classmethod
def validate_filename(cls, value: str) -> str:
value = value.strip()
if (
not value
or any(c in value for c in ("/", "\\", "\x00"))
or any(ord(c) < 32 or ord(c) == 127 for c in value)
):
raise ValueError("Unsafe filename")
return value
View File
+71
View File
@@ -0,0 +1,71 @@
import uuid
from sqlalchemy import select
from sqlalchemy.orm import Session, selectinload
from app.db.models import Chat, ChatMember
class ChatRepository:
def __init__(self, db: Session):
self.db = db
def get_by_id(self, chat_id: uuid.UUID) -> Chat | None:
statement = (
select(Chat)
.options(selectinload(Chat.members))
.where(Chat.id == chat_id)
)
return self.db.scalar(statement)
def get_by_user(
self,
user_id: uuid.UUID,
limit: int = 50,
) -> list[Chat]:
statement = (
select(Chat)
.join(ChatMember, ChatMember.chat_id == Chat.id)
.where(ChatMember.user_id == user_id)
.order_by(Chat.created_at.asc(), Chat.id.asc())
.limit(limit)
)
return list(self.db.scalars(statement).all())
def is_member(
self,
chat_id: uuid.UUID,
user_id: uuid.UUID,
) -> bool:
statement = select(ChatMember).where(
ChatMember.chat_id == chat_id,
ChatMember.user_id == user_id,
)
return self.db.scalar(statement) is not None
def create(
self,
title: str,
member_ids: list[uuid.UUID],
) -> Chat:
chat = Chat(title=title)
self.db.add(chat)
self.db.flush()
memberships = [
ChatMember(
chat_id=chat.id,
user_id=user_id,
)
for user_id in member_ids
]
self.db.add_all(memberships)
self.db.flush()
return chat
+73
View File
@@ -0,0 +1,73 @@
import uuid
from datetime import datetime
from sqlalchemy import and_, or_, select
from sqlalchemy.orm import Session
from app.db.models import Message
class MessageRepository:
def __init__(self, db: Session):
self.db = db
def get_by_id(
self,
message_id: uuid.UUID,
) -> Message | None:
statement = select(Message).where(
Message.id == message_id
)
return self.db.scalar(statement)
def create(
self,
chat_id: uuid.UUID,
sender_id: uuid.UUID,
text: str,
attachment_ids: list[str] | None = None,
) -> Message:
message = Message(
chat_id=chat_id,
sender_id=sender_id,
text=text,
attachment_ids=attachment_ids or [],
)
self.db.add(message)
self.db.flush()
return message
def get_history(
self,
chat_id: uuid.UUID,
limit: int = 50,
cursor_created_at: datetime | None = None,
cursor_id: uuid.UUID | None = None,
) -> list[Message]:
statement = select(Message).where(
Message.chat_id == chat_id
)
if cursor_created_at is not None and cursor_id is not None:
statement = statement.where(
or_(
Message.created_at > cursor_created_at,
and_(
Message.created_at == cursor_created_at,
Message.id > cursor_id,
),
)
)
statement = (
statement
.order_by(Message.created_at.asc(), Message.id.asc())
.limit(limit)
)
return list(self.db.scalars(statement).all())
+78
View File
@@ -0,0 +1,78 @@
import uuid
from datetime import datetime, timezone
from sqlalchemy import delete, select
from sqlalchemy.orm import Session
from app.db.models import Session as UserSession
class SessionRepository:
def __init__(self, db: Session):
self.db = db
def create(
self,
user_id: uuid.UUID,
token_hash: str,
expires_at: datetime,
) -> UserSession:
session = UserSession(
user_id=user_id,
token_hash=token_hash,
expires_at=expires_at,
)
self.db.add(session)
self.db.flush()
return session
def get_by_token_hash(
self,
token_hash: str,
) -> UserSession | None:
statement = select(UserSession).where(
UserSession.token_hash == token_hash
)
return self.db.scalar(statement)
def get_valid_session(
self,
token_hash: str,
) -> UserSession | None:
now = datetime.now(timezone.utc)
statement = select(UserSession).where(
UserSession.token_hash == token_hash,
UserSession.expires_at > now,
)
return self.db.scalar(statement)
def delete_by_token_hash(
self,
token_hash: str,
) -> bool:
statement = (
delete(UserSession)
.where(UserSession.token_hash == token_hash)
.returning(UserSession.id)
)
return self.db.scalar(statement) is not None
def delete_expired(self) -> int:
now = datetime.now(timezone.utc)
statement = (
delete(UserSession)
.where(UserSession.expires_at <= now)
)
result = self.db.execute(statement)
return result.rowcount
+41
View File
@@ -0,0 +1,41 @@
import uuid
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.db.models import User
class UserRepository:
def __init__(self, db: Session):
self.db = db
def get_by_id(self, user_id: uuid.UUID) -> User | None:
statement = select(User).where(User.id == user_id)
return self.db.scalar(statement)
def get_by_username(self, username: str) -> User | None:
statement = select(User).where(
User.username == username
)
return self.db.scalar(statement)
def create(
self,
username: str,
display_name: str,
password_hash: str,
) -> User:
user = User(
username=username,
display_name=display_name,
password_hash=password_hash,
)
self.db.add(user)
self.db.flush()
return user
View File
+66
View File
@@ -0,0 +1,66 @@
"""Best-effort cleanup for abandoned presigned uploads."""
import logging
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from app.core.config import settings
from app.db.models import AttachmentUpload
from app.db.session import SessionLocal
from app.services.object_storage import client
logger = logging.getLogger(__name__)
def cleanup_expired_uploads() -> int:
"""Delete old pending objects and mark their upload records expired.
A grace period protects clients completing an upload near URL expiry.
PostgreSQL SKIP LOCKED makes this safe across multiple API replicas.
"""
cutoff = datetime.now(timezone.utc) - timedelta(
seconds=settings.attachment_cleanup_grace_seconds
)
deleted_count = 0
with SessionLocal() as db:
uploads = db.scalars(
select(AttachmentUpload)
.where(
AttachmentUpload.status == "pending",
AttachmentUpload.expires_at <= cutoff,
)
.order_by(AttachmentUpload.expires_at.asc())
.limit(settings.attachment_cleanup_batch_size)
.with_for_update(skip_locked=True)
).all()
if not uploads:
return 0
s3 = client()
try:
for upload in uploads:
# DeleteObject is idempotent: a missing object is also safe.
s3.delete_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
upload.status = "expired"
deleted_count += 1
db.commit()
except Exception:
db.rollback()
logger.exception(
"Attachment cleanup batch failed; pending records will be retried"
)
return 0
if deleted_count:
logger.info(
"Expired abandoned attachment uploads cleaned: count=%d",
deleted_count,
)
return deleted_count
+159
View File
@@ -0,0 +1,159 @@
import hashlib
import secrets
from datetime import datetime, timedelta, timezone
from redis import Redis
from redis.backoff import NoBackoff
from redis.exceptions import RedisError
from redis.retry import Retry
from sqlalchemy.orm import Session
from app.core.config import settings
from app.db.models import User
from app.repositories.session_repository import SessionRepository
from app.repositories.user_repository import UserRepository
from app.services.user_service import password_hasher
class InvalidCredentialsError(Exception):
pass
class InvalidSessionError(Exception):
pass
class AuthService:
def __init__(self, db: Session):
self.db = db
self.sessions = SessionRepository(db)
self.users = UserRepository(db)
self._redis_available = True
self.redis = Redis.from_url(
settings.cache_url,
decode_responses=True,
socket_connect_timeout=0.5,
socket_timeout=0.5,
retry_on_timeout=False,
retry=Retry(NoBackoff(), 0),
)
@staticmethod
def _hash_token(token: str) -> str:
return hashlib.sha256(token.encode("utf-8")).hexdigest()
def create_session(self, username: str, password: str) -> dict:
user = self.users.get_by_username(username)
if user is None or not password_hasher.verify(
password, user.password_hash
):
raise InvalidCredentialsError("Invalid username or password")
token = secrets.token_urlsafe(32)
token_hash = self._hash_token(token)
expires_at = datetime.now(timezone.utc) + timedelta(
seconds=settings.session_ttl_seconds
)
try:
self.sessions.create(
user_id=user.id,
token_hash=token_hash,
expires_at=expires_at,
)
self.db.commit()
except Exception:
self.db.rollback()
raise
self._cache_session(token_hash, user.id, expires_at)
return {
"token": token,
"token_type": "Bearer",
"expires_at": expires_at,
"user": user,
}
def _cache_session(
self,
token_hash: str,
user_id,
expires_at,
) -> None:
if not self._redis_available:
return
ttl = max(
1,
int((expires_at - datetime.now(timezone.utc)).total_seconds()),
)
try:
self.redis.setex(
f"session:{token_hash}",
ttl,
str(user_id),
)
except RedisError:
self._redis_available = False
def authenticate(self, token: str) -> User:
token_hash = self._hash_token(token)
cached_user_id = None
if self._redis_available:
try:
cached_user_id = self.redis.get(
f"session:{token_hash}"
)
except RedisError:
self._redis_available = False
session = self.sessions.get_valid_session(token_hash)
if session is None:
if self._redis_available:
try:
self.redis.delete(f"session:{token_hash}")
except RedisError:
self._redis_available = False
raise InvalidSessionError(
"Session is invalid or expired"
)
if cached_user_id is None:
self._cache_session(
token_hash,
session.user_id,
session.expires_at,
)
return session.user
def delete_session(self, token: str) -> None:
token_hash = self._hash_token(token)
try:
deleted = self.sessions.delete_by_token_hash(token_hash)
if not deleted:
raise InvalidSessionError(
"Session is invalid or expired"
)
self.db.commit()
except Exception:
self.db.rollback()
raise
if self._redis_available:
try:
self.redis.delete(f"session:{token_hash}")
except RedisError:
self._redis_available = False
+94
View File
@@ -0,0 +1,94 @@
import uuid
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.db.models import Chat
from app.repositories.chat_repository import ChatRepository
from app.repositories.user_repository import UserRepository
class ChatNotFoundError(Exception):
pass
class ChatAccessError(Exception):
pass
class InvalidChatDataError(Exception):
pass
class ChatService:
def __init__(self, db: Session):
self.db = db
self.chats = ChatRepository(db)
self.users = UserRepository(db)
def create_chat(
self,
title: str,
member_ids: list[str],
creator_id: uuid.UUID,
) -> Chat:
title = title.strip()
if not title or len(title) > 100:
raise InvalidChatDataError("Invalid chat title")
if not member_ids or len(member_ids) > 100:
raise InvalidChatDataError("Invalid members")
try:
parsed_ids = [uuid.UUID(str(value)) for value in member_ids]
except (ValueError, TypeError, AttributeError):
raise InvalidChatDataError("Invalid member UUID") from None
if len(parsed_ids) != len(set(parsed_ids)):
raise InvalidChatDataError("Duplicate members")
if creator_id not in parsed_ids:
raise InvalidChatDataError("Creator must be a member")
for user_id in parsed_ids:
if self.users.get_by_id(user_id) is None:
raise InvalidChatDataError("Unknown member")
try:
chat = self.chats.create(
title=title,
member_ids=parsed_ids,
)
self.db.commit()
self.db.refresh(chat)
return chat
except IntegrityError as exc:
self.db.rollback()
raise InvalidChatDataError("Unable to create chat") from exc
except Exception:
self.db.rollback()
raise
def get_chat(self, chat_id: uuid.UUID, user_id: uuid.UUID) -> Chat:
chat = self.chats.get_by_id(chat_id)
if chat is None:
raise ChatNotFoundError()
if not self.chats.is_member(chat_id, user_id):
raise ChatNotFoundError()
return chat
def list_chats(self, user_id: uuid.UUID) -> list[Chat]:
return self.chats.get_by_user(user_id)
def check_membership(
self,
chat_id: uuid.UUID,
user_id: uuid.UUID,
) -> bool:
return self.chats.is_member(chat_id, user_id)
+281
View File
@@ -0,0 +1,281 @@
"""Deterministic image variants for the attachment pipeline."""
import hashlib
import io
import multiprocessing
import time
import warnings
from dataclasses import dataclass
from PIL import Image, ImageOps, UnidentifiedImageError
from app.core.config import settings
@dataclass(frozen=True)
class ProcessedVariant:
name: str
content_type: str
extension: str
width: int
height: int
data: bytes
sha256: str
class ImageProcessingTimeout(TimeoutError):
"""Raised when image processing exceeds its configured deadline."""
def _load_oriented(data: bytes) -> Image.Image:
if len(data) > settings.max_upload_bytes:
raise ValueError("image_too_large")
Image.MAX_IMAGE_PIXELS = settings.max_image_pixels
with warnings.catch_warnings():
warnings.simplefilter("error", Image.DecompressionBombWarning)
try:
source = Image.open(io.BytesIO(data))
if source.format not in {"JPEG", "PNG"}:
raise ValueError("unsupported_image")
source.verify()
source = Image.open(io.BytesIO(data))
image = ImageOps.exif_transpose(source)
image.load()
except (
UnidentifiedImageError,
OSError,
Image.DecompressionBombError,
Image.DecompressionBombWarning,
):
raise ValueError("invalid_image") from None
if image.width * image.height > settings.max_image_pixels:
raise ValueError("image_too_many_pixels")
return image
def _rgb_white(image: Image.Image) -> Image.Image:
if image.mode in ("RGBA", "LA") or (
image.mode == "P" and "transparency" in image.info
):
rgba = image.convert("RGBA")
background = Image.new(
"RGBA",
rgba.size,
(255, 255, 255, 255),
)
return Image.alpha_composite(background, rgba).convert("RGB")
return image.convert("RGB")
def _fit(
image: Image.Image,
box: tuple[int, int],
allow_upscale: bool = False,
) -> Image.Image:
image = image.copy()
image.thumbnail(box, Image.Resampling.LANCZOS)
if allow_upscale:
scale = min(
box[0] / image.width,
box[1] / image.height,
)
if scale > 1:
image = image.resize(
(
max(1, int(image.width * scale)),
max(1, int(image.height * scale)),
),
Image.Resampling.LANCZOS,
)
return image
def _encode(
name: str,
image: Image.Image,
fmt: str,
content_type: str,
extension: str,
**kwargs,
) -> ProcessedVariant:
output = io.BytesIO()
image.save(output, format=fmt, **kwargs)
data = output.getvalue()
return ProcessedVariant(
name=name,
content_type=content_type,
extension=extension,
width=image.width,
height=image.height,
data=data,
sha256=hashlib.sha256(data).hexdigest(),
)
def create_variants(original: bytes) -> list[ProcessedVariant]:
source = _load_oriented(original)
rgb = _rgb_white(source)
# 256x256 fit; never upscale; dimensions retain aspect ratio.
thumb = _fit(rgb, (256, 256))
thumbnail = _encode(
"thumbnail",
thumb,
"JPEG",
"image/jpeg",
"jpg",
quality=85,
optimize=True,
progressive=True,
)
# Center crop to square, 128x128. Upscaling is allowed by contract.
side = min(rgb.width, rgb.height)
left = (rgb.width - side) // 2
top = (rgb.height - side) // 2
crop = rgb.crop((left, top, left + side, top + side))
crop = crop.resize((128, 128), Image.Resampling.LANCZOS)
cropped = _encode(
"crop",
crop,
"JPEG",
"image/jpeg",
"jpg",
quality=85,
optimize=True,
progressive=True,
)
# Optimized WebP fit 1280x1280; never upscale.
webp_image = _fit(rgb, (1280, 1280))
webp = _encode(
"optimized",
webp_image,
"WEBP",
"image/webp",
"webp",
quality=82,
method=6,
)
return [thumbnail, cropped, webp]
def _variant_process_entry(connection, original: bytes) -> None:
"""Run Pillow processing in an isolated child process."""
try:
result = create_variants(original)
connection.send(("ok", result))
except ValueError as exc:
connection.send(("validation_error", str(exc)[:100]))
except Exception:
connection.send(("processing_error", "processing_failed"))
finally:
connection.close()
def create_variants_with_timeout(
original: bytes,
timeout_seconds: float,
) -> list[ProcessedVariant]:
"""Run image processing in a child process with a hard deadline."""
if timeout_seconds <= 0:
raise ValueError("invalid_processing_timeout")
context = multiprocessing.get_context("spawn")
receive_connection, send_connection = context.Pipe(duplex=False)
process = context.Process(
target=_variant_process_entry,
args=(send_connection, original),
name="image-processor",
)
try:
process.start()
send_connection.close()
deadline = time.monotonic() + timeout_seconds
message = None
while time.monotonic() < deadline:
if receive_connection.poll(0.1):
try:
message = receive_connection.recv()
except EOFError:
message = None
break
if not process.is_alive():
if receive_connection.poll():
try:
message = receive_connection.recv()
except EOFError:
message = None
break
if message is None and process.is_alive():
process.terminate()
process.join(timeout=2)
if process.is_alive():
process.kill()
process.join()
raise ImageProcessingTimeout(
"image_processing_timeout"
)
process.join(timeout=2)
if process.is_alive():
process.terminate()
process.join(timeout=2)
if message is None:
raise RuntimeError("image_processing_failed")
status, payload = message
if status == "ok":
return payload
if status == "validation_error":
raise ValueError(payload)
raise RuntimeError("image_processing_failed")
finally:
if process.is_alive():
process.terminate()
process.join(timeout=2)
if process.is_alive():
process.kill()
process.join()
receive_connection.close()
try:
send_connection.close()
except OSError:
pass
+41
View File
@@ -0,0 +1,41 @@
from io import BytesIO
from PIL import Image, UnidentifiedImageError
from fastapi import HTTPException
MAX_BYTES = 10 * 1024 * 1024
MAX_PIXELS = 40_000_000
ALLOWED = {"PNG": "image/png", "JPEG": "image/jpeg"}
def validate_image_file(fileobj, declared_type: str):
"""Validate a seekable, size-limited file without loading its full encoded bytes."""
try:
fileobj.seek(0)
with Image.open(fileobj) as image:
fmt = image.format
width, height = image.size
if fmt not in ALLOWED:
raise HTTPException(415, "Only PNG and JPEG are supported")
if width * height > MAX_PIXELS:
raise HTTPException(413, "Image exceeds 40 million pixels")
image.verify()
fileobj.seek(0)
with Image.open(fileobj) as image:
image.load()
except HTTPException:
raise
except (UnidentifiedImageError, OSError, ValueError, SyntaxError, Image.DecompressionBombError):
raise HTTPException(415, "Invalid or corrupted image") from None
actual_type = ALLOWED[fmt]
if declared_type != actual_type:
raise HTTPException(415, "Content-Type does not match image content")
fileobj.seek(0)
return actual_type, width, height
def validate_image(data: bytes, declared_type: str):
if not data:
raise HTTPException(415, "Empty image")
if len(data) > MAX_BYTES:
raise HTTPException(413, "Image exceeds 10 MiB")
return validate_image_file(BytesIO(data), declared_type)
+444
View File
@@ -0,0 +1,444 @@
import base64
import hashlib
import json
import time
import uuid
from datetime import datetime, timedelta, timezone
from types import SimpleNamespace
from redis import Redis
from redis.backoff import NoBackoff
from redis.exceptions import RedisError
from redis.retry import Retry
from sqlalchemy import delete, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app.core.config import settings
from app.db.models import Attachment, IdempotencyRecord, Message
from app.repositories.chat_repository import ChatRepository
from app.repositories.message_repository import MessageRepository
class MessageNotFoundError(Exception):
pass
class MessageAccessError(Exception):
pass
class InvalidMessageDataError(Exception):
pass
class IdempotencyConflictError(Exception):
pass
class MessageService:
def __init__(self, db: Session):
self.db = db
self.messages = MessageRepository(db)
self.chats = ChatRepository(db)
self._redis_available = True
self.redis = Redis.from_url(
settings.cache_url,
decode_responses=True,
socket_connect_timeout=0.5,
socket_timeout=0.5,
retry_on_timeout=False,
retry=Retry(NoBackoff(), 0),
)
@staticmethod
def encode_cursor(message: Message) -> str:
payload = {
"created_at": message.created_at.isoformat(),
"id": str(message.id),
}
raw = json.dumps(payload).encode("utf-8")
return (
base64.urlsafe_b64encode(raw)
.decode("ascii")
.rstrip("=")
)
@staticmethod
def decode_cursor(cursor: str | None):
if not cursor:
return None, None
try:
padded = cursor + "=" * (-len(cursor) % 4)
raw = base64.urlsafe_b64decode(padded.encode("ascii"))
payload = json.loads(raw.decode("utf-8"))
created_at = datetime.fromisoformat(payload["created_at"])
message_id = uuid.UUID(payload["id"])
if created_at.tzinfo is None:
raise ValueError("Cursor timestamp must include timezone")
return created_at, message_id
except (ValueError, TypeError, KeyError, UnicodeError):
raise InvalidMessageDataError("Invalid cursor") from None
def _version_key(self, chat_id: uuid.UUID) -> str:
return f"messages:version:{chat_id}"
def _cache_key(
self,
chat_id: uuid.UUID,
version: int,
limit: int,
cursor: str | None,
) -> str:
cursor_hash = hashlib.sha256(
(cursor or "first-page").encode("utf-8")
).hexdigest()[:16]
return (
f"messages:{chat_id}:v{version}:"
f"limit:{limit}:cursor:{cursor_hash}"
)
def _get_version(self, chat_id: uuid.UUID) -> int:
if not self._redis_available:
return 0
try:
value = self.redis.get(self._version_key(chat_id))
return int(value or 0)
except (RedisError, ValueError):
self._redis_available = False
return 0
def _bump_version(self, chat_id: uuid.UUID) -> None:
if not self._redis_available:
return
try:
self.redis.incr(self._version_key(chat_id))
except RedisError:
self._redis_available = False
@staticmethod
def _serialize(messages, next_cursor):
return json.dumps({
"items": [
{
"id": str(message.id),
"chat_id": str(message.chat_id),
"sender_id": str(message.sender_id),
"text": message.text,
"attachment_ids": message.attachment_ids or [],
"created_at": message.created_at.isoformat(),
}
for message in messages
],
"next_cursor": next_cursor,
})
@staticmethod
def _deserialize(payload: str):
data = json.loads(payload)
messages = [
SimpleNamespace(
id=uuid.UUID(item["id"]),
chat_id=uuid.UUID(item["chat_id"]),
sender_id=uuid.UUID(item["sender_id"]),
text=item["text"],
attachment_ids=item.get("attachment_ids", []),
created_at=datetime.fromisoformat(item["created_at"]),
)
for item in data["items"]
]
return messages, data["next_cursor"]
def _read_cache(self, key: str):
if not self._redis_available:
return None
try:
payload = self.redis.get(key)
if payload is None:
return None
return self._deserialize(payload)
except RedisError:
self._redis_available = False
return None
except (ValueError, TypeError, KeyError):
return None
def _load_from_database(
self,
chat_id,
limit,
cursor_created_at,
cursor_id,
):
rows = self.messages.get_history(
chat_id=chat_id,
limit=limit + 1,
cursor_created_at=cursor_created_at,
cursor_id=cursor_id,
)
has_more = len(rows) > limit
items = rows[:limit]
next_cursor = (
self.encode_cursor(items[-1])
if has_more and items
else None
)
return items, next_cursor
def create_message(
self,
chat_id: uuid.UUID,
sender_id: uuid.UUID,
text: str,
idempotency_key: str,
attachment_ids: list[str] | None = None,
) -> Message:
text = text.strip()
attachment_ids = attachment_ids or []
if (not text and not attachment_ids) or len(text) > 4000 or len(attachment_ids) > 10:
raise InvalidMessageDataError("Invalid message text")
# Access is checked for every retry, before consulting the key.
if not self.chats.is_member(chat_id, sender_id):
raise MessageNotFoundError()
if len(attachment_ids) != len(set(attachment_ids)):
raise InvalidMessageDataError("Duplicate attachment IDs")
if attachment_ids:
try:
parsed_attachment_ids = [uuid.UUID(value) for value in attachment_ids]
except (ValueError, TypeError):
raise InvalidMessageDataError("Invalid attachment UUID") from None
attachments = list(
self.db.scalars(
select(Attachment).where(
Attachment.id.in_(parsed_attachment_ids)
)
).all()
)
if (
len(attachments) != len(parsed_attachment_ids)
or any(attachment.chat_id != chat_id for attachment in attachments)
):
raise MessageNotFoundError()
method = "POST"
path = f"/api/v1/chats/{chat_id}/messages"
request_hash = hashlib.sha256(
json.dumps({"text": text, "attachment_ids": attachment_ids}, sort_keys=True, separators=(",", ":")).encode("utf-8")
).hexdigest()
now = datetime.now(timezone.utc)
expires_at = now + timedelta(seconds=max(settings.idempotency_ttl_seconds, 86400))
scope = (
IdempotencyRecord.user_id == sender_id,
IdempotencyRecord.method == method,
IdempotencyRecord.path == path,
IdempotencyRecord.idempotency_key == idempotency_key,
)
try:
# An expired key may be reused. Deletion and claim are serialized by
# PostgreSQL's unique constraint for the scope.
self.db.execute(
delete(IdempotencyRecord).where(*scope, IdempotencyRecord.expires_at <= now)
)
claim = (
pg_insert(IdempotencyRecord)
.values(
id=uuid.uuid4(), user_id=sender_id, method=method, path=path,
idempotency_key=idempotency_key, request_hash=request_hash,
message_id=None, created_at=now, expires_at=expires_at,
)
.on_conflict_do_nothing(constraint="uq_idempotency_scope_key")
.returning(IdempotencyRecord.id)
)
claimed_id = self.db.execute(claim).scalar_one_or_none()
if claimed_id is None:
existing = self.db.execute(
select(IdempotencyRecord).where(*scope).with_for_update()
).scalar_one_or_none()
if existing is None:
# Rare expiration race: retry once in a fresh transaction.
self.db.rollback()
return self.create_message(chat_id, sender_id, text, idempotency_key, attachment_ids)
if existing.request_hash != request_hash:
self.db.rollback()
raise IdempotencyConflictError()
if existing.message_id is None:
raise RuntimeError("Committed idempotency record has no message")
message = self.db.get(Message, existing.message_id)
if message is None:
raise RuntimeError("Idempotency record references a missing message")
self.db.commit()
return message
message = self.messages.create(
chat_id=chat_id, sender_id=sender_id, text=text, attachment_ids=attachment_ids
)
self.db.execute(
update(IdempotencyRecord)
.where(IdempotencyRecord.id == claimed_id)
.values(message_id=message.id)
)
self.db.commit()
self.db.refresh(message)
self._bump_version(chat_id)
return message
except IdempotencyConflictError:
raise
except Exception:
self.db.rollback()
raise
def get_history(
self,
chat_id: uuid.UUID,
user_id: uuid.UUID,
limit: int = 50,
cursor: str | None = None,
):
if not 1 <= limit <= 100:
raise InvalidMessageDataError(
"Limit must be between 1 and 100"
)
if not self.chats.is_member(chat_id, user_id):
raise MessageNotFoundError()
cursor_created_at, cursor_id = self.decode_cursor(cursor)
version = self._get_version(chat_id)
cache_key = self._cache_key(
chat_id,
version,
limit,
cursor,
)
cached = self._read_cache(cache_key)
if cached is not None:
return cached
if not self._redis_available:
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
lock_key = f"lock:{cache_key}"
lock_token = str(uuid.uuid4())
try:
acquired = self.redis.set(
lock_key,
lock_token,
nx=True,
ex=10,
)
except RedisError:
self._redis_available = False
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
if not acquired:
for _ in range(10):
time.sleep(0.05)
cached = self._read_cache(cache_key)
if cached is not None:
return cached
if not self._redis_available:
break
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
try:
cached = self._read_cache(cache_key)
if cached is not None:
return cached
if not self._redis_available:
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
items, next_cursor = self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
if self._redis_available and self._get_version(chat_id) == version:
try:
self.redis.setex(
cache_key,
settings.message_cache_ttl_seconds,
self._serialize(items, next_cursor),
)
except RedisError:
self._redis_available = False
return items, next_cursor
finally:
if self._redis_available:
try:
self.redis.eval(
"""
if redis.call('get', KEYS[1]) == ARGV[1]
then
return redis.call('del', KEYS[1])
else
return 0
end
""",
1,
lock_key,
lock_token,
)
except RedisError:
self._redis_available = False
+39
View File
@@ -0,0 +1,39 @@
import boto3
from botocore.config import Config
from app.core.config import settings
def client(endpoint_url=None):
return boto3.client(
"s3", endpoint_url=endpoint_url or settings.s3_endpoint_url,
aws_access_key_id=settings.s3_access_key,
aws_secret_access_key=settings.s3_secret_key,
region_name=settings.s3_region,
config=Config(signature_version="s3v4", connect_timeout=3, read_timeout=15, retries={"max_attempts": 2}, s3={"addressing_style": "path"}),
)
def ensure_bucket() -> None:
s3 = client()
try:
s3.head_bucket(Bucket=settings.s3_bucket)
except Exception:
s3.create_bucket(Bucket=settings.s3_bucket)
def presigned_client():
return client(endpoint_url=settings.s3_public_endpoint_url)
def create_presigned_upload_url(object_key: str, content_type: str) -> str:
s3 = presigned_client()
return s3.generate_presigned_url(
ClientMethod="put_object",
Params={
"Bucket": settings.s3_bucket,
"Key": object_key,
"ContentType": content_type,
},
ExpiresIn=settings.presigned_upload_ttl_seconds,
)
+102
View File
@@ -0,0 +1,102 @@
"""Publish durable PostgreSQL outbox events to RabbitMQ.
The database row is marked published only after RabbitMQ publisher confirmation.
If the process crashes after broker confirmation but before the DB commit, the
same event is published again; consumers must therefore be idempotent.
"""
import asyncio
import json
import logging
import random
from datetime import datetime, timezone
from typing import Any
import aio_pika
from aio_pika import DeliveryMode, Message
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.core.config import settings
from app.db.models import OutboxEvent
from app.db.session import SessionLocal
logger = logging.getLogger(__name__)
def utc_now() -> datetime:
return datetime.now(timezone.utc)
async def publish_once() -> int:
"""Publish one batch; return number of confirmed events."""
connection = await aio_pika.connect_robust(settings.rabbitmq_url)
try:
channel = await connection.channel(publisher_confirms=True)
exchange = await channel.declare_exchange(
settings.rabbitmq_exchange,
aio_pika.ExchangeType.DIRECT,
durable=True,
)
queue = await channel.declare_queue(settings.rabbitmq_queue, durable=True)
await queue.bind(exchange, routing_key=settings.rabbitmq_queue)
published = 0
db: Session = SessionLocal()
try:
# Lock a bounded batch. SKIP LOCKED allows multiple publisher replicas.
events = db.scalars(
select(OutboxEvent)
.where(OutboxEvent.status == "pending")
.order_by(OutboxEvent.created_at)
.with_for_update(skip_locked=True)
.limit(settings.outbox_batch_size)
).all()
for event in events:
body = json.dumps(event.payload, separators=(",", ":")).encode("utf-8")
message = Message(
body=body,
content_type="application/json",
delivery_mode=DeliveryMode.PERSISTENT,
message_id=str(event.id),
type=event.event_type,
timestamp=utc_now(),
)
await exchange.publish(message, routing_key=settings.rabbitmq_queue, mandatory=True)
event.status = "published"
event.published_at = utc_now()
event.attempts += 1
published += 1
db.commit()
return published
except Exception:
db.rollback()
raise
finally:
db.close()
finally:
await connection.close()
async def run_publisher() -> None:
delay = settings.outbox_poll_interval_seconds
while True:
try:
count = await publish_once()
if count:
logger.info("Published %s outbox events", count)
delay = settings.outbox_poll_interval_seconds
except asyncio.CancelledError:
raise
except Exception:
logger.exception("Outbox publisher iteration failed")
delay = min(max(delay * 2, settings.outbox_poll_interval_seconds), 15.0)
delay += random.uniform(0, delay * 0.2)
await asyncio.sleep(delay)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
asyncio.run(run_publisher())
+81
View File
@@ -0,0 +1,81 @@
import re
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from pwdlib import PasswordHash
from app.db.models import User
from app.repositories.user_repository import UserRepository
password_hasher = PasswordHash.recommended()
class UsernameAlreadyExistsError(Exception):
pass
class UserService:
def __init__(self, db: Session):
self.db = db
self.users = UserRepository(db)
def register(
self,
username: str,
display_name: str,
password: str,
) -> User:
if not re.fullmatch(r"[a-z0-9_]{3,32}", username):
raise ValueError("Invalid username")
if not display_name or not display_name.strip():
raise ValueError("Display name cannot be empty")
if len(display_name) > 100:
raise ValueError("Display name is too long")
if not 12 <= len(password) <= 128:
raise ValueError("Invalid password length")
if self.users.get_by_username(username) is not None:
raise UsernameAlreadyExistsError(
"Username already exists"
)
password_hash = password_hasher.hash(password)
try:
user = self.users.create(
username=username,
display_name=display_name,
password_hash=password_hash,
)
self.db.commit()
self.db.refresh(user)
return user
except IntegrityError as exc:
self.db.rollback()
raise UsernameAlreadyExistsError(
"Username already exists"
) from exc
except Exception:
self.db.rollback()
raise
def verify_password(
self,
user: User,
password: str,
) -> bool:
return password_hasher.verify(
password,
user.password_hash,
)
+193
View File
@@ -0,0 +1,193 @@
import base64
import bisect
import binascii
import json
import os
import threading
import uuid
from datetime import datetime, timezone
from typing import Dict, List, Optional, Tuple
from .models import Chat, Message, User
Key = Tuple[datetime, str]
def _encode_cursor(kind: str, scope: str, item) -> str:
payload = {
"v": 1,
"kind": kind,
"scope": scope,
"created_at": item.created_at.isoformat(),
"id": item.id,
}
raw = json.dumps(payload, separators=(",", ":")).encode("utf-8")
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
def _decode_cursor(cursor: str, kind: str, scope: str) -> Key:
try:
if not cursor or len(cursor) > 4096:
raise ValueError
padded = cursor + "=" * (-len(cursor) % 4)
raw = base64.b64decode(padded, altchars=b"-_", validate=True)
payload = json.loads(raw.decode("utf-8"))
if not isinstance(payload, dict) or payload.get("v") != 1:
raise ValueError
if payload.get("kind") != kind or payload.get("scope") != scope:
raise ValueError
item_id = payload.get("id")
timestamp = payload.get("created_at")
if not isinstance(item_id, str) or not isinstance(timestamp, str):
raise ValueError
created_at = datetime.fromisoformat(timestamp)
if created_at.tzinfo is None:
raise ValueError
created_at = created_at.astimezone(timezone.utc)
return created_at, item_id
except (
ValueError,
TypeError,
KeyError,
UnicodeDecodeError,
json.JSONDecodeError,
binascii.Error,
):
raise ValueError("invalid_request") from None
class InMemoryStore:
def __init__(self):
self._lock = threading.RLock()
self.users: Dict[str, User] = {}
self.chats: Dict[str, Chat] = {}
self.messages: Dict[str, List[Message]] = {}
self.usernames: Dict[str, str] = {}
self.message_keys: Dict[str, List[Key]] = {}
self.message_index_enabled = os.getenv("MESSAGE_INDEX_ENABLED", "1") != "0"
def create_user(self, username: str, display_name: str) -> User:
with self._lock:
if username in self.usernames:
raise ValueError("conflict")
user_id = str(uuid.uuid4())
user = User(
id=user_id,
username=username,
display_name=display_name,
created_at=datetime.now(timezone.utc),
)
self.users[user_id] = user
self.usernames[username] = user_id
return user
def get_user(self, user_id: str) -> Optional[User]:
with self._lock:
return self.users.get(user_id)
def create_chat(self, title: str, member_ids: List[str], creator_id: str) -> Chat:
with self._lock:
if creator_id not in member_ids or len(member_ids) != len(set(member_ids)):
raise ValueError("invalid_request")
if any(member_id not in self.users for member_id in member_ids):
raise ValueError("invalid_request")
chat_id = str(uuid.uuid4())
chat = Chat(
id=chat_id,
title=title,
member_ids=list(member_ids),
created_at=datetime.now(timezone.utc),
)
self.chats[chat_id] = chat
self.messages[chat_id] = []
self.message_keys[chat_id] = []
return chat
def get_chat(self, chat_id: str, user_id: str) -> Optional[Chat]:
with self._lock:
chat = self.chats.get(chat_id)
if chat is None or user_id not in chat.member_ids:
return None
return chat
def list_chats(
self, user_id: str, limit: int, cursor: Optional[str]
) -> Tuple[List[Chat], Optional[str]]:
with self._lock:
user_chats = [chat for chat in self.chats.values() if user_id in chat.member_ids]
user_chats.sort(key=lambda chat: (chat.created_at, chat.id))
keys = [(chat.created_at, chat.id) for chat in user_chats]
start_index = 0
if cursor:
target_key = _decode_cursor(cursor, kind="chats", scope=user_id)
start_index = bisect.bisect_right(keys, target_key)
page_with_extra = user_chats[start_index : start_index + limit + 1]
has_next = len(page_with_extra) > limit
page = page_with_extra[:limit]
next_cursor = None
if has_next and page:
next_cursor = _encode_cursor("chats", user_id, page[-1])
return page, next_cursor
def create_message(self, chat_id: str, sender_id: str, text: str) -> Message:
with self._lock:
chat = self.chats.get(chat_id)
if chat is None or sender_id not in chat.member_ids:
raise ValueError("not_found")
message = Message(
id=str(uuid.uuid4()),
chat_id=chat_id,
sender_id=sender_id,
text=text,
created_at=datetime.now(timezone.utc),
)
messages = self.messages[chat_id]
keys = self.message_keys[chat_id]
key = (message.created_at, message.id)
if self.message_index_enabled:
index = bisect.bisect_right(keys, key)
keys.insert(index, key)
messages.insert(index, message)
else:
current_keys = [(item.created_at, item.id) for item in messages]
index = bisect.bisect_right(current_keys, key)
messages.insert(index, message)
keys.insert(index, key)
return message
def list_messages(
self,
chat_id: str,
user_id: str,
limit: int,
cursor: Optional[str],
) -> Tuple[List[Message], Optional[str]]:
with self._lock:
chat = self.chats.get(chat_id)
if chat is None or user_id not in chat.member_ids:
raise ValueError("not_found")
messages = self.messages[chat_id]
if self.message_index_enabled:
keys = self.message_keys[chat_id]
else:
keys = [(item.created_at, item.id) for item in messages]
start_index = 0
if cursor:
target_key = _decode_cursor(cursor, kind="messages", scope=chat_id)
start_index = bisect.bisect_right(keys, target_key)
page_with_extra = messages[start_index : start_index + limit + 1]
has_next = len(page_with_extra) > limit
page = page_with_extra[:limit]
next_cursor = None
if has_next and page:
next_cursor = _encode_cursor("messages", chat_id, page[-1])
return page, next_cursor
+995
View File
@@ -0,0 +1,995 @@
"""RabbitMQ consumer process for durable image processing."""
import asyncio
import json
import logging
import random
import socket
import threading
import uuid
import time
from datetime import datetime, timedelta, timezone
import aio_pika
from aio_pika import DeliveryMode, Message
from botocore.exceptions import BotoCoreError, ClientError
from sqlalchemy import select, update
from sqlalchemy.exc import SQLAlchemyError
from app.core.config import settings
from app.db.models import Attachment, AttachmentVariant, ProcessingJob
from app.db.session import SessionLocal
from app.services.image_processor import (
ImageProcessingTimeout,
create_variants_with_timeout,
)
from app.services.object_storage import client as s3_client
from app.core.worker_telemetry import (
process_job_with_telemetry,
setup_worker_telemetry,
)
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
logger = logging.getLogger("attachment-worker")
WORKER_ID = (
f"{socket.gethostname()[:40]}-{uuid.uuid4().hex[:16]}"
)
DEAD_ROUTING_KEY = f"{settings.rabbitmq_queue}.dead"
PROCESSING_SEMAPHORE = None
def utc_now() -> datetime:
return datetime.now(timezone.utc)
def _safe_uuid(value) -> uuid.UUID:
return uuid.UUID(str(value))
def _claim_job(job_id, attachment_id, pipeline_version) -> dict:
"""Atomically claim a job or return the reason it cannot be claimed."""
job_id = _safe_uuid(job_id)
attachment_id = _safe_uuid(attachment_id)
pipeline_version = int(pipeline_version)
db = SessionLocal()
try:
job = db.scalar(
select(ProcessingJob)
.where(
ProcessingJob.id == job_id,
ProcessingJob.attachment_id == attachment_id,
ProcessingJob.pipeline_version == pipeline_version,
)
.with_for_update()
)
if job is None:
db.rollback()
return {"state": "missing"}
attachment = db.get(Attachment, attachment_id)
if attachment is None:
db.rollback()
return {"state": "missing"}
now = utc_now()
if attachment.status == "ready":
db.commit()
return {"state": "ready"}
if job.status == "failed":
db.commit()
return {"state": "failed"}
if (
job.status == "processing"
and job.lease_until is not None
and job.lease_until > now
):
delay = (job.lease_until - now).total_seconds() + 1
db.commit()
return {
"state": "busy",
"delay": max(1.0, delay),
}
if job.available_at > now:
delay = (job.available_at - now).total_seconds() + 0.5
db.commit()
return {
"state": "deferred",
"delay": max(1.0, delay),
}
if job.attempt >= settings.worker_max_attempts:
job.status = "failed"
job.worker_id = None
job.lease_until = None
job.last_error_code = "processing_attempts_exhausted"
attachment.status = "failed"
attachment.error_code = "processing_attempts_exhausted"
db.commit()
return {"state": "failed"}
claim_worker_id = f"{WORKER_ID}:{uuid.uuid4().hex[:16]}"
job.status = "processing"
job.worker_id = claim_worker_id
job.lease_until = now + timedelta(
seconds=settings.worker_lease_seconds
)
job.attempt += 1
attachment.status = "processing"
attachment.error_code = None
claimed_attempt = job.attempt
queue_lag_ms = max(
0.0,
(now - job.available_at).total_seconds() * 1000,
)
db.commit()
logger.info(
"Claimed job=%s attachment=%s attempt=%s queue_lag_ms=%.2f",
job_id,
attachment_id,
claimed_attempt,
queue_lag_ms,
)
return {
"state": "claimed",
"worker_id": claim_worker_id,
"attempt": claimed_attempt,
"queue_lag_ms": queue_lag_ms,
}
except Exception:
db.rollback()
raise
finally:
db.close()
def _renew_lease(
job_id: uuid.UUID,
worker_id: str,
attempt: int,
stop_event: threading.Event,
lost_event: threading.Event,
) -> None:
"""Renew the lease while image processing is running."""
interval = max(
5,
settings.worker_lease_seconds // 3,
)
while not stop_event.wait(interval):
db = SessionLocal()
try:
result = db.execute(
update(ProcessingJob)
.where(
ProcessingJob.id == job_id,
ProcessingJob.worker_id == worker_id,
ProcessingJob.attempt == attempt,
ProcessingJob.status == "processing",
)
.values(
lease_until=utc_now()
+ timedelta(seconds=settings.worker_lease_seconds)
)
)
db.commit()
if result.rowcount != 1:
lost_event.set()
logger.warning(
"Lease ownership lost: job=%s worker=%s",
job_id,
worker_id,
)
return
except Exception:
db.rollback()
lost_event.set()
logger.exception(
"Could not renew lease for job=%s",
job_id,
)
return
finally:
db.close()
def _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
error_code,
permanent=False,
) -> dict:
"""Record a failure only if this worker still owns the job."""
db = SessionLocal()
try:
job = db.scalar(
select(ProcessingJob)
.where(ProcessingJob.id == job_id)
.with_for_update()
)
attachment = db.scalar(
select(Attachment)
.where(Attachment.id == attachment_id)
.with_for_update()
)
if job is None or attachment is None:
db.rollback()
return {"state": "missing"}
if (
job.status != "processing"
or job.worker_id != worker_id
or job.attempt != attempt
):
db.rollback()
logger.warning(
"Ignoring failure from stale worker: job=%s worker=%s",
job_id,
worker_id,
)
return {
"state": "stale",
"delay": float(settings.worker_lease_seconds),
}
exhausted = job.attempt >= settings.worker_max_attempts
job.last_error_code = error_code
if permanent or exhausted:
job.status = "failed"
job.worker_id = None
job.lease_until = None
attachment.status = "failed"
attachment.error_code = error_code
db.commit()
logger.error(
"Job permanently failed: job=%s error=%s",
job_id,
error_code,
)
return {"state": "failed"}
delay = min(
settings.worker_retry_base_seconds
* (2 ** max(job.attempt - 1, 0)),
settings.worker_retry_max_seconds,
)
delay += random.uniform(0, delay * 0.25)
job.status = "queued"
job.available_at = utc_now() + timedelta(seconds=delay)
job.lease_until = None
job.worker_id = None
attachment.status = "queued"
attachment.error_code = error_code
db.commit()
logger.warning(
"Job scheduled for retry: job=%s delay=%.2fs error=%s",
job_id,
delay,
error_code,
)
return {
"state": "retry",
"delay": delay,
}
except Exception:
db.rollback()
raise
finally:
db.close()
def _process_impl(payload: dict) -> dict:
"""Process one image using a fenced, renewable job lease."""
job_id = _safe_uuid(payload["job_id"])
attachment_id = _safe_uuid(payload["attachment_id"])
pipeline_version = int(payload["pipeline_version"])
claim = _claim_job(
job_id,
attachment_id,
pipeline_version,
)
if claim["state"] != "claimed":
return claim
worker_id = claim["worker_id"]
attempt = claim["attempt"]
queue_lag_ms = claim["queue_lag_ms"]
processing_started_at = time.perf_counter()
stop_event = threading.Event()
lost_event = threading.Event()
heartbeat = threading.Thread(
target=_renew_lease,
args=(
job_id,
worker_id,
attempt,
stop_event,
lost_event,
),
daemon=True,
)
heartbeat.start()
staging_key = None
db = None
try:
db = SessionLocal()
attachment = db.get(Attachment, attachment_id)
if attachment is None:
raise ValueError("attachment_missing")
staging_key = attachment.staging_key
object_key = attachment.object_key
content_type = attachment.content_type
if not staging_key:
raise ValueError("staging_missing")
db.close()
db = None
if lost_event.is_set():
return {
"state": "retry",
"delay": float(settings.worker_lease_seconds),
}
s3 = s3_client()
response = s3.get_object(
Bucket=settings.s3_bucket,
Key=staging_key,
)
body = response["Body"]
try:
attachment_data = body.read(
settings.max_upload_bytes + 1
)
finally:
body.close()
if len(attachment_data) > settings.max_upload_bytes:
raise ValueError("image_too_large")
if lost_event.is_set():
return {
"state": "retry",
"delay": float(settings.worker_lease_seconds),
}
variants = create_variants_with_timeout(
attachment_data,
settings.image_processing_timeout_seconds,
)
# The original object uses a deterministic key.
s3.put_object(
Bucket=settings.s3_bucket,
Key=object_key,
Body=attachment_data,
ContentType=content_type,
)
variant_records = []
for variant in variants:
key = (
f"variants/{attachment_id}/"
f"v{pipeline_version}/"
f"{variant.name}.{variant.extension}"
)
s3.put_object(
Bucket=settings.s3_bucket,
Key=key,
Body=variant.data,
ContentType=variant.content_type,
)
variant_records.append(
{
"name": variant.name,
"key": key,
"content_type": variant.content_type,
"width": variant.width,
"height": variant.height,
"size_bytes": len(variant.data),
"sha256": variant.sha256,
}
)
if lost_event.is_set():
logger.warning(
"Worker lost lease before final commit: job=%s",
job_id,
)
return {
"state": "retry",
"delay": float(settings.worker_lease_seconds),
}
# Use a fresh transaction and verify the fencing token.
db = SessionLocal()
job = db.scalar(
select(ProcessingJob)
.where(ProcessingJob.id == job_id)
.with_for_update()
)
attachment = db.scalar(
select(Attachment)
.where(Attachment.id == attachment_id)
.with_for_update()
)
if job is None or attachment is None:
db.rollback()
return {"state": "missing"}
if (
job.status != "processing"
or job.worker_id != worker_id
or job.attempt != attempt
or job.lease_until is None
or job.lease_until <= utc_now()
or lost_event.is_set()
):
db.rollback()
logger.warning(
"Fencing check rejected stale worker: job=%s",
job_id,
)
return {
"state": "retry",
"delay": float(settings.worker_lease_seconds),
}
for item in variant_records:
existing = db.scalar(
select(AttachmentVariant)
.where(
AttachmentVariant.attachment_id == attachment_id,
AttachmentVariant.variant == item["name"],
)
)
if existing is None:
existing = AttachmentVariant(
attachment_id=attachment_id,
variant=item["name"],
object_key=item["key"],
content_type=item["content_type"],
width=item["width"],
height=item["height"],
size_bytes=item["size_bytes"],
sha256=item["sha256"],
)
db.add(existing)
else:
existing.object_key = item["key"]
existing.content_type = item["content_type"]
existing.width = item["width"]
existing.height = item["height"]
existing.size_bytes = item["size_bytes"]
existing.sha256 = item["sha256"]
job.status = "ready"
job.lease_until = None
job.worker_id = None
job.last_error_code = None
attachment.status = "ready"
attachment.error_code = None
attachment.updated_at = utc_now()
db.commit()
processing_elapsed_ms = (
time.perf_counter() - processing_started_at
) * 1000
logger.info(
(
"Image processing completed: attachment=%s "
"attempt=%s queue_lag_ms=%.2f processing_elapsed_ms=%.2f"
),
attachment_id,
attempt,
queue_lag_ms,
processing_elapsed_ms,
)
db.close()
db = None
# Staging cleanup is performed only after the final DB commit.
try:
s3.delete_object(
Bucket=settings.s3_bucket,
Key=staging_key,
)
cleanup_db = SessionLocal()
try:
cleanup_attachment = cleanup_db.scalar(
select(Attachment)
.where(Attachment.id == attachment_id)
.with_for_update()
)
if (
cleanup_attachment is not None
and cleanup_attachment.status == "ready"
and cleanup_attachment.staging_key == staging_key
):
cleanup_attachment.staging_key = None
cleanup_db.commit()
else:
cleanup_db.rollback()
finally:
cleanup_db.close()
except Exception:
logger.warning(
"Could not delete staging object for attachment=%s",
attachment_id,
exc_info=True,
)
return {"state": "ready"}
except ImageProcessingTimeout:
if db is not None:
db.rollback()
stop_event.set()
heartbeat.join(timeout=2)
logger.warning(
"Image processing timed out: job=%s attachment=%s timeout=%ss",
job_id,
attachment_id,
settings.image_processing_timeout_seconds,
)
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"image_processing_timeout",
permanent=False,
)
except ValueError as exc:
if db is not None:
db.rollback()
stop_event.set()
heartbeat.join(timeout=2)
code = str(exc)[:64] or "invalid_image"
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
code,
permanent=True,
)
except ClientError as exc:
if db is not None:
db.rollback()
stop_event.set()
heartbeat.join(timeout=2)
error = exc.response.get("Error", {})
error_code = error.get("Code", "")
if error_code in {
"NoSuchKey",
"NoSuchBucket",
"404",
"NotFound",
}:
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"staging_missing",
permanent=True,
)
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"storage_unavailable",
permanent=False,
)
except (BotoCoreError, SQLAlchemyError):
if db is not None:
db.rollback()
stop_event.set()
heartbeat.join(timeout=2)
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"processing_unavailable",
permanent=False,
)
except Exception:
if db is not None:
db.rollback()
logger.exception(
"Unexpected processing error: job=%s",
job_id,
)
stop_event.set()
heartbeat.join(timeout=2)
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"processing_failed",
permanent=False,
)
finally:
stop_event.set()
heartbeat.join(timeout=2)
if db is not None:
db.close()
def _process(payload: dict) -> dict:
return process_job_with_telemetry(
_process_impl,
payload,
)
async def _publish_delayed(
exchange,
payload: dict,
delay_seconds: float,
) -> None:
"""Publish a persistent message to the TTL retry queue."""
delay_ms = max(
1000,
int(delay_seconds * 1000),
)
message = Message(
body=json.dumps(
payload,
separators=(",", ":"),
).encode("utf-8"),
content_type="application/json",
delivery_mode=DeliveryMode.PERSISTENT,
message_id=str(uuid.uuid4()),
type="attachment.process.retry",
expiration=timedelta(milliseconds=delay_ms),
)
await exchange.publish(
message,
routing_key=f"{settings.rabbitmq_queue}.retry",
mandatory=True,
)
logger.info(
"Scheduled delayed delivery in %.2fs",
delay_ms / 1000,
)
async def _quarantine_message(exchange, message) -> None:
"""Move an invalid message to the durable dead-letter queue."""
dead_message = Message(
body=message.body,
content_type=message.content_type or "application/octet-stream",
delivery_mode=DeliveryMode.PERSISTENT,
message_id=message.message_id or str(uuid.uuid4()),
type="attachment.process.invalid",
)
await exchange.publish(
dead_message,
routing_key=DEAD_ROUTING_KEY,
mandatory=True,
)
await message.ack()
logger.error(
"Invalid message moved to dead-letter queue: message_id=%s",
message.message_id,
)
async def handle_message(
message: aio_pika.IncomingMessage,
retry_exchange,
dead_exchange,
semaphore: asyncio.Semaphore,
) -> None:
"""Consume, process, retry or quarantine one RabbitMQ message."""
async with semaphore:
try:
payload = json.loads(message.body.decode("utf-8"))
if not isinstance(payload, dict):
raise ValueError("invalid_payload")
for field in (
"job_id",
"attachment_id",
"pipeline_version",
):
if field not in payload:
raise ValueError("missing_payload_field")
_safe_uuid(payload["job_id"])
_safe_uuid(payload["attachment_id"])
int(payload["pipeline_version"])
except (ValueError, TypeError, KeyError, UnicodeDecodeError):
try:
await _quarantine_message(
dead_exchange,
message,
)
except Exception:
logger.exception(
"Could not quarantine invalid message"
)
await message.nack(requeue=True)
return
try:
result = await asyncio.to_thread(
_process,
payload,
)
state = result["state"]
if state in {
"busy",
"deferred",
"retry",
}:
await _publish_delayed(
retry_exchange,
payload,
result.get("delay", 1.0),
)
await message.ack()
logger.info(
"Message handled: job=%s state=%s",
payload["job_id"],
state,
)
except asyncio.CancelledError:
raise
except Exception:
logger.exception(
"Message processing failed: message_id=%s",
message.message_id,
)
# If delayed publication or DB processing fails,
# preserve the original message.
await message.nack(requeue=True)
async def main() -> None:
"""Start the durable RabbitMQ consumer."""
setup_worker_telemetry()
connection = await aio_pika.connect_robust(
settings.rabbitmq_url
)
async with connection:
channel = await connection.channel(
publisher_confirms=True
)
await channel.set_qos(
prefetch_count=settings.rabbitmq_prefetch
)
exchange = await channel.declare_exchange(
settings.rabbitmq_exchange,
aio_pika.ExchangeType.DIRECT,
durable=True,
)
queue = await channel.declare_queue(
settings.rabbitmq_queue,
durable=True,
)
await queue.bind(
exchange,
routing_key=settings.rabbitmq_queue,
)
# Delayed retry exchange and queue.
retry_exchange = await channel.declare_exchange(
f"{settings.rabbitmq_exchange}.retry",
aio_pika.ExchangeType.DIRECT,
durable=True,
)
retry_queue = await channel.declare_queue(
f"{settings.rabbitmq_queue}.retry",
durable=True,
arguments={
"x-dead-letter-exchange": settings.rabbitmq_exchange,
"x-dead-letter-routing-key": settings.rabbitmq_queue,
},
)
await retry_queue.bind(
retry_exchange,
routing_key=f"{settings.rabbitmq_queue}.retry",
)
# Durable quarantine queue for invalid messages.
dead_exchange = await channel.declare_exchange(
f"{settings.rabbitmq_exchange}.dead",
aio_pika.ExchangeType.DIRECT,
durable=True,
)
dead_queue = await channel.declare_queue(
f"{settings.rabbitmq_queue}.dead",
durable=True,
)
await dead_queue.bind(
dead_exchange,
routing_key=DEAD_ROUTING_KEY,
)
semaphore = asyncio.Semaphore(
settings.worker_concurrency
)
async def consumer(message):
await handle_message(
message,
retry_exchange,
dead_exchange,
semaphore,
)
logger.info(
"Worker %s consuming queue=%s concurrency=%s prefetch=%s",
WORKER_ID,
settings.rabbitmq_queue,
settings.worker_concurrency,
settings.rabbitmq_prefetch,
)
await queue.consume(consumer)
await asyncio.Future()
if __name__ == "__main__":
asyncio.run(main())
+161
View File
@@ -0,0 +1,161 @@
services:
postgres:
image: postgres:16.6-bookworm
environment:
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 3s
retries: 15
restart: unless-stopped
networks:
- backend
redis:
image: redis:7.4.2-alpine
command: ["redis-server", "--appendonly", "yes", "--save", "60", "1000"]
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
restart: unless-stopped
networks:
- backend
rabbitmq:
image: rabbitmq:4.0.5-management-alpine
environment:
RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
volumes:
- rabbitmq_data:/var/lib/rabbitmq
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
interval: 10s
timeout: 5s
retries: 10
restart: unless-stopped
networks:
- backend
seaweedfs:
image: chrislusf/seaweedfs:4.46
command: ["mini", "-dir=/data"]
environment:
AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY}
S3_BUCKET: ${S3_BUCKET}
volumes:
- seaweedfs_data:/data
ports:
- "127.0.0.1:8333:8333"
restart: unless-stopped
networks:
- backend
migrate:
build:
context: .
dockerfile: Dockerfile
env_file: .env
command: ["alembic", "upgrade", "head"]
depends_on:
postgres:
condition: service_healthy
healthcheck:
disable: true
restart: "no"
networks:
- backend
api:
build:
context: .
dockerfile: Dockerfile
env_file: .env
expose:
- "8080"
depends_on:
migrate:
condition: service_completed_successfully
seaweedfs:
condition: service_started
redis:
condition: service_healthy
restart: unless-stopped
networks:
- backend
outbox-publisher:
build:
context: .
dockerfile: Dockerfile
env_file: .env
command: ["python", "-m", "app.services.outbox_publisher"]
depends_on:
migrate:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
healthcheck:
disable: true
restart: unless-stopped
networks:
- backend
worker:
build:
context: .
dockerfile: Dockerfile
env_file: .env
cpus: 1.0
mem_limit: 1g
command: ["python", "-m", "app.worker"]
depends_on:
migrate:
condition: service_completed_successfully
seaweedfs:
condition: service_started
rabbitmq:
condition: service_healthy
healthcheck:
disable: true
restart: unless-stopped
networks:
- backend
nginx:
image: nginx:1.27.4-alpine
ports:
- "127.0.0.1:18081:80"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
api:
condition: service_healthy
healthcheck:
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1/health/live || exit 1"]
interval: 10s
timeout: 3s
retries: 10
restart: unless-stopped
networks:
- backend
volumes:
postgres_data:
redis_data:
rabbitmq_data:
seaweedfs_data:
networks:
backend:
driver: bridge
+258
View File
@@ -0,0 +1,258 @@
services:
postgres:
image: postgres:16.6-bookworm
environment:
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 3s
retries: 15
restart: unless-stopped
networks:
- backend
redis:
image: redis:7.4.2-alpine
command: ["redis-server", "--appendonly", "yes", "--save", "60", "1000"]
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
restart: unless-stopped
networks:
- backend
rabbitmq:
image: rabbitmq:4.0.5-management-alpine
environment:
RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
volumes:
- rabbitmq_data:/var/lib/rabbitmq
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
interval: 10s
timeout: 5s
retries: 10
restart: unless-stopped
networks:
- backend
seaweedfs:
image: chrislusf/seaweedfs:4.46
command: ["mini", "-dir=/data"]
environment:
AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY}
S3_BUCKET: ${S3_BUCKET}
volumes:
- seaweedfs_data:/data
ports:
- "127.0.0.1:8334:8333"
restart: unless-stopped
networks:
- backend
migrate:
build:
context: .
dockerfile: Dockerfile
env_file: .env
command: ["alembic", "upgrade", "head"]
depends_on:
postgres:
condition: service_healthy
healthcheck:
disable: true
restart: "no"
networks:
- backend
api:
build:
context: .
dockerfile: Dockerfile
env_file: .env
environment:
OTEL_SERVICE_NAME: messenger-api
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=lab
expose:
- "8080"
healthcheck:
test:
- CMD
- python
- -c
- "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=3)"
interval: 10s
timeout: 5s
retries: 10
start_period: 30s
depends_on:
migrate:
condition: service_completed_successfully
seaweedfs:
condition: service_started
redis:
condition: service_healthy
otel-collector:
condition: service_started
restart: unless-stopped
networks:
- backend
outbox-publisher:
build:
context: .
dockerfile: Dockerfile
env_file: .env
command: ["python", "-m", "app.services.outbox_publisher"]
depends_on:
migrate:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
healthcheck:
disable: true
restart: unless-stopped
networks:
- backend
worker:
build:
context: .
dockerfile: Dockerfile
env_file: .env
environment:
OTEL_SERVICE_NAME: messenger-worker
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=lab
cpus: 1.0
mem_limit: 1g
command: ["python", "-m", "app.worker"]
depends_on:
migrate:
condition: service_completed_successfully
seaweedfs:
condition: service_started
rabbitmq:
condition: service_healthy
otel-collector:
condition: service_started
healthcheck:
disable: true
restart: unless-stopped
networks:
- backend
nginx:
image: nginx:1.27.4-alpine
ports:
- "127.0.0.1:18082:80"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
api:
condition: service_healthy
healthcheck:
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1/health/live || exit 1"]
interval: 10s
timeout: 3s
retries: 10
restart: unless-stopped
networks:
- backend
otel-collector:
image: otel/opentelemetry-collector-contrib:0.116.1
command: ["--config=/etc/otelcol-contrib/config.yaml"]
volumes:
- ./observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
depends_on:
- prometheus
- loki
- tempo
restart: unless-stopped
networks:
- backend
prometheus:
image: prom/prometheus:v2.55.1
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=7d"
volumes:
- ./observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
ports:
- "127.0.0.1:19090:9090"
restart: unless-stopped
networks:
- backend
loki:
image: grafana/loki:3.3.2
command: ["-config.file=/etc/loki/local-config.yaml"]
volumes:
- ./observability/loki-config.yaml:/etc/loki/local-config.yaml:ro
- loki_data:/loki
restart: unless-stopped
networks:
- backend
tempo:
image: grafana/tempo:2.6.1
command: ["-config.file=/etc/tempo/tempo.yaml"]
volumes:
- ./observability/tempo-config.yaml:/etc/tempo/tempo.yaml:ro
- tempo_data:/var/tempo
restart: unless-stopped
networks:
- backend
grafana:
image: grafana/grafana:11.4.0
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-change_me}
GF_USERS_ALLOW_SIGN_UP: "false"
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro
- ./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards:ro
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
ports:
- "127.0.0.1:13000:3000"
depends_on:
- prometheus
- loki
- tempo
restart: unless-stopped
networks:
- backend
volumes:
postgres_data:
redis_data:
rabbitmq_data:
seaweedfs_data:
prometheus_data:
loki_data:
tempo_data:
grafana_data:
networks:
backend:
driver: bridge
+171
View File
@@ -0,0 +1,171 @@
{
"id": null,
"uid": "messenger-overview",
"title": "Messenger — Observability",
"tags": ["messenger", "lab6", "observability"],
"timezone": "browser",
"schemaVersion": 39,
"version": 1,
"refresh": "10s",
"time": {
"from": "now-1h",
"to": "now"
},
"panels": [
{
"id": 1,
"title": "API Requests Per Second",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"targets": [
{
"refId": "A",
"expr": "sum(rate(messenger_http_server_duration_milliseconds_count[5m]))",
"legendFormat": "Requests/sec"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"min": 0
},
"overrides": []
},
"options": {
"legend": {"displayMode": "list", "placement": "bottom"},
"tooltip": {"mode": "multi"}
}
},
{
"id": 2,
"title": "API Error Rate",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"targets": [
{
"refId": "A",
"expr": "100 * sum(rate(messenger_http_server_duration_milliseconds_count{http_response_status_code=~\"5..\"}[5m])) / clamp_min(sum(rate(messenger_http_server_duration_milliseconds_count[5m])), 0.001)",
"legendFormat": "5xx errors (%)"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {"displayMode": "list", "placement": "bottom"},
"tooltip": {"mode": "multi"}
}
},
{
"id": 3,
"title": "API Request Duration — p95",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_http_server_duration_milliseconds_bucket[5m])))",
"legendFormat": "p95"
}
],
"fieldConfig": {
"defaults": {
"unit": "ms",
"min": 0
},
"overrides": []
},
"options": {
"legend": {"displayMode": "list", "placement": "bottom"},
"tooltip": {"mode": "multi"}
}
},
{
"id": 4,
"title": "Successful Background Jobs",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 8},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"targets": [
{
"refId": "A",
"expr": "sum(rate(messenger_jobs_processed_total[5m]))",
"legendFormat": "Successful jobs/sec"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"min": 0
},
"overrides": []
},
"options": {
"legend": {"displayMode": "list", "placement": "bottom"},
"tooltip": {"mode": "multi"}
}
},
{
"id": 5,
"title": "Failed Background Jobs",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 16},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"targets": [
{
"refId": "A",
"expr": "sum(rate(messenger_jobs_failed_total[5m]))",
"legendFormat": "Failed jobs/sec"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"min": 0
},
"overrides": []
},
"options": {
"legend": {"displayMode": "list", "placement": "bottom"},
"tooltip": {"mode": "multi"}
}
},
{
"id": 6,
"title": "Job Processing Duration — p95",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 16},
"datasource": {"type": "prometheus", "uid": "prometheus"},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_job_processing_duration_seconds_bucket[5m])))",
"legendFormat": "p95"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"options": {
"legend": {"displayMode": "list", "placement": "bottom"},
"tooltip": {"mode": "multi"}
}
}
],
"annotations": {
"list": []
},
"links": []
}
@@ -0,0 +1,11 @@
apiVersion: 1
providers:
- name: Messenger
orgId: 1
folder: Messenger
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards
@@ -0,0 +1,30 @@
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
- name: Loki
uid: loki
type: loki
access: proxy
url: http://loki:3100
editable: false
jsonData:
derivedFields:
- name: TraceID
matcherRegex: '"trace_id":"([0-9a-f]{32})"'
datasourceUid: tempo
url: '$${__value.raw}'
- name: Tempo
uid: tempo
type: tempo
access: proxy
url: http://tempo:3200
editable: false
+1
View File
@@ -0,0 +1 @@
Generic single-database configuration.
+60
View File
@@ -0,0 +1,60 @@
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
from app.core.config import settings
from app.db.base import Base
from app.db import models
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
config.set_main_option(
"sqlalchemy.url",
settings.database_url.replace("%", "%%"),
)
target_metadata = Base.metadata
def run_migrations_offline() -> None:
url = settings.database_url
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
+26
View File
@@ -0,0 +1,26 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
@@ -0,0 +1,79 @@
"""create initial database schema
Revision ID: 21ad89656327
Revises:
Create Date: 2026-11-04 17:31:39.336368
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '21ad89656327'
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('chats',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('title', sa.String(length=100), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_table('users',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('username', sa.String(length=32), nullable=False),
sa.Column('display_name', sa.String(length=100), nullable=False),
sa.Column('password_hash', sa.String(length=255), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_users_username'), 'users', ['username'], unique=True)
op.create_table('chat_members',
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('chat_id', 'user_id')
)
op.create_table('messages',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('sender_id', sa.UUID(), nullable=False),
sa.Column('text', sa.Text(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['sender_id'], ['users.id'], ),
sa.PrimaryKeyConstraint('id')
)
op.create_index('ix_messages_chat_created_id', 'messages', ['chat_id', 'created_at', 'id'], unique=False)
op.create_table('sessions',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('token_hash', sa.String(length=64), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_sessions_token_hash'), 'sessions', ['token_hash'], unique=True)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_sessions_token_hash'), table_name='sessions')
op.drop_table('sessions')
op.drop_index('ix_messages_chat_created_id', table_name='messages')
op.drop_table('messages')
op.drop_table('chat_members')
op.drop_index(op.f('ix_users_username'), table_name='users')
op.drop_table('users')
op.drop_table('chats')
# ### end Alembic commands ###
@@ -0,0 +1,37 @@
"""add attachment metadata"""
from alembic import op
import sqlalchemy as sa
from typing import Sequence, Union
revision: str = "4d9a1f2c7b10"
down_revision: Union[str, None] = "8c3d9f4a21b7"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("messages", sa.Column("attachment_ids", sa.JSON(), nullable=False, server_default="[]"))
op.create_table(
"attachments",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("chat_id", sa.UUID(), nullable=False),
sa.Column("owner_id", sa.UUID(), nullable=False),
sa.Column("filename", sa.String(255), nullable=False),
sa.Column("content_type", sa.String(32), nullable=False),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("sha256", sa.String(64), nullable=False),
sa.Column("object_key", sa.String(512), nullable=False),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("error_code", sa.String(64), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["chat_id"], ["chats.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["owner_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key"),
)
op.create_index("ix_attachments_chat_id", "attachments", ["chat_id"])
def downgrade() -> None:
op.drop_index("ix_attachments_chat_id", table_name="attachments")
op.drop_table("attachments")
op.drop_column("messages", "attachment_ids")
@@ -0,0 +1,39 @@
"""add shared PostgreSQL idempotency records
Revision ID: 8c3d9f4a21b7
Revises: 21ad89656327
Create Date: 2026-11-05
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "8c3d9f4a21b7"
down_revision: Union[str, None] = "21ad89656327"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"idempotency_records",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("user_id", sa.UUID(), nullable=False),
sa.Column("method", sa.String(length=10), nullable=False),
sa.Column("path", sa.String(length=255), nullable=False),
sa.Column("idempotency_key", sa.String(length=128), nullable=False),
sa.Column("request_hash", sa.String(length=64), nullable=False),
sa.Column("message_id", sa.UUID(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("user_id", "method", "path", "idempotency_key", name="uq_idempotency_scope_key"),
)
op.create_index("ix_idempotency_records_expires_at", "idempotency_records", ["expires_at"], unique=False)
def downgrade() -> None:
op.drop_index("ix_idempotency_records_expires_at", table_name="idempotency_records")
op.drop_table("idempotency_records")
@@ -0,0 +1,78 @@
"""add attachment uploads
Revision ID: a7c2e91d5f40
Revises: 4d9a1f2c7b10
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "a7c2e91d5f40"
down_revision: Union[str, None] = "4d9a1f2c7b10"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"attachment_uploads",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("chat_id", sa.UUID(), nullable=False),
sa.Column("owner_id", sa.UUID(), nullable=False),
sa.Column("filename", sa.String(length=255), nullable=False),
sa.Column("content_type", sa.String(length=32), nullable=False),
sa.Column("expected_size_bytes", sa.Integer(), nullable=False),
sa.Column("expected_sha256", sa.String(length=64), nullable=False),
sa.Column("object_key", sa.String(length=512), nullable=False),
sa.Column("status", sa.String(length=16), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(
["chat_id"],
["chats.id"],
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["owner_id"],
["users.id"],
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key"),
)
op.create_index(
"ix_attachment_uploads_chat_id",
"attachment_uploads",
["chat_id"],
)
op.create_index(
"ix_attachment_uploads_owner_id",
"attachment_uploads",
["owner_id"],
)
op.create_index(
"ix_attachment_uploads_expires_at",
"attachment_uploads",
["expires_at"],
)
def downgrade() -> None:
op.drop_index(
"ix_attachment_uploads_expires_at",
table_name="attachment_uploads",
)
op.drop_index(
"ix_attachment_uploads_owner_id",
table_name="attachment_uploads",
)
op.drop_index(
"ix_attachment_uploads_chat_id",
table_name="attachment_uploads",
)
op.drop_table("attachment_uploads")
@@ -0,0 +1,210 @@
"""add durable image processing jobs"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "a9d670bf1c16"
down_revision: Union[str, None] = "a7c2e91d5f40"
branch_labels = None
depends_on = None
def upgrade() -> None:
# Дополнительные поля для хранения исходника и версии конвейера.
op.add_column(
"attachments",
sa.Column("staging_key", sa.String(length=512), nullable=True),
)
op.create_unique_constraint(
"uq_attachments_staging_key",
"attachments",
["staging_key"],
)
op.add_column(
"attachments",
sa.Column(
"pipeline_version",
sa.Integer(),
nullable=False,
server_default="1",
),
)
# Задания на асинхронную обработку.
op.create_table(
"processing_jobs",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("attachment_id", sa.UUID(), nullable=False),
sa.Column(
"pipeline_version",
sa.Integer(),
nullable=False,
server_default="1",
),
sa.Column(
"attempt",
sa.Integer(),
nullable=False,
server_default="1",
),
sa.Column(
"status",
sa.String(length=16),
nullable=False,
server_default="queued",
),
sa.Column(
"available_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.Column("lease_until", sa.DateTime(timezone=True), nullable=True),
sa.Column("worker_id", sa.String(length=128), nullable=True),
sa.Column(
"last_error_code",
sa.String(length=64),
nullable=True,
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.ForeignKeyConstraint(
["attachment_id"],
["attachments.id"],
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint(
"attachment_id",
"pipeline_version",
name="uq_processing_job_attachment_pipeline",
),
)
op.create_index(
"ix_processing_jobs_status_available",
"processing_jobs",
["status", "available_at"],
)
op.create_index(
"ix_processing_jobs_lease_until",
"processing_jobs",
["lease_until"],
)
# Созданные варианты изображений.
op.create_table(
"attachment_variants",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("attachment_id", sa.UUID(), nullable=False),
sa.Column("variant", sa.String(length=32), nullable=False),
sa.Column("object_key", sa.String(length=512), nullable=False),
sa.Column("content_type", sa.String(length=32), nullable=False),
sa.Column("width", sa.Integer(), nullable=False),
sa.Column("height", sa.Integer(), nullable=False),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("sha256", sa.String(length=64), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.ForeignKeyConstraint(
["attachment_id"],
["attachments.id"],
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key"),
sa.UniqueConstraint(
"attachment_id",
"variant",
name="uq_attachment_variant",
),
)
op.create_index(
"ix_attachment_variants_attachment_id",
"attachment_variants",
["attachment_id"],
)
# Transactional outbox: события публикуются в брокер
# отдельным процессом после фиксации транзакции.
op.create_table(
"outbox_events",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("event_type", sa.String(length=128), nullable=False),
sa.Column("aggregate_id", sa.UUID(), nullable=False),
sa.Column("payload", sa.JSON(), nullable=False),
sa.Column(
"status",
sa.String(length=16),
nullable=False,
server_default="pending",
),
sa.Column(
"attempts",
sa.Integer(),
nullable=False,
server_default="0",
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.Column(
"published_at",
sa.DateTime(timezone=True),
nullable=True,
),
sa.Column("last_error", sa.String(length=256), nullable=True),
sa.PrimaryKeyConstraint("id"),
)
op.create_index(
"ix_outbox_events_status_created",
"outbox_events",
["status", "created_at"],
)
def downgrade() -> None:
op.drop_index(
"ix_outbox_events_status_created",
table_name="outbox_events",
)
op.drop_table("outbox_events")
op.drop_index(
"ix_attachment_variants_attachment_id",
table_name="attachment_variants",
)
op.drop_table("attachment_variants")
op.drop_index(
"ix_processing_jobs_lease_until",
table_name="processing_jobs",
)
op.drop_index(
"ix_processing_jobs_status_available",
table_name="processing_jobs",
)
op.drop_table("processing_jobs")
op.drop_column("attachments", "pipeline_version")
op.drop_constraint(
"uq_attachments_staging_key",
"attachments",
type_="unique",
)
op.drop_column("attachments", "staging_key")
+37
View File
@@ -0,0 +1,37 @@
worker_processes auto;
events {
worker_connections 1024;
}
http {
resolver 127.0.0.11 valid=10s ipv6=off;
upstream api_backend {
zone api_backend 64k;
server api:8080 resolve;
}
server {
listen 80;
server_name _;
client_max_body_size 10m;
location / {
proxy_pass http://api_backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 5s;
proxy_send_timeout 120s;
proxy_read_timeout 120s;
proxy_next_upstream error timeout http_502 http_503 http_504;
}
}
}
+35
View File
@@ -0,0 +1,35 @@
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: "2024-01-01"
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
allow_structured_metadata: true
retention_period: 168h
compactor:
working_directory: /loki/compactor
retention_enabled: true
delete_request_store: filesystem
+48
View File
@@ -0,0 +1,48 @@
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 5s
limit_mib: 256
spike_limit_mib: 64
batch:
timeout: 5s
send_batch_size: 1024
exporters:
prometheus:
endpoint: 0.0.0.0:8889
namespace: messenger
send_timestamps: true
otlphttp/tempo:
endpoint: http://tempo:4318
otlphttp/loki:
endpoint: http://loki:3100/otlp
debug:
verbosity: basic
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/tempo, debug]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus, debug]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/loki, debug]
+9
View File
@@ -0,0 +1,9 @@
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: otel-collector
static_configs:
- targets:
- otel-collector:8889
+31
View File
@@ -0,0 +1,31 @@
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
ingester:
max_block_duration: 5m
compactor:
compaction:
block_retention: 168h
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
local:
path: /var/tempo/blocks
overrides:
defaults:
metrics_generator:
processors: []
+18
View File
@@ -0,0 +1,18 @@
fastapi==0.115.6
uvicorn[standard]==0.34.0
pydantic==2.10.4
pydantic-settings==2.7.1
SQLAlchemy==2.0.36
psycopg[binary]==3.2.3
alembic==1.14.0
redis==5.2.1
pwdlib[argon2]==0.2.1
boto3==1.35.99
Pillow==11.0.0
aio-pika==9.5.5
opentelemetry-api==1.29.0
opentelemetry-sdk==1.29.0
opentelemetry-exporter-otlp-proto-http==1.29.0
opentelemetry-instrumentation-fastapi==0.50b0
opentelemetry-instrumentation-sqlalchemy==0.50b0
opentelemetry-instrumentation-logging==0.50b0
+158
View File
@@ -0,0 +1,158 @@
"""Safely replay a failed attachment processing job.
Usage:
python scripts/replay_failed_job.py ATTACHMENT_UUID
"""
import argparse
import logging
import sys
import uuid
from datetime import datetime, timezone
from pathlib import Path
# Add project root to Python import path.
PROJECT_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(PROJECT_ROOT))
from sqlalchemy import select
from app.db.models import Attachment, OutboxEvent, ProcessingJob
from app.db.session import SessionLocal
logger = logging.getLogger("replay_failed_job")
EVENT_TYPE = "attachment.process.v1"
def utc_now() -> datetime:
return datetime.now(timezone.utc)
def replay_failed_job(attachment_id: uuid.UUID) -> None:
db = SessionLocal()
try:
# Lock the attachment first to serialize concurrent replay requests.
attachment = db.scalar(
select(Attachment)
.where(Attachment.id == attachment_id)
.with_for_update()
)
if attachment is None:
raise ValueError(f"Attachment not found: {attachment_id}")
job = db.scalar(
select(ProcessingJob)
.where(
ProcessingJob.attachment_id == attachment_id,
ProcessingJob.pipeline_version == attachment.pipeline_version,
)
.with_for_update()
)
if job is None:
raise ValueError(
f"Processing job not found for attachment: {attachment_id}"
)
# Only terminally failed jobs may be replayed.
if job.status != "failed":
raise ValueError(
f"Job cannot be replayed: current job status is '{job.status}'"
)
if attachment.status != "failed":
raise ValueError(
"Attachment and job states are inconsistent: "
f"attachment status is '{attachment.status}'"
)
# The worker needs the original staged file.
if not attachment.staging_key:
raise ValueError(
"Cannot replay: staging_key is empty. "
"The original file is not available for processing."
)
now = utc_now()
# Reset the existing job. Keep its ID and pipeline version.
job.attempt = 0
job.status = "queued"
job.available_at = now
job.lease_until = None
job.worker_id = None
job.last_error_code = None
# Return attachment metadata to the queued state.
attachment.status = "queued"
attachment.error_code = None
# Create a new durable outbox event in the same transaction.
event = OutboxEvent(
event_type=EVENT_TYPE,
aggregate_id=attachment.id,
payload={
"schema_version": 1,
"job_id": str(job.id),
"type": EVENT_TYPE,
"attachment_id": str(attachment.id),
"pipeline_version": job.pipeline_version,
"attempt": 1,
"created_at": now.isoformat(),
},
status="pending",
)
db.add(event)
db.commit()
print("Replay request successfully committed.")
print(f"Attachment ID: {attachment.id}")
print(f"Job ID: {job.id}")
print(f"Job status: {job.status}")
print(f"Attempt: {job.attempt}")
print(f"Outbox event: {event.id}")
print("Outbox status: pending")
print()
print("The existing outbox-publisher will publish the event.")
except Exception:
db.rollback()
raise
finally:
db.close()
def main() -> int:
parser = argparse.ArgumentParser(
description="Replay a terminally failed attachment processing job."
)
parser.add_argument(
"attachment_id",
help="UUID of the attachment to replay",
)
args = parser.parse_args()
try:
attachment_id = uuid.UUID(args.attachment_id)
replay_failed_job(attachment_id)
return 0
except ValueError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 1
except Exception:
logger.exception("Could not replay failed processing job")
return 1
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
raise SystemExit(main())