ЛР 5 — БВТ2302 — Вотчицева — на 4 #34

Closed
Tutuxa wants to merge 1 commits from Tutuxa/mtusi2026lab5:solution/bvt2302-Votchitseva into main
35 changed files with 1886 additions and 25 deletions
+29
View File
@@ -0,0 +1,29 @@
# Pinned base image. Use a digest for reproducibility if you have one available:
# FROM python:3.12.7-slim-bookworm@sha256:<digest>
FROM python:3.12.7-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_NO_CACHE_DIR=1 \
APP_HOST=0.0.0.0 \
APP_PORT=8080
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
RUN useradd --create-home --uid 10001 --shell /usr/sbin/nologin appuser \
&& chown -R appuser:appuser /app
USER appuser
EXPOSE 8080
HEALTHCHECK --interval=2s --timeout=3s --start-period=5s --retries=2 \
CMD python -c "import sys,urllib.request; \
sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=2).status==200 else 1)"
CMD ["python", "-m", "src"]
+44 -18
View File
@@ -1,37 +1,63 @@
# Отчёт — лабораторная 5
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- **Автор, группа:** Вотчицева Вероника, БВТ2302
- **Целевая оценка:** 4 (трек на 5 не заявлялся)
- **Commit/PR:** ТУТ_БУДЕТ_ССЫЛКА_НА_PR
- **Использованные источники, библиотеки и помощь ИИ:**
- Python 3.12, FastAPI, SQLAlchemy (asyncpg), Redis, HAProxy, PostgreSQL.
- Очереди: RabbitMQ 3.13, aio-pika.
- S3-хранилище: SeaweedFS 4.35, boto3.
- Обработка изображений: Pillow 11.3.0.
- При подготовке кода и инфраструктуры использовался ассистент (Claude).
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
**Prerequisites:** Docker Desktop (с поддержкой `--wait`), Python 3.9+ для тестов.
```powershell
Copy-Item .env.example .env
# Заполнить .env (POSTGRES_PASSWORD, S3_ROOT_USER и т.д. по примеру из предыдущих лаб)
docker compose up -d --build --wait --wait-timeout 150
python tests/smoke.py
python student-tests/test_attachments.py
```
**Остановка:** `docker compose down`. **Удаление данных:** `docker compose down -v`.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
**Что перенесено из lab4:** API (FastAPI), HAProxy, PostgreSQL, Redis, S3 (SeaweedFS), валидация картинок через Pillow.
**Что добавлено (Async Pipeline):**
1. **Broker (RabbitMQ):** API больше не обрабатывает картинки. Он валидирует файл, сохраняет оригинал в S3 и отправляет задачу в очередь.
2. **Transactional Outbox:** Для гарантии, что задача не потеряется между записью в БД и отправкой в RabbitMQ, я использую таблицу `outbox`. API в одной транзакции с созданием вложения записывает туда задачу. Отдельный фоновый процесс (relayer) выгребает её и публикует в RabbitMQ.
3. **Worker:** Отдельный контейнер, слушающий RabbitMQ. Он скачивает оригинал из S3, генерирует варианты (thumbnail, crop, optimized) и сохраняет их метаданные в таблицу `attachment_variants`, меняя статус на `ready`.
4. **Статусы:** API возвращает `202 Accepted` со статусом `queued`. Клиент опрашивает `GET /attachments/{id}` до статуса `ready`.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Публичный smoke (202 + готовность) | `python tests/smoke.py` | PASS: lab5 public API scenario | `evidence/smoke.txt` |
| Ошибки доступа и форматов | `python student-tests/test_attachments.py` | PASS | `evidence/test_attachments.txt` |
| 12 параллельных сообщений (Lab3) | `python student-tests/run.py` | PASS | `evidence/concurrency-lab5.txt` |
| Persistence | `python tests/resilience.py --mode persistence` | PASS | `evidence/persistence-lab5.txt` |
| Очередь при падении воркера | Остановить worker -> загрузить -> запустить worker | Файл дождался воркера и стал `ready` | `evidence/worker-resumed.txt` |
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
**1. Отказ Worker (Durable Queue).**
Я остановила контейнер `worker`, затем загрузила изображение. API вернул `202 Accepted` (статус `queued`), задача осталась в RabbitMQ. После перезапуска контейнера `worker`, задача была автоматически подхвачена, превью сгенерировано, и статус сменился на `ready`. Это доказывает, что API не делает resize синхронно, и недоступность воркера не приводит к потере принятых файлов.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
**2. Защита от зависаний (Timeouts и Retry).**
В `worker.py` настроено подтверждение (ack) сообщения только после успешного сохранения результатов в S3 и БД. Если воркер упадет во время обработки, RabbitMQ автоматически переотправит сообщение (redelivery). Ограничение количества повторных попыток предотвращает бесконечный цикл ошибок, переводя вложение в статус `failed` с кодом ошибки.
## Ограничения и следующие шаги
**3. Обработка изображений (Pillow).**
Для тестовой картинки 320×200 превью `thumbnail` генерируется как JPEG размером 256×160 (сохраняются пропорции). Также генерируются `crop` (128×128) и `optimized` (WebP).
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
## Ограничения
- **SPOF (Single Point of Failure):** RabbitMQ и Worker запущены в единственном экземпляре. При падении RabbitMQ (без volume) задачи могут быть потеряны.
- **Масштабирование Worker:** В текущей конфигурации запущен один воркер. В production-среде можно запустить несколько реплик воркера, так как RabbitMQ поддерживает конкурентное потребление (prefetch_count).
- **Retry:** В базовой реализации используется ограниченное количество повторов. Для production-среды потребуется настроить Dead Letter Queue (DLQ) и exponential backoff.
+169
View File
@@ -0,0 +1,169 @@
x-api: &api
build: .
image: mtusi-lab5-votchitseva-api:local
environment:
APP_HOST: 0.0.0.0
APP_PORT: 8080
DATABASE_URL: postgresql+asyncpg://user:${POSTGRES_PASSWORD:?Copy .env.example to .env}@db:5432/messenger
CACHE_URL: redis://redis:6379/0
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
S3_ENDPOINT: http://s3:8333
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
BROKER_URL: amqp://guest:guest@broker:5672/
depends_on:
migrate:
condition: service_completed_successfully
redis:
condition: service_healthy
s3:
condition: service_healthy
broker:
condition: service_healthy
networks: [front, data]
cpus: 0.5
mem_limit: 384m
stop_grace_period: 15s
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=4).close()"]
interval: 5s
timeout: 5s
retries: 16
start_period: 35s
services:
api1:
<<: *api
api2:
<<: *api
worker:
<<: *api
command: ["python", "-m", "src.worker"]
healthcheck:
disable: true
deploy:
replicas: 1
broker:
image: rabbitmq:3.13-alpine
entrypoint: ["/bin/sh", "-c", "rm -f /var/lib/rabbitmq/.erlang.cookie && exec /usr/local/bin/docker-entrypoint.sh rabbitmq-server"]
environment:
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_PASS: guest
volumes:
- broker_data:/var/lib/rabbitmq
networks: [data]
cpus: 0.5
mem_limit: 512m
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "check_running"]
interval: 10s
timeout: 5s
retries: 10
start_period: 30s
s3_config:
image: python:3.12.7-slim-bookworm
command: ["python", "/bootstrap/generate-s3-config.py"]
environment:
S3_ROOT_USER: ${S3_ROOT_USER:?Set S3_ROOT_USER in .env}
S3_ROOT_PASSWORD: ${S3_ROOT_PASSWORD:?Set S3_ROOT_PASSWORD in .env}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
volumes:
- ./infra/generate-s3-config.py:/bootstrap/generate-s3-config.py:ro
- s3_config_data:/s3-config
networks: [data]
restart: "no"
s3:
image: chrislusf/seaweedfs:4.35
command: ["mini", "-dir=/data", "-s3.config=/s3-config/s3.json", "-bucket=messenger-attachments"]
volumes:
- s3_data:/data
- s3_config_data:/s3-config:ro
depends_on:
s3_config: {condition: service_completed_successfully}
networks: [data]
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8333/status || exit 1"]
interval: 2s
timeout: 3s
retries: 15
start_period: 10s
migrate:
build: .
image: mtusi-lab5-votchitseva-api:local
command: ["python", "-m", "src.migrate"]
environment:
DATABASE_URL: postgresql+asyncpg://user:${POSTGRES_PASSWORD:?Copy .env.example to .env}@db:5432/messenger
depends_on:
db:
condition: service_healthy
networks: [data]
cpus: 0.5
mem_limit: 256m
healthcheck:
disable: true
lb:
image: haproxy:3.0.5-alpine
ports: ["127.0.0.1:8080:8080"]
volumes:
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
depends_on:
api1:
condition: service_healthy
api2:
condition: service_healthy
networks: [front]
cpus: 0.25
mem_limit: 128m
db:
image: postgres:16.4-alpine
environment:
POSTGRES_USER: user
POSTGRES_DB: messenger
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Copy .env.example to .env}
volumes: [pgdata:/var/lib/postgresql/data]
networks: [data]
cpus: 1.0
mem_limit: 512m
healthcheck:
test: ["CMD-SHELL", "pg_isready -U user -d messenger"]
interval: 2s
timeout: 2s
retries: 15
redis:
image: redis:7.4.1-alpine
command: ["redis-server", "--appendonly", "yes", "--appendfsync", "always"]
volumes: [redisdata:/data]
networks: [data]
cpus: 0.25
mem_limit: 128m
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 2s
timeout: 2s
retries: 15
networks:
front:
data:
internal: true
volumes:
pgdata:
redisdata:
s3_data:
s3_config_data:
broker_data:
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+30
View File
@@ -0,0 +1,30 @@
"""Bootstraps a private S3 identity configuration from non-committed .env.
The server reads it from a shared internal-only configuration volume.
"""
import json
import os
from pathlib import Path
bucket = os.environ["S3_BUCKET"]
assert bucket == "messenger-attachments", "Fixed bucket name; change policy and config deliberately"
config = {
"identities": [
{
"name": "lab4-admin",
"credentials": [{"accessKey": os.environ["S3_ROOT_USER"],
"secretKey": os.environ["S3_ROOT_PASSWORD"]}],
"actions": ["Admin", "Read", "Write", "List", "Tagging"],
},
{
"name": "lab4-api",
"credentials": [{"accessKey": os.environ["S3_ACCESS_KEY_ID"],
"secretKey": os.environ["S3_SECRET_ACCESS_KEY"]}],
"actions": [f"Read:{bucket}", f"Write:{bucket}", f"List:{bucket}"],
},
],
}
target = Path("/s3-config/s3.json")
target.write_text(json.dumps(config, indent=2), encoding="utf-8")
target.chmod(0o644) # Only mounted into S3 service; SeaweedFS runs non-root on some images.
print("PASS: private S3 identity config generated; credentials not logged")
+38
View File
@@ -0,0 +1,38 @@
global
log stdout format raw local0
maxconn 512
defaults
mode http
log global
option httplog
timeout connect 3s
timeout client 75s
timeout server 75s
timeout check 2s
retries 1
option redispatch
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 2
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold obsolete 1s
hold nx 1s
hold refused 1s
hold timeout 1s
hold other 1s
frontend public
bind :8080
default_backend api
backend api
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
default-server check inter 1s fall 2 rise 1 resolvers docker init-addr libc,none
server api1 api1:8080
server api2 api2:8080
+11
View File
@@ -0,0 +1,11 @@
fastapi==0.115.4
uvicorn==0.32.0
sqlalchemy==2.0.36
asyncpg==0.30.0
redis==5.2.0
passlib[bcrypt]==1.7.4
bcrypt==4.0.1
python-dotenv==1.0.1
boto3==1.40.67
Pillow==11.3.0
aio-pika==9.4.3
+4 -4
View File
@@ -8,8 +8,8 @@ import re
ROOT = Path(__file__).resolve().parents[1]
def main():
lab = json.loads((ROOT/'lab.json').read_text())['number']
spec = json.loads((ROOT/'contracts/openapi.json').read_text())
lab = json.loads((ROOT/'lab.json').read_text(encoding='utf-8'))['number']
spec = json.loads((ROOT/'contracts/openapi.json').read_text(encoding='utf-8'))
assert spec['openapi']=='3.1.0'
assert spec['info']['version']=='1.%d.0'%lab
operation_ids = set()
@@ -41,11 +41,11 @@ def main():
assert ('/api/v1/auth/sessions' in spec['paths']) == (lab>=2)
assert ('/api/v1/attachments/{attachment_id}' in spec['paths']) == (lab>=4)
for file in list((ROOT/'tests').glob('*.py'))+list((ROOT/'scripts').glob('*.py')):
ast.parse(file.read_text(),filename=str(file))
ast.parse(file.read_text(encoding='utf-8'),filename=str(file))
for file in ROOT.rglob('*.json'):
if any(part in {'.git','.venv','node_modules','vendor','output'} for part in file.parts):
continue
json.loads(file.read_text())
json.loads(file.read_text(encoding='utf-8'))
for required in ('README.md','COURSE.md','CONTRIBUTING.md','REPORT.md','contracts/README.md','tests/smoke.py','tests/README.md'):
assert (ROOT/required).is_file(), 'missing '+required
print('PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.')
+13
View File
@@ -0,0 +1,13 @@
"""Convenience runner. Does not fabricate evidence or control Docker itself."""
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
for command in (
[sys.executable, str(ROOT / "tests/smoke.py")],
[sys.executable, str(ROOT / "student-tests/run.py")],
[sys.executable, str(ROOT / "student-tests/test_attachments.py")],
):
print("RUN:", " ".join(command), flush=True)
subprocess.run(command, cwd=ROOT, check=True)
print("PASS: public smoke + lab3 concurrency + grade4 negative cases; run extra resilience and memory experiment separately")
+175
View File
@@ -0,0 +1,175 @@
#!/usr/bin/env python3
"""Stops each API separately and replaces both under a small closed-loop load."""
from concurrent.futures import ThreadPoolExecutor
import json
import math
import os
from pathlib import Path
import sys
import subprocess
import threading
import time
import uuid
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, Failure, arguments, check, run
from resilience import history
import ops
def exercise(c, name, action):
alice, bob, eve, chat = c.bootstrap()
initial_key = uuid.uuid4().hex
initial = c.message(alice, chat, "До воздействия", key=initial_key)["json"]
observed = {c.call("GET", "/api/v1/users/me", actor=alice)["headers"]["X-Instance-Id"]
for _ in range(20)}
check(len(observed) >= 2, "need two actual instances before " + name)
stop = threading.Event()
barrier = threading.Barrier(5)
def worker(number):
local = Client(c.base, timeout=2)
writes, cycles, errors = [], [], 0
barrier.wait()
key = uuid.uuid4().hex
body = "worker-%d-%s" % (number, key)
while not stop.is_set():
started = time.monotonic()
try:
response = local.message(alice, chat, body, key=key)
writes.append(response["json"])
# Read with the original session while another backend may serve it.
local.call("GET", "/api/v1/chats/" + chat["id"], actor=bob, schema="Chat")
finished = time.monotonic()
cycles.append((finished, finished - started,
response["headers"]["X-Instance-Id"]))
key = uuid.uuid4().hex
body = "worker-%d-%s" % (number, key)
except Failure:
errors += 1
# An uncertain write keeps its original key and body.
time.sleep(0.05)
return writes, cycles, errors
started = time.monotonic()
with ThreadPoolExecutor(max_workers=4) as pool:
futures = [pool.submit(worker, n) for n in range(4)]
barrier.wait()
try:
time.sleep(1)
action_started = time.monotonic()
phases = action()
action_finished = time.monotonic()
time.sleep(7)
finally:
stop.set()
results = [f.result() for f in futures]
elapsed = time.monotonic() - started
writes = [msg for messages, _, _ in results for msg in messages]
cycles = sorted(item for _, values, _ in results for item in values)
errors = sum(n for _, _, n in results)
# An in-flight POST can finish before the stop and its paired GET after it.
# Only cycles started after the action count as traffic to a surviving API.
recovered = [t for t, duration, _ in cycles if t - duration >= action_finished]
check(bool(recovered), "no reads/writes after " + name)
recovery = recovered[0] - action_started
check(recovery <= 10 if name.startswith("stop-") else True,
"failover exceeded 10 seconds")
times = [started] + [t for t, _, _ in cycles] + [time.monotonic()]
max_gap = max(b - a for a, b in zip(times, times[1:]))
check(max_gap <= 10, "no successful read/write cycle for more than 10 seconds")
during = [t for t, _, _ in cycles if action_started <= t <= action_finished]
if name == "rolling-update":
check(bool(during), "no successful traffic during rolling replacement")
for phase in phases or []:
check(any(phase["started"] <= t <= phase["finished"] for t in during),
"no successful traffic while replacing " + phase["service"])
stored = history(c, bob, chat)
ids = {m["id"] for m in stored}
check(len(ids) == len(stored), "duplicate message IDs")
check(len({m["text"] for m in stored}) == len(stored), "retry created duplicate text")
by_id = {m["id"]: m for m in stored}
check(all(by_id.get(m["id"]) == m for m in [initial] + writes), "confirmed data lost")
check(c.message(alice, chat, initial["text"], key=initial_key)["json"] == initial,
"pre-action idempotency result lost")
latency = sorted(d * 1000 for _, d, _ in cycles)
def percentile(p):
return round(latency[max(0, math.ceil(p * len(latency)) - 1)], 3)
result = {"scenario": name, "concurrency": 4, "elapsed_seconds": round(elapsed, 3),
"successful_read_write_cycles": len(cycles), "failed_attempts": errors,
"successful_http_requests": len(writes) + len(cycles),
"successful_rps": round((len(writes) + len(cycles)) / elapsed, 3),
"successful_cycles_per_second": round(len(cycles) / elapsed, 3),
"successful_cycles_during_action": len(during),
"max_success_gap_seconds": round(max_gap, 3),
"first_success_after_action_seconds": round(recovery, 3),
"latency_ms": {"p50": percentile(.5), "p95": percentile(.95), "p99": percentile(.99)},
"instances_before": sorted(observed),
"instances_after": sorted({i for t, duration, i in cycles
if t - duration >= action_finished}),
"confirmed_messages_preserved": len({m["id"] for m in [initial] + writes}),
"note": "Closed-loop: one POST + one GET per cycle. RPS counts confirmed HTTP responses, including partial cycles. Latency covers successful cycles only. Recovery includes action duration."}
print(json.dumps(result, ensure_ascii=False, indent=2), flush=True)
return result
def main():
args = arguments("Lab3 grade4 Compose experiments")
c = Client(args.base_url, args.ca_file)
ops.BASE_URL = c.base
dest = ROOT / "evidence"
dest.mkdir(exist_ok=True)
http_args = ["--base-url", c.base]
if args.ca_file:
http_args += ["--ca-file", args.ca_file]
def checked(script, filename, *extra):
result = subprocess.run([sys.executable, str(ROOT / script), *extra], cwd=ROOT,
text=True, encoding="utf-8", stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
env=dict(os.environ, PYTHONIOENCODING="utf-8"))
(dest / filename).write_text(result.stdout, encoding="utf-8")
print(result.stdout, end="", flush=True)
check(result.returncode == 0, script + " failed; see evidence/" + filename)
ops.restore()
checked("scripts/check_contract.py", "check.txt")
checked("tests/smoke.py", "smoke.txt", *http_args)
checked("student-tests/run.py", "student-tests.txt", *http_args)
results = []
processes = []
for service in ops.SERVICES:
ops.restore()
before = ops.state(service)
try:
result = exercise(c, "stop-" + service, lambda: ops.stop_one(service))
after = ops.state(service)
check(after["status"] == "exited", "stopped API restarted unexpectedly")
check(all(i.split(":", 1)[0] != before["hostname"] for i in result["instances_after"]),
"stopped replica still handles requests")
results.append(result)
processes.append({"action": "stop-" + service, "before": before, "after": after})
finally:
ops.restore()
before = [ops.state(s) for s in ops.SERVICES]
results.append(exercise(c, "rolling-update", ops.rolling_update))
after = [ops.state(s) for s in ops.SERVICES]
check(all(a["container"] != b["container"] for a, b in zip(before, after)),
"both API containers must be replaced")
processes.append({"action": "rolling-update", "before": before, "after": after})
(dest / "grade4.json").write_text(json.dumps(results, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
(dest / "processes.json").write_text(json.dumps(processes, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
suffix = ".cmd" if os.name == "nt" else ".sh"
try:
checked("tests/resilience.py", "failover.txt", "--mode", "failover",
"--action-script", str(ROOT / ("scripts/stop-one" + suffix)), *http_args)
finally:
ops.restore()
checked("tests/resilience.py", "persistence.txt", "--mode", "persistence",
"--action-script", str(ROOT / ("scripts/restart" + suffix)), *http_args)
print("PASS: both replica failures, rolling replacement and confirmed data")
if __name__ == "__main__":
run(main)
+71
View File
@@ -0,0 +1,71 @@
"""10 simultaneous real uploads + Docker RSS observations (NO invented metrics)."""
import concurrent.futures
import json
import subprocess
import sys
import threading
import time
from pathlib import Path
from statistics import median
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'tests'))
from client import Client, check, arguments, run
def mib(value):
value = value.strip()
for suffix, multiplier in (("GiB",1024),("MiB",1),("KiB",1/1024),("B",1/1048576)):
if value.endswith(suffix):
return float(value[:-len(suffix)]) * multiplier
return None
def main():
args = arguments('Concurrent uploads + observed API container memory')
client = Client(args.base_url, args.ca_file, timeout=30)
alice, bob, eve, chat = client.bootstrap()
file = (ROOT / 'tests/fixtures/sample.png').read_bytes()
results, memory = [], []
stop = threading.Event()
def observe():
while not stop.is_set():
r = subprocess.run(['docker','stats','--no-stream','--format','{{.Name}}|{{.MemUsage}}'],
capture_output=True, text=True, timeout=10)
if r.returncode == 0:
for line in r.stdout.splitlines():
if '|api1-' in line or '|api2-' in line:
pass
parts = line.split('|')
if len(parts)==2 and ('api1-' in parts[0] or 'api2-' in parts[0]):
amount = mib(parts[1].split('/')[0])
if amount is not None:
memory.append({'container': parts[0], 'mem_mib': round(amount,3)})
stop.wait(0.2)
monitor = threading.Thread(target=observe, daemon=True)
monitor.start()
try:
def worker(i):
local = Client(args.base_url,args.ca_file,timeout=30)
start=time.perf_counter()
resp=local.call('POST', f'/api/v1/chats/{chat["id"]}/attachments?filename=run{i}.png',
201, actor=alice, raw=file, headers={'Content-Type':'image/png'}, schema='Attachment')
return {'elapsed_ms':round((time.perf_counter()-start)*1000,2), 'instance': resp['headers'].get('X-Instance-Id')}
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
futures=[executor.submit(worker,i) for i in range(10)]
results=[f.result() for f in concurrent.futures.as_completed(futures)]
finally:
stop.set(); monitor.join(timeout=12)
output={'concurrency':10,'file_bytes':len(file),'completed':len(results),
'median_latency_ms':round(median(x['elapsed_ms'] for x in results),2),
'api_peak_mem_mib_observed':max([x['mem_mib'] for x in memory],default=None),
'memory_samples':len(memory),'instances':sorted(set(x['instance'] for x in results))}
target=ROOT / 'evidence' / 'uploads-10.json'
target.write_text(json.dumps(output,ensure_ascii=False,indent=2),encoding='utf-8')
check(len(results)==10, 'not all uploads completed')
print('PASS: 10 parallel uploads; metrics written:',target)
print(json.dumps(output, ensure_ascii=False, indent=2))
if not memory:
print('WARNING: no Docker memory samples; run docker stats separately for a valid grade-4 experiment')
if __name__ == '__main__':
run(main)
+112
View File
@@ -0,0 +1,112 @@
#!/usr/bin/env python3
"""Operator actions for the local Compose stand; volumes are kept."""
import argparse
import json
import os
from pathlib import Path
import subprocess
import time
import urllib.error
import urllib.request
ROOT = Path(__file__).resolve().parents[1]
SERVICES = ("api1", "api2")
BASE_URL = os.environ.get("BASE_URL", "http://localhost:8080").rstrip("/")
def compose(*args, capture=False):
result = subprocess.run(["docker", "compose", *args], cwd=ROOT, check=True,
text=True, stdout=subprocess.PIPE if capture else None)
return result.stdout.strip() if capture else None
def state(service):
cid = compose("ps", "--all", "--quiet", service, capture=True)
if not cid:
raise RuntimeError("No container for " + service)
result = subprocess.check_output(
["docker", "inspect", "--format",
'{{json .Config.Hostname}} {{json .State.Status}}', cid], text=True)
hostname, status = result.strip().split()
return {"service": service, "container": cid[:12],
"hostname": json.loads(hostname), "status": json.loads(status)}
def wait_instances(count=2, timeout=40):
deadline = time.monotonic() + timeout
seen = set()
while time.monotonic() < deadline:
try:
with urllib.request.urlopen(BASE_URL + "/health/ready", timeout=2) as r:
instance = r.headers.get("X-Instance-Id")
if instance:
seen.add(instance)
if len(seen) >= count:
return seen
except (OSError, urllib.error.URLError):
pass
time.sleep(0.2)
raise RuntimeError("Not enough ready API instances through LB")
def stop_one(service=None):
target = os.environ.get("TARGET_INSTANCE_ID", "").split(":", 1)[0]
if target:
service = next((s for s in SERVICES if state(s)["hostname"] == target), None)
if not service:
raise RuntimeError("TARGET_INSTANCE_ID is not an API container")
service = service or "api1"
if service not in SERVICES:
raise RuntimeError("Only API replicas may be stopped")
compose("kill", "--signal", "SIGKILL", service)
current = state(service)
if current["status"] != "exited":
raise RuntimeError("API did not stop")
print(json.dumps(current))
def restore():
compose("up", "-d", "--no-deps", "--no-build", "--wait", "--wait-timeout", "40", *SERVICES)
wait_instances()
def restart():
compose("down")
compose("up", "-d", "--no-build", "--wait", "--wait-timeout", "60")
wait_instances()
def rolling_update():
phases = []
compose("build", "api1")
# This migration only adds tables and is compatible with the old API.
compose("run", "--rm", "--no-deps", "migrate")
for service in SERVICES:
before = state(service)
started = time.monotonic()
compose("up", "-d", "--no-deps", "--no-build", "--force-recreate",
"--wait", "--wait-timeout", "40", service)
after = state(service)
if before["container"] == after["container"]:
raise RuntimeError("Container was not replaced")
seen = wait_instances()
if not any(i.split(":", 1)[0] == after["hostname"] for i in seen):
raise RuntimeError("LB has not discovered the replacement")
phases.append({"service": service, "started": started, "finished": time.monotonic()})
print(json.dumps({"replaced": service, "before": before, "after": after}))
return phases
def main():
parser = argparse.ArgumentParser()
parser.add_argument("action", choices=["restart", "stop-one", "restore", "rolling-update"])
parser.add_argument("--service", choices=SERVICES)
args = parser.parse_args()
if args.action == "stop-one":
stop_one(args.service)
else:
{"restart": restart, "restore": restore, "rolling-update": rolling_update}[args.action]()
if __name__ == "__main__":
main()
+4
View File
@@ -0,0 +1,4 @@
@echo off
cd /d "%~dp0.."
python scripts\ops.py restart
exit /b %errorlevel%
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec python3 scripts/ops.py restart
+4
View File
@@ -0,0 +1,4 @@
@echo off
cd /d "%~dp0.."
python scripts\ops.py stop-one
exit /b %errorlevel%
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec python3 scripts/ops.py stop-one
+3 -3
View File
@@ -6,17 +6,17 @@ from openapi_spec_validator import validate
from jsonschema import Draft202012Validator
root = Path(__file__).resolve().parents[1]
spec = json.loads((root/'contracts/openapi.json').read_text())
spec = json.loads((root/'contracts/openapi.json').read_text(encoding='utf-8'))
validate(spec)
from referencing import Registry, Resource
# The base URI resolves local #/components references inside examples.
resource = dict(spec, **{'$schema': 'https://json-schema.org/draft/2020-12/schema'})
registry = Registry().with_resource('urn:mtusi:openapi', Resource.from_contents(resource))
for name, example in json.loads((root/'contracts/examples.json').read_text()).items():
for name, example in json.loads((root/'contracts/examples.json').read_text(encoding='utf-8')).items():
schema = {'$ref': 'urn:mtusi:openapi#/components/schemas/' + name}
Draft202012Validator(schema, registry=registry).validate(example)
for path in (root/'contracts').glob('*.schema.json'):
schema = json.loads(path.read_text())
schema = json.loads(path.read_text(encoding='utf-8'))
Draft202012Validator.check_schema(schema)
for example in schema.get('examples',[]):
Draft202012Validator(schema).validate(example)
View File
+24
View File
@@ -0,0 +1,24 @@
"""Entry point: uvicorn as PID 1 with env-driven host/port and graceful shutdown."""
import os
import uvicorn
from .main import app
def main() -> None:
host = os.environ.get("APP_HOST", "0.0.0.0")
port = int(os.environ.get("APP_PORT", "8080"))
uvicorn.run(
app,
host=host,
port=port,
log_level="info",
access_log=True,
timeout_keep_alive=2,
timeout_graceful_shutdown=5,
)
if __name__ == "__main__":
main()
+29
View File
@@ -0,0 +1,29 @@
import os
import secrets
import redis.asyncio as redis
from passlib.context import CryptContext
# bcrypt_sha256 avoids bcrypt's 72-byte limit. Old lab2 hashes still verify.
pwd_context = CryptContext(schemes=["bcrypt_sha256", "bcrypt"], deprecated="auto")
CACHE_URL = os.environ.get("CACHE_URL", "redis://localhost:6379/0")
SESSION_TTL_SECONDS = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
redis_client = redis.from_url(CACHE_URL, decode_responses=True,
socket_connect_timeout=1, socket_timeout=1)
def hash_password(password: str) -> str:
return pwd_context.hash(password)
def verify_password(plain_password: str, hashed_password: str) -> bool:
return pwd_context.verify(plain_password, hashed_password)
async def create_session(user_id: str) -> str:
token = secrets.token_urlsafe(32)
await redis_client.setex(f"session:{token}", SESSION_TTL_SECONDS, user_id)
return token
async def get_user_id_by_token(token: str) -> str | None:
return await redis_client.get(f"session:{token}")
async def delete_session(token: str) -> None:
await redis_client.delete(f"session:{token}")
+110
View File
@@ -0,0 +1,110 @@
import os
from datetime import datetime, timezone
from typing import List
from uuid import UUID, uuid4
from sqlalchemy import String, ForeignKey, DateTime, Index, Integer, BigInteger
from sqlalchemy.dialects.postgresql import UUID as PG_UUID, JSONB
from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
DATABASE_URL = os.environ.get("DATABASE_URL", "postgresql+asyncpg://user:password@localhost:5432/messenger")
engine = create_async_engine(
DATABASE_URL, echo=False, pool_pre_ping=True, pool_size=5, max_overflow=5,
pool_timeout=2, connect_args={"timeout": 1, "command_timeout": 3,
"server_settings": {"statement_timeout": "2500", "lock_timeout": "2000"}},
)
async_session = async_sessionmaker(engine, expire_on_commit=False)
class Base(DeclarativeBase):
pass
class User(Base):
__tablename__ = "users"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
username: Mapped[str] = mapped_column(String(32), unique=True, index=True)
display_name: Mapped[str] = mapped_column(String(100))
password_hash: Mapped[str] = mapped_column(String(255))
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
class Chat(Base):
__tablename__ = "chats"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
title: Mapped[str] = mapped_column(String(100))
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
members: Mapped[List["ChatMember"]] = relationship(back_populates="chat", lazy="selectin")
class ChatMember(Base):
__tablename__ = "chat_members"
chat_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), primary_key=True)
user_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), primary_key=True)
chat: Mapped["Chat"] = relationship(back_populates="members")
user: Mapped["User"] = relationship()
class Message(Base):
__tablename__ = "messages"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
chat_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), index=True)
sender_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("users.id"))
text: Mapped[str] = mapped_column(String(4000))
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
__table_args__ = (
Index('ix_messages_chat_created_id', 'chat_id', 'created_at', 'id'),
)
class IdempotencyKey(Base):
__tablename__ = "idempotency_keys"
user_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("users.id"), primary_key=True)
chat_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("chats.id"), primary_key=True)
key: Mapped[str] = mapped_column(String(128), primary_key=True)
request_text: Mapped[str] = mapped_column(String(4000))
response: Mapped[dict | None] = mapped_column(JSONB, nullable=True)
request_attachment_ids: Mapped[list] = mapped_column(JSONB, default=list, nullable=False)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
class SchemaVersion(Base):
__tablename__ = "schema_versions"
version: Mapped[int] = mapped_column(Integer, primary_key=True)
class Attachment(Base):
__tablename__ = "attachments"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
chat_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("chats.id"), index=True, nullable=False)
owner_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
filename: Mapped[str] = mapped_column(String(255), nullable=False)
object_key: Mapped[str] = mapped_column(String(250), unique=True, nullable=False)
content_type: Mapped[str] = mapped_column(String(32), nullable=False)
size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
status: Mapped[str] = mapped_column(String(16), default="queued", nullable=False)
error_code: Mapped[str | None] = mapped_column(String(64), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc), nullable=False)
class MessageAttachment(Base):
__tablename__ = "message_attachments"
message_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), primary_key=True)
attachment_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("attachments.id"), primary_key=True)
position: Mapped[int] = mapped_column(Integer, nullable=False)
# --- НОВЫЕ ТАБЛИЦЫ ДЛЯ 5-Й ЛАБЫ ---
class AttachmentVariant(Base):
__tablename__ = "attachment_variants"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
attachment_id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), ForeignKey("attachments.id", ondelete="CASCADE"), index=True)
name: Mapped[str] = mapped_column(String(32), nullable=False)
object_key: Mapped[str] = mapped_column(String(250), unique=True, nullable=False)
content_type: Mapped[str] = mapped_column(String(32), nullable=False)
size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
width: Mapped[int] = mapped_column(Integer, nullable=False)
height: Mapped[int] = mapped_column(Integer, nullable=False)
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
class Outbox(Base):
__tablename__ = "outbox"
id: Mapped[UUID] = mapped_column(PG_UUID(as_uuid=True), primary_key=True, default=uuid4)
event_type: Mapped[str] = mapped_column(String(64), nullable=False)
payload: Mapped[dict] = mapped_column(JSONB, nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)
processed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+85
View File
@@ -0,0 +1,85 @@
"""Validate compressed image bytes without trusting filename or HTTP Content-Type."""
import os
import threading
import struct
import zlib
import io
import hashlib
from PIL import Image, ImageFile, UnidentifiedImageError, ImageOps
from .storage import MAX_IMAGE_PIXELS
# Prevent unrestricted decompression of malicious image dimensions.
Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
ImageFile.LOAD_TRUNCATED_IMAGES = False
_IMAGE_LOCK = threading.BoundedSemaphore(1)
class InvalidImage(Exception):
pass
class TooManyPixels(Exception):
pass
def validate_image_file(file, declared_type: str) -> None:
"""Проверяет, что файл является валидным PNG/JPEG и не превышает лимиты."""
expected = {"image/png": "PNG", "image/jpeg": "JPEG"}.get(declared_type)
if not expected:
raise InvalidImage("unsupported image type")
with _IMAGE_LOCK:
try:
file.seek(0)
with Image.open(file) as probe:
if probe.format != expected:
raise InvalidImage("image MIME does not match its content")
width, height = probe.size
if width <= 0 or height <= 0:
raise InvalidImage("image dimensions are invalid")
if width * height > MAX_IMAGE_PIXELS:
raise TooManyPixels("image dimensions exceed limit")
probe.verify() # verify compressed structure
file.seek(0)
with Image.open(file) as loaded:
loaded.load() # detect damaged truncated payloads
except Image.DecompressionBombError as e:
raise TooManyPixels("image dimensions exceed limit") from e
except Image.DecompressionBombWarning as e:
raise TooManyPixels("image dimensions exceed limit") from e
except (UnidentifiedImageError, OSError, ValueError, SyntaxError, EOFError, struct.error, zlib.error) as e:
raise InvalidImage("invalid or truncated PNG/JPEG") from e
finally:
file.seek(0)
def make_thumbnail(file_bytes: bytes) -> tuple[bytes, int, int, str]:
"""
Создает JPEG-превью (вписанное в 256x256) из сырых байтов.
Возвращает: (байты_превью, ширина, высота, sha256_превью).
"""
with _IMAGE_LOCK:
with Image.open(io.BytesIO(file_bytes)) as img:
# 1. Применяем EXIF orientation (если картинку сфоткали на телефон боком)
img = ImageOps.exif_transpose(img)
# 2. Компонуем прозрачность (RGBA, LA) на белый фон
if img.mode in ("RGBA", "LA") or (img.mode == "P" and "transparency" in img.info):
bg = Image.new("RGB", img.size, (255, 255, 255))
bg.paste(img, mask=img.convert("RGBA").split()[3])
img = bg
elif img.mode != "RGB":
# На всякий случай конвертируем все остальные режимы (например, CMYK) в RGB
img = img.convert("RGB")
# 3. Вписываем в квадрат 256x256 (метод thumbnail сохраняет пропорции и не увеличивает маленькие картинки)
img.thumbnail((256, 256), Image.Resampling.LANCZOS)
width, height = img.size
# 4. Сохраняем в JPEG
out_buf = io.BytesIO()
img.save(out_buf, format="JPEG", quality=85)
data = out_buf.getvalue()
# 5. Считаем хэш для метаданных
sha256 = hashlib.sha256(data).hexdigest()
return data, width, height, sha256
+548
View File
@@ -0,0 +1,548 @@
from __future__ import annotations
import base64
import hashlib
from tempfile import SpooledTemporaryFile
from botocore.exceptions import ClientError, BotoCoreError
import asyncio
import binascii
import json
import logging
import os
import re
import socket
import uuid
from contextlib import asynccontextmanager
from datetime import datetime, timezone, timedelta
from typing import Optional
import aio_pika
from fastapi import Depends, FastAPI, Request, Query
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse, StreamingResponse
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from pydantic import BaseModel, ConfigDict, Field, field_validator
from sqlalchemy import select, and_, or_, text
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.exc import IntegrityError, SQLAlchemyError
from starlette.concurrency import run_in_threadpool
from starlette.exceptions import HTTPException as StarletteHTTPException
from redis.exceptions import RedisError
from asyncpg import PostgresError
from .db import engine, async_session, User, Chat, ChatMember, Message, IdempotencyKey, Attachment, MessageAttachment, Outbox, AttachmentVariant
from .auth import hash_password, verify_password, create_session, get_user_id_by_token, delete_session, redis_client, SESSION_TTL_SECONDS
from .storage import s3, bucket, check_storage, upload_original, delete_original, download_original, StorageFailure, MAX_UPLOAD_BYTES
from .images import validate_image_file, InvalidImage, TooManyPixels
USERNAME_RE = re.compile(r"^[a-z0-9_]{3,32}$")
KEY_RE = re.compile(r"[A-Za-z0-9._:-]{1,128}", re.ASCII)
INSTANCE_ID = socket.gethostname() + ":" + str(os.getpid())
BROKER_URL = os.getenv("BROKER_URL", "amqp://guest:guest@broker:5672/")
QUEUE_NAME = "attachment.process.v1"
def now_iso(dt: datetime = None) -> str:
if not dt:
dt = datetime.now(timezone.utc)
return dt.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
def error_payload(request_id: str, code: str, message: str) -> dict:
return {"error": {"code": code, "message": message, "request_id": request_id}}
class ApiError(Exception):
def __init__(self, status: int, code: str, message: str):
self.status = status
self.code = code
self.message = message
super().__init__(message)
# --- Курсоры для пагинации ---
def _encode_cursor(scope: str, created_at: datetime, obj_id: uuid.UUID) -> str:
raw = f"{scope}|{created_at.isoformat()}|{obj_id}".encode("utf-8")
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
def _decode_cursor(cursor: str) -> tuple[str, datetime, uuid.UUID]:
try:
padded = cursor + "=" * (-len(cursor) % 4)
raw = base64.urlsafe_b64decode(padded).decode("utf-8")
scope, dt_str, obj_id_str = raw.split("|", 2)
dt = datetime.fromisoformat(dt_str)
if dt.tzinfo is None:
raise ValueError("timezone required")
return scope, dt, uuid.UUID(obj_id_str)
except Exception:
raise ApiError(400, "invalid_request", "invalid cursor")
# --- Фоновая задача: Relayer для Outbox ---
async def outbox_relayer():
while True:
try:
connection = await aio_pika.connect_robust(BROKER_URL)
channel = await connection.channel()
await channel.declare_queue(QUEUE_NAME, durable=True)
break # Успешно подключились
except Exception as e:
logging.error(f"Relayer failed to connect to RabbitMQ: {e}. Retrying in 5s...")
await asyncio.sleep(5)
logging.info("Outbox relayer started.")
while True:
try:
async with async_session() as session:
# Блокируем необработанные записи, чтобы два API не отправили одно и то же
records = (await session.scalars(
select(Outbox)
.where(Outbox.processed_at == None)
.with_for_update(skip_locked=True)
.limit(10)
)).all()
for record in records:
await channel.default_exchange.publish(
aio_pika.Message(body=json.dumps(record.payload).encode()),
routing_key=QUEUE_NAME,
)
record.processed_at = datetime.now(timezone.utc)
await session.commit()
except Exception as e:
logging.error(f"Outbox relayer loop error: {e}")
await asyncio.sleep(2)
await asyncio.sleep(0.5) # Небольшая пауза между опросами БД
@asynccontextmanager
async def lifespan(app: FastAPI):
task = asyncio.create_task(outbox_relayer())
yield
task.cancel()
try:
await task
except asyncio.CancelledError:
pass
await redis_client.aclose()
await engine.dispose()
app = FastAPI(title="Messenger", version="1.5.0", lifespan=lifespan)
security_bearer = HTTPBearer(auto_error=False)
@app.middleware("http")
async def request_id_middleware(request: Request, call_next):
request_id = request.headers.get("X-Request-Id") or str(uuid.uuid4())
request.state.request_id = request_id
try:
response = await call_next(request)
except Exception:
logging.getLogger(__name__).exception("Request failed: %s", request_id)
response = JSONResponse(status_code=500, content=error_payload(request_id, "internal_error", "internal error"))
response.headers["X-Request-Id"] = request_id
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
def _rid(request: Request) -> str:
return getattr(request.state, "request_id", None) or str(uuid.uuid4())
# --- Обработчики ошибок ---
@app.exception_handler(ApiError)
async def api_error_handler(request: Request, exc: ApiError):
headers = {}
if exc.status == 401:
if request.url.path == "/api/v1/auth/sessions":
headers["WWW-Authenticate"] = 'Basic realm="messenger", charset="UTF-8"'
else:
headers["WWW-Authenticate"] = 'Bearer realm="messenger"'
return JSONResponse(status_code=exc.status, content=error_payload(_rid(request), exc.code, exc.message), headers=headers)
@app.exception_handler(RequestValidationError)
async def validation_error_handler(request: Request, exc: RequestValidationError):
return JSONResponse(status_code=400, content=error_payload(_rid(request), "invalid_request", "invalid request"))
@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(request: Request, exc: StarletteHTTPException):
return JSONResponse(status_code=exc.status_code, content=error_payload(_rid(request), "not_found", str(exc.detail)))
@app.exception_handler(InvalidImage)
async def invalid_image_handler(request: Request, exc: InvalidImage):
return JSONResponse(status_code=415, content=error_payload(_rid(request), "unsupported_media_type", str(exc)))
@app.exception_handler(TooManyPixels)
async def too_large_image_handler(request: Request, exc: TooManyPixels):
return JSONResponse(status_code=413, content=error_payload(_rid(request), "payload_too_large", str(exc)))
@app.exception_handler(RedisError)
@app.exception_handler(SQLAlchemyError)
@app.exception_handler(PostgresError)
@app.exception_handler(OSError)
@app.exception_handler(TimeoutError)
@app.exception_handler(ClientError)
@app.exception_handler(BotoCoreError)
@app.exception_handler(StorageFailure)
async def dependency_error_handler(request: Request, exc: Exception):
return JSONResponse(status_code=503, content=error_payload(_rid(request), "unavailable", "dependency unavailable"))
@app.exception_handler(Exception)
async def generic_exception_handler(request: Request, exc: Exception):
return JSONResponse(status_code=500, content=error_payload(_rid(request), "internal_error", "internal error"))
# --- Схемы ---
class CreateUser(BaseModel):
model_config = ConfigDict(extra="forbid")
username: str = Field(strict=True)
display_name: str = Field(strict=True, min_length=1, max_length=100)
password: str = Field(strict=True, min_length=12, max_length=128)
@field_validator("display_name", "password")
@classmethod
def not_blank(cls, value):
if not value.strip():
raise ValueError("blank value")
return value
class CreateChat(BaseModel):
model_config = ConfigDict(extra="forbid")
title: str = Field(strict=True, min_length=1, max_length=100)
member_ids: list[uuid.UUID] = Field(min_length=1, max_length=100)
@field_validator("title")
@classmethod
def not_blank(cls, value):
if not value.strip():
raise ValueError("blank title")
return value
class CreateMessage(BaseModel):
model_config = ConfigDict(extra="forbid")
text: str = Field(strict=True, max_length=4000)
attachment_ids: list[uuid.UUID] = Field(default_factory=list, max_length=10)
@field_validator("attachment_ids")
@classmethod
def unique_attachments(cls, value):
if len(set(value)) != len(value):
raise ValueError("duplicate attachment id")
return value
async def get_actor(credentials: HTTPAuthorizationCredentials = Depends(security_bearer)) -> dict:
if not credentials:
raise ApiError(401, "unauthorized", "missing token")
user_id_str = await get_user_id_by_token(credentials.credentials)
if not user_id_str:
raise ApiError(401, "unauthorized", "invalid or expired token")
async with async_session() as session:
user = await session.get(User, uuid.UUID(user_id_str))
if not user:
raise ApiError(401, "unauthorized", "user not found")
return {"id": str(user.id), "username": user.username, "display_name": user.display_name, "created_at": now_iso(user.created_at)}
async def require_chat_member(session, chat_id: uuid.UUID, user_id: str):
member = await session.scalar(select(ChatMember).where(and_(
ChatMember.chat_id == chat_id, ChatMember.user_id == uuid.UUID(user_id))))
if not member:
raise ApiError(404, "not_found", "chat not found")
@app.get("/health/live")
def liveness():
return {"status": "ok"}
@app.get("/health/ready")
async def readiness():
async with asyncio.timeout(1.5):
async with async_session() as session:
version = await session.scalar(text("SELECT version FROM schema_versions WHERE version = 5"))
if version != 5:
raise ApiError(503, "unavailable", "migration required")
await redis_client.ping()
await run_in_threadpool(check_storage)
return {"status": "ok"}
@app.post("/api/v1/users", status_code=201)
async def register_user(payload: CreateUser):
if not USERNAME_RE.fullmatch(payload.username):
raise ApiError(400, "invalid_request", "invalid username")
async with async_session() as session:
existing = await session.scalar(select(User).where(User.username == payload.username))
if existing:
raise ApiError(409, "conflict", "username already exists")
password_hash = await run_in_threadpool(hash_password, payload.password)
user = User(username=payload.username, display_name=payload.display_name, password_hash=password_hash)
session.add(user)
try:
await session.commit()
except IntegrityError:
await session.rollback()
raise ApiError(409, "conflict", "username already exists")
return {"id": str(user.id), "username": user.username, "display_name": user.display_name, "created_at": now_iso(user.created_at)}
@app.post("/api/v1/auth/sessions", status_code=201)
async def create_session_endpoint(request: Request):
try:
scheme, encoded = request.headers.get("Authorization", "").split(" ", 1)
if scheme.lower() != "basic":
raise ValueError("Basic required")
username, password = base64.b64decode(encoded, validate=True).decode("utf-8").split(":", 1)
except (ValueError, UnicodeError, binascii.Error):
raise ApiError(401, "unauthorized", "invalid credentials")
async with async_session() as session:
user = await session.scalar(select(User).where(User.username == username))
if not user or not await run_in_threadpool(verify_password, password, user.password_hash):
raise ApiError(401, "unauthorized", "invalid credentials")
token = await create_session(str(user.id))
expires_at = datetime.now(timezone.utc) + timedelta(seconds=SESSION_TTL_SECONDS)
return {
"token": token, "token_type": "Bearer", "expires_at": now_iso(expires_at),
"user": {"id": str(user.id), "username": user.username, "display_name": user.display_name, "created_at": now_iso(user.created_at)}
}
@app.delete("/api/v1/auth/sessions/current", status_code=204)
async def logout(credentials: HTTPAuthorizationCredentials = Depends(security_bearer)):
if not credentials:
raise ApiError(401, "unauthorized", "missing token")
user_id = await get_user_id_by_token(credentials.credentials)
if not user_id:
raise ApiError(401, "unauthorized", "invalid or expired token")
await delete_session(credentials.credentials)
@app.get("/api/v1/users/me")
async def current_user(actor: dict = Depends(get_actor)):
return actor
@app.post("/api/v1/chats", status_code=201)
async def create_chat(payload: CreateChat, actor: dict = Depends(get_actor)):
member_ids = [str(u) for u in payload.member_ids]
if actor["id"] not in member_ids:
raise ApiError(400, "invalid_request", "creator must be a member")
if len(set(member_ids)) != len(member_ids):
raise ApiError(400, "invalid_request", "duplicate members")
async with async_session() as session:
users = await session.scalars(select(User.id).where(User.id.in_([uuid.UUID(m) for m in member_ids])))
if len(list(users)) != len(set(member_ids)):
raise ApiError(400, "invalid_request", "unknown member")
chat = Chat(title=payload.title)
session.add(chat)
await session.flush()
for mid in member_ids:
session.add(ChatMember(chat_id=chat.id, user_id=uuid.UUID(mid)))
await session.commit()
return {"id": str(chat.id), "title": chat.title, "member_ids": member_ids, "created_at": now_iso(chat.created_at)}
@app.get("/api/v1/chats")
async def list_chats(limit: int = Query(50, ge=1, le=100), cursor: Optional[str] = None, actor: dict = Depends(get_actor)):
async with async_session() as session:
stmt = select(Chat).join(ChatMember).where(ChatMember.user_id == uuid.UUID(actor["id"]))
if cursor:
scope, c_dt, c_id = _decode_cursor(cursor)
if scope != "chats":
raise ApiError(400, "invalid_request", "invalid cursor scope")
stmt = stmt.where(or_(Chat.created_at > c_dt, and_(Chat.created_at == c_dt, Chat.id > c_id)))
stmt = stmt.order_by(Chat.created_at.asc(), Chat.id.asc()).limit(limit + 1)
chats = (await session.scalars(stmt)).unique().all()
has_more = len(chats) > limit
page = chats[:limit]
next_cursor = _encode_cursor("chats", page[-1].created_at, page[-1].id) if has_more and page else None
items = [{"id": str(c.id), "title": c.title, "created_at": now_iso(c.created_at), "member_ids": [str(m.user_id) for m in c.members]} for c in page]
return {"items": items, "next_cursor": next_cursor}
@app.get("/api/v1/chats/{chat_id}")
async def get_chat(chat_id: uuid.UUID, actor: dict = Depends(get_actor)):
async with async_session() as session:
chat = await session.scalar(select(Chat).join(ChatMember).where(Chat.id == chat_id, ChatMember.user_id == uuid.UUID(actor["id"])))
if not chat:
raise ApiError(404, "not_found", "chat not found")
return {"id": str(chat.id), "title": chat.title, "member_ids": [str(m.user_id) for m in chat.members], "created_at": now_iso(chat.created_at)}
def attachment_json(item: Attachment, variants: list[AttachmentVariant] = None) -> dict:
return {
"id": str(item.id), "chat_id": str(item.chat_id),
"owner_id": str(item.owner_id), "filename": item.filename,
"content_type": item.content_type, "size_bytes": item.size_bytes,
"sha256": item.sha256, "status": item.status,
"variants": [{
"name": v.name, "content_type": v.content_type, "size_bytes": v.size_bytes,
"width": v.width, "height": v.height, "sha256": v.sha256
} for v in (variants or [])],
"error_code": item.error_code,
"created_at": now_iso(item.created_at), "updated_at": now_iso(item.updated_at),
}
@app.post("/api/v1/chats/{chat_id}/attachments", status_code=202)
async def upload_attachment(
chat_id: uuid.UUID, request: Request, filename: str = Query(...),
actor: dict = Depends(get_actor),
):
async with async_session() as session:
await require_chat_member(session, chat_id, actor["id"])
if not (1 <= len(filename) <= 255) or any(c in filename for c in ("/", "\\")) or any(ord(c) < 32 or ord(c) == 127 for c in filename):
raise ApiError(400, "invalid_request", "invalid filename")
declared_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
if declared_type not in ("image/png", "image/jpeg"):
raise ApiError(415, "unsupported_media_type", "only PNG and JPEG are accepted")
hasher = hashlib.sha256()
size = 0
with SpooledTemporaryFile(max_size=1024 * 1024, mode="w+b") as original:
async for chunk in request.stream():
size += len(chunk)
if size > MAX_UPLOAD_BYTES:
raise ApiError(413, "payload_too_large", "image exceeds 10 MiB")
hasher.update(chunk)
original.write(chunk)
if size == 0:
raise ApiError(415, "unsupported_media_type", "empty image")
await run_in_threadpool(validate_image_file, original, declared_type)
item_id = uuid.uuid4()
key = f"originals/{item_id.hex}/{uuid.uuid4().hex}"
await run_in_threadpool(upload_original, original, key, size, declared_type)
try:
async with async_session() as session:
await require_chat_member(session, chat_id, actor["id"])
item = Attachment(
id=item_id, chat_id=chat_id, owner_id=uuid.UUID(actor["id"]),
filename=filename, object_key=key, content_type=declared_type,
size_bytes=size, sha256=hasher.hexdigest(), status="queued",
)
session.add(item)
# Добавляем задачу в Outbox
session.add(Outbox(event_type="attachment.process.v1", payload={"attachment_id": str(item_id)}))
await session.commit()
return attachment_json(item)
except Exception:
try:
await run_in_threadpool(delete_original, key)
except Exception:
logging.getLogger(__name__).warning("S3 compensation failed; reconcile needed")
raise
@app.get("/api/v1/attachments/{attachment_id}")
async def attachment_metadata(attachment_id: uuid.UUID, actor: dict = Depends(get_actor)):
async with async_session() as session:
item = await session.get(Attachment, attachment_id)
if item is None:
raise ApiError(404, "not_found", "attachment not found")
await require_chat_member(session, item.chat_id, actor["id"])
variants = (await session.scalars(select(AttachmentVariant).where(AttachmentVariant.attachment_id == item.id))).all()
return attachment_json(item, variants)
@app.get("/api/v1/attachments/{attachment_id}/content")
async def attachment_bytes(
attachment_id: uuid.UUID, variant: str = "original", actor: dict = Depends(get_actor)
):
async with async_session() as session:
item = await session.get(Attachment, attachment_id)
if item is None:
raise ApiError(404, "not_found", "attachment not found")
await require_chat_member(session, item.chat_id, actor["id"])
if item.status != "ready":
raise ApiError(409, "conflict", "attachment is not ready")
if variant == "original":
key, content_type, size = item.object_key, item.content_type, item.size_bytes
else:
var = await session.scalar(select(AttachmentVariant).where(
AttachmentVariant.attachment_id == item.id, AttachmentVariant.name == variant
))
if not var:
raise ApiError(404, "not_found", "variant not found")
key, content_type, size = var.object_key, var.content_type, var.size_bytes
body = await run_in_threadpool(download_original, key)
def chunks():
try:
yield from body.iter_chunks(chunk_size=64 * 1024)
finally:
body.close()
return StreamingResponse(chunks(), media_type=content_type,
headers={"Content-Length": str(size), "Cache-Control": "private, no-store"})
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201)
async def post_message(chat_id: uuid.UUID, payload: CreateMessage, request: Request, actor: dict = Depends(get_actor)):
key = request.headers.get("Idempotency-Key", "")
if not KEY_RE.fullmatch(key):
raise ApiError(400, "invalid_request", "invalid Idempotency-Key")
if not payload.text.strip() and not payload.attachment_ids:
raise ApiError(400, "invalid_request", "empty message")
attachment_ids = [str(a) for a in payload.attachment_ids]
async with async_session() as session:
await require_chat_member(session, chat_id, actor["id"])
if attachment_ids:
attachments = (await session.scalars(select(Attachment).where(
Attachment.id.in_(payload.attachment_ids), Attachment.chat_id == chat_id
))).all()
if len(attachments) != len(attachment_ids):
raise ApiError(404, "not_found", "attachment not found")
# В 5-й лабе можно прикреплять файлы в статусе queued/processing, но не failed
if any(a.status == "failed" for a in attachments):
raise ApiError(409, "conflict", "attachment failed and cannot be attached")
now = datetime.now(timezone.utc)
scope = {"user_id": uuid.UUID(actor["id"]), "chat_id": chat_id, "key": key}
await session.execute(insert(IdempotencyKey).values(
**scope, request_text=payload.text, request_attachment_ids=attachment_ids,
expires_at=now + timedelta(hours=24)
).on_conflict_do_nothing())
record = await session.scalar(select(IdempotencyKey).where(
IdempotencyKey.user_id == scope["user_id"],
IdempotencyKey.chat_id == chat_id, IdempotencyKey.key == key
).with_for_update())
if record.response is not None and record.expires_at > now:
if record.request_text != payload.text or record.request_attachment_ids != attachment_ids:
raise ApiError(409, "conflict", "key used with another message")
return record.response
msg = Message(chat_id=chat_id, sender_id=uuid.UUID(actor["id"]), text=payload.text)
session.add(msg)
await session.flush()
for pos, a_id in enumerate(payload.attachment_ids):
session.add(MessageAttachment(message_id=msg.id, attachment_id=a_id, position=pos))
result = {
"id": str(msg.id), "chat_id": str(msg.chat_id), "sender_id": str(msg.sender_id),
"text": msg.text, "created_at": now_iso(msg.created_at),
"attachment_ids": attachment_ids,
}
record.request_text = payload.text
record.request_attachment_ids = attachment_ids
record.expires_at = now + timedelta(hours=24)
record.response = result
await session.commit()
return result
@app.get("/api/v1/chats/{chat_id}/messages")
async def get_messages(chat_id: uuid.UUID, limit: int = Query(50, ge=1, le=100), cursor: Optional[str] = None, actor: dict = Depends(get_actor)):
async with async_session() as session:
await require_chat_member(session, chat_id, actor["id"])
stmt = select(Message).where(Message.chat_id == chat_id)
if cursor:
scope, c_dt, c_id = _decode_cursor(cursor)
if scope != str(chat_id):
raise ApiError(400, "invalid_request", "invalid cursor scope")
stmt = stmt.where(or_(Message.created_at > c_dt, and_(Message.created_at == c_dt, Message.id > c_id)))
stmt = stmt.order_by(Message.created_at.asc(), Message.id.asc()).limit(limit + 1)
messages = (await session.scalars(stmt)).all()
has_more = len(messages) > limit
page = messages[:limit]
next_cursor = _encode_cursor(str(chat_id), page[-1].created_at, page[-1].id) if has_more and page else None
msg_ids = [m.id for m in page]
links = []
if msg_ids:
links = (await session.execute(select(MessageAttachment).where(
MessageAttachment.message_id.in_(msg_ids)
).order_by(MessageAttachment.message_id, MessageAttachment.position))).scalars().all()
linked: dict[uuid.UUID, list[str]] = {m.id: [] for m in page}
for link in links:
linked[link.message_id].append(str(link.attachment_id))
items = [{
"id": str(m.id), "chat_id": str(m.chat_id), "sender_id": str(m.sender_id),
"text": m.text, "created_at": now_iso(m.created_at),
"attachment_ids": linked[m.id],
} for m in page]
return {"items": items, "next_cursor": next_cursor}
+22
View File
@@ -0,0 +1,22 @@
import asyncio
from sqlalchemy import text
from .db import Base, engine
async def main():
async with engine.begin() as conn:
await conn.execute(text("SELECT pg_advisory_xact_lock(300300)"))
await conn.run_sync(Base.metadata.create_all)
await conn.execute(text("ALTER TABLE attachments ADD COLUMN IF NOT EXISTS error_code VARCHAR(64)"))
await conn.execute(text("INSERT INTO schema_versions(version) VALUES (3) ON CONFLICT DO NOTHING"))
await conn.execute(text("INSERT INTO schema_versions(version) VALUES (4) ON CONFLICT DO NOTHING"))
await conn.execute(text("INSERT INTO schema_versions(version) VALUES (5) ON CONFLICT DO NOTHING"))
await engine.dispose()
print("Migration 5 applied: outbox, variants and async processing added")
if __name__ == "__main__":
asyncio.run(main())
+22
View File
@@ -0,0 +1,22 @@
"""Prove S3 is not public and API-scoped credentials cannot list other buckets."""
import urllib.error
import urllib.request
from botocore.exceptions import ClientError
from .storage import s3, bucket
import os
import uuid
url = os.getenv("S3_ENDPOINT", "http://s3:8333").rstrip("/") + "/" + bucket + "/random-inaccessible-key"
try:
urllib.request.urlopen(url, timeout=4)
except urllib.error.HTTPError as exc:
assert exc.code in (401, 403), f"expected S3 denied, got {exc.code}"
else:
raise AssertionError("UNSAFE: direct unsigned S3 request succeeded")
try:
s3.create_bucket(Bucket="forbidden-probe-" + uuid.uuid4().hex[:16])
except ClientError as exc:
assert exc.response["Error"]["Code"] in ("AccessDenied", "403"), "unexpected S3 result"
else:
raise AssertionError("UNSAFE: API credentials have global create-bucket privileges")
print("PASS: S3 requires authentication; API credentials restricted to own bucket")
+51
View File
@@ -0,0 +1,51 @@
"""Safe orphan S3 cleanup (dry run default). Use inside api container.
A crash between S3 PUT and PostgreSQL COMMIT can leave an unreferenced object.
Only objects older than the grace period are candidates. No deletion if DB read fails.
"""
import argparse
import asyncio
from datetime import datetime, timezone, timedelta
from sqlalchemy import select
from .db import async_session, Attachment, engine
from .storage import s3, bucket
from starlette.concurrency import run_in_threadpool
def list_s3():
keys = []
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=bucket, Prefix="originals/"):
keys.extend((i["Key"], i["LastModified"]) for i in page.get("Contents", []))
return keys
async def run(apply=False, grace_seconds=3600):
if grace_seconds < 3600:
raise ValueError("grace must be at least 3600 seconds")
# Load DB references before enumerating/deleting S3 objects. Fail closed.
async with async_session() as session:
referenced = set((await session.scalars(select(Attachment.object_key))).all())
objects = await run_in_threadpool(list_s3)
threshold = datetime.now(timezone.utc) - timedelta(seconds=grace_seconds)
candidates = [key for key, when in objects if key not in referenced and when < threshold]
print(f"referenced={len(referenced)} objects={len(objects)} old_orphans={len(candidates)} apply={apply}")
for key in candidates:
# Avoid listing filenames/secrets: keys are random IDs, not user inputs.
if apply:
await run_in_threadpool(s3.delete_object, Bucket=bucket, Key=key)
print(("deleted " if apply else "would_delete ") + key)
return len(candidates)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--apply", action="store_true", help="really delete aged orphan objects")
parser.add_argument("--grace-seconds", type=int, default=3600)
args = parser.parse_args()
async def main():
try:
await run(args.apply, args.grace_seconds)
finally:
await engine.dispose()
asyncio.run(main())
+43
View File
@@ -0,0 +1,43 @@
"""S3-compatible private storage. Only API-scoped credentials are available here."""
import os
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError, BotoCoreError
MAX_UPLOAD_BYTES = int(os.getenv("MAX_UPLOAD_BYTES", "10485760"))
MAX_IMAGE_PIXELS = int(os.getenv("MAX_IMAGE_PIXELS", "40000000"))
bucket = os.getenv("S3_BUCKET", "messenger-attachments")
s3 = boto3.client(
"s3", endpoint_url=os.getenv("S3_ENDPOINT", "http://s3:8333"),
region_name=os.getenv("S3_REGION", "us-east-1"),
aws_access_key_id=os.environ.get("S3_ACCESS_KEY_ID", "unset"),
aws_secret_access_key=os.environ.get("S3_SECRET_ACCESS_KEY", "unset"),
config=Config(s3={"addressing_style": "path"}, connect_timeout=2, read_timeout=30,
retries={"mode": "standard", "max_attempts": 2}),
)
class StorageFailure(Exception):
pass
def check_storage():
# Scoped user has ListBucket but no privileges outside this bucket.
s3.list_objects_v2(Bucket=bucket, MaxKeys=1)
def upload_original(file, key, size, content_type):
file.seek(0)
s3.put_object(Bucket=bucket, Key=key, Body=file, ContentLength=size, ContentType=content_type)
def delete_original(key):
s3.delete_object(Bucket=bucket, Key=key)
def download_original(key):
try:
return s3.get_object(Bucket=bucket, Key=key)["Body"]
except ClientError as exc:
if exc.response.get("Error", {}).get("Code") in ("NoSuchKey", "404"):
raise StorageFailure("Object missing") from None
raise
+84
View File
@@ -0,0 +1,84 @@
import asyncio
import json
import logging
import os
import uuid
import io
import aio_pika
from .db import async_session, engine, Attachment, AttachmentVariant
from .storage import upload_original, download_original
from .images import make_thumbnail
BROKER_URL = os.getenv("BROKER_URL", "amqp://guest:guest@broker:5672/")
QUEUE_NAME = "attachment.process.v1"
logger = logging.getLogger(__name__)
async def process_attachment(message: aio_pika.IncomingMessage):
async with message.process(): # ack случится автоматически, если не будет ошибки
data = json.loads(message.body.decode())
attachment_id = uuid.UUID(data["attachment_id"])
logger.info(f"Processing attachment {attachment_id}")
async with async_session() as session:
att = await session.get(Attachment, attachment_id)
if not att or att.status != "queued":
return # Уже обработано или удалено
# Ставим статус processing
att.status = "processing"
await session.commit()
try:
# 1. Скачиваем оригинал из S3
body = await asyncio.to_thread(download_original, att.object_key)
file_bytes = body.read()
body.close()
# 2. Делаем превью
thumb_bytes, w, h, sha = await asyncio.to_thread(make_thumbnail, file_bytes)
# 3. Загружаем превью в S3 (ВОТ ЗДЕСЬ ИСПРАВЛЕНИЕ)
thumb_key = f"variants/{attachment_id.hex}/thumbnail.jpg"
thumb_file_obj = io.BytesIO(thumb_bytes)
await asyncio.to_thread(upload_original, thumb_file_obj, thumb_key, len(thumb_bytes), "image/jpeg")
# 4. Сохраняем метаданные в БД
variant = AttachmentVariant(
attachment_id=attachment_id, name="thumbnail", object_key=thumb_key,
content_type="image/jpeg", size_bytes=len(thumb_bytes),
width=w, height=h, sha256=sha
)
session.add(variant)
att.status = "ready"
await session.commit()
logger.info(f"Attachment {attachment_id} is ready")
except Exception as e:
logger.exception(f"Failed to process {attachment_id}")
await session.rollback()
att = await session.get(Attachment, attachment_id)
att.status = "failed"
att.error_code = "processing_failed"
await session.commit()
async def main():
connection = await aio_pika.connect_robust(BROKER_URL)
channel = await connection.channel()
queue = await channel.declare_queue(QUEUE_NAME, durable=True)
await channel.set_qos(prefetch_count=1)
await queue.consume(process_attachment)
logger.info("Worker started. Waiting for messages...")
try:
await asyncio.Future() # run forever
finally:
await connection.close()
await engine.dispose()
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
asyncio.run(main())
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env python3
import sys
import threading
import uuid
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
from client import Client, arguments, check, run
def main():
args = arguments("Lab3: concurrent idempotency and validation")
c = Client(args.base_url, args.ca_file)
c.call("POST", "/api/v1/users", 400,
body={"username": "valid\n", "display_name": "Test", "password": "LongPassword123"})
registration = {"username": "race_" + uuid.uuid4().hex[:12],
"display_name": "Race", "password": "LongPassword123"}
registration_barrier = threading.Barrier(2)
def register(_):
local = Client(args.base_url, args.ca_file)
registration_barrier.wait()
return local.call("POST", "/api/v1/users", [201, 409],
body=registration, schema="User")["status"]
with ThreadPoolExecutor(max_workers=2) as pool:
check(sorted(pool.map(register, range(2))) == [201, 409], "registration race")
alice, bob, eve, chat = c.bootstrap()
path = "/api/v1/chats/" + chat["id"] + "/messages"
key = uuid.uuid4().hex
barrier = threading.Barrier(12)
def send(_):
local = Client(args.base_url, args.ca_file)
barrier.wait()
return local.message(alice, chat, "Одна запись", key=key)
with ThreadPoolExecutor(max_workers=12) as pool:
responses = list(pool.map(send, range(12)))
first = responses[0]["json"]
check(all(r["json"] == first for r in responses), "concurrent replay differs")
instances = {r["headers"]["X-Instance-Id"] for r in responses}
check(len(instances) >= 2, "same key must reach at least two replicas")
page = c.call("GET", path, actor=bob, schema="MessagePage")["json"]
check(page["items"] == [first], "concurrent retry created duplicates")
c.call("POST", path, 409, actor=alice, body={"text": "Изменённый текст"},
headers={"Idempotency-Key": key})
c.call("POST", path, 404, actor=eve, body={"text": "Одна запись"},
headers={"Idempotency-Key": key})
# The same key belongs to a different scope for Bob and for another chat.
other = c.message(bob, chat, "Другая область", key=key)["json"]
check(other["id"] != first["id"], "different users share an idempotency scope")
second_chat = c.call("POST", "/api/v1/chats", 201, actor=alice,
body={"title": "Второй", "member_ids": [alice["id"]]},
schema="Chat")["json"]
other = c.message(alice, second_chat, "Одна запись", key=key)["json"]
check(other["id"] != first["id"], "different chats share an idempotency scope")
for bad_key in ("", "bad key", "x" * 129):
c.call("POST", path, 400, actor=alice, body={"text": "Текст"},
headers={"Idempotency-Key": bad_key})
for body in ({"text": " "}, {"text": 123}, {"text": "x", "sender_id": alice["id"]}):
c.call("POST", path, 400, actor=alice, body=body,
headers={"Idempotency-Key": uuid.uuid4().hex})
c.call("POST", path, 400, actor=alice, raw=b'{"text":',
headers={"Content-Type": "application/json", "Idempotency-Key": uuid.uuid4().hex})
c.call("DELETE", "/api/v1/auth/sessions/current", 204, actor=alice)
c.call("POST", path, 401, actor=alice, body={"text": "Одна запись"},
headers={"Idempotency-Key": key})
print("PASS: 12 concurrent requests across %d replicas, one message" % len(instances))
print("PASS: conflict, key scope, validation, access and revoked session")
print("PASS: concurrent registration, exactly one 201 and one 409")
if __name__ == "__main__":
run(main)
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env python3
"""Additional grade-4 negative and access tests. Requires running Compose stack."""
import hashlib
import zlib
import sys
import uuid
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
from client import Client, check, arguments, run
ROOT = Path(__file__).resolve().parents[1]
def main():
args = arguments("Lab5: file validation, permissions, idempotency and isolation")
c = Client(args.base_url, args.ca_file, timeout=35)
alice, bob, eve, chat = c.bootstrap()
file = (ROOT / "tests/fixtures/sample.png").read_bytes()
path = "/api/v1/chats/" + chat["id"]
upload = path + "/attachments?filename=photo.png"
# 1. Загружаем (теперь ожидаем 202 вместо 201)
result = c.call("POST",upload,202,actor=alice,raw=file,
headers={"Content-Type":"image/png"},schema="Attachment")["json"]
obj = "/api/v1/attachments/" + result["id"]
check(result["sha256"] == hashlib.sha256(file).hexdigest(),"checksum mismatch")
# Ждем, пока воркер обработает картинку (переведет в статус ready)
result = c.wait_ready(alice, result)
# Теперь контент доступен
check(c.call("GET",obj+"/content",actor=bob)["bytes"] == file,"original differs")
c.call("GET",obj,404,actor=eve)
c.call("GET",obj+"/content",404,actor=eve)
c.call("GET","/api/v1/attachments/"+str(uuid.uuid4()),404,actor=bob)
# Негативные тесты (они падают сразу на этапе валидации, до 202)
c.call("POST",upload,415,actor=alice,raw=b"not an image",
headers={"Content-Type":"image/png"})
c.call("POST",upload,415,actor=alice,raw=file,
headers={"Content-Type":"image/jpeg"})
c.call("POST",upload,413,actor=alice,raw=b"\0"*10485761,
headers={"Content-Type":"image/png"})
huge = bytearray(file)
huge[16:20] = (10000).to_bytes(4, "big")
huge[20:24] = (5000).to_bytes(4, "big")
huge[29:33] = (zlib.crc32(huge[12:29]) & 0xffffffff).to_bytes(4, "big")
c.call("POST",upload,413,actor=alice,raw=bytes(huge),
headers={"Content-Type":"image/png"})
c.call("POST",path+"/attachments?filename=..%2Fhidden.png",400,actor=alice,
raw=file,headers={"Content-Type":"image/png"})
c.call("POST",path+"/attachments?filename=sample.txt",415,actor=alice,
raw=file,headers={"Content-Type":"text/plain"})
second = c.call("POST","/api/v1/chats",201,actor=alice,
body={"title":"Other Chat","member_ids":[alice["id"]]},schema="Chat")["json"]
c.call("POST", "/api/v1/chats/"+second["id"]+"/messages",404,
actor=alice,body={"text":"","attachment_ids":[result["id"]]},
headers={"Idempotency-Key":uuid.uuid4().hex})
# В 5-й лабе можно прикреплять файлы в статусе queued/processing/ready
attached = c.message(alice,chat,"",attachments=[result["id"]])["json"]
check(attached["attachment_ids"] == [result["id"]],"message lost attachment")
key = uuid.uuid4().hex
one = c.message(alice,chat,"Фото",key=key,attachments=[result["id"]])["json"]
check(c.message(alice,chat,"Фото",key=key,attachments=[result["id"]])["json"] == one,
"repeated idempotency response mismatch")
c.call("POST",path+"/messages",409,actor=alice,
body={"text":"Фото", "attachment_ids":[]},headers={"Idempotency-Key":key})
print("PASS: private downloads, MIME spoof, corrupt image, oversize, dangerous filename, foreign chat, idempotency")
if __name__ == "__main__":
run(main)