forked from mtusi2026/mtusi2026lab4
Добавить прямую загрузку по короткой подписи.
Proxy endpoint сохранён. Complete сверяет владельца, размер и checksum, а сравнение трафика API на двух размерах лежит в evidence.
This commit is contained in:
@@ -32,3 +32,6 @@ S3_SECRET_ACCESS_KEY=lab4-dev-only-change-me
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
ORPHAN_GRACE_SECONDS=120
|
||||
# Host the client uses for a presigned PUT. Loopback only; the API itself uses S3_ENDPOINT.
|
||||
S3_PUBLIC_ENDPOINT=http://127.0.0.1:9000
|
||||
PRESIGN_TTL_SECONDS=60
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
## Лабораторная 4
|
||||
|
||||
Автор / группа: Булацев, БПИ2301
|
||||
Целевая оценка (3/4/5): 4
|
||||
Трек 5, если заявлен: нет
|
||||
Целевая оценка (3/4/5): 5
|
||||
Трек 5, если заявлен: 1, прямой upload
|
||||
|
||||
Что реализовано и что перенесено: стенд lab3 (Postgres, Redis, HAProxy, две реплики API, идемпотентность) и вложения lab4 в SeaweedFS. Метаданные в Postgres, байты в приватном bucket, ключ назначает сервер.
|
||||
Что реализовано и что перенесено: стенд lab3, вложения в SeaweedFS и отдельный presigned upload. Proxy `POST /attachments` сохранён. Метаданные в Postgres, байты в приватном bucket.
|
||||
Команды запуска из чистого clone: `cp .env.example .env`, затем `docker compose --env-file .env up -d --build --scale api=2`. Подробности в REPORT.md.
|
||||
Результаты проверок и ссылка на REPORT.md: smoke, persistence и failover прошли; негативные загрузки и orphan/память — в `evidence/`. См. REPORT.md.
|
||||
Ограничения / известные ошибки: один Docker-хост; трек 5 не делался; образ MinIO на Docker Hub недоступен, используется SeaweedFS 3.85.
|
||||
Ограничения / известные ошибки: один Docker-хост; complete прямой загрузки читает объект обратно в API; образ MinIO на Docker Hub недоступен, используется SeaweedFS 3.85. S3 открыт только на 127.0.0.1:9000.
|
||||
Источники и использование ИИ: контракт репозитория, документация SeaweedFS, boto3, Pillow. Код и прогон проверок — с помощью Cursor.
|
||||
|
||||
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
|
||||
@@ -32,3 +32,6 @@ test-failover:
|
||||
grade4:
|
||||
$(PYTHON) scripts/grade4_checks.py
|
||||
$(PYTHON) scripts/grade4_infra.py
|
||||
|
||||
track1:
|
||||
$(PYTHON) scripts/track1_experiment.py
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Отчёт — лабораторная 4
|
||||
|
||||
- Автор, группа: Булацев, БПИ2301
|
||||
- Целевая оценка; выбранный трек 5, если нужен: 4. Трек 5 не заявляется.
|
||||
- Целевая оценка; выбранный трек 5, если нужен: 5, трек 1 «Прямой upload».
|
||||
- Commit/PR: локальные коммиты после переноса lab3; PR не публиковался в этой сессии
|
||||
- Использованные источники, библиотеки и помощь ИИ: контракт репозитория, документация SeaweedFS S3, boto3, Pillow. Реализация и прогон проверок сделаны с помощью Cursor.
|
||||
|
||||
@@ -26,6 +26,7 @@ python -X utf8 tests/resilience.py --mode persistence --action-script scripts/re
|
||||
python -X utf8 tests/resilience.py --mode failover --action-script scripts/stop-one.sh --base-url http://127.0.0.1:8080
|
||||
python -X utf8 scripts/grade4_checks.py
|
||||
python -X utf8 scripts/grade4_infra.py
|
||||
python -X utf8 scripts/track1_experiment.py
|
||||
```
|
||||
|
||||
Остановка без удаления данных: `docker compose --env-file .env stop`. Полное удаление учебных volumes: `docker compose --env-file .env down -v`.
|
||||
@@ -44,6 +45,8 @@ python -X utf8 scripts/grade4_infra.py
|
||||
|
||||
`/health/ready` зависит от Postgres, Redis и `HeadBucket`. Сессии по-прежнему в Redis (`SESSION_STORE=redis`), не в Postgres.
|
||||
|
||||
Прямая загрузка не заменяет `POST /attachments`. `POST /chats/{id}/uploads` выдаёт PUT-подпись на 60 секунд и ключ `originals/<uuid>`. В JSON ключа нет, он есть только внутри подписанного URL. `POST /uploads/{id}/complete` принимает загрузку, только если её начал тот же пользователь, подпись не истекла, размер совпал с заявленным, sha256 совпал с байтами в bucket, а содержимое — целый PNG или JPEG. Чужой complete — 404. Повторный complete, complete без PUT и просроченная подпись — 409. Неверная сумма удаляет объект и больше не даёт дозавершить загрузку. Незаконченный ключ снимает тот же sweeper, что и orphan после grace. S3 с хоста доступен только на `127.0.0.1:9000`, чтобы клиент мог сделать PUT мимо API; анонимный запрос по-прежнему 403.
|
||||
|
||||
| Роль | Сервис | Образ | Проверка | Volume | Порт |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| api | api ×2 | сборка `mtusi-messenger:lab4` | `/health/ready` | нет | только внутри сети |
|
||||
@@ -71,6 +74,8 @@ python -X utf8 scripts/grade4_infra.py
|
||||
| Ограниченные ключи | тот же скрипт | `create_bucket` другого имени — `AccessDenied` | `evidence/grade4-infra.json` |
|
||||
| Компенсация orphan | тот же скрипт, grace 120 с | старый незакоммиченный объект удалён, свежий оставлен | `evidence/grade4-infra.json` |
|
||||
| Память 10 загрузок | тот же скрипт, docker stats | 10×230778 байт, все 201, пик API 61.7 МиБ при лимите 256 | `evidence/grade4-infra.json` |
|
||||
| Трек 1, негативные сценарии | `scripts/track1_experiment.py` | подделанная подпись 403; complete без PUT, чужой пользователь, повтор и просрочка — 404/409; неверная сумма 400 и объект больше не завершается; proxy endpoint по-прежнему 201 | `evidence/track1.json` |
|
||||
| Трек 1, трафик API | тот же скрипт, сумма RX `eth0` обеих реплик | файл 90243 байт: proxy RX 96204, прямой PUT RX 0, complete RX 94131. Файл 1891338 байт: proxy RX 1913606, прямой PUT RX 960, complete RX 1900090 | `evidence/track1.json` |
|
||||
|
||||
`Content-Length: 10485761` при коротком теле отвечает 413 до дочитывания заявленного размера.
|
||||
|
||||
@@ -78,6 +83,8 @@ python -X utf8 scripts/grade4_infra.py
|
||||
|
||||
Гипотеза уровня 4: десять одновременных загрузок около 230 КиБ не поднимают RSS реплики к лимиту 256 МиБ, потому что тело ограничено 10 МиБ и не декодируется целиком до проверки пикселей. Один прогон, без прогрева отдельной фазой. Это не профиль RPS сообщений.
|
||||
|
||||
Гипотеза трека 1: прямой PUT не заходит в API, поэтому RX API почти не зависит от размера файла, а proxy RX растёт вместе с телом. Complete один раз скачивает объект, чтобы сверить sha256, и этот трафик остаётся. Пик RSS по `docker stats` на файлах 90 КиБ и 1.9 МиБ остался около 61–62.5 МиБ: базовая память процесса больше короткого буфера, и по RSS выигрыш на этих размерах не виден. Выигрыш, который измерен, — это RX на фазе PUT.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| `scripts/restart.sh` (postgres, redis, s3, api), volumes сохранены | те же сессия, сообщение и байты | PASS публичного persistence | готовность вернулась в пределах ожидания теста (90 с) |
|
||||
@@ -91,6 +98,6 @@ python -X utf8 scripts/grade4_infra.py
|
||||
|
||||
Скачивание и загрузка держат в памяти процесса не больше лимита 10 МиБ плюс проверку Pillow. Это граница, а не потоковая отдача клиенту минуя буфер. Эксперимент памяти сделан на файлах ~226 КиБ, не на десяти одновременных файлах по 10 МиБ.
|
||||
|
||||
Окно между PUT и COMMIT закрывается немедленным удалением при ошибке и обходом раз в 30 с. Объект, умерший внутри grace 120 с, временно занимает место. Прямой presigned upload и отдельный reconciler всего bucket не делались: это трек 5.
|
||||
Окно между PUT и COMMIT закрывается немедленным удалением при ошибке и обходом раз в 30 с. Объект, умерший внутри grace 120 с, временно занимает место. Прямой upload добавлен отдельными маршрутами и не отменяет proxy. Complete всё равно читает объект в API, поэтому экономия трафика относится к фазе PUT, а не ко всей операции. Отдельный reconciler всего bucket не делался.
|
||||
|
||||
Случайный ключ и закрытый bucket не заменяют проверку участника чата: скачивание идёт через API и для постороннего даёт 404, даже если он угадает UUID.
|
||||
|
||||
@@ -178,6 +178,14 @@ async def sweep_orphans() -> int:
|
||||
continue
|
||||
await s3.delete_key(key)
|
||||
await conn.execute("DELETE FROM blob_writes WHERE object_key = $1", key)
|
||||
await conn.execute(
|
||||
"""
|
||||
UPDATE direct_uploads
|
||||
SET status = 'aborted'
|
||||
WHERE object_key = $1 AND status = 'pending'
|
||||
""",
|
||||
key,
|
||||
)
|
||||
deleted += 1
|
||||
if deleted:
|
||||
logger.info("removed %s uncommitted objects past grace", deleted)
|
||||
|
||||
@@ -37,6 +37,8 @@ LB_DRAIN_SECONDS = float(_env("LB_DRAIN_SECONDS", "2"))
|
||||
# container uses its own hostname; stop-one.sh maps that value to one replica.
|
||||
INSTANCE_ID = _env("INSTANCE_ID", socket.gethostname())
|
||||
S3_ENDPOINT = _env("S3_ENDPOINT", "http://s3:9000")
|
||||
S3_PUBLIC_ENDPOINT = _env("S3_PUBLIC_ENDPOINT", "http://127.0.0.1:9000")
|
||||
PRESIGN_TTL_SECONDS = min(max(int(_env("PRESIGN_TTL_SECONDS", "60")), 15), 300)
|
||||
S3_REGION = _env("S3_REGION", "us-east-1")
|
||||
S3_BUCKET = _env("S3_BUCKET", "messenger-attachments")
|
||||
S3_ACCESS_KEY_ID = _file_or_env("S3_ACCESS_KEY_ID")
|
||||
|
||||
+199
@@ -0,0 +1,199 @@
|
||||
"""Presigned PUT that does not replace the proxy upload endpoint."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from app import config, db, images, s3
|
||||
from app.attachments import _compensate, _public
|
||||
from app.errors import AppError
|
||||
from app.store import format_ts
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
if value.tzinfo is None:
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
return value
|
||||
|
||||
|
||||
async def begin_upload(
|
||||
chat_id: str,
|
||||
owner_id: str,
|
||||
filename: str,
|
||||
content_type: str,
|
||||
size_bytes: int,
|
||||
) -> dict:
|
||||
upload_id = uuid4()
|
||||
key = f"originals/{upload_id}"
|
||||
created = datetime.now(timezone.utc)
|
||||
expires_at = created + timedelta(seconds=config.PRESIGN_TTL_SECONDS)
|
||||
async with db.require().acquire() as conn:
|
||||
async with conn.transaction():
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO blob_writes (object_key, committed, created_at)
|
||||
VALUES ($1, FALSE, $2)
|
||||
""",
|
||||
key,
|
||||
created,
|
||||
)
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO direct_uploads (
|
||||
id, chat_id, owner_id, filename, content_type, size_bytes,
|
||||
object_key, status, expires_at, created_at
|
||||
)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, 'pending', $8, $9)
|
||||
""",
|
||||
upload_id,
|
||||
UUID(chat_id),
|
||||
UUID(owner_id),
|
||||
filename,
|
||||
content_type,
|
||||
size_bytes,
|
||||
key,
|
||||
expires_at,
|
||||
created,
|
||||
)
|
||||
try:
|
||||
url = await asyncio.to_thread(s3.presign_put, key, content_type, size_bytes)
|
||||
except Exception:
|
||||
await _compensate(key)
|
||||
async with db.require().acquire() as conn:
|
||||
await conn.execute(
|
||||
"UPDATE direct_uploads SET status = 'aborted' WHERE id = $1 AND status = 'pending'",
|
||||
upload_id,
|
||||
)
|
||||
raise
|
||||
return {
|
||||
"upload_id": str(upload_id),
|
||||
"method": "PUT",
|
||||
"url": url,
|
||||
"headers": {"Content-Type": content_type},
|
||||
"expires_at": format_ts(expires_at),
|
||||
"size_bytes": size_bytes,
|
||||
}
|
||||
|
||||
|
||||
async def _abort_pending(upload_id: UUID, key: str) -> None:
|
||||
async with db.require().acquire() as conn:
|
||||
changed = await conn.fetchval(
|
||||
"""
|
||||
UPDATE direct_uploads
|
||||
SET status = 'aborted'
|
||||
WHERE id = $1 AND status = 'pending'
|
||||
RETURNING id
|
||||
""",
|
||||
upload_id,
|
||||
)
|
||||
if changed is not None:
|
||||
await _compensate(key)
|
||||
|
||||
|
||||
async def complete_upload(upload_id: str, owner_id: str, claimed_sha: str) -> dict:
|
||||
async with db.require().acquire() as conn:
|
||||
row = await conn.fetchrow(
|
||||
"""
|
||||
SELECT id, chat_id, owner_id, filename, content_type, size_bytes,
|
||||
object_key, status, expires_at
|
||||
FROM direct_uploads
|
||||
WHERE id = $1 AND owner_id = $2
|
||||
""",
|
||||
UUID(upload_id),
|
||||
UUID(owner_id),
|
||||
)
|
||||
if row is None:
|
||||
raise AppError(404, "upload not found")
|
||||
if row["status"] != "pending":
|
||||
raise AppError(409, "upload is already finished")
|
||||
if _aware(row["expires_at"]) <= datetime.now(timezone.utc):
|
||||
await _abort_pending(row["id"], row["object_key"])
|
||||
raise AppError(409, "upload signature expired")
|
||||
|
||||
actual_size = await s3.head_size(row["object_key"])
|
||||
if actual_size is None:
|
||||
raise AppError(409, "upload is incomplete")
|
||||
if actual_size != row["size_bytes"] or actual_size > config.MAX_UPLOAD_BYTES:
|
||||
await _abort_pending(row["id"], row["object_key"])
|
||||
raise AppError(400, "uploaded size does not match")
|
||||
|
||||
data = await s3.get_bytes(row["object_key"])
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
if len(data) != row["size_bytes"] or digest != claimed_sha:
|
||||
await _abort_pending(row["id"], row["object_key"])
|
||||
raise AppError(400, "checksum does not match")
|
||||
try:
|
||||
images.inspect_image(data, row["content_type"])
|
||||
except images.ImageRejected as exc:
|
||||
await _abort_pending(row["id"], row["object_key"])
|
||||
raise AppError(exc.status, exc.message) from None
|
||||
|
||||
created = datetime.now(timezone.utc)
|
||||
try:
|
||||
async with db.require().acquire() as conn:
|
||||
async with conn.transaction():
|
||||
locked = await conn.fetchrow(
|
||||
"""
|
||||
SELECT status, object_key
|
||||
FROM direct_uploads
|
||||
WHERE id = $1 AND owner_id = $2
|
||||
FOR UPDATE
|
||||
""",
|
||||
row["id"],
|
||||
UUID(owner_id),
|
||||
)
|
||||
if locked is None or locked["status"] != "pending":
|
||||
raise AppError(409, "upload is already finished")
|
||||
intent = await conn.fetchrow(
|
||||
"SELECT committed FROM blob_writes WHERE object_key = $1 FOR UPDATE",
|
||||
row["object_key"],
|
||||
)
|
||||
if intent is None or intent["committed"]:
|
||||
raise AppError(409, "upload is already finished")
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO attachments (
|
||||
id, chat_id, owner_id, filename, content_type, size_bytes,
|
||||
sha256, object_key, status, error_code, created_at, updated_at
|
||||
)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 'ready', NULL, $9, $9)
|
||||
""",
|
||||
row["id"],
|
||||
row["chat_id"],
|
||||
row["owner_id"],
|
||||
row["filename"],
|
||||
row["content_type"],
|
||||
len(data),
|
||||
digest,
|
||||
row["object_key"],
|
||||
created,
|
||||
)
|
||||
await conn.execute(
|
||||
"UPDATE blob_writes SET committed = TRUE WHERE object_key = $1",
|
||||
row["object_key"],
|
||||
)
|
||||
await conn.execute(
|
||||
"UPDATE direct_uploads SET status = 'completed' WHERE id = $1",
|
||||
row["id"],
|
||||
)
|
||||
except AppError:
|
||||
raise
|
||||
except Exception:
|
||||
raise AppError(503, "service unavailable") from None
|
||||
return _public(
|
||||
{
|
||||
"id": row["id"],
|
||||
"chat_id": row["chat_id"],
|
||||
"owner_id": row["owner_id"],
|
||||
"filename": row["filename"],
|
||||
"content_type": row["content_type"],
|
||||
"size_bytes": len(data),
|
||||
"sha256": digest,
|
||||
"status": "ready",
|
||||
"created_at": created,
|
||||
"updated_at": created,
|
||||
}
|
||||
)
|
||||
@@ -17,6 +17,8 @@ from app.docs import openapi_json, swagger_ui
|
||||
from app.errors import AppError, error_response, from_app_error
|
||||
from app.middleware import RequestContextMiddleware
|
||||
from app.routes import (
|
||||
begin_direct_upload,
|
||||
complete_direct_upload,
|
||||
create_chat,
|
||||
create_message,
|
||||
create_session,
|
||||
@@ -93,6 +95,8 @@ def build_app() -> Starlette:
|
||||
Route("/api/v1/chats/{chat_id}/messages", create_message, methods=["POST"]),
|
||||
Route("/api/v1/chats/{chat_id}/messages", list_messages, methods=["GET"]),
|
||||
Route("/api/v1/chats/{chat_id}/attachments", upload_attachment, methods=["POST"]),
|
||||
Route("/api/v1/chats/{chat_id}/uploads", begin_direct_upload, methods=["POST"]),
|
||||
Route("/api/v1/uploads/{upload_id}/complete", complete_direct_upload, methods=["POST"]),
|
||||
Route("/api/v1/attachments/{attachment_id}", get_attachment, methods=["GET"]),
|
||||
Route("/api/v1/attachments/{attachment_id}/content", download_attachment, methods=["GET"]),
|
||||
Route("/api/v1/_lab/metrics", lab_metrics, methods=["GET"]),
|
||||
|
||||
+53
-1
@@ -1,10 +1,11 @@
|
||||
import base64
|
||||
import binascii
|
||||
import re
|
||||
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import JSONResponse, Response
|
||||
|
||||
from app import attachments, cache, config, db, images, rediscli, s3, sessions, store
|
||||
from app import attachments, cache, config, db, direct, images, rediscli, s3, sessions, store
|
||||
from app.cursor_codec import decode_cursor, encode_cursor
|
||||
from app.errors import BASIC_CHALLENGE, BEARER_CHALLENGE, AppError, error_response
|
||||
from app.passwords import dummy_hash, hash_password, verify_password
|
||||
@@ -311,6 +312,57 @@ async def upload_attachment(request: Request) -> Response:
|
||||
return _json(created, rid, 201)
|
||||
|
||||
|
||||
async def begin_direct_upload(request: Request) -> Response:
|
||||
rid = request.state.request_id
|
||||
user = await _bearer_user(request)
|
||||
chat_id = request.path_params["chat_id"]
|
||||
if validate_uuid(chat_id):
|
||||
return error_response(400, "invalid chat id", rid)
|
||||
raw = await request.body()
|
||||
body, err = parse_json_body(raw)
|
||||
if err:
|
||||
return error_response(400, err, rid)
|
||||
assert body is not None
|
||||
if require_exact_keys(body, {"filename", "content_type", "size_bytes"}, {"filename", "content_type", "size_bytes"}):
|
||||
return error_response(400, "invalid upload payload", rid)
|
||||
if validate_filename(body["filename"]):
|
||||
return error_response(400, "invalid filename", rid)
|
||||
content_type = body["content_type"]
|
||||
if content_type not in ("image/png", "image/jpeg"):
|
||||
return error_response(415, "unsupported media type", rid)
|
||||
size = body["size_bytes"]
|
||||
if isinstance(size, bool) or not isinstance(size, int):
|
||||
return error_response(400, "invalid size", rid)
|
||||
if size > config.MAX_UPLOAD_BYTES:
|
||||
return error_response(413, "file exceeds upload limit", rid)
|
||||
if size < 1:
|
||||
return error_response(400, "invalid size", rid)
|
||||
if await store.load_chat_for_member(chat_id, user["id"]) is None:
|
||||
return error_response(404, "chat not found", rid)
|
||||
started = await direct.begin_upload(chat_id, user["id"], body["filename"], content_type, size)
|
||||
return _json(started, rid, 201)
|
||||
|
||||
|
||||
async def complete_direct_upload(request: Request) -> Response:
|
||||
rid = request.state.request_id
|
||||
user = await _bearer_user(request)
|
||||
upload_id = request.path_params["upload_id"]
|
||||
if validate_uuid(upload_id):
|
||||
return error_response(400, "invalid upload id", rid)
|
||||
raw = await request.body()
|
||||
body, err = parse_json_body(raw)
|
||||
if err:
|
||||
return error_response(400, err, rid)
|
||||
assert body is not None
|
||||
if require_exact_keys(body, {"sha256"}, {"sha256"}):
|
||||
return error_response(400, "invalid upload payload", rid)
|
||||
digest = body["sha256"]
|
||||
if not isinstance(digest, str) or re.fullmatch(r"[a-f0-9]{64}", digest) is None:
|
||||
return error_response(400, "invalid checksum", rid)
|
||||
created = await direct.complete_upload(upload_id, user["id"], digest)
|
||||
return _json(created, rid, 201)
|
||||
|
||||
|
||||
async def get_attachment(request: Request) -> Response:
|
||||
rid = request.state.request_id
|
||||
user = await _bearer_user(request)
|
||||
|
||||
@@ -14,6 +14,7 @@ from app.errors import AppError
|
||||
|
||||
logger = logging.getLogger("messenger.s3")
|
||||
_client = None
|
||||
_signer = None
|
||||
|
||||
|
||||
def _client_or_create():
|
||||
@@ -38,6 +39,26 @@ def _client_or_create():
|
||||
return _client
|
||||
|
||||
|
||||
def _signer_or_create():
|
||||
"""Signs URLs for the host the browser/client can reach. It does not send traffic."""
|
||||
global _signer
|
||||
if _signer is None:
|
||||
if not config.S3_ACCESS_KEY_ID or not config.S3_SECRET_ACCESS_KEY:
|
||||
raise AppError(503, "service unavailable")
|
||||
_signer = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=config.S3_PUBLIC_ENDPOINT,
|
||||
aws_access_key_id=config.S3_ACCESS_KEY_ID,
|
||||
aws_secret_access_key=config.S3_SECRET_ACCESS_KEY,
|
||||
region_name=config.S3_REGION,
|
||||
config=Config(
|
||||
signature_version="s3v4",
|
||||
s3={"addressing_style": "path"},
|
||||
),
|
||||
)
|
||||
return _signer
|
||||
|
||||
|
||||
def _unavailable(exc: Exception) -> AppError:
|
||||
code = ""
|
||||
if isinstance(exc, ClientError):
|
||||
@@ -91,6 +112,39 @@ async def get_bytes(key: str) -> bytes:
|
||||
raise _unavailable(exc) from None
|
||||
|
||||
|
||||
def presign_put(key: str, content_type: str, size: int) -> str:
|
||||
return _signer_or_create().generate_presigned_url(
|
||||
"put_object",
|
||||
Params={
|
||||
"Bucket": config.S3_BUCKET,
|
||||
"Key": key,
|
||||
"ContentType": content_type,
|
||||
"ContentLength": size,
|
||||
},
|
||||
ExpiresIn=config.PRESIGN_TTL_SECONDS,
|
||||
HttpMethod="PUT",
|
||||
)
|
||||
|
||||
|
||||
async def head_size(key: str) -> int | None:
|
||||
def _head() -> int | None:
|
||||
try:
|
||||
response = _client_or_create().head_object(Bucket=config.S3_BUCKET, Key=key)
|
||||
except ClientError as exc:
|
||||
code = exc.response.get("Error", {}).get("Code", "")
|
||||
if code in {"404", "NoSuchKey", "NotFound"}:
|
||||
return None
|
||||
raise
|
||||
return int(response["ContentLength"])
|
||||
|
||||
try:
|
||||
return await asyncio.to_thread(_head)
|
||||
except AppError:
|
||||
raise
|
||||
except (BotoCoreError, ClientError) as exc:
|
||||
raise _unavailable(exc) from None
|
||||
|
||||
|
||||
async def delete_key(key: str) -> None:
|
||||
def _delete() -> None:
|
||||
try:
|
||||
|
||||
@@ -53,6 +53,8 @@ services:
|
||||
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||
ORPHAN_GRACE_SECONDS: ${ORPHAN_GRACE_SECONDS:-120}
|
||||
S3_SWEEP_INTERVAL_SECONDS: ${S3_SWEEP_INTERVAL_SECONDS:-30}
|
||||
S3_PUBLIC_ENDPOINT: ${S3_PUBLIC_ENDPOINT:-http://127.0.0.1:9000}
|
||||
PRESIGN_TTL_SECONDS: ${PRESIGN_TTL_SECONDS:-60}
|
||||
DB_POOL_MIN: "1"
|
||||
DB_POOL_MAX: ${DB_POOL_MAX:-4}
|
||||
DB_COMMAND_TIMEOUT: "5"
|
||||
@@ -123,6 +125,8 @@ services:
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY}
|
||||
ports:
|
||||
- "127.0.0.1:9000:9000"
|
||||
volumes:
|
||||
- s3_data:/data
|
||||
- ./infra/s3/seaweed-entrypoint.sh:/seaweed-entrypoint.sh:ro
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"openapi": "3.1.0",
|
||||
"info": {
|
||||
"title": "Messenger lab4 optional direct upload",
|
||||
"version": "1.4.0-track1"
|
||||
},
|
||||
"paths": {
|
||||
"/api/v1/chats/{chat_id}/uploads": {
|
||||
"post": {
|
||||
"operationId": "beginDirectUpload",
|
||||
"summary": "Issue a 60s PUT signature for one new object key. Does not replace POST /attachments."
|
||||
}
|
||||
},
|
||||
"/api/v1/uploads/{upload_id}/complete": {
|
||||
"post": {
|
||||
"operationId": "completeDirectUpload",
|
||||
"summary": "Check owner, size and sha256 of the object that arrived, then store ready metadata."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"hypothesis": "A presigned PUT does not enter the API, so API RX stays near zero at both sizes. complete reads the object once to check the checksum, and the proxy endpoint still receives the whole body.",
|
||||
"presign_ttl_seconds": 60,
|
||||
"sizes": [
|
||||
{
|
||||
"label": "small",
|
||||
"bytes": 90243,
|
||||
"proxy_rx_bytes": 96204,
|
||||
"proxy_peak_api_mib": 61.0,
|
||||
"direct_put_rx_bytes": 0,
|
||||
"direct_put_peak_api_mib": 61.4,
|
||||
"complete_rx_bytes": 94131,
|
||||
"complete_peak_api_mib": 61.4
|
||||
},
|
||||
{
|
||||
"label": "large",
|
||||
"bytes": 1891338,
|
||||
"proxy_rx_bytes": 1913606,
|
||||
"proxy_peak_api_mib": 62.4,
|
||||
"direct_put_rx_bytes": 960,
|
||||
"direct_put_peak_api_mib": 62.5,
|
||||
"complete_rx_bytes": 1900090,
|
||||
"complete_peak_api_mib": 62.1
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -51,3 +51,18 @@ Authorization: Bearer {{sessionToken}}
|
||||
Content-Type: image/png
|
||||
|
||||
< ../tests/fixtures/sample.png
|
||||
|
||||
### Optional direct upload. PUT the bytes to the returned url, then complete with the real sha256.
|
||||
POST {{baseUrl}}/api/v1/chats/{{chatId}}/uploads
|
||||
Authorization: Bearer {{sessionToken}}
|
||||
Content-Type: application/json
|
||||
|
||||
{"filename":"sample.png","content_type":"image/png","size_bytes":1}
|
||||
|
||||
### Complete only after the PUT. Replace the id and checksum locally.
|
||||
POST {{baseUrl}}/api/v1/uploads/REPLACE_WITH_UPLOAD_ID/complete
|
||||
Authorization: Bearer {{sessionToken}}
|
||||
Content-Type: application/json
|
||||
|
||||
{"sha256":"replace-with-lowercase-sha256-of-the-bytes"}
|
||||
|
||||
|
||||
+1
-1
@@ -6,4 +6,4 @@
|
||||
|
||||
Приложение строится из исходников. Не используйте заранее существующий образ преподавательского сервера вместо своей реализации. Restart hook перезапускает все соответствующие stores с volumes; failover hook останавливает только заданную API-реплику.
|
||||
|
||||
Выбран SeaweedFS `chrislusf/seaweedfs:3.85` (S3 gateway на порту 9000 внутри сети Compose). Образ MinIO на Docker Hub в этом окружении недоступен. Порты наружу не опубликованы, UI filer выключен (`-filer.exposeDirectoryData=false`). `infra/s3/seaweed-entrypoint.sh` поднимает одну ноду и выдаёт API-пользователю права только на bucket `messenger-attachments`. Анонимная identity без actions. `s3-init` идемпотентно создаёт bucket теми же ограниченными ключами.
|
||||
Выбран SeaweedFS `chrislusf/seaweedfs:3.85` (S3 gateway на порту 9000 внутри сети Compose). Образ MinIO на Docker Hub в этом окружении недоступен. Порты наружу не опубликованы, UI filer выключен (`-filer.exposeDirectoryData=false`). `infra/s3/seaweed-entrypoint.sh` поднимает одну ноду и выдаёт API-пользователю права только на bucket `messenger-attachments`. Анонимная identity без actions. `s3-init` идемпотентно создаёт bucket теми же ограниченными ключами. Для прямой загрузки S3 API слушает только `127.0.0.1:9000`; UI и остальные порты наружу не открыты. Подпись живёт `PRESIGN_TTL_SECONDS` (60).
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
-- Optional presigned upload. The signed URL dies quickly. The row stays pending
|
||||
-- until complete checks the object that actually landed in the bucket.
|
||||
-- Incomplete rows are aborted by the same grace sweeper as other uncommitted keys.
|
||||
|
||||
CREATE TABLE direct_uploads (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID NOT NULL REFERENCES chats (id) ON DELETE CASCADE,
|
||||
owner_id UUID NOT NULL REFERENCES users (id),
|
||||
filename TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
object_key TEXT NOT NULL UNIQUE,
|
||||
status TEXT NOT NULL,
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT direct_uploads_filename_len CHECK (char_length(filename) BETWEEN 1 AND 255),
|
||||
CONSTRAINT direct_uploads_size CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
CONSTRAINT direct_uploads_type CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||
CONSTRAINT direct_uploads_status CHECK (status IN ('pending', 'completed', 'aborted'))
|
||||
);
|
||||
|
||||
CREATE INDEX direct_uploads_pending_idx ON direct_uploads (expires_at) WHERE status = 'pending';
|
||||
@@ -211,13 +211,14 @@ def main():
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if published.returncode == 0 and published.stdout.strip():
|
||||
raise SystemExit("s3 port is published: " + published.stdout.strip())
|
||||
binding = published.stdout.strip()
|
||||
if published.returncode == 0 and binding and not binding.startswith("127.0.0.1:"):
|
||||
raise SystemExit("s3 port is published outside loopback: " + binding)
|
||||
summary = {
|
||||
"anonymous_bucket_status": anonymous_status(),
|
||||
"second_bucket": other_bucket_denied(),
|
||||
"sweep": plant_and_sweep(),
|
||||
"s3_port_published": False,
|
||||
"s3_port_published": binding or False,
|
||||
"memory": memory_sample(),
|
||||
}
|
||||
OUT.parent.mkdir(exist_ok=True)
|
||||
|
||||
@@ -0,0 +1,380 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Track 1: presigned upload, negative cases, API traffic and RAM at two sizes.
|
||||
|
||||
The signed URL is never written to the evidence file.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import uuid
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
OUT = ROOT / "evidence" / "track1.json"
|
||||
BASE = "http://127.0.0.1:8080"
|
||||
|
||||
|
||||
def expect(status, got, label):
|
||||
if got == 500:
|
||||
raise SystemExit("FAIL %s returned 500" % label)
|
||||
if got != status:
|
||||
raise SystemExit("FAIL %s expected %s got %s" % (label, status, got))
|
||||
print("PASS", label, got)
|
||||
|
||||
|
||||
class Api:
|
||||
def __init__(self) -> None:
|
||||
self.opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
|
||||
def call(self, method, path, body=None, token=None, raw=None, content_type=None):
|
||||
headers = {}
|
||||
data = raw
|
||||
if body is not None:
|
||||
data = json.dumps(body).encode("utf-8")
|
||||
headers["Content-Type"] = "application/json"
|
||||
if content_type:
|
||||
headers["Content-Type"] = content_type
|
||||
if token:
|
||||
headers["Authorization"] = "Bearer " + token
|
||||
request = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)
|
||||
try:
|
||||
response = self.opener.open(request, timeout=60)
|
||||
except urllib.error.HTTPError as error:
|
||||
response = error
|
||||
with response:
|
||||
payload = response.read()
|
||||
parsed = None
|
||||
if payload and response.headers.get_content_type() == "application/json":
|
||||
parsed = json.loads(payload)
|
||||
return response.status, parsed, payload
|
||||
|
||||
|
||||
def png(width: int, height: int) -> bytes:
|
||||
signature = b"\x89PNG\r\n\x1a\n"
|
||||
|
||||
def chunk(kind: bytes, data: bytes) -> bytes:
|
||||
return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)
|
||||
|
||||
ihdr = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)
|
||||
raw = b"".join(b"\x00" + os.urandom(width * 3) for _ in range(height))
|
||||
return signature + chunk(b"IHDR", ihdr) + chunk(b"IDAT", zlib.compress(raw, 1)) + chunk(b"IEND", b"")
|
||||
|
||||
|
||||
def put_signed(url: str, data: bytes, content_type: str) -> int:
|
||||
request = urllib.request.Request(url, data=data, method="PUT", headers={"Content-Type": content_type})
|
||||
try:
|
||||
response = urllib.request.urlopen(request, timeout=60)
|
||||
except urllib.error.HTTPError as error:
|
||||
response = error
|
||||
with response:
|
||||
response.read()
|
||||
return response.status
|
||||
|
||||
|
||||
def api_rx() -> int:
|
||||
ids = subprocess.check_output(
|
||||
["docker", "compose", "--env-file", ".env", "ps", "-q", "api"],
|
||||
cwd=ROOT,
|
||||
text=True,
|
||||
).split()
|
||||
total = 0
|
||||
for container_id in ids:
|
||||
raw = subprocess.check_output(["docker", "exec", container_id, "cat", "/proc/net/dev"], text=True)
|
||||
for line in raw.splitlines():
|
||||
if "eth0" not in line:
|
||||
continue
|
||||
total += int(line.split()[1])
|
||||
return total
|
||||
|
||||
|
||||
def peak_during(action):
|
||||
peaks = []
|
||||
stop = threading.Event()
|
||||
|
||||
def watch():
|
||||
while not stop.is_set():
|
||||
stats = subprocess.run(
|
||||
["docker", "stats", "--no-stream", "--format", "{{.Name}} {{.MemUsage}}"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
for line in stats.stdout.splitlines():
|
||||
if "mtusi2026lab4-api-" not in line:
|
||||
continue
|
||||
value = line.split()[1]
|
||||
number = float(value[:-3])
|
||||
if not value.lower().endswith("mib"):
|
||||
number = number / 1024
|
||||
peaks.append(number)
|
||||
time.sleep(0.25)
|
||||
|
||||
thread = threading.Thread(target=watch)
|
||||
thread.start()
|
||||
before = api_rx()
|
||||
result = action()
|
||||
after = api_rx()
|
||||
time.sleep(0.3)
|
||||
stop.set()
|
||||
thread.join()
|
||||
return {"rx_bytes": after - before, "peak_api_mib": round(max(peaks) if peaks else 0, 1), "result": result}
|
||||
|
||||
|
||||
def compose_psql(sql: str) -> None:
|
||||
subprocess.run(
|
||||
["docker", "compose", "--env-file", ".env", "exec", "-T", "postgres", "psql", "-U", "messenger", "-d", "messenger", "-c", sql],
|
||||
cwd=ROOT,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
|
||||
def negatives(api, alice_token, bob_token, eve_token, chat_id):
|
||||
small = png(16, 16)
|
||||
status, _, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/uploads" % chat_id,
|
||||
{"filename": "a/b.png", "content_type": "image/png", "size_bytes": len(small)},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(400, status, "direct filename")
|
||||
status, _, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/uploads" % chat_id,
|
||||
{"filename": "big.png", "content_type": "image/png", "size_bytes": 10485761},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(413, status, "direct declared size")
|
||||
status, _, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/uploads" % chat_id,
|
||||
{"filename": "x.png", "content_type": "image/png", "size_bytes": len(small)},
|
||||
token=eve_token,
|
||||
)
|
||||
expect(404, status, "outsider presign")
|
||||
|
||||
status, started, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/uploads" % chat_id,
|
||||
{"filename": "wait.png", "content_type": "image/png", "size_bytes": len(small)},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(201, status, "presign for incomplete")
|
||||
status, _, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/uploads/%s/complete" % started["upload_id"],
|
||||
{"sha256": hashlib.sha256(small).hexdigest()},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(409, status, "complete before PUT")
|
||||
status, _, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/uploads/%s/complete" % started["upload_id"],
|
||||
{"sha256": hashlib.sha256(small).hexdigest()},
|
||||
token=bob_token,
|
||||
)
|
||||
expect(404, status, "complete by another member")
|
||||
|
||||
tampered = put_signed(started["url"] + "x", small, "image/png")
|
||||
if tampered == 200:
|
||||
raise SystemExit("FAIL tampered signature was accepted")
|
||||
print("PASS tampered signature", tampered)
|
||||
|
||||
status, started, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/uploads" % chat_id,
|
||||
{"filename": "badsum.png", "content_type": "image/png", "size_bytes": len(small)},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(201, status, "presign for bad checksum")
|
||||
if put_signed(started["url"], small, "image/png") != 200:
|
||||
raise SystemExit("FAIL signed PUT")
|
||||
status, _, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/uploads/%s/complete" % started["upload_id"],
|
||||
{"sha256": "0" * 64},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(400, status, "wrong checksum")
|
||||
status, _, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/uploads/%s/complete" % started["upload_id"],
|
||||
{"sha256": hashlib.sha256(small).hexdigest()},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(409, status, "complete after rejected checksum")
|
||||
|
||||
status, started, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/uploads" % chat_id,
|
||||
{"filename": "late.png", "content_type": "image/png", "size_bytes": len(small)},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(201, status, "presign for expiry")
|
||||
compose_psql(
|
||||
"UPDATE direct_uploads SET expires_at = now() - interval '5 minutes' WHERE id = '%s'"
|
||||
% started["upload_id"]
|
||||
)
|
||||
status, _, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/uploads/%s/complete" % started["upload_id"],
|
||||
{"sha256": hashlib.sha256(small).hexdigest()},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(409, status, "expired upload")
|
||||
|
||||
digest = hashlib.sha256(small).hexdigest()
|
||||
status, started, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/uploads" % chat_id,
|
||||
{"filename": "ok.png", "content_type": "image/png", "size_bytes": len(small)},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(201, status, "presign happy path")
|
||||
if "originals/" in json.dumps({k: v for k, v in started.items() if k != "url"}):
|
||||
raise SystemExit("FAIL object key leaked outside the signed URL")
|
||||
if put_signed(started["url"], small, "image/png") != 200:
|
||||
raise SystemExit("FAIL happy PUT")
|
||||
status, attachment, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/uploads/%s/complete" % started["upload_id"],
|
||||
{"sha256": digest},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(201, status, "complete happy path")
|
||||
if attachment["sha256"] != digest or attachment["status"] != "ready":
|
||||
raise SystemExit("FAIL complete metadata")
|
||||
status, _, downloaded = api.call("GET", "/api/v1/attachments/%s/content" % attachment["id"], token=bob_token)
|
||||
expect(200, status, "member downloads direct upload")
|
||||
if downloaded != small:
|
||||
raise SystemExit("FAIL direct bytes")
|
||||
status, _, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/uploads/%s/complete" % started["upload_id"],
|
||||
{"sha256": digest},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(409, status, "second complete")
|
||||
status, proxy, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/attachments?filename=proxy.png" % chat_id,
|
||||
token=alice_token,
|
||||
raw=small,
|
||||
content_type="image/png",
|
||||
)
|
||||
expect(201, status, "proxy endpoint still works")
|
||||
return proxy["id"]
|
||||
|
||||
|
||||
def compare(api, token, chat_id):
|
||||
rows = []
|
||||
for label, image in (("small", png(200, 150)), ("large", png(900, 700))):
|
||||
digest = hashlib.sha256(image).hexdigest()
|
||||
proxy = peak_during(
|
||||
lambda image=image: api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/attachments?filename=%s.png" % (chat_id, label),
|
||||
token=token,
|
||||
raw=image,
|
||||
content_type="image/png",
|
||||
)
|
||||
)
|
||||
if proxy["result"][0] != 201:
|
||||
raise SystemExit("FAIL proxy %s %s" % (label, proxy["result"][0]))
|
||||
status, started, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats/%s/uploads" % chat_id,
|
||||
{"filename": label + "-direct.png", "content_type": "image/png", "size_bytes": len(image)},
|
||||
token=token,
|
||||
)
|
||||
expect(201, status, "presign " + label)
|
||||
direct_put = peak_during(lambda image=image, url=started["url"]: put_signed(url, image, "image/png"))
|
||||
if direct_put["result"] != 200:
|
||||
raise SystemExit("FAIL direct PUT %s %s" % (label, direct_put["result"]))
|
||||
direct_complete = peak_during(
|
||||
lambda digest=digest, upload_id=started["upload_id"]: api.call(
|
||||
"POST",
|
||||
"/api/v1/uploads/%s/complete" % upload_id,
|
||||
{"sha256": digest},
|
||||
token=token,
|
||||
)
|
||||
)
|
||||
if direct_complete["result"][0] != 201:
|
||||
raise SystemExit("FAIL complete %s %s" % (label, direct_complete["result"][0]))
|
||||
rows.append(
|
||||
{
|
||||
"label": label,
|
||||
"bytes": len(image),
|
||||
"proxy_rx_bytes": proxy["rx_bytes"],
|
||||
"proxy_peak_api_mib": proxy["peak_api_mib"],
|
||||
"direct_put_rx_bytes": direct_put["rx_bytes"],
|
||||
"direct_put_peak_api_mib": direct_put["peak_api_mib"],
|
||||
"complete_rx_bytes": direct_complete["rx_bytes"],
|
||||
"complete_peak_api_mib": direct_complete["peak_api_mib"],
|
||||
}
|
||||
)
|
||||
print("MEASURED", rows[-1])
|
||||
return rows
|
||||
|
||||
|
||||
def main():
|
||||
api = Api()
|
||||
|
||||
def register(label):
|
||||
username = "t1_" + label + "_" + uuid.uuid4().hex[:10]
|
||||
password = "T3st-" + uuid.uuid4().hex
|
||||
status, body, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/users",
|
||||
{"username": username, "display_name": "Track " + label, "password": password},
|
||||
)
|
||||
expect(201, status, "register " + label)
|
||||
return username, password, body
|
||||
|
||||
def login(username, password):
|
||||
basic = base64.b64encode((username + ":" + password).encode()).decode()
|
||||
request = urllib.request.Request(
|
||||
BASE + "/api/v1/auth/sessions",
|
||||
method="POST",
|
||||
headers={"Authorization": "Basic " + basic},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
return json.loads(response.read())["token"]
|
||||
|
||||
alice_name, alice_password, alice = register("alice")
|
||||
bob_name, bob_password, bob = register("bob")
|
||||
eve_name, eve_password, eve = register("eve")
|
||||
alice_token = login(alice_name, alice_password)
|
||||
bob_token = login(bob_name, bob_password)
|
||||
eve_token = login(eve_name, eve_password)
|
||||
status, chat, _ = api.call(
|
||||
"POST",
|
||||
"/api/v1/chats",
|
||||
{"title": "track1", "member_ids": [alice["id"], bob["id"]]},
|
||||
token=alice_token,
|
||||
)
|
||||
expect(201, status, "chat")
|
||||
negatives(api, alice_token, bob_token, eve_token, chat["id"])
|
||||
rows = compare(api, alice_token, chat["id"])
|
||||
summary = {
|
||||
"hypothesis": "A presigned PUT does not enter the API, so API RX stays near zero at both sizes. complete reads the object once to check the checksum, and the proxy endpoint still receives the whole body.",
|
||||
"presign_ttl_seconds": 60,
|
||||
"sizes": rows,
|
||||
}
|
||||
OUT.parent.mkdir(exist_ok=True)
|
||||
OUT.write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
|
||||
print(json.dumps(summary, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user