Добавить прямую загрузку по короткой подписи.

Proxy endpoint сохранён. Complete сверяет владельца, размер и checksum, а сравнение трафика API на двух размерах лежит в evidence.
This commit is contained in:
2026-11-09 00:36:52 +03:00
parent 8faf08d8eb
commit 4654de5837
18 changed files with 812 additions and 11 deletions
+3
View File
@@ -32,3 +32,6 @@ S3_SECRET_ACCESS_KEY=lab4-dev-only-change-me
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
ORPHAN_GRACE_SECONDS=120
# Host the client uses for a presigned PUT. Loopback only; the API itself uses S3_ENDPOINT.
S3_PUBLIC_ENDPOINT=http://127.0.0.1:9000
PRESIGN_TTL_SECONDS=60
+4 -4
View File
@@ -1,13 +1,13 @@
## Лабораторная 4
Автор / группа: Булацев, БПИ2301
Целевая оценка (3/4/5): 4
Трек 5, если заявлен: нет
Целевая оценка (3/4/5): 5
Трек 5, если заявлен: 1, прямой upload
Что реализовано и что перенесено: стенд lab3 (Postgres, Redis, HAProxy, две реплики API, идемпотентность) и вложения lab4 в SeaweedFS. Метаданные в Postgres, байты в приватном bucket, ключ назначает сервер.
Что реализовано и что перенесено: стенд lab3, вложения в SeaweedFS и отдельный presigned upload. Proxy `POST /attachments` сохранён. Метаданные в Postgres, байты в приватном bucket.
Команды запуска из чистого clone: `cp .env.example .env`, затем `docker compose --env-file .env up -d --build --scale api=2`. Подробности в REPORT.md.
Результаты проверок и ссылка на REPORT.md: smoke, persistence и failover прошли; негативные загрузки и orphan/память — в `evidence/`. См. REPORT.md.
Ограничения / известные ошибки: один Docker-хост; трек 5 не делался; образ MinIO на Docker Hub недоступен, используется SeaweedFS 3.85.
Ограничения / известные ошибки: один Docker-хост; complete прямой загрузки читает объект обратно в API; образ MinIO на Docker Hub недоступен, используется SeaweedFS 3.85. S3 открыт только на 127.0.0.1:9000.
Источники и использование ИИ: контракт репозитория, документация SeaweedFS, boto3, Pillow. Код и прогон проверок — с помощью Cursor.
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
+3
View File
@@ -32,3 +32,6 @@ test-failover:
grade4:
$(PYTHON) scripts/grade4_checks.py
$(PYTHON) scripts/grade4_infra.py
track1:
$(PYTHON) scripts/track1_experiment.py
+9 -2
View File
@@ -1,7 +1,7 @@
# Отчёт — лабораторная 4
- Автор, группа: Булацев, БПИ2301
- Целевая оценка; выбранный трек 5, если нужен: 4. Трек 5 не заявляется.
- Целевая оценка; выбранный трек 5, если нужен: 5, трек 1 «Прямой upload».
- Commit/PR: локальные коммиты после переноса lab3; PR не публиковался в этой сессии
- Использованные источники, библиотеки и помощь ИИ: контракт репозитория, документация SeaweedFS S3, boto3, Pillow. Реализация и прогон проверок сделаны с помощью Cursor.
@@ -26,6 +26,7 @@ python -X utf8 tests/resilience.py --mode persistence --action-script scripts/re
python -X utf8 tests/resilience.py --mode failover --action-script scripts/stop-one.sh --base-url http://127.0.0.1:8080
python -X utf8 scripts/grade4_checks.py
python -X utf8 scripts/grade4_infra.py
python -X utf8 scripts/track1_experiment.py
```
Остановка без удаления данных: `docker compose --env-file .env stop`. Полное удаление учебных volumes: `docker compose --env-file .env down -v`.
@@ -44,6 +45,8 @@ python -X utf8 scripts/grade4_infra.py
`/health/ready` зависит от Postgres, Redis и `HeadBucket`. Сессии по-прежнему в Redis (`SESSION_STORE=redis`), не в Postgres.
Прямая загрузка не заменяет `POST /attachments`. `POST /chats/{id}/uploads` выдаёт PUT-подпись на 60 секунд и ключ `originals/<uuid>`. В JSON ключа нет, он есть только внутри подписанного URL. `POST /uploads/{id}/complete` принимает загрузку, только если её начал тот же пользователь, подпись не истекла, размер совпал с заявленным, sha256 совпал с байтами в bucket, а содержимое — целый PNG или JPEG. Чужой complete — 404. Повторный complete, complete без PUT и просроченная подпись — 409. Неверная сумма удаляет объект и больше не даёт дозавершить загрузку. Незаконченный ключ снимает тот же sweeper, что и orphan после grace. S3 с хоста доступен только на `127.0.0.1:9000`, чтобы клиент мог сделать PUT мимо API; анонимный запрос по-прежнему 403.
| Роль | Сервис | Образ | Проверка | Volume | Порт |
| --- | --- | --- | --- | --- | --- |
| api | api ×2 | сборка `mtusi-messenger:lab4` | `/health/ready` | нет | только внутри сети |
@@ -71,6 +74,8 @@ python -X utf8 scripts/grade4_infra.py
| Ограниченные ключи | тот же скрипт | `create_bucket` другого имени — `AccessDenied` | `evidence/grade4-infra.json` |
| Компенсация orphan | тот же скрипт, grace 120 с | старый незакоммиченный объект удалён, свежий оставлен | `evidence/grade4-infra.json` |
| Память 10 загрузок | тот же скрипт, docker stats | 10×230778 байт, все 201, пик API 61.7 МиБ при лимите 256 | `evidence/grade4-infra.json` |
| Трек 1, негативные сценарии | `scripts/track1_experiment.py` | подделанная подпись 403; complete без PUT, чужой пользователь, повтор и просрочка — 404/409; неверная сумма 400 и объект больше не завершается; proxy endpoint по-прежнему 201 | `evidence/track1.json` |
| Трек 1, трафик API | тот же скрипт, сумма RX `eth0` обеих реплик | файл 90243 байт: proxy RX 96204, прямой PUT RX 0, complete RX 94131. Файл 1891338 байт: proxy RX 1913606, прямой PUT RX 960, complete RX 1900090 | `evidence/track1.json` |
`Content-Length: 10485761` при коротком теле отвечает 413 до дочитывания заявленного размера.
@@ -78,6 +83,8 @@ python -X utf8 scripts/grade4_infra.py
Гипотеза уровня 4: десять одновременных загрузок около 230 КиБ не поднимают RSS реплики к лимиту 256 МиБ, потому что тело ограничено 10 МиБ и не декодируется целиком до проверки пикселей. Один прогон, без прогрева отдельной фазой. Это не профиль RPS сообщений.
Гипотеза трека 1: прямой PUT не заходит в API, поэтому RX API почти не зависит от размера файла, а proxy RX растёт вместе с телом. Complete один раз скачивает объект, чтобы сверить sha256, и этот трафик остаётся. Пик RSS по `docker stats` на файлах 90 КиБ и 1.9 МиБ остался около 61–62.5 МиБ: базовая память процесса больше короткого буфера, и по RSS выигрыш на этих размерах не виден. Выигрыш, который измерен, — это RX на фазе PUT.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| `scripts/restart.sh` (postgres, redis, s3, api), volumes сохранены | те же сессия, сообщение и байты | PASS публичного persistence | готовность вернулась в пределах ожидания теста (90 с) |
@@ -91,6 +98,6 @@ python -X utf8 scripts/grade4_infra.py
Скачивание и загрузка держат в памяти процесса не больше лимита 10 МиБ плюс проверку Pillow. Это граница, а не потоковая отдача клиенту минуя буфер. Эксперимент памяти сделан на файлах ~226 КиБ, не на десяти одновременных файлах по 10 МиБ.
Окно между PUT и COMMIT закрывается немедленным удалением при ошибке и обходом раз в 30 с. Объект, умерший внутри grace 120 с, временно занимает место. Прямой presigned upload и отдельный reconciler всего bucket не делались: это трек 5.
Окно между PUT и COMMIT закрывается немедленным удалением при ошибке и обходом раз в 30 с. Объект, умерший внутри grace 120 с, временно занимает место. Прямой upload добавлен отдельными маршрутами и не отменяет proxy. Complete всё равно читает объект в API, поэтому экономия трафика относится к фазе PUT, а не ко всей операции. Отдельный reconciler всего bucket не делался.
Случайный ключ и закрытый bucket не заменяют проверку участника чата: скачивание идёт через API и для постороннего даёт 404, даже если он угадает UUID.
+8
View File
@@ -178,6 +178,14 @@ async def sweep_orphans() -> int:
continue
await s3.delete_key(key)
await conn.execute("DELETE FROM blob_writes WHERE object_key = $1", key)
await conn.execute(
"""
UPDATE direct_uploads
SET status = 'aborted'
WHERE object_key = $1 AND status = 'pending'
""",
key,
)
deleted += 1
if deleted:
logger.info("removed %s uncommitted objects past grace", deleted)
+2
View File
@@ -37,6 +37,8 @@ LB_DRAIN_SECONDS = float(_env("LB_DRAIN_SECONDS", "2"))
# container uses its own hostname; stop-one.sh maps that value to one replica.
INSTANCE_ID = _env("INSTANCE_ID", socket.gethostname())
S3_ENDPOINT = _env("S3_ENDPOINT", "http://s3:9000")
S3_PUBLIC_ENDPOINT = _env("S3_PUBLIC_ENDPOINT", "http://127.0.0.1:9000")
PRESIGN_TTL_SECONDS = min(max(int(_env("PRESIGN_TTL_SECONDS", "60")), 15), 300)
S3_REGION = _env("S3_REGION", "us-east-1")
S3_BUCKET = _env("S3_BUCKET", "messenger-attachments")
S3_ACCESS_KEY_ID = _file_or_env("S3_ACCESS_KEY_ID")
+199
View File
@@ -0,0 +1,199 @@
"""Presigned PUT that does not replace the proxy upload endpoint."""
from __future__ import annotations
import asyncio
import hashlib
from datetime import datetime, timedelta, timezone
from uuid import UUID, uuid4
from app import config, db, images, s3
from app.attachments import _compensate, _public
from app.errors import AppError
from app.store import format_ts
def _aware(value: datetime) -> datetime:
if value.tzinfo is None:
return value.replace(tzinfo=timezone.utc)
return value
async def begin_upload(
chat_id: str,
owner_id: str,
filename: str,
content_type: str,
size_bytes: int,
) -> dict:
upload_id = uuid4()
key = f"originals/{upload_id}"
created = datetime.now(timezone.utc)
expires_at = created + timedelta(seconds=config.PRESIGN_TTL_SECONDS)
async with db.require().acquire() as conn:
async with conn.transaction():
await conn.execute(
"""
INSERT INTO blob_writes (object_key, committed, created_at)
VALUES ($1, FALSE, $2)
""",
key,
created,
)
await conn.execute(
"""
INSERT INTO direct_uploads (
id, chat_id, owner_id, filename, content_type, size_bytes,
object_key, status, expires_at, created_at
)
VALUES ($1, $2, $3, $4, $5, $6, $7, 'pending', $8, $9)
""",
upload_id,
UUID(chat_id),
UUID(owner_id),
filename,
content_type,
size_bytes,
key,
expires_at,
created,
)
try:
url = await asyncio.to_thread(s3.presign_put, key, content_type, size_bytes)
except Exception:
await _compensate(key)
async with db.require().acquire() as conn:
await conn.execute(
"UPDATE direct_uploads SET status = 'aborted' WHERE id = $1 AND status = 'pending'",
upload_id,
)
raise
return {
"upload_id": str(upload_id),
"method": "PUT",
"url": url,
"headers": {"Content-Type": content_type},
"expires_at": format_ts(expires_at),
"size_bytes": size_bytes,
}
async def _abort_pending(upload_id: UUID, key: str) -> None:
async with db.require().acquire() as conn:
changed = await conn.fetchval(
"""
UPDATE direct_uploads
SET status = 'aborted'
WHERE id = $1 AND status = 'pending'
RETURNING id
""",
upload_id,
)
if changed is not None:
await _compensate(key)
async def complete_upload(upload_id: str, owner_id: str, claimed_sha: str) -> dict:
async with db.require().acquire() as conn:
row = await conn.fetchrow(
"""
SELECT id, chat_id, owner_id, filename, content_type, size_bytes,
object_key, status, expires_at
FROM direct_uploads
WHERE id = $1 AND owner_id = $2
""",
UUID(upload_id),
UUID(owner_id),
)
if row is None:
raise AppError(404, "upload not found")
if row["status"] != "pending":
raise AppError(409, "upload is already finished")
if _aware(row["expires_at"]) <= datetime.now(timezone.utc):
await _abort_pending(row["id"], row["object_key"])
raise AppError(409, "upload signature expired")
actual_size = await s3.head_size(row["object_key"])
if actual_size is None:
raise AppError(409, "upload is incomplete")
if actual_size != row["size_bytes"] or actual_size > config.MAX_UPLOAD_BYTES:
await _abort_pending(row["id"], row["object_key"])
raise AppError(400, "uploaded size does not match")
data = await s3.get_bytes(row["object_key"])
digest = hashlib.sha256(data).hexdigest()
if len(data) != row["size_bytes"] or digest != claimed_sha:
await _abort_pending(row["id"], row["object_key"])
raise AppError(400, "checksum does not match")
try:
images.inspect_image(data, row["content_type"])
except images.ImageRejected as exc:
await _abort_pending(row["id"], row["object_key"])
raise AppError(exc.status, exc.message) from None
created = datetime.now(timezone.utc)
try:
async with db.require().acquire() as conn:
async with conn.transaction():
locked = await conn.fetchrow(
"""
SELECT status, object_key
FROM direct_uploads
WHERE id = $1 AND owner_id = $2
FOR UPDATE
""",
row["id"],
UUID(owner_id),
)
if locked is None or locked["status"] != "pending":
raise AppError(409, "upload is already finished")
intent = await conn.fetchrow(
"SELECT committed FROM blob_writes WHERE object_key = $1 FOR UPDATE",
row["object_key"],
)
if intent is None or intent["committed"]:
raise AppError(409, "upload is already finished")
await conn.execute(
"""
INSERT INTO attachments (
id, chat_id, owner_id, filename, content_type, size_bytes,
sha256, object_key, status, error_code, created_at, updated_at
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 'ready', NULL, $9, $9)
""",
row["id"],
row["chat_id"],
row["owner_id"],
row["filename"],
row["content_type"],
len(data),
digest,
row["object_key"],
created,
)
await conn.execute(
"UPDATE blob_writes SET committed = TRUE WHERE object_key = $1",
row["object_key"],
)
await conn.execute(
"UPDATE direct_uploads SET status = 'completed' WHERE id = $1",
row["id"],
)
except AppError:
raise
except Exception:
raise AppError(503, "service unavailable") from None
return _public(
{
"id": row["id"],
"chat_id": row["chat_id"],
"owner_id": row["owner_id"],
"filename": row["filename"],
"content_type": row["content_type"],
"size_bytes": len(data),
"sha256": digest,
"status": "ready",
"created_at": created,
"updated_at": created,
}
)
+4
View File
@@ -17,6 +17,8 @@ from app.docs import openapi_json, swagger_ui
from app.errors import AppError, error_response, from_app_error
from app.middleware import RequestContextMiddleware
from app.routes import (
begin_direct_upload,
complete_direct_upload,
create_chat,
create_message,
create_session,
@@ -93,6 +95,8 @@ def build_app() -> Starlette:
Route("/api/v1/chats/{chat_id}/messages", create_message, methods=["POST"]),
Route("/api/v1/chats/{chat_id}/messages", list_messages, methods=["GET"]),
Route("/api/v1/chats/{chat_id}/attachments", upload_attachment, methods=["POST"]),
Route("/api/v1/chats/{chat_id}/uploads", begin_direct_upload, methods=["POST"]),
Route("/api/v1/uploads/{upload_id}/complete", complete_direct_upload, methods=["POST"]),
Route("/api/v1/attachments/{attachment_id}", get_attachment, methods=["GET"]),
Route("/api/v1/attachments/{attachment_id}/content", download_attachment, methods=["GET"]),
Route("/api/v1/_lab/metrics", lab_metrics, methods=["GET"]),
+53 -1
View File
@@ -1,10 +1,11 @@
import base64
import binascii
import re
from starlette.requests import Request
from starlette.responses import JSONResponse, Response
from app import attachments, cache, config, db, images, rediscli, s3, sessions, store
from app import attachments, cache, config, db, direct, images, rediscli, s3, sessions, store
from app.cursor_codec import decode_cursor, encode_cursor
from app.errors import BASIC_CHALLENGE, BEARER_CHALLENGE, AppError, error_response
from app.passwords import dummy_hash, hash_password, verify_password
@@ -311,6 +312,57 @@ async def upload_attachment(request: Request) -> Response:
return _json(created, rid, 201)
async def begin_direct_upload(request: Request) -> Response:
rid = request.state.request_id
user = await _bearer_user(request)
chat_id = request.path_params["chat_id"]
if validate_uuid(chat_id):
return error_response(400, "invalid chat id", rid)
raw = await request.body()
body, err = parse_json_body(raw)
if err:
return error_response(400, err, rid)
assert body is not None
if require_exact_keys(body, {"filename", "content_type", "size_bytes"}, {"filename", "content_type", "size_bytes"}):
return error_response(400, "invalid upload payload", rid)
if validate_filename(body["filename"]):
return error_response(400, "invalid filename", rid)
content_type = body["content_type"]
if content_type not in ("image/png", "image/jpeg"):
return error_response(415, "unsupported media type", rid)
size = body["size_bytes"]
if isinstance(size, bool) or not isinstance(size, int):
return error_response(400, "invalid size", rid)
if size > config.MAX_UPLOAD_BYTES:
return error_response(413, "file exceeds upload limit", rid)
if size < 1:
return error_response(400, "invalid size", rid)
if await store.load_chat_for_member(chat_id, user["id"]) is None:
return error_response(404, "chat not found", rid)
started = await direct.begin_upload(chat_id, user["id"], body["filename"], content_type, size)
return _json(started, rid, 201)
async def complete_direct_upload(request: Request) -> Response:
rid = request.state.request_id
user = await _bearer_user(request)
upload_id = request.path_params["upload_id"]
if validate_uuid(upload_id):
return error_response(400, "invalid upload id", rid)
raw = await request.body()
body, err = parse_json_body(raw)
if err:
return error_response(400, err, rid)
assert body is not None
if require_exact_keys(body, {"sha256"}, {"sha256"}):
return error_response(400, "invalid upload payload", rid)
digest = body["sha256"]
if not isinstance(digest, str) or re.fullmatch(r"[a-f0-9]{64}", digest) is None:
return error_response(400, "invalid checksum", rid)
created = await direct.complete_upload(upload_id, user["id"], digest)
return _json(created, rid, 201)
async def get_attachment(request: Request) -> Response:
rid = request.state.request_id
user = await _bearer_user(request)
+54
View File
@@ -14,6 +14,7 @@ from app.errors import AppError
logger = logging.getLogger("messenger.s3")
_client = None
_signer = None
def _client_or_create():
@@ -38,6 +39,26 @@ def _client_or_create():
return _client
def _signer_or_create():
"""Signs URLs for the host the browser/client can reach. It does not send traffic."""
global _signer
if _signer is None:
if not config.S3_ACCESS_KEY_ID or not config.S3_SECRET_ACCESS_KEY:
raise AppError(503, "service unavailable")
_signer = boto3.client(
"s3",
endpoint_url=config.S3_PUBLIC_ENDPOINT,
aws_access_key_id=config.S3_ACCESS_KEY_ID,
aws_secret_access_key=config.S3_SECRET_ACCESS_KEY,
region_name=config.S3_REGION,
config=Config(
signature_version="s3v4",
s3={"addressing_style": "path"},
),
)
return _signer
def _unavailable(exc: Exception) -> AppError:
code = ""
if isinstance(exc, ClientError):
@@ -91,6 +112,39 @@ async def get_bytes(key: str) -> bytes:
raise _unavailable(exc) from None
def presign_put(key: str, content_type: str, size: int) -> str:
return _signer_or_create().generate_presigned_url(
"put_object",
Params={
"Bucket": config.S3_BUCKET,
"Key": key,
"ContentType": content_type,
"ContentLength": size,
},
ExpiresIn=config.PRESIGN_TTL_SECONDS,
HttpMethod="PUT",
)
async def head_size(key: str) -> int | None:
def _head() -> int | None:
try:
response = _client_or_create().head_object(Bucket=config.S3_BUCKET, Key=key)
except ClientError as exc:
code = exc.response.get("Error", {}).get("Code", "")
if code in {"404", "NoSuchKey", "NotFound"}:
return None
raise
return int(response["ContentLength"])
try:
return await asyncio.to_thread(_head)
except AppError:
raise
except (BotoCoreError, ClientError) as exc:
raise _unavailable(exc) from None
async def delete_key(key: str) -> None:
def _delete() -> None:
try:
+4
View File
@@ -53,6 +53,8 @@ services:
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
ORPHAN_GRACE_SECONDS: ${ORPHAN_GRACE_SECONDS:-120}
S3_SWEEP_INTERVAL_SECONDS: ${S3_SWEEP_INTERVAL_SECONDS:-30}
S3_PUBLIC_ENDPOINT: ${S3_PUBLIC_ENDPOINT:-http://127.0.0.1:9000}
PRESIGN_TTL_SECONDS: ${PRESIGN_TTL_SECONDS:-60}
DB_POOL_MIN: "1"
DB_POOL_MAX: ${DB_POOL_MAX:-4}
DB_COMMAND_TIMEOUT: "5"
@@ -123,6 +125,8 @@ services:
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY}
ports:
- "127.0.0.1:9000:9000"
volumes:
- s3_data:/data
- ./infra/s3/seaweed-entrypoint.sh:/seaweed-entrypoint.sh:ro
+21
View File
@@ -0,0 +1,21 @@
{
"openapi": "3.1.0",
"info": {
"title": "Messenger lab4 optional direct upload",
"version": "1.4.0-track1"
},
"paths": {
"/api/v1/chats/{chat_id}/uploads": {
"post": {
"operationId": "beginDirectUpload",
"summary": "Issue a 60s PUT signature for one new object key. Does not replace POST /attachments."
}
},
"/api/v1/uploads/{upload_id}/complete": {
"post": {
"operationId": "completeDirectUpload",
"summary": "Check owner, size and sha256 of the object that arrived, then store ready metadata."
}
}
}
}
+26
View File
@@ -0,0 +1,26 @@
{
"hypothesis": "A presigned PUT does not enter the API, so API RX stays near zero at both sizes. complete reads the object once to check the checksum, and the proxy endpoint still receives the whole body.",
"presign_ttl_seconds": 60,
"sizes": [
{
"label": "small",
"bytes": 90243,
"proxy_rx_bytes": 96204,
"proxy_peak_api_mib": 61.0,
"direct_put_rx_bytes": 0,
"direct_put_peak_api_mib": 61.4,
"complete_rx_bytes": 94131,
"complete_peak_api_mib": 61.4
},
{
"label": "large",
"bytes": 1891338,
"proxy_rx_bytes": 1913606,
"proxy_peak_api_mib": 62.4,
"direct_put_rx_bytes": 960,
"direct_put_peak_api_mib": 62.5,
"complete_rx_bytes": 1900090,
"complete_peak_api_mib": 62.1
}
]
}
+15
View File
@@ -51,3 +51,18 @@ Authorization: Bearer {{sessionToken}}
Content-Type: image/png
< ../tests/fixtures/sample.png
### Optional direct upload. PUT the bytes to the returned url, then complete with the real sha256.
POST {{baseUrl}}/api/v1/chats/{{chatId}}/uploads
Authorization: Bearer {{sessionToken}}
Content-Type: application/json
{"filename":"sample.png","content_type":"image/png","size_bytes":1}
### Complete only after the PUT. Replace the id and checksum locally.
POST {{baseUrl}}/api/v1/uploads/REPLACE_WITH_UPLOAD_ID/complete
Authorization: Bearer {{sessionToken}}
Content-Type: application/json
{"sha256":"replace-with-lowercase-sha256-of-the-bytes"}
+1 -1
View File
@@ -6,4 +6,4 @@
Приложение строится из исходников. Не используйте заранее существующий образ преподавательского сервера вместо своей реализации. Restart hook перезапускает все соответствующие stores с volumes; failover hook останавливает только заданную API-реплику.
Выбран SeaweedFS `chrislusf/seaweedfs:3.85` (S3 gateway на порту 9000 внутри сети Compose). Образ MinIO на Docker Hub в этом окружении недоступен. Порты наружу не опубликованы, UI filer выключен (`-filer.exposeDirectoryData=false`). `infra/s3/seaweed-entrypoint.sh` поднимает одну ноду и выдаёт API-пользователю права только на bucket `messenger-attachments`. Анонимная identity без actions. `s3-init` идемпотентно создаёт bucket теми же ограниченными ключами.
Выбран SeaweedFS `chrislusf/seaweedfs:3.85` (S3 gateway на порту 9000 внутри сети Compose). Образ MinIO на Docker Hub в этом окружении недоступен. Порты наружу не опубликованы, UI filer выключен (`-filer.exposeDirectoryData=false`). `infra/s3/seaweed-entrypoint.sh` поднимает одну ноду и выдаёт API-пользователю права только на bucket `messenger-attachments`. Анонимная identity без actions. `s3-init` идемпотентно создаёт bucket теми же ограниченными ключами. Для прямой загрузки S3 API слушает только `127.0.0.1:9000`; UI и остальные порты наружу не открыты. Подпись живёт `PRESIGN_TTL_SECONDS` (60).
+22
View File
@@ -0,0 +1,22 @@
-- Optional presigned upload. The signed URL dies quickly. The row stays pending
-- until complete checks the object that actually landed in the bucket.
-- Incomplete rows are aborted by the same grace sweeper as other uncommitted keys.
CREATE TABLE direct_uploads (
id UUID PRIMARY KEY,
chat_id UUID NOT NULL REFERENCES chats (id) ON DELETE CASCADE,
owner_id UUID NOT NULL REFERENCES users (id),
filename TEXT NOT NULL,
content_type TEXT NOT NULL,
size_bytes INTEGER NOT NULL,
object_key TEXT NOT NULL UNIQUE,
status TEXT NOT NULL,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT direct_uploads_filename_len CHECK (char_length(filename) BETWEEN 1 AND 255),
CONSTRAINT direct_uploads_size CHECK (size_bytes BETWEEN 1 AND 10485760),
CONSTRAINT direct_uploads_type CHECK (content_type IN ('image/png', 'image/jpeg')),
CONSTRAINT direct_uploads_status CHECK (status IN ('pending', 'completed', 'aborted'))
);
CREATE INDEX direct_uploads_pending_idx ON direct_uploads (expires_at) WHERE status = 'pending';
+4 -3
View File
@@ -211,13 +211,14 @@ def main():
capture_output=True,
text=True,
)
if published.returncode == 0 and published.stdout.strip():
raise SystemExit("s3 port is published: " + published.stdout.strip())
binding = published.stdout.strip()
if published.returncode == 0 and binding and not binding.startswith("127.0.0.1:"):
raise SystemExit("s3 port is published outside loopback: " + binding)
summary = {
"anonymous_bucket_status": anonymous_status(),
"second_bucket": other_bucket_denied(),
"sweep": plant_and_sweep(),
"s3_port_published": False,
"s3_port_published": binding or False,
"memory": memory_sample(),
}
OUT.parent.mkdir(exist_ok=True)
+380
View File
@@ -0,0 +1,380 @@
#!/usr/bin/env python3
"""Track 1: presigned upload, negative cases, API traffic and RAM at two sizes.
The signed URL is never written to the evidence file.
"""
import base64
import hashlib
import json
import os
import struct
import subprocess
import threading
import time
import urllib.error
import urllib.request
import uuid
import zlib
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
OUT = ROOT / "evidence" / "track1.json"
BASE = "http://127.0.0.1:8080"
def expect(status, got, label):
if got == 500:
raise SystemExit("FAIL %s returned 500" % label)
if got != status:
raise SystemExit("FAIL %s expected %s got %s" % (label, status, got))
print("PASS", label, got)
class Api:
def __init__(self) -> None:
self.opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
def call(self, method, path, body=None, token=None, raw=None, content_type=None):
headers = {}
data = raw
if body is not None:
data = json.dumps(body).encode("utf-8")
headers["Content-Type"] = "application/json"
if content_type:
headers["Content-Type"] = content_type
if token:
headers["Authorization"] = "Bearer " + token
request = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)
try:
response = self.opener.open(request, timeout=60)
except urllib.error.HTTPError as error:
response = error
with response:
payload = response.read()
parsed = None
if payload and response.headers.get_content_type() == "application/json":
parsed = json.loads(payload)
return response.status, parsed, payload
def png(width: int, height: int) -> bytes:
signature = b"\x89PNG\r\n\x1a\n"
def chunk(kind: bytes, data: bytes) -> bytes:
return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)
ihdr = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)
raw = b"".join(b"\x00" + os.urandom(width * 3) for _ in range(height))
return signature + chunk(b"IHDR", ihdr) + chunk(b"IDAT", zlib.compress(raw, 1)) + chunk(b"IEND", b"")
def put_signed(url: str, data: bytes, content_type: str) -> int:
request = urllib.request.Request(url, data=data, method="PUT", headers={"Content-Type": content_type})
try:
response = urllib.request.urlopen(request, timeout=60)
except urllib.error.HTTPError as error:
response = error
with response:
response.read()
return response.status
def api_rx() -> int:
ids = subprocess.check_output(
["docker", "compose", "--env-file", ".env", "ps", "-q", "api"],
cwd=ROOT,
text=True,
).split()
total = 0
for container_id in ids:
raw = subprocess.check_output(["docker", "exec", container_id, "cat", "/proc/net/dev"], text=True)
for line in raw.splitlines():
if "eth0" not in line:
continue
total += int(line.split()[1])
return total
def peak_during(action):
peaks = []
stop = threading.Event()
def watch():
while not stop.is_set():
stats = subprocess.run(
["docker", "stats", "--no-stream", "--format", "{{.Name}} {{.MemUsage}}"],
capture_output=True,
text=True,
check=False,
)
for line in stats.stdout.splitlines():
if "mtusi2026lab4-api-" not in line:
continue
value = line.split()[1]
number = float(value[:-3])
if not value.lower().endswith("mib"):
number = number / 1024
peaks.append(number)
time.sleep(0.25)
thread = threading.Thread(target=watch)
thread.start()
before = api_rx()
result = action()
after = api_rx()
time.sleep(0.3)
stop.set()
thread.join()
return {"rx_bytes": after - before, "peak_api_mib": round(max(peaks) if peaks else 0, 1), "result": result}
def compose_psql(sql: str) -> None:
subprocess.run(
["docker", "compose", "--env-file", ".env", "exec", "-T", "postgres", "psql", "-U", "messenger", "-d", "messenger", "-c", sql],
cwd=ROOT,
check=True,
capture_output=True,
text=True,
)
def negatives(api, alice_token, bob_token, eve_token, chat_id):
small = png(16, 16)
status, _, _ = api.call(
"POST",
"/api/v1/chats/%s/uploads" % chat_id,
{"filename": "a/b.png", "content_type": "image/png", "size_bytes": len(small)},
token=alice_token,
)
expect(400, status, "direct filename")
status, _, _ = api.call(
"POST",
"/api/v1/chats/%s/uploads" % chat_id,
{"filename": "big.png", "content_type": "image/png", "size_bytes": 10485761},
token=alice_token,
)
expect(413, status, "direct declared size")
status, _, _ = api.call(
"POST",
"/api/v1/chats/%s/uploads" % chat_id,
{"filename": "x.png", "content_type": "image/png", "size_bytes": len(small)},
token=eve_token,
)
expect(404, status, "outsider presign")
status, started, _ = api.call(
"POST",
"/api/v1/chats/%s/uploads" % chat_id,
{"filename": "wait.png", "content_type": "image/png", "size_bytes": len(small)},
token=alice_token,
)
expect(201, status, "presign for incomplete")
status, _, _ = api.call(
"POST",
"/api/v1/uploads/%s/complete" % started["upload_id"],
{"sha256": hashlib.sha256(small).hexdigest()},
token=alice_token,
)
expect(409, status, "complete before PUT")
status, _, _ = api.call(
"POST",
"/api/v1/uploads/%s/complete" % started["upload_id"],
{"sha256": hashlib.sha256(small).hexdigest()},
token=bob_token,
)
expect(404, status, "complete by another member")
tampered = put_signed(started["url"] + "x", small, "image/png")
if tampered == 200:
raise SystemExit("FAIL tampered signature was accepted")
print("PASS tampered signature", tampered)
status, started, _ = api.call(
"POST",
"/api/v1/chats/%s/uploads" % chat_id,
{"filename": "badsum.png", "content_type": "image/png", "size_bytes": len(small)},
token=alice_token,
)
expect(201, status, "presign for bad checksum")
if put_signed(started["url"], small, "image/png") != 200:
raise SystemExit("FAIL signed PUT")
status, _, _ = api.call(
"POST",
"/api/v1/uploads/%s/complete" % started["upload_id"],
{"sha256": "0" * 64},
token=alice_token,
)
expect(400, status, "wrong checksum")
status, _, _ = api.call(
"POST",
"/api/v1/uploads/%s/complete" % started["upload_id"],
{"sha256": hashlib.sha256(small).hexdigest()},
token=alice_token,
)
expect(409, status, "complete after rejected checksum")
status, started, _ = api.call(
"POST",
"/api/v1/chats/%s/uploads" % chat_id,
{"filename": "late.png", "content_type": "image/png", "size_bytes": len(small)},
token=alice_token,
)
expect(201, status, "presign for expiry")
compose_psql(
"UPDATE direct_uploads SET expires_at = now() - interval '5 minutes' WHERE id = '%s'"
% started["upload_id"]
)
status, _, _ = api.call(
"POST",
"/api/v1/uploads/%s/complete" % started["upload_id"],
{"sha256": hashlib.sha256(small).hexdigest()},
token=alice_token,
)
expect(409, status, "expired upload")
digest = hashlib.sha256(small).hexdigest()
status, started, _ = api.call(
"POST",
"/api/v1/chats/%s/uploads" % chat_id,
{"filename": "ok.png", "content_type": "image/png", "size_bytes": len(small)},
token=alice_token,
)
expect(201, status, "presign happy path")
if "originals/" in json.dumps({k: v for k, v in started.items() if k != "url"}):
raise SystemExit("FAIL object key leaked outside the signed URL")
if put_signed(started["url"], small, "image/png") != 200:
raise SystemExit("FAIL happy PUT")
status, attachment, _ = api.call(
"POST",
"/api/v1/uploads/%s/complete" % started["upload_id"],
{"sha256": digest},
token=alice_token,
)
expect(201, status, "complete happy path")
if attachment["sha256"] != digest or attachment["status"] != "ready":
raise SystemExit("FAIL complete metadata")
status, _, downloaded = api.call("GET", "/api/v1/attachments/%s/content" % attachment["id"], token=bob_token)
expect(200, status, "member downloads direct upload")
if downloaded != small:
raise SystemExit("FAIL direct bytes")
status, _, _ = api.call(
"POST",
"/api/v1/uploads/%s/complete" % started["upload_id"],
{"sha256": digest},
token=alice_token,
)
expect(409, status, "second complete")
status, proxy, _ = api.call(
"POST",
"/api/v1/chats/%s/attachments?filename=proxy.png" % chat_id,
token=alice_token,
raw=small,
content_type="image/png",
)
expect(201, status, "proxy endpoint still works")
return proxy["id"]
def compare(api, token, chat_id):
rows = []
for label, image in (("small", png(200, 150)), ("large", png(900, 700))):
digest = hashlib.sha256(image).hexdigest()
proxy = peak_during(
lambda image=image: api.call(
"POST",
"/api/v1/chats/%s/attachments?filename=%s.png" % (chat_id, label),
token=token,
raw=image,
content_type="image/png",
)
)
if proxy["result"][0] != 201:
raise SystemExit("FAIL proxy %s %s" % (label, proxy["result"][0]))
status, started, _ = api.call(
"POST",
"/api/v1/chats/%s/uploads" % chat_id,
{"filename": label + "-direct.png", "content_type": "image/png", "size_bytes": len(image)},
token=token,
)
expect(201, status, "presign " + label)
direct_put = peak_during(lambda image=image, url=started["url"]: put_signed(url, image, "image/png"))
if direct_put["result"] != 200:
raise SystemExit("FAIL direct PUT %s %s" % (label, direct_put["result"]))
direct_complete = peak_during(
lambda digest=digest, upload_id=started["upload_id"]: api.call(
"POST",
"/api/v1/uploads/%s/complete" % upload_id,
{"sha256": digest},
token=token,
)
)
if direct_complete["result"][0] != 201:
raise SystemExit("FAIL complete %s %s" % (label, direct_complete["result"][0]))
rows.append(
{
"label": label,
"bytes": len(image),
"proxy_rx_bytes": proxy["rx_bytes"],
"proxy_peak_api_mib": proxy["peak_api_mib"],
"direct_put_rx_bytes": direct_put["rx_bytes"],
"direct_put_peak_api_mib": direct_put["peak_api_mib"],
"complete_rx_bytes": direct_complete["rx_bytes"],
"complete_peak_api_mib": direct_complete["peak_api_mib"],
}
)
print("MEASURED", rows[-1])
return rows
def main():
api = Api()
def register(label):
username = "t1_" + label + "_" + uuid.uuid4().hex[:10]
password = "T3st-" + uuid.uuid4().hex
status, body, _ = api.call(
"POST",
"/api/v1/users",
{"username": username, "display_name": "Track " + label, "password": password},
)
expect(201, status, "register " + label)
return username, password, body
def login(username, password):
basic = base64.b64encode((username + ":" + password).encode()).decode()
request = urllib.request.Request(
BASE + "/api/v1/auth/sessions",
method="POST",
headers={"Authorization": "Basic " + basic},
)
with urllib.request.urlopen(request, timeout=30) as response:
return json.loads(response.read())["token"]
alice_name, alice_password, alice = register("alice")
bob_name, bob_password, bob = register("bob")
eve_name, eve_password, eve = register("eve")
alice_token = login(alice_name, alice_password)
bob_token = login(bob_name, bob_password)
eve_token = login(eve_name, eve_password)
status, chat, _ = api.call(
"POST",
"/api/v1/chats",
{"title": "track1", "member_ids": [alice["id"], bob["id"]]},
token=alice_token,
)
expect(201, status, "chat")
negatives(api, alice_token, bob_token, eve_token, chat["id"])
rows = compare(api, alice_token, chat["id"])
summary = {
"hypothesis": "A presigned PUT does not enter the API, so API RX stays near zero at both sizes. complete reads the object once to check the checksum, and the proxy endpoint still receives the whole body.",
"presign_ttl_seconds": 60,
"sizes": rows,
}
OUT.parent.mkdir(exist_ok=True)
OUT.write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
print(json.dumps(summary, indent=2))
if __name__ == "__main__":
main()