forked from mtusi2026/mtusi2026lab4
lab4: вложения в приватном S3 (SeaweedFS), метаданные в PostgreSQL
- compose: сервис s3 (SeaweedFS 4.46 по digest, volume s3data, внутренняя сеть storage,
web-интерфейсы только на 127.0.0.1 внутри контейнера), одноразовый s3-init;
- три пары ключей S3: admin (только bootstrap), api (Read+Write своего bucket),
reconciler (+List); bootstrap идемпотентен и проверяет приватность и права;
- миграция 0004: attachments (pending -> ready, случайный object_key), message_attachments,
post_message_v2 с проверкой вложений в той же транзакции;
- POST /chats/{id}/attachments: лимиты до чтения/декодирования, потоковая проверка PNG
(CRC + zlib до точного размера), JPEG через djpeg -strict, семафор загрузок;
- GET метаданных и /content (стриминг из S3, без redirect), readiness с S3,
HAProxy: деградированный режим без S3 через /health/ready/core;
- janitor незавершённых загрузок с grace period.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
d916d09f7b
commit
335df4b2b2
@@ -24,7 +24,7 @@ def write(name: str, value: str) -> None:
|
||||
print(f"keep secrets/{name}")
|
||||
return
|
||||
path.write_text(value + "\n", encoding="utf-8", newline="\n")
|
||||
# Readable by the container users (postgres, valkey, app uid 10001) through the
|
||||
# Readable by the container users (postgres, valkey, seaweedfs, app uid 10001) through the
|
||||
# compose bind mount. Local synthetic lab credentials only.
|
||||
os.chmod(path, 0o644)
|
||||
print(f"created secrets/{name}")
|
||||
@@ -40,6 +40,11 @@ def main() -> int:
|
||||
write("valkey_password", secrets.token_urlsafe(24))
|
||||
write("database_url", f"postgresql://messenger:{read('postgres_password')}@postgres:5432/messenger")
|
||||
write("cache_url", f"redis://:{read('valkey_password')}@valkey:6379/0")
|
||||
# Lab 4: three separate S3 key pairs (SeaweedFS identities, infra/seaweedfs/start.sh):
|
||||
# admin = bucket bootstrap only; api = Read+Write on the bucket; reconciler = + List.
|
||||
for who in ("admin", "api", "reconciler"):
|
||||
write(f"s3_{who}_access_key_id", "AK" + secrets.token_hex(9).upper())
|
||||
write(f"s3_{who}_secret_access_key", secrets.token_hex(20))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
+5
-4
@@ -1,6 +1,6 @@
|
||||
#!/bin/sh
|
||||
# Restart hook for `make test-persistence ACTION_SCRIPT=scripts/restart.sh`.
|
||||
# Restarts EVERY process of the stand: LB, all API replicas, PostgreSQL and Valkey.
|
||||
# Restarts EVERY process of the stand: LB, all API replicas, PostgreSQL, Valkey and S3 (SeaweedFS).
|
||||
# Volumes, schema and data are preserved: no `down -v`, no FLUSH, no re-seed.
|
||||
# Output is suppressed by the test; run it by hand to see the steps.
|
||||
set -eu
|
||||
@@ -8,10 +8,11 @@ cd "$(dirname "$0")/.."
|
||||
|
||||
# 1. Entry point and API first: no request is served while the stores restart underneath.
|
||||
docker compose stop lb api
|
||||
# 2. Stores: clean shutdown (PostgreSQL checkpoint, Valkey AOF fsync), then start again.
|
||||
docker compose restart postgres valkey
|
||||
# 2. Stores: clean shutdown (PostgreSQL checkpoint, Valkey AOF fsync, SeaweedFS volume/filer
|
||||
# flush), then start again. The S3 bytes live on the named volume s3data.
|
||||
docker compose restart postgres valkey s3
|
||||
# 3. API replicas (wait for healthy stores via depends_on), then the LB. HAProxy resolves
|
||||
# the new replica addresses through Docker DNS by itself - no stale backend list.
|
||||
docker compose start api
|
||||
docker compose start lb
|
||||
echo "restart.sh: lb, api (all replicas), postgres and valkey restarted with volumes preserved"
|
||||
echo "restart.sh: lb, api (all replicas), postgres, valkey and s3 restarted with volumes preserved"
|
||||
|
||||
Reference in New Issue
Block a user