Files
mtusi2026lab5/infra/s3/README.md
T

4.2 KiB

Private object storage

s3 runs SeaweedFS 4.53 in one process. The image is pinned by digest in compose.yaml and internal/testenv/s3.go. seaweedfs.options is shared by the stand and the disposable integration-test server.

S3 HTTP (8333) and SeaweedFS's companion gRPC (18333) listen on the private container network. This image has no flag to disable the companion listener; its identity-cache mutations reject remote peers, which the integration test checks. Master, volume and filer HTTP/gRPC endpoints bind to loopback. No host port is published. /data contains both object volumes and the image's LevelDB filer metadata; the named s3data volume must be retained together with PostgreSQL data.

Run sh scripts/init-secrets.sh before starting the stand. It preserves the existing PostgreSQL and Valkey credentials and creates three independent S3 identities plus the static server configuration. Credential values are never printed. Repeated initialization keeps all existing keys. Set S3_BUCKET in the environment or the local .env before the first initialization; changing it later requires a planned migration of identities and retained data.

The server loads its static identity secret before serving requests. There is no anonymous identity. The bootstrap identity has administrative rights and its individual key files are mounted only in s3-init. The API identity has Read:<bucket> and Write:<bucket>; it cannot list objects or administer buckets. The reconciler identity adds List:<bucket>. All three identities are necessarily present in the S3 server's own identity configuration secret.

s3-init creates the bucket idempotently and waits for storage before API replicas start. An anonymous 403 and bucket metadata alone do not prove that retained volume locations have registered with the elected master. Bootstrap first reads a persistent .health/storage-v1 marker (creating it only when absent), then verifies a fresh write/read/delete. This startup-only barrier uses the caller's deadline, capped at 60 seconds; ordinary S3 operations and API readiness do not retry through it. The marker is outside the reconciler's staging/ and blobs/ namespaces. scripts/restart.sh already runs this bootstrap before restoring API replicas.

The bootstrap uses /usr/local/bin/s3init. The optional recon profile runs /usr/local/bin/reconcile, with reconciler credentials mapped to the standard S3_ACCESS_KEY_ID_FILE and S3_SECRET_ACCESS_KEY_FILE paths:

docker compose --profile recon run --rm reconcile
docker compose --profile recon run --rm reconcile --apply

The API receives its own standard S3 key files. Its background reconciliation worker must construct a separate store from S3_RECONCILE_ACCESS_KEY_ID_FILE and S3_RECONCILE_SECRET_ACCESS_KEY_FILE; listing with the API identity is denied. S3 is intentionally absent from API readiness so text traffic can remain available during an object-store outage.

The S3 cgroup is limited to one CPU, 512 MiB, and 256 processes. Upload limits are four concurrent requests and 64 MiB in flight; reader buffers have a 32 MiB budget and the chunk cache is disabled. Disk allocation is capped at 64 volumes of approximately 128 MiB each (with normal volume rollover overhead); this is a lab capacity limit, not a filesystem quota.

Each API replica has 2 GiB and GOMEMLIMIT=1536MiB, with one image decoder. Pixels alone are not the full budget: progressive JPEG retains coefficients, and a wide RGBA64 PNG keeps two large scanlines beside its pixel buffer. The 40 MP RGB/CMYK progressive JPEG and wide-PNG measurements are recorded in evidence/decoder-memory-*.json; this does not change the S3 service's 512 MiB limit.

Verify the adapter and real server with Docker running:

go test -race -count=1 ./internal/platform/s3store

testenv.StartS3(ctx) returns Store with reconciler rights, APIStore, Admin, endpoint and credentials. Recreate(ctx) replaces only its own container while keeping its named volume and does not recreate the bucket. It waits for the same retained-marker and write/read/delete barrier before returning. Start(ctx) does so as well. Always call Close() to remove the fixture's container and volume.