4.2 KiB
Private object storage
s3 runs SeaweedFS 4.53 in one process. The image is pinned by digest in
compose.yaml and internal/testenv/s3.go. seaweedfs.options is shared by
the stand and the disposable integration-test server.
S3 HTTP (8333) and SeaweedFS's companion gRPC (18333) listen on the private
container network. This image has no flag to disable the companion listener;
its identity-cache mutations reject remote peers, which the integration test
checks. Master, volume and filer HTTP/gRPC endpoints bind to loopback.
No host port is published. /data
contains both object volumes and the image's LevelDB filer metadata; the
named s3data volume must be retained together with PostgreSQL data.
Run sh scripts/init-secrets.sh before starting the stand. It preserves the
existing PostgreSQL and Valkey credentials and creates three independent S3
identities plus the static server configuration. Credential values are never
printed. Repeated initialization keeps all existing keys. Set S3_BUCKET in
the environment or the local .env before the first initialization; changing
it later requires a planned migration of identities and retained data.
The server loads its static identity secret before serving requests. There is
no anonymous identity. The bootstrap identity has administrative rights and
its individual key files are mounted only in s3-init. The API identity has
Read:<bucket> and Write:<bucket>; it cannot list objects or administer
buckets. The reconciler identity adds List:<bucket>. All three identities
are necessarily present in the S3 server's own identity configuration secret.
s3-init creates the bucket idempotently and waits for storage before API
replicas start. An anonymous 403 and bucket metadata alone do not prove that
retained volume locations have registered with the elected master. Bootstrap
first reads a persistent .health/storage-v1 marker (creating it only when
absent), then verifies a fresh write/read/delete. This startup-only barrier
uses the caller's deadline, capped at 60 seconds; ordinary S3 operations and
API readiness do not retry through it. The marker is outside the reconciler's
staging/ and blobs/ namespaces. scripts/restart.sh already runs this
bootstrap before restoring API replicas.
The bootstrap uses /usr/local/bin/s3init. The optional recon profile runs
/usr/local/bin/reconcile, with reconciler credentials mapped to the standard
S3_ACCESS_KEY_ID_FILE and S3_SECRET_ACCESS_KEY_FILE paths:
docker compose --profile recon run --rm reconcile
docker compose --profile recon run --rm reconcile --apply
The API receives its own standard S3 key files. Its background reconciliation
worker must construct a separate store from
S3_RECONCILE_ACCESS_KEY_ID_FILE and S3_RECONCILE_SECRET_ACCESS_KEY_FILE;
listing with the API identity is denied. S3 is intentionally absent from API
readiness so text traffic can remain available during an object-store outage.
The S3 cgroup is limited to one CPU, 512 MiB, and 256 processes. Upload limits are four concurrent requests and 64 MiB in flight; reader buffers have a 32 MiB budget and the chunk cache is disabled. Disk allocation is capped at 64 volumes of approximately 128 MiB each (with normal volume rollover overhead); this is a lab capacity limit, not a filesystem quota.
Each API replica has 2 GiB and GOMEMLIMIT=1536MiB, with one image decoder.
Pixels alone are not the full budget: progressive JPEG retains coefficients,
and a wide RGBA64 PNG keeps two large scanlines beside its pixel buffer.
The 40 MP RGB/CMYK progressive JPEG and wide-PNG measurements are recorded in
evidence/decoder-memory-*.json; this does not change the S3 service's 512 MiB limit.
Verify the adapter and real server with Docker running:
go test -race -count=1 ./internal/platform/s3store
testenv.StartS3(ctx) returns Store with reconciler rights, APIStore,
Admin, endpoint and credentials. Recreate(ctx) replaces only its own
container while keeping its named volume and does not recreate the bucket.
It waits for the same retained-marker and write/read/delete barrier before
returning. Start(ctx) does so as well. Always call Close() to remove the
fixture's container and volume.