The demo on a clean stand (61dcb6b) passes 13 of 13 checks: the dry-run
finds exactly the injected ids in every class, apply writes one
audit.attachment_discarded reason=reconcile per stale intent, the
original bytes leave the quarantine while random bytes of the same
length keep it, docker stop ends a pass hanging on a paused S3 in
0.11 s with exit code 143, and 74 passes during 3454 uploads lose no
ready attachment; 46 times the HEAD recheck kept an in-flight upload
out of the quarantine.
The cost matrix now reaches 100k: a shallow pass takes 0.108 / 1.032 /
3.761 / 20.211 s for 1k / 10k / 30k / 100k with ceil(N/1000) listings
and SQL pages and the same 87-88 MiB of RSS. t(100k)/t(10k) is 19.6,
outside the preregistered 10 +- 30 %, because a RustFS listing page
costs 91 ms at 1k and 197 ms at 100k objects in the prefix; the
reconciler's own CPU and SQL stay linear. The previous 30k series moves
to cost-b191623/, and the old after-checks give way to the full series
that follows the track.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The level 3 and 4 evidence was taken on 24f9d61 and a774a2f, before the
lab 3 fixes were carried over and before the reconciler. Take all of it
again in one series on 81dd816, whose application and stand code is the
final one: a fresh clone runs the launch block of the report from secrets
to reset-data, then one stand from clean volumes gives the smoke, stack,
limits, lb-check, failover of each replica and of the upload replica,
persistence, S3 outage, S3 privacy, lost-answer retry, upload faults, all
stack checks, storage layout and three runs of the upload memory matrix.
The three memory runs keep the peak with three slots at 297-345 MiB
against 512 MiB at the limit without them, but at 40 uploads of 10 MiB
the wait for a slot nears the 8 s request timeout and one run of three
gave 7 of 80 a 503. Those timeouts left 67 invisible intents; the record
shows the reconciler seeing them as fresh and the sweep removing all of
them after the grace period.
Message write load is taken with tests/load.py from a container on the
edge network: 2 106 RPS, no errors, replicas and PostgreSQL at their CPU
limits. The first try from the macOS host is kept as a discarded attempt,
it ran out of ephemeral ports while the replicas answered every request.
The report gets the defense answers, the load section, the evidence
commits, the upload limit order as the pipeline applies it, medians in
the track 2 cost table and the proposals over the word limit; the evidence
index lists every file with its command and commit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
REPORT.md gets the track 2 section: what the reconciler does and why it
is safe, the hypotheses written before the first run, the method of the
demo and the cost matrix with its screening, both result tables, the
verdicts and the limits. H1 holds. H2 holds for the pass cost: time is
linear (t(30k)/t(10k) = 3.73, 3.05 in the series of 29.10), the listing
is 95 % of it, list requests are ceil(N/1000) and SQL one more for the
empty last keyset page, RSS stays at 82.5-83.8 MiB; --deep is 4.3-5
times dearer per object, not ten. The criteria table, the failure table,
the limits, AGENTS.md and the openspec context name the track rows, and
evidence/reconcile/README.md describes every file.
On a stand from a clean state make test, test-persistence,
test-failover, restore, reconcile, lint, check, validate and
student-test (342 passed) all pass; the dry-run after these scenarios
finds 3 objects, 3 rows and no discrepancy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
make reconcile-cost on a fresh stand at b191623: RustFS 2 CPU / 3 GiB on
cpus 11-12, PostgreSQL on 13-14, the reconciler on 15, three dry-run
passes per point; no run was disturbed, foreign CPU stayed at most 0.2
cores in the VM and 0.01 on the pinned cores. Medians:
objects mode t, s lists sql gets rss, MiB
1000 shallow 0.105 1 2 0 82.5
10000 shallow 1.000 10 11 0 83.6
30000 shallow 3.729 30 31 0 83.8
10000 page100 2.058 100 101 0 81.0
1000 deep 0.493 1 2 1000 83.0
10000 deep 5.041 10 11 10000 84.1
30000 deep 16.150 30 31 30000 84.1
The listing takes 95 % of a shallow pass. The first full series of
29.10 (5ca2cdc, not screened, another lab under k6 at the time) is kept
in cost-2026-10-29 for comparison between days.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
make reconcile-demo on a fresh stand at 5c3c238, 12 of 12 checks true.
The dry-run found exactly what was injected on both sides (3 stale
orphans, 3 missing objects, 3 shorter and 3 same-size rewritten objects,
6 stale intents with and without an object, 2 fresh orphans and 2 fresh
intents) and changed nothing; apply fixed the same 18 and the second
apply did nothing. The independent check outside the API found no
violation and no lost ready attachment; the 9 damaged rows sit in
quarantine with their message links, the metadata answers 200 and the
content 404, and a restored object left quarantine. A pass during a held
lock was skipped, a pass with s3 stopped exited with 1 in 1.6 s and
changed nothing. 75 apply passes during 75 s of uploads through the
balancer removed only the 10 injected leftovers, saw 247 objects of
uploads in flight as fresh orphans and kept them: all 3418 uploads are
ready with matching SHA-256.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Filling RustFS 1.0.4 to 100k objects under one prefix did not finish in
two attempts: a write gets slower as the prefix grows (about 8 ms in an
empty prefix against 32 ms at 28k objects, 13 writes per second with 16
parallel writers), with enough memory and no background scanner load.
The cost matrix now stops at 30k objects with four parallel writers,
which still spans a factor of 30 for the linearity check; the design
and the spec of the change say so, and both aborted attempts with their
1k and 10k points and the diagnosis are kept in
evidence/reconcile/cost-aborted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Evidence comes from the running stand at a774a2f: upload-faults.txt
with the negative suite through the balancer (36 user errors, no 5xx
in the replica logs) and the compensation, crash, sweep, abort and
stall checks; upload-memory with per-run JSON and a summary for 64
and 3 slots, 1 and 10 MiB and 10, 20 and 40 concurrent uploads plus
the mmap threshold comparison; s3-security.txt with 23 checks; and
regression-level4.txt showing that smoke, persistence, failover of
each replica, the upload failover, S3 outage and abort scenarios, the
limits policy and all 15 stand checks still pass. The lint and own
test log now counts 335 tests. The upload-faults target stamps its
output with the date and commit.
REPORT.md gains rows 4.1 to 4.4 and a regression row, the compensation
and sweep mechanism with its grace period, where each upload limit and
buffer cap applies, the new variables, level 4 failure rows and the
memory experiment with hypothesis, method, results and limits. AGENTS
lists the new targets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Evidence comes from the stand started from an empty state and from a
fresh clone: the public smoke, the contract checks, lint and 325 own
tests, the storage layout with an out-of-API comparison of a stored
object against its metadata, failover of each replica under writes, the
upload failover scenario, persistence with StartedAt of every service
including S3, the S3 outage scenario and 22 S3 privacy and bootstrap
checks. The storage evidence script now covers the attachment tables
and the S3 secrets.
REPORT.md follows the lab 4 template: launch from a clean clone,
architecture with the source of truth for each fact, the write order
between PostgreSQL and S3 and what each failure leaves behind, where
each upload limit applies, roles and services, the .env.example
mapping, grade 3 rows and the grade 3 failure table. The design notes
the botocore Expect header that hung uploads behind the port proxy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>