Commit Graph
65 Commits
Author SHA1 Message Date
Artyom 3402fd69d4 Expose bounded seed progress for long reconciler experiments 2026-11-06 02:32:19 +03:00
Artyom 827b783be3 Reject disturbed reconciler measurements and preserve every series 2026-11-06 02:29:53 +03:00
Artyom 751aaed4b1 Preregister SQLAlchemy reconciler validation and full cost matrix 2026-11-06 02:28:14 +03:00
Artyom 57caeb6b5a Verify grade four faults memory cleanup and clean clone on SQLAlchemy 2026-11-06 02:27:08 +03:00
Artyom cf41f7c64d Separate historical results and audit every lab4 review finding 2026-11-06 02:07:59 +03:00
Artyom 6c706a727e Preserve evidence runs and volumes while propagating failures 2026-11-06 02:02:04 +03:00
Artyom c7e7a86627 Propose isolated grade four verification and review closure 2026-11-06 02:01:27 +03:00
Artyom 02f385866f Verify and archive lab4 grade three on SQLAlchemy 2026-11-06 02:00:43 +03:00
Artyom 763b025557 Fail attachment scenarios on invalid outcomes 2026-11-06 01:59:03 +03:00
Artyom 1aec062aba Capture isolated evidence with full provenance and exit codes 2026-11-06 01:54:30 +03:00
Artyom 5596ca935c Propose fresh lab4 grade three evidence on SQLAlchemy 2026-11-06 01:54:07 +03:00
Artyom 41700c27e5 Archive the lab3 foundation forward port 2026-11-06 01:48:09 +03:00
Artyom 8e4532f6f5 Forward-port lab3 SQLAlchemy and Alembic foundation
Carry source f591fa64e5 forward from 72a1e57. Preserve lab4 attachment transactions, reconciliation locks, quarantine, and existing schema data through verified legacy adoption. Keep report and evidence work in progress untouched.
2026-11-06 01:47:30 +03:00
Artyom dd4021b678 Propose the lab3 SQLAlchemy foundation forward port 2026-11-06 01:32:54 +03:00
ArtyomandClaude Opus 5.5 568ff6643e Archive the storage reconciler hardening change
A pass releases the quarantine only after the object hashes to the row
again, reads exactly ceil(N/page) SQL pages, publishes
audit.attachment_discarded reason=reconcile for the intents it drops,
is assembled by dishka with the same storage provider as the API and
ends any failure as reconcile.failed with exit code 1. The demo passes
13 of 13 checks, and the cost matrix up to 100k confirms the request
counts and the constant memory of H2 and refutes its linear time at
100k (19.6 instead of 10 +- 30 %, the RustFS listing page grows from
91 to 197 ms). student-test (348), test, persistence and failover pass
on the stand. The requirements are synced into lab4-storage-reconciler.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:54:40 +03:00
ArtyomandClaude Opus 5.5 d2e684f0cf Record the track 2 demo and the cost of a pass up to 100k
The demo on a clean stand (61dcb6b) passes 13 of 13 checks: the dry-run
finds exactly the injected ids in every class, apply writes one
audit.attachment_discarded reason=reconcile per stale intent, the
original bytes leave the quarantine while random bytes of the same
length keep it, docker stop ends a pass hanging on a paused S3 in
0.11 s with exit code 143, and 74 passes during 3454 uploads lose no
ready attachment; 46 times the HEAD recheck kept an in-flight upload
out of the quarantine.

The cost matrix now reaches 100k: a shallow pass takes 0.108 / 1.032 /
3.761 / 20.211 s for 1k / 10k / 30k / 100k with ceil(N/1000) listings
and SQL pages and the same 87-88 MiB of RSS. t(100k)/t(10k) is 19.6,
outside the preregistered 10 +- 30 %, because a RustFS listing page
costs 91 ms at 1k and 197 ms at 100k objects in the prefix; the
reconciler's own CPU and SQL stay linear. The previous 30k series moves
to cost-b191623/, and the old after-checks give way to the full series
that follows the track.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:54:29 +03:00
ArtyomandClaude Opus 5.5 6ffb90d1d1 Script the overload leftovers check with raw output
overload-leftovers.txt was assembled by hand with summarised reconciler
reports. make overload-leftovers now runs the dry-run, the status
counts, the wait for the sweep after the grace period, the audit and
swept counts of the replicas and the dry-run afterwards, and writes
their raw output.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:17:15 +03:00
ArtyomandClaude Opus 5.5 61dcb6bf13 Keep the audit journal of each reconciler pass in its report
The driver now stores the audit events a pass writes to stderr next to
its JSON report, and the apply check requires one
audit.attachment_discarded reason=reconcile for every injected stale
intent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:09:15 +03:00
ArtyomandClaude Opus 5.5 21937f920f Wait for S3 to turn healthy after the paused pass
docker compose up --wait failed at once after unpausing S3: the probes
that ran while it was paused had already marked it unhealthy. The demo
now polls the health status of the container until it is healthy again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:04:09 +03:00
ArtyomandClaude Opus 5.5 f1ee385a1b Tie the track tooling to keys, sets and a clean series
reconcile-inject.sh wrote an orphan under a key whose uuid differed
from the id it printed, so the demo could only compare counts; it now
uses one uuid for both, and the dry-run check compares the found and
the injected sets per class. The heal step returns the original bytes
under one quarantined key and random bytes of the same length under
another, and checks that only the first leaves the quarantine. The
load phase sums rechecked over its passes, and a new step stops a pass
hanging on a paused S3 with docker stop and checks exit code 143 within
seconds and a free lock afterwards.

reconcile_track.py cost clears the run files of the previous series and
measures 1k, 10k, 30k and 100k with --deep everywhere. The scenarios
record date, commit and docker inspect of the replicas or of S3 at the
moment they are down. make load-evidence and make clean-clone-evidence
rerun the message load and the clean clone block with raw output, and
make upload-memory passes ARGS to reproduce the run directories.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:01:39 +03:00
ArtyomandClaude Opus 5.5 3de1dcdc84 Release the quarantine only after the object hashes to the row again
A pass without --deep released the quarantine of a missing or resized
attachment as soon as an object of the same size appeared under its
key, so other bytes of the same length went back to /content. The pass
now streams and hashes the object of every quarantined row whose size
matches again, deep or not, and releases it only when the SHA-256
matches; other bytes keep the row quarantined. Consistent data still
costs no GET.

The keyset reads page + 1 rows, so a pass takes exactly ceil(N/page)
SQL pages instead of one more for the empty tail. The reconciler is
assembled by dishka from PlatformProvider and the attachment storage
provider that the API uses too, and it publishes the events of the
intents it drops, so audit.attachment_discarded reason=reconcile
reaches the journal. Any failure of a pass, not only Unavailable, ends
as reconcile.failed with the error type and exit code 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:01:39 +03:00
ArtyomandClaude Opus 5.5 b6c04f2296 Propose hardening the storage reconciler as an openspec change
The review of track 2 found that a pass without --deep released the
quarantine of a missing or resized attachment as soon as an object of
the same size reappeared, without checking its SHA-256; that the keyset
read an empty last page, one SQL request more than the spec promised;
that the discarded intents of a pass never reached the audit journal;
that the entry point duplicated the S3 client wiring and printed a
traceback for any failure other than Unavailable; and that the cost
matrix stopped at 30k while the preregistered H2 was about 100k.

The change hashes the object of a quarantined row before releasing it,
reads page + 1 rows, assembles the reconciler with dishka and publishes
its events, turns any pass failure into reconcile.failed, links the
injected orphans to the found ones by key and measures up to 100k.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:56:15 +03:00
ArtyomandClaude Opus 5.5 adca7c6edc Archive the attachment boundaries change
Upload checks the token before the declared size, a slow but steady
transfer outlives the request timeout while a stalled one is still cut,
the API and the reconciler hold separate minimal S3 keys, and the
one-off services run as non-root with an init process. On the stand a
3 MiB upload at 256 KiB/s took 12.1 s at the replica and got 201, a
3 MiB download at 192 KiB/s took 12.8 s and matched the SHA-256, the
API key is denied a bucket listing, and docker stop ends a hanging
reconciler pass in 0.13 s with exit code 143. student-test (345),
test-faults (19), persistence, failover and the lab 4 scenarios pass.
The requirements are synced into lab4-attachments, lab4-upload-resilience
and lab4-object-storage.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:53:53 +03:00
ArtyomandClaude Opus 5.5 2891b37e75 Split the S3 keys of the API and the reconciler, drop root from one-offs
The API key now has only GetObject, PutObject and DeleteObject on the
attachment objects: the server never lists the bucket, never runs
multipart uploads and has its region set. The reconciler gets its own
user messenger-reconciler with ListBucket, GetObject and DeleteObject,
created by s3-init from a secret of its own, so the API replicas can no
longer list the bucket. The cost seeding writes objects and therefore
runs in a one-off API container under the API key. s3-check proves the
API key cannot list and checks the reconciler key; its snapshot drops
dated fields with a JSON filter instead of grep, which left a dangling
comma, and its first line carries the date and the commit.

s3-init and s3-cli run as uid 65534, and migrate and reconciler get an
init process, so SIGTERM reaches python and docker stop ends a pass in
a fraction of a second instead of waiting for SIGKILL. limits.py and CI
check the users of the one-off services. Stack tests move a 3 MiB file
up and down through the balancer at 256 and 192 KiB/s, longer than the
request timeout, and check 401 before the declared size.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:53:39 +03:00
ArtyomandClaude Opus 5.5 54099e1da3 Check the token before the declared size and pace the request deadline
The pipeline now runs the guard first, so an anonymous upload with a too
large Content-Length gets 401 with WWW-Authenticate, as the contract
asks; the guard does not read the body, so 413 still comes before the
first body byte for an authenticated client.

The single request deadline cut slow but steady uploads and downloads:
a client slower than about 1.3 MB/s could not move 10 MiB in 8 s. Each
body chunk the application receives or sends now moves the deadline to
min(now, start + moved / MIN_TRANSFER_RATE_BYTES) + REQUEST_TIMEOUT,
128 KiB/s by default, like mod_reqtimeout with a ceiling: a steady
transfer lives as long as it moves, a stalled or crawling body is cut
after the request timeout as before, and the work after the last body
byte still gets at most the request timeout. The pipeline's own error
responses bypass the pacing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:53:39 +03:00
ArtyomandClaude Opus 5.5 96e934ed8f Propose tightening the lab 4 attachment boundaries as an openspec change
The review of lab 4 found four gaps at the attachment boundaries: an
anonymous upload with a too large Content-Length got 413 before 401,
the single request deadline cut slow but steady uploads and downloads,
the API S3 key could list the bucket and run multipart actions, and the
one-off services ran as root or as PID 1 without an init process.

The change checks the token first, paces the request deadline by the
bytes moved, splits the S3 keys of the API and of the reconciler and
runs the one-off services as non-root with an init process.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:41:27 +03:00
ArtyomandClaude Opus 5.5 c8cb263c77 Retake the lab 4 evidence on the final code and finish the report
The level 3 and 4 evidence was taken on 24f9d61 and a774a2f, before the
lab 3 fixes were carried over and before the reconciler. Take all of it
again in one series on 81dd816, whose application and stand code is the
final one: a fresh clone runs the launch block of the report from secrets
to reset-data, then one stand from clean volumes gives the smoke, stack,
limits, lb-check, failover of each replica and of the upload replica,
persistence, S3 outage, S3 privacy, lost-answer retry, upload faults, all
stack checks, storage layout and three runs of the upload memory matrix.

The three memory runs keep the peak with three slots at 297-345 MiB
against 512 MiB at the limit without them, but at 40 uploads of 10 MiB
the wait for a slot nears the 8 s request timeout and one run of three
gave 7 of 80 a 503. Those timeouts left 67 invisible intents; the record
shows the reconciler seeing them as fresh and the sweep removing all of
them after the grace period.

Message write load is taken with tests/load.py from a container on the
edge network: 2 106 RPS, no errors, replicas and PostgreSQL at their CPU
limits. The first try from the macOS host is kept as a discarded attempt,
it ran out of ephemeral ports while the replicas answered every request.

The report gets the defense answers, the load section, the evidence
commits, the upload limit order as the pipeline applies it, medians in
the track 2 cost table and the proposals over the word limit; the evidence
index lists every file with its command and commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 07:55:34 +03:00
ArtyomandClaude Opus 5.5 81dd816df2 Archive the storage reconciler change
Track 2 of lab 4 is closed. One reconciler pass merges the sorted bucket
listing with keyset pages of attachments by object key, finds objects
without a row, ready rows without an object or with other bytes and
abandoned intents, deletes only old objects without a row after a
second look at the row and old intents under SKIP LOCKED, and puts
damaged ready rows into a reversible quarantine instead of deleting
them. On the stand the dry-run found exactly the 22 injected
discrepancies of seven classes, apply fixed the 18 fixable ones, the
repeat did nothing, and 75 passes during 3418 uploads lost no ready
attachment. A shallow pass costs 0.105 / 1.000 / 3.729 s for 1k / 10k /
30k objects at a flat 83 MiB, 95 % of it in the RustFS listing; --deep
is 4.3-5 times dearer per object.

For the defence: make reconcile-demo shows injection, dry-run, apply,
the repeat, quarantine through the API, healing, the held lock, a
stopped S3 and the run under uploads in about four minutes; the cost
matrix is evidence/reconcile/cost/summary.txt (make reconcile-cost
repeats it on a fresh stand in about seven).

The lab4-storage-reconciler spec is added and the quarantine rule of
lab4-attachments is updated in the main specs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 07:19:42 +03:00
ArtyomandClaude Opus 5.5 58cae16ef8 Report track 2 and check levels 3 and 4 after it
REPORT.md gets the track 2 section: what the reconciler does and why it
is safe, the hypotheses written before the first run, the method of the
demo and the cost matrix with its screening, both result tables, the
verdicts and the limits. H1 holds. H2 holds for the pass cost: time is
linear (t(30k)/t(10k) = 3.73, 3.05 in the series of 29.10), the listing
is 95 % of it, list requests are ceil(N/1000) and SQL one more for the
empty last keyset page, RSS stays at 82.5-83.8 MiB; --deep is 4.3-5
times dearer per object, not ten. The criteria table, the failure table,
the limits, AGENTS.md and the openspec context name the track rows, and
evidence/reconcile/README.md describes every file.

On a stand from a clean state make test, test-persistence,
test-failover, restore, reconcile, lint, check, validate and
student-test (342 passed) all pass; the dry-run after these scenarios
finds 3 objects, 3 rows and no discrepancy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 07:19:16 +03:00
ArtyomandClaude Opus 5.5 8cb5a664c6 Keep the start allowance and retries in the cost summary
The edit that added them to the plan of the series missed after the
formatter split the line, so summary.json of df345b3 lists only the two
thresholds; that series ran with the defaults of b191623 (1.5 and 0.5
core-s, three retries). The summary of the next series records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:55:50 +03:00
ArtyomandClaude Opus 5.5 df345b32e1 Record the cost of a reconciler pass up to 30k objects
make reconcile-cost on a fresh stand at b191623: RustFS 2 CPU / 3 GiB on
cpus 11-12, PostgreSQL on 13-14, the reconciler on 15, three dry-run
passes per point; no run was disturbed, foreign CPU stayed at most 0.2
cores in the VM and 0.01 on the pinned cores. Medians:

  objects  mode     t, s    lists  sql  gets  rss, MiB
     1000  shallow  0.105       1    2     0      82.5
    10000  shallow  1.000      10   11     0      83.6
    30000  shallow  3.729      30   31     0      83.8
    10000  page100  2.058     100  101     0      81.0
     1000  deep     0.493       1    2  1000      83.0
    10000  deep     5.041      10   11 10000      84.1
    30000  deep    16.150      30   31 30000      84.1

The listing takes 95 % of a shallow pass. The first full series of
29.10 (5ca2cdc, not screened, another lab under k6 at the time) is kept
in cost-2026-10-29 for comparison between days.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:55:45 +03:00
ArtyomandClaude Opus 5.5 b1916236f4 Record the track 2 demo on the stand
make reconcile-demo on a fresh stand at 5c3c238, 12 of 12 checks true.
The dry-run found exactly what was injected on both sides (3 stale
orphans, 3 missing objects, 3 shorter and 3 same-size rewritten objects,
6 stale intents with and without an object, 2 fresh orphans and 2 fresh
intents) and changed nothing; apply fixed the same 18 and the second
apply did nothing. The independent check outside the API found no
violation and no lost ready attachment; the 9 damaged rows sit in
quarantine with their message links, the metadata answers 200 and the
content 404, and a restored object left quarantine. A pass during a held
lock was skipped, a pass with s3 stopped exited with 1 in 1.6 s and
changed nothing. 75 apply passes during 75 s of uploads through the
balancer removed only the 10 injected leftovers, saw 247 objects of
uploads in flight as fresh orphans and kept them: all 3418 uploads are
ready with matching SHA-256.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:48:07 +03:00
ArtyomandClaude Opus 5.5 5c3c238d17 Screen the cost runs for foreign load and check a pass without S3
The cost matrix shares the Docker VM with the stands and builds of other
labs. Each run now reads the VM-wide /proc/stat and the cpu.stat of the
s3 and postgres cgroups before and after the pass, and the reconciler
reports its own CPU time. Foreign CPU over the window is the busy time
minus ours minus the cost of starting a container (1.5 core-s in the VM
and 0.5 on the pinned cores, measured on the idle stand). A run with
more than 2 foreign cores in the VM or 0.25 on the pinned cores goes to
cost/disturbed and is repeated after 15 s of quiet, up to three times;
docker stats of the whole host are saved before and after the series.

The demo also runs a pass with s3 stopped: it must exit with code 1 in
bounded time, log reconcile.failed and leave PostgreSQL and the bucket
as they were.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:35:59 +03:00
ArtyomandClaude Opus 5.5 3b3cb5bd7f End a failed reconciler pass with the dependency error
The merge loop runs inside the TaskGroup that holds the --deep checks,
so a listing or page failure in the middle of a pass left the group as
an ExceptionGroup. The entry point catches Unavailable only: the pass
still exited with code 1, but through an uncaught traceback instead of
the reconcile.failed record. The pass now unwraps the group and raises
the Unavailable with its original cause; the API test fails the listing
on the second page of a deep apply pass and expects Unavailable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:31:46 +03:00
ArtyomandClaude Opus 5.5 8cd30488d2 Archive the carry over of the lab 3 fixes change
Sync the delta specs into the main specs: zones are linked by the
schema and the ports main.py assembles, the event bus and the network
dependencies name S3 and Valkey, an outsider does not wait for chat
locks, an empty cursor is rejected, a request cancelled at shutdown gets
503 in the contract format, the body limit counts received bytes per
route, logout withdraws the revocation mark on a database failure,
detection runs three times, failover-each records the stopped replica,
lost answers are retried on the stand, posting checks its input in a
fixed order and make up waits for every replica in the balancer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:25:44 +03:00
ArtyomandClaude Opus 5.5 e927afdb09 Carry over the lab 3 fixes made after 61c0c28
Bring the system changes of lab 3 from 61c0c28..72a1e57 onto the lab 4
code, resolving each conflict by meaning and keeping everything lab 4
added:

- a message is posted with one SQL statement under the chat row lock,
  and the attachment links go into the same statement; a foreign key
  violation on the attachment link still answers 404 and leaves nothing
  behind. Text and the attachment list are checked before the write, an
  outsider gets 404 for both, as before the carry over;
- the request pipeline counts the body bytes actually received against
  the limit of the route, so a chunked body over 1 MiB gets 413 and the
  upload route keeps its MAX_UPLOAD_BYTES limit;
- a request cancelled at shutdown gets 503 unavailable in the contract
  format, the worker waits a second less than the rest of the shutdown
  budget, SHUTDOWN_TIMEOUT_SECONDS becomes 6 s and the stop hooks 7 s;
- readiness sits behind a Readiness port with the draining flag in
  shared infrastructure, and the drain window is logged through Logger;
- logout revokes through SessionRevocations and withdraws the mark when
  PostgreSQL rejects the delete; an empty cursor is 400;
- EventBus.project is gone, the sweep test subscribes and waits on a
  queue; domain tests check attachment_list instead of Chat.post;
- make up waits for every replica in the balancer, BASE_URL follows the
  lb port, the image is tagged by the compose project, retry-check and
  faults-evidence are added, detection runs three times, failover-each
  and rolling-update record the stand usage, CI checks numeric uids.
  down also takes the tools and reconcile profiles of lab 4.

Track 1 tooling of lab 3 (scaling.py, its samplers, tools/ and the CPU
pinning of api, lb, valkey and k6) stays out; stand-usage.sh queries the
database size itself instead of the track 1 sampler.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:25:23 +03:00
ArtyomandClaude Opus 5.5 7ac1a4a11b Propose carrying over the lab 3 fixes as an openspec change
Lab 4 started from the lab 3 commit 61c0c28, and lab 3 changed up to
72a1e57 after its track and two reviews. The change lists what moves
over and how it meets the lab 4 code: the one-statement message write
takes the attachment links along, the body limit counts received bytes
per route, the shutdown, readiness and logout fixes come as they are,
and the stand tooling keeps the lab 4 profiles and CPU pinning. Delta
specs bring the requirements lab 3 changed, adapted to three zones and
the object store; track 1 tooling stays out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:15:48 +03:00
ArtyomandClaude Opus 5.5 5ca2cdc9af Measure the pass cost up to 30k objects
Filling RustFS 1.0.4 to 100k objects under one prefix did not finish in
two attempts: a write gets slower as the prefix grows (about 8 ms in an
empty prefix against 32 ms at 28k objects, 13 writes per second with 16
parallel writers), with enough memory and no background scanner load.
The cost matrix now stops at 30k objects with four parallel writers,
which still spans a factor of 30 for the linearity check; the design
and the spec of the change say so, and both aborted attempts with their
1k and 10k points and the diagnosis are kept in
evidence/reconcile/cost-aborted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 04:40:24 +03:00
ArtyomandClaude Opus 5.5 7dca40c55e Give the balancer time to see the replicas before it counts as unhealthy
The balancer reports itself unhealthy while no replica is up, and its
start period of 10 s was shorter than the time the replicas need on a
busy host: they wait for migrate and s3-init and have a 30 s start
period of their own. docker compose up --wait then failed on a cold
start although the stand came up seconds later. A 60 s start period
with 1 s probes keeps the balancer starting, not unhealthy, until the
first replica is in rotation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:52:27 +03:00
ArtyomandClaude Opus 5.5 de9dc61ebc Give RustFS room for the 100k point of the cost matrix
The first cost run stalled while filling the bucket to 100k objects:
RustFS keeps object metadata in memory, at about 45k objects it hit
its 1 GiB limit, the cgroup kept reclaiming and PutObject slowed from
about 240 to 11 objects per second. The S3 CPU limit is now taken from
S3_CPUS (default 1, as before), and the cost run gives RustFS 2 pinned
CPUs and 3 GiB for every point, so all N are measured with the same
limits; seeding uses 16 parallel writes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:50:30 +03:00
ArtyomandClaude Opus 5.5 a335f654ed Add the track 2 fault injection, invariant check and experiment driver
reconcile-inject.sh and reconcile_check.py come from the prepared kit:
the first breaks either side on purpose (an object without a row, a
removed object, another size, the same size with other bytes, an old
intent with and without its object), the second is an oracle outside
the API that classifies every row and object from psql and rc, hashes
objects with --deep and reports ready attachments lost against a saved
set.

reconcile_track.py demo uploads through the balancer, links some
attachments to messages, injects each class on both sides with old and
fresh variants, and checks that a deep dry-run finds exactly what was
injected and changes nothing, that apply fixes exactly that and a
second apply does nothing, that broken rows and their message links
survive in quarantine with metadata 200 and content 404, that a
restored object heals, that a second pass is skipped while the lock is
held, and that passes in a loop during 75 s of uploads lose nothing.
Intents are injected 300 s old, between the reconciler grace and the
API sweep grace, so only the reconciler removes them.

reconcile_track.py cost snapshots docker stats of the whole host, pins
S3, PostgreSQL and the reconciler to their own CPUs, fills the stand to
each N with seed_storage.py run in the API image (parallel PutObject and
COPY), and records three dry runs per N and mode with a median table.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:23:59 +03:00
ArtyomandClaude Opus 5.5 51fa19a8ee Run the reconciler as a one-off service of the stand
The reconciler service under the reconcile profile runs the API image
with the reconciler module as entrypoint and --dry-run as the default
command, so docker compose run --rm --no-deps reconciler takes only
flags. It gets the database URL and the limited API key of S3, which
already allows listing, reading and deleting objects, a read-only root,
1 CPU and 256 MiB, and a longer S3 read timeout for listing pages.
S3, PostgreSQL and the reconciler accept an optional cpuset from the
environment so measurements can pin them; the default is no pinning.

make reconcile ARGS=... runs a pass, make reconcile-demo and make
reconcile-cost drive the track experiments, and the limits policy now
covers the new service.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:17:54 +03:00
ArtyomandClaude Opus 5.5 91731fd317 Reconcile attachment metadata with the bucket in one merge pass
StorageReconciler walks the sorted bucket listing and keyset pages of
attachments ordered by object key side by side, so a pass is linear
and keeps one page of each side in memory. An object without a row is
deleted only when it is older than grace plus skew and a lookup by key
right before the delete still finds no row; a ready row whose object
is missing, has another size or, with the deep check, another digest
is put in quarantine after a HEAD recheck and is never deleted; a stale
intent is claimed with SKIP LOCKED and dropped object first; a
quarantined row whose object matches again is released. Deep checks
hash objects in a bounded task group, each action uses its own unit of
work, and failures of single actions are counted instead of aborting.
An advisory lock lets one pass run at a time. Content of a quarantined
attachment answers 404 without touching S3, its metadata is unchanged.

The storage port gains listing pages and HEAD, the repository keyset
pages, lookup by key, claim and quarantine marking; the platform logger
can write to another stream. The one-off entry point
python -m messenger.attachments.entrypoints.reconcile is a dry run
unless --apply is given, assembles the adapters with the API key of S3
and prints a JSON report with counts per class, samples and the cost
of the pass to stdout while the journal goes to stderr.

Tests run the reconciler against its own migrated database with every
class, apply and repeat, the row that appears after the listing, the
busy lock and the release, and check paged byte-order listing and HEAD
against RustFS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:12:10 +03:00
ArtyomandClaude Opus 5.5 62986245a2 Add the quarantine columns for the storage reconciler
Migration 0004 adds a nullable quarantine reason (missing, size or
digest mismatch) and the time it was set, both null or both present.
Nullable columns without a default only touch the catalogue, so the
running version keeps working while the migration is applied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:11:59 +03:00
ArtyomandClaude Opus 5.5 148868bdfb Propose the storage reconciler as an openspec change
Track 2 of grade 5: a one-off reconciler that merges the sorted bucket
listing with keyset pages of the attachments table, so a pass is linear
in time and holds one page per side in memory. It finds objects without
a row, ready rows without an object or with another size or digest, and
abandoned intents; by default it only reports, with --apply it deletes
old orphans after a row recheck, drops stale intents object first, and
quarantines broken ready rows instead of deleting them, so referenced
attachments survive and a restored object heals. A quarantined
attachment answers 404 on its content. An advisory lock keeps a single
pass at a time.

The design states the hypotheses on correctness and on the cost of a
pass, the reproducible method behind make reconcile-demo and make
reconcile-cost, and the acceptance criteria.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:05:42 +03:00
ArtyomandClaude Opus 5.5 d2744bc1ac Archive the upload resilience change
Grade 4 of lab 4 is closed. A failed ready commit after the object
write is compensated at once under a row lock, intents abandoned by a
crash or a lost write are swept with their objects after a 900 s grace
period by every replica with SKIP LOCKED, and nothing younger, nothing
ready and nothing referenced is touched. A client abort is a 400 and a
stalled body a 503 after 8 s, both leaving nothing behind. Three upload
slots per worker, capped socket buffers in the balancer and the API
and a fixed glibc mmap threshold keep 40 concurrent 10 MiB uploads at
330 of 512 MiB per replica with every upload succeeding, while without
the slots the same load hits the limit and loses 37 of 80. 36 user
errors through the balancer give no 5xx, and 23 checks prove the
bucket private, the API key limited, the console closed and the
bootstrap idempotent.

For the defence: run make upload-faults to show the negative suite,
the compensation, the killed replica with its leftover swept after the
grace period while the fresh one stays, and the abort and stall; show
the memory summary and make s3-check.

The lab4-upload-resilience spec is added, and the write order of
lab4-attachments and the console and API key requirements of
lab4-object-storage are updated in the main specs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:25:46 +03:00
ArtyomandClaude Opus 5.5 2b19277b1e Record level 4 evidence and fill in the report
Evidence comes from the running stand at a774a2f: upload-faults.txt
with the negative suite through the balancer (36 user errors, no 5xx
in the replica logs) and the compensation, crash, sweep, abort and
stall checks; upload-memory with per-run JSON and a summary for 64
and 3 slots, 1 and 10 MiB and 10, 20 and 40 concurrent uploads plus
the mmap threshold comparison; s3-security.txt with 23 checks; and
regression-level4.txt showing that smoke, persistence, failover of
each replica, the upload failover, S3 outage and abort scenarios, the
limits policy and all 15 stand checks still pass. The lint and own
test log now counts 335 tests. The upload-faults target stamps its
output with the date and commit.

REPORT.md gains rows 4.1 to 4.4 and a regression row, the compensation
and sweep mechanism with its grace period, where each upload limit and
buffer cap applies, the new variables, level 4 failure rows and the
memory experiment with hypothesis, method, results and limits. AGENTS
lists the new targets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:24:41 +03:00
ArtyomandClaude Opus 5.5 a774a2fbbd Measure upload memory and bound the buffers along the chain
upload_memory.py recreates the replicas and the balancer for every run
of 10, 20 and 40 concurrent uploads of 1 and 10 MiB with 64 slots
(effectively no limit) and with 3, drives them through lab4_seed.py
from the prepared tooling, and keeps per-run JSON plus a summary with
the cgroup memory.peak of each replica and of the balancer and the
oom_kill counters. The seed script now also records memory.current
before the load and reads X-Instance-Id regardless of case.

The first run showed that slots alone only move a waiting body out of
the worker: with 3 slots and 40 uploads of 10 MiB the kernel grew the
socket buffers of HAProxy to about 110 MB and the 128 MiB balancer was
killed. HAProxy now caps its client and server socket buffers at 64
KiB and the API replicas cap tcp_rmem and tcp_wmem at 256 KiB, which
keeps the balancer at 30 MiB. glibc raised its mmap threshold after
the first large free and kept freed 10 MiB bodies in the heap, so the
API fixes MALLOC_MMAP_THRESHOLD_ at 128 KiB.

With these settings 40 concurrent uploads of 10 MiB peak at 323 MiB of
512 per replica and all 80 succeed, while without the slot limit the
same load reaches 512 MiB, a worker is killed and only 31 of 80
succeed. UPLOAD_CONCURRENCY stays at 3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:14:09 +03:00
ArtyomandClaude Opus 5.5 98df351235 Check upload faults and user errors against the stand
test_attachment_faults.py runs through the balancer against the real
PostgreSQL and RustFS. The negative suite sends a body one byte over
the limit with Content-Length (answered before any body byte) and
chunked, a decompression bomb, truncated and disguised images, missing
and foreign content types, dangerous names, a stranger's reads and
references, an attachment of another chat and invalid lists, checks
each code and error body, and finds no 5xx in the replica logs and no
extra rows.

Faults come from triggers installed for the module and dropped after
it: an AFTER INSERT trigger remembers the id and key of each fault
intent, a BEFORE UPDATE trigger rejects the ready commit for
fault-fail names and holds it for 1.8 s for fault-crash names, while
the test finds the waiting backend's client address and kills that
replica with SIGKILL. The rejected commit leaves neither row nor
object right away and an audit line says compensation; a crash leaves
a pending row and an object, the one aged past the grace period is
swept with its object while the fresh one stays, and an aged ready
attachment in a message still downloads. A body cut after 256 KiB is
logged as 400 with nothing written, and a stalled body gets 503 after
8 s with nothing written.

make upload-faults stores the output in evidence/upload-faults.txt
without tracebacks of locals, and the CI acceptance job runs it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:48:46 +03:00
ArtyomandClaude Opus 5.5 28055a8551 Pass the upload limits to the API and check the console from the network
The API replicas get UPLOAD_CONCURRENCY, UPLOAD_GRACE_SECONDS,
UPLOAD_SWEEP_INTERVAL_SECONDS and UPLOAD_SWEEP_BATCH through the S3
anchor of compose.yaml with the same defaults as the settings, and
.env.example documents them. A value in the shell overrides the
default, which the memory experiment uses to recreate the replicas with
a different number of slots.

s3-check.sh now also asks the console port of the S3 service from a
separate container on the stand network and expects a refused
connection, so 23 checks cover ports, the console inside and outside
the container, the repeated bootstrap, the rights of the API key and
anonymous access.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:40:56 +03:00