Commit Graph
55 Commits
Author SHA1 Message Date
Artyom 75b93bbba5 Forward final lab3 session security and validation fixes 2026-11-06 12:07:00 +03:00
Artyom f0fc084fbb Propose forwarding final lab3 review fixes 2026-11-06 12:01:57 +03:00
Artyom d7478726a8 Complete SQLAlchemy reconciler evidence and close lab4 review findings 2026-11-06 03:37:31 +03:00
Artyom 3402fd69d4 Expose bounded seed progress for long reconciler experiments 2026-11-06 02:32:19 +03:00
Artyom 827b783be3 Reject disturbed reconciler measurements and preserve every series 2026-11-06 02:29:53 +03:00
Artyom 751aaed4b1 Preregister SQLAlchemy reconciler validation and full cost matrix 2026-11-06 02:28:14 +03:00
Artyom 57caeb6b5a Verify grade four faults memory cleanup and clean clone on SQLAlchemy 2026-11-06 02:27:08 +03:00
Artyom cf41f7c64d Separate historical results and audit every lab4 review finding 2026-11-06 02:07:59 +03:00
Artyom c7e7a86627 Propose isolated grade four verification and review closure 2026-11-06 02:01:27 +03:00
Artyom 02f385866f Verify and archive lab4 grade three on SQLAlchemy 2026-11-06 02:00:43 +03:00
Artyom 763b025557 Fail attachment scenarios on invalid outcomes 2026-11-06 01:59:03 +03:00
Artyom 1aec062aba Capture isolated evidence with full provenance and exit codes 2026-11-06 01:54:30 +03:00
Artyom 5596ca935c Propose fresh lab4 grade three evidence on SQLAlchemy 2026-11-06 01:54:07 +03:00
Artyom 41700c27e5 Archive the lab3 foundation forward port 2026-11-06 01:48:09 +03:00
Artyom 8e4532f6f5 Forward-port lab3 SQLAlchemy and Alembic foundation
Carry source f591fa64e525679a31c9db7f942d36c789e1a12f forward from 72a1e57. Preserve lab4 attachment transactions, reconciliation locks, quarantine, and existing schema data through verified legacy adoption. Keep report and evidence work in progress untouched.
2026-11-06 01:47:30 +03:00
Artyom dd4021b678 Propose the lab3 SQLAlchemy foundation forward port 2026-11-06 01:32:54 +03:00
ArtyomandClaude Opus 5.5 568ff6643e Archive the storage reconciler hardening change
A pass releases the quarantine only after the object hashes to the row
again, reads exactly ceil(N/page) SQL pages, publishes
audit.attachment_discarded reason=reconcile for the intents it drops,
is assembled by dishka with the same storage provider as the API and
ends any failure as reconcile.failed with exit code 1. The demo passes
13 of 13 checks, and the cost matrix up to 100k confirms the request
counts and the constant memory of H2 and refutes its linear time at
100k (19.6 instead of 10 +- 30 %, the RustFS listing page grows from
91 to 197 ms). student-test (348), test, persistence and failover pass
on the stand. The requirements are synced into lab4-storage-reconciler.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:54:40 +03:00
ArtyomandClaude Opus 5.5 f1ee385a1b Tie the track tooling to keys, sets and a clean series
reconcile-inject.sh wrote an orphan under a key whose uuid differed
from the id it printed, so the demo could only compare counts; it now
uses one uuid for both, and the dry-run check compares the found and
the injected sets per class. The heal step returns the original bytes
under one quarantined key and random bytes of the same length under
another, and checks that only the first leaves the quarantine. The
load phase sums rechecked over its passes, and a new step stops a pass
hanging on a paused S3 with docker stop and checks exit code 143 within
seconds and a free lock afterwards.

reconcile_track.py cost clears the run files of the previous series and
measures 1k, 10k, 30k and 100k with --deep everywhere. The scenarios
record date, commit and docker inspect of the replicas or of S3 at the
moment they are down. make load-evidence and make clean-clone-evidence
rerun the message load and the clean clone block with raw output, and
make upload-memory passes ARGS to reproduce the run directories.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:01:39 +03:00
ArtyomandClaude Opus 5.5 3de1dcdc84 Release the quarantine only after the object hashes to the row again
A pass without --deep released the quarantine of a missing or resized
attachment as soon as an object of the same size appeared under its
key, so other bytes of the same length went back to /content. The pass
now streams and hashes the object of every quarantined row whose size
matches again, deep or not, and releases it only when the SHA-256
matches; other bytes keep the row quarantined. Consistent data still
costs no GET.

The keyset reads page + 1 rows, so a pass takes exactly ceil(N/page)
SQL pages instead of one more for the empty tail. The reconciler is
assembled by dishka from PlatformProvider and the attachment storage
provider that the API uses too, and it publishes the events of the
intents it drops, so audit.attachment_discarded reason=reconcile
reaches the journal. Any failure of a pass, not only Unavailable, ends
as reconcile.failed with the error type and exit code 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:01:39 +03:00
ArtyomandClaude Opus 5.5 b6c04f2296 Propose hardening the storage reconciler as an openspec change
The review of track 2 found that a pass without --deep released the
quarantine of a missing or resized attachment as soon as an object of
the same size reappeared, without checking its SHA-256; that the keyset
read an empty last page, one SQL request more than the spec promised;
that the discarded intents of a pass never reached the audit journal;
that the entry point duplicated the S3 client wiring and printed a
traceback for any failure other than Unavailable; and that the cost
matrix stopped at 30k while the preregistered H2 was about 100k.

The change hashes the object of a quarantined row before releasing it,
reads page + 1 rows, assembles the reconciler with dishka and publishes
its events, turns any pass failure into reconcile.failed, links the
injected orphans to the found ones by key and measures up to 100k.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:56:15 +03:00
ArtyomandClaude Opus 5.5 adca7c6edc Archive the attachment boundaries change
Upload checks the token before the declared size, a slow but steady
transfer outlives the request timeout while a stalled one is still cut,
the API and the reconciler hold separate minimal S3 keys, and the
one-off services run as non-root with an init process. On the stand a
3 MiB upload at 256 KiB/s took 12.1 s at the replica and got 201, a
3 MiB download at 192 KiB/s took 12.8 s and matched the SHA-256, the
API key is denied a bucket listing, and docker stop ends a hanging
reconciler pass in 0.13 s with exit code 143. student-test (345),
test-faults (19), persistence, failover and the lab 4 scenarios pass.
The requirements are synced into lab4-attachments, lab4-upload-resilience
and lab4-object-storage.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:53:53 +03:00
ArtyomandClaude Opus 5.5 2891b37e75 Split the S3 keys of the API and the reconciler, drop root from one-offs
The API key now has only GetObject, PutObject and DeleteObject on the
attachment objects: the server never lists the bucket, never runs
multipart uploads and has its region set. The reconciler gets its own
user messenger-reconciler with ListBucket, GetObject and DeleteObject,
created by s3-init from a secret of its own, so the API replicas can no
longer list the bucket. The cost seeding writes objects and therefore
runs in a one-off API container under the API key. s3-check proves the
API key cannot list and checks the reconciler key; its snapshot drops
dated fields with a JSON filter instead of grep, which left a dangling
comma, and its first line carries the date and the commit.

s3-init and s3-cli run as uid 65534, and migrate and reconciler get an
init process, so SIGTERM reaches python and docker stop ends a pass in
a fraction of a second instead of waiting for SIGKILL. limits.py and CI
check the users of the one-off services. Stack tests move a 3 MiB file
up and down through the balancer at 256 and 192 KiB/s, longer than the
request timeout, and check 401 before the declared size.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:53:39 +03:00
ArtyomandClaude Opus 5.5 96e934ed8f Propose tightening the lab 4 attachment boundaries as an openspec change
The review of lab 4 found four gaps at the attachment boundaries: an
anonymous upload with a too large Content-Length got 413 before 401,
the single request deadline cut slow but steady uploads and downloads,
the API S3 key could list the bucket and run multipart actions, and the
one-off services ran as root or as PID 1 without an init process.

The change checks the token first, paces the request deadline by the
bytes moved, splits the S3 keys of the API and of the reconciler and
runs the one-off services as non-root with an init process.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:41:27 +03:00
ArtyomandClaude Opus 5.5 81dd816df2 Archive the storage reconciler change
Track 2 of lab 4 is closed. One reconciler pass merges the sorted bucket
listing with keyset pages of attachments by object key, finds objects
without a row, ready rows without an object or with other bytes and
abandoned intents, deletes only old objects without a row after a
second look at the row and old intents under SKIP LOCKED, and puts
damaged ready rows into a reversible quarantine instead of deleting
them. On the stand the dry-run found exactly the 22 injected
discrepancies of seven classes, apply fixed the 18 fixable ones, the
repeat did nothing, and 75 passes during 3418 uploads lost no ready
attachment. A shallow pass costs 0.105 / 1.000 / 3.729 s for 1k / 10k /
30k objects at a flat 83 MiB, 95 % of it in the RustFS listing; --deep
is 4.3-5 times dearer per object.

For the defence: make reconcile-demo shows injection, dry-run, apply,
the repeat, quarantine through the API, healing, the held lock, a
stopped S3 and the run under uploads in about four minutes; the cost
matrix is evidence/reconcile/cost/summary.txt (make reconcile-cost
repeats it on a fresh stand in about seven).

The lab4-storage-reconciler spec is added and the quarantine rule of
lab4-attachments is updated in the main specs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 07:19:42 +03:00
ArtyomandClaude Opus 5.5 58cae16ef8 Report track 2 and check levels 3 and 4 after it
REPORT.md gets the track 2 section: what the reconciler does and why it
is safe, the hypotheses written before the first run, the method of the
demo and the cost matrix with its screening, both result tables, the
verdicts and the limits. H1 holds. H2 holds for the pass cost: time is
linear (t(30k)/t(10k) = 3.73, 3.05 in the series of 29.10), the listing
is 95 % of it, list requests are ceil(N/1000) and SQL one more for the
empty last keyset page, RSS stays at 82.5-83.8 MiB; --deep is 4.3-5
times dearer per object, not ten. The criteria table, the failure table,
the limits, AGENTS.md and the openspec context name the track rows, and
evidence/reconcile/README.md describes every file.

On a stand from a clean state make test, test-persistence,
test-failover, restore, reconcile, lint, check, validate and
student-test (342 passed) all pass; the dry-run after these scenarios
finds 3 objects, 3 rows and no discrepancy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 07:19:16 +03:00
ArtyomandClaude Opus 5.5 df345b32e1 Record the cost of a reconciler pass up to 30k objects
make reconcile-cost on a fresh stand at b191623: RustFS 2 CPU / 3 GiB on
cpus 11-12, PostgreSQL on 13-14, the reconciler on 15, three dry-run
passes per point; no run was disturbed, foreign CPU stayed at most 0.2
cores in the VM and 0.01 on the pinned cores. Medians:

  objects  mode     t, s    lists  sql  gets  rss, MiB
     1000  shallow  0.105       1    2     0      82.5
    10000  shallow  1.000      10   11     0      83.6
    30000  shallow  3.729      30   31     0      83.8
    10000  page100  2.058     100  101     0      81.0
     1000  deep     0.493       1    2  1000      83.0
    10000  deep     5.041      10   11 10000      84.1
    30000  deep    16.150      30   31 30000      84.1

The listing takes 95 % of a shallow pass. The first full series of
29.10 (5ca2cdc, not screened, another lab under k6 at the time) is kept
in cost-2026-10-29 for comparison between days.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:55:45 +03:00
ArtyomandClaude Opus 5.5 b1916236f4 Record the track 2 demo on the stand
make reconcile-demo on a fresh stand at 5c3c238, 12 of 12 checks true.
The dry-run found exactly what was injected on both sides (3 stale
orphans, 3 missing objects, 3 shorter and 3 same-size rewritten objects,
6 stale intents with and without an object, 2 fresh orphans and 2 fresh
intents) and changed nothing; apply fixed the same 18 and the second
apply did nothing. The independent check outside the API found no
violation and no lost ready attachment; the 9 damaged rows sit in
quarantine with their message links, the metadata answers 200 and the
content 404, and a restored object left quarantine. A pass during a held
lock was skipped, a pass with s3 stopped exited with 1 in 1.6 s and
changed nothing. 75 apply passes during 75 s of uploads through the
balancer removed only the 10 injected leftovers, saw 247 objects of
uploads in flight as fresh orphans and kept them: all 3418 uploads are
ready with matching SHA-256.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:48:07 +03:00
ArtyomandClaude Opus 5.5 5c3c238d17 Screen the cost runs for foreign load and check a pass without S3
The cost matrix shares the Docker VM with the stands and builds of other
labs. Each run now reads the VM-wide /proc/stat and the cpu.stat of the
s3 and postgres cgroups before and after the pass, and the reconciler
reports its own CPU time. Foreign CPU over the window is the busy time
minus ours minus the cost of starting a container (1.5 core-s in the VM
and 0.5 on the pinned cores, measured on the idle stand). A run with
more than 2 foreign cores in the VM or 0.25 on the pinned cores goes to
cost/disturbed and is repeated after 15 s of quiet, up to three times;
docker stats of the whole host are saved before and after the series.

The demo also runs a pass with s3 stopped: it must exit with code 1 in
bounded time, log reconcile.failed and leave PostgreSQL and the bucket
as they were.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:35:59 +03:00
ArtyomandClaude Opus 5.5 3b3cb5bd7f End a failed reconciler pass with the dependency error
The merge loop runs inside the TaskGroup that holds the --deep checks,
so a listing or page failure in the middle of a pass left the group as
an ExceptionGroup. The entry point catches Unavailable only: the pass
still exited with code 1, but through an uncaught traceback instead of
the reconcile.failed record. The pass now unwraps the group and raises
the Unavailable with its original cause; the API test fails the listing
on the second page of a deep apply pass and expects Unavailable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:31:46 +03:00
ArtyomandClaude Opus 5.5 8cd30488d2 Archive the carry over of the lab 3 fixes change
Sync the delta specs into the main specs: zones are linked by the
schema and the ports main.py assembles, the event bus and the network
dependencies name S3 and Valkey, an outsider does not wait for chat
locks, an empty cursor is rejected, a request cancelled at shutdown gets
503 in the contract format, the body limit counts received bytes per
route, logout withdraws the revocation mark on a database failure,
detection runs three times, failover-each records the stopped replica,
lost answers are retried on the stand, posting checks its input in a
fixed order and make up waits for every replica in the balancer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:25:44 +03:00
ArtyomandClaude Opus 5.5 e927afdb09 Carry over the lab 3 fixes made after 61c0c28
Bring the system changes of lab 3 from 61c0c28..72a1e57 onto the lab 4
code, resolving each conflict by meaning and keeping everything lab 4
added:

- a message is posted with one SQL statement under the chat row lock,
  and the attachment links go into the same statement; a foreign key
  violation on the attachment link still answers 404 and leaves nothing
  behind. Text and the attachment list are checked before the write, an
  outsider gets 404 for both, as before the carry over;
- the request pipeline counts the body bytes actually received against
  the limit of the route, so a chunked body over 1 MiB gets 413 and the
  upload route keeps its MAX_UPLOAD_BYTES limit;
- a request cancelled at shutdown gets 503 unavailable in the contract
  format, the worker waits a second less than the rest of the shutdown
  budget, SHUTDOWN_TIMEOUT_SECONDS becomes 6 s and the stop hooks 7 s;
- readiness sits behind a Readiness port with the draining flag in
  shared infrastructure, and the drain window is logged through Logger;
- logout revokes through SessionRevocations and withdraws the mark when
  PostgreSQL rejects the delete; an empty cursor is 400;
- EventBus.project is gone, the sweep test subscribes and waits on a
  queue; domain tests check attachment_list instead of Chat.post;
- make up waits for every replica in the balancer, BASE_URL follows the
  lb port, the image is tagged by the compose project, retry-check and
  faults-evidence are added, detection runs three times, failover-each
  and rolling-update record the stand usage, CI checks numeric uids.
  down also takes the tools and reconcile profiles of lab 4.

Track 1 tooling of lab 3 (scaling.py, its samplers, tools/ and the CPU
pinning of api, lb, valkey and k6) stays out; stand-usage.sh queries the
database size itself instead of the track 1 sampler.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:25:23 +03:00
ArtyomandClaude Opus 5.5 7ac1a4a11b Propose carrying over the lab 3 fixes as an openspec change
Lab 4 started from the lab 3 commit 61c0c28, and lab 3 changed up to
72a1e57 after its track and two reviews. The change lists what moves
over and how it meets the lab 4 code: the one-statement message write
takes the attachment links along, the body limit counts received bytes
per route, the shutdown, readiness and logout fixes come as they are,
and the stand tooling keeps the lab 4 profiles and CPU pinning. Delta
specs bring the requirements lab 3 changed, adapted to three zones and
the object store; track 1 tooling stays out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:15:48 +03:00
ArtyomandClaude Opus 5.5 5ca2cdc9af Measure the pass cost up to 30k objects
Filling RustFS 1.0.4 to 100k objects under one prefix did not finish in
two attempts: a write gets slower as the prefix grows (about 8 ms in an
empty prefix against 32 ms at 28k objects, 13 writes per second with 16
parallel writers), with enough memory and no background scanner load.
The cost matrix now stops at 30k objects with four parallel writers,
which still spans a factor of 30 for the linearity check; the design
and the spec of the change say so, and both aborted attempts with their
1k and 10k points and the diagnosis are kept in
evidence/reconcile/cost-aborted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 04:40:24 +03:00
ArtyomandClaude Opus 5.5 a335f654ed Add the track 2 fault injection, invariant check and experiment driver
reconcile-inject.sh and reconcile_check.py come from the prepared kit:
the first breaks either side on purpose (an object without a row, a
removed object, another size, the same size with other bytes, an old
intent with and without its object), the second is an oracle outside
the API that classifies every row and object from psql and rc, hashes
objects with --deep and reports ready attachments lost against a saved
set.

reconcile_track.py demo uploads through the balancer, links some
attachments to messages, injects each class on both sides with old and
fresh variants, and checks that a deep dry-run finds exactly what was
injected and changes nothing, that apply fixes exactly that and a
second apply does nothing, that broken rows and their message links
survive in quarantine with metadata 200 and content 404, that a
restored object heals, that a second pass is skipped while the lock is
held, and that passes in a loop during 75 s of uploads lose nothing.
Intents are injected 300 s old, between the reconciler grace and the
API sweep grace, so only the reconciler removes them.

reconcile_track.py cost snapshots docker stats of the whole host, pins
S3, PostgreSQL and the reconciler to their own CPUs, fills the stand to
each N with seed_storage.py run in the API image (parallel PutObject and
COPY), and records three dry runs per N and mode with a median table.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:23:59 +03:00
ArtyomandClaude Opus 5.5 51fa19a8ee Run the reconciler as a one-off service of the stand
The reconciler service under the reconcile profile runs the API image
with the reconciler module as entrypoint and --dry-run as the default
command, so docker compose run --rm --no-deps reconciler takes only
flags. It gets the database URL and the limited API key of S3, which
already allows listing, reading and deleting objects, a read-only root,
1 CPU and 256 MiB, and a longer S3 read timeout for listing pages.
S3, PostgreSQL and the reconciler accept an optional cpuset from the
environment so measurements can pin them; the default is no pinning.

make reconcile ARGS=... runs a pass, make reconcile-demo and make
reconcile-cost drive the track experiments, and the limits policy now
covers the new service.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:17:54 +03:00
ArtyomandClaude Opus 5.5 91731fd317 Reconcile attachment metadata with the bucket in one merge pass
StorageReconciler walks the sorted bucket listing and keyset pages of
attachments ordered by object key side by side, so a pass is linear
and keeps one page of each side in memory. An object without a row is
deleted only when it is older than grace plus skew and a lookup by key
right before the delete still finds no row; a ready row whose object
is missing, has another size or, with the deep check, another digest
is put in quarantine after a HEAD recheck and is never deleted; a stale
intent is claimed with SKIP LOCKED and dropped object first; a
quarantined row whose object matches again is released. Deep checks
hash objects in a bounded task group, each action uses its own unit of
work, and failures of single actions are counted instead of aborting.
An advisory lock lets one pass run at a time. Content of a quarantined
attachment answers 404 without touching S3, its metadata is unchanged.

The storage port gains listing pages and HEAD, the repository keyset
pages, lookup by key, claim and quarantine marking; the platform logger
can write to another stream. The one-off entry point
python -m messenger.attachments.entrypoints.reconcile is a dry run
unless --apply is given, assembles the adapters with the API key of S3
and prints a JSON report with counts per class, samples and the cost
of the pass to stdout while the journal goes to stderr.

Tests run the reconciler against its own migrated database with every
class, apply and repeat, the row that appears after the listing, the
busy lock and the release, and check paged byte-order listing and HEAD
against RustFS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:12:10 +03:00
ArtyomandClaude Opus 5.5 62986245a2 Add the quarantine columns for the storage reconciler
Migration 0004 adds a nullable quarantine reason (missing, size or
digest mismatch) and the time it was set, both null or both present.
Nullable columns without a default only touch the catalogue, so the
running version keeps working while the migration is applied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:11:59 +03:00
ArtyomandClaude Opus 5.5 148868bdfb Propose the storage reconciler as an openspec change
Track 2 of grade 5: a one-off reconciler that merges the sorted bucket
listing with keyset pages of the attachments table, so a pass is linear
in time and holds one page per side in memory. It finds objects without
a row, ready rows without an object or with another size or digest, and
abandoned intents; by default it only reports, with --apply it deletes
old orphans after a row recheck, drops stale intents object first, and
quarantines broken ready rows instead of deleting them, so referenced
attachments survive and a restored object heals. A quarantined
attachment answers 404 on its content. An advisory lock keeps a single
pass at a time.

The design states the hypotheses on correctness and on the cost of a
pass, the reproducible method behind make reconcile-demo and make
reconcile-cost, and the acceptance criteria.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:05:42 +03:00
ArtyomandClaude Opus 5.5 d2744bc1ac Archive the upload resilience change
Grade 4 of lab 4 is closed. A failed ready commit after the object
write is compensated at once under a row lock, intents abandoned by a
crash or a lost write are swept with their objects after a 900 s grace
period by every replica with SKIP LOCKED, and nothing younger, nothing
ready and nothing referenced is touched. A client abort is a 400 and a
stalled body a 503 after 8 s, both leaving nothing behind. Three upload
slots per worker, capped socket buffers in the balancer and the API
and a fixed glibc mmap threshold keep 40 concurrent 10 MiB uploads at
330 of 512 MiB per replica with every upload succeeding, while without
the slots the same load hits the limit and loses 37 of 80. 36 user
errors through the balancer give no 5xx, and 23 checks prove the
bucket private, the API key limited, the console closed and the
bootstrap idempotent.

For the defence: run make upload-faults to show the negative suite,
the compensation, the killed replica with its leftover swept after the
grace period while the fresh one stays, and the abort and stall; show
the memory summary and make s3-check.

The lab4-upload-resilience spec is added, and the write order of
lab4-attachments and the console and API key requirements of
lab4-object-storage are updated in the main specs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:25:46 +03:00
ArtyomandClaude Opus 5.5 2b19277b1e Record level 4 evidence and fill in the report
Evidence comes from the running stand at a774a2f: upload-faults.txt
with the negative suite through the balancer (36 user errors, no 5xx
in the replica logs) and the compensation, crash, sweep, abort and
stall checks; upload-memory with per-run JSON and a summary for 64
and 3 slots, 1 and 10 MiB and 10, 20 and 40 concurrent uploads plus
the mmap threshold comparison; s3-security.txt with 23 checks; and
regression-level4.txt showing that smoke, persistence, failover of
each replica, the upload failover, S3 outage and abort scenarios, the
limits policy and all 15 stand checks still pass. The lint and own
test log now counts 335 tests. The upload-faults target stamps its
output with the date and commit.

REPORT.md gains rows 4.1 to 4.4 and a regression row, the compensation
and sweep mechanism with its grace period, where each upload limit and
buffer cap applies, the new variables, level 4 failure rows and the
memory experiment with hypothesis, method, results and limits. AGENTS
lists the new targets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:24:41 +03:00
ArtyomandClaude Opus 5.5 a774a2fbbd Measure upload memory and bound the buffers along the chain
upload_memory.py recreates the replicas and the balancer for every run
of 10, 20 and 40 concurrent uploads of 1 and 10 MiB with 64 slots
(effectively no limit) and with 3, drives them through lab4_seed.py
from the prepared tooling, and keeps per-run JSON plus a summary with
the cgroup memory.peak of each replica and of the balancer and the
oom_kill counters. The seed script now also records memory.current
before the load and reads X-Instance-Id regardless of case.

The first run showed that slots alone only move a waiting body out of
the worker: with 3 slots and 40 uploads of 10 MiB the kernel grew the
socket buffers of HAProxy to about 110 MB and the 128 MiB balancer was
killed. HAProxy now caps its client and server socket buffers at 64
KiB and the API replicas cap tcp_rmem and tcp_wmem at 256 KiB, which
keeps the balancer at 30 MiB. glibc raised its mmap threshold after
the first large free and kept freed 10 MiB bodies in the heap, so the
API fixes MALLOC_MMAP_THRESHOLD_ at 128 KiB.

With these settings 40 concurrent uploads of 10 MiB peak at 323 MiB of
512 per replica and all 80 succeed, while without the slot limit the
same load reaches 512 MiB, a worker is killed and only 31 of 80
succeed. UPLOAD_CONCURRENCY stays at 3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:14:09 +03:00
ArtyomandClaude Opus 5.5 98df351235 Check upload faults and user errors against the stand
test_attachment_faults.py runs through the balancer against the real
PostgreSQL and RustFS. The negative suite sends a body one byte over
the limit with Content-Length (answered before any body byte) and
chunked, a decompression bomb, truncated and disguised images, missing
and foreign content types, dangerous names, a stranger's reads and
references, an attachment of another chat and invalid lists, checks
each code and error body, and finds no 5xx in the replica logs and no
extra rows.

Faults come from triggers installed for the module and dropped after
it: an AFTER INSERT trigger remembers the id and key of each fault
intent, a BEFORE UPDATE trigger rejects the ready commit for
fault-fail names and holds it for 1.8 s for fault-crash names, while
the test finds the waiting backend's client address and kills that
replica with SIGKILL. The rejected commit leaves neither row nor
object right away and an audit line says compensation; a crash leaves
a pending row and an object, the one aged past the grace period is
swept with its object while the fresh one stays, and an aged ready
attachment in a message still downloads. A body cut after 256 KiB is
logged as 400 with nothing written, and a stalled body gets 503 after
8 s with nothing written.

make upload-faults stores the output in evidence/upload-faults.txt
without tracebacks of locals, and the CI acceptance job runs it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:48:46 +03:00
ArtyomandClaude Opus 5.5 28055a8551 Pass the upload limits to the API and check the console from the network
The API replicas get UPLOAD_CONCURRENCY, UPLOAD_GRACE_SECONDS,
UPLOAD_SWEEP_INTERVAL_SECONDS and UPLOAD_SWEEP_BATCH through the S3
anchor of compose.yaml with the same defaults as the settings, and
.env.example documents them. A value in the shell overrides the
default, which the memory experiment uses to recreate the replicas with
a different number of slots.

s3-check.sh now also asks the console port of the S3 service from a
separate container on the stand network and expects a refused
connection, so 23 checks cover ports, the console inside and outside
the container, the repeated bootstrap, the rights of the API key and
anonymous access.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:40:56 +03:00
ArtyomandClaude Opus 5.5 0e4eb1738c Compensate failed uploads and sweep abandoned intents
When the ready commit fails after the object was written, the upload
handler now locks the row, keeps it if it turned out ready, and
otherwise deletes the object first and the pending row second before
answering 503. A ready update that touches no row means the intent was
already swept, so it fails as unavailable and the object goes too;
a 201 without a row cannot happen.

Every worker runs an UploadSweeper that every 30 s removes pending
intents older than UPLOAD_GRACE_SECONDS (900 s, at least 60 s) one row
at a time under FOR UPDATE SKIP LOCKED over the partial pending index,
object first, and logs an audit event for each discard. Bodies are
buffered only inside one of UPLOAD_CONCURRENCY slots per worker and
released before the ready commit, and a client that disconnects in the
middle of a body becomes a 400 instead of an unhandled 500.

The tests run against a throwaway PostgreSQL: a trigger rejects the
ready commit and the compensation leaves neither row nor object, a
row removed during the write gives 503, a failed write stays until the
grace ends, fresh intents and an aged ready attachment in a message
survive a sweep, two concurrent sweeps share five intents exactly, the
background loop runs by itself, a second upload waits for the slot
without reading its body, and aborted and stalled bodies leave nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:39:51 +03:00
ArtyomandClaude Opus 5.5 a41580aceb Propose upload resilience as an openspec change
Grade 4 of lab 4 asks that a failure between the S3 write and the
PostgreSQL commit leaves no eternal orphans, that concurrent uploads
cannot take unbounded memory, that user errors never become a 500 and
that the API key and the private bucket are proven limited outside the
API. The change compensates a failed ready commit right away under a
row lock, sweeps abandoned intents after a grace period in every
replica with SKIP LOCKED, maps a client abort to a client error, bounds
buffered bodies per worker with upload slots, and checks all of it
against the stand: a negative suite through the balancer, faults
injected with a PostgreSQL trigger and a killed replica, and a memory
experiment with 10, 20 and 40 concurrent uploads.

The new lab4-upload-resilience capability holds these requirements;
the write order of lab4-attachments and the console and key checks of
lab4-object-storage are updated by deltas.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:34:39 +03:00
ArtyomandClaude Opus 5.5 e4253eb076 Archive the image attachments change
Grade 3 of lab 4 is closed. RustFS with a private bucket runs in the
stand, metadata lives in PostgreSQL and the bytes only in the bucket,
uploads answer 201 ready after the object write and the commit, and
the original is served byte for byte to chat members only. The smoke
test passes from a fresh clone; limits of 10 MiB and 40 MP, content
type by content and dangerous names are covered by 325 own tests; an
upload survives the stop of the replica that took it, both replicas
fail over under writes in under 3.1 s, a full restart keeps metadata
and bytes, and with S3 stopped uploads and downloads answer 503 within
2.6 s while texts, metadata and readiness keep working. 22 checks show
the bucket is private outside the API and the API key is limited.

For the defence: upload the fixture, send it in a message, download it
as the second member and compare SHA-256, get 404 as a stranger and
403 for an anonymous GET in S3, then run persistence, failover and the
upload failover scenario.

The lab4-attachments and lab4-object-storage specs are added, and
message sending, history, idempotent replay, the restart hook and stand
persistence are updated in the main specs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 22:20:38 +03:00
ArtyomandClaude Opus 5.5 db901d6401 Record level 3 evidence and fill in the report
Evidence comes from the stand started from an empty state and from a
fresh clone: the public smoke, the contract checks, lint and 325 own
tests, the storage layout with an out-of-API comparison of a stored
object against its metadata, failover of each replica under writes, the
upload failover scenario, persistence with StartedAt of every service
including S3, the S3 outage scenario and 22 S3 privacy and bootstrap
checks. The storage evidence script now covers the attachment tables
and the S3 secrets.

REPORT.md follows the lab 4 template: launch from a clean clone,
architecture with the source of truth for each fact, the write order
between PostgreSQL and S3 and what each failure leaves behind, where
each upload limit applies, roles and services, the .env.example
mapping, grade 3 rows and the grade 3 failure table. The design notes
the botocore Expect header that hung uploads behind the port proxy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 22:20:10 +03:00
ArtyomandClaude Opus 5.5 24f9d6195b Add the attachment failure scenarios and S3 checks
lab4_scenarios.py drives the public contract through tests/client.py:
failover-upload stops the replica that answered the upload with the
stop-one hook and downloads through another one comparing SHA-256, and
s3-outage stops S3 and records status and time of upload, download,
metadata, a text message and readiness before bringing it back. Both
print JSON with an ok field. s3-check.sh checks published ports, the
console, two more bootstrap runs without state changes, the allowed
and denied operations of the API key and anonymous access including a
plain HTTP GET from a separate container. evidence-attachments.sh
shows where the bytes live and compares a stored object with its
metadata outside the API.

Makefile gains scenario, s3-check and attachments-evidence, ruff covers
the new script, and the CI acceptance job runs both scenarios and the
S3 check after the smoke test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:50:03 +03:00
ArtyomandClaude Opus 5.5 5aa766acff Run RustFS with a private bucket in the stand
MinIO community images can no longer be pulled on this arm64 host, so
the stand runs RustFS 1.0.4 pinned by digest as non-root with a read
only root, no published port, the console off and data on the s3data
volume. A one-off s3-init with rc v0.1.36 creates the bucket, keeps it
private for anonymous clients, creates the API user with a policy on
the objects and listing of that one bucket, imports a lifecycle rule
for incomplete multipart uploads and checks itself; the API replicas
start only after it succeeded. Root keys reach only s3, s3-init and the
operator s3-cli; the API gets its own key and the S3 settings through
an anchor merged into its environment.

secrets.sh creates hex S3 keys, the restart hook restarts S3 next to
PostgreSQL and Valkey and waits until it is healthy, the limits policy
covers the S3 services, and the stack evidence lists the new packages.
The object storage adapter is tested against a throwaway RustFS of the
same image: a 10 MiB round trip, a missing object, an idempotent delete
and an unreachable endpoint failing within the request budget.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:50:03 +03:00
ArtyomandClaude Opus 5.5 f9760fc098 Link messages to attachments of the same chat
A message now carries attachment_ids in the request, in every response,
in the idempotent replay and in history, in the order they were sent.
The chat aggregate checks membership first, then refuses whitespace
text, more than ten or repeated attachments, and an empty text without
attachments, so an outsider still gets 404 before any 400.

The send statement inserts the links next to the message and the
idempotency key, and the foreign key from message_attachments to
attachments of the same chat turns an unknown or foreign attachment
into 404 and rolls the whole send back, without the messaging zone
reading the attachments table. Replay and history read the list with a
subquery on the primary key. The request fingerprint covers the list,
so a missing attachment_ids and [] are the same body while a different
order is a conflict.

The zone dependency test now covers three zones, and the SQL ownership
test checks that every zone touches only its own tables.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:45:19 +03:00