79 Commits
Author SHA1 Message Date
Artyom c67dd11029 Merge completed lab 4 for Gitea submission
Preserve the exact lab 4 tree and original history from 4ed889c1e4.
Connect it to the instructor master for a cross-fork pull request.
2026-11-08 00:57:05 +03:00
Artyom 4ed889c1e4 Validate final PNG workloads and retain applicable evidence 2026-11-06 22:21:55 +03:00
Artyom 5e49502ae7 Finalize lab4 report and evidence provenance after review 2026-11-06 21:53:18 +03:00
Artyom 7a0ebac2aa Archive the PNG header consistency change 2026-11-06 21:34:36 +03:00
Artyom 7c8a1a6489 Reject ambiguous PNG headers before pixel validation 2026-11-06 21:33:09 +03:00
Artyom bcbd4ebc02 Propose PNG header consistency as an OpenSpec change 2026-11-06 21:29:25 +03:00
Artyom 7342d5d97a Archive the PNG validation fix and record offline evidence 2026-11-06 13:00:51 +03:00
Artyom 9387c3d82a Reject corrupt PNG pixels with bounded stream validation 2026-11-06 12:57:32 +03:00
Artyom f0521fa1e0 Propose bounded PNG validation as an openspec change 2026-11-06 12:49:16 +03:00
Artyom 8bc2da2c09 Verify and archive final lab3 fixes for lab4 2026-11-06 12:33:49 +03:00
Artyom 75b93bbba5 Forward final lab3 session security and validation fixes 2026-11-06 12:07:00 +03:00
Artyom f0fc084fbb Propose forwarding final lab3 review fixes 2026-11-06 12:01:57 +03:00
Artyom d7478726a8 Complete SQLAlchemy reconciler evidence and close lab4 review findings 2026-11-06 03:37:31 +03:00
Artyom 504f77cb7c Validate reconciler evidence and expose isolated demo output 2026-11-06 03:32:06 +03:00
Artyom 3402fd69d4 Expose bounded seed progress for long reconciler experiments 2026-11-06 02:32:19 +03:00
Artyom 827b783be3 Reject disturbed reconciler measurements and preserve every series 2026-11-06 02:29:53 +03:00
Artyom 751aaed4b1 Preregister SQLAlchemy reconciler validation and full cost matrix 2026-11-06 02:28:14 +03:00
Artyom 57caeb6b5a Verify grade four faults memory cleanup and clean clone on SQLAlchemy 2026-11-06 02:27:08 +03:00
Artyom cf41f7c64d Separate historical results and audit every lab4 review finding 2026-11-06 02:07:59 +03:00
Artyom 6c706a727e Preserve evidence runs and volumes while propagating failures 2026-11-06 02:02:04 +03:00
Artyom c7e7a86627 Propose isolated grade four verification and review closure 2026-11-06 02:01:27 +03:00
Artyom 02f385866f Verify and archive lab4 grade three on SQLAlchemy 2026-11-06 02:00:43 +03:00
Artyom 763b025557 Fail attachment scenarios on invalid outcomes 2026-11-06 01:59:03 +03:00
Artyom 1aec062aba Capture isolated evidence with full provenance and exit codes 2026-11-06 01:54:30 +03:00
Artyom 5596ca935c Propose fresh lab4 grade three evidence on SQLAlchemy 2026-11-06 01:54:07 +03:00
Artyom 41700c27e5 Archive the lab3 foundation forward port 2026-11-06 01:48:09 +03:00
Artyom 8e4532f6f5 Forward-port lab3 SQLAlchemy and Alembic foundation
Carry source f591fa64e5 forward from 72a1e57. Preserve lab4 attachment transactions, reconciliation locks, quarantine, and existing schema data through verified legacy adoption. Keep report and evidence work in progress untouched.
2026-11-06 01:47:30 +03:00
Artyom dd4021b678 Propose the lab3 SQLAlchemy foundation forward port 2026-11-06 01:32:54 +03:00
ArtyomandClaude Opus 5.5 568ff6643e Archive the storage reconciler hardening change
A pass releases the quarantine only after the object hashes to the row
again, reads exactly ceil(N/page) SQL pages, publishes
audit.attachment_discarded reason=reconcile for the intents it drops,
is assembled by dishka with the same storage provider as the API and
ends any failure as reconcile.failed with exit code 1. The demo passes
13 of 13 checks, and the cost matrix up to 100k confirms the request
counts and the constant memory of H2 and refutes its linear time at
100k (19.6 instead of 10 +- 30 %, the RustFS listing page grows from
91 to 197 ms). student-test (348), test, persistence and failover pass
on the stand. The requirements are synced into lab4-storage-reconciler.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:54:40 +03:00
ArtyomandClaude Opus 5.5 d2e684f0cf Record the track 2 demo and the cost of a pass up to 100k
The demo on a clean stand (61dcb6b) passes 13 of 13 checks: the dry-run
finds exactly the injected ids in every class, apply writes one
audit.attachment_discarded reason=reconcile per stale intent, the
original bytes leave the quarantine while random bytes of the same
length keep it, docker stop ends a pass hanging on a paused S3 in
0.11 s with exit code 143, and 74 passes during 3454 uploads lose no
ready attachment; 46 times the HEAD recheck kept an in-flight upload
out of the quarantine.

The cost matrix now reaches 100k: a shallow pass takes 0.108 / 1.032 /
3.761 / 20.211 s for 1k / 10k / 30k / 100k with ceil(N/1000) listings
and SQL pages and the same 87-88 MiB of RSS. t(100k)/t(10k) is 19.6,
outside the preregistered 10 +- 30 %, because a RustFS listing page
costs 91 ms at 1k and 197 ms at 100k objects in the prefix; the
reconciler's own CPU and SQL stay linear. The previous 30k series moves
to cost-b191623/, and the old after-checks give way to the full series
that follows the track.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:54:29 +03:00
ArtyomandClaude Opus 5.5 6ffb90d1d1 Script the overload leftovers check with raw output
overload-leftovers.txt was assembled by hand with summarised reconciler
reports. make overload-leftovers now runs the dry-run, the status
counts, the wait for the sweep after the grace period, the audit and
swept counts of the replicas and the dry-run afterwards, and writes
their raw output.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:17:15 +03:00
ArtyomandClaude Opus 5.5 61dcb6bf13 Keep the audit journal of each reconciler pass in its report
The driver now stores the audit events a pass writes to stderr next to
its JSON report, and the apply check requires one
audit.attachment_discarded reason=reconcile for every injected stale
intent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:09:15 +03:00
ArtyomandClaude Opus 5.5 21937f920f Wait for S3 to turn healthy after the paused pass
docker compose up --wait failed at once after unpausing S3: the probes
that ran while it was paused had already marked it unhealthy. The demo
now polls the health status of the container until it is healthy again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:04:09 +03:00
ArtyomandClaude Opus 5.5 f1ee385a1b Tie the track tooling to keys, sets and a clean series
reconcile-inject.sh wrote an orphan under a key whose uuid differed
from the id it printed, so the demo could only compare counts; it now
uses one uuid for both, and the dry-run check compares the found and
the injected sets per class. The heal step returns the original bytes
under one quarantined key and random bytes of the same length under
another, and checks that only the first leaves the quarantine. The
load phase sums rechecked over its passes, and a new step stops a pass
hanging on a paused S3 with docker stop and checks exit code 143 within
seconds and a free lock afterwards.

reconcile_track.py cost clears the run files of the previous series and
measures 1k, 10k, 30k and 100k with --deep everywhere. The scenarios
record date, commit and docker inspect of the replicas or of S3 at the
moment they are down. make load-evidence and make clean-clone-evidence
rerun the message load and the clean clone block with raw output, and
make upload-memory passes ARGS to reproduce the run directories.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:01:39 +03:00
ArtyomandClaude Opus 5.5 3de1dcdc84 Release the quarantine only after the object hashes to the row again
A pass without --deep released the quarantine of a missing or resized
attachment as soon as an object of the same size appeared under its
key, so other bytes of the same length went back to /content. The pass
now streams and hashes the object of every quarantined row whose size
matches again, deep or not, and releases it only when the SHA-256
matches; other bytes keep the row quarantined. Consistent data still
costs no GET.

The keyset reads page + 1 rows, so a pass takes exactly ceil(N/page)
SQL pages instead of one more for the empty tail. The reconciler is
assembled by dishka from PlatformProvider and the attachment storage
provider that the API uses too, and it publishes the events of the
intents it drops, so audit.attachment_discarded reason=reconcile
reaches the journal. Any failure of a pass, not only Unavailable, ends
as reconcile.failed with the error type and exit code 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 09:01:39 +03:00
ArtyomandClaude Opus 5.5 b6c04f2296 Propose hardening the storage reconciler as an openspec change
The review of track 2 found that a pass without --deep released the
quarantine of a missing or resized attachment as soon as an object of
the same size reappeared, without checking its SHA-256; that the keyset
read an empty last page, one SQL request more than the spec promised;
that the discarded intents of a pass never reached the audit journal;
that the entry point duplicated the S3 client wiring and printed a
traceback for any failure other than Unavailable; and that the cost
matrix stopped at 30k while the preregistered H2 was about 100k.

The change hashes the object of a quarantined row before releasing it,
reads page + 1 rows, assembles the reconciler with dishka and publishes
its events, turns any pass failure into reconcile.failed, links the
injected orphans to the found ones by key and measures up to 100k.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:56:15 +03:00
ArtyomandClaude Opus 5.5 adca7c6edc Archive the attachment boundaries change
Upload checks the token before the declared size, a slow but steady
transfer outlives the request timeout while a stalled one is still cut,
the API and the reconciler hold separate minimal S3 keys, and the
one-off services run as non-root with an init process. On the stand a
3 MiB upload at 256 KiB/s took 12.1 s at the replica and got 201, a
3 MiB download at 192 KiB/s took 12.8 s and matched the SHA-256, the
API key is denied a bucket listing, and docker stop ends a hanging
reconciler pass in 0.13 s with exit code 143. student-test (345),
test-faults (19), persistence, failover and the lab 4 scenarios pass.
The requirements are synced into lab4-attachments, lab4-upload-resilience
and lab4-object-storage.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:53:53 +03:00
ArtyomandClaude Opus 5.5 2891b37e75 Split the S3 keys of the API and the reconciler, drop root from one-offs
The API key now has only GetObject, PutObject and DeleteObject on the
attachment objects: the server never lists the bucket, never runs
multipart uploads and has its region set. The reconciler gets its own
user messenger-reconciler with ListBucket, GetObject and DeleteObject,
created by s3-init from a secret of its own, so the API replicas can no
longer list the bucket. The cost seeding writes objects and therefore
runs in a one-off API container under the API key. s3-check proves the
API key cannot list and checks the reconciler key; its snapshot drops
dated fields with a JSON filter instead of grep, which left a dangling
comma, and its first line carries the date and the commit.

s3-init and s3-cli run as uid 65534, and migrate and reconciler get an
init process, so SIGTERM reaches python and docker stop ends a pass in
a fraction of a second instead of waiting for SIGKILL. limits.py and CI
check the users of the one-off services. Stack tests move a 3 MiB file
up and down through the balancer at 256 and 192 KiB/s, longer than the
request timeout, and check 401 before the declared size.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:53:39 +03:00
ArtyomandClaude Opus 5.5 54099e1da3 Check the token before the declared size and pace the request deadline
The pipeline now runs the guard first, so an anonymous upload with a too
large Content-Length gets 401 with WWW-Authenticate, as the contract
asks; the guard does not read the body, so 413 still comes before the
first body byte for an authenticated client.

The single request deadline cut slow but steady uploads and downloads:
a client slower than about 1.3 MB/s could not move 10 MiB in 8 s. Each
body chunk the application receives or sends now moves the deadline to
min(now, start + moved / MIN_TRANSFER_RATE_BYTES) + REQUEST_TIMEOUT,
128 KiB/s by default, like mod_reqtimeout with a ceiling: a steady
transfer lives as long as it moves, a stalled or crawling body is cut
after the request timeout as before, and the work after the last body
byte still gets at most the request timeout. The pipeline's own error
responses bypass the pacing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:53:39 +03:00
ArtyomandClaude Opus 5.5 96e934ed8f Propose tightening the lab 4 attachment boundaries as an openspec change
The review of lab 4 found four gaps at the attachment boundaries: an
anonymous upload with a too large Content-Length got 413 before 401,
the single request deadline cut slow but steady uploads and downloads,
the API S3 key could list the bucket and run multipart actions, and the
one-off services ran as root or as PID 1 without an init process.

The change checks the token first, paces the request deadline by the
bytes moved, splits the S3 keys of the API and of the reconciler and
runs the one-off services as non-root with an init process.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 08:41:27 +03:00
ArtyomandClaude Opus 5.5 c8cb263c77 Retake the lab 4 evidence on the final code and finish the report
The level 3 and 4 evidence was taken on 24f9d61 and a774a2f, before the
lab 3 fixes were carried over and before the reconciler. Take all of it
again in one series on 81dd816, whose application and stand code is the
final one: a fresh clone runs the launch block of the report from secrets
to reset-data, then one stand from clean volumes gives the smoke, stack,
limits, lb-check, failover of each replica and of the upload replica,
persistence, S3 outage, S3 privacy, lost-answer retry, upload faults, all
stack checks, storage layout and three runs of the upload memory matrix.

The three memory runs keep the peak with three slots at 297-345 MiB
against 512 MiB at the limit without them, but at 40 uploads of 10 MiB
the wait for a slot nears the 8 s request timeout and one run of three
gave 7 of 80 a 503. Those timeouts left 67 invisible intents; the record
shows the reconciler seeing them as fresh and the sweep removing all of
them after the grace period.

Message write load is taken with tests/load.py from a container on the
edge network: 2 106 RPS, no errors, replicas and PostgreSQL at their CPU
limits. The first try from the macOS host is kept as a discarded attempt,
it ran out of ephemeral ports while the replicas answered every request.

The report gets the defense answers, the load section, the evidence
commits, the upload limit order as the pipeline applies it, medians in
the track 2 cost table and the proposals over the word limit; the evidence
index lists every file with its command and commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 07:55:34 +03:00
ArtyomandClaude Opus 5.5 81dd816df2 Archive the storage reconciler change
Track 2 of lab 4 is closed. One reconciler pass merges the sorted bucket
listing with keyset pages of attachments by object key, finds objects
without a row, ready rows without an object or with other bytes and
abandoned intents, deletes only old objects without a row after a
second look at the row and old intents under SKIP LOCKED, and puts
damaged ready rows into a reversible quarantine instead of deleting
them. On the stand the dry-run found exactly the 22 injected
discrepancies of seven classes, apply fixed the 18 fixable ones, the
repeat did nothing, and 75 passes during 3418 uploads lost no ready
attachment. A shallow pass costs 0.105 / 1.000 / 3.729 s for 1k / 10k /
30k objects at a flat 83 MiB, 95 % of it in the RustFS listing; --deep
is 4.3-5 times dearer per object.

For the defence: make reconcile-demo shows injection, dry-run, apply,
the repeat, quarantine through the API, healing, the held lock, a
stopped S3 and the run under uploads in about four minutes; the cost
matrix is evidence/reconcile/cost/summary.txt (make reconcile-cost
repeats it on a fresh stand in about seven).

The lab4-storage-reconciler spec is added and the quarantine rule of
lab4-attachments is updated in the main specs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 07:19:42 +03:00
ArtyomandClaude Opus 5.5 58cae16ef8 Report track 2 and check levels 3 and 4 after it
REPORT.md gets the track 2 section: what the reconciler does and why it
is safe, the hypotheses written before the first run, the method of the
demo and the cost matrix with its screening, both result tables, the
verdicts and the limits. H1 holds. H2 holds for the pass cost: time is
linear (t(30k)/t(10k) = 3.73, 3.05 in the series of 29.10), the listing
is 95 % of it, list requests are ceil(N/1000) and SQL one more for the
empty last keyset page, RSS stays at 82.5-83.8 MiB; --deep is 4.3-5
times dearer per object, not ten. The criteria table, the failure table,
the limits, AGENTS.md and the openspec context name the track rows, and
evidence/reconcile/README.md describes every file.

On a stand from a clean state make test, test-persistence,
test-failover, restore, reconcile, lint, check, validate and
student-test (342 passed) all pass; the dry-run after these scenarios
finds 3 objects, 3 rows and no discrepancy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 07:19:16 +03:00
ArtyomandClaude Opus 5.5 8cb5a664c6 Keep the start allowance and retries in the cost summary
The edit that added them to the plan of the series missed after the
formatter split the line, so summary.json of df345b3 lists only the two
thresholds; that series ran with the defaults of b191623 (1.5 and 0.5
core-s, three retries). The summary of the next series records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:55:50 +03:00
ArtyomandClaude Opus 5.5 df345b32e1 Record the cost of a reconciler pass up to 30k objects
make reconcile-cost on a fresh stand at b191623: RustFS 2 CPU / 3 GiB on
cpus 11-12, PostgreSQL on 13-14, the reconciler on 15, three dry-run
passes per point; no run was disturbed, foreign CPU stayed at most 0.2
cores in the VM and 0.01 on the pinned cores. Medians:

  objects  mode     t, s    lists  sql  gets  rss, MiB
     1000  shallow  0.105       1    2     0      82.5
    10000  shallow  1.000      10   11     0      83.6
    30000  shallow  3.729      30   31     0      83.8
    10000  page100  2.058     100  101     0      81.0
     1000  deep     0.493       1    2  1000      83.0
    10000  deep     5.041      10   11 10000      84.1
    30000  deep    16.150      30   31 30000      84.1

The listing takes 95 % of a shallow pass. The first full series of
29.10 (5ca2cdc, not screened, another lab under k6 at the time) is kept
in cost-2026-10-29 for comparison between days.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:55:45 +03:00
ArtyomandClaude Opus 5.5 b1916236f4 Record the track 2 demo on the stand
make reconcile-demo on a fresh stand at 5c3c238, 12 of 12 checks true.
The dry-run found exactly what was injected on both sides (3 stale
orphans, 3 missing objects, 3 shorter and 3 same-size rewritten objects,
6 stale intents with and without an object, 2 fresh orphans and 2 fresh
intents) and changed nothing; apply fixed the same 18 and the second
apply did nothing. The independent check outside the API found no
violation and no lost ready attachment; the 9 damaged rows sit in
quarantine with their message links, the metadata answers 200 and the
content 404, and a restored object left quarantine. A pass during a held
lock was skipped, a pass with s3 stopped exited with 1 in 1.6 s and
changed nothing. 75 apply passes during 75 s of uploads through the
balancer removed only the 10 injected leftovers, saw 247 objects of
uploads in flight as fresh orphans and kept them: all 3418 uploads are
ready with matching SHA-256.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:48:07 +03:00
ArtyomandClaude Opus 5.5 5c3c238d17 Screen the cost runs for foreign load and check a pass without S3
The cost matrix shares the Docker VM with the stands and builds of other
labs. Each run now reads the VM-wide /proc/stat and the cpu.stat of the
s3 and postgres cgroups before and after the pass, and the reconciler
reports its own CPU time. Foreign CPU over the window is the busy time
minus ours minus the cost of starting a container (1.5 core-s in the VM
and 0.5 on the pinned cores, measured on the idle stand). A run with
more than 2 foreign cores in the VM or 0.25 on the pinned cores goes to
cost/disturbed and is repeated after 15 s of quiet, up to three times;
docker stats of the whole host are saved before and after the series.

The demo also runs a pass with s3 stopped: it must exit with code 1 in
bounded time, log reconcile.failed and leave PostgreSQL and the bucket
as they were.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:35:59 +03:00
ArtyomandClaude Opus 5.5 3b3cb5bd7f End a failed reconciler pass with the dependency error
The merge loop runs inside the TaskGroup that holds the --deep checks,
so a listing or page failure in the middle of a pass left the group as
an ExceptionGroup. The entry point catches Unavailable only: the pass
still exited with code 1, but through an uncaught traceback instead of
the reconcile.failed record. The pass now unwraps the group and raises
the Unavailable with its original cause; the API test fails the listing
on the second page of a deep apply pass and expects Unavailable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:31:46 +03:00
ArtyomandClaude Opus 5.5 8cd30488d2 Archive the carry over of the lab 3 fixes change
Sync the delta specs into the main specs: zones are linked by the
schema and the ports main.py assembles, the event bus and the network
dependencies name S3 and Valkey, an outsider does not wait for chat
locks, an empty cursor is rejected, a request cancelled at shutdown gets
503 in the contract format, the body limit counts received bytes per
route, logout withdraws the revocation mark on a database failure,
detection runs three times, failover-each records the stopped replica,
lost answers are retried on the stand, posting checks its input in a
fixed order and make up waits for every replica in the balancer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:25:44 +03:00
ArtyomandClaude Opus 5.5 e927afdb09 Carry over the lab 3 fixes made after 61c0c28
Bring the system changes of lab 3 from 61c0c28..72a1e57 onto the lab 4
code, resolving each conflict by meaning and keeping everything lab 4
added:

- a message is posted with one SQL statement under the chat row lock,
  and the attachment links go into the same statement; a foreign key
  violation on the attachment link still answers 404 and leaves nothing
  behind. Text and the attachment list are checked before the write, an
  outsider gets 404 for both, as before the carry over;
- the request pipeline counts the body bytes actually received against
  the limit of the route, so a chunked body over 1 MiB gets 413 and the
  upload route keeps its MAX_UPLOAD_BYTES limit;
- a request cancelled at shutdown gets 503 unavailable in the contract
  format, the worker waits a second less than the rest of the shutdown
  budget, SHUTDOWN_TIMEOUT_SECONDS becomes 6 s and the stop hooks 7 s;
- readiness sits behind a Readiness port with the draining flag in
  shared infrastructure, and the drain window is logged through Logger;
- logout revokes through SessionRevocations and withdraws the mark when
  PostgreSQL rejects the delete; an empty cursor is 400;
- EventBus.project is gone, the sweep test subscribes and waits on a
  queue; domain tests check attachment_list instead of Chat.post;
- make up waits for every replica in the balancer, BASE_URL follows the
  lb port, the image is tagged by the compose project, retry-check and
  faults-evidence are added, detection runs three times, failover-each
  and rolling-update record the stand usage, CI checks numeric uids.
  down also takes the tools and reconcile profiles of lab 4.

Track 1 tooling of lab 3 (scaling.py, its samplers, tools/ and the CPU
pinning of api, lb, valkey and k6) stays out; stand-usage.sh queries the
database size itself instead of the track 1 sampler.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:25:23 +03:00
ArtyomandClaude Opus 5.5 7ac1a4a11b Propose carrying over the lab 3 fixes as an openspec change
Lab 4 started from the lab 3 commit 61c0c28, and lab 3 changed up to
72a1e57 after its track and two reviews. The change lists what moves
over and how it meets the lab 4 code: the one-statement message write
takes the attachment links along, the body limit counts received bytes
per route, the shutdown, readiness and logout fixes come as they are,
and the stand tooling keeps the lab 4 profiles and CPU pinning. Delta
specs bring the requirements lab 3 changed, adapted to three zones and
the object store; track 1 tooling stays out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:15:48 +03:00
ArtyomandClaude Opus 5.5 5ca2cdc9af Measure the pass cost up to 30k objects
Filling RustFS 1.0.4 to 100k objects under one prefix did not finish in
two attempts: a write gets slower as the prefix grows (about 8 ms in an
empty prefix against 32 ms at 28k objects, 13 writes per second with 16
parallel writers), with enough memory and no background scanner load.
The cost matrix now stops at 30k objects with four parallel writers,
which still spans a factor of 30 for the linearity check; the design
and the spec of the change say so, and both aborted attempts with their
1k and 10k points and the diagnosis are kept in
evidence/reconcile/cost-aborted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 04:40:24 +03:00
ArtyomandClaude Opus 5.5 7dca40c55e Give the balancer time to see the replicas before it counts as unhealthy
The balancer reports itself unhealthy while no replica is up, and its
start period of 10 s was shorter than the time the replicas need on a
busy host: they wait for migrate and s3-init and have a 30 s start
period of their own. docker compose up --wait then failed on a cold
start although the stand came up seconds later. A 60 s start period
with 1 s probes keeps the balancer starting, not unhealthy, until the
first replica is in rotation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:52:27 +03:00
ArtyomandClaude Opus 5.5 de9dc61ebc Give RustFS room for the 100k point of the cost matrix
The first cost run stalled while filling the bucket to 100k objects:
RustFS keeps object metadata in memory, at about 45k objects it hit
its 1 GiB limit, the cgroup kept reclaiming and PutObject slowed from
about 240 to 11 objects per second. The S3 CPU limit is now taken from
S3_CPUS (default 1, as before), and the cost run gives RustFS 2 pinned
CPUs and 3 GiB for every point, so all N are measured with the same
limits; seeding uses 16 parallel writes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:50:30 +03:00
ArtyomandClaude Opus 5.5 a335f654ed Add the track 2 fault injection, invariant check and experiment driver
reconcile-inject.sh and reconcile_check.py come from the prepared kit:
the first breaks either side on purpose (an object without a row, a
removed object, another size, the same size with other bytes, an old
intent with and without its object), the second is an oracle outside
the API that classifies every row and object from psql and rc, hashes
objects with --deep and reports ready attachments lost against a saved
set.

reconcile_track.py demo uploads through the balancer, links some
attachments to messages, injects each class on both sides with old and
fresh variants, and checks that a deep dry-run finds exactly what was
injected and changes nothing, that apply fixes exactly that and a
second apply does nothing, that broken rows and their message links
survive in quarantine with metadata 200 and content 404, that a
restored object heals, that a second pass is skipped while the lock is
held, and that passes in a loop during 75 s of uploads lose nothing.
Intents are injected 300 s old, between the reconciler grace and the
API sweep grace, so only the reconciler removes them.

reconcile_track.py cost snapshots docker stats of the whole host, pins
S3, PostgreSQL and the reconciler to their own CPUs, fills the stand to
each N with seed_storage.py run in the API image (parallel PutObject and
COPY), and records three dry runs per N and mode with a median table.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:23:59 +03:00
ArtyomandClaude Opus 5.5 51fa19a8ee Run the reconciler as a one-off service of the stand
The reconciler service under the reconcile profile runs the API image
with the reconciler module as entrypoint and --dry-run as the default
command, so docker compose run --rm --no-deps reconciler takes only
flags. It gets the database URL and the limited API key of S3, which
already allows listing, reading and deleting objects, a read-only root,
1 CPU and 256 MiB, and a longer S3 read timeout for listing pages.
S3, PostgreSQL and the reconciler accept an optional cpuset from the
environment so measurements can pin them; the default is no pinning.

make reconcile ARGS=... runs a pass, make reconcile-demo and make
reconcile-cost drive the track experiments, and the limits policy now
covers the new service.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:17:54 +03:00
ArtyomandClaude Opus 5.5 91731fd317 Reconcile attachment metadata with the bucket in one merge pass
StorageReconciler walks the sorted bucket listing and keyset pages of
attachments ordered by object key side by side, so a pass is linear
and keeps one page of each side in memory. An object without a row is
deleted only when it is older than grace plus skew and a lookup by key
right before the delete still finds no row; a ready row whose object
is missing, has another size or, with the deep check, another digest
is put in quarantine after a HEAD recheck and is never deleted; a stale
intent is claimed with SKIP LOCKED and dropped object first; a
quarantined row whose object matches again is released. Deep checks
hash objects in a bounded task group, each action uses its own unit of
work, and failures of single actions are counted instead of aborting.
An advisory lock lets one pass run at a time. Content of a quarantined
attachment answers 404 without touching S3, its metadata is unchanged.

The storage port gains listing pages and HEAD, the repository keyset
pages, lookup by key, claim and quarantine marking; the platform logger
can write to another stream. The one-off entry point
python -m messenger.attachments.entrypoints.reconcile is a dry run
unless --apply is given, assembles the adapters with the API key of S3
and prints a JSON report with counts per class, samples and the cost
of the pass to stdout while the journal goes to stderr.

Tests run the reconciler against its own migrated database with every
class, apply and repeat, the row that appears after the listing, the
busy lock and the release, and check paged byte-order listing and HEAD
against RustFS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:12:10 +03:00
ArtyomandClaude Opus 5.5 62986245a2 Add the quarantine columns for the storage reconciler
Migration 0004 adds a nullable quarantine reason (missing, size or
digest mismatch) and the time it was set, both null or both present.
Nullable columns without a default only touch the catalogue, so the
running version keeps working while the migration is applied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:11:59 +03:00
ArtyomandClaude Opus 5.5 148868bdfb Propose the storage reconciler as an openspec change
Track 2 of grade 5: a one-off reconciler that merges the sorted bucket
listing with keyset pages of the attachments table, so a pass is linear
in time and holds one page per side in memory. It finds objects without
a row, ready rows without an object or with another size or digest, and
abandoned intents; by default it only reports, with --apply it deletes
old orphans after a row recheck, drops stale intents object first, and
quarantines broken ready rows instead of deleting them, so referenced
attachments survive and a restored object heals. A quarantined
attachment answers 404 on its content. An advisory lock keeps a single
pass at a time.

The design states the hypotheses on correctness and on the cost of a
pass, the reproducible method behind make reconcile-demo and make
reconcile-cost, and the acceptance criteria.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:05:42 +03:00
ArtyomandClaude Opus 5.5 d2744bc1ac Archive the upload resilience change
Grade 4 of lab 4 is closed. A failed ready commit after the object
write is compensated at once under a row lock, intents abandoned by a
crash or a lost write are swept with their objects after a 900 s grace
period by every replica with SKIP LOCKED, and nothing younger, nothing
ready and nothing referenced is touched. A client abort is a 400 and a
stalled body a 503 after 8 s, both leaving nothing behind. Three upload
slots per worker, capped socket buffers in the balancer and the API
and a fixed glibc mmap threshold keep 40 concurrent 10 MiB uploads at
330 of 512 MiB per replica with every upload succeeding, while without
the slots the same load hits the limit and loses 37 of 80. 36 user
errors through the balancer give no 5xx, and 23 checks prove the
bucket private, the API key limited, the console closed and the
bootstrap idempotent.

For the defence: run make upload-faults to show the negative suite,
the compensation, the killed replica with its leftover swept after the
grace period while the fresh one stays, and the abort and stall; show
the memory summary and make s3-check.

The lab4-upload-resilience spec is added, and the write order of
lab4-attachments and the console and API key requirements of
lab4-object-storage are updated in the main specs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:25:46 +03:00
ArtyomandClaude Opus 5.5 2b19277b1e Record level 4 evidence and fill in the report
Evidence comes from the running stand at a774a2f: upload-faults.txt
with the negative suite through the balancer (36 user errors, no 5xx
in the replica logs) and the compensation, crash, sweep, abort and
stall checks; upload-memory with per-run JSON and a summary for 64
and 3 slots, 1 and 10 MiB and 10, 20 and 40 concurrent uploads plus
the mmap threshold comparison; s3-security.txt with 23 checks; and
regression-level4.txt showing that smoke, persistence, failover of
each replica, the upload failover, S3 outage and abort scenarios, the
limits policy and all 15 stand checks still pass. The lint and own
test log now counts 335 tests. The upload-faults target stamps its
output with the date and commit.

REPORT.md gains rows 4.1 to 4.4 and a regression row, the compensation
and sweep mechanism with its grace period, where each upload limit and
buffer cap applies, the new variables, level 4 failure rows and the
memory experiment with hypothesis, method, results and limits. AGENTS
lists the new targets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:24:41 +03:00
ArtyomandClaude Opus 5.5 a774a2fbbd Measure upload memory and bound the buffers along the chain
upload_memory.py recreates the replicas and the balancer for every run
of 10, 20 and 40 concurrent uploads of 1 and 10 MiB with 64 slots
(effectively no limit) and with 3, drives them through lab4_seed.py
from the prepared tooling, and keeps per-run JSON plus a summary with
the cgroup memory.peak of each replica and of the balancer and the
oom_kill counters. The seed script now also records memory.current
before the load and reads X-Instance-Id regardless of case.

The first run showed that slots alone only move a waiting body out of
the worker: with 3 slots and 40 uploads of 10 MiB the kernel grew the
socket buffers of HAProxy to about 110 MB and the 128 MiB balancer was
killed. HAProxy now caps its client and server socket buffers at 64
KiB and the API replicas cap tcp_rmem and tcp_wmem at 256 KiB, which
keeps the balancer at 30 MiB. glibc raised its mmap threshold after
the first large free and kept freed 10 MiB bodies in the heap, so the
API fixes MALLOC_MMAP_THRESHOLD_ at 128 KiB.

With these settings 40 concurrent uploads of 10 MiB peak at 323 MiB of
512 per replica and all 80 succeed, while without the slot limit the
same load reaches 512 MiB, a worker is killed and only 31 of 80
succeed. UPLOAD_CONCURRENCY stays at 3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:14:09 +03:00
ArtyomandClaude Opus 5.5 98df351235 Check upload faults and user errors against the stand
test_attachment_faults.py runs through the balancer against the real
PostgreSQL and RustFS. The negative suite sends a body one byte over
the limit with Content-Length (answered before any body byte) and
chunked, a decompression bomb, truncated and disguised images, missing
and foreign content types, dangerous names, a stranger's reads and
references, an attachment of another chat and invalid lists, checks
each code and error body, and finds no 5xx in the replica logs and no
extra rows.

Faults come from triggers installed for the module and dropped after
it: an AFTER INSERT trigger remembers the id and key of each fault
intent, a BEFORE UPDATE trigger rejects the ready commit for
fault-fail names and holds it for 1.8 s for fault-crash names, while
the test finds the waiting backend's client address and kills that
replica with SIGKILL. The rejected commit leaves neither row nor
object right away and an audit line says compensation; a crash leaves
a pending row and an object, the one aged past the grace period is
swept with its object while the fresh one stays, and an aged ready
attachment in a message still downloads. A body cut after 256 KiB is
logged as 400 with nothing written, and a stalled body gets 503 after
8 s with nothing written.

make upload-faults stores the output in evidence/upload-faults.txt
without tracebacks of locals, and the CI acceptance job runs it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:48:46 +03:00
ArtyomandClaude Opus 5.5 28055a8551 Pass the upload limits to the API and check the console from the network
The API replicas get UPLOAD_CONCURRENCY, UPLOAD_GRACE_SECONDS,
UPLOAD_SWEEP_INTERVAL_SECONDS and UPLOAD_SWEEP_BATCH through the S3
anchor of compose.yaml with the same defaults as the settings, and
.env.example documents them. A value in the shell overrides the
default, which the memory experiment uses to recreate the replicas with
a different number of slots.

s3-check.sh now also asks the console port of the S3 service from a
separate container on the stand network and expects a refused
connection, so 23 checks cover ports, the console inside and outside
the container, the repeated bootstrap, the rights of the API key and
anonymous access.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:40:56 +03:00
ArtyomandClaude Opus 5.5 0e4eb1738c Compensate failed uploads and sweep abandoned intents
When the ready commit fails after the object was written, the upload
handler now locks the row, keeps it if it turned out ready, and
otherwise deletes the object first and the pending row second before
answering 503. A ready update that touches no row means the intent was
already swept, so it fails as unavailable and the object goes too;
a 201 without a row cannot happen.

Every worker runs an UploadSweeper that every 30 s removes pending
intents older than UPLOAD_GRACE_SECONDS (900 s, at least 60 s) one row
at a time under FOR UPDATE SKIP LOCKED over the partial pending index,
object first, and logs an audit event for each discard. Bodies are
buffered only inside one of UPLOAD_CONCURRENCY slots per worker and
released before the ready commit, and a client that disconnects in the
middle of a body becomes a 400 instead of an unhandled 500.

The tests run against a throwaway PostgreSQL: a trigger rejects the
ready commit and the compensation leaves neither row nor object, a
row removed during the write gives 503, a failed write stays until the
grace ends, fresh intents and an aged ready attachment in a message
survive a sweep, two concurrent sweeps share five intents exactly, the
background loop runs by itself, a second upload waits for the slot
without reading its body, and aborted and stalled bodies leave nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:39:51 +03:00
ArtyomandClaude Opus 5.5 70a0c6065e Keep events of committed transactions in multi-step commands
A command may now run several transactions: the upload compensation
commits a removal after the ready commit failed, and a sweep commits
one removal per abandoned intent and ends with a rolled back lookup.
The unit of work cleared every collected event on any rollback, and the
command bus published nothing when the handler raised, so the audit
trail lost discards that had really been committed.

A rollback now drops only the events recorded inside that transaction,
and the bus publishes what was collected even when the handler fails.
Events still come only from committed transactions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:39:31 +03:00
ArtyomandClaude Opus 5.5 a41580aceb Propose upload resilience as an openspec change
Grade 4 of lab 4 asks that a failure between the S3 write and the
PostgreSQL commit leaves no eternal orphans, that concurrent uploads
cannot take unbounded memory, that user errors never become a 500 and
that the API key and the private bucket are proven limited outside the
API. The change compensates a failed ready commit right away under a
row lock, sweeps abandoned intents after a grace period in every
replica with SKIP LOCKED, maps a client abort to a client error, bounds
buffered bodies per worker with upload slots, and checks all of it
against the stand: a negative suite through the balancer, faults
injected with a PostgreSQL trigger and a killed replica, and a memory
experiment with 10, 20 and 40 concurrent uploads.

The new lab4-upload-resilience capability holds these requirements;
the write order of lab4-attachments and the console and key checks of
lab4-object-storage are updated by deltas.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-28 22:34:39 +03:00
ArtyomandClaude Opus 5.5 e4253eb076 Archive the image attachments change
Grade 3 of lab 4 is closed. RustFS with a private bucket runs in the
stand, metadata lives in PostgreSQL and the bytes only in the bucket,
uploads answer 201 ready after the object write and the commit, and
the original is served byte for byte to chat members only. The smoke
test passes from a fresh clone; limits of 10 MiB and 40 MP, content
type by content and dangerous names are covered by 325 own tests; an
upload survives the stop of the replica that took it, both replicas
fail over under writes in under 3.1 s, a full restart keeps metadata
and bytes, and with S3 stopped uploads and downloads answer 503 within
2.6 s while texts, metadata and readiness keep working. 22 checks show
the bucket is private outside the API and the API key is limited.

For the defence: upload the fixture, send it in a message, download it
as the second member and compare SHA-256, get 404 as a stranger and
403 for an anonymous GET in S3, then run persistence, failover and the
upload failover scenario.

The lab4-attachments and lab4-object-storage specs are added, and
message sending, history, idempotent replay, the restart hook and stand
persistence are updated in the main specs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 22:20:38 +03:00
ArtyomandClaude Opus 5.5 db901d6401 Record level 3 evidence and fill in the report
Evidence comes from the stand started from an empty state and from a
fresh clone: the public smoke, the contract checks, lint and 325 own
tests, the storage layout with an out-of-API comparison of a stored
object against its metadata, failover of each replica under writes, the
upload failover scenario, persistence with StartedAt of every service
including S3, the S3 outage scenario and 22 S3 privacy and bootstrap
checks. The storage evidence script now covers the attachment tables
and the S3 secrets.

REPORT.md follows the lab 4 template: launch from a clean clone,
architecture with the source of truth for each fact, the write order
between PostgreSQL and S3 and what each failure leaves behind, where
each upload limit applies, roles and services, the .env.example
mapping, grade 3 rows and the grade 3 failure table. The design notes
the botocore Expect header that hung uploads behind the port proxy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 22:20:10 +03:00
ArtyomandClaude Opus 5.5 24f9d6195b Add the attachment failure scenarios and S3 checks
lab4_scenarios.py drives the public contract through tests/client.py:
failover-upload stops the replica that answered the upload with the
stop-one hook and downloads through another one comparing SHA-256, and
s3-outage stops S3 and records status and time of upload, download,
metadata, a text message and readiness before bringing it back. Both
print JSON with an ok field. s3-check.sh checks published ports, the
console, two more bootstrap runs without state changes, the allowed
and denied operations of the API key and anonymous access including a
plain HTTP GET from a separate container. evidence-attachments.sh
shows where the bytes live and compares a stored object with its
metadata outside the API.

Makefile gains scenario, s3-check and attachments-evidence, ruff covers
the new script, and the CI acceptance job runs both scenarios and the
S3 check after the smoke test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:50:03 +03:00
ArtyomandClaude Opus 5.5 5aa766acff Run RustFS with a private bucket in the stand
MinIO community images can no longer be pulled on this arm64 host, so
the stand runs RustFS 1.0.4 pinned by digest as non-root with a read
only root, no published port, the console off and data on the s3data
volume. A one-off s3-init with rc v0.1.36 creates the bucket, keeps it
private for anonymous clients, creates the API user with a policy on
the objects and listing of that one bucket, imports a lifecycle rule
for incomplete multipart uploads and checks itself; the API replicas
start only after it succeeded. Root keys reach only s3, s3-init and the
operator s3-cli; the API gets its own key and the S3 settings through
an anchor merged into its environment.

secrets.sh creates hex S3 keys, the restart hook restarts S3 next to
PostgreSQL and Valkey and waits until it is healthy, the limits policy
covers the S3 services, and the stack evidence lists the new packages.
The object storage adapter is tested against a throwaway RustFS of the
same image: a 10 MiB round trip, a missing object, an idempotent delete
and an unreachable endpoint failing within the request budget.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:50:03 +03:00
ArtyomandClaude Opus 5.5 f9760fc098 Link messages to attachments of the same chat
A message now carries attachment_ids in the request, in every response,
in the idempotent replay and in history, in the order they were sent.
The chat aggregate checks membership first, then refuses whitespace
text, more than ten or repeated attachments, and an empty text without
attachments, so an outsider still gets 404 before any 400.

The send statement inserts the links next to the message and the
idempotency key, and the foreign key from message_attachments to
attachments of the same chat turns an unknown or foreign attachment
into 404 and rolls the whole send back, without the messaging zone
reading the attachments table. Replay and history read the list with a
subquery on the primary key. The request fingerprint covers the list,
so a missing attachment_ids and [] are the same body while a different
order is a conflict.

The zone dependency test now covers three zones, and the SQL ownership
test checks that every zone touches only its own tables.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:45:19 +03:00
ArtyomandClaude Opus 5.5 00826a40d3 Serve attachment upload, metadata and download through the API
Add the adapters behind the attachments ports: PostgreSQL for the
pending and ready rows, with a foreign key violation on the chat member
turned into 404; Pillow that opens only PNG and JPEG, refuses a type
that differs from Content-Type, checks the pixel count from the header
before any decoding, verifies PNG chunks and decodes JPEG at 1/8 scale
in a worker thread under a semaphore; and S3 on aiobotocore with path
style, SigV4, short timeouts and checksums only when required. botocore
sends Expect: 100-continue for streamed bodies and RustFS behind the
Docker port proxy never answers it, so the PUT hung until the read
timeout; the handler that adds the header is unregistered.

The HTTP layer streams the raw body into the use case, returns the
contract Attachment and streams the original back with its content
type and length and no redirect. The request pipeline takes a body
limit per route, so uploads may carry 10 MiB while JSON stays at 1 MiB.
main.py wires the zone and answers its membership port from the
messaging chat directory.

Tests run the API with an in-memory object store: download by another
member, 404 for strangers, the exact 10 MiB limit and one byte over it
with Content-Length and chunked, 40 MP and above, spoofed and truncated
images, unsupported types, dangerous names, the variant, a failing
store and a lost object.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:44:32 +03:00
ArtyomandClaude Opus 5.5 76c9673ab3 Add the attachments zone with its domain and use cases
The zone owns the Attachment aggregate, which is staged as pending with
a server-assigned object key and becomes ready only when stored, the
Filename value that refuses empty, blank, overlong names and names with
a path separator or a control character, and the declared image type.

UploadAttachment checks membership first through the ChatMembership
port, then the name and the declared type, reads the body stream only
up to the limit while hashing it, inspects the image through a port,
commits the pending row, writes the object through the ObjectStorage
port and commits the ready row. GetAttachment and OpenAttachment answer
404 to strangers and to a ready row whose object is gone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:43:41 +03:00
ArtyomandClaude Opus 5.5 3794df20d4 Add the attachments schema as an additive migration
Migration 0003 creates attachments with a pending or ready status, a
server-assigned object key in byte order collation and a partial index
for pending rows, and message_attachments that keeps the order of the
attachments of a message. Foreign keys hold the links between zones:
an attachment belongs to a member of its chat, and a message may only
reference an attachment of its own chat, with RESTRICT so a referenced
attachment can never be deleted. The text check is relaxed to at most
4000 code points because an empty text is legal next to an attachment;
an empty message without attachments is still refused by the domain.

The migration test covers the new version, a rerun without changes and
both foreign keys.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:25:35 +03:00
ArtyomandClaude Opus 5.5 d7fb96bb70 Propose image attachments as an openspec change
Plan grade 3 of lab 4 as one change: a third domain zone for
attachments that takes the raw bytes of one PNG or JPEG, applies the
size limit before reading the whole body and the pixel limit before
decoding, checks the type by content, writes a pending row, the object
and then the ready row and only then answers 201, and serves metadata
and the original only to chat members through the API. Messages gain
attachment_ids, linked by a foreign key to attachments of the same chat
so the messaging zone never reads the attachments table, and the zone
asks messaging about membership through a port wired in main.py.

The stand gains RustFS with a private bucket, a one-off bootstrap with
a separate API key, and a restart hook that waits for S3. The specs add
lab4-attachments and lab4-object-storage and modify message sending,
history, idempotent replay, the restart hook and stand persistence.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:24:19 +03:00
ArtyomandClaude Opus 5.5 bf8e4e3cd1 Carry over the lab 3 solution
Bring the application, own tests, Dockerfile, toolchain, compose stack,
HAProxy and k6 configuration, migrations, hooks, evidence and level 4
scripts and the CI job from the lab 3 solution commit 61c0c28, merge
the lab 3 Makefile targets into the lab 4 skeleton Makefile without
touching its targets, and append the lab 3 solution variables to the
skeleton .env.example. The package and the application follow the
contract version 1.4.0, and the image and the throwaway test containers
are tagged lab4 so they do not collide with a lab 3 stand on the same
host.

This is the starting point of the lab 4 cycle: lint, own tests and the
offline skeleton check pass before any lab 4 change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:17:51 +03:00
ArtyomandClaude Opus 5.5 b0f463ac5c Carry over the agent context and openspec history from lab 3
Bring the Claude Code settings, the lint hook, the opsx commands and
skills, the architecture notes, the lab 1 to lab 3 specs and the
archived lab 1 to lab 3 changes into the lab 4 skeleton, so lab 4
continues the same spec history instead of starting from scratch. The
files come from the lab 3 solution commit 61c0c28.

Rewrite AGENTS.md and the openspec context and rules for lab 4: image
attachments with metadata in PostgreSQL and bytes in a private RustFS
bucket inside the stand, a third domain zone for attachments that asks
the messaging zone about chat membership through a port wired in
main.py, upload limits applied before unbounded reading or decoding,
201 only after both the object write and the commit, Out of scope moved
to labs 5-7 plus presigned uploads, and rules on the write order between
the two stores and on root S3 keys staying out of the API.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:16:50 +03:00
Mikhail Verkovykh ec540a1a72 Подготовить skeleton лабораторной 4 по мессенджеру 2026-09-10 18:01:35 +03:00
941 changed files with 78053 additions and 76 deletions
+181
View File
@@ -0,0 +1,181 @@
---
name: "OPSX: Apply"
description: "Implement tasks from an OpenSpec change (Experimental)"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "artifacts", "experimental"]
---
Implement tasks from an OpenSpec change.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: Optionally specify a change name (e.g., `/opsx:apply add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:apply <other>`).
2. **Check status to understand the schema**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to understand:
- `schemaName`: The workflow being used (e.g., "spec-driven")
- `planningHome`, `changeRoot`, and `actionContext`: planning scope and edit constraints
- Which artifact contains the tasks (typically "tasks" for spec-driven, check status for others)
3. **Get apply instructions**
```bash
openspec instructions apply --change "<name>" --json
```
This returns:
- `contextFiles`: artifact ID -> array of concrete file paths (varies by schema - could be proposal/specs/design/tasks or spec/tests/implementation/docs)
- Progress (total, complete, remaining)
- Task list with status
- Dynamic instruction based on current state
- Optional `context`: current required project instruction input from the selected root
- Optional `operationGuidance`: current advisory guidance for apply
**Handle states:**
- If `state: "blocked"` (missing artifacts): show message, suggest using `/opsx:continue` (if it is not installed, run `openspec status --change "<name>" --json` to see the next artifact and `openspec instructions <artifact-id> --change "<name>" --json` for how to create it)
- If `state: "all_done"`: congratulate, suggest archive
- Otherwise: proceed to implementation
Treat `context` as a required prompt-level input. Read and consider it, and
apply relevant project facts, conventions, and constraints while implementing.
Treat `operationGuidance` as optional additive advice. Read and consider every
entry, and follow entries that are applicable and compatible with the built-in
workflow.
Keep both fields separate from CLI-returned state, missing artifacts, tasks,
progress, `contextFiles`, and the built-in `instruction`. They are not
evidence of task completion, do not replace the built-in instruction, and do
not permit bypassing a blocked state. If context conflicts with the built-in
instruction, an explicit user choice, or a CLI-controlled value, report the
conflict and preserve the controlling value. If guidance is inapplicable or
conflicts with those controlling inputs, do not follow it and explain why.
These are prompt-level behavior contracts, not enforceable checks.
4. **Read context files**
Read every file path listed under `contextFiles` from the apply instructions output.
The files depend on the schema being used:
- **spec-driven**: proposal, specs, design, tasks
- Other schemas: follow the contextFiles from CLI output
Do not copy `context` or `operationGuidance` verbatim into implementation
files or planning artifacts unless the user separately asks for that content.
5. **Show current progress**
Display:
- Schema being used
- Progress: "N/M tasks complete"
- Remaining tasks overview
- Dynamic instruction from CLI
6. **Implement tasks (loop until done or blocked)**
For each pending task:
- Show which task is being worked on
- Make the code changes required
- Keep changes minimal and focused
- Mark task complete in the tasks file: `- [ ]` → `- [x]`
- Continue to next task
**Pause if:**
- Task is unclear → ask for clarification
- Implementation reveals a design issue → suggest updating artifacts
- Error or blocker encountered → report and wait for guidance
- User interrupts
7. **On completion or pause, show status**
Display:
- Tasks completed this session
- Overall progress: "N/M tasks complete"
- If all done: suggest archive
- If paused: explain why and wait for guidance
**Output During Implementation**
```
## Implementing: <change-name> (schema: <schema-name>)
Working on task 3/7: <task description>
[...implementation happening...]
✓ Task complete
Working on task 4/7: <task description>
[...implementation happening...]
✓ Task complete
```
**Output On Completion**
```
## Implementation Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Progress:** 7/7 tasks complete ✓
### Completed This Session
- [x] Task 1
- [x] Task 2
...
All tasks complete! You can archive this change with `/opsx:archive`.
```
**Output On Pause (Issue Encountered)**
```
## Implementation Paused
**Change:** <change-name>
**Schema:** <schema-name>
**Progress:** 4/7 tasks complete
### Issue Encountered
<description of the issue>
**Options:**
1. <option 1>
2. <option 2>
3. Other approach
What would you like to do?
```
**Guardrails**
- Keep going through tasks until done or blocked
- Always read context files before starting (from the apply instructions output)
- If task is ambiguous, pause and ask before implementing
- If implementation reveals issues, pause and suggest artifact updates
- Keep code changes minimal and scoped to each task
- Update task checkbox immediately after completing each task
- Pause on errors, blockers, or unclear requirements - don't guess
- Use contextFiles from CLI output, don't assume specific file names
- Do not use context or operation guidance as proof that a task is complete
- Apply relevant project context; report conflicts with controlling workflow inputs
- Consider every guidance entry; explain any inapplicable or conflicting advice
- Do not copy runtime context or operation guidance into implementation files or planning artifacts
- Preserve CLI-controlled blocked/ready/all-done behavior and completion criteria
**Fluid Workflow Integration**
This skill supports the "actions on a change" model:
- **Can be invoked anytime**: Before all artifacts are done (if tasks exist), after partial implementation, interleaved with other actions
- **Allows artifact updates**: If implementation reveals design issues, suggest updating artifacts - not phase-locked, work fluidly
+225
View File
@@ -0,0 +1,225 @@
---
name: "OPSX: Archive"
description: "Archive a completed change in the experimental workflow"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "archive", "experimental"]
---
Archive a completed change in the experimental workflow.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
**Input**: Optionally specify a change name after `/opsx:archive` (e.g., `/opsx:archive add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
When prompting, show only active changes (not already archived).
Include the schema used for each change if available.
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:archive <other>`).
**Load current archive inputs before the existing archive checks:**
After resolving the selected change and planning root, run:
```bash
openspec instructions archive --change "<name>" --json
```
Keep the same selected-root flags on this command. This lookup is advisory and
optional: it only supplies extra prompt inputs, so it must never block archiving.
If it exits non-zero or returns invalid JSON — for example on an older CLI that
does not support this command yet — continue the archive workflow with no
context and no operation guidance. Do not report an error and do not stop.
A successful response may omit both optional fields. Treat `context` as a
required prompt-level input: read and consider it, and apply relevant project
facts, conventions, and constraints. Treat `operationGuidance` as optional
additive advice: read and consider every entry, and follow entries that are
applicable and compatible with the built-in archive workflow.
Keep both fields separate from built-in steps, explicit user choices, resolved
paths, CLI checks, and command contracts. If context conflicts with one of those
controlling inputs, report the conflict and preserve the controlling value. If
guidance is inapplicable or conflicts with a controlling input, do not follow it
and explain why. Do not infer replacement paths, skipped prompts, or flags from
either field, and do not copy their text verbatim into specs, change artifacts,
or archive summaries unless the user separately asks for it. These are
prompt-level behavior contracts, not enforceable checks.
2. **Check artifact completion status**
Run `openspec status --change "<name>" --json` to check artifact completion.
Parse the JSON to understand:
- `schemaName`: The workflow being used
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context
- `artifacts`: List of artifacts with their status (`done`, `skipped`, or other)
**If any artifacts are neither `done` nor `skipped`** (skipped artifacts satisfy the requirement - the change declares skip_specs):
- Display warning listing incomplete artifacts
- Prompt user for confirmation to continue
- Proceed if user confirms
3. **Check task completion status**
Read the tasks file (typically `tasks.md`) to check for incomplete tasks.
Count tasks marked with `- [ ]` (incomplete) vs `- [x]` (complete).
**If incomplete tasks found:**
- Display warning showing count of incomplete tasks
- Prompt user for confirmation to continue
- Proceed if user confirms
**If no tasks file exists:** Proceed without task-related warning.
4. **Assess delta spec sync state**
Use `artifactPaths.specs.existingOutputPaths` from status JSON as the only
delta-spec source. If the `specs` entry is missing or
`existingOutputPaths` is empty, proceed without a sync prompt and do not infer
delta specs from other artifacts.
**If delta specs exist:**
- Compare each delta spec with its corresponding main spec at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (use the store-aware `planningHome.root` from step 2, not a hardcoded repo path)
- Determine what changes would be applied (adds, modifications, removals, renames)
- Show a combined summary before prompting
**Prompt options:**
- If changes needed: "Sync now (recommended)", "Archive without syncing"
- If already synced: "Archive now", "Sync anyway", "Cancel"
Route on the answer:
- "Cancel" — stop, do not archive
- "Archive without syncing" or "Archive now" — proceed to archive
- "Sync now" or "Sync anyway" — sync, then verify (below)
- Anything else — ask again rather than archiving
Before a selected sync writes any main spec, run
`openspec instructions specs --change "<name>" --json` once with the same
selected-root flags. Require a zero exit status and valid artifact-instruction
JSON. If the lookup fails or returns invalid JSON, report the error and stop
before writing any main spec or moving the change. A valid response with omitted
`rules` is the no-rules case. Apply returned `rules` only to the content and
form of main specs produced by this merge; do not use them as archive guidance,
change CLI behavior, or copy the rule text into any output file.
Then run the `/opsx:sync` workflow inline (agent-driven intelligent merge) for change '<name>', passing the delta spec analysis and the fetched specs-rule snapshot from above, and wait for it to finish. The inline sync must reuse that snapshot without fetching `specs` instructions again. Do not delegate it to a background task — step 5 would move `changeRoot` out from under a sync that is still reading it, leaving the change archived and the main specs never updated. If your agent can only run it by delegation, delegate synchronously and wait for the result.
Then re-run the comparison from the top of this step against every capability that has a delta spec in `artifactPaths.specs.existingOutputPaths` — not only the ones the sync reports it touched. A successful sync leaves nothing left to apply, so each capability must now read as already synced:
- ADDED requirements present
- MODIFIED requirements carrying the scenario and description changes named in the delta, with their other scenarios intact
- REMOVED requirements gone — and where this sync retired a capability (removed its last requirement, leaving `## Requirements` empty), its main spec deleted rather than left empty; a spec the sync deliberately kept and reported is also a match
- RENAMED requirements present under the new name and absent under the old one
If the sync failed, or any capability does not match, report what differs and stop — do not archive. Nothing has moved and `changeRoot` is intact, so the user can fix the mismatch or re-run the sync and start the archive again.
5. **Perform the archive**
Create an `archive` directory under `planningHome.changesDir` if it doesn't exist:
```bash
mkdir -p "<planningHome.changesDir>/archive"
```
Generate the target name: use the change name as-is when it already starts with a `YYYY-MM-DD-` prefix; otherwise prepend the current date as `YYYY-MM-DD-<change-name>`. Never stack a second date (same rule as `openspec archive`).
**Check if target already exists:**
- If yes: Fail with error, suggest renaming existing archive or using different date
- If no: Move `changeRoot` to the archive directory
```bash
mv "<changeRoot>" "<planningHome.changesDir>/archive/<target-name>"
```
6. **Display summary**
Show archive completion summary including:
- Change name
- Schema that was used
- Archive location
- Spec sync status (synced / sync skipped / no delta specs)
- Note about any warnings (incomplete artifacts/tasks)
**Output On Success**
```markdown
## Archive Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
**Specs:** ✓ Synced to main specs
All artifacts complete. All tasks complete.
```
**Output On Success (No Delta Specs)**
```markdown
## Archive Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
**Specs:** No delta specs
All artifacts complete. All tasks complete.
```
**Output On Success With Warnings**
```markdown
## Archive Complete (with warnings)
**Change:** <change-name>
**Schema:** <schema-name>
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
**Specs:** Sync skipped (user chose to skip)
**Warnings:**
- Archived with 2 incomplete artifacts
- Archived with 3 incomplete tasks
- Delta spec sync was skipped (user chose to skip)
Review the archive if this was not intentional.
```
**Output On Error (Archive Exists)**
```markdown
## Archive Failed
**Change:** <change-name>
**Target:** the archive path derived from `planningHome.changesDir`/<target-name>/
Target archive directory already exists.
**Options:**
1. Rename the existing archive
2. Delete the existing archive if it's a duplicate
3. Wait until a different date to archive
```
**Guardrails**
- Announce the selected change; prompt for selection when it is ambiguous
- Use artifact graph (openspec status --json) for completion checking
- Don't block archive on warnings - just inform and confirm
- Preserve .openspec.yaml when moving to archive (it moves with the directory)
- Show clear summary of what happened
- If sync is requested, run the `/opsx:sync` workflow inline (agent-driven)
- Never archive while a spec sync is still in flight — run the sync inline and verify the main specs before moving `changeRoot`
- If delta specs exist, always run the sync assessment and show the combined summary before prompting
- Apply relevant runtime context and report conflicts; operation guidance remains advisory
- Consider every guidance entry and explain any inapplicable or conflicting advice
- Existing CLI checks, resolved paths, prompts, and command contracts are unchanged
- Artifact rules constrain only the specs being written and are never operation guidance
- Never copy runtime context, operation guidance, or artifact-rule text verbatim into output files
+193
View File
@@ -0,0 +1,193 @@
---
name: "OPSX: Explore"
description: "Enter explore mode - think through ideas, investigate problems, clarify requirements"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "explore", "experimental", "thinking"]
---
Enter explore mode. Think deeply. Visualize freely. Follow the conversation wherever it goes.
**IMPORTANT: Explore mode is for thinking, not implementing.** You may read files, search code, and investigate the codebase, but you must NEVER write code or implement features. If the user asks you to implement something, remind them to exit explore mode first and create a change proposal. You MAY create OpenSpec artifacts (proposals, designs, specs) if the user asks—that's capturing thinking, not implementing. For a new change, scaffold it first as described below.
**This is a stance, not a workflow.** There are no fixed steps, no required sequence, no mandatory outputs. You're a thinking partner helping the user explore.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: The argument after `/opsx:explore` is whatever the user wants to think about. Could be:
- A vague idea: "real-time collaboration"
- A specific problem: "the auth system is getting unwieldy"
- A change name: "add-dark-mode" (to explore in context of that change)
- A comparison: "postgres vs sqlite for this"
- Nothing (just enter explore mode)
---
## The Stance
- **Curious, not prescriptive** - Ask questions that emerge naturally, don't follow a script
- **Open threads, not interrogations** - Surface multiple interesting directions and let the user follow what resonates. Don't funnel them through a single path of questions.
- **Visual** - Use ASCII diagrams liberally when they'd help clarify thinking
- **Adaptive** - Follow interesting threads, pivot when new information emerges
- **Patient** - Don't rush to conclusions, let the shape of the problem emerge
- **Grounded** - Explore the actual codebase when relevant, don't just theorize
---
## What You Might Do
Depending on what the user brings, you might:
**Explore the problem space**
- Ask clarifying questions that emerge from what they said
- Challenge assumptions
- Reframe the problem
- Find analogies
**Investigate the codebase**
- Map existing architecture relevant to the discussion
- Find integration points
- Identify patterns already in use
- Surface hidden complexity
**Compare options**
- Brainstorm multiple approaches
- Build comparison tables
- Sketch tradeoffs
- Recommend a path (if asked)
**Visualize**
```
┌─────────────────────────────────────────┐
│ Use ASCII diagrams liberally │
├─────────────────────────────────────────┤
│ │
│ ┌────────┐ ┌────────┐ │
│ │ State │────────▶│ State │ │
│ │ A │ │ B │ │
│ └────────┘ └────────┘ │
│ │
│ System diagrams, state machines, │
│ data flows, architecture sketches, │
│ dependency graphs, comparison tables │
│ │
└─────────────────────────────────────────┘
```
**Surface risks and unknowns**
- Identify what could go wrong
- Find gaps in understanding
- Suggest spikes or investigations
---
## OpenSpec Awareness
You have full context of the OpenSpec system. Use it naturally, don't force it.
### Check for context
At the start, quickly check what exists:
```bash
openspec list --json
```
This tells you:
- If there are active changes
- Their names, schemas, and status
- What the user might be working on
Then read the project's own context from the resolved root - `<root.path>/openspec/config.yaml` (or `config.yml`). Use the `root.path` returned above, and skip this if neither file exists:
- `context`: project background - tech stack, conventions, constraints
- `rules`: keyed by artifact id - the entries for an artifact apply only when you write that artifact
Ground your thinking in these. They are constraints for you to follow, not content to reproduce: do NOT copy them into the conversation or into any artifact you create.
If the user mentioned a specific change name, read its artifacts for context.
### When no change exists
Think freely. When insights crystallize, you might offer:
- "This feels solid enough to start a change. Want me to create a proposal?"
- Or keep exploring - no pressure to formalize
If the user asks you to capture the exploration as a new change, transition seamlessly into the requested capture:
1. Run `openspec new change "<name>"` (with `--store <id>` when applicable) before creating any artifacts. Never create a new change directory under `openspec/changes/` by hand; the CLI scaffold creates required metadata such as `.openspec.yaml`. Keep the selected `--store <id>` on every applicable follow-up `status` and `instructions` command.
2. Run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store), then process the requested artifacts in dependency order. For each requested artifact that is `ready`, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store). Before creating a requested artifact, evaluate any condition in its own `instruction` against the explored change; record a deliberate skip instead when the condition does not apply. If a requested artifact is blocked by a direct prerequisite the user did not request, run `openspec instructions "<prerequisite-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) for that prerequisite whether it is `ready` or `blocked`. If its own `instruction` states a condition, evaluate that condition against the explored change and record a deliberate skip only when the condition does not apply. If the condition applies, or the prerequisite is not conditional, treat it as a normal prerequisite and ask before expanding the capture. Do not create an unrequested prerequisite unless the user approves.
3. Follow the returned `template` and `instruction` fields. Read completed dependency files listed in `dependencies`, and apply `context` and `rules` as constraints without copying them into the artifact. If the instruction delegates creation to a specific skill or command, invoke it; otherwise write the artifact to `resolvedOutputPath`, using the instruction to choose a concrete path when it is a glob. Verify that the selected concrete output exists.
4. After creating each artifact, re-run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) and continue until every requested artifact is `done`, `skipped`, or was deliberately skipped because its own `instruction` stated a condition that did not apply. Tell the user about a deliberate conditional skip, remember it, and do not reconsider it. Dependencies are enablers, not gates: if a requested artifact is still `blocked` only because you deliberately skipped a conditional prerequisite, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) despite the blocked status, then create it using step 3 only when those recorded conditional skips are its sole missing dependencies. If a requested artifact is blocked by a prerequisite the user did not ask to capture and cannot be conditionally skipped, explain that dependency and ask before expanding the capture.
Capture the artifact(s) the user requested without asking them to invoke another workflow command. If they asked only to start a change, stop after scaffolding and show its status.
### When a change exists
If the user mentions a change or you detect one is relevant:
1. **Resolve and read existing artifacts for context**
- Run `openspec status --change "<name>" --json`.
- Use `changeRoot`, `artifactPaths`, and `actionContext` from the status JSON.
- Read existing files from `artifactPaths.<artifact>.existingOutputPaths`.
2. **Reference them naturally in conversation**
- "Your design mentions using Redis, but we just realized SQLite fits better..."
- "The proposal scopes this to premium users, but we're now thinking everyone..."
3. **Offer to capture when decisions are made**
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
| Insight Type | Where to Capture |
|----------------------------|-------------------------------------|
| New requirement discovered | `specs/<capability-path>/spec.md` |
| Requirement changed | `specs/<capability-path>/spec.md` |
| Design decision made | `design.md` |
| Scope changed | `proposal.md` |
| New work identified | `tasks.md` |
| Assumption invalidated | Relevant artifact |
Example offers:
- "That's a design decision. Capture it in design.md?"
- "This is a new requirement. Add it to specs?"
- "This changes scope. Update the proposal?"
4. **The user decides** - Offer and move on. Don't pressure. Don't auto-capture.
---
## What You Don't Have To Do
- Follow a script
- Ask the same questions every time
- Produce a specific artifact
- Reach a conclusion
- Stay on topic if a tangent is valuable
- Be brief (this is thinking time)
---
## Ending Discovery
There's no required ending. Discovery might:
- **Flow into a proposal**: "Ready to start? I can create a change proposal."
- **Result in artifact updates**: "Updated design.md with these decisions"
- **Just provide clarity**: User has what they need, moves on
- **Continue later**: "We can pick this up anytime"
When things crystallize, you might offer a summary - but it's optional. Sometimes the thinking IS the value.
---
## Guardrails
- **Don't implement** - Never write code or implement features. Creating OpenSpec artifacts is fine, writing application code is not.
- **Don't fake understanding** - If something is unclear, dig deeper
- **Don't rush** - Discovery is thinking time, not task time
- **Don't force structure** - Let patterns emerge naturally
- **Don't auto-capture** - Offer to save insights, don't just do it
- **Don't manually scaffold changes** - Never create a new change directory under `openspec/changes/` by hand. Always use `openspec new change "<name>"` (with `--store <id>` when applicable) so required metadata such as `.openspec.yaml` is created before writing artifacts.
- **Do visualize** - A good diagram is worth many paragraphs
- **Do explore the codebase** - Ground discussions in reality
- **Do question assumptions** - Including the user's and your own
+145
View File
@@ -0,0 +1,145 @@
---
name: "OPSX: Propose"
description: "Propose a new change - create it and generate all artifacts in one step"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "artifacts", "experimental"]
---
Propose a new change - create the change and generate all artifacts in one step.
**Planning boundary**: This workflow creates planning artifacts only. The user request that selected or triggered this workflow authorizes planning only, even if it asks to build or fix something. Do not edit project code. After the planning artifacts are complete, stop. Do not start implementation in the same response, even if the initial request asks for it. Wait for a new user request after the artifacts are presented; then start the apply workflow.
I'll create a change with the artifacts your schema defines. With the default spec-driven schema that is:
- proposal.md (what & why)
- `specs/<capability-path>/spec.md` (what the system must do - a delta, not the main spec)
- design.md (how)
- tasks.md (implementation steps)
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
When the user is ready to implement, they must start the apply workflow explicitly.
---
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: The argument after `/opsx:propose` is the change name (kebab-case), OR a description of what the user wants to build.
**Steps**
1. **Understand the request and clarify material ambiguity**
If no input is provided, ask the user (open-ended, no preset options):
> "What change do you want to work on? Describe what you want to build or fix."
From their description, derive a kebab-case name (e.g., "add user authentication" → `add-user-auth`).
**IMPORTANT**: Do NOT proceed without understanding what the user wants to build.
If the request contains ambiguity that would materially affect scope, externally observable behavior, compatibility, or acceptance criteria, ask the user before creating the change. For minor details, make a reasonable assumption and record it in the planning artifacts.
2. **Determine the workflow schema**
Use the configured default schema unless the user explicitly requests a different workflow.
**Use a different schema only if the user:**
- Explicitly requests a specific schema by name → use `--schema <schema-name>`
- Asks to "show workflows" or asks "what workflows" exist → resolve the authoritative root by running `openspec context --json` from the current working directory. If the user explicitly selected a registered store, use `openspec context --json --store "<store-id>"`. Then run `openspec schemas --json` with its working directory set to the returned `root.path` and let them choose. This preserves roots selected by a local `store:` pointer or the global `defaultStore`; `schemas` does not accept `--store`. If context reports only `no_openspec_root`, run `openspec schemas --json` from the current working directory instead. Do not use this fallback for invalid or unavailable stores.
Otherwise, omit `--schema` to preserve the configured default.
3. **Create the change directory**
Choose one schema form below. If a registered store is selected, append `--store "<store-id>"` to that command and each later OpenSpec command shown below that accepts `--store`.
Using the configured default:
```bash
openspec new change "<name>"
```
Using an explicitly requested schema:
```bash
openspec new change "<name>" --schema "<schema-name>"
```
This creates a scaffolded change in the planning home resolved by the CLI with `.openspec.yaml`.
4. **Get the artifact build order**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to get:
- `applyRequires`: array of artifact IDs needed before implementation (e.g., `["tasks"]`)
- `artifacts`: list of all artifacts, each with its `status` and its `requires` edges (the artifact IDs it directly depends on)
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
5. **Create every artifact in the required set**
Use a todo list to track progress through the artifacts.
Loop through artifacts in dependency order (artifacts with no pending dependencies first):
a. **For each artifact that is `ready` (dependencies satisfied)**:
- Get instructions:
```bash
openspec instructions <artifact-id> --change "<name>" --json
```
- The instructions JSON includes:
- `context`: Project background (constraints for you - do NOT include in output)
- `rules`: Artifact-specific rules (constraints for you - do NOT include in output)
- `template`: The structure to use for your output file
- `instruction`: Schema-specific guidance for this artifact type
- `skipped`/`warning`: present when the change declares skip_specs and this artifact must NOT be created - stop and pick another artifact
- `resolvedOutputPath`: Resolved path or pattern to write the artifact
- `dependencies`: Completed artifacts to read for context
- Read any completed dependency files for context - always re-read them from disk, even if you saw them earlier in the conversation (the user may have edited them)
- If the `instruction` field delegates creation to a specific skill or command, invoke it to produce the artifact instead of writing the file yourself, then verify the artifact file exists at `resolvedOutputPath`
- Otherwise create the artifact file using `template` as the structure and write it to `resolvedOutputPath`. If `resolvedOutputPath` is a glob, follow `instruction` to choose the concrete file path
- Apply `context` and `rules` as constraints - but do NOT copy them into the file
- Show brief progress: "Created <artifact-id>"
b. **Continue until every artifact in the required set exists (not just `apply.requires`)**
- After creating each artifact, re-run `openspec status --change "<name>" --json`
- The required set is `applyRequires` plus every artifact reachable from those by following the `requires` edges in `status --json` - walk them transitively (spec-driven closes over proposal, specs, design, tasks). Leave artifacts outside that set alone
- `status` is file-existence only, so an `applyRequires` artifact reading `done` does NOT mean its dependencies exist - writing `tasks.md` early marks `tasks` done while `specs` was never written. Use each artifact's `requires` edges, not its `status`, to build the required set: a `done` artifact still lists what it depends on
- An artifact already reading `status: "skipped"` is satisfied: the change declares `skip_specs` in `.openspec.yaml`, so its files must NOT exist. Never try to create one
- Create every artifact in the required set that is missing, then re-check - creating one can unblock others
- Skip one only when `status` already reports it `skipped`, or when its own `instruction` says it is conditional: run `openspec instructions <artifact-id> --change "<name>" --json` and skip only if its `instruction` field marks it optional (e.g. "create only if..."). Spec-driven's `design.md` qualifies; `specs` qualifies only via the `skipped` status above, never by your own judgment. Tell the user, and do not reconsider it
- Dependencies are enablers, not gates: if a required artifact is still `blocked` only because you skipped a conditional dependency, write it anyway
- Stop when every artifact in the required set is `done`, `skipped`, or was deliberately skipped
c. **If an artifact requires user input** (unclear context):
- Ask the user to clarify
- Then continue with creation
6. **Show final status**
```bash
openspec status --change "<name>"
```
**Output**
After completing all artifacts, summarize:
- Change name and location
- List of artifacts created with brief descriptions, plus any conditional artifact you skipped and why
- What's ready: "All artifacts needed for implementation are ready."
- Prompt: "The artifacts are ready for review. When you are ready, run `/opsx:apply`."
**Artifact Creation Guidelines**
- Follow the `instruction` field from `openspec instructions` for each artifact type - it is the authoritative guidance, even for familiar artifact names
- If the `instruction` field directs you to use a specific skill or command to create the artifact, invoke it instead of writing the artifact directly
- The schema defines what each artifact should contain - follow it
- Read dependency artifacts for context before creating new ones
- Use `template` as the structure for your output file - fill in its sections
- **IMPORTANT**: `context` and `rules` are constraints for YOU, not content for the file
- Do NOT copy `<context>`, `<rules>`, `<project_context>` blocks into the artifact
- These guide what you write, but should never appear in the output
**Guardrails**
- The request that invoked this workflow authorizes planning only. Any implementation or apply instruction in that request does not carry forward. Do NOT implement the change, start the apply workflow, or edit project code during this workflow. After presenting the artifacts, stop and wait for a new user request to start the apply workflow
- Create every artifact the apply phase transitively depends on, not just the ids listed in `apply.requires`
- Always read dependency artifacts before creating a new one - re-read from disk, not from conversation memory (files may have changed since you last saw them)
- Ask about ambiguities that would materially change scope, externally observable behavior, compatibility, or acceptance criteria; for minor details, make reasonable assumptions and record them
- If a change with that name already exists, ask if user wants to continue it or create a new one
- Verify each artifact file exists after writing before proceeding to next
+258
View File
@@ -0,0 +1,258 @@
---
name: "OPSX: Sync"
description: "Sync delta specs from a change to main specs"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "specs", "experimental"]
---
Sync delta specs from a change to main specs.
This is an **agent-driven** operation - you will read delta specs and directly edit main specs to apply the changes. This allows intelligent merging (e.g., adding a scenario without copying the entire requirement).
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
**Input**: Optionally specify a change name after `/opsx:sync` (e.g., `/opsx:sync add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
When prompting, show changes that have delta specs (under `specs/` directory).
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:sync <other>`).
2. **Resolve change context**
Run:
```bash
openspec status --change "<name>" --json
```
The JSON includes `planningHome.root`. Main specs live under `<planningHome.root>/openspec/specs/` — use that (store-aware) root for every main-spec path below, not a hardcoded repo path. When a store is selected it points at the store, not the current repository.
3. **Find delta specs**
Use `artifactPaths.specs.existingOutputPaths` from the status JSON as the
only source of delta spec paths. If the `specs` entry is missing or
`existingOutputPaths` is empty, report that there are no delta specs to sync,
do not infer them from other artifacts, and stop without requesting artifact
instructions or writing a main spec.
Sync every path in `existingOutputPaths` unless the caller narrowed the set.
A caller narrows it by naming an explicit list of complete entries from
`existingOutputPaths` — copy those absolute values verbatim. Archive does
this inline, and a user can too (for example, by selecting the entry ending
in `/specs/billing/invoices/spec.md`).
Then sync only the named paths and leave the remaining delta specs untouched:
bulk archive excludes a delta whose implementation it could not find, and
syncing it anyway would write a main spec the caller deliberately withheld.
Carry that narrowed selection through step 4; never widen it back to the full
list. If a named path is not in `existingOutputPaths`, do not sync it —
report it and stop, rather than dropping it silently. If the named list is
empty, report that there is nothing to sync and stop without writing a main
spec.
Each delta spec file contains sections like:
- `## ADDED Requirements` - New requirements to add
- `## MODIFIED Requirements` - Changes to existing requirements
- `## REMOVED Requirements` - Requirements to remove
- `## RENAMED Requirements` - Requirements to rename (FROM:/TO: format)
If no delta specs found, inform user and stop.
4. **For each delta spec, apply changes to main specs**
Before the first main-spec write, obtain one current specs-rule snapshot:
- If archive invoked this workflow inline and supplied a valid snapshot from
`openspec instructions specs --change "<name>" --json`, reuse it and do not
fetch the same instructions again.
- Otherwise run that command once now with the same selected-root flags.
- If the direct lookup exits non-zero or returns invalid artifact-instruction
JSON, report the error and stop before writing any main spec. Do not treat the
failure as an absent rule set.
- A valid response with omitted `rules` means no artifact rules are configured
and the existing semantic merge continues.
Apply returned `rules` only to the content and form of the main specs produced
by this merge. Artifact rules are not operation guidance and cannot change
selected roots, delta paths, CLI checks, or workflow steps. Use their text as
constraints without copying it verbatim into a main spec or summary.
For each capability delta spec path selected in step 3 — the full `existingOutputPaths` list, or the narrowed subset when a caller supplied one (these may belong to a selected store, not the repo):
a. **Read the delta spec** to understand the intended changes
b. **Read the main spec** at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (may not exist yet)
c. **Apply changes intelligently**:
**ADDED Requirements:**
- If requirement doesn't exist in main spec → add it
- If requirement already exists → update it to match (treat as implicit MODIFIED)
**MODIFIED Requirements:**
- Find the requirement in main spec
- Apply the changes - this can be:
- Adding new scenarios the main spec does not have yet
- Modifying existing scenarios
- Changing the requirement description
- Preserve scenarios/content not mentioned in the delta
**REMOVED Requirements:**
- Remove the entire requirement block from main spec
- Retiring the capability. Delete the whole `spec.md` - and the directory once
nothing else is left in it - only when ALL of these hold:
1. removing the requirements *this run* left no requirement blocks;
2. the rest of the spec is well-formed (it still has a `## Purpose`);
3. the main spec was not already empty before this sync - if you removed
nothing, change nothing;
4. every other nonblank line in the whole file is accounted for as the
title, Purpose, Requirements header, or a canonical requirement's
statement, scenarios, or fenced examples;
5. the change's `.openspec.yaml` declares `retire_capabilities: true`;
6. the `spec.md` resolves inside the real specs root (do not follow a
capability-directory symlink to delete an external file).
If removing the selected requirements would leave no requirement blocks and
any retirement condition is not satisfied, do not modify the main spec. Stop
the sync for that capability, report the blocking condition, and tell the user
how to resolve it. Never write or leave an empty `## Requirements` section.
When only the marker is missing, say that too - it is the one thing the user
can add to make the retirement go through.
- Deleting the file also deletes its `## Purpose`; any other section blocks
retirement. Name Purpose when you report the retirement. Include a pasteable
`git checkout` only when the spec lived in the caller's checkout;
otherwise give checkout-scoped recovery guidance.
**RENAMED Requirements:**
- Find the FROM requirement, rename to TO
**`## Purpose` in the delta:**
- The main spec already has one and it is authoritative - leave it alone
(this is what `openspec archive` does; it warns and moves on)
d. **Create new main spec** if capability doesn't exist yet:
- Create `<planningHome.root>/openspec/specs/<capability-path>/spec.md`
- Add Purpose section: copy the delta's `## Purpose` body verbatim when it has one
(this is what `openspec archive` does); only write a brief TBD placeholder when it does not
- Add Requirements section with the ADDED requirements
- Follow the **Main Spec Format Reference** below
5. **Validate updated main specs**
Run `openspec validate --specs` with the same selected-root flags used earlier.
If validation fails, report the problems and do not claim the sync succeeded.
6. **Show summary**
After applying all changes, summarize:
- Which capabilities were updated
- What changes were made (requirements added/modified/removed/renamed)
- Any new main spec left with a TBD Purpose placeholder, so it gets written
now rather than lingering
- Any capability retired, naming the deleted `spec.md`, its Purpose, and
either a pasteable `git checkout` or checkout-scoped recovery guidance
**Delta Spec Format Reference**
```markdown
## Purpose
Only on a delta that introduces a brand-new capability. Seeds the new main spec.
## ADDED Requirements
### Requirement: New Feature
The system SHALL do something new.
#### Scenario: Basic case
- **WHEN** user does X
- **THEN** system does Y
## MODIFIED Requirements
### Requirement: Existing Feature
The system SHALL keep doing the existing thing, now also handling A.
#### Scenario: Scenario the main spec already has
- **WHEN** user does X
- **THEN** system does Y
#### Scenario: New scenario to add
- **WHEN** user does A
- **THEN** system does B
## REMOVED Requirements
### Requirement: Deprecated Feature
## RENAMED Requirements
- FROM: `### Requirement: Old Name`
- TO: `### Requirement: New Name`
```
**Main Spec Format Reference**
Main specs are what the delta merges INTO. They must never contain delta operation headers (`## ADDED/MODIFIED/REMOVED/RENAMED Requirements`) - after syncing, every requirement lives under a single `## Requirements` section:
```markdown
# <capability> Specification
## Purpose
Short description of what this capability does and why it exists.
## Requirements
### Requirement: New Feature
The system SHALL do something new.
#### Scenario: Basic case
- **WHEN** user does X
- **THEN** system does Y
```
**Key Principle: Intelligent Merging**
Unlike programmatic merging, you merge rather than overwrite:
- A MODIFIED block carries the whole requirement - body plus every scenario that survives the change. `openspec validate` and `openspec archive` both reject one that drops a scenario the main spec still has.
- Keep anything the delta does not mention, in the main spec's existing order
- Use your judgment to merge changes sensibly
**Output On Success**
```markdown
## Specs Synced: <change-name>
Updated main specs:
**<capability-1>**:
- Added requirement: "New Feature"
- Modified requirement: "Existing Feature" (added 1 scenario)
**<capability-2>**:
- Created new spec file
- Added requirement: "Another Feature"
Main specs are now updated. The change remains active - archive when implementation is complete.
```
**Guardrails**
- Read both delta and main specs before making changes
- Preserve existing content not mentioned in delta
- Never copy a delta file into a main spec as-is - merge its content so the main spec keeps the Main Spec Format Reference structure, with no delta operation headers
- If something is unclear, ask for clarification
- Show what you're changing as you go
- The operation should be idempotent - running twice should give same result
- Use only `artifactPaths.specs.existingOutputPaths`; never infer delta specs from unrelated artifacts
- Honor a caller-supplied subset of `existingOutputPaths`; never widen it back to the full list
- Fetch specs instructions once for direct sync, or reuse the archive-supplied snapshot inline
- Stop before every main-spec write on a non-zero or invalid JSON specs-instruction response
- Artifact rules constrain only the specs being written and are never copied into output files
+87
View File
@@ -0,0 +1,87 @@
---
name: "OPSX: Update"
description: "Update a change - revise existing planning artifacts and keep them coherent (Experimental)"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "artifacts", "experimental"]
---
Revise a change's existing planning artifacts and keep them coherent. Never edit code.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: Optionally specify a change name after `/opsx:update` (e.g., `/opsx:update add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
`/opsx:continue` is an expanded-profile workflow and may not be installed. Before suggesting it anywhere below, verify that it is available. If it is unavailable, `openspec status --change "<name>" --json` shows the next artifact and `openspec instructions "<artifact-id>" --change "<name>" --json` explains how to create it.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes sorted by most recently modified, and ask the user to select one
When prompting, present the top 3-4 most recently modified changes as options, showing:
- Change name
- Schema (from `schema` field if present, otherwise "spec-driven")
- Status (e.g., "0/5 tasks", "complete", "no tasks")
- How recently it was modified (from `lastModified` field)
Mark the most recently modified change as "(Recommended)" since it's likely what the user wants to update.
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:update <other>`).
2. **Get the change's artifacts**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to understand current state. The response includes:
- `schemaName`: The workflow schema being used (e.g., "spec-driven")
- `artifacts`: Array of artifacts with their status ("done", "skipped", "ready", "blocked")
- `isPlanningComplete`: Boolean indicating if all planning artifacts are complete. Older CLI versions expose the same value as `isComplete`.
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
The artifact ids and paths come from the active schema - do NOT assume them, and do NOT branch on hardcoded artifact names. Custom schemas must work unchanged.
The files to edit are `artifactPaths.<id>.existingOutputPaths` - the concrete files that exist on disk, already glob-expanded for glob artifacts (e.g. `specs/**/*.md`). Do NOT write to `resolvedOutputPath`: for a glob artifact it is still the glob pattern, not a real file.
3. **Understand the request**
- If the user asked for a specific revision ("the design now uses X"), that is the starting edit.
- If they only said "update" / "make this coherent", treat it as a coherence review: read the existing artifacts and check them against each other for contradictions, gaps, and duplication.
4. **Read and reconcile**
- Read the artifact(s) the request touches and the change's other existing artifacts.
- Apply the requested edit. Then check every other existing artifact against it - in ANY direction: an edit to a later artifact may require revising an earlier one, not only the other way around. Build order is a useful reading order, not a constraint on which artifacts may be revised.
- Note everything that is now inconsistent, missing, or contradictory.
- Revise only files that already exist (`existingOutputPaths`). Do NOT create artifacts that don't exist yet, and do NOT invent new files under a glob artifact - note them and point the user to `/opsx:continue` to create them.
- If the change is already coherent, say so and make no edits.
5. **Confirm and apply, one artifact at a time**
- Show each proposed revision and why. Write only after the user confirms.
- If the user rejects a revision, do not write it - leave that artifact unchanged.
- When a substantial rewrite is needed, get that artifact's rules and template first:
```bash
openspec instructions "<artifact-id>" --change "<name>" --json
```
6. **Point to the next step (guidance only - NEVER act on it)**
- Artifacts still missing -> suggest `/opsx:continue` to create them.
- Change already implemented (tasks checked off / already applied) -> the code may no longer match the revised plan; suggest `/opsx:apply` to carry the delta into code.
- Everything done and implemented -> suggest `/opsx:archive`.
**Output**
After each invocation, show:
- Which artifacts were revised (and which proposed revisions were rejected)
- Anything deferred to `/opsx:continue` (not-yet-created artifacts or files)
- Where the change stands and the recommended next command
**Guardrails**
- Planning artifacts only - NEVER edit implementation code. If the revised plan implies code changes, stop and point to `/opsx:apply`.
- Use the artifact ids and paths reported by `openspec status`; never branch on hardcoded artifact names.
- Edit only the concrete files in `existingOutputPaths`; never write to a glob `resolvedOutputPath`.
- Do not advance the build frontier: no new artifacts, no new files under glob artifacts - that is `/opsx:continue`'s job.
- Confirm every edit with the user before writing.
- If the request changes the change's *intent* rather than refining it, first verify whether the expanded-profile `/opsx:new` workflow is available. If it is, recommend starting fresh with `/opsx:new` (the "Update vs. Start Fresh" heuristic). If it is unavailable, ask for a distinct unused change name and recommend `openspec new change "<new-change-name>"` instead.
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env bash
set -uo pipefail
ROOT="${CLAUDE_PROJECT_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"
PAYLOAD=$(cat)
FILE=$(printf '%s' "$PAYLOAD" | /usr/bin/python3 -c 'import json,sys
try:
d = json.load(sys.stdin)
except Exception:
sys.exit(0)
t = d.get("tool_input") or {}
r = d.get("tool_response") or {}
p = r.get("filePath") or t.get("file_path") or ""
print(p)
' 2>/dev/null)
case "$FILE" in
"") exit 0 ;;
*.py) ;;
*) exit 0 ;;
esac
case "$FILE" in
"$ROOT"/src/*) ;;
src/*) FILE="$ROOT/$FILE" ;;
*) exit 0 ;;
esac
[ -f "$FILE" ] || exit 0
[ -x "$ROOT/.venv/bin/ruff" ] || exit 0
OUT=$(
cd "$ROOT" || exit 0
"$ROOT/.venv/bin/ruff" check --fix --quiet "$FILE" 2>&1
"$ROOT/.venv/bin/ruff" format --quiet "$FILE" 2>&1
if [ -x "$ROOT/.venv/bin/ty" ]; then
"$ROOT/.venv/bin/ty" check --quiet "$FILE" 2>&1 | grep -v "^All checks passed" || true
fi
)
if [ -n "$OUT" ]; then
printf 'ruff/ty по %s:\n%s\n' "${FILE#"$ROOT"/}" "$OUT" >&2
exit 2
fi
exit 0
+17
View File
@@ -0,0 +1,17 @@
{
"hooks": {
"PostToolUse": [
{
"matcher": "Write|Edit|MultiEdit|NotebookEdit",
"hooks": [
{
"type": "command",
"command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/lint-python.sh",
"timeout": 60,
"statusMessage": "ruff + ty по src/"
}
]
}
]
}
}
@@ -0,0 +1,185 @@
---
name: openspec-apply-change
description: Implement tasks from an OpenSpec change. Use when the user wants to start implementing, continue implementation, or work through tasks.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Implement tasks from an OpenSpec change.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: Optionally specify a change name (e.g., `/opsx:apply add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:apply <other>`).
2. **Check status to understand the schema**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to understand:
- `schemaName`: The workflow being used (e.g., "spec-driven")
- `planningHome`, `changeRoot`, and `actionContext`: planning scope and edit constraints
- Which artifact contains the tasks (typically "tasks" for spec-driven, check status for others)
3. **Get apply instructions**
```bash
openspec instructions apply --change "<name>" --json
```
This returns:
- `contextFiles`: artifact ID -> array of concrete file paths (varies by schema - could be proposal/specs/design/tasks or spec/tests/implementation/docs)
- Progress (total, complete, remaining)
- Task list with status
- Dynamic instruction based on current state
- Optional `context`: current required project instruction input from the selected root
- Optional `operationGuidance`: current advisory guidance for apply
**Handle states:**
- If `state: "blocked"` (missing artifacts): show message, suggest using `/opsx:continue` (if it is not installed, run `openspec status --change "<name>" --json` to see the next artifact and `openspec instructions <artifact-id> --change "<name>" --json` for how to create it)
- If `state: "all_done"`: congratulate, suggest archive
- Otherwise: proceed to implementation
Treat `context` as a required prompt-level input. Read and consider it, and
apply relevant project facts, conventions, and constraints while implementing.
Treat `operationGuidance` as optional additive advice. Read and consider every
entry, and follow entries that are applicable and compatible with the built-in
workflow.
Keep both fields separate from CLI-returned state, missing artifacts, tasks,
progress, `contextFiles`, and the built-in `instruction`. They are not
evidence of task completion, do not replace the built-in instruction, and do
not permit bypassing a blocked state. If context conflicts with the built-in
instruction, an explicit user choice, or a CLI-controlled value, report the
conflict and preserve the controlling value. If guidance is inapplicable or
conflicts with those controlling inputs, do not follow it and explain why.
These are prompt-level behavior contracts, not enforceable checks.
4. **Read context files**
Read every file path listed under `contextFiles` from the apply instructions output.
The files depend on the schema being used:
- **spec-driven**: proposal, specs, design, tasks
- Other schemas: follow the contextFiles from CLI output
Do not copy `context` or `operationGuidance` verbatim into implementation
files or planning artifacts unless the user separately asks for that content.
5. **Show current progress**
Display:
- Schema being used
- Progress: "N/M tasks complete"
- Remaining tasks overview
- Dynamic instruction from CLI
6. **Implement tasks (loop until done or blocked)**
For each pending task:
- Show which task is being worked on
- Make the code changes required
- Keep changes minimal and focused
- Mark task complete in the tasks file: `- [ ]` → `- [x]`
- Continue to next task
**Pause if:**
- Task is unclear → ask for clarification
- Implementation reveals a design issue → suggest updating artifacts
- Error or blocker encountered → report and wait for guidance
- User interrupts
7. **On completion or pause, show status**
Display:
- Tasks completed this session
- Overall progress: "N/M tasks complete"
- If all done: suggest archive
- If paused: explain why and wait for guidance
**Output During Implementation**
```
## Implementing: <change-name> (schema: <schema-name>)
Working on task 3/7: <task description>
[...implementation happening...]
✓ Task complete
Working on task 4/7: <task description>
[...implementation happening...]
✓ Task complete
```
**Output On Completion**
```
## Implementation Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Progress:** 7/7 tasks complete ✓
### Completed This Session
- [x] Task 1
- [x] Task 2
...
All tasks complete! You can archive this change with `/opsx:archive`.
```
**Output On Pause (Issue Encountered)**
```
## Implementation Paused
**Change:** <change-name>
**Schema:** <schema-name>
**Progress:** 4/7 tasks complete
### Issue Encountered
<description of the issue>
**Options:**
1. <option 1>
2. <option 2>
3. Other approach
What would you like to do?
```
**Guardrails**
- Keep going through tasks until done or blocked
- Always read context files before starting (from the apply instructions output)
- If task is ambiguous, pause and ask before implementing
- If implementation reveals issues, pause and suggest artifact updates
- Keep code changes minimal and scoped to each task
- Update task checkbox immediately after completing each task
- Pause on errors, blockers, or unclear requirements - don't guess
- Use contextFiles from CLI output, don't assume specific file names
- Do not use context or operation guidance as proof that a task is complete
- Apply relevant project context; report conflicts with controlling workflow inputs
- Consider every guidance entry; explain any inapplicable or conflicting advice
- Do not copy runtime context or operation guidance into implementation files or planning artifacts
- Preserve CLI-controlled blocked/ready/all-done behavior and completion criteria
**Fluid Workflow Integration**
This skill supports the "actions on a change" model:
- **Can be invoked anytime**: Before all artifacts are done (if tasks exist), after partial implementation, interleaved with other actions
- **Allows artifact updates**: If implementation reveals design issues, suggest updating artifacts - not phase-locked, work fluidly
@@ -0,0 +1,182 @@
---
name: openspec-archive-change
description: Archive a completed change in the experimental workflow. Use when the user wants to finalize and archive a change after implementation is complete.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Archive a completed change in the experimental workflow.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
When prompting, show only active changes (not already archived).
Include the schema used for each change if available.
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:archive <other>`).
**Load current archive inputs before the existing archive checks:**
After resolving the selected change and planning root, run:
```bash
openspec instructions archive --change "<name>" --json
```
Keep the same selected-root flags on this command. This lookup is advisory and
optional: it only supplies extra prompt inputs, so it must never block archiving.
If it exits non-zero or returns invalid JSON — for example on an older CLI that
does not support this command yet — continue the archive workflow with no
context and no operation guidance. Do not report an error and do not stop.
A successful response may omit both optional fields. Treat `context` as a
required prompt-level input: read and consider it, and apply relevant project
facts, conventions, and constraints. Treat `operationGuidance` as optional
additive advice: read and consider every entry, and follow entries that are
applicable and compatible with the built-in archive workflow.
Keep both fields separate from built-in steps, explicit user choices, resolved
paths, CLI checks, and command contracts. If context conflicts with one of those
controlling inputs, report the conflict and preserve the controlling value. If
guidance is inapplicable or conflicts with a controlling input, do not follow it
and explain why. Do not infer replacement paths, skipped prompts, or flags from
either field, and do not copy their text verbatim into specs, change artifacts,
or archive summaries unless the user separately asks for it. These are
prompt-level behavior contracts, not enforceable checks.
2. **Check artifact completion status**
Run `openspec status --change "<name>" --json` to check artifact completion.
Parse the JSON to understand:
- `schemaName`: The workflow being used
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context
- `artifacts`: List of artifacts with their status (`done`, `skipped`, or other)
**If any artifacts are neither `done` nor `skipped`** (skipped artifacts satisfy the requirement - the change declares skip_specs):
- Display warning listing incomplete artifacts
- Ask the user to confirm they want to proceed
- Proceed if user confirms
3. **Check task completion status**
Read the tasks file (typically `tasks.md`) to check for incomplete tasks.
Count tasks marked with `- [ ]` (incomplete) vs `- [x]` (complete).
**If incomplete tasks found:**
- Display warning showing count of incomplete tasks
- Ask the user to confirm they want to proceed
- Proceed if user confirms
**If no tasks file exists:** Proceed without task-related warning.
4. **Assess delta spec sync state**
Use `artifactPaths.specs.existingOutputPaths` from status JSON as the only
delta-spec source. If the `specs` entry is missing or
`existingOutputPaths` is empty, proceed without a sync prompt and do not infer
delta specs from other artifacts.
**If delta specs exist:**
- Compare each delta spec with its corresponding main spec at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (use the store-aware `planningHome.root` from step 2, not a hardcoded repo path)
- Determine what changes would be applied (adds, modifications, removals, renames)
- Show a combined summary before prompting
**Prompt options:**
- If changes needed: "Sync now (recommended)", "Archive without syncing"
- If already synced: "Archive now", "Sync anyway", "Cancel"
Route on the answer:
- "Cancel" — stop, do not archive
- "Archive without syncing" or "Archive now" — proceed to archive
- "Sync now" or "Sync anyway" — sync, then verify (below)
- Anything else — ask again rather than archiving
Before a selected sync writes any main spec, run
`openspec instructions specs --change "<name>" --json` once with the same
selected-root flags. Require a zero exit status and valid artifact-instruction
JSON. If the lookup fails or returns invalid JSON, report the error and stop
before writing any main spec or moving the change. A valid response with omitted
`rules` is the no-rules case. Apply returned `rules` only to the content and
form of main specs produced by this merge; do not use them as archive guidance,
change CLI behavior, or copy the rule text into any output file.
Then run the `openspec-sync-specs` workflow inline (agent-driven intelligent merge) for change '<name>', passing the delta spec analysis and the fetched specs-rule snapshot from above, and wait for it to finish. The inline sync must reuse that snapshot without fetching `specs` instructions again. Do not delegate it to a background task — step 5 would move `changeRoot` out from under a sync that is still reading it, leaving the change archived and the main specs never updated. If your agent can only run it by delegation, delegate synchronously and wait for the result.
Then re-run the comparison from the top of this step against every capability that has a delta spec in `artifactPaths.specs.existingOutputPaths` — not only the ones the sync reports it touched. A successful sync leaves nothing left to apply, so each capability must now read as already synced:
- ADDED requirements present
- MODIFIED requirements carrying the scenario and description changes named in the delta, with their other scenarios intact
- REMOVED requirements gone — and where this sync retired a capability (removed its last requirement, leaving `## Requirements` empty), its main spec deleted rather than left empty; a spec the sync deliberately kept and reported is also a match
- RENAMED requirements present under the new name and absent under the old one
If the sync failed, or any capability does not match, report what differs and stop — do not archive. Nothing has moved and `changeRoot` is intact, so the user can fix the mismatch or re-run the sync and start the archive again.
5. **Perform the archive**
Create an `archive` directory under `planningHome.changesDir` if it doesn't exist:
```bash
mkdir -p "<planningHome.changesDir>/archive"
```
Generate the target name: use the change name as-is when it already starts with a `YYYY-MM-DD-` prefix; otherwise prepend the current date as `YYYY-MM-DD-<change-name>`. Never stack a second date (same rule as `openspec archive`).
**Check if target already exists:**
- If yes: Fail with error, suggest renaming existing archive or using different date
- If no: Move `changeRoot` to the archive directory
```bash
mv "<changeRoot>" "<planningHome.changesDir>/archive/<target-name>"
```
6. **Display summary**
Show archive completion summary including:
- Change name
- Schema that was used
- Archive location
- Whether specs were synced (if applicable)
- Note about any warnings (incomplete artifacts/tasks)
**Output On Success**
```markdown
## Archive Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
**Specs:** <"✓ Synced to main specs" only if the step 4 verification passed; otherwise "No delta specs" or "Sync skipped">
<"All artifacts complete. All tasks complete." — or, if archived with warnings, list them instead (e.g. "Archived with 2 incomplete tasks")>
```
**Guardrails**
- Announce the selected change; prompt for selection when it is ambiguous
- Use artifact graph (openspec status --json) for completion checking
- Don't block archive on warnings - just inform and confirm
- Preserve .openspec.yaml when moving to archive (it moves with the directory)
- Show clear summary of what happened
- If sync is requested, run the `openspec-sync-specs` workflow inline (agent-driven)
- Never archive while a spec sync is still in flight — run the sync inline and verify the main specs before moving `changeRoot`
- If delta specs exist, always run the sync assessment and show the combined summary before prompting
- Apply relevant runtime context and report conflicts; operation guidance remains advisory
- Consider every guidance entry and explain any inapplicable or conflicting advice
- Existing CLI checks, resolved paths, prompts, and command contracts are unchanged
- Artifact rules constrain only the specs being written and are never operation guidance
- Never copy runtime context, operation guidance, or artifact-rule text verbatim into output files
+308
View File
@@ -0,0 +1,308 @@
---
name: openspec-explore
description: Enter explore mode - a thinking partner for exploring ideas, investigating problems, and clarifying requirements. Use when the user wants to think through something before or during a change.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Enter explore mode. Think deeply. Visualize freely. Follow the conversation wherever it goes.
**IMPORTANT: Explore mode is for thinking, not implementing.** You may read files, search code, and investigate the codebase, but you must NEVER write code or implement features. If the user asks you to implement something, remind them to exit explore mode first and create a change proposal. You MAY create OpenSpec artifacts (proposals, designs, specs) if the user asks—that's capturing thinking, not implementing. For a new change, scaffold it first as described below.
**This is a stance, not a workflow.** There are no fixed steps, no required sequence, no mandatory outputs. You're a thinking partner helping the user explore.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
---
## The Stance
- **Curious, not prescriptive** - Ask questions that emerge naturally, don't follow a script
- **Open threads, not interrogations** - Surface multiple interesting directions and let the user follow what resonates. Don't funnel them through a single path of questions.
- **Visual** - Use ASCII diagrams liberally when they'd help clarify thinking
- **Adaptive** - Follow interesting threads, pivot when new information emerges
- **Patient** - Don't rush to conclusions, let the shape of the problem emerge
- **Grounded** - Explore the actual codebase when relevant, don't just theorize
---
## What You Might Do
Depending on what the user brings, you might:
**Explore the problem space**
- Ask clarifying questions that emerge from what they said
- Challenge assumptions
- Reframe the problem
- Find analogies
**Investigate the codebase**
- Map existing architecture relevant to the discussion
- Find integration points
- Identify patterns already in use
- Surface hidden complexity
**Compare options**
- Brainstorm multiple approaches
- Build comparison tables
- Sketch tradeoffs
- Recommend a path (if asked)
**Visualize**
```
┌─────────────────────────────────────────┐
│ Use ASCII diagrams liberally │
├─────────────────────────────────────────┤
│ │
│ ┌────────┐ ┌────────┐ │
│ │ State │────────▶│ State │ │
│ │ A │ │ B │ │
│ └────────┘ └────────┘ │
│ │
│ System diagrams, state machines, │
│ data flows, architecture sketches, │
│ dependency graphs, comparison tables │
│ │
└─────────────────────────────────────────┘
```
**Surface risks and unknowns**
- Identify what could go wrong
- Find gaps in understanding
- Suggest spikes or investigations
---
## OpenSpec Awareness
You have full context of the OpenSpec system. Use it naturally, don't force it.
### Check for context
At the start, quickly check what exists:
```bash
openspec list --json
```
This tells you:
- If there are active changes
- Their names, schemas, and status
- What the user might be working on
Then read the project's own context from the resolved root - `<root.path>/openspec/config.yaml` (or `config.yml`). Use the `root.path` returned above, and skip this if neither file exists:
- `context`: project background - tech stack, conventions, constraints
- `rules`: keyed by artifact id - the entries for an artifact apply only when you write that artifact
Ground your thinking in these. They are constraints for you to follow, not content to reproduce: do NOT copy them into the conversation or into any artifact you create.
### When no change exists
Think freely. When insights crystallize, you might offer:
- "This feels solid enough to start a change. Want me to create a proposal?"
- Or keep exploring - no pressure to formalize
If the user asks you to capture the exploration as a new change, transition seamlessly into the requested capture:
1. Run `openspec new change "<name>"` (with `--store <id>` when applicable) before creating any artifacts. Never create a new change directory under `openspec/changes/` by hand; the CLI scaffold creates required metadata such as `.openspec.yaml`. Keep the selected `--store <id>` on every applicable follow-up `status` and `instructions` command.
2. Run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store), then process the requested artifacts in dependency order. For each requested artifact that is `ready`, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store). Before creating a requested artifact, evaluate any condition in its own `instruction` against the explored change; record a deliberate skip instead when the condition does not apply. If a requested artifact is blocked by a direct prerequisite the user did not request, run `openspec instructions "<prerequisite-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) for that prerequisite whether it is `ready` or `blocked`. If its own `instruction` states a condition, evaluate that condition against the explored change and record a deliberate skip only when the condition does not apply. If the condition applies, or the prerequisite is not conditional, treat it as a normal prerequisite and ask before expanding the capture. Do not create an unrequested prerequisite unless the user approves.
3. Follow the returned `template` and `instruction` fields. Read completed dependency files listed in `dependencies`, and apply `context` and `rules` as constraints without copying them into the artifact. If the instruction delegates creation to a specific skill or command, invoke it; otherwise write the artifact to `resolvedOutputPath`, using the instruction to choose a concrete path when it is a glob. Verify that the selected concrete output exists.
4. After creating each artifact, re-run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) and continue until every requested artifact is `done`, `skipped`, or was deliberately skipped because its own `instruction` stated a condition that did not apply. Tell the user about a deliberate conditional skip, remember it, and do not reconsider it. Dependencies are enablers, not gates: if a requested artifact is still `blocked` only because you deliberately skipped a conditional prerequisite, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) despite the blocked status, then create it using step 3 only when those recorded conditional skips are its sole missing dependencies. If a requested artifact is blocked by a prerequisite the user did not ask to capture and cannot be conditionally skipped, explain that dependency and ask before expanding the capture.
Capture the artifact(s) the user requested without asking them to invoke another workflow command. If they asked only to start a change, stop after scaffolding and show its status.
### When a change exists
If the user mentions a change or you detect one is relevant:
1. **Resolve and read existing artifacts for context**
- Run `openspec status --change "<name>" --json`.
- Use `changeRoot`, `artifactPaths`, and `actionContext` from the status JSON.
- Read existing files from `artifactPaths.<artifact>.existingOutputPaths`.
2. **Reference them naturally in conversation**
- "Your design mentions using Redis, but we just realized SQLite fits better..."
- "The proposal scopes this to premium users, but we're now thinking everyone..."
3. **Offer to capture when decisions are made**
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
| Insight Type | Where to Capture |
|----------------------------|-------------------------------------|
| New requirement discovered | `specs/<capability-path>/spec.md` |
| Requirement changed | `specs/<capability-path>/spec.md` |
| Design decision made | `design.md` |
| Scope changed | `proposal.md` |
| New work identified | `tasks.md` |
| Assumption invalidated | Relevant artifact |
Example offers:
- "That's a design decision. Capture it in design.md?"
- "This is a new requirement. Add it to specs?"
- "This changes scope. Update the proposal?"
4. **The user decides** - Offer and move on. Don't pressure. Don't auto-capture.
---
## What You Don't Have To Do
- Follow a script
- Ask the same questions every time
- Produce a specific artifact
- Reach a conclusion
- Stay on topic if a tangent is valuable
- Be brief (this is thinking time)
---
## Handling Different Entry Points
**User brings a vague idea:**
```
User: I'm thinking about adding real-time collaboration
You: Real-time collab is a big space. Let me think about this...
COLLABORATION SPECTRUM
════════════════════════════════════════════
Awareness Coordination Sync
│ │ │
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│Presence│ │Cursors │ │ CRDT │
│ "3 │ │ Multi │ │Conflict│
│online" │ │ select │ │ free │
└────────┘ └────────┘ └────────┘
│ │ │
trivial moderate complex
Where's your head at?
```
**User brings a specific problem:**
```
User: The auth system is a mess
You: [reads codebase]
┌─────────────────────────────────────────────┐
│ CURRENT AUTH FLOW │
└─────────────────────────────────────────────┘
│
┌─────────────┼─────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Google │ │ GitHub │ │ Email │
│ OAuth │ │ OAuth │ │ Magic │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
└─────────────┼─────────────┘
▼
┌───────────┐
│ Session │
└─────┬─────┘
│
▼
┌───────────┐
│ Perms │
└───────────┘
I see three tangles. Which one's burning?
```
**User is stuck mid-implementation:**
```
User: /opsx:explore add-auth-system
The OAuth integration is more complex than expected
You: [reads change artifacts]
You're on task 4: "Implement OAuth flow"
Let me trace what's involved...
[draws diagram, explores options, suggests paths]
Want to update the design to reflect this?
Or add a spike task to investigate?
```
**User wants to compare options:**
```
User: Should we use Postgres or SQLite?
You: Generic answer is boring. What's the context?
User: A CLI tool that tracks local dev environments
You: That changes everything.
┌─────────────────────────────────────────────────┐
│ CLI TOOL DATA STORAGE │
└─────────────────────────────────────────────────┘
Key constraints:
• No daemon running
• Must work offline
• Single user
SQLite Postgres
Deployment embedded ✓ needs server ✗
Offline yes ✓ no ✗
Single file yes ✓ no ✗
SQLite. Not even close.
Unless... is there a sync component?
```
---
## Ending Discovery
There's no required ending. Discovery might:
- **Flow into a proposal**: "Ready to start? I can create a change proposal."
- **Result in artifact updates**: "Updated design.md with these decisions"
- **Just provide clarity**: User has what they need, moves on
- **Continue later**: "We can pick this up anytime"
When it feels like things are crystallizing, you might summarize:
```
## What We Figured Out
**The problem**: [crystallized understanding]
**The approach**: [if one emerged]
**Open questions**: [if any remain]
**Next steps** (if ready):
- Create a change proposal
- Keep exploring: just keep talking
```
But this summary is optional. Sometimes the thinking IS the value.
---
## Guardrails
- **Don't implement** - Never write code or implement features. Creating OpenSpec artifacts is fine, writing application code is not.
- **Don't fake understanding** - If something is unclear, dig deeper
- **Don't rush** - Discovery is thinking time, not task time
- **Don't force structure** - Let patterns emerge naturally
- **Don't auto-capture** - Offer to save insights, don't just do it
- **Don't manually scaffold changes** - Never create a new change directory under `openspec/changes/` by hand. Always use `openspec new change "<name>"` (with `--store <id>` when applicable) so required metadata such as `.openspec.yaml` is created before writing artifacts.
- **Do visualize** - A good diagram is worth many paragraphs
- **Do explore the codebase** - Ground discussions in reality
- **Do question assumptions** - Including the user's and your own
+149
View File
@@ -0,0 +1,149 @@
---
name: openspec-propose
description: Propose a new change with all artifacts generated in one step. Use when the user wants to quickly describe what they want to build and get a complete proposal with design, specs, and tasks ready for implementation.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Propose a new change - create the change and generate all artifacts in one step.
**Planning boundary**: This workflow creates planning artifacts only. The user request that selected or triggered this workflow authorizes planning only, even if it asks to build or fix something. Do not edit project code. After the planning artifacts are complete, stop. Do not start implementation in the same response, even if the initial request asks for it. Wait for a new user request after the artifacts are presented; then start the apply workflow.
I'll create a change with the artifacts your schema defines. With the default spec-driven schema that is:
- proposal.md (what & why)
- `specs/<capability-path>/spec.md` (what the system must do - a delta, not the main spec)
- design.md (how)
- tasks.md (implementation steps)
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
When the user is ready to implement, they must start the apply workflow explicitly.
---
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: The user's request should include a change name (kebab-case) OR a description of what they want to build.
**Steps**
1. **Understand the request and clarify material ambiguity**
If no clear input is provided, ask the user (open-ended, no preset options):
> "What change do you want to work on? Describe what you want to build or fix."
From their description, derive a kebab-case name (e.g., "add user authentication" → `add-user-auth`).
**IMPORTANT**: Do NOT proceed without understanding what the user wants to build.
If the request contains ambiguity that would materially affect scope, externally observable behavior, compatibility, or acceptance criteria, ask the user before creating the change. For minor details, make a reasonable assumption and record it in the planning artifacts.
2. **Determine the workflow schema**
Use the configured default schema unless the user explicitly requests a different workflow.
**Use a different schema only if the user:**
- Explicitly requests a specific schema by name → use `--schema <schema-name>`
- Asks to "show workflows" or asks "what workflows" exist → resolve the authoritative root by running `openspec context --json` from the current working directory. If the user explicitly selected a registered store, use `openspec context --json --store "<store-id>"`. Then run `openspec schemas --json` with its working directory set to the returned `root.path` and let them choose. This preserves roots selected by a local `store:` pointer or the global `defaultStore`; `schemas` does not accept `--store`. If context reports only `no_openspec_root`, run `openspec schemas --json` from the current working directory instead. Do not use this fallback for invalid or unavailable stores.
Otherwise, omit `--schema` to preserve the configured default.
3. **Create the change directory**
Choose one schema form below. If a registered store is selected, append `--store "<store-id>"` to that command and each later OpenSpec command shown below that accepts `--store`.
Using the configured default:
```bash
openspec new change "<name>"
```
Using an explicitly requested schema:
```bash
openspec new change "<name>" --schema "<schema-name>"
```
This creates a scaffolded change in the planning home resolved by the CLI with `.openspec.yaml`.
4. **Get the artifact build order**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to get:
- `applyRequires`: array of artifact IDs needed before implementation (e.g., `["tasks"]`)
- `artifacts`: list of all artifacts, each with its `status` and its `requires` edges (the artifact IDs it directly depends on)
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
5. **Create every artifact in the required set**
Use a todo list to track progress through the artifacts.
Loop through artifacts in dependency order (artifacts with no pending dependencies first):
a. **For each artifact that is `ready` (dependencies satisfied)**:
- Get instructions:
```bash
openspec instructions <artifact-id> --change "<name>" --json
```
- The instructions JSON includes:
- `context`: Project background (constraints for you - do NOT include in output)
- `rules`: Artifact-specific rules (constraints for you - do NOT include in output)
- `template`: The structure to use for your output file
- `instruction`: Schema-specific guidance for this artifact type
- `skipped`/`warning`: present when the change declares skip_specs and this artifact must NOT be created - stop and pick another artifact
- `resolvedOutputPath`: Resolved path or pattern to write the artifact
- `dependencies`: Completed artifacts to read for context
- Read any completed dependency files for context - always re-read them from disk, even if you saw them earlier in the conversation (the user may have edited them)
- If the `instruction` field delegates creation to a specific skill or command, invoke it to produce the artifact instead of writing the file yourself, then verify the artifact file exists at `resolvedOutputPath`
- Otherwise create the artifact file using `template` as the structure and write it to `resolvedOutputPath`. If `resolvedOutputPath` is a glob, follow `instruction` to choose the concrete file path
- Apply `context` and `rules` as constraints - but do NOT copy them into the file
- Show brief progress: "Created <artifact-id>"
b. **Continue until every artifact in the required set exists (not just `apply.requires`)**
- After creating each artifact, re-run `openspec status --change "<name>" --json`
- The required set is `applyRequires` plus every artifact reachable from those by following the `requires` edges in `status --json` - walk them transitively (spec-driven closes over proposal, specs, design, tasks). Leave artifacts outside that set alone
- `status` is file-existence only, so an `applyRequires` artifact reading `done` does NOT mean its dependencies exist - writing `tasks.md` early marks `tasks` done while `specs` was never written. Use each artifact's `requires` edges, not its `status`, to build the required set: a `done` artifact still lists what it depends on
- An artifact already reading `status: "skipped"` is satisfied: the change declares `skip_specs` in `.openspec.yaml`, so its files must NOT exist. Never try to create one
- Create every artifact in the required set that is missing, then re-check - creating one can unblock others
- Skip one only when `status` already reports it `skipped`, or when its own `instruction` says it is conditional: run `openspec instructions <artifact-id> --change "<name>" --json` and skip only if its `instruction` field marks it optional (e.g. "create only if..."). Spec-driven's `design.md` qualifies; `specs` qualifies only via the `skipped` status above, never by your own judgment. Tell the user, and do not reconsider it
- Dependencies are enablers, not gates: if a required artifact is still `blocked` only because you skipped a conditional dependency, write it anyway
- Stop when every artifact in the required set is `done`, `skipped`, or was deliberately skipped
c. **If an artifact requires user input** (unclear context):
- Ask the user to clarify
- Then continue with creation
6. **Show final status**
```bash
openspec status --change "<name>"
```
**Output**
After completing all artifacts, summarize:
- Change name and location
- List of artifacts created with brief descriptions, plus any conditional artifact you skipped and why
- What's ready: "All artifacts needed for implementation are ready."
- Prompt: "The artifacts are ready for review. When you are ready, run `/opsx:apply` or ask me to apply this change."
**Artifact Creation Guidelines**
- Follow the `instruction` field from `openspec instructions` for each artifact type - it is the authoritative guidance, even for familiar artifact names
- If the `instruction` field directs you to use a specific skill or command to create the artifact, invoke it instead of writing the artifact directly
- The schema defines what each artifact should contain - follow it
- Read dependency artifacts for context before creating new ones
- Use `template` as the structure for your output file - fill in its sections
- **IMPORTANT**: `context` and `rules` are constraints for YOU, not content for the file
- Do NOT copy `<context>`, `<rules>`, `<project_context>` blocks into the artifact
- These guide what you write, but should never appear in the output
**Guardrails**
- The request that invoked this workflow authorizes planning only. Any implementation or apply instruction in that request does not carry forward. Do NOT implement the change, start the apply workflow, or edit project code during this workflow. After presenting the artifacts, stop and wait for a new user request to start the apply workflow
- Create every artifact the apply phase transitively depends on, not just the ids listed in `apply.requires`
- Always read dependency artifacts before creating a new one - re-read from disk, not from conversation memory (files may have changed since you last saw them)
- Ask about ambiguities that would materially change scope, externally observable behavior, compatibility, or acceptance criteria; for minor details, make reasonable assumptions and record them
- If a change with that name already exists, ask if user wants to continue it or create a new one
- Verify each artifact file exists after writing before proceeding to next
+262
View File
@@ -0,0 +1,262 @@
---
name: openspec-sync-specs
description: Sync delta specs from a change to main specs. Use when the user wants to update main specs with changes from a delta spec, without archiving the change.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Sync delta specs from a change to main specs.
This is an **agent-driven** operation - you will read delta specs and directly edit main specs to apply the changes. This allows intelligent merging (e.g., adding a scenario without copying the entire requirement).
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
When prompting, show changes that have delta specs (under `specs/` directory).
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:sync <other>`).
2. **Resolve change context**
Run:
```bash
openspec status --change "<name>" --json
```
The JSON includes `planningHome.root`. Main specs live under `<planningHome.root>/openspec/specs/` — use that (store-aware) root for every main-spec path below, not a hardcoded repo path. When a store is selected it points at the store, not the current repository.
3. **Find delta specs**
Use `artifactPaths.specs.existingOutputPaths` from the status JSON as the
only source of delta spec paths. If the `specs` entry is missing or
`existingOutputPaths` is empty, report that there are no delta specs to sync,
do not infer them from other artifacts, and stop without requesting artifact
instructions or writing a main spec.
Sync every path in `existingOutputPaths` unless the caller narrowed the set.
A caller narrows it by naming an explicit list of complete entries from
`existingOutputPaths` — copy those absolute values verbatim. Archive does
this inline, and a user can too (for example, by selecting the entry ending
in `/specs/billing/invoices/spec.md`).
Then sync only the named paths and leave the remaining delta specs untouched:
bulk archive excludes a delta whose implementation it could not find, and
syncing it anyway would write a main spec the caller deliberately withheld.
Carry that narrowed selection through step 4; never widen it back to the full
list. If a named path is not in `existingOutputPaths`, do not sync it —
report it and stop, rather than dropping it silently. If the named list is
empty, report that there is nothing to sync and stop without writing a main
spec.
Each delta spec file contains sections like:
- `## ADDED Requirements` - New requirements to add
- `## MODIFIED Requirements` - Changes to existing requirements
- `## REMOVED Requirements` - Requirements to remove
- `## RENAMED Requirements` - Requirements to rename (FROM:/TO: format)
If no delta specs found, inform user and stop.
4. **For each delta spec, apply changes to main specs**
Before the first main-spec write, obtain one current specs-rule snapshot:
- If archive invoked this workflow inline and supplied a valid snapshot from
`openspec instructions specs --change "<name>" --json`, reuse it and do not
fetch the same instructions again.
- Otherwise run that command once now with the same selected-root flags.
- If the direct lookup exits non-zero or returns invalid artifact-instruction
JSON, report the error and stop before writing any main spec. Do not treat the
failure as an absent rule set.
- A valid response with omitted `rules` means no artifact rules are configured
and the existing semantic merge continues.
Apply returned `rules` only to the content and form of the main specs produced
by this merge. Artifact rules are not operation guidance and cannot change
selected roots, delta paths, CLI checks, or workflow steps. Use their text as
constraints without copying it verbatim into a main spec or summary.
For each capability delta spec path selected in step 3 — the full `existingOutputPaths` list, or the narrowed subset when a caller supplied one (these may belong to a selected store, not the repo):
a. **Read the delta spec** to understand the intended changes
b. **Read the main spec** at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (may not exist yet)
c. **Apply changes intelligently**:
**ADDED Requirements:**
- If requirement doesn't exist in main spec → add it
- If requirement already exists → update it to match (treat as implicit MODIFIED)
**MODIFIED Requirements:**
- Find the requirement in main spec
- Apply the changes - this can be:
- Adding new scenarios the main spec does not have yet
- Modifying existing scenarios
- Changing the requirement description
- Preserve scenarios/content not mentioned in the delta
**REMOVED Requirements:**
- Remove the entire requirement block from main spec
- Retiring the capability. Delete the whole `spec.md` - and the directory once
nothing else is left in it - only when ALL of these hold:
1. removing the requirements *this run* left no requirement blocks;
2. the rest of the spec is well-formed (it still has a `## Purpose`);
3. the main spec was not already empty before this sync - if you removed
nothing, change nothing;
4. every other nonblank line in the whole file is accounted for as the
title, Purpose, Requirements header, or a canonical requirement's
statement, scenarios, or fenced examples;
5. the change's `.openspec.yaml` declares `retire_capabilities: true`;
6. the `spec.md` resolves inside the real specs root (do not follow a
capability-directory symlink to delete an external file).
If removing the selected requirements would leave no requirement blocks and
any retirement condition is not satisfied, do not modify the main spec. Stop
the sync for that capability, report the blocking condition, and tell the user
how to resolve it. Never write or leave an empty `## Requirements` section.
When only the marker is missing, say that too - it is the one thing the user
can add to make the retirement go through.
- Deleting the file also deletes its `## Purpose`; any other section blocks
retirement. Name Purpose when you report the retirement. Include a pasteable
`git checkout` only when the spec lived in the caller's checkout;
otherwise give checkout-scoped recovery guidance.
**RENAMED Requirements:**
- Find the FROM requirement, rename to TO
**`## Purpose` in the delta:**
- The main spec already has one and it is authoritative - leave it alone
(this is what `openspec archive` does; it warns and moves on)
d. **Create new main spec** if capability doesn't exist yet:
- Create `<planningHome.root>/openspec/specs/<capability-path>/spec.md`
- Add Purpose section: copy the delta's `## Purpose` body verbatim when it has one
(this is what `openspec archive` does); only write a brief TBD placeholder when it does not
- Add Requirements section with the ADDED requirements
- Follow the **Main Spec Format Reference** below
5. **Validate updated main specs**
Run `openspec validate --specs` with the same selected-root flags used earlier.
If validation fails, report the problems and do not claim the sync succeeded.
6. **Show summary**
After applying all changes, summarize:
- Which capabilities were updated
- What changes were made (requirements added/modified/removed/renamed)
- Any new main spec left with a TBD Purpose placeholder, so it gets written
now rather than lingering
- Any capability retired, naming the deleted `spec.md`, its Purpose, and
either a pasteable `git checkout` or checkout-scoped recovery guidance
**Delta Spec Format Reference**
```markdown
## Purpose
Only on a delta that introduces a brand-new capability. Seeds the new main spec.
## ADDED Requirements
### Requirement: New Feature
The system SHALL do something new.
#### Scenario: Basic case
- **WHEN** user does X
- **THEN** system does Y
## MODIFIED Requirements
### Requirement: Existing Feature
The system SHALL keep doing the existing thing, now also handling A.
#### Scenario: Scenario the main spec already has
- **WHEN** user does X
- **THEN** system does Y
#### Scenario: New scenario to add
- **WHEN** user does A
- **THEN** system does B
## REMOVED Requirements
### Requirement: Deprecated Feature
## RENAMED Requirements
- FROM: `### Requirement: Old Name`
- TO: `### Requirement: New Name`
```
**Main Spec Format Reference**
Main specs are what the delta merges INTO. They must never contain delta operation headers (`## ADDED/MODIFIED/REMOVED/RENAMED Requirements`) - after syncing, every requirement lives under a single `## Requirements` section:
```markdown
# <capability> Specification
## Purpose
Short description of what this capability does and why it exists.
## Requirements
### Requirement: New Feature
The system SHALL do something new.
#### Scenario: Basic case
- **WHEN** user does X
- **THEN** system does Y
```
**Key Principle: Intelligent Merging**
Unlike programmatic merging, you merge rather than overwrite:
- A MODIFIED block carries the whole requirement - body plus every scenario that survives the change. `openspec validate` and `openspec archive` both reject one that drops a scenario the main spec still has.
- Keep anything the delta does not mention, in the main spec's existing order
- Use your judgment to merge changes sensibly
**Output On Success**
```markdown
## Specs Synced: <change-name>
Updated main specs:
**<capability-1>**:
- Added requirement: "New Feature"
- Modified requirement: "Existing Feature" (added 1 scenario)
**<capability-2>**:
- Created new spec file
- Added requirement: "Another Feature"
Main specs are now updated. The change remains active - archive when implementation is complete.
```
**Guardrails**
- Read both delta and main specs before making changes
- Preserve existing content not mentioned in delta
- Never copy a delta file into a main spec as-is - merge its content so the main spec keeps the Main Spec Format Reference structure, with no delta operation headers
- If something is unclear, ask for clarification
- Show what you're changing as you go
- The operation should be idempotent - running twice should give same result
- Use only `artifactPaths.specs.existingOutputPaths`; never infer delta specs from unrelated artifacts
- Honor a caller-supplied subset of `existingOutputPaths`; never widen it back to the full list
- Fetch specs instructions once for direct sync, or reuse the archive-supplied snapshot inline
- Stop before every main-spec write on a non-zero or invalid JSON specs-instruction response
- Artifact rules constrain only the specs being written and are never copied into output files
@@ -0,0 +1,91 @@
---
name: openspec-update-change
description: Update an OpenSpec change by revising its existing planning artifacts and keeping them coherent with one another. Use when the user wants to revise a change's plan, fold new decisions into it, or reconcile its artifacts after an edit. Never edits code.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Revise a change's existing planning artifacts and keep them coherent. Never edit code.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
`/opsx:continue` is an expanded-profile workflow and may not be installed. Before suggesting it anywhere below, verify that it is available. If it is unavailable, `openspec status --change "<name>" --json` shows the next artifact and `openspec instructions "<artifact-id>" --change "<name>" --json` explains how to create it.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes sorted by most recently modified, and ask the user to select one
When prompting, present the top 3-4 most recently modified changes as options, showing:
- Change name
- Schema (from `schema` field if present, otherwise "spec-driven")
- Status (e.g., "0/5 tasks", "complete", "no tasks")
- How recently it was modified (from `lastModified` field)
Mark the most recently modified change as "(Recommended)" since it's likely what the user wants to update.
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:update <other>`).
2. **Get the change's artifacts**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to understand current state. The response includes:
- `schemaName`: The workflow schema being used (e.g., "spec-driven")
- `artifacts`: Array of artifacts with their status ("done", "skipped", "ready", "blocked")
- `isPlanningComplete`: Boolean indicating if all planning artifacts are complete. Older CLI versions expose the same value as `isComplete`.
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
The artifact ids and paths come from the active schema - do NOT assume them, and do NOT branch on hardcoded artifact names. Custom schemas must work unchanged.
The files to edit are `artifactPaths.<id>.existingOutputPaths` - the concrete files that exist on disk, already glob-expanded for glob artifacts (e.g. `specs/**/*.md`). Do NOT write to `resolvedOutputPath`: for a glob artifact it is still the glob pattern, not a real file.
3. **Understand the request**
- If the user asked for a specific revision ("the design now uses X"), that is the starting edit.
- If they only said "update" / "make this coherent", treat it as a coherence review: read the existing artifacts and check them against each other for contradictions, gaps, and duplication.
4. **Read and reconcile**
- Read the artifact(s) the request touches and the change's other existing artifacts.
- Apply the requested edit. Then check every other existing artifact against it - in ANY direction: an edit to a later artifact may require revising an earlier one, not only the other way around. Build order is a useful reading order, not a constraint on which artifacts may be revised.
- Note everything that is now inconsistent, missing, or contradictory.
- Revise only files that already exist (`existingOutputPaths`). Do NOT create artifacts that don't exist yet, and do NOT invent new files under a glob artifact - note them and point the user to `/opsx:continue` to create them.
- If the change is already coherent, say so and make no edits.
5. **Confirm and apply, one artifact at a time**
- Show each proposed revision and why. Write only after the user confirms.
- If the user rejects a revision, do not write it - leave that artifact unchanged.
- When a substantial rewrite is needed, get that artifact's rules and template first:
```bash
openspec instructions "<artifact-id>" --change "<name>" --json
```
6. **Point to the next step (guidance only - NEVER act on it)**
- Artifacts still missing -> suggest `/opsx:continue` to create them.
- Change already implemented (tasks checked off / already applied) -> the code may no longer match the revised plan; suggest `/opsx:apply` to carry the delta into code.
- Everything done and implemented -> suggest `/opsx:archive`.
**Output**
After each invocation, show:
- Which artifacts were revised (and which proposed revisions were rejected)
- Anything deferred to `/opsx:continue` (not-yet-created artifacts or files)
- Where the change stands and the recommended next command
**Guardrails**
- Planning artifacts only - NEVER edit implementation code. If the revised plan implies code changes, stop and point to `/opsx:apply`.
- Use the artifact ids and paths reported by `openspec status`; never branch on hardcoded artifact names.
- Edit only the concrete files in `existingOutputPaths`; never write to a glob `resolvedOutputPath`.
- Do not advance the build frontier: no new artifacts, no new files under glob artifacts - that is `/opsx:continue`'s job.
- Confirm every edit with the user before writing.
- If the request changes the change's *intent* rather than refining it, first verify whether the expanded-profile `/opsx:new` workflow is available. If it is, recommend starting fresh with `/opsx:new` (the "Update vs. Start Fresh" heuristic). If it is unavailable, ask for a distinct unused change name and recommend `openspec new change "<new-change-name>"` instead.
+26
View File
@@ -14,7 +14,33 @@ backups
*.pfx *.pfx
*.jks *.jks
.venv .venv
venv
__pycache__ __pycache__
*.pyc
*.pyo
.pytest_cache
.ruff_cache
.mypy_cache
.ty_cache
node_modules node_modules
evidence evidence
openspec
student-tests
tests
scripts
examples
contracts
infra
*.md
!README.md
Makefile
Dockerfile*
compose*.yaml
.dockerignore
.gitignore
.DS_Store .DS_Store
lab.json
requirements-dev.txt
.claude
.verification/
+54
View File
@@ -9,3 +9,57 @@ DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey: # Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
S3_ENDPOINT=http://s3:9000
S3_REGION=us-east-1
S3_BUCKET=messenger-attachments
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
# Solution-specific: published LB ports, replica limits, workers and pool per replica,
# LB queue per replica, drain window before closing the listener on SIGTERM, PostgreSQL
# limits, load generator limits (profile bench), cursor key file. API_PORT/SHORTTTL_PORT of lab2 are gone:
# only the load balancer is published.
LB_HTTP_PORT=8080
LB_STATS_PORT=8404
API_CPUS=1
API_MEMORY=512m
WEB_CONCURRENCY=2
DB_POOL_MAX_SIZE=8
API_SERVER_MAXCONN=64
DRAIN_SECONDS=2
PG_MAX_CONNECTIONS=200
PG_CPUS=2
PG_MEMORY=1g
K6_CPUS=2
K6_MEMORY=512m
CURSOR_SECRET_FILE=/run/secrets/cursor_secret
# Solution-specific for lab4: S3 client timeouts (connect + read, two attempts stay below
# REQUEST_TIMEOUT_SECONDS=8), parallel image checks per worker, RustFS memory limit.
S3_CONNECT_TIMEOUT_SECONDS=1
S3_READ_TIMEOUT_SECONDS=2
IMAGE_CHECK_CONCURRENCY=2
S3_MEMORY=512m
# Solution-specific for lab4 grade 4: upload bodies buffered at once per worker, grace period
# before an abandoned pending upload is swept, sweep interval and rows per sweep pass.
UPLOAD_CONCURRENCY=3
UPLOAD_GRACE_SECONDS=900
UPLOAD_SWEEP_INTERVAL_SECONDS=30
UPLOAD_SWEEP_BATCH=100
# Fixed glibc mmap threshold for the API: buffers above 128 KiB are returned to the system on free.
MALLOC_MMAP_THRESHOLD_=131072
# Minimal client transfer rate, bytes/s: every body byte moved in either direction extends the
# request deadline by 1/rate, so slow but steady uploads and downloads outlive REQUEST_TIMEOUT_SECONDS.
MIN_TRANSFER_RATE_BYTES=131072
# Solution-specific for lab4 track 2: limits of the one-off reconciler (profile reconcile), its S3
# read timeout for listing pages, optional CPU pinning of the measured services (empty = no pinning)
# and the CPU limit of RustFS (make reconcile-cost raises it with its memory for the 100k point).
RECONCILE_CPUS=1
RECONCILE_MEMORY=256m
RECONCILE_S3_READ_TIMEOUT_SECONDS=10
RECONCILE_CPUSET=
S3_CPUSET=
PG_CPUSET=
S3_CPUS=1
+1 -1
View File
@@ -1,4 +1,4 @@
## Лабораторная 2 ## Лабораторная 4
Автор / группа: TODO Автор / группа: TODO
Целевая оценка (3/4/5): TODO Целевая оценка (3/4/5): TODO
+101
View File
@@ -0,0 +1,101 @@
name: Application
on: [push, pull_request]
permissions:
contents: read
jobs:
quality:
name: Линт, типы и собственные тесты
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
with:
enable-cache: true
- run: uv sync --locked --group dev
- run: make lint VENV=.venv
- run: make student-test VENV=.venv
acceptance:
name: Стенд Compose с RustFS за HAProxy, smoke, вложения, отказ реплики, рестарт и отказы зависимостей
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
with:
enable-cache: true
- run: uv sync --locked --group dev
- name: Поднять стенд из чистого clone
run: make up
- name: Публичный smoke через балансировщик
run: make test
- name: Вложение после остановки принявшей его реплики и при недоступном S3
run: |
make scenario SCENARIO=failover-upload
make scenario SCENARIO=s3-outage
- name: Приватный bucket, ограниченный ключ API и повторный bootstrap
run: make s3-check
- name: Негативный suite вложений, компенсация, уборка после grace, обрыв и зависание загрузки
run: make upload-faults
- name: Несколько реплик на одном токене и гонка одного ключа
run: make lb-check
- name: Повтор неподтверждённой записи после потерянного ответа и убитой реплики
run: make retry-check
- name: Остановка реплики под запросами, затем возврат
run: |
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
docker compose ps -a api
make restore
- name: Рестарт LB, API и хранилищ с сохранением volumes
run: make test-persistence ACTION_SCRIPT=scripts/restart.sh
- name: Отказы Valkey и PostgreSQL, гонка и rollback
run: make test-faults
- name: Лимиты, порты, секреты, volumes и сети по политике
run: make limits
- name: Время обнаружения отказа балансировщиком (SIGTERM, kill, pause, PostgreSQL)
run: make detection
- name: Наружу опубликован только балансировщик
run: |
docker compose ps --format '{{.Service}} {{.Ports}}' | awk '$2 != "" && $1 != "lb" {print; bad=1} END {exit bad}'
- name: Процессы работают не от root
run: |
for id in $(docker compose ps -q); do
docker top "$id" -eo pid,uid,comm | awk 'NR>1 && $2==0 {print; bad=1} END {exit bad}' || exit 1
done
- name: Одноразовые сервисы не от root
run: |
for service in migrate s3-init; do
user=$(docker inspect -f '{{.Config.User}}' "$(docker compose ps -aq "$service")")
echo "$service: $user"
case "${user%%:*}" in ""|0|root) exit 1 ;; esac
done
- name: Корректная остановка реплик API по SIGTERM
run: |
start=$(date +%s)
docker compose kill --signal SIGTERM api
codes=$(docker wait $(docker compose ps -aq api) | sort -u)
elapsed=$(( $(date +%s) - start ))
echo "остановка за ${elapsed} с, коды завершения ${codes}"
test "$elapsed" -le 10 && test "$codes" = "0"
- name: Журналы стенда
if: always()
run: docker compose logs --tail 80
+10
View File
@@ -30,3 +30,13 @@ coverage/
.vscode/ .vscode/
# Keep small, sanitized evidence summaries in evidence/ under version control. # Keep small, sanitized evidence summaries in evidence/ under version control.
# Inspect git diff --cached: ignore rules are not a secret scanner. # Inspect git diff --cached: ignore rules are not a secret scanner.
# Тулчейн студенческой реализации
.pytest_cache/
.ruff_cache/
.ty_cache/
# Локальные настройки агента
.claude/settings.local.json
.verification/
+82
View File
@@ -0,0 +1,82 @@
## Что это за репозиторий
Skeleton лабораторной 4 курса «Мессенджер под нагрузкой» (МТУСИ, 7 независимых лаб) и решение в ветке `solution/bpi2303-kostylev`. Skeleton даёт задание (`README.md`), нормативный контракт (`contracts/`) и публичные black-box проверки (`tests/`). Реализация API, Dockerfile, compose-стенд, конфигурация балансировщика и S3, миграции, хуки отказов и собственные тесты — студенческие; решение лабы 4 построено переносом решения лабы 3.
Лаба 4: вложения-изображения в сообщениях. Метаданные вложений живут в PostgreSQL, байты — в приватном bucket S3-совместимого хранилища **RustFS** внутри стенда, а не в БД и не на диске реплики. `POST /api/v1/chats/{chat_id}/attachments` принимает сырые байты одного PNG/JPEG (≤10 MiB, ≤40 млн пикселей, тип по содержимому) и отвечает `201 ready` только после записи объекта и фиксации метаданных; скачивание идёт только через API участникам чата. Сообщение несёт `attachment_ids`. Остаётся всё из лабы 3: HAProxy перед ≥2 репликами, `X-Instance-Id`, обязательный `Idempotency-Key`, хуки остановки реплики и рестарта стенда.
## Команды
```sh
make check # offline: структура skeleton, JSON, локальные $ref, синтаксис Python, версия 1.4.0
make validate # полная валидация OpenAPI 3.1: openapi-spec-validator из .venv (make install); PYTHON= — свой интерпретатор
make install # uv sync --group dev
make lint # ruff check + ruff format --check + ty check; обязательно перед commit
make student-test # собственные тесты (поднимают одноразовые PostgreSQL, Valkey и RustFS в Docker)
make student-test PYTEST_ARGS='-v -k drain' # выборка тестов с именами
make secrets # локальные secrets/* (включая root и API-ключи S3) и .env из .env.example
make up # docker compose up -d --build --wait: postgres, valkey, s3 → migrate, s3-init → api ×2 → lb; затем ждёт все реплики UP в LB
make test # публичный smoke через LB (BASE_URL по умолчанию — docker compose port lb 8080, без стенда http://localhost:8080)
make test-persistence ACTION_SCRIPT=scripts/restart.sh # рестарт lb, api, postgres, valkey, s3 с сохранением volumes
make test-failover ACTION_SCRIPT=scripts/stop-one.sh # остановка реплики TARGET_INSTANCE_ID под запросами
make test-faults # тесты стенда через LB (make faults-evidence → evidence/faults.txt)
make restore # вернуть остановленные реплики и дождаться их в ротации LB
make limits # лимиты, порты, секреты, volumes и сети всех сервисов против политики → evidence/limits.txt
make lb-check # один токен на обеих репликах и 20 гонок одного Idempotency-Key через LB
make retry-check # потерянный ответ (запись ждёт блокировку чата) на живой и на убитой реплике, повтор тем же ключом
make scenario SCENARIO=failover-upload # upload → стоп обслужившей реплики → download через другую, SHA-256
make scenario SCENARIO=s3-outage # S3 остановлен: upload/download 503, тексты работают
make s3-check # приватный bucket вне API, ограниченные ключи API и сверки, консоль, повторный bootstrap
make upload-faults # «4»: негативный suite через LB, компенсация, уборка после grace, обрыв и зависание тела
make upload-memory ARGS='--out evidence/upload-memory/run1' # «4»: пик памяти реплик и LB при 10/20/40 загрузках 1 и 10 MiB
make load-evidence # tests/load.py ×3 клиентом-контейнером в сети edge → evidence/load.txt, evidence/load/
make overload-leftovers # после upload-memory: намерения от 503, сверка и уборка после grace → evidence/overload-leftovers.txt
make clean-clone-evidence # блок «Запуск из чистого clone» REPORT.md в свежем клоне рядом → evidence/clean-clone.txt
make reconcile ARGS="--apply" # трек 2: сверка PostgreSQL ↔ S3 (без ARGS — dry-run), one-off сервис reconciler
make reconcile-demo # трек 2: внедрение ошибок, dry-run/apply/повтор, безопасность, нагрузка → evidence/reconcile/
make reconcile-cost # трек 2: стоимость обхода 1k/10k/30k/100k × 3 (чистый стенд) → evidence/reconcile/cost/
make down # остановка без удаления данных (с профилями bench, tools, reconcile)
```
`tests/*.py` и `scripts/*.py` — только stdlib, синтаксис Python 3.9 (`make check` разбирает их системным `python3`). Проверки сами не поднимают стенд. Хуки `scripts/stop-one.sh` и `scripts/restart.sh` запускаются тестом без shell, поэтому исполняемые и с shebang. Данные не удаляются автоматически: каждый прогон smoke создаёт новых синтетических пользователей и вложения.
## Архитектура skeleton
- `lab.json` — единственный источник номера лабы (4). От него зависят `scripts/check_contract.py` (`info.version == 1.4.0`, путь `/api/v1/attachments/{attachment_id}`, `X-Instance-Id` во всех ответах) и `tests/client.py` (ветки `LAB>=4`: upload `tests/fixtures/sample.png`, сообщение с `attachment_ids`, скачивание вторым участником, `404` постороннему).
- `tests/client.py` шлёт `Accept: application/json` на каждый запрос, включая скачивание картинки, и считает любой `30x` ошибкой: content negotiation и redirect на S3 не делаются.
- `tests/resilience.py`: перед хуком загружается `sample.png`; после `persistence` метаданные и `/content` запрашиваются **один раз** без повторов с таймаутом 2 с, поэтому хук рестарта ждёт здоровья S3. В `failover` `TARGET_INSTANCE_ID` — реплика, обслужившая первое **сообщение**, а не upload; остановку реплики, принявшей upload, доказывает `scripts/lab4_scenarios.py failover-upload`.
- `contracts/infrastructure.json` — роли `api` (≥2, без локального состояния), `postgres` (метаданные, persistent volume), `load_balancer`, `s3` (приватный bucket, persistent volume), опциональный `redis_or_valkey`. `compose.example.yaml` — только worksheet ролей.
## Ключевые правила контракта (lab4)
- Upload: `POST /api/v1/chats/{chat_id}/attachments?filename=…`, тело — сырые байты (не multipart), `Content-Type: image/png|image/jpeg`. Сначала `401`, затем объявленный `Content-Length` (`413`). Дедлайн запроса продлевается переданными байтами тела по `MIN_TRANSFER_RATE_BYTES` (128 KiB/s): медленный, но идущий upload и скачивание не обрываются. `>10485760` байт или `>40 000 000` пикселей — `413`; тип по содержимому не совпал, битый файл, иной `Content-Type` — `415`; `filename` пустой, пробельный, длиннее 255 code points или с `/`, `\`, NUL, управляющими символами — `400`. Object key назначает сервер. `201 Attachment` (`status=ready`, `variants=[]`, `error_code=null`, lowercase `sha256`, `size_bytes` оригинала) — только после записи в S3 и `COMMIT` метаданных. Кода `409` у upload нет.
- `GET /api/v1/attachments/{id}` и `/content?variant=original` — только участникам чата вложения, посторонний и неизвестный id — `404`. Скачивание отдаёт исходные байты с истинным `Content-Type`, без redirect, с `X-Request-Id` и `X-Instance-Id`.
- `Message.attachment_ids` обязателен в каждом ответе (`[]` у текстовых, в том числе в replay и истории). `CreateMessage.attachment_ids` — 0–10 уникальных UUID (иначе `400`), только вложения этого чата (иначе `404`); `text=""` допустим только с вложением. Отсутствующий `attachment_ids` и `[]` — одно тело для `Idempotency-Key`.
- Сохраняются правила lab3 (`X-Instance-Id`, `Idempotency-Key`, LB не повторяет `POST /users`, `/chats`, upload), lab2 (`Basic` только на входе, `Bearer` дальше, `WWW-Authenticate` по пути, logout `204`, TTL) и lab1 (формат ошибки, посторонний → `404`, keyset `(created_at, id)`, время RFC 3339 с `Z`).
## Правила работы с репозиторием
- Поздние исправления ЛР3 перенесены из immutable snapshot `ba78cf6da439a4ec60fd59389583b38285b0b2b1`; диапазон от `f591fa64e525679a31c9db7f942d36c789e1a12f` и применимость evidence описаны в `evidence/forward-port.md`.
- Только свой Compose project `mtusi2026lab4` либо `hlpipe-lab4-*` под разрешённым working_dir; HTTP 18084, остальные host-порты 24000–24999. `NO_PROXY` и `no_proxy`: localhost,127.0.0.1,::1. Чужие контейнеры не останавливать, WIP и volumes сохранять. Совместный Docker допустим; эксклюзивность только в финальном release.
- Runner назначает одного агента переноса и реализации 3→4→5 с evidence, затем review → отдельный fix при major → re-review, отдельный report и одного агента нагрузки/публикации. Грейды одной лабы последовательны; review/report предыдущей могут идти параллельно реализации следующей. Самостоятельных агентов, push и Telegram нет; runner/state не менять. Публикация только в назначенной фазе PUBLICATION в private MTUCI-Labs.
- Исправлять критичное: major — доказанная угроза безопасности, данным, доступности или неработающий обязательный сценарий. Прочие minor остаются в findings и ограничениях REPORT и не блокируют этап. Пригодные измерения сохраняются; повторяется только затронутая или признанная непригодной серия с указанной причиной и фоном нагрузки.
- Не менять и не ослаблять `contracts/`, `tests/`, `README.md`, `COURSE.md`, рубрику. Расхождение оформляется вопросом преподавателю.
- Код — `src/messenger/`, миграции — `alembic/versions/NNNN_*.py`, конфигурация LB, нагрузки и S3 — `infra/`, хуки и обвязка стенда — `scripts/`, собственные тесты — `student-tests/`, обезличенные доказательства — `evidence/`.
- PostgreSQL — SQLAlchemy 2 async с asyncpg как драйвером; ORM-модели внутри зон, Alembic принимает legacy-схему только после сверки журнала и каталога.
- Без комментариев в коде; отступы — пробелы (`[tool.ruff.format]`); Python 3.14 без `from __future__ import annotations` и `TYPE_CHECKING`.
- Порты реплик API и S3 наружу не публикуются: только LB. Нет `container_name`, `hostname` и общего `INSTANCE_ID` у сервиса `api`. Root-ключи S3 есть только у `s3`, `s3-init` и `s3-cli`; API получает отдельный ключ только на объекты bucket (`Get/Put/DeleteObject`), сверка — свой ключ (`ListBucket`, `Get/DeleteObject`). `s3-init` и `s3-cli` работают от uid 65534, `migrate` и `reconciler` — с `init: true`.
- `REPORT.md`: команды из чистого clone, архитектура, таблица «критерий → команда → наблюдение → файл», таблица отказов. PASS без команды не доказательство; для отказов — какой контейнер реально остановлен (`docker ps -a`).
- Отчётный этап меняет только `REPORT.md`, `AGENTS.md` и `evidence/**/*.md`; код, конфиги, OpenSpec и сырые измерения не меняет. Docker и нагрузку не запускает. `make student-test` поднимает зависимости: сохранённый результат или collection не выдаётся за новый полный прогон.
- Ветка `solution/bpi2303-kostylev`, коммиты на английском, перед каждым — `make lint`. Никогда не коммитить `.env`, `secrets/`, токены, ключи S3, volumes, дампы.
## Уровни оценки
Цель — **«5»**, трек 2 «Согласованность двух хранилищ (reconciler)». На snapshot `7a0ebac2aa9e1d8602054ca4c7c1b0c23d3bfad4` ревью прошло без открытых major. Финальная validation на `5e49502ae78fa48b992c97418e490666c699726d` с идентичным runtime закрыла динамический долг обоих PNG-фиксов: `evidence/load-validation.md`. Сохранён minor Unicode-surrogate → HTTP 500 (`src/messenger/messaging/entrypoints/schemas.py:35`, `evidence/forward-final-2026-11-06/known-minor-http.txt`).
Таблица критериев — в `REPORT.md`, команды и SHA — в `evidence/README.md`. Новая серия использует отдельный разрешённый project со свежими volumes; `reset-data`, `down -v`, reset/clean и удаление пользовательских данных запрещены. После Docker-этапа остановить только свои стенды, сохранив volumes. При изменениях проверять применимость по diff: повторять лишь затронутые/непригодные серии с причиной и фоном. Документационный коммит измерений не обесценивает.
Исторические upload-memory и конкурентные upload до PNG-фиксов не доказывают RAM/производительность нового инспектора. Финальная validation выполнена: 721 собственная проверка (719 + адресный повтор 2 setup errors RustFS), 7 stack, PNG/JPEG через LB/RustFS; матрица 3 слота × 1/10 MiB × 10/20/40 клиентов ×3: 840/840×201, API ≤440 MiB, OOM 0; 10 PNG 40 MP RGBA16: 10×201, API ≤270,828 MiB; upload при сверке: 3341×201, 67 проходов, lost_ready=0. Сырьё — `evidence/png-final-2026-11-06/`. Эти пригодные серии повторять без нового изменения/обоснованной непригодности запрещено. Контроль 64/3 оставлен историческим. 27 пригодных cost-замеров reconciler на `3402fd69d4922e500035a204d957be2cccaeb85a` сохраняются: измеряемый путь не менялся. Отрицательный вывод о линейности времени и разброс deep 100k не скрываются.
- **3** — RustFS и приватный bucket в общем стенде, метаданные в PostgreSQL; upload `201 ready` после записи и `COMMIT`, побайтное скачивание вторым участником, `404` постороннему, `403` анонимному GET в S3; лимиты 10 MiB / 40 MP, тип по содержимому, опасные `filename`; upload → стоп обслужившей реплики → download через другую; restart с volumes; недоступный S3 → `503` при работающих текстах; минимум lab3 сохранён.
- **4** — негативные тесты файлов без `500`; компенсация сбоя между S3 и БД под блокировкой строки и фоновая уборка намерений `pending` старше `UPLOAD_GRACE_SECONDS` (`SKIP LOCKED`, объект раньше строки); слоты загрузки на воркер, ограниченные буферы сокетов LB и API, фиксированный порог `mmap` glibc и замер ≥10 одновременных загрузок; идемпотентный bootstrap, ограниченные ключи API, закрытая консоль, проверка приватности вне API.
- **5** — трек 2: one-off `reconciler` (`python -m messenger.attachments.entrypoints.reconcile`, по умолчанию `--dry-run`) сливает отсортированный листинг bucket и keyset-страницы `attachments` по `object_key` (O(N) сравнений, O(страница) памяти; линейность времени в замерах не подтвердилась); объект без строки старше grace + skew удаляется после повторной проверки строки, готовые строки без объекта или с другими байтами уходят в карантин (миграция `0004`, `/content` → `404`) и не удаляются, `pending` старше grace удаляется (объект, затем строка), карантин снимается только после совпавшего SHA-256 объекта (и без `--deep`); удаление намерения — `audit.attachment_discarded reason=reconcile`; SQL-страниц `⌈N/страница⌉`; один проход за раз (`pg_try_advisory_lock`); повторный `--apply` — 0 действий; стоимость обхода 1k/10k/30k/100k объектов с закреплёнными CPU и отбраковкой прогонов с чужой нагрузкой (`evidence/reconcile/`).
+51
View File
@@ -0,0 +1,51 @@
FROM python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d AS builder
COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
ENV UV_COMPILE_BYTECODE=1 \
UV_LINK_MODE=copy \
UV_PYTHON_DOWNLOADS=never \
UV_PROJECT_ENVIRONMENT=/opt/venv
WORKDIR /build
COPY pyproject.toml uv.lock README.md ./
RUN uv sync --locked --no-dev --no-install-project --no-editable
COPY src ./src
RUN uv sync --locked --no-dev --no-editable
FROM python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d AS runtime
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PATH="/opt/venv/bin:$PATH" \
HOST=0.0.0.0 \
PORT=8080 \
LOG_LEVEL=INFO \
WEB_CONCURRENCY=2 \
REQUEST_TIMEOUT_SECONDS=8 \
DRAIN_SECONDS=2 \
SHUTDOWN_TIMEOUT_SECONDS=6
RUN groupadd --system --gid 10001 messenger \
&& useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
COPY --from=builder --chown=root:root /opt/venv /opt/venv
WORKDIR /app
COPY --chown=root:root alembic.ini ./
COPY --chown=root:root alembic ./alembic
USER 10001:10001
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
CMD ["python", "-c", "import urllib.request,sys;sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health/live',timeout=2).status==200 else 1)"]
CMD ["python", "-m", "gunicorn", \
"--config", "python:messenger.gunicorn_conf", \
"messenger.main:create_app()"]
+1 -1
View File
@@ -1,4 +1,4 @@
# Памятка преподавателю — лабораторная 2 # Памятка преподавателю — лабораторная 4
## Что подготовлено ## Что подготовлено
+187 -2
View File
@@ -1,17 +1,53 @@
PYTHON ?= python3 PYTHON ?= python3
BASE_URL ?= http://localhost:8080 BASE_URL ?= http://$(shell docker compose port lb 8080 2>/dev/null || echo localhost:8080)
CA_FILE ?= CA_FILE ?=
ACTION_SCRIPT ?= ACTION_SCRIPT ?=
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",) TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
.PHONY: help check validate test load test-persistence VENV ?= .venv
UV ?= uv
COMPOSE ?= docker compose
RATE ?= 100
FROM_REV ?=
SHIFT ?= 0
SCRIPTS_PY = scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py \
scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py
SCENARIO ?= failover-upload
PSQL = $(COMPOSE) exec -T postgres psql -U messenger -d messenger -X -q
PYTEST_ARGS ?=
.PHONY: help check validate test load test-persistence test-failover \
install lint format typecheck student-test \
secrets up down reset-data migrate ps logs restore lb-check failover-evidence stack-evidence \
persistence-evidence storage-evidence cache-evidence test-faults faults-evidence seed explain retry-check \
limits detection failover-each rolling-update level4 \
scenario s3-check attachments-evidence upload-faults upload-memory \
reconcile reconcile-demo reconcile-cost load-evidence clean-clone-evidence overload-leftovers
.NOTPARALLEL:
help: help:
@echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes" @echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes"
@echo "install/lint/typecheck/student-test: студенческий тулчейн"
@echo "secrets/up/down/migrate/ps/logs: стенд Docker Compose; reset-data: намеренное удаление volumes"
@echo "restore: вернуть остановленные реплики; lb-check: реплики на одном токене и 20 гонок одного ключа"
@echo "retry-check: потерянный ответ на живой и на убитой реплике, повтор тем же ключом"
@echo "level4 = limits, lb-check, detection, failover-each, rolling-update [FROM_REV=<коммит>]; RATE=$(RATE) rps"
@echo "scenario SCENARIO=failover-upload|s3-outage → evidence/<сценарий>.json; s3-check; attachments-evidence"
@echo "upload-faults: негативный suite, компенсация, уборка, обрыв и зависание загрузки против стенда"
@echo "upload-memory ARGS='--out evidence/upload-memory/runN': пик памяти реплик при 10/20/40 загрузках 1 и 10 MiB"
@echo "load-evidence: tests/load.py ×3 клиентом-контейнером в сети edge стенда → evidence/load.txt, evidence/load/"
@echo "overload-leftovers: после upload-memory — намерения от 503, сверка и уборка после grace → evidence/overload-leftovers.txt"
@echo "clean-clone-evidence: блок «Запуск из чистого clone» в свежем клоне рядом → evidence/clean-clone.txt"
@echo "reconcile ARGS='--apply --deep': сверка PostgreSQL ↔ S3 (без ARGS — dry-run); трек 2:"
@echo "reconcile-demo: внедрение ошибок, dry-run/apply/повтор, безопасность, нагрузка → evidence/reconcile/"
@echo "reconcile-cost: стоимость обхода 1k/10k/30k/100k объектов × 3 прогона на чистом стенде → evidence/reconcile/cost/"
@echo "BASE_URL=$(BASE_URL) (порт lb из docker compose); failover-evidence SHIFT=1 — сдвиг round robin на вторую реплику"
check: check:
$(PYTHON) scripts/check_contract.py $(PYTHON) scripts/check_contract.py
validate: PYTHON := $(if $(wildcard $(VENV)/bin/python),$(VENV)/bin/python,$(PYTHON))
validate: validate:
$(PYTHON) scripts/validate_openapi.py $(PYTHON) scripts/validate_openapi.py
@@ -24,3 +60,152 @@ load:
test-persistence: test-persistence:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable restart hook"; exit 2) @test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable restart hook"; exit 2)
$(PYTHON) tests/resilience.py --mode persistence --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS) $(PYTHON) tests/resilience.py --mode persistence --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
test-failover:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
install:
$(UV) sync --group dev
lint:
$(VENV)/bin/ruff check src student-tests $(SCRIPTS_PY)
$(VENV)/bin/ruff format --check src student-tests $(SCRIPTS_PY)
$(VENV)/bin/ty check src student-tests alembic
format:
$(VENV)/bin/ruff check --fix src student-tests $(SCRIPTS_PY)
$(VENV)/bin/ruff format src student-tests $(SCRIPTS_PY)
typecheck:
$(VENV)/bin/ty check src student-tests alembic
student-test:
$(VENV)/bin/pytest $(PYTEST_ARGS)
secrets:
sh scripts/secrets.sh
up: secrets
$(COMPOSE) up -d --build --wait
scripts/lb-wait.sh
down:
$(COMPOSE) --profile bench --profile tools --profile reconcile down
reset-data:
@echo "Удаляются volumes с учебными данными стенда"
$(COMPOSE) --profile bench --profile tools --profile reconcile down -v --remove-orphans
migrate:
$(COMPOSE) run --rm migrate
ps:
$(COMPOSE) ps
logs:
$(COMPOSE) logs --tail 200
restore:
scripts/restore.sh
lb-check:
$(PYTHON) scripts/probe.py check --rounds 20 --out evidence/lb-check.json
retry-check:
$(PYTHON) scripts/probe.py retry --out evidence/retry.json
limits:
$(PYTHON) scripts/limits.py --out evidence/limits.txt
detection:
rm -rf evidence/detection
for run in 1 2 3; do sh scripts/detection.sh evidence/detection/run$$run || exit 1; done
failover-each:
RATE=$(RATE) sh scripts/failover-each.sh evidence/failover-each
rolling-update:
ifneq ($(FROM_REV),)
git archive "$(FROM_REV)" | docker build -q -t "mtusi-messenger:$(FROM_REV)" -
API_IMAGE="mtusi-messenger:$(FROM_REV)" $(COMPOSE) up -d --no-deps --force-recreate --wait api
scripts/lb-wait.sh
endif
$(COMPOSE) build -q api
RATE=$(RATE) $(PYTHON) scripts/probe.py load --expect replace --duration 90 \
--out evidence/rolling$(if $(FROM_REV),-from-$(FROM_REV)) \
-- scripts/rolling-update.sh
level4: limits lb-check detection failover-each rolling-update
scenario:
$(PYTHON) scripts/lab4_scenarios.py $(SCENARIO) --base-url "$(BASE_URL)" --restore --output evidence/$(SCENARIO).json
scripts/lb-wait.sh
s3-check:
sh scripts/s3-check.sh > evidence/s3-security.txt; status=$$?; cat evidence/s3-security.txt; exit $$status
attachments-evidence:
sh scripts/evidence-attachments.sh
upload-faults:
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD) BASE_URL=$(BASE_URL)"; \
BASE_URL="$(BASE_URL)" $(VENV)/bin/pytest -m stack -v -s --no-header --tb=short \
-p no:cacheprovider student-tests/stack/test_attachment_faults.py; } > evidence/upload-faults.txt; \
status=$$?; cat evidence/upload-faults.txt; exit $$status
upload-memory:
$(PYTHON) scripts/upload_memory.py --base-url "$(BASE_URL)" $(ARGS)
load-evidence:
sh scripts/load-evidence.sh
overload-leftovers:
sh scripts/overload-leftovers.sh
clean-clone-evidence:
sh scripts/clean-clone.sh
reconcile:
$(COMPOSE) run --rm --no-deps reconciler $(ARGS)
reconcile-demo:
$(PYTHON) scripts/reconcile_track.py demo --base-url "$(BASE_URL)" $(ARGS)
reconcile-cost:
$(PYTHON) scripts/reconcile_track.py cost $(ARGS)
failover-evidence:
sh scripts/failover-evidence.sh "$(BASE_URL)" "$(SHIFT)"
stack-evidence:
sh scripts/evidence-stack.sh
persistence-evidence:
sh scripts/persistence-evidence.sh "$(BASE_URL)" > evidence/persistence.txt
cat evidence/persistence.txt
storage-evidence:
sh scripts/evidence-storage.sh
cache-evidence:
sh scripts/evidence-cache.sh > evidence/cache.txt
cat evidence/cache.txt
test-faults:
BASE_URL="$(BASE_URL)" $(VENV)/bin/pytest -m stack -v -s student-tests/stack
faults-evidence:
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD)"; echo "\$$ make test-faults"; \
$(MAKE) --no-print-directory test-faults 2>&1; } | sed 's#$(CURDIR)#<repo>#g' > evidence/faults.txt
cat evidence/faults.txt
grep -qE '^=+ [0-9]+ passed in' evidence/faults.txt
seed:
$(PSQL) -f - < scripts/sql/seed-history.sql
explain: seed
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD)"; \
echo "\$$ make explain # scripts/sql/explain-history.sql"; \
$(PSQL) -f - < scripts/sql/explain-history.sql; } > evidence/explain-history.txt
cat evidence/explain-history.txt
+33 -33
View File
@@ -1,6 +1,6 @@
# Лабораторная 2. PostgreSQL, сессии и Redis/Valkey # Лабораторная 4. S3-хранилище и вложения
Перенести состояние в постоянное хранилище и связать запросы с аутентифицированным пользователем. Подтверждённые сообщения и действующие сессии переживают перезапуск. Разделить метаданные и бинарное содержимое, добавить изображения в сообщения и сохранить доступ к ним при переключении API-реплики.
Это **skeleton**, а не готовый сервер. Здесь есть задание, контракт и публичные проверки. Реализацию приложения, Dockerfile и требуемую инфраструктуру пишет студент. Язык и framework свободные; UI не обязателен. Это **skeleton**, а не готовый сервер. Здесь есть задание, контракт и публичные проверки. Реализацию приложения, Dockerfile и требуемую инфраструктуру пишет студент. Язык и framework свободные; UI не обязателен.
@@ -8,41 +8,41 @@
## Откуда начинаем ## Откуда начинаем
Перенесите собственный минимум lab1. Изменения API: password при регистрации, Basic для создания сессии, Bearer вместо X-User-Id для бизнес-запросов. Перенесите минимум lab3. Дополняются поля сообщения и endpoint вложений; прежние текстовые POST остаются валидными.
**Архитектура:** Клиент → API → PostgreSQL; на 4 появляется Redis ИЛИ Valkey для сессий/кэша. **Архитектура:** Клиент → LB → API × 2+ → PostgreSQL (метаданные) + приватное S3-хранилище (байты).
## Что сделать ## Что сделать
1. Опишите таблицы, ограничения, индексы и транзакционные границы до написания запросов. Добавьте версионируемые миграции. 1. Выберите один S3-совместимый сервер: MinIO, SeaweedFS или RustFS. Закрепите проверенную версию образа и опишите создание приватного bucket.
2. Перенесите пользователей, чаты, членство и сообщения в PostgreSQL. Не используйте SQLite как замену PostgreSQL в принимаемом стенде. 2. Спроектируйте Attachment и связь с чатом/сообщением. S3 object key назначает сервер, filename используется только для отображения.
3. Реализуйте регистрацию с password KDF, вход через Basic, opaque Bearer-сессию, expiry и logout. Действующие сессии тоже должны сохраняться. 3. Реализуйте синхронный binary upload PNG/JPEG с ограничениями, сохранение метаданных и авторизованный download через API.
4. Подготовьте единый локальный запуск API и БД (Compose рекомендуется уже здесь), подключите именованные volumes. 4. Включите S3 и persistent volumes в общий Compose/Swarm. Проверьте, что другая API-реплика скачивает файл после остановки загрузившей.
5. Пройдите smoke и тест полного перезапуска. На 4 добавьте Redis/Valkey с конкретной ролью и повторите эксперимент отказа. 5. Проверьте недоступность S3, чужой чат и restart всего стенда.
## Оценка «3»: работающий минимум ## Оценка «3»: работающий минимум
- Минимум lab1 сохранён с объявленным переходом на новую аутентификацию. Все обязательные данные и ещё действующие сессии переживают рестарт API и хранилищ без удаления volumes. - Все обязательные возможности lab3 сохранены. S3-сервер находится в общем стенде; метаданные вложений живут в PostgreSQL, байты — в приватном bucket, не в БД или локальном каталоге API.
- PostgreSQL содержит пользователей, чаты, участников и сообщения. Есть первичные/внешние ключи, уникальность username и миграции с нуля. Составная операция создания чата выполняется атомарно. - POST вложения возвращает 201 ready после сохранения; оригинал скачивается побайтно. Message с attachment_ids читается участниками чата. Публичный smoke проверяет checksum и отсутствие доступа у постороннего.
- Basic применяется только на POST /auth/sessions, остальные endpoint требуют Bearer. Есть срок жизни сессии и отзыв; X-User-Id не даёт доступ. Пароли уже хешируются password KDF. - Принимаются реальные PNG/JPEG ≤10 MiB и ≤40 млн пикселей; MIME проверяется по содержимому, опасные filename отклоняются, лимиты применяются до неограниченного чтения/декодирования.
- make test и make test-persistence ACTION_SCRIPT=scripts/restart.sh проходят. Миграции повторно запускаются без дублирования данных. Secrets/volumes не попадают в Git. - Upload → остановка обслужившей API-реплики → download через другую возвращает тот же файл. Restart с сохранением volumes сохраняет метаданные и оригинал.
## Оценка «4»: уровень стажёра/джуна ## Оценка «4»: уровень стажёра/джуна
Весь уровень 3 **и все** пункты ниже: Весь уровень 3 **и все** пункты ниже:
- Redis ИЛИ Valkey реально используется для сессий с TTL либо осмысленного кэша. Опишите источник истины, политику expiry и инвалидации. Если это единственное хранилище сессий, включите persistence, выдерживающее проверяемый рестарт; один TTL не сохраняет данные. - Тесты покрывают слишком большой файл, повреждённое изображение, подмену MIME, чужое attachment_id и привязку файла другого чата. Пользовательская ошибка не превращается в 500.
- Используются параметризованные запросы/безопасный ORM, ограниченный connection pool и таймауты. Проверены гонка регистрации одинакового username и rollback составной операции. - Прерывание загрузки и отказ между записью S3/БД не оставляют вечных неучтённых объектов: есть документированный механизм компенсации/сверки/очистки с безопасным grace period.
- Индекс истории обоснован запросом и EXPLAIN (ANALYZE, BUFFERS) на ≥10 000 сообщениях; пагинация не использует полный scan всей истории на каждую страницу. - Лимиты/таймауты и потоковая обработка не позволяют одной загрузке занять неограниченную RAM. Проведён эксперимент ≥10 одновременных загрузок с измерением пика памяти.
- Есть автоматическая проверка expiry при коротком SESSION_TTL_SECONDS, logout, неверного пароля и недоступности Redis/БД. При потере сервиса приложение возвращает ограниченную ошибку/корректно обходит кэш, не принимает запрос от анонимного пользователя как доверенный. - Bucket bootstrap идемпотентен; API использует отдельные S3 credentials с ограниченными правами. Административная консоль не открыта в общей сети. Есть проверка private bucket вне API.
## Оценка «5»: исследование и доказанный результат ## Оценка «5»: исследование и доказанный результат
Весь уровень 4 **и один законченный трек на выбор**. Приведите гипотезу, повторяемую методику, результаты и ограничения. Весь уровень 4 **и один законченный трек на выбор**. Приведите гипотезу, повторяемую методику, результаты и ограничения.
**Трек 1. Согласованность кэша.** Покажите на конкурентном сценарии stale read или cache stampede, реализуйте защиту и измерьте число SQL-запросов/latency до и после. Докажите read-after-write из контракта и восстановление после рестарта кэша; одной установки Redis недостаточно. **Трек 1. Прямой upload.** Добавьте отдельный безопасный presigned-upload flow: короткая подпись, уникальный ключ, проверка принадлежности, размера/checksum при complete, обработка незавершённой загрузки. Сохраните обязательный proxy endpoint. Сравните трафик и RAM API при ≥2 размерах файлов, покажите негативные сценарии.
**Трек 2. План запроса и восстановление.** Исследуйте ≥100 000 сообщений с неравномерными размерами чатов, сравните ≥2 индекса/запроса и стоимость записи. Сделайте backup → восстановление в новое хранилище → сверку количества и выбранных сообщений. Обсудите время восстановления и границы потери данных. **Трек 2. Согласованность двух хранилищ.** Разработайте повторяемый reconciler для orphan/missing объектов и метаданных. Внедрите ошибки на обеих сторонах, покажите обнаружение и безопасное исправление с dry-run/повторным запуском. Докажите, что свежие и используемые вложения не удаляются; измерьте стоимость обхода.
## Как начать и проверить ## Как начать и проверить
@@ -78,28 +78,28 @@ Smoke не выставляет оценку автоматически. Про
## Сценарий защиты ## Сценарий защиты
1. Применить миграции к пустому volume, создать пользователей, войти, отправить сообщение. 1. Загрузить fixture, отправить message с attachment_ids, скачать как второй участник и сравнить SHA-256.
2. Запустить persistence-сценарий: он держит токен только в памяти тестового процесса; action script перезапускает API и все используемые хранилища, сохраняя volumes. 2. С третьим пользователем получить 404 для метаданных и bytes; direct unauthenticated S3 GET не отдаёт объект.
3. После готовности тот же токен читает те же данные; после logout — 401. Повторить с отдельным коротким TTL для проверки expiry. 3. Выполнить persistence и failover: общий тест начиная с lab4 сохраняет и повторно читает оригинал через API.
4. Для 4: показать роль Redis/Valkey, план SQL и ошибку зависимости. Для 5: выполнить выбранный эксперимент. 4. Показать, что на API нет обязательного локального файла оригинала. Для 4/5 — отказы и эксперименты.
## Что должно быть в PR ## Что должно быть в PR
Исходники, Dockerfile, миграции, compose.yaml либо эквивалентный воспроизводимый запуск, scripts/restart.sh, собственные тесты и отчёт. Дополненный стенд, настройка bucket/политики, миграции Attachment, тесты файлов, описание согласованности БД/S3 и результаты.
Заполните `REPORT.md` и [шаблон PR](.github/pull_request_template.md), укажите целевую оценку/трек. Не меняйте обязательный контракт и публичные tests ради зелёного результата. Сохраняйте возможности предыдущих лабораторных в пределах [объявленных переходов](COURSE.md). Заполните `REPORT.md` и [шаблон PR](.github/pull_request_template.md), укажите целевую оценку/трек. Не меняйте обязательный контракт и публичные tests ради зелёного результата. Сохраняйте возможности предыдущих лабораторных в пределах [объявленных переходов](COURSE.md).
## Вопросы на защите ## Вопросы на защите
- Что подтверждает COMMIT и какие гарантии зависят от настройки durability? - Почему файл в Docker volume одной API-реплики не равен объектному хранилищу?
- Где хранятся сессии и что будет после flush/restart Redis? - Когда можно честно вернуть 201?
- Чем аутентификация отличается от доступа к конкретному чату? - Что произойдёт, если S3 PUT прошёл, а COMMIT в PostgreSQL — нет?
- Почему hash пароля и быстрый checksum имеют разные задачи? - Почему случайный object key и приватный bucket не заменяют ACL приложения?
## Первичные материалы ## Первичные материалы
- [PostgreSQL transactions](https://www.postgresql.org/docs/current/tutorial-transactions.html) - [S3 object uploads](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html)
- [PostgreSQL EXPLAIN](https://www.postgresql.org/docs/current/using-explain.html) - [Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html)
- [Valkey persistence](https://valkey.io/topics/persistence/) - [MinIO](https://github.com/minio/minio)
- [Redis persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/) - [SeaweedFS](https://github.com/seaweedfs/seaweedfs)
- [Basic authentication](https://www.rfc-editor.org/info/rfc7617/) - [RustFS](https://docs.rustfs.com/)
+434 -21
View File
@@ -1,37 +1,450 @@
# Отчёт — лабораторная 2 # Отчёт — лабораторная 4 «S3-хранилище и вложения»
- Автор, группа: TODO Автор: **Костылев Артём, БПИ2303**. Ветка: `solution/bpi2303-kostylev`.
- Целевая оценка; выбранный трек 5, если нужен: TODO Цель — **«5», трек 2 «Согласованность двух хранилищ (reconciler)»**.
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO Финальная validation завершена 2026-11-06 на
`5e49502ae78fa48b992c97418e490666c699726d`: runtime совпадает с прошедшим независимое
ревью `7a0ebac2aa9e1d8602054ca4c7c1b0c23d3bfad4`. Динамический долг обоих PNG-фиксов
закрыт, открытых major не выявлено; известный minor сохранён в ограничениях.
**Заявлена «5», трек 2**, с отрицательным результатом гипотезы о линейном времени.
[Аудит применимости](evidence/load-validation.md) объясняет, какие старые серии приняты
без повторения и какие проверки выполнены заново. Публикация ещё не разрешена.
## Происхождение и применимость доказательств
| Состояние кода | SHA | Что подтверждено и что сохраняется |
| --- | --- | --- |
| Перенос основы ЛР3 на SQLAlchemy/Alembic | `8e4532f6f5c05d902b6e2c310f54320d95585904`, источник `f591fa64e525679a31c9db7f942d36c789e1a12f` | Полная серия уровней 3/4; историческая относительно поздних исправлений |
| Стоимость reconciler | `3402fd69d4922e500035a204d957be2cccaeb85a` | 27 пригодных замеров: измеряемый обход, SQL/S3-адаптеры, зависимости и стенд не менялись; сохраняются с исходным SHA |
| Поздние исправления ЛР3 | `75b93bbba5f29575d3fca1d013cc7e0770d88082`, источник `ba78cf6da439a4ec60fd59389583b38285b0b2b1` | Сессии, NUL/Basic, JSON ошибок LB; 492 теста, 19 stack-тестов, отказы и новые серии. Все 214 путей диапазона учтены в [манифесте переноса](evidence/forward-port.md) |
| Декодируемость PNG | `9387c3d82ab77ea376eecce19a36a22803d35efb` | Битый IDAT с корректными CRC отклоняется; [offline-доказательства](evidence/png-validation-fix/NOTES.md) |
| Единый IHDR | `7c8a1a64899dc0a1b5fa0949c0992ef40722eacf`, архив/ревью `7a0ebac2aa9e1d8602054ca4c7c1b0c23d3bfad4` | Повторный IHDR отклоняется; 407 offline-тестов. [Манифест](evidence/png-header-fix/manifest.json) совпадает с текущими семью файлами |
После `75b93bb…` в runtime изменились только `images.py` и новый `png.py` зоны attachments.
Поэтому старые upload-memory и загрузки во время сверки **не доказывают** память и скорость
нового инспектора. Результаты текстового пути, скачивания, persistence/failover, IAM/bootstrap
сохраняются для неизменённых механизмов; это не повторные прогоны на текущем HEAD.
Серия asyncpg 29–30 октября вынесена из основных таблиц в
[исторический индекс](evidence/README.md#историческая-реализация-asyncpg).
[Индекс evidence](evidence/README.md) связывает команды, SHA и файлы.
[Аудит отчётного этапа](evidence/report-audit-2026-11-06.md) содержит проверки исходных чисел,
ссылок и границ изменений. Сырые результаты не редактировались.
## Запуск из чистого clone ## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных. Нужны Git, Docker Compose v2, make, uv и Python 3.14 для приложения/собственных тестов;
публичным проверкам достаточно системного Python 3.9+. Из основного репозитория можно
создать свежий clone под разрешённым каталогом `.verification/`:
```sh
git clone --no-hardlinks --branch solution/bpi2303-kostylev . .verification/hlpipe-lab4-defense
cd .verification/hlpipe-lab4-defense
export COMPOSE_PROJECT_NAME=hlpipe-lab4-defense
export COMPOSE_FILE=compose.yaml
export LB_HTTP_PORT=18084 LB_STATS_PORT=24845
export BASE_URL=http://127.0.0.1:18084
export NO_PROXY=localhost,127.0.0.1,::1
export no_proxy=localhost,127.0.0.1,::1
make secrets
docker compose --env-file .env config --quiet
make up
make test
make test-persistence ACTION_SCRIPT=scripts/restart.sh
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
docker compose ps -a
make restore
make scenario SCENARIO=failover-upload
make restore
make scenario SCENARIO=s3-outage
make s3-check
make reconcile
make migrate
make down
```
Имя clone/project должно быть новым; порт 18084 должен быть свободен. Занявший его чужой
стенд не останавливается. `make secrets` создаёт локальные файлы и не перезаписывает
существующие; содержимое секретов выводить не нужно. `make up` ждёт PostgreSQL/Valkey/RustFS,
`alembic upgrade head`, bootstrap bucket и обе API-реплики в ротации LB. Повторный `make migrate`
не меняет актуальную схему. `make down` сохраняет постоянные volumes; `reset-data`, `down -v`
и удаление пользовательских данных в этом процессе не используются.
Сохранённое воспроизведение из свежего clone: **76,866 с, exit 0**, SHA
`cf41f7c64dcf1dea6da0643f5793bd2e5dd1f03e`, проект `hlpipe-lab4-clean-20261105231220`.
Smoke, persistence, failover, upload→stop→download, S3-outage, 31 проверка S3, dry-run и
повторная миграция прошли; после down остались три volumes.
[Полная команда и вывод](evidence/sqlalchemy-2026-11-06/grade4/clean-clone-command.txt).
Это запуск **до позднего переноса и PNG-фиксов**, а не доказательство запуска нынешнего SHA.
Проверки исходников выполняются отдельно:
```sh
make install
make check
make validate
make lint
make student-test
openspec validate --all --strict --no-interactive
```
`make student-test` использует одноразовые Docker-зависимости. На отчётном этапе выполнены
только check/validate/lint/OpenSpec и чтение сохранённых доказательств; тесты с Docker и
нагрузки не запускались. Сохранённые полные 492 теста относятся к `75b93bb…`, а 407 unit-тестов
после PNG-фиксов — к `7c8a1a6…`; складывать эти числа в число тестов одного прогона нельзя.
## Архитектура и решения ## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab1, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации. ```text
клиент → HAProxy :18084 → API ×2, по 2 UvicornWorker
├→ PostgreSQL: метаданные, сессии, сообщения
├→ Valkey: кэш сессий
└→ RustFS: байты оригиналов
migrate → PostgreSQL; s3-init → private bucket и IAM
reconciler → PostgreSQL + RustFS, одноразовый сервис
```
## Доказательства критериев Python 3.14, FastAPI, gunicorn/uvicorn-worker, dishka, Pydantic v2, structlog,
**SQLAlchemy 2.0.45 async, asyncpg как драйвер, Alembic 1.16.5**, aiobotocore и Pillow 13.0.0.
Версии зависимостей закреплены в [uv.lock](uv.lock), правила форматирования — пробелы в
[pyproject.toml](pyproject.toml). Образы закреплены digest в [compose.yaml](compose.yaml):
HAProxy 3.2.26, PostgreSQL 17.11, Valkey 8.1.11, RustFS 1.0.4, rc 0.1.36.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства | Зоны `identity`, `messaging`, `attachments` содержат домен, прикладные порты и адаптеры;
`shared` — платформу, `main.py` — сборку. Межзонная проверка членства идёт через порт
`ChatMembership`, не через импорт инфраструктуры соседней зоны. Pydantic остаётся на HTTP-границе,
ORM-модели `DeclarativeBase` — внутри зон. Сессии создаёт `async_sessionmaker`; UoW получает
свою `AsyncSession` на транзакцию и явно делает COMMIT. Незавершённая транзакция откатывается
при закрытии сессии. SQL строится ORM/Core с параметрами; `asyncpg` не используется как
отдельный прикладной слой запросов. Отправка сообщения — Core CTE под блокировкой строки чата:
квитанция `Idempotency-Key`, сообщение и ссылки на вложения фиксируются вместе.
Alembic [env.py](alembic/env.py) использует async engine и `run_sync`, advisory lock и
транзакции миграций. Ревизии: `0001` — базовая схема, `0002` — ключи идемпотентности,
`0003` — вложения/связи, `0004` — карантин. Legacy-схема принимается только после сверки
журнала и каталога в [legacy.py](alembic/legacy.py); произвольный stamp вместо миграции не применяется.
При подготовке отчёта через Context7 сверены закреплённые версии:
[SQLAlchemy, сессии](https://github.com/sqlalchemy/sqlalchemy/blob/rel_2_0_45/doc/build/orm/session_basics.rst),
[Alembic, async migrations](https://github.com/sqlalchemy/alembic/blob/rel_1_16_5/docs/build/cookbook.rst).
**Источники истины.** PostgreSQL хранит владельца/чат, имя, MIME, размер, SHA-256, размеры
изображения, статус, object key и ссылки сообщений. Байты — только в приватном RustFS,
ключ `attachments/<uuid>` назначает сервер. API read-only, без data volumes; `/tmp` — tmpfs.
`attachments.object_key` имеет UNIQUE и `COLLATE "C"`; частичный индекс pending обслуживает
уборку; FK `message_attachments (attachment_id, chat_id)` не даёт привязать чужой чат или
удалить используемое вложение. История использует keyset `(created_at, id)`.
**Upload.** Сначала аутентификация (`401`), объявленный размер (`413`), членство (`404`),
filename (`400`), MIME (`415`). Поток ограничен 10 485 760 байтами и считается также без
Content-Length; SHA-256 вычисляется при чтении. Тело собирается в ограниченный буфер под
`UPLOAD_CONCURRENCY=3` на worker; две проверки изображений на worker работают в потоках.
Лимит 40 млн пикселей проверяется до распаковки. PNG требует единственный первый IHDR,
проверку CRC и потоковую проверку zlib/Adler-32, длины строк и фильтров, включая Adam7.
Дополнительные блоки распаковки — по 64 KiB, полного массива пикселей нет. JPEG проверяется
через `draft` 1/8 и `load`. Ошибка изображения → `415` до намерения в БД/S3.
Оба PNG из ревью отклоняются с 415 в ASGI-регрессиях и через настоящий LB/RustFS.
После двух запросов число строк/объектов осталось 7/7; корректные PNG/JPEG скачаны
вторым участником побайтно: [финальная сверка](evidence/png-final-2026-11-06/png-roundtrip.json).
Порядок публикации: `INSERT pending + COMMIT → PUT → UPDATE ready + COMMIT → 201`.
Метаданные и скачивание доступны только участникам чата; чужой и неизвестный id дают `404`.
Скачивание идёт через API кусками 64 KiB, с исходными байтами и истинным MIME, без redirect.
`attachment_ids` присутствует во всех ответах сообщений, отсутствие и `[]` дают один fingerprint.
**Компенсация.** После отказа фиксации ready новая транзакция блокирует строку `FOR UPDATE`:
ready не трогается; у pending удаляется объект, затем строка. При отмене/неизвестном исходе
PUT намерение остаётся для уборки. Sweeper раз в 30 с выбирает до 100 pending старше 900 с
через `FOR UPDATE SKIP LOCKED`, удаляет объект раньше строки и публикует
`audit.attachment_discarded`. Готовые и используемые вложения не выбираются.
**Ресурсы и отказы.** У API 1 CPU/512 MiB на реплику, у LB 1 CPU/128 MiB,
PostgreSQL 2 CPU/1 GiB, RustFS 1 CPU/512 MiB, Valkey 0,5 CPU/128 MiB;
reconciler 1 CPU/256 MiB. Условия cost-эксперимента отличаются и указаны отдельно.
Слоты дополнены буферами HAProxy 64 KiB, максимумом TCP-буфера API 256 KiB и
`MALLOC_MMAP_THRESHOLD_=131072`. S3 connect/read = 1/2 с, две попытки; запрос — 8 с,
LB — 10 с простоя. Дедлайн продлевается переданными байтами при пороге 128 KiB/s,
поэтому идущая передача может длиться дольше 8 с. LB не повторяет upload.
Readiness проверяет PostgreSQL, но не S3: отказ хранилища изображений сохраняет текстовый путь.
**Доступ к S3.** Только LB публикует host-порты на loopback. RustFS в internal-сети,
консоль выключена. `s3-init` повторяемо создаёт private bucket, политики/пользователей и
lifecycle очистки незавершённых multipart. API получает только Get/Put/DeleteObject своего
bucket, reconciler — отдельный ключ ListBucket/Get/DeleteObject; root-ключи есть лишь у
`s3`, `s3-init`, `s3-cli`. `s3-init`/`s3-cli` — uid 65534, migrate/reconciler — `init: true`.
## Критерий → команда → наблюдение → файл
Номера ниже — строки этого отчёта, а не дополнительная рубрика. **F** означает SHA
`75b93bbba5f29575d3fca1d013cc7e0770d88082` до PNG-фиксов; **S** — исходную SQLAlchemy-серию,
точный SHA команды находится в её raw. **V** — финальная validation на
`5e49502ae78fa48b992c97418e490666c699726d`, runtime идентичен прошедшему review `7a0ebac…`.
[Новые команды и результаты](evidence/load-validation.md) дополняют сохранённые проверки.
| Критерий | Команда | Наблюдение и применимость | Файл |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO | | 3.1 Минимум ЛР3 | `make test`; `student-tests/tools/forward_network.py`; `make lb-check`; `make retry-check` | F: smoke, отзыв сессий на двух репликах, NUL/Basic PASS; S: 20×8 запросов с одним ключом, одна копия; потерянный ответ и kill без дублей | [smoke](evidence/forward-final-2026-11-06/smoke.txt), [сеть](evidence/forward-final-2026-11-06/network-regressions.txt), [LB](evidence/sqlalchemy-2026-11-06/grade3/lb-check-rerun.txt), [retry](evidence/sqlalchemy-2026-11-06/grade3/retry.txt) |
| Остальные пункты уровня 3 | TODO | TODO | TODO | | 3.2 Метаданные отдельно от байтов, приватность, stateless API | `make attachments-evidence`; `make limits` | S: внешняя сверка размера/SHA, изображения не в БД/API; F: лимиты, сети, mounts и порты PASS | [размещение](evidence/sqlalchemy-2026-11-06/grade3/storage-layout.txt), [лимиты](evidence/forward-final-2026-11-06/limits.txt) |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO | | 3.3 `201 ready`, сообщения, checksum, чужому `404` | `make test`; `make student-test` | F: PASS, включая реальные PostgreSQL/RustFS-тесты; V: smoke и 721 собственная проверка прошла (719 + адресный повтор 2 setup errors) | [smoke](evidence/forward-final-2026-11-06/smoke.txt), [492 теста](evidence/forward-final-2026-11-06/offline-checks-rerun.txt), [V smoke](evidence/png-final-2026-11-06/smoke.txt) |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO | | 3.4 PNG/JPEG, 10 MiB/40 MP, MIME/filename | `make test-faults`; `make student-test PYTEST_ARGS='-q --tb=short student-tests/unit'` | F: граничные файлы, MIME/имена → 4xx; текущие offline-регрессии: оба PNG из ревью → 415, без записи. V: оба файла → 415 через LB, число строк/объектов неизменно; PNG/JPEG скачаны побайтно | [старый suite](evidence/forward-final-2026-11-06/stack-faults.txt), [407 unit](evidence/png-header-fix/unit.txt), [регрессии](evidence/png-header-fix/targeted.txt), [V PNG/JPEG](evidence/png-final-2026-11-06/png-roundtrip.json) |
| 3.5 Upload → остановка принявшей реплики → download | `make scenario SCENARIO=failover-upload` | F: target exited, сосед running, SHA совпал, download через 2,762 с | [JSON](evidence/forward-final-2026-11-06/failover-upload.json) |
| 3.6 Restart с volumes и failover | `make persistence-evidence`; `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | F: данные/сессия сохранены; 130 записей, 0 ошибок, первая запись через 2,883 с | [persistence](evidence/forward-final-2026-11-06/persistence-rerun.txt), [failover](evidence/forward-final-2026-11-06/failover-d4c8d53d7433.txt) |
| Отказ S3 | `make scenario SCENARIO=s3-outage` | F: upload/download 503 за 4,990/5,329 с; текст 201, метаданные/readiness 200; после возврата checksum равен | [JSON](evidence/forward-final-2026-11-06/s3-outage.json) |
| 4.1 Пользовательские ошибки файлов не дают 500 | `make test-faults`; PNG ASGI-регрессии | F: 112 ответов в журнале негативного suite, 0×5xx; битый zlib и повторный IHDR добавлены позднее. V: 7 stack PASS, включая оба PNG; 54 ответа негативного теста, 0×5xx | [stack](evidence/forward-final-2026-11-06/stack-faults.txt), [PNG](evidence/png-header-fix/NOTES.md), [V suite](evidence/png-final-2026-11-06/upload-faults.txt) |
| 4.2 Компенсация, crash и безопасная уборка | `make test-faults`; `make overload-leftovers` | F: COMMIT fault → 503, строка/объект удалены; обе реплики убиты между PUT/COMMIT, свежие/ready сохранены. S: 30 pending убраны после реальных 900 с, 1669 ready/объектов совпали | [отказы](evidence/forward-final-2026-11-06/stack-faults.txt), [grace без сдвига времени](evidence/sqlalchemy-2026-11-06/grade4/overload-leftovers.txt) |
| 4.3 Ограниченная RAM, ≥10 загрузок | `make upload-memory ARGS='--slots 3 --out <новая-серия>'` | S: 36 точек; F: 3 точки, максимум 456,2 MiB, OOM 0, 238×201/2×503. V: 18 точек, 840/840×201, API ≤440 MiB, LB ≤31,1 MiB, OOM 0; 10×40 MP RGBA16 → 201, API ≤270,8 MiB | [S](evidence/sqlalchemy-2026-11-06/grade4/NOTES.md), [F](evidence/forward-port.md#повтор-затронутых-измерений), [V RAM](evidence/png-final-2026-11-06/memory-summary.json) |
| 4.4 Bootstrap/IAM/закрытая консоль/private вне API | `make s3-check` | S: 31 PASS, anonymous GET 403, API ListBucket запрещён, повтор bootstrap ×2 неизменен; конфигурация не менялась | [сырой вывод](evidence/sqlalchemy-2026-11-06/grade3/s3-security.txt) |
| 5.1 Внедрение, dry-run/apply/повтор, защита свежих и ссылок | F: `make reconcile-demo`; V: `python3 evidence/png-final-2026-11-06/concurrent-reconcile.py` | F: 13 PASS; 18 исправлений, повтор 0, ссылки сохранены; алгоритм не менялся. V: 3341×201 при 67 проходах, lost_ready=0, независимый SHA-аудит PASS | [demo](evidence/forward-final-2026-11-06/reconcile-demo/demo.json), [V load](evidence/png-final-2026-11-06/concurrent-reconcile/under-load.json) |
| 5.2 Стоимость обхода | `make reconcile-cost ARGS='--out <новая-серия>'` | 27 замеров на `3402fd6…`, 1k/10k/30k/100k; страницы и память подтверждены, линейное время отвергнуто | [методика](evidence/sqlalchemy-2026-11-06/track5/NOTES.md), [сырьё](evidence/sqlalchemy-2026-11-06/track5/measurements/cost/) |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены. ## Отказы: наблюдаемое время и потери
## Эксперименты и отказы Основная таблица — **F**, проект `hlpipe-lab4-forward-final-20261106`.
Времена — отдельные наблюдения, не SLA. Потерями считаются подтверждённые клиенту данные;
намеренно повреждённые байты в demo отдельно отмечены. Состояния контейнеров сохранены
в JSON/inspect и выводе хуков; неизвестные длительности не заменены нулём.
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения. | Воздействие / команда | Реально затронутый контейнер или объект | Наблюдение, время | Потери / файл |
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| TODO | TODO | TODO | TODO | | `scenario failover-upload` | API `d4c8d53d7433` exited; `da922ab2ea09` running | Хук 2,734 с; download через 2,762 с от старта | Байты совпали; [JSON](evidence/forward-final-2026-11-06/failover-upload.json) |
| `test-failover` | API `d4c8d53d7433`, подтверждён exited | 130 успешных записей, 0 failed_attempts; первая через 2,883 с | Дубликатов/потерь не обнаружено; [inspect и тест](evidence/forward-final-2026-11-06/failover-d4c8d53d7433.txt) |
| `persistence-evidence` | API ×2, LB `ea6930f64089`, PG `3f3a870d89c0`, Valkey `07ea16e2fcfe`, S3 `5921e788f03d`; StartedAt изменились | Вся команда 13,179 с; это не точный интервал недоступности | Однократное чтение после hook сохранило данные/сессию; [до/после](evidence/forward-final-2026-11-06/persistence-rerun.txt) |
| `scenario s3-outage` | S3 `5921e788f03d` exited | 503: upload 4,990 с, download 5,329 с; healthy через 5,760 с после старта | Готовый файл сохранился; текст 201; [JSON](evidence/forward-final-2026-11-06/s3-outage.json) |
| `make test-faults`, PostgreSQL stop | Сервис postgres своего project; отдельного inspect в suite нет | 503 за 0,002–0,015 с; готовность через 2,1 с после старта | Прежний токен работает, 40 регистраций через 2 реплики без рестарта API; [stack](evidence/forward-final-2026-11-06/stack-faults.txt) |
| `make test-faults`, Valkey stop | Сервис valkey своего project; отдельного inspect в suite нет | Живой токен 200 за 0,005 с; неизвестный 401 за 0,003 с; logout 503, после старта 204 | Отзыв не потерян; время восстановления не измерено; [stack](evidence/forward-final-2026-11-06/stack-faults.txt) |
| COMMIT fault после PUT | Триггер в синтетическом сценарии | 503; сразу строка отсутствует и HEAD 404; audit compensation | Готовые не затронуты; [stack](evidence/forward-final-2026-11-06/stack-faults.txt) |
| SIGKILL между PUT/COMMIT | По очереди API `da922ab2ea09`, `d4c8d53d7433` | Клиент 502; pending + объект. После сдвига за grace удаление через 26,1/1,5 с | Свежий остаток сохранён; ready в сообщении скачан с тем же SHA; [stack](evidence/forward-final-2026-11-06/stack-faults.txt) |
| Обрыв / зависание тела | 256 KiB из 5 MiB / 1 KiB | Обрыв: 400 в журнале, нет intent; stall: 503 через 8,02 с | Новых видимых вложений нет; [stack](evidence/forward-final-2026-11-06/stack-faults.txt) |
| Реальное ожидание grace, серия S | 30 pending: 27 после перегрузки, 3 после S3-outage | Наблюдение 847,812 с уже существовавших остатков; возраст не меняли, grace 900 с | 30 audit abandoned, затем 1669 ready = 1669 объектов; [вывод](evidence/sqlalchemy-2026-11-06/grade4/overload-leftovers.txt) |
| Reconciler demo, порча обеих сторон | 3 orphan, 9 ready с потерянными/изменёнными байтами, 6 pending | Apply: 18 действий; повтор 0; полный demo 256,6 с | 9 ready и 6 ссылок сохранены, байты сами не восстанавливаются; [demo](evidence/forward-final-2026-11-06/reconcile-demo/demo.json) |
| SIGTERM / S3 outage сверки | Одноразовый reconciler; S3 paused / остановлен | SIGTERM: 0,1 с, exit 143, lock освобождён; S3 outage: exit 1 за 2,32 с | Состояние не изменилось при outage; [demo](evidence/forward-final-2026-11-06/reconcile-demo/demo.json) |
## Ограничения и следующие шаги `failover-upload.json` содержит `restored=false`: сценарий сам не возвращал target;
[отдельный restore](evidence/forward-final-2026-11-06/restore.txt) возвращает обе реплики.
В конце фазы F все три её project остановлены, шесть постоянных volumes сохранены:
[shutdown-state](evidence/forward-final-2026-11-06/shutdown-state.txt).
Отчётный этап Docker не использовал. Состояние финальной validation-фазы и сохранённые
volumes зафиксированы отдельно в [аудите](evidence/load-validation.md).
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза. ## Память и запись сообщений
**Память S, до позднего переноса и PNG-фиксов.** Два API по 1 CPU/512 MiB, два worker,
LB 128 MiB, RustFS 1 CPU/512 MiB; 3/64 слота × PNG около 1/10 MiB × 10/20/40 клиентов,
три повтора каждой точки. Клиент с macOS, keep-alive, по два upload; перед точкой API/LB
пересоздаются для нового cgroup `memory.peak`. Пик включает процессы, page cache и сокеты;
p95 учитывает только 201. На сохранённом снимке фона — только свой project.
[Методика и 36 raw-точек](evidence/sqlalchemy-2026-11-06/grade4/NOTES.md).
| Конфигурация | Ответы в 18 точках | Максимум API | OOM API/LB |
| --- | --- | ---: | ---: |
| 3 слота | 840/840 ×201 | 448,0 MiB | 0 |
| 64 слота, исторический контроль | 813×201, 27×503 | 513,7 MiB | 0 |
**Память F, тоже до PNG-фиксов.** 40 клиентов, по два PNG 10 485 256 байт, 3 слота;
три повтора дали 80/80/78 успешных ответов и 0/0/2 ответа 503. Пики API:
456,2/438,1/450,4 MiB, LB: 31,8/28,5/29,1 MiB, OOM 0; p95 3641,3/4755,0/8091,8 мс.
Причина двух 503 отдельно не локализована; уборка их возможных остатков в этой серии не
измерялась. [Команды, фон и сырьё](evidence/forward-port.md#повтор-затронутых-измерений).
Контроль 64/3 остаётся историческим; новый инспектор измерен отдельно в серии V ниже.
**Память V, после обоих PNG-фиксов.** На том же распределении ресурсов проведены
ровно три новые серии штатных 3 слотов: ~1/10 MiB × 10/20/40 клиентов, два upload на
клиента. Всего **18 точек, 840/840×201**, ошибок/таймаутов/OOM-kill нет.
Максимум API **440,0 MiB / 512 MiB**, LB **31,1 MiB / 128 MiB**.
[Команды, сырьё и фон](evidence/load-validation.md#новая-матрица-памяти),
[пересчёт](evidence/png-final-2026-11-06/memory-summary.json).
| PNG / клиенты | Максимум API из 3 повторов, MiB | p95 успешных: медиана [min–max], мс |
| --- | ---: | ---: |
| ~1 MiB / 10 | 279,8 | 241,0 [134,3–257,7] |
| ~1 MiB / 20 | 284,5 | 243,8 [226,6–310,2] |
| ~1 MiB / 40 | 290,4 | 373,4 [367,6–417,6] |
| ~10 MiB / 10 | 394,8 | 1520,3 [1236,1–3287,8] |
| ~10 MiB / 20 | 440,0 | 2632,8 [2514,0–3632,6] |
| ~10 MiB / 40 | 439,2 | 5971,4 [4256,0–6534,9] |
Дополнительный PNG 8000×5000 RGBA16: 311 100 байт на проводе, 320 005 000 байт
распакованных строк. Перед нагрузкой файл независимо декодирован Pillow. 10 клиентов
стартовали через barrier: **10×201**, 10 побайтных скачиваний вторым участником,
API ≤270,828 MiB, LB ≤24,2 MiB, OOM 0. [Драйвер и raw](evidence/png-final-2026-11-06/high-depth.json).
Это один сильно сжатый паттерн; он не представляет все PNG и не является SLO.
После нагрузок shallow-проверка: 859 ready = 859 объектов, ни pending, ни orphan,
ни missing/size mismatch. SHA всех крупных объектов повторно не вычислялся.
[Сверка](evidence/png-final-2026-11-06/post-memory-invariants.json). На снимках фона —
только свой project; macOS, частоты, диск и кэш физического хоста не контролировались.
**Запись сообщений F.** `make load-evidence`: публичный `tests/load.py`, закрытый контур,
64 клиента, 8 с, новый горячий чат на прогон, три повтора с паузой 15 с. Клиент-контейнер
в edge-сети, 2 CPU/512 MiB; API ×2 по 1 CPU/512 MiB. На снимках до/после — только свой
project, host load average 3,42→4,69. Физический хост не был выделен эксклюзивно.
| Повтор | RPS | p95 успешных, мс | Ошибок |
| --- | ---: | ---: | ---: |
| 1 | 1428,577 | 100,062 | 0 |
| 2 | 1456,338 | 98,948 | 0 |
| 3 | 1529,320 | 98,475 | 0 |
Медиана **1456,338 RPS**, ошибок 0. [Команда](evidence/forward-final-2026-11-06/load-command.txt),
[сырые JSON](evidence/forward-final-2026-11-06/load/), [фон](evidence/forward-final-2026-11-06/performance-before.txt).
Это характеристика данного распределения ресурсов и одного чата, не SLO или предел архитектуры.
Относительно ранней SQLAlchemy-серии 1529,186 RPS разница −4,8%; причинное влияние
исправления из непарных серий не следует. PNG-изменения не вызываются текстовым запросом.
## Трек 2: согласованность двух хранилищ
`make reconcile` запускает `python -m messenger.attachments.entrypoints.reconcile`,
по умолчанию dry-run; `ARGS='--apply --deep'` включает исправления и хеширование.
Один проход сливает листинг S3 и keyset-страницы по `object_key` с индексом UNIQUE/COLLATE C.
Алгоритм делает O(N) сравнений и хранит O(страница); это не обещание линейного времени RustFS.
По умолчанию страница 1000, grace 3600 с + skew 60 с. Advisory lock разрешает один проход;
второй возвращает skipped. Deep хеширует потоками 64 KiB, по умолчанию до восьми объектов.
| Расхождение | Безопасное действие `--apply` |
| --- | --- |
| Старый orphan | Повторная проверка строки по ключу, затем удаление объекта; свежий не трогается |
| Старый pending | Повторная проверка под `FOR UPDATE SKIP LOCKED`, объект раньше строки, audit reason=reconcile |
| Ready без объекта / другой размер / SHA | Повторная проверка; карантин, без удаления ready и ссылок сообщений; `/content` → 404 |
| Объект карантина восстановлен | Снять карантин только после совпадения SHA-256, даже без `--deep` |
**Гипотезы до замеров** — [design на этапе SQLAlchemy](openspec/changes/archive/2026-11-06-verify-lab4-reconciler-sqlalchemy/design.md),
proposal `751aaed4b1baaa957c74bb8874adeb0eac4b0480`:
H1 — точное обнаружение/исправление, повтор 0 и сохранность свежих/используемых;
H2 — shallow t(100k)/t(10k) в [7,13], страницы ceil(N/page), RSS растёт менее чем на 10%,
deep на порядок дороже; H3 — после SQLAlchemy сохраняются страничность и ограниченная память.
Отрицательный результат H2 не меняет исходную гипотезу.
**Корректность F.** На свежем отдельном project, grace 60/skew 0 внедрены ошибки обеих сторон.
Dry-run не изменил 24 объекта/27 строк. Apply удалил 3 старых orphan и 6 pending, перевёл
9 ready в карантин; шесть audit-событий, все шесть ссылок сохранены. Повтор — 0 действий.
Возврат исходных байтов снял один карантин без deep, подмена той же длины — не сняла.
Всего 13 PASS; [первичный demo](evidence/forward-final-2026-11-06/reconcile-demo/demo.json).
При 75 с параллельных upload: 3364×201, 59 проходов, lost_ready=0, независимая проверка SHA
в [check-under-load](evidence/forward-final-2026-11-06/reconcile-demo/check-under-load.json).
**Конкурентная загрузка V после PNG-фиксов:** отдельный fresh project, 75 с / 3 потока,
3341/3341×201, 67 проходов, errors=0/skipped=0, lost_ready=0. Независимый deep-аудит
проверил SHA всех 3341 ready; каждый десятый upload привязан к сообщению. Удалены только
3 внедрённых orphan. [Команда](evidence/png-final-2026-11-06/concurrent-reconcile.txt),
[raw](evidence/png-final-2026-11-06/concurrent-reconcile/under-load.json),
[SHA-аудит](evidence/png-final-2026-11-06/concurrent-reconcile/check-under-load.json).
На пустом dataset инъекция pending создала 0 строк (ей нужна существующая строка-владелец);
этот сценарий повторно не доказан серией V и сохраняется из полного demo F.
Затронутый PNG-путь и сохранность всех 201 проверены; остальной алгоритм не менялся.
Повторную проверку строки перед удалением дополнительно покрывает
`test_row_written_after_listing_keeps_the_object`: наблюдение fresh не подменяет этот барьер.
**Стоимость на `3402fd69d4922e500035a204d957be2cccaeb85a`.** Свежий project
`hlpipe-lab4-cost-20261106`; 1k/10k/30k/100k объектов и столько же ready-строк, PNG 8×8 по 264 B
и один smoke-объект 111 817 B. Наполнение — четыре параллельных PUT и SQLAlchemy batch INSERT;
измерение начинается после COMMIT и тихого окна, время наполнения исключено.
API/LB/Valkey остановлены. RustFS 2 CPU/3 GiB, cpuset 11–12; PG 2 CPU/1 GiB, 13–14;
CLI 1 CPU/256 MiB, 15. Это отдельная конфигурация, не штатные лимиты API-стенда.
Перед каждым повтором фильтр чужого CPU: ≤2 ядра VM и ≤0,25 на закреплённых,
запас запуска 1,5/0,5 ядро·с; quiet 15 с, deadline 900 с, до трёх повторов отклонённой точки.
Приняты 27, отклонённых 0; максимум чужого CPU 0,25 ядра, pinned 0. По три shallow/deep
на каждом N и три page100 на 10k. Все raw с исходным SHA сохранены и пересчитаны при отчёте.
[Команда](evidence/sqlalchemy-2026-11-06/track5/cost-command.txt),
[полный протокол](evidence/sqlalchemy-2026-11-06/track5/NOTES.md),
[план/фон/сырьё](evidence/sqlalchemy-2026-11-06/track5/measurements/cost/).
| N | Режим | Медиана прохода [min–max], с | LIST / SQL-страниц / GET | RSS, MiB, медиана |
| ---: | --- | --- | --- | ---: |
| 1000 | shallow | 0,107 [0,105–0,113] | 1 / 1 / 0 | 109,8 |
| 1000 | deep | 0,499 [0,498–0,536] | 1 / 1 / 1000 | 110,2 |
| 10000 | shallow | 0,999 [0,996–1,077] | 10 / 10 / 0 | 110,9 |
| 10000 | deep | 4,998 [4,972–5,041] | 10 / 10 / 10000 | 111,3 |
| 10000 | page100 | 2,037 [2,035–2,113] | 100 / 100 / 0 | 107,9 |
| 30000 | shallow | 3,601 [3,567–3,633] | 30 / 30 / 0 | 110,9 |
| 30000 | deep | 15,665 [15,630–15,779] | 30 / 30 / 30000 | 111,6 |
| 100000 | shallow | 20,215 [20,206–21,335] | 100 / 100 / 0 | 111,1 |
| 100000 | deep | 110,910 [58,661–147,215] | 100 / 100 / 100000 | 111,4 |
SQL здесь считает запросы страниц, без служебных BEGIN/COMMIT/pre-ping/advisory lock;
HEAD/DELETE=0 во всех точках. `cost.seconds` — сам проход, CPU/wall в raw включают старт CLI.
**H2 о времени отвергнута:** t100k/t10k=20,235. На 100k листинг занимает медиану 19,538 с;
внутренняя причина его удорожания не установлена. Deep дороже в 4,35–5,49 раза, не на порядок.
Страницы точно ceil(N/page), shallow RSS 109,8→111,1 MiB (+1,18%): эти части H2 и H3
подтверждены в измеренной области. H1 подтверждена перечисленными сценариями на F.
Deep 100k: 147,215/110,910/58,661 с, разброс 79,84% медианы. Все повторы прошли заранее
заданный фильтр; лучший не выбран вместо остальных. Один хост/VM, тёплые данные, частоты,
диск и нагрузка macOS не контролировались. PNG 264 B не прогнозируют deep для 10 MiB,
выше 100k измерений нет. Сравнение с историческим asyncpg — не причинный A/B-эксперимент.
Сверка не восстанавливает утраченные байты и требует отдельного ручного/планового запуска.
## Завершение финальной validation
[Общий долг двух PNG-фиксов](evidence/png-header-fix/NOTES.md#обязательные-проверки-финального-этапа)
закрыт на V: API-тесты с PostgreSQL, upload faults через LB/RustFS, 18 точек штатной
матрицы RAM, десять одновременных 40 MP RGBA16 и upload при сверке с независимым SHA-аудитом.
Команды и сырые результаты — в [аудите](evidence/load-validation.md).
`make student-test` дал 719 PASS и 2 setup errors из-за неготового временного RustFS;
адресный повтор только этих двух тестов прошёл. Таким образом, все 721 выбранная
проверка исполнена успешно, первый полный запуск не объявляется зелёным. Отдельно
7 stack-тестов PASS. `make check`, `make validate`, `make lint` и OpenSpec strict (18)
прошли. Реализация, конфиги, нормативные файлы и OpenSpec не менялись.
Принятые 27 cost-замеров и 3 повтора текстовой нагрузки сохранены с исходными SHA: повтор
не требовался. Старые 64/3 и upload-memory до PNG-фиксов не используются для вывода о
новом инспекторе. После Docker-этапа оба собственных project остановлены, шесть постоянных
volumes сохранены; [инвентаризация](evidence/png-final-2026-11-06/shutdown-state.txt).
Публикация — только после отдельной команды координатора PUBLICATION; push не выполнялся.
## Ограничения
- **Minor, сохранён ревью:** непарные Unicode-суррогаты U+D800/U+DFFF в `text` вызывают
`PydanticSerializationError` в `fingerprint()` и HTTP 500. Файл
[schemas.py:35](src/messenger/messaging/entrypoints/schemas.py#L35),
[HTTP-доказательство](evidence/forward-final-2026-11-06/known-minor-http.txt).
Это дефект, не PASS; по правилу пользователя minor не требует fix и не блокирует этап.
- Один Docker-host, по одному PostgreSQL/RustFS/LB — точки отказа; репликация и резервные
копии байтов здесь не исследуются. При остановке всех API sweeper ждёт их возврата.
- Тело upload ограничено, но хранится целиком; слоты защищают память ценой очереди.
Два 503 в F сохранены без объяснения недоказанной причиной. В V все 840 запросов
успешны при пике ≤440 MiB; это измеренная область 10–40 клиентов, не общий предел RAM
и не обещание 100% успеха при любой нагрузке.
- При неизвестном результате PUT/COMMIT остаток ждёт grace; потерянный ответ upload
допускает повторное вложение, поскольку у этого endpoint нет Idempotency-Key.
- После отправки заголовков сбой S3 при download может оборвать поток; слишком медленная
передача и SIGTERM также ограничены таймаутами. Ранний отказ во время отправки большого
тела клиент иногда увидит как reset. Сессия проверяется раньше объявленного размера.
- В demo grace 60/skew 0, смещение часов не моделировалось; реальный grace API 900 с
отдельно подтверждён старой серией S. Reconciler не восстанавливает повреждённые байты.
- Нагрузка записи — закрытый контур и один чат, успешные latency имеют coordinated omission.
Причинный эффект SQLAlchemy/исправлений и предел системы из этих чисел не выводятся.
- Секреты — локальные файлы Compose вместо менеджера секретов; статические S3-ключи без
исследованной ротации. TLS, presigned flow, брокер и варианты изображений вне этой лабы.
- Исторические proposal `add-lab4-image-attachments` и `add-lab4-upload-resilience`
превышают рекомендацию 400 слов; история не переписана. `openspec/config.yaml` сверён
с текущим стеком, но в документационном этапе не редактировался по ограничению области работ.
## Ответы на вопросы защиты
**Почему volume одной API-реплики не равен объектному хранилищу?** Другая реплика не обязана
видеть её локальный volume. RustFS даёт всем репликам одно адресуемое ключами хранилище
с политиками доступа; остановка принявшей API не удаляет оригинал. В F это показали
разные `X-Instance-Id` и совпавший SHA-256 после остановки target.
**Когда можно вернуть 201?** После успешного PUT и COMMIT статуса ready. Pending уже
зафиксирован до PUT, но не виден пользователям. Потерянный ответ после COMMIT возможен;
это не позволяет заранее отвечать 201.
**Что если S3 PUT прошёл, а COMMIT — нет?** При известном отказе компенсация под блокировкой
удаляет объект, затем pending. При падении процесса/неизвестном исходе это делает sweeper
после grace. Уже ready не удаляется; независимые orphan и расхождения ищет reconciler.
**Почему случайный key/private bucket не заменяют ACL?** S3-ключ API разрешает доступ
ко всему bucket и не знает членства пользователей в чатах. Поэтому каждый запрос
метаданных/байтов проверяет членство; постороннему 404. Private bucket дополнительно
закрывает прямой анонимный GET, но не принимает решение за приложение.
Для защиты: `make test` показывает upload→message→download и чужому 404;
`make s3-check` — private вне API; persistence/failover — сохранность;
`make attachments-evidence` — отсутствие локальных файлов API; сценарии уровней 4/5 —
компенсацию, ограничения памяти и безопасную сверку. Следует показать исходный SHA каждого
результата и новые доказательства PNG-инспектора из серии V.
Решение перенесено из собственной ЛР3. При реализации и подготовке отчёта использован Codex;
автор работы — Костылев Артём. Нормативные README/COURSE/contracts/tests сохранены.
+4
View File
@@ -0,0 +1,4 @@
[alembic]
script_location = %(here)s/alembic
prepend_sys_path = %(here)s/src %(here)s/alembic
path_separator = space
+92
View File
@@ -0,0 +1,92 @@
import asyncio
from alembic import context
from alembic.script import ScriptDirectory
from legacy import adopt_legacy
from sqlalchemy import Connection, func, make_url, pool, select
from sqlalchemy.ext.asyncio import create_async_engine
from messenger.attachments.infrastructure.postgres import AttachmentRow
from messenger.identity.infrastructure.postgres import SessionRow, UserRow
from messenger.messaging.infrastructure.postgres import (
ChatMemberRow,
ChatRow,
IdempotencyKeyRow,
MessageAttachmentRow,
MessageRow,
)
from messenger.shared.config import PlatformSettings
LOCK_KEY = 20_260_002
MODELS = (
UserRow,
SessionRow,
ChatRow,
ChatMemberRow,
MessageRow,
IdempotencyKeyRow,
AttachmentRow,
MessageAttachmentRow,
)
target_metadata = UserRow.metadata
config = context.config
def include_object(_obj, name, _type, _reflected, _compare_to):
return name != "schema_migrations"
def run_migrations(connection: Connection) -> None:
with connection.engine.connect() as lock:
lock.execute(select(func.pg_advisory_lock(LOCK_KEY)))
lock.commit()
try:
with connection.begin():
adopt_legacy(connection, ScriptDirectory.from_config(config))
context.configure(
connection=connection,
target_metadata=target_metadata,
transaction_per_migration=True,
compare_type=True,
include_object=include_object,
)
with context.begin_transaction():
context.run_migrations()
finally:
lock.execute(select(func.pg_advisory_unlock(LOCK_KEY)))
lock.commit()
async def run_async_migrations() -> None:
dsn = config.attributes.get("database_url")
if dsn is None:
dsn = PlatformSettings().database_url.get_secret_value()
engine = create_async_engine(
make_url(dsn).set(drivername="postgresql+asyncpg"),
poolclass=pool.NullPool,
hide_parameters=True,
connect_args={
"timeout": 10,
"server_settings": {"lock_timeout": "10000", "statement_timeout": "60000"},
},
)
try:
async with engine.connect() as connection:
await connection.run_sync(run_migrations)
finally:
await engine.dispose()
if context.is_offline_mode():
context.configure(
dialect_name="postgresql",
target_metadata=target_metadata,
literal_binds=True,
transaction_per_migration=True,
)
with context.begin_transaction():
context.run_migrations()
elif config.attributes.get("connection") is not None:
run_migrations(config.attributes["connection"])
else:
asyncio.run(run_async_migrations())
+92
View File
@@ -0,0 +1,92 @@
from uuid import uuid4
from alembic.migration import MigrationContext
from alembic.operations import Operations
from alembic.script import ScriptDirectory
from sqlalchemy import Connection, MetaData, Table, func, inspect, select, text
from sqlalchemy.schema import CreateSchema
LEGACY = {
"0001_init": "9a13618f078ae9056431391e2e5197a31b981ae1fff047a9b3421ac750d3fd0d",
"0002_idempotency_keys": "d9159a35553249a1e51af5f2e4e7c39b014e97593c49748e13f3799839063dd0",
"0003_attachments": "c4417f560538a083efba1c67f660147dc94532fcbb2fa91f3f3e8258a0f01f2e",
"0004_attachment_quarantine": (
"541f5399d87fac3584c4102b4e0bf92c23cd372cc20a7d56dad03e771773202c"
),
}
SNAPSHOT = text("""
WITH relations AS (
SELECT c.* FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = :schema AND c.relkind IN ('r', 'p')
AND c.relname NOT IN ('schema_migrations', 'alembic_version')
)
SELECT 'table' AS kind, c.relname AS name,
concat_ws('|', c.relkind, c.relpersistence,
c.relrowsecurity, c.relforcerowsecurity) AS definition
FROM relations c
UNION ALL
SELECT 'column', c.relname || '.' || a.attname,
concat_ws('|', a.attnum, format_type(a.atttypid, a.atttypmod), a.attnotnull,
a.attidentity, a.attgenerated, a.attcollation::regcollation,
pg_get_expr(d.adbin, d.adrelid))
FROM relations c JOIN pg_attribute a ON a.attrelid = c.oid
LEFT JOIN pg_attrdef d ON d.adrelid = c.oid AND d.adnum = a.attnum
WHERE a.attnum > 0 AND NOT a.attisdropped
UNION ALL
SELECT 'constraint', c.relname || '.' || k.conname,
concat_ws('|', pg_get_constraintdef(k.oid), k.convalidated)
FROM relations c JOIN pg_constraint k ON k.conrelid = c.oid
UNION ALL
SELECT 'index', c.relname || '.' || x.relname,
concat_ws('|', pg_get_indexdef(i.indexrelid), i.indisvalid, i.indisready)
FROM relations c JOIN pg_index i ON i.indrelid = c.oid
JOIN pg_class x ON x.oid = i.indexrelid
UNION ALL
SELECT 'trigger', c.relname || '.' || t.tgname,
concat_ws('|', pg_get_triggerdef(t.oid), t.tgenabled)
FROM relations c JOIN pg_trigger t ON t.tgrelid = c.oid WHERE NOT t.tgisinternal
ORDER BY kind, name
""")
def snapshot(connection: Connection, schema: str) -> list[tuple[str, str, str]]:
return [
(kind, name, definition.replace(f"{schema}.", ""))
for kind, name, definition in connection.execute(SNAPSHOT, {"schema": schema})
]
def adopt_legacy(connection: Connection, scripts: ScriptDirectory) -> None:
migration = MigrationContext.configure(connection)
if migration.get_current_heads():
return
tables = set(inspect(connection).get_table_names(schema="public"))
if tables <= {"alembic_version"}:
return
if "schema_migrations" not in tables:
raise RuntimeError("Existing schema has no legacy migration journal")
journal = Table("schema_migrations", MetaData(), schema="public", autoload_with=connection)
applied = dict(connection.execute(select(journal.c.version, journal.c.checksum)).tuples().all())
expected = dict(list(LEGACY.items())[: len(applied)])
if applied != expected:
raise RuntimeError("Unknown or changed legacy migrations")
if not applied:
if tables - {"schema_migrations", "alembic_version"}:
raise RuntimeError("Existing schema has an empty migration journal")
return
revision = list(expected)[-1]
actual = snapshot(connection, "public")
reference = f"alembic_verify_{uuid4().hex}"
savepoint = connection.begin_nested()
try:
connection.execute(CreateSchema(reference))
connection.execute(select(func.set_config("search_path", reference, True)))
with Operations.context(MigrationContext.configure(connection)):
for script in reversed(list(scripts.walk_revisions(base="base", head=revision))):
script.module.upgrade()
wanted = snapshot(connection, reference)
finally:
savepoint.rollback()
if actual != wanted:
raise RuntimeError("Legacy schema differs from the recorded migrations")
migration.stamp(scripts, revision)
+16
View File
@@ -0,0 +1,16 @@
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
+87
View File
@@ -0,0 +1,87 @@
import sqlalchemy as sa
from alembic import op
revision = "0001_init"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"users",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("username", sa.Text(), nullable=False),
sa.Column("display_name", sa.Text(), nullable=False),
sa.Column("password_hash", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("password_hash LIKE '$argon2id$%'", name="users_password_hash_check"),
sa.CheckConstraint("username ~ '^[a-z0-9_]{3,32}$'", name="users_username_format"),
sa.CheckConstraint(
"char_length(display_name) BETWEEN 1 AND 100", name="users_display_name_check"
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("username", name="users_username_key"),
)
op.create_table(
"sessions",
sa.Column("token_hash", sa.LargeBinary(), nullable=False),
sa.Column("user_id", sa.Uuid(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("expires_at > created_at", name="sessions_check"),
sa.CheckConstraint("octet_length(token_hash) = 32", name="sessions_token_hash_check"),
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("token_hash"),
)
op.create_index("sessions_user_idx", "sessions", ["user_id"], unique=False)
op.create_table(
"chats",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("title", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("last_message_at", sa.DateTime(timezone=True), nullable=True),
sa.CheckConstraint("char_length(title) BETWEEN 1 AND 100", name="chats_title_check"),
sa.PrimaryKeyConstraint("id"),
)
op.create_table(
"chat_members",
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("user_id", sa.Uuid(), nullable=False),
sa.Column("position", sa.SmallInteger(), nullable=False),
sa.Column("chat_created_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("position BETWEEN 1 AND 100", name="chat_members_position_check"),
sa.ForeignKeyConstraint(["chat_id"], ["chats.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(
["user_id"],
["users.id"],
),
sa.PrimaryKeyConstraint("chat_id", "user_id"),
)
op.create_index(
"chat_members_user_idx",
"chat_members",
["user_id", "chat_created_at", "chat_id"],
unique=False,
)
op.create_table(
"messages",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("sender_id", sa.Uuid(), nullable=False),
sa.Column("text", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("char_length(text) BETWEEN 1 AND 4000", name="messages_text_check"),
sa.ForeignKeyConstraint(
["chat_id", "sender_id"],
["chat_members.chat_id", "chat_members.user_id"],
),
sa.PrimaryKeyConstraint("id"),
)
op.create_index(
"messages_history_idx", "messages", ["chat_id", "created_at", "id"], unique=False
)
def downgrade() -> None:
raise RuntimeError("Downgrade would delete application data")
+33
View File
@@ -0,0 +1,33 @@
import sqlalchemy as sa
from alembic import op
revision = "0002_idempotency_keys"
down_revision = "0001_init"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"idempotency_keys",
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("user_id", sa.Uuid(), nullable=False),
sa.Column("key", sa.Text(), nullable=False),
sa.Column("fingerprint", sa.LargeBinary(), nullable=False),
sa.Column("message_id", sa.Uuid(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("key ~ '^[A-Za-z0-9._:-]{1,128}$'", name="idempotency_keys_key_check"),
sa.CheckConstraint(
"octet_length(fingerprint) = 32", name="idempotency_keys_fingerprint_check"
),
sa.ForeignKeyConstraint(
["chat_id", "user_id"],
["chat_members.chat_id", "chat_members.user_id"],
),
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("chat_id", "user_id", "key"),
)
def downgrade() -> None:
raise RuntimeError("Downgrade would delete application data")
+77
View File
@@ -0,0 +1,77 @@
import sqlalchemy as sa
from alembic import op
revision = "0003_attachments"
down_revision = "0002_idempotency_keys"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.drop_constraint("messages_text_check", "messages", type_="check")
op.create_check_constraint("messages_text_check", "messages", "char_length(text) <= 4000")
op.create_table(
"attachments",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("owner_id", sa.Uuid(), nullable=False),
sa.Column("filename", sa.Text(), nullable=False),
sa.Column("content_type", sa.Text(), nullable=False),
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
sa.Column("sha256", sa.Text(), nullable=False),
sa.Column("width", sa.Integer(), nullable=False),
sa.Column("height", sa.Integer(), nullable=False),
sa.Column("object_key", sa.Text(collation="C"), nullable=False),
sa.Column("status", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key", name="attachments_object_key_key"),
sa.UniqueConstraint("id", "chat_id", name="attachments_chat_key"),
sa.ForeignKeyConstraint(
["chat_id", "owner_id"],
["chat_members.chat_id", "chat_members.user_id"],
name="attachments_owner_fkey",
),
sa.CheckConstraint(
"char_length(filename) BETWEEN 1 AND 255", name="attachments_filename_check"
),
sa.CheckConstraint(
"content_type IN ('image/png', 'image/jpeg')", name="attachments_content_type_check"
),
sa.CheckConstraint(
"size_bytes BETWEEN 1 AND 10485760", name="attachments_size_bytes_check"
),
sa.CheckConstraint("sha256 ~ '^[0-9a-f]{64}$'", name="attachments_sha256_check"),
sa.CheckConstraint("width > 0", name="attachments_width_check"),
sa.CheckConstraint("height > 0", name="attachments_height_check"),
sa.CheckConstraint("status IN ('pending', 'ready')", name="attachments_status_check"),
)
op.create_index(
"attachments_pending_idx",
"attachments",
["created_at"],
postgresql_where=sa.text("status = 'pending'"),
)
op.create_table(
"message_attachments",
sa.Column("message_id", sa.Uuid(), nullable=False),
sa.Column("position", sa.SmallInteger(), nullable=False),
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("attachment_id", sa.Uuid(), nullable=False),
sa.PrimaryKeyConstraint("message_id", "position"),
sa.UniqueConstraint("message_id", "attachment_id"),
sa.CheckConstraint("position BETWEEN 0 AND 9", name="message_attachments_position_check"),
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(
["attachment_id", "chat_id"],
["attachments.id", "attachments.chat_id"],
name="message_attachments_attachment_fkey",
ondelete="RESTRICT",
),
)
op.create_index("message_attachments_attachment_idx", "message_attachments", ["attachment_id"])
def downgrade() -> None:
raise RuntimeError("Downgrade would delete application data")
@@ -0,0 +1,28 @@
import sqlalchemy as sa
from alembic import op
revision = "0004_attachment_quarantine"
down_revision = "0003_attachments"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("attachments", sa.Column("quarantine_reason", sa.Text(), nullable=True))
op.create_check_constraint(
"attachments_quarantine_reason_check",
"attachments",
"quarantine_reason IN ('missing', 'size_mismatch', 'sha_mismatch')",
)
op.add_column(
"attachments", sa.Column("quarantined_at", sa.DateTime(timezone=True), nullable=True)
)
op.create_check_constraint(
"attachments_quarantine_check",
"attachments",
"(quarantine_reason IS NULL) = (quarantined_at IS NULL)",
)
def downgrade() -> None:
raise RuntimeError("Downgrade would delete quarantine data")
+3 -1
View File
@@ -1,10 +1,12 @@
# Topology worksheet, deliberately NOT a runnable stack (no application supplied). # Topology worksheet, deliberately NOT a runnable stack (no application supplied).
# Implement compose.yaml or stack.yaml; do not submit this empty file as deployment. # Implement compose.yaml or stack.yaml; do not submit this empty file as deployment.
# Service names below are logical roles, not imposed container names. # Service names below are logical roles, not imposed container names.
x-lab-number: 2 x-lab-number: 4
x-required-roles: x-required-roles:
- api - api
- postgres - postgres
- load_balancer
- s3
# TODO: implement real images/builds, network wiring, volumes, healthchecks, # TODO: implement real images/builds, network wiring, volumes, healthchecks,
# credentials and bootstrap. Details: contracts/infrastructure.json and README.md. # credentials and bootstrap. Details: contracts/infrastructure.json and README.md.
services: {} services: {}
+322
View File
@@ -0,0 +1,322 @@
x-app: &app
image: ${API_IMAGE:-mtusi-messenger:${COMPOSE_PROJECT_NAME}}
networks: [backend]
x-s3-env: &s3-env
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
S3_CONNECT_TIMEOUT_SECONDS: ${S3_CONNECT_TIMEOUT_SECONDS:-1}
S3_READ_TIMEOUT_SECONDS: ${S3_READ_TIMEOUT_SECONDS:-2}
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
IMAGE_CHECK_CONCURRENCY: ${IMAGE_CHECK_CONCURRENCY:-2}
UPLOAD_CONCURRENCY: ${UPLOAD_CONCURRENCY:-3}
UPLOAD_GRACE_SECONDS: ${UPLOAD_GRACE_SECONDS:-900}
UPLOAD_SWEEP_INTERVAL_SECONDS: ${UPLOAD_SWEEP_INTERVAL_SECONDS:-30}
UPLOAD_SWEEP_BATCH: ${UPLOAD_SWEEP_BATCH:-100}
x-s3-client: &s3-client
image: rustfs/rc:v0.1.36@sha256:ab024bfebee49a750ce886b4c70963ccd9ddaa03f491704a90710641d7a26699
read_only: true
tmpfs: ["/tmp:size=16m"]
environment:
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
RC_CONFIG_DIR: /tmp/rc
user: "65534:65534"
volumes: ["./infra/s3:/s3:ro"]
secrets:
- s3_root_user
- s3_root_password
- s3_access_key_id
- s3_secret_access_key
- s3_reconciler_access_key_id
- s3_reconciler_secret_access_key
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]
restart: "no"
networks: [backend]
deploy:
resources:
limits: {cpus: "0.5", memory: 64m}
services:
lb:
image: haproxy:3.2.26-alpine@sha256:5893840576071e6217cd52ebdcc14f454c6f7a631056fa6f4e0f40d3cac4215b
environment:
API_SERVER_MAXCONN: ${API_SERVER_MAXCONN:-64}
volumes: [./infra/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro]
ports:
- "127.0.0.1:${LB_HTTP_PORT:-8080}:8080"
- "127.0.0.1:${LB_STATS_PORT:-8404}:8404"
networks: [edge, backend]
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8404/healthz"]
interval: 2s
timeout: 2s
retries: 3
start_period: 60s
start_interval: 1s
sysctls:
net.ipv4.ip_local_port_range: "10240 65535"
net.ipv4.tcp_tw_reuse: "1"
stop_signal: SIGUSR1
stop_grace_period: 8s
restart: unless-stopped
read_only: true
cap_drop: [ALL]
deploy:
resources:
limits: {cpus: "${LB_CPUS:-1}", memory: 128m}
api:
<<: *app
build: .
environment:
<<: *s3-env
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
WEB_CONCURRENCY: ${WEB_CONCURRENCY:-2}
DB_POOL_MAX_SIZE: ${DB_POOL_MAX_SIZE:-8}
REQUEST_TIMEOUT_SECONDS: ${REQUEST_TIMEOUT_SECONDS:-8}
MIN_TRANSFER_RATE_BYTES: ${MIN_TRANSFER_RATE_BYTES:-131072}
DB_ACQUIRE_TIMEOUT_SECONDS: ${DB_ACQUIRE_TIMEOUT_SECONDS:-1}
DB_COMMAND_TIMEOUT_SECONDS: ${DB_COMMAND_TIMEOUT_SECONDS:-2}
DRAIN_SECONDS: ${DRAIN_SECONDS:-2}
SHUTDOWN_TIMEOUT_SECONDS: ${SHUTDOWN_TIMEOUT_SECONDS:-6}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
MALLOC_MMAP_THRESHOLD_: ${MALLOC_MMAP_THRESHOLD_:-131072}
secrets: [database_url, cache_url, cursor_secret, s3_access_key_id, s3_secret_access_key]
healthcheck:
test: ["CMD", "python", "-c", "import sys,urllib.request;sys.exit(urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=2).status!=200)"]
interval: 5s
timeout: 3s
retries: 3
start_period: 30s
start_interval: 200ms
stop_grace_period: 8s
restart: unless-stopped
read_only: true
tmpfs: [/tmp]
sysctls:
net.ipv4.tcp_rmem: "4096 65536 262144"
net.ipv4.tcp_wmem: "4096 65536 262144"
depends_on:
migrate:
condition: service_completed_successfully
valkey:
condition: service_started
s3-init:
condition: service_completed_successfully
deploy:
replicas: ${API_REPLICAS:-2}
resources:
limits: {cpus: "${API_CPUS:-1}", memory: "${API_MEMORY:-512m}"}
migrate:
<<: *app
pull_policy: never
init: true
command: ["alembic", "upgrade", "head"]
environment:
DATABASE_URL_FILE: /run/secrets/database_url
LOG_LEVEL: ${LOG_LEVEL:-INFO}
secrets: [database_url]
healthcheck:
disable: true
restart: "no"
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits: {cpus: "0.5", memory: 256m}
postgres:
image: postgres:17.11-alpine@sha256:266d3c257594e195b309ae9d1805e54c788f1d7b375f56bf6c367aac6c917e7b
user: "70:70"
command:
- postgres
- -c
- max_connections=${PG_MAX_CONNECTIONS:-200}
- -c
- shared_buffers=${PG_SHARED_BUFFERS:-256MB}
- -c
- shared_preload_libraries=pg_stat_statements
- -c
- pg_stat_statements.track=top
- -c
- track_io_timing=on
- -c
- idle_in_transaction_session_timeout=10s
environment:
POSTGRES_DB: messenger
POSTGRES_USER: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets: [postgres_password]
volumes: [pgdata:/var/lib/postgresql/data]
shm_size: 256m
cpuset: ${PG_CPUSET:-}
healthcheck:
test: ["CMD", "pg_isready", "-h", "127.0.0.1", "-U", "messenger", "-d", "messenger"]
interval: 2s
timeout: 3s
retries: 30
start_period: 10s
start_interval: 200ms
stop_grace_period: 10s
restart: unless-stopped
networks: [backend]
deploy:
resources:
limits: {cpus: "${PG_CPUS:-2}", memory: "${PG_MEMORY:-1g}"}
valkey:
image: valkey/valkey:8.1.11-alpine@sha256:2602ace2be6569bc2bbb06f873776516ab9089821525be34f4305c67eb9db02a
user: "999:1000"
command:
- valkey-server
- /run/secrets/valkey.conf
- --appendonly
- "yes"
- --appendfsync
- always
- --maxmemory
- 64mb
- --maxmemory-policy
- volatile-lru
secrets: [valkey.conf]
volumes: [valkeydata:/data]
healthcheck:
test: ["CMD-SHELL", "REDISCLI_AUTH=\"$$(sed -n 's/^requirepass //p' /run/secrets/valkey.conf)\" valkey-cli ping | grep -q PONG"]
interval: 2s
timeout: 3s
retries: 30
stop_grace_period: 10s
restart: unless-stopped
networks: [backend]
deploy:
resources:
limits: {cpus: "0.5", memory: 128m}
s3:
image: rustfs/rustfs:1.0.4@sha256:ed9210d6411d77ffceb70d94b964a3cb7d225983368bc25bebe01e995293dcc2
environment:
RUSTFS_ACCESS_KEY_FILE: /run/secrets/s3_root_user
RUSTFS_SECRET_KEY_FILE: /run/secrets/s3_root_password
RUSTFS_ADDRESS: ":9000"
RUSTFS_CONSOLE_ENABLE: "false"
RUSTFS_REGION: ${S3_REGION:-us-east-1}
RUSTFS_OBS_LOG_DIRECTORY: ""
RUSTFS_OBS_LOGGER_LEVEL: ${S3_LOG_LEVEL:-warn}
secrets: [s3_root_user, s3_root_password]
volumes: [s3data:/data]
cpuset: ${S3_CPUSET:-}
read_only: true
tmpfs: ["/tmp:size=64m"]
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]
healthcheck:
test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:9000/health/ready"]
interval: 2s
timeout: 3s
retries: 30
start_period: 20s
stop_grace_period: 10s
restart: unless-stopped
networks: [backend]
deploy:
resources:
limits: {cpus: "${S3_CPUS:-1}", memory: "${S3_MEMORY:-512m}"}
reconciler:
<<: *app
pull_policy: never
profiles: [reconcile]
init: true
entrypoint: ["python", "-m", "messenger.attachments.entrypoints.reconcile"]
command: ["--dry-run"]
environment:
<<: *s3-env
DATABASE_URL_FILE: /run/secrets/database_url
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_reconciler_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_reconciler_secret_access_key
S3_READ_TIMEOUT_SECONDS: ${RECONCILE_S3_READ_TIMEOUT_SECONDS:-10}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
secrets: [database_url, s3_reconciler_access_key_id, s3_reconciler_secret_access_key]
healthcheck:
disable: true
restart: "no"
read_only: true
cpuset: ${RECONCILE_CPUSET:-}
depends_on:
postgres:
condition: service_healthy
s3:
condition: service_healthy
deploy:
resources:
limits: {cpus: "${RECONCILE_CPUS:-1}", memory: "${RECONCILE_MEMORY:-256m}"}
s3-init:
<<: *s3-client
entrypoint: ["/bin/sh", "/s3/init.sh"]
depends_on:
s3:
condition: service_healthy
s3-cli:
<<: *s3-client
profiles: [tools]
entrypoint: ["/bin/sh", "/s3/cli.sh"]
k6:
image: grafana/k6:2.3.0@sha256:2dd4756ec6e14e62778916e1eb850108e4b499304c02df56760fa0bdb0fba0b8
profiles: [bench]
environment:
BASE_URL: http://lb:8080
volumes: [./infra/k6:/scripts:ro]
networks: [edge]
deploy:
resources:
limits: {cpus: "${K6_CPUS:-2}", memory: "${K6_MEMORY:-512m}"}
secrets:
postgres_password:
file: ./secrets/postgres_password
database_url:
file: ./secrets/database_url
valkey.conf:
file: ./secrets/valkey.conf
cache_url:
file: ./secrets/cache_url
cursor_secret:
file: ./secrets/cursor_secret
s3_root_user:
file: ./secrets/s3_root_user
s3_root_password:
file: ./secrets/s3_root_password
s3_access_key_id:
file: ./secrets/s3_access_key_id
s3_secret_access_key:
file: ./secrets/s3_secret_access_key
s3_reconciler_access_key_id:
file: ./secrets/s3_reconciler_access_key_id
s3_reconciler_secret_access_key:
file: ./secrets/s3_reconciler_secret_access_key
volumes:
pgdata:
valkeydata:
s3data:
networks:
edge:
backend:
internal: true
+4 -2
View File
@@ -34,7 +34,8 @@
"chat_id": "33333333-3333-4333-8333-333333333333", "chat_id": "33333333-3333-4333-8333-333333333333",
"sender_id": "11111111-1111-4111-8111-111111111111", "sender_id": "11111111-1111-4111-8111-111111111111",
"text": "Привет, Боб!", "text": "Привет, Боб!",
"created_at": "2026-09-01T12:00:00Z" "created_at": "2026-09-01T12:00:00Z",
"attachment_ids": []
}, },
"MessagePage": { "MessagePage": {
"items": [ "items": [
@@ -43,7 +44,8 @@
"chat_id": "33333333-3333-4333-8333-333333333333", "chat_id": "33333333-3333-4333-8333-333333333333",
"sender_id": "11111111-1111-4111-8111-111111111111", "sender_id": "11111111-1111-4111-8111-111111111111",
"text": "Привет, Боб!", "text": "Привет, Боб!",
"created_at": "2026-09-01T12:00:00Z" "created_at": "2026-09-01T12:00:00Z",
"attachment_ids": []
} }
], ],
"next_cursor": null "next_cursor": null
+17 -2
View File
@@ -1,10 +1,10 @@
{ {
"lab": 2, "lab": 4,
"kind": "requirements, not runnable deployment", "kind": "requirements, not runnable deployment",
"required_for_grade_3": [ "required_for_grade_3": [
{ {
"role": "api", "role": "api",
"count_min": 1, "count_min": 2,
"student_implements": true, "student_implements": true,
"persistent_local_state": false "persistent_local_state": false
}, },
@@ -12,6 +12,21 @@
"role": "postgres", "role": "postgres",
"purpose": "users/chats/membership/messages/metadata", "purpose": "users/chats/membership/messages/metadata",
"persistent_volume": true "persistent_volume": true
},
{
"role": "load_balancer",
"public_entrypoint": true,
"backend_readiness": true
},
{
"role": "s3",
"choose_one": [
"MinIO",
"SeaweedFS",
"RustFS"
],
"private_bucket": true,
"persistent_volume": true
} }
], ],
"optional_components": [ "optional_components": [
+1487 -7
View File
File diff suppressed because it is too large Load Diff
+110 -3
View File
@@ -1,5 +1,112 @@
# Доказательства # Индекс доказательств ЛР4
Храните небольшие обезличенные результаты: команды и stdout проверок, JSON/CSV агрегатов нагрузки, EXPLAIN, dashboard JSON, текст разбора отказа. Укажите дату, commit, аппаратные лимиты и профиль нагрузки. Финальный validation snapshot: `5e49502ae78fa48b992c97418e490666c699726d`, 2026-11-06.
Runtime совпадает с review `7a0ebac2aa9e1d8602054ca4c7c1b0c23d3bfad4`;
[аудит применимости и новые проверки](load-validation.md). Сохранённые cost и текстовая
нагрузка приняты с исходными SHA. Старые upload-memory остаются историческими.
Не добавляйте .env, session tokens, cookies, приватные ключи, credentials, реальные сообщения, raw dumps и signed URLs. Для скриншотов скрывайте секреты. Большие логи/бинарные traces сохраняйте отдельно по правилам преподавателя; одного скриншота без методики недостаточно. ## Финальная validation после PNG-фиксов
| Команда / назначение | Наблюдение | Файл |
| --- | --- | --- |
| `make student-test PYTEST_ARGS='-q --tb=short'`, затем адресный повтор двух S3-тестов | 719 PASS + 2 setup errors RustFS; только эти два повторены: 2 PASS. 721 выбранная проверка исполнена; первый запуск не объявлен зелёным | [полный](png-final-2026-11-06/student-test.txt), [повтор](png-final-2026-11-06/student-s3-retry.txt) |
| `make up`, `make test`, `scripts/limits.py` | Новый изолированный project, fresh volumes, штатные лимиты, smoke PASS | [up](png-final-2026-11-06/up.txt), [smoke](png-final-2026-11-06/smoke.txt), [лимиты](png-final-2026-11-06/limits.txt) |
| Прямой argv `make upload-faults`, новый путь захвата | 7 PASS; оба PNG → 415; негативный suite: 0×5xx | [raw](png-final-2026-11-06/upload-faults.txt) |
| `png-roundtrip.py` | 415 без новых строк/объектов; PNG/JPEG → 201 и побайтное скачивание | [raw](png-final-2026-11-06/png-roundtrip.json) |
| `make upload-memory ARGS='--slots 3 --out <новый-runN>'`, 3 серии | 18 точек, 840/840×201, API ≤440 MiB, LB ≤31,1 MiB, OOM 0 | [сводка](png-final-2026-11-06/memory-summary.json), [raw](png-final-2026-11-06/upload-memory/) |
| `high-depth.py` | 10 параллельных 40 MP RGBA16, 10×201, checksum совпал, API ≤270,828 MiB, OOM 0 | [raw](png-final-2026-11-06/high-depth.json) |
| `scripts/reconcile_check.py` после RAM | 859 ready = 859 объектов, расхождений/остатков нет; shallow | [raw](png-final-2026-11-06/post-memory-invariants.json) |
| `concurrent-reconcile.py` (только under_load) | 3341×201, 67 проходов, 3 orphan удалены; deep SHA/lost_ready=0. Pending инъекция 0 на пустом dataset, прежний сценарий F сохранён | [raw](png-final-2026-11-06/concurrent-reconcile/under-load.json), [oracle](png-final-2026-11-06/concurrent-reconcile/check-under-load.json) |
| Повторный аудит 27 cost-замеров | Приняты все 27; страницы/память подтверждены, линейность времени отвергнута | [аудит](png-final-2026-11-06/cost-reaudit.json), [diff/SHA](png-final-2026-11-06/retained-audit.json) |
## Offline-проверки до финальной validation
| Команда / назначение | Результат | SHA и файлы |
| --- | --- | --- |
| `make student-test PYTEST_ARGS='-q --tb=short student-tests/unit'` | 407 passed; оба повреждённых PNG отклоняются до записи | Код `7c8a1a64899dc0a1b5fa0949c0992ef40722eacf`; [unit](png-header-fix/unit.txt), [227 проверок инспектора/ASGI](png-header-fix/targeted.txt), [blob-манифест](png-header-fix/manifest.json) |
| `make check`, `make validate`, `make lint`, `openspec validate --all --strict --no-interactive` после фикса | PASS, 18 спецификаций после архивации | Архив `7a0ebac2aa9e1d8602054ca4c7c1b0c23d3bfad4`; [протокол](png-header-fix/NOTES.md), [OpenSpec](png-header-fix/openspec-final.txt) |
| Статическое/offline-ревью runner, round3 | Открытых major нет, один minor; Docker/нагрузки не выполнялись | Результат передан runner для `7a0ebac2aa9e1d8602054ca4c7c1b0c23d3bfad4`; отдельного raw-файла ревью в репозитории нет |
| Документационный аудит: check/validate/lint/OpenSpec, числа, ссылки, diff | Проверки отчётного этапа отдельно от предыдущих тестов | [Аудит](report-audit-2026-11-06.md); исходный snapshot указан выше |
| Финальные API/LB/RustFS-регрессии, память нового инспектора и upload при сверке | Выполненные результаты финальной фазы вынесены выше и в аудит применимости | [Общий долг двух PNG-фиксов](png-header-fix/NOTES.md#обязательные-проверки-финального-этапа) |
Первый PNG-фикс: `9387c3d82ab77ea376eecce19a36a22803d35efb`,
[причина и offline-доказательства](png-validation-fix/NOTES.md).
Второй фикс требует единственный IHDR; вместе они добавляют распаковку PNG ограниченными
блоками. Все предыдущие upload-memory и конкурентные upload в исторических разделах ниже
относятся к старому инспектору. Collection API/stack в `png-header-fix/` — сбор тестов, не их исполнение.
## После позднего переноса ЛР3, до PNG-фиксов
Runtime SHA **F**: `75b93bbba5f29575d3fca1d013cc7e0770d88082`, архив
`8bc2da2c094afc7e5f5e97c7beb5f278e76253fd`. Источник — immutable snapshot ЛР3
`ba78cf6da439a4ec60fd59389583b38285b0b2b1`, диапазон от `f591fa64e525679a31c9db7f942d36c789e1a12f`.
[Все 214 путей и область применимости](forward-port.md), [source-manifest](forward-final-2026-11-06/source-manifest.json).
Все успешные команды таблицы — на F; их полный argv, UTC, разрешённое окружение и exit code
находятся в raw. Пользователи, данные и файлы синтетические.
| Проверка / команда | Наблюдение | Файл |
| --- | --- | --- |
| `make check`, `make validate`, `make lint`, `make student-test` | PASS, 492 теста / 19 stack deselected | [offline-checks-rerun](forward-final-2026-11-06/offline-checks-rerun.txt) |
| `make up`, `make test` | Обе реплики healthy; smoke/checksum/ACL PASS | [up](forward-final-2026-11-06/up.txt), [smoke](forward-final-2026-11-06/smoke.txt), [после нагрузок](forward-final-2026-11-06/final-smoke.txt) |
| `sh scripts/persistence-evidence.sh http://127.0.0.1:18084` | PASS; StartedAt API/LB/PG/Valkey/S3 изменились | [persistence-rerun](forward-final-2026-11-06/persistence-rerun.txt) |
| `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | Target exited; 130 записей, 0 ошибок, 2,883 с до первой записи | [команда](forward-final-2026-11-06/failover-command.txt), [вывод/inspect](forward-final-2026-11-06/failover-d4c8d53d7433.txt) |
| `make scenario SCENARIO=failover-upload`; отдельный `make restore` | Другая реплика отдала те же байты; восстановление выполнено отдельно | [команда](forward-final-2026-11-06/failover-upload-command.txt), [JSON](forward-final-2026-11-06/failover-upload.json), [restore](forward-final-2026-11-06/restore.txt) |
| `make scenario SCENARIO=s3-outage` | 503 за 4,990/5,329 с; текст 201; checksum после старта совпал | [команда](forward-final-2026-11-06/s3-outage-command.txt), [JSON](forward-final-2026-11-06/s3-outage.json) |
| `make test-faults` | 19 PASS, 112 ответов негативной файловой проверки без 5xx; компенсация/crash/grace/stall/PG/Valkey | [stack-faults](forward-final-2026-11-06/stack-faults.txt) |
| `student-tests/tools/forward_network.py`; HAProxy fixture | Сессии/NUL/Basic PASS; 12 проверок реальных 502/503/504 и request_id | [сеть](forward-final-2026-11-06/network-regressions.txt), [LB](forward-final-2026-11-06/haproxy-errors.txt) |
| `make limits` | Ресурсы, порты, mounts, сети и роли секретов PASS | [limits](forward-final-2026-11-06/limits.txt) |
| `make load-evidence` | 3 повтора, медиана 1456,338 RPS, 0 ошибок; текстовый путь не менялся | [команда](forward-final-2026-11-06/load-command.txt), [вывод](forward-final-2026-11-06/load.txt), [raw JSON](forward-final-2026-11-06/load/) |
| `scripts/upload_memory.py --slots 3 --sizes 10485760 --levels 40 --out ...` | 3×80: 238×201, 2×503; пик API 456,2 MiB, OOM 0. Историческая память до PNG-фиксов | [run1](forward-final-2026-11-06/memory-1.txt), [run2](forward-final-2026-11-06/memory-2.txt), [run3](forward-final-2026-11-06/memory-3.txt), [raw](forward-final-2026-11-06/memory/) |
| `make reconcile-demo ARGS='--out evidence/forward-final-2026-11-06/reconcile-demo'` | 13 PASS; 3364 upload, 59 проходов, lost_ready 0. Конкурентные upload требуют повторения после PNG-фиксов | [команда](forward-final-2026-11-06/reconcile-demo.txt), [demo](forward-final-2026-11-06/reconcile-demo/demo.json), [deep-аудит](forward-final-2026-11-06/reconcile-demo/check-under-load.json) |
| `git diff --exit-code` измеряемых файлов | Стоимость reconciler сохранена с исходным SHA, не переснята на F | [retained-cost-audit](forward-final-2026-11-06/retained-cost-audit.txt) |
| Воспроизведение непарных surrogate в text | Два HTTP 500: известный **minor**, не успешный тест | [HTTP](forward-final-2026-11-06/known-minor-http.txt), [offline](forward-final-2026-11-06/known-minor.txt), [finding](forward-final-2026-11-06/findings.json) |
| `make down` своих project, fixture `docker compose down`, инвентаризация по labels | 0 контейнеров трёх project, 6 volumes сохранены | [main](forward-final-2026-11-06/main-down.txt), [demo](forward-final-2026-11-06/reconcile-down.txt), [итог](forward-final-2026-11-06/shutdown-state.txt) |
[Сводка чисел](forward-final-2026-11-06/measurement-summary.json),
[команда агрегации](forward-final-2026-11-06/measurement-summary.txt),
[фон до](forward-final-2026-11-06/performance-before.txt) и
[после](forward-final-2026-11-06/performance-after.txt).
Сохранены неуспешные попытки: [ty до исправления обвязки](forward-final-2026-11-06/offline-checks.txt)
и [неверный BASE_URL до hook](forward-final-2026-11-06/persistence-command.txt); они не считаются PASS.
## Исходная SQLAlchemy-серия
Runtime основы: `8e4532f6f5c05d902b6e2c310f54320d95585904`. Все результаты ниже сняты
**до F и обоих PNG-фиксов**. Обвязка менялась отдельно; точный SHA каждого захвата записан
в его первой строке/JSON. [Общий протокол](sqlalchemy-2026-11-06/README.md).
| Команда / критерий | Наблюдение | SHA обвязки и файлы |
| --- | --- | --- |
| `make test`, `make limits`, `make attachments-evidence`, LB/retry/отказы | Уровень 3 на SQLAlchemy, 404 собственных теста | `1aec062aba4560b51a6d9155571c9610bc777d7e`, `763b0255575366082c4c6731b54611af717685a5`; [grade3](sqlalchemy-2026-11-06/grade3/NOTES.md) |
| `make s3-check` | 31 PASS; private/IAM/bootstrap не менялись с этой серии | `763b0255575366082c4c6731b54611af717685a5`; [сырьё](sqlalchemy-2026-11-06/grade3/s3-security.txt) |
| `make upload-memory`, три полные матрицы | 36 точек, штатные 3 слота 840/840×201, пик 448 MiB, OOM 0; контроль 64 слота 27×503 | `6c706a727ec7b4c5f8254e19401705847e8e9c78`, `cf41f7c64dcf1dea6da0643f5793bd2e5dd1f03e`; [методика](sqlalchemy-2026-11-06/grade4/NOTES.md), [raw](sqlalchemy-2026-11-06/grade4/memory/) |
| `sh scripts/clean-clone.sh` | 76,866 с, проверки прошли, три volumes сохранены | `cf41f7c64dcf1dea6da0643f5793bd2e5dd1f03e`; [команда/вывод](sqlalchemy-2026-11-06/grade4/clean-clone-command.txt) |
| `make overload-leftovers` | Без сдвига времени убраны 30 pending после grace 900 с; 1669 ready/объектов совпали | SHA в [команде](sqlalchemy-2026-11-06/grade4/leftovers-command.txt); [вывод](sqlalchemy-2026-11-06/grade4/overload-leftovers.txt) |
| `make load-evidence` | 3 прогона, медиана 1529,186 RPS, 0 ошибок; историческая серия до F | SHA в [команде](sqlalchemy-2026-11-06/grade4/load-command.txt); [raw](sqlalchemy-2026-11-06/grade4/load/) |
| `make reconcile-demo`, отдельный отказ PG | 13 PASS, 3331 upload, lost_ready 0; обновлён demo серии F | `827b783be355be9fa60117dc3eda350af3dab454`, `3402fd69d4922e500035a204d957be2cccaeb85a`; [протокол](sqlalchemy-2026-11-06/track5/NOTES.md), [PG](sqlalchemy-2026-11-06/track5/pg-outage.txt) |
| `make reconcile-cost` | **Сохраняемые 27 замеров** до 100k; путь обхода неизменён | `3402fd69d4922e500035a204d957be2cccaeb85a`; [команда](sqlalchemy-2026-11-06/track5/cost-command.txt), [raw/план/фон](sqlalchemy-2026-11-06/track5/measurements/cost/), [расчёт](sqlalchemy-2026-11-06/track5/analysis.json) |
| Финальные проверки исходной реализации | 410 тестов, check/validate/lint/smoke/limits PASS; не тесты нового PNG | `504f77cb7cef28202e1830fc04e10e646488e778`; [вывод](sqlalchemy-2026-11-06/track5/final-checks.txt), [остановка 4 project / 12 volumes](sqlalchemy-2026-11-06/track5/shutdown-state.txt) |
[Закрытие 21 замечания исходной реализации](sqlalchemy-2026-11-06/REVIEW-CLOSURE.md)
относится к тому этапу. Оно предшествует независимым PNG-находкам и не отменяет их историю.
## Историческая реализация asyncpg
**Эти результаты не подтверждают нынешний SQLAlchemy-код или PNG-инспектор.**
Основная серия и старый clean-clone: `81dd816df21f820a7fd0fa4269cc375d8fc44155`;
reconciler: `61dcb6bf13f3d0853d36788930d79bca5f68ece6`. Дата и измеренный SHA
сохранены в raw. Старые команды удаления volumes — история, не инструкция для повторения.
| Историческая команда | Сохранённые файлы |
| --- | --- |
| `make check`, `make validate`, `make student-test`, `make test` | [контракт](contract-checks.txt), [тесты](student-tests.txt), [smoke](smoke.txt) |
| `make clean-clone-evidence`, `make stack-evidence`, `make limits`, `make attachments-evidence` | [clone](clean-clone.txt), [стенд](stack.txt), [окружение](environment.txt), [лимиты](limits.txt), [размещение](storage-layout.txt) |
| `make lb-check`, `make retry-check`, persistence и сценарии отказов | [LB](lb-check.json), [retry](retry.json), [persistence](persistence.txt), [upload failover](failover-upload.json), [S3 outage](s3-outage.json) |
| `make upload-faults`, `make faults-evidence`, `make s3-check` | [upload](upload-faults.txt), [stack](faults.txt), [S3](s3-security.txt) |
| `make upload-memory`, `make overload-leftovers`, `make load-evidence` | [память](upload-memory/), [остатки](overload-leftovers.txt), [нагрузка](load.txt), [raw](load/), [отброшенная попытка с хоста](load/host-attempt.txt) |
| `make reconcile-demo`, `make reconcile-cost` | [методика и все серии](reconcile/README.md), [raw](reconcile/), [cost](reconcile/cost/) |
Прежние cost-серии `b1916236f4391bfdf7d7718ebf741621047788cf` (лишняя SQL-страница),
`5ca2cdc9af6016df8d6f1be8bd4681f06fada334` (без фильтра фона) и прерванные наполнения
описаны в [историческом индексе reconciler](reconcile/README.md). Они не подмешиваются
к 27 принятым измерениям SQLAlchemy и не удаляются.
+132
View File
@@ -0,0 +1,132 @@
date: 2026-10-30T04:27:15Z commit: 81dd816 dir: mtusi2026lab4-clean (свой compose-проект и volumes)
# шаги — команды блока «Запуск из чистого clone» REPORT.md; строки с пометкой «наблюдение» — проверки между шагами
$ git clone <repo> mtusi2026lab4-clean && cd mtusi2026lab4-clean && git checkout solution/bpi2303-kostylev
HEAD 81dd816 Archive the storage reconciler change
$ docker volume ls -q --filter label=com.docker.compose.project=mtusi2026lab4-clean | wc -l | tr -d ' ' # наблюдение, не шаг инструкции
0
$ make secrets
sh scripts/secrets.sh
создан secrets/postgres_password
создан secrets/database_url
создан secrets/cursor_secret
создан secrets/valkey_password
создан secrets/valkey.conf
создан secrets/cache_url
создан secrets/s3_root_user
создан secrets/s3_root_password
создан secrets/s3_access_key_id
создан secrets/s3_secret_access_key
создан .env из .env.example
$ sed LB_HTTP_PORT=18084 LB_STATS_PORT=24084 .env # отступление: 8080 на этой машине занят стендами других лаб
$ git status --short --ignored secrets .env # наблюдение, не шаг инструкции
!! .env
!! secrets/
$ docker compose --env-file .env config --quiet && echo valid
valid
$ make up
Container mtusi2026lab4-clean-postgres-1 Healthy
Container mtusi2026lab4-clean-s3-1 Healthy
Container mtusi2026lab4-clean-s3-init-1 Exited
Container mtusi2026lab4-clean-migrate-1 Exited
Container mtusi2026lab4-clean-postgres-1 Healthy
Container mtusi2026lab4-clean-migrate-1 Exited
Container mtusi2026lab4-clean-valkey-1 Healthy
Container mtusi2026lab4-clean-s3-1 Healthy
Container mtusi2026lab4-clean-s3-init-1 Exited
Container mtusi2026lab4-clean-api-1 Healthy
Container mtusi2026lab4-clean-api-2 Healthy
Container mtusi2026lab4-clean-lb-1 Healthy
make up: 11 s (сборка образа из кэша слоёв)
$ docker compose ps -a --format '{{.Service}} {{.State}} {{.Status}} {{.Ports}}' # наблюдение, не шаг инструкции
api running Up 4 seconds (healthy)
api running Up 4 seconds (healthy)
lb running Up 10 seconds (healthy) 127.0.0.1:18084->8080/tcp, 127.0.0.1:24084->8404/tcp
migrate exited Exited (0) 8 seconds ago
postgres running Up 10 seconds (healthy)
s3 running Up 10 seconds (healthy)
s3-init exited Exited (0) 4 seconds ago
valkey running Up 10 seconds (healthy)
$ docker compose logs --no-log-prefix migrate s3-init | grep -v '^$' | tail -8 # наблюдение, не шаг инструкции
{"version":"0001_init","event":"migration.applied","level":"info","ts":"2026-10-30T04:27:18.183850Z"}
{"version":"0002_idempotency_keys","event":"migration.applied","level":"info","ts":"2026-10-30T04:27:18.183962Z"}
{"version":"0003_attachments","event":"migration.applied","level":"info","ts":"2026-10-30T04:27:18.184008Z"}
{"version":"0004_attachment_quarantine","event":"migration.applied","level":"info","ts":"2026-10-30T04:27:18.184039Z"}
{"reason":"applied","event":"migration.completed","level":"info","ts":"2026-10-30T04:27:18.184066Z"}
s3-init: bucket=messenger-attachments anonymous=private policy=messenger-api lifecycle=abort-incomplete-uploads
$ curl -s "http://$(docker compose port lb 8404)/stats;csv" | awk -F, '$1 == "api" && $2 != "BACKEND" {print $2, $18}' | grep -v MAINT # наблюдение, не шаг инструкции
api1 UP
api2 UP
$ make test
python3 tests/smoke.py --base-url "http://127.0.0.1:18084"
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18084"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
$ docker compose ps -a --format '{{.Service}} {{.State}} {{.RunningFor}} {{.Status}}' # наблюдение, не шаг инструкции
api running 24 seconds ago Up 3 seconds (healthy)
api running 24 seconds ago Up 3 seconds (healthy)
lb running 24 seconds ago Up 8 seconds (healthy)
migrate exited 24 seconds ago Exited (0) 7 seconds ago
postgres running 24 seconds ago Up 9 seconds (healthy)
s3 running 24 seconds ago Up 9 seconds (healthy)
s3-init exited 24 seconds ago Exited (0) 3 seconds ago
valkey running 24 seconds ago Up 9 seconds (healthy)
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18084"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 128, "failed_attempts": 0, "first_recovery_seconds": 2.948}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
$ docker compose ps -a --format '{{.Service}} {{.Name}} {{.State}} {{.Status}}' api # наблюдение, не шаг инструкции
api mtusi2026lab4-clean-api-1 running Up 19 seconds (healthy)
api mtusi2026lab4-clean-api-2 exited Exited (0) 12 seconds ago
$ make restore
scripts/restore.sh
Container mtusi2026lab4-clean-api-1 Running
Container mtusi2026lab4-clean-api-2 Starting
Container mtusi2026lab4-clean-api-2 Started
Container mtusi2026lab4-clean-api-1 Waiting
Container mtusi2026lab4-clean-api-2 Waiting
Container mtusi2026lab4-clean-api-1 Healthy
Container mtusi2026lab4-clean-api-2 Healthy
$ make scenario SCENARIO=failover-upload
{'ok': True, 'upload_instance': 'b21945b926b7', 'download_instance': '6c9bf1e3b1a6', 'sha256_equal': True, 'hook_seconds': 2.713}
$ make scenario SCENARIO=s3-outage
{'ok': True, 'upload': 503, 'metadata': 200, 'download': 503, 'text_message': 201, 'health_ready': 200}
$ make s3-check
23
== 2026-10-30T04:28:17Z S3 privacy and bootstrap checks
== IAM/bucket snapshot (без секретов)
$ make reconcile
dry-run {'objects': 5, 'rows': 6} {'pending_fresh': 1} errors 0
$ make migrate
docker compose run --rm migrate
{"reason":"up to date","event":"migration.completed","level":"info","ts":"2026-10-30T04:28:21.735485Z"}
$ docker volume ls --format '{{.Name}}' --filter label=com.docker.compose.project=mtusi2026lab4-clean # наблюдение, не шаг инструкции
mtusi2026lab4-clean_pgdata
mtusi2026lab4-clean_s3data
mtusi2026lab4-clean_valkeydata
$ make down
1 Container mtusi2026lab4-clean-api-1 Removed
1 Container mtusi2026lab4-clean-api-2 Removed
1 Container mtusi2026lab4-clean-lb-1 Removed
1 Container mtusi2026lab4-clean-migrate-1 Removed
1 Container mtusi2026lab4-clean-postgres-1 Removed
1 Container mtusi2026lab4-clean-s3-1 Removed
1 Container mtusi2026lab4-clean-s3-init-1 Removed
1 Container mtusi2026lab4-clean-valkey-1 Removed
1 Network mtusi2026lab4-clean_backend Removed
1 Network mtusi2026lab4-clean_edge Removed
$ docker volume ls --format '{{.Name}}' --filter label=com.docker.compose.project=mtusi2026lab4-clean # наблюдение, не шаг инструкции
mtusi2026lab4-clean_pgdata
mtusi2026lab4-clean_s3data
mtusi2026lab4-clean_valkeydata
$ make reset-data
Volume mtusi2026lab4-clean_pgdata Removing
Volume mtusi2026lab4-clean_s3data Removing
Volume mtusi2026lab4-clean_valkeydata Removing
Volume mtusi2026lab4-clean_pgdata Removed
Volume mtusi2026lab4-clean_s3data Removed
Volume mtusi2026lab4-clean_valkeydata Removed
$ docker volume ls -q --filter label=com.docker.compose.project=mtusi2026lab4-clean | wc -l | tr -d ' ' # наблюдение, не шаг инструкции
0
+12
View File
@@ -0,0 +1,12 @@
date: 2026-10-30T04:24:13Z commit: 81dd816
$ make check
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
$ make validate # openapi-spec-validator 0.7.2 из .venv (make install)
.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
$ git diff --stat upstream/main -- contracts tests README.md COURSE.md CONTRIBUTING.md MAINTAINERS.md lab.json compose.example.yaml Dockerfile.example # контракт, публичные тесты и задание не менялись
(пусто)
$ git diff --stat upstream/main -- .github # только свой CI job
.github/workflows/application.yml | 93 +++++++++++++++++++++++++++++++++++++++
1 file changed, 93 insertions(+)
+41
View File
@@ -0,0 +1,41 @@
date: 2026-10-30T04:29:05Z commit: 81dd816
host: Apple M4 Max, 16 CPU, 48 GiB RAM, macOS 15.1
docker: 28.3.2, VM: 16 CPU, 8217190400 bytes
compose: 2.38.2-desktop.1
images:
image: ${API_IMAGE:-mtusi-messenger:${COMPOSE_PROJECT_NAME}}
image: rustfs/rc:v0.1.36@sha256:ab024bfebee49a750ce886b4c70963ccd9ddaa03f491704a90710641d7a26699
image: haproxy:3.2.26-alpine@sha256:5893840576071e6217cd52ebdcc14f454c6f7a631056fa6f4e0f40d3cac4215b
image: postgres:17.11-alpine@sha256:266d3c257594e195b309ae9d1805e54c788f1d7b375f56bf6c367aac6c917e7b
image: valkey/valkey:8.1.11-alpine@sha256:2602ace2be6569bc2bbb06f873776516ab9089821525be34f4305c67eb9db02a
image: rustfs/rustfs:1.0.4@sha256:ed9210d6411d77ffceb70d94b964a3cb7d225983368bc25bebe01e995293dcc2
image: grafana/k6:2.3.0@sha256:2dd4756ec6e14e62778916e1eb850108e4b499304c02df56760fa0bdb0fba0b8
FROM python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d AS builder
COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
FROM python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d AS runtime
python in image: Python 3.14.7
haproxy: HAProxy version 3.2.26-3e0f11a 2026/10/14 - https://haproxy.org/
rustfs: rustfs 1.0.4
packages (uv.lock):
fastapi 0.141.1
uvicorn 0.54.0
uvicorn-worker 0.4.0
gunicorn 26.2.0
asyncpg 0.32.0
argon2-cffi 25.1.0
pydantic 2.13.5
pydantic-settings 2.15.0
dishka 1.10.1
structlog 26.1.0
redis 8.1.0
aiobotocore 3.10.0
botocore 1.43.117
aiohttp 3.14.5
pillow 13.0.0
process users (service uid command):
api 10001 python
lb 99 haproxy
postgres 70 postgres
s3 10001 rustfs
valkey 999 tini
valkey 999 valkey-server
+38
View File
@@ -0,0 +1,38 @@
date: 2026-10-30T04:29:33Z commit: 81dd816
== before: api replicas (hostname = X-Instance-Id), state, StartedAt, FinishedAt
0cc3ee358e59 running started=2026-10-30T04:28:59.851144131Z finished=0001-01-01T00:00:00Z
899a952a5a21 running started=2026-10-30T04:29:31.104320507Z finished=2026-10-30T04:29:18.153900709Z
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18084"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 128, "failed_attempts": 0, "first_recovery_seconds": 2.883}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: api replicas
0cc3ee358e59 exited started=2026-10-30T04:28:59.851144131Z finished=2026-10-30T04:29:36.57527926Z
899a952a5a21 running started=2026-10-30T04:29:31.104320507Z finished=2026-10-30T04:29:18.153900709Z
== stopped replica: 0cc3ee358e59; shutdown: signal, drain window with readiness 503, close, exit
{"event":"Handling signal: term","level":"info","ts":"2026-10-30T04:29:34.045936Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-10-30T04:29:34.072243Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-10-30T04:29:34.126665Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"6362781446fb427cbee670871ce5ef51","duration_ms":1.463,"event":"request.handled","level":"info","ts":"2026-10-30T04:29:34.446815Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"0d9cf1c47fca4ec98ed599b113cd93db","duration_ms":0.79,"event":"request.handled","level":"info","ts":"2026-10-30T04:29:34.950067Z"}
{"event":"Shutting down","level":"info","ts":"2026-10-30T04:29:36.079497Z"}
{"event":"Shutting down","level":"info","ts":"2026-10-30T04:29:36.138465Z"}
{"event":"service.stopped","level":"info","ts":"2026-10-30T04:29:36.188615Z"}
{"event":"Finished server process [20]","level":"info","ts":"2026-10-30T04:29:36.188874Z"}
{"event":"service.stopped","level":"info","ts":"2026-10-30T04:29:36.247500Z"}
{"event":"Finished server process [19]","level":"info","ts":"2026-10-30T04:29:36.247910Z"}
{"event":"Worker (pid:20) was sent SIGTERM!","level":"info","ts":"2026-10-30T04:29:36.250957Z"}
{"event":"Worker (pid:19) was sent SIGTERM!","level":"info","ts":"2026-10-30T04:29:36.357386Z"}
{"event":"Shutting down: Master","level":"info","ts":"2026-10-30T04:29:36.468491Z"}
$ scripts/restore.sh
Container mtusi2026lab4-api-2 Running
Container mtusi2026lab4-api-1 Starting
Container mtusi2026lab4-api-1 Started
Container mtusi2026lab4-api-2 Waiting
Container mtusi2026lab4-api-1 Waiting
Container mtusi2026lab4-api-2 Healthy
Container mtusi2026lab4-api-1 Healthy
== after restore
0cc3ee358e59 running started=2026-10-30T04:29:49.45750296Z finished=2026-10-30T04:29:36.57527926Z
899a952a5a21 running started=2026-10-30T04:29:31.104320507Z finished=2026-10-30T04:29:18.153900709Z
+39
View File
@@ -0,0 +1,39 @@
date: 2026-10-30T04:29:14Z commit: 81dd816
== before: api replicas (hostname = X-Instance-Id), state, StartedAt, FinishedAt
0cc3ee358e59 running started=2026-10-30T04:28:59.851144131Z finished=0001-01-01T00:00:00Z
899a952a5a21 running started=2026-10-30T04:29:11.286693179Z finished=2026-10-30T04:29:11.040183429Z
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18084"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 129, "failed_attempts": 0, "first_recovery_seconds": 2.885}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: api replicas
0cc3ee358e59 running started=2026-10-30T04:28:59.851144131Z finished=0001-01-01T00:00:00Z
899a952a5a21 exited started=2026-10-30T04:29:11.286693179Z finished=2026-10-30T04:29:18.153900709Z
== stopped replica: 899a952a5a21; shutdown: signal, drain window with readiness 503, close, exit
{"event":"Handling signal: term","level":"info","ts":"2026-10-30T04:29:15.605798Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-10-30T04:29:15.638898Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-10-30T04:29:15.701830Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"b416afb071ad426a9dde141e952d697f","duration_ms":0.823,"event":"request.handled","level":"info","ts":"2026-10-30T04:29:16.528606Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"3fe4a4c39cb24a219d3a393a1c0332f0","duration_ms":1.591,"event":"request.handled","level":"info","ts":"2026-10-30T04:29:17.038813Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"152273913ad34012bf5eaf31432490ae","duration_ms":0.263,"event":"request.handled","level":"info","ts":"2026-10-30T04:29:17.388797Z"}
{"event":"Shutting down","level":"info","ts":"2026-10-30T04:29:17.642635Z"}
{"event":"Shutting down","level":"info","ts":"2026-10-30T04:29:17.714902Z"}
{"event":"service.stopped","level":"info","ts":"2026-10-30T04:29:17.757838Z"}
{"event":"Finished server process [20]","level":"info","ts":"2026-10-30T04:29:17.758374Z"}
{"event":"Worker (pid:20) was sent SIGTERM!","level":"info","ts":"2026-10-30T04:29:17.826021Z"}
{"event":"service.stopped","level":"info","ts":"2026-10-30T04:29:17.832655Z"}
{"event":"Finished server process [13]","level":"info","ts":"2026-10-30T04:29:17.833005Z"}
{"event":"Worker (pid:13) was sent SIGTERM!","level":"info","ts":"2026-10-30T04:29:17.927620Z"}
{"event":"Shutting down: Master","level":"info","ts":"2026-10-30T04:29:18.029723Z"}
$ scripts/restore.sh
Container mtusi2026lab4-api-1 Running
Container mtusi2026lab4-api-2 Starting
Container mtusi2026lab4-api-2 Started
Container mtusi2026lab4-api-2 Waiting
Container mtusi2026lab4-api-1 Waiting
Container mtusi2026lab4-api-2 Healthy
Container mtusi2026lab4-api-1 Healthy
== after restore
0cc3ee358e59 running started=2026-10-30T04:28:59.851144131Z finished=0001-01-01T00:00:00Z
899a952a5a21 running started=2026-10-30T04:29:31.104320507Z finished=2026-10-30T04:29:18.153900709Z
+10
View File
@@ -0,0 +1,10 @@
{
"ok": true,
"scenario": "failover-upload",
"upload_instance": "899a952a5a21",
"download_instance": "0cc3ee358e59",
"sha256_equal": true,
"hook_seconds": 2.72,
"download_ok_after_seconds": 2.73,
"restored": true
}
+106
View File
@@ -0,0 +1,106 @@
date: 2026-10-30T04:31:11Z commit: 81dd816
$ make test-faults
BASE_URL="http://127.0.0.1:18084" .venv/bin/pytest -m stack -v -s student-tests/stack
============================= test session starts ==============================
platform darwin -- Python 3.14.5, pytest-9.1.1, pluggy-1.6.0 -- <repo>/.venv/bin/python
cachedir: .pytest_cache
rootdir: <repo>
configfile: pyproject.toml
plugins: asyncio-1.4.0, anyio-4.15.1
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collecting ... collected 17 items
student-tests/stack/test_attachment_faults.py::test_user_errors_never_become_500 10 MiB + 1 с Content-Length, тело не отправлено: 413
10 MiB + 1 chunked: 413
PNG 100000×100000 (бомба): 413
PNG 8001×5000: 413
обрезанный PNG: 415
обрезанный JPEG: 415
JPEG как image/png: 415
GIF как image/png: 415
WebP как image/png: 415
текст как image/png: 415
PNG как image/jpeg: 415
пустое тело: 415
application/octet-stream: 415
без Content-Type: 415
filename='../a.png': 400
filename='a/b.png': 400
filename='a\\b.png': 400
filename='a\x00.png': 400
filename='a\x01.png': 400
filename='a\x7f.png': 400
filename='': 400
filename=' ': 400
filename='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: 400
filename=None: 400
посторонний: upload в чужой чат: 404
посторонний: метаданные: 404
посторонний: /content: 404
посторонний: ссылка на чужое вложение: 404
вложение другого чата: 404
неизвестное вложение: 404
11 вложений: 400
повтор вложения: 400
невалидный UUID: 400
пустой текст без вложений: 400
невалидный id вложения: 400
variant=thumbnail: 400
строк вложений в чате: 2 (= ответов 201); ответов в журнале реплик: 54, из них 5xx: 0
PASSED
student-tests/stack/test_attachment_faults.py::test_commit_failure_is_compensated_at_once отказ COMMIT после записи объекта: 503 unavailable
сразу после ответа: строка нет, объект HEAD 404
журнал: audit.attachment_discarded reason=compensation
PASSED
student-tests/stack/test_attachment_faults.py::test_crash_between_put_and_commit_is_swept_after_grace реплика 899a952a5a21 убита между PUT и COMMIT, клиент получил 502
реплика 0cc3ee358e59 убита между PUT и COMMIT, клиент получил 502
остаток после падения: строка pending, объект HEAD 200
остаток после падения: строка pending, объект HEAD 200
старше grace: строка убрана через 26.2 с, объект HEAD 404
моложе grace: строка pending, объект HEAD 200
тот же после сдвига: убран через 1.0 с, объект HEAD 404
готовое вложение в сообщении со сдвинутым created_at скачивается, SHA-256 совпадает
PASSED
student-tests/stack/test_attachment_faults.py::test_aborted_upload_leaves_nothing обрыв после 256 KiB из 5 MiB: статус в журнале реплики [400], намерение None
PASSED
student-tests/stack/test_attachment_faults.py::test_stalled_upload_is_cut_and_leaves_nothing тело зависло после 1 KiB: 503 unavailable за 8.01 с
PASSED
student-tests/stack/test_faults.py::test_registration_race_yields_exactly_one_user гонка 50 регистраций: 201×1, 409×49
PASSED
student-tests/stack/test_faults.py::test_chat_creation_rolls_back_on_unknown_member чат с несуществующим участником: 400 invalid_request
PASSED
student-tests/stack/test_faults.py::test_wrong_password_and_unknown_user_get_same_basic_challenge неверный пароль: 401 за 0.011 с
неизвестный пользователь: 401 за 0.011 с
PASSED
student-tests/stack/test_faults.py::test_logout_then_token_is_dead logout: 204, затем 401, повтор 401
PASSED
student-tests/stack/test_faults.py::test_valkey_serves_repeated_checks 5 проверок токена: keyspace_hits 119 → 124
PASSED
student-tests/stack/test_faults.py::test_valkey_stopped_is_bypassed_but_logout_waits valkey остановлен, действующий токен: 200 за 0.004 с
valkey остановлен, неизвестный токен: 401 за 0.003 с
valkey остановлен: ready 200, вход 201, выход 503
valkey снова запущен: повторный выход 204
PASSED
student-tests/stack/test_faults.py::test_valkey_restart_keeps_cache_and_revocations рестарт valkey: ключей 45 → 45
рестарт valkey: живой 200, отозванный 401
PASSED
student-tests/stack/test_faults.py::test_valkey_flushall_loses_nothing FLUSHALL: живой 200, отозванный 401
PASSED
student-tests/stack/test_faults.py::test_postgres_stopped_gives_bounded_503_and_recovers postgres остановлен, ready: 503 за 0.004 с
postgres остановлен, me: 503 за 0.003 с
postgres остановлен, stranger: 503 за 0.002 с
postgres остановлен, login: 503 за 0.002 с
postgres остановлен, register: 503 за 0.015 с
live внутри реплики 200; LB исключил все реплики ещё через 1.16 с
live через LB без реплик: 503 за 0.000 с
postgres запущен: готовность через 1.6 с, прежний токен 200
PASSED
student-tests/stack/test_faults.py::test_api_restart_keeps_session_without_extending_it рестарт api: готовность через 2.3 с, токен 200
expires_at при входе 2026-10-30T05:32:12.249834Z, в БД после рестарта 2026-10-30T05:32:12.249834Z
PASSED
student-tests/stack/test_faults.py::test_full_restart_keeps_session_until_logout restart.sh: готовность через 0.0 с, прежний токен 200, выход 204, затем 401
PASSED
student-tests/stack/test_faults.py::test_session_expires_after_restart_at_original_time срок сессии сокращён до 20 с, рестарт api: готовность через 3.1 с, до срока 200, после срока 401
PASSED
======================== 17 passed in 98.43s (0:01:38) =========================
@@ -0,0 +1,43 @@
date: 2026-11-06T09:10:22Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ sh scripts/failover-evidence.sh http://127.0.0.1:18084
date: 2026-11-06T09:10:22Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
== before: api replicas (hostname = X-Instance-Id), state, StartedAt, FinishedAt
d4c8d53d7433 running started=2026-11-06T09:10:18.309662551Z finished=2026-11-06T09:10:12.020503506Z
da922ab2ea09 running started=2026-11-06T09:10:18.377623592Z finished=2026-11-06T09:10:12.004100798Z
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18084"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 130, "failed_attempts": 0, "first_recovery_seconds": 2.883}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: api replicas
d4c8d53d7433 exited started=2026-11-06T09:10:18.309662551Z finished=2026-11-06T09:10:25.313223596Z
da922ab2ea09 running started=2026-11-06T09:10:18.377623592Z finished=2026-11-06T09:10:12.004100798Z
== stopped replica: d4c8d53d7433; shutdown: signal, drain window with readiness 503, close, exit
{"event":"Handling signal: term","level":"info","ts":"2026-11-06T09:10:22.774795Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-11-06T09:10:22.776559Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-11-06T09:10:22.848442Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"c23539767f7c4ef5ba39f58a446754a2","duration_ms":1.481,"event":"request.handled","level":"info","ts":"2026-11-06T09:10:23.545632Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"60444a3eba784d839b8caccbaa24d1f2","duration_ms":0.278,"event":"request.handled","level":"info","ts":"2026-11-06T09:10:24.058530Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"5a247b2f9b304ff789354712b9c2d857","duration_ms":0.282,"event":"request.handled","level":"info","ts":"2026-11-06T09:10:24.545468Z"}
{"event":"Shutting down","level":"info","ts":"2026-11-06T09:10:24.779511Z"}
{"event":"Shutting down","level":"info","ts":"2026-11-06T09:10:24.852501Z"}
{"event":"service.stopped","level":"info","ts":"2026-11-06T09:10:24.884399Z"}
{"event":"Finished server process [19]","level":"info","ts":"2026-11-06T09:10:24.884667Z"}
{"event":"service.stopped","level":"info","ts":"2026-11-06T09:10:24.959721Z"}
{"event":"Finished server process [20]","level":"info","ts":"2026-11-06T09:10:24.959901Z"}
{"event":"Worker (pid:19) was sent SIGTERM!","level":"info","ts":"2026-11-06T09:10:24.961675Z"}
{"event":"Worker (pid:20) was sent SIGTERM!","level":"info","ts":"2026-11-06T09:10:25.072586Z"}
{"event":"Shutting down: Master","level":"info","ts":"2026-11-06T09:10:25.178811Z"}
$ scripts/restore.sh
Container hlpipe-lab4-forward-final-20261106-api-2 Running
Container hlpipe-lab4-forward-final-20261106-api-1 Starting
Container hlpipe-lab4-forward-final-20261106-api-1 Started
Container hlpipe-lab4-forward-final-20261106-api-1 Waiting
Container hlpipe-lab4-forward-final-20261106-api-2 Waiting
Container hlpipe-lab4-forward-final-20261106-api-1 Healthy
Container hlpipe-lab4-forward-final-20261106-api-2 Healthy
== after restore
d4c8d53d7433 running started=2026-11-06T09:10:38.256484379Z finished=2026-11-06T09:10:25.313223596Z
da922ab2ea09 running started=2026-11-06T09:10:18.377623592Z finished=2026-11-06T09:10:12.004100798Z
exit_code: 0 elapsed_seconds: 19.912
@@ -0,0 +1,39 @@
date: 2026-11-06T09:10:22Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
== before: api replicas (hostname = X-Instance-Id), state, StartedAt, FinishedAt
d4c8d53d7433 running started=2026-11-06T09:10:18.309662551Z finished=2026-11-06T09:10:12.020503506Z
da922ab2ea09 running started=2026-11-06T09:10:18.377623592Z finished=2026-11-06T09:10:12.004100798Z
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18084"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 130, "failed_attempts": 0, "first_recovery_seconds": 2.883}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: api replicas
d4c8d53d7433 exited started=2026-11-06T09:10:18.309662551Z finished=2026-11-06T09:10:25.313223596Z
da922ab2ea09 running started=2026-11-06T09:10:18.377623592Z finished=2026-11-06T09:10:12.004100798Z
== stopped replica: d4c8d53d7433; shutdown: signal, drain window with readiness 503, close, exit
{"event":"Handling signal: term","level":"info","ts":"2026-11-06T09:10:22.774795Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-11-06T09:10:22.776559Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-11-06T09:10:22.848442Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"c23539767f7c4ef5ba39f58a446754a2","duration_ms":1.481,"event":"request.handled","level":"info","ts":"2026-11-06T09:10:23.545632Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"60444a3eba784d839b8caccbaa24d1f2","duration_ms":0.278,"event":"request.handled","level":"info","ts":"2026-11-06T09:10:24.058530Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"5a247b2f9b304ff789354712b9c2d857","duration_ms":0.282,"event":"request.handled","level":"info","ts":"2026-11-06T09:10:24.545468Z"}
{"event":"Shutting down","level":"info","ts":"2026-11-06T09:10:24.779511Z"}
{"event":"Shutting down","level":"info","ts":"2026-11-06T09:10:24.852501Z"}
{"event":"service.stopped","level":"info","ts":"2026-11-06T09:10:24.884399Z"}
{"event":"Finished server process [19]","level":"info","ts":"2026-11-06T09:10:24.884667Z"}
{"event":"service.stopped","level":"info","ts":"2026-11-06T09:10:24.959721Z"}
{"event":"Finished server process [20]","level":"info","ts":"2026-11-06T09:10:24.959901Z"}
{"event":"Worker (pid:19) was sent SIGTERM!","level":"info","ts":"2026-11-06T09:10:24.961675Z"}
{"event":"Worker (pid:20) was sent SIGTERM!","level":"info","ts":"2026-11-06T09:10:25.072586Z"}
{"event":"Shutting down: Master","level":"info","ts":"2026-11-06T09:10:25.178811Z"}
$ scripts/restore.sh
Container hlpipe-lab4-forward-final-20261106-api-2 Running
Container hlpipe-lab4-forward-final-20261106-api-1 Starting
Container hlpipe-lab4-forward-final-20261106-api-1 Started
Container hlpipe-lab4-forward-final-20261106-api-1 Waiting
Container hlpipe-lab4-forward-final-20261106-api-2 Waiting
Container hlpipe-lab4-forward-final-20261106-api-1 Healthy
Container hlpipe-lab4-forward-final-20261106-api-2 Healthy
== after restore
d4c8d53d7433 running started=2026-11-06T09:10:38.256484379Z finished=2026-11-06T09:10:25.313223596Z
da922ab2ea09 running started=2026-11-06T09:10:18.377623592Z finished=2026-11-06T09:10:12.004100798Z
@@ -0,0 +1,33 @@
date: 2026-11-06T09:13:29Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/lab4_scenarios.py failover-upload --base-url http://127.0.0.1:18084 --output evidence/forward-final-2026-11-06/failover-upload.json
{
"ok": true,
"date": "2026-11-06T09:13:29Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"scenario": "failover-upload",
"upload_instance": "d4c8d53d7433",
"download_instance": "da922ab2ea09",
"sha256_equal": true,
"hook_seconds": 2.734,
"download_ok_after_seconds": 2.762,
"replicas_at_download": [
{
"hostname": "d4c8d53d7433",
"status": "exited",
"exit_code": "0",
"started_at": "2026-11-06T09:12:33.47280571Z",
"finished_at": "2026-11-06T09:13:32.422358668Z"
},
{
"hostname": "da922ab2ea09",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T09:12:33.439519377Z",
"finished_at": "2026-11-06T09:12:33.358178252Z"
}
],
"restored": false,
"replicas_after_restore": null
}
exit_code: 0 elapsed_seconds: 3.177
@@ -0,0 +1,29 @@
{
"ok": true,
"date": "2026-11-06T09:13:29Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"scenario": "failover-upload",
"upload_instance": "d4c8d53d7433",
"download_instance": "da922ab2ea09",
"sha256_equal": true,
"hook_seconds": 2.734,
"download_ok_after_seconds": 2.762,
"replicas_at_download": [
{
"hostname": "d4c8d53d7433",
"status": "exited",
"exit_code": "0",
"started_at": "2026-11-06T09:12:33.47280571Z",
"finished_at": "2026-11-06T09:13:32.422358668Z"
},
{
"hostname": "da922ab2ea09",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T09:12:33.439519377Z",
"finished_at": "2026-11-06T09:12:33.358178252Z"
}
],
"restored": false,
"replicas_after_restore": null
}
@@ -0,0 +1,35 @@
date: 2026-11-06T09:31:35Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {}
$ sh -c 'make check && make validate && make lint && openspec validate --all --strict'
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py
All checks passed!
.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py
163 files already formatted
.venv/bin/ty check src student-tests alembic
All checks passed!
- Validating...
✓ change/forward-lab3-final-review-fixes
✓ spec/lab1-event-publication
✓ spec/lab1-messenger-chats
✓ spec/lab1-messenger-history
✓ spec/lab1-messenger-identity
✓ spec/lab1-service-operability
✓ spec/lab2-auth-sessions
✓ spec/lab2-dependency-failures
✓ spec/lab2-history-index
✓ spec/lab2-persistent-storage
✓ spec/lab2-session-cache
✓ spec/lab3-failover-operations
✓ spec/lab3-idempotent-messages
✓ spec/lab3-load-balancing
✓ spec/lab4-attachments
✓ spec/lab4-evidence-runs
✓ spec/lab4-object-storage
✓ spec/lab4-storage-reconciler
✓ spec/lab4-upload-resilience
Totals: 19 passed, 0 failed (19 items)
exit_code: 0 elapsed_seconds: 1.660
@@ -0,0 +1,6 @@
date: 2026-11-06T09:18:22Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ make test
python3 tests/smoke.py --base-url "http://127.0.0.1:18084"
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
exit_code: 0 elapsed_seconds: 0.435
@@ -0,0 +1,7 @@
[
{
"severity": "minor",
"file": "src/messenger/messaging/entrypoints/schemas.py:35",
"problem": "Непарный Unicode surrogate U+D800/U+DFFF в text принимается схемой, fingerprint вызывает PydanticSerializationError; реальный HTTP возвращает 500 internal_error. Воспроизведено на 75b93bbba5f29575d3fca1d013cc7e0770d88082: evidence/forward-final-2026-11-06/known-minor-http.txt. Сохранено без исправления по правилу minor."
}
]
@@ -0,0 +1,49 @@
date: 2026-11-06T09:30:40Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {}
$ python3 -c 'import collections
import json
import subprocess
from pathlib import Path
source = Path("/Users/artemkostylev/projects/learning/HL/labs/.pipeline/.runs/labs/snapshots/lab3-ba78cf6da439a4ec60fd59389583b38285b0b2b1")
manifest = json.loads(Path("evidence/forward-final-2026-11-06/source-manifest.json").read_text())
def source_git(*args):
return subprocess.check_output(["git", "-C", str(source), *args], text=True)
assert source_git("rev-parse", "HEAD").strip() == manifest["source_to"]
changed = dict(line.split("\t", 1)[::-1] for line in source_git("diff", "--name-status", manifest["source_from"], manifest["source_to"]).splitlines())
assert changed == {row["file"]: row["status"] for row in manifest["files"]}
blobs = {}
for line in source_git("ls-tree", "-r", manifest["source_to"]).splitlines():
meta, path = line.split("\t", 1)
blobs[path] = meta.split()[2]
merged = {"infra/haproxy/haproxy.cfg", "src/messenger/messaging/domain/values.py"}
exact = 0
for row in manifest["files"]:
assert blobs[row["file"]] == row["source_blob"], row["file"]
if row["action"] == "copied-source-archive" or (row["action"] == "ported" and row["file"] not in merged):
assert Path(row["file"]).read_bytes() == (source / row["file"]).read_bytes(), row["file"]
exact += 1
print(json.dumps({"source_from":manifest["source_from"], "source_to":manifest["source_to"], "manifest_paths":len(changed), "decisions":dict(collections.Counter(row["action"] for row in manifest["files"])), "byte_exact_paths":exact, "manual_lab4_merges":sorted(merged)}, indent=2))
subprocess.run(["git", "diff", "--exit-code", manifest["target_base"], "--", "contracts", "tests", "README.md", "COURSE.md"], check=True)
subprocess.run(["git", "diff", "--exit-code", "75b93bbba5f29575d3fca1d013cc7e0770d88082", "--", "src", "student-tests", "infra", "scripts", "compose.yaml", "Dockerfile", "pyproject.toml", "uv.lock", "alembic"], check=True)
print("PASS: complete immutable-source manifest, preserved normative files, runtime unchanged since measurements")
'
{
"source_from": "f591fa64e525679a31c9db7f942d36c789e1a12f",
"source_to": "ba78cf6da439a4ec60fd59389583b38285b0b2b1",
"manifest_paths": 214,
"decisions": {
"adapted-context": 2,
"not-copied": 169,
"adapted-test-tool": 4,
"ported": 14,
"copied-source-archive": 19,
"merged-via-local-change": 6
},
"byte_exact_paths": 31,
"manual_lab4_merges": [
"infra/haproxy/haproxy.cfg",
"src/messenger/messaging/domain/values.py"
]
}
PASS: complete immutable-source manifest, preserved normative files, runtime unchanged since measurements
exit_code: 0 elapsed_seconds: 0.195
@@ -0,0 +1,48 @@
date: 2026-11-06T09:14:49Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:hlpipe-lab4-forward-final-20261106"}
$ sh -c '
set -e
fixture_project=hlpipe-lab4-forward-lb-20261106
fixture_file=student-tests/tools/lb_errors/compose.yaml
trap '"'"'docker compose -p "$fixture_project" -f "$fixture_file" --profile probe down'"'"' EXIT
docker compose -p "$fixture_project" -f "$fixture_file" up -d lb api
docker compose -p "$fixture_project" -f "$fixture_file" run --rm --no-deps probe 502
docker compose -p "$fixture_project" -f "$fixture_file" run --rm --no-deps probe 504
docker compose -p "$fixture_project" -f "$fixture_file" stop api
docker compose -p "$fixture_project" -f "$fixture_file" run --rm --no-deps probe 503
'
Network hlpipe-lab4-forward-lb-20261106_default Creating
Network hlpipe-lab4-forward-lb-20261106_default Created
Container hlpipe-lab4-forward-lb-20261106-lb-1 Creating
Container hlpipe-lab4-forward-lb-20261106-api-1 Creating
Container hlpipe-lab4-forward-lb-20261106-api-1 Created
Container hlpipe-lab4-forward-lb-20261106-lb-1 Created
Container hlpipe-lab4-forward-lb-20261106-api-1 Starting
Container hlpipe-lab4-forward-lb-20261106-lb-1 Starting
Container hlpipe-lab4-forward-lb-20261106-lb-1 Started
Container hlpipe-lab4-forward-lb-20261106-api-1 Started
{"status": 502, "request_id": "a\"b", "json_valid": true, "echo_matches": true, "seconds": 0.001}
{"status": 502, "request_id": "a\\b", "json_valid": true, "echo_matches": true, "seconds": 0.001}
{"status": 502, "request_id": "a\tb", "json_valid": true, "echo_matches": true, "seconds": 0.001}
{"status": 502, "request_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "json_valid": true, "echo_matches": true, "seconds": 0.001}
{"status": 504, "request_id": "a\"b", "json_valid": true, "echo_matches": true, "seconds": 10.024}
{"status": 504, "request_id": "a\\b", "json_valid": true, "echo_matches": true, "seconds": 10.005}
{"status": 504, "request_id": "a\tb", "json_valid": true, "echo_matches": true, "seconds": 10.022}
{"status": 504, "request_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "json_valid": true, "echo_matches": true, "seconds": 10.007}
Container hlpipe-lab4-forward-lb-20261106-api-1 Stopping
Container hlpipe-lab4-forward-lb-20261106-api-1 Stopped
{"status": 503, "request_id": "a\"b", "json_valid": true, "echo_matches": true, "seconds": 0.004}
{"status": 503, "request_id": "a\\b", "json_valid": true, "echo_matches": true, "seconds": 0.0}
{"status": 503, "request_id": "a\tb", "json_valid": true, "echo_matches": true, "seconds": 0.0}
{"status": 503, "request_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "json_valid": true, "echo_matches": true, "seconds": 0.0}
Container hlpipe-lab4-forward-lb-20261106-api-1 Stopping
Container hlpipe-lab4-forward-lb-20261106-lb-1 Stopping
Container hlpipe-lab4-forward-lb-20261106-api-1 Stopped
Container hlpipe-lab4-forward-lb-20261106-api-1 Removing
Container hlpipe-lab4-forward-lb-20261106-api-1 Removed
Container hlpipe-lab4-forward-lb-20261106-lb-1 Stopped
Container hlpipe-lab4-forward-lb-20261106-lb-1 Removing
Container hlpipe-lab4-forward-lb-20261106-lb-1 Removed
Network hlpipe-lab4-forward-lb-20261106_default Removing
Network hlpipe-lab4-forward-lb-20261106_default Removed
exit_code: 0 elapsed_seconds: 52.831
@@ -0,0 +1,24 @@
date: 2026-11-06T09:15:37Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {}
$ python3 -c 'import base64,http.client,json,uuid
conn=http.client.HTTPConnection("127.0.0.1",18084,timeout=10)
name="surrogate_"+uuid.uuid4().hex[:12]
password=uuid.uuid4().hex
def request(method,path,body=None,headers=None):
conn.request(method,path,None if body is None else json.dumps(body).encode(),{"Content-Type":"application/json",**(headers or {})})
response=conn.getresponse()
return response.status,json.loads(response.read() or "null")
status,user=request("POST","/api/v1/users",{"username":name,"password":password,"display_name":"Synthetic"})
assert status==201
status,session=request("POST","/api/v1/auth/sessions",headers={"Authorization":"Basic "+base64.b64encode((name+":"+password).encode()).decode()})
assert status==201
headers={"Authorization":"Bearer "+session["token"]}
status,chat=request("POST","/api/v1/chats",{"title":"Synthetic","member_ids":[user["id"]]},headers)
assert status==201
for value in [chr(0xd800),chr(0xdfff)]:
status,data=request("POST","/api/v1/chats/"+chat["id"]+"/messages",{"text":value},{**headers,"Idempotency-Key":uuid.uuid4().hex})
print(json.dumps({"codepoint":hex(ord(value)),"status":status,"error_code":data.get("error",{}).get("code")}))
conn.close()'
{"codepoint": "0xd800", "status": 500, "error_code": "internal_error"}
{"codepoint": "0xdfff", "status": 500, "error_code": "internal_error"}
exit_code: 0 elapsed_seconds: 0.114
@@ -0,0 +1,15 @@
date: 2026-11-06T09:09:38Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {}
$ .venv/bin/python -c 'import json
from messenger.messaging.entrypoints.schemas import CreateMessageRequest
for value in [chr(0xd800), chr(0xdfff)]:
body = CreateMessageRequest(text=value)
try:
body.fingerprint()
result = "accepted"
except Exception as error:
result = type(error).__name__
print(json.dumps({"codepoint": hex(ord(value)), "schema_accepted": True, "fingerprint_result": result}))'
{"codepoint": "0xd800", "schema_accepted": true, "fingerprint_result": "PydanticSerializationError"}
{"codepoint": "0xdfff", "schema_accepted": true, "fingerprint_result": "PydanticSerializationError"}
exit_code: 0 elapsed_seconds: 0.124
@@ -0,0 +1,29 @@
date: 2026-11-06T09:13:49Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/limits.py
date: 2026-11-06T09:13:49Z commit: 75b93bb
$ docker compose --profile bench --profile tools --profile reconcile config --format json (declared)
service cpus/mem ports secrets volumes networks ro user
---------- ---------- ------------------------------- ------------------------------------------------------------------------------------------------------------------------------- ---------- ---------------------- ----- -----------
api 1/512MiB - cache_url,cursor_secret,database_url,s3_access_key_id,s3_secret_access_key - backend(internal) true image
k6 2/512MiB - - - edge false image
lb 1/128MiB 127.0.0.1:18084,127.0.0.1:24844 - - backend(internal),edge true image
migrate 0.5/256MiB - database_url - backend(internal) false image
postgres 2/1024MiB - postgres_password pgdata backend(internal) false 70:70
reconciler 1/256MiB - database_url,s3_reconciler_access_key_id,s3_reconciler_secret_access_key - backend(internal) true image
s3 1/512MiB - s3_root_password,s3_root_user s3data backend(internal) true image
s3-cli 0.5/64MiB - s3_access_key_id,s3_reconciler_access_key_id,s3_reconciler_secret_access_key,s3_root_password,s3_root_user,s3_secret_access_key - backend(internal) true 65534:65534
s3-init 0.5/64MiB - s3_access_key_id,s3_reconciler_access_key_id,s3_reconciler_secret_access_key,s3_root_password,s3_root_user,s3_secret_access_key - backend(internal) true 65534:65534
valkey 0.5/128MiB - valkey.conf valkeydata backend(internal) false 999:1000
$ docker inspect (running containers: hostname = X-Instance-Id, applied limits)
service hostname cpus memory published
-------- ------------ ---- ------- ---------------------------------------------------
api da922ab2ea09 1 512MiB
lb ea6930f64089 1 128MiB 127.0.0.1:18084->8080/tcp,127.0.0.1:24844->8404/tcp
postgres 3f3a870d89c0 2 1024MiB
s3 5921e788f03d 1 512MiB
valkey 07ea16e2fcfe 0.5 128MiB
PASS: limits, ports, secrets, volumes and networks match the policy
exit_code: 0 elapsed_seconds: 0.308
@@ -0,0 +1,116 @@
date: 2026-11-06T09:16:16Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ make load-evidence
sh scripts/load-evidence.sh
date: 2026-11-06T09:16:17Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082 network: hlpipe-lab4-forward-final-20261106_edge
# make load-evidence (scripts/load-evidence.sh): закрытый контур tests/load.py, 64 потока, 8 с, все записи
# прогона в один новый чат, 3 повтора с паузой 15 с; клиент — одноразовый контейнер в сети <проект>_edge
# стенда (2 CPU, 512 MiB, базовый образ API по digest, tests/, contracts/ и lab.json только для чтения,
# tcp_tw_reuse=1), адрес http://lb:8080. На 4-й секунде — docker stats; после прогона — request.handled реплик.
$ docker ps --format '{{.Names}}' | grep -v '^hlpipe-lab4-forward-final-20261106-' # чужие контейнеры на VM во время серии
$ client() { docker run --rm --name hlpipe-lab4-forward-final-20261106-load-client --network hlpipe-lab4-forward-final-20261106_edge --cpus 2 --memory 512m --sysctl net.ipv4.tcp_tw_reuse=1 -v "$PWD/tests:/w/tests:ro" -v "$PWD/contracts:/w/contracts:ro" -v "$PWD/lab.json:/w/lab.json:ro" -v "$PWD/evidence/forward-final-2026-11-06/load:/out" python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d python /w/tests/load.py "$@"; }
$ client --base-url http://lb:8080 --duration 8 --concurrency 64 --output /out/run1.json
exit 0
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 8.0,
"elapsed_seconds": 8.238,
"concurrency": 64,
"successful_requests": 11768,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1428.577,
"successful_latency_ms": {
"p50": 40.163,
"p95": 100.062,
"p99": 141.558
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
$ docker stats --no-stream # 4-я секунда прогона: контейнер CPU% память
api-1 93.26% 271.2MiB / 512MiB
api-2 89.65% 307.4MiB / 512MiB
lb-1 46.78% 30.14MiB / 128MiB
load-client 60.53% 83.66MiB / 512MiB
postgres-1 157.07% 151.1MiB / 1GiB
s3-1 0.76% 69.52MiB / 512MiB
valkey-1 3.81% 13.9MiB / 128MiB
$ docker compose logs --since 2026-11-06T09:16:17Z api | request.handled: статус метод маршрут число
200 GET /health/live 1
200 GET /health/ready 22
201 POST /api/v1/auth/sessions 3
201 POST /api/v1/chats 1
201 POST /api/v1/chats/{chat_id}/messages 11768
201 POST /api/v1/users 3
$ client --base-url http://lb:8080 --duration 8 --concurrency 64 --output /out/run2.json
exit 0
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 8.0,
"elapsed_seconds": 8.229,
"concurrency": 64,
"successful_requests": 11984,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1456.338,
"successful_latency_ms": {
"p50": 37.442,
"p95": 98.948,
"p99": 134.678
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
$ docker stats --no-stream # 4-я секунда прогона: контейнер CPU% память
api-1 90.89% 272.5MiB / 512MiB
api-2 75.30% 310.1MiB / 512MiB
lb-1 39.94% 30.58MiB / 128MiB
load-client 46.50% 83.65MiB / 512MiB
postgres-1 121.64% 158.2MiB / 1GiB
s3-1 0.77% 69.37MiB / 512MiB
valkey-1 3.02% 13.87MiB / 128MiB
$ docker compose logs --since 2026-11-06T09:16:41Z api | request.handled: статус метод маршрут число
200 GET /health/live 1
200 GET /health/ready 23
201 POST /api/v1/auth/sessions 3
201 POST /api/v1/chats 1
201 POST /api/v1/chats/{chat_id}/messages 11984
201 POST /api/v1/users 3
$ client --base-url http://lb:8080 --duration 8 --concurrency 64 --output /out/run3.json
exit 0
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 8.0,
"elapsed_seconds": 8.21,
"concurrency": 64,
"successful_requests": 12555,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1529.32,
"successful_latency_ms": {
"p50": 33.478,
"p95": 98.475,
"p99": 138.028
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
$ docker stats --no-stream # 4-я секунда прогона: контейнер CPU% память
api-1 100.60% 273.6MiB / 512MiB
api-2 99.12% 310.2MiB / 512MiB
lb-1 58.14% 30.39MiB / 128MiB
load-client 66.66% 83.66MiB / 512MiB
postgres-1 168.89% 167.2MiB / 1GiB
s3-1 0.80% 71.19MiB / 512MiB
valkey-1 4.31% 13.5MiB / 128MiB
$ docker compose logs --since 2026-11-06T09:17:05Z api | request.handled: статус метод маршрут число
200 GET /health/live 1
200 GET /health/ready 24
201 POST /api/v1/auth/sessions 3
201 POST /api/v1/chats 1
201 POST /api/v1/chats/{chat_id}/messages 12555
201 POST /api/v1/users 3
$ docker compose exec -T postgres psql -c 'SELECT count(*) FROM messages'
36448
exit_code: 0 elapsed_seconds: 57.764
+111
View File
@@ -0,0 +1,111 @@
date: 2026-11-06T09:16:17Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082 network: hlpipe-lab4-forward-final-20261106_edge
# make load-evidence (scripts/load-evidence.sh): закрытый контур tests/load.py, 64 потока, 8 с, все записи
# прогона в один новый чат, 3 повтора с паузой 15 с; клиент — одноразовый контейнер в сети <проект>_edge
# стенда (2 CPU, 512 MiB, базовый образ API по digest, tests/, contracts/ и lab.json только для чтения,
# tcp_tw_reuse=1), адрес http://lb:8080. На 4-й секунде — docker stats; после прогона — request.handled реплик.
$ docker ps --format '{{.Names}}' | grep -v '^hlpipe-lab4-forward-final-20261106-' # чужие контейнеры на VM во время серии
$ client() { docker run --rm --name hlpipe-lab4-forward-final-20261106-load-client --network hlpipe-lab4-forward-final-20261106_edge --cpus 2 --memory 512m --sysctl net.ipv4.tcp_tw_reuse=1 -v "$PWD/tests:/w/tests:ro" -v "$PWD/contracts:/w/contracts:ro" -v "$PWD/lab.json:/w/lab.json:ro" -v "$PWD/evidence/forward-final-2026-11-06/load:/out" python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d python /w/tests/load.py "$@"; }
$ client --base-url http://lb:8080 --duration 8 --concurrency 64 --output /out/run1.json
exit 0
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 8.0,
"elapsed_seconds": 8.238,
"concurrency": 64,
"successful_requests": 11768,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1428.577,
"successful_latency_ms": {
"p50": 40.163,
"p95": 100.062,
"p99": 141.558
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
$ docker stats --no-stream # 4-я секунда прогона: контейнер CPU% память
api-1 93.26% 271.2MiB / 512MiB
api-2 89.65% 307.4MiB / 512MiB
lb-1 46.78% 30.14MiB / 128MiB
load-client 60.53% 83.66MiB / 512MiB
postgres-1 157.07% 151.1MiB / 1GiB
s3-1 0.76% 69.52MiB / 512MiB
valkey-1 3.81% 13.9MiB / 128MiB
$ docker compose logs --since 2026-11-06T09:16:17Z api | request.handled: статус метод маршрут число
200 GET /health/live 1
200 GET /health/ready 22
201 POST /api/v1/auth/sessions 3
201 POST /api/v1/chats 1
201 POST /api/v1/chats/{chat_id}/messages 11768
201 POST /api/v1/users 3
$ client --base-url http://lb:8080 --duration 8 --concurrency 64 --output /out/run2.json
exit 0
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 8.0,
"elapsed_seconds": 8.229,
"concurrency": 64,
"successful_requests": 11984,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1456.338,
"successful_latency_ms": {
"p50": 37.442,
"p95": 98.948,
"p99": 134.678
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
$ docker stats --no-stream # 4-я секунда прогона: контейнер CPU% память
api-1 90.89% 272.5MiB / 512MiB
api-2 75.30% 310.1MiB / 512MiB
lb-1 39.94% 30.58MiB / 128MiB
load-client 46.50% 83.65MiB / 512MiB
postgres-1 121.64% 158.2MiB / 1GiB
s3-1 0.77% 69.37MiB / 512MiB
valkey-1 3.02% 13.87MiB / 128MiB
$ docker compose logs --since 2026-11-06T09:16:41Z api | request.handled: статус метод маршрут число
200 GET /health/live 1
200 GET /health/ready 23
201 POST /api/v1/auth/sessions 3
201 POST /api/v1/chats 1
201 POST /api/v1/chats/{chat_id}/messages 11984
201 POST /api/v1/users 3
$ client --base-url http://lb:8080 --duration 8 --concurrency 64 --output /out/run3.json
exit 0
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 8.0,
"elapsed_seconds": 8.21,
"concurrency": 64,
"successful_requests": 12555,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1529.32,
"successful_latency_ms": {
"p50": 33.478,
"p95": 98.475,
"p99": 138.028
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
$ docker stats --no-stream # 4-я секунда прогона: контейнер CPU% память
api-1 100.60% 273.6MiB / 512MiB
api-2 99.12% 310.2MiB / 512MiB
lb-1 58.14% 30.39MiB / 128MiB
load-client 66.66% 83.66MiB / 512MiB
postgres-1 168.89% 167.2MiB / 1GiB
s3-1 0.80% 71.19MiB / 512MiB
valkey-1 4.31% 13.5MiB / 128MiB
$ docker compose logs --since 2026-11-06T09:17:05Z api | request.handled: статус метод маршрут число
200 GET /health/live 1
200 GET /health/ready 24
201 POST /api/v1/auth/sessions 3
201 POST /api/v1/chats 1
201 POST /api/v1/chats/{chat_id}/messages 12555
201 POST /api/v1/users 3
$ docker compose exec -T postgres psql -c 'SELECT count(*) FROM messages'
36448
@@ -0,0 +1,17 @@
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 8.0,
"elapsed_seconds": 8.238,
"concurrency": 64,
"successful_requests": 11768,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1428.577,
"successful_latency_ms": {
"p50": 40.163,
"p95": 100.062,
"p99": 141.558
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
@@ -0,0 +1,17 @@
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 8.0,
"elapsed_seconds": 8.229,
"concurrency": 64,
"successful_requests": 11984,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1456.338,
"successful_latency_ms": {
"p50": 37.442,
"p95": 98.948,
"p99": 134.678
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
@@ -0,0 +1,17 @@
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 8.0,
"elapsed_seconds": 8.21,
"concurrency": 64,
"successful_requests": 12555,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1529.32,
"successful_latency_ms": {
"p50": 33.478,
"p95": 98.475,
"p99": 138.028
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
@@ -0,0 +1,41 @@
date: 2026-11-06T09:18:23Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ make down
docker compose --profile bench --profile tools --profile reconcile down
Container hlpipe-lab4-forward-final-20261106-api-2 Stopping
Container hlpipe-lab4-forward-final-20261106-lb-1 Stopping
Container hlpipe-lab4-forward-final-20261106-api-1 Stopping
Container hlpipe-lab4-forward-final-20261106-lb-1 Stopped
Container hlpipe-lab4-forward-final-20261106-lb-1 Removing
Container hlpipe-lab4-forward-final-20261106-lb-1 Removed
Container hlpipe-lab4-forward-final-20261106-api-1 Stopped
Container hlpipe-lab4-forward-final-20261106-api-1 Removing
Container hlpipe-lab4-forward-final-20261106-api-1 Removed
Container hlpipe-lab4-forward-final-20261106-api-2 Stopped
Container hlpipe-lab4-forward-final-20261106-api-2 Removing
Container hlpipe-lab4-forward-final-20261106-api-2 Removed
Container hlpipe-lab4-forward-final-20261106-s3-init-1 Stopping
Container hlpipe-lab4-forward-final-20261106-migrate-1 Stopping
Container hlpipe-lab4-forward-final-20261106-valkey-1 Stopping
Container hlpipe-lab4-forward-final-20261106-s3-init-1 Stopped
Container hlpipe-lab4-forward-final-20261106-s3-init-1 Removing
Container hlpipe-lab4-forward-final-20261106-migrate-1 Stopped
Container hlpipe-lab4-forward-final-20261106-migrate-1 Removing
Container hlpipe-lab4-forward-final-20261106-s3-init-1 Removed
Container hlpipe-lab4-forward-final-20261106-s3-1 Stopping
Container hlpipe-lab4-forward-final-20261106-migrate-1 Removed
Container hlpipe-lab4-forward-final-20261106-postgres-1 Stopping
Container hlpipe-lab4-forward-final-20261106-valkey-1 Stopped
Container hlpipe-lab4-forward-final-20261106-valkey-1 Removing
Container hlpipe-lab4-forward-final-20261106-valkey-1 Removed
Container hlpipe-lab4-forward-final-20261106-s3-1 Stopped
Container hlpipe-lab4-forward-final-20261106-s3-1 Removing
Container hlpipe-lab4-forward-final-20261106-postgres-1 Stopped
Container hlpipe-lab4-forward-final-20261106-postgres-1 Removing
Container hlpipe-lab4-forward-final-20261106-s3-1 Removed
Container hlpipe-lab4-forward-final-20261106-postgres-1 Removed
Network hlpipe-lab4-forward-final-20261106_edge Removing
Network hlpipe-lab4-forward-final-20261106_backend Removing
Network hlpipe-lab4-forward-final-20261106_edge Removed
Network hlpipe-lab4-forward-final-20261106_backend Removed
exit_code: 0 elapsed_seconds: 3.283
@@ -0,0 +1,72 @@
{
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"load_rps": {
"all": [
1428.577,
1456.338,
1529.32
],
"median": 1456.338,
"min": 1428.577,
"max": 1529.32
},
"load_p95_ms": {
"all": [
100.062,
98.948,
98.475
],
"median": 98.948,
"min": 98.475,
"max": 100.062
},
"load_failed": 0,
"memory_statuses": [
{
"201": 80
},
{
"201": 80
},
{
"201": 78,
"503": 2
}
],
"body_bytes": [
10485256,
10485256,
10485256
],
"api_peak_mib": {
"all": [
456.2,
438.1,
450.4
],
"median": 450.4,
"min": 438.1,
"max": 456.2
},
"lb_peak_mib": {
"all": [
31.8,
28.5,
29.1
],
"median": 29.1,
"min": 28.5,
"max": 31.8
},
"upload_p95_ms": {
"all": [
3641.3,
4755.0,
8091.8
],
"median": 4755.0,
"min": 3641.3,
"max": 8091.8
},
"oom": 0
}
@@ -0,0 +1,87 @@
date: 2026-11-06T09:21:03Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {}
$ python3 -c 'import json,statistics,subprocess
from pathlib import Path
root=Path("evidence/forward-final-2026-11-06")
loads=[json.loads(p.read_text()) for p in sorted((root/"load").glob("run*.json"))]
memory=[json.loads(p.read_text()) for p in sorted((root/"memory").glob("run*/slots*.json"))]
assert len(loads)==len(memory)==3
assert all(r["failed_requests"]==0 for r in loads)
assert all(r["count"]==80 and r["concurrency"]==40 and r["upload_concurrency"]==3 for r in memory)
def stats(values):return {"all":values,"median":statistics.median(values),"min":min(values),"max":max(values)}
result={"commit":subprocess.check_output(["git","rev-parse","HEAD"],text=True).strip(),"load_rps":stats([r["successful_rps"] for r in loads]),"load_p95_ms":stats([r["successful_latency_ms"]["p95"] for r in loads]),"load_failed":sum(r["failed_requests"] for r in loads),"memory_statuses":[r["status_codes"] for r in memory],"body_bytes":[r["body_bytes"] for r in memory],"api_peak_mib":stats([max(v["cgroup_memory_peak_mib"] for v in r["memory"].values()) for r in memory]),"lb_peak_mib":stats([r["lb"]["cgroup_memory_peak_mib"] for r in memory]),"upload_p95_ms":stats([r["latency_ms"]["p95"] for r in memory]),"oom":sum(sum(v["oom_kill"] for v in r["memory"].values())+r["lb"]["oom_kill"] for r in memory)}
with (root/"measurement-summary.json").open("x") as stream:json.dump(result,stream,indent=2);stream.write("\n")
print(json.dumps(result,indent=2))'
{
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"load_rps": {
"all": [
1428.577,
1456.338,
1529.32
],
"median": 1456.338,
"min": 1428.577,
"max": 1529.32
},
"load_p95_ms": {
"all": [
100.062,
98.948,
98.475
],
"median": 98.948,
"min": 98.475,
"max": 100.062
},
"load_failed": 0,
"memory_statuses": [
{
"201": 80
},
{
"201": 80
},
{
"201": 78,
"503": 2
}
],
"body_bytes": [
10485256,
10485256,
10485256
],
"api_peak_mib": {
"all": [
456.2,
438.1,
450.4
],
"median": 450.4,
"min": 438.1,
"max": 456.2
},
"lb_peak_mib": {
"all": [
31.8,
28.5,
29.1
],
"median": 29.1,
"min": 28.5,
"max": 31.8
},
"upload_p95_ms": {
"all": [
3641.3,
4755.0,
8091.8
],
"median": 4755.0,
"min": 3641.3,
"max": 8091.8
},
"oom": 0
}
exit_code: 0 elapsed_seconds: 0.048
@@ -0,0 +1,7 @@
date: 2026-11-06T09:17:14Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/upload_memory.py --base-url http://127.0.0.1:18084 --slots 3 --sizes 10485760 --levels 40 --out evidence/forward-final-2026-11-06/memory/run1
# 2026-11-06T09:17:14Z commit 75b93bbba5f29575d3fca1d013cc7e0770d88082; api x2: 1 CPU, 512 MiB, 2 workers; MALLOC_MMAP_THRESHOLD_=131072; 2 uploads per client; peaks from cgroup memory.peak, MiB
slots size conc codes p50_ms p95_ms api_idle api_peak api_oom lb_peak lb_oom
3 10.0M 40 201x80 2587.5 3641.3 233.2 456.2 0 31.8 0
exit_code: 0 elapsed_seconds: 20.084
@@ -0,0 +1,7 @@
date: 2026-11-06T09:17:34Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/upload_memory.py --base-url http://127.0.0.1:18084 --slots 3 --sizes 10485760 --levels 40 --out evidence/forward-final-2026-11-06/memory/run2
# 2026-11-06T09:17:34Z commit 75b93bbba5f29575d3fca1d013cc7e0770d88082; api x2: 1 CPU, 512 MiB, 2 workers; MALLOC_MMAP_THRESHOLD_=131072; 2 uploads per client; peaks from cgroup memory.peak, MiB
slots size conc codes p50_ms p95_ms api_idle api_peak api_oom lb_peak lb_oom
3 10.0M 40 201x80 2770.4 4755.0 231.3 438.1 0 28.5 0
exit_code: 0 elapsed_seconds: 21.500
@@ -0,0 +1,7 @@
date: 2026-11-06T09:17:56Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/upload_memory.py --base-url http://127.0.0.1:18084 --slots 3 --sizes 10485760 --levels 40 --out evidence/forward-final-2026-11-06/memory/run3
# 2026-11-06T09:17:56Z commit 75b93bbba5f29575d3fca1d013cc7e0770d88082; api x2: 1 CPU, 512 MiB, 2 workers; MALLOC_MMAP_THRESHOLD_=131072; 2 uploads per client; peaks from cgroup memory.peak, MiB
slots size conc codes p50_ms p95_ms api_idle api_peak api_oom lb_peak lb_oom
3 10.0M 40 201x78,503x2 3560.9 8091.8 231.7 450.4 0 29.1 0
exit_code: 0 elapsed_seconds: 23.514
@@ -0,0 +1,40 @@
{
"date": "2026-11-06T09:17:21Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"upload_concurrency": 3,
"mode": "http",
"count": 80,
"concurrency": 40,
"body_bytes": 10485256,
"status_codes": {
"201": 80
},
"instances": 2,
"linked_in_messages": 0,
"elapsed_seconds": 6.143,
"successful_rps": 13.02,
"latency_ms": {
"p50": 2587.5,
"p95": 3641.3,
"p99": 4322.0,
"max": 4322.0
},
"memory": {
"35b198e7b5ca": {
"idle_mib": 230.7,
"docker_stats_peak_mib": 409.6,
"cgroup_memory_peak_mib": 424.4,
"oom_kill": 0
},
"93b12ae1c127": {
"idle_mib": 233.2,
"docker_stats_peak_mib": 419.5,
"cgroup_memory_peak_mib": 456.2,
"oom_kill": 0
}
},
"lb": {
"cgroup_memory_peak_mib": 31.8,
"oom_kill": 0
}
}
@@ -0,0 +1,3 @@
# 2026-11-06T09:17:14Z commit 75b93bbba5f29575d3fca1d013cc7e0770d88082; api x2: 1 CPU, 512 MiB, 2 workers; MALLOC_MMAP_THRESHOLD_=131072; 2 uploads per client; peaks from cgroup memory.peak, MiB
slots size conc codes p50_ms p95_ms api_idle api_peak api_oom lb_peak lb_oom
3 10.0M 40 201x80 2587.5 3641.3 233.2 456.2 0 31.8 0
@@ -0,0 +1,40 @@
{
"date": "2026-11-06T09:17:40Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"upload_concurrency": 3,
"mode": "http",
"count": 80,
"concurrency": 40,
"body_bytes": 10485256,
"status_codes": {
"201": 80
},
"instances": 2,
"linked_in_messages": 0,
"elapsed_seconds": 8.053,
"successful_rps": 9.93,
"latency_ms": {
"p50": 2770.4,
"p95": 4755.0,
"p99": 5507.3,
"max": 5507.3
},
"memory": {
"5df14feadc75": {
"idle_mib": 230.1,
"docker_stats_peak_mib": 411.6,
"cgroup_memory_peak_mib": 438.1,
"oom_kill": 0
},
"806f59cf0d4c": {
"idle_mib": 231.3,
"docker_stats_peak_mib": 401.0,
"cgroup_memory_peak_mib": 425.7,
"oom_kill": 0
}
},
"lb": {
"cgroup_memory_peak_mib": 28.5,
"oom_kill": 0
}
}
@@ -0,0 +1,3 @@
# 2026-11-06T09:17:34Z commit 75b93bbba5f29575d3fca1d013cc7e0770d88082; api x2: 1 CPU, 512 MiB, 2 workers; MALLOC_MMAP_THRESHOLD_=131072; 2 uploads per client; peaks from cgroup memory.peak, MiB
slots size conc codes p50_ms p95_ms api_idle api_peak api_oom lb_peak lb_oom
3 10.0M 40 201x80 2770.4 4755.0 231.3 438.1 0 28.5 0
@@ -0,0 +1,41 @@
{
"date": "2026-11-06T09:18:02Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"upload_concurrency": 3,
"mode": "http",
"count": 80,
"concurrency": 40,
"body_bytes": 10485256,
"status_codes": {
"201": 78,
"503": 2
},
"instances": 2,
"linked_in_messages": 0,
"elapsed_seconds": 9.545,
"successful_rps": 8.17,
"latency_ms": {
"p50": 3560.9,
"p95": 8091.8,
"p99": 8772.2,
"max": 8772.2
},
"memory": {
"798d7474c368": {
"idle_mib": 231.7,
"docker_stats_peak_mib": 428.2,
"cgroup_memory_peak_mib": 450.4,
"oom_kill": 0
},
"0b9a4f6359fb": {
"idle_mib": 179.4,
"docker_stats_peak_mib": 410.0,
"cgroup_memory_peak_mib": 430.7,
"oom_kill": 0
}
},
"lb": {
"cgroup_memory_peak_mib": 29.1,
"oom_kill": 0
}
}
@@ -0,0 +1,3 @@
# 2026-11-06T09:17:56Z commit 75b93bbba5f29575d3fca1d013cc7e0770d88082; api x2: 1 CPU, 512 MiB, 2 workers; MALLOC_MMAP_THRESHOLD_=131072; 2 uploads per client; peaks from cgroup memory.peak, MiB
slots size conc codes p50_ms p95_ms api_idle api_peak api_oom lb_peak lb_oom
3 10.0M 40 201x78,503x2 3560.9 8091.8 231.7 450.4 0 29.1 0
@@ -0,0 +1,154 @@
date: 2026-11-06T09:10:41Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ .venv/bin/python student-tests/tools/forward_network.py
{
"checks": [
{
"case": "non_ascii_basic",
"raw_hex": "42617369632080",
"status": 401,
"instance": "d4c8d53d7433"
},
{
"case": "non_ascii_basic",
"raw_hex": "426173696320ff",
"status": 401,
"instance": "da922ab2ea09"
},
{
"case": "non_ascii_basic",
"raw_hex": "426173696320c3a9",
"status": 401,
"instance": "d4c8d53d7433"
},
{
"case": "non_ascii_basic",
"raw_hex": "42617369632059577870593255366332566a636d5630ff",
"status": 401,
"instance": "da922ab2ea09"
},
{
"case": "nul_basic_display_title",
"position": "'\\x00'",
"status": [
401,
400,
201,
400
],
"instance": "d4c8d53d7433"
},
{
"case": "nul_basic_display_title",
"position": "'\\x00abc'",
"status": [
401,
400,
201,
400
],
"instance": "da922ab2ea09"
},
{
"case": "nul_basic_display_title",
"position": "'abc\\x00def'",
"status": [
401,
400,
201,
400
],
"instance": "d4c8d53d7433"
},
{
"case": "nul_basic_display_title",
"position": "'abc\\x00'",
"status": [
401,
400,
201,
400
],
"instance": "da922ab2ea09"
},
{
"case": "nul_message_corrected_retry",
"position": "'\\x00'",
"instances": [
"da922ab2ea09",
"d4c8d53d7433"
],
"status": [
400,
201,
201
],
"copies": 1
},
{
"case": "nul_message_corrected_retry",
"position": "'\\x00abc'",
"instances": [
"d4c8d53d7433",
"da922ab2ea09"
],
"status": [
400,
201,
201
],
"copies": 1
},
{
"case": "nul_message_corrected_retry",
"position": "'abc\\x00def'",
"instances": [
"da922ab2ea09",
"d4c8d53d7433"
],
"status": [
400,
201,
201
],
"copies": 1
},
{
"case": "nul_message_corrected_retry",
"position": "'abc\\x00'",
"instances": [
"d4c8d53d7433",
"da922ab2ea09"
],
"status": [
400,
201,
201
],
"copies": 1
},
{
"case": "nul_access",
"status": [
404,
404
]
},
{
"case": "logout_across_replicas",
"status": [
204,
401,
401,
401,
401
]
}
],
"instances": [
"d4c8d53d7433",
"da922ab2ea09"
],
"pass": true
}
exit_code: 0 elapsed_seconds: 0.403
@@ -0,0 +1,64 @@
date: 2026-11-06T09:07:00Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"NO_PROXY": "localhost,127.0.0.1,::1"}
$ sh -c 'make check && make validate && make lint && make student-test'
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py
All checks passed!
.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py
163 files already formatted
.venv/bin/ty check src student-tests alembic
All checks passed!
.venv/bin/pytest
============================= test session starts ==============================
platform darwin -- Python 3.14.5, pytest-9.1.1, pluggy-1.6.0
rootdir: <repo>
configfile: pyproject.toml
testpaths: student-tests
plugins: asyncio-1.4.0, anyio-4.15.1
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 511 items / 19 deselected / 492 selected
student-tests/adapters/test_s3_storage.py ... [ 0%]
student-tests/api/test_access.py ......... [ 2%]
student-tests/api/test_attachments.py .................................. [ 9%]
.... [ 10%]
student-tests/api/test_bounded_contexts.py ................... [ 14%]
student-tests/api/test_cache.py ............ [ 16%]
student-tests/api/test_concurrency.py .... [ 17%]
student-tests/api/test_consistency.py ........ [ 18%]
student-tests/api/test_drain.py .. [ 19%]
student-tests/api/test_events.py ...... [ 20%]
student-tests/api/test_idempotency.py ............... [ 23%]
student-tests/api/test_instance.py .... [ 24%]
student-tests/api/test_message_attachments.py ........... [ 26%]
student-tests/api/test_migrations.py ................................ [ 33%]
student-tests/api/test_negative.py ..................................... [ 40%]
[ 40%]
student-tests/api/test_nul_validation.py ................. [ 44%]
student-tests/api/test_operability.py .............. [ 46%]
student-tests/api/test_pagination.py .................. [ 50%]
student-tests/api/test_pool.py ............ [ 53%]
student-tests/api/test_reconciler.py .......... [ 55%]
student-tests/api/test_sessions.py ...................... [ 59%]
student-tests/api/test_timeout.py ........ [ 61%]
student-tests/api/test_upload_consistency.py ......... [ 63%]
student-tests/api/test_wire_format.py .. [ 63%]
student-tests/unit/test_attachments_domain.py .......................... [ 68%]
. [ 68%]
student-tests/unit/test_authentication.py ............................. [ 74%]
student-tests/unit/test_cursor.py ........... [ 77%]
student-tests/unit/test_domain.py ...................................... [ 84%]
.. [ 85%]
student-tests/unit/test_haproxy_errors.py ............... [ 88%]
student-tests/unit/test_nul_inputs.py ...................... [ 92%]
student-tests/unit/test_probe.py .............. [ 95%]
student-tests/unit/test_reconcile_measurements.py ...... [ 96%]
student-tests/unit/test_scaling.py ..... [ 97%]
student-tests/unit/test_session_cache.py .......... [ 99%]
student-tests/unit/test_timeouts.py . [100%]
================ 492 passed, 19 deselected in 88.06s (0:01:28) =================
exit_code: 0 elapsed_seconds: 90.288
@@ -0,0 +1,36 @@
date: 2026-11-06T09:06:38Z commit: f0fc084fbbc6a6084ac9526c50fcee45b2c9a277
environment: {"NO_PROXY": "localhost,127.0.0.1,::1"}
$ sh -c 'make check && make validate && make lint && make student-test'
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py
All checks passed!
.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py
163 files already formatted
.venv/bin/ty check src student-tests alembic
error[invalid-argument-type]: Argument to `HTTPConnection.__init__` is incorrect
--> student-tests/tools/forward_network.py:16:39
|
16 | conn = http.client.HTTPConnection(base.hostname, base.port, timeout=12)
| ^^^^^^^^^^^^^ Expected `str`, found `str | None`
info: element `None` of union `str | None` is not assignable to `str`
info: Method defined here
--> stdlib/http/client.pyi:336:13
|
336 | def __init__(
| ^^^^^^^^
337 | self,
338 | host: str,
| --------- Parameter declared here
error[not-subscriptable]: Cannot subscript object of type `None` with no `__getitem__` method
--> student-tests/tools/forward_network.py:28:16
|
28 | assert result["error"]["request_id"] == request_id
| ^^^^^^^^^^^^^^^
Found 2 diagnostics
make: *** [lint] Error 1
exit_code: 2 elapsed_seconds: 0.678
@@ -0,0 +1,7 @@
date: 2026-11-06T09:32:35Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {}
$ openspec archive forward-lab3-final-review-fixes --yes --skip-specs
Task status: ✓ Complete
Skipping spec updates (--skip-specs flag provided).
Change 'forward-lab3-final-review-fixes' archived as '2026-11-06-forward-lab3-final-review-fixes'.
exit_code: 0 elapsed_seconds: 0.776
@@ -0,0 +1,17 @@
date: 2026-11-06T09:18:19Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ sh -c 'uptime; docker ps --format "{{.Names}} {{.Status}}"; docker stats --no-stream --format "{{.Name}} {{.CPUPerc}} {{.MemUsage}}"'
12:18 up 37 days, 11:30, 6 users, load averages: 4.69 4.19 4.16
hlpipe-lab4-forward-final-20261106-api-1 Up 2 seconds (healthy)
hlpipe-lab4-forward-final-20261106-api-2 Up 2 seconds (healthy)
hlpipe-lab4-forward-final-20261106-lb-1 Up 2 seconds (healthy)
hlpipe-lab4-forward-final-20261106-postgres-1 Up 5 minutes (healthy)
hlpipe-lab4-forward-final-20261106-s3-1 Up 4 minutes (healthy)
hlpipe-lab4-forward-final-20261106-valkey-1 Up 5 minutes (healthy)
hlpipe-lab4-forward-final-20261106-api-1 0.49% 227.9MiB / 512MiB
hlpipe-lab4-forward-final-20261106-api-2 0.83% 227.9MiB / 512MiB
hlpipe-lab4-forward-final-20261106-lb-1 1.16% 15.97MiB / 128MiB
hlpipe-lab4-forward-final-20261106-postgres-1 2.18% 59.24MiB / 1GiB
hlpipe-lab4-forward-final-20261106-s3-1 1.32% 297.5MiB / 512MiB
hlpipe-lab4-forward-final-20261106-valkey-1 0.19% 13.54MiB / 128MiB
exit_code: 0 elapsed_seconds: 2.646
@@ -0,0 +1,18 @@
date: 2026-11-06T09:16:14Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ sh -c 'docker info --format "cpus={{.NCPU}} memory={{.MemTotal}}"; uptime; docker ps --format "{{.Names}} {{.Status}}"; docker stats --no-stream --format "{{.Name}} {{.CPUPerc}} {{.MemUsage}}"'
cpus=16 memory=8217190400
12:16 up 37 days, 11:28, 6 users, load averages: 3.42 4.16 4.16
hlpipe-lab4-forward-final-20261106-api-1 Up About a minute (healthy)
hlpipe-lab4-forward-final-20261106-api-2 Up 3 minutes (healthy)
hlpipe-lab4-forward-final-20261106-postgres-1 Up 3 minutes (healthy)
hlpipe-lab4-forward-final-20261106-lb-1 Up 3 minutes (healthy)
hlpipe-lab4-forward-final-20261106-s3-1 Up 2 minutes (healthy)
hlpipe-lab4-forward-final-20261106-valkey-1 Up 3 minutes (healthy)
hlpipe-lab4-forward-final-20261106-api-1 12.78% 258.1MiB / 512MiB
hlpipe-lab4-forward-final-20261106-api-2 0.83% 298.8MiB / 512MiB
hlpipe-lab4-forward-final-20261106-postgres-1 0.13% 40.44MiB / 1GiB
hlpipe-lab4-forward-final-20261106-lb-1 0.14% 24.3MiB / 128MiB
hlpipe-lab4-forward-final-20261106-s3-1 0.04% 69.52MiB / 512MiB
hlpipe-lab4-forward-final-20261106-valkey-1 1.25% 13.12MiB / 128MiB
exit_code: 0 elapsed_seconds: 2.724
@@ -0,0 +1,17 @@
date: 2026-11-06T09:09:23Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ sh scripts/persistence-evidence.sh
date: 2026-11-06T09:09:23Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
== before: service hostname StartedAt status
api d4c8d53d7433 2026-11-06T09:09:18.754909134Z running
api da922ab2ea09 2026-11-06T09:09:18.824598676Z running
lb ea6930f64089 2026-11-06T09:09:12.580488131Z running
migrate adc778a3d97c 2026-11-06T09:09:13.657768882Z exited
postgres 3f3a870d89c0 2026-11-06T09:09:12.588930798Z running
s3 5921e788f03d 2026-11-06T09:09:12.58822809Z running
s3-init c90ac1d6c503 2026-11-06T09:09:18.165289926Z exited
valkey 07ea16e2fcfe 2026-11-06T09:09:12.587649298Z running
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
FAIL: GET /health/live: connection/TLS failure (URLError)
make: *** [test-persistence] Error 1
exit_code: 2 elapsed_seconds: 0.381
@@ -0,0 +1,26 @@
date: 2026-11-06T09:10:08Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-final-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ sh scripts/persistence-evidence.sh http://127.0.0.1:18084
date: 2026-11-06T09:10:08Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
== before: service hostname StartedAt status
api d4c8d53d7433 2026-11-06T09:09:18.754909134Z running
api da922ab2ea09 2026-11-06T09:09:18.824598676Z running
lb ea6930f64089 2026-11-06T09:09:12.580488131Z running
migrate adc778a3d97c 2026-11-06T09:09:13.657768882Z exited
postgres 3f3a870d89c0 2026-11-06T09:09:12.588930798Z running
s3 5921e788f03d 2026-11-06T09:09:12.58822809Z running
s3-init c90ac1d6c503 2026-11-06T09:09:18.165289926Z exited
valkey 07ea16e2fcfe 2026-11-06T09:09:12.587649298Z running
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18084"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: service hostname StartedAt status
api d4c8d53d7433 2026-11-06T09:10:18.309662551Z running
api da922ab2ea09 2026-11-06T09:10:18.377623592Z running
lb ea6930f64089 2026-11-06T09:10:12.683407215Z running
migrate adc778a3d97c 2026-11-06T09:10:13.202156257Z exited
postgres 3f3a870d89c0 2026-11-06T09:10:12.374370839Z running
s3 5921e788f03d 2026-11-06T09:10:12.339453423Z running
s3-init c90ac1d6c503 2026-11-06T09:10:17.710432217Z exited
valkey 07ea16e2fcfe 2026-11-06T09:10:12.392600006Z running
exit_code: 0 elapsed_seconds: 13.179
@@ -0,0 +1,37 @@
date: 2026-11-06T09:33:10Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {}
$ sh -c 'make lint && openspec validate --all --strict && openspec list --json && git diff --check && git diff --exit-code d7478726a85791c21e065d58f610735a9e687433 -- contracts tests README.md COURSE.md'
.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py
All checks passed!
.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py
163 files already formatted
.venv/bin/ty check src student-tests alembic
All checks passed!
- Validating...
✓ spec/lab1-event-publication
✓ spec/lab1-messenger-chats
✓ spec/lab1-messenger-history
✓ spec/lab1-messenger-identity
✓ spec/lab1-service-operability
✓ spec/lab2-auth-sessions
✓ spec/lab2-dependency-failures
✓ spec/lab2-history-index
✓ spec/lab2-persistent-storage
✓ spec/lab2-session-cache
✓ spec/lab3-failover-operations
✓ spec/lab3-idempotent-messages
✓ spec/lab3-load-balancing
✓ spec/lab4-attachments
✓ spec/lab4-evidence-runs
✓ spec/lab4-object-storage
✓ spec/lab4-storage-reconciler
✓ spec/lab4-upload-resilience
Totals: 18 passed, 0 failed (18 items)
{
"changes": [],
"root": {
"path": "<repo>",
"source": "nearest"
}
}
exit_code: 0 elapsed_seconds: 1.921
@@ -0,0 +1,19 @@
date: 2026-11-06T09:19:41Z commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
environment: {"COMPOSE_PROJECT_NAME": "hlpipe-lab4-forward-reconcile-20261106", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18084", "LB_STATS_PORT": "24844", "BASE_URL": "http://127.0.0.1:18084", "EVIDENCE_DIR": "evidence/forward-final-2026-11-06", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:hlpipe-lab4-forward-final-20261106"}
$ make reconcile-demo 'ARGS=--out evidence/forward-final-2026-11-06/reconcile-demo'
python3 scripts/reconcile_track.py demo --base-url "http://127.0.0.1:18084" --out evidence/forward-final-2026-11-06/reconcile-demo
PASS dry-run находит каждый внедрённый класс, и найдено ровно внедрённое
PASS dry-run ничего не меняет
PASS apply исправляет ровно найденное dry-run, удаление намерений — в журнале аудита
PASS повторный apply — 0 действий
PASS независимая проверка после apply: нарушений нет, готовые не потеряны
PASS строки с расхождениями и ссылки сообщений сохранены
PASS карантин: метаданные 200, /content 404; нетронутое скачивается
PASS вернули те же байты — карантин снят; другие байты той же длины — остался
PASS второй одновременный проход пропущен
PASS SIGTERM: идущий проход останавливается за секунды, блокировка освобождена
PASS S3 остановлен: код 1 за ограниченное время, состояние не изменилось
PASS под нагрузкой: исправлены только внедрённые, ошибок нет
PASS под нагрузкой: все 201 готовы и совпадают по SHA-256, потерь 0
{"ok": true, "checks": 13}
exit_code: 0 elapsed_seconds: 257.493
@@ -0,0 +1,142 @@
{
"date": "2026-11-06T09:20:54Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"mode": "apply",
"skipped": false,
"grace_seconds": 60,
"skew_seconds": 0,
"deep": true,
"page_size": 1000,
"scanned": {
"objects": 24,
"rows": 27
},
"found": {
"orphan_stale": 3,
"orphan_fresh": 2,
"missing": 3,
"size_mismatch": 3,
"sha_mismatch": 3,
"pending_stale": 6,
"pending_fresh": 2,
"quarantined": 0,
"healed": 0
},
"fixed": {
"orphan_stale": 3,
"missing": 3,
"size_mismatch": 3,
"sha_mismatch": 3,
"pending_stale": 6,
"healed": 0
},
"rechecked": {},
"errors": 0,
"samples": {
"missing": [
"01a1b028-2293-743d-a00c-f7fec1a50808",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10",
"01a1b028-2306-74ed-9bff-7c2894a33621"
],
"size_mismatch": [
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-2316-7510-b41e-754b39583de9",
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d"
],
"sha_mismatch": [
"01a1b028-22c2-7713-adf4-93a25c06e07e",
"01a1b028-2327-7105-838f-3f0df5b10dc2",
"01a1b028-2330-7662-8f98-6b45a5c165a2"
],
"pending_stale": [
"2cc53ddd-063f-4edc-ae55-98f201965b7f",
"73fc548a-79a1-4396-a848-2e598eba75a1",
"848bed6c-437a-4169-8cca-940c1987055b",
"99d824a9-d16d-481e-9804-e209f2fa410b",
"a0a3b632-5fe3-447e-a93b-bbdce77c4fb1",
"d86c4cba-2836-4150-bf0e-d32e1ec442ec"
],
"orphan_stale": [
"attachments/35df74ac-5753-48e0-b106-d596db2ca8e6",
"attachments/8ba5a522-a4b8-40a8-b1f3-274401976db6",
"attachments/de3cc053-2a20-44ad-b223-e06818b039e7"
],
"orphan_fresh": [
"attachments/7b96d477-2dc4-4a8c-8703-e6176a7e46a6",
"attachments/d69d9843-ceab-4101-a278-7e5974071ac6"
],
"pending_fresh": [
"dfb03912-2304-4280-bfb7-7a7b9cb28597",
"f1cf4216-9e57-4217-866a-440aac9435f3"
]
},
"cost": {
"seconds": 0.119,
"requests": {
"s3_list": 1,
"s3_head": 6,
"s3_get": 13,
"s3_delete": 9,
"sql": 19
},
"waited_seconds": {
"s3_list": 0.018,
"sql": 0.006
},
"hashed_bytes": 1453621,
"cpu_seconds": 0.621,
"max_rss_mib": 107.6
},
"exit_code": 0,
"wall_seconds": 1.106,
"audit": [
{
"attachment_id": "2cc53ddd-063f-4edc-ae55-98f201965b7f",
"chat_id": "01a1b028-1d47-75b7-a3d7-f9dda0509d40",
"reason": "reconcile",
"event": "audit.attachment_discarded",
"level": "info",
"ts": "2026-11-06T09:20:54.712257Z"
},
{
"attachment_id": "73fc548a-79a1-4396-a848-2e598eba75a1",
"chat_id": "01a1b028-1d47-75b7-a3d7-f9dda0509d40",
"reason": "reconcile",
"event": "audit.attachment_discarded",
"level": "info",
"ts": "2026-11-06T09:20:54.724316Z"
},
{
"attachment_id": "848bed6c-437a-4169-8cca-940c1987055b",
"chat_id": "01a1b028-1d47-75b7-a3d7-f9dda0509d40",
"reason": "reconcile",
"event": "audit.attachment_discarded",
"level": "info",
"ts": "2026-11-06T09:20:54.732394Z"
},
{
"attachment_id": "99d824a9-d16d-481e-9804-e209f2fa410b",
"chat_id": "01a1b028-1d47-75b7-a3d7-f9dda0509d40",
"reason": "reconcile",
"event": "audit.attachment_discarded",
"level": "info",
"ts": "2026-11-06T09:20:54.741795Z"
},
{
"attachment_id": "a0a3b632-5fe3-447e-a93b-bbdce77c4fb1",
"chat_id": "01a1b028-1d47-75b7-a3d7-f9dda0509d40",
"reason": "reconcile",
"event": "audit.attachment_discarded",
"level": "info",
"ts": "2026-11-06T09:20:54.749494Z"
},
{
"attachment_id": "d86c4cba-2836-4150-bf0e-d32e1ec442ec",
"chat_id": "01a1b028-1d47-75b7-a3d7-f9dda0509d40",
"reason": "reconcile",
"event": "audit.attachment_discarded",
"level": "info",
"ts": "2026-11-06T09:20:54.757598Z"
}
]
}
@@ -0,0 +1,76 @@
{
"date": "2026-11-06T09:20:55Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"mode": "apply",
"skipped": false,
"grace_seconds": 60,
"skew_seconds": 0,
"deep": true,
"page_size": 1000,
"scanned": {
"objects": 18,
"rows": 21
},
"found": {
"orphan_stale": 0,
"orphan_fresh": 2,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"pending_fresh": 2,
"quarantined": 9,
"healed": 0
},
"fixed": {
"orphan_stale": 0,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"healed": 0
},
"rechecked": {},
"errors": 0,
"samples": {
"quarantined": [
"01a1b028-2293-743d-a00c-f7fec1a50808",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10",
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-2306-74ed-9bff-7c2894a33621",
"01a1b028-2316-7510-b41e-754b39583de9",
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d",
"01a1b028-22c2-7713-adf4-93a25c06e07e",
"01a1b028-2327-7105-838f-3f0df5b10dc2",
"01a1b028-2330-7662-8f98-6b45a5c165a2"
],
"orphan_fresh": [
"attachments/7b96d477-2dc4-4a8c-8703-e6176a7e46a6",
"attachments/d69d9843-ceab-4101-a278-7e5974071ac6"
],
"pending_fresh": [
"dfb03912-2304-4280-bfb7-7a7b9cb28597",
"f1cf4216-9e57-4217-866a-440aac9435f3"
]
},
"cost": {
"seconds": 0.043,
"requests": {
"s3_list": 1,
"s3_head": 0,
"s3_get": 13,
"s3_delete": 0,
"sql": 1
},
"waited_seconds": {
"s3_list": 0.017,
"sql": 0.006
},
"hashed_bytes": 1453621,
"cpu_seconds": 0.595,
"max_rss_mib": 107.4
},
"exit_code": 0,
"wall_seconds": 1.021,
"audit": []
}
@@ -0,0 +1,53 @@
{
"checked_at": "2026-11-06T09:20:57Z",
"grace_seconds": 60,
"deep": true,
"objects": 18,
"rows": 21,
"ready_healthy": 10,
"counts": {
"orphan_stale": 0,
"orphan_fresh": 2,
"missing_referenced": 0,
"missing_unreferenced": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale_with_object": 0,
"pending_stale_without_object": 0,
"pending_fresh": 2,
"quarantined": 9,
"incomplete_stale": 0,
"incomplete_fresh": 0,
"lost_ready": 0
},
"samples": {
"orphan_fresh": [
"attachments/7b96d477-2dc4-4a8c-8703-e6176a7e46a6",
"attachments/d69d9843-ceab-4101-a278-7e5974071ac6"
],
"pending_fresh": [
"dfb03912-2304-4280-bfb7-7a7b9cb28597",
"f1cf4216-9e57-4217-866a-440aac9435f3"
],
"quarantined": [
"01a1b028-2293-743d-a00c-f7fec1a50808",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10",
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-22c2-7713-adf4-93a25c06e07e",
"01a1b028-2306-74ed-9bff-7c2894a33621",
"01a1b028-2316-7510-b41e-754b39583de9",
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d",
"01a1b028-2327-7105-838f-3f0df5b10dc2",
"01a1b028-2330-7662-8f98-6b45a5c165a2"
]
},
"seconds": {
"s3_list": 0.3,
"s3_incomplete": 0.325,
"pg_query": 0.128,
"deep_hash": 0.593,
"total": 1.348
},
"violations": [],
"ok": true
}
@@ -0,0 +1,82 @@
{
"checked_at": "2026-11-06T09:20:51Z",
"grace_seconds": 60,
"deep": true,
"objects": 24,
"rows": 27,
"ready_healthy": 10,
"counts": {
"orphan_stale": 3,
"orphan_fresh": 2,
"missing_referenced": 2,
"missing_unreferenced": 1,
"size_mismatch": 3,
"sha_mismatch": 3,
"pending_stale_with_object": 3,
"pending_stale_without_object": 3,
"pending_fresh": 2,
"quarantined": 0,
"incomplete_stale": 0,
"incomplete_fresh": 0,
"lost_ready": 0
},
"samples": {
"orphan_stale": [
"attachments/35df74ac-5753-48e0-b106-d596db2ca8e6",
"attachments/8ba5a522-a4b8-40a8-b1f3-274401976db6",
"attachments/de3cc053-2a20-44ad-b223-e06818b039e7"
],
"orphan_fresh": [
"attachments/7b96d477-2dc4-4a8c-8703-e6176a7e46a6",
"attachments/d69d9843-ceab-4101-a278-7e5974071ac6"
],
"missing_referenced": [
"01a1b028-2293-743d-a00c-f7fec1a50808",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10"
],
"missing_unreferenced": [
"01a1b028-2306-74ed-9bff-7c2894a33621"
],
"size_mismatch": [
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-2316-7510-b41e-754b39583de9",
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d"
],
"sha_mismatch": [
"01a1b028-22c2-7713-adf4-93a25c06e07e",
"01a1b028-2327-7105-838f-3f0df5b10dc2",
"01a1b028-2330-7662-8f98-6b45a5c165a2"
],
"pending_stale_with_object": [
"848bed6c-437a-4169-8cca-940c1987055b",
"a0a3b632-5fe3-447e-a93b-bbdce77c4fb1",
"d86c4cba-2836-4150-bf0e-d32e1ec442ec"
],
"pending_stale_without_object": [
"2cc53ddd-063f-4edc-ae55-98f201965b7f",
"73fc548a-79a1-4396-a848-2e598eba75a1",
"99d824a9-d16d-481e-9804-e209f2fa410b"
],
"pending_fresh": [
"dfb03912-2304-4280-bfb7-7a7b9cb28597",
"f1cf4216-9e57-4217-866a-440aac9435f3"
]
},
"seconds": {
"s3_list": 0.304,
"s3_incomplete": 0.347,
"pg_query": 0.112,
"deep_hash": 0.556,
"total": 1.319
},
"violations": [
"orphan_stale",
"missing_referenced",
"missing_unreferenced",
"size_mismatch",
"sha_mismatch",
"pending_stale_with_object",
"pending_stale_without_object"
],
"ok": false
}
@@ -0,0 +1,44 @@
{
"checked_at": "2026-11-06T09:23:59Z",
"grace_seconds": 60,
"deep": true,
"objects": 3382,
"rows": 3383,
"ready_healthy": 3375,
"counts": {
"orphan_stale": 0,
"orphan_fresh": 0,
"missing_referenced": 0,
"missing_unreferenced": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale_with_object": 0,
"pending_stale_without_object": 0,
"pending_fresh": 0,
"quarantined": 8,
"incomplete_stale": 0,
"incomplete_fresh": 0,
"lost_ready": 0
},
"samples": {
"quarantined": [
"01a1b028-22ad-77fa-9b2e-d01e032a2c10",
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-22c2-7713-adf4-93a25c06e07e",
"01a1b028-2306-74ed-9bff-7c2894a33621",
"01a1b028-2316-7510-b41e-754b39583de9",
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d",
"01a1b028-2327-7105-838f-3f0df5b10dc2",
"01a1b028-2330-7662-8f98-6b45a5c165a2"
]
},
"seconds": {
"s3_list": 0.677,
"s3_incomplete": 0.367,
"pg_query": 0.149,
"deep_hash": 67.224,
"total": 68.428
},
"violations": [],
"ok": true
}
@@ -0,0 +1,245 @@
{
"date": "2026-11-06T09:23:59Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"started_at": "2026-11-06T09:19:42Z",
"faults": {
"injected": {
"orphan_stale": [
"35df74ac-5753-48e0-b106-d596db2ca8e6",
"de3cc053-2a20-44ad-b223-e06818b039e7",
"8ba5a522-a4b8-40a8-b1f3-274401976db6"
],
"missing": [
"01a1b028-2306-74ed-9bff-7c2894a33621",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10",
"01a1b028-2293-743d-a00c-f7fec1a50808"
],
"size_mismatch": [
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d",
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-2316-7510-b41e-754b39583de9"
],
"sha_mismatch": [
"01a1b028-2330-7662-8f98-6b45a5c165a2",
"01a1b028-2327-7105-838f-3f0df5b10dc2",
"01a1b028-22c2-7713-adf4-93a25c06e07e"
],
"pending_stale": [
"2cc53ddd-063f-4edc-ae55-98f201965b7f",
"99d824a9-d16d-481e-9804-e209f2fa410b",
"73fc548a-79a1-4396-a848-2e598eba75a1",
"a0a3b632-5fe3-447e-a93b-bbdce77c4fb1",
"848bed6c-437a-4169-8cca-940c1987055b",
"d86c4cba-2836-4150-bf0e-d32e1ec442ec"
],
"orphan_fresh": [
"d69d9843-ceab-4101-a278-7e5974071ac6",
"7b96d477-2dc4-4a8c-8703-e6176a7e46a6"
],
"pending_fresh": [
"f1cf4216-9e57-4217-866a-440aac9435f3",
"dfb03912-2304-4280-bfb7-7a7b9cb28597"
]
},
"targets": {
"missing": [
"01a1b028-2293-743d-a00c-f7fec1a50808",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10",
"01a1b028-2306-74ed-9bff-7c2894a33621"
],
"corrupt": [
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-2316-7510-b41e-754b39583de9",
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d"
],
"bitflip": [
"01a1b028-22c2-7713-adf4-93a25c06e07e",
"01a1b028-2327-7105-838f-3f0df5b10dc2",
"01a1b028-2330-7662-8f98-6b45a5c165a2"
]
},
"linked": [
"01a1b028-2293-743d-a00c-f7fec1a50808",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10",
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-22c2-7713-adf4-93a25c06e07e",
"01a1b028-22cb-73b4-92c8-0db140de50a1",
"01a1b028-22d2-7761-8bdd-8b9987dcd70f"
]
},
"seconds": 256.6,
"checks": [
{
"name": "dry-run находит каждый внедрённый класс, и найдено ровно внедрённое",
"ok": true,
"expected": {
"orphan_stale": 3,
"missing": 3,
"size_mismatch": 3,
"sha_mismatch": 3,
"pending_stale": 6,
"orphan_fresh": 2,
"pending_fresh": 2,
"quarantined": 0
},
"found": {
"orphan_stale": 3,
"orphan_fresh": 2,
"missing": 3,
"size_mismatch": 3,
"sha_mismatch": 3,
"pending_stale": 6,
"pending_fresh": 2,
"quarantined": 0,
"healed": 0
}
},
{
"name": "dry-run ничего не меняет",
"ok": true,
"objects": [
24,
24
],
"rows": [
27,
27
]
},
{
"name": "apply исправляет ровно найденное dry-run, удаление намерений — в журнале аудита",
"ok": true,
"dry_run": {
"orphan_stale": 3,
"missing": 3,
"size_mismatch": 3,
"sha_mismatch": 3,
"pending_stale": 6
},
"fixed": {
"orphan_stale": 3,
"missing": 3,
"size_mismatch": 3,
"sha_mismatch": 3,
"pending_stale": 6
},
"audit_discarded": 6
},
{
"name": "повторный apply — 0 действий",
"ok": true,
"fixed": {
"orphan_stale": 0,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"healed": 0
},
"found": {
"orphan_stale": 0,
"orphan_fresh": 2,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"pending_fresh": 2,
"quarantined": 9,
"healed": 0
}
},
{
"name": "независимая проверка после apply: нарушений нет, готовые не потеряны",
"ok": true,
"counts": {
"orphan_stale": 0,
"orphan_fresh": 2,
"missing_referenced": 0,
"missing_unreferenced": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale_with_object": 0,
"pending_stale_without_object": 0,
"pending_fresh": 2,
"quarantined": 9,
"incomplete_stale": 0,
"incomplete_fresh": 0,
"lost_ready": 0
}
},
{
"name": "строки с расхождениями и ссылки сообщений сохранены",
"ok": true,
"quarantine": {
"01a1b028-2293-743d-a00c-f7fec1a50808": "missing",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10": "missing",
"01a1b028-22ba-778b-99d5-eef1d99349f1": "size_mismatch",
"01a1b028-22c2-7713-adf4-93a25c06e07e": "sha_mismatch",
"01a1b028-2306-74ed-9bff-7c2894a33621": "missing",
"01a1b028-2316-7510-b41e-754b39583de9": "size_mismatch",
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d": "size_mismatch",
"01a1b028-2327-7105-838f-3f0df5b10dc2": "sha_mismatch",
"01a1b028-2330-7662-8f98-6b45a5c165a2": "sha_mismatch"
},
"message_links": 6
},
{
"name": "карантин: метаданные 200, /content 404; нетронутое скачивается",
"ok": true
},
{
"name": "вернули те же байты — карантин снят; другие байты той же длины — остался",
"ok": true,
"fixed": {
"orphan_stale": 0,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"healed": 1
},
"s3_get": 5
},
{
"name": "второй одновременный проход пропущен",
"ok": true
},
{
"name": "SIGTERM: идущий проход останавливается за секунды, блокировка освобождена",
"ok": true,
"s3": "paused",
"state_before_stop": "running",
"init": "true",
"stop_seconds": 0.1,
"exit_code": 143,
"next_pass_skipped": false
},
{
"name": "S3 остановлен: код 1 за ограниченное время, состояние не изменилось",
"ok": true,
"exit_code": 1,
"wall_seconds": 2.32
},
{
"name": "под нагрузкой: исправлены только внедрённые, ошибок нет",
"ok": true,
"fixed": {
"orphan_stale": 5,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 5
},
"runs": 59
},
{
"name": "под нагрузкой: все 201 готовы и совпадают по SHA-256, потерь 0",
"ok": true,
"uploads": {
"201": 3364
},
"lost_ready": 0
}
],
"ok": true
}
@@ -0,0 +1,93 @@
{
"date": "2026-11-06T09:20:52Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"mode": "dry-run",
"skipped": false,
"grace_seconds": 60,
"skew_seconds": 0,
"deep": true,
"page_size": 1000,
"scanned": {
"objects": 24,
"rows": 27
},
"found": {
"orphan_stale": 3,
"orphan_fresh": 2,
"missing": 3,
"size_mismatch": 3,
"sha_mismatch": 3,
"pending_stale": 6,
"pending_fresh": 2,
"quarantined": 0,
"healed": 0
},
"fixed": {
"orphan_stale": 0,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"healed": 0
},
"rechecked": {},
"errors": 0,
"samples": {
"missing": [
"01a1b028-2293-743d-a00c-f7fec1a50808",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10",
"01a1b028-2306-74ed-9bff-7c2894a33621"
],
"size_mismatch": [
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-2316-7510-b41e-754b39583de9",
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d"
],
"sha_mismatch": [
"01a1b028-22c2-7713-adf4-93a25c06e07e",
"01a1b028-2327-7105-838f-3f0df5b10dc2",
"01a1b028-2330-7662-8f98-6b45a5c165a2"
],
"pending_stale": [
"2cc53ddd-063f-4edc-ae55-98f201965b7f",
"73fc548a-79a1-4396-a848-2e598eba75a1",
"848bed6c-437a-4169-8cca-940c1987055b",
"99d824a9-d16d-481e-9804-e209f2fa410b",
"a0a3b632-5fe3-447e-a93b-bbdce77c4fb1",
"d86c4cba-2836-4150-bf0e-d32e1ec442ec"
],
"orphan_stale": [
"attachments/35df74ac-5753-48e0-b106-d596db2ca8e6",
"attachments/8ba5a522-a4b8-40a8-b1f3-274401976db6",
"attachments/de3cc053-2a20-44ad-b223-e06818b039e7"
],
"orphan_fresh": [
"attachments/7b96d477-2dc4-4a8c-8703-e6176a7e46a6",
"attachments/d69d9843-ceab-4101-a278-7e5974071ac6"
],
"pending_fresh": [
"dfb03912-2304-4280-bfb7-7a7b9cb28597",
"f1cf4216-9e57-4217-866a-440aac9435f3"
]
},
"cost": {
"seconds": 0.04,
"requests": {
"s3_list": 1,
"s3_head": 0,
"s3_get": 13,
"s3_delete": 0,
"sql": 1
},
"waited_seconds": {
"s3_list": 0.016,
"sql": 0.006
},
"hashed_bytes": 1453621,
"cpu_seconds": 0.594,
"max_rss_mib": 107.4
},
"exit_code": 0,
"wall_seconds": 1.013,
"audit": []
}
@@ -0,0 +1,78 @@
{
"date": "2026-11-06T09:20:59Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"mode": "apply",
"skipped": false,
"grace_seconds": 60,
"skew_seconds": 0,
"deep": false,
"page_size": 1000,
"scanned": {
"objects": 20,
"rows": 21
},
"found": {
"orphan_stale": 0,
"orphan_fresh": 2,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"pending_fresh": 2,
"quarantined": 8,
"healed": 1
},
"fixed": {
"orphan_stale": 0,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"healed": 1
},
"rechecked": {},
"errors": 0,
"samples": {
"quarantined": [
"01a1b028-22ba-778b-99d5-eef1d99349f1",
"01a1b028-2306-74ed-9bff-7c2894a33621",
"01a1b028-2316-7510-b41e-754b39583de9",
"01a1b028-231e-70ba-8a9e-1de6d00c2e0d",
"01a1b028-22ad-77fa-9b2e-d01e032a2c10",
"01a1b028-2330-7662-8f98-6b45a5c165a2",
"01a1b028-22c2-7713-adf4-93a25c06e07e",
"01a1b028-2327-7105-838f-3f0df5b10dc2"
],
"orphan_fresh": [
"attachments/7b96d477-2dc4-4a8c-8703-e6176a7e46a6",
"attachments/d69d9843-ceab-4101-a278-7e5974071ac6"
],
"pending_fresh": [
"dfb03912-2304-4280-bfb7-7a7b9cb28597",
"f1cf4216-9e57-4217-866a-440aac9435f3"
],
"healed": [
"01a1b028-2293-743d-a00c-f7fec1a50808"
]
},
"cost": {
"seconds": 0.04,
"requests": {
"s3_list": 1,
"s3_head": 0,
"s3_get": 5,
"s3_delete": 0,
"sql": 2
},
"waited_seconds": {
"s3_list": 0.019,
"sql": 0.006
},
"hashed_bytes": 559085,
"cpu_seconds": 0.607,
"max_rss_mib": 107.5
},
"exit_code": 0,
"wall_seconds": 1.074,
"audit": []
}
@@ -0,0 +1,53 @@
{
"date": "2026-11-06T09:21:14Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"mode": "apply",
"skipped": true,
"grace_seconds": 60,
"skew_seconds": 0,
"deep": false,
"page_size": 1000,
"scanned": {
"objects": 0,
"rows": 0
},
"found": {
"orphan_stale": 0,
"orphan_fresh": 0,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"pending_fresh": 0,
"quarantined": 0,
"healed": 0
},
"fixed": {
"orphan_stale": 0,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 0,
"healed": 0
},
"rechecked": {},
"errors": 0,
"samples": {},
"cost": {
"seconds": 0.008,
"requests": {
"s3_list": 0,
"s3_head": 0,
"s3_get": 0,
"s3_delete": 0,
"sql": 0
},
"waited_seconds": {},
"hashed_bytes": 0,
"cpu_seconds": 0.549,
"max_rss_mib": 106.5
},
"exit_code": 0,
"wall_seconds": 0.917,
"audit": []
}
@@ -0,0 +1 @@
{"01a1b028-1dde-7345-971e-c4c24622ef5d": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2293-743d-a00c-f7fec1a50808": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-22ad-77fa-9b2e-d01e032a2c10": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-22ba-778b-99d5-eef1d99349f1": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-22c2-7713-adf4-93a25c06e07e": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-22cb-73b4-92c8-0db140de50a1": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-22d2-7761-8bdd-8b9987dcd70f": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2306-74ed-9bff-7c2894a33621": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2316-7510-b41e-754b39583de9": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-231e-70ba-8a9e-1de6d00c2e0d": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2327-7105-838f-3f0df5b10dc2": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2330-7662-8f98-6b45a5c165a2": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2339-7091-b944-16c65cfce6f9": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2341-737d-b71f-ed06fdf083e5": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2349-7776-b1f8-5d8380dec796": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2351-7605-96d5-b39a1ad9df0f": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2359-72bc-add8-a22020ccf80a": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2361-71b8-b4bd-4227e8de54bc": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "01a1b028-2369-7670-9824-9c1871789b60": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,12 @@
{
"01a1b028-1dde-7345-971e-c4c24622ef5d": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"01a1b028-22cb-73b4-92c8-0db140de50a1": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"01a1b028-22d2-7761-8bdd-8b9987dcd70f": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"01a1b028-2339-7091-b944-16c65cfce6f9": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"01a1b028-2341-737d-b71f-ed06fdf083e5": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"01a1b028-2349-7776-b1f8-5d8380dec796": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"01a1b028-2351-7605-96d5-b39a1ad9df0f": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"01a1b028-2359-72bc-add8-a22020ccf80a": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"01a1b028-2361-71b8-b4bd-4227e8de54bc": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"01a1b028-2369-7670-9824-9c1871789b60": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
}
@@ -0,0 +1,10 @@
{
"date": "2026-11-06T09:21:32Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"exit_code": 1,
"wall_seconds": 2.32,
"stdout": "",
"log": [
"{\"error\":\"EndpointConnectionError\",\"event\":\"reconcile.failed\",\"level\":\"error\",\"ts\":\"2026-11-06T09:21:25.233096Z\"}"
]
}
@@ -0,0 +1,10 @@
{
"date": "2026-11-06T09:21:22Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"s3": "paused",
"state_before_stop": "running",
"init": "true",
"stop_seconds": 0.1,
"exit_code": 143,
"next_pass_skipped": false
}
@@ -0,0 +1,118 @@
{
"date": "2026-11-06T09:23:59Z",
"commit": "75b93bbba5f29575d3fca1d013cc7e0770d88082",
"seconds": 75,
"upload_threads": 3,
"uploads": {
"201": 3364
},
"created": 3364,
"linked_every": 10,
"reconciler_runs": 59,
"run_seconds": [
0.055,
0.048,
0.056,
0.057,
0.063,
0.067,
0.078,
0.082,
0.094,
0.093,
0.097,
0.102,
0.116,
0.144,
0.125,
0.131,
0.127,
0.15,
0.137,
0.155,
0.163,
0.161,
0.168,
0.183,
0.171,
0.177,
0.176,
0.195,
0.208,
0.197,
0.212,
0.217,
0.221,
0.222,
0.246,
0.236,
0.248,
0.247,
0.257,
0.277,
0.262,
0.3,
0.305,
0.289,
0.299,
0.297,
0.302,
0.358,
0.325,
0.325,
0.333,
0.352,
0.334,
0.346,
0.344,
0.346,
0.384,
0.38,
0.406
],
"fixed_total": {
"orphan_stale": 5,
"missing": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale": 5
},
"fresh_seen_total": {
"orphan_fresh": 175,
"pending_fresh": 42
},
"rechecked_total": {
"missing": 55
},
"errors": 0,
"skipped": 0,
"injected_before_load": {
"orphan": 3,
"pending_stale": 3
},
"left_from_faults": {
"orphan_stale": 2,
"pending_stale": 2
},
"check_after": {
"objects": 3382,
"rows": 3383,
"ready_healthy": 3375,
"counts": {
"orphan_stale": 0,
"orphan_fresh": 0,
"missing_referenced": 0,
"missing_unreferenced": 0,
"size_mismatch": 0,
"sha_mismatch": 0,
"pending_stale_with_object": 0,
"pending_stale_without_object": 0,
"pending_fresh": 0,
"quarantined": 8,
"incomplete_stale": 0,
"incomplete_fresh": 0,
"lost_ready": 0
},
"ok": true
}
}

Some files were not shown because too many files have changed in this diff Show More