Commit Graph
13 Commits
Author SHA1 Message Date
Artyom 9ba3076fa8 Preserve active staging and read published originals on reprocessing 2026-11-06 10:59:33 +03:00
Artyom 2e428cf8e6 Fix cold worker claim result mapping with Core updates 2026-11-06 10:35:08 +03:00
Artyom ac6c55809f Complete versioned attachment processing and fault invariants 2026-11-06 09:46:43 +03:00
Artyom 1217da5f5e Forward the lab 4 foundation and migrate the lab 5 pipeline to SQLAlchemy 2026-11-06 09:12:25 +03:00
ArtyomandClaude Opus 5.5 b9d28d190e Run the level 4 scenarios with one command
make level4 runs the level 3 and level 4 scenarios of the pipeline suite
and the invariants into evidence/level4.json. New scenarios: worker_limits
(a 40 Mpx render under a 0.5 s timeout and a 200 MiB address space fails
with processing_failed while the worker keeps running, then succeeds with
the default limits under the container limit), retry_backoff_jitter
(delays from the job journal stay within [d/2, d] and differ) and
failed_retention (failed staging is kept, removed after the retention,
then replay answers expired). The stalled holder must be fenced and the
republished copy must be a no-op. The faults override passes the stall,
render limits and retention through; the admin CLI is python -m
messenger.admin. make failed, make replay ID=... and make dlq are the
operator entry points; the lab 4 target becomes lab4-level4. CI runs
level4 instead of level3. The registry prints ISO timestamps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 03:19:28 +03:00
ArtyomandClaude Opus 5.5 b568fcf9c9 Add the failed registry, an explicit replay and failed staging retention
python -m messenger.admin is a new process assembly over the attachments
zone with only a database: "failed" lists failed jobs (served by the new
partial index jobs_failed_idx), "replay --attachment-id" moves a failed
attachment back to queued with the same job_id, attempt 1 and a fresh
outbox row, writing a replayed event with the operator. A ready, queued
or processing attachment is a noop (replay_noop event), a failed one
whose staging is already gone is expired. Every request is audited.

The relay now also removes the staging object of a failed attachment
once STAGING_RETENTION_SECONDS have passed since it failed; replay and
the cleanup take the same row lock.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 03:12:19 +03:00
ArtyomandClaude Opus 5.5 ecb4827e75 Renew job leases, spread retries with jitter and add crash points
Leases: the worker renews its lease before the first side effect and
every JOB_TIMEOUT_SECONDS/3 while the variants are produced; a renewal
that finds the lease taken cancels the work (and its render process)
and settles the job as fenced without writing anything.

Retries: the delay before attempt n+1 is drawn from [d/2, d] with
d = min(cap, base * 2^(n-1)) (equal jitter), the next attempt time is
computed by PostgreSQL and the delay is kept in the job event.

Faults: the Faults port is async; relay reaches after_commit_before_publish
and after_publish_before_confirm, worker reaches after_claim_before_side_effect,
stall_after_claim (once, FAULT_STALL_SECONDS), after_s3_write_before_commit
and after_commit_before_ack. Crashes are os._exit(137).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 03:08:31 +03:00
ArtyomandClaude Opus 5.5 f2242eb39d Render variants in killable processes with time and memory limits
The renderer adapter keeps one single-process pool per WORKER_CONCURRENCY
slot. A render that exceeds RENDER_TIMEOUT_SECONDS is killed and its slot
is respawned, the render process has RLIMIT_AS = RENDER_MEMORY_MB on
Linux, so both limits end in ImageTooDemanding, a permanent
processing_failed. A render process killed from outside is a transient
error. A cancelled render (lost lease, shutdown) kills its process too.
The worker container limit grows to 1g: two 448 MiB render processes
plus the worker itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 03:02:16 +03:00
ArtyomandClaude Opus 5.5 431b82fd43 Render crop and optimized variants next to the thumbnail
The worker decodes the original once (JPEG draft down to the optimized
box, EXIF orientation applied in place, alpha composed on white) and
encodes three variants: the 256 px thumbnail, a centred 128x128 JPEG crop
that may upscale and a WebP fitted into 1280x1280 without upscaling,
quality 80, method 4. No EXIF, ICC or XMP is written. The byte limit is
checked before decoding. ready is set by the transaction that stores all
three rows, so a partial set is never published.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 03:00:11 +03:00
ArtyomandClaude Opus 5.5 34741dc2d2 Refuse failed attachments in messages through a zone port
The messaging zone asks a new AttachmentStates port whether any listed
attachment of the chat has failed and answers 409 conflict before the
message is stored; queued and processing attachments can still be
sent right after the upload. The check runs only on the first execution
of a key, so a replay returns the original 201 even if the attachment
failed later. main.py wires the port to the attachments directory, so
neither zone reads the other's tables.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 01:09:31 +03:00
ArtyomandClaude Opus 5.5 b152b6e07e Process attachment jobs through an outbox, a relay and a worker
Upload now stages the checked original under staging/, and one
transaction moves the intent to queued, inserts the job (pipeline
version 1, attempt 1), its outbox row and an audit event; the 202 is
the in-memory snapshot, never a reread. Content answers 409 until the
attachment is ready, the thumbnail variant is served from the
published version and absent variants are 404.

The relay publishes outbox rows through FastStream with publisher
confirms, mandatory routing and persistent delivery, marks a row only
after the broker acked it, returns jobs whose lease expired and deletes
the staging object of ready attachments. The worker consumes the
quorum queue through a FastStream subscriber that declares nothing,
takes prefetch from WORKER_CONCURRENCY and acks manually after the
result commit, rejects messages off jobs.schema.json to the dead letter
exchange and requeues after an infrastructure error. A job is claimed
under a lease token; the original is copied to originals/ and a
256 px JPEG thumbnail is written under a content-addressed key; the
commit is fenced by the token, inserts variants with ON CONFLICT DO
NOTHING and publishes only a newer pipeline version. A transient error
reschedules the same job with an exponential delay and a new outbox
row, the last attempt fails with retry_exhausted, an undecodable image
fails at once with invalid_image, and a redelivered finished job is a
recorded no-op.

The worker and the relay are assembled in worker.py and relay.py with
the dishka FastStream integration and the platform providers of the
API; the shared core imports neither FastStream nor aio-pika. Tests
cover the thumbnail geometry and rendering, the job message against
the contract, the pipeline against PostgreSQL (duplicates, parallel
copies, retries, the attempt bound, leases and fencing) and the broker
adapters against a throwaway RabbitMQ with the stand's topology and
permissions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 01:08:06 +03:00
ArtyomandClaude Opus 5.5 64fdaa03f2 Add the jobs, outbox and variants schema as an additive migration
Migration 0004 widens the attachment states to queued, processing and
failed, adds the staging and original keys, the published pipeline
version and the error code, and moves the lab 4 object key into them
while leaving the old column nullable until a contract step. New tables
hold the variants under the effect key (attachment, pipeline version,
name), the jobs with attempt, claims and lease, the outbox rows the
relay publishes and the audit of job transitions. Partial indexes serve
the relay scan of unpublished rows, the search for expired leases and
the staging cleanup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 00:43:34 +03:00
ArtyomandClaude Opus 5.5 540fcb35ba Carry over the lab 4 solution
Bring the application, own tests, Dockerfile, toolchain, compose stack,
HAProxy, k6 and RustFS configuration, migrations, hooks, evidence and
scenario scripts and the CI job from the lab 4 solution commit d2744bc,
merge the lab 4 Makefile targets into the lab 5 skeleton Makefile
without touching its targets, and append the lab 4 solution variables
to the skeleton .env.example. The package and the application follow
the contract version 1.5.0, and the image and the throwaway test
containers are tagged lab5 so they do not collide with a lab 4 stand on
the same host.

This is the starting point of the lab 5 cycle: lint, own tests and the
offline skeleton check pass before any lab 5 change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 00:34:03 +03:00