Prove versioned pipeline recovery and audited replay (U4)

This commit is contained in:
Самохвалов Андрей Ремович
2026-10-02 14:44:24 +03:00
parent 03899d790b
commit ab88c5ba1b
7 changed files with 372 additions and 1 deletions
+36
View File
@@ -0,0 +1,36 @@
# Versioned pipeline fault injection — 2026-10-02
## Hypothesis and method
A process death before or after an S3 side effect, a duplicate delivery, an expired lease, and a reordered older-version delivery should leave one logical variant set per `(attachment_id, pipeline_version, name)`. A failed current job should replay only by an explicit audited operator action. RabbitMQ remains at-least-once: duplicate messages are expected; the durable PostgreSQL state and deterministic S3 keys make their effects idempotent.
`student-tests/test_pipeline_faults.py` creates synthetic 320×200 PNG uploads in the same PostgreSQL/MinIO stand as the application. It runs the real `src.worker.process_message` in child processes with `LAB5_FAULT_POINT` and `LAB5_FAULT_JOB_ID`; `os._exit(72)` terminates the child at the selected durable boundary. The parent checks database state, expires a simulated dead worker lease, runs recovery, redelivers the same logical job with the next managed attempt, and compares variant rows/checksums. The normal worker and publisher are stopped while injecting faults so their consumers cannot race the test. The test also forces a bounded temporary failure, verifies terminal `failed`, then uses the audited replay operation. No broker message is fabricated by the test; separate handoff/outbox tests exercise publication and confirm.
Fault points in worker code: `after_claim_before_s3`, `after_s3_before_result_commit`, `after_result_commit_before_ack`. Each is disabled unless both environment variables name a point and exact synthetic job ID. The test runs once on the existing educational volumes and once on a fresh, isolated Compose project. The isolated project had zero `attachment_jobs` and zero `attachment_job_replays` before the run; its volumes were removed afterward. Main lab5 volumes were preserved.
## Reproduce
```sh
# Existing local stand, with worker and publisher stopped:
docker compose --env-file .env stop worker publisher
docker compose --env-file .env run --rm --build --no-deps -T worker python - < student-tests/test_pipeline_faults.py
# Fresh, isolated PostgreSQL and S3 volumes; no API port and no normal consumer:
docker compose -p mtusi2026lab5fresh --env-file .env up -d --build postgres migrate s3-bootstrap
docker compose -p mtusi2026lab5fresh --env-file .env run --rm --build --no-deps -T worker python - < student-tests/test_pipeline_faults.py
docker compose -p mtusi2026lab5fresh --env-file .env down -v
```
On this host Docker builds used the ignored `compose.local.yaml` DNS override via `COMPOSE_FILE=compose.yaml:compose.local.yaml`; it does not alter application behavior. The first attempt to use `up --wait` on the isolated project exited 1 because Docker Compose treated the completed one-shot `migrate` service as not running; inspection showed `migrate` exited 0, PostgreSQL/S3 were healthy, and both relevant tables contained zero rows before the fault test.
## Observed result
Both runs returned the same anonymized raw summary (one JSON line per run):
```json
{"after_result_commit_before_ack_duplicate":"pass","after_s3_before_result_commit":"pass","before_side_effect_lease_recovery":"pass","bounded_transient_retry_then_replay":"pass","duplicate_reorder_new_version":"pass","failed_registry_operator_replay":"pass"}
```
For each ready version, the test found exactly the four expected variant names (`original`, `thumbnail`, `crop`, `optimized`), a matching original SHA-256, and the expected current `pipeline_version`. Replaying an already ready job returned `duplicate_or_stale`. After a second version became ready, an older delivery left its checksums and current version unchanged; version 1 rows remained available as history. The failed-job replay wrote one audit row and used a new attempt without deleting prior outbox history. The transient retry test exhausted a two-attempt budget, recorded `failed`, then succeeded only after replay reset that budget.
The controlled process death checks durable boundaries, not the probability of such a crash under load. The local stand has one MinIO and one PostgreSQL instance, so this result does not prove service availability during their outage or exactly-once delivery. The broker outage/confirm boundary is tested separately; this matrix checks the worker effect boundary against real S3 and PostgreSQL.
+16
View File
@@ -0,0 +1,16 @@
-- A replay starts a new bounded run without deleting earlier outbox history.
ALTER TABLE attachment_jobs
ADD COLUMN replay_start_attempt integer NOT NULL DEFAULT 1
CHECK (replay_start_attempt > 0 AND replay_start_attempt <= attempt);
CREATE TABLE attachment_job_replays (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
job_id uuid NOT NULL REFERENCES attachment_jobs(id) ON DELETE CASCADE,
previous_attempt integer NOT NULL CHECK (previous_attempt > 0),
new_attempt integer NOT NULL CHECK (new_attempt > previous_attempt),
previous_error_code text NOT NULL,
operator_name text NOT NULL CHECK (length(operator_name) BETWEEN 1 AND 80),
reason text NOT NULL CHECK (length(reason) BETWEEN 1 AND 200),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
UNIQUE (job_id, new_attempt)
);
+1 -1
View File
@@ -149,7 +149,7 @@ def claim_job(conn, job_id, delivery_attempt, lease_seconds=60):
def _reschedule_locked(conn, job, *, max_attempts, delay_seconds, error_code, permanent):
if max_attempts < 1 or delay_seconds < 0:
raise ValueError("invalid retry configuration")
failed = permanent or job["attempt"] >= max_attempts
failed = permanent or job["attempt"] - job["replay_start_attempt"] + 1 >= max_attempts
if failed:
code = error_code if permanent else "retry_exhausted"
conn.execute(
+64
View File
@@ -0,0 +1,64 @@
"""Explicit, audited replay of one failed current-version job."""
import argparse
from uuid import UUID
from .db import pool
def replay_failed(conn, job_id: UUID, operator: str, reason: str) -> int:
if not 1 <= len(operator) <= 80 or not 1 <= len(reason) <= 200:
raise ValueError("operator and reason are required")
with conn.transaction():
job = conn.execute(
"SELECT j.*,a.status AS attachment_status,"
"a.pipeline_version AS current_pipeline_version "
"FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id "
"WHERE j.id=%s FOR UPDATE OF j,a",
(job_id,),
).fetchone()
if (job is None or job["status"] != "failed" or
job["pipeline_version"] != job["current_pipeline_version"] or
job["attachment_status"] != "failed"):
raise ValueError("only a failed current-version job can be replayed")
attempt = job["attempt"] + 1
conn.execute(
"UPDATE attachment_jobs SET status='queued',attempt=%s,replay_start_attempt=%s,"
"next_attempt_at=clock_timestamp(),error_code=NULL,updated_at=clock_timestamp() "
"WHERE id=%s",
(attempt, attempt, job_id),
)
conn.execute(
"UPDATE attachments SET status='queued',error_code=NULL,updated_at=clock_timestamp() "
"WHERE id=%s AND pipeline_version=%s",
(job["attachment_id"], job["pipeline_version"]),
)
conn.execute(
"INSERT INTO attachment_outbox (job_id,attempt) VALUES (%s,%s)",
(job_id, attempt),
)
conn.execute(
"INSERT INTO attachment_job_replays "
"(job_id,previous_attempt,new_attempt,previous_error_code,operator_name,reason) "
"VALUES (%s,%s,%s,%s,%s,%s)",
(job_id, job["attempt"], attempt, job["error_code"], operator, reason),
)
return attempt
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("job_id", type=UUID)
parser.add_argument("--operator", required=True)
parser.add_argument("--reason", required=True)
args = parser.parse_args()
try:
with pool().connection() as conn:
attempt = replay_failed(conn, args.job_id, args.operator, args.reason)
except ValueError as exc:
parser.error(str(exc))
print(f"Queued job {args.job_id} attempt {attempt}")
if __name__ == "__main__":
main()
+57
View File
@@ -0,0 +1,57 @@
"""Start a new pipeline version from a verified, already ready original."""
import argparse
from uuid import UUID, uuid4
from .db import pool
def start_new_version(conn, attachment_id: UUID) -> tuple[int, UUID]:
with conn.transaction():
row = conn.execute(
"SELECT * FROM attachments WHERE id=%s FOR UPDATE", (attachment_id,)
).fetchone()
if row is None or row["status"] != "ready":
raise ValueError("only a ready attachment can start a new version")
original = conn.execute(
"SELECT object_key,size_bytes,sha256 FROM attachment_variants "
"WHERE attachment_id=%s AND pipeline_version=%s AND name='original'",
(attachment_id, row["pipeline_version"]),
).fetchone()
if (original is None or original["object_key"] != row["object_key"] or
original["size_bytes"] != row["size_bytes"] or
original["sha256"] != row["sha256"]):
raise ValueError("verified original is missing")
version = row["pipeline_version"] + 1
job_id = uuid4()
conn.execute(
"UPDATE attachments SET pipeline_version=%s,status='queued',error_code=NULL,"
"updated_at=clock_timestamp() WHERE id=%s",
(version, attachment_id),
)
conn.execute(
"INSERT INTO attachment_jobs (id,attachment_id,pipeline_version) "
"VALUES (%s,%s,%s)",
(job_id, attachment_id, version),
)
conn.execute(
"INSERT INTO attachment_outbox (job_id,attempt) VALUES (%s,1)",
(job_id,),
)
return version, job_id
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("attachment_id", type=UUID)
args = parser.parse_args()
try:
with pool().connection() as conn:
version, job_id = start_new_version(conn, args.attachment_id)
except ValueError as exc:
parser.error(str(exc))
print(f"Queued attachment {args.attachment_id} version {version} job {job_id}")
if __name__ == "__main__":
main()
+10
View File
@@ -104,6 +104,13 @@ def _delay(attempt: int) -> float:
return base + random.uniform(0, base / 2)
def _fault(point: str, job_id) -> None:
"""Test-only process death at a named durable boundary for one job."""
if (os.environ.get("LAB5_FAULT_POINT") == point and
os.environ.get("LAB5_FAULT_JOB_ID") == str(job_id)):
os._exit(72)
@contextmanager
def _deadline(seconds: int):
def expired(_signal, _frame):
@@ -124,13 +131,16 @@ def process_message(item: dict) -> str:
if job is None:
return "duplicate_or_stale"
token = job["lease_token"]
_fault("after_claim_before_s3", job["id"])
try:
with _deadline(LEASE_SECONDS):
data = _source_bytes(job)
variants = render(data, job["content_type"])
stored = _write_variants(job, variants)
_fault("after_s3_before_result_commit", job["id"])
with pool().connection() as conn:
current = complete_job(conn, job["id"], token, stored)
_fault("after_result_commit_before_ack", job["id"])
if current and job["staging_key"]:
try:
storage().remove_object(BUCKET, job["staging_key"])
+188
View File
@@ -0,0 +1,188 @@
#!/usr/bin/env python3
"""Deterministic fault windows for the versioned attachment pipeline.
Run in a one-shot worker image while the normal worker and publisher are stopped.
All rows and objects are synthetic and stay in the educational volumes.
"""
import hashlib
import json
import os
import subprocess
import sys
import time
from io import BytesIO
from uuid import uuid4
from PIL import Image
from src.attachments import BUCKET, storage
from src.db import pool
from src.pipeline_jobs import (
claim_job, finish_attempt, queue_attachment, recover_expired, reserve_upload,
)
from src.replay import replay_failed
from src.reprocess import start_new_version
from src.worker import process_message
image = BytesIO()
Image.new("RGB", (320, 200), "#2765a8").save(image, format="PNG")
DATA = image.getvalue()
SHA = hashlib.sha256(DATA).hexdigest()
def create_case():
user_id, chat_id, attachment_id = uuid4(), uuid4(), uuid4()
staging_key = f"staging/{attachment_id}"
with pool().connection() as conn:
with conn.transaction():
conn.execute(
"INSERT INTO users (id,username,display_name,password_hash) VALUES (%s,%s,%s,%s)",
(user_id, f"fault_{user_id.hex[:20]}", "Pipeline fault fixture", "synthetic"),
)
conn.execute("INSERT INTO chats (id,title) VALUES (%s,%s)", (chat_id, "Pipeline fault fixture"))
conn.execute(
"INSERT INTO memberships (chat_id,user_id,position) VALUES (%s,%s,0)",
(chat_id, user_id),
)
reserve_upload(conn, attachment_id, staging_key)
storage().put_object(BUCKET, staging_key, BytesIO(DATA), len(DATA), content_type="image/png")
with pool().connection() as conn:
_, job_id = queue_attachment(
conn, attachment_id=attachment_id, chat_id=chat_id, owner_id=user_id,
filename="fixture.png", staging_key=staging_key,
object_key=f"originals/{attachment_id}", content_type="image/png",
size_bytes=len(DATA), sha256=SHA,
)
return attachment_id, job_id
def item(job_id, attempt=1):
return {"job_id": str(job_id), "attempt": attempt}
def state(attachment_id, version):
with pool().connection() as conn:
attachment = conn.execute(
"SELECT status,pipeline_version,sha256 FROM attachments WHERE id=%s", (attachment_id,)
).fetchone()
rows = conn.execute(
"SELECT name,sha256 FROM attachment_variants "
"WHERE attachment_id=%s AND pipeline_version=%s ORDER BY name",
(attachment_id, version),
).fetchall()
return attachment, {row["name"]: row["sha256"] for row in rows}
def assert_ready(attachment_id, version):
attachment, variants = state(attachment_id, version)
assert attachment["status"] == "ready" and attachment["pipeline_version"] == version
assert attachment["sha256"] == SHA
assert set(variants) == {"original", "thumbnail", "crop", "optimized"}
assert variants["original"] == SHA
return variants
def expire_and_recover(job_id):
with pool().connection() as conn:
conn.execute(
"UPDATE attachment_jobs SET lease_expires_at=clock_timestamp()-interval '1 second' "
"WHERE id=%s AND status='processing'",
(job_id,),
)
recovered = recover_expired(conn, max_attempts=5)
assert (job_id, "queued") in recovered
def crash(job_id, point):
child = subprocess.run(
[sys.executable, "-c", "from src.worker import process_message; "
f"process_message({{'job_id': '{job_id}', 'attempt': 1}})"],
env={**os.environ, "LAB5_FAULT_POINT": point, "LAB5_FAULT_JOB_ID": str(job_id)},
capture_output=True, text=True,
)
assert child.returncode == 72, (point, child.returncode, child.stderr)
def main():
results = {}
attachment, job = create_case()
crash(job, "after_result_commit_before_ack")
first = assert_ready(attachment, 1)
assert process_message(item(job)) == "duplicate_or_stale"
assert assert_ready(attachment, 1) == first
results["after_result_commit_before_ack_duplicate"] = "pass"
attachment, job = create_case()
crash(job, "after_claim_before_s3")
expire_and_recover(job)
assert process_message(item(job, 2)) == "ready"
assert_ready(attachment, 1)
results["before_side_effect_lease_recovery"] = "pass"
attachment, job = create_case()
crash(job, "after_s3_before_result_commit")
expire_and_recover(job)
assert process_message(item(job, 2)) == "ready"
assert_ready(attachment, 1)
results["after_s3_before_result_commit"] = "pass"
attachment, old_job = create_case()
assert process_message(item(old_job)) == "ready"
with pool().connection() as conn:
version, new_job = start_new_version(conn, attachment)
assert version == 2
assert process_message(item(old_job)) == "duplicate_or_stale"
assert process_message(item(new_job)) == "ready"
newer = assert_ready(attachment, 2)
assert process_message(item(old_job)) == "duplicate_or_stale"
assert assert_ready(attachment, 2) == newer
assert len(state(attachment, 1)[1]) == 4
results["duplicate_reorder_new_version"] = "pass"
attachment, job = create_case()
with pool().connection() as conn:
claimed = claim_job(conn, job, 1)
assert finish_attempt(
conn, job, claimed["lease_token"], max_attempts=5,
error_code="invalid_image", permanent=True,
) == "failed"
attempt = replay_failed(conn, job, "lab5-test", "verified synthetic replay")
assert attempt == 2
assert process_message(item(job, attempt)) == "ready"
assert_ready(attachment, 1)
with pool().connection() as conn:
count = conn.execute(
"SELECT count(*) AS n FROM attachment_job_replays WHERE job_id=%s", (job,)
).fetchone()["n"]
assert count == 1
results["failed_registry_operator_replay"] = "pass"
attachment, job = create_case()
with pool().connection() as conn:
claimed = claim_job(conn, job, 1)
assert finish_attempt(
conn, job, claimed["lease_token"], max_attempts=2,
delay_seconds=0.01, error_code="storage_unavailable",
) == "queued"
time.sleep(0.03)
with pool().connection() as conn:
claimed = claim_job(conn, job, 2)
assert claimed is not None
assert finish_attempt(
conn, job, claimed["lease_token"], max_attempts=2,
error_code="storage_unavailable",
) == "failed"
attempt = replay_failed(conn, job, "lab5-test", "retry budget reset after repair")
assert attempt == 3
assert process_message(item(job, attempt)) == "ready"
assert_ready(attachment, 1)
results["bounded_transient_retry_then_replay"] = "pass"
print(json.dumps(results, sort_keys=True))
if __name__ == "__main__":
main()