Files
mtusi2026lab5/app/faults.py
T
alenchha 633fb96920 ЛР 5 — БВТ2301 — Курлова — на 5
RabbitMQ + transactional outbox, отдельный worker (thumbnail/crop/WebP), lease и heartbeat, retry с backoff+jitter, DLQ и replay, admission control; тесты отказов, трек 1 и отчёт.
2026-11-06 12:10:16 +03:00

39 lines
1.6 KiB
Python

"""Fault injection for the lab stand (FAULT_INJECTION=1 only; the default is off).
A fault is selected by the client-chosen display filename, e.g. `fault-worker_crash_after_put.png`.
Filenames never reach S3 keys or broker messages; the worker reads them from the DB like any metadata.
Crash faults fire only on the first attempt and first delivery, so recovery can be observed.
api_crash_after_commit API process exits right after COMMIT of attachment+job+outbox, before publish
worker_crash_after_put worker exits after writing final objects to S3, before the result COMMIT
worker_crash_after_commit worker exits after the result COMMIT, before broker ack
worker_hang worker blocks its event loop (no heartbeats) for 2.5 x lease on attempt 1
transientN attempts 1..N fail with a transient error (N = 1..9)
permanent permanent error on the first attempt -> failed without retries
"""
import logging
import os
import re
from typing import Optional
from app import config
log = logging.getLogger("messenger.faults")
TRANSIENT_RE = re.compile(r"fault-transient([1-9])")
def active(filename: Optional[str], name: str) -> bool:
return config.FAULT_INJECTION and bool(filename) and ("fault-" + name) in filename
def transient_until(filename: Optional[str]) -> int:
if not config.FAULT_INJECTION or not filename:
return 0
match = TRANSIENT_RE.search(filename)
return int(match.group(1)) if match else 0
def crash(point: str) -> None:
log.error("FAULT INJECTION: process exits at %s", point)
logging.shutdown()
os._exit(70)