Files
mtusi2026lab6/app/faults.py
alenchha 0638263029 ЛР 6 — БВТ2301 — Курлова — на 5 (трек 1 «Цена наблюдаемости»)
Перенесён минимум lab5 без изменения API и добавлена наблюдаемость:
- OTel SDK в API и worker: traces, metrics, logs по OTLP; W3C trace context
  API -> outbox -> RabbitMQ -> worker (producer/consumer spans, retry в той же трассе);
- OTel Collector (memory_limiter, scrub, batch, ограниченные очереди, профиль tail sampling),
  Prometheus, Loki, Tempo, Grafana с provisioning, dashboard и 6 alert rules с runbooks;
- проверки: наблюдаемость и приватность, инциденты A/B/C, насыщение worker,
  переполнение очереди exporter, эксперимент трека 1 (4 конфигурации x 3 прогона);
- CI job со сборкой и запуском стенда; REPORT.md с доказательствами в evidence/.
2026-11-11 09:24:27 +03:00

39 lines
1.6 KiB
Python

"""Fault injection for the lab stand (FAULT_INJECTION=1 only; the default is off).
A fault is selected by the client-chosen display filename, e.g. `fault-worker_crash_after_put.png`.
Filenames never reach S3 keys or broker messages; the worker reads them from the DB like any metadata.
Crash faults fire only on the first attempt and first delivery, so recovery can be observed.
api_crash_after_commit API process exits right after COMMIT of attachment+job+outbox, before publish
worker_crash_after_put worker exits after writing final objects to S3, before the result COMMIT
worker_crash_after_commit worker exits after the result COMMIT, before broker ack
worker_hang worker blocks its event loop (no heartbeats) for 2.5 x lease on attempt 1
transientN attempts 1..N fail with a transient error (N = 1..9)
permanent permanent error on the first attempt -> failed without retries
"""
import logging
import os
import re
from typing import Optional
from app import config
log = logging.getLogger("messenger.faults")
TRANSIENT_RE = re.compile(r"fault-transient([1-9])")
def active(filename: Optional[str], name: str) -> bool:
return config.FAULT_INJECTION and bool(filename) and ("fault-" + name) in filename
def transient_until(filename: Optional[str]) -> int:
if not config.FAULT_INJECTION or not filename:
return 0
match = TRANSIENT_RE.search(filename)
return int(match.group(1)) if match else 0
def crash(point: str) -> None:
log.error("FAULT INJECTION: process exits at %s", point)
logging.shutdown()
os._exit(70)