forked from mtusi2026/mtusi2026lab6
Перенесён минимум lab5 без изменения API и добавлена наблюдаемость: - OTel SDK в API и worker: traces, metrics, logs по OTLP; W3C trace context API -> outbox -> RabbitMQ -> worker (producer/consumer spans, retry в той же трассе); - OTel Collector (memory_limiter, scrub, batch, ограниченные очереди, профиль tail sampling), Prometheus, Loki, Tempo, Grafana с provisioning, dashboard и 6 alert rules с runbooks; - проверки: наблюдаемость и приватность, инциденты A/B/C, насыщение worker, переполнение очереди exporter, эксперимент трека 1 (4 конфигурации x 3 прогона); - CI job со сборкой и запуском стенда; REPORT.md с доказательствами в evidence/.
39 lines
1.6 KiB
Python
39 lines
1.6 KiB
Python
"""Fault injection for the lab stand (FAULT_INJECTION=1 only; the default is off).
|
|
|
|
A fault is selected by the client-chosen display filename, e.g. `fault-worker_crash_after_put.png`.
|
|
Filenames never reach S3 keys or broker messages; the worker reads them from the DB like any metadata.
|
|
Crash faults fire only on the first attempt and first delivery, so recovery can be observed.
|
|
api_crash_after_commit API process exits right after COMMIT of attachment+job+outbox, before publish
|
|
worker_crash_after_put worker exits after writing final objects to S3, before the result COMMIT
|
|
worker_crash_after_commit worker exits after the result COMMIT, before broker ack
|
|
worker_hang worker blocks its event loop (no heartbeats) for 2.5 x lease on attempt 1
|
|
transientN attempts 1..N fail with a transient error (N = 1..9)
|
|
permanent permanent error on the first attempt -> failed without retries
|
|
"""
|
|
import logging
|
|
import os
|
|
import re
|
|
from typing import Optional
|
|
|
|
from app import config
|
|
|
|
log = logging.getLogger("messenger.faults")
|
|
TRANSIENT_RE = re.compile(r"fault-transient([1-9])")
|
|
|
|
|
|
def active(filename: Optional[str], name: str) -> bool:
|
|
return config.FAULT_INJECTION and bool(filename) and ("fault-" + name) in filename
|
|
|
|
|
|
def transient_until(filename: Optional[str]) -> int:
|
|
if not config.FAULT_INJECTION or not filename:
|
|
return 0
|
|
match = TRANSIENT_RE.search(filename)
|
|
return int(match.group(1)) if match else 0
|
|
|
|
|
|
def crash(point: str) -> None:
|
|
log.error("FAULT INJECTION: process exits at %s", point)
|
|
logging.shutdown()
|
|
os._exit(70)
|