forked from mtusi2026/mtusi2026lab6
Один trace_id проходит от HTTP через outbox и Kafka до worker, S3 и JDBC, включая повтор. Collector держит короткую очередь и memory limiter, поэтому остановка Prometheus, Loki или Tempo не останавливает приём сообщений.
Автор / группа: ivanstorozhev, БВТ2304 Целевая оценка (3/4/5): 4 Трек 5, если заявлен: не заявлен Что реализовано и что перенесено: из lab5 перенесены Spring Boot API, PostgreSQL, Redis-кэш сессий, Argon2id, Liquibase через одноразовый сервис migrate, HAProxy без sticky session, две реплики, X-Instance-Id, обязательный Idempotency-Key, scripts/stop-one.sh и scripts/restart.sh, приватный SeaweedFS, ответ 202, staging, transactional outbox, отдельный worker, Kafka 3.9.1 (KRaft, volume kafka-data), состояния queued/processing/ready/failed, JPEG thumbnail и crop, lossy WebP, retry с backoff и jitter, топик attachments.failed, явный replay и lease. Оригинал принимается в S3 в одной транзакции со строкой outbox. Worker коммитит смещение только после записи вариантов в фиксированные ключи. Сообщение брокера содержит идентификаторы и W3C traceparent, без тела сообщения и ключа объекта. Добавлены OTLP из API и worker в Collector 0.139.0, Prometheus v3.2.1, Loki 3.4.2, Tempo 2.7.2 и Grafana 11.6.1. Producer-спан и consumer-спан связаны через заголовок Kafka. Повторная попытка остаётся в том же trace. Дашборд messenger-lab6 считает p50/p95/p99 через histogram_quantile по сумме бакетов и делит миллисекунды на 1000. SLI доступности — доля 5xx на обычных маршрутах, SLO 99% за 5 минут. SLI очереди — возраст самого старого queued или processing, SLO меньше 30 с. Алерты messenger-error-rate и messenger-queue-lag проверяются локально, без внешней доставки. memory_limiter 192 MiB и очереди экспортёров на 256 записей сбрасывают телеметрию, если хранилище недоступно. Команды запуска из чистого clone: cp .env.example .env docker compose --env-file .env config --quiet docker compose --env-file .env up -d --build curl -fsS http://127.0.0.1:8080/health/ready make test make test-persistence ACTION_SCRIPT=scripts/restart.sh make test-failover ACTION_SCRIPT=scripts/stop-one.sh docker compose --env-file .env start api python3 scripts/grade6-evidence.py Результаты проверок и ссылка на REPORT.md: подробности и числа в REPORT.md make check — PASS: offline skeleton structure, local refs and Python syntax; application NOT tested. make test — PASS: lab6 public API scenario; infrastructure and grading evidence remain separate. python3 scripts/grade6-evidence.py — PASS. Загрузка bf19675a-7d93-4379-a708-1e5efc7366a7, trace d27d941c741436ad51a3b35c2921ca4d: HTTP-спан родитель s3.PutObject, JDBC и attachments.process publish; process — ребёнок publish; S3 и JDBC worker — дети process. Тот же trace_id в логе API attachment.accepted и в логе worker job.ready, и в Loki. Повтор ac1e1581-4d24-4789-a887-c14a1852581a в trace 5540712fa2d49096df09db864f9ad9a8: два publish и два process, S3 worker только на второй попытке, итог ready. До аварий p50 0.039 с, p95 0.132 с, p99 0.134 с по гистограмме за 5 минут. slow-db: JDBC query 3013 мс; slow-handler: сервер 3026 мс и ни одного JDBC-спана. Остановка worker: возраст очереди 75.5 с, алерт Alerting, после старта то же вложение ready и алерт Normal. 608 ответов 500: доля 1 и Alerting, после тишины доля 0 и Normal. При остановленных Prometheus, Loki и Tempo сообщение получило 201, вложение стало ready, память collector 100 → 98.87 MiB при лимите 320 MiB, 32 строки ошибки экспорта. mvn -q test в backend/ — код выхода 0, 22 теста, 0 failures, 0 errors. Ограничения / известные ошибки: трек 5 не делался. Один Docker-хост, единственные PostgreSQL, Redis, SeaweedFS, Kafka, Loki, Tempo и Prometheus остаются точками отказа. Replication factor топиков равен 1. Доставка at least once: повтор перезаписывает те же ключи, но это не exactly-once на стороне S3. Пока хранилище телеметрии лежит, часть спанов и строк отбрасывается; после возвращения дырка в этом интервале остаётся. Шаг метрик 10 с, поэтому короткий pg_sleep может не попасть в hikaricp_connections_pending; различие снято по спанам. Staging failed-задания хранится сутки; позже replay уже не находит байты. Пока HAProxy не снял backend, часть запросов на останавливаемую реплику завершается ошибкой соединения; повтор с тем же Idempotency-Key безопасен для отправки сообщения. Источники и использование ИИ: контракт contracts/openapi.json, contracts/README.md, contracts/observability.md, contracts/jobs.schema.json; код lab5; Spring Boot 4.1.1, Spring Data JPA, spring-security-crypto, Spring Data Redis, Spring Kafka, Liquibase, Actuator, OpenTelemetry, opentelemetry-logback-appender 2.28.1-alpha, datasource-micrometer-spring-boot 2.3.0, MapStruct 1.6.3, OpenAPI Generator 7.17.0, AWS SDK for Java 2.31.50, Thumbnailator 0.4.20, webp-imageio 0.11.0, Bouncy Castle, PostgreSQL 16.10, Redis 7.4.6, HAProxy 3.2.6, SeaweedFS 4.52, Apache Kafka 3.9.1, Collector 0.139.0, Prometheus v3.2.1, Loki 3.4.2, Tempo 2.7.2, Grafana 11.6.1, Testcontainers. Помощь ИИ использована при переносе lab5, инструментации и отчёте. Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами. Контракт и публичные тесты не ослаблены. make check и make test выполнены; приложен реальный вывод. Сценарии отказов своего уровня воспроизведены. Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens. README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
This commit is contained in:
@@ -0,0 +1,278 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Grade-4 fault demos. Prints statuses and counts only; never prints tokens or passwords."""
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure # noqa: E402
|
||||
from image_probe import jpeg_size # noqa: E402
|
||||
|
||||
lines = []
|
||||
|
||||
|
||||
def log(message):
|
||||
print(message, flush=True)
|
||||
lines.append(message)
|
||||
|
||||
|
||||
def run(args, check=True):
|
||||
result = subprocess.run(args, cwd=ROOT, text=True, capture_output=True)
|
||||
if check and result.returncode != 0:
|
||||
raise SystemExit(args[0] + " failed: " + result.stderr[-400:])
|
||||
return result
|
||||
|
||||
|
||||
def compose(*args):
|
||||
run(["docker", "compose", "--env-file", ".env", *args])
|
||||
|
||||
|
||||
def demo_token():
|
||||
for line in (ROOT / ".env").read_text().splitlines():
|
||||
if line.startswith("DEMO_TOKEN="):
|
||||
return line.split("=", 1)[1].strip()
|
||||
return "lab-demo"
|
||||
|
||||
|
||||
def psql(sql):
|
||||
result = run([
|
||||
"docker", "exec", "messenger-lab6-postgres-1", "sh", "-c",
|
||||
'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -c "$1"', "psql", sql,
|
||||
])
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def topic_end(topic):
|
||||
result = run([
|
||||
"docker", "exec", "messenger-lab6-kafka-1",
|
||||
"/opt/kafka/bin/kafka-get-offsets.sh",
|
||||
"--bootstrap-server", "localhost:9092",
|
||||
"--topic", topic,
|
||||
])
|
||||
total = 0
|
||||
for line in result.stdout.splitlines():
|
||||
parts = line.strip().split(":")
|
||||
if len(parts) >= 3 and parts[-1].lstrip("-").isdigit():
|
||||
total += int(parts[-1])
|
||||
return total
|
||||
|
||||
|
||||
def offsets():
|
||||
return "attachments.process end=" + str(topic_end("attachments.process")) + "\nattachments.failed end=" + str(topic_end("attachments.failed"))
|
||||
|
||||
|
||||
def restart_count():
|
||||
result = run(["docker", "inspect", "-f", "{{.RestartCount}}", "messenger-lab6-worker-1"])
|
||||
return int(result.stdout.strip())
|
||||
|
||||
|
||||
def wait_health(client):
|
||||
deadline = time.monotonic() + 90
|
||||
while True:
|
||||
try:
|
||||
client.call("GET", "/health/ready", schema="Health")
|
||||
return
|
||||
except Failure:
|
||||
if time.monotonic() > deadline:
|
||||
raise
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def poll(client, actor, attachment, seconds=70):
|
||||
deadline = time.monotonic() + seconds
|
||||
last = attachment
|
||||
while time.monotonic() < deadline:
|
||||
last = client.call("GET", "/api/v1/attachments/" + attachment["id"], actor=actor, schema="Attachment")["json"]
|
||||
if last["status"] in ("ready", "failed"):
|
||||
return last
|
||||
time.sleep(0.4)
|
||||
raise SystemExit("attachment stayed " + last["status"])
|
||||
|
||||
|
||||
def summary(attachment):
|
||||
variants = [
|
||||
f"{item['name']}:{item['content_type']}:{item['width']}x{item['height']}"
|
||||
for item in attachment.get("variants") or []
|
||||
]
|
||||
return f"id={attachment['id']} status={attachment['status']} error={attachment.get('error_code')} variants={variants}"
|
||||
|
||||
|
||||
def webp_size(data):
|
||||
if data[:4] != b"RIFF" or data[8:12] != b"WEBP":
|
||||
raise SystemExit("optimized payload is not WebP")
|
||||
index = 12
|
||||
while index + 8 <= len(data):
|
||||
kind = data[index:index + 4]
|
||||
size = int.from_bytes(data[index + 4:index + 8], "little")
|
||||
payload = data[index + 8:index + 8 + size]
|
||||
if kind == b"VP8 ":
|
||||
width = int.from_bytes(payload[6:8], "little") & 0x3FFF
|
||||
height = int.from_bytes(payload[8:10], "little") & 0x3FFF
|
||||
return width, height
|
||||
if kind == b"VP8X":
|
||||
width = 1 + int.from_bytes(payload[4:7], "little")
|
||||
height = 1 + int.from_bytes(payload[7:10], "little")
|
||||
return width, height
|
||||
index += 8 + size + (size & 1)
|
||||
raise SystemExit("WebP dimensions not found")
|
||||
|
||||
|
||||
def content(client, actor, attachment, variant=None):
|
||||
path = "/api/v1/attachments/" + attachment["id"] + "/content"
|
||||
if variant:
|
||||
path += "?variant=" + variant
|
||||
return client.call("GET", path, actor=actor)
|
||||
|
||||
|
||||
def main():
|
||||
token = demo_token()
|
||||
client = Client("http://127.0.0.1:8080", timeout=15)
|
||||
wait_health(client)
|
||||
headers = {"X-Demo-Token": token}
|
||||
|
||||
log("== worker down, API restart, then worker ==")
|
||||
compose("stop", "worker")
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
attachment, original = client.attachment(alice, chat)
|
||||
client.message(alice, chat, "вложение ещё queued", attachments=[attachment["id"]])
|
||||
blocked = client.call("GET", "/api/v1/attachments/" + attachment["id"] + "/content", 409, actor=bob)
|
||||
log("queued upload " + summary(attachment) + " content=" + blocked["json"]["error"]["code"])
|
||||
compose("restart", "api")
|
||||
wait_health(client)
|
||||
queued = client.call("GET", "/api/v1/attachments/" + attachment["id"], actor=bob, schema="Attachment")["json"]
|
||||
log("after API restart " + summary(queued))
|
||||
if queued["status"] != "queued":
|
||||
raise SystemExit("job was lost while worker and API were restarted")
|
||||
compose("start", "worker")
|
||||
ready = poll(client, bob, queued)
|
||||
log("after worker start " + summary(ready))
|
||||
downloaded = content(client, bob, ready)
|
||||
if downloaded["bytes"] != original:
|
||||
raise SystemExit("original bytes changed")
|
||||
thumb = content(client, bob, ready, "thumbnail")
|
||||
crop = content(client, bob, ready, "crop")
|
||||
optimized = content(client, bob, ready, "optimized")
|
||||
log(f"bytes original={len(original)} thumbnail={jpeg_size(thumb['bytes'])} {thumb['headers'].get_content_type()} "
|
||||
f"crop={jpeg_size(crop['bytes'])} {crop['headers'].get_content_type()} "
|
||||
f"optimized={webp_size(optimized['bytes'])} {optimized['headers'].get_content_type()}")
|
||||
|
||||
log("== broker restart with durable topic ==")
|
||||
compose("stop", "worker")
|
||||
published_before = topic_end("attachments.process")
|
||||
held, _ = client.attachment(alice, chat)
|
||||
deadline = time.monotonic() + 20
|
||||
published = published_before
|
||||
while time.monotonic() < deadline:
|
||||
published = topic_end("attachments.process")
|
||||
if published > published_before:
|
||||
break
|
||||
time.sleep(0.5)
|
||||
log("topic end before kafka restart: " + str(published_before) + " -> " + str(published))
|
||||
if published <= published_before:
|
||||
raise SystemExit("accepted job was not waiting in the broker")
|
||||
compose("restart", "kafka")
|
||||
deadline = time.monotonic() + 60
|
||||
while time.monotonic() < deadline:
|
||||
state = run(["docker", "inspect", "-f", "{{.State.Health.Status}}", "messenger-lab6-kafka-1"], check=False)
|
||||
if state.stdout.strip() == "healthy":
|
||||
break
|
||||
time.sleep(1)
|
||||
retained = topic_end("attachments.process")
|
||||
log("topic end after kafka restart: " + str(retained))
|
||||
if retained < published:
|
||||
raise SystemExit("broker restart dropped the waiting job")
|
||||
compose("start", "worker")
|
||||
held_ready = poll(client, alice, held)
|
||||
log("after broker restart " + summary(held_ready))
|
||||
|
||||
log("== outbox paused across API kill ==")
|
||||
paused = json.loads(client.call("POST", "/internal/demo/outbox-pause", headers=headers)["bytes"])
|
||||
staged, _ = client.attachment(alice, chat)
|
||||
outbox = json.loads(client.call("GET", "/internal/demo/outbox", headers=headers)["bytes"])
|
||||
blocked = client.call("GET", "/api/v1/attachments/" + staged["id"] + "/content", 409, actor=alice)
|
||||
log(f"paused={paused['paused']} unpublished={outbox['unpublished']} content={blocked['json']['error']['code']} {summary(staged)}")
|
||||
compose("kill", "api")
|
||||
compose("start", "api")
|
||||
wait_health(client)
|
||||
outbox = json.loads(client.call("GET", "/internal/demo/outbox", headers=headers)["bytes"])
|
||||
staged = client.call("GET", "/api/v1/attachments/" + staged["id"], actor=alice, schema="Attachment")["json"]
|
||||
log(f"after API kill paused={outbox['paused']} unpublished={outbox['unpublished']} {summary(staged)}")
|
||||
if not outbox["paused"] or outbox["unpublished"] < 1 or staged["status"] != "queued":
|
||||
raise SystemExit("outbox intention did not survive API kill")
|
||||
client.call("POST", "/internal/demo/outbox-resume", headers=headers)
|
||||
staged = poll(client, alice, staged)
|
||||
log("after outbox resume " + summary(staged))
|
||||
|
||||
log("== worker exit after object write, before ack ==")
|
||||
before = restart_count()
|
||||
run(["docker", "exec", "messenger-lab6-worker-1", "touch", "/tmp/messenger-crash-after-write"])
|
||||
crashed, _ = client.attachment(alice, chat)
|
||||
deadline = time.monotonic() + 40
|
||||
while time.monotonic() < deadline and restart_count() <= before:
|
||||
time.sleep(0.5)
|
||||
after = restart_count()
|
||||
crashed = poll(client, alice, crashed)
|
||||
variants = psql("select name || ':' || count(*) from attachment_variants where attachment_id='"
|
||||
+ crashed["id"] + "' group by name order by name")
|
||||
log(f"restart_count {before}->{after} {summary(crashed)} db={variants}")
|
||||
replay = json.loads(client.call("POST", "/internal/demo/attachments/" + crashed["id"] + "/replay", headers=headers)["bytes"])
|
||||
log("replay of ready status=" + replay["status"] + " variants=" + str(len(replay.get("variants") or [])))
|
||||
|
||||
log("== permanent failure and explicit replay ==")
|
||||
failed_before = json.loads(client.call("GET", "/internal/demo/failed", headers=headers)["bytes"])["count"]
|
||||
run(["docker", "exec", "messenger-lab6-worker-1", "touch", "/tmp/messenger-fail-permanent"])
|
||||
broken, _ = client.attachment(alice, chat)
|
||||
broken = poll(client, alice, broken)
|
||||
denied = client.call("POST", "/api/v1/chats/" + chat["id"] + "/messages", 409, actor=alice,
|
||||
body={"text": "нельзя прикрепить failed", "attachment_ids": [broken["id"]]},
|
||||
headers={"Idempotency-Key": "fail-" + broken["id"]})
|
||||
failed_after = json.loads(client.call("GET", "/internal/demo/failed", headers=headers)["bytes"])["count"]
|
||||
log(f"failed registry {failed_before}->{failed_after} message={denied['json']['error']['code']} {summary(broken)}")
|
||||
log("topics:\n" + offsets())
|
||||
replay = json.loads(client.call("POST", "/internal/demo/attachments/" + broken["id"] + "/replay", headers=headers)["bytes"])
|
||||
log("replay of failed status=" + replay["status"])
|
||||
recovered = poll(client, alice, replay)
|
||||
variants = psql("select name || ':' || count(*) from attachment_variants where attachment_id='"
|
||||
+ recovered["id"] + "' group by name order by name")
|
||||
again = json.loads(client.call("POST", "/internal/demo/attachments/" + recovered["id"] + "/replay", headers=headers)["bytes"])
|
||||
log(f"after replay {summary(recovered)} db={variants} second_replay={again['status']}")
|
||||
|
||||
log("== rejected upload ==")
|
||||
rejected = client.call("POST", "/api/v1/chats/" + chat["id"] + "/attachments?filename=bad.png", 415,
|
||||
actor=alice, raw=b"not-an-image", headers={"Content-Type": "image/png"})
|
||||
log("corrupt png " + rejected["json"]["error"]["code"])
|
||||
|
||||
log("== runtime limits ==")
|
||||
ps = run(["docker", "ps", "--filter", "name=messenger-lab6", "--format", "{{.Names}} {{.Status}} {{.Ports}}"])
|
||||
log(ps.stdout.rstrip())
|
||||
for name in ("api-1", "api-2", "worker-1", "lb-1", "postgres-1", "redis-1", "kafka-1", "s3-1"):
|
||||
inspected = run(["docker", "inspect", "-f",
|
||||
"{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}} {{json .HostConfig.PortBindings}}",
|
||||
"messenger-lab6-" + name])
|
||||
log(name + " " + inspected.stdout.strip())
|
||||
anonymous = run(["docker", "exec", "messenger-lab6-api-1", "curl", "-s", "-o", "/dev/null", "-w", "%{http_code}",
|
||||
"http://s3:8333/messenger-attachments/originals/missing"])
|
||||
log("anonymous s3 status=" + anonymous.stdout.strip())
|
||||
log("PASS grade4 evidence")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (Failure, TimeoutError, OSError) as exc:
|
||||
log("FAIL " + str(exc))
|
||||
raise SystemExit(1)
|
||||
finally:
|
||||
subprocess.run(["docker", "exec", "messenger-lab6-worker-1", "rm", "-f",
|
||||
"/tmp/messenger-crash-after-write", "/tmp/messenger-fail-permanent"], cwd=ROOT)
|
||||
token = demo_token()
|
||||
subprocess.run(["curl", "-fsS", "-X", "POST", "-H", "X-Demo-Token: " + token,
|
||||
"http://127.0.0.1:8080/internal/demo/outbox-resume"], cwd=ROOT,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
subprocess.run(["docker", "compose", "--env-file", ".env", "start", "worker", "api"], cwd=ROOT,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
(ROOT / "evidence").mkdir(exist_ok=True)
|
||||
(ROOT / "evidence" / "grade4.txt").write_text("\n".join(lines) + "\n")
|
||||
@@ -0,0 +1,346 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Grade-6 observability evidence. Prints statuses, ids, and metric values only."""
|
||||
import base64
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure # noqa: E402
|
||||
|
||||
OUT = ROOT / "evidence" / "grade6.txt"
|
||||
lines = []
|
||||
|
||||
|
||||
def log(message):
|
||||
print(message, flush=True)
|
||||
lines.append(message)
|
||||
|
||||
|
||||
def run(args, check=True):
|
||||
result = subprocess.run(args, cwd=ROOT, text=True, capture_output=True)
|
||||
if check and result.returncode != 0:
|
||||
raise SystemExit(args[0] + " failed: " + (result.stderr or result.stdout)[-400:])
|
||||
return result
|
||||
|
||||
|
||||
def compose(*args):
|
||||
run(["docker", "compose", "--env-file", ".env", *args])
|
||||
|
||||
|
||||
def env_value(name, default):
|
||||
for line in (ROOT / ".env").read_text().splitlines():
|
||||
if line.startswith(name + "="):
|
||||
return line.split("=", 1)[1].strip()
|
||||
return default
|
||||
|
||||
|
||||
def prom(query):
|
||||
url = "http://127.0.0.1:9090/api/v1/query?" + urllib.parse.urlencode({"query": query})
|
||||
with urllib.request.urlopen(url, timeout=10) as response:
|
||||
data = json.loads(response.read().decode())
|
||||
if data.get("status") != "success":
|
||||
return "error"
|
||||
result = data["data"]["result"]
|
||||
if not result:
|
||||
return "empty"
|
||||
if len(result) == 1 and not result[0]["metric"]:
|
||||
return result[0]["value"][1]
|
||||
return "; ".join(
|
||||
",".join(f"{k}={v}" for k, v in item["metric"].items() if k != "__name__") + "=" + item["value"][1]
|
||||
for item in result[:6]
|
||||
)
|
||||
|
||||
|
||||
def grafana(path):
|
||||
user = env_value("GRAFANA_ADMIN_USER", "admin")
|
||||
password = env_value("GRAFANA_ADMIN_PASSWORD", "lab-grafana")
|
||||
token = base64.b64encode((user + ":" + password).encode()).decode()
|
||||
request = urllib.request.Request(
|
||||
"http://127.0.0.1:3000" + path,
|
||||
headers={"Authorization": "Basic " + token},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
return json.loads(response.read().decode())
|
||||
|
||||
|
||||
def alert_states():
|
||||
payload = grafana("/api/prometheus/grafana/api/v1/alerts")
|
||||
states = {}
|
||||
for alert in payload.get("data", {}).get("alerts", []):
|
||||
states[alert.get("labels", {}).get("alertname", "?")] = alert.get("state", "?")
|
||||
return states
|
||||
|
||||
|
||||
def wait_state(name, wanted, timeout):
|
||||
deadline = time.monotonic() + timeout
|
||||
last = {}
|
||||
while time.monotonic() < deadline:
|
||||
last = alert_states()
|
||||
state = last.get(name, "missing")
|
||||
if state in wanted:
|
||||
return state
|
||||
time.sleep(5)
|
||||
return "timeout:" + last.get(name, "missing")
|
||||
|
||||
|
||||
def container_logs(name, since):
|
||||
result = run(["docker", "logs", "--since", since, name], check=False)
|
||||
return result.stdout + result.stderr
|
||||
|
||||
|
||||
def trace_id_for(attachment_id=None, route=None, since="5m"):
|
||||
deadline = time.monotonic() + 25
|
||||
while time.monotonic() < deadline:
|
||||
blob = ""
|
||||
for name in ("messenger-lab6-api-1", "messenger-lab6-api-2", "messenger-lab6-worker-1"):
|
||||
blob += container_logs(name, since)
|
||||
for line in blob.splitlines():
|
||||
if "trace_id=" not in line:
|
||||
continue
|
||||
if attachment_id and attachment_id not in line:
|
||||
continue
|
||||
if route and route not in line:
|
||||
continue
|
||||
if "password" in line.lower() or "Bearer " in line:
|
||||
continue
|
||||
marker = line.split("trace_id=", 1)[1]
|
||||
return marker.split()[0]
|
||||
time.sleep(1)
|
||||
return None
|
||||
|
||||
|
||||
def fetch_trace(trace_id):
|
||||
last = ""
|
||||
for _ in range(8):
|
||||
result = run([
|
||||
"docker", "exec", "messenger-lab6-api-1",
|
||||
"curl", "-fsS", "http://tempo:3200/api/traces/" + trace_id,
|
||||
], check=False)
|
||||
if result.returncode == 0 and result.stdout.strip().startswith("{"):
|
||||
return json.loads(result.stdout)
|
||||
last = (result.stderr or result.stdout)[-200:]
|
||||
time.sleep(2)
|
||||
raise SystemExit("tempo trace " + trace_id + " unavailable: " + last)
|
||||
|
||||
|
||||
def hex_id(value):
|
||||
if not value:
|
||||
return ""
|
||||
return base64.b64decode(value).hex()
|
||||
|
||||
|
||||
def span_rows(trace):
|
||||
rows = {}
|
||||
for batch in trace.get("batches", []):
|
||||
service = "?"
|
||||
for attribute in batch.get("resource", {}).get("attributes", []):
|
||||
if attribute.get("key") == "service.name":
|
||||
service = attribute.get("value", {}).get("stringValue", "?")
|
||||
for scope in batch.get("scopeSpans", []):
|
||||
for span in scope.get("spans", []):
|
||||
start = int(span.get("startTimeUnixNano", "0"))
|
||||
end = int(span.get("endTimeUnixNano", "0"))
|
||||
status = (span.get("status") or {}).get("code", "")
|
||||
rows[hex_id(span.get("spanId"))] = {
|
||||
"service": service,
|
||||
"name": span.get("name", ""),
|
||||
"parent": hex_id(span.get("parentSpanId")),
|
||||
"ms": (end - start) / 1_000_000,
|
||||
"status": status,
|
||||
}
|
||||
return rows
|
||||
|
||||
|
||||
def describe(trace_id, interesting):
|
||||
rows = span_rows(fetch_trace(trace_id))
|
||||
log("trace " + trace_id + " spans=" + str(len(rows)))
|
||||
for sid, row in rows.items():
|
||||
if not any(token in row["name"] for token in interesting):
|
||||
continue
|
||||
parent = rows.get(row["parent"], {}).get("name", "-")
|
||||
log(
|
||||
" "
|
||||
+ row["service"]
|
||||
+ " "
|
||||
+ row["name"]
|
||||
+ " parent="
|
||||
+ parent
|
||||
+ " dur_ms="
|
||||
+ format(row["ms"], ".1f")
|
||||
+ (" " + row["status"] if row["status"] else "")
|
||||
)
|
||||
query_parents = {}
|
||||
for row in rows.values():
|
||||
if row["name"] != "query":
|
||||
continue
|
||||
parent = rows.get(row["parent"], {}).get("name", "?")
|
||||
query_parents[parent] = query_parents.get(parent, 0) + 1
|
||||
if query_parents:
|
||||
log(" jdbc query parents " + json.dumps(query_parents, ensure_ascii=False))
|
||||
return rows
|
||||
|
||||
|
||||
def stats():
|
||||
result = run([
|
||||
"docker", "stats", "--no-stream", "--format", "{{.Name}} {{.MemUsage}}",
|
||||
"messenger-lab6-otel-collector-1",
|
||||
"messenger-lab6-api-1",
|
||||
"messenger-lab6-api-2",
|
||||
"messenger-lab6-worker-1",
|
||||
])
|
||||
return " | ".join(line.strip() for line in result.stdout.splitlines())
|
||||
|
||||
|
||||
def main():
|
||||
OUT.parent.mkdir(exist_ok=True)
|
||||
log("make test")
|
||||
test = subprocess.run(["make", "test"], cwd=ROOT, text=True, capture_output=True)
|
||||
(ROOT / "evidence" / "results.txt").write_text(test.stdout + test.stderr)
|
||||
tail = [line for line in (test.stdout + test.stderr).splitlines() if line.strip()]
|
||||
log("make test exit=" + str(test.returncode) + " last=" + (tail[-1] if tail else ""))
|
||||
|
||||
client = Client("http://127.0.0.1:8080")
|
||||
alice, _bob, _eve, chat = client.bootstrap()
|
||||
token = env_value("DEMO_TOKEN", "lab-demo")
|
||||
|
||||
log("upload chain")
|
||||
attachment, _content = client.attachment(alice, chat)
|
||||
ready = client.wait_ready(alice, attachment)
|
||||
log("attachment " + ready["id"] + " status=" + ready["status"])
|
||||
trace = trace_id_for(attachment_id=attachment["id"])
|
||||
log("accepted trace_id=" + str(trace))
|
||||
if trace:
|
||||
describe(trace, ("http post", "attachments.process", "s3."))
|
||||
|
||||
log("transient retry")
|
||||
run(["docker", "exec", "messenger-lab6-worker-1", "touch", "/tmp/messenger-fail-transient"])
|
||||
retried, _content = client.attachment(alice, chat)
|
||||
retried_ready = client.wait_ready(alice, retried, wait_seconds=60)
|
||||
log("retry attachment " + retried_ready["id"] + " status=" + retried_ready["status"])
|
||||
retry_trace = trace_id_for(attachment_id=retried["id"])
|
||||
log("retry trace_id=" + str(retry_trace))
|
||||
if retry_trace:
|
||||
rows = describe(retry_trace, ("attachments.process", "s3."))
|
||||
consumers = [row for row in rows.values() if row["name"] == "attachments.process process"]
|
||||
publishers = [row for row in rows.values() if row["name"] == "attachments.process publish"]
|
||||
log("retry process spans=" + str(len(consumers)) + " publish spans=" + str(len(publishers)))
|
||||
|
||||
log("latency before faults p50_s=" + str(prom(
|
||||
'histogram_quantile(0.50, sum by (le) (rate(http_server_requests_milliseconds_bucket{uri!~"/health.*"}[5m]))) / 1000'
|
||||
)))
|
||||
log("latency before faults p95_s=" + str(prom(
|
||||
'histogram_quantile(0.95, sum by (le) (rate(http_server_requests_milliseconds_bucket{uri!~"/health.*"}[5m]))) / 1000'
|
||||
)))
|
||||
log("latency before faults p99_s=" + str(prom(
|
||||
'histogram_quantile(0.99, sum by (le) (rate(http_server_requests_milliseconds_bucket{uri!~"/health.*"}[5m]))) / 1000'
|
||||
)))
|
||||
|
||||
log("slow database vs slow handler")
|
||||
client.call(
|
||||
"POST", "/internal/demo/slow-db?seconds=3", expected=200, actor=alice,
|
||||
headers={"X-Demo-Token": token}, schema=None,
|
||||
)
|
||||
slow_db = trace_id_for(route="route=/internal/demo/slow-db")
|
||||
log("slow-db trace_id=" + str(slow_db))
|
||||
if slow_db:
|
||||
rows = describe(slow_db, ("http post",))
|
||||
slowest = max((row["ms"] for row in rows.values() if row["name"] == "query"), default=0)
|
||||
log("slow-db longest jdbc query ms=" + format(slowest, ".1f"))
|
||||
client.call(
|
||||
"POST", "/internal/demo/slow-handler?seconds=3", expected=200, actor=alice,
|
||||
headers={"X-Demo-Token": token}, schema=None,
|
||||
)
|
||||
slow_handler = trace_id_for(route="route=/internal/demo/slow-handler")
|
||||
log("slow-handler trace_id=" + str(slow_handler))
|
||||
if slow_handler:
|
||||
rows = describe(slow_handler, ("http post",))
|
||||
slowest = max((row["ms"] for row in rows.values() if row["name"] == "query"), default=0)
|
||||
server = max((row["ms"] for row in rows.values() if row["name"].startswith("http post")), default=0)
|
||||
log("slow-handler server ms=" + format(server, ".1f") + " longest jdbc query ms=" + format(slowest, ".1f"))
|
||||
|
||||
log("queue lag incident")
|
||||
compose("stop", "worker")
|
||||
queued, _content = client.attachment(alice, chat)
|
||||
log("queued while worker stopped " + queued["id"])
|
||||
deadline = time.monotonic() + 40
|
||||
age = "empty"
|
||||
while time.monotonic() < deadline:
|
||||
age = prom('max(messenger_queue_oldest_age_seconds{queue="attachments.process"})')
|
||||
try:
|
||||
if float(age) > 15:
|
||||
break
|
||||
except ValueError:
|
||||
pass
|
||||
time.sleep(5)
|
||||
log("queue age after wait s=" + str(age))
|
||||
state = wait_state("Attachment queue lag", {"Alerting"}, 150)
|
||||
log("queue alert while stopped=" + state + " age_s=" + str(prom(
|
||||
'max(messenger_queue_oldest_age_seconds{queue="attachments.process"})'
|
||||
)))
|
||||
compose("start", "worker")
|
||||
recovered = client.wait_ready(alice, queued, wait_seconds=90)
|
||||
log("queue recovered " + recovered["id"] + " status=" + recovered["status"])
|
||||
resolved = wait_state("Attachment queue lag", {"Normal", "Normal (NoData)"}, 90)
|
||||
log("queue alert after recovery=" + resolved + " age_s=" + str(prom(
|
||||
'max(messenger_queue_oldest_age_seconds{queue="attachments.process"})'
|
||||
)))
|
||||
|
||||
log("error rate incident")
|
||||
started = time.monotonic()
|
||||
sent = 0
|
||||
while time.monotonic() - started < 100:
|
||||
client.call("POST", "/internal/demo/error", expected=500, actor=alice, headers={"X-Demo-Token": token})
|
||||
sent += 1
|
||||
time.sleep(0.15)
|
||||
log("demo 500 responses=" + str(sent))
|
||||
error_state = wait_state("High business HTTP error rate", {"Alerting", "Pending"}, 40)
|
||||
ratio = prom(
|
||||
'sum(rate(http_server_requests_milliseconds_count{status=~"5..",uri!~"/health.*"}[2m]))'
|
||||
' / clamp_min(sum(rate(http_server_requests_milliseconds_count{uri!~"/health.*"}[2m])), 1e-9)'
|
||||
)
|
||||
log("error alert during burst=" + error_state + " ratio=" + str(ratio))
|
||||
resolved = wait_state("High business HTTP error rate", {"Normal", "Normal (NoData)"}, 200)
|
||||
log("error alert after quiet=" + resolved + " ratio=" + str(prom(
|
||||
'sum(rate(http_server_requests_milliseconds_count{status=~"5..",uri!~"/health.*"}[2m]))'
|
||||
' / clamp_min(sum(rate(http_server_requests_milliseconds_count{uri!~"/health.*"}[2m])), 1e-9)'
|
||||
)))
|
||||
|
||||
log("telemetry backends down")
|
||||
before = stats()
|
||||
log("mem before " + before)
|
||||
compose("stop", "prometheus", "loki", "tempo")
|
||||
message = client.message(alice, chat, "telemetry-down")
|
||||
log("message while stores down status=201 id=" + message["json"]["id"])
|
||||
held, _content = client.attachment(alice, chat)
|
||||
held_ready = client.wait_ready(alice, held, wait_seconds=60)
|
||||
log("attachment while stores down " + held_ready["id"] + " status=" + held_ready["status"])
|
||||
log("mem during " + stats())
|
||||
time.sleep(20)
|
||||
log("mem 20s later " + stats())
|
||||
collector = container_logs("messenger-lab6-otel-collector-1", "2m")
|
||||
drops = 0
|
||||
for line in collector.splitlines():
|
||||
lowered = line.lower()
|
||||
if "export" in lowered and ("fail" in lowered or "error" in lowered or "drop" in lowered):
|
||||
drops += 1
|
||||
log("collector export failure lines in last 2m=" + str(drops))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (Failure, OSError, subprocess.SubprocessError, json.JSONDecodeError) as exc:
|
||||
log("stopped: " + type(exc).__name__ + " " + str(exc).splitlines()[0][:300])
|
||||
raise
|
||||
finally:
|
||||
subprocess.run(
|
||||
["docker", "compose", "--env-file", ".env", "start", "worker", "prometheus", "loki", "tempo"],
|
||||
cwd=ROOT, text=True, capture_output=True,
|
||||
)
|
||||
OUT.write_text("\n".join(lines) + "\n")
|
||||
Executable
+22
@@ -0,0 +1,22 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
if [ -f .env ]; then
|
||||
docker compose --env-file .env restart lb api worker postgres redis s3 kafka
|
||||
else
|
||||
docker compose restart lb api worker postgres redis s3 kafka
|
||||
fi
|
||||
|
||||
i=0
|
||||
while [ "$i" -lt 60 ]; do
|
||||
if curl -fsS -m 2 http://127.0.0.1:8080/health/ready >/dev/null 2>&1; then
|
||||
# SeaweedFS answers master status before object reads fit the 2s client timeout.
|
||||
sleep 12
|
||||
curl -fsS -m 2 http://127.0.0.1:8080/health/ready >/dev/null 2>&1
|
||||
exit 0
|
||||
fi
|
||||
i=$((i + 1))
|
||||
sleep 1
|
||||
done
|
||||
echo "stack did not become ready" >&2
|
||||
exit 1
|
||||
Executable
+24
@@ -0,0 +1,24 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
: "${TARGET_INSTANCE_ID:?test provides target API instance alias}"
|
||||
|
||||
cid=""
|
||||
for id in $(docker ps -q --filter "label=com.docker.compose.project=messenger-lab6" --filter "label=com.docker.compose.service=api"); do
|
||||
full=$(docker inspect -f '{{.Id}}' "$id")
|
||||
host=$(docker inspect -f '{{.Config.Hostname}}' "$id")
|
||||
name=$(docker inspect -f '{{.Name}}' "$id")
|
||||
name=${name#/}
|
||||
short=$(printf '%s' "$full" | cut -c1-12)
|
||||
if [ "$host" = "$TARGET_INSTANCE_ID" ] || [ "$short" = "$TARGET_INSTANCE_ID" ] || [ "$name" = "$TARGET_INSTANCE_ID" ] || [ "$full" = "$TARGET_INSTANCE_ID" ]; then
|
||||
cid=$id
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$cid" ]; then
|
||||
echo "no running api replica matches TARGET_INSTANCE_ID" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
docker stop -t 6 "$cid" >/dev/null
|
||||
Reference in New Issue
Block a user