Files
mtusi2026lab6/scripts/grade4-evidence.py
T
ivanstorozhev 33bc3616f7 Один trace_id проходит от HTTP через outbox и Kafka до worker, S3 и JDBC, включая повтор. Collector держит короткую очередь и memory limiter, поэтому остановка Prometheus, Loki или Tempo не останавливает приём сообщений.
Автор / группа: ivanstorozhev, БВТ2304
Целевая оценка (3/4/5): 4
Трек 5, если заявлен: не заявлен

Что реализовано и что перенесено: из lab5 перенесены Spring Boot API, PostgreSQL, Redis-кэш сессий, Argon2id, Liquibase через одноразовый сервис migrate, HAProxy без sticky session, две реплики, X-Instance-Id, обязательный Idempotency-Key, scripts/stop-one.sh и scripts/restart.sh, приватный SeaweedFS, ответ 202, staging, transactional outbox, отдельный worker, Kafka 3.9.1 (KRaft, volume kafka-data), состояния queued/processing/ready/failed, JPEG thumbnail и crop, lossy WebP, retry с backoff и jitter, топик attachments.failed, явный replay и lease. Оригинал принимается в S3 в одной транзакции со строкой outbox. Worker коммитит смещение только после записи вариантов в фиксированные ключи. Сообщение брокера содержит идентификаторы и W3C traceparent, без тела сообщения и ключа объекта. Добавлены OTLP из API и worker в Collector 0.139.0, Prometheus v3.2.1, Loki 3.4.2, Tempo 2.7.2 и Grafana 11.6.1. Producer-спан и consumer-спан связаны через заголовок Kafka. Повторная попытка остаётся в том же trace. Дашборд messenger-lab6 считает p50/p95/p99 через histogram_quantile по сумме бакетов и делит миллисекунды на 1000. SLI доступности — доля 5xx на обычных маршрутах, SLO 99% за 5 минут. SLI очереди — возраст самого старого queued или processing, SLO меньше 30 с. Алерты messenger-error-rate и messenger-queue-lag проверяются локально, без внешней доставки. memory_limiter 192 MiB и очереди экспортёров на 256 записей сбрасывают телеметрию, если хранилище недоступно.

Команды запуска из чистого clone:
cp .env.example .env
docker compose --env-file .env config --quiet
docker compose --env-file .env up -d --build
curl -fsS http://127.0.0.1:8080/health/ready
make test
make test-persistence ACTION_SCRIPT=scripts/restart.sh
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
docker compose --env-file .env start api
python3 scripts/grade6-evidence.py

Результаты проверок и ссылка на REPORT.md: подробности и числа в REPORT.md
make check — PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
make test — PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
python3 scripts/grade6-evidence.py — PASS. Загрузка bf19675a-7d93-4379-a708-1e5efc7366a7, trace d27d941c741436ad51a3b35c2921ca4d: HTTP-спан родитель s3.PutObject, JDBC и attachments.process publish; process — ребёнок publish; S3 и JDBC worker — дети process. Тот же trace_id в логе API attachment.accepted и в логе worker job.ready, и в Loki. Повтор ac1e1581-4d24-4789-a887-c14a1852581a в trace 5540712fa2d49096df09db864f9ad9a8: два publish и два process, S3 worker только на второй попытке, итог ready. До аварий p50 0.039 с, p95 0.132 с, p99 0.134 с по гистограмме за 5 минут. slow-db: JDBC query 3013 мс; slow-handler: сервер 3026 мс и ни одного JDBC-спана. Остановка worker: возраст очереди 75.5 с, алерт Alerting, после старта то же вложение ready и алерт Normal. 608 ответов 500: доля 1 и Alerting, после тишины доля 0 и Normal. При остановленных Prometheus, Loki и Tempo сообщение получило 201, вложение стало ready, память collector 100 → 98.87 MiB при лимите 320 MiB, 32 строки ошибки экспорта.
mvn -q test в backend/ — код выхода 0, 22 теста, 0 failures, 0 errors.

Ограничения / известные ошибки: трек 5 не делался. Один Docker-хост, единственные PostgreSQL, Redis, SeaweedFS, Kafka, Loki, Tempo и Prometheus остаются точками отказа. Replication factor топиков равен 1. Доставка at least once: повтор перезаписывает те же ключи, но это не exactly-once на стороне S3. Пока хранилище телеметрии лежит, часть спанов и строк отбрасывается; после возвращения дырка в этом интервале остаётся. Шаг метрик 10 с, поэтому короткий pg_sleep может не попасть в hikaricp_connections_pending; различие снято по спанам. Staging failed-задания хранится сутки; позже replay уже не находит байты. Пока HAProxy не снял backend, часть запросов на останавливаемую реплику завершается ошибкой соединения; повтор с тем же Idempotency-Key безопасен для отправки сообщения.

Источники и использование ИИ: контракт contracts/openapi.json, contracts/README.md, contracts/observability.md, contracts/jobs.schema.json; код lab5; Spring Boot 4.1.1, Spring Data JPA, spring-security-crypto, Spring Data Redis, Spring Kafka, Liquibase, Actuator, OpenTelemetry, opentelemetry-logback-appender 2.28.1-alpha, datasource-micrometer-spring-boot 2.3.0, MapStruct 1.6.3, OpenAPI Generator 7.17.0, AWS SDK for Java 2.31.50, Thumbnailator 0.4.20, webp-imageio 0.11.0, Bouncy Castle, PostgreSQL 16.10, Redis 7.4.6, HAProxy 3.2.6, SeaweedFS 4.52, Apache Kafka 3.9.1, Collector 0.139.0, Prometheus v3.2.1, Loki 3.4.2, Tempo 2.7.2, Grafana 11.6.1, Testcontainers. Помощь ИИ использована при переносе lab5, инструментации и отчёте.

Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
Контракт и публичные тесты не ослаблены.
make check и make test выполнены; приложен реальный вывод.
Сценарии отказов своего уровня воспроизведены.
Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
2026-11-19 23:31:24 +03:00

279 lines
12 KiB
Python

#!/usr/bin/env python3
"""Grade-4 fault demos. Prints statuses and counts only; never prints tokens or passwords."""
import json
import subprocess
import sys
import time
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, Failure # noqa: E402
from image_probe import jpeg_size # noqa: E402
lines = []
def log(message):
print(message, flush=True)
lines.append(message)
def run(args, check=True):
result = subprocess.run(args, cwd=ROOT, text=True, capture_output=True)
if check and result.returncode != 0:
raise SystemExit(args[0] + " failed: " + result.stderr[-400:])
return result
def compose(*args):
run(["docker", "compose", "--env-file", ".env", *args])
def demo_token():
for line in (ROOT / ".env").read_text().splitlines():
if line.startswith("DEMO_TOKEN="):
return line.split("=", 1)[1].strip()
return "lab-demo"
def psql(sql):
result = run([
"docker", "exec", "messenger-lab6-postgres-1", "sh", "-c",
'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -c "$1"', "psql", sql,
])
return result.stdout.strip()
def topic_end(topic):
result = run([
"docker", "exec", "messenger-lab6-kafka-1",
"/opt/kafka/bin/kafka-get-offsets.sh",
"--bootstrap-server", "localhost:9092",
"--topic", topic,
])
total = 0
for line in result.stdout.splitlines():
parts = line.strip().split(":")
if len(parts) >= 3 and parts[-1].lstrip("-").isdigit():
total += int(parts[-1])
return total
def offsets():
return "attachments.process end=" + str(topic_end("attachments.process")) + "\nattachments.failed end=" + str(topic_end("attachments.failed"))
def restart_count():
result = run(["docker", "inspect", "-f", "{{.RestartCount}}", "messenger-lab6-worker-1"])
return int(result.stdout.strip())
def wait_health(client):
deadline = time.monotonic() + 90
while True:
try:
client.call("GET", "/health/ready", schema="Health")
return
except Failure:
if time.monotonic() > deadline:
raise
time.sleep(1)
def poll(client, actor, attachment, seconds=70):
deadline = time.monotonic() + seconds
last = attachment
while time.monotonic() < deadline:
last = client.call("GET", "/api/v1/attachments/" + attachment["id"], actor=actor, schema="Attachment")["json"]
if last["status"] in ("ready", "failed"):
return last
time.sleep(0.4)
raise SystemExit("attachment stayed " + last["status"])
def summary(attachment):
variants = [
f"{item['name']}:{item['content_type']}:{item['width']}x{item['height']}"
for item in attachment.get("variants") or []
]
return f"id={attachment['id']} status={attachment['status']} error={attachment.get('error_code')} variants={variants}"
def webp_size(data):
if data[:4] != b"RIFF" or data[8:12] != b"WEBP":
raise SystemExit("optimized payload is not WebP")
index = 12
while index + 8 <= len(data):
kind = data[index:index + 4]
size = int.from_bytes(data[index + 4:index + 8], "little")
payload = data[index + 8:index + 8 + size]
if kind == b"VP8 ":
width = int.from_bytes(payload[6:8], "little") & 0x3FFF
height = int.from_bytes(payload[8:10], "little") & 0x3FFF
return width, height
if kind == b"VP8X":
width = 1 + int.from_bytes(payload[4:7], "little")
height = 1 + int.from_bytes(payload[7:10], "little")
return width, height
index += 8 + size + (size & 1)
raise SystemExit("WebP dimensions not found")
def content(client, actor, attachment, variant=None):
path = "/api/v1/attachments/" + attachment["id"] + "/content"
if variant:
path += "?variant=" + variant
return client.call("GET", path, actor=actor)
def main():
token = demo_token()
client = Client("http://127.0.0.1:8080", timeout=15)
wait_health(client)
headers = {"X-Demo-Token": token}
log("== worker down, API restart, then worker ==")
compose("stop", "worker")
alice, bob, _eve, chat = client.bootstrap()
attachment, original = client.attachment(alice, chat)
client.message(alice, chat, "вложение ещё queued", attachments=[attachment["id"]])
blocked = client.call("GET", "/api/v1/attachments/" + attachment["id"] + "/content", 409, actor=bob)
log("queued upload " + summary(attachment) + " content=" + blocked["json"]["error"]["code"])
compose("restart", "api")
wait_health(client)
queued = client.call("GET", "/api/v1/attachments/" + attachment["id"], actor=bob, schema="Attachment")["json"]
log("after API restart " + summary(queued))
if queued["status"] != "queued":
raise SystemExit("job was lost while worker and API were restarted")
compose("start", "worker")
ready = poll(client, bob, queued)
log("after worker start " + summary(ready))
downloaded = content(client, bob, ready)
if downloaded["bytes"] != original:
raise SystemExit("original bytes changed")
thumb = content(client, bob, ready, "thumbnail")
crop = content(client, bob, ready, "crop")
optimized = content(client, bob, ready, "optimized")
log(f"bytes original={len(original)} thumbnail={jpeg_size(thumb['bytes'])} {thumb['headers'].get_content_type()} "
f"crop={jpeg_size(crop['bytes'])} {crop['headers'].get_content_type()} "
f"optimized={webp_size(optimized['bytes'])} {optimized['headers'].get_content_type()}")
log("== broker restart with durable topic ==")
compose("stop", "worker")
published_before = topic_end("attachments.process")
held, _ = client.attachment(alice, chat)
deadline = time.monotonic() + 20
published = published_before
while time.monotonic() < deadline:
published = topic_end("attachments.process")
if published > published_before:
break
time.sleep(0.5)
log("topic end before kafka restart: " + str(published_before) + " -> " + str(published))
if published <= published_before:
raise SystemExit("accepted job was not waiting in the broker")
compose("restart", "kafka")
deadline = time.monotonic() + 60
while time.monotonic() < deadline:
state = run(["docker", "inspect", "-f", "{{.State.Health.Status}}", "messenger-lab6-kafka-1"], check=False)
if state.stdout.strip() == "healthy":
break
time.sleep(1)
retained = topic_end("attachments.process")
log("topic end after kafka restart: " + str(retained))
if retained < published:
raise SystemExit("broker restart dropped the waiting job")
compose("start", "worker")
held_ready = poll(client, alice, held)
log("after broker restart " + summary(held_ready))
log("== outbox paused across API kill ==")
paused = json.loads(client.call("POST", "/internal/demo/outbox-pause", headers=headers)["bytes"])
staged, _ = client.attachment(alice, chat)
outbox = json.loads(client.call("GET", "/internal/demo/outbox", headers=headers)["bytes"])
blocked = client.call("GET", "/api/v1/attachments/" + staged["id"] + "/content", 409, actor=alice)
log(f"paused={paused['paused']} unpublished={outbox['unpublished']} content={blocked['json']['error']['code']} {summary(staged)}")
compose("kill", "api")
compose("start", "api")
wait_health(client)
outbox = json.loads(client.call("GET", "/internal/demo/outbox", headers=headers)["bytes"])
staged = client.call("GET", "/api/v1/attachments/" + staged["id"], actor=alice, schema="Attachment")["json"]
log(f"after API kill paused={outbox['paused']} unpublished={outbox['unpublished']} {summary(staged)}")
if not outbox["paused"] or outbox["unpublished"] < 1 or staged["status"] != "queued":
raise SystemExit("outbox intention did not survive API kill")
client.call("POST", "/internal/demo/outbox-resume", headers=headers)
staged = poll(client, alice, staged)
log("after outbox resume " + summary(staged))
log("== worker exit after object write, before ack ==")
before = restart_count()
run(["docker", "exec", "messenger-lab6-worker-1", "touch", "/tmp/messenger-crash-after-write"])
crashed, _ = client.attachment(alice, chat)
deadline = time.monotonic() + 40
while time.monotonic() < deadline and restart_count() <= before:
time.sleep(0.5)
after = restart_count()
crashed = poll(client, alice, crashed)
variants = psql("select name || ':' || count(*) from attachment_variants where attachment_id='"
+ crashed["id"] + "' group by name order by name")
log(f"restart_count {before}->{after} {summary(crashed)} db={variants}")
replay = json.loads(client.call("POST", "/internal/demo/attachments/" + crashed["id"] + "/replay", headers=headers)["bytes"])
log("replay of ready status=" + replay["status"] + " variants=" + str(len(replay.get("variants") or [])))
log("== permanent failure and explicit replay ==")
failed_before = json.loads(client.call("GET", "/internal/demo/failed", headers=headers)["bytes"])["count"]
run(["docker", "exec", "messenger-lab6-worker-1", "touch", "/tmp/messenger-fail-permanent"])
broken, _ = client.attachment(alice, chat)
broken = poll(client, alice, broken)
denied = client.call("POST", "/api/v1/chats/" + chat["id"] + "/messages", 409, actor=alice,
body={"text": "нельзя прикрепить failed", "attachment_ids": [broken["id"]]},
headers={"Idempotency-Key": "fail-" + broken["id"]})
failed_after = json.loads(client.call("GET", "/internal/demo/failed", headers=headers)["bytes"])["count"]
log(f"failed registry {failed_before}->{failed_after} message={denied['json']['error']['code']} {summary(broken)}")
log("topics:\n" + offsets())
replay = json.loads(client.call("POST", "/internal/demo/attachments/" + broken["id"] + "/replay", headers=headers)["bytes"])
log("replay of failed status=" + replay["status"])
recovered = poll(client, alice, replay)
variants = psql("select name || ':' || count(*) from attachment_variants where attachment_id='"
+ recovered["id"] + "' group by name order by name")
again = json.loads(client.call("POST", "/internal/demo/attachments/" + recovered["id"] + "/replay", headers=headers)["bytes"])
log(f"after replay {summary(recovered)} db={variants} second_replay={again['status']}")
log("== rejected upload ==")
rejected = client.call("POST", "/api/v1/chats/" + chat["id"] + "/attachments?filename=bad.png", 415,
actor=alice, raw=b"not-an-image", headers={"Content-Type": "image/png"})
log("corrupt png " + rejected["json"]["error"]["code"])
log("== runtime limits ==")
ps = run(["docker", "ps", "--filter", "name=messenger-lab6", "--format", "{{.Names}} {{.Status}} {{.Ports}}"])
log(ps.stdout.rstrip())
for name in ("api-1", "api-2", "worker-1", "lb-1", "postgres-1", "redis-1", "kafka-1", "s3-1"):
inspected = run(["docker", "inspect", "-f",
"{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}} {{json .HostConfig.PortBindings}}",
"messenger-lab6-" + name])
log(name + " " + inspected.stdout.strip())
anonymous = run(["docker", "exec", "messenger-lab6-api-1", "curl", "-s", "-o", "/dev/null", "-w", "%{http_code}",
"http://s3:8333/messenger-attachments/originals/missing"])
log("anonymous s3 status=" + anonymous.stdout.strip())
log("PASS grade4 evidence")
if __name__ == "__main__":
try:
main()
except (Failure, TimeoutError, OSError) as exc:
log("FAIL " + str(exc))
raise SystemExit(1)
finally:
subprocess.run(["docker", "exec", "messenger-lab6-worker-1", "rm", "-f",
"/tmp/messenger-crash-after-write", "/tmp/messenger-fail-permanent"], cwd=ROOT)
token = demo_token()
subprocess.run(["curl", "-fsS", "-X", "POST", "-H", "X-Demo-Token: " + token,
"http://127.0.0.1:8080/internal/demo/outbox-resume"], cwd=ROOT,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
subprocess.run(["docker", "compose", "--env-file", ".env", "start", "worker", "api"], cwd=ROOT,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
(ROOT / "evidence").mkdir(exist_ok=True)
(ROOT / "evidence" / "grade4.txt").write_text("\n".join(lines) + "\n")