lab5: RabbitMQ, relay и worker в стенде; reconciler lab4 убран

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

broker: RabbitMQ 4.3.7 (тег+digest), quorum queue + DLX/DLQ из definitions, пользователи
relay (только publish в exchange) и worker (только consume), без management UI и портов.
relay: outbox -> publish с publisher confirm, reaper истёкших lease, страховка от потерянных
сообщений. worker: claim + heartbeat lease, варианты, публикация с fencing, ack после COMMIT,
graceful stop, уборка staging и мусора попыток. S3: два bucket и четыре identity
(admin, api, worker, auditor) с проверкой модели доступа в s3-init. Отдельная сеть queue.
Reconciler lab4 (трек прошлой лабы) удалён: его роль выполняет уборка по журналу попыток.
This commit is contained in:
Alexey Bodanyuk
2026-10-31 16:12:52 +03:00
parent 655f264104
commit bc32a50967
15 changed files with 1058 additions and 141 deletions
+32 -5
View File
@@ -9,6 +9,9 @@ Delete ./secrets together with the volumes (`docker compose down -v`) to start o
"""
from __future__ import annotations
import base64
import hashlib
import json
import os
import secrets
import sys
@@ -24,8 +27,8 @@ def write(name: str, value: str) -> None:
print(f"keep secrets/{name}")
return
path.write_text(value + "\n", encoding="utf-8", newline="\n")
# Readable by the container users (postgres, valkey, seaweedfs, app uid 10001) through the
# compose bind mount. Local synthetic lab credentials only.
# Readable by the container users (postgres, valkey, seaweedfs, rabbitmq, app uid 10001)
# through the compose bind mount. Local synthetic lab credentials only.
os.chmod(path, 0o644)
print(f"created secrets/{name}")
@@ -34,17 +37,41 @@ def read(name: str) -> str:
return (SECRETS / name).read_text(encoding="utf-8").strip()
def rabbit_hash(password: str) -> str:
"""RabbitMQ `rabbit_password_hashing_sha256`: base64(salt + sha256(salt + password)), 4-byte salt.
The broker stores and compares only this hash; the plain password is in the client's URL."""
salt = secrets.token_bytes(4)
return base64.b64encode(salt + hashlib.sha256(salt + password.encode("utf-8")).digest()).decode("ascii")
def main() -> int:
SECRETS.mkdir(exist_ok=True)
write("postgres_password", secrets.token_urlsafe(24)) # [A-Za-z0-9_-]: safe inside URLs
write("valkey_password", secrets.token_urlsafe(24))
write("database_url", f"postgresql://messenger:{read('postgres_password')}@postgres:5432/messenger")
write("cache_url", f"redis://:{read('valkey_password')}@valkey:6379/0")
# Lab 4: three separate S3 key pairs (SeaweedFS identities, infra/seaweedfs/start.sh):
# admin = bucket bootstrap only; api = Read+Write on the bucket; reconciler = + List.
for who in ("admin", "api", "reconciler"):
# S3 key pairs (SeaweedFS identities, infra/seaweedfs/start.sh): admin = bucket bootstrap only;
# api = Write staging + Read final; worker = Read/Write both; auditor = Read + List (no Write).
for who in ("admin", "api", "worker", "auditor"):
write(f"s3_{who}_access_key_id", "AK" + secrets.token_hex(9).upper())
write(f"s3_{who}_secret_access_key", secrets.token_hex(20))
# Lab 5: RabbitMQ users. relay may only publish to the jobs exchange, worker may only consume
# the jobs queue; neither may declare/delete anything (configure "^$").
write("rabbitmq_relay_password", secrets.token_urlsafe(24))
write("rabbitmq_worker_password", secrets.token_urlsafe(24))
users = {
"users": [{"name": who, "password_hash": rabbit_hash(read(f"rabbitmq_{who}_password")),
"hashing_algorithm": "rabbit_password_hashing_sha256", "tags": []} for who in ("relay", "worker")],
"permissions": [
{"user": "relay", "vhost": "/", "configure": "^$", "write": "^messenger\\.jobs$", "read": "^$"},
{"user": "worker", "vhost": "/", "configure": "^$", "write": "^$", "read": "^messenger\\.attachment-jobs$"},
],
}
write("rabbitmq_users.json", json.dumps(users, indent=1))
for who in ("relay", "worker"):
# AMQP heartbeat 10 s: a frozen client loses its connection after ~2 missed heartbeats, and
# the broker returns its unacked messages to the queue for the other consumers.
write(f"broker_url_{who}", f"amqp://{who}:{read(f'rabbitmq_{who}_password')}@broker:5672/?heartbeat=10")
return 0
+12 -9
View File
@@ -1,18 +1,21 @@
#!/bin/sh
# Restart hook for `make test-persistence ACTION_SCRIPT=scripts/restart.sh`.
# Restarts EVERY process of the stand: LB, all API replicas, PostgreSQL, Valkey and S3 (SeaweedFS).
# Volumes, schema and data are preserved: no `down -v`, no FLUSH, no re-seed.
# Restarts EVERY process of the stand: LB, all API replicas, relay, worker(s), PostgreSQL, Valkey,
# S3 (SeaweedFS) and the broker (RabbitMQ). Volumes, schema, data and queued messages are
# preserved: no `down -v`, no FLUSH, no purge, no re-seed.
# Output is suppressed by the test; run it by hand to see the steps.
set -eu
cd "$(dirname "$0")/.."
# 1. Entry point and API first: no request is served while the stores restart underneath.
docker compose stop lb api
# 1. Entry point, API and the pipeline processes first: nothing is served or processed while the
# stores restart underneath.
docker compose stop lb api relay worker
# 2. Stores: clean shutdown (PostgreSQL checkpoint, Valkey AOF fsync, SeaweedFS volume/filer
# flush), then start again. The S3 bytes live on the named volume s3data.
docker compose restart postgres valkey s3
# 3. API replicas (wait for healthy stores via depends_on), then the LB. HAProxy resolves
# the new replica addresses through Docker DNS by itself - no stale backend list.
# flush, RabbitMQ quorum log), then start again. Bytes live on s3data, messages on rabbitmqdata.
docker compose restart postgres valkey s3 broker
# 3. API replicas (wait for healthy stores via depends_on), the LB, then relay and worker. HAProxy
# resolves the new replica addresses through Docker DNS by itself - no stale backend list.
docker compose start api
docker compose start lb
echo "restart.sh: lb, api (all replicas), postgres, valkey and s3 restarted with volumes preserved"
docker compose start relay worker
echo "restart.sh: lb, api (all replicas), relay, worker, postgres, valkey, s3 and broker restarted with volumes preserved"