ЛР 7 — БВТ2301 — Парамонов — на 5 #14

Closed
gleb_paramonov wants to merge 4 commits from gleb_paramonov/mtusi2026lab7:solution/bvt2301-paramonov into main
126 changed files with 10339 additions and 74 deletions
+3
View File
@@ -0,0 +1,3 @@
* text=auto eol=lf
*.png binary
*.jpg binary
+18
View File
@@ -0,0 +1,18 @@
# Official Docker Python image; immutable digest.
FROM python@sha256:5e2bdb1b0d542d5039cffb8181c65930093949194a65c365dfb1b0fe13a4fa5b
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONUTF8=1 \
APP_HOST=0.0.0.0 APP_PORT=8080
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
zlib1g=1:1.3.dfsg+really1.3.1-1+deb13u1 && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir pip==26.3 && pip install --no-cache-dir -r requirements.txt && pip uninstall -y setuptools && \
groupadd --gid 10001 messenger && useradd --uid 10001 --gid 10001 --no-create-home messenger
COPY src ./src
COPY migrations ./migrations
USER 10001:10001
EXPOSE 8080
STOPSIGNAL SIGTERM
HEALTHCHECK --interval=5s --timeout=6s --start-period=60s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=5)"
CMD ["python", "-m", "src"]
+219 -23
View File
@@ -1,37 +1,233 @@
# Отчёт — лабораторная 7
# ЛР7 — Парамонов Глеб, БВТ2301
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
Целевая оценка **5**, трек 1: **компрометация worker**. Ветка
`solution/bvt2301-paramonov`. Перенесена собственная реализация ЛР6: две API,
PostgreSQL, profile cache, приватный S3, transactional outbox, Valkey Streams,
worker с versioned image pipeline, reconciler, OTel/Grafana. Новый внешний API —
HTTPS. README задания, business contract и public tests не ослаблены; заполнен
предусмотренный заданием `contracts/trust-matrix.md`.
Код, исследование и оформление подготовлены с помощью OpenAI Codex. Результаты
в `evidence/` получены на реальном локальном стенде, а не сгенерированы как
пример. Это учебная реализация в заявленной модели угроз, не production seal.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Нужны Git, Docker Engine/Compose v2, Python 3.12+ и сеть для официальных images/
PyPI. Проверено на Windows10, Docker Desktop4.39 / Engine28 / Compose2.33, WSL2
4CPU/3GiB RAM/4GiB swap, i5-11400H/8GiB host RAM. Версии образов закреплены digest
в `compose.yaml`/`Dockerfile`; зависимости — точными версиями в requirements.
Предыдущие стенды остановить: административный Grafana port3000 общий.
## Архитектура и решения
```sh
python -m venv .venv
# Windows: вместо .venv/bin/python используйте .venv/Scripts/python.exe
.venv/bin/python -m pip install -r requirements-dev.txt -r requirements-test.txt
.venv/bin/python -X utf8 scripts/up.py
.venv/bin/python -X utf8 scripts/verify_public.py
```
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab6, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
`up.py` — явная операторская команда. Она создаёт локальные секреты, строит image,
согласованно останавливает старые процессы без удаления volumes, запускает Bao
и хранилища, выполняет init/unseal, migrations/GRANT, policy/KV provisioning,
создаёт private bucket, запускает остальное и ждёт verified HTTPS readiness.
Повтор сохраняет пароли/CA/data и проверяет hashes применённых migrations.
Leaf certificates с остатком <2h обновляются оператором. Это cold-start
workflow, бесшовный rollout не заявляется. На этой машине dependencies для
работы выполнялись также из соседнего существующего venv; для проверяющего
полностью достаточно собственного venv этого репозитория.
## Доказательства критериев
После запуска: API https://localhost:8443, Grafana https://localhost:3000
(login admin, пароль только в локальном `secrets/grafana-password`), Bao API
https://localhost:8200 (обязательный operator client cert). Не отключайте
проверку CA: для Python/public tests `--ca-file secrets/trust/ca.crt`. Для
браузера сначала установите публичный учебный CA вручную, если нужен UI.
HTTP на этих портах не является fallback.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Эквивалентные операции по шагам приведены в `scripts/up.py`. Обычный restart
API не требует unseal. После перезапуска Bao:
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
```sh
.venv/bin/python scripts/bao_operator.py unseal
```
## Эксперименты и отказы
Root token и 3 unseal shares (threshold2) находятся только в
`secrets/operator/bao-init.json`; CA key — рядом в `ca.key`. Их не передают в
PR/чат, не монтируют в runtime и не печатают. Windows bootstrap убирает inherited
ACL и оставляет owner/SYSTEM/Administrators; POSIX secrets0700/files0600.
Все доли на одном защищённом компьютере — ограничение учебного bootstrap,
не production secret custody. Резервные копии этих файлов должен хранить
оператор отдельно от data volume; потеря shares делает sealed data недоступными.
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Остановка с сохранением данных: `docker compose down`. Осознанное удаление
**всех учебных данных только этого проекта**: `docker compose down --volumes`.
После такого сброса следует создать новый clone/workdir и новые secrets;
старый bao-init не соответствует новому пустому raft volume. Команда удаления
не используется тестами. Restart hook делает stop/start в dependency order,
потому что приложение и его TLS sidecar разделяют network namespace.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
## Архитектура и эксплуатация
## Ограничения и следующие шаги
Клиент → TLS LB → mTLS API1/API2 → mTLS DB/S3/cache/Collector. Worker и dispatcher
используют отдельные broker identities. Collector экспортирует через mTLS в
Loki/Tempo; Prometheus читает защищённые scrape endpoints; Grafana читает data
sources через собственные mTLS tunnels. [Полная матрица](contracts/trust-matrix.md).
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
В backend namespace native listener только loopback. Прокси использует Python
stdlib/OpenSSL, проверяет CA, hostname и SAN allowlist; TLS>=1.2, CRL, максимум
64 активные/96 ожидающие задачи, 16KiB buffers, 60s connection lifetime. TLS
context cache <=1s, поэтому срок прекращения существующей связи <=61s. TLS
session tickets на server proxy выключены. Прокси не логирует payload. Native
TLS применяется у LB, Bao и Grafana. CA private key в services отсутствует.
Secret supervisor использует native mTLS cert auth OpenBao, а не постоянный
секрет «для получения секретов». Политика разрешает read только собственного
KV path. Token TTL/maxTTL120s, re-auth20s; после чтения token revoke-self.
Пароли БД/S3/Valkey — **static KV**, их реальная валидность заканчивается
ротацией backend. Локальный budget120s ограничивает продолжение работы child
при sealed/unavailable Bao, но не обезвреживает скопированный злоумышленником
пароль. Смена KV перезапускает child и закрывает старые pooled connections.
Crash child также обрабатывается supervisor без смены network namespace.
PostgreSQL — источник истины для пользователей, sessions, идемпотентности,
attachments/jobs/outbox. Native host authentication на loopback — SCRAM,
не trust. Только операторский Unix socket использует доверенный local role;
этот socket другим контейнерам не расшарен. Worker не читает users/sessions/
messages и не может DDL/role administration. API имеет business-table grants;
worker/dispatcher — jobs/attachments/outbox/audit, reconciler — attachments.
Dispatcher не получает S3 credentials. Valkey ACL ограничивает команды и key;
S3 policy — bucket prefixes. Отдельные credentials каждой роли лежат в Bao.
Argon2id: 32MiB, t2, p1, случайный salt. При успешном входе старые параметры
rehash с compare-and-swap. Session token — 256 случайных бит, в PostgreSQL
только SHA-256; expiry1h/logout проверяются по БД. KDF concurrency2/API, общий
limiter12 login attempts/10s и8 registration/10s. 429 содержит Retry-After10.
Это глобальный учебный бюджет, не распределённая антибот-система по пользователям.
Runtime non-privileged, capabilities dropped, no-new-privileges, read_only root
filesystem, RAM/CPU/pids limits, writable named volumes/tmpfs только где нужны.
Worker1CPU/768MiB, render subprocess имеет отдельный timeout/лимиты. Init jobs
кратковременно root с CHOWN/FOWNER/DAC_OVERRIDE для отдельных secret volumes;
в runtime этих capabilities нет. Stateful write paths: PG/Seaweed/Valkey data,
Bao raft+audit, Grafana/Prometheus/Loki/Tempo data. Наружу только loopback
8443/8200/3000. `.env.example` — исходный перечень ролей, фактические параметры
создаются `bootstrap.py`: `.env` только DB bootstrap, service env приходит из Bao,
S3 admin имеет отдельный `.env.storage-admin` лишь в operator profile.
## Проверки и доказательства
Финальная проверка выполнена из отдельного clone `clean-lab7-final` с собственным
venv, новой PKI/секретами и новыми volumes. Код соответствует commit `0bf1ff9`;
финальные evidence/документация зафиксированы отдельным последующим коммитом. Проверка публичного
набора 13.11.2026 прошла целиком: contract, OpenAPI, HTTPS smoke/security,
persistence, три failover-запуска с двумя различными container IDs. После fault
первые успешные записи восстановились за 3.390/2.953/2.610s. Первые два запуска
выбрали одну реплику, третий — другую; все реальные процессы остановлены SIGKILL,
без замены target теста. Перед следующим сценарием обе API и TLS sidecars
восстанавливаются согласованно. Публичные tests не изменены.
Полный security suite завершился PASS 13.11.2026 в 10:00:37 UTC: все 11 этапов,
включая TLS allow/deny matrix, Bao policies, worker permissions, API security,
компрометацию worker, sealed Bao, telemetry, hardening и 4 renderer unit tests.
При компрометации восстановлены 3/3 задания без изменения содержимого и образа;
старые certificate/Bao auth/DB password отклонены; полный эксперимент 35.61s.
При sealed Bao обе API child остановились через 123.86s (120s budget + grace),
восстановление после unseal 20.44s. Argon2 hash+verify median 0.207s,
maxRSS 53184KiB; 20 login за 4.578s дали 12×401 и 8×429 на двух API.
Это результаты одного запуска на указанном стенде, а не гарантированные SLO.
Сводка: [clean-clone.json](evidence/clean-clone.json),
[public suite](evidence/public-suite.txt), [security suite](evidence/security-suite.txt).
Image scan от 12.11 предшествует исправлению планирования TLS handshake;
зависимости не изменились, но scan не заявляется для точного app image ID
финального запуска (он отдельно записан в clean-clone.json).
```sh
.venv/bin/python scripts/verify_public.py
.venv/bin/python scripts/verify_security.py
# Ниже отдельные сценарии для воспроизведения/диагностики, уже включённые в suite:
.venv/bin/python scripts/test_identities.py
.venv/bin/python scripts/bao_operator.py publish-crl
docker compose run --rm security-check
# В Windows вывод перенаправлять в UTF-8, пример: > evidence/tls-matrix.json
docker compose run --rm security-check python /checks/bao_policy.py
.venv/bin/python checks/security_api.py
.venv/bin/python checks/compromised_worker.py
.venv/bin/python checks/sealed_bao.py
.venv/bin/python checks/telemetry_security.py
.venv/bin/python checks/hardening.py
docker compose exec -T worker python - < checks/worker_rights.py
docker compose run --rm --no-deps -v "$PWD/checks:/checks:ro" api1 python -m unittest discover -s /checks -p test_render.py
.venv/bin/python scripts/scan_images.py
```
Последний `run api1` использует только unittest renderer, service secret supervisor
не запускается (command переопределён). PowerShell stdin: `Get-Content
checks/worker_rights.py -Raw | docker compose exec -T worker python -`.
Перед повтором быстрых suites дайте limiter10s; не отключайте его ради PASS.
| Критерий | Сценарий / evidence |
| --- | --- |
| Неизменённый business contract/OpenAPI, HTTPS smoke | verify_public → check_contract.txt, validate_openapi.txt, smoke.txt, security.txt |
| Сохранность всех данных/сессии/файла после restart | persistence.txt + restart-processes.json, реальные StartedAt всех сервисов |
| Отказ обеих настоящих API-реплик | failover-*.txt и *-process.json, target из X-Instance-Id, distinct container IDs |
| Вся внутренняя TLS матрица и отрицательные identities | tls-matrix.json: все allow edges и no-cert/foreign CA/wrong SAN/expired/revoked/plaintext |
| Scoped cert auth/TTL/policy allow-deny | bao-policy.json |
| Пароли/session hashes/rehash/shared DoS budget/ACL | security-api.json |
| Реальный доступ с credentials worker | worker-rights.json |
| Отзыв identity, ротация cert+DB password, сохранность задач | compromised-worker.json |
| Sealed Bao и bounded expiry/recovery | sealed-bao.json |
| Метрики, сквозной upload trace, correlated logs без canaries | telemetry-security.json |
| Runtime isolation/permissions/mounts/ports | hardening.json |
| Dependencies и образы | dependency-audit.json, image-audit.json, security/SCAN_TRIAGE.md |
Модель угроз связывает 11 конкретных сценариев с этими проверками:
[security/THREAT_MODEL.md](security/THREAT_MODEL.md). Публичный smoke не используется
как доказательство всех security claims.
## Исследование: компрометация worker
Гипотеза: раздельная identity + least privilege ограничивают последствия захвата
worker обработкой вложений, а отзыв cert и связанного пароля останавливает новый
доступ; transactional outbox сохраняет принятые задания до восстановления.
Методика `checks/compromised_worker.py`: синтетические пользователи/чат; сравнение
operator session-read с denied worker session-read; revoke serial в proxy CRL и
Bao cert-auth CRL; завершение старых worker DB sessions; 3 загрузки при отозванном
worker; проверка queued; ALTER ROLE PASSWORD; новый24h leaf; secret-init; проверка
старого пароля на новом TLS соединении; обновление KV; ожидание ready всех3,
проверка distinct jobs и SHA-256. Образ worker до/после должен совпадать.
Первый завершённый запуск 04.11.2026: 3/3 задачи восстановлены без дублей, SHA-256
сохранены, образ не менялся, полный эксперимент39.16s. Это время включает fault
injection/provision и ожидание20s re-auth, не pure service recovery latency.
В ходе разработки тест нашёл loopback trust в PostgreSQL; после исправления на
SCRAM старый пароль действительно отклонён. Финальные повторные результаты
смотрите в JSON evidence (они заменяют предыдущий запуск).
Отдельный sealed-Bao эксперимент04.11: обе API child прекратили слушать через
106.58s от seal (остаток последнего120s budget), recovery23.44s после unseal.
Измерение Argon2 в API container: hash+verify median0.209s, maxRSS53168KiB;
20 конкурентных login завершились за2.031s, 12×401 и8×429, две instance identities.
Основная KDF memory bound двух API: 4×32MiB=128MiB плюс overhead процесса.
Эти числа — один контрольный запуск, не гарантия p99/capacity/устойчивости к DDoS.
## Ограничения и источники
Single host и single DB/S3/broker/Bao/LB: отказ хоста или этих зависимостей не
переживается без перерыва. Есть отказ отдельной API-реплики, но не полноценная
HA всех компонентов. Cert/CRL renewal операторский; безопасный CA rollover под
нагрузкой не заявляется (выбран трек1). Учебная PKI и все shares на одном хосте.
Worker сохраняет возможность повредить разрешённые prefixes/jobs; least privilege
ограничивает ущерб, не делает скомпрометированный процесс безопасным. Audit Bao
остаётся в operator volume, rotation/retention нужны для длительной эксплуатации.
Сканирование не является доказательством отсутствия уязвимостей; применимость и
оставшиеся findings разобраны отдельно. Более строгий deployment потребует
динамических DB leases, isolated worker VM/seccomp, сетевых политик и отдельной
защиты operator workstation.
Первичные источники: [OpenBao cert auth](https://openbao.org/docs/auth/cert/),
[declarative audit](https://openbao.org/docs/configuration/audit/),
[Python ssl](https://docs.python.org/3/library/ssl.html),
[OWASP Password Storage](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html),
[Pillow13 release](https://pillow.readthedocs.io/en/stable/releasenotes/13.0.0.html),
[Trivy](https://trivy.dev/docs/latest/getting-started/installation/).
+39
View File
@@ -0,0 +1,39 @@
import json
import ssl
import urllib.request
from urllib.error import HTTPError
from tls_matrix import context
def request(role,path,body=None,token=None):
headers={'Content-Type':'application/json'}
if token:headers['X-Vault-Token']=token
req=urllib.request.Request('https://openbao:8200/v1/'+path,headers=headers,
data=json.dumps(body).encode() if body is not None else None)
try:
with urllib.request.urlopen(req,context=context(role),timeout=5) as r:
raw=r.read();return r.status,json.loads(raw) if raw else {}
except HTTPError as e:return e.code,{}
results=[]
for role in ('api1','api2','worker','dispatcher','reconciler'):
code,data=request(role,'auth/cert/login',{'name':role})
assert code==200,(role,code)
auth=data['auth'];token=auth['client_token']
assert 0<auth['lease_duration']<=120 and 'root' not in auth['policies']
try:
assert request(role,'secret/data/'+role,token=token)[0]==200
other='worker' if role!='worker' else 'api1'
assert request(role,'secret/data/'+other,token=token)[0]==403
assert request(role,'sys/mounts',token=token)[0]==403
assert request(role,'auth/cert/login',{'name':other})[0] in (400,403)
results.append({'role':role,'own_secret':200,'other_secret':403,'admin':403,'TTL':auth['lease_duration'],'wrong_role_denied':True})
finally:request(role,'auth/token/revoke-self',{},token)
for role in (None,'outsider','revoked','expired','alien'):
try:assert request(role,'auth/cert/login',{'name':'api1'})[0] in (400,403)
except (ssl.SSLError,urllib.error.URLError) as exc:
# TLS handshake rejection also denies auth; network must already work above.
if not isinstance(exc,ssl.SSLError) and not isinstance(getattr(exc,'reason',None),ssl.SSLError):raise
results.append({'identity':role,'login_denied':True})
print(json.dumps({'passed':True,'results':results},indent=2))
+89
View File
@@ -0,0 +1,89 @@
"""Grade-5 track 1: revoke a real worker, rotate DB secret/leaf, preserve tasks.
Operator-only fault injection on a local synthetic workload. No public switch.
All backups contain private material and stay under ignored secrets/operator.
"""
import hashlib
import json
from pathlib import Path
import secrets
import shutil
import subprocess
import sys
import time
ROOT=Path(__file__).resolve().parents[1]
sys.path[:0]=[str(ROOT/'scripts'),str(ROOT/'tests')]
import pki
from bao_operator import request
from database_operator import sql
from client import Client
def docker(*args,input=None):
r=subprocess.run(['docker','compose',*args],cwd=ROOT,input=input,text=True,capture_output=True)
if r.returncode:raise RuntimeError('Compose experiment step failed: '+args[0])
return r.stdout.strip()
def main():
c=Client('https://127.0.0.1:8443',str(ROOT/'secrets/trust/ca.crt'))
a,b,e,chat=c.bootstrap()
baseline={'legacy_admin_can_read_sessions':int(sql('SELECT count(*) FROM sessions'))>0,
'scoped_worker_denied':['sessions','password_hashes','messages','DDL','role_admin','foreign_S3_prefix','bucket_admin','queue_publish','queue_admin']}
image_before=docker('images','-q','worker')
root=json.loads((ROOT/'secrets/operator/bao-init.json').read_text())['root_token']
identity=ROOT/'secrets/identities/worker'
backup=ROOT/'secrets/operator/worker-before-rotation';backup.mkdir(exist_ok=True)
for f in identity.iterdir():shutil.copy2(f,backup/f.name)
old_serial=pki.x509.load_pem_x509_certificate((identity/'cert.pem').read_bytes()).serial_number
start=time.monotonic()
pki.revoke('worker')
request('auth/cert/crls/operator',{'crl':(ROOT/'secrets/trust/revocations.pem').read_text()},root)
deny="import sys;sys.path.insert(0,'/checks');from tls_matrix import rejected;assert rejected('db',8443,'worker');print('old worker TLS denied')"
assert 'denied' in docker('run','--rm','--no-deps','security-check','python','-c',deny)
# Terminate old already-authenticated DB sessions as part of containment.
sql("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename='worker'")
queued=[c.attachment(a,chat) for _ in range(3)]
time.sleep(3)
ids=','.join("'"+att['id']+"'" for att,_ in queued)
assert sql(f"SELECT count(*) FROM attachments WHERE id IN ({ids}) AND status='queued'")=='3'
# Cert auth must reject the old identity even though signed by our CA.
rejection=docker('exec','-T','worker','python','-',input="from src.secret_agent import fetch\nfrom urllib.error import HTTPError\ntry: fetch()\nexcept HTTPError as e: assert e.code in (400,403); print('denied')\nelse: raise AssertionError('revoked worker got secrets')\n")
assert rejection=='denied'
cfg_path=ROOT/'secrets/operator/config.json';cfg=json.loads(cfg_path.read_text())
old_url=cfg['service_credentials']['worker']['DATABASE_URL']
password=secrets.token_hex(24)
sql(f"ALTER ROLE worker PASSWORD '{password}'")
cfg['database_passwords']['worker']=password
cfg['service_credentials']['worker']['DATABASE_URL']=f'postgresql://worker:{password}@127.0.0.1:5432/messenger'
cfg_path.write_text(json.dumps(cfg,indent=2));cfg_path.chmod(0o600)
cert=pki.issue('worker')
docker('run','--rm','--no-deps','secret-init')
# New TLS identity + old database secret is denied by PostgreSQL itself.
code="""import json,sys,psycopg
url=json.load(sys.stdin)['url']
try: psycopg.connect(url,connect_timeout=3)
except psycopg.OperationalError as e:
assert 'password authentication failed' in str(e);print('old password denied')
else:raise AssertionError('old database secret still valid')
"""
check=docker('exec','-T','worker','python','-c',code,input=json.dumps({'url':old_url}))
assert check=='old password denied'
request('secret/data/worker',{'data':cfg['service_credentials']['worker']},root)
for att,content in queued:
c.wait_ready(a,att,wait_seconds=90)
downloaded=c.call('GET','/api/v1/attachments/'+att['id']+'/content',actor=a)['bytes']
assert hashlib.sha256(downloaded).digest()==hashlib.sha256(content).digest()
assert sql(f"SELECT count(*) FROM jobs WHERE attachment_id IN ({ids}) AND status='done'")=='3'
assert sql(f"SELECT count(*) FROM jobs WHERE attachment_id IN ({ids})")=='3'
assert image_before==docker('images','-q','worker')
result={'passed':True,'baseline':baseline,'revoked_serial':str(old_serial),'new_serial':str(cert.serial_number),
'revoked_certificate_denied':True,'revoked_Bao_auth_denied':True,'rotated_old_DB_password_denied':True,
'queued_during_revocation':3,'completed_after_rotation':3,'distinct_jobs':3,'content_SHA256_preserved':True,
'image_unchanged':True,'total_seconds':round(time.monotonic()-start,2)}
(ROOT/'evidence/compromised-worker.json').write_text(json.dumps(result,indent=2)+'\n')
print('PASS: revoked worker -> queue preserved -> new cert/DB secret -> 3/3 jobs and hashes intact')
if __name__=='__main__':main()
+36
View File
@@ -0,0 +1,36 @@
import json
from pathlib import Path
import ssl
import subprocess
import urllib.request
ROOT=Path(__file__).resolve().parents[1]
def docker(*args):return subprocess.check_output(['docker',*args],cwd=ROOT,text=True,encoding='utf-8').strip()
rows=[]
ids=docker('compose','ps','-q').splitlines()
for cid in ids:
data=json.loads(docker('inspect',cid))[0]
role=data['Config']['Labels']['com.docker.compose.service']
if role in ('telemetry-init','secret-init'):continue
h=data['HostConfig'];mounts=data['Mounts'];user=data['Config']['User']
assert not h['Privileged'] and h['NetworkMode']!='host'
assert h['ReadonlyRootfs'] and 'ALL' in h['CapDrop'] and 'no-new-privileges:true' in h['SecurityOpt']
assert h['Memory']>0 and h['NanoCpus']>0 and h['PidsLimit']>0
assert not any('docker.sock' in m['Source'] or '/operator' in m['Source'].replace('\\','/') for m in mounts)
published=[]
for port,bindings in (h.get('PortBindings') or {}).items():
for bind in bindings or []:
assert bind['HostIp']=='127.0.0.1'
assert (role,bind['HostPort']) in {('lb','8443'),('openbao','8200'),('grafana','3000')}
published.append(bind['HostPort'])
if role in ('api1','api2','worker','dispatcher','reconciler'):
assert user=='10001:10001'
assert all(not e.startswith(('DATABASE_URL=','S3_SECRET_KEY=','BAO_TOKEN=','POSTGRES_PASSWORD=')) for e in data['Config']['Env'])
assert all(m['Destination'] in ('/identity','/trust','/tmp') for m in mounts)
rows.append({'service':role,'user':user,'read_only':True,'cap_drop':['ALL'],'memory_bytes':h['Memory'],'NanoCpus':h['NanoCpus'],'pids_limit':h['PidsLimit'],'published_loopback_ports':published,'operator_material_absent':True})
ctx=ssl.create_default_context(cafile=str(ROOT/'secrets/trust/ca.crt'))
with urllib.request.urlopen('https://127.0.0.1:3000/login',context=ctx,timeout=10) as r:assert r.status==200
raw=docker('compose','run','--rm','--no-deps','security-check','python','/checks/plaintext_isolation.py')
isolation=json.loads(raw);assert isolation['passed']
(ROOT/'evidence/hardening.json').write_text(json.dumps({'passed':True,'runtime':rows,'grafana_verified_HTTPS':True,'plaintext_isolation':isolation},indent=2)+'\n')
print('PASS: runtime privileges/resource limits/mounts/public ports and loopback-only plaintext')
+15
View File
@@ -0,0 +1,15 @@
import json
import socket
ports={'api1':[8080],'api2':[8080],'db':[5432],'s3':[8333,8888,9333,19333],
'cache':[6379],'broker':[6379],'otel-collector':[4318,9464,8888,13133],
'prometheus':[9090],'loki':[3100,9095],'tempo':[3200,4318,9095]}
results=[]
for host,values in ports.items():
for port in values:
with socket.socket() as s:
s.settimeout(2)
try:s.connect((host,port))
except ConnectionRefusedError:results.append({'host':host,'port':port,'refused':True})
else:raise AssertionError(f'Plaintext port reachable: {host}:{port}')
print(json.dumps({'passed':True,'probes':results},indent=2))
+46
View File
@@ -0,0 +1,46 @@
"""Finite supervisor lease under real sealed OpenBao, then explicit recovery."""
import json
from pathlib import Path
import ssl
import subprocess
import sys
import time
import urllib.request
from urllib.error import HTTPError,URLError
ROOT=Path(__file__).resolve().parents[1]
sys.path.insert(0,str(ROOT/'scripts'))
from bao_operator import request
ctx=ssl.create_default_context(cafile=str(ROOT/'secrets/trust/ca.crt'))
def status():
try:
with urllib.request.urlopen('https://127.0.0.1:8443/health/ready',context=ctx,timeout=3) as r:return r.status
except HTTPError as e:return e.code
assert status()==200
root=json.loads((ROOT/'secrets/operator/bao-init.json').read_text())['root_token']
start=time.monotonic();observations=[]
try:
request('sys/seal',{},root)
while time.monotonic()-start<150:
current=status();observations.append({'seconds':round(time.monotonic()-start,1),'HTTP':current})
if current==503:
# Both API children must actually have stopped, not just one LB check.
code="import socket; s=socket.socket();s.settimeout(1);assert s.connect_ex(('127.0.0.1',8080))!=0"
for role in ('api1','api2'):
subprocess.run(['docker','compose','exec','-T',role,'python','-c',code],cwd=ROOT,check=True,capture_output=True)
break
time.sleep(2)
else:raise AssertionError('Application exceeded finite secret availability budget')
stopped=time.monotonic()-start
finally:
subprocess.run([sys.executable,str(ROOT/'scripts/bao_operator.py'),'unseal'],cwd=ROOT,check=True)
recovery=time.monotonic()
while time.monotonic()-recovery<60:
if status()==200:break
time.sleep(1)
else:raise AssertionError('Unsealed system failed to recover')
result={'passed':True,'secret_refresh_seconds':20,'token_max_TTL_seconds':120,'both_API_children_stopped_seconds':round(stopped,2),'recovery_seconds':round(time.monotonic()-recovery,2),'observations':observations,'scope':'Supervisor bounds use during Bao outage; static backend passwords themselves require explicit rotation.'}
(ROOT/'evidence/sealed-bao.json').write_text(json.dumps(result,indent=2)+'\n')
print('PASS: sealed Bao -> finite lease -> both API children stopped -> unseal -> recovery')
+68
View File
@@ -0,0 +1,68 @@
"""HTTPS ACL/session/KDF/replica-shared login limits. Synthetic local test data."""
import base64
from concurrent.futures import ThreadPoolExecutor
import hashlib
import json
from pathlib import Path
import statistics
import subprocess
import sys
import time
ROOT=Path(__file__).resolve().parents[1]
sys.path[:0]=[str(ROOT/'tests'),str(ROOT/'scripts')]
from client import Client
from database_operator import sql
from argon2 import PasswordHasher
def client():return Client('https://127.0.0.1:8443',str(ROOT/'secrets/trust/ca.crt'))
def main():
c=client();a,b,e,chat=c.bootstrap();att,content=c.attachment(a,chat);c.wait_ready(a,att)
for suffix in ('','/content','/thumbnail'):
c.call('GET','/api/v1/attachments/'+att['id']+suffix,404,actor=e)
c.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=x.png',404,actor=e,raw=content,headers={'Content-Type':'image/png'})
c.call('GET','/api/v1/users/me',401,headers={'Authorization':'Bearer '+'x'*43})
# Upgrade a valid historical hash only after successful verification.
old=PasswordHasher(time_cost=2,memory_cost=19456,parallelism=1).hash(a['registration']['password'])
sql(f"UPDATE users SET password_hash='{old}' WHERE id='{a['id']}'")
credential=base64.b64encode((a['registration']['username']+':'+a['registration']['password']).encode()).decode()
session=c.call('POST','/api/v1/auth/sessions',201,headers={'Authorization':'Basic '+credential},schema='Session')['json']
stored=sql(f"SELECT password_hash FROM users WHERE id='{a['id']}'")
assert stored!=old and '$argon2id$' in stored and 'm=32768,t=2,p=1' in stored
digest=hashlib.sha256(session['token'].encode()).hexdigest()
assert sql(f"SELECT count(*) FROM sessions WHERE token_hash='{digest}'")=='1'
sql(f"UPDATE sessions SET expires_at=clock_timestamp()-interval '1 second' WHERE token_hash='{digest}'")
c.call('GET','/api/v1/users/me',401,actor={'token':session['token']})
c.call('DELETE','/api/v1/auth/sessions/current',204,actor=b)
c.call('GET','/api/v1/users/me',401,actor=b)
# Reset only the synthetic experiment's two limiter counters; limiter stays on.
sql('DELETE FROM login_budget')
wrong=base64.b64encode((a['registration']['username']+':Wrong-password-123').encode()).decode()
def login(_):
r=client().call('POST','/api/v1/auth/sessions',(401,429),headers={'Authorization':'Basic '+wrong})
if r['status']==429:assert int(r['headers']['Retry-After'])>0
assert a['registration']['password'].encode() not in r['bytes'] and b'argon2' not in r['bytes']
return {'status':r['status'],'seconds':round(r['elapsed'],4),'instance':r['headers']['X-Instance-Id']}
begin=time.monotonic()
with ThreadPoolExecutor(max_workers=20) as pool:attempts=list(pool.map(login,range(20)))
elapsed=time.monotonic()-begin
assert elapsed<10 and sum(r['status']==401 for r in attempts)==12 and sum(r['status']==429 for r in attempts)==8
assert len({r['instance'] for r in attempts})==2
# Measure actual Argon2 on the resource-limited API container, not the host.
bench="""import json,time,statistics,resource
from argon2 import PasswordHasher
p=PasswordHasher(time_cost=2,memory_cost=32768,parallelism=1);times=[]
for i in range(12):
t=time.perf_counter();h=p.hash('synthetic-benchmark-password');p.verify(h,'synthetic-benchmark-password');times.append(time.perf_counter()-t)
print(json.dumps({'hash_plus_verify_seconds':times,'median':statistics.median(times),'max_rss_kib':resource.getrusage(resource.RUSAGE_SELF).ru_maxrss}))
"""
kdf=json.loads(subprocess.check_output(['docker','compose','exec','-T','api1','python','-c',bench],cwd=ROOT,text=True))
result={'passed':True,'ACL_attachment_denials':4,'expired_session_denied':True,'logout_denied':True,'token_hash_sha256':True,'rehash':'19456 -> 32768 KiB','login_burst':attempts,'burst_seconds':round(elapsed,3),'KDF':kdf,'KDF_slots_per_API':2,'KDF_memory_bound_two_APIs_MiB':128}
(ROOT/'evidence/security-api.json').write_text(json.dumps(result,indent=2)+'\n')
print('PASS: ACL, expiry/logout, hash storage/rehash, shared 12/10s rate limit, measured Argon2')
if __name__=='__main__':main()
+31
View File
@@ -0,0 +1,31 @@
"""Read telemetry using the operator's mTLS identity; never emit secret canaries."""
import json
import sys
import time
import urllib.request
import urllib.parse
from urllib.error import HTTPError
from tls_matrix import context
args=json.load(sys.stdin);trace_id=args['trace_id']
def get(host,port,path):
with urllib.request.urlopen('https://'+host+':'+str(port)+path,context=context('operator'),timeout=8) as r:return json.load(r)
deadline=time.monotonic()+90
while True:
try:
trace=get('tempo',8444,'/api/traces/'+trace_id)
spans=[s for b in trace.get('batches',trace.get('resourceSpans',[])) for scope in b.get('scopeSpans',b.get('instrumentationLibrarySpans',[])) for s in scope.get('spans',[])]
names={s['name'] for s in spans}
query=urllib.parse.urlencode({'query':'{service_name=~"messenger-(api|worker|dispatcher)"} |= "'+trace_id+'"','limit':1000,'start':int((time.time()-600)*1e9),'end':int(time.time()*1e9)})
logs=get('loki',8443,'/loki/api/v1/query_range?'+query)
lines=[line for stream in logs['data']['result'] for _,line in stream['values']]
if {'attachment.consume','db.attachment.commit','attachment.publish','image.render'}<=names and any('job_finished' in line for line in lines):break
except HTTPError as e:
if e.code!=404:raise
if time.monotonic()>deadline:raise AssertionError('Telemetry did not arrive over TLS')
time.sleep(2)
raw=json.dumps([trace,logs])
assert all(secret not in raw for secret in args['canaries'])
metrics=get('prometheus',8443,'/api/v1/query?'+urllib.parse.urlencode({'query':'http_server_requests_total'}))
assert metrics['status']=='success' and metrics['data']['result']
print(json.dumps({'passed':True,'trace_id':trace_id,'span_count':len(spans),'span_names':sorted(names),'correlated_log_count':len(lines),'http_metric_series':len(metrics['data']['result']),'credential_canaries_absent':True,'all_queries':'mTLS verified operator identity'},indent=2))
+30
View File
@@ -0,0 +1,30 @@
import json
from pathlib import Path
import subprocess
import sys
import uuid
ROOT=Path(__file__).resolve().parents[1]
sys.path.insert(0,str(ROOT/'tests'))
from client import Client
c=Client('https://127.0.0.1:8443',str(ROOT/'secrets/trust/ca.crt'))
a=c.user('telemetry')
chat=c.call('POST','/api/v1/chats',201,actor=a,body={'title':'TLS telemetry','member_ids':[a['id']]},schema='Chat')['json']
trace=uuid.uuid4().hex
att=c.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=sample.png',202,actor=a,
raw=(ROOT/'tests/fixtures/sample.png').read_bytes(),headers={'Content-Type':'image/png','traceparent':'00-'+trace+'-'+uuid.uuid4().hex[:16]+'-01'},schema='Attachment')['json']
c.wait_ready(a,att)
cfg=json.loads((ROOT/'secrets/operator/config.json').read_text())
canaries=[a['token'],a['registration']['password'],*cfg['database_passwords'].values()]
for data in cfg['service_credentials'].values():
canaries.extend([data[k] for k in ('CURSOR_SECRET','S3_SECRET_KEY') if k in data])
result=subprocess.run(['docker','compose','run','--rm','-T','--no-deps','security-check','python','/checks/telemetry_probe.py'],cwd=ROOT,
input=json.dumps({'trace_id':trace,'canaries':canaries}),text=True,capture_output=True)
if result.returncode:raise RuntimeError('Telemetry check failed: '+result.stderr[-1500:])
summary=json.loads(result.stdout)
logs=subprocess.check_output(['docker','compose','logs','--no-color'],cwd=ROOT,text=True,encoding='utf-8',errors='replace')
assert all(secret not in logs for secret in canaries)
summary['container_logs_canaries_absent']=True
(ROOT/'evidence/telemetry-security.json').write_text(json.dumps(summary,indent=2)+'\n')
print('PASS: TLS metrics/traces/log correlation, credential canaries absent in telemetry/container logs')
+49
View File
@@ -0,0 +1,49 @@
import json
from pathlib import Path
import tempfile
import unittest
from PIL import Image
from src.render import render
class RenderTests(unittest.TestCase):
def generate(self, image, fmt='PNG', **save_args):
folder=tempfile.TemporaryDirectory(); self.addCleanup(folder.cleanup)
root=Path(folder.name); source=root/'input'
image.save(source,format=fmt,**save_args)
records=render(source,root,1)
return root,{row['name']:row for row in records}
def test_contract_dimensions(self):
root,rows=self.generate(Image.new('RGB',(320,200),'red'))
self.assertEqual((rows['thumbnail']['width'],rows['thumbnail']['height']),(256,160))
self.assertEqual((rows['crop']['width'],rows['crop']['height']),(128,128))
with Image.open(root/'optimized') as optimized:
self.assertEqual(optimized.format,'WEBP')
self.assertEqual(optimized.size,(320,200))
def test_alpha_white_and_small_image_not_enlarged(self):
root,rows=self.generate(Image.new('RGBA',(17,9),(255,0,0,0)))
with Image.open(root/'thumbnail') as thumb:
self.assertEqual(thumb.size,(17,9))
self.assertTrue(all(c>=250 for c in thumb.getpixel((0,0))))
self.assertEqual(rows['crop']['width'],128)
def test_exif_orientation_and_metadata_removed(self):
exif=Image.Exif(); exif[274]=6; exif[270]='synthetic metadata must disappear'
root,rows=self.generate(Image.new('RGB',(40,20),'blue'),'JPEG',exif=exif)
with Image.open(root/'thumbnail') as thumb:
self.assertEqual(thumb.size,(20,40))
self.assertFalse(thumb.getexif())
def test_centered_crop(self):
image=Image.new('RGB',(300,100),'red')
image.paste(Image.new('RGB',(100,100),'green'),(100,0))
root,_=self.generate(image)
with Image.open(root/'crop') as crop:
r,g,b=crop.getpixel((64,64))
self.assertGreater(g,r+60); self.assertGreater(g,b+60)
if __name__=='__main__': unittest.main()
+65
View File
@@ -0,0 +1,65 @@
"""Inside the isolated test network. Every permitted edge and rejected identity.
A timeout/refused connection is never counted as a valid negative TLS result.
"""
import json
from pathlib import Path
import socket
import ssl
import struct
import time
IDENTITIES=Path('/audit-identities')
TRUST=Path('/trust')
def context(role=None):
ctx=ssl.create_default_context(cafile=str(TRUST/'bundle.pem'))
ctx.verify_flags |= ssl.VERIFY_CRL_CHECK_LEAF
if role:ctx.load_cert_chain(IDENTITIES/role/'cert.pem',IDENTITIES/role/'key.pem')
return ctx
def probe(host,port,role,hostname=None):
ctx=context(role)
with socket.create_connection((host,port),timeout=5) as raw:
with ctx.wrap_socket(raw,server_hostname=hostname or host) as conn:
conn.settimeout(5)
if host=='db':payload=struct.pack('!II',8,80877103)
elif host in ('cache','broker'):payload=b'*1\r\n$4\r\nPING\r\n'
else:payload=b'GET /health/live HTTP/1.1\r\nHost: '+host.encode()+b'\r\nConnection: close\r\n\r\n'
conn.sendall(payload)
return conn.recv(4096)
def rejected(host,port,role):
try:return not probe(host,port,role)
except (ssl.SSLError,ConnectionResetError,BrokenPipeError):return True
def main():
start=time.monotonic();results=[]
for file in sorted(Path('/mesh').glob('*.json')):
host=file.stem
for route in json.loads(file.read_text()):
if route['mode']!='server':continue
port=route['listen']
for role in route['allow']:
assert probe(host,port,role),f'Permitted edge failed: {role}->{host}:{port}'
results.append({'source':role,'destination':host,'port':port,'allowed':True})
for role in (None,'outsider','alien','expired','revoked'):
assert rejected(host,port,role),f'Forbidden edge accepted: {role}->{host}:{port}'
# Same-CA hostname mismatch must fail client validation, not timeout.
try:probe(host,port,route['allow'][0],hostname='outsider')
except ssl.SSLCertVerificationError:pass
else:raise AssertionError('Wrong server identity accepted')
# Plaintext has no protocol response from a TLS-only ingress.
with socket.create_connection((host,port),timeout=5) as raw:
raw.settimeout(5);raw.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
try:assert raw.recv(100)==b''
except ConnectionResetError:pass
results.append({'destination':host,'port':port,'denied':['no_cert','wrong_CA','wrong_SAN','expired','revoked','wrong_server_SAN','plaintext']})
print(json.dumps({'passed':True,'seconds':round(time.monotonic()-start,2),'edges':results},indent=2))
if __name__=='__main__':main()
+37
View File
@@ -0,0 +1,37 @@
"""Execute inside the real worker container with its cert-authenticated credentials."""
import io
import json
import os
from uuid import uuid4
import psycopg
import redis
from botocore.exceptions import ClientError
from src.secret_agent import fetch
from src.jobs import database
from src.storage import Storage
from src.broker import connect,ensure_group
data,_=fetch();os.environ.update(data)
results={}
with database() as db:
assert db.execute('SELECT current_user AS role').fetchone()['role']=='worker'
db.execute('SELECT id FROM jobs LIMIT 1');results['jobs_read']='allowed'
for name,query in [('sessions','SELECT * FROM sessions'),('password_hashes','SELECT password_hash FROM users'),('messages','SELECT * FROM messages'),('schema','CREATE TABLE public.worker_escape(id int)'),('privilege','CREATE ROLE worker_escape SUPERUSER')]:
try:db.execute(query)
except psycopg.errors.InsufficientPrivilege:results[name]='denied'
else:raise AssertionError('Unexpected worker permission: '+name)
storage=Storage();key='variants/security-test-'+uuid4().hex+'/probe'
storage.client.put_object(Bucket=storage.bucket,Key=key,Body=b'test')
assert storage.client.get_object(Bucket=storage.bucket,Key=key)['Body'].read()==b'test'
storage.delete(key);results['variants_read_write_delete']='allowed'
for name,operation in [('foreign_prefix',lambda:storage.client.put_object(Bucket=storage.bucket,Key='private/forbidden',Body=b'test')),('bucket_admin',lambda:storage.client.create_bucket(Bucket='forbidden-'+uuid4().hex))]:
try:operation()
except ClientError as e:
assert e.response['ResponseMetadata']['HTTPStatusCode']==403;results[name]='denied'
else:raise AssertionError('Unexpected S3 permission: '+name)
queue=connect();ensure_group(queue);assert queue.ping();results['queue_group']='allowed'
for name,op in [('queue_publish',lambda:queue.xadd('attachment-jobs',{'forbidden':'test'})),('foreign_queue',lambda:queue.xreadgroup('image-workers','security-test',{'other-queue':'>'},count=1)),('queue_admin',lambda:queue.config_get('*'))]:
try:op()
except redis.exceptions.NoPermissionError:results[name]='denied'
else:raise AssertionError('Unexpected queue permission: '+name)
print(json.dumps({'passed':True,'rights':results},indent=2))
+1061
View File
File diff suppressed because it is too large Load Diff
+41 -31
View File
@@ -1,37 +1,47 @@
# Матрица доверия — заполнить в решении lab7
# Матрица доверия реализации ЛР7
Это обязательная матрица покрытия. Для каждой строки укажите реальный порт, endpoint, CA, service identity, проверку SAN/ACL, срок жизни credentials и команду положительного/отрицательного теста. TLS шифрует и подтверждает сервер; mTLS также проверяет клиента. При доверенном CA всё равно нужно ограничивать разрешённую identity. Plaintext hop допустим только внутри закрытого локального transport/proxy boundary, без опубликованного обхода; границу и rationale нужно доказать.
Обозначения: mTLS = TLS >=1.2 с CA, hostname/SAN сервера, обязательным клиентским
cert и точным SAN allowlist. CA общий, identity/key отдельны для каждой роли.
Полные allowlist/порты в `infra/mesh/*.json`; ниже перечислены все сетевые hops.
Внешние порты привязаны только к 127.0.0.1 хоста, внутренние не публикуются.
| Связь | Минимум на 3 | Дополнительно на 4 | Доказательство / конфиг |
| Связь | Transport / проверяемая identity | Прикладные права | Доказательство |
| --- | --- | --- | --- |
| Клиент → ingress/LB | TLS, CA + hostname, Bearer | Проверка expiry/revocation сессии и общего login rate limit | TODO |
| LB → API (все реплики) | mTLS + allowlist LB identity | Ротация leaf без обхода проверки | TODO |
| API/worker → PostgreSQL | TLS verify-full + раздельные ограниченные DB roles | Ротация credentials; mTLS — допустимое усиление | TODO |
| API/worker → S3 | TLS + разные scoped credentials/prefix rights | Проверка denied prefix и ротация | TODO |
| API/worker → broker | TLS + разные producer/consumer identities и ACL | mTLS + ограничение допустимых SAN/ролей | TODO |
| API/worker → Collector | TLS; сеть/аутентификация запрещает произвольный ingest | mTLS с раздельными identities | TODO |
| Collector → telemetry backends | TLS + scoped backend credentials или mTLS | Ротация и проверка least privilege | TODO |
| Grafana → data sources | TLS с проверкой сервера, ограниченная identity | Отрицательные проверки доступа | TODO |
| API/worker/agent → OpenBao | TLS + сервисный auth + scoped policy | Renewal/expiry/revocation | TODO |
| Оператор → OpenBao/Grafana/S3 admin | TLS + authentication, доступ локальный/через защищённый канал | Разделение operator/application roles | TODO |
| API → Redis/Valkey, если используется | TLS + ACL/пароль с минимальными правами | Ротация; mTLS по возможности | TODO |
| Replication/кластерные каналы, если добавлены | TLS/mTLS либо подтверждённый шифрованный transport | Проверка заявленной модели отказа | TODO |
| Клиент → LB:8443 | Native TLS, SAN localhost/127.0.0.1, клиент доверяет локальному CA | Basic только login; Bearer для business API | tests/smoke.py, tests/security.py |
| LB → api1/api2:8443 | mTLS, server SAN api1/api2, client SAN только lb | API:8080 только loopback | tls_matrix, no-cert/foreign-CA/wrong-SAN/expiry/revoke/plaintext |
| API/worker/dispatcher/reconciler → db:8443 | mTLS, SAN db, отдельные client SAN; operator для диагностики | SCRAM; отдельные PostgreSQL roles, не superuser; `database_operator.py` GRANT | tls_matrix, worker_rights, compromised_worker |
| API/worker/reconciler → s3:8443 | mTLS, SAN s3, соответствующие client SAN; dispatcher запрещён | SigV4 credentials отдельно. API: Read originals/variants, Write staging. Worker: Read/Write staging/originals/variants. Reconciler дополнительно List bucket | tls_matrix, worker_rights, public upload |
| worker/dispatcher → broker:8443 | mTLS, SAN broker, client worker/dispatcher; operator diagnostic | Valkey ACL: worker consume/ack/claim; dispatcher publish; только attachment-jobs | tls_matrix, worker_rights |
| api1/api2 → cache:8443 | mTLS, SAN cache, client api1/api2 | Именованные ACL GET/SET user:*; TTL 60s; truth в PostgreSQL | tls_matrix + smoke |
| API/worker/dispatcher/reconciler → otel-collector:8443 | mTLS, SAN otel-collector, allowlist этих identities | OTLP HTTP; без body/Authorization в telemetry. Cleanup пока не испускает сигналы, роль зарезервирована | tls_matrix, telemetry_security |
| Collector → tempo:8443 / loki:8443 | mTLS, server SAN tempo/loki; client otel-collector | OTLP; внутренние native backends без отдельного auth, authorizer TLS sidecar | tls_matrix, telemetry_security |
| Prometheus → collector:8444/8445 | mTLS, client prometheus; server otel-collector | Scrape application/exporter metrics, оба native listeners loopback | tls_matrix + telemetry_security |
| Grafana → prometheus:8443, loki:8443, tempo:8444 | mTLS, client grafana; отдельный server SAN | Queries через loopback outgoing tunnel; нет публичного доступа к backend | tls_matrix + telemetry_security |
| Service supervisor → OpenBao:8200 | Native mTLS; server SAN openbao; cert auth role требует SAN service-name | read только secret/data/role, token TTL/maxTTL 120s, re-auth 20s, revoke-self после чтения | bao_policy, sealed_bao |
| Оператор → OpenBao:127.0.0.1:8200 | Native mTLS operator + root token только для provision/seal | Init/unseal вручную с защищёнными локальными shares | bao_operator, restart-processes |
| Оператор → Grafana:127.0.0.1:3000 | Native HTTPS, SAN localhost/127.0.0.1 + уникальный admin password | Только локальный административный интерфейс | hardening audit / HTTPS login page |
| storage-init → S3 admin | Только loopback в namespace S3, одноразовый operator profile, собственного внешнего hop нет | Admin credential из отдельного локального файла, runtime его не получает | up.py private bucket bootstrap |
| database_operator → PostgreSQL | docker exec, Unix socket внутри DB, не сетевой hop | operator messenger role; socket не расшарен | миграции + сравнение privileged/scoped в track1 |
| secret-init → runtime secret volumes | Локальные mounts, одноразовый operator-owned процесс | Только service cert/config, chmod/chown; CA/unseal/root отсутствуют | hardening audit |
| Healthchecks | API/DB native loopback/Unix; LB → API по mTLS | /health/ready без секретов; public endpoint через LB HTTPS | smoke, verify_public |
| Репликация/кластер | N/A: single PostgreSQL, S3, broker, Bao raft node | Нет межузловых replication hops; HA dependencies не заявляется | compose.yaml |
Нет выбранного компонента — отметьте N/A с причиной, а не оставляйте пустую строку. Внутренний публичный health listener, если нужен, минимален и изолирован; через него нельзя обращаться к бизнес-API без mTLS. Отдельно задокументируйте scrape/health/bootstrap каналы: они тоже не должны создавать обход.
Каждый TLS proxy имеет только свой identity volume и публичный trust bundle.
Native plaintext слушает только loopback общей namespace с proxy; iptables и
«внутренняя сеть» не заменяют проверку SAN. Сетевые payloads в proxy не логируются.
Контекст TLS кэшируется максимум 1s; соединение максимум 60s, TLS tickets на
server proxy отключены. Отзыв нового подключения применяется <=1s, действующей
связи <=61s; эксперимент также завершает старые worker DB sessions. Native
HAProxy/OpenBao/Grafana перечитывают cert/config при restart/reload. CRL cert-auth
OpenBao отдельно публикуется `bao_operator.py publish-crl`.
## Обязательные отрицательные проверки
Отрицательные тесты не считают timeout/connection-refused правильным TLS отказом:
сначала положительно проверяется каждая разрешённая связь. В `bao_policy` допускается
именно SSL alert или HTTP deny, а не произвольная сетевая ошибка. Для API ACL,
OpenBao policy и прав backend есть отдельные прикладные тесты.
1. HTTPS с правильным CA/hostname проходит; с пустым/чужим CA и неправильным hostname — ошибка верификации, не «сервер недоступен».
2. LB→API: без client cert, с cert чужого CA, с запрещённой identity от доверенного CA — отказ; с разрешённой LB identity — успех.
3. На 4 повторите набор для broker и Collector; ограничение на network policy не заменяет проверку identity.
4. Пользователь вне чата не читает metadata/content и не прикрепляет чужой файл; поддельный X-User-Id не меняет Bearer identity.
5. OpenBao app policy читает только свой secret/path; запись policy и чтение чужого path запрещены. Worker не получает admin credentials API/БД/S3.
6. На 4 отзовите secret/cert и дождитесь контролируемого эффекта с учётом действующих соединений, TLS resumption и cache/TTL. Покажите новое подключение, иначе тест отзыва может ничего не проверить.
## Bootstrap и эксплуатация
В репозитории только открытые конфиги, policy templates и scripts. CA private key, leaf keys, root token, unseal shares и SecretID не коммитятся, не запекаются в образ и не печатаются в отчёте. Для локального стенда оператор создаёт их вне Git и передаёт через files/secrets с ограниченными правами. Опишите, кто имеет доступ к CA/unseal, что нужно после рестарта OpenBao и как восстановить работу.
OpenBao не использует dev-mode. Выберите persistent storage и auth метод (например AppRole с короткоживущим SecretID/response wrapping, cert auth или иная обоснованная identity). Не подменяйте root token постоянной переменной BAO_TOKEN в приложении. Рабочие tokens имеют policy, TTL и renewal/re-auth сценарий. Статические KV secrets на 3 допустимы при реальном чтении через сервисную identity; динамические DB credentials — хорошее расширение, но не обязательное название технологии.
Разрешённый компромисс учебного стенда: ручной operator init/unseal, если он безопасно документирован и не нужен при каждом обычном рестарте API. Автоматическое хранение всех unseal shares рядом с OpenBao без защиты не считается решённым bootstrap.
Root token/unseal shares/CA private key — только защищённые `secrets/operator`
на хосте, вне Git/image/container mounts. Static KV passwords не получают
автоматический backend TTL: supervisor прекращает их использование после
истечения локального бюджета, а украденные passwords отзываются ротацией.
Роли SQL/Valkey/S3 и доступы не объединяются общим service credential.
+65
View File
@@ -0,0 +1,65 @@
{
"passed": true,
"results": [
{
"role": "api1",
"own_secret": 200,
"other_secret": 403,
"admin": 403,
"TTL": 120,
"wrong_role_denied": true
},
{
"role": "api2",
"own_secret": 200,
"other_secret": 403,
"admin": 403,
"TTL": 120,
"wrong_role_denied": true
},
{
"role": "worker",
"own_secret": 200,
"other_secret": 403,
"admin": 403,
"TTL": 120,
"wrong_role_denied": true
},
{
"role": "dispatcher",
"own_secret": 200,
"other_secret": 403,
"admin": 403,
"TTL": 120,
"wrong_role_denied": true
},
{
"role": "reconciler",
"own_secret": 200,
"other_secret": 403,
"admin": 403,
"TTL": 120,
"wrong_role_denied": true
},
{
"identity": null,
"login_denied": true
},
{
"identity": "outsider",
"login_denied": true
},
{
"identity": "revoked",
"login_denied": true
},
{
"identity": "expired",
"login_denied": true
},
{
"identity": "alien",
"login_denied": true
}
]
}
+1
View File
@@ -0,0 +1 @@
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
+14
View File
@@ -0,0 +1,14 @@
{
"passed": true,
"completed_at_UTC": "2026-11-13T10:00:37.671336+00:00",
"source_commit": "0bf1ff93c71a6a8cf16edeceefc3b3fab8d23261",
"isolated_clone": true,
"own_venv": true,
"fresh_PKI_and_secrets": true,
"fresh_compose_volumes": true,
"public_suite": "PASS; 3 real SIGKILL scenarios cover 2 distinct API containers",
"security_suite": "PASS; all 11 stages",
"app_image_tested": "sha256:5d549e32c6a790e783550782db9b5c5993861fa5e7dac46039aaa9adb03e2eed",
"host": "Windows 10, Docker Desktop 4.39 / Engine 28 / Compose 2.33, WSL2 4 CPU / 3 GiB RAM / 4 GiB swap",
"scan_scope": "image-audit.json is the 12 November scan, preceding the TLS scheduling code fix; dependencies unchanged. It is not a scan of the exact app image above."
}
+1
View File
@@ -0,0 +1 @@
PASS: revoked worker -> queue preserved -> new cert/DB secret -> 3/3 jobs and hashes intact
+28
View File
@@ -0,0 +1,28 @@
{
"passed": true,
"baseline": {
"legacy_admin_can_read_sessions": true,
"scoped_worker_denied": [
"sessions",
"password_hashes",
"messages",
"DDL",
"role_admin",
"foreign_S3_prefix",
"bucket_admin",
"queue_publish",
"queue_admin"
]
},
"revoked_serial": "296803757360939619598658580576779705066316270051",
"new_serial": "582892826110773913646976171481035976112843893487",
"revoked_certificate_denied": true,
"revoked_Bao_auth_denied": true,
"rotated_old_DB_password_denied": true,
"queued_during_revocation": 3,
"completed_after_rotation": 3,
"distinct_jobs": 3,
"content_SHA256_preserved": true,
"image_unchanged": true,
"total_seconds": 35.61
}
+1
View File
@@ -0,0 +1 @@
{"dependencies": [{"name": "annotated-doc", "version": "0.0.5", "vulns": []}, {"name": "annotated-types", "version": "0.8.0", "vulns": []}, {"name": "anyio", "version": "4.15.1", "vulns": []}, {"name": "argon2-cffi", "version": "25.1.0", "vulns": []}, {"name": "argon2-cffi-bindings", "version": "26.1.0", "vulns": []}, {"name": "boto3", "version": "1.43.114", "vulns": []}, {"name": "botocore", "version": "1.43.114", "vulns": []}, {"name": "certifi", "version": "2026.10.23", "vulns": []}, {"name": "cffi", "version": "2.1.1", "vulns": []}, {"name": "charset-normalizer", "version": "3.5.2", "vulns": []}, {"name": "click", "version": "8.5.0", "vulns": []}, {"name": "fastapi", "version": "0.143.2", "vulns": []}, {"name": "googleapis-common-protos", "version": "1.75.5", "vulns": []}, {"name": "h11", "version": "0.16.0", "vulns": []}, {"name": "idna", "version": "3.20", "vulns": []}, {"name": "jmespath", "version": "1.1.0", "vulns": []}, {"name": "opentelemetry-api", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-exporter-http-transport", "version": "0.66b1", "vulns": []}, {"name": "opentelemetry-exporter-otlp-common", "version": "0.66b1", "vulns": []}, {"name": "opentelemetry-exporter-otlp-proto-common", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-exporter-otlp-proto-http", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-proto", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-sdk", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-semantic-conventions", "version": "0.66b1", "vulns": []}, {"name": "pillow", "version": "13.0.0", "vulns": []}, {"name": "protobuf", "version": "7.36.2", "vulns": []}, {"name": "psycopg", "version": "3.3.6", "vulns": []}, {"name": "psycopg-binary", "version": "3.3.6", "vulns": []}, {"name": "psycopg-pool", "version": "3.3.3", "vulns": []}, {"name": "pycparser", "version": "3.11", "vulns": []}, {"name": "pydantic", "version": "2.14.1", "vulns": []}, {"name": "pydantic-core", "version": "2.50.1", "vulns": []}, {"name": "python-dateutil", "version": "2.9.0.post0", "vulns": []}, {"name": "redis", "version": "8.1.0", "vulns": []}, {"name": "requests", "version": "2.34.2", "vulns": []}, {"name": "s3transfer", "version": "0.19.2", "vulns": []}, {"name": "six", "version": "1.17.0", "vulns": []}, {"name": "starlette", "version": "1.8.0", "vulns": []}, {"name": "typing-inspection", "version": "0.4.4", "vulns": []}, {"name": "typing-extensions", "version": "4.16.0", "vulns": []}, {"name": "tzdata", "version": "2026.5", "vulns": []}, {"name": "urllib3", "version": "2.8.0", "vulns": []}, {"name": "uvicorn", "version": "0.54.0", "vulns": []}], "fixes": []}
+11
View File
@@ -0,0 +1,11 @@
{
"target_instance": "b1a678ea159c4aaea77efbcc38b117e8",
"stopped_container": "0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3",
"before_containers": [
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
],
"after_running_containers": [
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7"
]
}
+3
View File
@@ -0,0 +1,3 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 77, "failed_attempts": 4, "first_recovery_seconds": 3.39}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+11
View File
@@ -0,0 +1,11 @@
{
"target_instance": "7f288f373c5140c3a3a1f8de74fa91f2",
"stopped_container": "0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3",
"before_containers": [
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
],
"after_running_containers": [
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7"
]
}
+3
View File
@@ -0,0 +1,3 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 83, "failed_attempts": 4, "first_recovery_seconds": 2.953}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+11
View File
@@ -0,0 +1,11 @@
{
"target_instance": "932b533b493e4dab8e09817e66f77901",
"stopped_container": "1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"before_containers": [
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
],
"after_running_containers": [
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
]
}
+3
View File
@@ -0,0 +1,3 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 79, "failed_attempts": 4, "first_recovery_seconds": 2.61}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+1
View File
@@ -0,0 +1 @@
Negative test identities generated
+1
View File
@@ -0,0 +1 @@
PASS: runtime privileges/resource limits/mounts/public ports and loopback-only plaintext
+502
View File
@@ -0,0 +1,502 @@
{
"passed": true,
"runtime": [
{
"service": "api1",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 536870912,
"NanoCpus": 500000000,
"pids_limit": 64,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "api1-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "api2",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 536870912,
"NanoCpus": 500000000,
"pids_limit": 64,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "api2-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "broker",
"user": "999:1000",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 134217728,
"NanoCpus": 250000000,
"pids_limit": 128,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "broker-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "cache",
"user": "999:1000",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 250000000,
"pids_limit": 128,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "cache-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "db",
"user": "999:999",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 268435456,
"NanoCpus": 1000000000,
"pids_limit": 128,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "db-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "dispatcher",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 134217728,
"NanoCpus": 250000000,
"pids_limit": 64,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "dispatcher-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "grafana",
"user": "472",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 536870912,
"NanoCpus": 1000000000,
"pids_limit": 128,
"published_loopback_ports": [
"3000"
],
"operator_material_absent": true
},
{
"service": "grafana-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "lb",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 500000000,
"pids_limit": 128,
"published_loopback_ports": [
"8443"
],
"operator_material_absent": true
},
{
"service": "loki",
"user": "10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 268435456,
"NanoCpus": 500000000,
"pids_limit": 128,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "loki-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "openbao",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 268435456,
"NanoCpus": 500000000,
"pids_limit": 128,
"published_loopback_ports": [
"8200"
],
"operator_material_absent": true
},
{
"service": "otel-collector",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 268435456,
"NanoCpus": 500000000,
"pids_limit": 128,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "otel-collector-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "prometheus",
"user": "nobody",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 268435456,
"NanoCpus": 500000000,
"pids_limit": 128,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "prometheus-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "reconciler",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 201326592,
"NanoCpus": 250000000,
"pids_limit": 64,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "reconciler-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "s3",
"user": "1000:1000",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 536870912,
"NanoCpus": 1000000000,
"pids_limit": 128,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "s3-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "tempo",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 268435456,
"NanoCpus": 500000000,
"pids_limit": 128,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "tempo-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "worker",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 805306368,
"NanoCpus": 1000000000,
"pids_limit": 64,
"published_loopback_ports": [],
"operator_material_absent": true
},
{
"service": "worker-tls",
"user": "10001:10001",
"read_only": true,
"cap_drop": [
"ALL"
],
"memory_bytes": 67108864,
"NanoCpus": 200000000,
"pids_limit": 32,
"published_loopback_ports": [],
"operator_material_absent": true
}
],
"grafana_verified_HTTPS": true,
"plaintext_isolation": {
"passed": true,
"probes": [
{
"host": "api1",
"port": 8080,
"refused": true
},
{
"host": "api2",
"port": 8080,
"refused": true
},
{
"host": "db",
"port": 5432,
"refused": true
},
{
"host": "s3",
"port": 8333,
"refused": true
},
{
"host": "s3",
"port": 8888,
"refused": true
},
{
"host": "s3",
"port": 9333,
"refused": true
},
{
"host": "s3",
"port": 19333,
"refused": true
},
{
"host": "cache",
"port": 6379,
"refused": true
},
{
"host": "broker",
"port": 6379,
"refused": true
},
{
"host": "otel-collector",
"port": 4318,
"refused": true
},
{
"host": "otel-collector",
"port": 9464,
"refused": true
},
{
"host": "otel-collector",
"port": 8888,
"refused": true
},
{
"host": "otel-collector",
"port": 13133,
"refused": true
},
{
"host": "prometheus",
"port": 9090,
"refused": true
},
{
"host": "loki",
"port": 3100,
"refused": true
},
{
"host": "loki",
"port": 9095,
"refused": true
},
{
"host": "tempo",
"port": 3200,
"refused": true
},
{
"host": "tempo",
"port": 4318,
"refused": true
},
{
"host": "tempo",
"port": 9095,
"refused": true
}
]
}
}
File diff suppressed because it is too large Load Diff
+2
View File
@@ -0,0 +1,2 @@
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+9
View File
@@ -0,0 +1,9 @@
scripts/check_contract.py: PASS
scripts/validate_openapi.py: PASS
tests/smoke.py: PASS
tests/security.py: PASS
persistence: PASS
failover-1: PASS
failover-2: PASS
failover-3: PASS
PASS: both distinct API containers failed over; all replicas restored
+1
View File
@@ -0,0 +1 @@
Operator action completed; no root token/unseal shares printed
+5
View File
@@ -0,0 +1,5 @@
....
----------------------------------------------------------------------
Ran 4 tests in 0.217s
OK
+735
View File
@@ -0,0 +1,735 @@
{
"before": {
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7": {
"service": "api1",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2835,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:55.809852669Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-13T09:52:33.85158481Z",
"End": "2026-11-13T09:52:34.414940821Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:39.416434594Z",
"End": "2026-11-13T09:52:40.03112071Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:45.032672537Z",
"End": "2026-11-13T09:52:45.684456356Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:50.685337932Z",
"End": "2026-11-13T09:52:51.337051054Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:55.883682844Z",
"End": "2026-11-13T09:52:56.745165826Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3": {
"service": "api2",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2820,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:55.797519555Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-13T09:52:35.50023978Z",
"End": "2026-11-13T09:52:35.92798761Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:40.929156411Z",
"End": "2026-11-13T09:52:41.906797728Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:46.908309853Z",
"End": "2026-11-13T09:52:47.31147321Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:51.873454972Z",
"End": "2026-11-13T09:52:52.395517815Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:57.397494814Z",
"End": "2026-11-13T09:52:57.897562413Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"76e0e9ee11f2acd4202aa4641b07c2af83558e6726dd26a3521b33bb520cf177": {
"service": "db",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 755,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:20.83333691Z",
"FinishedAt": "2026-11-13T09:50:28.225155745Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-13T09:52:50.379763185Z",
"End": "2026-11-13T09:52:50.456684316Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-13T09:52:52.018791868Z",
"End": "2026-11-13T09:52:52.157704126Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-13T09:52:54.159668931Z",
"End": "2026-11-13T09:52:54.239425636Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-13T09:52:56.241459415Z",
"End": "2026-11-13T09:52:56.374718752Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-13T09:52:58.375876362Z",
"End": "2026-11-13T09:52:58.425063406Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
]
}
}
},
"051a2a59f89f08acf5878b6ab22e2ad389bc09d4fa3d655604f27375b26922be": {
"service": "cache",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 840,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:21.347055009Z",
"FinishedAt": "2026-11-13T09:50:28.224207664Z"
}
},
"c0e3fc648ea6a23b643a0104ad9bf93f06bfd45e592fece24d8c4dc8788dc494": {
"service": "lb",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2892,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:55.925530847Z",
"FinishedAt": "2026-11-13T09:50:28.222150955Z"
}
},
"68c83b0d029545c4c8bfa67979e23fda461d7b9aea909d5e99e462f4858fcc1a": {
"service": "s3",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 863,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:21.411682013Z",
"FinishedAt": "2026-11-13T09:50:28.225741755Z"
}
},
"6db578724c16067c9101d8ffb911ae3bb1127c087e0b8a598c958983914676a7": {
"service": "reconciler",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2676,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:55.784764196Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}
},
"362e538c9c53fa7feb054689ed85f862d070518a168c0efaa6e184ad51bd34f5": {
"service": "broker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 879,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:21.392114019Z",
"FinishedAt": "2026-11-13T09:50:28.224003528Z"
}
},
"2e513af66f171401771b1a3680154f7edfc7472a5ee79e94041256efee78b2f0": {
"service": "worker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2709,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:55.951329668Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}
},
"c40815a4d87373db33cf7eb4b89d473b853dd7a9ece81fa73e052da54ced5ff4": {
"service": "dispatcher",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2629,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:55.684909892Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}
},
"ea547f7d390f190c74b9f60d5be466be91924df7ead9930e916f0e5bf9161ae0": {
"service": "openbao",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 940,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:21.574158786Z",
"FinishedAt": "2026-11-13T09:50:28.217091485Z"
}
},
"c605ad2ea7cfb8fdc37b85d5e8df440c0f9dfb1ba8317a8b87d84ce5ff38e2cb": {
"service": "otel-collector",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2156,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:52.584501439Z",
"FinishedAt": "2026-11-13T09:50:28.224053739Z"
}
},
"9b54b82ecd93b601a0ea977adedef1b81a1ff36eb925ab730684320a22113141": {
"service": "prometheus",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2352,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:53.97192011Z",
"FinishedAt": "2026-11-13T09:50:28.223361176Z"
}
},
"72ece8cb2659be6a83a2516dcb19b472d4ab9d6d1b50f20eca4f49799cd17131": {
"service": "loki",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2359,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:54.028034454Z",
"FinishedAt": "2026-11-13T09:50:28.224411699Z"
}
},
"574869ba08c1a34d0d30fa2da1e61c5a35dd66ee35afff36d18366e33658e621": {
"service": "tempo",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2288,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:53.935285487Z",
"FinishedAt": "2026-11-13T09:50:28.224318796Z"
}
},
"4ed391665684b152c494235932554b04ed24e82fc7213219f6cfc1dc54c3e4db": {
"service": "grafana",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2946,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:51:56.097459745Z",
"FinishedAt": "2026-11-13T09:50:28.225512752Z"
}
}
},
"after": {
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7": {
"service": "api1",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6310,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.810798832Z",
"FinishedAt": "2026-11-13T09:53:06.716904603Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-13T09:52:45.032672537Z",
"End": "2026-11-13T09:52:45.684456356Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:50.685337932Z",
"End": "2026-11-13T09:52:51.337051054Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:55.883682844Z",
"End": "2026-11-13T09:52:56.745165826Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:53:01.746116416Z",
"End": "2026-11-13T09:53:02.250230395Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:53:21.941393572Z",
"End": "2026-11-13T09:53:25.757437396Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
}
]
}
}
},
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3": {
"service": "api2",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6454,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.906757854Z",
"FinishedAt": "2026-11-13T09:53:08.979949677Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-13T09:52:46.908309853Z",
"End": "2026-11-13T09:52:47.31147321Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:51.873454972Z",
"End": "2026-11-13T09:52:52.395517815Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:52:57.397494814Z",
"End": "2026-11-13T09:52:57.897562413Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-13T09:53:02.902817408Z",
"End": "2026-11-13T09:53:06.233524775Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
},
{
"Start": "2026-11-13T09:53:22.46715569Z",
"End": "2026-11-13T09:53:25.869837051Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
}
]
}
}
},
"76e0e9ee11f2acd4202aa4641b07c2af83558e6726dd26a3521b33bb520cf177": {
"service": "db",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5985,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:14.810659332Z",
"FinishedAt": "2026-11-13T09:53:05.266445865Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-13T09:53:02.49033307Z",
"End": "2026-11-13T09:53:02.543395178Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-13T09:53:04.54387538Z",
"End": "2026-11-13T09:53:05.068330917Z",
"ExitCode": 1,
"Output": "/var/run/postgresql:5432 - rejecting connections\n"
},
{
"Start": "2026-11-13T09:53:18.178982393Z",
"End": "2026-11-13T09:53:20.786679419Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-13T09:53:22.800575887Z",
"End": "2026-11-13T09:53:24.416791098Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-13T09:53:26.420377151Z",
"End": "2026-11-13T09:53:26.842575441Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
]
}
}
},
"051a2a59f89f08acf5878b6ab22e2ad389bc09d4fa3d655604f27375b26922be": {
"service": "cache",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6487,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.945227154Z",
"FinishedAt": "2026-11-13T09:53:05.04199852Z"
}
},
"c0e3fc648ea6a23b643a0104ad9bf93f06bfd45e592fece24d8c4dc8788dc494": {
"service": "lb",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6479,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.9062729Z",
"FinishedAt": "2026-11-13T09:53:03.909611163Z"
}
},
"68c83b0d029545c4c8bfa67979e23fda461d7b9aea909d5e99e462f4858fcc1a": {
"service": "s3",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6375,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.849272466Z",
"FinishedAt": "2026-11-13T09:53:10.186208925Z"
}
},
"6db578724c16067c9101d8ffb911ae3bb1127c087e0b8a598c958983914676a7": {
"service": "reconciler",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6191,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.708018867Z",
"FinishedAt": "2026-11-13T09:53:09.175300496Z"
}
},
"362e538c9c53fa7feb054689ed85f862d070518a168c0efaa6e184ad51bd34f5": {
"service": "broker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6125,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.753030573Z",
"FinishedAt": "2026-11-13T09:53:05.167583954Z"
}
},
"2e513af66f171401771b1a3680154f7edfc7472a5ee79e94041256efee78b2f0": {
"service": "worker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6437,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.870744073Z",
"FinishedAt": "2026-11-13T09:53:09.192037057Z"
}
},
"c40815a4d87373db33cf7eb4b89d473b853dd7a9ece81fa73e052da54ced5ff4": {
"service": "dispatcher",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6311,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.806673713Z",
"FinishedAt": "2026-11-13T09:53:09.581490205Z"
}
},
"ea547f7d390f190c74b9f60d5be466be91924df7ead9930e916f0e5bf9161ae0": {
"service": "openbao",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6517,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.911866476Z",
"FinishedAt": "2026-11-13T09:53:03.972241104Z"
}
},
"c605ad2ea7cfb8fdc37b85d5e8df440c0f9dfb1ba8317a8b87d84ce5ff38e2cb": {
"service": "otel-collector",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5717,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:13.768718976Z",
"FinishedAt": "2026-11-13T09:53:05.062618953Z"
}
},
"9b54b82ecd93b601a0ea977adedef1b81a1ff36eb925ab730684320a22113141": {
"service": "prometheus",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5926,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:14.756925187Z",
"FinishedAt": "2026-11-13T09:53:05.244841875Z"
}
},
"72ece8cb2659be6a83a2516dcb19b472d4ab9d6d1b50f20eca4f49799cd17131": {
"service": "loki",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5961,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:14.793956064Z",
"FinishedAt": "2026-11-13T09:53:05.416062037Z"
}
},
"574869ba08c1a34d0d30fa2da1e61c5a35dd66ee35afff36d18366e33658e621": {
"service": "tempo",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5958,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:14.761611926Z",
"FinishedAt": "2026-11-13T09:53:05.299312685Z"
}
},
"4ed391665684b152c494235932554b04ed24e82fc7213219f6cfc1dc54c3e4db": {
"service": "grafana",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6442,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-13T09:53:15.90242133Z",
"FinishedAt": "2026-11-13T09:53:05.770095425Z"
}
}
},
"volumes_removed": false
}
+2
View File
@@ -0,0 +1,2 @@
Operator action completed; no root token/unseal shares printed
PASS: sealed Bao -> finite lease -> both API children stopped -> unseal -> recovery
+254
View File
@@ -0,0 +1,254 @@
{
"passed": true,
"secret_refresh_seconds": 20,
"token_max_TTL_seconds": 120,
"both_API_children_stopped_seconds": 123.86,
"recovery_seconds": 20.44,
"observations": [
{
"seconds": 1.3,
"HTTP": 200
},
{
"seconds": 3.3,
"HTTP": 200
},
{
"seconds": 5.4,
"HTTP": 200
},
{
"seconds": 7.4,
"HTTP": 200
},
{
"seconds": 9.4,
"HTTP": 200
},
{
"seconds": 11.5,
"HTTP": 200
},
{
"seconds": 13.5,
"HTTP": 200
},
{
"seconds": 15.5,
"HTTP": 200
},
{
"seconds": 17.5,
"HTTP": 200
},
{
"seconds": 19.6,
"HTTP": 200
},
{
"seconds": 21.6,
"HTTP": 200
},
{
"seconds": 23.6,
"HTTP": 200
},
{
"seconds": 25.7,
"HTTP": 200
},
{
"seconds": 27.7,
"HTTP": 200
},
{
"seconds": 29.7,
"HTTP": 200
},
{
"seconds": 31.8,
"HTTP": 200
},
{
"seconds": 33.8,
"HTTP": 200
},
{
"seconds": 35.8,
"HTTP": 200
},
{
"seconds": 37.9,
"HTTP": 200
},
{
"seconds": 39.9,
"HTTP": 200
},
{
"seconds": 42.0,
"HTTP": 200
},
{
"seconds": 44.0,
"HTTP": 200
},
{
"seconds": 46.0,
"HTTP": 200
},
{
"seconds": 48.0,
"HTTP": 200
},
{
"seconds": 50.1,
"HTTP": 200
},
{
"seconds": 52.1,
"HTTP": 200
},
{
"seconds": 54.1,
"HTTP": 200
},
{
"seconds": 56.2,
"HTTP": 200
},
{
"seconds": 58.2,
"HTTP": 200
},
{
"seconds": 60.2,
"HTTP": 200
},
{
"seconds": 62.3,
"HTTP": 200
},
{
"seconds": 64.3,
"HTTP": 200
},
{
"seconds": 66.3,
"HTTP": 200
},
{
"seconds": 68.4,
"HTTP": 200
},
{
"seconds": 70.4,
"HTTP": 200
},
{
"seconds": 72.4,
"HTTP": 200
},
{
"seconds": 74.5,
"HTTP": 200
},
{
"seconds": 76.5,
"HTTP": 200
},
{
"seconds": 78.5,
"HTTP": 200
},
{
"seconds": 80.5,
"HTTP": 200
},
{
"seconds": 82.6,
"HTTP": 200
},
{
"seconds": 84.6,
"HTTP": 200
},
{
"seconds": 86.7,
"HTTP": 200
},
{
"seconds": 88.7,
"HTTP": 200
},
{
"seconds": 90.8,
"HTTP": 200
},
{
"seconds": 92.8,
"HTTP": 200
},
{
"seconds": 94.8,
"HTTP": 200
},
{
"seconds": 96.8,
"HTTP": 200
},
{
"seconds": 98.8,
"HTTP": 200
},
{
"seconds": 100.9,
"HTTP": 200
},
{
"seconds": 102.9,
"HTTP": 200
},
{
"seconds": 104.9,
"HTTP": 200
},
{
"seconds": 107.0,
"HTTP": 200
},
{
"seconds": 109.0,
"HTTP": 200
},
{
"seconds": 111.0,
"HTTP": 200
},
{
"seconds": 113.1,
"HTTP": 200
},
{
"seconds": 115.1,
"HTTP": 200
},
{
"seconds": 117.1,
"HTTP": 200
},
{
"seconds": 119.2,
"HTTP": 502
},
{
"seconds": 121.2,
"HTTP": 502
},
{
"seconds": 123.2,
"HTTP": 503
}
],
"scope": "Supervisor bounds use during Bao outage; static backend passwords themselves require explicit rotation."
}
+1
View File
@@ -0,0 +1 @@
PASS: ACL, expiry/logout, hash storage/rehash, shared 12/10s rate limit, measured Argon2
+131
View File
@@ -0,0 +1,131 @@
{
"passed": true,
"ACL_attachment_denials": 4,
"expired_session_denied": true,
"logout_denied": true,
"token_hash_sha256": true,
"rehash": "19456 -> 32768 KiB",
"login_burst": [
{
"status": 401,
"seconds": 2.891,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.453,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 4.078,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 3.094,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.281,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 3.188,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 3.375,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.281,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 2.11,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 3.891,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 4.547,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 4.062,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 3.781,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.5,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 429,
"seconds": 0.531,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 3.125,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 429,
"seconds": 0.39,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 4.453,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 429,
"seconds": 0.39,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.453,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
}
],
"burst_seconds": 4.578,
"KDF": {
"hash_plus_verify_seconds": [
0.21583701900004826,
0.20667367100003275,
0.22015716700002486,
0.193256392999956,
0.20241064300000744,
0.20739852500003053,
0.20553775699994503,
0.2716375199999561,
0.2128679730000158,
0.22063045700002704,
0.20332950599998867,
0.20581266300001744
],
"median": 0.20703609800003164,
"max_rss_kib": 53184
},
"KDF_slots_per_API": 2,
"KDF_memory_bound_two_APIs_MiB": 128
}
+4
View File
@@ -0,0 +1,4 @@
{
"passed": true,
"completed_at_UTC": "2026-11-13T10:00:37.671336+00:00"
}
+11
View File
@@ -0,0 +1,11 @@
generate-test-identities: PASS
publish-crl: PASS
tls-matrix: PASS
bao-policy: PASS
worker-rights: PASS
security-api-run: PASS
compromised-worker-run: PASS
sealed-bao-run: PASS
telemetry-security-run: PASS
hardening-run: PASS
render-unit: PASS
+1
View File
@@ -0,0 +1 @@
PASS: trusted HTTPS works; untrusted CA rejected. This does NOT verify internal mTLS, OpenBao or hardening.
+1
View File
@@ -0,0 +1 @@
PASS: lab7 public API scenario; infrastructure and grading evidence remain separate.
+11
View File
@@ -0,0 +1,11 @@
{
"target_instance": "932b533b493e4dab8e09817e66f77901",
"stopped_container": "1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"before_containers": [
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
],
"after_running_containers": [
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
]
}
+1
View File
@@ -0,0 +1 @@
PASS: TLS metrics/traces/log correlation, credential canaries absent in telemetry/container logs
+21
View File
@@ -0,0 +1,21 @@
{
"passed": true,
"trace_id": "918070f9c17e4f2b9b8e079df1824a06",
"span_count": 13,
"span_names": [
"POST /api/v1/chats/{chat_id}/attachments",
"attachment.consume",
"attachment.durable_accept",
"attachment.publish",
"db.attachment.commit",
"db.session.authenticate",
"image.render",
"s3.get",
"s3.put"
],
"correlated_log_count": 3,
"http_metric_series": 26,
"credential_canaries_absent": true,
"all_queries": "mTLS verified operator identity",
"container_logs_canaries_absent": true
}
+397
View File
@@ -0,0 +1,397 @@
{
"passed": true,
"seconds": 2.49,
"edges": [
{
"source": "lb",
"destination": "api1",
"port": 8443,
"allowed": true
},
{
"destination": "api1",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "lb",
"destination": "api2",
"port": 8443,
"allowed": true
},
{
"destination": "api2",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "worker",
"destination": "broker",
"port": 8443,
"allowed": true
},
{
"source": "dispatcher",
"destination": "broker",
"port": 8443,
"allowed": true
},
{
"source": "operator",
"destination": "broker",
"port": 8443,
"allowed": true
},
{
"destination": "broker",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "api1",
"destination": "cache",
"port": 8443,
"allowed": true
},
{
"source": "api2",
"destination": "cache",
"port": 8443,
"allowed": true
},
{
"source": "operator",
"destination": "cache",
"port": 8443,
"allowed": true
},
{
"destination": "cache",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "api1",
"destination": "db",
"port": 8443,
"allowed": true
},
{
"source": "api2",
"destination": "db",
"port": 8443,
"allowed": true
},
{
"source": "worker",
"destination": "db",
"port": 8443,
"allowed": true
},
{
"source": "dispatcher",
"destination": "db",
"port": 8443,
"allowed": true
},
{
"source": "reconciler",
"destination": "db",
"port": 8443,
"allowed": true
},
{
"source": "operator",
"destination": "db",
"port": 8443,
"allowed": true
},
{
"destination": "db",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "otel-collector",
"destination": "loki",
"port": 8443,
"allowed": true
},
{
"source": "grafana",
"destination": "loki",
"port": 8443,
"allowed": true
},
{
"source": "operator",
"destination": "loki",
"port": 8443,
"allowed": true
},
{
"destination": "loki",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "api1",
"destination": "otel-collector",
"port": 8443,
"allowed": true
},
{
"source": "api2",
"destination": "otel-collector",
"port": 8443,
"allowed": true
},
{
"source": "worker",
"destination": "otel-collector",
"port": 8443,
"allowed": true
},
{
"source": "dispatcher",
"destination": "otel-collector",
"port": 8443,
"allowed": true
},
{
"source": "reconciler",
"destination": "otel-collector",
"port": 8443,
"allowed": true
},
{
"source": "operator",
"destination": "otel-collector",
"port": 8443,
"allowed": true
},
{
"destination": "otel-collector",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "prometheus",
"destination": "otel-collector",
"port": 8444,
"allowed": true
},
{
"source": "operator",
"destination": "otel-collector",
"port": 8444,
"allowed": true
},
{
"destination": "otel-collector",
"port": 8444,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "prometheus",
"destination": "otel-collector",
"port": 8445,
"allowed": true
},
{
"source": "operator",
"destination": "otel-collector",
"port": 8445,
"allowed": true
},
{
"destination": "otel-collector",
"port": 8445,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "grafana",
"destination": "prometheus",
"port": 8443,
"allowed": true
},
{
"source": "operator",
"destination": "prometheus",
"port": 8443,
"allowed": true
},
{
"destination": "prometheus",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "api1",
"destination": "s3",
"port": 8443,
"allowed": true
},
{
"source": "api2",
"destination": "s3",
"port": 8443,
"allowed": true
},
{
"source": "worker",
"destination": "s3",
"port": 8443,
"allowed": true
},
{
"source": "reconciler",
"destination": "s3",
"port": 8443,
"allowed": true
},
{
"source": "operator",
"destination": "s3",
"port": 8443,
"allowed": true
},
{
"destination": "s3",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "otel-collector",
"destination": "tempo",
"port": 8443,
"allowed": true
},
{
"destination": "tempo",
"port": 8443,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
},
{
"source": "grafana",
"destination": "tempo",
"port": 8444,
"allowed": true
},
{
"source": "operator",
"destination": "tempo",
"port": 8444,
"allowed": true
},
{
"destination": "tempo",
"port": 8444,
"denied": [
"no_cert",
"wrong_CA",
"wrong_SAN",
"expired",
"revoked",
"wrong_server_SAN",
"plaintext"
]
}
]
}
+1
View File
@@ -0,0 +1 @@
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
+18
View File
@@ -0,0 +1,18 @@
{
"passed": true,
"rights": {
"jobs_read": "allowed",
"sessions": "denied",
"password_hashes": "denied",
"messages": "denied",
"schema": "denied",
"privilege": "denied",
"variants_read_write_delete": "allowed",
"foreign_prefix": "denied",
"bucket_admin": "denied",
"queue_group": "allowed",
"queue_publish": "denied",
"foreign_queue": "denied",
"queue_admin": "denied"
}
}
+16 -4
View File
@@ -1,7 +1,19 @@
# Инфраструктура студента
# Infrastructure ЛР7
Здесь разместите конфигурации выбранных сервисов; основную топологию — в корневом compose.yaml/stack.yaml. `compose.example.yaml` — только worksheet ролей и намеренно не запускается. `contracts/infrastructure.json` задаёт минимальные роли, не готовые образы/пароли.
Исполняемый стенд: ../compose.yaml. Полная инструкция: ../REPORT.md.
Все versions закреплены digest. Подготовка: `python scripts/up.py` из корня.
В REPORT.md сопоставьте роль → service name → версия/digest → health/readiness → volume → сеть/порт → способ передачи credentials. Приведите bootstrap (schema, bucket, broker topology, а с lab6 provisioning) и безопасное повторение этих команд.
- api1/api2 — non-root ASGI + cert-auth secret supervisor, public LB только HTTPS8443.
- *-tls — отдельные identities, mTLS SAN allowlist, CA+CRL; shared namespace с backend.
- db — PostgreSQL/SCRAM, named pgdata, операторские migrations/GRANT.
- cache/broker — разные Valkey ACL/configs; persistent AOF только у broker.
- s3 — private SeaweedFS bucket, раздельные SigV4 users/prefix ACL, named s3data.
- openbao — native TLS+client verification, persistent raft, declarative audit.
- telemetry — Collector/Prometheus/Loki/Tempo/Grafana, persistent backend volumes.
- secret-init/telemetry-init/storage-init — ограниченные операторские bootstrap jobs.
Приложение строится из исходников. Не используйте заранее существующий образ преподавательского сервера вместо своей реализации. Restart hook перезапускает все соответствующие stores с volumes; failover hook останавливает только заданную API-реплику.
Матрица соединений: ../contracts/trust-matrix.md. Native plaintext listener
закрыт на loopback, TLS hop существует на каждом межконтейнерном соединении.
`docker compose restart api1` отдельно от sidecar не является штатным workflow:
используйте scripts/restart.py для согласованного stop/start shared namespaces.
Volume deletion не входит ни в запуск, ни в тесты.
+31
View File
@@ -0,0 +1,31 @@
groups:
- name: messenger-slo
interval: 5s
rules:
- alert: APIAvailabilityBurn
expr: |
(sum(rate(http_server_requests_total{status_class="5xx"}[20s])) / clamp_min(sum(rate(http_server_requests_total[20s])), 0.001) / 0.005 > 10)
and
(sum(rate(http_server_requests_total{status_class="5xx"}[60s])) / clamp_min(sum(rate(http_server_requests_total[60s])), 0.001) / 0.005 > 10)
for: 30s
labels: {severity: critical}
annotations:
summary: "API 99.5% availability budget burns >10x in both 20s and 60s windows"
runbook: "RUNBOOK.md#api"
- alert: AttachmentReadyBurn
expr: |
(sum(attachment_slo_bad{window="60"}) / clamp_min(sum(attachment_slo_eligible{window="60"}),1) / 0.01 > 10)
and
(sum(attachment_slo_bad{window="300"}) / clamp_min(sum(attachment_slo_eligible{window="300"}),1) / 0.01 > 10)
for: 15s
labels: {severity: critical}
annotations:
summary: "99% ready within 30s budget burns >10x in both 60s and 300s eligible cohorts"
runbook: "RUNBOOK.md#queue"
- alert: QueueLag
expr: max(queue_oldest_age_seconds) > 30 and sum(queue_depth) > 0
for: 10s
labels: {severity: warning}
annotations:
summary: "An unfinished durable job has waited over 30s"
runbook: "RUNBOOK.md#queue"
+669
View File
@@ -0,0 +1,669 @@
{
"uid": "mtusi-messenger",
"title": "MTUSI Messenger - signals and SLO",
"schemaVersion": 39,
"version": 1,
"editable": false,
"timezone": "browser",
"time": {
"from": "now-15m",
"to": "now"
},
"refresh": "15s",
"tags": [
"mtusi",
"lab6"
],
"panels": [
{
"id": 1,
"type": "timeseries",
"title": "API requests / second",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum by(method,route)(rate(http_server_requests_total[1m]))",
"legendFormat": "{{method}} {{route}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 2,
"type": "timeseries",
"title": "API 5xx / all business requests",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)",
"legendFormat": "5xx fraction"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 3,
"type": "timeseries",
"title": "API latency: aggregate buckets across replicas",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 4,
"type": "timeseries",
"title": "Jobs / second by outcome",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum by(outcome)(rate(jobs_processed_total[1m]))",
"legendFormat": "{{outcome}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 5,
"type": "timeseries",
"title": "Worker processing latency",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 6,
"type": "timeseries",
"title": "Acceptance to ready latency",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 7,
"type": "timeseries",
"title": "Unfinished durable jobs",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 24,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(queue_depth)",
"legendFormat": "queue depth"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 8,
"type": "timeseries",
"title": "Oldest unfinished job age",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 24,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "max(queue_oldest_age_seconds)",
"legendFormat": "oldest"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 9,
"type": "timeseries",
"title": "DB pool",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 32,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "db_pool_used",
"legendFormat": "used {{process_instance}}"
},
{
"refId": "B",
"expr": "db_pool_waiting",
"legendFormat": "waiting {{process_instance}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 10,
"type": "timeseries",
"title": "Worker saturation",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 32,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(worker_busy) / clamp_min(sum(worker_capacity),1)",
"legendFormat": "busy / capacity"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 11,
"type": "timeseries",
"title": "Current process RSS",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 40,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "process_rss_bytes",
"legendFormat": "{{service}} {{process_instance}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 12,
"type": "timeseries",
"title": "Process CPU seconds / wall second",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 40,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "clamp_min(deriv(process_cpu_time_seconds[1m]),0)",
"legendFormat": "{{service}} {{process_instance}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 13,
"type": "timeseries",
"title": "API availability burn rate; budget 0.5%",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[20s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[20s])),0.001) / 0.005",
"legendFormat": "20s"
},
{
"refId": "B",
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[60s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[60s])),0.001) / 0.005",
"legendFormat": "60s"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 14,
"type": "timeseries",
"title": "Ready <=30s burn rate; budget 1%",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(attachment_slo_bad{window=\"60\"}) / clamp_min(sum(attachment_slo_eligible{window=\"60\"}),1) / 0.01",
"legendFormat": "60s eligible cohort"
},
{
"refId": "B",
"expr": "sum(attachment_slo_bad{window=\"300\"}) / clamp_min(sum(attachment_slo_eligible{window=\"300\"}),1) / 0.01",
"legendFormat": "300s eligible cohort"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 15,
"type": "timeseries",
"title": "Collector exporter queue / capacity",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "otelcol_exporter_queue_size",
"legendFormat": "{{exporter}} size"
},
{
"refId": "B",
"expr": "otelcol_exporter_queue_capacity",
"legendFormat": "{{exporter}} capacity"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 16,
"type": "timeseries",
"title": "Collector failed exports",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "rate(otelcol_exporter_send_failed_spans_total[1m])",
"legendFormat": "{{exporter}} spans"
},
{
"refId": "B",
"expr": "rate(otelcol_exporter_send_failed_log_records_total[1m])",
"legendFormat": "{{exporter}} logs"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 99,
"type": "logs",
"title": "Structured application logs; TraceID links to Tempo",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"x": 0,
"y": 64,
"w": 24,
"h": 10
},
"targets": [
{
"refId": "A",
"expr": "{service_name=~\"messenger-(api|worker|dispatcher)\"}"
}
],
"options": {
"showTime": true,
"showLabels": false,
"wrapLogMessage": true,
"sortOrder": "Descending"
}
}
],
"annotations": {
"list": []
},
"templating": {
"list": []
}
}
@@ -0,0 +1,9 @@
apiVersion: 1
providers:
- name: Messenger
folder: MTUSI
type: file
disableDeletion: true
updateIntervalSeconds: 30
options:
path: /var/lib/grafana/dashboards
@@ -0,0 +1,31 @@
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
url: http://127.0.0.1:19090
isDefault: true
- name: Loki
uid: loki
type: loki
access: proxy
url: http://127.0.0.1:13100
jsonData:
derivedFields:
- name: TraceID
matcherRegex: '"trace_id": "([0-9a-f]{32})"'
datasourceUid: tempo
url: '$${__value.raw}'
- name: Tempo
uid: tempo
type: tempo
access: proxy
url: http://127.0.0.1:13200
jsonData:
tracesToLogsV2:
datasourceUid: loki
spanStartTimeShift: '-1m'
spanEndTimeShift: '1m'
filterByTraceID: true
nodeGraph: {enabled: true}
+32
View File
@@ -0,0 +1,32 @@
global
log stdout format raw local0
maxconn 256
defaults
mode http
log global
timeout connect 1s
timeout client 35s
timeout server 32s
timeout check 1s
timeout queue 2s
retries 0
option http-server-close
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold obsolete 1s
hold nx 1s
hold timeout 1s
frontend public
bind :8443 ssl crt /identity/combined.pem ssl-min-ver TLSv1.2
default_backend api
backend api
balance roundrobin
option log-health-checks
option httpchk GET /health/ready
http-check expect status 200
server api1 api1:8443 ssl verify required ca-file /trust/ca.crt crl-file /trust/revocations.pem crt /identity/combined.pem verifyhost api1 sni str(api1) check-sni api1 check inter 1s resolvers docker init-addr none
server api2 api2:8443 ssl verify required ca-file /trust/ca.crt crl-file /trust/revocations.pem crt /identity/combined.pem verifyhost api2 sni str(api2) check-sni api2 check inter 1s resolvers docker init-addr none
+36
View File
@@ -0,0 +1,36 @@
auth_enabled: false
server:
http_listen_address: 127.0.0.1
grpc_listen_address: 127.0.0.1
http_listen_port: 3100
common:
instance_addr: 127.0.0.1
path_prefix: /var/loki
replication_factor: 1
ring:
kvstore:
store: inmemory
storage:
filesystem:
chunks_directory: /var/loki/chunks
rules_directory: /var/loki/rules
schema_config:
configs:
- from: 2026-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
allow_structured_metadata: true
retention_period: 24h
ingestion_rate_mb: 2
ingestion_burst_size_mb: 4
compactor:
working_directory: /var/loki/compactor
retention_enabled: true
delete_request_store: filesystem
analytics:
reporting_enabled: false
+34
View File
@@ -0,0 +1,34 @@
[
{
"mode": "server",
"listen": 8443,
"target": 8080,
"allow": [
"lb"
]
},
{
"mode": "client",
"listen": 5432,
"host": "db",
"target": 8443
},
{
"mode": "client",
"listen": 8333,
"host": "s3",
"target": 8443
},
{
"mode": "client",
"listen": 4318,
"host": "otel-collector",
"target": 8443
},
{
"mode": "client",
"listen": 6379,
"host": "cache",
"target": 8443
}
]
+34
View File
@@ -0,0 +1,34 @@
[
{
"mode": "server",
"listen": 8443,
"target": 8080,
"allow": [
"lb"
]
},
{
"mode": "client",
"listen": 5432,
"host": "db",
"target": 8443
},
{
"mode": "client",
"listen": 8333,
"host": "s3",
"target": 8443
},
{
"mode": "client",
"listen": 4318,
"host": "otel-collector",
"target": 8443
},
{
"mode": "client",
"listen": 6379,
"host": "cache",
"target": 8443
}
]
+12
View File
@@ -0,0 +1,12 @@
[
{
"mode": "server",
"listen": 8443,
"target": 6379,
"allow": [
"worker",
"dispatcher",
"operator"
]
}
]
+12
View File
@@ -0,0 +1,12 @@
[
{
"mode": "server",
"listen": 8443,
"target": 6379,
"allow": [
"api1",
"api2",
"operator"
]
}
]
+15
View File
@@ -0,0 +1,15 @@
[
{
"mode": "server",
"listen": 8443,
"target": 5432,
"allow": [
"api1",
"api2",
"worker",
"dispatcher",
"reconciler",
"operator"
]
}
]
+20
View File
@@ -0,0 +1,20 @@
[
{
"mode": "client",
"listen": 5432,
"host": "db",
"target": 8443
},
{
"mode": "client",
"listen": 4318,
"host": "otel-collector",
"target": 8443
},
{
"mode": "client",
"listen": 6380,
"host": "broker",
"target": 8443
}
]
+20
View File
@@ -0,0 +1,20 @@
[
{
"mode": "client",
"listen": 19090,
"host": "prometheus",
"target": 8443
},
{
"mode": "client",
"listen": 13100,
"host": "loki",
"target": 8443
},
{
"mode": "client",
"listen": 13200,
"host": "tempo",
"target": 8444
}
]
+12
View File
@@ -0,0 +1,12 @@
[
{
"mode": "server",
"listen": 8443,
"target": 3100,
"allow": [
"otel-collector",
"grafana",
"operator"
]
}
]
+45
View File
@@ -0,0 +1,45 @@
[
{
"mode": "server",
"listen": 8443,
"target": 4318,
"allow": [
"api1",
"api2",
"worker",
"dispatcher",
"reconciler",
"operator"
]
},
{
"mode": "server",
"listen": 8444,
"target": 9464,
"allow": [
"prometheus",
"operator"
]
},
{
"mode": "server",
"listen": 8445,
"target": 8888,
"allow": [
"prometheus",
"operator"
]
},
{
"mode": "client",
"listen": 14318,
"host": "tempo",
"target": 8443
},
{
"mode": "client",
"listen": 13100,
"host": "loki",
"target": 8443
}
]
+23
View File
@@ -0,0 +1,23 @@
[
{
"mode": "server",
"listen": 8443,
"target": 9090,
"allow": [
"grafana",
"operator"
]
},
{
"mode": "client",
"listen": 19464,
"host": "otel-collector",
"target": 8444
},
{
"mode": "client",
"listen": 18888,
"host": "otel-collector",
"target": 8445
}
]
+20
View File
@@ -0,0 +1,20 @@
[
{
"mode": "client",
"listen": 5432,
"host": "db",
"target": 8443
},
{
"mode": "client",
"listen": 8333,
"host": "s3",
"target": 8443
},
{
"mode": "client",
"listen": 4318,
"host": "otel-collector",
"target": 8443
}
]
+14
View File
@@ -0,0 +1,14 @@
[
{
"mode": "server",
"listen": 8443,
"target": 8333,
"allow": [
"api1",
"api2",
"worker",
"reconciler",
"operator"
]
}
]
+19
View File
@@ -0,0 +1,19 @@
[
{
"mode": "server",
"listen": 8443,
"target": 4318,
"allow": [
"otel-collector"
]
},
{
"mode": "server",
"listen": 8444,
"target": 3200,
"allow": [
"grafana",
"operator"
]
}
]
+26
View File
@@ -0,0 +1,26 @@
[
{
"mode": "client",
"listen": 5432,
"host": "db",
"target": 8443
},
{
"mode": "client",
"listen": 8333,
"host": "s3",
"target": 8443
},
{
"mode": "client",
"listen": 4318,
"host": "otel-collector",
"target": 8443
},
{
"mode": "client",
"listen": 6380,
"host": "broker",
"target": 8443
}
]
+21
View File
@@ -0,0 +1,21 @@
ui = false
disable_mlock = true
api_addr = "https://openbao:8200"
cluster_addr = "https://127.0.0.1:8201"
audit "file" "operator" {
options {
file_path = "/bao/audit.json"
}
}
storage "raft" {
path = "/bao/data"
node_id = "lab7"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/identity/cert.pem"
tls_key_file = "/identity/key.pem"
tls_client_ca_file = "/trust/ca.crt"
tls_require_and_verify_client_cert = true
tls_min_version = "tls12"
}
+67
View File
@@ -0,0 +1,67 @@
receivers:
otlp:
protocols:
http:
endpoint: 127.0.0.1:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 192
spike_limit_mib: 48
batch:
timeout: 1s
send_batch_size: 64
send_batch_max_size: 128
exporters:
prometheus:
endpoint: 127.0.0.1:9464
enable_open_metrics: true
metric_expiration: 2m
otlphttp/tempo:
endpoint: http://127.0.0.1:14318
timeout: 2s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 32
retry_on_failure:
initial_interval: 1s
max_interval: 5s
max_elapsed_time: 30s
otlphttp/loki:
endpoint: http://127.0.0.1:13100/otlp
timeout: 2s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 32
retry_on_failure:
initial_interval: 1s
max_interval: 5s
max_elapsed_time: 30s
extensions:
health_check:
endpoint: 127.0.0.1:13133
service:
extensions: [health_check]
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus:
host: 127.0.0.1
port: 8888
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/loki]
+12 -5
View File
@@ -1,7 +1,14 @@
# Конфигурация наблюдаемости — реализовать
# Telemetry ЛР7
Добавьте `collector.yaml` с OTLP receivers, processors и exporters выбранной версии Collector. Укажите явные bind endpoints, внутренние сети, memory limits/batch и поведение при backpressure по уровню. Сопоставьте все три pipeline с реально развёрнутыми backend.
Перенесена реализация ЛР6. Collector принимает OTLP HTTP только на loopback;
сеть доступна через otel-collector:8443 с mTLS и SAN allowlist. Native экспорт
Collector → Tempo/Loki идёт в локальные tunnels, которые проверяют server CA/SAN
и предъявляют identity Collector. Prometheus scrape и Grafana queries также
используют разные mTLS identities. Backend ports не публикуются на хосте.
Добавьте конфигурации storage, Grafana provisioning/datasources, provisioning/dashboards и импортируемые dashboard JSON. На 4 — alert rules и runbooks. Пустой dashboard или debug exporter, который только печатает данные в stdout, не заменяют хранилища/доказательства из `contracts/observability.md`.
Не публикуйте OTLP/metrics/административные ports в общую сеть без необходимости. В lab7 реализуйте TLS/mTLS по trust matrix; экспорт Authorization и message bodies запрещён.
Provisioned dashboard: infra/grafana/dashboards/messenger.json. Grafana доступна
на https://localhost:3000, пароль admin создаёт bootstrap; анонимного доступа нет.
Секреты, bodies и Authorization не включены в application spans/logs.
`python checks/telemetry_security.py` проверяет upload trace, correlated logs,
HTTP metric series и отсутствие известных credential canaries. Подробности
в REPORT.md и contracts/trust-matrix.md.
+8
View File
@@ -0,0 +1,8 @@
# Operator-only Unix socket; no socket is shared with app containers.
local all messenger trust
local all all reject
# Plaintext exists only in the db TLS sidecar's shared loopback namespace.
# The TLS boundary enforces the peer identity; PostgreSQL ALSO verifies SCRAM.
host all all 127.0.0.1/32 scram-sha-256
host all all ::1/128 reject
host all all 0.0.0.0/0 reject
+12
View File
@@ -0,0 +1,12 @@
global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- /etc/prometheus/alerts.yaml
scrape_configs:
- job_name: messenger-otlp
static_configs:
- targets: [127.0.0.1:19464]
- job_name: collector
static_configs:
- targets: [127.0.0.1:18888]
+28
View File
@@ -0,0 +1,28 @@
server:
http_listen_address: 127.0.0.1
grpc_listen_address: 127.0.0.1
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
http:
endpoint: 127.0.0.1:4318
ingester:
trace_idle_period: 5s
max_block_duration: 1m
compactor:
compaction:
block_retention: 24h
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
local:
path: /var/tempo/blocks
overrides:
defaults:
ingestion:
rate_limit_bytes: 1000000
burst_size_bytes: 2000000
+35
View File
@@ -0,0 +1,35 @@
CREATE TABLE users (
id uuid PRIMARY KEY,
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
password_hash text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE sessions (
token_hash text PRIMARY KEY,
user_id uuid NOT NULL REFERENCES users(id),
expires_at timestamptz NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX sessions_expiry ON sessions(expires_at);
CREATE TABLE chats (
id uuid PRIMARY KEY,
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE members (
chat_id uuid NOT NULL REFERENCES chats(id),
user_id uuid NOT NULL REFERENCES users(id),
PRIMARY KEY(chat_id,user_id)
);
CREATE INDEX members_user_chat ON members(user_id,chat_id);
CREATE INDEX chats_order ON chats(created_at,id);
CREATE TABLE messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
sender_id uuid NOT NULL REFERENCES users(id),
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
FOREIGN KEY(chat_id,sender_id) REFERENCES members(chat_id,user_id)
);
CREATE INDEX messages_history ON messages(chat_id,created_at,id);
+10
View File
@@ -0,0 +1,10 @@
CREATE TABLE idempotency (
actor_id uuid NOT NULL REFERENCES users(id),
chat_id uuid NOT NULL REFERENCES chats(id),
key text NOT NULL CHECK (key ~ '^[A-Za-z0-9._:-]{1,128}$'),
request_hash text NOT NULL,
message_id uuid NOT NULL REFERENCES messages(id),
expires_at timestamptz NOT NULL,
PRIMARY KEY(actor_id,chat_id,key)
);
CREATE INDEX idempotency_expiry ON idempotency(expires_at);
+25
View File
@@ -0,0 +1,25 @@
CREATE TABLE attachments (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
owner_id uuid NOT NULL REFERENCES users(id),
filename text NOT NULL,
object_key text NOT NULL UNIQUE,
content_type text NOT NULL CHECK (content_type IN ('image/png','image/jpeg')),
size_bytes bigint NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
sha256 text NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
storage_state text NOT NULL DEFAULT 'ready' CHECK (storage_state IN ('ready','missing')),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
UNIQUE (id, chat_id),
FOREIGN KEY (chat_id, owner_id) REFERENCES members(chat_id,user_id)
);
CREATE INDEX attachments_chat ON attachments(chat_id,created_at,id);
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_check;
ALTER TABLE messages ADD CONSTRAINT messages_content_check
CHECK (length(text) <= 4000 AND (length(btrim(text)) > 0 OR cardinality(attachment_ids) > 0));
CREATE TABLE message_attachments (
message_id uuid NOT NULL REFERENCES messages(id),
attachment_id uuid NOT NULL REFERENCES attachments(id),
PRIMARY KEY(message_id,attachment_id)
);
+41
View File
@@ -0,0 +1,41 @@
ALTER TABLE attachments ADD COLUMN status text NOT NULL DEFAULT 'ready'
CHECK (status IN ('queued','processing','ready','failed'));
ALTER TABLE attachments ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1 CHECK (pipeline_version>0);
ALTER TABLE attachments ADD COLUMN staging_key text;
ALTER TABLE attachments ADD COLUMN variants jsonb NOT NULL DEFAULT '[]';
ALTER TABLE attachments ADD COLUMN error_code text;
ALTER TABLE attachments ADD COLUMN accepted_at timestamptz NOT NULL DEFAULT clock_timestamp();
CREATE TABLE jobs (
id uuid PRIMARY KEY,
attachment_id uuid NOT NULL REFERENCES attachments(id),
pipeline_version integer NOT NULL CHECK (pipeline_version IN (1,2)),
status text NOT NULL DEFAULT 'queued' CHECK (status IN ('queued','processing','done','failed','superseded')),
attempts integer NOT NULL DEFAULT 0 CHECK (attempts>=0),
lease_token uuid,
lease_until timestamptz,
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
error_code text,
traceparent text,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
UNIQUE (attachment_id,pipeline_version)
);
CREATE INDEX jobs_due ON jobs(status,next_attempt_at);
CREATE TABLE outbox (
job_id uuid PRIMARY KEY REFERENCES jobs(id),
published_at timestamptz,
publications integer NOT NULL DEFAULT 0
);
CREATE TABLE job_audit (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
job_id uuid NOT NULL REFERENCES jobs(id),
event text NOT NULL,
details jsonb NOT NULL DEFAULT '{}',
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE job_faults (
job_id uuid PRIMARY KEY REFERENCES jobs(id),
point text NOT NULL CHECK (point IN ('before_effect','after_effect','after_commit','transient','permanent')),
remaining integer NOT NULL CHECK (remaining>=0)
);
+1
View File
@@ -0,0 +1 @@
ALTER TABLE jobs ADD COLUMN IF NOT EXISTS tracestate text;
+6
View File
@@ -0,0 +1,6 @@
CREATE TABLE IF NOT EXISTS login_budget (
name text PRIMARY KEY,
window_start timestamptz NOT NULL,
used integer NOT NULL
);
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
+3
View File
@@ -0,0 +1,3 @@
-r requirements.txt
cryptography==50.0.2
PyYAML==6.0.3
+43
View File
@@ -0,0 +1,43 @@
annotated-doc==0.0.5
annotated-types==0.8.0
anyio==4.15.1
argon2-cffi==25.1.0
argon2-cffi-bindings==26.1.0
boto3==1.43.114
botocore==1.43.114
certifi==2026.10.23
cffi==2.1.1
charset-normalizer==3.5.2
click==8.5.0
fastapi==0.143.2
googleapis-common-protos==1.75.5
h11==0.16.0
idna==3.20
jmespath==1.1.0
opentelemetry-api==1.45.1
opentelemetry-exporter-http-transport==0.66b1
opentelemetry-exporter-otlp-common==0.66b1
opentelemetry-exporter-otlp-proto-common==1.45.1
opentelemetry-exporter-otlp-proto-http==1.45.1
opentelemetry-proto==1.45.1
opentelemetry-sdk==1.45.1
opentelemetry-semantic-conventions==0.66b1
pillow==13.0.0
protobuf==7.36.2
psycopg==3.3.6
psycopg-binary==3.3.6
psycopg-pool==3.3.3
pycparser==3.11
pydantic==2.14.1
pydantic_core==2.50.1
python-dateutil==2.9.0.post0
redis==8.1.0
requests==2.34.2
s3transfer==0.19.2
six==1.17.0
starlette==1.8.0
typing-inspection==0.4.4
typing_extensions==4.16.0
tzdata==2026.5
urllib3==2.8.0
uvicorn==0.54.0
+60
View File
@@ -0,0 +1,60 @@
"""Explicit operator init/unseal/provision. Shares never enter a service mount."""
import argparse
import json
from pathlib import Path
import ssl
import time
import urllib.request
from urllib.error import HTTPError, URLError
ROOT=Path(__file__).resolve().parents[1]
def request(path,body=None,token=None,method=None):
ctx=ssl.create_default_context(cafile=str(ROOT/'secrets/trust/ca.crt'))
ctx.load_cert_chain(ROOT/'secrets/identities/operator/cert.pem',ROOT/'secrets/identities/operator/key.pem')
headers={'Content-Type':'application/json'}
if token:headers['X-Vault-Token']=token
req=urllib.request.Request('https://127.0.0.1:8200/v1/'+path,data=json.dumps(body).encode() if body is not None else None,headers=headers,method=method)
with urllib.request.urlopen(req,context=ctx,timeout=15) as r:
raw=r.read();return json.loads(raw) if raw else {}
def main():
p=argparse.ArgumentParser();p.add_argument('action',choices=['init','unseal','provision','seal','publish-crl']);args=p.parse_args()
file=ROOT/'secrets/operator/bao-init.json'
if args.action=='init':
if request('sys/init')['initialized']:raise SystemExit('Already initialized; preserve existing operator shares')
result=request('sys/init',{'secret_shares':3,'secret_threshold':2})
with file.open('x',encoding='utf-8') as f:json.dump(result,f)
file.chmod(0o600)
print('Initialized; three shares stored in operator-only local file. Unseal explicitly.');return
operator=json.loads(file.read_text(encoding='utf-8'));token=operator['root_token']
if args.action=='unseal':
deadline=time.monotonic()+20
while True:
try:request('sys/seal-status');break
except (URLError,OSError):
if time.monotonic()>=deadline:raise
time.sleep(.5)
for share in operator['keys'][:2]:request('sys/unseal',{'key':share})
elif args.action=='seal':request('sys/seal',{},token)
elif args.action=='publish-crl':request('auth/cert/crls/operator',{'crl':(ROOT/'secrets/trust/revocations.pem').read_text()},token)
elif args.action=='provision':
mounts=request('sys/mounts',token=token)
if 'secret/' not in mounts:request('sys/mounts/secret',{'type':'kv','options':{'version':'2'}},token)
auth=request('sys/auth',token=token)
if 'cert/' not in auth:request('sys/auth/cert',{'type':'cert'},token)
config=json.loads((ROOT/'secrets/operator/config.json').read_text())
for role,data in config['service_credentials'].items():
policy=f'path "secret/data/{role}" {{ capabilities = ["read"] }}\n'
request('sys/policies/acl/'+role,{'policy':policy},token)
request('auth/cert/certs/'+role,{'certificate':(ROOT/'secrets/trust/ca.crt').read_text(),
'allowed_dns_sans':[role],'token_policies':[role],'token_ttl':120,'token_max_ttl':120,'token_no_default_policy':False},token)
request('secret/data/'+role,{'data':data},token)
# Audit is declarative in openbao.hcl; API audit creation remains disabled.
assert 'operator/' in request('sys/audit',token=token), 'Audit device missing'
print('Operator action completed; no root token/unseal shares printed')
if __name__=='__main__':main()
+81
View File
@@ -0,0 +1,81 @@
"""Generate isolated local credentials. Operator files are never app mounts."""
import json
from pathlib import Path
import secrets
import csv
import os
import subprocess
from datetime import datetime,timedelta,timezone
import pki
ROOT=Path(__file__).resolve().parents[1]
ROLES=['api1','api2','worker','dispatcher','reconciler']
def protect_local_files():
directory=ROOT/'secrets';directory.mkdir(exist_ok=True)
directory.chmod(0o700)
if os.name=='nt':
sid=next(csv.reader(subprocess.check_output(['whoami','/user','/fo','csv','/nh'],text=True).splitlines()))[1]
for path in (directory,ROOT/'.env',ROOT/'.env.storage-admin'):
if not path.exists():continue
flags='(OI)(CI)' if path.is_dir() else ''
result=subprocess.run(['icacls',str(path),'/inheritance:r','/grant:r',
'*'+sid+':'+flags+'F','*S-1-5-18:'+flags+'F','*S-1-5-32-544:'+flags+'F'],capture_output=True)
if result.returncode:raise RuntimeError('Could not restrict local secret ACLs')
def main():
protect_local_files()
pki.init()
pki.crl()
for role in ROLES+['db','s3','broker','cache','otel-collector','prometheus','loki','tempo','grafana','lb','openbao','operator','outsider']:
cert_path=ROOT/'secrets/identities'/role/'cert.pem'
if (not cert_path.exists() or pki.x509.load_pem_x509_certificate(cert_path.read_bytes()).not_valid_after_utc
< datetime.now(timezone.utc)+timedelta(hours=2)):
pki.issue(role)
target=ROOT/'secrets/operator/config.json'
if target.exists():
print('Existing local credentials preserved');return
password=secrets.token_hex(24)
db={role:secrets.token_hex(24) for role in ROLES}
broker={role:secrets.token_hex(24) for role in ('worker','dispatcher')}
cache={role:secrets.token_hex(24) for role in ('api1','api2')}
s3={role:(secrets.token_hex(12),secrets.token_hex(32)) for role in ROLES+['admin'] if role!='dispatcher'}
cursor=secrets.token_hex(32)
credentials={}
for role in ROLES:
credentials[role]={'DATABASE_URL':f'postgresql://{role}:{db[role]}@127.0.0.1:5432/messenger',
'CURSOR_SECRET':cursor,
'OTEL_EXPORTER_OTLP_ENDPOINT':'http://127.0.0.1:4318','TELEMETRY_ENABLED':'1','SESSION_TTL_SECONDS':'3600'}
if role in s3:
credentials[role].update({'S3_ENDPOINT':'http://127.0.0.1:8333','S3_BUCKET':'messenger',
'S3_ACCESS_KEY':s3[role][0],'S3_SECRET_KEY':s3[role][1]})
if role in cache: credentials[role]['REDIS_URL']=f'redis://{role}:{cache[role]}@127.0.0.1:6379/0'
if role in broker: credentials[role]['BROKER_URL']=f'redis://{role}:{broker[role]}@127.0.0.1:6380/0'
identities=[]
for role,(key,secret) in s3.items():
if role=='admin':actions=['Admin','Read','Write','List','Tagging']
elif role.startswith('api'):actions=['Read:messenger/originals/*','Read:messenger/variants/*','Write:messenger/staging/*']
else:
actions=['Read:messenger/staging/*','Read:messenger/originals/*','Read:messenger/variants/*','Write:messenger/staging/*','Write:messenger/originals/*','Write:messenger/variants/*']
if role=='reconciler':actions.append('List:messenger')
identities.append({'name':role,'credentials':[{'accessKey':key,'secretKey':secret}],'actions':actions})
files={'.env':'POSTGRES_PASSWORD='+password+'\n',
'.env.storage-admin':f'S3_ENDPOINT=http://127.0.0.1:8333\nS3_BUCKET=messenger\nS3_ACCESS_KEY={s3["admin"][0]}\nS3_SECRET_KEY={s3["admin"][1]}\n',
'secrets/s3.json':json.dumps({'identities':identities},indent=2),
'secrets/grafana-password':secrets.token_urlsafe(24)+'\n',
'secrets/cache.conf':'bind 127.0.0.1\nprotected-mode yes\nmaxmemory 32mb\nmaxmemory-policy allkeys-lru\nsave ""\nuser default off\n'+''.join(f'user {r} on >{v} ~user:* -@all +get +set +ping +hello +client|setinfo\n' for r,v in cache.items()),
'secrets/broker.conf':'bind 127.0.0.1\nprotected-mode yes\nappendonly yes\nappendfsync always\ndir /data\nsave ""\nmaxmemory 64mb\nmaxmemory-policy noeviction\nuser default off\n'+
f'user dispatcher on >{broker["dispatcher"]} ~attachment-jobs -@all +xadd +xgroup +ping +hello +client|setinfo\n'+
f'user worker on >{broker["worker"]} ~attachment-jobs -@all +xgroup +xautoclaim +xreadgroup +xack +xdel +multi +exec +ping +hello +client|setinfo\n',
'secrets/operator/config.json':json.dumps({'database_passwords':db,'service_credentials':credentials},indent=2)}
for name,value in files.items():
p=ROOT/name;p.parent.mkdir(parents=True,exist_ok=True)
with p.open('x',encoding='utf-8',newline='\n') as f:f.write(value)
p.chmod(0o600)
protect_local_files()
print('Generated separate identities and local secrets; values not printed')
if __name__=='__main__':main()
+37
View File
@@ -0,0 +1,37 @@
"""Generate the committed, provisioned dashboard from explicit PromQL."""
import json
from pathlib import Path
panels=[]
def panel(title,queries,unit='short'):
i=len(panels)
panels.append({'id':i+1,'type':'timeseries','title':title,'datasource':{'type':'prometheus','uid':'prometheus'},
'gridPos':{'x':(i%2)*12,'y':(i//2)*8,'w':12,'h':8},
'targets':[{'refId':chr(65+j),'expr':expr,'legendFormat':label} for j,(label,expr) in enumerate(queries)],
'fieldConfig':{'defaults':{'unit':unit,'min':0,**({'max':1} if unit=='percentunit' else {})},'overrides':[]},'options':{'legend':{'displayMode':'list','placement':'bottom'}}})
panel('API requests / second',[('{{method}} {{route}}','sum by(method,route)(rate(http_server_requests_total[1m]))')],'reqps')
panel('API 5xx / all business requests',[('5xx fraction','(sum(rate(http_server_requests_total{status_class="5xx"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)')],'percentunit')
panel('API latency: aggregate buckets across replicas',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(http_server_duration_seconds_bucket[1m])))') for p in (50,95,99)],'s')
panel('Jobs / second by outcome',[('{{outcome}}','sum by(outcome)(rate(jobs_processed_total[1m]))')],'ops')
panel('Worker processing latency',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))') for p in (50,95,99)],'s')
panel('Acceptance to ready latency',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))') for p in (50,95,99)],'s')
panel('Unfinished durable jobs',[('queue depth','sum(queue_depth)')])
panel('Oldest unfinished job age',[('oldest','max(queue_oldest_age_seconds)')],'s')
panel('DB pool',[('used {{process_instance}}','db_pool_used'),('waiting {{process_instance}}','db_pool_waiting')])
panel('Worker saturation',[('busy / capacity','sum(worker_busy) / clamp_min(sum(worker_capacity),1)')],'percentunit')
panel('Current process RSS',[('{{service}} {{process_instance}}','process_rss_bytes')],'bytes')
panel('Process CPU seconds / wall second',[('{{service}} {{process_instance}}','clamp_min(deriv(process_cpu_time_seconds[1m]),0)')],'short')
panel('API availability burn rate; budget 0.5%',[(f'{w}s',f'(sum(rate(http_server_requests_total{{status_class="5xx"}}[{w}s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[{w}s])),0.001) / 0.005') for w in (20,60)])
panel('Ready <=30s burn rate; budget 1%',[(f'{w}s eligible cohort',f'sum(attachment_slo_bad{{window="{w}"}}) / clamp_min(sum(attachment_slo_eligible{{window="{w}"}}),1) / 0.01') for w in (60,300)])
panel('Collector exporter queue / capacity',[('{{exporter}} size','otelcol_exporter_queue_size'),('{{exporter}} capacity','otelcol_exporter_queue_capacity')])
panel('Collector failed exports',[('{{exporter}} spans','rate(otelcol_exporter_send_failed_spans_total[1m])'),('{{exporter}} logs','rate(otelcol_exporter_send_failed_log_records_total[1m])')],'ops')
panels.append({'id':99,'type':'logs','title':'Structured application logs; TraceID links to Tempo','datasource':{'type':'loki','uid':'loki'},
'gridPos':{'x':0,'y':64,'w':24,'h':10},'targets':[{'refId':'A','expr':'{service_name=~"messenger-(api|worker|dispatcher)"}'}],
'options':{'showTime':True,'showLabels':False,'wrapLogMessage':True,'sortOrder':'Descending'}})
doc={'uid':'mtusi-messenger','title':'MTUSI Messenger - signals and SLO','schemaVersion':39,'version':1,
'editable':False,'timezone':'browser','time':{'from':'now-15m','to':'now'},'refresh':'15s','tags':['mtusi','lab6'],
'panels':panels,'annotations':{'list':[]},'templating':{'list':[]}}
path=Path(__file__).resolve().parents[1]/'infra/grafana/dashboards/messenger.json'
path.parent.mkdir(parents=True,exist_ok=True)
path.write_text(json.dumps(doc,ensure_ascii=False,indent=2)+'\n',encoding='utf-8')
+47
View File
@@ -0,0 +1,47 @@
"""Operator migrations/grants over the container-local PostgreSQL Unix socket.
SQL containing passwords travels on stdin, not argv/logs; no admin DB credential
is made available to an application. Existing migration hashes must match.
"""
import hashlib
import json
from pathlib import Path
import subprocess
ROOT=Path(__file__).resolve().parents[1]
def sql(query):
result=subprocess.run(['docker','compose','exec','-T','db','psql','-X','-qAt',
'-v','ON_ERROR_STOP=1','-U','messenger','-d','messenger'],input=query,
text=True,encoding='utf-8',capture_output=True,cwd=ROOT)
if result.returncode:
# Server error text may echo a credential-bearing statement.
raise RuntimeError('Operator SQL failed; no query or credentials printed')
return result.stdout.strip()
def main():
sql('CREATE TABLE IF NOT EXISTS schema_migrations (name text PRIMARY KEY, sha256 text NOT NULL, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())')
for file in sorted((ROOT/'migrations').glob('*.sql')):
digest=hashlib.sha256(file.read_bytes()).hexdigest()
old=sql(f"SELECT sha256 FROM schema_migrations WHERE name='{file.name}'")
if old:
assert old==digest, 'Applied migration changed: '+file.name
else:
sql('BEGIN;\n'+file.read_text()+f"\nINSERT INTO schema_migrations(name,sha256) VALUES ('{file.name}','{digest}'); COMMIT;")
print('Migration verified:',file.name)
passwords=json.loads((ROOT/'secrets/operator/config.json').read_text())['database_passwords']
for role,password in passwords.items():
assert role in ('api1','api2','worker','dispatcher','reconciler') and all(c in '0123456789abcdef' for c in password)
if not sql(f"SELECT 1 FROM pg_roles WHERE rolname='{role}'"):sql(f'CREATE ROLE {role} LOGIN')
sql(f"ALTER ROLE {role} PASSWORD '{password}' NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION; GRANT CONNECT ON DATABASE messenger TO {role}; GRANT USAGE ON SCHEMA public TO {role}; REVOKE ALL ON ALL TABLES IN SCHEMA public FROM {role}; REVOKE ALL ON ALL SEQUENCES IN SCHEMA public FROM {role};")
sql('REVOKE ALL ON DATABASE messenger FROM PUBLIC; REVOKE CREATE ON SCHEMA public FROM PUBLIC;')
sql('GRANT SELECT,INSERT,UPDATE,DELETE ON users,sessions,chats,members,messages,idempotency,attachments,message_attachments,jobs,outbox,login_budget TO api1,api2;')
sql('GRANT SELECT,UPDATE ON jobs,attachments,outbox,job_faults TO worker; GRANT INSERT ON job_audit TO worker; GRANT USAGE ON SEQUENCE job_audit_id_seq TO worker;')
sql('GRANT SELECT,UPDATE ON jobs,attachments,outbox TO dispatcher; GRANT INSERT ON job_audit TO dispatcher; GRANT USAGE ON SEQUENCE job_audit_id_seq TO dispatcher;')
sql('GRANT SELECT,UPDATE ON attachments TO reconciler;')
print('Scoped database roles applied')
if __name__=='__main__':main()
+84
View File
@@ -0,0 +1,84 @@
"""Operator-owned local PKI. CA private material is never mounted into services."""
from datetime import datetime, timedelta, timezone
import ipaddress
import json
from pathlib import Path
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.x509.oid import NameOID, ExtendedKeyUsageOID
ROOT=Path(__file__).resolve().parents[1]
PRIVATE=ROOT/'secrets/operator'
TRUST=ROOT/'secrets/trust'
def write(path,data):
path.parent.mkdir(parents=True,exist_ok=True)
path.write_bytes(data)
path.chmod(0o600 if 'key' in path.name or path.name == 'combined.pem' or path.parent == PRIVATE else 0o644)
def key_bytes(key):
return key.private_bytes(serialization.Encoding.PEM,serialization.PrivateFormat.PKCS8,serialization.NoEncryption())
def ca():
return (serialization.load_pem_private_key((PRIVATE/'ca.key').read_bytes(),None),
x509.load_pem_x509_certificate((TRUST/'ca.crt').read_bytes()))
def init():
if (PRIVATE/'ca.key').exists(): return
key=ec.generate_private_key(ec.SECP256R1()); now=datetime.now(timezone.utc)
name=x509.Name([x509.NameAttribute(NameOID.COMMON_NAME,'MTUSI local operator CA')])
cert=(x509.CertificateBuilder().subject_name(name).issuer_name(name).public_key(key.public_key())
.serial_number(x509.random_serial_number()).not_valid_before(now-timedelta(minutes=1))
.not_valid_after(now+timedelta(days=365)).add_extension(x509.BasicConstraints(ca=True,path_length=0),True)
.add_extension(x509.KeyUsage(False,False,False,False,False,True,True,None,None),True)
.add_extension(x509.SubjectKeyIdentifier.from_public_key(key.public_key()),False).sign(key,hashes.SHA256()))
write(PRIVATE/'ca.key',key_bytes(key));write(TRUST/'ca.crt',cert.public_bytes(serialization.Encoding.PEM))
write(PRIVATE/'revoked.json',b'[]'); crl()
def issue(role,hours=24,extra=()):
key,issuer=ca(); leaf=ec.generate_private_key(ec.SECP256R1()); now=datetime.now(timezone.utc)
sans=[x509.DNSName(role),*[x509.DNSName(n) for n in extra]]
if role in ('lb','openbao','grafana'): sans += [x509.DNSName('localhost'),x509.IPAddress(ipaddress.ip_address('127.0.0.1'))]
cert=(x509.CertificateBuilder().subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME,role)]))
.issuer_name(issuer.subject).public_key(leaf.public_key()).serial_number(x509.random_serial_number())
.not_valid_before(now-timedelta(minutes=1)).not_valid_after(now+timedelta(hours=hours))
.add_extension(x509.BasicConstraints(ca=False,path_length=None),True)
.add_extension(x509.SubjectAlternativeName(sans),False)
.add_extension(x509.ExtendedKeyUsage([ExtendedKeyUsageOID.SERVER_AUTH,ExtendedKeyUsageOID.CLIENT_AUTH]),False)
.add_extension(x509.AuthorityKeyIdentifier.from_issuer_public_key(key.public_key()),False)
.sign(key,hashes.SHA256()))
folder=ROOT/'secrets/identities'/role
write(folder/'key.pem',key_bytes(leaf));write(folder/'cert.pem',cert.public_bytes(serialization.Encoding.PEM))
write(folder/'combined.pem',cert.public_bytes(serialization.Encoding.PEM)+key_bytes(leaf))
return cert
def crl():
key,issuer=ca();now=datetime.now(timezone.utc)
doc=x509.CertificateRevocationListBuilder().issuer_name(issuer.subject).last_update(now-timedelta(minutes=1)).next_update(now+timedelta(days=7))
for serial in json.loads((PRIVATE/'revoked.json').read_text()):
doc=doc.add_revoked_certificate(x509.RevokedCertificateBuilder().serial_number(serial).revocation_date(now).build())
pem=doc.sign(key,hashes.SHA256()).public_bytes(serialization.Encoding.PEM)
write(TRUST/'revocations.pem',pem);write(TRUST/'bundle.pem',(TRUST/'ca.crt').read_bytes()+pem)
def revoke(role):
cert=x509.load_pem_x509_certificate((ROOT/'secrets/identities'/role/'cert.pem').read_bytes())
path=PRIVATE/'revoked.json';values=json.loads(path.read_text())
if cert.serial_number not in values: values.append(cert.serial_number)
write(path,json.dumps(values).encode());crl()
if __name__=='__main__':
import argparse
p=argparse.ArgumentParser();p.add_argument('action',choices=['issue','revoke']);p.add_argument('role');p.add_argument('--hours',type=float,default=24)
a=p.parse_args();init()
if a.action=='issue': issue(a.role,a.hours)
else: revoke(a.role)
print('PKI operation complete; no key material printed')
+3
View File
@@ -0,0 +1,3 @@
@echo off
if not defined LAB7_PYTHON set "LAB7_PYTHON=python"
"%LAB7_PYTHON%" "%~dp0restart.py"
+17
View File
@@ -0,0 +1,17 @@
"""Restart every running replica and dependency, preserving durable volumes."""
import json
import subprocess
import sys
from stack import ROOT, docker, state
before = state()
docker('compose', 'stop', '--timeout', '5')
docker('compose', 'start')
subprocess.run([sys.executable,str(ROOT/'scripts/bao_operator.py'),'unseal'],check=True)
after = state()
assert set(before) == set(after)
assert all(before[c]['state']['StartedAt'] != after[c]['state']['StartedAt'] for c in before)
(ROOT / 'evidence').mkdir(exist_ok=True)
(ROOT / 'evidence/restart-processes.json').write_text(
json.dumps({'before': before, 'after': after, 'volumes_removed': False}, indent=2) + '\n',
encoding='utf-8')
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
python3 scripts/restart.py
+39
View File
@@ -0,0 +1,39 @@
"""Scan exported image archives without mounting the Docker socket or host secrets."""
import argparse
import collections
from datetime import datetime,timezone
import hashlib
import json
from pathlib import Path
import subprocess
import tempfile
import yaml
ROOT=Path(__file__).resolve().parents[1]
TRIVY='aquasec/trivy@sha256:790fd66d1697395d8693d1acaa1ea702406a20f109b6fc7bac8db623eca6c06c'
def main():
parser=argparse.ArgumentParser();parser.add_argument('services',nargs='*',default=['api1','openbao','s3','lb']);args=parser.parse_args()
compose=yaml.safe_load((ROOT/'compose.yaml').read_text())
summary={'at_UTC':datetime.now(timezone.utc).isoformat(),'scanner':TRIVY,'scope':args.services,'images':[]}
with tempfile.TemporaryDirectory(prefix='lab7-image-scan-') as tmp:
for service in args.services:
image=compose['services'][service]['image']
image_id=subprocess.check_output(['docker','image','inspect',image,'--format','{{.Id}}'],text=True).strip()
archive=Path(tmp)/'image.tar';report=Path(tmp)/'report.json'
subprocess.run(['docker','save','--output',str(archive),image],check=True)
subprocess.run(['docker','run','--rm','--cpus','0.5','--memory','768m','--pids-limit','128',
'-v',tmp+':/scan','-v','mtusi-trivy-cache:/root/.cache/trivy',TRIVY,'image','--input','/scan/image.tar',
'--parallel','1','--scanners','vuln','--format','json','--output','/scan/report.json'],check=True)
data=json.loads(report.read_text());findings=[]
for result in data.get('Results',[]):
for v in result.get('Vulnerabilities',[]):
findings.append({'target':result.get('Target'),**{k:v.get(k) for k in ('VulnerabilityID','PkgName','InstalledVersion','FixedVersion','Severity','Title','PrimaryURL')}})
summary['images'].append({'service':service,'image':image,'image_id':image_id,'report_sha256':hashlib.sha256(report.read_bytes()).hexdigest(),
'counts':dict(collections.Counter(v['Severity'] for v in findings)),'findings':findings})
(ROOT/'evidence/image-audit.json').write_text(json.dumps(summary,indent=2)+'\n')
print(service,summary['images'][-1]['counts'],flush=True)
if __name__=='__main__':main()
+35
View File
@@ -0,0 +1,35 @@
"""Shared bounded Docker helpers for reproducible, scoped failure experiments."""
import json
from pathlib import Path
import subprocess
import time
ROOT=Path(__file__).resolve().parents[1]
def docker(*args):
return subprocess.check_output(['docker',*args],cwd=ROOT,text=True,encoding='utf-8',stderr=subprocess.PIPE).strip()
def replicas(): return docker('compose','ps','-q','api1','api2').splitlines()
def ready_replicas(count=2,timeout=60):
deadline=time.monotonic()+timeout
while time.monotonic()<deadline:
ids=replicas()
if len(ids)==count:
states=[json.loads(docker('inspect','--format','{{json .State}}',cid)) for cid in ids]
if all(s['Running'] and s.get('Health',{}).get('Status')=='healthy' for s in states): return
time.sleep(.5)
raise AssertionError('Actual replica health did not recover within timeout')
def instance(cid):
return docker('exec',cid,'python','-c',"import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/health/live',timeout=2).headers['X-Instance-Id'])")
def state():
result={}
for service in ('api1','api2','db','cache','lb','s3','reconciler','broker','worker','dispatcher','openbao','otel-collector','prometheus','loki','tempo','grafana'):
for cid in docker('compose','ps','-q',service).splitlines():
result[cid]={'service':service,'state':json.loads(docker('inspect','--format','{{json .State}}',cid))}
return result
def ready(client,timeout=60):
deadline=time.monotonic()+timeout
while time.monotonic()<deadline:
try:
if client.get('/health/ready').status_code==200: return
except Exception: pass
time.sleep(.2)
raise AssertionError('Readiness timeout')
+3
View File
@@ -0,0 +1,3 @@
@echo off
if not defined LAB7_PYTHON set "LAB7_PYTHON=python"
"%LAB7_PYTHON%" "%~dp0stop-one.py"
+15
View File
@@ -0,0 +1,15 @@
"""Stop the real process selected by X-Instance-Id; do not stop any dependency."""
import json
import os
from concurrent.futures import ThreadPoolExecutor
from stack import ROOT, docker, instance, replicas
target=os.environ['TARGET_INSTANCE_ID']
before=replicas()
with ThreadPoolExecutor(max_workers=2) as pool:
matches=[cid for cid,alias in zip(before,pool.map(instance,before)) if alias==target]
assert len(matches)==1,'Target does not uniquely identify a running API'
# Immediate process crash, without adding a graceful-drain delay to the fault.
docker('kill','--signal','KILL',matches[0])
assert docker('inspect','--format','{{.State.Running}}',matches[0])=='false'
(ROOT/'evidence').mkdir(exist_ok=True)
(ROOT/'evidence/stop-one.json').write_text(json.dumps({'target_instance':target,'stopped_container':matches[0],'before_containers':before,'after_running_containers':replicas()},indent=2)+'\n',encoding='utf-8')
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec python3 scripts/stop-one.py
+19
View File
@@ -0,0 +1,19 @@
"""Negative-test leaf identities; never used by a runtime service."""
from datetime import datetime,timedelta,timezone
from cryptography import x509
from cryptography.hazmat.primitives import hashes,serialization
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.x509.oid import NameOID
import pki
roles=['lb','api1','api2','worker','dispatcher','reconciler','operator','otel-collector','prometheus','grafana']
pki.issue('expired',hours=-.001,extra=roles)
pki.issue('revoked',extra=roles);pki.revoke('revoked')
key=ec.generate_private_key(ec.SECP256R1());now=datetime.now(timezone.utc)
name=x509.Name([x509.NameAttribute(NameOID.COMMON_NAME,'untrusted-test')])
cert=(x509.CertificateBuilder().subject_name(name).issuer_name(name).public_key(key.public_key())
.serial_number(x509.random_serial_number()).not_valid_before(now-timedelta(minutes=1))
.not_valid_after(now+timedelta(hours=1)).add_extension(x509.SubjectAlternativeName([x509.DNSName(n) for n in roles]),False).sign(key,hashes.SHA256()))
pki.write(pki.ROOT/'secrets/identities/alien/key.pem',pki.key_bytes(key))
pki.write(pki.ROOT/'secrets/identities/alien/cert.pem',cert.public_bytes(serialization.Encoding.PEM))
print('Negative test identities generated')

Some files were not shown because too many files have changed in this diff Show More