ЛР 7 — БВТ2301 — Парамонов — на 5 #14
@@ -0,0 +1,3 @@
|
||||
* text=auto eol=lf
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
# Official Docker Python image; immutable digest.
|
||||
FROM python@sha256:5e2bdb1b0d542d5039cffb8181c65930093949194a65c365dfb1b0fe13a4fa5b
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONUTF8=1 \
|
||||
APP_HOST=0.0.0.0 APP_PORT=8080
|
||||
WORKDIR /app
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
zlib1g=1:1.3.dfsg+really1.3.1-1+deb13u1 && rm -rf /var/lib/apt/lists/*
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir pip==26.3 && pip install --no-cache-dir -r requirements.txt && pip uninstall -y setuptools && \
|
||||
groupadd --gid 10001 messenger && useradd --uid 10001 --gid 10001 --no-create-home messenger
|
||||
COPY src ./src
|
||||
COPY migrations ./migrations
|
||||
USER 10001:10001
|
||||
EXPOSE 8080
|
||||
STOPSIGNAL SIGTERM
|
||||
HEALTHCHECK --interval=5s --timeout=6s --start-period=60s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=5)"
|
||||
CMD ["python", "-m", "src"]
|
||||
@@ -1,37 +1,233 @@
|
||||
# Отчёт — лабораторная 7
|
||||
# ЛР7 — Парамонов Глеб, БВТ2301
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
Целевая оценка **5**, трек 1: **компрометация worker**. Ветка
|
||||
`solution/bvt2301-paramonov`. Перенесена собственная реализация ЛР6: две API,
|
||||
PostgreSQL, profile cache, приватный S3, transactional outbox, Valkey Streams,
|
||||
worker с versioned image pipeline, reconciler, OTel/Grafana. Новый внешний API —
|
||||
HTTPS. README задания, business contract и public tests не ослаблены; заполнен
|
||||
предусмотренный заданием `contracts/trust-matrix.md`.
|
||||
|
||||
Код, исследование и оформление подготовлены с помощью OpenAI Codex. Результаты
|
||||
в `evidence/` получены на реальном локальном стенде, а не сгенерированы как
|
||||
пример. Это учебная реализация в заявленной модели угроз, не production seal.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Git, Docker Engine/Compose v2, Python 3.12+ и сеть для официальных images/
|
||||
PyPI. Проверено на Windows10, Docker Desktop4.39 / Engine28 / Compose2.33, WSL2
|
||||
4CPU/3GiB RAM/4GiB swap, i5-11400H/8GiB host RAM. Версии образов закреплены digest
|
||||
в `compose.yaml`/`Dockerfile`; зависимости — точными версиями в requirements.
|
||||
Предыдущие стенды остановить: административный Grafana port3000 общий.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
python -m venv .venv
|
||||
# Windows: вместо .venv/bin/python используйте .venv/Scripts/python.exe
|
||||
.venv/bin/python -m pip install -r requirements-dev.txt -r requirements-test.txt
|
||||
.venv/bin/python -X utf8 scripts/up.py
|
||||
.venv/bin/python -X utf8 scripts/verify_public.py
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab6, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
`up.py` — явная операторская команда. Она создаёт локальные секреты, строит image,
|
||||
согласованно останавливает старые процессы без удаления volumes, запускает Bao
|
||||
и хранилища, выполняет init/unseal, migrations/GRANT, policy/KV provisioning,
|
||||
создаёт private bucket, запускает остальное и ждёт verified HTTPS readiness.
|
||||
Повтор сохраняет пароли/CA/data и проверяет hashes применённых migrations.
|
||||
Leaf certificates с остатком <2h обновляются оператором. Это cold-start
|
||||
workflow, бесшовный rollout не заявляется. На этой машине dependencies для
|
||||
работы выполнялись также из соседнего существующего venv; для проверяющего
|
||||
полностью достаточно собственного venv этого репозитория.
|
||||
|
||||
## Доказательства критериев
|
||||
После запуска: API https://localhost:8443, Grafana https://localhost:3000
|
||||
(login admin, пароль только в локальном `secrets/grafana-password`), Bao API
|
||||
https://localhost:8200 (обязательный operator client cert). Не отключайте
|
||||
проверку CA: для Python/public tests `--ca-file secrets/trust/ca.crt`. Для
|
||||
браузера сначала установите публичный учебный CA вручную, если нужен UI.
|
||||
HTTP на этих портах не является fallback.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Эквивалентные операции по шагам приведены в `scripts/up.py`. Обычный restart
|
||||
API не требует unseal. После перезапуска Bao:
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
```sh
|
||||
.venv/bin/python scripts/bao_operator.py unseal
|
||||
```
|
||||
|
||||
## Эксперименты и отказы
|
||||
Root token и 3 unseal shares (threshold2) находятся только в
|
||||
`secrets/operator/bao-init.json`; CA key — рядом в `ca.key`. Их не передают в
|
||||
PR/чат, не монтируют в runtime и не печатают. Windows bootstrap убирает inherited
|
||||
ACL и оставляет owner/SYSTEM/Administrators; POSIX secrets0700/files0600.
|
||||
Все доли на одном защищённом компьютере — ограничение учебного bootstrap,
|
||||
не production secret custody. Резервные копии этих файлов должен хранить
|
||||
оператор отдельно от data volume; потеря shares делает sealed data недоступными.
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Остановка с сохранением данных: `docker compose down`. Осознанное удаление
|
||||
**всех учебных данных только этого проекта**: `docker compose down --volumes`.
|
||||
После такого сброса следует создать новый clone/workdir и новые secrets;
|
||||
старый bao-init не соответствует новому пустому raft volume. Команда удаления
|
||||
не используется тестами. Restart hook делает stop/start в dependency order,
|
||||
потому что приложение и его TLS sidecar разделяют network namespace.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
## Архитектура и эксплуатация
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
Клиент → TLS LB → mTLS API1/API2 → mTLS DB/S3/cache/Collector. Worker и dispatcher
|
||||
используют отдельные broker identities. Collector экспортирует через mTLS в
|
||||
Loki/Tempo; Prometheus читает защищённые scrape endpoints; Grafana читает data
|
||||
sources через собственные mTLS tunnels. [Полная матрица](contracts/trust-matrix.md).
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
В backend namespace native listener только loopback. Прокси использует Python
|
||||
stdlib/OpenSSL, проверяет CA, hostname и SAN allowlist; TLS>=1.2, CRL, максимум
|
||||
64 активные/96 ожидающие задачи, 16KiB buffers, 60s connection lifetime. TLS
|
||||
context cache <=1s, поэтому срок прекращения существующей связи <=61s. TLS
|
||||
session tickets на server proxy выключены. Прокси не логирует payload. Native
|
||||
TLS применяется у LB, Bao и Grafana. CA private key в services отсутствует.
|
||||
|
||||
Secret supervisor использует native mTLS cert auth OpenBao, а не постоянный
|
||||
секрет «для получения секретов». Политика разрешает read только собственного
|
||||
KV path. Token TTL/maxTTL120s, re-auth20s; после чтения token revoke-self.
|
||||
Пароли БД/S3/Valkey — **static KV**, их реальная валидность заканчивается
|
||||
ротацией backend. Локальный budget120s ограничивает продолжение работы child
|
||||
при sealed/unavailable Bao, но не обезвреживает скопированный злоумышленником
|
||||
пароль. Смена KV перезапускает child и закрывает старые pooled connections.
|
||||
Crash child также обрабатывается supervisor без смены network namespace.
|
||||
|
||||
PostgreSQL — источник истины для пользователей, sessions, идемпотентности,
|
||||
attachments/jobs/outbox. Native host authentication на loopback — SCRAM,
|
||||
не trust. Только операторский Unix socket использует доверенный local role;
|
||||
этот socket другим контейнерам не расшарен. Worker не читает users/sessions/
|
||||
messages и не может DDL/role administration. API имеет business-table grants;
|
||||
worker/dispatcher — jobs/attachments/outbox/audit, reconciler — attachments.
|
||||
Dispatcher не получает S3 credentials. Valkey ACL ограничивает команды и key;
|
||||
S3 policy — bucket prefixes. Отдельные credentials каждой роли лежат в Bao.
|
||||
|
||||
Argon2id: 32MiB, t2, p1, случайный salt. При успешном входе старые параметры
|
||||
rehash с compare-and-swap. Session token — 256 случайных бит, в PostgreSQL
|
||||
только SHA-256; expiry1h/logout проверяются по БД. KDF concurrency2/API, общий
|
||||
limiter12 login attempts/10s и8 registration/10s. 429 содержит Retry-After10.
|
||||
Это глобальный учебный бюджет, не распределённая антибот-система по пользователям.
|
||||
|
||||
Runtime non-privileged, capabilities dropped, no-new-privileges, read_only root
|
||||
filesystem, RAM/CPU/pids limits, writable named volumes/tmpfs только где нужны.
|
||||
Worker1CPU/768MiB, render subprocess имеет отдельный timeout/лимиты. Init jobs
|
||||
кратковременно root с CHOWN/FOWNER/DAC_OVERRIDE для отдельных secret volumes;
|
||||
в runtime этих capabilities нет. Stateful write paths: PG/Seaweed/Valkey data,
|
||||
Bao raft+audit, Grafana/Prometheus/Loki/Tempo data. Наружу только loopback
|
||||
8443/8200/3000. `.env.example` — исходный перечень ролей, фактические параметры
|
||||
создаются `bootstrap.py`: `.env` только DB bootstrap, service env приходит из Bao,
|
||||
S3 admin имеет отдельный `.env.storage-admin` лишь в operator profile.
|
||||
|
||||
## Проверки и доказательства
|
||||
|
||||
Финальная проверка выполнена из отдельного clone `clean-lab7-final` с собственным
|
||||
venv, новой PKI/секретами и новыми volumes. Код соответствует commit `0bf1ff9`;
|
||||
финальные evidence/документация зафиксированы отдельным последующим коммитом. Проверка публичного
|
||||
набора 13.11.2026 прошла целиком: contract, OpenAPI, HTTPS smoke/security,
|
||||
persistence, три failover-запуска с двумя различными container IDs. После fault
|
||||
первые успешные записи восстановились за 3.390/2.953/2.610s. Первые два запуска
|
||||
выбрали одну реплику, третий — другую; все реальные процессы остановлены SIGKILL,
|
||||
без замены target теста. Перед следующим сценарием обе API и TLS sidecars
|
||||
восстанавливаются согласованно. Публичные tests не изменены.
|
||||
|
||||
Полный security suite завершился PASS 13.11.2026 в 10:00:37 UTC: все 11 этапов,
|
||||
включая TLS allow/deny matrix, Bao policies, worker permissions, API security,
|
||||
компрометацию worker, sealed Bao, telemetry, hardening и 4 renderer unit tests.
|
||||
При компрометации восстановлены 3/3 задания без изменения содержимого и образа;
|
||||
старые certificate/Bao auth/DB password отклонены; полный эксперимент 35.61s.
|
||||
При sealed Bao обе API child остановились через 123.86s (120s budget + grace),
|
||||
восстановление после unseal 20.44s. Argon2 hash+verify median 0.207s,
|
||||
maxRSS 53184KiB; 20 login за 4.578s дали 12×401 и 8×429 на двух API.
|
||||
Это результаты одного запуска на указанном стенде, а не гарантированные SLO.
|
||||
Сводка: [clean-clone.json](evidence/clean-clone.json),
|
||||
[public suite](evidence/public-suite.txt), [security suite](evidence/security-suite.txt).
|
||||
Image scan от 12.11 предшествует исправлению планирования TLS handshake;
|
||||
зависимости не изменились, но scan не заявляется для точного app image ID
|
||||
финального запуска (он отдельно записан в clean-clone.json).
|
||||
|
||||
```sh
|
||||
.venv/bin/python scripts/verify_public.py
|
||||
.venv/bin/python scripts/verify_security.py
|
||||
# Ниже отдельные сценарии для воспроизведения/диагностики, уже включённые в suite:
|
||||
.venv/bin/python scripts/test_identities.py
|
||||
.venv/bin/python scripts/bao_operator.py publish-crl
|
||||
docker compose run --rm security-check
|
||||
# В Windows вывод перенаправлять в UTF-8, пример: > evidence/tls-matrix.json
|
||||
docker compose run --rm security-check python /checks/bao_policy.py
|
||||
.venv/bin/python checks/security_api.py
|
||||
.venv/bin/python checks/compromised_worker.py
|
||||
.venv/bin/python checks/sealed_bao.py
|
||||
.venv/bin/python checks/telemetry_security.py
|
||||
.venv/bin/python checks/hardening.py
|
||||
docker compose exec -T worker python - < checks/worker_rights.py
|
||||
docker compose run --rm --no-deps -v "$PWD/checks:/checks:ro" api1 python -m unittest discover -s /checks -p test_render.py
|
||||
.venv/bin/python scripts/scan_images.py
|
||||
```
|
||||
|
||||
Последний `run api1` использует только unittest renderer, service secret supervisor
|
||||
не запускается (command переопределён). PowerShell stdin: `Get-Content
|
||||
checks/worker_rights.py -Raw | docker compose exec -T worker python -`.
|
||||
Перед повтором быстрых suites дайте limiter10s; не отключайте его ради PASS.
|
||||
|
||||
| Критерий | Сценарий / evidence |
|
||||
| --- | --- |
|
||||
| Неизменённый business contract/OpenAPI, HTTPS smoke | verify_public → check_contract.txt, validate_openapi.txt, smoke.txt, security.txt |
|
||||
| Сохранность всех данных/сессии/файла после restart | persistence.txt + restart-processes.json, реальные StartedAt всех сервисов |
|
||||
| Отказ обеих настоящих API-реплик | failover-*.txt и *-process.json, target из X-Instance-Id, distinct container IDs |
|
||||
| Вся внутренняя TLS матрица и отрицательные identities | tls-matrix.json: все allow edges и no-cert/foreign CA/wrong SAN/expired/revoked/plaintext |
|
||||
| Scoped cert auth/TTL/policy allow-deny | bao-policy.json |
|
||||
| Пароли/session hashes/rehash/shared DoS budget/ACL | security-api.json |
|
||||
| Реальный доступ с credentials worker | worker-rights.json |
|
||||
| Отзыв identity, ротация cert+DB password, сохранность задач | compromised-worker.json |
|
||||
| Sealed Bao и bounded expiry/recovery | sealed-bao.json |
|
||||
| Метрики, сквозной upload trace, correlated logs без canaries | telemetry-security.json |
|
||||
| Runtime isolation/permissions/mounts/ports | hardening.json |
|
||||
| Dependencies и образы | dependency-audit.json, image-audit.json, security/SCAN_TRIAGE.md |
|
||||
|
||||
Модель угроз связывает 11 конкретных сценариев с этими проверками:
|
||||
[security/THREAT_MODEL.md](security/THREAT_MODEL.md). Публичный smoke не используется
|
||||
как доказательство всех security claims.
|
||||
|
||||
## Исследование: компрометация worker
|
||||
|
||||
Гипотеза: раздельная identity + least privilege ограничивают последствия захвата
|
||||
worker обработкой вложений, а отзыв cert и связанного пароля останавливает новый
|
||||
доступ; transactional outbox сохраняет принятые задания до восстановления.
|
||||
|
||||
Методика `checks/compromised_worker.py`: синтетические пользователи/чат; сравнение
|
||||
operator session-read с denied worker session-read; revoke serial в proxy CRL и
|
||||
Bao cert-auth CRL; завершение старых worker DB sessions; 3 загрузки при отозванном
|
||||
worker; проверка queued; ALTER ROLE PASSWORD; новый24h leaf; secret-init; проверка
|
||||
старого пароля на новом TLS соединении; обновление KV; ожидание ready всех3,
|
||||
проверка distinct jobs и SHA-256. Образ worker до/после должен совпадать.
|
||||
|
||||
Первый завершённый запуск 04.11.2026: 3/3 задачи восстановлены без дублей, SHA-256
|
||||
сохранены, образ не менялся, полный эксперимент39.16s. Это время включает fault
|
||||
injection/provision и ожидание20s re-auth, не pure service recovery latency.
|
||||
В ходе разработки тест нашёл loopback trust в PostgreSQL; после исправления на
|
||||
SCRAM старый пароль действительно отклонён. Финальные повторные результаты
|
||||
смотрите в JSON evidence (они заменяют предыдущий запуск).
|
||||
|
||||
Отдельный sealed-Bao эксперимент04.11: обе API child прекратили слушать через
|
||||
106.58s от seal (остаток последнего120s budget), recovery23.44s после unseal.
|
||||
Измерение Argon2 в API container: hash+verify median0.209s, maxRSS53168KiB;
|
||||
20 конкурентных login завершились за2.031s, 12×401 и8×429, две instance identities.
|
||||
Основная KDF memory bound двух API: 4×32MiB=128MiB плюс overhead процесса.
|
||||
Эти числа — один контрольный запуск, не гарантия p99/capacity/устойчивости к DDoS.
|
||||
|
||||
## Ограничения и источники
|
||||
|
||||
Single host и single DB/S3/broker/Bao/LB: отказ хоста или этих зависимостей не
|
||||
переживается без перерыва. Есть отказ отдельной API-реплики, но не полноценная
|
||||
HA всех компонентов. Cert/CRL renewal операторский; безопасный CA rollover под
|
||||
нагрузкой не заявляется (выбран трек1). Учебная PKI и все shares на одном хосте.
|
||||
Worker сохраняет возможность повредить разрешённые prefixes/jobs; least privilege
|
||||
ограничивает ущерб, не делает скомпрометированный процесс безопасным. Audit Bao
|
||||
остаётся в operator volume, rotation/retention нужны для длительной эксплуатации.
|
||||
Сканирование не является доказательством отсутствия уязвимостей; применимость и
|
||||
оставшиеся findings разобраны отдельно. Более строгий deployment потребует
|
||||
динамических DB leases, isolated worker VM/seccomp, сетевых политик и отдельной
|
||||
защиты operator workstation.
|
||||
|
||||
Первичные источники: [OpenBao cert auth](https://openbao.org/docs/auth/cert/),
|
||||
[declarative audit](https://openbao.org/docs/configuration/audit/),
|
||||
[Python ssl](https://docs.python.org/3/library/ssl.html),
|
||||
[OWASP Password Storage](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html),
|
||||
[Pillow13 release](https://pillow.readthedocs.io/en/stable/releasenotes/13.0.0.html),
|
||||
[Trivy](https://trivy.dev/docs/latest/getting-started/installation/).
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
import json
|
||||
import ssl
|
||||
import urllib.request
|
||||
from urllib.error import HTTPError
|
||||
from tls_matrix import context
|
||||
|
||||
|
||||
def request(role,path,body=None,token=None):
|
||||
headers={'Content-Type':'application/json'}
|
||||
if token:headers['X-Vault-Token']=token
|
||||
req=urllib.request.Request('https://openbao:8200/v1/'+path,headers=headers,
|
||||
data=json.dumps(body).encode() if body is not None else None)
|
||||
try:
|
||||
with urllib.request.urlopen(req,context=context(role),timeout=5) as r:
|
||||
raw=r.read();return r.status,json.loads(raw) if raw else {}
|
||||
except HTTPError as e:return e.code,{}
|
||||
|
||||
|
||||
results=[]
|
||||
for role in ('api1','api2','worker','dispatcher','reconciler'):
|
||||
code,data=request(role,'auth/cert/login',{'name':role})
|
||||
assert code==200,(role,code)
|
||||
auth=data['auth'];token=auth['client_token']
|
||||
assert 0<auth['lease_duration']<=120 and 'root' not in auth['policies']
|
||||
try:
|
||||
assert request(role,'secret/data/'+role,token=token)[0]==200
|
||||
other='worker' if role!='worker' else 'api1'
|
||||
assert request(role,'secret/data/'+other,token=token)[0]==403
|
||||
assert request(role,'sys/mounts',token=token)[0]==403
|
||||
assert request(role,'auth/cert/login',{'name':other})[0] in (400,403)
|
||||
results.append({'role':role,'own_secret':200,'other_secret':403,'admin':403,'TTL':auth['lease_duration'],'wrong_role_denied':True})
|
||||
finally:request(role,'auth/token/revoke-self',{},token)
|
||||
for role in (None,'outsider','revoked','expired','alien'):
|
||||
try:assert request(role,'auth/cert/login',{'name':'api1'})[0] in (400,403)
|
||||
except (ssl.SSLError,urllib.error.URLError) as exc:
|
||||
# TLS handshake rejection also denies auth; network must already work above.
|
||||
if not isinstance(exc,ssl.SSLError) and not isinstance(getattr(exc,'reason',None),ssl.SSLError):raise
|
||||
results.append({'identity':role,'login_denied':True})
|
||||
print(json.dumps({'passed':True,'results':results},indent=2))
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Grade-5 track 1: revoke a real worker, rotate DB secret/leaf, preserve tasks.
|
||||
|
||||
Operator-only fault injection on a local synthetic workload. No public switch.
|
||||
All backups contain private material and stay under ignored secrets/operator.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
sys.path[:0]=[str(ROOT/'scripts'),str(ROOT/'tests')]
|
||||
import pki
|
||||
from bao_operator import request
|
||||
from database_operator import sql
|
||||
from client import Client
|
||||
|
||||
|
||||
def docker(*args,input=None):
|
||||
r=subprocess.run(['docker','compose',*args],cwd=ROOT,input=input,text=True,capture_output=True)
|
||||
if r.returncode:raise RuntimeError('Compose experiment step failed: '+args[0])
|
||||
return r.stdout.strip()
|
||||
|
||||
|
||||
def main():
|
||||
c=Client('https://127.0.0.1:8443',str(ROOT/'secrets/trust/ca.crt'))
|
||||
a,b,e,chat=c.bootstrap()
|
||||
baseline={'legacy_admin_can_read_sessions':int(sql('SELECT count(*) FROM sessions'))>0,
|
||||
'scoped_worker_denied':['sessions','password_hashes','messages','DDL','role_admin','foreign_S3_prefix','bucket_admin','queue_publish','queue_admin']}
|
||||
image_before=docker('images','-q','worker')
|
||||
root=json.loads((ROOT/'secrets/operator/bao-init.json').read_text())['root_token']
|
||||
identity=ROOT/'secrets/identities/worker'
|
||||
backup=ROOT/'secrets/operator/worker-before-rotation';backup.mkdir(exist_ok=True)
|
||||
for f in identity.iterdir():shutil.copy2(f,backup/f.name)
|
||||
old_serial=pki.x509.load_pem_x509_certificate((identity/'cert.pem').read_bytes()).serial_number
|
||||
start=time.monotonic()
|
||||
pki.revoke('worker')
|
||||
request('auth/cert/crls/operator',{'crl':(ROOT/'secrets/trust/revocations.pem').read_text()},root)
|
||||
deny="import sys;sys.path.insert(0,'/checks');from tls_matrix import rejected;assert rejected('db',8443,'worker');print('old worker TLS denied')"
|
||||
assert 'denied' in docker('run','--rm','--no-deps','security-check','python','-c',deny)
|
||||
# Terminate old already-authenticated DB sessions as part of containment.
|
||||
sql("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename='worker'")
|
||||
queued=[c.attachment(a,chat) for _ in range(3)]
|
||||
time.sleep(3)
|
||||
ids=','.join("'"+att['id']+"'" for att,_ in queued)
|
||||
assert sql(f"SELECT count(*) FROM attachments WHERE id IN ({ids}) AND status='queued'")=='3'
|
||||
# Cert auth must reject the old identity even though signed by our CA.
|
||||
rejection=docker('exec','-T','worker','python','-',input="from src.secret_agent import fetch\nfrom urllib.error import HTTPError\ntry: fetch()\nexcept HTTPError as e: assert e.code in (400,403); print('denied')\nelse: raise AssertionError('revoked worker got secrets')\n")
|
||||
assert rejection=='denied'
|
||||
cfg_path=ROOT/'secrets/operator/config.json';cfg=json.loads(cfg_path.read_text())
|
||||
old_url=cfg['service_credentials']['worker']['DATABASE_URL']
|
||||
password=secrets.token_hex(24)
|
||||
sql(f"ALTER ROLE worker PASSWORD '{password}'")
|
||||
cfg['database_passwords']['worker']=password
|
||||
cfg['service_credentials']['worker']['DATABASE_URL']=f'postgresql://worker:{password}@127.0.0.1:5432/messenger'
|
||||
cfg_path.write_text(json.dumps(cfg,indent=2));cfg_path.chmod(0o600)
|
||||
cert=pki.issue('worker')
|
||||
docker('run','--rm','--no-deps','secret-init')
|
||||
# New TLS identity + old database secret is denied by PostgreSQL itself.
|
||||
code="""import json,sys,psycopg
|
||||
url=json.load(sys.stdin)['url']
|
||||
try: psycopg.connect(url,connect_timeout=3)
|
||||
except psycopg.OperationalError as e:
|
||||
assert 'password authentication failed' in str(e);print('old password denied')
|
||||
else:raise AssertionError('old database secret still valid')
|
||||
"""
|
||||
check=docker('exec','-T','worker','python','-c',code,input=json.dumps({'url':old_url}))
|
||||
assert check=='old password denied'
|
||||
request('secret/data/worker',{'data':cfg['service_credentials']['worker']},root)
|
||||
for att,content in queued:
|
||||
c.wait_ready(a,att,wait_seconds=90)
|
||||
downloaded=c.call('GET','/api/v1/attachments/'+att['id']+'/content',actor=a)['bytes']
|
||||
assert hashlib.sha256(downloaded).digest()==hashlib.sha256(content).digest()
|
||||
assert sql(f"SELECT count(*) FROM jobs WHERE attachment_id IN ({ids}) AND status='done'")=='3'
|
||||
assert sql(f"SELECT count(*) FROM jobs WHERE attachment_id IN ({ids})")=='3'
|
||||
assert image_before==docker('images','-q','worker')
|
||||
result={'passed':True,'baseline':baseline,'revoked_serial':str(old_serial),'new_serial':str(cert.serial_number),
|
||||
'revoked_certificate_denied':True,'revoked_Bao_auth_denied':True,'rotated_old_DB_password_denied':True,
|
||||
'queued_during_revocation':3,'completed_after_rotation':3,'distinct_jobs':3,'content_SHA256_preserved':True,
|
||||
'image_unchanged':True,'total_seconds':round(time.monotonic()-start,2)}
|
||||
(ROOT/'evidence/compromised-worker.json').write_text(json.dumps(result,indent=2)+'\n')
|
||||
print('PASS: revoked worker -> queue preserved -> new cert/DB secret -> 3/3 jobs and hashes intact')
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,36 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import ssl
|
||||
import subprocess
|
||||
import urllib.request
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
def docker(*args):return subprocess.check_output(['docker',*args],cwd=ROOT,text=True,encoding='utf-8').strip()
|
||||
rows=[]
|
||||
ids=docker('compose','ps','-q').splitlines()
|
||||
for cid in ids:
|
||||
data=json.loads(docker('inspect',cid))[0]
|
||||
role=data['Config']['Labels']['com.docker.compose.service']
|
||||
if role in ('telemetry-init','secret-init'):continue
|
||||
h=data['HostConfig'];mounts=data['Mounts'];user=data['Config']['User']
|
||||
assert not h['Privileged'] and h['NetworkMode']!='host'
|
||||
assert h['ReadonlyRootfs'] and 'ALL' in h['CapDrop'] and 'no-new-privileges:true' in h['SecurityOpt']
|
||||
assert h['Memory']>0 and h['NanoCpus']>0 and h['PidsLimit']>0
|
||||
assert not any('docker.sock' in m['Source'] or '/operator' in m['Source'].replace('\\','/') for m in mounts)
|
||||
published=[]
|
||||
for port,bindings in (h.get('PortBindings') or {}).items():
|
||||
for bind in bindings or []:
|
||||
assert bind['HostIp']=='127.0.0.1'
|
||||
assert (role,bind['HostPort']) in {('lb','8443'),('openbao','8200'),('grafana','3000')}
|
||||
published.append(bind['HostPort'])
|
||||
if role in ('api1','api2','worker','dispatcher','reconciler'):
|
||||
assert user=='10001:10001'
|
||||
assert all(not e.startswith(('DATABASE_URL=','S3_SECRET_KEY=','BAO_TOKEN=','POSTGRES_PASSWORD=')) for e in data['Config']['Env'])
|
||||
assert all(m['Destination'] in ('/identity','/trust','/tmp') for m in mounts)
|
||||
rows.append({'service':role,'user':user,'read_only':True,'cap_drop':['ALL'],'memory_bytes':h['Memory'],'NanoCpus':h['NanoCpus'],'pids_limit':h['PidsLimit'],'published_loopback_ports':published,'operator_material_absent':True})
|
||||
ctx=ssl.create_default_context(cafile=str(ROOT/'secrets/trust/ca.crt'))
|
||||
with urllib.request.urlopen('https://127.0.0.1:3000/login',context=ctx,timeout=10) as r:assert r.status==200
|
||||
raw=docker('compose','run','--rm','--no-deps','security-check','python','/checks/plaintext_isolation.py')
|
||||
isolation=json.loads(raw);assert isolation['passed']
|
||||
(ROOT/'evidence/hardening.json').write_text(json.dumps({'passed':True,'runtime':rows,'grafana_verified_HTTPS':True,'plaintext_isolation':isolation},indent=2)+'\n')
|
||||
print('PASS: runtime privileges/resource limits/mounts/public ports and loopback-only plaintext')
|
||||
@@ -0,0 +1,15 @@
|
||||
import json
|
||||
import socket
|
||||
|
||||
ports={'api1':[8080],'api2':[8080],'db':[5432],'s3':[8333,8888,9333,19333],
|
||||
'cache':[6379],'broker':[6379],'otel-collector':[4318,9464,8888,13133],
|
||||
'prometheus':[9090],'loki':[3100,9095],'tempo':[3200,4318,9095]}
|
||||
results=[]
|
||||
for host,values in ports.items():
|
||||
for port in values:
|
||||
with socket.socket() as s:
|
||||
s.settimeout(2)
|
||||
try:s.connect((host,port))
|
||||
except ConnectionRefusedError:results.append({'host':host,'port':port,'refused':True})
|
||||
else:raise AssertionError(f'Plaintext port reachable: {host}:{port}')
|
||||
print(json.dumps({'passed':True,'probes':results},indent=2))
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Finite supervisor lease under real sealed OpenBao, then explicit recovery."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import ssl
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from urllib.error import HTTPError,URLError
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0,str(ROOT/'scripts'))
|
||||
from bao_operator import request
|
||||
|
||||
ctx=ssl.create_default_context(cafile=str(ROOT/'secrets/trust/ca.crt'))
|
||||
def status():
|
||||
try:
|
||||
with urllib.request.urlopen('https://127.0.0.1:8443/health/ready',context=ctx,timeout=3) as r:return r.status
|
||||
except HTTPError as e:return e.code
|
||||
|
||||
assert status()==200
|
||||
root=json.loads((ROOT/'secrets/operator/bao-init.json').read_text())['root_token']
|
||||
start=time.monotonic();observations=[]
|
||||
try:
|
||||
request('sys/seal',{},root)
|
||||
while time.monotonic()-start<150:
|
||||
current=status();observations.append({'seconds':round(time.monotonic()-start,1),'HTTP':current})
|
||||
if current==503:
|
||||
# Both API children must actually have stopped, not just one LB check.
|
||||
code="import socket; s=socket.socket();s.settimeout(1);assert s.connect_ex(('127.0.0.1',8080))!=0"
|
||||
for role in ('api1','api2'):
|
||||
subprocess.run(['docker','compose','exec','-T',role,'python','-c',code],cwd=ROOT,check=True,capture_output=True)
|
||||
break
|
||||
time.sleep(2)
|
||||
else:raise AssertionError('Application exceeded finite secret availability budget')
|
||||
stopped=time.monotonic()-start
|
||||
finally:
|
||||
subprocess.run([sys.executable,str(ROOT/'scripts/bao_operator.py'),'unseal'],cwd=ROOT,check=True)
|
||||
recovery=time.monotonic()
|
||||
while time.monotonic()-recovery<60:
|
||||
if status()==200:break
|
||||
time.sleep(1)
|
||||
else:raise AssertionError('Unsealed system failed to recover')
|
||||
result={'passed':True,'secret_refresh_seconds':20,'token_max_TTL_seconds':120,'both_API_children_stopped_seconds':round(stopped,2),'recovery_seconds':round(time.monotonic()-recovery,2),'observations':observations,'scope':'Supervisor bounds use during Bao outage; static backend passwords themselves require explicit rotation.'}
|
||||
(ROOT/'evidence/sealed-bao.json').write_text(json.dumps(result,indent=2)+'\n')
|
||||
print('PASS: sealed Bao -> finite lease -> both API children stopped -> unseal -> recovery')
|
||||
@@ -0,0 +1,68 @@
|
||||
"""HTTPS ACL/session/KDF/replica-shared login limits. Synthetic local test data."""
|
||||
import base64
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
sys.path[:0]=[str(ROOT/'tests'),str(ROOT/'scripts')]
|
||||
from client import Client
|
||||
from database_operator import sql
|
||||
from argon2 import PasswordHasher
|
||||
|
||||
|
||||
def client():return Client('https://127.0.0.1:8443',str(ROOT/'secrets/trust/ca.crt'))
|
||||
|
||||
|
||||
def main():
|
||||
c=client();a,b,e,chat=c.bootstrap();att,content=c.attachment(a,chat);c.wait_ready(a,att)
|
||||
for suffix in ('','/content','/thumbnail'):
|
||||
c.call('GET','/api/v1/attachments/'+att['id']+suffix,404,actor=e)
|
||||
c.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=x.png',404,actor=e,raw=content,headers={'Content-Type':'image/png'})
|
||||
c.call('GET','/api/v1/users/me',401,headers={'Authorization':'Bearer '+'x'*43})
|
||||
# Upgrade a valid historical hash only after successful verification.
|
||||
old=PasswordHasher(time_cost=2,memory_cost=19456,parallelism=1).hash(a['registration']['password'])
|
||||
sql(f"UPDATE users SET password_hash='{old}' WHERE id='{a['id']}'")
|
||||
credential=base64.b64encode((a['registration']['username']+':'+a['registration']['password']).encode()).decode()
|
||||
session=c.call('POST','/api/v1/auth/sessions',201,headers={'Authorization':'Basic '+credential},schema='Session')['json']
|
||||
stored=sql(f"SELECT password_hash FROM users WHERE id='{a['id']}'")
|
||||
assert stored!=old and '$argon2id$' in stored and 'm=32768,t=2,p=1' in stored
|
||||
digest=hashlib.sha256(session['token'].encode()).hexdigest()
|
||||
assert sql(f"SELECT count(*) FROM sessions WHERE token_hash='{digest}'")=='1'
|
||||
sql(f"UPDATE sessions SET expires_at=clock_timestamp()-interval '1 second' WHERE token_hash='{digest}'")
|
||||
c.call('GET','/api/v1/users/me',401,actor={'token':session['token']})
|
||||
c.call('DELETE','/api/v1/auth/sessions/current',204,actor=b)
|
||||
c.call('GET','/api/v1/users/me',401,actor=b)
|
||||
# Reset only the synthetic experiment's two limiter counters; limiter stays on.
|
||||
sql('DELETE FROM login_budget')
|
||||
wrong=base64.b64encode((a['registration']['username']+':Wrong-password-123').encode()).decode()
|
||||
def login(_):
|
||||
r=client().call('POST','/api/v1/auth/sessions',(401,429),headers={'Authorization':'Basic '+wrong})
|
||||
if r['status']==429:assert int(r['headers']['Retry-After'])>0
|
||||
assert a['registration']['password'].encode() not in r['bytes'] and b'argon2' not in r['bytes']
|
||||
return {'status':r['status'],'seconds':round(r['elapsed'],4),'instance':r['headers']['X-Instance-Id']}
|
||||
begin=time.monotonic()
|
||||
with ThreadPoolExecutor(max_workers=20) as pool:attempts=list(pool.map(login,range(20)))
|
||||
elapsed=time.monotonic()-begin
|
||||
assert elapsed<10 and sum(r['status']==401 for r in attempts)==12 and sum(r['status']==429 for r in attempts)==8
|
||||
assert len({r['instance'] for r in attempts})==2
|
||||
# Measure actual Argon2 on the resource-limited API container, not the host.
|
||||
bench="""import json,time,statistics,resource
|
||||
from argon2 import PasswordHasher
|
||||
p=PasswordHasher(time_cost=2,memory_cost=32768,parallelism=1);times=[]
|
||||
for i in range(12):
|
||||
t=time.perf_counter();h=p.hash('synthetic-benchmark-password');p.verify(h,'synthetic-benchmark-password');times.append(time.perf_counter()-t)
|
||||
print(json.dumps({'hash_plus_verify_seconds':times,'median':statistics.median(times),'max_rss_kib':resource.getrusage(resource.RUSAGE_SELF).ru_maxrss}))
|
||||
"""
|
||||
kdf=json.loads(subprocess.check_output(['docker','compose','exec','-T','api1','python','-c',bench],cwd=ROOT,text=True))
|
||||
result={'passed':True,'ACL_attachment_denials':4,'expired_session_denied':True,'logout_denied':True,'token_hash_sha256':True,'rehash':'19456 -> 32768 KiB','login_burst':attempts,'burst_seconds':round(elapsed,3),'KDF':kdf,'KDF_slots_per_API':2,'KDF_memory_bound_two_APIs_MiB':128}
|
||||
(ROOT/'evidence/security-api.json').write_text(json.dumps(result,indent=2)+'\n')
|
||||
print('PASS: ACL, expiry/logout, hash storage/rehash, shared 12/10s rate limit, measured Argon2')
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,31 @@
|
||||
"""Read telemetry using the operator's mTLS identity; never emit secret canaries."""
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
from urllib.error import HTTPError
|
||||
from tls_matrix import context
|
||||
|
||||
args=json.load(sys.stdin);trace_id=args['trace_id']
|
||||
def get(host,port,path):
|
||||
with urllib.request.urlopen('https://'+host+':'+str(port)+path,context=context('operator'),timeout=8) as r:return json.load(r)
|
||||
deadline=time.monotonic()+90
|
||||
while True:
|
||||
try:
|
||||
trace=get('tempo',8444,'/api/traces/'+trace_id)
|
||||
spans=[s for b in trace.get('batches',trace.get('resourceSpans',[])) for scope in b.get('scopeSpans',b.get('instrumentationLibrarySpans',[])) for s in scope.get('spans',[])]
|
||||
names={s['name'] for s in spans}
|
||||
query=urllib.parse.urlencode({'query':'{service_name=~"messenger-(api|worker|dispatcher)"} |= "'+trace_id+'"','limit':1000,'start':int((time.time()-600)*1e9),'end':int(time.time()*1e9)})
|
||||
logs=get('loki',8443,'/loki/api/v1/query_range?'+query)
|
||||
lines=[line for stream in logs['data']['result'] for _,line in stream['values']]
|
||||
if {'attachment.consume','db.attachment.commit','attachment.publish','image.render'}<=names and any('job_finished' in line for line in lines):break
|
||||
except HTTPError as e:
|
||||
if e.code!=404:raise
|
||||
if time.monotonic()>deadline:raise AssertionError('Telemetry did not arrive over TLS')
|
||||
time.sleep(2)
|
||||
raw=json.dumps([trace,logs])
|
||||
assert all(secret not in raw for secret in args['canaries'])
|
||||
metrics=get('prometheus',8443,'/api/v1/query?'+urllib.parse.urlencode({'query':'http_server_requests_total'}))
|
||||
assert metrics['status']=='success' and metrics['data']['result']
|
||||
print(json.dumps({'passed':True,'trace_id':trace_id,'span_count':len(spans),'span_names':sorted(names),'correlated_log_count':len(lines),'http_metric_series':len(metrics['data']['result']),'credential_canaries_absent':True,'all_queries':'mTLS verified operator identity'},indent=2))
|
||||
@@ -0,0 +1,30 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0,str(ROOT/'tests'))
|
||||
from client import Client
|
||||
|
||||
c=Client('https://127.0.0.1:8443',str(ROOT/'secrets/trust/ca.crt'))
|
||||
a=c.user('telemetry')
|
||||
chat=c.call('POST','/api/v1/chats',201,actor=a,body={'title':'TLS telemetry','member_ids':[a['id']]},schema='Chat')['json']
|
||||
trace=uuid.uuid4().hex
|
||||
att=c.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=sample.png',202,actor=a,
|
||||
raw=(ROOT/'tests/fixtures/sample.png').read_bytes(),headers={'Content-Type':'image/png','traceparent':'00-'+trace+'-'+uuid.uuid4().hex[:16]+'-01'},schema='Attachment')['json']
|
||||
c.wait_ready(a,att)
|
||||
cfg=json.loads((ROOT/'secrets/operator/config.json').read_text())
|
||||
canaries=[a['token'],a['registration']['password'],*cfg['database_passwords'].values()]
|
||||
for data in cfg['service_credentials'].values():
|
||||
canaries.extend([data[k] for k in ('CURSOR_SECRET','S3_SECRET_KEY') if k in data])
|
||||
result=subprocess.run(['docker','compose','run','--rm','-T','--no-deps','security-check','python','/checks/telemetry_probe.py'],cwd=ROOT,
|
||||
input=json.dumps({'trace_id':trace,'canaries':canaries}),text=True,capture_output=True)
|
||||
if result.returncode:raise RuntimeError('Telemetry check failed: '+result.stderr[-1500:])
|
||||
summary=json.loads(result.stdout)
|
||||
logs=subprocess.check_output(['docker','compose','logs','--no-color'],cwd=ROOT,text=True,encoding='utf-8',errors='replace')
|
||||
assert all(secret not in logs for secret in canaries)
|
||||
summary['container_logs_canaries_absent']=True
|
||||
(ROOT/'evidence/telemetry-security.json').write_text(json.dumps(summary,indent=2)+'\n')
|
||||
print('PASS: TLS metrics/traces/log correlation, credential canaries absent in telemetry/container logs')
|
||||
@@ -0,0 +1,49 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from PIL import Image
|
||||
from src.render import render
|
||||
|
||||
|
||||
class RenderTests(unittest.TestCase):
|
||||
def generate(self, image, fmt='PNG', **save_args):
|
||||
folder=tempfile.TemporaryDirectory(); self.addCleanup(folder.cleanup)
|
||||
root=Path(folder.name); source=root/'input'
|
||||
image.save(source,format=fmt,**save_args)
|
||||
records=render(source,root,1)
|
||||
return root,{row['name']:row for row in records}
|
||||
|
||||
def test_contract_dimensions(self):
|
||||
root,rows=self.generate(Image.new('RGB',(320,200),'red'))
|
||||
self.assertEqual((rows['thumbnail']['width'],rows['thumbnail']['height']),(256,160))
|
||||
self.assertEqual((rows['crop']['width'],rows['crop']['height']),(128,128))
|
||||
with Image.open(root/'optimized') as optimized:
|
||||
self.assertEqual(optimized.format,'WEBP')
|
||||
self.assertEqual(optimized.size,(320,200))
|
||||
|
||||
def test_alpha_white_and_small_image_not_enlarged(self):
|
||||
root,rows=self.generate(Image.new('RGBA',(17,9),(255,0,0,0)))
|
||||
with Image.open(root/'thumbnail') as thumb:
|
||||
self.assertEqual(thumb.size,(17,9))
|
||||
self.assertTrue(all(c>=250 for c in thumb.getpixel((0,0))))
|
||||
self.assertEqual(rows['crop']['width'],128)
|
||||
|
||||
def test_exif_orientation_and_metadata_removed(self):
|
||||
exif=Image.Exif(); exif[274]=6; exif[270]='synthetic metadata must disappear'
|
||||
root,rows=self.generate(Image.new('RGB',(40,20),'blue'),'JPEG',exif=exif)
|
||||
with Image.open(root/'thumbnail') as thumb:
|
||||
self.assertEqual(thumb.size,(20,40))
|
||||
self.assertFalse(thumb.getexif())
|
||||
|
||||
def test_centered_crop(self):
|
||||
image=Image.new('RGB',(300,100),'red')
|
||||
image.paste(Image.new('RGB',(100,100),'green'),(100,0))
|
||||
root,_=self.generate(image)
|
||||
with Image.open(root/'crop') as crop:
|
||||
r,g,b=crop.getpixel((64,64))
|
||||
self.assertGreater(g,r+60); self.assertGreater(g,b+60)
|
||||
|
||||
|
||||
if __name__=='__main__': unittest.main()
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Inside the isolated test network. Every permitted edge and rejected identity.
|
||||
|
||||
A timeout/refused connection is never counted as a valid negative TLS result.
|
||||
"""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import ssl
|
||||
import struct
|
||||
import time
|
||||
|
||||
IDENTITIES=Path('/audit-identities')
|
||||
TRUST=Path('/trust')
|
||||
|
||||
|
||||
def context(role=None):
|
||||
ctx=ssl.create_default_context(cafile=str(TRUST/'bundle.pem'))
|
||||
ctx.verify_flags |= ssl.VERIFY_CRL_CHECK_LEAF
|
||||
if role:ctx.load_cert_chain(IDENTITIES/role/'cert.pem',IDENTITIES/role/'key.pem')
|
||||
return ctx
|
||||
|
||||
|
||||
def probe(host,port,role,hostname=None):
|
||||
ctx=context(role)
|
||||
with socket.create_connection((host,port),timeout=5) as raw:
|
||||
with ctx.wrap_socket(raw,server_hostname=hostname or host) as conn:
|
||||
conn.settimeout(5)
|
||||
if host=='db':payload=struct.pack('!II',8,80877103)
|
||||
elif host in ('cache','broker'):payload=b'*1\r\n$4\r\nPING\r\n'
|
||||
else:payload=b'GET /health/live HTTP/1.1\r\nHost: '+host.encode()+b'\r\nConnection: close\r\n\r\n'
|
||||
conn.sendall(payload)
|
||||
return conn.recv(4096)
|
||||
|
||||
|
||||
def rejected(host,port,role):
|
||||
try:return not probe(host,port,role)
|
||||
except (ssl.SSLError,ConnectionResetError,BrokenPipeError):return True
|
||||
|
||||
|
||||
def main():
|
||||
start=time.monotonic();results=[]
|
||||
for file in sorted(Path('/mesh').glob('*.json')):
|
||||
host=file.stem
|
||||
for route in json.loads(file.read_text()):
|
||||
if route['mode']!='server':continue
|
||||
port=route['listen']
|
||||
for role in route['allow']:
|
||||
assert probe(host,port,role),f'Permitted edge failed: {role}->{host}:{port}'
|
||||
results.append({'source':role,'destination':host,'port':port,'allowed':True})
|
||||
for role in (None,'outsider','alien','expired','revoked'):
|
||||
assert rejected(host,port,role),f'Forbidden edge accepted: {role}->{host}:{port}'
|
||||
# Same-CA hostname mismatch must fail client validation, not timeout.
|
||||
try:probe(host,port,route['allow'][0],hostname='outsider')
|
||||
except ssl.SSLCertVerificationError:pass
|
||||
else:raise AssertionError('Wrong server identity accepted')
|
||||
# Plaintext has no protocol response from a TLS-only ingress.
|
||||
with socket.create_connection((host,port),timeout=5) as raw:
|
||||
raw.settimeout(5);raw.sendall(b'GET / HTTP/1.1\r\nHost: x\r\n\r\n')
|
||||
try:assert raw.recv(100)==b''
|
||||
except ConnectionResetError:pass
|
||||
results.append({'destination':host,'port':port,'denied':['no_cert','wrong_CA','wrong_SAN','expired','revoked','wrong_server_SAN','plaintext']})
|
||||
print(json.dumps({'passed':True,'seconds':round(time.monotonic()-start,2),'edges':results},indent=2))
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Execute inside the real worker container with its cert-authenticated credentials."""
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from uuid import uuid4
|
||||
import psycopg
|
||||
import redis
|
||||
from botocore.exceptions import ClientError
|
||||
from src.secret_agent import fetch
|
||||
from src.jobs import database
|
||||
from src.storage import Storage
|
||||
from src.broker import connect,ensure_group
|
||||
|
||||
data,_=fetch();os.environ.update(data)
|
||||
results={}
|
||||
with database() as db:
|
||||
assert db.execute('SELECT current_user AS role').fetchone()['role']=='worker'
|
||||
db.execute('SELECT id FROM jobs LIMIT 1');results['jobs_read']='allowed'
|
||||
for name,query in [('sessions','SELECT * FROM sessions'),('password_hashes','SELECT password_hash FROM users'),('messages','SELECT * FROM messages'),('schema','CREATE TABLE public.worker_escape(id int)'),('privilege','CREATE ROLE worker_escape SUPERUSER')]:
|
||||
try:db.execute(query)
|
||||
except psycopg.errors.InsufficientPrivilege:results[name]='denied'
|
||||
else:raise AssertionError('Unexpected worker permission: '+name)
|
||||
storage=Storage();key='variants/security-test-'+uuid4().hex+'/probe'
|
||||
storage.client.put_object(Bucket=storage.bucket,Key=key,Body=b'test')
|
||||
assert storage.client.get_object(Bucket=storage.bucket,Key=key)['Body'].read()==b'test'
|
||||
storage.delete(key);results['variants_read_write_delete']='allowed'
|
||||
for name,operation in [('foreign_prefix',lambda:storage.client.put_object(Bucket=storage.bucket,Key='private/forbidden',Body=b'test')),('bucket_admin',lambda:storage.client.create_bucket(Bucket='forbidden-'+uuid4().hex))]:
|
||||
try:operation()
|
||||
except ClientError as e:
|
||||
assert e.response['ResponseMetadata']['HTTPStatusCode']==403;results[name]='denied'
|
||||
else:raise AssertionError('Unexpected S3 permission: '+name)
|
||||
queue=connect();ensure_group(queue);assert queue.ping();results['queue_group']='allowed'
|
||||
for name,op in [('queue_publish',lambda:queue.xadd('attachment-jobs',{'forbidden':'test'})),('foreign_queue',lambda:queue.xreadgroup('image-workers','security-test',{'other-queue':'>'},count=1)),('queue_admin',lambda:queue.config_get('*'))]:
|
||||
try:op()
|
||||
except redis.exceptions.NoPermissionError:results[name]='denied'
|
||||
else:raise AssertionError('Unexpected queue permission: '+name)
|
||||
print(json.dumps({'passed':True,'rights':results},indent=2))
|
||||
+1061
File diff suppressed because it is too large
Load Diff
+41
-31
@@ -1,37 +1,47 @@
|
||||
# Матрица доверия — заполнить в решении lab7
|
||||
# Матрица доверия реализации ЛР7
|
||||
|
||||
Это обязательная матрица покрытия. Для каждой строки укажите реальный порт, endpoint, CA, service identity, проверку SAN/ACL, срок жизни credentials и команду положительного/отрицательного теста. TLS шифрует и подтверждает сервер; mTLS также проверяет клиента. При доверенном CA всё равно нужно ограничивать разрешённую identity. Plaintext hop допустим только внутри закрытого локального transport/proxy boundary, без опубликованного обхода; границу и rationale нужно доказать.
|
||||
Обозначения: mTLS = TLS >=1.2 с CA, hostname/SAN сервера, обязательным клиентским
|
||||
cert и точным SAN allowlist. CA общий, identity/key отдельны для каждой роли.
|
||||
Полные allowlist/порты в `infra/mesh/*.json`; ниже перечислены все сетевые hops.
|
||||
Внешние порты привязаны только к 127.0.0.1 хоста, внутренние не публикуются.
|
||||
|
||||
| Связь | Минимум на 3 | Дополнительно на 4 | Доказательство / конфиг |
|
||||
| Связь | Transport / проверяемая identity | Прикладные права | Доказательство |
|
||||
| --- | --- | --- | --- |
|
||||
| Клиент → ingress/LB | TLS, CA + hostname, Bearer | Проверка expiry/revocation сессии и общего login rate limit | TODO |
|
||||
| LB → API (все реплики) | mTLS + allowlist LB identity | Ротация leaf без обхода проверки | TODO |
|
||||
| API/worker → PostgreSQL | TLS verify-full + раздельные ограниченные DB roles | Ротация credentials; mTLS — допустимое усиление | TODO |
|
||||
| API/worker → S3 | TLS + разные scoped credentials/prefix rights | Проверка denied prefix и ротация | TODO |
|
||||
| API/worker → broker | TLS + разные producer/consumer identities и ACL | mTLS + ограничение допустимых SAN/ролей | TODO |
|
||||
| API/worker → Collector | TLS; сеть/аутентификация запрещает произвольный ingest | mTLS с раздельными identities | TODO |
|
||||
| Collector → telemetry backends | TLS + scoped backend credentials или mTLS | Ротация и проверка least privilege | TODO |
|
||||
| Grafana → data sources | TLS с проверкой сервера, ограниченная identity | Отрицательные проверки доступа | TODO |
|
||||
| API/worker/agent → OpenBao | TLS + сервисный auth + scoped policy | Renewal/expiry/revocation | TODO |
|
||||
| Оператор → OpenBao/Grafana/S3 admin | TLS + authentication, доступ локальный/через защищённый канал | Разделение operator/application roles | TODO |
|
||||
| API → Redis/Valkey, если используется | TLS + ACL/пароль с минимальными правами | Ротация; mTLS по возможности | TODO |
|
||||
| Replication/кластерные каналы, если добавлены | TLS/mTLS либо подтверждённый шифрованный transport | Проверка заявленной модели отказа | TODO |
|
||||
| Клиент → LB:8443 | Native TLS, SAN localhost/127.0.0.1, клиент доверяет локальному CA | Basic только login; Bearer для business API | tests/smoke.py, tests/security.py |
|
||||
| LB → api1/api2:8443 | mTLS, server SAN api1/api2, client SAN только lb | API:8080 только loopback | tls_matrix, no-cert/foreign-CA/wrong-SAN/expiry/revoke/plaintext |
|
||||
| API/worker/dispatcher/reconciler → db:8443 | mTLS, SAN db, отдельные client SAN; operator для диагностики | SCRAM; отдельные PostgreSQL roles, не superuser; `database_operator.py` GRANT | tls_matrix, worker_rights, compromised_worker |
|
||||
| API/worker/reconciler → s3:8443 | mTLS, SAN s3, соответствующие client SAN; dispatcher запрещён | SigV4 credentials отдельно. API: Read originals/variants, Write staging. Worker: Read/Write staging/originals/variants. Reconciler дополнительно List bucket | tls_matrix, worker_rights, public upload |
|
||||
| worker/dispatcher → broker:8443 | mTLS, SAN broker, client worker/dispatcher; operator diagnostic | Valkey ACL: worker consume/ack/claim; dispatcher publish; только attachment-jobs | tls_matrix, worker_rights |
|
||||
| api1/api2 → cache:8443 | mTLS, SAN cache, client api1/api2 | Именованные ACL GET/SET user:*; TTL 60s; truth в PostgreSQL | tls_matrix + smoke |
|
||||
| API/worker/dispatcher/reconciler → otel-collector:8443 | mTLS, SAN otel-collector, allowlist этих identities | OTLP HTTP; без body/Authorization в telemetry. Cleanup пока не испускает сигналы, роль зарезервирована | tls_matrix, telemetry_security |
|
||||
| Collector → tempo:8443 / loki:8443 | mTLS, server SAN tempo/loki; client otel-collector | OTLP; внутренние native backends без отдельного auth, authorizer TLS sidecar | tls_matrix, telemetry_security |
|
||||
| Prometheus → collector:8444/8445 | mTLS, client prometheus; server otel-collector | Scrape application/exporter metrics, оба native listeners loopback | tls_matrix + telemetry_security |
|
||||
| Grafana → prometheus:8443, loki:8443, tempo:8444 | mTLS, client grafana; отдельный server SAN | Queries через loopback outgoing tunnel; нет публичного доступа к backend | tls_matrix + telemetry_security |
|
||||
| Service supervisor → OpenBao:8200 | Native mTLS; server SAN openbao; cert auth role требует SAN service-name | read только secret/data/role, token TTL/maxTTL 120s, re-auth 20s, revoke-self после чтения | bao_policy, sealed_bao |
|
||||
| Оператор → OpenBao:127.0.0.1:8200 | Native mTLS operator + root token только для provision/seal | Init/unseal вручную с защищёнными локальными shares | bao_operator, restart-processes |
|
||||
| Оператор → Grafana:127.0.0.1:3000 | Native HTTPS, SAN localhost/127.0.0.1 + уникальный admin password | Только локальный административный интерфейс | hardening audit / HTTPS login page |
|
||||
| storage-init → S3 admin | Только loopback в namespace S3, одноразовый operator profile, собственного внешнего hop нет | Admin credential из отдельного локального файла, runtime его не получает | up.py private bucket bootstrap |
|
||||
| database_operator → PostgreSQL | docker exec, Unix socket внутри DB, не сетевой hop | operator messenger role; socket не расшарен | миграции + сравнение privileged/scoped в track1 |
|
||||
| secret-init → runtime secret volumes | Локальные mounts, одноразовый operator-owned процесс | Только service cert/config, chmod/chown; CA/unseal/root отсутствуют | hardening audit |
|
||||
| Healthchecks | API/DB native loopback/Unix; LB → API по mTLS | /health/ready без секретов; public endpoint через LB HTTPS | smoke, verify_public |
|
||||
| Репликация/кластер | N/A: single PostgreSQL, S3, broker, Bao raft node | Нет межузловых replication hops; HA dependencies не заявляется | compose.yaml |
|
||||
|
||||
Нет выбранного компонента — отметьте N/A с причиной, а не оставляйте пустую строку. Внутренний публичный health listener, если нужен, минимален и изолирован; через него нельзя обращаться к бизнес-API без mTLS. Отдельно задокументируйте scrape/health/bootstrap каналы: они тоже не должны создавать обход.
|
||||
Каждый TLS proxy имеет только свой identity volume и публичный trust bundle.
|
||||
Native plaintext слушает только loopback общей namespace с proxy; iptables и
|
||||
«внутренняя сеть» не заменяют проверку SAN. Сетевые payloads в proxy не логируются.
|
||||
Контекст TLS кэшируется максимум 1s; соединение максимум 60s, TLS tickets на
|
||||
server proxy отключены. Отзыв нового подключения применяется <=1s, действующей
|
||||
связи <=61s; эксперимент также завершает старые worker DB sessions. Native
|
||||
HAProxy/OpenBao/Grafana перечитывают cert/config при restart/reload. CRL cert-auth
|
||||
OpenBao отдельно публикуется `bao_operator.py publish-crl`.
|
||||
|
||||
## Обязательные отрицательные проверки
|
||||
Отрицательные тесты не считают timeout/connection-refused правильным TLS отказом:
|
||||
сначала положительно проверяется каждая разрешённая связь. В `bao_policy` допускается
|
||||
именно SSL alert или HTTP deny, а не произвольная сетевая ошибка. Для API ACL,
|
||||
OpenBao policy и прав backend есть отдельные прикладные тесты.
|
||||
|
||||
1. HTTPS с правильным CA/hostname проходит; с пустым/чужим CA и неправильным hostname — ошибка верификации, не «сервер недоступен».
|
||||
2. LB→API: без client cert, с cert чужого CA, с запрещённой identity от доверенного CA — отказ; с разрешённой LB identity — успех.
|
||||
3. На 4 повторите набор для broker и Collector; ограничение на network policy не заменяет проверку identity.
|
||||
4. Пользователь вне чата не читает metadata/content и не прикрепляет чужой файл; поддельный X-User-Id не меняет Bearer identity.
|
||||
5. OpenBao app policy читает только свой secret/path; запись policy и чтение чужого path запрещены. Worker не получает admin credentials API/БД/S3.
|
||||
6. На 4 отзовите secret/cert и дождитесь контролируемого эффекта с учётом действующих соединений, TLS resumption и cache/TTL. Покажите новое подключение, иначе тест отзыва может ничего не проверить.
|
||||
|
||||
## Bootstrap и эксплуатация
|
||||
|
||||
В репозитории только открытые конфиги, policy templates и scripts. CA private key, leaf keys, root token, unseal shares и SecretID не коммитятся, не запекаются в образ и не печатаются в отчёте. Для локального стенда оператор создаёт их вне Git и передаёт через files/secrets с ограниченными правами. Опишите, кто имеет доступ к CA/unseal, что нужно после рестарта OpenBao и как восстановить работу.
|
||||
|
||||
OpenBao не использует dev-mode. Выберите persistent storage и auth метод (например AppRole с короткоживущим SecretID/response wrapping, cert auth или иная обоснованная identity). Не подменяйте root token постоянной переменной BAO_TOKEN в приложении. Рабочие tokens имеют policy, TTL и renewal/re-auth сценарий. Статические KV secrets на 3 допустимы при реальном чтении через сервисную identity; динамические DB credentials — хорошее расширение, но не обязательное название технологии.
|
||||
|
||||
Разрешённый компромисс учебного стенда: ручной operator init/unseal, если он безопасно документирован и не нужен при каждом обычном рестарте API. Автоматическое хранение всех unseal shares рядом с OpenBao без защиты не считается решённым bootstrap.
|
||||
Root token/unseal shares/CA private key — только защищённые `secrets/operator`
|
||||
на хосте, вне Git/image/container mounts. Static KV passwords не получают
|
||||
автоматический backend TTL: supervisor прекращает их использование после
|
||||
истечения локального бюджета, а украденные passwords отзываются ротацией.
|
||||
Роли SQL/Valkey/S3 и доступы не объединяются общим service credential.
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
{
|
||||
"passed": true,
|
||||
"results": [
|
||||
{
|
||||
"role": "api1",
|
||||
"own_secret": 200,
|
||||
"other_secret": 403,
|
||||
"admin": 403,
|
||||
"TTL": 120,
|
||||
"wrong_role_denied": true
|
||||
},
|
||||
{
|
||||
"role": "api2",
|
||||
"own_secret": 200,
|
||||
"other_secret": 403,
|
||||
"admin": 403,
|
||||
"TTL": 120,
|
||||
"wrong_role_denied": true
|
||||
},
|
||||
{
|
||||
"role": "worker",
|
||||
"own_secret": 200,
|
||||
"other_secret": 403,
|
||||
"admin": 403,
|
||||
"TTL": 120,
|
||||
"wrong_role_denied": true
|
||||
},
|
||||
{
|
||||
"role": "dispatcher",
|
||||
"own_secret": 200,
|
||||
"other_secret": 403,
|
||||
"admin": 403,
|
||||
"TTL": 120,
|
||||
"wrong_role_denied": true
|
||||
},
|
||||
{
|
||||
"role": "reconciler",
|
||||
"own_secret": 200,
|
||||
"other_secret": 403,
|
||||
"admin": 403,
|
||||
"TTL": 120,
|
||||
"wrong_role_denied": true
|
||||
},
|
||||
{
|
||||
"identity": null,
|
||||
"login_denied": true
|
||||
},
|
||||
{
|
||||
"identity": "outsider",
|
||||
"login_denied": true
|
||||
},
|
||||
{
|
||||
"identity": "revoked",
|
||||
"login_denied": true
|
||||
},
|
||||
{
|
||||
"identity": "expired",
|
||||
"login_denied": true
|
||||
},
|
||||
{
|
||||
"identity": "alien",
|
||||
"login_denied": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"passed": true,
|
||||
"completed_at_UTC": "2026-11-13T10:00:37.671336+00:00",
|
||||
"source_commit": "0bf1ff93c71a6a8cf16edeceefc3b3fab8d23261",
|
||||
"isolated_clone": true,
|
||||
"own_venv": true,
|
||||
"fresh_PKI_and_secrets": true,
|
||||
"fresh_compose_volumes": true,
|
||||
"public_suite": "PASS; 3 real SIGKILL scenarios cover 2 distinct API containers",
|
||||
"security_suite": "PASS; all 11 stages",
|
||||
"app_image_tested": "sha256:5d549e32c6a790e783550782db9b5c5993861fa5e7dac46039aaa9adb03e2eed",
|
||||
"host": "Windows 10, Docker Desktop 4.39 / Engine 28 / Compose 2.33, WSL2 4 CPU / 3 GiB RAM / 4 GiB swap",
|
||||
"scan_scope": "image-audit.json is the 12 November scan, preceding the TLS scheduling code fix; dependencies unchanged. It is not a scan of the exact app image above."
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: revoked worker -> queue preserved -> new cert/DB secret -> 3/3 jobs and hashes intact
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"passed": true,
|
||||
"baseline": {
|
||||
"legacy_admin_can_read_sessions": true,
|
||||
"scoped_worker_denied": [
|
||||
"sessions",
|
||||
"password_hashes",
|
||||
"messages",
|
||||
"DDL",
|
||||
"role_admin",
|
||||
"foreign_S3_prefix",
|
||||
"bucket_admin",
|
||||
"queue_publish",
|
||||
"queue_admin"
|
||||
]
|
||||
},
|
||||
"revoked_serial": "296803757360939619598658580576779705066316270051",
|
||||
"new_serial": "582892826110773913646976171481035976112843893487",
|
||||
"revoked_certificate_denied": true,
|
||||
"revoked_Bao_auth_denied": true,
|
||||
"rotated_old_DB_password_denied": true,
|
||||
"queued_during_revocation": 3,
|
||||
"completed_after_rotation": 3,
|
||||
"distinct_jobs": 3,
|
||||
"content_SHA256_preserved": true,
|
||||
"image_unchanged": true,
|
||||
"total_seconds": 35.61
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"dependencies": [{"name": "annotated-doc", "version": "0.0.5", "vulns": []}, {"name": "annotated-types", "version": "0.8.0", "vulns": []}, {"name": "anyio", "version": "4.15.1", "vulns": []}, {"name": "argon2-cffi", "version": "25.1.0", "vulns": []}, {"name": "argon2-cffi-bindings", "version": "26.1.0", "vulns": []}, {"name": "boto3", "version": "1.43.114", "vulns": []}, {"name": "botocore", "version": "1.43.114", "vulns": []}, {"name": "certifi", "version": "2026.10.23", "vulns": []}, {"name": "cffi", "version": "2.1.1", "vulns": []}, {"name": "charset-normalizer", "version": "3.5.2", "vulns": []}, {"name": "click", "version": "8.5.0", "vulns": []}, {"name": "fastapi", "version": "0.143.2", "vulns": []}, {"name": "googleapis-common-protos", "version": "1.75.5", "vulns": []}, {"name": "h11", "version": "0.16.0", "vulns": []}, {"name": "idna", "version": "3.20", "vulns": []}, {"name": "jmespath", "version": "1.1.0", "vulns": []}, {"name": "opentelemetry-api", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-exporter-http-transport", "version": "0.66b1", "vulns": []}, {"name": "opentelemetry-exporter-otlp-common", "version": "0.66b1", "vulns": []}, {"name": "opentelemetry-exporter-otlp-proto-common", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-exporter-otlp-proto-http", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-proto", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-sdk", "version": "1.45.1", "vulns": []}, {"name": "opentelemetry-semantic-conventions", "version": "0.66b1", "vulns": []}, {"name": "pillow", "version": "13.0.0", "vulns": []}, {"name": "protobuf", "version": "7.36.2", "vulns": []}, {"name": "psycopg", "version": "3.3.6", "vulns": []}, {"name": "psycopg-binary", "version": "3.3.6", "vulns": []}, {"name": "psycopg-pool", "version": "3.3.3", "vulns": []}, {"name": "pycparser", "version": "3.11", "vulns": []}, {"name": "pydantic", "version": "2.14.1", "vulns": []}, {"name": "pydantic-core", "version": "2.50.1", "vulns": []}, {"name": "python-dateutil", "version": "2.9.0.post0", "vulns": []}, {"name": "redis", "version": "8.1.0", "vulns": []}, {"name": "requests", "version": "2.34.2", "vulns": []}, {"name": "s3transfer", "version": "0.19.2", "vulns": []}, {"name": "six", "version": "1.17.0", "vulns": []}, {"name": "starlette", "version": "1.8.0", "vulns": []}, {"name": "typing-inspection", "version": "0.4.4", "vulns": []}, {"name": "typing-extensions", "version": "4.16.0", "vulns": []}, {"name": "tzdata", "version": "2026.5", "vulns": []}, {"name": "urllib3", "version": "2.8.0", "vulns": []}, {"name": "uvicorn", "version": "0.54.0", "vulns": []}], "fixes": []}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"target_instance": "b1a678ea159c4aaea77efbcc38b117e8",
|
||||
"stopped_container": "0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3",
|
||||
"before_containers": [
|
||||
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
|
||||
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
|
||||
],
|
||||
"after_running_containers": [
|
||||
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 77, "failed_attempts": 4, "first_recovery_seconds": 3.39}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"target_instance": "7f288f373c5140c3a3a1f8de74fa91f2",
|
||||
"stopped_container": "0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3",
|
||||
"before_containers": [
|
||||
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
|
||||
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
|
||||
],
|
||||
"after_running_containers": [
|
||||
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 83, "failed_attempts": 4, "first_recovery_seconds": 2.953}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"target_instance": "932b533b493e4dab8e09817e66f77901",
|
||||
"stopped_container": "1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
|
||||
"before_containers": [
|
||||
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
|
||||
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
|
||||
],
|
||||
"after_running_containers": [
|
||||
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 79, "failed_attempts": 4, "first_recovery_seconds": 2.61}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1 @@
|
||||
Negative test identities generated
|
||||
@@ -0,0 +1 @@
|
||||
PASS: runtime privileges/resource limits/mounts/public ports and loopback-only plaintext
|
||||
@@ -0,0 +1,502 @@
|
||||
{
|
||||
"passed": true,
|
||||
"runtime": [
|
||||
{
|
||||
"service": "api1",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 536870912,
|
||||
"NanoCpus": 500000000,
|
||||
"pids_limit": 64,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "api1-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "api2",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 536870912,
|
||||
"NanoCpus": 500000000,
|
||||
"pids_limit": 64,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "api2-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "broker",
|
||||
"user": "999:1000",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 134217728,
|
||||
"NanoCpus": 250000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "broker-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "cache",
|
||||
"user": "999:1000",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 250000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "cache-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "db",
|
||||
"user": "999:999",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 268435456,
|
||||
"NanoCpus": 1000000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "db-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "dispatcher",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 134217728,
|
||||
"NanoCpus": 250000000,
|
||||
"pids_limit": 64,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "dispatcher-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "grafana",
|
||||
"user": "472",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 536870912,
|
||||
"NanoCpus": 1000000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [
|
||||
"3000"
|
||||
],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "grafana-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "lb",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 500000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [
|
||||
"8443"
|
||||
],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "loki",
|
||||
"user": "10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 268435456,
|
||||
"NanoCpus": 500000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "loki-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "openbao",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 268435456,
|
||||
"NanoCpus": 500000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [
|
||||
"8200"
|
||||
],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "otel-collector",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 268435456,
|
||||
"NanoCpus": 500000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "otel-collector-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "prometheus",
|
||||
"user": "nobody",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 268435456,
|
||||
"NanoCpus": 500000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "prometheus-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "reconciler",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 201326592,
|
||||
"NanoCpus": 250000000,
|
||||
"pids_limit": 64,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "reconciler-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "s3",
|
||||
"user": "1000:1000",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 536870912,
|
||||
"NanoCpus": 1000000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "s3-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "tempo",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 268435456,
|
||||
"NanoCpus": 500000000,
|
||||
"pids_limit": 128,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "tempo-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "worker",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 805306368,
|
||||
"NanoCpus": 1000000000,
|
||||
"pids_limit": 64,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
},
|
||||
{
|
||||
"service": "worker-tls",
|
||||
"user": "10001:10001",
|
||||
"read_only": true,
|
||||
"cap_drop": [
|
||||
"ALL"
|
||||
],
|
||||
"memory_bytes": 67108864,
|
||||
"NanoCpus": 200000000,
|
||||
"pids_limit": 32,
|
||||
"published_loopback_ports": [],
|
||||
"operator_material_absent": true
|
||||
}
|
||||
],
|
||||
"grafana_verified_HTTPS": true,
|
||||
"plaintext_isolation": {
|
||||
"passed": true,
|
||||
"probes": [
|
||||
{
|
||||
"host": "api1",
|
||||
"port": 8080,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "api2",
|
||||
"port": 8080,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "db",
|
||||
"port": 5432,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "s3",
|
||||
"port": 8333,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "s3",
|
||||
"port": 8888,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "s3",
|
||||
"port": 9333,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "s3",
|
||||
"port": 19333,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "cache",
|
||||
"port": 6379,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "broker",
|
||||
"port": 6379,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "otel-collector",
|
||||
"port": 4318,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "otel-collector",
|
||||
"port": 9464,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "otel-collector",
|
||||
"port": 8888,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "otel-collector",
|
||||
"port": 13133,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "prometheus",
|
||||
"port": 9090,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "loki",
|
||||
"port": 3100,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "loki",
|
||||
"port": 9095,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "tempo",
|
||||
"port": 3200,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "tempo",
|
||||
"port": 4318,
|
||||
"refused": true
|
||||
},
|
||||
{
|
||||
"host": "tempo",
|
||||
"port": 9095,
|
||||
"refused": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,2 @@
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,9 @@
|
||||
scripts/check_contract.py: PASS
|
||||
scripts/validate_openapi.py: PASS
|
||||
tests/smoke.py: PASS
|
||||
tests/security.py: PASS
|
||||
persistence: PASS
|
||||
failover-1: PASS
|
||||
failover-2: PASS
|
||||
failover-3: PASS
|
||||
PASS: both distinct API containers failed over; all replicas restored
|
||||
@@ -0,0 +1 @@
|
||||
Operator action completed; no root token/unseal shares printed
|
||||
@@ -0,0 +1,5 @@
|
||||
....
|
||||
----------------------------------------------------------------------
|
||||
Ran 4 tests in 0.217s
|
||||
|
||||
OK
|
||||
@@ -0,0 +1,735 @@
|
||||
{
|
||||
"before": {
|
||||
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7": {
|
||||
"service": "api1",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2835,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:55.809852669Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-11-13T09:52:33.85158481Z",
|
||||
"End": "2026-11-13T09:52:34.414940821Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:39.416434594Z",
|
||||
"End": "2026-11-13T09:52:40.03112071Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:45.032672537Z",
|
||||
"End": "2026-11-13T09:52:45.684456356Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:50.685337932Z",
|
||||
"End": "2026-11-13T09:52:51.337051054Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:55.883682844Z",
|
||||
"End": "2026-11-13T09:52:56.745165826Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3": {
|
||||
"service": "api2",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2820,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:55.797519555Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-11-13T09:52:35.50023978Z",
|
||||
"End": "2026-11-13T09:52:35.92798761Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:40.929156411Z",
|
||||
"End": "2026-11-13T09:52:41.906797728Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:46.908309853Z",
|
||||
"End": "2026-11-13T09:52:47.31147321Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:51.873454972Z",
|
||||
"End": "2026-11-13T09:52:52.395517815Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:57.397494814Z",
|
||||
"End": "2026-11-13T09:52:57.897562413Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"76e0e9ee11f2acd4202aa4641b07c2af83558e6726dd26a3521b33bb520cf177": {
|
||||
"service": "db",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 755,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:20.83333691Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.225155745Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-11-13T09:52:50.379763185Z",
|
||||
"End": "2026-11-13T09:52:50.456684316Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:52.018791868Z",
|
||||
"End": "2026-11-13T09:52:52.157704126Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:54.159668931Z",
|
||||
"End": "2026-11-13T09:52:54.239425636Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:56.241459415Z",
|
||||
"End": "2026-11-13T09:52:56.374718752Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:58.375876362Z",
|
||||
"End": "2026-11-13T09:52:58.425063406Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"051a2a59f89f08acf5878b6ab22e2ad389bc09d4fa3d655604f27375b26922be": {
|
||||
"service": "cache",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 840,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:21.347055009Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.224207664Z"
|
||||
}
|
||||
},
|
||||
"c0e3fc648ea6a23b643a0104ad9bf93f06bfd45e592fece24d8c4dc8788dc494": {
|
||||
"service": "lb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2892,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:55.925530847Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.222150955Z"
|
||||
}
|
||||
},
|
||||
"68c83b0d029545c4c8bfa67979e23fda461d7b9aea909d5e99e462f4858fcc1a": {
|
||||
"service": "s3",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 863,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:21.411682013Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.225741755Z"
|
||||
}
|
||||
},
|
||||
"6db578724c16067c9101d8ffb911ae3bb1127c087e0b8a598c958983914676a7": {
|
||||
"service": "reconciler",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2676,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:55.784764196Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
}
|
||||
},
|
||||
"362e538c9c53fa7feb054689ed85f862d070518a168c0efaa6e184ad51bd34f5": {
|
||||
"service": "broker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 879,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:21.392114019Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.224003528Z"
|
||||
}
|
||||
},
|
||||
"2e513af66f171401771b1a3680154f7edfc7472a5ee79e94041256efee78b2f0": {
|
||||
"service": "worker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2709,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:55.951329668Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
}
|
||||
},
|
||||
"c40815a4d87373db33cf7eb4b89d473b853dd7a9ece81fa73e052da54ced5ff4": {
|
||||
"service": "dispatcher",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2629,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:55.684909892Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
}
|
||||
},
|
||||
"ea547f7d390f190c74b9f60d5be466be91924df7ead9930e916f0e5bf9161ae0": {
|
||||
"service": "openbao",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 940,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:21.574158786Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.217091485Z"
|
||||
}
|
||||
},
|
||||
"c605ad2ea7cfb8fdc37b85d5e8df440c0f9dfb1ba8317a8b87d84ce5ff38e2cb": {
|
||||
"service": "otel-collector",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2156,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:52.584501439Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.224053739Z"
|
||||
}
|
||||
},
|
||||
"9b54b82ecd93b601a0ea977adedef1b81a1ff36eb925ab730684320a22113141": {
|
||||
"service": "prometheus",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2352,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:53.97192011Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.223361176Z"
|
||||
}
|
||||
},
|
||||
"72ece8cb2659be6a83a2516dcb19b472d4ab9d6d1b50f20eca4f49799cd17131": {
|
||||
"service": "loki",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2359,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:54.028034454Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.224411699Z"
|
||||
}
|
||||
},
|
||||
"574869ba08c1a34d0d30fa2da1e61c5a35dd66ee35afff36d18366e33658e621": {
|
||||
"service": "tempo",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2288,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:53.935285487Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.224318796Z"
|
||||
}
|
||||
},
|
||||
"4ed391665684b152c494235932554b04ed24e82fc7213219f6cfc1dc54c3e4db": {
|
||||
"service": "grafana",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2946,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:51:56.097459745Z",
|
||||
"FinishedAt": "2026-11-13T09:50:28.225512752Z"
|
||||
}
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7": {
|
||||
"service": "api1",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6310,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.810798832Z",
|
||||
"FinishedAt": "2026-11-13T09:53:06.716904603Z",
|
||||
"Health": {
|
||||
"Status": "starting",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-11-13T09:52:45.032672537Z",
|
||||
"End": "2026-11-13T09:52:45.684456356Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:50.685337932Z",
|
||||
"End": "2026-11-13T09:52:51.337051054Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:55.883682844Z",
|
||||
"End": "2026-11-13T09:52:56.745165826Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:53:01.746116416Z",
|
||||
"End": "2026-11-13T09:53:02.250230395Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:53:21.941393572Z",
|
||||
"End": "2026-11-13T09:53:25.757437396Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3": {
|
||||
"service": "api2",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6454,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.906757854Z",
|
||||
"FinishedAt": "2026-11-13T09:53:08.979949677Z",
|
||||
"Health": {
|
||||
"Status": "starting",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-11-13T09:52:46.908309853Z",
|
||||
"End": "2026-11-13T09:52:47.31147321Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:51.873454972Z",
|
||||
"End": "2026-11-13T09:52:52.395517815Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:52:57.397494814Z",
|
||||
"End": "2026-11-13T09:52:57.897562413Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:53:02.902817408Z",
|
||||
"End": "2026-11-13T09:53:06.233524775Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:53:22.46715569Z",
|
||||
"End": "2026-11-13T09:53:25.869837051Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"76e0e9ee11f2acd4202aa4641b07c2af83558e6726dd26a3521b33bb520cf177": {
|
||||
"service": "db",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 5985,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:14.810659332Z",
|
||||
"FinishedAt": "2026-11-13T09:53:05.266445865Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-11-13T09:53:02.49033307Z",
|
||||
"End": "2026-11-13T09:53:02.543395178Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:53:04.54387538Z",
|
||||
"End": "2026-11-13T09:53:05.068330917Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "/var/run/postgresql:5432 - rejecting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:53:18.178982393Z",
|
||||
"End": "2026-11-13T09:53:20.786679419Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:53:22.800575887Z",
|
||||
"End": "2026-11-13T09:53:24.416791098Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-11-13T09:53:26.420377151Z",
|
||||
"End": "2026-11-13T09:53:26.842575441Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"051a2a59f89f08acf5878b6ab22e2ad389bc09d4fa3d655604f27375b26922be": {
|
||||
"service": "cache",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6487,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.945227154Z",
|
||||
"FinishedAt": "2026-11-13T09:53:05.04199852Z"
|
||||
}
|
||||
},
|
||||
"c0e3fc648ea6a23b643a0104ad9bf93f06bfd45e592fece24d8c4dc8788dc494": {
|
||||
"service": "lb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6479,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.9062729Z",
|
||||
"FinishedAt": "2026-11-13T09:53:03.909611163Z"
|
||||
}
|
||||
},
|
||||
"68c83b0d029545c4c8bfa67979e23fda461d7b9aea909d5e99e462f4858fcc1a": {
|
||||
"service": "s3",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6375,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.849272466Z",
|
||||
"FinishedAt": "2026-11-13T09:53:10.186208925Z"
|
||||
}
|
||||
},
|
||||
"6db578724c16067c9101d8ffb911ae3bb1127c087e0b8a598c958983914676a7": {
|
||||
"service": "reconciler",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6191,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.708018867Z",
|
||||
"FinishedAt": "2026-11-13T09:53:09.175300496Z"
|
||||
}
|
||||
},
|
||||
"362e538c9c53fa7feb054689ed85f862d070518a168c0efaa6e184ad51bd34f5": {
|
||||
"service": "broker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6125,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.753030573Z",
|
||||
"FinishedAt": "2026-11-13T09:53:05.167583954Z"
|
||||
}
|
||||
},
|
||||
"2e513af66f171401771b1a3680154f7edfc7472a5ee79e94041256efee78b2f0": {
|
||||
"service": "worker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6437,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.870744073Z",
|
||||
"FinishedAt": "2026-11-13T09:53:09.192037057Z"
|
||||
}
|
||||
},
|
||||
"c40815a4d87373db33cf7eb4b89d473b853dd7a9ece81fa73e052da54ced5ff4": {
|
||||
"service": "dispatcher",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6311,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.806673713Z",
|
||||
"FinishedAt": "2026-11-13T09:53:09.581490205Z"
|
||||
}
|
||||
},
|
||||
"ea547f7d390f190c74b9f60d5be466be91924df7ead9930e916f0e5bf9161ae0": {
|
||||
"service": "openbao",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6517,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.911866476Z",
|
||||
"FinishedAt": "2026-11-13T09:53:03.972241104Z"
|
||||
}
|
||||
},
|
||||
"c605ad2ea7cfb8fdc37b85d5e8df440c0f9dfb1ba8317a8b87d84ce5ff38e2cb": {
|
||||
"service": "otel-collector",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 5717,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:13.768718976Z",
|
||||
"FinishedAt": "2026-11-13T09:53:05.062618953Z"
|
||||
}
|
||||
},
|
||||
"9b54b82ecd93b601a0ea977adedef1b81a1ff36eb925ab730684320a22113141": {
|
||||
"service": "prometheus",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 5926,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:14.756925187Z",
|
||||
"FinishedAt": "2026-11-13T09:53:05.244841875Z"
|
||||
}
|
||||
},
|
||||
"72ece8cb2659be6a83a2516dcb19b472d4ab9d6d1b50f20eca4f49799cd17131": {
|
||||
"service": "loki",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 5961,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:14.793956064Z",
|
||||
"FinishedAt": "2026-11-13T09:53:05.416062037Z"
|
||||
}
|
||||
},
|
||||
"574869ba08c1a34d0d30fa2da1e61c5a35dd66ee35afff36d18366e33658e621": {
|
||||
"service": "tempo",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 5958,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:14.761611926Z",
|
||||
"FinishedAt": "2026-11-13T09:53:05.299312685Z"
|
||||
}
|
||||
},
|
||||
"4ed391665684b152c494235932554b04ed24e82fc7213219f6cfc1dc54c3e4db": {
|
||||
"service": "grafana",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6442,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-11-13T09:53:15.90242133Z",
|
||||
"FinishedAt": "2026-11-13T09:53:05.770095425Z"
|
||||
}
|
||||
}
|
||||
},
|
||||
"volumes_removed": false
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
Operator action completed; no root token/unseal shares printed
|
||||
PASS: sealed Bao -> finite lease -> both API children stopped -> unseal -> recovery
|
||||
@@ -0,0 +1,254 @@
|
||||
{
|
||||
"passed": true,
|
||||
"secret_refresh_seconds": 20,
|
||||
"token_max_TTL_seconds": 120,
|
||||
"both_API_children_stopped_seconds": 123.86,
|
||||
"recovery_seconds": 20.44,
|
||||
"observations": [
|
||||
{
|
||||
"seconds": 1.3,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 3.3,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 5.4,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 7.4,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 9.4,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 11.5,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 13.5,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 15.5,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 17.5,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 19.6,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 21.6,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 23.6,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 25.7,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 27.7,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 29.7,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 31.8,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 33.8,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 35.8,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 37.9,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 39.9,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 42.0,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 44.0,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 46.0,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 48.0,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 50.1,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 52.1,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 54.1,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 56.2,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 58.2,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 60.2,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 62.3,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 64.3,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 66.3,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 68.4,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 70.4,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 72.4,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 74.5,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 76.5,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 78.5,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 80.5,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 82.6,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 84.6,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 86.7,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 88.7,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 90.8,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 92.8,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 94.8,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 96.8,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 98.8,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 100.9,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 102.9,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 104.9,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 107.0,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 109.0,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 111.0,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 113.1,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 115.1,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 117.1,
|
||||
"HTTP": 200
|
||||
},
|
||||
{
|
||||
"seconds": 119.2,
|
||||
"HTTP": 502
|
||||
},
|
||||
{
|
||||
"seconds": 121.2,
|
||||
"HTTP": 502
|
||||
},
|
||||
{
|
||||
"seconds": 123.2,
|
||||
"HTTP": 503
|
||||
}
|
||||
],
|
||||
"scope": "Supervisor bounds use during Bao outage; static backend passwords themselves require explicit rotation."
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: ACL, expiry/logout, hash storage/rehash, shared 12/10s rate limit, measured Argon2
|
||||
@@ -0,0 +1,131 @@
|
||||
{
|
||||
"passed": true,
|
||||
"ACL_attachment_denials": 4,
|
||||
"expired_session_denied": true,
|
||||
"logout_denied": true,
|
||||
"token_hash_sha256": true,
|
||||
"rehash": "19456 -> 32768 KiB",
|
||||
"login_burst": [
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 2.891,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 429,
|
||||
"seconds": 0.453,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 4.078,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 3.094,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 429,
|
||||
"seconds": 0.281,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 3.188,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 3.375,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 429,
|
||||
"seconds": 0.281,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 2.11,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 3.891,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 4.547,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 4.062,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 3.781,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 429,
|
||||
"seconds": 0.5,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
},
|
||||
{
|
||||
"status": 429,
|
||||
"seconds": 0.531,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 3.125,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
},
|
||||
{
|
||||
"status": 429,
|
||||
"seconds": 0.39,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
},
|
||||
{
|
||||
"status": 401,
|
||||
"seconds": 4.453,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
},
|
||||
{
|
||||
"status": 429,
|
||||
"seconds": 0.39,
|
||||
"instance": "9a29628cf7004fbfb295d86c41370777"
|
||||
},
|
||||
{
|
||||
"status": 429,
|
||||
"seconds": 0.453,
|
||||
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
|
||||
}
|
||||
],
|
||||
"burst_seconds": 4.578,
|
||||
"KDF": {
|
||||
"hash_plus_verify_seconds": [
|
||||
0.21583701900004826,
|
||||
0.20667367100003275,
|
||||
0.22015716700002486,
|
||||
0.193256392999956,
|
||||
0.20241064300000744,
|
||||
0.20739852500003053,
|
||||
0.20553775699994503,
|
||||
0.2716375199999561,
|
||||
0.2128679730000158,
|
||||
0.22063045700002704,
|
||||
0.20332950599998867,
|
||||
0.20581266300001744
|
||||
],
|
||||
"median": 0.20703609800003164,
|
||||
"max_rss_kib": 53184
|
||||
},
|
||||
"KDF_slots_per_API": 2,
|
||||
"KDF_memory_bound_two_APIs_MiB": 128
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"passed": true,
|
||||
"completed_at_UTC": "2026-11-13T10:00:37.671336+00:00"
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
generate-test-identities: PASS
|
||||
publish-crl: PASS
|
||||
tls-matrix: PASS
|
||||
bao-policy: PASS
|
||||
worker-rights: PASS
|
||||
security-api-run: PASS
|
||||
compromised-worker-run: PASS
|
||||
sealed-bao-run: PASS
|
||||
telemetry-security-run: PASS
|
||||
hardening-run: PASS
|
||||
render-unit: PASS
|
||||
@@ -0,0 +1 @@
|
||||
PASS: trusted HTTPS works; untrusted CA rejected. This does NOT verify internal mTLS, OpenBao or hardening.
|
||||
@@ -0,0 +1 @@
|
||||
PASS: lab7 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"target_instance": "932b533b493e4dab8e09817e66f77901",
|
||||
"stopped_container": "1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
|
||||
"before_containers": [
|
||||
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
|
||||
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
|
||||
],
|
||||
"after_running_containers": [
|
||||
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: TLS metrics/traces/log correlation, credential canaries absent in telemetry/container logs
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"passed": true,
|
||||
"trace_id": "918070f9c17e4f2b9b8e079df1824a06",
|
||||
"span_count": 13,
|
||||
"span_names": [
|
||||
"POST /api/v1/chats/{chat_id}/attachments",
|
||||
"attachment.consume",
|
||||
"attachment.durable_accept",
|
||||
"attachment.publish",
|
||||
"db.attachment.commit",
|
||||
"db.session.authenticate",
|
||||
"image.render",
|
||||
"s3.get",
|
||||
"s3.put"
|
||||
],
|
||||
"correlated_log_count": 3,
|
||||
"http_metric_series": 26,
|
||||
"credential_canaries_absent": true,
|
||||
"all_queries": "mTLS verified operator identity",
|
||||
"container_logs_canaries_absent": true
|
||||
}
|
||||
@@ -0,0 +1,397 @@
|
||||
{
|
||||
"passed": true,
|
||||
"seconds": 2.49,
|
||||
"edges": [
|
||||
{
|
||||
"source": "lb",
|
||||
"destination": "api1",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "api1",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "lb",
|
||||
"destination": "api2",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "api2",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "worker",
|
||||
"destination": "broker",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "dispatcher",
|
||||
"destination": "broker",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "broker",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "broker",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "api1",
|
||||
"destination": "cache",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "api2",
|
||||
"destination": "cache",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "cache",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "cache",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "api1",
|
||||
"destination": "db",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "api2",
|
||||
"destination": "db",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "worker",
|
||||
"destination": "db",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "dispatcher",
|
||||
"destination": "db",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "reconciler",
|
||||
"destination": "db",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "db",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "db",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "otel-collector",
|
||||
"destination": "loki",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "grafana",
|
||||
"destination": "loki",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "loki",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "loki",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "api1",
|
||||
"destination": "otel-collector",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "api2",
|
||||
"destination": "otel-collector",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "worker",
|
||||
"destination": "otel-collector",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "dispatcher",
|
||||
"destination": "otel-collector",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "reconciler",
|
||||
"destination": "otel-collector",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "otel-collector",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "otel-collector",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "prometheus",
|
||||
"destination": "otel-collector",
|
||||
"port": 8444,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "otel-collector",
|
||||
"port": 8444,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "otel-collector",
|
||||
"port": 8444,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "prometheus",
|
||||
"destination": "otel-collector",
|
||||
"port": 8445,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "otel-collector",
|
||||
"port": 8445,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "otel-collector",
|
||||
"port": 8445,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "grafana",
|
||||
"destination": "prometheus",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "prometheus",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "prometheus",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "api1",
|
||||
"destination": "s3",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "api2",
|
||||
"destination": "s3",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "worker",
|
||||
"destination": "s3",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "reconciler",
|
||||
"destination": "s3",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "s3",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "s3",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "otel-collector",
|
||||
"destination": "tempo",
|
||||
"port": 8443,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "tempo",
|
||||
"port": 8443,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "grafana",
|
||||
"destination": "tempo",
|
||||
"port": 8444,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"source": "operator",
|
||||
"destination": "tempo",
|
||||
"port": 8444,
|
||||
"allowed": true
|
||||
},
|
||||
{
|
||||
"destination": "tempo",
|
||||
"port": 8444,
|
||||
"denied": [
|
||||
"no_cert",
|
||||
"wrong_CA",
|
||||
"wrong_SAN",
|
||||
"expired",
|
||||
"revoked",
|
||||
"wrong_server_SAN",
|
||||
"plaintext"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"passed": true,
|
||||
"rights": {
|
||||
"jobs_read": "allowed",
|
||||
"sessions": "denied",
|
||||
"password_hashes": "denied",
|
||||
"messages": "denied",
|
||||
"schema": "denied",
|
||||
"privilege": "denied",
|
||||
"variants_read_write_delete": "allowed",
|
||||
"foreign_prefix": "denied",
|
||||
"bucket_admin": "denied",
|
||||
"queue_group": "allowed",
|
||||
"queue_publish": "denied",
|
||||
"foreign_queue": "denied",
|
||||
"queue_admin": "denied"
|
||||
}
|
||||
}
|
||||
+16
-4
@@ -1,7 +1,19 @@
|
||||
# Инфраструктура студента
|
||||
# Infrastructure ЛР7
|
||||
|
||||
Здесь разместите конфигурации выбранных сервисов; основную топологию — в корневом compose.yaml/stack.yaml. `compose.example.yaml` — только worksheet ролей и намеренно не запускается. `contracts/infrastructure.json` задаёт минимальные роли, не готовые образы/пароли.
|
||||
Исполняемый стенд: ../compose.yaml. Полная инструкция: ../REPORT.md.
|
||||
Все versions закреплены digest. Подготовка: `python scripts/up.py` из корня.
|
||||
|
||||
В REPORT.md сопоставьте роль → service name → версия/digest → health/readiness → volume → сеть/порт → способ передачи credentials. Приведите bootstrap (schema, bucket, broker topology, а с lab6 provisioning) и безопасное повторение этих команд.
|
||||
- api1/api2 — non-root ASGI + cert-auth secret supervisor, public LB только HTTPS8443.
|
||||
- *-tls — отдельные identities, mTLS SAN allowlist, CA+CRL; shared namespace с backend.
|
||||
- db — PostgreSQL/SCRAM, named pgdata, операторские migrations/GRANT.
|
||||
- cache/broker — разные Valkey ACL/configs; persistent AOF только у broker.
|
||||
- s3 — private SeaweedFS bucket, раздельные SigV4 users/prefix ACL, named s3data.
|
||||
- openbao — native TLS+client verification, persistent raft, declarative audit.
|
||||
- telemetry — Collector/Prometheus/Loki/Tempo/Grafana, persistent backend volumes.
|
||||
- secret-init/telemetry-init/storage-init — ограниченные операторские bootstrap jobs.
|
||||
|
||||
Приложение строится из исходников. Не используйте заранее существующий образ преподавательского сервера вместо своей реализации. Restart hook перезапускает все соответствующие stores с volumes; failover hook останавливает только заданную API-реплику.
|
||||
Матрица соединений: ../contracts/trust-matrix.md. Native plaintext listener
|
||||
закрыт на loopback, TLS hop существует на каждом межконтейнерном соединении.
|
||||
`docker compose restart api1` отдельно от sidecar не является штатным workflow:
|
||||
используйте scripts/restart.py для согласованного stop/start shared namespaces.
|
||||
Volume deletion не входит ни в запуск, ни в тесты.
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
groups:
|
||||
- name: messenger-slo
|
||||
interval: 5s
|
||||
rules:
|
||||
- alert: APIAvailabilityBurn
|
||||
expr: |
|
||||
(sum(rate(http_server_requests_total{status_class="5xx"}[20s])) / clamp_min(sum(rate(http_server_requests_total[20s])), 0.001) / 0.005 > 10)
|
||||
and
|
||||
(sum(rate(http_server_requests_total{status_class="5xx"}[60s])) / clamp_min(sum(rate(http_server_requests_total[60s])), 0.001) / 0.005 > 10)
|
||||
for: 30s
|
||||
labels: {severity: critical}
|
||||
annotations:
|
||||
summary: "API 99.5% availability budget burns >10x in both 20s and 60s windows"
|
||||
runbook: "RUNBOOK.md#api"
|
||||
- alert: AttachmentReadyBurn
|
||||
expr: |
|
||||
(sum(attachment_slo_bad{window="60"}) / clamp_min(sum(attachment_slo_eligible{window="60"}),1) / 0.01 > 10)
|
||||
and
|
||||
(sum(attachment_slo_bad{window="300"}) / clamp_min(sum(attachment_slo_eligible{window="300"}),1) / 0.01 > 10)
|
||||
for: 15s
|
||||
labels: {severity: critical}
|
||||
annotations:
|
||||
summary: "99% ready within 30s budget burns >10x in both 60s and 300s eligible cohorts"
|
||||
runbook: "RUNBOOK.md#queue"
|
||||
- alert: QueueLag
|
||||
expr: max(queue_oldest_age_seconds) > 30 and sum(queue_depth) > 0
|
||||
for: 10s
|
||||
labels: {severity: warning}
|
||||
annotations:
|
||||
summary: "An unfinished durable job has waited over 30s"
|
||||
runbook: "RUNBOOK.md#queue"
|
||||
@@ -0,0 +1,669 @@
|
||||
{
|
||||
"uid": "mtusi-messenger",
|
||||
"title": "MTUSI Messenger - signals and SLO",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"editable": false,
|
||||
"timezone": "browser",
|
||||
"time": {
|
||||
"from": "now-15m",
|
||||
"to": "now"
|
||||
},
|
||||
"refresh": "15s",
|
||||
"tags": [
|
||||
"mtusi",
|
||||
"lab6"
|
||||
],
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "timeseries",
|
||||
"title": "API requests / second",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by(method,route)(rate(http_server_requests_total[1m]))",
|
||||
"legendFormat": "{{method}} {{route}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "timeseries",
|
||||
"title": "API 5xx / all business requests",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)",
|
||||
"legendFormat": "5xx fraction"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "timeseries",
|
||||
"title": "API latency: aggregate buckets across replicas",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "timeseries",
|
||||
"title": "Jobs / second by outcome",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by(outcome)(rate(jobs_processed_total[1m]))",
|
||||
"legendFormat": "{{outcome}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Worker processing latency",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Acceptance to ready latency",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "timeseries",
|
||||
"title": "Unfinished durable jobs",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(queue_depth)",
|
||||
"legendFormat": "queue depth"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "timeseries",
|
||||
"title": "Oldest unfinished job age",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(queue_oldest_age_seconds)",
|
||||
"legendFormat": "oldest"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "timeseries",
|
||||
"title": "DB pool",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "db_pool_used",
|
||||
"legendFormat": "used {{process_instance}}"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "db_pool_waiting",
|
||||
"legendFormat": "waiting {{process_instance}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"type": "timeseries",
|
||||
"title": "Worker saturation",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(worker_busy) / clamp_min(sum(worker_capacity),1)",
|
||||
"legendFormat": "busy / capacity"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"type": "timeseries",
|
||||
"title": "Current process RSS",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "process_rss_bytes",
|
||||
"legendFormat": "{{service}} {{process_instance}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "timeseries",
|
||||
"title": "Process CPU seconds / wall second",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "clamp_min(deriv(process_cpu_time_seconds[1m]),0)",
|
||||
"legendFormat": "{{service}} {{process_instance}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "timeseries",
|
||||
"title": "API availability burn rate; budget 0.5%",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[20s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[20s])),0.001) / 0.005",
|
||||
"legendFormat": "20s"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[60s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[60s])),0.001) / 0.005",
|
||||
"legendFormat": "60s"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"type": "timeseries",
|
||||
"title": "Ready <=30s burn rate; budget 1%",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(attachment_slo_bad{window=\"60\"}) / clamp_min(sum(attachment_slo_eligible{window=\"60\"}),1) / 0.01",
|
||||
"legendFormat": "60s eligible cohort"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "sum(attachment_slo_bad{window=\"300\"}) / clamp_min(sum(attachment_slo_eligible{window=\"300\"}),1) / 0.01",
|
||||
"legendFormat": "300s eligible cohort"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"type": "timeseries",
|
||||
"title": "Collector exporter queue / capacity",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "otelcol_exporter_queue_size",
|
||||
"legendFormat": "{{exporter}} size"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "otelcol_exporter_queue_capacity",
|
||||
"legendFormat": "{{exporter}} capacity"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"type": "timeseries",
|
||||
"title": "Collector failed exports",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(otelcol_exporter_send_failed_spans_total[1m])",
|
||||
"legendFormat": "{{exporter}} spans"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "rate(otelcol_exporter_send_failed_log_records_total[1m])",
|
||||
"legendFormat": "{{exporter}} logs"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 99,
|
||||
"type": "logs",
|
||||
"title": "Structured application logs; TraceID links to Tempo",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 64,
|
||||
"w": 24,
|
||||
"h": 10
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "{service_name=~\"messenger-(api|worker|dispatcher)\"}"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"templating": {
|
||||
"list": []
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: Messenger
|
||||
folder: MTUSI
|
||||
type: file
|
||||
disableDeletion: true
|
||||
updateIntervalSeconds: 30
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,31 @@
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
uid: prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://127.0.0.1:19090
|
||||
isDefault: true
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://127.0.0.1:13100
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: TraceID
|
||||
matcherRegex: '"trace_id": "([0-9a-f]{32})"'
|
||||
datasourceUid: tempo
|
||||
url: '$${__value.raw}'
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://127.0.0.1:13200
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: '-1m'
|
||||
spanEndTimeShift: '1m'
|
||||
filterByTraceID: true
|
||||
nodeGraph: {enabled: true}
|
||||
@@ -0,0 +1,32 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 256
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
timeout connect 1s
|
||||
timeout client 35s
|
||||
timeout server 32s
|
||||
timeout check 1s
|
||||
timeout queue 2s
|
||||
retries 0
|
||||
option http-server-close
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
hold obsolete 1s
|
||||
hold nx 1s
|
||||
hold timeout 1s
|
||||
frontend public
|
||||
bind :8443 ssl crt /identity/combined.pem ssl-min-ver TLSv1.2
|
||||
default_backend api
|
||||
backend api
|
||||
balance roundrobin
|
||||
option log-health-checks
|
||||
option httpchk GET /health/ready
|
||||
http-check expect status 200
|
||||
server api1 api1:8443 ssl verify required ca-file /trust/ca.crt crl-file /trust/revocations.pem crt /identity/combined.pem verifyhost api1 sni str(api1) check-sni api1 check inter 1s resolvers docker init-addr none
|
||||
server api2 api2:8443 ssl verify required ca-file /trust/ca.crt crl-file /trust/revocations.pem crt /identity/combined.pem verifyhost api2 sni str(api2) check-sni api2 check inter 1s resolvers docker init-addr none
|
||||
@@ -0,0 +1,36 @@
|
||||
auth_enabled: false
|
||||
server:
|
||||
http_listen_address: 127.0.0.1
|
||||
grpc_listen_address: 127.0.0.1
|
||||
http_listen_port: 3100
|
||||
common:
|
||||
instance_addr: 127.0.0.1
|
||||
path_prefix: /var/loki
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /var/loki/chunks
|
||||
rules_directory: /var/loki/rules
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2026-01-01
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
limits_config:
|
||||
allow_structured_metadata: true
|
||||
retention_period: 24h
|
||||
ingestion_rate_mb: 2
|
||||
ingestion_burst_size_mb: 4
|
||||
compactor:
|
||||
working_directory: /var/loki/compactor
|
||||
retention_enabled: true
|
||||
delete_request_store: filesystem
|
||||
analytics:
|
||||
reporting_enabled: false
|
||||
@@ -0,0 +1,34 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 8080,
|
||||
"allow": [
|
||||
"lb"
|
||||
]
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 5432,
|
||||
"host": "db",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 8333,
|
||||
"host": "s3",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 4318,
|
||||
"host": "otel-collector",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 6379,
|
||||
"host": "cache",
|
||||
"target": 8443
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,34 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 8080,
|
||||
"allow": [
|
||||
"lb"
|
||||
]
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 5432,
|
||||
"host": "db",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 8333,
|
||||
"host": "s3",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 4318,
|
||||
"host": "otel-collector",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 6379,
|
||||
"host": "cache",
|
||||
"target": 8443
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,12 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 6379,
|
||||
"allow": [
|
||||
"worker",
|
||||
"dispatcher",
|
||||
"operator"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,12 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 6379,
|
||||
"allow": [
|
||||
"api1",
|
||||
"api2",
|
||||
"operator"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,15 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 5432,
|
||||
"allow": [
|
||||
"api1",
|
||||
"api2",
|
||||
"worker",
|
||||
"dispatcher",
|
||||
"reconciler",
|
||||
"operator"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,20 @@
|
||||
[
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 5432,
|
||||
"host": "db",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 4318,
|
||||
"host": "otel-collector",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 6380,
|
||||
"host": "broker",
|
||||
"target": 8443
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,20 @@
|
||||
[
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 19090,
|
||||
"host": "prometheus",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 13100,
|
||||
"host": "loki",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 13200,
|
||||
"host": "tempo",
|
||||
"target": 8444
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,12 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 3100,
|
||||
"allow": [
|
||||
"otel-collector",
|
||||
"grafana",
|
||||
"operator"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,45 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 4318,
|
||||
"allow": [
|
||||
"api1",
|
||||
"api2",
|
||||
"worker",
|
||||
"dispatcher",
|
||||
"reconciler",
|
||||
"operator"
|
||||
]
|
||||
},
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8444,
|
||||
"target": 9464,
|
||||
"allow": [
|
||||
"prometheus",
|
||||
"operator"
|
||||
]
|
||||
},
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8445,
|
||||
"target": 8888,
|
||||
"allow": [
|
||||
"prometheus",
|
||||
"operator"
|
||||
]
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 14318,
|
||||
"host": "tempo",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 13100,
|
||||
"host": "loki",
|
||||
"target": 8443
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,23 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 9090,
|
||||
"allow": [
|
||||
"grafana",
|
||||
"operator"
|
||||
]
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 19464,
|
||||
"host": "otel-collector",
|
||||
"target": 8444
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 18888,
|
||||
"host": "otel-collector",
|
||||
"target": 8445
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,20 @@
|
||||
[
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 5432,
|
||||
"host": "db",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 8333,
|
||||
"host": "s3",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 4318,
|
||||
"host": "otel-collector",
|
||||
"target": 8443
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,14 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 8333,
|
||||
"allow": [
|
||||
"api1",
|
||||
"api2",
|
||||
"worker",
|
||||
"reconciler",
|
||||
"operator"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,19 @@
|
||||
[
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8443,
|
||||
"target": 4318,
|
||||
"allow": [
|
||||
"otel-collector"
|
||||
]
|
||||
},
|
||||
{
|
||||
"mode": "server",
|
||||
"listen": 8444,
|
||||
"target": 3200,
|
||||
"allow": [
|
||||
"grafana",
|
||||
"operator"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,26 @@
|
||||
[
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 5432,
|
||||
"host": "db",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 8333,
|
||||
"host": "s3",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 4318,
|
||||
"host": "otel-collector",
|
||||
"target": 8443
|
||||
},
|
||||
{
|
||||
"mode": "client",
|
||||
"listen": 6380,
|
||||
"host": "broker",
|
||||
"target": 8443
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,21 @@
|
||||
ui = false
|
||||
disable_mlock = true
|
||||
api_addr = "https://openbao:8200"
|
||||
cluster_addr = "https://127.0.0.1:8201"
|
||||
audit "file" "operator" {
|
||||
options {
|
||||
file_path = "/bao/audit.json"
|
||||
}
|
||||
}
|
||||
storage "raft" {
|
||||
path = "/bao/data"
|
||||
node_id = "lab7"
|
||||
}
|
||||
listener "tcp" {
|
||||
address = "0.0.0.0:8200"
|
||||
tls_cert_file = "/identity/cert.pem"
|
||||
tls_key_file = "/identity/key.pem"
|
||||
tls_client_ca_file = "/trust/ca.crt"
|
||||
tls_require_and_verify_client_cert = true
|
||||
tls_min_version = "tls12"
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 127.0.0.1:4318
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 192
|
||||
spike_limit_mib: 48
|
||||
batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 64
|
||||
send_batch_max_size: 128
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: 127.0.0.1:9464
|
||||
enable_open_metrics: true
|
||||
metric_expiration: 2m
|
||||
otlphttp/tempo:
|
||||
endpoint: http://127.0.0.1:14318
|
||||
timeout: 2s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 32
|
||||
retry_on_failure:
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 30s
|
||||
otlphttp/loki:
|
||||
endpoint: http://127.0.0.1:13100/otlp
|
||||
timeout: 2s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 32
|
||||
retry_on_failure:
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 30s
|
||||
extensions:
|
||||
health_check:
|
||||
endpoint: 127.0.0.1:13133
|
||||
service:
|
||||
extensions: [health_check]
|
||||
telemetry:
|
||||
metrics:
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus:
|
||||
host: 127.0.0.1
|
||||
port: 8888
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/tempo]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [prometheus]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/loki]
|
||||
+12
-5
@@ -1,7 +1,14 @@
|
||||
# Конфигурация наблюдаемости — реализовать
|
||||
# Telemetry ЛР7
|
||||
|
||||
Добавьте `collector.yaml` с OTLP receivers, processors и exporters выбранной версии Collector. Укажите явные bind endpoints, внутренние сети, memory limits/batch и поведение при backpressure по уровню. Сопоставьте все три pipeline с реально развёрнутыми backend.
|
||||
Перенесена реализация ЛР6. Collector принимает OTLP HTTP только на loopback;
|
||||
сеть доступна через otel-collector:8443 с mTLS и SAN allowlist. Native экспорт
|
||||
Collector → Tempo/Loki идёт в локальные tunnels, которые проверяют server CA/SAN
|
||||
и предъявляют identity Collector. Prometheus scrape и Grafana queries также
|
||||
используют разные mTLS identities. Backend ports не публикуются на хосте.
|
||||
|
||||
Добавьте конфигурации storage, Grafana provisioning/datasources, provisioning/dashboards и импортируемые dashboard JSON. На 4 — alert rules и runbooks. Пустой dashboard или debug exporter, который только печатает данные в stdout, не заменяют хранилища/доказательства из `contracts/observability.md`.
|
||||
|
||||
Не публикуйте OTLP/metrics/административные ports в общую сеть без необходимости. В lab7 реализуйте TLS/mTLS по trust matrix; экспорт Authorization и message bodies запрещён.
|
||||
Provisioned dashboard: infra/grafana/dashboards/messenger.json. Grafana доступна
|
||||
на https://localhost:3000, пароль admin создаёт bootstrap; анонимного доступа нет.
|
||||
Секреты, bodies и Authorization не включены в application spans/logs.
|
||||
`python checks/telemetry_security.py` проверяет upload trace, correlated logs,
|
||||
HTTP metric series и отсутствие известных credential canaries. Подробности
|
||||
в REPORT.md и contracts/trust-matrix.md.
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
# Operator-only Unix socket; no socket is shared with app containers.
|
||||
local all messenger trust
|
||||
local all all reject
|
||||
# Plaintext exists only in the db TLS sidecar's shared loopback namespace.
|
||||
# The TLS boundary enforces the peer identity; PostgreSQL ALSO verifies SCRAM.
|
||||
host all all 127.0.0.1/32 scram-sha-256
|
||||
host all all ::1/128 reject
|
||||
host all all 0.0.0.0/0 reject
|
||||
@@ -0,0 +1,12 @@
|
||||
global:
|
||||
scrape_interval: 5s
|
||||
evaluation_interval: 5s
|
||||
rule_files:
|
||||
- /etc/prometheus/alerts.yaml
|
||||
scrape_configs:
|
||||
- job_name: messenger-otlp
|
||||
static_configs:
|
||||
- targets: [127.0.0.1:19464]
|
||||
- job_name: collector
|
||||
static_configs:
|
||||
- targets: [127.0.0.1:18888]
|
||||
@@ -0,0 +1,28 @@
|
||||
server:
|
||||
http_listen_address: 127.0.0.1
|
||||
grpc_listen_address: 127.0.0.1
|
||||
http_listen_port: 3200
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 127.0.0.1:4318
|
||||
ingester:
|
||||
trace_idle_period: 5s
|
||||
max_block_duration: 1m
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 24h
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
local:
|
||||
path: /var/tempo/blocks
|
||||
overrides:
|
||||
defaults:
|
||||
ingestion:
|
||||
rate_limit_bytes: 1000000
|
||||
burst_size_bytes: 2000000
|
||||
@@ -0,0 +1,35 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE sessions (
|
||||
token_hash text PRIMARY KEY,
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
expires_at timestamptz NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX sessions_expiry ON sessions(expires_at);
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
PRIMARY KEY(chat_id,user_id)
|
||||
);
|
||||
CREATE INDEX members_user_chat ON members(user_id,chat_id);
|
||||
CREATE INDEX chats_order ON chats(created_at,id);
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
sender_id uuid NOT NULL REFERENCES users(id),
|
||||
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
FOREIGN KEY(chat_id,sender_id) REFERENCES members(chat_id,user_id)
|
||||
);
|
||||
CREATE INDEX messages_history ON messages(chat_id,created_at,id);
|
||||
@@ -0,0 +1,10 @@
|
||||
CREATE TABLE idempotency (
|
||||
actor_id uuid NOT NULL REFERENCES users(id),
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
key text NOT NULL CHECK (key ~ '^[A-Za-z0-9._:-]{1,128}$'),
|
||||
request_hash text NOT NULL,
|
||||
message_id uuid NOT NULL REFERENCES messages(id),
|
||||
expires_at timestamptz NOT NULL,
|
||||
PRIMARY KEY(actor_id,chat_id,key)
|
||||
);
|
||||
CREATE INDEX idempotency_expiry ON idempotency(expires_at);
|
||||
@@ -0,0 +1,25 @@
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
owner_id uuid NOT NULL REFERENCES users(id),
|
||||
filename text NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
content_type text NOT NULL CHECK (content_type IN ('image/png','image/jpeg')),
|
||||
size_bytes bigint NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 text NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
|
||||
storage_state text NOT NULL DEFAULT 'ready' CHECK (storage_state IN ('ready','missing')),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
UNIQUE (id, chat_id),
|
||||
FOREIGN KEY (chat_id, owner_id) REFERENCES members(chat_id,user_id)
|
||||
);
|
||||
CREATE INDEX attachments_chat ON attachments(chat_id,created_at,id);
|
||||
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_check;
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_content_check
|
||||
CHECK (length(text) <= 4000 AND (length(btrim(text)) > 0 OR cardinality(attachment_ids) > 0));
|
||||
CREATE TABLE message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages(id),
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
PRIMARY KEY(message_id,attachment_id)
|
||||
);
|
||||
@@ -0,0 +1,41 @@
|
||||
ALTER TABLE attachments ADD COLUMN status text NOT NULL DEFAULT 'ready'
|
||||
CHECK (status IN ('queued','processing','ready','failed'));
|
||||
ALTER TABLE attachments ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1 CHECK (pipeline_version>0);
|
||||
ALTER TABLE attachments ADD COLUMN staging_key text;
|
||||
ALTER TABLE attachments ADD COLUMN variants jsonb NOT NULL DEFAULT '[]';
|
||||
ALTER TABLE attachments ADD COLUMN error_code text;
|
||||
ALTER TABLE attachments ADD COLUMN accepted_at timestamptz NOT NULL DEFAULT clock_timestamp();
|
||||
|
||||
CREATE TABLE jobs (
|
||||
id uuid PRIMARY KEY,
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
pipeline_version integer NOT NULL CHECK (pipeline_version IN (1,2)),
|
||||
status text NOT NULL DEFAULT 'queued' CHECK (status IN ('queued','processing','done','failed','superseded')),
|
||||
attempts integer NOT NULL DEFAULT 0 CHECK (attempts>=0),
|
||||
lease_token uuid,
|
||||
lease_until timestamptz,
|
||||
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
error_code text,
|
||||
traceparent text,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
UNIQUE (attachment_id,pipeline_version)
|
||||
);
|
||||
CREATE INDEX jobs_due ON jobs(status,next_attempt_at);
|
||||
CREATE TABLE outbox (
|
||||
job_id uuid PRIMARY KEY REFERENCES jobs(id),
|
||||
published_at timestamptz,
|
||||
publications integer NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE TABLE job_audit (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
job_id uuid NOT NULL REFERENCES jobs(id),
|
||||
event text NOT NULL,
|
||||
details jsonb NOT NULL DEFAULT '{}',
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE job_faults (
|
||||
job_id uuid PRIMARY KEY REFERENCES jobs(id),
|
||||
point text NOT NULL CHECK (point IN ('before_effect','after_effect','after_commit','transient','permanent')),
|
||||
remaining integer NOT NULL CHECK (remaining>=0)
|
||||
);
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE jobs ADD COLUMN IF NOT EXISTS tracestate text;
|
||||
@@ -0,0 +1,6 @@
|
||||
CREATE TABLE IF NOT EXISTS login_budget (
|
||||
name text PRIMARY KEY,
|
||||
window_start timestamptz NOT NULL,
|
||||
used integer NOT NULL
|
||||
);
|
||||
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
|
||||
@@ -0,0 +1,3 @@
|
||||
-r requirements.txt
|
||||
cryptography==50.0.2
|
||||
PyYAML==6.0.3
|
||||
@@ -0,0 +1,43 @@
|
||||
annotated-doc==0.0.5
|
||||
annotated-types==0.8.0
|
||||
anyio==4.15.1
|
||||
argon2-cffi==25.1.0
|
||||
argon2-cffi-bindings==26.1.0
|
||||
boto3==1.43.114
|
||||
botocore==1.43.114
|
||||
certifi==2026.10.23
|
||||
cffi==2.1.1
|
||||
charset-normalizer==3.5.2
|
||||
click==8.5.0
|
||||
fastapi==0.143.2
|
||||
googleapis-common-protos==1.75.5
|
||||
h11==0.16.0
|
||||
idna==3.20
|
||||
jmespath==1.1.0
|
||||
opentelemetry-api==1.45.1
|
||||
opentelemetry-exporter-http-transport==0.66b1
|
||||
opentelemetry-exporter-otlp-common==0.66b1
|
||||
opentelemetry-exporter-otlp-proto-common==1.45.1
|
||||
opentelemetry-exporter-otlp-proto-http==1.45.1
|
||||
opentelemetry-proto==1.45.1
|
||||
opentelemetry-sdk==1.45.1
|
||||
opentelemetry-semantic-conventions==0.66b1
|
||||
pillow==13.0.0
|
||||
protobuf==7.36.2
|
||||
psycopg==3.3.6
|
||||
psycopg-binary==3.3.6
|
||||
psycopg-pool==3.3.3
|
||||
pycparser==3.11
|
||||
pydantic==2.14.1
|
||||
pydantic_core==2.50.1
|
||||
python-dateutil==2.9.0.post0
|
||||
redis==8.1.0
|
||||
requests==2.34.2
|
||||
s3transfer==0.19.2
|
||||
six==1.17.0
|
||||
starlette==1.8.0
|
||||
typing-inspection==0.4.4
|
||||
typing_extensions==4.16.0
|
||||
tzdata==2026.5
|
||||
urllib3==2.8.0
|
||||
uvicorn==0.54.0
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Explicit operator init/unseal/provision. Shares never enter a service mount."""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import ssl
|
||||
import time
|
||||
import urllib.request
|
||||
from urllib.error import HTTPError, URLError
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def request(path,body=None,token=None,method=None):
|
||||
ctx=ssl.create_default_context(cafile=str(ROOT/'secrets/trust/ca.crt'))
|
||||
ctx.load_cert_chain(ROOT/'secrets/identities/operator/cert.pem',ROOT/'secrets/identities/operator/key.pem')
|
||||
headers={'Content-Type':'application/json'}
|
||||
if token:headers['X-Vault-Token']=token
|
||||
req=urllib.request.Request('https://127.0.0.1:8200/v1/'+path,data=json.dumps(body).encode() if body is not None else None,headers=headers,method=method)
|
||||
with urllib.request.urlopen(req,context=ctx,timeout=15) as r:
|
||||
raw=r.read();return json.loads(raw) if raw else {}
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser();p.add_argument('action',choices=['init','unseal','provision','seal','publish-crl']);args=p.parse_args()
|
||||
file=ROOT/'secrets/operator/bao-init.json'
|
||||
if args.action=='init':
|
||||
if request('sys/init')['initialized']:raise SystemExit('Already initialized; preserve existing operator shares')
|
||||
result=request('sys/init',{'secret_shares':3,'secret_threshold':2})
|
||||
with file.open('x',encoding='utf-8') as f:json.dump(result,f)
|
||||
file.chmod(0o600)
|
||||
print('Initialized; three shares stored in operator-only local file. Unseal explicitly.');return
|
||||
operator=json.loads(file.read_text(encoding='utf-8'));token=operator['root_token']
|
||||
if args.action=='unseal':
|
||||
deadline=time.monotonic()+20
|
||||
while True:
|
||||
try:request('sys/seal-status');break
|
||||
except (URLError,OSError):
|
||||
if time.monotonic()>=deadline:raise
|
||||
time.sleep(.5)
|
||||
for share in operator['keys'][:2]:request('sys/unseal',{'key':share})
|
||||
elif args.action=='seal':request('sys/seal',{},token)
|
||||
elif args.action=='publish-crl':request('auth/cert/crls/operator',{'crl':(ROOT/'secrets/trust/revocations.pem').read_text()},token)
|
||||
elif args.action=='provision':
|
||||
mounts=request('sys/mounts',token=token)
|
||||
if 'secret/' not in mounts:request('sys/mounts/secret',{'type':'kv','options':{'version':'2'}},token)
|
||||
auth=request('sys/auth',token=token)
|
||||
if 'cert/' not in auth:request('sys/auth/cert',{'type':'cert'},token)
|
||||
config=json.loads((ROOT/'secrets/operator/config.json').read_text())
|
||||
for role,data in config['service_credentials'].items():
|
||||
policy=f'path "secret/data/{role}" {{ capabilities = ["read"] }}\n'
|
||||
request('sys/policies/acl/'+role,{'policy':policy},token)
|
||||
request('auth/cert/certs/'+role,{'certificate':(ROOT/'secrets/trust/ca.crt').read_text(),
|
||||
'allowed_dns_sans':[role],'token_policies':[role],'token_ttl':120,'token_max_ttl':120,'token_no_default_policy':False},token)
|
||||
request('secret/data/'+role,{'data':data},token)
|
||||
# Audit is declarative in openbao.hcl; API audit creation remains disabled.
|
||||
assert 'operator/' in request('sys/audit',token=token), 'Audit device missing'
|
||||
print('Operator action completed; no root token/unseal shares printed')
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Generate isolated local credentials. Operator files are never app mounts."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import csv
|
||||
import os
|
||||
import subprocess
|
||||
from datetime import datetime,timedelta,timezone
|
||||
import pki
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
ROLES=['api1','api2','worker','dispatcher','reconciler']
|
||||
|
||||
|
||||
def protect_local_files():
|
||||
directory=ROOT/'secrets';directory.mkdir(exist_ok=True)
|
||||
directory.chmod(0o700)
|
||||
if os.name=='nt':
|
||||
sid=next(csv.reader(subprocess.check_output(['whoami','/user','/fo','csv','/nh'],text=True).splitlines()))[1]
|
||||
for path in (directory,ROOT/'.env',ROOT/'.env.storage-admin'):
|
||||
if not path.exists():continue
|
||||
flags='(OI)(CI)' if path.is_dir() else ''
|
||||
result=subprocess.run(['icacls',str(path),'/inheritance:r','/grant:r',
|
||||
'*'+sid+':'+flags+'F','*S-1-5-18:'+flags+'F','*S-1-5-32-544:'+flags+'F'],capture_output=True)
|
||||
if result.returncode:raise RuntimeError('Could not restrict local secret ACLs')
|
||||
|
||||
|
||||
def main():
|
||||
protect_local_files()
|
||||
pki.init()
|
||||
pki.crl()
|
||||
for role in ROLES+['db','s3','broker','cache','otel-collector','prometheus','loki','tempo','grafana','lb','openbao','operator','outsider']:
|
||||
cert_path=ROOT/'secrets/identities'/role/'cert.pem'
|
||||
if (not cert_path.exists() or pki.x509.load_pem_x509_certificate(cert_path.read_bytes()).not_valid_after_utc
|
||||
< datetime.now(timezone.utc)+timedelta(hours=2)):
|
||||
pki.issue(role)
|
||||
target=ROOT/'secrets/operator/config.json'
|
||||
if target.exists():
|
||||
print('Existing local credentials preserved');return
|
||||
password=secrets.token_hex(24)
|
||||
db={role:secrets.token_hex(24) for role in ROLES}
|
||||
broker={role:secrets.token_hex(24) for role in ('worker','dispatcher')}
|
||||
cache={role:secrets.token_hex(24) for role in ('api1','api2')}
|
||||
s3={role:(secrets.token_hex(12),secrets.token_hex(32)) for role in ROLES+['admin'] if role!='dispatcher'}
|
||||
cursor=secrets.token_hex(32)
|
||||
credentials={}
|
||||
for role in ROLES:
|
||||
credentials[role]={'DATABASE_URL':f'postgresql://{role}:{db[role]}@127.0.0.1:5432/messenger',
|
||||
'CURSOR_SECRET':cursor,
|
||||
'OTEL_EXPORTER_OTLP_ENDPOINT':'http://127.0.0.1:4318','TELEMETRY_ENABLED':'1','SESSION_TTL_SECONDS':'3600'}
|
||||
if role in s3:
|
||||
credentials[role].update({'S3_ENDPOINT':'http://127.0.0.1:8333','S3_BUCKET':'messenger',
|
||||
'S3_ACCESS_KEY':s3[role][0],'S3_SECRET_KEY':s3[role][1]})
|
||||
if role in cache: credentials[role]['REDIS_URL']=f'redis://{role}:{cache[role]}@127.0.0.1:6379/0'
|
||||
if role in broker: credentials[role]['BROKER_URL']=f'redis://{role}:{broker[role]}@127.0.0.1:6380/0'
|
||||
identities=[]
|
||||
for role,(key,secret) in s3.items():
|
||||
if role=='admin':actions=['Admin','Read','Write','List','Tagging']
|
||||
elif role.startswith('api'):actions=['Read:messenger/originals/*','Read:messenger/variants/*','Write:messenger/staging/*']
|
||||
else:
|
||||
actions=['Read:messenger/staging/*','Read:messenger/originals/*','Read:messenger/variants/*','Write:messenger/staging/*','Write:messenger/originals/*','Write:messenger/variants/*']
|
||||
if role=='reconciler':actions.append('List:messenger')
|
||||
identities.append({'name':role,'credentials':[{'accessKey':key,'secretKey':secret}],'actions':actions})
|
||||
files={'.env':'POSTGRES_PASSWORD='+password+'\n',
|
||||
'.env.storage-admin':f'S3_ENDPOINT=http://127.0.0.1:8333\nS3_BUCKET=messenger\nS3_ACCESS_KEY={s3["admin"][0]}\nS3_SECRET_KEY={s3["admin"][1]}\n',
|
||||
'secrets/s3.json':json.dumps({'identities':identities},indent=2),
|
||||
'secrets/grafana-password':secrets.token_urlsafe(24)+'\n',
|
||||
'secrets/cache.conf':'bind 127.0.0.1\nprotected-mode yes\nmaxmemory 32mb\nmaxmemory-policy allkeys-lru\nsave ""\nuser default off\n'+''.join(f'user {r} on >{v} ~user:* -@all +get +set +ping +hello +client|setinfo\n' for r,v in cache.items()),
|
||||
'secrets/broker.conf':'bind 127.0.0.1\nprotected-mode yes\nappendonly yes\nappendfsync always\ndir /data\nsave ""\nmaxmemory 64mb\nmaxmemory-policy noeviction\nuser default off\n'+
|
||||
f'user dispatcher on >{broker["dispatcher"]} ~attachment-jobs -@all +xadd +xgroup +ping +hello +client|setinfo\n'+
|
||||
f'user worker on >{broker["worker"]} ~attachment-jobs -@all +xgroup +xautoclaim +xreadgroup +xack +xdel +multi +exec +ping +hello +client|setinfo\n',
|
||||
'secrets/operator/config.json':json.dumps({'database_passwords':db,'service_credentials':credentials},indent=2)}
|
||||
for name,value in files.items():
|
||||
p=ROOT/name;p.parent.mkdir(parents=True,exist_ok=True)
|
||||
with p.open('x',encoding='utf-8',newline='\n') as f:f.write(value)
|
||||
p.chmod(0o600)
|
||||
protect_local_files()
|
||||
print('Generated separate identities and local secrets; values not printed')
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Generate the committed, provisioned dashboard from explicit PromQL."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
panels=[]
|
||||
def panel(title,queries,unit='short'):
|
||||
i=len(panels)
|
||||
panels.append({'id':i+1,'type':'timeseries','title':title,'datasource':{'type':'prometheus','uid':'prometheus'},
|
||||
'gridPos':{'x':(i%2)*12,'y':(i//2)*8,'w':12,'h':8},
|
||||
'targets':[{'refId':chr(65+j),'expr':expr,'legendFormat':label} for j,(label,expr) in enumerate(queries)],
|
||||
'fieldConfig':{'defaults':{'unit':unit,'min':0,**({'max':1} if unit=='percentunit' else {})},'overrides':[]},'options':{'legend':{'displayMode':'list','placement':'bottom'}}})
|
||||
|
||||
panel('API requests / second',[('{{method}} {{route}}','sum by(method,route)(rate(http_server_requests_total[1m]))')],'reqps')
|
||||
panel('API 5xx / all business requests',[('5xx fraction','(sum(rate(http_server_requests_total{status_class="5xx"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)')],'percentunit')
|
||||
panel('API latency: aggregate buckets across replicas',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(http_server_duration_seconds_bucket[1m])))') for p in (50,95,99)],'s')
|
||||
panel('Jobs / second by outcome',[('{{outcome}}','sum by(outcome)(rate(jobs_processed_total[1m]))')],'ops')
|
||||
panel('Worker processing latency',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))') for p in (50,95,99)],'s')
|
||||
panel('Acceptance to ready latency',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))') for p in (50,95,99)],'s')
|
||||
panel('Unfinished durable jobs',[('queue depth','sum(queue_depth)')])
|
||||
panel('Oldest unfinished job age',[('oldest','max(queue_oldest_age_seconds)')],'s')
|
||||
panel('DB pool',[('used {{process_instance}}','db_pool_used'),('waiting {{process_instance}}','db_pool_waiting')])
|
||||
panel('Worker saturation',[('busy / capacity','sum(worker_busy) / clamp_min(sum(worker_capacity),1)')],'percentunit')
|
||||
panel('Current process RSS',[('{{service}} {{process_instance}}','process_rss_bytes')],'bytes')
|
||||
panel('Process CPU seconds / wall second',[('{{service}} {{process_instance}}','clamp_min(deriv(process_cpu_time_seconds[1m]),0)')],'short')
|
||||
panel('API availability burn rate; budget 0.5%',[(f'{w}s',f'(sum(rate(http_server_requests_total{{status_class="5xx"}}[{w}s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[{w}s])),0.001) / 0.005') for w in (20,60)])
|
||||
panel('Ready <=30s burn rate; budget 1%',[(f'{w}s eligible cohort',f'sum(attachment_slo_bad{{window="{w}"}}) / clamp_min(sum(attachment_slo_eligible{{window="{w}"}}),1) / 0.01') for w in (60,300)])
|
||||
panel('Collector exporter queue / capacity',[('{{exporter}} size','otelcol_exporter_queue_size'),('{{exporter}} capacity','otelcol_exporter_queue_capacity')])
|
||||
panel('Collector failed exports',[('{{exporter}} spans','rate(otelcol_exporter_send_failed_spans_total[1m])'),('{{exporter}} logs','rate(otelcol_exporter_send_failed_log_records_total[1m])')],'ops')
|
||||
panels.append({'id':99,'type':'logs','title':'Structured application logs; TraceID links to Tempo','datasource':{'type':'loki','uid':'loki'},
|
||||
'gridPos':{'x':0,'y':64,'w':24,'h':10},'targets':[{'refId':'A','expr':'{service_name=~"messenger-(api|worker|dispatcher)"}'}],
|
||||
'options':{'showTime':True,'showLabels':False,'wrapLogMessage':True,'sortOrder':'Descending'}})
|
||||
doc={'uid':'mtusi-messenger','title':'MTUSI Messenger - signals and SLO','schemaVersion':39,'version':1,
|
||||
'editable':False,'timezone':'browser','time':{'from':'now-15m','to':'now'},'refresh':'15s','tags':['mtusi','lab6'],
|
||||
'panels':panels,'annotations':{'list':[]},'templating':{'list':[]}}
|
||||
path=Path(__file__).resolve().parents[1]/'infra/grafana/dashboards/messenger.json'
|
||||
path.parent.mkdir(parents=True,exist_ok=True)
|
||||
path.write_text(json.dumps(doc,ensure_ascii=False,indent=2)+'\n',encoding='utf-8')
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Operator migrations/grants over the container-local PostgreSQL Unix socket.
|
||||
|
||||
SQL containing passwords travels on stdin, not argv/logs; no admin DB credential
|
||||
is made available to an application. Existing migration hashes must match.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def sql(query):
|
||||
result=subprocess.run(['docker','compose','exec','-T','db','psql','-X','-qAt',
|
||||
'-v','ON_ERROR_STOP=1','-U','messenger','-d','messenger'],input=query,
|
||||
text=True,encoding='utf-8',capture_output=True,cwd=ROOT)
|
||||
if result.returncode:
|
||||
# Server error text may echo a credential-bearing statement.
|
||||
raise RuntimeError('Operator SQL failed; no query or credentials printed')
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def main():
|
||||
sql('CREATE TABLE IF NOT EXISTS schema_migrations (name text PRIMARY KEY, sha256 text NOT NULL, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())')
|
||||
for file in sorted((ROOT/'migrations').glob('*.sql')):
|
||||
digest=hashlib.sha256(file.read_bytes()).hexdigest()
|
||||
old=sql(f"SELECT sha256 FROM schema_migrations WHERE name='{file.name}'")
|
||||
if old:
|
||||
assert old==digest, 'Applied migration changed: '+file.name
|
||||
else:
|
||||
sql('BEGIN;\n'+file.read_text()+f"\nINSERT INTO schema_migrations(name,sha256) VALUES ('{file.name}','{digest}'); COMMIT;")
|
||||
print('Migration verified:',file.name)
|
||||
passwords=json.loads((ROOT/'secrets/operator/config.json').read_text())['database_passwords']
|
||||
for role,password in passwords.items():
|
||||
assert role in ('api1','api2','worker','dispatcher','reconciler') and all(c in '0123456789abcdef' for c in password)
|
||||
if not sql(f"SELECT 1 FROM pg_roles WHERE rolname='{role}'"):sql(f'CREATE ROLE {role} LOGIN')
|
||||
sql(f"ALTER ROLE {role} PASSWORD '{password}' NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION; GRANT CONNECT ON DATABASE messenger TO {role}; GRANT USAGE ON SCHEMA public TO {role}; REVOKE ALL ON ALL TABLES IN SCHEMA public FROM {role}; REVOKE ALL ON ALL SEQUENCES IN SCHEMA public FROM {role};")
|
||||
sql('REVOKE ALL ON DATABASE messenger FROM PUBLIC; REVOKE CREATE ON SCHEMA public FROM PUBLIC;')
|
||||
sql('GRANT SELECT,INSERT,UPDATE,DELETE ON users,sessions,chats,members,messages,idempotency,attachments,message_attachments,jobs,outbox,login_budget TO api1,api2;')
|
||||
sql('GRANT SELECT,UPDATE ON jobs,attachments,outbox,job_faults TO worker; GRANT INSERT ON job_audit TO worker; GRANT USAGE ON SEQUENCE job_audit_id_seq TO worker;')
|
||||
sql('GRANT SELECT,UPDATE ON jobs,attachments,outbox TO dispatcher; GRANT INSERT ON job_audit TO dispatcher; GRANT USAGE ON SEQUENCE job_audit_id_seq TO dispatcher;')
|
||||
sql('GRANT SELECT,UPDATE ON attachments TO reconciler;')
|
||||
print('Scoped database roles applied')
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Operator-owned local PKI. CA private material is never mounted into services."""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import ipaddress
|
||||
import json
|
||||
from pathlib import Path
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes, serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import ec
|
||||
from cryptography.x509.oid import NameOID, ExtendedKeyUsageOID
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
PRIVATE=ROOT/'secrets/operator'
|
||||
TRUST=ROOT/'secrets/trust'
|
||||
|
||||
|
||||
def write(path,data):
|
||||
path.parent.mkdir(parents=True,exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
path.chmod(0o600 if 'key' in path.name or path.name == 'combined.pem' or path.parent == PRIVATE else 0o644)
|
||||
|
||||
|
||||
def key_bytes(key):
|
||||
return key.private_bytes(serialization.Encoding.PEM,serialization.PrivateFormat.PKCS8,serialization.NoEncryption())
|
||||
|
||||
|
||||
def ca():
|
||||
return (serialization.load_pem_private_key((PRIVATE/'ca.key').read_bytes(),None),
|
||||
x509.load_pem_x509_certificate((TRUST/'ca.crt').read_bytes()))
|
||||
|
||||
|
||||
def init():
|
||||
if (PRIVATE/'ca.key').exists(): return
|
||||
key=ec.generate_private_key(ec.SECP256R1()); now=datetime.now(timezone.utc)
|
||||
name=x509.Name([x509.NameAttribute(NameOID.COMMON_NAME,'MTUSI local operator CA')])
|
||||
cert=(x509.CertificateBuilder().subject_name(name).issuer_name(name).public_key(key.public_key())
|
||||
.serial_number(x509.random_serial_number()).not_valid_before(now-timedelta(minutes=1))
|
||||
.not_valid_after(now+timedelta(days=365)).add_extension(x509.BasicConstraints(ca=True,path_length=0),True)
|
||||
.add_extension(x509.KeyUsage(False,False,False,False,False,True,True,None,None),True)
|
||||
.add_extension(x509.SubjectKeyIdentifier.from_public_key(key.public_key()),False).sign(key,hashes.SHA256()))
|
||||
write(PRIVATE/'ca.key',key_bytes(key));write(TRUST/'ca.crt',cert.public_bytes(serialization.Encoding.PEM))
|
||||
write(PRIVATE/'revoked.json',b'[]'); crl()
|
||||
|
||||
|
||||
def issue(role,hours=24,extra=()):
|
||||
key,issuer=ca(); leaf=ec.generate_private_key(ec.SECP256R1()); now=datetime.now(timezone.utc)
|
||||
sans=[x509.DNSName(role),*[x509.DNSName(n) for n in extra]]
|
||||
if role in ('lb','openbao','grafana'): sans += [x509.DNSName('localhost'),x509.IPAddress(ipaddress.ip_address('127.0.0.1'))]
|
||||
cert=(x509.CertificateBuilder().subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME,role)]))
|
||||
.issuer_name(issuer.subject).public_key(leaf.public_key()).serial_number(x509.random_serial_number())
|
||||
.not_valid_before(now-timedelta(minutes=1)).not_valid_after(now+timedelta(hours=hours))
|
||||
.add_extension(x509.BasicConstraints(ca=False,path_length=None),True)
|
||||
.add_extension(x509.SubjectAlternativeName(sans),False)
|
||||
.add_extension(x509.ExtendedKeyUsage([ExtendedKeyUsageOID.SERVER_AUTH,ExtendedKeyUsageOID.CLIENT_AUTH]),False)
|
||||
.add_extension(x509.AuthorityKeyIdentifier.from_issuer_public_key(key.public_key()),False)
|
||||
.sign(key,hashes.SHA256()))
|
||||
folder=ROOT/'secrets/identities'/role
|
||||
write(folder/'key.pem',key_bytes(leaf));write(folder/'cert.pem',cert.public_bytes(serialization.Encoding.PEM))
|
||||
write(folder/'combined.pem',cert.public_bytes(serialization.Encoding.PEM)+key_bytes(leaf))
|
||||
return cert
|
||||
|
||||
|
||||
def crl():
|
||||
key,issuer=ca();now=datetime.now(timezone.utc)
|
||||
doc=x509.CertificateRevocationListBuilder().issuer_name(issuer.subject).last_update(now-timedelta(minutes=1)).next_update(now+timedelta(days=7))
|
||||
for serial in json.loads((PRIVATE/'revoked.json').read_text()):
|
||||
doc=doc.add_revoked_certificate(x509.RevokedCertificateBuilder().serial_number(serial).revocation_date(now).build())
|
||||
pem=doc.sign(key,hashes.SHA256()).public_bytes(serialization.Encoding.PEM)
|
||||
write(TRUST/'revocations.pem',pem);write(TRUST/'bundle.pem',(TRUST/'ca.crt').read_bytes()+pem)
|
||||
|
||||
|
||||
def revoke(role):
|
||||
cert=x509.load_pem_x509_certificate((ROOT/'secrets/identities'/role/'cert.pem').read_bytes())
|
||||
path=PRIVATE/'revoked.json';values=json.loads(path.read_text())
|
||||
if cert.serial_number not in values: values.append(cert.serial_number)
|
||||
write(path,json.dumps(values).encode());crl()
|
||||
|
||||
|
||||
if __name__=='__main__':
|
||||
import argparse
|
||||
p=argparse.ArgumentParser();p.add_argument('action',choices=['issue','revoke']);p.add_argument('role');p.add_argument('--hours',type=float,default=24)
|
||||
a=p.parse_args();init()
|
||||
if a.action=='issue': issue(a.role,a.hours)
|
||||
else: revoke(a.role)
|
||||
print('PKI operation complete; no key material printed')
|
||||
@@ -0,0 +1,3 @@
|
||||
@echo off
|
||||
if not defined LAB7_PYTHON set "LAB7_PYTHON=python"
|
||||
"%LAB7_PYTHON%" "%~dp0restart.py"
|
||||
@@ -0,0 +1,17 @@
|
||||
"""Restart every running replica and dependency, preserving durable volumes."""
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from stack import ROOT, docker, state
|
||||
|
||||
before = state()
|
||||
docker('compose', 'stop', '--timeout', '5')
|
||||
docker('compose', 'start')
|
||||
subprocess.run([sys.executable,str(ROOT/'scripts/bao_operator.py'),'unseal'],check=True)
|
||||
after = state()
|
||||
assert set(before) == set(after)
|
||||
assert all(before[c]['state']['StartedAt'] != after[c]['state']['StartedAt'] for c in before)
|
||||
(ROOT / 'evidence').mkdir(exist_ok=True)
|
||||
(ROOT / 'evidence/restart-processes.json').write_text(
|
||||
json.dumps({'before': before, 'after': after, 'volumes_removed': False}, indent=2) + '\n',
|
||||
encoding='utf-8')
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
python3 scripts/restart.py
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Scan exported image archives without mounting the Docker socket or host secrets."""
|
||||
import argparse
|
||||
import collections
|
||||
from datetime import datetime,timezone
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import yaml
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
TRIVY='aquasec/trivy@sha256:790fd66d1697395d8693d1acaa1ea702406a20f109b6fc7bac8db623eca6c06c'
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser();parser.add_argument('services',nargs='*',default=['api1','openbao','s3','lb']);args=parser.parse_args()
|
||||
compose=yaml.safe_load((ROOT/'compose.yaml').read_text())
|
||||
summary={'at_UTC':datetime.now(timezone.utc).isoformat(),'scanner':TRIVY,'scope':args.services,'images':[]}
|
||||
with tempfile.TemporaryDirectory(prefix='lab7-image-scan-') as tmp:
|
||||
for service in args.services:
|
||||
image=compose['services'][service]['image']
|
||||
image_id=subprocess.check_output(['docker','image','inspect',image,'--format','{{.Id}}'],text=True).strip()
|
||||
archive=Path(tmp)/'image.tar';report=Path(tmp)/'report.json'
|
||||
subprocess.run(['docker','save','--output',str(archive),image],check=True)
|
||||
subprocess.run(['docker','run','--rm','--cpus','0.5','--memory','768m','--pids-limit','128',
|
||||
'-v',tmp+':/scan','-v','mtusi-trivy-cache:/root/.cache/trivy',TRIVY,'image','--input','/scan/image.tar',
|
||||
'--parallel','1','--scanners','vuln','--format','json','--output','/scan/report.json'],check=True)
|
||||
data=json.loads(report.read_text());findings=[]
|
||||
for result in data.get('Results',[]):
|
||||
for v in result.get('Vulnerabilities',[]):
|
||||
findings.append({'target':result.get('Target'),**{k:v.get(k) for k in ('VulnerabilityID','PkgName','InstalledVersion','FixedVersion','Severity','Title','PrimaryURL')}})
|
||||
summary['images'].append({'service':service,'image':image,'image_id':image_id,'report_sha256':hashlib.sha256(report.read_bytes()).hexdigest(),
|
||||
'counts':dict(collections.Counter(v['Severity'] for v in findings)),'findings':findings})
|
||||
(ROOT/'evidence/image-audit.json').write_text(json.dumps(summary,indent=2)+'\n')
|
||||
print(service,summary['images'][-1]['counts'],flush=True)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Shared bounded Docker helpers for reproducible, scoped failure experiments."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
def docker(*args):
|
||||
return subprocess.check_output(['docker',*args],cwd=ROOT,text=True,encoding='utf-8',stderr=subprocess.PIPE).strip()
|
||||
def replicas(): return docker('compose','ps','-q','api1','api2').splitlines()
|
||||
|
||||
def ready_replicas(count=2,timeout=60):
|
||||
deadline=time.monotonic()+timeout
|
||||
while time.monotonic()<deadline:
|
||||
ids=replicas()
|
||||
if len(ids)==count:
|
||||
states=[json.loads(docker('inspect','--format','{{json .State}}',cid)) for cid in ids]
|
||||
if all(s['Running'] and s.get('Health',{}).get('Status')=='healthy' for s in states): return
|
||||
time.sleep(.5)
|
||||
raise AssertionError('Actual replica health did not recover within timeout')
|
||||
def instance(cid):
|
||||
return docker('exec',cid,'python','-c',"import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/health/live',timeout=2).headers['X-Instance-Id'])")
|
||||
def state():
|
||||
result={}
|
||||
for service in ('api1','api2','db','cache','lb','s3','reconciler','broker','worker','dispatcher','openbao','otel-collector','prometheus','loki','tempo','grafana'):
|
||||
for cid in docker('compose','ps','-q',service).splitlines():
|
||||
result[cid]={'service':service,'state':json.loads(docker('inspect','--format','{{json .State}}',cid))}
|
||||
return result
|
||||
def ready(client,timeout=60):
|
||||
deadline=time.monotonic()+timeout
|
||||
while time.monotonic()<deadline:
|
||||
try:
|
||||
if client.get('/health/ready').status_code==200: return
|
||||
except Exception: pass
|
||||
time.sleep(.2)
|
||||
raise AssertionError('Readiness timeout')
|
||||
@@ -0,0 +1,3 @@
|
||||
@echo off
|
||||
if not defined LAB7_PYTHON set "LAB7_PYTHON=python"
|
||||
"%LAB7_PYTHON%" "%~dp0stop-one.py"
|
||||
@@ -0,0 +1,15 @@
|
||||
"""Stop the real process selected by X-Instance-Id; do not stop any dependency."""
|
||||
import json
|
||||
import os
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from stack import ROOT, docker, instance, replicas
|
||||
target=os.environ['TARGET_INSTANCE_ID']
|
||||
before=replicas()
|
||||
with ThreadPoolExecutor(max_workers=2) as pool:
|
||||
matches=[cid for cid,alias in zip(before,pool.map(instance,before)) if alias==target]
|
||||
assert len(matches)==1,'Target does not uniquely identify a running API'
|
||||
# Immediate process crash, without adding a graceful-drain delay to the fault.
|
||||
docker('kill','--signal','KILL',matches[0])
|
||||
assert docker('inspect','--format','{{.State.Running}}',matches[0])=='false'
|
||||
(ROOT/'evidence').mkdir(exist_ok=True)
|
||||
(ROOT/'evidence/stop-one.json').write_text(json.dumps({'target_instance':target,'stopped_container':matches[0],'before_containers':before,'after_running_containers':replicas()},indent=2)+'\n',encoding='utf-8')
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
exec python3 scripts/stop-one.py
|
||||
@@ -0,0 +1,19 @@
|
||||
"""Negative-test leaf identities; never used by a runtime service."""
|
||||
from datetime import datetime,timedelta,timezone
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes,serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import ec
|
||||
from cryptography.x509.oid import NameOID
|
||||
import pki
|
||||
|
||||
roles=['lb','api1','api2','worker','dispatcher','reconciler','operator','otel-collector','prometheus','grafana']
|
||||
pki.issue('expired',hours=-.001,extra=roles)
|
||||
pki.issue('revoked',extra=roles);pki.revoke('revoked')
|
||||
key=ec.generate_private_key(ec.SECP256R1());now=datetime.now(timezone.utc)
|
||||
name=x509.Name([x509.NameAttribute(NameOID.COMMON_NAME,'untrusted-test')])
|
||||
cert=(x509.CertificateBuilder().subject_name(name).issuer_name(name).public_key(key.public_key())
|
||||
.serial_number(x509.random_serial_number()).not_valid_before(now-timedelta(minutes=1))
|
||||
.not_valid_after(now+timedelta(hours=1)).add_extension(x509.SubjectAlternativeName([x509.DNSName(n) for n in roles]),False).sign(key,hashes.SHA256()))
|
||||
pki.write(pki.ROOT/'secrets/identities/alien/key.pem',pki.key_bytes(key))
|
||||
pki.write(pki.ROOT/'secrets/identities/alien/cert.pem',cert.public_bytes(serialization.Encoding.PEM))
|
||||
print('Negative test identities generated')
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user