Record successful isolated lab7 verification and final evidence
Skeleton contracts / contracts (pull_request) Canceled after 0s

This commit is contained in:
2026-11-13 13:03:05 +03:00
parent 0bf1ff93c7
commit 15610061c9
19 changed files with 482 additions and 377 deletions
+25
View File
@@ -116,6 +116,31 @@ S3 admin имеет отдельный `.env.storage-admin` лишь в operator
## Проверки и доказательства
Финальная проверка выполнена из отдельного clone `clean-lab7-final` с собственным
venv, новой PKI/секретами и новыми volumes. Код соответствует commit `0bf1ff9`;
финальные evidence/документация зафиксированы отдельным последующим коммитом. Проверка публичного
набора 13.11.2026 прошла целиком: contract, OpenAPI, HTTPS smoke/security,
persistence, три failover-запуска с двумя различными container IDs. После fault
первые успешные записи восстановились за 3.390/2.953/2.610s. Первые два запуска
выбрали одну реплику, третий — другую; все реальные процессы остановлены SIGKILL,
без замены target теста. Перед следующим сценарием обе API и TLS sidecars
восстанавливаются согласованно. Публичные tests не изменены.
Полный security suite завершился PASS 13.11.2026 в 10:00:37 UTC: все 11 этапов,
включая TLS allow/deny matrix, Bao policies, worker permissions, API security,
компрометацию worker, sealed Bao, telemetry, hardening и 4 renderer unit tests.
При компрометации восстановлены 3/3 задания без изменения содержимого и образа;
старые certificate/Bao auth/DB password отклонены; полный эксперимент 35.61s.
При sealed Bao обе API child остановились через 123.86s (120s budget + grace),
восстановление после unseal 20.44s. Argon2 hash+verify median 0.207s,
maxRSS 53184KiB; 20 login за 4.578s дали 12×401 и 8×429 на двух API.
Это результаты одного запуска на указанном стенде, а не гарантированные SLO.
Сводка: [clean-clone.json](evidence/clean-clone.json),
[public suite](evidence/public-suite.txt), [security suite](evidence/security-suite.txt).
Image scan от 12.11 предшествует исправлению планирования TLS handshake;
зависимости не изменились, но scan не заявляется для точного app image ID
финального запуска (он отдельно записан в clean-clone.json).
```sh
.venv/bin/python scripts/verify_public.py
.venv/bin/python scripts/verify_security.py
+14
View File
@@ -0,0 +1,14 @@
{
"passed": true,
"completed_at_UTC": "2026-11-13T10:00:37.671336+00:00",
"source_commit": "0bf1ff93c71a6a8cf16edeceefc3b3fab8d23261",
"isolated_clone": true,
"own_venv": true,
"fresh_PKI_and_secrets": true,
"fresh_compose_volumes": true,
"public_suite": "PASS; 3 real SIGKILL scenarios cover 2 distinct API containers",
"security_suite": "PASS; all 11 stages",
"app_image_tested": "sha256:5d549e32c6a790e783550782db9b5c5993861fa5e7dac46039aaa9adb03e2eed",
"host": "Windows 10, Docker Desktop 4.39 / Engine 28 / Compose 2.33, WSL2 4 CPU / 3 GiB RAM / 4 GiB swap",
"scan_scope": "image-audit.json is the 12 November scan, preceding the TLS scheduling code fix; dependencies unchanged. It is not a scan of the exact app image above."
}
+3 -3
View File
@@ -14,8 +14,8 @@
"queue_admin"
]
},
"revoked_serial": "410801984061852326614470576717084328865603324645",
"new_serial": "206772288194023342549548212403414214329040324799",
"revoked_serial": "296803757360939619598658580576779705066316270051",
"new_serial": "582892826110773913646976171481035976112843893487",
"revoked_certificate_denied": true,
"revoked_Bao_auth_denied": true,
"rotated_old_DB_password_denied": true,
@@ -24,5 +24,5 @@
"distinct_jobs": 3,
"content_SHA256_preserved": true,
"image_unchanged": true,
"total_seconds": 36.89
"total_seconds": 35.61
}
+5 -5
View File
@@ -1,11 +1,11 @@
{
"target_instance": "c1f32a3a044049339e1ac7cbe56af4fa",
"stopped_container": "9b982ed0439d2b2a84a942b4a18cea8c11ee011e44ddcc3982a295404a1e0b65",
"target_instance": "b1a678ea159c4aaea77efbcc38b117e8",
"stopped_container": "0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3",
"before_containers": [
"4405619f699207cddbc05b1947545abf205d69867d859642b99bb43b4a417a12",
"9b982ed0439d2b2a84a942b4a18cea8c11ee011e44ddcc3982a295404a1e0b65"
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
],
"after_running_containers": [
"4405619f699207cddbc05b1947545abf205d69867d859642b99bb43b4a417a12"
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7"
]
}
+1 -1
View File
@@ -1,3 +1,3 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 71, "failed_attempts": 4, "first_recovery_seconds": 2.907}
{"observed_instances_before": 2, "successful_writes": 77, "failed_attempts": 4, "first_recovery_seconds": 3.39}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+5 -5
View File
@@ -1,11 +1,11 @@
{
"target_instance": "16a7fabd28ee40b4a4a7ddf7cc070896",
"stopped_container": "4405619f699207cddbc05b1947545abf205d69867d859642b99bb43b4a417a12",
"target_instance": "7f288f373c5140c3a3a1f8de74fa91f2",
"stopped_container": "0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3",
"before_containers": [
"4405619f699207cddbc05b1947545abf205d69867d859642b99bb43b4a417a12",
"9b982ed0439d2b2a84a942b4a18cea8c11ee011e44ddcc3982a295404a1e0b65"
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
],
"after_running_containers": [
"9b982ed0439d2b2a84a942b4a18cea8c11ee011e44ddcc3982a295404a1e0b65"
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7"
]
}
+1 -1
View File
@@ -1,3 +1,3 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 79, "failed_attempts": 4, "first_recovery_seconds": 3.125}
{"observed_instances_before": 2, "successful_writes": 83, "failed_attempts": 4, "first_recovery_seconds": 2.953}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+11
View File
@@ -0,0 +1,11 @@
{
"target_instance": "932b533b493e4dab8e09817e66f77901",
"stopped_container": "1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"before_containers": [
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
],
"after_running_containers": [
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
]
}
+3
View File
@@ -0,0 +1,3 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 79, "failed_attempts": 4, "first_recovery_seconds": 2.61}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+9
View File
@@ -0,0 +1,9 @@
scripts/check_contract.py: PASS
scripts/validate_openapi.py: PASS
tests/smoke.py: PASS
tests/security.py: PASS
persistence: PASS
failover-1: PASS
failover-2: PASS
failover-3: PASS
PASS: both distinct API containers failed over; all replicas restored
+1 -1
View File
@@ -1,5 +1,5 @@
....
----------------------------------------------------------------------
Ran 4 tests in 0.240s
Ran 4 tests in 0.217s
OK
+193 -193
View File
@@ -1,6 +1,6 @@
{
"before": {
"4405619f699207cddbc05b1947545abf205d69867d859642b99bb43b4a417a12": {
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7": {
"service": "api1",
"state": {
"Status": "running",
@@ -9,42 +9,42 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 28764,
"Pid": 2835,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:35:47.797150343Z",
"FinishedAt": "2026-11-11T11:34:23.558402295Z",
"StartedAt": "2026-11-13T09:51:55.809852669Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-11T11:36:04.021674433Z",
"End": "2026-11-11T11:36:04.483245883Z",
"Start": "2026-11-13T09:52:33.85158481Z",
"End": "2026-11-13T09:52:34.414940821Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:10.198239561Z",
"End": "2026-11-11T11:36:12.61719902Z",
"Start": "2026-11-13T09:52:39.416434594Z",
"End": "2026-11-13T09:52:40.03112071Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:17.619051776Z",
"End": "2026-11-11T11:36:18.229256415Z",
"Start": "2026-11-13T09:52:45.032672537Z",
"End": "2026-11-13T09:52:45.684456356Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:23.229801236Z",
"End": "2026-11-11T11:36:23.814745647Z",
"Start": "2026-11-13T09:52:50.685337932Z",
"End": "2026-11-13T09:52:51.337051054Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:28.815878749Z",
"End": "2026-11-11T11:36:29.290543895Z",
"Start": "2026-11-13T09:52:55.883682844Z",
"End": "2026-11-13T09:52:56.745165826Z",
"ExitCode": 0,
"Output": ""
}
@@ -52,7 +52,7 @@
}
}
},
"9b982ed0439d2b2a84a942b4a18cea8c11ee011e44ddcc3982a295404a1e0b65": {
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3": {
"service": "api2",
"state": {
"Status": "running",
@@ -61,42 +61,42 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23689,
"Pid": 2820,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.214133371Z",
"FinishedAt": "2026-11-11T11:32:11.072752249Z",
"StartedAt": "2026-11-13T09:51:55.797519555Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-11T11:36:05.917742644Z",
"End": "2026-11-11T11:36:06.347780428Z",
"Start": "2026-11-13T09:52:35.50023978Z",
"End": "2026-11-13T09:52:35.92798761Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:14.05284085Z",
"End": "2026-11-11T11:36:14.537050616Z",
"Start": "2026-11-13T09:52:40.929156411Z",
"End": "2026-11-13T09:52:41.906797728Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:19.538454996Z",
"End": "2026-11-11T11:36:20.105527493Z",
"Start": "2026-11-13T09:52:46.908309853Z",
"End": "2026-11-13T09:52:47.31147321Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:25.106180943Z",
"End": "2026-11-11T11:36:25.519708718Z",
"Start": "2026-11-13T09:52:51.873454972Z",
"End": "2026-11-13T09:52:52.395517815Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:30.521209633Z",
"End": "2026-11-11T11:36:31.035009116Z",
"Start": "2026-11-13T09:52:57.397494814Z",
"End": "2026-11-13T09:52:57.897562413Z",
"ExitCode": 0,
"Output": ""
}
@@ -104,7 +104,7 @@
}
}
},
"f741db3c2e8fa8f1bdc21d9acec6d44c99a3f4408ccd7c73aa1f93eae5d10b7f": {
"76e0e9ee11f2acd4202aa4641b07c2af83558e6726dd26a3521b33bb520cf177": {
"service": "db",
"state": {
"Status": "running",
@@ -113,42 +113,42 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 22996,
"Pid": 755,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:18.282052335Z",
"FinishedAt": "2026-11-11T11:32:08.104879831Z",
"StartedAt": "2026-11-13T09:51:20.83333691Z",
"FinishedAt": "2026-11-13T09:50:28.225155745Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-11T11:36:21.353237707Z",
"End": "2026-11-11T11:36:21.390132672Z",
"Start": "2026-11-13T09:52:50.379763185Z",
"End": "2026-11-13T09:52:50.456684316Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-11T11:36:23.390506113Z",
"End": "2026-11-11T11:36:23.486031011Z",
"Start": "2026-11-13T09:52:52.018791868Z",
"End": "2026-11-13T09:52:52.157704126Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-11T11:36:25.488420945Z",
"End": "2026-11-11T11:36:25.549786954Z",
"Start": "2026-11-13T09:52:54.159668931Z",
"End": "2026-11-13T09:52:54.239425636Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-11T11:36:27.550823558Z",
"End": "2026-11-11T11:36:27.588371214Z",
"Start": "2026-11-13T09:52:56.241459415Z",
"End": "2026-11-13T09:52:56.374718752Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-11T11:36:29.589462604Z",
"End": "2026-11-11T11:36:29.698878717Z",
"Start": "2026-11-13T09:52:58.375876362Z",
"End": "2026-11-13T09:52:58.425063406Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
@@ -156,7 +156,7 @@
}
}
},
"95bdec59b97137172ccff74afeae1b59a7314e835805f50edccf4a17b62b2184": {
"051a2a59f89f08acf5878b6ab22e2ad389bc09d4fa3d655604f27375b26922be": {
"service": "cache",
"state": {
"Status": "running",
@@ -165,14 +165,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23266,
"Pid": 840,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.08139663Z",
"FinishedAt": "2026-11-11T11:32:07.532320646Z"
"StartedAt": "2026-11-13T09:51:21.347055009Z",
"FinishedAt": "2026-11-13T09:50:28.224207664Z"
}
},
"9ac40379ff6275dcde3d4b2d317dcd9faa88760c6c7eec71cac36c79420407c7": {
"c0e3fc648ea6a23b643a0104ad9bf93f06bfd45e592fece24d8c4dc8788dc494": {
"service": "lb",
"state": {
"Status": "running",
@@ -181,14 +181,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23607,
"Pid": 2892,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.166253805Z",
"FinishedAt": "2026-11-11T11:32:06.491697298Z"
"StartedAt": "2026-11-13T09:51:55.925530847Z",
"FinishedAt": "2026-11-13T09:50:28.222150955Z"
}
},
"a02442c9d071c5eb11020c0bbcc6c751faf70103d001627fd0cf611447cd9066": {
"68c83b0d029545c4c8bfa67979e23fda461d7b9aea909d5e99e462f4858fcc1a": {
"service": "s3",
"state": {
"Status": "running",
@@ -197,14 +197,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23413,
"Pid": 863,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.130944852Z",
"FinishedAt": "2026-11-11T11:32:12.308681778Z"
"StartedAt": "2026-11-13T09:51:21.411682013Z",
"FinishedAt": "2026-11-13T09:50:28.225741755Z"
}
},
"b8191296010574753cf4e61b0a6605453a488df49ef03488fd8b40ab342724d4": {
"6db578724c16067c9101d8ffb911ae3bb1127c087e0b8a598c958983914676a7": {
"service": "reconciler",
"state": {
"Status": "running",
@@ -213,14 +213,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23485,
"Pid": 2676,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.180267156Z",
"FinishedAt": "2026-11-11T11:32:11.196101722Z"
"StartedAt": "2026-11-13T09:51:55.784764196Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}
},
"c8638dec3e880a026dd9d16fee7cc02b52c77eb4afa46a666ec5ce29454c2c13": {
"362e538c9c53fa7feb054689ed85f862d070518a168c0efaa6e184ad51bd34f5": {
"service": "broker",
"state": {
"Status": "running",
@@ -229,14 +229,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23386,
"Pid": 879,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.155114299Z",
"FinishedAt": "2026-11-11T11:32:07.537308968Z"
"StartedAt": "2026-11-13T09:51:21.392114019Z",
"FinishedAt": "2026-11-13T09:50:28.224003528Z"
}
},
"dcfba1124ca8ecdd54c327b95ea2f1f78194e50a9ab597d70a88fad735be05ce": {
"2e513af66f171401771b1a3680154f7edfc7472a5ee79e94041256efee78b2f0": {
"service": "worker",
"state": {
"Status": "running",
@@ -245,14 +245,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23640,
"Pid": 2709,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.22551692Z",
"FinishedAt": "2026-11-11T11:32:11.327585898Z"
"StartedAt": "2026-11-13T09:51:55.951329668Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}
},
"8356b6e281eb194949c11425f8312df9bcbb9bed34e52cc453884db43c974bfd": {
"c40815a4d87373db33cf7eb4b89d473b853dd7a9ece81fa73e052da54ced5ff4": {
"service": "dispatcher",
"state": {
"Status": "running",
@@ -261,14 +261,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23606,
"Pid": 2629,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.228113057Z",
"FinishedAt": "2026-11-11T11:32:11.704367169Z"
"StartedAt": "2026-11-13T09:51:55.684909892Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}
},
"3b624c924ff20f9bd2b9411028de506117558303eeb87fcfe2fe063a8dd64457": {
"ea547f7d390f190c74b9f60d5be466be91924df7ead9930e916f0e5bf9161ae0": {
"service": "openbao",
"state": {
"Status": "running",
@@ -277,14 +277,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23472,
"Pid": 940,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.196280925Z",
"FinishedAt": "2026-11-11T11:32:07.919079453Z"
"StartedAt": "2026-11-13T09:51:21.574158786Z",
"FinishedAt": "2026-11-13T09:50:28.217091485Z"
}
},
"bc79e839ab1ce72ee51c17c635c6aa2910984335f77d462d7a0eed91c98d71b1": {
"c605ad2ea7cfb8fdc37b85d5e8df440c0f9dfb1ba8317a8b87d84ce5ff38e2cb": {
"service": "otel-collector",
"state": {
"Status": "running",
@@ -293,14 +293,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 22826,
"Pid": 2156,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:16.701350044Z",
"FinishedAt": "2026-11-11T11:32:07.598332797Z"
"StartedAt": "2026-11-13T09:51:52.584501439Z",
"FinishedAt": "2026-11-13T09:50:28.224053739Z"
}
},
"7436a8dea1babed9d3ea56c4403e07a85a5a9f219b3b7a15470ed55ea7390d2d": {
"9b54b82ecd93b601a0ea977adedef1b81a1ff36eb925ab730684320a22113141": {
"service": "prometheus",
"state": {
"Status": "running",
@@ -309,14 +309,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23020,
"Pid": 2352,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:18.305444782Z",
"FinishedAt": "2026-11-11T11:32:07.463217934Z"
"StartedAt": "2026-11-13T09:51:53.97192011Z",
"FinishedAt": "2026-11-13T09:50:28.223361176Z"
}
},
"0535d5d36a7b064e43a75983a7216eb1c87191372f91eee2d51a689d43ca716a": {
"72ece8cb2659be6a83a2516dcb19b472d4ab9d6d1b50f20eca4f49799cd17131": {
"service": "loki",
"state": {
"Status": "running",
@@ -325,14 +325,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23088,
"Pid": 2359,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:18.314712046Z",
"FinishedAt": "2026-11-11T11:32:07.901781754Z"
"StartedAt": "2026-11-13T09:51:54.028034454Z",
"FinishedAt": "2026-11-13T09:50:28.224411699Z"
}
},
"f07d3b2989fd4a2a8b495aa72a49e2d2051ec0e8d5b1e1a708fec9aaf398fd44": {
"574869ba08c1a34d0d30fa2da1e61c5a35dd66ee35afff36d18366e33658e621": {
"service": "tempo",
"state": {
"Status": "running",
@@ -341,14 +341,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23064,
"Pid": 2288,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:18.325774562Z",
"FinishedAt": "2026-11-11T11:32:07.720095225Z"
"StartedAt": "2026-11-13T09:51:53.935285487Z",
"FinishedAt": "2026-11-13T09:50:28.224318796Z"
}
},
"e9763c9bf30fab2f3f98aca0b0d8328b7c0073f69d92311ff6fbfee95f3886c3": {
"4ed391665684b152c494235932554b04ed24e82fc7213219f6cfc1dc54c3e4db": {
"service": "grafana",
"state": {
"Status": "running",
@@ -357,16 +357,16 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23631,
"Pid": 2946,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:32:19.225167058Z",
"FinishedAt": "2026-11-11T11:32:08.747548051Z"
"StartedAt": "2026-11-13T09:51:56.097459745Z",
"FinishedAt": "2026-11-13T09:50:28.225512752Z"
}
}
},
"after": {
"4405619f699207cddbc05b1947545abf205d69867d859642b99bb43b4a417a12": {
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7": {
"service": "api1",
"state": {
"Status": "running",
@@ -375,42 +375,42 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31221,
"Pid": 6310,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:48.834579658Z",
"FinishedAt": "2026-11-11T11:36:37.19428573Z",
"StartedAt": "2026-11-13T09:53:15.810798832Z",
"FinishedAt": "2026-11-13T09:53:06.716904603Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-11T11:36:17.619051776Z",
"End": "2026-11-11T11:36:18.229256415Z",
"Start": "2026-11-13T09:52:45.032672537Z",
"End": "2026-11-13T09:52:45.684456356Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:23.229801236Z",
"End": "2026-11-11T11:36:23.814745647Z",
"Start": "2026-11-13T09:52:50.685337932Z",
"End": "2026-11-13T09:52:51.337051054Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:28.815878749Z",
"End": "2026-11-11T11:36:29.290543895Z",
"Start": "2026-11-13T09:52:55.883682844Z",
"End": "2026-11-13T09:52:56.745165826Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:34.291134004Z",
"End": "2026-11-11T11:36:34.789302313Z",
"Start": "2026-11-13T09:53:01.746116416Z",
"End": "2026-11-13T09:53:02.250230395Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:54.429174187Z",
"End": "2026-11-11T11:36:56.437530897Z",
"Start": "2026-11-13T09:53:21.941393572Z",
"End": "2026-11-13T09:53:25.757437396Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
}
@@ -418,7 +418,7 @@
}
}
},
"9b982ed0439d2b2a84a942b4a18cea8c11ee011e44ddcc3982a295404a1e0b65": {
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3": {
"service": "api2",
"state": {
"Status": "running",
@@ -427,42 +427,42 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31503,
"Pid": 6454,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:49.037618748Z",
"FinishedAt": "2026-11-11T11:36:39.478972969Z",
"StartedAt": "2026-11-13T09:53:15.906757854Z",
"FinishedAt": "2026-11-13T09:53:08.979949677Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-11T11:36:19.538454996Z",
"End": "2026-11-11T11:36:20.105527493Z",
"Start": "2026-11-13T09:52:46.908309853Z",
"End": "2026-11-13T09:52:47.31147321Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:25.106180943Z",
"End": "2026-11-11T11:36:25.519708718Z",
"Start": "2026-11-13T09:52:51.873454972Z",
"End": "2026-11-13T09:52:52.395517815Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:30.521209633Z",
"End": "2026-11-11T11:36:31.035009116Z",
"Start": "2026-11-13T09:52:57.397494814Z",
"End": "2026-11-13T09:52:57.897562413Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-11-11T11:36:36.038030025Z",
"End": "2026-11-11T11:36:37.222587794Z",
"Start": "2026-11-13T09:53:02.902817408Z",
"End": "2026-11-13T09:53:06.233524775Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
},
{
"Start": "2026-11-11T11:36:54.698838893Z",
"End": "2026-11-11T11:36:56.717109057Z",
"Start": "2026-11-13T09:53:22.46715569Z",
"End": "2026-11-13T09:53:25.869837051Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
}
@@ -470,7 +470,7 @@
}
}
},
"f741db3c2e8fa8f1bdc21d9acec6d44c99a3f4408ccd7c73aa1f93eae5d10b7f": {
"76e0e9ee11f2acd4202aa4641b07c2af83558e6726dd26a3521b33bb520cf177": {
"service": "db",
"state": {
"Status": "running",
@@ -479,42 +479,42 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31015,
"Pid": 5985,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:47.636821384Z",
"FinishedAt": "2026-11-11T11:36:36.448134287Z",
"StartedAt": "2026-11-13T09:53:14.810659332Z",
"FinishedAt": "2026-11-13T09:53:05.266445865Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-11-11T11:36:31.699738938Z",
"End": "2026-11-11T11:36:31.755213784Z",
"Start": "2026-11-13T09:53:02.49033307Z",
"End": "2026-11-13T09:53:02.543395178Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-11T11:36:33.755755475Z",
"End": "2026-11-11T11:36:33.823370987Z",
"Start": "2026-11-13T09:53:04.54387538Z",
"End": "2026-11-13T09:53:05.068330917Z",
"ExitCode": 1,
"Output": "/var/run/postgresql:5432 - rejecting connections\n"
},
{
"Start": "2026-11-13T09:53:18.178982393Z",
"End": "2026-11-13T09:53:20.786679419Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-11T11:36:35.824330395Z",
"End": "2026-11-11T11:36:36.016881636Z",
"Start": "2026-11-13T09:53:22.800575887Z",
"End": "2026-11-13T09:53:24.416791098Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-11-11T11:36:51.042630534Z",
"End": "2026-11-11T11:36:53.443029924Z",
"ExitCode": -1,
"Output": "Health check exceeded timeout (2s)"
},
{
"Start": "2026-11-11T11:36:55.444290175Z",
"End": "2026-11-11T11:36:56.386763854Z",
"Start": "2026-11-13T09:53:26.420377151Z",
"End": "2026-11-13T09:53:26.842575441Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
@@ -522,7 +522,7 @@
}
}
},
"95bdec59b97137172ccff74afeae1b59a7314e835805f50edccf4a17b62b2184": {
"051a2a59f89f08acf5878b6ab22e2ad389bc09d4fa3d655604f27375b26922be": {
"service": "cache",
"state": {
"Status": "running",
@@ -531,14 +531,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31404,
"Pid": 6487,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:49.015388053Z",
"FinishedAt": "2026-11-11T11:36:36.777818156Z"
"StartedAt": "2026-11-13T09:53:15.945227154Z",
"FinishedAt": "2026-11-13T09:53:05.04199852Z"
}
},
"9ac40379ff6275dcde3d4b2d317dcd9faa88760c6c7eec71cac36c79420407c7": {
"c0e3fc648ea6a23b643a0104ad9bf93f06bfd45e592fece24d8c4dc8788dc494": {
"service": "lb",
"state": {
"Status": "running",
@@ -547,14 +547,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31654,
"Pid": 6479,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:49.128762496Z",
"FinishedAt": "2026-11-11T11:36:35.203826427Z"
"StartedAt": "2026-11-13T09:53:15.9062729Z",
"FinishedAt": "2026-11-13T09:53:03.909611163Z"
}
},
"a02442c9d071c5eb11020c0bbcc6c751faf70103d001627fd0cf611447cd9066": {
"68c83b0d029545c4c8bfa67979e23fda461d7b9aea909d5e99e462f4858fcc1a": {
"service": "s3",
"state": {
"Status": "running",
@@ -563,14 +563,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31378,
"Pid": 6375,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:48.992942658Z",
"FinishedAt": "2026-11-11T11:36:41.576301537Z"
"StartedAt": "2026-11-13T09:53:15.849272466Z",
"FinishedAt": "2026-11-13T09:53:10.186208925Z"
}
},
"b8191296010574753cf4e61b0a6605453a488df49ef03488fd8b40ab342724d4": {
"6db578724c16067c9101d8ffb911ae3bb1127c087e0b8a598c958983914676a7": {
"service": "reconciler",
"state": {
"Status": "running",
@@ -579,14 +579,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31556,
"Pid": 6191,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:49.077635717Z",
"FinishedAt": "2026-11-11T11:36:41.898035512Z"
"StartedAt": "2026-11-13T09:53:15.708018867Z",
"FinishedAt": "2026-11-13T09:53:09.175300496Z"
}
},
"c8638dec3e880a026dd9d16fee7cc02b52c77eb4afa46a666ec5ce29454c2c13": {
"362e538c9c53fa7feb054689ed85f862d070518a168c0efaa6e184ad51bd34f5": {
"service": "broker",
"state": {
"Status": "running",
@@ -595,14 +595,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31537,
"Pid": 6125,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:49.0947063Z",
"FinishedAt": "2026-11-11T11:36:36.493690622Z"
"StartedAt": "2026-11-13T09:53:15.753030573Z",
"FinishedAt": "2026-11-13T09:53:05.167583954Z"
}
},
"dcfba1124ca8ecdd54c327b95ea2f1f78194e50a9ab597d70a88fad735be05ce": {
"2e513af66f171401771b1a3680154f7edfc7472a5ee79e94041256efee78b2f0": {
"service": "worker",
"state": {
"Status": "running",
@@ -611,14 +611,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31330,
"Pid": 6437,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:48.97384398Z",
"FinishedAt": "2026-11-11T11:36:41.462706667Z"
"StartedAt": "2026-11-13T09:53:15.870744073Z",
"FinishedAt": "2026-11-13T09:53:09.192037057Z"
}
},
"8356b6e281eb194949c11425f8312df9bcbb9bed34e52cc453884db43c974bfd": {
"c40815a4d87373db33cf7eb4b89d473b853dd7a9ece81fa73e052da54ced5ff4": {
"service": "dispatcher",
"state": {
"Status": "running",
@@ -627,14 +627,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31260,
"Pid": 6311,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:48.892574349Z",
"FinishedAt": "2026-11-11T11:36:41.425636939Z"
"StartedAt": "2026-11-13T09:53:15.806673713Z",
"FinishedAt": "2026-11-13T09:53:09.581490205Z"
}
},
"3b624c924ff20f9bd2b9411028de506117558303eeb87fcfe2fe063a8dd64457": {
"ea547f7d390f190c74b9f60d5be466be91924df7ead9930e916f0e5bf9161ae0": {
"service": "openbao",
"state": {
"Status": "running",
@@ -643,14 +643,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31763,
"Pid": 6517,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:49.125908742Z",
"FinishedAt": "2026-11-11T11:36:36.817840301Z"
"StartedAt": "2026-11-13T09:53:15.911866476Z",
"FinishedAt": "2026-11-13T09:53:03.972241104Z"
}
},
"bc79e839ab1ce72ee51c17c635c6aa2910984335f77d462d7a0eed91c98d71b1": {
"c605ad2ea7cfb8fdc37b85d5e8df440c0f9dfb1ba8317a8b87d84ce5ff38e2cb": {
"service": "otel-collector",
"state": {
"Status": "running",
@@ -659,14 +659,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 30797,
"Pid": 5717,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:46.394339379Z",
"FinishedAt": "2026-11-11T11:36:36.866663071Z"
"StartedAt": "2026-11-13T09:53:13.768718976Z",
"FinishedAt": "2026-11-13T09:53:05.062618953Z"
}
},
"7436a8dea1babed9d3ea56c4403e07a85a5a9f219b3b7a15470ed55ea7390d2d": {
"9b54b82ecd93b601a0ea977adedef1b81a1ff36eb925ab730684320a22113141": {
"service": "prometheus",
"state": {
"Status": "running",
@@ -675,14 +675,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31026,
"Pid": 5926,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:47.651135168Z",
"FinishedAt": "2026-11-11T11:36:36.814800908Z"
"StartedAt": "2026-11-13T09:53:14.756925187Z",
"FinishedAt": "2026-11-13T09:53:05.244841875Z"
}
},
"0535d5d36a7b064e43a75983a7216eb1c87191372f91eee2d51a689d43ca716a": {
"72ece8cb2659be6a83a2516dcb19b472d4ab9d6d1b50f20eca4f49799cd17131": {
"service": "loki",
"state": {
"Status": "running",
@@ -691,14 +691,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31005,
"Pid": 5961,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:47.641342828Z",
"FinishedAt": "2026-11-11T11:36:37.55838066Z"
"StartedAt": "2026-11-13T09:53:14.793956064Z",
"FinishedAt": "2026-11-13T09:53:05.416062037Z"
}
},
"f07d3b2989fd4a2a8b495aa72a49e2d2051ec0e8d5b1e1a708fec9aaf398fd44": {
"574869ba08c1a34d0d30fa2da1e61c5a35dd66ee35afff36d18366e33658e621": {
"service": "tempo",
"state": {
"Status": "running",
@@ -707,14 +707,14 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31037,
"Pid": 5958,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:47.645290633Z",
"FinishedAt": "2026-11-11T11:36:37.448506459Z"
"StartedAt": "2026-11-13T09:53:14.761611926Z",
"FinishedAt": "2026-11-13T09:53:05.299312685Z"
}
},
"e9763c9bf30fab2f3f98aca0b0d8328b7c0073f69d92311ff6fbfee95f3886c3": {
"4ed391665684b152c494235932554b04ed24e82fc7213219f6cfc1dc54c3e4db": {
"service": "grafana",
"state": {
"Status": "running",
@@ -723,11 +723,11 @@
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31643,
"Pid": 6442,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-11-11T11:36:49.142027651Z",
"FinishedAt": "2026-11-11T11:36:39.67135251Z"
"StartedAt": "2026-11-13T09:53:15.90242133Z",
"FinishedAt": "2026-11-13T09:53:05.770095425Z"
}
}
},
+84 -56
View File
@@ -2,223 +2,251 @@
"passed": true,
"secret_refresh_seconds": 20,
"token_max_TTL_seconds": 120,
"both_API_children_stopped_seconds": 109.66,
"recovery_seconds": 21.27,
"both_API_children_stopped_seconds": 123.86,
"recovery_seconds": 20.44,
"observations": [
{
"seconds": 0.8,
"seconds": 1.3,
"HTTP": 200
},
{
"seconds": 2.8,
"seconds": 3.3,
"HTTP": 200
},
{
"seconds": 4.9,
"seconds": 5.4,
"HTTP": 200
},
{
"seconds": 6.9,
"seconds": 7.4,
"HTTP": 200
},
{
"seconds": 8.9,
"seconds": 9.4,
"HTTP": 200
},
{
"seconds": 11.0,
"seconds": 11.5,
"HTTP": 200
},
{
"seconds": 13.0,
"seconds": 13.5,
"HTTP": 200
},
{
"seconds": 15.0,
"seconds": 15.5,
"HTTP": 200
},
{
"seconds": 17.1,
"seconds": 17.5,
"HTTP": 200
},
{
"seconds": 19.1,
"seconds": 19.6,
"HTTP": 200
},
{
"seconds": 21.1,
"seconds": 21.6,
"HTTP": 200
},
{
"seconds": 23.1,
"seconds": 23.6,
"HTTP": 200
},
{
"seconds": 25.2,
"seconds": 25.7,
"HTTP": 200
},
{
"seconds": 27.2,
"seconds": 27.7,
"HTTP": 200
},
{
"seconds": 29.2,
"seconds": 29.7,
"HTTP": 200
},
{
"seconds": 31.2,
"seconds": 31.8,
"HTTP": 200
},
{
"seconds": 33.3,
"seconds": 33.8,
"HTTP": 200
},
{
"seconds": 35.3,
"seconds": 35.8,
"HTTP": 200
},
{
"seconds": 37.4,
"seconds": 37.9,
"HTTP": 200
},
{
"seconds": 39.4,
"seconds": 39.9,
"HTTP": 200
},
{
"seconds": 41.4,
"seconds": 42.0,
"HTTP": 200
},
{
"seconds": 43.4,
"seconds": 44.0,
"HTTP": 200
},
{
"seconds": 45.5,
"seconds": 46.0,
"HTTP": 200
},
{
"seconds": 47.5,
"seconds": 48.0,
"HTTP": 200
},
{
"seconds": 49.5,
"seconds": 50.1,
"HTTP": 200
},
{
"seconds": 51.6,
"seconds": 52.1,
"HTTP": 200
},
{
"seconds": 53.6,
"seconds": 54.1,
"HTTP": 200
},
{
"seconds": 55.6,
"seconds": 56.2,
"HTTP": 200
},
{
"seconds": 57.7,
"seconds": 58.2,
"HTTP": 200
},
{
"seconds": 59.7,
"seconds": 60.2,
"HTTP": 200
},
{
"seconds": 61.7,
"seconds": 62.3,
"HTTP": 200
},
{
"seconds": 63.8,
"seconds": 64.3,
"HTTP": 200
},
{
"seconds": 65.8,
"seconds": 66.3,
"HTTP": 200
},
{
"seconds": 67.8,
"seconds": 68.4,
"HTTP": 200
},
{
"seconds": 69.8,
"seconds": 70.4,
"HTTP": 200
},
{
"seconds": 71.9,
"seconds": 72.4,
"HTTP": 200
},
{
"seconds": 73.9,
"seconds": 74.5,
"HTTP": 200
},
{
"seconds": 76.0,
"seconds": 76.5,
"HTTP": 200
},
{
"seconds": 78.0,
"seconds": 78.5,
"HTTP": 200
},
{
"seconds": 80.0,
"seconds": 80.5,
"HTTP": 200
},
{
"seconds": 82.0,
"seconds": 82.6,
"HTTP": 200
},
{
"seconds": 84.1,
"seconds": 84.6,
"HTTP": 200
},
{
"seconds": 86.1,
"seconds": 86.7,
"HTTP": 200
},
{
"seconds": 88.1,
"seconds": 88.7,
"HTTP": 200
},
{
"seconds": 90.2,
"seconds": 90.8,
"HTTP": 200
},
{
"seconds": 92.2,
"seconds": 92.8,
"HTTP": 200
},
{
"seconds": 94.2,
"seconds": 94.8,
"HTTP": 200
},
{
"seconds": 96.3,
"seconds": 96.8,
"HTTP": 200
},
{
"seconds": 98.3,
"seconds": 98.8,
"HTTP": 200
},
{
"seconds": 100.3,
"seconds": 100.9,
"HTTP": 200
},
{
"seconds": 102.4,
"seconds": 102.9,
"HTTP": 200
},
{
"seconds": 104.4,
"seconds": 104.9,
"HTTP": 200
},
{
"seconds": 107.0,
"HTTP": 200
},
{
"seconds": 109.0,
"HTTP": 200
},
{
"seconds": 111.0,
"HTTP": 200
},
{
"seconds": 113.1,
"HTTP": 200
},
{
"seconds": 115.1,
"HTTP": 200
},
{
"seconds": 117.1,
"HTTP": 200
},
{
"seconds": 119.2,
"HTTP": 502
},
{
"seconds": 106.5,
"seconds": 121.2,
"HTTP": 502
},
{
"seconds": 108.5,
"seconds": 123.2,
"HTTP": 503
}
],
+104 -104
View File
@@ -8,123 +8,123 @@
"login_burst": [
{
"status": 401,
"seconds": 0.469,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
},
{
"status": 401,
"seconds": 2.25,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
},
{
"status": 429,
"seconds": 0.5,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
},
{
"status": 429,
"seconds": 0.563,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
},
{
"status": 401,
"seconds": 1.829,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
},
{
"status": 401,
"seconds": 2.032,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
},
{
"status": 401,
"seconds": 1.657,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
},
{
"status": 401,
"seconds": 1.938,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
},
{
"status": 401,
"seconds": 1.954,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
},
{
"status": 401,
"seconds": 1.766,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
},
{
"status": 429,
"seconds": 0.469,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
},
{
"status": 401,
"seconds": 2.344,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
},
{
"status": 401,
"seconds": 2.532,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
},
{
"status": 429,
"seconds": 0.563,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
},
{
"status": 401,
"seconds": 2.125,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
},
{
"status": 429,
"seconds": 0.485,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
},
{
"status": 429,
"seconds": 0.547,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
},
{
"status": 401,
"seconds": 2.609,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
"seconds": 2.891,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.453,
"instance": "c330b523cae74f8c92f7bb395bba53f1"
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 4.078,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 3.094,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.281,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 3.188,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 3.375,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.281,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 2.11,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 3.891,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 4.547,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 4.062,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 3.781,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.5,
"instance": "ad91779af4504c6691ac2a1f9f7def96"
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 429,
"seconds": 0.531,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 401,
"seconds": 3.125,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 429,
"seconds": 0.39,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 401,
"seconds": 4.453,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
},
{
"status": 429,
"seconds": 0.39,
"instance": "9a29628cf7004fbfb295d86c41370777"
},
{
"status": 429,
"seconds": 0.453,
"instance": "8f1af19439ef4f8d9dbd95d4400217f7"
}
],
"burst_seconds": 2.64,
"burst_seconds": 4.578,
"KDF": {
"hash_plus_verify_seconds": [
0.2785999540000148,
0.24090507699997943,
0.35305704199993215,
0.40253531700000167,
0.5070911510000542,
0.21940843400000176,
0.19147180599998137,
0.20047324400002253,
0.2186746250000624,
0.19908314199994948,
0.20232280300001548,
0.2009935690000475
0.21583701900004826,
0.20667367100003275,
0.22015716700002486,
0.193256392999956,
0.20241064300000744,
0.20739852500003053,
0.20553775699994503,
0.2716375199999561,
0.2128679730000158,
0.22063045700002704,
0.20332950599998867,
0.20581266300001744
],
"median": 0.21904152950003208,
"max_rss_kib": 53164
"median": 0.20703609800003164,
"max_rss_kib": 53184
},
"KDF_slots_per_API": 2,
"KDF_memory_bound_two_APIs_MiB": 128
+4
View File
@@ -0,0 +1,4 @@
{
"passed": true,
"completed_at_UTC": "2026-11-13T10:00:37.671336+00:00"
}
+11
View File
@@ -0,0 +1,11 @@
generate-test-identities: PASS
publish-crl: PASS
tls-matrix: PASS
bao-policy: PASS
worker-rights: PASS
security-api-run: PASS
compromised-worker-run: PASS
sealed-bao-run: PASS
telemetry-security-run: PASS
hardening-run: PASS
render-unit: PASS
+5 -5
View File
@@ -1,11 +1,11 @@
{
"target_instance": "16a7fabd28ee40b4a4a7ddf7cc070896",
"stopped_container": "4405619f699207cddbc05b1947545abf205d69867d859642b99bb43b4a417a12",
"target_instance": "932b533b493e4dab8e09817e66f77901",
"stopped_container": "1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"before_containers": [
"4405619f699207cddbc05b1947545abf205d69867d859642b99bb43b4a417a12",
"9b982ed0439d2b2a84a942b4a18cea8c11ee011e44ddcc3982a295404a1e0b65"
"1570e3cf0bcaf52840456fe4527c951913c184c7a910035c662fd93573f1c1c7",
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
],
"after_running_containers": [
"9b982ed0439d2b2a84a942b4a18cea8c11ee011e44ddcc3982a295404a1e0b65"
"0a05a29d377fb35aa4676b597fa301dc1625ccfe208a7edbc6a74794659407b3"
]
}
+2 -2
View File
@@ -1,6 +1,6 @@
{
"passed": true,
"trace_id": "6b4f8b9212d0439e8056644eedfff17e",
"trace_id": "918070f9c17e4f2b9b8e079df1824a06",
"span_count": 13,
"span_names": [
"POST /api/v1/chats/{chat_id}/attachments",
@@ -14,7 +14,7 @@
"s3.put"
],
"correlated_log_count": 3,
"http_metric_series": 25,
"http_metric_series": 26,
"credential_canaries_absent": true,
"all_queries": "mTLS verified operator identity",
"container_logs_canaries_absent": true
+1 -1
View File
@@ -1,6 +1,6 @@
{
"passed": true,
"seconds": 3.9,
"seconds": 2.49,
"edges": [
{
"source": "lb",