evidence/level4.json: all 17 scenarios of levels 3 and 4 and the eight invariants pass on 0b85edf; evidence/operator.txt shows the failed registry, a replay to ready, a noop replay, an expired replay and the DLQ. Smoke, topology, level3, persistence, failover, lab 4 scenarios, limits, student tests and contract checks are refreshed on the same commit. REPORT.md describes the variants, worker limits, jitter, lease renewal, crash points, replay and retention, adds rows 4.1-4.4, the level 4 failure experiments and the updated limitations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
40 lines
3.2 KiB
Plaintext
40 lines
3.2 KiB
Plaintext
date: 2026-10-29T00:33:32Z commit: 0b85edf
|
|
== before: api replicas (hostname = X-Instance-Id), state, StartedAt, FinishedAt
|
|
095cb9cc85bd running started=2026-10-29T00:33:12.427394173Z finished=2026-10-29T00:33:03.853644169Z
|
|
70cfba801128 running started=2026-10-29T00:33:12.607365173Z finished=2026-10-29T00:33:03.666460502Z
|
|
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
|
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:18085"
|
|
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
|
{"observed_instances_before": 2, "successful_writes": 133, "failed_attempts": 0, "first_recovery_seconds": 3.005}
|
|
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
|
== after: api replicas
|
|
095cb9cc85bd running started=2026-10-29T00:33:12.427394173Z finished=2026-10-29T00:33:03.853644169Z
|
|
70cfba801128 exited started=2026-10-29T00:33:12.607365173Z finished=2026-10-29T00:33:39.052027754Z
|
|
== stopped replica: 70cfba801128; shutdown: signal, drain window with readiness 503, close, exit
|
|
{"event":"Handling signal: term","level":"info","ts":"2026-10-29T00:33:36.479119Z"}
|
|
{"event":"Draining for 2.0 s","level":"info","ts":"2026-10-29T00:33:36.486455Z"}
|
|
{"event":"Draining for 2.0 s","level":"info","ts":"2026-10-29T00:33:36.558375Z"}
|
|
{"method":"GET","route":"/health/ready","status":503,"request_id":"b3d9118f93b94471b698d536dda11072","duration_ms":0.772,"event":"request.handled","level":"info","ts":"2026-10-29T00:33:37.011026Z"}
|
|
{"method":"GET","route":"/health/ready","status":503,"request_id":"3c1324def7e74f3484c4fbd242f7f70b","duration_ms":0.655,"event":"request.handled","level":"info","ts":"2026-10-29T00:33:37.515877Z"}
|
|
{"method":"GET","route":"/health/ready","status":503,"request_id":"e56c9504272b45a89b7372020eb80d15","duration_ms":0.733,"event":"request.handled","level":"info","ts":"2026-10-29T00:33:37.588547Z"}
|
|
{"event":"Shutting down","level":"info","ts":"2026-10-29T00:33:38.487910Z"}
|
|
{"event":"Shutting down","level":"info","ts":"2026-10-29T00:33:38.560300Z"}
|
|
{"event":"service.stopped","level":"info","ts":"2026-10-29T00:33:38.592638Z"}
|
|
{"event":"Finished server process [13]","level":"info","ts":"2026-10-29T00:33:38.592934Z"}
|
|
{"event":"Worker (pid:13) was sent SIGTERM!","level":"info","ts":"2026-10-29T00:33:38.624866Z"}
|
|
{"event":"service.stopped","level":"info","ts":"2026-10-29T00:33:38.664775Z"}
|
|
{"event":"Finished server process [14]","level":"info","ts":"2026-10-29T00:33:38.665027Z"}
|
|
{"event":"Worker (pid:14) was sent SIGTERM!","level":"info","ts":"2026-10-29T00:33:38.725950Z"}
|
|
{"event":"Shutting down: Master","level":"info","ts":"2026-10-29T00:33:38.834487Z"}
|
|
$ scripts/restore.sh
|
|
Container mtusi2026lab5-api-1 Running
|
|
Container mtusi2026lab5-api-2 Starting
|
|
Container mtusi2026lab5-api-2 Started
|
|
Container mtusi2026lab5-api-1 Waiting
|
|
Container mtusi2026lab5-api-2 Waiting
|
|
Container mtusi2026lab5-api-1 Healthy
|
|
Container mtusi2026lab5-api-2 Healthy
|
|
== after restore
|
|
095cb9cc85bd running started=2026-10-29T00:33:12.427394173Z finished=2026-10-29T00:33:03.853644169Z
|
|
70cfba801128 running started=2026-10-29T00:33:51.687259594Z finished=2026-10-29T00:33:39.052027754Z
|