date: 2026-10-30T04:29:14Z  commit: 81dd816
== before: api replicas (hostname = X-Instance-Id), state, StartedAt, FinishedAt
0cc3ee358e59 running started=2026-10-30T04:28:59.851144131Z finished=0001-01-01T00:00:00Z
899a952a5a21 running started=2026-10-30T04:29:11.286693179Z finished=2026-10-30T04:29:11.040183429Z
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18084" 
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 129, "failed_attempts": 0, "first_recovery_seconds": 2.885}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: api replicas
0cc3ee358e59 running started=2026-10-30T04:28:59.851144131Z finished=0001-01-01T00:00:00Z
899a952a5a21 exited started=2026-10-30T04:29:11.286693179Z finished=2026-10-30T04:29:18.153900709Z
== stopped replica: 899a952a5a21; shutdown: signal, drain window with readiness 503, close, exit
{"event":"Handling signal: term","level":"info","ts":"2026-10-30T04:29:15.605798Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-10-30T04:29:15.638898Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-10-30T04:29:15.701830Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"b416afb071ad426a9dde141e952d697f","duration_ms":0.823,"event":"request.handled","level":"info","ts":"2026-10-30T04:29:16.528606Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"3fe4a4c39cb24a219d3a393a1c0332f0","duration_ms":1.591,"event":"request.handled","level":"info","ts":"2026-10-30T04:29:17.038813Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"152273913ad34012bf5eaf31432490ae","duration_ms":0.263,"event":"request.handled","level":"info","ts":"2026-10-30T04:29:17.388797Z"}
{"event":"Shutting down","level":"info","ts":"2026-10-30T04:29:17.642635Z"}
{"event":"Shutting down","level":"info","ts":"2026-10-30T04:29:17.714902Z"}
{"event":"service.stopped","level":"info","ts":"2026-10-30T04:29:17.757838Z"}
{"event":"Finished server process [20]","level":"info","ts":"2026-10-30T04:29:17.758374Z"}
{"event":"Worker (pid:20) was sent SIGTERM!","level":"info","ts":"2026-10-30T04:29:17.826021Z"}
{"event":"service.stopped","level":"info","ts":"2026-10-30T04:29:17.832655Z"}
{"event":"Finished server process [13]","level":"info","ts":"2026-10-30T04:29:17.833005Z"}
{"event":"Worker (pid:13) was sent SIGTERM!","level":"info","ts":"2026-10-30T04:29:17.927620Z"}
{"event":"Shutting down: Master","level":"info","ts":"2026-10-30T04:29:18.029723Z"}
$ scripts/restore.sh
 Container mtusi2026lab4-api-1  Running
 Container mtusi2026lab4-api-2  Starting
 Container mtusi2026lab4-api-2  Started
 Container mtusi2026lab4-api-2  Waiting
 Container mtusi2026lab4-api-1  Waiting
 Container mtusi2026lab4-api-2  Healthy
 Container mtusi2026lab4-api-1  Healthy
== after restore
0cc3ee358e59 running started=2026-10-30T04:28:59.851144131Z finished=0001-01-01T00:00:00Z
899a952a5a21 running started=2026-10-30T04:29:31.104320507Z finished=2026-10-30T04:29:18.153900709Z
