date: 2026-10-30T02:45:08Z  commit: e049c26
== before: api replicas (hostname = X-Instance-Id), state, StartedAt, FinishedAt
ff81f93e8b0d running started=2026-10-30T02:44:57.018195381Z finished=0001-01-01T00:00:00Z
fe75c6c37e5a running started=2026-10-30T02:44:57.104698047Z finished=0001-01-01T00:00:00Z
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18083" 
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 132, "failed_attempts": 0, "first_recovery_seconds": 2.858}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: api replicas
ff81f93e8b0d exited started=2026-10-30T02:44:57.018195381Z finished=2026-10-30T02:45:12.063064055Z
fe75c6c37e5a running started=2026-10-30T02:44:57.104698047Z finished=0001-01-01T00:00:00Z
== stopped replica: ff81f93e8b0d; shutdown: signal, drain window with readiness 503, close, exit
{"event":"Handling signal: term","level":"info","ts":"2026-10-30T02:45:09.656770Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-10-30T02:45:09.714534Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-10-30T02:45:09.722268Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"468d7db9c6d24e9797a43d901e9bdbb1","duration_ms":0.312,"event":"request.handled","level":"info","ts":"2026-10-30T02:45:10.589248Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"771ff57a02304724b1de9c40033c5157","duration_ms":0.304,"event":"request.handled","level":"info","ts":"2026-10-30T02:45:11.091968Z"}
{"event":"Shutting down","level":"info","ts":"2026-10-30T02:45:11.715907Z"}
{"event":"Shutting down","level":"info","ts":"2026-10-30T02:45:11.728294Z"}
{"event":"service.stopped","level":"info","ts":"2026-10-30T02:45:11.826861Z"}
{"event":"Finished server process [14]","level":"info","ts":"2026-10-30T02:45:11.827508Z"}
{"event":"service.stopped","level":"info","ts":"2026-10-30T02:45:11.835753Z"}
{"event":"Finished server process [13]","level":"info","ts":"2026-10-30T02:45:11.835868Z"}
{"event":"Worker (pid:13) was sent SIGTERM!","level":"info","ts":"2026-10-30T02:45:11.887319Z"}
{"event":"Worker (pid:14) was sent SIGTERM!","level":"info","ts":"2026-10-30T02:45:11.887404Z"}
{"event":"Shutting down: Master","level":"info","ts":"2026-10-30T02:45:11.990788Z"}
$ scripts/restore.sh
 Container mtusi2026lab3-api-2  Running
 Container mtusi2026lab3-api-1  Starting
 Container mtusi2026lab3-api-1  Started
 Container mtusi2026lab3-api-1  Waiting
 Container mtusi2026lab3-api-2  Waiting
 Container mtusi2026lab3-api-1  Healthy
 Container mtusi2026lab3-api-2  Healthy
== after restore
ff81f93e8b0d running started=2026-10-30T02:45:24.812924046Z finished=2026-10-30T02:45:12.063064055Z
fe75c6c37e5a running started=2026-10-30T02:44:57.104698047Z finished=0001-01-01T00:00:00Z
