date: 2026-11-06T09:10:22Z  commit: 75b93bbba5f29575d3fca1d013cc7e0770d88082
== before: api replicas (hostname = X-Instance-Id), state, StartedAt, FinishedAt
d4c8d53d7433 running started=2026-11-06T09:10:18.309662551Z finished=2026-11-06T09:10:12.020503506Z
da922ab2ea09 running started=2026-11-06T09:10:18.377623592Z finished=2026-11-06T09:10:12.004100798Z
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18084"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 130, "failed_attempts": 0, "first_recovery_seconds": 2.883}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: api replicas
d4c8d53d7433 exited started=2026-11-06T09:10:18.309662551Z finished=2026-11-06T09:10:25.313223596Z
da922ab2ea09 running started=2026-11-06T09:10:18.377623592Z finished=2026-11-06T09:10:12.004100798Z
== stopped replica: d4c8d53d7433; shutdown: signal, drain window with readiness 503, close, exit
{"event":"Handling signal: term","level":"info","ts":"2026-11-06T09:10:22.774795Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-11-06T09:10:22.776559Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-11-06T09:10:22.848442Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"c23539767f7c4ef5ba39f58a446754a2","duration_ms":1.481,"event":"request.handled","level":"info","ts":"2026-11-06T09:10:23.545632Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"60444a3eba784d839b8caccbaa24d1f2","duration_ms":0.278,"event":"request.handled","level":"info","ts":"2026-11-06T09:10:24.058530Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"5a247b2f9b304ff789354712b9c2d857","duration_ms":0.282,"event":"request.handled","level":"info","ts":"2026-11-06T09:10:24.545468Z"}
{"event":"Shutting down","level":"info","ts":"2026-11-06T09:10:24.779511Z"}
{"event":"Shutting down","level":"info","ts":"2026-11-06T09:10:24.852501Z"}
{"event":"service.stopped","level":"info","ts":"2026-11-06T09:10:24.884399Z"}
{"event":"Finished server process [19]","level":"info","ts":"2026-11-06T09:10:24.884667Z"}
{"event":"service.stopped","level":"info","ts":"2026-11-06T09:10:24.959721Z"}
{"event":"Finished server process [20]","level":"info","ts":"2026-11-06T09:10:24.959901Z"}
{"event":"Worker (pid:19) was sent SIGTERM!","level":"info","ts":"2026-11-06T09:10:24.961675Z"}
{"event":"Worker (pid:20) was sent SIGTERM!","level":"info","ts":"2026-11-06T09:10:25.072586Z"}
{"event":"Shutting down: Master","level":"info","ts":"2026-11-06T09:10:25.178811Z"}
$ scripts/restore.sh
 Container hlpipe-lab4-forward-final-20261106-api-2  Running
 Container hlpipe-lab4-forward-final-20261106-api-1  Starting
 Container hlpipe-lab4-forward-final-20261106-api-1  Started
 Container hlpipe-lab4-forward-final-20261106-api-1  Waiting
 Container hlpipe-lab4-forward-final-20261106-api-2  Waiting
 Container hlpipe-lab4-forward-final-20261106-api-1  Healthy
 Container hlpipe-lab4-forward-final-20261106-api-2  Healthy
== after restore
d4c8d53d7433 running started=2026-11-06T09:10:38.256484379Z finished=2026-11-06T09:10:25.313223596Z
da922ab2ea09 running started=2026-11-06T09:10:18.377623592Z finished=2026-11-06T09:10:12.004100798Z
