date: 2026-11-06T09:34:53Z  commit: 95511002b12510178a3cc2a22566b43e2b2b6c81
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ env EVIDENCE_DIR=evidence/grade3/failover-second make failover-evidence SHIFT=1
sh scripts/failover-evidence.sh "http://127.0.0.1:18086" "1"
date: 2026-11-06T09:34:53Z  commit: 95511002b12510178a3cc2a22566b43e2b2b6c81
== before: api replicas (hostname = X-Instance-Id), state, StartedAt, FinishedAt
9af2280c8af3 running started=2026-11-06T09:34:51.34604076Z finished=2026-11-06T09:34:38.360642671Z
943813ecb712 running started=2026-11-06T09:33:56.539288471Z finished=2026-11-06T09:33:50.07508976Z
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18086" 
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 133, "failed_attempts": 0, "first_recovery_seconds": 2.777}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: api replicas
9af2280c8af3 running started=2026-11-06T09:34:51.34604076Z finished=2026-11-06T09:34:38.360642671Z
943813ecb712 exited started=2026-11-06T09:33:56.539288471Z finished=2026-11-06T09:34:58.204116764Z
== stopped replica: 943813ecb712; shutdown: signal, drain window with readiness 503, close, exit
{"event":"Handling signal: term","level":"info","ts":"2026-11-06T09:34:55.700248Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-11-06T09:34:55.784538Z"}
{"limit":2.0,"event":"service.draining","level":"info","ts":"2026-11-06T09:34:55.795484Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"ff6952bfefbe4034ae30489d4e43b93c","duration_ms":0.567,"event":"request.handled","level":"info","ts":"2026-11-06T09:34:56.029040Z"}
{"method":"GET","route":"/health/ready","status":503,"request_id":"194add29fb3742db93424a18cf9ff680","duration_ms":1.596,"event":"request.handled","level":"info","ts":"2026-11-06T09:34:56.535013Z"}
{"event":"Shutting down","level":"info","ts":"2026-11-06T09:34:57.787659Z"}
{"event":"Shutting down","level":"info","ts":"2026-11-06T09:34:57.796586Z"}
{"event":"service.stopped","level":"info","ts":"2026-11-06T09:34:57.894245Z"}
{"event":"Finished server process [21]","level":"info","ts":"2026-11-06T09:34:57.896210Z"}
{"event":"service.stopped","level":"info","ts":"2026-11-06T09:34:57.903471Z"}
{"event":"Finished server process [23]","level":"info","ts":"2026-11-06T09:34:57.905749Z"}
{"event":"Worker (pid:21) was sent SIGTERM!","level":"info","ts":"2026-11-06T09:34:57.954114Z"}
{"event":"Worker (pid:23) was sent SIGTERM!","level":"info","ts":"2026-11-06T09:34:57.954212Z"}
{"event":"Shutting down: Master","level":"info","ts":"2026-11-06T09:34:58.057111Z"}
$ scripts/restore.sh
 Container mtusi2026lab6-api-1  Running
 Container mtusi2026lab6-api-2  Starting
 Container mtusi2026lab6-api-2  Started
 Container mtusi2026lab6-api-2  Waiting
 Container mtusi2026lab6-api-1  Waiting
 Container mtusi2026lab6-api-2  Healthy
 Container mtusi2026lab6-api-1  Healthy
== after restore
9af2280c8af3 running started=2026-11-06T09:34:51.34604076Z finished=2026-11-06T09:34:38.360642671Z
943813ecb712 running started=2026-11-06T09:35:11.260329797Z finished=2026-11-06T09:34:58.204116764Z
exit_code: 0  elapsed_seconds: 20.372
