Files
mtusi2026lab3/evidence/test-failover.txt
T
Alexey BodanyukandClaude Opus 5.5 c758e49cbb docs: REPORT lab3 — доказательства со стенда из чистого clone, уровень 4, трек 1
Второй прогон scripts/collect_evidence.py на новом clone (коммит 2708732):
make check/validate/test, test-persistence (новые PID у lb, api x2,
postgres, valkey), test-failover для обеих реплик по очереди (0 неудачных
попыток), 201 свой тест, серия отказов stop/kill/pause x 2 реплики,
rolling update под нагрузкой (0 ошибок), tests/load.py. Таблица
критерий -> команда -> результат -> файл, раздел уровня 4 с периодами
проверок и временем обнаружения.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 18:28:25 +03:00

75 lines
3.7 KiB
Plaintext

# make test-failover with scripts/stop-one: which container is really stopped, restore, repeat
# 2026-10-19T15:12:01Z, commit 2708732 fix: LB и на [::1] — localhost на Windows стоил 2 с на запрос; сдвиг цели failover
--- run 1; running API replicas BEFORE:
mtusi-lab3-api-1 host=94b38b941a8d running exit=0 pid=27148 started=2026-10-19T15:11:53.132
mtusi-lab3-api-2 host=a8d9cffd7a6e running exit=0 pid=27091 started=2026-10-19T15:11:52.824
$ make test-failover ACTION_SCRIPT=scripts/stop-one.cmd
python tests/resilience.py --mode failover --action-script "scripts/stop-one.cmd" --base-url "http://localhost:8080"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 131, "failed_attempts": 0, "first_recovery_seconds": 5.902}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
[exit 0, 16.4 s]
AFTER (the stopped replica, left stopped by the hook):
mtusi-lab3-api-1 host=94b38b941a8d exited exit=0 pid=0 started=2026-10-19T15:11:53.132
$ C:\Users\wfa22\AppData\Local\Python\pythoncore-3.14-64\python.exe scripts/lbctl.py status
api2 172.23.0.2 UP mtusi-lab3-api-2 X-Instance-Id=a8d9cffd7a6e
(other slots: MAINT, no container)
[exit 0, 0.7 s]
$ sh scripts/restore-api.sh
mtusi-lab3-api-1 Up 6 seconds (healthy)
mtusi-lab3-api-2 Up 33 seconds (healthy)
Container mtusi-lab3-api-2 Running
Container mtusi-lab3-api-1 Starting
Container mtusi-lab3-api-1 Started
Container mtusi-lab3-api-2 Waiting
Container mtusi-lab3-api-1 Waiting
Container mtusi-lab3-api-2 Healthy
Container mtusi-lab3-api-1 Healthy
[exit 0, 6.9 s]
$ C:\Users\wfa22\AppData\Local\Python\pythoncore-3.14-64\python.exe scripts/lbctl.py wait-up 2
2 API replica(s) UP in HAProxy after 0.7 s: ['94b38b941a8d', 'a8d9cffd7a6e']
[exit 0, 1.4 s]
--- run 2; running API replicas BEFORE:
mtusi-lab3-api-1 host=94b38b941a8d running exit=0 pid=28373 started=2026-10-19T15:12:20.518
mtusi-lab3-api-2 host=a8d9cffd7a6e running exit=0 pid=27091 started=2026-10-19T15:11:52.824
$ make test-failover ACTION_SCRIPT=scripts/stop-one.cmd
python tests/resilience.py --mode failover --action-script "scripts/stop-one.cmd" --base-url "http://localhost:8080"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 132, "failed_attempts": 0, "first_recovery_seconds": 5.667}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
[exit 0, 16.3 s]
AFTER (the stopped replica, left stopped by the hook):
mtusi-lab3-api-2 host=a8d9cffd7a6e exited exit=0 pid=0 started=2026-10-19T15:11:52.824
$ C:\Users\wfa22\AppData\Local\Python\pythoncore-3.14-64\python.exe scripts/lbctl.py status
api1 172.23.0.3 UP mtusi-lab3-api-1 X-Instance-Id=94b38b941a8d
(other slots: MAINT, no container)
[exit 0, 0.9 s]
$ sh scripts/restore-api.sh
mtusi-lab3-api-1 Up 32 seconds (healthy)
mtusi-lab3-api-2 Up 6 seconds (healthy)
Container mtusi-lab3-api-1 Running
Container mtusi-lab3-api-2 Starting
Container mtusi-lab3-api-2 Started
Container mtusi-lab3-api-2 Waiting
Container mtusi-lab3-api-1 Waiting
Container mtusi-lab3-api-2 Healthy
Container mtusi-lab3-api-1 Healthy
[exit 0, 7.0 s]
$ C:\Users\wfa22\AppData\Local\Python\pythoncore-3.14-64\python.exe scripts/lbctl.py wait-up 2
2 API replica(s) UP in HAProxy after 0.7 s: ['94b38b941a8d', 'a8d9cffd7a6e']
[exit 0, 1.6 s]
Replicas stopped by the public test across runs: ['mtusi-lab3-api-1', 'mtusi-lab3-api-2']