forked from mtusi2026/mtusi2026lab3
Второй прогон scripts/collect_evidence.py на новом clone (коммит 2708732):
make check/validate/test, test-persistence (новые PID у lb, api x2,
postgres, valkey), test-failover для обеих реплик по очереди (0 неудачных
попыток), 201 свой тест, серия отказов stop/kill/pause x 2 реплики,
rolling update под нагрузкой (0 ошибок), tests/load.py. Таблица
критерий -> команда -> результат -> файл, раздел уровня 4 с периодами
проверок и временем обнаружения.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
75 lines
3.7 KiB
Plaintext
75 lines
3.7 KiB
Plaintext
# make test-failover with scripts/stop-one: which container is really stopped, restore, repeat
|
|
# 2026-10-19T15:12:01Z, commit 2708732 fix: LB и на [::1] — localhost на Windows стоил 2 с на запрос; сдвиг цели failover
|
|
|
|
--- run 1; running API replicas BEFORE:
|
|
mtusi-lab3-api-1 host=94b38b941a8d running exit=0 pid=27148 started=2026-10-19T15:11:53.132
|
|
mtusi-lab3-api-2 host=a8d9cffd7a6e running exit=0 pid=27091 started=2026-10-19T15:11:52.824
|
|
|
|
$ make test-failover ACTION_SCRIPT=scripts/stop-one.cmd
|
|
python tests/resilience.py --mode failover --action-script "scripts/stop-one.cmd" --base-url "http://localhost:8080"
|
|
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
|
{"observed_instances_before": 2, "successful_writes": 131, "failed_attempts": 0, "first_recovery_seconds": 5.902}
|
|
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
|
[exit 0, 16.4 s]
|
|
|
|
AFTER (the stopped replica, left stopped by the hook):
|
|
mtusi-lab3-api-1 host=94b38b941a8d exited exit=0 pid=0 started=2026-10-19T15:11:53.132
|
|
|
|
$ C:\Users\wfa22\AppData\Local\Python\pythoncore-3.14-64\python.exe scripts/lbctl.py status
|
|
api2 172.23.0.2 UP mtusi-lab3-api-2 X-Instance-Id=a8d9cffd7a6e
|
|
(other slots: MAINT, no container)
|
|
[exit 0, 0.7 s]
|
|
|
|
$ sh scripts/restore-api.sh
|
|
mtusi-lab3-api-1 Up 6 seconds (healthy)
|
|
mtusi-lab3-api-2 Up 33 seconds (healthy)
|
|
Container mtusi-lab3-api-2 Running
|
|
Container mtusi-lab3-api-1 Starting
|
|
Container mtusi-lab3-api-1 Started
|
|
Container mtusi-lab3-api-2 Waiting
|
|
Container mtusi-lab3-api-1 Waiting
|
|
Container mtusi-lab3-api-2 Healthy
|
|
Container mtusi-lab3-api-1 Healthy
|
|
[exit 0, 6.9 s]
|
|
|
|
$ C:\Users\wfa22\AppData\Local\Python\pythoncore-3.14-64\python.exe scripts/lbctl.py wait-up 2
|
|
2 API replica(s) UP in HAProxy after 0.7 s: ['94b38b941a8d', 'a8d9cffd7a6e']
|
|
[exit 0, 1.4 s]
|
|
|
|
--- run 2; running API replicas BEFORE:
|
|
mtusi-lab3-api-1 host=94b38b941a8d running exit=0 pid=28373 started=2026-10-19T15:12:20.518
|
|
mtusi-lab3-api-2 host=a8d9cffd7a6e running exit=0 pid=27091 started=2026-10-19T15:11:52.824
|
|
|
|
$ make test-failover ACTION_SCRIPT=scripts/stop-one.cmd
|
|
python tests/resilience.py --mode failover --action-script "scripts/stop-one.cmd" --base-url "http://localhost:8080"
|
|
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
|
{"observed_instances_before": 2, "successful_writes": 132, "failed_attempts": 0, "first_recovery_seconds": 5.667}
|
|
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
|
[exit 0, 16.3 s]
|
|
|
|
AFTER (the stopped replica, left stopped by the hook):
|
|
mtusi-lab3-api-2 host=a8d9cffd7a6e exited exit=0 pid=0 started=2026-10-19T15:11:52.824
|
|
|
|
$ C:\Users\wfa22\AppData\Local\Python\pythoncore-3.14-64\python.exe scripts/lbctl.py status
|
|
api1 172.23.0.3 UP mtusi-lab3-api-1 X-Instance-Id=94b38b941a8d
|
|
(other slots: MAINT, no container)
|
|
[exit 0, 0.9 s]
|
|
|
|
$ sh scripts/restore-api.sh
|
|
mtusi-lab3-api-1 Up 32 seconds (healthy)
|
|
mtusi-lab3-api-2 Up 6 seconds (healthy)
|
|
Container mtusi-lab3-api-1 Running
|
|
Container mtusi-lab3-api-2 Starting
|
|
Container mtusi-lab3-api-2 Started
|
|
Container mtusi-lab3-api-2 Waiting
|
|
Container mtusi-lab3-api-1 Waiting
|
|
Container mtusi-lab3-api-2 Healthy
|
|
Container mtusi-lab3-api-1 Healthy
|
|
[exit 0, 7.0 s]
|
|
|
|
$ C:\Users\wfa22\AppData\Local\Python\pythoncore-3.14-64\python.exe scripts/lbctl.py wait-up 2
|
|
2 API replica(s) UP in HAProxy after 0.7 s: ['94b38b941a8d', 'a8d9cffd7a6e']
|
|
[exit 0, 1.6 s]
|
|
|
|
Replicas stopped by the public test across runs: ['mtusi-lab3-api-1', 'mtusi-lab3-api-2']
|