Add the stop-one, restart and restore hooks and an LB probe

scripts/stop-one.sh maps TARGET_INSTANCE_ID to the api container whose
hostname matches, stops only it with docker stop -t 6 and leaves it
stopped; an unknown id exits 3 and touches nothing. scripts/restart.sh
replaces the lab 2 hook: it stops lb and the replicas, restarts
PostgreSQL and Valkey with their volumes and brings the stand back
with up --wait, which also reruns the idempotent migrate job.
scripts/restore.sh starts stopped replicas without recreating them
and scripts/lb-wait.sh waits until the balancer statistics show every
replica UP again, so the next failover run sees two replicas.

scripts/probe.py check (stdlib, Python 3.9 syntax) counts X-Instance-Id
over 40 calls with one token and races 8 posts with one key through
the balancer. make restore and make lb-check run them, ruff covers the
probe, and the CI acceptance job runs the failover test and the probe.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Artyom
2026-10-28 11:43:52 +03:00
co-authored by Claude Opus 5.5
parent dc2015473f
commit ceb266ed05
9 changed files with 198 additions and 11 deletions
+10 -1
View File
@@ -19,7 +19,7 @@ jobs:
- run: make student-test VENV=.venv
acceptance:
name: Стенд Compose за HAProxy, smoke, рестарт и отказы зависимостей
name: Стенд Compose за HAProxy, smoke, отказ реплики, рестарт и отказы зависимостей
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
@@ -34,6 +34,15 @@ jobs:
- name: Публичный smoke через балансировщик
run: make test
- name: Несколько реплик на одном токене и гонка одного ключа
run: make lb-check
- name: Остановка реплики под запросами, затем возврат
run: |
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
docker compose ps -a api
make restore
- name: Рестарт LB, API и хранилищ с сохранением volumes
run: make test-persistence ACTION_SCRIPT=scripts/restart.sh