Keep the full output of a failed compose up in track 1 runs

The orchestrator printed only the last 500 characters of stderr before
retrying, which hid the cause of a flaky up on eight replicas. The whole
output of each failed attempt now goes to compose-up-failed.log next to
summary.csv.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
This commit is contained in:
Artyom
2026-10-29 23:12:55 +03:00
co-authored by Claude Opus 5.5
parent 0ce7d6ecf0
commit 16b6b28ea9
2 changed files with 8 additions and 4 deletions
@@ -9,7 +9,7 @@
- [ ] 2.1 `Makefile`: `BASE_URL` по порту `lb`, `API_IMAGE` по каталогу, `detection` трижды, `track1-control`
- [x] 2.2 `scripts/evidence-storage.sh` — volumes своего проекта; `scripts/limits.py` — коммит в заголовке
- [ ] 2.3 `scripts/scaling.py` — полный stderr сбоя `compose up` в `compose-up-failed.log`
- [x] 2.3 `scripts/scaling.py` — полный stderr сбоя `compose up` в `compose-up-failed.log`
- [ ] 2.4 `scripts/failover-each.sh` — `docker ps -a` реплик после хука; `scripts/failover-evidence.sh` — `SHIFT`
- [ ] 2.5 Проверка группы: `make lint`, `make check`, `bash -n`/`sh -n` скриптов
+7 -3
View File
@@ -208,14 +208,18 @@ def layout(config, args, variant):
return base, settings
def apply(base, settings, env):
def apply(base, settings, env, out):
env = dict(env, **settings)
for attempt in range(3):
try:
compose("up", "-d", "--wait", "--wait-timeout", "180", env=env)
break
except subprocess.CalledProcessError as error:
print(json.dumps({"compose_up_failed": attempt + 1, "stderr": error.stderr[-500:]}))
log = Path(out) / "compose-up-failed.log"
with log.open("a") as handle:
handle.write(f"== {time.strftime('%FT%T')} {base} attempt {attempt + 1}\n")
handle.write(error.stdout + error.stderr + "\n")
print(json.dumps({"compose_up_failed": attempt + 1, "log": str(log)}))
if attempt == 2:
raise
time.sleep(30)
@@ -338,7 +342,7 @@ def arguments():
def block(args, output, config, variant, repeats):
base, settings = layout(config, args, variant)
current, streak = apply(base, settings, args.base_env), 0
current, streak = apply(base, settings, args.base_env, args.out), 0
for rate in rates(args, base):
results = []
for repeat in repeats: