Keep the full output of a failed compose up in track 1 runs
The orchestrator printed only the last 500 characters of stderr before retrying, which hid the cause of a flaky up on eight replicas. The whole output of each failed attempt now goes to compose-up-failed.log next to summary.csv. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
0ce7d6ecf0
commit
16b6b28ea9
@@ -9,7 +9,7 @@
|
||||
|
||||
- [ ] 2.1 `Makefile`: `BASE_URL` по порту `lb`, `API_IMAGE` по каталогу, `detection` трижды, `track1-control`
|
||||
- [x] 2.2 `scripts/evidence-storage.sh` — volumes своего проекта; `scripts/limits.py` — коммит в заголовке
|
||||
- [ ] 2.3 `scripts/scaling.py` — полный stderr сбоя `compose up` в `compose-up-failed.log`
|
||||
- [x] 2.3 `scripts/scaling.py` — полный stderr сбоя `compose up` в `compose-up-failed.log`
|
||||
- [ ] 2.4 `scripts/failover-each.sh` — `docker ps -a` реплик после хука; `scripts/failover-evidence.sh` — `SHIFT`
|
||||
- [ ] 2.5 Проверка группы: `make lint`, `make check`, `bash -n`/`sh -n` скриптов
|
||||
|
||||
|
||||
+7
-3
@@ -208,14 +208,18 @@ def layout(config, args, variant):
|
||||
return base, settings
|
||||
|
||||
|
||||
def apply(base, settings, env):
|
||||
def apply(base, settings, env, out):
|
||||
env = dict(env, **settings)
|
||||
for attempt in range(3):
|
||||
try:
|
||||
compose("up", "-d", "--wait", "--wait-timeout", "180", env=env)
|
||||
break
|
||||
except subprocess.CalledProcessError as error:
|
||||
print(json.dumps({"compose_up_failed": attempt + 1, "stderr": error.stderr[-500:]}))
|
||||
log = Path(out) / "compose-up-failed.log"
|
||||
with log.open("a") as handle:
|
||||
handle.write(f"== {time.strftime('%FT%T')} {base} attempt {attempt + 1}\n")
|
||||
handle.write(error.stdout + error.stderr + "\n")
|
||||
print(json.dumps({"compose_up_failed": attempt + 1, "log": str(log)}))
|
||||
if attempt == 2:
|
||||
raise
|
||||
time.sleep(30)
|
||||
@@ -338,7 +342,7 @@ def arguments():
|
||||
|
||||
def block(args, output, config, variant, repeats):
|
||||
base, settings = layout(config, args, variant)
|
||||
current, streak = apply(base, settings, args.base_env), 0
|
||||
current, streak = apply(base, settings, args.base_env, args.out), 0
|
||||
for rate in rates(args, base):
|
||||
results = []
|
||||
for repeat in repeats:
|
||||
|
||||
Reference in New Issue
Block a user