Добавить автоматический эксперимент и изоляцию повторов

This commit is contained in:
TAskMAster339
2026-11-05 23:46:33 +03:00
parent c8ff31e5ff
commit 228b510ff3
10 changed files with 2069 additions and 3 deletions
+6 -1
View File
@@ -28,5 +28,10 @@ coverage/
.DS_Store
.idea/
.vscode/
# Keep small, sanitized evidence summaries in evidence/ under version control.
# Keep sanitized experiment evidence under version control.
# Inspect git diff --cached: ignore rules are not a secret scanner.
# Experiment evidence: keep public setup logs, exclude credentials and locks.
!evidence/**/*.log
evidence/**/.private/
evidence/**/.run.lock
+2 -2
View File
@@ -172,9 +172,9 @@ services:
- ./secrets/broker_user:/run/secrets/broker_user:ro
- ./secrets/broker_password:/run/secrets/broker_password:ro
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
test: ["CMD-SHELL", "rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_port_connectivity"]
interval: 5s
timeout: 5s
timeout: 10s
start_period: 20s
retries: 30
worker:
+43
View File
@@ -0,0 +1,43 @@
# Проверки инструментов и реализации
## Инструменты эксперимента
```sh
python3 -m unittest discover -s experiments/tests -v
```
Тесты проверяют воспроизводимость PNG и корректность их структуры, изоляцию
ресурсов Compose, размещение результатов в `evidence/idempotency`, правила
агрегации неполных и дублированных случаев, учёт OOM и исключение приватных
учётных данных из архива.
Проверка конфигурации созданного повтора:
```sh
docker compose -f evidence/idempotency/ИМЯ_СЕРИИ/.private/r1/compose.json config --quiet
```
В конфигурации должны отсутствовать общие с учебным стендом имена сетей и томов,
а пути файлов учётных данных должны вести в приватный каталог соответствующего повтора.
## Приложение
```sh
make check
docker compose --project-name highload-lab5 --profile test run --rm --build backend-tests
```
Backend-тесты включают проверку устаревшего lease token, публикации v2, запрета
понижения версии и повторного открытия старого задания. Тесты обработки изображений
проверяют геометрию, целостность исходника и различие результатов v1/v2.
Сборка и статическая проверка в WSL:
```sh
cd src
go build -o /tmp/highload-lab5-messenger ./backend
go vet ./backend/...
```
Сквозная проверка управляющего сценария выполняется пилотом. При дефекте инструмента
сохраняются артефакты пилота; после исправления создаётся новый снимок исходников.
+136
View File
@@ -0,0 +1,136 @@
# Запуск
## Окружение
Требования: WSL Ubuntu, Linux Docker Engine с интеграцией Docker Desktop,
Docker Compose с поддержкой `config --format json`, Python 3.9+ и Git.
При первой сборке нужен доступ к реестрам образов и пакетов.
```sh
python3 -m unittest discover -s experiments/tests -v
```
Порты 18085 и 18900 должны быть свободны. Для стенда рекомендуется выделить
около 6 GiB памяти Docker и несколько GiB диска. Первая сборка и основная серия
могут занять более часа. Параллельные сборки, нагрузочные тесты и другие серии
меняют условия выполнения и должны быть остановлены на время измерений.
До прогрева runner ожидает готовности приложения RabbitMQ и доступности списка
очередей. На запуск стенда и эту проверку отводится общий интервал 300 секунд.
Попытки записываются в `broker-startup.jsonl`, успешная проверка — в
`broker-readiness-r*.json`. Во время сценариев ошибка чтения очереди завершает
проверку и сохраняется в журнале команд.
Эксперимент использует отдельный Compose project. Учебный `highload-lab5` не
перезапускается. При нехватке ресурсов его можно остановить отдельно, сохранив тома.
## Снимок исходников
```sh
experiment_seal=$(python3 experiments/run.py seal)
printf '%s\n' "$experiment_seal"
```
Команда создаёт каталог `evidence/idempotency/seals/...` с `manifest.json` и
`source.zip`. В manifest записаны SHA-256 файлов и протокола, Git HEAD и признак
незакоммиченных изменений. Контейнеры при этом не запускаются.
Значение `experiment_seal` используется в следующих командах. В новом терминале
оно задаётся сохранённым путём:
```sh
experiment_seal=evidence/idempotency/seals/ИМЯ_СНИМКА
```
Runner проверяет контрольные суммы перед серией и каждым повтором. После изменения
исходников требуется новый снимок; после изменения методики — также новая версия
протокола. Существующие данные серий сохраняются.
## Пилот
```sh
python3 experiments/run.py run --phase pilot --seal "$experiment_seal"
```
Пилот выполняет девять сценариев для `rgb_small` в одном повторе. В конце выводится
путь публичного архива. Условия перехода к основной серии в `summary.json`:
```text
complete: true
completed: 9
violations: 0
invalid: 0
missing_artifacts: []
outcome: pilot_only_not_confirmation
```
При ошибке пилота основная серия откладывается до диагностики. Данные пилота
хранятся отдельно и не включаются в основную выборку.
## Основная серия
```sh
python3 experiments/run.py run --phase main --seal "$experiment_seal"
```
Runner выполняет 135 случаев. Каждый из пяти повторов получает новые тома,
сети и учётные данные. Порядок случаев и seed записываются в результаты.
После каждого повтора `docker compose down` удаляет его контейнеры и сети.
Именованные тома с данными сохраняются; параметр `--volumes` не используется.
Изменение кода во время серии или повторный запуск только неудачных случаев
нарушают сопоставимость данных. Исправленная реализация проверяется отдельной
серией с новым снимком исходников.
## Результаты
Все файлы размещаются в `evidence/idempotency/`:
| Путь внутри серии | Содержимое |
| --- | --- |
| `docs/`, `protocol.json`, `seal/` | Методика, параметры и исходники |
| `fixtures/` | Исходные PNG и контрольные суммы |
| `cases.jsonl`, `cases/` | Исходы случаев, наблюдения API, состояния БД, аудит и варианты изображений |
| `environment-r*.json`, `order-r*.json` | Образы, ресурсы и порядок выполнения |
| `docker-events-r*.jsonl` | События контейнеров, включая OOM и завершения |
| `command-errors.jsonl` | Ошибки команд с маскированием учётных данных |
| `summary.json` | Сводка по сценариям и профилям |
| `.private/` | Учётные данные и рабочие конфигурации стенда |
Архив для анализа создаётся рядом с каталогом серии:
`evidence/idempotency/main-...-public.zip` или `pilot-...-public.zip`.
В него не включается `.private`. Для передачи результатов используется этот архив;
полный каталог серии содержит пароли и строки подключения.
Публичные результаты и журналы запуска доступны Git. Корневой `.gitignore`
исключает только `.private/` и служебный `.run.lock` внутри `evidence/`.
Данные `evidence/` не включаются в новые снимки исходников.
Повторная агрегация без запуска стенда:
```sh
python3 experiments/run.py analyze evidence/idempotency/main-ИМЯ_СЕРИИ
```
Создание архива после прерванного запуска:
```sh
python3 experiments/run.py pack evidence/idempotency/main-ИМЯ_СЕРИИ
```
`pack` не перезаписывает существующий архив. Значения итогового поля `outcome`
описаны в [HYPOTHESIS.md](HYPOTHESIS.md).
## Прерывание и остановка
При Ctrl+C сохраняются собранные данные и признак прерванной серии.
После kill -9 или сбоя host автоматическая остановка может не выполниться.
Имена созданных project находятся в `manifest.json`. Остановка выбранного повтора:
```sh
docker compose --project-name ИМЯ_ИЗ_MANIFEST \
-f evidence/idempotency/ИМЯ_СЕРИИ/.private/r1/compose.json down
```
Тома сохраняются до завершения анализа. Глобальные команды `docker stop` и
`docker prune` для управления экспериментом не используются.
+524
View File
@@ -0,0 +1,524 @@
#!/usr/bin/env python3
"""Seal first; run pilot/main only on an explicit user command."""
import argparse
import datetime as dt
import json
import os
import platform
import random
import shutil
import socket
import subprocess
import sys
import time
import uuid
import zipfile
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent / "src"))
from support import (
PROTOCOL,
ROOT,
archive_public,
build_images,
digest,
dump,
ensure_local_ignore,
isolated_config,
png,
private_environment,
source_manifest,
verify_seal,
)
RESULTS = ROOT / "evidence/idempotency"
def stamp():
return (
dt.datetime.now(dt.timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+ "-"
+ uuid.uuid4().hex[:8]
)
def seal():
directory = RESULTS / "seals" / stamp()
directory.mkdir(parents=True, exist_ok=False)
files = source_manifest()
with zipfile.ZipFile(
directory / "source.zip", "x", zipfile.ZIP_DEFLATED
) as archive:
for name in files:
archive.write(ROOT / name, name)
value = {
"registered_utc": dt.datetime.now(dt.timezone.utc).isoformat(),
"protocol_sha256": digest(PROTOCOL.read_bytes()),
"files": files,
"git_head": subprocess.check_output(
["git", "rev-parse", "HEAD"], cwd=ROOT, text=True
).strip(),
"git_dirty": bool(
subprocess.check_output(["git", "status", "--porcelain"], cwd=ROOT)
),
}
dump(directory / "manifest.json", value)
print(directory.relative_to(ROOT))
def analyze(run):
protocol = json.loads((run / "protocol.json").read_text())
manifest = json.loads((run / "manifest.json").read_text())
records = (
[json.loads(line) for line in (run / "cases.jsonl").read_text().splitlines()]
if (run / "cases.jsonl").exists()
else []
)
violations = sum(row["status"] == "violation" for row in records)
invalid = sum(row["status"] == "invalid" for row in records)
missing = []
for row in records:
for filename in (
"events.jsonl",
"result.json",
"containers-before.json",
"containers-after.json",
"database-final.json",
"fault-markers.json",
):
if not (run / row["directory"] / filename).is_file():
missing.append(row["directory"] + "/" + filename)
expected = (
protocol["pass"]["cases"]
if manifest["phase"] == "main"
else len(protocol["scenarios"])
)
repetitions = (
range(1, protocol["repetitions"] + 1) if manifest["phase"] == "main" else [1]
)
profiles = (
protocol["profiles"]
if manifest["phase"] == "main"
else protocol["profiles"][:1]
)
planned = {
(r, p["name"], s)
for r in repetitions
for p in profiles
for s in protocol["scenarios"]
}
for repetition in repetitions:
for name in (
"docker-events-r" + str(repetition) + ".jsonl",
"environment-r" + str(repetition) + ".json",
"order-r" + str(repetition) + ".json",
):
if not (run / name).is_file():
missing.append(name)
keys = [(row["repetition"], row["profile"], row["scenario"]) for row in records]
invalid_matrix = len(keys) != len(set(keys)) or set(keys) != planned
oom_events = 0
for path in run.glob("docker-events-r*.jsonl"):
for line in path.read_text().splitlines():
if json.loads(line).get("action") == "oom":
oom_events += 1
for filename in ("seal/manifest.json", "seal/source.zip", "fixtures/manifest.json"):
if not (run / filename).exists():
missing.append(filename)
if manifest["phase"] != "main":
outcome = "pilot_only_not_confirmation"
elif violations or oom_events:
outcome = "refuted"
elif (
len(records) != expected
or invalid_matrix
or invalid
or missing
or manifest.get("interrupted")
or not manifest.get("finished_utc")
):
outcome = "inconclusive"
else:
outcome = "supported_in_tested_conditions"
summary = {
"phase": manifest["phase"],
"expected": expected,
"completed": len(records),
"violations": violations,
"complete": len(records) == expected
and not invalid_matrix
and not invalid
and not missing
and not oom_events
and not manifest.get("interrupted")
and bool(manifest.get("finished_utc")),
"oom_events": oom_events,
"invalid": invalid,
"invalid_matrix": invalid_matrix,
"missing_artifacts": missing,
"outcome": outcome,
"groups": {},
}
for row in records:
key = row["scenario"] + "/" + row["profile"]
group = summary["groups"].setdefault(
key, {"cases": 0, "violations": 0, "invalid": 0, "elapsed_seconds": []}
)
group["cases"] += 1
group["violations"] += row["status"] == "violation"
group["invalid"] += row["status"] == "invalid"
if row.get("elapsed_seconds") is not None:
group["elapsed_seconds"].append(row["elapsed_seconds"])
import statistics
for group in summary["groups"].values():
times = group["elapsed_seconds"]
group["diagnostic_time"] = (
None
if not times
else {
"min": min(times),
"median": statistics.median(times),
"max": max(times),
}
)
dump(run / "summary.json", summary)
return summary
def ports_available(protocol):
for port in protocol["ports"].values():
with socket.socket() as probe:
probe.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
try:
probe.bind(("127.0.0.1", port))
except OSError:
raise RuntimeError(
"Experiment port "
+ str(port)
+ " is occupied; do not stop unrelated services."
) from None
def run_series(seal_directory, phase):
if sys.platform != "linux":
raise RuntimeError("Run this command in WSL Ubuntu.")
import fcntl
from driver import Case, Failure, Stack, Violation
from events import LifecycleEvents
RESULTS.mkdir(parents=True, exist_ok=True)
lock = (RESULTS / ".run.lock").open("a")
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
frozen = verify_seal(seal_directory)
protocol = json.loads(PROTOCOL.read_text())
ports_available(protocol)
run = RESULTS / (phase + "-" + stamp())
run.mkdir(exist_ok=False)
shutil.copytree(seal_directory, run / "seal")
shutil.copytree(ROOT / "experiments/docs", run / "docs")
shutil.copy2(PROTOCOL, run / "protocol.json")
manifest = {
"phase": phase,
"started_utc": dt.datetime.now(dt.timezone.utc).isoformat(),
"protocol_sha256": frozen["protocol_sha256"],
"python": sys.version,
"platform": platform.platform(),
"git_head": frozen["git_head"],
"source_dirty": frozen["git_dirty"],
"docker": subprocess.check_output(
["docker", "version", "--format", "{{.Server.Version}}"], text=True
).strip(),
"compose": subprocess.check_output(
["docker", "compose", "version", "--short"], text=True
).strip(),
"projects": [],
"interrupted": False,
}
dump(run / "manifest.json", manifest)
print("Results:", run, flush=True)
app_image, s3_image = build_images(run)
manifest["application_image"] = app_image
manifest["s3_image"] = s3_image
fixtures = run / "fixtures"
fixtures.mkdir()
fixture_data = {}
fixture_manifest = {}
for profile in protocol["profiles"]:
data = png(profile, protocol["seed"])
fixture_data[profile["name"]] = data
(fixtures / (profile["name"] + ".png")).write_bytes(data)
fixture_manifest[profile["name"]] = {
"sha256": digest(data),
"bytes": len(data),
**profile,
}
dump(fixtures / "manifest.json", fixture_manifest)
profiles = protocol["profiles"] if phase == "main" else protocol["profiles"][:1]
repeats = protocol["repetitions"] if phase == "main" else 1
baseline_images = None
records = (run / "cases.jsonl").open("x")
try:
for repetition in range(1, repeats + 1):
verify_seal(seal_directory)
ports_available(protocol)
project = "lab5-idem-" + run.name.lower() + "-r" + str(repetition)
private = run / ".private" / ("r" + str(repetition))
private.parent.mkdir(exist_ok=True)
os.chmod(private.parent, 0o700)
values = private_environment(private, protocol)
env = {**os.environ, **values}
base = json.loads(
subprocess.check_output(
[
"docker",
"compose",
"--env-file",
str(private / "private.env"),
"config",
"--format",
"json",
],
cwd=ROOT,
env=env,
text=True,
)
)
configuration = isolated_config(
base, project, private, protocol, app_image, s3_image
)
dump(private / "compose.json", configuration)
os.chmod(private / "compose.json", 0o600)
stack = Stack(project, private / "compose.json", private, protocol)
manifest["projects"].append(project)
dump(run / "manifest.json", manifest)
print("Starting isolated repeat", repetition, "of", repeats, flush=True)
collector = LifecycleEvents(
project, run / ("docker-events-r" + str(repetition) + ".jsonl")
)
try:
setup_deadline = time.monotonic() + 300
with (run / ("boot-r" + str(repetition) + ".log")).open("w") as output:
subprocess.run(
stack.prefix + ["up", "-d", "--wait"],
cwd=ROOT,
stdout=output,
stderr=subprocess.STDOUT,
check=True,
timeout=300,
)
stack.wait_broker_ready(timeout=max(0, setup_deadline-time.monotonic()))
initial = stack.containers()
dump(run / ("environment-r" + str(repetition) + ".json"), initial)
images = {row["service"]: row["image"] for row in initial}
if baseline_images is None:
baseline_images = images
elif images != baseline_images:
raise RuntimeError("Image IDs changed between repetitions")
warmup = Case(
stack,
protocol["profiles"][0],
"control",
fixture_data[protocol["profiles"][0]["name"]],
run / "warmup" / ("r" + str(repetition)),
)
try:
dump(warmup.directory / "result.json", warmup.execute())
except Exception as exc:
dump(warmup.directory / "result.json", {
"status": "invalid", "error_type": type(exc).__name__,
"error": str(exc), "attachment_id": warmup.attachment,
})
if warmup.attachment:
try:
dump(warmup.directory / "database-final.json", stack.snapshot(warmup.attachment))
except Exception:
pass
raise
finally:
warmup.close()
order = [
(profile, scenario)
for profile in profiles
for scenario in protocol["scenarios"]
]
random.Random(protocol["seed"] + repetition).shuffle(order)
dump(
run / ("order-r" + str(repetition) + ".json"),
[{"profile": p["name"], "scenario": s} for p, s in order],
)
for index, (profile, scenario) in enumerate(order, 1):
relative = (
"cases/r"
+ str(repetition)
+ "/"
+ str(index).zfill(2)
+ "-"
+ profile["name"]
+ "-"
+ scenario
)
case = Case(
stack,
profile,
scenario,
fixture_data[profile["name"]],
run / relative,
)
row = {
"repetition": repetition,
"profile": profile["name"],
"scenario": scenario,
"directory": relative,
"status": "pass",
}
recovered = True
try:
if (
stack.sql(
"SELECT count(*) FROM attachments WHERE status IN ('queued','processing')"
)
!= "0"
or stack.queue_depth() != 0
):
raise RuntimeError("Non-quiescent initial state")
dump(
case.directory / "containers-before.json",
stack.containers(),
)
row.update(case.execute())
except KeyboardInterrupt:
row.update(
status="invalid",
error_type="KeyboardInterrupt",
error="Interrupted by user",
)
case.event("case_interrupted")
raise
except Exception as exc:
row.update(
status="violation"
if isinstance(exc, Violation)
or isinstance(exc, Failure)
and case.attachment is not None
else "invalid",
error_type=type(exc).__name__,
error=str(exc),
)
case.event(
"case_error", error_type=type(exc).__name__, error=str(exc)
)
try:
case.recover()
except Exception as recovery_error:
row["recovery_error_type"] = type(recovery_error).__name__
recovered = False
finally:
case.close()
try:
dump(
case.directory / "containers-after.json",
stack.containers(),
)
except Exception:
row["container_snapshot_missing"] = True
if case.attachment:
try:
dump(
case.directory / "database-final.json",
stack.snapshot(case.attachment),
)
except Exception:
row["database_snapshot_missing"] = True
markers = [
{"name": p.name, "mtime": p.stat().st_mtime}
for p in (private / "faults").glob(
"*" + case.attachment + "*"
)
]
dump(case.directory / "fault-markers.json", markers)
row.setdefault("elapsed_seconds", time.monotonic() - case.start)
dump(case.directory / "result.json", row)
records.write(json.dumps(row, ensure_ascii=False) + "\n")
records.flush()
if collector.failed:
raise RuntimeError(
"Docker lifecycle event stream ended unexpectedly"
)
print(
"Repeat",
repetition,
"case",
index,
profile["name"],
scenario,
row["status"],
flush=True,
)
if not recovered:
raise RuntimeError(
"Cannot restore quiescence; remaining cases not replaced"
)
finally:
# Remove this project's ephemeral networks before the next repeat.
# Named volumes retain the experiment data (no --volumes).
try:
subprocess.run(
stack.prefix + ["down", "--timeout", "30"],
cwd=ROOT,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
timeout=120,
check=True,
)
finally:
collector.close()
if collector.failed:
manifest["interrupted"] = True
except BaseException as exc:
manifest.update(interrupted=True, interruption_type=type(exc).__name__)
dump(
run / "interruption.json", {"type": type(exc).__name__, "message": str(exc)}
)
raise
finally:
records.close()
manifest["finished_utc"] = dt.datetime.now(dt.timezone.utc).isoformat()
dump(run / "manifest.json", manifest)
summary = analyze(run)
print("Outcome:", summary["outcome"], flush=True)
print("Return archive:", archive_public(run), flush=True)
fcntl.flock(lock, fcntl.LOCK_UN)
lock.close()
def main():
ensure_local_ignore()
parser = argparse.ArgumentParser(description=__doc__)
commands = parser.add_subparsers(dest="command", required=True)
commands.add_parser(
"seal", help="freeze source/protocol; no containers are started"
)
run = commands.add_parser("run")
run.add_argument("--phase", choices=("pilot", "main"), required=True)
run.add_argument("--seal", type=Path, required=True)
for name in ("analyze", "pack"):
command = commands.add_parser(name)
command.add_argument("directory", type=Path)
args = parser.parse_args()
if args.command == "seal":
seal()
elif args.command == "run":
run_series(args.seal.resolve(), args.phase)
elif args.command == "analyze":
print(
json.dumps(analyze(args.directory.resolve()), ensure_ascii=False, indent=2)
)
elif args.command == "pack":
print(archive_public(args.directory.resolve()))
if __name__ == "__main__":
main()
+693
View File
@@ -0,0 +1,693 @@
"""Controlled delivery/fault cases. Called only by an explicit experiment run."""
import datetime as dt
import hashlib
import hmac
import json
import subprocess
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET
from pathlib import Path
from support import ROOT, digest, dump
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, Failure
from image_probe import jpeg_size
class Violation(AssertionError):
pass
class HarnessError(RuntimeError):
pass
def require(condition, message):
if not condition:
raise Violation(message)
class Stack:
def __init__(self, project, config, private, protocol):
self.project, self.config, self.private, self.protocol = (
project,
config,
private,
protocol,
)
self.prefix = [
"docker",
"compose",
"--project-name",
project,
"-f",
str(config),
]
def call(self, *args):
result = subprocess.run(
self.prefix + list(args),
cwd=ROOT,
text=True,
capture_output=True,
timeout=60,
)
if result.returncode:
error = result.stderr
for path in (self.private / "secrets").iterdir():
value = path.read_text().strip()
if len(value) >= 12:
error = error.replace(value, "[REDACTED]")
with (self.private.parent.parent / "command-errors.jsonl").open(
"a"
) as output:
output.write(
json.dumps(
{
"project": self.project,
"args": args,
"exit_code": result.returncode,
"stderr": error,
}
)
+ "\n"
)
raise HarnessError("Compose command failed; see command-errors.jsonl")
return result.stdout.strip()
def wait_broker_ready(self, timeout=60):
"""Wait only during setup; runtime queue failures remain fatal."""
deadline = time.monotonic() + timeout
args = ["exec", "-T", "broker", "rabbitmqctl", "-q", "list_queues",
"name", "messages_ready", "messages_unacknowledged",
"--formatter", "json", "--timeout", "5"]
attempts = 0
while time.monotonic() < deadline:
attempts += 1
try:
result = subprocess.run(
self.prefix + args, cwd=ROOT, text=True, capture_output=True,
timeout=min(15, max(0.1, deadline-time.monotonic())),
)
if result.returncode == 0:
rows = json.loads(result.stdout)
if not isinstance(rows, list):
raise HarnessError("Unexpected RabbitMQ queue response")
dump(self.private.parent.parent / ("broker-readiness-"+self.private.name+".json"), {
"project": self.project, "attempts": attempts,
"ready": True,
})
return
error = result.stderr
# Auth, command syntax and stopped-container errors are not startup races.
if "requires the 'rabbit' app to be running" not in error:
raise HarnessError("RabbitMQ readiness probe failed: " + error.strip())
except subprocess.TimeoutExpired:
error = "RabbitMQ startup probe timed out"
with (self.private.parent.parent / "broker-startup.jsonl").open("a") as output:
output.write(json.dumps({"project": self.project, "attempt": attempts,
"stderr": error}) + "\n")
time.sleep(min(1, max(0, deadline-time.monotonic())))
raise HarnessError("RabbitMQ did not finish startup before the setup deadline")
def sql(self, query):
return self.call(
"exec",
"-T",
"db",
"sh",
"-c",
'exec psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atc "$1"',
"sh",
query,
)
def workers(self):
return self.call("ps", "-aq", "worker").splitlines()
def worker_action(self, action, ids):
for ident in ids:
owner = subprocess.check_output(
[
"docker",
"inspect",
"--format",
'{{index .Config.Labels "com.docker.compose.project"}}',
ident,
],
text=True,
).strip()
if owner != self.project:
raise HarnessError("Container belongs to another project")
if ids:
subprocess.run(
[
"docker",
action,
*(["--time", "5"] if action == "stop" else []),
*ids,
],
check=True,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
def queue_depth(self):
rows = json.loads(
self.call(
"exec",
"-T",
"broker",
"rabbitmqctl",
"-q",
"list_queues",
"name",
"messages_ready",
"messages_unacknowledged",
"--formatter",
"json",
)
)
return sum(
row["messages_ready"] + row["messages_unacknowledged"]
for row in rows
if row["name"] == "attachments.v1"
)
def operator(self, *args):
running = self.call("ps", "--status", "running", "-q", "dispatcher")
mode = (
["exec", "-T", "dispatcher"]
if running
else ["run", "--rm", "--no-deps", "-T", "dispatcher"]
)
return self.call(*mode, "./messenger", *args)
def admin(self, *args):
return json.loads(self.operator("jobs", *args))
def snapshot(self, attachment):
result = self.sql(
"SELECT json_build_object('desired',a.pipeline_version,'published',a.published_pipeline_version,'status',a.status,'jobs',(SELECT json_agg(json_build_object('job_id',j.id,'version',j.pipeline_version,'attempts',j.attempts,'superseded',j.superseded_at,'lease_until',j.lease_until) ORDER BY j.pipeline_version) FROM attachment_jobs j WHERE j.attachment_id=a.id),'attempts_audit',(SELECT COALESCE(json_agg(json_build_object('job_id',t.job_id,'generation',t.generation,'attempt',t.attempt,'started_at',t.started_at,'finished_at',t.finished_at,'error_code',t.error_code,'next_attempt_at',t.next_attempt_at) ORDER BY t.started_at),'[]') FROM attachment_job_attempts t JOIN attachment_jobs x ON x.id=t.job_id WHERE x.attachment_id=a.id),'publications',(SELECT COALESCE(json_agg(json_build_object('id',p.id,'version',p.pipeline_version,'variants',p.variants,'published_at',p.published_at) ORDER BY p.id),'[]') FROM attachment_publications p WHERE p.attachment_id=a.id)) FROM attachments a WHERE a.id='"
+ attachment
+ "'"
)
return json.loads(result)
def messages(self, attachment):
return json.loads(
self.sql(
"SELECT json_agg(json_build_object('schema_version',1,'job_id',id,'type','attachment.process.v1','attachment_id',attachment_id,'pipeline_version',pipeline_version,'attempt',1,'created_at',to_char(created_at AT TIME ZONE 'UTC','YYYY-MM-DD\"T\"HH24:MI:SS.US\"Z\"')) ORDER BY pipeline_version) FROM attachment_jobs WHERE attachment_id='"
+ attachment
+ "'"
)
)
def publish(self, job):
self.operator("broker-publish", json.dumps(job))
def containers(self):
ids = self.call("ps", "-aq").splitlines()
if not ids:
return []
# Never save Config.Env: it contains the experiment DB password.
rows = json.loads(
subprocess.check_output(["docker", "inspect", *ids], text=True)
)
return [
{
"id": row["Id"],
"image": row["Image"],
"service": row["Config"]["Labels"].get("com.docker.compose.service"),
"state": {
k: row["State"].get(k)
for k in (
"Status",
"ExitCode",
"OOMKilled",
"StartedAt",
"FinishedAt",
)
},
"restarts": row["RestartCount"],
"memory": row["HostConfig"]["Memory"],
"nano_cpus": row["HostConfig"]["NanoCpus"],
}
for row in rows
]
def s3(self, method, key="", query=None):
secret_dir = self.private / "secrets"
access = (secret_dir / "s3_api_access_key_id").read_text().strip()
secret = (secret_dir / "s3_api_secret_access_key").read_text().strip()
host = "127.0.0.1:" + str(self.protocol["ports"]["s3"])
uri = "/messenger-attachments/" + key
query_string = urllib.parse.urlencode(
sorted((query or {}).items()), quote_via=urllib.parse.quote
)
now = dt.datetime.now(dt.timezone.utc)
timestamp, day = now.strftime("%Y%m%dT%H%M%SZ"), now.strftime("%Y%m%d")
payload = digest(b"")
signed = "host;x-amz-content-sha256;x-amz-date"
headers = (
"host:"
+ host
+ "\nx-amz-content-sha256:"
+ payload
+ "\nx-amz-date:"
+ timestamp
+ "\n"
)
canonical = "\n".join([method, uri, query_string, headers, signed, payload])
scope = day + "/us-east-1/s3/aws4_request"
text = (
"AWS4-HMAC-SHA256\n"
+ timestamp
+ "\n"
+ scope
+ "\n"
+ digest(canonical.encode())
)
signing = ("AWS4" + secret).encode()
for part in (day, "us-east-1", "s3", "aws4_request"):
signing = hmac.new(signing, part.encode(), hashlib.sha256).digest()
signature = hmac.new(signing, text.encode(), hashlib.sha256).hexdigest()
request = urllib.request.Request(
"http://" + host + uri + ("?" + query_string if query_string else ""),
method=method,
headers={
"x-amz-date": timestamp,
"x-amz-content-sha256": payload,
"Authorization": "AWS4-HMAC-SHA256 Credential="
+ access
+ "/"
+ scope
+ ", SignedHeaders="
+ signed
+ ", Signature="
+ signature,
},
)
try:
response = urllib.request.urlopen(request, timeout=15)
except urllib.error.HTTPError as exc:
response = exc
with response:
code, body = response.code, response.read()
if code in (401, 403):
raise HarnessError("S3 measurement identity could not authenticate")
return code, body
def webp_size(data):
require(data[:4] == b"RIFF" and data[8:12] == b"WEBP", "invalid WebP container")
offset = 12
dimensions = None
while offset + 8 <= len(data):
kind = data[offset : offset + 4]
length = int.from_bytes(data[offset + 4 : offset + 8], "little")
body = data[offset + 8 : offset + 8 + length]
require(kind not in (b"EXIF", b"XMP ", b"ICCP"), "WebP metadata leaked")
if kind == b"VP8 ":
dimensions = (
int.from_bytes(body[6:8], "little") & 16383,
int.from_bytes(body[8:10], "little") & 16383,
)
if kind == b"VP8L":
bits = int.from_bytes(body[1:5], "little")
dimensions = ((bits & 16383) + 1, ((bits >> 14) & 16383) + 1)
if kind == b"VP8X":
dimensions = (
int.from_bytes(body[4:7], "little") + 1,
int.from_bytes(body[7:10], "little") + 1,
)
offset += 8 + length + (length % 2)
require(dimensions is not None, "WebP dimensions absent")
return dimensions
class Case:
def __init__(self, stack, profile, scenario, data, directory):
self.s, self.profile, self.scenario, self.data, self.directory = (
stack,
profile,
scenario,
data,
directory,
)
directory.mkdir(parents=True)
self.events = (directory / "events.jsonl").open("x")
self.start = time.monotonic()
self.last = None
self.gaps = []
self.attachment = None
self.verifications = 0
self.c = Client("http://127.0.0.1:" + str(stack.protocol["ports"]["api"]))
def event(self, kind, **values):
now = time.monotonic()
record = {
"utc": dt.datetime.now(dt.timezone.utc).isoformat(),
"elapsed": now - self.start,
"kind": kind,
**values,
}
self.events.write(json.dumps(record, ensure_ascii=False) + "\n")
self.events.flush()
def wait(self, fn, timeout=None, invariant=True):
deadline = time.monotonic() + (
timeout or self.s.protocol["case_timeout_seconds"]
)
while time.monotonic() < deadline:
value = fn()
if value and time.monotonic() <= deadline:
return value
time.sleep(self.s.protocol["poll_interval_seconds"])
if invariant:
raise Violation("completion/quiescence deadline exceeded")
raise HarnessError("required fault/precondition did not occur before deadline")
def observe(self):
now = time.monotonic()
if self.last is not None and now - self.last > 2:
self.gaps.append(now - self.last)
self.last = now
value = self.c.call(
"GET",
"/api/v1/attachments/" + self.attachment,
actor=self.alice,
schema="Attachment",
)["json"]
snapshot = self.s.snapshot(self.attachment)
self.event("observation", api=value, database=snapshot)
return value, snapshot
def ready(self, version):
def check():
value, snapshot = self.observe()
require(value["status"] != "failed", "valid fixture reached failed")
return (
(value, snapshot)
if value["status"] == "ready" and snapshot["published"] == version
else None
)
return self.wait(check)
def arm(self, point):
marker = self.s.private / "faults" / (point + "." + self.attachment)
marker.write_text("experiment " + self.scenario + "\n")
self.event("arm", point=point)
return Path(str(marker) + ".hit")
def upload(self):
self.alice, self.bob, self.eve, chat = self.c.bootstrap()
value = self.c.call(
"POST",
"/api/v1/chats/" + chat["id"] + "/attachments?filename=fixture.png",
202,
actor=self.alice,
raw=self.data,
headers={"Content-Type": "image/png"},
schema="Attachment",
)["json"]
self.attachment = value["id"]
require(
value["sha256"] == digest(self.data), "accepted original checksum mismatch"
)
self.event("accepted_202", attachment_id=self.attachment)
require(
len(self.s.messages(self.attachment)) == 1,
"accepted upload has no unique durable job",
)
def upgrade(self):
require(
self.s.admin(
"upgrade", self.attachment, "2", "Protocol v1 " + self.scenario
)["changed"],
"upgrade intent was not persisted",
)
self.event("upgrade_accepted", version=2)
def publish(self, job, count):
for _ in range(count):
self.s.publish(job)
self.event(
"explicit_delivery",
job_id=job["job_id"],
version=job["pipeline_version"],
count=count,
)
def verify(self, version):
value, snapshot = self.ready(version)
self.verifications += 1
observation = "verify-" + str(self.verifications).zfill(3) + "-v" + str(version)
objects = self.directory / "objects" / observation
objects.mkdir(parents=True)
require(
snapshot["desired"] == version, "desired version differs from publication"
)
original = self.c.call(
"GET", "/api/v1/attachments/" + self.attachment + "/content", actor=self.bob
)["bytes"]
require(original == self.data, "original changed")
require(
len(value["variants"]) == 3
and {v["name"] for v in value["variants"]}
== {"thumbnail", "crop", "optimized"},
"incomplete or duplicate variant set",
)
manifest = {}
for item in value["variants"]:
name = item["name"]
key = (
"variants/"
+ self.attachment
+ "/v"
+ str(version)
+ "/"
+ name
+ (".webp" if name == "optimized" else ".jpg")
)
response = self.c.call(
"GET",
"/api/v1/attachments/" + self.attachment + "/content?variant=" + name,
actor=self.bob,
)
require(
digest(response["bytes"]) == item["sha256"]
and len(response["bytes"]) == item["size_bytes"],
"variant checksum mismatch",
)
require(
response["headers"].get_content_type() == item["content_type"],
"variant MIME mismatch",
)
try:
dimensions = (
webp_size(response["bytes"])
if name == "optimized"
else jpeg_size(response["bytes"])
)
except Failure as exc:
raise Violation("actual variant decode failed") from exc
w, h = self.profile["width"], self.profile["height"]
limit = 1280 if name == "optimized" else 256
expected = (
(128, 128)
if name == "crop"
else (min(w, limit), max(1, h * limit // w))
if w >= h and w > limit
else (max(1, w * limit // h), limit)
if h > w and h > limit
else (w, h)
)
require(
dimensions == expected == (item["width"], item["height"]),
"actual variant dimensions mismatch",
)
code, body = self.s.s3("GET", key)
require(
code == 200 and body == response["bytes"],
"API exposes a different version than S3",
)
(
objects / (name + (".webp" if name == "optimized" else ".jpg"))
).write_bytes(body)
manifest[key] = digest(body)
code, body = self.s.s3(
"GET",
query={
"list-type": "2",
"prefix": "variants/" + self.attachment + "/v" + str(version) + "/",
},
)
require(code == 200, "cannot list variant set")
require(
len(ET.fromstring(body).findall("{*}Contents")) == 3,
"extra/missing object set",
)
history = snapshot["publications"]
versions = [p["version"] for p in history]
require(
versions == sorted(versions) and len(versions) == len(set(versions)),
"publication duplicated or rolled back",
)
require(versions[-1] == version, "publication history missing active version")
require(
all(
j["version"] == version or j["superseded"] is not None
for j in snapshot["jobs"]
),
"older accepted job has no durable disposition",
)
dump(self.directory / (observation + "-manifest.json"), manifest)
return manifest, snapshot
def execute(self):
self.s.worker_action("stop", self.s.workers())
self.s.call("stop", "dispatcher")
self.upload()
workers = self.s.workers()
require(len(workers) == 2, "two workers are required by protocol")
scenario = self.scenario
version = 1
first = None
if scenario == "commit_publish_gap":
marker = self.arm("after_commit_before_publish")
if scenario != "reorder":
self.s.call("start", "dispatcher")
if scenario == "commit_publish_gap":
self.wait(marker.exists, invariant=False)
if scenario == "duplicates":
self.publish(
self.s.messages(self.attachment)[0],
self.s.protocol["duplicates_before_ready"],
)
if scenario.startswith("crash_"):
point = {
"crash_before_effect": "before_effect",
"crash_after_effect": "after_object_before_commit",
"crash_before_ack": "after_commit_before_ack",
}[scenario]
marker = self.arm(point)
self.s.worker_action("start", [workers[0]])
self.wait(marker.exists, invariant=False)
self.event("fault_hit", point=point)
elif scenario == "reorder":
# No consumer is running; broker order is deliberately v2 then old v1.
self.upgrade()
jobs = self.s.messages(self.attachment)
self.publish(jobs[1], 1)
self.publish(jobs[0], 1)
version = 2
self.s.worker_action("start", workers)
elif scenario == "stale_lease":
marker = self.arm("after_claim")
self.s.worker_action("start", [workers[0]])
self.wait(marker.exists, invariant=False)
self.s.call("start", "dispatcher")
self.upgrade()
self.s.worker_action("start", [workers[1]])
first, _ = self.verify(2)
version = 2
release = (
self.s.private
/ "faults"
/ ("after_claim." + self.attachment + ".release")
)
release.write_text("release\n")
self.event("release_old_lease")
self.wait(
Path(str(release) + ".observed").exists, timeout=5, invariant=False
)
self.wait(
(
self.s.private
/ "faults"
/ ("stale_rejected." + self.attachment + ".observed")
).exists,
timeout=5,
invariant=False,
)
else:
self.s.worker_action("start", workers)
if scenario == "old_after_v2":
old_manifest, _ = self.verify(1)
old = self.s.messages(self.attachment)[0]
self.upgrade()
version = 2
first, _ = self.verify(2)
if not any(
old_manifest.get(key.replace("/v2/", "/v1/")) != value
for key, value in first.items()
):
raise HarnessError(
"Version fingerprints are indistinguishable for this fixture"
)
self.publish(old, self.s.protocol["duplicates_after_ready"])
for key, checksum in old_manifest.items():
code, body = self.s.s3("GET", key)
require(
code == 200 and digest(body) == checksum,
"v1 immutable object changed",
)
if scenario == "duplicates":
first, before = self.verify(1)
self.publish(
self.s.messages(self.attachment)[0],
self.s.protocol["duplicates_after_ready"],
)
if scenario != "reorder":
self.s.call("start", "dispatcher")
self.s.worker_action("start", workers)
self.wait(lambda: self.s.queue_depth() == 0)
# Let an already-delivered stale attempt leave its pause and hit the fence.
if scenario == "stale_lease":
time.sleep(1)
final, snapshot = self.verify(version)
if first is not None:
require(first == final, "old delivery changed published bytes")
if scenario in ("duplicates", "old_after_v2"):
require(
all(j["attempts"] == 1 for j in snapshot["jobs"]),
"terminal duplicate started another attempt",
)
if scenario == "reorder":
self.s.call("start", "dispatcher")
self.event("case_complete", version=version)
return {
"attachment_id": self.attachment,
"version": version,
"snapshot": snapshot,
"manifest": final,
"observation_gaps": self.gaps,
"elapsed_seconds": time.monotonic() - self.start,
}
def recover(self):
for marker in (self.s.private / "faults").glob("after_claim.*.hit"):
Path(str(marker)[:-4] + ".release").write_text("cleanup release\n")
self.s.call("start", "dispatcher")
self.s.worker_action("start", self.s.workers())
self.wait(
lambda: (
self.s.sql(
"SELECT count(*) FROM attachments WHERE status IN ('queued','processing')"
)
== "0"
and self.s.queue_depth() == 0
)
)
def close(self):
self.events.close()
+85
View File
@@ -0,0 +1,85 @@
"""Whitelisted Docker lifecycle events; never serialize environment or exec args."""
import json
import subprocess
import threading
class LifecycleEvents:
def __init__(self, project, path):
self.closing = False
self.oom = 0
self.failed = False
self.path = path
self.process = subprocess.Popen(
[
"docker",
"events",
"--filter",
"label=com.docker.compose.project=" + project,
"--format",
"{{json .}}",
],
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
text=True,
)
self.thread = threading.Thread(target=self.read, daemon=True)
self.thread.start()
def read(self):
try:
with self.path.open("x") as output:
for line in self.process.stdout:
try:
event = json.loads(line)
except json.JSONDecodeError:
if self.closing:
break
raise
action = event.get("Action", "")
if event.get("Type") != "container" or action not in (
"create",
"start",
"stop",
"kill",
"die",
"restart",
"destroy",
"oom",
"pause",
"unpause",
):
continue
actor = event.get("Actor", {})
attributes = actor.get("Attributes", {})
value = {
"time_nano": event.get("timeNano"),
"action": action,
"container_id": actor.get("ID"),
"service": attributes.get("com.docker.compose.service"),
"exit_code": attributes.get("exitCode"),
"signal": attributes.get("signal"),
}
if action == "oom":
self.oom += 1
output.write(json.dumps(value) + "\n")
output.flush()
except Exception:
self.failed = True
finally:
if not self.closing:
self.failed = True
def close(self):
self.closing = True
if self.process.poll() is None:
self.process.terminate()
try:
self.process.wait(timeout=5)
except subprocess.TimeoutExpired:
self.process.kill()
self.process.wait()
self.thread.join(timeout=5)
if self.thread.is_alive():
self.failed = True
+293
View File
@@ -0,0 +1,293 @@
"""Deterministic inputs, source sealing and private isolated Compose configuration."""
import hashlib
import json
import os
import secrets
import struct
import subprocess
import zipfile
import zlib
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
PROTOCOL = ROOT / "experiments/config/protocol.json"
def digest(data):
return hashlib.sha256(data).hexdigest()
def ensure_local_ignore():
path = Path(
subprocess.check_output(
["git", "rev-parse", "--git-path", "info/exclude"], cwd=ROOT, text=True
).strip()
)
if not path.is_absolute():
path = ROOT / path
text = path.read_text() if path.exists() else ""
# Migrate the old blanket exclusion; public results belong in Git.
lines = [line for line in text.splitlines() if line.strip() != "evidence/idempotency/"]
for pattern in ("experiments/results/", "evidence/idempotency/**/.private/", "evidence/idempotency/.run.lock"):
if pattern not in lines:
lines.append(pattern)
updated = "\n".join(lines).rstrip() + "\n"
if updated != text:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(updated)
def dump(path, value):
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
)
def source_files():
names = (
subprocess.check_output(
["git", "ls-files", "-z", "--cached", "--others", "--exclude-standard"],
cwd=ROOT,
)
.decode()
.split("\0")
)
return sorted(
{
name
for name in names
if name
and not name.startswith(
(
"experiments/results/",
"evidence/",
".memory/",
"secrets/",
".git/",
)
)
and (ROOT / name).is_file()
}
)
def source_manifest():
return {name: digest((ROOT / name).read_bytes()) for name in source_files()}
def verify_seal(seal):
value = json.loads((seal / "manifest.json").read_text())
if value["files"] != source_manifest():
raise RuntimeError("Source changed after seal; create a new seal/series.")
if value["protocol_sha256"] != digest(PROTOCOL.read_bytes()):
raise RuntimeError("Protocol changed after seal.")
return value
def png(profile, seed):
w, h = profile["width"], profile["height"]
encoder = zlib.compressobj(6)
parts = []
for y in range(h):
row = bytearray([0])
for x in range(w):
v = (x * 17 + y * 31 + seed + (x * y) % 251) & 255
row.extend(
(
v,
(v * 13 + x) & 255,
(v * 7 + y) & 255,
((x + y) * 11) & 255 if profile["alpha"] else 255,
)
)
parts.append(encoder.compress(row))
parts.append(encoder.flush())
def chunk(kind, data):
return (
struct.pack(">I", len(data))
+ kind
+ data
+ struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)
)
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 6, 0, 0, 0))
+ chunk(b"IDAT", b"".join(parts))
+ chunk(b"IEND", b"")
)
def private_environment(directory, protocol):
directory.mkdir(parents=True, exist_ok=False)
os.chmod(directory, 0o700)
secret_dir = directory / "secrets"
secret_dir.mkdir()
password = secrets.token_hex(24)
values = {
"POSTGRES_DB": "messenger",
"POSTGRES_USER": "messenger",
"POSTGRES_PASSWORD": password,
"APP_HOST": "0.0.0.0",
"APP_PORT": "8080",
"APP_ENV": "experiment",
"API_REPLICAS": "2",
"API_GOMAXPROCS": "1",
"SESSION_TTL_SECONDS": "7200",
"IDEMPOTENCY_TTL_SECONDS": "86400",
"S3_ENDPOINT": "http://s3:9000",
"S3_PUBLIC_ENDPOINT": "http://127.0.0.1:" + str(protocol["ports"]["s3"]),
"S3_BUCKET": "messenger-attachments",
"S3_REGION": "us-east-1",
"DATABASE_URL_FILE": "/run/secrets/database_url",
"CACHE_URL_FILE": "/run/secrets/cache_url",
"BROKER_URL_FILE": "/run/secrets/broker_url",
"S3_ACCESS_KEY_ID_FILE": "/run/secrets/s3_access_key_id",
"S3_SECRET_ACCESS_KEY_FILE": "/run/secrets/s3_secret_access_key",
"JOB_TIMEOUT_SECONDS": str(protocol["job_timeout_seconds"]),
"JOB_MAX_ATTEMPTS": str(protocol["job_max_attempts"]),
"WORKER_MAX_PIXELS": "8000000",
"UPLOAD_CLEANUP_GRACE_SECONDS": "600",
}
credentials = {
"postgres_password": password,
"database_url": "postgres://messenger:"
+ password
+ "@db:5432/messenger?sslmode=disable",
"cache_url": "redis://redis:6379/0",
"s3_access_key_id": secrets.token_hex(12),
"s3_secret_access_key": secrets.token_hex(32),
"s3_api_access_key_id": secrets.token_hex(12),
"s3_api_secret_access_key": secrets.token_hex(32),
"broker_user": "messenger",
"broker_password": secrets.token_hex(32),
}
credentials["broker_url"] = (
"amqp://messenger:" + credentials["broker_password"] + "@broker:5672/%2F"
)
for name, value in credentials.items():
(secret_dir / name).write_text(value + "\n")
os.chmod(secret_dir / name, 0o644)
env_file = directory / "private.env"
env_file.write_text("".join(k + "=" + v + "\n" for k, v in values.items()))
os.chmod(env_file, 0o600)
faults = directory / "faults"
faults.mkdir()
os.chmod(faults, 0o777)
return values
def isolated_config(base, project, directory, protocol, app_image, s3_image):
# config --format json resolves names: remove them to avoid sharing lab volumes.
base["name"] = project
for group in ("volumes", "networks"):
for value in base.get(group, {}).values():
if value.get("external"):
raise RuntimeError("External resources are forbidden in experiment.")
value.pop("name", None)
services = base["services"]
for name in ("frontend", "backend-tests"):
services.pop(name, None)
for name, service in services.items():
if "build" in service:
service.pop("build")
service["image"] = s3_image if name == "s3" else app_image
for mount in service.get("volumes", []):
if (
mount.get("type") == "bind"
and Path(mount["source"]).parent == ROOT / "secrets"
):
mount["source"] = str(
directory / "secrets" / Path(mount["source"]).name
)
services["s3"]["ports"] = [
{
"target": 9000,
"published": str(protocol["ports"]["s3"]),
"host_ip": "127.0.0.1",
"protocol": "tcp",
}
]
services["haproxy"]["ports"] = [
{
"target": 8080,
"published": str(protocol["ports"]["api"]),
"host_ip": "127.0.0.1",
"protocol": "tcp",
}
]
services["worker"].setdefault("deploy", {})["replicas"] = protocol["workers"]
for name in ("worker", "dispatcher"):
services[name]["environment"]["PIPELINE_FAULT_DIR"] = "/run/pipeline-faults"
services[name].setdefault("volumes", []).append(
{
"type": "bind",
"source": str(directory / "faults"),
"target": "/run/pipeline-faults",
}
)
return base
def build_images(public):
# Only builds images; does not start a container or collect an experiment case.
subprocess.run(
["docker", "build", "-t", "lab5-idem-app:prepared", str(ROOT / "src")],
check=True,
stdout=(public / "app-build.log").open("w"),
stderr=subprocess.STDOUT,
)
subprocess.run(
[
"docker",
"build",
"-f",
str(ROOT / "infra/minio.Dockerfile"),
"-t",
"lab5-idem-s3:prepared",
str(ROOT / "infra"),
],
check=True,
stdout=(public / "s3-build.log").open("w"),
stderr=subprocess.STDOUT,
)
def image(tag):
return subprocess.check_output(
["docker", "image", "inspect", "--format", "{{.Id}}", tag], text=True
).strip()
return image("lab5-idem-app:prepared"), image("lab5-idem-s3:prepared")
def archive_public(run):
target = run.parent / (run.name + "-public.zip")
secret_values = []
for directory in (run / ".private").glob("*/secrets"):
for path in directory.iterdir():
if any(
word in path.name
for word in (
"password",
"secret",
"access_key",
"database_url",
"broker_url",
)
):
value = path.read_bytes().strip()
if len(value) >= 16:
secret_values.append(value)
with zipfile.ZipFile(target, "x", zipfile.ZIP_DEFLATED) as archive:
for path in sorted(run.rglob("*")):
relative = path.relative_to(run)
if path.is_file() and ".private" not in relative.parts:
if any(value in path.read_bytes() for value in secret_values):
raise RuntimeError(
"Secret detected; do not share archive: " + str(relative)
)
archive.write(path, relative)
return target
@@ -0,0 +1,62 @@
"""Startup races must not be hidden during experimental cases."""
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from driver import Stack, HarnessError
class BrokerReadinessTests(unittest.TestCase):
def make_stack(self, root):
private = root / ".private" / "r1"
(private / "secrets").mkdir(parents=True)
return Stack("lab5-idem-unit", root / "config.json", private, {})
def test_booting_application_is_retried_before_warmup(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
stack = self.make_stack(root)
responses = [
subprocess.CompletedProcess([], 64, "", "this command requires the 'rabbit' app to be running"),
subprocess.CompletedProcess([], 0, "[]", ""),
]
with patch("driver.subprocess.run", side_effect=responses) as run, patch("driver.time.sleep"):
stack.wait_broker_ready(timeout=10)
self.assertEqual(run.call_count, 2)
self.assertEqual(json.loads((root / "broker-readiness-r1.json").read_text())["attempts"], 2)
self.assertEqual(len((root / "broker-startup.jsonl").read_text().splitlines()), 1)
def test_nonstartup_errors_fail_without_retry(self):
with tempfile.TemporaryDirectory() as temporary:
stack = self.make_stack(Path(temporary))
response = subprocess.CompletedProcess([], 70, "", "Authentication failed")
with patch("driver.subprocess.run", return_value=response) as run:
with self.assertRaises(HarnessError):
stack.wait_broker_ready(timeout=10)
self.assertEqual(run.call_count, 1)
def test_runtime_queue_probe_does_not_swallow_startup_or_outage(self):
with tempfile.TemporaryDirectory() as temporary:
stack = self.make_stack(Path(temporary))
response = subprocess.CompletedProcess([], 64, "", "requires the 'rabbit' app to be running")
with patch("driver.subprocess.run", return_value=response) as run:
with self.assertRaises(HarnessError):
stack.queue_depth()
self.assertEqual(run.call_count, 1)
def test_expired_setup_budget_does_not_start_a_probe(self):
with tempfile.TemporaryDirectory() as temporary:
stack = self.make_stack(Path(temporary))
with patch("driver.subprocess.run") as run:
with self.assertRaises(HarnessError):
stack.wait_broker_ready(timeout=0)
run.assert_not_called()
if __name__ == "__main__":
unittest.main()
+225
View File
@@ -0,0 +1,225 @@
"""Preparation checks with synthetic in-memory data; these do not run the series."""
import copy
import importlib.util
import json
import struct
import subprocess
import sys
import tempfile
import unittest
import zlib
from unittest.mock import patch
from pathlib import Path
EXPERIMENTS = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(EXPERIMENTS / "src"))
import support
spec = importlib.util.spec_from_file_location("experiment_cli", EXPERIMENTS / "run.py")
cli = importlib.util.module_from_spec(spec)
spec.loader.exec_module(cli)
class ToolTests(unittest.TestCase):
def test_git_keeps_public_evidence_and_hides_credentials_after_migration(self):
rules = (support.ROOT / ".gitignore").read_text()
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
subprocess.run(["git", "init", "-q", str(root)], check=True)
exclude = root / ".git/info/exclude"
exclude.write_text("evidence/idempotency/\ncustom-local-rule/\n")
public = root / "evidence/idempotency/main-test/summary.json"
private = public.parent / ".private/r1/private.env"
private.parent.mkdir(parents=True)
public.write_text("{}")
private.write_text("POSTGRES_PASSWORD=test-only\n")
(root / ".gitignore").write_text(rules)
log = public.parent / "boot-r1.log"
log.write_text("setup log\n")
with patch.object(support, "ROOT", root):
support.ensure_local_ignore()
first = exclude.read_text()
support.ensure_local_ignore()
self.assertEqual(first, exclude.read_text())
files = support.source_files()
self.assertIn("custom-local-rule/", first)
self.assertEqual(subprocess.run(["git", "check-ignore", "-q", str(public)], cwd=root).returncode, 1)
self.assertEqual(subprocess.run(["git", "check-ignore", "-q", str(private)], cwd=root).returncode, 0)
self.assertEqual(subprocess.run(["git", "check-ignore", "-q", str(log)], cwd=root).returncode, 1)
self.assertFalse(any(name.startswith("evidence/") for name in files))
def test_outputs_belong_to_root_evidence(self):
self.assertEqual(cli.RESULTS, support.ROOT / "evidence/idempotency")
def test_png_is_reproducible_and_decodable(self):
profile = {"width": 17, "height": 9, "alpha": True}
first = support.png(profile, 2303)
self.assertEqual(first, support.png(profile, 2303))
self.assertNotEqual(first, support.png(profile, 2304))
offset = 8
compressed = []
while offset < len(first):
size = struct.unpack(">I", first[offset : offset + 4])[0]
kind = first[offset + 4 : offset + 8]
data = first[offset + 8 : offset + 8 + size]
self.assertEqual(
zlib.crc32(kind + data) & 0xFFFFFFFF,
struct.unpack(">I", first[offset + 8 + size : offset + 12 + size])[0],
)
if kind == b"IDAT":
compressed.append(data)
offset += 12 + size
self.assertEqual(len(zlib.decompress(b"".join(compressed))), 9 * (17 * 4 + 1))
def test_isolation_removes_shared_resource_names_and_secret_paths(self):
service = {
"build": {"context": "x"},
"volumes": [
{
"type": "bind",
"source": str(support.ROOT / "secrets/database_url"),
"target": "/run/secrets/database_url",
}
],
"environment": {},
}
services = {
name: copy.deepcopy(service)
for name in (
"api",
"worker",
"dispatcher",
"s3",
"haproxy",
"frontend",
"backend-tests",
)
}
source = {
"services": services,
"volumes": {"db": {"name": "highload-lab5_db"}},
"networks": {"data": {"name": "highload-lab5_data"}},
}
protocol = json.loads(support.PROTOCOL.read_text())
private = Path("/tmp/private-example")
result = support.isolated_config(
source, "lab5-idem-unit", private, protocol, "sha256:app", "sha256:s3"
)
self.assertNotIn("name", result["volumes"]["db"])
self.assertNotIn("name", result["networks"]["data"])
self.assertNotIn("frontend", result["services"])
self.assertEqual(result["services"]["worker"]["deploy"]["replicas"], 2)
for entry in result["services"].values():
self.assertNotIn("build", entry)
self.assertTrue(
all(
not mount.get("source", "").startswith(
str(support.ROOT / "secrets")
)
for mount in entry["volumes"]
)
)
def test_analyzer_does_not_call_partial_or_duplicate_matrix_success(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
protocol = json.loads(support.PROTOCOL.read_text())
support.dump(root / "protocol.json", protocol)
support.dump(root / "manifest.json", {"phase": "main"})
(root / "cases.jsonl").write_text("")
self.assertEqual(cli.analyze(root)["outcome"], "inconclusive")
row = {
"repetition": 1,
"profile": "rgb_small",
"scenario": "control",
"directory": "cases/test",
"status": "violation",
}
(root / "cases.jsonl").write_text(json.dumps(row) + "\n")
self.assertEqual(cli.analyze(root)["outcome"], "refuted")
support.dump(root / "manifest.json", {"phase": "pilot"})
self.assertEqual(
cli.analyze(root)["outcome"], "pilot_only_not_confirmation"
)
def test_private_credentials_never_enter_share_archive(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary) / "case"
root.mkdir()
(root / ".private/r1/secrets").mkdir(parents=True)
(root / ".private/r1/secrets/broker_password").write_text(
"sensitive-password-value-123"
)
(root / "safe.json").write_text("{}")
archive = support.archive_public(root)
import zipfile
with zipfile.ZipFile(archive) as value:
self.assertEqual(value.namelist(), ["safe.json"])
def test_analyzer_requires_unique_complete_cases_and_counts_oom(self):
# A tiny artificial protocol tests the policy, never supplies experiment data.
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
protocol = json.loads(support.PROTOCOL.read_text())
protocol.update(
repetitions=1, profiles=protocol["profiles"][:1], scenarios=["control"]
)
protocol["pass"]["cases"] = 1
support.dump(root / "protocol.json", protocol)
support.dump(
root / "manifest.json", {"phase": "main", "finished_utc": "unit-test"}
)
for filename in (
"seal/manifest.json",
"seal/source.zip",
"fixtures/manifest.json",
"docker-events-r1.jsonl",
"environment-r1.json",
"order-r1.json",
):
path = root / filename
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("")
row = {
"repetition": 1,
"profile": "rgb_small",
"scenario": "control",
"directory": "cases/unit",
"status": "pass",
}
for filename in (
"events.jsonl",
"result.json",
"containers-before.json",
"containers-after.json",
"database-final.json",
"fault-markers.json",
):
path = root / row["directory"] / filename
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("{}")
(root / "cases.jsonl").write_text(json.dumps(row) + "\n")
self.assertEqual(
cli.analyze(root)["outcome"], "supported_in_tested_conditions"
)
(root / "cases.jsonl").write_text((json.dumps(row) + "\n") * 2)
self.assertEqual(cli.analyze(root)["outcome"], "inconclusive")
(root / "cases.jsonl").write_text(json.dumps(row) + "\n")
(root / "docker-events-r1.jsonl").write_text('{"action":"oom"}\n')
self.assertEqual(cli.analyze(root)["outcome"], "refuted")
def test_archive_refuses_a_public_file_with_local_secret(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary) / "case"
(root / ".private/r1/secrets").mkdir(parents=True)
secret = "local-sensitive-password-value"
(root / ".private/r1/secrets/postgres_password").write_text(secret)
(root / "leak.log").write_text(secret)
with self.assertRaises(RuntimeError):
support.archive_public(root)
if __name__ == "__main__":
unittest.main()