feat: add new experiment to test GOMAXPROCS and DB CPU limit

This commit is contained in:
TAskMAster339
2026-10-16 18:36:14 +03:00
parent a9557fa221
commit 14baedf5a1
8 changed files with 1074 additions and 0 deletions
+60
View File
@@ -0,0 +1,60 @@
# Масштабирование и GOMAXPROCS — v2
Это отдельная проверка замечаний к первому исследованию. Базовые конфигурации A1–B4, данные, запросы и SLO описаны в [исходной методике](../lab3-scale/docs/method.md). Ниже — только изменения. Старые результаты не заменяются.
## Что меняется и почему
1. **Предел нагрузки ищется на текущем компьютере.** Изначальный эксперимент определялся мощностью ПК, на котором запускался, из-за чего мощность самого приложения определялась некорректно. Теперь мы начинаем со 100 запросов/с и удваиваем нагрузку до первого нарушения SLO, максимум до 6400. Затем делим интервал пополам до ширины 10% или десяти точек на конфигурацию. Если уже 100 не выдерживается, снижаемся до 25. Это устраняет прежний потолок сетки 800: успешный запуск на 800 не означает предельную мощность 800. Если отказа не нашли, результат — только `C ≥ последнее успешное значение`.
2. **Подбор нагрузки отделён от проверки.** Пилот выбирает точки. После него программа сохраняет `confirmation.json` с контрольными нагрузками и порядком запусков, а затем проводит три новых блока для A1–B4. В каждом блоке проверяются 100 запросов/с и найденные нижняя и верхняя границы (без дубликатов). Гипотезы H1/H2 остаются прежними; интервалы отношений вычисляются только по новым блокам. Незамкнутая граница, пропуск или противоречие дают «данных недостаточно».
3. **Сравниваются явные `GOMAXPROCS=16` и `1`.** Для A4 выполняются пять пар при 25% и пять при 90% её пилотной нижней границы. Нагрузка внутри пары одинакова; порядок вариантов чередуется, порядок пар перемешан с фиксированным seed. Проверяем отдельно G_low и G_high: снижение p95 не менее 15%, p95 ≤500 мс, успешность ≥99% и не ниже контроля, подтверждённые сообщения сохранены. Критерий должен выполниться во всех пяти валидных парах. Пять отрицательных результатов опровергают гипотезу; смешанные результаты, пропуски или невалидные прогоны оставляют её неопределённой. Среднее, медиана и диапазон эффекта показываются дополнительно, без выбора удачных повторов. Если пилот A4 не нашёл обе границы, эти проверки пропускаются с указанием причины.
4. **Насыщение БД проверяется вмешательством.** На B4 сравниваем БД с 1 и 2 CPU при неизменных API, памяти и пуле соединений: три пары на нижней и три на верхней пилотной границе. Гипотеза DB_CPU: удвоение CPU БД возвращает SLO на верхней точке во всех трёх блоках, обе нижние точки проходят. Дополнительные условия: у исходной верхней точки средняя загрузка БД ≥90% лимита, API <85%, LB <90%. Без этих условий нельзя приписывать улучшение CPU БД. Отсутствие улучшения не исключает ограничений WAL, диска или блокировок. Для их разбора сохраняются счётчики WAL/I/O, ожидания PostgreSQL, ожидание пула и throttling. Если границы B4 нет, эта часть пропускается.
5. **Фиксируются условия повторения.** Перед запуском сохраняются исходники, их SHA-256, версия Docker, доступные CPU/RAM и ID единственного образа приложения. Фактические лимиты, размер пула и GOMAXPROCS проверяются на каждой точке. Телеметрия использует Docker context и число CPU Docker, без предположения о 16 процессорах хоста. Это позволяет повторить процедуру на другом компьютере, но не обещает одинаковый эффект на AMD и M4.
Длительность измерения — 30 с, прогрев — 15 с, пауза — 10 с. Перед каждой точкой восстанавливаются исходные данные и сессии, API пересоздаются. Порядок и seeds записаны в `design.json`. Расчёт p95 учитывает задержку отправки; отдельно сохраняются попытки, успешные запросы, ошибки, пропуски генератора и число ответов до конца окна. Пропуски >1% либо задержка отправки p95 >100 мс при CPU генератора ≥0,85 ядра останавливают серию как невалидную. Плохая производительность приложения сохраняется как результат, а не удаляется.
## Запуск в WSL
Из корня репозитория, с работающим Docker Desktop / Docker Engine:
```bash
# Только подготовка: сборка образа и фиксация методики, без нагрузки.
bash experiments/lab3-scale-v2/run.sh prepare scale-check-v2
```
Скрипт сам создаёт `.venv` в корне репозитория, устанавливает закреплённые зависимости и использует `.venv/bin/python`. Активировать окружение вручную не нужно. При следующих запусках оно используется повторно.
После подготовки не меняй код приложения и файлы обоих экспериментов до окончания серии. Изменения обнаруживаются по контрольным суммам; для другой версии нужен новый идентификатор с окончанием `-v2`.
```bash
bash experiments/lab3-scale-v2/run.sh run scale-check-v2
```
В другом терминале:
```bash
bash experiments/lab3-scale-v2/run.sh status scale-check-v2
# Пересчитать таблицы и графики из сохранённых запросов:
bash experiments/lab3-scale-v2/run.sh analyze scale-check-v2
# Если терминал был принудительно закрыт и приложение не восстановилось:
bash experiments/lab3-scale-v2/run.sh restore scale-check-v2
```
Ctrl+C завершает серию и запускает восстановление. Повторный запуск поверх начатой серии запрещён: для повторения выбери новый идентификатор и сначала выполни `prepare`.
## Где что лежит
Результаты появятся в `evidence/grade5-v2/scale-check-v2/`:
| Файл / папка | Содержимое |
| --- | --- |
| `design.json`, `design.sha256`, `source.tar.gz` | Зафиксированные правила, параметры и исходники |
| `environment-*.json` | Условия подготовки и запуска |
| `calibration/`, `calibration.json` | Пилотный поиск нагрузок; не входит в подтверждающую серию |
| `confirmation.json`, `confirmation.sha256` | План независимых проверок, сохранённый до их начала |
| `scaling/`, `gomax/`, `database/` | Новые измерения трёх частей исследования |
| `execution-order.json`, `status.json`, `restoration.json` | Порядок, прогресс и восстановление основного приложения |
| `summary.csv`, `analysis.json`, `RESULTS.md`, `*.png` | Пересчитанные метрики, решения по гипотезам и графики |
Внутри каждого измерения: `manifest.json` — параметры и контейнеры, `requests.jsonl.gz` — отдельные запросы, `telemetry.jsonl.gz` — ресурсы и БД, `pool.json` — пул/GOMAXPROCS, `summary.json` — метрики и проверка сохранности сообщений, `diagnostics.json` — диагностика. Прогрев хранится отдельно. Пароли находятся только в `.memory`.
`design.py` задаёт правила и порядок, `runner.py` управляет серией, `runtime.py` управляет отдельным стендом и телеметрией, `results.py` пересчитывает данные. Сам генератор запросов и SQL-набор данных используются из первого эксперимента. Команда `smoke` выполняет только короткую проверку инструментов и сохраняет её в `.memory`, отдельно от результатов исследования.
+81
View File
@@ -0,0 +1,81 @@
services:
db:
image: postgres:16.10-bookworm
cpus: ${BENCH_DB_CPU:-1}
mem_limit: 512m
command: [postgres, -c, track_io_timing=on, -c, track_wal_io_timing=on]
environment:
POSTGRES_DB: benchmark
POSTGRES_USER: benchmark
POSTGRES_PASSWORD: ${BENCH_PASSWORD:?private environment required}
volumes: [bench_db:/var/lib/postgresql/data]
networks: [data]
healthcheck:
test: [CMD-SHELL, 'pg_isready -U benchmark -d benchmark']
interval: 1s
timeout: 3s
retries: 60
redis:
image: redis:7.4.11-bookworm
cpus: 0.25
mem_limit: 128m
command: [redis-server, --appendonly, 'yes', --appendfsync, everysec, --dir, /data]
volumes: [bench_redis:/data]
networks: [data]
healthcheck:
test: [CMD, redis-cli, ping]
interval: 1s
timeout: 3s
retries: 60
migrate:
image: ${BENCH_IMAGE:?prepared image required}
cpus: 0.5
mem_limit: 128m
networks: [data]
environment:
DATABASE_URL: postgres://benchmark:${BENCH_PASSWORD}@db:5432/benchmark?sslmode=disable
command: [./messenger, migrate, up]
depends_on:
db: {condition: service_healthy}
api:
image: ${BENCH_IMAGE:?prepared image required}
cpus: ${BENCH_CPU:-0.5}
mem_limit: ${BENCH_MEMORY:-128m}
deploy:
replicas: ${BENCH_REPLICAS:-2}
stop_grace_period: 15s
networks: [proxy, data]
environment:
DATABASE_URL: postgres://benchmark:${BENCH_PASSWORD}@db:5432/benchmark?sslmode=disable
DATABASE_MAX_CONNS: ${BENCH_POOL:-6}
BENCH_POOL_STATS: '1'
GOMAXPROCS: ${BENCH_GOMAXPROCS:-16}
REDIS_ADDR: redis:6379
SESSION_TTL_SECONDS: 86400
APP_PORT: 8080
APP_HOST: 0.0.0.0
healthcheck:
test: [CMD, ./messenger, healthcheck]
interval: 1s
timeout: 3s
retries: 60
depends_on:
migrate: {condition: service_completed_successfully}
redis: {condition: service_healthy}
haproxy:
image: haproxy:3.2.25-alpine
cpus: 0.5
mem_limit: 128m
ports: ['127.0.0.1:18082:8080', '127.0.0.1:18406:8404']
networks: [edge, proxy]
volumes:
- ./experiments/lab3-scale/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
depends_on:
api: {condition: service_healthy}
volumes:
bench_db:
bench_redis:
networks:
edge:
proxy: {internal: true}
data: {internal: true}
+134
View File
@@ -0,0 +1,134 @@
"""Pure, deterministic rules for the preregistered v2 experiment."""
import random
import statistics
CONFIGS = {
"A1": {"replicas": 1, "cpu": 1, "memory": "256m", "pool": 12},
"A2": {"replicas": 2, "cpu": .5, "memory": "128m", "pool": 6},
"A4": {"replicas": 4, "cpu": .25, "memory": "64m", "pool": 3},
"B1": {"replicas": 1, "cpu": .5, "memory": "128m", "pool": 12},
"B2": {"replicas": 2, "cpu": .5, "memory": "128m", "pool": 6},
"B4": {"replicas": 4, "cpu": .5, "memory": "128m", "pool": 3},
}
DESIGN = {
"version": "v2.0",
"hypotheses": {
"H1": "Fixed total API budget: C4/C1 < 1.20 in all three blocks",
"H2": "Growing API budget: C4/C1 >= 1.50 in all three blocks",
"G_low": "At 25% of pilot A4 capacity, GOMAXPROCS=1 reduces p95 >=15% in all five pairs, retaining success and SLO",
"G_high": "At 90% of pilot A4 capacity, the same criteria hold in all five pairs",
"DB_CPU": "At the pilot B4 upper rate, increasing only DB CPU 1->2 restores SLO in all three blocks, with CPU-DB diagnostic prerequisites",
},
"duration_s": 30, "warmup_s": 15, "cooldown_s": 10,
"timeout_s": 5, "workload_seed": 230300, "inflight_limit": 8192,
"success_min": .99, "p95_max_ms": 500,
"workload": "80% new idempotent POST messages (128 ASCII bytes), 20% first history page GET limit=50",
"fixture": {"users": 200, "chats": 100, "messages": 10000},
"dependencies": {"db_cpu": 1, "db_memory": "512m", "redis_cpu": .25, "redis_memory": "128m", "lb_cpu": .5, "lb_memory": "128m"},
"pool_total": 12, "control_gomaxprocs": 16, "tuned_gomaxprocs": 1,
"search": {"start": 100, "floor": 25, "ceiling": 6400, "max_points": 10, "relative_width": .10},
"scaling_repeats": 3, "gomax_pairs": 5, "db_pairs": 3,
"calibration_seed": 230700, "scaling_seeds": [230701, 230702, 230703],
"gomax_order_seed": 230816, "db_order_seed": 230817,
"budget_seconds": 9000,
"generator_invalid": "dropped/scheduled >0.01 OR (dispatch p95 >100ms AND generator CPU >=0.85 cores)",
"db_cpu_prerequisites": {"db_cpu_min": .90, "api_cpu_max": .85, "lb_cpu_max": .90},
"aggregation": "Keep all pairs. Mean/median/range of paired percent reductions are descriptive, not a pooled p95. Mixed outcomes are inconclusive.",
"censoring": "No failed upper point means only a lower bound, never an exact capacity. Invalid generator stops the experiment; bad API performance and OOM are retained.",
"pilot_excluded_from_confirmation": True,
}
def config(name, gomax=16, db_cpu=1):
return {**CONFIGS[name], "gomaxprocs": gomax, "db_cpu": db_cpu}
def next_rate(observations, rules=None):
rules = rules or DESIGN["search"]
if any(not row["valid"] for row in observations):
return None
if not observations:
return rules["start"]
if len(observations) >= rules["max_points"]:
return None
passed = [row["rate"] for row in observations if row["slo_pass"]]
failed = [row["rate"] for row in observations if not row["slo_pass"]]
if passed and failed and max(passed) >= min(failed):
return None
if not passed:
value = max(rules["floor"], min(failed) // 2)
return value if value < min(failed) else None
lower = max(passed)
if not failed:
value = min(lower * 2, rules["ceiling"])
return value if value > lower else None
upper = min(failed)
if (upper - lower) / lower <= rules["relative_width"] or upper - lower <= 1:
return None
return (lower + upper) // 2
def bracket(observations):
passed = [row["rate"] for row in observations if row["valid"] and row["slo_pass"]]
failed = [row["rate"] for row in observations if row["valid"] and not row["slo_pass"]]
lower = max(passed, default=0)
upper = min(failed) if failed else None
invalid = any(not row["valid"] for row in observations)
nonmonotonic = upper is not None and lower >= upper
return {"lower": lower, "upper": upper, "invalid": invalid, "nonmonotonic": nonmonotonic,
"closed": bool(lower and upper and not invalid and not nonmonotonic)}
def freeze_confirmation(calibration):
bounds = {name: bracket(rows) for name, rows in calibration.items()}
rates = {}
for name, value in bounds.items():
candidates = [100, value["lower"], value["upper"]]
rates[name] = sorted({rate for rate in candidates if rate and rate <= DESIGN["search"]["ceiling"]})
scaling = []
for repeat, seed in enumerate(DESIGN["scaling_seeds"], 1):
points = [{"phase": "scaling", "base": name, "rate": rate, "repeat": repeat, "variant": name,
"config": config(name)} for name in CONFIGS for rate in rates[name]]
random.Random(seed).shuffle(points)
scaling.extend(points)
gomax = []
if bounds["A4"]["closed"]:
groups = []
for regime, fraction in (("low", .25), ("high", .90)):
rate = max(1, int(bounds["A4"]["lower"] * fraction))
for repeat in range(1, DESIGN["gomax_pairs"] + 1):
order = [16, 1] if repeat % 2 else [1, 16]
groups.append([{"phase": "gomax", "base": "A4", "regime": regime, "rate": rate,
"repeat": repeat, "variant": "I0" if value == 16 else "I1",
"config": config("A4", gomax=value)} for value in order])
random.Random(DESIGN["gomax_order_seed"]).shuffle(groups)
gomax = [point for group in groups for point in group]
database = []
if bounds["B4"]["closed"]:
groups = []
for repeat in range(1, DESIGN["db_pairs"] + 1):
for rate in (bounds["B4"]["lower"], bounds["B4"]["upper"]):
order = [1, 2] if repeat % 2 else [2, 1]
groups.append([{"phase": "database", "base": "B4", "rate": rate, "repeat": repeat,
"variant": "D0" if cpu == 1 else "D1", "config": config("B4", db_cpu=cpu)} for cpu in order])
random.Random(DESIGN["db_order_seed"]).shuffle(groups)
database = [point for group in groups for point in group]
return {"pilot_bounds": bounds, "rates_by_config": rates, "scaling": scaling,
"database": database, "gomax": gomax,
"skipped": {"gomax": None if gomax else "A4 pilot bracket not closed",
"database": None if database else "B4 pilot bracket not closed"}}
def pair_decision(pairs, expected=5):
if len(pairs) != expected or any(not pair["valid"] for pair in pairs):
result = "inconclusive"
elif all(pair["criteria_met"] for pair in pairs):
result = "supported"
elif all(not pair["criteria_met"] for pair in pairs):
result = "refuted"
else:
result = "inconclusive"
values = [pair["reduction_percent"] for pair in pairs if pair["reduction_percent"] is not None]
return {"result": result, "pairs": pairs, "reduction_mean_percent": statistics.mean(values) if values else None,
"reduction_median_percent": statistics.median(values) if values else None,
"reduction_range_percent": [min(values), max(values)] if values else None}
+210
View File
@@ -0,0 +1,210 @@
"""Recompute v2 results from individual requests; never touch v1 evidence."""
import csv
import gzip
import json
from collections import defaultdict
from design import CONFIGS, bracket, pair_decision
from runtime import digest, legacy_analysis, save, workload
def read(path):
return json.loads(path.read_text(encoding="utf-8"))
def jsonl(path):
with gzip.open(path, "rt", encoding="utf-8") as stream:
return [json.loads(line) for line in stream]
def audit(directory, manifest, design):
summary = read(directory / "summary.json")
rows = jsonl(directory / "requests.jsonl.gz")
successes = [row for row in rows if row["success"]]
sent = [row for row in rows if "start_s" in row]
duration = design["duration_s"]
calculated = {
"scheduled": len(rows), "sent": len(sent), "successes": len(successes),
"errors": len(rows) - len(successes), "generator_dropped": len(rows) - len(sent),
"success_ratio": len(successes) / max(1, len(rows)),
"goodput_rps": len(successes) / duration,
"window_goodput_rps": sum(row["end_s"] <= duration for row in successes) / duration,
"p95_ms": workload.percentile([row["latency_ms"] for row in successes], .95),
"dispatch_p95_ms": workload.percentile([row["dispatch_lag_ms"] for row in sent], .95),
}
mismatches = [key for key, value in calculated.items() if summary.get(key) != value]
if len(rows) != round(manifest["rate"] * duration):
mismatches.append("expected_request_count")
if manifest["protocol_sha256"] != digest(design):
mismatches.append("protocol_sha256")
if len({row["index"] for row in rows}) != len(rows):
mismatches.append("unique_request_index")
valid = summary["valid"] and not mismatches
slo = valid and summary["correctness"]["passed"] and calculated["success_ratio"] >= .99 and calculated["p95_ms"] is not None and calculated["p95_ms"] <= 500
diagnostic = legacy_analysis.diagnostics(directory, manifest)
return {**manifest, **summary, **calculated, "valid": valid, "slo_pass": slo,
"audit_mismatches": mismatches, "diagnostics": diagnostic,
"database_counters": database_counters(jsonl(directory / "telemetry.jsonl.gz"))}
def database_counters(samples):
values = [row["postgres"] for row in samples if "postgres" in row]
if len(values) < 2:
return {"available": False}
first, last = values[0], values[-1]
result = {"available": True, "note": "First-to-last sampled counters, not the full request window; waits are sample counts, not wait durations."}
for section, keys in (("wal", ("wal_records", "wal_bytes", "wal_write", "wal_sync", "wal_write_time", "wal_sync_time")),
("database", ("xact_commit", "xact_rollback", "blks_read", "blks_hit", "deadlocks"))):
reset = first[section].get("stats_reset") != last[section].get("stats_reset")
delta = {key: float(last[section][key]) - float(first[section][key]) for key in keys}
result[section] = {"reset_detected": reset or any(value < 0 for value in delta.values()), "delta": delta}
io = {}
old = {(row["object"], row["context"]): row for row in first["io"]}
for row in last["io"]:
prior = old.get((row["object"], row["context"]))
if prior:
keys = ("reads", "read_time", "writes", "write_time", "extends", "extend_time", "fsyncs", "fsync_time")
delta = {key: float(row[key] or 0) - float(prior[key] or 0) for key in keys}
io[row["object"] + "/" + row["context"]] = {"delta": delta, "reset_detected": row.get("stats_reset") != prior.get("stats_reset") or any(v < 0 for v in delta.values())}
result["io"] = io
return result
def capacities(runs, schedule):
result = []
for name in CONFIGS:
for repeat in (1, 2, 3):
points = [row for row in runs if row["phase"] == "scaling" and row["base"] == name and row["repeat"] == repeat]
bound = bracket(points)
expected = len([row for row in schedule if row["base"] == name and row["repeat"] == repeat])
incomplete = len(points) != expected or not expected
result.append({"config": name, "repeat": repeat, "capacity_lower": bound["lower"],
"capacity_upper": bound["upper"], "nonmonotonic": bound["nonmonotonic"],
"invalid_rates": [row["rate"] for row in points if not row["valid"]] + (["missing"] if incomplete else [])})
return result
def gomax_results(runs):
result = {}
for regime in ("low", "high"):
pairs = []
for repeat in range(1, 6):
values = {row["variant"]: row for row in runs if row["phase"] == "gomax" and row["regime"] == regime and row["repeat"] == repeat}
if "I0" not in values or "I1" not in values:
continue
control, tuned = values["I0"], values["I1"]
numeric = control["p95_ms"] is not None and control["p95_ms"] > 0 and tuned["p95_ms"] is not None
reduction = 100 * (1 - tuned["p95_ms"] / control["p95_ms"]) if numeric else None
valid = numeric and control["valid"] and tuned["valid"] and control["rate"] == tuned["rate"] and control["correctness"]["passed"] and tuned["correctness"]["passed"]
met = numeric and tuned["p95_ms"] <= .85 * control["p95_ms"] and tuned["p95_ms"] <= 500 and tuned["success_ratio"] >= max(.99, control["success_ratio"])
pairs.append({"repeat": repeat, "rate": control["rate"], "control_p95_ms": control["p95_ms"], "tuned_p95_ms": tuned["p95_ms"],
"valid": bool(valid), "criteria_met": bool(met), "reduction_percent": reduction})
result["G_" + regime] = pair_decision(pairs)
return result
def db_results(runs, confirmation):
bounds = confirmation.get("pilot_bounds", {}).get("B4", {})
lower, upper = bounds.get("lower"), bounds.get("upper")
blocks = []
for repeat in (1, 2, 3):
values = {(row["variant"], row["rate"]): row for row in runs if row["phase"] == "database" and row["repeat"] == repeat}
required = [(variant, rate) for variant in ("D0", "D1") for rate in (lower, upper)]
if not lower or not upper or any(key not in values for key in required):
continue
dl, du, tl, tu = (values[key] for key in (("D0", lower), ("D0", upper), ("D1", lower), ("D1", upper)))
cpu = du["diagnostics"]["cpu_fraction_of_limit_mean"]
eligible = cpu.get("db", -1) >= .90 and 0 <= cpu.get("api", -1) < .85 and 0 <= cpu.get("haproxy", -1) < .90
valid = all(values[key]["valid"] and values[key]["correctness"]["passed"] for key in required)
restored = dl["slo_pass"] and tl["slo_pass"] and not du["slo_pass"] and tu["slo_pass"]
blocks.append({"repeat": repeat, "valid": valid, "cpu_prerequisites_met": eligible,
"slo_restored": restored, "baseline_upper_cpu": cpu})
supported = len(blocks) == 3 and all(row["valid"] and row["cpu_prerequisites_met"] and row["slo_restored"] for row in blocks)
return {"result": "supported" if supported else "inconclusive", "blocks": blocks,
"note": "Doubling CPU tests a CPU bottleneck only. No effect does not disprove a WAL, storage, lock or pool bottleneck."}
def plots(directory, caps, hypotheses, runs):
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(10, 4))
for index, row in enumerate(caps):
lower, upper = row["capacity_lower"], row["capacity_upper"]
ax.plot(index, lower, "o", color="tab:blue")
if upper is not None:
ax.plot([index, index], [lower, upper], color="tab:blue")
ax.plot(index, upper, "x", color="tab:red")
else:
ax.annotate("≥", (index, lower), xytext=(0, 7), textcoords="offset points", ha="center")
ax.set_xticks(range(len(caps)), [r["config"] + "/" + str(r["repeat"]) for r in caps], rotation=60)
ax.set(ylabel="Offered requests/s", title="Confirmation capacity brackets (not confidence intervals)")
ax.grid(axis="y", alpha=.2)
fig.tight_layout(); fig.savefig(directory / "capacity.png", dpi=160); plt.close(fig)
fig, axes = plt.subplots(1, 2, figsize=(9, 4))
for ax, regime in zip(axes, ("low", "high")):
for pair in hypotheses["G_" + regime]["pairs"]:
ax.plot([0, 1], [pair["control_p95_ms"], pair["tuned_p95_ms"]], "o-", label="pair " + str(pair["repeat"]))
ax.set_xticks([0, 1], ["GOMAXPROCS=16", "GOMAXPROCS=1"])
ax.set(title=regime, ylabel="Successful request p95, ms")
ax.grid(alpha=.2)
fig.tight_layout(); fig.savefig(directory / "gomax.png", dpi=160); plt.close(fig)
points = [row for row in runs if row["phase"] == "database"]
if points:
fig, ax = plt.subplots(figsize=(8, 4))
for variant in ("D0", "D1"):
group = [row for row in points if row["variant"] == variant]
ax.scatter([r["rate"] for r in group], [r["p95_ms"] for r in group], label=variant)
ax.axhline(500, color="red", linestyle="--", label="p95 SLO (also success ≥99%)")
ax.set(xlabel="Offered requests/s", ylabel="Successful request p95, ms")
ax.legend(); fig.tight_layout(); fig.savefig(directory / "database.png", dpi=160); plt.close(fig)
def analyze(directory):
design = read(directory / "design.json")
if digest(design) != (directory / "design.sha256").read_text().strip():
raise ValueError("Design checksum mismatch")
confirmation = read(directory / "confirmation.json") if (directory / "confirmation.json").exists() else {}
if confirmation and (digest(confirmation) != (directory / "confirmation.sha256").read_text().strip() or confirmation["design_sha256"] != digest(design)):
raise ValueError("Confirmation schedule checksum mismatch")
runs, incomplete = [], []
for item in read(directory / "execution-order.json") if (directory / "execution-order.json").exists() else []:
target = directory / item["directory"]
if not (target / "summary.json").exists():
incomplete.append(item["directory"])
continue
manifest = read(target / "manifest.json")
run = audit(target, manifest, design)
if any(manifest.get(key) != item.get(key) for key in ("phase", "base", "rate", "repeat", "variant", "config", "sequence")):
run.update(valid=False, slo_pass=False)
run["audit_mismatches"].append("execution_order")
runs.append(run)
caps = capacities(runs, confirmation.get("scaling", []))
hypotheses = {**legacy_analysis.hypothesis_decisions(caps), **gomax_results(runs), "DB_CPU": db_results(runs, confirmation)}
if design["smoke_only"]:
hypotheses = {name: {"result": "not_tested_smoke_only"} for name in hypotheses}
result = {"series": design["series"], "smoke_only": design["smoke_only"], "status": read(directory / "status.json"),
"incomplete_runs": incomplete, "hypotheses": hypotheses, "capacities": caps, "runs": runs}
save(directory / "analysis.json", result)
keys = ["phase", "base", "variant", "repeat", "rate", "scheduled", "sent", "successes", "errors", "generator_dropped",
"success_ratio", "goodput_rps", "window_goodput_rps", "p95_ms", "dispatch_p95_ms", "valid", "slo_pass"]
with (directory / "summary.csv").open("w", newline="", encoding="utf-8") as stream:
writer = csv.DictWriter(stream, fieldnames=keys, extrasaction="ignore")
writer.writeheader(); writer.writerows(runs)
lines = ["# Результаты " + design["series"], "", "Статус: `" + result["status"]["status"] + "`. Завершённых измерений: " + str(len(runs)) + ".", ""]
if design["smoke_only"]:
lines += ["Короткая проверка инструментов. Эти данные не проверяют гипотезы и не входят в исследование.", ""]
else:
lines += ["| Гипотеза | Результат |", "| --- | --- |"] + [f"| {name} | {value['result']} |" for name, value in hypotheses.items()]
lines += ["", "supported — критерий выполнен во всех повторах; refuted — во всех парах получен противоположный результат; inconclusive — данных недостаточно или повторы расходятся.", "",
"[Сводная таблица](summary.csv), [полный расчёт и диагностика](analysis.json), [зафиксированная методика](design.json).",
"", "Пилотные измерения выбирают нагрузку и не входят в подтверждающую серию. Пределы мощности — границы на проверенной сетке, а не доверительные интервалы. Если верхняя граница не найдена, известно только C ≥ нижней границы.", "",
"GOMAXPROCS оценивается по отдельным парам. Среднее, медиана и диапазон процентов снижения p95 приведены в analysis.json; p95 разных прогонов не объединяются. Среднее не заменяет критерий пяти пар.", "",
"Счётчики WAL/I/O отражают разницу между первым и последним образцом телеметрии. Частота наблюдения ожиданий не равна времени ожидания. Удвоение CPU БД проверяет только ограничение CPU; отсутствие эффекта не исключает ограничения диска, WAL или блокировок.", "",
"![Границы мощности](capacity.png)", "", "![Парные сравнения](gomax.png)"]
if any(row["phase"] == "database" for row in runs):
lines += ["", "![Диагностика БД](database.png)"]
plots(directory, caps, hypotheses, runs)
(directory / "RESULTS.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
print("Analysis: " + str(directory / "RESULTS.md"), flush=True)
return result
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/../.."
action=${1:-run}
series=${2:-scale-check-v2}
python=.venv/bin/python
requirements=experiments/lab3-scale/requirements-lock.txt
if [ ! -x "$python" ]; then
python3 -m venv .venv
fi
if ! cmp -s "$requirements" .venv/lab3-scale-v2-requirements.txt; then
"$python" -m pip install -r "$requirements"
cp "$requirements" .venv/lab3-scale-v2-requirements.txt
fi
export PYTHONDONTWRITEBYTECODE=1
exec "$python" -u experiments/lab3-scale-v2/runner.py "$action" --series "$series"
+213
View File
@@ -0,0 +1,213 @@
"""Prepare once; the user starts the long preregistered experiment explicitly."""
import argparse
import copy
import fcntl
import json
import random
import re
import sys
import tarfile
import time
from datetime import datetime, timezone
from pathlib import Path
from design import CONFIGS, DESIGN, config, freeze_confirmation, next_rate
from runtime import HERE, ROOT, Runtime, check_ports, digest, environment, original, save, shell, sources
def validate_name(name):
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_-]{0,63}-v2", name):
raise ValueError("Series name must contain letters/digits/_/- and end with -v2")
return name
def directory_for(name, smoke=False):
validate_name(name)
return ROOT / (".memory/lab3-scale-v2-smoke" if smoke else "evidence/grade5-v2") / name
def prepare(name, smoke=False):
directory = directory_for(name, smoke)
fingerprint = sources()
if directory.exists():
existing = json.loads((directory / "design.json").read_text())
if existing["sources"] != fingerprint:
raise RuntimeError("Prepared sources changed; choose a new series ending in -v2")
if digest(existing) != (directory / "design.sha256").read_text().strip():
raise RuntimeError("Prepared design checksum differs")
if json.loads((directory / "status.json").read_text())["status"] != "prepared":
raise RuntimeError("Series already started or preparation was incomplete; choose a new series")
print("Already prepared: " + str(directory), flush=True)
return directory
check_ports()
env = environment()
image = "highload-lab3-bench-v2:" + env["app_code_sha256"][:16]
print("Building one image for every v2 condition", flush=True)
shell(["docker", "build", "-t", image, "src"])
image_id = shell(["docker", "image", "inspect", "--format", "{{.Id}}", image])
design = {**copy.deepcopy(DESIGN), "series": name, "smoke_only": smoke,
"sources": fingerprint, "image": image, "image_id": image_id,
"frozen_utc": datetime.now(timezone.utc).isoformat()}
if smoke:
design.update(duration_s=5, warmup_s=2, cooldown_s=1, budget_seconds=300)
directory.mkdir(parents=True)
save(directory / "design.json", design)
(directory / "design.sha256").write_text(digest(design) + "\n")
save(directory / "environment-prepared.json", env)
with tarfile.open(directory / "source.tar.gz", "w:gz") as archive:
for relative in fingerprint:
archive.add(ROOT / relative, arcname=relative, recursive=False)
save(directory / "status.json", {"status": "preparing", "long_experiment_started": False,
"smoke_only": smoke})
# Credentials remain outside evidence and the source archive.
runtime = Runtime(name, image, env)
runtime.compose(["config", "--quiet"])
save(directory / "status.json", {"status": "prepared", "long_experiment_started": False,
"smoke_only": smoke})
print("PREPARED: " + str(directory), flush=True)
return directory
class StopExperiment(RuntimeError):
pass
class Experiment:
def __init__(self, directory):
self.directory = directory
self.design = json.loads((directory / "design.json").read_text())
if digest(self.design) != (directory / "design.sha256").read_text().strip():
raise RuntimeError("Frozen design checksum differs")
if self.design["sources"] != sources():
raise RuntimeError("Code/instrument changed after preparation; choose a new series")
current_image = shell(["docker", "image", "inspect", "--format", "{{.Id}}", self.design["image"]])
if current_image != self.design["image_id"]:
raise RuntimeError("Prepared Docker image changed")
self.protocol_sha = digest(self.design)
self.started = time.monotonic()
self.sequence = 0
self.runtime = Runtime(self.design["series"], self.design["image"], environment())
self.order = []
def run_point(self, point):
# Budget stops are explicit and keep every completed or partial run.
needed = self.design["warmup_s"] + self.design["duration_s"] + self.design["cooldown_s"] + 15
if time.monotonic() - self.started + needed > self.design["budget_seconds"]:
raise StopExperiment("budget_exhausted")
if self.design["sources"] != sources():
raise StopExperiment("sources_changed_during_experiment")
self.sequence += 1
name = f"{self.sequence:03d}-{point['variant']}-r{point['repeat']}-rate{point['rate']}"
if point.get("regime"):
name += "-" + point["regime"]
target = self.directory / point["phase"] / name
self.order.append({"sequence": self.sequence, "directory": str(target.relative_to(self.directory)), **point})
save(self.directory / "execution-order.json", self.order)
save(self.directory / "status.json", {"status": "running", "current": self.order[-1],
"elapsed_s": round(time.monotonic() - self.started, 1), "completed_runs": self.sequence - 1,
"smoke_only": self.design["smoke_only"]})
print(f"START {point['phase']} {name}", flush=True)
result = self.runtime.execute(point, target, self.protocol_sha, self.design, self.sequence)
if not result["valid"]:
cause = "generator_limited" if result["generator_limited"] else "instrument_or_runtime_invalid"
raise StopExperiment(cause)
if not result["correctness"]["passed"]:
raise StopExperiment("confirmed_data_not_preserved")
time.sleep(self.design["cooldown_s"])
return result
def execute(self):
old = json.loads((self.directory / "status.json").read_text())
if old["status"] != "prepared":
raise RuntimeError("Series has already started; raw data are retained. Prepare a new series.")
check_ports()
status, error, restored = "failed", None, False
try:
self.runtime.suspend_main()
save(self.directory / "environment-run.json", self.runtime.env_info)
if self.design["smoke_only"]:
for variant, settings in (("I0", config("A4", 16)), ("I1", config("A4", 1)), ("D1", config("A4", 16, 2))):
result = self.run_point({"phase": "smoke", "base": "A4", "rate": 20, "repeat": 1,
"variant": variant, "config": settings})
if not result["slo_pass"]:
raise StopExperiment("smoke_slo_not_met")
status = "smoke_complete"
else:
calibration = {name: [] for name in CONFIGS}
names = list(CONFIGS)
random.Random(self.design["calibration_seed"]).shuffle(names)
for name in names:
while True:
rate = next_rate(calibration[name], self.design["search"])
if rate is None:
break
result = self.run_point({"phase": "calibration", "base": name, "variant": name,
"rate": rate, "repeat": len(calibration[name]) + 1, "config": config(name)})
calibration[name].append({"rate": rate, "valid": result["valid"], "slo_pass": result["slo_pass"]})
save(self.directory / "calibration.json", calibration)
confirmation = freeze_confirmation(calibration)
confirmation["design_sha256"] = self.protocol_sha
save(self.directory / "confirmation.json", confirmation)
(self.directory / "confirmation.sha256").write_text(digest(confirmation) + "\n")
# This schedule is frozen before seeing any confirmation outcome.
for phase in ("scaling", "database", "gomax"):
for point in confirmation[phase]:
self.run_point(point)
status = "complete"
except StopExperiment as failure:
status, error = str(failure), str(failure)
except KeyboardInterrupt:
status, error = "interrupted", "KeyboardInterrupt"
except Exception as failure:
status, error = "failed", str(failure)
finally:
restored = self.runtime.restore()
save(self.directory / "restoration.json", json.loads((self.runtime.private / "restoration.json").read_text()))
save(self.directory / "status.json", {"status": status, "error": error, "restoration_ok": restored,
"elapsed_s": round(time.monotonic() - self.started, 1), "scheduled_or_started_runs": self.sequence,
"smoke_only": self.design["smoke_only"]})
print(f"FINISHED: {status}; original application restored={restored}", flush=True)
from results import analyze
analyze(self.directory)
return 0 if status in ("complete", "smoke_complete") and restored else 1
def main():
parser = argparse.ArgumentParser(__doc__)
parser.add_argument("action", choices=("prepare", "run", "smoke", "status", "analyze", "restore"))
parser.add_argument("--series", default="scale-check-v2")
args = parser.parse_args()
validate_name(args.series)
smoke = args.action == "smoke"
directory = directory_for(args.series, smoke)
if args.action == "status":
print((directory / "status.json").read_text())
return 0
if args.action == "analyze":
from results import analyze
analyze(directory)
return 0
lock_path = ROOT / ".memory/benchmark/experiment.lock"
lock_path.parent.mkdir(parents=True, exist_ok=True)
with lock_path.open("a") as lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
parser.error("Another v1/v2 experiment is running")
if args.action == "prepare":
prepare(args.series)
return 0
if args.action == "restore":
design = json.loads((directory / "design.json").read_text())
return 0 if Runtime(args.series, design["image"], environment()).restore() else 1
if smoke:
prepare(args.series, smoke=True)
return Experiment(directory).execute()
if __name__ == "__main__":
try:
sys.exit(main())
except Exception as error:
print("ERROR: " + str(error), file=sys.stderr)
sys.exit(1)
+276
View File
@@ -0,0 +1,276 @@
"""Isolated v2 Docker runtime; reuse the v1 workload and fixture unchanged."""
import asyncio
import csv
import hashlib
import importlib.util
import io
import json
import os
import platform
import secrets
import socket
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
HERE = Path(__file__).resolve().parent
ROOT = HERE.parents[1]
V1 = HERE.parent / "lab3-scale"
sys.path.insert(0, str(V1))
import common as original
import load as workload
import aiohttp
def module(path, name):
spec = importlib.util.spec_from_file_location(name, path)
result = importlib.util.module_from_spec(spec)
spec.loader.exec_module(result)
return result
legacy_analysis = module(V1 / "analyze.py", "scale_v1_analysis")
def save(path, data):
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def digest(data):
return hashlib.sha256(json.dumps(data, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
def sources():
paths = [p for base in (ROOT / "src/backend", V1, HERE) for p in base.rglob("*")
if p.is_file() and "__pycache__" not in p.parts and p.suffix.lower() != ".md"]
paths += [ROOT / name for name in ("src/go.mod", "src/go.sum", "src/Dockerfile")]
return {str(path.relative_to(ROOT)): hashlib.sha256(path.read_bytes()).hexdigest() for path in sorted(paths)}
def shell(args):
result = subprocess.run(args, cwd=ROOT, text=True, capture_output=True)
if result.returncode:
# Avoid echoing expanded connection strings or private Docker output.
raise RuntimeError("Command failed: " + " ".join(args[:3]))
return result.stdout.strip()
def docker_endpoint():
context = json.loads(shell(["docker", "context", "inspect"]))[0]
endpoint = os.environ.get("DOCKER_HOST") or context["Endpoints"]["docker"]["Host"]
if not endpoint.startswith("unix://"):
raise RuntimeError("v2 telemetry requires a local unix:// Docker context (WSL/Linux/macOS)")
path = endpoint.removeprefix("unix://")
if not Path(path).exists():
raise RuntimeError("Docker socket from the current context is unavailable")
return path
def check_ports():
for port in (18082, 18406):
with socket.socket() as probe:
try:
probe.bind(("127.0.0.1", port))
except OSError as error:
raise RuntimeError(f"Port {port} is occupied; no unrelated container will be stopped") from error
def environment():
info = json.loads(shell(["docker", "info", "--format",
'{"cpus":{{.NCPU}},"memory":{{.MemTotal}},"architecture":{{json .Architecture}},"os":{{json .OperatingSystem}}}']))
return {"time_utc": datetime.now(timezone.utc).isoformat(), "platform": platform.platform(),
"python": platform.python_version(), "docker_resources": info,
"docker_version": json.loads(shell(["docker", "version", "--format", "{{json .Server}}"])),
"compose_version": shell(["docker", "compose", "version"]),
"git_commit": shell(["git", "rev-parse", "HEAD"]),
"worktree_files": shell(["git", "status", "--porcelain"]).splitlines(),
"docker_socket": docker_endpoint(), "app_code_sha256": original.code_hash()}
DB_SAMPLE = """SELECT json_build_object(
'active',(SELECT count(*) FROM pg_stat_activity WHERE datname='benchmark' AND state='active' AND pid<>pg_backend_pid()),
'waits',(SELECT coalesce(json_agg(json_build_object('event_type',wait_event_type,'event',wait_event,'count',n)),'[]'::json)
FROM (SELECT wait_event_type,wait_event,count(*) n FROM pg_stat_activity WHERE datname='benchmark' AND pid<>pg_backend_pid() GROUP BY 1,2) t),
'database',(SELECT row_to_json(d) FROM (SELECT xact_commit,xact_rollback,blks_read,blks_hit,tup_inserted,deadlocks FROM pg_stat_database WHERE datname='benchmark') d),
'wal',(SELECT row_to_json(w) FROM pg_stat_wal w),
'io',(SELECT coalesce(json_agg(i),'[]'::json) FROM pg_stat_io i WHERE backend_type='client backend')
);"""
class Runtime:
def __init__(self, series, image, env_info):
self.series = series
self.private = ROOT / ".memory/lab3-scale-v2" / series
self.private.mkdir(parents=True, exist_ok=True)
self.project = "highload-lab3-v2-" + hashlib.sha256(series.encode()).hexdigest()[:10]
self.image = image
self.env_info = env_info
self.socket = env_info["docker_socket"]
self.db_cpu = None
credentials = self.private / "environment"
if not credentials.exists():
credentials.write_text("BENCH_PASSWORD=" + secrets.token_hex(24) + "\n")
credentials.chmod(0o600)
# Only this process is adapted. No v1 source or historical evidence changes.
original.compose = self.compose
original.PROJECT = self.project
workload.BASE = "http://127.0.0.1:18082"
workload.STATS = "http://127.0.0.1:18406/stats;csv"
workload.collect = self.collect
def compose(self, args, config=None):
config = config or {"cpu": .5, "memory": "128m", "replicas": 2, "pool": 6,
"gomaxprocs": 16, "db_cpu": 1}
env = {**os.environ, "BENCH_CPU": str(config["cpu"]), "BENCH_MEMORY": config["memory"],
"BENCH_REPLICAS": str(config["replicas"]), "BENCH_POOL": str(config["pool"]),
"BENCH_GOMAXPROCS": str(config["gomaxprocs"]), "BENCH_DB_CPU": str(config["db_cpu"]),
"BENCH_IMAGE": self.image}
result = subprocess.run(["docker", "compose", "--project-name", self.project, "--project-directory", str(ROOT),
"--env-file", str(self.private / "environment"), "-f", str(HERE / "compose.yml"), *args],
cwd=ROOT, env=env, text=True, capture_output=True)
if result.returncode:
(self.private / "compose-error.log").write_text(result.stdout + result.stderr)
raise RuntimeError("v2 Compose failed; see " + str(self.private / "compose-error.log"))
return result.stdout.strip()
def suspend_main(self):
ids = shell(["docker", "ps", "-q", "--filter", "label=com.docker.compose.project=highload-lab3"]).split()
records = []
for container in ids:
data = json.loads(shell(["docker", "inspect", "--format",
'{"id":{{json .Id}},"service":{{json (index .Config.Labels "com.docker.compose.service")}}}', container]))
records.append(data)
save(self.private / "original-state.json", records)
if ids:
shell(["docker", "stop", *ids])
def restore(self):
errors = []
try:
self.compose(["stop"])
except Exception as error:
errors.append(str(error))
path = self.private / "original-state.json"
records = json.loads(path.read_text()) if path.exists() else []
for group in (("db", "redis"), ("api",), ("haproxy", "frontend")):
for record in records:
if record["service"] not in group:
continue
try:
project = shell(["docker", "inspect", "--format", '{{index .Config.Labels "com.docker.compose.project"}}', record["id"]])
if project != "highload-lab3":
raise RuntimeError("Original container identity changed")
shell(["docker", "start", record["id"]])
deadline = time.monotonic() + 60
while time.monotonic() < deadline:
state = json.loads(shell(["docker", "inspect", "--format", "{{json .State}}", record["id"]]))
if state["Running"] and state.get("Health", {}).get("Status", "healthy") == "healthy":
break
time.sleep(1)
else:
raise RuntimeError("Original container did not become healthy: " + record["service"])
except Exception as error:
errors.append(str(error))
save(self.private / "restoration.json", {"ok": not errors, "errors": errors})
return not errors
def prepare_services(self, config):
if self.db_cpu != config["db_cpu"]:
self.compose(["up", "-d", "--wait", "db", "redis"], config)
# One explicit migration job, not migrations in each API.
self.compose(["run", "--rm", "--no-deps", "migrate"], config)
self.db_cpu = config["db_cpu"]
self.compose(["up", "-d", "--no-deps", "--force-recreate", "--wait", "api"], config)
self.compose(["up", "-d", "--no-deps", "haproxy"], config)
def execute(self, point, directory, protocol_sha, timings, sequence):
if directory.exists():
raise RuntimeError("Run already exists; raw measurements are never overwritten: " + str(directory))
directory.mkdir(parents=True)
config = point["config"]
self.prepare_services(config)
actors = original.restore_fixture()
actual = original.current_containers()
apis = [row for row in actual if row["service"] == "api"]
for name, cpu, memory in (("api", config["cpu"], int(config["memory"].removesuffix("m")) * 1048576),
("db", config["db_cpu"], 512 * 1048576), ("redis", .25, 128 * 1048576),
("haproxy", .5, 128 * 1048576)):
rows = [row for row in actual if row["service"] == name]
if not rows or any(row["cpu"] != cpu * 1e9 or row["memory"] != memory for row in rows):
raise RuntimeError("Effective limits differ from protocol: " + name)
if len(apis) != config["replicas"] or any(row["ports"] for row in apis):
raise RuntimeError("API replica count or exposed ports differ")
manifest = {**point, "run_id": directory.name, "sequence": sequence, "protocol_sha256": protocol_sha,
"code_sha256": original.code_hash(), "containers": actual,
"docker_cpus": self.env_info["docker_resources"]["cpus"],
"started_utc": datetime.now(timezone.utc).isoformat()}
save(directory / "manifest.json", manifest)
result = asyncio.run(workload.measure(actors, directory, config, rate=point["rate"],
duration=timings["duration_s"], warmup=timings["warmup_s"], seed=230300,
prefix=self.series + ":" + str(sequence), limit=8192))
samples = json.loads((directory / "pool.json").read_text())
expected = {row["instance"] for row in apis}
observed = {row["instance"] for row in samples}
runtime_ok = observed == expected and all(row.get("gomaxprocs") == config["gomaxprocs"] and row["max"] == config["pool"] for row in samples)
result["runtime_settings_verified"] = runtime_ok
result["valid"] = result["valid"] and runtime_ok
result["slo_pass"] = result["slo_pass"] and result["valid"]
result["rate"] = point["rate"]
save(directory / "summary.json", result)
diagnostic = legacy_analysis.diagnostics(directory, manifest)
save(directory / "diagnostics.json", diagnostic)
print(json.dumps({"run": directory.name, "rate": point["rate"], "p95_ms": result["p95_ms"],
"success": result["success_ratio"], "valid": result["valid"], "slo_pass": result["slo_pass"]}), flush=True)
return result
async def sample_db(self, container):
process = await asyncio.create_subprocess_exec("docker", "exec", "-i", "-e", "PGOPTIONS=-c statement_timeout=2000", container,
"psql", "-X", "-q", "-t", "-A", "-v", "ON_ERROR_STOP=1", "-U", "benchmark", "-d", "benchmark",
stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
try:
stdout, _ = await asyncio.wait_for(process.communicate(DB_SAMPLE.encode()), 4)
except BaseException:
if process.returncode is None:
process.kill()
await process.communicate()
raise
if process.returncode:
raise RuntimeError("DB telemetry unavailable")
return json.loads(stdout)
async def collect(self, stop, containers, rows):
db = next(row["id"] for row in containers if row["service"] == "db")
async with aiohttp.ClientSession(connector=aiohttp.UnixConnector(path=self.socket), timeout=aiohttp.ClientTimeout(total=4)) as engine, \
aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=4)) as http:
async def stats(container):
async with engine.get(f"http://docker/v1.47/containers/{container['id']}/stats?stream=false&one-shot=true") as response:
data = await response.json()
cpu = dict(data.get("cpu_stats") or {})
observed = cpu.get("online_cpus")
cpu["online_cpus"] = observed or self.env_info["docker_resources"]["cpus"]
return {"instance": container["instance"], "service": container["service"], "cpu_stats": cpu,
"cpu_count_source": "stats" if observed else "docker_info", "memory_stats": data.get("memory_stats"),
"blkio_stats": data.get("blkio_stats"), "stats_available": response.status == 200}
while not stop.is_set():
begin = time.monotonic()
try:
resources, database = await asyncio.gather(asyncio.gather(*(stats(row) for row in containers)), self.sample_db(db))
haproxy, stats_error = [], None
try:
async with http.get(workload.STATS) as response:
haproxy = [row for row in csv.DictReader(io.StringIO((await response.text()).removeprefix("# ")))
if row["pxname"] in ("api_replicas", "api_entry")]
except (aiohttp.ClientError, TimeoutError):
stats_error = "unavailable"
rows.append({"time": datetime.now(timezone.utc).isoformat(), "resources": resources,
"postgres": database, "haproxy": haproxy, "haproxy_stats_error": stats_error,
"generator_cpu_seconds": time.process_time(), "collection_seconds": time.monotonic() - begin})
except Exception as error:
rows.append({"time": datetime.now(timezone.utc).isoformat(), "error": type(error).__name__})
try:
await asyncio.wait_for(stop.wait(), max(.01, 1 - (time.monotonic() - begin)))
except TimeoutError:
pass
@@ -0,0 +1,84 @@
"""Checks for scientific decision gates, censoring and paired design."""
import unittest
from design import CONFIGS, bracket, config, freeze_confirmation, next_rate, pair_decision
from results import capacities, db_results, gomax_results
from runtime import legacy_analysis
def observation(rate, passed=True, valid=True):
return {"rate": rate, "slo_pass": passed, "valid": valid}
class DesignTests(unittest.TestCase):
def test_800_is_not_a_ceiling(self):
points = [observation(rate) for rate in (100, 200, 400, 800)]
self.assertEqual(next_rate(points), 1600)
self.assertIsNone(bracket(points)["upper"])
points += [observation(1600), observation(3200), observation(6400)]
self.assertIsNone(next_rate(points))
self.assertFalse(bracket(points)["closed"])
def test_search_floor_bisection_invalid_and_nonmonotonic(self):
self.assertEqual(next_rate([observation(100, False)]), 50)
self.assertIsNone(next_rate([observation(25, False)]))
self.assertEqual(next_rate([observation(100), observation(200, False)]), 150)
self.assertIsNone(next_rate([observation(100), observation(109, False)]))
self.assertIsNone(next_rate([observation(100, valid=False)]))
self.assertTrue(bracket([observation(100, False), observation(200)])["nonmonotonic"])
def test_confirmation_is_fresh_and_pairs_change_one_factor(self):
pilot = {name: [observation(400), observation(440, False)] for name in CONFIGS}
plan = freeze_confirmation(pilot)
self.assertEqual(len(plan["scaling"]), 54)
self.assertEqual(len(plan["gomax"]), 20)
self.assertEqual(len(plan["database"]), 12)
self.assertEqual(plan, freeze_confirmation(pilot))
for phase, field in (("gomax", "gomaxprocs"), ("database", "db_cpu")):
for index in range(0, len(plan[phase]), 2):
a, b = plan[phase][index:index + 2]
self.assertEqual(a["rate"], b["rate"])
self.assertEqual(a["repeat"], b["repeat"])
self.assertEqual({k: v for k, v in a["config"].items() if k != field}, {k: v for k, v in b["config"].items() if k != field})
self.assertNotEqual(a["config"][field], b["config"][field])
self.assertEqual({p["config"]["gomaxprocs"] for p in plan["gomax"]}, {1, 16})
def test_open_bracket_skips_causal_tests(self):
plan = freeze_confirmation({name: [observation(800)] for name in CONFIGS})
self.assertFalse(plan["gomax"])
self.assertFalse(plan["database"])
def test_mean_does_not_hide_a_failed_pair(self):
pairs = [{"valid": True, "criteria_met": effect >= 15, "reduction_percent": effect} for effect in (60, 60, 60, 60, 0)]
self.assertEqual(pair_decision(pairs)["result"], "inconclusive")
self.assertEqual(pair_decision(pairs)["reduction_mean_percent"], 48)
self.assertEqual(pair_decision(pairs[:4])["result"], "inconclusive")
self.assertEqual(pair_decision([dict(pairs[0]) for _ in range(5)])["result"], "supported")
self.assertEqual(pair_decision([dict(pairs[-1]) for _ in range(5)])["result"], "refuted")
pairs[0]["valid"] = False
self.assertEqual(pair_decision(pairs)["result"], "inconclusive")
def test_missing_or_censored_scaling_is_inconclusive(self):
runs = [{**observation(800), "phase": "scaling", "base": name, "repeat": repeat} for name in CONFIGS for repeat in (1, 2, 3)]
caps = capacities(runs, runs)
decisions = legacy_analysis.hypothesis_decisions(caps)
self.assertEqual(decisions["H2"]["result"], "inconclusive")
self.assertTrue(all(row["capacity_upper"] is None for row in caps))
self.assertEqual(legacy_analysis.hypothesis_decisions(capacities(runs[:-1], runs))["H2"]["result"], "inconclusive")
def test_db_needs_cpu_evidence_not_just_faster_result(self):
points = []
for repeat in (1, 2, 3):
for variant in ("D0", "D1"):
for rate in (400, 440):
points.append({"phase": "database", "variant": variant, "rate": rate, "repeat": repeat,
"valid": True, "correctness": {"passed": True}, "slo_pass": rate == 400 or variant == "D1",
"diagnostics": {"cpu_fraction_of_limit_mean": {"db": .95, "api": .6, "haproxy": .2}}})
plan = {"pilot_bounds": {"B4": {"lower": 400, "upper": 440}}}
self.assertEqual(db_results(points, plan)["result"], "supported")
points[1]["diagnostics"]["cpu_fraction_of_limit_mean"]["db"] = .4
self.assertEqual(db_results(points, plan)["result"], "inconclusive")
if __name__ == "__main__":
unittest.main()