feat: add new experiment to test GOMAXPROCS and DB CPU limit
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
# Масштабирование и GOMAXPROCS — v2
|
||||
|
||||
Это отдельная проверка замечаний к первому исследованию. Базовые конфигурации A1–B4, данные, запросы и SLO описаны в [исходной методике](../lab3-scale/docs/method.md). Ниже — только изменения. Старые результаты не заменяются.
|
||||
|
||||
## Что меняется и почему
|
||||
|
||||
1. **Предел нагрузки ищется на текущем компьютере.** Изначальный эксперимент определялся мощностью ПК, на котором запускался, из-за чего мощность самого приложения определялась некорректно. Теперь мы начинаем со 100 запросов/с и удваиваем нагрузку до первого нарушения SLO, максимум до 6400. Затем делим интервал пополам до ширины 10% или десяти точек на конфигурацию. Если уже 100 не выдерживается, снижаемся до 25. Это устраняет прежний потолок сетки 800: успешный запуск на 800 не означает предельную мощность 800. Если отказа не нашли, результат — только `C ≥ последнее успешное значение`.
|
||||
2. **Подбор нагрузки отделён от проверки.** Пилот выбирает точки. После него программа сохраняет `confirmation.json` с контрольными нагрузками и порядком запусков, а затем проводит три новых блока для A1–B4. В каждом блоке проверяются 100 запросов/с и найденные нижняя и верхняя границы (без дубликатов). Гипотезы H1/H2 остаются прежними; интервалы отношений вычисляются только по новым блокам. Незамкнутая граница, пропуск или противоречие дают «данных недостаточно».
|
||||
3. **Сравниваются явные `GOMAXPROCS=16` и `1`.** Для A4 выполняются пять пар при 25% и пять при 90% её пилотной нижней границы. Нагрузка внутри пары одинакова; порядок вариантов чередуется, порядок пар перемешан с фиксированным seed. Проверяем отдельно G_low и G_high: снижение p95 не менее 15%, p95 ≤500 мс, успешность ≥99% и не ниже контроля, подтверждённые сообщения сохранены. Критерий должен выполниться во всех пяти валидных парах. Пять отрицательных результатов опровергают гипотезу; смешанные результаты, пропуски или невалидные прогоны оставляют её неопределённой. Среднее, медиана и диапазон эффекта показываются дополнительно, без выбора удачных повторов. Если пилот A4 не нашёл обе границы, эти проверки пропускаются с указанием причины.
|
||||
4. **Насыщение БД проверяется вмешательством.** На B4 сравниваем БД с 1 и 2 CPU при неизменных API, памяти и пуле соединений: три пары на нижней и три на верхней пилотной границе. Гипотеза DB_CPU: удвоение CPU БД возвращает SLO на верхней точке во всех трёх блоках, обе нижние точки проходят. Дополнительные условия: у исходной верхней точки средняя загрузка БД ≥90% лимита, API <85%, LB <90%. Без этих условий нельзя приписывать улучшение CPU БД. Отсутствие улучшения не исключает ограничений WAL, диска или блокировок. Для их разбора сохраняются счётчики WAL/I/O, ожидания PostgreSQL, ожидание пула и throttling. Если границы B4 нет, эта часть пропускается.
|
||||
5. **Фиксируются условия повторения.** Перед запуском сохраняются исходники, их SHA-256, версия Docker, доступные CPU/RAM и ID единственного образа приложения. Фактические лимиты, размер пула и GOMAXPROCS проверяются на каждой точке. Телеметрия использует Docker context и число CPU Docker, без предположения о 16 процессорах хоста. Это позволяет повторить процедуру на другом компьютере, но не обещает одинаковый эффект на AMD и M4.
|
||||
|
||||
Длительность измерения — 30 с, прогрев — 15 с, пауза — 10 с. Перед каждой точкой восстанавливаются исходные данные и сессии, API пересоздаются. Порядок и seeds записаны в `design.json`. Расчёт p95 учитывает задержку отправки; отдельно сохраняются попытки, успешные запросы, ошибки, пропуски генератора и число ответов до конца окна. Пропуски >1% либо задержка отправки p95 >100 мс при CPU генератора ≥0,85 ядра останавливают серию как невалидную. Плохая производительность приложения сохраняется как результат, а не удаляется.
|
||||
|
||||
## Запуск в WSL
|
||||
|
||||
Из корня репозитория, с работающим Docker Desktop / Docker Engine:
|
||||
|
||||
```bash
|
||||
# Только подготовка: сборка образа и фиксация методики, без нагрузки.
|
||||
bash experiments/lab3-scale-v2/run.sh prepare scale-check-v2
|
||||
```
|
||||
|
||||
Скрипт сам создаёт `.venv` в корне репозитория, устанавливает закреплённые зависимости и использует `.venv/bin/python`. Активировать окружение вручную не нужно. При следующих запусках оно используется повторно.
|
||||
|
||||
После подготовки не меняй код приложения и файлы обоих экспериментов до окончания серии. Изменения обнаруживаются по контрольным суммам; для другой версии нужен новый идентификатор с окончанием `-v2`.
|
||||
|
||||
```bash
|
||||
bash experiments/lab3-scale-v2/run.sh run scale-check-v2
|
||||
```
|
||||
|
||||
В другом терминале:
|
||||
|
||||
```bash
|
||||
bash experiments/lab3-scale-v2/run.sh status scale-check-v2
|
||||
# Пересчитать таблицы и графики из сохранённых запросов:
|
||||
bash experiments/lab3-scale-v2/run.sh analyze scale-check-v2
|
||||
# Если терминал был принудительно закрыт и приложение не восстановилось:
|
||||
bash experiments/lab3-scale-v2/run.sh restore scale-check-v2
|
||||
```
|
||||
|
||||
Ctrl+C завершает серию и запускает восстановление. Повторный запуск поверх начатой серии запрещён: для повторения выбери новый идентификатор и сначала выполни `prepare`.
|
||||
|
||||
## Где что лежит
|
||||
|
||||
Результаты появятся в `evidence/grade5-v2/scale-check-v2/`:
|
||||
|
||||
| Файл / папка | Содержимое |
|
||||
| --- | --- |
|
||||
| `design.json`, `design.sha256`, `source.tar.gz` | Зафиксированные правила, параметры и исходники |
|
||||
| `environment-*.json` | Условия подготовки и запуска |
|
||||
| `calibration/`, `calibration.json` | Пилотный поиск нагрузок; не входит в подтверждающую серию |
|
||||
| `confirmation.json`, `confirmation.sha256` | План независимых проверок, сохранённый до их начала |
|
||||
| `scaling/`, `gomax/`, `database/` | Новые измерения трёх частей исследования |
|
||||
| `execution-order.json`, `status.json`, `restoration.json` | Порядок, прогресс и восстановление основного приложения |
|
||||
| `summary.csv`, `analysis.json`, `RESULTS.md`, `*.png` | Пересчитанные метрики, решения по гипотезам и графики |
|
||||
|
||||
Внутри каждого измерения: `manifest.json` — параметры и контейнеры, `requests.jsonl.gz` — отдельные запросы, `telemetry.jsonl.gz` — ресурсы и БД, `pool.json` — пул/GOMAXPROCS, `summary.json` — метрики и проверка сохранности сообщений, `diagnostics.json` — диагностика. Прогрев хранится отдельно. Пароли находятся только в `.memory`.
|
||||
|
||||
`design.py` задаёт правила и порядок, `runner.py` управляет серией, `runtime.py` управляет отдельным стендом и телеметрией, `results.py` пересчитывает данные. Сам генератор запросов и SQL-набор данных используются из первого эксперимента. Команда `smoke` выполняет только короткую проверку инструментов и сохраняет её в `.memory`, отдельно от результатов исследования.
|
||||
@@ -0,0 +1,81 @@
|
||||
services:
|
||||
db:
|
||||
image: postgres:16.10-bookworm
|
||||
cpus: ${BENCH_DB_CPU:-1}
|
||||
mem_limit: 512m
|
||||
command: [postgres, -c, track_io_timing=on, -c, track_wal_io_timing=on]
|
||||
environment:
|
||||
POSTGRES_DB: benchmark
|
||||
POSTGRES_USER: benchmark
|
||||
POSTGRES_PASSWORD: ${BENCH_PASSWORD:?private environment required}
|
||||
volumes: [bench_db:/var/lib/postgresql/data]
|
||||
networks: [data]
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'pg_isready -U benchmark -d benchmark']
|
||||
interval: 1s
|
||||
timeout: 3s
|
||||
retries: 60
|
||||
redis:
|
||||
image: redis:7.4.11-bookworm
|
||||
cpus: 0.25
|
||||
mem_limit: 128m
|
||||
command: [redis-server, --appendonly, 'yes', --appendfsync, everysec, --dir, /data]
|
||||
volumes: [bench_redis:/data]
|
||||
networks: [data]
|
||||
healthcheck:
|
||||
test: [CMD, redis-cli, ping]
|
||||
interval: 1s
|
||||
timeout: 3s
|
||||
retries: 60
|
||||
migrate:
|
||||
image: ${BENCH_IMAGE:?prepared image required}
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
networks: [data]
|
||||
environment:
|
||||
DATABASE_URL: postgres://benchmark:${BENCH_PASSWORD}@db:5432/benchmark?sslmode=disable
|
||||
command: [./messenger, migrate, up]
|
||||
depends_on:
|
||||
db: {condition: service_healthy}
|
||||
api:
|
||||
image: ${BENCH_IMAGE:?prepared image required}
|
||||
cpus: ${BENCH_CPU:-0.5}
|
||||
mem_limit: ${BENCH_MEMORY:-128m}
|
||||
deploy:
|
||||
replicas: ${BENCH_REPLICAS:-2}
|
||||
stop_grace_period: 15s
|
||||
networks: [proxy, data]
|
||||
environment:
|
||||
DATABASE_URL: postgres://benchmark:${BENCH_PASSWORD}@db:5432/benchmark?sslmode=disable
|
||||
DATABASE_MAX_CONNS: ${BENCH_POOL:-6}
|
||||
BENCH_POOL_STATS: '1'
|
||||
GOMAXPROCS: ${BENCH_GOMAXPROCS:-16}
|
||||
REDIS_ADDR: redis:6379
|
||||
SESSION_TTL_SECONDS: 86400
|
||||
APP_PORT: 8080
|
||||
APP_HOST: 0.0.0.0
|
||||
healthcheck:
|
||||
test: [CMD, ./messenger, healthcheck]
|
||||
interval: 1s
|
||||
timeout: 3s
|
||||
retries: 60
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
redis: {condition: service_healthy}
|
||||
haproxy:
|
||||
image: haproxy:3.2.25-alpine
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
ports: ['127.0.0.1:18082:8080', '127.0.0.1:18406:8404']
|
||||
networks: [edge, proxy]
|
||||
volumes:
|
||||
- ./experiments/lab3-scale/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
depends_on:
|
||||
api: {condition: service_healthy}
|
||||
volumes:
|
||||
bench_db:
|
||||
bench_redis:
|
||||
networks:
|
||||
edge:
|
||||
proxy: {internal: true}
|
||||
data: {internal: true}
|
||||
@@ -0,0 +1,134 @@
|
||||
"""Pure, deterministic rules for the preregistered v2 experiment."""
|
||||
import random
|
||||
import statistics
|
||||
|
||||
CONFIGS = {
|
||||
"A1": {"replicas": 1, "cpu": 1, "memory": "256m", "pool": 12},
|
||||
"A2": {"replicas": 2, "cpu": .5, "memory": "128m", "pool": 6},
|
||||
"A4": {"replicas": 4, "cpu": .25, "memory": "64m", "pool": 3},
|
||||
"B1": {"replicas": 1, "cpu": .5, "memory": "128m", "pool": 12},
|
||||
"B2": {"replicas": 2, "cpu": .5, "memory": "128m", "pool": 6},
|
||||
"B4": {"replicas": 4, "cpu": .5, "memory": "128m", "pool": 3},
|
||||
}
|
||||
DESIGN = {
|
||||
"version": "v2.0",
|
||||
"hypotheses": {
|
||||
"H1": "Fixed total API budget: C4/C1 < 1.20 in all three blocks",
|
||||
"H2": "Growing API budget: C4/C1 >= 1.50 in all three blocks",
|
||||
"G_low": "At 25% of pilot A4 capacity, GOMAXPROCS=1 reduces p95 >=15% in all five pairs, retaining success and SLO",
|
||||
"G_high": "At 90% of pilot A4 capacity, the same criteria hold in all five pairs",
|
||||
"DB_CPU": "At the pilot B4 upper rate, increasing only DB CPU 1->2 restores SLO in all three blocks, with CPU-DB diagnostic prerequisites",
|
||||
},
|
||||
"duration_s": 30, "warmup_s": 15, "cooldown_s": 10,
|
||||
"timeout_s": 5, "workload_seed": 230300, "inflight_limit": 8192,
|
||||
"success_min": .99, "p95_max_ms": 500,
|
||||
"workload": "80% new idempotent POST messages (128 ASCII bytes), 20% first history page GET limit=50",
|
||||
"fixture": {"users": 200, "chats": 100, "messages": 10000},
|
||||
"dependencies": {"db_cpu": 1, "db_memory": "512m", "redis_cpu": .25, "redis_memory": "128m", "lb_cpu": .5, "lb_memory": "128m"},
|
||||
"pool_total": 12, "control_gomaxprocs": 16, "tuned_gomaxprocs": 1,
|
||||
"search": {"start": 100, "floor": 25, "ceiling": 6400, "max_points": 10, "relative_width": .10},
|
||||
"scaling_repeats": 3, "gomax_pairs": 5, "db_pairs": 3,
|
||||
"calibration_seed": 230700, "scaling_seeds": [230701, 230702, 230703],
|
||||
"gomax_order_seed": 230816, "db_order_seed": 230817,
|
||||
"budget_seconds": 9000,
|
||||
"generator_invalid": "dropped/scheduled >0.01 OR (dispatch p95 >100ms AND generator CPU >=0.85 cores)",
|
||||
"db_cpu_prerequisites": {"db_cpu_min": .90, "api_cpu_max": .85, "lb_cpu_max": .90},
|
||||
"aggregation": "Keep all pairs. Mean/median/range of paired percent reductions are descriptive, not a pooled p95. Mixed outcomes are inconclusive.",
|
||||
"censoring": "No failed upper point means only a lower bound, never an exact capacity. Invalid generator stops the experiment; bad API performance and OOM are retained.",
|
||||
"pilot_excluded_from_confirmation": True,
|
||||
}
|
||||
|
||||
|
||||
def config(name, gomax=16, db_cpu=1):
|
||||
return {**CONFIGS[name], "gomaxprocs": gomax, "db_cpu": db_cpu}
|
||||
|
||||
|
||||
def next_rate(observations, rules=None):
|
||||
rules = rules or DESIGN["search"]
|
||||
if any(not row["valid"] for row in observations):
|
||||
return None
|
||||
if not observations:
|
||||
return rules["start"]
|
||||
if len(observations) >= rules["max_points"]:
|
||||
return None
|
||||
passed = [row["rate"] for row in observations if row["slo_pass"]]
|
||||
failed = [row["rate"] for row in observations if not row["slo_pass"]]
|
||||
if passed and failed and max(passed) >= min(failed):
|
||||
return None
|
||||
if not passed:
|
||||
value = max(rules["floor"], min(failed) // 2)
|
||||
return value if value < min(failed) else None
|
||||
lower = max(passed)
|
||||
if not failed:
|
||||
value = min(lower * 2, rules["ceiling"])
|
||||
return value if value > lower else None
|
||||
upper = min(failed)
|
||||
if (upper - lower) / lower <= rules["relative_width"] or upper - lower <= 1:
|
||||
return None
|
||||
return (lower + upper) // 2
|
||||
|
||||
|
||||
def bracket(observations):
|
||||
passed = [row["rate"] for row in observations if row["valid"] and row["slo_pass"]]
|
||||
failed = [row["rate"] for row in observations if row["valid"] and not row["slo_pass"]]
|
||||
lower = max(passed, default=0)
|
||||
upper = min(failed) if failed else None
|
||||
invalid = any(not row["valid"] for row in observations)
|
||||
nonmonotonic = upper is not None and lower >= upper
|
||||
return {"lower": lower, "upper": upper, "invalid": invalid, "nonmonotonic": nonmonotonic,
|
||||
"closed": bool(lower and upper and not invalid and not nonmonotonic)}
|
||||
|
||||
|
||||
def freeze_confirmation(calibration):
|
||||
bounds = {name: bracket(rows) for name, rows in calibration.items()}
|
||||
rates = {}
|
||||
for name, value in bounds.items():
|
||||
candidates = [100, value["lower"], value["upper"]]
|
||||
rates[name] = sorted({rate for rate in candidates if rate and rate <= DESIGN["search"]["ceiling"]})
|
||||
scaling = []
|
||||
for repeat, seed in enumerate(DESIGN["scaling_seeds"], 1):
|
||||
points = [{"phase": "scaling", "base": name, "rate": rate, "repeat": repeat, "variant": name,
|
||||
"config": config(name)} for name in CONFIGS for rate in rates[name]]
|
||||
random.Random(seed).shuffle(points)
|
||||
scaling.extend(points)
|
||||
gomax = []
|
||||
if bounds["A4"]["closed"]:
|
||||
groups = []
|
||||
for regime, fraction in (("low", .25), ("high", .90)):
|
||||
rate = max(1, int(bounds["A4"]["lower"] * fraction))
|
||||
for repeat in range(1, DESIGN["gomax_pairs"] + 1):
|
||||
order = [16, 1] if repeat % 2 else [1, 16]
|
||||
groups.append([{"phase": "gomax", "base": "A4", "regime": regime, "rate": rate,
|
||||
"repeat": repeat, "variant": "I0" if value == 16 else "I1",
|
||||
"config": config("A4", gomax=value)} for value in order])
|
||||
random.Random(DESIGN["gomax_order_seed"]).shuffle(groups)
|
||||
gomax = [point for group in groups for point in group]
|
||||
database = []
|
||||
if bounds["B4"]["closed"]:
|
||||
groups = []
|
||||
for repeat in range(1, DESIGN["db_pairs"] + 1):
|
||||
for rate in (bounds["B4"]["lower"], bounds["B4"]["upper"]):
|
||||
order = [1, 2] if repeat % 2 else [2, 1]
|
||||
groups.append([{"phase": "database", "base": "B4", "rate": rate, "repeat": repeat,
|
||||
"variant": "D0" if cpu == 1 else "D1", "config": config("B4", db_cpu=cpu)} for cpu in order])
|
||||
random.Random(DESIGN["db_order_seed"]).shuffle(groups)
|
||||
database = [point for group in groups for point in group]
|
||||
return {"pilot_bounds": bounds, "rates_by_config": rates, "scaling": scaling,
|
||||
"database": database, "gomax": gomax,
|
||||
"skipped": {"gomax": None if gomax else "A4 pilot bracket not closed",
|
||||
"database": None if database else "B4 pilot bracket not closed"}}
|
||||
|
||||
|
||||
def pair_decision(pairs, expected=5):
|
||||
if len(pairs) != expected or any(not pair["valid"] for pair in pairs):
|
||||
result = "inconclusive"
|
||||
elif all(pair["criteria_met"] for pair in pairs):
|
||||
result = "supported"
|
||||
elif all(not pair["criteria_met"] for pair in pairs):
|
||||
result = "refuted"
|
||||
else:
|
||||
result = "inconclusive"
|
||||
values = [pair["reduction_percent"] for pair in pairs if pair["reduction_percent"] is not None]
|
||||
return {"result": result, "pairs": pairs, "reduction_mean_percent": statistics.mean(values) if values else None,
|
||||
"reduction_median_percent": statistics.median(values) if values else None,
|
||||
"reduction_range_percent": [min(values), max(values)] if values else None}
|
||||
@@ -0,0 +1,210 @@
|
||||
"""Recompute v2 results from individual requests; never touch v1 evidence."""
|
||||
import csv
|
||||
import gzip
|
||||
import json
|
||||
from collections import defaultdict
|
||||
|
||||
from design import CONFIGS, bracket, pair_decision
|
||||
from runtime import digest, legacy_analysis, save, workload
|
||||
|
||||
|
||||
def read(path):
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def jsonl(path):
|
||||
with gzip.open(path, "rt", encoding="utf-8") as stream:
|
||||
return [json.loads(line) for line in stream]
|
||||
|
||||
|
||||
def audit(directory, manifest, design):
|
||||
summary = read(directory / "summary.json")
|
||||
rows = jsonl(directory / "requests.jsonl.gz")
|
||||
successes = [row for row in rows if row["success"]]
|
||||
sent = [row for row in rows if "start_s" in row]
|
||||
duration = design["duration_s"]
|
||||
calculated = {
|
||||
"scheduled": len(rows), "sent": len(sent), "successes": len(successes),
|
||||
"errors": len(rows) - len(successes), "generator_dropped": len(rows) - len(sent),
|
||||
"success_ratio": len(successes) / max(1, len(rows)),
|
||||
"goodput_rps": len(successes) / duration,
|
||||
"window_goodput_rps": sum(row["end_s"] <= duration for row in successes) / duration,
|
||||
"p95_ms": workload.percentile([row["latency_ms"] for row in successes], .95),
|
||||
"dispatch_p95_ms": workload.percentile([row["dispatch_lag_ms"] for row in sent], .95),
|
||||
}
|
||||
mismatches = [key for key, value in calculated.items() if summary.get(key) != value]
|
||||
if len(rows) != round(manifest["rate"] * duration):
|
||||
mismatches.append("expected_request_count")
|
||||
if manifest["protocol_sha256"] != digest(design):
|
||||
mismatches.append("protocol_sha256")
|
||||
if len({row["index"] for row in rows}) != len(rows):
|
||||
mismatches.append("unique_request_index")
|
||||
valid = summary["valid"] and not mismatches
|
||||
slo = valid and summary["correctness"]["passed"] and calculated["success_ratio"] >= .99 and calculated["p95_ms"] is not None and calculated["p95_ms"] <= 500
|
||||
diagnostic = legacy_analysis.diagnostics(directory, manifest)
|
||||
return {**manifest, **summary, **calculated, "valid": valid, "slo_pass": slo,
|
||||
"audit_mismatches": mismatches, "diagnostics": diagnostic,
|
||||
"database_counters": database_counters(jsonl(directory / "telemetry.jsonl.gz"))}
|
||||
|
||||
|
||||
def database_counters(samples):
|
||||
values = [row["postgres"] for row in samples if "postgres" in row]
|
||||
if len(values) < 2:
|
||||
return {"available": False}
|
||||
first, last = values[0], values[-1]
|
||||
result = {"available": True, "note": "First-to-last sampled counters, not the full request window; waits are sample counts, not wait durations."}
|
||||
for section, keys in (("wal", ("wal_records", "wal_bytes", "wal_write", "wal_sync", "wal_write_time", "wal_sync_time")),
|
||||
("database", ("xact_commit", "xact_rollback", "blks_read", "blks_hit", "deadlocks"))):
|
||||
reset = first[section].get("stats_reset") != last[section].get("stats_reset")
|
||||
delta = {key: float(last[section][key]) - float(first[section][key]) for key in keys}
|
||||
result[section] = {"reset_detected": reset or any(value < 0 for value in delta.values()), "delta": delta}
|
||||
io = {}
|
||||
old = {(row["object"], row["context"]): row for row in first["io"]}
|
||||
for row in last["io"]:
|
||||
prior = old.get((row["object"], row["context"]))
|
||||
if prior:
|
||||
keys = ("reads", "read_time", "writes", "write_time", "extends", "extend_time", "fsyncs", "fsync_time")
|
||||
delta = {key: float(row[key] or 0) - float(prior[key] or 0) for key in keys}
|
||||
io[row["object"] + "/" + row["context"]] = {"delta": delta, "reset_detected": row.get("stats_reset") != prior.get("stats_reset") or any(v < 0 for v in delta.values())}
|
||||
result["io"] = io
|
||||
return result
|
||||
|
||||
|
||||
def capacities(runs, schedule):
|
||||
result = []
|
||||
for name in CONFIGS:
|
||||
for repeat in (1, 2, 3):
|
||||
points = [row for row in runs if row["phase"] == "scaling" and row["base"] == name and row["repeat"] == repeat]
|
||||
bound = bracket(points)
|
||||
expected = len([row for row in schedule if row["base"] == name and row["repeat"] == repeat])
|
||||
incomplete = len(points) != expected or not expected
|
||||
result.append({"config": name, "repeat": repeat, "capacity_lower": bound["lower"],
|
||||
"capacity_upper": bound["upper"], "nonmonotonic": bound["nonmonotonic"],
|
||||
"invalid_rates": [row["rate"] for row in points if not row["valid"]] + (["missing"] if incomplete else [])})
|
||||
return result
|
||||
|
||||
|
||||
def gomax_results(runs):
|
||||
result = {}
|
||||
for regime in ("low", "high"):
|
||||
pairs = []
|
||||
for repeat in range(1, 6):
|
||||
values = {row["variant"]: row for row in runs if row["phase"] == "gomax" and row["regime"] == regime and row["repeat"] == repeat}
|
||||
if "I0" not in values or "I1" not in values:
|
||||
continue
|
||||
control, tuned = values["I0"], values["I1"]
|
||||
numeric = control["p95_ms"] is not None and control["p95_ms"] > 0 and tuned["p95_ms"] is not None
|
||||
reduction = 100 * (1 - tuned["p95_ms"] / control["p95_ms"]) if numeric else None
|
||||
valid = numeric and control["valid"] and tuned["valid"] and control["rate"] == tuned["rate"] and control["correctness"]["passed"] and tuned["correctness"]["passed"]
|
||||
met = numeric and tuned["p95_ms"] <= .85 * control["p95_ms"] and tuned["p95_ms"] <= 500 and tuned["success_ratio"] >= max(.99, control["success_ratio"])
|
||||
pairs.append({"repeat": repeat, "rate": control["rate"], "control_p95_ms": control["p95_ms"], "tuned_p95_ms": tuned["p95_ms"],
|
||||
"valid": bool(valid), "criteria_met": bool(met), "reduction_percent": reduction})
|
||||
result["G_" + regime] = pair_decision(pairs)
|
||||
return result
|
||||
|
||||
|
||||
def db_results(runs, confirmation):
|
||||
bounds = confirmation.get("pilot_bounds", {}).get("B4", {})
|
||||
lower, upper = bounds.get("lower"), bounds.get("upper")
|
||||
blocks = []
|
||||
for repeat in (1, 2, 3):
|
||||
values = {(row["variant"], row["rate"]): row for row in runs if row["phase"] == "database" and row["repeat"] == repeat}
|
||||
required = [(variant, rate) for variant in ("D0", "D1") for rate in (lower, upper)]
|
||||
if not lower or not upper or any(key not in values for key in required):
|
||||
continue
|
||||
dl, du, tl, tu = (values[key] for key in (("D0", lower), ("D0", upper), ("D1", lower), ("D1", upper)))
|
||||
cpu = du["diagnostics"]["cpu_fraction_of_limit_mean"]
|
||||
eligible = cpu.get("db", -1) >= .90 and 0 <= cpu.get("api", -1) < .85 and 0 <= cpu.get("haproxy", -1) < .90
|
||||
valid = all(values[key]["valid"] and values[key]["correctness"]["passed"] for key in required)
|
||||
restored = dl["slo_pass"] and tl["slo_pass"] and not du["slo_pass"] and tu["slo_pass"]
|
||||
blocks.append({"repeat": repeat, "valid": valid, "cpu_prerequisites_met": eligible,
|
||||
"slo_restored": restored, "baseline_upper_cpu": cpu})
|
||||
supported = len(blocks) == 3 and all(row["valid"] and row["cpu_prerequisites_met"] and row["slo_restored"] for row in blocks)
|
||||
return {"result": "supported" if supported else "inconclusive", "blocks": blocks,
|
||||
"note": "Doubling CPU tests a CPU bottleneck only. No effect does not disprove a WAL, storage, lock or pool bottleneck."}
|
||||
|
||||
|
||||
def plots(directory, caps, hypotheses, runs):
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
fig, ax = plt.subplots(figsize=(10, 4))
|
||||
for index, row in enumerate(caps):
|
||||
lower, upper = row["capacity_lower"], row["capacity_upper"]
|
||||
ax.plot(index, lower, "o", color="tab:blue")
|
||||
if upper is not None:
|
||||
ax.plot([index, index], [lower, upper], color="tab:blue")
|
||||
ax.plot(index, upper, "x", color="tab:red")
|
||||
else:
|
||||
ax.annotate("≥", (index, lower), xytext=(0, 7), textcoords="offset points", ha="center")
|
||||
ax.set_xticks(range(len(caps)), [r["config"] + "/" + str(r["repeat"]) for r in caps], rotation=60)
|
||||
ax.set(ylabel="Offered requests/s", title="Confirmation capacity brackets (not confidence intervals)")
|
||||
ax.grid(axis="y", alpha=.2)
|
||||
fig.tight_layout(); fig.savefig(directory / "capacity.png", dpi=160); plt.close(fig)
|
||||
fig, axes = plt.subplots(1, 2, figsize=(9, 4))
|
||||
for ax, regime in zip(axes, ("low", "high")):
|
||||
for pair in hypotheses["G_" + regime]["pairs"]:
|
||||
ax.plot([0, 1], [pair["control_p95_ms"], pair["tuned_p95_ms"]], "o-", label="pair " + str(pair["repeat"]))
|
||||
ax.set_xticks([0, 1], ["GOMAXPROCS=16", "GOMAXPROCS=1"])
|
||||
ax.set(title=regime, ylabel="Successful request p95, ms")
|
||||
ax.grid(alpha=.2)
|
||||
fig.tight_layout(); fig.savefig(directory / "gomax.png", dpi=160); plt.close(fig)
|
||||
points = [row for row in runs if row["phase"] == "database"]
|
||||
if points:
|
||||
fig, ax = plt.subplots(figsize=(8, 4))
|
||||
for variant in ("D0", "D1"):
|
||||
group = [row for row in points if row["variant"] == variant]
|
||||
ax.scatter([r["rate"] for r in group], [r["p95_ms"] for r in group], label=variant)
|
||||
ax.axhline(500, color="red", linestyle="--", label="p95 SLO (also success ≥99%)")
|
||||
ax.set(xlabel="Offered requests/s", ylabel="Successful request p95, ms")
|
||||
ax.legend(); fig.tight_layout(); fig.savefig(directory / "database.png", dpi=160); plt.close(fig)
|
||||
|
||||
|
||||
def analyze(directory):
|
||||
design = read(directory / "design.json")
|
||||
if digest(design) != (directory / "design.sha256").read_text().strip():
|
||||
raise ValueError("Design checksum mismatch")
|
||||
confirmation = read(directory / "confirmation.json") if (directory / "confirmation.json").exists() else {}
|
||||
if confirmation and (digest(confirmation) != (directory / "confirmation.sha256").read_text().strip() or confirmation["design_sha256"] != digest(design)):
|
||||
raise ValueError("Confirmation schedule checksum mismatch")
|
||||
runs, incomplete = [], []
|
||||
for item in read(directory / "execution-order.json") if (directory / "execution-order.json").exists() else []:
|
||||
target = directory / item["directory"]
|
||||
if not (target / "summary.json").exists():
|
||||
incomplete.append(item["directory"])
|
||||
continue
|
||||
manifest = read(target / "manifest.json")
|
||||
run = audit(target, manifest, design)
|
||||
if any(manifest.get(key) != item.get(key) for key in ("phase", "base", "rate", "repeat", "variant", "config", "sequence")):
|
||||
run.update(valid=False, slo_pass=False)
|
||||
run["audit_mismatches"].append("execution_order")
|
||||
runs.append(run)
|
||||
caps = capacities(runs, confirmation.get("scaling", []))
|
||||
hypotheses = {**legacy_analysis.hypothesis_decisions(caps), **gomax_results(runs), "DB_CPU": db_results(runs, confirmation)}
|
||||
if design["smoke_only"]:
|
||||
hypotheses = {name: {"result": "not_tested_smoke_only"} for name in hypotheses}
|
||||
result = {"series": design["series"], "smoke_only": design["smoke_only"], "status": read(directory / "status.json"),
|
||||
"incomplete_runs": incomplete, "hypotheses": hypotheses, "capacities": caps, "runs": runs}
|
||||
save(directory / "analysis.json", result)
|
||||
keys = ["phase", "base", "variant", "repeat", "rate", "scheduled", "sent", "successes", "errors", "generator_dropped",
|
||||
"success_ratio", "goodput_rps", "window_goodput_rps", "p95_ms", "dispatch_p95_ms", "valid", "slo_pass"]
|
||||
with (directory / "summary.csv").open("w", newline="", encoding="utf-8") as stream:
|
||||
writer = csv.DictWriter(stream, fieldnames=keys, extrasaction="ignore")
|
||||
writer.writeheader(); writer.writerows(runs)
|
||||
lines = ["# Результаты " + design["series"], "", "Статус: `" + result["status"]["status"] + "`. Завершённых измерений: " + str(len(runs)) + ".", ""]
|
||||
if design["smoke_only"]:
|
||||
lines += ["Короткая проверка инструментов. Эти данные не проверяют гипотезы и не входят в исследование.", ""]
|
||||
else:
|
||||
lines += ["| Гипотеза | Результат |", "| --- | --- |"] + [f"| {name} | {value['result']} |" for name, value in hypotheses.items()]
|
||||
lines += ["", "supported — критерий выполнен во всех повторах; refuted — во всех парах получен противоположный результат; inconclusive — данных недостаточно или повторы расходятся.", "",
|
||||
"[Сводная таблица](summary.csv), [полный расчёт и диагностика](analysis.json), [зафиксированная методика](design.json).",
|
||||
"", "Пилотные измерения выбирают нагрузку и не входят в подтверждающую серию. Пределы мощности — границы на проверенной сетке, а не доверительные интервалы. Если верхняя граница не найдена, известно только C ≥ нижней границы.", "",
|
||||
"GOMAXPROCS оценивается по отдельным парам. Среднее, медиана и диапазон процентов снижения p95 приведены в analysis.json; p95 разных прогонов не объединяются. Среднее не заменяет критерий пяти пар.", "",
|
||||
"Счётчики WAL/I/O отражают разницу между первым и последним образцом телеметрии. Частота наблюдения ожиданий не равна времени ожидания. Удвоение CPU БД проверяет только ограничение CPU; отсутствие эффекта не исключает ограничения диска, WAL или блокировок.", "",
|
||||
"", "", ""]
|
||||
if any(row["phase"] == "database" for row in runs):
|
||||
lines += ["", ""]
|
||||
plots(directory, caps, hypotheses, runs)
|
||||
(directory / "RESULTS.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
print("Analysis: " + str(directory / "RESULTS.md"), flush=True)
|
||||
return result
|
||||
@@ -0,0 +1,16 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/../.."
|
||||
action=${1:-run}
|
||||
series=${2:-scale-check-v2}
|
||||
python=.venv/bin/python
|
||||
requirements=experiments/lab3-scale/requirements-lock.txt
|
||||
if [ ! -x "$python" ]; then
|
||||
python3 -m venv .venv
|
||||
fi
|
||||
if ! cmp -s "$requirements" .venv/lab3-scale-v2-requirements.txt; then
|
||||
"$python" -m pip install -r "$requirements"
|
||||
cp "$requirements" .venv/lab3-scale-v2-requirements.txt
|
||||
fi
|
||||
export PYTHONDONTWRITEBYTECODE=1
|
||||
exec "$python" -u experiments/lab3-scale-v2/runner.py "$action" --series "$series"
|
||||
@@ -0,0 +1,213 @@
|
||||
"""Prepare once; the user starts the long preregistered experiment explicitly."""
|
||||
import argparse
|
||||
import copy
|
||||
import fcntl
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
import sys
|
||||
import tarfile
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from design import CONFIGS, DESIGN, config, freeze_confirmation, next_rate
|
||||
from runtime import HERE, ROOT, Runtime, check_ports, digest, environment, original, save, shell, sources
|
||||
|
||||
|
||||
def validate_name(name):
|
||||
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_-]{0,63}-v2", name):
|
||||
raise ValueError("Series name must contain letters/digits/_/- and end with -v2")
|
||||
return name
|
||||
|
||||
|
||||
def directory_for(name, smoke=False):
|
||||
validate_name(name)
|
||||
return ROOT / (".memory/lab3-scale-v2-smoke" if smoke else "evidence/grade5-v2") / name
|
||||
|
||||
|
||||
def prepare(name, smoke=False):
|
||||
directory = directory_for(name, smoke)
|
||||
fingerprint = sources()
|
||||
if directory.exists():
|
||||
existing = json.loads((directory / "design.json").read_text())
|
||||
if existing["sources"] != fingerprint:
|
||||
raise RuntimeError("Prepared sources changed; choose a new series ending in -v2")
|
||||
if digest(existing) != (directory / "design.sha256").read_text().strip():
|
||||
raise RuntimeError("Prepared design checksum differs")
|
||||
if json.loads((directory / "status.json").read_text())["status"] != "prepared":
|
||||
raise RuntimeError("Series already started or preparation was incomplete; choose a new series")
|
||||
print("Already prepared: " + str(directory), flush=True)
|
||||
return directory
|
||||
check_ports()
|
||||
env = environment()
|
||||
image = "highload-lab3-bench-v2:" + env["app_code_sha256"][:16]
|
||||
print("Building one image for every v2 condition", flush=True)
|
||||
shell(["docker", "build", "-t", image, "src"])
|
||||
image_id = shell(["docker", "image", "inspect", "--format", "{{.Id}}", image])
|
||||
design = {**copy.deepcopy(DESIGN), "series": name, "smoke_only": smoke,
|
||||
"sources": fingerprint, "image": image, "image_id": image_id,
|
||||
"frozen_utc": datetime.now(timezone.utc).isoformat()}
|
||||
if smoke:
|
||||
design.update(duration_s=5, warmup_s=2, cooldown_s=1, budget_seconds=300)
|
||||
directory.mkdir(parents=True)
|
||||
save(directory / "design.json", design)
|
||||
(directory / "design.sha256").write_text(digest(design) + "\n")
|
||||
save(directory / "environment-prepared.json", env)
|
||||
with tarfile.open(directory / "source.tar.gz", "w:gz") as archive:
|
||||
for relative in fingerprint:
|
||||
archive.add(ROOT / relative, arcname=relative, recursive=False)
|
||||
save(directory / "status.json", {"status": "preparing", "long_experiment_started": False,
|
||||
"smoke_only": smoke})
|
||||
# Credentials remain outside evidence and the source archive.
|
||||
runtime = Runtime(name, image, env)
|
||||
runtime.compose(["config", "--quiet"])
|
||||
save(directory / "status.json", {"status": "prepared", "long_experiment_started": False,
|
||||
"smoke_only": smoke})
|
||||
print("PREPARED: " + str(directory), flush=True)
|
||||
return directory
|
||||
|
||||
|
||||
class StopExperiment(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class Experiment:
|
||||
def __init__(self, directory):
|
||||
self.directory = directory
|
||||
self.design = json.loads((directory / "design.json").read_text())
|
||||
if digest(self.design) != (directory / "design.sha256").read_text().strip():
|
||||
raise RuntimeError("Frozen design checksum differs")
|
||||
if self.design["sources"] != sources():
|
||||
raise RuntimeError("Code/instrument changed after preparation; choose a new series")
|
||||
current_image = shell(["docker", "image", "inspect", "--format", "{{.Id}}", self.design["image"]])
|
||||
if current_image != self.design["image_id"]:
|
||||
raise RuntimeError("Prepared Docker image changed")
|
||||
self.protocol_sha = digest(self.design)
|
||||
self.started = time.monotonic()
|
||||
self.sequence = 0
|
||||
self.runtime = Runtime(self.design["series"], self.design["image"], environment())
|
||||
self.order = []
|
||||
|
||||
def run_point(self, point):
|
||||
# Budget stops are explicit and keep every completed or partial run.
|
||||
needed = self.design["warmup_s"] + self.design["duration_s"] + self.design["cooldown_s"] + 15
|
||||
if time.monotonic() - self.started + needed > self.design["budget_seconds"]:
|
||||
raise StopExperiment("budget_exhausted")
|
||||
if self.design["sources"] != sources():
|
||||
raise StopExperiment("sources_changed_during_experiment")
|
||||
self.sequence += 1
|
||||
name = f"{self.sequence:03d}-{point['variant']}-r{point['repeat']}-rate{point['rate']}"
|
||||
if point.get("regime"):
|
||||
name += "-" + point["regime"]
|
||||
target = self.directory / point["phase"] / name
|
||||
self.order.append({"sequence": self.sequence, "directory": str(target.relative_to(self.directory)), **point})
|
||||
save(self.directory / "execution-order.json", self.order)
|
||||
save(self.directory / "status.json", {"status": "running", "current": self.order[-1],
|
||||
"elapsed_s": round(time.monotonic() - self.started, 1), "completed_runs": self.sequence - 1,
|
||||
"smoke_only": self.design["smoke_only"]})
|
||||
print(f"START {point['phase']} {name}", flush=True)
|
||||
result = self.runtime.execute(point, target, self.protocol_sha, self.design, self.sequence)
|
||||
if not result["valid"]:
|
||||
cause = "generator_limited" if result["generator_limited"] else "instrument_or_runtime_invalid"
|
||||
raise StopExperiment(cause)
|
||||
if not result["correctness"]["passed"]:
|
||||
raise StopExperiment("confirmed_data_not_preserved")
|
||||
time.sleep(self.design["cooldown_s"])
|
||||
return result
|
||||
|
||||
def execute(self):
|
||||
old = json.loads((self.directory / "status.json").read_text())
|
||||
if old["status"] != "prepared":
|
||||
raise RuntimeError("Series has already started; raw data are retained. Prepare a new series.")
|
||||
check_ports()
|
||||
status, error, restored = "failed", None, False
|
||||
try:
|
||||
self.runtime.suspend_main()
|
||||
save(self.directory / "environment-run.json", self.runtime.env_info)
|
||||
if self.design["smoke_only"]:
|
||||
for variant, settings in (("I0", config("A4", 16)), ("I1", config("A4", 1)), ("D1", config("A4", 16, 2))):
|
||||
result = self.run_point({"phase": "smoke", "base": "A4", "rate": 20, "repeat": 1,
|
||||
"variant": variant, "config": settings})
|
||||
if not result["slo_pass"]:
|
||||
raise StopExperiment("smoke_slo_not_met")
|
||||
status = "smoke_complete"
|
||||
else:
|
||||
calibration = {name: [] for name in CONFIGS}
|
||||
names = list(CONFIGS)
|
||||
random.Random(self.design["calibration_seed"]).shuffle(names)
|
||||
for name in names:
|
||||
while True:
|
||||
rate = next_rate(calibration[name], self.design["search"])
|
||||
if rate is None:
|
||||
break
|
||||
result = self.run_point({"phase": "calibration", "base": name, "variant": name,
|
||||
"rate": rate, "repeat": len(calibration[name]) + 1, "config": config(name)})
|
||||
calibration[name].append({"rate": rate, "valid": result["valid"], "slo_pass": result["slo_pass"]})
|
||||
save(self.directory / "calibration.json", calibration)
|
||||
confirmation = freeze_confirmation(calibration)
|
||||
confirmation["design_sha256"] = self.protocol_sha
|
||||
save(self.directory / "confirmation.json", confirmation)
|
||||
(self.directory / "confirmation.sha256").write_text(digest(confirmation) + "\n")
|
||||
# This schedule is frozen before seeing any confirmation outcome.
|
||||
for phase in ("scaling", "database", "gomax"):
|
||||
for point in confirmation[phase]:
|
||||
self.run_point(point)
|
||||
status = "complete"
|
||||
except StopExperiment as failure:
|
||||
status, error = str(failure), str(failure)
|
||||
except KeyboardInterrupt:
|
||||
status, error = "interrupted", "KeyboardInterrupt"
|
||||
except Exception as failure:
|
||||
status, error = "failed", str(failure)
|
||||
finally:
|
||||
restored = self.runtime.restore()
|
||||
save(self.directory / "restoration.json", json.loads((self.runtime.private / "restoration.json").read_text()))
|
||||
save(self.directory / "status.json", {"status": status, "error": error, "restoration_ok": restored,
|
||||
"elapsed_s": round(time.monotonic() - self.started, 1), "scheduled_or_started_runs": self.sequence,
|
||||
"smoke_only": self.design["smoke_only"]})
|
||||
print(f"FINISHED: {status}; original application restored={restored}", flush=True)
|
||||
from results import analyze
|
||||
analyze(self.directory)
|
||||
return 0 if status in ("complete", "smoke_complete") and restored else 1
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(__doc__)
|
||||
parser.add_argument("action", choices=("prepare", "run", "smoke", "status", "analyze", "restore"))
|
||||
parser.add_argument("--series", default="scale-check-v2")
|
||||
args = parser.parse_args()
|
||||
validate_name(args.series)
|
||||
smoke = args.action == "smoke"
|
||||
directory = directory_for(args.series, smoke)
|
||||
if args.action == "status":
|
||||
print((directory / "status.json").read_text())
|
||||
return 0
|
||||
if args.action == "analyze":
|
||||
from results import analyze
|
||||
analyze(directory)
|
||||
return 0
|
||||
lock_path = ROOT / ".memory/benchmark/experiment.lock"
|
||||
lock_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with lock_path.open("a") as lock:
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
parser.error("Another v1/v2 experiment is running")
|
||||
if args.action == "prepare":
|
||||
prepare(args.series)
|
||||
return 0
|
||||
if args.action == "restore":
|
||||
design = json.loads((directory / "design.json").read_text())
|
||||
return 0 if Runtime(args.series, design["image"], environment()).restore() else 1
|
||||
if smoke:
|
||||
prepare(args.series, smoke=True)
|
||||
return Experiment(directory).execute()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except Exception as error:
|
||||
print("ERROR: " + str(error), file=sys.stderr)
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,276 @@
|
||||
"""Isolated v2 Docker runtime; reuse the v1 workload and fixture unchanged."""
|
||||
import asyncio
|
||||
import csv
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import secrets
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
ROOT = HERE.parents[1]
|
||||
V1 = HERE.parent / "lab3-scale"
|
||||
sys.path.insert(0, str(V1))
|
||||
import common as original
|
||||
import load as workload
|
||||
import aiohttp
|
||||
|
||||
|
||||
def module(path, name):
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
result = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(result)
|
||||
return result
|
||||
|
||||
|
||||
legacy_analysis = module(V1 / "analyze.py", "scale_v1_analysis")
|
||||
|
||||
|
||||
def save(path, data):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def digest(data):
|
||||
return hashlib.sha256(json.dumps(data, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
|
||||
|
||||
|
||||
def sources():
|
||||
paths = [p for base in (ROOT / "src/backend", V1, HERE) for p in base.rglob("*")
|
||||
if p.is_file() and "__pycache__" not in p.parts and p.suffix.lower() != ".md"]
|
||||
paths += [ROOT / name for name in ("src/go.mod", "src/go.sum", "src/Dockerfile")]
|
||||
return {str(path.relative_to(ROOT)): hashlib.sha256(path.read_bytes()).hexdigest() for path in sorted(paths)}
|
||||
|
||||
|
||||
def shell(args):
|
||||
result = subprocess.run(args, cwd=ROOT, text=True, capture_output=True)
|
||||
if result.returncode:
|
||||
# Avoid echoing expanded connection strings or private Docker output.
|
||||
raise RuntimeError("Command failed: " + " ".join(args[:3]))
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def docker_endpoint():
|
||||
context = json.loads(shell(["docker", "context", "inspect"]))[0]
|
||||
endpoint = os.environ.get("DOCKER_HOST") or context["Endpoints"]["docker"]["Host"]
|
||||
if not endpoint.startswith("unix://"):
|
||||
raise RuntimeError("v2 telemetry requires a local unix:// Docker context (WSL/Linux/macOS)")
|
||||
path = endpoint.removeprefix("unix://")
|
||||
if not Path(path).exists():
|
||||
raise RuntimeError("Docker socket from the current context is unavailable")
|
||||
return path
|
||||
|
||||
|
||||
def check_ports():
|
||||
for port in (18082, 18406):
|
||||
with socket.socket() as probe:
|
||||
try:
|
||||
probe.bind(("127.0.0.1", port))
|
||||
except OSError as error:
|
||||
raise RuntimeError(f"Port {port} is occupied; no unrelated container will be stopped") from error
|
||||
|
||||
|
||||
def environment():
|
||||
info = json.loads(shell(["docker", "info", "--format",
|
||||
'{"cpus":{{.NCPU}},"memory":{{.MemTotal}},"architecture":{{json .Architecture}},"os":{{json .OperatingSystem}}}']))
|
||||
return {"time_utc": datetime.now(timezone.utc).isoformat(), "platform": platform.platform(),
|
||||
"python": platform.python_version(), "docker_resources": info,
|
||||
"docker_version": json.loads(shell(["docker", "version", "--format", "{{json .Server}}"])),
|
||||
"compose_version": shell(["docker", "compose", "version"]),
|
||||
"git_commit": shell(["git", "rev-parse", "HEAD"]),
|
||||
"worktree_files": shell(["git", "status", "--porcelain"]).splitlines(),
|
||||
"docker_socket": docker_endpoint(), "app_code_sha256": original.code_hash()}
|
||||
|
||||
|
||||
DB_SAMPLE = """SELECT json_build_object(
|
||||
'active',(SELECT count(*) FROM pg_stat_activity WHERE datname='benchmark' AND state='active' AND pid<>pg_backend_pid()),
|
||||
'waits',(SELECT coalesce(json_agg(json_build_object('event_type',wait_event_type,'event',wait_event,'count',n)),'[]'::json)
|
||||
FROM (SELECT wait_event_type,wait_event,count(*) n FROM pg_stat_activity WHERE datname='benchmark' AND pid<>pg_backend_pid() GROUP BY 1,2) t),
|
||||
'database',(SELECT row_to_json(d) FROM (SELECT xact_commit,xact_rollback,blks_read,blks_hit,tup_inserted,deadlocks FROM pg_stat_database WHERE datname='benchmark') d),
|
||||
'wal',(SELECT row_to_json(w) FROM pg_stat_wal w),
|
||||
'io',(SELECT coalesce(json_agg(i),'[]'::json) FROM pg_stat_io i WHERE backend_type='client backend')
|
||||
);"""
|
||||
|
||||
|
||||
class Runtime:
|
||||
def __init__(self, series, image, env_info):
|
||||
self.series = series
|
||||
self.private = ROOT / ".memory/lab3-scale-v2" / series
|
||||
self.private.mkdir(parents=True, exist_ok=True)
|
||||
self.project = "highload-lab3-v2-" + hashlib.sha256(series.encode()).hexdigest()[:10]
|
||||
self.image = image
|
||||
self.env_info = env_info
|
||||
self.socket = env_info["docker_socket"]
|
||||
self.db_cpu = None
|
||||
credentials = self.private / "environment"
|
||||
if not credentials.exists():
|
||||
credentials.write_text("BENCH_PASSWORD=" + secrets.token_hex(24) + "\n")
|
||||
credentials.chmod(0o600)
|
||||
# Only this process is adapted. No v1 source or historical evidence changes.
|
||||
original.compose = self.compose
|
||||
original.PROJECT = self.project
|
||||
workload.BASE = "http://127.0.0.1:18082"
|
||||
workload.STATS = "http://127.0.0.1:18406/stats;csv"
|
||||
workload.collect = self.collect
|
||||
|
||||
def compose(self, args, config=None):
|
||||
config = config or {"cpu": .5, "memory": "128m", "replicas": 2, "pool": 6,
|
||||
"gomaxprocs": 16, "db_cpu": 1}
|
||||
env = {**os.environ, "BENCH_CPU": str(config["cpu"]), "BENCH_MEMORY": config["memory"],
|
||||
"BENCH_REPLICAS": str(config["replicas"]), "BENCH_POOL": str(config["pool"]),
|
||||
"BENCH_GOMAXPROCS": str(config["gomaxprocs"]), "BENCH_DB_CPU": str(config["db_cpu"]),
|
||||
"BENCH_IMAGE": self.image}
|
||||
result = subprocess.run(["docker", "compose", "--project-name", self.project, "--project-directory", str(ROOT),
|
||||
"--env-file", str(self.private / "environment"), "-f", str(HERE / "compose.yml"), *args],
|
||||
cwd=ROOT, env=env, text=True, capture_output=True)
|
||||
if result.returncode:
|
||||
(self.private / "compose-error.log").write_text(result.stdout + result.stderr)
|
||||
raise RuntimeError("v2 Compose failed; see " + str(self.private / "compose-error.log"))
|
||||
return result.stdout.strip()
|
||||
|
||||
def suspend_main(self):
|
||||
ids = shell(["docker", "ps", "-q", "--filter", "label=com.docker.compose.project=highload-lab3"]).split()
|
||||
records = []
|
||||
for container in ids:
|
||||
data = json.loads(shell(["docker", "inspect", "--format",
|
||||
'{"id":{{json .Id}},"service":{{json (index .Config.Labels "com.docker.compose.service")}}}', container]))
|
||||
records.append(data)
|
||||
save(self.private / "original-state.json", records)
|
||||
if ids:
|
||||
shell(["docker", "stop", *ids])
|
||||
|
||||
def restore(self):
|
||||
errors = []
|
||||
try:
|
||||
self.compose(["stop"])
|
||||
except Exception as error:
|
||||
errors.append(str(error))
|
||||
path = self.private / "original-state.json"
|
||||
records = json.loads(path.read_text()) if path.exists() else []
|
||||
for group in (("db", "redis"), ("api",), ("haproxy", "frontend")):
|
||||
for record in records:
|
||||
if record["service"] not in group:
|
||||
continue
|
||||
try:
|
||||
project = shell(["docker", "inspect", "--format", '{{index .Config.Labels "com.docker.compose.project"}}', record["id"]])
|
||||
if project != "highload-lab3":
|
||||
raise RuntimeError("Original container identity changed")
|
||||
shell(["docker", "start", record["id"]])
|
||||
deadline = time.monotonic() + 60
|
||||
while time.monotonic() < deadline:
|
||||
state = json.loads(shell(["docker", "inspect", "--format", "{{json .State}}", record["id"]]))
|
||||
if state["Running"] and state.get("Health", {}).get("Status", "healthy") == "healthy":
|
||||
break
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise RuntimeError("Original container did not become healthy: " + record["service"])
|
||||
except Exception as error:
|
||||
errors.append(str(error))
|
||||
save(self.private / "restoration.json", {"ok": not errors, "errors": errors})
|
||||
return not errors
|
||||
|
||||
def prepare_services(self, config):
|
||||
if self.db_cpu != config["db_cpu"]:
|
||||
self.compose(["up", "-d", "--wait", "db", "redis"], config)
|
||||
# One explicit migration job, not migrations in each API.
|
||||
self.compose(["run", "--rm", "--no-deps", "migrate"], config)
|
||||
self.db_cpu = config["db_cpu"]
|
||||
self.compose(["up", "-d", "--no-deps", "--force-recreate", "--wait", "api"], config)
|
||||
self.compose(["up", "-d", "--no-deps", "haproxy"], config)
|
||||
|
||||
def execute(self, point, directory, protocol_sha, timings, sequence):
|
||||
if directory.exists():
|
||||
raise RuntimeError("Run already exists; raw measurements are never overwritten: " + str(directory))
|
||||
directory.mkdir(parents=True)
|
||||
config = point["config"]
|
||||
self.prepare_services(config)
|
||||
actors = original.restore_fixture()
|
||||
actual = original.current_containers()
|
||||
apis = [row for row in actual if row["service"] == "api"]
|
||||
for name, cpu, memory in (("api", config["cpu"], int(config["memory"].removesuffix("m")) * 1048576),
|
||||
("db", config["db_cpu"], 512 * 1048576), ("redis", .25, 128 * 1048576),
|
||||
("haproxy", .5, 128 * 1048576)):
|
||||
rows = [row for row in actual if row["service"] == name]
|
||||
if not rows or any(row["cpu"] != cpu * 1e9 or row["memory"] != memory for row in rows):
|
||||
raise RuntimeError("Effective limits differ from protocol: " + name)
|
||||
if len(apis) != config["replicas"] or any(row["ports"] for row in apis):
|
||||
raise RuntimeError("API replica count or exposed ports differ")
|
||||
manifest = {**point, "run_id": directory.name, "sequence": sequence, "protocol_sha256": protocol_sha,
|
||||
"code_sha256": original.code_hash(), "containers": actual,
|
||||
"docker_cpus": self.env_info["docker_resources"]["cpus"],
|
||||
"started_utc": datetime.now(timezone.utc).isoformat()}
|
||||
save(directory / "manifest.json", manifest)
|
||||
result = asyncio.run(workload.measure(actors, directory, config, rate=point["rate"],
|
||||
duration=timings["duration_s"], warmup=timings["warmup_s"], seed=230300,
|
||||
prefix=self.series + ":" + str(sequence), limit=8192))
|
||||
samples = json.loads((directory / "pool.json").read_text())
|
||||
expected = {row["instance"] for row in apis}
|
||||
observed = {row["instance"] for row in samples}
|
||||
runtime_ok = observed == expected and all(row.get("gomaxprocs") == config["gomaxprocs"] and row["max"] == config["pool"] for row in samples)
|
||||
result["runtime_settings_verified"] = runtime_ok
|
||||
result["valid"] = result["valid"] and runtime_ok
|
||||
result["slo_pass"] = result["slo_pass"] and result["valid"]
|
||||
result["rate"] = point["rate"]
|
||||
save(directory / "summary.json", result)
|
||||
diagnostic = legacy_analysis.diagnostics(directory, manifest)
|
||||
save(directory / "diagnostics.json", diagnostic)
|
||||
print(json.dumps({"run": directory.name, "rate": point["rate"], "p95_ms": result["p95_ms"],
|
||||
"success": result["success_ratio"], "valid": result["valid"], "slo_pass": result["slo_pass"]}), flush=True)
|
||||
return result
|
||||
|
||||
async def sample_db(self, container):
|
||||
process = await asyncio.create_subprocess_exec("docker", "exec", "-i", "-e", "PGOPTIONS=-c statement_timeout=2000", container,
|
||||
"psql", "-X", "-q", "-t", "-A", "-v", "ON_ERROR_STOP=1", "-U", "benchmark", "-d", "benchmark",
|
||||
stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
|
||||
try:
|
||||
stdout, _ = await asyncio.wait_for(process.communicate(DB_SAMPLE.encode()), 4)
|
||||
except BaseException:
|
||||
if process.returncode is None:
|
||||
process.kill()
|
||||
await process.communicate()
|
||||
raise
|
||||
if process.returncode:
|
||||
raise RuntimeError("DB telemetry unavailable")
|
||||
return json.loads(stdout)
|
||||
|
||||
async def collect(self, stop, containers, rows):
|
||||
db = next(row["id"] for row in containers if row["service"] == "db")
|
||||
async with aiohttp.ClientSession(connector=aiohttp.UnixConnector(path=self.socket), timeout=aiohttp.ClientTimeout(total=4)) as engine, \
|
||||
aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=4)) as http:
|
||||
async def stats(container):
|
||||
async with engine.get(f"http://docker/v1.47/containers/{container['id']}/stats?stream=false&one-shot=true") as response:
|
||||
data = await response.json()
|
||||
cpu = dict(data.get("cpu_stats") or {})
|
||||
observed = cpu.get("online_cpus")
|
||||
cpu["online_cpus"] = observed or self.env_info["docker_resources"]["cpus"]
|
||||
return {"instance": container["instance"], "service": container["service"], "cpu_stats": cpu,
|
||||
"cpu_count_source": "stats" if observed else "docker_info", "memory_stats": data.get("memory_stats"),
|
||||
"blkio_stats": data.get("blkio_stats"), "stats_available": response.status == 200}
|
||||
while not stop.is_set():
|
||||
begin = time.monotonic()
|
||||
try:
|
||||
resources, database = await asyncio.gather(asyncio.gather(*(stats(row) for row in containers)), self.sample_db(db))
|
||||
haproxy, stats_error = [], None
|
||||
try:
|
||||
async with http.get(workload.STATS) as response:
|
||||
haproxy = [row for row in csv.DictReader(io.StringIO((await response.text()).removeprefix("# ")))
|
||||
if row["pxname"] in ("api_replicas", "api_entry")]
|
||||
except (aiohttp.ClientError, TimeoutError):
|
||||
stats_error = "unavailable"
|
||||
rows.append({"time": datetime.now(timezone.utc).isoformat(), "resources": resources,
|
||||
"postgres": database, "haproxy": haproxy, "haproxy_stats_error": stats_error,
|
||||
"generator_cpu_seconds": time.process_time(), "collection_seconds": time.monotonic() - begin})
|
||||
except Exception as error:
|
||||
rows.append({"time": datetime.now(timezone.utc).isoformat(), "error": type(error).__name__})
|
||||
try:
|
||||
await asyncio.wait_for(stop.wait(), max(.01, 1 - (time.monotonic() - begin)))
|
||||
except TimeoutError:
|
||||
pass
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Checks for scientific decision gates, censoring and paired design."""
|
||||
import unittest
|
||||
|
||||
from design import CONFIGS, bracket, config, freeze_confirmation, next_rate, pair_decision
|
||||
from results import capacities, db_results, gomax_results
|
||||
from runtime import legacy_analysis
|
||||
|
||||
|
||||
def observation(rate, passed=True, valid=True):
|
||||
return {"rate": rate, "slo_pass": passed, "valid": valid}
|
||||
|
||||
|
||||
class DesignTests(unittest.TestCase):
|
||||
def test_800_is_not_a_ceiling(self):
|
||||
points = [observation(rate) for rate in (100, 200, 400, 800)]
|
||||
self.assertEqual(next_rate(points), 1600)
|
||||
self.assertIsNone(bracket(points)["upper"])
|
||||
points += [observation(1600), observation(3200), observation(6400)]
|
||||
self.assertIsNone(next_rate(points))
|
||||
self.assertFalse(bracket(points)["closed"])
|
||||
|
||||
def test_search_floor_bisection_invalid_and_nonmonotonic(self):
|
||||
self.assertEqual(next_rate([observation(100, False)]), 50)
|
||||
self.assertIsNone(next_rate([observation(25, False)]))
|
||||
self.assertEqual(next_rate([observation(100), observation(200, False)]), 150)
|
||||
self.assertIsNone(next_rate([observation(100), observation(109, False)]))
|
||||
self.assertIsNone(next_rate([observation(100, valid=False)]))
|
||||
self.assertTrue(bracket([observation(100, False), observation(200)])["nonmonotonic"])
|
||||
|
||||
def test_confirmation_is_fresh_and_pairs_change_one_factor(self):
|
||||
pilot = {name: [observation(400), observation(440, False)] for name in CONFIGS}
|
||||
plan = freeze_confirmation(pilot)
|
||||
self.assertEqual(len(plan["scaling"]), 54)
|
||||
self.assertEqual(len(plan["gomax"]), 20)
|
||||
self.assertEqual(len(plan["database"]), 12)
|
||||
self.assertEqual(plan, freeze_confirmation(pilot))
|
||||
for phase, field in (("gomax", "gomaxprocs"), ("database", "db_cpu")):
|
||||
for index in range(0, len(plan[phase]), 2):
|
||||
a, b = plan[phase][index:index + 2]
|
||||
self.assertEqual(a["rate"], b["rate"])
|
||||
self.assertEqual(a["repeat"], b["repeat"])
|
||||
self.assertEqual({k: v for k, v in a["config"].items() if k != field}, {k: v for k, v in b["config"].items() if k != field})
|
||||
self.assertNotEqual(a["config"][field], b["config"][field])
|
||||
self.assertEqual({p["config"]["gomaxprocs"] for p in plan["gomax"]}, {1, 16})
|
||||
|
||||
def test_open_bracket_skips_causal_tests(self):
|
||||
plan = freeze_confirmation({name: [observation(800)] for name in CONFIGS})
|
||||
self.assertFalse(plan["gomax"])
|
||||
self.assertFalse(plan["database"])
|
||||
|
||||
def test_mean_does_not_hide_a_failed_pair(self):
|
||||
pairs = [{"valid": True, "criteria_met": effect >= 15, "reduction_percent": effect} for effect in (60, 60, 60, 60, 0)]
|
||||
self.assertEqual(pair_decision(pairs)["result"], "inconclusive")
|
||||
self.assertEqual(pair_decision(pairs)["reduction_mean_percent"], 48)
|
||||
self.assertEqual(pair_decision(pairs[:4])["result"], "inconclusive")
|
||||
self.assertEqual(pair_decision([dict(pairs[0]) for _ in range(5)])["result"], "supported")
|
||||
self.assertEqual(pair_decision([dict(pairs[-1]) for _ in range(5)])["result"], "refuted")
|
||||
pairs[0]["valid"] = False
|
||||
self.assertEqual(pair_decision(pairs)["result"], "inconclusive")
|
||||
|
||||
def test_missing_or_censored_scaling_is_inconclusive(self):
|
||||
runs = [{**observation(800), "phase": "scaling", "base": name, "repeat": repeat} for name in CONFIGS for repeat in (1, 2, 3)]
|
||||
caps = capacities(runs, runs)
|
||||
decisions = legacy_analysis.hypothesis_decisions(caps)
|
||||
self.assertEqual(decisions["H2"]["result"], "inconclusive")
|
||||
self.assertTrue(all(row["capacity_upper"] is None for row in caps))
|
||||
self.assertEqual(legacy_analysis.hypothesis_decisions(capacities(runs[:-1], runs))["H2"]["result"], "inconclusive")
|
||||
|
||||
def test_db_needs_cpu_evidence_not_just_faster_result(self):
|
||||
points = []
|
||||
for repeat in (1, 2, 3):
|
||||
for variant in ("D0", "D1"):
|
||||
for rate in (400, 440):
|
||||
points.append({"phase": "database", "variant": variant, "rate": rate, "repeat": repeat,
|
||||
"valid": True, "correctness": {"passed": True}, "slo_pass": rate == 400 or variant == "D1",
|
||||
"diagnostics": {"cpu_fraction_of_limit_mean": {"db": .95, "api": .6, "haproxy": .2}}})
|
||||
plan = {"pilot_bounds": {"B4": {"lower": 400, "upper": 440}}}
|
||||
self.assertEqual(db_results(points, plan)["result"], "supported")
|
||||
points[1]["diagnostics"]["cpu_fraction_of_limit_mean"]["db"] = .4
|
||||
self.assertEqual(db_results(points, plan)["result"], "inconclusive")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user