diff --git a/experiments/lab3-scale-v2/README.md b/experiments/lab3-scale-v2/README.md new file mode 100644 index 00000000..ffafbdbd --- /dev/null +++ b/experiments/lab3-scale-v2/README.md @@ -0,0 +1,60 @@ +# Масштабирование и GOMAXPROCS — v2 + +Это отдельная проверка замечаний к первому исследованию. Базовые конфигурации A1–B4, данные, запросы и SLO описаны в [исходной методике](../lab3-scale/docs/method.md). Ниже — только изменения. Старые результаты не заменяются. + +## Что меняется и почему + +1. **Предел нагрузки ищется на текущем компьютере.** Изначальный эксперимент определялся мощностью ПК, на котором запускался, из-за чего мощность самого приложения определялась некорректно. Теперь мы начинаем со 100 запросов/с и удваиваем нагрузку до первого нарушения SLO, максимум до 6400. Затем делим интервал пополам до ширины 10% или десяти точек на конфигурацию. Если уже 100 не выдерживается, снижаемся до 25. Это устраняет прежний потолок сетки 800: успешный запуск на 800 не означает предельную мощность 800. Если отказа не нашли, результат — только `C ≥ последнее успешное значение`. +2. **Подбор нагрузки отделён от проверки.** Пилот выбирает точки. После него программа сохраняет `confirmation.json` с контрольными нагрузками и порядком запусков, а затем проводит три новых блока для A1–B4. В каждом блоке проверяются 100 запросов/с и найденные нижняя и верхняя границы (без дубликатов). Гипотезы H1/H2 остаются прежними; интервалы отношений вычисляются только по новым блокам. Незамкнутая граница, пропуск или противоречие дают «данных недостаточно». +3. **Сравниваются явные `GOMAXPROCS=16` и `1`.** Для A4 выполняются пять пар при 25% и пять при 90% её пилотной нижней границы. Нагрузка внутри пары одинакова; порядок вариантов чередуется, порядок пар перемешан с фиксированным seed. Проверяем отдельно G_low и G_high: снижение p95 не менее 15%, p95 ≤500 мс, успешность ≥99% и не ниже контроля, подтверждённые сообщения сохранены. Критерий должен выполниться во всех пяти валидных парах. Пять отрицательных результатов опровергают гипотезу; смешанные результаты, пропуски или невалидные прогоны оставляют её неопределённой. Среднее, медиана и диапазон эффекта показываются дополнительно, без выбора удачных повторов. Если пилот A4 не нашёл обе границы, эти проверки пропускаются с указанием причины. +4. **Насыщение БД проверяется вмешательством.** На B4 сравниваем БД с 1 и 2 CPU при неизменных API, памяти и пуле соединений: три пары на нижней и три на верхней пилотной границе. Гипотеза DB_CPU: удвоение CPU БД возвращает SLO на верхней точке во всех трёх блоках, обе нижние точки проходят. Дополнительные условия: у исходной верхней точки средняя загрузка БД ≥90% лимита, API <85%, LB <90%. Без этих условий нельзя приписывать улучшение CPU БД. Отсутствие улучшения не исключает ограничений WAL, диска или блокировок. Для их разбора сохраняются счётчики WAL/I/O, ожидания PostgreSQL, ожидание пула и throttling. Если границы B4 нет, эта часть пропускается. +5. **Фиксируются условия повторения.** Перед запуском сохраняются исходники, их SHA-256, версия Docker, доступные CPU/RAM и ID единственного образа приложения. Фактические лимиты, размер пула и GOMAXPROCS проверяются на каждой точке. Телеметрия использует Docker context и число CPU Docker, без предположения о 16 процессорах хоста. Это позволяет повторить процедуру на другом компьютере, но не обещает одинаковый эффект на AMD и M4. + +Длительность измерения — 30 с, прогрев — 15 с, пауза — 10 с. Перед каждой точкой восстанавливаются исходные данные и сессии, API пересоздаются. Порядок и seeds записаны в `design.json`. Расчёт p95 учитывает задержку отправки; отдельно сохраняются попытки, успешные запросы, ошибки, пропуски генератора и число ответов до конца окна. Пропуски >1% либо задержка отправки p95 >100 мс при CPU генератора ≥0,85 ядра останавливают серию как невалидную. Плохая производительность приложения сохраняется как результат, а не удаляется. + +## Запуск в WSL + +Из корня репозитория, с работающим Docker Desktop / Docker Engine: + +```bash +# Только подготовка: сборка образа и фиксация методики, без нагрузки. +bash experiments/lab3-scale-v2/run.sh prepare scale-check-v2 +``` + +Скрипт сам создаёт `.venv` в корне репозитория, устанавливает закреплённые зависимости и использует `.venv/bin/python`. Активировать окружение вручную не нужно. При следующих запусках оно используется повторно. + +После подготовки не меняй код приложения и файлы обоих экспериментов до окончания серии. Изменения обнаруживаются по контрольным суммам; для другой версии нужен новый идентификатор с окончанием `-v2`. + +```bash +bash experiments/lab3-scale-v2/run.sh run scale-check-v2 +``` + +В другом терминале: + +```bash +bash experiments/lab3-scale-v2/run.sh status scale-check-v2 +# Пересчитать таблицы и графики из сохранённых запросов: +bash experiments/lab3-scale-v2/run.sh analyze scale-check-v2 +# Если терминал был принудительно закрыт и приложение не восстановилось: +bash experiments/lab3-scale-v2/run.sh restore scale-check-v2 +``` + +Ctrl+C завершает серию и запускает восстановление. Повторный запуск поверх начатой серии запрещён: для повторения выбери новый идентификатор и сначала выполни `prepare`. + +## Где что лежит + +Результаты появятся в `evidence/grade5-v2/scale-check-v2/`: + +| Файл / папка | Содержимое | +| --- | --- | +| `design.json`, `design.sha256`, `source.tar.gz` | Зафиксированные правила, параметры и исходники | +| `environment-*.json` | Условия подготовки и запуска | +| `calibration/`, `calibration.json` | Пилотный поиск нагрузок; не входит в подтверждающую серию | +| `confirmation.json`, `confirmation.sha256` | План независимых проверок, сохранённый до их начала | +| `scaling/`, `gomax/`, `database/` | Новые измерения трёх частей исследования | +| `execution-order.json`, `status.json`, `restoration.json` | Порядок, прогресс и восстановление основного приложения | +| `summary.csv`, `analysis.json`, `RESULTS.md`, `*.png` | Пересчитанные метрики, решения по гипотезам и графики | + +Внутри каждого измерения: `manifest.json` — параметры и контейнеры, `requests.jsonl.gz` — отдельные запросы, `telemetry.jsonl.gz` — ресурсы и БД, `pool.json` — пул/GOMAXPROCS, `summary.json` — метрики и проверка сохранности сообщений, `diagnostics.json` — диагностика. Прогрев хранится отдельно. Пароли находятся только в `.memory`. + +`design.py` задаёт правила и порядок, `runner.py` управляет серией, `runtime.py` управляет отдельным стендом и телеметрией, `results.py` пересчитывает данные. Сам генератор запросов и SQL-набор данных используются из первого эксперимента. Команда `smoke` выполняет только короткую проверку инструментов и сохраняет её в `.memory`, отдельно от результатов исследования. diff --git a/experiments/lab3-scale-v2/compose.yml b/experiments/lab3-scale-v2/compose.yml new file mode 100644 index 00000000..f95f6980 --- /dev/null +++ b/experiments/lab3-scale-v2/compose.yml @@ -0,0 +1,81 @@ +services: + db: + image: postgres:16.10-bookworm + cpus: ${BENCH_DB_CPU:-1} + mem_limit: 512m + command: [postgres, -c, track_io_timing=on, -c, track_wal_io_timing=on] + environment: + POSTGRES_DB: benchmark + POSTGRES_USER: benchmark + POSTGRES_PASSWORD: ${BENCH_PASSWORD:?private environment required} + volumes: [bench_db:/var/lib/postgresql/data] + networks: [data] + healthcheck: + test: [CMD-SHELL, 'pg_isready -U benchmark -d benchmark'] + interval: 1s + timeout: 3s + retries: 60 + redis: + image: redis:7.4.11-bookworm + cpus: 0.25 + mem_limit: 128m + command: [redis-server, --appendonly, 'yes', --appendfsync, everysec, --dir, /data] + volumes: [bench_redis:/data] + networks: [data] + healthcheck: + test: [CMD, redis-cli, ping] + interval: 1s + timeout: 3s + retries: 60 + migrate: + image: ${BENCH_IMAGE:?prepared image required} + cpus: 0.5 + mem_limit: 128m + networks: [data] + environment: + DATABASE_URL: postgres://benchmark:${BENCH_PASSWORD}@db:5432/benchmark?sslmode=disable + command: [./messenger, migrate, up] + depends_on: + db: {condition: service_healthy} + api: + image: ${BENCH_IMAGE:?prepared image required} + cpus: ${BENCH_CPU:-0.5} + mem_limit: ${BENCH_MEMORY:-128m} + deploy: + replicas: ${BENCH_REPLICAS:-2} + stop_grace_period: 15s + networks: [proxy, data] + environment: + DATABASE_URL: postgres://benchmark:${BENCH_PASSWORD}@db:5432/benchmark?sslmode=disable + DATABASE_MAX_CONNS: ${BENCH_POOL:-6} + BENCH_POOL_STATS: '1' + GOMAXPROCS: ${BENCH_GOMAXPROCS:-16} + REDIS_ADDR: redis:6379 + SESSION_TTL_SECONDS: 86400 + APP_PORT: 8080 + APP_HOST: 0.0.0.0 + healthcheck: + test: [CMD, ./messenger, healthcheck] + interval: 1s + timeout: 3s + retries: 60 + depends_on: + migrate: {condition: service_completed_successfully} + redis: {condition: service_healthy} + haproxy: + image: haproxy:3.2.25-alpine + cpus: 0.5 + mem_limit: 128m + ports: ['127.0.0.1:18082:8080', '127.0.0.1:18406:8404'] + networks: [edge, proxy] + volumes: + - ./experiments/lab3-scale/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro + depends_on: + api: {condition: service_healthy} +volumes: + bench_db: + bench_redis: +networks: + edge: + proxy: {internal: true} + data: {internal: true} diff --git a/experiments/lab3-scale-v2/design.py b/experiments/lab3-scale-v2/design.py new file mode 100644 index 00000000..27471874 --- /dev/null +++ b/experiments/lab3-scale-v2/design.py @@ -0,0 +1,134 @@ +"""Pure, deterministic rules for the preregistered v2 experiment.""" +import random +import statistics + +CONFIGS = { + "A1": {"replicas": 1, "cpu": 1, "memory": "256m", "pool": 12}, + "A2": {"replicas": 2, "cpu": .5, "memory": "128m", "pool": 6}, + "A4": {"replicas": 4, "cpu": .25, "memory": "64m", "pool": 3}, + "B1": {"replicas": 1, "cpu": .5, "memory": "128m", "pool": 12}, + "B2": {"replicas": 2, "cpu": .5, "memory": "128m", "pool": 6}, + "B4": {"replicas": 4, "cpu": .5, "memory": "128m", "pool": 3}, +} +DESIGN = { + "version": "v2.0", + "hypotheses": { + "H1": "Fixed total API budget: C4/C1 < 1.20 in all three blocks", + "H2": "Growing API budget: C4/C1 >= 1.50 in all three blocks", + "G_low": "At 25% of pilot A4 capacity, GOMAXPROCS=1 reduces p95 >=15% in all five pairs, retaining success and SLO", + "G_high": "At 90% of pilot A4 capacity, the same criteria hold in all five pairs", + "DB_CPU": "At the pilot B4 upper rate, increasing only DB CPU 1->2 restores SLO in all three blocks, with CPU-DB diagnostic prerequisites", + }, + "duration_s": 30, "warmup_s": 15, "cooldown_s": 10, + "timeout_s": 5, "workload_seed": 230300, "inflight_limit": 8192, + "success_min": .99, "p95_max_ms": 500, + "workload": "80% new idempotent POST messages (128 ASCII bytes), 20% first history page GET limit=50", + "fixture": {"users": 200, "chats": 100, "messages": 10000}, + "dependencies": {"db_cpu": 1, "db_memory": "512m", "redis_cpu": .25, "redis_memory": "128m", "lb_cpu": .5, "lb_memory": "128m"}, + "pool_total": 12, "control_gomaxprocs": 16, "tuned_gomaxprocs": 1, + "search": {"start": 100, "floor": 25, "ceiling": 6400, "max_points": 10, "relative_width": .10}, + "scaling_repeats": 3, "gomax_pairs": 5, "db_pairs": 3, + "calibration_seed": 230700, "scaling_seeds": [230701, 230702, 230703], + "gomax_order_seed": 230816, "db_order_seed": 230817, + "budget_seconds": 9000, + "generator_invalid": "dropped/scheduled >0.01 OR (dispatch p95 >100ms AND generator CPU >=0.85 cores)", + "db_cpu_prerequisites": {"db_cpu_min": .90, "api_cpu_max": .85, "lb_cpu_max": .90}, + "aggregation": "Keep all pairs. Mean/median/range of paired percent reductions are descriptive, not a pooled p95. Mixed outcomes are inconclusive.", + "censoring": "No failed upper point means only a lower bound, never an exact capacity. Invalid generator stops the experiment; bad API performance and OOM are retained.", + "pilot_excluded_from_confirmation": True, +} + + +def config(name, gomax=16, db_cpu=1): + return {**CONFIGS[name], "gomaxprocs": gomax, "db_cpu": db_cpu} + + +def next_rate(observations, rules=None): + rules = rules or DESIGN["search"] + if any(not row["valid"] for row in observations): + return None + if not observations: + return rules["start"] + if len(observations) >= rules["max_points"]: + return None + passed = [row["rate"] for row in observations if row["slo_pass"]] + failed = [row["rate"] for row in observations if not row["slo_pass"]] + if passed and failed and max(passed) >= min(failed): + return None + if not passed: + value = max(rules["floor"], min(failed) // 2) + return value if value < min(failed) else None + lower = max(passed) + if not failed: + value = min(lower * 2, rules["ceiling"]) + return value if value > lower else None + upper = min(failed) + if (upper - lower) / lower <= rules["relative_width"] or upper - lower <= 1: + return None + return (lower + upper) // 2 + + +def bracket(observations): + passed = [row["rate"] for row in observations if row["valid"] and row["slo_pass"]] + failed = [row["rate"] for row in observations if row["valid"] and not row["slo_pass"]] + lower = max(passed, default=0) + upper = min(failed) if failed else None + invalid = any(not row["valid"] for row in observations) + nonmonotonic = upper is not None and lower >= upper + return {"lower": lower, "upper": upper, "invalid": invalid, "nonmonotonic": nonmonotonic, + "closed": bool(lower and upper and not invalid and not nonmonotonic)} + + +def freeze_confirmation(calibration): + bounds = {name: bracket(rows) for name, rows in calibration.items()} + rates = {} + for name, value in bounds.items(): + candidates = [100, value["lower"], value["upper"]] + rates[name] = sorted({rate for rate in candidates if rate and rate <= DESIGN["search"]["ceiling"]}) + scaling = [] + for repeat, seed in enumerate(DESIGN["scaling_seeds"], 1): + points = [{"phase": "scaling", "base": name, "rate": rate, "repeat": repeat, "variant": name, + "config": config(name)} for name in CONFIGS for rate in rates[name]] + random.Random(seed).shuffle(points) + scaling.extend(points) + gomax = [] + if bounds["A4"]["closed"]: + groups = [] + for regime, fraction in (("low", .25), ("high", .90)): + rate = max(1, int(bounds["A4"]["lower"] * fraction)) + for repeat in range(1, DESIGN["gomax_pairs"] + 1): + order = [16, 1] if repeat % 2 else [1, 16] + groups.append([{"phase": "gomax", "base": "A4", "regime": regime, "rate": rate, + "repeat": repeat, "variant": "I0" if value == 16 else "I1", + "config": config("A4", gomax=value)} for value in order]) + random.Random(DESIGN["gomax_order_seed"]).shuffle(groups) + gomax = [point for group in groups for point in group] + database = [] + if bounds["B4"]["closed"]: + groups = [] + for repeat in range(1, DESIGN["db_pairs"] + 1): + for rate in (bounds["B4"]["lower"], bounds["B4"]["upper"]): + order = [1, 2] if repeat % 2 else [2, 1] + groups.append([{"phase": "database", "base": "B4", "rate": rate, "repeat": repeat, + "variant": "D0" if cpu == 1 else "D1", "config": config("B4", db_cpu=cpu)} for cpu in order]) + random.Random(DESIGN["db_order_seed"]).shuffle(groups) + database = [point for group in groups for point in group] + return {"pilot_bounds": bounds, "rates_by_config": rates, "scaling": scaling, + "database": database, "gomax": gomax, + "skipped": {"gomax": None if gomax else "A4 pilot bracket not closed", + "database": None if database else "B4 pilot bracket not closed"}} + + +def pair_decision(pairs, expected=5): + if len(pairs) != expected or any(not pair["valid"] for pair in pairs): + result = "inconclusive" + elif all(pair["criteria_met"] for pair in pairs): + result = "supported" + elif all(not pair["criteria_met"] for pair in pairs): + result = "refuted" + else: + result = "inconclusive" + values = [pair["reduction_percent"] for pair in pairs if pair["reduction_percent"] is not None] + return {"result": result, "pairs": pairs, "reduction_mean_percent": statistics.mean(values) if values else None, + "reduction_median_percent": statistics.median(values) if values else None, + "reduction_range_percent": [min(values), max(values)] if values else None} diff --git a/experiments/lab3-scale-v2/results.py b/experiments/lab3-scale-v2/results.py new file mode 100644 index 00000000..5e3c3fc1 --- /dev/null +++ b/experiments/lab3-scale-v2/results.py @@ -0,0 +1,210 @@ +"""Recompute v2 results from individual requests; never touch v1 evidence.""" +import csv +import gzip +import json +from collections import defaultdict + +from design import CONFIGS, bracket, pair_decision +from runtime import digest, legacy_analysis, save, workload + + +def read(path): + return json.loads(path.read_text(encoding="utf-8")) + + +def jsonl(path): + with gzip.open(path, "rt", encoding="utf-8") as stream: + return [json.loads(line) for line in stream] + + +def audit(directory, manifest, design): + summary = read(directory / "summary.json") + rows = jsonl(directory / "requests.jsonl.gz") + successes = [row for row in rows if row["success"]] + sent = [row for row in rows if "start_s" in row] + duration = design["duration_s"] + calculated = { + "scheduled": len(rows), "sent": len(sent), "successes": len(successes), + "errors": len(rows) - len(successes), "generator_dropped": len(rows) - len(sent), + "success_ratio": len(successes) / max(1, len(rows)), + "goodput_rps": len(successes) / duration, + "window_goodput_rps": sum(row["end_s"] <= duration for row in successes) / duration, + "p95_ms": workload.percentile([row["latency_ms"] for row in successes], .95), + "dispatch_p95_ms": workload.percentile([row["dispatch_lag_ms"] for row in sent], .95), + } + mismatches = [key for key, value in calculated.items() if summary.get(key) != value] + if len(rows) != round(manifest["rate"] * duration): + mismatches.append("expected_request_count") + if manifest["protocol_sha256"] != digest(design): + mismatches.append("protocol_sha256") + if len({row["index"] for row in rows}) != len(rows): + mismatches.append("unique_request_index") + valid = summary["valid"] and not mismatches + slo = valid and summary["correctness"]["passed"] and calculated["success_ratio"] >= .99 and calculated["p95_ms"] is not None and calculated["p95_ms"] <= 500 + diagnostic = legacy_analysis.diagnostics(directory, manifest) + return {**manifest, **summary, **calculated, "valid": valid, "slo_pass": slo, + "audit_mismatches": mismatches, "diagnostics": diagnostic, + "database_counters": database_counters(jsonl(directory / "telemetry.jsonl.gz"))} + + +def database_counters(samples): + values = [row["postgres"] for row in samples if "postgres" in row] + if len(values) < 2: + return {"available": False} + first, last = values[0], values[-1] + result = {"available": True, "note": "First-to-last sampled counters, not the full request window; waits are sample counts, not wait durations."} + for section, keys in (("wal", ("wal_records", "wal_bytes", "wal_write", "wal_sync", "wal_write_time", "wal_sync_time")), + ("database", ("xact_commit", "xact_rollback", "blks_read", "blks_hit", "deadlocks"))): + reset = first[section].get("stats_reset") != last[section].get("stats_reset") + delta = {key: float(last[section][key]) - float(first[section][key]) for key in keys} + result[section] = {"reset_detected": reset or any(value < 0 for value in delta.values()), "delta": delta} + io = {} + old = {(row["object"], row["context"]): row for row in first["io"]} + for row in last["io"]: + prior = old.get((row["object"], row["context"])) + if prior: + keys = ("reads", "read_time", "writes", "write_time", "extends", "extend_time", "fsyncs", "fsync_time") + delta = {key: float(row[key] or 0) - float(prior[key] or 0) for key in keys} + io[row["object"] + "/" + row["context"]] = {"delta": delta, "reset_detected": row.get("stats_reset") != prior.get("stats_reset") or any(v < 0 for v in delta.values())} + result["io"] = io + return result + + +def capacities(runs, schedule): + result = [] + for name in CONFIGS: + for repeat in (1, 2, 3): + points = [row for row in runs if row["phase"] == "scaling" and row["base"] == name and row["repeat"] == repeat] + bound = bracket(points) + expected = len([row for row in schedule if row["base"] == name and row["repeat"] == repeat]) + incomplete = len(points) != expected or not expected + result.append({"config": name, "repeat": repeat, "capacity_lower": bound["lower"], + "capacity_upper": bound["upper"], "nonmonotonic": bound["nonmonotonic"], + "invalid_rates": [row["rate"] for row in points if not row["valid"]] + (["missing"] if incomplete else [])}) + return result + + +def gomax_results(runs): + result = {} + for regime in ("low", "high"): + pairs = [] + for repeat in range(1, 6): + values = {row["variant"]: row for row in runs if row["phase"] == "gomax" and row["regime"] == regime and row["repeat"] == repeat} + if "I0" not in values or "I1" not in values: + continue + control, tuned = values["I0"], values["I1"] + numeric = control["p95_ms"] is not None and control["p95_ms"] > 0 and tuned["p95_ms"] is not None + reduction = 100 * (1 - tuned["p95_ms"] / control["p95_ms"]) if numeric else None + valid = numeric and control["valid"] and tuned["valid"] and control["rate"] == tuned["rate"] and control["correctness"]["passed"] and tuned["correctness"]["passed"] + met = numeric and tuned["p95_ms"] <= .85 * control["p95_ms"] and tuned["p95_ms"] <= 500 and tuned["success_ratio"] >= max(.99, control["success_ratio"]) + pairs.append({"repeat": repeat, "rate": control["rate"], "control_p95_ms": control["p95_ms"], "tuned_p95_ms": tuned["p95_ms"], + "valid": bool(valid), "criteria_met": bool(met), "reduction_percent": reduction}) + result["G_" + regime] = pair_decision(pairs) + return result + + +def db_results(runs, confirmation): + bounds = confirmation.get("pilot_bounds", {}).get("B4", {}) + lower, upper = bounds.get("lower"), bounds.get("upper") + blocks = [] + for repeat in (1, 2, 3): + values = {(row["variant"], row["rate"]): row for row in runs if row["phase"] == "database" and row["repeat"] == repeat} + required = [(variant, rate) for variant in ("D0", "D1") for rate in (lower, upper)] + if not lower or not upper or any(key not in values for key in required): + continue + dl, du, tl, tu = (values[key] for key in (("D0", lower), ("D0", upper), ("D1", lower), ("D1", upper))) + cpu = du["diagnostics"]["cpu_fraction_of_limit_mean"] + eligible = cpu.get("db", -1) >= .90 and 0 <= cpu.get("api", -1) < .85 and 0 <= cpu.get("haproxy", -1) < .90 + valid = all(values[key]["valid"] and values[key]["correctness"]["passed"] for key in required) + restored = dl["slo_pass"] and tl["slo_pass"] and not du["slo_pass"] and tu["slo_pass"] + blocks.append({"repeat": repeat, "valid": valid, "cpu_prerequisites_met": eligible, + "slo_restored": restored, "baseline_upper_cpu": cpu}) + supported = len(blocks) == 3 and all(row["valid"] and row["cpu_prerequisites_met"] and row["slo_restored"] for row in blocks) + return {"result": "supported" if supported else "inconclusive", "blocks": blocks, + "note": "Doubling CPU tests a CPU bottleneck only. No effect does not disprove a WAL, storage, lock or pool bottleneck."} + + +def plots(directory, caps, hypotheses, runs): + import matplotlib + matplotlib.use("Agg") + import matplotlib.pyplot as plt + fig, ax = plt.subplots(figsize=(10, 4)) + for index, row in enumerate(caps): + lower, upper = row["capacity_lower"], row["capacity_upper"] + ax.plot(index, lower, "o", color="tab:blue") + if upper is not None: + ax.plot([index, index], [lower, upper], color="tab:blue") + ax.plot(index, upper, "x", color="tab:red") + else: + ax.annotate("≥", (index, lower), xytext=(0, 7), textcoords="offset points", ha="center") + ax.set_xticks(range(len(caps)), [r["config"] + "/" + str(r["repeat"]) for r in caps], rotation=60) + ax.set(ylabel="Offered requests/s", title="Confirmation capacity brackets (not confidence intervals)") + ax.grid(axis="y", alpha=.2) + fig.tight_layout(); fig.savefig(directory / "capacity.png", dpi=160); plt.close(fig) + fig, axes = plt.subplots(1, 2, figsize=(9, 4)) + for ax, regime in zip(axes, ("low", "high")): + for pair in hypotheses["G_" + regime]["pairs"]: + ax.plot([0, 1], [pair["control_p95_ms"], pair["tuned_p95_ms"]], "o-", label="pair " + str(pair["repeat"])) + ax.set_xticks([0, 1], ["GOMAXPROCS=16", "GOMAXPROCS=1"]) + ax.set(title=regime, ylabel="Successful request p95, ms") + ax.grid(alpha=.2) + fig.tight_layout(); fig.savefig(directory / "gomax.png", dpi=160); plt.close(fig) + points = [row for row in runs if row["phase"] == "database"] + if points: + fig, ax = plt.subplots(figsize=(8, 4)) + for variant in ("D0", "D1"): + group = [row for row in points if row["variant"] == variant] + ax.scatter([r["rate"] for r in group], [r["p95_ms"] for r in group], label=variant) + ax.axhline(500, color="red", linestyle="--", label="p95 SLO (also success ≥99%)") + ax.set(xlabel="Offered requests/s", ylabel="Successful request p95, ms") + ax.legend(); fig.tight_layout(); fig.savefig(directory / "database.png", dpi=160); plt.close(fig) + + +def analyze(directory): + design = read(directory / "design.json") + if digest(design) != (directory / "design.sha256").read_text().strip(): + raise ValueError("Design checksum mismatch") + confirmation = read(directory / "confirmation.json") if (directory / "confirmation.json").exists() else {} + if confirmation and (digest(confirmation) != (directory / "confirmation.sha256").read_text().strip() or confirmation["design_sha256"] != digest(design)): + raise ValueError("Confirmation schedule checksum mismatch") + runs, incomplete = [], [] + for item in read(directory / "execution-order.json") if (directory / "execution-order.json").exists() else []: + target = directory / item["directory"] + if not (target / "summary.json").exists(): + incomplete.append(item["directory"]) + continue + manifest = read(target / "manifest.json") + run = audit(target, manifest, design) + if any(manifest.get(key) != item.get(key) for key in ("phase", "base", "rate", "repeat", "variant", "config", "sequence")): + run.update(valid=False, slo_pass=False) + run["audit_mismatches"].append("execution_order") + runs.append(run) + caps = capacities(runs, confirmation.get("scaling", [])) + hypotheses = {**legacy_analysis.hypothesis_decisions(caps), **gomax_results(runs), "DB_CPU": db_results(runs, confirmation)} + if design["smoke_only"]: + hypotheses = {name: {"result": "not_tested_smoke_only"} for name in hypotheses} + result = {"series": design["series"], "smoke_only": design["smoke_only"], "status": read(directory / "status.json"), + "incomplete_runs": incomplete, "hypotheses": hypotheses, "capacities": caps, "runs": runs} + save(directory / "analysis.json", result) + keys = ["phase", "base", "variant", "repeat", "rate", "scheduled", "sent", "successes", "errors", "generator_dropped", + "success_ratio", "goodput_rps", "window_goodput_rps", "p95_ms", "dispatch_p95_ms", "valid", "slo_pass"] + with (directory / "summary.csv").open("w", newline="", encoding="utf-8") as stream: + writer = csv.DictWriter(stream, fieldnames=keys, extrasaction="ignore") + writer.writeheader(); writer.writerows(runs) + lines = ["# Результаты " + design["series"], "", "Статус: `" + result["status"]["status"] + "`. Завершённых измерений: " + str(len(runs)) + ".", ""] + if design["smoke_only"]: + lines += ["Короткая проверка инструментов. Эти данные не проверяют гипотезы и не входят в исследование.", ""] + else: + lines += ["| Гипотеза | Результат |", "| --- | --- |"] + [f"| {name} | {value['result']} |" for name, value in hypotheses.items()] + lines += ["", "supported — критерий выполнен во всех повторах; refuted — во всех парах получен противоположный результат; inconclusive — данных недостаточно или повторы расходятся.", "", + "[Сводная таблица](summary.csv), [полный расчёт и диагностика](analysis.json), [зафиксированная методика](design.json).", + "", "Пилотные измерения выбирают нагрузку и не входят в подтверждающую серию. Пределы мощности — границы на проверенной сетке, а не доверительные интервалы. Если верхняя граница не найдена, известно только C ≥ нижней границы.", "", + "GOMAXPROCS оценивается по отдельным парам. Среднее, медиана и диапазон процентов снижения p95 приведены в analysis.json; p95 разных прогонов не объединяются. Среднее не заменяет критерий пяти пар.", "", + "Счётчики WAL/I/O отражают разницу между первым и последним образцом телеметрии. Частота наблюдения ожиданий не равна времени ожидания. Удвоение CPU БД проверяет только ограничение CPU; отсутствие эффекта не исключает ограничения диска, WAL или блокировок.", "", + "![Границы мощности](capacity.png)", "", "![Парные сравнения](gomax.png)"] + if any(row["phase"] == "database" for row in runs): + lines += ["", "![Диагностика БД](database.png)"] + plots(directory, caps, hypotheses, runs) + (directory / "RESULTS.md").write_text("\n".join(lines) + "\n", encoding="utf-8") + print("Analysis: " + str(directory / "RESULTS.md"), flush=True) + return result diff --git a/experiments/lab3-scale-v2/run.sh b/experiments/lab3-scale-v2/run.sh new file mode 100644 index 00000000..5fe20370 --- /dev/null +++ b/experiments/lab3-scale-v2/run.sh @@ -0,0 +1,16 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/../.." +action=${1:-run} +series=${2:-scale-check-v2} +python=.venv/bin/python +requirements=experiments/lab3-scale/requirements-lock.txt +if [ ! -x "$python" ]; then + python3 -m venv .venv +fi +if ! cmp -s "$requirements" .venv/lab3-scale-v2-requirements.txt; then + "$python" -m pip install -r "$requirements" + cp "$requirements" .venv/lab3-scale-v2-requirements.txt +fi +export PYTHONDONTWRITEBYTECODE=1 +exec "$python" -u experiments/lab3-scale-v2/runner.py "$action" --series "$series" diff --git a/experiments/lab3-scale-v2/runner.py b/experiments/lab3-scale-v2/runner.py new file mode 100644 index 00000000..92f5c3f2 --- /dev/null +++ b/experiments/lab3-scale-v2/runner.py @@ -0,0 +1,213 @@ +"""Prepare once; the user starts the long preregistered experiment explicitly.""" +import argparse +import copy +import fcntl +import json +import random +import re +import sys +import tarfile +import time +from datetime import datetime, timezone +from pathlib import Path + +from design import CONFIGS, DESIGN, config, freeze_confirmation, next_rate +from runtime import HERE, ROOT, Runtime, check_ports, digest, environment, original, save, shell, sources + + +def validate_name(name): + if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_-]{0,63}-v2", name): + raise ValueError("Series name must contain letters/digits/_/- and end with -v2") + return name + + +def directory_for(name, smoke=False): + validate_name(name) + return ROOT / (".memory/lab3-scale-v2-smoke" if smoke else "evidence/grade5-v2") / name + + +def prepare(name, smoke=False): + directory = directory_for(name, smoke) + fingerprint = sources() + if directory.exists(): + existing = json.loads((directory / "design.json").read_text()) + if existing["sources"] != fingerprint: + raise RuntimeError("Prepared sources changed; choose a new series ending in -v2") + if digest(existing) != (directory / "design.sha256").read_text().strip(): + raise RuntimeError("Prepared design checksum differs") + if json.loads((directory / "status.json").read_text())["status"] != "prepared": + raise RuntimeError("Series already started or preparation was incomplete; choose a new series") + print("Already prepared: " + str(directory), flush=True) + return directory + check_ports() + env = environment() + image = "highload-lab3-bench-v2:" + env["app_code_sha256"][:16] + print("Building one image for every v2 condition", flush=True) + shell(["docker", "build", "-t", image, "src"]) + image_id = shell(["docker", "image", "inspect", "--format", "{{.Id}}", image]) + design = {**copy.deepcopy(DESIGN), "series": name, "smoke_only": smoke, + "sources": fingerprint, "image": image, "image_id": image_id, + "frozen_utc": datetime.now(timezone.utc).isoformat()} + if smoke: + design.update(duration_s=5, warmup_s=2, cooldown_s=1, budget_seconds=300) + directory.mkdir(parents=True) + save(directory / "design.json", design) + (directory / "design.sha256").write_text(digest(design) + "\n") + save(directory / "environment-prepared.json", env) + with tarfile.open(directory / "source.tar.gz", "w:gz") as archive: + for relative in fingerprint: + archive.add(ROOT / relative, arcname=relative, recursive=False) + save(directory / "status.json", {"status": "preparing", "long_experiment_started": False, + "smoke_only": smoke}) + # Credentials remain outside evidence and the source archive. + runtime = Runtime(name, image, env) + runtime.compose(["config", "--quiet"]) + save(directory / "status.json", {"status": "prepared", "long_experiment_started": False, + "smoke_only": smoke}) + print("PREPARED: " + str(directory), flush=True) + return directory + + +class StopExperiment(RuntimeError): + pass + + +class Experiment: + def __init__(self, directory): + self.directory = directory + self.design = json.loads((directory / "design.json").read_text()) + if digest(self.design) != (directory / "design.sha256").read_text().strip(): + raise RuntimeError("Frozen design checksum differs") + if self.design["sources"] != sources(): + raise RuntimeError("Code/instrument changed after preparation; choose a new series") + current_image = shell(["docker", "image", "inspect", "--format", "{{.Id}}", self.design["image"]]) + if current_image != self.design["image_id"]: + raise RuntimeError("Prepared Docker image changed") + self.protocol_sha = digest(self.design) + self.started = time.monotonic() + self.sequence = 0 + self.runtime = Runtime(self.design["series"], self.design["image"], environment()) + self.order = [] + + def run_point(self, point): + # Budget stops are explicit and keep every completed or partial run. + needed = self.design["warmup_s"] + self.design["duration_s"] + self.design["cooldown_s"] + 15 + if time.monotonic() - self.started + needed > self.design["budget_seconds"]: + raise StopExperiment("budget_exhausted") + if self.design["sources"] != sources(): + raise StopExperiment("sources_changed_during_experiment") + self.sequence += 1 + name = f"{self.sequence:03d}-{point['variant']}-r{point['repeat']}-rate{point['rate']}" + if point.get("regime"): + name += "-" + point["regime"] + target = self.directory / point["phase"] / name + self.order.append({"sequence": self.sequence, "directory": str(target.relative_to(self.directory)), **point}) + save(self.directory / "execution-order.json", self.order) + save(self.directory / "status.json", {"status": "running", "current": self.order[-1], + "elapsed_s": round(time.monotonic() - self.started, 1), "completed_runs": self.sequence - 1, + "smoke_only": self.design["smoke_only"]}) + print(f"START {point['phase']} {name}", flush=True) + result = self.runtime.execute(point, target, self.protocol_sha, self.design, self.sequence) + if not result["valid"]: + cause = "generator_limited" if result["generator_limited"] else "instrument_or_runtime_invalid" + raise StopExperiment(cause) + if not result["correctness"]["passed"]: + raise StopExperiment("confirmed_data_not_preserved") + time.sleep(self.design["cooldown_s"]) + return result + + def execute(self): + old = json.loads((self.directory / "status.json").read_text()) + if old["status"] != "prepared": + raise RuntimeError("Series has already started; raw data are retained. Prepare a new series.") + check_ports() + status, error, restored = "failed", None, False + try: + self.runtime.suspend_main() + save(self.directory / "environment-run.json", self.runtime.env_info) + if self.design["smoke_only"]: + for variant, settings in (("I0", config("A4", 16)), ("I1", config("A4", 1)), ("D1", config("A4", 16, 2))): + result = self.run_point({"phase": "smoke", "base": "A4", "rate": 20, "repeat": 1, + "variant": variant, "config": settings}) + if not result["slo_pass"]: + raise StopExperiment("smoke_slo_not_met") + status = "smoke_complete" + else: + calibration = {name: [] for name in CONFIGS} + names = list(CONFIGS) + random.Random(self.design["calibration_seed"]).shuffle(names) + for name in names: + while True: + rate = next_rate(calibration[name], self.design["search"]) + if rate is None: + break + result = self.run_point({"phase": "calibration", "base": name, "variant": name, + "rate": rate, "repeat": len(calibration[name]) + 1, "config": config(name)}) + calibration[name].append({"rate": rate, "valid": result["valid"], "slo_pass": result["slo_pass"]}) + save(self.directory / "calibration.json", calibration) + confirmation = freeze_confirmation(calibration) + confirmation["design_sha256"] = self.protocol_sha + save(self.directory / "confirmation.json", confirmation) + (self.directory / "confirmation.sha256").write_text(digest(confirmation) + "\n") + # This schedule is frozen before seeing any confirmation outcome. + for phase in ("scaling", "database", "gomax"): + for point in confirmation[phase]: + self.run_point(point) + status = "complete" + except StopExperiment as failure: + status, error = str(failure), str(failure) + except KeyboardInterrupt: + status, error = "interrupted", "KeyboardInterrupt" + except Exception as failure: + status, error = "failed", str(failure) + finally: + restored = self.runtime.restore() + save(self.directory / "restoration.json", json.loads((self.runtime.private / "restoration.json").read_text())) + save(self.directory / "status.json", {"status": status, "error": error, "restoration_ok": restored, + "elapsed_s": round(time.monotonic() - self.started, 1), "scheduled_or_started_runs": self.sequence, + "smoke_only": self.design["smoke_only"]}) + print(f"FINISHED: {status}; original application restored={restored}", flush=True) + from results import analyze + analyze(self.directory) + return 0 if status in ("complete", "smoke_complete") and restored else 1 + + +def main(): + parser = argparse.ArgumentParser(__doc__) + parser.add_argument("action", choices=("prepare", "run", "smoke", "status", "analyze", "restore")) + parser.add_argument("--series", default="scale-check-v2") + args = parser.parse_args() + validate_name(args.series) + smoke = args.action == "smoke" + directory = directory_for(args.series, smoke) + if args.action == "status": + print((directory / "status.json").read_text()) + return 0 + if args.action == "analyze": + from results import analyze + analyze(directory) + return 0 + lock_path = ROOT / ".memory/benchmark/experiment.lock" + lock_path.parent.mkdir(parents=True, exist_ok=True) + with lock_path.open("a") as lock: + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + parser.error("Another v1/v2 experiment is running") + if args.action == "prepare": + prepare(args.series) + return 0 + if args.action == "restore": + design = json.loads((directory / "design.json").read_text()) + return 0 if Runtime(args.series, design["image"], environment()).restore() else 1 + if smoke: + prepare(args.series, smoke=True) + return Experiment(directory).execute() + + +if __name__ == "__main__": + try: + sys.exit(main()) + except Exception as error: + print("ERROR: " + str(error), file=sys.stderr) + sys.exit(1) diff --git a/experiments/lab3-scale-v2/runtime.py b/experiments/lab3-scale-v2/runtime.py new file mode 100644 index 00000000..cc036a22 --- /dev/null +++ b/experiments/lab3-scale-v2/runtime.py @@ -0,0 +1,276 @@ +"""Isolated v2 Docker runtime; reuse the v1 workload and fixture unchanged.""" +import asyncio +import csv +import hashlib +import importlib.util +import io +import json +import os +import platform +import secrets +import socket +import subprocess +import sys +import time +from datetime import datetime, timezone +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[1] +V1 = HERE.parent / "lab3-scale" +sys.path.insert(0, str(V1)) +import common as original +import load as workload +import aiohttp + + +def module(path, name): + spec = importlib.util.spec_from_file_location(name, path) + result = importlib.util.module_from_spec(spec) + spec.loader.exec_module(result) + return result + + +legacy_analysis = module(V1 / "analyze.py", "scale_v1_analysis") + + +def save(path, data): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + + +def digest(data): + return hashlib.sha256(json.dumps(data, sort_keys=True, separators=(",", ":")).encode()).hexdigest() + + +def sources(): + paths = [p for base in (ROOT / "src/backend", V1, HERE) for p in base.rglob("*") + if p.is_file() and "__pycache__" not in p.parts and p.suffix.lower() != ".md"] + paths += [ROOT / name for name in ("src/go.mod", "src/go.sum", "src/Dockerfile")] + return {str(path.relative_to(ROOT)): hashlib.sha256(path.read_bytes()).hexdigest() for path in sorted(paths)} + + +def shell(args): + result = subprocess.run(args, cwd=ROOT, text=True, capture_output=True) + if result.returncode: + # Avoid echoing expanded connection strings or private Docker output. + raise RuntimeError("Command failed: " + " ".join(args[:3])) + return result.stdout.strip() + + +def docker_endpoint(): + context = json.loads(shell(["docker", "context", "inspect"]))[0] + endpoint = os.environ.get("DOCKER_HOST") or context["Endpoints"]["docker"]["Host"] + if not endpoint.startswith("unix://"): + raise RuntimeError("v2 telemetry requires a local unix:// Docker context (WSL/Linux/macOS)") + path = endpoint.removeprefix("unix://") + if not Path(path).exists(): + raise RuntimeError("Docker socket from the current context is unavailable") + return path + + +def check_ports(): + for port in (18082, 18406): + with socket.socket() as probe: + try: + probe.bind(("127.0.0.1", port)) + except OSError as error: + raise RuntimeError(f"Port {port} is occupied; no unrelated container will be stopped") from error + + +def environment(): + info = json.loads(shell(["docker", "info", "--format", + '{"cpus":{{.NCPU}},"memory":{{.MemTotal}},"architecture":{{json .Architecture}},"os":{{json .OperatingSystem}}}'])) + return {"time_utc": datetime.now(timezone.utc).isoformat(), "platform": platform.platform(), + "python": platform.python_version(), "docker_resources": info, + "docker_version": json.loads(shell(["docker", "version", "--format", "{{json .Server}}"])), + "compose_version": shell(["docker", "compose", "version"]), + "git_commit": shell(["git", "rev-parse", "HEAD"]), + "worktree_files": shell(["git", "status", "--porcelain"]).splitlines(), + "docker_socket": docker_endpoint(), "app_code_sha256": original.code_hash()} + + +DB_SAMPLE = """SELECT json_build_object( +'active',(SELECT count(*) FROM pg_stat_activity WHERE datname='benchmark' AND state='active' AND pid<>pg_backend_pid()), +'waits',(SELECT coalesce(json_agg(json_build_object('event_type',wait_event_type,'event',wait_event,'count',n)),'[]'::json) + FROM (SELECT wait_event_type,wait_event,count(*) n FROM pg_stat_activity WHERE datname='benchmark' AND pid<>pg_backend_pid() GROUP BY 1,2) t), +'database',(SELECT row_to_json(d) FROM (SELECT xact_commit,xact_rollback,blks_read,blks_hit,tup_inserted,deadlocks FROM pg_stat_database WHERE datname='benchmark') d), +'wal',(SELECT row_to_json(w) FROM pg_stat_wal w), +'io',(SELECT coalesce(json_agg(i),'[]'::json) FROM pg_stat_io i WHERE backend_type='client backend') +);""" + + +class Runtime: + def __init__(self, series, image, env_info): + self.series = series + self.private = ROOT / ".memory/lab3-scale-v2" / series + self.private.mkdir(parents=True, exist_ok=True) + self.project = "highload-lab3-v2-" + hashlib.sha256(series.encode()).hexdigest()[:10] + self.image = image + self.env_info = env_info + self.socket = env_info["docker_socket"] + self.db_cpu = None + credentials = self.private / "environment" + if not credentials.exists(): + credentials.write_text("BENCH_PASSWORD=" + secrets.token_hex(24) + "\n") + credentials.chmod(0o600) + # Only this process is adapted. No v1 source or historical evidence changes. + original.compose = self.compose + original.PROJECT = self.project + workload.BASE = "http://127.0.0.1:18082" + workload.STATS = "http://127.0.0.1:18406/stats;csv" + workload.collect = self.collect + + def compose(self, args, config=None): + config = config or {"cpu": .5, "memory": "128m", "replicas": 2, "pool": 6, + "gomaxprocs": 16, "db_cpu": 1} + env = {**os.environ, "BENCH_CPU": str(config["cpu"]), "BENCH_MEMORY": config["memory"], + "BENCH_REPLICAS": str(config["replicas"]), "BENCH_POOL": str(config["pool"]), + "BENCH_GOMAXPROCS": str(config["gomaxprocs"]), "BENCH_DB_CPU": str(config["db_cpu"]), + "BENCH_IMAGE": self.image} + result = subprocess.run(["docker", "compose", "--project-name", self.project, "--project-directory", str(ROOT), + "--env-file", str(self.private / "environment"), "-f", str(HERE / "compose.yml"), *args], + cwd=ROOT, env=env, text=True, capture_output=True) + if result.returncode: + (self.private / "compose-error.log").write_text(result.stdout + result.stderr) + raise RuntimeError("v2 Compose failed; see " + str(self.private / "compose-error.log")) + return result.stdout.strip() + + def suspend_main(self): + ids = shell(["docker", "ps", "-q", "--filter", "label=com.docker.compose.project=highload-lab3"]).split() + records = [] + for container in ids: + data = json.loads(shell(["docker", "inspect", "--format", + '{"id":{{json .Id}},"service":{{json (index .Config.Labels "com.docker.compose.service")}}}', container])) + records.append(data) + save(self.private / "original-state.json", records) + if ids: + shell(["docker", "stop", *ids]) + + def restore(self): + errors = [] + try: + self.compose(["stop"]) + except Exception as error: + errors.append(str(error)) + path = self.private / "original-state.json" + records = json.loads(path.read_text()) if path.exists() else [] + for group in (("db", "redis"), ("api",), ("haproxy", "frontend")): + for record in records: + if record["service"] not in group: + continue + try: + project = shell(["docker", "inspect", "--format", '{{index .Config.Labels "com.docker.compose.project"}}', record["id"]]) + if project != "highload-lab3": + raise RuntimeError("Original container identity changed") + shell(["docker", "start", record["id"]]) + deadline = time.monotonic() + 60 + while time.monotonic() < deadline: + state = json.loads(shell(["docker", "inspect", "--format", "{{json .State}}", record["id"]])) + if state["Running"] and state.get("Health", {}).get("Status", "healthy") == "healthy": + break + time.sleep(1) + else: + raise RuntimeError("Original container did not become healthy: " + record["service"]) + except Exception as error: + errors.append(str(error)) + save(self.private / "restoration.json", {"ok": not errors, "errors": errors}) + return not errors + + def prepare_services(self, config): + if self.db_cpu != config["db_cpu"]: + self.compose(["up", "-d", "--wait", "db", "redis"], config) + # One explicit migration job, not migrations in each API. + self.compose(["run", "--rm", "--no-deps", "migrate"], config) + self.db_cpu = config["db_cpu"] + self.compose(["up", "-d", "--no-deps", "--force-recreate", "--wait", "api"], config) + self.compose(["up", "-d", "--no-deps", "haproxy"], config) + + def execute(self, point, directory, protocol_sha, timings, sequence): + if directory.exists(): + raise RuntimeError("Run already exists; raw measurements are never overwritten: " + str(directory)) + directory.mkdir(parents=True) + config = point["config"] + self.prepare_services(config) + actors = original.restore_fixture() + actual = original.current_containers() + apis = [row for row in actual if row["service"] == "api"] + for name, cpu, memory in (("api", config["cpu"], int(config["memory"].removesuffix("m")) * 1048576), + ("db", config["db_cpu"], 512 * 1048576), ("redis", .25, 128 * 1048576), + ("haproxy", .5, 128 * 1048576)): + rows = [row for row in actual if row["service"] == name] + if not rows or any(row["cpu"] != cpu * 1e9 or row["memory"] != memory for row in rows): + raise RuntimeError("Effective limits differ from protocol: " + name) + if len(apis) != config["replicas"] or any(row["ports"] for row in apis): + raise RuntimeError("API replica count or exposed ports differ") + manifest = {**point, "run_id": directory.name, "sequence": sequence, "protocol_sha256": protocol_sha, + "code_sha256": original.code_hash(), "containers": actual, + "docker_cpus": self.env_info["docker_resources"]["cpus"], + "started_utc": datetime.now(timezone.utc).isoformat()} + save(directory / "manifest.json", manifest) + result = asyncio.run(workload.measure(actors, directory, config, rate=point["rate"], + duration=timings["duration_s"], warmup=timings["warmup_s"], seed=230300, + prefix=self.series + ":" + str(sequence), limit=8192)) + samples = json.loads((directory / "pool.json").read_text()) + expected = {row["instance"] for row in apis} + observed = {row["instance"] for row in samples} + runtime_ok = observed == expected and all(row.get("gomaxprocs") == config["gomaxprocs"] and row["max"] == config["pool"] for row in samples) + result["runtime_settings_verified"] = runtime_ok + result["valid"] = result["valid"] and runtime_ok + result["slo_pass"] = result["slo_pass"] and result["valid"] + result["rate"] = point["rate"] + save(directory / "summary.json", result) + diagnostic = legacy_analysis.diagnostics(directory, manifest) + save(directory / "diagnostics.json", diagnostic) + print(json.dumps({"run": directory.name, "rate": point["rate"], "p95_ms": result["p95_ms"], + "success": result["success_ratio"], "valid": result["valid"], "slo_pass": result["slo_pass"]}), flush=True) + return result + + async def sample_db(self, container): + process = await asyncio.create_subprocess_exec("docker", "exec", "-i", "-e", "PGOPTIONS=-c statement_timeout=2000", container, + "psql", "-X", "-q", "-t", "-A", "-v", "ON_ERROR_STOP=1", "-U", "benchmark", "-d", "benchmark", + stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) + try: + stdout, _ = await asyncio.wait_for(process.communicate(DB_SAMPLE.encode()), 4) + except BaseException: + if process.returncode is None: + process.kill() + await process.communicate() + raise + if process.returncode: + raise RuntimeError("DB telemetry unavailable") + return json.loads(stdout) + + async def collect(self, stop, containers, rows): + db = next(row["id"] for row in containers if row["service"] == "db") + async with aiohttp.ClientSession(connector=aiohttp.UnixConnector(path=self.socket), timeout=aiohttp.ClientTimeout(total=4)) as engine, \ + aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=4)) as http: + async def stats(container): + async with engine.get(f"http://docker/v1.47/containers/{container['id']}/stats?stream=false&one-shot=true") as response: + data = await response.json() + cpu = dict(data.get("cpu_stats") or {}) + observed = cpu.get("online_cpus") + cpu["online_cpus"] = observed or self.env_info["docker_resources"]["cpus"] + return {"instance": container["instance"], "service": container["service"], "cpu_stats": cpu, + "cpu_count_source": "stats" if observed else "docker_info", "memory_stats": data.get("memory_stats"), + "blkio_stats": data.get("blkio_stats"), "stats_available": response.status == 200} + while not stop.is_set(): + begin = time.monotonic() + try: + resources, database = await asyncio.gather(asyncio.gather(*(stats(row) for row in containers)), self.sample_db(db)) + haproxy, stats_error = [], None + try: + async with http.get(workload.STATS) as response: + haproxy = [row for row in csv.DictReader(io.StringIO((await response.text()).removeprefix("# "))) + if row["pxname"] in ("api_replicas", "api_entry")] + except (aiohttp.ClientError, TimeoutError): + stats_error = "unavailable" + rows.append({"time": datetime.now(timezone.utc).isoformat(), "resources": resources, + "postgres": database, "haproxy": haproxy, "haproxy_stats_error": stats_error, + "generator_cpu_seconds": time.process_time(), "collection_seconds": time.monotonic() - begin}) + except Exception as error: + rows.append({"time": datetime.now(timezone.utc).isoformat(), "error": type(error).__name__}) + try: + await asyncio.wait_for(stop.wait(), max(.01, 1 - (time.monotonic() - begin))) + except TimeoutError: + pass diff --git a/experiments/lab3-scale-v2/test_experiment.py b/experiments/lab3-scale-v2/test_experiment.py new file mode 100644 index 00000000..a6713f12 --- /dev/null +++ b/experiments/lab3-scale-v2/test_experiment.py @@ -0,0 +1,84 @@ +"""Checks for scientific decision gates, censoring and paired design.""" +import unittest + +from design import CONFIGS, bracket, config, freeze_confirmation, next_rate, pair_decision +from results import capacities, db_results, gomax_results +from runtime import legacy_analysis + + +def observation(rate, passed=True, valid=True): + return {"rate": rate, "slo_pass": passed, "valid": valid} + + +class DesignTests(unittest.TestCase): + def test_800_is_not_a_ceiling(self): + points = [observation(rate) for rate in (100, 200, 400, 800)] + self.assertEqual(next_rate(points), 1600) + self.assertIsNone(bracket(points)["upper"]) + points += [observation(1600), observation(3200), observation(6400)] + self.assertIsNone(next_rate(points)) + self.assertFalse(bracket(points)["closed"]) + + def test_search_floor_bisection_invalid_and_nonmonotonic(self): + self.assertEqual(next_rate([observation(100, False)]), 50) + self.assertIsNone(next_rate([observation(25, False)])) + self.assertEqual(next_rate([observation(100), observation(200, False)]), 150) + self.assertIsNone(next_rate([observation(100), observation(109, False)])) + self.assertIsNone(next_rate([observation(100, valid=False)])) + self.assertTrue(bracket([observation(100, False), observation(200)])["nonmonotonic"]) + + def test_confirmation_is_fresh_and_pairs_change_one_factor(self): + pilot = {name: [observation(400), observation(440, False)] for name in CONFIGS} + plan = freeze_confirmation(pilot) + self.assertEqual(len(plan["scaling"]), 54) + self.assertEqual(len(plan["gomax"]), 20) + self.assertEqual(len(plan["database"]), 12) + self.assertEqual(plan, freeze_confirmation(pilot)) + for phase, field in (("gomax", "gomaxprocs"), ("database", "db_cpu")): + for index in range(0, len(plan[phase]), 2): + a, b = plan[phase][index:index + 2] + self.assertEqual(a["rate"], b["rate"]) + self.assertEqual(a["repeat"], b["repeat"]) + self.assertEqual({k: v for k, v in a["config"].items() if k != field}, {k: v for k, v in b["config"].items() if k != field}) + self.assertNotEqual(a["config"][field], b["config"][field]) + self.assertEqual({p["config"]["gomaxprocs"] for p in plan["gomax"]}, {1, 16}) + + def test_open_bracket_skips_causal_tests(self): + plan = freeze_confirmation({name: [observation(800)] for name in CONFIGS}) + self.assertFalse(plan["gomax"]) + self.assertFalse(plan["database"]) + + def test_mean_does_not_hide_a_failed_pair(self): + pairs = [{"valid": True, "criteria_met": effect >= 15, "reduction_percent": effect} for effect in (60, 60, 60, 60, 0)] + self.assertEqual(pair_decision(pairs)["result"], "inconclusive") + self.assertEqual(pair_decision(pairs)["reduction_mean_percent"], 48) + self.assertEqual(pair_decision(pairs[:4])["result"], "inconclusive") + self.assertEqual(pair_decision([dict(pairs[0]) for _ in range(5)])["result"], "supported") + self.assertEqual(pair_decision([dict(pairs[-1]) for _ in range(5)])["result"], "refuted") + pairs[0]["valid"] = False + self.assertEqual(pair_decision(pairs)["result"], "inconclusive") + + def test_missing_or_censored_scaling_is_inconclusive(self): + runs = [{**observation(800), "phase": "scaling", "base": name, "repeat": repeat} for name in CONFIGS for repeat in (1, 2, 3)] + caps = capacities(runs, runs) + decisions = legacy_analysis.hypothesis_decisions(caps) + self.assertEqual(decisions["H2"]["result"], "inconclusive") + self.assertTrue(all(row["capacity_upper"] is None for row in caps)) + self.assertEqual(legacy_analysis.hypothesis_decisions(capacities(runs[:-1], runs))["H2"]["result"], "inconclusive") + + def test_db_needs_cpu_evidence_not_just_faster_result(self): + points = [] + for repeat in (1, 2, 3): + for variant in ("D0", "D1"): + for rate in (400, 440): + points.append({"phase": "database", "variant": variant, "rate": rate, "repeat": repeat, + "valid": True, "correctness": {"passed": True}, "slo_pass": rate == 400 or variant == "D1", + "diagnostics": {"cpu_fraction_of_limit_mean": {"db": .95, "api": .6, "haproxy": .2}}}) + plan = {"pilot_bounds": {"B4": {"lower": 400, "upper": 440}}} + self.assertEqual(db_results(points, plan)["result"], "supported") + points[1]["diagnostics"]["cpu_fraction_of_limit_mean"]["db"] = .4 + self.assertEqual(db_results(points, plan)["result"], "inconclusive") + + +if __name__ == "__main__": + unittest.main()