forked from mtusi2026/mtusi2026lab2
Measure upload memory and bound the buffers along the chain
upload_memory.py recreates the replicas and the balancer for every run of 10, 20 and 40 concurrent uploads of 1 and 10 MiB with 64 slots (effectively no limit) and with 3, drives them through lab4_seed.py from the prepared tooling, and keeps per-run JSON plus a summary with the cgroup memory.peak of each replica and of the balancer and the oom_kill counters. The seed script now also records memory.current before the load and reads X-Instance-Id regardless of case. The first run showed that slots alone only move a waiting body out of the worker: with 3 slots and 40 uploads of 10 MiB the kernel grew the socket buffers of HAProxy to about 110 MB and the 128 MiB balancer was killed. HAProxy now caps its client and server socket buffers at 64 KiB and the API replicas cap tcp_rmem and tcp_wmem at 256 KiB, which keeps the balancer at 30 MiB. glibc raised its mmap threshold after the first large free and kept freed 10 MiB bodies in the heap, so the API fixes MALLOC_MMAP_THRESHOLD_ at 128 KiB. With these settings 40 concurrent uploads of 10 MiB peak at 323 MiB of 512 per replica and all 80 succeed, while without the slot limit the same load reaches 512 MiB, a worker is killed and only 31 of 80 succeed. UPLOAD_CONCURRENCY stays at 3. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
98df351235
commit
a774a2fbbd
@@ -48,3 +48,5 @@ UPLOAD_CONCURRENCY=3
|
||||
UPLOAD_GRACE_SECONDS=900
|
||||
UPLOAD_SWEEP_INTERVAL_SECONDS=30
|
||||
UPLOAD_SWEEP_BATCH=100
|
||||
# Fixed glibc mmap threshold for the API: buffers above 128 KiB are returned to the system on free.
|
||||
MALLOC_MMAP_THRESHOLD_=131072
|
||||
|
||||
@@ -10,7 +10,8 @@ COMPOSE ?= docker compose
|
||||
RATE ?= 100
|
||||
API_IMAGE ?= mtusi-messenger:lab4
|
||||
FROM_REV ?=
|
||||
SCRIPTS_PY = scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py
|
||||
SCRIPTS_PY = scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py \
|
||||
scripts/upload_memory.py
|
||||
SCENARIO ?= failover-upload
|
||||
PSQL = $(COMPOSE) exec -T postgres psql -U messenger -d messenger -X -q
|
||||
|
||||
@@ -19,7 +20,7 @@ PSQL = $(COMPOSE) exec -T postgres psql -U messenger -d messenger -X -q
|
||||
secrets up down reset-data migrate ps logs restore lb-check failover-evidence stack-evidence \
|
||||
persistence-evidence storage-evidence cache-evidence test-faults seed explain \
|
||||
limits detection failover-each rolling-update level4 \
|
||||
scenario s3-check attachments-evidence upload-faults
|
||||
scenario s3-check attachments-evidence upload-faults upload-memory
|
||||
|
||||
help:
|
||||
@echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes"
|
||||
@@ -29,6 +30,7 @@ help:
|
||||
@echo "level4 = limits, lb-check, detection, failover-each, rolling-update [FROM_REV=<коммит>]; RATE=$(RATE) rps"
|
||||
@echo "scenario SCENARIO=failover-upload|s3-outage → evidence/<сценарий>.json; s3-check; attachments-evidence"
|
||||
@echo "upload-faults: негативный suite, компенсация, уборка, обрыв и зависание загрузки против стенда"
|
||||
@echo "upload-memory: пик памяти реплик при 10/20/40 одновременных загрузках 1 и 10 MiB → evidence/upload-memory/"
|
||||
|
||||
check:
|
||||
$(PYTHON) scripts/check_contract.py
|
||||
@@ -133,6 +135,9 @@ upload-faults:
|
||||
student-tests/stack/test_attachment_faults.py > evidence/upload-faults.txt; \
|
||||
status=$$?; cat evidence/upload-faults.txt; exit $$status
|
||||
|
||||
upload-memory:
|
||||
$(PYTHON) scripts/upload_memory.py --base-url "$(BASE_URL)"
|
||||
|
||||
failover-evidence:
|
||||
sh scripts/failover-evidence.sh "$(BASE_URL)"
|
||||
|
||||
|
||||
@@ -82,6 +82,7 @@ services:
|
||||
DRAIN_SECONDS: ${DRAIN_SECONDS:-2}
|
||||
SHUTDOWN_TIMEOUT_SECONDS: ${SHUTDOWN_TIMEOUT_SECONDS:-5}
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
MALLOC_MMAP_THRESHOLD_: ${MALLOC_MMAP_THRESHOLD_:-131072}
|
||||
secrets: [database_url, cache_url, cursor_secret, s3_access_key_id, s3_secret_access_key]
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import sys,urllib.request;sys.exit(urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=2).status!=200)"]
|
||||
@@ -94,6 +95,9 @@ services:
|
||||
restart: unless-stopped
|
||||
read_only: true
|
||||
tmpfs: [/tmp]
|
||||
sysctls:
|
||||
net.ipv4.tcp_rmem: "4096 65536 262144"
|
||||
net.ipv4.tcp_wmem: "4096 65536 262144"
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
|
||||
@@ -3,6 +3,10 @@ global
|
||||
maxconn 4096
|
||||
hard-stop-after 5s
|
||||
tune.bufsize 32768
|
||||
tune.rcvbuf.client 65536
|
||||
tune.rcvbuf.server 65536
|
||||
tune.sndbuf.client 65536
|
||||
tune.sndbuf.server 65536
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
|
||||
@@ -57,7 +57,14 @@ Grace period `UPLOAD_GRACE_SECONDS=900`, не меньше 60 (валидаци
|
||||
|
||||
`UploadSlots` — `asyncio.Semaphore(UPLOAD_CONCURRENCY)` на воркер (scope `APP`). Обработчик сначала проверяет членство, `filename` и `Content-Type` (дёшево, без тела), затем под слотом читает тело, проверяет изображение, пишет `pending` и делает `PutObject`. Буфер живёт только внутри метода записи и освобождается до перевода в `ready`. Без слота тело не читается: uvicorn приостанавливает чтение сокета, когда в его буфере больше 64 KiB, поэтому ждущая загрузка занимает десятки KiB; очередь ограничена `maxconn 64` HAProxy на реплику и таймаутом запроса (дольше 8 с — `503`).
|
||||
|
||||
Оценка: одна загрузка 10 MiB — тело (10 MiB) плюс склейка (+10 MiB на миг) плюс копия в клиенте S3 — до ~30 MiB. `UPLOAD_CONCURRENCY=3` → 2 воркера × 3 × 30 MiB ≈ 180 MiB поверх простоя реплики при лимите 512 MiB. Значение подтверждается замером D5, при расхождении меняется по его результату.
|
||||
Оценка до замера: одна загрузка 10 MiB — тело (10 MiB) плюс склейка (+10 MiB на миг) плюс копия в клиенте S3 — до ~30 MiB; `UPLOAD_CONCURRENCY=3` → 2 воркера × 3 × 30 MiB ≈ 180 MiB поверх простоя реплики при лимите 512 MiB.
|
||||
|
||||
Замер D5 уточнил картину в двух местах, и решение дополнено:
|
||||
|
||||
- **Ожидающая загрузка живёт в буферах сокетов.** Слот держит тело вне памяти воркера, но клиент продолжает слать, и байты копятся в ядре по цепочке: автонастройка TCP растит буфер приёма до 6 MiB на соединение, а память сокетов учитывается в cgroup контейнера. Первый прогон с 3 слотами при 40 одновременных загрузках по 10 MiB убил HAProxy по OOM (лимит 128 MiB, `memory.stat sock` ≈ 110 MB), соединения оборвались. Поэтому HAProxy получает `tune.rcvbuf.client/server` и `tune.sndbuf.client/server` по 64 KiB, а у `api` — `net.ipv4.tcp_rmem`/`tcp_wmem` с максимумом 256 KiB (sysctl сетевого namespace в `compose.yaml`). Пик LB после этого — 30 MiB; на пропускную способность внутри одного хоста буферы не влияют. Ожидающая загрузка стоит ~0,3 MiB на LB и столько же на реплике; число таких соединений ограничивает `maxconn 64` на реплику.
|
||||
- **Аллокатор glibc не отдаёт освобождённые тела.** Тела больше 128 KiB берутся у `malloc`; после первого освобождения крупного блока glibc поднимает порог `mmap` до 32 MiB, и следующие буферы по 10 MiB остаются в куче процесса после освобождения. `MALLOC_MMAP_THRESHOLD_=131072` у `api` фиксирует порог: большие буферы выделяются `mmap` и возвращаются системе. При 3 слотах и 40 загрузках по 10 MiB пик реплики — 323 MiB против 364 MiB с порогом 32 MiB.
|
||||
|
||||
Итог замера: одна загрузка 10 MiB в полёте стоит ≈ 34 MiB, 6 слотов на реплику — ≈ 200 MiB поверх простоя ~120 MiB, пик 323 MiB (63 % лимита) при любой одновременности от 20 до 40. Без лимита (64 слота) те же 40 загрузок упираются в 512 MiB, ядро убивает воркер, и успешны лишь 31 из 80. Значение `UPLOAD_CONCURRENCY=3` оставлено.
|
||||
|
||||
### D4. Обрыв и зависание тела
|
||||
|
||||
@@ -72,7 +79,7 @@ Grace period `UPLOAD_GRACE_SECONDS=900`, не меньше 60 (валидаци
|
||||
- `created_at` брошенного намерения сдвигается на 2 × grace назад — так проверка не ждёт 15 минут; свежее намерение остаётся и переживает проход;
|
||||
- обрыв и зависание тела — сырым сокетом через LB с собственным `X-Request-Id`, статус ищется в журнале реплики.
|
||||
|
||||
Замер памяти: `scripts/lab4_seed.py` из заготовки (режим `http`, `--sample`: `docker stats` в фоне и `memory.peak` cgroup в конце; добавлен `memory.current` до нагрузки) и `scripts/upload-memory.sh`: для `UPLOAD_CONCURRENCY` ∈ {64 (фактически без лимита), 3} × размер ∈ {1, 10 MiB} × одновременность ∈ {10, 20, 40} пересоздаёт реплики (`memory.peak` на ядре Docker Desktop не сбрасывается), ждёт LB и делает 2 × c загрузок; сводка — `evidence/upload-memory/summary.txt`.
|
||||
Замер памяти: `scripts/lab4_seed.py` из заготовки (режим `http`, `--sample`: `docker stats` в фоне и `memory.peak` cgroup в конце; добавлены `memory.current` до нагрузки и счётчик `oom_kill` из `memory.events`, заголовок `X-Instance-Id` читается без учёта регистра) и `scripts/upload_memory.py` (Python вместо задуманного `upload-memory.sh`: сводка и вызов генератора проще без оболочки): для `UPLOAD_CONCURRENCY` ∈ {64 (фактически без лимита), 3} × размер ∈ {1, 10 MiB} × одновременность ∈ {10, 20, 40} пересоздаёт реплики и LB (`memory.peak` на ядре Docker Desktop не сбрасывается), ждёт LB и делает 2 × c загрузок; пик LB снимается отдельно; сводка — `evidence/upload-memory/summary.txt`, сравнение порога `mmap` — `evidence/upload-memory/mmap-32m/`.
|
||||
|
||||
### D6. Проверка S3
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@
|
||||
## Impact
|
||||
|
||||
- Код: зона `attachments` (домен, обработчики, адаптер PostgreSQL, настройки, фоновая уборка, HTTP-слой), `main.py` (запуск уборки); схема БД не меняется.
|
||||
- Стенд: `compose.yaml`, `.env.example`, `scripts/lab4_seed.py` и `scripts/upload-memory.sh` из заготовки, `scripts/s3-check.sh`, `Makefile`, CI.
|
||||
- Стенд: `compose.yaml` (буферы сокетов и порог `mmap` у `api`), `infra/haproxy/haproxy.cfg` (буферы сокетов LB), `.env.example`, `scripts/lab4_seed.py` из заготовки и `scripts/upload_memory.py`, `scripts/s3-check.sh`, `Makefile`, CI.
|
||||
- Тесты: `student-tests/api` (компенсация, уборка, слоты, обрыв, таймаут), `student-tests/stack` (стенд).
|
||||
- Отчёт: строки 4.1–4.4, таблица отказов, раздел эксперимента, `evidence/`.
|
||||
|
||||
|
||||
@@ -29,8 +29,9 @@
|
||||
|
||||
## 5. Замер памяти
|
||||
|
||||
- [ ] 5.1 `scripts/lab4_seed.py` из заготовки (`memory.current` до нагрузки), `scripts/upload-memory.sh`, цель `make upload-memory`, ruff для скрипта
|
||||
- [ ] 5.2 Проверка группы: прогон 64/3 слота × 1/10 MiB × 10/20/40; выбор `UPLOAD_CONCURRENCY` по результату
|
||||
- [x] 5.1 `scripts/lab4_seed.py` из заготовки (`memory.current` до нагрузки, `oom_kill`), `scripts/upload_memory.py`, цель `make upload-memory`, ruff для скриптов
|
||||
- [x] 5.2 Буферы сокетов по цепочке (HAProxy `tune.*buf`, `tcp_rmem`/`tcp_wmem` у `api`) и фиксированный порог `mmap` glibc по результатам первого прогона
|
||||
- [x] 5.3 Проверка группы: прогон 64/3 слота × 1/10 MiB × 10/20/40 и сравнение порога `mmap`; `UPLOAD_CONCURRENCY=3` подтверждено
|
||||
|
||||
## 6. Отчёт
|
||||
|
||||
|
||||
Executable
+352
@@ -0,0 +1,352 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import base64
|
||||
import hashlib
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAX_UPLOAD = 10 * 1024 * 1024
|
||||
COLUMNS = (
|
||||
"id",
|
||||
"chat_id",
|
||||
"owner_id",
|
||||
"filename",
|
||||
"content_type",
|
||||
"size_bytes",
|
||||
"sha256",
|
||||
"width",
|
||||
"height",
|
||||
"object_key",
|
||||
"status",
|
||||
)
|
||||
|
||||
|
||||
def chunk(tag, data):
|
||||
return (
|
||||
struct.pack(">I", len(data))
|
||||
+ tag
|
||||
+ data
|
||||
+ struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF)
|
||||
)
|
||||
|
||||
|
||||
def png(width, height):
|
||||
raw = b"".join(b"\x00" + os.urandom(width * 3) for _ in range(height))
|
||||
header = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)
|
||||
return (
|
||||
b"\x89PNG\r\n\x1a\n"
|
||||
+ chunk(b"IHDR", header)
|
||||
+ chunk(b"IDAT", zlib.compress(raw, 1))
|
||||
+ chunk(b"IEND", b"")
|
||||
)
|
||||
|
||||
|
||||
def sized_png(target):
|
||||
height = max(1, target // (1024 * 3 + 1))
|
||||
body = png(1024, height)
|
||||
while len(body) > min(target, MAX_UPLOAD) and height > 1:
|
||||
height -= max(1, (len(body) - target) // 3073 + 1)
|
||||
body = png(1024, height)
|
||||
return body
|
||||
|
||||
|
||||
def run(cmd, stdin=None):
|
||||
out = subprocess.run(cmd, cwd=ROOT, input=stdin, capture_output=True, check=False)
|
||||
if out.returncode:
|
||||
raise SystemExit(
|
||||
f"command failed: {' '.join(cmd[:6])}\n{out.stderr.decode(errors='replace')[-500:]}"
|
||||
)
|
||||
return out.stdout.decode()
|
||||
|
||||
|
||||
def psql(sql, stdin=None):
|
||||
user, db = os.environ.get("PGUSER", "messenger"), os.environ.get("PGDATABASE", "messenger")
|
||||
psql = [
|
||||
"docker",
|
||||
"compose",
|
||||
"exec",
|
||||
"-T",
|
||||
"postgres",
|
||||
"psql",
|
||||
"-v",
|
||||
"ON_ERROR_STOP=1",
|
||||
"-qAt",
|
||||
"-F",
|
||||
" ",
|
||||
]
|
||||
return run([*psql, "-U", user, "-d", db, "-c", sql], stdin)
|
||||
|
||||
|
||||
def direct(args):
|
||||
bucket = os.environ.get("S3_BUCKET", "messenger-attachments")
|
||||
prefix = os.environ.get("S3_PREFIX", "attachments/")
|
||||
if args.chat_id and args.owner_id:
|
||||
chat_id, owner_id = args.chat_id, args.owner_id
|
||||
else:
|
||||
found = psql(
|
||||
"SELECT chat_id, owner_id FROM attachments ORDER BY created_at LIMIT 1"
|
||||
).split()
|
||||
if len(found) != 2:
|
||||
raise SystemExit(
|
||||
"no attachment rows yet: upload one via API or pass --chat-id and --owner-id"
|
||||
)
|
||||
chat_id, owner_id = found
|
||||
rows = []
|
||||
started = time.monotonic()
|
||||
with tempfile.TemporaryDirectory(prefix="lab4-seed-") as tmp:
|
||||
Path(tmp).chmod(0o755)
|
||||
for i in range(args.count):
|
||||
aid = str(uuid.uuid4())
|
||||
body = png(8, 8)
|
||||
path = Path(tmp) / aid
|
||||
path.write_bytes(body)
|
||||
path.chmod(0o644)
|
||||
digest = hashlib.sha256(body).hexdigest()
|
||||
rows.append(
|
||||
f"{aid},{chat_id},{owner_id},seed-{i}.png,image/png,{len(body)},{digest},8,8,{prefix}{aid},ready"
|
||||
)
|
||||
generated = time.monotonic()
|
||||
cli = [
|
||||
"docker",
|
||||
"compose",
|
||||
"run",
|
||||
"--rm",
|
||||
"--no-deps",
|
||||
"-T",
|
||||
"-v",
|
||||
f"{tmp}:/seed:ro",
|
||||
"s3-cli",
|
||||
]
|
||||
run([*cli, "put", "--recursive", "--no-progress", "/seed/", f"root/{bucket}/{prefix}"])
|
||||
uploaded = time.monotonic()
|
||||
columns = ", ".join(COLUMNS)
|
||||
psql(
|
||||
f"COPY attachments ({columns}) FROM STDIN WITH (FORMAT csv)",
|
||||
"".join(f"{r}\n" for r in rows).encode(),
|
||||
)
|
||||
seconds = {
|
||||
"generate": round(generated - started, 3),
|
||||
"s3_put": round(uploaded - generated, 3),
|
||||
"pg_copy": round(time.monotonic() - uploaded, 3),
|
||||
}
|
||||
return {"mode": "direct", "count": args.count, "seconds": seconds}
|
||||
|
||||
|
||||
class Http:
|
||||
def __init__(self, base, timeout):
|
||||
parts = urlsplit(base)
|
||||
self.cls = (
|
||||
http.client.HTTPSConnection if parts.scheme == "https" else http.client.HTTPConnection
|
||||
)
|
||||
self.host, self.timeout, self.local = parts.netloc, timeout, threading.local()
|
||||
|
||||
def call(self, method, path, body=None, headers=None, token=None):
|
||||
conn = getattr(self.local, "conn", None) or self.cls(self.host, timeout=self.timeout)
|
||||
self.local.conn = conn
|
||||
h = {"Accept": "application/json"}
|
||||
if token:
|
||||
h["Authorization"] = "Bearer " + token
|
||||
if isinstance(body, dict):
|
||||
body, h["Content-Type"] = json.dumps(body).encode(), "application/json"
|
||||
h.update(headers or {})
|
||||
try:
|
||||
conn.request(method, path, body=body, headers=h)
|
||||
response = conn.getresponse()
|
||||
data = response.read()
|
||||
except (OSError, http.client.HTTPException):
|
||||
conn.close()
|
||||
self.local.conn = None
|
||||
return 0, {}, b""
|
||||
return response.status, {k.lower(): v for k, v in response.getheaders()}, data
|
||||
|
||||
|
||||
def login(api, label):
|
||||
name = f"seed_{label}_{uuid.uuid4().hex[:12]}"
|
||||
password = "Seed-" + uuid.uuid4().hex
|
||||
body = {"username": name, "display_name": label, "password": password}
|
||||
status, _, data = api.call("POST", "/api/v1/users", body)
|
||||
if status != 201:
|
||||
raise SystemExit(f"user create failed: {status}")
|
||||
user = json.loads(data)
|
||||
basic = base64.b64encode(f"{name}:{password}".encode()).decode()
|
||||
status, _, data = api.call(
|
||||
"POST", "/api/v1/auth/sessions", headers={"Authorization": f"Basic {basic}"}
|
||||
)
|
||||
if status != 201:
|
||||
raise SystemExit(f"login failed: {status}")
|
||||
return user["id"], json.loads(data)["token"]
|
||||
|
||||
|
||||
def mib(text):
|
||||
m = re.match(r"([\d.]+)\s*([KMG]i?B)", text)
|
||||
if not m:
|
||||
return 0.0
|
||||
scale = {"KiB": 1 / 1024, "KB": 1 / 1024, "GiB": 1024, "GB": 1024}
|
||||
return float(m.group(1)) * scale.get(m.group(2), 1)
|
||||
|
||||
|
||||
def sampler(ids, peaks, stop):
|
||||
while not stop.is_set():
|
||||
cmd = ["docker", "stats", "--no-stream", "--format", "{{.ID}} {{.MemUsage}}", *ids]
|
||||
out = subprocess.run(cmd, capture_output=True, text=True, check=False).stdout
|
||||
for line in out.splitlines():
|
||||
cid, usage = line.split(" ", 1)
|
||||
peaks[cid] = max(peaks.get(cid, 0.0), mib(usage))
|
||||
|
||||
|
||||
def cgroup(cid, name):
|
||||
cmd = ["docker", "exec", cid, "cat", f"/sys/fs/cgroup/{name}"]
|
||||
return subprocess.run(cmd, capture_output=True, text=True, check=False).stdout.strip()
|
||||
|
||||
|
||||
def percentile(values, q):
|
||||
return (
|
||||
round(sorted(values)[min(len(values) - 1, int(q * len(values)))] * 1000, 1)
|
||||
if values
|
||||
else None
|
||||
)
|
||||
|
||||
|
||||
def over_http(args):
|
||||
api = Http(args.base_url, args.timeout)
|
||||
alice, token = login(api, "alice")
|
||||
bob, _ = login(api, "bob")
|
||||
status, _, data = api.call(
|
||||
"POST", "/api/v1/chats", {"title": "seed", "member_ids": [alice, bob]}, token=token
|
||||
)
|
||||
if status != 201:
|
||||
raise SystemExit(f"chat create failed: {status}")
|
||||
chat = json.loads(data)["id"]
|
||||
fixed = sized_png(args.size) if args.size else None
|
||||
counter, lock, results = iter(range(args.count)), threading.Lock(), []
|
||||
ids = run(["docker", "compose", "ps", "-q", args.service]).split() if args.sample else []
|
||||
idle = {cid: round(int(cgroup(cid, "memory.current") or 0) / 1048576, 1) for cid in ids}
|
||||
peaks, stop = {}, threading.Event()
|
||||
watcher = threading.Thread(target=sampler, args=(ids, peaks, stop), daemon=True)
|
||||
if ids:
|
||||
watcher.start()
|
||||
|
||||
def worker():
|
||||
while True:
|
||||
with lock:
|
||||
i = next(counter, None)
|
||||
if i is None:
|
||||
return
|
||||
body = fixed or png(8, 8)
|
||||
began = time.monotonic()
|
||||
path = f"/api/v1/chats/{chat}/attachments?filename=seed-{i}.png"
|
||||
status, headers, data = api.call(
|
||||
"POST", path, body, {"Content-Type": "image/png"}, token
|
||||
)
|
||||
created = json.loads(data)["id"] if status == 201 else None
|
||||
results.append(
|
||||
(status, time.monotonic() - began, headers.get("x-instance-id", ""), created)
|
||||
)
|
||||
|
||||
started = time.monotonic()
|
||||
threads = [threading.Thread(target=worker) for _ in range(args.concurrency)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
elapsed = time.monotonic() - started
|
||||
stop.set()
|
||||
if ids:
|
||||
watcher.join()
|
||||
created = [r[3] for r in results if r[3]]
|
||||
for start in range(0, min(args.link, len(created)), 10):
|
||||
body = {"text": "", "attachment_ids": created[start : min(start + 10, args.link)]}
|
||||
api.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat}/messages",
|
||||
body,
|
||||
{"Idempotency-Key": uuid.uuid4().hex},
|
||||
token,
|
||||
)
|
||||
if args.ids_out:
|
||||
Path(args.ids_out).write_text("".join(f"{i}\n" for i in created))
|
||||
codes, instances = {}, {}
|
||||
for status, _, instance, _ in results:
|
||||
codes[str(status)] = codes.get(str(status), 0) + 1
|
||||
instances[instance] = instances.get(instance, 0) + 1
|
||||
ok = [r[1] for r in results if r[0] == 201]
|
||||
memory = {}
|
||||
for cid in ids:
|
||||
peak = cgroup(cid, "memory.peak")
|
||||
events = dict(line.split() for line in cgroup(cid, "memory.events").splitlines())
|
||||
memory[cid[:12]] = {
|
||||
"idle_mib": idle[cid],
|
||||
"docker_stats_peak_mib": round(peaks.get(cid[:12], 0.0), 1),
|
||||
"cgroup_memory_peak_mib": round(int(peak) / 1048576, 1) if peak.isdigit() else None,
|
||||
"oom_kill": int(events.get("oom_kill", 0)),
|
||||
}
|
||||
return {
|
||||
"mode": "http",
|
||||
"count": args.count,
|
||||
"concurrency": args.concurrency,
|
||||
"body_bytes": len(fixed) if fixed else "~200 (unique 8x8 PNG)",
|
||||
"status_codes": codes,
|
||||
"instances": len([i for i in instances if i]),
|
||||
"linked_in_messages": min(args.link, len(created)),
|
||||
"elapsed_seconds": round(elapsed, 3),
|
||||
"successful_rps": round(len(ok) / elapsed, 2) if elapsed else None,
|
||||
"latency_ms": {
|
||||
"p50": percentile(ok, 0.5),
|
||||
"p95": percentile(ok, 0.95),
|
||||
"p99": percentile(ok, 0.99),
|
||||
"max": percentile(ok, 1.0),
|
||||
},
|
||||
"memory": memory,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(
|
||||
description="Lab4 attachment volume generator: HTTP uploads or direct S3+PG seeding"
|
||||
)
|
||||
p.add_argument("--mode", choices=["http", "direct"], default="http")
|
||||
p.add_argument("--count", type=int, default=1000)
|
||||
p.add_argument("--concurrency", type=int, default=16)
|
||||
p.add_argument(
|
||||
"--size",
|
||||
type=int,
|
||||
default=0,
|
||||
help="http: bytes per upload (one PNG reused); 0 = unique 8x8 PNGs",
|
||||
)
|
||||
p.add_argument(
|
||||
"--link", type=int, default=0, help="http: reference the first N attachments from messages"
|
||||
)
|
||||
p.add_argument(
|
||||
"--sample",
|
||||
action="store_true",
|
||||
help="http: sample docker stats and memory.peak of API containers",
|
||||
)
|
||||
p.add_argument("--service", default="api")
|
||||
p.add_argument("--base-url", default=os.environ.get("BASE_URL", "http://localhost:8080"))
|
||||
p.add_argument("--timeout", type=float, default=60)
|
||||
p.add_argument("--ids-out", metavar="FILE")
|
||||
p.add_argument("--chat-id")
|
||||
p.add_argument("--owner-id")
|
||||
p.add_argument("--output", metavar="FILE")
|
||||
args = p.parse_args()
|
||||
report = direct(args) if args.mode == "direct" else over_http(args)
|
||||
text = json.dumps(report, ensure_ascii=False, indent=2)
|
||||
if args.output:
|
||||
Path(args.output).write_text(text + "\n")
|
||||
print(text)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+117
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "scripts"))
|
||||
seed = __import__("lab4_seed")
|
||||
FORMAT = "{:>5} {:>5} {:>4} {:<18} {:>7} {:>7} {:>8} {:>8} {:>7} {:>8} {:>6}"
|
||||
COLUMNS = FORMAT.format(
|
||||
"slots",
|
||||
"size",
|
||||
"conc",
|
||||
"codes",
|
||||
"p50_ms",
|
||||
"p95_ms",
|
||||
"api_idle",
|
||||
"api_peak",
|
||||
"api_oom",
|
||||
"lb_peak",
|
||||
"lb_oom",
|
||||
)
|
||||
|
||||
|
||||
OVERRIDES = ("UPLOAD_CONCURRENCY", "MALLOC_MMAP_THRESHOLD_")
|
||||
|
||||
|
||||
def recreate(slots):
|
||||
env = dict(os.environ, UPLOAD_CONCURRENCY=str(slots))
|
||||
if not slots:
|
||||
env = {k: v for k, v in os.environ.items() if k not in OVERRIDES}
|
||||
cmd = ["docker", "compose", "up", "-d", "--no-deps", "--force-recreate", "--wait", "api", "lb"]
|
||||
subprocess.run(cmd, cwd=ROOT, env=env, check=True, capture_output=True)
|
||||
subprocess.run([str(ROOT / "scripts/lb-wait.sh")], cwd=ROOT, check=True)
|
||||
|
||||
|
||||
def balancer():
|
||||
cid = seed.run(["docker", "compose", "ps", "-q", "lb"]).strip()
|
||||
events = dict(row.split() for row in seed.cgroup(cid, "memory.events").splitlines())
|
||||
peak = seed.cgroup(cid, "memory.peak")
|
||||
return {
|
||||
"cgroup_memory_peak_mib": round(int(peak) / 1048576, 1),
|
||||
"oom_kill": int(events["oom_kill"]),
|
||||
}
|
||||
|
||||
|
||||
def line(report):
|
||||
memory = report["memory"].values()
|
||||
codes = ",".join(f"{k}x{v}" for k, v in sorted(report["status_codes"].items()))
|
||||
return FORMAT.format(
|
||||
report["upload_concurrency"],
|
||||
f"{report['body_bytes'] / 1048576:.1f}M",
|
||||
report["concurrency"],
|
||||
codes,
|
||||
report["latency_ms"]["p50"],
|
||||
report["latency_ms"]["p95"],
|
||||
max(m["idle_mib"] for m in memory),
|
||||
max(m["cgroup_memory_peak_mib"] or 0 for m in memory),
|
||||
sum(m["oom_kill"] for m in memory),
|
||||
report["lb"]["cgroup_memory_peak_mib"],
|
||||
report["lb"]["oom_kill"],
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description="Peak memory of API replicas under concurrent uploads")
|
||||
p.add_argument("--base-url", default=os.environ.get("BASE_URL", "http://localhost:8080"))
|
||||
p.add_argument("--slots", type=int, nargs="+", default=[64, 3])
|
||||
p.add_argument("--sizes", type=int, nargs="+", default=[1 << 20, 10 << 20])
|
||||
p.add_argument("--levels", type=int, nargs="+", default=[10, 20, 40])
|
||||
p.add_argument("--rounds", type=int, default=2, help="uploads per concurrent client")
|
||||
p.add_argument("--out", default="evidence/upload-memory")
|
||||
args = p.parse_args()
|
||||
out = ROOT / args.out
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
commit = seed.run(["git", "rev-parse", "--short", "HEAD"]).strip()
|
||||
threshold = os.environ.get("MALLOC_MMAP_THRESHOLD_", "131072")
|
||||
lines = [
|
||||
f"# {time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())} commit {commit}; api x2: 1 CPU,"
|
||||
f" 512 MiB, 2 workers; MALLOC_MMAP_THRESHOLD_={threshold}; {args.rounds} uploads"
|
||||
" per client; peaks from cgroup memory.peak, MiB",
|
||||
COLUMNS,
|
||||
]
|
||||
print("\n".join(lines), flush=True)
|
||||
try:
|
||||
for slots in args.slots:
|
||||
for size in args.sizes:
|
||||
for level in args.levels:
|
||||
recreate(slots)
|
||||
run = argparse.Namespace(
|
||||
base_url=args.base_url,
|
||||
timeout=60,
|
||||
size=size,
|
||||
count=level * args.rounds,
|
||||
concurrency=level,
|
||||
sample=True,
|
||||
service="api",
|
||||
link=0,
|
||||
ids_out=None,
|
||||
)
|
||||
report = {"upload_concurrency": slots, **seed.over_http(run)}
|
||||
report["lb"] = balancer()
|
||||
name = f"slots{slots}-{size >> 20}mib-c{level}.json"
|
||||
(out / name).write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n")
|
||||
lines.append(line(report))
|
||||
print(lines[-1], flush=True)
|
||||
finally:
|
||||
recreate(None)
|
||||
(out / "summary.txt").write_text("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user