ЛР 5 — БВТ2301 — Владимирцева — на 5 #23
@@ -0,0 +1,30 @@
|
||||
---
|
||||
name: lab5-bench
|
||||
description: Load measurements of lab 5 for grade 5, track 1 "queue under overload" — worker concurrency/prefetch settings on small, large and huge-pixel images (profile queue), with throughput, queue lag, p95 time-to-ready, worker memory, backpressure (429 at the backlog limit) and recovery. Use when asked to run benchmarks, make bench-queue (or bench-queue-quick), or fill REPORT.md with measured numbers and the outcomes of hypotheses Q1–Q5.
|
||||
---
|
||||
|
||||
# Замеры лабы 5 (трек 1)
|
||||
|
||||
Сначала прочитай целиком (`cat`): **`bench/RUNBOOK.md`** (порядок шагов),
|
||||
`bench/AGENTS.md` (правила стенда), корневой `AGENTS.md` (разделы 2,
|
||||
8–11 действуют всегда).
|
||||
|
||||
Коротко:
|
||||
|
||||
1. Шаг 0 RUNBOOK: ресурсы ВМ Docker, самопроверки разбора
|
||||
(`bench/queue.py --selftest` и остальные), `make test-isolated`,
|
||||
`make failure-checks`, `make stop`, самопроверка `make bench-queue-quick`.
|
||||
Остановка и отчёт пользователю.
|
||||
2. Трек 1: `make bench-queue` → `evidence/queue/baseline/`; таблицы
|
||||
классов и итоги Q1–Q5 из `summary.md` — в `REPORT.md`.
|
||||
3. Последний шаг RUNBOOK: строки `TODO (Codex)` в `REPORT.md`, раздел
|
||||
«Железо и стенд замеров», `make lint`, `make check`, `make
|
||||
test-isolated`, уборка `vlad-bench*`.
|
||||
|
||||
Нельзя: переписывать гипотезы и пороги после замеров, менять код
|
||||
приложения или профиль ради результата, править файлы меток руками,
|
||||
перезаписывать или удалять каталоги меток без согласия, считать числа
|
||||
вручную вместо `summary.md`. Долгое — в фоне с журналом (`nohup make … >
|
||||
/tmp/bench-*.log 2>&1 &`), docker — вне sandbox с одобрения
|
||||
пользователя. `aborted`, `host_warnings`, упавшие проверки — остановиться
|
||||
и сказать пользователю.
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
name: messenger-stand
|
||||
description: Run, check and operate the lab 5 messenger stand — HAProxy with API replicas, PostgreSQL, Valkey, SeaweedFS S3, RabbitMQ, relay, image worker, cleaner. Use when asked to start or stop the stand, run make test / student-test / test-isolated / failure-checks / s3-check, show the defense scenario (worker stopped → 202 → restart → ready), inspect the pipeline state, list or replay failed jobs, stop one replica, run cleanup or reconcile, do a rolling update, or reproduce an API bug over HTTP.
|
||||
---
|
||||
|
||||
# Стенд мессенджера лабы 5
|
||||
|
||||
Команды — корневой `AGENTS.md`, раздел 4; устройство стенда, брокер, S3,
|
||||
worker, таймауты, скрипты — `infra/AGENTS.md`; pipeline —
|
||||
`src/services/AGENTS.md`; тесты — `student-tests/AGENTS.md`.
|
||||
|
||||
| Задача | Команда |
|
||||
|---|---|
|
||||
| поднять стенд целиком | `make up` → `127.0.0.1:${HOST_PORT:-8080}` |
|
||||
| публичный smoke / свои тесты | `make test`, `make student-test` |
|
||||
| всё на чистом отдельном стенде | `make test-isolated` |
|
||||
| проверки отказов (в том числе pipeline) | `make failure-checks [FAULT_ARGS=-rP]` |
|
||||
| сценарий защиты, шаги 2–3 | `make defense` |
|
||||
| задания по состояниям и очередь брокера | `make pipeline-state` |
|
||||
| реестр failed / возврат | `make replay [ERROR_CODE=…]` / `make replay APPLY=1 ATTACHMENT=<id> OPERATOR=<кто> REASON=<зачем>` |
|
||||
| остановить / запустить worker | `make stop-worker` / `make start-worker` |
|
||||
| журнал relay и worker | `make logs-pipeline` |
|
||||
| приватность bucket вне API | `make s3-check` |
|
||||
| вложение после остановки принявшей реплики | `make attachment-failover [STOP_MODE=kill]` |
|
||||
| реплики за LB | `make instances`, `make lb-stats`, `make failover-each` |
|
||||
| уборка / сверка (dry-run) | `make cleanup-once` / `make reconcile [STRATEGY=head] [DEEP=1]` |
|
||||
| обновить код без простоя | `make rolling-update` |
|
||||
| остановить / удалить контейнеры | `make stop` / `make down` (volume остаются) |
|
||||
|
||||
Правила:
|
||||
|
||||
- Порт занят другим сервисом — не убивать процесс: другие порты в `.env`
|
||||
(`make env HOST_PORT=… POSTGRES_HOST_PORT=… VALKEY_HOST_PORT=…
|
||||
S3_HOST_PORT=… BROKER_HOST_PORT=…` до первого `make up`) и
|
||||
`BASE_URL=http://localhost:<порт>` у тестов и `make defense`.
|
||||
- `make defense` останавливает worker и перезапускает хранилища и брокер
|
||||
рабочего стенда (`scripts/restart.sh`); worker запускается при любом
|
||||
исходе.
|
||||
- `make replay APPLY=1` и `make reconcile APPLY=1` меняют рабочий стенд —
|
||||
только с согласия пользователя; без `APPLY` — только показ.
|
||||
- `make wipe`, `docker compose down -v` и любое удаление volume — только
|
||||
с согласия пользователя.
|
||||
- Пароли, ключи S3 и брокера из `.env`, токены не печатать; в
|
||||
`evidence/` — только обезличенный вывод (`evidence/AGENTS.md`).
|
||||
@@ -0,0 +1,187 @@
|
||||
# Правила команд Codex для проекта (execpolicy, Starlark).
|
||||
#
|
||||
# Действуют на команды, которые Codex запускает вне sandbox (docker, сеть):
|
||||
# allow — без вопроса;
|
||||
# prompt — спросить пользователя перед каждым запуском;
|
||||
# forbidden — не запускать никогда.
|
||||
# Несколько совпавших правил — побеждает самое строгое (forbidden →
|
||||
# prompt → allow). pattern — точный префикс аргументов команды; список в
|
||||
# позиции — варианты. match / not_match — примеры, которые Codex
|
||||
# проверяет при загрузке правил.
|
||||
#
|
||||
# Файл загружается, только если проект доверенный (слой .codex/ проекта
|
||||
# trusted). После правки — перезапустить Codex. Проверка правила (команда —
|
||||
# без «--» перед ней: с ним ничего не сопоставляется):
|
||||
# codex execpolicy check --pretty --rules "$PWD/.codex/rules/messenger.rules" git push
|
||||
#
|
||||
# Источник запретов — корневой AGENTS.md, раздел 2 «Нельзя»; правила их
|
||||
# дублируют на уровне команд, но не заменяют.
|
||||
|
||||
# --- git: только смотреть; изменения — по просьбе пользователя -------------
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["git", ["status", "diff", "log", "show", "rev-parse", "ls-files"]],
|
||||
decision = "allow",
|
||||
justification = "Read-only git commands are always fine.",
|
||||
match = ["git status", "git diff --stat", "git log --oneline -5"],
|
||||
not_match = ["git commit -m x"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["git", ["commit", "checkout", "switch", "reset", "stash", "rebase", "merge", "restore", "tag", "branch"]],
|
||||
decision = "prompt",
|
||||
justification = "AGENTS.md: git changes only when the user asked for them (commit message in English, conventional style).",
|
||||
match = ["git commit -m 'feat: x'", "git reset --hard", "git checkout main"],
|
||||
not_match = ["git status"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["git", ["push", "clean"]],
|
||||
decision = "forbidden",
|
||||
justification = "Never push or clean the working tree: the user does it; git clean would delete evidence and local files.",
|
||||
match = ["git push", "git push origin main", "git clean -fdx"],
|
||||
)
|
||||
|
||||
# --- данные стендов: удаление только с согласия -----------------------------
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["make", "wipe"],
|
||||
decision = "prompt",
|
||||
justification = "make wipe deletes the working stand volume (all data). Ask the user first (AGENTS.md, section 2, item 8).",
|
||||
match = ["make wipe"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", "compose", "down", "-v"],
|
||||
decision = "prompt",
|
||||
justification = "down -v deletes volumes. The bench and test stands remove their own projects themselves; for the working stand ask the user.",
|
||||
match = ["docker compose down -v", "docker compose down -v --remove-orphans"],
|
||||
not_match = ["docker compose down"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", "volume", ["rm", "prune"]],
|
||||
decision = "prompt",
|
||||
justification = "Deleting volumes deletes data: only bench leftovers (vlad-bench*) and only with the user's consent.",
|
||||
match = ["docker volume rm vlad-bench_pgdata", "docker volume prune -f"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", ["rm", "kill", "stop", "pause"]],
|
||||
decision = "prompt",
|
||||
justification = "Only own containers (vladimirceva-lab5*, vlad-bench*); never containers of other projects.",
|
||||
match = ["docker rm -f vlad-bench-gen", "docker stop some-container"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", ["system", "builder", "image", "container", "network"], "prune"],
|
||||
decision = "forbidden",
|
||||
justification = "Prune touches every project on this machine. Remove own leftovers by name instead (bench/AGENTS.md).",
|
||||
match = ["docker system prune -af", "docker image prune -a"],
|
||||
)
|
||||
|
||||
# --- процессы: не убивать по порту вслепую ------------------------------------
|
||||
|
||||
prefix_rule(
|
||||
pattern = [["kill", "pkill", "killall"]],
|
||||
decision = "prompt",
|
||||
justification = "Kill only a PID whose command is `python -m src` (AGENTS.md, section 10). A published port on macOS belongs to com.docker.backend: killing it stops Docker Desktop.",
|
||||
match = ["kill 12345", "pkill -f uvicorn"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["rm", ["-rf", "-fr", "-r"]],
|
||||
decision = "prompt",
|
||||
justification = "Recursive delete: label folders in evidence/ are removed only with the user's consent (bench/AGENTS.md).",
|
||||
match = ["rm -rf evidence/scaling/quick-x"],
|
||||
)
|
||||
|
||||
# --- проект: проверки, стенд, замеры — без вопроса ----------------------------
|
||||
|
||||
prefix_rule(
|
||||
pattern = [
|
||||
"make",
|
||||
[
|
||||
"install", "lint", "fmt", "check", "validate", "test", "load",
|
||||
"test-persistence", "test-failover", "student-test", "test-isolated",
|
||||
"failure-checks", "requirements", "requirements-check", "env",
|
||||
"docker-build", "up", "stop", "down", "logs", "db-up", "run",
|
||||
"migrate-local", "replicas", "instances", "lb-stats",
|
||||
"failover-each", "rolling-update",
|
||||
"attachment-failover", "s3-check", "cleanup-once",
|
||||
"stop-worker", "start-worker", "defense", "pipeline-state",
|
||||
"logs-pipeline", "run-relay", "run-worker",
|
||||
"bench-queue", "bench-queue-quick",
|
||||
"bench-uploads", "bench-uploads-quick", "bench-reconcile",
|
||||
"bench-reconcile-quick",
|
||||
"bench-failover", "bench-failover-quick", "bench-rollout",
|
||||
"bench-rollout-quick", "bench-scaling", "bench-scaling-quick",
|
||||
"bench-compare", "bench-plots",
|
||||
],
|
||||
],
|
||||
decision = "allow",
|
||||
justification = "Project targets: checks, the working stand without deleting data, bench profiles (bench/RUNBOOK.md).",
|
||||
match = ["make lint", "make test-isolated", "make bench-queue", "make defense", "make failover-each STOP_MODE=kill"],
|
||||
not_match = ["make wipe", "make reconcile APPLY=1", "make replay APPLY=1"],
|
||||
)
|
||||
|
||||
# реестр failed: показ ничего не меняет
|
||||
prefix_rule(
|
||||
pattern = ["make", "replay"],
|
||||
decision = "allow",
|
||||
justification = "Without APPLY=1 replay only lists failed jobs.",
|
||||
match = ["make replay", "make replay ERROR_CODE=retry_exhausted"],
|
||||
)
|
||||
|
||||
# возврат failed в очередь меняет рабочую базу: спросить пользователя
|
||||
prefix_rule(
|
||||
pattern = ["make", "replay", "APPLY=1"],
|
||||
decision = "prompt",
|
||||
justification = "replay --apply returns failed jobs to the queue of the working stand and writes an audit row: ask the user first.",
|
||||
match = ["make replay APPLY=1 ATTACHMENT=x OPERATOR=me REASON=s3"],
|
||||
)
|
||||
|
||||
# make reconcile без APPLY — dry-run: ничего не меняет
|
||||
prefix_rule(
|
||||
pattern = ["make", "reconcile"],
|
||||
decision = "allow",
|
||||
justification = "Dry-run reconcile only reports discrepancies (exit 0 or 3).",
|
||||
match = ["make reconcile", "make reconcile STRATEGY=head DEEP=1"],
|
||||
)
|
||||
|
||||
# с APPLY=1 сверка удаляет объекты без строки и помечает потерянные
|
||||
# оригиналы в рабочей базе: спросить пользователя
|
||||
prefix_rule(
|
||||
pattern = ["make", "reconcile", "APPLY=1"],
|
||||
decision = "prompt",
|
||||
justification = "reconcile --apply deletes orphan objects and marks lost originals in the working stand: ask the user first.",
|
||||
match = ["make reconcile APPLY=1", "make reconcile APPLY=1 MAX_CHANGES=10"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", ["ps", "info", "version", "inspect", "logs", "images", "events", "exec", "cp", "build", "tag", "run", "start", "unpause"]],
|
||||
decision = "allow",
|
||||
justification = "Inspecting and running own stand containers; deletion is covered by stricter rules above.",
|
||||
match = ["docker ps -a", "docker info", "docker logs -t vlad-bench-lb"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", "compose"],
|
||||
decision = "allow",
|
||||
justification = "Compose of this project; down -v is prompted by the stricter rule above.",
|
||||
match = ["docker compose ps", "docker compose logs api"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["uv", ["run", "sync", "lock", "export"]],
|
||||
decision = "allow",
|
||||
justification = "Project environment from uv.lock; new dependencies still need the user's consent (AGENTS.md, section 2, item 11).",
|
||||
match = ["uv run pytest", "uv sync", "uv run python bench/reconcile.py --selftest"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = [["python3", "python"], ["scripts/lab5.py", "student-tests/stand.py", "tests/smoke.py", "tests/resilience.py", "scripts/check_contract.py"]],
|
||||
decision = "allow",
|
||||
justification = "Project scripts on the host Python (stdlib only).",
|
||||
match = ["python3 scripts/lab5.py s3-check", "python3 student-tests/stand.py isolated"],
|
||||
)
|
||||
@@ -18,3 +18,17 @@ __pycache__
|
||||
node_modules
|
||||
evidence
|
||||
.DS_Store
|
||||
.codex
|
||||
.agents
|
||||
tests
|
||||
student-tests
|
||||
bench
|
||||
contracts
|
||||
infra
|
||||
scripts
|
||||
**/AGENTS.md
|
||||
**/__pycache__
|
||||
**/*.pyc
|
||||
**/.DS_Store
|
||||
*.md
|
||||
**/*.md
|
||||
|
||||
+35
-1
@@ -1,6 +1,7 @@
|
||||
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
|
||||
# These are logical interface suggestions; map actual framework names in REPORT.md.
|
||||
APP_HOST=0.0.0.0
|
||||
# Host process (make run); the container binds 0.0.0.0 in Dockerfile.
|
||||
APP_HOST=127.0.0.1
|
||||
APP_PORT=8080
|
||||
APP_ENV=lab
|
||||
SESSION_TTL_SECONDS=3600
|
||||
@@ -11,6 +12,8 @@ SESSION_STORE=postgres
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
# Lab 4 logical names (skeleton). Containers get S3_ENDPOINT from compose.yaml,
|
||||
# make run overrides it with 127.0.0.1:S3_HOST_PORT; keys come from the lines below.
|
||||
S3_ENDPOINT=http://s3:9000
|
||||
S3_REGION=us-east-1
|
||||
S3_BUCKET=messenger-attachments
|
||||
@@ -18,8 +21,39 @@ S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
# Lab 5 logical names (skeleton). BROKER_URL_FILE is not used: relay and worker
|
||||
# get BROKER_HOST/BROKER_PASSWORD from compose.yaml (REPORT.md, «Конфигурация»).
|
||||
BROKER_URL_FILE=/run/secrets/broker_url
|
||||
WORKER_CONCURRENCY=2
|
||||
JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
|
||||
# Compose and make run. make env fills the empty passwords and keys locally.
|
||||
HOST_PORT=8080
|
||||
POSTGRES_USER=messenger
|
||||
POSTGRES_DB=messenger
|
||||
POSTGRES_PASSWORD=
|
||||
POSTGRES_HOST_PORT=5433
|
||||
VALKEY_PASSWORD=
|
||||
VALKEY_HOST_PORT=6380
|
||||
LOG_LEVEL=INFO
|
||||
SHUTDOWN_TIMEOUT=5
|
||||
# S3 (SeaweedFS): the root pair and the signing key stay with the s3 server;
|
||||
# api, worker and cleaner have their own users created by s3-bootstrap.
|
||||
S3_HOST_PORT=8334
|
||||
S3_ROOT_ACCESS_KEY=
|
||||
S3_ROOT_SECRET_KEY=
|
||||
S3_JWT_SIGNING_KEY=
|
||||
S3_API_ACCESS_KEY=
|
||||
S3_API_SECRET_KEY=
|
||||
S3_WORKER_ACCESS_KEY=
|
||||
S3_WORKER_SECRET_KEY=
|
||||
S3_CLEANER_ACCESS_KEY=
|
||||
S3_CLEANER_SECRET_KEY=
|
||||
# RabbitMQ (lab 5): only relay and worker get the password.
|
||||
BROKER_PASSWORD=
|
||||
BROKER_HOST_PORT=5673
|
||||
WORKER_PREFETCH=2
|
||||
RENDER_MEMORY_MB=768
|
||||
UPLOAD_MAX_UNFINISHED=300
|
||||
|
||||
@@ -25,8 +25,17 @@ bin/
|
||||
obj/
|
||||
coverage/
|
||||
*.log
|
||||
!evidence/**/raw/*.log
|
||||
.DS_Store
|
||||
.idea/
|
||||
.vscode/
|
||||
# Temporary benchmark state and self-check results are not submission evidence.
|
||||
bench/.bench.lock
|
||||
evidence/failover/quick-*/
|
||||
evidence/rollout/quick-*/
|
||||
evidence/scaling/quick-*/
|
||||
evidence/uploads/quick-*/
|
||||
evidence/reconcile/quick-*/
|
||||
evidence/queue/quick-*/
|
||||
# Keep small, sanitized evidence summaries in evidence/ under version control.
|
||||
# Inspect git diff --cached: ignore rules are not a secret scanner.
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
3.13
|
||||
@@ -0,0 +1,309 @@
|
||||
# AGENTS.md — мессенджер MTUSI, лабораторная 5
|
||||
|
||||
REST API мессенджера с вложениями-изображениями и фоновой обработкой:
|
||||
реплики API за HAProxy; PostgreSQL — источник истины (данные, сессии,
|
||||
ключи идемпотентности, метаданные вложений, **задания обработки**);
|
||||
Valkey — кэш карточек чатов; S3 (SeaweedFS) — staging и опубликованные
|
||||
файлы; RabbitMQ — сообщения о заданиях; relay — outbox → брокер; worker —
|
||||
обработка изображений в отдельном процессе; cleaner — уборка и сверка.
|
||||
Загрузка отвечает **202 `queued`**, готовность клиент узнаёт из метаданных
|
||||
(`ready` с вариантами `thumbnail`, `crop`, `optimized` или `failed` с
|
||||
`error_code`). Код перенесён из лабы 4 (`mtusi2026lab4-vladimirceva`). Цель
|
||||
— **«5»**: уровни 3 и 4 `README.md` и трек 1 «Очередь под перегрузкой».
|
||||
|
||||
**Состояние (6 ноября 2026).** Код, тесты, стенд и все проверки без
|
||||
нагрузки готовы и сняты на машине разработки (`evidence/`, таблица
|
||||
доказательств `REPORT.md`): `make test`, `make test-isolated`, `make
|
||||
failure-checks`, `make s3-check`, `make defense`. Трек 1 завершён:
|
||||
`evidence/queue/queue10/`, 24 ячейки и 120 фаз, 14185 принятых загрузок,
|
||||
5xx / failed / missing — 0; Q1–Q5 подтверждены, итоги в `REPORT.md`.
|
||||
Гипотезы и пороги записаны до замеров — не переписывать.
|
||||
|
||||
**Подготовка на этой машине (5 ноября 2026).** Хост и ВМ Docker имеют
|
||||
10 CPU и 7,65 ГиБ памяти ВМ. Бюджет `queue` — 9,5 CPU / 6,0625 ГиБ:
|
||||
PostgreSQL 1 CPU, HAProxy 0,5 CPU, S3 1 CPU / 768 МиБ; worker 2 CPU / 1536 МиБ во всех
|
||||
ячейках. Полный замер — `evidence/queue/queue10/`.
|
||||
При сверке с лабой 4 найден неперенесённый `a2047b5` (пустой JPEG-скан),
|
||||
описан в `KNOWN_ISSUES.md`. Проверки подготовки —
|
||||
`evidence/tests/measurement-host.txt`: `make test-isolated` — 140 passed,
|
||||
все `results` 0; `make failure-checks` — 34 passed. На этом хосте для
|
||||
проверок использован временный Compose 5.6.1 вместо системного 2.34.
|
||||
|
||||
Стек: Python 3.13, FastAPI, uvicorn, pydantic v2, SQLAlchemy Core 2.1 +
|
||||
asyncpg, alembic, argon2-cffi, redis-py, aiobotocore 3.x, aio-pika 10,
|
||||
Pillow 12 (только в процессе рендера); PostgreSQL 18.6, Valkey 9.1.2,
|
||||
HAProxy 3.4.6, SeaweedFS 4.48, RabbitMQ 4.3.7; uv, pip + `requirements.txt`
|
||||
с хешами (в образе), pytest + httpx.
|
||||
|
||||
## 1. Где инструкции и как их читать
|
||||
|
||||
Codex сам загружает `AGENTS.md` от корня до рабочего каталога (вместе — не
|
||||
больше 32 КиБ). Правила слоёв лежат глубже, поэтому **перед правкой файла
|
||||
прочитай каждый `AGENTS.md` на пути к нему** (`cat`).
|
||||
|
||||
| Файл | Когда читать |
|
||||
|---|---|
|
||||
| `AGENTS.md` (этот) | всегда: запреты, команды, карта, Definition of Done |
|
||||
| `src/AGENTS.md` | любая правка в `src/`: слои, импорты, процессы, ошибки, конфиг (таблица переменных), логи |
|
||||
| `src/api/AGENTS.md` | роуты, схемы, middleware, вход, `X-Instance-Id`, `Idempotency-Key`, вложения |
|
||||
| `src/services/AGENTS.md` | бизнес-правила, загрузка, worker, relay, replay, уборка и сверка |
|
||||
| `src/storage/AGENTS.md` | PostgreSQL: транзакции, гонки, вложения, задания и fencing; кэш; объекты S3 |
|
||||
| `migrations/AGENTS.md` | схема, новые ревизии, совместимость с работающей версией |
|
||||
| `infra/AGENTS.md` | `Dockerfile`, `compose.yaml`, S3, брокер, relay, worker, HAProxy, `scripts/*` |
|
||||
| `student-tests/AGENTS.md` | свои тесты, отдельные стенды, проверки отказов |
|
||||
| `evidence/AGENTS.md` | доказательства для `REPORT.md` |
|
||||
| `bench/AGENTS.md`, `bench/RUNBOOK.md` | нагрузочный стенд; замеры трека 1 по шагам |
|
||||
|
||||
Ещё Codex читает **правила команд** `.codex/rules/messenger.rules` (только у
|
||||
доверенного проекта; после правки — перезапуск Codex; проверка — `codex
|
||||
execpolicy check --pretty --rules "$PWD/.codex/rules/messenger.rules" make
|
||||
wipe`) и **навыки** `.agents/skills/`: `lab5-bench` (замеры по RUNBOOK),
|
||||
`messenger-stand` (стенд, проверки, демонстрация). Они дублируют раздел 2,
|
||||
но не заменяют его.
|
||||
|
||||
Конфликт: глубже — уточняет, но не отменяет раздел 2. Прямая просьба
|
||||
пользователя важнее этих файлов, кроме пунктов «спросить».
|
||||
|
||||
## 2. Нельзя
|
||||
|
||||
1. **Менять skeleton преподавателя** (спросить): `contracts/`, `tests/`,
|
||||
`scripts/` (кроме своих `restart.sh`, `stop-one.sh`,
|
||||
`rolling-update.sh`, `lab5.py`), `lab.json`, `README.md`, `COURSE.md`,
|
||||
`CONTRIBUTING.md`, `MAINTAINERS.md`, `Dockerfile.example`,
|
||||
`compose.example.yaml`, `infra/README.md`, `requirements-dev.txt`,
|
||||
`evidence/README.md`, `examples/`, `.github/`, верх `Makefile` до строки
|
||||
`=====`. Публичный тест упал — чиним приложение.
|
||||
2. **Подгонять тесты под код**: ослаблять `assert`, менять ожидаемый
|
||||
статус, `skip`/`xfail`, удалять тест.
|
||||
3. **Состояние в памяти процесса, от которого зависит корректность**:
|
||||
данные, сессии, ключи, метаданные и **задания** — только PostgreSQL;
|
||||
байты — только S3; сообщение брокера — только подсказка «посмотри
|
||||
задание». `--workers`, gunicorn, `reload`; sticky session. Уборка и
|
||||
сверка — только `cleaner`, публикация — только `relay`, обработка —
|
||||
только `worker`.
|
||||
4. **Порты реплик наружу и `container_name` у `api`**; консоль, master,
|
||||
volume, filer SeaweedFS, мониторинг LB, management RabbitMQ — наружу.
|
||||
5. **Файлы конфигурации приложения**: настройки — только переменные
|
||||
окружения (`src/config.py`).
|
||||
6. **JSON-логи, structlog, `print`** вместо логгера (`src/log.py`).
|
||||
7. **Писать в лог** текст сообщений, `display_name`, `title`, username,
|
||||
`filename` (только длину), пароль, токен, ключи, их хэши,
|
||||
`Authorization`, `Idempotency-Key`, тела, параметры SQL, оператора и
|
||||
причину replay (только длины). Можно id, ключ объекта, размеры.
|
||||
8. **Git без просьбы**: `commit`, `push`, `checkout`, `reset`, `stash`,
|
||||
`rebase`, `clean`. Смотреть можно.
|
||||
9. **Удалять данные стенда без просьбы** (спросить): `make wipe`, `docker
|
||||
compose down -v`, `docker volume rm`, очистку очереди брокера,
|
||||
`FLUSHALL`, `TRUNCATE`/`DROP`, удаление объектов мимо API, каталоги
|
||||
меток в `evidence/`; `make replay APPLY=1`, `make reconcile APPLY=1` на
|
||||
рабочем стенде.
|
||||
10. **Секреты и `.env`** в git, образ, `evidence/`, вывод и отчёт; `COPY .
|
||||
.`. Административную пару S3 — никому, кроме `s3`; пароль брокера —
|
||||
только `rabbitmq`, `relay`, `worker`.
|
||||
11. **Незакреплённые версии**: `latest`, тег без digest, пакет без lock.
|
||||
12. **Новая зависимость** (спросить).
|
||||
13. **Миграция, которая ломает работающую версию**: только expand;
|
||||
применённую ревизию не править.
|
||||
14. **`HTTPException`** в сервисах и хранилище, **403**, **422** наружу;
|
||||
прозрачные повторы `POST`.
|
||||
15. **Обходить sandbox Codex** (раздел 10).
|
||||
16. **Выдумывать результаты**: в отчёт и `evidence/` — только реальный
|
||||
вывод команд, которые ты запустил. Гипотезы Q1–Q5 не переписывать.
|
||||
17. **Чинить попутно найденное** без просьбы: записать в `KNOWN_ISSUES.md`.
|
||||
18. **Менять код приложения ради замеров** (спросить).
|
||||
|
||||
## 3. Порядок работы
|
||||
|
||||
1. Откуда требование: `README.md` (рубрика), `contracts/README.md`
|
||||
(«Фоновая обработка (с lab5)»), `contracts/attachment-states.md`,
|
||||
`contracts/jobs.schema.json`, `contracts/openapi.json`, `COURSE.md`.
|
||||
Неясно — спроси. Ответ преподавателя — в раздел 12.
|
||||
2. Прочитай нужные `AGENTS.md` и затрагиваемые файлы целиком.
|
||||
3. Задача больше пары файлов — короткий план в чате.
|
||||
4. Минимальный дифф. 5. Проверки раздела 8. 6. Отчёт — раздел 11.
|
||||
|
||||
## 4. Команды
|
||||
|
||||
| Команда | Что делает |
|
||||
|---|---|
|
||||
| `make install` / `make env [HOST_PORT=…]` | `uv sync` / `.env` со случайными паролями и ключами (существующий дополняется) |
|
||||
| `make up` | postgres, valkey, s3, rabbitmq → migrate, s3-bootstrap → api × `API_REPLICAS` (2), relay, worker, cleaner → haproxy; ждёт healthy |
|
||||
| `make stop` / `make down` / `make logs` / `make logs-pipeline` | остановить / удалить контейнеры (volume остаются) / журнал реплик / relay и worker |
|
||||
| `make wipe` | удалить контейнеры и volume `pgdata`, `s3data`, `rmqdata`: только после `yes` |
|
||||
| `make test [BASE_URL=...]` | публичный smoke (202, ready, оригинал, thumbnail 256×160) |
|
||||
| `make defense` | шаги 2–3 защиты: worker стоп → 202 → ссылка → 409 → `restart.sh` → worker → ready |
|
||||
| `make stop-worker` / `make start-worker` / `make pipeline-state` | worker; задания по состояниям и очередь брокера |
|
||||
| `make replay [ERROR_CODE=…] [ATTACHMENT=…] [APPLY=1 OPERATOR=… REASON=…]` | реестр failed; с `APPLY=1` — возврат с аудитом (коды 0/3/1/2) |
|
||||
| `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | данные, токен, оригинал после рестарта LB, реплик, хранилищ и брокера |
|
||||
| `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` / `make failover-each` | отказ одной реплики |
|
||||
| `make student-test [BASE_URL=…] [EXPECT_REPLICAS=…]` | свои тесты против стенда (без отказов) |
|
||||
| `make test-isolated` / `make failure-checks [FAULT_ARGS=-rP]` | отдельный стенд: всё выше / проверки отказов (в том числе pipeline) |
|
||||
| `make attachment-failover` / `make s3-check` | вложение после остановки принявшей реплики / приватность bucket |
|
||||
| `make replicas` / `make instances` / `make lb-stats` / `make rolling-update` | реплики за LB |
|
||||
| `make cleanup-once` / `make reconcile [APPLY=1] …` | уборщик (с лабы 5 — и staging) / сверка (dry-run по умолчанию) |
|
||||
| `make db-up` / `make run` / `make run-relay` / `make run-worker` / `make migrate-local` | процессы на хосте рядом с хранилищами стенда |
|
||||
| `make lint` / `make fmt` / `make check` / `make validate PYTHON=.venv/bin/python` | ruff; офлайн-проверки skeleton |
|
||||
| `make requirements` / `make requirements-check` | `requirements.txt` из `uv.lock` |
|
||||
| `make bench-queue [QUEUE_LABEL=…]` / `make bench-queue-quick` | трек 1 под нагрузкой (`bench/RUNBOOK.md`); `quick-*` в отчёт не идут |
|
||||
| `make bench-uploads`, `bench-reconcile`, `bench-failover`, `bench-rollout`, `bench-scaling` | профили лаб 3–4 (сохранены, в лабе 5 не обязательны) |
|
||||
|
||||
Проверки ходят на `http://localhost:8080` (HAProxy). Внешние порты в
|
||||
`.env`: `HOST_PORT`, `POSTGRES_HOST_PORT` (5433), `VALKEY_HOST_PORT` (6380),
|
||||
`S3_HOST_PORT` (8334), `BROKER_HOST_PORT` (5673); разовая подмена — `make
|
||||
test BASE_URL=http://localhost:8090`. Тесты против рабочего стенда
|
||||
оставляют данные; чистый прогон — `make test-isolated`.
|
||||
|
||||
## 5. Карта кода
|
||||
|
||||
```
|
||||
src/ python -m src (__main__ → early_stop.install() → main.run()): API
|
||||
main.py create_app(): Db, кэш, хранилище, S3, места загрузок, пароли, роутеры
|
||||
config.py Settings.from_env(): Database, Cache, ObjectStorage, Uploads, Cleaner, Broker, Pipeline, Relay
|
||||
constants.py, log.py, errors.py, clock.py, domain.py, pagination.py, healthcheck.py, early_stop.py
|
||||
images.py структурная проверка PNG/JPEG на входе API (stdlib)
|
||||
db.py, cache.py, s3.py, uploads.py, passwords.py инфраструктура процесса
|
||||
broker.py RabbitMQ (aio-pika): JobPublisher (confirms) и JobConsumer (qos, ack), формат сообщения
|
||||
renderer.py ImageRenderer: процесс обработки на задание, лимиты, коды выхода → исходы
|
||||
worker/ python -m src.worker: потребитель + WORKER_CONCURRENCY задач; render.py — дочерний процесс Pillow
|
||||
relay/ python -m src.relay: outbox → брокер, аренды, потерянные; `replay` — реестр failed
|
||||
cleaner/ python -m src.cleaner: брошенные загрузки, staging, сверка
|
||||
api/ routes/, schemas, deps, identity, request_id, handlers
|
||||
services/ User/Session/Chat/Message/Attachment, Processing, Relay, Replay, Cleanup, Reconcile
|
||||
storage/ base.py (Storage, MaintenanceStorage, PipelineStorage), postgres.py, pipeline.py, cached.py, tables.py, objects.py
|
||||
migrations/ 0001 лаба 2, 0002 ключи, 0003 вложения, 0004 задания, варианты, журнал replay (expand)
|
||||
student-tests/ тесты по HTTP; test_processing.py, test_pipeline.py (лаба 5); stand.py, fault_hook.py
|
||||
bench/ нагрузочный стенд; профиль queue — трек 1 (bench/AGENTS.md)
|
||||
infra/ haproxy/, valkey/, s3/bootstrap.sh, rabbitmq/rabbitmq.conf
|
||||
scripts/ restart.sh, stop-one.sh, rolling-update.sh, lab5.py (свои); остальное — skeleton
|
||||
.codex/rules/, .agents/skills/, evidence/, Dockerfile, compose.yaml, pyproject.toml, uv.lock, requirements.txt
|
||||
```
|
||||
|
||||
Поток: HAProxy → реплика (401, 400, 404, 415, 413, 429 предела, 429 места,
|
||||
тело, структура) → строка `pending` → PUT в staging → `queued` + задание
|
||||
(одна транзакция) → **202**. Дальше без клиента: relay публикует задание с
|
||||
подтверждением → worker захватывает с арендой → GET staging, рендер в
|
||||
отдельном процессе, PUT оригинала и вариантов → фиксация `ready`
|
||||
(fencing по номеру попытки) → ack. Подробно — `src/services/AGENTS.md`.
|
||||
|
||||
## 6. Контракт: самое важное
|
||||
|
||||
- Каждый ответ — `X-Request-Id` и `X-Instance-Id`; ошибка — `{"error":
|
||||
{"code", "message", "request_id"}}`; 422 и 403 — никогда.
|
||||
- Загрузка — сырые байты PNG/JPEG ≤ 10 MiB, ≤ 40 млн пикселей; **202** со
|
||||
снимком `queued` только после staging и задания в PostgreSQL. Задание
|
||||
ждёт и при остановленных брокере и worker.
|
||||
- Метаданные: `queued`/`processing`/`ready`/`failed`, `variants` (thumbnail
|
||||
JPEG ≤ 256×256, crop JPEG 128×128, optimized WebP ≤ 1280×1280),
|
||||
`error_code` у `failed` (`invalid_image`, `image_too_large`,
|
||||
`source_missing`, `processing_failed`, `retry_exhausted`).
|
||||
- Байты (`/content?variant=`): не готово или `failed` — 409; варианта у
|
||||
готового нет — 404.
|
||||
- Сообщение: `attachment_ids` до 10 видимых вложений этого чата (`queued`,
|
||||
`processing`, `ready`); `failed` — 409. Пустой текст — только с вложением.
|
||||
|
||||
## 7. Конфигурация, логи, Docker, зависимости
|
||||
|
||||
**Переменные** читает только `src/config.py`; неверное значение — код 2.
|
||||
Группы: `APP_*`, `LOG_LEVEL`, `DB_*`, `CACHE_*`, `S3_*`, `UPLOAD_*`
|
||||
(`UPLOAD_MAX_UNFINISHED` — предел незавершённых заданий), `CLEANER_*`,
|
||||
`STAGING_RETENTION_SECONDS`, `RECONCILE_*`, `BROKER_*`, `WORKER_*`,
|
||||
`JOB_*`, `RENDER_MEMORY_MB`, `RELAY_*`. Таблица — `src/AGENTS.md`.
|
||||
`WORKER_FAULT`, `RELAY_FAULT` — точки сбоя только для проверок отказов.
|
||||
|
||||
**Логи**: строка на событие `время уровень логгер [rid=...] сообщение
|
||||
key=value`. Строки `attachment processed …`, `job delivery skipped …`,
|
||||
`reconcile pass finished …` разбирают `bench/` и тесты — имена полей не
|
||||
менять.
|
||||
|
||||
**Docker, compose, S3, брокер, HAProxy** — `infra/AGENTS.md`. Числа менять
|
||||
только вместе: аренда задания `JOB_TIMEOUT_SECONDS + JOB_LEASE_MARGIN_SECONDS`
|
||||
(60 + 20 с) > времени рендера и записи; `timeout server` пути загрузки 40 с
|
||||
> 5 + 30 с; `CLEANER_GRACE` 900 с > 21 с; дренаж замены 90 с > 40 с.
|
||||
|
||||
**Зависимости**: `pyproject.toml` → `uv lock` → `make requirements` → `make
|
||||
requirements-check` → пересборка образа.
|
||||
|
||||
## 8. Definition of Done
|
||||
|
||||
- [ ] `make fmt`, затем `make lint` — без ошибок;
|
||||
- [ ] `make test-isolated` — в `results` все 0;
|
||||
- [ ] правка хранилища, вложений, pipeline, уборщика, сверки, сессий,
|
||||
ключей, кэша, клиентов, таймаутов — ещё `make failure-checks`;
|
||||
- [ ] правка `compose.yaml`, `haproxy.cfg`, `infra/`, `scripts/*` — `make
|
||||
up`, `make test`, `make defense`, `make s3-check`, `make
|
||||
attachment-failover`, `make test-persistence
|
||||
ACTION_SCRIPT=scripts/restart.sh`, `make failover-each`;
|
||||
- [ ] правка схемы — новая expand-ревизия, `alembic check` без расхождений;
|
||||
- [ ] правка `Dockerfile`, зависимостей — `make up`, всё healthy, `make
|
||||
test`, `make requirements-check`;
|
||||
- [ ] правка `bench/` — самопроверки разбора и `make bench-queue-quick`;
|
||||
- [ ] новое поведение из рубрики — тест в `student-tests/`;
|
||||
- [ ] изменилось описанное в `REPORT.md`, `evidence/`, `AGENTS.md` —
|
||||
обновлено там же;
|
||||
- [ ] остановлены серверы и контейнеры, которые ты запустил.
|
||||
|
||||
Не смог выполнить пункт — так и напиши, с причиной.
|
||||
|
||||
## 9. Если тест или проверка упали
|
||||
|
||||
1. Воспроизведи и прочитай вывод целиком.
|
||||
2. Причина: **дефект приложения** → код; **ошибка теста** (противоречит
|
||||
контракту или зависит от случайного) → тест, с объяснением; **стенд**
|
||||
(чужой сервис на порту, старый образ, не тот `BASE_URL`, остатки
|
||||
прерванного прогона, брокер или S3 ещё не healthy) → стенд.
|
||||
3. Неясно — остановись и спроси.
|
||||
|
||||
Тест `tests/` не правится никогда. Проверки отказов меряют время с запасом;
|
||||
на загруженном хосте возможно ложное падение — один перезапуск и сказать.
|
||||
|
||||
## 10. Особенности Codex
|
||||
|
||||
- В sandbox нет сети и часто нет docker: запроси запуск вне sandbox
|
||||
(правила `.codex/rules/`) или попроси пользователя; команду — дословно.
|
||||
- Долгие команды (`make bench-queue` — ~1,5 ч, `make failure-checks` —
|
||||
~8 мин) — в фоне: `nohup make … > /tmp/x.log 2>&1 &`, затем `tail`.
|
||||
- Не убивать процесс по порту вслепую: порт контейнера на macOS держит
|
||||
`com.docker.backend`. Чужие контейнеры не трогать — другой порт (`make
|
||||
env HOST_PORT=… … BROKER_HOST_PORT=…` до первого `make up`).
|
||||
- Держи этот файл ≤ ~28 КБ (`wc -c`). Новое правило слоя — во вложенный
|
||||
`AGENTS.md`.
|
||||
|
||||
## 11. Отчёт пользователю
|
||||
|
||||
По-русски, коротко: что сделано; какие проверки и с каким итогом
|
||||
(`N passed`, `PASS: lab5 ...`, `results: {...}`, метки `evidence/…`); что
|
||||
не проверено и почему; что записано в `KNOWN_ISSUES.md`.
|
||||
|
||||
## 12. Согласования с преподавателем
|
||||
|
||||
Договорённости, которых нет в skeleton. Каждую упомянуть в `REPORT.md`.
|
||||
|
||||
| Дата | Договорённость | Где учтено |
|
||||
|---|---|---|
|
||||
| 2026-10-20 | Лаба 4: «пустой текст» сообщения — и `""`, и строка из одних пробелов. `" "` с ≥ 1 вложением — 201, без вложений — 400 | `src/api/schemas.py` (`CreateMessage`), `student-tests/test_attachments.py`, `REPORT.md` |
|
||||
|
||||
## 13. Стиль кода
|
||||
|
||||
- ruff: строка 88, py313; формат — `make fmt`. Аннотации везде. Скрипты на
|
||||
`python3` хоста (`student-tests/stand.py`, `fault_hook.py`,
|
||||
`scripts/lab5.py`, `bench/`) — stdlib, Python 3.9+.
|
||||
- Методы сервисов и изменяющие методы хранилища — keyword-only.
|
||||
- Комментарии и докстринги — короткие, по-русски, «почему». Логи и
|
||||
`error.message` — по-английски.
|
||||
- Коммиты (только по просьбе): английский, conventional.
|
||||
|
||||
## 14. Документы решения
|
||||
|
||||
`REPORT.md` (отчёт, гипотезы Q1–Q5, ответы защиты), `evidence/`,
|
||||
`KNOWN_ISSUES.md`, `bench/RUNBOOK.md`. Доказательства лаб 3–4 — в их
|
||||
репозиториях.
|
||||
|
||||
## 15. Дальше: лаба 6
|
||||
|
||||
Новый skeleton — отдельный репозиторий; сначала его `COURSE.md`
|
||||
(телеметрия). Переносить своё (код, тесты, миграции, `bench/`, `infra/`
|
||||
кроме `README.md`, свои `scripts/*`, Dockerfile, compose, lock-файлы,
|
||||
`AGENTS.md`, `.codex/`, `.agents/`, `KNOWN_ISSUES.md`); `Makefile`,
|
||||
`.env.example`, `.gitignore`, `.dockerignore` — сливать; skeleton, `.env`,
|
||||
volumes, `REPORT.md`, `evidence/` — не копировать. `name:` compose —
|
||||
`vladimirceva-lab6`.
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
# Один образ на все процессы лабы 5: API (`python -m src`), миграции,
|
||||
# relay (`-m src.relay`), worker (`-m src.worker`, Pillow — только в его
|
||||
# дочернем `-m src.worker.render`) и уборщик (`-m src.cleaner`). Процесс
|
||||
# выбирает entrypoint сервиса compose. Правила — infra/AGENTS.md.
|
||||
#
|
||||
# Базовый образ один на оба stage и закреплён точной патч-версией плюс
|
||||
# digest мультиарх-индекса (amd64 и arm64): venv из deps работает только с
|
||||
# тем же интерпретатором. Чтобы обновить, взять новый тег и значение поля
|
||||
# Digest, которое печатает buildx imagetools inspect для этого тега.
|
||||
|
||||
# deps: зависимости строго по requirements.txt (экспорт uv.lock с хешами).
|
||||
# У asyncpg, greenlet и argon2-cffi-bindings есть готовые колёса для
|
||||
# linux amd64/arm64: компилятор в образе не нужен
|
||||
FROM python:3.13.15-slim-trixie@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b AS deps
|
||||
|
||||
ENV PIP_NO_CACHE_DIR=1 \
|
||||
PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||
|
||||
RUN python -m venv /opt/venv
|
||||
COPY requirements.txt /tmp/requirements.txt
|
||||
# --require-hashes: пакет с другим хешем или без пина — сборка падает;
|
||||
# --no-deps: ставится ровно список из файла, без догрузки зависимостей;
|
||||
# --only-binary: сборка из исходников (sdist) запрещена
|
||||
RUN /opt/venv/bin/pip install --require-hashes --no-deps --only-binary=:all: \
|
||||
-r /tmp/requirements.txt
|
||||
|
||||
|
||||
# runtime: интерпретатор базового образа, venv и код; без pip-кэша и тестов
|
||||
FROM python:3.13.15-slim-trixie@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b AS runtime
|
||||
|
||||
RUN groupadd --system --gid 10001 messenger \
|
||||
&& useradd --system --uid 10001 --gid messenger --no-create-home \
|
||||
--home-dir /nonexistent --shell /usr/sbin/nologin messenger
|
||||
|
||||
# Фиксированный порог mmap в 128 KiB: всё крупнее glibc выделяет через
|
||||
# mmap, и BytesIO с телом загрузки (до 10 MiB) расширяется mremap без
|
||||
# копирования. По умолчанию порог плавающий — после первой загрузки glibc
|
||||
# задирает его почти до 10 MiB, буфер переезжает в кучу, и при realloc в
|
||||
# памяти на мгновение оказываются две копии тела
|
||||
ENV PATH=/opt/venv/bin:$PATH \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
MALLOC_MMAP_THRESHOLD_=131072 \
|
||||
APP_HOST=0.0.0.0 \
|
||||
APP_PORT=8080
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# владелец файлов — root, процесс от messenger может их только читать.
|
||||
# Миграции едут в том же образе: сервис migrate запускает его с командой
|
||||
# `python -m alembic upgrade head`
|
||||
COPY --from=deps /opt/venv /opt/venv
|
||||
COPY src ./src
|
||||
COPY alembic.ini ./
|
||||
COPY migrations ./migrations
|
||||
|
||||
USER messenger:messenger
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=10s --timeout=3s --start-period=10s --start-interval=1s \
|
||||
--retries=3 CMD ["python", "-m", "src.healthcheck"]
|
||||
|
||||
# без shell-обёртки: python становится PID 1, и SIGTERM приходит прямо ему
|
||||
CMD ["python", "-m", "src"]
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
# Известные проблемы
|
||||
|
||||
Найдены попутно и не исправлены без отдельной просьбы (AGENTS.md,
|
||||
раздел 2, п. 17). Формат: что, где, симптом, предложение. Исправленное
|
||||
удаляется отсюда.
|
||||
|
||||
## Лаба 5
|
||||
|
||||
### SeaweedFS с лимитом 512 МиБ может погибнуть на крупных загрузках
|
||||
|
||||
Метка `evidence/queue/baseline/` прервана на пятой ячейке. В
|
||||
`c4p4-l-r1` API получил 308 ответов 5xx в перегрузке и 49 при
|
||||
восстановлении; 14 принятых заданий завершились `retry_exhausted`
|
||||
из-за недоступного S3. Журнал ядра Docker от 2026-11-05T20:43:04Z:
|
||||
`weed invoked oom-killer`, память cgroup 524292 kB при лимите
|
||||
524288 kB, процесс `weed` убит. Это ограничение инфраструктуры S3;
|
||||
при выборе ресурсов рабочего стенда нужно учитывать нагрузку крупных
|
||||
объектов. Доказательство — `evidence/stand/s3-load.txt` и прерванная
|
||||
метка. В профиле очереди S3 выделены 768 МиБ, `GOMEMLIMIT=400MiB`.
|
||||
|
||||
### Из лабы 4 не перенесён запрет JPEG с пустым сканом
|
||||
|
||||
Сверка с соседним репозиторием лабы 4 после её замеров обнаружила
|
||||
коммит `a2047b5` (`fix: reject JPEG scans without image data`), отсутствующий
|
||||
в лабе 5: в `src/images.py` нет `_JPEG_EMPTY_SCAN`, в
|
||||
`student-tests/test_attachments.py` — четырёх случаев
|
||||
`test_jpeg_without_scan_data_is_415`, в `student-tests/images.py` —
|
||||
генератора `jpeg_with_scan_data`. Проверка текущего `inspect_image`
|
||||
на JPEG только из заголовков возвращает `ImageInfo` 1×1 вместо
|
||||
`DamagedImage`; данные скана из одних маркеров перезапуска также не
|
||||
отсекаются этой проверкой. Предложение: отдельно перенести исправление
|
||||
и четыре теста, затем выполнить изолированные проверки. Самостоятельно
|
||||
не переносилось: RUNBOOK требует только перечислить пропущенные правки.
|
||||
|
||||
### Зависание Docker credential helper при загрузке публичного образа
|
||||
|
||||
На текущем хосте `make test-isolated` остановился на `rabbitmq Pulling`:
|
||||
дочерний `docker-credential-desktop` несколько минут не завершался,
|
||||
контейнеры ещё не создавались. Docker Hub с хоста доступен (HTTP 401
|
||||
на `/v2/`), а анонимный pull того же закреплённого digest с временным
|
||||
`DOCKER_CONFIG` прошёл. Первый прогон завершён с ошибкой окружения
|
||||
после остановки его helper, тестовый скрипт выполнил cleanup. Для
|
||||
следующего прогона использована отдельная конфигурация в `/private/tmp/`
|
||||
без credential helper и существующий Docker socket; пользовательская
|
||||
конфигурация и связка ключей не изменялись. Причина зависания helper
|
||||
не выяснена; при обычном запуске может повториться.
|
||||
|
||||
### Системный Compose 2.34 не проходит подготовку изолированного стенда
|
||||
|
||||
При `student-tests/stand.py:start_lb` команда `up --wait api relay
|
||||
worker haproxy` на Compose 2.34.0-desktop.1 завершается с
|
||||
`container ...-relay-1 has no healthcheck configured`. Реплики API,
|
||||
хранилища и HAProxy healthy, worker уже потребляет очередь; pytest
|
||||
не запускается, `results: {'setup': 1}`. В `compose.yaml` у relay и
|
||||
worker healthcheck намеренно отключён. Код не менялся; для следующего
|
||||
прогона взят официальный Compose 5.6.1 (версия машины разработки),
|
||||
его SHA-256 проверен, бинарник и конфигурация помещены в `/private/tmp/`.
|
||||
Системный Compose остался прежним. Следует явно указывать совместимую
|
||||
версию Compose при повторении проверок на этой машине. На временном
|
||||
Compose 5.6.1 `make test-isolated` прошёл: 140 passed, все пять
|
||||
значений `results` — 0 (`evidence/tests/measurement-host.txt`).
|
||||
|
||||
### Смесь версий при замене реплик лабы 4 на лабу 5
|
||||
|
||||
`scripts/rolling-update.sh` меняет реплики по одной. Пока работает хоть
|
||||
одна реплика лабы 4, её загрузки отвечают 201 `ready`, задания не пишутся,
|
||||
вариантов у таких вложений нет (метаданные `ready` с `variants: []`,
|
||||
`?variant=thumbnail` — 404). Схема 0004 это допускает (expand), но
|
||||
клиент на время замены видит два поведения. Предложение: первый релиз
|
||||
лабы 5 выкатывать `make down` + `make up` (с простоем) или дописать
|
||||
разовую команду, которая ставит задания готовым вложениям без вариантов.
|
||||
|
||||
### JPEG без таблиц квантования проходит проверку API
|
||||
|
||||
`src/images.py:_inspect_jpeg` проверяет маркеры, кадр и скан, но не
|
||||
наличие DQT. Такой файл получает 202, а worker переводит его в `failed`
|
||||
с `invalid_image` (`student-tests/test_processing.py::
|
||||
test_undecodable_image_fails_with_safe_code`). Контракт требует 415 для
|
||||
заведомо повреждённых файлов на входе. Предложение: в `_inspect_jpeg`
|
||||
собирать номера таблиц DQT и требовать, чтобы каждая компонента кадра
|
||||
ссылалась на определённую до SOS.
|
||||
|
||||
### Эвристика потерянных сообщений видит только «ready» в очереди
|
||||
|
||||
`src/services/relay.py`: `republish_lost` срабатывает, когда в очереди нет
|
||||
сообщений, готовых к выдаче (`message_count` пассивного объявления), а
|
||||
задание опубликовано дольше `RELAY_REPUBLISH_AFTER`. Сообщения, уже
|
||||
выданные worker и ждущие во внутренней очереди процесса (prefetch больше
|
||||
concurrency), в этот счёт не входят. При очень большом prefetch и долгих
|
||||
заданиях relay опубликует их повторно: исход верный (старая попытка
|
||||
станет `stale`), но работа брокера и worker лишняя. Предложение: брать
|
||||
ещё `messages_unacknowledged` (management API) или не республиковать,
|
||||
пока у очереди есть потребители с неподтверждёнными сообщениями.
|
||||
|
||||
### Долгий отказ PostgreSQL и предел доставок quorum-очереди
|
||||
|
||||
`src/worker/main.py:_one`: PostgreSQL не ответил на захвате — сообщение
|
||||
возвращается в очередь через 1 с. У quorum-очереди RabbitMQ 4 есть
|
||||
предел повторных доставок (по умолчанию 20), после него сообщение
|
||||
отбрасывается. Задание при этом цело в PostgreSQL и вернётся через
|
||||
`republish_lost`, но не раньше `RELAY_REPUBLISH_AFTER`. Предложение:
|
||||
пауза перед возвратом с ростом или собственный `x-delivery-limit` под
|
||||
ожидаемую длину отказа.
|
||||
|
||||
### Шум aiormq в журнале при остановке брокера
|
||||
|
||||
При `docker compose stop rabbitmq` библиотека aiormq пишет `ERROR` и
|
||||
traceback `CONNECTION_FORCED` в журнал relay и worker, хотя дальше их
|
||||
код штатно переподключается (строки `broker … failed`, затем `broker
|
||||
consuming`). Предложение: поднять уровень логгера `aiormq.connection` до
|
||||
`CRITICAL` в `src/log.py` и оставить только свои строки.
|
||||
|
||||
### SAWarning в `alembic check`
|
||||
|
||||
`alembic check` (alembic 1.20.0, SQLAlchemy 2.1.1) печатает `SAWarning:
|
||||
Can't validate argument 'dialect_options'; can't locate any SQLAlchemy
|
||||
dialect named 'dialect'` из `migrations/env.py:54`, затем «No new upgrade
|
||||
operations detected» (`evidence/stand/migrations.txt`). Предупреждение
|
||||
появляется при сравнении отражённых индексов, на результат проверки не
|
||||
влияет; источник не выяснен. Предложение: проверить на минимальной схеме
|
||||
с одним частичным индексом и, если это библиотека, закрепить фильтр
|
||||
`warnings` только для этой строки в `env.py` или дождаться исправления.
|
||||
|
||||
## Перенесено из лабы 3 (bench/, серии трека 1 лабы 3)
|
||||
|
||||
Профили лабы 3 в лабе 5 сохранены, но не обязательны.
|
||||
|
||||
### Возобновление прерванной серии требует доработки
|
||||
|
||||
Статический анализ `bench/run.py:run_scaling` и `scaling_cell`: ветка
|
||||
`resume` сохраняет прежний `meta.aborted`, а финальная проверка этого
|
||||
поля возвращает ошибку даже после завершения оставшихся ячеек. Повтор
|
||||
незавершённой ячейки добавляет строки с теми же номерами. Предложение:
|
||||
восстанавливать только незавершённую ячейку и снимать `aborted` после
|
||||
успешного продолжения.
|
||||
|
||||
### Подпись насыщения для A/B с одной конфигурацией
|
||||
|
||||
В `bench/plots.py:saturation` единственная точка возвращает `None`, а
|
||||
`main` трактует это как продолжающийся рост ≥ 10 %. Предложение:
|
||||
отдельно печатать «недостаточно конфигураций» при числе точек меньше
|
||||
двух.
|
||||
@@ -0,0 +1,59 @@
|
||||
# ЛР 5 — БВТ2301 — Владимирцева — на 5
|
||||
|
||||
Автор / группа: Владимирцева, БВТ2301.
|
||||
Целевая оценка: **5**.
|
||||
Трек: **1 — «Очередь под перегрузкой»**.
|
||||
|
||||
Загрузка вложения отвечает `202 queued` после сохранения оригинала в
|
||||
staging S3 и задания в PostgreSQL. Отдельный worker обрабатывает
|
||||
изображение; клиент получает состояние `ready` с вариантами thumbnail,
|
||||
crop и optimized либо `failed` с безопасным кодом ошибки.
|
||||
|
||||
Перенесён мессенджер лабораторных 2–4: API за HAProxy, PostgreSQL,
|
||||
Valkey, приватное S3, сессии, идемпотентность, вложения и проверки
|
||||
сохранности данных. Реализованы transactional outbox и relay с publisher
|
||||
confirms, RabbitMQ quorum queue, ручной ack после фиксации результата,
|
||||
аренды и fencing, ограниченные повторы с backoff, реестр failed и replay
|
||||
с аудитом. Рендер выполняется в отдельном процессе с лимитами памяти,
|
||||
CPU и времени; staging убирает cleaner.
|
||||
|
||||
Запуск из чистого clone (Docker Compose 5.6.1 и uv):
|
||||
|
||||
```sh
|
||||
make install
|
||||
make env
|
||||
make up
|
||||
make test
|
||||
```
|
||||
|
||||
Проверки и доказательства:
|
||||
|
||||
- `make fmt`, `make lint`, `make check`, `make requirements-check` — PASS.
|
||||
- `make test-isolated` — **140 passed**, smoke, failover,
|
||||
attachment-failover и persistence: все пять значений `results` — 0.
|
||||
- `make failure-checks` — **34 passed**; окна отказов relay/worker,
|
||||
рестарт брокера, повторы, replay и отказы хранилищ проверены.
|
||||
- `make bench-queue QUEUE_LABEL=queue10` — **24 ячейки, 120 фаз,
|
||||
14185 загрузок**, 5xx / failed / missing — 0. Все Q1–Q5 подтверждены.
|
||||
Сравниваются четыре настройки worker на трёх классах изображений,
|
||||
по два повтора; хост — Apple M4, Docker VM 10 CPU / 7,65 ГиБ.
|
||||
Предел принятой работы соблюдён, восстановление доказано.
|
||||
- Отчёт: [REPORT.md](REPORT.md). Полный замер:
|
||||
[summary.md](evidence/queue/queue10/summary.md),
|
||||
[meta.json](evidence/queue/queue10/meta.json) и исходные CSV/журналы.
|
||||
|
||||
Ограничения: один Docker-host и по одному PostgreSQL, S3 и брокеру;
|
||||
результаты нагрузки относятся к указанному оборудованию. Известные
|
||||
краевые случаи проверки JPEG и ограничения инфраструктуры перечислены
|
||||
в [KNOWN_ISSUES.md](KNOWN_ISSUES.md).
|
||||
|
||||
Источники и ИИ: документация RabbitMQ, PostgreSQL, aio-pika и Pillow;
|
||||
Claude — реализация и документация под проверку автора, Codex —
|
||||
замеры и оформление результатов. Подробнее — раздел 8 отчёта.
|
||||
|
||||
- [x] Обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [x] Контракт и публичные тесты не ослаблены.
|
||||
- [x] `make check` и `make test` выполнены; реальный вывод приложен.
|
||||
- [x] Сценарии отказов и безопасности воспроизведены.
|
||||
- [x] Секреты и приватные ключи находятся вне Git; отчёт не содержит токенов.
|
||||
- [x] README задания сохранён; REPORT.md описывает запуск решения.
|
||||
@@ -28,3 +28,403 @@ test-persistence:
|
||||
test-failover:
|
||||
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
|
||||
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Собственные цели решения. Цели skeleton выше (help, check, validate, test,
|
||||
# load, test-persistence, test-failover) не менять: в следующей лабе
|
||||
# Makefile придёт новый, свои цели переносятся в него слиянием.
|
||||
# ===========================================================================
|
||||
|
||||
# .env: подстановки compose и окружение приложения на хосте. Без него —
|
||||
# значения по умолчанию ниже (пароли и ключи пустые: сначала `make env`)
|
||||
-include .env
|
||||
|
||||
IMAGE ?= mtusi-messenger-vladimirceva:lab5
|
||||
# порт HAProxy на 127.0.0.1 — единственный порт API стенда на хосте
|
||||
HOST_PORT ?= 8080
|
||||
# порт сервиса на хосте (make run): другой, чем у стенда, — оба могут
|
||||
# работать одновременно
|
||||
RUN_PORT ?= 8081
|
||||
POSTGRES_USER ?= messenger
|
||||
POSTGRES_DB ?= messenger
|
||||
POSTGRES_HOST_PORT ?= 5433
|
||||
VALKEY_HOST_PORT ?= 6380
|
||||
S3_HOST_PORT ?= 8334
|
||||
BROKER_HOST_PORT ?= 5673
|
||||
SESSION_TTL_SECONDS ?= 3600
|
||||
# реплик API за HAProxy; разово — `make replicas API_REPLICAS=3`
|
||||
API_REPLICAS ?= 2
|
||||
DC = docker compose
|
||||
UV_EXPORT = uv export --frozen --no-dev --no-emit-project --no-header --format requirements.txt
|
||||
# долгие замеры на macOS — под caffeinate: ноутбук не уснёт по простою
|
||||
CAFFEINATE := $(shell command -v caffeinate >/dev/null 2>&1 && echo caffeinate -isu)
|
||||
|
||||
# --- приложение на хосте ----------------------------------------------------
|
||||
|
||||
.PHONY: install
|
||||
install:
|
||||
uv sync
|
||||
|
||||
# окружение сервиса и миграций на хосте: хранилища стенда на 127.0.0.1,
|
||||
# пароли и ключи пользователя API в S3 из .env. .env читается в рецепте, а
|
||||
# не при разборе Makefile: так первый запуск видит и файл, только что
|
||||
# созданный `make env`. Рецепт без эха — секреты не печатаются
|
||||
LOCAL_ENV = set -a; . ./.env; set +a; \
|
||||
export DB_HOST=127.0.0.1 DB_PORT="$$POSTGRES_HOST_PORT" DB_NAME="$$POSTGRES_DB" \
|
||||
DB_USER="$$POSTGRES_USER" DB_PASSWORD="$$POSTGRES_PASSWORD" CACHE_HOST=127.0.0.1 \
|
||||
CACHE_PORT="$$VALKEY_HOST_PORT" CACHE_PASSWORD="$$VALKEY_PASSWORD" \
|
||||
S3_ENDPOINT="http://127.0.0.1:$$S3_HOST_PORT" S3_ACCESS_KEY_ID="$$S3_API_ACCESS_KEY" \
|
||||
S3_SECRET_ACCESS_KEY="$$S3_API_SECRET_KEY" BROKER_HOST=127.0.0.1 \
|
||||
BROKER_PORT="$$BROKER_HOST_PORT" BROKER_PASSWORD="$$BROKER_PASSWORD"
|
||||
# worker на хосте — ключи своего пользователя S3, а не API
|
||||
WORKER_ENV = $(LOCAL_ENV); export S3_ACCESS_KEY_ID="$$S3_WORKER_ACCESS_KEY" \
|
||||
S3_SECRET_ACCESS_KEY="$$S3_WORKER_SECRET_KEY"
|
||||
|
||||
# PostgreSQL, Valkey и S3 из compose, миграции с хоста, затем один процесс
|
||||
# сервиса на 127.0.0.1:RUN_PORT без LB, текстовые логи в stdout; стоп —
|
||||
# Ctrl+C. Срок сессии — `make run SESSION_TTL_SECONDS=2`
|
||||
.PHONY: run
|
||||
run: migrate-local
|
||||
@echo "uv run python -m src # 127.0.0.1:$(RUN_PORT), SESSION_TTL_SECONDS=$(SESSION_TTL_SECONDS)"
|
||||
@$(LOCAL_ENV); APP_PORT=$(RUN_PORT) SESSION_TTL_SECONDS=$(SESSION_TTL_SECONDS) \
|
||||
uv run python -m src
|
||||
|
||||
# relay и worker на хосте рядом с `make run` (брокер и хранилища — из
|
||||
# compose); стоп — Ctrl+C. Сервисы compose relay и worker при этом лучше
|
||||
# остановить: make stop-worker, docker compose stop relay
|
||||
.PHONY: run-relay run-worker
|
||||
run-relay: migrate-local
|
||||
@$(LOCAL_ENV); uv run python -m src.relay
|
||||
|
||||
run-worker: migrate-local
|
||||
@$(WORKER_ENV); uv run python -m src.worker
|
||||
|
||||
# миграции с хоста в PostgreSQL стенда (127.0.0.1:POSTGRES_HOST_PORT)
|
||||
.PHONY: migrate-local
|
||||
migrate-local: db-up
|
||||
@echo "uv run alembic upgrade head # 127.0.0.1:$(POSTGRES_HOST_PORT)"
|
||||
@$(LOCAL_ENV); uv run alembic upgrade head
|
||||
|
||||
.PHONY: lint
|
||||
lint:
|
||||
uv run ruff format --check .
|
||||
uv run ruff check .
|
||||
|
||||
.PHONY: fmt
|
||||
fmt:
|
||||
uv run ruff format .
|
||||
uv run ruff check --fix .
|
||||
|
||||
# собственные тесты против запущенного стенда (BASE_URL, по умолчанию
|
||||
# HAProxy рабочего стенда); EXPECT_REPLICAS — сколько реплик за BASE_URL
|
||||
# (против `make run` — 1: тесты реплик пропускаются с причиной).
|
||||
# Тесты с маркером fault сюда не входят, для них есть make failure-checks
|
||||
EXPECT_REPLICAS ?= $(API_REPLICAS)
|
||||
.PHONY: student-test
|
||||
student-test:
|
||||
BASE_URL=$(BASE_URL) EXPECT_REPLICAS=$(EXPECT_REPLICAS) uv run pytest
|
||||
|
||||
# отдельный стенд из того же compose.yaml (свои PostgreSQL, Valkey, S3,
|
||||
# volume, HAProxy и две реплики, свои пароли и ключи, без .env): сборка,
|
||||
# миграции и bootstrap S3, make test и make student-test через его LB,
|
||||
# test-failover, test-persistence, затем уборка при любом исходе. Рабочий
|
||||
# стенд не трогается. $(PYTHON), а не uv run: uv повторно шлёт SIGINT, и
|
||||
# Ctrl+C обрывал бы уборку
|
||||
.PHONY: test-isolated
|
||||
test-isolated:
|
||||
$(PYTHON) student-tests/stand.py isolated
|
||||
|
||||
# pytest -m fault на ещё одном одноразовом стенде: собственные хранилища и
|
||||
# S3, уборщик с grace 25 с, несколько API с разными настройками (обычный,
|
||||
# «тесный», «узкий по загрузкам») и hook student-tests/fault_hook.py.
|
||||
# Дополнительные аргументы pytest — через FAULT_ARGS
|
||||
.PHONY: failure-checks
|
||||
failure-checks:
|
||||
$(PYTHON) student-tests/stand.py failures $(FAULT_ARGS)
|
||||
|
||||
# requirements.txt для образа — только из uv.lock, с хешами
|
||||
.PHONY: requirements
|
||||
requirements:
|
||||
$(UV_EXPORT) --output-file requirements.txt
|
||||
|
||||
.PHONY: requirements-check
|
||||
requirements-check:
|
||||
$(UV_EXPORT) | diff -u requirements.txt - && echo "requirements.txt matches uv.lock"
|
||||
|
||||
# --- Docker и compose ---------------------------------------------------------
|
||||
|
||||
# .env из шаблона со случайными паролями PostgreSQL и Valkey и ключами S3:
|
||||
# в шаблоне они пустые, общего пароля стенда в git нет. Существующий .env
|
||||
# не перезаписывается — дописываются только переменные шаблона, которых в
|
||||
# нём нет (секреты — случайными). Печатаются только имена
|
||||
.PHONY: env
|
||||
env:
|
||||
@$(PYTHON) scripts/lab5.py env
|
||||
|
||||
.PHONY: docker-build
|
||||
docker-build:
|
||||
docker build -t $(IMAGE) .
|
||||
|
||||
# весь стенд: хранилища и брокер -> migrate, s3-bootstrap -> api ×
|
||||
# API_REPLICAS, relay, worker, cleaner -> haproxy; --wait ждёт healthy.
|
||||
# Если образ поменялся, compose пересоздаст все реплики сразу и API на
|
||||
# это время пропадёт; плавная замена — через make rolling-update
|
||||
.PHONY: up
|
||||
up: env
|
||||
$(DC) up -d --build --wait api relay worker cleaner haproxy
|
||||
|
||||
# остановить контейнеры (SIGTERM); данные и контейнеры остаются. LB
|
||||
# останавливается раньше реплик (depends_on); S3 — плавно, ~15 с
|
||||
.PHONY: stop
|
||||
stop:
|
||||
$(DC) stop
|
||||
|
||||
# удалить контейнеры; volume pgdata и s3data с данными остаются
|
||||
# (удаление — wipe)
|
||||
.PHONY: down
|
||||
down:
|
||||
$(DC) down
|
||||
|
||||
# журнал всех реплик; префикс строки — имя контейнера реплики
|
||||
.PHONY: logs
|
||||
logs:
|
||||
$(DC) logs -f api
|
||||
|
||||
# журнал фона: relay и worker
|
||||
.PHONY: logs-pipeline
|
||||
logs-pipeline:
|
||||
$(DC) logs -f relay worker
|
||||
|
||||
# хранилища стенда с ожиданием healthcheck и bootstrap S3: для make run.
|
||||
# --wait одноразовый сервис не ждёт — код bootstrap отдельно
|
||||
.PHONY: db-up
|
||||
db-up: env
|
||||
$(DC) up -d --wait postgres valkey s3 rabbitmq
|
||||
$(DC) up --no-deps --exit-code-from s3-bootstrap s3-bootstrap
|
||||
|
||||
# снести стенд вместе с volume (данные пропадут); спрашивает подтверждение yes
|
||||
.PHONY: wipe
|
||||
wipe:
|
||||
@read -p "Удалить все данные стенда (volume pgdata, s3data, rmqdata)? Введите yes: " answer; \
|
||||
if [ "$$answer" = "yes" ]; then \
|
||||
$(DC) down -v --remove-orphans; \
|
||||
else \
|
||||
echo "cancelled, nothing removed"; exit 1; \
|
||||
fi
|
||||
|
||||
# --- Реплики за HAProxy (лаба 3) --------------------------------------------
|
||||
|
||||
# вернуть остановленные реплики (те же контейнеры и X-Instance-Id) и
|
||||
# выставить их число API_REPLICAS; ждёт healthy
|
||||
.PHONY: replicas
|
||||
replicas:
|
||||
$(DC) up -d --no-deps --no-recreate --wait --scale api=$(API_REPLICAS) api
|
||||
|
||||
# вход одним синтетическим пользователем и серия GET /users/me с одним
|
||||
# токеном через LB: X-Instance-Id каждого ответа и итог по репликам
|
||||
.PHONY: instances
|
||||
instances:
|
||||
$(PYTHON) scripts/lab5.py instances --base-url "$(BASE_URL)"
|
||||
|
||||
# таблица слотов backend api: IP реплики, статус (во время замены бывает
|
||||
# DRAIN/MAINT) и счётчики запросов, сбоев соединения, retry и redispatch
|
||||
.PHONY: lb-stats
|
||||
lb-stats:
|
||||
$(PYTHON) scripts/lab5.py lb-stats
|
||||
|
||||
# make test-failover для каждой реплики по очереди с возвратом
|
||||
# предыдущей (make replicas): какой контейнер остановлен, строка
|
||||
# STOP_ONE_LOG и итог теста. STOP_MODE=kill — без graceful
|
||||
STOP_MODE ?= stop
|
||||
.PHONY: failover-each
|
||||
failover-each:
|
||||
$(PYTHON) scripts/lab5.py failover-each --base-url "$(BASE_URL)" \
|
||||
--replicas $(API_REPLICAS) --mode $(STOP_MODE)
|
||||
|
||||
# выкатить текущий код без простоя: собрать образ, один раз прогнать
|
||||
# миграции и менять реплики по одной, выводя старую через drain в LB
|
||||
.PHONY: rolling-update
|
||||
rolling-update:
|
||||
API_REPLICAS=$(API_REPLICAS) scripts/rolling-update.sh
|
||||
|
||||
# --- S3 и вложения (с лабы 4) -------------------------------------------------
|
||||
|
||||
# вложение через LB → остановка реплики, принявшей загрузку (stop-one.sh по
|
||||
# её X-Instance-Id) → скачивание через другую, тот же SHA-256 → 404
|
||||
# постороннему → make replicas. STOP_MODE=kill — без graceful
|
||||
.PHONY: attachment-failover
|
||||
attachment-failover:
|
||||
$(PYTHON) scripts/lab5.py attachment-failover --base-url "$(BASE_URL)" \
|
||||
--replicas $(API_REPLICAS) --mode $(STOP_MODE)
|
||||
|
||||
# приватность bucket вне API: права ключей API, worker и уборщика (bootstrap в
|
||||
# режиме check), анонимные запросы с хоста и из сети стенда, порты
|
||||
# контейнера S3; затем уборка проверочных объектов. Код 1 при нарушении
|
||||
.PHONY: s3-check
|
||||
s3-check:
|
||||
$(PYTHON) scripts/lab5.py s3-check
|
||||
|
||||
# разовый проход уборщика незавершённых загрузок: 0 — проход дошёл до
|
||||
# конца, 1 — прерван отказом, 2 — конфигурация
|
||||
.PHONY: cleanup-once
|
||||
cleanup-once:
|
||||
$(DC) run --rm --no-deps cleaner --once
|
||||
|
||||
# разовая сверка bucket и БД (трек 2) в контейнере уборщика. По умолчанию
|
||||
# dry-run; APPLY=1 — исправить безопасное; MAX_CHANGES, STRATEGY=head,
|
||||
# DEEP=1 (SHA-256 содержимого), RATE (объектов S3/с, 0 — без предела).
|
||||
# Выход 0 — всё сошлось, 3 — есть неисправленные расхождения, 1 — проход
|
||||
# не дошёл до конца, 2 — ошибка в настройках
|
||||
APPLY ?=
|
||||
MAX_CHANGES ?=
|
||||
STRATEGY ?=
|
||||
DEEP ?=
|
||||
RATE ?=
|
||||
.PHONY: reconcile
|
||||
reconcile:
|
||||
$(DC) run --rm --no-deps cleaner reconcile $(if $(APPLY),--apply,) \
|
||||
$(if $(MAX_CHANGES),--max-changes $(MAX_CHANGES),) \
|
||||
$(if $(STRATEGY),--strategy $(STRATEGY),) $(if $(DEEP),--deep,) \
|
||||
$(if $(RATE),--rate $(RATE),)
|
||||
|
||||
# --- Брокер, relay, worker (лаба 5) -------------------------------------------
|
||||
|
||||
# worker стенда: остановить (шаг 2 сценария защиты) / снова запустить
|
||||
.PHONY: stop-worker start-worker
|
||||
stop-worker:
|
||||
$(DC) stop worker
|
||||
|
||||
start-worker:
|
||||
$(DC) up -d --no-deps worker
|
||||
|
||||
# сценарий защиты, шаги 2–3, одной командой: worker стоит → 202 → сообщение
|
||||
# → 409 → scripts/restart.sh → worker → ready и thumbnail 256×160
|
||||
.PHONY: defense
|
||||
defense:
|
||||
$(PYTHON) scripts/lab5.py defense --base-url "$(BASE_URL)"
|
||||
|
||||
# задания по состояниям и очередь брокера
|
||||
.PHONY: pipeline-state
|
||||
pipeline-state:
|
||||
@$(PYTHON) scripts/lab5.py pipeline
|
||||
|
||||
# реестр failed и replay в контейнере relay. Без APPLY=1 — только показ;
|
||||
# с APPLY=1 нужен выбор (ATTACHMENT=<id> или ERROR_CODE=...) и OPERATOR,
|
||||
# REASON. Коды: 0 — готово, 3 — часть отказана, 1 — PostgreSQL, 2 — аргументы
|
||||
ATTACHMENT ?=
|
||||
ERROR_CODE ?=
|
||||
LIMIT ?= 20
|
||||
OPERATOR ?=
|
||||
REASON ?=
|
||||
.PHONY: replay
|
||||
replay:
|
||||
$(DC) run --rm --no-deps relay replay $(if $(ATTACHMENT),--attachment $(ATTACHMENT),) \
|
||||
$(if $(ERROR_CODE),--error-code $(ERROR_CODE),) --limit $(LIMIT) \
|
||||
$(if $(APPLY),--apply --operator "$(OPERATOR)" --reason "$(REASON)",)
|
||||
|
||||
# --- Нагрузочный стенд (лаба 5, трек 1 — очередь под перегрузкой; профили
|
||||
# лаб 3 и 4 сохранены: отказ и замена под нагрузкой, память загрузок,
|
||||
# стоимость сверки) --
|
||||
# Порядок работы — bench/RUNBOOK.md, правила — bench/AGENTS.md. Свой проект
|
||||
# compose vlad-bench, рабочий стенд перед замером — make stop.
|
||||
|
||||
QUEUE_LABEL ?= baseline
|
||||
FAILOVER_LABEL ?= baseline
|
||||
ROLLOUT_LABEL ?= baseline
|
||||
UPLOADS_LABEL ?= baseline
|
||||
RECONCILE_LABEL ?= baseline
|
||||
# серия трека 1 лабы 3 (сохраняется): a, b, db, ab, quick
|
||||
SERIES ?=
|
||||
SCALING_LABEL ?=
|
||||
LABELS ?=
|
||||
PLOTS ?= scaling
|
||||
BENCH = $(CAFFEINATE) uv run python bench/run.py
|
||||
|
||||
# лаба 5, трек 1: настройки concurrency/prefetch worker × классы изображений;
|
||||
# ёмкость, ровная нагрузка, перегрузка при пределе незавершённых и
|
||||
# восстановление; пишет evidence/queue/<метка>/
|
||||
.PHONY: bench-queue
|
||||
bench-queue:
|
||||
$(BENCH) queue --label "$(QUEUE_LABEL)"
|
||||
|
||||
# самопроверка профиля очереди (~9 мин): c1p1 и c4p4 × классы s и x, один
|
||||
# повтор, короткие окна; метка quick-queue-<UTC> в отчёт не идёт
|
||||
.PHONY: bench-queue-quick
|
||||
bench-queue-quick:
|
||||
$(BENCH) queue-quick
|
||||
|
||||
# уровень 4 лабы 3: две реплики под постоянной нагрузкой, каждая в режимах
|
||||
# stop, kill, pause с возвратом; пишет evidence/failover/<метка>/
|
||||
.PHONY: bench-failover
|
||||
bench-failover:
|
||||
$(BENCH) failover --label "$(FAILOVER_LABEL)"
|
||||
|
||||
# проверка стенда отказа (~2 мин); метка quick-failover-<UTC> в отчёт не идёт
|
||||
.PHONY: bench-failover-quick
|
||||
bench-failover-quick:
|
||||
$(BENCH) failover-quick
|
||||
|
||||
# уровень 4 лабы 3: поочерёдная замена (scripts/rolling-update.sh) под той
|
||||
# же нагрузкой, с новой expand-ревизией; пишет evidence/rollout/<метка>/
|
||||
.PHONY: bench-rollout
|
||||
bench-rollout:
|
||||
$(BENCH) rollout --label "$(ROLLOUT_LABEL)"
|
||||
|
||||
.PHONY: bench-rollout-quick
|
||||
bench-rollout-quick:
|
||||
$(BENCH) rollout-quick
|
||||
|
||||
# лаба 4, уровень 4: сколько памяти занимает реплика, когда параллельно
|
||||
# грузят 1, 10, 20 или 40 клиентов тела ~1 и ~10 MiB; сравниваются места
|
||||
# по умолчанию и режим без лимита. Результат — evidence/uploads/<метка>/
|
||||
.PHONY: bench-uploads
|
||||
bench-uploads:
|
||||
$(BENCH) uploads --label "$(UPLOADS_LABEL)"
|
||||
|
||||
# самопроверка профиля загрузок (~5 мин); метка quick-uploads-<UTC>
|
||||
.PHONY: bench-uploads-quick
|
||||
bench-uploads-quick:
|
||||
$(BENCH) uploads-quick
|
||||
|
||||
# уровень 5, трек 2: стоимость сверки bucket и БД по размеру набора,
|
||||
# стратегиям и под фоновой нагрузкой; пишет evidence/reconcile/<метка>/
|
||||
.PHONY: bench-reconcile
|
||||
bench-reconcile:
|
||||
$(BENCH) reconcile --label "$(RECONCILE_LABEL)"
|
||||
|
||||
# самопроверка профиля сверки (~6 мин); метка quick-reconcile-<UTC>
|
||||
.PHONY: bench-reconcile-quick
|
||||
bench-reconcile-quick:
|
||||
$(BENCH) reconcile-quick
|
||||
|
||||
# трек 1 лабы 3 (сохраняется для сравнения, в лабе 4 не обязателен):
|
||||
# серия конфигураций, на каждую ячейку свежий стенд, открытая модель,
|
||||
# поиск R*; пишет evidence/scaling/<метка>/. CONFIGS="4x1-pg0.5 4x1" —
|
||||
# ячейки вместо профильных; RESUME=1 — продолжить прерванную метку
|
||||
CONFIGS ?=
|
||||
.PHONY: bench-scaling
|
||||
bench-scaling:
|
||||
@test -n "$(SERIES)" || (echo "usage: make bench-scaling SERIES=a|b|db|ab [SCALING_LABEL=...] [CONFIGS=...] [RESUME=1]"; exit 2)
|
||||
$(BENCH) scaling --series "$(SERIES)" $(if $(SCALING_LABEL),--label "$(SCALING_LABEL)",) \
|
||||
$(if $(CONFIGS),--configs "$(CONFIGS)",) $(if $(RESUME),--resume,)
|
||||
|
||||
# самопроверка серий (~5 мин): 1 × 1 и 2 × 1, короткие окна
|
||||
.PHONY: bench-scaling-quick
|
||||
bench-scaling-quick:
|
||||
$(BENCH) scaling --series quick
|
||||
|
||||
# сравнение двух меток серий (по общим конфигурациям); AA — метка того же
|
||||
# кода, что BEFORE: уровень шума
|
||||
.PHONY: bench-compare
|
||||
bench-compare:
|
||||
@test -n "$(BEFORE)" -a -n "$(AFTER)" || \
|
||||
(echo "usage: make bench-compare BEFORE=<label> AFTER=<label> [AA=<label>]"; exit 2)
|
||||
uv run python bench/compare.py --before "$(BEFORE)" --after "$(AFTER)" \
|
||||
$(if $(AA),--aa "$(AA)",)
|
||||
|
||||
# графики SVG и сводка серий трека 1: evidence/plots/<PLOTS>/
|
||||
.PHONY: bench-plots
|
||||
bench-plots:
|
||||
@test -n "$(LABELS)" || (echo "usage: make bench-plots LABELS=\"scaling-a scaling-b\" [PLOTS=scaling]"; exit 2)
|
||||
uv run python bench/plots.py --out "$(PLOTS)" $(LABELS)
|
||||
|
||||
@@ -1,37 +1,518 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Владимирцева, БВТ2301
|
||||
- Целевая оценка: **5**; трек 5 — **трек 1 «Очередь под перегрузкой»**
|
||||
- Версия кода: `src_sha256 4c4bb305b2747d154105720b93261ef476675670470bfa7a10b338dd79514da9`
|
||||
- Источники и помощь ИИ — раздел 8
|
||||
|
||||
## Запуск из чистого clone
|
||||
Состояние: всё, что не требует нагрузки (уровни 3 и 4 целиком, проверки
|
||||
отказов, сценарий защиты), снято на машине разработки — раздел 3 и
|
||||
`evidence/`. Стенд замеров трека 1 — ВМ Docker с 10 CPU и 7,65 ГиБ RAM.
|
||||
Подробности — раздел 4.3 и `evidence/tests/measurement-host.txt`.
|
||||
Трек 1 завершён: [queue10](evidence/queue/queue10/summary.md),
|
||||
24 ячейки; Q1–Q5 подтверждены. Результаты — раздел 4.2.
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
## 1. Запуск и демонстрация
|
||||
|
||||
## Архитектура и решения
|
||||
Нужны Docker Desktop (Compose v2), `uv`, Python 3.9+ для проверок
|
||||
skeleton. Версии: Python 3.13.15 (образ), PostgreSQL 18.6, Valkey 9.1.2,
|
||||
HAProxy 3.4.6, SeaweedFS 4.48, RabbitMQ 4.3.7 — все образы по тегу и
|
||||
digest (`compose.yaml`, `Dockerfile`).
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
```sh
|
||||
make install # uv sync: тесты и линтер на хосте
|
||||
make env # .env со случайными паролями и ключами (только имена в выводе)
|
||||
make up # сборка, миграции, bootstrap S3, 2 реплики, relay, worker, cleaner, LB
|
||||
make test # публичный smoke лабы 5
|
||||
make student-test # свои тесты против стенда
|
||||
make test-isolated # то же на отдельном стенде + failover + persistence
|
||||
make failure-checks # проверки отказов (в том числе pipeline) на отдельном стенде
|
||||
make defense # шаги 2–3 сценария защиты одной командой
|
||||
make stop # остановить; данные в volume остаются
|
||||
make wipe # намеренно удалить данные (спросит yes)
|
||||
```
|
||||
|
||||
## Доказательства критериев
|
||||
Порты заняты — `make env HOST_PORT=8180 POSTGRES_HOST_PORT=5533
|
||||
VALKEY_HOST_PORT=6480 S3_HOST_PORT=8434 BROKER_HOST_PORT=5773` до первого
|
||||
`make up` и `BASE_URL=http://localhost:8180` у проверок.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
**Сценарий защиты.**
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
1. `make test` → `PASS: lab5 public API scenario`: 202, ready ≤ 60 с,
|
||||
оригинал по SHA-256 и настоящий JPEG thumbnail 256×160.
|
||||
2–3. `make defense`: worker остановлен → загрузка 202 `queued` →
|
||||
сообщение со ссылкой 201 → байты 409 → текст 201 → `scripts/restart.sh`
|
||||
(LB, реплики, PostgreSQL, Valkey, S3, RabbitMQ с volume) → вложение всё
|
||||
ещё `queued` → worker запущен → `ready`, оригинал и thumbnail; печатает
|
||||
id и `StartedAt` до и после (`evidence/attachments/defense.txt`).
|
||||
4. Уровень 4: окна отказа воспроизводит `make failure-checks`
|
||||
(`student-tests/test_pipeline.py`): kill worker после записи объектов
|
||||
до фиксации и после фиксации до ack, relay остановлен (COMMIT →
|
||||
publish) и убит после подтверждения, повторы до бюджета и replay
|
||||
`failed`. Вручную: `make replay`, `make replay APPLY=1
|
||||
ATTACHMENT=<id> OPERATOR=<кто> REASON=<зачем>`. Для 5 — `make
|
||||
bench-queue-quick` (самопроверка профиля, ~10 мин).
|
||||
|
||||
## Эксперименты и отказы
|
||||
## 2. Архитектура и решения
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
```
|
||||
клиент → HAProxy → api × 2 ─┬─ PostgreSQL: строки, задания (outbox), варианты, журнал replay
|
||||
├─ S3: <база>/staging/<id> (принятый оригинал)
|
||||
└─ Valkey: карточки чатов
|
||||
relay → PostgreSQL (outbox) → RabbitMQ (quorum, <база>.attachment-jobs)
|
||||
worker ← RabbitMQ; → PostgreSQL (захват, итог) и S3 (<база>/attachments/<id>/original, …/p1/<вариант>)
|
||||
└─ процесс рендера на задание (Pillow, RLIMIT_AS, RLIMIT_CPU, без секретов)
|
||||
cleaner → PostgreSQL, S3: брошенные загрузки, staging, сверка
|
||||
```
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
**Источник истины — PostgreSQL.** Сообщение брокера — только подсказка
|
||||
«посмотри задание X, попытка N» (`contracts/jobs.schema.json`: id, тип,
|
||||
версия pipeline, попытка, время). Что делать, решает строка
|
||||
`attachment_jobs` под `FOR UPDATE`.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
**Брокер — RabbitMQ 4**: quorum-очередь хранит сообщения на диске
|
||||
(`rmqdata`), publisher confirms и ручной ack есть из коробки, `basic.qos`
|
||||
даёт prefetch — рычаг трека 1. Valkey Streams отпал: у стенда Valkey —
|
||||
кэш без persistence, второй экземпляр с AOF ради очереди дороже.
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
### Приём (уровень 3)
|
||||
|
||||
Порядок в `AttachmentService.upload`: проверки → предел незавершённой
|
||||
работы → место загрузки → тело → структура и SHA-256 → строка `pending`
|
||||
→ PUT в staging → `queued` + задание одной транзакцией → **202**. В
|
||||
момент 202 сохранены: полный проверенный оригинал в приватном S3
|
||||
(staging), метаданные и намерение обработки (строка задания). Брокера в
|
||||
пути запроса нет: у реплик API нет ни адреса, ни пароля брокера.
|
||||
Остановленные брокер, relay или worker не мешают загрузке, тексту,
|
||||
истории, метаданным и готовым байтам; `/health/ready` проверяет только
|
||||
PostgreSQL.
|
||||
|
||||
### Outbox и relay (уровень 4)
|
||||
|
||||
Разрыв «COMMIT в БД, затем publish» закрыт transactional outbox: задание
|
||||
пишется в той же транзакции, что `queued`; публикует отдельный процесс
|
||||
`relay` (`PostgresPipeline.publish_due`): выборка `FOR UPDATE SKIP
|
||||
LOCKED`, публикация пачки с подтверждением брокера **внутри**
|
||||
транзакции, отметка `published_at` и номер попытки — после подтверждения.
|
||||
Relay умер до publish — задание в outbox, новый relay его отдаст. Умер
|
||||
после confirm до commit — та же попытка уйдёт ещё раз: дубль, который
|
||||
worker отбросит. Ещё relay возвращает истёкшие аренды и раз в
|
||||
`RELAY_REPUBLISH_AFTER/4` переопубликовывает «потерянное» (опубликовано
|
||||
давно, а очередь пуста — например, потерян volume брокера).
|
||||
|
||||
### Worker, аренда, fencing, идемпотентность (уровни 3–4)
|
||||
|
||||
`claim_job` переводит задание в `processing` с арендой `JOB_TIMEOUT +
|
||||
JOB_LEASE_MARGIN` (80 с), только если сообщение — про текущую
|
||||
опубликованную попытку. Дубль при живой аренде, старая попытка, раннее
|
||||
сообщение, готовое или `failed` задание — no-op с ack. Результат
|
||||
(`finish_job`) фиксируется одной транзакцией — варианты по ключу эффекта
|
||||
`(attachment_id, pipeline_version, variant)` с `ON CONFLICT DO NOTHING`,
|
||||
`ready` вложения и задания — и только пока попытка ещё текущая (номер
|
||||
попытки в `WHERE` — fencing). Ключи объектов детерминированы: повтор
|
||||
перезаписывает те же байты. Ack — только после записанного исхода.
|
||||
Пульса аренды нет: срок рендера ограничен `JOB_TIMEOUT_SECONDS`, а запас
|
||||
покрывает S3 и PostgreSQL до и после.
|
||||
|
||||
Зависший worker от долгого: долгий укладывается в аренду (срок рендера
|
||||
меньше её, процесс убивается по сроку), зависший её теряет — relay
|
||||
возвращает задание, а поздний результат старой попытки не зафиксируется
|
||||
(fencing). Двух неконтролируемых исполнений нет: второй захват возможен
|
||||
только после истечения аренды, и итог фиксирует только одна попытка.
|
||||
|
||||
### Повторы, ошибки, реестр и replay (уровень 4)
|
||||
|
||||
Временные ошибки (S3, срок рендера по часам, сбой процесса) — `queued` с
|
||||
`next_attempt_at = now + delay`, `delay = d/2 + U(0, d/2)`, `d = min(120,
|
||||
2 · 2^(n−1))` (equal jitter); бюджет — `JOB_MAX_ATTEMPTS` (5), затем
|
||||
`failed` `retry_exhausted`. Постоянные — сразу `failed`: не декодируется
|
||||
(`invalid_image`), больше лимита пикселей или памяти (`image_too_large`),
|
||||
нет staging (`source_missing`), процесс съел весь `RLIMIT_CPU`
|
||||
(`processing_failed`). Реестр — строки `failed` (`make replay`). Replay —
|
||||
`make replay APPLY=1 …`: одной транзакцией запись в
|
||||
`attachment_job_replays` (кто, зачем, прежняя ошибка и попытка),
|
||||
`attempt_base = attempt` (новый бюджет; номер попытки продолжается, и
|
||||
старые сообщения остаются «чужими»), `queued`. Готовое повторно не
|
||||
исполняется (`already_ready`, код 3); без staging — `source_gone`.
|
||||
|
||||
### Варианты и лимиты обработки (уровни 3–4)
|
||||
|
||||
Процесс рендера на задание (`src/worker/render.py`): первыми строками
|
||||
`RLIMIT_AS` = `RENDER_MEMORY_MB` (768 МиБ), `RLIMIT_CPU` = срок задания,
|
||||
`RLIMIT_CORE` = 0, затем Pillow; `MAX_IMAGE_PIXELS` = 40 млн, бомба
|
||||
распаковки — отказ; окружение собрано заново (без паролей и ключей);
|
||||
байты — stdin, результат — stdout; срок по часам — kill. Байты на входе
|
||||
ограничены 10 MiB ещё в API и повторно при чтении staging. JPEG
|
||||
декодируется с `draft` до размера, который нужен вариантам.
|
||||
|
||||
- thumbnail — JPEG q=85, вписан в 256×256 без увеличения, округление вниз,
|
||||
минимум 1 px (320×200 → 256×160);
|
||||
- crop — JPEG q=85, центральный квадрат → ровно 128×128;
|
||||
- optimized — WebP q=80, method=4, вписан в 1280×1280 без увеличения,
|
||||
альфа сохраняется, EXIF/ICC не переносятся (файл может оказаться больше
|
||||
исходника — не обещаем уменьшения);
|
||||
- EXIF orientation применяется до всего; прозрачность в JPEG — на белом.
|
||||
|
||||
`ready` ставится только после оригинала и всех трёх вариантов.
|
||||
|
||||
### Staging и уборка (уровень 4)
|
||||
|
||||
Staging убирает только `cleaner`: у `ready` — следующим проходом (60 с), у
|
||||
`failed` — через `STAGING_RETENTION_SECONDS` (сутки, окно replay) вместе с
|
||||
объектами неудачных попыток. Строка — под `FOR NO KEY UPDATE SKIP
|
||||
LOCKED`; replay держит тот же замок, поэтому источник не исчезнет посреди
|
||||
возврата. Брошенные загрузки (`pending` старше 900 с) — как в лабе 4, но
|
||||
удаляется и staging. Сверка лабы 4 знает ключи вариантов.
|
||||
|
||||
### Предел принятой работы (трек 1)
|
||||
|
||||
`UPLOAD_MAX_UNFINISHED` (300 на рабочем стенде, 60 в профиле `queue`):
|
||||
если незавершённых заданий стенда (`queued`
|
||||
+ `processing`, все реплики, одним запросом по частичному индексу с
|
||||
`LIMIT`) не меньше предела — 429 `rate_limited`, `Retry-After: 5` до
|
||||
чтения тела; ни строки, ни объекта. Предел мягкий: одновременные
|
||||
загрузки после проверки превышают его не больше чем на число мест.
|
||||
|
||||
### Лимиты, пулы, порты и сроки
|
||||
|
||||
Лимиты контейнеров — `infra/AGENTS.md`. Соединения PostgreSQL: реплики 2 ×
|
||||
15 + migrate + cleaner + relay + worker 3 = 36 из 100. Наружу — только
|
||||
127.0.0.1: LB, PostgreSQL, Valkey, API S3, AMQP. Аренда 80 с > срок
|
||||
рендера 60 с; `stop_grace_period` worker 12 с > `WORKER_STOP_TIMEOUT` 8 с.
|
||||
|
||||
### Миграции и совместимость
|
||||
|
||||
`0004` — только расширение (колонки NULL, новые таблицы, `CHECK` с `NOT
|
||||
VALID`, индекс на рабочей таблице `CONCURRENTLY`). Реплики лабы 4 на ней
|
||||
работают, но отвечают 201 без заданий (`KNOWN_ISSUES.md`, «Смесь версий»).
|
||||
|
||||
### Конфигурация: логические имена `.env.example`
|
||||
|
||||
| Skeleton | Здесь |
|
||||
|---|---|
|
||||
| `BROKER_URL_FILE` | `BROKER_HOST`, `BROKER_PORT`, `BROKER_USER`, `BROKER_PASSWORD` (только relay и worker, из `.env` через compose) |
|
||||
| `WORKER_CONCURRENCY` | `WORKER_CONCURRENCY` (2) и `WORKER_PREFETCH` (2) |
|
||||
| `JOB_MAX_ATTEMPTS` | `JOB_MAX_ATTEMPTS` (5) |
|
||||
| `JOB_TIMEOUT_SECONDS` | `JOB_TIMEOUT_SECONDS` (60) + `JOB_LEASE_MARGIN_SECONDS` (20) = аренда |
|
||||
| `STAGING_RETENTION_SECONDS` | `STAGING_RETENTION_SECONDS` (86400) у cleaner |
|
||||
| `MAX_UPLOAD_BYTES`, `MAX_IMAGE_PIXELS` | константы контракта `src/constants.py` (окружение их не меняет) |
|
||||
| остальные (лабы 2–4) | как в отчёте лабы 4: `DB_*`, `CACHE_*`, `S3_*`, `API_REPLICAS`, `IDEMPOTENCY_TTL_SECONDS` |
|
||||
|
||||
### Согласования с преподавателем
|
||||
|
||||
2026-10-20 (лаба 4, действует): пустой текст — и `""`, и пробелы; с ≥ 1
|
||||
вложением — 201, без — 400.
|
||||
|
||||
## 3. Доказательства критериев
|
||||
|
||||
Доказательства без нагрузки сняты 5 ноября 2026; паспорта окружения —
|
||||
в шапках файлов. Один и тот же код —
|
||||
`src_sha256 4c4bb305…514da9` в шапке каждого файла. Рабочий стенд — LB на
|
||||
127.0.0.1:8180 (8080 на этой машине занят).
|
||||
|
||||
| Критерий | Команда | Наблюдали | Файл |
|
||||
|---|---|---|---|
|
||||
| 3: минимум лабы 4 сохранён, 201 → 202 | `make test-isolated` | `results` все 0: smoke, 140 своих тестов, failover (122 записи, 0 ошибок, восстановление 1,8 с), attachment-failover, persistence | `evidence/tests/test-isolated.txt` |
|
||||
| 3: публичный smoke | `make test BASE_URL=…:8180` | `PASS: lab5 public API scenario` | `evidence/attachments/smoke.txt` |
|
||||
| 3: 202 только после staging и задания; остановка API не теряет работу | `make defense` | worker стоит → 202 `queued`; рестарт LB, реплик, PostgreSQL, Valkey, S3, RabbitMQ (`StartedAt` позже) → всё ещё `queued` → worker → `ready` | `evidence/attachments/defense.txt` |
|
||||
| 3: отдельные broker и worker; API без брокера | `docker compose ps`, `config` | сервисы `rabbitmq`, `relay`, `worker`; `BROKER_PASSWORD` только у relay и worker; у api — ни адреса, ни пароля | `evidence/stand/stand.txt` |
|
||||
| 3: durable, at least once, ack после эффекта | `make failure-checks` | kill worker после фиксации до ack → повторная доставка no-op, та же попытка 1 | `evidence/tests/failure-checks.txt` |
|
||||
| 3: состояния в API; оригинал неизменен; thumbnail по контракту | `make test-isolated` (`test_processing.py`) | 320×200 → thumbnail 256×160, crop 128×128, optimized WebP; SHA-256 оригинала прежний; JPEG проверен декодером | `evidence/tests/test-isolated.txt` |
|
||||
| 3: ограниченный retry, `failed` с безопасным кодом; второго набора вариантов нет | `make failure-checks` | 3 попытки → `failed` `retry_exhausted`; после kill и повторов вариантов ровно 3 | `evidence/tests/failure-checks.txt` |
|
||||
| 3: демонстрация, задание переживает рестарт брокера | `make defense`, проба рестарта RabbitMQ | после stop/up брокера сообщение в очереди 15 с из 15, worker обработал его с `attempt=1` | `evidence/attachments/defense.txt`, `evidence/broker/restart.txt` |
|
||||
| 4: crop, optimized; лимиты bytes, пикселей, времени, памяти | `make test-isolated`; проба процесса рендера в образе | 413 на > 10 MiB и > 40 Мпикс; рендер: 42 Мпикс → код 4, 36 Мпикс при `RLIMIT_AS` 256 МиБ → 4, `RLIMIT_CPU` 1 с → SIGXCPU, не картинка → 3 | `evidence/tests/test-isolated.txt`, `evidence/worker/limits.txt` |
|
||||
| 4: outbox; crash между COMMIT и publish | `make failure-checks` (`test_relay_down_keeps_intent_in_outbox`, `test_relay_crash_after_confirm`) | relay стоит → задание в outbox, запуск relay → `ready` через 3,4 с; relay убит после confirm (код 137) → задание не отмечено, новый relay → `ready`, вариантов 3 | `evidence/tests/failure-checks.txt` |
|
||||
| 4: backoff с jitter, постоянные и временные, реестр, replay; готовое не повторяется | `make failure-checks` (`test_transient_errors_exhaust_then_replay`, `test_permanent_error_fails_at_first_attempt`) | временные S3 → 3 попытки → `retry_exhausted`; replay с журналом → `ready`; replay готового → `already_ready`, код 3; постоянная — `failed` с первой попытки | `evidence/tests/failure-checks.txt` |
|
||||
| 4: worker убит после записи объекта до ack; lease; prefetch; уборка staging | `make failure-checks` | kill после объектов до COMMIT → аренда истекла → `ready` второй попыткой через 13,4 с; staging убран через 1,0 с после `ready`; `prefetch=2` в журнале worker | `evidence/tests/failure-checks.txt`, `evidence/broker/restart.txt` |
|
||||
| перенос лаб 3–4: схема только expand | миграции на пустой базе, `alembic check` | `0001 → 0004`, повтор no-op, `downgrade -1` и обратно; новые `CHECK` — `NOT VALID`; расхождений нет | `evidence/stand/migrations.txt` |
|
||||
| перенос лаб 2–4: persistence, failover вложения, приватность S3 | `make test-persistence`, `make attachment-failover`, `make s3-check` | PASS; скачивание после остановки принявшей реплики за 0,51 с; 79 проверок прав, 0 расхождений (ключ worker: put/get, без delete и list) | `evidence/stand/persistence.txt`, `evidence/attachments/failover.txt`, `evidence/s3/privacy.txt` |
|
||||
| 5, трек 1 | `make bench-queue QUEUE_LABEL=queue10` | 24 ячейки, 120 фаз, 14185 принятых; 5xx / failed / missing 0; Q1–Q5 подтверждены | `evidence/queue/queue10/summary.md`, `meta.json`, `cells.csv`, `timeline.csv`, `jobs.csv.gz` |
|
||||
|
||||
## 4. Эксперименты и отказы
|
||||
|
||||
### 4.1. Отказы без нагрузки
|
||||
|
||||
`make failure-checks FAULT_ARGS=-rP` — 34 passed за 7 мин 21 с, отдельный
|
||||
стенд с короткими сроками (аренда 10 + 4 с, 3 попытки); числа —
|
||||
`evidence/tests/failure-checks.txt`.
|
||||
|
||||
| Воздействие | Ожидаем | Наблюдали |
|
||||
|---|---|---|
|
||||
| worker стоит, рестарт API и брокера | 202, ссылка 201, байты 409; после запуска worker — `ready` | `ready`, `attempt` 1 |
|
||||
| рестарт RabbitMQ с volume после публикации | сообщение цело, повторной публикации нет | в очереди через 0,6 с после старта; `ready` попыткой 1 |
|
||||
| relay стоит после COMMIT | намерение в outbox, загрузка не ждёт | запуск relay → `ready` через 3,4 с |
|
||||
| relay убит после confirm до отметки | та же попытка уйдёт ещё раз, обработка одна | код 137, задание `attempt` 0; затем `ready`, вариантов 3 |
|
||||
| worker убит после записи объектов до COMMIT | аренда истекает, relay возвращает, результат тот же | объекты записаны, `processing`; `ready` попыткой 2 через 13,4 с |
|
||||
| worker убит после COMMIT до ack | повторная доставка — no-op | `ready`, `attempt` 1, очередь 0 0 |
|
||||
| временная ошибка чтения staging, каждый раз | повторы с backoff до бюджета, затем `failed` | `retry_exhausted` после 3 попыток; replay → `ready` |
|
||||
| незавершённых заданий не меньше предела | 429 `Retry-After: 5`, без строки | PASS (`test_backlog_limit_is_429`) |
|
||||
| `ready` зафиксирован | staging убирает cleaner | убран через 1,0 с, остались оригинал и 3 варианта |
|
||||
| PostgreSQL остановлен / заморожен | быстрый 503 / 503 по таймауту | 0,01 с / 4,02 с и 1,01 с при занятом пуле |
|
||||
| Valkey остановлен / заморожен | обход кэша / один таймаут | 200 и 201 / 0,206 с, затем 0,003 с |
|
||||
| S3 остановлен / заморожен | 503 на загрузке и скачивании | 0,058 с / 5,02 и 5,03 с |
|
||||
| kill реплики между PUT staging и `queued` | клиент видит обрыв; уборщик убирает строку и объект | `RemoteProtocolError`; убрано через 20,0 с |
|
||||
| сверка с обеих сторон | dry-run находит, apply чинит | 5 видов расхождений найдено, после исправления 0 |
|
||||
|
||||
Отдельно — процесс рендера в образе стенда (`evidence/worker/limits.txt`):
|
||||
36 Мпикс укладываются в 768 МиБ (пик RSS 300–410 МиБ, 1,1–1,4 с), при
|
||||
256 МиБ — код 4 (`image_too_large`); заголовок на 42 Мпикс отбит за 0,13 с;
|
||||
`RLIMIT_CPU` 1 с — SIGXCPU (`processing_failed`).
|
||||
|
||||
### 4.2. Трек 1: очередь под перегрузкой
|
||||
|
||||
**Вопрос.** Как concurrency и prefetch worker при фиксированном бюджете
|
||||
(2 CPU, 1,5 ГиБ) влияют на пропускную способность, задержку до `ready` и
|
||||
память на изображениях разной тяжести; держит ли предел принятой работы
|
||||
перегрузку и как быстро система возвращается.
|
||||
|
||||
**Методика** (`bench/RUNBOOK.md`, профиль `queue`, параметры целиком — в
|
||||
`meta.json` метки): стенд `vlad-bench` — LB, 2 реплики API (1 CPU / 512
|
||||
МиБ), PostgreSQL 1 / 1 ГиБ, Valkey, HAProxy 0,5 / 256 МиБ,
|
||||
S3 1 / 768 МиБ, RabbitMQ 1 / 512 МиБ,
|
||||
relay 0,25 / 192 МиБ, один worker **2 CPU / 1536 МиБ для всех настроек**,
|
||||
генератор 1 / 512 МиБ, без уборщика; предел `UPLOAD_MAX_UNFINISHED` = 60,
|
||||
`RENDER_MEMORY_MB` = 768. Настройки (concurrency, prefetch): c1p1, c2p2,
|
||||
c4p4, c2p32. Классы PNG: `s` — 320×200 шум (~190 КБ), `l` — 1800×1800 шум
|
||||
(~9,7 МБ), `x` — 6000×6000 однотонный (36 Мпикс, файл ~120 КБ, тяжёлое
|
||||
декодирование). Ячейка — настройка × класс × повтор (2 повтора, порядок
|
||||
перемешан с фиксированным зерном); на повтор — свежий стенд, на ячейку —
|
||||
пересозданный worker. Фазы ячейки:
|
||||
|
||||
- **A «разбор»** — worker остановлен, N загрузок (`s` 50, `l` 40, `x` 20),
|
||||
запуск; пропускная способность T = N / время от готовности потребителя
|
||||
до последнего `ready`;
|
||||
- **B «норма»** — открытая модель 0,6 × T, окно 40 с;
|
||||
- **«fixed»** — одна интенсивность для всех настроек класса (`s` 2,5/с,
|
||||
`l` 0,8/с, `x` 0,5/с — около половины T у c1p1), окно 40 с: сравнение
|
||||
задержки при равной нагрузке;
|
||||
- **C «перегрузка»** — 2 × T, окно `min(150, max(40, 1,5 × 60 / T))` с,
|
||||
чтобы предел достигался на каждом классе;
|
||||
- **D «восстановление»** — время до нуля незавершённых, затем норма 0,6 ×
|
||||
T на 20 с.
|
||||
|
||||
Время до `ready` — `attachment_jobs.updated_at − attachments.created_at`
|
||||
(часы сервера), очередь (queue lag) — `waited` из журнала worker
|
||||
(приём → захват), незавершённые — точный счёт по событиям и выборки раз
|
||||
в секунду, память — `memory.peak` контейнера worker за ячейку и пик RSS
|
||||
процесса рендера.
|
||||
|
||||
**Гипотезы (записаны до замеров).**
|
||||
|
||||
- **Q1.** На тяжёлых классах (`l`, `x`) c2p2 даёт ≥ 1,6 × пропускной
|
||||
способности c1p1, а c4p4 — не больше 1,15 × c2p2: у worker 2 CPU,
|
||||
рендер упирается в CPU.
|
||||
- **Q2.** Prefetch сам по себе пропускную способность не меняет: c2p32 в
|
||||
пределах ±10 % от c2p2 на каждом классе, p95 фазы «fixed» — в пределах
|
||||
±20 %.
|
||||
- **Q3.** Память растёт с concurrency на тяжёлых пикселях: на `x` пик
|
||||
worker при c4p4 ≥ 2,5 × c1p1; на `s` — не больше 1,5 ×.
|
||||
- **Q4.** Под перегрузкой предел держит принятую работу: незавершённых не
|
||||
больше 60 + 2 × 4 (реплики × места загрузки), доля 429 после достижения
|
||||
предела ≥ 30 %, 5xx нет.
|
||||
- **Q5.** После перегрузки очередь разбирается не дольше 1,3 × 60 / T + 5
|
||||
с, p95 нормы после — не больше 1,3 × p95 нормы до + 0,5 с, ни одна
|
||||
принятая загрузка не потеряна и не `failed`.
|
||||
|
||||
**Правило рекомендации** (тоже до замеров): на класс — настройка с
|
||||
наименьшим p95 фазы «fixed» среди тех, у кого пик worker ≤ 80 % лимита
|
||||
(1229 МиБ); при разнице p95 до 10 % — меньший пик.
|
||||
|
||||
Самопроверка на машине разработки (`make bench-queue-quick`, c1p1 и c4p4
|
||||
на `s` и `x`, метка `quick-*`, в отчёт как результат не идёт) показала,
|
||||
что профиль доходит до конца и предел достигается на всех ячейках:
|
||||
пропускная способность `s` 5,6 и 11,2 гот./с, `x` 1,08 и 1,68 гот./с;
|
||||
пик worker на `x` c4p4 — 1244 МиБ (4 процесса рендера по ~300 МиБ).
|
||||
|
||||
**Результаты.** Полная метка [queue10](evidence/queue/queue10/summary.md):
|
||||
24 ячейки (4 настройки × 3 класса × 2 повтора), все 120 фаз завершены.
|
||||
Принятых загрузок — 14185; failed, missing, 5xx и пропусков расписания
|
||||
генератора — 0. Таблицы приведены из `summary.md`;
|
||||
значения — медиана (min–max) двух повторов.
|
||||
Вердикты вычислены по исходной точности `cells.csv`; округление относится
|
||||
к отображению сводки. В CSV также сохранены p50/p95/p99 времени до ready.
|
||||
|
||||
#### Класс `s`
|
||||
|
||||
| Настройка | Пропускная, гот./с | CPU worker, ядер | p95 до ready (0,6 ёмкости), с | queue lag p95, с | p95 fixed, с | Окно перегрузки, с | 429 за окно, % | 429 после предела, % | Retry-After 5 / 1 | Макс. незавершённых | Восстановление, с | p95 после, с | Пик worker, MiB | Пик рендера, MiB | failed / missing | Пометки |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| c1p1 | 7.049 (7.041–7.057) | 0.95 (0.95–0.96) | 0.65 (0.65–0.65) | 0.50 (0.50–0.50) | 0.67 (0.67–0.68) | 40 (40–40) | 40 (40–40) | 50 (50–50) | 225 (224–226) / 0 (0–0) | 60 (60–60) | 8.4 (8.4–8.4) | 0.65 (0.65–0.65) | 100 (100–100) | 97 (96–97) | 0 / 0 | — |
|
||||
| c2p2 | 13.451 (13.404–13.499) | 1.65 (1.58–1.73) | 0.65 (0.65–0.66) | 0.50 (0.50–0.50) | 0.66 (0.65–0.67) | 40 (40–40) | 45 (45–45) | 50 (50–50) | 484 (479–490) / 0 (0–0) | 60 (60–60) | 4.3 (4.3–4.3) | 0.65 (0.65–0.66) | 116 (116–116) | 98 (97–98) | 0 / 0 | — |
|
||||
| c4p4 | 13.418 (13.362–13.473) | 1.79 (1.67–1.91) | 0.77 (0.77–0.77) | 0.50 (0.50–0.51) | 0.68 (0.68–0.68) | 40 (40–40) | 45 (45–46) | 50 (50–51) | 488 (485–491) / 0 (0–0) | 60 (60–61) | 4.3 (4.3–4.4) | 0.77 (0.76–0.77) | 146 (146–146) | 98 (97–98) | 0 / 0 | — |
|
||||
| c2p32 | 13.552 (13.545–13.558) | 1.79 (1.78–1.80) | 0.65 (0.65–0.66) | 0.50 (0.50–0.50) | 0.67 (0.67–0.67) | 40 (40–40) | 45 (45–45) | 50 (50–50) | 492 (491–492) / 0 (0–0) | 60 (60–60) | 4.3 (4.3–4.4) | 0.65 (0.65–0.66) | 116 (116–116) | 98 (97–98) | 0 / 0 | — |
|
||||
|
||||
#### Класс `l`
|
||||
|
||||
| Настройка | Пропускная, гот./с | CPU worker, ядер | p95 до ready (0,6 ёмкости), с | queue lag p95, с | p95 fixed, с | Окно перегрузки, с | 429 за окно, % | 429 после предела, % | Retry-After 5 / 1 | Макс. незавершённых | Восстановление, с | p95 после, с | Пик worker, MiB | Пик рендера, MiB | failed / missing | Пометки |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| c1p1 | 2.303 (2.253–2.353) | 0.84 (0.82–0.86) | 1.14 (0.99–1.29) | 0.59 (0.56–0.61) | 0.99 (0.97–1.00) | 40 (40–40) | 22 (18–25) | 54 (51–58) | 40 (33–48) / 0 (0–0) | 63 (62–64) | 25.2 (25.0–25.4) | 1.16 (1.11–1.21) | 165 (165–165) | 119 (119–119) | 0 / 0 | — |
|
||||
| c2p2 | 3.811 (3.699–3.924) | 1.42 (1.36–1.47) | 1.36 (1.09–1.62) | 0.70 (0.59–0.82) | 0.98 (0.97–0.98) | 40 (40–40) | 40 (39–40) | 53 (51–55) | 121 (119–123) / 0 (0–0) | 63 (62–64) | 16.0 (15.2–16.7) | 1.44 (1.28–1.61) | 246 (246–246) | 139 (137–140) | 0 / 0 | — |
|
||||
| c4p4 | 3.867 (3.732–4.002) | 1.54 (1.49–1.60) | 1.61 (1.59–1.63) | 0.75 (0.65–0.85) | 0.98 (0.98–0.99) | 40 (40–40) | 41 (38–44) | 56 (55–57) | 126 (113–140) / 0 (0–0) | 67 (66–68) | 14.1 (13.3–14.8) | 1.75 (1.67–1.84) | 408 (407–408) | 178 (175–179) | 0 / 0 | — |
|
||||
| c2p32 | 3.722 (3.642–3.802) | 1.38 (1.36–1.41) | 1.60 (1.32–1.88) | 0.91 (0.66–1.15) | 0.98 (0.98–0.98) | 40 (40–40) | 37 (36–38) | 53 (52–54) | 110 (104–116) / 0 (0–0) | 65 (64–66) | 16.9 (15.9–17.9) | 1.54 (1.12–1.95) | 246 (246–247) | 139 (138–140) | 0 / 0 | — |
|
||||
|
||||
#### Класс `x`
|
||||
|
||||
| Настройка | Пропускная, гот./с | CPU worker, ядер | p95 до ready (0,6 ёмкости), с | queue lag p95, с | p95 fixed, с | Окно перегрузки, с | 429 за окно, % | 429 после предела, % | Retry-After 5 / 1 | Макс. незавершённых | Восстановление, с | p95 после, с | Пик worker, MiB | Пик рендера, MiB | failed / missing | Пометки |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| c1p1 | 1.526 (1.526–1.527) | 0.98 (0.97–0.99) | 1.16 (1.14–1.17) | 0.48 (0.47–0.49) | 1.11 (1.08–1.14) | 59 (59–59) | 18 (18–18) | 51 (51–51) | 32 (32–32) / 0 (0–0) | 60 (60–60) | 38.5 (38.3–38.7) | 1.18 (1.18–1.18) | 376 (376–376) | 299 (299–300) | 0 / 0 | — |
|
||||
| c2p2 | 2.907 (2.846–2.968) | 1.95 (1.92–1.97) | 1.16 (1.16–1.17) | 0.49 (0.49–0.50) | 1.14 (1.14–1.14) | 40 (40–40) | 27 (26–28) | 52 (52–53) | 63 (60–66) / 0 (0–0) | 60 (60–60) | 21.2 (21.0–21.4) | 1.17 (1.16–1.18) | 668 (668–669) | 299 (299–300) | 0 / 0 | — |
|
||||
| c4p4 | 2.763 (2.760–2.766) | 1.84 (1.83–1.85) | 1.16 (1.15–1.17) | 0.49 (0.49–0.49) | 1.13 (1.11–1.16) | 40 (40–40) | 25 (24–25) | 50 (50–50) | 54 (54–55) / 0 (0–0) | 60 (60–60) | 20.7 (20.7–20.8) | 1.16 (1.14–1.18) | 1252 (1252–1252) | 299 (299–300) | 0 / 0 | — |
|
||||
| c2p32 | 2.912 (2.895–2.929) | 1.91 (1.85–1.97) | 1.16 (1.16–1.16) | 0.49 (0.49–0.50) | 1.14 (1.13–1.14) | 40 (40–40) | 28 (28–28) | 53 (52–53) | 64 (64–65) / 0 (0–0) | 60 (60–60) | 21.1 (21.1–21.2) | 1.17 (1.16–1.17) | 669 (669–669) | 300 (299–300) | 0 / 0 | — |
|
||||
|
||||
**Итог Q1.** Подтверждена. l: c2p2/c1p1 = 3.811/2.303 = 1.66 (≥ 1.6), c4p4/c2p2 = 3.867/3.811 = 1.01 (≤ 1.15) гот./с; x: c2p2/c1p1 = 2.907/1.526 = 1.90 (≥ 1.6), c4p4/c2p2 = 2.763/2.907 = 0.95 (≤ 1.15) гот./с
|
||||
|
||||
На классе `x` c2p2 использует около двух ядер worker (1,95 CPU):
|
||||
два рендера одновременно почти удваивают пропускную способность.
|
||||
Четыре процесса делят тот же бюджет 2 CPU и дополнительной скорости
|
||||
не дают; на `l` выигрыш c4p4 относительно c2p2 также близок к нулю.
|
||||
|
||||
**Итог Q2.** Подтверждена. s: пропускная 13.552 к 13.451 (+1%, допуск ±10%), p95 0.67 к 0.66 с (+1%, допуск ±20%); l: пропускная 3.722 к 3.811 (-2%, допуск ±10%), p95 0.98 к 0.98 с (+0%, допуск ±20%); x: пропускная 2.912 к 2.907 (+0%, допуск ±10%), p95 1.14 к 1.14 с (-0%, допуск ±20%)
|
||||
|
||||
При одном worker prefetch 32 не добавляет исполняющих процессов:
|
||||
concurrency остаётся 2. Сообщения ожидают внутри потребителя;
|
||||
пропускная способность и задержка fixed практически те же,
|
||||
что при prefetch 2. Для этого стенда prefetch 2 достаточен.
|
||||
|
||||
**Итог Q3.** Подтверждена. x: пик c4p4/c1p1 = 1252/376 MiB = 3.33 (≥ 2.5); s: пик c4p4/c1p1 = 146/100 MiB = 1.47 (≤ 1.5)
|
||||
|
||||
На `x` один процесс рендера достигает примерно 299–300 МиБ RSS.
|
||||
Одновременные процессы поднимают пик контейнера до 1252 МиБ.
|
||||
У малых изображений прирост существенно меньше. Пик контейнера
|
||||
из cgroup включает общие страницы и кэш; сумма RSS процессов
|
||||
не равна этому показателю.
|
||||
|
||||
**Итог Q4.** Подтверждена. макс. незавершённых 68 (граница 60 + 8 = 68); 429 после предела: min 50 % (порог 30%) в 24 ячейках; за всё окно 18%–46%; 5xx 0
|
||||
|
||||
Предел мягкий: до восьми загрузок могут уже пройти проверку
|
||||
незавершённой работы на двух репликах с четырьмя местами каждая.
|
||||
Наблюдаемый максимум 68 точно укладывается в эту границу.
|
||||
Отказы после достижения предела имеют Retry-After 5; приём лишней
|
||||
работы прекращается до чтения тела, поэтому очередь остаётся ограниченной.
|
||||
|
||||
**Итог Q5.** Подтверждена. 24 ячеек; худшее опустошение — 0.70 границы, худший p95 после — 0.72 границы; failed + missing 0
|
||||
|
||||
После прекращения перегрузки worker завершает ограниченный остаток
|
||||
работы; время разбора согласуется с оценкой порядка 60 / T.
|
||||
Задержка нормы после перегрузки возвращается в заданную границу,
|
||||
и все принятые вложения доходят до ready.
|
||||
|
||||
**Компромисс latency/ресурсы и рекомендация.**
|
||||
|
||||
- рекомендация (наименьший p95 фазы fixed при пике worker ≤ 80% лимита, 1229 MiB; p95 в пределах 10% — меньший пик): s — c1p1 (p95 fixed 0.67 с, пропускная 7.049 гот./с, пик 100 MiB; p95 fixed: c2p2 0.66 с, c2p32 0.67 с, c1p1 0.67 с, c4p4 0.68 с); l — c1p1 (p95 fixed 0.99 с, пропускная 2.303 гот./с, пик 165 MiB; p95 fixed: c2p2 0.98 с, c2p32 0.98 с, c4p4 0.98 с, c1p1 0.99 с); x — c1p1 (p95 fixed 1.11 с, пропускная 1.526 гот./с, пик 376 MiB; p95 fixed: c1p1 1.11 с, c2p32 1.14 с, c2p2 1.14 с)
|
||||
|
||||
Это рекомендация при фиксированной невысокой интенсивности класса:
|
||||
разница p95 между настройками мала, поэтому правило выбирает меньший
|
||||
пик памяти c1p1. Если нужен запас пропускной способности при росте
|
||||
очереди, c2p2 даёт разумный компромисс: на тяжёлых классах около
|
||||
1,66–1,90 скорости c1p1 при меньшем пике памяти, чем c4p4.
|
||||
Для `x` c4p4 достигает 1252 МиБ, выше порога рекомендации 1229 МиБ,
|
||||
при этом пропускная способность ниже c2p2. Большой prefetch
|
||||
не компенсирует ограничение CPU.
|
||||
|
||||
### 4.3. Железо и стенд замеров
|
||||
|
||||
Apple M4, 10 CPU, 16 ГиБ RAM, macOS 26.6.2 arm64. ВМ Docker Desktop —
|
||||
10 CPU, 8218034176 байт RAM (7,65 ГиБ), Linux 6.10.14-linuxkit,
|
||||
cgroup v2 / cgroupfs. Docker 28.0.4, Compose 5.6.1.
|
||||
Питание от сети; посторонних контейнеров нет, предупреждений о хосте нет,
|
||||
swapouts_delta = 0.
|
||||
|
||||
Команда: `make bench-queue QUEUE_LABEL=queue10`.
|
||||
Начало — 2026-11-05T21:01:09Z, конец — 2026-11-05T22:13:53Z (UTC),
|
||||
длительность 72 мин 44 с. Методика прогрева, интенсивности и окна —
|
||||
раздел 4.2; точные параметры — [meta.json](evidence/queue/queue10/meta.json).
|
||||
Контейнеры без swap, порты стенда на хост не публикуются.
|
||||
|
||||
| Компонент | CPU | RAM, МиБ |
|
||||
|---|---:|---:|
|
||||
| API × 2 | по 1 | по 512 |
|
||||
| PostgreSQL | 1 | 1024 |
|
||||
| Valkey | 0,5 | 128 |
|
||||
| HAProxy | 0,5 | 256 |
|
||||
| S3 | 1 | 768 |
|
||||
| RabbitMQ | 1 | 512 |
|
||||
| relay | 0,25 | 192 |
|
||||
| worker | 2 | 1536 |
|
||||
| генератор | 1 | 512 |
|
||||
| сэмплер | 0,25 | 256 |
|
||||
|
||||
Сумма лимитов — 9,5 CPU и 6,0625 ГиБ; резерв проверки памяти — 1,5 ГиБ.
|
||||
В каждой ячейке worker имеет один и тот же бюджет, меняются только
|
||||
concurrency и prefetch.
|
||||
|
||||
| Образ | Версия и digest |
|
||||
|---|---|
|
||||
| Python | `python:3.13.15-slim-trixie@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b` |
|
||||
| postgres | `postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722` |
|
||||
| valkey | `valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd` |
|
||||
| haproxy | `haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26` |
|
||||
| s3 | `chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d` |
|
||||
| rabbitmq | `rabbitmq:4.3.7-alpine@sha256:b1f997f74103d08fc1f20a13890dc44fe6cfa51d528cf521b480ada988d3ffa5` |
|
||||
|
||||
Образ приложения — `mtusi-messenger-vladimirceva:bench-queue10`,
|
||||
ID `sha256:eec175e93886926e10260e6cdf6464e6e21b4111fd44d5f62c798e8e4158f4d4`.
|
||||
Python 3.13.15; версии библиотек и настройки PostgreSQL —
|
||||
в паспорте метки. Проверки подготовки: 140 passed в `make test-isolated`,
|
||||
34 passed в `make failure-checks FAULT_ARGS=-rP`;
|
||||
вывод — [measurement-host.txt](evidence/tests/measurement-host.txt).
|
||||
|
||||
## 5. Ограничения
|
||||
|
||||
- Один Docker-host, один брокер и одна PostgreSQL: отказ хоста, LB,
|
||||
PostgreSQL или S3 стенд не переживает; брокер и worker — переживает.
|
||||
- At least once + идемпотентный эффект — не exactly once: дубли доставки
|
||||
возможны (relay после confirm, ack после commit), их отбрасывает
|
||||
строка задания.
|
||||
- Пульса аренды нет: задание дольше аренды будет выполнено повторно
|
||||
(результат один — fencing). Аренда выбрана заведомо больше срока
|
||||
рендера.
|
||||
- Структурная проверка API пропускает JPEG без таблиц квантования — он
|
||||
становится `failed` в worker, а не 415 (`KNOWN_ISSUES.md`).
|
||||
- Сверка с лабой 4 обнаружила неперенесённое исправление `a2047b5`:
|
||||
текущая структурная проверка пропускает JPEG с пустым сканом.
|
||||
Пропущенные код и четыре теста перечислены в `KNOWN_ISSUES.md`;
|
||||
при подготовке замеров они не переносились.
|
||||
- Эвристика «потерянных» сообщений не видит выданные worker, но не
|
||||
взятые (`KNOWN_ISSUES.md`).
|
||||
- Pipeline версии 1; несколько версий — трек 2, не выбран.
|
||||
- Замер выполнен на одном Apple M4, по два повтора на ячейку:
|
||||
перенос абсолютных скоростей на другое оборудование требует замера.
|
||||
- Для полной серии используются S3 768 МиБ и worker 1536 МиБ.
|
||||
Пик c4p4 на `x` превышает 80 % памяти worker; запаса для этой
|
||||
настройки мало.
|
||||
- Прерванная метка `baseline` не участвует в сравнительных таблицах.
|
||||
Её отказ SeaweedFS по памяти подтверждён в
|
||||
[s3-load.txt](evidence/stand/s3-load.txt); полный результат — `queue10`.
|
||||
|
||||
## 6. Ответы на вопросы защиты
|
||||
|
||||
- **Publisher confirm и consumer ack.** Confirm — брокер принял и записал
|
||||
сообщение (relay отмечает задание опубликованным только после него).
|
||||
Ack — потребитель закончил: worker подтверждает после того, как исход
|
||||
записан в PostgreSQL. Первое — граница «relay → брокер», второе —
|
||||
«брокер → результат».
|
||||
- **Что сохранено в момент 202.** Полный проверенный оригинал в staging
|
||||
S3, строка вложения `queued` с размером и SHA-256 и строка задания
|
||||
(outbox) — одной транзакцией после записи staging.
|
||||
- **Почему повторная доставка нормальна и где ключ идемпотентности.**
|
||||
Доставка at least once: сообщение повторится после падения до ack или
|
||||
повторной публикации. Ключ — строка задания (`job_id`, номер попытки,
|
||||
состояние) для решения «делать или нет» и ключ эффекта
|
||||
`(attachment_id, pipeline_version, variant)` для результата.
|
||||
- **Зависший и долгий worker.** Аренда больше срока рендера: долгий
|
||||
успевает. Зависший теряет аренду, задание возвращает relay; поздний
|
||||
результат прежней попытки не пройдёт fencing, а объекты он пишет в те
|
||||
же ключи тем же содержимым.
|
||||
|
||||
## 7. Что ещё проверить на защите
|
||||
|
||||
`make pipeline-state` во время перегрузки; `make replay` на рабочем
|
||||
стенде; `make s3-check` (права пользователя worker); `make
|
||||
test-persistence ACTION_SCRIPT=scripts/restart.sh` с брокером.
|
||||
|
||||
## 8. Источники и помощь ИИ
|
||||
|
||||
Документация RabbitMQ (reliability, confirms, quorum queues), aio-pika,
|
||||
Pillow (`draft`, `exif_transpose`, `DecompressionBombError`), PostgreSQL
|
||||
(`SKIP LOCKED`, блокировки строк). Помощь ИИ: Claude (Anthropic) — код,
|
||||
тесты, стенд и документация по моим требованиям и под мою проверку;
|
||||
Codex — замеры трека 1 по `bench/RUNBOOK.md` и итоги в этом отчёте.
|
||||
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Миграции alembic; правила — migrations/AGENTS.md.
|
||||
# URL подключения migrations/env.py собирает из тех же переменных DB_*,
|
||||
# что и приложение (src/config.py), поэтому sqlalchemy.url здесь нет.
|
||||
# Секций logging тоже нет: журнал идёт текстовым форматом src/log.py.
|
||||
|
||||
[alembic]
|
||||
script_location = %(here)s/migrations
|
||||
# имя ревизии: <номер>_<slug>.py, номер задаётся при создании (--rev-id)
|
||||
file_template = %%(rev)s_%%(slug)s
|
||||
truncate_slug_length = 40
|
||||
# импорт src из корня репозитория и из /app в образе
|
||||
prepend_sys_path = .
|
||||
path_separator = os
|
||||
+254
@@ -0,0 +1,254 @@
|
||||
# bench/ — нагрузочный стенд лаб 3–5
|
||||
|
||||
В лабе 5 стенд даёт результат трека 1 «Очередь под перегрузкой»
|
||||
(`README.md`): ≥ 3 настройки concurrency/prefetch на разных размерах
|
||||
изображений, throughput, queue lag, p95 до `ready`, RSS, backpressure и
|
||||
предел принятой работы, восстановление после перегрузки — профиль
|
||||
`queue`.
|
||||
|
||||
Профили лабы 4 (`uploads`, `reconcile`) и лабы 3 (`failover`, `rollout`,
|
||||
серии `scaling`) сохранены и работают на коде лабы 5 (загрузка теперь
|
||||
202), но в лабе 5 не обязательны: их метки и числа остались в
|
||||
репозиториях прошлых лаб.
|
||||
|
||||
Порядок работы по шагам — `bench/RUNBOOK.md`. Здесь — правила, которые
|
||||
действуют на каждом шаге.
|
||||
|
||||
## Что есть
|
||||
|
||||
| Файл | Команда | Что делает |
|
||||
|---|---|---|
|
||||
| `run.py` | `make bench-queue`, `bench-queue-quick`; лаба 4 — `bench-uploads`, `bench-reconcile`, `*-quick`; лаба 3 — `bench-failover`, `bench-rollout`, `bench-scaling` | оркестратор: стенд, профили, файлы метки; у `queue` — поток шкалы (SELECT состояния заданий раз в секунду) |
|
||||
| `common.py` | — | docker, стенд compose `vlad-bench` (с лабы 4 — S3, bootstrap, уборщик; с лабы 5 — брокер, relay, worker), паспорт, хост, бюджет, сэмплер `peak`, чтение файла cgroup через сэмплер |
|
||||
| `queue.py` | `uv run python bench/queue.py --selftest` | профиль `queue` без docker: план ячеек кругами, разбор журнала worker, журнала генератора и строк SQL, строки `cells.csv`, `timeline.csv`, `jobs.csv.gz`, вердикты Q1–Q5, рекомендация, сводка. Имя заслоняет stdlib `queue` внутри bench/: модули стенда его не импортируют |
|
||||
| `uploads.py` | `uv run python bench/uploads.py --selftest` | профиль `uploads` без docker: план кругами, разбор сэмплера `peak`, строка `runs.csv`, сводка, вердикты U1–U4 |
|
||||
| `reconcile.py` | `uv run python bench/reconcile.py --selftest` | профиль `reconcile` без docker: модель набора (id, содержимое и класс по номеру и зерну), строки `COPY`, план, разбор итога уборщика, сверка с моделью, окна фоновой нагрузки, вердикты H1–H5, сводка |
|
||||
| `fill_objects.py` | — (контейнер `vlad-bench-fill`) | PUT объектов набора мимо API: botocore образа стенда, ключи пользователя API из окружения контейнера |
|
||||
| `analysis.py` | `uv run python bench/analysis.py --selftest` | разбор без docker: журналы запросов, события Docker, журнал HAProxy, агрегаты сэмплера, признаки узкого места, поиск R* |
|
||||
| `loadgen.py` | — (`docker exec` в `vlad-bench-gen`) | генератор на stdlib: `setup`, `step`, `constant` (с лабы 4 — скачивания вложений и `--stop-file`), `retry`, `body` (PNG из шума; с лабы 5 — `--width`/`--height` и одноцветный `--flat`), `uploads` (закрытая модель загрузок, принято — 201 или 202), `queue` (лаба 5: открытая модель загрузок без повторов, `--count`, журнал с id принятых) |
|
||||
| `sampler.py` | — (контейнер `vlad-bench-sampler`) | `list`/`scan` — CPU и память cgroup раз в интервал (`scan` — и у `queue`: worker пересоздаётся на ячейку); `peak` (лаба 4) — сброс `memory.peak` своим дескриптором, выборки `current`/`anon`/`sock`/`oom_kill` раз в 0,2 с |
|
||||
| `compare.py`, `plots.py` | `make bench-compare`, `make bench-plots` | лаба 3: «до/после» серий и графики SVG |
|
||||
| `bench.compose.yml`, `bench.env` | — | override стенда: имена, лимиты `BENCH_*`, без портов и swap; S3 и уборщик с лимитами рабочего стенда; брокер (`BENCH_MQ_*`), relay (0,25 / 192 МБ), worker (`BENCH_WORKER_*`: лимиты, concurrency, prefetch, `RENDER_MEMORY_MB`); `UPLOAD_MAX_UNFINISHED` реплик — `BENCH_UPLOAD_MAX_UNFINISHED`, по умолчанию 0 (без предела) |
|
||||
|
||||
Профили (поля — `QueueProfile`, `UploadsProfile`, `ReconcileProfile` в
|
||||
`run.py`, целиком в `meta.json`):
|
||||
|
||||
| Профиль | Что | Время |
|
||||
|---|---|---|
|
||||
| `queue` | LB, две реплики 1 CPU / 512 МБ, PostgreSQL, Valkey, S3, брокер 1 / 512 МБ, relay, один worker 2 CPU / 1536 МБ на все настройки, генератор 1 / 512 МБ; `UPLOAD_MAX_UNFINISHED=60`. Настройки c1p1, c2p2, c4p4, c2p32 (concurrency/prefetch) × классы `s` (шум 320 × 200), `l` (шум 1800 × 1800, ~9,7 МБ), `x` (один цвет 6000 × 6000) × 2 повтора, ячейки кругами; стенд новый на повтор, worker — на ячейку. Фазы ячейки: ёмкость (worker стоит, N = 50 / 40 / 20 тел — меньше предела, старт worker, ready/с), ровная 0,6 ёмкости 40 с, `fixed` — общая на класс интенсивность (s 2,5, l 0,8, x 0,5 загрузки/с) 40 с, перегрузка 2 ёмкости в окне `min(150 с, max(40 с, 1,5 × 60 / ёмкость))`, восстановление (до пустой очереди, затем 0,6 ёмкости 20 с) | ~1,5 ч (оценка по `queue-quick`: ~3–4,5 мин на ячейку, дольше всех `x` c1p1) |
|
||||
| `queue-quick` | c1p1 и c4p4 × `s` и `x`, один повтор, окно 15 с (с `l` вышло бы ~14 мин) | ~9 мин |
|
||||
| `uploads` | LB, одна реплика 1 CPU / 512 МБ без swap, S3 1 CPU / 512 МБ; варианты «с лимитом» (места по умолчанию: 4 + очередь 16) и «без лимита» (`UPLOAD_MAX_CONCURRENT=1000`, `UPLOAD_MAX_WAITING=0`) × тела ~1 и ~10 MiB × 1/10/20/40 клиентов × 3 повтора = 48 прогонов | ~45 мин |
|
||||
| `uploads-quick` | оба варианта, 10 MiB, 1/10/40 клиентов, один повтор, окно 10 с | ~5 мин |
|
||||
| `reconcile` | LB, две реплики, S3; этапы: A — 1 000 × 1 MiB (`batch`, `deep`, `deep-fast`); B — 10 000 × 4 KiB (`batch`, `head`, `deep`); C — тот же стенд +20 000 (`batch`, `head`); D — +70 000 (`batch` и четыре ячейки под фоновой нагрузкой); E — свежий, 10 000 согласованных + 1 000 строк без объекта (`batch`, `refuse`); 3 повтора кругами | ~45 мин |
|
||||
| `reconcile-quick` | 2 000 × 4 KiB (`batch`, `head`, `deep`, `load-stand`, `load-fast`) и 500 + 100 (`batch`, `refuse`), один повтор | ~6 мин |
|
||||
|
||||
Ячейки сверки (`reconcile.CELLS`): `batch` — выборка строк на страницу
|
||||
списка; `head` — выборка на объект и HEAD на строку; `deep` — SHA-256
|
||||
содержимого; `-fast` — уборщик 1 CPU вместо 0,25; `load-*` — проход под
|
||||
фоновой нагрузкой (скачивания 200/с, запись 50/с, чтение 50/с через LB):
|
||||
`load-stand` — предел скорости по умолчанию (`RECONCILE_RATE`, 2000
|
||||
объектов/с) и 0,25 CPU, `load-unlimited` — без предела, `load-fast` —
|
||||
без предела и 1 CPU, `load-fast-limited` — с пределом и 1 CPU; `refuse` —
|
||||
`--apply --max-changes 100` (быстрый — 10) при большем числе кандидатов.
|
||||
Все прочие ячейки — dry-run.
|
||||
|
||||
ВМ профиля `queue` — 10 CPU / 7,65 ГиБ. Сумма лимитов — 9,5 CPU /
|
||||
6,0625 ГиБ: PostgreSQL 1 CPU / 1 ГиБ, HAProxy 0,5 CPU / 256 МиБ,
|
||||
S3 1 CPU / 768 МиБ,
|
||||
worker 2 CPU / 1536 МиБ, две реплики по 1 CPU / 512 МиБ; остальные
|
||||
лимиты — из профиля. Запас проверки памяти — 1,5 ГиБ.
|
||||
|
||||
Стенд — свой проект compose `vlad-bench`: тот же `compose.yaml` с
|
||||
override `bench.compose.yml` и подстановками `bench.env`, а не `.env`
|
||||
рабочего стенда. Контейнеры: `vlad-bench-postgres`, `vlad-bench-valkey`,
|
||||
`vlad-bench-s3`, `vlad-bench-lb`, реплики `vlad-bench-api-N`, генератор
|
||||
`vlad-bench-gen`, сэмплер `vlad-bench-sampler`, загрузчик
|
||||
`vlad-bench-fill`, разовые проходы `vlad-bench-reconcile-<n>`; с лабы 5 —
|
||||
`vlad-bench-mq`, `vlad-bench-relay`, `vlad-bench-worker`; сеть
|
||||
`vlad-bench-net`, volume `vlad-bench_pgdata`, `vlad-bench_s3data`,
|
||||
`vlad-bench_rmqdata`. Генератор ходит только через LB
|
||||
(`http://haproxy:8080`).
|
||||
|
||||
Метрики профиля `queue` (часы ВМ Docker у всех источников):
|
||||
|
||||
- пропускная — ready/с фазы ёмкости: N тел ждут в брокере, отсчёт от
|
||||
строки `broker consuming` после старта worker до последнего `ready`
|
||||
(старт процесса — отдельно, `startup_s`);
|
||||
- время до ready — `attachment_jobs.updated_at − attachments.created_at`
|
||||
(SQL); queue lag — `waited` строки `attachment processed` журнала
|
||||
worker (приём → захват);
|
||||
- незавершённых — точный счёт по созданию и завершению заданий ячейки;
|
||||
шкала раз в секунду (`unfinished`, outbox, в брокере, `processing`,
|
||||
возраст старейшего) — для графика;
|
||||
- p95 фазы `fixed` — та же метрика при одной интенсивности на класс для
|
||||
всех настроек: по ней Q2 (задержка) и рекомендация; p95 фазы `steady`
|
||||
(0,6 своей ёмкости) — для Q5;
|
||||
- окно перегрузки — `min(150 с, max(окно, 1,5 × предел / ёмкость))`,
|
||||
факт — `window_s` строки `overload`;
|
||||
- 429 — доля за всё окно перегрузки и после первого отказа с
|
||||
`Retry-After: 5` (предел работы достигнут); Q4 проверяет вторую;
|
||||
- восстановление — от конца окна перегрузки до последнего `ready`;
|
||||
- пик worker — `memory.peak` cgroup контейнера в конце ячейки через
|
||||
сэмплер (контейнер новый на ячейку, перезапуск в фазе ёмкости заводит
|
||||
cgroup заново); пик рендера — наибольший `render_peak_kib`;
|
||||
- рекомендация на класс — наименьший p95 фазы `fixed` среди настроек с
|
||||
пиком worker ≤ 80 % лимита; p95 в пределах 10 % от лучшего равны, из
|
||||
них — меньший пик.
|
||||
|
||||
Файлы меток:
|
||||
|
||||
- `evidence/queue/<метка>/`: `summary.md` (таблица на класс: настройки ×
|
||||
пропускная, p95 до ready, lag, доли 429, максимум незавершённых,
|
||||
восстановление, пики; вердикты Q1–Q5 и рекомендация), `cells.csv`
|
||||
(строка на ячейку × фазу), `timeline.csv` (раз в секунду: состояние
|
||||
заданий, CPU и память worker, ответы генератора), `jobs.csv.gz`
|
||||
(задание: фаза, приём, ready, попытка, `waited`, `took`), `meta.json`
|
||||
(профиль, план, тела с SHA-256, паспорт, `app_env` worker по
|
||||
настройкам, отметки фаз ячеек), `raw/` (итоги и журналы генератора по
|
||||
фазам, журнал worker ячейки, строки сэмплера повтора `.jsonl.gz`);
|
||||
|
||||
- `evidence/uploads/<метка>/`: `summary.md` (ячейки — медиана и min–max
|
||||
повторов, вердикты U1–U4), `runs.csv` (строка на прогон: память до
|
||||
окна, пик окна, пик старта, `anon`, `sock`, OOM, ответы по кодам,
|
||||
загрузок/с, p50/p95/p99, пики S3 и LB, пометки), `meta.json` (профиль,
|
||||
план, тела с SHA-256, паспорт стенда, `app_env` по вариантам), `raw/`
|
||||
(`<прогон>-gen.json` — итог генератора, `<прогон>-peak.jsonl` — строки
|
||||
сэмплера);
|
||||
- `evidence/reconcile/<метка>/`: `summary.md` (проходы, фоновая нагрузка,
|
||||
вердикты H1–H5), `passes.csv` (строка на проход: найдено по классам,
|
||||
стоимость — вызовы S3 по операциям, ключи списка, прочитанные байты,
|
||||
запросы к БД, длительность шагов, CPU и пик RSS уборщика, CPU S3,
|
||||
PostgreSQL и реплик за окно прохода), `windows.csv` (окно × вид
|
||||
запроса фоновой нагрузки), `meta.json` (план, модели наборов,
|
||||
наполнение), `raw/` (журнал каждого прохода, журнал запросов фоновой
|
||||
нагрузки `.csv.gz`).
|
||||
|
||||
Хеш кода `src_sha256` (по `src/` и `migrations/` **внутри образа**),
|
||||
версии пакетов и значения по умолчанию (`app_defaults`: пул, мест
|
||||
загрузок, сверки, `MALLOC_MMAP_THRESHOLD_`) — в паспорте метки. Образ
|
||||
метки `mtusi-messenger-vladimirceva:bench-<метка>` остаётся в docker;
|
||||
образы `quick-*` стенд удаляет.
|
||||
|
||||
## Методика не меняется между метками
|
||||
|
||||
Не меняются (поменял — новая методика, метки переснимаются, в
|
||||
`REPORT.md` — что и почему):
|
||||
|
||||
- **машина и сессия**: метки одного профиля — на одной машине, рабочий
|
||||
стенд остановлен (`make stop`), посторонних контейнеров нет;
|
||||
- **профиль** (`run.py`): лимиты (реплика 1 CPU / 512 МиБ без swap,
|
||||
PostgreSQL 2 / 1 ГиБ (у `queue` 1 / 1 ГиБ), Valkey 0,5 / 128 МиБ,
|
||||
HAProxy 1 / 256 МиБ (у `queue` 0,5 / 256 МиБ), S3 1 /
|
||||
512 МиБ (у `queue` 1 / 768 МиБ), уборщик 0,25 / 256 МиБ, генератор 2 / 1 ГиБ — у `queue` 1 /
|
||||
512 МиБ, сэмплер 0,25 / 256 МиБ, брокер 1 / 512 МиБ, relay 0,25 / 192
|
||||
МиБ, worker 2 / 1536 МиБ), тела и их зерно, клиенты, окна, наборы,
|
||||
классы и их доли (0,25 % на класс), этапы и ячейки, фоновая нагрузка,
|
||||
настройки worker, доли ёмкости, интенсивности `fixed`, правило окна
|
||||
перегрузки, предел незавершённых, повторы, зерно
|
||||
порядка. Новый параметр — поле профиля, а не константа в коде;
|
||||
- **хранилища, S3 и LB**: те же пины образов и аргументы, что у рабочего
|
||||
стенда (`compose.yaml`), тот же `haproxy.cfg`, `valkey.conf`,
|
||||
`infra/s3/bootstrap.sh`; bucket `bench` (`bench.env`);
|
||||
- **приложение**: значения по умолчанию `src/config.py`; окружение
|
||||
стенда — срок сессии, `UPLOAD_MAX_UNFINISHED` (0, у `queue` — поле
|
||||
профиля), у варианта «без лимита» — два `UPLOAD_*` (override
|
||||
`variant-unlimited.compose.yml` во временном каталоге), у проходов
|
||||
сверки — `CLEANER_GRACE=30`, у worker профиля `queue` —
|
||||
`WORKER_CONCURRENCY`, `WORKER_PREFETCH` ячейки и `RENDER_MEMORY_MB`;
|
||||
- **пороги вердиктов** (`U*` в `uploads.py`, `H*_` в `reconcile.py`,
|
||||
`Q*_` в `queue.py`) записаны до замеров вместе с гипотезами в
|
||||
`REPORT.md`. Их изменение — смена методики, а не подгонка вывода.
|
||||
|
||||
## Честность результатов
|
||||
|
||||
- В `evidence/queue/<метка>/`, `evidence/uploads/<метка>/` и
|
||||
`evidence/reconcile/<метка>/` пишет только `run.py`. Руками эти файлы
|
||||
не править и не пересобирать.
|
||||
- Каталог метки не перезаписывается. Неудачная метка (`aborted`,
|
||||
предупреждения о хосте, пометки) не удаляется молча и не выдаётся за
|
||||
результат: сними новую метку и скажи пользователю, почему старая не
|
||||
годится. Удалять каталог метки — только с согласия пользователя.
|
||||
- Числа в `REPORT.md` переносятся из `summary.md` и `*.csv` как есть.
|
||||
- Гипотезы Q1–Q5 (U1–U4 и H1–H5 прошлой лабы) записываются в
|
||||
`REPORT.md` **до** замеров и после них не переписываются; итог —
|
||||
отдельным абзацем ниже. Опровергнутая гипотеза — тоже результат.
|
||||
- Проект не под git: измеренный код называет `src_sha256` в паспорте.
|
||||
|
||||
## Как устроен стенд (не ломать)
|
||||
|
||||
- Чёрный ящик: `run.py` не импортирует `src`; запросы к API — только
|
||||
генератор по HTTP через LB.
|
||||
- Исключения, только на стенде `vlad-bench`:
|
||||
- PostgreSQL стенда — `docker exec … psql` от пользователя контейнера,
|
||||
SQL через stdin: `VACUUM (ANALYZE)`, `CHECKPOINT`, число строк,
|
||||
ревизия; у профиля `reconcile` — ещё `COPY attachments FROM STDIN`
|
||||
(метаданные набора; схему не трогает, файла CSV нет — строки
|
||||
генерирует `reconcile.rows_csv`);
|
||||
- объекты набора `reconcile` — `fill_objects.py` в `vlad-bench-fill`
|
||||
(ключи API — `docker run -e <имя>` без значения в argv);
|
||||
- S3 профиля `uploads` — перед каждым прогоном удаляется вместе с
|
||||
volume и поднимается заново с bootstrap; к bucket оркестратор не
|
||||
обращается;
|
||||
- сэмплер `peak` — `--user 0`, cgroupfs на запись, без сети: запись
|
||||
`reset` в `memory.peak` своим дескриптором — управление счётчиком
|
||||
ядра, а не приложением;
|
||||
- проход сверки — процесс уборщика из образа (`docker compose run
|
||||
cleaner reconcile …`), результат — его итоговая строка `reconcile
|
||||
pass finished` (`raw/<проход>-cleaner.log`);
|
||||
- профиль `queue`: SELECT состояния `attachment_jobs` и вложений
|
||||
ячейки (без записи); worker пересоздаётся на ячейку (`docker compose
|
||||
up --no-deps --force-recreate worker` с `BENCH_WORKER_*`), в фазе
|
||||
ёмкости — `docker stop`/`docker start`; журнал worker (`docker
|
||||
logs`) — источник `waited`, `took`, `render_peak_kib` и начала
|
||||
потребления; `memory.peak` читает сэмплер `scan` (cgroupfs хоста).
|
||||
- Найденное каждым проходом сверяется с моделью набора
|
||||
(`reconcile.found_problems`): расхождение — отказ профиля с проходом,
|
||||
классом и обоими числами. Это проверка корректности сверки, а не
|
||||
пометка.
|
||||
- Все вызовы compose — с `-p vlad-bench`, обоими `-f` (у варианта —
|
||||
третьим) и `--env-file bench/bench.env`. Пароли хранилищ и брокера и
|
||||
девять секретов S3 (с лабы 5 — и ключи worker) — новые на прогон,
|
||||
никуда не пишутся; compose требует их и у профилей, которые S3 и
|
||||
брокер не запускают.
|
||||
- Пароли и токены пользователей стенда живут только в генераторе; в
|
||||
stdout, результаты и паспорт не попадают. В паспорт окружения реплик не
|
||||
попадают имена с `PASSWORD`, `SECRET`, `ACCESS_KEY`.
|
||||
- Все отметки времени — часы ВМ Docker: генератор, сэмплер, журнал
|
||||
уборщика (`src/log.py`, UTC с мс).
|
||||
- Только стандартная библиотека Python на хосте и в генераторе.
|
||||
- Один прогон за раз: блокировка `bench/.bench.lock`. Остатки
|
||||
прерванного прогона `run.py` не удаляет сам: печатает команды уборки.
|
||||
- Чужие контейнеры не останавливать и не удалять.
|
||||
- Столбцы `cells.csv`, `timeline.csv`, `jobs.csv.gz`, `runs.csv`,
|
||||
`passes.csv`, `windows.csv` не переименовывать и не удалять. Новая
|
||||
метрика — новый столбец в конце.
|
||||
|
||||
## Пометки и что с ними делать
|
||||
|
||||
| Где | Пометка | Значит | Действие |
|
||||
|---|---|---|---|
|
||||
| `meta.json` | `aborted` | прогон прерван | новая метка после разбора `app-tail.log` |
|
||||
| `meta.json` | `host_warnings` | батарея, давление на память, посторонние контейнеры, выгрузка страниц, сон | устранить и переснять |
|
||||
| `runs.csv` | `oom` | реплику завершило ядро по памяти | это результат варианта (ожидаем у «без лимита»), не сбой |
|
||||
| `runs.csv` | `peak_sampled` | `memory.peak` не сброшен (ядро < 6.12) | пик — по выборкам раз в 0,2 с: сказать в отчёте |
|
||||
| `runs.csv` | `host_swap` | хост выгружал страницы во время прогона | переснять метку |
|
||||
| `passes.csv` | `cpu_limit` | лимит CPU прохода не тот, что у ячейки | дефект стенда: разобраться, не переносить в отчёт |
|
||||
| `passes.csv` | `exit` | код уборщика не 3 | набор с расхождениями даёт 3: смотреть `raw/<проход>-cleaner.log` |
|
||||
| `passes.csv` | `bg_dropped` | генератор фоновой нагрузки не отправил часть запросов по расписанию | H4 по такой метке не подтверждается: сказать пользователю, переснять на тихом хосте |
|
||||
| профиль падает | `found differs from the model` | сверка нашла не то, что внесено | дефект сверки или стенда: остановиться, сказать пользователю |
|
||||
| `cells.csv` | `drain_rejected` | не все N тел фазы ёмкости получили 202 | пропускная по принятым; повторилось — разобрать `raw/<ячейка>-drain-gen.csv` |
|
||||
| `cells.csv` | `failed`, `missing` | принятая загрузка стала `failed` или не дошла до `ready` | Q5 опровергнута по ячейке: смотреть `raw/<ячейка>-worker.log` (OOM рендера, `error_code`) |
|
||||
| `cells.csv` | `5xx` | ответы 5xx в фазе | Q4 опровергнута: найти причину (`raw/<ячейка>-<фаза>-gen.csv`, X-Instance-Id), прежде чем переносить в отчёт |
|
||||
| `cells.csv` | `gen_dropped` | генератор не отправил часть загрузок по расписанию | интенсивность фазы не та: переснять на тихом хосте |
|
||||
|
||||
## Проверка правок стенда
|
||||
|
||||
`make fmt && make lint`; `uv run python bench/queue.py --selftest`,
|
||||
`uv run python bench/analysis.py --selftest`, `uv run python
|
||||
bench/uploads.py --selftest`, `uv run python bench/reconcile.py
|
||||
--selftest`; затем `make bench-queue-quick` (лаба 5), при правке общего —
|
||||
ещё `make bench-uploads-quick` и `make bench-reconcile-quick`: прогоны
|
||||
доходят до конца, `aborted` нет.
|
||||
Правка общего (`common.py`, `loadgen.py`, `sampler.py`) — ещё
|
||||
`make bench-failover-quick`. Docker недоступен из sandbox — попросить
|
||||
пользователя запустить команду и прислать вывод.
|
||||
@@ -0,0 +1,176 @@
|
||||
# Лаба 5: замеры трека 1 — порядок работы
|
||||
|
||||
Для того, кто снимает замеры на машине прогонов, — человека или Codex.
|
||||
Правила стенда — `bench/AGENTS.md`, общие — корневой `AGENTS.md`.
|
||||
Прочитать целиком до первого `make bench-*`.
|
||||
|
||||
**Цель — «5»** (`README.md`): уровни 3 и 4 и законченный трек 1 «Очередь
|
||||
под перегрузкой». Код, тесты, стенд и все проверки без нагрузки готовы и
|
||||
сняты на машине разработки (`evidence/`, раздел 3 `REPORT.md`). Здесь
|
||||
остаётся один профиль под нагрузкой, его итоги и отчёт. **Код приложения
|
||||
и профиль не меняются**: замер проверяет записанные гипотезы Q1–Q5.
|
||||
|
||||
**Что требует трек.** «Сравните ≥ 3 настройки concurrency/prefetch на
|
||||
разных размерах изображений. Измерьте throughput, queue lag, p95 времени
|
||||
до ready и RSS; покажите backpressure и ограничение количества принятой
|
||||
работы. Докажите восстановление после перегрузки, объясните компромисс
|
||||
latency/ресурсы». И из `COURSE.md`: версии и образы, железо, лимиты,
|
||||
команда, прогрев, длительность, интенсивность, ошибки, p50/p95/p99,
|
||||
CPU/RAM — всё это пишет стенд в `meta.json` и `summary.md`.
|
||||
|
||||
## Словарь
|
||||
|
||||
- **Метка** — один прогон профиля; каталог `evidence/queue/<метка>/` не
|
||||
перезаписывается.
|
||||
- **Настройка** — `c<concurrency>p<prefetch>` worker: `c1p1`, `c2p2`,
|
||||
`c4p4`, `c2p32`. Ресурсы worker одинаковы у всех (2 CPU / 1536 МиБ).
|
||||
- **Класс** — изображение: `s` (PNG 320×200 из шума, ~190 КБ), `l` (PNG
|
||||
1800×1800 из шума, ~9,7 МБ), `x` (однотонный PNG 6000×6000, 36 Мпикс,
|
||||
маленький файл, но тяжёлое декодирование).
|
||||
- **Ячейка** — настройка × класс × повтор; фазы: A «разбор» (worker стоит,
|
||||
N загрузок, запуск — пропускная способность T), B «норма» (0,6 × T),
|
||||
«fixed» (одна интенсивность на класс: `s` 2,5/с, `l` 0,8/с, `x` 0,5/с),
|
||||
C «перегрузка» (2 × T, окно до 150 с — пока предел не достигнут), D
|
||||
«восстановление» (до нуля незавершённых, затем короткая норма).
|
||||
- **Время до ready** — `attachment_jobs.updated_at − attachments.created_at`
|
||||
(часы сервера); **очередь** (`waited`) — от приёма до захвата worker
|
||||
(журнал worker); **предел** — `UPLOAD_MAX_UNFINISHED` = 60.
|
||||
|
||||
## Сколько это занимает
|
||||
|
||||
| Шаг | Что | Время |
|
||||
|---|---|---|
|
||||
| 0 | подготовка, проверки (`failure-checks` ~8 мин), самопроверка (~9 мин) | 40–50 мин |
|
||||
| 1 | `make bench-queue`: 24 ячейки по 3–4,5 мин | ~1,5 ч + 30 мин отчёт |
|
||||
| 2 | отчёт и финальные проверки | 30 мин |
|
||||
|
||||
Шаг 1 — в одной сессии устройства, без перезагрузки и сна; устройством
|
||||
не пользоваться. Долгое — в фоне с журналом:
|
||||
|
||||
```sh
|
||||
nohup make bench-queue QUEUE_LABEL=queue10 > /tmp/bench-queue.log 2>&1 &
|
||||
tail -n 20 /tmp/bench-queue.log # раз в несколько минут
|
||||
```
|
||||
|
||||
**Точки остановки.** После шагов 0 и 1 — короткий отчёт пользователю
|
||||
(что снято, пометки, главные числа) и ожидание «продолжай». Что-то не по
|
||||
плану (`aborted`, `host_warnings`, упавшие проверки) — остановиться сразу,
|
||||
не переснимать молча.
|
||||
|
||||
---
|
||||
|
||||
## Шаг 0. Подготовка
|
||||
|
||||
- [ ] **Сверка с её репозиторием.** Если на этой машине есть её
|
||||
сданная лаба 4 (после замеров Codex), сравни её `src/`,
|
||||
`student-tests/`, `bench/`, `infra/`, `scripts/` с тем, что пришло в
|
||||
лабу 5 (`diff -r`): правки после замеров лабы 4, которых здесь нет, —
|
||||
список пользователю, без самостоятельного переноса.
|
||||
- [ ] Docker доступен (`docker version`; из sandbox — запуск вне него с
|
||||
одобрения пользователя). Ресурсы ВМ: `docker info --format
|
||||
'{{.NCPU}} CPU, {{.MemTotal}} bytes {{.KernelVersion}}'` — нужно **не
|
||||
меньше 10 CPU и 7,6 ГиБ** (лимиты стенда — 9,5 CPU и 6,0625 ГиБ,
|
||||
запас памяти 1,5 ГиБ; 0,5 CPU ВМ вне лимитов контейнеров).
|
||||
PostgreSQL — 1 CPU / 1 ГиБ, HAProxy — 0,5 CPU / 256 МиБ,
|
||||
S3 — 1 CPU / 768 МиБ,
|
||||
worker — 2 CPU / 1536 МиБ; ресурсы одинаковы во всех ячейках.
|
||||
Меньше — Settings → Resources Docker Desktop, перезапуск
|
||||
Docker; не помещается — сказать пользователю, лимиты профиля не
|
||||
уменьшать.
|
||||
- [ ] `make install`, `make lint`, самопроверки разбора:
|
||||
|
||||
```sh
|
||||
uv run python bench/queue.py --selftest
|
||||
uv run python bench/analysis.py --selftest
|
||||
uv run python bench/uploads.py --selftest
|
||||
uv run python bench/reconcile.py --selftest
|
||||
```
|
||||
|
||||
- [ ] `make env` (если `.env` нет), `make up`, `make test`, `make
|
||||
test-isolated` (в `results` все 0), `make failure-checks` (все
|
||||
passed). Красное — разобраться (корневой `AGENTS.md`, раздел 9),
|
||||
замеры не начинать.
|
||||
- [ ] Рабочий стенд остановлен: `make stop`; `docker ps` — пусто, чужие
|
||||
контейнеры не трогать (если есть — спросить пользователя).
|
||||
- [ ] Питание от сети, тяжёлые программы закрыты.
|
||||
- [ ] Самопроверка (метка `quick-queue-*` в отчёт не идёт):
|
||||
|
||||
```sh
|
||||
make bench-queue-quick
|
||||
```
|
||||
|
||||
В `summary.md` метки: нет `aborted`, раздел предупреждений о хосте
|
||||
пуст, в каждой ячейке есть все фазы, принятых загрузок без `ready`
|
||||
— 0, предел достигнут («429 после предела» не пусто). На машине
|
||||
разработки самопроверка дала: `s` T 5,6 / 11,2 гот./с (c1p1 / c4p4),
|
||||
`x` 1,08 / 1,68 гот./с, пик worker на `x` c4p4 1244 МиБ — порядок
|
||||
чисел должен быть похожим.
|
||||
- [ ] Гипотезы Q1–Q5 уже записаны в `REPORT.md` (4.2) **до** замеров —
|
||||
не переписывать; итоги — отдельными абзацами «Итог Q…».
|
||||
- [ ] Проект не под git: измеренный код называет `src_sha256` в
|
||||
паспорте метки. Если пользователь сделал git-репозиторий — коммит
|
||||
перед замером только с его согласия.
|
||||
|
||||
**Остановка:** отчёт пользователю — ресурсы ВМ, результаты проверок и
|
||||
самопроверки (пропускная способность классов в ней).
|
||||
|
||||
## Шаг 1. Трек 1: очередь под перегрузкой
|
||||
|
||||
```sh
|
||||
nohup make bench-queue QUEUE_LABEL=queue10 > /tmp/bench-queue.log 2>&1 &
|
||||
```
|
||||
|
||||
Метка `queue10` (`evidence/queue/queue10/`). Для нового прогона —
|
||||
уникальный `QUEUE_LABEL`: существующие метки не перезаписываются.
|
||||
|
||||
Проверь `summary.md`:
|
||||
- [ ] `aborted` нет, предупреждений о хосте нет;
|
||||
- [ ] все 4 настройки × 3 класса × 2 повтора; у каждой ячейки фазы A–D;
|
||||
- [ ] принятых без `ready` (failed или пропавших) — 0; если нет — это
|
||||
опровержение Q5 (записать), а не сбой стенда;
|
||||
- [ ] у класса `x` при `c4p4` нет OOM worker; если есть — это результат
|
||||
Q3 (записать), не переснимать.
|
||||
|
||||
В `REPORT.md`, раздел 4.2 (строки `TODO (Codex)`):
|
||||
- таблицы классов из `summary.md` как есть и ссылка на метку;
|
||||
- итог каждой гипотезы Q1–Q5 — строка вердикта из `summary.md` с
|
||||
числами, затем абзац объяснения. Опора объяснения: у worker 2 CPU, а
|
||||
рендер в отдельном процессе упирается в CPU (concurrency выше 2 даёт
|
||||
очередь за CPU, а не скорость); prefetch держит сообщения во внутренней
|
||||
очереди worker и не ускоряет (зато забирает работу у других
|
||||
потребителей); память — сумма пиков одновременных процессов рендера
|
||||
(36 Мпикс — ~300 МиБ на процесс, `RENDER_MEMORY_MB` 768); предел 60
|
||||
держит очередь и время ожидания `≈ 60 / T`; фаза «fixed» сравнивает
|
||||
настройки при равной нагрузке, B — при равной загрузке (0,6 × T);
|
||||
- абзац «Компромисс latency/ресурсы и рекомендация» — строка
|
||||
рекомендации из `summary.md` и объяснение;
|
||||
- раздел 4.3 «Железо и стенд замеров» — из `meta.json` (`host`,
|
||||
`docker`, образы, `budget`);
|
||||
- строка таблицы доказательств «Трек 1» → `evidence/queue/queue10/`.
|
||||
|
||||
**Остановка:** отчёт пользователю — пропускная способность и p95 по
|
||||
классам и настройкам, доля 429, время восстановления, пики памяти,
|
||||
итоги Q1–Q5.
|
||||
|
||||
## Шаг 2. Отчёт и проверка
|
||||
|
||||
- [ ] `REPORT.md`: строк `TODO (Codex)` не осталось (`grep -n "TODO
|
||||
(Codex)" REPORT.md` — пусто), гипотезы не переписаны, ограничения
|
||||
дополнены тем, что обнаружено.
|
||||
- [ ] Корневой `AGENTS.md`: абзац «Состояние» — что снято, где.
|
||||
- [ ] `make lint`, `make check`, `make test-isolated` — после правок.
|
||||
- [ ] Остановить всё своё: `docker ps -a | grep vlad-bench` — пусто;
|
||||
рабочий стенд — как был до тебя.
|
||||
- [ ] Отчёт пользователю (корневой `AGENTS.md`, раздел 11).
|
||||
|
||||
## Если что-то пошло не так
|
||||
|
||||
| Симптом | Что делать |
|
||||
|---|---|
|
||||
| `leftovers of an interrupted bench run` | выполнить напечатанные команды уборки (только `vlad-bench*`) и повторить |
|
||||
| `another bench run holds bench/.bench.lock` | идёт другой прогон: дождаться |
|
||||
| `limits need N CPU, docker VM has M` | ресурсы ВМ (шаг 0); лимиты профиля не уменьшать |
|
||||
| загрузки фазы A не 202 | стенд не готов или предел занят остатком: `app-tail.log` метки, повтор один раз |
|
||||
| `aborted` | новая метка после разбора `app-tail.log` и `raw/` |
|
||||
| `host_warnings` | устранить причину и переснять под новым именем (`QUEUE_LABEL=baseline2`) |
|
||||
| OOM worker | результат Q3 — записать; переснимать не нужно |
|
||||
@@ -0,0 +1,546 @@
|
||||
"""Разбор результатов стенда без docker: журналы, окна, признаки, поиск R*.
|
||||
|
||||
Чистые функции над строками журналов генератора, событиями Docker,
|
||||
журналом HAProxy и тиками сэмплера. Их зовёт `run.py`; самопроверка —
|
||||
`uv run python bench/analysis.py --selftest`. Только stdlib.
|
||||
|
||||
Все отметки времени — часы ВМ Docker в миллисекундах эпохи: генератор
|
||||
(`time.time()` в контейнере), события Docker (`timeNano`), журнал HAProxy
|
||||
(`docker logs -t`), сэмплер. Часы хоста с ними не сравниваются.
|
||||
"""
|
||||
|
||||
import dataclasses
|
||||
import datetime
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import sys
|
||||
|
||||
# --- журнал запросов генератора (профили failover и rollout) ----------------
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Request:
|
||||
scheduled: float
|
||||
sent: float | None
|
||||
done: float | None
|
||||
kind: str
|
||||
status: int
|
||||
instance: str
|
||||
key: str
|
||||
error: str
|
||||
|
||||
@property
|
||||
def ok(self) -> bool:
|
||||
return self.status == (201 if self.kind == "write" else 200)
|
||||
|
||||
@property
|
||||
def latency(self) -> float | None:
|
||||
"""Задержка от момента по расписанию до ответа, мс."""
|
||||
return None if self.done is None else self.done - self.scheduled
|
||||
|
||||
|
||||
def parse_requests(text: str) -> list[Request]:
|
||||
out = []
|
||||
for line in text.splitlines():
|
||||
parts = line.split(",")
|
||||
if len(parts) != 9 or parts[0] != "fo":
|
||||
continue
|
||||
out.append(
|
||||
Request(
|
||||
scheduled=float(parts[1]),
|
||||
sent=float(parts[2]) if parts[2] else None,
|
||||
done=float(parts[3]) if parts[3] else None,
|
||||
kind=parts[4],
|
||||
status=int(parts[5]),
|
||||
instance=parts[6],
|
||||
key=parts[7],
|
||||
error=parts[8],
|
||||
),
|
||||
)
|
||||
out.sort(key=lambda r: r.scheduled)
|
||||
return out
|
||||
|
||||
|
||||
def percentile(values: list[float], fraction: float) -> float | None:
|
||||
if not values:
|
||||
return None
|
||||
ordered = sorted(values)
|
||||
return ordered[max(1, math.ceil(fraction * len(ordered))) - 1]
|
||||
|
||||
|
||||
def error_label(request: Request) -> str:
|
||||
return request.error or str(request.status)
|
||||
|
||||
|
||||
def window_stats(requests: list[Request], start: float, end: float) -> dict:
|
||||
"""Запросы окна [start, end) по моменту расписания."""
|
||||
inside = [r for r in requests if start <= r.scheduled < end]
|
||||
seconds = max((end - start) / 1000, 1e-9)
|
||||
ok = [r for r in inside if r.ok]
|
||||
errors: dict[str, int] = {}
|
||||
for request in inside:
|
||||
if not request.ok:
|
||||
errors[error_label(request)] = errors.get(error_label(request), 0) + 1
|
||||
latencies = [r.latency for r in ok if r.latency is not None]
|
||||
return {
|
||||
"seconds": round(seconds, 2),
|
||||
"requests": len(inside),
|
||||
"ok": len(ok),
|
||||
"ok_rps": round(len(ok) / seconds, 1),
|
||||
"ok_write_rps": round(sum(r.kind == "write" for r in ok) / seconds, 1),
|
||||
"ok_read_rps": round(sum(r.kind == "read" for r in ok) / seconds, 1),
|
||||
"errors": sum(errors.values()),
|
||||
"error_kinds": errors,
|
||||
"p50_ms": _r(percentile(latencies, 0.5)),
|
||||
"p95_ms": _r(percentile(latencies, 0.95)),
|
||||
"p99_ms": _r(percentile(latencies, 0.99)),
|
||||
"max_ms": _r(max(latencies) if latencies else None),
|
||||
}
|
||||
|
||||
|
||||
def _r(value: float | None, digits: int = 1) -> float | None:
|
||||
return None if value is None else round(value, digits)
|
||||
|
||||
|
||||
def timeline(
|
||||
requests: list[Request],
|
||||
origin: float,
|
||||
start: float,
|
||||
end: float,
|
||||
groups: dict[str, set[str]] | None = None,
|
||||
) -> list[dict]:
|
||||
"""Посекундная шкала от `origin`: успешные по видам, ошибки, реплики.
|
||||
|
||||
`groups` — именованные наборы X-Instance-Id (цель и остальные, старые
|
||||
и новые): у каждого — число успешных ответов в секунду.
|
||||
"""
|
||||
first = math.floor((start - origin) / 1000)
|
||||
last = math.ceil((end - origin) / 1000)
|
||||
rows = []
|
||||
for second in range(first, last):
|
||||
lo, hi = origin + second * 1000, origin + (second + 1) * 1000
|
||||
inside = [r for r in requests if lo <= r.scheduled < hi]
|
||||
ok = [r for r in inside if r.ok]
|
||||
errors: dict[str, int] = {}
|
||||
for request in inside:
|
||||
if not request.ok:
|
||||
errors[error_label(request)] = errors.get(error_label(request), 0) + 1
|
||||
latencies = [r.latency for r in ok if r.latency is not None]
|
||||
row = {
|
||||
"second": second,
|
||||
"sent": len(inside),
|
||||
"ok_write": sum(r.kind == "write" for r in ok),
|
||||
"ok_read": sum(r.kind == "read" for r in ok),
|
||||
"errors": sum(errors.values()),
|
||||
"error_kinds": ";".join(f"{k}:{v}" for k, v in sorted(errors.items())),
|
||||
"p95_ms": _r(percentile(latencies, 0.95)),
|
||||
"max_ms": _r(max(latencies) if latencies else None),
|
||||
}
|
||||
for name, ids in (groups or {}).items():
|
||||
row[f"ok_{name}"] = sum(r.instance in ids for r in ok)
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def recovery_seconds(
|
||||
requests: list[Request],
|
||||
action: float,
|
||||
end: float,
|
||||
offered_per_s: float,
|
||||
) -> float | None:
|
||||
"""Время восстановления, с: от действия до начала первой секунды, после
|
||||
которой до `end` нет ни одной ошибки и успешных не меньше 90 % от
|
||||
поданной нагрузки в каждой секунде. None — не восстановилось."""
|
||||
rows = timeline(requests, action, action, end)
|
||||
good = [
|
||||
row["errors"] == 0 and row["ok_write"] + row["ok_read"] >= 0.9 * offered_per_s
|
||||
for row in rows
|
||||
]
|
||||
for index in range(len(good)):
|
||||
if all(good[index:]):
|
||||
return float(rows[index]["second"])
|
||||
return None
|
||||
|
||||
|
||||
# --- события Docker и журнал HAProxy ------------------------------------------------
|
||||
|
||||
|
||||
def parse_events(text: str) -> list[tuple[float, str, str]]:
|
||||
"""`docker events --format '{{json .}}'`: (мс, действие, id контейнера)."""
|
||||
events = []
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
data = json.loads(line)
|
||||
action = data.get("Action") or data.get("status") or ""
|
||||
actor = (data.get("Actor") or {}).get("ID") or data.get("id") or ""
|
||||
events.append((int(data["timeNano"]) / 1e6, action.split(":")[0], actor))
|
||||
events.sort()
|
||||
return events
|
||||
|
||||
|
||||
_LB_STATE = re.compile(
|
||||
r"Server api/(?P<slot>api-\d+)\b.*?(?P<state>is DOWN|is UP|is going DOWN"
|
||||
r"|enters drain|enters maintenance|is leaving forced drain|is leaving maintenance"
|
||||
r"|administratively READY)",
|
||||
)
|
||||
|
||||
|
||||
def parse_docker_time(stamp: str) -> float:
|
||||
"""RFC 3339 с наносекундами из `docker logs -t` → мс эпохи."""
|
||||
stamp = stamp.rstrip("Z")
|
||||
main, _, fraction = stamp.partition(".")
|
||||
base = datetime.datetime.fromisoformat(main).replace(tzinfo=datetime.UTC)
|
||||
return base.timestamp() * 1000 + float(f"0.{fraction or 0}") * 1000
|
||||
|
||||
|
||||
def parse_lb_log(text: str) -> list[tuple[float, str, str]]:
|
||||
"""Смены состояния слотов: (мс, слот, состояние). Дубли строк HAProxy
|
||||
(мастер и воркер пишут одну и ту же) схлопываются."""
|
||||
seen = set()
|
||||
out = []
|
||||
for line in text.splitlines():
|
||||
stamp, _, message = line.partition(" ")
|
||||
match = _LB_STATE.search(message)
|
||||
if not match:
|
||||
continue
|
||||
when = parse_docker_time(stamp)
|
||||
item = (round(when / 50), match.group("slot"), match.group("state"))
|
||||
if item in seen:
|
||||
continue
|
||||
seen.add(item)
|
||||
out.append((when, match.group("slot"), match.group("state")))
|
||||
out.sort()
|
||||
return out
|
||||
|
||||
|
||||
def lb_http_errors(text: str) -> dict[str, int]:
|
||||
"""Ответы с ошибкой в журнале HAProxy (option dontlog-normal пишет
|
||||
только их): статус → число."""
|
||||
counts: dict[str, int] = {}
|
||||
for line in text.splitlines():
|
||||
match = re.search(r'\s(\d{3}) \d+ - - (\S{4}) .*"(GET|POST) ', line)
|
||||
if match and match.group(1)[0] in "45":
|
||||
counts[match.group(1)] = counts.get(match.group(1), 0) + 1
|
||||
return counts
|
||||
|
||||
|
||||
# --- сэмплер ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def window_ticks(ticks: list[dict], start: float, end: float) -> list[dict]:
|
||||
"""Тики окна [start, end] (секунды эпохи) плюс по одному снаружи:
|
||||
приращения считаются по краям окна."""
|
||||
inside = [t for t in ticks if start <= t["ts"] <= end]
|
||||
before = [t for t in ticks if t["ts"] < start]
|
||||
after = [t for t in ticks if t["ts"] > end]
|
||||
return ([before[-1]] if before else []) + inside + ([after[0]] if after else [])
|
||||
|
||||
|
||||
def cgroup_usage(ticks: list[dict], container: str) -> dict | None:
|
||||
"""CPU (ядер), доля троттлинга и память контейнера по тикам окна.
|
||||
|
||||
Каждый соседний интервал считается отдельно: новая cgroup (контейнер
|
||||
перезапущен) даёт отрицательное приращение — такой интервал
|
||||
пропускается, а не обнуляет окно.
|
||||
"""
|
||||
points = [(t["ts"], t["cg"].get(container[:12])) for t in ticks]
|
||||
points = [(ts, line) for ts, line in points if line]
|
||||
if len(points) < 2:
|
||||
return None
|
||||
cpu_us = seconds = periods = throttled = 0.0
|
||||
memory = 0
|
||||
for (t0, a), (t1, b) in zip(points, points[1:], strict=False):
|
||||
if b[0] < a[0]:
|
||||
continue
|
||||
cpu_us += b[0] - a[0]
|
||||
seconds += t1 - t0
|
||||
periods += b[1] - a[1]
|
||||
throttled += b[2] - a[2]
|
||||
memory = max(memory, a[4], b[4])
|
||||
if seconds <= 0:
|
||||
return None
|
||||
return {
|
||||
"cpu_cores": round(cpu_us / seconds / 1e6, 3),
|
||||
"throttled_share": round(throttled / periods, 4) if periods else 0.0,
|
||||
"memory_mib": round(memory / 2**20, 1),
|
||||
}
|
||||
|
||||
|
||||
def vm_busy(ticks: list[dict]) -> float | None:
|
||||
if len(ticks) < 2:
|
||||
return None
|
||||
busy = ticks[-1]["vm"][0] - ticks[0]["vm"][0]
|
||||
total = ticks[-1]["vm"][1] - ticks[0]["vm"][1]
|
||||
return round(busy / total, 4) if total else None
|
||||
|
||||
|
||||
def lb_delta(ticks: list[dict], field: str) -> int | None:
|
||||
"""Приращение счётчика HAProxy по всем слотам за окно."""
|
||||
with_lb = [t for t in ticks if t.get("lb")]
|
||||
if len(with_lb) < 2:
|
||||
return None
|
||||
|
||||
def total(tick: dict) -> int:
|
||||
return sum(
|
||||
row.get(field, 0) for name, row in tick["lb"].items() if name != "FRONTEND"
|
||||
)
|
||||
|
||||
return total(with_lb[-1]) - total(with_lb[0])
|
||||
|
||||
|
||||
def pg_stats(ticks: list[dict], replica_addrs: set[str]) -> dict:
|
||||
"""Соединения приложения и счётчики базы за окно."""
|
||||
with_pg = [t for t in ticks if t.get("pg")]
|
||||
if not with_pg:
|
||||
return {}
|
||||
per_tick_total, per_replica_max, active, lock_waits = [], 0, [], []
|
||||
for tick in with_pg:
|
||||
by_addr: dict[str, int] = {}
|
||||
busy = waiting = 0
|
||||
for addr, state, wait, count in tick["pg"]["conn"]:
|
||||
by_addr[addr] = by_addr.get(addr, 0) + count
|
||||
if state == "active":
|
||||
busy += count
|
||||
if wait == "Lock":
|
||||
waiting += count
|
||||
per_tick_total.append(sum(by_addr.values()))
|
||||
known = [
|
||||
n for a, n in by_addr.items() if not replica_addrs or a in replica_addrs
|
||||
]
|
||||
per_replica_max = max([per_replica_max, *known])
|
||||
active.append(busy)
|
||||
lock_waits.append(waiting)
|
||||
first, last = with_pg[0]["pg"]["db"], with_pg[-1]["pg"]["db"]
|
||||
seconds = max(with_pg[-1]["ts"] - with_pg[0]["ts"], 1e-9)
|
||||
return {
|
||||
"pg_conn_max": max(per_tick_total),
|
||||
"pg_conn_replica_max": per_replica_max,
|
||||
"pg_active_mean": round(sum(active) / len(active), 2),
|
||||
"pg_lock_wait_mean": round(sum(lock_waits) / len(lock_waits), 2),
|
||||
"pg_new_sessions": last["sessions"] - first["sessions"],
|
||||
"pg_commits_per_s": round(
|
||||
(last["xact_commit"] - first["xact_commit"]) / seconds, 1
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# --- признаки узкого места (серии трека 1) ------------------------------------------
|
||||
|
||||
# Пороги заданы до прогонов (REPORT.md, гипотезы трека 1). Сработавший
|
||||
# признак на первой нарушившей ступени называет, во что упёрлась система
|
||||
THRESHOLDS = {
|
||||
# реплика: доля своего лимита CPU или заметный троттлинг CFS
|
||||
"app_cpu_share": 0.85,
|
||||
"app_throttled_share": 0.20,
|
||||
# PostgreSQL: доля лимита CPU, средние ожидания блокировок
|
||||
"db_cpu_share": 0.85,
|
||||
"db_lock_wait_mean": 1.0,
|
||||
# пул: соединений одной реплики не меньше доли её предела (размер + переполнение)
|
||||
"pool_share": 0.90,
|
||||
# переоткрытие соединений: новых сессий PostgreSQL за окно ступени
|
||||
"churn_sessions": 20,
|
||||
# LB: доля его лимита CPU
|
||||
"lb_cpu_share": 0.85,
|
||||
# ВМ Docker занята почти целиком: реплики ждут CPU хоста
|
||||
"host_busy_share": 0.90,
|
||||
# генератор: доля лимита CPU или отставание от расписания
|
||||
"gen_cpu_share": 0.90,
|
||||
"gen_lag_p99_ms": 50.0,
|
||||
}
|
||||
|
||||
|
||||
def signals(row: dict, limits: dict, pool_cap: int) -> list[str]:
|
||||
"""Признаки узкого места ступени по агрегатам `row` (см. run.py)."""
|
||||
found = []
|
||||
if (row.get("api_cpu_max") or 0) >= THRESHOLDS["app_cpu_share"] * limits[
|
||||
"api_cpus"
|
||||
] or (row.get("api_throttled_max") or 0) >= THRESHOLDS["app_throttled_share"]:
|
||||
found.append("app")
|
||||
if (row.get("db_cpu_cores") or 0) >= THRESHOLDS["db_cpu_share"] * limits["db_cpus"]:
|
||||
found.append("db_cpu")
|
||||
if (row.get("pg_lock_wait_mean") or 0) >= THRESHOLDS["db_lock_wait_mean"]:
|
||||
found.append("db_lock")
|
||||
if (
|
||||
pool_cap
|
||||
and (row.get("pg_conn_replica_max") or 0) >= THRESHOLDS["pool_share"] * pool_cap
|
||||
):
|
||||
found.append("pool")
|
||||
if (row.get("pg_new_sessions") or 0) >= THRESHOLDS["churn_sessions"]:
|
||||
found.append("churn")
|
||||
if (row.get("lb_cpu_cores") or 0) >= THRESHOLDS["lb_cpu_share"] * limits["lb_cpus"]:
|
||||
found.append("lb")
|
||||
if (row.get("vm_busy_share") or 0) >= THRESHOLDS["host_busy_share"]:
|
||||
found.append("host")
|
||||
if (
|
||||
(row.get("gen_cpu_cores") or 0)
|
||||
>= THRESHOLDS["gen_cpu_share"] * limits["gen_cpus"]
|
||||
or (row.get("lag_p99_ms") or 0) >= THRESHOLDS["gen_lag_p99_ms"]
|
||||
or (row.get("dropped") or 0) > 0
|
||||
):
|
||||
found.append("generator")
|
||||
return found
|
||||
|
||||
|
||||
# --- поиск R* ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class Search:
|
||||
"""Автомат поиска R*: наибольшей интенсивности, на которой ступень
|
||||
выдерживает SLO.
|
||||
|
||||
Сначала фиксированные уровни (сравнение конфигураций на одной
|
||||
нагрузке), затем лестница `base × factor^i`: начало — первая ступенька
|
||||
не ниже `start`, подъём до первой нарушившей (если нарушена первая —
|
||||
спуск до первой прошедшей), затем `bisections` делений интервала
|
||||
«прошла — нарушила» по среднему геометрическому, с округлением до
|
||||
целого. R* — наибольшая прошедшая ступень поиска.
|
||||
"""
|
||||
|
||||
fixed: tuple[float, ...]
|
||||
base: float
|
||||
factor: float
|
||||
start: float
|
||||
bisections: int
|
||||
cap: float
|
||||
floor: float = 25.0
|
||||
results: list[tuple[float, bool, str]] = dataclasses.field(default_factory=list)
|
||||
|
||||
def ladder(self, rate: float) -> float:
|
||||
steps = math.ceil(math.log(rate / self.base, self.factor) - 1e-9)
|
||||
return round(self.base * self.factor ** max(steps, 0))
|
||||
|
||||
def record(self, rate: float, passed: bool, phase: str) -> None:
|
||||
self.results.append((rate, passed, phase))
|
||||
|
||||
def next_rate(self) -> tuple[float, str] | None:
|
||||
done_fixed = [r for r in self.results if r[2] == "fixed"]
|
||||
if len(done_fixed) < len(self.fixed):
|
||||
return self.fixed[len(done_fixed)], "fixed"
|
||||
search = [r for r in self.results if r[2] != "fixed"]
|
||||
passed = [rate for rate, ok, _ in search if ok]
|
||||
failed = [rate for rate, ok, _ in search if not ok]
|
||||
if not search:
|
||||
fixed_pass = [rate for rate, ok, _ in done_fixed if ok]
|
||||
start = max([self.start, *fixed_pass])
|
||||
return min(self.ladder(start), self.cap), "ladder"
|
||||
phase = search[-1][2]
|
||||
if phase == "ladder":
|
||||
last_rate, last_ok, _ = search[-1]
|
||||
if last_ok and not failed:
|
||||
if last_rate >= self.cap:
|
||||
return None
|
||||
return min(round(last_rate * self.factor), self.cap), "ladder"
|
||||
if not last_ok and not passed:
|
||||
lower = round(last_rate / self.factor)
|
||||
return (lower, "ladder") if lower >= self.floor else None
|
||||
# интервал найден: деления
|
||||
if not passed or not failed:
|
||||
return None
|
||||
bisections = sum(1 for r in search if r[2] == "bisect")
|
||||
if bisections >= self.bisections:
|
||||
return None
|
||||
lo = max(passed)
|
||||
hi = (
|
||||
min(rate for rate in failed if rate > lo)
|
||||
if any(r > lo for r in failed)
|
||||
else None
|
||||
)
|
||||
if hi is None:
|
||||
return None
|
||||
middle = round(math.sqrt(lo * hi))
|
||||
if middle <= lo or middle >= hi:
|
||||
return None
|
||||
return middle, "bisect"
|
||||
|
||||
def r_star(self) -> float | None:
|
||||
passed = [rate for rate, ok, phase in self.results if ok and phase != "fixed"]
|
||||
return max(passed) if passed else None
|
||||
|
||||
def first_fail(self) -> float | None:
|
||||
r_star = self.r_star() or 0
|
||||
failed = [
|
||||
rate for rate, ok, phase in self.results if not ok and phase != "fixed"
|
||||
]
|
||||
above = [rate for rate in failed if rate > r_star]
|
||||
return min(above) if above else None
|
||||
|
||||
|
||||
# --- --selftest: разбор на синтетических данных ---------------------------------------
|
||||
|
||||
|
||||
def _selftest() -> None:
|
||||
# поиск: подъём, нарушение, два деления
|
||||
search = Search(
|
||||
fixed=(250, 500), base=100, factor=1.25, start=600, bisections=2, cap=8000
|
||||
)
|
||||
capacity = 1000
|
||||
plan = []
|
||||
while (step := search.next_rate()) is not None:
|
||||
rate, phase = step
|
||||
plan.append((rate, phase))
|
||||
search.record(rate, rate <= capacity, phase)
|
||||
assert plan[:2] == [(250, "fixed"), (500, "fixed")], plan
|
||||
# лестница 100 × 1,25^i: первая ступенька не ниже 600 — 745
|
||||
assert plan[2] == (745, "ladder"), plan
|
||||
assert search.r_star() is not None, plan
|
||||
assert search.r_star() <= capacity, plan
|
||||
assert search.first_fail() > capacity, plan
|
||||
assert sum(phase == "bisect" for _, phase in plan) == 2, plan
|
||||
# первая ступень поиска нарушена — спуск
|
||||
search = Search(fixed=(), base=100, factor=1.25, start=300, bisections=1, cap=8000)
|
||||
plan = []
|
||||
while (step := search.next_rate()) is not None:
|
||||
plan.append(step)
|
||||
search.record(step[0], step[0] <= 200, step[1])
|
||||
assert plan[0] == (305, "ladder"), plan
|
||||
assert plan[1] == (244, "ladder"), plan
|
||||
assert search.r_star() <= 200, plan
|
||||
|
||||
# окно и время восстановления
|
||||
reqs = [
|
||||
Request(
|
||||
1000 + i * 10, 1000 + i * 10, 1005 + i * 10, "write", 201, "a", f"k{i}", ""
|
||||
)
|
||||
for i in range(500)
|
||||
]
|
||||
reqs[200] = dataclasses.replace(reqs[200], status=502)
|
||||
stats = window_stats(reqs, 1000, 6000)
|
||||
assert stats["errors"] == 1, stats
|
||||
assert stats["ok"] == 499, stats
|
||||
assert recovery_seconds(reqs, 1000, 6000, 100) == 3.0
|
||||
rows = timeline(reqs, 1000, 1000, 6000, {"a": {"a"}})
|
||||
assert rows[2]["errors"] == 1, rows[:3]
|
||||
assert rows[0]["ok_a"] == 100, rows[:3]
|
||||
|
||||
# сэмплер: приращения, перезапуск контейнера
|
||||
ticks = [
|
||||
{"ts": 0.0, "cg": {"abc": [0, 0, 0, 0, 100]}, "vm": [0, 100]},
|
||||
{"ts": 1.0, "cg": {"abc": [500_000, 10, 1, 0, 200]}, "vm": [50, 200]},
|
||||
{"ts": 2.0, "cg": {"abc": [100_000, 2, 0, 0, 50]}, "vm": [60, 300]},
|
||||
{"ts": 3.0, "cg": {"abc": [600_000, 12, 0, 0, 80]}, "vm": [160, 400]},
|
||||
]
|
||||
usage = cgroup_usage(ticks, "abc")
|
||||
assert usage["cpu_cores"] == 0.5, usage
|
||||
assert vm_busy(ticks) == 0.5333, vm_busy(ticks)
|
||||
|
||||
# журнал LB: дубли схлопываются
|
||||
log = (
|
||||
"2026-10-14T10:00:00.100000000Z Server api/api-1 is DOWN, reason: x\n"
|
||||
"2026-10-14T10:00:00.101000000Z [WARNING] (8) : Server api/api-1 is DOWN, reason: x\n"
|
||||
"2026-10-14T10:00:03.000000000Z Server api/api-1 is UP, reason: y\n"
|
||||
)
|
||||
parsed = parse_lb_log(log)
|
||||
assert [p[2] for p in parsed] == ["is DOWN", "is UP"], parsed
|
||||
print("analysis selftest: ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if sys.argv[1:] == ["--selftest"]:
|
||||
_selftest()
|
||||
else:
|
||||
print("usage: python bench/analysis.py --selftest", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
@@ -0,0 +1,153 @@
|
||||
# Override нагрузочного стенда (bench/run.py): тот же compose.yaml под
|
||||
# проектом vlad-bench, со своими volume, сетью, именами и лимитами. Пины
|
||||
# хранилищ и LB, healthcheck хранилищ, valkey.conf, haproxy.cfg и
|
||||
# миграции — из основного файла. Подстановки — bench/bench.env, а не .env
|
||||
# рабочего стенда; пароли, образ и BENCH_* передаёт run.py (поля профиля),
|
||||
# без них стенд не поднимается. Что не меняется между метками —
|
||||
# bench/AGENTS.md.
|
||||
#
|
||||
# Лимиты — в той же форме deploy.resources.limits, что у основного файла:
|
||||
# cpus / mem_limit рядом с ними compose не принимает. Swap выключен
|
||||
# (memswap_limit = memory). Портов на хост нет: генератор и сэмплер ходят
|
||||
# внутри сети vlad-bench-net.
|
||||
#
|
||||
# S3, s3-bootstrap и уборщик (лаба 4) запускают только профили uploads и
|
||||
# reconcile: остальные поднимают сервисы списком и реплики с --no-deps, их
|
||||
# методика не меняется. Секреты S3 compose требует при разборе файла (:?),
|
||||
# поэтому run.py генерирует их на каждый прогон любого профиля.
|
||||
#
|
||||
# Лаба 5: брокер, relay и worker поднимает только профиль queue. Пароль
|
||||
# брокера, ключи worker в S3 и BENCH_MQ_*, BENCH_WORKER_* run.py тоже
|
||||
# передаёт всем профилям: файл разбирается целиком. Предел незавершённых
|
||||
# заданий у реплик по умолчанию снят (0): профили лабы 4 мерят загрузку,
|
||||
# а не очередь.
|
||||
services:
|
||||
postgres:
|
||||
container_name: vlad-bench-postgres
|
||||
# без автоперезапуска: если база упадёт, run.py поставит прогону stand_down
|
||||
restart: "no"
|
||||
command: ["postgres", "-c", "max_connections=${BENCH_DB_MAX_CONNECTIONS:?run make bench-*}"]
|
||||
ports: !reset []
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_DB_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_DB_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_DB_MEMORY:?run make bench-*}
|
||||
|
||||
valkey:
|
||||
container_name: vlad-bench-valkey
|
||||
restart: "no"
|
||||
ports: !reset []
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_CACHE_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_CACHE_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_CACHE_MEMORY:?run make bench-*}
|
||||
|
||||
# образ, аргументы, healthcheck и GOMEMLIMIT — из основного файла;
|
||||
# volume — свой у проекта (vlad-bench_s3data), профиль uploads заводит
|
||||
# новый на каждый прогон
|
||||
s3:
|
||||
container_name: vlad-bench-s3
|
||||
restart: "no"
|
||||
ports: !reset []
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_S3_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_S3_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_S3_MEMORY:?run make bench-*}
|
||||
|
||||
migrate:
|
||||
image: ${BENCH_IMAGE:?run make bench-*}
|
||||
|
||||
# только разовые проходы профиля reconcile (docker compose run): лимит CPU
|
||||
# прохода — BENCH_CLEANER_CPUS вызова
|
||||
cleaner:
|
||||
image: ${BENCH_IMAGE:?run make bench-*}
|
||||
restart: "no"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_CLEANER_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_CLEANER_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_CLEANER_MEMORY:?run make bench-*}
|
||||
|
||||
# реплики без container_name: их число — BENCH_API_REPLICAS (--scale)
|
||||
api:
|
||||
image: ${BENCH_IMAGE:?run make bench-*}
|
||||
restart: "no"
|
||||
# проверка образа — второй процесс Python раз в несколько секунд в квоте
|
||||
# CPU реплики; готовность и так проверяет LB раз в секунду
|
||||
healthcheck:
|
||||
disable: true
|
||||
environment:
|
||||
SESSION_TTL_SECONDS: ${BENCH_SESSION_TTL:?run make bench-*}
|
||||
UPLOAD_MAX_UNFINISHED: ${BENCH_UPLOAD_MAX_UNFINISHED:-0}
|
||||
deploy:
|
||||
replicas: ${BENCH_API_REPLICAS:?run make bench-*}
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_API_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_API_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_API_MEMORY:?run make bench-*}
|
||||
|
||||
haproxy:
|
||||
container_name: vlad-bench-lb
|
||||
restart: "no"
|
||||
ports: !reset []
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_LB_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_LB_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_LB_MEMORY:?run make bench-*}
|
||||
|
||||
# брокер: образ, healthcheck, rabbitmq.conf и hostname — из основного
|
||||
# файла; volume свой у проекта (vlad-bench_rmqdata), down -v его удаляет
|
||||
rabbitmq:
|
||||
container_name: vlad-bench-mq
|
||||
restart: "no"
|
||||
ports: !reset []
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_MQ_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_MQ_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_MQ_MEMORY:?run make bench-*}
|
||||
|
||||
# relay — лимиты рабочего стенда, в профиле queue не меняются
|
||||
relay:
|
||||
container_name: vlad-bench-relay
|
||||
image: ${BENCH_IMAGE:?run make bench-*}
|
||||
restart: "no"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 192m
|
||||
memswap_limit: 192m
|
||||
|
||||
# worker: лимиты одни на все настройки профиля queue, меняются только
|
||||
# concurrency и prefetch (пересоздание сервиса на ячейку)
|
||||
worker:
|
||||
container_name: vlad-bench-worker
|
||||
image: ${BENCH_IMAGE:?run make bench-*}
|
||||
restart: "no"
|
||||
environment:
|
||||
WORKER_CONCURRENCY: ${BENCH_WORKER_CONCURRENCY:?run make bench-*}
|
||||
WORKER_PREFETCH: ${BENCH_WORKER_PREFETCH:?run make bench-*}
|
||||
RENDER_MEMORY_MB: ${BENCH_WORKER_RENDER_MEMORY_MB:?run make bench-*}
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_WORKER_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_WORKER_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_WORKER_MEMORY:?run make bench-*}
|
||||
|
||||
networks:
|
||||
default:
|
||||
# генератор и сэмплер подключаются к сети стенда по имени
|
||||
name: vlad-bench-net
|
||||
@@ -0,0 +1,8 @@
|
||||
# Подстановки compose для нагрузочного стенда вместо .env рабочего стенда
|
||||
# (docker compose --env-file bench/bench.env). Только несекретные имена:
|
||||
# пароли и ключи S3 генерирует bench/run.py на каждый прогон, лимиты, образ,
|
||||
# число реплик и срок сессии приходят из профиля (BENCH_*).
|
||||
POSTGRES_USER=bench
|
||||
POSTGRES_DB=bench
|
||||
LOG_LEVEL=INFO
|
||||
S3_BUCKET=bench
|
||||
+1137
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,248 @@
|
||||
"""Сравнение двух меток серий трека 1: «до» и «после» одного изменения.
|
||||
|
||||
uv run python bench/compare.py --before scaling-ab-before \\
|
||||
--after scaling-ab-after [--aa scaling-ab-aa]
|
||||
→ evidence/compare/<до>__<после>.md
|
||||
|
||||
По общим конфигурациям: R* (медиана, min–max, Δ), задержки на
|
||||
фиксированных уровнях, CPU на запрос реплик и PostgreSQL, новые сессии
|
||||
PostgreSQL и соединения на ступени R* и на первой нарушившей. С `--aa`
|
||||
(метка того же кода, что «до») — уровень шума: наибольшее |Δ медианы R*|
|
||||
между «до» и A/A по общим конфигурациям.
|
||||
|
||||
Изменение R* — эффект, только если: разбросы повторов не пересекаются,
|
||||
|Δ медианы| больше шума A/A и не меньше двух ступеней сетки поиска
|
||||
(R* после / R* до ≥ √factor ≈ 1,118 при factor 1,25: шаг сетки после двух
|
||||
делений — factor^¼, одна ступень — в пределах разрешения поиска). Иначе
|
||||
— «шум» или «в пределах разрешения».
|
||||
|
||||
Методика меток должна совпадать: профиль (кроме списка ячеек серии ab),
|
||||
лимиты, образы хранилищ, хост. Различаться может только код приложения
|
||||
(`src_sha256`). Иначе — отказ.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SCALING = ROOT / "evidence" / "scaling"
|
||||
OUT = ROOT / "evidence" / "compare"
|
||||
|
||||
|
||||
def read_csv(path: Path) -> list[dict]:
|
||||
with path.open(encoding="utf-8") as handle:
|
||||
return list(csv.DictReader(handle))
|
||||
|
||||
|
||||
def num(value) -> float | None:
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def load(label: str) -> dict:
|
||||
folder = SCALING / label
|
||||
if not (folder / "meta.json").exists():
|
||||
raise SystemExit(f"no label {label} in evidence/scaling")
|
||||
return {
|
||||
"label": label,
|
||||
"meta": json.loads((folder / "meta.json").read_text(encoding="utf-8")),
|
||||
"searches": read_csv(folder / "searches.csv"),
|
||||
"runs": read_csv(folder / "runs.csv"),
|
||||
}
|
||||
|
||||
|
||||
def methodology(meta: dict) -> dict:
|
||||
"""Что должно совпасть у сравниваемых меток."""
|
||||
profile = dict(meta["profile"])
|
||||
profile.pop("configs", None)
|
||||
stand = meta.get("stand") or {}
|
||||
return {
|
||||
"profile": profile,
|
||||
"series_name": profile.get("name"),
|
||||
"configs": meta.get("configs"),
|
||||
"cpu_model": meta["host"].get("cpu_model"),
|
||||
"vm_cpus": meta["docker"].get("NCPU"),
|
||||
"postgres_image": (stand.get("postgres") or {}).get("image"),
|
||||
}
|
||||
|
||||
|
||||
def check_methodology(before: dict, after: dict) -> list[str]:
|
||||
a, b = methodology(before["meta"]), methodology(after["meta"])
|
||||
problems = []
|
||||
if a["profile"] != b["profile"]:
|
||||
problems.append("profile differs")
|
||||
if a["cpu_model"] != b["cpu_model"] or a["vm_cpus"] != b["vm_cpus"]:
|
||||
problems.append("host or docker VM differs")
|
||||
common = set(a["configs"] or {}) & set(b["configs"] or {})
|
||||
for config in sorted(common):
|
||||
if a["configs"][config] != b["configs"][config]:
|
||||
problems.append(f"limits of {config} differ")
|
||||
return problems
|
||||
|
||||
|
||||
def stats_by_config(data: dict) -> dict:
|
||||
out = {}
|
||||
for row in data["searches"]:
|
||||
entry = out.setdefault(row["config"], {"r": [], "cells": []})
|
||||
if num(row["r_star"]) is not None:
|
||||
entry["r"].append(num(row["r_star"]))
|
||||
entry["cells"].append(row)
|
||||
for entry in out.values():
|
||||
values = entry["r"] or [0.0]
|
||||
entry["median"] = statistics.median(values)
|
||||
entry["min"], entry["max"] = min(values), max(values)
|
||||
return out
|
||||
|
||||
|
||||
def step_metric(data: dict, config: str, which: str, key: str) -> float | None:
|
||||
"""Медиана поля на ступени R* (`star`) или первой нарушившей (`fail`)."""
|
||||
values = []
|
||||
for search in data["searches"]:
|
||||
if search["config"] != config:
|
||||
continue
|
||||
rate = num(search["r_star"] if which == "star" else search["first_fail"])
|
||||
row = next(
|
||||
(
|
||||
r
|
||||
for r in data["runs"]
|
||||
if r["cell"] == search["cell"]
|
||||
and r["phase"] != "fixed"
|
||||
and num(r["rate"]) == rate
|
||||
),
|
||||
None,
|
||||
)
|
||||
if row is not None and num(row.get(key)) is not None:
|
||||
values.append(num(row[key]))
|
||||
return statistics.median(values) if values else None
|
||||
|
||||
|
||||
def fixed_metric(data: dict, config: str, rate: float, key: str) -> float | None:
|
||||
values = [
|
||||
num(r[key])
|
||||
for r in data["runs"]
|
||||
if r["config"] == config
|
||||
and r["phase"] == "fixed"
|
||||
and num(r["rate"]) == rate
|
||||
and num(r.get(key)) is not None
|
||||
]
|
||||
return statistics.median(values) if values else None
|
||||
|
||||
|
||||
def fmt(value: float | None, digits: int = 1) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
return f"{round(value, digits):g}"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--before", required=True)
|
||||
parser.add_argument("--after", required=True)
|
||||
parser.add_argument("--aa")
|
||||
args = parser.parse_args()
|
||||
before, after = load(args.before), load(args.after)
|
||||
aa = load(args.aa) if args.aa else None
|
||||
problems = check_methodology(before, after)
|
||||
if aa:
|
||||
problems += [f"A/A: {p}" for p in check_methodology(before, aa)]
|
||||
if aa["meta"]["code"]["src_sha256"] != before["meta"]["code"]["src_sha256"]:
|
||||
problems.append("A/A label has different code than BEFORE (src_sha256)")
|
||||
if problems:
|
||||
raise SystemExit("labels are not comparable: " + "; ".join(problems))
|
||||
|
||||
factor = before["meta"]["profile"]["ladder_factor"]
|
||||
two_steps = math.sqrt(factor)
|
||||
sb, sa = stats_by_config(before), stats_by_config(after)
|
||||
common = [c for c in before["meta"]["profile"]["configs"] if c in sb and c in sa]
|
||||
noise = None
|
||||
if aa:
|
||||
saa = stats_by_config(aa)
|
||||
deltas = [
|
||||
abs(saa[c]["median"] - sb[c]["median"]) / sb[c]["median"]
|
||||
for c in common
|
||||
if c in saa and sb[c]["median"]
|
||||
]
|
||||
noise = max(deltas) if deltas else None
|
||||
|
||||
same_code = (
|
||||
before["meta"]["code"]["src_sha256"] == after["meta"]["code"]["src_sha256"]
|
||||
)
|
||||
lines = [
|
||||
f"# Сравнение: {args.before} → {args.after}",
|
||||
"",
|
||||
f"- Код «до»: src_sha256 `{before['meta']['code']['src_sha256'][:16]}`; «после»: "
|
||||
f"`{after['meta']['code']['src_sha256'][:16]}`"
|
||||
+ (" — **код одинаковый: это A/A, а не изменение**" if same_code else "")
|
||||
+ ".",
|
||||
f"- Пул приложения «до»: {before['meta']['image'].get('app_defaults')}; «после»: "
|
||||
f"{after['meta']['image'].get('app_defaults')}.",
|
||||
f"- Шум A/A ({args.aa}): "
|
||||
+ (
|
||||
f"наибольшее |Δ медианы R*| = {noise * 100:.1f} %"
|
||||
if noise is not None
|
||||
else "не задан"
|
||||
)
|
||||
+ ".",
|
||||
f"- Порог эффекта: разбросы не пересекаются, |Δ| > шума и R* после / до ≥ "
|
||||
f"{two_steps:.3f} (две ступени сетки поиска; одна ступень — {factor**0.25:.3f}).",
|
||||
"",
|
||||
"| Конфигурация | R* до, медиана (min–max) | R* после | Δ | Вывод | p95 до / после на фиксированных, мс | API мкс/запрос до / после | PostgreSQL мкс/запрос на R* до / после | Новых сессий за окно на первой нарушившей до / после | Соединений на R* до / после |",
|
||||
"|---|---|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
rates = before["meta"]["profile"]["fixed_rates"]
|
||||
for config in common:
|
||||
b, a = sb[config], sa[config]
|
||||
delta = (a["median"] - b["median"]) / b["median"] if b["median"] else None
|
||||
overlap = not (a["min"] > b["max"] or a["max"] < b["min"])
|
||||
ratio = a["median"] / b["median"] if b["median"] else None
|
||||
if delta is None:
|
||||
verdict = "нет данных"
|
||||
elif overlap:
|
||||
verdict = "шум (разбросы пересекаются)"
|
||||
elif noise is not None and abs(delta) <= noise:
|
||||
verdict = "шум (в пределах A/A)"
|
||||
elif ratio is not None and (ratio >= two_steps or ratio <= 1 / two_steps):
|
||||
verdict = "эффект"
|
||||
else:
|
||||
verdict = "в пределах разрешения поиска"
|
||||
p95 = " ; ".join(
|
||||
f"{r:g}: {fmt(fixed_metric(before, config, r, 'p95_ms'))} / {fmt(fixed_metric(after, config, r, 'p95_ms'))}"
|
||||
for r in rates
|
||||
)
|
||||
api_us = (
|
||||
f"{fmt(fixed_metric(before, config, rates[-1], 'api_cpu_us_per_req'))} / "
|
||||
f"{fmt(fixed_metric(after, config, rates[-1], 'api_cpu_us_per_req'))}"
|
||||
)
|
||||
db_us = (
|
||||
f"{fmt(step_metric(before, config, 'star', 'db_cpu_us_per_req'))} / "
|
||||
f"{fmt(step_metric(after, config, 'star', 'db_cpu_us_per_req'))}"
|
||||
)
|
||||
sessions = (
|
||||
f"{fmt(step_metric(before, config, 'fail', 'pg_new_sessions'), 0)} / "
|
||||
f"{fmt(step_metric(after, config, 'fail', 'pg_new_sessions'), 0)}"
|
||||
)
|
||||
conns = (
|
||||
f"{fmt(step_metric(before, config, 'star', 'pg_conn_max'), 0)} / "
|
||||
f"{fmt(step_metric(after, config, 'star', 'pg_conn_max'), 0)}"
|
||||
)
|
||||
lines.append(
|
||||
f"| {config} | {b['median']:g} ({b['min']:g}–{b['max']:g}) | "
|
||||
f"{a['median']:g} ({a['min']:g}–{a['max']:g}) | "
|
||||
f"{fmt(delta * 100 if delta is not None else None)} % | {verdict} | {p95} | "
|
||||
f"{api_us} | {db_us} | {sessions} | {conns} |"
|
||||
)
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
path = OUT / f"{args.before}__{args.after}.md"
|
||||
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
print("\n".join(lines))
|
||||
print(f"\nwritten: {path.relative_to(ROOT)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,119 @@
|
||||
"""Загрузчик объектов профиля reconcile: PUT в S3 стенда мимо API.
|
||||
|
||||
Запускается исключительно внутри `vlad-bench-fill`, контейнера на образе
|
||||
стенда: botocore там уже есть, потому что от него зависит aiobotocore, а
|
||||
пакет `src` скрипт не трогает. Номера, ключи и байты берутся из
|
||||
`bench/reconcile.py` — `bench/run.py` кладёт его рядом через `docker cp`.
|
||||
Ключи пользователя API приходят только из окружения контейнера
|
||||
(`S3_ACCESS_KEY_ID` и `S3_SECRET_ACCESS_KEY`, передаются как `docker run
|
||||
-e <имя>`, без значения в командной строке) и никуда не печатаются.
|
||||
|
||||
Пишутся те номера из [--start, --end), которым модель назначает объект.
|
||||
Работа делится между процессами, у каждого свой клиент и пул потоков. На
|
||||
выходе одна JSON-строка с полями objects, put_per_s и seconds; хотя бы
|
||||
один неудачный PUT даёт код 1.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from multiprocessing import Pool
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
import reconcile # noqa: E402
|
||||
|
||||
KEY_ENV = ("S3_ACCESS_KEY_ID", "S3_SECRET_ACCESS_KEY")
|
||||
|
||||
|
||||
def client(endpoint: str, threads: int):
|
||||
import botocore.session # noqa: PLC0415 - только в контейнере стенда
|
||||
from botocore.config import Config # noqa: PLC0415
|
||||
|
||||
return botocore.session.get_session().create_client(
|
||||
"s3",
|
||||
endpoint_url=endpoint,
|
||||
region_name="us-east-1",
|
||||
aws_access_key_id=os.environ[KEY_ENV[0]],
|
||||
aws_secret_access_key=os.environ[KEY_ENV[1]],
|
||||
config=Config(
|
||||
s3={"addressing_style": "path"},
|
||||
retries={"total_max_attempts": 3},
|
||||
max_pool_connections=threads,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def put_chunk(job: tuple[dict, list[int]]) -> int:
|
||||
"""Тело рабочего процесса: создать клиента, раздать номера потокам, вернуть их число."""
|
||||
args, indices = job
|
||||
s3 = client(args["endpoint"], args["threads"])
|
||||
|
||||
def put(index: int) -> None:
|
||||
attachment_id = reconcile.object_id(args["seed"], index)
|
||||
s3.put_object(
|
||||
Bucket=args["bucket"],
|
||||
Key=reconcile.object_key(args["namespace"], attachment_id),
|
||||
Body=reconcile.content(args["seed"], index, args["size"]),
|
||||
ContentType=reconcile.CONTENT_TYPE,
|
||||
)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=args["threads"]) as pool:
|
||||
for _ in pool.map(put, indices):
|
||||
pass
|
||||
return len(indices)
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--endpoint", required=True)
|
||||
parser.add_argument("--bucket", required=True)
|
||||
parser.add_argument("--namespace", required=True)
|
||||
parser.add_argument("--seed", type=int, required=True)
|
||||
parser.add_argument("--start", type=int, required=True)
|
||||
parser.add_argument("--end", type=int, required=True)
|
||||
parser.add_argument("--size", type=int, required=True)
|
||||
parser.add_argument("--force", choices=(reconcile.FORCE_OK,))
|
||||
parser.add_argument("--processes", type=int, default=3)
|
||||
parser.add_argument("--threads", type=int, default=16)
|
||||
args = parser.parse_args(argv)
|
||||
if not Path("/.dockerenv").exists():
|
||||
print(json.dumps({"error": "run only in the bench container"}), flush=True)
|
||||
return 2
|
||||
if any(not os.environ.get(name) for name in KEY_ENV):
|
||||
print(json.dumps({"error": f"{' and '.join(KEY_ENV)} required"}), flush=True)
|
||||
return 2
|
||||
indices = reconcile.object_indices(args.start, args.end, args.force)
|
||||
parts = [indices[i :: args.processes] for i in range(args.processes)]
|
||||
shared = {
|
||||
key: getattr(args, key)
|
||||
for key in ("endpoint", "bucket", "namespace", "seed", "size", "threads")
|
||||
}
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with Pool(args.processes) as pool:
|
||||
done = sum(pool.map(put_chunk, [(shared, p) for p in parts if p]))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# печатаем лишь тип: сообщение botocore может содержать URL и заголовки
|
||||
print(json.dumps({"error": f"put failed ({type(exc).__name__})"}), flush=True)
|
||||
return 1
|
||||
seconds = time.monotonic() - started
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"objects": done,
|
||||
"put_per_s": round(done / seconds, 1) if seconds else 0.0,
|
||||
"seconds": round(seconds, 2),
|
||||
},
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0 if done == len(indices) else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
+1457
File diff suppressed because it is too large
Load Diff
+412
@@ -0,0 +1,412 @@
|
||||
"""Графики серий трека 1: SVG на стандартной библиотеке.
|
||||
|
||||
uv run python bench/plots.py --out scaling scaling-a scaling-b scaling-db
|
||||
→ evidence/plots/<out>/: rstar-by-config.svg, rstar-by-replicas.svg,
|
||||
summary.md (R*, эффективность, точка насыщения)
|
||||
|
||||
`run.py` после серии зовёт `render_label()`: в каталоге метки —
|
||||
`p95-by-rate.svg` (p95 от поданной интенсивности по ячейкам) и
|
||||
`cpu-by-component.svg` (CPU компонентов на ступени R*). Числа берутся из
|
||||
`searches.csv` и `runs.csv` меток как есть.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import html
|
||||
import math
|
||||
import re
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SCALING = ROOT / "evidence" / "scaling"
|
||||
PLOTS = ROOT / "evidence" / "plots"
|
||||
COLORS = (
|
||||
"#2f6fbf",
|
||||
"#d0752a",
|
||||
"#3c9a5f",
|
||||
"#b8404a",
|
||||
"#7a5cb5",
|
||||
"#8c6d3f",
|
||||
"#c25f9c",
|
||||
"#5f8f9c",
|
||||
)
|
||||
COMPONENTS = (
|
||||
("api_cpu_cores", "API (сумма реплик)", "#2f6fbf"),
|
||||
("db_cpu_cores", "PostgreSQL", "#b8404a"),
|
||||
("lb_cpu_cores", "HAProxy", "#3c9a5f"),
|
||||
("cache_cpu_cores", "Valkey", "#7a5cb5"),
|
||||
("gen_cpu_cores", "генератор", "#8c6d3f"),
|
||||
)
|
||||
CONFIG_RE = re.compile(r"(\d+)x([\d.]+)(?:-pg([\d.]+))?")
|
||||
|
||||
|
||||
def read_csv(path: Path) -> list[dict]:
|
||||
if not path.exists():
|
||||
return []
|
||||
with path.open(encoding="utf-8") as handle:
|
||||
return list(csv.DictReader(handle))
|
||||
|
||||
|
||||
def num(value) -> float | None:
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def nice_step(raw: float) -> float:
|
||||
"""Круглый шаг сетки не меньше raw: 1, 2, 2,5, 5 или 10 × 10^k."""
|
||||
magnitude = 10 ** math.floor(math.log10(raw))
|
||||
return next(m * magnitude for m in (1, 2, 2.5, 5, 10) if m * magnitude >= raw)
|
||||
|
||||
|
||||
def nice_ticks(top: float, count: int = 6) -> list[float]:
|
||||
step = nice_step(top / count) if top > 0 else 1.0
|
||||
return [round(step * i, 6) for i in range(int(top / step) + 1)]
|
||||
|
||||
|
||||
class Svg:
|
||||
"""Поле графика с осями: x и y в единицах данных."""
|
||||
|
||||
def __init__(self, width: int, height: int, title: str) -> None:
|
||||
self.width, self.height = width, height
|
||||
# справа — место под легенду: имена меток длинные
|
||||
self.left, self.right, self.top, self.bottom = 70, 300, 40, 50
|
||||
self.parts = [
|
||||
f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" '
|
||||
f'viewBox="0 0 {width} {height}" font-family="sans-serif" font-size="12">',
|
||||
f'<rect width="{width}" height="{height}" fill="white"/>',
|
||||
f'<text x="{width / 2}" y="22" text-anchor="middle" font-size="14">'
|
||||
f"{html.escape(title)}</text>",
|
||||
]
|
||||
self.x0 = self.x1 = self.y0 = self.y1 = 0.0
|
||||
self.ystep = 1.0
|
||||
self.legend_row = 0
|
||||
|
||||
def scale(self, x0: float, x1: float, y0: float, y1: float) -> None:
|
||||
"""Ось y — от y0 до круглого значения не меньше y1 (шаг сетки
|
||||
1, 2, 2,5 или 5 × 10^k, пять–шесть делений)."""
|
||||
self.ystep = nice_step(max(y1 - y0, 1e-9) / 5)
|
||||
top = y0 + self.ystep * math.ceil(max(y1 - y0, 1e-9) / self.ystep)
|
||||
self.x0, self.x1, self.y0, self.y1 = (
|
||||
x0,
|
||||
x1 if x1 > x0 else x0 + 1,
|
||||
y0,
|
||||
top,
|
||||
)
|
||||
|
||||
def px(self, x: float) -> float:
|
||||
span = self.width - self.left - self.right
|
||||
return self.left + (x - self.x0) / (self.x1 - self.x0) * span
|
||||
|
||||
def py(self, y: float) -> float:
|
||||
span = self.height - self.top - self.bottom
|
||||
return self.height - self.bottom - (y - self.y0) / (self.y1 - self.y0) * span
|
||||
|
||||
def axes(self, xlabel: str, ylabel: str, xticks: list[tuple[float, str]]) -> None:
|
||||
bottom, left = self.height - self.bottom, self.left
|
||||
right = self.width - self.right
|
||||
self.parts.append(
|
||||
f'<line x1="{left}" y1="{bottom}" x2="{right}" y2="{bottom}" stroke="#333"/>'
|
||||
)
|
||||
self.parts.append(
|
||||
f'<line x1="{left}" y1="{self.top}" x2="{left}" y2="{bottom}" stroke="#333"/>'
|
||||
)
|
||||
for x, text in xticks:
|
||||
self.parts.append(
|
||||
f'<text x="{self.px(x):.1f}" y="{bottom + 16}" text-anchor="middle">{html.escape(text)}</text>'
|
||||
)
|
||||
for index in range(round((self.y1 - self.y0) / self.ystep) + 1):
|
||||
value = round(self.y0 + self.ystep * index, 6)
|
||||
y = self.py(value)
|
||||
self.parts.append(
|
||||
f'<line x1="{left}" y1="{y:.1f}" x2="{right}" y2="{y:.1f}" stroke="#eee"/>'
|
||||
)
|
||||
self.parts.append(
|
||||
f'<text x="{left - 6}" y="{y + 4:.1f}" text-anchor="end">{value:g}</text>'
|
||||
)
|
||||
self.parts.append(
|
||||
f'<text x="{(left + right) / 2}" y="{self.height - 12}" text-anchor="middle">{html.escape(xlabel)}</text>'
|
||||
)
|
||||
self.parts.append(
|
||||
f'<text x="16" y="{(self.top + bottom) / 2}" text-anchor="middle" '
|
||||
f'transform="rotate(-90 16 {(self.top + bottom) / 2})">{html.escape(ylabel)}</text>'
|
||||
)
|
||||
|
||||
def line(
|
||||
self, points: list[tuple[float, float]], color: str, dash: str = ""
|
||||
) -> None:
|
||||
if not points:
|
||||
return
|
||||
path = " ".join(f"{self.px(x):.1f},{self.py(y):.1f}" for x, y in points)
|
||||
extra = f' stroke-dasharray="{dash}"' if dash else ""
|
||||
self.parts.append(
|
||||
f'<polyline points="{path}" fill="none" stroke="{color}" stroke-width="2"{extra}/>'
|
||||
)
|
||||
|
||||
def dot(self, x: float, y: float, color: str, hollow: bool = False) -> None:
|
||||
fill = "white" if hollow else color
|
||||
self.parts.append(
|
||||
f'<circle cx="{self.px(x):.1f}" cy="{self.py(y):.1f}" r="3.5" fill="{fill}" stroke="{color}"/>'
|
||||
)
|
||||
|
||||
def whisker(self, x: float, lo: float, hi: float, color: str) -> None:
|
||||
self.parts.append(
|
||||
f'<line x1="{self.px(x):.1f}" y1="{self.py(lo):.1f}" x2="{self.px(x):.1f}" '
|
||||
f'y2="{self.py(hi):.1f}" stroke="{color}" stroke-width="1.5"/>'
|
||||
)
|
||||
|
||||
def bar(self, x: float, width: float, y0: float, y1: float, color: str) -> None:
|
||||
left = self.px(x - width / 2)
|
||||
right = self.px(x + width / 2)
|
||||
self.parts.append(
|
||||
f'<rect x="{left:.1f}" y="{self.py(y1):.1f}" width="{right - left:.1f}" '
|
||||
f'height="{self.py(y0) - self.py(y1):.1f}" fill="{color}" opacity="0.85"/>'
|
||||
)
|
||||
|
||||
def legend(self, text: str, color: str, dash: str = "") -> None:
|
||||
x = self.width - self.right + 14
|
||||
y = self.top + 14 + 18 * self.legend_row
|
||||
extra = f' stroke-dasharray="{dash}"' if dash else ""
|
||||
self.parts.append(
|
||||
f'<line x1="{x}" y1="{y - 4}" x2="{x + 18}" y2="{y - 4}" stroke="{color}" stroke-width="3"{extra}/>'
|
||||
)
|
||||
self.parts.append(f'<text x="{x + 24}" y="{y}">{html.escape(text)}</text>')
|
||||
self.legend_row += 1
|
||||
|
||||
def save(self, path: Path) -> None:
|
||||
path.write_text("\n".join([*self.parts, "</svg>"]) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def render_label(folder: Path) -> None:
|
||||
"""Графики одной метки серии: p95 от интенсивности, CPU на ступени R*."""
|
||||
runs = read_csv(folder / "runs.csv")
|
||||
searches = read_csv(folder / "searches.csv")
|
||||
if not runs:
|
||||
return
|
||||
configs = list(dict.fromkeys(row["config"] for row in runs))
|
||||
rates = [num(r["rate"]) or 0 for r in runs]
|
||||
p95s = [min(num(r["p95_ms"]) or 0, 1000) for r in runs]
|
||||
svg = Svg(1000, 440, f"p95 успешных от поданной интенсивности: {folder.name}")
|
||||
svg.scale(0, max(rates) * 1.05, 0, max(max(p95s) * 1.1, 250))
|
||||
ticks = [(x, f"{x:g}") for x in nice_ticks(max(rates))]
|
||||
svg.axes("запросов в секунду (по расписанию)", "p95, мс (≤ 1000)", ticks)
|
||||
svg.line([(0, 200), (max(rates) * 1.05, 200)], "#999", "4 4")
|
||||
for index, config in enumerate(configs):
|
||||
color = COLORS[index % len(COLORS)]
|
||||
cells = sorted(
|
||||
{(r["cell"], r["repeat"]) for r in runs if r["config"] == config}
|
||||
)
|
||||
for cell, _ in cells:
|
||||
points = sorted(
|
||||
(num(r["rate"]) or 0, min(num(r["p95_ms"]) or 1000, 1000))
|
||||
for r in runs
|
||||
if r["cell"] == cell
|
||||
)
|
||||
svg.line(points, color)
|
||||
for r in runs:
|
||||
if r["cell"] == cell:
|
||||
svg.dot(
|
||||
num(r["rate"]) or 0,
|
||||
min(num(r["p95_ms"]) or 1000, 1000),
|
||||
color,
|
||||
hollow=r["slo_pass"] != "True",
|
||||
)
|
||||
svg.legend(config, color)
|
||||
svg.legend("SLO 200 мс", "#999", "4 4")
|
||||
svg.save(folder / "p95-by-rate.svg")
|
||||
|
||||
# CPU компонентов на ступени R* каждой ячейки (медиана по повторам)
|
||||
stacks = []
|
||||
for config in configs:
|
||||
values = {key: [] for key, _, _ in COMPONENTS}
|
||||
for search in [s for s in searches if s["config"] == config]:
|
||||
star = num(search["r_star"])
|
||||
row = next(
|
||||
(
|
||||
r
|
||||
for r in runs
|
||||
if r["cell"] == search["cell"]
|
||||
and r["phase"] != "fixed"
|
||||
and num(r["rate"]) == star
|
||||
),
|
||||
None,
|
||||
)
|
||||
if row is None:
|
||||
continue
|
||||
for key, _, _ in COMPONENTS:
|
||||
if num(row.get(key)) is not None:
|
||||
values[key].append(num(row[key]))
|
||||
stacks.append(
|
||||
(config, {k: statistics.median(v) if v else 0 for k, v in values.items()})
|
||||
)
|
||||
if not stacks:
|
||||
return
|
||||
top = max(sum(s.values()) for _, s in stacks) or 1
|
||||
svg = Svg(
|
||||
1000, 440, f"CPU компонентов на ступени R* (медиана), ядер: {folder.name}"
|
||||
)
|
||||
svg.scale(0, len(stacks) + 1, 0, top * 1.15)
|
||||
svg.axes("конфигурация", "ядер", [(i + 1, c) for i, (c, _) in enumerate(stacks)])
|
||||
for index, (_, stack) in enumerate(stacks):
|
||||
base = 0.0
|
||||
for key, _, color in COMPONENTS:
|
||||
svg.bar(index + 1, 0.5, base, base + stack[key], color)
|
||||
base += stack[key]
|
||||
for _, name, color in COMPONENTS:
|
||||
svg.legend(name, color)
|
||||
svg.save(folder / "cpu-by-component.svg")
|
||||
|
||||
|
||||
def label_rows(label: str) -> list[dict]:
|
||||
rows = read_csv(SCALING / label / "searches.csv")
|
||||
for row in rows:
|
||||
row["label"] = label
|
||||
return rows
|
||||
|
||||
|
||||
def summarize(rows: list[dict]) -> dict[tuple[str, str], dict]:
|
||||
"""(метка, конфигурация) → медиана, min, max R*, реплик, CPU."""
|
||||
out: dict[tuple[str, str], dict] = {}
|
||||
for row in rows:
|
||||
key = (row["label"], row["config"])
|
||||
entry = out.setdefault(
|
||||
key,
|
||||
{
|
||||
"values": [],
|
||||
"replicas": int(row["replicas"]),
|
||||
"api_cpus": num(row["api_cpus"]),
|
||||
"db_cpus": num(row["db_cpus"]),
|
||||
"signals": {},
|
||||
},
|
||||
)
|
||||
if num(row["r_star"]) is not None:
|
||||
entry["values"].append(num(row["r_star"]))
|
||||
for signal in (row["signals_at_fail"] or "").split(";"):
|
||||
if signal:
|
||||
entry["signals"][signal] = entry["signals"].get(signal, 0) + 1
|
||||
for entry in out.values():
|
||||
values = entry["values"] or [0]
|
||||
entry["median"] = statistics.median(values)
|
||||
entry["min"], entry["max"] = min(values), max(values)
|
||||
return out
|
||||
|
||||
|
||||
def saturation(points: list[tuple[int, float]]) -> int | None:
|
||||
"""Наименьшее k, после которого следующая реплика даёт < +10 % R*."""
|
||||
points = sorted(points)
|
||||
for (k, r), (_, r_next) in zip(points, points[1:], strict=False):
|
||||
if r_next < 1.10 * r:
|
||||
return k
|
||||
return None
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--out", required=True)
|
||||
parser.add_argument("labels", nargs="+")
|
||||
args = parser.parse_args()
|
||||
if not re.fullmatch(r"[A-Za-z0-9._-]{1,48}", args.out):
|
||||
raise SystemExit("--out must be a simple name")
|
||||
rows = [row for label in args.labels for row in label_rows(label)]
|
||||
if not rows:
|
||||
raise SystemExit("no searches.csv in the given labels")
|
||||
stats = summarize(rows)
|
||||
folder = PLOTS / args.out
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# R* по конфигурациям всех меток: медиана и min–max
|
||||
items = list(stats.items())
|
||||
top = max(entry["max"] for _, entry in items) or 1
|
||||
svg = Svg(
|
||||
max(1000, 90 * len(items) + 380), 460, "R* по конфигурациям: медиана и min–max"
|
||||
)
|
||||
svg.scale(0, len(items) + 1, 0, top * 1.15)
|
||||
svg.axes(
|
||||
"метка / конфигурация",
|
||||
"R*, запросов/с",
|
||||
[(i + 1, cfg) for i, ((_, cfg), _) in enumerate(items)],
|
||||
)
|
||||
labels = list(dict.fromkeys(label for (label, _), _ in items))
|
||||
for index, ((label, _), entry) in enumerate(items):
|
||||
color = COLORS[labels.index(label) % len(COLORS)]
|
||||
svg.bar(index + 1, 0.55, 0, entry["median"], color)
|
||||
svg.whisker(index + 1, entry["min"], entry["max"], "#222")
|
||||
for label in labels:
|
||||
svg.legend(label, COLORS[labels.index(label) % len(COLORS)])
|
||||
svg.save(folder / "rstar-by-config.svg")
|
||||
|
||||
# R* от числа реплик по 1 CPU — по меткам, где такие конфигурации есть
|
||||
svg = Svg(
|
||||
860, 440, "R* от числа реплик (по 1 CPU): медиана, min–max, линейный рост"
|
||||
)
|
||||
curves = {}
|
||||
for (label, config), entry in items:
|
||||
match = CONFIG_RE.fullmatch(config)
|
||||
if match and float(match.group(2)) == 1.0:
|
||||
curves.setdefault((label, match.group(3) or "2"), []).append(
|
||||
(entry["replicas"], entry)
|
||||
)
|
||||
if curves:
|
||||
max_k = max(k for points in curves.values() for k, _ in points)
|
||||
# линия линейного роста от 1 × 1 тоже должна поместиться
|
||||
ideal = [
|
||||
e["median"] * max_k
|
||||
for points in curves.values()
|
||||
for k, e in points
|
||||
if k == 1
|
||||
]
|
||||
svg.scale(0, max_k + 0.5, 0, max([top, *ideal]) * 1.05)
|
||||
svg.axes("реплик", "R*, запросов/с", [(k, str(k)) for k in range(1, max_k + 1)])
|
||||
for index, ((label, pg), points) in enumerate(sorted(curves.items())):
|
||||
color = COLORS[index % len(COLORS)]
|
||||
points.sort(key=lambda p: p[0])
|
||||
svg.line([(k, e["median"]) for k, e in points], color)
|
||||
for k, entry in points:
|
||||
svg.dot(k, entry["median"], color)
|
||||
svg.whisker(k, entry["min"], entry["max"], color)
|
||||
one = next((e["median"] for k, e in points if k == 1), None)
|
||||
if one:
|
||||
svg.line([(1, one), (max_k, one * max_k)], color, "3 4")
|
||||
svg.legend(f"{label}, PostgreSQL {pg} CPU", color)
|
||||
svg.save(folder / "rstar-by-replicas.svg")
|
||||
|
||||
lines = [
|
||||
f"# Графики трека 1: {args.out}",
|
||||
"",
|
||||
f"Метки: {', '.join(args.labels)}. Числа — `searches.csv` меток.",
|
||||
"",
|
||||
"| Метка | Конфигурация | Реплик | CPU реплики | PostgreSQL CPU | R* медиана | min–max | Эффективность к 1 × 1 метки | Признаки на первой нарушившей |",
|
||||
"|---|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
for (label, config), entry in items:
|
||||
one = stats.get((label, "1x1"))
|
||||
eff = (
|
||||
f"{entry['median'] / (entry['replicas'] * one['median']):.2f}"
|
||||
if one and one["median"] and config != "1x1"
|
||||
else "—"
|
||||
)
|
||||
signals = (
|
||||
", ".join(f"{k} ×{v}" for k, v in sorted(entry["signals"].items())) or "—"
|
||||
)
|
||||
lines.append(
|
||||
f"| {label} | {config} | {entry['replicas']} | {entry['api_cpus']:g} | "
|
||||
f"{entry['db_cpus']:g} | {entry['median']:g} | {entry['min']:g}–{entry['max']:g} | "
|
||||
f"{eff} | {signals} |"
|
||||
)
|
||||
lines += ["", "## Точка насыщения", ""]
|
||||
for (label, pg), points in sorted(curves.items()):
|
||||
k = saturation([(k, e["median"]) for k, e in points])
|
||||
verdict = (
|
||||
f"после {k} реплик(и) следующая даёт меньше +10 % R*"
|
||||
if k
|
||||
else "в пределах снятых конфигураций рост не останавливается (≥ +10 % на реплику)"
|
||||
)
|
||||
lines.append(f"- {label}, PostgreSQL {pg} CPU: {verdict}.")
|
||||
(folder / "summary.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
print(f"written: {folder.relative_to(ROOT)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+1482
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,952 @@
|
||||
"""Профиль reconcile: набор, план, разбор и сводка без docker. Только stdlib.
|
||||
|
||||
Сам запуск проходов делает `run.py` (профили `reconcile` и
|
||||
`reconcile-quick`), а этот модуль содержит только расчёты без побочных
|
||||
эффектов: модель набора (по номеру и зерну — id, байты и класс
|
||||
объекта), строки CSV метаданных для `COPY`, план проходов кругами,
|
||||
разбор итоговой строки уборщика `reconcile pass finished …`, сверка
|
||||
найденного с моделью, окна фоновой нагрузки, строки `passes.csv` и
|
||||
`windows.csv`, вердикты H1–H5 (гипотезы внесены в `REPORT.md` заранее)
|
||||
и `summary.md`. Проверка без стенда — запуск файла с `--selftest`.
|
||||
|
||||
Этот же модуль импортирует `bench/fill_objects.py` внутри контейнера на
|
||||
образе стенда, поэтому на верхнем уровне допустим только stdlib.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import random
|
||||
import re
|
||||
import statistics
|
||||
import sys
|
||||
import uuid
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
|
||||
# остаток i mod PERIOD задаёт, что сломано у номера i: 0 — в S3 есть файл,
|
||||
# а строки нет; 1 — строка ready есть, файла нет; 2 — size_bytes на 1
|
||||
# больше настоящего; 3 — pending, брошенный вместе с файлом. Каждый класс
|
||||
# занимает 1/400 = 0,25 % набора
|
||||
PERIOD = 400
|
||||
CLASSES = ("orphan_object", "missing_object", "size_mismatch", "stale_pending")
|
||||
# найденное уборщиком (src/services/reconcile.py, KINDS)
|
||||
KINDS = (
|
||||
"orphan_object",
|
||||
"stale_pending",
|
||||
"missing_object",
|
||||
"size_mismatch",
|
||||
"checksum_mismatch",
|
||||
"restored_object",
|
||||
"foreign_object",
|
||||
)
|
||||
# переопределить класс сразу для всего диапазона: либо все номера исправны,
|
||||
# либо у всех нет файла — так строится набор для проверки отказа по пределу
|
||||
FORCE_OK = "ok"
|
||||
FORCE_MISSING = "missing_object"
|
||||
CONTENT_TYPE = "image/png"
|
||||
KIB = 1024
|
||||
MIB = 2**20
|
||||
COPY_COLUMNS = (
|
||||
"id",
|
||||
"chat_id",
|
||||
"owner_id",
|
||||
"filename",
|
||||
"content_type",
|
||||
"size_bytes",
|
||||
"sha256",
|
||||
"object_key",
|
||||
"status",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
)
|
||||
# параметры ячейки: strategy, deep, cpu уборщика ("stand" = 0,25 CPU как на
|
||||
# рабочем стенде, "fast" = 1 CPU для сравнения), rate ("default" берёт
|
||||
# RECONCILE_RATE из настроек, число — явный предел), load и apply
|
||||
CELLS: dict[str, dict] = {
|
||||
"batch": {"strategy": "batch", "deep": False, "cpu": "stand", "rate": 0},
|
||||
"head": {"strategy": "head", "deep": False, "cpu": "stand", "rate": 0},
|
||||
"deep": {"strategy": "batch", "deep": True, "cpu": "stand", "rate": 0},
|
||||
"deep-fast": {"strategy": "batch", "deep": True, "cpu": "fast", "rate": 0},
|
||||
"load-stand": {"strategy": "batch", "cpu": "stand", "rate": "default", "load": 1},
|
||||
"load-unlimited": {"strategy": "batch", "cpu": "stand", "rate": 0, "load": 1},
|
||||
"load-fast": {"strategy": "batch", "cpu": "fast", "rate": 0, "load": 1},
|
||||
"load-fast-limited": {
|
||||
"strategy": "batch",
|
||||
"cpu": "fast",
|
||||
"rate": "default",
|
||||
"load": 1,
|
||||
},
|
||||
# отказ по пределу: --apply --max-changes 100 при 1 000 кандидатов
|
||||
"refuse": {"strategy": "batch", "cpu": "stand", "rate": 0, "apply": 1},
|
||||
}
|
||||
LOAD_CELLS = ("load-stand", "load-unlimited", "load-fast", "load-fast-limited")
|
||||
LIMITED_LOAD = ("load-stand", "load-fast-limited")
|
||||
WINDOWS = ("before", "pass", "after")
|
||||
LOAD_KINDS = ("download", "write", "read")
|
||||
FINISHED = "reconcile pass finished"
|
||||
S3_OPERATIONS = ("list", "head", "get", "delete")
|
||||
PHASES = ("list", "rows", "stale", "recheck", "act")
|
||||
ACTIONS = ("cleared_lost", "deleted_object", "marked_lost", "removed_pending")
|
||||
# пороги гипотез — до замеров (REPORT.md, «Трек 2: гипотезы»)
|
||||
H1_SPREAD = 0.30
|
||||
H1_BYTES_PER_KEY = 300
|
||||
H2_TIME_RATIO = 5
|
||||
H2_RSS_GROWTH_MIB = 5
|
||||
H3_DEEP_RATIO = 5
|
||||
H3_FAST_RATIO = 1.5
|
||||
H4_THRESHOLD = 1.5
|
||||
H4_RATE_SHARE = 0.9
|
||||
H5_SPREAD = 0.20
|
||||
|
||||
PASS_COLUMNS = (
|
||||
"pass",
|
||||
"stage",
|
||||
"cell",
|
||||
"repeat",
|
||||
"order",
|
||||
"n",
|
||||
"size",
|
||||
"strategy",
|
||||
"deep",
|
||||
"apply",
|
||||
"rate",
|
||||
"cleaner_cpus",
|
||||
"nano_cpus",
|
||||
"exit_code",
|
||||
"refused",
|
||||
"listed_objects",
|
||||
"checked_rows",
|
||||
*(f"found_{kind}" for kind in KINDS),
|
||||
*(f"fixed_{action}" for action in ACTIONS),
|
||||
"remaining",
|
||||
*(f"s3_{op}" for op in S3_OPERATIONS),
|
||||
"listed_keys",
|
||||
"read_bytes",
|
||||
"db_queries",
|
||||
*(f"phase_{name}_ms" for name in PHASES),
|
||||
"cpu_ms",
|
||||
"maxrss_kib",
|
||||
"took_ms",
|
||||
"ms_per_1k",
|
||||
"cpu_share",
|
||||
"touched_per_s",
|
||||
"s3_cpu_cores",
|
||||
"db_cpu_cores",
|
||||
"api_cpu_cores",
|
||||
"model_ok",
|
||||
"flags",
|
||||
)
|
||||
WINDOW_COLUMNS = (
|
||||
"pass",
|
||||
"cell",
|
||||
"repeat",
|
||||
"window",
|
||||
"kind",
|
||||
"seconds",
|
||||
"requests",
|
||||
"ok",
|
||||
"errors",
|
||||
"errors_5xx",
|
||||
"p50_ms",
|
||||
"p95_ms",
|
||||
"p99_ms",
|
||||
"ratio_p95",
|
||||
)
|
||||
|
||||
|
||||
# --- набор ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def object_class(index: int, force: str | None = None) -> str | None:
|
||||
"""Какое расхождение заложено в номер `index` (None — никакого)."""
|
||||
if force == FORCE_OK:
|
||||
return None
|
||||
if force is not None:
|
||||
return force
|
||||
slot = index % PERIOD
|
||||
return CLASSES[slot] if slot < len(CLASSES) else None
|
||||
|
||||
|
||||
def object_id(seed: int, index: int) -> str:
|
||||
"""Детерминированный id: первые 16 байт SHA-256 от `<зерно>:<номер>` как UUID4."""
|
||||
digest = hashlib.sha256(f"{seed}:{index}".encode()).digest()
|
||||
return str(uuid.UUID(bytes=digest[:16], version=4))
|
||||
|
||||
|
||||
def content(seed: int, index: int, size: int) -> bytes:
|
||||
"""Байты объекта: 32-байтный хэш от `<зерно>:<номер>:c`, размноженный до `size`."""
|
||||
block = hashlib.sha256(f"{seed}:{index}:c".encode()).digest()
|
||||
return (block * (size // len(block) + 1))[:size]
|
||||
|
||||
|
||||
def object_key(namespace: str, attachment_id: str) -> str:
|
||||
"""Ключ оригинала — как `AttachmentObjects.key_of` (src/storage/objects.py)."""
|
||||
return f"{namespace}/attachments/{attachment_id}/original"
|
||||
|
||||
|
||||
def has_object(kind: str | None) -> bool:
|
||||
return kind != "missing_object"
|
||||
|
||||
|
||||
def has_row(kind: str | None) -> bool:
|
||||
return kind != "orphan_object"
|
||||
|
||||
|
||||
def object_indices(start: int, end: int, force: str | None = None) -> list[int]:
|
||||
"""Какие номера из [start, end) загрузчик должен положить в S3."""
|
||||
return [i for i in range(start, end) if has_object(object_class(i, force))]
|
||||
|
||||
|
||||
def rows_csv(
|
||||
*,
|
||||
seed: int,
|
||||
start: int,
|
||||
end: int,
|
||||
size: int,
|
||||
namespace: str,
|
||||
chat_id: str,
|
||||
owner_id: str,
|
||||
created_at: str,
|
||||
force: str | None = None,
|
||||
) -> str:
|
||||
"""CSV для `COPY attachments` по колонкам COPY_COLUMNS. Статус `ready`
|
||||
получают согласованные номера, номера «нет объекта» и «не тот размер»
|
||||
(у последних `size_bytes` завышен на 1); зависшая загрузка идёт как
|
||||
`pending`, а для «объекта без строки» строка не пишется вовсе.
|
||||
SHA-256 считается от тех же байт, которые запишет загрузчик."""
|
||||
lines = []
|
||||
for index in range(start, end):
|
||||
kind = object_class(index, force)
|
||||
if not has_row(kind):
|
||||
continue
|
||||
attachment_id = object_id(seed, index)
|
||||
digest = hashlib.sha256(content(seed, index, size)).hexdigest()
|
||||
size_bytes = size + 1 if kind == "size_mismatch" else size
|
||||
status = "pending" if kind == "stale_pending" else "ready"
|
||||
lines.append(
|
||||
",".join(
|
||||
(
|
||||
attachment_id,
|
||||
chat_id,
|
||||
owner_id,
|
||||
f"bench-{index}.png",
|
||||
CONTENT_TYPE,
|
||||
str(size_bytes),
|
||||
f"\\x{digest}",
|
||||
object_key(namespace, attachment_id),
|
||||
status,
|
||||
created_at,
|
||||
created_at,
|
||||
),
|
||||
),
|
||||
)
|
||||
return "\n".join(lines) + ("\n" if lines else "")
|
||||
|
||||
|
||||
def class_count(n: int, slot: int) -> int:
|
||||
"""Число номеров из range(n), попадающих в остаток `slot` по PERIOD."""
|
||||
return n // PERIOD + (1 if n % PERIOD > slot else 0)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Model:
|
||||
"""Что должна увидеть сверка: сколько файлов в S3, строк каждого
|
||||
статуса, расхождений каждого класса и полностью исправных пар."""
|
||||
|
||||
objects: int
|
||||
ready_rows: int
|
||||
pending_rows: int
|
||||
found: dict[str, int]
|
||||
consistent: int
|
||||
|
||||
@property
|
||||
def candidates(self) -> int:
|
||||
return sum(self.found.values())
|
||||
|
||||
|
||||
def model(n: int, *, clean: int = 0, missing: int = 0) -> Model:
|
||||
"""Набор из `n` объектов с классами по номеру; набор отказа по пределу
|
||||
— `clean` согласованных и `missing` строк без объекта."""
|
||||
if clean or missing:
|
||||
return Model(
|
||||
objects=clean,
|
||||
ready_rows=clean + missing,
|
||||
pending_rows=0,
|
||||
found={"missing_object": missing} if missing else {},
|
||||
consistent=clean,
|
||||
)
|
||||
found = {kind: class_count(n, slot) for slot, kind in enumerate(CLASSES)}
|
||||
return Model(
|
||||
objects=n - found["missing_object"],
|
||||
ready_rows=n - found["orphan_object"] - found["stale_pending"],
|
||||
pending_rows=found["stale_pending"],
|
||||
found={kind: count for kind, count in found.items() if count},
|
||||
consistent=n - sum(found.values()),
|
||||
)
|
||||
|
||||
|
||||
def downloads(seed: int, n: int, count: int) -> list[str]:
|
||||
"""Случайная (по зерну) выборка исправных вложений, которые будет качать фон."""
|
||||
pool = [i for i in range(n) if object_class(i) is None]
|
||||
chosen = random.Random(seed).sample(pool, min(count, len(pool)))
|
||||
return [object_id(seed, i) for i in chosen]
|
||||
|
||||
|
||||
# --- план --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def plan_passes(
|
||||
stages: Iterable[tuple[str, tuple[str, ...]]], repeats: int, seed: int
|
||||
) -> list[dict]:
|
||||
"""Расписание проходов. Каждый этап повторяется `repeats` кругами; в
|
||||
круге ячейки перемешаны `Random(seed + 100 × номер этапа + r)`, и
|
||||
новый круг не стартует той ячейкой, на которой закончился предыдущий."""
|
||||
plan: list[dict] = []
|
||||
for stage_no, (stage, cells) in enumerate(stages):
|
||||
previous = None
|
||||
for round_no in range(1, repeats + 1):
|
||||
order = list(range(len(cells)))
|
||||
random.Random(seed + stage_no * 100 + round_no).shuffle(order)
|
||||
if len(order) > 1 and order[0] == previous:
|
||||
order[0], order[1] = order[1], order[0]
|
||||
for index in order:
|
||||
cell = cells[index]
|
||||
plan.append(
|
||||
{
|
||||
"pass": f"{stage}-{cell}-r{round_no}",
|
||||
"stage": stage,
|
||||
"cell": cell,
|
||||
"repeat": round_no,
|
||||
"order": len(plan) + 1,
|
||||
},
|
||||
)
|
||||
previous = order[-1]
|
||||
return plan
|
||||
|
||||
|
||||
# --- разбор ------------------------------------------------------------------------
|
||||
|
||||
|
||||
_PAIR = re.compile(r"(\w+)=(\S*)")
|
||||
|
||||
|
||||
def _value(text: str) -> object:
|
||||
if text in ("true", "false"):
|
||||
return text == "true"
|
||||
if text == "None":
|
||||
return None
|
||||
if text.endswith("ms") and text[:-2].isdigit():
|
||||
return int(text[:-2])
|
||||
try:
|
||||
return int(text)
|
||||
except ValueError:
|
||||
pass
|
||||
try:
|
||||
return float(text)
|
||||
except ValueError:
|
||||
return text
|
||||
|
||||
|
||||
def parse_finished(log: str) -> dict | None:
|
||||
"""Итог прохода из журнала уборщика: время строки (секунды эпохи, часы
|
||||
ВМ) и поля `key=value`; `took` — в мс. Нет строки — `None`."""
|
||||
for line in reversed(log.splitlines()):
|
||||
if FINISHED not in line:
|
||||
continue
|
||||
stamp = line.split(" ", 1)[0]
|
||||
fields = {
|
||||
key: _value(value)
|
||||
for key, value in _PAIR.findall(line.split(FINISHED, 1)[1])
|
||||
}
|
||||
fields["ts"] = parse_log_time(stamp)
|
||||
fields["took_ms"] = fields.pop("took", None)
|
||||
return fields
|
||||
return None
|
||||
|
||||
|
||||
def parse_log_time(stamp: str) -> float:
|
||||
"""`2026-10-23T12:00:00.123Z` (src/log.py) → секунды эпохи."""
|
||||
moment = datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%S.%fZ")
|
||||
return moment.replace(tzinfo=UTC).timestamp()
|
||||
|
||||
|
||||
def found_problems(fields: dict, expected: Model, *, deep: bool) -> list[str]:
|
||||
"""Расхождения итога прохода с моделью набора: класс, найдено, ожидалось."""
|
||||
problems = []
|
||||
for kind in KINDS:
|
||||
want = expected.found.get(kind, 0)
|
||||
got = fields.get(f"found_{kind}", 0)
|
||||
if kind == "checksum_mismatch" and not deep:
|
||||
want = 0
|
||||
if got != want:
|
||||
problems.append(f"found_{kind}={got} expected={want}")
|
||||
for name, want in (
|
||||
("listed_objects", expected.objects),
|
||||
("checked_rows", expected.ready_rows),
|
||||
):
|
||||
if fields.get(name) != want:
|
||||
problems.append(f"{name}={fields.get(name)} expected={want}")
|
||||
return problems
|
||||
|
||||
|
||||
def touched(fields: dict) -> int:
|
||||
"""Объектов S3, затронутых проходом: ключи списка и вызовы на объект."""
|
||||
return int(fields.get("listed_keys") or 0) + sum(
|
||||
int(fields.get(f"s3_{op}") or 0) for op in ("head", "get", "delete")
|
||||
)
|
||||
|
||||
|
||||
def pass_row(
|
||||
item: dict,
|
||||
*,
|
||||
n: int,
|
||||
size: int,
|
||||
cell: dict,
|
||||
cleaner_cpus: float,
|
||||
rate: float,
|
||||
fields: dict | None,
|
||||
exit_code: int | None,
|
||||
nano_cpus: int | None,
|
||||
usage: dict,
|
||||
problems: list[str],
|
||||
extra_flags: Iterable[str] = (),
|
||||
) -> dict:
|
||||
"""Строка `passes.csv`. Пометки: `no_result` — итоговой строки нет,
|
||||
`model` — найденное разошлось с моделью, `cpu_limit` — лимит CPU
|
||||
контейнера прохода не тот, `exit` — неожиданный код выхода;
|
||||
`extra_flags` — от оркестратора (`bg_dropped` — генератор фоновой
|
||||
нагрузки не отправил часть запросов по расписанию)."""
|
||||
fields = fields or {}
|
||||
took = fields.get("took_ms")
|
||||
cpu = fields.get("cpu_ms")
|
||||
flags = list(extra_flags)
|
||||
if not fields:
|
||||
flags.append("no_result")
|
||||
if problems:
|
||||
flags.append("model")
|
||||
if nano_cpus is not None and nano_cpus != round(cleaner_cpus * 1e9):
|
||||
flags.append("cpu_limit")
|
||||
# в каждом наборе профиля расхождения есть: dry-run и отказ — код 3
|
||||
if exit_code != 3:
|
||||
flags.append("exit")
|
||||
return {
|
||||
**{key: item[key] for key in ("pass", "stage", "cell", "repeat", "order")},
|
||||
"n": n,
|
||||
"size": size,
|
||||
"strategy": cell["strategy"],
|
||||
"deep": bool(cell.get("deep")),
|
||||
"apply": bool(cell.get("apply")),
|
||||
"rate": rate,
|
||||
"cleaner_cpus": cleaner_cpus,
|
||||
"nano_cpus": nano_cpus,
|
||||
"exit_code": exit_code,
|
||||
**{
|
||||
key: fields.get(key)
|
||||
for key in PASS_COLUMNS
|
||||
if key.startswith(("found_", "fixed_", "s3_", "phase_"))
|
||||
and key not in ("s3_cpu_cores",)
|
||||
},
|
||||
"refused": fields.get("refused"),
|
||||
"listed_objects": fields.get("listed_objects"),
|
||||
"checked_rows": fields.get("checked_rows"),
|
||||
"remaining": fields.get("remaining"),
|
||||
"listed_keys": fields.get("listed_keys"),
|
||||
"read_bytes": fields.get("read_bytes"),
|
||||
"db_queries": fields.get("db_queries"),
|
||||
"cpu_ms": cpu,
|
||||
"maxrss_kib": fields.get("maxrss_kib"),
|
||||
"took_ms": took,
|
||||
"ms_per_1k": round(took / n * 1000, 2) if took is not None and n else None,
|
||||
"cpu_share": round(cpu / took, 3) if cpu is not None and took else None,
|
||||
"touched_per_s": (
|
||||
round(touched(fields) / took * 1000, 1) if fields and took else None
|
||||
),
|
||||
"s3_cpu_cores": usage.get("s3_cpu_cores"),
|
||||
"db_cpu_cores": usage.get("db_cpu_cores"),
|
||||
"api_cpu_cores": usage.get("api_cpu_cores"),
|
||||
"model_ok": not problems,
|
||||
"flags": " ".join(flags + [f"model:{p}" for p in problems[:3]]),
|
||||
}
|
||||
|
||||
|
||||
# --- фоновая нагрузка --------------------------------------------------------------
|
||||
|
||||
|
||||
def window_rows(
|
||||
item: dict,
|
||||
requests: list,
|
||||
bounds: dict[str, tuple[float, float]],
|
||||
stats_fn,
|
||||
) -> list[dict]:
|
||||
"""Строки `windows.csv` прохода под нагрузкой: окно × вид запроса;
|
||||
`ratio_p95` у окна `pass` — p95 прохода / p95 тихих окон (до и после
|
||||
вместе). `stats_fn` — `analysis.window_stats`, `bounds` — мс эпохи."""
|
||||
rows = []
|
||||
for kind in LOAD_KINDS:
|
||||
of_kind = [r for r in requests if r.kind == kind]
|
||||
quiet = [
|
||||
r
|
||||
for name in ("before", "after")
|
||||
for r in of_kind
|
||||
if bounds[name][0] <= r.scheduled < bounds[name][1]
|
||||
]
|
||||
quiet_p95 = stats_fn(quiet, 0, float("inf"))["p95_ms"] if quiet else None
|
||||
for window in WINDOWS:
|
||||
start, end = bounds[window]
|
||||
stats = stats_fn(of_kind, start, end)
|
||||
ratio = None
|
||||
if window == "pass" and stats["p95_ms"] and quiet_p95:
|
||||
ratio = round(stats["p95_ms"] / quiet_p95, 3)
|
||||
rows.append(
|
||||
{
|
||||
"pass": item["pass"],
|
||||
"cell": item["cell"],
|
||||
"repeat": item["repeat"],
|
||||
"window": window,
|
||||
"kind": kind,
|
||||
"seconds": stats["seconds"],
|
||||
"requests": stats["requests"],
|
||||
"ok": stats["ok"],
|
||||
"errors": stats["errors"],
|
||||
"errors_5xx": sum(
|
||||
count
|
||||
for label, count in stats["error_kinds"].items()
|
||||
if label.startswith("5")
|
||||
),
|
||||
"p50_ms": stats["p50_ms"],
|
||||
"p95_ms": stats["p95_ms"],
|
||||
"p99_ms": stats["p99_ms"],
|
||||
"ratio_p95": ratio,
|
||||
},
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
# --- вердикты ----------------------------------------------------------------------
|
||||
|
||||
|
||||
CONFIRMED = "подтверждена"
|
||||
REFUTED = "опровергнута"
|
||||
UNTESTED = "нет данных"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Verdict:
|
||||
hypothesis: str
|
||||
result: str
|
||||
numbers: str
|
||||
|
||||
|
||||
def num(value) -> float | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def med(values: Iterable) -> float | None:
|
||||
present = [v for v in (num(x) for x in values) if v is not None]
|
||||
return statistics.median(present) if present else None
|
||||
|
||||
|
||||
def spread(values: Iterable, digits: int = 0) -> str:
|
||||
present = [v for v in (num(x) for x in values) if v is not None]
|
||||
if not present:
|
||||
return "—"
|
||||
mid = statistics.median(present)
|
||||
if len(present) == 1:
|
||||
return f"{mid:.{digits}f}"
|
||||
return f"{mid:.{digits}f} ({min(present):.{digits}f}–{max(present):.{digits}f})"
|
||||
|
||||
|
||||
def fmt(value: float | None, digits: int = 2) -> str:
|
||||
return "—" if value is None else f"{value:.{digits}f}"
|
||||
|
||||
|
||||
def cell_rows(passes: list[dict], stage: str, cell: str) -> list[dict]:
|
||||
return [
|
||||
p
|
||||
for p in passes
|
||||
if p["stage"] == stage and p["cell"] == cell and not p.get("flags")
|
||||
]
|
||||
|
||||
|
||||
def cell_median(passes: list[dict], stage: str, cell: str, column: str) -> float | None:
|
||||
return med(p.get(column) for p in cell_rows(passes, stage, cell))
|
||||
|
||||
|
||||
def verdict_h1(stages: dict[str, dict], passes: list[dict]) -> Verdict:
|
||||
"""batch линейна: мс на 1 000 объектов по этапам 4 KiB — в пределах
|
||||
±30 %, рост RSS ≤ 300 байт на ключ."""
|
||||
# ряд по N — наборы с классами по номеру; набор отказа по пределу
|
||||
# (строки без объекта сверх набора) — другая модель, в ряд не идёт
|
||||
small = [
|
||||
s
|
||||
for s, info in stages.items()
|
||||
if info["size"] <= 64 * KIB and not info.get("missing")
|
||||
]
|
||||
per_k = {s: cell_median(passes, s, "batch", "ms_per_1k") for s in small}
|
||||
rss = {s: cell_median(passes, s, "batch", "maxrss_kib") for s in small}
|
||||
present = {s: v for s, v in per_k.items() if v}
|
||||
if len(present) < 2:
|
||||
return Verdict("H1", UNTESTED, "нужно ≥ 2 этапов batch с 4 KiB")
|
||||
low, high = min(present.values()), max(present.values())
|
||||
spread_ok = high / low - 1 <= H1_SPREAD
|
||||
first = min(present, key=lambda s: stages[s]["n"])
|
||||
last = max(present, key=lambda s: stages[s]["n"])
|
||||
keys = stages[last]["n"] - stages[first]["n"]
|
||||
growth = None
|
||||
if rss.get(first) and rss.get(last) and keys > 0:
|
||||
growth = (rss[last] - rss[first]) * 1024 / keys
|
||||
rss_ok = growth is not None and growth <= H1_BYTES_PER_KEY
|
||||
numbers = (
|
||||
"мс на 1 000 объектов: "
|
||||
+ ", ".join(f"{stages[s]['n']} — {fmt(v)}" for s, v in present.items())
|
||||
+ f" (разброс {high / low - 1:.0%}, порог {H1_SPREAD:.0%}); рост RSS "
|
||||
f"{fmt(growth, 0)} Б на ключ ({stages[first]['n']} → {stages[last]['n']}, "
|
||||
f"порог {H1_BYTES_PER_KEY})"
|
||||
)
|
||||
return Verdict("H1", CONFIRMED if spread_ok and rss_ok else REFUTED, numbers)
|
||||
|
||||
|
||||
def verdict_h2(stages: dict[str, dict], passes: list[dict]) -> Verdict:
|
||||
"""head: время ≥ 5 × batch на тех же N, запросов к БД не меньше
|
||||
объектов, RSS почти не растёт."""
|
||||
parts, checks = [], []
|
||||
rss = {}
|
||||
for stage, info in stages.items():
|
||||
head = cell_median(passes, stage, "head", "took_ms")
|
||||
batch = cell_median(passes, stage, "batch", "took_ms")
|
||||
if not head or not batch:
|
||||
continue
|
||||
queries = cell_median(passes, stage, "head", "db_queries")
|
||||
listed = cell_median(passes, stage, "head", "listed_objects")
|
||||
rss[info["n"]] = cell_median(passes, stage, "head", "maxrss_kib")
|
||||
ratio = head / batch
|
||||
checks.append(ratio >= H2_TIME_RATIO)
|
||||
checks.append(queries is not None and listed is not None and queries >= listed)
|
||||
parts.append(
|
||||
f"N={info['n']}: head {head:.0f} мс / batch {batch:.0f} мс = "
|
||||
f"{ratio:.1f}×, запросов БД {fmt(queries, 0)} на {fmt(listed, 0)} объектов"
|
||||
)
|
||||
if not parts:
|
||||
return Verdict("H2", UNTESTED, "нет пар head и batch")
|
||||
present = {n: v for n, v in rss.items() if v}
|
||||
if len(present) >= 2:
|
||||
growth = (present[max(present)] - present[min(present)]) / 1024
|
||||
checks.append(growth <= H2_RSS_GROWTH_MIB)
|
||||
parts.append(
|
||||
f"рост RSS head {min(present)} → {max(present)}: {growth:.1f} MiB "
|
||||
f"(порог {H2_RSS_GROWTH_MIB})"
|
||||
)
|
||||
return Verdict("H2", CONFIRMED if all(checks) else REFUTED, "; ".join(parts))
|
||||
|
||||
|
||||
def verdict_h3(
|
||||
stages: dict[str, dict], passes: list[dict], models: dict[str, Model]
|
||||
) -> Verdict:
|
||||
"""deep: прочитано ровно согласованные × размер; на 1 MiB — ≥ 5 × batch;
|
||||
1 CPU быстрее 0,25 CPU ≥ 1,5 раза."""
|
||||
parts, checks = [], []
|
||||
for stage, info in stages.items():
|
||||
rows = cell_rows(passes, stage, "deep") + cell_rows(passes, stage, "deep-fast")
|
||||
if not rows:
|
||||
continue
|
||||
want = models[stage].consistent * info["size"]
|
||||
exact = all(int(num(r["read_bytes"]) or -1) == want for r in rows)
|
||||
checks.append(exact)
|
||||
parts.append(
|
||||
f"N={info['n']} × {info['size'] // KIB} KiB: read_bytes = {want} "
|
||||
f"{'во всех' if exact else 'не во всех'} {len(rows)} проходах"
|
||||
)
|
||||
if info["size"] >= MIB:
|
||||
deep = cell_median(passes, stage, "deep", "took_ms")
|
||||
batch = cell_median(passes, stage, "batch", "took_ms")
|
||||
fast = cell_median(passes, stage, "deep-fast", "took_ms")
|
||||
share = cell_median(passes, stage, "deep", "cpu_share")
|
||||
if deep and batch:
|
||||
checks.append(deep / batch >= H3_DEEP_RATIO)
|
||||
parts.append(
|
||||
f"deep / batch = {deep / batch:.1f}× (порог {H3_DEEP_RATIO})"
|
||||
)
|
||||
if deep and fast:
|
||||
checks.append(deep / fast >= H3_FAST_RATIO)
|
||||
parts.append(
|
||||
f"0,25 CPU / 1 CPU = {deep / fast:.2f}× (порог {H3_FAST_RATIO}), "
|
||||
f"CPU уборщика / время при 0,25 CPU = {fmt(share)}"
|
||||
)
|
||||
if not parts:
|
||||
return Verdict("H3", UNTESTED, "нет глубоких проходов")
|
||||
return Verdict("H3", CONFIRMED if all(checks) else REFUTED, "; ".join(parts))
|
||||
|
||||
|
||||
def verdict_h4(passes: list[dict], windows: list[dict]) -> Verdict:
|
||||
"""Под нагрузкой: с пределом скорости p95 скачиваний в проходе ≤ 1,5 ×
|
||||
тихих, 5xx нет; предел соблюдён (затронуто/с ≤ предела / 0,9); без
|
||||
предела при 1 CPU отношение больше, чем с ним."""
|
||||
ratios: dict[str, float | None] = {}
|
||||
for cell in LOAD_CELLS:
|
||||
ratios[cell] = med(
|
||||
w["ratio_p95"]
|
||||
for w in windows
|
||||
if w["cell"] == cell and w["window"] == "pass" and w["kind"] == "download"
|
||||
)
|
||||
if not any(ratios.values()):
|
||||
return Verdict("H4", UNTESTED, "нет проходов под нагрузкой")
|
||||
fives = sum(int(num(w["errors_5xx"]) or 0) for w in windows)
|
||||
# генератор не успевал по расписанию — отношение p95 не о сверке
|
||||
dropped = sum("bg_dropped" in str(p.get("flags") or "") for p in passes)
|
||||
checks = [fives == 0, dropped == 0]
|
||||
for cell in LIMITED_LOAD:
|
||||
if ratios.get(cell) is not None:
|
||||
checks.append(ratios[cell] <= H4_THRESHOLD)
|
||||
rates = []
|
||||
for p in passes:
|
||||
if p["cell"] in LIMITED_LOAD and num(p.get("rate")):
|
||||
observed = num(p.get("touched_per_s"))
|
||||
rates.append(observed)
|
||||
checks.append(
|
||||
observed is not None and observed <= num(p["rate"]) / H4_RATE_SHARE
|
||||
)
|
||||
if ratios.get("load-fast") and ratios.get("load-fast-limited"):
|
||||
checks.append(ratios["load-fast"] > ratios["load-fast-limited"])
|
||||
numbers = (
|
||||
"p95 скачиваний в проходе / в тихих окнах: "
|
||||
+ ", ".join(f"{cell} {fmt(value)}" for cell, value in ratios.items())
|
||||
+ f" (порог для ячеек с пределом {H4_THRESHOLD}); 5xx во всех окнах: {fives}; "
|
||||
f"проходов с bg_dropped: {dropped}; затронуто объектов/с с пределом: "
|
||||
f"{', '.join(fmt(r, 0) for r in rates) or '—'}"
|
||||
)
|
||||
return Verdict("H4", CONFIRMED if all(checks) else REFUTED, numbers)
|
||||
|
||||
|
||||
def verdict_h5(passes: list[dict]) -> Verdict:
|
||||
"""Отказ по пределу ничего не меняет и стоит как dry-run того же набора."""
|
||||
stage = next((p["stage"] for p in passes if p["cell"] == "refuse"), None)
|
||||
if stage is None:
|
||||
return Verdict("H5", UNTESTED, "нет проходов отказа")
|
||||
refuse = [p for p in passes if p["stage"] == stage and p["cell"] == "refuse"]
|
||||
fixed = sum(
|
||||
int(num(p.get(f"fixed_{action}")) or 0) for p in refuse for action in ACTIONS
|
||||
)
|
||||
codes = sorted({p.get("exit_code") for p in refuse}, key=str)
|
||||
refused = all(p.get("refused") in (True, "True") for p in refuse)
|
||||
took_r = cell_median(passes, stage, "refuse", "took_ms")
|
||||
took_d = cell_median(passes, stage, "batch", "took_ms")
|
||||
calls = {
|
||||
cell: cell_median(passes, stage, cell, "s3_head")
|
||||
for cell in ("refuse", "batch")
|
||||
}
|
||||
checks = [fixed == 0, refused, codes == [3]]
|
||||
numbers = (
|
||||
f"исправлено {fixed}, коды {codes}, refused во всех: {refused}; время "
|
||||
f"отказа {fmt(took_r, 0)} мс / dry-run {fmt(took_d, 0)} мс; HEAD "
|
||||
f"{fmt(calls['refuse'], 0)} / {fmt(calls['batch'], 0)}"
|
||||
)
|
||||
if took_r and took_d:
|
||||
checks.append(abs(took_r / took_d - 1) <= H5_SPREAD)
|
||||
checks.append(calls["refuse"] is not None and calls["refuse"] == calls["batch"])
|
||||
return Verdict("H5", CONFIRMED if all(checks) else REFUTED, numbers)
|
||||
|
||||
|
||||
# --- сводка ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def render_summary(
|
||||
meta: dict,
|
||||
passes: list[dict],
|
||||
windows: list[dict],
|
||||
verdicts: list[Verdict],
|
||||
) -> str:
|
||||
profile = meta["profile"]
|
||||
lines = [
|
||||
f"# Сверка bucket и БД: метка `{meta['label']}`",
|
||||
"",
|
||||
f"- Статус: **{'прерван: ' + meta['aborted'] if meta.get('aborted') else 'готово'}**",
|
||||
f"- Код (`src_sha256` образа): `{meta.get('code', {}).get('src_sha256', '—')}`",
|
||||
f"- Профиль `{profile['name']}`: повторов {profile['repeats']}, уборщик "
|
||||
f"{profile['cleaner_cpus']} CPU (управляемый опыт — {profile['fast_cpus']}), "
|
||||
f"предел скорости по умолчанию {profile['default_rate']:g} объектов/с, "
|
||||
f"grace прохода {profile['grace_s']} с, страница {profile['page_size']}",
|
||||
f"- Фоновая нагрузка (H4): скачиваний {profile['load_download_rate']:g}/с, "
|
||||
f"записей {profile['load_write_rate']:g}/с, чтений {profile['load_read_rate']:g}/с; "
|
||||
f"тихие окна {profile['quiet_s']:g} с до и после прохода",
|
||||
"- Значения — медиана (min–max) повторов. Стоимость — итоговая строка "
|
||||
"уборщика (счётчики клиента S3, запросы к БД, `cpu_ms` — CPU самого "
|
||||
"прохода без старта процесса, `maxrss_kib` — пик RSS процесса); CPU S3, "
|
||||
"PostgreSQL и реплик — сэмплер за окно прохода. LIST включает проверку "
|
||||
"bucket перед каждым «объекта нет» (список с одним ключом)",
|
||||
"",
|
||||
"## Проходы",
|
||||
"",
|
||||
"| Этап | N | Размер | Ячейка | Время, мс | мс на 1 000 | LIST | HEAD | GET "
|
||||
"| Запросов БД | Прочитано, MiB | CPU, мс | RSS, MiB | S3 CPU | Пометки |",
|
||||
"|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|",
|
||||
]
|
||||
seen: list[tuple[str, str]] = []
|
||||
for p in passes:
|
||||
key = (p["stage"], p["cell"])
|
||||
if key not in seen:
|
||||
seen.append(key)
|
||||
for stage, cell in seen:
|
||||
rows = [p for p in passes if p["stage"] == stage and p["cell"] == cell]
|
||||
first = rows[0]
|
||||
flags = sorted({f for r in rows for f in str(r.get("flags") or "").split()})
|
||||
lines.append(
|
||||
f"| {stage} | {first['n']} | {int(first['size']) // KIB} KiB | {cell} | "
|
||||
f"{spread(r['took_ms'] for r in rows)} | "
|
||||
f"{spread((r['ms_per_1k'] for r in rows), 2)} | "
|
||||
f"{spread(r['s3_list'] for r in rows)} | {spread(r['s3_head'] for r in rows)} | "
|
||||
f"{spread(r['s3_get'] for r in rows)} | {spread(r['db_queries'] for r in rows)} | "
|
||||
f"{spread(((num(r['read_bytes']) or 0) / MIB for r in rows), 1)} | "
|
||||
f"{spread(r['cpu_ms'] for r in rows)} | "
|
||||
f"{spread(((num(r['maxrss_kib']) or 0) / 1024 for r in rows), 1)} | "
|
||||
f"{spread((r['s3_cpu_cores'] for r in rows), 2)} | {' '.join(flags) or '—'} |"
|
||||
)
|
||||
if windows:
|
||||
lines += [
|
||||
"",
|
||||
"## Фоновая нагрузка",
|
||||
"",
|
||||
"| Ячейка | Окно | Скачиваний | p95 скачиваний, мс | p95 записи, мс | "
|
||||
"Ошибок | p95 прохода / тихих |",
|
||||
"|---|---|---:|---:|---:|---:|---:|",
|
||||
]
|
||||
for cell in LOAD_CELLS:
|
||||
for window in WINDOWS:
|
||||
rows = [
|
||||
w for w in windows if w["cell"] == cell and w["window"] == window
|
||||
]
|
||||
if not rows:
|
||||
continue
|
||||
down = [w for w in rows if w["kind"] == "download"]
|
||||
write = [w for w in rows if w["kind"] == "write"]
|
||||
lines.append(
|
||||
f"| {cell} | {window} | {spread(w['requests'] for w in down)} | "
|
||||
f"{spread((w['p95_ms'] for w in down), 1)} | "
|
||||
f"{spread((w['p95_ms'] for w in write), 1)} | "
|
||||
f"{sum(int(num(w['errors']) or 0) for w in rows)} | "
|
||||
f"{spread((w['ratio_p95'] for w in down), 2) if window == 'pass' else '—'} |"
|
||||
)
|
||||
lines += [
|
||||
"",
|
||||
"## Гипотезы",
|
||||
"",
|
||||
"Тексты и пороги — `REPORT.md`, «Трек 2: гипотезы» (записаны до замеров). "
|
||||
"Итог по числам этой метки; вывод — в `REPORT.md`.",
|
||||
"",
|
||||
"| Гипотеза | Итог | Числа |",
|
||||
"|---|---|---|",
|
||||
]
|
||||
lines += [f"| {v.hypothesis} | {v.result} | {v.numbers} |" for v in verdicts]
|
||||
warnings = meta.get("host_warnings") or []
|
||||
if warnings:
|
||||
lines += ["", "## Предупреждения о хосте", "", *(f"- {w}" for w in warnings)]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# --- --selftest: проверка расчётов без стенда --------------------------------------
|
||||
|
||||
|
||||
def selftest() -> None:
|
||||
m = model(2000)
|
||||
assert m.found == {
|
||||
"orphan_object": 5,
|
||||
"missing_object": 5,
|
||||
"size_mismatch": 5,
|
||||
"stale_pending": 5,
|
||||
}, m.found
|
||||
assert (m.objects, m.ready_rows, m.pending_rows) == (1995, 1990, 5)
|
||||
assert m.consistent == 1980
|
||||
assert (class_count(401, 0), class_count(401, 1)) == (2, 1)
|
||||
s4 = model(0, clean=500, missing=100)
|
||||
assert (s4.objects, s4.ready_rows, s4.candidates) == (500, 600, 100)
|
||||
assert object_id(1, 2) == object_id(1, 2) != object_id(1, 3)
|
||||
assert uuid.UUID(object_id(1, 2)).version == 4
|
||||
assert len(content(1, 2, 4096)) == 4096
|
||||
assert content(1, 2, 10) == content(1, 2, 10)
|
||||
assert len(object_indices(0, 400)) == 399
|
||||
assert object_indices(0, 10, FORCE_MISSING) == []
|
||||
csv_text = rows_csv(
|
||||
seed=1,
|
||||
start=0,
|
||||
end=400,
|
||||
size=8,
|
||||
namespace="bench",
|
||||
chat_id="c",
|
||||
owner_id="o",
|
||||
created_at="2026-10-22T00:00:00Z",
|
||||
)
|
||||
rows = csv_text.splitlines()
|
||||
assert len(rows) == 399, len(rows)
|
||||
assert ",ready," in rows[0]
|
||||
assert ",9,\\x" in rows[1]
|
||||
assert ",pending," in rows[2]
|
||||
assert rows[0].count(",") == len(COPY_COLUMNS) - 1
|
||||
plan = plan_passes([("a", ("batch", "deep")), ("b", ("batch",))], 2, 7)
|
||||
assert [p["order"] for p in plan] == [1, 2, 3, 4, 5, 6]
|
||||
assert {p["pass"] for p in plan if p["stage"] == "a"} == {
|
||||
"a-batch-r1",
|
||||
"a-deep-r1",
|
||||
"a-batch-r2",
|
||||
"a-deep-r2",
|
||||
}
|
||||
assert plan[1]["cell"] != plan[2]["cell"]
|
||||
log = (
|
||||
"2026-10-23T12:00:00.000Z INFO src.cleaner [rid=-] cleaner started mode=reconcile\n"
|
||||
"2026-10-23T12:00:05.250Z WARNING src.services.reconcile [rid=-] "
|
||||
"reconcile pass finished apply=false strategy=batch deep=false rate=0 "
|
||||
"listed_objects=1995 checked_rows=1990 fresh_objects=0 fresh_rows=0 lost_known=0 "
|
||||
"found_orphan_object=5 found_stale_pending=5 found_missing_object=5 "
|
||||
"found_size_mismatch=5 found_checksum_mismatch=0 found_restored_object=0 "
|
||||
"found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 "
|
||||
"fixed_marked_lost=0 fixed_removed_pending=0 remaining=20 refused=false "
|
||||
"unlogged=0 interrupted_by=None s3_list=7 s3_head=15 s3_get=0 s3_delete=0 "
|
||||
"listed_keys=2000 read_bytes=0 db_queries=30 phase_list_ms=100 phase_rows_ms=50 "
|
||||
"phase_stale_ms=2 phase_recheck_ms=40 phase_act_ms=0 cpu_ms=900 maxrss_kib=90000 "
|
||||
"took=1200ms\n"
|
||||
)
|
||||
fields = parse_finished(log)
|
||||
assert fields is not None
|
||||
assert (fields["took_ms"], fields["refused"]) == (1200, False)
|
||||
assert (fields["interrupted_by"], fields["s3_list"]) == (None, 7)
|
||||
assert abs(fields["ts"] - 1_792_756_805.25) < 1e-3, fields["ts"]
|
||||
assert found_problems(fields, m, deep=False) == []
|
||||
bad = dict(fields, found_orphan_object=4)
|
||||
assert found_problems(bad, m, deep=False) == ["found_orphan_object=4 expected=5"]
|
||||
assert touched(fields) == 2015
|
||||
row = pass_row(
|
||||
{"pass": "q-batch-r1", "stage": "q", "cell": "batch", "repeat": 1, "order": 1},
|
||||
n=2000,
|
||||
size=4096,
|
||||
cell=CELLS["batch"],
|
||||
cleaner_cpus=0.25,
|
||||
rate=0,
|
||||
fields=fields,
|
||||
exit_code=3,
|
||||
nano_cpus=250_000_000,
|
||||
usage={},
|
||||
problems=[],
|
||||
)
|
||||
assert (row["ms_per_1k"], row["cpu_share"], row["flags"]) == (600.0, 0.75, "")
|
||||
assert set(row) <= set(PASS_COLUMNS), set(row) - set(PASS_COLUMNS)
|
||||
passes = [
|
||||
dict(row, stage=s, n=n, ms_per_1k=v, maxrss_kib=r)
|
||||
for s, n, v, r in (("b", 10_000, 10.0, 90_000), ("d", 100_000, 11.0, 110_000))
|
||||
]
|
||||
stages = {"b": {"n": 10_000, "size": 4096}, "d": {"n": 100_000, "size": 4096}}
|
||||
h1 = verdict_h1(stages, passes)
|
||||
assert h1.result == CONFIRMED, h1
|
||||
passes[1]["maxrss_kib"] = 200_000
|
||||
assert verdict_h1(stages, passes).result == REFUTED
|
||||
assert verdict_h5([]).result == UNTESTED
|
||||
print("reconcile selftest: ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if sys.argv[1:] == ["--selftest"]:
|
||||
selftest()
|
||||
else:
|
||||
raise SystemExit("usage: uv run python bench/reconcile.py --selftest")
|
||||
+3275
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,274 @@
|
||||
"""Сэмплер стенда: раз в секунду — ресурсы контейнеров, LB и PostgreSQL.
|
||||
|
||||
Работает только внутри контейнера `vlad-bench-sampler` (образ
|
||||
приложения: в нём есть Python и asyncpg) с `--cgroupns host`: тогда в
|
||||
`/sys/fs/cgroup/docker/<id>/` видны cgroup всех контейнеров ВМ Docker.
|
||||
Его запускает `bench/run.py` (`Stand.start_sampler`), на хосте он не
|
||||
работает. Пишет строки JSON в файл `--out`; время — часы ВМ (`time.time()`),
|
||||
те же, что у генератора.
|
||||
|
||||
Строка тика:
|
||||
{"ts": ..., "cg": {"<id12>": [usage_usec, nr_periods, nr_throttled,
|
||||
throttled_usec, memory_current]},
|
||||
"vm": [занято_jiffies, всего_jiffies],
|
||||
"lb": {"<слот>": {...}, "FRONTEND": {...}}, # режим list
|
||||
"pg": {"conn": [[адрес, состояние, ожидание, n], ...],
|
||||
"db": {xact_commit, xact_rollback, sessions, numbackends}}}
|
||||
|
||||
Режимы:
|
||||
list — заданные контейнеры (`--container id` …), опрос статистики
|
||||
HAProxy (`/stats;csv` на :8404 внутри сети) и одно постоянное
|
||||
соединение с PostgreSQL стенда (`pg_stat_activity`,
|
||||
`pg_stat_database`); его слот занимает одно из
|
||||
`max_connections`. Серии трека 1;
|
||||
scan — все каталоги контейнеров в cgroup на каждом тике (новые
|
||||
реплики видны с первого тика), без сети, LB и PostgreSQL:
|
||||
наблюдатель не обращается к стенду. Профили failover, rollout
|
||||
и reconcile;
|
||||
peak — пиковая память заданных контейнеров (`--container роль=id`) за
|
||||
окно прогона профиля uploads (лаба 4). Контейнер с `--user 0`,
|
||||
cgroupfs на запись, без сети. `memory.peak` каждого контейнера
|
||||
открывается на чтение и запись: прежнее значение — пик с
|
||||
запуска, запись `reset` в свой дескриптор сбрасывает пик для
|
||||
этого дескриптора (ядро ≥ 6.12). Раз в `--interval` —
|
||||
`memory.current`, `anon` и `sock` из `memory.stat`, `oom_kill`
|
||||
из `memory.events`. Появился `--stop-file` — пик через тот же
|
||||
дескриптор и выход.
|
||||
|
||||
Строки режима peak:
|
||||
{"start": {"<роль>": байт|null}, "reset": {"<роль>": true|false}}
|
||||
{"ts": ..., "pk": {"<роль>": [current, anon, sock, oom_kill]|null}}
|
||||
{"ts": ..., "peak": {"<роль>": байт|null}, "end": true}
|
||||
|
||||
Пароль PostgreSQL — только из окружения контейнера (`PGPASSWORD`), в
|
||||
вывод не попадает.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
CGROUP = Path("/sys/fs/cgroup/docker")
|
||||
HEX64 = re.compile(r"[0-9a-f]{64}")
|
||||
LB_STATS = "http://haproxy:8404/stats;csv"
|
||||
LB_FIELDS = ("scur", "stot", "econ", "eresp", "wretr", "wredis", "qcur", "rate")
|
||||
# по этому application_name отбираем соединения приложения; значение
|
||||
# продублировано из src/constants.py, потому что импортировать src нельзя
|
||||
APP_NAME = "mtusi-messenger"
|
||||
PG_CONN_SQL = """
|
||||
SELECT coalesce(host(client_addr), '-'), coalesce(state, '-'),
|
||||
coalesce(wait_event_type, 'CPU'), count(*)
|
||||
FROM pg_stat_activity
|
||||
WHERE backend_type = 'client backend' AND application_name = $1
|
||||
GROUP BY 1, 2, 3
|
||||
"""
|
||||
PG_DB_SQL = """
|
||||
SELECT xact_commit, xact_rollback, sessions, numbackends
|
||||
FROM pg_stat_database WHERE datname = current_database()
|
||||
"""
|
||||
|
||||
|
||||
def cgroup_line(container: str) -> list[int] | None:
|
||||
"""usage_usec, nr_periods, nr_throttled, throttled_usec, memory.current."""
|
||||
folder = CGROUP / container
|
||||
try:
|
||||
stat = (folder / "cpu.stat").read_text()
|
||||
memory = int((folder / "memory.current").read_text())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
values = dict(line.split() for line in stat.splitlines() if line)
|
||||
return [
|
||||
int(values.get("usage_usec", 0)),
|
||||
int(values.get("nr_periods", 0)),
|
||||
int(values.get("nr_throttled", 0)),
|
||||
int(values.get("throttled_usec", 0)),
|
||||
memory,
|
||||
]
|
||||
|
||||
|
||||
def vm_line() -> list[int]:
|
||||
fields = Path("/proc/stat").read_text().splitlines()[0].split()[1:9]
|
||||
user, nice, system, idle, iowait, irq, softirq, steal = (int(f) for f in fields)
|
||||
total = user + nice + system + idle + iowait + irq + softirq + steal
|
||||
return [total - idle - iowait, total]
|
||||
|
||||
|
||||
def lb_stats() -> dict | None:
|
||||
try:
|
||||
with urllib.request.urlopen(LB_STATS, timeout=0.5) as response: # noqa: S310
|
||||
text = response.read().decode()
|
||||
except OSError:
|
||||
return None
|
||||
lines = text.splitlines()
|
||||
header = lines[0].lstrip("# ").split(",")
|
||||
out = {}
|
||||
for line in lines[1:]:
|
||||
row = dict(zip(header, line.split(","), strict=False))
|
||||
if row.get("pxname") != "api" and row.get("pxname") != "entry":
|
||||
continue
|
||||
name = "FRONTEND" if row["pxname"] == "entry" else row["svname"]
|
||||
if name == "FRONTEND" and row["svname"] != "FRONTEND":
|
||||
continue
|
||||
out[name] = {
|
||||
"status": row.get("status", ""),
|
||||
"addr": row.get("addr", ""),
|
||||
**{field: int(row[field]) for field in LB_FIELDS if row.get(field)},
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
def _memory_line(container: str) -> list[int] | None:
|
||||
"""memory.current, anon, sock, oom_kill контейнера; `None` — его нет."""
|
||||
folder = CGROUP / container
|
||||
try:
|
||||
current = int((folder / "memory.current").read_text())
|
||||
stat = dict(
|
||||
line.split() for line in (folder / "memory.stat").read_text().splitlines()
|
||||
)
|
||||
events = dict(
|
||||
line.split() for line in (folder / "memory.events").read_text().splitlines()
|
||||
)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
return [
|
||||
current,
|
||||
int(stat.get("anon", 0)),
|
||||
int(stat.get("sock", 0)),
|
||||
int(events.get("oom_kill", 0)),
|
||||
]
|
||||
|
||||
|
||||
def _read_fd(fd: int) -> int | None:
|
||||
try:
|
||||
os.lseek(fd, 0, os.SEEK_SET)
|
||||
return int(os.read(fd, 64).decode().strip())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def peak(args: argparse.Namespace) -> None:
|
||||
"""Режим peak: сброс пика своим дескриптором, выборки, пик в конце."""
|
||||
targets = dict(item.split("=", 1) for item in args.container)
|
||||
handles: dict[str, int] = {}
|
||||
start: dict[str, int | None] = {}
|
||||
reset: dict[str, bool] = {}
|
||||
for role, container in targets.items():
|
||||
try:
|
||||
fd = os.open(CGROUP / container / "memory.peak", os.O_RDWR)
|
||||
except OSError:
|
||||
start[role], reset[role] = None, False
|
||||
continue
|
||||
start[role] = _read_fd(fd)
|
||||
try:
|
||||
os.lseek(fd, 0, os.SEEK_SET)
|
||||
os.write(fd, b"reset")
|
||||
reset[role] = True
|
||||
handles[role] = fd
|
||||
except OSError:
|
||||
reset[role] = False
|
||||
os.close(fd)
|
||||
with open(args.out, "a", encoding="ascii") as out:
|
||||
out.write(json.dumps({"start": start, "reset": reset}) + "\n")
|
||||
out.flush()
|
||||
next_tick = time.monotonic()
|
||||
while not os.path.exists(args.stop_file):
|
||||
tick = {
|
||||
"ts": round(time.time(), 3),
|
||||
"pk": {role: _memory_line(c) for role, c in targets.items()},
|
||||
}
|
||||
out.write(json.dumps(tick, separators=(",", ":")) + "\n")
|
||||
out.flush()
|
||||
next_tick += args.interval
|
||||
time.sleep(max(0.0, next_tick - time.monotonic()))
|
||||
final = {
|
||||
"ts": round(time.time(), 3),
|
||||
"peak": {role: _read_fd(fd) for role, fd in handles.items()},
|
||||
"end": True,
|
||||
}
|
||||
out.write(json.dumps(final) + "\n")
|
||||
|
||||
|
||||
async def main(args: argparse.Namespace) -> None:
|
||||
interval = args.interval
|
||||
connection = None
|
||||
if args.mode == "list":
|
||||
import asyncpg # noqa: PLC0415 - нужен только режиму list
|
||||
|
||||
for _ in range(20):
|
||||
try:
|
||||
connection = await asyncpg.connect(
|
||||
host="postgres",
|
||||
user=os.environ["PGUSER"],
|
||||
database=os.environ["PGDATABASE"],
|
||||
password=os.environ["PGPASSWORD"],
|
||||
server_settings={"application_name": "bench-sampler"},
|
||||
timeout=3,
|
||||
)
|
||||
break
|
||||
except (OSError, asyncpg.PostgresError):
|
||||
await asyncio.sleep(0.5)
|
||||
out = open(args.out, "a", encoding="ascii") # noqa: ASYNC230, SIM115
|
||||
next_tick = time.monotonic()
|
||||
while True:
|
||||
ts = time.time()
|
||||
if args.mode == "scan":
|
||||
# файлы cgroup — псевдофайловая система ядра: чтение без ожидания
|
||||
try:
|
||||
ids = [
|
||||
p.name
|
||||
for p in CGROUP.iterdir() # noqa: ASYNC240
|
||||
if HEX64.fullmatch(p.name)
|
||||
]
|
||||
except OSError:
|
||||
ids = []
|
||||
else:
|
||||
ids = args.container
|
||||
tick: dict = {"ts": round(ts, 3), "cg": {}, "vm": vm_line()}
|
||||
for container in ids:
|
||||
line = cgroup_line(container)
|
||||
if line is not None:
|
||||
tick["cg"][container[:12]] = line
|
||||
if args.mode == "list":
|
||||
tick["lb"] = await asyncio.to_thread(lb_stats)
|
||||
if connection is not None:
|
||||
with contextlib.suppress(Exception):
|
||||
rows = await connection.fetch(PG_CONN_SQL, APP_NAME)
|
||||
db = await connection.fetchrow(PG_DB_SQL)
|
||||
# соединения приложения: адрес реплики, состояние, ожидание
|
||||
tick["pg"] = {
|
||||
"conn": [list(row) for row in rows],
|
||||
"db": dict(db),
|
||||
}
|
||||
out.write(json.dumps(tick, separators=(",", ":")) + "\n")
|
||||
out.flush()
|
||||
next_tick += interval
|
||||
await asyncio.sleep(max(0.0, next_tick - time.monotonic()))
|
||||
|
||||
|
||||
def parser() -> argparse.ArgumentParser:
|
||||
root = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
root.add_argument("--mode", choices=("list", "scan", "peak"), required=True)
|
||||
root.add_argument("--interval", type=float, default=1.0)
|
||||
root.add_argument("--out", required=True)
|
||||
# list — id контейнера; peak — роль=id
|
||||
root.add_argument("--container", action="append", default=[])
|
||||
root.add_argument("--stop-file", default="/tmp/peak.stop")
|
||||
return root
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
with contextlib.suppress(KeyboardInterrupt):
|
||||
arguments = parser().parse_args()
|
||||
# peak — без event loop: только файлы cgroup и пауза между выборками
|
||||
if arguments.mode == "peak":
|
||||
peak(arguments)
|
||||
else:
|
||||
asyncio.run(main(arguments))
|
||||
sys.exit(0)
|
||||
@@ -0,0 +1,583 @@
|
||||
"""Профиль uploads: план, разбор сэмплера peak, сводка без docker. Только stdlib.
|
||||
|
||||
Контейнерами управляет `run.py` (профили `uploads` и `uploads-quick`),
|
||||
а тут собраны расчёты без побочных эффектов: порядок прогонов по кругам,
|
||||
пик памяти из вывода `bench/sampler.py` в режиме `peak`, строка
|
||||
`runs.csv`, сводка по ячейкам и вердикты U1–U4 (сами гипотезы внесены в
|
||||
`REPORT.md` заранее). Проверить расчёты без стенда: запустить этот файл
|
||||
с ключом `--selftest` через `uv run python`.
|
||||
"""
|
||||
|
||||
import random
|
||||
import statistics
|
||||
import sys
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass
|
||||
|
||||
MIB = 2**20
|
||||
VARIANTS = ("limited", "unlimited")
|
||||
VARIANT_NAMES = {"limited": "с лимитом", "unlimited": "без лимита"}
|
||||
ROLES = ("api", "s3", "lb")
|
||||
# пороги гипотез — до замеров (REPORT.md, «Память загрузок: гипотезы»)
|
||||
U1_HALF = 0.5
|
||||
U1_GROWTH_MIB = 1.0
|
||||
U2_BODIES_PER_CLIENT = 0.8
|
||||
U2_NEAR_LIMIT = 0.9
|
||||
U3_SHARE = 0.9
|
||||
|
||||
RUN_COLUMNS = (
|
||||
"run",
|
||||
"variant",
|
||||
"size",
|
||||
"body_bytes",
|
||||
"clients",
|
||||
"repeat",
|
||||
"order",
|
||||
"mem_before_mib",
|
||||
"start_peak_mib",
|
||||
"peak_mib",
|
||||
"peak_source",
|
||||
"anon_max_mib",
|
||||
"sock_max_mib",
|
||||
"oom_kill",
|
||||
"oom_killed",
|
||||
"exit_code",
|
||||
"running_after",
|
||||
"started",
|
||||
"ok",
|
||||
"rejected_429",
|
||||
"errors_5xx",
|
||||
"other_errors",
|
||||
"statuses",
|
||||
"uploads_per_s",
|
||||
"mib_per_s",
|
||||
"p50_ms",
|
||||
"p95_ms",
|
||||
"p99_ms",
|
||||
"max_ms",
|
||||
"s3_peak_mib",
|
||||
"s3_anon_max_mib",
|
||||
"lb_peak_mib",
|
||||
"lb_sock_max_mib",
|
||||
"gen_cpu_s",
|
||||
"host_swapouts",
|
||||
"flags",
|
||||
)
|
||||
|
||||
|
||||
def plan_runs(
|
||||
variants: Iterable[str],
|
||||
sizes: Iterable[str],
|
||||
clients: Iterable[int],
|
||||
repeats: int,
|
||||
seed: int,
|
||||
) -> list[dict]:
|
||||
"""Расписание: все сочетания «вариант, размер, число клиентов» в каждом
|
||||
из `repeats` кругов, перемешанные `Random(seed + r)`. Если круг
|
||||
начался бы той же ячейкой, что закончил прошлый, первые две меняются."""
|
||||
cells = [(v, s, c) for v in variants for s in sizes for c in clients]
|
||||
plan: list[dict] = []
|
||||
previous = None
|
||||
for round_no in range(1, repeats + 1):
|
||||
order = list(range(len(cells)))
|
||||
random.Random(seed + round_no).shuffle(order)
|
||||
if len(order) > 1 and order[0] == previous:
|
||||
order[0], order[1] = order[1], order[0]
|
||||
for index in order:
|
||||
variant, size, count = cells[index]
|
||||
plan.append(
|
||||
{
|
||||
"run": f"{variant}-{size}-c{count}-r{round_no}",
|
||||
"variant": variant,
|
||||
"size": size,
|
||||
"clients": count,
|
||||
"repeat": round_no,
|
||||
"order": len(plan) + 1,
|
||||
},
|
||||
)
|
||||
previous = order[-1]
|
||||
return plan
|
||||
|
||||
|
||||
@dataclass
|
||||
class Peak:
|
||||
"""Накопитель по одному контейнеру: что сэмплер `peak` увидел за окно."""
|
||||
|
||||
start_peak: int | None = None
|
||||
reset: bool = False
|
||||
peak: int | None = None
|
||||
first_current: int | None = None
|
||||
max_current: int = 0
|
||||
max_anon: int = 0
|
||||
max_sock: int = 0
|
||||
oom_first: int | None = None
|
||||
oom_last: int | None = None
|
||||
samples: int = 0
|
||||
missing: int = 0
|
||||
|
||||
@property
|
||||
def oom_kills(self) -> int:
|
||||
if self.oom_first is None or self.oom_last is None:
|
||||
return 0
|
||||
return self.oom_last - self.oom_first
|
||||
|
||||
@property
|
||||
def window_peak(self) -> int | None:
|
||||
"""Лучшая доступная оценка пика: `memory.peak` после нашего сброса,
|
||||
а если сбросить или дочитать не удалось (контейнер умер) — максимум
|
||||
из выборок."""
|
||||
if self.reset and self.peak is not None:
|
||||
return max(self.peak, self.max_current)
|
||||
return self.max_current or None
|
||||
|
||||
@property
|
||||
def source(self) -> str:
|
||||
return "reset" if self.reset and self.peak is not None else "sampled"
|
||||
|
||||
|
||||
def parse_peak(lines: list[dict]) -> dict[str, Peak]:
|
||||
"""Строки сэмплера `peak` → память по ролям."""
|
||||
roles: dict[str, Peak] = {}
|
||||
|
||||
def role(name: str) -> Peak:
|
||||
return roles.setdefault(name, Peak())
|
||||
|
||||
for line in lines:
|
||||
if "reset" in line:
|
||||
for name, value in line.get("start", {}).items():
|
||||
role(name).start_peak = value
|
||||
for name, ok in line["reset"].items():
|
||||
role(name).reset = bool(ok)
|
||||
elif "pk" in line:
|
||||
for name, values in line["pk"].items():
|
||||
item = role(name)
|
||||
if values is None:
|
||||
item.missing += 1
|
||||
continue
|
||||
current, anon, sock, oom = values
|
||||
item.samples += 1
|
||||
if item.first_current is None:
|
||||
item.first_current = current
|
||||
item.max_current = max(item.max_current, current)
|
||||
item.max_anon = max(item.max_anon, anon)
|
||||
item.max_sock = max(item.max_sock, sock)
|
||||
if item.oom_first is None:
|
||||
item.oom_first = oom
|
||||
item.oom_last = oom
|
||||
elif "peak" in line:
|
||||
for name, value in line["peak"].items():
|
||||
role(name).peak = value
|
||||
return roles
|
||||
|
||||
|
||||
def mib(value: int | float | None) -> float | None:
|
||||
return None if value is None else round(value / MIB, 1)
|
||||
|
||||
|
||||
def run_row(
|
||||
run: dict,
|
||||
*,
|
||||
body_bytes: int,
|
||||
peaks: dict[str, Peak],
|
||||
result: dict,
|
||||
state: dict,
|
||||
swapouts: int | None,
|
||||
) -> dict:
|
||||
"""Собрать строку `runs.csv` и флаги к ней.
|
||||
|
||||
- `oom`: ядро убило реплику по памяти (`OOMKilled` у контейнера или
|
||||
счётчик `oom_kill` в cgroup); для варианта это исход, не поломка;
|
||||
- `host_swap`: пока шёл прогон, у хоста росли swapouts;
|
||||
- `peak_sampled`: сбросить `memory.peak` не вышло, и пик взят как
|
||||
максимум выборок;
|
||||
- `no_result`: итог генератора не пришёл."""
|
||||
api = peaks.get("api", Peak())
|
||||
s3 = peaks.get("s3", Peak())
|
||||
lb = peaks.get("lb", Peak())
|
||||
oom = bool(state.get("oom_killed")) or api.oom_kills > 0
|
||||
flags = []
|
||||
if oom:
|
||||
flags.append("oom")
|
||||
if swapouts:
|
||||
flags.append("host_swap")
|
||||
if api.source == "sampled":
|
||||
flags.append("peak_sampled")
|
||||
if not result:
|
||||
flags.append("no_result")
|
||||
return {
|
||||
**{key: run[key] for key in ("run", "variant", "size", "clients", "repeat")},
|
||||
"order": run["order"],
|
||||
"body_bytes": body_bytes,
|
||||
"mem_before_mib": mib(api.first_current),
|
||||
"start_peak_mib": mib(api.start_peak),
|
||||
"peak_mib": mib(api.window_peak),
|
||||
"peak_source": api.source,
|
||||
"anon_max_mib": mib(api.max_anon),
|
||||
"sock_max_mib": mib(api.max_sock),
|
||||
"oom_kill": api.oom_kills,
|
||||
"oom_killed": bool(state.get("oom_killed")),
|
||||
"exit_code": state.get("exit_code"),
|
||||
"running_after": state.get("running"),
|
||||
"started": result.get("started"),
|
||||
"ok": result.get("ok"),
|
||||
"rejected_429": result.get("rejected_429"),
|
||||
"errors_5xx": result.get("errors_5xx"),
|
||||
"other_errors": result.get("other_errors"),
|
||||
"statuses": " ".join(
|
||||
f"{k}:{v}" for k, v in sorted(result.get("statuses", {}).items())
|
||||
),
|
||||
**{
|
||||
key: result.get(key)
|
||||
for key in (
|
||||
"uploads_per_s",
|
||||
"mib_per_s",
|
||||
"p50_ms",
|
||||
"p95_ms",
|
||||
"p99_ms",
|
||||
"max_ms",
|
||||
"gen_cpu_s",
|
||||
)
|
||||
},
|
||||
"s3_peak_mib": mib(s3.window_peak),
|
||||
"s3_anon_max_mib": mib(s3.max_anon),
|
||||
"lb_peak_mib": mib(lb.window_peak),
|
||||
"lb_sock_max_mib": mib(lb.max_sock),
|
||||
"host_swapouts": swapouts,
|
||||
"flags": " ".join(flags),
|
||||
}
|
||||
|
||||
|
||||
# --- сводка ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def num(value) -> float | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def median(values: Iterable) -> float | None:
|
||||
present = [v for v in (num(x) for x in values) if v is not None]
|
||||
return statistics.median(present) if present else None
|
||||
|
||||
|
||||
def spread(values: Iterable, digits: int = 1) -> str:
|
||||
present = [v for v in (num(x) for x in values) if v is not None]
|
||||
if not present:
|
||||
return "—"
|
||||
mid = statistics.median(present)
|
||||
if len(present) == 1:
|
||||
return f"{mid:.{digits}f}"
|
||||
return f"{mid:.{digits}f} ({min(present):.{digits}f}–{max(present):.{digits}f})"
|
||||
|
||||
|
||||
def fmt(value: float | None, digits: int = 1) -> str:
|
||||
return "—" if value is None else f"{value:.{digits}f}"
|
||||
|
||||
|
||||
def is_oom(row: dict) -> bool:
|
||||
return "oom" in str(row.get("flags") or "").split()
|
||||
|
||||
|
||||
def cells(rows: list[dict]) -> dict[tuple[str, str, int], list[dict]]:
|
||||
grouped: dict[tuple[str, str, int], list[dict]] = {}
|
||||
for row in rows:
|
||||
key = (row["variant"], row["size"], int(row["clients"]))
|
||||
grouped.setdefault(key, []).append(row)
|
||||
return grouped
|
||||
|
||||
|
||||
def per_client_bodies(row: dict) -> float | None:
|
||||
"""(пик − память до окна) / (размер тела × клиенты): сколько тел на клиента."""
|
||||
peak, before = num(row.get("peak_mib")), num(row.get("mem_before_mib"))
|
||||
body, clients = num(row.get("body_bytes")), int(row["clients"])
|
||||
if peak is None or before is None or not body or not clients:
|
||||
return None
|
||||
return (peak - before) * MIB / body / clients
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Verdict:
|
||||
hypothesis: str
|
||||
result: str
|
||||
numbers: str
|
||||
|
||||
|
||||
CONFIRMED = "подтверждена"
|
||||
REFUTED = "опровергнута"
|
||||
UNTESTED = "нет данных"
|
||||
|
||||
|
||||
def verdicts(rows: list[dict], *, limit_mib: float, big: str) -> list[Verdict]:
|
||||
grouped = cells(rows)
|
||||
|
||||
def group(variant: str, size: str, clients: int) -> list[dict]:
|
||||
return grouped.get((variant, size, clients), [])
|
||||
|
||||
out = []
|
||||
# U1: с лимитом пик ≤ половины лимита при C = 10, 20, 40 (большое тело),
|
||||
# рост сверх C = 10 ≤ 1 MiB на клиента; OOM и 5xx нет ни в одном прогоне
|
||||
levels = [c for c in (10, 20, 40) if group("limited", big, c)]
|
||||
limited = [r for r in rows if r["variant"] == "limited"]
|
||||
if levels:
|
||||
peaks = {
|
||||
c: median(r["peak_mib"] for r in group("limited", big, c)) for c in levels
|
||||
}
|
||||
base = peaks.get(10)
|
||||
growth = {
|
||||
c: (peaks[c] - base) / (c - 10)
|
||||
for c in levels
|
||||
if c > 10 and base is not None and peaks[c] is not None
|
||||
}
|
||||
ooms = sum(is_oom(r) for r in limited)
|
||||
fives = sum(int(num(r.get("errors_5xx")) or 0) for r in limited)
|
||||
ok = (
|
||||
all(v is not None and v <= limit_mib * U1_HALF for v in peaks.values())
|
||||
and all(v <= U1_GROWTH_MIB for v in growth.values())
|
||||
and not ooms
|
||||
and not fives
|
||||
)
|
||||
out.append(
|
||||
Verdict(
|
||||
"U1",
|
||||
CONFIRMED if ok else REFUTED,
|
||||
"пик, MiB: "
|
||||
+ ", ".join(f"C={c} {fmt(v)}" for c, v in peaks.items())
|
||||
+ f" (половина лимита {limit_mib * U1_HALF:g}); сверх C=10, MiB "
|
||||
"на клиента: "
|
||||
+ (", ".join(f"C={c} {fmt(v, 2)}" for c, v in growth.items()) or "—")
|
||||
+ f"; OOM {ooms}, 5xx {fives} во всех прогонах «с лимитом»",
|
||||
),
|
||||
)
|
||||
else:
|
||||
out.append(Verdict("U1", UNTESTED, "нет ячеек «с лимитом» C ≥ 10"))
|
||||
# U2: без лимита рост ≥ 0,8 тела на клиента при C = 10; при C = 40 —
|
||||
# OOM или пик ≥ 90 % лимита
|
||||
ten = group("unlimited", big, 10)
|
||||
forty = group("unlimited", big, 40)
|
||||
if ten or forty:
|
||||
slope = median(per_client_bodies(r) for r in ten if not is_oom(r))
|
||||
near = all(
|
||||
is_oom(r) or (num(r["peak_mib"]) or 0) >= limit_mib * U2_NEAR_LIMIT
|
||||
for r in forty
|
||||
)
|
||||
ok = (
|
||||
slope is not None and slope >= U2_BODIES_PER_CLIENT and bool(forty) and near
|
||||
)
|
||||
out.append(
|
||||
Verdict(
|
||||
"U2",
|
||||
CONFIRMED if ok else REFUTED,
|
||||
f"C=10: {fmt(slope, 2)} тела на клиента (порог {U2_BODIES_PER_CLIENT}); "
|
||||
f"C=40: OOM {sum(is_oom(r) for r in forty)}/{len(forty)}, пик "
|
||||
f"{spread(r['peak_mib'] for r in forty)} MiB (90 % лимита "
|
||||
f"{limit_mib * U2_NEAR_LIMIT:g})",
|
||||
),
|
||||
)
|
||||
else:
|
||||
out.append(Verdict("U2", UNTESTED, "нет ячеек «без лимита»"))
|
||||
# U3: загрузок/с с лимитом ≥ 90 % без лимита при C ≥ 10, пока тот жив
|
||||
ratios = []
|
||||
for variant_size in sorted({r["size"] for r in rows}):
|
||||
for clients in (10, 20, 40):
|
||||
lim = group("limited", variant_size, clients)
|
||||
unl = group("unlimited", variant_size, clients)
|
||||
if not lim or not unl or any(is_oom(r) for r in unl):
|
||||
continue
|
||||
a = median(r["uploads_per_s"] for r in lim)
|
||||
b = median(r["uploads_per_s"] for r in unl)
|
||||
if a is not None and b:
|
||||
ratios.append((variant_size, clients, a, b, a / b))
|
||||
if ratios:
|
||||
out.append(
|
||||
Verdict(
|
||||
"U3",
|
||||
CONFIRMED if all(r[4] >= U3_SHARE for r in ratios) else REFUTED,
|
||||
"; ".join(
|
||||
f"{s} C={c}: {a:.2f} / {b:.2f} = {ratio:.0%}"
|
||||
for s, c, a, b, ratio in ratios
|
||||
),
|
||||
),
|
||||
)
|
||||
else:
|
||||
out.append(Verdict("U3", UNTESTED, "нет пар ячеек без OOM"))
|
||||
# U4: с лимитом 429 нет при C ≤ 20 (4 места + 16 в очереди), есть при C = 40
|
||||
low = [r for r in limited if int(r["clients"]) <= 20]
|
||||
high = [r for r in limited if int(r["clients"]) >= 40]
|
||||
if low and high:
|
||||
low_429 = sum(int(num(r.get("rejected_429")) or 0) for r in low)
|
||||
high_429 = [int(num(r.get("rejected_429")) or 0) for r in high]
|
||||
ok = low_429 == 0 and all(v > 0 for v in high_429)
|
||||
out.append(
|
||||
Verdict(
|
||||
"U4",
|
||||
CONFIRMED if ok else REFUTED,
|
||||
f"429 при C ≤ 20: {low_429} в {len(low)} прогонах; при C = 40: "
|
||||
+ ", ".join(map(str, high_429)),
|
||||
),
|
||||
)
|
||||
else:
|
||||
out.append(Verdict("U4", UNTESTED, "нет ячеек «с лимитом» C ≤ 20 и C = 40"))
|
||||
return out
|
||||
|
||||
|
||||
def render_summary(meta: dict, rows: list[dict]) -> str:
|
||||
profile = meta["profile"]
|
||||
bodies = meta.get("bodies", {})
|
||||
big = max(bodies, key=lambda k: bodies[k]["bytes"]) if bodies else ""
|
||||
limit_mib = float(profile["api_memory"].rstrip("m"))
|
||||
lines = [
|
||||
f"# Память загрузок: метка `{meta['label']}`",
|
||||
"",
|
||||
f"- Статус: **{'прерван: ' + meta['aborted'] if meta.get('aborted') else 'готово'}**",
|
||||
f"- Код (`src_sha256` образа): `{meta.get('code', {}).get('src_sha256', '—')}`",
|
||||
f"- Профиль `{profile['name']}`: варианты "
|
||||
+ ", ".join(f"«{VARIANT_NAMES[v]}»" for v in profile["variants"])
|
||||
+ f"; клиенты {', '.join(map(str, profile['clients']))}; повторов "
|
||||
f"{profile['repeats']}; окно {profile['window_s']:g} с после "
|
||||
f"{profile['warmup_s']:g} с разгона",
|
||||
"- Тела: "
|
||||
+ ", ".join(
|
||||
f"{name} — {item['bytes']} байт ({item['width']} × {item['height']}), "
|
||||
f"SHA-256 `{item['sha256'][:16]}…`"
|
||||
for name, item in bodies.items()
|
||||
),
|
||||
f"- Реплика {profile['api_cpus']} CPU / {profile['api_memory']} без swap, "
|
||||
f"буфер приёма `{meta.get('stand', {}).get('api_tcp_rmem', '—')}`; S3 "
|
||||
f"{profile['s3_cpus']} CPU / {profile['s3_memory']}; «без лимита» — "
|
||||
+ ", ".join(f"`{k}={v}`" for k, v in profile["unlimited_env"]),
|
||||
"- Пик — `memory.peak` cgroup, сброшенный своим дескриптором после "
|
||||
"разогрева (`peak_source` = reset); без сброса — наибольшая выборка "
|
||||
f"раз в {profile['sample_interval_s']:g} с (`peak_sampled`). «До окна» — "
|
||||
"первая выборка. Ответы по кодам и задержки — загрузки, начатые в окне. "
|
||||
"Значения — медиана (min–max) повторов. Пик S3 включает страничный кэш "
|
||||
"записанных объектов и упирается в лимит; память самого шлюза — `anon`",
|
||||
"",
|
||||
"## Ячейки",
|
||||
"",
|
||||
"| Вариант | Тело | C | Пик старта, MiB | До окна, MiB | Пик, MiB | anon, MiB "
|
||||
"| sock, MiB | OOM | 201/202 | 429 | 5xx | прочие | Загрузок/с | p50, мс | "
|
||||
"p95, мс | S3 пик / anon, MiB | LB пик, MiB | Пометки |",
|
||||
"|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|"
|
||||
"---:|---:|---:|---|",
|
||||
]
|
||||
grouped = cells(rows)
|
||||
for (variant, size, clients), items in sorted(
|
||||
grouped.items(), key=lambda kv: (kv[0][0], kv[0][1], kv[0][2])
|
||||
):
|
||||
flags = sorted({f for r in items for f in str(r.get("flags") or "").split()})
|
||||
lines.append(
|
||||
f"| {VARIANT_NAMES.get(variant, variant)} | {size} | {clients} | "
|
||||
f"{spread(r['start_peak_mib'] for r in items)} | "
|
||||
f"{spread(r['mem_before_mib'] for r in items)} | "
|
||||
f"{spread(r['peak_mib'] for r in items)} | "
|
||||
f"{spread(r['anon_max_mib'] for r in items)} | "
|
||||
f"{spread(r['sock_max_mib'] for r in items)} | "
|
||||
f"{sum(is_oom(r) for r in items)}/{len(items)} | "
|
||||
f"{spread((r['ok'] for r in items), 0)} | "
|
||||
f"{spread((r['rejected_429'] for r in items), 0)} | "
|
||||
f"{spread((r['errors_5xx'] for r in items), 0)} | "
|
||||
f"{spread((r['other_errors'] for r in items), 0)} | "
|
||||
f"{spread((r['uploads_per_s'] for r in items), 2)} | "
|
||||
f"{spread((r['p50_ms'] for r in items), 0)} | "
|
||||
f"{spread((r['p95_ms'] for r in items), 0)} | "
|
||||
f"{spread(r['s3_peak_mib'] for r in items)} / "
|
||||
f"{spread(r['s3_anon_max_mib'] for r in items)} | "
|
||||
f"{spread(r['lb_peak_mib'] for r in items)} | {' '.join(flags) or '—'} |"
|
||||
)
|
||||
lines += [
|
||||
"",
|
||||
"## Гипотезы",
|
||||
"",
|
||||
"Тексты и пороги — `REPORT.md`, «Память загрузок: гипотезы» (записаны до "
|
||||
f"замеров); большое тело — `{big}`, лимит памяти реплики {limit_mib:g} MiB.",
|
||||
"",
|
||||
"| Гипотеза | Итог | Числа |",
|
||||
"|---|---|---|",
|
||||
]
|
||||
lines += [
|
||||
f"| {v.hypothesis} | {v.result} | {v.numbers} |"
|
||||
for v in verdicts(rows, limit_mib=limit_mib, big=big)
|
||||
]
|
||||
warnings = meta.get("host_warnings") or []
|
||||
if warnings:
|
||||
lines += ["", "## Предупреждения о хосте", "", *(f"- {w}" for w in warnings)]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# --- --selftest: проверка расчётов без стенда --------------------------------------
|
||||
|
||||
|
||||
def selftest() -> None:
|
||||
plan = plan_runs(VARIANTS, ("1mib", "10mib"), (1, 10), 2, 5)
|
||||
assert [p["order"] for p in plan] == list(range(1, 17))
|
||||
assert plan[7]["run"] != plan[8]["run"]
|
||||
assert len({p["run"] for p in plan}) == 16
|
||||
lines = [
|
||||
{"start": {"api": 300 * MIB, "s3": None}, "reset": {"api": True, "s3": False}},
|
||||
{"ts": 1.0, "pk": {"api": [100 * MIB, 50 * MIB, 1 * MIB, 0], "s3": None}},
|
||||
{"ts": 1.2, "pk": {"api": [180 * MIB, 120 * MIB, 3 * MIB, 0], "s3": None}},
|
||||
{"ts": 1.4, "pk": {"api": [150 * MIB, 90 * MIB, 2 * MIB, 1], "s3": None}},
|
||||
{"ts": 1.6, "peak": {"api": 200 * MIB}, "end": True},
|
||||
]
|
||||
peaks = parse_peak(lines)
|
||||
api = peaks["api"]
|
||||
assert (api.start_peak, api.window_peak) == (300 * MIB, 200 * MIB)
|
||||
assert (api.first_current, api.max_sock) == (100 * MIB, 3 * MIB)
|
||||
assert (api.oom_kills, api.source) == (1, "reset")
|
||||
assert (peaks["s3"].missing, peaks["s3"].source) == (3, "sampled")
|
||||
run = {"run": "limited-10mib-c10-r1", "variant": "limited", "size": "10mib"}
|
||||
run |= {"clients": 10, "repeat": 1, "order": 1}
|
||||
result = {"ok": 40, "rejected_429": 0, "errors_5xx": 0, "statuses": {"201": 40}}
|
||||
row = run_row(
|
||||
run,
|
||||
body_bytes=10 * MIB,
|
||||
peaks=peaks,
|
||||
result=result,
|
||||
state={"oom_killed": False, "exit_code": 0, "running": True},
|
||||
swapouts=0,
|
||||
)
|
||||
assert (row["peak_mib"], row["mem_before_mib"]) == (200.0, 100.0)
|
||||
assert row["flags"] == "oom", row["flags"]
|
||||
assert set(row) <= set(RUN_COLUMNS), set(row) - set(RUN_COLUMNS)
|
||||
assert per_client_bodies(row) == 1.0
|
||||
rows = []
|
||||
for clients, peak in ((10, 100.0), (20, 105.0), (40, 125.0)):
|
||||
rows.append(
|
||||
{
|
||||
"variant": "limited",
|
||||
"size": "10mib",
|
||||
"clients": clients,
|
||||
"body_bytes": 10 * MIB,
|
||||
"peak_mib": peak,
|
||||
"mem_before_mib": 80.0,
|
||||
"errors_5xx": 0,
|
||||
"rejected_429": 0 if clients <= 20 else 7,
|
||||
"uploads_per_s": 4.0,
|
||||
"flags": "",
|
||||
},
|
||||
)
|
||||
rows.append(dict(rows[0], variant="unlimited", peak_mib=200.0, uploads_per_s=4.1))
|
||||
rows.append(dict(rows[2], variant="unlimited", flags="oom"))
|
||||
result_by = {
|
||||
v.hypothesis: v.result for v in verdicts(rows, limit_mib=512, big="10mib")
|
||||
}
|
||||
assert result_by == {
|
||||
"U1": CONFIRMED,
|
||||
"U2": CONFIRMED,
|
||||
"U3": CONFIRMED,
|
||||
"U4": CONFIRMED,
|
||||
}, result_by
|
||||
rows[2]["peak_mib"] = 400.0
|
||||
assert verdicts(rows, limit_mib=512, big="10mib")[0].result == REFUTED
|
||||
print("uploads selftest: ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if sys.argv[1:] == ["--selftest"]:
|
||||
selftest()
|
||||
else:
|
||||
raise SystemExit("usage: uv run python bench/uploads.py --selftest")
|
||||
+526
@@ -0,0 +1,526 @@
|
||||
# Стенд лабы 5. Вход — HAProxy, за ним API_REPLICAS реплик API.
|
||||
# Хранилища: PostgreSQL (источник истины, в том числе задания обработки),
|
||||
# Valkey (кэш карточек чатов), SeaweedFS (staging и опубликованные файлы),
|
||||
# RabbitMQ (сообщения о заданиях). Фон: relay (outbox → брокер), worker
|
||||
# (обработка изображений), cleaner (уборка и сверка). Одноразовые:
|
||||
# migrate, s3-bootstrap. ${...} — из .env (`make env`). Наружу — только
|
||||
# 127.0.0.1. Правила — infra/AGENTS.md; запуск — `make up`.
|
||||
name: vladimirceva-lab5
|
||||
|
||||
# Блоки окружения ниже разбиты по хранилищам, чтобы каждый сервис
|
||||
# подключал только свои секреты. К базе ходят api, cleaner, relay,
|
||||
# worker и migrate
|
||||
x-db-env: &db-env
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
DB_HOST: postgres
|
||||
DB_PORT: "5432"
|
||||
DB_NAME: ${POSTGRES_DB:?POSTGRES_DB is not set, run make env}
|
||||
DB_USER: ${POSTGRES_USER:?POSTGRES_USER is not set, run make env}
|
||||
DB_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is empty, run make env}
|
||||
|
||||
# кэш нужен одному api
|
||||
x-cache-env: &cache-env
|
||||
CACHE_HOST: valkey
|
||||
CACHE_PORT: "6379"
|
||||
CACHE_PASSWORD: ${VALKEY_PASSWORD:?VALKEY_PASSWORD is empty, run make env}
|
||||
|
||||
# общий для api, worker и cleaner адрес S3 и имя bucket; ключи каждый
|
||||
# получает свои (их заводит infra/s3/bootstrap.sh), root-пары нет ни у кого
|
||||
x-s3-env: &s3-env
|
||||
S3_ENDPOINT: http://s3:8333
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
|
||||
# RabbitMQ — только relay и worker; у реплик API брокера нет вовсе
|
||||
x-broker-env: &broker-env
|
||||
BROKER_HOST: rabbitmq
|
||||
BROKER_PORT: "5672"
|
||||
BROKER_USER: messenger
|
||||
BROKER_PASSWORD: ${BROKER_PASSWORD:?BROKER_PASSWORD is empty, run make env}
|
||||
|
||||
# параметры pipeline: читают worker (повторы, аренда) и relay (бюджет при
|
||||
# возврате истёкшей аренды)
|
||||
x-pipeline-env: &pipeline-env
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
|
||||
# один образ SeaweedFS на сервер и bootstrap; digest мультиарх-индекса,
|
||||
# поэтому закрепление работает и на amd64, и на arm64
|
||||
x-s3-image: &s3-image chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d
|
||||
|
||||
# ужесточение контейнеров приложения: файловая система только для
|
||||
# чтения, без capabilities и повышения привилегий
|
||||
x-hardened: &hardened
|
||||
read_only: true
|
||||
cap_drop:
|
||||
- ALL
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_USER: ${POSTGRES_USER:?POSTGRES_USER is not set, run make env}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is empty, run make env}
|
||||
POSTGRES_DB: ${POSTGRES_DB:?POSTGRES_DB is not set, run make env}
|
||||
ports:
|
||||
# только localhost: к нему подключается `make run` и миграции с
|
||||
# хоста. API наружу так не публикуется — только через LB
|
||||
- "127.0.0.1:${POSTGRES_HOST_PORT:-5433}:5432"
|
||||
volumes:
|
||||
# с 18-й версии кластер лежит в подкаталоге 18/docker, поэтому
|
||||
# монтируем родительский каталог — тот, что объявлен в Config.Volumes
|
||||
- pgdata:/var/lib/postgresql
|
||||
healthcheck:
|
||||
# -h 127.0.0.1 обязателен: init-скрипты образа идут на временном
|
||||
# сервере, который слушает лишь unix-сокет, и проверка по сокету
|
||||
# отпустила бы зависимых до настоящего старта. $$ раскрывает shell
|
||||
# контейнера, а не compose
|
||||
test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U \"$$POSTGRES_USER\" -d \"$$POSTGRES_DB\""]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
start_interval: 1s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
# соединения ≤ max_connections (100): реплики 2 × 15, migrate 1,
|
||||
# cleaner 1, relay 1, worker 3 — 36; на пике замены реплик 51
|
||||
# (REPORT.md, «Лимиты и пулы»)
|
||||
cpus: "2.0"
|
||||
memory: 1G
|
||||
|
||||
# Valkey держит только кэш карточек чатов и ничего не пишет на диск
|
||||
# (valkey.conf): рестарт обнуляет кэш, а источник истины — PostgreSQL
|
||||
valkey:
|
||||
image: valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd
|
||||
restart: unless-stopped
|
||||
# пароль подставляет shell контейнера ($$), в описании сервиса его не
|
||||
# видно; entrypoint образа сбрасывает права до пользователя valkey
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- exec docker-entrypoint.sh valkey-server /usr/local/etc/valkey/valkey.conf --requirepass "$$VALKEY_PASSWORD"
|
||||
environment:
|
||||
VALKEY_PASSWORD: ${VALKEY_PASSWORD:?VALKEY_PASSWORD is empty, run make env}
|
||||
# valkey-cli читает пароль отсюда, и healthcheck обходится без -a
|
||||
VALKEYCLI_AUTH: ${VALKEY_PASSWORD:?VALKEY_PASSWORD is empty, run make env}
|
||||
ports:
|
||||
# только localhost: к нему подключается `make run`
|
||||
- "127.0.0.1:${VALKEY_HOST_PORT:-6380}:6379"
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ./infra/valkey/valkey.conf
|
||||
target: /usr/local/etc/valkey/valkey.conf
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
healthcheck:
|
||||
# с неверным паролем ответ будет NOAUTH, так что PONG значит
|
||||
# «сервер жив и пароль совпал»
|
||||
test: ["CMD-SHELL", 'test "$$(valkey-cli ping)" = PONG']
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 10s
|
||||
start_interval: 1s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
# 64mb под данные (maxmemory) и запас на сам процесс
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
|
||||
# SeaweedFS хранит байты вложений (infra/AGENTS.md, «S3»). Команда weed
|
||||
# server запускает в одном процессе master, volume, filer и S3-шлюз.
|
||||
# Внутренние компоненты сидят на 127.0.0.1 контейнера; наружу, в сеть
|
||||
# стенда, смотрят только S3 API (8333) и gRPC шлюза (18333) под
|
||||
# JWT-подписью. Переход на пользователя seaweed и -dir=/data делает
|
||||
# entrypoint образа
|
||||
s3:
|
||||
image: *s3-image
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- server
|
||||
# внутренние серверы SeaweedFS привязаны к loopback
|
||||
- -ip=127.0.0.1
|
||||
- -ip.bind=127.0.0.1
|
||||
# иначе master отправляет телеметрию разработчикам SeaweedFS
|
||||
- -master.telemetry=false
|
||||
# с одним master выбирать лидера не из кого; рестарт до healthy
|
||||
# занимает ~7 с вместо ~24
|
||||
- -master.electionTimeout=1s
|
||||
# задержка остановки тома имеет смысл только в многоузловом кластере
|
||||
- -volume.preStopSeconds=0
|
||||
- -filer
|
||||
- -s3
|
||||
- -s3.ip.bind=0.0.0.0
|
||||
- -s3.port=8333
|
||||
# гасим порты Iceberg (8181) и Lance (9101), открытые по умолчанию
|
||||
- -s3.port.iceberg=0
|
||||
- -s3.port.lance=0
|
||||
# пользователей заводит weed shell в bootstrap, HTTP IAM на 8333 лишний
|
||||
- -s3.iam=false
|
||||
# суммарный объём параллельных загрузок в шлюзе ≤ 64 МБ, сверх — очередь
|
||||
- -s3.concurrentUploadLimitMB=64
|
||||
environment:
|
||||
# root-ключи заданы сразу, поэтому S3 ни секунды не работает без
|
||||
# аутентификации. Они есть только здесь, другим сервисам не
|
||||
# передаются
|
||||
AWS_ACCESS_KEY_ID: ${S3_ROOT_ACCESS_KEY:?S3_ROOT_ACCESS_KEY is empty, run make env}
|
||||
AWS_SECRET_ACCESS_KEY: ${S3_ROOT_SECRET_KEY:?S3_ROOT_SECRET_KEY is empty, run make env}
|
||||
# gRPC шлюза открыт в сеть стенда; JWT-подпись не даёт соседнему
|
||||
# контейнеру действовать от имени администратора
|
||||
WEED_JWT_FILER_SIGNING_KEY: ${S3_JWT_SIGNING_KEY:?S3_JWT_SIGNING_KEY is empty, run make env}
|
||||
# мягкий предел кучи Go заставляет GC срабатывать раньше; вместе с
|
||||
# concurrentUploadLimitMB это держит двадцать параллельных PUT по
|
||||
# 10 MiB в лимите 512 МБ
|
||||
GOMEMLIMIT: 400MiB
|
||||
ports:
|
||||
# только localhost и только API S3: проверки приватности с хоста
|
||||
# (make s3-check) и make run; консоль, UI и служебные серверы не
|
||||
# публикуются
|
||||
- "127.0.0.1:${S3_HOST_PORT:-8334}:8333"
|
||||
volumes:
|
||||
# /data объявлен в образе: данные томов, метаданные filer, пользователи
|
||||
- s3data:/data
|
||||
healthcheck:
|
||||
# четыре условия по loopback: у master есть лидер, в /dir/status уже
|
||||
# виден узел данных, filer и шлюз отвечают. Без проверки DataNodes
|
||||
# healthy приходит ~4 с до регистрации тома, и первый PUT висел бы
|
||||
# почти до своего таймаута
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:9333/cluster/healthz && wget -q -O - http://127.0.0.1:9333/dir/status | grep -qF '\"DataNodes\":[{' && wget -q -O /dev/null http://127.0.0.1:8888/healthz && wget -q -O /dev/null http://127.0.0.1:8333/healthz"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
start_interval: 1s
|
||||
# filer на SIGTERM ждёт подписчиков около 15 с (константа в коде
|
||||
# SeaweedFS); с grace 10 с контейнер добивался бы SIGKILL и кодом 137
|
||||
stop_grace_period: 20s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
|
||||
# Разовая настройка S3 перед стартом реплик, по аналогии с migrate:
|
||||
# приватный bucket, пользователи api, worker и cleaner с узкими правами
|
||||
# и проверка, что анонимно ничего не читается. Запускать можно сколько
|
||||
# угодно раз — ключи и политики каждый раз приводятся к .env.
|
||||
# network_mode: service:s3 даёт общий loopback с сервером, без него до
|
||||
# master и filer не достучаться. make s3-check — тот же скрипт с check
|
||||
s3-bootstrap:
|
||||
image: *s3-image
|
||||
network_mode: service:s3
|
||||
user: seaweed
|
||||
entrypoint: ["sh", "/opt/s3/bootstrap.sh"]
|
||||
command: ["apply"]
|
||||
restart: "no"
|
||||
environment:
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
# weed shell обращается к IAM filer только с JWT этой подписи
|
||||
WEED_JWT_FILER_SIGNING_KEY: ${S3_JWT_SIGNING_KEY:?S3_JWT_SIGNING_KEY is empty, run make env}
|
||||
S3_API_ACCESS_KEY: ${S3_API_ACCESS_KEY:?S3_API_ACCESS_KEY is empty, run make env}
|
||||
S3_API_SECRET_KEY: ${S3_API_SECRET_KEY:?S3_API_SECRET_KEY is empty, run make env}
|
||||
S3_WORKER_ACCESS_KEY: ${S3_WORKER_ACCESS_KEY:?S3_WORKER_ACCESS_KEY is empty, run make env}
|
||||
S3_WORKER_SECRET_KEY: ${S3_WORKER_SECRET_KEY:?S3_WORKER_SECRET_KEY is empty, run make env}
|
||||
S3_CLEANER_ACCESS_KEY: ${S3_CLEANER_ACCESS_KEY:?S3_CLEANER_ACCESS_KEY is empty, run make env}
|
||||
S3_CLEANER_SECRET_KEY: ${S3_CLEANER_SECRET_KEY:?S3_CLEANER_SECRET_KEY is empty, run make env}
|
||||
# политика cleaner разрешает ListBucket лишь под этим префиксом; это
|
||||
# не секрет — cleaner вычисляет такой же из DB_NAME
|
||||
S3_CLEANER_LIST_PREFIX: ${POSTGRES_DB:?POSTGRES_DB is not set, run make env}/attachments/
|
||||
HOME: /tmp
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ./infra/s3/bootstrap.sh
|
||||
target: /opt/s3/bootstrap.sh
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
# tmpfs: временные файлы с ключами и политиками не попадают на диск, а
|
||||
# объявленный в образе /data не плодит анонимный volume на каждый запуск
|
||||
tmpfs:
|
||||
- /tmp
|
||||
- /data
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
|
||||
# миграции до старта реплик: тот же образ, другая команда. Один раз на
|
||||
# `up`, реплики ждут его завершения; сами реплики миграции не запускают
|
||||
# (их несколько — они гонялись бы за одну схему). Повторный запуск —
|
||||
# no-op: alembic применяет только недостающие ревизии
|
||||
migrate:
|
||||
image: mtusi-messenger-vladimirceva:lab5
|
||||
# образ собирает сервис api (make up, make docker-build)
|
||||
pull_policy: never
|
||||
command: ["python", "-m", "alembic", "upgrade", "head"]
|
||||
restart: "no"
|
||||
# migrate получает одно окружение базы
|
||||
environment: *db-env
|
||||
<<: *hardened
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 256M
|
||||
|
||||
# API масштабируется числом реплик, поэтому у сервиса нет ни ports, ни
|
||||
# container_name — с ними --scale не работает; клиенты ходят только
|
||||
# через haproxy. X-Instance-Id берётся из hostname, а это короткий ID
|
||||
# контейнера. Временно сменить число — `make replicas API_REPLICAS=N`
|
||||
api:
|
||||
build: .
|
||||
# тот же тег, что у make docker-build
|
||||
image: mtusi-messenger-vladimirceva:lab5
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
<<: [*db-env, *cache-env, *s3-env]
|
||||
# пользователь API в S3: чтение, запись и удаление объектов своего
|
||||
# bucket; списка и прав на bucket нет
|
||||
S3_ACCESS_KEY_ID: ${S3_API_ACCESS_KEY:?S3_API_ACCESS_KEY is empty, run make env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_API_SECRET_KEY:?S3_API_SECRET_KEY is empty, run make env}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
SHUTDOWN_TIMEOUT: ${SHUTDOWN_TIMEOUT:-5}
|
||||
# предел незавершённых заданий стенда; 0 — без предела
|
||||
UPLOAD_MAX_UNFINISHED: ${UPLOAD_MAX_UNFINISHED:-300}
|
||||
<<: *hardened
|
||||
# приложение ничего не пишет на диск, кроме временных файлов Python
|
||||
tmpfs:
|
||||
- /tmp
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
# API переживает отказ Valkey, но стенд стартует сразу с рабочим кэшем
|
||||
valkey:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
# к первому запросу ключи API в S3 уже заведены
|
||||
s3-bootstrap:
|
||||
condition: service_completed_successfully
|
||||
# проверка здоровья задана в Dockerfile. Grace 10 с больше
|
||||
# SHUTDOWN_TIMEOUT (≤ 9 с), чтобы uvicorn закрылся сам, без SIGKILL
|
||||
stop_grace_period: 10s
|
||||
# потолок приёмного буфера TCP 512 KiB вместо 32 MiB по умолчанию в ВМ
|
||||
# Docker. Пока реплика не читает тело (ждёт место загрузки или уже
|
||||
# отказала), байты лежат в этом буфере и идут в лимит памяти
|
||||
# контейнера. Действует на все копии api, включая тестовые стенды и bench/
|
||||
sysctls:
|
||||
net.ipv4.tcp_rmem: "4096 131072 524288"
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
resources:
|
||||
limits:
|
||||
# на event loop хватает одного ядра; только Argon2id в потоках
|
||||
# мог бы взять больше, и под лимитом вход просто медленнее.
|
||||
# 512M складываются из четырёх Argon2id по 64 MiB, самого
|
||||
# интерпретатора и четырёх мест загрузки по телу до 10 MiB
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
|
||||
# cleaner: фоновая уборка брошенных загрузок и сверка S3 с базой. Образ
|
||||
# тот же, что у api, точка входа своя; стартует после migrate и
|
||||
# s3-bootstrap, от него никто не зависит, портов у него нет.
|
||||
# Разовые запуски из Makefile: make cleanup-once и make reconcile
|
||||
cleaner:
|
||||
image: mtusi-messenger-vladimirceva:lab5
|
||||
pull_policy: never
|
||||
# точка входа, а не command: аргументы `docker compose run cleaner …`
|
||||
# — флаги уборщика, а не замена команды
|
||||
entrypoint: ["python", "-m", "src.cleaner"]
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
<<: [*db-env, *s3-env]
|
||||
# проходы идут по одному, хватает одного соединения
|
||||
DB_POOL_SIZE: "1"
|
||||
DB_POOL_OVERFLOW: "0"
|
||||
# сколько держать staging и обрывки failed (окно для replay)
|
||||
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||
# ключи cleaner в S3: читать, удалять и листать префикс своей базы;
|
||||
# чужих ключей и пароля Valkey он не видит
|
||||
S3_ACCESS_KEY_ID: ${S3_CLEANER_ACCESS_KEY:?S3_CLEANER_ACCESS_KEY is empty, run make env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_CLEANER_SECRET_KEY:?S3_CLEANER_SECRET_KEY is empty, run make env}
|
||||
<<: *hardened
|
||||
tmpfs:
|
||||
- /tmp
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-bootstrap:
|
||||
condition: service_completed_successfully
|
||||
# унаследованный от образа healthcheck стучится в /health/ready, а у
|
||||
# cleaner нет HTTP; следить за ним — по журналу
|
||||
healthcheck:
|
||||
disable: true
|
||||
# на SIGTERM cleaner дорабатывает шаг (до 6 с) и выходит с кодом 0
|
||||
stop_grace_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 256M
|
||||
|
||||
# брокер: одна quorum-очередь <база>.attachment-jobs (объявляют relay и
|
||||
# worker). hostname фиксирован: по нему RabbitMQ называет узел и каталог
|
||||
# данных в volume — с новым именем старые сообщения не нашлись бы
|
||||
rabbitmq:
|
||||
image: rabbitmq:4.3.7-alpine@sha256:b1f997f74103d08fc1f20a13890dc44fe6cfa51d528cf521b480ada988d3ffa5
|
||||
hostname: rabbitmq
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
# пользователь создаётся при первом старте на пустом volume; смена
|
||||
# пароля потом — rabbitmqctl change_password (REPORT.md)
|
||||
RABBITMQ_DEFAULT_USER: messenger
|
||||
RABBITMQ_DEFAULT_PASS: ${BROKER_PASSWORD:?BROKER_PASSWORD is empty, run make env}
|
||||
ports:
|
||||
# только localhost: make run-worker / run-relay с хоста
|
||||
- "127.0.0.1:${BROKER_HOST_PORT:-5673}:5672"
|
||||
volumes:
|
||||
- rmqdata:/var/lib/rabbitmq
|
||||
- type: bind
|
||||
source: ./infra/rabbitmq/rabbitmq.conf
|
||||
target: /etc/rabbitmq/conf.d/90-messenger.conf
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
start_period: 30s
|
||||
start_interval: 2s
|
||||
stop_grace_period: 20s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
|
||||
# relay: забирает намерения из attachment_jobs и публикует с
|
||||
# подтверждением; возвращает истёкшие аренды. S3 и Valkey ему не нужны
|
||||
relay:
|
||||
image: mtusi-messenger-vladimirceva:lab5
|
||||
pull_policy: never
|
||||
entrypoint: ["python", "-m", "src.relay"]
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
<<: [*db-env, *broker-env, *pipeline-env]
|
||||
DB_POOL_SIZE: "1"
|
||||
DB_POOL_OVERFLOW: "0"
|
||||
<<: *hardened
|
||||
tmpfs:
|
||||
- /tmp
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
disable: true
|
||||
stop_grace_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 192M
|
||||
|
||||
# worker: обработка изображений. Pillow работает только в его дочерних
|
||||
# процессах (src/worker/render.py) с RLIMIT_AS = RENDER_MEMORY_MB и
|
||||
# RLIMIT_CPU; лимит контейнера — на concurrency таких процессов сразу
|
||||
worker:
|
||||
image: mtusi-messenger-vladimirceva:lab5
|
||||
pull_policy: never
|
||||
entrypoint: ["python", "-m", "src.worker"]
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
<<: [*db-env, *s3-env, *broker-env, *pipeline-env]
|
||||
# пользователь worker в S3: чтение staging, запись результата
|
||||
S3_ACCESS_KEY_ID: ${S3_WORKER_ACCESS_KEY:?S3_WORKER_ACCESS_KEY is empty, run make env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_WORKER_SECRET_KEY:?S3_WORKER_SECRET_KEY is empty, run make env}
|
||||
DB_POOL_SIZE: "2"
|
||||
DB_POOL_OVERFLOW: "1"
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||
WORKER_PREFETCH: ${WORKER_PREFETCH:-2}
|
||||
RENDER_MEMORY_MB: ${RENDER_MEMORY_MB:-768}
|
||||
<<: *hardened
|
||||
tmpfs:
|
||||
- /tmp
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-bootstrap:
|
||||
condition: service_completed_successfully
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
disable: true
|
||||
# SIGTERM: начатые задания ждут WORKER_STOP_TIMEOUT (8 с)
|
||||
stop_grace_period: 12s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "2.0"
|
||||
memory: 1536M
|
||||
|
||||
# HAProxy — единственный опубликованный вход API. Список реплик берёт из
|
||||
# DNS Docker и опрашивает их /health/ready (infra/haproxy/haproxy.cfg);
|
||||
# секретов в окружении нет
|
||||
haproxy:
|
||||
image: haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
# публикуем на 127.0.0.1: учебный стенд, из внешней сети не нужен
|
||||
- "127.0.0.1:${HOST_PORT:-8080}:8080"
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ./infra/haproxy/haproxy.cfg
|
||||
target: /usr/local/etc/haproxy/haproxy.cfg
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
<<: *hardened
|
||||
# из-за зависимости LB стартует после реплик, а при остановке стенда
|
||||
# выключается первым
|
||||
depends_on:
|
||||
- api
|
||||
healthcheck:
|
||||
# /lb/ready вернёт 200, только если в backend есть живая реплика,
|
||||
# то есть LB правда может ответить. Запрос — через /dev/tcp bash:
|
||||
# curl в образе отсутствует
|
||||
test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/8404 && printf 'GET /lb/ready HTTP/1.0\\r\\n\\r\\n' >&3 && head -n 1 <&3 | grep -q ' 200 '"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
start_interval: 1s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
# HAProxy запущен с одним потоком (nbthread 1)
|
||||
cpus: "1.0"
|
||||
memory: 256M
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
s3data:
|
||||
rmqdata:
|
||||
@@ -0,0 +1,61 @@
|
||||
# evidence/ — доказательства для REPORT.md
|
||||
|
||||
Требования преподавателя — `evidence/README.md` (skeleton, не менять).
|
||||
Каждый файл — подтверждение строки «Доказательства критериев» в
|
||||
`REPORT.md`. Доказательства лаб 3 и 4 остались в их репозиториях: сюда
|
||||
ничего не копируется, всё снято заново на стенде лабы 5.
|
||||
|
||||
## Правила
|
||||
|
||||
- **Только реальный вывод** команд, которые ты сам запустил. Не дописывать
|
||||
«ожидаемое», не править числа, не склеивать прогоны без пометки. Нечем
|
||||
проверить — не создавать файл, а сказать пользователю.
|
||||
- Шапка — строки `#`: что доказывает (пункт рубрики), дата UTC, окружение
|
||||
(ОС, Docker, Compose, Python), стенд (рабочий и его порт или отдельный),
|
||||
код (`src_sha256` — SHA-256 по отсортированным путям и содержимому
|
||||
`src/` и `migrations/` без `__pycache__`, как в паспорте `bench/`).
|
||||
Как повторить — команды.
|
||||
- Команда — строкой `$ ...`, под ней вывод; сокращать можно с пометкой
|
||||
`...`, не трогая строк, на которые ссылается отчёт.
|
||||
- Итог — строкой `#` после вывода: что видно и почему это закрывает
|
||||
пункт.
|
||||
- Без секретов, токенов, ключей, `.env`, текстов сообщений; домашний
|
||||
каталог — `<репозиторий>`; у переменных окружения — только имена.
|
||||
- Файл пишется инструментом правки (`apply_patch`), а не `>` из shell.
|
||||
|
||||
## Состав
|
||||
|
||||
Сняты на машине разработки (без нагрузки), 5 ноября 2026:
|
||||
|
||||
| Файл | Пункт рубрики |
|
||||
|---|---|
|
||||
| `stand/stand.txt` | стенд: отдельные broker и worker, relay, LB — единственный вход API, порты только на loopback, лимиты у всех, секреты по сервисам, образы по digest |
|
||||
| `stand/migrations.txt` | миграции `0001 → 0004` с нуля, повтор — no-op, `alembic check` |
|
||||
| `stand/s3-load.txt` | диагностика прерванного прогона: OOM SeaweedFS по журналу ядра, сохранённая метка `baseline` |
|
||||
| `stand/persistence.txt` | `make test-persistence ACTION_SCRIPT=scripts/restart.sh`: данные, токен и оригинал после рестарта LB, реплик, PostgreSQL, Valkey, S3 и RabbitMQ (`StartedAt` до и после) |
|
||||
| `attachments/smoke.txt` | публичный smoke лабы 5 |
|
||||
| `attachments/defense.txt` | `make defense`: worker стоит → 202 → ссылка 201 → 409 → текст → рестарт с volume → ready, thumbnail 256×160 |
|
||||
| `attachments/failover.txt` | `make attachment-failover`: вложение после остановки принявшей реплики |
|
||||
| `broker/restart.txt` | опубликованное задание переживает рестарт RabbitMQ с volume (та же попытка, без повторной публикации) |
|
||||
| `worker/limits.txt` | процесс рендера: пиксели, память (`RLIMIT_AS`), CPU (`RLIMIT_CPU`), не декодируется — коды выхода и исходы |
|
||||
| `s3/privacy.txt` | `make s3-check`: права ключей API, worker, уборщика, аноним — 403 |
|
||||
| `tests/test-isolated.txt` | smoke, свои тесты через LB, failover, attachment-failover, persistence на отдельном стенде |
|
||||
| `tests/failure-checks.txt` | проверки отказов с числами (`-rP`): окна отказа pipeline, повторы и replay, предел 429, staging, отказы хранилищ |
|
||||
| `tests/measurement-host.txt` | подготовка на машине с 10 CPU: бюджет queue 9,5 CPU, самопроверки, проверки приложения, сверка с лабой 4; без результатов нагрузки |
|
||||
|
||||
Снято на машине с 10 CPU (`bench/RUNBOOK.md`), 5 ноября 2026 UTC:
|
||||
|
||||
| Каталог | Пункт |
|
||||
|---|---|
|
||||
| `queue/queue10/` | трек 1: 24 ячейки, 120 фаз, 14185 принятых, 5xx / failed / missing 0; Q1–Q5 подтверждены; `summary.md`, `meta.json`, `cells.csv`, `timeline.csv`, `jobs.csv.gz` |
|
||||
|
||||
Поменялось показанное поведение — перезапусти проверку и обнови файл
|
||||
целиком, вместе с датой.
|
||||
|
||||
## Каталоги меток — исключение
|
||||
|
||||
`queue/<метка>/` (и профилей прошлых лаб: `uploads/`, `reconcile/`,
|
||||
`failover/`, `rollout/`, `scaling/`) пишет только стенд (`make bench-*`):
|
||||
руками не править и не создавать. Каталог метки не перезаписывается и не
|
||||
удаляется без согласия пользователя. `quick-*` — самопроверки, в git и
|
||||
отчёт не идут.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Доказывает (уровень 3, «Демонстрация»; шаги 2–3 сценария защиты): worker
|
||||
# остановлен → загрузка 202 queued → сообщение со ссылкой на queued 201 →
|
||||
# байты 409 → текст работает → рестарт LB, реплик API, PostgreSQL, Valkey,
|
||||
# S3 и RabbitMQ с теми же volume → вложение всё ещё queued (задание не
|
||||
# исчезло) → worker запущен → ready, оригинал и thumbnail 256×160.
|
||||
# Дата: 2026-11-05, 17:51–17:52 UTC.
|
||||
# Окружение: macOS 26.7.1, Docker 29.9.2, Compose 5.6.1, Python хоста 3.11.9.
|
||||
# Стенд: рабочий (make up), LB на 127.0.0.1:8180 (порт 8080 занят чужим
|
||||
# сервисом на этой машине).
|
||||
# Код: src_sha256 4c4bb305b2747d154105720b93261ef476675670470bfa7a10b338dd79514da9
|
||||
# Повторить: make defense BASE_URL=http://localhost:8180
|
||||
|
||||
$ make defense BASE_URL=http://localhost:8180
|
||||
python3 scripts/lab5.py defense --base-url "http://localhost:8180"
|
||||
$ docker compose stop worker
|
||||
worker stopped; upload -> 202 queued
|
||||
attachment_id=01fe582c-ae6e-4393-b744-53c5dfea2a1c
|
||||
message with queued attachment -> 201
|
||||
content while worker is stopped -> 409
|
||||
plain text message meanwhile -> 201
|
||||
StartedAt before restart:
|
||||
vladimirceva-lab5-api-1 2026-11-05T17:51:19.186978472Z; vladimirceva-lab5-api-2 2026-11-05T17:51:19.09613518Z
|
||||
vladimirceva-lab5-postgres-1 2026-11-05T16:45:55.248846086Z
|
||||
vladimirceva-lab5-valkey-1 2026-11-05T16:45:55.387099711Z
|
||||
vladimirceva-lab5-s3-1 2026-11-05T16:45:55.691076878Z
|
||||
vladimirceva-lab5-rabbitmq-1 2026-11-05T16:46:08.595203301Z
|
||||
worker: not running
|
||||
vladimirceva-lab5-relay-1 2026-11-05T17:51:18.354131013Z
|
||||
$ scripts/restart.sh -> exit 0
|
||||
after restart, worker still stopped: queued
|
||||
$ docker compose start worker
|
||||
after worker start: ready error_code=None
|
||||
original -> 200, sha256 == fixture
|
||||
thumbnail -> 200, JPEG (256, 160)
|
||||
StartedAt after:
|
||||
vladimirceva-lab5-api-1 2026-11-05T17:51:56.624640669Z; vladimirceva-lab5-api-2 2026-11-05T17:51:56.519461128Z
|
||||
vladimirceva-lab5-postgres-1 2026-11-05T17:51:32.808124047Z
|
||||
vladimirceva-lab5-valkey-1 2026-11-05T17:51:32.771200047Z
|
||||
vladimirceva-lab5-s3-1 2026-11-05T17:51:47.798918721Z
|
||||
vladimirceva-lab5-rabbitmq-1 2026-11-05T17:51:33.79540259Z
|
||||
vladimirceva-lab5-worker-1 2026-11-05T17:51:58.637002629Z
|
||||
vladimirceva-lab5-relay-1 2026-11-05T17:51:18.354131013Z
|
||||
PASS
|
||||
# exit 0
|
||||
|
||||
# Итог: StartedAt реплик, PostgreSQL, Valkey, S3 и RabbitMQ после —
|
||||
# позже, чем до: все они действительно перезапущены (relay не трогался по
|
||||
# замыслу restart.sh). Принятая работа пережила рестарт и остановку
|
||||
# worker: вложение осталось queued и стало ready после запуска worker;
|
||||
# thumbnail — настоящий JPEG 256×160 (размеры прочитаны из маркера SOF).
|
||||
# Ссылку на queued-вложение сообщение принимает (контракт: queued/processing
|
||||
# прикреплять можно), байты до готовности — 409.
|
||||
@@ -0,0 +1,75 @@
|
||||
# Доказывает (переносимое требование лаб 4–5): вложение, принятое одной
|
||||
# репликой и обработанное worker, скачивается через LB после остановки
|
||||
# именно этой реплики — байты совпадают по SHA-256; постороннему —
|
||||
# 404 и на метаданные, и на байты; реплика потом возвращена.
|
||||
# Дата: 2026-11-05, 17:52 UTC.
|
||||
# Окружение: macOS 26.7.1, Docker 29.9.2, Compose 5.6.1, Python хоста 3.11.9.
|
||||
# Стенд: рабочий (make up), LB на 127.0.0.1:8180, две реплики api.
|
||||
# Код: src_sha256 4c4bb305b2747d154105720b93261ef476675670470bfa7a10b338dd79514da9
|
||||
# Повторить: make attachment-failover BASE_URL=http://localhost:8180
|
||||
|
||||
$ make attachment-failover BASE_URL=http://localhost:8180
|
||||
python3 scripts/lab5.py attachment-failover --base-url "http://localhost:8180" \
|
||||
--replicas 2 --mode stop
|
||||
upload: 202 queued X-Instance-Id=686b3cd1f413 size=111817 sha256=99ea7533c441f79c… (fixture 99ea7533c441f79c…)
|
||||
processed: ready variants=3
|
||||
message with attachment: 201 attachment_ids=['5d46c446-c3f6-46ef-8a76-98623f934ab4']
|
||||
=== docker compose ps before
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
vladimirceva-lab5-api-1 3c1fcdf16b1f running Up 6 seconds (healthy)
|
||||
vladimirceva-lab5-api-2 686b3cd1f413 running Up 6 seconds (healthy)
|
||||
vladimirceva-lab5-cleaner-1 eb670d93c750 running Up 44 seconds
|
||||
vladimirceva-lab5-haproxy-1 28ad51aacfab running Up 6 seconds (healthy)
|
||||
vladimirceva-lab5-migrate-1 de187eda1dc1 exited Exited (0) 8 seconds ago
|
||||
vladimirceva-lab5-postgres-1 af42297aa67c running Up 30 seconds (healthy)
|
||||
vladimirceva-lab5-rabbitmq-1 cb3f59bde4b1 running Up 29 seconds (healthy)
|
||||
vladimirceva-lab5-relay-1 092f510a5e46 running Up 45 seconds
|
||||
vladimirceva-lab5-s3-1 f02a71e04dae running Up 15 seconds (healthy)
|
||||
vladimirceva-lab5-s3-bootstrap-1 279095908b5f exited Exited (0) 7 seconds ago
|
||||
vladimirceva-lab5-valkey-1 f6a287a02f57 running Up 30 seconds (healthy)
|
||||
vladimirceva-lab5-worker-1 0f8f0f2370a3 running Up 4 seconds
|
||||
|
||||
$ TARGET_INSTANCE_ID=686b3cd1f413 STOP_MODE=stop scripts/stop-one.sh
|
||||
exit 0; STOP_ONE_LOG: 2026-11-05T17:52:04Z TARGET_INSTANCE_ID=686b3cd1f413 container=686b3cd1f41322231088be18b06e46e248177ac104ed79d9c2ce2557a16b5cc3 mode=stop
|
||||
download by bob after stop: 200 X-Instance-Id=3c1fcdf16b1f after 0.51s, bytes=111817, sha256 == fixture, content-type=image/png
|
||||
outsider GET /attachments/{id}: 404
|
||||
outsider GET /attachments/{id}/content: 404
|
||||
=== docker compose ps after
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
vladimirceva-lab5-api-1 3c1fcdf16b1f running Up 8 seconds (healthy)
|
||||
vladimirceva-lab5-api-2 686b3cd1f413 exited Exited (0) 1 second ago
|
||||
vladimirceva-lab5-cleaner-1 eb670d93c750 running Up 46 seconds
|
||||
vladimirceva-lab5-haproxy-1 28ad51aacfab running Up 8 seconds (healthy)
|
||||
vladimirceva-lab5-migrate-1 de187eda1dc1 exited Exited (0) 10 seconds ago
|
||||
vladimirceva-lab5-postgres-1 af42297aa67c running Up 32 seconds (healthy)
|
||||
vladimirceva-lab5-rabbitmq-1 cb3f59bde4b1 running Up 31 seconds (healthy)
|
||||
vladimirceva-lab5-relay-1 092f510a5e46 running Up 46 seconds
|
||||
vladimirceva-lab5-s3-1 f02a71e04dae running Up 17 seconds (healthy)
|
||||
vladimirceva-lab5-s3-bootstrap-1 279095908b5f exited Exited (0) 9 seconds ago
|
||||
vladimirceva-lab5-valkey-1 f6a287a02f57 running Up 32 seconds (healthy)
|
||||
vladimirceva-lab5-worker-1 0f8f0f2370a3 running Up 6 seconds
|
||||
|
||||
replicas restored:
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
vladimirceva-lab5-api-1 3c1fcdf16b1f running Up 11 seconds (healthy)
|
||||
vladimirceva-lab5-api-2 686b3cd1f413 running Up 2 seconds (healthy)
|
||||
vladimirceva-lab5-cleaner-1 eb670d93c750 running Up 49 seconds
|
||||
vladimirceva-lab5-haproxy-1 28ad51aacfab running Up 11 seconds (healthy)
|
||||
vladimirceva-lab5-migrate-1 de187eda1dc1 exited Exited (0) 12 seconds ago
|
||||
vladimirceva-lab5-postgres-1 af42297aa67c running Up 35 seconds (healthy)
|
||||
vladimirceva-lab5-rabbitmq-1 cb3f59bde4b1 running Up 34 seconds (healthy)
|
||||
vladimirceva-lab5-relay-1 092f510a5e46 running Up 49 seconds
|
||||
vladimirceva-lab5-s3-1 f02a71e04dae running Up 20 seconds (healthy)
|
||||
vladimirceva-lab5-s3-bootstrap-1 279095908b5f exited Exited (0) 11 seconds ago
|
||||
vladimirceva-lab5-valkey-1 f6a287a02f57 running Up 35 seconds (healthy)
|
||||
vladimirceva-lab5-worker-1 0f8f0f2370a3 running Up 9 seconds
|
||||
|
||||
PASS
|
||||
# exit 0
|
||||
|
||||
# Итог: загрузку принял api-2 (686b3cd1f413), worker довёл её до ready с
|
||||
# тремя вариантами; после остановки api-2 участник чата скачал оригинал
|
||||
# через api-1 (3c1fcdf16b1f) за 0,51 с, SHA-256 совпал с фикстурой —
|
||||
# байты лежат в S3, а не на реплике. Посторонний получает 404 (не 403).
|
||||
# Прогон шёл сразу после make defense, поэтому «Up N seconds» у
|
||||
# хранилищ малы — они только что перезапущены restart.sh.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Доказывает (уровень 3, «Публичный smoke проходит»): публичный сценарий
|
||||
# лабы 5 через LB рабочего стенда — загрузка 202 queued, ready ≤ 60 с,
|
||||
# оригинал по SHA-256, настоящий JPEG thumbnail 256×160.
|
||||
# Дата: 2026-11-05, 17:51 UTC.
|
||||
# Окружение: macOS 26.7.1, Docker 29.9.2, Compose 5.6.1, Python хоста 3.11.9.
|
||||
# Стенд: рабочий (make up), LB на 127.0.0.1:8180.
|
||||
# Код: src_sha256 4c4bb305b2747d154105720b93261ef476675670470bfa7a10b338dd79514da9
|
||||
# Повторить: make test BASE_URL=http://localhost:8180
|
||||
|
||||
$ make test BASE_URL=http://localhost:8180
|
||||
python3 tests/smoke.py --base-url "http://localhost:8180"
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
# exit 0
|
||||
|
||||
# Итог: PASS публичной проверки skeleton (tests/smoke.py, профиль lab.json =
|
||||
# 5): 202, ready, оригинал, thumbnail 256×160 по метаданным и по маркеру
|
||||
# SOF самого JPEG, 404 постороннему.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Доказывает (уровень 4, «Брокер переживает рестарт с volume»; «Ни одно
|
||||
# принятое задание не теряется»): опубликованное relay задание лежит в
|
||||
# quorum-очереди и после stop/up RabbitMQ с тем же volume rmqdata; worker
|
||||
# берёт его с той же попыткой (attempt = 1) — повторной публикации и
|
||||
# лишней попытки нет.
|
||||
# Дата: 2026-11-05, 17:51–17:53 UTC.
|
||||
# Окружение: macOS 26.7.1, Docker 29.9.2, Compose 5.6.1, Python хоста 3.11.9.
|
||||
# Стенд: рабочий (make up), LB на 127.0.0.1:8180.
|
||||
# Код: src_sha256 4c4bb305b2747d154105720b93261ef476675670470bfa7a10b338dd79514da9
|
||||
# Повторить (то же, что делает проверка test_broker_restart_keeps_published_job
|
||||
# в student-tests/test_pipeline.py, но на рабочем стенде и с печатью):
|
||||
# docker compose stop worker → регистрация, чат, загрузка фикстуры через
|
||||
# LB → ждать published_at → docker compose stop rabbitmq → docker compose
|
||||
# up -d --wait --no-recreate rabbitmq → 15 раз раз в секунду: rabbitmqctl
|
||||
# list_queues name messages_ready messages_unacknowledged и строка задания
|
||||
# (state|attempt|published) → docker compose up -d --no-deps worker →
|
||||
# ждать ready → журнал relay и worker с момента пробы.
|
||||
|
||||
upload 202 a5835d46-6bc9-4fa4-828f-4e71df0986c9
|
||||
job before queued|1|t queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 0 0
|
||||
stop 0 1.4s
|
||||
up 0 3.3s
|
||||
t+0s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+1s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+2s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+3s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+4s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+5s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+6s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+7s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+8s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+9s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+10s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+11s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+12s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+13s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
t+14s queue name messages_ready messages_unacknowledged | messenger.attachment-jobs 1 0 job queued|1|t
|
||||
final job ready|1|t
|
||||
relay (since probe):
|
||||
2026-11-05T17:51:29.752Z INFO src.services.relay [rid=-] relay published jobs=1
|
||||
2026-11-05T17:51:31.272Z INFO src.services.relay [rid=-] relay published jobs=1
|
||||
2026-11-05T17:51:32.796Z WARNING src.relay [rid=-] relay pass failed unavailable=database retry_in=1s
|
||||
2026-11-05T17:52:02.875Z INFO src.services.relay [rid=-] relay published jobs=1
|
||||
2026-11-05T17:52:14.537Z INFO src.services.relay [rid=-] relay published jobs=1
|
||||
2026-11-05T17:52:16.571Z WARNING src.relay [rid=-] relay pass failed unavailable=database retry_in=1s
|
||||
2026-11-05T17:52:51.663Z INFO src.services.relay [rid=-] relay published jobs=1
|
||||
worker (since probe):
|
||||
2026-11-05T17:51:59.482Z INFO src.broker [rid=-] broker consuming queue=messenger.attachment-jobs prefetch=2
|
||||
2026-11-05T17:52:20.367Z INFO src.broker [rid=-] broker consuming queue=messenger.attachment-jobs prefetch=2
|
||||
2026-11-05T17:53:27.176Z INFO src.broker [rid=-] broker consuming queue=messenger.attachment-jobs prefetch=2
|
||||
2026-11-05T17:53:27.392Z INFO src.services.processing [rid=-] attachment processed attachment_id=a5835d46-6bc9-4fa4-828f-4e71df0986c9 job_id=3de62c1c-9a3b-45e4-9eb2-e2fbcc56b356 attempt=1 variants=3 render_peak_kib=98216 took=205ms waited=35952ms
|
||||
|
||||
# Итог: задание опубликовано (published = t, attempt = 1) при остановленном
|
||||
# worker; после stop/up RabbitMQ сообщение в очереди (messages_ready = 1)
|
||||
# все 15 секунд, задание всё ещё queued с attempt = 1. Worker после
|
||||
# запуска обработал его с attempt=1: дошло то самое сообщение, без
|
||||
# повторной публикации и новой попытки. Ноль в строке «job before» —
|
||||
# задержка статистики rabbitmqctl (~5 с после publish), а не пустая
|
||||
# очередь: строка взята через ~0,5 с после published_at. Журнал relay
|
||||
# за 2 минуты включает и прогоны defense/failover/persistence до пробы:
|
||||
# последняя публикация (17:52:51) — это задание пробы, после неё relay
|
||||
# ничего не публиковал. «relay pass failed unavailable=database» — рестарт
|
||||
# PostgreSQL в тех прогонах (restart.sh), relay повторил проход сам.
|
||||
@@ -0,0 +1,21 @@
|
||||
cell,config,concurrency,prefetch,size_class,body_bytes,repeat,order,phase,rate,window_s,scheduled,sent,accepted,rejected_429,retry_after_5,retry_after_1,errors_5xx,other_errors,dropped,share_429,share_429_saturated,backlog_reached,throughput,drain_s,startup_s,ttr_p50_ms,ttr_p95_ms,ttr_p99_ms,ttr_max_ms,lag_p50_ms,lag_p95_ms,lag_max_ms,took_p50_ms,took_p95_ms,max_unfinished,max_unfinished_sampled,max_outbox,max_in_broker,max_oldest_age_s,recovery_s,jobs,ready,failed,missing,unknown_jobs,retries,skipped,worker_cpu_cores,worker_mem_max_mib,worker_peak_mib,render_peak_mib,upload_p95_ms,flags
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,drain,10.0,3.0,20,20,20,0,0,0,0,0,0,0.0,,False,2.968,6.738,0.621,6007.5,8220.4,8298.7,8298.7,5332.0,7566.0,7636.0,665.0,701.0,,20,2,20,7.75,,20,20,0,0,0,0,0,1.926,530.5,669.0,299.4,102.64,
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,steady,1.781,40.0,72,72,72,0,0,0,0,0,0,0.0,,False,,,,914.0,1149.9,1164.9,1164.9,260.0,484.0,507.0,657.0,667.0,,2,1,1,1.12,,72,72,0,0,0,0,0,1.12,376.4,669.0,299.5,130.75,
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,fixed,0.5,40.0,20,20,20,0,0,0,0,0,0,0.0,,False,,,,916.0,1137.7,1155.9,1155.9,251.0,491.0,494.0,661.0,675.0,,1,0,0,0.51,,20,20,0,0,0,0,0,0.325,267.8,669.0,299.5,133.33,
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,overload,5.936,40.0,238,238,173,65,65,0,0,0,0,0.2731,0.5242,True,,,,16011.4,20892.1,20943.9,20955.3,15340.0,20198.0,20260.0,695.0,719.0,60,60,2,58,20.28,,173,173,0,0,0,0,0,1.924,668.4,669.0,299.5,116.96,
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,recovery,1.781,20.0,36,36,36,0,0,0,0,0,0,0.0,,False,,,,923.8,1157.7,1161.1,1161.1,263.0,479.0,503.0,661.0,681.0,,59,1,56,20.87,20.809,36,36,0,0,0,0,0,1.497,668.8,669.0,299.4,133.38,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,13.413,3.728,0.613,3833.9,4630.5,4664.1,4664.1,3558.0,4365.0,4495.0,291.0,320.0,,50,10,50,4.59,,50,50,0,0,0,0,0,1.874,139.5,146.4,97.1,14.34,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,steady,8.048,40.0,322,322,322,0,0,0,0,0,0,0.0,,False,,,,555.3,760.0,815.0,826.5,282.0,506.0,520.0,280.0,320.0,,7,4,2,0.78,,322,322,0,0,0,0,0,1.197,145.5,146.4,97.6,18.01,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,421.7,654.8,692.5,696.5,260.0,510.0,531.0,159.0,166.0,,2,2,0,0.6,,100,100,0,0,0,0,0,0.367,112.4,146.4,97.6,26.53,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,overload,26.825,40.0,1074,1074,586,488,488,0,0,0,0,0.4544,0.5067,True,,,,4465.8,4564.4,4586.8,4595.4,4174.0,4268.0,4303.0,297.0,310.0,60,60,14,56,4.55,,586,586,0,0,0,0,0,1.954,145.6,146.4,98.1,16.36,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,recovery,8.048,20.0,161,161,161,0,0,0,0,0,0,0.0,,False,,,,532.8,763.4,815.4,823.2,272.0,501.0,517.0,283.0,324.0,,48,4,45,4.41,4.244,161,161,0,0,0,0,0,1.305,145.8,146.4,98.1,18.47,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,13.577,3.683,0.639,4398.6,5157.4,5227.7,5227.7,4253.0,5016.0,5081.0,145.0,150.0,,50,5,50,4.82,,50,50,0,0,0,0,0,1.771,112.7,116.0,97.4,14.02,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,steady,8.146,40.0,326,326,326,0,0,0,0,0,0,0.0,,False,,,,501.2,655.4,669.0,690.9,355.0,501.0,541.0,147.0,161.0,,6,4,3,0.63,,326,326,0,0,0,0,0,1.092,114.6,116.0,97.6,14.52,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,441.9,675.5,681.9,698.6,280.0,511.0,526.0,160.0,168.0,,2,2,0,0.58,,100,100,0,0,0,0,0,0.372,115.2,116.0,97.6,26.62,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,overload,27.153,40.0,1087,1087,596,491,491,0,0,0,0,0.4517,0.5015,True,,,,4382.2,4426.6,4435.0,4437.6,4237.0,4282.0,4293.0,147.0,152.0,60,60,7,58,4.41,,596,596,0,0,0,0,0,1.797,115.2,116.0,97.9,10.52,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,recovery,8.146,20.0,163,163,163,0,0,0,0,0,0,0.0,,False,,,,498.7,646.1,668.4,669.7,347.0,495.0,517.0,148.0,161.0,,46,4,44,4.39,4.335,163,163,0,0,0,0,0,1.217,115.6,116.0,97.9,14.19,
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,drain,4.0,11.0,40,40,40,0,0,0,0,0,0,0.0,,False,4.025,9.938,0.628,11422.5,11830.8,12062.4,12062.4,10453.0,10870.0,11077.0,1001.0,1113.0,,40,4,40,11.33,,40,40,0,0,0,0,0,1.58,405.8,406.6,175.6,380.96,
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,steady,2.415,40.0,97,97,97,0,0,0,0,0,0,0.0,,False,,,,920.2,1670.7,1960.3,1960.3,380.0,827.0,1238.0,510.0,1371.0,,4,2,0,1.74,,97,97,0,0,0,0,0,0.875,323.3,406.6,177.8,715.01,
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,fixed,0.8,40.0,32,32,32,0,0,0,0,0,0,0.0,,False,,,,764.1,1019.7,1564.1,1564.1,325.0,563.0,582.0,437.0,470.0,,1,1,0,1.48,,32,32,0,0,0,0,0,0.295,164.2,406.6,177.8,128.71,
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,overload,8.05,40.0,322,322,14,0,0,0,308,0,0,0.0,,False,,,,,,,,,,,,,14,14,14,7,32.98,,322,0,14,0,308,56,0,0.055,355.8,406.6,,637.54,failed 5xx
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,recovery,2.415,20.0,49,49,0,0,0,0,49,0,0,0.0,,False,,,,,,,,,,,,,,0,0,0,0.0,0.0,49,0,0,0,49,0,0,0.0,84.2,406.6,,,5xx
|
||||
|
Binary file not shown.
@@ -0,0 +1,679 @@
|
||||
{
|
||||
"kind": "queue",
|
||||
"label": "baseline",
|
||||
"started": "2026-11-05T20:32:47Z",
|
||||
"profile": {
|
||||
"name": "queue",
|
||||
"configs": [
|
||||
[
|
||||
"c1p1",
|
||||
1,
|
||||
1
|
||||
],
|
||||
[
|
||||
"c2p2",
|
||||
2,
|
||||
2
|
||||
],
|
||||
[
|
||||
"c4p4",
|
||||
4,
|
||||
4
|
||||
],
|
||||
[
|
||||
"c2p32",
|
||||
2,
|
||||
32
|
||||
]
|
||||
],
|
||||
"classes": [
|
||||
[
|
||||
"s",
|
||||
"noise",
|
||||
320,
|
||||
200,
|
||||
50
|
||||
],
|
||||
[
|
||||
"l",
|
||||
"noise",
|
||||
1800,
|
||||
1800,
|
||||
40
|
||||
],
|
||||
[
|
||||
"x",
|
||||
"flat",
|
||||
6000,
|
||||
6000,
|
||||
20
|
||||
]
|
||||
],
|
||||
"repeats": 2,
|
||||
"window_s": 40.0,
|
||||
"steady_factor": 0.6,
|
||||
"overload_factor": 2.0,
|
||||
"overload_fill": 1.5,
|
||||
"overload_max_s": 150.0,
|
||||
"fixed_rates": [
|
||||
[
|
||||
"s",
|
||||
2.5
|
||||
],
|
||||
[
|
||||
"l",
|
||||
0.8
|
||||
],
|
||||
[
|
||||
"x",
|
||||
0.5
|
||||
]
|
||||
],
|
||||
"backlog_limit": 60,
|
||||
"fill_rates": [
|
||||
[
|
||||
"s",
|
||||
25.0
|
||||
],
|
||||
[
|
||||
"l",
|
||||
4.0
|
||||
],
|
||||
[
|
||||
"x",
|
||||
10.0
|
||||
]
|
||||
],
|
||||
"upload_timeout_s": 60.0,
|
||||
"settle_timeout_s": 900.0,
|
||||
"sample_interval_s": 1.0,
|
||||
"seed": 20261105,
|
||||
"api_replicas": 2,
|
||||
"api_cpus": 1.0,
|
||||
"api_memory": "512m",
|
||||
"db_cpus": 1.0,
|
||||
"lb_cpus": 0.5,
|
||||
"s3_cpus": 1.0,
|
||||
"s3_memory": "512m",
|
||||
"mq_cpus": 1.0,
|
||||
"mq_memory": "512m",
|
||||
"worker_cpus": 2.0,
|
||||
"worker_memory": "1536m",
|
||||
"render_memory_mb": 768,
|
||||
"gen_cpus": 1.0,
|
||||
"gen_memory": "512m",
|
||||
"session_ttl": 21600
|
||||
},
|
||||
"host": {
|
||||
"platform": "macOS-26.6.2-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"cpu_model": "Apple M4",
|
||||
"cpu_count": 10,
|
||||
"ram_gb": 16.0,
|
||||
"swap_used_mib": 4679.06,
|
||||
"power": "AC Power",
|
||||
"memory_pressure": null,
|
||||
"memory_free_percent": 51,
|
||||
"swapouts": 950094,
|
||||
"loadavg_1m": 2.25
|
||||
},
|
||||
"docker": {
|
||||
"ServerVersion": "28.0.4",
|
||||
"OperatingSystem": "Docker Desktop",
|
||||
"KernelVersion": "6.10.14-linuxkit",
|
||||
"Architecture": "aarch64",
|
||||
"NCPU": 10,
|
||||
"MemTotal": 8218034176,
|
||||
"CgroupVersion": "2",
|
||||
"CgroupDriver": "cgroupfs",
|
||||
"compose_version": "5.6.1",
|
||||
"other_running_containers": []
|
||||
},
|
||||
"budget": {
|
||||
"limits_cpus": 9.5,
|
||||
"limits_memory_gib": 5.81,
|
||||
"vm_cpus": 10,
|
||||
"vm_memory_gib": 7.65
|
||||
},
|
||||
"host_warnings": [],
|
||||
"image": {
|
||||
"tag": "mtusi-messenger-vladimirceva:bench-baseline",
|
||||
"id": "sha256:eec175e93886926e10260e6cdf6464e6e21b4111fd44d5f62c798e8e4158f4d4",
|
||||
"created": "2026-11-05T19:51:54.027533094Z",
|
||||
"src_sha256": "4c4bb305b2747d154105720b93261ef476675670470bfa7a10b338dd79514da9",
|
||||
"versions": {
|
||||
"python": "3.13.15",
|
||||
"fastapi": "0.142.1",
|
||||
"starlette": "1.7.0",
|
||||
"uvicorn": "0.54.0",
|
||||
"pydantic": "2.13.5",
|
||||
"sqlalchemy": "2.1.1",
|
||||
"asyncpg": "0.31.0",
|
||||
"alembic": "1.20.0",
|
||||
"redis": "8.1.0",
|
||||
"argon2-cffi": "25.1.0",
|
||||
"aiobotocore": "3.10.0",
|
||||
"botocore": "1.43.117",
|
||||
"aiohttp": "3.14.5"
|
||||
},
|
||||
"app_defaults": {
|
||||
"pool_size": 10,
|
||||
"pool_overflow": 5,
|
||||
"pool_timeout": 5.0,
|
||||
"operation_timeout": 6.0,
|
||||
"statement_timeout_ms": 5000,
|
||||
"kdf_concurrency": 4,
|
||||
"keep_alive": 60,
|
||||
"shutdown_timeout": 5,
|
||||
"s3_timeout": 5.0,
|
||||
"s3_max_connections": 10,
|
||||
"uploads": {
|
||||
"max_concurrent": 4,
|
||||
"max_waiting": 16,
|
||||
"wait_timeout": 5.0,
|
||||
"body_timeout": 30.0,
|
||||
"max_unfinished": 300
|
||||
},
|
||||
"cleaner": {
|
||||
"interval": 60.0,
|
||||
"grace": 900,
|
||||
"batch": 100,
|
||||
"reconcile_interval": 3600,
|
||||
"reconcile_page": 1000,
|
||||
"reconcile_max_changes": 100,
|
||||
"reconcile_rate": 2000.0,
|
||||
"staging_retention": 86400
|
||||
},
|
||||
"pipeline": {
|
||||
"concurrency": 2,
|
||||
"prefetch": 2,
|
||||
"max_attempts": 5,
|
||||
"job_timeout": 60.0,
|
||||
"lease_margin": 20.0,
|
||||
"retry_base": 2.0,
|
||||
"retry_max": 120.0,
|
||||
"render_memory_mb": 768,
|
||||
"stop_timeout": 8.0,
|
||||
"fault": ""
|
||||
},
|
||||
"relay": {
|
||||
"interval": 0.5,
|
||||
"batch": 100,
|
||||
"republish_after": 120.0,
|
||||
"fault": ""
|
||||
},
|
||||
"malloc_mmap_threshold": "131072"
|
||||
}
|
||||
},
|
||||
"code": {
|
||||
"src_sha256": "4c4bb305b2747d154105720b93261ef476675670470bfa7a10b338dd79514da9",
|
||||
"git_commit": null,
|
||||
"git_dirty": null
|
||||
},
|
||||
"plan": [
|
||||
{
|
||||
"cell": "c2p2-x-r1",
|
||||
"config": "c2p2",
|
||||
"concurrency": 2,
|
||||
"prefetch": 2,
|
||||
"size_class": "x",
|
||||
"repeat": 1,
|
||||
"order": 1
|
||||
},
|
||||
{
|
||||
"cell": "c4p4-s-r1",
|
||||
"config": "c4p4",
|
||||
"concurrency": 4,
|
||||
"prefetch": 4,
|
||||
"size_class": "s",
|
||||
"repeat": 1,
|
||||
"order": 2
|
||||
},
|
||||
{
|
||||
"cell": "c2p32-s-r1",
|
||||
"config": "c2p32",
|
||||
"concurrency": 2,
|
||||
"prefetch": 32,
|
||||
"size_class": "s",
|
||||
"repeat": 1,
|
||||
"order": 3
|
||||
},
|
||||
{
|
||||
"cell": "c4p4-l-r1",
|
||||
"config": "c4p4",
|
||||
"concurrency": 4,
|
||||
"prefetch": 4,
|
||||
"size_class": "l",
|
||||
"repeat": 1,
|
||||
"order": 4
|
||||
},
|
||||
{
|
||||
"cell": "c4p4-x-r1",
|
||||
"config": "c4p4",
|
||||
"concurrency": 4,
|
||||
"prefetch": 4,
|
||||
"size_class": "x",
|
||||
"repeat": 1,
|
||||
"order": 5
|
||||
},
|
||||
{
|
||||
"cell": "c2p2-l-r1",
|
||||
"config": "c2p2",
|
||||
"concurrency": 2,
|
||||
"prefetch": 2,
|
||||
"size_class": "l",
|
||||
"repeat": 1,
|
||||
"order": 6
|
||||
},
|
||||
{
|
||||
"cell": "c2p32-x-r1",
|
||||
"config": "c2p32",
|
||||
"concurrency": 2,
|
||||
"prefetch": 32,
|
||||
"size_class": "x",
|
||||
"repeat": 1,
|
||||
"order": 7
|
||||
},
|
||||
{
|
||||
"cell": "c2p2-s-r1",
|
||||
"config": "c2p2",
|
||||
"concurrency": 2,
|
||||
"prefetch": 2,
|
||||
"size_class": "s",
|
||||
"repeat": 1,
|
||||
"order": 8
|
||||
},
|
||||
{
|
||||
"cell": "c1p1-x-r1",
|
||||
"config": "c1p1",
|
||||
"concurrency": 1,
|
||||
"prefetch": 1,
|
||||
"size_class": "x",
|
||||
"repeat": 1,
|
||||
"order": 9
|
||||
},
|
||||
{
|
||||
"cell": "c1p1-l-r1",
|
||||
"config": "c1p1",
|
||||
"concurrency": 1,
|
||||
"prefetch": 1,
|
||||
"size_class": "l",
|
||||
"repeat": 1,
|
||||
"order": 10
|
||||
},
|
||||
{
|
||||
"cell": "c1p1-s-r1",
|
||||
"config": "c1p1",
|
||||
"concurrency": 1,
|
||||
"prefetch": 1,
|
||||
"size_class": "s",
|
||||
"repeat": 1,
|
||||
"order": 11
|
||||
},
|
||||
{
|
||||
"cell": "c2p32-l-r1",
|
||||
"config": "c2p32",
|
||||
"concurrency": 2,
|
||||
"prefetch": 32,
|
||||
"size_class": "l",
|
||||
"repeat": 1,
|
||||
"order": 12
|
||||
},
|
||||
{
|
||||
"cell": "c2p32-x-r2",
|
||||
"config": "c2p32",
|
||||
"concurrency": 2,
|
||||
"prefetch": 32,
|
||||
"size_class": "x",
|
||||
"repeat": 2,
|
||||
"order": 13
|
||||
},
|
||||
{
|
||||
"cell": "c2p2-s-r2",
|
||||
"config": "c2p2",
|
||||
"concurrency": 2,
|
||||
"prefetch": 2,
|
||||
"size_class": "s",
|
||||
"repeat": 2,
|
||||
"order": 14
|
||||
},
|
||||
{
|
||||
"cell": "c4p4-x-r2",
|
||||
"config": "c4p4",
|
||||
"concurrency": 4,
|
||||
"prefetch": 4,
|
||||
"size_class": "x",
|
||||
"repeat": 2,
|
||||
"order": 15
|
||||
},
|
||||
{
|
||||
"cell": "c1p1-x-r2",
|
||||
"config": "c1p1",
|
||||
"concurrency": 1,
|
||||
"prefetch": 1,
|
||||
"size_class": "x",
|
||||
"repeat": 2,
|
||||
"order": 16
|
||||
},
|
||||
{
|
||||
"cell": "c2p2-l-r2",
|
||||
"config": "c2p2",
|
||||
"concurrency": 2,
|
||||
"prefetch": 2,
|
||||
"size_class": "l",
|
||||
"repeat": 2,
|
||||
"order": 17
|
||||
},
|
||||
{
|
||||
"cell": "c2p2-x-r2",
|
||||
"config": "c2p2",
|
||||
"concurrency": 2,
|
||||
"prefetch": 2,
|
||||
"size_class": "x",
|
||||
"repeat": 2,
|
||||
"order": 18
|
||||
},
|
||||
{
|
||||
"cell": "c1p1-l-r2",
|
||||
"config": "c1p1",
|
||||
"concurrency": 1,
|
||||
"prefetch": 1,
|
||||
"size_class": "l",
|
||||
"repeat": 2,
|
||||
"order": 19
|
||||
},
|
||||
{
|
||||
"cell": "c1p1-s-r2",
|
||||
"config": "c1p1",
|
||||
"concurrency": 1,
|
||||
"prefetch": 1,
|
||||
"size_class": "s",
|
||||
"repeat": 2,
|
||||
"order": 20
|
||||
},
|
||||
{
|
||||
"cell": "c4p4-l-r2",
|
||||
"config": "c4p4",
|
||||
"concurrency": 4,
|
||||
"prefetch": 4,
|
||||
"size_class": "l",
|
||||
"repeat": 2,
|
||||
"order": 21
|
||||
},
|
||||
{
|
||||
"cell": "c4p4-s-r2",
|
||||
"config": "c4p4",
|
||||
"concurrency": 4,
|
||||
"prefetch": 4,
|
||||
"size_class": "s",
|
||||
"repeat": 2,
|
||||
"order": 22
|
||||
},
|
||||
{
|
||||
"cell": "c2p32-l-r2",
|
||||
"config": "c2p32",
|
||||
"concurrency": 2,
|
||||
"prefetch": 32,
|
||||
"size_class": "l",
|
||||
"repeat": 2,
|
||||
"order": 23
|
||||
},
|
||||
{
|
||||
"cell": "c2p32-s-r2",
|
||||
"config": "c2p32",
|
||||
"concurrency": 2,
|
||||
"prefetch": 32,
|
||||
"size_class": "s",
|
||||
"repeat": 2,
|
||||
"order": 24
|
||||
}
|
||||
],
|
||||
"upload_slack": 8,
|
||||
"bodies": {
|
||||
"s": {
|
||||
"kind": "noise",
|
||||
"width": 320,
|
||||
"height": 200,
|
||||
"bytes": 192323,
|
||||
"sha256": "dd245db29b537d420390fac805135b7a6a93c04d289e7d8d8265deb28f27d4df"
|
||||
},
|
||||
"l": {
|
||||
"kind": "noise",
|
||||
"width": 1800,
|
||||
"height": 1800,
|
||||
"bytes": 9724823,
|
||||
"sha256": "cd5dd6eae6c4a6c88adcde2276157b649970a3d65f8f0cc69e2c5eeed341ad32"
|
||||
},
|
||||
"x": {
|
||||
"kind": "flat",
|
||||
"width": 6000,
|
||||
"height": 6000,
|
||||
"bytes": 119755,
|
||||
"sha256": "0652f72ddb1d240172e8f0f4598ab72c29a0da365c9e822a5842b64ec6109a9a"
|
||||
}
|
||||
},
|
||||
"app_env": {
|
||||
"c2p2": {
|
||||
"DB_POOL_OVERFLOW": "1",
|
||||
"DB_POOL_SIZE": "2",
|
||||
"JOB_MAX_ATTEMPTS": "5",
|
||||
"JOB_TIMEOUT_SECONDS": "60",
|
||||
"LOG_LEVEL": "INFO",
|
||||
"RENDER_MEMORY_MB": "768",
|
||||
"WORKER_CONCURRENCY": "2",
|
||||
"WORKER_PREFETCH": "2"
|
||||
},
|
||||
"c4p4": {
|
||||
"DB_POOL_OVERFLOW": "1",
|
||||
"DB_POOL_SIZE": "2",
|
||||
"JOB_MAX_ATTEMPTS": "5",
|
||||
"JOB_TIMEOUT_SECONDS": "60",
|
||||
"LOG_LEVEL": "INFO",
|
||||
"RENDER_MEMORY_MB": "768",
|
||||
"WORKER_CONCURRENCY": "4",
|
||||
"WORKER_PREFETCH": "4"
|
||||
},
|
||||
"c2p32": {
|
||||
"DB_POOL_OVERFLOW": "1",
|
||||
"DB_POOL_SIZE": "2",
|
||||
"JOB_MAX_ATTEMPTS": "5",
|
||||
"JOB_TIMEOUT_SECONDS": "60",
|
||||
"LOG_LEVEL": "INFO",
|
||||
"RENDER_MEMORY_MB": "768",
|
||||
"WORKER_CONCURRENCY": "2",
|
||||
"WORKER_PREFETCH": "32"
|
||||
}
|
||||
},
|
||||
"cells": {
|
||||
"c2p2-x-r1": {
|
||||
"worker_id": "475b24d8d650e944d58284859cfed6aeab2d4eab9948ca7abcde73906d903a1f",
|
||||
"marks": {
|
||||
"drain": 1793910786.366664,
|
||||
"worker_start": 1793910789.779219,
|
||||
"steady": 1793910797.644604,
|
||||
"fixed": 1793910839.410008,
|
||||
"overload": 1793910878.737991,
|
||||
"recovery": 1793910919.344358,
|
||||
"end": 1793910962.209227
|
||||
},
|
||||
"consuming_ts": 1793910790.4,
|
||||
"memory_peak_bytes": 701472768,
|
||||
"timeline_samples": 176,
|
||||
"timeline_failures": 0
|
||||
},
|
||||
"c4p4-s-r1": {
|
||||
"worker_id": "3e76e350120f7f6cb81e19d99c917283f9f3dd2d31c791ebff2257fd9dec9cab",
|
||||
"marks": {
|
||||
"drain": 1793910963.754187,
|
||||
"worker_start": 1793910966.572253,
|
||||
"steady": 1793910971.611535,
|
||||
"fixed": 1793911012.768243,
|
||||
"overload": 1793911053.589357,
|
||||
"recovery": 1793911094.181323,
|
||||
"end": 1793911119.979236
|
||||
},
|
||||
"consuming_ts": 1793910967.185,
|
||||
"memory_peak_bytes": 153550848,
|
||||
"timeline_samples": 157,
|
||||
"timeline_failures": 0
|
||||
},
|
||||
"c2p32-s-r1": {
|
||||
"worker_id": "9a548bdb2c458b8b6bb7d8feb6cccbf020445f31a10cb651157b804149277e71",
|
||||
"marks": {
|
||||
"drain": 1793911121.512778,
|
||||
"worker_start": 1793911124.909323,
|
||||
"steady": 1793911129.983547,
|
||||
"fixed": 1793911171.162871,
|
||||
"overload": 1793911212.058685,
|
||||
"recovery": 1793911252.635956,
|
||||
"end": 1793911278.524287
|
||||
},
|
||||
"consuming_ts": 1793911125.548,
|
||||
"memory_peak_bytes": 121618432,
|
||||
"timeline_samples": 158,
|
||||
"timeline_failures": 0
|
||||
},
|
||||
"c4p4-l-r1": {
|
||||
"worker_id": "b76a7e231790c51cf8daa5907884c35fd733aa185b31d6a7502c5ff1ffffab16",
|
||||
"marks": {
|
||||
"drain": 1793911280.005287,
|
||||
"worker_start": 1793911290.689234,
|
||||
"steady": 1793911301.48384,
|
||||
"fixed": 1793911342.536057,
|
||||
"overload": 1793911382.63656,
|
||||
"recovery": 1793911423.144251,
|
||||
"end": 1793911443.773926
|
||||
},
|
||||
"consuming_ts": 1793911291.317,
|
||||
"memory_peak_bytes": 426397696,
|
||||
"timeline_samples": 164,
|
||||
"timeline_failures": 0
|
||||
}
|
||||
},
|
||||
"stand": {
|
||||
"postgres": {
|
||||
"image": "postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722",
|
||||
"version": "18.6 (Debian 18.6-1.pgdg13+2)",
|
||||
"settings": {
|
||||
"checkpoint_timeout": "5min",
|
||||
"fsync": "on",
|
||||
"jit": "on",
|
||||
"max_connections": "100",
|
||||
"max_wal_size": "1GB",
|
||||
"password_encryption": "scram-sha-256",
|
||||
"shared_buffers": "128MB",
|
||||
"synchronous_commit": "on",
|
||||
"wal_level": "replica"
|
||||
}
|
||||
},
|
||||
"valkey": {
|
||||
"image": "valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd",
|
||||
"version": "9.1.2"
|
||||
},
|
||||
"haproxy": {
|
||||
"image": "haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26"
|
||||
},
|
||||
"limits": {
|
||||
"api": {
|
||||
"nano_cpus": 1000000000,
|
||||
"memory": 536870912,
|
||||
"memory_swap": 536870912,
|
||||
"published_ports": {}
|
||||
},
|
||||
"postgres": {
|
||||
"nano_cpus": 1000000000,
|
||||
"memory": 1073741824,
|
||||
"memory_swap": 1073741824,
|
||||
"published_ports": {}
|
||||
},
|
||||
"valkey": {
|
||||
"nano_cpus": 500000000,
|
||||
"memory": 134217728,
|
||||
"memory_swap": 134217728,
|
||||
"published_ports": {}
|
||||
},
|
||||
"haproxy": {
|
||||
"nano_cpus": 500000000,
|
||||
"memory": 268435456,
|
||||
"memory_swap": 268435456,
|
||||
"published_ports": {}
|
||||
},
|
||||
"s3": {
|
||||
"nano_cpus": 1000000000,
|
||||
"memory": 536870912,
|
||||
"memory_swap": 536870912,
|
||||
"published_ports": {}
|
||||
},
|
||||
"rabbitmq": {
|
||||
"nano_cpus": 1000000000,
|
||||
"memory": 536870912,
|
||||
"memory_swap": 536870912,
|
||||
"published_ports": {}
|
||||
},
|
||||
"relay": {
|
||||
"nano_cpus": 250000000,
|
||||
"memory": 201326592,
|
||||
"memory_swap": 201326592,
|
||||
"published_ports": {}
|
||||
},
|
||||
"worker": {
|
||||
"nano_cpus": 2000000000,
|
||||
"memory": 1610612736,
|
||||
"memory_swap": 1610612736,
|
||||
"published_ports": {}
|
||||
}
|
||||
},
|
||||
"api_replicas": 2,
|
||||
"app_env": {
|
||||
"APP_HOST": "0.0.0.0",
|
||||
"APP_PORT": "8080",
|
||||
"CACHE_HOST": "valkey",
|
||||
"CACHE_PORT": "6379",
|
||||
"DB_HOST": "postgres",
|
||||
"DB_NAME": "bench",
|
||||
"DB_PORT": "5432",
|
||||
"DB_USER": "bench",
|
||||
"IDEMPOTENCY_TTL_SECONDS": "86400",
|
||||
"LOG_LEVEL": "INFO",
|
||||
"MALLOC_MMAP_THRESHOLD_": "131072",
|
||||
"S3_BUCKET": "bench",
|
||||
"S3_ENDPOINT": "http://s3:8333",
|
||||
"SESSION_TTL_SECONDS": "21600",
|
||||
"SHUTDOWN_TIMEOUT": "5",
|
||||
"UPLOAD_MAX_UNFINISHED": "60"
|
||||
},
|
||||
"api_tcp_rmem": "4096 131072 524288",
|
||||
"s3": {
|
||||
"image": "chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d",
|
||||
"command": [
|
||||
"server",
|
||||
"-ip=127.0.0.1",
|
||||
"-ip.bind=127.0.0.1",
|
||||
"-master.telemetry=false",
|
||||
"-master.electionTimeout=1s",
|
||||
"-volume.preStopSeconds=0",
|
||||
"-filer",
|
||||
"-s3",
|
||||
"-s3.ip.bind=0.0.0.0",
|
||||
"-s3.port=8333",
|
||||
"-s3.port.iceberg=0",
|
||||
"-s3.port.lance=0",
|
||||
"-s3.iam=false",
|
||||
"-s3.concurrentUploadLimitMB=64"
|
||||
]
|
||||
},
|
||||
"rabbitmq": {
|
||||
"image": "rabbitmq:4.3.7-alpine@sha256:b1f997f74103d08fc1f20a13890dc44fe6cfa51d528cf521b480ada988d3ffa5",
|
||||
"version": "4.3.7"
|
||||
}
|
||||
},
|
||||
"aborted": "StandError: c4p4-x-r1: no ready job after the drain:\n2026-11-05T20:44:04.880Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=4 prefetch=4 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-\n2026-11-05T20:44:04.887Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=4\n2026-11-05T20:44:05.280Z INFO src.worker [rid=-] worker stopped exit_code=0 cancelled=0 returned=0\n2026-11-05T20:44:08.660Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=4 prefetch=4 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-\n2026-11-05T20:44:08.666Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=4\n",
|
||||
"finished": "2026-11-05T20:44:11Z",
|
||||
"host_state_end": {
|
||||
"slept_seconds": 0.0,
|
||||
"swapouts_start": 950094,
|
||||
"swapouts_end": 950094,
|
||||
"power_end": "AC Power",
|
||||
"swapouts_delta": 0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
q,1793910786926.5,1793910786929.4,1793910787073.5,202,,8c6c9422-399e-4f4d-984b-0adedf5615e2,39e797bfa6f6,
|
||||
q,1793910787026.5,1793910787028.3,1793910787130.9,202,,1fb82fef-091d-48ec-a8a9-30d72345f966,7bbaa8668c87,
|
||||
q,1793910787126.5,1793910787127.3,1793910787223.3,202,,8f0cbefc-5a22-46bd-a126-7b64dd65030d,39e797bfa6f6,
|
||||
q,1793910787226.5,1793910787228.2,1793910787322.0,202,,5fce6b55-8b7c-4235-9528-dbbc59ca5a95,7bbaa8668c87,
|
||||
q,1793910787326.5,1793910787328.1,1793910787425.5,202,,544ee407-97f7-4561-9887-e80400cbd8e0,39e797bfa6f6,
|
||||
q,1793910787426.5,1793910787425.6,1793910787519.5,202,,164a0c4a-dbc2-4d6d-a1e8-89fab877ba82,7bbaa8668c87,
|
||||
q,1793910787526.5,1793910787527.7,1793910787624.7,202,,a2380693-dbde-4c8c-a918-dbd294fa12d1,39e797bfa6f6,
|
||||
q,1793910787626.5,1793910787628.1,1793910787724.4,202,,9d1807fa-0441-454d-8a7e-06e4c1863df7,7bbaa8668c87,
|
||||
q,1793910787726.5,1793910787728.7,1793910787825.6,202,,d3ed11d8-f556-4b18-a981-09d2c05207b5,39e797bfa6f6,
|
||||
q,1793910787826.5,1793910787825.6,1793910787921.5,202,,17c307b8-3fce-4cf7-93c6-c6f94150b62c,7bbaa8668c87,
|
||||
q,1793910787926.5,1793910787927.6,1793910788024.3,202,,8e316bef-326b-4dfc-ab77-c4e5c7e90169,39e797bfa6f6,
|
||||
q,1793910788026.5,1793910788028.8,1793910788124.5,202,,e483c11f-b585-442b-96d5-ae047cfeb573,7bbaa8668c87,
|
||||
q,1793910788126.5,1793910788127.1,1793910788224.3,202,,ce13c278-c136-45d0-91d8-401f108626ba,39e797bfa6f6,
|
||||
q,1793910788226.5,1793910788228.1,1793910788321.3,202,,dbb7dc38-8a0e-4f9c-9ca9-de72e47789c7,7bbaa8668c87,
|
||||
q,1793910788326.5,1793910788328.3,1793910788426.1,202,,d42c12ae-35e1-4000-9906-01b609a870f0,39e797bfa6f6,
|
||||
q,1793910788426.5,1793910788426.1,1793910788521.6,202,,39f1260a-2c56-4d73-88c8-f1271d8d2ebe,7bbaa8668c87,
|
||||
q,1793910788526.5,1793910788527.3,1793910788623.5,202,,47e62309-c5d5-4ee9-b859-c9c491596072,39e797bfa6f6,
|
||||
q,1793910788626.5,1793910788627.1,1793910788723.7,202,,9eabb7e1-728d-4c7b-ab9e-f5d514b776f4,7bbaa8668c87,
|
||||
q,1793910788726.5,1793910788727.9,1793910788825.9,202,,6df0d6ac-5fbc-497e-ad2e-402a63495493,39e797bfa6f6,
|
||||
q,1793910788826.5,1793910788826.0,1793910788923.2,202,,66f98ee2-ca95-407a-a2a1-128fc36afbfa,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 10.0,
|
||||
"duration_s": 3.0,
|
||||
"start": 1793910786.926,
|
||||
"end": 1793910788.926,
|
||||
"finished": 1793910788.923,
|
||||
"scheduled": 20,
|
||||
"sent": 20,
|
||||
"accepted": 20,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 20
|
||||
},
|
||||
"upload_p50_ms": 96.62,
|
||||
"upload_p95_ms": 102.64,
|
||||
"upload_max_ms": 144.13,
|
||||
"peak_inflight": 2,
|
||||
"gen_cpu_s": 0.018,
|
||||
"log": "/tmp/c2p2-x-r1-drain.log"
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
q,1793910839775.0,1793910839776.8,1793910839910.2,202,,f6b1e01c-a46d-4648-824f-1268c002e5b4,39e797bfa6f6,
|
||||
q,1793910841775.0,1793910841783.4,1793910841913.0,202,,573484f2-59d3-4a02-b033-0477d64ea3f4,7bbaa8668c87,
|
||||
q,1793910843775.0,1793910843777.1,1793910843906.6,202,,4f35f3d3-e838-4ca1-805e-08046b526d49,39e797bfa6f6,
|
||||
q,1793910845775.0,1793910845779.1,1793910845880.0,202,,43986b63-3b78-4f05-9c3b-03bd04523aaf,7bbaa8668c87,
|
||||
q,1793910847775.0,1793910847782.2,1793910847911.9,202,,26f611a2-a59a-452b-9ebc-df7fe04eb68f,39e797bfa6f6,
|
||||
q,1793910849775.0,1793910849781.4,1793910849911.3,202,,f6c2b007-9dcb-4bdc-aefa-0b8818d8b425,7bbaa8668c87,
|
||||
q,1793910851775.0,1793910851781.0,1793910851909.2,202,,e7cb9fc0-a6fa-47e7-a561-032f09f4045b,39e797bfa6f6,
|
||||
q,1793910853775.0,1793910853778.5,1793910853907.9,202,,be79a66a-f540-4d72-99dd-02c5efa7db17,7bbaa8668c87,
|
||||
q,1793910855775.0,1793910855776.8,1793910855907.7,202,,bb27dd73-bbdd-4e23-8568-06ddc9ee993b,39e797bfa6f6,
|
||||
q,1793910857775.0,1793910857778.3,1793910857900.9,202,,f747dd41-b784-4bd7-a8d0-2e5a85d4e6d2,7bbaa8668c87,
|
||||
q,1793910859775.0,1793910859779.3,1793910859908.0,202,,d716aefb-b545-4e74-86a5-3b1ac6525c9e,39e797bfa6f6,
|
||||
q,1793910861775.0,1793910861779.0,1793910861913.0,202,,806de701-a53c-4204-966c-bad5e2e35865,7bbaa8668c87,
|
||||
q,1793910863775.0,1793910863781.1,1793910863913.8,202,,c4ed389d-8930-4d0b-a5dd-52603add4a74,39e797bfa6f6,
|
||||
q,1793910865775.0,1793910865780.1,1793910865910.4,202,,28dc600c-2cb8-4ca5-bb8d-87966c1d20a7,7bbaa8668c87,
|
||||
q,1793910867775.0,1793910867783.2,1793910867905.7,202,,b05c0d67-ac51-4fcf-a32d-d7c742b1a58f,39e797bfa6f6,
|
||||
q,1793910869775.0,1793910869779.5,1793910869902.4,202,,0af33e81-b4cb-441c-89ff-66b1a2e3834b,7bbaa8668c87,
|
||||
q,1793910871775.0,1793910871783.5,1793910871909.3,202,,2b403af2-14bf-422f-9c53-c209c6d431dc,39e797bfa6f6,
|
||||
q,1793910873775.0,1793910873776.9,1793910873906.4,202,,65c19aa9-b11a-469e-9dc9-b1234ec68b1a,7bbaa8668c87,
|
||||
q,1793910875775.0,1793910875779.6,1793910875909.7,202,,1a871ebc-21b3-43ef-836d-31739e0dcc94,39e797bfa6f6,
|
||||
q,1793910877775.0,1793910877783.5,1793910877913.5,202,,327c403f-1b16-41c4-bdd2-6431c4961497,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 0.5,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793910839.775,
|
||||
"end": 1793910879.775,
|
||||
"finished": 1793910877.913,
|
||||
"scheduled": 20,
|
||||
"sent": 20,
|
||||
"accepted": 20,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 20
|
||||
},
|
||||
"upload_p50_ms": 129.5,
|
||||
"upload_p95_ms": 133.33,
|
||||
"upload_max_ms": 134.02,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.054,
|
||||
"log": "/tmp/c2p2-x-r1-fixed.log"
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
q,1793910879131.7,1793910879137.8,1793910879269.9,202,,3130ea87-4583-4bc7-b032-2588ff35a8cf,39e797bfa6f6,
|
||||
q,1793910879300.2,1793910879302.8,1793910879407.0,202,,12e62e0a-526a-4d4c-8afd-04aab001878c,7bbaa8668c87,
|
||||
q,1793910879468.6,1793910879469.8,1793910879574.7,202,,41afbe08-ebc4-420e-88b9-4f19a88361d6,39e797bfa6f6,
|
||||
q,1793910879637.1,1793910879638.6,1793910879746.1,202,,38aecf82-ab2d-454c-a879-169eb740bcdc,7bbaa8668c87,
|
||||
q,1793910879805.6,1793910879807.3,1793910879909.3,202,,bdbf4811-0ad6-4f1f-b955-09b2dbd2f9e3,39e797bfa6f6,
|
||||
q,1793910879974.0,1793910879975.3,1793910880078.5,202,,5dadb0e0-bd32-4ab4-9aaa-a262f350b71f,7bbaa8668c87,
|
||||
q,1793910880142.5,1793910880144.6,1793910880247.9,202,,84233de3-4619-464c-a152-44667ba2633a,39e797bfa6f6,
|
||||
q,1793910880311.0,1793910880312.9,1793910880420.8,202,,e725b708-f720-420d-a572-5f02c8415ac1,7bbaa8668c87,
|
||||
q,1793910880479.4,1793910880481.6,1793910880584.5,202,,2903d9be-e649-4b48-a7ca-f5a688ca41a9,39e797bfa6f6,
|
||||
q,1793910880647.9,1793910880652.6,1793910880756.3,202,,062ebbfc-696b-4e68-8cb3-1e6fa4464554,7bbaa8668c87,
|
||||
q,1793910880816.3,1793910880818.7,1793910880922.7,202,,8dc716e0-9eb8-4b45-9a55-7dcfe7a84d1d,39e797bfa6f6,
|
||||
q,1793910880984.8,1793910880986.8,1793910881096.1,202,,342e77fe-3000-480c-9bc4-abf5b34504e9,7bbaa8668c87,
|
||||
q,1793910881153.3,1793910881154.8,1793910881258.1,202,,4575c606-b171-47a8-bd53-127dec61fde9,39e797bfa6f6,
|
||||
q,1793910881321.7,1793910881323.7,1793910881430.5,202,,b65ed9f9-41f6-4272-af4d-97b820122ded,7bbaa8668c87,
|
||||
q,1793910881490.2,1793910881492.9,1793910881596.2,202,,2ae8603a-9a0c-48e7-b067-ce7e4c07f9e3,39e797bfa6f6,
|
||||
q,1793910881658.6,1793910881659.7,1793910881768.4,202,,f2f7cf73-dac3-40a5-a8d9-1cdf3e21e0a6,7bbaa8668c87,
|
||||
q,1793910881827.1,1793910881828.8,1793910881931.5,202,,83b68628-2b1b-4125-94ba-4750dc5df729,39e797bfa6f6,
|
||||
q,1793910881995.6,1793910881997.6,1793910882100.3,202,,a39a275a-2f08-46aa-b041-413178f10f9b,7bbaa8668c87,
|
||||
q,1793910882164.0,1793910882164.6,1793910882267.6,202,,86bb437d-2e55-433b-892d-9803fdd95334,39e797bfa6f6,
|
||||
q,1793910882332.5,1793910882334.7,1793910882448.4,202,,df9c563a-42e3-48b6-b17a-dfbf9c7abf04,7bbaa8668c87,
|
||||
q,1793910882500.9,1793910882501.6,1793910882605.2,202,,3ba7d028-3664-48d9-a698-f1356e852c1b,39e797bfa6f6,
|
||||
q,1793910882669.4,1793910882670.6,1793910882774.6,202,,0d534025-c8fd-4ee3-84de-85e0e95e0281,7bbaa8668c87,
|
||||
q,1793910882837.9,1793910882839.8,1793910882942.0,202,,3f6355d2-33f0-43fb-8bd8-26812a2207c1,39e797bfa6f6,
|
||||
q,1793910883006.3,1793910883008.3,1793910883116.7,202,,510fedf8-c9e8-4b6b-a72c-e922b08effc6,7bbaa8668c87,
|
||||
q,1793910883174.8,1793910883177.8,1793910883279.2,202,,1b00f69f-c2d6-40f1-b12a-c9ea9aa0d503,39e797bfa6f6,
|
||||
q,1793910883343.2,1793910883343.6,1793910883451.1,202,,1de47f93-3c7f-4b94-bc9c-e5915aad65c5,7bbaa8668c87,
|
||||
q,1793910883511.7,1793910883512.6,1793910883619.3,202,,ae21d05c-1d37-46b4-a928-fcd3d5303490,39e797bfa6f6,
|
||||
q,1793910883680.2,1793910883680.8,1793910883790.4,202,,15a15e20-e42a-4d92-ae6a-7d78e4add863,7bbaa8668c87,
|
||||
q,1793910883848.6,1793910883851.7,1793910883959.5,202,,8527ce8c-cce6-4d01-bd2a-5fbdd40cfcbc,39e797bfa6f6,
|
||||
q,1793910884017.1,1793910884020.1,1793910884124.3,202,,b9d6dd6b-63e9-4aba-90f5-1a012b9deceb,7bbaa8668c87,
|
||||
q,1793910884185.5,1793910884186.6,1793910884290.0,202,,78e3c7ff-b420-48f9-b328-7d32337c28a6,39e797bfa6f6,
|
||||
q,1793910884354.0,1793910884354.6,1793910884464.5,202,,4da83b40-e8e3-4559-b752-4bcc6d709270,7bbaa8668c87,
|
||||
q,1793910884522.5,1793910884523.9,1793910884633.0,202,,a344ae1f-8a57-471d-b58c-aee492d91095,39e797bfa6f6,
|
||||
q,1793910884690.9,1793910884691.9,1793910884798.7,202,,17906a5e-bedf-4eab-8664-9da70647518f,7bbaa8668c87,
|
||||
q,1793910884859.4,1793910884860.6,1793910884964.2,202,,f5cfba83-a72e-441b-971f-5433ec299e83,39e797bfa6f6,
|
||||
q,1793910885027.8,1793910885028.8,1793910885135.9,202,,c70919aa-fda7-4de6-b199-aa78a01a0ee4,7bbaa8668c87,
|
||||
q,1793910885196.3,1793910885197.6,1793910885302.3,202,,9f4463cb-1925-4ffe-996a-a6eede1e4165,39e797bfa6f6,
|
||||
q,1793910885364.8,1793910885365.7,1793910885470.5,202,,88fc08d1-cd5c-447c-881a-247aa6d6d0e4,7bbaa8668c87,
|
||||
q,1793910885533.2,1793910885534.6,1793910885639.6,202,,da16b6c2-edc6-411f-866e-22b912742c20,39e797bfa6f6,
|
||||
q,1793910885701.7,1793910885704.2,1793910885809.9,202,,1cfbccad-9570-4314-8ead-93f8347b090e,7bbaa8668c87,
|
||||
q,1793910885870.1,1793910885871.7,1793910885978.6,202,,ae9fc576-8133-4bfd-97b3-1f78886df7a8,39e797bfa6f6,
|
||||
q,1793910886038.6,1793910886039.6,1793910886141.8,202,,ee43baf6-edb9-4ddd-8498-511f6c520100,7bbaa8668c87,
|
||||
q,1793910886207.1,1793910886208.9,1793910886313.1,202,,5640ec64-b875-442a-b9b2-41924627a9ce,39e797bfa6f6,
|
||||
q,1793910886375.5,1793910886376.6,1793910886484.1,202,,f3d5e18e-9f4d-44e0-91d7-e104a013198a,7bbaa8668c87,
|
||||
q,1793910886544.0,1793910886544.6,1793910886649.5,202,,eb7182be-6517-4b42-9628-f27de798ff8c,39e797bfa6f6,
|
||||
q,1793910886712.4,1793910886713.6,1793910886816.7,202,,5d2fe0f4-c3e8-4e92-8c55-9264f7868fa8,7bbaa8668c87,
|
||||
q,1793910886880.9,1793910886885.8,1793910886989.8,202,,4c41e4a3-1b88-4a76-9748-dd5097c1613c,39e797bfa6f6,
|
||||
q,1793910887049.4,1793910887051.0,1793910887157.8,202,,97dd5c8c-ee10-4a35-a7a4-cf22ca5b133a,7bbaa8668c87,
|
||||
q,1793910887217.8,1793910887218.7,1793910887324.8,202,,81c176d0-a34b-4833-aa7b-b41f76daa768,39e797bfa6f6,
|
||||
q,1793910887386.3,1793910887388.2,1793910887496.7,202,,8fc479f6-c3c1-4bb9-9f5d-f748004180da,7bbaa8668c87,
|
||||
q,1793910887554.8,1793910887555.6,1793910887659.8,202,,a9dafb84-ef9a-4389-9218-494be124bf1a,39e797bfa6f6,
|
||||
q,1793910887723.2,1793910887724.6,1793910887830.7,202,,16829385-49da-4438-a6a8-2ec501bcde6b,7bbaa8668c87,
|
||||
q,1793910887891.7,1793910887895.3,1793910888000.2,202,,e5560c3e-719e-4c47-81e7-69b2901985ce,39e797bfa6f6,
|
||||
q,1793910888060.1,1793910888062.6,1793910888168.6,202,,6d9b808b-30a1-46c0-8887-dc47cd2c4eef,7bbaa8668c87,
|
||||
q,1793910888228.6,1793910888230.3,1793910888333.6,202,,aada7922-e9ad-4a4c-988a-00df28937250,39e797bfa6f6,
|
||||
q,1793910888397.1,1793910888399.6,1793910888505.6,202,,fe838370-46a6-45ed-8c35-7ab69c0490c9,7bbaa8668c87,
|
||||
q,1793910888565.5,1793910888568.8,1793910888673.5,202,,bb630203-1fab-4f2b-9a0c-768f5fd7849e,39e797bfa6f6,
|
||||
q,1793910888734.0,1793910888736.5,1793910888840.2,202,,c121677d-6dd1-4c33-8440-ff703a622404,7bbaa8668c87,
|
||||
q,1793910888902.4,1793910888904.8,1793910889011.3,202,,4c568e8c-d283-4317-bada-1c3b6cd2481f,39e797bfa6f6,
|
||||
q,1793910889070.9,1793910889071.6,1793910889180.1,202,,f555859d-db7a-4396-9eea-0b993ed24698,7bbaa8668c87,
|
||||
q,1793910889239.4,1793910889240.6,1793910889352.8,202,,fc670baa-cf5f-41fd-84e3-5a0da0b87d7d,39e797bfa6f6,
|
||||
q,1793910889407.8,1793910889408.7,1793910889522.9,202,,be3c815d-ec72-4739-840c-63be1e125323,7bbaa8668c87,
|
||||
q,1793910889576.3,1793910889577.6,1793910889681.3,202,,a0d71ff3-7f8d-4075-a46e-fcc2373d4ce8,39e797bfa6f6,
|
||||
q,1793910889744.7,1793910889746.3,1793910889860.3,202,,d7430d6e-bfcf-4155-b51d-b61c9f9aa465,7bbaa8668c87,
|
||||
q,1793910889913.2,1793910889913.9,1793910890021.1,202,,b571c029-c394-4a4f-9b6a-5940760844cf,39e797bfa6f6,
|
||||
q,1793910890081.7,1793910890083.3,1793910890193.8,202,,73349b4c-2afe-4be4-8b28-9a9c5c9e9bc2,7bbaa8668c87,
|
||||
q,1793910890250.1,1793910890251.7,1793910890357.7,202,,a55ebc78-2167-4057-8ada-88f1f13974b6,39e797bfa6f6,
|
||||
q,1793910890418.6,1793910890420.0,1793910890525.1,202,,56a5e97b-8e8a-4a24-b543-a40c1688b8b3,7bbaa8668c87,
|
||||
q,1793910890587.0,1793910890587.8,1793910890703.2,202,,f0a74a60-a1bd-4aca-8b21-ded449d5ccc0,39e797bfa6f6,
|
||||
q,1793910890755.5,1793910890761.6,1793910890872.2,202,,21d64a7b-c189-4d7c-bf15-5f8ad7fd86a7,7bbaa8668c87,
|
||||
q,1793910890924.0,1793910890925.7,1793910891031.2,202,,568bb57e-c23b-4643-ad65-c82fe0b31fa2,39e797bfa6f6,
|
||||
q,1793910891092.4,1793910891093.8,1793910891197.7,202,,6a4dfb63-d746-4d99-bfd9-40d7ef06f766,7bbaa8668c87,
|
||||
q,1793910891260.9,1793910891266.7,1793910891374.2,202,,734d898b-5739-411d-bdad-424236fa8f25,39e797bfa6f6,
|
||||
q,1793910891429.3,1793910891431.6,1793910891537.2,202,,7e6d116c-824c-48ca-97aa-d78c7b34ba9e,7bbaa8668c87,
|
||||
q,1793910891597.8,1793910891600.0,1793910891705.9,202,,85dbf075-ac05-4135-a45f-e08340579089,39e797bfa6f6,
|
||||
q,1793910891766.3,1793910891772.6,1793910891877.8,202,,81f016d1-fef5-481d-8a39-0875d59ca836,7bbaa8668c87,
|
||||
q,1793910891934.7,1793910891936.6,1793910892043.2,202,,082aa566-b596-40b0-bbe4-042e24a96562,39e797bfa6f6,
|
||||
q,1793910892103.2,1793910892108.6,1793910892211.9,202,,44ce1067-1f0f-4c16-887d-c06df8d05f82,7bbaa8668c87,
|
||||
q,1793910892271.6,1793910892273.8,1793910892384.0,202,,d0b2e75c-183d-4c5b-a87b-b8912d3110ed,39e797bfa6f6,
|
||||
q,1793910892440.1,1793910892441.8,1793910892545.6,202,,035bfed0-7772-4738-954b-ccbbbb0db189,7bbaa8668c87,
|
||||
q,1793910892608.6,1793910892612.8,1793910892719.3,202,,79148b35-7f7f-46d8-bac5-e202658c6760,39e797bfa6f6,
|
||||
q,1793910892777.0,1793910892777.8,1793910892886.6,202,,00260b16-ef74-4f2c-bd02-9d32a7f3eee3,7bbaa8668c87,
|
||||
q,1793910892945.5,1793910892947.7,1793910893056.6,202,,5a01b98b-c40c-4628-a3f5-0c7f506fd7d7,39e797bfa6f6,
|
||||
q,1793910893113.9,1793910893114.7,1793910893220.1,202,,40399572-0b69-4a84-bf8c-e7e48fddaeab,7bbaa8668c87,
|
||||
q,1793910893282.4,1793910893284.7,1793910893394.3,202,,672b4e92-bedf-412d-bf8c-910552ab4c2c,39e797bfa6f6,
|
||||
q,1793910893450.9,1793910893456.6,1793910893565.5,202,,31b12660-973e-40e8-b0e8-84efc44270ac,7bbaa8668c87,
|
||||
q,1793910893619.3,1793910893620.8,1793910893737.8,202,,f6e7d88d-229d-495e-8a0d-dd3ff6a77f96,39e797bfa6f6,
|
||||
q,1793910893787.8,1793910893793.6,1793910893897.8,202,,13822f63-8720-4c93-a810-b3b061a9f3a5,7bbaa8668c87,
|
||||
q,1793910893956.2,1793910893959.3,1793910894066.0,202,,b19c3e12-2a73-49d3-8211-887c273b923a,39e797bfa6f6,
|
||||
q,1793910894124.7,1793910894127.8,1793910894233.2,202,,167ab4f0-f7b0-4167-b921-b8ebaae1675b,7bbaa8668c87,
|
||||
q,1793910894293.2,1793910894293.8,1793910894409.2,202,,f5d8b9ec-2293-420c-b8aa-aa2bef22ed78,39e797bfa6f6,
|
||||
q,1793910894461.6,1793910894465.6,1793910894571.1,202,,2c8ba7cf-2020-4fcc-a7ca-d8909bc5e726,7bbaa8668c87,
|
||||
q,1793910894630.1,1793910894631.7,1793910894740.6,202,,21cda083-80ed-4b85-8e37-0bae5992ab05,39e797bfa6f6,
|
||||
q,1793910894798.6,1793910894800.3,1793910894917.3,202,,4394a81c-78fe-4191-abb6-31ccad42c601,7bbaa8668c87,
|
||||
q,1793910894967.0,1793910894967.6,1793910895084.0,202,,b89f1b03-3064-44bd-9384-ef675fb2048d,39e797bfa6f6,
|
||||
q,1793910895135.5,1793910895137.6,1793910895241.6,202,,6a8e143d-7a9c-438d-bd16-5688e293d187,7bbaa8668c87,
|
||||
q,1793910895303.9,1793910895307.3,1793910895417.8,202,,a334dab9-882f-443b-aff1-4a5625163e1e,39e797bfa6f6,
|
||||
q,1793910895472.4,1793910895473.8,1793910895583.2,202,,43e56ba8-e6f8-408a-b058-c34987e945f3,7bbaa8668c87,
|
||||
q,1793910895640.9,1793910895642.3,1793910895750.7,202,,c76d5a2a-2d67-4882-88f2-532635007c52,39e797bfa6f6,
|
||||
q,1793910895809.3,1793910895813.8,1793910895917.7,202,,bbaa5046-8d98-43c4-ae0e-57de3052d62f,7bbaa8668c87,
|
||||
q,1793910895977.8,1793910895981.0,1793910896084.3,202,,f9832245-8489-49cd-93eb-bde73739ada6,39e797bfa6f6,
|
||||
q,1793910896146.2,1793910896149.3,1793910896255.9,202,,ca4f0b64-8ae9-4ede-b965-edb55e7353b8,7bbaa8668c87,
|
||||
q,1793910896314.7,1793910896317.7,1793910896435.9,202,,36965987-a5a2-473d-b10f-252d04dcbf85,39e797bfa6f6,
|
||||
q,1793910896483.2,1793910896486.5,1793910896592.1,202,,70651bd6-4413-4f60-bfeb-fdcc475ec76c,7bbaa8668c87,
|
||||
q,1793910896651.6,1793910896656.8,1793910896759.8,202,,e7d210d8-272f-48a3-9fee-ec0036cc65ff,39e797bfa6f6,
|
||||
q,1793910896820.1,1793910896823.0,1793910896931.0,202,,2202d22b-2d4f-4dcd-ad59-d19edd318678,7bbaa8668c87,
|
||||
q,1793910896988.5,1793910896994.6,1793910897101.8,202,,50f0810f-0c8b-4752-99f5-e8289462eb2c,39e797bfa6f6,
|
||||
q,1793910897157.0,1793910897160.6,1793910897265.5,202,,822666b6-0d7c-4975-aaf0-8835b538aa54,7bbaa8668c87,
|
||||
q,1793910897325.5,1793910897329.5,1793910897436.8,202,,2f88e3b7-7177-488b-b59f-c60300c4c286,39e797bfa6f6,
|
||||
q,1793910897493.9,1793910897497.2,1793910897606.1,202,,6a63a5c3-8657-45f5-ad1a-a9180c786c05,7bbaa8668c87,
|
||||
q,1793910897662.4,1793910897668.5,1793910897777.4,202,,d0b6d584-56fa-4adc-9f8b-3543c61feb82,39e797bfa6f6,
|
||||
q,1793910897830.8,1793910897837.5,1793910897948.3,202,,99052150-8787-4b6e-8a47-00ef9ce8d7d1,7bbaa8668c87,
|
||||
q,1793910897999.3,1793910898002.0,1793910898106.0,202,,a43a9632-cfe6-44d4-babb-586ae402b046,39e797bfa6f6,
|
||||
q,1793910898167.8,1793910898169.3,1793910898274.6,202,,491709a5-c784-4cb7-9e1b-211d7e817d83,7bbaa8668c87,
|
||||
q,1793910898336.2,1793910898337.2,1793910898341.1,429,5,,39e797bfa6f6,
|
||||
q,1793910898504.7,1793910898506.2,1793910898508.9,429,5,,7bbaa8668c87,
|
||||
q,1793910898673.1,1793910898675.7,1793910898678.4,429,5,,39e797bfa6f6,
|
||||
q,1793910898841.6,1793910898843.8,1793910898949.7,202,,ce959cc7-d675-4dc9-b4f7-bdf517b58de0,7bbaa8668c87,
|
||||
q,1793910899010.1,1793910899012.0,1793910899121.4,202,,61bf8b0b-57d1-4d80-bf44-1330397d62aa,39e797bfa6f6,
|
||||
q,1793910899178.5,1793910899180.8,1793910899183.6,429,5,,7bbaa8668c87,
|
||||
q,1793910899347.0,1793910899348.7,1793910899351.7,429,5,,39e797bfa6f6,
|
||||
q,1793910899515.4,1793910899518.4,1793910899626.8,202,,b0cd40c7-a682-4acc-ad1d-5e3b892a5c37,7bbaa8668c87,
|
||||
q,1793910899683.9,1793910899687.0,1793910899795.4,202,,88ef0076-ee18-43f1-98d1-913a1e577a76,39e797bfa6f6,
|
||||
q,1793910899852.4,1793910899852.8,1793910899855.6,429,5,,7bbaa8668c87,
|
||||
q,1793910900020.8,1793910900025.0,1793910900027.9,429,5,,39e797bfa6f6,
|
||||
q,1793910900189.3,1793910900190.8,1793910900302.5,202,,1fdc70f4-680f-467a-8cb6-7e6910a77656,7bbaa8668c87,
|
||||
q,1793910900357.7,1793910900359.6,1793910900478.3,202,,4a4c15cd-626e-4cca-abd0-d776f651752f,39e797bfa6f6,
|
||||
q,1793910900526.2,1793910900526.6,1793910900530.3,429,5,,7bbaa8668c87,
|
||||
q,1793910900694.7,1793910900698.3,1793910900701.1,429,5,,39e797bfa6f6,
|
||||
q,1793910900863.1,1793910900864.3,1793910900968.7,202,,9f9c79be-0f26-4a31-a30b-a85c57acb53c,7bbaa8668c87,
|
||||
q,1793910901031.6,1793910901033.6,1793910901149.7,202,,95f5e24c-424d-4845-b18d-5380614f9e0c,39e797bfa6f6,
|
||||
q,1793910901200.1,1793910901205.3,1793910901209.5,429,5,,7bbaa8668c87,
|
||||
q,1793910901368.5,1793910901368.8,1793910901373.5,429,5,,39e797bfa6f6,
|
||||
q,1793910901537.0,1793910901538.6,1793910901648.4,202,,484fb7eb-e37c-4f5c-87dc-cc4c7e79d1d9,7bbaa8668c87,
|
||||
q,1793910901705.4,1793910901706.8,1793910901819.5,202,,45235b9b-4f65-453e-b014-8ecf75ba8f78,39e797bfa6f6,
|
||||
q,1793910901873.9,1793910901875.6,1793910901879.5,429,5,,7bbaa8668c87,
|
||||
q,1793910902042.4,1793910902044.0,1793910902050.0,429,5,,39e797bfa6f6,
|
||||
q,1793910902210.8,1793910902211.3,1793910902214.2,429,5,,7bbaa8668c87,
|
||||
q,1793910902379.3,1793910902379.7,1793910902496.6,202,,383ed374-119e-4130-8aab-e3404b8f744b,39e797bfa6f6,
|
||||
q,1793910902547.7,1793910902555.6,1793910902672.1,202,,193d6a57-1381-4285-991a-9e6cd74e2361,7bbaa8668c87,
|
||||
q,1793910902716.2,1793910902718.0,1793910902721.2,429,5,,39e797bfa6f6,
|
||||
q,1793910902884.7,1793910902886.0,1793910902888.5,429,5,,7bbaa8668c87,
|
||||
q,1793910903053.1,1793910903056.3,1793910903160.3,202,,e660c9ba-0f4c-4be4-84b4-fd3dbb32d464,39e797bfa6f6,
|
||||
q,1793910903221.6,1793910903222.8,1793910903335.1,202,,5a651399-e565-4b6c-bae9-3b48ae64ed60,7bbaa8668c87,
|
||||
q,1793910903390.0,1793910903390.4,1793910903393.4,429,5,,39e797bfa6f6,
|
||||
q,1793910903558.5,1793910903565.6,1793910903568.7,429,5,,7bbaa8668c87,
|
||||
q,1793910903727.0,1793910903728.7,1793910903831.4,202,,6c3413cb-3de8-4bbf-9028-44583efe3607,39e797bfa6f6,
|
||||
q,1793910903895.4,1793910903896.8,1793910904006.6,202,,3b6c611b-ba3a-4389-b24d-c05ec58ec8d5,7bbaa8668c87,
|
||||
q,1793910904063.9,1793910904070.1,1793910904073.1,429,5,,39e797bfa6f6,
|
||||
q,1793910904232.3,1793910904233.6,1793910904236.6,429,5,,7bbaa8668c87,
|
||||
q,1793910904400.8,1793910904403.3,1793910904508.4,202,,b4005a2c-a35e-4985-805d-1b76f8d4d6a5,39e797bfa6f6,
|
||||
q,1793910904569.3,1793910904571.9,1793910904684.5,202,,7f5e79f1-ad3f-474d-8a62-be323a79e064,7bbaa8668c87,
|
||||
q,1793910904737.7,1793910904739.8,1793910904742.5,429,5,,39e797bfa6f6,
|
||||
q,1793910904906.2,1793910904907.6,1793910904910.7,429,5,,7bbaa8668c87,
|
||||
q,1793910905074.6,1793910905075.6,1793910905179.3,202,,2cb1fbd5-7536-45ee-a858-59d680bd8eb6,39e797bfa6f6,
|
||||
q,1793910905243.1,1793910905243.6,1793910905357.1,202,,de1e043e-afc4-4deb-8837-11961cf58684,7bbaa8668c87,
|
||||
q,1793910905411.6,1793910905417.6,1793910905420.7,429,5,,39e797bfa6f6,
|
||||
q,1793910905580.0,1793910905582.1,1793910905585.4,429,5,,7bbaa8668c87,
|
||||
q,1793910905748.5,1793910905749.6,1793910905856.4,202,,ef977c70-447d-4c2d-adc6-36bdfdcbef4c,39e797bfa6f6,
|
||||
q,1793910905916.9,1793910905917.6,1793910906038.3,202,,99b24604-2c76-4396-8680-c66ea01d4ea2,7bbaa8668c87,
|
||||
q,1793910906085.4,1793910906087.1,1793910906089.8,429,5,,39e797bfa6f6,
|
||||
q,1793910906253.9,1793910906256.6,1793910906259.7,429,5,,7bbaa8668c87,
|
||||
q,1793910906422.3,1793910906424.3,1793910906528.9,202,,35af00e6-e1aa-4c57-9439-026d4ce456c2,39e797bfa6f6,
|
||||
q,1793910906590.8,1793910906596.6,1793910906701.7,202,,5aca7369-f06c-4b77-aace-f5f9bb1ecd4c,7bbaa8668c87,
|
||||
q,1793910906759.2,1793910906765.6,1793910906768.5,429,5,,39e797bfa6f6,
|
||||
q,1793910906927.7,1793910906929.8,1793910906932.7,429,5,,7bbaa8668c87,
|
||||
q,1793910907096.2,1793910907097.6,1793910907100.6,429,5,,39e797bfa6f6,
|
||||
q,1793910907264.6,1793910907267.0,1793910907372.9,202,,d039d677-57b5-40db-9083-61b4b16dbb11,7bbaa8668c87,
|
||||
q,1793910907433.1,1793910907435.3,1793910907552.7,202,,7d5f62c6-3e88-4465-965e-e681ba7efd1c,39e797bfa6f6,
|
||||
q,1793910907601.5,1793910907604.8,1793910907607.9,429,5,,7bbaa8668c87,
|
||||
q,1793910907770.0,1793910907773.0,1793910907775.9,429,5,,39e797bfa6f6,
|
||||
q,1793910907938.5,1793910907939.8,1793910908043.1,202,,f53aefd7-36d4-4703-bb8b-96499ec75aff,7bbaa8668c87,
|
||||
q,1793910908106.9,1793910908109.5,1793910908221.3,202,,0f47d4bc-4204-4230-a4b5-16ec026d1ea5,39e797bfa6f6,
|
||||
q,1793910908275.4,1793910908276.8,1793910908279.6,429,5,,7bbaa8668c87,
|
||||
q,1793910908443.9,1793910908447.5,1793910908450.4,429,5,,39e797bfa6f6,
|
||||
q,1793910908612.3,1793910908612.8,1793910908716.8,202,,b7f2cf8e-b1c9-42eb-8fa1-3d7506908280,7bbaa8668c87,
|
||||
q,1793910908780.8,1793910908781.8,1793910908902.0,202,,fb4b77cd-27b2-4ccf-b6fa-be6ecd0095b2,39e797bfa6f6,
|
||||
q,1793910908949.2,1793910908951.6,1793910908954.2,429,5,,7bbaa8668c87,
|
||||
q,1793910909117.7,1793910909122.1,1793910909125.3,429,5,,39e797bfa6f6,
|
||||
q,1793910909286.2,1793910909286.5,1793910909394.2,202,,b287eed0-dd43-4ec6-bc94-15027c4f1b8e,7bbaa8668c87,
|
||||
q,1793910909454.6,1793910909459.8,1793910909569.2,202,,fc4800a1-c7d0-4866-8b73-56b02ddaef11,39e797bfa6f6,
|
||||
q,1793910909623.1,1793910909626.5,1793910909629.2,429,5,,7bbaa8668c87,
|
||||
q,1793910909791.5,1793910909797.6,1793910909800.7,429,5,,39e797bfa6f6,
|
||||
q,1793910909960.0,1793910909961.9,1793910910069.1,202,,ca1352de-6a88-43b4-8f98-566c4bf84084,7bbaa8668c87,
|
||||
q,1793910910128.5,1793910910129.7,1793910910244.6,202,,497592a3-4517-450b-8fd3-d3b0b342aeef,39e797bfa6f6,
|
||||
q,1793910910296.9,1793910910300.3,1793910910303.0,429,5,,7bbaa8668c87,
|
||||
q,1793910910465.4,1793910910467.0,1793910910470.4,429,5,,39e797bfa6f6,
|
||||
q,1793910910633.8,1793910910634.6,1793910910738.8,202,,2e053036-2f2a-4553-bc71-d3c58ce04754,7bbaa8668c87,
|
||||
q,1793910910802.3,1793910910803.6,1793910910918.7,202,,1c34eb1e-0c8c-477a-b2b6-dcd55e32b346,39e797bfa6f6,
|
||||
q,1793910910970.8,1793910910975.0,1793910910978.1,429,5,,7bbaa8668c87,
|
||||
q,1793910911139.2,1793910911140.6,1793910911143.7,429,5,,39e797bfa6f6,
|
||||
q,1793910911307.7,1793910911308.7,1793910911416.3,202,,b98ca6ca-7966-4136-bdbf-b426313b1e21,7bbaa8668c87,
|
||||
q,1793910911476.1,1793910911476.6,1793910911588.4,202,,ce315f4f-0211-4eff-a9e6-f55f66597335,39e797bfa6f6,
|
||||
q,1793910911644.6,1793910911645.6,1793910911650.2,429,5,,7bbaa8668c87,
|
||||
q,1793910911813.1,1793910911813.7,1793910911816.8,429,5,,39e797bfa6f6,
|
||||
q,1793910911981.5,1793910911983.2,1793910911986.1,429,5,,7bbaa8668c87,
|
||||
q,1793910912150.0,1793910912151.4,1793910912259.7,202,,1a439d67-c654-47a8-8589-f8737ed42f15,39e797bfa6f6,
|
||||
q,1793910912318.4,1793910912321.0,1793910912430.9,202,,280fcf9a-7407-4014-97b9-eca05257db86,7bbaa8668c87,
|
||||
q,1793910912486.9,1793910912488.1,1793910912490.9,429,5,,39e797bfa6f6,
|
||||
q,1793910912655.4,1793910912658.5,1793910912661.5,429,5,,7bbaa8668c87,
|
||||
q,1793910912823.8,1793910912825.8,1793910912929.0,202,,5933ad00-47b5-4c7b-aa0b-d4431b271ab0,39e797bfa6f6,
|
||||
q,1793910912992.3,1793910912997.3,1793910913110.8,202,,f25d824d-c017-471d-8dbc-60a429a59736,7bbaa8668c87,
|
||||
q,1793910913160.7,1793910913164.6,1793910913167.3,429,5,,39e797bfa6f6,
|
||||
q,1793910913329.2,1793910913335.3,1793910913338.0,429,5,,7bbaa8668c87,
|
||||
q,1793910913497.7,1793910913500.3,1793910913603.9,202,,dad24888-7633-4bd0-a2f0-dd1d9d6ae309,39e797bfa6f6,
|
||||
q,1793910913666.1,1793910913671.9,1793910913779.8,202,,60ead31c-dc83-4447-b284-d63f9f9bf524,7bbaa8668c87,
|
||||
q,1793910913834.6,1793910913840.2,1793910913843.0,429,5,,39e797bfa6f6,
|
||||
q,1793910914003.0,1793910914009.6,1793910914012.5,429,5,,7bbaa8668c87,
|
||||
q,1793910914171.5,1793910914176.3,1793910914279.8,202,,91011ec0-8031-4e9c-91a9-33df28fd3719,39e797bfa6f6,
|
||||
q,1793910914340.0,1793910914342.1,1793910914453.2,202,,c2c23e97-6442-4b0b-a077-dd5ade365ca0,7bbaa8668c87,
|
||||
q,1793910914508.4,1793910914511.7,1793910914514.4,429,5,,39e797bfa6f6,
|
||||
q,1793910914676.9,1793910914683.6,1793910914686.5,429,5,,7bbaa8668c87,
|
||||
q,1793910914845.4,1793910914846.6,1793910914950.8,202,,4ea86425-461c-4f27-8a8c-af16de25dc38,39e797bfa6f6,
|
||||
q,1793910915013.8,1793910915014.6,1793910915122.1,202,,f706425b-a5d2-467a-89d7-92372531f6a1,7bbaa8668c87,
|
||||
q,1793910915182.3,1793910915183.8,1793910915187.1,429,5,,39e797bfa6f6,
|
||||
q,1793910915350.7,1793910915351.6,1793910915354.8,429,5,,7bbaa8668c87,
|
||||
q,1793910915519.2,1793910915522.0,1793910915524.8,429,5,,39e797bfa6f6,
|
||||
q,1793910915687.7,1793910915691.6,1793910915797.3,202,,c9e9bbba-fed6-4225-8ca8-fcf6e0e11c71,7bbaa8668c87,
|
||||
q,1793910915856.1,1793910915861.9,1793910915971.9,202,,033fa897-3e89-4a47-b264-ae39fa98cb85,39e797bfa6f6,
|
||||
q,1793910916024.6,1793910916027.1,1793910916029.9,429,5,,7bbaa8668c87,
|
||||
q,1793910916193.0,1793910916194.6,1793910916197.5,429,5,,39e797bfa6f6,
|
||||
q,1793910916361.5,1793910916362.7,1793910916471.6,202,,64ff448b-c98b-479a-b333-67eb5547d27c,7bbaa8668c87,
|
||||
q,1793910916530.0,1793910916537.6,1793910916650.8,202,,2ea6f033-4f60-4cb9-ad58-112d8ab4383a,39e797bfa6f6,
|
||||
q,1793910916698.4,1793910916702.0,1793910916705.3,429,5,,7bbaa8668c87,
|
||||
q,1793910916866.9,1793910916867.6,1793910916871.1,429,5,,39e797bfa6f6,
|
||||
q,1793910917035.3,1793910917036.6,1793910917143.5,202,,f788fe31-31cc-4095-9575-77a8a2057fc6,7bbaa8668c87,
|
||||
q,1793910917203.8,1793910917206.3,1793910917325.9,202,,d1153ecb-244c-4904-bc37-8b21c00e3826,39e797bfa6f6,
|
||||
q,1793910917372.3,1793910917373.7,1793910917377.1,429,5,,7bbaa8668c87,
|
||||
q,1793910917540.7,1793910917541.8,1793910917544.6,429,5,,39e797bfa6f6,
|
||||
q,1793910917709.2,1793910917710.7,1793910917814.8,202,,08c9d35b-7727-4c83-805f-e769f1255d82,7bbaa8668c87,
|
||||
q,1793910917877.6,1793910917878.9,1793910917989.7,202,,1716e0b2-f677-47d3-b43a-1d71ee34984c,39e797bfa6f6,
|
||||
q,1793910918046.1,1793910918047.8,1793910918050.8,429,5,,7bbaa8668c87,
|
||||
q,1793910918214.6,1793910918216.7,1793910918219.3,429,5,,39e797bfa6f6,
|
||||
q,1793910918383.0,1793910918383.6,1793910918490.3,202,,aa1cb19a-ccf8-45c5-9d07-224faa082342,7bbaa8668c87,
|
||||
q,1793910918551.5,1793910918552.6,1793910918662.8,202,,518a7653-02af-4155-b392-511c576b0fba,39e797bfa6f6,
|
||||
q,1793910918719.9,1793910918722.1,1793910918725.0,429,5,,7bbaa8668c87,
|
||||
q,1793910918888.4,1793910918891.2,1793910918894.0,429,5,,39e797bfa6f6,
|
||||
q,1793910919056.9,1793910919057.9,1793910919161.4,202,,a35baac4-a262-46e1-919e-c08fea7aed95,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"rate": 5.93611,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793910879.132,
|
||||
"end": 1793910919.225,
|
||||
"finished": 1793910919.159,
|
||||
"scheduled": 238,
|
||||
"sent": 238,
|
||||
"accepted": 173,
|
||||
"rejected_429": 65,
|
||||
"retry_after": {
|
||||
"5": 65
|
||||
},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 173,
|
||||
"429": 65
|
||||
},
|
||||
"upload_p50_ms": 106.75,
|
||||
"upload_p95_ms": 116.96,
|
||||
"upload_max_ms": 132.08,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.188,
|
||||
"log": "/tmp/c2p2-x-r1-overload.log"
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
q,1793910941015.5,1793910941018.4,1793910941160.0,202,,605cb49d-e96a-4987-b14d-e8dab87a67e9,39e797bfa6f6,
|
||||
q,1793910941577.1,1793910941580.2,1793910941713.6,202,,4a518ac2-a5e0-476d-b1fe-2db037049ebf,7bbaa8668c87,
|
||||
q,1793910942138.6,1793910942139.9,1793910942244.3,202,,8a79b646-0935-42c7-b798-9d45c2adb85f,39e797bfa6f6,
|
||||
q,1793910942700.2,1793910942702.6,1793910942805.2,202,,7ac8bfd8-685b-44de-a82d-0cd620e30d83,7bbaa8668c87,
|
||||
q,1793910943261.7,1793910943263.4,1793910943367.7,202,,d5182065-3f4b-4e97-9493-fc3997469324,39e797bfa6f6,
|
||||
q,1793910943823.2,1793910943825.1,1793910943929.0,202,,fb1240e4-0645-4258-84b0-7ab46d33e7e4,7bbaa8668c87,
|
||||
q,1793910944384.8,1793910944385.9,1793910944493.4,202,,caa8ab17-03a0-49a3-b97b-f60d91efea98,39e797bfa6f6,
|
||||
q,1793910944946.3,1793910944948.4,1793910945050.1,202,,50ccb7d4-af96-4755-8835-101367323897,7bbaa8668c87,
|
||||
q,1793910945507.8,1793910945508.9,1793910945613.4,202,,06a324ee-90c6-47b0-9d94-6b706c3bba38,39e797bfa6f6,
|
||||
q,1793910946069.4,1793910946072.9,1793910946174.6,202,,06fdc278-6e9d-4346-b585-231a16d5fe01,7bbaa8668c87,
|
||||
q,1793910946630.9,1793910946633.3,1793910946765.1,202,,8450b4da-f5ce-4ef3-ad6c-e26e4f04eb18,39e797bfa6f6,
|
||||
q,1793910947192.4,1793910947194.1,1793910947298.2,202,,e29f4610-27d1-4d4d-82ce-a7a662c3b329,7bbaa8668c87,
|
||||
q,1793910947754.0,1793910947754.9,1793910947857.7,202,,29919874-ebe6-4d7a-9a12-a1e9c2f85460,39e797bfa6f6,
|
||||
q,1793910948315.5,1793910948316.9,1793910948423.2,202,,6c534d76-c612-47be-ab5b-727ccc5d4c11,7bbaa8668c87,
|
||||
q,1793910948877.1,1793910948878.4,1793910948981.3,202,,6bf86c2b-8e16-4429-bbfa-54813a486189,39e797bfa6f6,
|
||||
q,1793910949438.6,1793910949439.9,1793910949544.7,202,,8c0d64a1-2bbe-4a2c-a12e-3746828b2c10,7bbaa8668c87,
|
||||
q,1793910950000.1,1793910950006.9,1793910950112.1,202,,bfa1b66a-d362-4176-a0b0-88bb2c214db5,39e797bfa6f6,
|
||||
q,1793910950561.7,1793910950564.2,1793910950665.3,202,,b1d6a29c-ed38-49b6-a373-2c323490afb8,7bbaa8668c87,
|
||||
q,1793910951123.2,1793910951129.9,1793910951231.6,202,,ee498ef4-58bc-4efe-9e23-207ed6aeed15,39e797bfa6f6,
|
||||
q,1793910951684.7,1793910951687.3,1793910951789.5,202,,90fa21e9-ef85-4103-a678-823696c7cbb0,7bbaa8668c87,
|
||||
q,1793910952246.3,1793910952248.0,1793910952357.7,202,,012223b0-7c61-488f-b0ac-a8b470756abf,39e797bfa6f6,
|
||||
q,1793910952807.8,1793910952810.1,1793910952913.1,202,,84b699f1-16f3-4359-adbc-10fab0d5154c,7bbaa8668c87,
|
||||
q,1793910953369.3,1793910953371.1,1793910953476.6,202,,5d58bc86-8f45-4788-a0f6-ee6beb24bf5f,39e797bfa6f6,
|
||||
q,1793910953930.9,1793910953933.3,1793910954036.1,202,,98c8ca16-a97f-4fdf-9f09-675d6da5ddd2,7bbaa8668c87,
|
||||
q,1793910954492.4,1793910954493.9,1793910954595.4,202,,8ab8b585-5643-4c00-9529-a8a402bb63bf,39e797bfa6f6,
|
||||
q,1793910955053.9,1793910955054.9,1793910955162.7,202,,c8ef65da-88d7-4c5d-b3b7-d02f78f9be72,7bbaa8668c87,
|
||||
q,1793910955615.5,1793910955617.4,1793910955720.4,202,,81a1d4bf-4f6c-4696-8c7b-acf47a40bd92,39e797bfa6f6,
|
||||
q,1793910956177.0,1793910956177.9,1793910956279.8,202,,1b133395-e55e-40ee-9c4b-5f21a617f524,7bbaa8668c87,
|
||||
q,1793910956738.6,1793910956739.9,1793910956842.0,202,,36e4d217-8cf5-490c-8808-f9821bda08ea,39e797bfa6f6,
|
||||
q,1793910957300.1,1793910957303.7,1793910957436.6,202,,56d2b494-e16e-47e5-8862-e3a695cff913,7bbaa8668c87,
|
||||
q,1793910957861.6,1793910957867.3,1793910957971.0,202,,2d926ceb-67a6-441e-854f-2a312e12ea2c,39e797bfa6f6,
|
||||
q,1793910958423.2,1793910958426.6,1793910958528.7,202,,e04647bb-d20f-4576-b1d1-ddfe6b90452a,7bbaa8668c87,
|
||||
q,1793910958984.7,1793910958986.1,1793910959088.6,202,,bc1d6d3c-b4a9-45b6-83ae-5031bb5682c7,39e797bfa6f6,
|
||||
q,1793910959546.2,1793910959550.0,1793910959652.7,202,,57001115-2125-455d-8bd2-3280fdd8d6a5,7bbaa8668c87,
|
||||
q,1793910960107.8,1793910960110.2,1793910960212.4,202,,3f0d9517-dd77-4279-952d-9865574424e1,39e797bfa6f6,
|
||||
q,1793910960669.3,1793910960672.0,1793910960775.6,202,,cdaf1912-b85c-4b6e-a270-687a78c05191,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 1.78083,
|
||||
"duration_s": 20.0,
|
||||
"start": 1793910941.016,
|
||||
"end": 1793910961.231,
|
||||
"finished": 1793910960.776,
|
||||
"scheduled": 36,
|
||||
"sent": 36,
|
||||
"accepted": 36,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 36
|
||||
},
|
||||
"upload_p50_ms": 103.03,
|
||||
"upload_p95_ms": 133.38,
|
||||
"upload_max_ms": 141.56,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.038,
|
||||
"log": "/tmp/c2p2-x-r1-recovery.log"
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
q,1793910798041.3,1793910798043.1,1793910798171.8,202,,80bb53ca-a739-4ce2-8dc6-c4d2181e1f9d,39e797bfa6f6,
|
||||
q,1793910798602.8,1793910798603.7,1793910798705.5,202,,83aa49ff-0a65-469f-b15b-876935cae985,7bbaa8668c87,
|
||||
q,1793910799164.3,1793910799169.2,1793910799270.7,202,,c8cdd5c8-9e99-4655-88f7-dd712d80af93,39e797bfa6f6,
|
||||
q,1793910799725.9,1793910799727.2,1793910799858.9,202,,415a2a6d-6ce0-421b-96a0-8eb20a7eeb43,7bbaa8668c87,
|
||||
q,1793910800287.4,1793910800288.7,1793910800394.8,202,,bd9d99f1-6235-47ad-8a1f-8a965f1ba00e,39e797bfa6f6,
|
||||
q,1793910800848.9,1793910800851.5,1793910800954.3,202,,b23cc782-a205-41da-bdb3-70ccaa5f05c9,7bbaa8668c87,
|
||||
q,1793910801410.5,1793910801412.8,1793910801516.9,202,,2ee2ba5c-8f5a-427f-879b-684a89e36b51,39e797bfa6f6,
|
||||
q,1793910801972.0,1793910801973.9,1793910802076.9,202,,69746071-a25b-4d1c-9b9a-cf6c27c5aff4,7bbaa8668c87,
|
||||
q,1793910802533.5,1793910802534.7,1793910802637.4,202,,93eac001-18ad-4741-9dbe-b0a0a55aa373,39e797bfa6f6,
|
||||
q,1793910803095.1,1793910803097.2,1793910803199.3,202,,5963ed26-e532-4efc-9920-ea7bbb62eae0,7bbaa8668c87,
|
||||
q,1793910803656.6,1793910803660.0,1793910803760.5,202,,89e325e0-d466-484e-950a-03dffb6ed9ba,39e797bfa6f6,
|
||||
q,1793910804218.2,1793910804220.2,1793910804321.7,202,,8d01ef0f-ddcc-4d71-bd63-995b3d75ab2b,7bbaa8668c87,
|
||||
q,1793910804779.7,1793910804781.5,1793910804883.0,202,,c2489035-7c8c-418a-880e-24d40a3667f7,39e797bfa6f6,
|
||||
q,1793910805341.2,1793910805342.9,1793910805473.8,202,,38fe9b5b-d88a-4cc0-a872-f8a7e3ff493e,7bbaa8668c87,
|
||||
q,1793910805902.8,1793910805903.3,1793910806005.9,202,,5a28a291-fd9e-43c3-a360-4ceac6491a27,39e797bfa6f6,
|
||||
q,1793910806464.3,1793910806466.0,1793910806572.2,202,,e02afa0e-4b01-4919-8b80-22c28faceb58,7bbaa8668c87,
|
||||
q,1793910807025.8,1793910807027.9,1793910807131.6,202,,2e7548da-75d2-43b4-a0f3-6c0b4df1bc35,39e797bfa6f6,
|
||||
q,1793910807587.4,1793910807588.7,1793910807691.1,202,,88d148f6-ca72-4565-b38b-f1ee3ff60090,7bbaa8668c87,
|
||||
q,1793910808148.9,1793910808151.0,1793910808252.7,202,,19168109-6245-4af0-a9a2-4bd01891cf49,39e797bfa6f6,
|
||||
q,1793910808710.4,1793910808711.8,1793910808814.7,202,,136114c2-86b1-4612-a01d-fc85d58b7138,7bbaa8668c87,
|
||||
q,1793910809272.0,1793910809275.8,1793910809381.2,202,,7d5eb73d-d179-4af8-849f-abfe9bcafb9a,39e797bfa6f6,
|
||||
q,1793910809833.5,1793910809838.7,1793910809939.6,202,,8e91d004-666f-462c-9d53-5bd57ee24f7d,7bbaa8668c87,
|
||||
q,1793910810395.0,1793910810396.9,1793910810514.2,202,,43a7fd22-b65d-4de5-9f5a-4aaf472307a7,39e797bfa6f6,
|
||||
q,1793910810956.6,1793910810960.0,1793910811063.7,202,,06afbb2e-4b89-4f5a-a4e5-abb5026d7a8d,7bbaa8668c87,
|
||||
q,1793910811518.1,1793910811520.0,1793910811625.2,202,,b574099d-4f6a-460f-9a65-6c763f0b6abc,39e797bfa6f6,
|
||||
q,1793910812079.7,1793910812080.8,1793910812183.3,202,,03d20e3c-918b-42ad-933a-995afe2792af,7bbaa8668c87,
|
||||
q,1793910812641.2,1793910812641.8,1793910812743.6,202,,0f5c0134-9047-495e-93bb-2c408f17d9d1,39e797bfa6f6,
|
||||
q,1793910813202.7,1793910813205.5,1793910813309.1,202,,d5ee40b9-b15d-4afc-ba25-bc0dbc6aeef5,7bbaa8668c87,
|
||||
q,1793910813764.3,1793910813768.0,1793910813869.9,202,,97e77d83-4da2-495e-b701-3ca94248c8e3,39e797bfa6f6,
|
||||
q,1793910814325.8,1793910814328.7,1793910814431.4,202,,5f5707da-607a-43be-83ce-4efddea06283,7bbaa8668c87,
|
||||
q,1793910814887.3,1793910814890.0,1793910814990.6,202,,f871a697-04a3-4d08-87fe-4d6ea6533e4e,39e797bfa6f6,
|
||||
q,1793910815448.9,1793910815451.1,1793910815573.4,202,,fcd0e40a-f734-48b5-b3a9-c90b3ce70c0d,7bbaa8668c87,
|
||||
q,1793910816010.4,1793910816011.7,1793910816115.4,202,,314d6a2f-e9f8-4fc6-831b-de19791ac3d0,39e797bfa6f6,
|
||||
q,1793910816571.9,1793910816574.2,1793910816677.3,202,,4231c639-2cb0-4eba-b2a6-be487e21c369,7bbaa8668c87,
|
||||
q,1793910817133.5,1793910817136.0,1793910817240.5,202,,d06bc99a-e045-4fdc-90a4-75bfbdbf5589,39e797bfa6f6,
|
||||
q,1793910817695.0,1793910817695.8,1793910817797.8,202,,c8e5296e-2bd9-4212-bd50-8387a7bb1669,7bbaa8668c87,
|
||||
q,1793910818256.6,1793910818257.3,1793910818361.5,202,,17282371-f6b9-400a-a9e1-9b044dd699fe,39e797bfa6f6,
|
||||
q,1793910818818.1,1793910818819.8,1793910818923.0,202,,dcf2a7ea-7a2f-4c8a-8ce9-b139838a8cd2,7bbaa8668c87,
|
||||
q,1793910819379.6,1793910819380.9,1793910819482.7,202,,a192a851-cd45-49fe-90cd-e5657f53a5f9,39e797bfa6f6,
|
||||
q,1793910819941.2,1793910819947.7,1793910820048.8,202,,61297f70-9c72-4a91-b224-db8a56dd60de,7bbaa8668c87,
|
||||
q,1793910820502.7,1793910820504.4,1793910820617.9,202,,3128b432-69ed-42f8-a07d-5945d1adc298,39e797bfa6f6,
|
||||
q,1793910821064.2,1793910821067.2,1793910821169.9,202,,123c6248-a1b4-47bb-8b91-22579a4dc016,7bbaa8668c87,
|
||||
q,1793910821625.8,1793910821629.2,1793910821731.8,202,,d2a4a9a4-1ae6-4a0b-8ec2-f4e8b22d5cb3,39e797bfa6f6,
|
||||
q,1793910822187.3,1793910822193.7,1793910822296.8,202,,9899bfcb-cbf2-46ed-9602-c230e82313e9,7bbaa8668c87,
|
||||
q,1793910822748.8,1793910822750.7,1793910822852.7,202,,a3120be3-dabf-400f-adcc-93a4f1bc3a56,39e797bfa6f6,
|
||||
q,1793910823310.4,1793910823315.7,1793910823419.6,202,,5e9a004d-25a6-4f53-a34a-6b1c58f10457,7bbaa8668c87,
|
||||
q,1793910823871.9,1793910823874.5,1793910823978.0,202,,fe7d9e1a-f236-43d9-b0bf-536ca9c95d55,39e797bfa6f6,
|
||||
q,1793910824433.4,1793910824437.0,1793910824538.3,202,,d127d65c-f927-4fd1-8a4a-8b954e9e7ac1,7bbaa8668c87,
|
||||
q,1793910824995.0,1793910824998.9,1793910825100.0,202,,7e307e35-04e4-465a-92d4-1fd0e5ce398f,39e797bfa6f6,
|
||||
q,1793910825556.5,1793910825558.5,1793910825659.6,202,,25b27488-8a83-4b66-841c-421fc2562fff,7bbaa8668c87,
|
||||
q,1793910826118.1,1793910826120.7,1793910826251.4,202,,c4df8783-baba-490b-9435-ab24fa321f3e,39e797bfa6f6,
|
||||
q,1793910826679.6,1793910826681.7,1793910826785.5,202,,b01e93c1-077f-44f5-b8b0-df9ab81a82a2,7bbaa8668c87,
|
||||
q,1793910827241.1,1793910827242.7,1793910827351.5,202,,1d9152c2-48c2-417b-800d-c90498b473e4,39e797bfa6f6,
|
||||
q,1793910827802.7,1793910827806.0,1793910827909.7,202,,73e45df9-fe52-4bee-a2a6-91d566b3c8a1,7bbaa8668c87,
|
||||
q,1793910828364.2,1793910828365.9,1793910828469.4,202,,ab7e8625-fde4-4dbd-a654-93a062534d02,39e797bfa6f6,
|
||||
q,1793910828925.7,1793910828926.7,1793910829029.5,202,,ec01f049-48db-4c35-a65b-fc9c42a7f84b,7bbaa8668c87,
|
||||
q,1793910829487.3,1793910829490.6,1793910829594.0,202,,949ebffa-a091-47f1-9140-79d2bf5a51bf,39e797bfa6f6,
|
||||
q,1793910830048.8,1793910830052.4,1793910830152.9,202,,cfd6034e-0e3e-4f61-8de8-e4de8e8c31eb,7bbaa8668c87,
|
||||
q,1793910830610.3,1793910830613.8,1793910830714.6,202,,5c9d44ac-8d93-4203-b818-ee9ef2f793dd,39e797bfa6f6,
|
||||
q,1793910831171.9,1793910831174.0,1793910831274.4,202,,afc38d75-27f7-473f-ad1a-96b31b01b321,7bbaa8668c87,
|
||||
q,1793910831733.4,1793910831735.6,1793910831863.3,202,,93faa552-4bdc-4ee1-99ef-0917548ce4b0,39e797bfa6f6,
|
||||
q,1793910832295.0,1793910832299.4,1793910832406.2,202,,7425c1d0-698e-4ecf-95a0-07572fd05763,7bbaa8668c87,
|
||||
q,1793910832856.5,1793910832859.7,1793910832964.5,202,,85f7630a-911e-4ac8-899c-4fed3a831e57,39e797bfa6f6,
|
||||
q,1793910833418.0,1793910833421.2,1793910833525.0,202,,bcb5b5d5-ad13-4177-9550-1090b7e8c303,7bbaa8668c87,
|
||||
q,1793910833979.6,1793910833980.8,1793910834083.8,202,,7d325fc1-1724-4f3b-841f-1fec766f2cc2,39e797bfa6f6,
|
||||
q,1793910834541.1,1793910834543.8,1793910834646.2,202,,70089b84-dc92-47d8-b3db-a03dd36086df,7bbaa8668c87,
|
||||
q,1793910835102.6,1793910835106.7,1793910835210.9,202,,c178e1f1-b68f-4be6-b932-b6b64e9c8436,39e797bfa6f6,
|
||||
q,1793910835664.2,1793910835670.7,1793910835773.7,202,,01520b9b-a1c0-43ea-b4bc-2b1cff621cfb,7bbaa8668c87,
|
||||
q,1793910836225.7,1793910836227.7,1793910836329.3,202,,11f1c955-af72-4bda-acb3-bfa3a2828a22,39e797bfa6f6,
|
||||
q,1793910836787.2,1793910836790.0,1793910836922.2,202,,cae89961-263f-478d-b657-d5a062d1d417,7bbaa8668c87,
|
||||
q,1793910837348.8,1793910837350.9,1793910837457.7,202,,91006be4-c017-495b-b551-b8709d24bb97,39e797bfa6f6,
|
||||
q,1793910837910.3,1793910837914.0,1793910838018.3,202,,7e4cd9a6-a002-41de-a2a9-5609ef0a64bd,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 1.78083,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793910798.041,
|
||||
"end": 1793910838.472,
|
||||
"finished": 1793910838.018,
|
||||
"scheduled": 72,
|
||||
"sent": 72,
|
||||
"accepted": 72,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 72
|
||||
},
|
||||
"upload_p50_ms": 102.94,
|
||||
"upload_p95_ms": 130.75,
|
||||
"upload_max_ms": 132.25,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.064,
|
||||
"log": "/tmp/c2p2-x-r1-steady.log"
|
||||
}
|
||||
@@ -0,0 +1,326 @@
|
||||
2026-11-05T20:33:06.058Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=2 prefetch=2 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-
|
||||
2026-11-05T20:33:06.065Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=2
|
||||
2026-11-05T20:33:06.388Z INFO src.worker [rid=-] worker stopped exit_code=0 cancelled=0 returned=0
|
||||
2026-11-05T20:33:10.392Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=2 prefetch=2 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-
|
||||
2026-11-05T20:33:10.400Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=2
|
||||
2026-11-05T20:33:11.107Z INFO src.services.processing [rid=-] attachment processed attachment_id=1fb82fef-091d-48ec-a8a9-30d72345f966 job_id=9899e504-1b44-4f5b-9bba-7e263efbc192 attempt=1 variants=3 render_peak_kib=306472 took=701ms waited=3286ms
|
||||
2026-11-05T20:33:11.108Z INFO src.services.processing [rid=-] attachment processed attachment_id=8c6c9422-399e-4f4d-984b-0adedf5615e2 job_id=6de501aa-f0b8-4a0c-8e47-7eb29d9e8b79 attempt=1 variants=3 render_peak_kib=306492 took=703ms waited=3344ms
|
||||
2026-11-05T20:33:11.785Z INFO src.services.processing [rid=-] attachment processed attachment_id=5fce6b55-8b7c-4235-9528-dbbc59ca5a95 job_id=2089dc89-e3c8-4bc4-9337-f411c1bc0507 attempt=1 variants=3 render_peak_kib=306440 took=676ms waited=3792ms
|
||||
2026-11-05T20:33:11.787Z INFO src.services.processing [rid=-] attachment processed attachment_id=8f0cbefc-5a22-46bd-a126-7b64dd65030d job_id=aa45eef7-0f62-48eb-ae9d-d1399a20029f attempt=1 variants=3 render_peak_kib=306528 took=678ms waited=3891ms
|
||||
2026-11-05T20:33:12.483Z INFO src.services.processing [rid=-] attachment processed attachment_id=164a0c4a-dbc2-4d6d-a1e8-89fab877ba82 job_id=cf148048-d614-4401-ae6d-7263e9562a48 attempt=1 variants=3 render_peak_kib=306512 took=674ms waited=4295ms
|
||||
2026-11-05T20:33:12.483Z INFO src.services.processing [rid=-] attachment processed attachment_id=544ee407-97f7-4561-9887-e80400cbd8e0 job_id=7594d24b-483c-4f9f-a2e5-36f515ffe827 attempt=1 variants=3 render_peak_kib=306460 took=674ms waited=4389ms
|
||||
2026-11-05T20:33:13.150Z INFO src.services.processing [rid=-] attachment processed attachment_id=a2380693-dbde-4c8c-a918-dbd294fa12d1 job_id=0dcad1aa-d6a8-4207-8c57-91a05c873317 attempt=1 variants=3 render_peak_kib=306452 took=665ms waited=4866ms
|
||||
2026-11-05T20:33:13.150Z INFO src.services.processing [rid=-] attachment processed attachment_id=9d1807fa-0441-454d-8a7e-06e4c1863df7 job_id=d2aeeff7-3d08-4829-865a-e674e15e33cc attempt=1 variants=3 render_peak_kib=306548 took=665ms waited=4767ms
|
||||
2026-11-05T20:33:13.831Z INFO src.services.processing [rid=-] attachment processed attachment_id=d3ed11d8-f556-4b18-a981-09d2c05207b5 job_id=886ee070-ec30-4c08-a123-88dd888c62ca attempt=1 variants=3 render_peak_kib=306540 took=679ms waited=5332ms
|
||||
2026-11-05T20:33:13.835Z INFO src.services.processing [rid=-] attachment processed attachment_id=17c307b8-3fce-4cf7-93c6-c6f94150b62c job_id=278ea91d-1785-4e17-a171-c0aeda3059fc attempt=1 variants=3 render_peak_kib=306468 took=682ms waited=5236ms
|
||||
2026-11-05T20:33:14.498Z INFO src.services.processing [rid=-] attachment processed attachment_id=e483c11f-b585-442b-96d5-ae047cfeb573 job_id=e3c161e4-3dc2-4a7f-b530-122697530ce0 attempt=1 variants=3 render_peak_kib=306576 took=662ms waited=5717ms
|
||||
2026-11-05T20:33:14.506Z INFO src.services.processing [rid=-] attachment processed attachment_id=8e316bef-326b-4dfc-ab77-c4e5c7e90169 job_id=7265dab0-dd55-400b-8270-6fa0828ce9fa attempt=1 variants=3 render_peak_kib=306488 took=673ms waited=5814ms
|
||||
2026-11-05T20:33:15.136Z INFO src.services.processing [rid=-] attachment processed attachment_id=ce13c278-c136-45d0-91d8-401f108626ba job_id=21b25670-da24-4a5b-b902-f857744587a2 attempt=1 variants=3 render_peak_kib=306532 took=636ms waited=6280ms
|
||||
2026-11-05T20:33:15.163Z INFO src.services.processing [rid=-] attachment processed attachment_id=dbb7dc38-8a0e-4f9c-9ca9-de72e47789c7 job_id=d192bb90-f26f-4cd8-a086-c9d3d558f7a8 attempt=1 variants=3 render_peak_kib=306568 took=656ms waited=6191ms
|
||||
2026-11-05T20:33:15.796Z INFO src.services.processing [rid=-] attachment processed attachment_id=d42c12ae-35e1-4000-9906-01b609a870f0 job_id=65d9fb85-78af-421e-a0db-51101f8a8605 attempt=1 variants=3 render_peak_kib=306548 took=659ms waited=6717ms
|
||||
2026-11-05T20:33:15.824Z INFO src.services.processing [rid=-] attachment processed attachment_id=39f1260a-2c56-4d73-88c8-f1271d8d2ebe job_id=7dc13025-a5c8-48ff-9bbb-325f1dfb1295 attempt=1 variants=3 render_peak_kib=306520 took=659ms waited=6648ms
|
||||
2026-11-05T20:33:16.455Z INFO src.services.processing [rid=-] attachment processed attachment_id=47e62309-c5d5-4ee9-b859-c9c491596072 job_id=80bfe3ac-d1fe-45c0-bf1b-9db95ac5fa69 attempt=1 variants=3 render_peak_kib=306520 took=658ms waited=7179ms
|
||||
2026-11-05T20:33:16.482Z INFO src.services.processing [rid=-] attachment processed attachment_id=9eabb7e1-728d-4c7b-ab9e-f5d514b776f4 job_id=fce1a9da-3976-4a63-999b-6c65e6a899c9 attempt=1 variants=3 render_peak_kib=306572 took=657ms waited=7106ms
|
||||
2026-11-05T20:33:17.122Z INFO src.services.processing [rid=-] attachment processed attachment_id=6df0d6ac-5fbc-497e-ad2e-402a63495493 job_id=17fe452a-eb36-4e1c-bba1-ca0b751c18e4 attempt=1 variants=3 render_peak_kib=306548 took=665ms waited=7636ms
|
||||
2026-11-05T20:33:17.140Z INFO src.services.processing [rid=-] attachment processed attachment_id=66f98ee2-ca95-407a-a2a1-128fc36afbfa job_id=877622cb-5896-40b4-bdd2-53da6ad4d2bd attempt=1 variants=3 render_peak_kib=306544 took=657ms waited=7566ms
|
||||
2026-11-05T20:33:18.901Z INFO src.services.processing [rid=-] attachment processed attachment_id=80bb53ca-a739-4ce2-8dc6-c4d2181e1f9d job_id=5da970ad-98c0-4f35-ae37-920e5d5fb9b0 attempt=1 variants=3 render_peak_kib=306496 took=653ms waited=81ms
|
||||
2026-11-05T20:33:19.408Z INFO src.services.processing [rid=-] attachment processed attachment_id=83aa49ff-0a65-469f-b15b-876935cae985 job_id=5aedbf32-37e8-4f90-aff7-6585220ac48e attempt=1 variants=3 render_peak_kib=306508 took=653ms waited=55ms
|
||||
2026-11-05T20:33:20.423Z INFO src.services.processing [rid=-] attachment processed attachment_id=c8cdd5c8-9e99-4655-88f7-dd712d80af93 job_id=2058346e-0e3d-41b9-a003-743ed06bb2e1 attempt=1 variants=3 render_peak_kib=306588 took=657ms waited=501ms
|
||||
2026-11-05T20:33:20.929Z INFO src.services.processing [rid=-] attachment processed attachment_id=415a2a6d-6ce0-421b-96a0-8eb20a7eeb43 job_id=d80a4715-912c-4e41-a05b-bb426fe62cd5 attempt=1 variants=3 render_peak_kib=306476 took=656ms waited=420ms
|
||||
2026-11-05T20:33:21.439Z INFO src.services.processing [rid=-] attachment processed attachment_id=bd9d99f1-6235-47ad-8a1f-8a965f1ba00e job_id=d6034e37-4663-443b-945f-976fec655850 attempt=1 variants=3 render_peak_kib=306512 took=658ms waited=392ms
|
||||
2026-11-05T20:33:21.948Z INFO src.services.processing [rid=-] attachment processed attachment_id=b23cc782-a205-41da-bdb3-70ccaa5f05c9 job_id=6daeb88f-4436-4e1b-8916-6a4c85cbef12 attempt=1 variants=3 render_peak_kib=306448 took=658ms waited=341ms
|
||||
2026-11-05T20:33:22.455Z INFO src.services.processing [rid=-] attachment processed attachment_id=2ee2ba5c-8f5a-427f-879b-684a89e36b51 job_id=171060fc-9ff5-46a9-b86a-de84b39b3f67 attempt=1 variants=3 render_peak_kib=306564 took=660ms waited=285ms
|
||||
2026-11-05T20:33:22.965Z INFO src.services.processing [rid=-] attachment processed attachment_id=69746071-a25b-4d1c-9b9a-cf6c27c5aff4 job_id=1579f9a7-5dfa-4d77-b922-70ea94d44cca attempt=1 variants=3 render_peak_kib=306572 took=662ms waited=233ms
|
||||
2026-11-05T20:33:23.467Z INFO src.services.processing [rid=-] attachment processed attachment_id=93eac001-18ad-4741-9dbe-b0a0a55aa373 job_id=a75040b1-7f8b-487c-a025-92c6d425e700 attempt=1 variants=3 render_peak_kib=306472 took=656ms waited=179ms
|
||||
2026-11-05T20:33:23.975Z INFO src.services.processing [rid=-] attachment processed attachment_id=5963ed26-e532-4efc-9920-ea7bbb62eae0 job_id=55b6a191-b4e6-47c3-b2e6-34e236d7a867 attempt=1 variants=3 render_peak_kib=306492 took=659ms waited=123ms
|
||||
2026-11-05T20:33:24.482Z INFO src.services.processing [rid=-] attachment processed attachment_id=89e325e0-d466-484e-950a-03dffb6ed9ba job_id=a1bc3b8f-907b-4221-ad32-81c843011ce8 attempt=1 variants=3 render_peak_kib=306520 took=660ms waited=68ms
|
||||
2026-11-05T20:33:24.978Z INFO src.services.processing [rid=-] attachment processed attachment_id=8d01ef0f-ddcc-4d71-bd63-995b3d75ab2b job_id=cfa2f193-bf80-4070-a439-38b7769d09ea attempt=1 variants=3 render_peak_kib=306436 took=651ms waited=11ms
|
||||
2026-11-05T20:33:26.026Z INFO src.services.processing [rid=-] attachment processed attachment_id=c2489035-7c8c-418a-880e-24d40a3667f7 job_id=b6c9bf39-887a-4961-91eb-622fc88efe51 attempt=1 variants=3 render_peak_kib=306580 took=678ms waited=470ms
|
||||
2026-11-05T20:33:26.509Z INFO src.services.processing [rid=-] attachment processed attachment_id=38fe9b5b-d88a-4cc0-a872-f8a7e3ff493e job_id=4e213a31-db1e-4c0e-b7f9-3063cd0aac11 attempt=1 variants=3 render_peak_kib=306460 took=653ms waited=387ms
|
||||
2026-11-05T20:33:27.017Z INFO src.services.processing [rid=-] attachment processed attachment_id=5a28a291-fd9e-43c3-a360-4ceac6491a27 job_id=3bbed0c1-28fd-459b-bb6c-d9aa27d04e13 attempt=1 variants=3 render_peak_kib=306648 took=656ms waited=361ms
|
||||
2026-11-05T20:33:27.524Z INFO src.services.processing [rid=-] attachment processed attachment_id=e02afa0e-4b01-4919-8b80-22c28faceb58 job_id=e25990d4-a5b6-4b19-b725-4ad7382f24fe attempt=1 variants=3 render_peak_kib=306480 took=656ms waited=303ms
|
||||
2026-11-05T20:33:28.039Z INFO src.services.processing [rid=-] attachment processed attachment_id=2e7548da-75d2-43b4-a0f3-6c0b4df1bc35 job_id=be119ac1-d301-4def-be51-a0ee853be77e attempt=1 variants=3 render_peak_kib=306544 took=667ms waited=246ms
|
||||
2026-11-05T20:33:28.536Z INFO src.services.processing [rid=-] attachment processed attachment_id=88d148f6-ca72-4565-b38b-f1ee3ff60090 job_id=a0e51eb4-6d0e-457e-9b30-d31295200dc6 attempt=1 variants=3 render_peak_kib=306576 took=659ms waited=192ms
|
||||
2026-11-05T20:33:29.036Z INFO src.services.processing [rid=-] attachment processed attachment_id=19168109-6245-4af0-a9a2-4bd01891cf49 job_id=15a1d630-a37c-4916-b7d8-eb0a115c2f20 attempt=1 variants=3 render_peak_kib=306500 took=654ms waited=135ms
|
||||
2026-11-05T20:33:29.545Z INFO src.services.processing [rid=-] attachment processed attachment_id=136114c2-86b1-4612-a01d-fc85d58b7138 job_id=bca1089b-821f-4e62-9555-2534f9681174 attempt=1 variants=3 render_peak_kib=306560 took=658ms waited=78ms
|
||||
2026-11-05T20:33:30.045Z INFO src.services.processing [rid=-] attachment processed attachment_id=7d5eb73d-d179-4af8-849f-abfe9bcafb9a job_id=93dc6c0e-8067-4c89-a02b-dc3ce9e2ee4f attempt=1 variants=3 render_peak_kib=306536 took=652ms waited=18ms
|
||||
2026-11-05T20:33:31.068Z INFO src.services.processing [rid=-] attachment processed attachment_id=8e91d004-666f-462c-9d53-5bd57ee24f7d job_id=b31f12df-19ee-422f-9940-2ab8e5c0ee33 attempt=1 variants=3 render_peak_kib=306520 took=659ms waited=474ms
|
||||
2026-11-05T20:33:31.574Z INFO src.services.processing [rid=-] attachment processed attachment_id=43a7fd22-b65d-4de5-9f5a-4aaf472307a7 job_id=67bf6a34-f381-4372-86e0-3a373dd8c57a attempt=1 variants=3 render_peak_kib=306468 took=654ms waited=411ms
|
||||
2026-11-05T20:33:32.084Z INFO src.services.processing [rid=-] attachment processed attachment_id=06afbb2e-4b89-4f5a-a4e5-abb5026d7a8d job_id=0b6b43e7-f235-4284-86d0-99999a68e189 attempt=1 variants=3 render_peak_kib=306548 took=659ms waited=367ms
|
||||
2026-11-05T20:33:32.593Z INFO src.services.processing [rid=-] attachment processed attachment_id=b574099d-4f6a-460f-9a65-6c763f0b6abc job_id=73ae4859-c905-4dad-bf9b-2bf12ad163ce attempt=1 variants=3 render_peak_kib=306460 took=659ms waited=317ms
|
||||
2026-11-05T20:33:33.094Z INFO src.services.processing [rid=-] attachment processed attachment_id=03d20e3c-918b-42ad-933a-995afe2792af job_id=c5ce4ba6-74bc-4e1d-8e3e-0929896d6ed2 attempt=1 variants=3 render_peak_kib=306504 took=655ms waited=261ms
|
||||
2026-11-05T20:33:33.595Z INFO src.services.processing [rid=-] attachment processed attachment_id=0f5c0134-9047-495e-93bb-2c408f17d9d1 job_id=fc90b498-89e9-4dc5-b59b-ccd9c499bd1e attempt=1 variants=3 render_peak_kib=306464 took=651ms waited=205ms
|
||||
2026-11-05T20:33:34.106Z INFO src.services.processing [rid=-] attachment processed attachment_id=d5ee40b9-b15d-4afc-ba25-bc0dbc6aeef5 job_id=50234839-577c-4526-9127-cc18ac038659 attempt=1 variants=3 render_peak_kib=306476 took=657ms waited=146ms
|
||||
2026-11-05T20:33:34.607Z INFO src.services.processing [rid=-] attachment processed attachment_id=97e77d83-4da2-495e-b701-3ca94248c8e3 job_id=9d28c37d-026a-4a73-b53a-0a1ff4163dc3 attempt=1 variants=3 render_peak_kib=306464 took=654ms waited=89ms
|
||||
2026-11-05T20:33:35.112Z INFO src.services.processing [rid=-] attachment processed attachment_id=5f5707da-607a-43be-83ce-4efddea06283 job_id=ea3e2ad5-99ba-4644-ab1a-e018298b8c2f attempt=1 variants=3 render_peak_kib=306556 took=653ms waited=33ms
|
||||
2026-11-05T20:33:36.124Z INFO src.services.processing [rid=-] attachment processed attachment_id=f871a697-04a3-4d08-87fe-4d6ea6533e4e job_id=3b368d93-1633-4dcb-b649-874bc8eac94d attempt=1 variants=3 render_peak_kib=306524 took=654ms waited=484ms
|
||||
2026-11-05T20:33:36.632Z INFO src.services.processing [rid=-] attachment processed attachment_id=fcd0e40a-f734-48b5-b3a9-c90b3ce70c0d job_id=a200e96a-f113-4b98-bb50-5f1e1e1ad659 attempt=1 variants=3 render_peak_kib=306572 took=651ms waited=414ms
|
||||
2026-11-05T20:33:37.153Z INFO src.services.processing [rid=-] attachment processed attachment_id=314d6a2f-e9f8-4fc6-831b-de19791ac3d0 job_id=a44dadff-4ec9-427d-b0a7-b290e1a42700 attempt=1 variants=3 render_peak_kib=306452 took=662ms waited=382ms
|
||||
2026-11-05T20:33:37.659Z INFO src.services.processing [rid=-] attachment processed attachment_id=4231c639-2cb0-4eba-b2a6-be487e21c369 job_id=5741d256-6d7e-443e-a1d2-200a3e45227b attempt=1 variants=3 render_peak_kib=306644 took=658ms waited=329ms
|
||||
2026-11-05T20:33:38.163Z INFO src.services.processing [rid=-] attachment processed attachment_id=d06bc99a-e045-4fdc-90a4-75bfbdbf5589 job_id=6952230b-a0b0-43f4-9ca3-8fb5516055db attempt=1 variants=3 render_peak_kib=306484 took=653ms waited=276ms
|
||||
2026-11-05T20:33:38.673Z INFO src.services.processing [rid=-] attachment processed attachment_id=c8e5296e-2bd9-4212-bd50-8387a7bb1669 job_id=54895f76-efc7-4b32-8830-6ab729633d0c attempt=1 variants=3 render_peak_kib=306568 took=657ms waited=224ms
|
||||
2026-11-05T20:33:39.182Z INFO src.services.processing [rid=-] attachment processed attachment_id=17282371-f6b9-400a-a9e1-9b044dd699fe job_id=ce4d2059-791b-4a69-8053-9775a6685eac attempt=1 variants=3 render_peak_kib=306556 took=661ms waited=165ms
|
||||
2026-11-05T20:33:39.688Z INFO src.services.processing [rid=-] attachment processed attachment_id=dcf2a7ea-7a2f-4c8a-8ce9-b139838a8cd2 job_id=70c1d8b2-8f76-40e9-918f-c38ef668bb0f attempt=1 variants=3 render_peak_kib=306500 took=659ms waited=112ms
|
||||
2026-11-05T20:33:40.182Z INFO src.services.processing [rid=-] attachment processed attachment_id=a192a851-cd45-49fe-90cd-e5657f53a5f9 job_id=0d3a4248-5b16-4559-bed5-3e8fdd463b34 attempt=1 variants=3 render_peak_kib=306500 took=649ms waited=56ms
|
||||
2026-11-05T20:33:41.195Z INFO src.services.processing [rid=-] attachment processed attachment_id=61297f70-9c72-4a91-b224-db8a56dd60de job_id=f992e5b6-e810-438d-b704-16b8a277c03b attempt=1 variants=3 render_peak_kib=306504 took=650ms waited=502ms
|
||||
2026-11-05T20:33:41.709Z INFO src.services.processing [rid=-] attachment processed attachment_id=3128b432-69ed-42f8-a07d-5945d1adc298 job_id=fa9b63f2-9eb4-497f-8322-a59898a71309 attempt=1 variants=3 render_peak_kib=306496 took=659ms waited=439ms
|
||||
2026-11-05T20:33:42.215Z INFO src.services.processing [rid=-] attachment processed attachment_id=123c6248-a1b4-47bb-8b91-22579a4dc016 job_id=14747da5-df13-4866-af9c-202730163f9a attempt=1 variants=3 render_peak_kib=306528 took=656ms waited=396ms
|
||||
2026-11-05T20:33:42.720Z INFO src.services.processing [rid=-] attachment processed attachment_id=d2a4a9a4-1ae6-4a0b-8ec2-f4e8b22d5cb3 job_id=ffd98f35-1d88-4917-b1a5-748c0d749621 attempt=1 variants=3 render_peak_kib=306540 took=651ms waited=343ms
|
||||
2026-11-05T20:33:43.228Z INFO src.services.processing [rid=-] attachment processed attachment_id=9899bfcb-cbf2-46ed-9602-c230e82313e9 job_id=f1a48871-f39d-4084-84d3-ff4306f3bfb9 attempt=1 variants=3 render_peak_kib=306608 took=654ms waited=284ms
|
||||
2026-11-05T20:33:43.735Z INFO src.services.processing [rid=-] attachment processed attachment_id=a3120be3-dabf-400f-adcc-93a4f1bc3a56 job_id=0a29259c-96d6-4a1d-9326-d6d1e49a7105 attempt=1 variants=3 render_peak_kib=306480 took=656ms waited=232ms
|
||||
2026-11-05T20:33:44.249Z INFO src.services.processing [rid=-] attachment processed attachment_id=5e9a004d-25a6-4f53-a34a-6b1c58f10457 job_id=6fc7dbe6-a62d-4e59-82c8-2b5e84192d36 attempt=1 variants=3 render_peak_kib=306556 took=664ms waited=174ms
|
||||
2026-11-05T20:33:44.749Z INFO src.services.processing [rid=-] attachment processed attachment_id=fe7d9e1a-f236-43d9-b0bf-536ca9c95d55 job_id=9b9e10ae-2aab-45ed-adf6-29c4cdf6ac3d attempt=1 variants=3 render_peak_kib=306512 took=642ms waited=136ms
|
||||
2026-11-05T20:33:45.276Z INFO src.services.processing [rid=-] attachment processed attachment_id=d127d65c-f927-4fd1-8a4a-8b954e9e7ac1 job_id=a1282392-3daa-4381-89c8-01e2d915323d attempt=1 variants=3 render_peak_kib=306516 took=659ms waited=86ms
|
||||
2026-11-05T20:33:45.776Z INFO src.services.processing [rid=-] attachment processed attachment_id=7e307e35-04e4-465a-92d4-1fd0e5ce398f job_id=378d056e-4324-4127-a684-592921071105 attempt=1 variants=3 render_peak_kib=306532 took=653ms waited=30ms
|
||||
2026-11-05T20:33:46.805Z INFO src.services.processing [rid=-] attachment processed attachment_id=25b27488-8a83-4b66-841c-421fc2562fff job_id=8331854a-bc7a-41f8-bc44-bff4410eced1 attempt=1 variants=3 render_peak_kib=306488 took=667ms waited=484ms
|
||||
2026-11-05T20:33:47.305Z INFO src.services.processing [rid=-] attachment processed attachment_id=c4df8783-baba-490b-9435-ab24fa321f3e job_id=f9ed0bea-b0d9-4f03-aa0b-3baa23b78b3c attempt=1 variants=3 render_peak_kib=306580 took=661ms waited=399ms
|
||||
2026-11-05T20:33:47.813Z INFO src.services.processing [rid=-] attachment processed attachment_id=b01e93c1-077f-44f5-b8b0-df9ab81a82a2 job_id=e57e43cc-9ebe-4abb-95ab-efb9e2db60ee attempt=1 variants=3 render_peak_kib=306472 took=664ms waited=369ms
|
||||
2026-11-05T20:33:48.305Z INFO src.services.processing [rid=-] attachment processed attachment_id=1d9152c2-48c2-417b-800d-c90498b473e4 job_id=6ff6992d-77c8-45c6-bbce-6077e66e6635 attempt=1 variants=3 render_peak_kib=306504 took=651ms waited=314ms
|
||||
2026-11-05T20:33:48.819Z INFO src.services.processing [rid=-] attachment processed attachment_id=73e45df9-fe52-4bee-a2a6-91d566b3c8a1 job_id=6c6365f6-f5d2-41a4-97ff-29efa428cfcb attempt=1 variants=3 render_peak_kib=306472 took=656ms waited=259ms
|
||||
2026-11-05T20:33:49.327Z INFO src.services.processing [rid=-] attachment processed attachment_id=ab7e8625-fde4-4dbd-a654-93a062534d02 job_id=3b7f4725-89b2-47f1-8ca2-c13ddd5fd0dd attempt=1 variants=3 render_peak_kib=306552 took=654ms waited=210ms
|
||||
2026-11-05T20:33:49.837Z INFO src.services.processing [rid=-] attachment processed attachment_id=ec01f049-48db-4c35-a65b-fc9c42a7f84b job_id=08142853-74b7-4e6a-92dd-4952027a669a attempt=1 variants=3 render_peak_kib=306504 took=659ms waited=155ms
|
||||
2026-11-05T20:33:50.363Z INFO src.services.processing [rid=-] attachment processed attachment_id=949ebffa-a091-47f1-9140-79d2bf5a51bf job_id=7329735b-cc8d-4a2b-aff8-470cc369373b attempt=1 variants=3 render_peak_kib=306568 took=656ms waited=119ms
|
||||
2026-11-05T20:33:50.873Z INFO src.services.processing [rid=-] attachment processed attachment_id=cfd6034e-0e3e-4f61-8de8-e4de8e8c31eb job_id=416595c9-9dce-4e9c-910b-6de37540a07b attempt=1 variants=3 render_peak_kib=306556 took=657ms waited=70ms
|
||||
2026-11-05T20:33:51.379Z INFO src.services.processing [rid=-] attachment processed attachment_id=5c9d44ac-8d93-4203-b818-ee9ef2f793dd job_id=fb8614f9-49c9-4060-9bef-993164b65e48 attempt=1 variants=3 render_peak_kib=306492 took=657ms waited=14ms
|
||||
2026-11-05T20:33:52.421Z INFO src.services.processing [rid=-] attachment processed attachment_id=afc38d75-27f7-473f-ad1a-96b31b01b321 job_id=963ef5d6-226b-4d58-b959-90c0cebc00e1 attempt=1 variants=3 render_peak_kib=306496 took=680ms waited=473ms
|
||||
2026-11-05T20:33:52.902Z INFO src.services.processing [rid=-] attachment processed attachment_id=93faa552-4bdc-4ee1-99ef-0917548ce4b0 job_id=12ebabe0-5216-48a3-9300-f9acf6b433fd attempt=1 variants=3 render_peak_kib=306468 took=655ms waited=390ms
|
||||
2026-11-05T20:33:53.409Z INFO src.services.processing [rid=-] attachment processed attachment_id=7425c1d0-698e-4ecf-95a0-07572fd05763 job_id=af7b0217-db32-4438-ab79-829cdcd46a10 attempt=1 variants=3 render_peak_kib=306520 took=652ms waited=357ms
|
||||
2026-11-05T20:33:53.931Z INFO src.services.processing [rid=-] attachment processed attachment_id=85f7630a-911e-4ac8-899c-4fed3a831e57 job_id=a54ec29b-a3e5-453e-b6a6-0b4cec9862ef attempt=1 variants=3 render_peak_kib=306472 took=657ms waited=318ms
|
||||
2026-11-05T20:33:54.436Z INFO src.services.processing [rid=-] attachment processed attachment_id=bcb5b5d5-ad13-4177-9550-1090b7e8c303 job_id=1fb79ef2-51cc-4bf2-90cf-492002a003f0 attempt=1 variants=3 render_peak_kib=306512 took=657ms waited=260ms
|
||||
2026-11-05T20:33:54.943Z INFO src.services.processing [rid=-] attachment processed attachment_id=7d325fc1-1724-4f3b-841f-1fec766f2cc2 job_id=e219964d-ba0b-473a-8774-114ff3d103d6 attempt=1 variants=3 render_peak_kib=306512 took=656ms waited=210ms
|
||||
2026-11-05T20:33:55.460Z INFO src.services.processing [rid=-] attachment processed attachment_id=70089b84-dc92-47d8-b3db-a03dd36086df job_id=5ec09d57-1c3d-4799-86e8-16bd8887ac69 attempt=1 variants=3 render_peak_kib=306504 took=661ms waited=160ms
|
||||
2026-11-05T20:33:55.971Z INFO src.services.processing [rid=-] attachment processed attachment_id=c178e1f1-b68f-4be6-b932-b6b64e9c8436 job_id=13bf946f-4be3-4ef5-88c0-b5b451b6096f attempt=1 variants=3 render_peak_kib=306540 took=662ms waited=105ms
|
||||
2026-11-05T20:33:56.474Z INFO src.services.processing [rid=-] attachment processed attachment_id=01520b9b-a1c0-43ea-b4bc-2b1cff621cfb job_id=ef6f3bf2-10ad-4b5b-a94e-a0d196f43f96 attempt=1 variants=3 render_peak_kib=306496 took=657ms waited=50ms
|
||||
2026-11-05T20:33:57.490Z INFO src.services.processing [rid=-] attachment processed attachment_id=11f1c955-af72-4bda-acb3-bfa3a2828a22 job_id=5a0719de-19aa-42dc-979a-ab2a7585f994 attempt=1 variants=3 render_peak_kib=306452 took=660ms waited=507ms
|
||||
2026-11-05T20:33:58.000Z INFO src.services.processing [rid=-] attachment processed attachment_id=cae89961-263f-478d-b657-d5a062d1d417 job_id=bc19c84f-72e2-43e0-a702-79ba6aa6e8be attempt=1 variants=3 render_peak_kib=306472 took=661ms waited=424ms
|
||||
2026-11-05T20:33:58.510Z INFO src.services.processing [rid=-] attachment processed attachment_id=91006be4-c017-495b-b551-b8709d24bb97 job_id=d39f1b27-52f1-476a-b57c-a3439de4d11d attempt=1 variants=3 render_peak_kib=306516 took=659ms waited=400ms
|
||||
2026-11-05T20:33:59.012Z INFO src.services.processing [rid=-] attachment processed attachment_id=7e4cd9a6-a002-41de-a2a9-5609ef0a64bd job_id=7cf4b244-5175-43db-8742-d86b71f4dc86 attempt=1 variants=3 render_peak_kib=306456 took=655ms waited=344ms
|
||||
2026-11-05T20:34:01.024Z INFO src.services.processing [rid=-] attachment processed attachment_id=f6b1e01c-a46d-4648-824f-1268c002e5b4 job_id=9e64fd26-d5b5-47c7-8062-f4476bd6cf60 attempt=1 variants=3 render_peak_kib=306540 took=651ms waited=468ms
|
||||
2026-11-05T20:34:03.065Z INFO src.services.processing [rid=-] attachment processed attachment_id=573484f2-59d3-4a02-b033-0477d64ea3f4 job_id=0fdabe95-baab-441f-b4f2-5e28dcf2b773 attempt=1 variants=3 render_peak_kib=306480 took=667ms waited=491ms
|
||||
2026-11-05T20:34:04.557Z INFO src.services.processing [rid=-] attachment processed attachment_id=4f35f3d3-e838-4ca1-805e-08046b526d49 job_id=247c52de-8f93-458f-8da6-cc192bf0c628 attempt=1 variants=3 render_peak_kib=306524 took=646ms waited=10ms
|
||||
2026-11-05T20:34:06.590Z INFO src.services.processing [rid=-] attachment processed attachment_id=43986b63-3b78-4f05-9c3b-03bd04523aaf job_id=1baf67fa-432f-4de1-9239-c06970c8f721 attempt=1 variants=3 render_peak_kib=306524 took=653ms waited=63ms
|
||||
2026-11-05T20:34:08.631Z INFO src.services.processing [rid=-] attachment processed attachment_id=26f611a2-a59a-452b-9ebc-df7fe04eb68f job_id=ad1690b4-e094-4792-adeb-09ff0d0a9a17 attempt=1 variants=3 render_peak_kib=306540 took=661ms waited=65ms
|
||||
2026-11-05T20:34:10.648Z INFO src.services.processing [rid=-] attachment processed attachment_id=f6c2b007-9dcb-4bdc-aefa-0b8818d8b425 job_id=85d4ee12-022d-4dd8-90bc-d941e059c95d attempt=1 variants=3 render_peak_kib=306544 took=651ms waited=92ms
|
||||
2026-11-05T20:34:12.674Z INFO src.services.processing [rid=-] attachment processed attachment_id=e7cb9fc0-a6fa-47e7-a561-032f09f4045b job_id=640940d2-0d16-4240-a9d5-6c8cb9edbd45 attempt=1 variants=3 render_peak_kib=306496 took=652ms waited=119ms
|
||||
2026-11-05T20:34:14.722Z INFO src.services.processing [rid=-] attachment processed attachment_id=be79a66a-f540-4d72-99dd-02c5efa7db17 job_id=71d4276d-5d25-4a5c-8c1b-3149e433b7c8 attempt=1 variants=3 render_peak_kib=306484 took=662ms waited=158ms
|
||||
2026-11-05T20:34:16.756Z INFO src.services.processing [rid=-] attachment processed attachment_id=bb27dd73-bbdd-4e23-8568-06ddc9ee993b job_id=a54c5535-e9a8-40f6-9c2f-03bca5b70460 attempt=1 variants=3 render_peak_kib=306448 took=667ms waited=187ms
|
||||
2026-11-05T20:34:18.800Z INFO src.services.processing [rid=-] attachment processed attachment_id=f747dd41-b784-4bd7-a8d0-2e5a85d4e6d2 job_id=17b1c313-945d-4d48-9e3b-b2c9ff41bd90 attempt=1 variants=3 render_peak_kib=306456 took=675ms waited=230ms
|
||||
2026-11-05T20:34:20.821Z INFO src.services.processing [rid=-] attachment processed attachment_id=d716aefb-b545-4e74-86a5-3b1ac6525c9e job_id=e273d567-438e-4a90-9156-fb360e10a991 attempt=1 variants=3 render_peak_kib=306476 took=668ms waited=251ms
|
||||
2026-11-05T20:34:22.824Z INFO src.services.processing [rid=-] attachment processed attachment_id=806de701-a53c-4204-966c-bad5e2e35865 job_id=da457057-bf9e-4e8c-ba86-69aae4599ffa attempt=1 variants=3 render_peak_kib=306540 took=644ms waited=274ms
|
||||
2026-11-05T20:34:24.887Z INFO src.services.processing [rid=-] attachment processed attachment_id=c4ed389d-8930-4d0b-a5dd-52603add4a74 job_id=9571c296-453b-49b9-be96-77dc27bec9c1 attempt=1 variants=3 render_peak_kib=306412 took=666ms waited=316ms
|
||||
2026-11-05T20:34:26.922Z INFO src.services.processing [rid=-] attachment processed attachment_id=28dc600c-2cb8-4ca5-bb8d-87966c1d20a7 job_id=835235af-f94d-4531-9beb-c60578860bda attempt=1 variants=3 render_peak_kib=306484 took=667ms waited=351ms
|
||||
2026-11-05T20:34:28.960Z INFO src.services.processing [rid=-] attachment processed attachment_id=b05c0d67-ac51-4fcf-a32d-d7c742b1a58f job_id=cc717eaa-cc5c-4897-96b7-67986a45b733 attempt=1 variants=3 render_peak_kib=306652 took=675ms waited=385ms
|
||||
2026-11-05T20:34:30.984Z INFO src.services.processing [rid=-] attachment processed attachment_id=0af33e81-b4cb-441c-89ff-66b1a2e3834b job_id=0e269531-a9a7-4f3f-9c6e-dd37221f9f89 attempt=1 variants=3 render_peak_kib=306512 took=667ms waited=420ms
|
||||
2026-11-05T20:34:33.025Z INFO src.services.processing [rid=-] attachment processed attachment_id=2b403af2-14bf-422f-9c53-c209c6d431dc job_id=c7823a38-da7c-4abc-a793-2fb0e8a7f719 attempt=1 variants=3 render_peak_kib=306452 took=676ms waited=446ms
|
||||
2026-11-05T20:34:35.028Z INFO src.services.processing [rid=-] attachment processed attachment_id=65c19aa9-b11a-469e-9dc9-b1234ec68b1a job_id=f1bfe066-90a3-4be1-a96b-5ac6cc5479ad attempt=1 variants=3 render_peak_kib=306580 took=656ms waited=472ms
|
||||
2026-11-05T20:34:37.043Z INFO src.services.processing [rid=-] attachment processed attachment_id=1a871ebc-21b3-43ef-836d-31739e0dcc94 job_id=e032ce52-2942-47fb-8844-17d0a22ef388 attempt=1 variants=3 render_peak_kib=306456 took=646ms waited=494ms
|
||||
2026-11-05T20:34:38.559Z INFO src.services.processing [rid=-] attachment processed attachment_id=327c403f-1b16-41c4-bdd2-6431c4961497 job_id=db0f60d3-00a1-48dd-b08f-021395cd4b83 attempt=1 variants=3 render_peak_kib=306632 took=643ms waited=9ms
|
||||
2026-11-05T20:34:40.110Z INFO src.services.processing [rid=-] attachment processed attachment_id=12e62e0a-526a-4d4c-8afd-04aab001878c job_id=a7b7df80-6eaf-40c2-b340-755f61094fd5 attempt=1 variants=3 render_peak_kib=306592 took=683ms waited=27ms
|
||||
2026-11-05T20:34:40.112Z INFO src.services.processing [rid=-] attachment processed attachment_id=3130ea87-4583-4bc7-b032-2588ff35a8cf job_id=da646cc8-b788-48e0-8154-ab5be934fb18 attempt=1 variants=3 render_peak_kib=306528 took=686ms waited=164ms
|
||||
2026-11-05T20:34:40.785Z INFO src.services.processing [rid=-] attachment processed attachment_id=41afbe08-ebc4-420e-88b9-4f19a88361d6 job_id=e7c61f60-0612-4fda-b202-6d6f58e54a1c attempt=1 variants=3 render_peak_kib=306640 took=673ms waited=543ms
|
||||
2026-11-05T20:34:40.788Z INFO src.services.processing [rid=-] attachment processed attachment_id=38aecf82-ab2d-454c-a879-169eb740bcdc job_id=33038e65-f0b6-47b7-bd3c-f2d39b657a0a attempt=1 variants=3 render_peak_kib=306504 took=674ms waited=374ms
|
||||
2026-11-05T20:34:41.461Z INFO src.services.processing [rid=-] attachment processed attachment_id=bdbf4811-0ad6-4f1f-b955-09b2dbd2f9e3 job_id=d94a7166-eca9-47fd-ab94-c8fe3b4e00c4 attempt=1 variants=3 render_peak_kib=306544 took=675ms waited=883ms
|
||||
2026-11-05T20:34:41.461Z INFO src.services.processing [rid=-] attachment processed attachment_id=5dadb0e0-bd32-4ab4-9aaa-a262f350b71f job_id=ef3ba475-85eb-4857-8d51-1171a2e9b3ab attempt=1 variants=3 render_peak_kib=306472 took=673ms waited=716ms
|
||||
2026-11-05T20:34:42.136Z INFO src.services.processing [rid=-] attachment processed attachment_id=84233de3-4619-464c-a152-44667ba2633a job_id=21f50e44-dce6-49b1-8a3e-8697efcade5e attempt=1 variants=3 render_peak_kib=306504 took=674ms waited=1221ms
|
||||
2026-11-05T20:34:42.136Z INFO src.services.processing [rid=-] attachment processed attachment_id=e725b708-f720-420d-a572-5f02c8415ac1 job_id=ae80328a-e2b5-4ce8-8edb-1853af408c4c attempt=1 variants=3 render_peak_kib=306540 took=674ms waited=1048ms
|
||||
2026-11-05T20:34:42.812Z INFO src.services.processing [rid=-] attachment processed attachment_id=062ebbfc-696b-4e68-8cb3-1e6fa4464554 job_id=aa4e0c24-a96c-459d-bb9f-dde7906aff0c attempt=1 variants=3 render_peak_kib=306528 took=674ms waited=1388ms
|
||||
2026-11-05T20:34:42.813Z INFO src.services.processing [rid=-] attachment processed attachment_id=2903d9be-e649-4b48-a7ca-f5a688ca41a9 job_id=3f99a99c-00a7-4304-8e25-5af32eaf900a attempt=1 variants=3 render_peak_kib=306564 took=676ms waited=1559ms
|
||||
2026-11-05T20:34:43.489Z INFO src.services.processing [rid=-] attachment processed attachment_id=342e77fe-3000-480c-9bc4-abf5b34504e9 job_id=42ba5b58-1445-4ad0-9849-dff863431c7d attempt=1 variants=3 render_peak_kib=306548 took=675ms waited=1725ms
|
||||
2026-11-05T20:34:43.489Z INFO src.services.processing [rid=-] attachment processed attachment_id=8dc716e0-9eb8-4b45-9a55-7dcfe7a84d1d job_id=f1b7cc56-5448-4575-8b0a-315f0f9d0c2a attempt=1 variants=3 render_peak_kib=306472 took=676ms waited=1897ms
|
||||
2026-11-05T20:34:44.179Z INFO src.services.processing [rid=-] attachment processed attachment_id=b65ed9f9-41f6-4272-af4d-97b820122ded job_id=0d827b0e-ee28-4963-acb3-fe674b1abd6e attempt=1 variants=3 render_peak_kib=306456 took=688ms waited=2067ms
|
||||
2026-11-05T20:34:44.183Z INFO src.services.processing [rid=-] attachment processed attachment_id=4575c606-b171-47a8-bd53-127dec61fde9 job_id=1e142e33-7f42-427b-985a-a1caef037c85 attempt=1 variants=3 render_peak_kib=306540 took=692ms waited=2238ms
|
||||
2026-11-05T20:34:44.858Z INFO src.services.processing [rid=-] attachment processed attachment_id=2ae8603a-9a0c-48e7-b067-ce7e4c07f9e3 job_id=4dd98ce8-34e6-4c36-8923-21e3b61fee88 attempt=1 variants=3 render_peak_kib=306464 took=678ms waited=2590ms
|
||||
2026-11-05T20:34:44.859Z INFO src.services.processing [rid=-] attachment processed attachment_id=f2f7cf73-dac3-40a5-a8d9-1cdf3e21e0a6 job_id=732b5c40-41ac-4f73-8695-72153669ccd5 attempt=1 variants=3 render_peak_kib=306652 took=675ms waited=2421ms
|
||||
2026-11-05T20:34:45.534Z INFO src.services.processing [rid=-] attachment processed attachment_id=83b68628-2b1b-4125-94ba-4750dc5df729 job_id=ffeaa174-cf30-4852-9fb7-174c2d26deed attempt=1 variants=3 render_peak_kib=306464 took=675ms waited=2934ms
|
||||
2026-11-05T20:34:45.540Z INFO src.services.processing [rid=-] attachment processed attachment_id=a39a275a-2f08-46aa-b041-413178f10f9b job_id=01dcbd7a-c902-42c2-b307-9a903e522ea0 attempt=1 variants=3 render_peak_kib=306548 took=680ms waited=2765ms
|
||||
2026-11-05T20:34:46.215Z INFO src.services.processing [rid=-] attachment processed attachment_id=86bb437d-2e55-433b-892d-9803fdd95334 job_id=a8469850-5341-4745-9b17-edb8fa4a9f5f attempt=1 variants=3 render_peak_kib=306540 took=680ms waited=3274ms
|
||||
2026-11-05T20:34:46.217Z INFO src.services.processing [rid=-] attachment processed attachment_id=df9c563a-42e3-48b6-b17a-dfbf9c7abf04 job_id=4b49ecb8-fdd3-46d1-90e0-87f3987a6e7d attempt=1 variants=3 render_peak_kib=306524 took=670ms waited=3105ms
|
||||
2026-11-05T20:34:46.894Z INFO src.services.processing [rid=-] attachment processed attachment_id=0d534025-c8fd-4ee3-84de-85e0e95e0281 job_id=81590602-7764-47b0-8fee-d16b42e01706 attempt=1 variants=3 render_peak_kib=306652 took=676ms waited=3450ms
|
||||
2026-11-05T20:34:46.896Z INFO src.services.processing [rid=-] attachment processed attachment_id=3ba7d028-3664-48d9-a698-f1356e852c1b job_id=7bdecfda-4783-4c15-97ae-b0514b6d475e attempt=1 variants=3 render_peak_kib=306568 took=678ms waited=3619ms
|
||||
2026-11-05T20:34:47.584Z INFO src.services.processing [rid=-] attachment processed attachment_id=510fedf8-c9e8-4b6b-a72c-e922b08effc6 job_id=4c8b7cb0-70fb-4852-949e-f2d12cdd7563 attempt=1 variants=3 render_peak_kib=306444 took=687ms waited=3786ms
|
||||
2026-11-05T20:34:47.584Z INFO src.services.processing [rid=-] attachment processed attachment_id=3f6355d2-33f0-43fb-8bd8-26812a2207c1 job_id=f3a95aca-b84f-4c9e-997a-7267849905dc attempt=1 variants=3 render_peak_kib=306528 took=688ms waited=3960ms
|
||||
2026-11-05T20:34:48.265Z INFO src.services.processing [rid=-] attachment processed attachment_id=1b00f69f-c2d6-40f1-b12a-c9ea9aa0d503 job_id=2b03a7fb-e80f-44c8-a0a8-8579f589bf3e attempt=1 variants=3 render_peak_kib=306528 took=680ms waited=4312ms
|
||||
2026-11-05T20:34:48.272Z INFO src.services.processing [rid=-] attachment processed attachment_id=1de47f93-3c7f-4b94-bc9c-e5915aad65c5 job_id=7bf4add4-65e5-451b-b730-add628534599 attempt=1 variants=3 render_peak_kib=306472 took=687ms waited=4141ms
|
||||
2026-11-05T20:34:48.938Z INFO src.services.processing [rid=-] attachment processed attachment_id=ae21d05c-1d37-46b4-a928-fcd3d5303490 job_id=0080c7de-9859-4221-ba3a-0095dfd50856 attempt=1 variants=3 render_peak_kib=306504 took=671ms waited=4655ms
|
||||
2026-11-05T20:34:48.947Z INFO src.services.processing [rid=-] attachment processed attachment_id=15a15e20-e42a-4d92-ae6a-7d78e4add863 job_id=8ee996a5-1044-4ca7-a230-7aa1b624be69 attempt=1 variants=3 render_peak_kib=306536 took=674ms waited=4490ms
|
||||
2026-11-05T20:34:49.641Z INFO src.services.processing [rid=-] attachment processed attachment_id=8527ce8c-cce6-4d01-bd2a-5fbdd40cfcbc job_id=7e674473-0432-4995-9f8c-5af338196b7f attempt=1 variants=3 render_peak_kib=306500 took=701ms waited=4989ms
|
||||
2026-11-05T20:34:49.642Z INFO src.services.processing [rid=-] attachment processed attachment_id=b9d6dd6b-63e9-4aba-90f5-1a012b9deceb job_id=ee405d1f-4ccc-41dc-ae8a-d1328296aced attempt=1 variants=3 render_peak_kib=306492 took=694ms waited=4831ms
|
||||
2026-11-05T20:34:50.341Z INFO src.services.processing [rid=-] attachment processed attachment_id=78e3c7ff-b420-48f9-b328-7d32337c28a6 job_id=7a013115-bc8b-49e2-960c-f7637d92815f attempt=1 variants=3 render_peak_kib=306560 took=698ms waited=5359ms
|
||||
2026-11-05T20:34:50.346Z INFO src.services.processing [rid=-] attachment processed attachment_id=4da83b40-e8e3-4559-b752-4bcc6d709270 job_id=6e332293-fd95-493d-9284-e355c6550e04 attempt=1 variants=3 render_peak_kib=306632 took=702ms waited=5186ms
|
||||
2026-11-05T20:34:51.050Z INFO src.services.processing [rid=-] attachment processed attachment_id=a344ae1f-8a57-471d-b58c-aee492d91095 job_id=1ee595ca-7d01-4aac-a3a5-b942e95ff405 attempt=1 variants=3 render_peak_kib=306512 took=704ms waited=5719ms
|
||||
2026-11-05T20:34:51.050Z INFO src.services.processing [rid=-] attachment processed attachment_id=17906a5e-bedf-4eab-8664-9da70647518f job_id=e8201cd0-1477-44f4-8307-c0971f581b58 attempt=1 variants=3 render_peak_kib=306484 took=703ms waited=5555ms
|
||||
2026-11-05T20:34:51.749Z INFO src.services.processing [rid=-] attachment processed attachment_id=f5cfba83-a72e-441b-971f-5433ec299e83 job_id=79092a27-c9bc-41f4-af2e-97dd09af6af5 attempt=1 variants=3 render_peak_kib=306472 took=698ms waited=6094ms
|
||||
2026-11-05T20:34:51.750Z INFO src.services.processing [rid=-] attachment processed attachment_id=c70919aa-fda7-4de6-b199-aa78a01a0ee4 job_id=d6a9d2a1-5d9c-443d-9229-b9b7981ab032 attempt=1 variants=3 render_peak_kib=306484 took=699ms waited=5922ms
|
||||
2026-11-05T20:34:52.433Z INFO src.services.processing [rid=-] attachment processed attachment_id=9f4463cb-1925-4ffe-996a-a6eede1e4165 job_id=5fb79a42-fc12-42c6-935f-f187e4e48040 attempt=1 variants=3 render_peak_kib=306452 took=682ms waited=6454ms
|
||||
2026-11-05T20:34:52.433Z INFO src.services.processing [rid=-] attachment processed attachment_id=88fc08d1-cd5c-447c-881a-247aa6d6d0e4 job_id=2101835b-cfb8-44da-a997-77832eaccdc7 attempt=1 variants=3 render_peak_kib=306520 took=682ms waited=6287ms
|
||||
2026-11-05T20:34:53.119Z INFO src.services.processing [rid=-] attachment processed attachment_id=da16b6c2-edc6-411f-866e-22b912742c20 job_id=70f0450a-a57f-408d-a647-3575ff02c4b1 attempt=1 variants=3 render_peak_kib=306496 took=685ms waited=6801ms
|
||||
2026-11-05T20:34:53.120Z INFO src.services.processing [rid=-] attachment processed attachment_id=1cfbccad-9570-4314-8ead-93f8347b090e job_id=a10441cf-4698-445b-8c78-0d7fe3e091cd attempt=1 variants=3 render_peak_kib=306492 took=686ms waited=6631ms
|
||||
2026-11-05T20:34:53.823Z INFO src.services.processing [rid=-] attachment processed attachment_id=ee43baf6-edb9-4ddd-8498-511f6c520100 job_id=130a2d81-2162-4ebc-8f30-b7edef180c71 attempt=1 variants=3 render_peak_kib=306516 took=701ms waited=6986ms
|
||||
2026-11-05T20:34:53.823Z INFO src.services.processing [rid=-] attachment processed attachment_id=ae9fc576-8133-4bfd-97b3-1f78886df7a8 job_id=5f6a0800-61bb-4554-aa56-39aa0614c8db attempt=1 variants=3 render_peak_kib=306572 took=702ms waited=7149ms
|
||||
2026-11-05T20:34:54.509Z INFO src.services.processing [rid=-] attachment processed attachment_id=f3d5e18e-9f4d-44e0-91d7-e104a013198a job_id=8b405a9f-c133-4be6-91d3-e8da34df85df attempt=1 variants=3 render_peak_kib=306476 took=684ms waited=7347ms
|
||||
2026-11-05T20:34:54.512Z INFO src.services.processing [rid=-] attachment processed attachment_id=5640ec64-b875-442a-b9b2-41924627a9ce job_id=fdba9172-da4c-45bc-a045-3f540fbd476c attempt=1 variants=3 render_peak_kib=306560 took=688ms waited=7518ms
|
||||
2026-11-05T20:34:55.214Z INFO src.services.processing [rid=-] attachment processed attachment_id=eb7182be-6517-4b42-9628-f27de798ff8c job_id=f7855117-5eb8-4755-96f8-4af8f9311f5f attempt=1 variants=3 render_peak_kib=306572 took=704ms waited=7867ms
|
||||
2026-11-05T20:34:55.215Z INFO src.services.processing [rid=-] attachment processed attachment_id=5d2fe0f4-c3e8-4e92-8c55-9264f7868fa8 job_id=5fed4847-8130-46c0-8b21-54539cfeb8eb attempt=1 variants=3 render_peak_kib=306496 took=701ms waited=7703ms
|
||||
2026-11-05T20:34:55.888Z INFO src.services.processing [rid=-] attachment processed attachment_id=4c41e4a3-1b88-4a76-9748-dd5097c1613c job_id=8348a2cf-daf2-4ae5-8a4e-3ddcbac0413c attempt=1 variants=3 render_peak_kib=306544 took=673ms waited=8232ms
|
||||
2026-11-05T20:34:55.897Z INFO src.services.processing [rid=-] attachment processed attachment_id=97dd5c8c-ee10-4a35-a7a4-cf22ca5b133a job_id=8977e71c-14b7-4a38-9c72-633bc08a81e0 attempt=1 variants=3 render_peak_kib=306504 took=681ms waited=8064ms
|
||||
2026-11-05T20:34:56.591Z INFO src.services.processing [rid=-] attachment processed attachment_id=8fc479f6-c3c1-4bb9-9f5d-f748004180da job_id=2f300b08-1aa0-4ce3-a06f-3e923a1592b3 attempt=1 variants=3 render_peak_kib=306536 took=693ms waited=8408ms
|
||||
2026-11-05T20:34:56.595Z INFO src.services.processing [rid=-] attachment processed attachment_id=81c176d0-a34b-4833-aa7b-b41f76daa768 job_id=c1d539e1-e444-4cb7-b7bd-1e08f73bd88c attempt=1 variants=3 render_peak_kib=306540 took=704ms waited=8572ms
|
||||
2026-11-05T20:34:57.285Z INFO src.services.processing [rid=-] attachment processed attachment_id=16829385-49da-4438-a6a8-2ec501bcde6b job_id=cb98e127-4d32-41d7-868b-9a6c6a788fbe attempt=1 variants=3 render_peak_kib=306560 took=689ms waited=8771ms
|
||||
2026-11-05T20:34:57.286Z INFO src.services.processing [rid=-] attachment processed attachment_id=a9dafb84-ef9a-4389-9218-494be124bf1a job_id=fd724160-04b1-4e2c-bad2-d899c8251a5f attempt=1 variants=3 render_peak_kib=306496 took=694ms waited=8939ms
|
||||
2026-11-05T20:34:57.983Z INFO src.services.processing [rid=-] attachment processed attachment_id=6d9b808b-30a1-46c0-8887-dc47cd2c4eef job_id=fc710b02-8bb7-4816-88a7-6be0174d3ea5 attempt=1 variants=3 render_peak_kib=306492 took=696ms waited=9125ms
|
||||
2026-11-05T20:34:57.983Z INFO src.services.processing [rid=-] attachment processed attachment_id=e5560c3e-719e-4c47-81e7-69b2901985ce job_id=350bccf1-b0fe-4e2f-8631-cdc4483e30c8 attempt=1 variants=3 render_peak_kib=306572 took=697ms waited=9292ms
|
||||
2026-11-05T20:34:58.681Z INFO src.services.processing [rid=-] attachment processed attachment_id=fe838370-46a6-45ed-8c35-7ab69c0490c9 job_id=781e3a58-293f-449c-a9f0-c0766344f63b attempt=1 variants=3 render_peak_kib=306484 took=697ms waited=9485ms
|
||||
2026-11-05T20:34:58.681Z INFO src.services.processing [rid=-] attachment processed attachment_id=aada7922-e9ad-4a4c-988a-00df28937250 job_id=f06eade6-16f5-4089-a0b3-0be658006055 attempt=1 variants=3 render_peak_kib=306564 took=697ms waited=9657ms
|
||||
2026-11-05T20:34:59.382Z INFO src.services.processing [rid=-] attachment processed attachment_id=bb630203-1fab-4f2b-9a0c-768f5fd7849e job_id=74655172-2eb8-4822-b986-5d13247e87cb attempt=1 variants=3 render_peak_kib=306552 took=699ms waited=10015ms
|
||||
2026-11-05T20:34:59.382Z INFO src.services.processing [rid=-] attachment processed attachment_id=c121677d-6dd1-4c33-8440-ff703a622404 job_id=aa0a10ae-ba64-455d-b169-a2f1920f1a6b attempt=1 variants=3 render_peak_kib=306464 took=699ms waited=9849ms
|
||||
2026-11-05T20:35:00.078Z INFO src.services.processing [rid=-] attachment processed attachment_id=4c568e8c-d283-4317-bada-1c3b6cd2481f job_id=c82202d9-5900-4834-90b5-63bbd9631cda attempt=1 variants=3 render_peak_kib=306524 took=695ms waited=10380ms
|
||||
2026-11-05T20:35:00.080Z INFO src.services.processing [rid=-] attachment processed attachment_id=f555859d-db7a-4396-9eea-0b993ed24698 job_id=93fbbf44-b3c3-4a56-a7b9-7b062bb33b70 attempt=1 variants=3 render_peak_kib=306528 took=696ms waited=10211ms
|
||||
2026-11-05T20:35:00.801Z INFO src.services.processing [rid=-] attachment processed attachment_id=fc670baa-cf5f-41fd-84e3-5a0da0b87d7d job_id=29a6a1cf-f061-4900-a282-375193ea14b4 attempt=1 variants=3 render_peak_kib=306552 took=721ms waited=10736ms
|
||||
2026-11-05T20:35:00.803Z INFO src.services.processing [rid=-] attachment processed attachment_id=be3c815d-ec72-4739-840c-63be1e125323 job_id=c97b9425-7e43-43dd-90d0-5b31edca2b0f attempt=1 variants=3 render_peak_kib=306544 took=722ms waited=10566ms
|
||||
2026-11-05T20:35:01.523Z INFO src.services.processing [rid=-] attachment processed attachment_id=d7430d6e-bfcf-4155-b51d-b61c9f9aa465 job_id=115c9752-d0c9-41d5-aaf7-9c5d0d81fc22 attempt=1 variants=3 render_peak_kib=306452 took=719ms waited=10958ms
|
||||
2026-11-05T20:35:01.524Z INFO src.services.processing [rid=-] attachment processed attachment_id=a0d71ff3-7f8d-4075-a46e-fcc2373d4ce8 job_id=2d1dc0d0-f2b4-41ed-95bb-df72675b2c1d attempt=1 variants=3 render_peak_kib=306452 took=721ms waited=11128ms
|
||||
2026-11-05T20:35:02.247Z INFO src.services.processing [rid=-] attachment processed attachment_id=73349b4c-2afe-4be4-8b28-9a9c5c9e9bc2 job_id=5830a914-59f9-4670-96b7-a85ca942dac5 attempt=1 variants=3 render_peak_kib=306532 took=721ms waited=11338ms
|
||||
2026-11-05T20:35:02.248Z INFO src.services.processing [rid=-] attachment processed attachment_id=b571c029-c394-4a4f-9b6a-5940760844cf job_id=7c2c7e7d-058b-4986-8b1a-8a26ba961d01 attempt=1 variants=3 render_peak_kib=306468 took=723ms waited=11510ms
|
||||
2026-11-05T20:35:02.937Z INFO src.services.processing [rid=-] attachment processed attachment_id=56a5e97b-8e8a-4a24-b543-a40c1688b8b3 job_id=9f32bbb9-5753-468f-a7e1-54e292e7c5e0 attempt=1 variants=3 render_peak_kib=306540 took=686ms waited=11733ms
|
||||
2026-11-05T20:35:02.964Z INFO src.services.processing [rid=-] attachment processed attachment_id=a55ebc78-2167-4057-8ada-88f1f13974b6 job_id=4b5fe12d-37dc-41ed-9e11-8a67edec712a attempt=1 variants=3 render_peak_kib=306460 took=716ms waited=11896ms
|
||||
2026-11-05T20:35:03.638Z INFO src.services.processing [rid=-] attachment processed attachment_id=f0a74a60-a1bd-4aca-8b21-ded449d5ccc0 job_id=364cf86b-d904-48bb-b632-7aff75544e34 attempt=1 variants=3 render_peak_kib=306492 took=700ms waited=12245ms
|
||||
2026-11-05T20:35:03.666Z INFO src.services.processing [rid=-] attachment processed attachment_id=21d64a7b-c189-4d7c-bf15-5f8ad7fd86a7 job_id=eaec263b-5264-41d9-ba24-c6c70f066d71 attempt=1 variants=3 render_peak_kib=306524 took=701ms waited=12099ms
|
||||
2026-11-05T20:35:04.327Z INFO src.services.processing [rid=-] attachment processed attachment_id=568bb57e-c23b-4643-ad65-c82fe0b31fa2 job_id=be67e91e-a096-4830-b0ac-ef2dde35cfb0 attempt=1 variants=3 render_peak_kib=306516 took=688ms waited=12614ms
|
||||
2026-11-05T20:35:04.352Z INFO src.services.processing [rid=-] attachment processed attachment_id=6a4dfb63-d746-4d99-bfd9-40d7ef06f766 job_id=6430ae7d-e9bb-4be7-85c4-3f2bbe335c34 attempt=1 variants=3 render_peak_kib=306480 took=684ms waited=12477ms
|
||||
2026-11-05T20:35:05.007Z INFO src.services.processing [rid=-] attachment processed attachment_id=734d898b-5739-411d-bdad-424236fa8f25 job_id=f4ea173e-61db-4689-9e9b-2013dae21738 attempt=1 variants=3 render_peak_kib=306480 took=679ms waited=12961ms
|
||||
2026-11-05T20:35:05.048Z INFO src.services.processing [rid=-] attachment processed attachment_id=7e6d116c-824c-48ca-97aa-d78c7b34ba9e job_id=cce63606-beb0-458b-bae4-f67dbbface32 attempt=1 variants=3 render_peak_kib=306504 took=694ms waited=12822ms
|
||||
2026-11-05T20:35:05.695Z INFO src.services.processing [rid=-] attachment processed attachment_id=85dbf075-ac05-4135-a45f-e08340579089 job_id=db956d26-3f50-49e3-95cd-408fbe08528d attempt=1 variants=3 render_peak_kib=306536 took=687ms waited=13309ms
|
||||
2026-11-05T20:35:05.758Z INFO src.services.processing [rid=-] attachment processed attachment_id=81f016d1-fef5-481d-8a39-0875d59ca836 job_id=e34bc0cc-607f-4622-9b95-b01da94545ca attempt=1 variants=3 render_peak_kib=306560 took=709ms waited=13178ms
|
||||
2026-11-05T20:35:06.403Z INFO src.services.processing [rid=-] attachment processed attachment_id=082aa566-b596-40b0-bbe4-042e24a96562 job_id=d72b02cc-958c-4500-ac78-0f0cc7d584b1 attempt=1 variants=3 render_peak_kib=306440 took=706ms waited=13659ms
|
||||
2026-11-05T20:35:06.445Z INFO src.services.processing [rid=-] attachment processed attachment_id=44ce1067-1f0f-4c16-887d-c06df8d05f82 job_id=6abbadba-51d2-439d-b2cd-32d720aa27e6 attempt=1 variants=3 render_peak_kib=306572 took=683ms waited=13558ms
|
||||
2026-11-05T20:35:07.100Z INFO src.services.processing [rid=-] attachment processed attachment_id=d0b2e75c-183d-4c5b-a87b-b8912d3110ed job_id=cbd8ee0a-1f61-4227-941a-63b35b2b1b31 attempt=1 variants=3 render_peak_kib=306452 took=696ms waited=14027ms
|
||||
2026-11-05T20:35:07.138Z INFO src.services.processing [rid=-] attachment processed attachment_id=035bfed0-7772-4738-954b-ccbbbb0db189 job_id=c531222a-684a-4e12-9eb4-4438acfc5059 attempt=1 variants=3 render_peak_kib=306532 took=686ms waited=13913ms
|
||||
2026-11-05T20:35:07.799Z INFO src.services.processing [rid=-] attachment processed attachment_id=79148b35-7f7f-46d8-bac5-e202658c6760 job_id=4b895691-f09c-4f99-ab07-94fba7ed7833 attempt=1 variants=3 render_peak_kib=306464 took=698ms waited=14388ms
|
||||
2026-11-05T20:35:07.848Z INFO src.services.processing [rid=-] attachment processed attachment_id=00260b16-ef74-4f2c-bd02-9d32a7f3eee3 job_id=03af46bc-646e-4fdd-8ff8-04c8ca9ce65b attempt=1 variants=3 render_peak_kib=306648 took=709ms waited=14260ms
|
||||
2026-11-05T20:35:08.486Z INFO src.services.processing [rid=-] attachment processed attachment_id=5a01b98b-c40c-4628-a3f5-0c7f506fd7d7 job_id=5f86fb4b-a426-417a-8e20-465cebf92344 attempt=1 variants=3 render_peak_kib=306540 took=685ms waited=14750ms
|
||||
2026-11-05T20:35:08.535Z INFO src.services.processing [rid=-] attachment processed attachment_id=40399572-0b69-4a84-bf8c-e7e48fddaeab job_id=14884aa2-1db2-4f3d-ba7a-2bec1173b989 attempt=1 variants=3 render_peak_kib=306520 took=686ms waited=14637ms
|
||||
2026-11-05T20:35:09.184Z INFO src.services.processing [rid=-] attachment processed attachment_id=672b4e92-bedf-412d-bf8c-910552ab4c2c job_id=75db33b8-04ba-42bb-b29e-0ecda0bc498f attempt=1 variants=3 render_peak_kib=306552 took=697ms waited=15100ms
|
||||
2026-11-05T20:35:09.230Z INFO src.services.processing [rid=-] attachment processed attachment_id=31b12660-973e-40e8-b0e8-84efc44270ac job_id=c54bfe18-e252-4597-b08a-f8535657c8a7 attempt=1 variants=3 render_peak_kib=306524 took=693ms waited=14979ms
|
||||
2026-11-05T20:35:09.864Z INFO src.services.processing [rid=-] attachment processed attachment_id=f6e7d88d-229d-495e-8a0d-dd3ff6a77f96 job_id=90df670f-bd72-4716-8bee-ae34bb39a982 attempt=1 variants=3 render_peak_kib=306512 took=678ms waited=15454ms
|
||||
2026-11-05T20:35:09.905Z INFO src.services.processing [rid=-] attachment processed attachment_id=13822f63-8720-4c93-a810-b3b061a9f3a5 job_id=495f3320-d77b-4a6d-a395-dea52c91be42 attempt=1 variants=3 render_peak_kib=306472 took=674ms waited=15340ms
|
||||
2026-11-05T20:35:10.559Z INFO src.services.processing [rid=-] attachment processed attachment_id=b19c3e12-2a73-49d3-8211-887c273b923a job_id=f860bc70-a403-430d-af88-febeca7b2136 attempt=1 variants=3 render_peak_kib=306468 took=696ms waited=15805ms
|
||||
2026-11-05T20:35:10.587Z INFO src.services.processing [rid=-] attachment processed attachment_id=167ab4f0-f7b0-4167-b921-b8ebaae1675b job_id=d83f74d0-de08-44a8-bd01-5688f04fa114 attempt=1 variants=3 render_peak_kib=306476 took=682ms waited=15679ms
|
||||
2026-11-05T20:35:11.277Z INFO src.services.processing [rid=-] attachment processed attachment_id=f5d8b9ec-2293-420c-b8aa-aa2bef22ed78 job_id=40180d60-c51d-43a3-9498-8e15255e5107 attempt=1 variants=3 render_peak_kib=306492 took=717ms waited=16161ms
|
||||
2026-11-05T20:35:11.295Z INFO src.services.processing [rid=-] attachment processed attachment_id=2c8ba7cf-2020-4fcc-a7ca-d8909bc5e726 job_id=1b95266c-1c53-42ec-90d0-918492ffb1d8 attempt=1 variants=3 render_peak_kib=306500 took=707ms waited=16024ms
|
||||
2026-11-05T20:35:12.039Z INFO src.services.processing [rid=-] attachment processed attachment_id=4394a81c-78fe-4191-abb6-31ccad42c601 job_id=0cafdf75-c2a8-4e8f-bd5a-1b8364685acb attempt=1 variants=3 render_peak_kib=306488 took=735ms waited=16395ms
|
||||
2026-11-05T20:35:12.039Z INFO src.services.processing [rid=-] attachment processed attachment_id=21cda083-80ed-4b85-8e37-0bae5992ab05 job_id=5c93d12b-dcbb-49f2-ae95-e28a1b2d56ef attempt=1 variants=3 render_peak_kib=306456 took=735ms waited=16571ms
|
||||
2026-11-05T20:35:12.739Z INFO src.services.processing [rid=-] attachment processed attachment_id=b89f1b03-3064-44bd-9384-ef675fb2048d job_id=50f77ead-5229-4bec-bd0e-313ac484688c attempt=1 variants=3 render_peak_kib=306572 took=698ms waited=16965ms
|
||||
2026-11-05T20:35:12.745Z INFO src.services.processing [rid=-] attachment processed attachment_id=6a8e143d-7a9c-438d-bd16-5688e293d187 job_id=eab09b6d-cdd6-4cde-8642-38778ed4be08 attempt=1 variants=3 render_peak_kib=306572 took=704ms waited=16806ms
|
||||
2026-11-05T20:35:13.435Z INFO src.services.processing [rid=-] attachment processed attachment_id=43e56ba8-e6f8-408a-b058-c34987e945f3 job_id=b4be1cf4-2f1b-4588-b3f5-0709b248bdd3 attempt=1 variants=3 render_peak_kib=306460 took=689ms waited=17169ms
|
||||
2026-11-05T20:35:13.436Z INFO src.services.processing [rid=-] attachment processed attachment_id=a334dab9-882f-443b-aff1-4a5625163e1e job_id=0426fdf9-60b7-4a11-b450-166c34e4414b attempt=1 variants=3 render_peak_kib=306488 took=696ms waited=17333ms
|
||||
2026-11-05T20:35:14.131Z INFO src.services.processing [rid=-] attachment processed attachment_id=c76d5a2a-2d67-4882-88f2-532635007c52 job_id=296fd3f4-87a0-4d4f-a076-5cc5ae10bda8 attempt=1 variants=3 render_peak_kib=306552 took=695ms waited=17693ms
|
||||
2026-11-05T20:35:14.132Z INFO src.services.processing [rid=-] attachment processed attachment_id=bbaa5046-8d98-43c4-ae0e-57de3052d62f job_id=e24b0670-b8b6-4b2f-8967-e446cb50405d attempt=1 variants=3 render_peak_kib=306560 took=694ms waited=17526ms
|
||||
2026-11-05T20:35:14.845Z INFO src.services.processing [rid=-] attachment processed attachment_id=ca4f0b64-8ae9-4ede-b965-edb55e7353b8 job_id=e11023a3-d6f2-4789-8524-7eb1046dd458 attempt=1 variants=3 render_peak_kib=306516 took=712ms waited=17884ms
|
||||
2026-11-05T20:35:14.848Z INFO src.services.processing [rid=-] attachment processed attachment_id=f9832245-8489-49cd-93eb-bde73739ada6 job_id=ad3098ff-67e4-45bc-bc82-f4a3d20fde01 attempt=1 variants=3 render_peak_kib=306576 took=715ms waited=18055ms
|
||||
2026-11-05T20:35:15.552Z INFO src.services.processing [rid=-] attachment processed attachment_id=70651bd6-4413-4f60-bfeb-fdcc475ec76c job_id=d5f37cda-4360-4d8d-8c13-5390bf8b49f8 attempt=1 variants=3 render_peak_kib=306568 took=703ms waited=18263ms
|
||||
2026-11-05T20:35:15.552Z INFO src.services.processing [rid=-] attachment processed attachment_id=36965987-a5a2-473d-b10f-252d04dcbf85 job_id=a6eaf512-a3df-4fb1-8b2e-9a5c15c9a775 attempt=1 variants=3 render_peak_kib=306500 took=705ms waited=18418ms
|
||||
2026-11-05T20:35:16.252Z INFO src.services.processing [rid=-] attachment processed attachment_id=2202d22b-2d4f-4dcd-ad59-d19edd318678 job_id=80814c48-a05c-47d4-af57-9b5d15e88d2c attempt=1 variants=3 render_peak_kib=306572 took=698ms waited=18629ms
|
||||
2026-11-05T20:35:16.253Z INFO src.services.processing [rid=-] attachment processed attachment_id=e7d210d8-272f-48a3-9fee-ec0036cc65ff job_id=6f2e3d8f-6741-465f-b613-1e3f3545753d attempt=1 variants=3 render_peak_kib=306464 took=699ms waited=18800ms
|
||||
2026-11-05T20:35:16.953Z INFO src.services.processing [rid=-] attachment processed attachment_id=50f0810f-0c8b-4752-99f5-e8289462eb2c job_id=a604266d-2f5f-4086-98ec-5e7bac56516d attempt=1 variants=3 render_peak_kib=306568 took=700ms waited=19158ms
|
||||
2026-11-05T20:35:16.965Z INFO src.services.processing [rid=-] attachment processed attachment_id=822666b6-0d7c-4975-aaf0-8835b538aa54 job_id=76a947e3-da49-4ddf-a728-8ca2186f0e77 attempt=1 variants=3 render_peak_kib=306544 took=711ms waited=18994ms
|
||||
2026-11-05T20:35:17.662Z INFO src.services.processing [rid=-] attachment processed attachment_id=2f88e3b7-7177-488b-b59f-c60300c4c286 job_id=a4119024-5697-452f-8fc4-cf8326c85838 attempt=1 variants=3 render_peak_kib=306644 took=707ms waited=19525ms
|
||||
2026-11-05T20:35:17.668Z INFO src.services.processing [rid=-] attachment processed attachment_id=6a63a5c3-8657-45f5-ad1a-a9180c786c05 job_id=12dce7d0-9498-47c5-b7bb-c9b8810bd4a8 attempt=1 variants=3 render_peak_kib=306444 took=702ms waited=19367ms
|
||||
2026-11-05T20:35:18.358Z INFO src.services.processing [rid=-] attachment processed attachment_id=d0b6d584-56fa-4adc-9f8b-3543c61feb82 job_id=47013865-4038-49af-bc44-709331923c13 attempt=1 variants=3 render_peak_kib=306460 took=695ms waited=19892ms
|
||||
2026-11-05T20:35:18.367Z INFO src.services.processing [rid=-] attachment processed attachment_id=99052150-8787-4b6e-8a47-00ef9ce8d7d1 job_id=aeeb6d4d-7546-4fbf-8a33-425a05fbbadc attempt=1 variants=3 render_peak_kib=306580 took=698ms waited=19727ms
|
||||
2026-11-05T20:35:19.046Z INFO src.services.processing [rid=-] attachment processed attachment_id=a43a9632-cfe6-44d4-babb-586ae402b046 job_id=2aa8ec7b-6001-438f-a3c8-dc1ef24145a6 attempt=1 variants=3 render_peak_kib=306508 took=686ms waited=20260ms
|
||||
2026-11-05T20:35:19.055Z INFO src.services.processing [rid=-] attachment processed attachment_id=491709a5-c784-4cb7-9e1b-211d7e817d83 job_id=22a5ffa5-7513-4b71-a4f9-049792287b99 attempt=1 variants=3 render_peak_kib=306536 took=687ms waited=20099ms
|
||||
2026-11-05T20:35:19.766Z INFO src.services.processing [rid=-] attachment processed attachment_id=ce959cc7-d675-4dc9-b4f7-bdf517b58de0 job_id=a2680306-b36e-4672-88ac-e1fa3072eea6 attempt=1 variants=3 render_peak_kib=306540 took=718ms waited=20105ms
|
||||
2026-11-05T20:35:19.769Z INFO src.services.processing [rid=-] attachment processed attachment_id=61bf8b0b-57d1-4d80-bf44-1330397d62aa job_id=721b31c3-cf0a-4970-b410-76edf32eb0dc attempt=1 variants=3 render_peak_kib=306564 took=712ms waited=19942ms
|
||||
2026-11-05T20:35:20.458Z INFO src.services.processing [rid=-] attachment processed attachment_id=b0cd40c7-a682-4acc-ad1d-5e3b892a5c37 job_id=41bd42a0-0ff2-4378-82a6-98d31e4ba4c3 attempt=1 variants=3 render_peak_kib=306444 took=691ms waited=20147ms
|
||||
2026-11-05T20:35:20.468Z INFO src.services.processing [rid=-] attachment processed attachment_id=88ef0076-ee18-43f1-98d1-913a1e577a76 job_id=0a08205d-95a2-4017-91ce-950cb5b4ec1d attempt=1 variants=3 render_peak_kib=306544 took=698ms waited=19981ms
|
||||
2026-11-05T20:35:21.187Z INFO src.services.processing [rid=-] attachment processed attachment_id=1fdc70f4-680f-467a-8cb6-7e6910a77656 job_id=71bdbf50-f467-42c4-a9cc-4da6f8dc5cfa attempt=1 variants=3 render_peak_kib=306468 took=727ms waited=20167ms
|
||||
2026-11-05T20:35:21.188Z INFO src.services.processing [rid=-] attachment processed attachment_id=4a4c15cd-626e-4cca-abd0-d776f651752f job_id=304c5ff7-7022-4bc7-b50e-968637122b2e attempt=1 variants=3 render_peak_kib=306512 took=719ms waited=19998ms
|
||||
2026-11-05T20:35:21.879Z INFO src.services.processing [rid=-] attachment processed attachment_id=95f5e24c-424d-4845-b18d-5380614f9e0c job_id=678af9b9-14b5-437c-9518-ee8dfa246541 attempt=1 variants=3 render_peak_kib=306524 took=690ms waited=20047ms
|
||||
2026-11-05T20:35:21.890Z INFO src.services.processing [rid=-] attachment processed attachment_id=9f9c79be-0f26-4a31-a30b-a85c57acb53c job_id=06f01ec9-1f9c-4659-9101-e94c74077a2b attempt=1 variants=3 render_peak_kib=306516 took=702ms waited=20226ms
|
||||
2026-11-05T20:35:22.580Z INFO src.services.processing [rid=-] attachment processed attachment_id=484fb7eb-e37c-4f5c-87dc-cc4c7e79d1d9 job_id=f1e89b75-448a-415f-b4ee-837a587fae69 attempt=1 variants=3 render_peak_kib=306452 took=700ms waited=20241ms
|
||||
2026-11-05T20:35:22.588Z INFO src.services.processing [rid=-] attachment processed attachment_id=45235b9b-4f65-453e-b014-8ecf75ba8f78 job_id=cfc954f4-62e6-4e78-b4f1-48a9c6f82d7b attempt=1 variants=3 render_peak_kib=306516 took=697ms waited=20080ms
|
||||
2026-11-05T20:35:23.274Z INFO src.services.processing [rid=-] attachment processed attachment_id=383ed374-119e-4130-8aab-e3404b8f744b job_id=461efb7e-f16c-45f6-b4f2-89117cc6249f attempt=1 variants=3 render_peak_kib=306540 took=692ms waited=20093ms
|
||||
2026-11-05T20:35:23.296Z INFO src.services.processing [rid=-] attachment processed attachment_id=193d6a57-1381-4285-991a-9e6cd74e2361 job_id=e58ecf2e-5503-42a1-b76b-9d3294dc03d0 attempt=1 variants=3 render_peak_kib=306524 took=690ms waited=19942ms
|
||||
2026-11-05T20:35:23.982Z INFO src.services.processing [rid=-] attachment processed attachment_id=e660c9ba-0f4c-4be4-84b4-fd3dbb32d464 job_id=53322cdd-f9f5-4a16-85b4-bcee87837bfa attempt=1 variants=3 render_peak_kib=306552 took=707ms waited=20121ms
|
||||
2026-11-05T20:35:24.002Z INFO src.services.processing [rid=-] attachment processed attachment_id=5a651399-e565-4b6c-bae9-3b48ae64ed60 job_id=2ca4e821-ebf4-4a9d-9809-96968040fb97 attempt=1 variants=3 render_peak_kib=306500 took=704ms waited=19969ms
|
||||
2026-11-05T20:35:24.664Z INFO src.services.processing [rid=-] attachment processed attachment_id=6c3413cb-3de8-4bbf-9028-44583efe3607 job_id=adca7945-9df7-4170-923f-1fa5ce22450d attempt=1 variants=3 render_peak_kib=306472 took=680ms waited=20158ms
|
||||
2026-11-05T20:35:24.681Z INFO src.services.processing [rid=-] attachment processed attachment_id=3b6c611b-ba3a-4389-b24d-c05ec58ec8d5 job_id=555115ac-17df-40ee-ab7c-0f75125899ae attempt=1 variants=3 render_peak_kib=306492 took=678ms waited=20005ms
|
||||
2026-11-05T20:35:25.350Z INFO src.services.processing [rid=-] attachment processed attachment_id=b4005a2c-a35e-4985-805d-1b76f8d4d6a5 job_id=970eebbd-5409-426e-9163-d6bb2baae13e attempt=1 variants=3 render_peak_kib=306532 took=685ms waited=20163ms
|
||||
2026-11-05T20:35:25.374Z INFO src.services.processing [rid=-] attachment processed attachment_id=7f5e79f1-ad3f-474d-8a62-be323a79e064 job_id=67e76c55-04da-44c1-a2bd-8d55152545e6 attempt=1 variants=3 render_peak_kib=306544 took=691ms waited=20004ms
|
||||
2026-11-05T20:35:26.040Z INFO src.services.processing [rid=-] attachment processed attachment_id=2cb1fbd5-7536-45ee-a858-59d680bd8eb6 job_id=2f65edcc-898c-4610-a95f-bac99ce503c0 attempt=1 variants=3 render_peak_kib=306484 took=688ms waited=20178ms
|
||||
2026-11-05T20:35:26.062Z INFO src.services.processing [rid=-] attachment processed attachment_id=de1e043e-afc4-4deb-8837-11961cf58684 job_id=d0692292-a594-4823-a464-42ebdedcc439 attempt=1 variants=3 render_peak_kib=306500 took=688ms waited=20028ms
|
||||
2026-11-05T20:35:26.723Z INFO src.services.processing [rid=-] attachment processed attachment_id=ef977c70-447d-4c2d-adc6-36bdfdcbef4c job_id=b8b1764a-9134-4be1-90dc-e930701209b4 attempt=1 variants=3 render_peak_kib=306532 took=681ms waited=20193ms
|
||||
2026-11-05T20:35:26.747Z INFO src.services.processing [rid=-] attachment processed attachment_id=99b24604-2c76-4396-8680-c66ea01d4ea2 job_id=d89f3e9d-b1c9-4473-864d-692b546e2a4a attempt=1 variants=3 render_peak_kib=306504 took=683ms waited=20033ms
|
||||
2026-11-05T20:35:27.421Z INFO src.services.processing [rid=-] attachment processed attachment_id=35af00e6-e1aa-4c57-9439-026d4ce456c2 job_id=8e8f7875-41dd-47ac-962f-d0423629b7d2 attempt=1 variants=3 render_peak_kib=306552 took=697ms waited=20202ms
|
||||
2026-11-05T20:35:27.441Z INFO src.services.processing [rid=-] attachment processed attachment_id=5aca7369-f06c-4b77-aace-f5f9bb1ecd4c job_id=92d42806-a297-4218-884b-2e401882cfc3 attempt=1 variants=3 render_peak_kib=306504 took=693ms waited=20053ms
|
||||
2026-11-05T20:35:28.133Z INFO src.services.processing [rid=-] attachment processed attachment_id=d039d677-57b5-40db-9083-61b4b16dbb11 job_id=f2f570ed-a437-4357-ab10-c580ae5bc513 attempt=1 variants=3 render_peak_kib=306544 took=711ms waited=20056ms
|
||||
2026-11-05T20:35:28.154Z INFO src.services.processing [rid=-] attachment processed attachment_id=7d5f62c6-3e88-4465-965e-e681ba7efd1c job_id=bab584ff-6e06-4796-a18a-6b8c8254cd63 attempt=1 variants=3 render_peak_kib=306520 took=711ms waited=19896ms
|
||||
2026-11-05T20:35:28.827Z INFO src.services.processing [rid=-] attachment processed attachment_id=f53aefd7-36d4-4703-bb8b-96499ec75aff job_id=efc54ca5-c3e1-4091-b8f3-1e1651044f81 attempt=1 variants=3 render_peak_kib=306516 took=692ms waited=20098ms
|
||||
2026-11-05T20:35:28.842Z INFO src.services.processing [rid=-] attachment processed attachment_id=0f47d4bc-4204-4230-a4b5-16ec026d1ea5 job_id=1344e4f9-edb6-4036-a0f0-8ecf6a5ef438 attempt=1 variants=3 render_peak_kib=306472 took=687ms waited=19940ms
|
||||
2026-11-05T20:35:29.541Z INFO src.services.processing [rid=-] attachment processed attachment_id=b7f2cf8e-b1c9-42eb-8fa1-3d7506908280 job_id=f3d77dd8-8ce8-4ed7-89bd-0ab7d9056bcc attempt=1 variants=3 render_peak_kib=306528 took=713ms waited=20118ms
|
||||
2026-11-05T20:35:29.554Z INFO src.services.processing [rid=-] attachment processed attachment_id=fb4b77cd-27b2-4ccf-b6fa-be6ecd0095b2 job_id=d586dd35-938b-4895-87fc-3926df77459e attempt=1 variants=3 render_peak_kib=306568 took=712ms waited=19947ms
|
||||
2026-11-05T20:35:30.237Z INFO src.services.processing [rid=-] attachment processed attachment_id=b287eed0-dd43-4ec6-bc94-15027c4f1b8e job_id=8df4a153-b061-43a2-a956-374ff01b49f0 attempt=1 variants=3 render_peak_kib=306524 took=694ms waited=20156ms
|
||||
2026-11-05T20:35:30.244Z INFO src.services.processing [rid=-] attachment processed attachment_id=fc4800a1-c7d0-4866-8b73-56b02ddaef11 job_id=dc73e43d-31eb-4113-9ec7-67c600b1a20b attempt=1 variants=3 render_peak_kib=306496 took=689ms waited=19993ms
|
||||
2026-11-05T20:35:30.954Z INFO src.services.processing [rid=-] attachment processed attachment_id=497592a3-4517-450b-8fd3-d3b0b342aeef job_id=01c232cd-c4a7-43be-af62-44fcddf68f66 attempt=1 variants=3 render_peak_kib=306464 took=709ms waited=20010ms
|
||||
2026-11-05T20:35:30.954Z INFO src.services.processing [rid=-] attachment processed attachment_id=ca1352de-6a88-43b4-8f98-566c4bf84084 job_id=a4a10535-fe15-4f55-b194-4c6ca09848cc attempt=1 variants=3 render_peak_kib=306516 took=716ms waited=20179ms
|
||||
2026-11-05T20:35:31.653Z INFO src.services.processing [rid=-] attachment processed attachment_id=2e053036-2f2a-4553-bc71-d3c58ce04754 job_id=83b2d4df-5dc1-45eb-9a26-f4e3585004d8 attempt=1 variants=3 render_peak_kib=306500 took=697ms waited=20225ms
|
||||
2026-11-05T20:35:31.658Z INFO src.services.processing [rid=-] attachment processed attachment_id=1c34eb1e-0c8c-477a-b2b6-dcd55e32b346 job_id=d1ecce98-6afe-420b-9022-8146635ebd93 attempt=1 variants=3 render_peak_kib=306512 took=702ms waited=20046ms
|
||||
2026-11-05T20:35:32.365Z INFO src.services.processing [rid=-] attachment processed attachment_id=ce315f4f-0211-4eff-a9e6-f55f66597335 job_id=5ff72a9e-1bdd-4b73-bd64-aac2dde26f7d attempt=1 variants=3 render_peak_kib=306572 took=706ms waited=20081ms
|
||||
2026-11-05T20:35:32.365Z INFO src.services.processing [rid=-] attachment processed attachment_id=b98ca6ca-7966-4136-bdbf-b426313b1e21 job_id=b2b6a45a-ef46-43f0-9f9a-8d0f7b439336 attempt=1 variants=3 render_peak_kib=306504 took=711ms waited=20247ms
|
||||
2026-11-05T20:35:33.073Z INFO src.services.processing [rid=-] attachment processed attachment_id=280fcf9a-7407-4014-97b9-eca05257db86 job_id=5fa070f7-04f0-42b5-88bd-4dccd4bf6079 attempt=1 variants=3 render_peak_kib=306500 took=706ms waited=19944ms
|
||||
2026-11-05T20:35:33.075Z INFO src.services.processing [rid=-] attachment processed attachment_id=1a439d67-c654-47a8-8589-f8737ed42f15 job_id=4cb1b37b-8c2b-4bfb-b419-1721223e01fa attempt=1 variants=3 render_peak_kib=306552 took=708ms waited=20118ms
|
||||
2026-11-05T20:35:33.783Z INFO src.services.processing [rid=-] attachment processed attachment_id=5933ad00-47b5-4c7b-aa0b-d4431b271ab0 job_id=e5671efc-618d-4f63-a262-78be9601738d attempt=1 variants=3 render_peak_kib=306516 took=709ms waited=20153ms
|
||||
2026-11-05T20:35:33.784Z INFO src.services.processing [rid=-] attachment processed attachment_id=f25d824d-c017-471d-8dbc-60a429a59736 job_id=64ff4279-ee30-4acf-9e93-6492431e03dd attempt=1 variants=3 render_peak_kib=306460 took=708ms waited=19973ms
|
||||
2026-11-05T20:35:34.468Z INFO src.services.processing [rid=-] attachment processed attachment_id=60ead31c-dc83-4447-b284-d63f9f9bf524 job_id=65df926f-5b6a-461c-b932-4f99b2cf646a attempt=1 variants=3 render_peak_kib=306544 took=683ms waited=20013ms
|
||||
2026-11-05T20:35:34.469Z INFO src.services.processing [rid=-] attachment processed attachment_id=dad24888-7633-4bd0-a2f0-dd1d9d6ae309 job_id=b741e881-2065-4f29-8134-a6c49f303c5a attempt=1 variants=3 render_peak_kib=306476 took=684ms waited=20188ms
|
||||
2026-11-05T20:35:35.166Z INFO src.services.processing [rid=-] attachment processed attachment_id=c2c23e97-6442-4b0b-a077-dd5ade365ca0 job_id=95a2ae68-65a0-4823-9e72-6cec0315b873 attempt=1 variants=3 render_peak_kib=306504 took=695ms waited=20026ms
|
||||
2026-11-05T20:35:35.177Z INFO src.services.processing [rid=-] attachment processed attachment_id=91011ec0-8031-4e9c-91a9-33df28fd3719 job_id=38704a4a-5af1-444b-be82-50ad988c435e attempt=1 variants=3 render_peak_kib=306508 took=708ms waited=20198ms
|
||||
2026-11-05T20:35:35.869Z INFO src.services.processing [rid=-] attachment processed attachment_id=4ea86425-461c-4f27-8a8c-af16de25dc38 job_id=a98d68d0-057e-4aa4-835f-8e9347fa7a47 attempt=1 variants=3 render_peak_kib=306488 took=702ms waited=20224ms
|
||||
2026-11-05T20:35:35.884Z INFO src.services.processing [rid=-] attachment processed attachment_id=f706425b-a5d2-467a-89d7-92372531f6a1 job_id=504d55d8-6e67-4f89-90b5-5cf7a1eef669 attempt=1 variants=3 render_peak_kib=306560 took=705ms waited=20064ms
|
||||
2026-11-05T20:35:36.560Z INFO src.services.processing [rid=-] attachment processed attachment_id=c9e9bbba-fed6-4225-8ca8-fcf6e0e11c71 job_id=ba950bd1-6385-455a-aa4e-6ac67a3d7da0 attempt=1 variants=3 render_peak_kib=306476 took=689ms waited=20081ms
|
||||
2026-11-05T20:35:36.576Z INFO src.services.processing [rid=-] attachment processed attachment_id=033fa897-3e89-4a47-b264-ae39fa98cb85 job_id=734e6345-2c1c-4975-9bb2-dbe397cc3a29 attempt=1 variants=3 render_peak_kib=306480 took=691ms waited=19921ms
|
||||
2026-11-05T20:35:37.256Z INFO src.services.processing [rid=-] attachment processed attachment_id=64ff448b-c98b-479a-b333-67eb5547d27c job_id=c5e22f17-bec9-47cd-8374-4eb116d2de58 attempt=1 variants=3 render_peak_kib=306484 took=694ms waited=20102ms
|
||||
2026-11-05T20:35:37.270Z INFO src.services.processing [rid=-] attachment processed attachment_id=2ea6f033-4f60-4cb9-ad58-112d8ab4383a job_id=a933a743-375c-4276-af8e-6c979c1ac377 attempt=1 variants=3 render_peak_kib=306512 took=693ms waited=19934ms
|
||||
2026-11-05T20:35:37.955Z INFO src.services.processing [rid=-] attachment processed attachment_id=f788fe31-31cc-4095-9575-77a8a2057fc6 job_id=847bb8df-91a4-4a62-b02b-82802a7954ba attempt=1 variants=3 render_peak_kib=306516 took=698ms waited=20124ms
|
||||
2026-11-05T20:35:37.965Z INFO src.services.processing [rid=-] attachment processed attachment_id=d1153ecb-244c-4904-bc37-8b21c00e3826 job_id=a4ae10a6-94df-40a9-afff-65494d5acd18 attempt=1 variants=3 render_peak_kib=306548 took=694ms waited=19956ms
|
||||
2026-11-05T20:35:38.650Z INFO src.services.processing [rid=-] attachment processed attachment_id=08c9d35b-7727-4c83-805f-e769f1255d82 job_id=6935f615-1e33-4f7f-bff2-3e3d9434f06e attempt=1 variants=3 render_peak_kib=306464 took=694ms waited=20150ms
|
||||
2026-11-05T20:35:38.652Z INFO src.services.processing [rid=-] attachment processed attachment_id=1716e0b2-f677-47d3-b43a-1d71ee34984c job_id=5d683736-1489-4131-a33a-555d3b4f1e69 attempt=1 variants=3 render_peak_kib=306580 took=685ms waited=19985ms
|
||||
2026-11-05T20:35:39.365Z INFO src.services.processing [rid=-] attachment processed attachment_id=518a7653-02af-4155-b392-511c576b0fba job_id=45b67b72-1d2a-4dc3-b8f7-9c5b4b39c9ae attempt=1 variants=3 render_peak_kib=306492 took=712ms waited=19999ms
|
||||
2026-11-05T20:35:39.366Z INFO src.services.processing [rid=-] attachment processed attachment_id=aa1cb19a-ccf8-45c5-9d07-224faa082342 job_id=f7ddee9b-9ed2-42ce-a316-b51227d6e272 attempt=1 variants=3 render_peak_kib=306524 took=714ms waited=20170ms
|
||||
2026-11-05T20:35:40.036Z INFO src.services.processing [rid=-] attachment processed attachment_id=a35baac4-a262-46e1-919e-c08fea7aed95 job_id=f5ba8d74-a8c8-4669-b9e8-b32beabda8a4 attempt=1 variants=3 render_peak_kib=306516 took=669ms waited=20213ms
|
||||
2026-11-05T20:35:42.274Z INFO src.services.processing [rid=-] attachment processed attachment_id=605cb49d-e96a-4987-b14d-e8dab87a67e9 job_id=e662faab-5ad5-40eb-a9bb-9d219b59e36b attempt=1 variants=3 render_peak_kib=306528 took=680ms waited=448ms
|
||||
2026-11-05T20:35:42.762Z INFO src.services.processing [rid=-] attachment processed attachment_id=4a518ac2-a5e0-476d-b1fe-2db037049ebf job_id=702b3909-5324-4817-9129-f711b1d7e089 attempt=1 variants=3 render_peak_kib=306520 took=660ms waited=396ms
|
||||
2026-11-05T20:35:43.275Z INFO src.services.processing [rid=-] attachment processed attachment_id=8a79b646-0935-42c7-b798-9d45c2adb85f job_id=28c130a6-b465-4c80-86e3-f5e7821e7876 attempt=1 variants=3 render_peak_kib=306484 took=663ms waited=373ms
|
||||
2026-11-05T20:35:43.772Z INFO src.services.processing [rid=-] attachment processed attachment_id=7ac8bfd8-685b-44de-a82d-0cd620e30d83 job_id=da85627a-8f46-419e-be17-331ef6e81f57 attempt=1 variants=3 render_peak_kib=306520 took=655ms waited=317ms
|
||||
2026-11-05T20:35:44.287Z INFO src.services.processing [rid=-] attachment processed attachment_id=d5182065-3f4b-4e97-9493-fc3997469324 job_id=200ede28-ee7c-449a-b999-138fb2f1ebc6 attempt=1 variants=3 render_peak_kib=306516 took=663ms waited=263ms
|
||||
2026-11-05T20:35:44.814Z INFO src.services.processing [rid=-] attachment processed attachment_id=fb1240e4-0645-4258-84b0-7ab46d33e7e4 job_id=b8c6a398-ea85-449e-bae5-db64713635a5 attempt=1 variants=3 render_peak_kib=306500 took=683ms waited=207ms
|
||||
2026-11-05T20:35:45.300Z INFO src.services.processing [rid=-] attachment processed attachment_id=caa8ab17-03a0-49a3-b97b-f60d91efea98 job_id=1460e967-0f6f-4315-abb5-e860e3a9ef0a attempt=1 variants=3 render_peak_kib=306520 took=660ms waited=153ms
|
||||
2026-11-05T20:35:45.805Z INFO src.services.processing [rid=-] attachment processed attachment_id=50ccb7d4-af96-4755-8835-101367323897 job_id=a8f1ee4f-6dea-475f-9810-99d0fe3f598e attempt=1 variants=3 render_peak_kib=306468 took=654ms waited=106ms
|
||||
2026-11-05T20:35:46.313Z INFO src.services.processing [rid=-] attachment processed attachment_id=06a324ee-90c6-47b0-9d94-6b706c3bba38 job_id=18ebd537-bc27-49e4-8e75-d6c716737360 attempt=1 variants=3 render_peak_kib=306500 took=652ms waited=53ms
|
||||
2026-11-05T20:35:47.332Z INFO src.services.processing [rid=-] attachment processed attachment_id=06fdc278-6e9d-4346-b585-231a16d5fe01 job_id=60e60e73-0d88-48fe-800a-e03cd864f238 attempt=1 variants=3 render_peak_kib=306504 took=661ms waited=503ms
|
||||
2026-11-05T20:35:47.840Z INFO src.services.processing [rid=-] attachment processed attachment_id=8450b4da-f5ce-4ef3-ad6c-e26e4f04eb18 job_id=5b812c0b-0484-43d8-be14-b2da9b24a125 attempt=1 variants=3 render_peak_kib=306484 took=663ms waited=418ms
|
||||
2026-11-05T20:35:48.344Z INFO src.services.processing [rid=-] attachment processed attachment_id=e29f4610-27d1-4d4d-82ce-a7a662c3b329 job_id=5cb9a95d-a6c9-434c-8180-51ebf09b125b attempt=1 variants=3 render_peak_kib=306456 took=660ms waited=391ms
|
||||
2026-11-05T20:35:48.852Z INFO src.services.processing [rid=-] attachment processed attachment_id=29919874-ebe6-4d7a-9a12-a1e9c2f85460 job_id=ba15c503-5f46-44eb-be45-d47d2fee5ec2 attempt=1 variants=3 render_peak_kib=306552 took=663ms waited=337ms
|
||||
2026-11-05T20:35:49.355Z INFO src.services.processing [rid=-] attachment processed attachment_id=6c534d76-c612-47be-ab5b-727ccc5d4c11 job_id=cdaa85fa-65ab-4289-9bf9-14fff82c5500 attempt=1 variants=3 render_peak_kib=306512 took=656ms waited=282ms
|
||||
2026-11-05T20:35:49.871Z INFO src.services.processing [rid=-] attachment processed attachment_id=6bf86c2b-8e16-4429-bbfa-54813a486189 job_id=abdc5509-3275-4622-bcae-82e5bf86ffdc attempt=1 variants=3 render_peak_kib=306468 took=665ms waited=232ms
|
||||
2026-11-05T20:35:50.388Z INFO src.services.processing [rid=-] attachment processed attachment_id=8c0d64a1-2bbe-4a2c-a12e-3746828b2c10 job_id=df3ff14f-1e22-4c2d-89fd-bd4dcaa53d55 attempt=1 variants=3 render_peak_kib=306492 took=677ms waited=175ms
|
||||
2026-11-05T20:35:50.880Z INFO src.services.processing [rid=-] attachment processed attachment_id=bfa1b66a-d362-4176-a0b0-88bb2c214db5 job_id=1e64ba16-e094-444b-8138-7b95c936f413 attempt=1 variants=3 render_peak_kib=306532 took=663ms waited=112ms
|
||||
2026-11-05T20:35:51.390Z INFO src.services.processing [rid=-] attachment processed attachment_id=b1d6a29c-ed38-49b6-a373-2c323490afb8 job_id=506467b0-3955-4308-9b0b-a41439be2296 attempt=1 variants=3 render_peak_kib=306544 took=662ms waited=68ms
|
||||
2026-11-05T20:35:51.892Z INFO src.services.processing [rid=-] attachment processed attachment_id=ee498ef4-58bc-4efe-9e23-207ed6aeed15 job_id=7da72c98-3f2c-49d9-913f-c43ee504e8a1 attempt=1 variants=3 render_peak_kib=306528 took=659ms waited=7ms
|
||||
2026-11-05T20:35:52.907Z INFO src.services.processing [rid=-] attachment processed attachment_id=90fa21e9-ef85-4103-a678-823696c7cbb0 job_id=5c619e3f-9c18-494a-86f7-fd0e7b85920f attempt=1 variants=3 render_peak_kib=306532 took=661ms waited=462ms
|
||||
2026-11-05T20:35:53.413Z INFO src.services.processing [rid=-] attachment processed attachment_id=012223b0-7c61-488f-b0ac-a8b470756abf job_id=fc4e1f8b-ea79-4de8-b8a8-8874d8239751 attempt=1 variants=3 render_peak_kib=306516 took=658ms waited=403ms
|
||||
2026-11-05T20:35:53.922Z INFO src.services.processing [rid=-] attachment processed attachment_id=84b699f1-16f3-4359-adbc-10fab0d5154c job_id=adfa1aec-aa0c-4a4c-ba45-113dc36aa607 attempt=1 variants=3 render_peak_kib=306448 took=661ms waited=352ms
|
||||
2026-11-05T20:35:54.432Z INFO src.services.processing [rid=-] attachment processed attachment_id=5d58bc86-8f45-4788-a0f6-ee6beb24bf5f job_id=13e13bea-3557-4bc2-802f-fab9df835645 attempt=1 variants=3 render_peak_kib=306556 took=661ms waited=300ms
|
||||
2026-11-05T20:35:54.948Z INFO src.services.processing [rid=-] attachment processed attachment_id=98c8ca16-a97f-4fdf-9f09-675d6da5ddd2 job_id=ec91013d-2469-4fab-bf05-1f99449cc204 attempt=1 variants=3 render_peak_kib=306524 took=668ms waited=250ms
|
||||
2026-11-05T20:35:55.426Z INFO src.services.processing [rid=-] attachment processed attachment_id=8ab8b585-5643-4c00-9529-a8a402bb63bf job_id=33e36f5a-0aa6-48e1-996a-981dd4ff579d attempt=1 variants=3 render_peak_kib=306536 took=642ms waited=195ms
|
||||
2026-11-05T20:35:55.960Z INFO src.services.processing [rid=-] attachment processed attachment_id=c8ef65da-88d7-4c5d-b3b7-d02f78f9be72 job_id=ef407b77-9835-4659-9199-d53e72187ab2 attempt=1 variants=3 render_peak_kib=306468 took=670ms waited=134ms
|
||||
2026-11-05T20:35:56.460Z INFO src.services.processing [rid=-] attachment processed attachment_id=81a1d4bf-4f6c-4696-8c7b-acf47a40bd92 job_id=73b1fff4-1e60-492b-a794-d070a1a337e6 attempt=1 variants=3 render_peak_kib=306460 took=664ms waited=82ms
|
||||
2026-11-05T20:35:56.952Z INFO src.services.processing [rid=-] attachment processed attachment_id=1b133395-e55e-40ee-9c4b-5f21a617f524 job_id=269df868-7a77-4bb5-8d8e-d27df41c3255 attempt=1 variants=3 render_peak_kib=306476 took=651ms waited=27ms
|
||||
2026-11-05T20:35:57.996Z INFO src.services.processing [rid=-] attachment processed attachment_id=36e4d217-8cf5-490c-8808-f9821bda08ea job_id=67dec927-70d3-4c08-92e6-0d137200b747 attempt=1 variants=3 render_peak_kib=306548 took=681ms waited=479ms
|
||||
2026-11-05T20:35:58.485Z INFO src.services.processing [rid=-] attachment processed attachment_id=56d2b494-e16e-47e5-8862-e3a695cff913 job_id=1f934604-e418-4544-a44d-e945a8ea0a66 attempt=1 variants=3 render_peak_kib=306516 took=659ms waited=396ms
|
||||
2026-11-05T20:35:58.992Z INFO src.services.processing [rid=-] attachment processed attachment_id=2d926ceb-67a6-441e-854f-2a312e12ea2c job_id=aa7a9cd2-220d-4cf8-bed2-4ce553433964 attempt=1 variants=3 render_peak_kib=306540 took=655ms waited=371ms
|
||||
2026-11-05T20:35:59.502Z INFO src.services.processing [rid=-] attachment processed attachment_id=e04647bb-d20f-4576-b1d1-ddfe6b90452a job_id=a9f10306-c53e-4fe5-a79c-aa6c31384f28 attempt=1 variants=3 render_peak_kib=306568 took=656ms waited=323ms
|
||||
2026-11-05T20:36:00.015Z INFO src.services.processing [rid=-] attachment processed attachment_id=bc1d6d3c-b4a9-45b6-83ae-5031bb5682c7 job_id=ecf7132e-4f31-4ae2-9236-3fbeffa229fa attempt=1 variants=3 render_peak_kib=306548 took=663ms waited=269ms
|
||||
2026-11-05T20:36:00.526Z INFO src.services.processing [rid=-] attachment processed attachment_id=57001115-2125-455d-8bd2-3280fdd8d6a5 job_id=7a1b988c-02c1-4293-84a3-45aed750f48b attempt=1 variants=3 render_peak_kib=306456 took=668ms waited=211ms
|
||||
2026-11-05T20:36:01.031Z INFO src.services.processing [rid=-] attachment processed attachment_id=3f0d9517-dd77-4279-952d-9865574424e1 job_id=3e40271d-3a85-4019-bfa6-3a8a0fb33022 attempt=1 variants=3 render_peak_kib=306540 took=666ms waited=158ms
|
||||
2026-11-05T20:36:01.533Z INFO src.services.processing [rid=-] attachment processed attachment_id=cdaf1912-b85c-4b6e-a270-687a78c05191 job_id=8bca512e-6c14-4965-a1f2-05d05e282706 attempt=1 variants=3 render_peak_kib=306460 took=663ms waited=100ms
|
||||
@@ -0,0 +1,50 @@
|
||||
q,1793911122078.6,1793911122088.1,1793911122116.5,202,,7c023664-e29b-4455-b199-787002a68578,7bbaa8668c87,
|
||||
q,1793911122118.6,1793911122120.9,1793911122135.7,202,,17eabde3-4808-4dde-a214-bfbb5b806926,39e797bfa6f6,
|
||||
q,1793911122158.6,1793911122160.8,1793911122173.2,202,,312ce6d3-3151-48b9-adb1-20b5817f882b,7bbaa8668c87,
|
||||
q,1793911122198.6,1793911122201.9,1793911122215.0,202,,568334ae-ce0f-4094-b44a-c32b7b3e93d6,39e797bfa6f6,
|
||||
q,1793911122238.6,1793911122241.1,1793911122254.6,202,,8b10d802-4b4a-4c05-8723-c74dda9bb8ec,7bbaa8668c87,
|
||||
q,1793911122278.6,1793911122280.2,1793911122292.7,202,,c60e09b1-2b8c-4cdf-8451-73f29773cc54,39e797bfa6f6,
|
||||
q,1793911122318.6,1793911122327.7,1793911122341.3,202,,13f60ef1-0839-4b40-8ee4-6bb9629bf0e9,7bbaa8668c87,
|
||||
q,1793911122358.6,1793911122362.0,1793911122374.3,202,,0a43e4ad-a656-497d-907d-49587d16f784,39e797bfa6f6,
|
||||
q,1793911122398.6,1793911122401.4,1793911122411.6,202,,49b744df-eb1f-4f2a-9828-a6a424fe3bab,7bbaa8668c87,
|
||||
q,1793911122438.6,1793911122440.6,1793911122450.7,202,,e5335dac-c345-473b-b088-f166a72f7972,39e797bfa6f6,
|
||||
q,1793911122478.6,1793911122479.8,1793911122491.4,202,,a9864799-3d34-4909-ab40-79f12896668b,7bbaa8668c87,
|
||||
q,1793911122518.6,1793911122521.5,1793911122531.8,202,,bdcdc60d-c28d-41da-9d1a-2e66c5903be3,39e797bfa6f6,
|
||||
q,1793911122558.6,1793911122559.9,1793911122569.4,202,,9acdf189-d812-4412-90a6-d0e551f4131b,7bbaa8668c87,
|
||||
q,1793911122598.6,1793911122601.2,1793911122610.9,202,,448f9483-ea7b-40ea-a25b-08f30943ea30,39e797bfa6f6,
|
||||
q,1793911122638.6,1793911122644.3,1793911122655.1,202,,f3626baf-c306-4ae3-8584-2114a92a408c,7bbaa8668c87,
|
||||
q,1793911122678.6,1793911122679.6,1793911122690.1,202,,e7f38ff4-4b51-44e6-b1fc-1c22896dd248,39e797bfa6f6,
|
||||
q,1793911122718.6,1793911122722.6,1793911122734.3,202,,17b88801-d3c0-44c8-b552-ac7c626ecdf3,7bbaa8668c87,
|
||||
q,1793911122758.6,1793911122764.3,1793911122778.3,202,,49e05501-b5c0-4473-9360-f79c1316bbc2,39e797bfa6f6,
|
||||
q,1793911122798.6,1793911122803.4,1793911122815.8,202,,335f62a8-1121-4246-98ae-82a2da58bbab,7bbaa8668c87,
|
||||
q,1793911122838.6,1793911122841.0,1793911122854.1,202,,707428bf-ada6-423d-b743-89e0c62afbda,39e797bfa6f6,
|
||||
q,1793911122878.6,1793911122879.5,1793911122889.0,202,,a8b757f3-d176-4644-a056-a1a87974c926,7bbaa8668c87,
|
||||
q,1793911122918.6,1793911122921.3,1793911122931.4,202,,3d60ef8b-a1b1-4ff8-81bd-d25d63ee12c6,39e797bfa6f6,
|
||||
q,1793911122958.6,1793911122963.2,1793911122974.0,202,,f9e5368e-6de7-4493-8b31-c30cba4c7cfb,7bbaa8668c87,
|
||||
q,1793911122998.6,1793911123001.2,1793911123012.3,202,,778c4e2d-726e-4119-aab2-959087b05418,39e797bfa6f6,
|
||||
q,1793911123038.6,1793911123039.9,1793911123051.4,202,,c85d9e1c-6d79-4b74-8163-711155cca1f9,7bbaa8668c87,
|
||||
q,1793911123078.6,1793911123083.6,1793911123095.5,202,,df4f32b3-bb83-4517-a358-8baebd12abc6,39e797bfa6f6,
|
||||
q,1793911123118.6,1793911123120.2,1793911123133.3,202,,3351a655-323f-4e8d-9ed0-03234055ee01,7bbaa8668c87,
|
||||
q,1793911123158.6,1793911123162.3,1793911123175.1,202,,65bb22a7-3bf8-40dc-aa20-c66249621e18,39e797bfa6f6,
|
||||
q,1793911123198.6,1793911123203.3,1793911123216.2,202,,334fce4d-fb2b-4946-bbb6-474b4f6b78a4,7bbaa8668c87,
|
||||
q,1793911123238.6,1793911123240.8,1793911123253.6,202,,45f8ebdb-ee2d-4a5d-850d-935c5c041dcf,39e797bfa6f6,
|
||||
q,1793911123278.6,1793911123280.0,1793911123291.4,202,,e4cf5f84-eccd-47c2-9476-3b414dcc3d4b,7bbaa8668c87,
|
||||
q,1793911123318.6,1793911123326.8,1793911123339.4,202,,a74ab383-860b-48be-853d-81c5dd019436,39e797bfa6f6,
|
||||
q,1793911123358.6,1793911123369.1,1793911123380.7,202,,6885a84e-6905-4a0f-8c39-d58cb4e37591,7bbaa8668c87,
|
||||
q,1793911123398.6,1793911123400.3,1793911123410.7,202,,47c82382-f841-4546-b2f1-b051f52e046f,39e797bfa6f6,
|
||||
q,1793911123438.6,1793911123444.9,1793911123455.5,202,,1f0bd67b-39e3-47de-90f7-449dc629d5d2,7bbaa8668c87,
|
||||
q,1793911123478.6,1793911123480.9,1793911123493.4,202,,e1f2b391-b9b7-41bf-acf6-f764671ad8bb,39e797bfa6f6,
|
||||
q,1793911123518.6,1793911123520.8,1793911123531.5,202,,3d32239f-6409-4d21-bb6b-445a33f37b67,7bbaa8668c87,
|
||||
q,1793911123558.6,1793911123561.0,1793911123571.0,202,,11ff31e6-980b-4310-b608-e060ba2d2db5,39e797bfa6f6,
|
||||
q,1793911123598.6,1793911123606.1,1793911123616.2,202,,4793b7e1-1f4c-48ff-a220-36c79920f3b3,7bbaa8668c87,
|
||||
q,1793911123638.6,1793911123641.7,1793911123651.6,202,,fdf038a0-e506-4734-a4b2-59a0848904fb,39e797bfa6f6,
|
||||
q,1793911123678.6,1793911123683.3,1793911123693.6,202,,c18530c3-ae2a-4616-8340-5c3ced63eba8,7bbaa8668c87,
|
||||
q,1793911123718.6,1793911123721.2,1793911123732.9,202,,2b8f8b76-5754-47c1-8211-cf93f6a4bd49,39e797bfa6f6,
|
||||
q,1793911123758.6,1793911123761.8,1793911123774.3,202,,9ec2d5d5-9ba6-4d9f-954e-278d3a2bb477,7bbaa8668c87,
|
||||
q,1793911123798.6,1793911123801.3,1793911123815.0,202,,16a7ca97-1234-4c23-8153-c3bef91f626f,39e797bfa6f6,
|
||||
q,1793911123838.6,1793911123840.6,1793911123851.4,202,,7b40dd36-9deb-4a27-b045-e367fa34378a,7bbaa8668c87,
|
||||
q,1793911123878.6,1793911123880.5,1793911123888.9,202,,cf4fff2a-a402-4c2f-806b-5134f6d5b2e4,39e797bfa6f6,
|
||||
q,1793911123918.6,1793911123919.6,1793911123928.0,202,,97e935e4-ced9-42c6-b138-c7bb978b357b,7bbaa8668c87,
|
||||
q,1793911123958.6,1793911123959.7,1793911123967.8,202,,2d263a78-262c-41b2-adbb-92afb269ca87,39e797bfa6f6,
|
||||
q,1793911123998.6,1793911123999.9,1793911124008.0,202,,20333f44-117b-4b8f-ad33-f4dded0f2ae1,7bbaa8668c87,
|
||||
q,1793911124038.6,1793911124039.5,1793911124047.7,202,,8f676bbd-4a05-4a29-a242-9a30a611704b,39e797bfa6f6,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 25.0,
|
||||
"duration_s": 3.0,
|
||||
"start": 1793911122.079,
|
||||
"end": 1793911124.079,
|
||||
"finished": 1793911124.048,
|
||||
"scheduled": 50,
|
||||
"sent": 50,
|
||||
"accepted": 50,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 50
|
||||
},
|
||||
"upload_p50_ms": 11.38,
|
||||
"upload_p95_ms": 14.02,
|
||||
"upload_max_ms": 28.38,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.046,
|
||||
"log": "/tmp/c2p32-s-r1-drain.log"
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
q,1793911171541.4,1793911171542.1,1793911171559.5,202,,acd731d7-372b-4ece-856d-01d2c5795328,7bbaa8668c87,
|
||||
q,1793911171941.4,1793911171941.9,1793911171950.0,202,,2980d8e3-b2e4-466d-a156-2b6f0058dd39,39e797bfa6f6,
|
||||
q,1793911172341.4,1793911172344.9,1793911172353.3,202,,e64cc430-55d0-45bd-9535-038dabfd1f72,7bbaa8668c87,
|
||||
q,1793911172741.4,1793911172742.9,1793911172752.0,202,,e4e181ce-faac-441f-b3f2-2c6caac5c97f,39e797bfa6f6,
|
||||
q,1793911173141.4,1793911173145.5,1793911173175.2,202,,500702ad-fdee-4610-bdc2-da14a3c71a0c,7bbaa8668c87,
|
||||
q,1793911173541.4,1793911173545.2,1793911173553.9,202,,f1815c2d-b2aa-47b3-af59-e594ab7cf469,39e797bfa6f6,
|
||||
q,1793911173941.4,1793911173944.3,1793911173951.3,202,,a561657f-ec5d-4c70-a0c3-98d2431126bc,7bbaa8668c87,
|
||||
q,1793911174341.4,1793911174343.3,1793911174352.3,202,,54562b58-579a-40e6-be89-b8ad58ac512b,39e797bfa6f6,
|
||||
q,1793911174741.4,1793911174743.6,1793911174770.5,202,,2fd7262c-9be0-4e8d-ab95-a246c5883380,7bbaa8668c87,
|
||||
q,1793911175141.4,1793911175143.8,1793911175161.8,202,,f0462029-5df9-4f80-8822-45bdc8fabba8,39e797bfa6f6,
|
||||
q,1793911175541.4,1793911175545.9,1793911175554.2,202,,87017dff-b63c-44fe-8256-eaef503f5b5d,7bbaa8668c87,
|
||||
q,1793911175941.4,1793911175943.5,1793911175952.0,202,,7f22c1d5-c0ee-4ce2-a4f4-bed928031918,39e797bfa6f6,
|
||||
q,1793911176341.4,1793911176343.3,1793911176367.0,202,,b3776a18-8174-4961-9b6f-bdf0cd6a60dd,7bbaa8668c87,
|
||||
q,1793911176741.4,1793911176744.6,1793911176769.7,202,,17d0e30c-76d1-4647-9e23-e73eb124ffb8,39e797bfa6f6,
|
||||
q,1793911177141.4,1793911177147.3,1793911177160.2,202,,2def0998-6a6c-4128-96e1-29e7e4296e5d,7bbaa8668c87,
|
||||
q,1793911177541.4,1793911177543.1,1793911177551.8,202,,3f531825-967f-4d78-ae2f-e9cf1596c779,39e797bfa6f6,
|
||||
q,1793911177941.4,1793911177943.0,1793911177953.0,202,,effe787c-cbae-461c-b3c7-cfae903272e2,7bbaa8668c87,
|
||||
q,1793911178341.4,1793911178344.4,1793911178352.5,202,,2ad933d3-7083-43ff-b9d7-85948f13efcf,39e797bfa6f6,
|
||||
q,1793911178741.4,1793911178747.1,1793911178770.2,202,,f930282c-71d4-4961-b81f-1488bbf7d5f4,7bbaa8668c87,
|
||||
q,1793911179141.4,1793911179145.2,1793911179154.0,202,,9661f472-9909-4e65-aa83-5d67a922bcd7,39e797bfa6f6,
|
||||
q,1793911179541.4,1793911179542.9,1793911179551.7,202,,f5cf83f1-9c3b-4af2-aaaa-9309fc27eaf4,7bbaa8668c87,
|
||||
q,1793911179941.4,1793911179944.3,1793911179969.3,202,,28ac618e-52ca-4276-813d-db7ccb10ae41,39e797bfa6f6,
|
||||
q,1793911180341.4,1793911180351.8,1793911180377.1,202,,de6fabe8-79e7-416e-bcca-c1bcea69e7b6,7bbaa8668c87,
|
||||
q,1793911180741.4,1793911180742.5,1793911180752.9,202,,3134e2ac-1825-4ca8-9461-19622f77f4fd,39e797bfa6f6,
|
||||
q,1793911181141.4,1793911181142.9,1793911181151.6,202,,ce36ed2f-1f3b-4a93-bfa9-5773299d18de,7bbaa8668c87,
|
||||
q,1793911181541.4,1793911181543.8,1793911181554.6,202,,3f29de86-c96a-48e1-938a-7d1a6e38f61a,39e797bfa6f6,
|
||||
q,1793911181941.4,1793911181943.2,1793911181968.7,202,,58fe21bd-3b9d-419d-96cb-97a355d85ee1,7bbaa8668c87,
|
||||
q,1793911182341.4,1793911182353.0,1793911182361.2,202,,9d680ddb-4f75-4cbb-a6ef-80fb2dd425d1,39e797bfa6f6,
|
||||
q,1793911182741.4,1793911182743.1,1793911182752.5,202,,69240322-27d6-45c3-a07b-cd435ed955d4,7bbaa8668c87,
|
||||
q,1793911183141.4,1793911183143.9,1793911183152.3,202,,62639ddb-f32d-4cb3-85c8-ce182dddfe7d,39e797bfa6f6,
|
||||
q,1793911183541.4,1793911183542.9,1793911183551.8,202,,1300d6f8-1e8a-4d5f-b4d8-d6bf5bdb227e,7bbaa8668c87,
|
||||
q,1793911183941.4,1793911183943.6,1793911183969.5,202,,7b0e7707-5566-444d-b2a5-780ca34f42d6,39e797bfa6f6,
|
||||
q,1793911184341.4,1793911184342.7,1793911184354.5,202,,5ccdba7b-e508-40c5-9163-03916c0696d6,7bbaa8668c87,
|
||||
q,1793911184741.4,1793911184744.0,1793911184751.8,202,,81c487a6-21cd-4cca-9745-52ca7ed6a26d,39e797bfa6f6,
|
||||
q,1793911185141.4,1793911185142.9,1793911185152.1,202,,1a806d04-42a1-45b9-9243-9b5b62fcfbc9,7bbaa8668c87,
|
||||
q,1793911185541.4,1793911185543.1,1793911185557.2,202,,7738c7f2-4720-4ea0-a916-e16fdef73bfc,39e797bfa6f6,
|
||||
q,1793911185941.4,1793911185945.1,1793911185968.2,202,,e8a05338-01c6-434a-aaec-d4bbc273393e,7bbaa8668c87,
|
||||
q,1793911186341.4,1793911186353.0,1793911186362.9,202,,4fdb8013-0f45-40cf-82ec-652e82713be3,39e797bfa6f6,
|
||||
q,1793911186741.4,1793911186752.9,1793911186761.2,202,,8d2e8731-3152-42e2-b5c1-0cb68ea3b6fb,7bbaa8668c87,
|
||||
q,1793911187141.4,1793911187143.9,1793911187164.8,202,,81a109d0-38e6-4d50-8e26-88cb39753cad,39e797bfa6f6,
|
||||
q,1793911187541.4,1793911187543.1,1793911187556.3,202,,af314e16-1691-4cf3-ad49-fa7f0ade00ad,7bbaa8668c87,
|
||||
q,1793911187941.4,1793911187945.0,1793911187958.8,202,,2be32e54-2694-4998-8e02-ba745a74ec71,39e797bfa6f6,
|
||||
q,1793911188341.4,1793911188344.0,1793911188351.4,202,,d72be787-bbe0-432a-b186-7bdda80a234b,7bbaa8668c87,
|
||||
q,1793911188741.4,1793911188741.9,1793911188751.4,202,,ef2564e0-f7b9-44b6-889f-c9dfaa56ec3b,39e797bfa6f6,
|
||||
q,1793911189141.4,1793911189145.1,1793911189154.3,202,,344f9d4b-0a65-4000-a86c-028251c58975,7bbaa8668c87,
|
||||
q,1793911189541.4,1793911189542.9,1793911189554.2,202,,642bd277-82c9-4b48-a012-82a85e0a296c,39e797bfa6f6,
|
||||
q,1793911189941.4,1793911189945.6,1793911189954.1,202,,a3b2af03-6324-475c-b828-1b8a44214f1e,7bbaa8668c87,
|
||||
q,1793911190341.4,1793911190342.9,1793911190351.0,202,,3bc7e22c-0b7a-416d-944e-4796bb2bf5c5,39e797bfa6f6,
|
||||
q,1793911190741.4,1793911190745.1,1793911190770.8,202,,f386e8de-0301-4e7d-8e1e-216036abf07a,7bbaa8668c87,
|
||||
q,1793911191141.4,1793911191145.6,1793911191171.3,202,,d310b7d7-2a2c-47c0-9e63-a1a5d91e2d1e,39e797bfa6f6,
|
||||
q,1793911191541.4,1793911191544.9,1793911191553.3,202,,4b8fe177-adaa-4ddd-9163-34b9b1faf02c,7bbaa8668c87,
|
||||
q,1793911191941.4,1793911191942.9,1793911191951.5,202,,12c17651-fd40-4b3f-b31f-62e272b9335b,39e797bfa6f6,
|
||||
q,1793911192341.4,1793911192342.4,1793911192352.2,202,,4ea440f3-ec95-451e-8212-549f5e3f6037,7bbaa8668c87,
|
||||
q,1793911192741.4,1793911192742.9,1793911192751.8,202,,1f851064-6aa8-4c16-9356-0952bc5d0abb,39e797bfa6f6,
|
||||
q,1793911193141.4,1793911193143.4,1793911193162.0,202,,6ee344d9-1617-4378-bcfa-bac5d08a4053,7bbaa8668c87,
|
||||
q,1793911193541.4,1793911193543.4,1793911193551.6,202,,49a2796f-0f7b-451f-a505-916a5ff47bbf,39e797bfa6f6,
|
||||
q,1793911193941.4,1793911193942.9,1793911193951.1,202,,165097b6-aa3e-406d-b234-ad7e58437652,7bbaa8668c87,
|
||||
q,1793911194341.4,1793911194342.9,1793911194352.4,202,,828461ab-194a-4ae3-b724-3a56b3c403b8,39e797bfa6f6,
|
||||
q,1793911194741.4,1793911194742.7,1793911194769.3,202,,23fd66cc-c08e-44f2-bc9b-62c76f241fe0,7bbaa8668c87,
|
||||
q,1793911195141.4,1793911195144.6,1793911195158.1,202,,9514827f-81ae-45e0-af3a-1727b44e7c76,39e797bfa6f6,
|
||||
q,1793911195541.4,1793911195543.7,1793911195550.9,202,,39a9cc86-0493-42b0-ba81-78befbf3c588,7bbaa8668c87,
|
||||
q,1793911195941.4,1793911195943.6,1793911195952.3,202,,e902f547-46d0-4be5-8841-4e8a1c6d6a26,39e797bfa6f6,
|
||||
q,1793911196341.4,1793911196343.3,1793911196378.9,202,,e51dfddd-d837-401d-afdc-db80b0017c9c,7bbaa8668c87,
|
||||
q,1793911196741.4,1793911196745.8,1793911196768.5,202,,981e8f2c-ed96-4af0-8b43-3c50d6eecd18,39e797bfa6f6,
|
||||
q,1793911197141.4,1793911197142.9,1793911197151.7,202,,bb838257-9421-4a55-ae63-75660531618a,7bbaa8668c87,
|
||||
q,1793911197541.4,1793911197541.9,1793911197550.5,202,,5c56631a-8584-418e-83f4-ab08b8ffe85c,39e797bfa6f6,
|
||||
q,1793911197941.4,1793911197942.0,1793911197960.5,202,,a36844b3-00eb-49eb-a52a-7206df68a36e,7bbaa8668c87,
|
||||
q,1793911198341.4,1793911198347.7,1793911198373.0,202,,30e0658f-4961-424f-8926-279e41512ab6,39e797bfa6f6,
|
||||
q,1793911198741.4,1793911198745.3,1793911198757.5,202,,39c6a559-62dc-4106-bfea-95880190e06d,7bbaa8668c87,
|
||||
q,1793911199141.4,1793911199142.3,1793911199149.9,202,,9d625280-cd44-4da5-afbd-d4748c21a7cd,39e797bfa6f6,
|
||||
q,1793911199541.4,1793911199543.2,1793911199552.8,202,,a883b66f-6bba-46b0-a133-e917ca6a85de,7bbaa8668c87,
|
||||
q,1793911199941.4,1793911199943.6,1793911199954.6,202,,1d57ab99-dd55-4016-80c7-0f92dbc31174,39e797bfa6f6,
|
||||
q,1793911200341.4,1793911200343.9,1793911200367.4,202,,7e8dbf75-3c84-40b6-90e9-e0eb1708861f,7bbaa8668c87,
|
||||
q,1793911200741.4,1793911200745.8,1793911200755.6,202,,958e77a5-8db4-4a6c-8002-765df4512dd7,39e797bfa6f6,
|
||||
q,1793911201141.4,1793911201141.9,1793911201151.0,202,,e802bc7a-1559-452c-b857-551335c7fefb,7bbaa8668c87,
|
||||
q,1793911201541.4,1793911201543.1,1793911201556.0,202,,95b16c89-c0e8-4e94-953b-a35e8b31d8c7,39e797bfa6f6,
|
||||
q,1793911201941.4,1793911201943.3,1793911201968.9,202,,6c063240-4af9-4371-9f64-8a4cffce71d2,7bbaa8668c87,
|
||||
q,1793911202341.4,1793911202348.9,1793911202361.2,202,,ab64bbb2-997d-40a1-bc9d-b152f296b465,39e797bfa6f6,
|
||||
q,1793911202741.4,1793911202747.6,1793911202756.0,202,,5192bed5-df10-4a77-bf22-c126c54e7884,7bbaa8668c87,
|
||||
q,1793911203141.4,1793911203152.9,1793911203161.4,202,,12e9cb8a-115d-4dd8-9906-7804e432cc1a,39e797bfa6f6,
|
||||
q,1793911203541.4,1793911203545.5,1793911203556.3,202,,8ed0b265-295e-442f-b517-2300e77467ac,7bbaa8668c87,
|
||||
q,1793911203941.4,1793911203952.2,1793911203978.0,202,,7ee0ac06-ebd2-4afa-8e59-51b97120cdb3,39e797bfa6f6,
|
||||
q,1793911204341.4,1793911204353.0,1793911204364.6,202,,ffb5ae64-fee1-406f-8fc2-b6fbfc3d3a89,7bbaa8668c87,
|
||||
q,1793911204741.4,1793911204743.9,1793911204751.1,202,,d5b32026-a44c-4585-bb53-d74dbf710eea,39e797bfa6f6,
|
||||
q,1793911205141.4,1793911205143.0,1793911205152.1,202,,85d10179-74e9-4248-bdfd-cbdcd9f0e6cf,7bbaa8668c87,
|
||||
q,1793911205541.4,1793911205544.1,1793911205557.5,202,,fbcd4815-513d-4b5e-8111-cf22c8b761ea,39e797bfa6f6,
|
||||
q,1793911205941.4,1793911205945.3,1793911205964.0,202,,fa7e6d4e-8aee-4750-b2e0-0bb521ad1275,7bbaa8668c87,
|
||||
q,1793911206341.4,1793911206344.8,1793911206353.6,202,,3f7a9a25-f297-4f5e-83ac-4f6af9078405,39e797bfa6f6,
|
||||
q,1793911206741.4,1793911206742.9,1793911206751.3,202,,724ace3e-cfdd-4b82-85e4-79124313fae0,7bbaa8668c87,
|
||||
q,1793911207141.4,1793911207144.9,1793911207170.6,202,,203dfbfc-50e7-44cd-88e7-72ae357f04bf,39e797bfa6f6,
|
||||
q,1793911207541.4,1793911207543.7,1793911207553.3,202,,b52a1887-8d5a-4c0f-9135-c14a02bb98ce,7bbaa8668c87,
|
||||
q,1793911207941.4,1793911207948.8,1793911207962.6,202,,763db4e2-6bf6-48fc-b70e-4edd0fef3001,39e797bfa6f6,
|
||||
q,1793911208341.4,1793911208341.9,1793911208349.4,202,,cd434702-d2de-42db-963f-69fbf791f861,7bbaa8668c87,
|
||||
q,1793911208741.4,1793911208747.4,1793911208756.4,202,,6ad4a07b-fac2-4829-8483-95cfeb19bf4a,39e797bfa6f6,
|
||||
q,1793911209141.4,1793911209153.3,1793911209180.2,202,,e0b95105-2fdf-42f8-85c1-6d2f782b397f,7bbaa8668c87,
|
||||
q,1793911209541.4,1793911209543.9,1793911209554.1,202,,d5caf79b-a2b2-484a-915a-e4f099795083,39e797bfa6f6,
|
||||
q,1793911209941.4,1793911209952.0,1793911209961.7,202,,62eba477-3ef3-456f-8c87-04fa70432eba,7bbaa8668c87,
|
||||
q,1793911210341.4,1793911210342.9,1793911210352.4,202,,7240811b-9e47-4428-8cb1-ca7086251833,39e797bfa6f6,
|
||||
q,1793911210741.4,1793911210743.0,1793911210754.1,202,,02b5e5e6-b5d1-4791-9ab2-f6186d500174,7bbaa8668c87,
|
||||
q,1793911211141.4,1793911211147.0,1793911211174.4,202,,dfdcd327-638a-4729-8e12-3183ab5d3f49,39e797bfa6f6,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 2.5,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793911171.541,
|
||||
"end": 1793911211.541,
|
||||
"finished": 1793911211.173,
|
||||
"scheduled": 100,
|
||||
"sent": 100,
|
||||
"accepted": 100,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 100
|
||||
},
|
||||
"upload_p50_ms": 9.83,
|
||||
"upload_p95_ms": 26.62,
|
||||
"upload_max_ms": 35.58,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.131,
|
||||
"log": "/tmp/c2p32-s-r1-fixed.log"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"rate": 27.1532,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793911212.46,
|
||||
"end": 1793911252.492,
|
||||
"finished": 1793911252.471,
|
||||
"scheduled": 1087,
|
||||
"sent": 1087,
|
||||
"accepted": 596,
|
||||
"rejected_429": 491,
|
||||
"retry_after": {
|
||||
"5": 491
|
||||
},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 596,
|
||||
"429": 491
|
||||
},
|
||||
"upload_p50_ms": 8.3,
|
||||
"upload_p95_ms": 10.52,
|
||||
"upload_max_ms": 32.88,
|
||||
"peak_inflight": 2,
|
||||
"gen_cpu_s": 0.769,
|
||||
"log": "/tmp/c2p32-s-r1-overload.log"
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
q,1793911257760.2,1793911257761.8,1793911257791.2,202,,326f2344-6b6a-481d-948b-43c30ca7d79c,39e797bfa6f6,
|
||||
q,1793911257883.0,1793911257884.8,1793911257897.5,202,,110a4e6b-a980-4f6d-bbe8-c16fe37eca72,7bbaa8668c87,
|
||||
q,1793911258005.7,1793911258007.7,1793911258016.0,202,,c3504f17-5ccb-4be9-85e0-8f1e8a19d6de,39e797bfa6f6,
|
||||
q,1793911258128.5,1793911258130.9,1793911258141.5,202,,8e9cb73e-8687-4213-9522-3c8825188804,7bbaa8668c87,
|
||||
q,1793911258251.2,1793911258252.1,1793911258265.4,202,,b4706623-684b-4c9a-bfc4-627dbdd0fffe,39e797bfa6f6,
|
||||
q,1793911258374.0,1793911258376.1,1793911258395.4,202,,c8386d16-6b0a-4d8a-98aa-e4153be205ce,7bbaa8668c87,
|
||||
q,1793911258496.8,1793911258498.5,1793911258508.4,202,,fe31171f-1397-4e26-a747-0314ef9e7fe0,39e797bfa6f6,
|
||||
q,1793911258619.5,1793911258623.7,1793911258632.4,202,,19ebdf0f-b086-4a81-9f26-58e3606884c1,7bbaa8668c87,
|
||||
q,1793911258742.3,1793911258753.2,1793911258761.2,202,,b9c4b7dc-eae3-478b-85a7-2ca85b9d50de,39e797bfa6f6,
|
||||
q,1793911258865.0,1793911258868.8,1793911258875.9,202,,e104ee3b-b756-457e-97ea-41e43716fd74,7bbaa8668c87,
|
||||
q,1793911258987.8,1793911258991.7,1793911259000.0,202,,6a81d106-44b5-45e8-b664-f02c0fcf5f12,39e797bfa6f6,
|
||||
q,1793911259110.6,1793911259114.2,1793911259123.1,202,,f9f39b6a-cfb3-45a3-8b14-40a2403e31bb,7bbaa8668c87,
|
||||
q,1793911259233.3,1793911259245.2,1793911259253.6,202,,dbb433c7-9af4-42ec-8512-442544704b7b,39e797bfa6f6,
|
||||
q,1793911259356.1,1793911259358.2,1793911259367.5,202,,8c516796-571b-4f7b-925b-d3594a14de9c,7bbaa8668c87,
|
||||
q,1793911259478.8,1793911259480.2,1793911259489.7,202,,8a3662ba-52bc-4f40-a307-a4cb82e693f4,39e797bfa6f6,
|
||||
q,1793911259601.6,1793911259602.2,1793911259612.5,202,,d7748b2e-acca-4588-85ac-76934aa2a992,7bbaa8668c87,
|
||||
q,1793911259724.4,1793911259725.3,1793911259733.9,202,,c7a3b48d-44b0-4e0a-b224-40e635ec3c59,39e797bfa6f6,
|
||||
q,1793911259847.1,1793911259858.3,1793911259871.0,202,,10b990a1-be89-4e9c-ba81-4669a7d26e26,7bbaa8668c87,
|
||||
q,1793911259969.9,1793911259971.3,1793911259980.5,202,,a143cd75-5f71-4068-93ac-a13bd7b3f06c,39e797bfa6f6,
|
||||
q,1793911260092.6,1793911260095.1,1793911260103.0,202,,7a747a00-38fe-44b6-8d64-27007c0ce37e,7bbaa8668c87,
|
||||
q,1793911260215.4,1793911260218.4,1793911260228.0,202,,6fac7ff6-620a-4632-8423-f626c4f79db2,39e797bfa6f6,
|
||||
q,1793911260338.2,1793911260340.6,1793911260352.2,202,,9f0c6275-abe8-4ecc-8b6a-d0e9c4031f02,7bbaa8668c87,
|
||||
q,1793911260460.9,1793911260464.3,1793911260474.4,202,,4584a9ca-c335-4a9a-a1d0-47ec70fdb991,39e797bfa6f6,
|
||||
q,1793911260583.7,1793911260595.3,1793911260605.5,202,,3d880466-753d-40e9-9ed5-16497f7ee7cb,7bbaa8668c87,
|
||||
q,1793911260706.4,1793911260709.7,1793911260718.2,202,,f882151a-feea-4adc-982c-088eb2a3931e,39e797bfa6f6,
|
||||
q,1793911260829.2,1793911260839.2,1793911260849.4,202,,3de41058-191e-415a-85ca-7bd155bdfeca,7bbaa8668c87,
|
||||
q,1793911260952.0,1793911260954.3,1793911260968.2,202,,0b3dae62-a9e2-450e-a50c-d916125335bd,39e797bfa6f6,
|
||||
q,1793911261074.7,1793911261077.2,1793911261085.7,202,,c1de651e-1316-4654-bc6b-69f6ae56f613,7bbaa8668c87,
|
||||
q,1793911261197.5,1793911261198.3,1793911261206.6,202,,08b5d4df-9083-410d-a97e-4d72a8a2ae96,39e797bfa6f6,
|
||||
q,1793911261320.2,1793911261327.2,1793911261337.1,202,,929a3bb2-69e7-42b8-b050-5176f29f17d4,7bbaa8668c87,
|
||||
q,1793911261443.0,1793911261445.5,1793911261463.1,202,,0acd4549-3fe0-470a-92ef-ee32f3265fe9,39e797bfa6f6,
|
||||
q,1793911261565.8,1793911261569.7,1793911261578.6,202,,625ea00e-ac25-46f0-bab5-0edc127ce925,7bbaa8668c87,
|
||||
q,1793911261688.5,1793911261691.7,1793911261700.2,202,,882f8406-534d-4d67-b226-6f52866b0a34,39e797bfa6f6,
|
||||
q,1793911261811.3,1793911261813.4,1793911261822.3,202,,3d198165-ccb2-42af-9361-e6d93b23500b,7bbaa8668c87,
|
||||
q,1793911261934.0,1793911261936.6,1793911261950.7,202,,a4517caa-7c0f-435a-8139-d891c76414c1,39e797bfa6f6,
|
||||
q,1793911262056.8,1793911262058.6,1793911262067.1,202,,247f5a79-f3b3-4427-9793-770db7285c1f,7bbaa8668c87,
|
||||
q,1793911262179.6,1793911262181.2,1793911262189.7,202,,c9447afc-3bbb-4cc1-bfa6-ebceeb177506,39e797bfa6f6,
|
||||
q,1793911262302.3,1793911262305.1,1793911262313.0,202,,7782f945-3516-4368-8eaa-7275b454e05b,7bbaa8668c87,
|
||||
q,1793911262425.1,1793911262429.0,1793911262436.7,202,,1007d7d4-a1b2-47cd-a483-575b62d80415,39e797bfa6f6,
|
||||
q,1793911262547.8,1793911262549.2,1793911262558.4,202,,ee6af335-b9ae-45a8-9038-6ea19373d4be,7bbaa8668c87,
|
||||
q,1793911262670.6,1793911262672.2,1793911262680.4,202,,bc9edad4-62e1-4b3b-859b-c6c477f91cbe,39e797bfa6f6,
|
||||
q,1793911262793.4,1793911262796.0,1793911262803.6,202,,6f68c8b5-4d8f-4af6-b66a-961c3a388a36,7bbaa8668c87,
|
||||
q,1793911262916.1,1793911262917.2,1793911262927.4,202,,28d7dfc0-5ce4-4437-b462-46f19d565bd9,39e797bfa6f6,
|
||||
q,1793911263038.9,1793911263040.7,1793911263049.4,202,,8c7c705a-b6f1-4aae-9cc0-1cd5751fac07,7bbaa8668c87,
|
||||
q,1793911263161.6,1793911263165.0,1793911263173.7,202,,35c00aa9-ab74-404c-bad6-8d242a933761,39e797bfa6f6,
|
||||
q,1793911263284.4,1793911263285.5,1793911263293.4,202,,fca05985-adb6-4452-ba8d-4b3c713249d0,7bbaa8668c87,
|
||||
q,1793911263407.2,1793911263408.2,1793911263419.2,202,,a8050504-346c-4b8f-ae87-3df5136f49e9,39e797bfa6f6,
|
||||
q,1793911263529.9,1793911263534.4,1793911263543.3,202,,ece4da70-63f5-4ee2-b88f-794e3a4e9c9a,7bbaa8668c87,
|
||||
q,1793911263652.7,1793911263654.4,1793911263662.6,202,,84142471-1d8e-410f-bc16-fdfd3eee8502,39e797bfa6f6,
|
||||
q,1793911263775.5,1793911263776.2,1793911263785.2,202,,eab8e849-0f71-429a-9295-4e626da4da2e,7bbaa8668c87,
|
||||
q,1793911263898.2,1793911263899.2,1793911263910.7,202,,9d75427e-292c-465f-9732-4e48660d50d9,39e797bfa6f6,
|
||||
q,1793911264021.0,1793911264022.5,1793911264033.4,202,,c6974b6d-d555-4e00-a733-19d393a82504,7bbaa8668c87,
|
||||
q,1793911264143.7,1793911264145.2,1793911264154.1,202,,e0356036-f45c-4e79-84f5-ee48628a60b6,39e797bfa6f6,
|
||||
q,1793911264266.5,1793911264268.2,1793911264276.6,202,,5935c453-8c66-43be-9ef7-89533a9f5981,7bbaa8668c87,
|
||||
q,1793911264389.3,1793911264390.3,1793911264399.8,202,,4a008c91-f674-4227-97ef-f28e6f96e84a,39e797bfa6f6,
|
||||
q,1793911264512.0,1793911264513.5,1793911264526.3,202,,492e09fd-4aad-47f2-88e0-fa4a37e74081,7bbaa8668c87,
|
||||
q,1793911264634.8,1793911264636.7,1793911264645.6,202,,ae6e629f-a8af-4607-a41c-aa9ae888dafa,39e797bfa6f6,
|
||||
q,1793911264757.5,1793911264758.5,1793911264767.6,202,,6ed9666f-cf01-408f-a00f-081cf7c8d8a0,7bbaa8668c87,
|
||||
q,1793911264880.3,1793911264883.1,1793911264891.3,202,,58096dd9-c20a-476e-bcd9-ed5c27206069,39e797bfa6f6,
|
||||
q,1793911265003.1,1793911265004.1,1793911265012.7,202,,ee6997aa-ee28-496c-a4bf-839552d3a690,7bbaa8668c87,
|
||||
q,1793911265125.8,1793911265128.4,1793911265138.3,202,,546ed200-08c9-4b97-959b-3023b9cde533,39e797bfa6f6,
|
||||
q,1793911265248.6,1793911265260.2,1793911265268.8,202,,170ebe72-5996-4caf-b6d5-8fc6e32bc1a0,7bbaa8668c87,
|
||||
q,1793911265371.3,1793911265374.2,1793911265382.5,202,,d74f5607-3a1d-4fb2-b52e-d342052eb5ba,39e797bfa6f6,
|
||||
q,1793911265494.1,1793911265495.3,1793911265504.2,202,,42f615b7-8ef0-4f83-869b-453d7e9b26b9,7bbaa8668c87,
|
||||
q,1793911265616.9,1793911265618.2,1793911265628.1,202,,41b351db-485f-4ec1-9168-d9e034c3a748,39e797bfa6f6,
|
||||
q,1793911265739.6,1793911265741.4,1793911265751.1,202,,14ea9515-26b6-4bd7-89f6-d70ba272e2ba,7bbaa8668c87,
|
||||
q,1793911265862.4,1793911265874.3,1793911265882.6,202,,28103722-a100-45d2-8754-3817ea082f59,39e797bfa6f6,
|
||||
q,1793911265985.1,1793911265989.3,1793911266002.1,202,,413ef148-a363-4104-914b-f6855bb741f3,7bbaa8668c87,
|
||||
q,1793911266107.9,1793911266109.2,1793911266117.7,202,,39e326b3-21c3-4799-9eb6-64bed2fef399,39e797bfa6f6,
|
||||
q,1793911266230.7,1793911266231.3,1793911266239.9,202,,c1e1fb3b-1734-440c-80f5-8dc87d9e00a1,7bbaa8668c87,
|
||||
q,1793911266353.4,1793911266356.4,1793911266363.8,202,,492c60c1-3cfe-47ce-9216-3e1dd551c8cf,39e797bfa6f6,
|
||||
q,1793911266476.2,1793911266477.3,1793911266492.2,202,,62330779-51a0-49c0-9f0e-3b37b117cc91,7bbaa8668c87,
|
||||
q,1793911266598.9,1793911266601.7,1793911266610.4,202,,43d818d9-9d43-4b45-b3fe-a076741f471f,39e797bfa6f6,
|
||||
q,1793911266721.7,1793911266723.7,1793911266732.2,202,,f8c94c3f-db6d-451b-b0a4-9391058cfc5f,7bbaa8668c87,
|
||||
q,1793911266844.5,1793911266846.0,1793911266854.1,202,,934bede6-7b63-4c84-8a89-8f357430c59e,39e797bfa6f6,
|
||||
q,1793911266967.2,1793911266969.7,1793911266982.0,202,,76a87438-971a-4a61-9c79-0abd9adeb96d,7bbaa8668c87,
|
||||
q,1793911267090.0,1793911267092.2,1793911267100.8,202,,3b893350-2cc5-473c-9707-48fa69ae23fb,39e797bfa6f6,
|
||||
q,1793911267212.7,1793911267213.3,1793911267231.0,202,,9f5724ad-a741-42e5-bfa7-f071237adec2,7bbaa8668c87,
|
||||
q,1793911267335.5,1793911267337.4,1793911267346.5,202,,e1e17e3c-8b6b-4018-8a31-4724ed5c2d25,39e797bfa6f6,
|
||||
q,1793911267458.3,1793911267459.8,1793911267470.7,202,,fc6fb71e-57ca-4516-85b3-353f2854f5f6,7bbaa8668c87,
|
||||
q,1793911267581.0,1793911267584.7,1793911267593.7,202,,83e7a96e-7c85-4913-afdf-a74621227cdb,39e797bfa6f6,
|
||||
q,1793911267703.8,1793911267705.5,1793911267715.6,202,,3a937967-82ff-41bb-b325-3416ef83c7bc,7bbaa8668c87,
|
||||
q,1793911267826.5,1793911267829.7,1793911267839.2,202,,a1515544-ffbf-47d5-a24f-cafef0914e85,39e797bfa6f6,
|
||||
q,1793911267949.3,1793911267950.6,1793911267959.6,202,,8f9148f6-24b2-4922-bae5-8f932f01b3d1,7bbaa8668c87,
|
||||
q,1793911268072.1,1793911268073.4,1793911268089.4,202,,ee46744f-7308-4ec6-8350-bffff60c47ef,39e797bfa6f6,
|
||||
q,1793911268194.8,1793911268197.7,1793911268206.7,202,,4f1c0698-59ea-40e3-8f73-b800d945c540,7bbaa8668c87,
|
||||
q,1793911268317.6,1793911268319.2,1793911268328.9,202,,1ae44fd6-fe1d-4a42-8489-74c073d7c6c8,39e797bfa6f6,
|
||||
q,1793911268440.3,1793911268443.1,1793911268452.2,202,,768d1d8b-7023-4582-bbf3-1b741a88f4af,7bbaa8668c87,
|
||||
q,1793911268563.1,1793911268565.1,1793911268575.1,202,,0ea6d46b-e799-4e25-bb86-c5562be5fe44,39e797bfa6f6,
|
||||
q,1793911268685.9,1793911268692.1,1793911268701.0,202,,a68e39d7-73d8-4f6e-b649-eec56b7cbb67,7bbaa8668c87,
|
||||
q,1793911268808.6,1793911268813.2,1793911268821.9,202,,04ce52c2-df19-42ee-a57e-723d3568f671,39e797bfa6f6,
|
||||
q,1793911268931.4,1793911268932.2,1793911268940.3,202,,39c39c79-75f0-4845-bde0-bdb25ddda481,7bbaa8668c87,
|
||||
q,1793911269054.1,1793911269056.7,1793911269064.9,202,,d6a79e28-ebbe-4bef-b60b-dddea0e2cfca,39e797bfa6f6,
|
||||
q,1793911269176.9,1793911269178.5,1793911269186.9,202,,4c2b1ccc-b98f-4c86-8561-c4b49ddcfdee,7bbaa8668c87,
|
||||
q,1793911269299.7,1793911269300.7,1793911269309.2,202,,f11cf6f6-bc60-4929-9a99-f3e6837d1550,39e797bfa6f6,
|
||||
q,1793911269422.4,1793911269425.8,1793911269433.7,202,,2933f7b9-0410-4ead-97bc-d4f6b64c0555,7bbaa8668c87,
|
||||
q,1793911269545.2,1793911269547.3,1793911269556.5,202,,01fb79c7-20f8-4173-811c-30f22bee6296,39e797bfa6f6,
|
||||
q,1793911269668.0,1793911269669.4,1793911269678.2,202,,627ae6e0-e346-402a-8ba3-a48c8f32079d,7bbaa8668c87,
|
||||
q,1793911269790.7,1793911269793.2,1793911269803.2,202,,84ca0589-dd5f-444f-8ac3-964d6b314260,39e797bfa6f6,
|
||||
q,1793911269913.5,1793911269916.1,1793911269923.8,202,,97109492-1dcb-44c2-ac12-ba0378fc78bb,7bbaa8668c87,
|
||||
q,1793911270036.2,1793911270038.2,1793911270050.5,202,,a6fc05c1-2243-4ac2-945b-69326bad2a18,39e797bfa6f6,
|
||||
q,1793911270159.0,1793911270160.2,1793911270169.5,202,,ff3c2a94-9aa3-484e-9c89-e5b9c35cb7e3,7bbaa8668c87,
|
||||
q,1793911270281.8,1793911270283.2,1793911270291.3,202,,4af437f8-2302-4656-a2c8-daec51b4909b,39e797bfa6f6,
|
||||
q,1793911270404.5,1793911270406.2,1793911270416.1,202,,b4e97f00-8ba3-498e-86c8-24565cf54be7,7bbaa8668c87,
|
||||
q,1793911270527.3,1793911270529.3,1793911270538.0,202,,2c9ba206-364a-4021-8e8b-d55c87a187e9,39e797bfa6f6,
|
||||
q,1793911270650.0,1793911270654.7,1793911270663.2,202,,d209c4c8-3036-4932-9749-2f479f73e53c,7bbaa8668c87,
|
||||
q,1793911270772.8,1793911270774.2,1793911270782.8,202,,efc556f8-d41a-45a2-a06d-a858a47b0889,39e797bfa6f6,
|
||||
q,1793911270895.6,1793911270896.2,1793911270904.9,202,,eef64d1a-4af5-4d3f-8962-6fab9177de3f,7bbaa8668c87,
|
||||
q,1793911271018.3,1793911271021.3,1793911271033.0,202,,15835551-6551-4568-b2fb-4c4f2bd1053b,39e797bfa6f6,
|
||||
q,1793911271141.1,1793911271142.6,1793911271153.7,202,,707b7a1f-4225-4838-8c1b-a1a919b936f7,7bbaa8668c87,
|
||||
q,1793911271263.8,1793911271270.5,1793911271279.1,202,,d7d3a3ea-5de5-4616-bc25-29c3159fecf0,39e797bfa6f6,
|
||||
q,1793911271386.6,1793911271388.7,1793911271397.4,202,,7876eb8a-26e3-464b-bbfe-33851fd46084,7bbaa8668c87,
|
||||
q,1793911271509.4,1793911271510.3,1793911271519.5,202,,920b0429-cfc0-4b5d-bbab-17479b1a2209,39e797bfa6f6,
|
||||
q,1793911271632.1,1793911271635.6,1793911271649.8,202,,3ff7991c-38d6-4216-afb9-fbf5b44350c6,7bbaa8668c87,
|
||||
q,1793911271754.9,1793911271760.2,1793911271769.0,202,,7148b290-945e-4fc5-ad83-be6946468075,39e797bfa6f6,
|
||||
q,1793911271877.6,1793911271881.2,1793911271889.7,202,,2459bfee-c8b5-4e68-ba99-4944433499c1,7bbaa8668c87,
|
||||
q,1793911272000.4,1793911272005.3,1793911272014.3,202,,6631f15b-de77-41a8-a8d3-67d0de825cfd,39e797bfa6f6,
|
||||
q,1793911272123.2,1793911272126.7,1793911272142.5,202,,6de4f480-80ff-4bab-9589-4e5720c90b0d,7bbaa8668c87,
|
||||
q,1793911272245.9,1793911272250.2,1793911272258.9,202,,506a996a-47c7-4a71-a725-e46f16fcd6c3,39e797bfa6f6,
|
||||
q,1793911272368.7,1793911272371.2,1793911272379.7,202,,ca51bb80-f5cf-4867-b62a-20f27a866df7,7bbaa8668c87,
|
||||
q,1793911272491.4,1793911272492.2,1793911272500.6,202,,f1f1d77e-e5a7-452d-91b2-64b0201ea7b7,39e797bfa6f6,
|
||||
q,1793911272614.2,1793911272617.5,1793911272624.9,202,,c5d92ad2-137c-486a-bee0-fd7e054a5770,7bbaa8668c87,
|
||||
q,1793911272737.0,1793911272747.3,1793911272755.7,202,,f5cdd711-5a54-4462-8637-ed8fe9a31e11,39e797bfa6f6,
|
||||
q,1793911272859.7,1793911272861.7,1793911272870.4,202,,5af89c05-e483-4462-9af9-1c8efd5827ec,7bbaa8668c87,
|
||||
q,1793911272982.5,1793911272985.1,1793911272993.0,202,,392b5d91-b618-4be2-bfc9-d6fa83a0dcf2,39e797bfa6f6,
|
||||
q,1793911273105.2,1793911273107.3,1793911273118.9,202,,8024a3b0-3dbe-4255-b638-f3b86e983c54,7bbaa8668c87,
|
||||
q,1793911273228.0,1793911273239.2,1793911273248.1,202,,ea300a06-86ad-41c5-bc28-42799c7bccc6,39e797bfa6f6,
|
||||
q,1793911273350.8,1793911273356.7,1793911273365.0,202,,6ab92a2d-73c0-4812-9ace-397b5ceb689c,7bbaa8668c87,
|
||||
q,1793911273473.5,1793911273474.3,1793911273484.2,202,,fe2aebab-446f-4cea-a4f2-01d740a84ffb,39e797bfa6f6,
|
||||
q,1793911273596.3,1793911273605.6,1793911273617.6,202,,9a585ced-ed97-44c0-a65b-ee16a6aa62d2,7bbaa8668c87,
|
||||
q,1793911273719.0,1793911273721.2,1793911273730.0,202,,b8b08ade-3ee6-4441-a721-aba27aeb36ca,39e797bfa6f6,
|
||||
q,1793911273841.8,1793911273842.6,1793911273850.6,202,,965a9555-e43e-4da5-91c2-cdbc0b887b72,7bbaa8668c87,
|
||||
q,1793911273964.6,1793911273966.7,1793911273976.5,202,,fc205b35-a9d3-4f9b-a4f1-c60006fe7d54,39e797bfa6f6,
|
||||
q,1793911274087.3,1793911274090.4,1793911274102.7,202,,e64ba5da-f05f-4bc4-8c50-13dbe316cb99,7bbaa8668c87,
|
||||
q,1793911274210.1,1793911274211.3,1793911274221.6,202,,544d0340-d5a3-4c91-ab39-b76178036313,39e797bfa6f6,
|
||||
q,1793911274332.8,1793911274335.2,1793911274343.9,202,,964ca384-3f6f-4a17-9225-f90ca5e6099c,7bbaa8668c87,
|
||||
q,1793911274455.6,1793911274456.8,1793911274465.4,202,,ee79122f-ec19-4e50-8178-997e516ea1c5,39e797bfa6f6,
|
||||
q,1793911274578.4,1793911274580.2,1793911274590.1,202,,801de897-78f0-416c-bc10-30a2719941a3,7bbaa8668c87,
|
||||
q,1793911274701.1,1793911274702.3,1793911274715.5,202,,93750772-9212-41f3-adbe-5e88d8d6cc1a,39e797bfa6f6,
|
||||
q,1793911274823.9,1793911274825.2,1793911274834.4,202,,b8a5be7b-751b-4ee5-ab88-200deed6ff68,7bbaa8668c87,
|
||||
q,1793911274946.6,1793911274948.2,1793911274957.5,202,,f96347b8-7daa-405e-883b-e5a0895cda14,39e797bfa6f6,
|
||||
q,1793911275069.4,1793911275071.0,1793911275079.4,202,,b39b3426-c5b9-4717-8190-c62be0968044,7bbaa8668c87,
|
||||
q,1793911275192.2,1793911275195.1,1793911275209.2,202,,c4053d6d-97fd-40f5-b9d1-57bf32753a69,39e797bfa6f6,
|
||||
q,1793911275314.9,1793911275316.2,1793911275324.8,202,,4f807314-afca-49b1-ad39-1e66da3b5217,7bbaa8668c87,
|
||||
q,1793911275437.7,1793911275438.2,1793911275447.1,202,,90fa5951-cd46-494b-bd45-79454668e08f,39e797bfa6f6,
|
||||
q,1793911275560.4,1793911275561.4,1793911275570.7,202,,7f9ad84d-8156-4935-8890-73fe13480018,7bbaa8668c87,
|
||||
q,1793911275683.2,1793911275684.9,1793911275693.0,202,,b8ec6eae-633b-4c90-a93e-85ae364a52dd,39e797bfa6f6,
|
||||
q,1793911275806.0,1793911275806.3,1793911275817.0,202,,d992ac11-4b26-4393-b469-dbedf2887997,7bbaa8668c87,
|
||||
q,1793911275928.7,1793911275930.5,1793911275945.7,202,,e6036ebd-8986-4c30-8d92-4864d6ae6391,39e797bfa6f6,
|
||||
q,1793911276051.5,1793911276052.4,1793911276060.4,202,,2d479e45-7f98-4e02-a1a7-03adb87de114,7bbaa8668c87,
|
||||
q,1793911276174.3,1793911276177.0,1793911276185.7,202,,e6f11364-188e-4f2c-98c7-c04d6fde5808,39e797bfa6f6,
|
||||
q,1793911276297.0,1793911276298.2,1793911276307.1,202,,cb2d0c7e-e3cb-4de5-9bad-9318c7767e5a,7bbaa8668c87,
|
||||
q,1793911276419.8,1793911276423.2,1793911276431.5,202,,1c274380-9f82-4717-88ff-9ac6f6fbe9ed,39e797bfa6f6,
|
||||
q,1793911276542.5,1793911276545.8,1793911276553.4,202,,86df6b30-2197-49c0-b98d-6788b90359af,7bbaa8668c87,
|
||||
q,1793911276665.3,1793911276669.3,1793911276681.6,202,,e0fba22d-522f-4d5f-a6f2-4a64c3d7070b,39e797bfa6f6,
|
||||
q,1793911276788.1,1793911276791.4,1793911276801.3,202,,87e3839e-773a-420d-bb87-7ae604165780,7bbaa8668c87,
|
||||
q,1793911276910.8,1793911276912.3,1793911276920.8,202,,6c49a3df-ccf3-49d9-8ded-61506addae10,39e797bfa6f6,
|
||||
q,1793911277033.6,1793911277036.2,1793911277045.2,202,,f834a7c3-c19a-49f7-bab7-84bcf6105758,7bbaa8668c87,
|
||||
q,1793911277156.3,1793911277168.3,1793911277179.8,202,,c7dee6b0-af01-4951-a755-75b63c1a15d7,39e797bfa6f6,
|
||||
q,1793911277279.1,1793911277280.7,1793911277293.7,202,,660e2d32-60dd-45af-9499-a67e7b77d05e,7bbaa8668c87,
|
||||
q,1793911277401.9,1793911277403.7,1793911277412.0,202,,312e250f-a5d4-4428-b076-82ddb9493911,39e797bfa6f6,
|
||||
q,1793911277524.6,1793911277527.7,1793911277536.9,202,,cf366acf-229a-4fe0-b2b6-394136ec72fa,7bbaa8668c87,
|
||||
q,1793911277647.4,1793911277658.3,1793911277667.7,202,,8a7954d7-beb2-47e6-badc-678cd81347ea,39e797bfa6f6,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 8.14595,
|
||||
"duration_s": 20.0,
|
||||
"start": 1793911257.76,
|
||||
"end": 1793911277.77,
|
||||
"finished": 1793911277.667,
|
||||
"scheduled": 163,
|
||||
"sent": 163,
|
||||
"accepted": 163,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 163
|
||||
},
|
||||
"upload_p50_ms": 8.93,
|
||||
"upload_p95_ms": 14.19,
|
||||
"upload_max_ms": 29.45,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.139,
|
||||
"log": "/tmp/c2p32-s-r1-recovery.log"
|
||||
}
|
||||
@@ -0,0 +1,326 @@
|
||||
q,1793911130379.8,1793911130380.8,1793911130393.6,202,,44e1fe81-57d4-4e38-bd26-815bbd9f278d,7bbaa8668c87,
|
||||
q,1793911130502.6,1793911130505.0,1793911130514.4,202,,60d65f99-c9e3-4e6a-9e74-c70d92313990,39e797bfa6f6,
|
||||
q,1793911130625.3,1793911130629.4,1793911130637.3,202,,be5a0eda-eb97-4dd7-b92d-9300e6c1c6d3,7bbaa8668c87,
|
||||
q,1793911130748.1,1793911130751.9,1793911130765.2,202,,fc92c570-62b1-4c73-bedf-7a9f053f14f1,39e797bfa6f6,
|
||||
q,1793911130870.9,1793911130871.5,1793911130888.7,202,,712628a0-e1f8-45dc-b512-83f1f974f52f,7bbaa8668c87,
|
||||
q,1793911130993.6,1793911130996.4,1793911131004.9,202,,ebe000f7-da7a-4fe4-b9f8-91fb4d754d2b,39e797bfa6f6,
|
||||
q,1793911131116.4,1793911131117.6,1793911131125.5,202,,9633f209-e14a-4096-baf5-f52c28d0eb53,7bbaa8668c87,
|
||||
q,1793911131239.1,1793911131246.6,1793911131255.1,202,,b2927dac-df5c-4401-9427-892da3f389a9,39e797bfa6f6,
|
||||
q,1793911131361.9,1793911131372.5,1793911131384.8,202,,1854288b-634f-46f4-87ed-cf257250bd81,7bbaa8668c87,
|
||||
q,1793911131484.7,1793911131485.5,1793911131499.5,202,,17a2bd26-060a-4a8d-a2e3-6e0499ef0ed9,39e797bfa6f6,
|
||||
q,1793911131607.4,1793911131609.5,1793911131618.5,202,,51da4b18-f259-4987-a00f-2ffe0d7bb6ec,7bbaa8668c87,
|
||||
q,1793911131730.2,1793911131731.0,1793911131739.7,202,,7d9c198c-e08a-4870-9a5f-97fd5478cda4,39e797bfa6f6,
|
||||
q,1793911131852.9,1793911131864.6,1793911131873.9,202,,b2027e7c-996e-42f6-bcad-b9bb64a6783b,7bbaa8668c87,
|
||||
q,1793911131975.7,1793911131979.8,1793911131994.5,202,,70a9dddc-f5c6-4ae5-a540-a78ddcb9856e,39e797bfa6f6,
|
||||
q,1793911132098.5,1793911132100.9,1793911132109.4,202,,2c3ee49a-38fd-4745-a7e5-61946eb9c0d1,7bbaa8668c87,
|
||||
q,1793911132221.2,1793911132225.9,1793911132234.6,202,,07159bc8-0d8d-4335-8a1d-65b57f64d44d,39e797bfa6f6,
|
||||
q,1793911132344.0,1793911132350.6,1793911132359.5,202,,c5c58703-118e-42de-b838-eda51330576b,7bbaa8668c87,
|
||||
q,1793911132466.8,1793911132476.3,1793911132493.8,202,,ae2d2957-060c-4a80-9637-047213d4f488,39e797bfa6f6,
|
||||
q,1793911132589.5,1793911132590.5,1793911132599.2,202,,0a910490-5f4f-4709-a107-093643d75c71,7bbaa8668c87,
|
||||
q,1793911132712.3,1793911132714.2,1793911132722.8,202,,62a1c0c8-2498-4e3c-856d-453b0a7fff51,39e797bfa6f6,
|
||||
q,1793911132835.0,1793911132838.8,1793911132846.6,202,,8a3835c0-9bc3-461e-b42c-9813ebe206ca,7bbaa8668c87,
|
||||
q,1793911132957.8,1793911132967.8,1793911132983.8,202,,5a7293e5-b25a-4194-89bd-103a27b6501f,39e797bfa6f6,
|
||||
q,1793911133080.6,1793911133082.2,1793911133090.6,202,,b97870c5-b6a1-4fe2-ab60-a3672d8b41f6,7bbaa8668c87,
|
||||
q,1793911133203.3,1793911133206.5,1793911133215.3,202,,ad65d852-2807-4748-a69c-99fd55a4cb5c,39e797bfa6f6,
|
||||
q,1793911133326.1,1793911133327.6,1793911133335.3,202,,000b673b-4c0a-45f0-9ff3-3c2baff2336c,7bbaa8668c87,
|
||||
q,1793911133448.8,1793911133452.3,1793911133459.5,202,,dca6ecea-acd1-48b5-be2d-fca6847b2c89,39e797bfa6f6,
|
||||
q,1793911133571.6,1793911133573.9,1793911133582.4,202,,9644181c-3ffe-4934-a82d-8e996117b591,7bbaa8668c87,
|
||||
q,1793911133694.4,1793911133697.5,1793911133705.6,202,,3f427d5b-805b-45a9-9496-4d83a48c761a,39e797bfa6f6,
|
||||
q,1793911133817.1,1793911133820.0,1793911133827.5,202,,5010d6d8-47d8-4379-bbd4-f7930b1f9212,7bbaa8668c87,
|
||||
q,1793911133939.9,1793911133942.8,1793911133952.7,202,,d901e98c-3526-4713-bd21-17a32d15314b,39e797bfa6f6,
|
||||
q,1793911134062.6,1793911134065.0,1793911134073.3,202,,216875d0-9924-44f3-a1c9-0896018f2962,7bbaa8668c87,
|
||||
q,1793911134185.4,1793911134186.5,1793911134194.6,202,,8571edad-f0a7-43c8-8c23-35736088adc4,39e797bfa6f6,
|
||||
q,1793911134308.2,1793911134309.8,1793911134317.0,202,,2e5e5422-a7e3-4b72-bc56-fc8d6e44d6ff,7bbaa8668c87,
|
||||
q,1793911134430.9,1793911134432.7,1793911134444.4,202,,40364cff-d983-410c-9721-7156da7d1e70,39e797bfa6f6,
|
||||
q,1793911134553.7,1793911134556.9,1793911134565.4,202,,2762d8e2-1c87-45ff-8bc4-a5fead595960,7bbaa8668c87,
|
||||
q,1793911134676.4,1793911134677.5,1793911134687.4,202,,da27c281-d904-4d37-b9e6-1a6035e6ece1,39e797bfa6f6,
|
||||
q,1793911134799.2,1793911134800.5,1793911134811.0,202,,74bf818b-60ac-4706-b3bc-b98308609a76,7bbaa8668c87,
|
||||
q,1793911134922.0,1793911134922.5,1793911134931.3,202,,75edf182-cc92-47ba-8b03-2c43d2f942e4,39e797bfa6f6,
|
||||
q,1793911135044.7,1793911135050.2,1793911135058.5,202,,8ca6d8f1-9c82-4cc7-ac60-f9ccd50a2d77,7bbaa8668c87,
|
||||
q,1793911135167.5,1793911135169.0,1793911135193.2,202,,71a8910b-35f0-4c85-bbd4-3584d2b94a90,39e797bfa6f6,
|
||||
q,1793911135290.2,1793911135291.9,1793911135300.7,202,,26851df8-8727-47ae-a4df-f50328ceb3ac,7bbaa8668c87,
|
||||
q,1793911135413.0,1793911135416.0,1793911135426.7,202,,4031ce3f-c9e2-4c7b-8ca7-b67f44971954,39e797bfa6f6,
|
||||
q,1793911135535.8,1793911135537.7,1793911135547.9,202,,151b0e75-0127-4a06-8f12-e17b130ce605,7bbaa8668c87,
|
||||
q,1793911135658.5,1793911135659.5,1793911135668.6,202,,4f453f09-f982-4760-9b5a-d7e67d9c0b87,39e797bfa6f6,
|
||||
q,1793911135781.3,1793911135783.0,1793911135791.5,202,,e505f112-fb77-4c06-84de-24bfb8c73559,7bbaa8668c87,
|
||||
q,1793911135904.0,1793911135906.9,1793911135916.6,202,,46eaa12d-fd19-4c53-a2f0-9e3257053074,39e797bfa6f6,
|
||||
q,1793911136026.8,1793911136028.3,1793911136039.8,202,,e706f8c1-adcf-45da-9826-3617c79f603b,7bbaa8668c87,
|
||||
q,1793911136149.6,1793911136150.7,1793911136159.2,202,,9ff011fc-472a-4e47-a7d7-8fd16a57708d,39e797bfa6f6,
|
||||
q,1793911136272.3,1793911136273.5,1793911136282.0,202,,06e0f407-378f-4bfb-8edb-e67f56e1987e,7bbaa8668c87,
|
||||
q,1793911136395.1,1793911136396.5,1793911136405.2,202,,401712e7-2239-4f14-b35e-70b200046604,39e797bfa6f6,
|
||||
q,1793911136517.8,1793911136518.9,1793911136526.6,202,,e9385963-0e6b-4e91-8477-4b564446706b,7bbaa8668c87,
|
||||
q,1793911136640.6,1793911136641.5,1793911136650.1,202,,96e189aa-b257-41ae-9bf1-a6313053ed42,39e797bfa6f6,
|
||||
q,1793911136763.4,1793911136765.0,1793911136773.4,202,,0bcda0dc-4360-41ce-9794-22efade3e713,7bbaa8668c87,
|
||||
q,1793911136886.1,1793911136887.9,1793911136896.1,202,,000e1cce-ca2f-44aa-94ba-7aca891de18e,39e797bfa6f6,
|
||||
q,1793911137008.9,1793911137010.5,1793911137024.9,202,,07bd93b6-08d8-4eed-b24d-5b890dec49df,7bbaa8668c87,
|
||||
q,1793911137131.6,1793911137132.8,1793911137142.1,202,,680e2a93-034f-426a-b9fe-de7f026ebda8,39e797bfa6f6,
|
||||
q,1793911137254.4,1793911137256.0,1793911137264.1,202,,1bdd0f7e-9905-46ef-b193-1aa0e8ad47c3,7bbaa8668c87,
|
||||
q,1793911137377.2,1793911137378.5,1793911137386.2,202,,33938da9-e149-4d29-b5b3-a3bf61700a47,39e797bfa6f6,
|
||||
q,1793911137499.9,1793911137501.5,1793911137511.4,202,,2e415e6b-1b7a-4b1a-93ff-d9ee75dc0f01,7bbaa8668c87,
|
||||
q,1793911137622.7,1793911137624.5,1793911137634.3,202,,f8482791-34bc-4ce9-b3f3-2bdf7b300e0b,39e797bfa6f6,
|
||||
q,1793911137745.4,1793911137747.8,1793911137756.0,202,,ccf2b3a6-9be0-4fd0-96b6-3a19afcb4058,7bbaa8668c87,
|
||||
q,1793911137868.2,1793911137869.5,1793911137876.9,202,,adb4a809-3443-4138-9f58-e23e799b85bf,39e797bfa6f6,
|
||||
q,1793911137991.0,1793911137994.6,1793911138007.3,202,,2763151d-4b0d-4145-8467-2b4d172c4e77,7bbaa8668c87,
|
||||
q,1793911138113.7,1793911138115.5,1793911138124.8,202,,44f0efbb-9954-43cd-8756-655a3b02fe41,39e797bfa6f6,
|
||||
q,1793911138236.5,1793911138237.5,1793911138246.0,202,,98244284-28c7-4909-a4e1-60df570cf73e,7bbaa8668c87,
|
||||
q,1793911138359.2,1793911138360.6,1793911138369.0,202,,78c33961-64e5-49b2-a9de-72679528e38c,39e797bfa6f6,
|
||||
q,1793911138482.0,1793911138483.1,1793911138496.2,202,,d0cd97dd-66e8-488a-a28b-0d652df06a4b,7bbaa8668c87,
|
||||
q,1793911138604.8,1793911138605.5,1793911138615.3,202,,d2a5ef43-7cb7-41f3-a7c5-36a0086b1387,39e797bfa6f6,
|
||||
q,1793911138727.5,1793911138730.5,1793911138740.4,202,,44dcb20a-71e7-4c34-8198-dd5468976acc,7bbaa8668c87,
|
||||
q,1793911138850.3,1793911138851.9,1793911138860.9,202,,5eecbb4d-1d6d-4060-af20-1f0721aa9bee,39e797bfa6f6,
|
||||
q,1793911138973.1,1793911138977.5,1793911138987.1,202,,ad60bd49-0e3a-4c79-a000-65b653c3b19f,7bbaa8668c87,
|
||||
q,1793911139095.8,1793911139106.6,1793911139128.2,202,,641985f1-830b-4ff6-8384-fb01574d51ea,39e797bfa6f6,
|
||||
q,1793911139218.6,1793911139221.1,1793911139229.9,202,,a9c2795b-d1c5-4cf2-a1a6-0320529010e1,7bbaa8668c87,
|
||||
q,1793911139341.3,1793911139342.5,1793911139351.1,202,,a1fc37c4-2f92-4cc6-9bf3-dbba717e3744,39e797bfa6f6,
|
||||
q,1793911139464.1,1793911139466.3,1793911139474.6,202,,e10899c4-8fe0-4787-b7d3-5c8796955c02,7bbaa8668c87,
|
||||
q,1793911139586.9,1793911139598.6,1793911139610.3,202,,5d4c976d-89e5-406c-8582-43b3a0f70018,39e797bfa6f6,
|
||||
q,1793911139709.6,1793911139715.2,1793911139724.0,202,,f68ea388-0ab1-4f32-9fb7-e616c968f35e,7bbaa8668c87,
|
||||
q,1793911139832.4,1793911139838.8,1793911139847.1,202,,20e29c81-e839-4841-bed2-fd23d7afb175,39e797bfa6f6,
|
||||
q,1793911139955.1,1793911139957.5,1793911139965.5,202,,62e168f9-dd68-43bd-be2f-37a156fb14c2,7bbaa8668c87,
|
||||
q,1793911140077.9,1793911140079.5,1793911140087.9,202,,b8ca0dc5-6677-4ac0-beb3-01fd9eadc3b7,39e797bfa6f6,
|
||||
q,1793911140200.7,1793911140201.9,1793911140212.5,202,,67009462-0a7d-4782-808d-261ced886dec,7bbaa8668c87,
|
||||
q,1793911140323.4,1793911140324.6,1793911140334.9,202,,4a294fb2-57d4-4e5c-9001-da10612b4526,39e797bfa6f6,
|
||||
q,1793911140446.2,1793911140447.9,1793911140456.0,202,,617c66a1-522b-4982-bda4-bfbc46c611a2,7bbaa8668c87,
|
||||
q,1793911140568.9,1793911140573.7,1793911140582.4,202,,fc6ce00c-8e34-4b0b-9ac0-2a25159c1fa0,39e797bfa6f6,
|
||||
q,1793911140691.7,1793911140693.5,1793911140702.3,202,,01b6aab6-c47d-4fa5-900d-ad40c3ae65ea,7bbaa8668c87,
|
||||
q,1793911140814.5,1793911140818.4,1793911140827.3,202,,2734b012-52eb-46a7-95ed-cd175041b23b,39e797bfa6f6,
|
||||
q,1793911140937.2,1793911140939.8,1793911140948.4,202,,867fd8da-8c92-4b83-b9e6-ffebb77fab88,7bbaa8668c87,
|
||||
q,1793911141060.0,1793911141062.6,1793911141072.4,202,,108021b6-7938-427b-9043-833eef496b86,39e797bfa6f6,
|
||||
q,1793911141182.7,1793911141186.5,1793911141199.2,202,,81f898ff-af6a-4ab4-a481-a3e135b72565,7bbaa8668c87,
|
||||
q,1793911141305.5,1793911141308.3,1793911141316.1,202,,bd78ae5b-d66b-4068-83f6-c1c65cf97c3a,39e797bfa6f6,
|
||||
q,1793911141428.3,1793911141429.8,1793911141439.4,202,,242383e6-8d93-472a-8180-57eeeb959677,7bbaa8668c87,
|
||||
q,1793911141551.0,1793911141555.8,1793911141565.3,202,,b09dabc0-83d8-44ff-b103-87c891fc5795,39e797bfa6f6,
|
||||
q,1793911141673.8,1793911141674.8,1793911141685.2,202,,c10a7782-ec6f-45cd-814e-94d995cad140,7bbaa8668c87,
|
||||
q,1793911141796.5,1793911141797.5,1793911141806.6,202,,572ee5d2-49b8-4553-936f-3a6735b51aad,39e797bfa6f6,
|
||||
q,1793911141919.3,1793911141920.5,1793911141929.8,202,,08d01552-e929-4635-a704-837ae70aad61,7bbaa8668c87,
|
||||
q,1793911142042.1,1793911142044.9,1793911142055.1,202,,b6abcdfc-40a3-45df-a40f-aac9f8be215e,39e797bfa6f6,
|
||||
q,1793911142164.8,1793911142166.4,1793911142179.7,202,,a2c40270-b52e-40b8-8710-162ed9a6b39d,7bbaa8668c87,
|
||||
q,1793911142287.6,1793911142290.9,1793911142299.3,202,,ed395338-9e40-47ee-896c-92f77b943323,39e797bfa6f6,
|
||||
q,1793911142410.3,1793911142415.3,1793911142424.8,202,,76e2cf6f-ef2d-4c5c-8c42-a6a202668508,7bbaa8668c87,
|
||||
q,1793911142533.1,1793911142535.4,1793911142543.5,202,,ee315abb-c578-43b8-903c-7fb33cbdfd49,39e797bfa6f6,
|
||||
q,1793911142655.9,1793911142660.5,1793911142674.0,202,,ac4deeeb-adc0-4bba-9c8d-03d3cfaae0c6,7bbaa8668c87,
|
||||
q,1793911142778.6,1793911142780.7,1793911142789.3,202,,0729fb07-74a8-4279-b3ee-72487edcfbab,39e797bfa6f6,
|
||||
q,1793911142901.4,1793911142908.0,1793911142916.2,202,,1b974c8a-7d06-4619-bc72-d9f9dcd7a23c,7bbaa8668c87,
|
||||
q,1793911143024.1,1793911143031.0,1793911143040.4,202,,a4e5a9cb-2b0b-49c3-b1b1-37c091b9f829,39e797bfa6f6,
|
||||
q,1793911143146.9,1793911143154.8,1793911143169.3,202,,5e82c45b-7d7e-4e81-a87b-075c69986d76,7bbaa8668c87,
|
||||
q,1793911143269.7,1793911143273.4,1793911143281.9,202,,993469c4-a626-461f-8a66-07b50e325374,39e797bfa6f6,
|
||||
q,1793911143392.4,1793911143394.4,1793911143403.2,202,,277e7466-6343-4aea-a994-d72163505258,7bbaa8668c87,
|
||||
q,1793911143515.2,1793911143515.8,1793911143524.0,202,,5816ce76-c751-46d2-b035-fc9dcc1e182f,39e797bfa6f6,
|
||||
q,1793911143637.9,1793911143638.3,1793911143645.7,202,,a26435fd-456d-43e8-a05b-594a8a7b9c2b,7bbaa8668c87,
|
||||
q,1793911143760.7,1793911143762.5,1793911143771.9,202,,dcf81050-f4a9-47fe-b5eb-3872320cc402,39e797bfa6f6,
|
||||
q,1793911143883.5,1793911143887.8,1793911143895.8,202,,0b4a48bb-d5a6-44ce-9c92-e71b93507409,7bbaa8668c87,
|
||||
q,1793911144006.2,1793911144015.5,1793911144023.5,202,,38ffa05e-34eb-4a27-81f8-d024bd0c1afd,39e797bfa6f6,
|
||||
q,1793911144129.0,1793911144140.5,1793911144154.6,202,,6840d205-4418-4c91-9330-2a1b2904d8c4,7bbaa8668c87,
|
||||
q,1793911144251.7,1793911144254.9,1793911144263.1,202,,cc1cb2a5-d727-4c86-a219-977b58b13cdd,39e797bfa6f6,
|
||||
q,1793911144374.5,1793911144376.8,1793911144384.8,202,,29eeb95a-e960-4cc7-a3f9-bc024871d360,7bbaa8668c87,
|
||||
q,1793911144497.3,1793911144498.5,1793911144507.8,202,,b86758e1-0eae-422c-aa50-54151144d469,39e797bfa6f6,
|
||||
q,1793911144620.0,1793911144621.8,1793911144634.1,202,,08a5322c-0ecf-48f4-b1b5-c87e2d3654b4,7bbaa8668c87,
|
||||
q,1793911144742.8,1793911144743.9,1793911144754.1,202,,c369e776-8cad-46a7-8269-19dc1e3a58fa,39e797bfa6f6,
|
||||
q,1793911144865.6,1793911144867.5,1793911144875.8,202,,650ab703-cec0-4eb3-8669-dd88a5acfa09,7bbaa8668c87,
|
||||
q,1793911144988.3,1793911144989.5,1793911144998.4,202,,0d1dea89-02e0-4180-8a67-16b789dc6fc5,39e797bfa6f6,
|
||||
q,1793911145111.1,1793911145114.9,1793911145126.3,202,,6843b529-a0d6-4df6-ada4-d0a0b7b9d2a0,7bbaa8668c87,
|
||||
q,1793911145233.8,1793911145235.5,1793911145247.8,202,,c7eb5ba0-4c60-454f-9814-ecf789a58fc7,39e797bfa6f6,
|
||||
q,1793911145356.6,1793911145357.7,1793911145365.8,202,,1989f4ed-b3f8-4395-b44d-897b7d089767,7bbaa8668c87,
|
||||
q,1793911145479.4,1793911145480.9,1793911145492.7,202,,8fa1d7b5-516c-4420-a19f-00e6212075d6,39e797bfa6f6,
|
||||
q,1793911145602.1,1793911145604.5,1793911145613.0,202,,c8a244f9-cf88-4f07-b095-ae0df861d4d9,7bbaa8668c87,
|
||||
q,1793911145724.9,1793911145725.6,1793911145735.5,202,,8558ead2-5910-4419-a1e7-4b3f689f8e6b,39e797bfa6f6,
|
||||
q,1793911145847.6,1793911145849.5,1793911145858.6,202,,201f44c6-ca33-4a6e-8012-e2d586bc61fe,7bbaa8668c87,
|
||||
q,1793911145970.4,1793911145971.5,1793911145980.3,202,,372b174f-a043-458b-85f8-80675c6f519a,39e797bfa6f6,
|
||||
q,1793911146093.2,1793911146094.6,1793911146103.4,202,,a2e0ecd9-48cf-4e2c-b89e-e6be9477b683,7bbaa8668c87,
|
||||
q,1793911146215.9,1793911146222.1,1793911146261.0,202,,bec9648f-013f-4ed6-a218-e093679287c2,39e797bfa6f6,
|
||||
q,1793911146338.7,1793911146340.0,1793911146348.6,202,,48ac9145-442e-465a-80da-c0eadfb3057e,7bbaa8668c87,
|
||||
q,1793911146461.4,1793911146462.7,1793911146476.5,202,,6fed5ce2-e775-4eae-9a2f-13fbf8dd4d02,39e797bfa6f6,
|
||||
q,1793911146584.2,1793911146585.5,1793911146593.5,202,,8dbcb477-015b-4e28-b97e-d75dd4f1cd2a,7bbaa8668c87,
|
||||
q,1793911146707.0,1793911146709.1,1793911146722.5,202,,4261726b-1bbc-48fd-a4d2-6d32780b0787,39e797bfa6f6,
|
||||
q,1793911146829.7,1793911146831.5,1793911146840.7,202,,48ccaec0-c5e9-4b44-a044-0f71af70b80e,7bbaa8668c87,
|
||||
q,1793911146952.5,1793911146954.0,1793911146962.5,202,,616c57be-e1ea-456e-a214-0170407d4f2a,39e797bfa6f6,
|
||||
q,1793911147075.2,1793911147075.8,1793911147083.7,202,,21deae8e-560d-4f56-9b1b-181f0219092e,7bbaa8668c87,
|
||||
q,1793911147198.0,1793911147198.9,1793911147207.8,202,,84cf8cd5-2cab-4087-a1c7-c5fdf12d69e7,39e797bfa6f6,
|
||||
q,1793911147320.8,1793911147321.8,1793911147330.2,202,,4220c0b0-6656-4c22-8253-457b74684e5c,7bbaa8668c87,
|
||||
q,1793911147443.5,1793911147444.8,1793911147452.9,202,,ad4da290-bde8-4d27-9d80-36c27aa90714,39e797bfa6f6,
|
||||
q,1793911147566.3,1793911147567.8,1793911147576.5,202,,968d2397-72cc-49a1-bdee-f91455627d92,7bbaa8668c87,
|
||||
q,1793911147689.0,1793911147690.6,1793911147701.8,202,,a656feb6-99d4-4905-bff0-6b938e3ee6bf,39e797bfa6f6,
|
||||
q,1793911147811.8,1793911147814.4,1793911147824.1,202,,555b4e9a-254f-4587-b14c-028aefecdb9b,7bbaa8668c87,
|
||||
q,1793911147934.6,1793911147935.5,1793911147944.5,202,,ec851382-7059-4258-b29f-729a7cf77813,39e797bfa6f6,
|
||||
q,1793911148057.3,1793911148061.4,1793911148070.8,202,,7f8c3e19-e42d-4efe-aad3-1a7e0f2bda7b,7bbaa8668c87,
|
||||
q,1793911148180.1,1793911148182.3,1793911148192.6,202,,235eb93b-6d31-44a7-954c-8cc386910b69,39e797bfa6f6,
|
||||
q,1793911148302.8,1793911148305.2,1793911148316.2,202,,2f0769a8-956a-4e55-b28d-9842f0ff3841,7bbaa8668c87,
|
||||
q,1793911148425.6,1793911148426.9,1793911148435.0,202,,5dce86e6-85aa-40e5-a548-6c851933b55f,39e797bfa6f6,
|
||||
q,1793911148548.4,1793911148550.5,1793911148559.5,202,,12467512-2056-4c40-89ae-e59ba431577e,7bbaa8668c87,
|
||||
q,1793911148671.1,1793911148676.6,1793911148686.3,202,,17f0b306-5378-4cef-9343-9e704aa8a55e,39e797bfa6f6,
|
||||
q,1793911148793.9,1793911148799.4,1793911148817.8,202,,5b6b36d3-639d-499b-a1ff-d08592f3e7a5,7bbaa8668c87,
|
||||
q,1793911148916.6,1793911148923.2,1793911148931.7,202,,663de52b-7997-4296-be43-4a6baaa587ae,39e797bfa6f6,
|
||||
q,1793911149039.4,1793911149048.5,1793911149056.9,202,,20d07d40-aafc-4f93-8cbb-fb9db20fcfd9,7bbaa8668c87,
|
||||
q,1793911149162.2,1793911149163.9,1793911149172.0,202,,ea85e06c-b5ad-4ba0-917b-7e443e0ca2f9,39e797bfa6f6,
|
||||
q,1793911149284.9,1793911149295.5,1793911149311.7,202,,07aa1761-b0be-43f6-94ca-ec4af9e5c8ec,7bbaa8668c87,
|
||||
q,1793911149407.7,1793911149408.7,1793911149417.6,202,,12677ba6-acd2-45ca-a417-09f7875dace6,39e797bfa6f6,
|
||||
q,1793911149530.4,1793911149534.3,1793911149543.5,202,,e74812ce-e686-421d-bdf6-f0658cdf92a4,7bbaa8668c87,
|
||||
q,1793911149653.2,1793911149661.9,1793911149669.7,202,,6a1a1dcb-c3d8-4d0b-a071-e8a68751f747,39e797bfa6f6,
|
||||
q,1793911149776.0,1793911149781.1,1793911149788.8,202,,0d56fc00-7fb3-4762-b5bb-bd6b82d7a4bc,7bbaa8668c87,
|
||||
q,1793911149898.7,1793911149900.4,1793911149908.6,202,,f7220d80-0e5a-4bae-9a15-052bdbc4937e,39e797bfa6f6,
|
||||
q,1793911150021.5,1793911150028.7,1793911150037.1,202,,c7ce841a-cf2b-4827-b73b-bff06d9d25a2,7bbaa8668c87,
|
||||
q,1793911150144.2,1793911150147.0,1793911150154.4,202,,e2b61a2a-f6fb-47d2-bf40-6ac6e0118915,39e797bfa6f6,
|
||||
q,1793911150267.0,1793911150278.5,1793911150292.4,202,,2933782c-1031-4f8f-ad14-a4c47907b961,7bbaa8668c87,
|
||||
q,1793911150389.8,1793911150394.2,1793911150402.7,202,,8af9ff3d-ce46-40bb-bbcf-15ee36b06599,39e797bfa6f6,
|
||||
q,1793911150512.5,1793911150513.7,1793911150524.9,202,,c12065b7-115e-4dfc-97f9-680908cba8ff,7bbaa8668c87,
|
||||
q,1793911150635.3,1793911150637.0,1793911150646.7,202,,36973438-1476-4f20-8869-307a110014ea,39e797bfa6f6,
|
||||
q,1793911150758.0,1793911150769.6,1793911150782.7,202,,fb1ff674-9008-4df8-9bb6-26b2db767bce,7bbaa8668c87,
|
||||
q,1793911150880.8,1793911150882.5,1793911150899.1,202,,dd1812e4-c653-43cc-9b1c-85847d5973c6,39e797bfa6f6,
|
||||
q,1793911151003.6,1793911151004.5,1793911151014.2,202,,246d9517-6888-46df-b581-8a3d0d294be3,7bbaa8668c87,
|
||||
q,1793911151126.3,1793911151128.7,1793911151142.1,202,,ce1c5448-c4bc-45b2-b990-19b3de942dcf,39e797bfa6f6,
|
||||
q,1793911151249.1,1793911151249.5,1793911151258.4,202,,3b903406-8fcf-4a91-9990-880b62b22c10,7bbaa8668c87,
|
||||
q,1793911151371.9,1793911151379.2,1793911151388.0,202,,7e5524c3-c0d4-487a-9cd9-6cec6a16dae7,39e797bfa6f6,
|
||||
q,1793911151494.6,1793911151496.9,1793911151506.9,202,,59f712e4-e131-4588-9651-1be79ecff5ed,7bbaa8668c87,
|
||||
q,1793911151617.4,1793911151618.5,1793911151627.0,202,,3a1e6dab-afb6-470c-818f-98203d7f9191,39e797bfa6f6,
|
||||
q,1793911151740.1,1793911151742.9,1793911151753.8,202,,e4d99caf-aa3e-4b97-92e9-fb3e96c6bdae,7bbaa8668c87,
|
||||
q,1793911151862.9,1793911151864.5,1793911151877.1,202,,46232b97-a87a-46ee-9907-395c0eecb98f,39e797bfa6f6,
|
||||
q,1793911151985.7,1793911151986.9,1793911151995.9,202,,cc35cc89-5d92-4b41-bea1-e7323c1a514a,7bbaa8668c87,
|
||||
q,1793911152108.4,1793911152109.7,1793911152118.5,202,,d0e99a6c-e6f9-4693-9052-922cd82a10a8,39e797bfa6f6,
|
||||
q,1793911152231.2,1793911152231.7,1793911152240.1,202,,3a200b07-7717-41f8-b72f-3328d7075c08,7bbaa8668c87,
|
||||
q,1793911152353.9,1793911152354.8,1793911152369.3,202,,427c6c7e-6e69-47cd-9245-95d219f15f06,39e797bfa6f6,
|
||||
q,1793911152476.7,1793911152478.7,1793911152489.5,202,,cf225a82-0c61-4b17-afd6-e47603b3ee02,7bbaa8668c87,
|
||||
q,1793911152599.5,1793911152602.5,1793911152611.3,202,,52b77c5c-73be-4e80-841e-7210f79ed3b1,39e797bfa6f6,
|
||||
q,1793911152722.2,1793911152724.9,1793911152732.9,202,,b9371f50-e5ac-47e3-b59d-83ddd088e71c,7bbaa8668c87,
|
||||
q,1793911152845.0,1793911152846.9,1793911152854.7,202,,f91d7e98-481b-4b64-aecd-abb5a647554d,39e797bfa6f6,
|
||||
q,1793911152967.7,1793911152969.7,1793911152978.9,202,,de882511-a3f8-4bb1-a75e-25a89b48a248,7bbaa8668c87,
|
||||
q,1793911153090.5,1793911153095.7,1793911153104.2,202,,706e1715-5adb-4a6e-bccf-5ad89b55789c,39e797bfa6f6,
|
||||
q,1793911153213.3,1793911153215.3,1793911153223.1,202,,75dbc8b7-15b2-4b12-842e-617f5cb0229d,7bbaa8668c87,
|
||||
q,1793911153336.0,1793911153339.0,1793911153353.1,202,,09b928a3-a9cf-4b69-8eae-224927ef6622,39e797bfa6f6,
|
||||
q,1793911153458.8,1793911153463.9,1793911153472.6,202,,a2f48af5-156f-4c92-9e67-1605e615893a,7bbaa8668c87,
|
||||
q,1793911153581.5,1793911153582.5,1793911153590.6,202,,7cfa17ab-e8b8-4843-906b-339435251bb8,39e797bfa6f6,
|
||||
q,1793911153704.3,1793911153706.3,1793911153713.7,202,,735594fb-99bd-4551-a3d5-20db31041013,7bbaa8668c87,
|
||||
q,1793911153827.1,1793911153829.7,1793911153841.7,202,,a8416c81-7a70-46a5-bc0e-2c61a5ed033e,39e797bfa6f6,
|
||||
q,1793911153949.8,1793911153952.5,1793911153961.1,202,,07359b4b-be66-4d70-b4e9-a1b3e46fb83a,7bbaa8668c87,
|
||||
q,1793911154072.6,1793911154075.9,1793911154084.2,202,,d3900874-9d0f-4c24-8b98-77e3cc960d4d,39e797bfa6f6,
|
||||
q,1793911154195.3,1793911154196.5,1793911154204.7,202,,5ab221b8-1905-406c-afd1-f7a37747bcd5,7bbaa8668c87,
|
||||
q,1793911154318.1,1793911154326.6,1793911154338.9,202,,3fc67441-9997-44d6-b14c-2bb303f8ecc9,39e797bfa6f6,
|
||||
q,1793911154440.9,1793911154451.5,1793911154460.9,202,,5f0f2fa1-ecc1-4a25-ae09-1c91319556d7,7bbaa8668c87,
|
||||
q,1793911154563.6,1793911154565.8,1793911154574.3,202,,c86a9f6f-0590-476f-bd22-39b902eabcd9,39e797bfa6f6,
|
||||
q,1793911154686.4,1793911154698.5,1793911154708.4,202,,2dc4f475-514f-4cfc-9d8e-761e98ca68b0,7bbaa8668c87,
|
||||
q,1793911154809.1,1793911154813.6,1793911154824.9,202,,a3c19b8b-674e-42f4-935d-95fc78581e66,39e797bfa6f6,
|
||||
q,1793911154931.9,1793911154933.0,1793911154943.8,202,,4b85ba86-c9d0-4447-a7fb-12eb4b49513f,7bbaa8668c87,
|
||||
q,1793911155054.7,1793911155056.5,1793911155065.3,202,,83cfb186-62bf-4888-b126-081e4a5de8fb,39e797bfa6f6,
|
||||
q,1793911155177.4,1793911155180.5,1793911155188.9,202,,9bd2c45b-1aca-4f9c-b7db-95acf62e1d48,7bbaa8668c87,
|
||||
q,1793911155300.2,1793911155305.6,1793911155316.0,202,,b3418060-c991-4570-800d-0010ff7754d3,39e797bfa6f6,
|
||||
q,1793911155422.9,1793911155426.8,1793911155445.3,202,,d960dcc7-15e4-41c3-9a80-3072786bc6be,7bbaa8668c87,
|
||||
q,1793911155545.7,1793911155546.9,1793911155555.8,202,,d5692c9d-40c1-43d2-8c74-dae55a104250,39e797bfa6f6,
|
||||
q,1793911155668.5,1793911155670.0,1793911155678.4,202,,35cb4a02-0585-4f50-82b2-3bf1179898a1,7bbaa8668c87,
|
||||
q,1793911155791.2,1793911155800.3,1793911155808.9,202,,2ea86153-f037-462e-b9e3-baeacd191863,39e797bfa6f6,
|
||||
q,1793911155914.0,1793911155916.1,1793911155929.6,202,,85f09b34-fe5d-417c-9f01-d30111673805,7bbaa8668c87,
|
||||
q,1793911156036.7,1793911156038.8,1793911156047.7,202,,b17fd80e-5c8d-48e2-80bd-e26b5da64f3d,39e797bfa6f6,
|
||||
q,1793911156159.5,1793911156163.4,1793911156171.8,202,,9017617e-d775-40e6-ac73-5ccf1f94bead,7bbaa8668c87,
|
||||
q,1793911156282.3,1793911156283.8,1793911156292.3,202,,b2dd20eb-3fa3-4214-bb52-3d2d280ea7f8,39e797bfa6f6,
|
||||
q,1793911156405.0,1793911156406.1,1793911156414.2,202,,f9bdd52c-f4e1-497a-bda3-39f49c4efd3d,7bbaa8668c87,
|
||||
q,1793911156527.8,1793911156529.7,1793911156540.8,202,,c682e3a4-53b1-4089-9754-08b5c75fe21c,39e797bfa6f6,
|
||||
q,1793911156650.5,1793911156653.0,1793911156663.4,202,,5f8dd6b7-361b-4ab1-bd70-c5efec5f4ff3,7bbaa8668c87,
|
||||
q,1793911156773.3,1793911156776.2,1793911156783.7,202,,c67b3703-c491-4da2-b82e-4bf4c144f23b,39e797bfa6f6,
|
||||
q,1793911156896.1,1793911156902.6,1793911156913.4,202,,d08bff6b-df66-41af-8fe4-dbbefbc66adb,7bbaa8668c87,
|
||||
q,1793911157018.8,1793911157020.5,1793911157028.8,202,,8858eb7f-a698-4763-97c9-266a08d7692f,39e797bfa6f6,
|
||||
q,1793911157141.6,1793911157142.5,1793911157169.8,202,,875bc9d6-4c76-46ff-97a5-bdd63dcd35a5,7bbaa8668c87,
|
||||
q,1793911157264.4,1793911157266.0,1793911157272.9,202,,880b4a23-f9a3-40fa-8969-34904eb7fe6c,39e797bfa6f6,
|
||||
q,1793911157387.1,1793911157389.8,1793911157403.0,202,,f6a6fbaa-1bc0-4910-a962-13c31fa4dfbf,7bbaa8668c87,
|
||||
q,1793911157509.9,1793911157510.6,1793911157521.8,202,,8f7a9eab-9a8b-4607-a0ce-8cbb6e536428,39e797bfa6f6,
|
||||
q,1793911157632.6,1793911157634.3,1793911157642.8,202,,21eb32d4-324a-49b8-b194-d33627ffa11e,7bbaa8668c87,
|
||||
q,1793911157755.4,1793911157756.5,1793911157766.6,202,,c34fa70c-0e45-41e1-b498-0d3bc3249c16,39e797bfa6f6,
|
||||
q,1793911157878.2,1793911157882.4,1793911157894.0,202,,8535c067-c36b-4129-b126-2ca29b6b07f9,7bbaa8668c87,
|
||||
q,1793911158000.9,1793911158001.5,1793911158011.1,202,,be970120-d405-41db-afba-2255c8836115,39e797bfa6f6,
|
||||
q,1793911158123.7,1793911158126.4,1793911158135.2,202,,bca29bcf-9455-4185-bf13-80782af2bd76,7bbaa8668c87,
|
||||
q,1793911158246.4,1793911158248.5,1793911158257.1,202,,2cdd75e1-10c8-4541-b3c3-0f762e5aab16,39e797bfa6f6,
|
||||
q,1793911158369.2,1793911158371.4,1793911158381.5,202,,c2ffd8cb-bef1-474a-a445-4edc14a4bda2,7bbaa8668c87,
|
||||
q,1793911158492.0,1793911158493.6,1793911158509.7,202,,199e129b-156b-40a3-bca3-31733e5f497e,39e797bfa6f6,
|
||||
q,1793911158614.7,1793911158617.6,1793911158626.6,202,,92d10033-4777-4c08-b7b0-ebf1e1e6db34,7bbaa8668c87,
|
||||
q,1793911158737.5,1793911158739.4,1793911158748.7,202,,af5c05e4-304b-4a22-9a5e-b257c638f301,39e797bfa6f6,
|
||||
q,1793911158860.2,1793911158864.9,1793911158874.4,202,,e53ef1ee-590a-4842-beca-6dc2678cc4d6,7bbaa8668c87,
|
||||
q,1793911158983.0,1793911158985.4,1793911159002.9,202,,5d10c2a8-6fa7-459d-9050-65d8d13c3271,39e797bfa6f6,
|
||||
q,1793911159105.8,1793911159108.0,1793911159117.7,202,,19066012-2d47-4ba8-a70e-f733d760cc8b,7bbaa8668c87,
|
||||
q,1793911159228.5,1793911159230.9,1793911159240.0,202,,944e6d2d-8d58-47ad-af0f-cf68f04d9264,39e797bfa6f6,
|
||||
q,1793911159351.3,1793911159353.9,1793911159362.1,202,,e4f31f03-45fe-4f86-87ed-b7adb531d8c3,7bbaa8668c87,
|
||||
q,1793911159474.0,1793911159477.2,1793911159486.5,202,,35a07be8-f03d-45dd-a977-59158756b86e,39e797bfa6f6,
|
||||
q,1793911159596.8,1793911159601.0,1793911159610.0,202,,91455ffa-d1fe-447c-abd3-70f3b2b181a3,7bbaa8668c87,
|
||||
q,1793911159719.6,1793911159720.5,1793911159729.2,202,,93349d8a-1986-4ee0-9d81-8bc1fd5020b9,39e797bfa6f6,
|
||||
q,1793911159842.3,1793911159847.0,1793911159855.0,202,,a2de623f-7f0c-4ced-8613-52c51fc07f51,7bbaa8668c87,
|
||||
q,1793911159965.1,1793911159971.2,1793911159985.3,202,,4ab5b35f-779a-4f42-a5b6-e6707338d9ed,39e797bfa6f6,
|
||||
q,1793911160087.8,1793911160091.5,1793911160099.7,202,,bb13ca33-76c2-4e3c-b180-146955ae3b1c,7bbaa8668c87,
|
||||
q,1793911160210.6,1793911160212.9,1793911160221.1,202,,3fce7218-9099-4077-8a7d-09208aaedfc4,39e797bfa6f6,
|
||||
q,1793911160333.4,1793911160334.6,1793911160341.7,202,,4f6ae4fc-3993-40d4-a140-c00c59dc91e7,7bbaa8668c87,
|
||||
q,1793911160456.1,1793911160467.7,1793911160480.7,202,,72c7d6af-6445-4c27-a9bf-1be46640be2c,39e797bfa6f6,
|
||||
q,1793911160578.9,1793911160584.4,1793911160594.5,202,,7bf6daa8-8b51-4a07-871e-8e8c9e76bb14,7bbaa8668c87,
|
||||
q,1793911160701.6,1793911160703.7,1793911160712.3,202,,4829ffd0-a994-4b85-8e50-778d44e6cb45,39e797bfa6f6,
|
||||
q,1793911160824.4,1793911160826.9,1793911160834.9,202,,c77acc8b-f5c1-4ae5-9695-87aaebd5c687,7bbaa8668c87,
|
||||
q,1793911160947.2,1793911160949.8,1793911160962.1,202,,b5da82fb-2cbf-4563-9b7e-10aa61935d49,39e797bfa6f6,
|
||||
q,1793911161069.9,1793911161070.5,1793911161083.9,202,,13d260e5-7994-45f4-8477-b33940055b27,7bbaa8668c87,
|
||||
q,1793911161192.7,1793911161198.6,1793911161210.2,202,,f5aa9299-44bc-48ec-accc-db63606fc262,39e797bfa6f6,
|
||||
q,1793911161315.4,1793911161317.4,1793911161326.7,202,,9a35bb11-2f5a-430b-af8f-2b2186dd1957,7bbaa8668c87,
|
||||
q,1793911161438.2,1793911161449.5,1793911161457.7,202,,fd61bc04-089d-4e7f-9ef9-28547b0b1670,39e797bfa6f6,
|
||||
q,1793911161561.0,1793911161563.5,1793911161571.8,202,,2dee559d-1b97-426d-ad78-96be758b6914,7bbaa8668c87,
|
||||
q,1793911161683.7,1793911161684.5,1793911161694.9,202,,a3b46f4e-2462-4ef4-9a24-13905982596f,39e797bfa6f6,
|
||||
q,1793911161806.5,1793911161807.7,1793911161818.8,202,,feaf9584-e54b-4035-ae9c-7a96ac504bf9,7bbaa8668c87,
|
||||
q,1793911161929.2,1793911161930.5,1793911161939.0,202,,f298829d-7eeb-4616-b37f-031e108afe9a,39e797bfa6f6,
|
||||
q,1793911162052.0,1793911162052.6,1793911162062.4,202,,cbbec792-00ee-40c0-b349-279086568c3d,7bbaa8668c87,
|
||||
q,1793911162174.8,1793911162176.5,1793911162184.8,202,,4c092398-cf3c-40dc-a6bf-224b4ba7a91e,39e797bfa6f6,
|
||||
q,1793911162297.5,1793911162298.9,1793911162307.6,202,,cb5a7dda-643e-48f2-a121-f1ff90752892,7bbaa8668c87,
|
||||
q,1793911162420.3,1793911162423.8,1793911162433.5,202,,073f14b2-8f28-43ff-b21e-29fa2e8b94c5,39e797bfa6f6,
|
||||
q,1793911162543.0,1793911162545.2,1793911162555.7,202,,4fc71429-6f5c-4318-bcb2-2addab31f27d,7bbaa8668c87,
|
||||
q,1793911162665.8,1793911162670.6,1793911162678.8,202,,d3871567-0f6a-4d41-b600-7d690070d2ad,39e797bfa6f6,
|
||||
q,1793911162788.6,1793911162791.4,1793911162800.3,202,,7cce7b35-a6ba-45f5-a4fa-9f904263fb2f,7bbaa8668c87,
|
||||
q,1793911162911.3,1793911162914.2,1793911162922.5,202,,b6b9f548-bcb9-4001-8d6e-81df2424e118,39e797bfa6f6,
|
||||
q,1793911163034.1,1793911163037.0,1793911163045.8,202,,fd115fdb-8f58-47d5-96cd-1c1182bb0fd5,7bbaa8668c87,
|
||||
q,1793911163156.8,1793911163160.4,1793911163169.2,202,,747954cf-b9f2-41b4-9a81-55c77ec1d50c,39e797bfa6f6,
|
||||
q,1793911163279.6,1793911163280.8,1793911163289.0,202,,9835db20-7631-4565-b027-2c13741cc552,7bbaa8668c87,
|
||||
q,1793911163402.4,1793911163404.0,1793911163411.9,202,,374d6b2c-504b-4654-8eed-7ad19cedbd57,39e797bfa6f6,
|
||||
q,1793911163525.1,1793911163527.5,1793911163536.6,202,,fb570ba9-0131-4bb4-a8fb-2520b564827e,7bbaa8668c87,
|
||||
q,1793911163647.9,1793911163649.5,1793911163658.0,202,,126b7ca4-23ad-4af2-b233-56f3d8fd24d2,39e797bfa6f6,
|
||||
q,1793911163770.7,1793911163772.0,1793911163780.1,202,,a71f57e0-e8b3-4ccb-bf60-2aaf2616d85a,7bbaa8668c87,
|
||||
q,1793911163893.4,1793911163895.3,1793911163902.9,202,,9c4cf9ee-209f-419a-9857-4f8eaf07ad67,39e797bfa6f6,
|
||||
q,1793911164016.2,1793911164019.9,1793911164032.2,202,,544495c3-693f-4656-8320-f46b5e9a110b,7bbaa8668c87,
|
||||
q,1793911164138.9,1793911164139.9,1793911164148.9,202,,75cdd6a9-074d-4540-bee1-603bf8172c0b,39e797bfa6f6,
|
||||
q,1793911164261.7,1793911164262.5,1793911164270.5,202,,7c919b07-abdb-4b01-b61a-0ba93c914766,7bbaa8668c87,
|
||||
q,1793911164384.5,1793911164385.5,1793911164393.4,202,,1d1f9a47-2722-404d-b9d7-4f38b80c3edc,39e797bfa6f6,
|
||||
q,1793911164507.2,1793911164508.9,1793911164519.4,202,,246b37a6-a453-4c37-b277-2ba0ac036aa6,7bbaa8668c87,
|
||||
q,1793911164630.0,1793911164632.9,1793911164641.1,202,,c781b515-54cc-4cc4-9e74-c11a2bd811bd,39e797bfa6f6,
|
||||
q,1793911164752.7,1793911164753.8,1793911164761.8,202,,336cdad4-6c99-45cc-824f-0785de639740,7bbaa8668c87,
|
||||
q,1793911164875.5,1793911164879.5,1793911164888.5,202,,8ae99f64-8b14-4dcc-a6da-d6ae14e48880,39e797bfa6f6,
|
||||
q,1793911164998.3,1793911165000.3,1793911165011.0,202,,ab6e9400-d34f-4d65-a0aa-37244d00546e,7bbaa8668c87,
|
||||
q,1793911165121.0,1793911165121.6,1793911165134.7,202,,f134ac4a-3ea0-4618-8b0c-c9af7a4591d7,39e797bfa6f6,
|
||||
q,1793911165243.8,1793911165246.9,1793911165256.6,202,,7b5f4a82-8084-4c9d-b839-29fdadfd38b3,7bbaa8668c87,
|
||||
q,1793911165366.5,1793911165367.5,1793911165376.1,202,,3cbb80da-bda6-4b4e-80ae-a368c6f33eb6,39e797bfa6f6,
|
||||
q,1793911165489.3,1793911165490.5,1793911165500.9,202,,26169ec2-8303-41b7-999b-fe6851ca3d94,7bbaa8668c87,
|
||||
q,1793911165612.1,1793911165623.0,1793911165636.5,202,,a21a53f6-8614-49b4-97b0-5999d8055542,39e797bfa6f6,
|
||||
q,1793911165734.8,1793911165741.3,1793911165750.2,202,,3578b4d9-8fe4-44fe-931f-d739764cf1fe,7bbaa8668c87,
|
||||
q,1793911165857.6,1793911165862.5,1793911165871.7,202,,66e980a4-302f-43ff-86ea-7c2135229a0a,39e797bfa6f6,
|
||||
q,1793911165980.3,1793911165984.4,1793911165993.2,202,,62252cc2-829a-4d37-aa33-a56b93ab0ccc,7bbaa8668c87,
|
||||
q,1793911166103.1,1793911166103.9,1793911166117.4,202,,81d70cc7-2582-4296-bf45-7f9b2af9b7fd,39e797bfa6f6,
|
||||
q,1793911166225.9,1793911166229.9,1793911166238.8,202,,35d4a45e-9a3b-4623-b753-28a9d5c511e1,7bbaa8668c87,
|
||||
q,1793911166348.6,1793911166351.5,1793911166359.8,202,,c6ccecc4-736a-41ed-a576-fbda6c49e7cd,39e797bfa6f6,
|
||||
q,1793911166471.4,1793911166474.5,1793911166483.1,202,,cb2fbb40-c4fa-4c64-8403-cf246da3c28c,7bbaa8668c87,
|
||||
q,1793911166594.1,1793911166597.1,1793911166605.5,202,,81bec245-d80f-4081-b5ab-dd844e2b100a,39e797bfa6f6,
|
||||
q,1793911166716.9,1793911166718.6,1793911166727.0,202,,a0e24c98-f0c3-46b5-92e6-c672a2f7a506,7bbaa8668c87,
|
||||
q,1793911166839.7,1793911166848.3,1793911166856.7,202,,d1ac21b7-c185-4ced-8cd5-15f079491572,39e797bfa6f6,
|
||||
q,1793911166962.4,1793911166971.0,1793911166978.5,202,,cffb3c26-8f4a-4409-a7a5-3247a14c3aa1,7bbaa8668c87,
|
||||
q,1793911167085.2,1793911167086.9,1793911167097.0,202,,8e063440-2f82-4b72-a8f5-c1cb9be467a7,39e797bfa6f6,
|
||||
q,1793911167207.9,1793911167208.5,1793911167219.2,202,,10bc1d66-7a73-47e7-83d2-cca8afe6a644,7bbaa8668c87,
|
||||
q,1793911167330.7,1793911167331.5,1793911167342.1,202,,7f4248b8-2af7-4bfe-a66b-6c3e57b0e4b8,39e797bfa6f6,
|
||||
q,1793911167453.5,1793911167454.6,1793911167462.8,202,,5fceaddf-5195-47e2-8cae-2ed1d9e8d161,7bbaa8668c87,
|
||||
q,1793911167576.2,1793911167583.3,1793911167592.0,202,,da750cdd-7dd6-4cd2-8120-e3942871c313,39e797bfa6f6,
|
||||
q,1793911167699.0,1793911167704.5,1793911167712.9,202,,6a171040-c9c9-43d6-b1bb-010b8f356268,7bbaa8668c87,
|
||||
q,1793911167821.7,1793911167822.5,1793911167830.3,202,,05824239-0a0a-4e9d-91ee-2b6c1019ec77,39e797bfa6f6,
|
||||
q,1793911167944.5,1793911167946.5,1793911167957.2,202,,f837afa3-3353-4f58-81ba-aca3654521a9,7bbaa8668c87,
|
||||
q,1793911168067.3,1793911168078.6,1793911168112.7,202,,f0d13112-377a-4ec3-bf5f-f5e9338fe288,39e797bfa6f6,
|
||||
q,1793911168190.0,1793911168194.7,1793911168203.9,202,,ead68dd3-ea6d-4b4e-8c56-551f620881e2,7bbaa8668c87,
|
||||
q,1793911168312.8,1793911168313.8,1793911168322.5,202,,0a02f7ab-fbc2-4aae-86c4-adc0c280d9bf,39e797bfa6f6,
|
||||
q,1793911168435.5,1793911168436.8,1793911168445.2,202,,aaeab563-ce28-4680-a9f2-8a5f2fb8f8c5,7bbaa8668c87,
|
||||
q,1793911168558.3,1793911168560.1,1793911168568.9,202,,5e6570d9-55fa-4079-aa58-f15209333edb,39e797bfa6f6,
|
||||
q,1793911168681.1,1793911168682.5,1793911168693.5,202,,5d0cbe47-72c3-4447-977c-a60a1881d973,7bbaa8668c87,
|
||||
q,1793911168803.8,1793911168807.0,1793911168815.2,202,,d84a2cf7-11f0-4494-b070-378402a06a53,39e797bfa6f6,
|
||||
q,1793911168926.6,1793911168927.5,1793911168936.1,202,,71476548-6bf6-4747-aedc-16299f501afa,7bbaa8668c87,
|
||||
q,1793911169049.3,1793911169052.7,1793911169062.6,202,,dedfac9d-981a-4b8c-a9dd-bf1bb9f2d7a2,39e797bfa6f6,
|
||||
q,1793911169172.1,1793911169176.0,1793911169191.8,202,,cd2acec2-e12a-4fd6-8697-28ad3dbec21e,7bbaa8668c87,
|
||||
q,1793911169294.9,1793911169298.0,1793911169306.8,202,,3943d3c7-ba46-4949-af4a-09961c87fba6,39e797bfa6f6,
|
||||
q,1793911169417.6,1793911169421.5,1793911169429.9,202,,7a5d6d87-e248-42ec-baa4-c2c28a0bc468,7bbaa8668c87,
|
||||
q,1793911169540.4,1793911169542.6,1793911169550.7,202,,2b00324b-dfb1-40ed-83c9-109e5fbcf6d2,39e797bfa6f6,
|
||||
q,1793911169663.1,1793911169665.4,1793911169678.5,202,,8b5b97ac-488b-4a08-a1ba-e6db99d4ab1a,7bbaa8668c87,
|
||||
q,1793911169785.9,1793911169786.7,1793911169796.0,202,,43558fa7-0c49-4bf8-92a0-cd2bf519aad2,39e797bfa6f6,
|
||||
q,1793911169908.7,1793911169910.5,1793911169919.0,202,,7fa22608-203e-40da-b200-7ed0286122dc,7bbaa8668c87,
|
||||
q,1793911170031.4,1793911170033.5,1793911170041.6,202,,4ee98d0d-ba81-43e5-855f-97c1962abb57,39e797bfa6f6,
|
||||
q,1793911170154.2,1793911170158.5,1793911170165.9,202,,072e12fd-6ac6-4870-a6f9-6fd7f676e7e9,7bbaa8668c87,
|
||||
q,1793911170277.0,1793911170279.5,1793911170288.0,202,,f3c0dc2f-6cce-4fa9-90b2-d79c7c32abb5,39e797bfa6f6,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 8.14595,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793911130.38,
|
||||
"end": 1793911170.4,
|
||||
"finished": 1793911170.287,
|
||||
"scheduled": 326,
|
||||
"sent": 326,
|
||||
"accepted": 326,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 326
|
||||
},
|
||||
"upload_p50_ms": 8.84,
|
||||
"upload_p95_ms": 14.52,
|
||||
"upload_max_ms": 38.87,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.262,
|
||||
"log": "/tmp/c2p32-s-r1-steady.log"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,40 @@
|
||||
q,1793911280582.0,1793911280584.2,1793911280688.7,202,,49549fee-5b11-4479-b836-c487c6637b71,7bbaa8668c87,
|
||||
q,1793911280832.0,1793911280835.9,1793911280926.3,202,,65d1d867-76a5-4b5e-a0ff-1f38dd65b67e,39e797bfa6f6,
|
||||
q,1793911281082.0,1793911281083.6,1793911281162.6,202,,a491897b-510b-4155-b5f8-b205f4df32ba,7bbaa8668c87,
|
||||
q,1793911281332.0,1793911281333.5,1793911281407.7,202,,022f5f01-3536-4bae-a7eb-914d34af5e8c,39e797bfa6f6,
|
||||
q,1793911281582.0,1793911281586.8,1793911281679.0,202,,da181c03-4baa-49de-a8db-2e240f879487,7bbaa8668c87,
|
||||
q,1793911281832.0,1793911281833.5,1793911281920.8,202,,dbe7f009-fbe1-44af-af52-dee7770c5f03,39e797bfa6f6,
|
||||
q,1793911282582.0,1793911282587.0,1793911282967.9,202,,0721fcfd-4662-4ac4-a456-b0729067e2c0,7bbaa8668c87,
|
||||
q,1793911282082.0,1793911282086.0,1793911282984.1,202,,f9e286a4-653c-4af9-9e53-8fd85797cc6f,7bbaa8668c87,
|
||||
q,1793911282332.0,1793911282342.0,1793911283013.0,202,,5d9f3e43-d067-43ec-b2c3-7750c2f4625d,39e797bfa6f6,
|
||||
q,1793911282832.0,1793911282835.4,1793911283016.4,202,,5d0d049a-981f-4ac7-974a-6fe8551555e9,39e797bfa6f6,
|
||||
q,1793911283082.0,1793911283083.6,1793911283183.0,202,,c4f79ddb-bef8-43ac-a6ce-8430426b40e5,7bbaa8668c87,
|
||||
q,1793911283332.0,1793911283342.5,1793911283444.1,202,,6172e2e2-860c-4aba-a27c-d98080487ffa,39e797bfa6f6,
|
||||
q,1793911283582.0,1793911283583.8,1793911283675.2,202,,f5263c27-4a82-4f25-a7cd-b4808fe0ab44,7bbaa8668c87,
|
||||
q,1793911283832.0,1793911283837.6,1793911283936.0,202,,741f4f0e-4e23-4ba3-8e3a-4e553b202eb9,39e797bfa6f6,
|
||||
q,1793911284082.0,1793911284092.6,1793911284173.6,202,,03ec7e14-83fd-4803-86f7-98d8ed6e2086,7bbaa8668c87,
|
||||
q,1793911284332.0,1793911284336.0,1793911284429.6,202,,908db4b0-84a1-4777-be51-63f5668b982a,39e797bfa6f6,
|
||||
q,1793911284582.0,1793911284590.6,1793911284680.2,202,,f0edabb3-98cb-4125-94da-e1bb4f6a2c08,7bbaa8668c87,
|
||||
q,1793911284832.0,1793911284834.8,1793911284921.4,202,,93ea2cf7-1dc9-4e83-88ed-fb7ef9c45215,39e797bfa6f6,
|
||||
q,1793911285082.0,1793911285084.9,1793911285186.6,202,,f180408f-a109-461e-9160-382604bd9cdb,7bbaa8668c87,
|
||||
q,1793911285332.0,1793911285340.6,1793911285432.0,202,,759567b7-4f39-4201-98c8-067224f7d4df,39e797bfa6f6,
|
||||
q,1793911285582.0,1793911285584.1,1793911285677.6,202,,90539c88-f789-4710-884f-00f183558f6d,7bbaa8668c87,
|
||||
q,1793911285832.0,1793911285834.4,1793911285926.3,202,,f39bc846-9447-48ea-b77b-62de889204b2,39e797bfa6f6,
|
||||
q,1793911286082.0,1793911286084.5,1793911286165.0,202,,6d18ba0b-394f-4fdc-b7d8-743e19ba9210,7bbaa8668c87,
|
||||
q,1793911286332.0,1793911286332.9,1793911286408.1,202,,24c79088-c500-443e-a125-04d32a29b956,39e797bfa6f6,
|
||||
q,1793911286582.0,1793911286585.3,1793911286656.9,202,,19dfe59f-c4f4-4705-af68-cbc5583995be,7bbaa8668c87,
|
||||
q,1793911286832.0,1793911286834.5,1793911286912.1,202,,826eb603-29b0-483f-84fc-d110bac3aa41,39e797bfa6f6,
|
||||
q,1793911287082.0,1793911287082.7,1793911287181.9,202,,fdcb3b24-447e-4215-bb35-1a9032d02d0f,7bbaa8668c87,
|
||||
q,1793911287332.0,1793911287337.5,1793911287436.2,202,,2a5d3296-5b4a-47ec-a0b7-80a46d685837,39e797bfa6f6,
|
||||
q,1793911287582.0,1793911287588.5,1793911287688.7,202,,e8f302f6-0ba9-49c5-bfde-c5512700d50d,7bbaa8668c87,
|
||||
q,1793911287832.0,1793911287844.5,1793911287940.8,202,,37ff09b9-be68-4523-99b8-6fa6f1d8a08a,39e797bfa6f6,
|
||||
q,1793911288082.0,1793911288085.0,1793911288166.1,202,,61ef985a-9a9d-4f57-8dc1-5dade858ba75,7bbaa8668c87,
|
||||
q,1793911288332.0,1793911288336.3,1793911288431.3,202,,cbd723eb-2664-4ae1-80a2-46e95e4e182d,39e797bfa6f6,
|
||||
q,1793911288582.0,1793911288586.0,1793911288675.5,202,,4e3c879c-eb25-470b-9145-aa64d9abf94e,7bbaa8668c87,
|
||||
q,1793911288832.0,1793911288843.8,1793911288935.1,202,,67dc5b17-ab0b-466e-b32a-70f56722879c,39e797bfa6f6,
|
||||
q,1793911289082.0,1793911289082.9,1793911289189.9,202,,e9b7e577-24a7-4a44-9e6d-5b807e1e7279,7bbaa8668c87,
|
||||
q,1793911289332.0,1793911289339.4,1793911289433.4,202,,60b74a6c-6725-4439-897b-99a2252a5bbf,39e797bfa6f6,
|
||||
q,1793911289582.0,1793911289589.1,1793911289688.4,202,,92e6ac5e-321f-45b9-91de-567c446bfe35,7bbaa8668c87,
|
||||
q,1793911289832.0,1793911289839.5,1793911289933.6,202,,6f02cdb0-833d-49c6-a285-79ccd88d11ce,39e797bfa6f6,
|
||||
q,1793911290082.0,1793911290084.0,1793911290164.3,202,,2f7e7b12-030d-44df-9898-206a2aef2dcb,7bbaa8668c87,
|
||||
q,1793911290332.0,1793911290336.6,1793911290433.4,202,,9a9579f8-85d1-4171-9f92-e33b0c539942,39e797bfa6f6,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 4.0,
|
||||
"duration_s": 11.0,
|
||||
"start": 1793911280.582,
|
||||
"end": 1793911290.582,
|
||||
"finished": 1793911290.434,
|
||||
"scheduled": 40,
|
||||
"sent": 40,
|
||||
"accepted": 40,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 40
|
||||
},
|
||||
"upload_p50_ms": 93.43,
|
||||
"upload_p95_ms": 380.96,
|
||||
"upload_max_ms": 898.1,
|
||||
"peak_inflight": 4,
|
||||
"gen_cpu_s": 0.138,
|
||||
"log": "/tmp/c4p4-l-r1-drain.log"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
q,1793911342943.0,1793911342944.1,1793911343052.0,202,,329f53d0-b6fb-4789-bd98-64a4a8a82d29,39e797bfa6f6,
|
||||
q,1793911344193.0,1793911344202.2,1793911344326.7,202,,46144556-944e-44a9-a2ef-f639ef98146d,7bbaa8668c87,
|
||||
q,1793911345443.0,1793911345446.2,1793911345553.8,202,,8d741696-fe85-4f9e-801b-bb7fe520c8e3,39e797bfa6f6,
|
||||
q,1793911346693.0,1793911346706.9,1793911346824.3,202,,f6bff17b-c05a-411d-9464-930960e3fd8e,7bbaa8668c87,
|
||||
q,1793911347943.0,1793911347954.9,1793911348064.3,202,,89629992-31d8-4d98-b376-caf62d89dbed,39e797bfa6f6,
|
||||
q,1793911349193.0,1793911349204.7,1793911349311.1,202,,5caf39b5-cd80-4286-ad04-33d397adc8c6,7bbaa8668c87,
|
||||
q,1793911350443.0,1793911350444.9,1793911350553.0,202,,68fe736e-982a-47f6-a2f4-c35169c7aab9,39e797bfa6f6,
|
||||
q,1793911351693.0,1793911351696.7,1793911351773.5,202,,def63feb-d77b-4d4d-8a7b-b1ca4f08dea7,7bbaa8668c87,
|
||||
q,1793911352943.0,1793911352948.2,1793911353053.8,202,,f56f864a-f7fc-4c52-b09c-f5b5d7923e6b,39e797bfa6f6,
|
||||
q,1793911354193.0,1793911354206.4,1793911354316.0,202,,8b408361-2e56-415c-89f5-3faa4c5528cd,7bbaa8668c87,
|
||||
q,1793911355443.0,1793911355455.0,1793911355566.1,202,,0ea365cc-f028-432d-a3e2-5b23f20355e7,39e797bfa6f6,
|
||||
q,1793911356693.0,1793911356695.8,1793911356799.6,202,,828dd5e3-06f8-48d3-8da0-d936afb40939,7bbaa8668c87,
|
||||
q,1793911357943.0,1793911357949.7,1793911358061.4,202,,882dc9fb-fa80-4dc7-b651-dc60c374247e,39e797bfa6f6,
|
||||
q,1793911359193.0,1793911359195.8,1793911359272.6,202,,ef9139bb-93fb-4c41-ae05-ed0323ae4eb7,7bbaa8668c87,
|
||||
q,1793911360443.0,1793911360447.8,1793911360570.5,202,,5ca0868b-6c3e-4f51-a276-6b5e5422e9f5,39e797bfa6f6,
|
||||
q,1793911361693.0,1793911361696.0,1793911361800.2,202,,c0569b14-f9c3-45b9-94d9-334949c9685d,7bbaa8668c87,
|
||||
q,1793911362943.0,1793911362943.6,1793911363072.3,202,,0c006a4e-d22c-4373-9d50-de9f15cb22fd,39e797bfa6f6,
|
||||
q,1793911364193.0,1793911364200.1,1793911364304.0,202,,91f9df6e-2bc8-45ff-9dd7-f504e48e2604,7bbaa8668c87,
|
||||
q,1793911365443.0,1793911365454.9,1793911365557.8,202,,303f62a7-1f63-4c7d-83e3-272966e4075c,39e797bfa6f6,
|
||||
q,1793911366693.0,1793911366699.4,1793911367056.3,202,,177dafb5-3a93-44e1-861d-48d8de5f17e9,7bbaa8668c87,
|
||||
q,1793911367943.0,1793911367944.6,1793911368025.4,202,,2f3b1673-a0b4-4f71-82ed-8343c9b5f7ea,39e797bfa6f6,
|
||||
q,1793911369193.0,1793911369194.0,1793911369298.4,202,,4ebf0ba5-cb5a-4d6e-a4c8-1bdaa13932de,7bbaa8668c87,
|
||||
q,1793911370443.0,1793911370444.2,1793911370551.1,202,,c4a78ec8-03d3-497d-affd-a9cfb09f5327,39e797bfa6f6,
|
||||
q,1793911371693.0,1793911371699.4,1793911371814.3,202,,7ab649d1-0209-420c-8798-fe85806ccbb2,7bbaa8668c87,
|
||||
q,1793911372943.0,1793911372950.6,1793911373053.3,202,,5a65d0cc-d52a-4097-9109-17839e940a66,39e797bfa6f6,
|
||||
q,1793911374193.0,1793911374204.7,1793911374323.9,202,,5426d41b-70c0-44d2-9115-d658ffffd438,7bbaa8668c87,
|
||||
q,1793911375443.0,1793911375454.7,1793911375547.9,202,,d67ca25e-0c4c-44df-840c-ae61551a9a37,39e797bfa6f6,
|
||||
q,1793911376693.0,1793911376698.6,1793911376808.4,202,,409e63e7-f33c-47aa-a20e-49ed0a08dd4f,7bbaa8668c87,
|
||||
q,1793911377943.0,1793911377955.9,1793911378069.7,202,,fd694b8b-3b45-4425-bfd2-2e7ae9a95342,39e797bfa6f6,
|
||||
q,1793911379193.0,1793911379197.4,1793911379295.3,202,,e6f3e1f6-16b1-44ab-be34-cacfea25e4b5,7bbaa8668c87,
|
||||
q,1793911380443.0,1793911380446.6,1793911380528.6,202,,12ee82ae-64ea-44a2-82c0-befe59e3e032,39e797bfa6f6,
|
||||
q,1793911381693.0,1793911381695.3,1793911381797.7,202,,756b2029-5934-4ea9-af56-cfffd647d82f,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 0.8,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793911342.943,
|
||||
"end": 1793911382.943,
|
||||
"finished": 1793911381.797,
|
||||
"scheduled": 32,
|
||||
"sent": 32,
|
||||
"accepted": 32,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 32
|
||||
},
|
||||
"upload_p50_ms": 106.91,
|
||||
"upload_p95_ms": 128.71,
|
||||
"upload_max_ms": 356.87,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.261,
|
||||
"log": "/tmp/c4p4-l-r1-fixed.log"
|
||||
}
|
||||
@@ -0,0 +1,322 @@
|
||||
q,1793911383032.8,1793911383035.2,1793911383130.0,202,,df1f5b63-d4da-456d-b6bb-3e3945343691,39e797bfa6f6,
|
||||
q,1793911383157.1,1793911383167.0,1793911383234.8,202,,fe6095c5-2a30-483f-843c-42649ae60159,7bbaa8668c87,
|
||||
q,1793911383281.3,1793911383285.0,1793911383358.7,202,,e0eed9b3-05e4-47f6-a7a5-974124ce1cf8,39e797bfa6f6,
|
||||
q,1793911383405.5,1793911383407.3,1793911383481.1,202,,f9846259-e980-4d07-b3be-aa5b6505081b,7bbaa8668c87,
|
||||
q,1793911383529.7,1793911383531.5,1793911383618.2,202,,a2333ad4-a052-4d41-8b85-b338097a956a,39e797bfa6f6,
|
||||
q,1793911383654.0,1793911383656.4,1793911383755.3,202,,9d400afa-8545-41ed-9fb7-81bb7f2929df,7bbaa8668c87,
|
||||
q,1793911383778.2,1793911383779.3,1793911383857.1,202,,ca657cd7-40c3-41fd-a39c-22f395768efa,39e797bfa6f6,
|
||||
q,1793911383902.4,1793911383903.4,1793911384002.0,202,,f9d84ef9-b923-4c03-aae5-3076362dd493,7bbaa8668c87,
|
||||
q,1793911384026.6,1793911384028.0,1793911384123.9,202,,a0d084cc-7693-46b6-b6b5-bdc500c3c390,39e797bfa6f6,
|
||||
q,1793911384150.9,1793911384152.0,1793911384246.7,202,,67a8d18f-e423-46a0-a579-8bdef1db2f02,7bbaa8668c87,
|
||||
q,1793911384275.1,1793911384277.1,1793911384350.2,202,,2fe8e7c5-238a-4fa2-b7c8-580eea9a6777,39e797bfa6f6,
|
||||
q,1793911384399.3,1793911384401.0,1793911385038.5,202,,ef4af91f-c9b5-42e1-92af-ddd2658fc11c,7bbaa8668c87,
|
||||
q,1793911384647.8,1793911384649.4,1793911385053.4,202,,a5356872-d138-4c70-9c6d-453a7e59c949,7bbaa8668c87,
|
||||
q,1793911384523.5,1793911384525.2,1793911385054.5,202,,aabc6e22-b955-4841-88b3-3c54f367c723,39e797bfa6f6,
|
||||
q,1793911384896.2,1793911384899.0,1793911385712.2,503,,,7bbaa8668c87,
|
||||
q,1793911385517.4,1793911385519.4,1793911385712.3,503,,,39e797bfa6f6,
|
||||
q,1793911385393.1,1793911385395.3,1793911385712.3,503,,,7bbaa8668c87,
|
||||
q,1793911385020.5,1793911385022.2,1793911385712.3,503,,,39e797bfa6f6,
|
||||
q,1793911385144.7,1793911385146.4,1793911385712.7,503,,,7bbaa8668c87,
|
||||
q,1793911385268.9,1793911385270.3,1793911385712.8,503,,,39e797bfa6f6,
|
||||
q,1793911384772.0,1793911384776.2,1793911385712.8,503,,,39e797bfa6f6,
|
||||
q,1793911385641.6,1793911385643.1,1793911385713.2,503,,,7bbaa8668c87,
|
||||
q,1793911385765.8,1793911385768.0,1793911386796.1,503,,,39e797bfa6f6,
|
||||
q,1793911385890.0,1793911385892.9,1793911386926.5,503,,,7bbaa8668c87,
|
||||
q,1793911386014.3,1793911386015.9,1793911387050.3,503,,,39e797bfa6f6,
|
||||
q,1793911386138.5,1793911386146.6,1793911387186.4,503,,,7bbaa8668c87,
|
||||
q,1793911386262.7,1793911386264.7,1793911387304.0,503,,,39e797bfa6f6,
|
||||
q,1793911386386.9,1793911386388.7,1793911387430.0,503,,,7bbaa8668c87,
|
||||
q,1793911386511.2,1793911386514.4,1793911387559.5,503,,,39e797bfa6f6,
|
||||
q,1793911386635.4,1793911386638.4,1793911387680.5,503,,,7bbaa8668c87,
|
||||
q,1793911386759.6,1793911386761.8,1793911387828.4,503,,,39e797bfa6f6,
|
||||
q,1793911386883.9,1793911386890.1,1793911387963.9,503,,,7bbaa8668c87,
|
||||
q,1793911387008.1,1793911387010.0,1793911388083.0,503,,,39e797bfa6f6,
|
||||
q,1793911387132.3,1793911387136.0,1793911388224.3,503,,,7bbaa8668c87,
|
||||
q,1793911387256.5,1793911387259.6,1793911388344.8,503,,,39e797bfa6f6,
|
||||
q,1793911387380.8,1793911387381.6,1793911388471.6,503,,,7bbaa8668c87,
|
||||
q,1793911387505.0,1793911387506.4,1793911388597.7,503,,,39e797bfa6f6,
|
||||
q,1793911387629.2,1793911387633.7,1793911388726.7,503,,,7bbaa8668c87,
|
||||
q,1793911387753.4,1793911387764.1,1793911388861.0,503,,,39e797bfa6f6,
|
||||
q,1793911387877.7,1793911387879.0,1793911389003.8,503,,,7bbaa8668c87,
|
||||
q,1793911388001.9,1793911388003.1,1793911389124.6,503,,,39e797bfa6f6,
|
||||
q,1793911388126.1,1793911388127.3,1793911389268.0,503,,,7bbaa8668c87,
|
||||
q,1793911388250.3,1793911388252.2,1793911389383.8,503,,,39e797bfa6f6,
|
||||
q,1793911388374.6,1793911388377.6,1793911389511.8,503,,,7bbaa8668c87,
|
||||
q,1793911388498.8,1793911388500.2,1793911389637.9,503,,,39e797bfa6f6,
|
||||
q,1793911388623.0,1793911388630.0,1793911389767.6,503,,,7bbaa8668c87,
|
||||
q,1793911388747.2,1793911388751.5,1793911389899.6,503,,,39e797bfa6f6,
|
||||
q,1793911388871.5,1793911388870.9,1793911390032.5,503,,,7bbaa8668c87,
|
||||
q,1793911388995.7,1793911388998.0,1793911390160.6,503,,,39e797bfa6f6,
|
||||
q,1793911389119.9,1793911389123.8,1793911390306.2,503,,,7bbaa8668c87,
|
||||
q,1793911389244.2,1793911389248.7,1793911390427.4,503,,,39e797bfa6f6,
|
||||
q,1793911389368.4,1793911389376.9,1793911390539.5,503,,,7bbaa8668c87,
|
||||
q,1793911389492.6,1793911389495.4,1793911390668.5,503,,,39e797bfa6f6,
|
||||
q,1793911389616.8,1793911389618.0,1793911390801.0,503,,,7bbaa8668c87,
|
||||
q,1793911389741.1,1793911389742.1,1793911390933.4,503,,,39e797bfa6f6,
|
||||
q,1793911389865.3,1793911389869.2,1793911391064.2,503,,,7bbaa8668c87,
|
||||
q,1793911389989.5,1793911389991.3,1793911391198.2,503,,,39e797bfa6f6,
|
||||
q,1793911390113.7,1793911390115.0,1793911391343.8,503,,,7bbaa8668c87,
|
||||
q,1793911390238.0,1793911390242.8,1793911391465.9,503,,,39e797bfa6f6,
|
||||
q,1793911390362.2,1793911390371.0,1793911391581.6,503,,,7bbaa8668c87,
|
||||
q,1793911390486.4,1793911390498.0,1793911391704.1,503,,,39e797bfa6f6,
|
||||
q,1793911390610.6,1793911390622.0,1793911391835.4,503,,,7bbaa8668c87,
|
||||
q,1793911390734.9,1793911390738.3,1793911391962.4,503,,,39e797bfa6f6,
|
||||
q,1793911390859.1,1793911390861.1,1793911392090.8,503,,,7bbaa8668c87,
|
||||
q,1793911390983.3,1793911390984.4,1793911392235.0,503,,,39e797bfa6f6,
|
||||
q,1793911391107.5,1793911391109.7,1793911392387.9,503,,,7bbaa8668c87,
|
||||
q,1793911391231.8,1793911391233.5,1793911392511.4,503,,,39e797bfa6f6,
|
||||
q,1793911391356.0,1793911391355.3,1793911392623.8,503,,,7bbaa8668c87,
|
||||
q,1793911391480.2,1793911391482.2,1793911392745.7,503,,,39e797bfa6f6,
|
||||
q,1793911391604.5,1793911391607.9,1793911392881.7,503,,,7bbaa8668c87,
|
||||
q,1793911391728.7,1793911391730.0,1793911393008.3,503,,,39e797bfa6f6,
|
||||
q,1793911391852.9,1793911391854.4,1793911393125.2,503,,,7bbaa8668c87,
|
||||
q,1793911391977.1,1793911391979.0,1793911393277.1,503,,,39e797bfa6f6,
|
||||
q,1793911392101.4,1793911392101.2,1793911393431.4,503,,,7bbaa8668c87,
|
||||
q,1793911392225.6,1793911392227.3,1793911393552.6,503,,,39e797bfa6f6,
|
||||
q,1793911392349.8,1793911392352.3,1793911393659.8,503,,,7bbaa8668c87,
|
||||
q,1793911392474.0,1793911392476.7,1793911393786.6,503,,,39e797bfa6f6,
|
||||
q,1793911392598.3,1793911392603.9,1793911393923.0,503,,,7bbaa8668c87,
|
||||
q,1793911392722.5,1793911392726.3,1793911394043.4,503,,,39e797bfa6f6,
|
||||
q,1793911392846.7,1793911392848.9,1793911394167.1,503,,,7bbaa8668c87,
|
||||
q,1793911392970.9,1793911392974.0,1793911394312.8,503,,,39e797bfa6f6,
|
||||
q,1793911393095.2,1793911393097.4,1793911394474.2,503,,,7bbaa8668c87,
|
||||
q,1793911393219.4,1793911393221.2,1793911394593.7,503,,,39e797bfa6f6,
|
||||
q,1793911393343.6,1793911393347.0,1793911394700.7,503,,,7bbaa8668c87,
|
||||
q,1793911393467.8,1793911393474.7,1793911394831.0,503,,,39e797bfa6f6,
|
||||
q,1793911393592.1,1793911393599.6,1793911394964.5,503,,,7bbaa8668c87,
|
||||
q,1793911394585.9,1793911394592.6,1793911394994.3,503,,,7bbaa8668c87,
|
||||
q,1793911394834.3,1793911394835.0,1793911395022.5,503,,,7bbaa8668c87,
|
||||
q,1793911393716.3,1793911393719.0,1793911395074.8,503,,,39e797bfa6f6,
|
||||
q,1793911394710.1,1793911394710.0,1793911395103.8,503,,,39e797bfa6f6,
|
||||
q,1793911395082.8,1793911395081.9,1793911395113.3,503,,,7bbaa8668c87,
|
||||
q,1793911394958.6,1793911394961.0,1793911395132.2,503,,,39e797bfa6f6,
|
||||
q,1793911393840.5,1793911393846.6,1793911395204.5,503,,,7bbaa8668c87,
|
||||
q,1793911395207.0,1793911395207.9,1793911395238.9,503,,,39e797bfa6f6,
|
||||
q,1793911393964.8,1793911393966.4,1793911395354.8,503,,,39e797bfa6f6,
|
||||
q,1793911395331.2,1793911395342.0,1793911395383.3,503,,,7bbaa8668c87,
|
||||
q,1793911395455.5,1793911395459.4,1793911395498.7,503,,,39e797bfa6f6,
|
||||
q,1793911394089.0,1793911394090.0,1793911395522.1,503,,,7bbaa8668c87,
|
||||
q,1793911394213.2,1793911394215.9,1793911395633.6,503,,,39e797bfa6f6,
|
||||
q,1793911395579.7,1793911395591.0,1793911395633.6,503,,,7bbaa8668c87,
|
||||
q,1793911394337.4,1793911394340.0,1793911395738.9,503,,,7bbaa8668c87,
|
||||
q,1793911395703.9,1793911395716.0,1793911395758.0,503,,,39e797bfa6f6,
|
||||
q,1793911394461.7,1793911394471.6,1793911395871.8,503,,,39e797bfa6f6,
|
||||
q,1793911395828.2,1793911395841.0,1793911395881.7,503,,,7bbaa8668c87,
|
||||
q,1793911395952.4,1793911395960.2,1793911396001.8,503,,,39e797bfa6f6,
|
||||
q,1793911396076.6,1793911396079.6,1793911396115.7,503,,,7bbaa8668c87,
|
||||
q,1793911396200.8,1793911396205.1,1793911396245.3,503,,,39e797bfa6f6,
|
||||
q,1793911396325.1,1793911396328.0,1793911396367.5,503,,,7bbaa8668c87,
|
||||
q,1793911396449.3,1793911396453.5,1793911396494.0,503,,,39e797bfa6f6,
|
||||
q,1793911396573.5,1793911396575.5,1793911396614.7,503,,,7bbaa8668c87,
|
||||
q,1793911396697.7,1793911396702.4,1793911396741.1,503,,,39e797bfa6f6,
|
||||
q,1793911396822.0,1793911396832.3,1793911396873.6,503,,,7bbaa8668c87,
|
||||
q,1793911396946.2,1793911396947.3,1793911396987.5,503,,,39e797bfa6f6,
|
||||
q,1793911397070.4,1793911397071.5,1793911397110.9,503,,,7bbaa8668c87,
|
||||
q,1793911397194.6,1793911397195.7,1793911397237.1,503,,,39e797bfa6f6,
|
||||
q,1793911397318.9,1793911397321.0,1793911397360.7,503,,,7bbaa8668c87,
|
||||
q,1793911397443.1,1793911397447.8,1793911397488.3,503,,,39e797bfa6f6,
|
||||
q,1793911397567.3,1793911397568.3,1793911397605.8,503,,,7bbaa8668c87,
|
||||
q,1793911397691.5,1793911397694.1,1793911397733.8,503,,,39e797bfa6f6,
|
||||
q,1793911397815.8,1793911397818.2,1793911397858.7,503,,,7bbaa8668c87,
|
||||
q,1793911397940.0,1793911397941.9,1793911397982.2,503,,,39e797bfa6f6,
|
||||
q,1793911398064.2,1793911398067.4,1793911398102.7,503,,,7bbaa8668c87,
|
||||
q,1793911398188.5,1793911398190.0,1793911398229.1,503,,,39e797bfa6f6,
|
||||
q,1793911398312.7,1793911398313.8,1793911398352.6,503,,,7bbaa8668c87,
|
||||
q,1793911398436.9,1793911398438.1,1793911398478.4,503,,,39e797bfa6f6,
|
||||
q,1793911398561.1,1793911398562.9,1793911398603.7,503,,,7bbaa8668c87,
|
||||
q,1793911398685.4,1793911398687.5,1793911398727.9,503,,,39e797bfa6f6,
|
||||
q,1793911398809.6,1793911398813.8,1793911398855.1,503,,,7bbaa8668c87,
|
||||
q,1793911398933.8,1793911398935.6,1793911398978.7,503,,,39e797bfa6f6,
|
||||
q,1793911399058.0,1793911399059.0,1793911399093.7,503,,,7bbaa8668c87,
|
||||
q,1793911399182.3,1793911399184.2,1793911399221.6,503,,,39e797bfa6f6,
|
||||
q,1793911399306.5,1793911399308.5,1793911399340.8,503,,,7bbaa8668c87,
|
||||
q,1793911399430.7,1793911399435.0,1793911399474.5,503,,,39e797bfa6f6,
|
||||
q,1793911399554.9,1793911399563.7,1793911399606.9,503,,,7bbaa8668c87,
|
||||
q,1793911399679.2,1793911399682.4,1793911399722.3,503,,,39e797bfa6f6,
|
||||
q,1793911399803.4,1793911399807.3,1793911399847.5,503,,,7bbaa8668c87,
|
||||
q,1793911399927.6,1793911399931.6,1793911399973.2,503,,,39e797bfa6f6,
|
||||
q,1793911400051.8,1793911400054.6,1793911400087.9,503,,,7bbaa8668c87,
|
||||
q,1793911400176.1,1793911400178.9,1793911400218.2,503,,,39e797bfa6f6,
|
||||
q,1793911400300.3,1793911400302.1,1793911400333.1,503,,,7bbaa8668c87,
|
||||
q,1793911400424.5,1793911400426.0,1793911400457.1,503,,,39e797bfa6f6,
|
||||
q,1793911400548.8,1793911400550.0,1793911400581.3,503,,,7bbaa8668c87,
|
||||
q,1793911400673.0,1793911400684.0,1793911400722.9,503,,,39e797bfa6f6,
|
||||
q,1793911400797.2,1793911400799.0,1793911400831.9,503,,,7bbaa8668c87,
|
||||
q,1793911400921.4,1793911400926.2,1793911400964.2,503,,,39e797bfa6f6,
|
||||
q,1793911401045.7,1793911401046.7,1793911401080.6,503,,,7bbaa8668c87,
|
||||
q,1793911401169.9,1793911401172.2,1793911401202.7,503,,,39e797bfa6f6,
|
||||
q,1793911401294.1,1793911401295.5,1793911401327.5,503,,,7bbaa8668c87,
|
||||
q,1793911401418.3,1793911401419.8,1793911401455.2,503,,,39e797bfa6f6,
|
||||
q,1793911401542.6,1793911401545.0,1793911401585.9,503,,,7bbaa8668c87,
|
||||
q,1793911401666.8,1793911401672.0,1793911401714.2,503,,,39e797bfa6f6,
|
||||
q,1793911401791.0,1793911401797.8,1793911401838.3,503,,,7bbaa8668c87,
|
||||
q,1793911401915.2,1793911401919.0,1793911401959.3,503,,,39e797bfa6f6,
|
||||
q,1793911402039.5,1793911402043.8,1793911402077.2,503,,,7bbaa8668c87,
|
||||
q,1793911402163.7,1793911402167.2,1793911402206.4,503,,,39e797bfa6f6,
|
||||
q,1793911402287.9,1793911402292.5,1793911402332.2,503,,,7bbaa8668c87,
|
||||
q,1793911402412.1,1793911402413.0,1793911402455.2,503,,,39e797bfa6f6,
|
||||
q,1793911402536.4,1793911402540.4,1793911402582.4,503,,,7bbaa8668c87,
|
||||
q,1793911402660.6,1793911402662.4,1793911402699.2,503,,,39e797bfa6f6,
|
||||
q,1793911402784.8,1793911402786.3,1793911402825.5,503,,,7bbaa8668c87,
|
||||
q,1793911402909.1,1793911402912.5,1793911402951.7,503,,,39e797bfa6f6,
|
||||
q,1793911403033.3,1793911403034.1,1793911403067.2,503,,,7bbaa8668c87,
|
||||
q,1793911403157.5,1793911403159.3,1793911403197.8,503,,,39e797bfa6f6,
|
||||
q,1793911403281.7,1793911403282.4,1793911403321.6,503,,,7bbaa8668c87,
|
||||
q,1793911403406.0,1793911403412.3,1793911403454.1,503,,,39e797bfa6f6,
|
||||
q,1793911403530.2,1793911403534.0,1793911403574.2,503,,,7bbaa8668c87,
|
||||
q,1793911403654.4,1793911403658.0,1793911403698.6,503,,,39e797bfa6f6,
|
||||
q,1793911403778.6,1793911403785.5,1793911403826.0,503,,,7bbaa8668c87,
|
||||
q,1793911403902.9,1793911403904.8,1793911403944.6,503,,,39e797bfa6f6,
|
||||
q,1793911404027.1,1793911404028.1,1793911404060.1,503,,,7bbaa8668c87,
|
||||
q,1793911404151.3,1793911404153.0,1793911404190.0,503,,,39e797bfa6f6,
|
||||
q,1793911404275.5,1793911404277.4,1793911404316.2,503,,,7bbaa8668c87,
|
||||
q,1793911404399.8,1793911404401.5,1793911404441.8,503,,,39e797bfa6f6,
|
||||
q,1793911404524.0,1793911404527.8,1793911404572.3,503,,,7bbaa8668c87,
|
||||
q,1793911404648.2,1793911404650.8,1793911404689.5,503,,,39e797bfa6f6,
|
||||
q,1793911404772.4,1793911404774.6,1793911404816.2,503,,,7bbaa8668c87,
|
||||
q,1793911404896.7,1793911404908.1,1793911404950.6,503,,,39e797bfa6f6,
|
||||
q,1793911405020.9,1793911405025.2,1793911405056.0,503,,,7bbaa8668c87,
|
||||
q,1793911405145.1,1793911405152.6,1793911405183.7,503,,,39e797bfa6f6,
|
||||
q,1793911405269.4,1793911405275.0,1793911405305.8,503,,,7bbaa8668c87,
|
||||
q,1793911405393.6,1793911405396.6,1793911405442.8,503,,,39e797bfa6f6,
|
||||
q,1793911405517.8,1793911405529.0,1793911405573.1,503,,,7bbaa8668c87,
|
||||
q,1793911405642.0,1793911405643.8,1793911405677.3,503,,,39e797bfa6f6,
|
||||
q,1793911405766.3,1793911405767.1,1793911405796.4,503,,,7bbaa8668c87,
|
||||
q,1793911405890.5,1793911405898.2,1793911405930.1,503,,,39e797bfa6f6,
|
||||
q,1793911406014.7,1793911406018.8,1793911406048.0,503,,,7bbaa8668c87,
|
||||
q,1793911406138.9,1793911406140.8,1793911406179.3,503,,,39e797bfa6f6,
|
||||
q,1793911406263.2,1793911406270.1,1793911406310.3,503,,,7bbaa8668c87,
|
||||
q,1793911406387.4,1793911406392.0,1793911406433.5,503,,,39e797bfa6f6,
|
||||
q,1793911406511.6,1793911406519.0,1793911406560.6,503,,,7bbaa8668c87,
|
||||
q,1793911406635.8,1793911406637.4,1793911406679.6,503,,,39e797bfa6f6,
|
||||
q,1793911406760.1,1793911406761.4,1793911406803.2,503,,,7bbaa8668c87,
|
||||
q,1793911406884.3,1793911406896.0,1793911406939.3,503,,,39e797bfa6f6,
|
||||
q,1793911407008.5,1793911407010.1,1793911407046.1,503,,,7bbaa8668c87,
|
||||
q,1793911407132.8,1793911407137.3,1793911407176.3,503,,,39e797bfa6f6,
|
||||
q,1793911407257.0,1793911407268.0,1793911407309.3,503,,,7bbaa8668c87,
|
||||
q,1793911407381.2,1793911407385.9,1793911407425.0,503,,,39e797bfa6f6,
|
||||
q,1793911407505.4,1793911407509.7,1793911407550.1,503,,,7bbaa8668c87,
|
||||
q,1793911407629.7,1793911407631.3,1793911407668.6,503,,,39e797bfa6f6,
|
||||
q,1793911407753.9,1793911407755.3,1793911407794.8,503,,,7bbaa8668c87,
|
||||
q,1793911407878.1,1793911407881.6,1793911407923.2,503,,,39e797bfa6f6,
|
||||
q,1793911408002.3,1793911408004.1,1793911408041.4,503,,,7bbaa8668c87,
|
||||
q,1793911408126.6,1793911408128.9,1793911408166.8,503,,,39e797bfa6f6,
|
||||
q,1793911408250.8,1793911408252.3,1793911408292.5,503,,,7bbaa8668c87,
|
||||
q,1793911408375.0,1793911408377.5,1793911408417.1,503,,,39e797bfa6f6,
|
||||
q,1793911408499.2,1793911408501.5,1793911408542.6,503,,,7bbaa8668c87,
|
||||
q,1793911408623.5,1793911408628.2,1793911408668.8,503,,,39e797bfa6f6,
|
||||
q,1793911408747.7,1793911408750.0,1793911408789.6,503,,,7bbaa8668c87,
|
||||
q,1793911408871.9,1793911408873.3,1793911408909.3,503,,,39e797bfa6f6,
|
||||
q,1793911408996.1,1793911408998.0,1793911409033.6,503,,,7bbaa8668c87,
|
||||
q,1793911409120.4,1793911409122.3,1793911409155.8,503,,,39e797bfa6f6,
|
||||
q,1793911409244.6,1793911409248.0,1793911409286.7,503,,,7bbaa8668c87,
|
||||
q,1793911409368.8,1793911409370.2,1793911409407.3,503,,,39e797bfa6f6,
|
||||
q,1793911409493.1,1793911409499.8,1793911409539.7,503,,,7bbaa8668c87,
|
||||
q,1793911409617.3,1793911409620.1,1793911409661.8,503,,,39e797bfa6f6,
|
||||
q,1793911409741.5,1793911409743.9,1793911409783.4,503,,,7bbaa8668c87,
|
||||
q,1793911409865.7,1793911409867.3,1793911409909.0,503,,,39e797bfa6f6,
|
||||
q,1793911409990.0,1793911409991.0,1793911410029.2,503,,,7bbaa8668c87,
|
||||
q,1793911410114.2,1793911410115.3,1793911410153.6,503,,,39e797bfa6f6,
|
||||
q,1793911410238.4,1793911410249.0,1793911410291.7,503,,,7bbaa8668c87,
|
||||
q,1793911410362.6,1793911410374.0,1793911410416.5,503,,,39e797bfa6f6,
|
||||
q,1793911410486.9,1793911410491.6,1793911410530.3,503,,,7bbaa8668c87,
|
||||
q,1793911410611.1,1793911410615.7,1793911410646.1,503,,,39e797bfa6f6,
|
||||
q,1793911410735.3,1793911410738.4,1793911410784.7,503,,,7bbaa8668c87,
|
||||
q,1793911410859.5,1793911410863.9,1793911410907.1,503,,,39e797bfa6f6,
|
||||
q,1793911410983.8,1793911410984.3,1793911411020.2,503,,,7bbaa8668c87,
|
||||
q,1793911411108.0,1793911411109.4,1793911411140.0,503,,,39e797bfa6f6,
|
||||
q,1793911411232.2,1793911411234.0,1793911411265.3,503,,,7bbaa8668c87,
|
||||
q,1793911411356.4,1793911411358.3,1793911411388.1,503,,,39e797bfa6f6,
|
||||
q,1793911411480.7,1793911411482.0,1793911411511.8,503,,,7bbaa8668c87,
|
||||
q,1793911411604.9,1793911411615.5,1793911411652.6,503,,,39e797bfa6f6,
|
||||
q,1793911411729.1,1793911411731.6,1793911411769.4,503,,,7bbaa8668c87,
|
||||
q,1793911411853.4,1793911411864.0,1793911411896.1,503,,,39e797bfa6f6,
|
||||
q,1793911411977.6,1793911411979.2,1793911412015.4,503,,,7bbaa8668c87,
|
||||
q,1793911412101.8,1793911412108.8,1793911412147.9,503,,,39e797bfa6f6,
|
||||
q,1793911412226.0,1793911412236.5,1793911412279.0,503,,,7bbaa8668c87,
|
||||
q,1793911412350.3,1793911412354.9,1793911412394.8,503,,,39e797bfa6f6,
|
||||
q,1793911412474.5,1793911412485.0,1793911412524.2,503,,,7bbaa8668c87,
|
||||
q,1793911412598.7,1793911412608.0,1793911412649.7,503,,,39e797bfa6f6,
|
||||
q,1793911412722.9,1793911412725.5,1793911412762.7,503,,,7bbaa8668c87,
|
||||
q,1793911412847.2,1793911412851.0,1793911412892.6,503,,,39e797bfa6f6,
|
||||
q,1793911412971.4,1793911412975.2,1793911413011.2,503,,,7bbaa8668c87,
|
||||
q,1793911413095.6,1793911413100.1,1793911413137.2,503,,,39e797bfa6f6,
|
||||
q,1793911413219.8,1793911413227.5,1793911413268.2,503,,,7bbaa8668c87,
|
||||
q,1793911413344.1,1793911413345.9,1793911413388.4,503,,,39e797bfa6f6,
|
||||
q,1793911413468.3,1793911413473.5,1793911413516.5,503,,,7bbaa8668c87,
|
||||
q,1793911413592.5,1793911413593.5,1793911413632.3,503,,,39e797bfa6f6,
|
||||
q,1793911413716.7,1793911413718.8,1793911413760.7,503,,,7bbaa8668c87,
|
||||
q,1793911413841.0,1793911413852.0,1793911413895.3,503,,,39e797bfa6f6,
|
||||
q,1793911413965.2,1793911413972.6,1793911414010.5,503,,,7bbaa8668c87,
|
||||
q,1793911414089.4,1793911414090.2,1793911414125.8,503,,,39e797bfa6f6,
|
||||
q,1793911414213.7,1793911414218.2,1793911414254.3,503,,,7bbaa8668c87,
|
||||
q,1793911414337.9,1793911414340.8,1793911414379.8,503,,,39e797bfa6f6,
|
||||
q,1793911414462.1,1793911414465.3,1793911414500.1,503,,,7bbaa8668c87,
|
||||
q,1793911414586.3,1793911414589.3,1793911414629.8,503,,,39e797bfa6f6,
|
||||
q,1793911414710.6,1793911414714.5,1793911414755.1,503,,,7bbaa8668c87,
|
||||
q,1793911414834.8,1793911414836.7,1793911414876.7,503,,,39e797bfa6f6,
|
||||
q,1793911414959.0,1793911414960.7,1793911415001.4,503,,,7bbaa8668c87,
|
||||
q,1793911415083.2,1793911415084.5,1793911415122.9,503,,,39e797bfa6f6,
|
||||
q,1793911415207.5,1793911415217.8,1793911415259.9,503,,,7bbaa8668c87,
|
||||
q,1793911415331.7,1793911415335.8,1793911415375.6,503,,,39e797bfa6f6,
|
||||
q,1793911415455.9,1793911415457.6,1793911415494.1,503,,,7bbaa8668c87,
|
||||
q,1793911415580.1,1793911415589.7,1793911415629.6,503,,,39e797bfa6f6,
|
||||
q,1793911415704.4,1793911415711.0,1793911415750.9,503,,,7bbaa8668c87,
|
||||
q,1793911415828.6,1793911415832.2,1793911415872.7,503,,,39e797bfa6f6,
|
||||
q,1793911415952.8,1793911415954.3,1793911415993.3,503,,,7bbaa8668c87,
|
||||
q,1793911416077.1,1793911416087.9,1793911416124.3,503,,,39e797bfa6f6,
|
||||
q,1793911416201.3,1793911416205.0,1793911416244.5,503,,,7bbaa8668c87,
|
||||
q,1793911416325.5,1793911416328.1,1793911416369.1,503,,,39e797bfa6f6,
|
||||
q,1793911416449.7,1793911416451.0,1793911416481.2,503,,,7bbaa8668c87,
|
||||
q,1793911416574.0,1793911416576.3,1793911416605.1,503,,,39e797bfa6f6,
|
||||
q,1793911416698.2,1793911416700.1,1793911416732.9,503,,,7bbaa8668c87,
|
||||
q,1793911416822.4,1793911416829.5,1793911416868.1,503,,,39e797bfa6f6,
|
||||
q,1793911416946.6,1793911416948.9,1793911416989.3,503,,,7bbaa8668c87,
|
||||
q,1793911417070.9,1793911417076.4,1793911417112.6,503,,,39e797bfa6f6,
|
||||
q,1793911417195.1,1793911417197.4,1793911417235.9,503,,,7bbaa8668c87,
|
||||
q,1793911417319.3,1793911417326.9,1793911417368.7,503,,,39e797bfa6f6,
|
||||
q,1793911417443.5,1793911417454.0,1793911417495.4,503,,,7bbaa8668c87,
|
||||
q,1793911417567.8,1793911417579.0,1793911417620.0,503,,,39e797bfa6f6,
|
||||
q,1793911417692.0,1793911417695.3,1793911417731.2,503,,,7bbaa8668c87,
|
||||
q,1793911417816.2,1793911417820.3,1793911417861.7,503,,,39e797bfa6f6,
|
||||
q,1793911417940.4,1793911417944.9,1793911417986.7,503,,,7bbaa8668c87,
|
||||
q,1793911418064.7,1793911418067.3,1793911418102.7,503,,,39e797bfa6f6,
|
||||
q,1793911418188.9,1793911418197.4,1793911418237.2,503,,,7bbaa8668c87,
|
||||
q,1793911418313.1,1793911418318.0,1793911418360.6,503,,,39e797bfa6f6,
|
||||
q,1793911418437.4,1793911418448.0,1793911418490.3,503,,,7bbaa8668c87,
|
||||
q,1793911418561.6,1793911418564.7,1793911418605.6,503,,,39e797bfa6f6,
|
||||
q,1793911418685.8,1793911418693.5,1793911418734.2,503,,,7bbaa8668c87,
|
||||
q,1793911418810.0,1793911418811.0,1793911418852.3,503,,,39e797bfa6f6,
|
||||
q,1793911418934.3,1793911418936.0,1793911418977.8,503,,,7bbaa8668c87,
|
||||
q,1793911419058.5,1793911419060.8,1793911419094.4,503,,,39e797bfa6f6,
|
||||
q,1793911419182.7,1793911419184.5,1793911419217.7,503,,,7bbaa8668c87,
|
||||
q,1793911419306.9,1793911419310.8,1793911419350.1,503,,,39e797bfa6f6,
|
||||
q,1793911419431.2,1793911419434.6,1793911419474.5,503,,,7bbaa8668c87,
|
||||
q,1793911419555.4,1793911419556.9,1793911419596.6,503,,,39e797bfa6f6,
|
||||
q,1793911419679.6,1793911419682.6,1793911419723.4,503,,,7bbaa8668c87,
|
||||
q,1793911419803.8,1793911419810.0,1793911419851.5,503,,,39e797bfa6f6,
|
||||
q,1793911419928.1,1793911419930.6,1793911419971.2,503,,,7bbaa8668c87,
|
||||
q,1793911420052.3,1793911420054.3,1793911420088.6,503,,,39e797bfa6f6,
|
||||
q,1793911420176.5,1793911420177.6,1793911420215.6,503,,,7bbaa8668c87,
|
||||
q,1793911420300.7,1793911420302.3,1793911420343.4,503,,,39e797bfa6f6,
|
||||
q,1793911420425.0,1793911420429.1,1793911420469.1,503,,,7bbaa8668c87,
|
||||
q,1793911420549.2,1793911420551.8,1793911420589.4,503,,,39e797bfa6f6,
|
||||
q,1793911420673.4,1793911420674.1,1793911420714.5,503,,,7bbaa8668c87,
|
||||
q,1793911420797.7,1793911420804.9,1793911420846.3,503,,,39e797bfa6f6,
|
||||
q,1793911420921.9,1793911420924.5,1793911420963.8,503,,,7bbaa8668c87,
|
||||
q,1793911421046.1,1793911421050.8,1793911421083.7,503,,,39e797bfa6f6,
|
||||
q,1793911421170.3,1793911421172.5,1793911421209.5,503,,,7bbaa8668c87,
|
||||
q,1793911421294.6,1793911421296.2,1793911421336.6,503,,,39e797bfa6f6,
|
||||
q,1793911421418.8,1793911421424.0,1793911421465.9,503,,,7bbaa8668c87,
|
||||
q,1793911421543.0,1793911421551.8,1793911421594.3,503,,,39e797bfa6f6,
|
||||
q,1793911421667.2,1793911421670.2,1793911421710.1,503,,,7bbaa8668c87,
|
||||
q,1793911421791.5,1793911421796.9,1793911421827.9,503,,,39e797bfa6f6,
|
||||
q,1793911421915.7,1793911421917.4,1793911421949.1,503,,,7bbaa8668c87,
|
||||
q,1793911422039.9,1793911422041.0,1793911422072.3,503,,,39e797bfa6f6,
|
||||
q,1793911422164.1,1793911422165.5,1793911422194.9,503,,,7bbaa8668c87,
|
||||
q,1793911422288.4,1793911422296.5,1793911422331.9,503,,,39e797bfa6f6,
|
||||
q,1793911422412.6,1793911422416.4,1793911422454.3,503,,,7bbaa8668c87,
|
||||
q,1793911422536.8,1793911422541.1,1793911422581.1,503,,,39e797bfa6f6,
|
||||
q,1793911422661.0,1793911422672.0,1793911422713.6,503,,,7bbaa8668c87,
|
||||
q,1793911422785.3,1793911422787.7,1793911422826.6,503,,,39e797bfa6f6,
|
||||
q,1793911422909.5,1793911422910.5,1793911422952.2,503,,,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"rate": 8.04982,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793911383.033,
|
||||
"end": 1793911423.034,
|
||||
"finished": 1793911422.951,
|
||||
"scheduled": 322,
|
||||
"sent": 322,
|
||||
"accepted": 14,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 308,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 14,
|
||||
"503": 308
|
||||
},
|
||||
"upload_p50_ms": 94.66,
|
||||
"upload_p95_ms": 637.54,
|
||||
"upload_max_ms": 637.54,
|
||||
"peak_inflight": 12,
|
||||
"gen_cpu_s": 0.741,
|
||||
"log": "/tmp/c4p4-l-r1-overload.log"
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
q,1793911423625.9,1793911423631.7,1793911423693.8,503,,,39e797bfa6f6,
|
||||
q,1793911424040.0,1793911424043.9,1793911424082.2,503,,,7bbaa8668c87,
|
||||
q,1793911424454.0,1793911424456.6,1793911424516.2,503,,,39e797bfa6f6,
|
||||
q,1793911424868.1,1793911424874.1,1793911424934.1,503,,,7bbaa8668c87,
|
||||
q,1793911425282.2,1793911425285.3,1793911425340.0,503,,,39e797bfa6f6,
|
||||
q,1793911425696.3,1793911425700.1,1793911425761.6,503,,,7bbaa8668c87,
|
||||
q,1793911426110.4,1793911426114.9,1793911426157.7,503,,,39e797bfa6f6,
|
||||
q,1793911426524.5,1793911426527.7,1793911426586.8,503,,,7bbaa8668c87,
|
||||
q,1793911426938.6,1793911426940.4,1793911427004.8,503,,,39e797bfa6f6,
|
||||
q,1793911427352.7,1793911427355.3,1793911427385.3,503,,,7bbaa8668c87,
|
||||
q,1793911427766.7,1793911427772.8,1793911427818.9,503,,,39e797bfa6f6,
|
||||
q,1793911428180.8,1793911428187.7,1793911428251.3,503,,,7bbaa8668c87,
|
||||
q,1793911428594.9,1793911428604.2,1793911428667.0,503,,,39e797bfa6f6,
|
||||
q,1793911429009.0,1793911429011.2,1793911429058.0,503,,,7bbaa8668c87,
|
||||
q,1793911429423.1,1793911429426.1,1793911429483.8,503,,,39e797bfa6f6,
|
||||
q,1793911429837.2,1793911429841.0,1793911429900.1,503,,,7bbaa8668c87,
|
||||
q,1793911430251.3,1793911430258.4,1793911430318.5,503,,,39e797bfa6f6,
|
||||
q,1793911430665.4,1793911430668.9,1793911430729.1,503,,,7bbaa8668c87,
|
||||
q,1793911431079.4,1793911431082.0,1793911431120.6,503,,,39e797bfa6f6,
|
||||
q,1793911431493.5,1793911431496.0,1793911431555.1,503,,,7bbaa8668c87,
|
||||
q,1793911431907.6,1793911431909.5,1793911431969.5,503,,,39e797bfa6f6,
|
||||
q,1793911432321.7,1793911432325.9,1793911432385.8,503,,,7bbaa8668c87,
|
||||
q,1793911432735.8,1793911432738.3,1793911432771.5,503,,,39e797bfa6f6,
|
||||
q,1793911433149.9,1793911433152.2,1793911433198.8,503,,,7bbaa8668c87,
|
||||
q,1793911433564.0,1793911433568.1,1793911433629.8,503,,,39e797bfa6f6,
|
||||
q,1793911433978.1,1793911433981.2,1793911434042.1,503,,,7bbaa8668c87,
|
||||
q,1793911434392.1,1793911434396.2,1793911434425.9,503,,,39e797bfa6f6,
|
||||
q,1793911434806.2,1793911434808.7,1793911434870.7,503,,,7bbaa8668c87,
|
||||
q,1793911435220.3,1793911435233.2,1793911435294.6,503,,,39e797bfa6f6,
|
||||
q,1793911435634.4,1793911435637.6,1793911435698.6,503,,,7bbaa8668c87,
|
||||
q,1793911436048.5,1793911436051.3,1793911436087.2,503,,,39e797bfa6f6,
|
||||
q,1793911436462.6,1793911436464.9,1793911436524.5,503,,,7bbaa8668c87,
|
||||
q,1793911436876.7,1793911436880.5,1793911436940.4,503,,,39e797bfa6f6,
|
||||
q,1793911437290.8,1793911437293.3,1793911437354.3,503,,,7bbaa8668c87,
|
||||
q,1793911437704.8,1793911437710.9,1793911437770.6,503,,,39e797bfa6f6,
|
||||
q,1793911438118.9,1793911438120.9,1793911438151.1,503,,,7bbaa8668c87,
|
||||
q,1793911438533.0,1793911438535.8,1793911438585.3,503,,,39e797bfa6f6,
|
||||
q,1793911438947.1,1793911438952.7,1793911439011.9,503,,,7bbaa8668c87,
|
||||
q,1793911439361.2,1793911439364.2,1793911439423.2,503,,,39e797bfa6f6,
|
||||
q,1793911439775.3,1793911439787.1,1793911439850.7,503,,,7bbaa8668c87,
|
||||
q,1793911440189.4,1793911440193.0,1793911440255.5,503,,,39e797bfa6f6,
|
||||
q,1793911440603.4,1793911440612.0,1793911440650.9,503,,,7bbaa8668c87,
|
||||
q,1793911441017.5,1793911441019.7,1793911441065.9,503,,,39e797bfa6f6,
|
||||
q,1793911441431.6,1793911441434.1,1793911441495.5,503,,,7bbaa8668c87,
|
||||
q,1793911441845.7,1793911441848.5,1793911441909.2,503,,,39e797bfa6f6,
|
||||
q,1793911442259.8,1793911442265.1,1793911442326.3,503,,,7bbaa8668c87,
|
||||
q,1793911442673.9,1793911442681.8,1793911442742.9,503,,,39e797bfa6f6,
|
||||
q,1793911443088.0,1793911443091.5,1793911443127.6,503,,,7bbaa8668c87,
|
||||
q,1793911443502.1,1793911443505.8,1793911443536.6,503,,,39e797bfa6f6,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 2.41495,
|
||||
"duration_s": 20.0,
|
||||
"start": 1793911423.626,
|
||||
"end": 1793911443.916,
|
||||
"finished": 1793911443.537,
|
||||
"scheduled": 49,
|
||||
"sent": 49,
|
||||
"accepted": 0,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 49,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"503": 49
|
||||
},
|
||||
"upload_p50_ms": null,
|
||||
"upload_p95_ms": null,
|
||||
"upload_max_ms": null,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.347,
|
||||
"log": "/tmp/c4p4-l-r1-recovery.log"
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
q,1793911301883.6,1793911301891.1,1793911302003.3,202,,eb27bc7d-9be2-4241-8bbb-15b1612629bb,7bbaa8668c87,
|
||||
q,1793911302297.7,1793911302301.1,1793911302388.2,202,,432bc22c-9bb1-4570-be79-00e90227fc2f,39e797bfa6f6,
|
||||
q,1793911302711.7,1793911302713.6,1793911302793.2,202,,d78419a9-459b-4f46-b548-aa4b805f8d3f,7bbaa8668c87,
|
||||
q,1793911303125.8,1793911303127.1,1793911303204.8,202,,ded7572d-881e-4e9d-aabc-d00dbc1ed26e,39e797bfa6f6,
|
||||
q,1793911303539.9,1793911303543.2,1793911303618.6,202,,c0bdf4ac-0e5c-45d3-8e9b-0b40841b9c00,7bbaa8668c87,
|
||||
q,1793911303954.0,1793911303966.1,1793911304053.2,202,,9ec00db0-30d2-4b7f-8ce6-9a8668a0cb55,39e797bfa6f6,
|
||||
q,1793911304368.1,1793911304379.1,1793911305135.8,202,,f02263df-bc2f-447f-9ee7-88979f4f6a82,7bbaa8668c87,
|
||||
q,1793911304782.2,1793911304783.5,1793911305143.2,202,,de55992f-7394-4488-94a2-ac3e1f08df58,39e797bfa6f6,
|
||||
q,1793911305196.3,1793911305202.9,1793911305279.5,202,,2da19f82-bbca-41f6-93de-28c6a648810d,7bbaa8668c87,
|
||||
q,1793911305610.4,1793911305617.6,1793911305746.7,202,,abb8f2a0-10a9-4176-a039-fa16d2de12fd,39e797bfa6f6,
|
||||
q,1793911306024.4,1793911306026.6,1793911306112.1,202,,1c620fdf-eb71-4f79-8e61-7f18ea358e70,7bbaa8668c87,
|
||||
q,1793911306438.5,1793911306445.8,1793911306523.1,202,,981423c3-d7e3-4c89-9bf3-a06b440760f9,39e797bfa6f6,
|
||||
q,1793911306852.6,1793911306854.6,1793911306930.4,202,,b7d2455f-ea4a-47d3-8542-4f23dd6ebfcb,7bbaa8668c87,
|
||||
q,1793911307266.7,1793911307268.1,1793911307557.9,202,,3a31e724-faa1-4276-9a04-94c2b184dff7,39e797bfa6f6,
|
||||
q,1793911307680.8,1793911307682.2,1793911307756.4,202,,eb174c65-4265-40ad-8808-b75be3d1eb52,7bbaa8668c87,
|
||||
q,1793911308094.9,1793911308098.1,1793911308189.4,202,,04a2312f-26de-4379-8841-cbfd303e06eb,39e797bfa6f6,
|
||||
q,1793911308509.0,1793911308510.2,1793911308632.2,202,,8af604ae-5d8f-4520-b6e9-83e3319c1193,7bbaa8668c87,
|
||||
q,1793911308923.1,1793911308924.4,1793911309004.7,202,,443aa3f0-b9e3-48a0-a345-daad28d8711b,39e797bfa6f6,
|
||||
q,1793911309337.1,1793911309340.1,1793911309413.5,202,,6ae0e8ae-977f-4631-b7b8-8d65475b3c09,7bbaa8668c87,
|
||||
q,1793911309751.2,1793911309752.1,1793911309836.5,202,,56a96602-d2fa-462c-91d2-14f8bd5c5cd3,39e797bfa6f6,
|
||||
q,1793911310165.3,1793911310167.5,1793911310241.9,202,,1148ae9b-ae15-442b-b3c6-c50042ae4d65,7bbaa8668c87,
|
||||
q,1793911310579.4,1793911310581.2,1793911310667.6,202,,576b36fa-0fe9-4c33-8c63-30d09a4b0a4b,39e797bfa6f6,
|
||||
q,1793911310993.5,1793911310997.5,1793911311071.5,202,,e2a6029e-5ff9-450d-835b-24a34afd28bf,7bbaa8668c87,
|
||||
q,1793911311407.6,1793911311410.1,1793911311486.8,202,,b0d1b03f-0281-4283-a871-001898906ffe,39e797bfa6f6,
|
||||
q,1793911311821.7,1793911311823.1,1793911311898.8,202,,999c3ee3-fdff-41ef-9161-34682d480161,7bbaa8668c87,
|
||||
q,1793911312235.8,1793911312239.4,1793911312316.6,202,,b0ac2d0e-c3b6-47fe-96ae-d4c30c506eab,39e797bfa6f6,
|
||||
q,1793911312649.8,1793911312651.4,1793911312732.6,202,,1dba1b56-8705-4082-8659-1df0f50262bd,7bbaa8668c87,
|
||||
q,1793911313063.9,1793911313068.2,1793911313675.3,202,,4412ae27-edbc-4636-bdd4-e02b7da541b9,39e797bfa6f6,
|
||||
q,1793911313478.0,1793911313480.2,1793911313675.7,202,,5e8f8a3c-ff2e-4abe-9b7f-c6b2aa240296,7bbaa8668c87,
|
||||
q,1793911313892.1,1793911313896.2,1793911313984.7,202,,d64f9609-b313-4108-95b3-4b20b553db16,39e797bfa6f6,
|
||||
q,1793911314306.2,1793911314309.0,1793911314402.3,202,,49b8784b-5fdb-4b2e-8be0-29d770e911cc,7bbaa8668c87,
|
||||
q,1793911314720.3,1793911314721.9,1793911314829.8,202,,27aa2e13-ed20-4732-b300-c86398d42ad7,39e797bfa6f6,
|
||||
q,1793911315134.4,1793911315136.1,1793911315213.7,202,,18c54fef-f4cf-4ac6-b11c-23de1f4d8cd2,7bbaa8668c87,
|
||||
q,1793911315548.5,1793911315551.3,1793911315625.1,202,,942844a1-1ca8-4129-9c01-4d845e4f6bbc,39e797bfa6f6,
|
||||
q,1793911315962.5,1793911315964.3,1793911316044.8,202,,2e72f489-245a-44d8-8d91-d029f1ecdacc,7bbaa8668c87,
|
||||
q,1793911316376.6,1793911316386.1,1793911316464.8,202,,20887da4-e5e5-4cef-ac26-e2c7dc54324b,39e797bfa6f6,
|
||||
q,1793911316790.7,1793911316792.4,1793911316867.5,202,,ee31c9e2-ad03-42ad-9ac1-d030796ac6e1,7bbaa8668c87,
|
||||
q,1793911317204.8,1793911317206.1,1793911317301.4,202,,842610b4-f468-4650-81df-74593e2ceaab,39e797bfa6f6,
|
||||
q,1793911317618.9,1793911317623.2,1793911317704.2,202,,12403742-eca4-466d-a47b-08519e82b321,7bbaa8668c87,
|
||||
q,1793911318033.0,1793911318034.5,1793911318109.9,202,,237fc839-9497-437a-b97a-8a724636207c,39e797bfa6f6,
|
||||
q,1793911318447.1,1793911318451.9,1793911318525.1,202,,6c91722f-1cf0-4313-a21a-fe0582c2d574,7bbaa8668c87,
|
||||
q,1793911318861.2,1793911318872.9,1793911318949.4,202,,850fc5db-a88d-4cb5-bd84-56d4e4ba5845,39e797bfa6f6,
|
||||
q,1793911319689.3,1793911319691.2,1793911319821.2,202,,cdf8afda-55bb-4beb-b551-c26af14b76fd,39e797bfa6f6,
|
||||
q,1793911319275.2,1793911319276.1,1793911319821.4,202,,54fbc54d-07a7-4501-8b3a-a9f9f5738e9a,7bbaa8668c87,
|
||||
q,1793911320103.4,1793911320105.8,1793911320206.0,202,,0a9fa68b-3880-4287-83f4-927aed5ac414,7bbaa8668c87,
|
||||
q,1793911320517.5,1793911320519.2,1793911320637.6,202,,1e2fd017-d6ee-4945-952c-d6ada93064d3,39e797bfa6f6,
|
||||
q,1793911320931.6,1793911320932.1,1793911321022.4,202,,f452d3a7-c189-4a2a-b363-32311cb44906,7bbaa8668c87,
|
||||
q,1793911321345.7,1793911321347.6,1793911321433.6,202,,1eb1ffad-d136-486c-80d0-6395f7791ffc,39e797bfa6f6,
|
||||
q,1793911321759.8,1793911321762.1,1793911321837.1,202,,14cff46d-8f99-4f5e-9ea6-2b04d99311a5,7bbaa8668c87,
|
||||
q,1793911322173.8,1793911322175.1,1793911322248.6,202,,cc796a73-2fd7-42fa-90a2-c5ccb51b5402,39e797bfa6f6,
|
||||
q,1793911322587.9,1793911322599.1,1793911322675.4,202,,ce5fe396-b8e3-4a09-85c6-5840c0425835,7bbaa8668c87,
|
||||
q,1793911323002.0,1793911323005.2,1793911323082.0,202,,57bc8297-4581-40fa-a396-5d70d4a1b004,39e797bfa6f6,
|
||||
q,1793911323416.1,1793911323418.6,1793911323930.6,202,,6f05b811-cca3-406e-aca5-a3b43154a052,7bbaa8668c87,
|
||||
q,1793911323830.2,1793911323831.6,1793911323960.4,202,,b6b9721e-31f8-48c1-90e9-03bfc5c7f813,39e797bfa6f6,
|
||||
q,1793911324244.3,1793911324251.0,1793911324346.3,202,,3ec55ab0-6794-42b3-a02e-444bbbb56c73,7bbaa8668c87,
|
||||
q,1793911324658.4,1793911324659.4,1793911324738.9,202,,d459d073-b2a2-4425-9b0d-951df19bcc14,39e797bfa6f6,
|
||||
q,1793911325072.5,1793911325076.6,1793911325153.6,202,,a1b2bbe4-259d-4491-8649-e5a5cfdeefd1,7bbaa8668c87,
|
||||
q,1793911325486.5,1793911325488.0,1793911325568.4,202,,4f3f5956-48a2-45e7-9d94-ebb5f47515d1,39e797bfa6f6,
|
||||
q,1793911325900.6,1793911325903.1,1793911326053.4,202,,2689ece4-9123-4fa6-a1e3-2925fa575f54,7bbaa8668c87,
|
||||
q,1793911326314.7,1793911326320.3,1793911326395.6,202,,975522c2-6197-4c88-8dd6-822940a3c2d3,39e797bfa6f6,
|
||||
q,1793911326728.8,1793911326731.4,1793911326804.2,202,,5d3ec9ec-dd55-4735-a597-79df3cd1b663,7bbaa8668c87,
|
||||
q,1793911327142.9,1793911327145.2,1793911328017.6,202,,09207c25-6e4f-4d1d-b843-820898901f53,39e797bfa6f6,
|
||||
q,1793911327557.0,1793911327558.4,1793911328019.6,202,,cd350747-5bfd-427e-82bf-de0358bca59b,7bbaa8668c87,
|
||||
q,1793911327971.1,1793911327973.2,1793911328126.8,202,,23a8175f-9008-4b02-9de9-17b186e54656,39e797bfa6f6,
|
||||
q,1793911328385.2,1793911328386.1,1793911328482.8,202,,4f3dbbb1-79b7-412a-972a-a94af6950e97,7bbaa8668c87,
|
||||
q,1793911328799.2,1793911328804.6,1793911328884.8,202,,d85df1a4-cbbb-4604-a027-360e365e33c3,39e797bfa6f6,
|
||||
q,1793911329213.3,1793911329214.2,1793911329291.9,202,,cffb0609-291a-4a86-b729-4c444fb943a6,7bbaa8668c87,
|
||||
q,1793911329627.4,1793911329629.1,1793911329705.3,202,,896b3633-b957-4b65-b35e-f4d97a7a155e,39e797bfa6f6,
|
||||
q,1793911330041.5,1793911330043.1,1793911330224.4,202,,f597ffb8-5929-45b1-8b5a-52c97f19937e,7bbaa8668c87,
|
||||
q,1793911330455.6,1793911330467.1,1793911330541.9,202,,26be7851-94e7-457e-9e3d-49e0352ceeb7,39e797bfa6f6,
|
||||
q,1793911330869.7,1793911330878.6,1793911330961.9,202,,22d11913-e0c0-4816-81e1-93afc8485463,7bbaa8668c87,
|
||||
q,1793911331697.9,1793911331699.5,1793911332139.6,202,,026c83e9-bdde-46a2-9c23-7dd0d95f1f4e,7bbaa8668c87,
|
||||
q,1793911331283.8,1793911331285.6,1793911332143.8,202,,d2f213c8-2af5-4749-a237-b6623cfdc71b,39e797bfa6f6,
|
||||
q,1793911332111.9,1793911332114.5,1793911332237.9,202,,8ee35c71-1409-4c93-9db0-700631f0fc55,39e797bfa6f6,
|
||||
q,1793911332526.0,1793911332529.4,1793911332719.8,202,,23e759fb-26f1-4a23-a8e8-301a758a6ad0,7bbaa8668c87,
|
||||
q,1793911332940.1,1793911332942.1,1793911333014.9,202,,a0929f18-8182-496e-a961-28bbc7504569,39e797bfa6f6,
|
||||
q,1793911333354.2,1793911333355.8,1793911334131.0,202,,6353ae93-df87-4b0a-9986-8bebce24078c,7bbaa8668c87,
|
||||
q,1793911333768.3,1793911333769.6,1793911334131.7,202,,10db0a87-87a6-4393-be31-29f045043ac6,39e797bfa6f6,
|
||||
q,1793911334182.4,1793911334183.1,1793911334270.3,202,,77bbe73e-13cf-4cab-af28-d073e487deeb,7bbaa8668c87,
|
||||
q,1793911334596.5,1793911334598.5,1793911334672.4,202,,df6fa898-4132-44a2-8ea1-5de0c1704c20,39e797bfa6f6,
|
||||
q,1793911335010.6,1793911335012.2,1793911335083.5,202,,0a4749b7-ce47-4efb-8522-ea0a1f63656c,7bbaa8668c87,
|
||||
q,1793911335424.6,1793911335426.3,1793911336141.3,202,,f547c6a9-5b30-46c7-826e-3d576126e1b1,39e797bfa6f6,
|
||||
q,1793911335838.7,1793911335840.5,1793911336141.4,202,,76d07f06-5796-49ae-8527-c30ae2242e82,7bbaa8668c87,
|
||||
q,1793911336252.8,1793911336259.1,1793911336361.5,202,,eabba86d-5656-4aa8-82a3-d31c3b6d734b,39e797bfa6f6,
|
||||
q,1793911336666.9,1793911336668.4,1793911336750.3,202,,b0314e02-a503-416c-9ec9-41a9ef9a283a,7bbaa8668c87,
|
||||
q,1793911337081.0,1793911337084.1,1793911337158.7,202,,7fdd71d7-317f-472c-9745-3e6901c4e067,39e797bfa6f6,
|
||||
q,1793911337495.1,1793911337502.4,1793911337575.1,202,,f49bb1b0-05eb-477d-a188-f828aea2eb03,7bbaa8668c87,
|
||||
q,1793911337909.2,1793911337911.1,1793911337986.8,202,,8a58ae16-7538-4443-830e-58cf2a697d6b,39e797bfa6f6,
|
||||
q,1793911338323.3,1793911338325.5,1793911338405.7,202,,538840fa-20cf-47b5-b344-05ff2d5b0ced,7bbaa8668c87,
|
||||
q,1793911338737.3,1793911338739.8,1793911338840.0,202,,83d1bbbf-34f6-4b88-92f1-ec86930b23e9,39e797bfa6f6,
|
||||
q,1793911339151.4,1793911339156.6,1793911339273.7,202,,1cf8f464-fd23-44ea-9a0a-adac99236394,7bbaa8668c87,
|
||||
q,1793911339565.5,1793911339568.1,1793911339649.7,202,,ad046f5c-10d5-4994-a9eb-06910e60e18b,39e797bfa6f6,
|
||||
q,1793911339979.6,1793911339981.1,1793911340060.9,202,,0eb14ad1-8b51-4592-81e1-0d6b39ad1e09,7bbaa8668c87,
|
||||
q,1793911340393.7,1793911340403.1,1793911340481.2,202,,a4ef68e7-c1f8-4d22-96f0-60bd4b9b7da8,39e797bfa6f6,
|
||||
q,1793911340807.8,1793911340809.2,1793911340887.0,202,,4bae8377-9b5a-4290-b1da-8ee90d48e601,7bbaa8668c87,
|
||||
q,1793911341221.9,1793911341223.1,1793911341365.5,202,,efb0e1a8-6d20-4a62-8f2c-ee01d19f5276,39e797bfa6f6,
|
||||
q,1793911341635.9,1793911341637.6,1793911341717.9,202,,d2ca4d85-4bcd-4011-803c-9f976f79d345,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 2.41495,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793911301.884,
|
||||
"end": 1793911342.05,
|
||||
"finished": 1793911341.717,
|
||||
"scheduled": 97,
|
||||
"sent": 97,
|
||||
"accepted": 97,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 97
|
||||
},
|
||||
"upload_p50_ms": 81.57,
|
||||
"upload_p95_ms": 715.01,
|
||||
"upload_max_ms": 872.44,
|
||||
"peak_inflight": 3,
|
||||
"gen_cpu_s": 0.257,
|
||||
"log": "/tmp/c4p4-l-r1-steady.log"
|
||||
}
|
||||
@@ -0,0 +1,314 @@
|
||||
2026-11-05T20:41:19.659Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=4 prefetch=4 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-
|
||||
2026-11-05T20:41:19.666Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=4
|
||||
2026-11-05T20:41:20.032Z INFO src.worker [rid=-] worker stopped exit_code=0 cancelled=0 returned=0
|
||||
2026-11-05T20:41:31.310Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=4 prefetch=4 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-
|
||||
2026-11-05T20:41:31.317Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=4
|
||||
2026-11-05T20:41:32.359Z INFO src.services.processing [rid=-] attachment processed attachment_id=022f5f01-3536-4bae-a7eb-914d34af5e8c job_id=e09d01d2-b44b-4c31-b1a8-a931e988825d attempt=1 variants=3 render_peak_kib=127920 took=1033ms waited=9972ms
|
||||
2026-11-05T20:41:32.362Z INFO src.services.processing [rid=-] attachment processed attachment_id=49549fee-5b11-4479-b836-c487c6637b71 job_id=c983cf27-b5b4-4ac7-b38b-906611bf6677 attempt=1 variants=3 render_peak_kib=154976 took=1042ms waited=10696ms
|
||||
2026-11-05T20:41:32.377Z INFO src.services.processing [rid=-] attachment processed attachment_id=65d1d867-76a5-4b5e-a0ff-1f38dd65b67e job_id=0459d397-47d6-4491-9add-56c4dfc75edc attempt=1 variants=3 render_peak_kib=131008 took=1057ms waited=10456ms
|
||||
2026-11-05T20:41:32.431Z INFO src.services.processing [rid=-] attachment processed attachment_id=a491897b-510b-4155-b5f8-b205f4df32ba job_id=f823d3fd-5a7e-4c68-a8e2-a17380acddcf attempt=1 variants=3 render_peak_kib=143984 took=1105ms waited=10222ms
|
||||
2026-11-05T20:41:33.455Z INFO src.services.processing [rid=-] attachment processed attachment_id=da181c03-4baa-49de-a8db-2e240f879487 job_id=103b995e-5b59-44a9-b6d4-e2c382f79afd attempt=1 variants=3 render_peak_kib=178324 took=1091ms waited=10742ms
|
||||
2026-11-05T20:41:33.472Z INFO src.services.processing [rid=-] attachment processed attachment_id=dbe7f009-fbe1-44af-af52-dee7770c5f03 job_id=df0663d9-319b-4b5f-abff-d09229c43267 attempt=1 variants=3 render_peak_kib=178324 took=1107ms waited=10498ms
|
||||
2026-11-05T20:41:33.473Z INFO src.services.processing [rid=-] attachment processed attachment_id=0721fcfd-4662-4ac4-a456-b0729067e2c0 job_id=5d203e1d-bfc9-455b-8951-839a34f1e02b attempt=1 variants=3 render_peak_kib=178324 took=1095ms waited=9760ms
|
||||
2026-11-05T20:41:33.475Z INFO src.services.processing [rid=-] attachment processed attachment_id=f9e286a4-653c-4af9-9e53-8fd85797cc6f job_id=40ad97f3-6fdd-4938-a742-d78bc0ea66e8 attempt=1 variants=3 render_peak_kib=178324 took=1039ms waited=10325ms
|
||||
2026-11-05T20:41:34.476Z INFO src.services.processing [rid=-] attachment processed attachment_id=5d9f3e43-d067-43ec-b2c3-7750c2f4625d job_id=112a5c12-e99f-421b-9eb6-4b86a7bbc64f attempt=1 variants=3 render_peak_kib=178672 took=1018ms waited=11077ms
|
||||
2026-11-05T20:41:34.477Z INFO src.services.processing [rid=-] attachment processed attachment_id=6172e2e2-860c-4aba-a27c-d98080487ffa job_id=47dee9a9-64af-40d7-8c1a-b1f831f8cc1c attempt=1 variants=3 render_peak_kib=178672 took=998ms waited=10100ms
|
||||
2026-11-05T20:41:34.478Z INFO src.services.processing [rid=-] attachment processed attachment_id=c4f79ddb-bef8-43ac-a6ce-8430426b40e5 job_id=a3752697-22c9-400b-956d-f0aef6d8817e attempt=1 variants=3 render_peak_kib=178672 took=1002ms waited=10373ms
|
||||
2026-11-05T20:41:34.498Z INFO src.services.processing [rid=-] attachment processed attachment_id=5d0d049a-981f-4ac7-974a-6fe8551555e9 job_id=bda16671-4db5-483a-b09e-3ac034799b3c attempt=1 variants=3 render_peak_kib=178672 took=1024ms waited=10608ms
|
||||
2026-11-05T20:41:35.569Z INFO src.services.processing [rid=-] attachment processed attachment_id=03ec7e14-83fd-4803-86f7-98d8ed6e2086 job_id=2c271c98-c299-4c4a-bcd0-af4c96c9edac attempt=1 variants=3 render_peak_kib=178672 took=1090ms waited=10363ms
|
||||
2026-11-05T20:41:35.613Z INFO src.services.processing [rid=-] attachment processed attachment_id=908db4b0-84a1-4777-be51-63f5668b982a job_id=80f2a5b9-ee02-4408-b522-7b9e545a8e84 attempt=1 variants=3 render_peak_kib=178672 took=1113ms waited=10129ms
|
||||
2026-11-05T20:41:35.619Z INFO src.services.processing [rid=-] attachment processed attachment_id=741f4f0e-4e23-4ba3-8e3a-4e553b202eb9 job_id=6a187790-959e-491e-b771-493faa40fbd3 attempt=1 variants=3 render_peak_kib=178672 took=1140ms waited=10607ms
|
||||
2026-11-05T20:41:35.622Z INFO src.services.processing [rid=-] attachment processed attachment_id=f5263c27-4a82-4f25-a7cd-b4808fe0ab44 job_id=ba6986c4-94ad-4cd0-b0d4-175fbec8a3d4 attempt=1 variants=3 render_peak_kib=178672 took=1143ms waited=10870ms
|
||||
2026-11-05T20:41:36.369Z INFO src.services.processing [rid=-] attachment processed attachment_id=f0edabb3-98cb-4125-94da-e1bb4f6a2c08 job_id=e4a5243a-040f-4198-bf99-432194e0d96b attempt=1 variants=3 render_peak_kib=178808 took=798ms waited=10945ms
|
||||
2026-11-05T20:41:36.575Z INFO src.services.processing [rid=-] attachment processed attachment_id=f180408f-a109-461e-9160-382604bd9cdb job_id=83a226fe-1305-4f13-9e86-ebbe67957b6f attempt=1 variants=3 render_peak_kib=178808 took=953ms waited=10512ms
|
||||
2026-11-05T20:41:36.575Z INFO src.services.processing [rid=-] attachment processed attachment_id=93ea2cf7-1dc9-4e83-88ed-fb7ef9c45215 job_id=781e7688-e375-4436-ad8a-d2fea9276efa attempt=1 variants=3 render_peak_kib=178808 took=956ms waited=10752ms
|
||||
2026-11-05T20:41:36.579Z INFO src.services.processing [rid=-] attachment processed attachment_id=759567b7-4f39-4201-98c8-067224f7d4df job_id=65c3f59f-615d-47e6-add5-e89db6aa4bce attempt=1 variants=3 render_peak_kib=178808 took=943ms waited=10265ms
|
||||
2026-11-05T20:41:37.056Z INFO src.services.processing [rid=-] attachment processed attachment_id=90539c88-f789-4710-884f-00f183558f6d job_id=75810a89-ba2c-47fc-b291-9c6336d77825 attempt=1 variants=3 render_peak_kib=178808 took=674ms waited=10767ms
|
||||
2026-11-05T20:41:37.553Z INFO src.services.processing [rid=-] attachment processed attachment_id=24c79088-c500-443e-a125-04d32a29b956 job_id=2ec4e3e0-18a6-4c98-a0b6-23832b6a56ab attempt=1 variants=3 render_peak_kib=178808 took=972ms waited=10227ms
|
||||
2026-11-05T20:41:37.553Z INFO src.services.processing [rid=-] attachment processed attachment_id=f39bc846-9447-48ea-b77b-62de889204b2 job_id=0c466eb1-bb81-4bca-a339-b091921752e8 attempt=1 variants=3 render_peak_kib=178808 took=976ms waited=10710ms
|
||||
2026-11-05T20:41:37.558Z INFO src.services.processing [rid=-] attachment processed attachment_id=6d18ba0b-394f-4fdc-b7d8-743e19ba9210 job_id=26e20f0d-7376-430a-a55a-7497d6effa82 attempt=1 variants=3 render_peak_kib=178808 took=981ms waited=10468ms
|
||||
2026-11-05T20:41:37.675Z INFO src.services.processing [rid=-] attachment processed attachment_id=19dfe59f-c4f4-4705-af68-cbc5583995be job_id=47c1571c-30cb-493b-a3ab-25712cdfd360 attempt=1 variants=3 render_peak_kib=178808 took=617ms waited=10454ms
|
||||
2026-11-05T20:41:38.561Z INFO src.services.processing [rid=-] attachment processed attachment_id=fdcb3b24-447e-4215-bb35-1a9032d02d0f job_id=09c73459-a3d7-412b-bd40-e5badbe2dfa9 attempt=1 variants=3 render_peak_kib=178808 took=1005ms waited=10448ms
|
||||
2026-11-05T20:41:38.562Z INFO src.services.processing [rid=-] attachment processed attachment_id=826eb603-29b0-483f-84fc-d110bac3aa41 job_id=ec75e342-c2f5-4270-a4f0-8df65773298a attempt=1 variants=3 render_peak_kib=178808 took=1006ms waited=10699ms
|
||||
2026-11-05T20:41:38.562Z INFO src.services.processing [rid=-] attachment processed attachment_id=2a5d3296-5b4a-47ec-a0b7-80a46d685837 job_id=509f2557-8585-4025-b741-00e8fba85d2d attempt=1 variants=3 render_peak_kib=178808 took=1001ms waited=10189ms
|
||||
2026-11-05T20:41:38.563Z INFO src.services.processing [rid=-] attachment processed attachment_id=e8f302f6-0ba9-49c5-bfde-c5512700d50d job_id=05c49ecc-bd55-4f18-9697-0fd855449cd4 attempt=1 variants=3 render_peak_kib=178808 took=886ms waited=10056ms
|
||||
2026-11-05T20:41:39.548Z INFO src.services.processing [rid=-] attachment processed attachment_id=cbd723eb-2664-4ae1-80a2-46e95e4e182d job_id=a145357a-922a-48e5-86ae-d98cb9a16a6b attempt=1 variants=3 render_peak_kib=179144 took=984ms waited=10190ms
|
||||
2026-11-05T20:41:39.560Z INFO src.services.processing [rid=-] attachment processed attachment_id=4e3c879c-eb25-470b-9145-aa64d9abf94e job_id=808dc470-ed73-4f0b-8895-c3a3760082ef attempt=1 variants=3 render_peak_kib=179144 took=996ms waited=9947ms
|
||||
2026-11-05T20:41:39.571Z INFO src.services.processing [rid=-] attachment processed attachment_id=61ef985a-9a9d-4f57-8dc1-5dade858ba75 job_id=0082828c-1379-4341-931e-16c1da73b361 attempt=1 variants=3 render_peak_kib=179144 took=1008ms waited=10453ms
|
||||
2026-11-05T20:41:39.588Z INFO src.services.processing [rid=-] attachment processed attachment_id=37ff09b9-be68-4523-99b8-6fa6f1d8a08a job_id=30fa7518-6b6c-49a7-903b-c2e6ac312c02 attempt=1 variants=3 render_peak_kib=179144 took=1025ms waited=10684ms
|
||||
2026-11-05T20:41:40.558Z INFO src.services.processing [rid=-] attachment processed attachment_id=67dc5b17-ab0b-466e-b32a-70f56722879c job_id=c45d0312-0ebf-4f26-988a-f92abb808b9e attempt=1 variants=3 render_peak_kib=179144 took=1010ms waited=10671ms
|
||||
2026-11-05T20:41:40.559Z INFO src.services.processing [rid=-] attachment processed attachment_id=e9b7e577-24a7-4a44-9e6d-5b807e1e7279 job_id=1634c4c7-466b-41d7-9225-88c3734f45ec attempt=1 variants=3 render_peak_kib=179144 took=987ms waited=10469ms
|
||||
2026-11-05T20:41:40.559Z INFO src.services.processing [rid=-] attachment processed attachment_id=60b74a6c-6725-4439-897b-99a2252a5bbf job_id=50c32b70-cd96-4bf5-b4b4-8f260f961f71 attempt=1 variants=3 render_peak_kib=179144 took=986ms waited=10203ms
|
||||
2026-11-05T20:41:40.561Z INFO src.services.processing [rid=-] attachment processed attachment_id=92e6ac5e-321f-45b9-91de-567c446bfe35 job_id=dbd222ca-809e-48e2-b536-8313f69ce62c attempt=1 variants=3 render_peak_kib=179144 took=966ms waited=9974ms
|
||||
2026-11-05T20:41:41.254Z INFO src.services.processing [rid=-] attachment processed attachment_id=6f02cdb0-833d-49c6-a285-79ccd88d11ce job_id=30585e31-dd25-48a5-8864-2f9aa7766bd7 attempt=1 variants=3 render_peak_kib=179828 took=694ms waited=10690ms
|
||||
2026-11-05T20:41:41.254Z INFO src.services.processing [rid=-] attachment processed attachment_id=9a9579f8-85d1-4171-9f92-e33b0c539942 job_id=6b6ce906-fd0a-4111-9766-ffbbd6d898ed attempt=1 variants=3 render_peak_kib=179828 took=693ms waited=10190ms
|
||||
2026-11-05T20:41:41.256Z INFO src.services.processing [rid=-] attachment processed attachment_id=2f7e7b12-030d-44df-9898-206a2aef2dcb job_id=250959fb-96c1-4bd6-b07c-bc001ad3ad32 attempt=1 variants=3 render_peak_kib=179828 took=696ms waited=10452ms
|
||||
2026-11-05T20:41:42.516Z INFO src.services.processing [rid=-] attachment processed attachment_id=eb27bc7d-9be2-4241-8bbb-15b1612629bb job_id=1a6fcbbf-ff63-42d1-b7e0-ba062489b349 attempt=1 variants=3 render_peak_kib=179828 took=423ms waited=156ms
|
||||
2026-11-05T20:41:43.031Z INFO src.services.processing [rid=-] attachment processed attachment_id=432bc22c-9bb1-4570-be79-00e90227fc2f job_id=6dee1336-c210-4db9-b434-d920fd86831b attempt=1 variants=3 render_peak_kib=179828 took=432ms waited=279ms
|
||||
2026-11-05T20:41:43.539Z INFO src.services.processing [rid=-] attachment processed attachment_id=d78419a9-459b-4f46-b548-aa4b805f8d3f job_id=caf44d7f-1561-4a0a-8bbe-043dd5c16f99 attempt=1 variants=3 render_peak_kib=179828 took=434ms waited=372ms
|
||||
2026-11-05T20:41:44.059Z INFO src.services.processing [rid=-] attachment processed attachment_id=ded7572d-881e-4e9d-aabc-d00dbc1ed26e job_id=b433926c-757c-414d-8212-5a21406b6793 attempt=1 variants=3 render_peak_kib=179828 took=449ms waited=463ms
|
||||
2026-11-05T20:41:45.161Z INFO src.services.processing [rid=-] attachment processed attachment_id=9ec00db0-30d2-4b7f-8ce6-9a8668a0cb55 job_id=7772d14f-2967-4255-8d36-0905b284a7fd attempt=1 variants=3 render_peak_kib=179828 took=1046ms waited=129ms
|
||||
2026-11-05T20:41:45.161Z INFO src.services.processing [rid=-] attachment processed attachment_id=c0bdf4ac-0e5c-45d3-8e9b-0b40841b9c00 job_id=eb6b6856-5f27-4a9a-9741-927ade9e4a18 attempt=1 variants=3 render_peak_kib=179828 took=1047ms waited=552ms
|
||||
2026-11-05T20:41:46.364Z INFO src.services.processing [rid=-] attachment processed attachment_id=f02263df-bc2f-447f-9ee7-88979f4f6a82 job_id=9fffe5fd-90ed-4aad-a3e1-6d32db7e2c62 attempt=1 variants=3 render_peak_kib=179828 took=728ms waited=1238ms
|
||||
2026-11-05T20:41:46.365Z INFO src.services.processing [rid=-] attachment processed attachment_id=de55992f-7394-4488-94a2-ac3e1f08df58 job_id=0ea86cac-0c37-4b1f-9161-18ddb2f86dc4 attempt=1 variants=3 render_peak_kib=179828 took=727ms waited=827ms
|
||||
2026-11-05T20:41:46.373Z INFO src.services.processing [rid=-] attachment processed attachment_id=2da19f82-bbca-41f6-93de-28c6a648810d job_id=a97021c8-b2bc-4f2a-8429-16753114878c attempt=1 variants=3 render_peak_kib=179828 took=735ms waited=412ms
|
||||
2026-11-05T20:41:46.608Z INFO src.services.processing [rid=-] attachment processed attachment_id=abb8f2a0-10a9-4176-a039-fa16d2de12fd job_id=04bac8cf-30dd-42fe-ba6e-388a6abb61bd attempt=1 variants=3 render_peak_kib=179828 took=461ms waited=485ms
|
||||
2026-11-05T20:41:46.828Z INFO src.services.processing [rid=-] attachment processed attachment_id=1c620fdf-eb71-4f79-8e61-7f18ea358e70 job_id=32e9708b-39d5-452c-9565-523518811199 attempt=1 variants=3 render_peak_kib=182032 took=462ms waited=319ms
|
||||
2026-11-05T20:41:47.068Z INFO src.services.processing [rid=-] attachment processed attachment_id=981423c3-d7e3-4c89-9bf3-a06b440760f9 job_id=79769616-5011-4a01-a56a-ba99240236b4 attempt=1 variants=3 render_peak_kib=182032 took=417ms waited=185ms
|
||||
2026-11-05T20:41:47.582Z INFO src.services.processing [rid=-] attachment processed attachment_id=b7d2455f-ea4a-47d3-8542-4f23dd6ebfcb job_id=8878d482-6790-4e84-a432-80dc6a4ebc8e attempt=1 variants=3 render_peak_kib=182032 took=423ms waited=285ms
|
||||
2026-11-05T20:41:48.085Z INFO src.services.processing [rid=-] attachment processed attachment_id=3a31e724-faa1-4276-9a04-94c2b184dff7 job_id=9e6c0ead-c610-4704-be4d-5f2afa884d07 attempt=1 variants=3 render_peak_kib=182032 took=418ms waited=380ms
|
||||
2026-11-05T20:41:48.644Z INFO src.services.processing [rid=-] attachment processed attachment_id=eb174c65-4265-40ad-8808-b75be3d1eb52 job_id=8b49222b-adc6-4c9f-861b-da6cd3c2c135 attempt=1 variants=3 render_peak_kib=182032 took=455ms waited=486ms
|
||||
2026-11-05T20:41:49.166Z INFO src.services.processing [rid=-] attachment processed attachment_id=8af604ae-5d8f-4520-b6e9-83e3319c1193 job_id=95de639c-0613-468a-9216-4a0fdcec36bc attempt=1 variants=3 render_peak_kib=182032 took=465ms waited=171ms
|
||||
2026-11-05T20:41:49.166Z INFO src.services.processing [rid=-] attachment processed attachment_id=04a2312f-26de-4379-8841-cbfd303e06eb job_id=b8a7d82f-9d91-4525-9926-3bdf55531d4f attempt=1 variants=3 render_peak_kib=182032 took=466ms waited=583ms
|
||||
2026-11-05T20:41:49.633Z INFO src.services.processing [rid=-] attachment processed attachment_id=443aa3f0-b9e3-48a0-a345-daad28d8711b job_id=c16fee89-f33f-4e75-9d14-57e755eb24e9 attempt=1 variants=3 render_peak_kib=182032 took=428ms waited=261ms
|
||||
2026-11-05T20:41:50.139Z INFO src.services.processing [rid=-] attachment processed attachment_id=6ae0e8ae-977f-4631-b7b8-8d65475b3c09 job_id=fa673260-f0c5-4024-b254-36537cd7deb2 attempt=1 variants=3 render_peak_kib=182032 took=423ms waited=356ms
|
||||
2026-11-05T20:41:50.667Z INFO src.services.processing [rid=-] attachment processed attachment_id=56a96602-d2fa-462c-91d2-14f8bd5c5cd3 job_id=77d5cc6f-e219-4e80-9476-01c814ad17a3 attempt=1 variants=3 render_peak_kib=182032 took=447ms waited=449ms
|
||||
2026-11-05T20:41:51.263Z INFO src.services.processing [rid=-] attachment processed attachment_id=576b36fa-0fe9-4c33-8c63-30d09a4b0a4b job_id=8cf186fb-0db6-4837-b988-cc2ba35d232a attempt=1 variants=3 render_peak_kib=182032 took=529ms waited=133ms
|
||||
2026-11-05T20:41:51.276Z INFO src.services.processing [rid=-] attachment processed attachment_id=1148ae9b-ae15-442b-b3c6-c50042ae4d65 job_id=5a3f3300-6787-4575-9c55-fe3b5c89b969 attempt=1 variants=3 render_peak_kib=182032 took=542ms waited=547ms
|
||||
2026-11-05T20:41:51.695Z INFO src.services.processing [rid=-] attachment processed attachment_id=e2a6029e-5ff9-450d-835b-24a34afd28bf job_id=cd362dea-d1b5-43ff-be73-5910808c293b attempt=1 variants=3 render_peak_kib=182032 took=432ms waited=247ms
|
||||
2026-11-05T20:41:52.201Z INFO src.services.processing [rid=-] attachment processed attachment_id=b0d1b03f-0281-4283-a871-001898906ffe job_id=a32b309d-2f34-4a5b-a138-12ead1c6bcc1 attempt=1 variants=3 render_peak_kib=182032 took=422ms waited=350ms
|
||||
2026-11-05T20:41:52.733Z INFO src.services.processing [rid=-] attachment processed attachment_id=999c3ee3-fdff-41ef-9161-34682d480161 job_id=678cc1f6-fd8e-463a-9a18-90acd16865cb attempt=1 variants=3 render_peak_kib=182032 took=448ms waited=442ms
|
||||
2026-11-05T20:41:53.697Z INFO src.services.processing [rid=-] attachment processed attachment_id=b0ac2d0e-c3b6-47fe-96ae-d4c30c506eab job_id=a2714401-273e-4611-a2fd-7c8788ec9ac5 attempt=1 variants=3 render_peak_kib=182032 took=906ms waited=533ms
|
||||
2026-11-05T20:41:53.697Z INFO src.services.processing [rid=-] attachment processed attachment_id=1dba1b56-8705-4082-8659-1df0f50262bd job_id=78dd482c-6c2b-42f9-98ad-254177395684 attempt=1 variants=3 render_peak_kib=182032 took=906ms waited=120ms
|
||||
2026-11-05T20:41:54.338Z INFO src.services.processing [rid=-] attachment processed attachment_id=5e8f8a3c-ff2e-4abe-9b7f-c6b2aa240296 job_id=74ff6129-e3e9-45a7-a16d-9734930ec159 attempt=1 variants=3 render_peak_kib=182032 took=537ms waited=292ms
|
||||
2026-11-05T20:41:54.338Z INFO src.services.processing [rid=-] attachment processed attachment_id=4412ae27-edbc-4636-bdd4-e02b7da541b9 job_id=c7345fe8-19ff-427b-bb61-5a2d13e3efb9 attempt=1 variants=3 render_peak_kib=182032 took=538ms waited=714ms
|
||||
2026-11-05T20:41:54.753Z INFO src.services.processing [rid=-] attachment processed attachment_id=d64f9609-b313-4108-95b3-4b20b553db16 job_id=422d8df2-474f-450e-b449-c9f7ecb1f3d3 attempt=1 variants=3 render_peak_kib=182032 took=414ms waited=422ms
|
||||
2026-11-05T20:41:55.343Z INFO src.services.processing [rid=-] attachment processed attachment_id=27aa2e13-ed20-4732-b300-c86398d42ad7 job_id=e323de45-98cc-4b5b-8d00-5db88204698f attempt=1 variants=3 render_peak_kib=182032 took=496ms waited=105ms
|
||||
2026-11-05T20:41:55.343Z INFO src.services.processing [rid=-] attachment processed attachment_id=49b8784b-5fdb-4b2e-8be0-29d770e911cc job_id=b1c6a527-c059-4c1e-b5d4-86eb1908f50a attempt=1 variants=3 render_peak_kib=182032 took=497ms waited=517ms
|
||||
2026-11-05T20:41:55.800Z INFO src.services.processing [rid=-] attachment processed attachment_id=18c54fef-f4cf-4ac6-b11c-23de1f4d8cd2 job_id=108cf22f-5b8f-47d1-acb2-7845bc338eb8 attempt=1 variants=3 render_peak_kib=182032 took=447ms waited=198ms
|
||||
2026-11-05T20:41:56.296Z INFO src.services.processing [rid=-] attachment processed attachment_id=942844a1-1ca8-4129-9c01-4d845e4f6bbc job_id=504bc07e-7384-4f3f-aa37-196ca57edffa attempt=1 variants=3 render_peak_kib=182032 took=427ms waited=299ms
|
||||
2026-11-05T20:41:56.819Z INFO src.services.processing [rid=-] attachment processed attachment_id=2e72f489-245a-44d8-8d91-d029f1ecdacc job_id=b588c959-6fdf-4146-a966-f7dc3dede0d9 attempt=1 variants=3 render_peak_kib=182032 took=434ms waited=401ms
|
||||
2026-11-05T20:41:57.426Z INFO src.services.processing [rid=-] attachment processed attachment_id=20887da4-e5e5-4cef-ac26-e2c7dc54324b job_id=9f6047d2-f2db-4402-9fa4-d59efcb61616 attempt=1 variants=3 render_peak_kib=182032 took=531ms waited=486ms
|
||||
2026-11-05T20:41:57.435Z INFO src.services.processing [rid=-] attachment processed attachment_id=ee31c9e2-ad03-42ad-9ac1-d030796ac6e1 job_id=28c549a9-55fa-4a24-9d21-1673d2763455 attempt=1 variants=3 render_peak_kib=182032 took=539ms waited=83ms
|
||||
2026-11-05T20:41:57.832Z INFO src.services.processing [rid=-] attachment processed attachment_id=842610b4-f468-4650-81df-74593e2ceaab job_id=0c4d21d6-1121-4b1d-9004-e6b22095c1b3 attempt=1 variants=3 render_peak_kib=182032 took=424ms waited=182ms
|
||||
2026-11-05T20:41:58.339Z INFO src.services.processing [rid=-] attachment processed attachment_id=12403742-eca4-466d-a47b-08519e82b321 job_id=48425dce-6e8b-4d7e-9e10-113cb3d39bb0 attempt=1 variants=3 render_peak_kib=182032 took=425ms waited=270ms
|
||||
2026-11-05T20:41:58.831Z INFO src.services.processing [rid=-] attachment processed attachment_id=237fc839-9497-437a-b97a-8a724636207c job_id=b396f741-544e-4aee-9e77-d3cb75efab88 attempt=1 variants=3 render_peak_kib=182032 took=413ms waited=366ms
|
||||
2026-11-05T20:41:59.834Z INFO src.services.processing [rid=-] attachment processed attachment_id=6c91722f-1cf0-4313-a21a-fe0582c2d574 job_id=8a546dfd-de96-458b-b678-00bcf9f8b977 attempt=1 variants=3 render_peak_kib=182032 took=908ms waited=455ms
|
||||
2026-11-05T20:42:00.307Z INFO src.services.processing [rid=-] attachment processed attachment_id=850fc5db-a88d-4cb5-bd84-56d4e4ba5845 job_id=1c3ae611-896b-4f9f-88f8-96223e936870 attempt=1 variants=3 render_peak_kib=182032 took=854ms waited=560ms
|
||||
2026-11-05T20:42:00.545Z INFO src.services.processing [rid=-] attachment processed attachment_id=54fbc54d-07a7-4501-8b3a-a9f9f5738e9a job_id=88890237-c7d8-4595-b415-7352e7802ced attempt=1 variants=3 render_peak_kib=182032 took=587ms waited=662ms
|
||||
2026-11-05T20:42:00.556Z INFO src.services.processing [rid=-] attachment processed attachment_id=cdf8afda-55bb-4beb-b551-c26af14b76fd job_id=32cf1728-26f7-4740-bd39-59b7d7259f3a attempt=1 variants=3 render_peak_kib=182032 took=598ms waited=240ms
|
||||
2026-11-05T20:42:00.950Z INFO src.services.processing [rid=-] attachment processed attachment_id=0a9fa68b-3880-4287-83f4-927aed5ac414 job_id=9cd81d06-02d6-4375-9fb2-d1ac6641003d attempt=1 variants=3 render_peak_kib=182032 took=484ms waited=342ms
|
||||
2026-11-05T20:42:01.440Z INFO src.services.processing [rid=-] attachment processed attachment_id=1e2fd017-d6ee-4945-952c-d6ada93064d3 job_id=cc239b65-a166-4e50-b898-f56fca6b5a33 attempt=1 variants=3 render_peak_kib=182032 took=468ms waited=432ms
|
||||
2026-11-05T20:42:02.039Z INFO src.services.processing [rid=-] attachment processed attachment_id=1eb1ffad-d136-486c-80d0-6395f7791ffc job_id=b9fefe16-8aea-42ad-a2cc-01af3cd1c5fb attempt=1 variants=3 render_peak_kib=182032 took=560ms waited=112ms
|
||||
2026-11-05T20:42:02.039Z INFO src.services.processing [rid=-] attachment processed attachment_id=f452d3a7-c189-4a2a-b363-32311cb44906 job_id=5051c19d-ff71-4f1c-8950-4c2db6310a2b attempt=1 variants=3 render_peak_kib=182032 took=561ms waited=527ms
|
||||
2026-11-05T20:42:02.430Z INFO src.services.processing [rid=-] attachment processed attachment_id=14cff46d-8f99-4f5e-9ea6-2b04d99311a5 job_id=159fdb39-b22c-4eb0-8ce7-7447bb12cc5f attempt=1 variants=3 render_peak_kib=182032 took=434ms waited=215ms
|
||||
2026-11-05T20:42:02.923Z INFO src.services.processing [rid=-] attachment processed attachment_id=cc796a73-2fd7-42fa-90a2-c5ccb51b5402 job_id=97e2bd1f-92f5-461e-886e-5846ed9c6463 attempt=1 variants=3 render_peak_kib=182032 took=418ms waited=310ms
|
||||
2026-11-05T20:42:03.963Z INFO src.services.processing [rid=-] attachment processed attachment_id=ce5fe396-b8e3-4a09-85c6-5840c0425835 job_id=b0b3d2eb-b278-4cef-81e1-f148f5e41f22 attempt=1 variants=3 render_peak_kib=182032 took=951ms waited=392ms
|
||||
2026-11-05T20:42:04.467Z INFO src.services.processing [rid=-] attachment processed attachment_id=57bc8297-4581-40fa-a396-5d70d4a1b004 job_id=f80284c1-a8dc-4aa4-9506-1067a56abb34 attempt=1 variants=3 render_peak_kib=182032 took=949ms waited=490ms
|
||||
2026-11-05T20:42:04.652Z INFO src.services.processing [rid=-] attachment processed attachment_id=6f05b811-cca3-406e-aca5-a3b43154a052 job_id=4a1a79c4-afbc-4f57-b1b6-405721cc4ad7 attempt=1 variants=3 render_peak_kib=182032 took=629ms waited=584ms
|
||||
2026-11-05T20:42:04.655Z INFO src.services.processing [rid=-] attachment processed attachment_id=b6b9721e-31f8-48c1-90e9-03bfc5c7f813 job_id=36569d43-8d5d-45d0-b787-2a74d2c4123c attempt=1 variants=3 render_peak_kib=182032 took=630ms waited=151ms
|
||||
2026-11-05T20:42:05.066Z INFO src.services.processing [rid=-] attachment processed attachment_id=3ec55ab0-6794-42b3-a02e-444bbbb56c73 job_id=ee004ef0-ecc7-4df9-a991-f650273bd39c attempt=1 variants=3 render_peak_kib=182032 took=533ms waited=262ms
|
||||
2026-11-05T20:42:05.471Z INFO src.services.processing [rid=-] attachment processed attachment_id=d459d073-b2a2-4425-9b0d-951df19bcc14 job_id=4b573195-abb0-4393-9c7f-2c6096bab018 attempt=1 variants=3 render_peak_kib=182032 took=434ms waited=359ms
|
||||
2026-11-05T20:42:06.053Z INFO src.services.processing [rid=-] attachment processed attachment_id=a1b2bbe4-259d-4491-8649-e5a5cfdeefd1 job_id=41fda878-cdea-45cb-98f3-ad8e54cc6b38 attempt=1 variants=3 render_peak_kib=182032 took=510ms waited=447ms
|
||||
2026-11-05T20:42:06.475Z INFO src.services.processing [rid=-] attachment processed attachment_id=4f3f5956-48a2-45e7-9d94-ebb5f47515d1 job_id=7ee13cfe-45d8-475e-927c-884813dc8020 attempt=1 variants=3 render_peak_kib=182032 took=421ms waited=546ms
|
||||
2026-11-05T20:42:07.035Z INFO src.services.processing [rid=-] attachment processed attachment_id=975522c2-6197-4c88-8dd6-822940a3c2d3 job_id=c1c40a5f-36ae-4d51-9849-14ff1f0a5fad attempt=1 variants=3 render_peak_kib=182032 took=474ms waited=221ms
|
||||
2026-11-05T20:42:07.041Z INFO src.services.processing [rid=-] attachment processed attachment_id=2689ece4-9123-4fa6-a1e3-2925fa575f54 job_id=18b8307a-1571-4434-afa4-c8936f39a130 attempt=1 variants=3 render_peak_kib=182032 took=481ms waited=637ms
|
||||
2026-11-05T20:42:08.059Z INFO src.services.processing [rid=-] attachment processed attachment_id=5d3ec9ec-dd55-4735-a597-79df3cd1b663 job_id=086e77b2-891e-4ed9-8914-1dd2f11386d3 attempt=1 variants=3 render_peak_kib=182032 took=993ms waited=315ms
|
||||
2026-11-05T20:42:08.663Z INFO src.services.processing [rid=-] attachment processed attachment_id=09207c25-6e4f-4d1d-b843-820898901f53 job_id=c8b63c96-c255-4e0b-9a6c-75e7b69bbeea attempt=1 variants=3 render_peak_kib=182032 took=589ms waited=909ms
|
||||
2026-11-05T20:42:08.663Z INFO src.services.processing [rid=-] attachment processed attachment_id=cd350747-5bfd-427e-82bf-de0358bca59b job_id=0c131158-4444-4903-87fe-09a183f00833 attempt=1 variants=3 render_peak_kib=182032 took=589ms waited=494ms
|
||||
2026-11-05T20:42:09.145Z INFO src.services.processing [rid=-] attachment processed attachment_id=4f3dbbb1-79b7-412a-972a-a94af6950e97 job_id=80bb32bc-fe9f-463a-814c-00c5834c0058 attempt=1 variants=3 render_peak_kib=182032 took=561ms waited=179ms
|
||||
2026-11-05T20:42:09.148Z INFO src.services.processing [rid=-] attachment processed attachment_id=23a8175f-9008-4b02-9de9-17b186e54656 job_id=b1fdf99c-cc61-4608-a846-1f22778ac02f attempt=1 variants=3 render_peak_kib=182032 took=564ms waited=581ms
|
||||
2026-11-05T20:42:09.532Z INFO src.services.processing [rid=-] attachment processed attachment_id=d85df1a4-cbbb-4604-a027-360e365e33c3 job_id=e9d78d3d-0ced-4b09-97c5-2c09ddbb65c1 attempt=1 variants=3 render_peak_kib=182032 took=426ms waited=282ms
|
||||
2026-11-05T20:42:10.029Z INFO src.services.processing [rid=-] attachment processed attachment_id=cffb0609-291a-4a86-b729-4c444fb943a6 job_id=b51b3565-9473-4680-b449-34b0ac1a82a0 attempt=1 variants=3 render_peak_kib=182032 took=418ms waited=376ms
|
||||
2026-11-05T20:42:10.635Z INFO src.services.processing [rid=-] attachment processed attachment_id=896b3633-b957-4b65-b35e-f4d97a7a155e job_id=94a83273-944a-4f22-bae0-68d7c2d2df7d attempt=1 variants=3 render_peak_kib=182032 took=505ms waited=481ms
|
||||
2026-11-05T20:42:11.724Z INFO src.services.processing [rid=-] attachment processed attachment_id=f597ffb8-5929-45b1-8b5a-52c97f19937e job_id=fc9d2afc-b040-47b9-9499-7819699640e5 attempt=1 variants=3 render_peak_kib=182032 took=1088ms waited=573ms
|
||||
2026-11-05T20:42:12.160Z INFO src.services.processing [rid=-] attachment processed attachment_id=26be7851-94e7-457e-9e3d-49e0352ceeb7 job_id=6f2d769d-d9f5-4447-8315-4de26d3e0374 attempt=1 variants=3 render_peak_kib=182032 took=1524ms waited=151ms
|
||||
2026-11-05T20:42:12.160Z INFO src.services.processing [rid=-] attachment processed attachment_id=22d11913-e0c0-4816-81e1-93afc8485463 job_id=74af1ca8-6c5f-447e-ab0e-585bbb068a76 attempt=1 variants=3 render_peak_kib=182032 took=1005ms waited=259ms
|
||||
2026-11-05T20:42:12.737Z INFO src.services.processing [rid=-] attachment processed attachment_id=d2f213c8-2af5-4749-a237-b6623cfdc71b job_id=7b5e3cc9-5760-488d-b7a6-f6a0439c052a attempt=1 variants=3 render_peak_kib=182032 took=574ms waited=858ms
|
||||
2026-11-05T20:42:12.738Z INFO src.services.processing [rid=-] attachment processed attachment_id=026c83e9-bdde-46a2-9c23-7dd0d95f1f4e job_id=136b2287-0a28-4fbd-a24d-df046bd64c5c attempt=1 variants=3 render_peak_kib=182032 took=573ms waited=444ms
|
||||
2026-11-05T20:42:13.780Z INFO src.services.processing [rid=-] attachment processed attachment_id=8ee35c71-1409-4c93-9db0-700631f0fc55 job_id=183c0eb9-2ade-4ad2-93b4-a10155384396 attempt=1 variants=3 render_peak_kib=182032 took=1103ms waited=530ms
|
||||
2026-11-05T20:42:14.222Z INFO src.services.processing [rid=-] attachment processed attachment_id=23e759fb-26f1-4a23-a8e8-301a758a6ad0 job_id=a64f0bfa-e2f0-41b5-9e50-644906ef23c7 attempt=1 variants=3 render_peak_kib=182032 took=1012ms waited=661ms
|
||||
2026-11-05T20:42:14.642Z INFO src.services.processing [rid=-] attachment processed attachment_id=a0929f18-8182-496e-a961-28bbc7504569 job_id=32bf4cc5-1ee9-478d-b74c-4995a4b25b3e attempt=1 variants=3 render_peak_kib=182032 took=1431ms waited=250ms
|
||||
2026-11-05T20:42:14.849Z INFO src.services.processing [rid=-] attachment processed attachment_id=10db0a87-87a6-4393-be31-29f045043ac6 job_id=587d3e21-833f-42e9-ad3f-7e4dfddb0feb attempt=1 variants=3 render_peak_kib=182032 took=628ms waited=421ms
|
||||
2026-11-05T20:42:14.855Z INFO src.services.processing [rid=-] attachment processed attachment_id=6353ae93-df87-4b0a-9986-8bebce24078c job_id=153be5f0-1769-4272-b87a-aae36e6f0a95 attempt=1 variants=3 render_peak_kib=182032 took=634ms waited=836ms
|
||||
2026-11-05T20:42:16.164Z INFO src.services.processing [rid=-] attachment processed attachment_id=77bbe73e-13cf-4cab-af28-d073e487deeb job_id=3861946d-1513-49c3-9e77-166bc6a19839 attempt=1 variants=3 render_peak_kib=182032 took=1427ms waited=534ms
|
||||
2026-11-05T20:42:16.165Z INFO src.services.processing [rid=-] attachment processed attachment_id=df6fa898-4132-44a2-8ea1-5de0c1704c20 job_id=77335afa-7bd2-440f-9da4-731dc88531db attempt=1 variants=3 render_peak_kib=182032 took=1427ms waited=118ms
|
||||
2026-11-05T20:42:16.635Z INFO src.services.processing [rid=-] attachment processed attachment_id=0a4749b7-ce47-4efb-8522-ea0a1f63656c job_id=4d99cf69-a1af-4a6f-a600-44729f106bf1 attempt=1 variants=3 render_peak_kib=182032 took=1371ms waited=232ms
|
||||
2026-11-05T20:42:16.939Z INFO src.services.processing [rid=-] attachment processed attachment_id=f547c6a9-5b30-46c7-826e-3d576126e1b1 job_id=0c4b0939-3f1b-494d-aa8c-986140279f22 attempt=1 variants=3 render_peak_kib=182032 took=667ms waited=826ms
|
||||
2026-11-05T20:42:16.940Z INFO src.services.processing [rid=-] attachment processed attachment_id=76d07f06-5796-49ae-8527-c30ae2242e82 job_id=c19f7122-336c-4f71-bc5e-21a5b8b7bdbd attempt=1 variants=3 render_peak_kib=182032 took=668ms waited=409ms
|
||||
2026-11-05T20:42:17.333Z INFO src.services.processing [rid=-] attachment processed attachment_id=b0314e02-a503-416c-9ec9-41a9ef9a283a job_id=db940d7c-2c0e-4aed-b3e9-92f03ef4407c attempt=1 variants=3 render_peak_kib=182032 took=554ms waited=92ms
|
||||
2026-11-05T20:42:17.338Z INFO src.services.processing [rid=-] attachment processed attachment_id=eabba86d-5656-4aa8-82a3-d31c3b6d734b job_id=4ad82c04-8b0b-4889-9bf9-618f8d57f628 attempt=1 variants=3 render_peak_kib=182032 took=559ms waited=500ms
|
||||
2026-11-05T20:42:17.731Z INFO src.services.processing [rid=-] attachment processed attachment_id=7fdd71d7-317f-472c-9745-3e6901c4e067 job_id=4acf7334-bd3e-4c24-a1b7-1517a6daab0b attempt=1 variants=3 render_peak_kib=182032 took=437ms waited=191ms
|
||||
2026-11-05T20:42:18.405Z INFO src.services.processing [rid=-] attachment processed attachment_id=f49bb1b0-05eb-477d-a188-f828aea2eb03 job_id=92a1e1b2-199e-40c5-b696-3cabae8b17f2 attempt=1 variants=3 render_peak_kib=182032 took=605ms waited=279ms
|
||||
2026-11-05T20:42:18.848Z INFO src.services.processing [rid=-] attachment processed attachment_id=8a58ae16-7538-4443-830e-58cf2a697d6b job_id=4774ddab-9716-4e22-8cb3-c4751da4d78a attempt=1 variants=3 render_peak_kib=182032 took=518ms waited=400ms
|
||||
2026-11-05T20:42:19.341Z INFO src.services.processing [rid=-] attachment processed attachment_id=538840fa-20cf-47b5-b344-05ff2d5b0ced job_id=31c23bb0-8191-457d-a9f6-f7487f47203e attempt=1 variants=3 render_peak_kib=182032 took=504ms waited=491ms
|
||||
2026-11-05T20:42:19.850Z INFO src.services.processing [rid=-] attachment processed attachment_id=83d1bbbf-34f6-4b88-92f1-ec86930b23e9 job_id=0773469f-db92-44d7-93da-7db62139a8cd attempt=1 variants=3 render_peak_kib=182032 took=507ms waited=583ms
|
||||
2026-11-05T20:42:19.851Z INFO src.services.processing [rid=-] attachment processed attachment_id=1cf8f464-fd23-44ea-9a0a-adac99236394 job_id=c5092c36-947c-4305-945d-52778088fb14 attempt=1 variants=3 render_peak_kib=182032 took=508ms waited=168ms
|
||||
2026-11-05T20:42:20.273Z INFO src.services.processing [rid=-] attachment processed attachment_id=ad046f5c-10d5-4994-a9eb-06910e60e18b job_id=0b1a42dc-3df8-4041-b2e5-31745a0adae4 attempt=1 variants=3 render_peak_kib=182032 took=422ms waited=265ms
|
||||
2026-11-05T20:42:20.778Z INFO src.services.processing [rid=-] attachment processed attachment_id=0eb14ad1-8b51-4592-81e1-0d6b39ad1e09 job_id=956a49cd-11e4-40f5-acb5-2fb940ef5df5 attempt=1 variants=3 render_peak_kib=182032 took=421ms waited=357ms
|
||||
2026-11-05T20:42:21.364Z INFO src.services.processing [rid=-] attachment processed attachment_id=a4ef68e7-c1f8-4d22-96f0-60bd4b9b7da8 job_id=78ec4f8e-0eac-4794-90a5-c58563d77012 attempt=1 variants=3 render_peak_kib=182032 took=498ms waited=444ms
|
||||
2026-11-05T20:42:21.858Z INFO src.services.processing [rid=-] attachment processed attachment_id=efb0e1a8-6d20-4a62-8f2c-ee01d19f5276 job_id=a3324d5f-4382-488f-893f-2714bf1d8828 attempt=1 variants=3 render_peak_kib=182032 took=481ms waited=134ms
|
||||
2026-11-05T20:42:21.860Z INFO src.services.processing [rid=-] attachment processed attachment_id=4bae8377-9b5a-4290-b1da-8ee90d48e601 job_id=3576cd2a-bbf0-49a9-ae4e-ac6dd4351236 attempt=1 variants=3 render_peak_kib=182032 took=484ms waited=548ms
|
||||
2026-11-05T20:42:22.297Z INFO src.services.processing [rid=-] attachment processed attachment_id=d2ca4d85-4bcd-4011-803c-9f976f79d345 job_id=68c818b2-d8a2-48e0-9c5b-e5339b66a004 attempt=1 variants=3 render_peak_kib=182032 took=413ms waited=228ms
|
||||
2026-11-05T20:42:23.853Z INFO src.services.processing [rid=-] attachment processed attachment_id=329f53d0-b6fb-4789-bd98-64a4a8a82d29 job_id=acbc8436-744e-45aa-abf0-c26c38533df3 attempt=1 variants=3 render_peak_kib=182032 took=431ms waited=431ms
|
||||
2026-11-05T20:42:24.881Z INFO src.services.processing [rid=-] attachment processed attachment_id=46144556-944e-44a9-a2ef-f639ef98146d job_id=31fb4a4a-26e1-4955-a2fb-9cfcff5f4fe9 attempt=1 variants=3 render_peak_kib=182032 took=441ms waited=192ms
|
||||
2026-11-05T20:42:26.415Z INFO src.services.processing [rid=-] attachment processed attachment_id=8d741696-fe85-4f9e-801b-bb7fe520c8e3 job_id=57a557ff-f2cc-40d1-9ec5-d3a382243271 attempt=1 variants=3 render_peak_kib=182032 took=443ms waited=479ms
|
||||
2026-11-05T20:42:27.408Z INFO src.services.processing [rid=-] attachment processed attachment_id=f6bff17b-c05a-411d-9464-930960e3fd8e job_id=6667bd4d-9ad4-4e86-bccb-cf087e909bec attempt=1 variants=3 render_peak_kib=182032 took=425ms waited=235ms
|
||||
2026-11-05T20:42:28.964Z INFO src.services.processing [rid=-] attachment processed attachment_id=89629992-31d8-4d98-b376-caf62d89dbed job_id=de28ab7e-815c-44c8-a700-52b5eda48f71 attempt=1 variants=3 render_peak_kib=182032 took=445ms waited=519ms
|
||||
2026-11-05T20:42:29.992Z INFO src.services.processing [rid=-] attachment processed attachment_id=5caf39b5-cd80-4286-ad04-33d397adc8c6 job_id=28191dde-09f9-41b1-ab90-ccb3f03f05a5 attempt=1 variants=3 render_peak_kib=182032 took=459ms waited=301ms
|
||||
2026-11-05T20:42:31.487Z INFO src.services.processing [rid=-] attachment processed attachment_id=68fe736e-982a-47f6-a2f4-c35169c7aab9 job_id=55143291-3c43-4616-a88b-178ae7f7c749 attempt=1 variants=3 render_peak_kib=182032 took=437ms waited=563ms
|
||||
2026-11-05T20:42:32.505Z INFO src.services.processing [rid=-] attachment processed attachment_id=def63feb-d77b-4d4d-8a7b-b1ca4f08dea7 job_id=05209fd8-65a4-4587-a7af-34ce36f7a5ad attempt=1 variants=3 render_peak_kib=182032 took=442ms waited=347ms
|
||||
2026-11-05T20:42:33.477Z INFO src.services.processing [rid=-] attachment processed attachment_id=f56f864a-f7fc-4c52-b09c-f5b5d7923e6b job_id=b7b095d3-33ba-4c84-81bd-a9b7c9d969d8 attempt=1 variants=3 render_peak_kib=182032 took=406ms waited=77ms
|
||||
2026-11-05T20:42:35.031Z INFO src.services.processing [rid=-] attachment processed attachment_id=8b408361-2e56-415c-89f5-3faa4c5528cd job_id=a67004a2-4d4e-46da-8233-e4b6191a779c attempt=1 variants=3 render_peak_kib=182032 took=441ms waited=339ms
|
||||
2026-11-05T20:42:36.017Z INFO src.services.processing [rid=-] attachment processed attachment_id=0ea365cc-f028-432d-a3e2-5b23f20355e7 job_id=13c26c71-acb2-4fd4-9c73-570db92cc828 attempt=1 variants=3 render_peak_kib=182032 took=407ms waited=110ms
|
||||
2026-11-05T20:42:37.565Z INFO src.services.processing [rid=-] attachment processed attachment_id=828dd5e3-06f8-48d3-8da0-d936afb40939 job_id=bf47c2b0-4d17-4678-8d02-697c53270953 attempt=1 variants=3 render_peak_kib=182032 took=428ms waited=399ms
|
||||
2026-11-05T20:42:38.588Z INFO src.services.processing [rid=-] attachment processed attachment_id=882dc9fb-fa80-4dc7-b651-dc60c374247e job_id=b6f6f78f-5ffb-454e-87d4-1d1f74e4e337 attempt=1 variants=3 render_peak_kib=182032 took=440ms waited=154ms
|
||||
2026-11-05T20:42:40.085Z INFO src.services.processing [rid=-] attachment processed attachment_id=ef9139bb-93fb-4c41-ae05-ed0323ae4eb7 job_id=ad770b8c-e37b-4f83-8d33-8a12f6ba9775 attempt=1 variants=3 render_peak_kib=182032 took=417ms waited=449ms
|
||||
2026-11-05T20:42:41.111Z INFO src.services.processing [rid=-] attachment processed attachment_id=5ca0868b-6c3e-4f51-a276-6b5e5422e9f5 job_id=f2358a4e-376c-4692-ba94-6fa7d10149d8 attempt=1 variants=3 render_peak_kib=182032 took=415ms waited=204ms
|
||||
2026-11-05T20:42:42.677Z INFO src.services.processing [rid=-] attachment processed attachment_id=c0569b14-f9c3-45b9-94d9-334949c9685d job_id=88dc8a4c-bf1d-4c91-9d70-da582979763c attempt=1 variants=3 render_peak_kib=182032 took=447ms waited=491ms
|
||||
2026-11-05T20:42:43.666Z INFO src.services.processing [rid=-] attachment processed attachment_id=0c006a4e-d22c-4373-9d50-de9f15cb22fd job_id=5376f8c3-a0cb-4b02-a2f3-a1d95d69bc04 attempt=1 variants=3 render_peak_kib=182032 took=423ms waited=255ms
|
||||
2026-11-05T20:42:45.184Z INFO src.services.processing [rid=-] attachment processed attachment_id=91f9df6e-2bc8-45ff-9dd7-f504e48e2604 job_id=8b311a08-f427-4c4a-ab9b-7fb4e20d30e9 attempt=1 variants=3 render_peak_kib=182032 took=422ms waited=518ms
|
||||
2026-11-05T20:42:47.066Z INFO src.services.processing [rid=-] attachment processed attachment_id=303f62a7-1f63-4c7d-83e3-272966e4075c job_id=074c428e-5b45-4b4d-81b1-54c3e23f48c1 attempt=1 variants=3 render_peak_kib=182032 took=1276ms waited=291ms
|
||||
2026-11-05T20:42:47.766Z INFO src.services.processing [rid=-] attachment processed attachment_id=177dafb5-3a93-44e1-861d-48d8de5f17e9 job_id=566c1d12-7bb1-4b5b-9eff-0669a47bf421 attempt=1 variants=3 render_peak_kib=182032 took=440ms waited=582ms
|
||||
2026-11-05T20:42:48.808Z INFO src.services.processing [rid=-] attachment processed attachment_id=2f3b1673-a0b4-4f71-82ed-8343c9b5f7ea job_id=ee5fe1a5-4c10-4b53-8968-0a7f9e9c8dde attempt=1 variants=3 render_peak_kib=182032 took=459ms waited=386ms
|
||||
2026-11-05T20:42:49.766Z INFO src.services.processing [rid=-] attachment processed attachment_id=4ebf0ba5-cb5a-4d6e-a4c8-1bdaa13932de job_id=277dbe64-a1c9-4bdb-a67a-9b5b6f8aca75 attempt=1 variants=3 render_peak_kib=182032 took=405ms waited=125ms
|
||||
2026-11-05T20:42:51.369Z INFO src.services.processing [rid=-] attachment processed attachment_id=c4a78ec8-03d3-497d-affd-a9cfb09f5327 job_id=1576d426-97ce-406a-bdd9-88ada7d5c405 attempt=1 variants=3 render_peak_kib=182032 took=470ms waited=413ms
|
||||
2026-11-05T20:42:52.325Z INFO src.services.processing [rid=-] attachment processed attachment_id=7ab649d1-0209-420c-8798-fe85806ccbb2 job_id=01fdc5b1-54d6-4444-b315-bcc14b91159a attempt=1 variants=3 render_peak_kib=182032 took=416ms waited=166ms
|
||||
2026-11-05T20:42:53.860Z INFO src.services.processing [rid=-] attachment processed attachment_id=5a65d0cc-d52a-4097-9109-17839e940a66 job_id=a0093674-7f1d-4733-83a9-00e6d2e56b5f attempt=1 variants=3 render_peak_kib=182032 took=430ms waited=442ms
|
||||
2026-11-05T20:42:54.865Z INFO src.services.processing [rid=-] attachment processed attachment_id=5426d41b-70c0-44d2-9115-d658ffffd438 job_id=807c2313-2484-4632-92ca-3bc88ba0f0ad attempt=1 variants=3 render_peak_kib=182032 took=419ms waited=198ms
|
||||
2026-11-05T20:42:56.411Z INFO src.services.processing [rid=-] attachment processed attachment_id=d67ca25e-0c4c-44df-840c-ae61551a9a37 job_id=0f91baaf-9b84-44f4-9652-5ac1f36f7ed9 attempt=1 variants=3 render_peak_kib=182032 took=441ms waited=482ms
|
||||
2026-11-05T20:42:57.440Z INFO src.services.processing [rid=-] attachment processed attachment_id=409e63e7-f33c-47aa-a20e-49ed0a08dd4f job_id=9cf00252-483a-46ea-be4e-99bf9e330cfc attempt=1 variants=3 render_peak_kib=182032 took=446ms waited=251ms
|
||||
2026-11-05T20:42:58.948Z INFO src.services.processing [rid=-] attachment processed attachment_id=fd694b8b-3b45-4425-bfd2-2e7ae9a95342 job_id=51cdd01b-f25e-4a3e-93b7-9d406b8dad6e attempt=1 variants=3 render_peak_kib=182032 took=440ms waited=508ms
|
||||
2026-11-05T20:42:59.966Z INFO src.services.processing [rid=-] attachment processed attachment_id=e6f3e1f6-16b1-44ab-be34-cacfea25e4b5 job_id=52d2836c-e546-40b1-8db7-8737e8095bc0 attempt=1 variants=3 render_peak_kib=182032 took=442ms waited=291ms
|
||||
2026-11-05T20:43:00.947Z INFO src.services.processing [rid=-] attachment processed attachment_id=12ee82ae-64ea-44a2-82c0-befe59e3e032 job_id=4de39e69-0966-49d4-b439-78aa9d538d16 attempt=1 variants=3 render_peak_kib=182032 took=411ms waited=67ms
|
||||
2026-11-05T20:43:02.495Z INFO src.services.processing [rid=-] attachment processed attachment_id=756b2029-5934-4ea9-af56-cfffd647d82f job_id=74f0aa29-de53-4a92-9b2c-f5676f36ca47 attempt=1 variants=3 render_peak_kib=182032 took=431ms waited=325ms
|
||||
2026-11-05T20:43:05.712Z WARNING src.s3 [rid=-] object storage unavailable operation=put_object endpoint=http://s3:8333 bucket=bench key=bench/attachments/f9846259-e980-4d07-b3be-aa5b6505081b/p1/optimized.webp took=596ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:05.712Z WARNING src.s3 [rid=-] object storage unavailable operation=put_object endpoint=http://s3:8333 bucket=bench key=bench/attachments/df1f5b63-d4da-456d-b6bb-3e3945343691/p1/crop.jpg took=485ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:05.713Z WARNING src.s3 [rid=-] object storage unavailable operation=put_object endpoint=http://s3:8333 bucket=bench key=bench/attachments/e0eed9b3-05e4-47f6-a7a5-974124ce1cf8/p1/crop.jpg took=588ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:05.713Z WARNING src.s3 [rid=-] object storage unavailable operation=put_object endpoint=http://s3:8333 bucket=bench key=bench/attachments/fe6095c5-2a30-483f-843c-42649ae60159/p1/optimized.webp took=598ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:05.725Z INFO src.services.processing [rid=-] job retry scheduled job_id=851414b0-992c-4a23-98a7-4acbaabe61da attachment_id=f9846259-e980-4d07-b3be-aa5b6505081b attempt=1 reason=s3_write delay=1.3s fenced=False
|
||||
2026-11-05T20:43:05.725Z INFO src.services.processing [rid=-] job retry scheduled job_id=9813f6f7-5cfd-4a44-8860-850f8a641897 attachment_id=df1f5b63-d4da-456d-b6bb-3e3945343691 attempt=1 reason=s3_write delay=1.6s fenced=False
|
||||
2026-11-05T20:43:05.728Z INFO src.services.processing [rid=-] job retry scheduled job_id=c22789af-ffc5-4869-b6be-08786fd7ab2b attachment_id=fe6095c5-2a30-483f-843c-42649ae60159 attempt=1 reason=s3_write delay=1.4s fenced=False
|
||||
2026-11-05T20:43:05.729Z INFO src.services.processing [rid=-] job retry scheduled job_id=b4ce1903-600c-443e-8259-f5ed7db015c7 attachment_id=e0eed9b3-05e4-47f6-a7a5-974124ce1cf8 attempt=1 reason=s3_write delay=1.8s fenced=False
|
||||
2026-11-05T20:43:06.734Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a2333ad4-a052-4d41-8b85-b338097a956a took=1005ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:06.734Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/9d400afa-8545-41ed-9fb7-81bb7f2929df took=1004ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:06.735Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/f9d84ef9-b923-4c03-aae5-3076362dd493 took=1003ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:06.735Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ca657cd7-40c3-41fd-a39c-22f395768efa took=1003ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:06.750Z INFO src.services.processing [rid=-] job retry scheduled job_id=d7c50c24-45e1-43a9-a4e7-e99e5230f08e attachment_id=a2333ad4-a052-4d41-8b85-b338097a956a attempt=1 reason=s3_read delay=1.3s fenced=False
|
||||
2026-11-05T20:43:06.751Z INFO src.services.processing [rid=-] job retry scheduled job_id=8da89f75-a677-428e-8b9e-b89c144fcc19 attachment_id=9d400afa-8545-41ed-9fb7-81bb7f2929df attempt=1 reason=s3_read delay=1.2s fenced=False
|
||||
2026-11-05T20:43:06.753Z INFO src.services.processing [rid=-] job retry scheduled job_id=2f9de399-4298-4d91-85eb-9366e0184e83 attachment_id=ca657cd7-40c3-41fd-a39c-22f395768efa attempt=1 reason=s3_read delay=1.7s fenced=False
|
||||
2026-11-05T20:43:06.755Z INFO src.services.processing [rid=-] job retry scheduled job_id=3961deda-5fbc-436d-a294-76a80a68ac29 attachment_id=f9d84ef9-b923-4c03-aae5-3076362dd493 attempt=1 reason=s3_read delay=1.7s fenced=False
|
||||
2026-11-05T20:43:07.764Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a0d084cc-7693-46b6-b6b5-bdc500c3c390 took=1010ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:07.764Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/67a8d18f-e423-46a0-a579-8bdef1db2f02 took=1009ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:07.764Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 took=1008ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:07.765Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ef4af91f-c9b5-42e1-92af-ddd2658fc11c took=1006ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:07.781Z INFO src.services.processing [rid=-] job retry scheduled job_id=94d9896a-4244-4490-a4e7-c3fd1ff48b95 attachment_id=a0d084cc-7693-46b6-b6b5-bdc500c3c390 attempt=1 reason=s3_read delay=2.0s fenced=False
|
||||
2026-11-05T20:43:07.781Z INFO src.services.processing [rid=-] job retry scheduled job_id=960c2afd-cf5f-4545-884a-3c6ef5c6ee25 attachment_id=67a8d18f-e423-46a0-a579-8bdef1db2f02 attempt=1 reason=s3_read delay=1.7s fenced=False
|
||||
2026-11-05T20:43:07.783Z INFO src.services.processing [rid=-] job retry scheduled job_id=1fbb50d3-cece-4d62-9501-3494a791fc46 attachment_id=ef4af91f-c9b5-42e1-92af-ddd2658fc11c attempt=1 reason=s3_read delay=2.0s fenced=False
|
||||
2026-11-05T20:43:07.785Z INFO src.services.processing [rid=-] job retry scheduled job_id=e89fd2b7-4856-474a-9256-f3bb81c72dab attachment_id=2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 attempt=1 reason=s3_read delay=1.5s fenced=False
|
||||
2026-11-05T20:43:08.797Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/aabc6e22-b955-4841-88b3-3c54f367c723 took=1011ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:08.797Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a5356872-d138-4c70-9c6d-453a7e59c949 took=1011ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:08.797Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/fe6095c5-2a30-483f-843c-42649ae60159 took=1009ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:08.798Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/f9846259-e980-4d07-b3be-aa5b6505081b took=1008ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:08.812Z INFO src.services.processing [rid=-] job retry scheduled job_id=ab17455e-20da-44fb-bea8-e56895e257db attachment_id=a5356872-d138-4c70-9c6d-453a7e59c949 attempt=1 reason=s3_read delay=1.5s fenced=False
|
||||
2026-11-05T20:43:08.813Z INFO src.services.processing [rid=-] job retry scheduled job_id=bbd8a220-f6cc-4296-9ced-cf2f6b991d43 attachment_id=aabc6e22-b955-4841-88b3-3c54f367c723 attempt=1 reason=s3_read delay=1.6s fenced=False
|
||||
2026-11-05T20:43:08.814Z INFO src.services.processing [rid=-] job retry scheduled job_id=851414b0-992c-4a23-98a7-4acbaabe61da attachment_id=f9846259-e980-4d07-b3be-aa5b6505081b attempt=2 reason=s3_read delay=3.7s fenced=False
|
||||
2026-11-05T20:43:08.817Z INFO src.services.processing [rid=-] job retry scheduled job_id=c22789af-ffc5-4869-b6be-08786fd7ab2b attachment_id=fe6095c5-2a30-483f-843c-42649ae60159 attempt=2 reason=s3_read delay=2.7s fenced=False
|
||||
2026-11-05T20:43:09.818Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/df1f5b63-d4da-456d-b6bb-3e3945343691 took=1002ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:09.818Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/e0eed9b3-05e4-47f6-a7a5-974124ce1cf8 took=1001ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:09.819Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/9d400afa-8545-41ed-9fb7-81bb7f2929df took=1000ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:09.820Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a2333ad4-a052-4d41-8b85-b338097a956a took=1000ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:09.821Z INFO src.services.processing [rid=-] job retry scheduled job_id=9813f6f7-5cfd-4a44-8860-850f8a641897 attachment_id=df1f5b63-d4da-456d-b6bb-3e3945343691 attempt=2 reason=s3_read delay=3.4s fenced=False
|
||||
2026-11-05T20:43:09.821Z INFO src.services.processing [rid=-] job retry scheduled job_id=b4ce1903-600c-443e-8259-f5ed7db015c7 attachment_id=e0eed9b3-05e4-47f6-a7a5-974124ce1cf8 attempt=2 reason=s3_read delay=2.0s fenced=False
|
||||
2026-11-05T20:43:09.832Z INFO src.services.processing [rid=-] job retry scheduled job_id=d7c50c24-45e1-43a9-a4e7-e99e5230f08e attachment_id=a2333ad4-a052-4d41-8b85-b338097a956a attempt=2 reason=s3_read delay=3.0s fenced=False
|
||||
2026-11-05T20:43:09.835Z INFO src.services.processing [rid=-] job retry scheduled job_id=8da89f75-a677-428e-8b9e-b89c144fcc19 attachment_id=9d400afa-8545-41ed-9fb7-81bb7f2929df attempt=2 reason=s3_read delay=3.0s fenced=False
|
||||
2026-11-05T20:43:10.835Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ca657cd7-40c3-41fd-a39c-22f395768efa took=1002ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:10.835Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/f9d84ef9-b923-4c03-aae5-3076362dd493 took=1002ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:10.835Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 took=1000ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:10.836Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/67a8d18f-e423-46a0-a579-8bdef1db2f02 took=1000ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:10.847Z INFO src.services.processing [rid=-] job retry scheduled job_id=3961deda-5fbc-436d-a294-76a80a68ac29 attachment_id=f9d84ef9-b923-4c03-aae5-3076362dd493 attempt=2 reason=s3_read delay=3.4s fenced=False
|
||||
2026-11-05T20:43:10.847Z INFO src.services.processing [rid=-] job retry scheduled job_id=2f9de399-4298-4d91-85eb-9366e0184e83 attachment_id=ca657cd7-40c3-41fd-a39c-22f395768efa attempt=2 reason=s3_read delay=2.0s fenced=False
|
||||
2026-11-05T20:43:10.849Z INFO src.services.processing [rid=-] job retry scheduled job_id=960c2afd-cf5f-4545-884a-3c6ef5c6ee25 attachment_id=67a8d18f-e423-46a0-a579-8bdef1db2f02 attempt=2 reason=s3_read delay=3.4s fenced=False
|
||||
2026-11-05T20:43:10.850Z INFO src.services.processing [rid=-] job retry scheduled job_id=e89fd2b7-4856-474a-9256-f3bb81c72dab attachment_id=2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 attempt=2 reason=s3_read delay=2.3s fenced=False
|
||||
2026-11-05T20:43:11.853Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ef4af91f-c9b5-42e1-92af-ddd2658fc11c took=1002ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:11.853Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a0d084cc-7693-46b6-b6b5-bdc500c3c390 took=1001ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:11.853Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/aabc6e22-b955-4841-88b3-3c54f367c723 took=1000ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:11.855Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a5356872-d138-4c70-9c6d-453a7e59c949 took=1000ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:11.868Z INFO src.services.processing [rid=-] job retry scheduled job_id=94d9896a-4244-4490-a4e7-c3fd1ff48b95 attachment_id=a0d084cc-7693-46b6-b6b5-bdc500c3c390 attempt=2 reason=s3_read delay=2.1s fenced=False
|
||||
2026-11-05T20:43:11.869Z INFO src.services.processing [rid=-] job retry scheduled job_id=1fbb50d3-cece-4d62-9501-3494a791fc46 attachment_id=ef4af91f-c9b5-42e1-92af-ddd2658fc11c attempt=2 reason=s3_read delay=3.8s fenced=False
|
||||
2026-11-05T20:43:11.871Z INFO src.services.processing [rid=-] job retry scheduled job_id=ab17455e-20da-44fb-bea8-e56895e257db attachment_id=a5356872-d138-4c70-9c6d-453a7e59c949 attempt=2 reason=s3_read delay=2.9s fenced=False
|
||||
2026-11-05T20:43:11.872Z INFO src.services.processing [rid=-] job retry scheduled job_id=bbd8a220-f6cc-4296-9ced-cf2f6b991d43 attachment_id=aabc6e22-b955-4841-88b3-3c54f367c723 attempt=2 reason=s3_read delay=2.2s fenced=False
|
||||
2026-11-05T20:43:12.875Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/fe6095c5-2a30-483f-843c-42649ae60159 took=1002ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:12.878Z INFO src.services.processing [rid=-] job retry scheduled job_id=c22789af-ffc5-4869-b6be-08786fd7ab2b attachment_id=fe6095c5-2a30-483f-843c-42649ae60159 attempt=3 reason=s3_read delay=6.4s fenced=False
|
||||
2026-11-05T20:43:13.179Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/e0eed9b3-05e4-47f6-a7a5-974124ce1cf8 took=1003ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:13.181Z INFO src.services.processing [rid=-] job retry scheduled job_id=b4ce1903-600c-443e-8259-f5ed7db015c7 attachment_id=e0eed9b3-05e4-47f6-a7a5-974124ce1cf8 attempt=3 reason=s3_read delay=6.0s fenced=False
|
||||
2026-11-05T20:43:13.691Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/f9846259-e980-4d07-b3be-aa5b6505081b took=1005ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:13.692Z INFO src.services.processing [rid=-] job retry scheduled job_id=851414b0-992c-4a23-98a7-4acbaabe61da attachment_id=f9846259-e980-4d07-b3be-aa5b6505081b attempt=3 reason=s3_read delay=6.8s fenced=False
|
||||
2026-11-05T20:43:14.195Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a2333ad4-a052-4d41-8b85-b338097a956a took=1004ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:14.195Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/9d400afa-8545-41ed-9fb7-81bb7f2929df took=1003ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:14.197Z INFO src.services.processing [rid=-] job retry scheduled job_id=d7c50c24-45e1-43a9-a4e7-e99e5230f08e attachment_id=a2333ad4-a052-4d41-8b85-b338097a956a attempt=3 reason=s3_read delay=6.4s fenced=False
|
||||
2026-11-05T20:43:14.197Z INFO src.services.processing [rid=-] job retry scheduled job_id=8da89f75-a677-428e-8b9e-b89c144fcc19 attachment_id=9d400afa-8545-41ed-9fb7-81bb7f2929df attempt=3 reason=s3_read delay=4.6s fenced=False
|
||||
2026-11-05T20:43:14.210Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ca657cd7-40c3-41fd-a39c-22f395768efa took=1004ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:14.211Z INFO src.services.processing [rid=-] job retry scheduled job_id=2f9de399-4298-4d91-85eb-9366e0184e83 attachment_id=ca657cd7-40c3-41fd-a39c-22f395768efa attempt=3 reason=s3_read delay=6.5s fenced=False
|
||||
2026-11-05T20:43:14.699Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 took=1004ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:14.703Z INFO src.services.processing [rid=-] job retry scheduled job_id=e89fd2b7-4856-474a-9256-f3bb81c72dab attachment_id=2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 attempt=3 reason=s3_read delay=4.3s fenced=False
|
||||
2026-11-05T20:43:15.203Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/df1f5b63-d4da-456d-b6bb-3e3945343691 took=1003ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:15.203Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a0d084cc-7693-46b6-b6b5-bdc500c3c390 took=1002ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:15.205Z INFO src.services.processing [rid=-] job retry scheduled job_id=9813f6f7-5cfd-4a44-8860-850f8a641897 attachment_id=df1f5b63-d4da-456d-b6bb-3e3945343691 attempt=3 reason=s3_read delay=7.3s fenced=False
|
||||
2026-11-05T20:43:15.205Z INFO src.services.processing [rid=-] job retry scheduled job_id=94d9896a-4244-4490-a4e7-c3fd1ff48b95 attachment_id=a0d084cc-7693-46b6-b6b5-bdc500c3c390 attempt=3 reason=s3_read delay=7.5s fenced=False
|
||||
2026-11-05T20:43:15.215Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/aabc6e22-b955-4841-88b3-3c54f367c723 took=1001ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:15.217Z INFO src.services.processing [rid=-] job retry scheduled job_id=bbd8a220-f6cc-4296-9ced-cf2f6b991d43 attachment_id=aabc6e22-b955-4841-88b3-3c54f367c723 attempt=3 reason=s3_read delay=7.1s fenced=False
|
||||
2026-11-05T20:43:15.715Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/f9d84ef9-b923-4c03-aae5-3076362dd493 took=1008ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:15.719Z INFO src.services.processing [rid=-] job retry scheduled job_id=3961deda-5fbc-436d-a294-76a80a68ac29 attachment_id=f9d84ef9-b923-4c03-aae5-3076362dd493 attempt=3 reason=s3_read delay=4.6s fenced=False
|
||||
2026-11-05T20:43:15.726Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ef4af91f-c9b5-42e1-92af-ddd2658fc11c took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:15.728Z INFO src.services.processing [rid=-] job retry scheduled job_id=1fbb50d3-cece-4d62-9501-3494a791fc46 attachment_id=ef4af91f-c9b5-42e1-92af-ddd2658fc11c attempt=3 reason=s3_read delay=6.6s fenced=False
|
||||
2026-11-05T20:43:16.211Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/67a8d18f-e423-46a0-a579-8bdef1db2f02 took=1004ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:16.213Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a5356872-d138-4c70-9c6d-453a7e59c949 took=1000ms error_type=ConnectTimeoutError
|
||||
2026-11-05T20:43:16.214Z INFO src.services.processing [rid=-] job retry scheduled job_id=960c2afd-cf5f-4545-884a-3c6ef5c6ee25 attachment_id=67a8d18f-e423-46a0-a579-8bdef1db2f02 attempt=3 reason=s3_read delay=7.6s fenced=False
|
||||
2026-11-05T20:43:16.215Z INFO src.services.processing [rid=-] job retry scheduled job_id=ab17455e-20da-44fb-bea8-e56895e257db attachment_id=a5356872-d138-4c70-9c6d-453a7e59c949 attempt=3 reason=s3_read delay=7.5s fenced=False
|
||||
2026-11-05T20:43:19.268Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/9d400afa-8545-41ed-9fb7-81bb7f2929df took=11ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:19.271Z INFO src.services.processing [rid=-] job retry scheduled job_id=8da89f75-a677-428e-8b9e-b89c144fcc19 attachment_id=9d400afa-8545-41ed-9fb7-81bb7f2929df attempt=4 reason=s3_read delay=10.7s fenced=False
|
||||
2026-11-05T20:43:19.272Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/fe6095c5-2a30-483f-843c-42649ae60159 took=1ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:19.272Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:19.274Z INFO src.services.processing [rid=-] job retry scheduled job_id=e89fd2b7-4856-474a-9256-f3bb81c72dab attachment_id=2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 attempt=4 reason=s3_read delay=9.3s fenced=False
|
||||
2026-11-05T20:43:19.275Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/e0eed9b3-05e4-47f6-a7a5-974124ce1cf8 took=2ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:19.275Z INFO src.services.processing [rid=-] job retry scheduled job_id=c22789af-ffc5-4869-b6be-08786fd7ab2b attachment_id=fe6095c5-2a30-483f-843c-42649ae60159 attempt=4 reason=s3_read delay=10.0s fenced=False
|
||||
2026-11-05T20:43:19.276Z INFO src.services.processing [rid=-] job retry scheduled job_id=b4ce1903-600c-443e-8259-f5ed7db015c7 attachment_id=e0eed9b3-05e4-47f6-a7a5-974124ce1cf8 attempt=4 reason=s3_read delay=12.4s fenced=False
|
||||
2026-11-05T20:43:20.794Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/f9846259-e980-4d07-b3be-aa5b6505081b took=12ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:20.794Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/f9d84ef9-b923-4c03-aae5-3076362dd493 took=13ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:20.797Z INFO src.services.processing [rid=-] job retry scheduled job_id=851414b0-992c-4a23-98a7-4acbaabe61da attachment_id=f9846259-e980-4d07-b3be-aa5b6505081b attempt=4 reason=s3_read delay=12.0s fenced=False
|
||||
2026-11-05T20:43:20.797Z INFO src.services.processing [rid=-] job retry scheduled job_id=3961deda-5fbc-436d-a294-76a80a68ac29 attachment_id=f9d84ef9-b923-4c03-aae5-3076362dd493 attempt=4 reason=s3_read delay=12.7s fenced=False
|
||||
2026-11-05T20:43:20.798Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ca657cd7-40c3-41fd-a39c-22f395768efa took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:20.801Z INFO src.services.processing [rid=-] job retry scheduled job_id=2f9de399-4298-4d91-85eb-9366e0184e83 attachment_id=ca657cd7-40c3-41fd-a39c-22f395768efa attempt=4 reason=s3_read delay=13.3s fenced=False
|
||||
2026-11-05T20:43:20.802Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a2333ad4-a052-4d41-8b85-b338097a956a took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:20.803Z INFO src.services.processing [rid=-] job retry scheduled job_id=d7c50c24-45e1-43a9-a4e7-e99e5230f08e attachment_id=a2333ad4-a052-4d41-8b85-b338097a956a attempt=4 reason=s3_read delay=11.4s fenced=False
|
||||
2026-11-05T20:43:22.306Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/aabc6e22-b955-4841-88b3-3c54f367c723 took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:22.308Z INFO src.services.processing [rid=-] job retry scheduled job_id=bbd8a220-f6cc-4296-9ced-cf2f6b991d43 attachment_id=aabc6e22-b955-4841-88b3-3c54f367c723 attempt=4 reason=s3_read delay=11.9s fenced=False
|
||||
2026-11-05T20:43:22.823Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/df1f5b63-d4da-456d-b6bb-3e3945343691 took=11ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:22.823Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ef4af91f-c9b5-42e1-92af-ddd2658fc11c took=13ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:22.825Z INFO src.services.processing [rid=-] job retry scheduled job_id=1fbb50d3-cece-4d62-9501-3494a791fc46 attachment_id=ef4af91f-c9b5-42e1-92af-ddd2658fc11c attempt=4 reason=s3_read delay=15.2s fenced=False
|
||||
2026-11-05T20:43:22.825Z INFO src.services.processing [rid=-] job retry scheduled job_id=9813f6f7-5cfd-4a44-8860-850f8a641897 attachment_id=df1f5b63-d4da-456d-b6bb-3e3945343691 attempt=4 reason=s3_read delay=8.5s fenced=False
|
||||
2026-11-05T20:43:22.829Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a0d084cc-7693-46b6-b6b5-bdc500c3c390 took=2ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:22.830Z INFO src.services.processing [rid=-] job retry scheduled job_id=94d9896a-4244-4490-a4e7-c3fd1ff48b95 attachment_id=a0d084cc-7693-46b6-b6b5-bdc500c3c390 attempt=4 reason=s3_read delay=12.9s fenced=False
|
||||
2026-11-05T20:43:23.821Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/67a8d18f-e423-46a0-a579-8bdef1db2f02 took=2ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:23.822Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a5356872-d138-4c70-9c6d-453a7e59c949 took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:23.823Z INFO src.services.processing [rid=-] job retry scheduled job_id=ab17455e-20da-44fb-bea8-e56895e257db attachment_id=a5356872-d138-4c70-9c6d-453a7e59c949 attempt=4 reason=s3_read delay=14.4s fenced=False
|
||||
2026-11-05T20:43:23.823Z INFO src.services.processing [rid=-] job retry scheduled job_id=960c2afd-cf5f-4545-884a-3c6ef5c6ee25 attachment_id=67a8d18f-e423-46a0-a579-8bdef1db2f02 attempt=4 reason=s3_read delay=13.7s fenced=False
|
||||
2026-11-05T20:43:28.863Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:28.865Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=2fe8e7c5-238a-4fa2-b7c8-580eea9a6777 job_id=e89fd2b7-4856-474a-9256-f3bb81c72dab attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:29.373Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/fe6095c5-2a30-483f-843c-42649ae60159 took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:29.375Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=fe6095c5-2a30-483f-843c-42649ae60159 job_id=c22789af-ffc5-4869-b6be-08786fd7ab2b attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:30.386Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/9d400afa-8545-41ed-9fb7-81bb7f2929df took=4ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:30.388Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=9d400afa-8545-41ed-9fb7-81bb7f2929df job_id=8da89f75-a677-428e-8b9e-b89c144fcc19 attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:31.396Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/df1f5b63-d4da-456d-b6bb-3e3945343691 took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:31.397Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=df1f5b63-d4da-456d-b6bb-3e3945343691 job_id=9813f6f7-5cfd-4a44-8860-850f8a641897 attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:31.900Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/e0eed9b3-05e4-47f6-a7a5-974124ce1cf8 took=2ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:31.902Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=e0eed9b3-05e4-47f6-a7a5-974124ce1cf8 job_id=b4ce1903-600c-443e-8259-f5ed7db015c7 attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:32.407Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a2333ad4-a052-4d41-8b85-b338097a956a took=2ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:32.408Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=a2333ad4-a052-4d41-8b85-b338097a956a job_id=d7c50c24-45e1-43a9-a4e7-e99e5230f08e attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:32.914Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/f9846259-e980-4d07-b3be-aa5b6505081b took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:32.915Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=f9846259-e980-4d07-b3be-aa5b6505081b job_id=851414b0-992c-4a23-98a7-4acbaabe61da attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:33.924Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/f9d84ef9-b923-4c03-aae5-3076362dd493 took=2ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:33.926Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=f9d84ef9-b923-4c03-aae5-3076362dd493 job_id=3961deda-5fbc-436d-a294-76a80a68ac29 attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:34.434Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/aabc6e22-b955-4841-88b3-3c54f367c723 took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:34.434Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ca657cd7-40c3-41fd-a39c-22f395768efa took=4ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:34.436Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=ca657cd7-40c3-41fd-a39c-22f395768efa job_id=2f9de399-4298-4d91-85eb-9366e0184e83 attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:34.436Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=aabc6e22-b955-4841-88b3-3c54f367c723 job_id=bbd8a220-f6cc-4296-9ced-cf2f6b991d43 attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:35.960Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a0d084cc-7693-46b6-b6b5-bdc500c3c390 took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:35.962Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=a0d084cc-7693-46b6-b6b5-bdc500c3c390 job_id=94d9896a-4244-4490-a4e7-c3fd1ff48b95 attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:37.980Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/67a8d18f-e423-46a0-a579-8bdef1db2f02 took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:37.981Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=67a8d18f-e423-46a0-a579-8bdef1db2f02 job_id=960c2afd-cf5f-4545-884a-3c6ef5c6ee25 attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:38.490Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/a5356872-d138-4c70-9c6d-453a7e59c949 took=2ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:38.490Z WARNING src.s3 [rid=-] object storage unavailable operation=get_object endpoint=http://s3:8333 bucket=bench key=bench/staging/ef4af91f-c9b5-42e1-92af-ddd2658fc11c took=3ms error_type=EndpointConnectionError
|
||||
2026-11-05T20:43:38.491Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=a5356872-d138-4c70-9c6d-453a7e59c949 job_id=ab17455e-20da-44fb-bea8-e56895e257db attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
2026-11-05T20:43:38.492Z WARNING src.services.processing [rid=-] attachment processing failed attachment_id=ef4af91f-c9b5-42e1-92af-ddd2658fc11c job_id=1fbb50d3-cece-4d62-9501-3494a791fc46 attempt=5 error_code=retry_exhausted detail=s3_read fenced=False
|
||||
@@ -0,0 +1,50 @@
|
||||
q,1793910964317.1,1793910964319.4,1793910964347.1,202,,624f0dd7-0c2d-4c83-8843-bbe06d9d791a,39e797bfa6f6,
|
||||
q,1793910964357.1,1793910964358.9,1793910964373.2,202,,6cdd56df-e112-4bba-a89d-146b0c762bf0,7bbaa8668c87,
|
||||
q,1793910964397.1,1793910964398.4,1793910964410.4,202,,bb2686b4-0a75-4a6e-89fc-1ed6c71c52e4,39e797bfa6f6,
|
||||
q,1793910964437.1,1793910964438.0,1793910964449.5,202,,b6893618-ee77-4af5-b6a6-19cbe471be41,7bbaa8668c87,
|
||||
q,1793910964477.1,1793910964478.4,1793910964490.2,202,,b6f9c97c-2e86-4b10-b70b-6622810b65d0,39e797bfa6f6,
|
||||
q,1793910964517.1,1793910964518.6,1793910964530.6,202,,a0741b9c-e98a-430a-aedb-3ceb93d61478,7bbaa8668c87,
|
||||
q,1793910964557.1,1793910964558.6,1793910964571.1,202,,66078edf-7a38-4b83-b398-c9a75d3a65f0,39e797bfa6f6,
|
||||
q,1793910964597.1,1793910964600.7,1793910964613.7,202,,6776d34f-02e2-45dc-9d6d-fe3cda6be637,7bbaa8668c87,
|
||||
q,1793910964637.1,1793910964638.9,1793910964651.3,202,,22a98928-c5e1-475c-86ad-211c049b321d,39e797bfa6f6,
|
||||
q,1793910964677.1,1793910964680.1,1793910964692.1,202,,ecdca22c-9106-45c7-b92e-c0a4b67424eb,7bbaa8668c87,
|
||||
q,1793910964717.1,1793910964719.3,1793910964735.4,202,,80bd1cfa-d71f-4b8a-9548-bca24a721711,39e797bfa6f6,
|
||||
q,1793910964757.1,1793910964758.2,1793910964768.2,202,,0d146a44-ff90-4d4a-a258-71c289c896ba,7bbaa8668c87,
|
||||
q,1793910964797.1,1793910964799.2,1793910964808.0,202,,84010548-12c7-4f12-a02d-e4ffff48f95f,39e797bfa6f6,
|
||||
q,1793910964837.1,1793910964838.9,1793910964849.2,202,,513e2a1e-30e3-48c6-a33e-c4525ec0fcec,7bbaa8668c87,
|
||||
q,1793910964877.1,1793910964878.1,1793910964887.2,202,,09631d36-bfc5-4972-a9b1-38ffa5e920ac,39e797bfa6f6,
|
||||
q,1793910964917.1,1793910964918.1,1793910964927.1,202,,7be4a741-8389-4699-a691-90d332efe36c,7bbaa8668c87,
|
||||
q,1793910964957.1,1793910964959.0,1793910964969.1,202,,755a16ba-4650-42ae-b2a0-5c10c28f9296,39e797bfa6f6,
|
||||
q,1793910964997.1,1793910964998.1,1793910965008.2,202,,47720528-a34f-4586-a62d-442472568b3f,7bbaa8668c87,
|
||||
q,1793910965037.1,1793910965038.9,1793910965048.6,202,,3e9a9e83-745d-45e2-82d4-e58bb169db0c,39e797bfa6f6,
|
||||
q,1793910965077.1,1793910965078.2,1793910965088.1,202,,b5523944-a456-4799-94c2-334194e1a1dd,7bbaa8668c87,
|
||||
q,1793910965117.1,1793910965118.3,1793910965128.5,202,,af991f3c-7ecf-4a58-8bd1-30bf6506014e,39e797bfa6f6,
|
||||
q,1793910965157.1,1793910965159.3,1793910965170.0,202,,0853a82b-e51f-4e74-937d-126054f3376e,7bbaa8668c87,
|
||||
q,1793910965197.1,1793910965199.7,1793910965210.3,202,,c85db1d7-7c6a-41fe-886a-a0becb914c0a,39e797bfa6f6,
|
||||
q,1793910965237.1,1793910965239.9,1793910965252.1,202,,3e3e037e-b1ce-45f8-b190-fcddf689ac95,7bbaa8668c87,
|
||||
q,1793910965277.1,1793910965277.9,1793910965289.8,202,,e3c28c07-6e54-42fb-96ca-db263b19cc7e,39e797bfa6f6,
|
||||
q,1793910965317.1,1793910965318.9,1793910965331.5,202,,cb916baf-a790-49fd-ad19-8106ccfad440,7bbaa8668c87,
|
||||
q,1793910965357.1,1793910965359.6,1793910965371.7,202,,3fd48af8-409f-4bfc-b3b5-c950c3f4f3ed,39e797bfa6f6,
|
||||
q,1793910965397.1,1793910965398.9,1793910965410.3,202,,108c1b29-ebba-4ca3-bc1c-3af1eafaa111,7bbaa8668c87,
|
||||
q,1793910965437.1,1793910965438.1,1793910965447.0,202,,2a8863d8-b07f-4be1-a045-cebc19f7d957,39e797bfa6f6,
|
||||
q,1793910965477.1,1793910965479.0,1793910965488.4,202,,81ee07e6-e7f5-4bfa-bdb8-24ba70fd570e,7bbaa8668c87,
|
||||
q,1793910965517.1,1793910965518.9,1793910965528.4,202,,59ad672c-de93-4181-9cd2-49ca313c98ab,39e797bfa6f6,
|
||||
q,1793910965557.1,1793910965561.0,1793910965571.2,202,,d21337ea-350c-4e2c-89aa-89991fbd4119,7bbaa8668c87,
|
||||
q,1793910965597.1,1793910965599.8,1793910965610.5,202,,75446bc9-2f40-4a72-9380-931ce73f6c44,39e797bfa6f6,
|
||||
q,1793910965637.1,1793910965638.1,1793910965649.1,202,,54ae72dd-3536-4979-bdbf-1b5c99fa5e79,7bbaa8668c87,
|
||||
q,1793910965677.1,1793910965678.6,1793910965690.1,202,,7903803a-0e1a-427d-9723-4370be34e281,39e797bfa6f6,
|
||||
q,1793910965717.1,1793910965719.3,1793910965733.6,202,,c8982d53-b3b6-4840-bed9-b20bcafb572c,7bbaa8668c87,
|
||||
q,1793910965757.1,1793910965757.9,1793910965767.9,202,,2d2b0a37-6b59-4ff5-9aa3-decb60be7e88,39e797bfa6f6,
|
||||
q,1793910965797.1,1793910965799.1,1793910965807.1,202,,fefb8638-ff6d-4f9b-b4a9-c555e46ecb1c,7bbaa8668c87,
|
||||
q,1793910965837.1,1793910965837.9,1793910965846.1,202,,9d4df388-46ec-4b99-a187-c22683334ad2,39e797bfa6f6,
|
||||
q,1793910965877.1,1793910965877.9,1793910965885.6,202,,bdcc42e5-70f5-4fd4-b2bf-bf7b7bfe4c02,7bbaa8668c87,
|
||||
q,1793910965917.1,1793910965918.3,1793910965927.0,202,,72f5bd97-34f8-4437-a04f-0c27ab20f238,39e797bfa6f6,
|
||||
q,1793910965957.1,1793910965958.9,1793910965967.1,202,,038fb213-b100-4445-85e1-a83c721d277a,7bbaa8668c87,
|
||||
q,1793910965997.1,1793910965998.6,1793910966006.7,202,,a9240246-08f1-4a17-80df-46d47b4ee240,39e797bfa6f6,
|
||||
q,1793910966037.1,1793910966037.9,1793910966046.0,202,,8902d068-c3ef-4c59-ae57-c7bb8835d09c,7bbaa8668c87,
|
||||
q,1793910966077.1,1793910966079.6,1793910966087.0,202,,a6bc3ad9-8089-45bc-b099-e79ccb0df65e,39e797bfa6f6,
|
||||
q,1793910966117.1,1793910966118.9,1793910966127.0,202,,f6920723-f44d-4b78-a7dc-785f455d4cba,7bbaa8668c87,
|
||||
q,1793910966157.1,1793910966158.1,1793910966166.6,202,,e25db83c-62ed-48c5-8562-349d237425f1,39e797bfa6f6,
|
||||
q,1793910966197.1,1793910966198.4,1793910966206.3,202,,a250a98d-2907-451e-82ed-161ccf8bb620,7bbaa8668c87,
|
||||
q,1793910966237.1,1793910966237.7,1793910966246.2,202,,aff57b70-f114-41c2-bd4f-d54b9fbc5a19,39e797bfa6f6,
|
||||
q,1793910966277.1,1793910966278.7,1793910966288.5,202,,a6c8552d-95ed-42c5-ae5e-077e56869533,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 25.0,
|
||||
"duration_s": 3.0,
|
||||
"start": 1793910964.317,
|
||||
"end": 1793910966.317,
|
||||
"finished": 1793910966.289,
|
||||
"scheduled": 50,
|
||||
"sent": 50,
|
||||
"accepted": 50,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 50
|
||||
},
|
||||
"upload_p50_ms": 10.07,
|
||||
"upload_p95_ms": 14.34,
|
||||
"upload_max_ms": 27.62,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.046,
|
||||
"log": "/tmp/c4p4-s-r1-drain.log"
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
q,1793911013158.8,1793911013159.9,1793911013189.5,202,,a831007d-e7fd-4f6d-bf14-9ec2733ab876,39e797bfa6f6,
|
||||
q,1793911013558.8,1793911013561.9,1793911013585.1,202,,952746a0-e352-4de9-8d74-1f2cee982ede,7bbaa8668c87,
|
||||
q,1793911013958.8,1793911013959.8,1793911013968.3,202,,532e4fb7-de6c-4975-8144-2536454982f6,39e797bfa6f6,
|
||||
q,1793911014358.8,1793911014362.8,1793911014371.0,202,,bea08d69-4e36-4bf9-a79f-1933593ef703,7bbaa8668c87,
|
||||
q,1793911014758.8,1793911014759.8,1793911014769.0,202,,4b39a85d-1083-44fc-9715-2840ae976869,39e797bfa6f6,
|
||||
q,1793911015158.8,1793911015159.6,1793911015169.1,202,,6ce3c2df-f171-42bf-9b07-76919b96f0a6,7bbaa8668c87,
|
||||
q,1793911015558.8,1793911015560.5,1793911015573.5,202,,da9c220d-4c77-49d9-9cd0-d891408734e4,39e797bfa6f6,
|
||||
q,1793911015958.8,1793911015960.0,1793911015967.5,202,,49b3b146-5d0d-4503-949c-abe71104e713,7bbaa8668c87,
|
||||
q,1793911016358.8,1793911016362.0,1793911016370.2,202,,76986bd1-9130-4689-8bea-1f18101e170c,39e797bfa6f6,
|
||||
q,1793911016758.8,1793911016760.5,1793911016776.5,202,,dc2322a6-f642-4a16-84de-11dd824aef47,7bbaa8668c87,
|
||||
q,1793911017158.8,1793911017162.1,1793911017187.4,202,,9f5fb0a1-68d6-41a3-85c4-7002f3910b41,39e797bfa6f6,
|
||||
q,1793911017558.8,1793911017562.4,1793911017572.9,202,,1c0765bc-bfca-4719-9427-b000ffcf5477,7bbaa8668c87,
|
||||
q,1793911017958.8,1793911017960.5,1793911017968.4,202,,b6d65008-6c71-4bd6-a7d9-4e0141ff3cc6,39e797bfa6f6,
|
||||
q,1793911018358.8,1793911018361.2,1793911018374.1,202,,51da2ed2-ad98-496f-9c4e-e67a0bdf607c,7bbaa8668c87,
|
||||
q,1793911018758.8,1793911018760.2,1793911018775.0,202,,402f085e-95ea-4c31-bbbf-759a32ea8f7c,39e797bfa6f6,
|
||||
q,1793911019158.8,1793911019163.9,1793911019179.3,202,,d7710c74-67b1-4822-b9fc-e2c1f35b05a0,7bbaa8668c87,
|
||||
q,1793911019558.8,1793911019561.5,1793911019569.0,202,,d835a534-80db-4c53-89b0-1685fcc37ac3,39e797bfa6f6,
|
||||
q,1793911019958.8,1793911019959.8,1793911019967.8,202,,98d0c0a9-b94b-4758-8b70-500c6d6b3684,7bbaa8668c87,
|
||||
q,1793911020358.8,1793911020359.8,1793911020368.7,202,,589c49f3-eb1a-431e-b96d-b5c91ab28564,39e797bfa6f6,
|
||||
q,1793911020758.8,1793911020761.0,1793911020773.3,202,,8ec84633-e0ee-48c0-86ac-b0c738b5a8ea,7bbaa8668c87,
|
||||
q,1793911021158.8,1793911021162.6,1793911021174.0,202,,b2bba72d-5137-4efb-b163-c8898093f6e6,39e797bfa6f6,
|
||||
q,1793911021558.8,1793911021559.8,1793911021568.3,202,,cc252e93-3ac8-49a0-86ce-dbb5db0038fc,7bbaa8668c87,
|
||||
q,1793911021958.8,1793911021963.0,1793911021975.4,202,,8678a9b2-8222-4791-a871-97cc9fd9acbc,39e797bfa6f6,
|
||||
q,1793911022358.8,1793911022362.8,1793911022389.1,202,,deeaa0cd-8d97-48d7-92a8-e0a43cff1bba,7bbaa8668c87,
|
||||
q,1793911022758.8,1793911022759.6,1793911022770.7,202,,3596ce42-2c22-4f0b-85df-f863dd24532a,39e797bfa6f6,
|
||||
q,1793911023158.8,1793911023164.8,1793911023172.8,202,,c0952f44-4285-4b38-9c07-0ff80befe16e,7bbaa8668c87,
|
||||
q,1793911023558.8,1793911023559.8,1793911023567.6,202,,ec988501-e1c6-4296-8d1b-e30f0b9c7b0b,39e797bfa6f6,
|
||||
q,1793911023958.8,1793911023966.1,1793911023993.3,202,,d08892eb-d3d3-490a-8667-a89ad85d884b,7bbaa8668c87,
|
||||
q,1793911024358.8,1793911024361.9,1793911024386.1,202,,d9eaadde-925a-4c20-9961-7c1dbd3714ba,39e797bfa6f6,
|
||||
q,1793911024758.8,1793911024760.5,1793911024769.8,202,,08abea28-b00e-4b7a-bcf5-ea1d10c46c1f,7bbaa8668c87,
|
||||
q,1793911025158.8,1793911025159.8,1793911025167.9,202,,b63f86dd-2290-4bbf-99c9-da9640bc110a,39e797bfa6f6,
|
||||
q,1793911025558.8,1793911025560.0,1793911025569.0,202,,390661fe-04e9-4b06-93de-3abcfc054967,7bbaa8668c87,
|
||||
q,1793911025958.8,1793911025959.3,1793911025968.6,202,,ff537618-aa88-485a-a373-b5a15b3e16d3,39e797bfa6f6,
|
||||
q,1793911026358.8,1793911026361.6,1793911026379.1,202,,cbece15c-1179-48e3-bb6a-5be184c803e1,7bbaa8668c87,
|
||||
q,1793911026758.8,1793911026759.8,1793911026767.9,202,,33932658-f46c-4777-85e9-37ccac5d7701,39e797bfa6f6,
|
||||
q,1793911027158.8,1793911027159.5,1793911027167.9,202,,bb1082b4-b4d9-4492-af8b-ca6e89832964,7bbaa8668c87,
|
||||
q,1793911027558.8,1793911027561.2,1793911027587.4,202,,d6ef4f16-c901-4e8f-ab63-0ca7559d16bf,39e797bfa6f6,
|
||||
q,1793911027958.8,1793911027961.9,1793911027984.1,202,,14ab154f-31d5-4604-bc27-aff3eddd2bc9,7bbaa8668c87,
|
||||
q,1793911028358.8,1793911028364.9,1793911028376.3,202,,fa41832b-ec0d-42f6-aa74-f78ec3fc1ab0,39e797bfa6f6,
|
||||
q,1793911028758.8,1793911028760.9,1793911028769.8,202,,b698eabc-a031-4f88-8312-2a6aaea897ad,7bbaa8668c87,
|
||||
q,1793911029158.8,1793911029160.5,1793911029172.8,202,,28819e72-6f48-4778-8ae3-a368d49c1a8c,39e797bfa6f6,
|
||||
q,1793911029558.8,1793911029561.0,1793911029587.3,202,,12b952fb-aa1d-4578-9a69-9431b312559b,7bbaa8668c87,
|
||||
q,1793911029958.8,1793911029961.5,1793911029976.1,202,,aab2b742-c75d-40a2-9d55-6a205a15a910,39e797bfa6f6,
|
||||
q,1793911030358.8,1793911030359.8,1793911030367.8,202,,fa75982f-b109-4832-8f85-f5db7ee873ed,7bbaa8668c87,
|
||||
q,1793911030758.8,1793911030759.9,1793911030768.1,202,,cf3f4b81-b6ad-4774-8334-848043e9d785,39e797bfa6f6,
|
||||
q,1793911031158.8,1793911031159.8,1793911031168.9,202,,b5787893-2ffe-4eca-b565-8efbfe7d858b,7bbaa8668c87,
|
||||
q,1793911031558.8,1793911031560.3,1793911031586.2,202,,dfe371cc-ff4f-47ab-ad9d-f4ac0a134772,39e797bfa6f6,
|
||||
q,1793911031958.8,1793911031961.8,1793911031972.8,202,,6a4ea0d3-f885-4e36-a546-82cd260f3b80,7bbaa8668c87,
|
||||
q,1793911032358.8,1793911032361.2,1793911032369.3,202,,611ce066-c68a-4cfd-b42d-c9921a08104d,39e797bfa6f6,
|
||||
q,1793911032758.8,1793911032760.0,1793911032770.2,202,,408b02bb-fe6a-451d-ab11-f8137bbb52a8,7bbaa8668c87,
|
||||
q,1793911033158.8,1793911033162.9,1793911033187.1,202,,d3240294-e018-40df-bec3-b2dc76480714,39e797bfa6f6,
|
||||
q,1793911033558.8,1793911033559.9,1793911033573.6,202,,7c3a6d26-e369-430e-afe1-0f8cc66ea7cf,7bbaa8668c87,
|
||||
q,1793911033958.8,1793911033960.3,1793911033968.6,202,,78615553-4c5b-4e78-8804-bcc2908bf1bd,39e797bfa6f6,
|
||||
q,1793911034358.8,1793911034362.1,1793911034370.2,202,,e04528ca-1a3d-4ac7-92f5-60580318a08d,7bbaa8668c87,
|
||||
q,1793911034758.8,1793911034760.1,1793911034776.2,202,,d04a1297-7302-4c93-9085-2fa75dce3a0a,39e797bfa6f6,
|
||||
q,1793911035158.8,1793911035166.1,1793911035192.6,202,,545f9a1f-7b42-4a14-9f41-df7827ae0886,7bbaa8668c87,
|
||||
q,1793911035558.8,1793911035565.9,1793911035578.3,202,,a55bff29-e8d8-4e29-bc69-0c3d7ebae193,39e797bfa6f6,
|
||||
q,1793911035958.8,1793911035963.7,1793911035972.9,202,,c2e2fcda-3fce-4f30-a03f-0f417afda120,7bbaa8668c87,
|
||||
q,1793911036358.8,1793911036360.0,1793911036369.6,202,,ab2cbb02-e893-4d24-b508-b161e1a55211,39e797bfa6f6,
|
||||
q,1793911036758.8,1793911036760.2,1793911036770.7,202,,a2a748b9-39bd-4348-b609-db70896ebfad,7bbaa8668c87,
|
||||
q,1793911037158.8,1793911037166.1,1793911037191.4,202,,4194c2b1-fb71-4f79-a3ff-53d214f5f09c,39e797bfa6f6,
|
||||
q,1793911037558.8,1793911037565.8,1793911037575.8,202,,b1510c60-c83b-40d8-aa12-c6696d1d5f7e,7bbaa8668c87,
|
||||
q,1793911037958.8,1793911037961.0,1793911037969.9,202,,b5e1fbf5-0d43-49de-956f-99f3f8459d24,39e797bfa6f6,
|
||||
q,1793911038358.8,1793911038360.0,1793911038373.4,202,,fe18b3d0-160a-4e87-bd8a-cc78439194a8,7bbaa8668c87,
|
||||
q,1793911038758.8,1793911038759.9,1793911038773.2,202,,6edd11d3-1d9e-48b1-ba59-75f821413853,39e797bfa6f6,
|
||||
q,1793911039158.8,1793911039160.7,1793911039175.3,202,,88b83e4e-601c-47a0-b234-4414630224e0,7bbaa8668c87,
|
||||
q,1793911039558.8,1793911039563.8,1793911039571.3,202,,414da303-4403-4031-b24a-9b4fb7e0f362,39e797bfa6f6,
|
||||
q,1793911039958.8,1793911039961.1,1793911039969.8,202,,cb10a2c7-d6fb-43d7-b444-dcbc1df90de5,7bbaa8668c87,
|
||||
q,1793911040358.8,1793911040360.0,1793911040376.6,202,,569ae45a-d257-4c97-b440-d8a06164ed12,39e797bfa6f6,
|
||||
q,1793911040758.8,1793911040761.0,1793911040772.5,202,,5c215b9b-7976-4928-8301-cc9211833e14,7bbaa8668c87,
|
||||
q,1793911041158.8,1793911041162.6,1793911041174.2,202,,640d2c85-aece-4599-93f3-0cdcf094923a,39e797bfa6f6,
|
||||
q,1793911041558.8,1793911041564.8,1793911041572.8,202,,9e934aeb-2e91-4136-a4c0-bc901ff6e479,7bbaa8668c87,
|
||||
q,1793911041958.8,1793911041959.8,1793911041968.7,202,,43773a20-4c7a-4c5f-86a9-23c2b0964722,39e797bfa6f6,
|
||||
q,1793911042358.8,1793911042364.7,1793911042391.4,202,,a65ebf39-38d0-4b48-9b86-c3227b04f956,7bbaa8668c87,
|
||||
q,1793911042758.8,1793911042761.3,1793911042773.0,202,,135fdd67-7f2a-44c5-8d66-84a8bb5624fc,39e797bfa6f6,
|
||||
q,1793911043158.8,1793911043163.0,1793911043170.9,202,,78a77577-b70c-431f-b411-68ca7ef9535f,7bbaa8668c87,
|
||||
q,1793911043558.8,1793911043564.8,1793911043573.0,202,,11ddd766-d1f1-4e08-a8fe-4b361ba83841,39e797bfa6f6,
|
||||
q,1793911043958.8,1793911043965.1,1793911043992.1,202,,53a3b1f1-ee2c-4378-a89e-0b77cb935e93,7bbaa8668c87,
|
||||
q,1793911044358.8,1793911044366.0,1793911044390.4,202,,680908f0-da47-4c72-9f04-2c3901ebd347,39e797bfa6f6,
|
||||
q,1793911044758.8,1793911044761.3,1793911044771.2,202,,7e02f4b0-daa9-4ead-ab85-5bdfef2da47a,7bbaa8668c87,
|
||||
q,1793911045158.8,1793911045161.8,1793911045169.7,202,,7de68e29-97eb-42dd-972f-06f909fd947e,39e797bfa6f6,
|
||||
q,1793911045558.8,1793911045561.9,1793911045571.5,202,,be359b50-06b7-4a44-83ce-72548c96e587,7bbaa8668c87,
|
||||
q,1793911045958.8,1793911045966.1,1793911045990.9,202,,a3df0369-920a-41fc-946a-839a7b173d69,39e797bfa6f6,
|
||||
q,1793911046358.8,1793911046360.8,1793911046370.2,202,,b67ff36e-2748-42e1-8a34-718a25dca3f4,7bbaa8668c87,
|
||||
q,1793911046758.8,1793911046760.0,1793911046767.8,202,,9222ef96-8b11-436e-88d0-214f5d4306a0,39e797bfa6f6,
|
||||
q,1793911047158.8,1793911047159.8,1793911047169.2,202,,52a897dc-d528-4cfe-83b1-682a68ebb74f,7bbaa8668c87,
|
||||
q,1793911047558.8,1793911047564.9,1793911047578.9,202,,a2273c37-a279-412f-8148-a56116bc9e91,39e797bfa6f6,
|
||||
q,1793911047958.8,1793911047964.2,1793911047990.4,202,,cd9993a5-11a5-4dbf-bef6-e8caf0e35a22,7bbaa8668c87,
|
||||
q,1793911048358.8,1793911048365.9,1793911048377.5,202,,2fa57a4f-d2dd-4e2c-8a3e-a38cb74afc83,39e797bfa6f6,
|
||||
q,1793911048758.8,1793911048759.9,1793911048768.0,202,,8ae7e7b9-38ed-4811-ba72-490208eeb646,7bbaa8668c87,
|
||||
q,1793911049158.8,1793911049164.8,1793911049174.3,202,,c770dbe6-41da-441e-a3f8-0e522401fb81,39e797bfa6f6,
|
||||
q,1793911049558.8,1793911049561.9,1793911049604.5,202,,5100eb40-6ed8-471a-bf45-d7d4b0f22c90,7bbaa8668c87,
|
||||
q,1793911049958.8,1793911049966.0,1793911049986.6,202,,ddc17138-b8f6-4b5d-9bf4-81eb4b8adbca,39e797bfa6f6,
|
||||
q,1793911050358.8,1793911050361.4,1793911050369.8,202,,ac9edcdb-eca1-4b09-b04d-7b06c15a7871,7bbaa8668c87,
|
||||
q,1793911050758.8,1793911050759.9,1793911050768.8,202,,91c81888-069e-4fdd-817b-3610ad5fa661,39e797bfa6f6,
|
||||
q,1793911051158.8,1793911051159.9,1793911051170.5,202,,6017af39-230f-46a8-b9c0-47741e253d88,7bbaa8668c87,
|
||||
q,1793911051558.8,1793911051560.1,1793911051585.5,202,,1f3cfca6-d3e8-449b-8e77-860485bedd2f,39e797bfa6f6,
|
||||
q,1793911051958.8,1793911051963.0,1793911051975.0,202,,28badf43-b850-4092-9946-4e58caa4d51f,7bbaa8668c87,
|
||||
q,1793911052358.8,1793911052359.8,1793911052367.2,202,,62028f82-90ae-4031-a8b4-4d5a64c310f7,39e797bfa6f6,
|
||||
q,1793911052758.8,1793911052759.8,1793911052769.5,202,,69e22e08-2e5a-41cb-a398-d6ee82593165,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 2.5,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793911013.159,
|
||||
"end": 1793911053.159,
|
||||
"finished": 1793911052.769,
|
||||
"scheduled": 100,
|
||||
"sent": 100,
|
||||
"accepted": 100,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 100
|
||||
},
|
||||
"upload_p50_ms": 10.47,
|
||||
"upload_p95_ms": 26.53,
|
||||
"upload_max_ms": 42.68,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.132,
|
||||
"log": "/tmp/c4p4-s-r1-fixed.log"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"rate": 26.8251,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793911053.984,
|
||||
"end": 1793911094.021,
|
||||
"finished": 1793911093.988,
|
||||
"scheduled": 1074,
|
||||
"sent": 1074,
|
||||
"accepted": 586,
|
||||
"rejected_429": 488,
|
||||
"retry_after": {
|
||||
"5": 488
|
||||
},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 586,
|
||||
"429": 488
|
||||
},
|
||||
"upload_p50_ms": 11.88,
|
||||
"upload_p95_ms": 16.36,
|
||||
"upload_max_ms": 39.72,
|
||||
"peak_inflight": 2,
|
||||
"gen_cpu_s": 0.999,
|
||||
"log": "/tmp/c4p4-s-r1-overload.log"
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
q,1793911099230.9,1793911099232.6,1793911099259.4,202,,e42db4f4-b2f7-4630-a448-8c48d9d67e03,39e797bfa6f6,
|
||||
q,1793911099355.1,1793911099356.4,1793911099383.0,202,,3a1a547f-53f1-42f3-84ce-34353d3af1e2,7bbaa8668c87,
|
||||
q,1793911099479.4,1793911099483.0,1793911099492.0,202,,c1956beb-7c60-43ed-b976-6a9d08ae0fca,39e797bfa6f6,
|
||||
q,1793911099603.7,1793911099605.5,1793911099613.2,202,,f714d191-04fc-4fe7-b9c1-1dbf3933445b,7bbaa8668c87,
|
||||
q,1793911099727.9,1793911099728.5,1793911099740.0,202,,7fd4dd45-0bb9-4af6-9f62-262cc0773a64,39e797bfa6f6,
|
||||
q,1793911099852.2,1793911099859.9,1793911099874.5,202,,334eb878-3d6b-4cd7-ac83-a71102f608a4,7bbaa8668c87,
|
||||
q,1793911099976.5,1793911099977.4,1793911099992.8,202,,b8188360-fb3b-42e6-8571-8aba9b273108,39e797bfa6f6,
|
||||
q,1793911100100.7,1793911100103.2,1793911100119.0,202,,b06846fd-74c4-41f4-a37f-d1ca85021d24,7bbaa8668c87,
|
||||
q,1793911100225.0,1793911100229.5,1793911100237.0,202,,dba26df9-3741-4bfb-b737-35ffdbc90baa,39e797bfa6f6,
|
||||
q,1793911100349.2,1793911100354.2,1793911100368.4,202,,a484fd22-1384-49bf-9d5a-af667dfdb8e4,7bbaa8668c87,
|
||||
q,1793911100473.5,1793911100475.4,1793911100490.7,202,,0f4ed578-5da0-4b1e-b054-aba676b80ab5,39e797bfa6f6,
|
||||
q,1793911100597.8,1793911100599.5,1793911100617.9,202,,0da06e99-c00f-429b-8436-796629dfdebc,7bbaa8668c87,
|
||||
q,1793911100722.0,1793911100724.4,1793911100734.1,202,,1b0e1bca-92f0-418a-8bf3-fb8e961a46fa,39e797bfa6f6,
|
||||
q,1793911100846.3,1793911100850.3,1793911100860.8,202,,c1c57f6a-cce4-4f4f-9cf8-f24e8ee58de0,7bbaa8668c87,
|
||||
q,1793911100970.6,1793911100971.0,1793911100983.2,202,,79838373-6216-4b4d-92fc-c333085591d5,39e797bfa6f6,
|
||||
q,1793911101094.8,1793911101096.0,1793911101112.1,202,,7554ac06-ca6b-46a5-bd6d-e02c941aa7a2,7bbaa8668c87,
|
||||
q,1793911101219.1,1793911101220.0,1793911101230.4,202,,85579c31-1295-4a99-ade9-26d1e47f194e,39e797bfa6f6,
|
||||
q,1793911101343.3,1793911101347.0,1793911101358.6,202,,6a4ffc98-eeaf-445f-a453-a987ee94c3fe,7bbaa8668c87,
|
||||
q,1793911101467.6,1793911101468.1,1793911101488.4,202,,69625112-95c3-41fe-babc-74927ea6e046,39e797bfa6f6,
|
||||
q,1793911101591.9,1793911101594.1,1793911101608.6,202,,11b321f1-b1e1-45b8-961f-6e0be945c7a3,7bbaa8668c87,
|
||||
q,1793911101716.1,1793911101717.5,1793911101736.0,202,,ae325fc1-5675-4ef9-a1c4-ca5b544dd3a5,39e797bfa6f6,
|
||||
q,1793911101840.4,1793911101842.6,1793911101851.0,202,,2167f2a2-aacb-483f-a008-496ab570bf4d,7bbaa8668c87,
|
||||
q,1793911101964.6,1793911101967.6,1793911101981.5,202,,65b22b44-8a2d-41c8-946f-68bb35de7061,39e797bfa6f6,
|
||||
q,1793911102088.9,1793911102090.1,1793911102105.6,202,,481fbf80-48a1-4ae6-95e2-080ba3cabc40,7bbaa8668c87,
|
||||
q,1793911102213.2,1793911102215.6,1793911102231.3,202,,941268c6-4f23-4859-989b-4180c97f48ed,39e797bfa6f6,
|
||||
q,1793911102337.4,1793911102339.0,1793911102348.0,202,,d33c0e3e-3a3a-4062-a146-b0d73963232a,7bbaa8668c87,
|
||||
q,1793911102461.7,1793911102463.4,1793911102471.1,202,,b94420c5-b228-44eb-aad7-86e673500786,39e797bfa6f6,
|
||||
q,1793911102586.0,1793911102592.0,1793911102606.8,202,,aaf91194-03c1-4d87-8434-fbef751f4100,7bbaa8668c87,
|
||||
q,1793911102710.2,1793911102713.0,1793911102729.6,202,,b7c86428-0604-44a6-a98c-1de44afbebbc,39e797bfa6f6,
|
||||
q,1793911102834.5,1793911102837.2,1793911102847.6,202,,7506ba80-4bec-4424-9b2c-9736995f6d46,7bbaa8668c87,
|
||||
q,1793911102958.7,1793911102960.3,1793911102976.0,202,,c0645782-9e7e-4798-9753-02ff267064b3,39e797bfa6f6,
|
||||
q,1793911103083.0,1793911103085.4,1793911103099.8,202,,869774c5-89b1-42f6-83b3-8715c9b6c484,7bbaa8668c87,
|
||||
q,1793911103207.3,1793911103208.0,1793911103222.4,202,,35761dac-6b42-4453-b043-6d58781a8947,39e797bfa6f6,
|
||||
q,1793911103331.5,1793911103332.8,1793911103340.6,202,,3cdf940b-b546-44a5-84c1-fc8613768327,7bbaa8668c87,
|
||||
q,1793911103455.8,1793911103458.4,1793911103473.2,202,,167f5ecd-222d-4e57-94e3-c9a3b8e9bacd,39e797bfa6f6,
|
||||
q,1793911103580.0,1793911103583.7,1793911103596.6,202,,d95e27af-28fa-42ce-8cfb-d4221faa8fb3,7bbaa8668c87,
|
||||
q,1793911103704.3,1793911103705.6,1793911103724.6,202,,6ebb4630-c804-4393-b34a-5fe884489a74,39e797bfa6f6,
|
||||
q,1793911103828.6,1793911103834.0,1793911103841.7,202,,4bb59998-3d3b-4796-b672-3746b262072a,7bbaa8668c87,
|
||||
q,1793911103952.8,1793911103955.2,1793911103967.6,202,,6f3431ad-8bf4-40dc-bae4-a0160cbf26f4,39e797bfa6f6,
|
||||
q,1793911104077.1,1793911104078.1,1793911104093.1,202,,84be2778-ac8e-4749-b82b-906d34770953,7bbaa8668c87,
|
||||
q,1793911104201.4,1793911104204.1,1793911104219.0,202,,11a0a675-89de-4c03-a945-0954a88a02a2,39e797bfa6f6,
|
||||
q,1793911104325.6,1793911104334.9,1793911104343.9,202,,b9d7c1ad-853c-4c50-9b68-c4995d423fd7,7bbaa8668c87,
|
||||
q,1793911104449.9,1793911104453.6,1793911104468.0,202,,501e8f7d-6c78-4b88-97cc-2e326fa1a530,39e797bfa6f6,
|
||||
q,1793911104574.1,1793911104576.4,1793911104590.4,202,,91e0312d-50cf-409d-ae99-e3e24cf6a1cd,7bbaa8668c87,
|
||||
q,1793911104698.4,1793911104700.4,1793911104718.1,202,,27030d5e-429a-4953-ace4-0e6dff52378a,39e797bfa6f6,
|
||||
q,1793911104822.7,1793911104825.3,1793911104833.0,202,,55fc5bdb-3f13-4454-9276-10619ea270c4,7bbaa8668c87,
|
||||
q,1793911104946.9,1793911104958.0,1793911104970.9,202,,bad5051d-62bd-4079-91a7-8045212eb6df,39e797bfa6f6,
|
||||
q,1793911105071.2,1793911105073.1,1793911105086.5,202,,6d5776e0-2204-44e5-ace7-fa0263d3d40f,7bbaa8668c87,
|
||||
q,1793911105195.4,1793911105196.5,1793911105210.9,202,,58b76d96-5b5c-4041-a5c2-44841c205438,39e797bfa6f6,
|
||||
q,1793911105319.7,1793911105321.0,1793911105329.2,202,,97a69394-8bfb-47f2-8202-606e6702dee7,7bbaa8668c87,
|
||||
q,1793911105444.0,1793911105448.9,1793911105460.0,202,,a60ff527-66b7-431c-9f07-6af808a97bee,39e797bfa6f6,
|
||||
q,1793911105568.2,1793911105570.8,1793911105583.4,202,,a39cd094-4b81-49f5-8f96-dbea6ab8ba40,7bbaa8668c87,
|
||||
q,1793911105692.5,1793911105693.0,1793911105710.2,202,,e2fd7f8d-454b-4685-b1eb-3bc4fe899b00,39e797bfa6f6,
|
||||
q,1793911105816.8,1793911105818.0,1793911105828.7,202,,acb2ea45-457b-4c1a-a077-9ea228fee1d8,7bbaa8668c87,
|
||||
q,1793911105941.0,1793911105951.4,1793911105962.8,202,,9f78e3a1-9dde-4c99-bc87-87009ac368c4,39e797bfa6f6,
|
||||
q,1793911106065.3,1793911106071.3,1793911106091.4,202,,082542b8-4759-463c-aaea-284df3065877,7bbaa8668c87,
|
||||
q,1793911106189.5,1793911106191.0,1793911106205.7,202,,34c26702-122c-4978-a772-906134bffcb7,39e797bfa6f6,
|
||||
q,1793911106313.8,1793911106317.8,1793911106331.0,202,,20edc6b0-d348-4f89-bb78-d16b76082799,7bbaa8668c87,
|
||||
q,1793911106438.1,1793911106439.5,1793911106447.5,202,,e6863146-4688-4cb1-b515-60b29cfaf10a,39e797bfa6f6,
|
||||
q,1793911106562.3,1793911106563.4,1793911106578.4,202,,9ab50018-7ab0-4fc6-9a0e-44dede4701d0,7bbaa8668c87,
|
||||
q,1793911106686.6,1793911106688.8,1793911106703.4,202,,335623e3-7180-49be-8ff5-fb57696a4b94,39e797bfa6f6,
|
||||
q,1793911106810.8,1793911106814.4,1793911106828.8,202,,81297701-6ae1-4c53-9d5d-c2bbf703b37b,7bbaa8668c87,
|
||||
q,1793911106935.1,1793911106937.4,1793911106945.1,202,,ba816040-c1f9-4c3f-9213-f0822460198c,39e797bfa6f6,
|
||||
q,1793911107059.4,1793911107067.6,1793911107075.2,202,,d33b8214-1ff4-4f82-9523-252655db9739,7bbaa8668c87,
|
||||
q,1793911107183.6,1793911107186.2,1793911107198.0,202,,5b5027e7-a106-41d2-8146-f6d47de0a718,39e797bfa6f6,
|
||||
q,1793911107307.9,1793911107309.9,1793911107325.2,202,,6b140d9e-3c3b-4fd0-997b-4dc2b204eea6,7bbaa8668c87,
|
||||
q,1793911107432.2,1793911107443.1,1793911107453.9,202,,fc9aa824-9929-43a0-8508-ef9a933d98d3,39e797bfa6f6,
|
||||
q,1793911107556.4,1793911107559.1,1793911107568.3,202,,dbc1802c-261f-4f1d-bdd6-e65371206a35,7bbaa8668c87,
|
||||
q,1793911107680.7,1793911107684.3,1793911107699.3,202,,b491dc8d-3833-4a17-82e7-ae53fa3cb6b2,39e797bfa6f6,
|
||||
q,1793911107804.9,1793911107806.0,1793911107823.3,202,,395c0058-c64f-44c5-b224-1793780d7c15,7bbaa8668c87,
|
||||
q,1793911107929.2,1793911107930.0,1793911107938.8,202,,d152c078-4fff-4ec8-b906-e565ff888a28,39e797bfa6f6,
|
||||
q,1793911108053.5,1793911108061.8,1793911108075.2,202,,ce81f9c6-7c66-4fa9-bfab-6a4766c486e5,7bbaa8668c87,
|
||||
q,1793911108177.7,1793911108181.0,1793911108192.0,202,,6c01e15f-9545-46a6-ae51-06413568048b,39e797bfa6f6,
|
||||
q,1793911108302.0,1793911108303.1,1793911108318.7,202,,26df5e88-2961-42dd-b7b3-77e9a16b650e,7bbaa8668c87,
|
||||
q,1793911108426.2,1793911108431.0,1793911108439.6,202,,2682c93f-13d9-4783-922a-6f7bffd19266,39e797bfa6f6,
|
||||
q,1793911108550.5,1793911108553.1,1793911108568.7,202,,1ef29e99-e096-441c-a02b-908fe05904b9,7bbaa8668c87,
|
||||
q,1793911108674.8,1793911108675.3,1793911108691.0,202,,51baf9de-bd43-4c50-be8b-5524c9caaeee,39e797bfa6f6,
|
||||
q,1793911108799.0,1793911108801.2,1793911108816.5,202,,04f8ec7d-f2d5-4867-b25e-1ef5b8d2078e,7bbaa8668c87,
|
||||
q,1793911108923.3,1793911108924.4,1793911108931.7,202,,f37bbdeb-8fcb-4be6-9b8d-e146fc4ca3ab,39e797bfa6f6,
|
||||
q,1793911109047.6,1793911109048.9,1793911109056.9,202,,06284f6f-33c1-407d-8408-63f58947caad,7bbaa8668c87,
|
||||
q,1793911109171.8,1793911109174.2,1793911109189.4,202,,573c0c11-7f61-4eaa-a3aa-c9b245c30b7b,39e797bfa6f6,
|
||||
q,1793911109296.1,1793911109299.5,1793911109315.3,202,,89e7ab17-2d77-4e7e-ab0c-7459c8d40ea8,7bbaa8668c87,
|
||||
q,1793911109420.3,1793911109421.0,1793911109429.0,202,,b9cfb225-7ada-4c84-9f97-f7dff2b8bbba,39e797bfa6f6,
|
||||
q,1793911109544.6,1793911109550.0,1793911109562.8,202,,a7f88695-a97b-4434-8c7f-33e5179dd166,7bbaa8668c87,
|
||||
q,1793911109668.9,1793911109669.3,1793911109682.3,202,,fbe62447-634f-4aff-a361-5e285cca1cf4,39e797bfa6f6,
|
||||
q,1793911109793.1,1793911109794.2,1793911109809.9,202,,8a400a05-41d0-49fb-9952-5fa91fd213ce,7bbaa8668c87,
|
||||
q,1793911109917.4,1793911109919.3,1793911109928.7,202,,99f74fc4-c3b8-4462-a897-55d045185547,39e797bfa6f6,
|
||||
q,1793911110041.6,1793911110044.2,1793911110055.6,202,,8ffe556c-94b6-43cf-9b0f-9e16d636dd89,7bbaa8668c87,
|
||||
q,1793911110165.9,1793911110167.3,1793911110177.6,202,,46710988-0dca-42fc-9c69-63c13dbea760,39e797bfa6f6,
|
||||
q,1793911110290.2,1793911110291.2,1793911110307.1,202,,9b7bc491-a7eb-4d1a-9d83-594bac293ece,7bbaa8668c87,
|
||||
q,1793911110414.4,1793911110416.1,1793911110430.5,202,,cf7de1b0-7fa9-44e7-b35f-9a58025494a3,39e797bfa6f6,
|
||||
q,1793911110538.7,1793911110546.1,1793911110556.3,202,,60f15e78-49f3-47be-a2cb-fe61333152d7,7bbaa8668c87,
|
||||
q,1793911110663.0,1793911110665.8,1793911110698.5,202,,a59bd30b-73d7-416a-9e55-ab7ccd999f46,39e797bfa6f6,
|
||||
q,1793911110787.2,1793911110788.1,1793911110804.1,202,,1b91a60e-5616-4459-ba5e-f352fe78a025,7bbaa8668c87,
|
||||
q,1793911110911.5,1793911110914.0,1793911110932.3,202,,9c302535-2d3f-43f3-a9db-d71cc32672ea,39e797bfa6f6,
|
||||
q,1793911111035.7,1793911111039.8,1793911111048.0,202,,2d4787b2-0960-4616-9451-763d86b9442d,7bbaa8668c87,
|
||||
q,1793911111160.0,1793911111163.9,1793911111178.7,202,,26aadb90-88cc-436b-9a33-3836d013eaf4,39e797bfa6f6,
|
||||
q,1793911111284.3,1793911111287.1,1793911111303.2,202,,7b310778-01a9-43ce-b1eb-5a4153c6bcd3,7bbaa8668c87,
|
||||
q,1793911111408.5,1793911111410.5,1793911111424.9,202,,abf05dd4-3caa-4719-8065-0c76c8910be3,39e797bfa6f6,
|
||||
q,1793911111532.8,1793911111534.4,1793911111543.3,202,,c609c01d-e527-4c38-99e5-485d2ec47ec6,7bbaa8668c87,
|
||||
q,1793911111657.0,1793911111658.5,1793911111668.9,202,,79527b27-80b4-49ee-9bc3-7ccb0fb12bbd,39e797bfa6f6,
|
||||
q,1793911111781.3,1793911111784.5,1793911111800.0,202,,abb7173f-2c61-46cf-a20d-09e3875a4f7d,7bbaa8668c87,
|
||||
q,1793911111905.6,1793911111907.0,1793911111921.9,202,,257e354f-5b86-4595-8bb6-7f6151f2397c,39e797bfa6f6,
|
||||
q,1793911112029.8,1793911112031.2,1793911112039.3,202,,101e4dc4-6e59-4a1a-b966-0301881eeed0,7bbaa8668c87,
|
||||
q,1793911112154.1,1793911112156.5,1793911112164.2,202,,84e7d40a-efe4-4ff2-a445-fa745ba0eec7,39e797bfa6f6,
|
||||
q,1793911112278.4,1793911112284.0,1793911112296.1,202,,eb47755f-f423-4301-90fa-13fc749dd6be,7bbaa8668c87,
|
||||
q,1793911112402.6,1793911112405.1,1793911112420.5,202,,fb425651-cba2-48b6-85d8-b7535b545db9,39e797bfa6f6,
|
||||
q,1793911112526.9,1793911112532.3,1793911112542.4,202,,ec5e7f7e-c728-4850-821d-e163887727fb,7bbaa8668c87,
|
||||
q,1793911112651.1,1793911112662.1,1793911112682.5,202,,116e500c-a34f-41cf-b196-899d42cbff4d,39e797bfa6f6,
|
||||
q,1793911112775.4,1793911112776.6,1793911112784.5,202,,339544ec-e35e-4456-83cf-66e9400bb96a,7bbaa8668c87,
|
||||
q,1793911112899.7,1793911112901.1,1793911112916.7,202,,e2d4985a-01a4-420b-8858-02a36c2945ba,39e797bfa6f6,
|
||||
q,1793911113023.9,1793911113026.2,1793911113034.6,202,,17f94fbb-578a-4930-be39-6d460bd9f81d,7bbaa8668c87,
|
||||
q,1793911113148.2,1793911113149.1,1793911113158.7,202,,09f6add0-90ed-427c-a996-de13d5c7bee5,39e797bfa6f6,
|
||||
q,1793911113272.4,1793911113273.0,1793911113281.1,202,,e1487b2f-7eb7-4766-8dc6-a5ff1c3c6d03,7bbaa8668c87,
|
||||
q,1793911113396.7,1793911113398.1,1793911113413.0,202,,3c723851-4dae-4199-b738-dbb7b2b7e288,39e797bfa6f6,
|
||||
q,1793911113521.0,1793911113524.5,1793911113532.8,202,,aebdf7bb-54ce-4141-8be1-dc756eaac3cc,7bbaa8668c87,
|
||||
q,1793911113645.2,1793911113649.1,1793911113663.6,202,,27d992c7-6524-47bd-a429-c5531e2277e3,39e797bfa6f6,
|
||||
q,1793911113769.5,1793911113770.0,1793911113783.4,202,,18f0ad5f-7741-4abf-bf97-8958779b5efe,7bbaa8668c87,
|
||||
q,1793911113893.8,1793911113902.1,1793911113915.7,202,,0c0afdd8-f859-4ac2-bc27-6256da5890d7,39e797bfa6f6,
|
||||
q,1793911114018.0,1793911114020.1,1793911114027.3,202,,d891ce5a-eb22-45a0-aea6-8b13c638a2d9,7bbaa8668c87,
|
||||
q,1793911114142.3,1793911114143.9,1793911114157.0,202,,22520492-8ce8-40e4-b3b1-25082ae07ac5,39e797bfa6f6,
|
||||
q,1793911114266.5,1793911114268.0,1793911114281.4,202,,41e7f8f8-0782-4290-a515-02b427984613,7bbaa8668c87,
|
||||
q,1793911114390.8,1793911114392.2,1793911114406.4,202,,50c96288-a567-4a1c-bef0-0bc4898b2ae5,39e797bfa6f6,
|
||||
q,1793911114515.1,1793911114516.0,1793911114524.2,202,,e699d9da-b47f-4ee1-bb1a-dfb8935a6501,7bbaa8668c87,
|
||||
q,1793911114639.3,1793911114644.1,1793911114657.8,202,,7e49868c-021f-4a7b-a0a2-a40bcf01fded,39e797bfa6f6,
|
||||
q,1793911114763.6,1793911114764.2,1793911114781.4,202,,507ef441-942a-4e2a-8b5f-d9233e656a09,7bbaa8668c87,
|
||||
q,1793911114887.8,1793911114889.9,1793911114905.1,202,,5d3d50c6-df61-4b91-9213-35d0d6d6f519,39e797bfa6f6,
|
||||
q,1793911115012.1,1793911115013.2,1793911115023.3,202,,b678d793-015f-4625-b237-030a9cbbe411,7bbaa8668c87,
|
||||
q,1793911115136.4,1793911115139.6,1793911115151.6,202,,744f16cb-d6df-468a-bb9b-a04e2fd8a6c5,39e797bfa6f6,
|
||||
q,1793911115260.6,1793911115262.0,1793911115274.8,202,,c2843fcb-7859-4763-b8d8-4827872a2635,7bbaa8668c87,
|
||||
q,1793911115384.9,1793911115387.4,1793911115401.7,202,,ac570293-8e2f-4237-8f9d-d18f45549071,39e797bfa6f6,
|
||||
q,1793911115509.2,1793911115511.0,1793911115525.2,202,,190ec6de-f853-4fbe-aaab-251be4d8647c,7bbaa8668c87,
|
||||
q,1793911115633.4,1793911115635.5,1793911115645.6,202,,2f251127-961e-453f-864e-12d1759b89d5,39e797bfa6f6,
|
||||
q,1793911115757.7,1793911115759.4,1793911115773.0,202,,85cb6f96-d790-47c3-8049-252d52d6d9d0,7bbaa8668c87,
|
||||
q,1793911115881.9,1793911115884.2,1793911115896.6,202,,76a92e43-097d-4029-bc25-a89249b06cf2,39e797bfa6f6,
|
||||
q,1793911116006.2,1793911116007.5,1793911116022.8,202,,e723e140-e3b6-4ef8-bec0-d7c9e2a9c38e,7bbaa8668c87,
|
||||
q,1793911116130.5,1793911116132.5,1793911116141.6,202,,8ef723b2-e361-4595-8353-3c71b52891cf,39e797bfa6f6,
|
||||
q,1793911116254.7,1793911116256.1,1793911116274.0,202,,d088b63e-f128-4ec2-bc3b-76b02fce85b5,7bbaa8668c87,
|
||||
q,1793911116379.0,1793911116382.8,1793911116396.2,202,,bd7270a9-a261-41df-8a4f-dfae0f8785a4,39e797bfa6f6,
|
||||
q,1793911116503.2,1793911116504.5,1793911116519.5,202,,38187720-b7d7-4edb-9ef9-791735bdeebf,7bbaa8668c87,
|
||||
q,1793911116627.5,1793911116628.3,1793911116637.1,202,,7dcba070-5178-4313-990b-7da828ddc2a4,39e797bfa6f6,
|
||||
q,1793911116751.8,1793911116755.3,1793911116765.4,202,,f4fa8b76-f39f-4fc0-89ec-f0f7de8a7073,7bbaa8668c87,
|
||||
q,1793911116876.0,1793911116879.9,1793911116890.6,202,,ed40865a-b71e-4be0-b306-7f24e03dec3b,39e797bfa6f6,
|
||||
q,1793911117000.3,1793911117003.0,1793911117018.8,202,,49e9b3f8-677e-4894-aa17-9a2f6d5926e0,7bbaa8668c87,
|
||||
q,1793911117124.6,1793911117126.4,1793911117134.9,202,,2337d811-98a4-4c36-9ff8-3efa17933173,39e797bfa6f6,
|
||||
q,1793911117248.8,1793911117251.0,1793911117259.1,202,,5c1c1a6e-4ab5-47fe-adc9-d560d9b9ccb5,7bbaa8668c87,
|
||||
q,1793911117373.1,1793911117373.6,1793911117381.7,202,,43f3e0e0-2be1-4482-91f1-4a6247dd545c,39e797bfa6f6,
|
||||
q,1793911117497.3,1793911117498.3,1793911117513.0,202,,9743f1db-5fc8-4bd5-90e1-18cbce5784b8,7bbaa8668c87,
|
||||
q,1793911117621.6,1793911117623.5,1793911117632.9,202,,7d5b64b3-93e0-49b6-a8f1-c8b890fc3636,39e797bfa6f6,
|
||||
q,1793911117745.9,1793911117747.3,1793911117760.6,202,,78e6b005-9702-4140-a354-0e233538ace2,7bbaa8668c87,
|
||||
q,1793911117870.1,1793911117873.4,1793911117881.4,202,,efa2423c-004e-4a0a-8fe7-152aaca3586b,39e797bfa6f6,
|
||||
q,1793911117994.4,1793911117995.5,1793911118013.6,202,,6de70091-3e52-4424-8745-893bd3ceaab0,7bbaa8668c87,
|
||||
q,1793911118118.6,1793911118120.2,1793911118128.4,202,,1442fe77-9da8-4a72-abb8-d1f4a85f3977,39e797bfa6f6,
|
||||
q,1793911118242.9,1793911118246.0,1793911118260.9,202,,f4a2bc88-29c8-400b-91ae-0ec2d3c9b3be,7bbaa8668c87,
|
||||
q,1793911118367.2,1793911118369.0,1793911118379.3,202,,c0afc217-9ca4-4262-8cdb-ea5fbd007427,39e797bfa6f6,
|
||||
q,1793911118491.4,1793911118495.2,1793911118512.0,202,,8f5cb83d-22fb-4289-b022-242d4734c087,7bbaa8668c87,
|
||||
q,1793911118615.7,1793911118618.0,1793911118626.1,202,,05a858b3-2334-4c25-9194-f43134b02b3f,39e797bfa6f6,
|
||||
q,1793911118740.0,1793911118741.0,1793911118751.0,202,,b44eaadc-b5f6-4214-a120-55dcf4928d22,7bbaa8668c87,
|
||||
q,1793911118864.2,1793911118867.1,1793911118880.1,202,,67e00d6e-b6c2-4466-b4e7-b4051a0a90b2,39e797bfa6f6,
|
||||
q,1793911118988.5,1793911118991.0,1793911119004.7,202,,57679283-2417-47f9-87e3-d1118125cbfa,7bbaa8668c87,
|
||||
q,1793911119112.7,1793911119115.3,1793911119124.4,202,,51cbf17d-8940-40f8-9167-ee5f0005c700,39e797bfa6f6,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 8.04754,
|
||||
"duration_s": 20.0,
|
||||
"start": 1793911099.231,
|
||||
"end": 1793911119.237,
|
||||
"finished": 1793911119.125,
|
||||
"scheduled": 161,
|
||||
"sent": 161,
|
||||
"accepted": 161,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 161
|
||||
},
|
||||
"upload_p50_ms": 13.36,
|
||||
"upload_p95_ms": 18.47,
|
||||
"upload_max_ms": 32.72,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.169,
|
||||
"log": "/tmp/c4p4-s-r1-recovery.log"
|
||||
}
|
||||
@@ -0,0 +1,322 @@
|
||||
q,1793910972008.2,1793910972009.9,1793910972038.6,202,,b28703c3-cc05-4c8e-a931-c5edfc49c95a,39e797bfa6f6,
|
||||
q,1793910972132.5,1793910972134.0,1793910972158.8,202,,a4a288cb-d478-447e-9451-29468251f38d,7bbaa8668c87,
|
||||
q,1793910972256.8,1793910972258.8,1793910972282.8,202,,fd0cc02f-39c6-40d5-b7d8-240593f22551,39e797bfa6f6,
|
||||
q,1793910972381.0,1793910972383.9,1793910972409.6,202,,74fb9bf4-a67d-41aa-ae70-5039a7ef7c81,7bbaa8668c87,
|
||||
q,1793910972505.3,1793910972507.1,1793910972520.7,202,,98df3a90-0249-4c9f-ac98-d373fb3ce8ef,39e797bfa6f6,
|
||||
q,1793910972629.5,1793910972630.9,1793910972640.9,202,,1d9a59b0-8113-406b-97fb-005c38f5b27c,7bbaa8668c87,
|
||||
q,1793910972753.8,1793910972755.3,1793910972764.8,202,,b10f3643-f498-4961-9809-4f627a7cb4e8,39e797bfa6f6,
|
||||
q,1793910972878.1,1793910972880.1,1793910972889.7,202,,9d1f62ea-5f40-4082-a274-ed9f81e261b3,7bbaa8668c87,
|
||||
q,1793910973002.3,1793910973003.4,1793910973013.7,202,,c1c0c35a-abd6-4088-9549-a80e3779d1f2,39e797bfa6f6,
|
||||
q,1793910973126.6,1793910973127.4,1793910973141.4,202,,ee938915-e0ab-4714-b025-0eaf293ba85d,7bbaa8668c87,
|
||||
q,1793910973250.9,1793910973252.6,1793910973260.6,202,,3789cb5b-3b26-451b-b816-e1e6e2f1c9b6,39e797bfa6f6,
|
||||
q,1793910973375.1,1793910973376.7,1793910973384.8,202,,55045a60-2390-4ba1-be4c-1ae4ce5da2fb,7bbaa8668c87,
|
||||
q,1793910973499.4,1793910973500.5,1793910973510.5,202,,0ee664ff-400b-4210-81f2-08910a44b7dc,39e797bfa6f6,
|
||||
q,1793910973623.6,1793910973624.8,1793910973638.7,202,,51f69835-6454-4d6a-a117-9d26deb9d56c,7bbaa8668c87,
|
||||
q,1793910973747.9,1793910973749.3,1793910973757.6,202,,0ab5a76f-bd6e-4d92-9097-97a93ed7174b,39e797bfa6f6,
|
||||
q,1793910973872.2,1793910973874.2,1793910973884.6,202,,43ce0a65-d2e7-4f5f-99d6-b0bf2158588c,7bbaa8668c87,
|
||||
q,1793910973996.4,1793910974001.7,1793910974019.7,202,,b31fbadd-812d-4dca-9036-e2a7451679fd,39e797bfa6f6,
|
||||
q,1793910974120.7,1793910974121.6,1793910974135.4,202,,dc5ad778-7a04-4d3a-ab76-acbb4f0adee6,7bbaa8668c87,
|
||||
q,1793910974244.9,1793910974246.5,1793910974253.8,202,,87c82491-0ba6-4f4c-991b-eb284c9a6772,39e797bfa6f6,
|
||||
q,1793910974369.2,1793910974371.7,1793910974381.0,202,,dac0d697-b64d-44db-8a8a-87140014fb09,7bbaa8668c87,
|
||||
q,1793910974493.5,1793910974499.4,1793910974533.6,202,,357f9a88-322c-4c18-a726-b7acae439af1,39e797bfa6f6,
|
||||
q,1793910974617.7,1793910974619.4,1793910974634.8,202,,a6bc15f0-daf1-4fab-9f59-b8c75f677a27,7bbaa8668c87,
|
||||
q,1793910974742.0,1793910974743.3,1793910974752.2,202,,1fbd1839-007d-4d3b-b3e0-d873aff7aadf,39e797bfa6f6,
|
||||
q,1793910974866.3,1793910974872.4,1793910974884.0,202,,695591d7-774d-495a-8996-e052fa1819b3,7bbaa8668c87,
|
||||
q,1793910974990.5,1793910974992.7,1793910975006.5,202,,cd32cf3f-4308-4d2f-b732-bdffcf0bd9e3,39e797bfa6f6,
|
||||
q,1793910975114.8,1793910975117.2,1793910975132.8,202,,196319b0-528b-4c5b-9491-8f76bd4d7b1a,7bbaa8668c87,
|
||||
q,1793910975239.0,1793910975240.8,1793910975249.1,202,,c7a01bd5-00b2-4788-af9a-c54d70bef5c0,39e797bfa6f6,
|
||||
q,1793910975363.3,1793910975367.4,1793910975376.0,202,,4e38f786-3b7b-46bd-8813-202eeb4b073f,7bbaa8668c87,
|
||||
q,1793910975487.6,1793910975488.8,1793910975498.4,202,,ded7b4dc-057d-4db1-bb7d-975946360df6,39e797bfa6f6,
|
||||
q,1793910975611.8,1793910975618.6,1793910975635.5,202,,d2422172-bcf7-4902-b0c8-40371251cd4c,7bbaa8668c87,
|
||||
q,1793910975736.1,1793910975737.5,1793910975749.2,202,,eb682482-1a28-4157-a179-c70faba7ce6a,39e797bfa6f6,
|
||||
q,1793910975860.3,1793910975862.8,1793910975871.9,202,,4cc13c26-bde8-4ca8-b974-a62853e97861,7bbaa8668c87,
|
||||
q,1793910975984.6,1793910975990.6,1793910976008.5,202,,0ec33bb3-453a-4dc2-9fd5-d65d90944503,39e797bfa6f6,
|
||||
q,1793910976108.9,1793910976110.4,1793910976125.8,202,,19e706e2-5422-42d2-9e68-335bf0c652b0,7bbaa8668c87,
|
||||
q,1793910976233.1,1793910976235.1,1793910976245.0,202,,b592c630-987c-4673-b6c8-fceb25eb5f6f,39e797bfa6f6,
|
||||
q,1793910976357.4,1793910976363.4,1793910976371.4,202,,164c3895-21e2-463e-8c18-0b3f17c11cab,7bbaa8668c87,
|
||||
q,1793910976481.7,1793910976488.3,1793910976500.3,202,,ee43a008-15a0-49f7-b0d2-df6451466f03,39e797bfa6f6,
|
||||
q,1793910976605.9,1793910976607.4,1793910976623.3,202,,8113368e-8361-4da2-a6d4-44b2c343aaae,7bbaa8668c87,
|
||||
q,1793910976730.2,1793910976731.6,1793910976744.2,202,,248bb26a-ef19-433a-ae9d-da4bf165cb5b,39e797bfa6f6,
|
||||
q,1793910976854.4,1793910976856.4,1793910976864.6,202,,b7a58a2e-516e-41f5-9b0b-b29e0c056994,7bbaa8668c87,
|
||||
q,1793910976978.7,1793910976982.6,1793910976991.0,202,,dd0adb95-632c-4200-ae71-6234da9b0e3c,39e797bfa6f6,
|
||||
q,1793910977103.0,1793910977103.4,1793910977118.4,202,,9d029d86-8896-47b0-aa6c-c08549574f22,7bbaa8668c87,
|
||||
q,1793910977227.2,1793910977228.8,1793910977240.7,202,,0a9ff6a6-8704-4e08-871c-f50bb214c9aa,39e797bfa6f6,
|
||||
q,1793910977351.5,1793910977352.6,1793910977360.3,202,,1bb23435-934e-432e-aedb-17bc9ebad48e,7bbaa8668c87,
|
||||
q,1793910977475.7,1793910977480.2,1793910977493.2,202,,2b751cbc-e234-4dba-aa00-faa70f9871bc,39e797bfa6f6,
|
||||
q,1793910977600.0,1793910977601.4,1793910977615.1,202,,e64c2f98-7ec5-42a1-a1c9-184d2c83b352,7bbaa8668c87,
|
||||
q,1793910977724.3,1793910977725.4,1793910977739.0,202,,35771a71-968a-4306-9e27-406fdfd5b125,39e797bfa6f6,
|
||||
q,1793910977848.5,1793910977850.3,1793910977858.4,202,,96cfd44c-784b-4385-b0e9-fe03e89ac45b,7bbaa8668c87,
|
||||
q,1793910977972.8,1793910977975.5,1793910977984.7,202,,c6e2fed9-9b00-450e-9cc7-51200814ae64,39e797bfa6f6,
|
||||
q,1793910978097.1,1793910978098.6,1793910978109.2,202,,65481ff5-e1e8-42ee-9c3f-d6194bb61bce,7bbaa8668c87,
|
||||
q,1793910978221.3,1793910978223.6,1793910978257.4,202,,de38afba-04a2-49ae-9015-fca5273f17ce,39e797bfa6f6,
|
||||
q,1793910978345.6,1793910978348.3,1793910978355.6,202,,b3a42369-8f1f-4440-adf9-e97716788288,7bbaa8668c87,
|
||||
q,1793910978469.8,1793910978472.4,1793910978483.5,202,,abfac0f1-5bc7-48cc-83ed-700a557dc8ec,39e797bfa6f6,
|
||||
q,1793910978594.1,1793910978595.5,1793910978608.6,202,,b0ef2b44-f137-4605-9469-0310ded4bb25,7bbaa8668c87,
|
||||
q,1793910978718.4,1793910978719.6,1793910978737.9,202,,2135032b-c0a1-466a-b048-d669e566aefd,39e797bfa6f6,
|
||||
q,1793910978842.6,1793910978844.1,1793910978852.6,202,,220062eb-f437-48df-a63f-1db9255c1af4,7bbaa8668c87,
|
||||
q,1793910978966.9,1793910978969.1,1793910978979.0,202,,e4571f7b-b481-4502-9faf-d2bb5a4dec89,39e797bfa6f6,
|
||||
q,1793910979091.1,1793910979093.0,1793910979110.8,202,,41b8eb73-8c8b-40f0-be91-c76ea6a4a942,7bbaa8668c87,
|
||||
q,1793910979215.4,1793910979216.4,1793910979231.8,202,,2e0b452a-ebce-448f-963b-b70ab034172d,39e797bfa6f6,
|
||||
q,1793910979339.7,1793910979340.4,1793910979349.4,202,,01b612fa-a1a1-466a-80d7-e5a93cde4e47,7bbaa8668c87,
|
||||
q,1793910979463.9,1793910979465.2,1793910979475.4,202,,2883166a-ace6-4784-a3ae-3991354e1846,39e797bfa6f6,
|
||||
q,1793910979588.2,1793910979589.5,1793910979606.5,202,,6422551e-7d35-436a-88eb-77d35337f7dd,7bbaa8668c87,
|
||||
q,1793910979712.5,1793910979713.6,1793910979728.7,202,,1f0aab67-a130-4983-b455-38fb0bd10eda,39e797bfa6f6,
|
||||
q,1793910979836.7,1793910979837.4,1793910979846.2,202,,304a1456-0559-4608-bdd1-56c2e15244db,7bbaa8668c87,
|
||||
q,1793910979961.0,1793910979963.1,1793910979970.7,202,,3f7de732-6fbf-45c7-9100-ee82a135d9d3,39e797bfa6f6,
|
||||
q,1793910980085.2,1793910980086.6,1793910980094.2,202,,934c58c3-1c51-4eab-90c6-849218e05fc8,7bbaa8668c87,
|
||||
q,1793910980209.5,1793910980211.7,1793910980228.4,202,,71826769-08cb-4f27-a7b7-d05954bc4aeb,39e797bfa6f6,
|
||||
q,1793910980333.8,1793910980335.5,1793910980344.2,202,,13a16f02-a0b5-412f-9165-9180da622dc2,7bbaa8668c87,
|
||||
q,1793910980458.0,1793910980460.1,1793910980470.8,202,,cb4c1b95-0996-4012-867f-d2f8c23f96ca,39e797bfa6f6,
|
||||
q,1793910980582.3,1793910980583.6,1793910980591.7,202,,095f1e22-c63d-41ba-a82a-5d5c07fa75cb,7bbaa8668c87,
|
||||
q,1793910980706.5,1793910980707.6,1793910980723.8,202,,d0fe47b3-b1a0-446d-9707-7cc10dbad29b,39e797bfa6f6,
|
||||
q,1793910980830.8,1793910980832.1,1793910980843.6,202,,af86a748-e350-4f86-b256-3397290a11a3,7bbaa8668c87,
|
||||
q,1793910980955.1,1793910980961.4,1793910980970.9,202,,fb2cd1b4-3c9d-47cd-874c-eb3ef81f9a8d,39e797bfa6f6,
|
||||
q,1793910981079.3,1793910981085.5,1793910981103.0,202,,62b027d8-12cd-401d-9662-05744644cf55,7bbaa8668c87,
|
||||
q,1793910981203.6,1793910981204.9,1793910981220.1,202,,28247df4-dce4-49ce-a2df-f93fe6c43dff,39e797bfa6f6,
|
||||
q,1793910981327.9,1793910981330.1,1793910981342.7,202,,17330eaa-fcc7-451a-adab-40d990dd136b,7bbaa8668c87,
|
||||
q,1793910981452.1,1793910981458.4,1793910981466.4,202,,6a9af19d-7e90-4933-aa3a-a8a5c91a898e,39e797bfa6f6,
|
||||
q,1793910981576.4,1793910981582.4,1793910981597.4,202,,d9867edc-5293-44e1-ac96-76c0cf8d8af0,7bbaa8668c87,
|
||||
q,1793910981700.6,1793910981703.4,1793910981720.1,202,,b6c96bf1-35f3-433d-82a5-29deec8a726f,39e797bfa6f6,
|
||||
q,1793910981824.9,1793910981825.6,1793910981839.4,202,,41895109-ba00-4863-9783-a720261fa16f,7bbaa8668c87,
|
||||
q,1793910981949.2,1793910981951.1,1793910981959.2,202,,94c9ae9e-c864-4dd3-9e7d-48e4f8781fe4,39e797bfa6f6,
|
||||
q,1793910982073.4,1793910982075.8,1793910982083.7,202,,4e5f7e41-e37a-473c-952c-7244c395cbd2,7bbaa8668c87,
|
||||
q,1793910982197.7,1793910982199.1,1793910982215.2,202,,0cda94c4-6a3f-4a44-adbf-e30dbe39d360,39e797bfa6f6,
|
||||
q,1793910982321.9,1793910982323.5,1793910982338.9,202,,b8363466-ee77-459f-b780-d01738026d8b,7bbaa8668c87,
|
||||
q,1793910982446.2,1793910982447.8,1793910982454.8,202,,edb3b6b2-72e2-4624-8349-4d57a5cbb392,39e797bfa6f6,
|
||||
q,1793910982570.5,1793910982576.4,1793910982589.0,202,,93d9da09-1ebb-428f-b1d1-42bb779d924d,7bbaa8668c87,
|
||||
q,1793910982694.7,1793910982695.3,1793910982707.0,202,,9f6ea436-b0cc-4dbe-a5d9-fc4ffea47bc5,39e797bfa6f6,
|
||||
q,1793910982819.0,1793910982820.6,1793910982835.5,202,,85f04242-ccb1-4fb4-bebd-73b4dae03bd9,7bbaa8668c87,
|
||||
q,1793910982943.3,1793910982944.4,1793910982951.7,202,,c0260951-89a6-4ae7-b438-6469ee85d400,39e797bfa6f6,
|
||||
q,1793910983067.5,1793910983068.1,1793910983078.7,202,,20ae5e9d-ae10-4e12-b36a-3bd00e2d66dc,7bbaa8668c87,
|
||||
q,1793910983191.8,1793910983193.5,1793910983206.5,202,,2f6284b4-1915-4aff-a64a-b4b6ddb76fba,39e797bfa6f6,
|
||||
q,1793910983316.0,1793910983317.4,1793910983332.0,202,,d63ee0c9-56b2-4231-9861-9ce4314c1cc7,7bbaa8668c87,
|
||||
q,1793910983440.3,1793910983441.6,1793910983450.2,202,,543606a9-d1bf-4910-a9f2-1562105e1158,39e797bfa6f6,
|
||||
q,1793910983564.6,1793910983566.7,1793910983574.4,202,,be489d73-875f-4c95-a261-53c6d4303f57,7bbaa8668c87,
|
||||
q,1793910983688.8,1793910983692.1,1793910983702.5,202,,f4c2ca12-d1ee-412f-9618-fe85e0829fc0,39e797bfa6f6,
|
||||
q,1793910983813.1,1793910983814.4,1793910983829.9,202,,12556d47-4f01-48bc-ac2a-89c45c955ff5,7bbaa8668c87,
|
||||
q,1793910983937.4,1793910983938.5,1793910983946.9,202,,92d51a3b-9cb4-4294-818e-0179e433578c,39e797bfa6f6,
|
||||
q,1793910984061.6,1793910984062.6,1793910984074.0,202,,a331bbc8-1cf5-4045-b6e1-74c2bf3e6951,7bbaa8668c87,
|
||||
q,1793910984185.9,1793910984186.5,1793910984208.2,202,,7bb20f1e-88ec-413c-877f-6a8aad84f00d,39e797bfa6f6,
|
||||
q,1793910984310.1,1793910984311.4,1793910984326.4,202,,7710d183-c408-459f-b590-1f881557cb41,7bbaa8668c87,
|
||||
q,1793910984434.4,1793910984435.6,1793910984446.1,202,,3aaa16df-f763-41d3-8bd2-41542a32dd57,39e797bfa6f6,
|
||||
q,1793910984558.7,1793910984559.4,1793910984568.9,202,,aadb2b82-b4ef-4a12-8d2d-6dca95674940,7bbaa8668c87,
|
||||
q,1793910984682.9,1793910984684.6,1793910984699.2,202,,ac3573fc-78cb-4660-b9b1-60e43475355c,39e797bfa6f6,
|
||||
q,1793910984807.2,1793910984808.4,1793910984822.9,202,,3ff6e0c6-4b18-4ccb-9b61-93eb388a2c6e,7bbaa8668c87,
|
||||
q,1793910984931.4,1793910984932.6,1793910984942.7,202,,caa73ad3-0ead-43a4-bb0c-210635865bc6,39e797bfa6f6,
|
||||
q,1793910985055.7,1793910985056.4,1793910985064.5,202,,ed8cbdc5-3df4-412d-9c9a-6b1f020b925b,7bbaa8668c87,
|
||||
q,1793910985180.0,1793910985181.7,1793910985190.3,202,,df688c01-5e2d-4d86-89e1-69adba67aecb,39e797bfa6f6,
|
||||
q,1793910985304.2,1793910985304.6,1793910985320.2,202,,8941fb11-ec5d-47e6-bcb8-9d8e31da33b6,7bbaa8668c87,
|
||||
q,1793910985428.5,1793910985431.1,1793910985440.0,202,,99a2a07f-5557-4c21-9ab8-3ab066c8ad63,39e797bfa6f6,
|
||||
q,1793910985552.8,1793910985555.6,1793910985565.7,202,,90f888d9-89b8-4e10-b7bb-8b712113df2d,7bbaa8668c87,
|
||||
q,1793910985677.0,1793910985681.6,1793910985693.4,202,,9448f336-f876-467b-a5ff-f5814161096b,39e797bfa6f6,
|
||||
q,1793910985801.3,1793910985802.5,1793910985817.6,202,,92b56cc9-d1bd-49a8-a1bb-ea18f65c20ff,7bbaa8668c87,
|
||||
q,1793910985925.5,1793910985927.4,1793910985939.4,202,,679a8c7d-acc0-4073-a445-ebc217156623,39e797bfa6f6,
|
||||
q,1793910986049.8,1793910986054.6,1793910986062.8,202,,e4235e4f-a042-44ac-94a2-ad0646b3000c,7bbaa8668c87,
|
||||
q,1793910986174.1,1793910986177.3,1793910986191.2,202,,89d61d3a-3080-491f-9d10-5defdc2987fd,39e797bfa6f6,
|
||||
q,1793910986298.3,1793910986300.5,1793910986314.9,202,,eb2924a8-1488-4f4e-8b0b-39c1421bf6ad,7bbaa8668c87,
|
||||
q,1793910986422.6,1793910986423.5,1793910986438.9,202,,4903c75a-b28c-48f8-9630-8dc8f2358c12,39e797bfa6f6,
|
||||
q,1793910986546.8,1793910986549.0,1793910986556.4,202,,3ea14b28-c02f-43c3-8fa2-85354b4a336e,7bbaa8668c87,
|
||||
q,1793910986671.1,1793910986677.3,1793910986691.2,202,,9864fd6d-6a4b-434b-ba1e-97b894b0d6c9,39e797bfa6f6,
|
||||
q,1793910986795.4,1793910986797.5,1793910986812.1,202,,827664ea-e440-4929-9c5c-44844f1c3086,7bbaa8668c87,
|
||||
q,1793910986919.6,1793910986921.9,1793910986937.5,202,,f690f923-4258-4bd2-ad48-4ff0d1eae6f4,39e797bfa6f6,
|
||||
q,1793910987043.9,1793910987047.5,1793910987054.5,202,,e63da156-de42-4a9d-b897-41a48979a59d,7bbaa8668c87,
|
||||
q,1793910987168.2,1793910987170.9,1793910987183.5,202,,21dce0ee-3e49-4cdc-a980-d1d865c7c585,39e797bfa6f6,
|
||||
q,1793910987292.4,1793910987294.6,1793910987310.0,202,,859f0e3e-2ddd-4bf7-bc7a-88271323dbec,7bbaa8668c87,
|
||||
q,1793910987416.7,1793910987417.4,1793910987432.6,202,,ee38abab-69ff-4c6a-ba6a-9c712d1638e0,39e797bfa6f6,
|
||||
q,1793910987540.9,1793910987542.4,1793910987551.8,202,,645f903e-fa24-4aec-acf1-f80b79644274,7bbaa8668c87,
|
||||
q,1793910987665.2,1793910987666.4,1793910987675.0,202,,1bc18979-8026-469a-8871-eda43fc3cbca,39e797bfa6f6,
|
||||
q,1793910987789.5,1793910987790.5,1793910987806.0,202,,3ed7e6b3-f123-4370-a124-70da702b86bc,7bbaa8668c87,
|
||||
q,1793910987913.7,1793910987914.6,1793910987929.9,202,,7fac7049-95f5-426e-8395-040f84d20663,39e797bfa6f6,
|
||||
q,1793910988038.0,1793910988038.6,1793910988045.6,202,,a8c780df-60a0-423a-b8f6-5ee754d048a5,7bbaa8668c87,
|
||||
q,1793910988162.2,1793910988168.4,1793910988180.8,202,,2111ef25-0f44-45d3-a4e3-8456f7807335,39e797bfa6f6,
|
||||
q,1793910988286.5,1793910988287.8,1793910988302.5,202,,6675429c-4e4e-454b-8183-fcab72da4094,7bbaa8668c87,
|
||||
q,1793910988410.8,1793910988412.6,1793910988428.3,202,,b97a3f1b-4262-4017-9e31-b3e42fce997b,39e797bfa6f6,
|
||||
q,1793910988535.0,1793910988536.4,1793910988544.9,202,,8df7e29c-043f-41a3-897b-61b7d37615b2,7bbaa8668c87,
|
||||
q,1793910988659.3,1793910988665.4,1793910988675.0,202,,b77c5abd-ed5d-4710-91d2-947c73fcddfc,39e797bfa6f6,
|
||||
q,1793910988783.6,1793910988784.4,1793910988793.4,202,,c631e3f6-6ff8-4133-9bdb-8e7f94e0939d,7bbaa8668c87,
|
||||
q,1793910988907.8,1793910988908.6,1793910988924.6,202,,e5ff79e7-bf3c-485b-bf3a-6839c6840d65,39e797bfa6f6,
|
||||
q,1793910989032.1,1793910989033.5,1793910989042.3,202,,23e76d27-411a-4d8d-bdcd-271277a3e03f,7bbaa8668c87,
|
||||
q,1793910989156.3,1793910989163.4,1793910989177.8,202,,3a2c7228-e99a-42d6-b0ca-fe063b693b3c,39e797bfa6f6,
|
||||
q,1793910989280.6,1793910989282.6,1793910989299.7,202,,032f0819-c325-4e77-bd09-3ae84175ce80,7bbaa8668c87,
|
||||
q,1793910989404.9,1793910989406.4,1793910989421.9,202,,ca1c394a-72ba-408f-be3e-be01327342e7,39e797bfa6f6,
|
||||
q,1793910989529.1,1793910989530.4,1793910989541.6,202,,6a5486f1-35a6-4f91-9595-78cd50d7ec71,7bbaa8668c87,
|
||||
q,1793910989653.4,1793910989654.9,1793910989663.5,202,,9e00cfe0-58c7-473b-831e-3decd05c6502,39e797bfa6f6,
|
||||
q,1793910989777.6,1793910989779.7,1793910989788.9,202,,0a1b1334-84dc-4f20-8f88-95677aa4e8ea,7bbaa8668c87,
|
||||
q,1793910989901.9,1793910989902.5,1793910989916.4,202,,65bd5fbf-5d26-46a1-b4ff-cd7ee2c5c96d,39e797bfa6f6,
|
||||
q,1793910990026.2,1793910990027.4,1793910990038.3,202,,cb942e99-7d64-4b75-8ff1-b853c9bb7ec1,7bbaa8668c87,
|
||||
q,1793910990150.4,1793910990152.3,1793910990161.7,202,,2dd91202-7542-473c-a646-1e549567258d,39e797bfa6f6,
|
||||
q,1793910990274.7,1793910990276.4,1793910990284.6,202,,e78c3bc7-8bbb-4a67-8d7c-1f53cc847c76,7bbaa8668c87,
|
||||
q,1793910990399.0,1793910990401.6,1793910990417.5,202,,0afddad0-4f46-4d11-b48b-b2d3293750de,39e797bfa6f6,
|
||||
q,1793910990523.2,1793910990525.1,1793910990539.0,202,,a8b0fbc9-1bb9-4515-b8c6-1eb2fcae329b,7bbaa8668c87,
|
||||
q,1793910990647.5,1793910990649.5,1793910990657.9,202,,29bc1d0c-904e-4c2f-ad05-a01a86b6d500,39e797bfa6f6,
|
||||
q,1793910990771.7,1793910990772.6,1793910990781.2,202,,05456ec5-9f51-4334-8c33-ad6d3c104097,7bbaa8668c87,
|
||||
q,1793910990896.0,1793910990898.1,1793910990913.8,202,,64274d65-b97e-473c-9da6-4d71b7d404ca,39e797bfa6f6,
|
||||
q,1793910991020.3,1793910991021.4,1793910991032.4,202,,0bd4b8b4-40dc-4693-bcd8-d15493efec5d,7bbaa8668c87,
|
||||
q,1793910991144.5,1793910991147.5,1793910991179.1,202,,c8afbfc2-e1a2-47bb-af1f-8cf2035de3b0,39e797bfa6f6,
|
||||
q,1793910991268.8,1793910991269.5,1793910991281.4,202,,35136e85-3579-42b9-acdf-28183aa17c5c,7bbaa8668c87,
|
||||
q,1793910991393.0,1793910991394.4,1793910991408.4,202,,dabf14c2-d9c7-4bd4-add8-1ab1c05ac97c,39e797bfa6f6,
|
||||
q,1793910991517.3,1793910991519.6,1793910991534.8,202,,c918ef2b-1620-4111-9fe3-e49fdbf222e9,7bbaa8668c87,
|
||||
q,1793910991641.6,1793910991642.6,1793910991651.6,202,,1166db8e-e110-4370-8d60-fcc18fc9c990,39e797bfa6f6,
|
||||
q,1793910991765.8,1793910991768.3,1793910991780.5,202,,7379aa30-18b6-4651-b9c1-962c41acaebf,7bbaa8668c87,
|
||||
q,1793910991890.1,1793910991892.1,1793910991906.8,202,,9bd89740-9555-4284-9f8f-5a78eae22508,39e797bfa6f6,
|
||||
q,1793910992014.4,1793910992015.6,1793910992029.8,202,,724463ca-92b1-46dc-a4e0-0a23fda4bbda,7bbaa8668c87,
|
||||
q,1793910992138.6,1793910992141.0,1793910992150.3,202,,7c5af94e-2d90-4b21-89c4-0889cb2f1f88,39e797bfa6f6,
|
||||
q,1793910992262.9,1793910992265.5,1793910992281.5,202,,489c9729-5ae8-4b03-92e7-efb451ea37d8,7bbaa8668c87,
|
||||
q,1793910992387.1,1793910992388.5,1793910992403.5,202,,065a028d-c18a-4c36-b1d3-cbf515e67802,39e797bfa6f6,
|
||||
q,1793910992511.4,1793910992513.4,1793910992530.7,202,,9726f2dc-0e96-42e0-afd6-6072cbf13429,7bbaa8668c87,
|
||||
q,1793910992635.7,1793910992637.9,1793910992645.2,202,,c53a39bb-9b26-4713-b21c-d1d6bef4125b,39e797bfa6f6,
|
||||
q,1793910992759.9,1793910992762.2,1793910992772.2,202,,990fc5bf-0421-496a-88dc-b0e24beaae12,7bbaa8668c87,
|
||||
q,1793910992884.2,1793910992885.5,1793910992900.5,202,,f385223d-c1ac-4ac1-a5fb-bd1bc38db7bf,39e797bfa6f6,
|
||||
q,1793910993008.4,1793910993009.6,1793910993027.4,202,,3c25b737-3536-4604-a296-890a3f6652b0,7bbaa8668c87,
|
||||
q,1793910993132.7,1793910993134.4,1793910993143.9,202,,24203eea-e8b3-4b08-bf67-349f8c29caa7,39e797bfa6f6,
|
||||
q,1793910993257.0,1793910993257.3,1793910993266.1,202,,665df63f-2526-4e8e-8558-054c177faf68,7bbaa8668c87,
|
||||
q,1793910993381.2,1793910993382.4,1793910993395.4,202,,5546e3df-e665-4022-aadf-028fea84c481,39e797bfa6f6,
|
||||
q,1793910993505.5,1793910993508.6,1793910993522.5,202,,98ad8586-3bdd-475f-8c22-287527157ea7,7bbaa8668c87,
|
||||
q,1793910993629.8,1793910993631.4,1793910993639.5,202,,75a7e706-6576-443b-9794-405125c9f5f5,39e797bfa6f6,
|
||||
q,1793910993754.0,1793910993757.1,1793910993766.9,202,,a70985f6-188c-4422-bf77-fe5a4e38665d,7bbaa8668c87,
|
||||
q,1793910993878.3,1793910993879.4,1793910993890.3,202,,bf977cce-9e28-4eba-a2e8-d85076245e3e,39e797bfa6f6,
|
||||
q,1793910994002.5,1793910994003.9,1793910994019.4,202,,a30eba83-9990-420e-8c9b-b7bcc3e73d83,7bbaa8668c87,
|
||||
q,1793910994126.8,1793910994128.4,1793910994141.2,202,,577e0382-81ee-40d7-92ce-03fa8caff398,39e797bfa6f6,
|
||||
q,1793910994251.1,1793910994254.1,1793910994261.7,202,,8748e2f0-60c8-4160-87ca-aa2fd82044fd,7bbaa8668c87,
|
||||
q,1793910994375.3,1793910994376.3,1793910994385.9,202,,b723674f-c61a-4542-a924-16fd382752a4,39e797bfa6f6,
|
||||
q,1793910994499.6,1793910994502.8,1793910994519.8,202,,f493c392-038a-4197-a431-581632ea4483,7bbaa8668c87,
|
||||
q,1793910994623.8,1793910994625.4,1793910994635.8,202,,d48a5bdb-a342-49e7-be1c-ee5f35ad2aaf,39e797bfa6f6,
|
||||
q,1793910994748.1,1793910994749.7,1793910994760.7,202,,b710e573-db9f-4d71-84bb-9511b8587d10,7bbaa8668c87,
|
||||
q,1793910994872.4,1793910994876.9,1793910994892.6,202,,b906fab8-2ba8-4704-9e80-eea0b1614c68,39e797bfa6f6,
|
||||
q,1793910994996.6,1793910994998.4,1793910995013.6,202,,0b4c0954-e2ae-4eba-8895-42ede37fc561,7bbaa8668c87,
|
||||
q,1793910995120.9,1793910995121.9,1793910995135.4,202,,efc530a4-22eb-4eb7-9f65-b6a506ed263a,39e797bfa6f6,
|
||||
q,1793910995245.2,1793910995247.3,1793910995256.1,202,,6b8e6a4b-5d2a-4b62-8546-16e766d2847c,7bbaa8668c87,
|
||||
q,1793910995369.4,1793910995373.8,1793910995388.9,202,,7d19100a-271c-432e-a830-46e0b1f731b0,39e797bfa6f6,
|
||||
q,1793910995493.7,1793910995494.9,1793910995510.8,202,,40babcc2-6a5f-4732-beba-d67200cea9f0,7bbaa8668c87,
|
||||
q,1793910995617.9,1793910995619.6,1793910995633.7,202,,095546b9-29f5-492d-b4ce-0a922ec8b134,39e797bfa6f6,
|
||||
q,1793910995742.2,1793910995743.5,1793910995752.0,202,,df3c804d-7e4f-4ab6-aec3-f63fedff0340,7bbaa8668c87,
|
||||
q,1793910995866.5,1793910995868.2,1793910995875.9,202,,dfddde97-5c0a-4ee0-93e7-137d88428a16,39e797bfa6f6,
|
||||
q,1793910995990.7,1793910995991.4,1793910996006.1,202,,8364b631-ea06-413b-8c4f-b90bf632ba68,7bbaa8668c87,
|
||||
q,1793910996115.0,1793910996116.6,1793910996130.5,202,,b10b4aea-8889-4bdf-a36d-ab98cb1ebff9,39e797bfa6f6,
|
||||
q,1793910996239.2,1793910996243.4,1793910996284.8,202,,a0c17049-8ef0-433f-be57-be1f68b4af02,7bbaa8668c87,
|
||||
q,1793910996363.5,1793910996365.0,1793910996375.4,202,,1c4919b3-c01f-46cf-b24e-e7c226659618,39e797bfa6f6,
|
||||
q,1793910996487.8,1793910996489.4,1793910996503.3,202,,c8ac4553-8815-4081-87ab-a06d530789e5,7bbaa8668c87,
|
||||
q,1793910996612.0,1793910996613.6,1793910996627.8,202,,204d9d4f-483d-4a0b-a072-f18ff5f4f0b0,39e797bfa6f6,
|
||||
q,1793910996736.3,1793910996737.6,1793910996745.7,202,,e8ffb02f-6d86-4dbb-b15d-90b765ba4683,7bbaa8668c87,
|
||||
q,1793910996860.6,1793910996861.5,1793910996873.9,202,,6f91edaf-de58-4725-baa9-41070a2e6ce1,39e797bfa6f6,
|
||||
q,1793910996984.8,1793910996986.1,1793910996995.2,202,,877b0463-996a-4d57-b059-02468bb50428,7bbaa8668c87,
|
||||
q,1793910997109.1,1793910997110.8,1793910997125.9,202,,16d4401d-fa3a-488d-8fa5-6e1c21e115cf,39e797bfa6f6,
|
||||
q,1793910997233.3,1793910997235.3,1793910997244.0,202,,f686efa1-9545-4dc3-88f1-21ea281498ba,7bbaa8668c87,
|
||||
q,1793910997357.6,1793910997358.9,1793910997374.0,202,,e3a9a72c-304e-408c-84c9-f7a591b15f31,39e797bfa6f6,
|
||||
q,1793910997481.9,1793910997482.4,1793910997498.0,202,,321dbaf9-bf02-47ce-96ae-740254960375,7bbaa8668c87,
|
||||
q,1793910997606.1,1793910997607.4,1793910997660.0,202,,23f19b42-c5bf-4b16-a488-9c74463daa83,39e797bfa6f6,
|
||||
q,1793910997730.4,1793910997732.6,1793910997740.2,202,,ee3d5db9-ad9b-4831-84c6-53a3ebdcc609,7bbaa8668c87,
|
||||
q,1793910997854.6,1793910997856.9,1793910997868.4,202,,372a8b51-ba91-467e-8c2b-2c7477710d97,39e797bfa6f6,
|
||||
q,1793910997978.9,1793910997980.6,1793910997993.3,202,,6f967663-a672-4181-9d0a-e7ca8e0820c6,7bbaa8668c87,
|
||||
q,1793910998103.2,1793910998104.5,1793910998119.9,202,,a39b449f-2190-45cf-9933-2a749c0bf8e9,39e797bfa6f6,
|
||||
q,1793910998227.4,1793910998228.3,1793910998236.0,202,,57ce821a-e3c5-4867-95d4-60a811b0b6b2,7bbaa8668c87,
|
||||
q,1793910998351.7,1793910998352.8,1793910998362.3,202,,bdc8a4a5-785e-4047-99e3-3dd11a673813,39e797bfa6f6,
|
||||
q,1793910998476.0,1793910998477.4,1793910998492.8,202,,70f1c0e4-bc7f-495b-aba7-50046a977466,7bbaa8668c87,
|
||||
q,1793910998600.2,1793910998602.6,1793910998619.1,202,,5f486aa4-45c8-44b9-b4b1-1e60240d79fb,39e797bfa6f6,
|
||||
q,1793910998724.5,1793910998726.4,1793910998735.5,202,,36f6e6aa-84f2-4b71-976d-aac167ea2b89,7bbaa8668c87,
|
||||
q,1793910998848.7,1793910998855.4,1793910998866.1,202,,8761314d-cd9d-4049-a1cb-d378a9391739,39e797bfa6f6,
|
||||
q,1793910998973.0,1793910998974.6,1793910998985.5,202,,89db4e9a-3350-4dd9-a04c-3bed34b5963a,7bbaa8668c87,
|
||||
q,1793910999097.3,1793910999098.4,1793910999112.9,202,,fc59dd65-f9ca-4886-96ca-a279870b9c8d,39e797bfa6f6,
|
||||
q,1793910999221.5,1793910999222.9,1793910999235.8,202,,b2503cf9-c186-4515-8fbe-bfe10480bf49,7bbaa8668c87,
|
||||
q,1793910999345.8,1793910999346.6,1793910999355.0,202,,03616bd8-b15c-4379-aa74-c518dc05c25c,39e797bfa6f6,
|
||||
q,1793910999470.0,1793910999476.4,1793910999490.9,202,,0344b7bd-b348-4d4b-91f7-ac32211999ba,7bbaa8668c87,
|
||||
q,1793910999594.3,1793910999596.4,1793910999609.8,202,,041833ca-6d80-41de-ac3d-05e470f29127,39e797bfa6f6,
|
||||
q,1793910999718.6,1793910999719.5,1793910999737.0,202,,067c1012-d6a3-448d-a9ff-bfa3628d657e,7bbaa8668c87,
|
||||
q,1793910999842.8,1793910999844.7,1793910999853.0,202,,fa574acb-53ce-428d-b03f-9844feee5451,39e797bfa6f6,
|
||||
q,1793910999967.1,1793910999972.8,1793910999987.7,202,,f933e133-f02f-4c47-aaae-b96f2be79413,7bbaa8668c87,
|
||||
q,1793911000091.4,1793911000094.3,1793911000109.0,202,,9f84f22e-ae30-48f8-9a07-19ab21771961,39e797bfa6f6,
|
||||
q,1793911000215.6,1793911000216.5,1793911000233.2,202,,08ccc498-68c7-4281-9e40-367e8021f37b,7bbaa8668c87,
|
||||
q,1793911000339.9,1793911000341.6,1793911000350.4,202,,0fcae547-fdb8-4bf9-a550-ab3353accc20,39e797bfa6f6,
|
||||
q,1793911000464.1,1793911000470.6,1793911000488.6,202,,beb31c4e-dfa7-4cd6-945d-31d76290619f,7bbaa8668c87,
|
||||
q,1793911000588.4,1793911000589.8,1793911000606.4,202,,d2620ab4-d998-4af1-b045-78c3b382865b,39e797bfa6f6,
|
||||
q,1793911000712.7,1793911000714.4,1793911000731.4,202,,a39b4418-dea0-40de-921e-9c2c6565cf5e,7bbaa8668c87,
|
||||
q,1793911000836.9,1793911000839.5,1793911000847.2,202,,0518d105-8d5c-42e4-a46d-8b7e9b601f7f,39e797bfa6f6,
|
||||
q,1793911000961.2,1793911000964.1,1793911000971.7,202,,10da257f-c6b3-44c9-b945-3e78b727542a,7bbaa8668c87,
|
||||
q,1793911001085.4,1793911001088.9,1793911001102.3,202,,0aa1ff34-1063-453d-9752-9718625b75a4,39e797bfa6f6,
|
||||
q,1793911001209.7,1793911001211.5,1793911001227.7,202,,833dc025-66ca-438f-9a8f-393a3aa6c8e8,7bbaa8668c87,
|
||||
q,1793911001334.0,1793911001336.5,1793911001345.4,202,,621b60f6-7adc-4bcd-86da-7bf792284451,39e797bfa6f6,
|
||||
q,1793911001458.2,1793911001464.5,1793911001483.6,202,,eca07243-837f-4897-b75e-214034279c45,7bbaa8668c87,
|
||||
q,1793911001582.5,1793911001584.9,1793911001598.4,202,,4a3218af-79fb-4806-809b-2246da955f5a,39e797bfa6f6,
|
||||
q,1793911001706.8,1793911001709.1,1793911001725.7,202,,d48352b5-4964-4d2b-93b3-5b826264c589,7bbaa8668c87,
|
||||
q,1793911001831.0,1793911001832.4,1793911001840.0,202,,89ac90b7-6fd7-4d34-a772-4649441eda17,39e797bfa6f6,
|
||||
q,1793911001955.3,1793911001956.9,1793911001969.4,202,,dcdeb9dc-f099-45ba-b64c-86f3e27c9f7a,7bbaa8668c87,
|
||||
q,1793911002079.5,1793911002081.4,1793911002094.0,202,,45ee9b13-963c-4f93-a13e-5d28c6fb7b61,39e797bfa6f6,
|
||||
q,1793911002203.8,1793911002206.6,1793911002222.0,202,,fae652e1-17c8-46e1-95a7-fd60fa5c4107,7bbaa8668c87,
|
||||
q,1793911002328.1,1793911002328.9,1793911002337.0,202,,d1c7cba9-2aa1-4e85-b75a-8ecbc359759d,39e797bfa6f6,
|
||||
q,1793911002452.3,1793911002453.4,1793911002467.9,202,,acd9e26d-1c80-48ce-a459-41ac3f11d8f0,7bbaa8668c87,
|
||||
q,1793911002576.6,1793911002577.4,1793911002589.8,202,,6e94ba2c-c13d-4079-8c42-b7114dc46927,39e797bfa6f6,
|
||||
q,1793911002700.8,1793911002701.6,1793911002716.5,202,,1ae268a3-79ba-46c9-975d-d7f944012977,7bbaa8668c87,
|
||||
q,1793911002825.1,1793911002826.3,1793911002833.6,202,,ca4efeff-620d-422b-822b-1db262fe2189,39e797bfa6f6,
|
||||
q,1793911002949.4,1793911002955.4,1793911002967.6,202,,e3c0d7e9-88da-453d-81d1-f6cf0a117f1a,7bbaa8668c87,
|
||||
q,1793911003073.6,1793911003075.4,1793911003088.0,202,,fcafb6cb-7864-46cc-97f8-8ce71c54b327,39e797bfa6f6,
|
||||
q,1793911003197.9,1793911003198.5,1793911003234.3,202,,56bcdc0b-5d2c-461b-b415-5d223c13fa0a,7bbaa8668c87,
|
||||
q,1793911003322.2,1793911003323.3,1793911003334.5,202,,9882b701-c2b8-48ad-8607-890cb6c9e144,39e797bfa6f6,
|
||||
q,1793911003446.4,1793911003448.1,1793911003459.9,202,,81353699-c3ad-425e-84bd-8457a1e4a5de,7bbaa8668c87,
|
||||
q,1793911003570.7,1793911003571.6,1793911003583.6,202,,08449dfe-fa53-4ad4-bf27-e85eb83b31e4,39e797bfa6f6,
|
||||
q,1793911003694.9,1793911003700.0,1793911003716.1,202,,547c2f6f-3f9d-4839-8dae-5de4c3496183,7bbaa8668c87,
|
||||
q,1793911003819.2,1793911003821.4,1793911003837.4,202,,0c45f57b-4d76-4402-adb3-55287b036eb3,39e797bfa6f6,
|
||||
q,1793911003943.5,1793911003943.7,1793911003951.7,202,,b4451638-436f-4154-be6f-5152b787c7d8,7bbaa8668c87,
|
||||
q,1793911004067.7,1793911004068.3,1793911004077.0,202,,4583d7e7-b8db-47f7-a794-86eb798646a5,39e797bfa6f6,
|
||||
q,1793911004192.0,1793911004192.4,1793911004207.1,202,,4a96f8d7-1338-44a3-ad1e-ac0e4f154348,7bbaa8668c87,
|
||||
q,1793911004316.2,1793911004318.6,1793911004328.8,202,,ecea97b6-1435-4ee4-bf7a-7a13f949a2d3,39e797bfa6f6,
|
||||
q,1793911004440.5,1793911004441.3,1793911004449.5,202,,bab1ae68-8ccb-4df5-a72b-2e71be427932,7bbaa8668c87,
|
||||
q,1793911004564.8,1793911004567.0,1793911004576.0,202,,e61b8415-5516-4d1c-b52b-4c7663209d7b,39e797bfa6f6,
|
||||
q,1793911004689.0,1793911004690.5,1793911004706.7,202,,e9c87b1a-1459-49f0-824a-ba91599a3ec1,7bbaa8668c87,
|
||||
q,1793911004813.3,1793911004814.6,1793911004825.0,202,,b7c6ed65-e8c5-4e54-a289-13160345aa1a,39e797bfa6f6,
|
||||
q,1793911004937.6,1793911004940.3,1793911004951.8,202,,9676970a-2d20-46d2-bbf2-092571cb325e,7bbaa8668c87,
|
||||
q,1793911005061.8,1793911005064.5,1793911005090.0,202,,23ae799a-e5ff-4a7e-b588-74ef0fbb46bc,39e797bfa6f6,
|
||||
q,1793911005186.1,1793911005187.9,1793911005204.0,202,,5d659f50-3080-4b2d-9f45-d139bc669a65,7bbaa8668c87,
|
||||
q,1793911005310.3,1793911005312.2,1793911005328.4,202,,ab352cd5-0701-41e1-b2d9-43f6531992c8,39e797bfa6f6,
|
||||
q,1793911005434.6,1793911005435.9,1793911005444.8,202,,2fefff88-53da-4380-9b4e-34573223ad03,7bbaa8668c87,
|
||||
q,1793911005558.9,1793911005565.5,1793911005584.7,202,,b01e544e-f514-4ee4-a992-1cb8bdc69250,39e797bfa6f6,
|
||||
q,1793911005683.1,1793911005686.7,1793911005703.6,202,,2ce25f72-22ce-4f33-b218-80e6df98e5ea,7bbaa8668c87,
|
||||
q,1793911005807.4,1793911005809.6,1793911005824.7,202,,4197f3e0-61bd-44fd-9df8-70607a670424,39e797bfa6f6,
|
||||
q,1793911005931.6,1793911005932.8,1793911005940.6,202,,d6299892-83d1-48f1-b3ed-3c5f19445880,7bbaa8668c87,
|
||||
q,1793911006055.9,1793911006058.6,1793911006065.9,202,,25a79c09-3e7d-4d04-8464-74a3eead2ef1,39e797bfa6f6,
|
||||
q,1793911006180.2,1793911006182.9,1793911006194.5,202,,28017b8a-9202-4605-a62e-5720cc57deda,7bbaa8668c87,
|
||||
q,1793911006304.4,1793911006306.4,1793911006322.3,202,,d69d1b2d-412f-4dfa-9e17-d927dbc365ae,39e797bfa6f6,
|
||||
q,1793911006428.7,1793911006433.7,1793911006445.6,202,,ac17fcc2-711c-4c88-adaa-b18fc41733ce,7bbaa8668c87,
|
||||
q,1793911006553.0,1793911006555.4,1793911006571.2,202,,76685979-e0f9-4cf4-99bb-bc404727c0c7,39e797bfa6f6,
|
||||
q,1793911006677.2,1793911006679.7,1793911006687.7,202,,7db05970-6c65-41c7-81f1-92bb1e1a509a,7bbaa8668c87,
|
||||
q,1793911006801.5,1793911006802.7,1793911006818.9,202,,b844dc42-f46b-4e5f-9be3-19342680c392,39e797bfa6f6,
|
||||
q,1793911006925.7,1793911006927.7,1793911006935.7,202,,3c0deab8-2e61-4a90-891e-e3601d4f566e,7bbaa8668c87,
|
||||
q,1793911007050.0,1793911007053.4,1793911007068.7,202,,18a8b6db-ba10-4c30-8ef0-cca901758d13,39e797bfa6f6,
|
||||
q,1793911007174.3,1793911007175.9,1793911007191.5,202,,cfeff061-7d3a-4243-90e0-57a02bf2489a,7bbaa8668c87,
|
||||
q,1793911007298.5,1793911007299.8,1793911007314.4,202,,d99487db-68b0-48ee-97cb-6565630324d3,39e797bfa6f6,
|
||||
q,1793911007422.8,1793911007424.1,1793911007431.3,202,,ad0689c0-85a9-4043-addd-4b226a27f6da,7bbaa8668c87,
|
||||
q,1793911007547.0,1793911007549.2,1793911007562.7,202,,1955e80e-21c8-4bb0-be71-00bc17b17295,39e797bfa6f6,
|
||||
q,1793911007671.3,1793911007672.4,1793911007687.7,202,,24750373-3456-4f8f-a423-fdcdf4e17207,7bbaa8668c87,
|
||||
q,1793911007795.6,1793911007796.9,1793911007809.9,202,,030dd035-b91a-477f-85ef-3788ed7d4a5a,39e797bfa6f6,
|
||||
q,1793911007919.8,1793911007920.5,1793911007928.5,202,,a997a401-507b-48ff-b23b-297cc8010ecf,7bbaa8668c87,
|
||||
q,1793911008044.1,1793911008045.4,1793911008058.7,202,,80357c20-5806-4fae-b5cc-a52bbefbcded,39e797bfa6f6,
|
||||
q,1793911008168.4,1793911008169.4,1793911008183.3,202,,184e5980-50ae-435d-b049-aec74edc700f,7bbaa8668c87,
|
||||
q,1793911008292.6,1793911008294.4,1793911008309.2,202,,6afde3ef-649f-43d8-b086-1e7abd4967c2,39e797bfa6f6,
|
||||
q,1793911008416.9,1793911008418.3,1793911008425.7,202,,2ea89737-b580-494b-aabc-a960dbaf8c68,7bbaa8668c87,
|
||||
q,1793911008541.1,1793911008542.8,1793911008554.2,202,,5f418304-5e05-49ea-92bb-4068b393c391,39e797bfa6f6,
|
||||
q,1793911008665.4,1793911008667.6,1793911008678.6,202,,f9824031-8f9c-4b57-ab34-582e07aa7771,7bbaa8668c87,
|
||||
q,1793911008789.7,1793911008792.1,1793911008806.8,202,,86aeaa84-ec3c-4e54-aabc-3acaae572d18,39e797bfa6f6,
|
||||
q,1793911008913.9,1793911008915.4,1793911008927.2,202,,3cdd4f75-d099-4315-b3c7-9b7a57cc0fb2,7bbaa8668c87,
|
||||
q,1793911009038.2,1793911009040.8,1793911009051.7,202,,a2ec6209-dda6-4ad7-9246-ab80f0f880c8,39e797bfa6f6,
|
||||
q,1793911009162.5,1793911009163.4,1793911009175.0,202,,43c58be1-8492-4069-8fbe-dbec0d3126be,7bbaa8668c87,
|
||||
q,1793911009286.7,1793911009287.4,1793911009302.9,202,,836723b9-cc7c-4e19-984b-c628cf442aa9,39e797bfa6f6,
|
||||
q,1793911009411.0,1793911009411.7,1793911009427.4,202,,b731041c-5b77-491d-a1a4-4de7cd5fe235,7bbaa8668c87,
|
||||
q,1793911009535.2,1793911009536.4,1793911009545.1,202,,0ee02718-6052-433a-beca-865d370d8f55,39e797bfa6f6,
|
||||
q,1793911009659.5,1793911009662.2,1793911009670.7,202,,9e563ef0-c007-4c5c-88ea-4a0874638bc0,7bbaa8668c87,
|
||||
q,1793911009783.8,1793911009784.6,1793911009796.6,202,,dd7bb93f-551b-4f61-ab67-d71238d94707,39e797bfa6f6,
|
||||
q,1793911009908.0,1793911009909.3,1793911009920.8,202,,58054e69-f91a-4c1f-bebc-97cd759cccc6,7bbaa8668c87,
|
||||
q,1793911010032.3,1793911010038.4,1793911010049.2,202,,88f3df7d-1c5e-40d3-971d-038b70c7811e,39e797bfa6f6,
|
||||
q,1793911010156.5,1793911010158.4,1793911010175.3,202,,0563d5aa-ddd9-467d-a956-3a5355d2c641,7bbaa8668c87,
|
||||
q,1793911010280.8,1793911010281.6,1793911010291.6,202,,f0234d74-5ce3-49fc-acc8-2146b2f47f9e,39e797bfa6f6,
|
||||
q,1793911010405.1,1793911010407.5,1793911010422.2,202,,7a89d260-1e80-49ff-9c71-70f06a04c114,7bbaa8668c87,
|
||||
q,1793911010529.3,1793911010532.6,1793911010541.1,202,,e081c2f7-853c-47d3-8c44-84e5cc51e12a,39e797bfa6f6,
|
||||
q,1793911010653.6,1793911010659.6,1793911010681.0,202,,7a70dda6-1b47-4fc6-84ad-38d731b39469,7bbaa8668c87,
|
||||
q,1793911010777.9,1793911010779.6,1793911010787.7,202,,3e419585-d6c1-4e2b-97e5-92823dc2b1ed,39e797bfa6f6,
|
||||
q,1793911010902.1,1793911010903.7,1793911010919.1,202,,aa6f4f00-8b3a-4ac4-ba4f-b3b033a4f53f,7bbaa8668c87,
|
||||
q,1793911011026.4,1793911011030.1,1793911011038.7,202,,8a4b219a-a1a9-4334-afd7-f3b8b4ecef4e,39e797bfa6f6,
|
||||
q,1793911011150.6,1793911011156.4,1793911011172.4,202,,f88210b9-1c0d-481a-8fd7-bab548997d7e,7bbaa8668c87,
|
||||
q,1793911011274.9,1793911011279.6,1793911011288.3,202,,57f25bce-1be5-4582-a39f-2e9826cd5cc2,39e797bfa6f6,
|
||||
q,1793911011399.2,1793911011400.5,1793911011414.4,202,,d35679d4-62d0-40c9-907d-da4eaa0f3855,7bbaa8668c87,
|
||||
q,1793911011523.4,1793911011525.8,1793911011533.5,202,,bed33a8c-cc01-445f-b4d5-784b64965aad,39e797bfa6f6,
|
||||
q,1793911011647.7,1793911011649.5,1793911011656.9,202,,15d161f1-f322-4290-8da1-66cfef7b3dc6,7bbaa8668c87,
|
||||
q,1793911011771.9,1793911011772.6,1793911011780.4,202,,eedb361f-8663-4ca8-8163-c7aa69c32b22,39e797bfa6f6,
|
||||
q,1793911011896.2,1793911011897.4,1793911011913.3,202,,f16e0a6c-4cdf-4ee4-965b-ffed860f4358,7bbaa8668c87,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 8.04754,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793910972.008,
|
||||
"end": 1793911012.02,
|
||||
"finished": 1793911011.913,
|
||||
"scheduled": 322,
|
||||
"sent": 322,
|
||||
"accepted": 322,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 322
|
||||
},
|
||||
"upload_p50_ms": 12.43,
|
||||
"upload_p95_ms": 18.01,
|
||||
"upload_max_ms": 52.63,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.341,
|
||||
"log": "/tmp/c4p4-s-r1-steady.log"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,20 @@
|
||||
q,1793911445821.2,1793911445828.4,1793911445959.5,503,,,7bbaa8668c87,
|
||||
q,1793911445921.2,1793911445922.9,1793911446019.0,503,,,39e797bfa6f6,
|
||||
q,1793911446021.2,1793911446023.2,1793911446117.6,503,,,7bbaa8668c87,
|
||||
q,1793911446121.2,1793911446122.3,1793911446219.7,503,,,39e797bfa6f6,
|
||||
q,1793911446221.2,1793911446221.8,1793911446316.8,503,,,7bbaa8668c87,
|
||||
q,1793911446321.2,1793911446324.3,1793911446419.1,503,,,39e797bfa6f6,
|
||||
q,1793911446421.2,1793911446422.9,1793911446515.8,503,,,7bbaa8668c87,
|
||||
q,1793911446521.2,1793911446523.6,1793911446617.0,503,,,39e797bfa6f6,
|
||||
q,1793911446621.2,1793911446624.6,1793911446719.7,503,,,7bbaa8668c87,
|
||||
q,1793911446721.2,1793911446723.2,1793911446817.6,503,,,39e797bfa6f6,
|
||||
q,1793911446821.2,1793911446824.0,1793911446919.4,503,,,7bbaa8668c87,
|
||||
q,1793911446921.2,1793911446923.1,1793911447016.5,503,,,39e797bfa6f6,
|
||||
q,1793911447021.2,1793911447024.3,1793911447118.5,503,,,7bbaa8668c87,
|
||||
q,1793911447121.2,1793911447123.5,1793911447226.2,503,,,39e797bfa6f6,
|
||||
q,1793911447221.2,1793911447222.9,1793911447322.7,503,,,7bbaa8668c87,
|
||||
q,1793911447321.2,1793911447322.0,1793911447417.8,503,,,39e797bfa6f6,
|
||||
q,1793911447421.2,1793911447423.1,1793911447519.2,503,,,7bbaa8668c87,
|
||||
q,1793911447521.2,1793911447522.2,1793911447616.6,503,,,39e797bfa6f6,
|
||||
q,1793911447621.2,1793911447624.4,1793911447719.1,503,,,7bbaa8668c87,
|
||||
q,1793911447721.2,1793911447723.2,1793911447817.6,503,,,39e797bfa6f6,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 10.0,
|
||||
"duration_s": 3.0,
|
||||
"start": 1793911445.821,
|
||||
"end": 1793911447.821,
|
||||
"finished": 1793911447.818,
|
||||
"scheduled": 20,
|
||||
"sent": 20,
|
||||
"accepted": 0,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 20,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"503": 20
|
||||
},
|
||||
"upload_p50_ms": null,
|
||||
"upload_p95_ms": null,
|
||||
"upload_max_ms": null,
|
||||
"peak_inflight": 2,
|
||||
"gen_cpu_s": 0.016,
|
||||
"log": "/tmp/c4p4-x-r1-drain.log"
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
# Очередь под перегрузкой: метка `baseline`
|
||||
|
||||
- Статус: **прерван: StandError: c4p4-x-r1: no ready job after the drain:
|
||||
2026-11-05T20:44:04.880Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=4 prefetch=4 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-
|
||||
2026-11-05T20:44:04.887Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=4
|
||||
2026-11-05T20:44:05.280Z INFO src.worker [rid=-] worker stopped exit_code=0 cancelled=0 returned=0
|
||||
2026-11-05T20:44:08.660Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=4 prefetch=4 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-
|
||||
2026-11-05T20:44:08.666Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=4
|
||||
**
|
||||
- Код (`src_sha256` образа): `4c4bb305b2747d154105720b93261ef476675670470bfa7a10b338dd79514da9`
|
||||
- Профиль `queue`: настройки c1p1 (concurrency 1, prefetch 1), c2p2 (concurrency 2, prefetch 2), c4p4 (concurrency 4, prefetch 4), c2p32 (concurrency 2, prefetch 32); повторов 2; окно 40 с, ровная 0.6 ёмкости ячейки, общая интенсивность класса (fixed) s 2.5/с, l 0.8/с, x 0.5/с, перегрузка 2 ёмкости в окне min(150 с, max(окно, 1.5 × предел / ёмкость)), после перегрузки — ровная 20 с; предел незавершённых `UPLOAD_MAX_UNFINISHED=60`
|
||||
- Тела: s — noise 320 × 200, 192323 байт, SHA-256 `dd245db29b537d42…`, l — noise 1800 × 1800, 9724823 байт, SHA-256 `cd5dd6eae6c4a6c8…`, x — flat 6000 × 6000, 119755 байт, SHA-256 `0652f72ddb1d2401…`
|
||||
- Стенд: LB, 2 реплики по 1 CPU / 512m, worker 2 CPU / 1536m (`RENDER_MEMORY_MB=768`), брокер 1 CPU / 512m, S3 1 CPU / 512m
|
||||
- Пропускная — ready/с фазы ёмкости: N тел ждут в брокере, отсчёт — от строки `broker consuming` worker до последнего ready. Время до ready — `attachment_jobs.updated_at − attachments.created_at` (часы PostgreSQL); queue lag — `waited` журнала worker (приём → захват). Пик worker — `memory.peak` cgroup контейнера за ячейку (контейнер новый на ячейку), пик рендера — наибольший `render_peak_kib`. Незавершённых — точный счёт по созданию и завершению заданий. Восстановление — от конца окна перегрузки до последнего ready. Значения — медиана (min–max) повторов
|
||||
|
||||
## Класс `s`
|
||||
|
||||
| Настройка | Пропускная, гот./с | CPU worker, ядер | p95 до ready (0,6 ёмкости), с | queue lag p95, с | p95 fixed, с | Окно перегрузки, с | 429 за окно, % | 429 после предела, % | Retry-After 5 / 1 | Макс. незавершённых | Восстановление, с | p95 после, с | Пик worker, MiB | Пик рендера, MiB | failed / missing | Пометки |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| c4p4 | 13.413 | 1.87 | 0.76 | 0.51 | 0.65 | 40 | 45 | 51 | 488 / 0 | 60 | 4.2 | 0.76 | 146 | 98 (97–98) | 0 / 0 | — |
|
||||
| c2p32 | 13.577 | 1.77 | 0.66 | 0.50 | 0.68 | 40 | 45 | 50 | 491 / 0 | 60 | 4.3 | 0.65 | 116 | 98 (97–98) | 0 / 0 | — |
|
||||
|
||||
## Класс `l`
|
||||
|
||||
| Настройка | Пропускная, гот./с | CPU worker, ядер | p95 до ready (0,6 ёмкости), с | queue lag p95, с | p95 fixed, с | Окно перегрузки, с | 429 за окно, % | 429 после предела, % | Retry-After 5 / 1 | Макс. незавершённых | Восстановление, с | p95 после, с | Пик worker, MiB | Пик рендера, MiB | failed / missing | Пометки |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| c4p4 | 4.025 | 1.58 | 1.67 | 0.83 | 1.02 | 40 | 0 | — | 0 / 0 | 14 | 0.0 | — | 407 | 178 (176–178) | 14 / 0 | 5xx failed |
|
||||
|
||||
## Класс `x`
|
||||
|
||||
| Настройка | Пропускная, гот./с | CPU worker, ядер | p95 до ready (0,6 ёмкости), с | queue lag p95, с | p95 fixed, с | Окно перегрузки, с | 429 за окно, % | 429 после предела, % | Retry-After 5 / 1 | Макс. незавершённых | Восстановление, с | p95 после, с | Пик worker, MiB | Пик рендера, MiB | failed / missing | Пометки |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| c2p2 | 2.968 | 1.93 | 1.15 | 0.48 | 1.14 | 40 | 27 | 52 | 65 / 0 | 60 | 20.8 | 1.16 | 669 | 300 (299–300) | 0 / 0 | — |
|
||||
|
||||
## Гипотезы
|
||||
|
||||
Тексты и пороги — `REPORT.md`, «Очередь: гипотезы» (записаны до замеров); граница незавершённых — предел 60 + места загрузок стенда 8.
|
||||
|
||||
| Гипотеза | Итог | Числа |
|
||||
|---|---|---|
|
||||
| Q1 | нет данных | l: нет ячеек c1p1, c2p2 и c4p4; x: нет ячеек c1p1, c2p2 и c4p4 |
|
||||
| Q2 | нет данных | s: нет ячеек c2p2 и c2p32; l: нет ячеек c2p2 и c2p32; x: нет ячеек c2p2 и c2p32 |
|
||||
| Q3 | нет данных | x: нет пиков c1p1 и c4p4; s: нет пиков c1p1 и c4p4 |
|
||||
| Q4 | опровергнута | макс. незавершённых 60 (граница 60 + 8 = 68); 429 после предела: min 50 % (порог 30%) в 3 ячейках; за всё окно 0%–45%; 5xx 308; предел не достигнут: c4p4-l-r1 |
|
||||
| Q5 | нет данных | 3 ячеек; худшее опустошение — 0.67 границы, худший p95 после — 0.58 границы; failed + missing 0; c4p4-l-r1: нет чисел |
|
||||
|
||||
- рекомендация (наименьший p95 фазы fixed при пике worker ≤ 80% лимита, 1229 MiB; p95 в пределах 10% — меньший пик): s — c2p32 (p95 fixed 0.68 с, пропускная 13.577 гот./с, пик 116 MiB; p95 fixed: c4p4 0.65 с, c2p32 0.68 с); l — c4p4 (p95 fixed 1.02 с, пропускная 4.025 гот./с, пик 407 MiB; p95 fixed: c4p4 1.02 с); x — c2p2 (p95 fixed 1.14 с, пропускная 2.968 гот./с, пик 669 MiB; p95 fixed: c2p2 1.14 с)
|
||||
@@ -0,0 +1,656 @@
|
||||
cell,config,size_class,repeat,phase,ts,t_s,unfinished,outbox,in_broker,processing,oldest_age_s,worker_cpu_cores,worker_mem_mib,accepted,rejected_429,errors
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910786.367,0.0,0,0,0,0,0.0,,80.7,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910787.384,1.02,4,2,2,0,0.31,,80.7,4,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910788.382,2.02,14,2,12,0,1.31,,80.7,10,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910789.394,3.03,20,0,20,0,2.32,,,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910790.383,4.02,20,0,20,0,3.31,,22.8,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910791.389,5.02,18,0,16,2,4.17,1.666,99.1,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910792.38,6.01,16,0,14,2,4.96,1.938,398.4,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910793.404,7.04,12,0,10,2,5.58,1.976,459.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910794.384,8.02,10,0,8,2,6.36,1.96,397.6,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910795.387,9.02,6,0,4,2,6.96,1.985,272.5,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910796.376,10.01,4,0,2,2,7.75,1.98,530.5,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,drain,1793910797.386,11.02,0,0,0,0,0.0,1.976,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910798.393,12.03,1,0,0,1,0.22,0.005,83.6,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910799.397,13.03,2,1,0,1,0.69,1.03,375.7,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910800.374,14.01,2,0,0,2,1.11,0.638,373.8,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910801.376,15.01,2,0,0,2,0.98,1.29,373.4,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910802.392,16.03,2,0,0,2,0.88,1.293,373.0,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910803.39,17.02,2,0,0,2,0.75,1.294,241.7,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910804.387,18.02,2,0,0,2,0.63,1.292,241.1,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910805.412,19.05,1,0,0,1,0.53,0.991,84.1,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910806.38,20.01,2,0,0,2,0.91,0.952,240.5,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910807.38,21.01,2,0,0,2,0.81,1.288,238.8,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910808.383,22.02,2,1,0,1,0.69,1.305,238.9,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910809.389,23.02,2,1,0,1,0.58,1.294,238.6,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910810.411,24.04,1,0,1,0,0.47,1.056,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910811.385,25.02,2,1,0,1,0.87,0.869,237.9,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910812.383,26.02,2,1,0,1,0.76,1.291,362.1,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910813.374,27.01,2,1,0,1,0.63,1.296,240.6,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910814.381,28.01,1,0,0,1,0.51,1.292,220.5,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910815.426,29.06,1,1,0,0,0.44,1.118,84.0,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910816.387,30.02,2,1,0,1,0.82,0.802,180.6,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910817.382,31.02,2,1,0,1,0.71,1.27,342.7,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910818.382,32.02,2,1,0,1,0.59,1.274,324.9,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910819.376,33.01,1,0,0,1,0.46,1.278,286.5,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910820.419,34.05,1,1,0,0,0.37,1.198,272.0,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910821.388,35.02,2,1,0,1,0.77,0.717,285.1,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910822.388,36.02,2,1,0,1,0.66,1.266,276.5,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910823.381,37.01,1,0,0,1,0.53,1.279,274.3,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910824.38,38.01,1,0,0,1,0.4,1.258,269.0,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910825.384,39.02,1,0,0,1,0.29,1.248,259.5,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910826.385,40.02,2,1,0,1,0.73,0.783,84.1,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910827.391,41.02,2,1,0,1,0.61,1.149,246.3,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910828.382,42.02,1,0,0,1,0.47,1.306,244.3,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910829.362,43.0,1,0,0,1,0.33,1.281,239.5,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910830.38,44.01,1,0,0,1,0.23,1.269,375.7,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910831.377,45.01,2,1,0,1,0.66,1.292,376.2,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910832.396,46.03,2,0,0,2,1.12,0.634,374.4,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910833.381,47.01,2,0,0,2,0.98,1.312,376.4,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910834.391,48.02,2,0,0,2,0.87,1.279,373.9,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910835.383,49.02,2,0,0,2,0.74,1.285,241.6,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910836.383,50.02,2,1,0,1,0.61,1.295,241.6,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910837.386,51.02,2,0,0,2,1.06,0.639,240.5,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910838.377,52.01,2,0,0,2,0.92,1.293,240.8,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,steady,1793910839.383,53.02,0,0,0,0,0.0,1.022,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910840.376,54.01,1,0,0,1,0.47,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910841.426,55.06,0,0,0,0,0.0,0.646,84.0,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910842.417,56.05,1,0,0,1,0.51,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910843.424,57.06,0,0,0,0,0.0,0.664,84.2,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910844.376,58.01,1,0,0,1,0.47,0.224,236.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910845.432,59.07,0,0,0,0,0.0,0.419,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910846.387,60.02,1,0,0,1,0.51,0.194,267.8,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910847.415,61.05,0,0,0,0,0.0,0.453,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910848.385,62.02,1,0,0,1,0.47,0.162,196.4,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910849.414,63.05,0,0,0,0,0.0,0.492,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910850.378,64.01,1,0,0,1,0.47,0.134,158.5,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910851.378,65.01,0,0,0,0,0.0,0.513,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910852.38,66.01,1,0,0,1,0.47,0.11,118.6,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910853.418,67.05,0,0,0,0,0.0,0.538,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910854.38,68.01,1,0,0,1,0.47,0.07,93.7,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910855.418,69.05,0,0,0,0,0.0,0.586,84.0,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910856.384,70.02,1,0,0,1,0.48,0.041,92.0,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910857.416,71.05,0,0,0,0,0.0,0.62,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910858.383,72.02,1,0,0,1,0.48,0.006,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910859.396,73.03,0,0,0,0,0.0,0.664,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910860.386,74.02,1,0,0,1,0.48,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910861.421,75.05,0,0,0,0,0.0,0.665,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910862.378,76.01,1,0,0,1,0.47,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910863.418,77.05,0,0,0,0,0.0,0.637,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910864.374,78.01,1,0,0,1,0.46,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910865.421,79.05,0,0,0,0,0.0,0.661,84.1,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910866.389,80.02,1,0,0,1,0.48,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910867.381,81.01,0,0,0,0,0.0,0.662,84.1,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910868.385,82.02,1,0,0,1,0.48,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910869.397,83.03,0,0,0,0,0.0,0.672,83.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910870.399,84.03,1,0,0,1,0.5,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910871.411,85.04,0,0,0,0,0.0,0.664,84.0,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910872.408,86.04,1,0,0,1,0.5,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910873.415,87.05,0,0,0,0,0.0,0.671,84.0,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910874.404,88.04,1,0,0,1,0.5,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910875.42,89.05,0,0,0,0,0.0,0.65,84.4,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910876.412,90.05,1,0,0,1,0.5,0.0,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910877.417,91.05,0,0,0,0,0.0,0.64,84.1,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,fixed,1793910878.375,92.01,1,0,0,1,0.46,0.215,238.1,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910879.37,93.0,1,1,0,0,0.1,0.424,83.9,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910880.376,94.01,5,2,1,2,0.8,1.392,100.3,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910881.386,95.02,9,2,5,2,1.48,1.987,531.1,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910882.393,96.03,11,2,7,2,1.81,1.976,84.9,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910883.395,97.03,15,2,11,2,2.47,1.97,398.6,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910884.391,98.02,17,2,13,2,2.8,1.979,462.7,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910885.375,99.01,21,2,17,2,3.45,1.968,396.5,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910886.378,100.01,23,2,19,2,3.78,1.98,451.9,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910887.378,101.01,27,2,23,2,4.44,1.969,389.4,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910888.377,102.01,29,2,25,2,4.76,1.979,427.9,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910889.375,103.01,33,2,29,2,5.42,1.967,359.2,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910890.377,104.01,35,2,31,2,5.75,1.984,395.3,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910891.392,105.03,39,2,35,2,6.43,1.949,107.2,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910892.393,106.03,43,2,39,2,7.09,1.985,666.6,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910893.387,107.02,44,1,41,2,7.41,1.969,91.7,5,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910894.39,108.02,48,1,45,2,8.08,1.979,396.8,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910895.384,109.02,50,1,47,2,8.4,1.987,452.6,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910896.381,110.01,54,1,51,2,9.06,1.975,391.9,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910897.378,111.01,56,1,53,2,9.38,1.983,419.7,6,0,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910898.377,112.01,60,1,57,2,10.05,1.969,267.1,6,1,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910899.384,113.02,58,0,58,0,10.38,1.98,667.5,2,4,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910900.379,114.01,59,1,56,2,11.03,1.963,103.7,3,2,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910901.406,115.04,60,0,58,2,11.73,1.984,398.2,3,4,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910902.392,116.03,58,0,56,2,12.04,1.981,443.2,2,3,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910903.375,117.01,60,1,57,2,12.68,1.967,370.7,4,2,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910904.371,118.0,58,0,56,2,13.0,1.988,533.9,2,4,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910905.386,119.02,60,1,57,2,13.68,1.979,152.8,4,2,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910906.381,120.01,60,0,58,2,14.34,1.984,667.9,2,4,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910907.386,121.02,59,1,56,2,14.67,1.975,92.3,3,3,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910908.383,122.02,60,0,58,2,15.33,1.984,398.0,3,3,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910909.387,123.02,58,0,56,2,15.65,1.978,454.2,2,3,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910910.375,124.01,60,0,58,2,16.31,1.981,394.4,4,3,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910911.395,125.03,58,0,56,2,16.66,1.991,424.0,2,3,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910912.384,126.02,59,0,57,2,17.3,1.931,112.3,3,3,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910913.383,127.02,60,0,58,2,17.97,1.985,668.3,3,4,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910914.392,128.03,59,0,57,2,18.31,1.974,84.8,3,2,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910915.387,129.02,60,0,58,2,18.95,1.968,394.0,3,4,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910916.391,130.02,58,1,55,2,19.29,1.98,434.1,2,3,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910917.38,131.01,60,2,56,2,19.95,1.967,315.1,4,3,0
|
||||
c2p2-x-r1,c2p2,x,1,overload,1793910918.381,132.01,58,1,55,2,20.28,1.983,668.4,2,3,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910919.384,133.02,59,1,56,2,20.44,1.976,108.4,3,2,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910920.386,134.02,57,0,55,2,20.76,1.972,663.3,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910921.389,135.02,53,0,51,2,20.42,1.983,460.0,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910922.373,136.01,51,0,49,2,20.73,1.972,394.7,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910923.382,137.02,47,0,45,2,20.22,1.974,419.6,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910924.375,138.01,45,0,43,2,20.55,1.978,237.2,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910925.383,139.02,41,0,39,2,20.21,1.984,621.6,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910926.385,140.02,39,0,37,2,20.53,1.97,108.4,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910927.393,141.03,37,0,35,2,20.87,1.982,665.1,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910928.383,142.02,33,0,31,2,20.34,1.986,273.0,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910929.376,143.01,31,0,29,2,20.66,1.981,397.8,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910930.385,144.02,27,0,25,2,20.32,1.984,441.1,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910931.373,145.01,25,0,23,2,20.64,1.969,337.2,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910932.38,146.01,21,0,19,2,20.12,1.975,668.8,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910933.368,147.0,19,0,17,2,20.44,1.967,103.0,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910934.375,148.01,17,0,15,2,20.77,1.984,554.7,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910935.379,149.01,13,0,11,2,20.43,1.98,460.9,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910936.378,150.01,11,0,9,2,20.58,1.975,394.4,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910937.393,151.03,7,0,5,2,20.25,1.986,429.8,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910938.382,152.02,5,0,3,2,20.57,1.98,339.6,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910939.379,153.01,1,0,0,1,20.22,1.982,668.8,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910940.399,154.03,0,0,0,0,0.0,1.119,84.1,0,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910941.404,155.04,1,1,0,0,0.25,0.0,84.1,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910942.384,156.02,2,1,0,1,0.67,0.558,260.4,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910943.382,157.01,2,1,0,1,0.58,1.302,259.7,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910944.375,158.01,1,0,0,1,0.45,1.283,251.0,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910945.376,159.01,1,0,0,1,0.33,1.322,246.9,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910946.378,160.01,1,1,0,0,0.2,1.286,242.1,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910947.388,161.02,2,1,0,1,0.62,0.638,236.6,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910948.388,162.02,1,0,0,1,0.53,1.303,376.3,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910949.373,163.01,1,0,0,1,0.39,1.297,376.1,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910950.377,164.01,2,0,0,2,0.84,1.301,375.6,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910951.379,165.01,2,0,0,2,0.72,1.316,375.9,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910952.386,166.02,2,1,0,1,0.6,0.911,84.2,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910953.378,167.01,2,0,0,2,1.02,1.033,374.8,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910954.382,168.02,2,0,0,2,0.91,1.295,375.9,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910955.374,169.01,2,0,0,2,0.78,1.307,374.3,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910956.386,170.02,2,0,0,2,0.67,1.283,241.9,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910957.39,171.02,1,0,0,1,0.55,0.973,84.1,1,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910958.391,172.02,2,0,0,2,0.96,0.981,241.4,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910959.373,173.01,2,0,0,2,0.85,1.287,241.0,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910960.364,174.0,2,1,0,1,0.71,1.3,240.7,2,0,0
|
||||
c2p2-x-r1,c2p2,x,1,recovery,1793910961.382,175.02,1,0,0,1,0.61,1.314,239.5,1,0,0
|
||||
c4p4-s-r1,c4p4,s,1,drain,1793910963.755,0.0,0,0,0,0,0.0,,22.6,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,drain,1793910964.778,1.02,12,10,2,0,0.43,,22.6,12,0,0
|
||||
c4p4-s-r1,c4p4,s,1,drain,1793910965.769,2.01,37,9,28,0,1.43,,,25,0,0
|
||||
c4p4-s-r1,c4p4,s,1,drain,1793910966.749,3.0,50,0,50,0,2.41,,,13,0,0
|
||||
c4p4-s-r1,c4p4,s,1,drain,1793910967.758,4.0,46,0,42,4,3.27,,64.1,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,drain,1793910968.775,5.02,30,0,26,4,3.65,1.934,103.1,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,drain,1793910969.746,5.99,18,0,14,4,4.14,2.098,139.5,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,drain,1793910970.75,7.0,3,0,0,3,4.59,1.97,84.8,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910971.772,8.02,0,0,0,0,0.0,1.496,83.3,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910972.762,9.01,6,2,0,4,0.73,0.0,83.3,6,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910973.764,10.01,6,3,0,3,0.62,1.014,139.9,9,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910974.75,11.0,4,2,0,2,0.5,1.231,140.1,7,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910975.753,12.0,6,2,0,4,0.62,1.188,130.0,9,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910976.748,12.99,6,2,0,4,0.63,1.26,122.2,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910977.759,14.0,6,2,0,4,0.64,1.172,122.2,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910978.762,15.01,6,2,0,4,0.65,1.129,113.1,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910979.757,16.0,6,1,1,4,0.65,1.153,86.3,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910980.755,17.0,5,1,0,4,0.53,1.35,84.0,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910981.764,18.01,5,1,0,4,0.55,1.193,83.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910982.766,19.01,5,1,0,4,0.55,1.238,83.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910983.762,20.01,5,1,0,4,0.56,1.229,83.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910984.773,21.02,5,0,1,4,0.57,1.221,83.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910985.777,22.02,4,0,0,4,0.46,1.371,83.6,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910986.766,23.01,4,0,2,2,0.45,1.209,83.7,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910987.763,24.01,4,4,0,0,0.46,1.235,83.6,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910988.778,25.02,4,4,0,0,0.48,1.236,83.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910989.776,26.02,4,4,0,0,0.48,1.24,83.7,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910990.769,27.01,3,3,0,0,0.36,1.281,93.6,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910991.778,28.02,3,3,0,0,0.37,1.288,83.7,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910992.771,29.02,3,3,0,0,0.37,1.205,83.7,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910993.76,30.01,3,3,0,0,0.37,1.258,99.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910994.765,31.01,4,4,0,0,0.38,1.201,145.2,9,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910995.752,32.0,4,3,0,1,0.37,1.201,115.1,7,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910996.759,33.0,3,3,0,0,0.26,1.352,100.6,9,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910997.752,34.0,3,3,0,0,0.26,1.202,145.5,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910998.769,35.02,3,3,0,0,0.28,1.147,142.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793910999.751,36.0,7,3,0,4,0.77,1.103,140.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911000.762,37.01,7,2,1,4,0.78,1.264,136.6,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911001.749,38.0,6,2,0,4,0.65,1.42,136.8,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911002.756,39.0,6,2,0,4,0.67,1.221,137.1,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911003.762,40.01,6,2,0,4,0.68,1.124,122.5,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911004.752,41.0,6,2,0,4,0.68,1.223,119.9,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911005.75,42.0,6,1,1,4,0.66,1.107,114.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911006.764,43.01,5,1,0,4,0.57,1.348,107.3,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911007.766,44.01,5,1,0,4,0.58,1.161,84.7,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911008.762,45.01,5,1,0,4,0.58,1.231,83.6,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911009.76,46.01,5,1,0,4,0.59,1.255,83.6,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911010.761,47.01,5,1,0,4,0.59,1.239,83.6,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911011.766,48.01,4,0,0,4,0.48,1.22,83.9,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,steady,1793911012.769,49.01,0,0,0,0,0.0,1.346,83.6,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911013.792,50.04,1,0,0,1,0.21,0.295,83.6,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911014.75,51.0,1,1,0,0,0.38,0.302,83.6,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911015.795,52.04,1,0,0,1,0.22,0.299,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911016.795,53.04,2,2,0,0,0.43,0.422,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911017.789,54.04,1,1,0,0,0.22,0.433,83.7,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911018.781,55.03,2,2,0,0,0.41,0.297,83.9,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911019.78,56.03,1,1,0,0,0.21,0.435,83.7,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911020.786,57.03,1,1,0,0,0.01,0.305,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911021.772,58.02,1,1,0,0,0.21,0.415,83.9,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911022.776,59.02,1,1,0,0,0.01,0.316,83.9,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911023.766,60.01,1,1,0,0,0.2,0.461,84.0,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911024.768,61.01,0,0,0,0,0.0,0.294,99.1,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911025.77,62.02,1,1,0,0,0.2,0.426,97.7,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911026.76,63.01,0,0,0,0,0.0,0.361,112.4,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911027.764,64.01,1,1,0,0,0.18,0.324,91.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911028.76,65.01,1,0,0,1,0.39,0.3,93.1,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911029.759,66.0,2,0,0,2,0.59,0.417,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911030.756,67.0,1,0,0,1,0.39,0.462,83.6,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911031.766,68.01,1,0,0,1,0.19,0.314,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911032.788,69.03,2,1,0,1,0.42,0.431,83.7,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911033.79,70.04,1,0,0,1,0.22,0.314,83.8,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911034.783,71.03,2,0,0,2,0.42,0.446,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911035.768,72.01,1,1,0,0,0.19,0.44,83.9,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911036.775,73.02,2,2,0,0,0.41,0.283,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911037.799,74.04,1,1,0,0,0.23,0.419,83.9,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911038.778,75.02,2,2,0,0,0.41,0.295,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911039.777,76.02,1,1,0,0,0.21,0.444,84.0,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911040.769,77.02,0,0,0,0,0.0,0.305,84.1,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911041.752,78.0,1,1,0,0,0.18,0.441,83.8,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911042.782,79.03,1,1,0,0,0.01,0.293,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911043.771,80.02,1,1,0,0,0.2,0.456,84.1,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911044.775,81.02,1,1,0,0,0.01,0.289,99.1,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911045.773,82.02,1,1,0,0,0.2,0.424,97.6,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911046.754,83.0,0,0,0,0,0.0,0.353,102.8,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911047.754,84.0,1,1,0,0,0.18,0.341,91.5,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911048.757,85.0,1,0,0,1,0.38,0.302,100.9,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911049.757,86.0,2,0,0,2,0.58,0.41,83.8,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911050.757,87.0,1,0,0,1,0.39,0.441,83.6,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911051.766,88.01,2,0,0,2,0.6,0.309,83.6,3,0,0
|
||||
c4p4-s-r1,c4p4,s,1,fixed,1793911052.755,89.0,1,0,0,1,0.39,0.445,83.8,2,0,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911053.767,90.01,0,0,0,0,0.0,0.301,83.6,1,0,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911054.744,90.99,14,1,11,2,0.49,0.157,83.6,21,0,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911055.753,92.0,29,0,25,4,1.05,1.935,125.0,27,0,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911056.759,93.0,44,14,26,4,1.61,2.015,145.6,27,0,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911057.755,94.0,54,13,37,4,2.01,1.93,113.8,26,0,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911058.769,95.02,60,7,49,4,2.57,2.06,129.6,18,10,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911059.752,96.0,60,5,51,4,3.07,1.998,133.3,13,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911060.753,97.0,59,6,49,4,3.51,1.98,110.7,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911061.759,98.01,60,5,51,4,4.08,2.028,137.4,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911062.753,99.0,60,4,52,4,4.43,1.996,137.9,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911063.752,100.0,58,4,50,4,4.31,1.99,117.4,13,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911064.755,101.0,60,6,50,4,4.35,2.001,114.4,15,12,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911065.75,102.0,60,4,52,4,4.45,2.015,138.6,12,15,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911066.759,103.0,59,5,50,4,4.41,1.956,116.8,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911067.755,104.0,60,5,51,4,4.41,2.022,119.9,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911068.782,105.03,60,4,52,4,4.46,2.018,139.0,12,15,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911069.758,106.0,59,4,51,4,4.47,1.98,111.0,13,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911070.75,107.0,60,4,52,4,4.52,2.017,126.4,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911071.749,108.0,60,4,52,4,4.45,2.0,127.6,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911072.754,109.0,59,3,52,4,4.44,1.984,113.9,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911073.763,110.01,60,3,54,3,4.52,2.015,119.9,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911074.755,111.0,59,2,53,4,4.43,1.981,129.5,12,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911075.751,112.0,60,3,53,4,4.45,2.01,111.6,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911076.75,113.0,60,3,53,4,4.49,2.009,131.9,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911077.757,114.0,59,2,53,4,4.45,1.984,127.3,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911078.748,114.99,60,3,53,4,4.47,2.018,128.3,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911079.755,116.0,60,1,55,4,4.44,2.001,126.0,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911080.746,116.99,59,2,53,4,4.42,1.983,121.7,13,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911081.747,117.99,60,1,55,4,4.41,1.998,128.0,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911082.752,119.0,60,1,55,4,4.41,2.015,131.1,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911083.755,120.0,60,2,54,4,4.45,1.993,118.4,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911084.76,121.01,59,1,54,4,4.33,1.994,104.5,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911085.768,122.01,60,2,54,4,4.44,2.014,131.5,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911086.756,123.0,60,0,56,4,4.51,1.998,130.0,13,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911087.756,124.0,60,6,50,4,4.5,1.998,131.3,13,14,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911088.768,125.01,58,6,48,4,4.43,1.989,111.5,12,15,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911089.759,126.0,60,6,50,4,4.52,2.015,117.9,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911090.761,127.01,60,7,49,4,4.44,1.985,128.2,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911091.761,128.01,60,6,50,4,4.55,2.024,120.3,12,15,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911092.764,129.01,60,6,50,4,4.43,1.992,129.6,14,13,0
|
||||
c4p4-s-r1,c4p4,s,1,overload,1793911093.749,129.99,59,6,49,4,4.42,1.958,114.6,13,13,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911094.758,131.0,48,0,45,3,4.34,2.044,106.0,3,4,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911095.76,132.01,36,0,32,4,4.41,1.991,130.8,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911096.751,133.0,22,0,18,4,4.4,2.0,128.9,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911097.751,134.0,8,0,4,4,4.31,1.996,112.7,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911098.767,135.01,0,0,0,0,0.0,2.005,104.6,0,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911099.777,136.02,3,3,0,0,0.29,0.183,83.8,5,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911100.753,137.0,3,3,0,0,0.27,0.853,145.8,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911101.763,138.01,7,3,0,4,0.78,1.112,133.3,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911102.76,139.01,2,2,0,0,0.16,1.271,137.2,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911103.752,140.0,6,2,0,4,0.66,1.364,141.0,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911104.763,141.01,6,2,0,4,0.67,1.246,136.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911105.748,141.99,6,2,0,4,0.66,1.149,122.9,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911106.759,143.0,6,1,1,4,0.67,1.117,116.2,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911107.75,144.0,5,1,0,4,0.55,1.374,113.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911108.761,145.01,5,1,0,4,0.57,1.127,89.4,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911109.755,146.0,5,1,0,4,0.57,1.233,83.9,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911110.758,147.0,5,1,0,4,0.58,1.23,83.9,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911111.761,148.01,5,0,1,4,0.58,1.261,83.9,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911112.768,149.01,4,0,0,4,0.48,1.364,84.0,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911113.775,150.02,4,0,0,4,0.5,1.222,83.9,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911114.777,151.02,4,0,0,4,0.5,1.239,84.1,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911115.782,152.03,5,1,2,2,0.51,1.246,84.1,9,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911116.761,153.01,4,4,0,0,0.49,1.248,84.1,7,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911117.777,154.02,4,4,0,0,0.4,1.273,93.6,9,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911118.753,155.0,4,4,0,0,0.38,1.296,84.0,8,0,0
|
||||
c4p4-s-r1,c4p4,s,1,recovery,1793911119.781,156.03,0,0,0,0,0.0,1.237,83.9,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,drain,1793911121.513,0.0,0,0,0,0,0.0,,67.2,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,drain,1793911122.525,1.01,11,4,7,0,0.41,,67.2,11,0,0
|
||||
c2p32-s-r1,c2p32,s,1,drain,1793911123.535,2.02,37,5,32,0,1.42,,67.2,26,0,0
|
||||
c2p32-s-r1,c2p32,s,1,drain,1793911124.557,3.04,50,0,50,0,2.44,,,13,0,0
|
||||
c2p32-s-r1,c2p32,s,1,drain,1793911125.52,4.01,50,0,50,0,3.41,,16.7,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,drain,1793911126.497,4.98,38,0,36,2,3.93,1.474,112.7,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,drain,1793911127.506,5.99,24,0,22,2,4.37,1.865,111.3,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,drain,1793911128.511,7.0,10,0,8,2,4.82,1.869,102.9,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,drain,1793911129.496,7.98,0,0,0,0,0.0,1.876,100.1,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911130.519,9.01,2,1,0,1,0.13,0.197,83.9,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911131.531,10.02,5,1,2,2,0.53,0.452,92.9,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911132.527,11.01,5,1,2,2,0.54,1.097,114.6,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911133.516,12.0,4,0,2,2,0.43,1.084,113.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911134.539,13.03,4,0,2,2,0.47,1.22,113.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911135.536,14.02,4,4,0,0,0.48,1.085,111.7,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911136.516,15.0,3,3,0,0,0.36,1.069,105.6,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911137.534,16.02,4,4,0,0,0.39,1.202,103.2,9,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911138.537,17.02,4,4,0,0,0.41,1.067,99.5,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911139.524,18.01,4,4,0,0,0.4,1.06,88.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911140.518,19.01,3,3,0,0,0.31,1.258,84.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911141.515,20.0,3,3,0,0,0.32,1.099,84.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911142.511,21.0,3,3,0,0,0.33,1.127,84.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911143.512,22.0,3,2,0,1,0.34,1.126,84.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911144.511,23.0,3,3,0,0,0.25,1.245,84.1,9,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911145.508,24.0,5,3,0,2,0.51,1.136,84.3,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911146.52,25.01,5,3,0,2,0.54,1.131,84.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911147.517,26.0,4,2,0,2,0.43,1.189,93.7,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911148.523,27.01,4,2,0,2,0.45,1.198,84.3,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911149.522,28.01,4,1,1,2,0.47,1.142,84.2,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911150.516,29.0,3,1,0,2,0.36,1.244,84.4,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911151.513,30.0,4,2,0,2,0.37,1.11,113.9,9,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911152.504,30.99,6,1,3,2,0.63,1.104,113.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911153.514,32.0,5,1,2,2,0.54,1.231,112.4,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911154.51,33.0,5,1,2,2,0.55,1.079,105.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911155.517,34.0,5,1,2,2,0.58,1.09,102.6,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911156.509,35.0,4,0,2,2,0.46,1.102,99.6,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911157.516,36.0,4,0,2,2,0.49,1.237,99.4,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911158.546,37.03,5,1,2,2,0.54,1.109,85.3,9,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911159.522,38.01,4,0,2,2,0.41,1.092,113.5,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911160.531,39.02,4,4,0,0,0.43,1.209,112.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911161.522,40.01,4,4,0,0,0.44,1.08,103.3,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911162.526,41.01,4,4,0,0,0.47,1.122,104.8,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911163.527,42.01,3,3,0,0,0.36,1.204,103.4,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911164.53,43.02,4,4,0,0,0.38,1.069,99.9,9,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911165.528,44.02,4,4,0,0,0.4,1.065,89.8,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911166.519,45.01,4,3,0,1,0.4,1.109,84.1,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911167.504,45.99,3,3,0,0,0.29,1.235,84.1,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911168.519,47.01,3,3,0,0,0.32,1.118,84.1,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911169.517,48.0,3,3,0,0,0.33,1.115,84.1,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,steady,1793911170.489,48.98,3,1,0,2,0.45,1.256,84.9,7,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911171.56,50.05,1,1,0,0,0.0,0.72,84.1,1,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911172.525,51.01,1,1,0,0,0.17,0.139,84.1,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911173.531,52.02,0,0,0,0,0.0,0.283,84.1,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911174.517,53.0,1,1,0,0,0.17,0.461,84.1,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911175.517,54.0,0,0,0,0,0.0,0.458,84.4,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911176.511,55.0,2,1,0,1,0.56,0.316,84.5,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911177.52,56.01,1,0,0,1,0.36,0.469,84.6,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911178.516,57.0,2,0,0,2,0.56,0.326,84.4,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911179.522,58.01,1,0,0,1,0.37,0.44,84.6,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911180.536,59.02,2,0,0,2,0.57,0.314,84.5,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911181.545,60.03,1,0,0,1,0.4,0.446,98.7,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911182.52,61.01,1,1,0,0,0.16,0.38,112.3,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911183.514,62.0,1,1,0,0,0.36,0.314,93.6,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911184.552,63.04,1,1,0,0,0.2,0.418,92.0,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911185.56,64.05,2,2,0,0,0.41,0.266,90.7,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911186.551,65.04,1,1,0,0,0.19,0.424,85.8,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911187.551,66.04,1,1,0,0,0.39,0.287,84.3,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911188.536,67.02,1,1,0,0,0.19,0.457,84.2,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911189.521,68.01,0,0,0,0,0.0,0.317,84.1,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911190.528,69.02,1,1,0,0,0.18,0.437,84.2,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911191.532,70.02,0,0,0,0,0.0,0.31,84.3,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911192.531,71.02,1,1,0,0,0.18,0.462,84.2,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911193.525,72.01,0,0,0,0,0.0,0.433,84.7,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911194.505,72.99,1,1,0,0,0.15,0.318,84.2,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911195.529,74.02,1,0,0,1,0.37,0.447,84.2,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911196.53,75.02,2,0,0,2,0.58,0.317,84.4,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911197.513,76.0,1,0,0,1,0.36,0.438,84.2,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911198.523,77.01,2,0,0,2,0.57,0.3,84.5,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911199.525,78.01,1,0,0,1,0.38,0.464,99.6,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911200.56,79.05,1,0,0,1,0.2,0.416,115.2,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911201.556,80.04,2,1,0,1,0.41,0.278,94.7,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911202.523,81.01,1,1,0,0,0.16,0.351,103.3,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911203.525,82.01,1,1,0,0,0.37,0.365,93.8,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911204.549,83.04,1,1,0,0,0.19,0.398,92.9,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911205.551,84.04,1,1,0,0,0.4,0.282,91.7,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911206.525,85.01,1,1,0,0,0.17,0.406,84.2,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911207.526,86.01,1,1,0,0,0.36,0.305,84.3,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911208.53,87.02,1,1,0,0,0.18,0.441,84.2,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911209.531,88.02,0,0,0,0,0.0,0.308,84.4,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911210.514,89.0,1,1,0,0,0.16,0.449,84.2,3,0,0
|
||||
c2p32-s-r1,c2p32,s,1,fixed,1793911211.498,89.99,0,0,0,0,0.0,0.441,84.4,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911212.529,91.02,2,2,0,0,0.03,0.145,84.2,2,0,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911213.511,92.0,21,6,13,2,0.75,0.621,99.8,27,0,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911214.507,92.99,34,6,26,2,1.23,1.879,84.5,27,0,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911215.505,93.99,47,5,40,2,1.71,1.878,113.4,27,0,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911216.508,95.0,60,3,55,2,2.27,1.844,103.1,25,2,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911217.51,96.0,60,2,56,2,2.75,1.865,100.0,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911218.52,97.01,60,2,56,2,3.25,1.878,84.9,14,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911219.512,98.0,59,2,55,2,3.73,1.868,113.2,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911220.509,99.0,58,1,55,2,4.21,1.862,104.9,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911221.511,100.0,60,2,56,2,4.4,1.873,99.9,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911222.507,100.99,60,2,56,2,4.33,1.874,85.4,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911223.513,102.0,59,1,56,2,4.3,1.877,113.4,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911224.527,103.01,59,1,56,2,4.29,1.875,112.8,14,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911225.509,104.0,60,0,58,2,4.39,1.875,103.5,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911226.51,105.0,60,0,58,2,4.35,1.871,100.0,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911227.511,106.0,60,0,58,2,4.33,1.88,114.6,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911228.527,107.01,58,0,56,2,4.31,1.828,104.2,12,15,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911229.513,108.0,60,6,52,2,4.41,1.877,100.4,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911230.512,109.0,60,7,51,2,4.38,1.874,101.4,14,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911231.514,110.0,60,7,51,2,4.35,1.882,115.1,13,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911232.519,111.01,59,7,50,2,4.29,1.867,112.2,14,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911233.513,112.0,58,6,50,2,4.25,1.87,103.4,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911234.519,113.01,60,6,52,2,4.37,1.869,100.1,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911235.518,114.01,60,6,52,2,4.34,1.871,115.2,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911236.523,115.01,60,6,52,2,4.31,1.876,112.9,14,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911237.511,116.0,58,4,52,2,4.27,1.876,112.9,12,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911238.512,117.0,60,5,53,2,4.39,1.864,100.4,14,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911239.509,118.0,60,5,53,2,4.36,1.867,93.1,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911240.506,118.99,60,6,52,2,4.32,1.898,115.0,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911241.514,120.0,59,5,52,2,4.3,1.871,113.3,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911242.514,121.0,60,4,54,2,4.41,1.818,103.0,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911243.525,122.01,60,4,54,2,4.39,1.86,91.4,14,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911244.512,123.0,60,4,54,2,4.35,1.887,114.9,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911245.513,124.0,59,4,53,2,4.32,1.87,112.4,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911246.515,125.0,58,3,53,2,4.25,1.868,103.7,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911247.518,126.01,60,4,54,2,4.37,1.876,100.0,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911248.501,126.99,60,4,54,2,4.32,1.873,85.1,14,13,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911249.508,128.0,59,3,54,2,4.3,1.873,112.2,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911250.512,129.0,58,2,54,2,4.27,1.873,105.2,13,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911251.513,130.0,60,2,56,2,4.39,1.871,100.4,14,14,0
|
||||
c2p32-s-r1,c2p32,s,1,overload,1793911252.522,131.01,60,2,56,2,4.37,1.873,93.9,14,12,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911253.514,132.0,46,0,44,2,4.33,1.88,115.6,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911254.511,133.0,32,0,30,2,4.3,1.864,112.2,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911255.515,134.0,18,0,16,2,4.27,1.865,103.8,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911256.526,135.01,6,0,4,2,4.39,1.873,103.7,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911257.528,136.02,0,0,0,0,0.0,1.287,85.1,0,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911258.519,137.01,6,1,3,2,0.62,0.144,84.7,7,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911259.514,138.0,5,1,2,2,0.52,1.108,105.2,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911260.518,139.0,5,1,2,2,0.54,1.069,100.2,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911261.516,140.0,5,1,2,2,0.55,1.108,94.4,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911262.536,141.02,4,0,2,2,0.47,1.113,85.6,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911263.523,142.01,4,0,2,2,0.47,1.23,85.6,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911264.539,143.03,5,1,2,2,0.51,1.074,113.1,9,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911265.495,143.98,3,3,0,0,0.36,1.112,112.4,7,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911266.533,145.02,4,4,0,0,0.42,1.211,113.0,9,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911267.513,146.0,4,4,0,0,0.41,1.078,103.6,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911268.526,147.01,4,4,0,0,0.44,1.077,99.9,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911269.531,148.02,3,3,0,0,0.35,1.224,100.8,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911270.518,149.0,3,3,0,0,0.35,1.062,93.4,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911271.524,150.01,4,4,0,0,0.37,1.097,84.5,9,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911272.527,151.01,4,3,0,1,0.39,1.133,84.5,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911273.503,151.99,3,3,0,0,0.26,1.243,84.5,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911274.509,153.0,3,3,0,0,0.29,1.127,84.7,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911275.517,154.0,3,2,0,1,0.31,1.143,84.5,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911276.516,155.0,4,2,0,2,0.46,1.27,84.5,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911277.518,156.0,4,2,0,2,0.48,1.123,85.0,8,0,0
|
||||
c2p32-s-r1,c2p32,s,1,recovery,1793911278.525,157.01,0,0,0,0,0.0,1.133,85.2,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911280.006,0.0,0,0,0,0,0.0,,3.4,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911281.013,1.01,2,1,1,0,0.33,,3.4,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911282.02,2.02,6,1,5,0,1.33,,,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911283.025,3.02,10,4,6,0,2.34,,,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911284.017,4.01,14,1,13,0,3.33,,,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911285.024,5.02,18,1,17,0,4.34,,,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911286.026,6.02,22,1,21,0,5.34,,,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911287.009,7.0,26,0,26,0,6.32,,,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911288.023,8.02,30,0,30,0,7.34,,,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911289.019,9.01,34,0,34,0,8.33,,,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911290.027,10.02,38,0,38,0,9.34,,,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911291.007,11.0,40,0,40,0,10.32,,,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911292.008,12.0,40,0,36,4,11.32,,40.2,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911293.01,13.0,36,0,32,4,11.33,1.767,165.1,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911294.015,14.01,32,0,28,4,11.0,1.475,267.7,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911294.997,14.99,28,0,24,4,11.33,1.432,405.8,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911296.007,16.0,24,0,20,4,11.33,1.492,400.5,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911296.994,16.99,20,0,16,4,11.32,1.38,363.4,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911298.007,18.0,15,0,11,4,11.1,1.734,321.8,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911299.007,19.0,11,0,7,4,11.07,1.877,385.7,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911300.015,20.01,7,0,3,4,11.08,1.561,388.5,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,drain,1793911301.015,21.01,3,0,0,3,11.09,1.551,386.8,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911301.999,21.99,0,0,0,0,0.0,1.534,146.4,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911303.006,23.0,2,1,0,1,0.63,0.04,106.8,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911304.016,24.01,2,1,0,1,0.81,0.722,104.3,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911305.016,25.01,2,0,0,2,1.4,0.726,90.3,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911306.019,26.01,4,1,0,3,0.89,0.711,122.2,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911307.014,27.01,2,1,0,1,0.49,0.977,323.3,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911308.012,28.01,2,1,0,1,0.49,1.293,83.2,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911309.009,29.0,3,1,0,2,0.84,0.739,83.3,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911310.015,30.01,2,1,0,1,0.6,1.095,122.3,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911311.018,31.01,2,0,0,2,0.78,0.726,93.9,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911312.017,32.01,2,1,0,1,0.54,1.084,115.6,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911313.018,33.01,2,0,0,2,0.7,0.728,155.0,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911314.014,34.01,3,1,0,2,0.34,1.015,244.6,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911315.016,35.01,2,0,0,2,0.62,0.741,244.6,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911316.027,36.02,1,0,0,1,0.4,1.035,237.6,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911317.017,37.01,2,0,0,2,0.55,0.812,154.9,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911318.006,38.0,1,0,0,1,0.31,0.94,213.9,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911319.011,39.01,2,1,0,1,0.49,0.892,145.2,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911320.01,40.01,3,0,0,3,1.06,0.706,144.1,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911321.015,41.01,1,0,0,1,0.38,0.719,233.6,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911322.015,42.01,3,0,0,3,1.0,1.09,126.8,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911323.02,43.01,1,0,0,1,0.35,1.067,116.0,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911324.01,44.0,3,2,0,1,0.93,0.726,116.1,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911325.014,45.01,2,1,0,1,0.68,0.571,197.8,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911326.006,46.0,2,1,0,1,0.85,1.289,112.3,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911327.003,47.0,3,1,0,2,0.97,0.727,111.5,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911328.005,48.0,1,0,0,1,1.2,1.115,111.0,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911329.006,49.0,3,1,0,2,0.88,0.312,87.6,5,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911329.996,49.99,2,1,0,1,0.71,1.477,115.1,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911331.002,51.0,3,1,0,2,0.78,0.734,83.4,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911332.025,52.02,2,0,0,2,1.49,1.084,113.1,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911333.011,53.01,2,1,0,1,0.78,0.39,122.3,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911334.013,54.01,2,0,0,2,1.3,1.101,94.5,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911335.01,55.0,2,0,0,2,0.74,0.366,168.1,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911336.009,56.0,3,0,0,3,1.74,1.791,244.8,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911337.01,57.01,2,0,0,2,0.67,0.095,130.4,4,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911338.012,58.01,2,1,0,1,0.44,1.717,240.3,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911339.003,59.0,2,1,0,1,0.6,0.859,164.1,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911340.01,60.01,1,0,0,1,0.37,0.713,161.4,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911341.003,61.0,2,1,0,1,0.52,1.116,159.0,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,steady,1793911342.024,62.02,1,0,0,1,0.31,0.713,153.6,2,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911343.02,63.01,0,0,0,0,0.0,1.102,154.7,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911344.033,64.03,0,0,0,0,0.0,0.119,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911345.023,65.02,0,0,0,0,0.0,0.378,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911346.032,66.03,1,0,0,1,0.48,0.386,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911347.028,67.02,1,0,0,1,0.22,0.138,120.6,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911348.027,68.02,0,0,0,0,0.0,0.379,128.6,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911349.011,69.01,0,0,0,0,0.0,0.234,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911350.011,70.01,0,0,0,0,0.0,0.386,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911351.046,71.04,1,1,0,0,0.5,0.391,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911352.021,72.02,1,1,0,0,0.25,0.069,111.6,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911353.017,73.01,0,0,0,0,0.0,0.375,111.7,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911354.055,74.05,0,0,0,0,0.0,0.371,112.0,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911355.014,75.01,1,0,0,1,0.7,0.301,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911356.007,76.0,1,0,0,1,0.44,0.374,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911357.007,77.0,1,1,0,0,0.21,0.357,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911358.019,78.01,0,0,0,0,0.0,0.0,83.4,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911359.055,79.05,0,0,0,0,0.0,0.361,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911360.021,80.02,1,0,0,1,0.75,0.384,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911361.008,81.0,1,0,0,1,0.45,0.368,83.6,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911362.034,82.03,1,1,0,0,0.24,0.364,83.7,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911363.021,83.02,0,0,0,0,0.0,0.001,83.4,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911364.042,84.04,0,0,0,0,0.0,0.389,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911365.012,85.01,1,0,0,1,0.71,0.367,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911366.004,86.0,1,0,0,1,0.45,0.354,122.1,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911367.04,87.03,1,0,0,1,1.48,0.337,164.2,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911368.0,87.99,0,0,0,0,0.0,0.071,83.6,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911369.039,89.03,0,0,0,0,0.0,0.381,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911370.021,90.02,0,0,0,0,0.0,0.373,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911371.011,91.01,1,0,0,1,0.46,0.358,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911372.019,92.01,1,0,0,1,0.21,0.211,144.7,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911373.008,93.0,0,0,0,0,0.0,0.391,145.5,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911374.026,94.02,0,0,0,0,0.0,0.155,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911375.029,95.02,0,0,0,0,0.0,0.375,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911376.03,96.02,1,0,0,1,0.49,0.36,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911377.032,97.03,1,0,0,1,0.23,0.142,124.5,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911378.023,98.02,0,0,0,0,0.0,0.366,116.1,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911379.015,99.01,0,0,0,0,0.0,0.259,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911379.993,99.99,0,0,0,0,0.0,0.386,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911381.019,101.01,0,0,0,0,0.0,0.375,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,fixed,1793911381.984,101.98,1,1,0,0,0.19,0.361,83.4,1,0,0
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911383.047,103.04,0,0,0,0,0.0,0.055,109.9,0,0,0
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911384.004,104.0,8,4,0,4,0.88,0.306,83.4,8,0,0
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911385.009,105.0,11,0,7,4,1.89,1.054,355.8,3,0,0
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911386.025,106.02,14,4,6,4,2.9,0.569,126.2,3,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911387.025,107.02,14,8,2,4,3.9,0.024,126.1,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911388.016,108.01,14,8,2,4,4.89,0.025,126.1,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911389.005,109.0,14,8,2,4,5.88,0.026,83.6,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911390.033,110.03,14,10,0,4,6.91,0.023,83.6,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911391.017,111.01,14,10,0,4,7.89,0.019,83.6,0,0,7
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911392.027,112.02,14,13,0,1,8.9,0.019,83.7,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911393.017,113.01,14,12,0,2,9.89,0.02,83.7,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911393.993,113.99,14,9,1,4,10.87,0.007,83.7,0,0,7
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911395.005,115.0,14,9,1,4,11.88,0.021,83.7,0,0,9
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911396.017,116.01,14,12,0,2,12.89,0.013,83.8,0,0,18
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911397.014,117.01,14,14,0,0,13.89,0.015,83.8,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911398.019,118.01,14,14,0,0,14.9,0.004,83.8,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911399.022,119.02,14,14,0,0,15.9,0.0,83.8,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911400.025,120.02,14,14,0,0,16.9,0.0,83.8,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911401.017,121.01,14,14,0,0,17.89,0.022,83.9,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911402.013,122.01,14,14,0,0,18.89,0.024,83.9,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911403.005,123.0,14,14,0,0,19.88,0.0,83.9,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911404.026,124.02,14,14,0,0,20.9,0.026,83.9,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911405.008,125.0,14,14,0,0,21.89,0.006,83.9,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911406.005,126.0,14,14,0,0,22.88,0.0,83.9,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911407.024,127.02,14,14,0,0,23.9,0.0,83.9,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911408.019,128.01,14,14,0,0,24.9,0.0,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911409.031,129.03,13,13,0,0,25.91,0.0,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911410.023,130.02,12,12,0,0,26.9,0.004,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911411.007,131.0,11,11,0,0,27.88,0.005,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911412.022,132.02,9,9,0,0,28.54,0.006,84.2,0,0,9
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911413.022,133.02,7,7,0,0,29.17,0.006,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911414.019,134.01,6,6,0,0,30.17,0.007,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911415.016,135.01,4,4,0,0,30.9,0.003,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911416.014,136.01,3,3,0,0,31.77,0.007,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911417.019,137.01,3,3,0,0,32.78,0.005,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911418.012,138.01,2,2,0,0,32.98,0.0,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911419.012,139.01,0,0,0,0,0.0,0.004,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911420.012,140.01,0,0,0,0,0.0,0.005,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911421.023,141.02,0,0,0,0,0.0,0.0,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911422.0,141.99,0,0,0,0,0.0,0.0,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,overload,1793911423.016,143.01,0,0,0,0,0.0,0.0,84.2,0,0,8
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911424.049,144.04,0,0,0,0,0.0,0.0,84.2,0,0,1
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911425.023,145.02,0,0,0,0,0.0,0.0,84.2,0,0,3
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911426.055,146.05,0,0,0,0,0.0,0.0,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911427.023,147.02,0,0,0,0,0.0,0.0,84.2,0,0,3
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911428.049,148.04,0,0,0,0,0.0,0.0,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911429.044,149.04,0,0,0,0,0.0,0.0,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911430.029,150.02,0,0,0,0,0.0,0.0,84.2,0,0,3
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911431.056,151.05,0,0,0,0,0.0,0.0,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911432.019,152.01,0,0,0,0,0.0,0.0,84.2,0,0,3
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911433.009,153.0,0,0,0,0,0.0,0.001,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911434.043,154.04,0,0,0,0,0.0,0.0,84.2,0,0,3
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911435.045,155.04,0,0,0,0,0.0,0.0,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911436.045,156.04,0,0,0,0,0.0,0.0,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911437.017,157.01,0,0,0,0,0.0,0.0,84.2,0,0,3
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911438.017,158.01,0,0,0,0,0.0,0.0,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911439.017,159.01,0,0,0,0,0.0,0.0,84.2,0,0,3
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911440.04,160.03,0,0,0,0,0.0,0.0,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911441.046,161.04,0,0,0,0,0.0,0.0,84.2,0,0,2
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911442.043,162.04,0,0,0,0,0.0,0.0,84.2,0,0,3
|
||||
c4p4-l-r1,c4p4,l,1,recovery,1793911443.015,163.01,0,0,0,0,0.0,0.001,84.2,0,0,2
|
||||
|
@@ -0,0 +1,121 @@
|
||||
cell,config,concurrency,prefetch,size_class,body_bytes,repeat,order,phase,rate,window_s,scheduled,sent,accepted,rejected_429,retry_after_5,retry_after_1,errors_5xx,other_errors,dropped,share_429,share_429_saturated,backlog_reached,throughput,drain_s,startup_s,ttr_p50_ms,ttr_p95_ms,ttr_p99_ms,ttr_max_ms,lag_p50_ms,lag_p95_ms,lag_max_ms,took_p50_ms,took_p95_ms,max_unfinished,max_unfinished_sampled,max_outbox,max_in_broker,max_oldest_age_s,recovery_s,jobs,ready,failed,missing,unknown_jobs,retries,skipped,worker_cpu_cores,worker_mem_max_mib,worker_peak_mib,render_peak_mib,upload_p95_ms,flags
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,drain,10.0,3.0,20,20,20,0,0,0,0,0,0,0.0,,False,2.968,6.738,0.605,5333.0,7567.1,7652.3,7652.3,4663.0,6906.0,6985.0,665.0,737.0,,20,4,20,7.55,,20,20,0,0,0,0,0,1.97,458.9,667.8,299.4,104.22,
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,steady,1.781,40.0,72,72,72,0,0,0,0,0,0,0.0,,False,,,,915.7,1157.1,1170.7,1170.7,261.0,489.0,511.0,656.0,669.0,,3,1,0,1.14,,72,72,0,0,0,0,0,1.13,376.0,667.8,299.5,130.97,
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,fixed,0.5,40.0,20,20,20,0,0,0,0,0,0,0.0,,False,,,,903.2,1137.6,1144.6,1144.6,253.0,474.0,493.0,656.0,674.0,,1,1,0,1.08,,20,20,0,0,0,0,0,0.325,375.5,667.8,299.4,131.99,
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,overload,5.936,40.0,238,238,172,66,66,0,0,0,0,0.2773,0.528,True,,,,16106.1,21323.8,21376.3,21379.9,15398.0,20621.0,20681.0,701.0,738.0,60,60,3,58,20.6,,172,172,0,0,0,0,0,1.919,560.4,667.8,299.5,117.11,
|
||||
c2p2-x-r1,c2p2,2,2,x,119755,1,1,recovery,1.781,20.0,36,36,36,0,0,0,0,0,0,0.0,,False,,,,913.6,1155.9,1161.9,1161.9,253.0,489.0,510.0,662.0,678.0,,59,1,56,21.35,20.989,36,36,0,0,0,0,0,1.515,563.2,667.8,299.4,132.35,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,13.473,3.711,0.628,3883.9,4611.2,4652.0,4652.0,3594.0,4321.0,4453.0,293.0,311.0,,50,11,50,4.49,,50,50,0,0,0,0,0,1.907,139.6,146.2,96.8,13.71,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,steady,8.084,40.0,324,324,324,0,0,0,0,0,0,0.0,,False,,,,548.7,765.8,793.0,820.6,280.0,503.0,519.0,279.0,328.0,,7,5,1,0.74,,324,324,0,0,0,0,0,1.203,142.3,146.2,97.4,18.97,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,433.1,676.4,686.9,695.4,271.0,505.0,542.0,160.0,168.0,,2,1,0,0.53,,100,100,0,0,0,0,0,0.377,114.5,146.2,97.4,26.61,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,overload,26.946,40.0,1078,1078,587,491,491,0,0,0,0,0.4555,0.5062,True,,,,4443.8,4538.3,4572.0,4575.0,4156.0,4246.0,4286.0,297.0,326.0,61,60,13,56,4.57,,587,587,0,0,0,0,0,1.935,142.2,146.2,97.9,16.05,
|
||||
c4p4-s-r1,c4p4,4,4,s,192323,1,2,recovery,8.084,20.0,162,162,162,0,0,0,0,0,0,0.0,,False,,,,537.5,762.3,784.3,787.1,274.0,498.0,521.0,279.0,320.0,,52,4,48,4.52,4.334,162,162,0,0,0,0,0,1.302,142.3,146.2,97.9,17.18,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,13.558,3.688,0.644,3851.6,4616.2,4683.7,4683.7,3711.0,4472.0,4540.0,146.0,150.0,,50,9,50,4.56,,50,50,0,0,0,0,0,1.782,112.5,116.1,97.2,14.51,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,steady,8.135,40.0,326,326,326,0,0,0,0,0,0,0.0,,False,,,,502.9,655.1,668.4,674.5,353.0,502.0,521.0,148.0,160.0,,6,4,3,0.64,,326,326,0,0,0,0,0,1.116,114.8,116.1,97.7,15.22,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,425.2,667.6,682.8,685.0,273.0,506.0,521.0,158.0,169.0,,2,2,0,0.64,,100,100,0,0,0,0,0,0.366,103.2,116.1,97.7,27.66,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,overload,27.117,40.0,1085,1085,594,491,491,0,0,0,0,0.4525,0.5005,True,,,,4380.1,4425.1,4455.7,4463.7,4235.0,4281.0,4319.0,147.0,152.0,60,60,7,58,4.41,,594,594,0,0,0,0,0,1.824,115.0,116.1,98.1,10.79,
|
||||
c2p32-s-r1,c2p32,2,32,s,192323,1,3,recovery,8.135,20.0,163,163,163,0,0,0,0,0,0,0.0,,False,,,,503.1,652.2,661.3,661.5,357.0,500.0,517.0,148.0,162.0,,58,5,56,4.4,4.288,163,163,0,0,0,0,0,1.214,115.1,116.1,98.1,14.34,
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,drain,4.0,11.0,40,40,40,0,0,0,0,0,0,0.0,,False,3.732,10.717,0.631,11773.0,12337.3,12443.1,12443.1,10715.0,11370.0,11464.0,1072.0,1141.0,,40,2,40,12.33,,40,40,0,0,0,0,0,1.485,329.3,408.4,175.6,119.8,
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,steady,2.239,40.0,90,90,90,0,0,0,0,0,0,0.0,,False,,,,811.5,1625.2,2131.9,2131.9,366.0,653.0,1259.0,459.0,1075.0,,5,3,0,1.29,,90,90,0,0,0,0,0,0.8,325.3,408.4,177.6,373.99,
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,fixed,0.8,40.0,32,32,32,0,0,0,0,0,0,0.0,,False,,,,784.1,993.7,1025.5,1025.5,351.0,566.0,578.0,436.0,470.0,,1,1,0,0.76,,32,32,0,0,0,0,0,0.295,161.6,408.4,177.6,112.22,
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,overload,7.465,40.0,299,299,186,113,113,0,0,0,0,0.3779,0.5485,True,,,,15997.1,17364.9,18960.7,19242.1,14996.0,16230.0,16833.0,1076.0,2203.0,66,63,5,59,17.83,,186,186,0,0,0,0,0,1.272,406.4,408.4,177.9,977.07,
|
||||
c4p4-l-r1,c4p4,4,4,l,9724823,1,4,recovery,2.239,20.0,45,45,45,0,0,0,0,0,0,0.0,,False,,,,924.1,1667.4,2100.7,2100.7,374.0,867.0,1192.0,497.0,1101.0,,57,2,53,16.13,14.8,45,45,0,0,0,0,0,1.076,337.0,408.4,178.4,676.66,
|
||||
c4p4-x-r1,c4p4,4,4,x,119755,1,5,drain,10.0,3.0,20,20,20,0,0,0,0,0,0,0.0,,False,2.766,7.232,0.629,6784.6,8826.2,8908.0,8908.0,5404.0,7428.0,7509.0,1398.0,1617.0,,20,1,20,8.74,,20,20,0,0,0,0,0,1.847,1248.8,1251.8,299.4,104.22,
|
||||
c4p4-x-r1,c4p4,4,4,x,119755,1,5,steady,1.659,40.0,67,67,67,0,0,0,0,0,0,0.0,,False,,,,915.1,1152.1,1194.9,1194.9,255.0,486.0,512.0,662.0,685.0,,2,1,0,1.06,,67,67,0,0,0,0,0,1.068,375.4,1251.8,299.5,113.8,
|
||||
c4p4-x-r1,c4p4,4,4,x,119755,1,5,fixed,0.5,40.0,20,20,20,0,0,0,0,0,0,0.0,,False,,,,932.0,1112.1,1156.8,1156.8,265.0,449.0,485.0,668.0,680.0,,1,0,0,0.79,,20,20,0,0,0,0,0,0.325,375.4,1251.8,299.4,126.56,
|
||||
c4p4-x-r1,c4p4,4,4,x,119755,1,5,overload,5.531,40.0,222,222,167,55,55,0,0,0,0,0.2477,0.5,True,,,,16299.1,21462.2,21509.4,21512.4,14881.0,20042.0,20159.0,1423.0,1493.0,60,60,3,56,20.47,,167,167,0,0,0,0,0,1.921,1032.5,1251.8,299.5,118.99,
|
||||
c4p4-x-r1,c4p4,4,4,x,119755,1,5,recovery,1.659,20.0,34,34,34,0,0,0,0,0,0,0.0,,False,,,,897.5,1137.9,1177.1,1177.1,235.0,479.0,498.0,665.0,682.0,,57,1,53,21.4,20.657,34,34,0,0,0,0,0,1.517,889.7,1251.8,299.4,128.54,
|
||||
c2p2-l-r1,c2p2,2,2,l,9724823,1,6,drain,4.0,11.0,40,40,40,0,0,0,0,0,0,0.0,,False,3.699,10.814,0.625,12327.2,12625.1,12778.8,12778.8,11790.0,12111.0,12294.0,505.0,581.0,,40,1,40,12.38,,40,40,0,0,0,0,0,1.362,245.3,246.5,138.3,487.4,
|
||||
c2p2-l-r1,c2p2,2,2,l,9724823,1,6,steady,2.219,40.0,89,89,89,0,0,0,0,0,0,0.0,,False,,,,843.0,1618.7,1739.4,1739.4,398.0,817.0,1073.0,475.0,1173.0,,4,2,2,1.66,,89,89,0,0,0,0,0,0.782,245.6,246.5,139.2,359.36,
|
||||
c2p2-l-r1,c2p2,2,2,l,9724823,1,6,fixed,0.8,40.0,32,32,32,0,0,0,0,0,0,0.0,,False,,,,756.3,967.4,1005.0,1005.0,324.0,534.0,570.0,433.0,468.0,,1,1,0,0.63,,32,32,0,0,0,0,0,0.302,164.6,246.5,139.2,138.87,
|
||||
c2p2-l-r1,c2p2,2,2,l,9724823,1,6,overload,7.398,40.0,296,296,177,119,119,0,0,0,0,0.402,0.5459,True,,,,16680.1,18223.5,18372.0,18484.5,16179.0,17496.0,17962.0,503.0,1435.0,64,64,8,62,18.23,,177,177,0,0,0,0,0,1.09,245.6,246.5,139.5,899.15,
|
||||
c2p2-l-r1,c2p2,2,2,l,9724823,1,6,recovery,2.219,20.0,45,45,45,0,0,0,0,0,0,0.0,,False,,,,878.2,1607.1,1724.5,1724.5,415.0,796.0,1161.0,445.0,1249.0,,57,1,55,17.76,16.708,45,45,0,0,0,0,0,1.012,245.8,246.5,139.5,492.04,
|
||||
c2p32-x-r1,c2p32,2,32,x,119755,1,7,drain,10.0,3.0,20,20,20,0,0,0,0,0,0,0.0,,False,2.895,6.909,0.646,6047.0,8456.6,8518.6,8518.6,5372.0,7751.0,7812.0,686.0,725.0,,20,1,20,8.43,,20,20,0,0,0,0,0,1.968,664.4,669.3,299.5,105.1,
|
||||
c2p32-x-r1,c2p32,2,32,x,119755,1,7,steady,1.737,40.0,70,70,70,0,0,0,0,0,0,0.0,,False,,,,924.3,1161.7,1175.8,1175.8,263.0,496.0,511.0,663.0,677.0,,2,1,0,1.06,,70,70,0,0,0,0,0,1.122,376.3,669.3,299.5,131.46,
|
||||
c2p32-x-r1,c2p32,2,32,x,119755,1,7,fixed,0.5,40.0,20,20,20,0,0,0,0,0,0,0.0,,False,,,,998.4,1130.6,1156.4,1156.4,337.0,482.0,492.0,662.0,674.0,,1,1,1,0.46,,20,20,0,0,0,0,0,0.325,375.9,669.3,299.5,134.1,
|
||||
c2p32-x-r1,c2p32,2,32,x,119755,1,7,overload,5.79,40.0,232,232,168,64,64,0,0,0,0,0.2759,0.5246,True,,,,16440.3,21621.8,21674.7,21674.8,15685.0,20899.0,20969.0,720.0,747.0,60,60,3,58,20.81,,168,168,0,0,0,0,0,1.897,668.0,669.3,299.5,123.0,
|
||||
c2p32-x-r1,c2p32,2,32,x,119755,1,7,recovery,1.737,20.0,35,35,35,0,0,0,0,0,0,0.0,,False,,,,938.9,1164.2,1165.4,1165.4,272.0,487.0,502.0,673.0,695.0,,58,1,56,21.61,21.154,35,35,0,0,0,0,0,1.548,668.8,669.3,299.5,134.84,
|
||||
c2p2-s-r1,c2p2,2,2,s,192323,1,8,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,13.404,3.73,0.631,4500.8,5211.9,5287.7,5287.7,4353.0,5071.0,5145.0,145.0,155.0,,50,6,50,4.9,,50,50,0,0,0,0,0,1.577,109.1,116.1,97.4,13.47,
|
||||
c2p2-s-r1,c2p2,2,2,s,192323,1,8,steady,8.042,40.0,322,322,322,0,0,0,0,0,0,0.0,,False,,,,499.4,655.2,670.1,674.9,354.0,503.0,524.0,148.0,161.0,,6,5,2,0.66,,322,322,0,0,0,0,0,1.105,114.7,116.1,97.6,14.92,
|
||||
c2p2-s-r1,c2p2,2,2,s,192323,1,8,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,423.9,671.2,691.0,695.6,266.0,508.0,531.0,159.0,168.0,,2,1,0,0.53,,100,100,0,0,0,0,0,0.366,113.6,116.1,98.0,26.92,
|
||||
c2p2-s-r1,c2p2,2,2,s,192323,1,8,overload,26.808,40.0,1073,1073,594,479,479,0,0,0,0,0.4464,0.4964,True,,,,4397.1,4447.5,4453.6,4456.5,4254.0,4303.0,4311.0,146.0,151.0,60,60,7,58,4.44,,594,594,0,0,0,0,0,1.832,115.2,116.1,98.1,10.82,
|
||||
c2p2-s-r1,c2p2,2,2,s,192323,1,8,recovery,8.042,20.0,161,161,161,0,0,0,0,0,0,0.0,,False,,,,496.3,650.5,669.6,674.7,346.0,498.0,515.0,148.0,160.0,,46,4,44,4.37,4.265,161,161,0,0,0,0,0,1.216,115.1,116.1,98.1,14.31,
|
||||
c1p1-x-r1,c1p1,1,1,x,119755,1,9,drain,10.0,3.0,20,20,20,0,0,0,0,0,0,0.0,,False,1.527,13.098,0.624,8449.7,13442.9,13992.0,13992.0,7793.0,12786.0,13348.0,652.0,662.0,,20,4,20,13.41,,20,20,0,0,0,0,0,0.973,374.9,376.3,299.4,102.43,
|
||||
c1p1-x-r1,c1p1,1,1,x,119755,1,9,steady,0.916,40.0,37,37,37,0,0,0,0,0,0,0.0,,False,,,,946.7,1171.7,1182.1,1182.1,286.0,494.0,512.0,672.0,685.0,,2,1,0,1.1,,37,37,0,0,0,0,0,0.6,375.1,376.3,299.4,132.65,
|
||||
c1p1-x-r1,c1p1,1,1,x,119755,1,9,fixed,0.5,40.0,20,20,20,0,0,0,0,0,0,0.0,,False,,,,872.1,1139.5,1160.0,1160.0,219.0,468.0,494.0,660.0,679.0,,1,1,0,1.15,,20,20,0,0,0,0,0,0.328,271.3,376.3,299.4,132.13,
|
||||
c1p1-x-r1,c1p1,1,1,x,119755,1,9,overload,3.054,58.943,181,181,149,32,32,0,0,0,0,0.1768,0.5079,True,,,,25087.3,39231.3,39241.7,39242.0,24438.0,38583.0,38593.0,653.0,665.0,60,60,2,59,29.42,,149,149,0,0,0,0,0,0.972,375.5,376.3,299.5,105.66,
|
||||
c1p1-x-r1,c1p1,1,1,x,119755,1,9,recovery,0.916,20.0,19,19,19,0,0,0,0,0,0,0.0,,False,,,,971.7,1177.8,1177.8,1177.8,314.0,504.0,504.0,674.0,685.0,,59,1,58,39.18,38.675,19,19,0,0,0,0,0,0.852,375.6,376.3,299.5,130.44,
|
||||
c1p1-l-r1,c1p1,1,1,l,9724823,1,10,drain,4.0,11.0,40,40,40,0,0,0,0,0,0,0.0,,False,2.253,17.751,0.627,15126.9,18340.0,18711.7,18711.7,14715.0,17908.0,18276.0,421.0,466.0,,40,1,40,18.59,,40,40,0,0,0,0,0,0.824,162.9,164.7,118.7,123.16,
|
||||
c1p1-l-r1,c1p1,1,1,l,9724823,1,10,steady,1.352,40.0,55,55,55,0,0,0,0,0,0,0.0,,False,,,,756.5,994.4,1043.7,1043.7,331.0,560.0,595.0,437.0,498.0,,2,1,0,0.89,,55,55,0,0,0,0,0,0.493,164.4,164.7,119.0,135.6,
|
||||
c1p1-l-r1,c1p1,1,1,l,9724823,1,10,fixed,0.8,40.0,32,32,32,0,0,0,0,0,0,0.0,,False,,,,749.0,1003.9,1004.9,1004.9,317.0,567.0,592.0,432.0,475.0,,1,1,0,0.76,,32,32,0,0,0,0,0,0.294,159.1,164.7,119.0,142.42,
|
||||
c1p1-l-r1,c1p1,1,1,l,9724823,1,10,overload,4.507,40.0,181,181,148,33,33,0,0,0,0,0.1823,0.5077,True,,,,17253.7,26179.5,26399.0,26605.8,16800.0,25754.0,26184.0,426.0,492.0,62,60,2,59,19.97,,148,148,0,0,0,0,0,0.797,164.0,164.7,119.1,115.25,
|
||||
c1p1-l-r1,c1p1,1,1,l,9724823,1,10,recovery,1.352,20.0,28,28,28,0,0,0,0,0,0,0.0,,False,,,,809.1,1113.7,1126.6,1126.6,319.0,568.0,692.0,437.0,704.0,,59,1,58,26.09,25.362,28,28,0,0,0,0,0,0.68,164.0,164.7,119.1,360.12,
|
||||
c1p1-s-r1,c1p1,1,1,s,192323,1,11,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,7.057,7.085,0.62,6115.2,8439.0,8640.0,8640.0,5980.0,8303.0,8503.0,139.0,146.0,,50,2,50,8.49,,50,50,0,0,0,0,0,0.951,98.6,99.5,95.9,15.46,
|
||||
c1p1-s-r1,c1p1,1,1,s,192323,1,11,steady,4.234,40.0,170,170,170,0,0,0,0,0,0,0.0,,False,,,,502.9,647.2,661.9,669.5,359.0,496.0,520.0,142.0,161.0,,3,2,2,0.65,,170,170,0,0,0,0,0,0.57,98.6,99.5,96.5,20.31,
|
||||
c1p1-s-r1,c1p1,1,1,s,192323,1,11,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,437.1,670.9,685.9,695.8,288.0,513.0,550.0,148.0,165.0,,2,1,1,0.51,,100,100,0,0,0,0,0,0.358,98.5,99.5,96.8,25.58,
|
||||
c1p1-s-r1,c1p1,1,1,s,192323,1,11,overload,14.114,40.0,565,565,339,226,226,0,0,0,0,0.4,0.5011,True,,,,8451.7,8539.0,8547.9,8553.2,8314.0,8402.0,8417.0,140.0,146.0,60,60,5,59,8.49,,339,339,0,0,0,0,0,0.927,98.8,99.5,96.9,9.97,
|
||||
c1p1-s-r1,c1p1,1,1,s,192323,1,11,recovery,4.234,20.0,85,85,85,0,0,0,0,0,0,0.0,,False,,,,495.7,649.8,659.0,659.0,351.0,503.0,516.0,142.0,162.0,,55,2,54,8.42,8.399,85,85,0,0,0,0,0,0.667,98.7,99.5,96.9,22.03,
|
||||
c2p32-l-r1,c2p32,2,32,l,9724823,1,12,drain,4.0,11.0,40,40,40,0,0,0,0,0,0,0.0,,False,3.802,10.519,0.626,11736.0,12305.5,12319.3,12319.3,11241.0,11610.0,11819.0,483.0,639.0,,40,2,40,11.88,,40,40,0,0,0,0,0,1.405,245.4,247.1,138.2,590.02,
|
||||
c2p32-l-r1,c2p32,2,32,l,9724823,1,12,steady,2.281,40.0,92,92,92,0,0,0,0,0,0,0.0,,False,,,,1087.5,1878.1,2093.4,2093.4,481.0,1152.0,1328.0,486.0,1368.0,,4,1,1,1.72,,92,92,0,0,0,0,0,0.814,245.2,247.1,139.6,777.16,
|
||||
c2p32-l-r1,c2p32,2,32,l,9724823,1,12,fixed,0.8,40.0,32,32,32,0,0,0,0,0,0,0.0,,False,,,,708.1,975.9,978.3,978.3,277.0,550.0,551.0,430.0,483.0,,1,1,0,0.68,,32,32,0,0,0,0,0,0.299,164.8,247.1,139.6,133.58,
|
||||
c2p32-l-r1,c2p32,2,32,l,9724823,1,12,overload,7.605,40.0,305,305,189,116,116,0,0,0,0,0.3803,0.5395,True,,,,16339.0,19281.5,19840.6,19930.3,15826.0,18800.0,19449.0,502.0,1430.0,66,63,6,60,17.47,,189,189,0,0,0,0,0,1.171,229.3,247.1,139.9,880.55,
|
||||
c2p32-l-r1,c2p32,2,32,l,9724823,1,12,recovery,2.281,20.0,46,46,46,0,0,0,0,0,0,0.0,,False,,,,1082.6,1952.9,2190.0,2190.0,487.0,1188.0,1701.0,504.0,1312.0,,61,1,59,18.98,17.851,46,46,0,0,0,0,0,1.016,245.9,247.1,139.9,944.87,
|
||||
c2p32-x-r2,c2p32,2,32,x,119755,2,13,drain,10.0,3.0,20,20,20,0,0,0,0,0,0,0.0,,False,2.929,6.829,0.617,5404.2,7709.9,7802.0,7802.0,4722.0,7036.0,7134.0,678.0,701.0,,20,4,20,7.57,,20,20,0,0,0,0,0,1.85,435.6,669.2,299.5,109.68,
|
||||
c2p32-x-r2,c2p32,2,32,x,119755,2,13,steady,1.757,40.0,71,71,71,0,0,0,0,0,0,0.0,,False,,,,945.6,1159.0,1173.7,1173.7,282.0,493.0,507.0,659.0,679.0,,2,1,0,1.06,,71,71,0,0,0,0,0,1.125,376.2,669.2,299.5,131.76,
|
||||
c2p32-x-r2,c2p32,2,32,x,119755,2,13,fixed,0.5,40.0,20,20,20,0,0,0,0,0,0,0.0,,False,,,,937.7,1143.2,1180.0,1180.0,283.0,490.0,510.0,657.0,674.0,,1,0,0,0.93,,20,20,0,0,0,0,0,0.336,375.5,669.2,299.4,135.33,
|
||||
c2p32-x-r2,c2p32,2,32,x,119755,2,13,overload,5.858,40.0,235,235,170,65,65,0,0,0,0,0.2766,0.5285,True,,,,16253.2,21520.5,21565.5,21573.8,15542.0,20792.0,20847.0,716.0,744.0,60,60,2,58,20.83,,170,170,0,0,0,0,0,1.933,668.3,669.2,299.5,120.95,
|
||||
c2p32-x-r2,c2p32,2,32,x,119755,2,13,recovery,1.757,20.0,36,36,36,0,0,0,0,0,0,0.0,,False,,,,936.6,1168.2,1201.2,1201.2,250.0,496.0,509.0,674.0,710.0,,58,1,56,21.46,21.124,36,36,0,0,0,0,0,1.53,669.0,669.2,299.5,132.35,
|
||||
c2p2-s-r2,c2p2,2,2,s,192323,2,14,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,13.499,3.704,0.611,3797.9,4573.9,4637.4,4637.4,3654.0,4427.0,4495.0,145.0,150.0,,50,9,50,4.56,,50,50,0,0,0,0,0,1.725,100.4,116.1,97.2,13.7,
|
||||
c2p2-s-r2,c2p2,2,2,s,192323,2,14,steady,8.099,40.0,324,324,324,0,0,0,0,0,0,0.0,,False,,,,496.9,652.9,665.9,676.1,350.0,500.0,520.0,147.0,160.0,,6,4,3,0.63,,324,324,0,0,0,0,0,1.103,114.8,116.1,97.5,15.3,
|
||||
c2p2-s-r2,c2p2,2,2,s,192323,2,14,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,430.3,653.3,669.3,672.2,272.0,502.0,509.0,160.0,168.0,,2,2,0,0.51,,100,100,0,0,0,0,0,0.371,99.6,116.1,97.8,27.56,
|
||||
c2p2-s-r2,c2p2,2,2,s,192323,2,14,overload,26.998,40.0,1080,1080,590,490,490,0,0,0,0,0.4537,0.5031,True,,,,4415.6,4485.7,4514.3,4516.0,4272.0,4340.0,4373.0,146.0,152.0,60,60,9,58,4.48,,590,590,0,0,0,0,0,1.787,115.1,116.1,98.0,10.57,
|
||||
c2p2-s-r2,c2p2,2,2,s,192323,2,14,recovery,8.099,20.0,162,162,162,0,0,0,0,0,0,0.0,,False,,,,498.6,655.0,674.1,677.4,347.0,503.0,524.0,148.0,162.0,,56,4,54,4.4,4.294,162,162,0,0,0,0,0,1.214,113.7,116.1,98.0,15.23,
|
||||
c4p4-x-r2,c4p4,4,4,x,119755,2,15,drain,10.0,3.0,20,20,20,0,0,0,0,0,0,0.0,,False,2.76,7.246,0.627,6860.6,8889.3,8943.4,8943.4,5453.0,7532.0,7567.0,1401.0,1607.0,,20,1,20,8.68,,20,20,0,0,0,0,0,1.829,1248.6,1251.5,299.4,107.36,
|
||||
c4p4-x-r2,c4p4,4,4,x,119755,2,15,steady,1.656,40.0,67,67,67,0,0,0,0,0,0,0.0,,False,,,,933.4,1170.3,1194.8,1194.8,279.0,493.0,514.0,664.0,683.0,,2,1,0,1.12,,67,67,0,0,0,0,0,1.068,375.8,1251.5,299.5,118.16,
|
||||
c4p4-x-r2,c4p4,4,4,x,119755,2,15,fixed,0.5,40.0,20,20,20,0,0,0,0,0,0,0.0,,False,,,,862.7,1156.4,1160.5,1160.5,196.0,472.0,490.0,660.0,681.0,,1,0,0,0.65,,20,20,0,0,0,0,0,0.336,375.9,1251.5,299.5,133.75,
|
||||
c4p4-x-r2,c4p4,4,4,x,119755,2,15,overload,5.52,40.0,221,221,167,54,54,0,0,0,0,0.2443,0.5047,True,,,,16025.0,21686.6,21774.7,21797.4,14550.0,20249.0,20357.0,1428.0,1499.0,60,60,3,56,20.24,,167,167,0,0,0,0,0,1.929,870.3,1251.5,299.5,120.38,
|
||||
c4p4-x-r2,c4p4,4,4,x,119755,2,15,recovery,1.656,20.0,34,34,34,0,0,0,0,0,0,0.0,,False,,,,946.9,1182.0,1215.8,1215.8,269.0,494.0,508.0,672.0,695.0,,57,1,53,21.71,20.798,34,34,0,0,0,0,0,1.499,927.4,1251.5,299.4,118.01,
|
||||
c1p1-x-r2,c1p1,1,1,x,119755,2,16,drain,10.0,3.0,20,20,20,0,0,0,0,0,0,0.0,,False,1.526,13.106,0.627,9079.2,14057.1,14611.8,14611.8,8425.0,13436.0,13961.0,653.0,666.0,,20,1,20,14.45,,20,20,0,0,0,0,0,0.991,373.5,375.8,299.4,104.68,
|
||||
c1p1-x-r2,c1p1,1,1,x,119755,2,16,steady,0.916,40.0,37,37,37,0,0,0,0,0,0,0.0,,False,,,,958.1,1140.8,1161.7,1161.7,289.0,466.0,487.0,670.0,681.0,,1,1,0,0.91,,37,37,0,0,0,0,0,0.587,374.4,375.8,299.4,133.54,
|
||||
c1p1-x-r2,c1p1,1,1,x,119755,2,16,fixed,0.5,40.0,20,20,20,0,0,0,0,0,0,0.0,,False,,,,875.1,1079.0,1148.7,1148.7,216.0,424.0,493.0,662.0,684.0,,1,0,0,0.69,,20,20,0,0,0,0,0,0.337,375.4,375.8,299.4,134.16,
|
||||
c1p1-x-r2,c1p1,1,1,x,119755,2,16,overload,3.052,58.9765,180,180,148,32,32,0,0,0,0,0.1778,0.5079,True,,,,24880.3,38872.2,38881.2,38888.8,24225.0,38224.0,38237.0,653.0,664.0,60,60,2,59,29.5,,148,148,0,0,0,0,0,0.969,375.4,375.8,299.5,105.33,
|
||||
c1p1-x-r2,c1p1,1,1,x,119755,2,16,recovery,0.916,20.0,19,19,19,0,0,0,0,0,0,0.0,,False,,,,989.9,1178.3,1178.3,1178.3,320.0,504.0,504.0,669.0,682.0,,58,1,57,38.86,38.252,19,19,0,0,0,0,0,0.849,375.2,375.8,299.4,137.05,
|
||||
c2p2-l-r2,c2p2,2,2,l,9724823,2,17,drain,4.0,11.0,40,40,40,0,0,0,0,0,0,0.0,,False,3.924,10.193,0.63,11174.2,11425.5,11449.6,11449.6,10684.0,10935.0,10941.0,503.0,545.0,,40,2,40,11.36,,40,40,0,0,0,0,0,1.469,244.5,246.1,137.4,115.75,
|
||||
c2p2-l-r2,c2p2,2,2,l,9724823,2,17,steady,2.355,40.0,95,95,95,0,0,0,0,0,0,0.0,,False,,,,756.7,1093.3,1183.2,1183.2,335.0,586.0,654.0,474.0,596.0,,3,1,1,1.03,,95,95,0,0,0,0,0,0.847,244.5,246.1,137.7,182.14,
|
||||
c2p2-l-r2,c2p2,2,2,l,9724823,2,17,fixed,0.8,40.0,32,32,32,0,0,0,0,0,0,0.0,,False,,,,755.9,984.0,1011.6,1011.6,317.0,555.0,579.0,432.0,467.0,,1,1,0,0.93,,32,32,0,0,0,0,0,0.296,164.3,246.1,137.7,123.37,
|
||||
c2p2-l-r2,c2p2,2,2,l,9724823,2,17,overload,7.848,40.0,314,314,191,123,123,0,0,0,0,0.3917,0.5146,True,,,,15049.5,15391.5,16014.7,16291.4,14549.0,14888.0,15819.0,502.0,882.0,62,60,6,58,15.26,,191,191,0,0,0,0,0,1.233,244.7,246.1,138.9,874.97,
|
||||
c2p2-l-r2,c2p2,2,2,l,9724823,2,17,recovery,2.355,20.0,48,48,48,0,0,0,0,0,0,0.0,,False,,,,915.9,1275.5,1521.1,1521.1,424.0,783.0,914.0,510.0,654.0,,58,2,56,15.29,15.211,48,48,0,0,0,0,0,1.056,245.1,246.1,138.9,388.19,
|
||||
c2p2-x-r2,c2p2,2,2,x,119755,2,18,drain,10.0,3.0,20,20,20,0,0,0,0,0,0,0.0,,False,2.846,7.027,0.626,6093.1,8546.4,8648.2,8648.2,5406.0,7856.0,7951.0,695.0,731.0,,20,2,20,8.51,,20,20,0,0,0,0,0,1.923,666.4,669.1,299.4,105.93,
|
||||
c2p2-x-r2,c2p2,2,2,x,119755,2,18,steady,1.708,40.0,69,69,69,0,0,0,0,0,0,0.0,,False,,,,919.9,1167.2,1173.2,1173.2,262.0,496.0,510.0,666.0,681.0,,2,1,0,1.06,,69,69,0,0,0,0,0,1.11,375.9,669.1,299.5,131.75,
|
||||
c2p2-x-r2,c2p2,2,2,x,119755,2,18,fixed,0.5,40.0,20,20,20,0,0,0,0,0,0,0.0,,False,,,,947.2,1137.3,1137.5,1137.5,282.0,466.0,488.0,671.0,676.0,,1,0,0,0.92,,20,20,0,0,0,0,0,0.333,238.0,669.1,299.4,133.56,
|
||||
c2p2-x-r2,c2p2,2,2,x,119755,2,18,overload,5.692,40.0,228,228,168,60,60,0,0,0,0,0.2632,0.5172,True,,,,16334.0,21746.8,21796.4,21824.3,15605.0,21019.0,21088.0,724.0,748.0,60,60,3,58,20.97,,168,168,0,0,0,0,0,1.908,668.4,669.1,299.5,121.39,
|
||||
c2p2-x-r2,c2p2,2,2,x,119755,2,18,recovery,1.708,20.0,35,35,35,0,0,0,0,0,0,0.0,,False,,,,986.4,1183.1,1186.8,1186.8,300.0,512.0,518.0,676.0,690.0,,58,1,56,21.66,21.424,35,35,0,0,0,0,0,1.519,668.4,669.1,299.5,133.14,
|
||||
c1p1-l-r2,c1p1,1,1,l,9724823,2,19,drain,4.0,11.0,40,40,40,0,0,0,0,0,0,0.0,,False,2.353,16.998,0.66,14517.8,17680.1,18027.5,18027.5,14097.0,17269.0,17600.0,419.0,456.0,,40,2,40,17.57,,40,40,0,0,0,0,0,0.862,163.7,164.9,118.8,121.74,
|
||||
c1p1-l-r2,c1p1,1,1,l,9724823,2,19,steady,1.412,40.0,57,57,57,0,0,0,0,0,0,0.0,,False,,,,829.5,1290.9,1410.8,1410.8,356.0,613.0,875.0,441.0,495.0,,2,1,0,1.16,,57,57,0,0,0,0,0,0.512,163.9,164.9,119.0,317.92,
|
||||
c1p1-l-r2,c1p1,1,1,l,9724823,2,19,fixed,0.8,40.0,32,32,32,0,0,0,0,0,0,0.0,,False,,,,746.8,972.9,988.3,988.3,318.0,540.0,569.0,428.0,453.0,,1,1,0,0.67,,32,32,0,0,0,0,0,0.302,162.9,164.9,119.0,147.46,
|
||||
c1p1-l-r2,c1p1,1,1,l,9724823,2,19,overload,4.707,40.0,189,189,141,48,48,0,0,0,0,0.254,0.5783,True,,,,20203.4,27898.3,28718.0,28759.3,19121.0,27477.0,28332.0,425.0,550.0,64,62,4,61,22.52,,141,141,0,0,0,0,0,0.737,163.9,164.9,119.3,625.79,
|
||||
c1p1-l-r2,c1p1,1,1,l,9724823,2,19,recovery,1.412,20.0,29,29,29,0,0,0,0,0,0,0.0,,False,,,,833.9,1207.5,1967.9,1967.9,384.0,579.0,1559.0,433.0,520.0,,58,1,57,27.74,25.017,29,29,0,0,0,0,0,0.683,164.1,164.9,119.3,178.88,
|
||||
c1p1-s-r2,c1p1,1,1,s,192323,2,20,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,7.041,7.101,0.621,5511.3,7861.6,8063.3,8063.3,5372.0,7728.0,7926.0,139.0,147.0,,50,12,50,7.61,,50,50,0,0,0,0,0,0.956,97.2,99.8,96.0,13.67,
|
||||
c1p1-s-r2,c1p1,1,1,s,192323,2,20,steady,4.225,40.0,169,169,169,0,0,0,0,0,0,0.0,,False,,,,507.7,649.1,678.1,682.4,366.0,503.0,531.0,141.0,162.0,,3,3,2,0.62,,169,169,0,0,0,0,0,0.578,98.6,99.8,96.3,21.35,
|
||||
c1p1-s-r2,c1p1,1,1,s,192323,2,20,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,445.0,676.8,687.1,691.0,290.0,519.0,529.0,153.0,165.0,,2,2,1,0.63,,100,100,0,0,0,0,0,0.358,98.6,99.8,96.5,27.63,
|
||||
c1p1-s-r2,c1p1,1,1,s,192323,2,20,overload,14.083,40.0,564,564,340,224,224,0,0,0,0,0.3972,0.4978,True,,,,8423.4,8496.5,8527.1,8530.9,8287.0,8360.0,8392.0,139.0,145.0,60,60,4,59,8.51,,340,340,0,0,0,0,0,0.934,98.8,99.8,96.8,9.66,
|
||||
c1p1-s-r2,c1p1,1,1,s,192323,2,20,recovery,4.225,20.0,85,85,85,0,0,0,0,0,0,0.0,,False,,,,518.5,649.1,680.5,680.5,369.0,500.0,529.0,141.0,163.0,,53,2,52,8.43,8.359,85,85,0,0,0,0,0,0.677,99.0,99.8,96.8,20.27,
|
||||
c4p4-l-r2,c4p4,4,4,l,9724823,2,21,drain,4.0,11.0,40,40,40,0,0,0,0,0,0,0.0,,False,4.002,9.996,0.621,11096.8,11633.0,11807.7,11807.7,10146.0,10707.0,10852.0,874.0,1669.0,,40,1,40,11.35,,40,40,0,0,0,0,0,1.604,317.4,407.3,175.1,131.62,
|
||||
c4p4-l-r2,c4p4,4,4,l,9724823,2,21,steady,2.401,40.0,97,97,97,0,0,0,0,0,0,0.0,,False,,,,941.8,1588.2,1996.5,1996.5,387.0,853.0,1112.0,514.0,963.0,,4,2,0,1.14,,97,97,0,0,0,0,0,0.873,244.7,407.3,177.7,493.85,
|
||||
c4p4-l-r2,c4p4,4,4,l,9724823,2,21,fixed,0.8,40.0,32,32,32,0,0,0,0,0,0,0.0,,False,,,,765.9,976.3,999.8,999.8,336.0,541.0,564.0,435.0,476.0,,1,1,0,0.77,,32,32,0,0,0,0,0,0.297,164.4,407.3,177.7,116.91,
|
||||
c4p4-l-r2,c4p4,4,4,l,9724823,2,21,overload,8.004,40.0,321,321,181,140,140,0,0,0,0,0.4361,0.5714,True,,,,14471.0,18019.9,18586.2,18746.3,13410.0,16933.0,17753.0,1000.0,2697.0,68,64,4,60,17.84,,181,181,0,0,0,0,0,1.21,370.7,407.3,177.8,1267.06,
|
||||
c4p4-l-r2,c4p4,4,4,l,9724823,2,21,recovery,2.401,20.0,49,49,49,0,0,0,0,0,0,0.0,,False,,,,918.0,1842.3,2045.3,2045.3,396.0,770.0,1115.0,505.0,934.0,,57,2,52,15.92,13.322,49,49,0,0,0,0,0,1.159,372.9,407.3,178.7,357.7,
|
||||
c4p4-s-r2,c4p4,4,4,s,192323,2,22,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,13.362,3.742,0.627,4506.3,5257.5,5293.6,5293.6,4221.0,4975.0,5115.0,291.0,335.0,,50,2,50,4.96,,50,50,0,0,0,0,0,1.665,138.8,146.1,96.7,13.86,
|
||||
c4p4-s-r2,c4p4,4,4,s,192323,2,22,steady,8.017,40.0,321,321,321,0,0,0,0,0,0,0.0,,False,,,,548.3,774.3,821.7,831.5,280.0,505.0,538.0,284.0,329.0,,7,4,2,0.75,,321,321,0,0,0,0,0,1.212,143.9,146.1,97.5,16.86,
|
||||
c4p4-s-r2,c4p4,4,4,s,192323,2,22,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,443.0,680.9,687.5,688.0,287.0,518.0,526.0,160.0,167.0,,2,2,0,0.63,,100,100,0,0,0,0,0,0.367,114.3,146.1,97.5,28.24,
|
||||
c4p4-s-r2,c4p4,4,4,s,192323,2,22,overload,26.724,40.0,1069,1069,584,485,485,0,0,0,0,0.4537,0.5031,True,,,,4470.7,4538.3,4570.2,4580.5,4177.0,4251.0,4282.0,298.0,311.0,60,60,10,56,4.51,,584,584,0,0,0,0,0,1.948,138.4,146.1,97.8,15.47,
|
||||
c4p4-s-r2,c4p4,4,4,s,192323,2,22,recovery,8.017,20.0,161,161,161,0,0,0,0,0,0,0.0,,False,,,,540.4,769.0,791.2,793.0,281.0,500.0,519.0,283.0,324.0,,48,4,44,4.52,4.36,161,161,0,0,0,0,0,1.328,139.7,146.1,97.8,17.23,
|
||||
c2p32-l-r2,c2p32,2,32,l,9724823,2,23,drain,4.0,11.0,40,40,40,0,0,0,0,0,0,0.0,,False,3.642,10.984,0.622,12573.0,12837.3,12845.7,12845.7,11481.0,12342.0,12370.0,502.0,551.0,,40,2,40,12.81,,40,40,0,0,0,0,0,1.359,244.1,245.6,137.5,116.37,
|
||||
c2p32-l-r2,c2p32,2,32,l,9724823,2,23,steady,2.185,40.0,88,88,88,0,0,0,0,0,0,0.0,,False,,,,831.9,1324.3,1873.3,1873.3,386.0,664.0,1334.0,460.0,608.0,,3,3,0,1.06,,88,88,0,0,0,0,0,0.789,227.3,245.6,137.8,250.25,
|
||||
c2p32-l-r2,c2p32,2,32,l,9724823,2,23,fixed,0.8,40.0,32,32,32,0,0,0,0,0,0,0.0,,False,,,,718.0,980.9,1009.5,1009.5,291.0,546.0,578.0,433.0,472.0,,1,1,0,0.86,,32,32,0,0,0,0,0,0.298,164.5,245.6,137.8,113.04,
|
||||
c2p32-l-r2,c2p32,2,32,l,9724823,2,23,overload,7.283,40.0,292,292,188,104,104,0,0,0,0,0.3562,0.5174,True,,,,16103.2,17658.8,17898.0,18324.6,15526.0,17118.0,17797.0,512.0,1183.0,64,62,6,59,17.33,,188,188,0,0,0,0,0,1.187,244.8,245.6,138.9,849.87,
|
||||
c2p32-l-r2,c2p32,2,32,l,9724823,2,23,recovery,2.185,20.0,44,44,44,0,0,0,0,0,0,0.0,,False,,,,818.9,1119.1,1233.2,1233.2,377.0,646.0,716.0,443.0,573.0,,53,2,51,17.38,15.933,44,44,0,0,0,0,0,1.045,245.2,245.6,138.9,203.53,
|
||||
c2p32-s-r2,c2p32,2,32,s,192323,2,24,drain,25.0,3.0,50,50,50,0,0,0,0,0,0,0.0,,False,13.545,3.691,0.616,3827.7,4602.3,4663.4,4663.4,3683.0,4456.0,4519.0,146.0,151.0,,50,10,50,4.59,,50,50,0,0,0,0,0,1.797,112.6,116.1,97.4,14.49,
|
||||
c2p32-s-r2,c2p32,2,32,s,192323,2,24,steady,8.127,40.0,326,326,326,0,0,0,0,0,0,0.0,,False,,,,503.5,654.8,673.6,692.8,354.0,501.0,547.0,149.0,162.0,,6,4,3,0.63,,326,326,0,0,0,0,0,1.098,115.1,116.1,97.8,14.71,
|
||||
c2p32-s-r2,c2p32,2,32,s,192323,2,24,fixed,2.5,40.0,100,100,100,0,0,0,0,0,0,0.0,,False,,,,429.3,665.9,689.1,703.9,273.0,505.0,539.0,161.0,168.0,,2,2,0,0.67,,100,100,0,0,0,0,0,0.385,113.6,116.1,97.9,26.71,
|
||||
c2p32-s-r2,c2p32,2,32,s,192323,2,24,overload,27.091,40.0,1084,1084,592,492,492,0,0,0,0,0.4539,0.502,True,,,,4403.8,4472.2,4485.2,4487.1,4259.0,4328.0,4343.0,147.0,152.0,60,60,7,58,4.46,,592,592,0,0,0,0,0,1.795,115.2,116.1,98.0,10.86,
|
||||
c2p32-s-r2,c2p32,2,32,s,192323,2,24,recovery,8.127,20.0,163,163,163,0,0,0,0,0,0,0.0,,False,,,,507.7,657.0,665.1,666.7,356.0,499.0,515.0,148.0,161.0,,52,4,50,4.42,4.359,163,163,0,0,0,0,0,1.212,115.2,116.1,98.2,14.83,
|
||||
|
Binary file not shown.
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,40 @@
|
||||
q,1793914087506.5,1793914087508.7,1793914087626.0,202,,5a308166-b42e-4188-a912-c5e6549cce8c,9a65a1fbb253,
|
||||
q,1793914087756.5,1793914087761.5,1793914087843.8,202,,e21c8d74-8ca1-45f0-babe-772f5425768c,768a936eebef,
|
||||
q,1793914088006.5,1793914088008.3,1793914088083.8,202,,40c53fb7-3161-47f3-be15-df4e709b8c17,9a65a1fbb253,
|
||||
q,1793914088256.5,1793914088267.6,1793914088361.2,202,,3631842e-5eb5-49db-b0fe-4faccb256137,768a936eebef,
|
||||
q,1793914088506.5,1793914088514.6,1793914088608.6,202,,821b78f7-0246-4788-b4bc-f0660584f39d,9a65a1fbb253,
|
||||
q,1793914088756.5,1793914088758.0,1793914088847.8,202,,d9d686ee-47ed-4efd-be66-fdd81b5d7832,768a936eebef,
|
||||
q,1793914089006.5,1793914089011.8,1793914089098.4,202,,8d9ed1e3-7d40-4105-9955-6d1527ba6556,9a65a1fbb253,
|
||||
q,1793914089256.5,1793914089265.2,1793914089351.8,202,,1e2a8018-6868-4d9c-b9bc-3b7f81e1679f,768a936eebef,
|
||||
q,1793914089506.5,1793914089510.6,1793914089635.1,202,,805848dd-02d5-4bf2-a546-c77a9a5f6165,9a65a1fbb253,
|
||||
q,1793914089756.5,1793914089757.6,1793914089835.7,202,,1704b183-2502-4953-85de-5ea479b95828,768a936eebef,
|
||||
q,1793914090006.5,1793914090009.6,1793914090092.8,202,,8d642490-e7b7-47c1-9e1e-5d4e44b31312,9a65a1fbb253,
|
||||
q,1793914090256.5,1793914090258.7,1793914090351.6,202,,17f8533b-e2bf-4eb8-b37e-23064cadac98,768a936eebef,
|
||||
q,1793914090506.5,1793914090508.0,1793914090602.6,202,,e6515040-051c-4db9-9122-d4b9226c3bd6,9a65a1fbb253,
|
||||
q,1793914090756.5,1793914090758.2,1793914090832.6,202,,6027d468-7fcc-4230-b388-088793d7578f,768a936eebef,
|
||||
q,1793914091006.5,1793914091018.6,1793914091105.9,202,,6b1465f9-0cf0-4f04-99e6-7a670a9e2096,9a65a1fbb253,
|
||||
q,1793914091256.5,1793914091260.5,1793914091355.3,202,,32660c7a-218c-4e28-89a4-4ac5d90c53f4,768a936eebef,
|
||||
q,1793914091506.5,1793914091508.9,1793914091632.1,202,,bb335d70-295a-41bf-a0bf-25ebdeb259da,9a65a1fbb253,
|
||||
q,1793914091756.5,1793914091757.6,1793914091845.5,202,,fac777e1-1ea6-427b-a97c-7fc0d7e2bc1e,768a936eebef,
|
||||
q,1793914092006.5,1793914092007.0,1793914092089.8,202,,be6cad29-8613-4875-99ae-927ab3141b6f,9a65a1fbb253,
|
||||
q,1793914092256.5,1793914092259.9,1793914092354.3,202,,ad7f7875-1a91-41cf-8bf2-6019042b3701,768a936eebef,
|
||||
q,1793914092506.5,1793914092510.3,1793914092603.6,202,,ffad38b2-1809-4b10-8618-22fb9db74b93,9a65a1fbb253,
|
||||
q,1793914092756.5,1793914092762.5,1793914092854.5,202,,7b379a66-6576-463e-8e72-619c46deb461,768a936eebef,
|
||||
q,1793914093006.5,1793914093007.9,1793914093083.5,202,,dd88e413-42f7-44bf-89d6-81f448a0ec91,9a65a1fbb253,
|
||||
q,1793914093256.5,1793914093264.2,1793914093342.4,202,,f49d510e-4017-44d9-8852-32a62196271b,768a936eebef,
|
||||
q,1793914093506.5,1793914093517.8,1793914093635.4,202,,1c448a36-5ff3-4f48-8ba9-6656fd0d363e,9a65a1fbb253,
|
||||
q,1793914093756.5,1793914093766.5,1793914093852.5,202,,c815b02a-f8de-42a0-a315-be20c9de8e33,768a936eebef,
|
||||
q,1793914094006.5,1793914094017.6,1793914094095.5,202,,12a870ae-105c-42e0-ac33-f350dea246fb,9a65a1fbb253,
|
||||
q,1793914094256.5,1793914094262.3,1793914094353.3,202,,a204aa4e-c522-4452-9b8d-77dc41bb0728,768a936eebef,
|
||||
q,1793914094506.5,1793914094510.0,1793914094592.5,202,,b0d90900-0c38-45d9-8d21-b97a3da265a0,9a65a1fbb253,
|
||||
q,1793914094756.5,1793914094767.6,1793914094847.8,202,,087deb87-e7e7-4550-9253-1d83d3d7e438,768a936eebef,
|
||||
q,1793914095006.5,1793914095017.6,1793914095096.2,202,,5bfb1b40-01ab-4977-b151-8afaa6884a3b,9a65a1fbb253,
|
||||
q,1793914095256.5,1793914095266.2,1793914095366.2,202,,c980019d-736d-463f-b93c-a8f5b852ef19,768a936eebef,
|
||||
q,1793914095506.5,1793914095507.8,1793914095631.9,202,,146d34eb-4ae5-42b7-80c6-1fbb4e778f42,9a65a1fbb253,
|
||||
q,1793914095756.5,1793914095758.2,1793914095845.8,202,,06f1aa0b-a6c7-496d-b50b-218d92f806a8,768a936eebef,
|
||||
q,1793914096006.5,1793914096008.9,1793914096081.9,202,,d142684c-2b4a-4587-ae0c-174f7a6487fc,9a65a1fbb253,
|
||||
q,1793914096256.5,1793914096268.5,1793914096369.8,202,,2cddd77e-365b-49d1-a70d-ac6f12377f5d,768a936eebef,
|
||||
q,1793914096506.5,1793914096510.0,1793914096604.9,202,,39cd9589-96c4-4cff-80c8-71a801433b5e,9a65a1fbb253,
|
||||
q,1793914096756.5,1793914096757.7,1793914096846.8,202,,b2a25818-0470-4637-8654-2e0f63898e16,768a936eebef,
|
||||
q,1793914097006.5,1793914097009.2,1793914097090.7,202,,57a5c5cc-e4f5-46be-b8cc-c1795774a545,9a65a1fbb253,
|
||||
q,1793914097256.5,1793914097260.5,1793914097350.9,202,,f2546864-1ed9-4889-9352-df5801f792fb,768a936eebef,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 4.0,
|
||||
"duration_s": 11.0,
|
||||
"start": 1793914087.507,
|
||||
"end": 1793914097.507,
|
||||
"finished": 1793914097.351,
|
||||
"scheduled": 40,
|
||||
"sent": 40,
|
||||
"accepted": 40,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 40
|
||||
},
|
||||
"upload_p50_ms": 87.96,
|
||||
"upload_p95_ms": 123.16,
|
||||
"upload_max_ms": 124.49,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.13,
|
||||
"log": "/tmp/c1p1-l-r1-drain.log"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
q,1793914158850.5,1793914158856.1,1793914158969.7,202,,1d3a3996-a1ef-4b41-9325-693d3e286f98,768a936eebef,
|
||||
q,1793914160100.5,1793914160108.4,1793914160192.1,202,,c2a87351-95fe-46f0-9467-594ae6d8766d,9a65a1fbb253,
|
||||
q,1793914161350.5,1793914161359.1,1793914161464.4,202,,9ad73bcf-2082-43c3-9300-f4ed1ab4d09a,768a936eebef,
|
||||
q,1793914162600.5,1793914162605.0,1793914162738.0,202,,42d8d5d7-5cc3-47f4-8513-6e82a8a2dcb4,9a65a1fbb253,
|
||||
q,1793914163850.5,1793914163863.3,1793914163973.1,202,,c62aa28c-ed5f-4bc8-94e3-fb60fa3c7546,768a936eebef,
|
||||
q,1793914165100.5,1793914165101.4,1793914165180.3,202,,0259593e-0212-452c-ac9c-216549ad5004,9a65a1fbb253,
|
||||
q,1793914166350.5,1793914166354.7,1793914166454.6,202,,51655fbe-f4cc-48fe-86fb-9b83f9be1e6b,768a936eebef,
|
||||
q,1793914167600.5,1793914167606.1,1793914167728.7,202,,0da28333-0b65-408b-ba9e-d33e593d9923,9a65a1fbb253,
|
||||
q,1793914168850.5,1793914168860.1,1793914168940.0,202,,2e7654c8-0c38-4100-a807-c53330a601c6,768a936eebef,
|
||||
q,1793914170100.5,1793914170103.6,1793914170208.0,202,,d599319d-e4ec-4478-b00c-5f5a38a95720,9a65a1fbb253,
|
||||
q,1793914171350.5,1793914171354.0,1793914171450.1,202,,604f1fc0-7737-4aef-a563-1b79d2f07970,768a936eebef,
|
||||
q,1793914172600.5,1793914172603.6,1793914172734.2,202,,21bb01f6-8869-4653-9db2-321b1d2f5628,9a65a1fbb253,
|
||||
q,1793914173850.5,1793914173863.3,1793914173968.6,202,,bbc216c1-d3fa-46d3-99c9-c1b10d08c9f1,768a936eebef,
|
||||
q,1793914175100.5,1793914175108.7,1793914175218.5,202,,a111346e-b96a-4bb9-823f-c7537a7b72c6,9a65a1fbb253,
|
||||
q,1793914176350.5,1793914176362.3,1793914176475.0,202,,1433eeb5-e303-4d4e-a4da-8461583093cc,768a936eebef,
|
||||
q,1793914177600.5,1793914177608.7,1793914177751.1,202,,510df726-0522-4946-a1ec-8abc82606086,9a65a1fbb253,
|
||||
q,1793914178850.5,1793914178853.2,1793914178958.2,202,,d1329b7f-edf4-4ee9-8ad9-7b5be0147217,768a936eebef,
|
||||
q,1793914180100.5,1793914180103.6,1793914180197.1,202,,93a62187-833d-407e-8506-e52f99f89838,9a65a1fbb253,
|
||||
q,1793914181350.5,1793914181354.7,1793914181464.9,202,,1d869ffc-d464-446f-969a-90494404f51e,768a936eebef,
|
||||
q,1793914182600.5,1793914182608.3,1793914182717.4,202,,be4aa6e3-74ad-4d03-bbe3-f069095c5707,9a65a1fbb253,
|
||||
q,1793914183850.5,1793914183859.4,1793914183969.0,202,,802a50c0-6cb4-424b-9e9a-50df49ccda8f,768a936eebef,
|
||||
q,1793914185100.5,1793914185102.1,1793914185183.2,202,,6a334490-54d7-4757-86c0-138951cb961a,9a65a1fbb253,
|
||||
q,1793914186350.5,1793914186354.8,1793914186499.6,202,,667dec15-a660-454e-b1f9-7ee21c686ade,768a936eebef,
|
||||
q,1793914187600.5,1793914187612.3,1793914187715.2,202,,282f48a3-02dd-43be-8f1a-1295a9af331f,9a65a1fbb253,
|
||||
q,1793914188850.5,1793914188855.2,1793914188944.8,202,,02a3826a-2000-4782-815c-24ab72d21d4c,768a936eebef,
|
||||
q,1793914190100.5,1793914190112.7,1793914190213.3,202,,784cbf52-e6b7-473d-b925-16c14722472b,9a65a1fbb253,
|
||||
q,1793914191350.5,1793914191364.2,1793914191479.9,202,,5e93c43a-1237-4c11-a5c3-93097ce589bb,768a936eebef,
|
||||
q,1793914192600.5,1793914192611.3,1793914192718.4,202,,9e5ef38f-6d13-4f35-bdb5-f6787f9ef7a8,9a65a1fbb253,
|
||||
q,1793914193850.5,1793914193854.6,1793914193963.1,202,,413e71d7-cf7b-4553-b972-ab3d83e90398,768a936eebef,
|
||||
q,1793914195100.5,1793914195110.4,1793914195219.4,202,,e46f6e93-9f02-4d66-8823-22a87fb3a241,9a65a1fbb253,
|
||||
q,1793914196350.5,1793914196354.1,1793914196456.5,202,,b667ebc3-1675-44a1-ba50-19e6447b3590,768a936eebef,
|
||||
q,1793914197600.5,1793914197608.4,1793914197718.6,202,,b3291604-e78a-41ea-b136-85344e014df1,9a65a1fbb253,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 0.8,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793914158.851,
|
||||
"end": 1793914198.851,
|
||||
"finished": 1793914197.719,
|
||||
"scheduled": 32,
|
||||
"sent": 32,
|
||||
"accepted": 32,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 32
|
||||
},
|
||||
"upload_p50_ms": 107.12,
|
||||
"upload_p95_ms": 142.42,
|
||||
"upload_max_ms": 144.74,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.258,
|
||||
"log": "/tmp/c1p1-l-r1-fixed.log"
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
q,1793914199580.3,1793914199589.9,1793914199669.7,202,,ea3c87f2-39e1-4e9c-b0ae-1fee2cec7d1c,768a936eebef,
|
||||
q,1793914199802.2,1793914199812.7,1793914199903.2,202,,98e19e6d-2f5b-43f1-bdc5-063c73d44f21,9a65a1fbb253,
|
||||
q,1793914200024.1,1793914200025.5,1793914200105.8,202,,87f16673-bc77-46e0-8dba-4be898e51bb8,768a936eebef,
|
||||
q,1793914200245.9,1793914200247.1,1793914200326.6,202,,9c03eeda-2a90-4493-852a-9eda327e9ffd,9a65a1fbb253,
|
||||
q,1793914200467.8,1793914200470.6,1793914200599.0,202,,80bf1371-fb58-4f21-b814-90ba2de38ce8,768a936eebef,
|
||||
q,1793914200689.7,1793914200698.8,1793914200771.4,202,,a4cf6272-efed-44cb-8887-40faa31b3f6e,9a65a1fbb253,
|
||||
q,1793914200911.6,1793914200913.9,1793914201030.1,202,,19abd1d5-0e43-497e-8388-8b635eb391ae,768a936eebef,
|
||||
q,1793914201133.5,1793914201136.1,1793914201218.8,202,,84d29ad2-3b29-474a-9a81-9bc79d757e69,9a65a1fbb253,
|
||||
q,1793914201355.4,1793914201361.3,1793914201443.3,202,,cef5a248-654b-459c-b9c7-920fd8687a80,768a936eebef,
|
||||
q,1793914201577.2,1793914201579.5,1793914201654.9,202,,404355e6-0395-4734-ba69-be7e867b6c02,9a65a1fbb253,
|
||||
q,1793914201799.1,1793914201810.6,1793914201925.8,202,,922d7c02-f79a-445d-8d10-27ecf8ad3a6a,768a936eebef,
|
||||
q,1793914202021.0,1793914202030.6,1793914202115.6,202,,e2b3136e-c239-4585-ac84-e07ff5c77681,9a65a1fbb253,
|
||||
q,1793914202242.9,1793914202243.6,1793914202321.2,202,,348d2a2e-0e0f-4c9e-9c7b-41d644eda6e1,768a936eebef,
|
||||
q,1793914202464.8,1793914202470.6,1793914202544.9,202,,be94e0bf-c62d-4588-8d97-c2799a7ce2d6,9a65a1fbb253,
|
||||
q,1793914202686.7,1793914202698.6,1793914202788.1,202,,6aa9ce20-4664-4808-9615-022dadbc522c,768a936eebef,
|
||||
q,1793914202908.6,1793914202909.6,1793914202987.5,202,,726502cd-44a4-4065-afc2-03a376c63a05,9a65a1fbb253,
|
||||
q,1793914203130.4,1793914203135.1,1793914203210.5,202,,6fbe8878-576c-43d6-a46d-2d7f21dc9556,768a936eebef,
|
||||
q,1793914203352.3,1793914203357.1,1793914203431.6,202,,a4228c8a-a867-478a-8ae0-d9fdb9e636c7,9a65a1fbb253,
|
||||
q,1793914203574.2,1793914203578.6,1793914203779.4,202,,73a255a5-64c9-4ed3-8990-4a64b5ba7f3d,768a936eebef,
|
||||
q,1793914203796.1,1793914203797.4,1793914203875.0,202,,b95417f4-0d4d-4a39-abc8-fa4330000c5d,9a65a1fbb253,
|
||||
q,1793914204018.0,1793914204029.6,1793914204167.0,202,,17f79187-3199-402b-91b7-b2bc1e65b87c,768a936eebef,
|
||||
q,1793914204239.9,1793914204246.4,1793914204324.8,202,,9c821c7e-09dc-492b-8482-128a89baf0b0,9a65a1fbb253,
|
||||
q,1793914204461.8,1793914204463.1,1793914204544.3,202,,0e976230-89ea-4b0d-a4f0-6544996ff9db,768a936eebef,
|
||||
q,1793914204683.6,1793914204691.5,1793914204764.6,202,,9c5702ad-855e-41d5-a0eb-ef637f707d8d,9a65a1fbb253,
|
||||
q,1793914204905.5,1793914204906.9,1793914204981.5,202,,92adbc91-ff9a-4ef0-a2a4-8ed1b130c08c,768a936eebef,
|
||||
q,1793914205127.4,1793914205132.1,1793914205207.7,202,,5bdf42e2-f87c-4785-a7fc-1557ddd4ee6a,9a65a1fbb253,
|
||||
q,1793914205349.3,1793914205351.6,1793914205436.7,202,,1f94f71d-08ea-41bb-875b-c83104c07792,768a936eebef,
|
||||
q,1793914205571.2,1793914205582.6,1793914205656.1,202,,15bd3c5d-c58a-4dd5-9ff9-07cb06887ecb,9a65a1fbb253,
|
||||
q,1793914205793.1,1793914205793.9,1793914205906.1,202,,bc452357-c6d1-494b-9dea-af2ff44fbadc,768a936eebef,
|
||||
q,1793914206014.9,1793914206022.6,1793914206104.5,202,,64ba839c-9fd3-421a-9750-146170ddac34,9a65a1fbb253,
|
||||
q,1793914206236.8,1793914206248.6,1793914206329.2,202,,bc132980-6983-46ac-bac5-e8666d599bbb,768a936eebef,
|
||||
q,1793914206458.7,1793914206460.5,1793914206542.3,202,,8eedfd35-94ca-4bca-90ab-846f2a16197b,9a65a1fbb253,
|
||||
q,1793914206680.6,1793914206683.2,1793914206759.3,202,,3ff9c5e8-70ba-42b6-8924-8730fc3e81d5,768a936eebef,
|
||||
q,1793914206902.5,1793914206904.7,1793914206983.5,202,,7d8ff3e3-a8fb-46b7-a331-52b233cadc36,9a65a1fbb253,
|
||||
q,1793914207124.4,1793914207126.1,1793914207202.5,202,,402e4f79-3f97-4126-85d8-09efe2cf0c4e,768a936eebef,
|
||||
q,1793914207346.3,1793914207349.0,1793914207424.9,202,,e9f80072-16df-4355-a299-b1eba460cc8e,9a65a1fbb253,
|
||||
q,1793914207568.1,1793914207577.1,1793914207652.0,202,,195ea3f6-03f3-4a79-a229-ca6e199a995c,768a936eebef,
|
||||
q,1793914207790.0,1793914207791.0,1793914207874.7,202,,931260aa-9a17-4bce-8b94-c478da3785c2,9a65a1fbb253,
|
||||
q,1793914208011.9,1793914208023.6,1793914208103.6,202,,e830cb84-dda0-4ba0-81ae-31b168e53fb3,768a936eebef,
|
||||
q,1793914208233.8,1793914208236.1,1793914208316.1,202,,ff1c87e4-efa5-4e8a-bf7c-c995ee1afdc2,9a65a1fbb253,
|
||||
q,1793914208455.7,1793914208466.6,1793914208549.0,202,,aa227dd7-d2b8-46c1-8d90-d6895190d689,768a936eebef,
|
||||
q,1793914208677.6,1793914208682.9,1793914208758.6,202,,7855df4d-ec67-4c54-98c3-15654855f2ee,9a65a1fbb253,
|
||||
q,1793914208899.5,1793914208899.9,1793914208975.3,202,,551d87b3-c119-4618-a9be-2fe564e0fc88,768a936eebef,
|
||||
q,1793914209121.3,1793914209123.1,1793914209201.0,202,,30644b4c-76e4-4233-ab5e-33cf5ced1535,9a65a1fbb253,
|
||||
q,1793914209343.2,1793914209351.6,1793914209426.6,202,,65aeaaa8-615f-4c45-9e4b-ca13cb1362a8,768a936eebef,
|
||||
q,1793914209565.1,1793914209571.6,1793914209649.6,202,,17b0d3d3-2526-496c-b493-4b7a5f86e731,9a65a1fbb253,
|
||||
q,1793914209787.0,1793914209791.0,1793914209894.4,202,,827047d6-f7a1-4ec9-bd52-e9c3fcd3cd9a,768a936eebef,
|
||||
q,1793914210008.9,1793914210011.5,1793914210094.1,202,,b4d05a95-56bf-49fe-ba61-ee3e0945f6f1,9a65a1fbb253,
|
||||
q,1793914210230.8,1793914210241.6,1793914210318.9,202,,d892aa6c-e56c-4591-aea6-13dd3c0cdcbf,768a936eebef,
|
||||
q,1793914210452.6,1793914210461.6,1793914210539.2,202,,2d79e1eb-4828-4cc1-9af6-05cb82c9b1ce,9a65a1fbb253,
|
||||
q,1793914210674.5,1793914210678.6,1793914210749.4,202,,cf5f3dd2-8bb3-41e0-9dd2-e8bc70bb39da,768a936eebef,
|
||||
q,1793914210896.4,1793914210897.6,1793914210972.8,202,,29d5df30-0e28-43c2-8290-ff7d1c9b0f3e,9a65a1fbb253,
|
||||
q,1793914211118.3,1793914211126.6,1793914211202.8,202,,db37ad3b-a964-4908-92bc-394809bd4094,768a936eebef,
|
||||
q,1793914211340.2,1793914211343.5,1793914211420.1,202,,348b74c8-9d12-44c6-a377-6b158f4546b1,9a65a1fbb253,
|
||||
q,1793914211562.1,1793914211566.6,1793914211666.1,202,,5b18a010-a128-4050-9d77-47e82f6078bf,768a936eebef,
|
||||
q,1793914211784.0,1793914211785.6,1793914211865.3,202,,7ffe5557-1b25-43d5-a43b-e5523afa9428,9a65a1fbb253,
|
||||
q,1793914212005.8,1793914212006.6,1793914212086.6,202,,d499578e-2bc7-4027-b501-5860ea55d5ca,768a936eebef,
|
||||
q,1793914212227.7,1793914212230.8,1793914212317.8,202,,faa70be1-308a-412c-9b10-90a3b838cf4e,9a65a1fbb253,
|
||||
q,1793914212449.6,1793914212452.5,1793914212533.7,202,,ad6bfcea-b064-495a-a67b-7c612c85cbf1,768a936eebef,
|
||||
q,1793914212671.5,1793914212674.5,1793914212746.6,202,,f2d01b23-9f49-45ca-9446-752ee4146848,9a65a1fbb253,
|
||||
q,1793914212893.4,1793914212897.0,1793914212979.2,202,,710f6915-6047-4806-8723-d1f5755ff9f8,768a936eebef,
|
||||
q,1793914213115.3,1793914213116.6,1793914213193.0,202,,f4adb4b7-377e-4191-a14e-1229c3667a3b,9a65a1fbb253,
|
||||
q,1793914213337.2,1793914213338.8,1793914213414.1,202,,f3e14ab1-94d4-49c9-9cfc-595a7a2eef90,768a936eebef,
|
||||
q,1793914213559.0,1793914213569.6,1793914213647.6,202,,b2ba729e-971e-489b-8dc7-34ace34f8b9f,9a65a1fbb253,
|
||||
q,1793914213780.9,1793914213793.6,1793914213898.0,202,,f94c62a2-db1a-4ca3-a820-4699397d2a30,768a936eebef,
|
||||
q,1793914214002.8,1793914214003.9,1793914214082.5,202,,f6cd9884-cbe6-4eac-b16b-5cc56cf20e9c,9a65a1fbb253,
|
||||
q,1793914214224.7,1793914214229.6,1793914214315.9,202,,892a949e-51cb-4d79-b2ce-1d073aee5573,768a936eebef,
|
||||
q,1793914214446.6,1793914214452.6,1793914214530.7,202,,06c1310f-d407-4f6f-a7d9-a48bd17b575a,9a65a1fbb253,
|
||||
q,1793914214668.5,1793914214670.3,1793914214752.9,202,,a496c633-d035-4de2-bd10-9fd758589b12,768a936eebef,
|
||||
q,1793914214890.3,1793914214895.6,1793914214973.9,202,,4b2537cf-fd56-4a26-bea9-2fd0d35bdc3a,9a65a1fbb253,
|
||||
q,1793914215112.2,1793914215123.6,1793914215197.6,202,,2ce4672b-8cc2-4819-b6ea-c2748b0d2d50,768a936eebef,
|
||||
q,1793914215334.1,1793914215336.0,1793914215408.3,202,,0b1d79f6-6a86-4232-a2f2-67dbb8b35b02,9a65a1fbb253,
|
||||
q,1793914215556.0,1793914215561.1,1793914215633.4,202,,8f21b643-8745-44eb-bd39-3b6fe2516e6a,768a936eebef,
|
||||
q,1793914215777.9,1793914215780.6,1793914215863.2,202,,0a62f19c-cb60-4ebb-8a7e-d4a1d818f67e,9a65a1fbb253,
|
||||
q,1793914215999.8,1793914216000.6,1793914216082.2,202,,1d9ffd1a-1e8d-4a82-81ba-96404add3618,768a936eebef,
|
||||
q,1793914216221.7,1793914216223.3,1793914216304.7,202,,e2fe3159-c26c-4f46-8d79-adfb684d79e0,9a65a1fbb253,
|
||||
q,1793914216443.5,1793914216452.6,1793914216529.7,202,,15be21b6-c7ec-4c40-8430-e367a30e590c,768a936eebef,
|
||||
q,1793914216665.4,1793914216667.0,1793914216743.8,202,,0c6d9cb2-95bc-42e8-b53d-069b677ca771,9a65a1fbb253,
|
||||
q,1793914216887.3,1793914216892.6,1793914216969.9,202,,26a90e04-898d-491f-b478-eead78126c64,768a936eebef,
|
||||
q,1793914217109.2,1793914217113.0,1793914217182.8,202,,45a09851-fa4a-4be5-bca2-adaf54d2650a,9a65a1fbb253,
|
||||
q,1793914217331.1,1793914217336.5,1793914217412.6,202,,b4e1ee90-2e56-430b-bbf8-05d6c992512b,768a936eebef,
|
||||
q,1793914217553.0,1793914217555.6,1793914217634.1,202,,8505fdc8-8637-453a-b102-acf23ad0c7d3,9a65a1fbb253,
|
||||
q,1793914217774.9,1793914217775.7,1793914217883.4,202,,05203ba0-96b7-4692-85e1-1e848c62fdc2,768a936eebef,
|
||||
q,1793914217996.7,1793914218000.1,1793914218087.9,202,,aba31ae4-f7b7-4d78-a86e-956df9dd2998,9a65a1fbb253,
|
||||
q,1793914218218.6,1793914218227.0,1793914218304.2,202,,11c98c96-4c13-45a0-8112-e26d1ed6bda3,768a936eebef,
|
||||
q,1793914218440.5,1793914218442.1,1793914218518.5,202,,c4668afc-8373-4fea-afc7-e64e92c1ca89,9a65a1fbb253,
|
||||
q,1793914218662.4,1793914218674.6,1793914218751.1,202,,9ce2e47c-80b4-4b08-837e-b71460a28ae0,768a936eebef,
|
||||
q,1793914218884.3,1793914218886.6,1793914218963.7,202,,7fd5498b-e040-4f80-8d6d-061f35ac74c3,9a65a1fbb253,
|
||||
q,1793914219106.2,1793914219115.4,1793914219190.1,202,,030394ca-1319-491b-98d3-b30547da390f,768a936eebef,
|
||||
q,1793914219328.0,1793914219328.6,1793914219413.0,202,,9ae936d9-f835-48c7-a4d6-258a3dd6f326,9a65a1fbb253,
|
||||
q,1793914219549.9,1793914219560.6,1793914219642.7,202,,37e3ad18-cae3-44c9-8b8a-c5f136f0dc56,768a936eebef,
|
||||
q,1793914219771.8,1793914219773.4,1793914219852.5,202,,37496f4c-3ff8-42dc-b98b-26e40c6f7ab3,9a65a1fbb253,
|
||||
q,1793914219993.7,1793914220001.2,1793914220079.0,202,,9909caef-a4e1-44eb-b9c4-033e78cd4e5a,768a936eebef,
|
||||
q,1793914220215.6,1793914220216.8,1793914220300.8,202,,a57b1791-f9a4-42a6-84a5-286e1c87318a,9a65a1fbb253,
|
||||
q,1793914220437.5,1793914220446.5,1793914220527.5,202,,2bbeb076-e9bb-4c8b-aeaf-93ffe6e72281,768a936eebef,
|
||||
q,1793914220659.4,1793914220663.3,1793914220744.2,202,,3e5fc06f-578f-40ea-adab-397af3ec811d,9a65a1fbb253,
|
||||
q,1793914220881.2,1793914220882.6,1793914220955.1,202,,c4cc883f-62cb-4c6f-85d1-4cf046103ac4,768a936eebef,
|
||||
q,1793914221103.1,1793914221106.6,1793914221182.9,202,,6eaab446-100d-40b6-a709-97d58921aa83,9a65a1fbb253,
|
||||
q,1793914221325.0,1793914221328.4,1793914221401.0,202,,60c4b08c-30ab-4741-b427-00c192ca8fb6,768a936eebef,
|
||||
q,1793914221546.9,1793914221557.6,1793914221632.4,202,,a2c9949c-925c-4713-91fa-f61e8f61cdab,9a65a1fbb253,
|
||||
q,1793914221768.8,1793914221780.2,1793914221885.9,202,,b461cb22-893a-41a3-b5d4-7abb55082f7c,768a936eebef,
|
||||
q,1793914221990.7,1793914221992.3,1793914222067.2,202,,fab39c99-dc95-4bff-8e0c-01cc206f231a,9a65a1fbb253,
|
||||
q,1793914222212.5,1793914222223.6,1793914222304.6,202,,83b45a7f-147e-400b-986a-a23538361e31,768a936eebef,
|
||||
q,1793914222434.4,1793914222435.6,1793914222510.0,202,,1cf6f6e1-80dc-4a31-8638-e2fa59e345d9,9a65a1fbb253,
|
||||
q,1793914222656.3,1793914222667.4,1793914222744.5,202,,f202f9b8-3815-4890-bf32-d42ed8b96ffd,768a936eebef,
|
||||
q,1793914222878.2,1793914222880.6,1793914222956.6,202,,1438b5de-a438-4049-b6e5-027a6ce41c58,9a65a1fbb253,
|
||||
q,1793914223100.1,1793914223102.6,1793914223178.9,202,,c18f3276-ebbc-4e94-b988-f83592816257,768a936eebef,
|
||||
q,1793914223322.0,1793914223324.0,1793914223402.8,202,,238cc8ed-3849-42ac-9905-ad5fe2234478,9a65a1fbb253,
|
||||
q,1793914223543.9,1793914223553.3,1793914223624.9,202,,6497b36b-c08f-48cb-a34f-f856e51cc1b3,768a936eebef,
|
||||
q,1793914223765.7,1793914223768.1,1793914223853.8,202,,fbbe378a-88df-4b39-903e-4fa9f486cec9,9a65a1fbb253,
|
||||
q,1793914223987.6,1793914223991.6,1793914224072.4,202,,6d2a58e3-9a35-4e5f-9aac-45f54c730abc,768a936eebef,
|
||||
q,1793914224209.5,1793914224212.9,1793914224302.1,202,,df73151a-f7b4-423b-b23b-b5c0bca927d4,9a65a1fbb253,
|
||||
q,1793914224431.4,1793914224442.6,1793914224521.4,202,,957938b3-a008-4653-966b-1415c7a713d3,768a936eebef,
|
||||
q,1793914224653.3,1793914224653.9,1793914224737.0,202,,2f1c9ea7-e917-41c8-b82e-2944f0055813,9a65a1fbb253,
|
||||
q,1793914224875.2,1793914224878.6,1793914224955.1,202,,ca2acb29-3ece-458e-99bf-b4d45dbbcae8,768a936eebef,
|
||||
q,1793914225097.1,1793914225098.6,1793914225173.8,202,,760a64af-3c1a-40a9-994e-0920e44d35f8,9a65a1fbb253,
|
||||
q,1793914225318.9,1793914225323.6,1793914225326.4,429,5,,768a936eebef,
|
||||
q,1793914225540.8,1793914225552.6,1793914225630.2,202,,d36acb87-a393-4038-930b-ff29565b0f07,9a65a1fbb253,
|
||||
q,1793914225762.7,1793914225765.8,1793914225768.6,429,5,,768a936eebef,
|
||||
q,1793914225984.6,1793914225987.0,1793914226062.2,202,,0334cc33-70ba-40f9-b073-eb578e8e241a,9a65a1fbb253,
|
||||
q,1793914226206.5,1793914226217.6,1793914226220.3,429,5,,768a936eebef,
|
||||
q,1793914226428.4,1793914226429.9,1793914226511.5,202,,c06ad9a5-2354-4f2e-a236-8a44b00911bd,9a65a1fbb253,
|
||||
q,1793914226650.2,1793914226655.4,1793914226658.2,429,5,,768a936eebef,
|
||||
q,1793914226872.1,1793914226883.6,1793914226960.4,202,,281e8193-f4b1-44e7-990a-f0cdc7275a42,9a65a1fbb253,
|
||||
q,1793914227094.0,1793914227099.4,1793914227101.9,429,5,,768a936eebef,
|
||||
q,1793914227315.9,1793914227318.6,1793914227396.9,202,,5e91f376-8e2f-461a-a02e-a57f9ca8b883,9a65a1fbb253,
|
||||
q,1793914227537.8,1793914227538.6,1793914227541.4,429,5,,768a936eebef,
|
||||
q,1793914227759.7,1793914227763.8,1793914227844.1,202,,320955eb-e871-4f85-8e15-6698afe4a1be,9a65a1fbb253,
|
||||
q,1793914227981.6,1793914227985.6,1793914227988.4,429,5,,768a936eebef,
|
||||
q,1793914228203.4,1793914228204.6,1793914228279.0,202,,3db997e6-a32c-40a2-a45e-4c9c2cd7e502,9a65a1fbb253,
|
||||
q,1793914228425.3,1793914228426.6,1793914228429.7,429,5,,768a936eebef,
|
||||
q,1793914228647.2,1793914228648.6,1793914229224.5,202,,cadcc207-8611-4cc9-b60f-1dd8c7d4bd55,9a65a1fbb253,
|
||||
q,1793914228869.1,1793914228875.3,1793914229224.6,202,,0b83619a-7ea9-45d4-866e-c6c16bc93c48,768a936eebef,
|
||||
q,1793914229091.0,1793914229093.9,1793914229227.2,202,,622914c0-e3e6-4c30-adc3-956d56fe16fe,9a65a1fbb253,
|
||||
q,1793914229312.9,1793914229314.0,1793914229316.8,429,5,,768a936eebef,
|
||||
q,1793914229534.8,1793914229545.6,1793914229548.4,429,5,,9a65a1fbb253,
|
||||
q,1793914229756.6,1793914229768.6,1793914229771.8,429,5,,768a936eebef,
|
||||
q,1793914229978.5,1793914229981.2,1793914229984.1,429,5,,9a65a1fbb253,
|
||||
q,1793914230200.4,1793914230201.6,1793914230275.8,202,,5668ab16-c8a9-405b-9941-d93e79dc8d54,768a936eebef,
|
||||
q,1793914230422.3,1793914230431.3,1793914230434.1,429,5,,9a65a1fbb253,
|
||||
q,1793914230644.2,1793914230644.7,1793914230754.7,202,,39b17f68-2a3c-432a-bff6-788c87bce24a,768a936eebef,
|
||||
q,1793914230866.1,1793914230867.6,1793914230870.7,429,5,,9a65a1fbb253,
|
||||
q,1793914231087.9,1793914231089.6,1793914231169.6,202,,c0f3b973-98d4-477b-8990-69a9d5803010,768a936eebef,
|
||||
q,1793914231309.8,1793914231311.6,1793914231314.7,429,5,,9a65a1fbb253,
|
||||
q,1793914231531.7,1793914231539.4,1793914231618.4,202,,5769a304-8f7f-44cf-b308-0621c5e105fb,768a936eebef,
|
||||
q,1793914231753.6,1793914231756.0,1793914231758.8,429,5,,9a65a1fbb253,
|
||||
q,1793914231975.5,1793914231978.5,1793914232050.5,202,,859202b7-8e79-4b00-8379-fa0c6d6cc2cb,768a936eebef,
|
||||
q,1793914232197.4,1793914232202.0,1793914232281.9,202,,eba96fcb-0fda-4225-aeff-bb122f52a7b3,9a65a1fbb253,
|
||||
q,1793914232419.3,1793914232430.6,1793914232433.3,429,5,,768a936eebef,
|
||||
q,1793914232641.1,1793914232643.9,1793914232756.4,202,,ae1613e5-33fc-4427-9917-979b0e305a1c,9a65a1fbb253,
|
||||
q,1793914232863.0,1793914232870.0,1793914232872.7,429,5,,768a936eebef,
|
||||
q,1793914233084.9,1793914233087.0,1793914233165.5,202,,cec6265f-7d89-422c-b98c-ecdc43009b8d,9a65a1fbb253,
|
||||
q,1793914233306.8,1793914233310.7,1793914233313.5,429,5,,768a936eebef,
|
||||
q,1793914233528.7,1793914233532.1,1793914233606.4,202,,9d0334f7-7465-4aa9-bda5-2f4044ffaff9,9a65a1fbb253,
|
||||
q,1793914233750.6,1793914233752.8,1793914233755.8,429,5,,768a936eebef,
|
||||
q,1793914233972.5,1793914233973.6,1793914234050.9,202,,acee1608-55c9-4e7f-ace8-8370e2f34165,9a65a1fbb253,
|
||||
q,1793914234194.3,1793914234198.1,1793914234201.2,429,5,,768a936eebef,
|
||||
q,1793914234416.2,1793914234419.6,1793914234497.6,202,,b94d4aa2-ed2a-4e01-85ad-130ab52146c5,9a65a1fbb253,
|
||||
q,1793914234638.1,1793914234639.6,1793914234642.9,429,5,,768a936eebef,
|
||||
q,1793914234860.0,1793914234871.7,1793914234876.6,429,5,,9a65a1fbb253,
|
||||
q,1793914235081.9,1793914235085.0,1793914235165.8,202,,54c035c9-005e-4efa-9e11-46deee08750c,768a936eebef,
|
||||
q,1793914235303.8,1793914235314.6,1793914235318.2,429,5,,9a65a1fbb253,
|
||||
q,1793914235525.6,1793914235528.6,1793914235607.7,202,,85da543e-f025-44c0-aa64-e650a6d68af9,768a936eebef,
|
||||
q,1793914235747.5,1793914235753.5,1793914235756.6,429,5,,9a65a1fbb253,
|
||||
q,1793914235969.4,1793914235973.5,1793914236044.5,202,,ca073e64-501a-4a49-8c41-d1e60627f282,768a936eebef,
|
||||
q,1793914236191.3,1793914236193.0,1793914236195.9,429,5,,9a65a1fbb253,
|
||||
q,1793914236413.2,1793914236424.6,1793914236500.2,202,,9eabc651-7683-4303-a7a6-d4448a1f752b,768a936eebef,
|
||||
q,1793914236635.1,1793914236636.8,1793914236640.0,429,5,,9a65a1fbb253,
|
||||
q,1793914236857.0,1793914236867.6,1793914236955.6,202,,cbcdcde0-a222-44ac-88b4-d403cacd372c,768a936eebef,
|
||||
q,1793914237078.8,1793914237081.0,1793914237083.8,429,5,,9a65a1fbb253,
|
||||
q,1793914237300.7,1793914237307.6,1793914237386.6,202,,0f1fb4ed-5f74-4840-b1f8-57e558e328eb,768a936eebef,
|
||||
q,1793914237522.6,1793914237523.6,1793914237526.3,429,5,,9a65a1fbb253,
|
||||
q,1793914237744.5,1793914237748.8,1793914237830.7,202,,528513a2-d9df-4ef4-a0df-9c03749d13c5,768a936eebef,
|
||||
q,1793914237966.4,1793914237967.1,1793914237970.5,429,5,,9a65a1fbb253,
|
||||
q,1793914238188.3,1793914238190.6,1793914238265.6,202,,4bb82a54-c4d5-45b2-9509-6fa5f4c2a7e0,768a936eebef,
|
||||
q,1793914238410.2,1793914238410.8,1793914238483.2,202,,1f951299-75cb-43e5-af82-8c11cee315f4,9a65a1fbb253,
|
||||
q,1793914238632.0,1793914238636.6,1793914238639.3,429,5,,768a936eebef,
|
||||
q,1793914238853.9,1793914238858.8,1793914238939.9,202,,fb9db74c-1eb2-4c20-b055-aaee76a46df5,9a65a1fbb253,
|
||||
q,1793914239075.8,1793914239078.0,1793914239080.8,429,5,,768a936eebef,
|
||||
q,1793914239297.7,1793914239298.8,1793914239303.1,429,5,,9a65a1fbb253,
|
||||
q,1793914239519.6,1793914239519.9,1793914239590.5,202,,0e321f5d-3743-4d79-b984-66ce9945a306,768a936eebef,
|
||||
|
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"rate": 4.50684,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793914199.58,
|
||||
"end": 1793914239.741,
|
||||
"finished": 1793914239.591,
|
||||
"scheduled": 181,
|
||||
"sent": 181,
|
||||
"accepted": 148,
|
||||
"rejected_429": 33,
|
||||
"retry_after": {
|
||||
"5": 33
|
||||
},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 148,
|
||||
"429": 33
|
||||
},
|
||||
"upload_p50_ms": 78.29,
|
||||
"upload_p95_ms": 115.25,
|
||||
"upload_max_ms": 575.89,
|
||||
"peak_inflight": 3,
|
||||
"gen_cpu_s": 0.323,
|
||||
"log": "/tmp/c1p1-l-r1-overload.log"
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
q,1793914266093.3,1793914266104.2,1793914266201.3,202,,42c65662-cc4a-4561-8a72-962e4a1a3026,9a65a1fbb253,
|
||||
q,1793914266832.9,1793914266841.5,1793914266944.9,202,,299d2026-67cb-4baf-9ed1-d7e2b0d9afb8,768a936eebef,
|
||||
q,1793914267572.5,1793914267574.6,1793914267666.7,202,,e458f350-92b9-496b-9478-7b681acbcb95,9a65a1fbb253,
|
||||
q,1793914268312.1,1793914268324.1,1793914268727.6,202,,c90e4642-aaea-414b-a4f8-fc8cd0b2f0c1,768a936eebef,
|
||||
q,1793914269051.8,1793914269062.7,1793914269140.0,202,,13a201e2-02ec-40a7-aa1e-e659231b048f,9a65a1fbb253,
|
||||
q,1793914269791.4,1793914269796.0,1793914269873.1,202,,55ede26f-0269-4e8c-908f-8e602b46fcc0,768a936eebef,
|
||||
q,1793914270531.0,1793914270539.1,1793914270644.4,202,,ee7aaeac-3239-453d-8b61-7fadaf32a975,9a65a1fbb253,
|
||||
q,1793914271270.6,1793914271283.1,1793914271358.9,202,,c61e8fca-ebb3-4c16-9095-1376d2c75e48,768a936eebef,
|
||||
q,1793914272010.2,1793914272011.1,1793914272089.2,202,,c3652aa0-3cbf-4adf-9019-ad899be8d744,9a65a1fbb253,
|
||||
q,1793914272749.8,1793914272752.6,1793914272857.4,202,,8cb2e149-e29d-4c38-84a8-d516da1402d3,768a936eebef,
|
||||
q,1793914273489.5,1793914273491.1,1793914273620.6,202,,7db1afc6-e5fb-462e-8392-ffc96364c9a7,9a65a1fbb253,
|
||||
q,1793914274229.1,1793914274231.2,1793914274591.4,202,,ce54d6eb-fa3c-413a-ac3f-983020545c33,768a936eebef,
|
||||
q,1793914274968.7,1793914274970.5,1793914275120.6,202,,caafbeaa-2411-4277-b980-da1b3ff39433,9a65a1fbb253,
|
||||
q,1793914275708.3,1793914275718.3,1793914275810.8,202,,4de49e9f-5d93-4731-bc38-5a95a1d9e911,768a936eebef,
|
||||
q,1793914276447.9,1793914276449.1,1793914276524.4,202,,07c73dda-da7c-42da-8571-ad82ba9d6901,9a65a1fbb253,
|
||||
q,1793914277187.5,1793914277191.5,1793914277273.6,202,,115cd1fc-baf4-4217-9b78-15cae48f779e,768a936eebef,
|
||||
q,1793914277927.2,1793914277929.6,1793914278010.7,202,,e27e4185-70cd-46e9-bd19-b5a60022aae1,9a65a1fbb253,
|
||||
q,1793914278666.8,1793914278670.5,1793914278741.2,202,,89f837c6-3d93-45cd-b3cb-f4142c9ff51b,768a936eebef,
|
||||
q,1793914279406.4,1793914279409.1,1793914279486.0,202,,2707099a-b489-41d0-b3bb-374ee75e9bae,9a65a1fbb253,
|
||||
q,1793914280146.0,1793914280148.5,1793914280228.2,202,,384256b1-47e0-4965-ac5e-6f16ef69fccd,768a936eebef,
|
||||
q,1793914280885.6,1793914280891.6,1793914280968.2,202,,f8fba439-fc86-4088-a396-e9a7bf2a9fee,9a65a1fbb253,
|
||||
q,1793914281625.3,1793914281626.1,1793914281740.8,202,,46cf924a-b5da-48b0-bb14-70bc1e338bc9,768a936eebef,
|
||||
q,1793914282364.9,1793914282367.1,1793914282471.1,202,,808c30ef-f7e1-4365-a710-e1fe724abe9b,9a65a1fbb253,
|
||||
q,1793914283104.5,1793914283116.1,1793914283256.5,202,,d82001f7-3513-4eca-b1e2-a32426b525bc,768a936eebef,
|
||||
q,1793914283844.1,1793914283849.7,1793914283942.0,202,,b035103b-4926-4c3a-9e50-4eea8b2780cb,9a65a1fbb253,
|
||||
q,1793914284583.7,1793914284590.8,1793914284665.5,202,,35bc6a63-8d96-440b-8802-fcb6c90d26dc,768a936eebef,
|
||||
q,1793914285323.3,1793914285324.5,1793914285405.7,202,,84ddc3d0-ca5f-4753-8ada-182afa43ced6,9a65a1fbb253,
|
||||
q,1793914286063.0,1793914286066.6,1793914286137.8,202,,bd794a67-3495-4eb7-8d69-544564e1b0dd,768a936eebef,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 1.35205,
|
||||
"duration_s": 20.0,
|
||||
"start": 1793914266.093,
|
||||
"end": 1793914286.803,
|
||||
"finished": 1793914286.138,
|
||||
"scheduled": 28,
|
||||
"sent": 28,
|
||||
"accepted": 28,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 28
|
||||
},
|
||||
"upload_p50_ms": 82.16,
|
||||
"upload_p95_ms": 360.12,
|
||||
"upload_max_ms": 403.57,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.083,
|
||||
"log": "/tmp/c1p1-l-r1-recovery.log"
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
q,1793914116995.5,1793914116998.6,1793914117114.9,202,,7cbded1f-ae22-4d77-bf30-18288b7c11c6,9a65a1fbb253,
|
||||
q,1793914117735.1,1793914117740.3,1793914117817.1,202,,75b5b578-07af-4559-9877-81ca259424d0,768a936eebef,
|
||||
q,1793914118474.7,1793914118477.1,1793914118553.2,202,,22773ed8-1f4a-4d48-91c2-fa0992642c1d,9a65a1fbb253,
|
||||
q,1793914119214.4,1793914119222.9,1793914119326.1,202,,abad6fd5-3000-4ecd-b225-ba03f49ebe13,768a936eebef,
|
||||
q,1793914119954.0,1793914119956.2,1793914120091.8,202,,b24e71eb-1168-4e9e-a1a8-f9a28cd1f153,9a65a1fbb253,
|
||||
q,1793914120693.6,1793914120695.0,1793914120789.8,202,,152d1150-ccf3-4f18-a130-9366eee2ca53,768a936eebef,
|
||||
q,1793914121433.2,1793914121444.8,1793914121518.3,202,,8b6f6ec7-fad2-443e-b4e9-a630fb99adcc,9a65a1fbb253,
|
||||
q,1793914122172.8,1793914122178.3,1793914122261.3,202,,afbe917b-a016-458c-9bb1-799124d6729c,768a936eebef,
|
||||
q,1793914122912.4,1793914122915.2,1793914123015.7,202,,fac33f0e-afdc-41f6-bf8a-e9102a4f6211,9a65a1fbb253,
|
||||
q,1793914123652.1,1793914123655.4,1793914123735.0,202,,131f3d03-b326-48eb-bb2b-2caf4d132ff0,768a936eebef,
|
||||
q,1793914124391.7,1793914124399.8,1793914124470.1,202,,eeb30a8d-5976-4145-b7e4-cc65ee95f771,9a65a1fbb253,
|
||||
q,1793914125131.3,1793914125134.1,1793914125216.5,202,,d59fcab6-262b-421f-8138-5452e2915353,768a936eebef,
|
||||
q,1793914125870.9,1793914125874.4,1793914125980.9,202,,fb1d4426-6631-42ca-a374-650b0980117b,9a65a1fbb253,
|
||||
q,1793914126610.5,1793914126613.8,1793914126734.5,202,,bf3b2255-f225-48d4-8491-b26ae574b96d,768a936eebef,
|
||||
q,1793914127350.1,1793914127362.1,1793914127459.2,202,,16a98364-be3a-48e1-83c3-a6fd54615df8,9a65a1fbb253,
|
||||
q,1793914128089.8,1793914128094.8,1793914128169.1,202,,79efc1c1-18ab-4d00-841d-3e4eb2ebd19a,768a936eebef,
|
||||
q,1793914128829.4,1793914128833.7,1793914128919.6,202,,6dc68b7b-9347-4231-9f2d-66b7a4c73992,9a65a1fbb253,
|
||||
q,1793914129569.0,1793914129571.8,1793914129651.7,202,,d04e23f7-403b-4e1e-8508-5338cdc0d1f6,768a936eebef,
|
||||
q,1793914130308.6,1793914130312.0,1793914130394.7,202,,42a87dbc-032e-4002-88bd-ac5bfdf1f1af,9a65a1fbb253,
|
||||
q,1793914131048.2,1793914131050.2,1793914131126.5,202,,cdfa74e2-065e-4988-ab0d-c2ae75f3d98e,768a936eebef,
|
||||
q,1793914131787.9,1793914131792.3,1793914131868.1,202,,b6ab2216-713a-4eca-a416-db09c7d7aa2b,9a65a1fbb253,
|
||||
q,1793914132527.5,1793914132528.8,1793914132607.3,202,,168640b5-ccfc-4c34-99c7-a769da4ca36c,768a936eebef,
|
||||
q,1793914133267.1,1793914133269.1,1793914133349.6,202,,671c09b6-9b8b-43a9-a1ca-da45350c838d,9a65a1fbb253,
|
||||
q,1793914134006.7,1793914134015.3,1793914134103.3,202,,a332cbcb-0948-4c22-8687-29700c2fcadb,768a936eebef,
|
||||
q,1793914134746.3,1793914134752.7,1793914134908.8,202,,c5716d2c-21c7-475f-87bf-75fbc65fd6ef,9a65a1fbb253,
|
||||
q,1793914135485.9,1793914135489.1,1793914135580.1,202,,18b72eb7-bf13-4fc1-a1cb-0ae1c5f09c4d,768a936eebef,
|
||||
q,1793914136225.6,1793914136236.8,1793914136339.6,202,,f515323c-2b55-4da1-a2dd-b4943492e898,9a65a1fbb253,
|
||||
q,1793914136965.2,1793914136968.4,1793914137045.2,202,,ae64100e-fe44-44d8-a6e8-589a0a55c287,768a936eebef,
|
||||
q,1793914137704.8,1793914137707.8,1793914137784.7,202,,836410ae-ee89-4329-82a0-4613f92b7c77,9a65a1fbb253,
|
||||
q,1793914138444.4,1793914138447.2,1793914138520.9,202,,b0e55e5e-bfdd-41d8-9013-adac0183eb57,768a936eebef,
|
||||
q,1793914139184.0,1793914139193.8,1793914139295.6,202,,e2ef7ce1-dee5-4dac-9262-ab4988ae4b01,9a65a1fbb253,
|
||||
q,1793914139923.6,1793914139924.8,1793914139995.9,202,,91777370-070c-4e67-9ba8-de1b31842645,768a936eebef,
|
||||
q,1793914140663.3,1793914140665.6,1793914140760.8,202,,d7283a9c-457f-4be8-abc2-6b95c560195d,9a65a1fbb253,
|
||||
q,1793914141402.9,1793914141407.4,1793914141537.1,202,,715e00b1-cd81-4f1b-84a5-122e1b6b3571,768a936eebef,
|
||||
q,1793914142142.5,1793914142145.5,1793914142227.3,202,,54d59e18-2246-409b-9fc3-8cd0a79837e5,9a65a1fbb253,
|
||||
q,1793914142882.1,1793914142885.8,1793914142958.1,202,,85747e1b-1450-408e-8104-d65fca6f72f9,768a936eebef,
|
||||
q,1793914143621.7,1793914143624.5,1793914143705.2,202,,2295423b-7df6-4ef3-b470-7ac7a3efd92b,9a65a1fbb253,
|
||||
q,1793914144361.4,1793914144371.3,1793914144443.3,202,,de1169a5-0249-45dd-aac7-63c55b6f3cd9,768a936eebef,
|
||||
q,1793914145101.0,1793914145103.2,1793914145183.6,202,,fb5d9994-d0ed-4215-8dcc-64bf880837a4,9a65a1fbb253,
|
||||
q,1793914145840.6,1793914145842.0,1793914145921.0,202,,36e54cf0-02e0-4c92-bdf3-280ffd45c8ab,768a936eebef,
|
||||
q,1793914146580.2,1793914146584.4,1793914146661.7,202,,785da709-f5ba-4e2d-bc31-48d5519edd56,9a65a1fbb253,
|
||||
q,1793914147319.8,1793914147321.8,1793914147397.0,202,,92c71f0d-6283-41ce-8eb6-d0e2955dea5f,768a936eebef,
|
||||
q,1793914148059.4,1793914148060.9,1793914148164.2,202,,c0f551d8-416b-4480-b29c-0542241177cc,9a65a1fbb253,
|
||||
q,1793914148799.1,1793914148803.1,1793914148909.3,202,,9616d8ca-9404-4f9c-89c4-88b6d4cc18c9,768a936eebef,
|
||||
q,1793914149538.7,1793914149540.8,1793914149656.1,202,,37380371-7362-4268-bd79-a39a1de82504,9a65a1fbb253,
|
||||
q,1793914150278.3,1793914150287.8,1793914150378.8,202,,1c0bcaaa-5e06-4318-a15c-32916a4ba5c3,768a936eebef,
|
||||
q,1793914151017.9,1793914151029.8,1793914151105.6,202,,965ff6b4-db17-4d0d-8d5c-036fc40cbd90,9a65a1fbb253,
|
||||
q,1793914151757.5,1793914151764.4,1793914151841.1,202,,7041bf4a-6bce-4768-88e0-1472e66d91f0,768a936eebef,
|
||||
q,1793914152497.1,1793914152509.8,1793914152585.3,202,,90aa7f30-27ca-402c-bc66-5ecf39097e01,9a65a1fbb253,
|
||||
q,1793914153236.8,1793914153248.8,1793914153321.0,202,,02953bd1-b8b6-4bfd-b1d4-c721d8766279,768a936eebef,
|
||||
q,1793914153976.4,1793914153981.8,1793914154053.0,202,,896e93bd-ded8-4646-824b-f49e5d995287,9a65a1fbb253,
|
||||
q,1793914154716.0,1793914154717.8,1793914154845.7,202,,e9b09c26-748a-4714-b702-8ce6eca9557f,768a936eebef,
|
||||
q,1793914155455.6,1793914155462.1,1793914155540.5,202,,9fbe6ded-2289-41c0-bdad-b8e74ba5d980,9a65a1fbb253,
|
||||
q,1793914156195.2,1793914156197.2,1793914156334.2,202,,cdb317ee-14d3-431d-9768-44df864641e7,768a936eebef,
|
||||
q,1793914156934.9,1793914156938.1,1793914157022.2,202,,e7a29672-8ba7-456c-bf2b-a321d393063e,9a65a1fbb253,
|
||||
|
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"rate": 1.35205,
|
||||
"duration_s": 40.0,
|
||||
"start": 1793914116.995,
|
||||
"end": 1793914157.674,
|
||||
"finished": 1793914157.023,
|
||||
"scheduled": 55,
|
||||
"sent": 55,
|
||||
"accepted": 55,
|
||||
"rejected_429": 0,
|
||||
"retry_after": {},
|
||||
"errors_5xx": 0,
|
||||
"other_errors": 0,
|
||||
"dropped": 0,
|
||||
"statuses": {
|
||||
"202": 55
|
||||
},
|
||||
"upload_p50_ms": 80.72,
|
||||
"upload_p95_ms": 135.6,
|
||||
"upload_max_ms": 156.08,
|
||||
"peak_inflight": 1,
|
||||
"gen_cpu_s": 0.146,
|
||||
"log": "/tmp/c1p1-l-r1-steady.log"
|
||||
}
|
||||
@@ -0,0 +1,308 @@
|
||||
2026-11-05T21:28:06.546Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=1 prefetch=1 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-
|
||||
2026-11-05T21:28:06.552Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=1
|
||||
2026-11-05T21:28:06.964Z INFO src.worker [rid=-] worker stopped exit_code=0 cancelled=0 returned=0
|
||||
2026-11-05T21:28:18.247Z INFO src.worker [rid=-] worker started queue=bench.attachment-jobs concurrency=1 prefetch=1 max_attempts=5 job_timeout=60s lease=80s retry=2..120s render_memory=768MiB db=postgres:5432/bench pool=2 s3=http://s3:8333 broker=rabbitmq:5672 fault=-
|
||||
2026-11-05T21:28:18.254Z INFO src.broker [rid=-] broker consuming queue=bench.attachment-jobs prefetch=1
|
||||
2026-11-05T21:28:18.723Z INFO src.services.processing [rid=-] attachment processed attachment_id=5a308166-b42e-4188-a912-c5e6549cce8c job_id=c4425329-c319-46cd-8184-5c69db19603a attempt=1 variants=3 render_peak_kib=107372 took=466ms waited=10701ms
|
||||
2026-11-05T21:28:19.138Z INFO src.services.processing [rid=-] attachment processed attachment_id=e21c8d74-8ca1-45f0-babe-772f5425768c job_id=3973369b-dc2a-4283-a7d1-51cd5d6c7fc2 attempt=1 variants=3 render_peak_kib=121436 took=413ms waited=10944ms
|
||||
2026-11-05T21:28:19.549Z INFO src.services.processing [rid=-] attachment processed attachment_id=40c53fb7-3161-47f3-be15-df4e709b8c17 job_id=30b7cb02-04df-4c24-a6cd-09a74b4fd145 attempt=1 variants=3 render_peak_kib=121436 took=410ms waited=11112ms
|
||||
2026-11-05T21:28:19.964Z INFO src.services.processing [rid=-] attachment processed attachment_id=3631842e-5eb5-49db-b0fe-4faccb256137 job_id=09540fd6-8f95-49e6-873e-6b8b9f7ffd15 attempt=1 variants=3 render_peak_kib=121436 took=414ms waited=11248ms
|
||||
2026-11-05T21:28:20.377Z INFO src.services.processing [rid=-] attachment processed attachment_id=821b78f7-0246-4788-b4bc-f0660584f39d job_id=e9806e51-f4dd-4417-8269-c1be8fde5ad8 attempt=1 variants=3 render_peak_kib=121436 took=411ms waited=11418ms
|
||||
2026-11-05T21:28:20.801Z INFO src.services.processing [rid=-] attachment processed attachment_id=d9d686ee-47ed-4efd-be66-fdd81b5d7832 job_id=1f7945ae-b1cb-43a9-871f-196a54a79f1e attempt=1 variants=3 render_peak_kib=121436 took=422ms waited=11590ms
|
||||
2026-11-05T21:28:21.226Z INFO src.services.processing [rid=-] attachment processed attachment_id=8d9ed1e3-7d40-4105-9955-6d1527ba6556 job_id=8ea25c8f-b7e3-4606-918c-61734ad80a14 attempt=1 variants=3 render_peak_kib=121436 took=424ms waited=11764ms
|
||||
2026-11-05T21:28:21.649Z INFO src.services.processing [rid=-] attachment processed attachment_id=1e2a8018-6868-4d9c-b9bc-3b7f81e1679f job_id=9598b2f1-b347-4437-a24a-813421efe32d attempt=1 variants=3 render_peak_kib=121436 took=421ms waited=11934ms
|
||||
2026-11-05T21:28:22.747Z INFO src.services.processing [rid=-] attachment processed attachment_id=805848dd-02d5-4bf2-a546-c77a9a5f6165 job_id=22f8dc14-11ab-4464-8283-86e729637750 attempt=1 variants=3 render_peak_kib=121436 took=1097ms waited=12116ms
|
||||
2026-11-05T21:28:23.168Z INFO src.services.processing [rid=-] attachment processed attachment_id=1704b183-2502-4953-85de-5ea479b95828 job_id=e510221f-962c-4608-b5ca-42bdedebc8b8 attempt=1 variants=3 render_peak_kib=121436 took=420ms waited=12965ms
|
||||
2026-11-05T21:28:23.600Z INFO src.services.processing [rid=-] attachment processed attachment_id=8d642490-e7b7-47c1-9e1e-5d4e44b31312 job_id=eeab157e-4189-4401-9079-2b752daa4fd3 attempt=1 variants=3 render_peak_kib=121436 took=431ms waited=13134ms
|
||||
2026-11-05T21:28:24.033Z INFO src.services.processing [rid=-] attachment processed attachment_id=17f8533b-e2bf-4eb8-b37e-23064cadac98 job_id=758806c6-a00a-47a3-b103-456090021f7c attempt=1 variants=3 render_peak_kib=121436 took=431ms waited=13310ms
|
||||
2026-11-05T21:28:24.456Z INFO src.services.processing [rid=-] attachment processed attachment_id=e6515040-051c-4db9-9122-d4b9226c3bd6 job_id=27066743-56b1-4752-9f29-8d6834cab8b2 attempt=1 variants=3 render_peak_kib=121436 took=422ms waited=13493ms
|
||||
2026-11-05T21:28:24.874Z INFO src.services.processing [rid=-] attachment processed attachment_id=6027d468-7fcc-4230-b388-088793d7578f job_id=28a008aa-ea0b-4871-b287-1d6f10b046d4 attempt=1 variants=3 render_peak_kib=121436 took=417ms waited=13680ms
|
||||
2026-11-05T21:28:25.293Z INFO src.services.processing [rid=-] attachment processed attachment_id=6b1465f9-0cf0-4f04-99e6-7a670a9e2096 job_id=fbee465e-06d2-4e66-8aaa-0addd35d9f48 attempt=1 variants=3 render_peak_kib=121436 took=417ms waited=13828ms
|
||||
2026-11-05T21:28:25.741Z INFO src.services.processing [rid=-] attachment processed attachment_id=32660c7a-218c-4e28-89a4-4ac5d90c53f4 job_id=79617860-81e6-4e38-93f8-1d76f4f7cae1 attempt=1 variants=3 render_peak_kib=121436 took=446ms waited=14000ms
|
||||
2026-11-05T21:28:26.161Z INFO src.services.processing [rid=-] attachment processed attachment_id=bb335d70-295a-41bf-a0bf-25ebdeb259da job_id=44606326-5197-4bab-af03-8ddf8a376caa attempt=1 variants=3 render_peak_kib=121436 took=419ms waited=14199ms
|
||||
2026-11-05T21:28:26.588Z INFO src.services.processing [rid=-] attachment processed attachment_id=fac777e1-1ea6-427b-a97c-7fc0d7e2bc1e job_id=4c8b6d37-8821-4cbf-9a47-af2abac1b49f attempt=1 variants=3 render_peak_kib=121436 took=426ms waited=14375ms
|
||||
2026-11-05T21:28:27.008Z INFO src.services.processing [rid=-] attachment processed attachment_id=be6cad29-8613-4875-99ae-927ab3141b6f job_id=2f4abe75-648c-49c5-8999-d1afb7ad987a attempt=1 variants=3 render_peak_kib=121436 took=419ms waited=14557ms
|
||||
2026-11-05T21:28:27.424Z INFO src.services.processing [rid=-] attachment processed attachment_id=ad7f7875-1a91-41cf-8bf2-6019042b3701 job_id=3767e0b6-a9bc-4949-8da0-29bc72307cbc attempt=1 variants=3 render_peak_kib=121436 took=414ms waited=14715ms
|
||||
2026-11-05T21:28:27.851Z INFO src.services.processing [rid=-] attachment processed attachment_id=ffad38b2-1809-4b10-8618-22fb9db74b93 job_id=d588210e-5ffc-402d-859d-d63da27a5fe1 attempt=1 variants=3 render_peak_kib=121436 took=426ms waited=14881ms
|
||||
2026-11-05T21:28:28.264Z INFO src.services.processing [rid=-] attachment processed attachment_id=7b379a66-6576-463e-8e72-619c46deb461 job_id=5380fbca-2dfc-44dc-903c-db234572bcf7 attempt=1 variants=3 render_peak_kib=121440 took=412ms waited=15054ms
|
||||
2026-11-05T21:28:28.706Z INFO src.services.processing [rid=-] attachment processed attachment_id=dd88e413-42f7-44bf-89d6-81f448a0ec91 job_id=b9bfce49-6a6a-495b-a505-50a1248b7262 attempt=1 variants=3 render_peak_kib=121440 took=441ms waited=15234ms
|
||||
2026-11-05T21:28:29.132Z INFO src.services.processing [rid=-] attachment processed attachment_id=f49d510e-4017-44d9-8852-32a62196271b job_id=6578f74c-4724-4ac5-bd80-1bc5d9f062ad attempt=1 variants=3 render_peak_kib=121440 took=424ms waited=15424ms
|
||||
2026-11-05T21:28:29.564Z INFO src.services.processing [rid=-] attachment processed attachment_id=1c448a36-5ff3-4f48-8ba9-6656fd0d363e job_id=aaf309e5-f09b-4296-9804-7815a12e5be8 attempt=1 variants=3 render_peak_kib=121440 took=431ms waited=15585ms
|
||||
2026-11-05T21:28:29.995Z INFO src.services.processing [rid=-] attachment processed attachment_id=c815b02a-f8de-42a0-a315-be20c9de8e33 job_id=71f0de98-4855-4961-b54e-4d2a344c540d attempt=1 variants=3 render_peak_kib=121440 took=429ms waited=15770ms
|
||||
2026-11-05T21:28:30.413Z INFO src.services.processing [rid=-] attachment processed attachment_id=12a870ae-105c-42e0-ac33-f350dea246fb job_id=b07ea594-ed84-4c9a-a9e9-8cb1fc439558 attempt=1 variants=3 render_peak_kib=121440 took=417ms waited=15953ms
|
||||
2026-11-05T21:28:30.836Z INFO src.services.processing [rid=-] attachment processed attachment_id=a204aa4e-c522-4452-9b8d-77dc41bb0728 job_id=37dd9c7e-485e-4183-9b71-a49bef9ec1e9 attempt=1 variants=3 render_peak_kib=121472 took=421ms waited=16116ms
|
||||
2026-11-05T21:28:31.252Z INFO src.services.processing [rid=-] attachment processed attachment_id=b0d90900-0c38-45d9-8d21-b97a3da265a0 job_id=118e879f-e3c9-468d-996a-947d7eab5830 attempt=1 variants=3 render_peak_kib=121472 took=415ms waited=16298ms
|
||||
2026-11-05T21:28:31.662Z INFO src.services.processing [rid=-] attachment processed attachment_id=087deb87-e7e7-4550-9253-1d83d3d7e438 job_id=f154bd94-f313-48a9-8dbc-b425df7f552a attempt=1 variants=3 render_peak_kib=121472 took=409ms waited=16461ms
|
||||
2026-11-05T21:28:32.082Z INFO src.services.processing [rid=-] attachment processed attachment_id=5bfb1b40-01ab-4977-b151-8afaa6884a3b job_id=e7b63d5e-ffa7-4cd1-b3a8-7e2bcbd8b5ed attempt=1 variants=3 render_peak_kib=121480 took=418ms waited=16626ms
|
||||
2026-11-05T21:28:32.493Z INFO src.services.processing [rid=-] attachment processed attachment_id=c980019d-736d-463f-b93c-a8f5b852ef19 job_id=720d5927-6f41-43ae-bd73-733d3800185c attempt=1 variants=3 render_peak_kib=121480 took=410ms waited=16780ms
|
||||
2026-11-05T21:28:32.978Z INFO src.services.processing [rid=-] attachment processed attachment_id=146d34eb-4ae5-42b7-80c6-1fbb4e778f42 job_id=cb97cf06-40fd-4461-894d-3cdefc668fe2 attempt=1 variants=3 render_peak_kib=121480 took=484ms waited=16955ms
|
||||
2026-11-05T21:28:33.410Z INFO src.services.processing [rid=-] attachment processed attachment_id=06f1aa0b-a6c7-496d-b50b-218d92f806a8 job_id=7d40e212-9b6c-4bde-8142-73a8cedcf15b attempt=1 variants=3 render_peak_kib=121484 took=429ms waited=17193ms
|
||||
2026-11-05T21:28:33.832Z INFO src.services.processing [rid=-] attachment processed attachment_id=d142684c-2b4a-4587-ae0c-174f7a6487fc job_id=91fbfad5-c291-4477-935a-5909c0604e81 attempt=1 variants=3 render_peak_kib=121512 took=421ms waited=17380ms
|
||||
2026-11-05T21:28:34.253Z INFO src.services.processing [rid=-] attachment processed attachment_id=2cddd77e-365b-49d1-a70d-ac6f12377f5d job_id=8f29ef21-2f74-42bb-b14c-335402a89d03 attempt=1 variants=3 render_peak_kib=121512 took=419ms waited=17524ms
|
||||
2026-11-05T21:28:34.694Z INFO src.services.processing [rid=-] attachment processed attachment_id=39cd9589-96c4-4cff-80c8-71a801433b5e job_id=cd9a2969-dd82-46a5-91eb-a92a50dd8f19 attempt=1 variants=3 render_peak_kib=121536 took=440ms waited=17713ms
|
||||
2026-11-05T21:28:35.130Z INFO src.services.processing [rid=-] attachment processed attachment_id=b2a25818-0470-4637-8654-2e0f63898e16 job_id=7e72eccd-a9c4-453d-9ad9-d0490401b4ce attempt=1 variants=3 render_peak_kib=121548 took=435ms waited=17908ms
|
||||
2026-11-05T21:28:35.568Z INFO src.services.processing [rid=-] attachment processed attachment_id=57a5c5cc-e4f5-46be-b8cc-c1795774a545 job_id=3ad614ec-51fb-48fc-a210-ee41ec9f6e60 attempt=1 variants=3 render_peak_kib=121548 took=436ms waited=18097ms
|
||||
2026-11-05T21:28:36.007Z INFO src.services.processing [rid=-] attachment processed attachment_id=f2546864-1ed9-4889-9352-df5801f792fb job_id=13210c42-64eb-44f1-95a9-83682162fee7 attempt=1 variants=3 render_peak_kib=121548 took=438ms waited=18276ms
|
||||
2026-11-05T21:28:37.991Z INFO src.services.processing [rid=-] attachment processed attachment_id=7cbded1f-ae22-4d77-bf30-18288b7c11c6 job_id=70b3f4e1-5c91-4cd5-be1a-95dde031cb70 attempt=1 variants=3 render_peak_kib=121548 took=437ms waited=527ms
|
||||
2026-11-05T21:28:38.482Z INFO src.services.processing [rid=-] attachment processed attachment_id=75b5b578-07af-4559-9877-81ca259424d0 job_id=9e6de04d-5c1b-4b74-9b3b-7cfd76a2d9af attempt=1 variants=3 render_peak_kib=121548 took=410ms waited=312ms
|
||||
2026-11-05T21:28:38.996Z INFO src.services.processing [rid=-] attachment processed attachment_id=22773ed8-1f4a-4d48-91c2-fa0992642c1d job_id=acdc8e41-d15f-4523-a690-ae6f431be6c3 attempt=1 variants=3 render_peak_kib=121548 took=412ms waited=88ms
|
||||
2026-11-05T21:28:40.121Z INFO src.services.processing [rid=-] attachment processed attachment_id=abad6fd5-3000-4ecd-b225-ba03f49ebe13 job_id=f427b24b-b970-4623-ac38-030119095be7 attempt=1 variants=3 render_peak_kib=121548 took=511ms waited=342ms
|
||||
2026-11-05T21:28:40.542Z INFO src.services.processing [rid=-] attachment processed attachment_id=b24e71eb-1168-4e9e-a1a8-f9a28cd1f153 job_id=713dbbb8-326e-4087-8b1a-b8559af952a6 attempt=1 variants=3 render_peak_kib=121548 took=420ms waited=146ms
|
||||
2026-11-05T21:28:41.603Z INFO src.services.processing [rid=-] attachment processed attachment_id=152d1150-ccf3-4f18-a130-9366eee2ca53 job_id=01dcf2ea-7b1d-4c30-9aea-ddd0ed5de69e attempt=1 variants=3 render_peak_kib=121548 took=453ms waited=423ms
|
||||
2026-11-05T21:28:42.069Z INFO src.services.processing [rid=-] attachment processed attachment_id=8b6f6ec7-fad2-443e-b4e9-a630fb99adcc job_id=bb7e5acc-1628-40f5-a2a4-184f4d166377 attempt=1 variants=3 render_peak_kib=121548 took=412ms waited=192ms
|
||||
2026-11-05T21:28:43.131Z INFO src.services.processing [rid=-] attachment processed attachment_id=afbe917b-a016-458c-9bb1-799124d6729c job_id=0feda5c8-f7a6-46c1-a6b5-091c974c3243 attempt=1 variants=3 render_peak_kib=121548 took=443ms waited=482ms
|
||||
2026-11-05T21:28:43.616Z INFO src.services.processing [rid=-] attachment processed attachment_id=fac33f0e-afdc-41f6-bf8a-e9102a4f6211 job_id=ccb0b2cc-0ed1-40c3-bc94-8e9e490e1a2c attempt=1 variants=3 render_peak_kib=121548 took=422ms waited=259ms
|
||||
2026-11-05T21:28:44.655Z INFO src.services.processing [rid=-] attachment processed attachment_id=131f3d03-b326-48eb-bb2b-2caf4d132ff0 job_id=6de00e66-2d8a-481b-9b79-ed750eba5273 attempt=1 variants=3 render_peak_kib=121548 took=438ms waited=542ms
|
||||
2026-11-05T21:28:45.142Z INFO src.services.processing [rid=-] attachment processed attachment_id=eeb30a8d-5976-4145-b7e4-cc65ee95f771 job_id=5c9dbc2b-2c84-41b3-a52f-88b48aa2950e attempt=1 variants=3 render_peak_kib=121548 took=415ms waited=308ms
|
||||
2026-11-05T21:28:45.651Z INFO src.services.processing [rid=-] attachment processed attachment_id=d59fcab6-262b-421f-8138-5452e2915353 job_id=6faff5aa-c687-4402-a3ee-06f96583cf56 attempt=1 variants=3 render_peak_kib=121548 took=420ms waited=78ms
|
||||
2026-11-05T21:28:46.743Z INFO src.services.processing [rid=-] attachment processed attachment_id=fb1d4426-6631-42ca-a374-650b0980117b job_id=99636445-fa9a-4fc8-a23f-569f587a9d98 attempt=1 variants=3 render_peak_kib=121548 took=498ms waited=349ms
|
||||
2026-11-05T21:28:47.180Z INFO src.services.processing [rid=-] attachment processed attachment_id=bf3b2255-f225-48d4-8491-b26ae574b96d job_id=4b62fd19-bece-4ac9-bb78-ff6dd9fa7df2 attempt=1 variants=3 render_peak_kib=121548 took=430ms waited=116ms
|
||||
2026-11-05T21:28:48.240Z INFO src.services.processing [rid=-] attachment processed attachment_id=16a98364-be3a-48e1-83c3-a6fd54615df8 job_id=c338d6dc-eaec-409b-8ae9-cfedb3852e87 attempt=1 variants=3 render_peak_kib=121548 took=454ms waited=384ms
|
||||
2026-11-05T21:28:48.741Z INFO src.services.processing [rid=-] attachment processed attachment_id=79efc1c1-18ab-4d00-841d-3e4eb2ebd19a job_id=d46c1c24-c205-48e2-918e-0d9b6cfb1500 attempt=1 variants=3 render_peak_kib=121548 took=448ms waited=179ms
|
||||
2026-11-05T21:28:49.761Z INFO src.services.processing [rid=-] attachment processed attachment_id=6dc68b7b-9347-4231-9f2d-66b7a4c73992 job_id=0038ca24-1b8d-4fa4-84da-8cd991209d89 attempt=1 variants=3 render_peak_kib=121548 took=440ms waited=463ms
|
||||
2026-11-05T21:28:50.239Z INFO src.services.processing [rid=-] attachment processed attachment_id=d04e23f7-403b-4e1e-8508-5338cdc0d1f6 job_id=dd96d1fb-c18f-48ea-8349-a1badc8825b0 attempt=1 variants=3 render_peak_kib=121548 took=410ms waited=238ms
|
||||
2026-11-05T21:28:51.291Z INFO src.services.processing [rid=-] attachment processed attachment_id=42a87dbc-032e-4002-88bd-ac5bfdf1f1af job_id=1f31c566-d743-4e28-97a9-89813d73eaff attempt=1 variants=3 render_peak_kib=121548 took=447ms waited=510ms
|
||||
2026-11-05T21:28:51.766Z INFO src.services.processing [rid=-] attachment processed attachment_id=cdfa74e2-065e-4988-ab0d-c2ae75f3d98e job_id=1915b202-71d4-4bdd-8e76-a8e21e041a34 attempt=1 variants=3 render_peak_kib=121548 took=411ms waited=285ms
|
||||
2026-11-05T21:28:52.856Z INFO src.services.processing [rid=-] attachment processed attachment_id=b6ab2216-713a-4eca-a416-db09c7d7aa2b job_id=c23a2484-24c3-4620-8691-032050a689ba attempt=1 variants=3 render_peak_kib=121552 took=484ms waited=560ms
|
||||
2026-11-05T21:28:53.311Z INFO src.services.processing [rid=-] attachment processed attachment_id=168640b5-ccfc-4c34-99c7-a769da4ca36c job_id=4098724a-2b48-4ca8-91ba-6fd11874f70b attempt=1 variants=3 render_peak_kib=121552 took=428ms waited=334ms
|
||||
2026-11-05T21:28:53.808Z INFO src.services.processing [rid=-] attachment processed attachment_id=671c09b6-9b8b-43a9-a1ca-da45350c838d job_id=b5b38a1c-9311-4998-8a44-e7e2e2377f19 attempt=1 variants=3 render_peak_kib=121552 took=418ms waited=100ms
|
||||
2026-11-05T21:28:54.921Z INFO src.services.processing [rid=-] attachment processed attachment_id=a332cbcb-0948-4c22-8687-29700c2fcadb job_id=8ef7e06c-90bb-4bdd-a789-bb35bb72f0a7 attempt=1 variants=3 render_peak_kib=121552 took=509ms waited=367ms
|
||||
2026-11-05T21:28:55.338Z INFO src.services.processing [rid=-] attachment processed attachment_id=c5716d2c-21c7-475f-87bf-75fbc65fd6ef job_id=58dc8250-e22a-4688-8486-00d3a027feb0 attempt=1 variants=3 render_peak_kib=121552 took=416ms waited=150ms
|
||||
2026-11-05T21:28:56.406Z INFO src.services.processing [rid=-] attachment processed attachment_id=18b72eb7-bf13-4fc1-a1cb-0ae1c5f09c4d job_id=f2184e11-815f-49c0-aff1-83292d586dab attempt=1 variants=3 render_peak_kib=121640 took=467ms waited=418ms
|
||||
2026-11-05T21:28:56.888Z INFO src.services.processing [rid=-] attachment processed attachment_id=f515323c-2b55-4da1-a2dd-b4943492e898 job_id=238079e0-8a9d-4aa4-acca-22c5dea2517c attempt=1 variants=3 render_peak_kib=121644 took=419ms waited=213ms
|
||||
2026-11-05T21:28:57.942Z INFO src.services.processing [rid=-] attachment processed attachment_id=ae64100e-fe44-44d8-a6e8-589a0a55c287 job_id=c42fb9f4-7cb6-4d6e-b991-23b9cd322c4d attempt=1 variants=3 render_peak_kib=121652 took=447ms waited=506ms
|
||||
2026-11-05T21:28:58.427Z INFO src.services.processing [rid=-] attachment processed attachment_id=836410ae-ee89-4329-82a0-4613f92b7c77 job_id=efd34b09-f671-4374-bee1-fa2f8b69d640 attempt=1 variants=3 render_peak_kib=121828 took=426ms waited=274ms
|
||||
2026-11-05T21:28:59.466Z INFO src.services.processing [rid=-] attachment processed attachment_id=b0e55e5e-bfdd-41d8-9013-adac0183eb57 job_id=92856cf5-e8ef-4cc1-a42c-a5ecd9584b6a attempt=1 variants=3 render_peak_kib=121828 took=451ms waited=549ms
|
||||
2026-11-05T21:28:59.936Z INFO src.services.processing [rid=-] attachment processed attachment_id=e2ef7ce1-dee5-4dac-9262-ab4988ae4b01 job_id=d327e1ce-5e76-4322-b137-bb0ff0836843 attempt=1 variants=3 render_peak_kib=121828 took=416ms waited=307ms
|
||||
2026-11-05T21:29:00.438Z INFO src.services.processing [rid=-] attachment processed attachment_id=91777370-070c-4e67-9ba8-de1b31842645 job_id=db28ed40-7fab-4a9b-a302-def3e07d5196 attempt=1 variants=3 render_peak_kib=121828 took=407ms waited=85ms
|
||||
2026-11-05T21:29:01.545Z INFO src.services.processing [rid=-] attachment processed attachment_id=d7283a9c-457f-4be8-abc2-6b95c560195d job_id=6e9e9b4c-95fd-4793-bccd-2eb24b4ca5e7 attempt=1 variants=3 render_peak_kib=121828 took=492ms waited=351ms
|
||||
2026-11-05T21:29:02.028Z INFO src.services.processing [rid=-] attachment processed attachment_id=715e00b1-cd81-4f1b-84a5-122e1b6b3571 job_id=6a2a19b5-532f-4e34-b434-5524eb1564e0 attempt=1 variants=3 render_peak_kib=121828 took=466ms waited=136ms
|
||||
2026-11-05T21:29:03.037Z INFO src.services.processing [rid=-] attachment processed attachment_id=54d59e18-2246-409b-9fc3-8cd0a79837e5 job_id=a653e067-b183-4baf-989a-e2c7cfa765b9 attempt=1 variants=3 render_peak_kib=121828 took=451ms waited=422ms
|
||||
2026-11-05T21:29:03.509Z INFO src.services.processing [rid=-] attachment processed attachment_id=85747e1b-1450-408e-8104-d65fca6f72f9 job_id=79b1b345-1f44-412a-bf73-2e6dff3777ab attempt=1 variants=3 render_peak_kib=121828 took=414ms waited=189ms
|
||||
2026-11-05T21:29:04.554Z INFO src.services.processing [rid=-] attachment processed attachment_id=2295423b-7df6-4ef3-b470-7ac7a3efd92b job_id=034d652f-592e-4cbf-b444-264a1f318504 attempt=1 variants=3 render_peak_kib=121828 took=434ms waited=468ms
|
||||
2026-11-05T21:29:05.100Z INFO src.services.processing [rid=-] attachment processed attachment_id=de1169a5-0249-45dd-aac7-63c55b6f3cd9 job_id=39b7892f-4d65-4c8c-8693-01bb71da3a4a attempt=1 variants=3 render_peak_kib=121828 took=473ms waited=236ms
|
||||
2026-11-05T21:29:06.084Z INFO src.services.processing [rid=-] attachment processed attachment_id=fb5d9994-d0ed-4215-8dcc-64bf880837a4 job_id=f65feed0-f634-4e06-94bc-fa13064928c2 attempt=1 variants=3 render_peak_kib=121828 took=440ms waited=523ms
|
||||
2026-11-05T21:29:06.563Z INFO src.services.processing [rid=-] attachment processed attachment_id=36e54cf0-02e0-4c92-bdf3-280ffd45c8ab job_id=1e8ecd22-087c-4980-8722-80eafc34ff00 attempt=1 variants=3 render_peak_kib=121828 took=408ms waited=293ms
|
||||
2026-11-05T21:29:07.599Z INFO src.services.processing [rid=-] attachment processed attachment_id=785da709-f5ba-4e2d-bc31-48d5519edd56 job_id=aaf0366c-5aec-4e78-b1c0-2a38a0e8130c attempt=1 variants=3 render_peak_kib=121828 took=435ms waited=561ms
|
||||
2026-11-05T21:29:08.109Z INFO src.services.processing [rid=-] attachment processed attachment_id=92c71f0d-6283-41ce-8eb6-d0e2955dea5f job_id=7c141f44-4323-43d6-be72-3dfd00e71404 attempt=1 variants=3 render_peak_kib=121828 took=436ms waited=331ms
|
||||
2026-11-05T21:29:08.593Z INFO src.services.processing [rid=-] attachment processed attachment_id=c0f551d8-416b-4480-b29c-0542241177cc job_id=8f0de4c7-50cf-4160-ad76-4df6012270bd attempt=1 variants=3 render_peak_kib=121828 took=413ms waited=98ms
|
||||
2026-11-05T21:29:09.659Z INFO src.services.processing [rid=-] attachment processed attachment_id=9616d8ca-9404-4f9c-89c4-88b6d4cc18c9 job_id=e04ccd36-ab8d-467b-9663-206506e37caa attempt=1 variants=3 render_peak_kib=121828 took=462ms waited=350ms
|
||||
2026-11-05T21:29:10.123Z INFO src.services.processing [rid=-] attachment processed attachment_id=37380371-7362-4268-bd79-a39a1de82504 job_id=29778bd2-1831-4a31-8d06-b0c0530538c5 attempt=1 variants=3 render_peak_kib=121828 took=421ms waited=142ms
|
||||
2026-11-05T21:29:11.155Z INFO src.services.processing [rid=-] attachment processed attachment_id=1c0bcaaa-5e06-4318-a15c-32916a4ba5c3 job_id=cfc4670c-ffd8-4a73-b7d9-5321d1f73961 attempt=1 variants=3 render_peak_kib=121828 took=425ms waited=409ms
|
||||
2026-11-05T21:29:11.677Z INFO src.services.processing [rid=-] attachment processed attachment_id=965ff6b4-db17-4d0d-8d5c-036fc40cbd90 job_id=c30fb800-db3d-4525-8b4a-ac1ffc9c3959 attempt=1 variants=3 render_peak_kib=121832 took=440ms waited=189ms
|
||||
2026-11-05T21:29:12.722Z INFO src.services.processing [rid=-] attachment processed attachment_id=7041bf4a-6bce-4768-88e0-1472e66d91f0 job_id=741eb6b4-b00d-473b-8b13-241722a0b966 attempt=1 variants=3 render_peak_kib=121832 took=449ms waited=484ms
|
||||
2026-11-05T21:29:13.201Z INFO src.services.processing [rid=-] attachment processed attachment_id=90aa7f30-27ca-402c-bc66-5ecf39097e01 job_id=ca8d11fa-65f3-4168-a1ae-a7aec857bb07 attempt=1 variants=3 render_peak_kib=121832 took=422ms waited=250ms
|
||||
2026-11-05T21:29:14.245Z INFO src.services.processing [rid=-] attachment processed attachment_id=02953bd1-b8b6-4bfd-b1d4-c721d8766279 job_id=b94c3bd1-0553-432d-8045-69e4c761f2c1 attempt=1 variants=3 render_peak_kib=121832 took=441ms waited=535ms
|
||||
2026-11-05T21:29:14.799Z INFO src.services.processing [rid=-] attachment processed attachment_id=896e93bd-ded8-4646-824b-f49e5d995287 job_id=ccf80374-ec93-49fe-9482-9d680ca2cde6 attempt=1 variants=3 render_peak_kib=121832 took=481ms waited=317ms
|
||||
2026-11-05T21:29:15.787Z INFO src.services.processing [rid=-] attachment processed attachment_id=e9b09c26-748a-4714-b702-8ce6eca9557f job_id=f9201469-6fd1-4771-81c4-4e0361ebbe7a attempt=1 variants=3 render_peak_kib=121832 took=451ms waited=595ms
|
||||
2026-11-05T21:29:16.342Z INFO src.services.processing [rid=-] attachment processed attachment_id=9fbe6ded-2289-41c0-bdad-b8e74ba5d980 job_id=7a8884bc-c986-48b5-be6e-28b920202be3 attempt=1 variants=3 render_peak_kib=121832 took=495ms waited=365ms
|
||||
2026-11-05T21:29:16.754Z INFO src.services.processing [rid=-] attachment processed attachment_id=cdb317ee-14d3-431d-9768-44df864641e7 job_id=d1402217-0741-4d51-b166-9c72b692b6eb attempt=1 variants=3 render_peak_kib=121832 took=404ms waited=132ms
|
||||
2026-11-05T21:29:17.815Z INFO src.services.processing [rid=-] attachment processed attachment_id=e7a29672-8ba7-456c-bf2b-a321d393063e job_id=d130ba32-0a96-4c1c-9269-e7daf826ed8b attempt=1 variants=3 render_peak_kib=121832 took=442ms waited=410ms
|
||||
2026-11-05T21:29:19.836Z INFO src.services.processing [rid=-] attachment processed attachment_id=1d3a3996-a1ef-4b41-9325-693d3e286f98 job_id=d96f9458-44b1-4c9f-b1f9-482b3d53442b attempt=1 variants=3 render_peak_kib=121832 took=430ms waited=502ms
|
||||
2026-11-05T21:29:20.863Z INFO src.services.processing [rid=-] attachment processed attachment_id=c2a87351-95fe-46f0-9467-594ae6d8766d job_id=00046d09-72be-4de7-8ee8-461b3a36b6c8 attempt=1 variants=3 render_peak_kib=121832 took=433ms waited=293ms
|
||||
2026-11-05T21:29:22.380Z INFO src.services.processing [rid=-] attachment processed attachment_id=9ad73bcf-2082-43c3-9300-f4ed1ab4d09a job_id=fde67459-f65e-4b35-a88d-09ab51179de4 attempt=1 variants=3 render_peak_kib=121832 took=432ms waited=542ms
|
||||
2026-11-05T21:29:23.406Z INFO src.services.processing [rid=-] attachment processed attachment_id=42d8d5d7-5cc3-47f4-8513-6e82a8a2dcb4 job_id=323a7349-b0e8-436c-a9b2-27d0ced87cd9 attempt=1 variants=3 render_peak_kib=121832 took=439ms waited=317ms
|
||||
2026-11-05T21:29:24.386Z INFO src.services.processing [rid=-] attachment processed attachment_id=c62aa28c-ed5f-4bc8-94e3-fb60fa3c7546 job_id=08f19d51-8104-4edf-bedd-9dd91491b50e attempt=1 variants=3 render_peak_kib=121832 took=408ms waited=69ms
|
||||
2026-11-05T21:29:25.960Z INFO src.services.processing [rid=-] attachment processed attachment_id=0259593e-0212-452c-ac9c-216549ad5004 job_id=9bc4feee-f388-4a41-9a36-16b03599ef58 attempt=1 variants=3 render_peak_kib=121832 took=453ms waited=383ms
|
||||
2026-11-05T21:29:26.933Z INFO src.services.processing [rid=-] attachment processed attachment_id=51655fbe-f4cc-48fe-86fb-9b83f9be1e6b job_id=4166a3ef-0ad3-4b11-b2fc-003d2d087133 attempt=1 variants=3 render_peak_kib=121832 took=412ms waited=130ms
|
||||
2026-11-05T21:29:28.500Z INFO src.services.processing [rid=-] attachment processed attachment_id=0da28333-0b65-408b-ba9e-d33e593d9923 job_id=9abfbcb9-3eaf-4666-b235-dea4b5034d1e attempt=1 variants=3 render_peak_kib=121832 took=433ms waited=424ms
|
||||
2026-11-05T21:29:29.493Z INFO src.services.processing [rid=-] attachment processed attachment_id=2e7654c8-0c38-4100-a807-c53330a601c6 job_id=ffdf8ef0-3f7f-4cff-8672-47ee9dbf6500 attempt=1 variants=3 render_peak_kib=121832 took=412ms waited=202ms
|
||||
2026-11-05T21:29:31.044Z INFO src.services.processing [rid=-] attachment processed attachment_id=d599319d-e4ec-4478-b00c-5f5a38a95720 job_id=d42c7012-d372-4d4e-8125-bfd417b170fe attempt=1 variants=3 render_peak_kib=121832 took=433ms waited=465ms
|
||||
2026-11-05T21:29:32.061Z INFO src.services.processing [rid=-] attachment processed attachment_id=604f1fc0-7737-4aef-a563-1b79d2f07970 job_id=c20be6c5-0d9d-44f4-800b-052cd62950ce attempt=1 variants=3 render_peak_kib=121832 took=432ms waited=240ms
|
||||
2026-11-05T21:29:33.593Z INFO src.services.processing [rid=-] attachment processed attachment_id=21bb01f6-8869-4653-9db2-321b1d2f5628 job_id=28ccd0ea-6229-4ade-aa46-221bf3033953 attempt=1 variants=3 render_peak_kib=121832 took=438ms waited=506ms
|
||||
2026-11-05T21:29:34.597Z INFO src.services.processing [rid=-] attachment processed attachment_id=bbc216c1-d3fa-46d3-99c9-c1b10d08c9f1 job_id=8b2ebc61-1e7c-4c63-8793-e4598f0eb8ae attempt=1 variants=3 render_peak_kib=121832 took=423ms waited=264ms
|
||||
2026-11-05T21:29:36.159Z INFO src.services.processing [rid=-] attachment processed attachment_id=a111346e-b96a-4bb9-823f-c7537a7b72c6 job_id=3a398445-c3aa-487f-bbf3-5abb5b4b66bc attempt=1 variants=3 render_peak_kib=121832 took=456ms waited=550ms
|
||||
2026-11-05T21:29:37.154Z INFO src.services.processing [rid=-] attachment processed attachment_id=1433eeb5-e303-4d4e-a4da-8461583093cc job_id=cf115829-3038-4fd6-860a-c6eb855fac9c attempt=1 variants=3 render_peak_kib=121832 took=434ms waited=312ms
|
||||
2026-11-05T21:29:38.660Z INFO src.services.processing [rid=-] attachment processed attachment_id=510df726-0522-4946-a1ec-8abc82606086 job_id=34e32974-7f71-4afb-88bf-1a73257d9575 attempt=1 variants=3 render_peak_kib=121832 took=415ms waited=592ms
|
||||
2026-11-05T21:29:39.725Z INFO src.services.processing [rid=-] attachment processed attachment_id=d1329b7f-edf4-4ee9-8ad9-7b5be0147217 job_id=b7914598-ce86-4701-a68a-6cbea3be3c05 attempt=1 variants=3 render_peak_kib=121832 took=454ms waited=374ms
|
||||
2026-11-05T21:29:40.694Z INFO src.services.processing [rid=-] attachment processed attachment_id=93a62187-833d-407e-8506-e52f99f89838 job_id=d812dd6d-d195-4fbc-a18c-704a5f885105 attempt=1 variants=3 render_peak_kib=121832 took=410ms waited=144ms
|
||||
2026-11-05T21:29:42.289Z INFO src.services.processing [rid=-] attachment processed attachment_id=1d869ffc-d464-446f-969a-90494404f51e job_id=0f473d4b-8a66-4ea8-8f73-6ec29d0aa3da attempt=1 variants=3 render_peak_kib=121832 took=475ms waited=414ms
|
||||
2026-11-05T21:29:43.269Z INFO src.services.processing [rid=-] attachment processed attachment_id=be4aa6e3-74ad-4d03-bbe3-f069095c5707 job_id=bd8caa02-6481-48e9-877c-a458f1528e9d attempt=1 variants=3 render_peak_kib=121832 took=428ms waited=187ms
|
||||
2026-11-05T21:29:44.819Z INFO src.services.processing [rid=-] attachment processed attachment_id=802a50c0-6cb4-424b-9e9a-50df49ccda8f job_id=b7ef4cef-7f11-45ed-b2c0-5ec8655d46fb attempt=1 variants=3 render_peak_kib=121832 took=438ms waited=475ms
|
||||
2026-11-05T21:29:45.823Z INFO src.services.processing [rid=-] attachment processed attachment_id=6a334490-54d7-4757-86c0-138951cb961a job_id=d21759de-9b8f-462a-be3e-edbc3ef70b6e attempt=1 variants=3 render_peak_kib=121832 took=428ms waited=274ms
|
||||
2026-11-05T21:29:47.360Z INFO src.services.processing [rid=-] attachment processed attachment_id=667dec15-a660-454e-b1f9-7ee21c686ade job_id=60d49572-1877-46de-aadd-c3cd66842f2e attempt=1 variants=3 render_peak_kib=121832 took=438ms waited=521ms
|
||||
2026-11-05T21:29:48.358Z INFO src.services.processing [rid=-] attachment processed attachment_id=282f48a3-02dd-43be-8f1a-1295a9af331f job_id=682349d4-46af-4e46-b17f-412bf2c27842 attempt=1 variants=3 render_peak_kib=121832 took=422ms waited=279ms
|
||||
2026-11-05T21:29:49.386Z INFO src.services.processing [rid=-] attachment processed attachment_id=02a3826a-2000-4782-815c-24ab72d21d4c job_id=3a110350-351f-49e4-93ba-86ff6e66923d attempt=1 variants=3 render_peak_kib=121832 took=436ms waited=63ms
|
||||
2026-11-05T21:29:50.918Z INFO src.services.processing [rid=-] attachment processed attachment_id=784cbf52-e6b7-473d-b925-16c14722472b job_id=b5514f96-41b8-404f-8b55-71e1f6a8c92a attempt=1 variants=3 render_peak_kib=121832 took=453ms waited=309ms
|
||||
2026-11-05T21:29:52.400Z INFO src.services.processing [rid=-] attachment processed attachment_id=5e93c43a-1237-4c11-a5c3-93097ce589bb job_id=f74222d0-73a3-44ac-bcc9-98d4d9f18088 attempt=1 variants=3 render_peak_kib=121832 took=423ms waited=567ms
|
||||
2026-11-05T21:29:53.408Z INFO src.services.processing [rid=-] attachment processed attachment_id=9e5ef38f-6d13-4f35-bdb5-f6787f9ef7a8 job_id=1f2570b4-e1b7-4adb-a0ed-93e0612609fe attempt=1 variants=3 render_peak_kib=121832 took=421ms waited=331ms
|
||||
2026-11-05T21:29:54.417Z INFO src.services.processing [rid=-] attachment processed attachment_id=413e71d7-cf7b-4553-b972-ab3d83e90398 job_id=26b3a411-ff2f-433d-839c-2016a07a992e attempt=1 variants=3 render_peak_kib=121832 took=406ms waited=111ms
|
||||
2026-11-05T21:29:56.032Z INFO src.services.processing [rid=-] attachment processed attachment_id=e46f6e93-9f02-4d66-8823-22a87fb3a241 job_id=1345f01d-6b60-479a-9877-d26398513c5d attempt=1 variants=3 render_peak_kib=121832 took=494ms waited=381ms
|
||||
2026-11-05T21:29:56.972Z INFO src.services.processing [rid=-] attachment processed attachment_id=b667ebc3-1675-44a1-ba50-19e6447b3590 job_id=03cc8c3b-9f5e-4c38-b86e-b75cdc0bd8fa attempt=1 variants=3 render_peak_kib=121832 took=424ms waited=148ms
|
||||
2026-11-05T21:29:58.517Z INFO src.services.processing [rid=-] attachment processed attachment_id=b3291604-e78a-41ea-b136-85344e014df1 job_id=69c0b9d6-a554-4fa3-82e8-ed3570e715b8 attempt=1 variants=3 render_peak_kib=121832 took=430ms waited=433ms
|
||||
2026-11-05T21:30:00.598Z INFO src.services.processing [rid=-] attachment processed attachment_id=ea3c87f2-39e1-4e9c-b0ae-1fee2cec7d1c job_id=d09e57c4-1465-449b-854f-1736e059d47f attempt=1 variants=3 render_peak_kib=121832 took=493ms waited=491ms
|
||||
2026-11-05T21:30:01.040Z INFO src.services.processing [rid=-] attachment processed attachment_id=98e19e6d-2f5b-43f1-bdc5-063c73d44f21 job_id=e417e2d9-86c2-46d5-98c3-d81f5590a942 attempt=1 variants=3 render_peak_kib=121832 took=441ms waited=756ms
|
||||
2026-11-05T21:30:01.530Z INFO src.services.processing [rid=-] attachment processed attachment_id=87f16673-bc77-46e0-8dba-4be898e51bb8 job_id=c095ad16-1914-43d7-acbf-3cdaf62573d2 attempt=1 variants=3 render_peak_kib=121832 took=489ms waited=992ms
|
||||
2026-11-05T21:30:01.970Z INFO src.services.processing [rid=-] attachment processed attachment_id=9c03eeda-2a90-4493-852a-9eda327e9ffd job_id=88e57bc0-be59-47bf-9dfa-9b67ba0cfc1a attempt=1 variants=3 render_peak_kib=121832 took=439ms waited=1265ms
|
||||
2026-11-05T21:30:02.395Z INFO src.services.processing [rid=-] attachment processed attachment_id=80bf1371-fb58-4f21-b814-90ba2de38ce8 job_id=cd855b77-0c09-4783-a6d4-2143d976b2f7 attempt=1 variants=3 render_peak_kib=121832 took=423ms waited=1482ms
|
||||
2026-11-05T21:30:02.838Z INFO src.services.processing [rid=-] attachment processed attachment_id=a4cf6272-efed-44cb-8887-40faa31b3f6e job_id=f255df83-0d3e-4536-8298-265e84ad87c4 attempt=1 variants=3 render_peak_kib=121832 took=442ms waited=1677ms
|
||||
2026-11-05T21:30:03.264Z INFO src.services.processing [rid=-] attachment processed attachment_id=19abd1d5-0e43-497e-8388-8b635eb391ae job_id=0b509f4e-ed04-42af-9757-eb7bc5c8f6a7 attempt=1 variants=3 render_peak_kib=121832 took=424ms waited=1907ms
|
||||
2026-11-05T21:30:03.793Z INFO src.services.processing [rid=-] attachment processed attachment_id=84d29ad2-3b29-474a-9a81-9bc79d757e69 job_id=18c42156-a4be-462d-83ae-2549e142812d attempt=1 variants=3 render_peak_kib=121832 took=526ms waited=2109ms
|
||||
2026-11-05T21:30:04.218Z INFO src.services.processing [rid=-] attachment processed attachment_id=cef5a248-654b-459c-b9c7-920fd8687a80 job_id=40ea1ac0-63c0-410b-812f-41d0a4b524dd attempt=1 variants=3 render_peak_kib=121832 took=423ms waited=2414ms
|
||||
2026-11-05T21:30:04.635Z INFO src.services.processing [rid=-] attachment processed attachment_id=404355e6-0395-4734-ba69-be7e867b6c02 job_id=c963abe6-e237-442c-9349-51c49aae9923 attempt=1 variants=3 render_peak_kib=121832 took=415ms waited=2620ms
|
||||
2026-11-05T21:30:05.053Z INFO src.services.processing [rid=-] attachment processed attachment_id=922d7c02-f79a-445d-8d10-27ecf8ad3a6a job_id=02fb4eae-20de-4283-95cd-ea7b5496a286 attempt=1 variants=3 render_peak_kib=121832 took=417ms waited=2806ms
|
||||
2026-11-05T21:30:05.472Z INFO src.services.processing [rid=-] attachment processed attachment_id=e2b3136e-c239-4585-ac84-e07ff5c77681 job_id=7f04c8d7-a7b8-4f25-bd23-7bd297c2d5d4 attempt=1 variants=3 render_peak_kib=121832 took=418ms waited=3005ms
|
||||
2026-11-05T21:30:05.947Z INFO src.services.processing [rid=-] attachment processed attachment_id=348d2a2e-0e0f-4c9e-9c7b-41d644eda6e1 job_id=9b3b44ab-05dc-4283-a484-05d326caee0a attempt=1 variants=3 render_peak_kib=121832 took=474ms waited=3211ms
|
||||
2026-11-05T21:30:06.419Z INFO src.services.processing [rid=-] attachment processed attachment_id=be94e0bf-c62d-4588-8d97-c2799a7ce2d6 job_id=2223afa5-6545-400a-a0e0-07c458848b4c attempt=1 variants=3 render_peak_kib=121832 took=470ms waited=3459ms
|
||||
2026-11-05T21:30:06.872Z INFO src.services.processing [rid=-] attachment processed attachment_id=6aa9ce20-4664-4808-9615-022dadbc522c job_id=8c1bee76-109c-4d94-96ed-089d810e17e3 attempt=1 variants=3 render_peak_kib=121832 took=452ms waited=3703ms
|
||||
2026-11-05T21:30:07.294Z INFO src.services.processing [rid=-] attachment processed attachment_id=726502cd-44a4-4065-afc2-03a376c63a05 job_id=be079ba8-0159-42b4-9110-a8ea88e54dd4 attempt=1 variants=3 render_peak_kib=121832 took=420ms waited=3945ms
|
||||
2026-11-05T21:30:07.744Z INFO src.services.processing [rid=-] attachment processed attachment_id=6fbe8878-576c-43d6-a46d-2d7f21dc9556 job_id=d5c09417-17b3-4ef8-90a4-17b4a347b26d attempt=1 variants=3 render_peak_kib=121832 took=449ms waited=4141ms
|
||||
2026-11-05T21:30:08.170Z INFO src.services.processing [rid=-] attachment processed attachment_id=a4228c8a-a867-478a-8ae0-d9fdb9e636c7 job_id=9e922583-56d1-42f9-a016-77820bbeebdd attempt=1 variants=3 render_peak_kib=121832 took=424ms waited=4368ms
|
||||
2026-11-05T21:30:08.639Z INFO src.services.processing [rid=-] attachment processed attachment_id=73a255a5-64c9-4ed3-8990-4a64b5ba7f3d job_id=511c37da-0fc0-4dac-ad46-0de2a32b71ed attempt=1 variants=3 render_peak_kib=121832 took=467ms waited=4574ms
|
||||
2026-11-05T21:30:09.058Z INFO src.services.processing [rid=-] attachment processed attachment_id=b95417f4-0d4d-4a39-abc8-fa4330000c5d job_id=0995d6de-c8be-4ff3-83be-3065c32790c5 attempt=1 variants=3 render_peak_kib=121832 took=417ms waited=4823ms
|
||||
2026-11-05T21:30:09.530Z INFO src.services.processing [rid=-] attachment processed attachment_id=17f79187-3199-402b-91b7-b2bc1e65b87c job_id=cdd22a64-e6a5-438f-b4fa-dad8db868eff attempt=1 variants=3 render_peak_kib=121832 took=471ms waited=5010ms
|
||||
2026-11-05T21:30:09.955Z INFO src.services.processing [rid=-] attachment processed attachment_id=9c821c7e-09dc-492b-8482-128a89baf0b0 job_id=4edaa1a3-d4ab-4e7c-a991-62c9bda77283 attempt=1 variants=3 render_peak_kib=121832 took=424ms waited=5265ms
|
||||
2026-11-05T21:30:10.388Z INFO src.services.processing [rid=-] attachment processed attachment_id=0e976230-89ea-4b0d-a4f0-6544996ff9db job_id=810ff249-b5f2-4c86-89db-952adb256813 attempt=1 variants=3 render_peak_kib=121832 took=431ms waited=5475ms
|
||||
2026-11-05T21:30:10.832Z INFO src.services.processing [rid=-] attachment processed attachment_id=9c5702ad-855e-41d5-a0eb-ef637f707d8d job_id=bb36fe15-7d77-4a68-b538-b51d8b41926c attempt=1 variants=3 render_peak_kib=121832 took=442ms waited=5679ms
|
||||
2026-11-05T21:30:11.256Z INFO src.services.processing [rid=-] attachment processed attachment_id=92adbc91-ff9a-4ef0-a2a4-8ed1b130c08c job_id=9d9d8cf8-d721-4495-9a95-d7f5a080f1fc attempt=1 variants=3 render_peak_kib=121832 took=422ms waited=5908ms
|
||||
2026-11-05T21:30:11.776Z INFO src.services.processing [rid=-] attachment processed attachment_id=5bdf42e2-f87c-4785-a7fc-1557ddd4ee6a job_id=c82098ac-2571-445b-ad5a-5866877f8391 attempt=1 variants=3 render_peak_kib=121832 took=519ms waited=6105ms
|
||||
2026-11-05T21:30:12.202Z INFO src.services.processing [rid=-] attachment processed attachment_id=1f94f71d-08ea-41bb-875b-c83104c07792 job_id=58fe58cb-f972-4861-9b95-b0dedb6a8394 attempt=1 variants=3 render_peak_kib=121832 took=424ms waited=6407ms
|
||||
2026-11-05T21:30:12.636Z INFO src.services.processing [rid=-] attachment processed attachment_id=15bd3c5d-c58a-4dd5-9ff9-07cb06887ecb job_id=04fbaac4-14ea-4214-9e25-b772d0b54210 attempt=1 variants=3 render_peak_kib=121832 took=433ms waited=6602ms
|
||||
2026-11-05T21:30:13.053Z INFO src.services.processing [rid=-] attachment processed attachment_id=bc452357-c6d1-494b-9dea-af2ff44fbadc job_id=c795f7bc-066f-462c-aa9d-731f7a142a72 attempt=1 variants=3 render_peak_kib=121832 took=415ms waited=6826ms
|
||||
2026-11-05T21:30:13.469Z INFO src.services.processing [rid=-] attachment processed attachment_id=64ba839c-9fd3-421a-9750-146170ddac34 job_id=0622e8b4-b0b7-44b0-aa04-f278ea69c4d6 attempt=1 variants=3 render_peak_kib=121832 took=415ms waited=7011ms
|
||||
2026-11-05T21:30:13.948Z INFO src.services.processing [rid=-] attachment processed attachment_id=bc132980-6983-46ac-bac5-e8666d599bbb job_id=d1713717-88dc-4729-94bc-4de1a2f7a341 attempt=1 variants=3 render_peak_kib=121832 took=478ms waited=7203ms
|
||||
2026-11-05T21:30:14.365Z INFO src.services.processing [rid=-] attachment processed attachment_id=8eedfd35-94ca-4bca-90ab-846f2a16197b job_id=0a6f1a22-2140-4b80-9aac-65e55df94560 attempt=1 variants=3 render_peak_kib=121832 took=415ms waited=7470ms
|
||||
2026-11-05T21:30:14.848Z INFO src.services.processing [rid=-] attachment processed attachment_id=3ff9c5e8-70ba-42b6-8924-8730fc3e81d5 job_id=60e0f402-43d5-4775-b529-ec0a66b1a155 attempt=1 variants=3 render_peak_kib=121832 took=481ms waited=7664ms
|
||||
2026-11-05T21:30:15.284Z INFO src.services.processing [rid=-] attachment processed attachment_id=7d8ff3e3-a8fb-46b7-a331-52b233cadc36 job_id=605d7d16-69c9-4171-bb19-b27d036105f5 attempt=1 variants=3 render_peak_kib=121832 took=435ms waited=7923ms
|
||||
2026-11-05T21:30:15.777Z INFO src.services.processing [rid=-] attachment processed attachment_id=402e4f79-3f97-4126-85d8-09efe2cf0c4e job_id=96e647f7-4a65-49db-9f1d-ff9bb2184b14 attempt=1 variants=3 render_peak_kib=121832 took=492ms waited=8140ms
|
||||
2026-11-05T21:30:16.217Z INFO src.services.processing [rid=-] attachment processed attachment_id=e9f80072-16df-4355-a299-b1eba460cc8e job_id=9a76a3d7-6b18-4556-a933-d0329f4d0d04 attempt=1 variants=3 render_peak_kib=121832 took=438ms waited=8411ms
|
||||
2026-11-05T21:30:16.657Z INFO src.services.processing [rid=-] attachment processed attachment_id=195ea3f6-03f3-4a79-a229-ca6e199a995c job_id=5682afdf-1c47-455d-9ba4-5413ac5543af attempt=1 variants=3 render_peak_kib=121832 took=440ms waited=8622ms
|
||||
2026-11-05T21:30:17.076Z INFO src.services.processing [rid=-] attachment processed attachment_id=931260aa-9a17-4bce-8b94-c478da3785c2 job_id=5e5c0e5a-78ff-46b4-bec9-0333659f1b40 attempt=1 variants=3 render_peak_kib=121832 took=417ms waited=8849ms
|
||||
2026-11-05T21:30:17.498Z INFO src.services.processing [rid=-] attachment processed attachment_id=e830cb84-dda0-4ba0-81ae-31b168e53fb3 job_id=97dbb513-5a81-4bf4-9504-bdf44180e000 attempt=1 variants=3 render_peak_kib=121832 took=421ms waited=9035ms
|
||||
2026-11-05T21:30:17.946Z INFO src.services.processing [rid=-] attachment processed attachment_id=ff1c87e4-efa5-4e8a-bf7c-c995ee1afdc2 job_id=2a60514b-d8bc-4a26-9587-d8c6203f9084 attempt=1 variants=3 render_peak_kib=121832 took=445ms waited=9245ms
|
||||
2026-11-05T21:30:18.371Z INFO src.services.processing [rid=-] attachment processed attachment_id=aa227dd7-d2b8-46c1-8d90-d6895190d689 job_id=e0c1c383-a2fd-47b4-93c9-5e4552945008 attempt=1 variants=3 render_peak_kib=121832 took=423ms waited=9462ms
|
||||
2026-11-05T21:30:18.838Z INFO src.services.processing [rid=-] attachment processed attachment_id=7855df4d-ec67-4c54-98c3-15654855f2ee job_id=048c7d1a-b110-4e10-85bf-d62ca502f2cb attempt=1 variants=3 render_peak_kib=121832 took=465ms waited=9671ms
|
||||
2026-11-05T21:30:19.264Z INFO src.services.processing [rid=-] attachment processed attachment_id=551d87b3-c119-4618-a9be-2fe564e0fc88 job_id=8501a0ab-9c95-4514-a5b7-dd9f020ef02e attempt=1 variants=3 render_peak_kib=121832 took=425ms waited=9920ms
|
||||
2026-11-05T21:30:19.766Z INFO src.services.processing [rid=-] attachment processed attachment_id=30644b4c-76e4-4233-ab5e-33cf5ced1535 job_id=3790dd44-8426-4e25-a8d6-8e5969776fec attempt=1 variants=3 render_peak_kib=121832 took=502ms waited=10122ms
|
||||
2026-11-05T21:30:20.198Z INFO src.services.processing [rid=-] attachment processed attachment_id=65aeaaa8-615f-4c45-9e4b-ca13cb1362a8 job_id=3461bfe8-a20b-43f7-9966-cf7a89388c20 attempt=1 variants=3 render_peak_kib=121832 took=430ms waited=10397ms
|
||||
2026-11-05T21:30:20.633Z INFO src.services.processing [rid=-] attachment processed attachment_id=17b0d3d3-2526-496c-b493-4b7a5f86e731 job_id=fdbc461c-0fa6-4e06-a435-2f07583b5f35 attempt=1 variants=3 render_peak_kib=121832 took=434ms waited=10609ms
|
||||
2026-11-05T21:30:21.061Z INFO src.services.processing [rid=-] attachment processed attachment_id=827047d6-f7a1-4ec9-bd52-e9c3fcd3cd9a job_id=90211616-d3a6-40dd-b668-07233c4a66eb attempt=1 variants=3 render_peak_kib=121832 took=426ms waited=10825ms
|
||||
2026-11-05T21:30:21.476Z INFO src.services.processing [rid=-] attachment processed attachment_id=b4d05a95-56bf-49fe-ba61-ee3e0945f6f1 job_id=c28ddd0f-501a-441c-88b7-52dc84569e31 attempt=1 variants=3 render_peak_kib=121832 took=414ms waited=11031ms
|
||||
2026-11-05T21:30:21.949Z INFO src.services.processing [rid=-] attachment processed attachment_id=d892aa6c-e56c-4591-aea6-13dd3c0cdcbf job_id=f7209f53-dcfa-430c-90bc-3df9a9ea569c attempt=1 variants=3 render_peak_kib=121832 took=471ms waited=11217ms
|
||||
2026-11-05T21:30:22.388Z INFO src.services.processing [rid=-] attachment processed attachment_id=2d79e1eb-4828-4cc1-9af6-05cb82c9b1ce job_id=19eaff02-4c77-442e-8344-49039b71d3e6 attempt=1 variants=3 render_peak_kib=121832 took=437ms waited=11469ms
|
||||
2026-11-05T21:30:22.833Z INFO src.services.processing [rid=-] attachment processed attachment_id=cf5f3dd2-8bb3-41e0-9dd2-e8bc70bb39da job_id=4b2b7d5e-a22a-42f4-b2ab-b9d9a63c0dfc attempt=1 variants=3 render_peak_kib=121832 took=444ms waited=11691ms
|
||||
2026-11-05T21:30:23.273Z INFO src.services.processing [rid=-] attachment processed attachment_id=29d5df30-0e28-43c2-8290-ff7d1c9b0f3e job_id=236c1b69-8b40-467d-923d-ccdeb23e9a24 attempt=1 variants=3 render_peak_kib=121832 took=439ms waited=11917ms
|
||||
2026-11-05T21:30:23.790Z INFO src.services.processing [rid=-] attachment processed attachment_id=db37ad3b-a964-4908-92bc-394809bd4094 job_id=fc57b624-bcb3-44b1-81f5-e92d3bc603e7 attempt=1 variants=3 render_peak_kib=121832 took=515ms waited=12129ms
|
||||
2026-11-05T21:30:24.240Z INFO src.services.processing [rid=-] attachment processed attachment_id=348b74c8-9d12-44c6-a377-6b158f4546b1 job_id=b0b752dd-5787-40b1-8ed6-dfffec57ebc1 attempt=1 variants=3 render_peak_kib=121832 took=442ms waited=12433ms
|
||||
2026-11-05T21:30:24.669Z INFO src.services.processing [rid=-] attachment processed attachment_id=5b18a010-a128-4050-9d77-47e82f6078bf job_id=19e203dc-1ae0-43b4-a3c0-89a40c1f79b2 attempt=1 variants=3 render_peak_kib=121832 took=428ms waited=12655ms
|
||||
2026-11-05T21:30:25.090Z INFO src.services.processing [rid=-] attachment processed attachment_id=7ffe5557-1b25-43d5-a43b-e5523afa9428 job_id=19529b02-2c9d-42e3-bae5-aeaabd175e60 attempt=1 variants=3 render_peak_kib=121832 took=419ms waited=12865ms
|
||||
2026-11-05T21:30:25.504Z INFO src.services.processing [rid=-] attachment processed attachment_id=d499578e-2bc7-4027-b501-5860ea55d5ca job_id=eece0b19-021b-4b1e-b448-aafc8f76edaa attempt=1 variants=3 render_peak_kib=121832 took=413ms waited=13064ms
|
||||
2026-11-05T21:30:25.964Z INFO src.services.processing [rid=-] attachment processed attachment_id=faa70be1-308a-412c-9b10-90a3b838cf4e job_id=0c240b0b-bdb0-4bbe-a710-46a066e67aab attempt=1 variants=3 render_peak_kib=121832 took=459ms waited=13254ms
|
||||
2026-11-05T21:30:26.399Z INFO src.services.processing [rid=-] attachment processed attachment_id=ad6bfcea-b064-495a-a67b-7c612c85cbf1 job_id=184149ca-e408-4124-a497-9753bbf6c210 attempt=1 variants=3 render_peak_kib=121832 took=434ms waited=13493ms
|
||||
2026-11-05T21:30:26.825Z INFO src.services.processing [rid=-] attachment processed attachment_id=f2d01b23-9f49-45ca-9446-752ee4146848 job_id=3ea6570b-ef83-4659-a3a1-7327282482cc attempt=1 variants=3 render_peak_kib=121832 took=424ms waited=13706ms
|
||||
2026-11-05T21:30:27.248Z INFO src.services.processing [rid=-] attachment processed attachment_id=710f6915-6047-4806-8723-d1f5755ff9f8 job_id=66392556-ad37-4166-8247-ad93beb8659f attempt=1 variants=3 render_peak_kib=121832 took=422ms waited=13908ms
|
||||
2026-11-05T21:30:27.665Z INFO src.services.processing [rid=-] attachment processed attachment_id=f4adb4b7-377e-4191-a14e-1229c3667a3b job_id=b228afd4-4f1d-41c6-b5e3-f515cb3d132e attempt=1 variants=3 render_peak_kib=121832 took=416ms waited=14113ms
|
||||
2026-11-05T21:30:28.079Z INFO src.services.processing [rid=-] attachment processed attachment_id=f3e14ab1-94d4-49c9-9cfc-595a7a2eef90 job_id=41c244c4-f9d5-4948-82de-d109470763f4 attempt=1 variants=3 render_peak_kib=121832 took=413ms waited=14308ms
|
||||
2026-11-05T21:30:28.518Z INFO src.services.processing [rid=-] attachment processed attachment_id=b2ba729e-971e-489b-8dc7-34ace34f8b9f job_id=ced8978e-9964-4ee5-b116-f4bb570ed2e9 attempt=1 variants=3 render_peak_kib=121832 took=436ms waited=14493ms
|
||||
2026-11-05T21:30:29.242Z INFO src.services.processing [rid=-] attachment processed attachment_id=f94c62a2-db1a-4ca3-a820-4699397d2a30 job_id=79ee36d1-8148-4f48-bdda-d4cd09d3e8d7 attempt=1 variants=3 render_peak_kib=121832 took=722ms waited=14707ms
|
||||
2026-11-05T21:30:29.663Z INFO src.services.processing [rid=-] attachment processed attachment_id=f6cd9884-cbe6-4eac-b16b-5cc56cf20e9c job_id=450f9ece-07b5-45a8-b54f-8e886bbb74d9 attempt=1 variants=3 render_peak_kib=121832 took=421ms waited=15219ms
|
||||
2026-11-05T21:30:30.077Z INFO src.services.processing [rid=-] attachment processed attachment_id=892a949e-51cb-4d79-b2ce-1d073aee5573 job_id=5d2e66fd-7557-44ca-93f1-108546253f19 attempt=1 variants=3 render_peak_kib=121916 took=412ms waited=15416ms
|
||||
2026-11-05T21:30:30.493Z INFO src.services.processing [rid=-] attachment processed attachment_id=06c1310f-d407-4f6f-a7d9-a48bd17b575a job_id=66a4f96d-c159-4e77-9c73-24f37c369022 attempt=1 variants=3 render_peak_kib=121916 took=415ms waited=15606ms
|
||||
2026-11-05T21:30:30.926Z INFO src.services.processing [rid=-] attachment processed attachment_id=a496c633-d035-4de2-bd10-9fd758589b12 job_id=ac60b10b-41ce-44d0-b187-de6cbf06a97c attempt=1 variants=3 render_peak_kib=121916 took=431ms waited=15805ms
|
||||
2026-11-05T21:30:31.347Z INFO src.services.processing [rid=-] attachment processed attachment_id=4b2537cf-fd56-4a26-bea9-2fd0d35bdc3a job_id=0562246e-ee02-4171-897e-82029053105a attempt=1 variants=3 render_peak_kib=121916 took=420ms waited=16010ms
|
||||
2026-11-05T21:30:31.765Z INFO src.services.processing [rid=-] attachment processed attachment_id=2ce4672b-8cc2-4819-b6ea-c2748b0d2d50 job_id=497e8427-61d0-41d0-89b2-4ed72e0f6c58 attempt=1 variants=3 render_peak_kib=121916 took=416ms waited=16206ms
|
||||
2026-11-05T21:30:32.183Z INFO src.services.processing [rid=-] attachment processed attachment_id=0b1d79f6-6a86-4232-a2f2-67dbb8b35b02 job_id=6ed3ecc7-2fea-4251-84fb-a4b891aeb9af attempt=1 variants=3 render_peak_kib=121916 took=417ms waited=16410ms
|
||||
2026-11-05T21:30:32.600Z INFO src.services.processing [rid=-] attachment processed attachment_id=8f21b643-8745-44eb-bd39-3b6fe2516e6a job_id=1e56a6cc-bfec-4a52-9ebe-8663079f2b47 attempt=1 variants=3 render_peak_kib=121916 took=416ms waited=16603ms
|
||||
2026-11-05T21:30:33.063Z INFO src.services.processing [rid=-] attachment processed attachment_id=0a62f19c-cb60-4ebb-8a7e-d4a1d818f67e job_id=0fb30084-5503-430e-a516-3ddcb7e14d40 attempt=1 variants=3 render_peak_kib=121916 took=462ms waited=16800ms
|
||||
2026-11-05T21:30:33.480Z INFO src.services.processing [rid=-] attachment processed attachment_id=1d9ffd1a-1e8d-4a82-81ba-96404add3618 job_id=988625eb-44f1-4ae4-aff7-5871d0c8ae5b attempt=1 variants=3 render_peak_kib=121916 took=416ms waited=17044ms
|
||||
2026-11-05T21:30:33.934Z INFO src.services.processing [rid=-] attachment processed attachment_id=e2fe3159-c26c-4f46-8d79-adfb684d79e0 job_id=a7b4094d-8d77-46c2-a629-2db3041f14f0 attempt=1 variants=3 render_peak_kib=121916 took=452ms waited=17236ms
|
||||
2026-11-05T21:30:34.385Z INFO src.services.processing [rid=-] attachment processed attachment_id=15be21b6-c7ec-4c40-8430-e367a30e590c job_id=1e077c09-901d-43e5-b54c-fb38d7f68aa1 attempt=1 variants=3 render_peak_kib=121916 took=450ms waited=17463ms
|
||||
2026-11-05T21:30:34.992Z INFO src.services.processing [rid=-] attachment processed attachment_id=0c6d9cb2-95bc-42e8-b53d-069b677ca771 job_id=d279e5d4-f17f-47c8-987b-d6abcb2298ef attempt=1 variants=3 render_peak_kib=121916 took=606ms waited=17698ms
|
||||
2026-11-05T21:30:35.423Z INFO src.services.processing [rid=-] attachment processed attachment_id=26a90e04-898d-491f-b478-eead78126c64 job_id=5969e6d9-c761-4f8e-996b-f83ebc0d4443 attempt=1 variants=3 render_peak_kib=121916 took=429ms waited=18081ms
|
||||
2026-11-05T21:30:35.841Z INFO src.services.processing [rid=-] attachment processed attachment_id=45a09851-fa4a-4be5-bca2-adaf54d2650a job_id=2adfdd24-93b3-45af-917b-740884ebb87f attempt=1 variants=3 render_peak_kib=121964 took=417ms waited=18292ms
|
||||
2026-11-05T21:30:36.264Z INFO src.services.processing [rid=-] attachment processed attachment_id=b4e1ee90-2e56-430b-bbf8-05d6c992512b job_id=b6db3002-8a18-43fb-8bd5-c8574cd3ab75 attempt=1 variants=3 render_peak_kib=121964 took=422ms waited=18486ms
|
||||
2026-11-05T21:30:36.712Z INFO src.services.processing [rid=-] attachment processed attachment_id=8505fdc8-8637-453a-b102-acf23ad0c7d3 job_id=f5923643-7833-4892-86ae-c54ba43ffc35 attempt=1 variants=3 render_peak_kib=121964 took=447ms waited=18690ms
|
||||
2026-11-05T21:30:37.145Z INFO src.services.processing [rid=-] attachment processed attachment_id=05203ba0-96b7-4692-85e1-1e848c62fdc2 job_id=14bca4f3-9770-4dbf-af34-be36b7f8f53e attempt=1 variants=3 render_peak_kib=121964 took=431ms waited=18918ms
|
||||
2026-11-05T21:30:37.574Z INFO src.services.processing [rid=-] attachment processed attachment_id=aba31ae4-f7b7-4d78-a86e-956df9dd2998 job_id=2734fc53-4a44-4702-b7ae-c6180f1619cb attempt=1 variants=3 render_peak_kib=121964 took=427ms waited=19127ms
|
||||
2026-11-05T21:30:37.990Z INFO src.services.processing [rid=-] attachment processed attachment_id=11c98c96-4c13-45a0-8112-e26d1ed6bda3 job_id=cce19af7-ed7e-45ac-afd7-61feac9b18ea attempt=1 variants=3 render_peak_kib=121964 took=416ms waited=19329ms
|
||||
2026-11-05T21:30:38.406Z INFO src.services.processing [rid=-] attachment processed attachment_id=c4668afc-8373-4fea-afc7-e64e92c1ca89 job_id=265b9fa3-e0e2-418c-a490-02b9ee139e5b attempt=1 variants=3 render_peak_kib=121964 took=414ms waited=19530ms
|
||||
2026-11-05T21:30:38.859Z INFO src.services.processing [rid=-] attachment processed attachment_id=9ce2e47c-80b4-4b08-837e-b71460a28ae0 job_id=68bdeca7-85d1-400f-852b-ab4537833d28 attempt=1 variants=3 render_peak_kib=121964 took=452ms waited=19714ms
|
||||
2026-11-05T21:30:39.339Z INFO src.services.processing [rid=-] attachment processed attachment_id=7fd5498b-e040-4f80-8d6d-061f35ac74c3 job_id=9b15dead-a965-43a5-b71e-6c351e27582c attempt=1 variants=3 render_peak_kib=121964 took=479ms waited=19953ms
|
||||
2026-11-05T21:30:39.775Z INFO src.services.processing [rid=-] attachment processed attachment_id=030394ca-1319-491b-98d3-b30547da390f job_id=0094a2c6-0ca5-4928-816b-328a2c955994 attempt=1 variants=3 render_peak_kib=121964 took=434ms waited=20206ms
|
||||
2026-11-05T21:30:40.192Z INFO src.services.processing [rid=-] attachment processed attachment_id=9ae936d9-f835-48c7-a4d6-258a3dd6f326 job_id=bcce88dc-3b93-4047-a92d-01594934c498 attempt=1 variants=3 render_peak_kib=121964 took=416ms waited=20427ms
|
||||
2026-11-05T21:30:40.611Z INFO src.services.processing [rid=-] attachment processed attachment_id=37e3ad18-cae3-44c9-8b8a-c5f136f0dc56 job_id=a6dc93a6-b78a-4760-b89a-9024fe5a7b24 attempt=1 variants=3 render_peak_kib=121964 took=417ms waited=20613ms
|
||||
2026-11-05T21:30:41.037Z INFO src.services.processing [rid=-] attachment processed attachment_id=37496f4c-3ff8-42dc-b98b-26e40c6f7ab3 job_id=5ad62776-f782-4bac-8ef3-8595cb84dcb8 attempt=1 variants=3 render_peak_kib=121964 took=425ms waited=20819ms
|
||||
2026-11-05T21:30:41.452Z INFO src.services.processing [rid=-] attachment processed attachment_id=9909caef-a4e1-44eb-b9c4-033e78cd4e5a job_id=c5e944ef-6b46-44b4-a49c-20ba728f45e3 attempt=1 variants=3 render_peak_kib=121964 took=414ms waited=21018ms
|
||||
2026-11-05T21:30:41.905Z INFO src.services.processing [rid=-] attachment processed attachment_id=a57b1791-f9a4-42a6-84a5-286e1c87318a job_id=b3765e44-bc73-4ce0-abb9-cfa337ca1b59 attempt=1 variants=3 render_peak_kib=121964 took=451ms waited=21217ms
|
||||
2026-11-05T21:30:42.357Z INFO src.services.processing [rid=-] attachment processed attachment_id=2bbeb076-e9bb-4c8b-aeaf-93ffe6e72281 job_id=a20da119-87c6-4f57-817b-9c7f2dbf9618 attempt=1 variants=3 render_peak_kib=121964 took=449ms waited=21441ms
|
||||
2026-11-05T21:30:42.776Z INFO src.services.processing [rid=-] attachment processed attachment_id=3e5fc06f-578f-40ea-adab-397af3ec811d job_id=b76dbfee-b5b0-4304-b3c8-b093e7c15c70 attempt=1 variants=3 render_peak_kib=121964 took=417ms waited=21675ms
|
||||
2026-11-05T21:30:43.191Z INFO src.services.processing [rid=-] attachment processed attachment_id=c4cc883f-62cb-4c6f-85d1-4cf046103ac4 job_id=152759df-7829-4cd1-8097-f7e9ba39d274 attempt=1 variants=3 render_peak_kib=121964 took=414ms waited=21875ms
|
||||
2026-11-05T21:30:43.609Z INFO src.services.processing [rid=-] attachment processed attachment_id=6eaab446-100d-40b6-a709-97d58921aa83 job_id=9109be29-6506-4ce5-94a7-cac364a1f7cd attempt=1 variants=3 render_peak_kib=121964 took=415ms waited=22068ms
|
||||
2026-11-05T21:30:44.059Z INFO src.services.processing [rid=-] attachment processed attachment_id=60c4b08c-30ab-4741-b427-00c192ca8fb6 job_id=326c9166-9cee-440c-9932-57564597242d attempt=1 variants=3 render_peak_kib=121964 took=448ms waited=22263ms
|
||||
2026-11-05T21:30:44.481Z INFO src.services.processing [rid=-] attachment processed attachment_id=a2c9949c-925c-4713-91fa-f61e8f61cdab job_id=8b0ab8aa-0754-41b5-bd89-dab593cd2baa attempt=1 variants=3 render_peak_kib=121964 took=421ms waited=22483ms
|
||||
2026-11-05T21:30:44.925Z INFO src.services.processing [rid=-] attachment processed attachment_id=b461cb22-893a-41a3-b5d4-7abb55082f7c job_id=a0f32305-8c0a-41ca-a5d5-e1674a45ee8c attempt=1 variants=3 render_peak_kib=121964 took=441ms waited=22684ms
|
||||
2026-11-05T21:30:45.351Z INFO src.services.processing [rid=-] attachment processed attachment_id=fab39c99-dc95-4bff-8e0c-01cc206f231a job_id=fc86ff57-f981-4c71-940b-bde300f48b1f attempt=1 variants=3 render_peak_kib=121964 took=424ms waited=22915ms
|
||||
2026-11-05T21:30:45.800Z INFO src.services.processing [rid=-] attachment processed attachment_id=83b45a7f-147e-400b-986a-a23538361e31 job_id=ed060b43-2379-4f23-8900-3f8bab8a5ea9 attempt=1 variants=3 render_peak_kib=121964 took=448ms waited=23109ms
|
||||
2026-11-05T21:30:46.223Z INFO src.services.processing [rid=-] attachment processed attachment_id=1cf6f6e1-80dc-4a31-8638-e2fa59e345d9 job_id=7feb25a0-ea6a-4c47-a997-fa506754ed43 attempt=1 variants=3 render_peak_kib=121964 took=420ms waited=23347ms
|
||||
2026-11-05T21:30:46.648Z INFO src.services.processing [rid=-] attachment processed attachment_id=f202f9b8-3815-4890-bf32-d42ed8b96ffd job_id=99d014c2-ca4f-4f02-9443-76a4523c3fa0 attempt=1 variants=3 render_peak_kib=121964 took=424ms waited=23537ms
|
||||
2026-11-05T21:30:47.071Z INFO src.services.processing [rid=-] attachment processed attachment_id=1438b5de-a438-4049-b6e5-027a6ce41c58 job_id=d8df02cb-b46a-4f5e-9631-ebe25523b71e attempt=1 variants=3 render_peak_kib=121964 took=422ms waited=23748ms
|
||||
2026-11-05T21:30:47.485Z INFO src.services.processing [rid=-] attachment processed attachment_id=c18f3276-ebbc-4e94-b988-f83592816257 job_id=b403c8e2-41a5-4994-b68a-b14a5e80f6f4 attempt=1 variants=3 render_peak_kib=121964 took=414ms waited=23949ms
|
||||
2026-11-05T21:30:47.913Z INFO src.services.processing [rid=-] attachment processed attachment_id=238cc8ed-3849-42ac-9905-ad5fe2234478 job_id=8a20d4de-7622-4c61-8b0b-863671e380b2 attempt=1 variants=3 render_peak_kib=121964 took=427ms waited=24143ms
|
||||
2026-11-05T21:30:48.346Z INFO src.services.processing [rid=-] attachment processed attachment_id=6497b36b-c08f-48cb-a34f-f856e51cc1b3 job_id=08bcfb87-a157-4636-a3e1-2cd876fa7f92 attempt=1 variants=3 render_peak_kib=121964 took=431ms waited=24343ms
|
||||
2026-11-05T21:30:48.800Z INFO src.services.processing [rid=-] attachment processed attachment_id=fbbe378a-88df-4b39-903e-4fa9f486cec9 job_id=93144847-995b-458d-936f-7682d263e37b attempt=1 variants=3 render_peak_kib=121964 took=452ms waited=24559ms
|
||||
2026-11-05T21:30:49.223Z INFO src.services.processing [rid=-] attachment processed attachment_id=6d2a58e3-9a35-4e5f-9aac-45f54c730abc job_id=681d5995-9dc5-4f6a-a69a-124e76b8d3be attempt=1 variants=3 render_peak_kib=121964 took=422ms waited=24790ms
|
||||
2026-11-05T21:30:49.656Z INFO src.services.processing [rid=-] attachment processed attachment_id=df73151a-f7b4-423b-b23b-b5c0bca927d4 job_id=2b44c325-3943-4c79-90c9-77c0eaead340 attempt=1 variants=3 render_peak_kib=121964 took=430ms waited=24992ms
|
||||
2026-11-05T21:30:50.087Z INFO src.services.processing [rid=-] attachment processed attachment_id=957938b3-a008-4653-966b-1415c7a713d3 job_id=24e9e711-bb8e-45f8-880a-84fe45323458 attempt=1 variants=3 render_peak_kib=121964 took=430ms waited=25196ms
|
||||
2026-11-05T21:30:50.530Z INFO src.services.processing [rid=-] attachment processed attachment_id=2f1c9ea7-e917-41c8-b82e-2944f0055813 job_id=52ee4936-1c7a-4173-b8fd-7cdee0563c8a attempt=1 variants=3 render_peak_kib=121964 took=442ms waited=25414ms
|
||||
2026-11-05T21:30:50.962Z INFO src.services.processing [rid=-] attachment processed attachment_id=ca2acb29-3ece-458e-99bf-b4d45dbbcae8 job_id=1b27f1b2-f62d-4850-b91a-b8b9476f24b5 attempt=1 variants=3 render_peak_kib=121964 took=431ms waited=25632ms
|
||||
2026-11-05T21:30:51.374Z INFO src.services.processing [rid=-] attachment processed attachment_id=760a64af-3c1a-40a9-994e-0920e44d35f8 job_id=8dadbd30-c982-4807-a860-7ece5963aa8b attempt=1 variants=3 render_peak_kib=121964 took=411ms waited=25845ms
|
||||
2026-11-05T21:30:51.795Z INFO src.services.processing [rid=-] attachment processed attachment_id=d36acb87-a393-4038-930b-ff29565b0f07 job_id=ec861452-952e-4928-95c1-849d8f8dfb1d attempt=1 variants=3 render_peak_kib=121964 took=413ms waited=25810ms
|
||||
2026-11-05T21:30:52.217Z INFO src.services.processing [rid=-] attachment processed attachment_id=0334cc33-70ba-40f9-b073-eb578e8e241a job_id=7a8c4eca-da1f-4efb-804f-b1b44995f675 attempt=1 variants=3 render_peak_kib=121964 took=421ms waited=25789ms
|
||||
2026-11-05T21:30:52.633Z INFO src.services.processing [rid=-] attachment processed attachment_id=c06ad9a5-2354-4f2e-a236-8a44b00911bd job_id=58992f32-af82-457c-9967-01ae32dbc149 attempt=1 variants=3 render_peak_kib=121964 took=415ms waited=25768ms
|
||||
2026-11-05T21:30:53.109Z INFO src.services.processing [rid=-] attachment processed attachment_id=281e8193-f4b1-44e7-990a-f0cdc7275a42 job_id=e5c7be01-7a89-46fb-91aa-bca587bb5b4f attempt=1 variants=3 render_peak_kib=121964 took=475ms waited=25729ms
|
||||
2026-11-05T21:30:53.535Z INFO src.services.processing [rid=-] attachment processed attachment_id=5e91f376-8e2f-461a-a02e-a57f9ca8b883 job_id=285f98bc-5a33-4662-8d84-e0b0f5b2b631 attempt=1 variants=3 render_peak_kib=121964 took=425ms waited=25773ms
|
||||
2026-11-05T21:30:53.955Z INFO src.services.processing [rid=-] attachment processed attachment_id=320955eb-e871-4f85-8e15-6698afe4a1be job_id=20b0495f-84af-4703-bf1e-c5f0c0cdde98 attempt=1 variants=3 render_peak_kib=121964 took=417ms waited=25754ms
|
||||
2026-11-05T21:30:54.406Z INFO src.services.processing [rid=-] attachment processed attachment_id=3db997e6-a32c-40a2-a45e-4c9c2cd7e502 job_id=017bf1ad-976b-4e7b-9fe2-aa971f1f3dae attempt=1 variants=3 render_peak_kib=121964 took=450ms waited=25732ms
|
||||
2026-11-05T21:30:54.838Z INFO src.services.processing [rid=-] attachment processed attachment_id=cadcc207-8611-4cc9-b60f-1dd8c7d4bd55 job_id=e0a3b430-348a-4146-960e-4d57fd9250b6 attempt=1 variants=3 render_peak_kib=121964 took=431ms waited=25740ms
|
||||
2026-11-05T21:30:55.297Z INFO src.services.processing [rid=-] attachment processed attachment_id=0b83619a-7ea9-45d4-866e-c6c16bc93c48 job_id=87dfc1b1-99a3-472e-affd-4004ed0cd865 attempt=1 variants=3 render_peak_kib=121964 took=457ms waited=25945ms
|
||||
2026-11-05T21:30:55.723Z INFO src.services.processing [rid=-] attachment processed attachment_id=622914c0-e3e6-4c30-adc3-956d56fe16fe job_id=bc44b1d9-c63a-4321-8610-fea3238a1cc4 attempt=1 variants=3 render_peak_kib=121964 took=425ms waited=26184ms
|
||||
2026-11-05T21:30:56.145Z INFO src.services.processing [rid=-] attachment processed attachment_id=5668ab16-c8a9-405b-9941-d93e79dc8d54 job_id=6cf974f8-a928-40f8-a4e9-24c69f4d03b6 attempt=1 variants=3 render_peak_kib=121964 took=421ms waited=25503ms
|
||||
2026-11-05T21:30:56.560Z INFO src.services.processing [rid=-] attachment processed attachment_id=39b17f68-2a3c-432a-bff6-788c87bce24a job_id=0db9e848-5bb7-459d-bcfa-0f09d496c913 attempt=1 variants=3 render_peak_kib=121964 took=414ms waited=25482ms
|
||||
2026-11-05T21:30:57.012Z INFO src.services.processing [rid=-] attachment processed attachment_id=c0f3b973-98d4-477b-8990-69a9d5803010 job_id=776b40da-dcaf-4465-8eb0-e7444761dc2c attempt=1 variants=3 render_peak_kib=121964 took=450ms waited=25452ms
|
||||
2026-11-05T21:30:57.436Z INFO src.services.processing [rid=-] attachment processed attachment_id=5769a304-8f7f-44cf-b308-0621c5e105fb job_id=4ecb8974-f081-478f-9565-3bab2140b1b0 attempt=1 variants=3 render_peak_kib=121964 took=423ms waited=25454ms
|
||||
2026-11-05T21:30:57.869Z INFO src.services.processing [rid=-] attachment processed attachment_id=859202b7-8e79-4b00-8379-fa0c6d6cc2cb job_id=12be6c25-c4a6-419e-a641-7f7307f0e1e3 attempt=1 variants=3 render_peak_kib=121964 took=432ms waited=25440ms
|
||||
2026-11-05T21:30:58.283Z INFO src.services.processing [rid=-] attachment processed attachment_id=eba96fcb-0fda-4225-aeff-bb122f52a7b3 job_id=7fc27b84-8c98-4594-be0d-f7fa53dea514 attempt=1 variants=3 render_peak_kib=121964 took=413ms waited=25649ms
|
||||
2026-11-05T21:30:58.700Z INFO src.services.processing [rid=-] attachment processed attachment_id=ae1613e5-33fc-4427-9917-979b0e305a1c job_id=6643b134-eb36-43f9-b361-b7661d70a4d2 attempt=1 variants=3 render_peak_kib=121964 took=415ms waited=25621ms
|
||||
2026-11-05T21:30:59.116Z INFO src.services.processing [rid=-] attachment processed attachment_id=cec6265f-7d89-422c-b98c-ecdc43009b8d job_id=1b7b744e-a532-4c91-8259-c28bc3141707 attempt=1 variants=3 render_peak_kib=121964 took=415ms waited=25594ms
|
||||
2026-11-05T21:30:59.531Z INFO src.services.processing [rid=-] attachment processed attachment_id=9d0334f7-7465-4aa9-bda5-2f4044ffaff9 job_id=97abb300-f290-4ecd-af2d-67f530f299ca attempt=1 variants=3 render_peak_kib=121964 took=413ms waited=25566ms
|
||||
2026-11-05T21:30:59.950Z INFO src.services.processing [rid=-] attachment processed attachment_id=acee1608-55c9-4e7f-ace8-8370e2f34165 job_id=008ba604-3abb-402d-bc4c-c0eba3de92bb attempt=1 variants=3 render_peak_kib=121964 took=417ms waited=25538ms
|
||||
2026-11-05T21:31:00.388Z INFO src.services.processing [rid=-] attachment processed attachment_id=b94d4aa2-ed2a-4e01-85ad-130ab52146c5 job_id=ca946eeb-932c-407c-acf8-7a298b62d361 attempt=1 variants=3 render_peak_kib=121964 took=437ms waited=25511ms
|
||||
2026-11-05T21:31:00.815Z INFO src.services.processing [rid=-] attachment processed attachment_id=54c035c9-005e-4efa-9e11-46deee08750c job_id=33fc7e01-80c8-406a-8273-67fb200b3ebc attempt=1 variants=3 render_peak_kib=121964 took=426ms waited=25282ms
|
||||
2026-11-05T21:31:01.247Z INFO src.services.processing [rid=-] attachment processed attachment_id=85da543e-f025-44c0-aa64-e650a6d68af9 job_id=56d540d8-ba36-4962-a12a-e8f8e69552a0 attempt=1 variants=3 render_peak_kib=121964 took=431ms waited=25265ms
|
||||
2026-11-05T21:31:01.665Z INFO src.services.processing [rid=-] attachment processed attachment_id=ca073e64-501a-4a49-8c41-d1e60627f282 job_id=f6e467f0-e315-4595-a093-f5b4621cd850 attempt=1 variants=3 render_peak_kib=121964 took=416ms waited=25255ms
|
||||
2026-11-05T21:31:02.083Z INFO src.services.processing [rid=-] attachment processed attachment_id=9eabc651-7683-4303-a7a6-d4448a1f752b job_id=56807084-fdcf-4d17-a769-85da17c1fea2 attempt=1 variants=3 render_peak_kib=121964 took=417ms waited=25222ms
|
||||
2026-11-05T21:31:02.512Z INFO src.services.processing [rid=-] attachment processed attachment_id=cbcdcde0-a222-44ac-88b4-d403cacd372c job_id=e5f70002-1483-4ad4-9766-f763d572a18d attempt=1 variants=3 render_peak_kib=121964 took=428ms waited=25198ms
|
||||
2026-11-05T21:31:02.931Z INFO src.services.processing [rid=-] attachment processed attachment_id=0f1fb4ed-5f74-4840-b1f8-57e558e328eb job_id=559c0d86-f356-41ad-8702-bb8be4d675dd attempt=1 variants=3 render_peak_kib=121964 took=417ms waited=25186ms
|
||||
2026-11-05T21:31:03.380Z INFO src.services.processing [rid=-] attachment processed attachment_id=528513a2-d9df-4ef4-a0df-9c03749d13c5 job_id=df2cc2b5-2288-492b-863e-b6411d603753 attempt=1 variants=3 render_peak_kib=121964 took=448ms waited=25163ms
|
||||
2026-11-05T21:31:03.841Z INFO src.services.processing [rid=-] attachment processed attachment_id=4bb82a54-c4d5-45b2-9509-6fa5f4c2a7e0 job_id=6a9a9640-6279-4a61-9ac0-bddf60dca593 attempt=1 variants=3 render_peak_kib=121964 took=460ms waited=25171ms
|
||||
2026-11-05T21:31:04.268Z INFO src.services.processing [rid=-] attachment processed attachment_id=1f951299-75cb-43e5-af82-8c11cee315f4 job_id=af321317-6e10-4369-a468-bb75ac8a49e4 attempt=1 variants=3 render_peak_kib=121964 took=425ms waited=25411ms
|
||||
2026-11-05T21:31:04.687Z INFO src.services.processing [rid=-] attachment processed attachment_id=fb9db74c-1eb2-4c20-b055-aaee76a46df5 job_id=a5fa5ab5-235e-4b19-b598-d2c055928f85 attempt=1 variants=3 render_peak_kib=121964 took=417ms waited=25390ms
|
||||
2026-11-05T21:31:05.105Z INFO src.services.processing [rid=-] attachment processed attachment_id=0e321f5d-3743-4d79-b984-66ce9945a306 job_id=4d3720a3-11ed-4fc3-847b-ceae09e1a7d0 attempt=1 variants=3 render_peak_kib=121964 took=417ms waited=25148ms
|
||||
2026-11-05T21:31:06.953Z INFO src.services.processing [rid=-] attachment processed attachment_id=42c65662-cc4a-4561-8a72-962e4a1a3026 job_id=67d46d27-b526-4ab7-9865-ec71f22d59b2 attempt=1 variants=3 render_peak_kib=121964 took=455ms waited=356ms
|
||||
2026-11-05T21:31:07.434Z INFO src.services.processing [rid=-] attachment processed attachment_id=299d2026-67cb-4baf-9ed1-d7e2b0d9afb8 job_id=fff61c13-cc43-4bff-b266-a9754a4ace1f attempt=1 variants=3 render_peak_kib=121964 took=425ms waited=148ms
|
||||
2026-11-05T21:31:08.726Z INFO src.services.processing [rid=-] attachment processed attachment_id=e458f350-92b9-496b-9478-7b681acbcb95 job_id=534946d9-616b-4d0f-9ee3-d189f13fc8e1 attempt=1 variants=3 render_peak_kib=121964 took=704ms waited=419ms
|
||||
2026-11-05T21:31:09.474Z INFO src.services.processing [rid=-] attachment processed attachment_id=c90e4642-aaea-414b-a4f8-fc8cd0b2f0c1 job_id=f3e5ccd6-1dbf-4e60-a7b8-a58dc6b9b2d9 attempt=1 variants=3 render_peak_kib=121964 took=438ms waited=692ms
|
||||
2026-11-05T21:31:09.969Z INFO src.services.processing [rid=-] attachment processed attachment_id=13a201e2-02ec-40a7-aa1e-e659231b048f job_id=398205af-ba14-4c8d-8610-fcf7bd2d5e3e attempt=1 variants=3 render_peak_kib=121964 took=418ms waited=466ms
|
||||
2026-11-05T21:31:10.490Z INFO src.services.processing [rid=-] attachment processed attachment_id=55ede26f-0269-4e8c-908f-8e602b46fcc0 job_id=d168b3ad-6a93-4212-bab9-39f1ce1a6481 attempt=1 variants=3 render_peak_kib=121964 took=427ms waited=247ms
|
||||
2026-11-05T21:31:11.501Z INFO src.services.processing [rid=-] attachment processed attachment_id=ee7aaeac-3239-453d-8b61-7fadaf32a975 job_id=dae913b2-5ad3-40c5-8716-454027dcc743 attempt=1 variants=3 render_peak_kib=121964 took=419ms waited=522ms
|
||||
2026-11-05T21:31:12.025Z INFO src.services.processing [rid=-] attachment processed attachment_id=c61e8fca-ebb3-4c16-9095-1376d2c75e48 job_id=1d9f56b1-dc46-4897-aa28-354e724552ab attempt=1 variants=3 render_peak_kib=121964 took=433ms waited=289ms
|
||||
2026-11-05T21:31:12.524Z INFO src.services.processing [rid=-] attachment processed attachment_id=c3652aa0-3cbf-4adf-9019-ad899be8d744 job_id=e74f572e-58cd-43de-82b3-ab2117f9b506 attempt=1 variants=3 render_peak_kib=121964 took=428ms waited=65ms
|
||||
2026-11-05T21:31:13.628Z INFO src.services.processing [rid=-] attachment processed attachment_id=8cb2e149-e29d-4c38-84a8-d516da1402d3 job_id=549c5c4a-b015-4c16-9f1d-eaab50f7822a attempt=1 variants=3 render_peak_kib=121968 took=512ms waited=319ms
|
||||
2026-11-05T21:31:14.608Z INFO src.services.processing [rid=-] attachment processed attachment_id=7db1afc6-e5fb-462e-8392-ffc96364c9a7 job_id=ba514e3d-95a2-4780-b650-0ab748bf6926 attempt=1 variants=3 render_peak_kib=121968 took=979ms waited=116ms
|
||||
2026-11-05T21:31:15.141Z INFO src.services.processing [rid=-] attachment processed attachment_id=ce54d6eb-fa3c-413a-ac3f-983020545c33 job_id=594de798-4b19-48b4-a7f4-71183d3a592f attempt=1 variants=3 render_peak_kib=121968 took=507ms waited=376ms
|
||||
2026-11-05T21:31:15.561Z INFO src.services.processing [rid=-] attachment processed attachment_id=caafbeaa-2411-4277-b980-da1b3ff39433 job_id=daedeedf-83cf-4c27-a6e5-afbace0c5a74 attempt=1 variants=3 render_peak_kib=121968 took=420ms waited=152ms
|
||||
2026-11-05T21:31:16.595Z INFO src.services.processing [rid=-] attachment processed attachment_id=4de49e9f-5d93-4731-bc38-5a95a1d9e911 job_id=dd968735-c53d-4387-804d-3d42eeed2f63 attempt=1 variants=3 render_peak_kib=121968 took=439ms waited=404ms
|
||||
2026-11-05T21:31:17.127Z INFO src.services.processing [rid=-] attachment processed attachment_id=07c73dda-da7c-42da-8571-ad82ba9d6901 job_id=9a29ce2e-df79-473c-a5e5-c07f0069426c attempt=1 variants=3 render_peak_kib=121968 took=455ms waited=204ms
|
||||
2026-11-05T21:31:18.136Z INFO src.services.processing [rid=-] attachment processed attachment_id=115cd1fc-baf4-4217-9b78-15cae48f779e job_id=9780a46a-260a-45a1-a014-bfb7271cd50a attempt=1 variants=3 render_peak_kib=121968 took=449ms waited=477ms
|
||||
2026-11-05T21:31:18.629Z INFO src.services.processing [rid=-] attachment processed attachment_id=e27e4185-70cd-46e9-bd19-b5a60022aae1 job_id=88dfed47-130c-4c50-9240-9421771f25cc attempt=1 variants=3 render_peak_kib=121968 took=437ms waited=243ms
|
||||
2026-11-05T21:31:19.647Z INFO src.services.processing [rid=-] attachment processed attachment_id=89f837c6-3d93-45cd-b3cb-f4142c9ff51b job_id=1663aa2d-7bf0-460e-a04f-5bafc57431af attempt=1 variants=3 render_peak_kib=121968 took=436ms waited=520ms
|
||||
2026-11-05T21:31:20.157Z INFO src.services.processing [rid=-] attachment processed attachment_id=2707099a-b489-41d0-b3bb-374ee75e9bae job_id=93a34b6c-dfe4-41f5-9f5e-06b41127976c attempt=1 variants=3 render_peak_kib=121968 took=436ms waited=293ms
|
||||
2026-11-05T21:31:21.173Z INFO src.services.processing [rid=-] attachment processed attachment_id=384256b1-47e0-4965-ac5e-6f16ef69fccd job_id=b4e336b2-0ac2-427c-ab93-e7bb8dbcfd82 attempt=1 variants=3 render_peak_kib=121968 took=437ms waited=568ms
|
||||
2026-11-05T21:31:21.658Z INFO src.services.processing [rid=-] attachment processed attachment_id=f8fba439-fc86-4088-a396-e9a7bf2a9fee job_id=31d0c402-84fd-4da9-b5c2-ccbcd881f4e6 attempt=1 variants=3 render_peak_kib=121968 took=414ms waited=331ms
|
||||
2026-11-05T21:31:22.158Z INFO src.services.processing [rid=-] attachment processed attachment_id=46cf924a-b5da-48b0-bb14-70bc1e338bc9 job_id=aeceba11-d9d2-4837-9a3b-7f2b86596fa5 attempt=1 variants=3 render_peak_kib=121968 took=409ms waited=103ms
|
||||
2026-11-05T21:31:23.267Z INFO src.services.processing [rid=-] attachment processed attachment_id=808c30ef-f7e1-4365-a710-e1fe724abe9b job_id=147580ad-69b6-48f2-b7b1-0d38a433685b attempt=1 variants=3 render_peak_kib=121968 took=490ms waited=374ms
|
||||
2026-11-05T21:31:23.694Z INFO src.services.processing [rid=-] attachment processed attachment_id=d82001f7-3513-4eca-b1e2-a32426b525bc job_id=cdda7ae1-5d89-4af4-a9ca-700b4b469843 attempt=1 variants=3 render_peak_kib=121968 took=409ms waited=149ms
|
||||
2026-11-05T21:31:24.742Z INFO src.services.processing [rid=-] attachment processed attachment_id=b035103b-4926-4c3a-9e50-4eea8b2780cb job_id=c61f6f26-08ea-4888-874c-aff4fa7d8f05 attempt=1 variants=3 render_peak_kib=121968 took=440ms waited=420ms
|
||||
2026-11-05T21:31:25.249Z INFO src.services.processing [rid=-] attachment processed attachment_id=35bc6a63-8d96-440b-8802-fcb6c90d26dc job_id=710be2fe-fa32-4b48-a144-b204c39871e2 attempt=1 variants=3 render_peak_kib=121968 took=440ms waited=198ms
|
||||
2026-11-05T21:31:26.288Z INFO src.services.processing [rid=-] attachment processed attachment_id=84ddc3d0-ca5f-4753-8ada-182afa43ced6 job_id=8b46d417-e4e4-4b2f-827a-282b7102d185 attempt=1 variants=3 render_peak_kib=121968 took=457ms waited=485ms
|
||||
2026-11-05T21:31:26.804Z INFO src.services.processing [rid=-] attachment processed attachment_id=bd794a67-3495-4eb7-8d69-544564e1b0dd job_id=0680e2da-738d-4521-bdcc-8ae1ed05cf3f attempt=1 variants=3 render_peak_kib=121968 took=463ms waited=255ms
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user