diff --git a/.env.example b/.env.example index 119384b5..00fb2703 100644 --- a/.env.example +++ b/.env.example @@ -11,7 +11,7 @@ SESSION_STORE=postgres CACHE_URL_FILE=/run/secrets/cache_url API_REPLICAS=2 IDEMPOTENCY_TTL_SECONDS=86400 -S3_ENDPOINT=http://s3:9000 +S3_ENDPOINT=http://s3:8333 S3_REGION=us-east-1 S3_BUCKET=messenger-attachments S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id @@ -20,6 +20,9 @@ MAX_UPLOAD_BYTES=10485760 MAX_IMAGE_PIXELS=40000000 BROKER_URL_FILE=/run/secrets/broker_url WORKER_CONCURRENCY=2 +WORKER_PREFETCH=2 JOB_MAX_ATTEMPTS=5 JOB_TIMEOUT_SECONDS=60 STAGING_RETENTION_SECONDS=86400 +MAX_PENDING_JOBS=100 +RENDER_MEMORY_MB=768 diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 7d1ff4a4..3cbab7a0 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,18 +1,76 @@ -## Лабораторная 5 +## Лабораторная 5: асинхронная обработка изображений -Автор / группа: TODO -Целевая оценка (3/4/5): TODO -Трек 5, если заявлен: TODO +Автор / группа: Матвей Козырев / BVT2301. +Целевая оценка: 5. Трек 1 — «Очередь под перегрузкой». +Ветка решения: `solution/BVT2301-Kozyrev`. -Что реализовано и что перенесено: TODO -Команды запуска из чистого clone: TODO -Результаты проверок и ссылка на REPORT.md: TODO -Ограничения / известные ошибки: TODO -Источники и использование ИИ: TODO +Upload принимает исходный файл в устойчивое S3 staging и возвращает `202 queued` после фиксации attachment, job и outbox в PostgreSQL. Отдельный worker создаёт варианты изображения; клиент получает состояние через метаданные. Доказательства уровней 3–5 и ограничения исследования приведены в REPORT.md и evidence. -- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами. -- [ ] Контракт и публичные тесты не ослаблены. -- [ ] make check и make test выполнены; приложен реальный вывод. -- [ ] Сценарии отказов/безопасности своего уровня воспроизведены. -- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens. -- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения. +## Реализация + +- Перенесена база lab4: FastAPI, PostgreSQL, сессии, чаты/сообщения, идемпотентность сообщений, приватное S3-хранилище SeaweedFS, две API-реплики за HAProxy. +- Добавлены RabbitMQ с persistent volume, transactional outbox и отдельные dispatcher/worker. +- Реализованы состояния `queued/processing/ready/failed`, thumbnail JPEG, crop JPEG 128×128 и optimized WebP. +- Добавлены ограниченные retries с backoff/jitter, классификация ошибок, lease recovery, защита от устаревших попыток, реестр failed и операторский replay. Некорректные сообщения направляются в DLQ. +- Рендер выполняется в отдельном процессе с timeout и Linux-лимитом памяти. Worker имеет heartbeat healthcheck; staging и непривязанные объекты очищаются с блокировкой активных writers. +- Добавлены глобальный лимит принятых jobs с ответом `429`, отдельные concurrency/prefetch и генератор исследования с измерением queue wait, времени до ready, throughput, RSS и проверкой восстановления. + +## Запуск и проверка + +Нужны работающий Docker Engine с Compose v2, Python 3.12+, uv и свободный порт `127.0.0.1:8080`. Из корня clone: + +```sh +git switch solution/BVT2301-Kozyrev +python3 scripts/init-secrets.py +docker compose config --quiet +docker compose up -d --build +docker compose ps -a +curl --fail http://localhost:8080/health/ready +make test +``` + +Миграции и bootstrap bucket выполняются сервисами `migrate` и `s3-init`. Остановка с сохранением данных: `docker compose down`. + +Локальные регрессии: + +```sh +uv sync --locked --no-dev +make check +.venv/bin/python student-tests/pipeline_critical.py +.venv/bin/python student-tests/remaining_regressions.py +``` + +Выполнено локально: `make check`, `docker compose config --quiet`, `git diff --check`; 8 тестов критических ошибок и 7 дополнительных регрессионных тестов прошли. Часть сценариев использует mocks и не проверяет SQL на реальном PostgreSQL. + +Подробные команды, результаты, архитектура и матрица критериев: [REPORT.md](../REPORT.md). + +## Интеграционные проверки + +- Публичный smoke, persistence и failover обеих API-реплик. +- Worker stop → upload 202 → сообщение с вложением → restart API/broker → ready. +- Дубли доставки; crash между COMMIT и publish; crash worker после S3 до COMMIT/ack; lease recovery. +- Исчерпание retry, permanent failure, реестр failed, явный replay и отказ replay для ready. +- Реальные SQL-проверки backoff, счётчика lease, fencing и watchdog зависших queued jobs. +- Linux memory/timeout, приватность S3, ACL вариантов, безопасная уборка stale-объектов. +- Трек 1: concurrency/prefetch 1/1, 2/2, 4/4 на двух размерах PNG; throughput, p95, queue wait, RSS, 429, drain и контрольный upload. + +Повторяемые команды и результаты: [REPORT.md](../REPORT.md), [pipeline-integration.json](../evidence/pipeline-integration.json), [pipeline-invariants.json](../evidence/pipeline-invariants.json), [resilience.json](../evidence/resilience.json). + +Исправлены запуск RabbitMQ с Docker secret, обход backoff/lease дублем и отсутствие повторной публикации для зависших queued jobs. Контракты и публичные тесты не менялись. + +Ограничения: один Docker-host; closed-loop burst и polling; один прогон на профиль, без статистической оценки; RSS может учитывать shared pages повторно и пропускать пики. Failed staging сохраняется для replay. Чистый clone отдельно не запускался; исторические student-tests lab4 вне перечисленного набора не заявляются пройденными. Оценку определяет преподаватель. + +## Источники и использование ИИ + +Основа — README задания, обязательные контракты и публичные тесты; версии библиотек зафиксированы в `pyproject.toml`/`uv.lock`, образы — в `compose.yaml`/`Dockerfile`. Источники и детали перечислены в [REPORT.md](../REPORT.md). + +Codex использовался для переноса кода, реализации, аудита, исправлений, тестов и документации. Mock-регрессии отделены от реальных интеграционных и нагрузочных результатов. + +- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами. +- [x] Контракт и публичные тесты не ослаблены. +- [x] make check и make test выполнены; приложен реальный вывод. +- [x] Сценарии отказов/безопасности своего уровня воспроизведены. +- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens. +- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения. + +Проверка секретов: совпадений с локальными credentials и проверенными форматами ключей не обнаружено в отслеживаемых файлах и доступной Git-истории. Область и ограничения проверки: [evidence/checklist-review.txt](../evidence/checklist-review.txt). diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 00000000..ab272d34 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,23 @@ +FROM python:3.12.12-slim + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 + +WORKDIR /app + +RUN pip install --no-cache-dir uv==0.11.28 + +COPY pyproject.toml uv.lock ./ +RUN uv sync --frozen --no-dev --no-install-project + +COPY src ./src +COPY migrations ./migrations + +USER 65534:65534 + +EXPOSE 8080 + +HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \ + CMD ["/app/.venv/bin/python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"] + +CMD ["/app/.venv/bin/python", "-m", "src.main"] diff --git a/Makefile b/Makefile index 5d86d171..da8c6dee 100644 --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ CA_FILE ?= ACTION_SCRIPT ?= TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",) -.PHONY: help check validate test load test-persistence test-failover +.PHONY: help check validate test load track1 test-persistence test-failover help: @echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes" @@ -21,6 +21,10 @@ test: load: $(PYTHON) tests/load.py --base-url "$(BASE_URL)" $(TLS_ARGS) +track1: + @test -n "$(TRACK_ARGS)" || (echo "Set TRACK_ARGS, e.g. --run-name c1-p1 --workers 1 --prefetch 1 --max-pending 20"; exit 2) + .venv/bin/python scripts/track1.py --base-url "$(BASE_URL)" $(TLS_ARGS) $(TRACK_ARGS) + test-persistence: @test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable restart hook"; exit 2) $(PYTHON) tests/resilience.py --mode persistence --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS) diff --git a/REPORT.md b/REPORT.md index e98f75e9..1e4d366a 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,37 +1,186 @@ # Отчёт — лабораторная 5 -- Автор, группа: TODO -- Целевая оценка; выбранный трек 5, если нужен: TODO -- Commit/PR: TODO после публикации -- Использованные источники, библиотеки и помощь ИИ: TODO +> Проверка выполняется на реальном PostgreSQL/S3/RabbitMQ. Результаты отказов: [pipeline-integration.json](evidence/pipeline-integration.json), SQL/безопасности: [pipeline-invariants.json](evidence/pipeline-invariants.json), persistence/failover: [resilience.json](evidence/resilience.json). История первоначальных проверок: [checklist-review.txt](evidence/checklist-review.txt). + +- Автор: Матвей Козырев, BVT2301. +- Целевая оценка: 5, трек 1 «Очередь под перегрузкой». +- Ветка: `solution/BVT2301-Kozyrev`. HEAD перед текущими исправлениями: `a1bb6f8` (`refactor`); результаты относятся к рабочему дереву, а не только этому commit. Ссылка на PR не предоставлена. +- Даты локальной проверки: 26–27 октября 2026 года. +- Статус: перечисленные ниже проверки уровней 3–4 и исследование трека 1 уровня 5 выполнены; результаты и границы проверки приложены. Это не автоматическое выставление оценки. + +## Исходная база, источники и помощь ИИ + +Из собственной lab4 перенесены API на FastAPI, PostgreSQL-миграции, сессии, идемпотентность сообщений, S3-адаптер, две API-реплики и HAProxy. Основание проверки: [README](README.md), [семантика API](contracts/README.md), [состояния вложений](contracts/attachment-states.md), [OpenAPI](contracts/openapi.json), [инфраструктура](contracts/infrastructure.json), [публичные тесты](tests/README.md). + +Использован Codex для переноса исходников, подготовки реализации, аудита, исправлений и оформления отчёта. Локальные проверки не заменяют интеграционную демонстрацию. Справочные первоисточники задания: [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability), [publisher confirms и consumer acknowledgements](https://www.rabbitmq.com/docs/confirms). + +Зависимости зафиксированы в [pyproject.toml](pyproject.toml) и [uv.lock](uv.lock): FastAPI 0.141.1, Uvicorn 0.40.0, asyncpg 0.31.0, argon2-cffi 25.1.0, boto3 1.43.115, Pillow 13.0.0, pika 1.3.2. Имя пакета пока сохранено от lab4. ## Запуск из чистого clone -TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных. +Нужны Git, Python 3.12+ для приложения, uv и работающий Docker Engine с Compose v2. REST доступен на `127.0.0.1:8080`; порт должен быть свободен, в том числе от стенда lab4. Используются синтетические данные. + +В [Dockerfile](Dockerfile) и [Compose](compose.yaml) закреплены Python 3.12.12-slim, uv 0.11.28, PostgreSQL 17.6-alpine, SeaweedFS 4.17 с digest, RabbitMQ 4.1.2-alpine, HAProxy 3.0.11-alpine. + +Из корня полученного clone: + +```sh +git switch solution/BVT2301-Kozyrev +python3 scripts/init-secrets.py +docker compose config --quiet +docker compose up -d --build +docker compose ps -a +curl --fail http://localhost:8080/health/ready +make test +``` + +Локальные credentials создаются в игнорируемом `secrets/`. Их содержимое нельзя публиковать. Стандартный Compose использует значения по умолчанию; отдельный .env не обязателен. Сервис migrate применяет SQL из migrations, s3-init создаёт приватный bucket и проверяет запись. API и worker зависят от успешного завершения этих сервисов. + +Эта инструкция пока не подтверждена запуском из чистого clone. Успешный `up -d` сам по себе не доказывает готовность: нужны проверка процессов, readiness и smoke. Worker использует heartbeat AMQP event loop; outbox/reconcile не наследуют HTTP healthcheck. + +Локальные проверки: + +```sh +uv sync --locked --no-dev +make check +.venv/bin/python student-tests/pipeline_critical.py +# Для полной OpenAPI-валидации: +uv pip install --python .venv/bin/python -r requirements-dev.txt +make validate PYTHON=.venv/bin/python +``` + +Остановка с сохранением volumes: `docker compose down`. Намеренный сброс учебных данных: `docker compose down -v`; удаляет volumes PostgreSQL, S3 и RabbitMQ данного проекта. В ходе проверки сброс не выполнялся. ## Архитектура и решения -TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации. +Клиент → HAProxy → API → PostgreSQL + S3 staging. Фоновый путь: PostgreSQL outbox → dispatcher → RabbitMQ → worker → final S3 objects + метаданные PostgreSQL. Реализация: [attachments.py](src/attachments.py), [postgres.py](src/storage/postgres.py), [outbox.py](src/outbox.py), [broker.py](src/broker.py), [worker.py](src/worker.py). -## Доказательства критериев +PostgreSQL хранит пользователей, чаты, сообщения, сессии, attachments, jobs и outbox. S3 хранит байты. Брокер передаёт служебные ID, без файлов и credentials. -| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства | -| --- | --- | --- | --- | -| Публичный smoke | make test | TODO | TODO | -| Остальные пункты уровня 3 | TODO | TODO | TODO | -| Все пункты уровня 4, если заявлен | TODO | TODO | TODO | -| Выбранный трек 5, если заявлен | TODO | TODO | TODO | +Перед 202 API полностью принимает HTTP body, проверяет изображение, записывает исходник в staging и фиксирует attachment/job/outbox одной DB-транзакцией. S3 PUT не входит в DB-транзакцию: при rollback возможен непривязанный объект. Resize выполняет worker. Клиент получает состояние через GET метаданных; content неготового вложения возвращает 409. -TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены. +Dispatcher декодирует JSONB и публикует persistent сообщение с publisher confirm, затем отмечает event опубликованным. Crash после confirm до отметки допускает дубль. Восстановление после crash API между COMMIT и publish проверено с остановленным dispatcher: в БД остаётся неопубликованный outbox, после запуска задача достигает ready без создания новой job. -## Эксперименты и отказы +Watchdog повторно создаёт outbox-intent для queued-job, не захваченной за 60 секунд после публикации. Он не увеличивает attempt: это повтор доставки, а не новая попытка обработки. После намерения updated_at обновляется, поэтому повтор не происходит каждый цикл dispatcher. Это также восстанавливает доставку, если durable job существует, но брокер больше не содержит сообщения; дубли защищены состоянием job. -TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения. +Worker захватывает только queued-job, для которой не действует отложенный retry. Просроченный processing восстанавливает dispatcher с увеличением attempt; дубль не продлевает lease и не обходит backoff. Завершение проверяет состояние, время захвата processing_started_at и срок lease. Попытки пишут по разным S3-ключам; только действующая попытка публикует свои ключи в БД. Отклонение устаревших completion/retry проверено на PostgreSQL. Логический ключ варианта — (attachment_id, pipeline_version, name); физические непривязанные объекты требуют уборки. -| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных | -| --- | --- | --- | --- | -| TODO | TODO | TODO | TODO | +Ack выполняется после устойчивого результата или записи retry/failed; при сбое записи используется nack/requeue. Terminal-job повторно не обрабатывается. Это at-least-once доставка, не exactly-once. Реестр failed хранится в PostgreSQL; DLQ используется для некорректных сообщений. -## Ограничения и следующие шаги +Ошибка удаления staging после ready журналируется и не откатывает результат. Сервис reconcile раз в минуту удаляет непривязанные объекты старше retention из staging/originals/variants. Общая advisory-блокировка исключает удаление во время upload и работы worker. Staging failed-задач сохраняется для явного replay; автоматического удаления этих исходников нет. Lease recovery учитывает попытки; retry использует exponential backoff с jitter. Невалидное изображение и превышение памяти рендера переводят задачу в failed без повторов. Ошибки зависимостей и timeout повторяются с ограничением попыток. -TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза. +Явный операторский replay только failed-job: + +```sh +docker compose run --rm outbox /app/.venv/bin/python -m src.replay ATTACHMENT_UUID +``` + +Replay сохраняет replayed_at; полного журнала операторских действий нет. + +## Конфигурация и варианты изображений + +| Параметр | Соответствие реализации | +| --- | --- | +| SESSION_TTL_SECONDS / IDEMPOTENCY_TTL_SECONDS | По умолчанию 3600 / 86400 секунд, PostgreSQL | +| S3_ENDPOINT | Compose и .env.example используют http://s3:8333 | +| DATABASE_URL_FILE, CURSOR_SECRET_FILE, S3/Broker *_FILE | Credentials подключаются через Compose secrets | +| MAX_UPLOAD_BYTES / MAX_IMAGE_PIXELS | 10 485 760 байт / 40 000 000 пикселей; 10 MiB = 10 × 1024 × 1024 байт | +| WORKER_CONCURRENCY / WORKER_PREFETCH | По умолчанию 2 / 2, потоки и prefetch | +| JOB_MAX_ATTEMPTS / JOB_TIMEOUT_SECONDS | По умолчанию 5 / 60; рендер идёт в отдельном subprocess, принудительно завершаемом по timeout | +| MAX_PENDING_JOBS | По умолчанию 100 queued/processing jobs; глобальная DB-блокировка при admission, отказ 429 | +| STAGING_RETENTION_SECONDS | Порог возраста непривязанных объектов для плановой уборки, по умолчанию 86400 секунд | +| OUTBOX_FAULT_AFTER_COMMIT / WORKER_FAULT_AFTER_S3_PUT | По умолчанию 0; включаются только отказными тестами. Worker падает один раз на контейнер, после записи вариантов до COMMIT/ack; маркер находится в /tmp | +| API_REPLICAS, SESSION_STORE, CACHE_URL_FILE | Не управляют текущим Compose: явно заданы api-a/api-b и PostgreSQL без Redis | + +Worker применяет EXIF orientation, композитит alpha на белый, создаёт thumbnail JPEG до 256×256, центрированный crop JPEG 128×128, optimized WebP до 1280×1280. JPEG quality=85, WebP quality=80/method=4; явного переноса входных EXIF нет. + +Округление исправлено: исходник 300×200 даёт 256×170; высота вычисляется как floor(200 × 256 / 300). Маленькие изображения не увеличиваются. Linux renderer ограничен RLIMIT_AS=768 MiB; worker-контейнер ограничен 4 GiB RAM и 2 CPU. RSS собирается отдельно, RLIMIT_AS ограничивает виртуальную память. + +## Выполненные проверки и доказательства критериев + +Локально прошли `make check`, полная OpenAPI-валидация и 15 регрессионных тестов (8 + 7). Контракт, README задания и публичные tests не изменены относительно skeleton `276dbda`. Mock-числа не используются как нагрузочные результаты. + +На контейнерном стенде прошли публичный smoke, дополнительные проверки attachments, конкурентная идемпотентность и S3 permissions. Команды и первоначальный вывод: [checklist-review.txt](evidence/checklist-review.txt). + +| Критерий README | Проверка и доказательство | +| --- | --- | +| 3: минимум lab4, durable upload 202 | Smoke, attachments, persistence; HTTP передаёт все байты, исходник после рестарта совпадает | +| 3: broker/worker, доставка, ack | Отдельные контейнеры; broker restart с volume; queued → ready после запуска worker | +| 3: состояния, оригинал, thumbnail | HTTP-сценарии, фактические форматы/размеры и неизменность оригинала | +| 3: retries, failed, идемпотентность | Исчерпание двух попыток в тестовой конфигурации, failed/replay; дубль ready не создаёт объектов | +| 3: worker stop → upload → сообщение → restart API/broker → ready | PASS в [pipeline-integration.json](evidence/pipeline-integration.json) | +| 4: crop/WebP и bytes/pixels/time/memory | Attachments проверяет границы bytes/pixels; invariants — варианты и Linux timeout; integration — Linux memory limit | +| 4: outbox, crash COMMIT/publish | Реальный exit API при остановленном outbox; существующая job достигает ready после восстановления | +| 4: backoff/jitter, классификация, failed/replay | Permanent invalid_image на первой попытке; transient retry исчерпывается; явный replay; SQL-тест запрещает обход backoff дублем | +| 4: crash после S3, lease, cleanup | Worker exit до COMMIT/ack; attempt=2; четыре stale-объекта удалены, четыре объекта актуального результата сохранены | +| 5: три настройки, разные изображения | Серия 1/1, 2/2, 4/4 для 512×512 и 2400×2400; результаты ниже | +| 5: backpressure и восстановление | Генератор требует 429, соблюдение лимита в замерах, ready всех принятых jobs, пустую очередь и успешный контрольный upload | + +Основные источники: [pipeline-integration.json](evidence/pipeline-integration.json), [pipeline-invariants.json](evidence/pipeline-invariants.json), [resilience.json](evidence/resilience.json). Последний содержит состояния контейнеров до/после: сохранение данных проверяется вместе с фактом перезапуска, а не только возвратом hook=0. [job-contract.json](evidence/job-contract.json) подтверждает проверку 425 реальных outbox payloads по обязательной схеме, включая запрет дополнительных полей, и no-op повторной доставки полного сохранённого сообщения. + +Последний публичный failover для обеих реплик: восстановление записи за 2,436 и 4,675 с при требовании ≤10 с; соответственно 113 и 95 подтверждённых записей, 1 и 2 неуспешные переходные попытки. Подтверждённые сообщения, исходный файл и прежняя сессия сохранились. Persistence прошёл с изменившимися StartedAt всех проверяемых API/хранилищ/worker/outbox. + +Воспроизводимый запуск, строго последовательно на отдельном учебном стенде: + +```sh +uv sync --locked --no-dev +uv pip install --python .venv/bin/python -r requirements-dev.txt +make check +make validate PYTHON=.venv/bin/python +make test PYTHON=.venv/bin/python +.venv/bin/python student-tests/attachments.py +.venv/bin/python student-tests/s3_permissions.py +.venv/bin/python student-tests/idempotency_concurrency.py +.venv/bin/python student-tests/pipeline_critical.py +.venv/bin/python student-tests/remaining_regressions.py +.venv/bin/python student-tests/pipeline_invariants.py +.venv/bin/python student-tests/pipeline_integration.py +.venv/bin/python student-tests/job_contract.py +.venv/bin/python scripts/verify-resilience.py +.venv/bin/python scripts/run-track1-profiles.py +.venv/bin/python scripts/record-environment.py +``` + +Отказные тесты временно останавливают/пересоздают сервисы и изменяют только свои синтетические вложения. В finally восстанавливают обычные настройки; volumes не удаляют. Проверка cleanup удалила только четыре явно выбранных непривязанных объекта аварийной попытки; актуальные объекты остались. Не запускайте эти тесты одновременно с нагрузкой. + +## Эксперимент: очередь под перегрузкой + +Гипотеза: увеличение concurrency/prefetch сокращает ожидание в очереди до насыщения CPU, но повышает потребление памяти; admission ограничивает принятую работу и позволяет восстановиться после burst. + +Методика: [run-track1-profiles.py](scripts/run-track1-profiles.py) последовательно выставляет 1/1, 2/2, 4/4. Для каждого профиля выполняются два burst по 160 запросов с 16 клиентскими потоками: PNG 512×512 (1890 байт) и 2400×2400 (22196 байт). Изображения синтетические, с повторяемым рисунком; их сжатый размер не отражает стоимость декодирования. Лимит queued/processing — 32, worker — 2 CPU и 4 GiB RAM. Перед burst проверяются реальные настройки контейнеров и отсутствие pending jobs. Отдельного прогрева и статистических повторений нет: по одному прогону на комбинацию, не измерение предельной производительности. + +Среда и хеши исходников: [environment.json](evidence/environment.json). Docker Desktop на macOS ARM64; VM имеет 10 CPU и 10 416 644 096 байт RAM. Это ресурсы VM, не выделенные worker. + +Источники результатов: [1/1, 512](evidence/real-c1-p1-s512.json), [1/1, 2400](evidence/real-c1-p1-s2400.json), [2/2, 512](evidence/real-c2-p2-s512.json), [2/2, 2400](evidence/real-c2-p2-s2400.json), [4/4, 512](evidence/real-c4-p4-s512.json), [4/4, 2400](evidence/real-c4-p4-s2400.json). + +| Concurrency/prefetch | PNG, px | 202 / 429 | Длительность, с | Ready/с | p95 ready, с | p95 queue wait, с | RSS max, MiB | +| --- | --- | --- | --- | --- | --- | --- | --- | +| 1/1 | 512×512 | 33 / 127 | 25.89 | 1.275 | 24.481 | 23.709 | 134.0 | +| 1/1 | 2400×2400 | 37 / 123 | 35.50 | 1.042 | 30.436 | 29.505 | 240.0 | +| 2/2 | 512×512 | 34 / 126 | 16.24 | 2.093 | 14.519 | 13.572 | 157.9 | +| 2/2 | 2400×2400 | 44 / 116 | 25.81 | 1.705 | 18.617 | 17.474 | 406.4 | +| 4/4 | 512×512 | 32 / 128 | 15.49 | 2.066 | 14.970 | 13.602 | 286.6 | +| 4/4 | 2400×2400 | 40 / 120 | 24.31 | 1.646 | 18.508 | 16.166 | 518.4 | + +Итого: 6 × 160 = 960 запросов, 220 приняты и завершены ready, 740 получили ожидаемый 429; failed, timeout и ошибки метрик — 0. Во всех шести прогонах очередь полностью опустела и контрольный upload достиг ready. Максимум pending в сэмплах — 32; на прогон получено 5–14 сэмплов. + +Вывод для этой серии: 2/2 быстрее 1/1 на обоих размерах, но использует больше памяти. Переход 2/2 → 4/4 не увеличил измеренный throughput: 2,093 → 2,066 ready/с для 512 и 1,705 → 1,646 для 2400. При этом RSS вырос с 157,9 до 286,6 MiB и с 406,4 до 518,4 MiB соответственно. На лимите 2 CPU профиль 2/2 выглядит более экономным выбором для этой нагрузки; статистически доказанного оптимума по одному прогону нет. p95 включает накопление очереди, поэтому это не время одного изолированного resize. + +Формулы: throughput = ready / elapsed_seconds; p95 — элемент ceil(0,95 × N) отсортированной выборки; queue wait = first_started_at − created_at в PostgreSQL; RSS MiB = max_sampled_rss_bytes / 1 048 576. Время ready измеряется от ответа 202 до наблюдения ready. В JSON сохранены p50/p95/p99, все запросы, длительность, замеры очереди и число отказов. + +Это closed-loop burst: coordinated omission, polling и накладные расходы клиента ограничивают выводы. RSS — сумма RSS процессов worker, shared pages могут учитываться несколько раз; пики между замерами пропускаются. Сэмплы pending подтверждают наблюдаемый предел, но не заменяют атомарную admission-блокировку. Число принятых за весь прогон может превышать 32: завершённые jobs освобождают ёмкость. Ответы 429 ожидаемы и учитываются отдельно от транспортных ошибок и failed jobs. + +## Исправления по результатам проверки + +- RabbitMQ 4.1.2 отвергал RABBITMQ_DEFAULT_PASS_FILE. [Стартовый скрипт](infra/rabbitmq-secret-entrypoint.sh) читает Docker secret внутри контейнера и вызывает штатный entrypoint; пароль не записан в Compose. +- Дубль мог обойти backoff и повторно захватить просроченный processing без увеличения attempt. [Захват в PostgreSQL](src/storage/postgres.py) теперь разрешён только для queued без действующей задержки; lease восстанавливает dispatcher. Реальный SQL-тест покрывает обе ситуации и устаревшие completion/retry. +- После выключения Docker обнаружены queued jobs с published outbox при пустой очереди брокера. Причина исчезновения сообщений не установлена; установлен дефект восстановления: next_attempt_at=infinity исключал такие jobs из повторной публикации навсегда. Добавлен 60-секундный watchdog; реальный SQL-тест проверяет повторную публикацию без увеличения attempt и без создания события на каждом проходе. +- Добавлен отключённый по умолчанию одноразовый crash-hook worker после S3; нужны реальные проверки этой границы, а не только mocks. + +## Ограничения + +Один Docker-host, одна БД, брокер, S3 и LB не обеспечивают устойчивость к потере host/хранилищ. Проверены конкретные процессные отказы и перезапуски с сохранёнными volumes. At-least-once не означает exactly-once. + +Failed staging сохраняется бессрочно для явного replay; автоматической политики удаления таких исходников нет. Полного аудита операторских действий нет, сохраняется replayed_at. Исторические student-tests lab4, не перечисленные в командах выше, не заявляются как пройденные. + +Запуск из отдельного чистого clone не проверялся: сборка, миграции и bootstrap проверялись в текущем рабочем дереве. Изменения не закоммичены и PR автоматически не создан. Окончательную оценку определяет преподаватель с учётом защиты; успешные тесты не являются доказательством отсутствия любых ошибок. diff --git a/compose.yaml b/compose.yaml new file mode 100644 index 00000000..8b0b05d6 --- /dev/null +++ b/compose.yaml @@ -0,0 +1,311 @@ +x-api: &api + build: + context: . + environment: + APP_HOST: 0.0.0.0 + APP_PORT: 8080 + APP_ENV: lab5 + DATABASE_URL_FILE: /run/secrets/database_url + SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600} + IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400} + CURSOR_SECRET_FILE: /run/secrets/cursor_secret + S3_ENDPOINT: http://s3:8333 + S3_BUCKET: messenger-attachments + S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id + S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key + BROKER_URL_FILE: /run/secrets/broker_url + MAX_UPLOAD_BYTES: "10485760" + MAX_IMAGE_PIXELS: "40000000" + UPLOAD_CONCURRENCY: "1" + UPLOAD_TIMEOUT_SECONDS: "30" + UPLOAD_FAULT_AFTER_S3_PUT: ${UPLOAD_FAULT_AFTER_S3_PUT:-0} + OUTBOX_FAULT_AFTER_COMMIT: ${OUTBOX_FAULT_AFTER_COMMIT:-0} + MAX_PENDING_JOBS: ${MAX_PENDING_JOBS:-100} + secrets: + - database_url + - cursor_secret + - s3_access_key_id + - s3_secret_access_key + - broker_url + depends_on: + s3-init: + condition: service_completed_successfully + migrate: + condition: service_completed_successfully + broker: + condition: service_healthy + expose: + - "8080" + stop_grace_period: 10s + healthcheck: + test: ["CMD", "/app/.venv/bin/python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"] + interval: 5s + timeout: 3s + retries: 12 + start_period: 10s + deploy: + resources: + limits: + cpus: "0.50" + memory: 512M + +services: + broker: + image: rabbitmq:4.1.2-alpine + environment: + RABBITMQ_DEFAULT_USER: messenger + entrypoint: ["/bin/sh", "/usr/local/bin/rabbitmq-secret-entrypoint.sh"] + command: ["rabbitmq-server"] + secrets: + - rabbitmq_password + volumes: + - rabbitmq_data:/var/lib/rabbitmq + - ./infra/rabbitmq-secret-entrypoint.sh:/usr/local/bin/rabbitmq-secret-entrypoint.sh:ro + healthcheck: + test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"] + interval: 5s + timeout: 3s + retries: 20 + start_period: 10s + deploy: + resources: + limits: + cpus: "0.50" + memory: 256M + + s3: + image: chrislusf/seaweedfs:4.17@sha256:186de7ef977a20343ee9a5544073f081976a29e2d29ecf8379891e7bf177fbe9 + command: ["server", "-dir=/data", "-ip=s3", "-ip.bind=0.0.0.0", "-s3", "-s3.config=/run/secrets/s3_config", "-master.volumeSizeLimitMB=128", "-volume.max=64", "-s3.port.iceberg=0", "-filer.exposeDirectoryData=false"] + secrets: + - s3_config + volumes: + - s3_data:/data + - ./infra/filer.toml:/etc/seaweedfs/filer.toml:ro + healthcheck: + test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://localhost:9333/cluster/status"] + interval: 5s + timeout: 3s + retries: 20 + start_period: 15s + deploy: + resources: + limits: + cpus: "1.0" + memory: 512M + + s3-init: + build: + context: . + command: ["/app/.venv/bin/python", "-m", "src.bootstrap_s3"] + environment: + S3_ENDPOINT: http://s3:8333 + S3_BUCKET: messenger-attachments + S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_admin_key + S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_admin_secret + secrets: + - s3_admin_key + - s3_admin_secret + depends_on: + s3: + condition: service_healthy + restart: "no" + + postgres: + image: postgres:17.6-alpine + environment: + POSTGRES_DB: messenger + POSTGRES_USER: messenger + POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password + secrets: + - postgres_password + volumes: + - postgres_data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"] + interval: 5s + timeout: 3s + retries: 12 + start_period: 5s + deploy: + resources: + limits: + cpus: "0.75" + memory: 512M + + migrate: + build: + context: . + command: ["/app/.venv/bin/python", "-m", "src.migrate"] + environment: + DATABASE_URL_FILE: /run/secrets/database_url + CURSOR_SECRET_FILE: /run/secrets/cursor_secret + secrets: + - database_url + - cursor_secret + depends_on: + postgres: + condition: service_healthy + restart: "no" + deploy: + resources: + limits: + cpus: "0.25" + memory: 128M + + reconcile: + healthcheck: + disable: true + build: + context: . + entrypoint: ["/app/.venv/bin/python", "-m", "src.reconcile"] + command: ["--apply", "--loop"] + restart: unless-stopped + environment: + DATABASE_URL_FILE: /run/secrets/database_url + CURSOR_SECRET_FILE: /run/secrets/cursor_secret + S3_ENDPOINT: http://s3:8333 + S3_BUCKET: messenger-attachments + S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_admin_key + S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_admin_secret + STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400} + secrets: + - database_url + - cursor_secret + - s3_admin_key + - s3_admin_secret + depends_on: + postgres: + condition: service_healthy + s3: + condition: service_healthy + deploy: + resources: + limits: + cpus: "0.25" + memory: 128M + + api-a: + <<: *api + + api-b: + <<: *api + + worker: + restart: unless-stopped + healthcheck: + test: ["CMD", "/app/.venv/bin/python", "-c", "import time,pathlib; assert time.time()-float(pathlib.Path('/tmp/worker-heartbeat').read_text()) < 15"] + interval: 5s + timeout: 3s + retries: 3 + deploy: + resources: + limits: + cpus: "2.0" + memory: 4G + build: + context: . + command: ["/app/.venv/bin/python", "-m", "src.worker"] + environment: + APP_ENV: lab5 + DATABASE_URL_FILE: /run/secrets/database_url + CURSOR_SECRET_FILE: /run/secrets/cursor_secret + S3_ENDPOINT: http://s3:8333 + S3_BUCKET: messenger-attachments + S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id + S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key + BROKER_URL_FILE: /run/secrets/broker_url + WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2} + WORKER_FAULT_AFTER_S3_PUT: ${WORKER_FAULT_AFTER_S3_PUT:-0} + RENDER_MEMORY_MB: ${RENDER_MEMORY_MB:-768} + WORKER_PREFETCH: ${WORKER_PREFETCH:-2} + JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5} + JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60} + STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400} + secrets: + - database_url + - cursor_secret + - s3_access_key_id + - s3_secret_access_key + - broker_url + depends_on: + s3-init: + condition: service_completed_successfully + migrate: + condition: service_completed_successfully + broker: + condition: service_healthy + stop_grace_period: 30s + + outbox: + healthcheck: + disable: true + build: + context: . + command: ["/app/.venv/bin/python", "-m", "src.outbox"] + environment: + APP_ENV: lab5 + DATABASE_URL_FILE: /run/secrets/database_url + CURSOR_SECRET_FILE: /run/secrets/cursor_secret + BROKER_URL_FILE: /run/secrets/broker_url + OUTBOX_DISPATCH_INTERVAL_SECONDS: "1" + JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5} + secrets: + - database_url + - cursor_secret + - broker_url + depends_on: + migrate: + condition: service_completed_successfully + broker: + condition: service_healthy + restart: unless-stopped + + load_balancer: + image: haproxy:3.0.11-alpine + ports: + - "127.0.0.1:8080:8080" + volumes: + - ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro + depends_on: + api-a: + condition: service_healthy + api-b: + condition: service_healthy + healthcheck: + test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:8080/health/ready"] + interval: 5s + timeout: 3s + retries: 12 + start_period: 5s + deploy: + resources: + limits: + cpus: "0.25" + memory: 128M + +secrets: + s3_config: + file: ./secrets/s3.json + s3_admin_key: + file: ./secrets/s3_admin_key + s3_admin_secret: + file: ./secrets/s3_admin_secret + s3_access_key_id: + file: ./secrets/s3_access_key_id + s3_secret_access_key: + file: ./secrets/s3_secret_access_key + postgres_password: + file: ./secrets/postgres_password + database_url: + file: ./secrets/database_url + cursor_secret: + file: ./secrets/cursor_secret + rabbitmq_password: + file: ./secrets/rabbitmq_password + broker_url: + file: ./secrets/broker_url + +volumes: + s3_data: + postgres_data: + rabbitmq_data: diff --git a/evidence/README.md b/evidence/README.md index 8df70015..5a4980a3 100644 --- a/evidence/README.md +++ b/evidence/README.md @@ -3,3 +3,15 @@ Храните небольшие обезличенные результаты: команды и stdout проверок, JSON/CSV агрегатов нагрузки, EXPLAIN, dashboard JSON, текст разбора отказа. Укажите дату, commit, аппаратные лимиты и профиль нагрузки. Не добавляйте .env, session tokens, cookies, приватные ключи, credentials, реальные сообщения, raw dumps и signed URLs. Для скриншотов скрывайте секреты. Большие логи/бинарные traces сохраняйте отдельно по правилам преподавателя; одного скриншота без методики недостаточно. + +## Проверки решения, 26–27 октября 2026 + +- [environment.json](environment.json): среда, лимиты, образы и хеши исходников рабочего дерева. +- [checklist-review.txt](checklist-review.txt): хронология проверок, включая первоначальные ошибки; ранние ограничения не описывают финальное состояние. +- [pipeline-integration.json](pipeline-integration.json): stop/restart, crash COMMIT/publish и S3/ack, lease, retry/replay, cleanup, Linux memory. +- [pipeline-invariants.json](pipeline-invariants.json): реальные SQL-инварианты, watchdog, ACL, форматы вариантов, cleanup locking, DLQ, timeout. +- [job-contract.json](job-contract.json): проверка реальных outbox payloads по обязательной схеме и redelivery полного сообщения. +- [resilience.json](resilience.json): stdout публичных persistence/failover и состояния контейнеров до/после. +- `real-c{1,2,4}-p{1,2,4}-s{512,2400}.json`: шесть профилей нагрузки; параметры, каждый запрос, метрики, backpressure и восстановление. + +Методика, таблица результатов и ограничения: [REPORT.md](../REPORT.md). Mock-метрики локальных регрессий в исследование не включены. Данные всех пользователей/вложений синтетические; сессии и credentials не сохранялись. diff --git a/evidence/checklist-review.txt b/evidence/checklist-review.txt new file mode 100644 index 00000000..72a52404 --- /dev/null +++ b/evidence/checklist-review.txt @@ -0,0 +1,113 @@ +PR checklist review, 2026-10-26 + +make check: PASS (offline skeleton check; application not tested). +python student-tests/pipeline_critical.py: 8 tests, OK. +python student-tests/remaining_regressions.py: 7 tests, OK. +docker compose config --quiet: exit 0. +git diff 276dbda -- README.md contracts tests: empty. +git ls-files secrets .env: empty. +git check-ignore .env secrets/postgres_password: both ignored. + +Secret scan scope: +- 90 Git-tracked working-tree files. +- 128 blobs reachable through git rev-list --objects --all. +- Compared against 10 locally configured secret values, read without printing. +- Searched private-key headers, AWS keys, GitHub/OpenAI token formats, + JWT-shaped tokens, and literal PostgreSQL/AMQP/Redis credential URLs. +- No findings. Template substitutions in credential URLs were excluded. +- This is a scoped heuristic check, not proof that every possible secret format is absent. + +Mock benchmark numbers printed by the regression suite are not load-test evidence. + +Integration attempt: +- Started installed Docker Desktop; Docker Engine 29.8.0 became available. +- make validate PYTHON=.venv/bin/python: PASS, full OpenAPI 3.1 / JSON Schema. +- docker compose up -d --build: images built, exit 1 during startup. +- migrate: exited 0. +- s3-init: PASS: private S3 bucket ready; exited 0. +- broker (rabbitmq:4.1.2-alpine): exited 1. + error: RABBITMQ_DEFAULT_PASS_FILE is set but deprecated + error: deprecated environment variables detected + Please use a configuration file instead +- API, worker, outbox and load_balancer remained Created, not running. +- Public smoke and failure experiments cannot run until broker starts. +- Volumes were preserved. PostgreSQL, S3 and reconcile were left running. + +Follow-up after RabbitMQ startup fix, 2026-10-26: +- Removed deprecated RABBITMQ_DEFAULT_PASS_FILE from Compose. +- Read password from Docker secret in infra/rabbitmq-secret-entrypoint.sh; + export inside container and exec stock entrypoint with rabbitmq-server. +- docker compose up -d: exit 0. +- docker compose ps -a: broker, api-a, api-b, worker, load_balancer, + postgres and s3 healthy; outbox/reconcile running; migrate/s3-init exited 0. + +Actual check output (exit 0): +python3 scripts/check_contract.py +PASS: offline skeleton structure, local refs and Python syntax; application NOT tested. +.venv/bin/python tests/smoke.py --base-url "http://localhost:8080" +PASS: lab5 public API scenario; infrastructure and grading evidence remain separate. + +.venv/bin/python student-tests/attachments.py (exit 0): +PASS: PNG/JPEG originals, metadata, attachment-only messages and ACL +PASS: filename, MIME, malformed image, byte/pixel and chunked limits +PASS: attachment scope, duplicates and idempotency normalization +PASS: inclusive 10 MiB and 40 million pixel boundaries + +.venv/bin/python student-tests/pipeline_critical.py (exit 0): +Ran 8 tests in 0.012s +OK +.venv/bin/python student-tests/remaining_regressions.py (exit 0): +Ran 7 tests in 0.576s +OK + +Mock benchmark output is excluded: it is not real load evidence. +Failure/restart experiments and real load profiles remain unverified. +Stack left running; no volumes deleted. + +Follow-up verification, 2026-10-26/27: +- Real fault suite: evidence/pipeline-integration.json. +- Real SQL, Linux, ACL and cleanup locking checks: evidence/pipeline-invariants.json. +- Public persistence and both API failovers with before/after container states: + evidence/resilience.json. Initial isolated run passed; overlapping exploratory + restart/failover attempts are not counted as successful evidence. +- S3 permissions: PASS: API identity cannot create buckets; maintenance identity has separate admin access. +- Idempotency: PASS: concurrent idempotency through load balancer. +- After Docker was stopped, 10 queued jobs had published outbox records but + RabbitMQ reported 0 ready and 0 unacknowledged messages. Original disappearance + cause is not established. Added 60s unclaimed-job publication watchdog. +- Fixed redelivery bypassing retry delay / expired lease attempt accounting. +- Current-worktree secret scan: 107 tracked/nonignored files, 9 local single-line + secret values >=12 bytes and private-key/GitHub/AWS patterns; no findings. + This complements the earlier history scan; neither is a universal guarantee. +- Test cleanup deleted four explicitly selected orphan objects from its own + crashed attempt. Active result objects and all volumes were preserved. + +Final real load series, 2026-10-27: +- Six runs: workers/prefetch 1/1, 2/2, 4/4; PNG 512 and 2400 square. +- Each run: 160 uploads, client concurrency 16, MAX_PENDING_JOBS=32. +- Total 960 requests = 220 accepted (all ready) + 740 expected HTTP 429. +- Each run: failed=0, timed_out=0, metric_errors=[], drained=true, + recovery_upload_ready=true. Observed pending never exceeded 32. +- Source records: real-c*-p*-s*.json; table/formulas/limitations in REPORT.md. +- Earlier small runs were exploratory and replaced by this comparable series. +- Job schema: 425 real outbox payloads validated; persisted redelivery is no-op. + Detailed result: job-contract.json. +- Offline check, full OpenAPI validation, 8 + 7 local regressions passed again + after the final PostgreSQL watchdog change. Mock throughput is not evidence. +- Final public persistence: PASS, actual service StartedAt values changed. +- Final public failovers: PASS for both API replicas. First recovery times: + 2.436s (113 successful writes, 1 failed attempt) and + 4.675s (95 successful writes, 2 failed attempts). +- Final public smoke after those failures: PASS. + +Final confirmation on the watchdog build, 2026-10-27: +- All seven pipeline integration stages passed again; pipeline-integration.json + contains the final run, not the earlier exploratory attempts. +- Attachments, S3 permissions, concurrent idempotency and public smoke passed again. +- Worker metrics: pending=0, concurrency=2, prefetch=2. +- Working RabbitMQ queue: messages_ready=0, messages_unacknowledged=0. +- DLQ contains 3 deliberately malformed test messages (not pending image work). +- broker, worker, both APIs, PostgreSQL, S3 and LB healthy; outbox/reconcile running; + migrate and s3-init exited 0. Normal configuration restored; volumes preserved. +- Final working-tree secret scan: 108 files, 9 local values plus key patterns, + no findings. git diff --check passed; README/contracts/public tests unchanged. diff --git a/evidence/environment.json b/evidence/environment.json new file mode 100644 index 00000000..85931c61 --- /dev/null +++ b/evidence/environment.json @@ -0,0 +1,170 @@ +{ + "recorded_at_utc": "2026-10-27T09:21:55.484450+00:00", + "base_commit": "a1bb6f8bded13c0fae2a525dcdbda8d6a890b971", + "branch": "solution/BVT2301-Kozyrev", + "note": "Results apply to this working tree, not only the base commit. Per-profile load settings are recorded in real-c*-p*-s*.json.", + "host": "macOS-26.1-arm64-arm-64bit-Mach-O", + "docker": { + "server": "29.8.0", + "cpus": 10, + "memory_bytes": 10416644096 + }, + "compose_version": "5.5.1", + "sources_sha256": { + "Dockerfile": "efede941e2ccf21cd1e70ebbf8af7647209bd030b2962f9446bee497a2ed6a8c", + "Makefile": "46681d5c9aa3b09be285ce5b6db39bb9e3c4348e3918505f916b865b8831a681", + "compose.yaml": "5e1a5813ee0f663334214e243593edee3528e4d49cbe1c89fbb26a3a5ed50d89", + "infra/README.md": "0e804fa51debba4c38d3cc84abf4704897665e0335b272060363f03aadd6354f", + "infra/filer.toml": "ae3c01bf7708f5d4a319fdb454f753be0f137fe7c31269f1c413cbccc860f5c7", + "infra/haproxy.cfg": "23c9b869170633e3c6a8c41bbd3f1102bc97bf213e0d7cfed9d461af91216fd3", + "infra/rabbitmq-secret-entrypoint.sh": "94ab6376d9fa6e08452172f69cbf467735f6edd781e07d368041769e76b3ac19", + "migrations/001_initial.sql": "062e58f5535d1ab6cd89d2347479a2a6b08a09e04665f7428ce41f89b39a19d1", + "migrations/002_history_cache_version.sql": "826c8243bc148caaa7339070826a7c2928073315a40dba7fa5a3d7421d980a4d", + "migrations/003_idempotency.sql": "d650ea7c1379928a013e035977a77026d2b8b09ad5b2ea97eecb545e7450353a", + "migrations/004_attachments.sql": "d760453ba130c94c403f5b04222649fc131ab4d091edd0267ede901fdf82a79e", + "migrations/005_async_attachments.sql": "77d0e180bc5661a1824224bd24152992b55e3ec19afa8fcbde66e7ff49216f59", + "migrations/006_level4_reliability.sql": "de6a2c3ce574abda4ddea1835c30c2845dead50f1c33f705a3b82f381bc6d1fe", + "migrations/007_job_measurements.sql": "d520a39c5a98ed471faef8371da9521bab249a53e190a7a6719788da1242029d", + "pyproject.toml": "056adefdef2f9a924bbf8e05b680a0e2dac553a097b2f351d2daecfa0a168ed2", + "scripts/audit.py": "5c695c70058b24e975ceca310b2bdc879858081e5ce82b444ea9eab6b9c0d2ed", + "scripts/check_contract.py": "7c4402c2a6ab80d3b4676e1866d6ecc87b6daea14c080bcb9be9a7e8b959d671", + "scripts/init-secrets.py": "f83905a882c82a48e5f0c04fc83f839c43d0c1bc1f7b3cd74aa70e8562bce2af", + "scripts/record-environment.py": "3a97282e349f99e8245f757599ed45e72c240a9445b577a22fdf13a4d941010e", + "scripts/restart.example.sh": "0328246ba75defbced992104e986fde973f63e3830d83251eced5953263e944b", + "scripts/restart.sh": "2424309b58bf9670c0c39d52c0b02d0bbc5b3454b774106ee69ef323705e5170", + "scripts/run-track1-profiles.py": "7d59285bac1ce4db391eafaf47fd7d8e8904e4792ac6580fa48347ebcc7b0deb", + "scripts/stop-one.example.sh": "1bcf34394520b89d95b7605ec56675be799eea920e83cfbba306d5dc0a6d9811", + "scripts/stop-one.sh": "c69fdb1891e0c49f41d3892f2d108ecd38b2f59d505c8fe991023bb5c9b08a5e", + "scripts/track1.py": "c5edbd3c6aa68e3a9b638dab2572239806b8d4605b5feebaab6e05cd8ccb9fc7", + "scripts/validate_openapi.py": "4e981acbfc10dac58817bf922d5896e996798551fedc1b2385130e1cddb13d54", + "scripts/verify-persistence.py": "6c366bf4d85802bcffe65ae3d45d00736b184033313dabb60b80e1270d8874f1", + "scripts/verify-resilience.py": "13d9d056638fcfed6616ac159a5d149cfdd26f6090c51fabbb3ab0cde9ac8199", + "src/__init__.py": "01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b", + "src/attachments.py": "43421b711a5be717c49522688c2fc5035f5ddb7898d02006e51791e66861b26b", + "src/bootstrap_s3.py": "2c43f5540e63f004a88e08961a72293546fbc5210f46bc719b90b9f5c0f3f9bd", + "src/broker.py": "551fa504d9bfdf3871b1a1cada8210ad73bd75fef5c5a1737c958ec2d53a3587", + "src/config.py": "db51a87564d3e84feddc9e824e37088a0a164af38361e2d131db34de6ec66542", + "src/cursor.py": "a9192f2445042b8d4ddac57031718ef558990a52eb2922827074479b7f686775", + "src/errors.py": "07f8860b45cadd242d0d9f268c4b806e35353f33b11bbde0ab3d50acda86a80a", + "src/main.py": "6980cdbb09c3b356ff870ee7fc972d85e835c58228d3c0ea27de8a668dc41ece", + "src/migrate.py": "1d164c8921df2b5369e30b1f8936c2b71858c5279070f0fd593f62f66e1f0b7d", + "src/models.py": "924ace12fb9bcdd222ff58ea4d40cd57ba40290c8fbdb335c267ec35f3bad930", + "src/outbox.py": "3fbe66c516e16efa00982f6cd4c4c46fbe04d7f5f98f48d76cece3ae1911d9ad", + "src/pipeline_metrics.py": "61df2e1f5d802ac21d03a3a8e0a653a25b84d0cabfdcccd0aab6327bbb0da27d", + "src/reconcile.py": "e36163601f3acead6d698fbad747044b7026b1af07cdea5223486bf5e1b43d53", + "src/render.py": "a2eb4001d934b09f39cc555cf56c06159dd6b40a59bc6a8c3ba64ae2612dabe5", + "src/replay.py": "5490e6fd42052a6a0f1449bef3ddd4ffbe3f847141743e6b43e28c858d510dc1", + "src/routes.py": "d629efcdea1c0d2b2202a57ea6a1804164fb8cc220de3b32850c30bdf97bbf11", + "src/storage/__init__.py": "05ef66318c4c436eaafc493072640817219ab37bce464a265a2306233389064a", + "src/storage/postgres.py": "07de530b2287cbfb358fc677f71dbdd9b15387f7312a3d40df095cc853fe5ad4", + "src/storage/s3.py": "b9b36ab1e86d084b58bffc2f55b16d260d234b2beeb35f8461e9f05222d4b843", + "src/worker.py": "e72ac3629f7f2976571a9958fcb0590289d388c0b031c9529f342857731e3303", + "student-tests/attachment_resilience.py": "8a4ad367263a5d5a597fcd4a2a657c80946d19a3bbedeebdfffbf75f45531e5d", + "student-tests/attachments.py": "f80ef87aba3d409b5689e1597e40f15428d8e65211529cef2dbc4b853682e995", + "student-tests/idempotency_concurrency.py": "db5d139009e4660ac1801af45d10e24665aa81cba737731e527c0f5b4152231e", + "student-tests/job_contract.py": "4b65bae8500c2f6b40ce5faf5435e0dfbb98c71c7c17dd6789c505d923d30273", + "student-tests/pipeline_critical.py": "49a6ee8f64bd75c9890e4a9b7767595ddaa38af3a4abfc07131855ea7a1308f5", + "student-tests/pipeline_integration.py": "12dec32832afb4f7392188f6e265c0d51c0f0dc8b4adb221b9a979e6e6f07550", + "student-tests/pipeline_invariants.py": "49b846db34327fcbcabf4dc2f0a81359a0072a263ece262a435e28e4aca868af", + "student-tests/readiness_failover.py": "0ef8355d85be66597d7d44eb7308dcb58c6c9a174d18f360b2d6d22fdcfbed14", + "student-tests/readiness_unavailable.py": "46e3f680a3ff9cb79e95568004d97f1752dbdcd88fd4b5f160a9cc38af65a237", + "student-tests/reconcile.py": "9c82be28eb5a41b38c9ddd63dbf65d3edc238ca09ffa4612fc4b679d64487173", + "student-tests/reconcile_audit.py": "1bcca4da1030901e45c1369672c8363f9a5b08513c0270dc8a6f73b1f0b74221", + "student-tests/reconcile_benchmark.py": "9ae60097cb70e63baf5653559e727e02da246827a256e02580f03b5d47e1ca15", + "student-tests/remaining_regressions.py": "a5e61fd4a2c6a92d52e1b0d051b3931f699d8b0b2d73d2ce5c2553e313efe3e6", + "student-tests/review_regressions.py": "de58f63b2c794c2775247ddd3fccc523870ccbe673ffa404fdd566274d1d41de", + "student-tests/s3_permissions.py": "8c42c8ef3f7abb73bfdaa9d395874ae44041501f540213a1276c5a8bbe5e9c80", + "student-tests/session_expiry.py": "ba8ead4356e4d12c4047ac2b5704d3d3342f71c37b3c13f65a0a9d7c37186ad6", + "student-tests/upload_memory.py": "ed6c013b9e29a826191e1ca904e0546449f0e6ed9ecddbf5aa256d0768f64604", + "uv.lock": "908af5e1ce932a1fe9ccd60401c28957d001e846d8d9b524f2e688534373c9de" + }, + "services": { + "postgres": { + "image": "postgres:17.6-alpine", + "image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94", + "state": "running", + "health": "healthy", + "memory_limit_bytes": 536870912, + "nano_cpus_limit": 750000000, + "volumes": [ + "mtusi2026lab5_postgres_data" + ] + }, + "s3": { + "image": "chrislusf/seaweedfs:4.17@sha256:186de7ef977a20343ee9a5544073f081976a29e2d29ecf8379891e7bf177fbe9", + "image_id": "sha256:186de7ef977a20343ee9a5544073f081976a29e2d29ecf8379891e7bf177fbe9", + "state": "running", + "health": "healthy", + "memory_limit_bytes": 536870912, + "nano_cpus_limit": 1000000000, + "volumes": [ + "mtusi2026lab5_s3_data" + ] + }, + "broker": { + "image": "rabbitmq:4.1.2-alpine", + "image_id": "sha256:8f1a4555a52cb5bbd91fcacfed98f230a1762ef7a08cc05d7f711b8a31fb5ca5", + "state": "running", + "health": "healthy", + "memory_limit_bytes": 268435456, + "nano_cpus_limit": 500000000, + "volumes": [ + "mtusi2026lab5_rabbitmq_data" + ] + }, + "worker": { + "image": "mtusi2026lab5-worker", + "image_id": "sha256:869ddc8aed23c7d0d6d387c7f2e88bdfdb3bbfc70d8dd2c2571508051da43132", + "state": "running", + "health": "healthy", + "memory_limit_bytes": 4294967296, + "nano_cpus_limit": 2000000000, + "volumes": [] + }, + "outbox": { + "image": "mtusi2026lab5-outbox", + "image_id": "sha256:6f97c6b53c9616356ea32cc60c27da36c5e6049e34ffdcce4e3a98d6ab255fb2", + "state": "running", + "health": null, + "memory_limit_bytes": 0, + "nano_cpus_limit": 0, + "volumes": [] + }, + "reconcile": { + "image": "mtusi2026lab5-reconcile", + "image_id": "sha256:b03833302fe18069ab432b09cf75a79ec80ee11d5640f2e441f308bcfcde9453", + "state": "running", + "health": null, + "memory_limit_bytes": 134217728, + "nano_cpus_limit": 250000000, + "volumes": [] + }, + "api-a": { + "image": "mtusi2026lab5-api-a", + "image_id": "sha256:dbe50ae4bd1c27d0c3861f3ecbeedc0c73101e7ac14ed0393cccf4114046f042", + "state": "running", + "health": "healthy", + "memory_limit_bytes": 536870912, + "nano_cpus_limit": 500000000, + "volumes": [] + }, + "api-b": { + "image": "mtusi2026lab5-api-b", + "image_id": "sha256:f0ce767b22b37349e57232dbb5fc6b17a0b16c75b60a39088eae16549800b1c7", + "state": "running", + "health": "healthy", + "memory_limit_bytes": 536870912, + "nano_cpus_limit": 500000000, + "volumes": [] + }, + "load_balancer": { + "image": "haproxy:3.0.11-alpine", + "image_id": "sha256:e49b7beb96f4dba79f9a671515b469a6e516a2b88ff0681a820b184bba1effbd", + "state": "running", + "health": "healthy", + "memory_limit_bytes": 134217728, + "nano_cpus_limit": 250000000, + "volumes": [] + } + } +} diff --git a/evidence/job-contract.json b/evidence/job-contract.json new file mode 100644 index 00000000..5b39afb8 --- /dev/null +++ b/evidence/job-contract.json @@ -0,0 +1,6 @@ +{ + "status": "PASS", + "validated_outbox_payloads": 425, + "extra_payload_fields": false, + "persisted_redelivery_noop": true +} diff --git a/evidence/pipeline-integration.json b/evidence/pipeline-integration.json new file mode 100644 index 00000000..757a4281 --- /dev/null +++ b/evidence/pipeline-integration.json @@ -0,0 +1,38 @@ +[ + { + "test": "stopped worker, queued message, broker/API restart, ready", + "status": "PASS", + "attachment_id": "ec83e811-1507-40a3-9cec-d79d501e6a4f" + }, + { + "test": "duplicate delivery and replay of ready are no-ops", + "status": "PASS", + "object_count": 4 + }, + { + "test": "crash after COMMIT before publish recovered from existing outbox", + "status": "PASS", + "attachment_id": "357956fa-c571-45fa-a0d9-18ce24f382eb" + }, + { + "test": "worker crash after S3 before commit/ack, lease recovery", + "status": "PASS", + "attempt": 2, + "objects_before_cleanup": 8 + }, + { + "test": "targeted orphan cleanup preserves committed attempt", + "status": "PASS", + "deleted": 4 + }, + { + "test": "bounded transient retry, failed registry and explicit replay", + "status": "PASS", + "attempts_before_replay": 2 + }, + { + "test": "Linux renderer memory limit", + "status": "PASS", + "returncode": 21 + } +] diff --git a/evidence/pipeline-invariants.json b/evidence/pipeline-invariants.json new file mode 100644 index 00000000..29812389 --- /dev/null +++ b/evidence/pipeline-invariants.json @@ -0,0 +1,11 @@ +[ + "actual thumbnail/crop/WebP formats, dimensions and removed EXIF", + "anonymous S3 403 on existing object; variant ACL 404", + { + "maintenance_wait_seconds": 1.0019877920000226 + }, + "real SQL: queued watchdog republishes without attempt increment/flood; duplicate respects backoff; expired lease counted; stale completion/retry fenced", + "permanent invalid_image reaches failed on attempt 1", + "malformed broker message delivered to DLQ", + "Linux render subprocess killed/reaped on deadline" +] diff --git a/evidence/real-c1-p1-s2400.json b/evidence/real-c1-p1-s2400.json new file mode 100644 index 00000000..df014794 --- /dev/null +++ b/evidence/real-c1-p1-s2400.json @@ -0,0 +1,3586 @@ +{ + "run_name": "real-c1-p1-s2400", + "settings": { + "workers": 1, + "prefetch": 1, + "max_pending": 32, + "worker_limits": { + "Memory": 4294967296, + "NanoCpus": 2000000000 + } + }, + "workload": { + "images": 160, + "concurrency": 16, + "size": 2400, + "bytes": 22196, + "type": "closed-loop burst" + }, + "results": { + "accepted": 37, + "rejected_429": 123, + "ready": 37, + "failed": 0, + "timed_out": 0, + "elapsed_seconds": 35.50305537500026, + "ready_per_second": 1.042163825315548, + "ready_latency_ms": { + "0.5": 17455.534, + "0.95": 30435.623, + "0.99": 30481.177 + }, + "queue_wait_ms": { + "0.5": 16404.236, + "0.95": 29505.011, + "0.99": 29518.091 + }, + "max_sampled_rss_bytes": 251621376, + "max_sampled_pending": 32, + "sample_count": 14, + "drained": true, + "recovery_upload_ready": true + }, + "records": [ + { + "index": 0, + "status": 202, + "upload_seconds": 0.06751475000010032, + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "terminal": "ready", + "ready_seconds": 1.6049423330000536 + }, + { + "index": 2, + "status": 202, + "upload_seconds": 0.07451191700010895, + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "terminal": "ready", + "ready_seconds": 2.5987475829997493 + }, + { + "index": 3, + "status": 202, + "upload_seconds": 0.15237262500022553, + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "terminal": "ready", + "ready_seconds": 3.044046832999811 + }, + { + "index": 1, + "status": 202, + "upload_seconds": 0.15928762500016092, + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "terminal": "ready", + "ready_seconds": 4.0280258329999015 + }, + { + "index": 6, + "status": 202, + "upload_seconds": 0.1937505420000889, + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "terminal": "ready", + "ready_seconds": 5.380929375000051 + }, + { + "index": 5, + "status": 202, + "upload_seconds": 0.25458195800001704, + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "terminal": "ready", + "ready_seconds": 5.926423834000161 + }, + { + "index": 9, + "status": 202, + "upload_seconds": 0.2665866669999559, + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "terminal": "ready", + "ready_seconds": 6.856949083000018 + }, + { + "index": 8, + "status": 202, + "upload_seconds": 0.27834416700034126, + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "terminal": "ready", + "ready_seconds": 7.6031478749996495 + }, + { + "index": 11, + "status": 202, + "upload_seconds": 0.34458629200025825, + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "terminal": "ready", + "ready_seconds": 8.480898750000051 + }, + { + "index": 10, + "status": 202, + "upload_seconds": 0.38872983300007036, + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "terminal": "ready", + "ready_seconds": 9.317503833000046 + }, + { + "index": 4, + "status": 202, + "upload_seconds": 0.45357995899985326, + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "terminal": "ready", + "ready_seconds": 10.317705458000091 + }, + { + "index": 13, + "status": 202, + "upload_seconds": 0.4587804170000709, + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "terminal": "ready", + "ready_seconds": 11.226020000000062 + }, + { + "index": 7, + "status": 202, + "upload_seconds": 0.4824979580002946, + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "terminal": "ready", + "ready_seconds": 12.050059208999755 + }, + { + "index": 15, + "status": 202, + "upload_seconds": 0.5459110000001601, + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "terminal": "ready", + "ready_seconds": 12.942837208999663 + }, + { + "index": 12, + "status": 202, + "upload_seconds": 0.5708613749998221, + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "terminal": "ready", + "ready_seconds": 13.75311499999998 + }, + { + "index": 17, + "status": 202, + "upload_seconds": 0.5155874170000061, + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "terminal": "ready", + "ready_seconds": 14.716898958000002 + }, + { + "index": 14, + "status": 202, + "upload_seconds": 0.6448091249999379, + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "terminal": "ready", + "ready_seconds": 15.549343209000199 + }, + { + "index": 19, + "status": 202, + "upload_seconds": 0.49970345900010216, + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "terminal": "ready", + "ready_seconds": 16.348253000000113 + }, + { + "index": 16, + "status": 202, + "upload_seconds": 0.6207672089999505, + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "terminal": "ready", + "ready_seconds": 17.45553374999963 + }, + { + "index": 22, + "status": 202, + "upload_seconds": 0.49496875000022555, + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "terminal": "ready", + "ready_seconds": 18.299862207999922 + }, + { + "index": 18, + "status": 202, + "upload_seconds": 0.6193955420003476, + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "terminal": "ready", + "ready_seconds": 19.244367915999646 + }, + { + "index": 24, + "status": 202, + "upload_seconds": 0.5143247089999932, + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "terminal": "ready", + "ready_seconds": 20.117251458000283 + }, + { + "index": 20, + "status": 202, + "upload_seconds": 0.6738845830000173, + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "terminal": "ready", + "ready_seconds": 21.204034374999992 + }, + { + "index": 26, + "status": 202, + "upload_seconds": 0.4356542079999599, + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "terminal": "ready", + "ready_seconds": 22.060257416999775 + }, + { + "index": 21, + "status": 202, + "upload_seconds": 0.6971938329998011, + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "terminal": "ready", + "ready_seconds": 22.966771750000134 + }, + { + "index": 28, + "status": 202, + "upload_seconds": 0.48609683299991957, + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "terminal": "ready", + "ready_seconds": 23.781251124999926 + }, + { + "index": 23, + "status": 202, + "upload_seconds": 0.7013296250001986, + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "terminal": "ready", + "ready_seconds": 24.740702791999865 + }, + { + "index": 29, + "status": 202, + "upload_seconds": 0.5200753330000225, + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "terminal": "ready", + "ready_seconds": 25.61710149999999 + }, + { + "index": 25, + "status": 202, + "upload_seconds": 0.6878797080003096, + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "terminal": "ready", + "ready_seconds": 26.538217667000026 + }, + { + "index": 31, + "status": 202, + "upload_seconds": 0.5770468749997235, + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "terminal": "ready", + "ready_seconds": 27.440981917000045 + }, + { + "index": 27, + "status": 202, + "upload_seconds": 0.7179129169999214, + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "terminal": "ready", + "ready_seconds": 28.320231749999948 + }, + { + "index": 33, + "status": 202, + "upload_seconds": 0.537874833999922, + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "terminal": "ready", + "ready_seconds": 29.281564708000133 + }, + { + "index": 30, + "status": 429, + "upload_seconds": 0.6264232500002436 + }, + { + "index": 32, + "status": 429, + "upload_seconds": 0.6135735830002886 + }, + { + "index": 34, + "status": 429, + "upload_seconds": 0.57848120799963 + }, + { + "index": 35, + "status": 429, + "upload_seconds": 0.5214576670000497 + }, + { + "index": 38, + "status": 429, + "upload_seconds": 0.4232839580004111 + }, + { + "index": 36, + "status": 429, + "upload_seconds": 0.5944752499999595 + }, + { + "index": 40, + "status": 202, + "upload_seconds": 0.43313837499999863, + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "terminal": "ready", + "ready_seconds": 30.481176667 + }, + { + "index": 37, + "status": 429, + "upload_seconds": 0.5279237079998893 + }, + { + "index": 41, + "status": 429, + "upload_seconds": 0.4874647090000508 + }, + { + "index": 46, + "status": 429, + "upload_seconds": 0.2976544580001246 + }, + { + "index": 39, + "status": 429, + "upload_seconds": 0.5910501660000591 + }, + { + "index": 47, + "status": 429, + "upload_seconds": 0.3643132920001335 + }, + { + "index": 42, + "status": 429, + "upload_seconds": 0.5871979580001607 + }, + { + "index": 48, + "status": 429, + "upload_seconds": 0.38167995800040444 + }, + { + "index": 43, + "status": 429, + "upload_seconds": 0.5213576670003022 + }, + { + "index": 44, + "status": 429, + "upload_seconds": 0.5978834580000694 + }, + { + "index": 49, + "status": 429, + "upload_seconds": 0.4228952080002273 + }, + { + "index": 51, + "status": 429, + "upload_seconds": 0.48186116700026105 + }, + { + "index": 45, + "status": 429, + "upload_seconds": 0.6031653330001063 + }, + { + "index": 52, + "status": 429, + "upload_seconds": 0.4970078330002252 + }, + { + "index": 50, + "status": 429, + "upload_seconds": 0.5275511250001728 + }, + { + "index": 53, + "status": 429, + "upload_seconds": 0.5119440830003441 + }, + { + "index": 57, + "status": 429, + "upload_seconds": 0.40361562500038417 + }, + { + "index": 54, + "status": 429, + "upload_seconds": 0.573893458999919 + }, + { + "index": 60, + "status": 429, + "upload_seconds": 0.40328020800006925 + }, + { + "index": 62, + "status": 429, + "upload_seconds": 0.4000605840001299 + }, + { + "index": 55, + "status": 429, + "upload_seconds": 0.6692029999999249 + }, + { + "index": 56, + "status": 429, + "upload_seconds": 0.6225064579998616 + }, + { + "index": 63, + "status": 429, + "upload_seconds": 0.40598258299996814 + }, + { + "index": 58, + "status": 429, + "upload_seconds": 0.6847872919997826 + }, + { + "index": 64, + "status": 429, + "upload_seconds": 0.47814983299986125 + }, + { + "index": 59, + "status": 429, + "upload_seconds": 0.6276995829998668 + }, + { + "index": 61, + "status": 429, + "upload_seconds": 0.6777354999999261 + }, + { + "index": 66, + "status": 429, + "upload_seconds": 0.4940775419995589 + }, + { + "index": 68, + "status": 429, + "upload_seconds": 0.4973452500003077 + }, + { + "index": 65, + "status": 202, + "upload_seconds": 0.5221591250001438, + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "terminal": "ready", + "ready_seconds": 30.397425250000197 + }, + { + "index": 70, + "status": 429, + "upload_seconds": 0.4808669999997619 + }, + { + "index": 67, + "status": 429, + "upload_seconds": 0.575548875000095 + }, + { + "index": 72, + "status": 429, + "upload_seconds": 0.4118760840001414 + }, + { + "index": 69, + "status": 429, + "upload_seconds": 0.577730832999805 + }, + { + "index": 73, + "status": 429, + "upload_seconds": 0.4716782079999575 + }, + { + "index": 71, + "status": 429, + "upload_seconds": 0.5931040000000394 + }, + { + "index": 76, + "status": 429, + "upload_seconds": 0.48203154100019674 + }, + { + "index": 74, + "status": 429, + "upload_seconds": 0.5135427089999212 + }, + { + "index": 77, + "status": 429, + "upload_seconds": 0.43694883300031506 + }, + { + "index": 75, + "status": 429, + "upload_seconds": 0.5743062090000421 + }, + { + "index": 78, + "status": 429, + "upload_seconds": 0.5255246250003438 + }, + { + "index": 79, + "status": 429, + "upload_seconds": 0.5039793329997337 + }, + { + "index": 81, + "status": 429, + "upload_seconds": 0.5166306669998448 + }, + { + "index": 80, + "status": 429, + "upload_seconds": 0.5993904590000056 + }, + { + "index": 85, + "status": 429, + "upload_seconds": 0.5032475419998264 + }, + { + "index": 82, + "status": 429, + "upload_seconds": 0.6057980830000815 + }, + { + "index": 87, + "status": 429, + "upload_seconds": 0.5053580839999086 + }, + { + "index": 89, + "status": 429, + "upload_seconds": 0.433452374999888 + }, + { + "index": 83, + "status": 429, + "upload_seconds": 0.699554791999617 + }, + { + "index": 91, + "status": 429, + "upload_seconds": 0.5004729590000352 + }, + { + "index": 84, + "status": 429, + "upload_seconds": 0.7285058750003373 + }, + { + "index": 93, + "status": 429, + "upload_seconds": 0.5093171670000629 + }, + { + "index": 86, + "status": 429, + "upload_seconds": 0.7937231250002696 + }, + { + "index": 95, + "status": 429, + "upload_seconds": 0.49584137500005454 + }, + { + "index": 88, + "status": 202, + "upload_seconds": 0.8076417089996539, + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "terminal": "ready", + "ready_seconds": 30.316999583000324 + }, + { + "index": 96, + "status": 429, + "upload_seconds": 0.492497917000037 + }, + { + "index": 90, + "status": 429, + "upload_seconds": 0.7331488750000972 + }, + { + "index": 99, + "status": 429, + "upload_seconds": 0.45177912500003004 + }, + { + "index": 101, + "status": 429, + "upload_seconds": 0.38794695899969156 + }, + { + "index": 92, + "status": 429, + "upload_seconds": 0.7862456660000134 + }, + { + "index": 102, + "status": 429, + "upload_seconds": 0.4716022919997158 + }, + { + "index": 103, + "status": 429, + "upload_seconds": 0.41004866699995546 + }, + { + "index": 94, + "status": 429, + "upload_seconds": 0.7964041660002295 + }, + { + "index": 97, + "status": 429, + "upload_seconds": 0.7136342909998348 + }, + { + "index": 104, + "status": 429, + "upload_seconds": 0.48929604100021606 + }, + { + "index": 106, + "status": 429, + "upload_seconds": 0.4875204160002795 + }, + { + "index": 98, + "status": 429, + "upload_seconds": 0.7973849999998492 + }, + { + "index": 107, + "status": 429, + "upload_seconds": 0.500636083000245 + }, + { + "index": 100, + "status": 429, + "upload_seconds": 0.7970687500001077 + }, + { + "index": 108, + "status": 429, + "upload_seconds": 0.5011185410003236 + }, + { + "index": 105, + "status": 429, + "upload_seconds": 0.6304047910002737 + }, + { + "index": 109, + "status": 429, + "upload_seconds": 0.6026061249999657 + }, + { + "index": 111, + "status": 429, + "upload_seconds": 0.5336569580003925 + }, + { + "index": 110, + "status": 429, + "upload_seconds": 0.6685325419998662 + }, + { + "index": 112, + "status": 429, + "upload_seconds": 0.596721916000206 + }, + { + "index": 113, + "status": 429, + "upload_seconds": 0.6035968330002106 + }, + { + "index": 116, + "status": 429, + "upload_seconds": 0.570689499999844 + }, + { + "index": 114, + "status": 429, + "upload_seconds": 0.6114308329997584 + }, + { + "index": 118, + "status": 202, + "upload_seconds": 0.5652103329998681, + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "terminal": "ready", + "ready_seconds": 30.4356227090002 + }, + { + "index": 115, + "status": 429, + "upload_seconds": 0.6625353330000507 + }, + { + "index": 120, + "status": 429, + "upload_seconds": 0.5013821669999743 + }, + { + "index": 117, + "status": 429, + "upload_seconds": 0.5982385830002386 + }, + { + "index": 119, + "status": 429, + "upload_seconds": 0.527991750000183 + }, + { + "index": 121, + "status": 429, + "upload_seconds": 0.5643502920001993 + }, + { + "index": 123, + "status": 429, + "upload_seconds": 0.4965356249999786 + }, + { + "index": 122, + "status": 429, + "upload_seconds": 0.5161449170000196 + }, + { + "index": 124, + "status": 429, + "upload_seconds": 0.56276783300018 + }, + { + "index": 125, + "status": 429, + "upload_seconds": 0.4845628329999272 + }, + { + "index": 126, + "status": 429, + "upload_seconds": 0.5025779170000533 + }, + { + "index": 127, + "status": 429, + "upload_seconds": 0.49424216699981116 + }, + { + "index": 132, + "status": 429, + "upload_seconds": 0.32049087500035967 + }, + { + "index": 128, + "status": 429, + "upload_seconds": 0.5886672910000925 + }, + { + "index": 134, + "status": 429, + "upload_seconds": 0.39353779200018835 + }, + { + "index": 135, + "status": 429, + "upload_seconds": 0.41075466599977517 + }, + { + "index": 137, + "status": 429, + "upload_seconds": 0.410953999999947 + }, + { + "index": 129, + "status": 429, + "upload_seconds": 0.6807109169999421 + }, + { + "index": 139, + "status": 429, + "upload_seconds": 0.47732537499996397 + }, + { + "index": 144, + "status": 429, + "upload_seconds": 0.3798191659998338 + }, + { + "index": 130, + "status": 429, + "upload_seconds": 0.8001185419998365 + }, + { + "index": 146, + "status": 429, + "upload_seconds": 0.3966425000003255 + }, + { + "index": 131, + "status": 429, + "upload_seconds": 0.8961517500001719 + }, + { + "index": 147, + "status": 429, + "upload_seconds": 0.4783827910000582 + }, + { + "index": 149, + "status": 429, + "upload_seconds": 0.4900604590002331 + }, + { + "index": 133, + "status": 429, + "upload_seconds": 1.0226498339998216 + }, + { + "index": 136, + "status": 202, + "upload_seconds": 1.0718137499998193, + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "terminal": "ready", + "ready_seconds": 30.18230529099992 + }, + { + "index": 152, + "status": 429, + "upload_seconds": 0.40827208300015627 + }, + { + "index": 138, + "status": 429, + "upload_seconds": 1.0942369589997725 + }, + { + "index": 154, + "status": 429, + "upload_seconds": 0.39713483299965446 + }, + { + "index": 140, + "status": 429, + "upload_seconds": 1.0298455410002134 + }, + { + "index": 155, + "status": 429, + "upload_seconds": 0.3951938329996665 + }, + { + "index": 158, + "status": 429, + "upload_seconds": 0.2184658330002094 + }, + { + "index": 141, + "status": 429, + "upload_seconds": 1.12382370899968 + }, + { + "index": 142, + "status": 429, + "upload_seconds": 1.180578166000032 + }, + { + "index": 143, + "status": 429, + "upload_seconds": 1.1277323329995852 + }, + { + "index": 145, + "status": 429, + "upload_seconds": 1.1127691249998861 + }, + { + "index": 148, + "status": 429, + "upload_seconds": 1.0283722500003023 + }, + { + "index": 150, + "status": 429, + "upload_seconds": 1.0152499999999236 + }, + { + "index": 151, + "status": 429, + "upload_seconds": 0.9384003749996737 + }, + { + "index": 153, + "status": 429, + "upload_seconds": 0.9105748749998384 + }, + { + "index": 156, + "status": 429, + "upload_seconds": 0.7260491249999177 + }, + { + "index": 157, + "status": 429, + "upload_seconds": 0.7935182080000232 + }, + { + "index": 159, + "status": 429, + "upload_seconds": 0.7168919999999162 + } + ], + "samples": [ + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 8, + "rss_bytes": 77721600, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 7, + "messages_unacknowledged": 1 + }, + "seconds": 1.6022961250000662 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "processing", + "queue_wait_seconds": 2.210975 + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 32, + "rss_bytes": 143298560, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 30, + "messages_unacknowledged": 1 + }, + "seconds": 4.902898875000119 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "processing", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 31, + "rss_bytes": 93421568, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 29, + "messages_unacknowledged": 1 + }, + "seconds": 7.304545583000163 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "processing", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + } + ], + "pending": 29, + "rss_bytes": 247091200, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 27, + "messages_unacknowledged": 1 + }, + "seconds": 9.710651249999955 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "processing", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + } + ], + "pending": 26, + "rss_bytes": 119484416, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 23, + "messages_unacknowledged": 1 + }, + "seconds": 12.711902708000252 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "ready", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "ready", + "queue_wait_seconds": 11.13794 + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "ready", + "queue_wait_seconds": 12.015013 + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "processing", + "queue_wait_seconds": 12.912578 + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + } + ], + "pending": 23, + "rss_bytes": 215371776, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 21, + "messages_unacknowledged": 1 + }, + "seconds": 15.116820125000231 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "ready", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "ready", + "queue_wait_seconds": 11.13794 + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "ready", + "queue_wait_seconds": 12.015013 + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "ready", + "queue_wait_seconds": 13.73859 + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "ready", + "queue_wait_seconds": 14.61538 + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "processing", + "queue_wait_seconds": 15.479344 + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "ready", + "queue_wait_seconds": 12.912578 + } + ], + "pending": 20, + "rss_bytes": 112168960, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 17, + "messages_unacknowledged": 1 + }, + "seconds": 18.510802250000324 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "ready", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "ready", + "queue_wait_seconds": 11.13794 + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "ready", + "queue_wait_seconds": 12.015013 + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "ready", + "queue_wait_seconds": 13.73859 + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "ready", + "queue_wait_seconds": 14.61538 + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "ready", + "queue_wait_seconds": 15.479344 + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "ready", + "queue_wait_seconds": 17.415371 + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "processing", + "queue_wait_seconds": 18.325502 + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "ready", + "queue_wait_seconds": 12.912578 + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "ready", + "queue_wait_seconds": 16.404236 + } + ], + "pending": 17, + "rss_bytes": 251621376, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 14, + "messages_unacknowledged": 1 + }, + "seconds": 21.138074917000267 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "ready", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "ready", + "queue_wait_seconds": 11.13794 + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "ready", + "queue_wait_seconds": 12.015013 + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "ready", + "queue_wait_seconds": 13.73859 + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "ready", + "queue_wait_seconds": 14.61538 + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "ready", + "queue_wait_seconds": 15.479344 + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "ready", + "queue_wait_seconds": 17.415371 + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "ready", + "queue_wait_seconds": 18.325502 + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "ready", + "queue_wait_seconds": 19.234062 + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "processing", + "queue_wait_seconds": 21.10443 + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "ready", + "queue_wait_seconds": 20.213834 + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "ready", + "queue_wait_seconds": 12.912578 + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "ready", + "queue_wait_seconds": 16.404236 + } + ], + "pending": 14, + "rss_bytes": 126009344, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 12, + "messages_unacknowledged": 1 + }, + "seconds": 23.515174791999925 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "ready", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "ready", + "queue_wait_seconds": 11.13794 + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "ready", + "queue_wait_seconds": 12.015013 + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "ready", + "queue_wait_seconds": 13.73859 + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "ready", + "queue_wait_seconds": 14.61538 + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "ready", + "queue_wait_seconds": 15.479344 + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "ready", + "queue_wait_seconds": 17.415371 + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "ready", + "queue_wait_seconds": 22.908989 + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "ready", + "queue_wait_seconds": 18.325502 + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "ready", + "queue_wait_seconds": 19.234062 + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "ready", + "queue_wait_seconds": 21.10443 + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "ready", + "queue_wait_seconds": 20.213834 + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "ready", + "queue_wait_seconds": 22.016404 + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "processing", + "queue_wait_seconds": 23.754009 + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "ready", + "queue_wait_seconds": 12.912578 + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "ready", + "queue_wait_seconds": 16.404236 + } + ], + "pending": 11, + "rss_bytes": 93618176, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 9, + "messages_unacknowledged": 1 + }, + "seconds": 25.912013000000115 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "ready", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "ready", + "queue_wait_seconds": 11.13794 + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "ready", + "queue_wait_seconds": 12.015013 + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "ready", + "queue_wait_seconds": 13.73859 + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "ready", + "queue_wait_seconds": 14.61538 + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "ready", + "queue_wait_seconds": 15.479344 + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "ready", + "queue_wait_seconds": 17.415371 + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "ready", + "queue_wait_seconds": 22.908989 + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "ready", + "queue_wait_seconds": 24.698151 + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "ready", + "queue_wait_seconds": 18.325502 + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "processing", + "queue_wait_seconds": 25.529356 + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "ready", + "queue_wait_seconds": 19.234062 + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "ready", + "queue_wait_seconds": 21.10443 + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "ready", + "queue_wait_seconds": 20.213834 + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "ready", + "queue_wait_seconds": 22.016404 + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "ready", + "queue_wait_seconds": 23.754009 + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "ready", + "queue_wait_seconds": 12.912578 + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "ready", + "queue_wait_seconds": 16.404236 + } + ], + "pending": 9, + "rss_bytes": 141860864, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 7, + "messages_unacknowledged": 1 + }, + "seconds": 28.308871125000223 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "ready", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "ready", + "queue_wait_seconds": 11.13794 + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "ready", + "queue_wait_seconds": 12.015013 + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "ready", + "queue_wait_seconds": 13.73859 + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "ready", + "queue_wait_seconds": 14.61538 + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "ready", + "queue_wait_seconds": 15.479344 + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "ready", + "queue_wait_seconds": 17.415371 + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "ready", + "queue_wait_seconds": 22.908989 + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "ready", + "queue_wait_seconds": 24.698151 + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "ready", + "queue_wait_seconds": 26.514711 + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "processing", + "queue_wait_seconds": 28.320098 + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "ready", + "queue_wait_seconds": 18.325502 + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "ready", + "queue_wait_seconds": 25.529356 + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "ready", + "queue_wait_seconds": 27.488247 + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "ready", + "queue_wait_seconds": 19.234062 + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "ready", + "queue_wait_seconds": 21.10443 + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "ready", + "queue_wait_seconds": 20.213834 + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "ready", + "queue_wait_seconds": 22.016404 + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "ready", + "queue_wait_seconds": 23.754009 + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "ready", + "queue_wait_seconds": 12.912578 + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "ready", + "queue_wait_seconds": 16.404236 + } + ], + "pending": 6, + "rss_bytes": 93745152, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 3, + "messages_unacknowledged": 1 + }, + "seconds": 31.99964666699998 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "ready", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "ready", + "queue_wait_seconds": 11.13794 + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "ready", + "queue_wait_seconds": 12.015013 + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "ready", + "queue_wait_seconds": 13.73859 + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "ready", + "queue_wait_seconds": 14.61538 + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "ready", + "queue_wait_seconds": 15.479344 + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "ready", + "queue_wait_seconds": 17.415371 + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "ready", + "queue_wait_seconds": 22.908989 + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "ready", + "queue_wait_seconds": 24.698151 + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "ready", + "queue_wait_seconds": 26.514711 + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "ready", + "queue_wait_seconds": 28.320098 + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "ready", + "queue_wait_seconds": 18.325502 + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "ready", + "queue_wait_seconds": 25.529356 + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "ready", + "queue_wait_seconds": 27.488247 + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "ready", + "queue_wait_seconds": 19.234062 + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "ready", + "queue_wait_seconds": 21.10443 + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "ready", + "queue_wait_seconds": 20.213834 + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "ready", + "queue_wait_seconds": 22.016404 + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "ready", + "queue_wait_seconds": 23.754009 + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "ready", + "queue_wait_seconds": 29.414892 + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "ready", + "queue_wait_seconds": 29.505011 + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "processing", + "queue_wait_seconds": 29.434334 + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "ready", + "queue_wait_seconds": 12.912578 + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "ready", + "queue_wait_seconds": 16.404236 + } + ], + "pending": 3, + "rss_bytes": 149323776, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 1, + "messages_unacknowledged": 1 + }, + "seconds": 34.41366366700004 + }, + { + "jobs": [ + { + "id": "023aef23-0d1b-470f-82c4-e532f4a8a84e", + "status": "ready", + "queue_wait_seconds": 6.738933 + }, + { + "id": "772903cb-2e8e-4859-ad17-5b7098b77448", + "status": "ready", + "queue_wait_seconds": 7.621757 + }, + { + "id": "b3da1679-c85a-47e9-afc1-ef9250a29139", + "status": "ready", + "queue_wait_seconds": 8.438357 + }, + { + "id": "519b820e-0b7a-4eb5-b154-fce92caaf302", + "status": "ready", + "queue_wait_seconds": 1.371153 + }, + { + "id": "3583b51b-3f45-4224-9704-cbceab1da8b1", + "status": "ready", + "queue_wait_seconds": 2.210975 + }, + { + "id": "637e41cf-6175-46e1-b27d-21fb02dcf655", + "status": "ready", + "queue_wait_seconds": 4.004568 + }, + { + "id": "645036f9-4ac0-42af-a529-69ee2166560e", + "status": "ready", + "queue_wait_seconds": 5.018808 + }, + { + "id": "a8d50ceb-d5c0-4211-a635-d5e230d19338", + "status": "ready", + "queue_wait_seconds": 10.202127 + }, + { + "id": "6a9d9300-dd02-4360-937b-62852535338c", + "status": "ready", + "queue_wait_seconds": 11.13794 + }, + { + "id": "2146b856-9e0e-430b-ba58-bfbc7c688a4c", + "status": "ready", + "queue_wait_seconds": 12.015013 + }, + { + "id": "2030200c-9d41-46cf-bc94-28f32cb19a0e", + "status": "ready", + "queue_wait_seconds": 13.73859 + }, + { + "id": "b73072a4-1bb6-448b-83ad-aa33d023aa46", + "status": "ready", + "queue_wait_seconds": 14.61538 + }, + { + "id": "425915f4-39d5-49bd-92f8-b8187be4656d", + "status": "ready", + "queue_wait_seconds": 15.479344 + }, + { + "id": "e1018ad8-5133-4bd3-a0ba-f567272d2057", + "status": "ready", + "queue_wait_seconds": 17.415371 + }, + { + "id": "80fd2278-465b-4943-b253-9e0cb8031f3d", + "status": "ready", + "queue_wait_seconds": 22.908989 + }, + { + "id": "60b82ba1-da75-4a17-a798-2b9b2faad839", + "status": "ready", + "queue_wait_seconds": 24.698151 + }, + { + "id": "c0fc299e-713b-4ee3-afb9-8ad74a74ebe7", + "status": "ready", + "queue_wait_seconds": 26.514711 + }, + { + "id": "37ea5de6-03a5-4469-9d16-34b7d5552817", + "status": "ready", + "queue_wait_seconds": 28.320098 + }, + { + "id": "04207e69-27a8-46bc-8407-d1624c04d033", + "status": "ready", + "queue_wait_seconds": 18.325502 + }, + { + "id": "04d6876a-21fd-462b-81cb-6054e88df0ec", + "status": "ready", + "queue_wait_seconds": 25.529356 + }, + { + "id": "312e9d57-5801-426c-9ed5-8e18cb2c6037", + "status": "ready", + "queue_wait_seconds": 27.488247 + }, + { + "id": "20c606b0-0761-4fbd-ab55-382c832f6bbc", + "status": "ready", + "queue_wait_seconds": 19.234062 + }, + { + "id": "bfc93eb8-8b5e-4871-9c4f-b33aa708a429", + "status": "ready", + "queue_wait_seconds": 21.10443 + }, + { + "id": "aaba2508-bb88-4940-9eb5-73364b94b20f", + "status": "ready", + "queue_wait_seconds": 20.213834 + }, + { + "id": "bd8e07fd-bf68-4590-9298-53d47ea11b3d", + "status": "ready", + "queue_wait_seconds": 22.016404 + }, + { + "id": "84ef2ec8-c8ae-423b-906f-42be35996a73", + "status": "ready", + "queue_wait_seconds": 23.754009 + }, + { + "id": "5820a0d8-407b-4a74-a5e2-0c290af90e72", + "status": "ready", + "queue_wait_seconds": 0.392166 + }, + { + "id": "fa4c6b97-b914-4750-b7f0-c4117d300861", + "status": "ready", + "queue_wait_seconds": 29.414892 + }, + { + "id": "26c85b9b-6123-47d7-b256-b7fe0b128a28", + "status": "ready", + "queue_wait_seconds": 29.505011 + }, + { + "id": "c6f8159c-0f1c-4164-8728-9267ff93b53f", + "status": "ready", + "queue_wait_seconds": 29.434334 + }, + { + "id": "81c058d2-aa50-47a9-bd81-d4ef8db25efb", + "status": "ready", + "queue_wait_seconds": 3.131054 + }, + { + "id": "b34fef33-2e92-4242-9271-c904d3320801", + "status": "ready", + "queue_wait_seconds": 29.518091 + }, + { + "id": "aa78c25b-9eef-47d7-8a2e-695cef21a77b", + "status": "ready", + "queue_wait_seconds": 29.259553 + }, + { + "id": "b666d9e9-7c1e-41c8-bdba-ea0b559a1628", + "status": "ready", + "queue_wait_seconds": 5.856984 + }, + { + "id": "110fba68-a7a6-4f8f-8047-a812d4c70c3c", + "status": "ready", + "queue_wait_seconds": 9.340853 + }, + { + "id": "420141d7-5051-4c96-aacb-8d3cd080ecc7", + "status": "ready", + "queue_wait_seconds": 12.912578 + }, + { + "id": "5e2d380a-c7ec-46d2-bfea-c69a4a40bc3a", + "status": "ready", + "queue_wait_seconds": 16.404236 + } + ], + "pending": 0, + "rss_bytes": 77819904, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 0, + "messages_unacknowledged": 0 + }, + "seconds": 36.704709083000125 + } + ], + "metric_errors": [], + "limitations": "Closed-loop; ready time is polling upper bound. RSS is sum of worker/container processes, shared pages may be counted twice. Sampling may miss peaks. Queue wait is first claim minus durable job creation." +} diff --git a/evidence/real-c1-p1-s512.json b/evidence/real-c1-p1-s512.json new file mode 100644 index 00000000..41a6d884 --- /dev/null +++ b/evidence/real-c1-p1-s512.json @@ -0,0 +1,2618 @@ +{ + "run_name": "real-c1-p1-s512", + "settings": { + "workers": 1, + "prefetch": 1, + "max_pending": 32, + "worker_limits": { + "Memory": 4294967296, + "NanoCpus": 2000000000 + } + }, + "workload": { + "images": 160, + "concurrency": 16, + "size": 512, + "bytes": 1890, + "type": "closed-loop burst" + }, + "results": { + "accepted": 33, + "rejected_429": 127, + "ready": 33, + "failed": 0, + "timed_out": 0, + "elapsed_seconds": 25.88860091699962, + "ready_per_second": 1.2746922904717772, + "ready_latency_ms": { + "0.5": 13256.203, + "0.95": 24480.634, + "0.99": 24722.491 + }, + "queue_wait_ms": { + "0.5": 12498.609, + "0.95": 23708.966, + "0.99": 24014.379 + }, + "max_sampled_rss_bytes": 140509184, + "max_sampled_pending": 28, + "sample_count": 10, + "drained": true, + "recovery_upload_ready": true + }, + "records": [ + { + "index": 5, + "status": 202, + "upload_seconds": 0.1515077920003023, + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "terminal": "ready", + "ready_seconds": 1.2060929169997507 + }, + { + "index": 0, + "status": 202, + "upload_seconds": 0.17243608300032065, + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "terminal": "ready", + "ready_seconds": 1.7519195830000172 + }, + { + "index": 10, + "status": 202, + "upload_seconds": 0.20669429200006562, + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "terminal": "ready", + "ready_seconds": 2.633656707999762 + }, + { + "index": 7, + "status": 202, + "upload_seconds": 0.24114520800003447, + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "terminal": "ready", + "ready_seconds": 3.394811749999917 + }, + { + "index": 1, + "status": 202, + "upload_seconds": 0.258484832999784, + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "terminal": "ready", + "ready_seconds": 4.059611209000195 + }, + { + "index": 15, + "status": 202, + "upload_seconds": 0.25915191599960963, + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "terminal": "ready", + "ready_seconds": 4.790080666999984 + }, + { + "index": 4, + "status": 202, + "upload_seconds": 0.27212433300019256, + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "terminal": "ready", + "ready_seconds": 5.644111291999707 + }, + { + "index": 11, + "status": 202, + "upload_seconds": 0.2773440410001058, + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "terminal": "ready", + "ready_seconds": 6.311223209000218 + }, + { + "index": 3, + "status": 202, + "upload_seconds": 0.28588220799974806, + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "terminal": "ready", + "ready_seconds": 7.019885666999926 + }, + { + "index": 6, + "status": 202, + "upload_seconds": 0.34075545799987594, + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "terminal": "ready", + "ready_seconds": 7.819338667000011 + }, + { + "index": 2, + "status": 202, + "upload_seconds": 0.35241945799998575, + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "terminal": "ready", + "ready_seconds": 8.612079374999666 + }, + { + "index": 8, + "status": 202, + "upload_seconds": 0.36015725000015664, + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "terminal": "ready", + "ready_seconds": 9.298493792000045 + }, + { + "index": 13, + "status": 202, + "upload_seconds": 0.36359600000014325, + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "terminal": "ready", + "ready_seconds": 10.115750082999966 + }, + { + "index": 17, + "status": 202, + "upload_seconds": 0.20047779200012883, + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "terminal": "ready", + "ready_seconds": 11.025289082999734 + }, + { + "index": 14, + "status": 202, + "upload_seconds": 0.37420062499995765, + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "terminal": "ready", + "ready_seconds": 11.807512333999966 + }, + { + "index": 18, + "status": 202, + "upload_seconds": 0.1760354160001043, + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "terminal": "ready", + "ready_seconds": 12.561547792000056 + }, + { + "index": 9, + "status": 202, + "upload_seconds": 0.4432161660001839, + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "terminal": "ready", + "ready_seconds": 13.25620329200001 + }, + { + "index": 19, + "status": 202, + "upload_seconds": 0.20694687499963038, + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "terminal": "ready", + "ready_seconds": 13.940520458000265 + }, + { + "index": 16, + "status": 202, + "upload_seconds": 0.30164095800000723, + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "terminal": "ready", + "ready_seconds": 14.665956250000363 + }, + { + "index": 21, + "status": 202, + "upload_seconds": 0.19466324999984863, + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "terminal": "ready", + "ready_seconds": 15.515610583000125 + }, + { + "index": 12, + "status": 202, + "upload_seconds": 0.4630282910002279, + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "terminal": "ready", + "ready_seconds": 16.350001583999983 + }, + { + "index": 23, + "status": 202, + "upload_seconds": 0.19391383299989684, + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "terminal": "ready", + "ready_seconds": 17.007574459000352 + }, + { + "index": 20, + "status": 202, + "upload_seconds": 0.223310875000152, + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "terminal": "ready", + "ready_seconds": 17.824724250000145 + }, + { + "index": 25, + "status": 202, + "upload_seconds": 0.19780524999987392, + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "terminal": "ready", + "ready_seconds": 18.57707179199997 + }, + { + "index": 22, + "status": 202, + "upload_seconds": 0.27231787499977145, + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "terminal": "ready", + "ready_seconds": 19.243424334000338 + }, + { + "index": 28, + "status": 202, + "upload_seconds": 0.20971320899980128, + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "terminal": "ready", + "ready_seconds": 19.96837675000006 + }, + { + "index": 24, + "status": 202, + "upload_seconds": 0.3004352919997473, + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "terminal": "ready", + "ready_seconds": 20.711411290999877 + }, + { + "index": 30, + "status": 202, + "upload_seconds": 0.2757489999999052, + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "terminal": "ready", + "ready_seconds": 21.368605791999926 + }, + { + "index": 26, + "status": 202, + "upload_seconds": 0.31110070800014, + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "terminal": "ready", + "ready_seconds": 22.201705249999577 + }, + { + "index": 32, + "status": 202, + "upload_seconds": 0.2237908330002938, + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "terminal": "ready", + "ready_seconds": 22.891766541999914 + }, + { + "index": 27, + "status": 202, + "upload_seconds": 0.31318975000021965, + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "terminal": "ready", + "ready_seconds": 23.709088333999716 + }, + { + "index": 34, + "status": 202, + "upload_seconds": 0.22858258399992337, + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "terminal": "ready", + "ready_seconds": 24.480633790999946 + }, + { + "index": 29, + "status": 429, + "upload_seconds": 0.3107968340000298 + }, + { + "index": 31, + "status": 429, + "upload_seconds": 0.30156650000026275 + }, + { + "index": 36, + "status": 429, + "upload_seconds": 0.22144750000006752 + }, + { + "index": 38, + "status": 429, + "upload_seconds": 0.21124912499999482 + }, + { + "index": 33, + "status": 429, + "upload_seconds": 0.2434027080003034 + }, + { + "index": 39, + "status": 429, + "upload_seconds": 0.15663841700006742 + }, + { + "index": 35, + "status": 429, + "upload_seconds": 0.23858995799992044 + }, + { + "index": 41, + "status": 429, + "upload_seconds": 0.12418854099996679 + }, + { + "index": 37, + "status": 429, + "upload_seconds": 0.22730249999995067 + }, + { + "index": 40, + "status": 429, + "upload_seconds": 0.15962500000023283 + }, + { + "index": 47, + "status": 429, + "upload_seconds": 0.023495916000229045 + }, + { + "index": 42, + "status": 429, + "upload_seconds": 0.12416483299966785 + }, + { + "index": 49, + "status": 429, + "upload_seconds": 0.022527582999828155 + }, + { + "index": 43, + "status": 429, + "upload_seconds": 0.059076749999803724 + }, + { + "index": 52, + "status": 429, + "upload_seconds": 0.05421075000003839 + }, + { + "index": 53, + "status": 429, + "upload_seconds": 0.05277812499980428 + }, + { + "index": 44, + "status": 429, + "upload_seconds": 0.09077612500004761 + }, + { + "index": 56, + "status": 429, + "upload_seconds": 0.05337350000036167 + }, + { + "index": 45, + "status": 429, + "upload_seconds": 0.09234162500024468 + }, + { + "index": 57, + "status": 429, + "upload_seconds": 0.05563941599984901 + }, + { + "index": 46, + "status": 429, + "upload_seconds": 0.09104404100025931 + }, + { + "index": 60, + "status": 429, + "upload_seconds": 0.05430454199995438 + }, + { + "index": 48, + "status": 429, + "upload_seconds": 0.08733362499970099 + }, + { + "index": 65, + "status": 429, + "upload_seconds": 0.016065167000306246 + }, + { + "index": 50, + "status": 429, + "upload_seconds": 0.0877424999998766 + }, + { + "index": 64, + "status": 429, + "upload_seconds": 0.022171500000240485 + }, + { + "index": 51, + "status": 429, + "upload_seconds": 0.09011870799986355 + }, + { + "index": 69, + "status": 429, + "upload_seconds": 0.01701458300021841 + }, + { + "index": 73, + "status": 429, + "upload_seconds": 0.01128354100001161 + }, + { + "index": 54, + "status": 429, + "upload_seconds": 0.09170166700005211 + }, + { + "index": 55, + "status": 429, + "upload_seconds": 0.09230654199973287 + }, + { + "index": 74, + "status": 429, + "upload_seconds": 0.062350791999961075 + }, + { + "index": 58, + "status": 429, + "upload_seconds": 0.14047329200002423 + }, + { + "index": 59, + "status": 429, + "upload_seconds": 0.1392597079998268 + }, + { + "index": 76, + "status": 429, + "upload_seconds": 0.06148600000005899 + }, + { + "index": 61, + "status": 429, + "upload_seconds": 0.14121045799993226 + }, + { + "index": 78, + "status": 429, + "upload_seconds": 0.061845291999816254 + }, + { + "index": 63, + "status": 429, + "upload_seconds": 0.10801129099991158 + }, + { + "index": 79, + "status": 429, + "upload_seconds": 0.014398000000255706 + }, + { + "index": 62, + "status": 429, + "upload_seconds": 0.11688145800007987 + }, + { + "index": 82, + "status": 429, + "upload_seconds": 0.01313283399986176 + }, + { + "index": 67, + "status": 429, + "upload_seconds": 0.10564666699974623 + }, + { + "index": 84, + "status": 429, + "upload_seconds": 0.012718041999960406 + }, + { + "index": 66, + "status": 429, + "upload_seconds": 0.11113966699986122 + }, + { + "index": 86, + "status": 429, + "upload_seconds": 0.01238062499987791 + }, + { + "index": 68, + "status": 429, + "upload_seconds": 0.11080579100007526 + }, + { + "index": 88, + "status": 429, + "upload_seconds": 0.014717000000018743 + }, + { + "index": 70, + "status": 429, + "upload_seconds": 0.11145049999959156 + }, + { + "index": 90, + "status": 429, + "upload_seconds": 0.01667700000007244 + }, + { + "index": 71, + "status": 429, + "upload_seconds": 0.11536829200031207 + }, + { + "index": 92, + "status": 429, + "upload_seconds": 0.01737291599965829 + }, + { + "index": 72, + "status": 429, + "upload_seconds": 0.11601158399980704 + }, + { + "index": 75, + "status": 429, + "upload_seconds": 0.11367070799997236 + }, + { + "index": 77, + "status": 429, + "upload_seconds": 0.11212150000028487 + }, + { + "index": 95, + "status": 429, + "upload_seconds": 0.062290000000302825 + }, + { + "index": 96, + "status": 429, + "upload_seconds": 0.060525709000103234 + }, + { + "index": 99, + "status": 429, + "upload_seconds": 0.0574664999999186 + }, + { + "index": 80, + "status": 429, + "upload_seconds": 0.1077512920001027 + }, + { + "index": 100, + "status": 429, + "upload_seconds": 0.05760254200004056 + }, + { + "index": 81, + "status": 429, + "upload_seconds": 0.11005895800008147 + }, + { + "index": 101, + "status": 429, + "upload_seconds": 0.05901241599985951 + }, + { + "index": 103, + "status": 429, + "upload_seconds": 0.0169482500000413 + }, + { + "index": 83, + "status": 429, + "upload_seconds": 0.11185533299976669 + }, + { + "index": 85, + "status": 429, + "upload_seconds": 0.11177841700009594 + }, + { + "index": 106, + "status": 429, + "upload_seconds": 0.015621624999766937 + }, + { + "index": 107, + "status": 429, + "upload_seconds": 0.015429000000040105 + }, + { + "index": 87, + "status": 429, + "upload_seconds": 0.11452150000013717 + }, + { + "index": 109, + "status": 429, + "upload_seconds": 0.015608000000156608 + }, + { + "index": 89, + "status": 429, + "upload_seconds": 0.11843962500006455 + }, + { + "index": 91, + "status": 429, + "upload_seconds": 0.11700137500019991 + }, + { + "index": 111, + "status": 429, + "upload_seconds": 0.017707999999856838 + }, + { + "index": 93, + "status": 429, + "upload_seconds": 0.11532237499977782 + }, + { + "index": 94, + "status": 429, + "upload_seconds": 0.11550666700031798 + }, + { + "index": 114, + "status": 429, + "upload_seconds": 0.06642287500017119 + }, + { + "index": 115, + "status": 429, + "upload_seconds": 0.06651504199999181 + }, + { + "index": 120, + "status": 429, + "upload_seconds": 0.05537083300032464 + }, + { + "index": 97, + "status": 429, + "upload_seconds": 0.16557062499987296 + }, + { + "index": 118, + "status": 429, + "upload_seconds": 0.06625354100015102 + }, + { + "index": 98, + "status": 429, + "upload_seconds": 0.16641000000026906 + }, + { + "index": 121, + "status": 429, + "upload_seconds": 0.013595667000117828 + }, + { + "index": 102, + "status": 429, + "upload_seconds": 0.11890187500011962 + }, + { + "index": 104, + "status": 429, + "upload_seconds": 0.1304275420002341 + }, + { + "index": 123, + "status": 202, + "upload_seconds": 0.030111209000097006, + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "terminal": "ready", + "ready_seconds": 24.722491375000118 + }, + { + "index": 105, + "status": 429, + "upload_seconds": 0.13306945799968162 + }, + { + "index": 124, + "status": 429, + "upload_seconds": 0.03546820800011119 + }, + { + "index": 108, + "status": 429, + "upload_seconds": 0.12907741699973485 + }, + { + "index": 110, + "status": 429, + "upload_seconds": 0.12747804199989332 + }, + { + "index": 127, + "status": 429, + "upload_seconds": 0.034468040999854566 + }, + { + "index": 112, + "status": 429, + "upload_seconds": 0.12559795799961648 + }, + { + "index": 113, + "status": 429, + "upload_seconds": 0.12379454099982468 + }, + { + "index": 116, + "status": 429, + "upload_seconds": 0.11790420799979984 + }, + { + "index": 117, + "status": 429, + "upload_seconds": 0.11693912500004444 + }, + { + "index": 119, + "status": 429, + "upload_seconds": 0.11561145899986514 + }, + { + "index": 128, + "status": 429, + "upload_seconds": 0.08585570800005371 + }, + { + "index": 122, + "status": 429, + "upload_seconds": 0.10295945799998663 + }, + { + "index": 129, + "status": 429, + "upload_seconds": 0.07168899999987843 + }, + { + "index": 125, + "status": 429, + "upload_seconds": 0.10284645799993086 + }, + { + "index": 132, + "status": 429, + "upload_seconds": 0.07029099999999744 + }, + { + "index": 126, + "status": 429, + "upload_seconds": 0.10420654200015633 + }, + { + "index": 133, + "status": 429, + "upload_seconds": 0.07461874999989959 + }, + { + "index": 130, + "status": 429, + "upload_seconds": 0.08516820799968627 + }, + { + "index": 138, + "status": 429, + "upload_seconds": 0.06728424999982963 + }, + { + "index": 131, + "status": 429, + "upload_seconds": 0.08576104200028567 + }, + { + "index": 140, + "status": 429, + "upload_seconds": 0.06875520800031154 + }, + { + "index": 135, + "status": 429, + "upload_seconds": 0.08457187500016516 + }, + { + "index": 134, + "status": 429, + "upload_seconds": 0.0880942909998339 + }, + { + "index": 137, + "status": 429, + "upload_seconds": 0.08638970799984236 + }, + { + "index": 136, + "status": 429, + "upload_seconds": 0.09174199999961274 + }, + { + "index": 139, + "status": 429, + "upload_seconds": 0.08418666599982316 + }, + { + "index": 141, + "status": 429, + "upload_seconds": 0.04333891699980086 + }, + { + "index": 142, + "status": 429, + "upload_seconds": 0.041392415999780496 + }, + { + "index": 145, + "status": 429, + "upload_seconds": 0.03814300000021831 + }, + { + "index": 146, + "status": 429, + "upload_seconds": 0.03727591699998811 + }, + { + "index": 143, + "status": 429, + "upload_seconds": 0.08993129199961913 + }, + { + "index": 144, + "status": 429, + "upload_seconds": 0.08829049999985727 + }, + { + "index": 147, + "status": 429, + "upload_seconds": 0.08331075000023702 + }, + { + "index": 148, + "status": 429, + "upload_seconds": 0.08366537499978222 + }, + { + "index": 150, + "status": 429, + "upload_seconds": 0.07892354100022203 + }, + { + "index": 151, + "status": 429, + "upload_seconds": 0.0778832500000135 + }, + { + "index": 149, + "status": 429, + "upload_seconds": 0.08456445799993162 + }, + { + "index": 152, + "status": 429, + "upload_seconds": 0.07809341599977415 + }, + { + "index": 153, + "status": 429, + "upload_seconds": 0.07691716599993015 + }, + { + "index": 155, + "status": 429, + "upload_seconds": 0.0715253749999647 + }, + { + "index": 154, + "status": 429, + "upload_seconds": 0.07583329199997024 + }, + { + "index": 158, + "status": 429, + "upload_seconds": 0.06990616700022656 + }, + { + "index": 156, + "status": 429, + "upload_seconds": 0.076692125000136 + }, + { + "index": 157, + "status": 429, + "upload_seconds": 0.07521083299980091 + }, + { + "index": 159, + "status": 429, + "upload_seconds": 0.07508783299999777 + } + ], + "samples": [ + { + "jobs": [ + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 9, + "rss_bytes": 65892352, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 24, + "messages_unacknowledged": 1 + }, + "seconds": 3.312588207999852 + }, + { + "jobs": [ + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "processing", + "queue_wait_seconds": 3.970311 + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "ready", + "queue_wait_seconds": 0.209376 + }, + { + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "ready", + "queue_wait_seconds": 0.951509 + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "ready", + "queue_wait_seconds": 1.731303 + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "ready", + "queue_wait_seconds": 2.495207 + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "ready", + "queue_wait_seconds": 3.285734 + } + ], + "pending": 28, + "rss_bytes": 117329920, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 26, + "messages_unacknowledged": 1 + }, + "seconds": 5.694464249999783 + }, + { + "jobs": [ + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "ready", + "queue_wait_seconds": 3.970311 + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "ready", + "queue_wait_seconds": 4.741965 + }, + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "processing", + "queue_wait_seconds": 6.303542 + }, + { + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "ready", + "queue_wait_seconds": 5.523898 + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "ready", + "queue_wait_seconds": 0.209376 + }, + { + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "ready", + "queue_wait_seconds": 0.951509 + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "ready", + "queue_wait_seconds": 1.731303 + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "ready", + "queue_wait_seconds": 2.495207 + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "ready", + "queue_wait_seconds": 3.285734 + } + ], + "pending": 25, + "rss_bytes": 121466880, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 23, + "messages_unacknowledged": 1 + }, + "seconds": 8.106282082999769 + }, + { + "jobs": [ + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "ready", + "queue_wait_seconds": 6.303542 + }, + { + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "status": "processing", + "queue_wait_seconds": 8.555495 + }, + { + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "ready", + "queue_wait_seconds": 1.731303 + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "ready", + "queue_wait_seconds": 2.495207 + }, + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "ready", + "queue_wait_seconds": 3.970311 + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "ready", + "queue_wait_seconds": 4.741965 + }, + { + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "status": "ready", + "queue_wait_seconds": 7.019249 + }, + { + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "status": "ready", + "queue_wait_seconds": 7.813889 + }, + { + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "ready", + "queue_wait_seconds": 0.209376 + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "ready", + "queue_wait_seconds": 3.285734 + }, + { + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "ready", + "queue_wait_seconds": 0.951509 + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "ready", + "queue_wait_seconds": 5.523898 + } + ], + "pending": 22, + "rss_bytes": 122712064, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 20, + "messages_unacknowledged": 1 + }, + "seconds": 10.50490820799996 + }, + { + "jobs": [ + { + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "status": "ready", + "queue_wait_seconds": 8.555495 + }, + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "ready", + "queue_wait_seconds": 6.303542 + }, + { + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "status": "ready", + "queue_wait_seconds": 9.327098 + }, + { + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "ready", + "queue_wait_seconds": 1.731303 + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "ready", + "queue_wait_seconds": 2.495207 + }, + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "ready", + "queue_wait_seconds": 3.970311 + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "ready", + "queue_wait_seconds": 4.741965 + }, + { + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "status": "ready", + "queue_wait_seconds": 7.019249 + }, + { + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "status": "ready", + "queue_wait_seconds": 7.813889 + }, + { + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "status": "ready", + "queue_wait_seconds": 10.129006 + }, + { + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "status": "processing", + "queue_wait_seconds": 10.908842 + }, + { + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "ready", + "queue_wait_seconds": 0.209376 + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "ready", + "queue_wait_seconds": 3.285734 + }, + { + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "ready", + "queue_wait_seconds": 0.951509 + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "ready", + "queue_wait_seconds": 5.523898 + } + ], + "pending": 19, + "rss_bytes": 124387328, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 17, + "messages_unacknowledged": 1 + }, + "seconds": 12.819240999999693 + }, + { + "jobs": [ + { + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "status": "ready", + "queue_wait_seconds": 8.555495 + }, + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "ready", + "queue_wait_seconds": 6.303542 + }, + { + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "status": "ready", + "queue_wait_seconds": 9.327098 + }, + { + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "status": "ready", + "queue_wait_seconds": 12.498609 + }, + { + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "ready", + "queue_wait_seconds": 1.731303 + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "ready", + "queue_wait_seconds": 2.495207 + }, + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "ready", + "queue_wait_seconds": 3.970311 + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "ready", + "queue_wait_seconds": 4.741965 + }, + { + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "status": "ready", + "queue_wait_seconds": 7.019249 + }, + { + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "status": "ready", + "queue_wait_seconds": 7.813889 + }, + { + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "status": "ready", + "queue_wait_seconds": 10.129006 + }, + { + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "status": "ready", + "queue_wait_seconds": 10.908842 + }, + { + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "status": "ready", + "queue_wait_seconds": 11.726486 + }, + { + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "status": "processing", + "queue_wait_seconds": 13.219188 + }, + { + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "ready", + "queue_wait_seconds": 0.209376 + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "ready", + "queue_wait_seconds": 3.285734 + }, + { + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "ready", + "queue_wait_seconds": 0.951509 + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "ready", + "queue_wait_seconds": 5.523898 + } + ], + "pending": 16, + "rss_bytes": 124952576, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 12, + "messages_unacknowledged": 1 + }, + "seconds": 16.509460624999974 + }, + { + "jobs": [ + { + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "status": "ready", + "queue_wait_seconds": 8.555495 + }, + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "ready", + "queue_wait_seconds": 6.303542 + }, + { + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "status": "ready", + "queue_wait_seconds": 9.327098 + }, + { + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "status": "ready", + "queue_wait_seconds": 12.498609 + }, + { + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "status": "ready", + "queue_wait_seconds": 13.962375 + }, + { + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "status": "ready", + "queue_wait_seconds": 15.55325 + }, + { + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "status": "ready", + "queue_wait_seconds": 16.31806 + }, + { + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "status": "processing", + "queue_wait_seconds": 17.018301 + }, + { + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "ready", + "queue_wait_seconds": 1.731303 + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "ready", + "queue_wait_seconds": 2.495207 + }, + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "ready", + "queue_wait_seconds": 3.970311 + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "ready", + "queue_wait_seconds": 4.741965 + }, + { + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "status": "ready", + "queue_wait_seconds": 7.019249 + }, + { + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "status": "ready", + "queue_wait_seconds": 7.813889 + }, + { + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "status": "ready", + "queue_wait_seconds": 10.129006 + }, + { + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "status": "ready", + "queue_wait_seconds": 10.908842 + }, + { + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "status": "ready", + "queue_wait_seconds": 11.726486 + }, + { + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "status": "ready", + "queue_wait_seconds": 13.219188 + }, + { + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "status": "ready", + "queue_wait_seconds": 14.752463 + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "ready", + "queue_wait_seconds": 0.209376 + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "ready", + "queue_wait_seconds": 3.285734 + }, + { + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "ready", + "queue_wait_seconds": 0.951509 + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "ready", + "queue_wait_seconds": 5.523898 + } + ], + "pending": 11, + "rss_bytes": 110653440, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 9, + "messages_unacknowledged": 1 + }, + "seconds": 18.91084945800003 + }, + { + "jobs": [ + { + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "status": "ready", + "queue_wait_seconds": 8.555495 + }, + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "ready", + "queue_wait_seconds": 6.303542 + }, + { + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "status": "ready", + "queue_wait_seconds": 9.327098 + }, + { + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "status": "ready", + "queue_wait_seconds": 12.498609 + }, + { + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "status": "ready", + "queue_wait_seconds": 13.962375 + }, + { + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "status": "ready", + "queue_wait_seconds": 15.55325 + }, + { + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "status": "ready", + "queue_wait_seconds": 16.31806 + }, + { + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "status": "ready", + "queue_wait_seconds": 17.018301 + }, + { + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "status": "ready", + "queue_wait_seconds": 17.822672 + }, + { + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "status": "ready", + "queue_wait_seconds": 18.585936 + }, + { + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "status": "processing", + "queue_wait_seconds": 19.241094 + }, + { + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "ready", + "queue_wait_seconds": 1.731303 + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "ready", + "queue_wait_seconds": 2.495207 + }, + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "ready", + "queue_wait_seconds": 3.970311 + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "ready", + "queue_wait_seconds": 4.741965 + }, + { + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "status": "ready", + "queue_wait_seconds": 7.019249 + }, + { + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "status": "ready", + "queue_wait_seconds": 7.813889 + }, + { + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "status": "ready", + "queue_wait_seconds": 10.129006 + }, + { + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "status": "ready", + "queue_wait_seconds": 10.908842 + }, + { + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "status": "ready", + "queue_wait_seconds": 11.726486 + }, + { + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "status": "ready", + "queue_wait_seconds": 13.219188 + }, + { + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "status": "ready", + "queue_wait_seconds": 14.752463 + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "ready", + "queue_wait_seconds": 0.209376 + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "ready", + "queue_wait_seconds": 3.285734 + }, + { + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "ready", + "queue_wait_seconds": 0.951509 + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "ready", + "queue_wait_seconds": 5.523898 + } + ], + "pending": 8, + "rss_bytes": 119222272, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 6, + "messages_unacknowledged": 1 + }, + "seconds": 21.22383291699998 + }, + { + "jobs": [ + { + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "status": "ready", + "queue_wait_seconds": 8.555495 + }, + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "ready", + "queue_wait_seconds": 6.303542 + }, + { + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "status": "ready", + "queue_wait_seconds": 9.327098 + }, + { + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "status": "ready", + "queue_wait_seconds": 12.498609 + }, + { + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "status": "ready", + "queue_wait_seconds": 13.962375 + }, + { + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "status": "ready", + "queue_wait_seconds": 15.55325 + }, + { + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "status": "ready", + "queue_wait_seconds": 16.31806 + }, + { + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "status": "ready", + "queue_wait_seconds": 17.018301 + }, + { + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "status": "ready", + "queue_wait_seconds": 17.822672 + }, + { + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "status": "ready", + "queue_wait_seconds": 18.585936 + }, + { + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "status": "ready", + "queue_wait_seconds": 19.241094 + }, + { + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "status": "ready", + "queue_wait_seconds": 20.024996 + }, + { + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "status": "ready", + "queue_wait_seconds": 20.702635 + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "ready", + "queue_wait_seconds": 1.731303 + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "ready", + "queue_wait_seconds": 2.495207 + }, + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "ready", + "queue_wait_seconds": 3.970311 + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "ready", + "queue_wait_seconds": 4.741965 + }, + { + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "status": "ready", + "queue_wait_seconds": 7.019249 + }, + { + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "status": "ready", + "queue_wait_seconds": 7.813889 + }, + { + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "status": "ready", + "queue_wait_seconds": 10.129006 + }, + { + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "status": "ready", + "queue_wait_seconds": 10.908842 + }, + { + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "status": "ready", + "queue_wait_seconds": 11.726486 + }, + { + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "status": "ready", + "queue_wait_seconds": 13.219188 + }, + { + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "status": "ready", + "queue_wait_seconds": 14.752463 + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "ready", + "queue_wait_seconds": 0.209376 + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "ready", + "queue_wait_seconds": 3.285734 + }, + { + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "status": "processing", + "queue_wait_seconds": 21.426501 + }, + { + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "ready", + "queue_wait_seconds": 0.951509 + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "ready", + "queue_wait_seconds": 5.523898 + } + ], + "pending": 5, + "rss_bytes": 127066112, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 2, + "messages_unacknowledged": 1 + }, + "seconds": 23.708276166999894 + }, + { + "jobs": [ + { + "id": "6f7750d5-8782-4fbf-948d-0da0149484d3", + "status": "ready", + "queue_wait_seconds": 8.555495 + }, + { + "id": "84141acd-79c7-4907-bbe2-302d93a472f0", + "status": "ready", + "queue_wait_seconds": 6.303542 + }, + { + "id": "c5c0d028-aae4-4e5d-bd34-6343c147f31b", + "status": "ready", + "queue_wait_seconds": 9.327098 + }, + { + "id": "e8125669-0ef6-40a2-ae56-21b9f0a3b3c8", + "status": "ready", + "queue_wait_seconds": 12.498609 + }, + { + "id": "f9abee28-cd49-4544-8d5f-7bf6c89d7fae", + "status": "ready", + "queue_wait_seconds": 13.962375 + }, + { + "id": "cbb07e3b-3bd2-4aa9-bced-b829453cd69b", + "status": "ready", + "queue_wait_seconds": 15.55325 + }, + { + "id": "6b34b540-9632-4ed5-b83f-5eefc4a8e9f9", + "status": "ready", + "queue_wait_seconds": 16.31806 + }, + { + "id": "307ab90b-9e6c-46cf-82cb-78e11fc2e554", + "status": "ready", + "queue_wait_seconds": 17.018301 + }, + { + "id": "d06d213a-e090-4607-9234-82eb3dc4a12d", + "status": "ready", + "queue_wait_seconds": 17.822672 + }, + { + "id": "ceefe6cb-6cda-4387-95b1-a171b764b31a", + "status": "ready", + "queue_wait_seconds": 18.585936 + }, + { + "id": "4370e594-1069-4929-9c47-822854cfb30b", + "status": "ready", + "queue_wait_seconds": 19.241094 + }, + { + "id": "8e8ca509-9d13-4ccd-8898-11a0fe553cae", + "status": "ready", + "queue_wait_seconds": 20.024996 + }, + { + "id": "aecd569f-5cd0-4c23-9255-758971f87ca8", + "status": "ready", + "queue_wait_seconds": 20.702635 + }, + { + "id": "41f91f2d-daf5-4b86-a7a2-d4c4b3d08da0", + "status": "ready", + "queue_wait_seconds": 1.731303 + }, + { + "id": "4af48971-b90a-4bbf-9e77-c03eafb92a2f", + "status": "ready", + "queue_wait_seconds": 2.495207 + }, + { + "id": "e62f6f3d-14bf-4eb1-ac6e-dba09cb7f18b", + "status": "ready", + "queue_wait_seconds": 3.970311 + }, + { + "id": "eba38ebf-2dde-446c-8b75-d8d954e70c37", + "status": "ready", + "queue_wait_seconds": 4.741965 + }, + { + "id": "2edef5e7-013f-4b0c-99fd-86df0c775b7b", + "status": "ready", + "queue_wait_seconds": 7.019249 + }, + { + "id": "13213c8a-4539-4e7c-bd02-d360f2397557", + "status": "ready", + "queue_wait_seconds": 7.813889 + }, + { + "id": "0a80f633-a6d0-42d6-b8ec-a225ade86ddd", + "status": "ready", + "queue_wait_seconds": 10.129006 + }, + { + "id": "6039bdd6-286d-41be-8c68-8cfefeaa2d1f", + "status": "ready", + "queue_wait_seconds": 10.908842 + }, + { + "id": "115c4822-a5f2-4ada-bb1c-18b8b395e1a4", + "status": "ready", + "queue_wait_seconds": 11.726486 + }, + { + "id": "22adfc04-c47d-483f-9076-52bba751e132", + "status": "ready", + "queue_wait_seconds": 13.219188 + }, + { + "id": "d776cbc8-4ba8-40cb-be22-65acb15c7e9c", + "status": "ready", + "queue_wait_seconds": 14.752463 + }, + { + "id": "1763589f-cd6e-4670-8079-a5cf9a62e30b", + "status": "ready", + "queue_wait_seconds": 0.209376 + }, + { + "id": "b4d5bd67-a002-4a16-a25a-b3e217a508fc", + "status": "ready", + "queue_wait_seconds": 3.285734 + }, + { + "id": "da34cfe8-c71f-4c21-ac47-441450bb7247", + "status": "ready", + "queue_wait_seconds": 21.426501 + }, + { + "id": "f247aca0-03e0-4a01-903b-a6c088fe48ee", + "status": "ready", + "queue_wait_seconds": 22.920227 + }, + { + "id": "58852ce8-232b-4bac-99ef-7fafd4db5382", + "status": "ready", + "queue_wait_seconds": 22.143713 + }, + { + "id": "0fe86f08-9562-4213-8195-197ccbd43a9f", + "status": "processing", + "queue_wait_seconds": 23.708966 + }, + { + "id": "2e50c0df-be3f-401d-948b-74da9beebbc0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "924ea00d-60d0-477a-948c-65714ae837ba", + "status": "ready", + "queue_wait_seconds": 0.951509 + }, + { + "id": "3fe199b4-aa5a-45e0-9d98-d0429b5897a7", + "status": "ready", + "queue_wait_seconds": 5.523898 + } + ], + "pending": 2, + "rss_bytes": 140509184, + "concurrency": 1, + "prefetch": 1, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 0, + "messages_unacknowledged": 0 + }, + "seconds": 26.020062332999714 + } + ], + "metric_errors": [], + "limitations": "Closed-loop; ready time is polling upper bound. RSS is sum of worker/container processes, shared pages may be counted twice. Sampling may miss peaks. Queue wait is first claim minus durable job creation." +} diff --git a/evidence/real-c2-p2-s2400.json b/evidence/real-c2-p2-s2400.json new file mode 100644 index 00000000..2e8dd129 --- /dev/null +++ b/evidence/real-c2-p2-s2400.json @@ -0,0 +1,2643 @@ +{ + "run_name": "real-c2-p2-s2400", + "settings": { + "workers": 2, + "prefetch": 2, + "max_pending": 32, + "worker_limits": { + "Memory": 4294967296, + "NanoCpus": 2000000000 + } + }, + "workload": { + "images": 160, + "concurrency": 16, + "size": 2400, + "bytes": 22196, + "type": "closed-loop burst" + }, + "results": { + "accepted": 44, + "rejected_429": 116, + "ready": 44, + "failed": 0, + "timed_out": 0, + "elapsed_seconds": 25.81285724999998, + "ready_per_second": 1.7045768925871247, + "ready_latency_ms": { + "0.5": 12286.007, + "0.95": 18617.316, + "0.99": 18681.231 + }, + "queue_wait_ms": { + "0.5": 11213.428, + "0.95": 17474.298, + "0.99": 17540.778 + }, + "max_sampled_rss_bytes": 426147840, + "max_sampled_pending": 32, + "sample_count": 8, + "drained": true, + "recovery_upload_ready": true + }, + "records": [ + { + "index": 4, + "status": 202, + "upload_seconds": 0.048076250000121945, + "id": "e6220851-907e-4781-a419-3d412c538dd4", + "terminal": "ready", + "ready_seconds": 1.7805040000002919 + }, + { + "index": 0, + "status": 202, + "upload_seconds": 0.11845995900011985, + "id": "71e52817-7638-4a35-a03f-8c4685d8eff7", + "terminal": "ready", + "ready_seconds": 1.7162398329996904 + }, + { + "index": 5, + "status": 202, + "upload_seconds": 0.12580133299979934, + "id": "44b27028-d2f3-441c-a4cb-61705a2bb4d4", + "terminal": "ready", + "ready_seconds": 3.006989542000156 + }, + { + "index": 1, + "status": 202, + "upload_seconds": 0.21564712500003225, + "id": "854d10e1-c62b-4b11-bf5e-84b79d33d242", + "terminal": "ready", + "ready_seconds": 2.924300167000183 + }, + { + "index": 3, + "status": 202, + "upload_seconds": 0.22265766700002132, + "id": "65ac34b4-8281-4ca7-98d3-8a7f82243f8e", + "terminal": "ready", + "ready_seconds": 3.944052792000093 + }, + { + "index": 2, + "status": 202, + "upload_seconds": 0.2544706670000778, + "id": "2862b00e-8286-4691-a75c-e0d78ddbf7df", + "terminal": "ready", + "ready_seconds": 3.915943999999854 + }, + { + "index": 9, + "status": 202, + "upload_seconds": 0.2585247920001166, + "id": "142ca5f5-e7dd-47cf-9e00-bc971cd47f5d", + "terminal": "ready", + "ready_seconds": 5.673512291999941 + }, + { + "index": 14, + "status": 202, + "upload_seconds": 0.30869145799988473, + "id": "97ba2f88-15eb-46f4-96ca-987c2767f020", + "terminal": "ready", + "ready_seconds": 5.627181542000017 + }, + { + "index": 10, + "status": 202, + "upload_seconds": 0.33927591599967855, + "id": "79a508e5-811f-441c-b024-4c9d55bcf4f8", + "terminal": "ready", + "ready_seconds": 6.11677683400012 + }, + { + "index": 11, + "status": 202, + "upload_seconds": 0.3538874579999174, + "id": "c1bfc527-41ed-494b-98ed-6f615706b5d0", + "terminal": "ready", + "ready_seconds": 6.106359124999926 + }, + { + "index": 15, + "status": 202, + "upload_seconds": 0.4135475419998329, + "id": "a2ada32f-005b-47d1-9696-4a6b66347466", + "terminal": "ready", + "ready_seconds": 7.1153355410001495 + }, + { + "index": 13, + "status": 202, + "upload_seconds": 0.42790341699992496, + "id": "bfec05f2-66de-4bd8-946e-1e994ed19a11", + "terminal": "ready", + "ready_seconds": 7.107039875000282 + }, + { + "index": 12, + "status": 202, + "upload_seconds": 0.44114029200000004, + "id": "630c060a-86ea-42e0-b542-735b79195ec4", + "terminal": "ready", + "ready_seconds": 8.085497167000085 + }, + { + "index": 8, + "status": 202, + "upload_seconds": 0.5100770420003755, + "id": "6d4abbd7-a2aa-4311-bf8b-9c8a3c2436a9", + "terminal": "ready", + "ready_seconds": 8.022380375000012 + }, + { + "index": 6, + "status": 202, + "upload_seconds": 0.5357521670002825, + "id": "0faba61a-74a8-4c97-9004-fe21c673a39c", + "terminal": "ready", + "ready_seconds": 9.028811457999836 + }, + { + "index": 7, + "status": 202, + "upload_seconds": 0.5444544580000183, + "id": "5b49ce17-f650-4b73-bf81-1a53e5b8a8ee", + "terminal": "ready", + "ready_seconds": 9.186855916999775 + }, + { + "index": 16, + "status": 202, + "upload_seconds": 0.5663733749997846, + "id": "74e6b6af-7b57-4208-9ab7-69ef32dd7c2e", + "terminal": "ready", + "ready_seconds": 10.100664042000062 + }, + { + "index": 18, + "status": 202, + "upload_seconds": 0.5054580830001214, + "id": "123953ed-7dea-4c1a-95e7-450e36391ccd", + "terminal": "ready", + "ready_seconds": 10.271500166999886 + }, + { + "index": 17, + "status": 202, + "upload_seconds": 0.5986915839998801, + "id": "5004d920-834c-4245-abe7-e3a375cd386e", + "terminal": "ready", + "ready_seconds": 11.219001666000167 + }, + { + "index": 21, + "status": 202, + "upload_seconds": 0.47858416700000816, + "id": "7a0324c8-7c1e-4d8c-b9d5-adb8971f4cd7", + "terminal": "ready", + "ready_seconds": 11.206318208000084 + }, + { + "index": 19, + "status": 202, + "upload_seconds": 0.6187557920002291, + "id": "ffdaa446-d902-4bd1-b190-48f20c5660ea", + "terminal": "ready", + "ready_seconds": 12.286007207999774 + }, + { + "index": 23, + "status": 202, + "upload_seconds": 0.5287360000002082, + "id": "8a0d5a87-96ea-485e-bf82-2e716e8733e7", + "terminal": "ready", + "ready_seconds": 12.282156957999632 + }, + { + "index": 20, + "status": 202, + "upload_seconds": 0.6908607079999456, + "id": "3b807fc4-0b69-4ced-a278-8fa651951b21", + "terminal": "ready", + "ready_seconds": 13.460399250000137 + }, + { + "index": 24, + "status": 202, + "upload_seconds": 0.5853720420000172, + "id": "ebd49517-f5f8-4a57-912a-5de09b590211", + "terminal": "ready", + "ready_seconds": 13.450247833000049 + }, + { + "index": 22, + "status": 202, + "upload_seconds": 0.6991159580002204, + "id": "fa42c4f3-bfd6-4ea9-85d6-5abd2a5d0de0", + "terminal": "ready", + "ready_seconds": 14.663882750000084 + }, + { + "index": 27, + "status": 202, + "upload_seconds": 0.6016568330001064, + "id": "c6aa1109-dd87-4a4a-9963-a75ad569ac3d", + "terminal": "ready", + "ready_seconds": 14.594127542000024 + }, + { + "index": 25, + "status": 202, + "upload_seconds": 0.7569196250001369, + "id": "565b6d76-ec2b-44fc-8f7e-cf1071af6b97", + "terminal": "ready", + "ready_seconds": 15.759266374999697 + }, + { + "index": 34, + "status": 202, + "upload_seconds": 0.4290847910001503, + "id": "d13ba3da-2024-4524-8440-091c9e66dfae", + "terminal": "ready", + "ready_seconds": 15.730743791999885 + }, + { + "index": 26, + "status": 202, + "upload_seconds": 0.8117197920000763, + "id": "0aa92496-ca9b-46cf-9f1f-3b7a7852e3e6", + "terminal": "ready", + "ready_seconds": 16.685640207999768 + }, + { + "index": 36, + "status": 202, + "upload_seconds": 0.4841380420002679, + "id": "d35549e1-f2ca-4107-9441-2f7c17a6841c", + "terminal": "ready", + "ready_seconds": 16.59910895799976 + }, + { + "index": 28, + "status": 202, + "upload_seconds": 0.9014012910001838, + "id": "eaed41cd-ee20-4aa3-bdad-842d688f5ed8", + "terminal": "ready", + "ready_seconds": 17.706869791999907 + }, + { + "index": 38, + "status": 202, + "upload_seconds": 0.4486818749996928, + "id": "2cccdb9e-14ee-4cae-90c9-96c4e9f1bce8", + "terminal": "ready", + "ready_seconds": 17.841155709000304 + }, + { + "index": 29, + "status": 429, + "upload_seconds": 0.8594525829998929 + }, + { + "index": 46, + "status": 429, + "upload_seconds": 0.10282891599990762 + }, + { + "index": 30, + "status": 429, + "upload_seconds": 0.9181065839998155 + }, + { + "index": 47, + "status": 429, + "upload_seconds": 0.17937725000001592 + }, + { + "index": 31, + "status": 429, + "upload_seconds": 0.9977930000000015 + }, + { + "index": 50, + "status": 202, + "upload_seconds": 0.17748645800020313, + "id": "93dcb0b4-9500-41d4-8efb-b8ed9102245a", + "terminal": "ready", + "ready_seconds": 18.48120079199998 + }, + { + "index": 32, + "status": 202, + "upload_seconds": 1.040037792000021, + "id": "f3296a6d-585e-4a54-993b-5b22fae14d79", + "terminal": "ready", + "ready_seconds": 18.617316458000005 + }, + { + "index": 49, + "status": 429, + "upload_seconds": 0.20609425000020565 + }, + { + "index": 33, + "status": 429, + "upload_seconds": 1.096328000000085 + }, + { + "index": 51, + "status": 429, + "upload_seconds": 0.1883593339998697 + }, + { + "index": 35, + "status": 429, + "upload_seconds": 1.021885167000164 + }, + { + "index": 53, + "status": 429, + "upload_seconds": 0.12703379200002018 + }, + { + "index": 54, + "status": 429, + "upload_seconds": 0.18533500000012282 + }, + { + "index": 37, + "status": 429, + "upload_seconds": 1.0091110000003027 + }, + { + "index": 55, + "status": 429, + "upload_seconds": 0.28139487499993265 + }, + { + "index": 39, + "status": 429, + "upload_seconds": 1.0322164999997767 + }, + { + "index": 57, + "status": 429, + "upload_seconds": 0.2928190830002677 + }, + { + "index": 40, + "status": 429, + "upload_seconds": 1.0924864589997014 + }, + { + "index": 59, + "status": 429, + "upload_seconds": 0.36383116600018184 + }, + { + "index": 41, + "status": 429, + "upload_seconds": 1.1262857910001003 + }, + { + "index": 61, + "status": 429, + "upload_seconds": 0.37982483299992964 + }, + { + "index": 42, + "status": 429, + "upload_seconds": 1.1417122080001718 + }, + { + "index": 64, + "status": 429, + "upload_seconds": 0.30003766600020754 + }, + { + "index": 43, + "status": 429, + "upload_seconds": 1.1932412500000282 + }, + { + "index": 44, + "status": 429, + "upload_seconds": 1.1403329999998277 + }, + { + "index": 67, + "status": 429, + "upload_seconds": 0.2618021250000311 + }, + { + "index": 45, + "status": 429, + "upload_seconds": 1.132940165999571 + }, + { + "index": 71, + "status": 429, + "upload_seconds": 0.18453820800004905 + }, + { + "index": 48, + "status": 429, + "upload_seconds": 1.1916312500002277 + }, + { + "index": 73, + "status": 429, + "upload_seconds": 0.20020037499989485 + }, + { + "index": 74, + "status": 429, + "upload_seconds": 0.1966466659996513 + }, + { + "index": 52, + "status": 429, + "upload_seconds": 1.1151706250002462 + }, + { + "index": 75, + "status": 202, + "upload_seconds": 0.2186567919998197, + "id": "ec5db85a-966e-4a3d-a206-fe02eed0e829", + "terminal": "ready", + "ready_seconds": 18.35815966600012 + }, + { + "index": 56, + "status": 202, + "upload_seconds": 1.0291547500000888, + "id": "7cb2fae6-e9ad-4ef2-9de1-59a940631061", + "terminal": "ready", + "ready_seconds": 18.634665916999893 + }, + { + "index": 58, + "status": 429, + "upload_seconds": 1.0813023750001776 + }, + { + "index": 60, + "status": 429, + "upload_seconds": 1.0256571250001798 + }, + { + "index": 77, + "status": 429, + "upload_seconds": 0.3133569169999646 + }, + { + "index": 62, + "status": 429, + "upload_seconds": 1.0147090830000707 + }, + { + "index": 79, + "status": 429, + "upload_seconds": 0.3626584589997037 + }, + { + "index": 78, + "status": 429, + "upload_seconds": 0.4038203329996577 + }, + { + "index": 63, + "status": 429, + "upload_seconds": 1.0955599580001945 + }, + { + "index": 80, + "status": 429, + "upload_seconds": 0.3963498749999417 + }, + { + "index": 65, + "status": 429, + "upload_seconds": 1.1005212499999288 + }, + { + "index": 81, + "status": 429, + "upload_seconds": 0.4779017079999903 + }, + { + "index": 66, + "status": 429, + "upload_seconds": 1.1236495409998497 + }, + { + "index": 83, + "status": 429, + "upload_seconds": 0.45475691599995116 + }, + { + "index": 68, + "status": 429, + "upload_seconds": 1.121875166999871 + }, + { + "index": 84, + "status": 429, + "upload_seconds": 0.49239070800012996 + }, + { + "index": 69, + "status": 429, + "upload_seconds": 1.1870458329999565 + }, + { + "index": 87, + "status": 429, + "upload_seconds": 0.46637129199962146 + }, + { + "index": 70, + "status": 429, + "upload_seconds": 1.2146258749999106 + }, + { + "index": 90, + "status": 429, + "upload_seconds": 0.39783049999959985 + }, + { + "index": 91, + "status": 429, + "upload_seconds": 0.4025347920000968 + }, + { + "index": 72, + "status": 429, + "upload_seconds": 1.2730600419999973 + }, + { + "index": 76, + "status": 429, + "upload_seconds": 1.164409749999777 + }, + { + "index": 96, + "status": 429, + "upload_seconds": 0.28618704100017567 + }, + { + "index": 82, + "status": 202, + "upload_seconds": 0.976462999999967, + "id": "37798bcf-523f-488c-b924-1cbf1401d10f", + "terminal": "ready", + "ready_seconds": 18.46186149999994 + }, + { + "index": 97, + "status": 202, + "upload_seconds": 0.33711458300012964, + "id": "1fc1ecf2-5e90-47e2-8a65-5a4c2ab2a79f", + "terminal": "ready", + "ready_seconds": 18.681231000000025 + }, + { + "index": 85, + "status": 429, + "upload_seconds": 0.906199457999719 + }, + { + "index": 100, + "status": 429, + "upload_seconds": 0.2805070839999644 + }, + { + "index": 86, + "status": 429, + "upload_seconds": 0.9126533329999802 + }, + { + "index": 103, + "status": 429, + "upload_seconds": 0.21831262500018056 + }, + { + "index": 88, + "status": 429, + "upload_seconds": 0.9689429160002874 + }, + { + "index": 105, + "status": 429, + "upload_seconds": 0.18128495900009511 + }, + { + "index": 107, + "status": 429, + "upload_seconds": 0.2108627500001603 + }, + { + "index": 89, + "status": 429, + "upload_seconds": 1.0199856670001282 + }, + { + "index": 108, + "status": 429, + "upload_seconds": 0.29148562499995023 + }, + { + "index": 92, + "status": 429, + "upload_seconds": 1.0041438329999437 + }, + { + "index": 110, + "status": 429, + "upload_seconds": 0.30862241700015147 + }, + { + "index": 93, + "status": 429, + "upload_seconds": 1.0299879580002198 + }, + { + "index": 112, + "status": 429, + "upload_seconds": 0.3024873330000446 + }, + { + "index": 94, + "status": 429, + "upload_seconds": 1.103745957999763 + }, + { + "index": 113, + "status": 429, + "upload_seconds": 0.3658512500001052 + }, + { + "index": 95, + "status": 429, + "upload_seconds": 1.1139779160002945 + }, + { + "index": 117, + "status": 429, + "upload_seconds": 0.26349824999988414 + }, + { + "index": 98, + "status": 429, + "upload_seconds": 1.0940686250000908 + }, + { + "index": 119, + "status": 429, + "upload_seconds": 0.18302300000004834 + }, + { + "index": 99, + "status": 429, + "upload_seconds": 1.1084201660000872 + }, + { + "index": 121, + "status": 429, + "upload_seconds": 0.17867712499992194 + }, + { + "index": 101, + "status": 429, + "upload_seconds": 1.0954202910002095 + }, + { + "index": 124, + "status": 429, + "upload_seconds": 0.1159228329997859 + }, + { + "index": 102, + "status": 429, + "upload_seconds": 1.0323737920002714 + }, + { + "index": 126, + "status": 429, + "upload_seconds": 0.11884741599988047 + }, + { + "index": 104, + "status": 429, + "upload_seconds": 1.112994333000188 + }, + { + "index": 129, + "status": 429, + "upload_seconds": 0.18248741600018548 + }, + { + "index": 130, + "status": 202, + "upload_seconds": 0.2931947920001221, + "id": "9f35b50c-1004-4b04-9517-0b57acf82ca7", + "terminal": "ready", + "ready_seconds": 18.302858041000036 + }, + { + "index": 106, + "status": 202, + "upload_seconds": 1.2888164579999284, + "id": "9eb70243-7438-4793-be4a-f66af995365c", + "terminal": "ready", + "ready_seconds": 18.5428895 + }, + { + "index": 109, + "status": 429, + "upload_seconds": 1.39322808299994 + }, + { + "index": 131, + "status": 429, + "upload_seconds": 0.48574533300006806 + }, + { + "index": 134, + "status": 429, + "upload_seconds": 0.46707845800028736 + }, + { + "index": 111, + "status": 429, + "upload_seconds": 1.5877682089999325 + }, + { + "index": 114, + "status": 429, + "upload_seconds": 1.5180629159999626 + }, + { + "index": 136, + "status": 429, + "upload_seconds": 0.40611200000012104 + }, + { + "index": 137, + "status": 429, + "upload_seconds": 0.22647604200028582 + }, + { + "index": 115, + "status": 429, + "upload_seconds": 1.5930597500000658 + }, + { + "index": 138, + "status": 429, + "upload_seconds": 0.3184854589999304 + }, + { + "index": 116, + "status": 429, + "upload_seconds": 1.6908943749999708 + }, + { + "index": 139, + "status": 429, + "upload_seconds": 0.2975272089997816 + }, + { + "index": 118, + "status": 429, + "upload_seconds": 1.6236075409997284 + }, + { + "index": 120, + "status": 202, + "upload_seconds": 1.629679374999796, + "id": "3f829f5f-2e3b-4b57-b077-8f0415e17749", + "terminal": "ready", + "ready_seconds": 18.371231375000207 + }, + { + "index": 140, + "status": 429, + "upload_seconds": 0.3915386670000771 + }, + { + "index": 122, + "status": 429, + "upload_seconds": 1.6336919160003163 + }, + { + "index": 142, + "status": 202, + "upload_seconds": 0.4794339999998556, + "id": "90d0ca39-6a54-4efe-a8e5-678a61abcca2", + "terminal": "ready", + "ready_seconds": 18.429761333999977 + }, + { + "index": 123, + "status": 429, + "upload_seconds": 1.721432625000034 + }, + { + "index": 145, + "status": 429, + "upload_seconds": 0.37514145800014376 + }, + { + "index": 125, + "status": 429, + "upload_seconds": 1.7810077499998442 + }, + { + "index": 148, + "status": 429, + "upload_seconds": 0.23658520800017868 + }, + { + "index": 127, + "status": 429, + "upload_seconds": 1.7905991249999715 + }, + { + "index": 152, + "status": 429, + "upload_seconds": 0.1318766660001529 + }, + { + "index": 128, + "status": 429, + "upload_seconds": 1.8717867499999556 + }, + { + "index": 155, + "status": 429, + "upload_seconds": 0.11615970799994102 + }, + { + "index": 132, + "status": 429, + "upload_seconds": 1.7131286660001024 + }, + { + "index": 156, + "status": 429, + "upload_seconds": 0.18156408300001203 + }, + { + "index": 133, + "status": 429, + "upload_seconds": 1.624520042000313 + }, + { + "index": 157, + "status": 429, + "upload_seconds": 0.19793895900011194 + }, + { + "index": 135, + "status": 429, + "upload_seconds": 1.5168750420002652 + }, + { + "index": 158, + "status": 429, + "upload_seconds": 0.2729759579997335 + }, + { + "index": 141, + "status": 429, + "upload_seconds": 1.101664540999991 + }, + { + "index": 143, + "status": 429, + "upload_seconds": 1.0194447910002964 + }, + { + "index": 144, + "status": 429, + "upload_seconds": 1.0371107909995771 + }, + { + "index": 146, + "status": 429, + "upload_seconds": 1.0915329170002224 + }, + { + "index": 147, + "status": 429, + "upload_seconds": 1.082200499999999 + }, + { + "index": 149, + "status": 202, + "upload_seconds": 1.0937565829999585, + "id": "b0dad738-3a16-4f70-90d7-7a41bbb7b796", + "terminal": "ready", + "ready_seconds": 18.316909499999838 + }, + { + "index": 150, + "status": 429, + "upload_seconds": 1.0937106249998578 + }, + { + "index": 151, + "status": 202, + "upload_seconds": 1.1536285829997723, + "id": "a06f93b8-9e6a-449a-a7e2-b9a50abe98a1", + "terminal": "ready", + "ready_seconds": 18.25718737499983 + }, + { + "index": 153, + "status": 429, + "upload_seconds": 1.100675166000201 + }, + { + "index": 154, + "status": 429, + "upload_seconds": 1.1878124999998363 + }, + { + "index": 159, + "status": 429, + "upload_seconds": 1.0022970830000304 + } + ], + "samples": [ + { + "jobs": [ + { + "id": "e6220851-907e-4781-a419-3d412c538dd4", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "44b27028-d2f3-441c-a4cb-61705a2bb4d4", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "65ac34b4-8281-4ca7-98d3-8a7f82243f8e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "71e52817-7638-4a35-a03f-8c4685d8eff7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "854d10e1-c62b-4b11-bf5e-84b79d33d242", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2862b00e-8286-4691-a75c-e0d78ddbf7df", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "97ba2f88-15eb-46f4-96ca-987c2767f020", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "142ca5f5-e7dd-47cf-9e00-bc971cd47f5d", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 8, + "rss_bytes": 79306752, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 24, + "messages_unacknowledged": 2 + }, + "seconds": 3.2913291670001854 + }, + { + "jobs": [ + { + "id": "44b27028-d2f3-441c-a4cb-61705a2bb4d4", + "status": "ready", + "queue_wait_seconds": 1.569802 + }, + { + "id": "65ac34b4-8281-4ca7-98d3-8a7f82243f8e", + "status": "ready", + "queue_wait_seconds": 2.636425 + }, + { + "id": "8a0d5a87-96ea-485e-bf82-2e716e8733e7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ffdaa446-d902-4bd1-b190-48f20c5660ea", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "71e52817-7638-4a35-a03f-8c4685d8eff7", + "status": "ready", + "queue_wait_seconds": 0.459176 + }, + { + "id": "854d10e1-c62b-4b11-bf5e-84b79d33d242", + "status": "ready", + "queue_wait_seconds": 1.532247 + }, + { + "id": "2862b00e-8286-4691-a75c-e0d78ddbf7df", + "status": "ready", + "queue_wait_seconds": 2.548381 + }, + { + "id": "97ba2f88-15eb-46f4-96ca-987c2767f020", + "status": "processing", + "queue_wait_seconds": 3.60771 + }, + { + "id": "c1bfc527-41ed-494b-98ed-6f615706b5d0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "bfec05f2-66de-4bd8-946e-1e994ed19a11", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6d4abbd7-a2aa-4311-bf8b-9c8a3c2436a9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5b49ce17-f650-4b73-bf81-1a53e5b8a8ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "74e6b6af-7b57-4208-9ab7-69ef32dd7c2e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5004d920-834c-4245-abe7-e3a375cd386e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "7cb2fae6-e9ad-4ef2-9de1-59a940631061", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37798bcf-523f-488c-b924-1cbf1401d10f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "142ca5f5-e7dd-47cf-9e00-bc971cd47f5d", + "status": "processing", + "queue_wait_seconds": 3.603931 + }, + { + "id": "79a508e5-811f-441c-b024-4c9d55bcf4f8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a2ada32f-005b-47d1-9696-4a6b66347466", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "630c060a-86ea-42e0-b542-735b79195ec4", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0faba61a-74a8-4c97-9004-fe21c673a39c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "123953ed-7dea-4c1a-95e7-450e36391ccd", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "7a0324c8-7c1e-4d8c-b9d5-adb8971f4cd7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ec5db85a-966e-4a3d-a206-fe02eed0e829", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "fa42c4f3-bfd6-4ea9-85d6-5abd2a5d0de0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3b807fc4-0b69-4ced-a278-8fa651951b21", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ebd49517-f5f8-4a57-912a-5de09b590211", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6aa1109-dd87-4a4a-9963-a75ad569ac3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d13ba3da-2024-4524-8440-091c9e66dfae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d35549e1-f2ca-4107-9441-2f7c17a6841c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2cccdb9e-14ee-4cae-90c9-96c4e9f1bce8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1fc1ecf2-5e90-47e2-8a65-5a4c2ab2a79f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "565b6d76-ec2b-44fc-8f7e-cf1071af6b97", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0aa92496-ca9b-46cf-9f1f-3b7a7852e3e6", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "eaed41cd-ee20-4aa3-bdad-842d688f5ed8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f3296a6d-585e-4a54-993b-5b22fae14d79", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e6220851-907e-4781-a419-3d412c538dd4", + "status": "ready", + "queue_wait_seconds": 0.454261 + }, + { + "id": "93dcb0b4-9500-41d4-8efb-b8ed9102245a", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 32, + "rss_bytes": 385646592, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 30, + "messages_unacknowledged": 2 + }, + "seconds": 6.206476792000103 + }, + { + "jobs": [ + { + "id": "a06f93b8-9e6a-449a-a7e2-b9a50abe98a1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "44b27028-d2f3-441c-a4cb-61705a2bb4d4", + "status": "ready", + "queue_wait_seconds": 1.569802 + }, + { + "id": "65ac34b4-8281-4ca7-98d3-8a7f82243f8e", + "status": "ready", + "queue_wait_seconds": 2.636425 + }, + { + "id": "8a0d5a87-96ea-485e-bf82-2e716e8733e7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ffdaa446-d902-4bd1-b190-48f20c5660ea", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "71e52817-7638-4a35-a03f-8c4685d8eff7", + "status": "ready", + "queue_wait_seconds": 0.459176 + }, + { + "id": "854d10e1-c62b-4b11-bf5e-84b79d33d242", + "status": "ready", + "queue_wait_seconds": 1.532247 + }, + { + "id": "2862b00e-8286-4691-a75c-e0d78ddbf7df", + "status": "ready", + "queue_wait_seconds": 2.548381 + }, + { + "id": "97ba2f88-15eb-46f4-96ca-987c2767f020", + "status": "ready", + "queue_wait_seconds": 3.60771 + }, + { + "id": "6d4abbd7-a2aa-4311-bf8b-9c8a3c2436a9", + "status": "processing", + "queue_wait_seconds": 7.017188 + }, + { + "id": "5b49ce17-f650-4b73-bf81-1a53e5b8a8ee", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "74e6b6af-7b57-4208-9ab7-69ef32dd7c2e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5004d920-834c-4245-abe7-e3a375cd386e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9eb70243-7438-4793-be4a-f66af995365c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "7cb2fae6-e9ad-4ef2-9de1-59a940631061", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37798bcf-523f-488c-b924-1cbf1401d10f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3f829f5f-2e3b-4b57-b077-8f0415e17749", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "142ca5f5-e7dd-47cf-9e00-bc971cd47f5d", + "status": "ready", + "queue_wait_seconds": 3.603931 + }, + { + "id": "79a508e5-811f-441c-b024-4c9d55bcf4f8", + "status": "ready", + "queue_wait_seconds": 4.761281 + }, + { + "id": "a2ada32f-005b-47d1-9696-4a6b66347466", + "status": "ready", + "queue_wait_seconds": 5.845664 + }, + { + "id": "630c060a-86ea-42e0-b542-735b79195ec4", + "status": "processing", + "queue_wait_seconds": 6.919159 + }, + { + "id": "0faba61a-74a8-4c97-9004-fe21c673a39c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "123953ed-7dea-4c1a-95e7-450e36391ccd", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "7a0324c8-7c1e-4d8c-b9d5-adb8971f4cd7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ec5db85a-966e-4a3d-a206-fe02eed0e829", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9f35b50c-1004-4b04-9517-0b57acf82ca7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "fa42c4f3-bfd6-4ea9-85d6-5abd2a5d0de0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3b807fc4-0b69-4ced-a278-8fa651951b21", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ebd49517-f5f8-4a57-912a-5de09b590211", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6aa1109-dd87-4a4a-9963-a75ad569ac3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d13ba3da-2024-4524-8440-091c9e66dfae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d35549e1-f2ca-4107-9441-2f7c17a6841c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2cccdb9e-14ee-4cae-90c9-96c4e9f1bce8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1fc1ecf2-5e90-47e2-8a65-5a4c2ab2a79f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "90d0ca39-6a54-4efe-a8e5-678a61abcca2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "565b6d76-ec2b-44fc-8f7e-cf1071af6b97", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0aa92496-ca9b-46cf-9f1f-3b7a7852e3e6", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "eaed41cd-ee20-4aa3-bdad-842d688f5ed8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f3296a6d-585e-4a54-993b-5b22fae14d79", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b0dad738-3a16-4f70-90d7-7a41bbb7b796", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e6220851-907e-4781-a419-3d412c538dd4", + "status": "ready", + "queue_wait_seconds": 0.454261 + }, + { + "id": "93dcb0b4-9500-41d4-8efb-b8ed9102245a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c1bfc527-41ed-494b-98ed-6f615706b5d0", + "status": "ready", + "queue_wait_seconds": 4.843346 + }, + { + "id": "bfec05f2-66de-4bd8-946e-1e994ed19a11", + "status": "ready", + "queue_wait_seconds": 5.990047 + } + ], + "pending": 32, + "rss_bytes": 106242048, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 26, + "messages_unacknowledged": 2 + }, + "seconds": 10.475131666999914 + }, + { + "jobs": [ + { + "id": "a06f93b8-9e6a-449a-a7e2-b9a50abe98a1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "44b27028-d2f3-441c-a4cb-61705a2bb4d4", + "status": "ready", + "queue_wait_seconds": 1.569802 + }, + { + "id": "65ac34b4-8281-4ca7-98d3-8a7f82243f8e", + "status": "ready", + "queue_wait_seconds": 2.636425 + }, + { + "id": "8a0d5a87-96ea-485e-bf82-2e716e8733e7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ffdaa446-d902-4bd1-b190-48f20c5660ea", + "status": "processing", + "queue_wait_seconds": 11.129503 + }, + { + "id": "71e52817-7638-4a35-a03f-8c4685d8eff7", + "status": "ready", + "queue_wait_seconds": 0.459176 + }, + { + "id": "854d10e1-c62b-4b11-bf5e-84b79d33d242", + "status": "ready", + "queue_wait_seconds": 1.532247 + }, + { + "id": "2862b00e-8286-4691-a75c-e0d78ddbf7df", + "status": "ready", + "queue_wait_seconds": 2.548381 + }, + { + "id": "97ba2f88-15eb-46f4-96ca-987c2767f020", + "status": "ready", + "queue_wait_seconds": 3.60771 + }, + { + "id": "5b49ce17-f650-4b73-bf81-1a53e5b8a8ee", + "status": "ready", + "queue_wait_seconds": 8.039887 + }, + { + "id": "74e6b6af-7b57-4208-9ab7-69ef32dd7c2e", + "status": "ready", + "queue_wait_seconds": 8.989359 + }, + { + "id": "5004d920-834c-4245-abe7-e3a375cd386e", + "status": "ready", + "queue_wait_seconds": 10.034437 + }, + { + "id": "9eb70243-7438-4793-be4a-f66af995365c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "7cb2fae6-e9ad-4ef2-9de1-59a940631061", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37798bcf-523f-488c-b924-1cbf1401d10f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3f829f5f-2e3b-4b57-b077-8f0415e17749", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "142ca5f5-e7dd-47cf-9e00-bc971cd47f5d", + "status": "ready", + "queue_wait_seconds": 3.603931 + }, + { + "id": "79a508e5-811f-441c-b024-4c9d55bcf4f8", + "status": "ready", + "queue_wait_seconds": 4.761281 + }, + { + "id": "a2ada32f-005b-47d1-9696-4a6b66347466", + "status": "ready", + "queue_wait_seconds": 5.845664 + }, + { + "id": "630c060a-86ea-42e0-b542-735b79195ec4", + "status": "ready", + "queue_wait_seconds": 6.919159 + }, + { + "id": "0faba61a-74a8-4c97-9004-fe21c673a39c", + "status": "ready", + "queue_wait_seconds": 7.94663 + }, + { + "id": "123953ed-7dea-4c1a-95e7-450e36391ccd", + "status": "ready", + "queue_wait_seconds": 9.033254 + }, + { + "id": "7a0324c8-7c1e-4d8c-b9d5-adb8971f4cd7", + "status": "processing", + "queue_wait_seconds": 10.041972 + }, + { + "id": "ec5db85a-966e-4a3d-a206-fe02eed0e829", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9f35b50c-1004-4b04-9517-0b57acf82ca7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6d4abbd7-a2aa-4311-bf8b-9c8a3c2436a9", + "status": "ready", + "queue_wait_seconds": 7.017188 + }, + { + "id": "fa42c4f3-bfd6-4ea9-85d6-5abd2a5d0de0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3b807fc4-0b69-4ced-a278-8fa651951b21", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ebd49517-f5f8-4a57-912a-5de09b590211", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c6aa1109-dd87-4a4a-9963-a75ad569ac3d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d13ba3da-2024-4524-8440-091c9e66dfae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d35549e1-f2ca-4107-9441-2f7c17a6841c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2cccdb9e-14ee-4cae-90c9-96c4e9f1bce8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1fc1ecf2-5e90-47e2-8a65-5a4c2ab2a79f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "90d0ca39-6a54-4efe-a8e5-678a61abcca2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "565b6d76-ec2b-44fc-8f7e-cf1071af6b97", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0aa92496-ca9b-46cf-9f1f-3b7a7852e3e6", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "eaed41cd-ee20-4aa3-bdad-842d688f5ed8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f3296a6d-585e-4a54-993b-5b22fae14d79", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b0dad738-3a16-4f70-90d7-7a41bbb7b796", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e6220851-907e-4781-a419-3d412c538dd4", + "status": "ready", + "queue_wait_seconds": 0.454261 + }, + { + "id": "93dcb0b4-9500-41d4-8efb-b8ed9102245a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c1bfc527-41ed-494b-98ed-6f615706b5d0", + "status": "ready", + "queue_wait_seconds": 4.843346 + }, + { + "id": "bfec05f2-66de-4bd8-946e-1e994ed19a11", + "status": "ready", + "queue_wait_seconds": 5.990047 + } + ], + "pending": 25, + "rss_bytes": 82882560, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 20, + "messages_unacknowledged": 2 + }, + "seconds": 13.371319917000164 + }, + { + "jobs": [ + { + "id": "a06f93b8-9e6a-449a-a7e2-b9a50abe98a1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "44b27028-d2f3-441c-a4cb-61705a2bb4d4", + "status": "ready", + "queue_wait_seconds": 1.569802 + }, + { + "id": "65ac34b4-8281-4ca7-98d3-8a7f82243f8e", + "status": "ready", + "queue_wait_seconds": 2.636425 + }, + { + "id": "71e52817-7638-4a35-a03f-8c4685d8eff7", + "status": "ready", + "queue_wait_seconds": 0.459176 + }, + { + "id": "854d10e1-c62b-4b11-bf5e-84b79d33d242", + "status": "ready", + "queue_wait_seconds": 1.532247 + }, + { + "id": "2862b00e-8286-4691-a75c-e0d78ddbf7df", + "status": "ready", + "queue_wait_seconds": 2.548381 + }, + { + "id": "97ba2f88-15eb-46f4-96ca-987c2767f020", + "status": "ready", + "queue_wait_seconds": 3.60771 + }, + { + "id": "5b49ce17-f650-4b73-bf81-1a53e5b8a8ee", + "status": "ready", + "queue_wait_seconds": 8.039887 + }, + { + "id": "74e6b6af-7b57-4208-9ab7-69ef32dd7c2e", + "status": "ready", + "queue_wait_seconds": 8.989359 + }, + { + "id": "5004d920-834c-4245-abe7-e3a375cd386e", + "status": "ready", + "queue_wait_seconds": 10.034437 + }, + { + "id": "9eb70243-7438-4793-be4a-f66af995365c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "7cb2fae6-e9ad-4ef2-9de1-59a940631061", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37798bcf-523f-488c-b924-1cbf1401d10f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3f829f5f-2e3b-4b57-b077-8f0415e17749", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "142ca5f5-e7dd-47cf-9e00-bc971cd47f5d", + "status": "ready", + "queue_wait_seconds": 3.603931 + }, + { + "id": "79a508e5-811f-441c-b024-4c9d55bcf4f8", + "status": "ready", + "queue_wait_seconds": 4.761281 + }, + { + "id": "a2ada32f-005b-47d1-9696-4a6b66347466", + "status": "ready", + "queue_wait_seconds": 5.845664 + }, + { + "id": "630c060a-86ea-42e0-b542-735b79195ec4", + "status": "ready", + "queue_wait_seconds": 6.919159 + }, + { + "id": "0faba61a-74a8-4c97-9004-fe21c673a39c", + "status": "ready", + "queue_wait_seconds": 7.94663 + }, + { + "id": "123953ed-7dea-4c1a-95e7-450e36391ccd", + "status": "ready", + "queue_wait_seconds": 9.033254 + }, + { + "id": "7a0324c8-7c1e-4d8c-b9d5-adb8971f4cd7", + "status": "ready", + "queue_wait_seconds": 10.041972 + }, + { + "id": "ec5db85a-966e-4a3d-a206-fe02eed0e829", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9f35b50c-1004-4b04-9517-0b57acf82ca7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6d4abbd7-a2aa-4311-bf8b-9c8a3c2436a9", + "status": "ready", + "queue_wait_seconds": 7.017188 + }, + { + "id": "fa42c4f3-bfd6-4ea9-85d6-5abd2a5d0de0", + "status": "processing", + "queue_wait_seconds": 13.411448 + }, + { + "id": "c6aa1109-dd87-4a4a-9963-a75ad569ac3d", + "status": "processing", + "queue_wait_seconds": 13.348482 + }, + { + "id": "d13ba3da-2024-4524-8440-091c9e66dfae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d35549e1-f2ca-4107-9441-2f7c17a6841c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2cccdb9e-14ee-4cae-90c9-96c4e9f1bce8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1fc1ecf2-5e90-47e2-8a65-5a4c2ab2a79f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "90d0ca39-6a54-4efe-a8e5-678a61abcca2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ffdaa446-d902-4bd1-b190-48f20c5660ea", + "status": "ready", + "queue_wait_seconds": 11.129503 + }, + { + "id": "565b6d76-ec2b-44fc-8f7e-cf1071af6b97", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0aa92496-ca9b-46cf-9f1f-3b7a7852e3e6", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "eaed41cd-ee20-4aa3-bdad-842d688f5ed8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f3296a6d-585e-4a54-993b-5b22fae14d79", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b0dad738-3a16-4f70-90d7-7a41bbb7b796", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8a0d5a87-96ea-485e-bf82-2e716e8733e7", + "status": "ready", + "queue_wait_seconds": 11.213428 + }, + { + "id": "e6220851-907e-4781-a419-3d412c538dd4", + "status": "ready", + "queue_wait_seconds": 0.454261 + }, + { + "id": "3b807fc4-0b69-4ced-a278-8fa651951b21", + "status": "ready", + "queue_wait_seconds": 12.233335 + }, + { + "id": "93dcb0b4-9500-41d4-8efb-b8ed9102245a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ebd49517-f5f8-4a57-912a-5de09b590211", + "status": "ready", + "queue_wait_seconds": 12.217281 + }, + { + "id": "c1bfc527-41ed-494b-98ed-6f615706b5d0", + "status": "ready", + "queue_wait_seconds": 4.843346 + }, + { + "id": "bfec05f2-66de-4bd8-946e-1e994ed19a11", + "status": "ready", + "queue_wait_seconds": 5.990047 + } + ], + "pending": 20, + "rss_bytes": 152965120, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 14, + "messages_unacknowledged": 2 + }, + "seconds": 17.371419708000303 + }, + { + "jobs": [ + { + "id": "a06f93b8-9e6a-449a-a7e2-b9a50abe98a1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "44b27028-d2f3-441c-a4cb-61705a2bb4d4", + "status": "ready", + "queue_wait_seconds": 1.569802 + }, + { + "id": "65ac34b4-8281-4ca7-98d3-8a7f82243f8e", + "status": "ready", + "queue_wait_seconds": 2.636425 + }, + { + "id": "71e52817-7638-4a35-a03f-8c4685d8eff7", + "status": "ready", + "queue_wait_seconds": 0.459176 + }, + { + "id": "854d10e1-c62b-4b11-bf5e-84b79d33d242", + "status": "ready", + "queue_wait_seconds": 1.532247 + }, + { + "id": "2862b00e-8286-4691-a75c-e0d78ddbf7df", + "status": "ready", + "queue_wait_seconds": 2.548381 + }, + { + "id": "97ba2f88-15eb-46f4-96ca-987c2767f020", + "status": "ready", + "queue_wait_seconds": 3.60771 + }, + { + "id": "5b49ce17-f650-4b73-bf81-1a53e5b8a8ee", + "status": "ready", + "queue_wait_seconds": 8.039887 + }, + { + "id": "74e6b6af-7b57-4208-9ab7-69ef32dd7c2e", + "status": "ready", + "queue_wait_seconds": 8.989359 + }, + { + "id": "5004d920-834c-4245-abe7-e3a375cd386e", + "status": "ready", + "queue_wait_seconds": 10.034437 + }, + { + "id": "9eb70243-7438-4793-be4a-f66af995365c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "7cb2fae6-e9ad-4ef2-9de1-59a940631061", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37798bcf-523f-488c-b924-1cbf1401d10f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3f829f5f-2e3b-4b57-b077-8f0415e17749", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "142ca5f5-e7dd-47cf-9e00-bc971cd47f5d", + "status": "ready", + "queue_wait_seconds": 3.603931 + }, + { + "id": "79a508e5-811f-441c-b024-4c9d55bcf4f8", + "status": "ready", + "queue_wait_seconds": 4.761281 + }, + { + "id": "a2ada32f-005b-47d1-9696-4a6b66347466", + "status": "ready", + "queue_wait_seconds": 5.845664 + }, + { + "id": "630c060a-86ea-42e0-b542-735b79195ec4", + "status": "ready", + "queue_wait_seconds": 6.919159 + }, + { + "id": "0faba61a-74a8-4c97-9004-fe21c673a39c", + "status": "ready", + "queue_wait_seconds": 7.94663 + }, + { + "id": "123953ed-7dea-4c1a-95e7-450e36391ccd", + "status": "ready", + "queue_wait_seconds": 9.033254 + }, + { + "id": "7a0324c8-7c1e-4d8c-b9d5-adb8971f4cd7", + "status": "ready", + "queue_wait_seconds": 10.041972 + }, + { + "id": "ec5db85a-966e-4a3d-a206-fe02eed0e829", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9f35b50c-1004-4b04-9517-0b57acf82ca7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6d4abbd7-a2aa-4311-bf8b-9c8a3c2436a9", + "status": "ready", + "queue_wait_seconds": 7.017188 + }, + { + "id": "c6aa1109-dd87-4a4a-9963-a75ad569ac3d", + "status": "ready", + "queue_wait_seconds": 13.348482 + }, + { + "id": "d35549e1-f2ca-4107-9441-2f7c17a6841c", + "status": "ready", + "queue_wait_seconds": 15.563603 + }, + { + "id": "2cccdb9e-14ee-4cae-90c9-96c4e9f1bce8", + "status": "processing", + "queue_wait_seconds": 16.608765 + }, + { + "id": "1fc1ecf2-5e90-47e2-8a65-5a4c2ab2a79f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "90d0ca39-6a54-4efe-a8e5-678a61abcca2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ffdaa446-d902-4bd1-b190-48f20c5660ea", + "status": "ready", + "queue_wait_seconds": 11.129503 + }, + { + "id": "565b6d76-ec2b-44fc-8f7e-cf1071af6b97", + "status": "ready", + "queue_wait_seconds": 14.655773 + }, + { + "id": "0aa92496-ca9b-46cf-9f1f-3b7a7852e3e6", + "status": "ready", + "queue_wait_seconds": 15.635309 + }, + { + "id": "eaed41cd-ee20-4aa3-bdad-842d688f5ed8", + "status": "processing", + "queue_wait_seconds": 16.598575 + }, + { + "id": "f3296a6d-585e-4a54-993b-5b22fae14d79", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b0dad738-3a16-4f70-90d7-7a41bbb7b796", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8a0d5a87-96ea-485e-bf82-2e716e8733e7", + "status": "ready", + "queue_wait_seconds": 11.213428 + }, + { + "id": "e6220851-907e-4781-a419-3d412c538dd4", + "status": "ready", + "queue_wait_seconds": 0.454261 + }, + { + "id": "3b807fc4-0b69-4ced-a278-8fa651951b21", + "status": "ready", + "queue_wait_seconds": 12.233335 + }, + { + "id": "93dcb0b4-9500-41d4-8efb-b8ed9102245a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ebd49517-f5f8-4a57-912a-5de09b590211", + "status": "ready", + "queue_wait_seconds": 12.217281 + }, + { + "id": "c1bfc527-41ed-494b-98ed-6f615706b5d0", + "status": "ready", + "queue_wait_seconds": 4.843346 + }, + { + "id": "fa42c4f3-bfd6-4ea9-85d6-5abd2a5d0de0", + "status": "ready", + "queue_wait_seconds": 13.411448 + }, + { + "id": "bfec05f2-66de-4bd8-946e-1e994ed19a11", + "status": "ready", + "queue_wait_seconds": 5.990047 + }, + { + "id": "d13ba3da-2024-4524-8440-091c9e66dfae", + "status": "ready", + "queue_wait_seconds": 14.561299 + } + ], + "pending": 14, + "rss_bytes": 426147840, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 9, + "messages_unacknowledged": 2 + }, + "seconds": 20.19001366700013 + }, + { + "jobs": [ + { + "id": "a06f93b8-9e6a-449a-a7e2-b9a50abe98a1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "44b27028-d2f3-441c-a4cb-61705a2bb4d4", + "status": "ready", + "queue_wait_seconds": 1.569802 + }, + { + "id": "65ac34b4-8281-4ca7-98d3-8a7f82243f8e", + "status": "ready", + "queue_wait_seconds": 2.636425 + }, + { + "id": "71e52817-7638-4a35-a03f-8c4685d8eff7", + "status": "ready", + "queue_wait_seconds": 0.459176 + }, + { + "id": "854d10e1-c62b-4b11-bf5e-84b79d33d242", + "status": "ready", + "queue_wait_seconds": 1.532247 + }, + { + "id": "2862b00e-8286-4691-a75c-e0d78ddbf7df", + "status": "ready", + "queue_wait_seconds": 2.548381 + }, + { + "id": "97ba2f88-15eb-46f4-96ca-987c2767f020", + "status": "ready", + "queue_wait_seconds": 3.60771 + }, + { + "id": "5b49ce17-f650-4b73-bf81-1a53e5b8a8ee", + "status": "ready", + "queue_wait_seconds": 8.039887 + }, + { + "id": "74e6b6af-7b57-4208-9ab7-69ef32dd7c2e", + "status": "ready", + "queue_wait_seconds": 8.989359 + }, + { + "id": "5004d920-834c-4245-abe7-e3a375cd386e", + "status": "ready", + "queue_wait_seconds": 10.034437 + }, + { + "id": "9eb70243-7438-4793-be4a-f66af995365c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "7cb2fae6-e9ad-4ef2-9de1-59a940631061", + "status": "ready", + "queue_wait_seconds": 17.503307 + }, + { + "id": "37798bcf-523f-488c-b924-1cbf1401d10f", + "status": "processing", + "queue_wait_seconds": 17.332744 + }, + { + "id": "3f829f5f-2e3b-4b57-b077-8f0415e17749", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "142ca5f5-e7dd-47cf-9e00-bc971cd47f5d", + "status": "ready", + "queue_wait_seconds": 3.603931 + }, + { + "id": "79a508e5-811f-441c-b024-4c9d55bcf4f8", + "status": "ready", + "queue_wait_seconds": 4.761281 + }, + { + "id": "a2ada32f-005b-47d1-9696-4a6b66347466", + "status": "ready", + "queue_wait_seconds": 5.845664 + }, + { + "id": "630c060a-86ea-42e0-b542-735b79195ec4", + "status": "ready", + "queue_wait_seconds": 6.919159 + }, + { + "id": "0faba61a-74a8-4c97-9004-fe21c673a39c", + "status": "ready", + "queue_wait_seconds": 7.94663 + }, + { + "id": "123953ed-7dea-4c1a-95e7-450e36391ccd", + "status": "ready", + "queue_wait_seconds": 9.033254 + }, + { + "id": "7a0324c8-7c1e-4d8c-b9d5-adb8971f4cd7", + "status": "ready", + "queue_wait_seconds": 10.041972 + }, + { + "id": "ec5db85a-966e-4a3d-a206-fe02eed0e829", + "status": "ready", + "queue_wait_seconds": 17.324338 + }, + { + "id": "9f35b50c-1004-4b04-9517-0b57acf82ca7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6d4abbd7-a2aa-4311-bf8b-9c8a3c2436a9", + "status": "ready", + "queue_wait_seconds": 7.017188 + }, + { + "id": "c6aa1109-dd87-4a4a-9963-a75ad569ac3d", + "status": "ready", + "queue_wait_seconds": 13.348482 + }, + { + "id": "d35549e1-f2ca-4107-9441-2f7c17a6841c", + "status": "ready", + "queue_wait_seconds": 15.563603 + }, + { + "id": "2cccdb9e-14ee-4cae-90c9-96c4e9f1bce8", + "status": "ready", + "queue_wait_seconds": 16.608765 + }, + { + "id": "1fc1ecf2-5e90-47e2-8a65-5a4c2ab2a79f", + "status": "processing", + "queue_wait_seconds": 17.474298 + }, + { + "id": "90d0ca39-6a54-4efe-a8e5-678a61abcca2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ffdaa446-d902-4bd1-b190-48f20c5660ea", + "status": "ready", + "queue_wait_seconds": 11.129503 + }, + { + "id": "565b6d76-ec2b-44fc-8f7e-cf1071af6b97", + "status": "ready", + "queue_wait_seconds": 14.655773 + }, + { + "id": "0aa92496-ca9b-46cf-9f1f-3b7a7852e3e6", + "status": "ready", + "queue_wait_seconds": 15.635309 + }, + { + "id": "eaed41cd-ee20-4aa3-bdad-842d688f5ed8", + "status": "ready", + "queue_wait_seconds": 16.598575 + }, + { + "id": "f3296a6d-585e-4a54-993b-5b22fae14d79", + "status": "ready", + "queue_wait_seconds": 17.540778 + }, + { + "id": "b0dad738-3a16-4f70-90d7-7a41bbb7b796", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8a0d5a87-96ea-485e-bf82-2e716e8733e7", + "status": "ready", + "queue_wait_seconds": 11.213428 + }, + { + "id": "e6220851-907e-4781-a419-3d412c538dd4", + "status": "ready", + "queue_wait_seconds": 0.454261 + }, + { + "id": "3b807fc4-0b69-4ced-a278-8fa651951b21", + "status": "ready", + "queue_wait_seconds": 12.233335 + }, + { + "id": "93dcb0b4-9500-41d4-8efb-b8ed9102245a", + "status": "ready", + "queue_wait_seconds": 17.417498 + }, + { + "id": "ebd49517-f5f8-4a57-912a-5de09b590211", + "status": "ready", + "queue_wait_seconds": 12.217281 + }, + { + "id": "c1bfc527-41ed-494b-98ed-6f615706b5d0", + "status": "ready", + "queue_wait_seconds": 4.843346 + }, + { + "id": "fa42c4f3-bfd6-4ea9-85d6-5abd2a5d0de0", + "status": "ready", + "queue_wait_seconds": 13.411448 + }, + { + "id": "bfec05f2-66de-4bd8-946e-1e994ed19a11", + "status": "ready", + "queue_wait_seconds": 5.990047 + }, + { + "id": "d13ba3da-2024-4524-8440-091c9e66dfae", + "status": "ready", + "queue_wait_seconds": 14.561299 + } + ], + "pending": 8, + "rss_bytes": 161169408, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 4, + "messages_unacknowledged": 2 + }, + "seconds": 23.17149562500026 + }, + { + "jobs": [ + { + "id": "a06f93b8-9e6a-449a-a7e2-b9a50abe98a1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3f829f5f-2e3b-4b57-b077-8f0415e17749", + "status": "ready", + "queue_wait_seconds": 17.216875 + }, + { + "id": "44b27028-d2f3-441c-a4cb-61705a2bb4d4", + "status": "ready", + "queue_wait_seconds": 1.569802 + }, + { + "id": "65ac34b4-8281-4ca7-98d3-8a7f82243f8e", + "status": "ready", + "queue_wait_seconds": 2.636425 + }, + { + "id": "71e52817-7638-4a35-a03f-8c4685d8eff7", + "status": "ready", + "queue_wait_seconds": 0.459176 + }, + { + "id": "854d10e1-c62b-4b11-bf5e-84b79d33d242", + "status": "ready", + "queue_wait_seconds": 1.532247 + }, + { + "id": "2862b00e-8286-4691-a75c-e0d78ddbf7df", + "status": "ready", + "queue_wait_seconds": 2.548381 + }, + { + "id": "97ba2f88-15eb-46f4-96ca-987c2767f020", + "status": "ready", + "queue_wait_seconds": 3.60771 + }, + { + "id": "5b49ce17-f650-4b73-bf81-1a53e5b8a8ee", + "status": "ready", + "queue_wait_seconds": 8.039887 + }, + { + "id": "74e6b6af-7b57-4208-9ab7-69ef32dd7c2e", + "status": "ready", + "queue_wait_seconds": 8.989359 + }, + { + "id": "5004d920-834c-4245-abe7-e3a375cd386e", + "status": "ready", + "queue_wait_seconds": 10.034437 + }, + { + "id": "9eb70243-7438-4793-be4a-f66af995365c", + "status": "ready", + "queue_wait_seconds": 17.447926 + }, + { + "id": "7cb2fae6-e9ad-4ef2-9de1-59a940631061", + "status": "ready", + "queue_wait_seconds": 17.503307 + }, + { + "id": "37798bcf-523f-488c-b924-1cbf1401d10f", + "status": "ready", + "queue_wait_seconds": 17.332744 + }, + { + "id": "142ca5f5-e7dd-47cf-9e00-bc971cd47f5d", + "status": "ready", + "queue_wait_seconds": 3.603931 + }, + { + "id": "79a508e5-811f-441c-b024-4c9d55bcf4f8", + "status": "ready", + "queue_wait_seconds": 4.761281 + }, + { + "id": "a2ada32f-005b-47d1-9696-4a6b66347466", + "status": "ready", + "queue_wait_seconds": 5.845664 + }, + { + "id": "630c060a-86ea-42e0-b542-735b79195ec4", + "status": "ready", + "queue_wait_seconds": 6.919159 + }, + { + "id": "0faba61a-74a8-4c97-9004-fe21c673a39c", + "status": "ready", + "queue_wait_seconds": 7.94663 + }, + { + "id": "123953ed-7dea-4c1a-95e7-450e36391ccd", + "status": "ready", + "queue_wait_seconds": 9.033254 + }, + { + "id": "7a0324c8-7c1e-4d8c-b9d5-adb8971f4cd7", + "status": "ready", + "queue_wait_seconds": 10.041972 + }, + { + "id": "ec5db85a-966e-4a3d-a206-fe02eed0e829", + "status": "ready", + "queue_wait_seconds": 17.324338 + }, + { + "id": "9f35b50c-1004-4b04-9517-0b57acf82ca7", + "status": "ready", + "queue_wait_seconds": 17.211469 + }, + { + "id": "6d4abbd7-a2aa-4311-bf8b-9c8a3c2436a9", + "status": "ready", + "queue_wait_seconds": 7.017188 + }, + { + "id": "c6aa1109-dd87-4a4a-9963-a75ad569ac3d", + "status": "ready", + "queue_wait_seconds": 13.348482 + }, + { + "id": "d35549e1-f2ca-4107-9441-2f7c17a6841c", + "status": "ready", + "queue_wait_seconds": 15.563603 + }, + { + "id": "2cccdb9e-14ee-4cae-90c9-96c4e9f1bce8", + "status": "ready", + "queue_wait_seconds": 16.608765 + }, + { + "id": "1fc1ecf2-5e90-47e2-8a65-5a4c2ab2a79f", + "status": "ready", + "queue_wait_seconds": 17.474298 + }, + { + "id": "90d0ca39-6a54-4efe-a8e5-678a61abcca2", + "status": "processing", + "queue_wait_seconds": 17.362298 + }, + { + "id": "ffdaa446-d902-4bd1-b190-48f20c5660ea", + "status": "ready", + "queue_wait_seconds": 11.129503 + }, + { + "id": "565b6d76-ec2b-44fc-8f7e-cf1071af6b97", + "status": "ready", + "queue_wait_seconds": 14.655773 + }, + { + "id": "0aa92496-ca9b-46cf-9f1f-3b7a7852e3e6", + "status": "ready", + "queue_wait_seconds": 15.635309 + }, + { + "id": "eaed41cd-ee20-4aa3-bdad-842d688f5ed8", + "status": "ready", + "queue_wait_seconds": 16.598575 + }, + { + "id": "f3296a6d-585e-4a54-993b-5b22fae14d79", + "status": "ready", + "queue_wait_seconds": 17.540778 + }, + { + "id": "b0dad738-3a16-4f70-90d7-7a41bbb7b796", + "status": "processing", + "queue_wait_seconds": 17.147551 + }, + { + "id": "8a0d5a87-96ea-485e-bf82-2e716e8733e7", + "status": "ready", + "queue_wait_seconds": 11.213428 + }, + { + "id": "e6220851-907e-4781-a419-3d412c538dd4", + "status": "ready", + "queue_wait_seconds": 0.454261 + }, + { + "id": "3b807fc4-0b69-4ced-a278-8fa651951b21", + "status": "ready", + "queue_wait_seconds": 12.233335 + }, + { + "id": "93dcb0b4-9500-41d4-8efb-b8ed9102245a", + "status": "ready", + "queue_wait_seconds": 17.417498 + }, + { + "id": "ebd49517-f5f8-4a57-912a-5de09b590211", + "status": "ready", + "queue_wait_seconds": 12.217281 + }, + { + "id": "c1bfc527-41ed-494b-98ed-6f615706b5d0", + "status": "ready", + "queue_wait_seconds": 4.843346 + }, + { + "id": "fa42c4f3-bfd6-4ea9-85d6-5abd2a5d0de0", + "status": "ready", + "queue_wait_seconds": 13.411448 + }, + { + "id": "bfec05f2-66de-4bd8-946e-1e994ed19a11", + "status": "ready", + "queue_wait_seconds": 5.990047 + }, + { + "id": "d13ba3da-2024-4524-8440-091c9e66dfae", + "status": "ready", + "queue_wait_seconds": 14.561299 + } + ], + "pending": 3, + "rss_bytes": 274051072, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 0, + "messages_unacknowledged": 0 + }, + "seconds": 25.793395458000305 + } + ], + "metric_errors": [], + "limitations": "Closed-loop; ready time is polling upper bound. RSS is sum of worker/container processes, shared pages may be counted twice. Sampling may miss peaks. Queue wait is first claim minus durable job creation." +} diff --git a/evidence/real-c2-p2-s512.json b/evidence/real-c2-p2-s512.json new file mode 100644 index 00000000..feafc217 --- /dev/null +++ b/evidence/real-c2-p2-s512.json @@ -0,0 +1,1920 @@ +{ + "run_name": "real-c2-p2-s512", + "settings": { + "workers": 2, + "prefetch": 2, + "max_pending": 32, + "worker_limits": { + "Memory": 4294967296, + "NanoCpus": 2000000000 + } + }, + "workload": { + "images": 160, + "concurrency": 16, + "size": 512, + "bytes": 1890, + "type": "closed-loop burst" + }, + "results": { + "accepted": 34, + "rejected_429": 126, + "ready": 34, + "failed": 0, + "timed_out": 0, + "elapsed_seconds": 16.242352083999776, + "ready_per_second": 2.093292881730667, + "ready_latency_ms": { + "0.5": 8214.26, + "0.95": 14518.909, + "0.99": 14526.648 + }, + "queue_wait_ms": { + "0.5": 7368.448, + "0.95": 13571.623, + "0.99": 13572.687 + }, + "max_sampled_rss_bytes": 165621760, + "max_sampled_pending": 26, + "sample_count": 6, + "drained": true, + "recovery_upload_ready": true + }, + "records": [ + { + "index": 3, + "status": 202, + "upload_seconds": 0.12917175000029602, + "id": "a84ca007-7400-48eb-8a8a-6ea4d8d10764", + "terminal": "ready", + "ready_seconds": 1.6046167499998774 + }, + { + "index": 0, + "status": 202, + "upload_seconds": 0.15294637499982855, + "id": "ea038f65-febd-4f5a-8629-34e33e7d3896", + "terminal": "ready", + "ready_seconds": 1.5883122500003992 + }, + { + "index": 2, + "status": 202, + "upload_seconds": 0.22641291699983412, + "id": "2365078f-c7cd-4bbb-974b-66cc7c27a8b7", + "terminal": "ready", + "ready_seconds": 2.242202833000192 + }, + { + "index": 9, + "status": 202, + "upload_seconds": 0.2302292919998763, + "id": "ba9f5a1c-270f-4c54-80e7-363b70205259", + "terminal": "ready", + "ready_seconds": 2.2385437080001793 + }, + { + "index": 5, + "status": 202, + "upload_seconds": 0.2409649169999284, + "id": "0c3181b1-b01f-498a-89a6-f404f05575e6", + "terminal": "ready", + "ready_seconds": 3.1129356249998636 + }, + { + "index": 1, + "status": 202, + "upload_seconds": 0.24977929199985738, + "id": "b03cff8e-576e-426a-9bbb-c11ba1d225bf", + "terminal": "ready", + "ready_seconds": 3.110197624999728 + }, + { + "index": 7, + "status": 202, + "upload_seconds": 0.25349850000020524, + "id": "40849cc4-696e-4202-8f7d-861bd692ba90", + "terminal": "ready", + "ready_seconds": 4.045680833999995 + }, + { + "index": 4, + "status": 202, + "upload_seconds": 0.3135128750000149, + "id": "0263ce9f-8c4c-43a8-885d-0be62e08d8fd", + "terminal": "ready", + "ready_seconds": 3.9903834579999966 + }, + { + "index": 6, + "status": 202, + "upload_seconds": 0.3302817080002569, + "id": "b023a107-e51c-4714-bdbb-1ca6346f5654", + "terminal": "ready", + "ready_seconds": 4.831816458000048 + }, + { + "index": 8, + "status": 202, + "upload_seconds": 0.33681625000008353, + "id": "f9a273e0-3ac2-4cc4-b4f6-648df6b93f71", + "terminal": "ready", + "ready_seconds": 4.831954708000012 + }, + { + "index": 14, + "status": 202, + "upload_seconds": 0.3457225409997591, + "id": "f7792a5e-d4da-47fe-8817-eb68d08420a3", + "terminal": "ready", + "ready_seconds": 5.751106209000227 + }, + { + "index": 10, + "status": 202, + "upload_seconds": 0.42660366599966437, + "id": "1734ef1f-7449-4dc9-a238-2d895b7933f0", + "terminal": "ready", + "ready_seconds": 5.676397500000348 + }, + { + "index": 15, + "status": 202, + "upload_seconds": 0.4359824580001259, + "id": "f141ec67-c5ac-4ff8-8d86-977d0bf8d2d6", + "terminal": "ready", + "ready_seconds": 6.410713999999643 + }, + { + "index": 13, + "status": 202, + "upload_seconds": 0.445325208000213, + "id": "23c140b7-2b0c-46fb-88ef-79b141200e09", + "terminal": "ready", + "ready_seconds": 6.407209459000114 + }, + { + "index": 12, + "status": 202, + "upload_seconds": 0.5144284580001113, + "id": "228b4f66-8b1e-4b8a-933f-a1cd75aca8db", + "terminal": "ready", + "ready_seconds": 7.367557332999695 + }, + { + "index": 11, + "status": 202, + "upload_seconds": 0.5186863330000051, + "id": "7546c2c1-0322-4c7c-b0af-82c53c6bda0f", + "terminal": "ready", + "ready_seconds": 7.365603624999949 + }, + { + "index": 17, + "status": 202, + "upload_seconds": 0.3798642080000718, + "id": "a53d81be-6ca2-42a1-b5b3-f7f088501717", + "terminal": "ready", + "ready_seconds": 8.214260249999825 + }, + { + "index": 16, + "status": 202, + "upload_seconds": 0.4072741249997307, + "id": "e635f5c9-37f7-4f9a-b1e3-87fbc8b6b137", + "terminal": "ready", + "ready_seconds": 8.214776875000098 + }, + { + "index": 19, + "status": 202, + "upload_seconds": 0.30660641700023916, + "id": "f47ac0e4-1765-4a50-bd05-16fbd1cab48e", + "terminal": "ready", + "ready_seconds": 9.025760666999759 + }, + { + "index": 18, + "status": 202, + "upload_seconds": 0.32717766699988715, + "id": "10834475-21f8-4eed-bbbc-851359677ec9", + "terminal": "ready", + "ready_seconds": 9.01737862500022 + }, + { + "index": 21, + "status": 202, + "upload_seconds": 0.3869310420000147, + "id": "71a80c0b-7732-4d22-8b7b-2c2d2f5eaaf4", + "terminal": "ready", + "ready_seconds": 9.89056629200013 + }, + { + "index": 20, + "status": 202, + "upload_seconds": 0.40076808300000266, + "id": "60b11a2c-1abe-439b-a017-d539e8b6cae6", + "terminal": "ready", + "ready_seconds": 9.889409333999993 + }, + { + "index": 23, + "status": 202, + "upload_seconds": 0.3348280830000476, + "id": "ede9d41c-0ae6-4aa9-b863-48c6f97e6b06", + "terminal": "ready", + "ready_seconds": 10.760795124999731 + }, + { + "index": 22, + "status": 202, + "upload_seconds": 0.40038170800016815, + "id": "a04e0db9-273d-454a-961c-b9f559838e8c", + "terminal": "ready", + "ready_seconds": 10.756632874999923 + }, + { + "index": 27, + "status": 202, + "upload_seconds": 0.2850536250002733, + "id": "e34722a1-983e-4fe5-9095-d94c45578213", + "terminal": "ready", + "ready_seconds": 11.514665457999854 + }, + { + "index": 24, + "status": 202, + "upload_seconds": 0.3895994999998038, + "id": "aa4a7430-b50a-4b1d-95a5-8243e93a44dc", + "terminal": "ready", + "ready_seconds": 11.512222792000102 + }, + { + "index": 29, + "status": 202, + "upload_seconds": 0.2759498329996859, + "id": "c4933534-3cdf-4868-b408-4740fcf4ce49", + "terminal": "ready", + "ready_seconds": 12.463432209000075 + }, + { + "index": 25, + "status": 202, + "upload_seconds": 0.39353587499999776, + "id": "5f437b62-c9e4-4b54-819f-1ed5cf971451", + "terminal": "ready", + "ready_seconds": 12.459671916999923 + }, + { + "index": 31, + "status": 202, + "upload_seconds": 0.21517233299982763, + "id": "f021eec7-4f30-4c07-b808-3832ea84ec42", + "terminal": "ready", + "ready_seconds": 13.555949957999928 + }, + { + "index": 26, + "status": 202, + "upload_seconds": 0.3996991250000974, + "id": "770aa1ca-017d-48b3-ba75-b0c193019735", + "terminal": "ready", + "ready_seconds": 13.54779974999974 + }, + { + "index": 33, + "status": 202, + "upload_seconds": 0.22272112500013463, + "id": "8fe5707f-def4-4d86-9cd2-6ebfb820ba0c", + "terminal": "ready", + "ready_seconds": 14.501860541000042 + }, + { + "index": 28, + "status": 202, + "upload_seconds": 0.40217966599993815, + "id": "0fcb3a86-365b-42a2-b25f-8f7e7469bfa9", + "terminal": "ready", + "ready_seconds": 14.299741541999992 + }, + { + "index": 35, + "status": 429, + "upload_seconds": 0.29108162500006074 + }, + { + "index": 36, + "status": 429, + "upload_seconds": 0.21654262499987453 + }, + { + "index": 30, + "status": 429, + "upload_seconds": 0.33678091699994184 + }, + { + "index": 37, + "status": 429, + "upload_seconds": 0.21945070800029498 + }, + { + "index": 32, + "status": 429, + "upload_seconds": 0.3333044579999296 + }, + { + "index": 34, + "status": 429, + "upload_seconds": 0.3318352500000401 + }, + { + "index": 38, + "status": 429, + "upload_seconds": 0.22342429099990113 + }, + { + "index": 39, + "status": 429, + "upload_seconds": 0.22433783299993593 + }, + { + "index": 41, + "status": 429, + "upload_seconds": 0.1630755000001045 + }, + { + "index": 40, + "status": 429, + "upload_seconds": 0.2058624580004107 + }, + { + "index": 42, + "status": 429, + "upload_seconds": 0.2000157920001584 + }, + { + "index": 43, + "status": 429, + "upload_seconds": 0.19792220800036375 + }, + { + "index": 44, + "status": 429, + "upload_seconds": 0.1964950419996967 + }, + { + "index": 45, + "status": 429, + "upload_seconds": 0.1869903339998018 + }, + { + "index": 47, + "status": 429, + "upload_seconds": 0.10014587500018024 + }, + { + "index": 46, + "status": 429, + "upload_seconds": 0.1863770420000037 + }, + { + "index": 48, + "status": 429, + "upload_seconds": 0.11036904100001266 + }, + { + "index": 50, + "status": 429, + "upload_seconds": 0.10014579100015908 + }, + { + "index": 49, + "status": 429, + "upload_seconds": 0.10872033300029216 + }, + { + "index": 52, + "status": 429, + "upload_seconds": 0.100687999999991 + }, + { + "index": 51, + "status": 429, + "upload_seconds": 0.10719833400025891 + }, + { + "index": 54, + "status": 429, + "upload_seconds": 0.10036358399975143 + }, + { + "index": 55, + "status": 429, + "upload_seconds": 0.1445132079998075 + }, + { + "index": 53, + "status": 429, + "upload_seconds": 0.15387408399965352 + }, + { + "index": 57, + "status": 429, + "upload_seconds": 0.11083912499998405 + }, + { + "index": 56, + "status": 429, + "upload_seconds": 0.14729929099985384 + }, + { + "index": 60, + "status": 429, + "upload_seconds": 0.10116270800017446 + }, + { + "index": 58, + "status": 429, + "upload_seconds": 0.1153104169998187 + }, + { + "index": 62, + "status": 429, + "upload_seconds": 0.09897995800019999 + }, + { + "index": 59, + "status": 429, + "upload_seconds": 0.11751304200015511 + }, + { + "index": 64, + "status": 429, + "upload_seconds": 0.09450020800022685 + }, + { + "index": 61, + "status": 429, + "upload_seconds": 0.11443654200002129 + }, + { + "index": 67, + "status": 429, + "upload_seconds": 0.08928904200001853 + }, + { + "index": 63, + "status": 429, + "upload_seconds": 0.11487845800002106 + }, + { + "index": 68, + "status": 429, + "upload_seconds": 0.093265208000048 + }, + { + "index": 65, + "status": 429, + "upload_seconds": 0.11132516600036979 + }, + { + "index": 70, + "status": 429, + "upload_seconds": 0.04351945799999157 + }, + { + "index": 66, + "status": 429, + "upload_seconds": 0.15564237500029776 + }, + { + "index": 73, + "status": 429, + "upload_seconds": 0.09278012499999022 + }, + { + "index": 69, + "status": 429, + "upload_seconds": 0.15309237500014206 + }, + { + "index": 71, + "status": 429, + "upload_seconds": 0.10693475000016406 + }, + { + "index": 74, + "status": 429, + "upload_seconds": 0.09867783400022745 + }, + { + "index": 76, + "status": 429, + "upload_seconds": 0.09871441599989339 + }, + { + "index": 72, + "status": 429, + "upload_seconds": 0.10997062499973254 + }, + { + "index": 78, + "status": 429, + "upload_seconds": 0.09929504199999428 + }, + { + "index": 75, + "status": 429, + "upload_seconds": 0.11062137499993696 + }, + { + "index": 80, + "status": 429, + "upload_seconds": 0.10318233399993915 + }, + { + "index": 77, + "status": 429, + "upload_seconds": 0.11292566700012685 + }, + { + "index": 82, + "status": 429, + "upload_seconds": 0.10396441700004289 + }, + { + "index": 79, + "status": 429, + "upload_seconds": 0.115623499999856 + }, + { + "index": 81, + "status": 429, + "upload_seconds": 0.16020670799980508 + }, + { + "index": 84, + "status": 429, + "upload_seconds": 0.15430979199982175 + }, + { + "index": 83, + "status": 429, + "upload_seconds": 0.16093991700017796 + }, + { + "index": 87, + "status": 429, + "upload_seconds": 0.1078883750001296 + }, + { + "index": 85, + "status": 429, + "upload_seconds": 0.11655912500009435 + }, + { + "index": 88, + "status": 429, + "upload_seconds": 0.10763095799984512 + }, + { + "index": 86, + "status": 429, + "upload_seconds": 0.1166600409997045 + }, + { + "index": 89, + "status": 429, + "upload_seconds": 0.11480037499995888 + }, + { + "index": 91, + "status": 429, + "upload_seconds": 0.10773066599995218 + }, + { + "index": 90, + "status": 429, + "upload_seconds": 0.11548649999986083 + }, + { + "index": 93, + "status": 429, + "upload_seconds": 0.1068061249998209 + }, + { + "index": 94, + "status": 429, + "upload_seconds": 0.10609625000006417 + }, + { + "index": 92, + "status": 429, + "upload_seconds": 0.1718456249996052 + }, + { + "index": 95, + "status": 429, + "upload_seconds": 0.163274709000234 + }, + { + "index": 97, + "status": 429, + "upload_seconds": 0.15980420799996864 + }, + { + "index": 96, + "status": 429, + "upload_seconds": 0.16382266699974934 + }, + { + "index": 100, + "status": 429, + "upload_seconds": 0.10853450000013254 + }, + { + "index": 98, + "status": 429, + "upload_seconds": 0.11782604199970592 + }, + { + "index": 101, + "status": 429, + "upload_seconds": 0.10783325000011246 + }, + { + "index": 104, + "status": 429, + "upload_seconds": 0.10794595900006243 + }, + { + "index": 99, + "status": 429, + "upload_seconds": 0.12741620799988596 + }, + { + "index": 102, + "status": 429, + "upload_seconds": 0.12278420800021195 + }, + { + "index": 105, + "status": 429, + "upload_seconds": 0.10986741600027017 + }, + { + "index": 103, + "status": 429, + "upload_seconds": 0.12497666599983859 + }, + { + "index": 106, + "status": 429, + "upload_seconds": 0.12057433400013906 + }, + { + "index": 108, + "status": 429, + "upload_seconds": 0.1594689170001402 + }, + { + "index": 107, + "status": 429, + "upload_seconds": 0.16620966699974815 + }, + { + "index": 111, + "status": 429, + "upload_seconds": 0.10261899999977686 + }, + { + "index": 109, + "status": 429, + "upload_seconds": 0.1667495830001826 + }, + { + "index": 112, + "status": 429, + "upload_seconds": 0.10666212500018446 + }, + { + "index": 115, + "status": 429, + "upload_seconds": 0.09983962500018606 + }, + { + "index": 110, + "status": 429, + "upload_seconds": 0.11967950000007477 + }, + { + "index": 116, + "status": 429, + "upload_seconds": 0.10513341700016099 + }, + { + "index": 113, + "status": 429, + "upload_seconds": 0.11632737499985524 + }, + { + "index": 114, + "status": 429, + "upload_seconds": 0.12158854200015412 + }, + { + "index": 117, + "status": 429, + "upload_seconds": 0.109022707999884 + }, + { + "index": 118, + "status": 429, + "upload_seconds": 0.11288891599997442 + }, + { + "index": 119, + "status": 429, + "upload_seconds": 0.15883933399982197 + }, + { + "index": 121, + "status": 429, + "upload_seconds": 0.15077804200018363 + }, + { + "index": 120, + "status": 429, + "upload_seconds": 0.1647544590000507 + }, + { + "index": 122, + "status": 429, + "upload_seconds": 0.16401112499988812 + }, + { + "index": 124, + "status": 429, + "upload_seconds": 0.11339029100008702 + }, + { + "index": 123, + "status": 429, + "upload_seconds": 0.1210008339999149 + }, + { + "index": 125, + "status": 429, + "upload_seconds": 0.11970100000007733 + }, + { + "index": 126, + "status": 429, + "upload_seconds": 0.11676124999985404 + }, + { + "index": 128, + "status": 429, + "upload_seconds": 0.11316583300003913 + }, + { + "index": 127, + "status": 429, + "upload_seconds": 0.12119091700014906 + }, + { + "index": 131, + "status": 429, + "upload_seconds": 0.1097740000000158 + }, + { + "index": 129, + "status": 429, + "upload_seconds": 0.11966004099986094 + }, + { + "index": 132, + "status": 429, + "upload_seconds": 0.10448008299999856 + }, + { + "index": 130, + "status": 429, + "upload_seconds": 0.1176833749996149 + }, + { + "index": 135, + "status": 429, + "upload_seconds": 0.04971216599960826 + }, + { + "index": 133, + "status": 429, + "upload_seconds": 0.11015466600019863 + }, + { + "index": 136, + "status": 429, + "upload_seconds": 0.05554258300026049 + }, + { + "index": 134, + "status": 429, + "upload_seconds": 0.10946441599980972 + }, + { + "index": 137, + "status": 429, + "upload_seconds": 0.08972470899971086 + }, + { + "index": 139, + "status": 429, + "upload_seconds": 0.07857033399977809 + }, + { + "index": 140, + "status": 429, + "upload_seconds": 0.07967250000001513 + }, + { + "index": 138, + "status": 429, + "upload_seconds": 0.08657020800001192 + }, + { + "index": 143, + "status": 429, + "upload_seconds": 0.07128558300018994 + }, + { + "index": 141, + "status": 429, + "upload_seconds": 0.07956545799970627 + }, + { + "index": 144, + "status": 202, + "upload_seconds": 0.07929708399979063, + "id": "a25433e1-ad0f-4687-b526-1b3b53ccdf65", + "terminal": "ready", + "ready_seconds": 14.526648250000108 + }, + { + "index": 142, + "status": 202, + "upload_seconds": 0.09708766600033414, + "id": "d570f247-7b62-4fa4-acfe-51b8b38bcb52", + "terminal": "ready", + "ready_seconds": 14.518908541999735 + }, + { + "index": 147, + "status": 429, + "upload_seconds": 0.08362375000024258 + }, + { + "index": 145, + "status": 429, + "upload_seconds": 0.09248937499978638 + }, + { + "index": 148, + "status": 429, + "upload_seconds": 0.08598129199981486 + }, + { + "index": 146, + "status": 429, + "upload_seconds": 0.09503562500003682 + }, + { + "index": 150, + "status": 429, + "upload_seconds": 0.0858608749999803 + }, + { + "index": 149, + "status": 429, + "upload_seconds": 0.09222666599998774 + }, + { + "index": 151, + "status": 429, + "upload_seconds": 0.0878139999999803 + }, + { + "index": 152, + "status": 429, + "upload_seconds": 0.0895354999997835 + }, + { + "index": 153, + "status": 429, + "upload_seconds": 0.04900874999975713 + }, + { + "index": 154, + "status": 429, + "upload_seconds": 0.04847933299970464 + }, + { + "index": 155, + "status": 429, + "upload_seconds": 0.04700687500007916 + }, + { + "index": 156, + "status": 429, + "upload_seconds": 0.0464057919998595 + }, + { + "index": 157, + "status": 429, + "upload_seconds": 0.044527291000122204 + }, + { + "index": 158, + "status": 429, + "upload_seconds": 0.04334416700021393 + }, + { + "index": 159, + "status": 429, + "upload_seconds": 0.035500750000210246 + } + ], + "samples": [ + { + "jobs": [ + { + "id": "a84ca007-7400-48eb-8a8a-6ea4d8d10764", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2365078f-c7cd-4bbb-974b-66cc7c27a8b7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ea038f65-febd-4f5a-8629-34e33e7d3896", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ba9f5a1c-270f-4c54-80e7-363b70205259", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b03cff8e-576e-426a-9bbb-c11ba1d225bf", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0c3181b1-b01f-498a-89a6-f404f05575e6", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "40849cc4-696e-4202-8f7d-861bd692ba90", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 7, + "rss_bytes": 51228672, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 28, + "messages_unacknowledged": 2 + }, + "seconds": 3.0829907499996807 + }, + { + "jobs": [ + { + "id": "ea038f65-febd-4f5a-8629-34e33e7d3896", + "status": "ready", + "queue_wait_seconds": 0.603772 + }, + { + "id": "ba9f5a1c-270f-4c54-80e7-363b70205259", + "status": "ready", + "queue_wait_seconds": 1.535945 + }, + { + "id": "b03cff8e-576e-426a-9bbb-c11ba1d225bf", + "status": "ready", + "queue_wait_seconds": 2.2873 + }, + { + "id": "0263ce9f-8c4c-43a8-885d-0be62e08d8fd", + "status": "ready", + "queue_wait_seconds": 3.144025 + }, + { + "id": "f9a273e0-3ac2-4cc4-b4f6-648df6b93f71", + "status": "processing", + "queue_wait_seconds": 3.95698 + }, + { + "id": "1734ef1f-7449-4dc9-a238-2d895b7933f0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "23c140b7-2b0c-46fb-88ef-79b141200e09", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "7546c2c1-0322-4c7c-b0af-82c53c6bda0f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "e635f5c9-37f7-4f9a-b1e3-87fbc8b6b137", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "10834475-21f8-4eed-bbbc-851359677ec9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b11a2c-1abe-439b-a017-d539e8b6cae6", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a04e0db9-273d-454a-961c-b9f559838e8c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa4a7430-b50a-4b1d-95a5-8243e93a44dc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5f437b62-c9e4-4b54-819f-1ed5cf971451", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "770aa1ca-017d-48b3-ba75-b0c193019735", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fcb3a86-365b-42a2-b25f-8f7e7469bfa9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a25433e1-ad0f-4687-b526-1b3b53ccdf65", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0c3181b1-b01f-498a-89a6-f404f05575e6", + "status": "ready", + "queue_wait_seconds": 2.293722 + }, + { + "id": "40849cc4-696e-4202-8f7d-861bd692ba90", + "status": "ready", + "queue_wait_seconds": 3.139273 + }, + { + "id": "b023a107-e51c-4714-bdbb-1ca6346f5654", + "status": "processing", + "queue_wait_seconds": 4.011168 + }, + { + "id": "f141ec67-c5ac-4ff8-8d86-977d0bf8d2d6", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "228b4f66-8b1e-4b8a-933f-a1cd75aca8db", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a53d81be-6ca2-42a1-b5b3-f7f088501717", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f47ac0e4-1765-4a50-bd05-16fbd1cab48e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d570f247-7b62-4fa4-acfe-51b8b38bcb52", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f7792a5e-d4da-47fe-8817-eb68d08420a3", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "71a80c0b-7732-4d22-8b7b-2c2d2f5eaaf4", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ede9d41c-0ae6-4aa9-b863-48c6f97e6b06", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a84ca007-7400-48eb-8a8a-6ea4d8d10764", + "status": "ready", + "queue_wait_seconds": 0.625877 + }, + { + "id": "e34722a1-983e-4fe5-9095-d94c45578213", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c4933534-3cdf-4868-b408-4740fcf4ce49", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f021eec7-4f30-4c07-b808-3832ea84ec42", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8fe5707f-def4-4d86-9cd2-6ebfb820ba0c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2365078f-c7cd-4bbb-974b-66cc7c27a8b7", + "status": "ready", + "queue_wait_seconds": 1.560049 + } + ], + "pending": 26, + "rss_bytes": 109887488, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 22, + "messages_unacknowledged": 2 + }, + "seconds": 5.793001791999814 + }, + { + "jobs": [ + { + "id": "ea038f65-febd-4f5a-8629-34e33e7d3896", + "status": "ready", + "queue_wait_seconds": 0.603772 + }, + { + "id": "ba9f5a1c-270f-4c54-80e7-363b70205259", + "status": "ready", + "queue_wait_seconds": 1.535945 + }, + { + "id": "b03cff8e-576e-426a-9bbb-c11ba1d225bf", + "status": "ready", + "queue_wait_seconds": 2.2873 + }, + { + "id": "0263ce9f-8c4c-43a8-885d-0be62e08d8fd", + "status": "ready", + "queue_wait_seconds": 3.144025 + }, + { + "id": "f9a273e0-3ac2-4cc4-b4f6-648df6b93f71", + "status": "ready", + "queue_wait_seconds": 3.95698 + }, + { + "id": "1734ef1f-7449-4dc9-a238-2d895b7933f0", + "status": "ready", + "queue_wait_seconds": 4.862639 + }, + { + "id": "23c140b7-2b0c-46fb-88ef-79b141200e09", + "status": "ready", + "queue_wait_seconds": 5.627001 + }, + { + "id": "7546c2c1-0322-4c7c-b0af-82c53c6bda0f", + "status": "processing", + "queue_wait_seconds": 6.455358 + }, + { + "id": "e635f5c9-37f7-4f9a-b1e3-87fbc8b6b137", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "10834475-21f8-4eed-bbbc-851359677ec9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "60b11a2c-1abe-439b-a017-d539e8b6cae6", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a04e0db9-273d-454a-961c-b9f559838e8c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa4a7430-b50a-4b1d-95a5-8243e93a44dc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5f437b62-c9e4-4b54-819f-1ed5cf971451", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "770aa1ca-017d-48b3-ba75-b0c193019735", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fcb3a86-365b-42a2-b25f-8f7e7469bfa9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a25433e1-ad0f-4687-b526-1b3b53ccdf65", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0c3181b1-b01f-498a-89a6-f404f05575e6", + "status": "ready", + "queue_wait_seconds": 2.293722 + }, + { + "id": "40849cc4-696e-4202-8f7d-861bd692ba90", + "status": "ready", + "queue_wait_seconds": 3.139273 + }, + { + "id": "b023a107-e51c-4714-bdbb-1ca6346f5654", + "status": "ready", + "queue_wait_seconds": 4.011168 + }, + { + "id": "f141ec67-c5ac-4ff8-8d86-977d0bf8d2d6", + "status": "ready", + "queue_wait_seconds": 5.703673 + }, + { + "id": "228b4f66-8b1e-4b8a-933f-a1cd75aca8db", + "status": "processing", + "queue_wait_seconds": 6.46576 + }, + { + "id": "a53d81be-6ca2-42a1-b5b3-f7f088501717", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f47ac0e4-1765-4a50-bd05-16fbd1cab48e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d570f247-7b62-4fa4-acfe-51b8b38bcb52", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "71a80c0b-7732-4d22-8b7b-2c2d2f5eaaf4", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ede9d41c-0ae6-4aa9-b863-48c6f97e6b06", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a84ca007-7400-48eb-8a8a-6ea4d8d10764", + "status": "ready", + "queue_wait_seconds": 0.625877 + }, + { + "id": "e34722a1-983e-4fe5-9095-d94c45578213", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c4933534-3cdf-4868-b408-4740fcf4ce49", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f021eec7-4f30-4c07-b808-3832ea84ec42", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8fe5707f-def4-4d86-9cd2-6ebfb820ba0c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2365078f-c7cd-4bbb-974b-66cc7c27a8b7", + "status": "ready", + "queue_wait_seconds": 1.560049 + }, + { + "id": "f7792a5e-d4da-47fe-8817-eb68d08420a3", + "status": "ready", + "queue_wait_seconds": 4.867845 + } + ], + "pending": 20, + "rss_bytes": 139767808, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 16, + "messages_unacknowledged": 2 + }, + "seconds": 8.494727583999975 + }, + { + "jobs": [ + { + "id": "ea038f65-febd-4f5a-8629-34e33e7d3896", + "status": "ready", + "queue_wait_seconds": 0.603772 + }, + { + "id": "ba9f5a1c-270f-4c54-80e7-363b70205259", + "status": "ready", + "queue_wait_seconds": 1.535945 + }, + { + "id": "b03cff8e-576e-426a-9bbb-c11ba1d225bf", + "status": "ready", + "queue_wait_seconds": 2.2873 + }, + { + "id": "0263ce9f-8c4c-43a8-885d-0be62e08d8fd", + "status": "ready", + "queue_wait_seconds": 3.144025 + }, + { + "id": "f9a273e0-3ac2-4cc4-b4f6-648df6b93f71", + "status": "ready", + "queue_wait_seconds": 3.95698 + }, + { + "id": "1734ef1f-7449-4dc9-a238-2d895b7933f0", + "status": "ready", + "queue_wait_seconds": 4.862639 + }, + { + "id": "23c140b7-2b0c-46fb-88ef-79b141200e09", + "status": "ready", + "queue_wait_seconds": 5.627001 + }, + { + "id": "7546c2c1-0322-4c7c-b0af-82c53c6bda0f", + "status": "ready", + "queue_wait_seconds": 6.455358 + }, + { + "id": "e635f5c9-37f7-4f9a-b1e3-87fbc8b6b137", + "status": "ready", + "queue_wait_seconds": 7.368448 + }, + { + "id": "10834475-21f8-4eed-bbbc-851359677ec9", + "status": "ready", + "queue_wait_seconds": 8.163975 + }, + { + "id": "60b11a2c-1abe-439b-a017-d539e8b6cae6", + "status": "processing", + "queue_wait_seconds": 8.952436 + }, + { + "id": "a04e0db9-273d-454a-961c-b9f559838e8c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "aa4a7430-b50a-4b1d-95a5-8243e93a44dc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5f437b62-c9e4-4b54-819f-1ed5cf971451", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "770aa1ca-017d-48b3-ba75-b0c193019735", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fcb3a86-365b-42a2-b25f-8f7e7469bfa9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a25433e1-ad0f-4687-b526-1b3b53ccdf65", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0c3181b1-b01f-498a-89a6-f404f05575e6", + "status": "ready", + "queue_wait_seconds": 2.293722 + }, + { + "id": "40849cc4-696e-4202-8f7d-861bd692ba90", + "status": "ready", + "queue_wait_seconds": 3.139273 + }, + { + "id": "b023a107-e51c-4714-bdbb-1ca6346f5654", + "status": "ready", + "queue_wait_seconds": 4.011168 + }, + { + "id": "f141ec67-c5ac-4ff8-8d86-977d0bf8d2d6", + "status": "ready", + "queue_wait_seconds": 5.703673 + }, + { + "id": "228b4f66-8b1e-4b8a-933f-a1cd75aca8db", + "status": "ready", + "queue_wait_seconds": 6.46576 + }, + { + "id": "a53d81be-6ca2-42a1-b5b3-f7f088501717", + "status": "ready", + "queue_wait_seconds": 7.372371 + }, + { + "id": "f47ac0e4-1765-4a50-bd05-16fbd1cab48e", + "status": "ready", + "queue_wait_seconds": 8.165016 + }, + { + "id": "d570f247-7b62-4fa4-acfe-51b8b38bcb52", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "71a80c0b-7732-4d22-8b7b-2c2d2f5eaaf4", + "status": "processing", + "queue_wait_seconds": 8.963767 + }, + { + "id": "ede9d41c-0ae6-4aa9-b863-48c6f97e6b06", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a84ca007-7400-48eb-8a8a-6ea4d8d10764", + "status": "ready", + "queue_wait_seconds": 0.625877 + }, + { + "id": "e34722a1-983e-4fe5-9095-d94c45578213", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c4933534-3cdf-4868-b408-4740fcf4ce49", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f021eec7-4f30-4c07-b808-3832ea84ec42", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8fe5707f-def4-4d86-9cd2-6ebfb820ba0c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2365078f-c7cd-4bbb-974b-66cc7c27a8b7", + "status": "ready", + "queue_wait_seconds": 1.560049 + }, + { + "id": "f7792a5e-d4da-47fe-8817-eb68d08420a3", + "status": "ready", + "queue_wait_seconds": 4.867845 + } + ], + "pending": 14, + "rss_bytes": 151478272, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 10, + "messages_unacknowledged": 2 + }, + "seconds": 11.27948699999979 + }, + { + "jobs": [ + { + "id": "ea038f65-febd-4f5a-8629-34e33e7d3896", + "status": "ready", + "queue_wait_seconds": 0.603772 + }, + { + "id": "ba9f5a1c-270f-4c54-80e7-363b70205259", + "status": "ready", + "queue_wait_seconds": 1.535945 + }, + { + "id": "b03cff8e-576e-426a-9bbb-c11ba1d225bf", + "status": "ready", + "queue_wait_seconds": 2.2873 + }, + { + "id": "0263ce9f-8c4c-43a8-885d-0be62e08d8fd", + "status": "ready", + "queue_wait_seconds": 3.144025 + }, + { + "id": "f9a273e0-3ac2-4cc4-b4f6-648df6b93f71", + "status": "ready", + "queue_wait_seconds": 3.95698 + }, + { + "id": "1734ef1f-7449-4dc9-a238-2d895b7933f0", + "status": "ready", + "queue_wait_seconds": 4.862639 + }, + { + "id": "23c140b7-2b0c-46fb-88ef-79b141200e09", + "status": "ready", + "queue_wait_seconds": 5.627001 + }, + { + "id": "7546c2c1-0322-4c7c-b0af-82c53c6bda0f", + "status": "ready", + "queue_wait_seconds": 6.455358 + }, + { + "id": "e635f5c9-37f7-4f9a-b1e3-87fbc8b6b137", + "status": "ready", + "queue_wait_seconds": 7.368448 + }, + { + "id": "10834475-21f8-4eed-bbbc-851359677ec9", + "status": "ready", + "queue_wait_seconds": 8.163975 + }, + { + "id": "60b11a2c-1abe-439b-a017-d539e8b6cae6", + "status": "ready", + "queue_wait_seconds": 8.952436 + }, + { + "id": "a04e0db9-273d-454a-961c-b9f559838e8c", + "status": "ready", + "queue_wait_seconds": 9.849386 + }, + { + "id": "aa4a7430-b50a-4b1d-95a5-8243e93a44dc", + "status": "ready", + "queue_wait_seconds": 10.717404 + }, + { + "id": "5f437b62-c9e4-4b54-819f-1ed5cf971451", + "status": "processing", + "queue_wait_seconds": 11.475121 + }, + { + "id": "770aa1ca-017d-48b3-ba75-b0c193019735", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fcb3a86-365b-42a2-b25f-8f7e7469bfa9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a25433e1-ad0f-4687-b526-1b3b53ccdf65", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0c3181b1-b01f-498a-89a6-f404f05575e6", + "status": "ready", + "queue_wait_seconds": 2.293722 + }, + { + "id": "40849cc4-696e-4202-8f7d-861bd692ba90", + "status": "ready", + "queue_wait_seconds": 3.139273 + }, + { + "id": "b023a107-e51c-4714-bdbb-1ca6346f5654", + "status": "ready", + "queue_wait_seconds": 4.011168 + }, + { + "id": "f141ec67-c5ac-4ff8-8d86-977d0bf8d2d6", + "status": "ready", + "queue_wait_seconds": 5.703673 + }, + { + "id": "228b4f66-8b1e-4b8a-933f-a1cd75aca8db", + "status": "ready", + "queue_wait_seconds": 6.46576 + }, + { + "id": "a53d81be-6ca2-42a1-b5b3-f7f088501717", + "status": "ready", + "queue_wait_seconds": 7.372371 + }, + { + "id": "f47ac0e4-1765-4a50-bd05-16fbd1cab48e", + "status": "ready", + "queue_wait_seconds": 8.165016 + }, + { + "id": "d570f247-7b62-4fa4-acfe-51b8b38bcb52", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a84ca007-7400-48eb-8a8a-6ea4d8d10764", + "status": "ready", + "queue_wait_seconds": 0.625877 + }, + { + "id": "c4933534-3cdf-4868-b408-4740fcf4ce49", + "status": "processing", + "queue_wait_seconds": 11.484755 + }, + { + "id": "f021eec7-4f30-4c07-b808-3832ea84ec42", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8fe5707f-def4-4d86-9cd2-6ebfb820ba0c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2365078f-c7cd-4bbb-974b-66cc7c27a8b7", + "status": "ready", + "queue_wait_seconds": 1.560049 + }, + { + "id": "f7792a5e-d4da-47fe-8817-eb68d08420a3", + "status": "ready", + "queue_wait_seconds": 4.867845 + }, + { + "id": "71a80c0b-7732-4d22-8b7b-2c2d2f5eaaf4", + "status": "ready", + "queue_wait_seconds": 8.963767 + }, + { + "id": "ede9d41c-0ae6-4aa9-b863-48c6f97e6b06", + "status": "ready", + "queue_wait_seconds": 9.859237 + }, + { + "id": "e34722a1-983e-4fe5-9095-d94c45578213", + "status": "ready", + "queue_wait_seconds": 10.701347 + } + ], + "pending": 8, + "rss_bytes": 165621760, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 4, + "messages_unacknowledged": 2 + }, + "seconds": 13.96383650000007 + }, + { + "jobs": [ + { + "id": "8fe5707f-def4-4d86-9cd2-6ebfb820ba0c", + "status": "ready", + "queue_wait_seconds": 13.522461 + }, + { + "id": "ea038f65-febd-4f5a-8629-34e33e7d3896", + "status": "ready", + "queue_wait_seconds": 0.603772 + }, + { + "id": "ba9f5a1c-270f-4c54-80e7-363b70205259", + "status": "ready", + "queue_wait_seconds": 1.535945 + }, + { + "id": "b03cff8e-576e-426a-9bbb-c11ba1d225bf", + "status": "ready", + "queue_wait_seconds": 2.2873 + }, + { + "id": "0263ce9f-8c4c-43a8-885d-0be62e08d8fd", + "status": "ready", + "queue_wait_seconds": 3.144025 + }, + { + "id": "f9a273e0-3ac2-4cc4-b4f6-648df6b93f71", + "status": "ready", + "queue_wait_seconds": 3.95698 + }, + { + "id": "1734ef1f-7449-4dc9-a238-2d895b7933f0", + "status": "ready", + "queue_wait_seconds": 4.862639 + }, + { + "id": "23c140b7-2b0c-46fb-88ef-79b141200e09", + "status": "ready", + "queue_wait_seconds": 5.627001 + }, + { + "id": "7546c2c1-0322-4c7c-b0af-82c53c6bda0f", + "status": "ready", + "queue_wait_seconds": 6.455358 + }, + { + "id": "e635f5c9-37f7-4f9a-b1e3-87fbc8b6b137", + "status": "ready", + "queue_wait_seconds": 7.368448 + }, + { + "id": "10834475-21f8-4eed-bbbc-851359677ec9", + "status": "ready", + "queue_wait_seconds": 8.163975 + }, + { + "id": "60b11a2c-1abe-439b-a017-d539e8b6cae6", + "status": "ready", + "queue_wait_seconds": 8.952436 + }, + { + "id": "a04e0db9-273d-454a-961c-b9f559838e8c", + "status": "ready", + "queue_wait_seconds": 9.849386 + }, + { + "id": "aa4a7430-b50a-4b1d-95a5-8243e93a44dc", + "status": "ready", + "queue_wait_seconds": 10.717404 + }, + { + "id": "5f437b62-c9e4-4b54-819f-1ed5cf971451", + "status": "ready", + "queue_wait_seconds": 11.475121 + }, + { + "id": "770aa1ca-017d-48b3-ba75-b0c193019735", + "status": "ready", + "queue_wait_seconds": 12.741935 + }, + { + "id": "0fcb3a86-365b-42a2-b25f-8f7e7469bfa9", + "status": "ready", + "queue_wait_seconds": 13.534135 + }, + { + "id": "a25433e1-ad0f-4687-b526-1b3b53ccdf65", + "status": "processing", + "queue_wait_seconds": 13.572687 + }, + { + "id": "0c3181b1-b01f-498a-89a6-f404f05575e6", + "status": "ready", + "queue_wait_seconds": 2.293722 + }, + { + "id": "40849cc4-696e-4202-8f7d-861bd692ba90", + "status": "ready", + "queue_wait_seconds": 3.139273 + }, + { + "id": "b023a107-e51c-4714-bdbb-1ca6346f5654", + "status": "ready", + "queue_wait_seconds": 4.011168 + }, + { + "id": "f141ec67-c5ac-4ff8-8d86-977d0bf8d2d6", + "status": "ready", + "queue_wait_seconds": 5.703673 + }, + { + "id": "228b4f66-8b1e-4b8a-933f-a1cd75aca8db", + "status": "ready", + "queue_wait_seconds": 6.46576 + }, + { + "id": "a53d81be-6ca2-42a1-b5b3-f7f088501717", + "status": "ready", + "queue_wait_seconds": 7.372371 + }, + { + "id": "f47ac0e4-1765-4a50-bd05-16fbd1cab48e", + "status": "ready", + "queue_wait_seconds": 8.165016 + }, + { + "id": "d570f247-7b62-4fa4-acfe-51b8b38bcb52", + "status": "processing", + "queue_wait_seconds": 13.571623 + }, + { + "id": "a84ca007-7400-48eb-8a8a-6ea4d8d10764", + "status": "ready", + "queue_wait_seconds": 0.625877 + }, + { + "id": "2365078f-c7cd-4bbb-974b-66cc7c27a8b7", + "status": "ready", + "queue_wait_seconds": 1.560049 + }, + { + "id": "f7792a5e-d4da-47fe-8817-eb68d08420a3", + "status": "ready", + "queue_wait_seconds": 4.867845 + }, + { + "id": "71a80c0b-7732-4d22-8b7b-2c2d2f5eaaf4", + "status": "ready", + "queue_wait_seconds": 8.963767 + }, + { + "id": "ede9d41c-0ae6-4aa9-b863-48c6f97e6b06", + "status": "ready", + "queue_wait_seconds": 9.859237 + }, + { + "id": "e34722a1-983e-4fe5-9095-d94c45578213", + "status": "ready", + "queue_wait_seconds": 10.701347 + }, + { + "id": "c4933534-3cdf-4868-b408-4740fcf4ce49", + "status": "ready", + "queue_wait_seconds": 11.484755 + }, + { + "id": "f021eec7-4f30-4c07-b808-3832ea84ec42", + "status": "ready", + "queue_wait_seconds": 12.748428 + } + ], + "pending": 2, + "rss_bytes": 112926720, + "concurrency": 2, + "prefetch": 2, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 0, + "messages_unacknowledged": 0 + }, + "seconds": 17.06889337499979 + } + ], + "metric_errors": [], + "limitations": "Closed-loop; ready time is polling upper bound. RSS is sum of worker/container processes, shared pages may be counted twice. Sampling may miss peaks. Queue wait is first claim minus durable job creation." +} diff --git a/evidence/real-c4-p4-s2400.json b/evidence/real-c4-p4-s2400.json new file mode 100644 index 00000000..3391a7e4 --- /dev/null +++ b/evidence/real-c4-p4-s2400.json @@ -0,0 +1,2476 @@ +{ + "run_name": "real-c4-p4-s2400", + "settings": { + "workers": 4, + "prefetch": 4, + "max_pending": 32, + "worker_limits": { + "Memory": 4294967296, + "NanoCpus": 2000000000 + } + }, + "workload": { + "images": 160, + "concurrency": 16, + "size": 2400, + "bytes": 22196, + "type": "closed-loop burst" + }, + "results": { + "accepted": 40, + "rejected_429": 120, + "ready": 40, + "failed": 0, + "timed_out": 0, + "elapsed_seconds": 24.30633354199972, + "ready_per_second": 1.6456616104145314, + "ready_latency_ms": { + "0.5": 12408.961, + "0.95": 18507.818, + "0.99": 18790.189 + }, + "queue_wait_ms": { + "0.5": 10312.805, + "0.95": 16166.105, + "0.99": 16624.492 + }, + "max_sampled_rss_bytes": 543588352, + "max_sampled_pending": 32, + "sample_count": 8, + "drained": true, + "recovery_upload_ready": true + }, + "records": [ + { + "index": 0, + "status": 202, + "upload_seconds": 0.05747733399994104, + "id": "666f0462-8bde-499c-943a-d1a9a52ebee0", + "terminal": "ready", + "ready_seconds": 4.027561207999952 + }, + { + "index": 1, + "status": 202, + "upload_seconds": 0.06882850000010876, + "id": "51fd30af-2c11-4fc0-8df7-bed99c51d0b3", + "terminal": "ready", + "ready_seconds": 4.025526583000101 + }, + { + "index": 5, + "status": 202, + "upload_seconds": 0.15581120800015924, + "id": "61eeb2df-ae10-4683-975d-1988ff0bdfd7", + "terminal": "ready", + "ready_seconds": 4.11275012499982 + }, + { + "index": 3, + "status": 202, + "upload_seconds": 0.1640389590002087, + "id": "24a8e5b5-48a3-44b7-bc83-af8218c9ac86", + "terminal": "ready", + "ready_seconds": 4.116358582999965 + }, + { + "index": 6, + "status": 202, + "upload_seconds": 0.18950695800003814, + "id": "c60fcbbf-afaa-4bb6-8ab0-a35b01058a27", + "terminal": "ready", + "ready_seconds": 6.020101500000237 + }, + { + "index": 8, + "status": 202, + "upload_seconds": 0.2529424590002236, + "id": "9000f960-8632-431e-81f7-81e7c6e14b10", + "terminal": "ready", + "ready_seconds": 5.9591944999997395 + }, + { + "index": 2, + "status": 202, + "upload_seconds": 0.26110300000027564, + "id": "c0b2efd1-b1df-457b-8bf5-ea42ea43c6b1", + "terminal": "ready", + "ready_seconds": 5.956588874999852 + }, + { + "index": 7, + "status": 202, + "upload_seconds": 0.2982125419998738, + "id": "dc5a345a-2890-4221-9a12-666f17a97528", + "terminal": "ready", + "ready_seconds": 5.920130874999813 + }, + { + "index": 10, + "status": 202, + "upload_seconds": 0.3556584590000966, + "id": "eb974799-6d8d-4e0e-8083-cf06cbadf551", + "terminal": "ready", + "ready_seconds": 7.708901458000128 + }, + { + "index": 13, + "status": 202, + "upload_seconds": 0.37174533400002474, + "id": "ba5f12d9-4be2-4c84-b4d2-94d1601748ef", + "terminal": "ready", + "ready_seconds": 7.944927333000123 + }, + { + "index": 4, + "status": 202, + "upload_seconds": 0.4525333330002468, + "id": "b05bbb3a-2aef-419b-b0dd-525698eba0f1", + "terminal": "ready", + "ready_seconds": 8.070154583999738 + }, + { + "index": 12, + "status": 202, + "upload_seconds": 0.45617270899992945, + "id": "a45073f1-59bf-47b0-bb00-1cdc4d5a97cc", + "terminal": "ready", + "ready_seconds": 8.273260832999767 + }, + { + "index": 9, + "status": 202, + "upload_seconds": 0.4885836670000572, + "id": "b44c6091-c4a4-413f-8d16-2b50adadd4d2", + "terminal": "ready", + "ready_seconds": 9.82335749999993 + }, + { + "index": 11, + "status": 202, + "upload_seconds": 0.49368862499977695, + "id": "2e4fea6b-7a05-47df-b433-a4a413d399e0", + "terminal": "ready", + "ready_seconds": 9.823378875000344 + }, + { + "index": 16, + "status": 202, + "upload_seconds": 0.5221212500000547, + "id": "6f95f22c-940b-4190-8b1d-aa7608985848", + "terminal": "ready", + "ready_seconds": 10.483770165999886 + }, + { + "index": 15, + "status": 202, + "upload_seconds": 0.5865937910002685, + "id": "6f56f236-1859-41b9-acd6-8cbf7268ff92", + "terminal": "ready", + "ready_seconds": 10.4761267089998 + }, + { + "index": 18, + "status": 202, + "upload_seconds": 0.5029819169999428, + "id": "4be0e3c5-3444-4a99-90cc-5fde4336a43c", + "terminal": "ready", + "ready_seconds": 11.896967000000132 + }, + { + "index": 14, + "status": 202, + "upload_seconds": 0.6613184580000961, + "id": "1e00cefc-ce86-42e0-b17f-97a7166f17b2", + "terminal": "ready", + "ready_seconds": 11.892707125000015 + }, + { + "index": 20, + "status": 202, + "upload_seconds": 0.49663941700009673, + "id": "9bba8073-6595-48ac-95d1-411f47eb06d8", + "terminal": "ready", + "ready_seconds": 12.40871908300005 + }, + { + "index": 17, + "status": 202, + "upload_seconds": 0.6260004589998971, + "id": "72791978-e968-44e0-accd-45a1dc2f3644", + "terminal": "ready", + "ready_seconds": 12.408960582999953 + }, + { + "index": 21, + "status": 202, + "upload_seconds": 0.5179503750000549, + "id": "13c0f2dd-67f8-452e-9ae1-7032cd1c2fd2", + "terminal": "ready", + "ready_seconds": 14.216128291000132 + }, + { + "index": 19, + "status": 202, + "upload_seconds": 0.6195237500000985, + "id": "4ce61788-027d-4109-a333-cb23cb33d305", + "terminal": "ready", + "ready_seconds": 14.215918208999938 + }, + { + "index": 23, + "status": 202, + "upload_seconds": 0.557807916999991, + "id": "77349e6b-f4bf-4c87-bbd6-e63c1d1f5dbf", + "terminal": "ready", + "ready_seconds": 14.636029416999918 + }, + { + "index": 22, + "status": 202, + "upload_seconds": 0.6023047919998135, + "id": "f99ad64e-7ddc-4f6c-8826-36edbf5d999c", + "terminal": "ready", + "ready_seconds": 14.635965082999974 + }, + { + "index": 24, + "status": 202, + "upload_seconds": 0.5213137499999903, + "id": "44917eba-369e-4f00-b14e-d90a72996a59", + "terminal": "ready", + "ready_seconds": 15.926922000000104 + }, + { + "index": 26, + "status": 202, + "upload_seconds": 0.4523000829999546, + "id": "fb7d9ddd-81ff-4659-a9bf-80dcbe1b2692", + "terminal": "ready", + "ready_seconds": 16.05335370800003 + }, + { + "index": 25, + "status": 202, + "upload_seconds": 0.5877269999996315, + "id": "9e88ac8c-209a-4dac-9fcd-424820b5847c", + "terminal": "ready", + "ready_seconds": 16.769380958000056 + }, + { + "index": 28, + "status": 202, + "upload_seconds": 0.4847945000001346, + "id": "9e232507-1b4d-4ea9-94de-626c6e28eeda", + "terminal": "ready", + "ready_seconds": 16.597695250000015 + }, + { + "index": 27, + "status": 202, + "upload_seconds": 0.5340564999996786, + "id": "a2608bb2-8a02-4996-ac8f-c4cc9f412e0d", + "terminal": "ready", + "ready_seconds": 18.3290873750002 + }, + { + "index": 29, + "status": 202, + "upload_seconds": 0.5558498339996731, + "id": "30d1137f-2ca7-4442-8e82-e16867c24155", + "terminal": "ready", + "ready_seconds": 18.275010208000367 + }, + { + "index": 30, + "status": 202, + "upload_seconds": 0.4988747500001409, + "id": "c2265ee3-ee0b-4983-8296-ca2ac0e82e96", + "terminal": "ready", + "ready_seconds": 18.790189290999933 + }, + { + "index": 33, + "status": 202, + "upload_seconds": 0.41763499999979103, + "id": "40fe12b2-f03d-4263-b3e0-07716dffc53e", + "terminal": "ready", + "ready_seconds": 18.788450208000086 + }, + { + "index": 31, + "status": 429, + "upload_seconds": 0.5176506660000086 + }, + { + "index": 35, + "status": 429, + "upload_seconds": 0.4721334579999166 + }, + { + "index": 32, + "status": 429, + "upload_seconds": 0.5214307920000465 + }, + { + "index": 37, + "status": 429, + "upload_seconds": 0.4064060829996379 + }, + { + "index": 39, + "status": 429, + "upload_seconds": 0.34858966700039673 + }, + { + "index": 34, + "status": 429, + "upload_seconds": 0.582176583999626 + }, + { + "index": 36, + "status": 429, + "upload_seconds": 0.5904807499996423 + }, + { + "index": 41, + "status": 429, + "upload_seconds": 0.46556779200000165 + }, + { + "index": 43, + "status": 429, + "upload_seconds": 0.4251493329998084 + }, + { + "index": 38, + "status": 429, + "upload_seconds": 0.6050344579998637 + }, + { + "index": 44, + "status": 429, + "upload_seconds": 0.4964398750003056 + }, + { + "index": 40, + "status": 429, + "upload_seconds": 0.6105557920000138 + }, + { + "index": 42, + "status": 429, + "upload_seconds": 0.6072146660003455 + }, + { + "index": 45, + "status": 429, + "upload_seconds": 0.5414415830000507 + }, + { + "index": 46, + "status": 429, + "upload_seconds": 0.5269656249997752 + }, + { + "index": 47, + "status": 429, + "upload_seconds": 0.598630625000169 + }, + { + "index": 48, + "status": 429, + "upload_seconds": 0.5747869159999937 + }, + { + "index": 49, + "status": 429, + "upload_seconds": 0.5970194169999559 + }, + { + "index": 50, + "status": 429, + "upload_seconds": 0.5949379159997079 + }, + { + "index": 53, + "status": 429, + "upload_seconds": 0.5328158329998587 + }, + { + "index": 51, + "status": 429, + "upload_seconds": 0.6770308339996518 + }, + { + "index": 54, + "status": 429, + "upload_seconds": 0.5337664590001623 + }, + { + "index": 52, + "status": 429, + "upload_seconds": 0.759974541000247 + }, + { + "index": 55, + "status": 429, + "upload_seconds": 0.603516333999778 + }, + { + "index": 57, + "status": 429, + "upload_seconds": 0.596160915999917 + }, + { + "index": 56, + "status": 429, + "upload_seconds": 0.6719817079997483 + }, + { + "index": 58, + "status": 429, + "upload_seconds": 0.6022002499998962 + }, + { + "index": 59, + "status": 429, + "upload_seconds": 0.6092106249998324 + }, + { + "index": 61, + "status": 429, + "upload_seconds": 0.5854576669999005 + }, + { + "index": 60, + "status": 429, + "upload_seconds": 0.6275084579997383 + }, + { + "index": 62, + "status": 429, + "upload_seconds": 0.6581309170001077 + }, + { + "index": 63, + "status": 429, + "upload_seconds": 0.6113555839997389 + }, + { + "index": 64, + "status": 429, + "upload_seconds": 0.6738223749998724 + }, + { + "index": 66, + "status": 429, + "upload_seconds": 0.6103316250000717 + }, + { + "index": 65, + "status": 429, + "upload_seconds": 0.6263138340000296 + }, + { + "index": 68, + "status": 429, + "upload_seconds": 0.6027756250000493 + }, + { + "index": 67, + "status": 429, + "upload_seconds": 0.6788702920002834 + }, + { + "index": 69, + "status": 429, + "upload_seconds": 0.6650337920000311 + }, + { + "index": 70, + "status": 429, + "upload_seconds": 0.6099658749999435 + }, + { + "index": 72, + "status": 429, + "upload_seconds": 0.6034837910001443 + }, + { + "index": 71, + "status": 429, + "upload_seconds": 0.7030028340000172 + }, + { + "index": 75, + "status": 429, + "upload_seconds": 0.672948500000075 + }, + { + "index": 73, + "status": 429, + "upload_seconds": 0.7059914169999502 + }, + { + "index": 74, + "status": 429, + "upload_seconds": 0.7788630000000012 + }, + { + "index": 76, + "status": 429, + "upload_seconds": 0.6945285000001604 + }, + { + "index": 77, + "status": 429, + "upload_seconds": 0.7148655830001189 + }, + { + "index": 79, + "status": 429, + "upload_seconds": 0.6818583330000365 + }, + { + "index": 78, + "status": 429, + "upload_seconds": 0.7229228330002115 + }, + { + "index": 80, + "status": 429, + "upload_seconds": 0.7159322090001297 + }, + { + "index": 81, + "status": 429, + "upload_seconds": 0.6935698749998664 + }, + { + "index": 84, + "status": 429, + "upload_seconds": 0.6196588750003684 + }, + { + "index": 82, + "status": 429, + "upload_seconds": 0.7715660829999251 + }, + { + "index": 87, + "status": 429, + "upload_seconds": 0.530849666999984 + }, + { + "index": 83, + "status": 429, + "upload_seconds": 0.7255615829999442 + }, + { + "index": 85, + "status": 429, + "upload_seconds": 0.7060491670004012 + }, + { + "index": 88, + "status": 429, + "upload_seconds": 0.5979354169999169 + }, + { + "index": 86, + "status": 429, + "upload_seconds": 0.7127409999998235 + }, + { + "index": 89, + "status": 429, + "upload_seconds": 0.5993293750002522 + }, + { + "index": 90, + "status": 429, + "upload_seconds": 0.5942680829998608 + }, + { + "index": 92, + "status": 429, + "upload_seconds": 0.5355847920000087 + }, + { + "index": 91, + "status": 429, + "upload_seconds": 0.5960489579997557 + }, + { + "index": 93, + "status": 429, + "upload_seconds": 0.5792056249997586 + }, + { + "index": 97, + "status": 202, + "upload_seconds": 0.5776968750001288, + "id": "37484fa0-a23d-4ee9-b688-f9cf59e727a2", + "terminal": "ready", + "ready_seconds": 17.868646291999994 + }, + { + "index": 94, + "status": 202, + "upload_seconds": 0.6988310830001865, + "id": "2aa39587-9a7b-4adf-be73-70174684c6f9", + "terminal": "ready", + "ready_seconds": 17.854372082999816 + }, + { + "index": 95, + "status": 202, + "upload_seconds": 0.7703467500000443, + "id": "92bda7ec-45b2-4733-9931-a605b1e43842", + "terminal": "ready", + "ready_seconds": 18.507818290999694 + }, + { + "index": 99, + "status": 202, + "upload_seconds": 0.6099736669998492, + "id": "f0d760ed-6d3e-45ea-8854-5acec6e66ba0", + "terminal": "ready", + "ready_seconds": 18.377560791000178 + }, + { + "index": 96, + "status": 429, + "upload_seconds": 0.7103495000001203 + }, + { + "index": 98, + "status": 429, + "upload_seconds": 0.7070418339999378 + }, + { + "index": 100, + "status": 429, + "upload_seconds": 0.7121545830000287 + }, + { + "index": 102, + "status": 429, + "upload_seconds": 0.6871527080002124 + }, + { + "index": 105, + "status": 429, + "upload_seconds": 0.6935586249996959 + }, + { + "index": 101, + "status": 429, + "upload_seconds": 0.8776165829999627 + }, + { + "index": 110, + "status": 429, + "upload_seconds": 0.5006673749999209 + }, + { + "index": 111, + "status": 429, + "upload_seconds": 0.590699042000324 + }, + { + "index": 103, + "status": 429, + "upload_seconds": 1.0026990830001523 + }, + { + "index": 113, + "status": 429, + "upload_seconds": 0.5075017079998361 + }, + { + "index": 114, + "status": 429, + "upload_seconds": 0.5838066669998625 + }, + { + "index": 118, + "status": 429, + "upload_seconds": 0.33380558299995755 + }, + { + "index": 120, + "status": 429, + "upload_seconds": 0.3212936249997256 + }, + { + "index": 104, + "status": 429, + "upload_seconds": 1.2680248750002647 + }, + { + "index": 122, + "status": 429, + "upload_seconds": 0.3054895420000321 + }, + { + "index": 124, + "status": 429, + "upload_seconds": 0.3021243749999485 + }, + { + "index": 106, + "status": 429, + "upload_seconds": 1.3005834999999024 + }, + { + "index": 125, + "status": 429, + "upload_seconds": 0.35967683399985617 + }, + { + "index": 107, + "status": 429, + "upload_seconds": 1.383920875000058 + }, + { + "index": 127, + "status": 429, + "upload_seconds": 0.29778499999974883 + }, + { + "index": 108, + "status": 429, + "upload_seconds": 1.421576499999901 + }, + { + "index": 130, + "status": 429, + "upload_seconds": 0.2904971250000017 + }, + { + "index": 109, + "status": 429, + "upload_seconds": 1.4880274160000226 + }, + { + "index": 131, + "status": 429, + "upload_seconds": 0.29417866699986917 + }, + { + "index": 133, + "status": 429, + "upload_seconds": 0.22435616600023423 + }, + { + "index": 112, + "status": 429, + "upload_seconds": 1.331625250000343 + }, + { + "index": 134, + "status": 429, + "upload_seconds": 0.2861254590002318 + }, + { + "index": 115, + "status": 429, + "upload_seconds": 1.3702650830000493 + }, + { + "index": 136, + "status": 429, + "upload_seconds": 0.2834125839999615 + }, + { + "index": 117, + "status": 429, + "upload_seconds": 1.321286332999989 + }, + { + "index": 137, + "status": 429, + "upload_seconds": 0.3054844170001161 + }, + { + "index": 116, + "status": 429, + "upload_seconds": 1.4806589580002765 + }, + { + "index": 139, + "status": 429, + "upload_seconds": 0.3768667090002964 + }, + { + "index": 119, + "status": 429, + "upload_seconds": 1.4008834160003971 + }, + { + "index": 140, + "status": 429, + "upload_seconds": 0.3107364580000649 + }, + { + "index": 143, + "status": 429, + "upload_seconds": 0.28850087499995425 + }, + { + "index": 121, + "status": 429, + "upload_seconds": 1.3158427910002501 + }, + { + "index": 145, + "status": 429, + "upload_seconds": 0.28050970899994354 + }, + { + "index": 123, + "status": 429, + "upload_seconds": 1.4152941669999564 + }, + { + "index": 147, + "status": 429, + "upload_seconds": 0.22654599999987113 + }, + { + "index": 126, + "status": 202, + "upload_seconds": 1.3709077500002422, + "id": "95741924-c00b-4a1c-ae77-eeeefc56eb0f", + "terminal": "ready", + "ready_seconds": 17.854421042000013 + }, + { + "index": 150, + "status": 429, + "upload_seconds": 0.27064129100017453 + }, + { + "index": 128, + "status": 202, + "upload_seconds": 1.3654873329996917, + "id": "abf827fa-1e39-48a2-99d3-1e357b309cef", + "terminal": "ready", + "ready_seconds": 17.872440375000224 + }, + { + "index": 151, + "status": 429, + "upload_seconds": 0.2851489589997982 + }, + { + "index": 129, + "status": 429, + "upload_seconds": 1.305922499999724 + }, + { + "index": 155, + "status": 429, + "upload_seconds": 0.12406854200025919 + }, + { + "index": 158, + "status": 202, + "upload_seconds": 0.10026170799983447, + "id": "34edf97f-f853-404a-9d16-79951cd28cac", + "terminal": "ready", + "ready_seconds": 18.188908874999925 + }, + { + "index": 132, + "status": 429, + "upload_seconds": 1.2856076660000326 + }, + { + "index": 135, + "status": 202, + "upload_seconds": 1.2286266249998334, + "id": "5ef0393b-c667-4b1a-938c-4ce6623b2fbb", + "terminal": "ready", + "ready_seconds": 18.0753997920001 + }, + { + "index": 138, + "status": 429, + "upload_seconds": 1.1866848750000827 + }, + { + "index": 141, + "status": 429, + "upload_seconds": 1.1998106249998273 + }, + { + "index": 142, + "status": 429, + "upload_seconds": 1.210984209000344 + }, + { + "index": 144, + "status": 429, + "upload_seconds": 1.1988117500000044 + }, + { + "index": 146, + "status": 429, + "upload_seconds": 1.1316604590001589 + }, + { + "index": 148, + "status": 429, + "upload_seconds": 1.1997857500000464 + }, + { + "index": 149, + "status": 429, + "upload_seconds": 1.2118705829998362 + }, + { + "index": 152, + "status": 429, + "upload_seconds": 1.1804036250000536 + }, + { + "index": 153, + "status": 429, + "upload_seconds": 1.2026392500001748 + }, + { + "index": 154, + "status": 429, + "upload_seconds": 1.14636633300006 + }, + { + "index": 156, + "status": 429, + "upload_seconds": 1.1298792500001582 + }, + { + "index": 157, + "status": 429, + "upload_seconds": 1.2229275419999794 + }, + { + "index": 159, + "status": 429, + "upload_seconds": 1.2867647500002022 + } + ], + "samples": [ + { + "jobs": [ + { + "id": "51fd30af-2c11-4fc0-8df7-bed99c51d0b3", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "24a8e5b5-48a3-44b7-bc83-af8218c9ac86", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9000f960-8632-431e-81f7-81e7c6e14b10", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c60fcbbf-afaa-4bb6-8ab0-a35b01058a27", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0b2efd1-b1df-457b-8bf5-ea42ea43c6b1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "666f0462-8bde-499c-943a-d1a9a52ebee0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "61eeb2df-ae10-4683-975d-1988ff0bdfd7", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 7, + "rss_bytes": 99078144, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 16, + "messages_unacknowledged": 4 + }, + "seconds": 1.6153807919999963 + }, + { + "jobs": [ + { + "id": "51fd30af-2c11-4fc0-8df7-bed99c51d0b3", + "status": "ready", + "queue_wait_seconds": 1.037055 + }, + { + "id": "40fe12b2-f03d-4263-b3e0-07716dffc53e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f99ad64e-7ddc-4f6c-8826-36edbf5d999c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "24a8e5b5-48a3-44b7-bc83-af8218c9ac86", + "status": "processing", + "queue_wait_seconds": 1.063697 + }, + { + "id": "9000f960-8632-431e-81f7-81e7c6e14b10", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "eb974799-6d8d-4e0e-8083-cf06cbadf551", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b05bbb3a-2aef-419b-b0dd-525698eba0f1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b44c6091-c4a4-413f-8d16-2b50adadd4d2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4ce61788-027d-4109-a333-cb23cb33d305", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "fb7d9ddd-81ff-4659-a9bf-80dcbe1b2692", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9e232507-1b4d-4ea9-94de-626c6e28eeda", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "30d1137f-2ca7-4442-8e82-e16867c24155", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c60fcbbf-afaa-4bb6-8ab0-a35b01058a27", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0b2efd1-b1df-457b-8bf5-ea42ea43c6b1", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "dc5a345a-2890-4221-9a12-666f17a97528", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ba5f12d9-4be2-4c84-b4d2-94d1601748ef", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "61eeb2df-ae10-4683-975d-1988ff0bdfd7", + "status": "processing", + "queue_wait_seconds": 1.071236 + }, + { + "id": "a45073f1-59bf-47b0-bb00-1cdc4d5a97cc", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2e4fea6b-7a05-47df-b433-a4a413d399e0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6f56f236-1859-41b9-acd6-8cbf7268ff92", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1e00cefc-ce86-42e0-b17f-97a7166f17b2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "72791978-e968-44e0-accd-45a1dc2f3644", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "666f0462-8bde-499c-943a-d1a9a52ebee0", + "status": "ready", + "queue_wait_seconds": 1.02112 + }, + { + "id": "9bba8073-6595-48ac-95d1-411f47eb06d8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "13c0f2dd-67f8-452e-9ae1-7032cd1c2fd2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "77349e6b-f4bf-4c87-bbd6-e63c1d1f5dbf", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "44917eba-369e-4f00-b14e-d90a72996a59", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6f95f22c-940b-4190-8b1d-aa7608985848", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4be0e3c5-3444-4a99-90cc-5fde4336a43c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9e88ac8c-209a-4dac-9fcd-424820b5847c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a2608bb2-8a02-4996-ac8f-c4cc9f412e0d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c2265ee3-ee0b-4983-8296-ca2ac0e82e96", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 30, + "rss_bytes": 99299328, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 28, + "messages_unacknowledged": 4 + }, + "seconds": 5.625640291999844 + }, + { + "jobs": [ + { + "id": "51fd30af-2c11-4fc0-8df7-bed99c51d0b3", + "status": "ready", + "queue_wait_seconds": 1.037055 + }, + { + "id": "61eeb2df-ae10-4683-975d-1988ff0bdfd7", + "status": "ready", + "queue_wait_seconds": 1.071236 + }, + { + "id": "9000f960-8632-431e-81f7-81e7c6e14b10", + "status": "ready", + "queue_wait_seconds": 3.440742 + }, + { + "id": "c60fcbbf-afaa-4bb6-8ab0-a35b01058a27", + "status": "ready", + "queue_wait_seconds": 3.473585 + }, + { + "id": "abf827fa-1e39-48a2-99d3-1e357b309cef", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0b2efd1-b1df-457b-8bf5-ea42ea43c6b1", + "status": "ready", + "queue_wait_seconds": 3.526715 + }, + { + "id": "dc5a345a-2890-4221-9a12-666f17a97528", + "status": "ready", + "queue_wait_seconds": 3.49971 + }, + { + "id": "40fe12b2-f03d-4263-b3e0-07716dffc53e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2aa39587-9a7b-4adf-be73-70174684c6f9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "92bda7ec-45b2-4733-9931-a605b1e43842", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f99ad64e-7ddc-4f6c-8826-36edbf5d999c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "24a8e5b5-48a3-44b7-bc83-af8218c9ac86", + "status": "ready", + "queue_wait_seconds": 1.063697 + }, + { + "id": "eb974799-6d8d-4e0e-8083-cf06cbadf551", + "status": "processing", + "queue_wait_seconds": 5.53124 + }, + { + "id": "b05bbb3a-2aef-419b-b0dd-525698eba0f1", + "status": "processing", + "queue_wait_seconds": 5.715995 + }, + { + "id": "b44c6091-c4a4-413f-8d16-2b50adadd4d2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4ce61788-027d-4109-a333-cb23cb33d305", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "fb7d9ddd-81ff-4659-a9bf-80dcbe1b2692", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9e232507-1b4d-4ea9-94de-626c6e28eeda", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "30d1137f-2ca7-4442-8e82-e16867c24155", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a45073f1-59bf-47b0-bb00-1cdc4d5a97cc", + "status": "processing", + "queue_wait_seconds": 5.809955 + }, + { + "id": "2e4fea6b-7a05-47df-b433-a4a413d399e0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6f56f236-1859-41b9-acd6-8cbf7268ff92", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1e00cefc-ce86-42e0-b17f-97a7166f17b2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "72791978-e968-44e0-accd-45a1dc2f3644", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "666f0462-8bde-499c-943a-d1a9a52ebee0", + "status": "ready", + "queue_wait_seconds": 1.02112 + }, + { + "id": "f0d760ed-6d3e-45ea-8854-5acec6e66ba0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "95741924-c00b-4a1c-ae77-eeeefc56eb0f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "34edf97f-f853-404a-9d16-79951cd28cac", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9bba8073-6595-48ac-95d1-411f47eb06d8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "13c0f2dd-67f8-452e-9ae1-7032cd1c2fd2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "77349e6b-f4bf-4c87-bbd6-e63c1d1f5dbf", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "44917eba-369e-4f00-b14e-d90a72996a59", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6f95f22c-940b-4190-8b1d-aa7608985848", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "4be0e3c5-3444-4a99-90cc-5fde4336a43c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9e88ac8c-209a-4dac-9fcd-424820b5847c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a2608bb2-8a02-4996-ac8f-c4cc9f412e0d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c2265ee3-ee0b-4983-8296-ca2ac0e82e96", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37484fa0-a23d-4ee9-b688-f9cf59e727a2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ba5f12d9-4be2-4c84-b4d2-94d1601748ef", + "status": "processing", + "queue_wait_seconds": 5.560767 + }, + { + "id": "5ef0393b-c667-4b1a-938c-4ce6623b2fbb", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 32, + "rss_bytes": 530886656, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 24, + "messages_unacknowledged": 4 + }, + "seconds": 8.938039791999927 + }, + { + "jobs": [ + { + "id": "ba5f12d9-4be2-4c84-b4d2-94d1601748ef", + "status": "ready", + "queue_wait_seconds": 5.560767 + }, + { + "id": "51fd30af-2c11-4fc0-8df7-bed99c51d0b3", + "status": "ready", + "queue_wait_seconds": 1.037055 + }, + { + "id": "61eeb2df-ae10-4683-975d-1988ff0bdfd7", + "status": "ready", + "queue_wait_seconds": 1.071236 + }, + { + "id": "9000f960-8632-431e-81f7-81e7c6e14b10", + "status": "ready", + "queue_wait_seconds": 3.440742 + }, + { + "id": "c60fcbbf-afaa-4bb6-8ab0-a35b01058a27", + "status": "ready", + "queue_wait_seconds": 3.473585 + }, + { + "id": "abf827fa-1e39-48a2-99d3-1e357b309cef", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0b2efd1-b1df-457b-8bf5-ea42ea43c6b1", + "status": "ready", + "queue_wait_seconds": 3.526715 + }, + { + "id": "dc5a345a-2890-4221-9a12-666f17a97528", + "status": "ready", + "queue_wait_seconds": 3.49971 + }, + { + "id": "40fe12b2-f03d-4263-b3e0-07716dffc53e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2aa39587-9a7b-4adf-be73-70174684c6f9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "92bda7ec-45b2-4733-9931-a605b1e43842", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f99ad64e-7ddc-4f6c-8826-36edbf5d999c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "24a8e5b5-48a3-44b7-bc83-af8218c9ac86", + "status": "ready", + "queue_wait_seconds": 1.063697 + }, + { + "id": "eb974799-6d8d-4e0e-8083-cf06cbadf551", + "status": "ready", + "queue_wait_seconds": 5.53124 + }, + { + "id": "b05bbb3a-2aef-419b-b0dd-525698eba0f1", + "status": "ready", + "queue_wait_seconds": 5.715995 + }, + { + "id": "b44c6091-c4a4-413f-8d16-2b50adadd4d2", + "status": "ready", + "queue_wait_seconds": 7.500136 + }, + { + "id": "4ce61788-027d-4109-a333-cb23cb33d305", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "fb7d9ddd-81ff-4659-a9bf-80dcbe1b2692", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9e232507-1b4d-4ea9-94de-626c6e28eeda", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "30d1137f-2ca7-4442-8e82-e16867c24155", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a45073f1-59bf-47b0-bb00-1cdc4d5a97cc", + "status": "ready", + "queue_wait_seconds": 5.809955 + }, + { + "id": "2e4fea6b-7a05-47df-b433-a4a413d399e0", + "status": "ready", + "queue_wait_seconds": 7.689263 + }, + { + "id": "6f56f236-1859-41b9-acd6-8cbf7268ff92", + "status": "processing", + "queue_wait_seconds": 8.135161 + }, + { + "id": "1e00cefc-ce86-42e0-b17f-97a7166f17b2", + "status": "processing", + "queue_wait_seconds": 9.785291 + }, + { + "id": "72791978-e968-44e0-accd-45a1dc2f3644", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "666f0462-8bde-499c-943a-d1a9a52ebee0", + "status": "ready", + "queue_wait_seconds": 1.02112 + }, + { + "id": "f0d760ed-6d3e-45ea-8854-5acec6e66ba0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "95741924-c00b-4a1c-ae77-eeeefc56eb0f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "34edf97f-f853-404a-9d16-79951cd28cac", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9bba8073-6595-48ac-95d1-411f47eb06d8", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "13c0f2dd-67f8-452e-9ae1-7032cd1c2fd2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "77349e6b-f4bf-4c87-bbd6-e63c1d1f5dbf", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "44917eba-369e-4f00-b14e-d90a72996a59", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6f95f22c-940b-4190-8b1d-aa7608985848", + "status": "processing", + "queue_wait_seconds": 8.010705 + }, + { + "id": "4be0e3c5-3444-4a99-90cc-5fde4336a43c", + "status": "processing", + "queue_wait_seconds": 9.608512 + }, + { + "id": "9e88ac8c-209a-4dac-9fcd-424820b5847c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a2608bb2-8a02-4996-ac8f-c4cc9f412e0d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c2265ee3-ee0b-4983-8296-ca2ac0e82e96", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37484fa0-a23d-4ee9-b688-f9cf59e727a2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5ef0393b-c667-4b1a-938c-4ce6623b2fbb", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 26, + "rss_bytes": 442515456, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 18, + "messages_unacknowledged": 4 + }, + "seconds": 12.644221624999773 + }, + { + "jobs": [ + { + "id": "ba5f12d9-4be2-4c84-b4d2-94d1601748ef", + "status": "ready", + "queue_wait_seconds": 5.560767 + }, + { + "id": "51fd30af-2c11-4fc0-8df7-bed99c51d0b3", + "status": "ready", + "queue_wait_seconds": 1.037055 + }, + { + "id": "61eeb2df-ae10-4683-975d-1988ff0bdfd7", + "status": "ready", + "queue_wait_seconds": 1.071236 + }, + { + "id": "9000f960-8632-431e-81f7-81e7c6e14b10", + "status": "ready", + "queue_wait_seconds": 3.440742 + }, + { + "id": "c60fcbbf-afaa-4bb6-8ab0-a35b01058a27", + "status": "ready", + "queue_wait_seconds": 3.473585 + }, + { + "id": "abf827fa-1e39-48a2-99d3-1e357b309cef", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0b2efd1-b1df-457b-8bf5-ea42ea43c6b1", + "status": "ready", + "queue_wait_seconds": 3.526715 + }, + { + "id": "dc5a345a-2890-4221-9a12-666f17a97528", + "status": "ready", + "queue_wait_seconds": 3.49971 + }, + { + "id": "4be0e3c5-3444-4a99-90cc-5fde4336a43c", + "status": "ready", + "queue_wait_seconds": 9.608512 + }, + { + "id": "1e00cefc-ce86-42e0-b17f-97a7166f17b2", + "status": "ready", + "queue_wait_seconds": 9.785291 + }, + { + "id": "40fe12b2-f03d-4263-b3e0-07716dffc53e", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2aa39587-9a7b-4adf-be73-70174684c6f9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "92bda7ec-45b2-4733-9931-a605b1e43842", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f99ad64e-7ddc-4f6c-8826-36edbf5d999c", + "status": "processing", + "queue_wait_seconds": 12.26839 + }, + { + "id": "24a8e5b5-48a3-44b7-bc83-af8218c9ac86", + "status": "ready", + "queue_wait_seconds": 1.063697 + }, + { + "id": "eb974799-6d8d-4e0e-8083-cf06cbadf551", + "status": "ready", + "queue_wait_seconds": 5.53124 + }, + { + "id": "b05bbb3a-2aef-419b-b0dd-525698eba0f1", + "status": "ready", + "queue_wait_seconds": 5.715995 + }, + { + "id": "b44c6091-c4a4-413f-8d16-2b50adadd4d2", + "status": "ready", + "queue_wait_seconds": 7.500136 + }, + { + "id": "4ce61788-027d-4109-a333-cb23cb33d305", + "status": "processing", + "queue_wait_seconds": 11.720122 + }, + { + "id": "fb7d9ddd-81ff-4659-a9bf-80dcbe1b2692", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9e232507-1b4d-4ea9-94de-626c6e28eeda", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "30d1137f-2ca7-4442-8e82-e16867c24155", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a45073f1-59bf-47b0-bb00-1cdc4d5a97cc", + "status": "ready", + "queue_wait_seconds": 5.809955 + }, + { + "id": "2e4fea6b-7a05-47df-b433-a4a413d399e0", + "status": "ready", + "queue_wait_seconds": 7.689263 + }, + { + "id": "6f56f236-1859-41b9-acd6-8cbf7268ff92", + "status": "ready", + "queue_wait_seconds": 8.135161 + }, + { + "id": "72791978-e968-44e0-accd-45a1dc2f3644", + "status": "ready", + "queue_wait_seconds": 10.304263 + }, + { + "id": "666f0462-8bde-499c-943a-d1a9a52ebee0", + "status": "ready", + "queue_wait_seconds": 1.02112 + }, + { + "id": "f0d760ed-6d3e-45ea-8854-5acec6e66ba0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "95741924-c00b-4a1c-ae77-eeeefc56eb0f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "34edf97f-f853-404a-9d16-79951cd28cac", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9bba8073-6595-48ac-95d1-411f47eb06d8", + "status": "ready", + "queue_wait_seconds": 10.312805 + }, + { + "id": "13c0f2dd-67f8-452e-9ae1-7032cd1c2fd2", + "status": "processing", + "queue_wait_seconds": 11.695531 + }, + { + "id": "77349e6b-f4bf-4c87-bbd6-e63c1d1f5dbf", + "status": "processing", + "queue_wait_seconds": 12.274454 + }, + { + "id": "44917eba-369e-4f00-b14e-d90a72996a59", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6f95f22c-940b-4190-8b1d-aa7608985848", + "status": "ready", + "queue_wait_seconds": 8.010705 + }, + { + "id": "9e88ac8c-209a-4dac-9fcd-424820b5847c", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a2608bb2-8a02-4996-ac8f-c4cc9f412e0d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c2265ee3-ee0b-4983-8296-ca2ac0e82e96", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "37484fa0-a23d-4ee9-b688-f9cf59e727a2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5ef0393b-c667-4b1a-938c-4ce6623b2fbb", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 20, + "rss_bytes": 543588352, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 12, + "messages_unacknowledged": 4 + }, + "seconds": 16.017298166999808 + }, + { + "jobs": [ + { + "id": "ba5f12d9-4be2-4c84-b4d2-94d1601748ef", + "status": "ready", + "queue_wait_seconds": 5.560767 + }, + { + "id": "51fd30af-2c11-4fc0-8df7-bed99c51d0b3", + "status": "ready", + "queue_wait_seconds": 1.037055 + }, + { + "id": "61eeb2df-ae10-4683-975d-1988ff0bdfd7", + "status": "ready", + "queue_wait_seconds": 1.071236 + }, + { + "id": "9000f960-8632-431e-81f7-81e7c6e14b10", + "status": "ready", + "queue_wait_seconds": 3.440742 + }, + { + "id": "c60fcbbf-afaa-4bb6-8ab0-a35b01058a27", + "status": "ready", + "queue_wait_seconds": 3.473585 + }, + { + "id": "abf827fa-1e39-48a2-99d3-1e357b309cef", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0b2efd1-b1df-457b-8bf5-ea42ea43c6b1", + "status": "ready", + "queue_wait_seconds": 3.526715 + }, + { + "id": "dc5a345a-2890-4221-9a12-666f17a97528", + "status": "ready", + "queue_wait_seconds": 3.49971 + }, + { + "id": "4be0e3c5-3444-4a99-90cc-5fde4336a43c", + "status": "ready", + "queue_wait_seconds": 9.608512 + }, + { + "id": "1e00cefc-ce86-42e0-b17f-97a7166f17b2", + "status": "ready", + "queue_wait_seconds": 9.785291 + }, + { + "id": "4ce61788-027d-4109-a333-cb23cb33d305", + "status": "ready", + "queue_wait_seconds": 11.720122 + }, + { + "id": "44917eba-369e-4f00-b14e-d90a72996a59", + "status": "ready", + "queue_wait_seconds": 13.956924 + }, + { + "id": "2aa39587-9a7b-4adf-be73-70174684c6f9", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "92bda7ec-45b2-4733-9931-a605b1e43842", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f99ad64e-7ddc-4f6c-8826-36edbf5d999c", + "status": "ready", + "queue_wait_seconds": 12.26839 + }, + { + "id": "24a8e5b5-48a3-44b7-bc83-af8218c9ac86", + "status": "ready", + "queue_wait_seconds": 1.063697 + }, + { + "id": "eb974799-6d8d-4e0e-8083-cf06cbadf551", + "status": "ready", + "queue_wait_seconds": 5.53124 + }, + { + "id": "b05bbb3a-2aef-419b-b0dd-525698eba0f1", + "status": "ready", + "queue_wait_seconds": 5.715995 + }, + { + "id": "b44c6091-c4a4-413f-8d16-2b50adadd4d2", + "status": "ready", + "queue_wait_seconds": 7.500136 + }, + { + "id": "fb7d9ddd-81ff-4659-a9bf-80dcbe1b2692", + "status": "ready", + "queue_wait_seconds": 14.021433 + }, + { + "id": "9e232507-1b4d-4ea9-94de-626c6e28eeda", + "status": "ready", + "queue_wait_seconds": 14.589506 + }, + { + "id": "30d1137f-2ca7-4442-8e82-e16867c24155", + "status": "processing", + "queue_wait_seconds": 15.992084 + }, + { + "id": "a45073f1-59bf-47b0-bb00-1cdc4d5a97cc", + "status": "ready", + "queue_wait_seconds": 5.809955 + }, + { + "id": "2e4fea6b-7a05-47df-b433-a4a413d399e0", + "status": "ready", + "queue_wait_seconds": 7.689263 + }, + { + "id": "6f56f236-1859-41b9-acd6-8cbf7268ff92", + "status": "ready", + "queue_wait_seconds": 8.135161 + }, + { + "id": "72791978-e968-44e0-accd-45a1dc2f3644", + "status": "ready", + "queue_wait_seconds": 10.304263 + }, + { + "id": "666f0462-8bde-499c-943a-d1a9a52ebee0", + "status": "ready", + "queue_wait_seconds": 1.02112 + }, + { + "id": "f0d760ed-6d3e-45ea-8854-5acec6e66ba0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "95741924-c00b-4a1c-ae77-eeeefc56eb0f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "34edf97f-f853-404a-9d16-79951cd28cac", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9bba8073-6595-48ac-95d1-411f47eb06d8", + "status": "ready", + "queue_wait_seconds": 10.312805 + }, + { + "id": "13c0f2dd-67f8-452e-9ae1-7032cd1c2fd2", + "status": "ready", + "queue_wait_seconds": 11.695531 + }, + { + "id": "77349e6b-f4bf-4c87-bbd6-e63c1d1f5dbf", + "status": "ready", + "queue_wait_seconds": 12.274454 + }, + { + "id": "6f95f22c-940b-4190-8b1d-aa7608985848", + "status": "ready", + "queue_wait_seconds": 8.010705 + }, + { + "id": "9e88ac8c-209a-4dac-9fcd-424820b5847c", + "status": "ready", + "queue_wait_seconds": 14.523599 + }, + { + "id": "a2608bb2-8a02-4996-ac8f-c4cc9f412e0d", + "status": "processing", + "queue_wait_seconds": 15.926957 + }, + { + "id": "c2265ee3-ee0b-4983-8296-ca2ac0e82e96", + "status": "processing", + "queue_wait_seconds": 16.595398 + }, + { + "id": "37484fa0-a23d-4ee9-b688-f9cf59e727a2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5ef0393b-c667-4b1a-938c-4ce6623b2fbb", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "40fe12b2-f03d-4263-b3e0-07716dffc53e", + "status": "processing", + "queue_wait_seconds": 16.624492 + } + ], + "pending": 12, + "rss_bytes": 169066496, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 8, + "messages_unacknowledged": 4 + }, + "seconds": 19.03751737499988 + }, + { + "jobs": [ + { + "id": "ba5f12d9-4be2-4c84-b4d2-94d1601748ef", + "status": "ready", + "queue_wait_seconds": 5.560767 + }, + { + "id": "51fd30af-2c11-4fc0-8df7-bed99c51d0b3", + "status": "ready", + "queue_wait_seconds": 1.037055 + }, + { + "id": "61eeb2df-ae10-4683-975d-1988ff0bdfd7", + "status": "ready", + "queue_wait_seconds": 1.071236 + }, + { + "id": "9000f960-8632-431e-81f7-81e7c6e14b10", + "status": "ready", + "queue_wait_seconds": 3.440742 + }, + { + "id": "c60fcbbf-afaa-4bb6-8ab0-a35b01058a27", + "status": "ready", + "queue_wait_seconds": 3.473585 + }, + { + "id": "abf827fa-1e39-48a2-99d3-1e357b309cef", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0b2efd1-b1df-457b-8bf5-ea42ea43c6b1", + "status": "ready", + "queue_wait_seconds": 3.526715 + }, + { + "id": "dc5a345a-2890-4221-9a12-666f17a97528", + "status": "ready", + "queue_wait_seconds": 3.49971 + }, + { + "id": "4be0e3c5-3444-4a99-90cc-5fde4336a43c", + "status": "ready", + "queue_wait_seconds": 9.608512 + }, + { + "id": "1e00cefc-ce86-42e0-b17f-97a7166f17b2", + "status": "ready", + "queue_wait_seconds": 9.785291 + }, + { + "id": "4ce61788-027d-4109-a333-cb23cb33d305", + "status": "ready", + "queue_wait_seconds": 11.720122 + }, + { + "id": "44917eba-369e-4f00-b14e-d90a72996a59", + "status": "ready", + "queue_wait_seconds": 13.956924 + }, + { + "id": "2aa39587-9a7b-4adf-be73-70174684c6f9", + "status": "processing", + "queue_wait_seconds": 15.797952 + }, + { + "id": "92bda7ec-45b2-4733-9931-a605b1e43842", + "status": "processing", + "queue_wait_seconds": 16.166105 + }, + { + "id": "f99ad64e-7ddc-4f6c-8826-36edbf5d999c", + "status": "ready", + "queue_wait_seconds": 12.26839 + }, + { + "id": "24a8e5b5-48a3-44b7-bc83-af8218c9ac86", + "status": "ready", + "queue_wait_seconds": 1.063697 + }, + { + "id": "eb974799-6d8d-4e0e-8083-cf06cbadf551", + "status": "ready", + "queue_wait_seconds": 5.53124 + }, + { + "id": "b05bbb3a-2aef-419b-b0dd-525698eba0f1", + "status": "ready", + "queue_wait_seconds": 5.715995 + }, + { + "id": "b44c6091-c4a4-413f-8d16-2b50adadd4d2", + "status": "ready", + "queue_wait_seconds": 7.500136 + }, + { + "id": "fb7d9ddd-81ff-4659-a9bf-80dcbe1b2692", + "status": "ready", + "queue_wait_seconds": 14.021433 + }, + { + "id": "9e232507-1b4d-4ea9-94de-626c6e28eeda", + "status": "ready", + "queue_wait_seconds": 14.589506 + }, + { + "id": "a45073f1-59bf-47b0-bb00-1cdc4d5a97cc", + "status": "ready", + "queue_wait_seconds": 5.809955 + }, + { + "id": "2e4fea6b-7a05-47df-b433-a4a413d399e0", + "status": "ready", + "queue_wait_seconds": 7.689263 + }, + { + "id": "6f56f236-1859-41b9-acd6-8cbf7268ff92", + "status": "ready", + "queue_wait_seconds": 8.135161 + }, + { + "id": "72791978-e968-44e0-accd-45a1dc2f3644", + "status": "ready", + "queue_wait_seconds": 10.304263 + }, + { + "id": "666f0462-8bde-499c-943a-d1a9a52ebee0", + "status": "ready", + "queue_wait_seconds": 1.02112 + }, + { + "id": "f0d760ed-6d3e-45ea-8854-5acec6e66ba0", + "status": "processing", + "queue_wait_seconds": 16.117251 + }, + { + "id": "95741924-c00b-4a1c-ae77-eeeefc56eb0f", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "34edf97f-f853-404a-9d16-79951cd28cac", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9bba8073-6595-48ac-95d1-411f47eb06d8", + "status": "ready", + "queue_wait_seconds": 10.312805 + }, + { + "id": "13c0f2dd-67f8-452e-9ae1-7032cd1c2fd2", + "status": "ready", + "queue_wait_seconds": 11.695531 + }, + { + "id": "77349e6b-f4bf-4c87-bbd6-e63c1d1f5dbf", + "status": "ready", + "queue_wait_seconds": 12.274454 + }, + { + "id": "6f95f22c-940b-4190-8b1d-aa7608985848", + "status": "ready", + "queue_wait_seconds": 8.010705 + }, + { + "id": "9e88ac8c-209a-4dac-9fcd-424820b5847c", + "status": "ready", + "queue_wait_seconds": 14.523599 + }, + { + "id": "a2608bb2-8a02-4996-ac8f-c4cc9f412e0d", + "status": "ready", + "queue_wait_seconds": 15.926957 + }, + { + "id": "c2265ee3-ee0b-4983-8296-ca2ac0e82e96", + "status": "ready", + "queue_wait_seconds": 16.595398 + }, + { + "id": "37484fa0-a23d-4ee9-b688-f9cf59e727a2", + "status": "processing", + "queue_wait_seconds": 15.716894 + }, + { + "id": "5ef0393b-c667-4b1a-938c-4ce6623b2fbb", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "40fe12b2-f03d-4263-b3e0-07716dffc53e", + "status": "ready", + "queue_wait_seconds": 16.624492 + }, + { + "id": "30d1137f-2ca7-4442-8e82-e16867c24155", + "status": "ready", + "queue_wait_seconds": 15.992084 + } + ], + "pending": 8, + "rss_bytes": 477585408, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 0, + "messages_unacknowledged": 4 + }, + "seconds": 22.34022991699976 + }, + { + "jobs": [ + { + "id": "ba5f12d9-4be2-4c84-b4d2-94d1601748ef", + "status": "ready", + "queue_wait_seconds": 5.560767 + }, + { + "id": "51fd30af-2c11-4fc0-8df7-bed99c51d0b3", + "status": "ready", + "queue_wait_seconds": 1.037055 + }, + { + "id": "61eeb2df-ae10-4683-975d-1988ff0bdfd7", + "status": "ready", + "queue_wait_seconds": 1.071236 + }, + { + "id": "9000f960-8632-431e-81f7-81e7c6e14b10", + "status": "ready", + "queue_wait_seconds": 3.440742 + }, + { + "id": "c60fcbbf-afaa-4bb6-8ab0-a35b01058a27", + "status": "ready", + "queue_wait_seconds": 3.473585 + }, + { + "id": "c0b2efd1-b1df-457b-8bf5-ea42ea43c6b1", + "status": "ready", + "queue_wait_seconds": 3.526715 + }, + { + "id": "dc5a345a-2890-4221-9a12-666f17a97528", + "status": "ready", + "queue_wait_seconds": 3.49971 + }, + { + "id": "4be0e3c5-3444-4a99-90cc-5fde4336a43c", + "status": "ready", + "queue_wait_seconds": 9.608512 + }, + { + "id": "1e00cefc-ce86-42e0-b17f-97a7166f17b2", + "status": "ready", + "queue_wait_seconds": 9.785291 + }, + { + "id": "4ce61788-027d-4109-a333-cb23cb33d305", + "status": "ready", + "queue_wait_seconds": 11.720122 + }, + { + "id": "44917eba-369e-4f00-b14e-d90a72996a59", + "status": "ready", + "queue_wait_seconds": 13.956924 + }, + { + "id": "2aa39587-9a7b-4adf-be73-70174684c6f9", + "status": "ready", + "queue_wait_seconds": 15.797952 + }, + { + "id": "f99ad64e-7ddc-4f6c-8826-36edbf5d999c", + "status": "ready", + "queue_wait_seconds": 12.26839 + }, + { + "id": "24a8e5b5-48a3-44b7-bc83-af8218c9ac86", + "status": "ready", + "queue_wait_seconds": 1.063697 + }, + { + "id": "eb974799-6d8d-4e0e-8083-cf06cbadf551", + "status": "ready", + "queue_wait_seconds": 5.53124 + }, + { + "id": "b05bbb3a-2aef-419b-b0dd-525698eba0f1", + "status": "ready", + "queue_wait_seconds": 5.715995 + }, + { + "id": "b44c6091-c4a4-413f-8d16-2b50adadd4d2", + "status": "ready", + "queue_wait_seconds": 7.500136 + }, + { + "id": "fb7d9ddd-81ff-4659-a9bf-80dcbe1b2692", + "status": "ready", + "queue_wait_seconds": 14.021433 + }, + { + "id": "9e232507-1b4d-4ea9-94de-626c6e28eeda", + "status": "ready", + "queue_wait_seconds": 14.589506 + }, + { + "id": "a45073f1-59bf-47b0-bb00-1cdc4d5a97cc", + "status": "ready", + "queue_wait_seconds": 5.809955 + }, + { + "id": "2e4fea6b-7a05-47df-b433-a4a413d399e0", + "status": "ready", + "queue_wait_seconds": 7.689263 + }, + { + "id": "6f56f236-1859-41b9-acd6-8cbf7268ff92", + "status": "ready", + "queue_wait_seconds": 8.135161 + }, + { + "id": "72791978-e968-44e0-accd-45a1dc2f3644", + "status": "ready", + "queue_wait_seconds": 10.304263 + }, + { + "id": "666f0462-8bde-499c-943a-d1a9a52ebee0", + "status": "ready", + "queue_wait_seconds": 1.02112 + }, + { + "id": "f0d760ed-6d3e-45ea-8854-5acec6e66ba0", + "status": "ready", + "queue_wait_seconds": 16.117251 + }, + { + "id": "95741924-c00b-4a1c-ae77-eeeefc56eb0f", + "status": "ready", + "queue_wait_seconds": 15.72333 + }, + { + "id": "34edf97f-f853-404a-9d16-79951cd28cac", + "status": "processing", + "queue_wait_seconds": 16.092245 + }, + { + "id": "9bba8073-6595-48ac-95d1-411f47eb06d8", + "status": "ready", + "queue_wait_seconds": 10.312805 + }, + { + "id": "13c0f2dd-67f8-452e-9ae1-7032cd1c2fd2", + "status": "ready", + "queue_wait_seconds": 11.695531 + }, + { + "id": "77349e6b-f4bf-4c87-bbd6-e63c1d1f5dbf", + "status": "ready", + "queue_wait_seconds": 12.274454 + }, + { + "id": "6f95f22c-940b-4190-8b1d-aa7608985848", + "status": "ready", + "queue_wait_seconds": 8.010705 + }, + { + "id": "9e88ac8c-209a-4dac-9fcd-424820b5847c", + "status": "ready", + "queue_wait_seconds": 14.523599 + }, + { + "id": "a2608bb2-8a02-4996-ac8f-c4cc9f412e0d", + "status": "ready", + "queue_wait_seconds": 15.926957 + }, + { + "id": "c2265ee3-ee0b-4983-8296-ca2ac0e82e96", + "status": "ready", + "queue_wait_seconds": 16.595398 + }, + { + "id": "5ef0393b-c667-4b1a-938c-4ce6623b2fbb", + "status": "processing", + "queue_wait_seconds": 16.099341 + }, + { + "id": "40fe12b2-f03d-4263-b3e0-07716dffc53e", + "status": "ready", + "queue_wait_seconds": 16.624492 + }, + { + "id": "30d1137f-2ca7-4442-8e82-e16867c24155", + "status": "ready", + "queue_wait_seconds": 15.992084 + }, + { + "id": "37484fa0-a23d-4ee9-b688-f9cf59e727a2", + "status": "ready", + "queue_wait_seconds": 15.716894 + }, + { + "id": "abf827fa-1e39-48a2-99d3-1e357b309cef", + "status": "ready", + "queue_wait_seconds": 15.802003 + }, + { + "id": "92bda7ec-45b2-4733-9931-a605b1e43842", + "status": "ready", + "queue_wait_seconds": 16.166105 + } + ], + "pending": 2, + "rss_bytes": 457875456, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 0, + "messages_unacknowledged": 0 + }, + "seconds": 25.01005570899997 + } + ], + "metric_errors": [], + "limitations": "Closed-loop; ready time is polling upper bound. RSS is sum of worker/container processes, shared pages may be counted twice. Sampling may miss peaks. Queue wait is first claim minus durable job creation." +} diff --git a/evidence/real-c4-p4-s512.json b/evidence/real-c4-p4-s512.json new file mode 100644 index 00000000..d5f85590 --- /dev/null +++ b/evidence/real-c4-p4-s512.json @@ -0,0 +1,1760 @@ +{ + "run_name": "real-c4-p4-s512", + "settings": { + "workers": 4, + "prefetch": 4, + "max_pending": 32, + "worker_limits": { + "Memory": 4294967296, + "NanoCpus": 2000000000 + } + }, + "workload": { + "images": 160, + "concurrency": 16, + "size": 512, + "bytes": 1890, + "type": "closed-loop burst" + }, + "results": { + "accepted": 32, + "rejected_429": 128, + "ready": 32, + "failed": 0, + "timed_out": 0, + "elapsed_seconds": 15.487373459000082, + "ready_per_second": 2.066199287097583, + "ready_latency_ms": { + "0.5": 8368.798, + "0.95": 14969.782, + "0.99": 14970.546 + }, + "queue_wait_ms": { + "0.5": 6616.724, + "0.95": 13602.215, + "0.99": 13612.007 + }, + "max_sampled_rss_bytes": 300544000, + "max_sampled_pending": 28, + "sample_count": 5, + "drained": true, + "recovery_upload_ready": true + }, + "records": [ + { + "index": 1, + "status": 202, + "upload_seconds": 0.04753729200001544, + "id": "e6ab4777-a2a7-4dd1-b744-ca0e3bb4b016", + "terminal": "ready", + "ready_seconds": 2.9483837499997207 + }, + { + "index": 0, + "status": 202, + "upload_seconds": 0.058448124999813444, + "id": "cd3bd3f0-d5e3-4a01-996b-2396320c9435", + "terminal": "ready", + "ready_seconds": 2.941110292000303 + }, + { + "index": 3, + "status": 202, + "upload_seconds": 0.06258716699994693, + "id": "315beea2-9aa1-43a7-abbb-ad3f1898dffe", + "terminal": "ready", + "ready_seconds": 2.938533500000176 + }, + { + "index": 5, + "status": 202, + "upload_seconds": 0.06796133300031215, + "id": "36496779-4f8b-400f-bbd5-1bd069aa9d80", + "terminal": "ready", + "ready_seconds": 2.935570082999675 + }, + { + "index": 2, + "status": 202, + "upload_seconds": 0.07536149999987174, + "id": "9cb3b8b5-02dd-4d7c-8fec-5617f65bdb57", + "terminal": "ready", + "ready_seconds": 4.8476730829997905 + }, + { + "index": 4, + "status": 202, + "upload_seconds": 0.08149491599988323, + "id": "f5b4dab7-b060-4c06-a458-276f808fb113", + "terminal": "ready", + "ready_seconds": 4.8436147500001425 + }, + { + "index": 7, + "status": 202, + "upload_seconds": 0.08933875000002445, + "id": "5e086ebb-f46d-4bf2-8e6c-700327b5becb", + "terminal": "ready", + "ready_seconds": 4.836734041999989 + }, + { + "index": 6, + "status": 202, + "upload_seconds": 0.14503208299993275, + "id": "9df958ad-72be-4244-8035-9bd3a4428362", + "terminal": "ready", + "ready_seconds": 4.782915458999923 + }, + { + "index": 9, + "status": 202, + "upload_seconds": 0.1517882089997329, + "id": "ac26ac0d-9ec3-44e1-81f8-9988299342ae", + "terminal": "ready", + "ready_seconds": 6.554945625000073 + }, + { + "index": 8, + "status": 202, + "upload_seconds": 0.15745708300028127, + "id": "9c4c8f65-c6cc-44e5-9ffc-6e6a85c71fc2", + "terminal": "ready", + "ready_seconds": 6.551889625000058 + }, + { + "index": 10, + "status": 202, + "upload_seconds": 0.16311474999974962, + "id": "139cb0e9-93a0-4a00-9b44-47d2ac2b770d", + "terminal": "ready", + "ready_seconds": 6.5495444170001065 + }, + { + "index": 11, + "status": 202, + "upload_seconds": 0.17345383400015635, + "id": "2f3d268b-6a01-4401-b040-539c57720468", + "terminal": "ready", + "ready_seconds": 6.5411967909999476 + }, + { + "index": 14, + "status": 202, + "upload_seconds": 0.1769182500001989, + "id": "b198ed01-d7f6-49a7-b02e-a23e5263e66a", + "terminal": "ready", + "ready_seconds": 8.252714917000048 + }, + { + "index": 12, + "status": 202, + "upload_seconds": 0.18675541700031317, + "id": "544ee42a-de19-47a1-ae80-2519b975c230", + "terminal": "ready", + "ready_seconds": 8.246630082999673 + }, + { + "index": 13, + "status": 202, + "upload_seconds": 0.23851254099963626, + "id": "1924353a-e5cc-4351-a90d-3e97a5550ac2", + "terminal": "ready", + "ready_seconds": 8.368798459000118 + }, + { + "index": 15, + "status": 202, + "upload_seconds": 0.2530379169998014, + "id": "f92dd859-7dad-4903-a765-609f3909a5b5", + "terminal": "ready", + "ready_seconds": 8.356482041000163 + }, + { + "index": 17, + "status": 202, + "upload_seconds": 0.20464787499986414, + "id": "adf5b8b6-06d8-4d15-a030-cdbcc1abb628", + "terminal": "ready", + "ready_seconds": 9.86125279099997 + }, + { + "index": 16, + "status": 202, + "upload_seconds": 0.2223134169998957, + "id": "3ebfdf6a-8cfe-49fd-95e1-86b2e71115e7", + "terminal": "ready", + "ready_seconds": 10.163923957999941 + }, + { + "index": 19, + "status": 202, + "upload_seconds": 0.20582066599990867, + "id": "dc1cce7e-e39b-43dd-936e-16e9d8c2c6c0", + "terminal": "ready", + "ready_seconds": 9.999229666999781 + }, + { + "index": 18, + "status": 202, + "upload_seconds": 0.21822870799996963, + "id": "c0d36822-0f7a-480d-af44-db1ea433ad19", + "terminal": "ready", + "ready_seconds": 10.309353000000101 + }, + { + "index": 21, + "status": 202, + "upload_seconds": 0.20664875000011307, + "id": "5b407ae5-3918-415d-a898-57a680915bfb", + "terminal": "ready", + "ready_seconds": 11.328423292000025 + }, + { + "index": 20, + "status": 202, + "upload_seconds": 0.270221415999913, + "id": "11ac2a26-b680-4e35-a192-dabc04e22cae", + "terminal": "ready", + "ready_seconds": 11.560134709000067 + }, + { + "index": 24, + "status": 202, + "upload_seconds": 0.1977571249999528, + "id": "ec05ff96-98c2-46df-9701-1589263a990d", + "terminal": "ready", + "ready_seconds": 11.828857999999855 + }, + { + "index": 22, + "status": 202, + "upload_seconds": 0.26920650000010937, + "id": "d75b7e65-235d-45d7-9332-09dc776b1644", + "terminal": "ready", + "ready_seconds": 11.826973167000233 + }, + { + "index": 26, + "status": 202, + "upload_seconds": 0.20661937500017302, + "id": "0fefa5c8-4756-424a-96fb-59408ff618de", + "terminal": "ready", + "ready_seconds": 12.739058541999839 + }, + { + "index": 23, + "status": 202, + "upload_seconds": 0.23995704099979775, + "id": "093fb4c2-87cd-43ca-8a73-3f03a976fd86", + "terminal": "ready", + "ready_seconds": 13.395899458999793 + }, + { + "index": 28, + "status": 202, + "upload_seconds": 0.2597166250002374, + "id": "6e32c3c3-4742-4fa5-95c9-ec99c08206f7", + "terminal": "ready", + "ready_seconds": 13.585583167000095 + }, + { + "index": 25, + "status": 202, + "upload_seconds": 0.28609745899984773, + "id": "938a80d0-e52f-4b29-b14b-425789ad0d48", + "terminal": "ready", + "ready_seconds": 13.584023875000184 + }, + { + "index": 32, + "status": 202, + "upload_seconds": 0.18776133400024264, + "id": "07881790-7f8f-4911-9231-d1611866cddd", + "terminal": "ready", + "ready_seconds": 14.525396457999705 + }, + { + "index": 27, + "status": 202, + "upload_seconds": 0.282620124999994, + "id": "8be53303-9c94-432a-8dac-d16555ba2ec5", + "terminal": "ready", + "ready_seconds": 14.758370625000225 + }, + { + "index": 34, + "status": 202, + "upload_seconds": 0.1891129999999066, + "id": "1caf5d0c-4dd3-45c7-8d5f-cd46c5c2aa82", + "terminal": "ready", + "ready_seconds": 14.970545500000298 + }, + { + "index": 29, + "status": 202, + "upload_seconds": 0.27686441699961506, + "id": "a448ff44-a164-4467-aa3a-10ab224dd952", + "terminal": "ready", + "ready_seconds": 14.969781750000038 + }, + { + "index": 36, + "status": 429, + "upload_seconds": 0.1788566670002183 + }, + { + "index": 30, + "status": 429, + "upload_seconds": 0.2286883330002638 + }, + { + "index": 38, + "status": 429, + "upload_seconds": 0.11974287500015635 + }, + { + "index": 31, + "status": 429, + "upload_seconds": 0.21622341700003744 + }, + { + "index": 42, + "status": 429, + "upload_seconds": 0.03318649999982881 + }, + { + "index": 44, + "status": 429, + "upload_seconds": 0.028098333999878378 + }, + { + "index": 33, + "status": 429, + "upload_seconds": 0.2093574999998964 + }, + { + "index": 35, + "status": 429, + "upload_seconds": 0.20181345899982261 + }, + { + "index": 46, + "status": 429, + "upload_seconds": 0.019832292000046436 + }, + { + "index": 48, + "status": 429, + "upload_seconds": 0.02027704200008884 + }, + { + "index": 37, + "status": 429, + "upload_seconds": 0.14265770800011524 + }, + { + "index": 39, + "status": 429, + "upload_seconds": 0.13151087500000358 + }, + { + "index": 49, + "status": 429, + "upload_seconds": 0.020244875000116735 + }, + { + "index": 52, + "status": 429, + "upload_seconds": 0.019746584000131406 + }, + { + "index": 40, + "status": 429, + "upload_seconds": 0.122717250000278 + }, + { + "index": 54, + "status": 429, + "upload_seconds": 0.018651750000117318 + }, + { + "index": 41, + "status": 429, + "upload_seconds": 0.11198070899990853 + }, + { + "index": 56, + "status": 429, + "upload_seconds": 0.017220166999777575 + }, + { + "index": 43, + "status": 429, + "upload_seconds": 0.055698375000247324 + }, + { + "index": 58, + "status": 429, + "upload_seconds": 0.01582166700018206 + }, + { + "index": 45, + "status": 429, + "upload_seconds": 0.045816500000000815 + }, + { + "index": 60, + "status": 429, + "upload_seconds": 0.015798291999999492 + }, + { + "index": 47, + "status": 429, + "upload_seconds": 0.040965624999898864 + }, + { + "index": 61, + "status": 429, + "upload_seconds": 0.015320291000080033 + }, + { + "index": 50, + "status": 429, + "upload_seconds": 0.039519165999990946 + }, + { + "index": 64, + "status": 429, + "upload_seconds": 0.051109833999817056 + }, + { + "index": 66, + "status": 429, + "upload_seconds": 0.05044879199977004 + }, + { + "index": 67, + "status": 429, + "upload_seconds": 0.050756749999891326 + }, + { + "index": 51, + "status": 429, + "upload_seconds": 0.08278720799989969 + }, + { + "index": 70, + "status": 429, + "upload_seconds": 0.05099541699973997 + }, + { + "index": 53, + "status": 429, + "upload_seconds": 0.08233166699983485 + }, + { + "index": 72, + "status": 429, + "upload_seconds": 0.0524012920000132 + }, + { + "index": 55, + "status": 429, + "upload_seconds": 0.08193108299974483 + }, + { + "index": 57, + "status": 429, + "upload_seconds": 0.07990879099997983 + }, + { + "index": 73, + "status": 429, + "upload_seconds": 0.018355084000177158 + }, + { + "index": 75, + "status": 429, + "upload_seconds": 0.01638008300005822 + }, + { + "index": 59, + "status": 429, + "upload_seconds": 0.0813407499999812 + }, + { + "index": 78, + "status": 429, + "upload_seconds": 0.013762750000296364 + }, + { + "index": 62, + "status": 429, + "upload_seconds": 0.08210583399977622 + }, + { + "index": 79, + "status": 429, + "upload_seconds": 0.01703524999993533 + }, + { + "index": 63, + "status": 429, + "upload_seconds": 0.08378549999997631 + }, + { + "index": 82, + "status": 429, + "upload_seconds": 0.015090875000169035 + }, + { + "index": 65, + "status": 429, + "upload_seconds": 0.08440045799989093 + }, + { + "index": 84, + "status": 429, + "upload_seconds": 0.01574366699969687 + }, + { + "index": 68, + "status": 429, + "upload_seconds": 0.08552383300002475 + }, + { + "index": 69, + "status": 429, + "upload_seconds": 0.08528116699972088 + }, + { + "index": 86, + "status": 429, + "upload_seconds": 0.015591208999921946 + }, + { + "index": 71, + "status": 429, + "upload_seconds": 0.08522816700042313 + }, + { + "index": 87, + "status": 429, + "upload_seconds": 0.01475241600019217 + }, + { + "index": 74, + "status": 429, + "upload_seconds": 0.04690420799988715 + }, + { + "index": 76, + "status": 429, + "upload_seconds": 0.04536020900013682 + }, + { + "index": 77, + "status": 429, + "upload_seconds": 0.09670958300011989 + }, + { + "index": 89, + "status": 429, + "upload_seconds": 0.07204133400000501 + }, + { + "index": 80, + "status": 429, + "upload_seconds": 0.0932608330003859 + }, + { + "index": 92, + "status": 429, + "upload_seconds": 0.06803954199995133 + }, + { + "index": 93, + "status": 429, + "upload_seconds": 0.06962916700013011 + }, + { + "index": 81, + "status": 429, + "upload_seconds": 0.09479862500029412 + }, + { + "index": 83, + "status": 429, + "upload_seconds": 0.09380266600010145 + }, + { + "index": 85, + "status": 429, + "upload_seconds": 0.09493295900028897 + }, + { + "index": 88, + "status": 429, + "upload_seconds": 0.09251587500011738 + }, + { + "index": 90, + "status": 429, + "upload_seconds": 0.09410416599985183 + }, + { + "index": 91, + "status": 429, + "upload_seconds": 0.09533774999999878 + }, + { + "index": 94, + "status": 429, + "upload_seconds": 0.09469170800002757 + }, + { + "index": 96, + "status": 429, + "upload_seconds": 0.09571354099989549 + }, + { + "index": 97, + "status": 429, + "upload_seconds": 0.09629025000003821 + }, + { + "index": 100, + "status": 429, + "upload_seconds": 0.041100749999714026 + }, + { + "index": 102, + "status": 429, + "upload_seconds": 0.04219512500003475 + }, + { + "index": 103, + "status": 429, + "upload_seconds": 0.09371987499980605 + }, + { + "index": 95, + "status": 429, + "upload_seconds": 0.16535937500020736 + }, + { + "index": 107, + "status": 429, + "upload_seconds": 0.08581404100004875 + }, + { + "index": 108, + "status": 429, + "upload_seconds": 0.08434166600000026 + }, + { + "index": 98, + "status": 429, + "upload_seconds": 0.1636935420001464 + }, + { + "index": 110, + "status": 429, + "upload_seconds": 0.08051787500016871 + }, + { + "index": 99, + "status": 429, + "upload_seconds": 0.11403116699966631 + }, + { + "index": 112, + "status": 429, + "upload_seconds": 0.077150999999958 + }, + { + "index": 101, + "status": 429, + "upload_seconds": 0.11568770800022321 + }, + { + "index": 114, + "status": 429, + "upload_seconds": 0.07843283300007897 + }, + { + "index": 104, + "status": 429, + "upload_seconds": 0.11823620799987111 + }, + { + "index": 116, + "status": 429, + "upload_seconds": 0.023709499999768013 + }, + { + "index": 106, + "status": 429, + "upload_seconds": 0.11704491700038488 + }, + { + "index": 117, + "status": 429, + "upload_seconds": 0.029177124999932857 + }, + { + "index": 120, + "status": 429, + "upload_seconds": 0.02551233400026831 + }, + { + "index": 121, + "status": 429, + "upload_seconds": 0.026515291000123398 + }, + { + "index": 123, + "status": 429, + "upload_seconds": 0.025847416000033263 + }, + { + "index": 124, + "status": 429, + "upload_seconds": 0.022417165999740973 + }, + { + "index": 127, + "status": 429, + "upload_seconds": 0.017101667000133602 + }, + { + "index": 128, + "status": 429, + "upload_seconds": 0.01693241599969042 + }, + { + "index": 113, + "status": 429, + "upload_seconds": 0.15219195800000307 + }, + { + "index": 109, + "status": 429, + "upload_seconds": 0.17179320799959896 + }, + { + "index": 130, + "status": 429, + "upload_seconds": 0.05976387500004421 + }, + { + "index": 105, + "status": 429, + "upload_seconds": 0.19575412499989397 + }, + { + "index": 132, + "status": 429, + "upload_seconds": 0.05909445899987986 + }, + { + "index": 111, + "status": 429, + "upload_seconds": 0.17344275000004927 + }, + { + "index": 134, + "status": 429, + "upload_seconds": 0.05773333300021477 + }, + { + "index": 115, + "status": 429, + "upload_seconds": 0.11181125000030079 + }, + { + "index": 136, + "status": 429, + "upload_seconds": 0.0130131670002811 + }, + { + "index": 118, + "status": 429, + "upload_seconds": 0.10940054200000304 + }, + { + "index": 139, + "status": 429, + "upload_seconds": 0.011305541999718116 + }, + { + "index": 140, + "status": 429, + "upload_seconds": 0.012991125000098691 + }, + { + "index": 119, + "status": 429, + "upload_seconds": 0.11497391600005358 + }, + { + "index": 142, + "status": 429, + "upload_seconds": 0.015366832999916369 + }, + { + "index": 122, + "status": 429, + "upload_seconds": 0.1138162080001166 + }, + { + "index": 145, + "status": 429, + "upload_seconds": 0.013733416999912151 + }, + { + "index": 146, + "status": 429, + "upload_seconds": 0.01257383399979517 + }, + { + "index": 125, + "status": 429, + "upload_seconds": 0.11253662499984785 + }, + { + "index": 149, + "status": 429, + "upload_seconds": 0.009620583000014449 + }, + { + "index": 126, + "status": 429, + "upload_seconds": 0.11382695800011788 + }, + { + "index": 133, + "status": 429, + "upload_seconds": 0.1429283749998831 + }, + { + "index": 131, + "status": 429, + "upload_seconds": 0.1515150000000176 + }, + { + "index": 150, + "status": 429, + "upload_seconds": 0.06583062499976222 + }, + { + "index": 129, + "status": 429, + "upload_seconds": 0.16118008300009024 + }, + { + "index": 135, + "status": 429, + "upload_seconds": 0.10885229199993773 + }, + { + "index": 153, + "status": 429, + "upload_seconds": 0.06510295800035237 + }, + { + "index": 137, + "status": 429, + "upload_seconds": 0.10635516599995753 + }, + { + "index": 156, + "status": 429, + "upload_seconds": 0.013374209000176052 + }, + { + "index": 138, + "status": 429, + "upload_seconds": 0.10548737499993877 + }, + { + "index": 159, + "status": 429, + "upload_seconds": 0.009730250000302476 + }, + { + "index": 141, + "status": 429, + "upload_seconds": 0.10453824999967765 + }, + { + "index": 143, + "status": 429, + "upload_seconds": 0.10606179199976395 + }, + { + "index": 144, + "status": 429, + "upload_seconds": 0.10541479099993012 + }, + { + "index": 147, + "status": 429, + "upload_seconds": 0.10280137500012643 + }, + { + "index": 148, + "status": 429, + "upload_seconds": 0.10188541700017595 + }, + { + "index": 151, + "status": 429, + "upload_seconds": 0.0972808750002514 + }, + { + "index": 152, + "status": 429, + "upload_seconds": 0.09751054199978171 + }, + { + "index": 154, + "status": 429, + "upload_seconds": 0.09700299999985873 + }, + { + "index": 155, + "status": 429, + "upload_seconds": 0.049579999999878055 + }, + { + "index": 158, + "status": 429, + "upload_seconds": 0.04376775000037014 + }, + { + "index": 157, + "status": 429, + "upload_seconds": 0.09170020800002021 + } + ], + "samples": [ + { + "jobs": [ + { + "id": "e6ab4777-a2a7-4dd1-b744-ca0e3bb4b016", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "315beea2-9aa1-43a7-abbb-ad3f1898dffe", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "cd3bd3f0-d5e3-4a01-996b-2396320c9435", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "36496779-4f8b-400f-bbd5-1bd069aa9d80", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f5b4dab7-b060-4c06-a458-276f808fb113", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9df958ad-72be-4244-8035-9bd3a4428362", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9cb3b8b5-02dd-4d7c-8fec-5617f65bdb57", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5e086ebb-f46d-4bf2-8e6c-700327b5becb", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ac26ac0d-9ec3-44e1-81f8-9988299342ae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9c4c8f65-c6cc-44e5-9ffc-6e6a85c71fc2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2f3d268b-6a01-4401-b040-539c57720468", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "544ee42a-de19-47a1-ae80-2519b975c230", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f92dd859-7dad-4903-a765-609f3909a5b5", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "139cb0e9-93a0-4a00-9b44-47d2ac2b770d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b198ed01-d7f6-49a7-b02e-a23e5263e66a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1924353a-e5cc-4351-a90d-3e97a5550ac2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "adf5b8b6-06d8-4d15-a030-cdbcc1abb628", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "dc1cce7e-e39b-43dd-936e-16e9d8c2c6c0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5b407ae5-3918-415d-a898-57a680915bfb", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "3ebfdf6a-8cfe-49fd-95e1-86b2e71115e7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0d36822-0f7a-480d-af44-db1ea433ad19", + "status": "queued", + "queue_wait_seconds": null + } + ], + "pending": 21, + "rss_bytes": 51228672, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 16, + "messages_unacknowledged": 4 + }, + "seconds": 1.7026757089997773 + }, + { + "jobs": [ + { + "id": "cd3bd3f0-d5e3-4a01-996b-2396320c9435", + "status": "ready", + "queue_wait_seconds": 1.095668 + }, + { + "id": "e6ab4777-a2a7-4dd1-b744-ca0e3bb4b016", + "status": "ready", + "queue_wait_seconds": 1.101685 + }, + { + "id": "36496779-4f8b-400f-bbd5-1bd069aa9d80", + "status": "ready", + "queue_wait_seconds": 1.142645 + }, + { + "id": "9df958ad-72be-4244-8035-9bd3a4428362", + "status": "processing", + "queue_wait_seconds": 2.981451 + }, + { + "id": "ac26ac0d-9ec3-44e1-81f8-9988299342ae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "6e32c3c3-4742-4fa5-95c9-ec99c08206f7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "07881790-7f8f-4911-9231-d1611866cddd", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1caf5d0c-4dd3-45c7-8d5f-cd46c5c2aa82", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "315beea2-9aa1-43a7-abbb-ad3f1898dffe", + "status": "ready", + "queue_wait_seconds": 1.070289 + }, + { + "id": "9c4c8f65-c6cc-44e5-9ffc-6e6a85c71fc2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "2f3d268b-6a01-4401-b040-539c57720468", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "544ee42a-de19-47a1-ae80-2519b975c230", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f92dd859-7dad-4903-a765-609f3909a5b5", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "938a80d0-e52f-4b29-b14b-425789ad0d48", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8be53303-9c94-432a-8dac-d16555ba2ec5", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a448ff44-a164-4467-aa3a-10ab224dd952", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9cb3b8b5-02dd-4d7c-8fec-5617f65bdb57", + "status": "processing", + "queue_wait_seconds": 2.967999 + }, + { + "id": "139cb0e9-93a0-4a00-9b44-47d2ac2b770d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "b198ed01-d7f6-49a7-b02e-a23e5263e66a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1924353a-e5cc-4351-a90d-3e97a5550ac2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "adf5b8b6-06d8-4d15-a030-cdbcc1abb628", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "dc1cce7e-e39b-43dd-936e-16e9d8c2c6c0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5b407ae5-3918-415d-a898-57a680915bfb", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ec05ff96-98c2-46df-9701-1589263a990d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fefa5c8-4756-424a-96fb-59408ff618de", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f5b4dab7-b060-4c06-a458-276f808fb113", + "status": "processing", + "queue_wait_seconds": 3.000177 + }, + { + "id": "3ebfdf6a-8cfe-49fd-95e1-86b2e71115e7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0d36822-0f7a-480d-af44-db1ea433ad19", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "11ac2a26-b680-4e35-a192-dabc04e22cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d75b7e65-235d-45d7-9332-09dc776b1644", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "093fb4c2-87cd-43ca-8a73-3f03a976fd86", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5e086ebb-f46d-4bf2-8e6c-700327b5becb", + "status": "processing", + "queue_wait_seconds": 3.000233 + } + ], + "pending": 28, + "rss_bytes": 92037120, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 20, + "messages_unacknowledged": 4 + }, + "seconds": 4.914294374999827 + }, + { + "jobs": [ + { + "id": "cd3bd3f0-d5e3-4a01-996b-2396320c9435", + "status": "ready", + "queue_wait_seconds": 1.095668 + }, + { + "id": "e6ab4777-a2a7-4dd1-b744-ca0e3bb4b016", + "status": "ready", + "queue_wait_seconds": 1.101685 + }, + { + "id": "36496779-4f8b-400f-bbd5-1bd069aa9d80", + "status": "ready", + "queue_wait_seconds": 1.142645 + }, + { + "id": "9df958ad-72be-4244-8035-9bd3a4428362", + "status": "ready", + "queue_wait_seconds": 2.981451 + }, + { + "id": "ac26ac0d-9ec3-44e1-81f8-9988299342ae", + "status": "processing", + "queue_wait_seconds": 4.761922 + }, + { + "id": "6e32c3c3-4742-4fa5-95c9-ec99c08206f7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "07881790-7f8f-4911-9231-d1611866cddd", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1caf5d0c-4dd3-45c7-8d5f-cd46c5c2aa82", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "315beea2-9aa1-43a7-abbb-ad3f1898dffe", + "status": "ready", + "queue_wait_seconds": 1.070289 + }, + { + "id": "9c4c8f65-c6cc-44e5-9ffc-6e6a85c71fc2", + "status": "ready", + "queue_wait_seconds": 4.726148 + }, + { + "id": "2f3d268b-6a01-4401-b040-539c57720468", + "status": "ready", + "queue_wait_seconds": 4.724874 + }, + { + "id": "544ee42a-de19-47a1-ae80-2519b975c230", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f92dd859-7dad-4903-a765-609f3909a5b5", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "938a80d0-e52f-4b29-b14b-425789ad0d48", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8be53303-9c94-432a-8dac-d16555ba2ec5", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a448ff44-a164-4467-aa3a-10ab224dd952", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9cb3b8b5-02dd-4d7c-8fec-5617f65bdb57", + "status": "ready", + "queue_wait_seconds": 2.967999 + }, + { + "id": "139cb0e9-93a0-4a00-9b44-47d2ac2b770d", + "status": "processing", + "queue_wait_seconds": 4.707635 + }, + { + "id": "b198ed01-d7f6-49a7-b02e-a23e5263e66a", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1924353a-e5cc-4351-a90d-3e97a5550ac2", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "adf5b8b6-06d8-4d15-a030-cdbcc1abb628", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "dc1cce7e-e39b-43dd-936e-16e9d8c2c6c0", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5b407ae5-3918-415d-a898-57a680915bfb", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ec05ff96-98c2-46df-9701-1589263a990d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fefa5c8-4756-424a-96fb-59408ff618de", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f5b4dab7-b060-4c06-a458-276f808fb113", + "status": "ready", + "queue_wait_seconds": 3.000177 + }, + { + "id": "3ebfdf6a-8cfe-49fd-95e1-86b2e71115e7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "c0d36822-0f7a-480d-af44-db1ea433ad19", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "11ac2a26-b680-4e35-a192-dabc04e22cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d75b7e65-235d-45d7-9332-09dc776b1644", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "093fb4c2-87cd-43ca-8a73-3f03a976fd86", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5e086ebb-f46d-4bf2-8e6c-700327b5becb", + "status": "ready", + "queue_wait_seconds": 3.000233 + } + ], + "pending": 22, + "rss_bytes": 93237248, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 16, + "messages_unacknowledged": 4 + }, + "seconds": 7.998253458999898 + }, + { + "jobs": [ + { + "id": "cd3bd3f0-d5e3-4a01-996b-2396320c9435", + "status": "ready", + "queue_wait_seconds": 1.095668 + }, + { + "id": "e6ab4777-a2a7-4dd1-b744-ca0e3bb4b016", + "status": "ready", + "queue_wait_seconds": 1.101685 + }, + { + "id": "36496779-4f8b-400f-bbd5-1bd069aa9d80", + "status": "ready", + "queue_wait_seconds": 1.142645 + }, + { + "id": "9df958ad-72be-4244-8035-9bd3a4428362", + "status": "ready", + "queue_wait_seconds": 2.981451 + }, + { + "id": "ac26ac0d-9ec3-44e1-81f8-9988299342ae", + "status": "ready", + "queue_wait_seconds": 4.761922 + }, + { + "id": "6e32c3c3-4742-4fa5-95c9-ec99c08206f7", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "07881790-7f8f-4911-9231-d1611866cddd", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "1caf5d0c-4dd3-45c7-8d5f-cd46c5c2aa82", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "315beea2-9aa1-43a7-abbb-ad3f1898dffe", + "status": "ready", + "queue_wait_seconds": 1.070289 + }, + { + "id": "9c4c8f65-c6cc-44e5-9ffc-6e6a85c71fc2", + "status": "ready", + "queue_wait_seconds": 4.726148 + }, + { + "id": "2f3d268b-6a01-4401-b040-539c57720468", + "status": "ready", + "queue_wait_seconds": 4.724874 + }, + { + "id": "544ee42a-de19-47a1-ae80-2519b975c230", + "status": "ready", + "queue_wait_seconds": 6.566292 + }, + { + "id": "f92dd859-7dad-4903-a765-609f3909a5b5", + "status": "ready", + "queue_wait_seconds": 6.616724 + }, + { + "id": "938a80d0-e52f-4b29-b14b-425789ad0d48", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "8be53303-9c94-432a-8dac-d16555ba2ec5", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a448ff44-a164-4467-aa3a-10ab224dd952", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9cb3b8b5-02dd-4d7c-8fec-5617f65bdb57", + "status": "ready", + "queue_wait_seconds": 2.967999 + }, + { + "id": "139cb0e9-93a0-4a00-9b44-47d2ac2b770d", + "status": "ready", + "queue_wait_seconds": 4.707635 + }, + { + "id": "b198ed01-d7f6-49a7-b02e-a23e5263e66a", + "status": "ready", + "queue_wait_seconds": 6.554121 + }, + { + "id": "1924353a-e5cc-4351-a90d-3e97a5550ac2", + "status": "ready", + "queue_wait_seconds": 6.582615 + }, + { + "id": "adf5b8b6-06d8-4d15-a030-cdbcc1abb628", + "status": "processing", + "queue_wait_seconds": 8.157628 + }, + { + "id": "dc1cce7e-e39b-43dd-936e-16e9d8c2c6c0", + "status": "processing", + "queue_wait_seconds": 8.25348 + }, + { + "id": "5b407ae5-3918-415d-a898-57a680915bfb", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "ec05ff96-98c2-46df-9701-1589263a990d", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "0fefa5c8-4756-424a-96fb-59408ff618de", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "f5b4dab7-b060-4c06-a458-276f808fb113", + "status": "ready", + "queue_wait_seconds": 3.000177 + }, + { + "id": "3ebfdf6a-8cfe-49fd-95e1-86b2e71115e7", + "status": "processing", + "queue_wait_seconds": 8.224918 + }, + { + "id": "c0d36822-0f7a-480d-af44-db1ea433ad19", + "status": "processing", + "queue_wait_seconds": 8.321014 + }, + { + "id": "11ac2a26-b680-4e35-a192-dabc04e22cae", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "d75b7e65-235d-45d7-9332-09dc776b1644", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "093fb4c2-87cd-43ca-8a73-3f03a976fd86", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "5e086ebb-f46d-4bf2-8e6c-700327b5becb", + "status": "ready", + "queue_wait_seconds": 3.000233 + } + ], + "pending": 16, + "rss_bytes": 300544000, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 6, + "messages_unacknowledged": 4 + }, + "seconds": 12.100205249999817 + }, + { + "jobs": [ + { + "id": "cd3bd3f0-d5e3-4a01-996b-2396320c9435", + "status": "ready", + "queue_wait_seconds": 1.095668 + }, + { + "id": "3ebfdf6a-8cfe-49fd-95e1-86b2e71115e7", + "status": "ready", + "queue_wait_seconds": 8.224918 + }, + { + "id": "093fb4c2-87cd-43ca-8a73-3f03a976fd86", + "status": "ready", + "queue_wait_seconds": 11.554636 + }, + { + "id": "e6ab4777-a2a7-4dd1-b744-ca0e3bb4b016", + "status": "ready", + "queue_wait_seconds": 1.101685 + }, + { + "id": "36496779-4f8b-400f-bbd5-1bd069aa9d80", + "status": "ready", + "queue_wait_seconds": 1.142645 + }, + { + "id": "9df958ad-72be-4244-8035-9bd3a4428362", + "status": "ready", + "queue_wait_seconds": 2.981451 + }, + { + "id": "ac26ac0d-9ec3-44e1-81f8-9988299342ae", + "status": "ready", + "queue_wait_seconds": 4.761922 + }, + { + "id": "6e32c3c3-4742-4fa5-95c9-ec99c08206f7", + "status": "processing", + "queue_wait_seconds": 11.788202 + }, + { + "id": "07881790-7f8f-4911-9231-d1611866cddd", + "status": "processing", + "queue_wait_seconds": 12.726998 + }, + { + "id": "1caf5d0c-4dd3-45c7-8d5f-cd46c5c2aa82", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "315beea2-9aa1-43a7-abbb-ad3f1898dffe", + "status": "ready", + "queue_wait_seconds": 1.070289 + }, + { + "id": "9c4c8f65-c6cc-44e5-9ffc-6e6a85c71fc2", + "status": "ready", + "queue_wait_seconds": 4.726148 + }, + { + "id": "2f3d268b-6a01-4401-b040-539c57720468", + "status": "ready", + "queue_wait_seconds": 4.724874 + }, + { + "id": "544ee42a-de19-47a1-ae80-2519b975c230", + "status": "ready", + "queue_wait_seconds": 6.566292 + }, + { + "id": "f92dd859-7dad-4903-a765-609f3909a5b5", + "status": "ready", + "queue_wait_seconds": 6.616724 + }, + { + "id": "938a80d0-e52f-4b29-b14b-425789ad0d48", + "status": "processing", + "queue_wait_seconds": 11.774602 + }, + { + "id": "8be53303-9c94-432a-8dac-d16555ba2ec5", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "a448ff44-a164-4467-aa3a-10ab224dd952", + "status": "queued", + "queue_wait_seconds": null + }, + { + "id": "9cb3b8b5-02dd-4d7c-8fec-5617f65bdb57", + "status": "ready", + "queue_wait_seconds": 2.967999 + }, + { + "id": "139cb0e9-93a0-4a00-9b44-47d2ac2b770d", + "status": "ready", + "queue_wait_seconds": 4.707635 + }, + { + "id": "b198ed01-d7f6-49a7-b02e-a23e5263e66a", + "status": "ready", + "queue_wait_seconds": 6.554121 + }, + { + "id": "1924353a-e5cc-4351-a90d-3e97a5550ac2", + "status": "ready", + "queue_wait_seconds": 6.582615 + }, + { + "id": "adf5b8b6-06d8-4d15-a030-cdbcc1abb628", + "status": "ready", + "queue_wait_seconds": 8.157628 + }, + { + "id": "dc1cce7e-e39b-43dd-936e-16e9d8c2c6c0", + "status": "ready", + "queue_wait_seconds": 8.25348 + }, + { + "id": "5b407ae5-3918-415d-a898-57a680915bfb", + "status": "ready", + "queue_wait_seconds": 9.891625 + }, + { + "id": "ec05ff96-98c2-46df-9701-1589263a990d", + "status": "ready", + "queue_wait_seconds": 10.081661 + }, + { + "id": "0fefa5c8-4756-424a-96fb-59408ff618de", + "status": "ready", + "queue_wait_seconds": 11.320219 + }, + { + "id": "f5b4dab7-b060-4c06-a458-276f808fb113", + "status": "ready", + "queue_wait_seconds": 3.000177 + }, + { + "id": "c0d36822-0f7a-480d-af44-db1ea433ad19", + "status": "ready", + "queue_wait_seconds": 8.321014 + }, + { + "id": "11ac2a26-b680-4e35-a192-dabc04e22cae", + "status": "ready", + "queue_wait_seconds": 10.021064 + }, + { + "id": "d75b7e65-235d-45d7-9332-09dc776b1644", + "status": "ready", + "queue_wait_seconds": 10.153745 + }, + { + "id": "5e086ebb-f46d-4bf2-8e6c-700327b5becb", + "status": "ready", + "queue_wait_seconds": 3.000233 + } + ], + "pending": 6, + "rss_bytes": 132493312, + "concurrency": 4, + "prefetch": 4, + "queue": { + "name": "attachment.process.v1", + "messages_ready": 0, + "messages_unacknowledged": 3 + }, + "seconds": 15.211442291999901 + } + ], + "metric_errors": [], + "limitations": "Closed-loop; ready time is polling upper bound. RSS is sum of worker/container processes, shared pages may be counted twice. Sampling may miss peaks. Queue wait is first claim minus durable job creation." +} diff --git a/evidence/resilience.json b/evidence/resilience.json new file mode 100644 index 00000000..12fa6df7 --- /dev/null +++ b/evidence/resilience.json @@ -0,0 +1,359 @@ +[ + { + "mode": "persistence", + "returncode": 0, + "output": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.\n", + "before": { + "api-a": { + "id": "c3f5c47431e1", + "started_at": "2026-10-27T09:15:45.83814246Z", + "status": "running" + }, + "api-b": { + "id": "378998cb6e67", + "started_at": "2026-10-27T09:15:45.995940169Z", + "status": "running" + }, + "broker": { + "id": "15b4f76f796e", + "started_at": "2026-10-27T09:06:18.060378836Z", + "status": "running" + }, + "load_balancer": { + "id": "75a26443cd80", + "started_at": "2026-10-27T09:06:40.813835416Z", + "status": "running" + }, + "migrate": { + "id": "4752a39b2c74", + "started_at": "2026-10-27T09:10:25.395828423Z", + "status": "exited" + }, + "outbox": { + "id": "9b88bcba1d4a", + "started_at": "2026-10-27T09:10:07.674024346Z", + "status": "running" + }, + "postgres": { + "id": "e40600648a2a", + "started_at": "2026-10-27T09:06:17.86004142Z", + "status": "running" + }, + "reconcile": { + "id": "e97099d7b46b", + "started_at": "2026-10-27T09:10:07.611048596Z", + "status": "running" + }, + "s3": { + "id": "d9ccfafd22ca", + "started_at": "2026-10-27T09:06:17.939983378Z", + "status": "running" + }, + "s3-init": { + "id": "3e40ef3cf827", + "started_at": "2026-10-27T09:10:25.451664548Z", + "status": "exited" + }, + "worker": { + "id": "fc199d14fc86", + "started_at": "2026-10-27T09:15:45.913794835Z", + "status": "running" + } + }, + "after": { + "api-a": { + "id": "c3f5c47431e1", + "started_at": "2026-10-27T09:16:27.015462382Z", + "status": "running" + }, + "api-b": { + "id": "378998cb6e67", + "started_at": "2026-10-27T09:16:26.933943216Z", + "status": "running" + }, + "broker": { + "id": "15b4f76f796e", + "started_at": "2026-10-27T09:16:07.882420138Z", + "status": "running" + }, + "load_balancer": { + "id": "75a26443cd80", + "started_at": "2026-10-27T09:16:26.847483466Z", + "status": "running" + }, + "migrate": { + "id": "4752a39b2c74", + "started_at": "2026-10-27T09:16:57.480130299Z", + "status": "exited" + }, + "outbox": { + "id": "9b88bcba1d4a", + "started_at": "2026-10-27T09:16:29.790017884Z", + "status": "running" + }, + "postgres": { + "id": "e40600648a2a", + "started_at": "2026-10-27T09:16:06.788383595Z", + "status": "running" + }, + "reconcile": { + "id": "e97099d7b46b", + "started_at": "2026-10-27T09:16:29.811798967Z", + "status": "running" + }, + "s3": { + "id": "d9ccfafd22ca", + "started_at": "2026-10-27T09:16:09.773875388Z", + "status": "running" + }, + "s3-init": { + "id": "3e40ef3cf827", + "started_at": "2026-10-27T09:16:57.533971841Z", + "status": "exited" + }, + "worker": { + "id": "fc199d14fc86", + "started_at": "2026-10-27T09:16:56.780625216Z", + "status": "running" + } + } + }, + { + "mode": "failover", + "returncode": 0, + "output": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 113, \"failed_attempts\": 1, \"first_recovery_seconds\": 2.436}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.\n", + "before": { + "api-a": { + "id": "c3f5c47431e1", + "started_at": "2026-10-27T09:16:27.015462382Z", + "status": "running" + }, + "api-b": { + "id": "378998cb6e67", + "started_at": "2026-10-27T09:16:26.933943216Z", + "status": "running" + }, + "broker": { + "id": "15b4f76f796e", + "started_at": "2026-10-27T09:16:07.882420138Z", + "status": "running" + }, + "load_balancer": { + "id": "75a26443cd80", + "started_at": "2026-10-27T09:16:26.847483466Z", + "status": "running" + }, + "migrate": { + "id": "4752a39b2c74", + "started_at": "2026-10-27T09:16:57.480130299Z", + "status": "exited" + }, + "outbox": { + "id": "9b88bcba1d4a", + "started_at": "2026-10-27T09:16:29.790017884Z", + "status": "running" + }, + "postgres": { + "id": "e40600648a2a", + "started_at": "2026-10-27T09:16:06.788383595Z", + "status": "running" + }, + "reconcile": { + "id": "e97099d7b46b", + "started_at": "2026-10-27T09:16:29.811798967Z", + "status": "running" + }, + "s3": { + "id": "d9ccfafd22ca", + "started_at": "2026-10-27T09:16:09.773875388Z", + "status": "running" + }, + "s3-init": { + "id": "3e40ef3cf827", + "started_at": "2026-10-27T09:16:57.533971841Z", + "status": "exited" + }, + "worker": { + "id": "fc199d14fc86", + "started_at": "2026-10-27T09:16:56.780625216Z", + "status": "running" + } + }, + "after": { + "api-a": { + "id": "c3f5c47431e1", + "started_at": "2026-10-27T09:16:27.015462382Z", + "status": "exited" + }, + "api-b": { + "id": "378998cb6e67", + "started_at": "2026-10-27T09:16:26.933943216Z", + "status": "running" + }, + "broker": { + "id": "15b4f76f796e", + "started_at": "2026-10-27T09:16:07.882420138Z", + "status": "running" + }, + "load_balancer": { + "id": "75a26443cd80", + "started_at": "2026-10-27T09:16:26.847483466Z", + "status": "running" + }, + "migrate": { + "id": "4752a39b2c74", + "started_at": "2026-10-27T09:16:57.480130299Z", + "status": "exited" + }, + "outbox": { + "id": "9b88bcba1d4a", + "started_at": "2026-10-27T09:16:29.790017884Z", + "status": "running" + }, + "postgres": { + "id": "e40600648a2a", + "started_at": "2026-10-27T09:16:06.788383595Z", + "status": "running" + }, + "reconcile": { + "id": "e97099d7b46b", + "started_at": "2026-10-27T09:16:29.811798967Z", + "status": "running" + }, + "s3": { + "id": "d9ccfafd22ca", + "started_at": "2026-10-27T09:16:09.773875388Z", + "status": "running" + }, + "s3-init": { + "id": "3e40ef3cf827", + "started_at": "2026-10-27T09:16:57.533971841Z", + "status": "exited" + }, + "worker": { + "id": "fc199d14fc86", + "started_at": "2026-10-27T09:16:56.780625216Z", + "status": "running" + } + } + }, + { + "mode": "failover", + "returncode": 0, + "output": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 95, \"failed_attempts\": 2, \"first_recovery_seconds\": 4.675}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.\n", + "before": { + "api-a": { + "id": "c3f5c47431e1", + "started_at": "2026-10-27T09:17:21.770393088Z", + "status": "running" + }, + "api-b": { + "id": "378998cb6e67", + "started_at": "2026-10-27T09:16:26.933943216Z", + "status": "running" + }, + "broker": { + "id": "15b4f76f796e", + "started_at": "2026-10-27T09:16:07.882420138Z", + "status": "running" + }, + "load_balancer": { + "id": "75a26443cd80", + "started_at": "2026-10-27T09:16:26.847483466Z", + "status": "running" + }, + "migrate": { + "id": "4752a39b2c74", + "started_at": "2026-10-27T09:16:57.480130299Z", + "status": "exited" + }, + "outbox": { + "id": "9b88bcba1d4a", + "started_at": "2026-10-27T09:16:29.790017884Z", + "status": "running" + }, + "postgres": { + "id": "e40600648a2a", + "started_at": "2026-10-27T09:16:06.788383595Z", + "status": "running" + }, + "reconcile": { + "id": "e97099d7b46b", + "started_at": "2026-10-27T09:16:29.811798967Z", + "status": "running" + }, + "s3": { + "id": "d9ccfafd22ca", + "started_at": "2026-10-27T09:16:09.773875388Z", + "status": "running" + }, + "s3-init": { + "id": "3e40ef3cf827", + "started_at": "2026-10-27T09:16:57.533971841Z", + "status": "exited" + }, + "worker": { + "id": "fc199d14fc86", + "started_at": "2026-10-27T09:16:56.780625216Z", + "status": "running" + } + }, + "after": { + "api-a": { + "id": "c3f5c47431e1", + "started_at": "2026-10-27T09:17:21.770393088Z", + "status": "running" + }, + "api-b": { + "id": "378998cb6e67", + "started_at": "2026-10-27T09:16:26.933943216Z", + "status": "exited" + }, + "broker": { + "id": "15b4f76f796e", + "started_at": "2026-10-27T09:16:07.882420138Z", + "status": "running" + }, + "load_balancer": { + "id": "75a26443cd80", + "started_at": "2026-10-27T09:16:26.847483466Z", + "status": "running" + }, + "migrate": { + "id": "4752a39b2c74", + "started_at": "2026-10-27T09:16:57.480130299Z", + "status": "exited" + }, + "outbox": { + "id": "9b88bcba1d4a", + "started_at": "2026-10-27T09:16:29.790017884Z", + "status": "running" + }, + "postgres": { + "id": "e40600648a2a", + "started_at": "2026-10-27T09:16:06.788383595Z", + "status": "running" + }, + "reconcile": { + "id": "e97099d7b46b", + "started_at": "2026-10-27T09:16:29.811798967Z", + "status": "running" + }, + "s3": { + "id": "d9ccfafd22ca", + "started_at": "2026-10-27T09:16:09.773875388Z", + "status": "running" + }, + "s3-init": { + "id": "3e40ef3cf827", + "started_at": "2026-10-27T09:16:57.533971841Z", + "status": "exited" + }, + "worker": { + "id": "fc199d14fc86", + "started_at": "2026-10-27T09:16:56.780625216Z", + "status": "running" + } + } + } +] diff --git a/infra/filer.toml b/infra/filer.toml new file mode 100644 index 00000000..85b2149c --- /dev/null +++ b/infra/filer.toml @@ -0,0 +1,3 @@ +[leveldb2] +enabled = true +dir = "/data/filer" diff --git a/infra/haproxy.cfg b/infra/haproxy.cfg new file mode 100644 index 00000000..00f0b6c7 --- /dev/null +++ b/infra/haproxy.cfg @@ -0,0 +1,35 @@ +global + log stdout format raw local0 + +defaults + mode http + log global + option httplog + option forwardfor + timeout connect 1s + timeout client 30s + timeout server 10s + timeout check 2s + retries 2 + +resolvers docker + nameserver dns 127.0.0.11:53 + resolve_retries 3 + timeout resolve 1s + timeout retry 1s + hold valid 2s + hold obsolete 2s + +frontend http + bind :8080 + default_backend api + +backend api + balance roundrobin + option httpchk + http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host localhost + http-check expect status 200 + default-server check inter 2s fall 1 rise 2 resolvers docker resolve-prefer ipv4 init-addr libc,none + server api-a api-a:8080 + server api-b api-b:8080 + diff --git a/infra/rabbitmq-secret-entrypoint.sh b/infra/rabbitmq-secret-entrypoint.sh new file mode 100644 index 00000000..8ed866ff --- /dev/null +++ b/infra/rabbitmq-secret-entrypoint.sh @@ -0,0 +1,12 @@ +#!/bin/sh +set -eu + +# RabbitMQ 4 no longer supports *_FILE. Keep the password in a Docker +# secret and export it only inside the running container, never in Compose. +RABBITMQ_DEFAULT_PASS="$(cat /run/secrets/rabbitmq_password)" +if [ -z "$RABBITMQ_DEFAULT_PASS" ]; then + echo "RabbitMQ password secret must not be empty" >&2 + exit 1 +fi +export RABBITMQ_DEFAULT_PASS +exec /usr/local/bin/docker-entrypoint.sh "$@" diff --git a/migrations/001_initial.sql b/migrations/001_initial.sql new file mode 100644 index 00000000..c03d5fd9 --- /dev/null +++ b/migrations/001_initial.sql @@ -0,0 +1,37 @@ +CREATE TABLE users ( + id uuid PRIMARY KEY, + username varchar(32) NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'), + display_name varchar(100) NOT NULL CHECK (char_length(btrim(display_name)) > 0), + password_hash text NOT NULL, + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE chats ( + id uuid PRIMARY KEY, + title varchar(100) NOT NULL CHECK (char_length(btrim(title)) > 0), + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE chat_members ( + chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE, + user_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT, + PRIMARY KEY (chat_id, user_id) +); +CREATE INDEX chat_members_user_chat_idx ON chat_members (user_id, chat_id); + +CREATE TABLE messages ( + id uuid PRIMARY KEY, + chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE, + sender_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT, + text varchar(4000) NOT NULL CHECK (char_length(btrim(text)) > 0), + created_at timestamptz NOT NULL DEFAULT now() +); +CREATE INDEX messages_chat_created_id_idx ON messages (chat_id, created_at, id); + +CREATE TABLE sessions ( + token_hash char(64) PRIMARY KEY, + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + expires_at timestamptz NOT NULL, + created_at timestamptz NOT NULL DEFAULT now() +); +CREATE INDEX sessions_expiry_idx ON sessions (expires_at); diff --git a/migrations/002_history_cache_version.sql b/migrations/002_history_cache_version.sql new file mode 100644 index 00000000..7b7972c1 --- /dev/null +++ b/migrations/002_history_cache_version.sql @@ -0,0 +1 @@ +ALTER TABLE chats ADD COLUMN history_version bigint NOT NULL DEFAULT 0; diff --git a/migrations/003_idempotency.sql b/migrations/003_idempotency.sql new file mode 100644 index 00000000..7fcf6650 --- /dev/null +++ b/migrations/003_idempotency.sql @@ -0,0 +1,12 @@ +CREATE TABLE idempotency_records ( + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + method varchar(8) NOT NULL, + path text NOT NULL, + idempotency_key varchar(128) NOT NULL, + request_hash char(64) NOT NULL, + message_id uuid REFERENCES messages(id) ON DELETE RESTRICT, + created_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (user_id, method, path, idempotency_key) +); + +CREATE INDEX idempotency_records_created_at_idx ON idempotency_records (created_at); diff --git a/migrations/004_attachments.sql b/migrations/004_attachments.sql new file mode 100644 index 00000000..2fe0f9e2 --- /dev/null +++ b/migrations/004_attachments.sql @@ -0,0 +1,18 @@ +CREATE TABLE attachments ( + id uuid PRIMARY KEY, + chat_id uuid NOT NULL REFERENCES chats(id), + owner_id uuid NOT NULL REFERENCES users(id), + filename varchar(255) NOT NULL, + content_type text NOT NULL CHECK (content_type IN ('image/png', 'image/jpeg')), + size_bytes integer NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760), + sha256 char(64) NOT NULL, + object_key text NOT NULL UNIQUE, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now(), + UNIQUE (id, chat_id) +); +ALTER TABLE messages DROP CONSTRAINT messages_text_check; +ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}'; +ALTER TABLE messages ADD CONSTRAINT messages_content_check + CHECK (char_length(btrim(text)) > 0 OR cardinality(attachment_ids) > 0); +CREATE INDEX attachments_chat_idx ON attachments(chat_id); diff --git a/migrations/005_async_attachments.sql b/migrations/005_async_attachments.sql new file mode 100644 index 00000000..fa3bccaf --- /dev/null +++ b/migrations/005_async_attachments.sql @@ -0,0 +1,34 @@ +ALTER TABLE attachments + RENAME COLUMN object_key TO staging_key; +ALTER TABLE attachments + ADD COLUMN original_key text, + ADD COLUMN status text NOT NULL DEFAULT 'queued' + CHECK (status IN ('queued', 'processing', 'ready', 'failed')), + ADD COLUMN error_code text, + ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1, + ADD COLUMN processing_started_at timestamptz; + +CREATE TABLE attachment_variants ( + attachment_id uuid NOT NULL REFERENCES attachments(id) ON DELETE CASCADE, + pipeline_version integer NOT NULL, + name text NOT NULL CHECK (name IN ('thumbnail', 'crop', 'optimized')), + object_key text NOT NULL UNIQUE, + content_type text NOT NULL, + size_bytes integer NOT NULL CHECK (size_bytes > 0), + width integer NOT NULL CHECK (width > 0), + height integer NOT NULL CHECK (height > 0), + sha256 char(64) NOT NULL, + PRIMARY KEY (attachment_id, pipeline_version, name) +); + +CREATE TABLE attachment_jobs ( + job_id uuid PRIMARY KEY, + attachment_id uuid NOT NULL UNIQUE REFERENCES attachments(id) ON DELETE CASCADE, + pipeline_version integer NOT NULL DEFAULT 1, + attempt integer NOT NULL DEFAULT 1 CHECK (attempt >= 1), + status text NOT NULL DEFAULT 'queued' CHECK (status IN ('queued', 'processing', 'ready', 'failed')), + last_error text, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); +CREATE INDEX attachment_jobs_status_idx ON attachment_jobs(status, updated_at); diff --git a/migrations/006_level4_reliability.sql b/migrations/006_level4_reliability.sql new file mode 100644 index 00000000..3be954c0 --- /dev/null +++ b/migrations/006_level4_reliability.sql @@ -0,0 +1,17 @@ +ALTER TABLE attachment_jobs + ADD COLUMN next_attempt_at timestamptz NOT NULL DEFAULT now(), + ADD COLUMN lease_until timestamptz, + ADD COLUMN replayed_at timestamptz; +CREATE INDEX attachment_jobs_due_idx ON attachment_jobs(status, next_attempt_at); +CREATE INDEX attachment_jobs_lease_idx ON attachment_jobs(status, lease_until); + +CREATE TABLE attachment_outbox ( + id uuid PRIMARY KEY, + job_id uuid NOT NULL REFERENCES attachment_jobs(job_id) ON DELETE CASCADE, + payload jsonb NOT NULL, + created_at timestamptz NOT NULL DEFAULT now(), + published_at timestamptz, + attempts integer NOT NULL DEFAULT 0, + UNIQUE (job_id, published_at) +); +CREATE INDEX attachment_outbox_pending_idx ON attachment_outbox(created_at) WHERE published_at IS NULL; diff --git a/migrations/007_job_measurements.sql b/migrations/007_job_measurements.sql new file mode 100644 index 00000000..12fc1930 --- /dev/null +++ b/migrations/007_job_measurements.sql @@ -0,0 +1 @@ +ALTER TABLE attachment_jobs ADD COLUMN first_started_at timestamptz; diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 00000000..20c78ef2 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,15 @@ +[project] +name = "mtusi2026lab4" +version = "0.1.0" +description = "MTUSI high-load messenger, laboratory 4" +readme = "README.md" +requires-python = ">=3.12" +dependencies = [ + "argon2-cffi==25.1.0", + "asyncpg==0.31.0", + "boto3==1.43.115", + "fastapi==0.141.1", + "pillow==13.0.0", + "pika==1.3.2", + "uvicorn==0.40.0", +] diff --git a/scripts/audit.py b/scripts/audit.py new file mode 100644 index 00000000..5c15de21 --- /dev/null +++ b/scripts/audit.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +"""Sequential lab3-5 verification; capture actual output and exit status, fail fast.""" +from datetime import datetime, timezone +import argparse +import json +from pathlib import Path +import subprocess +import sys +import time + +ROOT = Path(__file__).resolve().parents[1] +DESTINATION = ROOT / "evidence/audit" +CHECKS = [ + ("contracts", ["make", "check", "validate", "PYTHON=.venv/bin/python"]), + ("compose", ["docker", "compose", "config", "--quiet"]), + ("smoke", ["make", "test"]), + ("attachments", ["make", "test-attachments"]), + ("idempotency", [sys.executable, "student-tests/idempotency_concurrency.py"]), + ("regressions", [sys.executable, "student-tests/review_regressions.py"]), + ("private-bucket", ["make", "test-private"]), + ("s3-permissions", ["make", "test-s3-permissions"]), + ("bootstrap", ["docker", "compose", "run", "--rm", "--no-deps", "s3-init"]), + ("upload-failover", ["make", "test-upload-failover"]), + ("public-failover", ["make", "test-failover", "ACTION_SCRIPT=scripts/stop-one.sh"]), + ("restore-replicas", ["docker", "compose", "up", "-d", "--wait", "api-a", "api-b", "load_balancer"]), + ("s3-outage", ["make", "test-s3-outage"]), + ("persistence", [sys.executable, "scripts/verify-persistence.py"]), + ("reconcile", ["make", "test-reconcile"]), + ("reconcile-audit", ["make", "test-reconcile-audit"]), + ("upload-memory", ["make", "test-upload-memory"]), + ("reconcile-benchmark", ["make", "test-reconcile-benchmark"]), + ("final-smoke", ["make", "test"]), + ("environment", [sys.executable, "scripts/record-environment.py"]), +] + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--resume-from', choices=[name for name, _ in CHECKS]) + args = parser.parse_args() + DESTINATION.mkdir(parents=True, exist_ok=True) + results = [] + checks = CHECKS + if args.resume_from: + offset = [name for name, _ in CHECKS].index(args.resume_from) + previous = json.loads((DESTINATION / 'results.json').read_text())['checks'] + results = previous[:offset] + if [item['name'] for item in results] != [name for name, _ in CHECKS[:offset]] or any(item['exit_code'] for item in results): + parser.error('resume requires successful evidence for every earlier step') + checks = CHECKS[offset:] + for name, command in checks: + print(f"RUN {name}", flush=True) + started = time.monotonic() + with (DESTINATION / f"{name}.txt").open("w") as stream: + result = subprocess.run(command, cwd=ROOT, stdout=stream, stderr=subprocess.STDOUT) + results.append({"name": name, "command": command, "exit_code": result.returncode, + "seconds": round(time.monotonic() - started, 3)}) + (DESTINATION / "results.json").write_text(json.dumps({ + "recorded_at_utc": datetime.now(timezone.utc).isoformat(), + "checks": results, "complete": len(results) == len(CHECKS), + }, indent=2) + "\n") + print(f"{'PASS' if result.returncode == 0 else 'FAIL'} {name}: exit={result.returncode}", flush=True) + if result.returncode: + print((DESTINATION / f"{name}.txt").read_text()[-8000:], flush=True) + return result.returncode + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/init-secrets.py b/scripts/init-secrets.py new file mode 100644 index 00000000..a4160497 --- /dev/null +++ b/scripts/init-secrets.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python3 +"""Create local credentials without printing or overwriting existing values.""" +import json +from pathlib import Path +import secrets + +root = Path(__file__).resolve().parents[1] / "secrets" +root.mkdir(mode=0o700, exist_ok=True) + + +def write(name, value): + path = root / name + if path.exists(): + return path.read_text().strip() + with path.open("x") as stream: + path.chmod(0o600) + stream.write(value + "\n") + return value + + +password = write("postgres_password", secrets.token_hex(32)) +write("database_url", f"postgresql://messenger:{password}@postgres:5432/messenger") +write("cursor_secret", secrets.token_hex(32)) +admin_key = write("s3_admin_key", secrets.token_hex(16)) +admin_secret = write("s3_admin_secret", secrets.token_hex(32)) +app_key = write("s3_access_key_id", secrets.token_hex(16)) +app_secret = write("s3_secret_access_key", secrets.token_hex(32)) +rabbitmq_password = write("rabbitmq_password", secrets.token_urlsafe(32)) +write("broker_url", f"amqp://messenger:{rabbitmq_password}@broker:5672/%2F") +config_path = root / "s3.json" +config_path.write_text(json.dumps({"identities": [ + {"name": "bootstrap", "credentials": [{"accessKey": admin_key, "secretKey": admin_secret}], + "actions": ["Admin", "Read", "Write", "List", "Tagging"]}, + {"name": "api", "credentials": [{"accessKey": app_key, "secretKey": app_secret}], + "actions": ["Read:messenger-attachments", "Write:messenger-attachments", "List:messenger-attachments"]}, +]}, indent=2) + "\n") +config_path.chmod(0o600) +print("Local secrets ready; existing values preserved.") diff --git a/scripts/record-environment.py b/scripts/record-environment.py new file mode 100644 index 00000000..961dc2fd --- /dev/null +++ b/scripts/record-environment.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +"""Sanitized environment and source fingerprint for local test evidence.""" +from datetime import datetime, timezone +from hashlib import sha256 +import json +from pathlib import Path +import platform +import subprocess + +root = Path(__file__).resolve().parents[1] + + +def command(*args): + return subprocess.check_output(args, cwd=root, text=True).strip() + + +paths = [root / name for name in ('Dockerfile', 'compose.yaml', 'pyproject.toml', 'uv.lock', 'Makefile')] +for directory in ('src', 'migrations', 'infra', 'scripts', 'student-tests'): + paths.extend(path for path in (root / directory).rglob('*') + if path.is_file() and '__pycache__' not in path.parts) +report = { + 'recorded_at_utc': datetime.now(timezone.utc).isoformat(), + 'base_commit': command('git', 'rev-parse', 'HEAD'), + 'branch': command('git', 'branch', '--show-current'), + 'note': 'Results apply to this working tree, not only the base commit. Per-profile load settings are recorded in real-c*-p*-s*.json.', + 'host': platform.platform(), + 'docker': json.loads(command('docker', 'info', '--format', + '{"server":"{{.ServerVersion}}","cpus":{{.NCPU}},"memory_bytes":{{.MemTotal}}}')), + 'compose_version': command('docker', 'compose', 'version', '--short'), + 'sources_sha256': {str(path.relative_to(root)): sha256(path.read_bytes()).hexdigest() for path in sorted(paths)}, + 'services': {}, +} +for service in ('postgres', 's3', 'broker', 'worker', 'outbox', 'reconcile', 'api-a', 'api-b', 'load_balancer'): + container = command('docker', 'compose', 'ps', '-q', service) + info = json.loads(command('docker', 'inspect', container))[0] + report['services'][service] = { + 'image': info['Config']['Image'], 'image_id': info['Image'], + 'state': info['State']['Status'], 'health': info['State'].get('Health', {}).get('Status'), + 'memory_limit_bytes': info['HostConfig']['Memory'], + 'nano_cpus_limit': info['HostConfig']['NanoCpus'], + 'volumes': [mount['Name'] for mount in info['Mounts'] if mount['Type'] == 'volume'], + } +(root / 'evidence/environment.json').write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n') +print('PASS: sanitized environment and source hashes saved') diff --git a/scripts/restart.sh b/scripts/restart.sh new file mode 100755 index 00000000..958b700d --- /dev/null +++ b/scripts/restart.sh @@ -0,0 +1,10 @@ +#!/bin/sh +set -eu +cd "$(dirname "$0")/.." + +# Deliberately preserves named volumes and test data. +docker compose restart postgres s3 broker +docker compose up -d --wait postgres s3 broker +docker compose run --rm --no-deps s3-init +docker compose restart worker outbox reconcile api-a api-b load_balancer +docker compose up -d --wait worker outbox reconcile api-a api-b load_balancer diff --git a/scripts/run-track1-profiles.py b/scripts/run-track1-profiles.py new file mode 100644 index 00000000..ba3f1405 --- /dev/null +++ b/scripts/run-track1-profiles.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python3 +"""Run real, sequential Docker profiles; restore normal settings on exit.""" +import os +from pathlib import Path +import subprocess +import sys + +ROOT = Path(__file__).resolve().parents[1] + + +def main(): + try: + for workers in (1, 2, 4): + env = dict(os.environ, WORKER_CONCURRENCY=str(workers), + WORKER_PREFETCH=str(workers), MAX_PENDING_JOBS='32') + subprocess.run(['docker', 'compose', 'up', '-d', '--no-deps', '--wait', + 'worker', 'api-a', 'api-b'], cwd=ROOT, env=env, check=True) + for size in (512, 2400): + subprocess.run([sys.executable, 'scripts/track1.py', + '--run-name', f'real-c{workers}-p{workers}-s{size}', + '--workers', str(workers), '--prefetch', str(workers), + '--max-pending', '32', '--images', '160', + '--concurrency', '16', '--size', str(size), + '--require-backpressure'], cwd=ROOT, check=True) + finally: + subprocess.run(['docker', 'compose', 'up', '-d', '--no-deps', '--wait', + 'worker', 'api-a', 'api-b'], cwd=ROOT, check=True) + + +if __name__ == '__main__': + main() diff --git a/scripts/stop-one.sh b/scripts/stop-one.sh new file mode 100755 index 00000000..befd6cfb --- /dev/null +++ b/scripts/stop-one.sh @@ -0,0 +1,18 @@ +#!/bin/sh +set -eu + +: "${TARGET_INSTANCE_ID:?test provides target API instance alias}" +cd "$(dirname "$0")/.." + +for service in api-a api-b; do + container="$(docker compose ps -q "$service")" + [ -n "$container" ] || continue + hostname="$(docker inspect --format '{{.Config.Hostname}}' "$container")" + if [ "$hostname" = "$TARGET_INSTANCE_ID" ]; then + docker compose stop "$service" + exit 0 + fi +done + +printf '%s\n' 'TARGET_INSTANCE_ID does not match a running API replica' >&2 +exit 2 diff --git a/scripts/track1.py b/scripts/track1.py new file mode 100755 index 00000000..d72240d5 --- /dev/null +++ b/scripts/track1.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +"""Closed-loop upload burst with concurrent readiness observations and drain proof.""" +from concurrent.futures import ThreadPoolExecutor, as_completed +from io import BytesIO +import json +import math +from pathlib import Path +import subprocess +import sys +import threading +import time + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tests")) +from client import Client, Failure, arguments, check, run + + +def percentile(values, point): + values = sorted(values) + return round(values[max(0, math.ceil(len(values) * point) - 1)] * 1000, 3) if values else None + + +def command_json(*args): + result = subprocess.run(args, cwd=ROOT, text=True, capture_output=True, timeout=20) + if result.returncode: + raise Failure("metrics command failed: " + args[0]) + try: + return json.loads(result.stdout) + except ValueError: + raise Failure("metrics command returned invalid JSON") from None + + +def metrics(chat_id): + return command_json("docker", "compose", "exec", "-T", "worker", + "/app/.venv/bin/python", "-m", "src.pipeline_metrics", chat_id) + + +def broker_depth(): + rows = command_json("docker", "compose", "exec", "-T", "broker", + "rabbitmqctl", "list_queues", "name", "messages_ready", + "messages_unacknowledged", "--formatter", "json") + return next(row for row in rows if row["name"] == "attachment.process.v1") + + +def container_info(service): + result = subprocess.run(['docker', 'compose', 'ps', '-q', service], cwd=ROOT, + capture_output=True, text=True, timeout=10, check=True) + ids = result.stdout.split() + check(len(ids) == 1, 'expected exactly one container: ' + service) + return command_json('docker', 'inspect', ids[0])[0] + + +def main(): + def configure(p): + p.add_argument("--run-name", required=True) + p.add_argument("--workers", type=int, required=True) + p.add_argument("--prefetch", type=int, required=True) + p.add_argument("--max-pending", type=int, required=True) + p.add_argument("--images", type=int, default=40) + p.add_argument("--concurrency", type=int, default=8) + p.add_argument("--size", type=int, default=1600) + p.add_argument("--max-ready-seconds", type=float, default=180) + p.add_argument("--require-backpressure", action="store_true") + p.add_argument("--output") + args = arguments(__doc__, configure) + check(1 <= args.images <= 1000 and 1 <= args.concurrency <= 128, "invalid workload") + check(1 <= args.size <= 6000 and 0 < args.max_ready_seconds <= 3600, "invalid size/deadline") + check(args.workers > 0 and args.prefetch > 0 and args.max_pending > 0, "invalid profile") + from PIL import Image + # Reproducible, nonuniform synthetic image; no real personal data. + row = bytes(v for x in range(args.size) for v in ((x * 13) % 256, (x * 37) % 256, (x * 71) % 256)) + image = Image.frombytes("RGB", (args.size, args.size), row * args.size) + stream = BytesIO() + image.save(stream, "PNG") + data = stream.getvalue() + client = Client(args.base_url, args.ca_file, timeout=30) + alice, bob, _, chat = client.bootstrap() + baseline = metrics(chat["id"]) + check(baseline["concurrency"] == args.workers and baseline["prefetch"] == args.prefetch, + "worker settings do not match requested profile") + check(baseline["pending"] == 0, "use an idle isolated stack") + for service in ("api-a", "api-b"): + env = dict(item.split('=', 1) for item in container_info(service)['Config']['Env']) + check(int(env['MAX_PENDING_JOBS']) == args.max_pending, + 'running API MAX_PENDING_JOBS does not match profile') + resources = container_info('worker')['HostConfig'] + path = "/api/v1/chats/" + chat["id"] + "/attachments?filename=track1.png" + pending, records, samples, errors = {}, [], [], [] + lock, finished = threading.Lock(), threading.Event() + started = time.monotonic() + def observe(): + local = Client(args.base_url, args.ca_file, timeout=5) + while not finished.wait(.1): + with lock: + batch = list(pending.items()) + for key, record in batch: + if finished.is_set(): + break + try: + att = local.call("GET", "/api/v1/attachments/" + key, actor=bob, schema="Attachment")["json"] + if att["status"] in ("ready", "failed"): + with lock: + record["terminal"] = att["status"] + record["terminal_at"] = time.monotonic() + pending.pop(key, None) + except Failure: + continue + def sample(): + while not finished.is_set(): + try: + sample = metrics(chat["id"]) + sample["queue"] = broker_depth() + sample["seconds"] = time.monotonic() - started + with lock: + samples.append(sample) + except (Failure, OSError, subprocess.TimeoutExpired) as exc: + with lock: + errors.append(type(exc).__name__) + if finished.wait(1): + break + observer = threading.Thread(target=observe, daemon=True) + sampler = threading.Thread(target=sample, daemon=True) + observer.start() + sampler.start() + def upload(index): + local = Client(args.base_url, args.ca_file, timeout=30) + before = time.monotonic() + try: + response = local.call("POST", path, (202, 429), actor=alice, raw=data, + headers={"Content-Type": "image/png"}, schema="Attachment") + received = time.monotonic() + record = {"index": index, "status": response["status"], "upload_seconds": received - before} + if response["status"] == 202: + record.update(id=response["json"]["id"], accepted_at=received) + with lock: + records.append(record) + if response["status"] == 202: + pending[record["id"]] = record + except Failure: + with lock: + records.append({"index": index, "status": "transport_or_http_error"}) + try: + with ThreadPoolExecutor(max_workers=args.concurrency) as pool: + list(pool.map(upload, range(args.images))) + upload_finished = time.monotonic() + deadline = upload_finished + args.max_ready_seconds + while time.monotonic() < deadline: + with lock: + empty = not pending + if empty: + break + time.sleep(.1) + finally: + measured_end = time.monotonic() + finished.set() + observer.join(timeout=10) + sampler.join(timeout=45) + elapsed = measured_end - started + final = metrics(chat["id"]) + queue = broker_depth() + for _ in range(20): + if queue["messages_ready"] == 0 and queue["messages_unacknowledged"] == 0: + break + time.sleep(.25) + queue = broker_depth() + accepted = [r for r in records if r["status"] == 202] + ready = [r for r in accepted if r.get("terminal") == "ready"] + latencies = [r["terminal_at"] - r["accepted_at"] for r in ready] + waits = [r["queue_wait_seconds"] for r in final["jobs"] if r["queue_wait_seconds"] is not None] + drained = (bool(accepted) and len(ready) == len(accepted) and final["pending"] == 0 + and queue["messages_ready"] == 0 and queue["messages_unacknowledged"] == 0) + recovery = False + if drained: + try: + att, _ = client.attachment(alice, chat) + client.wait_ready(bob, att, wait_seconds=args.max_ready_seconds) + recovery = True + except Failure: + pass + rejected = sum(r["status"] == 429 for r in records) + report = { + "run_name": args.run_name, + "settings": {"workers": baseline["concurrency"], "prefetch": baseline["prefetch"], + "max_pending": args.max_pending, + "worker_limits": {key: resources[key] for key in ('Memory', 'NanoCpus')}}, + "workload": {"images": args.images, "concurrency": args.concurrency, + "size": args.size, "bytes": len(data), "type": "closed-loop burst"}, + "results": {"accepted": len(accepted), "rejected_429": rejected, "ready": len(ready), + "failed": sum(r.get("terminal") == "failed" for r in accepted), + "timed_out": len(pending), "elapsed_seconds": elapsed, + "ready_per_second": len(ready) / elapsed, + "ready_latency_ms": {str(p): percentile(latencies, p) for p in (.5, .95, .99)}, + "queue_wait_ms": {str(p): percentile(waits, p) for p in (.5, .95, .99)}, + "max_sampled_rss_bytes": max((s["rss_bytes"] for s in samples), default=None), + "max_sampled_pending": max((s["pending"] for s in samples), default=None), + "sample_count": len(samples), + "drained": drained, "recovery_upload_ready": recovery}, + "records": [{k: v for k, v in r.items() if k not in ("accepted_at", "terminal_at")} | + ({"ready_seconds": r["terminal_at"] - r["accepted_at"]} if "terminal_at" in r else {}) + for r in records], + "samples": samples, "metric_errors": errors, + "limitations": "Closed-loop; ready time is polling upper bound. RSS is sum of worker/container processes, shared pages may be counted twice. Sampling may miss peaks. Queue wait is first claim minus durable job creation.", + } + target = Path(args.output or ROOT / "evidence" / (args.run_name + ".json")) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(report, indent=2) + "\n") + print(json.dumps(report["results"], indent=2)) + check(not errors and bool(samples), "incomplete metrics; result saved but not valid evidence") + check(all(s['pending'] <= args.max_pending for s in samples), "admission limit exceeded") + check(all(r["status"] in (202, 429) for r in records), "unexpected HTTP/transport failures") + check(drained and recovery, "pipeline did not drain/recover; see saved result") + if args.require_backpressure: + check(rejected > 0, "backpressure was not observed") + + +if __name__ == "__main__": + run(main) diff --git a/scripts/verify-persistence.py b/scripts/verify-persistence.py new file mode 100644 index 00000000..5bfc5a44 --- /dev/null +++ b/scripts/verify-persistence.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python3 +"""Public persistence test plus proof of actual container restarts.""" +import json +from pathlib import Path +import subprocess +import sys + +ROOT = Path(__file__).resolve().parents[1] + + +def snapshot(): + result = {} + for service in ('postgres', 's3', 'api-a', 'api-b', 'load_balancer'): + container = subprocess.check_output(['docker', 'compose', 'ps', '-q', service], cwd=ROOT, text=True).strip() + started = subprocess.check_output(['docker', 'inspect', '--format', '{{.State.StartedAt}}', container], text=True).strip() + result[service] = {'container': container[:12], 'started_at': started} + return result + + +before = snapshot() +print('Before:', json.dumps(before, sort_keys=True), flush=True) +subprocess.run([sys.executable, 'tests/resilience.py', '--mode', 'persistence', + '--action-script', 'scripts/restart.sh'], cwd=ROOT, check=True) +after = snapshot() +print('After:', json.dumps(after, sort_keys=True), flush=True) +assert all(before[key]['started_at'] != after[key]['started_at'] for key in before) +print('PASS: PostgreSQL, S3, both API replicas and LB actually restarted; volumes preserved') diff --git a/scripts/verify-resilience.py b/scripts/verify-resilience.py new file mode 100644 index 00000000..02ccd000 --- /dev/null +++ b/scripts/verify-resilience.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +"""Sequential public tests with safe container-state evidence; no credentials.""" +import json +from pathlib import Path +import subprocess +import sys + +ROOT = Path(__file__).resolve().parents[1] + + +def states(): + ids = subprocess.check_output(['docker', 'compose', 'ps', '-aq'], text=True).split() + rows = json.loads(subprocess.check_output(['docker', 'inspect', *ids], text=True)) + return {r['Config']['Labels']['com.docker.compose.service']: + {'id': r['Id'][:12], 'started_at': r['State']['StartedAt'], 'status': r['State']['Status']} + for r in rows} + + +def main(): + results, stopped = [], set() + try: + for mode in ['persistence'] + ['failover'] * 6: + before = states() + hook = 'restart' if mode == 'persistence' else 'stop-one' + result = subprocess.run([sys.executable, 'tests/resilience.py', '--mode', mode, + '--action-script', 'scripts/' + hook + '.sh'], + text=True, capture_output=True, timeout=200) + after = states() + results.append(dict(mode=mode, returncode=result.returncode, + output=result.stdout + result.stderr, before=before, after=after)) + (ROOT / 'evidence/resilience.json').write_text(json.dumps(results, indent=2) + '\n') + print(result.stdout + result.stderr, flush=True) + assert result.returncode == 0, 'public resilience failed' + if mode == 'persistence': + for service in ('postgres', 's3', 'broker', 'worker', 'outbox', 'api-a', 'api-b'): + assert before[service]['started_at'] != after[service]['started_at'], service + else: + stopped.update(s for s in ('api-a', 'api-b') if after[s]['status'] == 'exited') + subprocess.run(['docker', 'compose', 'up', '-d', '--no-deps', '--wait', 'api-a', 'api-b'], check=True) + if len(stopped) == 2: + break + assert len(stopped) == 2, 'did not exercise both API replicas' + finally: + subprocess.run(['docker', 'compose', 'up', '-d', '--no-deps', '--wait', 'api-a', 'api-b'], check=True) + + +if __name__ == '__main__': + main() diff --git a/src/__init__.py b/src/__init__.py new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/src/__init__.py @@ -0,0 +1 @@ + diff --git a/src/attachments.py b/src/attachments.py new file mode 100644 index 00000000..ea496eeb --- /dev/null +++ b/src/attachments.py @@ -0,0 +1,141 @@ +import asyncio +from hashlib import sha256 +import os +from tempfile import TemporaryFile +import unicodedata +from uuid import UUID, uuid4 +import warnings + +from fastapi import APIRouter, Depends, Query, Request, Response +from PIL import Image, UnidentifiedImageError +from starlette.concurrency import run_in_threadpool +from starlette.requests import ClientDisconnect + +from .errors import ApiError +from .models import Attachment +from .routes import get_current_user, get_store +from .storage.postgres import PostgresStore + +router = APIRouter() +Image.MAX_IMAGE_PIXELS = 40000000 +warnings.filterwarnings("error", category=Image.DecompressionBombWarning) + + +def inspect_image(file, content_type, max_pixels): + try: + file.seek(0) + with Image.open(file, formats=("PNG", "JPEG")) as image: + if image.width * image.height > max_pixels: + raise ApiError(413, "payload_too_large", "Слишком много пикселей") + actual = {"PNG": "image/png", "JPEG": "image/jpeg"}[image.format] + if actual != content_type or getattr(image, "n_frames", 1) != 1: + raise ApiError(415, "unsupported_media_type", "Нужен обычный PNG или JPEG") + image.verify() + # verify checks PNG CRC; load also catches truncated pixel streams/JPEGs. + file.seek(0) + with Image.open(file, formats=("PNG", "JPEG")) as image: + image.load() + except (Image.DecompressionBombError, Image.DecompressionBombWarning) as error: + raise ApiError(413, "payload_too_large", "Слишком много пикселей") from error + except (UnidentifiedImageError, OSError, ValueError, SyntaxError) as error: + raise ApiError(415, "unsupported_media_type", "Повреждённое изображение") from error + file.seek(0) + + +@router.post("/api/v1/chats/{chat_id}/attachments", response_model=Attachment, status_code=202) +async def upload(chat_id: UUID, request: Request, filename: str = Query(...), + user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)): + await store.get_chat_for_member(chat_id, user["id"]) + if (not 1 <= len(filename) <= 255 or not filename.strip() or + any(c in "/\\" or unicodedata.category(c) in {"Cc", "Cs"} for c in filename)): + raise ApiError(400, "invalid_request", "Некорректное имя файла") + content_type = request.headers.get("content-type", "").lower() + if content_type not in {"image/png", "image/jpeg"}: + raise ApiError(415, "unsupported_media_type", "Нужен PNG или JPEG") + s3 = request.app.state.s3 + limit = s3.settings.max_bytes + if "content-length" in request.headers: + try: + length = int(request.headers["content-length"]) + if length < 0: + raise ValueError() + except ValueError: + raise ApiError(400, "invalid_request", "Некорректная длина") from None + if length > limit: + raise ApiError(413, "payload_too_large", "Файл больше лимита") + try: + timeout = int(os.environ.get("UPLOAD_TIMEOUT_SECONDS", "30")) + if not 1 <= timeout <= 120: + raise ApiError(503, "unavailable", "Некорректный лимит времени загрузки") + async with asyncio.timeout(timeout): + async with request.app.state.upload_slot: + # Temporary scratch space only; deleted on every exit, never used for downloads. + with TemporaryFile() as file: + size = 0 + digest = sha256() + async for chunk in request.stream(): + size += len(chunk) + if size > limit: + raise ApiError(413, "payload_too_large", "Файл больше лимита") + digest.update(chunk) + await run_in_threadpool(file.write, chunk) + await run_in_threadpool(inspect_image, file, content_type, s3.settings.max_pixels) + attachment_id, job_id = uuid4(), uuid4() + key = f"staging/{attachment_id}/original" + async with store.attachment_transaction() as connection: + try: + max_pending = int(os.environ.get("MAX_PENDING_JOBS", "100")) + except ValueError as error: + raise ApiError(503, "unavailable", "Некорректный лимит очереди") from error + if max_pending < 1: + raise ApiError(503, "unavailable", "Некорректный лимит очереди") + await store.ensure_attachment_queue_capacity(connection, max_pending) + await s3.put(key, file, size, content_type) + if os.environ.get("UPLOAD_FAULT_AFTER_S3_PUT") == "1": + raise ApiError(503, "unavailable", "Инъецирован отказ после S3 PUT") + attachment = await store.create_attachment( + attachment_id, chat_id, user["id"], filename, content_type, + size, digest.hexdigest(), key, job_id, connection=connection, + ) + # Fault-injection for level-4 proof: durable outbox must recover this commit. + if os.environ.get("OUTBOX_FAULT_AFTER_COMMIT") == "1": + os._exit(70) + attachment["variants"] = [] + return attachment + except ClientDisconnect: + raise ApiError(400, "invalid_request", "Загрузка прервана") from None + except TimeoutError: + raise ApiError(503, "unavailable", "Время загрузки истекло") from None + + +@router.get("/api/v1/attachments/{attachment_id}", response_model=Attachment) +async def metadata(attachment_id: UUID, user: dict = Depends(get_current_user), + store: PostgresStore = Depends(get_store)): + return await store.attachment_for_member(attachment_id, user["id"]) + + +@router.get("/api/v1/attachments/{attachment_id}/content") +async def download(attachment_id: UUID, request: Request, + variant: str = Query("original", pattern="^(original|thumbnail|crop|optimized)$"), + user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)): + attachment = await store.attachment_for_member(attachment_id, user["id"]) + if attachment["status"] != "ready": + raise ApiError(409, "conflict", "Вложение ещё не готово") + if variant == "original": + data = await request.app.state.s3.download(attachment) + content_type = attachment["content_type"] + else: + matching = [item for item in attachment["variants"] if item["name"] == variant] + if not matching: + raise ApiError(404, "not_found", "Вариант не найден") + item = matching[0] + object_key = await store.pool.fetchval( + 'SELECT object_key FROM attachment_variants WHERE attachment_id=$1 AND pipeline_version=$2 AND name=$3', + attachment_id, attachment['pipeline_version'], variant, + ) + data = await request.app.state.s3.download_key( + object_key, item["size_bytes"], item["sha256"] + ) + content_type = item["content_type"] + return Response(data, media_type=content_type, + headers={"X-Content-Type-Options": "nosniff", "Cache-Control": "private, no-store"}) diff --git a/src/bootstrap_s3.py b/src/bootstrap_s3.py new file mode 100644 index 00000000..28f2c8b0 --- /dev/null +++ b/src/bootstrap_s3.py @@ -0,0 +1,51 @@ +"""Idempotent private bucket bootstrap; executed with separate admin credentials.""" +import time +from uuid import uuid4 +from botocore.exceptions import BotoCoreError, ClientError +from .storage.s3 import S3Settings, S3Store + + +def main(): + store = S3Store(S3Settings.load()) + try: + for attempt in range(60): + try: + store.client.list_buckets() + break + except (BotoCoreError, ClientError): + if attempt == 59: + raise RuntimeError("S3 startup timed out") from None + time.sleep(1) + bucket = store.settings.bucket + try: + store.client.head_bucket(Bucket=bucket) + except ClientError as error: + if str(error.response["Error"]["Code"]) not in {"404", "NoSuchBucket", "NotFound"}: + raise + store.client.create_bucket(Bucket=bucket) + # No anonymous identity; revoke any previously configured public grants/policy. + store.client.put_bucket_acl(Bucket=bucket, ACL="private") + try: + store.client.delete_bucket_policy(Bucket=bucket) + except ClientError as error: + if error.response["Error"]["Code"] != "NoSuchBucketPolicy": + raise + # The master health endpoint can become available before a volume accepts + # writes after a restart. Verify the actual data path before APIs start. + probe_key = f"_bootstrap/write-probe-{uuid4()}" + for attempt in range(30): + try: + store.client.put_object(Bucket=bucket, Key=probe_key, Body=b"ready") + store.client.delete_object(Bucket=bucket, Key=probe_key) + break + except (BotoCoreError, ClientError): + if attempt == 29: + raise RuntimeError("S3 write path startup timed out") from None + time.sleep(1) + print("PASS: private S3 bucket ready") + finally: + store.client.close() + + +if __name__ == "__main__": + main() diff --git a/src/broker.py b/src/broker.py new file mode 100644 index 00000000..4cd980d1 --- /dev/null +++ b/src/broker.py @@ -0,0 +1,83 @@ +"""Small durable RabbitMQ adapter; broker messages contain no attachment bytes.""" +import json +import os +from pathlib import Path + +import pika +from starlette.concurrency import run_in_threadpool + +from .errors import ApiError + +QUEUE = "attachment.process.v1" +DLQ = "attachment.process.v1.dlq" + + +def broker_url() -> str: + path = os.environ.get("BROKER_URL_FILE") + value = Path(path).read_text(encoding="utf-8").strip() if path else os.environ.get("BROKER_URL", "") + if not value.startswith(("amqp://", "amqps://")): + raise ValueError("BROKER_URL или BROKER_URL_FILE должен содержать AMQP URL") + return value + + +def _connection() -> pika.BlockingConnection: + connection = pika.BlockingConnection(pika.URLParameters(broker_url())) + channel = connection.channel() + channel.queue_declare(queue=QUEUE, durable=True) + channel.queue_declare(queue=DLQ, durable=True) + return connection + + +def _publish(payload: dict) -> None: + connection = _connection() + try: + channel = connection.channel() + channel.confirm_delivery() + channel.basic_publish( + exchange="", routing_key=QUEUE, + body=json.dumps(payload, separators=(",", ":")).encode("utf-8"), + properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent, content_type="application/json"), + mandatory=True, + ) + finally: + connection.close() + + +async def publish(payload: dict) -> None: + try: + await run_in_threadpool(_publish, payload) + except (pika.exceptions.AMQPError, OSError, ValueError) as error: + raise ApiError(503, "unavailable", "Брокер недоступен") from error + + +async def publish_dlq(payload: dict) -> None: + payload = {**payload, "failure": payload.get('failure', 'retry_exhausted')} + try: + await run_in_threadpool(_publish_to, DLQ, payload) + except (pika.exceptions.AMQPError, OSError, ValueError) as error: + raise RuntimeError("DLQ is unavailable") from error + + +def _publish_to(queue: str, payload: dict) -> None: + connection = _connection() + try: + channel = connection.channel() + channel.confirm_delivery() + channel.basic_publish( + exchange="", routing_key=queue, body=json.dumps(payload, separators=(",", ":")).encode("utf-8"), + properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent, content_type="application/json"), + mandatory=True, + ) + finally: + connection.close() + + +async def ready() -> None: + connection = None + try: + connection = await run_in_threadpool(_connection) + except (pika.exceptions.AMQPError, OSError, ValueError) as error: + raise ApiError(503, "unavailable", "Брокер недоступен") from error + finally: + if connection is not None: + await run_in_threadpool(connection.close) diff --git a/src/config.py b/src/config.py new file mode 100644 index 00000000..320063a0 --- /dev/null +++ b/src/config.py @@ -0,0 +1,125 @@ +from dataclasses import dataclass +import os +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +_CONFIGURATION_KEYS = { + "APP_HOST", + "APP_PORT", + "APP_ENV", + "DATABASE_URL", + "DATABASE_URL_FILE", + "SESSION_TTL_SECONDS", + "IDEMPOTENCY_TTL_SECONDS", + "INSTANCE_ID", + "CURSOR_SECRET", + "CURSOR_SECRET_FILE", + "S3_ENDPOINT", "S3_REGION", "S3_BUCKET", + "S3_ACCESS_KEY_ID", "S3_ACCESS_KEY_ID_FILE", + "S3_SECRET_ACCESS_KEY", "S3_SECRET_ACCESS_KEY_FILE", + "BROKER_URL", "BROKER_URL_FILE", + "MAX_UPLOAD_BYTES", "MAX_IMAGE_PIXELS", + "UPLOAD_CONCURRENCY", "UPLOAD_TIMEOUT_SECONDS", "UPLOAD_FAULT_AFTER_S3_PUT", + "WORKER_CONCURRENCY", "JOB_MAX_ATTEMPTS", "JOB_TIMEOUT_SECONDS", "STAGING_RETENTION_SECONDS", + "WORKER_PREFETCH", "MAX_PENDING_JOBS", "RENDER_MEMORY_MB", +} + + +@dataclass(frozen=True, slots=True) +class Settings: + """Проверенная конфигурация запуска HTTP-сервера.""" + + host: str + port: int + environment: str + database_url: str + session_ttl_seconds: int + idempotency_ttl_seconds: int + instance_id: str + cursor_secret: bytes + + +def load_settings(env_file: Path = ROOT / ".env") -> Settings: + """Читает разрешённые ключи .env, не заменяя явно переданное окружение.""" + + if env_file.is_file(): + for raw_line in env_file.read_text(encoding="utf-8").splitlines(): + line = raw_line.strip() + if not line or line.startswith("#"): + continue + key, separator, value = line.partition("=") + if not separator or key not in _CONFIGURATION_KEYS: + continue + value = value.strip() + if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}: + value = value[1:-1] + os.environ.setdefault(key, value) + + def value(name: str, default: str = "") -> str: + return os.environ.get(name, default).strip() + + def number(name: str, default: int) -> int: + try: + result = int(value(name, str(default))) + except ValueError as error: + raise ValueError(f"{name} должен быть целым числом") from error + if result < 1: + raise ValueError(f"{name} должен быть положительным") + return result + + def url(name: str, file_name: str, prefixes: tuple[str, ...]) -> str: + result = value(name) + path = value(file_name) + if path: + try: + result = Path(path).read_text(encoding="utf-8").strip() + except OSError as error: + raise ValueError(f"{file_name} недоступен") from error + if result and not result.startswith(prefixes): + raise ValueError(f"{name} содержит неверный URL") + return result + + def secret(name: str, file_name: str) -> bytes: + result = value(name) + path = value(file_name) + if path: + try: + result = Path(path).read_text(encoding="utf-8").strip() + except OSError as error: + raise ValueError(f"{file_name} недоступен") from error + encoded = result.encode("utf-8") + if len(encoded) < 32: + raise ValueError(f"{name} должен содержать не менее 32 байт") + return encoded + + host = value("APP_HOST", "0.0.0.0") + environment = value("APP_ENV", "lab") + if not host: + raise ValueError("APP_HOST не может быть пустым") + if not environment: + raise ValueError("APP_ENV не может быть пустым") + port = number("APP_PORT", 8080) + if port > 65535: + raise ValueError("APP_PORT должен находиться в диапазоне 1..65535") + database_url = url( + "DATABASE_URL", "DATABASE_URL_FILE", ("postgresql://", "postgres://") + ) + if not database_url: + raise ValueError("Нужен PostgreSQL DATABASE_URL или DATABASE_URL_FILE") + instance_id = value("INSTANCE_ID") or os.uname().nodename + if not instance_id: + raise ValueError("INSTANCE_ID не может быть пустым") + idempotency_ttl_seconds = number("IDEMPOTENCY_TTL_SECONDS", 86400) + if idempotency_ttl_seconds < 86400: + raise ValueError("IDEMPOTENCY_TTL_SECONDS должен быть не менее 86400") + return Settings( + host=host, + port=port, + environment=environment, + database_url=database_url, + session_ttl_seconds=number("SESSION_TTL_SECONDS", 3600), + idempotency_ttl_seconds=idempotency_ttl_seconds, + instance_id=instance_id, + cursor_secret=secret("CURSOR_SECRET", "CURSOR_SECRET_FILE"), + ) diff --git a/src/cursor.py b/src/cursor.py new file mode 100644 index 00000000..5dd8eab3 --- /dev/null +++ b/src/cursor.py @@ -0,0 +1,81 @@ +import base64 +from datetime import UTC, datetime +import hashlib +import hmac +import json +from uuid import UUID + + +_secret: bytes | None = None + + +class CursorError(ValueError): + """Недопустимый, повреждённый или применённый не к тому ресурсу курсор.""" + + +def configure_secret(secret: bytes) -> None: + """Устанавливает общий для всех реплик ключ подписи курсоров.""" + global _secret + if len(secret) < 32: + raise ValueError("cursor secret must contain at least 32 bytes") + _secret = secret + + +def signing_secret() -> bytes: + if _secret is None: + raise RuntimeError("cursor secret has not been configured") + return _secret + + +def _encode_bytes(value: bytes) -> str: + return base64.urlsafe_b64encode(value).decode().rstrip("=") + + +def _decode_bytes(value: str) -> bytes: + try: + return base64.urlsafe_b64decode(value + "=" * (-len(value) % 4)) + except (ValueError, UnicodeEncodeError) as error: + raise CursorError from error + + +def encode_cursor(scope: str, created_at: datetime, item_id: str) -> str: + payload = json.dumps( + { + "scope": scope, + "created_at": created_at.astimezone(UTC) + .isoformat() + .replace("+00:00", "Z"), + "id": item_id, + }, + separators=(",", ":"), + sort_keys=True, + ).encode() + signature = hmac.new(signing_secret(), payload, hashlib.sha256).digest() + return _encode_bytes(payload) + "." + _encode_bytes(signature) + + +def decode_cursor(cursor: str, expected_scope: str) -> tuple[datetime, str]: + try: + encoded_payload, encoded_signature = cursor.split(".") + payload = _decode_bytes(encoded_payload) + signature = _decode_bytes(encoded_signature) + except ValueError as error: + raise CursorError from error + + expected_signature = hmac.new(signing_secret(), payload, hashlib.sha256).digest() + if not hmac.compare_digest(signature, expected_signature): + raise CursorError + + try: + data = json.loads(payload) + if set(data) != {"scope", "created_at", "id"}: + raise CursorError + if data["scope"] != expected_scope: + raise CursorError + item_id = str(UUID(data["id"])) + created_at = datetime.fromisoformat(data["created_at"].replace("Z", "+00:00")) + if created_at.tzinfo is None: + raise CursorError + return created_at.astimezone(UTC), item_id + except (TypeError, ValueError, KeyError) as error: + raise CursorError from error diff --git a/src/errors.py b/src/errors.py new file mode 100644 index 00000000..258d6996 --- /dev/null +++ b/src/errors.py @@ -0,0 +1,12 @@ +"""Ошибки, которые преобразуются в контрактные HTTP-ответы.""" + + +class ApiError(Exception): + + def __init__( + self, status_code: int, code: str, message: str, headers: dict[str, str] | None = None + ) -> None: + self.status_code = status_code + self.code = code + self.message = message + self.headers = headers or {} diff --git a/src/main.py b/src/main.py new file mode 100644 index 00000000..f3559329 --- /dev/null +++ b/src/main.py @@ -0,0 +1,172 @@ +import asyncio +import logging +import os +import sys +import time +from contextlib import asynccontextmanager +from uuid import uuid4 + +from fastapi import FastAPI, Request +from fastapi.exceptions import RequestValidationError +import uvicorn +from starlette.responses import JSONResponse, Response +from starlette.exceptions import HTTPException + +from .errors import ApiError +from .cursor import configure_secret +from .models import ErrorDetails, ErrorResponse +from .config import load_settings +from .routes import router +from .attachments import router as attachment_router +from .storage.s3 import S3Settings, S3Store +from .storage.postgres import PostgresStore + + +def logger(name: str) -> logging.Logger: + result = logging.getLogger(name) + if result.handlers: + return result + handler = logging.StreamHandler(sys.stdout) + handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")) + result.addHandler(handler) + result.setLevel(logging.INFO) + result.propagate = False + return result + + +access_logger = logger("messenger.access") + + +@asynccontextmanager +async def lifespan(application: FastAPI): + settings = load_settings() + configure_secret(settings.cursor_secret) + store = PostgresStore(settings.database_url) + await store.connect() + application.state.store = store + application.state.settings = settings + application.state.s3 = S3Store(S3Settings.load()) + concurrency = int(os.environ.get("UPLOAD_CONCURRENCY", "1")) + if not 1 <= concurrency <= 8: + raise ValueError("UPLOAD_CONCURRENCY must be 1..8") + application.state.upload_slot = asyncio.Semaphore(concurrency) + application.state.accepting_requests = True + try: + yield + finally: + application.state.accepting_requests = False + application.state.s3.client.close() + await store.close() + + +app = FastAPI( + title="MTUSI Messenger", + version="1.4.0", + redoc_url=None, + lifespan=lifespan, +) + + +@app.middleware("http") +async def add_request_id(request: Request, call_next) -> Response: + """Добавляет ID запроса и пишет безопасную строку access-лога.""" + + request.state.request_id = str(uuid4()) + started_at = time.monotonic() + status_code = 500 + try: + response = await call_next(request) + status_code = response.status_code + response.headers["X-Request-Id"] = request.state.request_id + response.headers["X-Instance-Id"] = request.app.state.settings.instance_id + return response + finally: + access_logger.info( + "request_id=%s method=%s path=%s status=%s elapsed_ms=%.3f", + request.state.request_id, + request.method, + request.url.path, + status_code, + (time.monotonic() - started_at) * 1000, + ) + + +def error_response( + request: Request, status_code: int, code: str, message: str, headers: dict[str, str] | None = None +) -> JSONResponse: + request_id = request.state.request_id + body = ErrorResponse( + error=ErrorDetails(code=code, message=message, request_id=request_id) + ) + # Неожиданная ошибка обрабатывается за пределами HTTP-middleware. + # Поэтому её ответ тоже должен сам получить заголовок из контракта. + return JSONResponse( + status_code=status_code, + content=body.model_dump(), + headers={ + "X-Request-Id": request_id, + "X-Instance-Id": request.app.state.settings.instance_id, + **(headers or {}), + }, + ) + + +@app.exception_handler(ApiError) +async def handle_api_error(request: Request, error: ApiError) -> JSONResponse: + """Преобразует контролируемую ошибку API в стандартный JSON-ответ.""" + + return error_response(request, error.status_code, error.code, error.message, error.headers) + + +@app.exception_handler(RequestValidationError) +async def handle_validation_error( + request: Request, error: RequestValidationError +) -> JSONResponse: + """Преобразует ошибки JSON и Pydantic в обязательный ответ 400.""" + + return error_response(request, 400, "invalid_request", "Некорректный запрос") + + +@app.exception_handler(HTTPException) +async def handle_http_error(request: Request, error: HTTPException) -> JSONResponse: + """Включает ошибки декодирования тела, возникающие до Pydantic.""" + codes = { + 400: "invalid_request", + 401: "unauthorized", + 403: "forbidden", + 404: "not_found", + 405: "method_not_allowed", + 503: "unavailable", + } + code = codes.get(error.status_code, "internal_error") + return error_response( + request, error.status_code, code, + "Некорректный запрос" if error.status_code < 500 else "Ошибка сервера", + error.headers, + ) + + +@app.exception_handler(Exception) +async def handle_unexpected_error(request: Request, error: Exception) -> JSONResponse: + """Не раскрывает внутренние детали при непредвиденной ошибке приложения.""" + + return error_response(request, 500, "internal_error", "Внутренняя ошибка сервера") + + +app.include_router(router) +app.include_router(attachment_router) + + +def main() -> None: + settings = load_settings() + uvicorn.run( + app, + host=settings.host, + port=settings.port, + access_log=False, + timeout_graceful_shutdown=8, + ) + + +if __name__ == "__main__": + main() diff --git a/src/migrate.py b/src/migrate.py new file mode 100644 index 00000000..4376e8d6 --- /dev/null +++ b/src/migrate.py @@ -0,0 +1,18 @@ +import asyncio + +from .config import load_settings +from .storage.postgres import PostgresStore + + +async def run() -> None: + settings = load_settings() + store = PostgresStore(settings.database_url) + await store.connect() + try: + await store.migrate() + finally: + await store.close() + + +if __name__ == "__main__": + asyncio.run(run()) diff --git a/src/models.py b/src/models.py new file mode 100644 index 00000000..15f38109 --- /dev/null +++ b/src/models.py @@ -0,0 +1,151 @@ +from datetime import datetime +from uuid import UUID + +from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator + +class CreateUserRequest(BaseModel): + model_config = ConfigDict(extra="forbid", strict=True) + + username: str = Field(min_length=3, max_length=32, pattern=r"^[a-z0-9_]{3,32}$") + display_name: str = Field(min_length=1, max_length=100) + password: str = Field(min_length=12, max_length=128) + + @field_validator("display_name") + @classmethod + def display_name_must_not_be_whitespace(cls, value: str) -> str: + """Не допускает обязательное имя, состоящее только из пробелов.""" + + if not value.strip(): + raise ValueError("display_name не может состоять только из пробелов") + return value + + +class CreateChatRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + + title: str = Field(min_length=1, max_length=100) + member_ids: list[UUID] = Field(min_length=1, max_length=100) + + @field_validator("title") + @classmethod + def title_must_not_be_whitespace(cls, value: str) -> str: + """Не допускает обязательное название, состоящее только из пробелов.""" + + if not value.strip(): + raise ValueError("title не может состоять только из пробелов") + return value + + +class ChatPageResponse(BaseModel): + model_config = ConfigDict(extra="forbid") + + items: list["Chat"] + next_cursor: str | None + + +class CreateMessageRequest(BaseModel): + model_config = ConfigDict(extra="forbid", strict=True) + + text: str = Field(max_length=4000) + attachment_ids: list[str] = Field(default_factory=list, max_length=10) + + @field_validator("attachment_ids") + @classmethod + def valid_ids(cls, values: list[str]) -> list[str]: + normalized = [str(UUID(value)) for value in values] + if len(normalized) != len(set(normalized)): + raise ValueError("Повторяющиеся attachment_ids") + return normalized + + @model_validator(mode="after") + def has_content(self): + if not self.text.strip() and not self.attachment_ids: + raise ValueError("Сообщение без содержимого") + return self + + +class MessagePageResponse(BaseModel): + model_config = ConfigDict(extra="forbid") + + items: list["Message"] + next_cursor: str | None + + +class ErrorDetails(BaseModel): + model_config = ConfigDict(extra="forbid") + + code: str + message: str + request_id: str + + +class ErrorResponse(BaseModel): + model_config = ConfigDict(extra="forbid") + + error: ErrorDetails + + +class User(BaseModel): + model_config = ConfigDict(extra="forbid") + + id: UUID + username: str + display_name: str + created_at: datetime + + +class Chat(BaseModel): + model_config = ConfigDict(extra="forbid") + + id: UUID + title: str + member_ids: list[UUID] + created_at: datetime + + +class Message(BaseModel): + model_config = ConfigDict(extra="forbid") + + id: UUID + chat_id: UUID + sender_id: UUID + text: str + attachment_ids: list[UUID] = Field(default_factory=list) + created_at: datetime + + +class Session(BaseModel): + model_config = ConfigDict(extra="forbid") + + token: str = Field(min_length=32) + token_type: str = "Bearer" + expires_at: datetime + user: User + + +class Attachment(BaseModel): + model_config = ConfigDict(extra="ignore") + + id: UUID + chat_id: UUID + owner_id: UUID + filename: str + content_type: str + size_bytes: int + sha256: str + status: str + variants: list["Variant"] = Field(default_factory=list) + error_code: str | None = None + created_at: datetime + updated_at: datetime + + +class Variant(BaseModel): + model_config = ConfigDict(extra="forbid") + + name: str + content_type: str + size_bytes: int + width: int + height: int + sha256: str diff --git a/src/outbox.py b/src/outbox.py new file mode 100644 index 00000000..60f00997 --- /dev/null +++ b/src/outbox.py @@ -0,0 +1,33 @@ +"""Transactional-outbox dispatcher; safe to run as one or more replicas.""" +import asyncio +import logging +import os + +from .broker import publish +from .config import load_settings +from .storage.postgres import PostgresStore + +logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s outbox %(message)s") + + +async def run() -> None: + settings = load_settings() + store = PostgresStore(settings.database_url) + await store.connect() + interval = max(0.1, float(os.environ.get("OUTBOX_DISPATCH_INTERVAL_SECONDS", "1"))) + try: + while True: + try: + await store.schedule_due_jobs(int(os.environ.get('JOB_MAX_ATTEMPTS', '5'))) + for event in await store.due_outbox_events(): + await publish(event["payload"]) + await store.mark_outbox_published(event["id"]) + except Exception: + logging.exception("outbox dispatch failed; durable event remains pending") + await asyncio.sleep(interval) + finally: + await store.close() + + +if __name__ == "__main__": + asyncio.run(run()) diff --git a/src/pipeline_metrics.py b/src/pipeline_metrics.py new file mode 100644 index 00000000..93cde0b3 --- /dev/null +++ b/src/pipeline_metrics.py @@ -0,0 +1,37 @@ +"""Operator-only metrics for one synthetic chat; never exposes credentials.""" +import asyncio +import json +import os +from pathlib import Path +import sys +from uuid import UUID +from .config import load_settings +from .storage.postgres import PostgresStore + + +async def main(chat): + store = PostgresStore(load_settings().database_url) + await store.connect() + try: + rows = await store.pool.fetch('SELECT a.id, a.status, extract(epoch from (j.first_started_at-j.created_at))::float8 AS queue_wait_seconds FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id WHERE a.chat_id=$1', chat) + jobs = [dict(row, id=str(row['id'])) for row in rows] + pending = await store.pool.fetchval("SELECT count(*) FROM attachment_jobs WHERE status IN ('queued','processing')") + finally: + await store.close() + rss = 0 + for path in Path('/proc').glob('[0-9]*/status'): + try: + if int(path.parent.name) == os.getpid(): + continue + for line in path.read_text().splitlines(): + if line.startswith('VmRSS:'): + rss += int(line.split()[1]) * 1024 + except (OSError, ValueError): + pass + print(json.dumps({'jobs': jobs, 'pending': pending, 'rss_bytes': rss, + 'concurrency': int(os.environ['WORKER_CONCURRENCY']), + 'prefetch': int(os.environ['WORKER_PREFETCH'])})) + + +if __name__ == '__main__': + asyncio.run(main(UUID(sys.argv[1]))) diff --git a/src/reconcile.py b/src/reconcile.py new file mode 100644 index 00000000..1b60bb87 --- /dev/null +++ b/src/reconcile.py @@ -0,0 +1,104 @@ +"""Safely reconcile attachment metadata and original S3 objects. + +The command is dry-run by default. It only deletes unreferenced objects older +than a configured grace period; missing objects are reported and never cause a +metadata delete. That keeps a delayed PostgreSQL commit and active attachment +records safe from a maintenance run. +""" +import argparse +import asyncio +from datetime import datetime, timedelta, timezone +import json +import os +from time import perf_counter + +from .config import load_settings +from .storage.postgres import PostgresStore +from .storage.s3 import S3Settings, S3Store + + +async def run(args) -> dict: + settings = load_settings() + db = PostgresStore(settings.database_url) + s3 = S3Store(S3Settings.load()) + try: + await db.connect() + async with db.attachment_transaction(maintenance=True): + return await scan(args, db, s3) + finally: + s3.client.close() + await db.close() + + +async def scan(args, db, s3) -> dict: + # Age is measured at scan start, not after a potentially slow inventory. + cutoff = datetime.now(timezone.utc) - timedelta(seconds=args.grace_seconds) + started = perf_counter() + objects, s3_pages = {}, 0 + for prefix in ('staging/', 'originals/', 'variants/'): + entries, pages = await s3.list_with_pages(prefix) + objects.update(entries) + s3_pages += pages + listed_at = perf_counter() + metadata = await db.attachment_keys() + metadata_at = perf_counter() + orphan = sorted(key for key, modified in objects.items() + if key not in metadata and modified <= cutoff) + fresh = sorted(key for key, modified in objects.items() + if key not in metadata and modified > cutoff) + missing = sorted({str(attachment_id): key for key, attachment_id in metadata.items() + if key not in objects}.items()) + deleted = [] + selected = set(args.object_key) if args.object_key else None + if args.apply: + for key in orphan: + if selected is not None and key not in selected: + continue + await s3.delete(key) + deleted.append(key) + finished = perf_counter() + return { + "mode": "apply" if args.apply else "dry-run", + "grace_seconds": args.grace_seconds, + "s3_objects": len(objects), + "metadata_rows": len(metadata), + "orphan_candidates": orphan, + "fresh_unreferenced": fresh, + "missing_objects": [{"attachment_id": item[0], "object_key": item[1]} for item in missing], + "deleted": deleted, + "selected_keys": sorted(selected) if selected is not None else None, + "scan_metrics": { + "s3_list_pages": s3_pages, + "s3_list_seconds": round(listed_at - started, 6), + "postgres_metadata_seconds": round(metadata_at - listed_at, 6), + "total_seconds": round(finished - started, 6), + }, + } + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--apply", action="store_true", help="delete only aged orphan objects") + parser.add_argument("--grace-seconds", type=int, default=int(os.environ.get('STAGING_RETENTION_SECONDS', '86400'))) + parser.add_argument('--loop', action='store_true') + parser.add_argument("--object-key", action="append", default=[], + help="restrict apply to these exact keys; repeatable, inventory stays complete") + args = parser.parse_args() + if not 0 <= args.grace_seconds <= 7 * 24 * 3600: + parser.error("--grace-seconds must be 0..604800") + if any(not key.startswith(('originals/', 'staging/', 'variants/')) for key in args.object_key): + parser.error('unsupported object prefix') + if args.loop: + import time + while True: + try: + print(json.dumps(asyncio.run(run(args)), ensure_ascii=False, sort_keys=True)) + except Exception: + print('cleanup deferred: dependency unavailable', flush=True) + time.sleep(60) + else: + print(json.dumps(asyncio.run(run(args)), ensure_ascii=False, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/src/render.py b/src/render.py new file mode 100644 index 00000000..dbb1d1df --- /dev/null +++ b/src/render.py @@ -0,0 +1,27 @@ +"""Isolated image renderer. Parent enforces wall time and kills this process.""" +import os +import pickle +import sys + + +def main(): + if sys.platform == 'linux': + import resource + limit = int(os.environ.get('RENDER_MEMORY_MB', '768')) * 1024 * 1024 + resource.setrlimit(resource.RLIMIT_AS, (limit, limit)) + from .worker import variants + from PIL import UnidentifiedImageError, Image + try: + data = sys.stdin.buffer.read(10485761) + if len(data) > 10485760: + raise ValueError('bytes limit') + result = variants(data, int(os.environ.get('MAX_IMAGE_PIXELS', '40000000'))) + pickle.dump(result, sys.stdout.buffer) + except MemoryError: + sys.exit(21) + except (ValueError, OSError, SyntaxError, UnidentifiedImageError, Image.DecompressionBombError): + sys.exit(20) + + +if __name__ == '__main__': + main() diff --git a/src/replay.py b/src/replay.py new file mode 100644 index 00000000..2360cb5e --- /dev/null +++ b/src/replay.py @@ -0,0 +1,24 @@ +"""Operator-only safe replay for a failed attachment job.""" +import argparse +import asyncio +from uuid import UUID + +from .config import load_settings +from .storage.postgres import PostgresStore + + +async def run(attachment_id: UUID) -> None: + store = PostgresStore(load_settings().database_url) + await store.connect() + try: + if not await store.replay_failed_attachment(attachment_id): + raise SystemExit("attachment is not in failed state") + finally: + await store.close() + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("attachment_id", type=UUID) + args = parser.parse_args() + asyncio.run(run(args.attachment_id)) diff --git a/src/routes.py b/src/routes.py new file mode 100644 index 00000000..fcd2e74f --- /dev/null +++ b/src/routes.py @@ -0,0 +1,185 @@ +import base64 +import re +from datetime import datetime +from uuid import UUID + +from fastapi import APIRouter, Depends, Header, Query, Request, Response + +from .cursor import CursorError, decode_cursor, encode_cursor +from .errors import ApiError +from .storage.postgres import PostgresStore +from .broker import ready as broker_ready +from .models import ( + Chat, + ChatPageResponse, + CreateChatRequest, + CreateMessageRequest, + CreateUserRequest, + Message, + MessagePageResponse, + Session, + User, +) + +router = APIRouter() +BEARER_CHALLENGE = {"WWW-Authenticate": 'Bearer realm="messenger"'} +BASIC_CHALLENGE = {"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'} +IDEMPOTENCY_KEY = re.compile(r"^[A-Za-z0-9._:-]{1,128}$") + +def get_store(request: Request) -> PostgresStore: + return request.app.state.store + + +async def get_current_user( + authorization: str | None = Header(default=None), + store: PostgresStore = Depends(get_store), +) -> dict: + if authorization is None or not authorization.startswith("Bearer "): + raise ApiError(401, "unauthorized", "Требуется Bearer-сессия", BEARER_CHALLENGE) + token = authorization[7:] + if not token or not token.isascii() or token.strip() != token: + raise ApiError(401, "unauthorized", "Требуется Bearer-сессия", BEARER_CHALLENGE) + user = await store.user_for_token(token) + if user is None: + raise ApiError(401, "unauthorized", "Недействительная сессия", BEARER_CHALLENGE) + return user + +def parse_basic(authorization: str | None) -> tuple[str, str]: + if authorization is None or not authorization.startswith("Basic "): + raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE) + try: + decoded = base64.b64decode(authorization[6:], validate=True) + username_bytes, password_bytes = decoded.split(b":", 1) + username = username_bytes.decode("ascii") + password = password_bytes.decode("utf-8") + except (UnicodeDecodeError, ValueError) as error: + raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE) from error + if not username or not (12 <= len(password) <= 128): + raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE) + return username, password + + +def cursor_position(cursor: str | None, scope: str) -> tuple[datetime, UUID] | None: + if cursor is None: + return None + try: + created_at, item_id = decode_cursor(cursor, scope) + return created_at, UUID(item_id) + except (CursorError, ValueError) as error: + raise ApiError(400, "invalid_request", "Некорректный cursor") from error + + +def next_page_cursor(rows: list[dict], limit: int, scope: str) -> str | None: + if len(rows) <= limit: + return None + last = rows[limit - 1] + return encode_cursor(scope, last["created_at"], str(last["id"])) + +@router.get("/health/live") +async def liveness() -> dict[str, str]: + return {"status": "ok"} + +@router.get("/health/ready") +async def readiness(request: Request, store: PostgresStore = Depends(get_store)) -> dict[str, str]: + if not request.app.state.accepting_requests: + raise ApiError(503, "unavailable", "API завершает работу") + await store.ready() + await request.app.state.s3.ready() + await broker_ready() + return {"status": "ok"} + +@router.post("/api/v1/users", response_model=User, status_code=201) +async def register_user( + payload: CreateUserRequest, store: PostgresStore = Depends(get_store) +) -> dict: + return await store.create_user(payload.username, payload.display_name, payload.password) + +@router.post("/api/v1/auth/sessions", response_model=Session, status_code=201) +async def create_session( + request: Request, + authorization: str | None = Header(default=None), + store: PostgresStore = Depends(get_store), +) -> dict: + if await request.body(): + raise ApiError(400, "invalid_request", "Тело запроса не требуется") + username, password = parse_basic(authorization) + user = await store.authenticate(username, password) + if user is None: + raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE) + return await store.create_session(user, request.app.state.settings.session_ttl_seconds) + +@router.delete("/api/v1/auth/sessions/current", status_code=204) +async def revoke_session( + authorization: str | None = Header(default=None), + store: PostgresStore = Depends(get_store), + user: dict = Depends(get_current_user), +) -> Response: + del user + await store.revoke_session(authorization[7:]) + return Response(status_code=204) + +@router.get("/api/v1/users/me", response_model=User) +async def current_user(user: dict = Depends(get_current_user)) -> dict: + return user + +@router.post("/api/v1/chats", response_model=Chat, status_code=201) +async def create_chat( + payload: CreateChatRequest, + user: dict = Depends(get_current_user), + store: PostgresStore = Depends(get_store), +) -> dict: + return await store.create_chat(user["id"], payload.title, payload.member_ids) + +@router.get("/api/v1/chats", response_model=ChatPageResponse) +async def list_chats( + user: dict = Depends(get_current_user), + store: PostgresStore = Depends(get_store), + limit: int = Query(default=50, ge=1, le=100), + cursor: str | None = Query(default=None, min_length=1), +) -> ChatPageResponse: + scope = "chats:" + str(user["id"]) + rows = await store.list_chats_for_member(user["id"], cursor_position(cursor, scope), limit) + page = rows[:limit] + return ChatPageResponse(items=page, next_cursor=next_page_cursor(rows, limit, scope)) + +@router.get("/api/v1/chats/{chat_id}", response_model=Chat) +async def get_chat( + chat_id: UUID, + user: dict = Depends(get_current_user), + store: PostgresStore = Depends(get_store), +) -> dict: + return await store.get_chat_for_member(chat_id, user["id"]) + +@router.post("/api/v1/chats/{chat_id}/messages", response_model=Message, status_code=201) +async def create_message( + chat_id: UUID, + payload: CreateMessageRequest, + request: Request, + idempotency_key: str | None = Header(default=None, alias="Idempotency-Key"), + user: dict = Depends(get_current_user), + store: PostgresStore = Depends(get_store), +) -> dict: + if idempotency_key is None or IDEMPOTENCY_KEY.fullmatch(idempotency_key) is None: + raise ApiError(400, "invalid_request", "Некорректный Idempotency-Key") + return await store.create_message_idempotent( + chat_id, + user["id"], + payload.text, + idempotency_key, + request.app.state.settings.idempotency_ttl_seconds, + [UUID(value) for value in payload.attachment_ids], + ) + +@router.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePageResponse) +async def list_messages( + chat_id: UUID, + user: dict = Depends(get_current_user), + store: PostgresStore = Depends(get_store), + limit: int = Query(default=50, ge=1, le=100), + cursor: str | None = Query(default=None, min_length=1), +) -> MessagePageResponse: + scope = "messages:" + str(chat_id) + after = cursor_position(cursor, scope) + rows = await store.list_messages_for_member(chat_id, user["id"], after, limit) + page = rows[:limit] + return MessagePageResponse(items=page, next_cursor=next_page_cursor(rows, limit, scope)) diff --git a/src/storage/__init__.py b/src/storage/__init__.py new file mode 100644 index 00000000..e2e04193 --- /dev/null +++ b/src/storage/__init__.py @@ -0,0 +1,2 @@ +"""Слой in-memory хранения для приложения первой лабораторной работы.""" +"""Persistent storage adapters.""" diff --git a/src/storage/postgres.py b/src/storage/postgres.py new file mode 100644 index 00000000..d83f93dc --- /dev/null +++ b/src/storage/postgres.py @@ -0,0 +1,531 @@ +from datetime import datetime, timedelta, timezone +from contextlib import asynccontextmanager +from hashlib import sha256 +import json +from pathlib import Path +from secrets import token_urlsafe +from uuid import UUID, uuid4 + +import asyncpg +from argon2 import PasswordHasher +from argon2.exceptions import InvalidHashError, VerificationError + +from ..errors import ApiError + + +ROOT = Path(__file__).resolve().parents[2] +MIGRATIONS = ROOT / "migrations" +PASSWORD_HASHER = PasswordHasher() +DB_ERRORS = (asyncpg.PostgresError, asyncpg.InterfaceError, OSError, TimeoutError) + + +class PostgresStore: + def __init__(self, database_url: str) -> None: + self._database_url = database_url + self._pool: asyncpg.Pool | None = None + + async def connect(self) -> None: + try: + self._pool = await asyncpg.create_pool( + self._database_url, min_size=1, max_size=10, command_timeout=5 + ) + except DB_ERRORS as error: + raise RuntimeError("PostgreSQL is unavailable") from error + + async def close(self) -> None: + if self._pool is not None: + await self._pool.close() + + @property + def pool(self) -> asyncpg.Pool: + if self._pool is None: + raise RuntimeError("PostgreSQL pool has not been initialized") + return self._pool + + async def migrate(self) -> None: + async with self.pool.acquire() as connection: + await connection.execute( + "CREATE TABLE IF NOT EXISTS schema_migrations (version text PRIMARY KEY, applied_at timestamptz NOT NULL DEFAULT now())" + ) + for migration in sorted(MIGRATIONS.glob("*.sql")): + applied = await connection.fetchval( + "SELECT 1 FROM schema_migrations WHERE version = $1", migration.name + ) + if applied: + continue + async with connection.transaction(): + await connection.execute(migration.read_text(encoding="utf-8")) + await connection.execute( + "INSERT INTO schema_migrations (version) VALUES ($1)", migration.name + ) + + @staticmethod + def _user(row: asyncpg.Record) -> dict: + return { + "id": row["id"], + "username": row["username"], + "display_name": row["display_name"], + "created_at": row["created_at"], + } + + @staticmethod + def _chat(row: asyncpg.Record) -> dict: + return { + "id": row["id"], + "title": row["title"], + "member_ids": row["member_ids"], + "created_at": row["created_at"], + } + + @staticmethod + def _message(row: asyncpg.Record) -> dict: + return { + "id": row["id"], + "chat_id": row["chat_id"], + "sender_id": row["sender_id"], + "text": row["text"], + "attachment_ids": row["attachment_ids"], + "created_at": row["created_at"], + } + + async def ready(self) -> None: + try: + await self.pool.fetchval("SELECT 1") + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + + async def create_user(self, username: str, display_name: str, password: str) -> dict: + try: + row = await self.pool.fetchrow( + "INSERT INTO users (id, username, display_name, password_hash) VALUES ($1, $2, $3, $4) RETURNING id, username, display_name, created_at", + uuid4(), + username, + display_name, + PASSWORD_HASHER.hash(password), + ) + except asyncpg.UniqueViolationError as error: + raise ApiError(409, "conflict", "Username уже занят") from error + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + return self._user(row) + + async def authenticate(self, username: str, password: str) -> dict | None: + try: + row = await self.pool.fetchrow( + "SELECT id, username, display_name, created_at, password_hash FROM users WHERE username = $1", + username, + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + if row is None: + return None + try: + valid = PASSWORD_HASHER.verify(row["password_hash"], password) + except (InvalidHashError, VerificationError): + valid = False + return self._user(row) if valid else None + + async def create_session(self, user: dict, ttl_seconds: int) -> dict: + token = token_urlsafe(32) + token_hash = sha256(token.encode("ascii")).hexdigest() + try: + row = await self.pool.fetchrow( + "INSERT INTO sessions (token_hash, user_id, expires_at) VALUES ($1, $2, now() + $3 * interval '1 second') RETURNING expires_at", + token_hash, + user["id"], + ttl_seconds, + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + return {"token": token, "token_type": "Bearer", "expires_at": row["expires_at"], "user": user} + + async def user_for_token(self, token: str) -> dict | None: + if not token.isascii(): + return None + token_hash = sha256(token.encode("ascii")).hexdigest() + try: + row = await self.pool.fetchrow( + "SELECT u.id, u.username, u.display_name, u.created_at FROM sessions s JOIN users u ON u.id = s.user_id WHERE s.token_hash = $1 AND s.expires_at > now()", + token_hash, + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + return self._user(row) if row else None + + async def revoke_session(self, token: str) -> None: + try: + await self.pool.execute( + "DELETE FROM sessions WHERE token_hash = $1", sha256(token.encode("ascii")).hexdigest() + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + + async def create_chat(self, creator_id: UUID, title: str, member_ids: list[UUID]) -> dict: + if len(member_ids) != len(set(member_ids)) or creator_id not in member_ids: + raise ApiError(400, "invalid_request", "Некорректный список участников") + try: + async with self.pool.acquire() as connection, connection.transaction(): + existing = await connection.fetchval( + "SELECT count(*) FROM users WHERE id = ANY($1::uuid[])", member_ids + ) + if existing != len(member_ids): + raise ApiError(400, "invalid_request", "Один или несколько участников не существуют") + row = await connection.fetchrow( + "INSERT INTO chats (id, title) VALUES ($1, $2) RETURNING id, title, created_at", + uuid4(), + title, + ) + await connection.executemany( + "INSERT INTO chat_members (chat_id, user_id) VALUES ($1, $2)", + [(row["id"], member_id) for member_id in member_ids], + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + return { + "id": row["id"], + "title": row["title"], + "member_ids": member_ids, + "created_at": row["created_at"], + } + + async def get_chat_for_member(self, chat_id: UUID, user_id: UUID) -> dict: + try: + row = await self.pool.fetchrow( + "SELECT c.id, c.title, c.created_at, array_agg(cm_all.user_id ORDER BY cm_all.user_id) AS member_ids FROM chats c JOIN chat_members mine ON mine.chat_id = c.id AND mine.user_id = $2 JOIN chat_members cm_all ON cm_all.chat_id = c.id WHERE c.id = $1 GROUP BY c.id", + chat_id, + user_id, + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + if row is None: + raise ApiError(404, "not_found", "Чат не найден") + return self._chat(row) + + async def list_chats_for_member( + self, user_id: UUID, after: tuple[datetime, UUID] | None, limit: int + ) -> list[dict]: + query = "SELECT c.id, c.title, c.created_at, array_agg(cm_all.user_id ORDER BY cm_all.user_id) AS member_ids FROM chats c JOIN chat_members mine ON mine.chat_id = c.id AND mine.user_id = $1 JOIN chat_members cm_all ON cm_all.chat_id = c.id" + args: list = [user_id] + if after is not None: + query += " WHERE (c.created_at, c.id) > ($2, $3)" + args.extend(after) + query += " GROUP BY c.id ORDER BY c.created_at, c.id LIMIT $" + str(len(args) + 1) + args.append(limit + 1) + try: + rows = await self.pool.fetch(query, *args) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + return [self._chat(row) for row in rows] + + async def create_message_idempotent( + self, chat_id: UUID, sender_id: UUID, text: str, idempotency_key: str, ttl_seconds: int, attachment_ids: list[UUID] | None = None + ) -> dict: + """Создаёт сообщение и фиксирует результат retry в одной транзакции.""" + path = f"/api/v1/chats/{chat_id}/messages" + attachment_ids = attachment_ids or [] + # Preserve hashes of text-only requests already accepted by lab3. + canonical = {"text": text} + if attachment_ids: + canonical["attachment_ids"] = [str(value) for value in attachment_ids] + request_hash = sha256( + json.dumps(canonical, ensure_ascii=False, separators=(",", ":"), sort_keys=True).encode("utf-8") + ).hexdigest() + try: + async with self.pool.acquire() as connection, connection.transaction(): + member = await connection.fetchval( + "SELECT 1 FROM chat_members WHERE chat_id = $1 AND user_id = $2", chat_id, sender_id + ) + if member is None: + raise ApiError(404, "not_found", "Чат не найден") + await connection.execute( + "DELETE FROM idempotency_records WHERE created_at <= now() - $1 * interval '1 second'", ttl_seconds + ) + if attachment_ids: + count = await connection.fetchval( + "SELECT count(*) FROM attachments WHERE chat_id = $1 AND id = ANY($2::uuid[])", + chat_id, attachment_ids, + ) + if count != len(attachment_ids): + raise ApiError(404, "not_found", "Вложение не найдено") + failed = await connection.fetchval( + "SELECT 1 FROM attachments WHERE id=ANY($1::uuid[]) AND status='failed' LIMIT 1", attachment_ids + ) + if failed: + raise ApiError(409, 'conflict', 'Вложение завершилось ошибкой') + claimed = await connection.fetchrow( + "INSERT INTO idempotency_records (user_id, method, path, idempotency_key, request_hash) " + "VALUES ($1, 'POST', $2, $3, $4) ON CONFLICT DO NOTHING RETURNING message_id", + sender_id, path, idempotency_key, request_hash, + ) + if claimed is None: + recorded = await connection.fetchrow( + "SELECT request_hash, message_id FROM idempotency_records " + "WHERE user_id = $1 AND method = 'POST' AND path = $2 AND idempotency_key = $3 FOR UPDATE", + sender_id, path, idempotency_key, + ) + if recorded is None: + raise ApiError(503, "unavailable", "Не удалось получить результат идемпотентного запроса") + if recorded["request_hash"] != request_hash: + raise ApiError(409, "conflict", "Idempotency-Key использован с другим телом") + if recorded["message_id"] is None: + raise ApiError(503, "unavailable", "Не удалось получить результат идемпотентного запроса") + row = await connection.fetchrow( + "SELECT id, chat_id, sender_id, text, attachment_ids, created_at FROM messages WHERE id = $1", recorded["message_id"] + ) + if row is None: + raise ApiError(503, "unavailable", "Не удалось получить сохранённое сообщение") + return self._message(row) + row = await connection.fetchrow( + "INSERT INTO messages (id, chat_id, sender_id, text, attachment_ids) VALUES ($1, $2, $3, $4, $5) " + "RETURNING id, chat_id, sender_id, text, attachment_ids, created_at", + uuid4(), chat_id, sender_id, text, attachment_ids, + ) + await connection.execute( + "UPDATE idempotency_records SET message_id = $1 WHERE user_id = $2 AND method = 'POST' " + "AND path = $3 AND idempotency_key = $4", + row["id"], sender_id, path, idempotency_key, + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + return self._message(row) + + async def list_messages_for_member( + self, chat_id: UUID, user_id: UUID, after: tuple[datetime, UUID] | None, limit: int + ) -> list[dict]: + await self.get_chat_for_member(chat_id, user_id) + query = "SELECT id, chat_id, sender_id, text, attachment_ids, created_at FROM messages WHERE chat_id = $1" + args: list = [chat_id] + if after is not None: + query += " AND (created_at, id) > ($2, $3)" + args.extend(after) + query += " ORDER BY created_at, id LIMIT $" + str(len(args) + 1) + args.append(limit + 1) + try: + rows = await self.pool.fetch(query, *args) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + return [self._message(row) for row in rows] + + + @asynccontextmanager + async def attachment_transaction(self, *, maintenance=False): + """Shared writer / exclusive reconciler lock across all API replicas. + + Writers hold the transaction from before PUT through metadata COMMIT. + Maintenance holds the exclusive lock through inventory and deletion. + All attachment writers must participate in this protocol. + """ + try: + async with self.pool.acquire() as connection, connection.transaction(): + await connection.execute("SET LOCAL lock_timeout = '5s'") + lock = "pg_advisory_xact_lock" if maintenance else "pg_advisory_xact_lock_shared" + await connection.execute(f"SELECT {lock}(41004, 1)") + yield connection + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + + async def create_attachment(self, attachment_id, chat_id, owner_id, filename, + content_type, size_bytes, digest, staging_key, job_id, *, connection=None) -> dict: + try: + row = await (connection or self.pool).fetchrow( + "INSERT INTO attachments (id, chat_id, owner_id, filename, content_type, size_bytes, sha256, staging_key) " + "VALUES ($1,$2,$3,$4,$5,$6,$7,$8) RETURNING *", + attachment_id, chat_id, owner_id, filename, content_type, size_bytes, digest, staging_key, + ) + await (connection or self.pool).execute( + "INSERT INTO attachment_jobs (job_id, attachment_id) VALUES ($1, $2)", job_id, attachment_id + ) + payload = { + "schema_version": 1, "job_id": str(job_id), "type": "attachment.process.v1", + "attachment_id": str(attachment_id), "pipeline_version": 1, "attempt": 1, + "created_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), + } + await (connection or self.pool).execute( + "INSERT INTO attachment_outbox (id, job_id, payload) VALUES ($1, $2, $3::jsonb)", + uuid4(), job_id, json.dumps(payload, separators=(",", ":")), + ) + await (connection or self.pool).execute( + "UPDATE attachment_jobs SET next_attempt_at = 'infinity' WHERE job_id = $1", job_id + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + return dict(row) + + async def ensure_attachment_queue_capacity(self, connection: asyncpg.Connection, maximum: int) -> None: + """Globally serialize admission so replicas cannot over-accept queued work.""" + try: + await connection.execute("SELECT pg_advisory_xact_lock(41005, 1)") + pending = await connection.fetchval( + "SELECT count(*) FROM attachment_jobs WHERE status IN ('queued', 'processing')" + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + if pending >= maximum: + raise ApiError(429, "rate_limited", "Очередь обработки заполнена") + + async def attachment_for_member(self, attachment_id: UUID, user_id: UUID) -> dict: + try: + row = await self.pool.fetchrow( + "SELECT a.* FROM attachments a JOIN chat_members m ON m.chat_id = a.chat_id " + "WHERE a.id = $1 AND m.user_id = $2", attachment_id, user_id, + ) + except DB_ERRORS as error: + raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error + if row is None: + raise ApiError(404, "not_found", "Вложение не найдено") + return await self._attachment_with_variants(row) + + async def _attachment_with_variants(self, row: asyncpg.Record | dict) -> dict: + result = dict(row) + variants = await self.pool.fetch( + "SELECT name, content_type, size_bytes, width, height, sha256 FROM attachment_variants " + "WHERE attachment_id = $1 AND pipeline_version = $2 ORDER BY name", + result["id"], result["pipeline_version"], + ) + result["variants"] = [dict(item) for item in variants] + return result + + async def claim_attachment_job(self, attachment_id: UUID, lease_seconds: int) -> dict | None: + try: + async with self.pool.acquire() as connection, connection.transaction(): + row = await connection.fetchrow( + "SELECT a.*, j.job_id, j.attempt, j.lease_until, " + "(j.next_attempt_at <> 'infinity'::timestamptz AND j.next_attempt_at > now()) AS retry_waiting " + "FROM attachments a JOIN attachment_jobs j ON j.attachment_id = a.id " + "WHERE a.id = $1 FOR UPDATE", attachment_id + ) + # Only the dispatcher recovers an expired lease and increments + # its attempt. Redelivery must not renew it or bypass backoff. + if row is None or row["status"] != "queued" or row["retry_waiting"]: + return None + await connection.execute( + "UPDATE attachments SET status = 'processing', processing_started_at = now(), updated_at = now() WHERE id = $1", + attachment_id, + ) + await connection.execute( + "UPDATE attachment_jobs SET status = 'processing', first_started_at=COALESCE(first_started_at, now()), lease_until = now() + $2 * interval '1 second', updated_at = now() WHERE attachment_id = $1", + attachment_id, lease_seconds, + ) + result = dict(row) + result['claim_time'] = await connection.fetchval('SELECT processing_started_at FROM attachments WHERE id=$1', attachment_id) + return result + except DB_ERRORS as error: + raise RuntimeError("PostgreSQL is unavailable") from error + + async def complete_attachment_job(self, attachment_id: UUID, original_key: str, variants: list[dict], claim_time) -> bool: + try: + async with self.pool.acquire() as connection, connection.transaction(): + valid = await connection.fetchval( + "SELECT 1 FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id WHERE a.id=$1 AND a.status='processing' AND a.processing_started_at=$2 AND j.lease_until > now() FOR UPDATE", attachment_id, claim_time + ) + if not valid: + return False + for variant in variants: + await connection.execute( + "INSERT INTO attachment_variants (attachment_id, pipeline_version, name, object_key, content_type, size_bytes, width, height, sha256) " + "VALUES ($1, 1, $2, $3, $4, $5, $6, $7, $8) " + "ON CONFLICT (attachment_id, pipeline_version, name) DO UPDATE SET object_key = EXCLUDED.object_key, content_type = EXCLUDED.content_type, size_bytes = EXCLUDED.size_bytes, width = EXCLUDED.width, height = EXCLUDED.height, sha256 = EXCLUDED.sha256", + attachment_id, variant["name"], variant["object_key"], variant["content_type"], variant["size_bytes"], variant["width"], variant["height"], variant["sha256"], + ) + await connection.execute( + "UPDATE attachments SET status = 'ready', original_key = $2, error_code = NULL, updated_at = now() WHERE id = $1", + attachment_id, original_key, + ) + await connection.execute("UPDATE attachment_jobs SET status = 'ready', lease_until = NULL, updated_at = now() WHERE attachment_id = $1", attachment_id) + return True + except DB_ERRORS as error: + raise RuntimeError("PostgreSQL is unavailable") from error + + async def retry_attachment_job(self, attachment_id: UUID, max_attempts: int, delay_seconds: float, claim_time, permanent_code=None) -> bool: + """Return true only when a bounded at-least-once retry was recorded.""" + try: + async with self.pool.acquire() as connection, connection.transaction(): + job = await connection.fetchrow( + "SELECT j.attempt FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE a.id=$1 AND a.status='processing' AND a.processing_started_at=$2 FOR UPDATE", attachment_id, claim_time + ) + if job is None: + return False + if permanent_code is None and job["attempt"] < max_attempts: + await connection.execute( + "UPDATE attachment_jobs SET attempt = attempt + 1, status = 'queued', last_error = 'processing_failed', lease_until = NULL, next_attempt_at = now() + $2 * interval '1 second', updated_at = now() WHERE attachment_id = $1", + attachment_id, delay_seconds, + ) + await connection.execute( + "UPDATE attachments SET status = 'queued', updated_at = now() WHERE id = $1", attachment_id + ) + return True + code = permanent_code or 'retry_exhausted' + await connection.execute("UPDATE attachments SET status='failed', error_code=$2, updated_at=now() WHERE id=$1", attachment_id, code) + await connection.execute("UPDATE attachment_jobs SET status='failed', last_error=$2, lease_until=NULL, updated_at=now() WHERE attachment_id=$1", attachment_id, code) + return False + except DB_ERRORS as error: + raise RuntimeError("PostgreSQL is unavailable") from error + + async def due_outbox_events(self, limit: int = 20) -> list[dict]: + try: + return [{**dict(row), 'payload': json.loads(row['payload']) if isinstance(row['payload'], str) else row['payload']} for row in await self.pool.fetch( + "SELECT id, payload FROM attachment_outbox WHERE published_at IS NULL ORDER BY created_at LIMIT $1", limit + )] + except DB_ERRORS as error: + raise RuntimeError("PostgreSQL is unavailable") from error + + async def mark_outbox_published(self, event_id: UUID) -> None: + await self.pool.execute("UPDATE attachment_outbox SET published_at = now(), attempts = attempts + 1 WHERE id = $1", event_id) + + async def schedule_due_jobs(self, max_attempts: int = 5) -> None: + """Outbox dispatcher also restores expired leases and scheduled retries.""" + try: + async with self.pool.acquire() as connection, connection.transaction(): + rows = await connection.fetch( + "SELECT job_id, attachment_id, pipeline_version, attempt, status FROM attachment_jobs " + "WHERE (status = 'queued' AND (next_attempt_at <= now() OR " + "(next_attempt_at = 'infinity'::timestamptz AND updated_at < now() - interval '60 seconds'))) " + "OR (status = 'processing' AND lease_until < now()) " + "FOR UPDATE SKIP LOCKED" + ) + for row in rows: + row = dict(row) + if row["status"] == "processing": + if row['attempt'] >= max_attempts: + await connection.execute("UPDATE attachment_jobs SET status='failed', lease_until=NULL, last_error='retry_exhausted', updated_at=now() WHERE job_id=$1", row['job_id']) + await connection.execute("UPDATE attachments SET status='failed', error_code='retry_exhausted', updated_at=now() WHERE id=$1", row['attachment_id']) + continue + row['attempt'] += 1 + await connection.execute( + "UPDATE attachment_jobs SET status = 'queued', attempt=$2, lease_until = NULL, next_attempt_at = now(), last_error = 'lease_expired', updated_at = now() WHERE job_id = $1", row["job_id"], row['attempt'] + ) + await connection.execute("UPDATE attachments SET status = 'queued', updated_at = now() WHERE id = $1", row["attachment_id"]) + payload = {"schema_version": 1, "job_id": str(row["job_id"]), "type": "attachment.process.v1", "attachment_id": str(row["attachment_id"]), "pipeline_version": row["pipeline_version"], "attempt": row["attempt"], "created_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")} + await connection.execute("INSERT INTO attachment_outbox (id, job_id, payload) VALUES ($1, $2, $3::jsonb)", uuid4(), row["job_id"], json.dumps(payload, separators=(",", ":"))) + # Refresh the watchdog on publication intent, not on claim. + # Unclaimed jobs are republished after 60s. At-least-once + # duplicates are safe, and a published-but-lost delivery + # can no longer strand durable accepted work forever. + await connection.execute("UPDATE attachment_jobs SET next_attempt_at = 'infinity', updated_at=now() WHERE job_id = $1", row["job_id"]) + except DB_ERRORS as error: + raise RuntimeError("PostgreSQL is unavailable") from error + + async def replay_failed_attachment(self, attachment_id: UUID) -> bool: + """Explicit operator-only replay; never requeues an already successful job.""" + try: + async with self.pool.acquire() as connection, connection.transaction(): + job = await connection.fetchrow( + "SELECT job_id, status FROM attachment_jobs WHERE attachment_id = $1 FOR UPDATE", attachment_id + ) + if job is None or job["status"] != "failed": + return False + await connection.execute( + "UPDATE attachment_jobs SET status = 'queued', attempt = 1, next_attempt_at = now(), lease_until = NULL, replayed_at = now(), last_error = NULL, updated_at = now() WHERE attachment_id = $1", attachment_id + ) + await connection.execute("UPDATE attachments SET status = 'queued', error_code = NULL, updated_at = now() WHERE id = $1", attachment_id) + return True + except DB_ERRORS as error: + raise RuntimeError("PostgreSQL is unavailable") from error + + async def attachment_keys(self) -> dict[str, UUID]: + """Inventory used only by the maintenance reconciler.""" + try: + rows = await self.pool.fetch("SELECT staging_key AS object_key, id FROM attachments WHERE status <> 'ready' UNION ALL SELECT original_key, id FROM attachments WHERE original_key IS NOT NULL UNION ALL SELECT object_key, attachment_id AS id FROM attachment_variants") + except DB_ERRORS as error: + raise RuntimeError("PostgreSQL is unavailable") from error + return {row["object_key"]: row["id"] for row in rows} diff --git a/src/storage/s3.py b/src/storage/s3.py new file mode 100644 index 00000000..f154df96 --- /dev/null +++ b/src/storage/s3.py @@ -0,0 +1,120 @@ +"""Bounded S3 operations; synchronous SDK calls run outside the event loop.""" +from dataclasses import dataclass +from hashlib import sha256 +import os +from pathlib import Path + +import boto3 +from botocore.config import Config +from botocore.exceptions import BotoCoreError, ClientError +from starlette.concurrency import run_in_threadpool + +from ..errors import ApiError + + +@dataclass(frozen=True) +class S3Settings: + endpoint: str + region: str + bucket: str + access_key: str + secret_key: str + max_bytes: int + max_pixels: int + + @classmethod + def load(cls): + def secret(name): + path = os.environ.get(name + "_FILE") + result = Path(path).read_text().strip() if path else os.environ.get(name, "") + if not result: + raise ValueError(f"Missing {name}") + return result + + max_bytes = int(os.environ.get("MAX_UPLOAD_BYTES", "10485760")) + max_pixels = int(os.environ.get("MAX_IMAGE_PIXELS", "40000000")) + if not 1 <= max_bytes <= 10485760 or not 1 <= max_pixels <= 40000000: + raise ValueError("Image limits exceed contract") + return cls(os.environ.get("S3_ENDPOINT", "http://s3:8333"), + os.environ.get("S3_REGION", "us-east-1"), + os.environ.get("S3_BUCKET", "messenger-attachments"), + secret("S3_ACCESS_KEY_ID"), secret("S3_SECRET_ACCESS_KEY"), + max_bytes, max_pixels) + + +class S3Store: + def __init__(self, settings): + self.settings = settings + self.client = boto3.client( + "s3", endpoint_url=settings.endpoint, region_name=settings.region, + aws_access_key_id=settings.access_key, aws_secret_access_key=settings.secret_key, + config=Config(signature_version="s3v4", connect_timeout=2, read_timeout=5, + retries={"total_max_attempts": 1}, + s3={"addressing_style": "path"}, + request_checksum_calculation="when_required", + response_checksum_validation="when_required"), + ) + + async def call(self, method, **kwargs): + try: + return await run_in_threadpool(getattr(self.client, method), **kwargs) + except (BotoCoreError, ClientError, OSError) as error: + raise ApiError(503, "unavailable", "S3 недоступно") from error + + async def ready(self): + await self.call("head_bucket", Bucket=self.settings.bucket) + + async def put(self, key, file, size, content_type): + file.seek(0) + await self.call("put_object", Bucket=self.settings.bucket, Key=key, + Body=file, ContentLength=size, ContentType=content_type) + + async def copy(self, source_key: str, destination_key: str, content_type: str) -> None: + await self.call( + "copy_object", Bucket=self.settings.bucket, Key=destination_key, + CopySource={"Bucket": self.settings.bucket, "Key": source_key}, + ContentType=content_type, MetadataDirective="REPLACE", + ) + + async def list(self, prefix: str) -> dict[str, object]: + """Return every object under a controlled prefix, handling S3 pagination.""" + objects, _ = await self.list_with_pages(prefix) + return objects + + async def list_with_pages(self, prefix: str) -> tuple[dict[str, object], int]: + """Return inventory and number of S3 ListObjectsV2 pages consumed.""" + result: dict[str, object] = {} + token = None + pages = 0 + while True: + args = {"Bucket": self.settings.bucket, "Prefix": prefix} + if token: + args["ContinuationToken"] = token + page = await self.call("list_objects_v2", **args) + pages += 1 + for item in page.get("Contents", []): + result[item["Key"]] = item["LastModified"] + if not page.get("IsTruncated"): + return result, pages + token = page["NextContinuationToken"] + + async def delete(self, key: str) -> None: + await self.call("delete_object", Bucket=self.settings.bucket, Key=key) + + async def download(self, attachment): + return await self.download_key(attachment["original_key"], attachment["size_bytes"], attachment["sha256"]) + + async def download_key(self, key: str, expected_size: int | None = None, + expected_sha256: str | None = None) -> bytes: + response = await self.call("get_object", Bucket=self.settings.bucket, Key=key) + body = response["Body"] + try: + data = await run_in_threadpool(body.read, self.settings.max_bytes + 1) + except (BotoCoreError, OSError) as error: + raise ApiError(503, "unavailable", "S3 недоступно") from error + finally: + body.close() + if ((expected_size is not None and len(data) != expected_size) or + (expected_sha256 is not None and sha256(data).hexdigest() != expected_sha256)): + raise ApiError(503, "unavailable", "Оригинал недоступен") + return data diff --git a/src/worker.py b/src/worker.py new file mode 100644 index 00000000..d355b887 --- /dev/null +++ b/src/worker.py @@ -0,0 +1,234 @@ +"""RabbitMQ consumer for the version-1 attachment pipeline.""" +import asyncio +from hashlib import sha256 +from io import BytesIO +import json +import logging +import os +import random +import pickle +import subprocess +import sys +import time +from pathlib import Path +from concurrent.futures import ThreadPoolExecutor +from uuid import UUID, uuid4 + +from PIL import Image, ImageOps + +from .broker import QUEUE, _connection, publish_dlq +from .config import load_settings +from .storage.postgres import PostgresStore +from .storage.s3 import S3Settings, S3Store + +logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s worker %(message)s") + + +class PermanentFailure(Exception): + pass + + +def fit(image, bound): + longest = max(image.size) + if longest <= bound: + return image.copy() + return image.resize((max(1, image.width * bound // longest), + max(1, image.height * bound // longest)), Image.Resampling.LANCZOS) + + +def render_isolated(data, timeout): + try: + result = subprocess.run([sys.executable, '-m', 'src.render'], input=data, + stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=timeout) + except subprocess.TimeoutExpired as error: + raise TimeoutError('render deadline exceeded') from error + if result.returncode in (20, 21): + raise PermanentFailure('invalid_image' if result.returncode == 20 else 'image_resource_limit') + if result.returncode: + raise RuntimeError('renderer exited') + # Only our own child process produces this payload; no external pickle input. + return pickle.loads(result.stdout) + + +def thumbnail(data: bytes) -> tuple[bytes, int, int]: + with Image.open(BytesIO(data), formats=("PNG", "JPEG")) as source: + image = ImageOps.exif_transpose(source) + if image.mode in {"RGBA", "LA"} or (image.mode == "P" and "transparency" in image.info): + background = Image.new("RGBA", image.size, "white") + background.alpha_composite(image.convert("RGBA")) + image = background.convert("RGB") + else: + image = image.convert("RGB") + image = fit(image, 256) + output = BytesIO() + image.save(output, format="JPEG", quality=85, optimize=True) + return output.getvalue(), image.width, image.height + + +def variants(data: bytes, max_pixels: int = 40_000_000) -> list[tuple[str, bytes, str, int, int]]: + """Render required variants without preserving EXIF or other input metadata.""" + with Image.open(BytesIO(data), formats=("PNG", "JPEG")) as source: + if source.width * source.height > max_pixels: + raise ValueError("image exceeds configured pixel limit") + image = ImageOps.exif_transpose(source).convert("RGBA") + white = Image.new("RGBA", image.size, "white") + white.alpha_composite(image) + rgb = white.convert("RGB") + + thumb = fit(rgb, 256) + thumbnail_out = BytesIO(); thumb.save(thumbnail_out, "JPEG", quality=85, optimize=True) + + side = min(rgb.width, rgb.height) + left, top = (rgb.width - side) // 2, (rgb.height - side) // 2 + crop = rgb.crop((left, top, left + side, top + side)).resize((128, 128), Image.Resampling.LANCZOS) + crop_out = BytesIO(); crop.save(crop_out, "JPEG", quality=85, optimize=True) + + optimized = fit(rgb, 1280) + optimized_out = BytesIO(); optimized.save(optimized_out, "WEBP", quality=80, method=4) + return [ + ("thumbnail", thumbnail_out.getvalue(), "image/jpeg", thumb.width, thumb.height), + ("crop", crop_out.getvalue(), "image/jpeg", 128, 128), + ("optimized", optimized_out.getvalue(), "image/webp", optimized.width, optimized.height), + ] + + +async def process(store: PostgresStore, s3: S3Store, payload: dict, lease_seconds: int) -> None: + # Excludes cleanup for the whole lifetime of S3 effects, including stale attempts. + async with store.attachment_transaction(): + await process_locked(store, s3, payload, lease_seconds) + + +async def process_locked(store: PostgresStore, s3: S3Store, payload: dict, lease_seconds: int) -> None: + attachment_id = UUID(payload["attachment_id"]) + attachment = await store.claim_attachment_job(attachment_id, lease_seconds) + if attachment is None: + return + payload['_claim_time'] = attachment['claim_time'] + payload['attempt'] = attachment.get('attempt', payload.get('attempt', 1)) + attempt_key = uuid4().hex + original_key = f"originals/{attachment_id}/{attempt_key}" + await s3.copy(attachment["staging_key"], original_key, attachment["content_type"]) + original = await s3.download_key(attachment["staging_key"], attachment["size_bytes"], attachment["sha256"]) + rendered = await asyncio.to_thread(render_isolated, original, lease_seconds) + metadata = [] + for name, content, content_type, width, height in rendered: + extension = "webp" if name == "optimized" else "jpg" + object_key = f"variants/{attachment_id}/{attempt_key}/{name}.{extension}" + await s3.put(object_key, BytesIO(content), len(content), content_type) + metadata.append({"name": name, "object_key": object_key, "content_type": content_type, + "size_bytes": len(content), "width": width, "height": height, + "sha256": sha256(content).hexdigest()}) + # Explicit, opt-in integration fault. Marker survives container restart, + # so one crash exercises lease recovery without a permanent crash loop. + if os.environ.get('WORKER_FAULT_AFTER_S3_PUT') == '1': + marker = Path('/tmp/worker-fault-after-s3') + try: + with marker.open('x') as stream: + stream.write(str(attachment_id)) + except FileExistsError: + pass + else: + os._exit(70) + committed = await store.complete_attachment_job(attachment_id, original_key, metadata, attachment['claim_time']) + if committed: + try: + await s3.delete(attachment["staging_key"]) + except Exception: + logging.warning('staging cleanup deferred attachment_id=%s', attachment_id) + + +def settle_delivery(channel, delivery_tag, future): + """Only a successful durable disposition permits acknowledgement.""" + try: + success = future.result() is True + except Exception: + success = False + if channel.is_open: + if success: + channel.basic_ack(delivery_tag) + else: + channel.basic_nack(delivery_tag, requeue=True) + + +def main() -> None: + settings = load_settings() + connection = _connection() + channel = connection.channel() + workers = max(1, int(os.environ.get("WORKER_CONCURRENCY", "2"))) + channel.basic_qos(prefetch_count=max(1, int(os.environ.get("WORKER_PREFETCH", str(workers))))) + + async def handle_async(payload): + """Each executor thread owns its event loop and its DB/S3 clients.""" + store = PostgresStore(settings.database_url) + s3 = S3Store(S3Settings.load()) + await store.connect() + try: + await process(store, s3, payload, max(1, int(os.environ.get("JOB_TIMEOUT_SECONDS", "60")))) + finally: + s3.client.close() + await store.close() + + async def schedule_retry(attachment_id, delay, claim_time, permanent_code=None): + retry_store = PostgresStore(settings.database_url) + await retry_store.connect() + try: + return await retry_store.retry_attachment_job( + attachment_id, max(1, int(os.environ.get("JOB_MAX_ATTEMPTS", "5"))), delay, claim_time, permanent_code + ) + finally: + await retry_store.close() + + def handle(body): + payload = None + try: + payload = json.loads(body) + if not isinstance(payload, dict): + raise ValueError('job must be an object') + if payload["type"] != "attachment.process.v1" or payload["schema_version"] != 1: + raise ValueError("unexpected broker payload") + UUID(payload['attachment_id']) + UUID(payload['job_id']) + except (ValueError, KeyError, TypeError): + try: + asyncio.run(publish_dlq({'failure': 'invalid_job'})) + return True + except Exception: + return False + try: + asyncio.run(handle_async(payload)) + except Exception as error: + logging.warning("job failed attachment_id=%s", payload['attachment_id']) + if '_claim_time' not in payload: + return False + try: + attachment_id = UUID(payload["attachment_id"]) + attempt = int(payload.get("attempt", 1)) + delay = min(60.0, float(2 ** min(attempt, 6))) * random.uniform(0.5, 1.5) + asyncio.run(schedule_retry(attachment_id, delay, payload['_claim_time'], + str(error) if isinstance(error, PermanentFailure) else None)) + except Exception: + return False + return True + + def consume(ch, method, properties, body): + del properties + future = executor.submit(handle, body) + def acknowledged(_future): + if connection.is_open: + connection.add_callback_threadsafe(lambda: settle_delivery(ch, method.delivery_tag, _future)) + future.add_done_callback(acknowledged) + + try: + def heartbeat(): + Path('/tmp/worker-heartbeat').write_text(str(time.time())) + connection.call_later(2, heartbeat) + heartbeat() + with ThreadPoolExecutor(max_workers=workers, thread_name_prefix="attachment-worker") as executor: + channel.basic_consume(queue=QUEUE, on_message_callback=consume, auto_ack=False) + channel.start_consuming() + finally: + connection.close() + + +if __name__ == "__main__": + main() diff --git a/student-tests/attachment_resilience.py b/student-tests/attachment_resilience.py new file mode 100644 index 00000000..4f72d9ff --- /dev/null +++ b/student-tests/attachment_resilience.py @@ -0,0 +1,120 @@ +#!/usr/bin/env python3 +"""Explicit local Docker tests: private bucket, uploader failover, S3 outage.""" +import json +from pathlib import Path +import subprocess +import sys +import time + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / 'tests')) +from client import Client, Failure, arguments, check, run + + +def compose(*args, input=None): + return subprocess.check_output(['docker', 'compose', *args], cwd=ROOT, input=input, + text=True, stderr=subprocess.DEVNULL) + + +def wait_ready(c): + deadline = time.monotonic() + 90 + while True: + try: + c.call('GET', '/health/ready', schema='Health') + return + except Failure: + check(time.monotonic() < deadline, 'stack did not recover') + time.sleep(1) + + +def main(): + def configure(parser): + parser.add_argument('--mode', required=True, choices=['private', 'failover', 's3-outage']) + args = arguments(__doc__, configure) + c = Client(args.base_url, args.ca_file) + alice, bob, eve, chat = c.bootstrap() + data = (ROOT / 'tests/fixtures/sample.png').read_bytes() + uploaded = c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=sample.png', + 201, actor=alice, raw=data, headers={'Content-Type': 'image/png'}, schema='Attachment') + att = uploaded['json'] + content_path = '/api/v1/attachments/' + att['id'] + '/content' + if args.mode == 'private': + script = '''import sys, urllib.request, urllib.error +url = "http://s3:8333/messenger-attachments/originals/" + sys.stdin.read().strip() +try: + urllib.request.urlopen(url, timeout=5) +except urllib.error.HTTPError as error: + assert error.code == 403, error.code + print("PASS: anonymous S3 GET returns 403 for an existing object") +else: + raise AssertionError("private object leaked") +''' + print(compose('exec', '-T', 'api-a', '/app/.venv/bin/python', '-c', script, input=att['id']).strip()) + elif args.mode == 'failover': + target = uploaded['headers']['X-Instance-Id'] + service = None + for candidate in ('api-a', 'api-b'): + container = compose('ps', '-q', candidate).strip() + hostname = subprocess.check_output(['docker', 'inspect', '--format', '{{.Config.Hostname}}', container], text=True).strip() + if hostname == target: + service = candidate + check(service is not None, 'uploading replica not found') + try: + compose('stop', service) + deadline = time.monotonic() + 15 + while True: + try: + download = c.call('GET', content_path, actor=bob) + break + except Failure: + check(time.monotonic() < deadline, 'download did not recover') + time.sleep(0.5) + check(download['headers']['X-Instance-Id'] != target, 'download used stopped uploader') + check(download['bytes'] == data, 'original changed after stopping uploader') + print('PASS: uploaded via ' + service + '; stopped that container; original downloaded through another replica') + finally: + compose('start', service) + wait_ready(c) + else: + try: + compose('stop', 's3') + # Direct API request inside the network: LB excludes unready backends. + script = '''import json, sys, urllib.request, urllib.error +payload = json.load(sys.stdin) +for method, path, body, headers in [ + ("GET", "/health/ready", None, {}), + ("GET", payload["path"], None, {"Authorization": "Bearer " + payload["token"]}), + ("POST", payload["upload_path"], bytes.fromhex(payload["bytes"]), + {"Authorization": "Bearer " + payload["token"], "Content-Type": "image/png"}), +]: + req = urllib.request.Request("http://localhost:8080" + path, data=body, headers=headers, method=method) + try: + urllib.request.urlopen(req, timeout=15) + except urllib.error.HTTPError as error: + assert error.code == 503, error.code + document = json.load(error) + assert document["error"]["code"] == "unavailable" + assert document["error"]["request_id"] == error.headers["X-Request-Id"] + else: + raise AssertionError("S3 outage was not reported") +assert urllib.request.urlopen("http://localhost:8080/health/live").status == 200 +print("PASS: S3 outage gives 503 for readiness, download and upload; liveness stays 200") +''' + payload = {'path': content_path, 'token': bob['token'], 'bytes': data.hex(), + 'upload_path': '/api/v1/chats/' + chat['id'] + '/attachments?filename=outage.png'} + print(compose('exec', '-T', 'api-a', '/app/.venv/bin/python', '-c', script, + input=json.dumps(payload)).strip()) + finally: + compose('start', 's3') + # HeadBucket/read recovery alone does not prove the write path is ready. + # Use the same write-probing bootstrap as the documented restart hook. + compose('run', '--rm', '--no-deps', 's3-init') + wait_ready(c) + check(c.call('GET', content_path, actor=bob)['bytes'] == data, 'original lost after S3 outage') + c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=after-recovery.png', + 201, actor=alice, raw=data, headers={'Content-Type': 'image/png'}, schema='Attachment') + print('PASS: original available and new upload accepted after S3 restart/bootstrap') + + +if __name__ == '__main__': + run(main) diff --git a/student-tests/attachments.py b/student-tests/attachments.py new file mode 100644 index 00000000..f9f7531e --- /dev/null +++ b/student-tests/attachments.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 +"""Attachment contract checks against the running lab5 stack.""" +import hashlib +import http.client +from io import BytesIO +from pathlib import Path +import struct +import sys +import uuid +import zlib +from urllib.parse import quote, urlsplit + +from PIL import Image + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests")) +from client import Client, arguments, check, run + + +def main(): + args = arguments(__doc__) + c = Client(args.base_url, args.ca_file, timeout=30) + alice, bob, eve, chat = c.bootstrap() + path = '/api/v1/chats/' + chat['id'] + '/attachments?filename=' + png = (Path(__file__).resolve().parents[1] / 'tests/fixtures/sample.png').read_bytes() + + def upload(data, mime='image/png', filename='photo.png', status=202, actor=alice): + return c.call('POST', path + quote(filename, safe=''), status, actor=actor, raw=data, + headers={'Content-Type': mime}, schema='Attachment' if status == 202 else None) + + for fmt, mime in [('PNG', 'image/png'), ('JPEG', 'image/jpeg')]: + buf = BytesIO() + Image.new('RGB', (32, 20), 'blue').save(buf, fmt) + data = buf.getvalue() + att = upload(data, mime)['json'] + check(att['variants'] == [] and att['status'] == 'queued', 'wrong attachment state') + att = c.wait_ready(bob, att) + check(att['sha256'] == hashlib.sha256(data).hexdigest(), 'wrong checksum') + content_path = '/api/v1/attachments/' + att['id'] + check(c.call('GET', content_path, actor=bob, schema='Attachment')['json'] == att, + 'metadata changed') + check(c.call('GET', content_path + '/content', actor=bob)['bytes'] == data, 'bytes changed') + c.call('GET', content_path, 404, actor=eve) + c.call('GET', content_path + '/content', 404, actor=eve) + c.call('GET', content_path + '/content', 401) + c.call('GET', content_path + '/content?variant=thumbnail', actor=bob) + message = c.message(bob, chat, '', attachments=[att['id']])['json'] + history = c.call('GET', '/api/v1/chats/' + chat['id'] + '/messages', + actor=alice, schema='MessagePage')['json']['items'] + check(message in history, 'attachment message missing in history') + print('PASS: PNG/JPEG originals, metadata, attachment-only messages and ACL') + + for name in ('../x.png', 'x/y.png', 'x\\y.png', 'x\x00.png', 'x\n.png', 'x\x7f.png', ' ', 'x' * 256): + upload(png, filename=name, status=400) + upload(png, mime='image/jpeg', status=415) + upload(b'not an image', status=415) + upload(png[:30], status=415) + upload(png, mime='application/octet-stream', status=415) + upload(png, status=404, actor=eve) + upload(b'x' * (10485760 + 1), status=413) + + # Valid PNG header with an oversized canvas: reject before attempting pixel decoding. + def chunk(kind, data): + return struct.pack('>I', len(data)) + kind + data + struct.pack('>I', zlib.crc32(kind + data)) + huge = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', 8001, 5000, 8, 2, 0, 0, 0)) + huge += chunk(b'IDAT', zlib.compress(b'')) + chunk(b'IEND', b'') + upload(huge, status=413) + + # No Content-Length: limits must also work for chunked requests. + url = urlsplit(c.base) + connection = http.client.HTTPConnection(url.hostname, url.port, timeout=30) + connection.request('POST', path + 'chunked.png', body=iter([b'x' * 65536] * 161), + headers={'Authorization': 'Bearer ' + alice['token'], 'Content-Type': 'image/png'}, + encode_chunked=True) + response = connection.getresponse() + check(response.status == 413, 'chunked upload bypassed byte limit') + response.read() + connection.close() + print('PASS: filename, MIME, malformed image, byte/pixel and chunked limits') + + att = upload(png)['json'] + other_chat = c.call('POST', '/api/v1/chats', 201, actor=alice, + body={'title': 'other', 'member_ids': [alice['id']]}, schema='Chat')['json'] + for target, ids, code in ((chat, [str(uuid.uuid4())], 404), + (other_chat, [att['id']], 404), + (chat, [att['id'], att['id']], 400)): + c.call('POST', '/api/v1/chats/' + target['id'] + '/messages', code, actor=alice, + body={'text': '', 'attachment_ids': ids}, headers={'Idempotency-Key': uuid.uuid4().hex}) + key = uuid.uuid4().hex + first = c.message(alice, chat, 'same', key=key)['json'] + check(c.message(alice, chat, 'same', key=key, attachments=[])['json'] == first, + 'omitted and empty attachment_ids differ') + key = uuid.uuid4().hex + first = c.message(alice, chat, '', key=key, attachments=[att['id']])['json'] + check(c.message(alice, chat, '', key=key, attachments=[att['id']])['json'] == first, 'replay differs') + c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 409, actor=alice, + body={'text': 'changed', 'attachment_ids': [att['id']]}, headers={'Idempotency-Key': key}) + print('PASS: attachment scope, duplicates and idempotency normalization') + + # Exact limits are accepted; original bytes (including PNG trailing data) survive. + exact = png + b'\0' * (10485760 - len(png)) + att = upload(exact)['json'] + c.wait_ready(bob, att) + check(c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes'] == exact, + '10 MiB original changed') + buf = BytesIO() + Image.new('RGB', (8000, 5000)).save(buf, 'PNG') + att = upload(buf.getvalue())['json'] + check(att['size_bytes'] == len(buf.getvalue()), '40M pixel upload failed') + print('PASS: inclusive 10 MiB and 40 million pixel boundaries') + + +if __name__ == '__main__': + run(main) diff --git a/student-tests/idempotency_concurrency.py b/student-tests/idempotency_concurrency.py new file mode 100755 index 00000000..fcf958f4 --- /dev/null +++ b/student-tests/idempotency_concurrency.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +"""Один ключ одновременно через LB создаёт ровно одно сообщение.""" +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +import sys +import uuid + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests")) +from client import Client, check, run # noqa: E402 + + +def main() -> None: + client = Client() + alice, _bob, _eve, chat = client.bootstrap() + key = uuid.uuid4().hex + text = "concurrent idempotency " + key + + def send(_number: int): + return Client().message(alice, chat, text, key=key)["json"] + + with ThreadPoolExecutor(max_workers=16) as pool: + replies = list(pool.map(send, range(16))) + check(all(reply == replies[0] for reply in replies), "replays returned different messages") + + path = "/api/v1/chats/" + chat["id"] + "/messages?limit=100" + history = client.call("GET", path, actor=alice, schema="MessagePage")["json"]["items"] + check(sum(message["id"] == replies[0]["id"] for message in history) == 1, "concurrent retries created duplicates") + print("PASS: concurrent idempotency through load balancer") + + +if __name__ == "__main__": + run(main) diff --git a/student-tests/job_contract.py b/student-tests/job_contract.py new file mode 100644 index 00000000..961c0f98 --- /dev/null +++ b/student-tests/job_contract.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python3 +"""Check real producer payloads and redeliver a complete persisted message.""" +import json +import time +from jsonschema import Draft202012Validator, FormatChecker +from pipeline_integration import ROOT, Client, remote, duplicate, snapshot, wait, check + + +def main(): + c = Client('http://localhost:8080', timeout=30) + actor, _, _, chat = c.bootstrap() + att, _ = c.attachment(actor, chat) + wait(c, actor, att) + payloads = remote('''rows = await db.pool.fetch('SELECT payload FROM attachment_outbox') +print(json.dumps([json.loads(row['payload']) if isinstance(row['payload'], str) else row['payload'] for row in rows]))''') + validator = Draft202012Validator(json.loads((ROOT / 'contracts/jobs.schema.json').read_text()), format_checker=FormatChecker()) + for payload in payloads: + validator.validate(payload) + before = snapshot(att) + duplicate(att) + duplicate(att) + time.sleep(3) + check(snapshot(att) == before, 'persisted redelivery changed result') + result = dict(status='PASS', validated_outbox_payloads=len(payloads), + extra_payload_fields=False, persisted_redelivery_noop=True) + (ROOT / 'evidence/job-contract.json').write_text(json.dumps(result, indent=2) + '\n') + print(json.dumps(result)) + + +if __name__ == '__main__': + main() diff --git a/student-tests/pipeline_critical.py b/student-tests/pipeline_critical.py new file mode 100644 index 00000000..dfb716ff --- /dev/null +++ b/student-tests/pipeline_critical.py @@ -0,0 +1,107 @@ +"""Offline regression checks for delivery and terminal-state safety.""" +import asyncio +from contextlib import asynccontextmanager +from datetime import datetime, timezone +from io import BytesIO +import json +from pathlib import Path +import sys +from types import SimpleNamespace +import unittest +from unittest.mock import AsyncMock, MagicMock, patch +from uuid import uuid4 + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from src.storage.postgres import PostgresStore +from src.worker import process_locked as process, settle_delivery +from concurrent.futures import Future +from src.broker import _publish_to + + +class CriticalTests(unittest.IsolatedAsyncioTestCase): + def test_failed_or_unrecorded_work_is_requeued(self): + for result in (False, None, RuntimeError('database unavailable')): + future, channel = Future(), MagicMock() + if isinstance(result, Exception): + future.set_exception(result) + else: + future.set_result(result) + settle_delivery(channel, 7, future) + channel.basic_ack.assert_not_called() + channel.basic_nack.assert_called_once_with(7, requeue=True) + + def test_durable_success_is_acked(self): + future, channel = Future(), MagicMock() + future.set_result(True) + settle_delivery(channel, 7, future) + channel.basic_ack.assert_called_once_with(7) + channel.basic_nack.assert_not_called() + + async def test_outbox_decodes_jsonb(self): + store = PostgresStore('unused') + store._pool = AsyncMock() + payload = {'type': 'attachment.process.v1', 'attachment_id': str(uuid4())} + store.pool.fetch.return_value = [{'id': uuid4(), 'payload': json.dumps(payload)}] + event, = await store.due_outbox_events() + self.assertEqual(json.loads(json.dumps(event['payload'])), payload) + + async def run_processing(self, committed): + store, s3 = AsyncMock(), AsyncMock() + stamp = datetime.now(timezone.utc) + store.claim_attachment_job.return_value = { + 'claim_time': stamp, 'staging_key': 'staging/test', + 'content_type': 'image/png', 'size_bytes': 1, 'sha256': 'unused', + } + store.complete_attachment_job.return_value = committed + s3.settings = SimpleNamespace(max_pixels=40000000) + s3.download_key.return_value = b'fixture' + s3.delete.side_effect = RuntimeError('storage temporarily unavailable') + payload = {'attachment_id': str(uuid4())} + with patch('src.worker.render_isolated', return_value=[('thumbnail', b'jpeg', 'image/jpeg', 1, 1)]): + await process(store, s3, payload, 60) + self.assertEqual(store.complete_attachment_job.call_args.args[-1], stamp) + return store, s3 + + async def test_cleanup_error_does_not_retry_ready(self): + store, s3 = await self.run_processing(True) + s3.delete.assert_awaited_once() + store.retry_attachment_job.assert_not_awaited() + + async def test_stale_completion_keeps_staging(self): + _, s3 = await self.run_processing(False) + s3.delete.assert_not_awaited() + + async def test_stale_claim_cannot_publish_metadata(self): + store = PostgresStore('unused') + connection = AsyncMock() + connection.fetchval.return_value = None + @asynccontextmanager + async def context(): + yield connection + connection.transaction = context + store._pool = SimpleNamespace(acquire=context) + self.assertFalse(await store.complete_attachment_job(uuid4(), 'original', [], datetime.now(timezone.utc))) + connection.execute.assert_not_awaited() + + async def test_retry_of_terminal_or_stale_attempt_is_noop(self): + store = PostgresStore('unused') + connection = AsyncMock() + connection.fetchrow.return_value = None + @asynccontextmanager + async def context(): + yield connection + connection.transaction = context + store._pool = SimpleNamespace(acquire=context) + self.assertFalse(await store.retry_attachment_job(uuid4(), 5, 1, datetime.now(timezone.utc))) + connection.execute.assert_not_awaited() + + def test_dlq_waits_for_publisher_confirm(self): + connection = MagicMock() + with patch('src.broker._connection', return_value=connection): + _publish_to('dlq', {'failure': 'invalid_job'}) + connection.channel.return_value.confirm_delivery.assert_called_once() + self.assertTrue(connection.channel.return_value.basic_publish.call_args.kwargs['mandatory']) + + +if __name__ == '__main__': + unittest.main() diff --git a/student-tests/pipeline_integration.py b/student-tests/pipeline_integration.py new file mode 100644 index 00000000..c2304405 --- /dev/null +++ b/student-tests/pipeline_integration.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +"""Destructive-to-availability tests on the local lab stack; volumes preserved. + +Only synthetic attachments created here are mutated. Tokens stay in memory. +Run sequentially, never alongside load or another fault test. +""" +import json +import os +from pathlib import Path +import subprocess +import sys +import time + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / 'tests')) +from client import Client, Failure, check + +RESULTS = [] + + +def compose(*args, env=None, source=None): + result = subprocess.run(['docker', 'compose', *args], cwd=ROOT, + env=dict(os.environ, **(env or {})), input=source, + text=True, capture_output=True, timeout=150) + check(result.returncode == 0, 'compose failed: ' + ' '.join(args[:4]) + '\n' + result.stderr[-1000:]) + return result.stdout.strip() + + +def remote(body, **values): + source = '''import asyncio, json, os +from uuid import UUID +from src.config import load_settings +from src.storage.postgres import PostgresStore +from src.storage.s3 import S3Store, S3Settings +from src.broker import publish +V = json.loads(%r) +async def main(): + db = PostgresStore(load_settings().database_url) + await db.connect() + s3 = S3Store(S3Settings.load()) + try: +%s + finally: + s3.client.close() + await db.close() +asyncio.run(main()) +''' % (json.dumps(values), '\n'.join(' ' + line for line in body.splitlines())) + return json.loads(compose('exec', '-T', 'api-a', '/app/.venv/bin/python', '-', source=source)) + + +def record(name, **facts): + RESULTS.append(dict(test=name, status='PASS', **facts)) + print('PASS: ' + name, flush=True) + (ROOT / 'evidence/pipeline-integration.json').write_text(json.dumps(RESULTS, indent=2) + '\n') + + +def ready(c): + deadline = time.monotonic() + 90 + successes = 0 + while time.monotonic() < deadline: + try: + c.call('GET', '/health/ready') + successes += 1 + if successes >= 5: + return + time.sleep(.5) + except Failure: + successes = 0 + time.sleep(.5) + raise Failure('API did not recover') + + +def wait(c, actor, att, status='ready', timeout=120): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + value = c.call('GET', '/api/v1/attachments/' + att['id'], actor=actor, schema='Attachment')['json'] + except Failure: + time.sleep(.25) + continue + if value['status'] == status: + return value + check(value['status'] != 'failed' or status == 'failed', 'unexpected failed attachment') + time.sleep(.25) + raise Failure('attachment did not reach ' + status) + + +def snapshot(att): + return remote('''row = await db.pool.fetchrow("SELECT a.status, j.attempt, j.job_id, a.original_key, a.staging_key FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id WHERE a.id=$1", UUID(V['id'])) +objects = sorted(list(await s3.list('originals/' + V['id'] + '/')) + list(await s3.list('variants/' + V['id'] + '/'))) +print(json.dumps(dict(row) | {'objects': objects}, default=str))''', id=att['id']) + + +def duplicate(att): + return remote('''payload = await db.pool.fetchval("SELECT o.payload FROM attachment_outbox o JOIN attachment_jobs j ON j.job_id=o.job_id WHERE j.attachment_id=$1 ORDER BY o.created_at LIMIT 1", UUID(V['id'])) +await publish(json.loads(payload) if isinstance(payload, str) else payload) +print('true')''', id=att['id']) + + +def main(): + c = Client('http://localhost:8080', timeout=30) + alice, bob, _, chat = c.bootstrap() + + # No worker: HTTP admission and message creation must remain asynchronous. + compose('stop', '-t', '2', 'worker') + att, data = c.attachment(alice, chat) + check(att['status'] == 'queued', 'upload is not queued with stopped worker') + c.message(alice, chat, 'queued attachment', attachments=[att['id']]) + time.sleep(2) + compose('restart', 'broker') + compose('up', '-d', '--no-deps', '--wait', 'broker') + compose('restart', 'api-a', 'api-b') + ready(c) + compose('start', 'worker') + completed = wait(c, bob, att) + check(c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes'] == data, 'original changed') + check({v['name'] for v in completed['variants']} == {'thumbnail', 'crop', 'optimized'}, 'variants missing') + record('stopped worker, queued message, broker/API restart, ready', attachment_id=att['id']) + + before = snapshot(att) + duplicate(att) + duplicate(att) + time.sleep(3) + check(snapshot(att) == before, 'duplicate changed result or created S3 objects') + refused = remote("print(json.dumps(await db.replay_failed_attachment(UUID(V['id']))))", id=att['id']) + check(refused is False, 'ready replay accepted') + record('duplicate delivery and replay of ready are no-ops', object_count=len(before['objects'])) + + # Pause publication so the API crash is demonstrably before any publish. + compose('stop', '-t', '2', 'outbox') + compose('up', '-d', '--no-deps', '--wait', 'api-a', 'api-b', env={'OUTBOX_FAULT_AFTER_COMMIT': '1'}) + ready(c) + fault_actor, _, _, fault_chat = c.bootstrap() + try: + c.attachment(fault_actor, fault_chat) + raise AssertionError('crash hook did not interrupt upload') + except Failure: + pass + compose('up', '-d', '--no-deps', '--wait', 'api-a', 'api-b') + ready(c) + row = remote('''rows = await db.pool.fetch("SELECT a.id, o.published_at FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id JOIN attachment_outbox o ON o.job_id=j.job_id WHERE a.chat_id=$1", UUID(V['chat'])) +assert len(rows) == 1 and rows[0]['published_at'] is None +print(json.dumps({'id': str(rows[0]['id'])}))''', chat=fault_chat['id']) + compose('start', 'outbox') + wait(c, fault_actor, row) + record('crash after COMMIT before publish recovered from existing outbox', attachment_id=row['id']) + + compose('up', '-d', '--no-deps', '--wait', 'worker', env={'WORKER_FAULT_AFTER_S3_PUT': '1', 'JOB_TIMEOUT_SECONDS': '10'}) + att, _ = c.attachment(alice, chat) + wait(c, alice, att) + snap = snapshot(att) + check(snap['attempt'] >= 2 and len(snap['objects']) >= 8, 'side effect crash/lease retry not observed') + marker = compose('exec', '-T', 'worker', 'cat', '/tmp/worker-fault-after-s3') + check(marker == att['id'], 'wrong fault target') + record('worker crash after S3 before commit/ack, lease recovery', attempt=snap['attempt'], objects_before_cleanup=len(snap['objects'])) + compose('up', '-d', '--no-deps', '--wait', 'worker') + + # Select only this test's unreferenced attempt keys for cleanup. + cleaned = remote('''from argparse import Namespace +from src.reconcile import run +keys = V['keys'] +result = await run(Namespace(apply=True, grace_seconds=0, object_key=keys)) +print(json.dumps(result['deleted']))''', keys=snap['objects']) + check(len(cleaned) == 4, 'stale attempt objects not cleaned safely') + check(len(snapshot(att)['objects']) == 4, 'live attempt affected by cleanup') + wait(c, alice, att) + record('targeted orphan cleanup preserves committed attempt', deleted=len(cleaned)) + + # Lose only this synthetic staging object to exercise transient retry exhaustion. + compose('stop', '-t', '2', 'worker') + att, data = c.attachment(alice, chat) + remote("await s3.delete('staging/' + V['id'] + '/original')\nprint('true')", id=att['id']) + compose('up', '-d', '--no-deps', '--wait', 'worker', env={'JOB_MAX_ATTEMPTS': '2'}) + failed = wait(c, alice, att, 'failed') + snap = snapshot(att) + check(snap['attempt'] == 2 and failed['error_code'] == 'retry_exhausted', 'unbounded/wrong retry') + # Restore exactly the bytes originally accepted, then explicit failed-only replay. + import base64 + replayed = remote('''from io import BytesIO +import base64 +content = base64.b64decode(V['data']) +await s3.put('staging/' + V['id'] + '/original', BytesIO(content), len(content), 'image/png') +print(json.dumps(await db.replay_failed_attachment(UUID(V['id']))))''', id=att['id'], data=base64.b64encode(data).decode()) + check(replayed, 'failed replay refused') + wait(c, alice, att) + record('bounded transient retry, failed registry and explicit replay', attempts_before_replay=snap['attempt']) + compose('up', '-d', '--no-deps', '--wait', 'worker') + + # Real Linux child with deliberately insufficient address space. + memory = remote('''import subprocess +from io import BytesIO +from PIL import Image +buf = BytesIO(); Image.new('RGB', (2400, 2400)).save(buf, 'PNG') +env = dict(os.environ, RENDER_MEMORY_MB='120') +p = subprocess.run(['/app/.venv/bin/python', '-m', 'src.render'], input=buf.getvalue(), capture_output=True, env=env, timeout=30) +print(json.dumps({'returncode': p.returncode}))''') + check(memory['returncode'] == 21, 'Linux memory limit did not produce bounded resource failure') + record('Linux renderer memory limit', **memory) + + +if __name__ == '__main__': + try: + main() + finally: + compose('up', '-d', '--no-deps', '--wait', 'api-a', 'api-b', 'worker', 'outbox') diff --git a/student-tests/pipeline_invariants.py b/student-tests/pipeline_invariants.py new file mode 100644 index 00000000..e5715514 --- /dev/null +++ b/student-tests/pipeline_invariants.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +"""Real PostgreSQL/S3/RabbitMQ invariant and security checks, synthetic data only.""" +import json +from pathlib import Path +import time +from pipeline_integration import Client, check, compose, remote, wait, ROOT + + +def main(): + results = [] + c = Client('http://localhost:8080', timeout=30) + alice, bob, eve, chat = c.bootstrap() + att, _ = c.attachment(alice, chat) + wait(c, bob, att) + from io import BytesIO + from PIL import Image + for name, fmt, bound in (('thumbnail', 'JPEG', 256), ('crop', 'JPEG', 128), ('optimized', 'WEBP', 1280)): + content = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=' + name, actor=bob)['bytes'] + with Image.open(BytesIO(content)) as image: + check(image.format == fmt and max(image.size) <= bound and not image.getexif(), 'variant format/size/EXIF') + if name == 'crop': + check(image.size == (128, 128), 'crop dimensions') + results.append('actual thumbnail/crop/WebP formats, dimensions and removed EXIF') + # Verify actual private object key, not a nonexistent legacy key. + private = remote('''import urllib.request, urllib.error +key = await db.pool.fetchval('SELECT original_key FROM attachments WHERE id=$1', UUID(V['id'])) +try: + urllib.request.urlopen('http://s3:8333/' + s3.settings.bucket + '/' + key, timeout=5) +except urllib.error.HTTPError as error: + assert error.code == 403, error.code +else: + raise AssertionError('anonymous object leaked') +print('true')''', id=att['id']) + check(private, 'private S3 check failed') + for variant in ('thumbnail', 'crop', 'optimized'): + c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=' + variant, 404, actor=eve) + results.append('anonymous S3 403 on existing object; variant ACL 404') + + lock = remote('''import time +entered = asyncio.Event() +async def writer(): + async with db.attachment_transaction(): + entered.set() + await asyncio.sleep(1) +task = asyncio.create_task(writer()) +await entered.wait() +started = time.monotonic() +async with db.attachment_transaction(maintenance=True): + waited = time.monotonic() - started +await task +assert waited >= .9, waited +print(json.dumps({'maintenance_wait_seconds': waited}))''') + results.append(lock) + + # Poison staging only for a newly created synthetic attachment and update its + # digest so invalid-image classification (not transport checksum) is exercised. + compose('stop', '-t', '2', 'worker') + guarded, _ = c.attachment(alice, chat) + fence = remote('''key = UUID(V['id']) +job_id = await db.pool.fetchval('SELECT job_id FROM attachment_jobs WHERE attachment_id=$1', key) +before = await db.pool.fetchval('SELECT count(*) FROM attachment_outbox WHERE job_id=$1', job_id) +await db.pool.execute("UPDATE attachment_jobs SET updated_at=now()-interval '61 seconds' WHERE attachment_id=$1", key) +await db.schedule_due_jobs() +after = await db.pool.fetchval('SELECT count(*) FROM attachment_outbox WHERE job_id=$1', job_id) +assert after == before + 1, 'unclaimed published job stranded' +assert await db.pool.fetchval('SELECT attempt FROM attachment_jobs WHERE attachment_id=$1', key) == 1 +await db.schedule_due_jobs() +assert await db.pool.fetchval('SELECT count(*) FROM attachment_outbox WHERE job_id=$1', job_id) == after, 'watchdog floods outbox' +first = await db.claim_attachment_job(key, 30) +assert first is not None +assert await db.retry_attachment_job(key, 5, 30, first['claim_time']) +assert await db.claim_attachment_job(key, 30) is None, 'duplicate bypassed retry delay' +await db.pool.execute("UPDATE attachment_jobs SET next_attempt_at=now() WHERE attachment_id=$1", key) +await db.schedule_due_jobs() +second = await db.claim_attachment_job(key, 30) +assert second is not None and second['attempt'] == 2 +assert not await db.complete_attachment_job(key, 'invalid-stale-key', [], first['claim_time']) +assert not await db.retry_attachment_job(key, 5, 0, first['claim_time']) +await db.pool.execute("UPDATE attachment_jobs SET lease_until=now()-interval '1 second' WHERE attachment_id=$1", key) +assert await db.claim_attachment_job(key, 30) is None, 'duplicate bypassed lease attempt accounting' +await db.schedule_due_jobs() +third = await db.claim_attachment_job(key, 30) +assert third['attempt'] == 3 +assert await db.retry_attachment_job(key, 5, 0, third['claim_time']) +print('true')''', id=guarded['id']) + check(fence, 'SQL fencing/backoff invariant') + results.append('real SQL: queued watchdog republishes without attempt increment/flood; duplicate respects backoff; expired lease counted; stale completion/retry fenced') + att, _ = c.attachment(alice, chat) + remote('''from io import BytesIO +from hashlib import sha256 +data = b'invalid image payload' +await s3.put('staging/' + V['id'] + '/original', BytesIO(data), len(data), 'image/png') +await db.pool.execute('UPDATE attachments SET size_bytes=$2, sha256=$3 WHERE id=$1', UUID(V['id']), len(data), sha256(data).hexdigest()) +print('true')''', id=att['id']) + compose('start', 'worker') + wait(c, alice, guarded) + failed = wait(c, alice, att, 'failed') + check(failed['error_code'] == 'invalid_image', 'permanent classification failed') + attempt = remote("print(await db.pool.fetchval('SELECT attempt FROM attachment_jobs WHERE attachment_id=$1', UUID(V['id'])))", id=att['id']) + check(attempt == 1, 'permanent error retried') + results.append('permanent invalid_image reaches failed on attempt 1') + + # DLQ uses confirmed persistent publication; count change avoids consuming + # messages belonging to a different test. + dlq = remote('''from src.broker import _connection, _publish, DLQ +connection = _connection(); channel = connection.channel() +before = channel.queue_declare(queue=DLQ, durable=True).method.message_count +connection.close() +_publish({'malformed': True}) +print(json.dumps({'before': before}))''') + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + count = remote('''from src.broker import _connection, DLQ +connection = _connection() +print(connection.channel().queue_declare(queue=DLQ, durable=True).method.message_count) +connection.close()''') + if count > dlq['before']: + break + time.sleep(.5) + check(count == dlq['before'] + 1, 'invalid job not delivered to DLQ') + results.append('malformed broker message delivered to DLQ') + + # Actual Linux child timeout: no renderer can finish within this deadline. + timeout = remote('''from src.worker import render_isolated +try: + render_isolated(b'x', .000001) +except TimeoutError: + print('true') +else: + raise AssertionError('timeout not enforced')''') + check(timeout, 'timeout missing') + results.append('Linux render subprocess killed/reaped on deadline') + (ROOT / 'evidence/pipeline-invariants.json').write_text(json.dumps(results, indent=2) + '\n') + print(json.dumps(results, indent=2), flush=True) + + +if __name__ == '__main__': + try: + main() + finally: + compose('up', '-d', '--no-deps', '--wait', 'worker') diff --git a/student-tests/readiness_failover.py b/student-tests/readiness_failover.py new file mode 100644 index 00000000..cdb6c0a6 --- /dev/null +++ b/student-tests/readiness_failover.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +"""Запрет БД только для api-a через pg_hba; восстановление в finally.""" +import ipaddress +import json +from pathlib import Path +import subprocess +import sys +import time +import uuid + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tests")) +from client import Client, check, run + + +def compose(*args): + return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, text=True).strip() + + +def sql(statement): + return compose("exec", "-T", "postgres", "psql", "-U", "messenger", "-d", + "messenger", "-v", "ON_ERROR_STOP=1", "-Atc", statement) + + +def ready(service): + code = """import urllib.request, urllib.error +try: + print(urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=7).status) +except urllib.error.HTTPError as error: + print(error.code) +""" + return compose("exec", "-T", service, "/app/.venv/bin/python", "-c", code) + + +def main(): + client = Client() + container = json.loads(subprocess.check_output( + ["docker", "inspect", compose("ps", "-q", "api-a")], text=True))[0] + addresses = list(container["NetworkSettings"]["Networks"].values()) + check(len(addresses) == 1, "test expects one API network") + address = str(ipaddress.IPv4Address(addresses[0]["IPAddress"])) + bad_id = container["Config"]["Hostname"] + before = {client.call("GET", "/health/live")["headers"]["X-Instance-Id"] for _ in range(40)} + check(len(before) == 2 and bad_id in before, "both API replicas must be reachable") + check(ready("api-a") == ready("api-b") == "200", "both APIs must be ready") + # Сохраняем исходный pg_hba; чужую оставшуюся резервную копию не перезаписываем. + compose("exec", "-T", "postgres", "sh", "-ec", + 'test ! -e "$PGDATA/pg_hba.conf.readiness-backup"; ' + 'cp -p "$PGDATA/pg_hba.conf" "$PGDATA/pg_hba.conf.readiness-backup"') + try: + compose("exec", "-T", "postgres", "sh", "-ec", + '{ printf "host all all %s/32 reject\\n" "$1"; ' + 'cat "$PGDATA/pg_hba.conf.readiness-backup"; } > "$PGDATA/pg_hba.conf"', + "sh", address) + sql("SELECT pg_reload_conf()") + time.sleep(0.5) + sql(f"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE client_addr = '{address}'::inet") + check(ready("api-a") == "503", "api-a must be unready") + check(ready("api-b") == "200", "api-b must remain ready") + # 2s interval + 2s check timeout, с запасом на планировщик Docker Desktop. + time.sleep(6) + for _ in range(40): + result = client.call("GET", "/health/live", 200) + check(result["headers"]["X-Instance-Id"] != bad_id, "LB selected unready api-a") + for _ in range(20): + result = client.call("POST", "/api/v1/users", 201, body={ + "username": "r_" + uuid.uuid4().hex[:16], + "display_name": "Readiness regression", + "password": "Test-password-" + uuid.uuid4().hex, + }, schema="User") + check(result["headers"]["X-Instance-Id"] != bad_id, "registration reached api-a") + print("PASS: api-a ready=503, api-b ready=200; 40/40 live and 20/20 registrations use api-b") + finally: + compose("exec", "-T", "postgres", "sh", "-ec", + 'mv "$PGDATA/pg_hba.conf.readiness-backup" "$PGDATA/pg_hba.conf"') + sql("SELECT pg_reload_conf()") + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + observed = {client.call("GET", "/health/live")["headers"]["X-Instance-Id"] for _ in range(10)} + if observed == before: + check(ready("api-a") == "200", "restored api-a must be ready") + print("PASS: database access restored; both API replicas returned to LB") + return + time.sleep(0.5) + raise AssertionError("api-a did not return to LB") + + +if __name__ == "__main__": + run(main) diff --git a/student-tests/readiness_unavailable.py b/student-tests/readiness_unavailable.py new file mode 100644 index 00000000..cf4bffa1 --- /dev/null +++ b/student-tests/readiness_unavailable.py @@ -0,0 +1,16 @@ +#!/usr/bin/env python3 +"""Проверка 503 readiness при остановленном PostgreSQL.""" +from pathlib import Path +import sys + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests")) +from client import Client, run # noqa: E402 + + +def main() -> None: + Client().call("GET", "/health/ready", 503) + print("PASS: readiness rejects unavailable PostgreSQL") + + +if __name__ == "__main__": + run(main) diff --git a/student-tests/reconcile.py b/student-tests/reconcile.py new file mode 100644 index 00000000..04a8fd7d --- /dev/null +++ b/student-tests/reconcile.py @@ -0,0 +1,152 @@ +#!/usr/bin/env python3 +"""Inject S3/DB disagreement and prove dry-run, grace period and safe repair.""" +import json +import base64 +from pathlib import Path +import subprocess +import sys +import time +import uuid + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tests")) +from client import Client, Failure, arguments, check, run + + +def compose(*args, input=None, env=None) -> str: + return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, text=True, + input=input, env=env) + + +def reconcile(*args) -> dict: + output = compose("--profile", "maintenance", "run", "--rm", "--no-deps", "reconcile", *args) + return json.loads(next(line for line in reversed(output.splitlines()) if line.startswith("{"))) + + +def s3_code(action: str, key: str, data: bytes | None = None) -> None: + code = """from src.storage.s3 import S3Settings, S3Store +import base64, json, sys +s3 = S3Store(S3Settings.load()) +payload = json.load(sys.stdin) +key = payload['key'] +try: + if sys.argv[1] == 'put': + s3.client.put_object(Bucket=s3.settings.bucket, Key=key, + Body=base64.b64decode(payload.get('data', 'b3JwaGFu'))) + else: + s3.client.delete_object(Bucket=s3.settings.bucket, Key=key) +finally: + s3.client.close() +""" + payload = {"key": key} + if data is not None: + payload["data"] = base64.b64encode(data).decode() + compose("run", "--rm", "--no-deps", "s3-init", "/app/.venv/bin/python", "-c", code, action, + input=json.dumps(payload)) + + +def wait_ready(client: Client, env=None) -> None: + subprocess.run(["docker", "compose", "up", "-d", "--wait", "--no-deps", "api-a"], + cwd=ROOT, env=env, check=True, stdout=subprocess.DEVNULL) + deadline = time.monotonic() + 15 + while True: + try: + client.call("GET", "/health/ready", schema="Health") + return + except Failure: + check(time.monotonic() < deadline, "stack did not recover") + time.sleep(.5) + + +def main(): + args = arguments(__doc__) + client = Client(args.base_url, args.ca_file, timeout=30) + alice, bob, _eve, chat = client.bootstrap() + image = (ROOT / "tests/fixtures/sample.png").read_bytes() + path = "/api/v1/chats/" + chat["id"] + "/attachments?filename=" + + def upload(name, expected=201): + return client.call("POST", path + name, expected, actor=alice, raw=image, + headers={"Content-Type": "image/png"}, + schema="Attachment" if expected == 201 else None) + + # A reference object must survive an apply sweep, even with zero grace. + protected = upload("protected.png")["json"] + message = client.message(alice, chat, "protected", attachments=[protected["id"]])["json"] + orphan = "originals/test-orphan-" + uuid.uuid4().hex + s3_code("put", orphan) + + report = reconcile("--grace-seconds", "3600") + check(orphan in report["fresh_unreferenced"], "fresh orphan missing from dry-run") + check(not report["deleted"], "dry-run deleted an object") + report = reconcile("--apply", "--grace-seconds", "3600", "--object-key", orphan) + check(orphan in report["fresh_unreferenced"] and not report["deleted"], "grace period ignored") + report = reconcile("--apply", "--grace-seconds", "0", "--object-key", orphan) + check(report["deleted"] == [orphan], "aged orphan was not deleted safely") + report = reconcile("--apply", "--grace-seconds", "0", "--object-key", orphan) + check(orphan not in report["deleted"] and orphan not in report["orphan_candidates"], + "second repair run was not idempotent") + check(client.call("GET", "/api/v1/attachments/" + protected["id"] + "/content", actor=bob)["bytes"] == image, + "reconciler deleted a referenced object") + + # This models loss after metadata has committed: report it, do not delete metadata. + missing = upload("missing.png")["json"] + missing_key = "originals/" + missing["id"] + s3_code("delete", missing_key) + try: + report = reconcile("--apply", "--grace-seconds", "0", "--object-key", missing_key) + expected = {"attachment_id": missing["id"], "object_key": missing_key} + check(expected in report["missing_objects"], "missing metadata/object pair not reported") + client.call("GET", "/api/v1/attachments/" + missing["id"], actor=bob, schema="Attachment") + client.call("GET", "/api/v1/attachments/" + missing["id"] + "/content", 503, actor=bob) + finally: + s3_code("put", missing_key, image) + check(client.call("GET", "/api/v1/attachments/" + missing["id"] + "/content", actor=bob)["bytes"] == image, + "test object was not restored") + report = reconcile("--grace-seconds", "0") + check(expected not in report["missing_objects"], "restored object remained missing on second scan") + print("PASS: S3-side missing object, dry-run, grace period and idempotent orphan repair", flush=True) + + # Inject a process-equivalent failure after PUT directly into the configured + # replica. The public load balancer may correctly select the healthy peer, + # which would not exercise this boundary. + environment = dict(__import__("os").environ, UPLOAD_FAULT_AFTER_S3_PUT="1") + before_fault = set(reconcile("--grace-seconds", "0")["orphan_candidates"]) + try: + subprocess.run(["docker", "compose", "up", "-d", "--force-recreate", "--no-deps", "api-a"], + cwd=ROOT, env=environment, check=True, stdout=subprocess.DEVNULL) + wait_ready(client, environment) + fault = """import base64, json, sys, urllib.error, urllib.request +payload = json.load(sys.stdin) +request = urllib.request.Request('http://127.0.0.1:8080/api/v1/chats/' + payload['chat_id'] + + '/attachments?filename=fault-after-put.png', data=base64.b64decode(payload['image']), + method='POST', headers={'Authorization': 'Bearer ' + payload['token'], 'Content-Type': 'image/png'}) +try: + urllib.request.urlopen(request, timeout=30) +except urllib.error.HTTPError as error: + assert error.code == 503, error.code + assert json.load(error)['error']['code'] == 'unavailable' +else: + raise AssertionError('fault injection unexpectedly returned success') +""" + compose("exec", "-T", "api-a", "/app/.venv/bin/python", "-c", fault, + input=json.dumps({"chat_id": chat["id"], "token": alice["token"], + "image": base64.b64encode(image).decode()})) + report = reconcile("--grace-seconds", "0") + injected = sorted(set(report["orphan_candidates"]) - before_fault) + check(len(injected) == 1, "fault must create exactly one new orphan; run without other writers") + report = reconcile("--apply", "--grace-seconds", "0", "--object-key", injected[0]) + check(set(injected).issubset(report["deleted"]), "reconciler did not clean injected orphan") + history = client.call("GET", "/api/v1/chats/" + chat["id"] + "/messages", actor=bob, schema="MessagePage")["json"] + check(message in history["items"], "message reference was lost") + check(client.call("GET", "/api/v1/attachments/" + protected["id"] + "/content", actor=bob)["bytes"] == image, + "used original was lost") + print("PASS: injected pre-INSERT failure repaired; message and used original survive", flush=True) + finally: + environment["UPLOAD_FAULT_AFTER_S3_PUT"] = "0" + subprocess.run(["docker", "compose", "up", "-d", "--wait", "--force-recreate", "--no-deps", "api-a"], + cwd=ROOT, env=environment, check=True, stdout=subprocess.DEVNULL) + + +if __name__ == "__main__": + run(main) diff --git a/student-tests/reconcile_audit.py b/student-tests/reconcile_audit.py new file mode 100644 index 00000000..8c1e0469 --- /dev/null +++ b/student-tests/reconcile_audit.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python3 +"""Real database rejection, concurrent commit, interrupted upload and pagination.""" +import base64 +import json +from pathlib import Path +import subprocess +import sys + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tests")) +from client import Client, arguments, run + +PROBE = r''' +import asyncio, base64, hashlib, http.client, json, sys, urllib.request, urllib.error +from types import SimpleNamespace +from uuid import UUID, uuid4 +from src.config import load_settings +from src.storage.postgres import PostgresStore +from src.storage.s3 import S3Store, S3Settings +from src.reconcile import run as reconcile, scan + +payload = json.load(sys.stdin) +image = base64.b64decode(payload['image']) +chat_id, user_id = UUID(payload['chat_id']), UUID(payload['user_id']) + +def upload(filename): + req = urllib.request.Request('http://api-a:8080/api/v1/chats/' + str(chat_id) + + '/attachments?filename=' + filename, data=image, method='POST', + headers={'Authorization': 'Bearer ' + payload['token'], 'Content-Type': 'image/png'}) + try: + with urllib.request.urlopen(req, timeout=30) as response: + return response.status, json.load(response) + except urllib.error.HTTPError as error: + return error.code, json.load(error) + +async def main(): + db = PostgresStore(load_settings().database_url) + s3 = S3Store(S3Settings.load()) + await db.connect() + def args(key): + return SimpleNamespace(apply=True, grace_seconds=0, object_key=[key]) + try: + # A failed metadata inventory must not authorize any S3 deletion. + key = 'originals/audit-inventory-' + uuid4().hex + await s3.call('put_object', Bucket=s3.settings.bucket, Key=key, Body=image) + class BrokenInventory: + async def attachment_keys(self): + # Real SQL failure, not an empty inventory on error. + return await db.pool.fetch('SELECT * FROM audit_missing_table_' + uuid4().hex) + try: + async with db.attachment_transaction(maintenance=True): + try: + await scan(args(key), BrokenInventory(), s3) + except Exception: + pass + else: + raise AssertionError('failed inventory did not abort') + assert key in await s3.list('originals/') + finally: + await s3.delete(key) + print('PASS: failed PostgreSQL inventory aborts apply without deleting the orphan', flush=True) + + # An actual PostgreSQL trigger rejects only this test chat's INSERT. + suffix = uuid4().hex + name = 'audit_reject_' + suffix + before = await s3.list('originals/') + await db.pool.execute(f"""CREATE FUNCTION {name}() RETURNS trigger LANGUAGE plpgsql AS $$ + BEGIN IF NEW.chat_id = '{chat_id}'::uuid THEN + RAISE EXCEPTION 'synthetic audit rejection' USING ERRCODE = '23514'; + END IF; RETURN NEW; END $$""") + try: + await db.pool.execute(f'CREATE TRIGGER {name} BEFORE INSERT ON attachments FOR EACH ROW EXECUTE FUNCTION {name}()') + status, body = await asyncio.to_thread(upload, 'db-failure.png') + assert status == 503 and body['error']['code'] == 'unavailable', (status, body) + assert await db.pool.fetchval('SELECT count(*) FROM attachments WHERE chat_id=$1', chat_id) == 0 + finally: + await db.pool.execute(f'DROP TRIGGER IF EXISTS {name} ON attachments') + await db.pool.execute(f'DROP FUNCTION {name}()') + created = set(await s3.list('originals/')) - set(before) + assert len(created) == 1, created + key = created.pop() + report = await reconcile(SimpleNamespace(apply=False, grace_seconds=0, object_key=[key])) + assert key in report['orphan_candidates'] and not report['deleted'] + assert (await reconcile(args(key)))['deleted'] == [key] + assert not (await reconcile(args(key)))['deleted'] + print('PASS: actual PostgreSQL INSERT rejection -> 503, dry-run detection, scoped repair, repeat no-op', flush=True) + + # Writer holds a shared lock after PUT; cleanup must wait for COMMIT. + identifier = uuid4() + key = 'originals/' + str(identifier) + async with db.attachment_transaction() as connection: + await s3.call('put_object', Bucket=s3.settings.bucket, Key=key, Body=image) + sweep = asyncio.create_task(reconcile(args(key))) + for _ in range(40): + waiting = await db.pool.fetchval("SELECT count(*) FROM pg_locks WHERE locktype='advisory' AND classid=41004 AND objid=1 AND NOT granted") + if waiting: + break + await asyncio.sleep(.05) + assert waiting and not sweep.done(), 'maintenance failed to wait for active writer' + await db.create_attachment(identifier, chat_id, user_id, 'concurrent.png', 'image/png', + len(image), hashlib.sha256(image).hexdigest(), key, connection=connection) + report = await sweep + assert key not in report['deleted'] and key not in report['orphan_candidates'] + row = await db.attachment_for_member(identifier, user_id) + assert await s3.download(row) == image + print('PASS: apply with zero grace waits for in-flight PUT/COMMIT; committed original survives', flush=True) + + # Abrupt client close before all bytes arrive must not persist a file. + before = await s3.list('originals/') + rows = await db.attachment_keys() + def interrupt(): + conn = http.client.HTTPConnection('api-a', 8080, timeout=10) + conn.putrequest('POST', '/api/v1/chats/' + str(chat_id) + '/attachments?filename=interrupted.png') + conn.putheader('Authorization', 'Bearer ' + payload['token']) + conn.putheader('Content-Type', 'image/png') + conn.putheader('Content-Length', str(len(image) + 100000)) + conn.endheaders() + conn.send(image[:32]) + conn.close() + await asyncio.to_thread(interrupt) + await asyncio.sleep(1) + assert await db.attachment_keys() == rows + assert await s3.list('originals/') == before + status, body = await asyncio.to_thread(upload, 'after-disconnect.png') + assert status == 201, (status, body) + print('PASS: interrupted upload leaves no metadata/object and next upload succeeds', flush=True) + + # Exercise pagination on real S3 without adding thousands of fixtures. + original_call = s3.call + async def small_pages(method, **kwargs): + if method == 'list_objects_v2': + kwargs['MaxKeys'] = 7 + return await original_call(method, **kwargs) + expected = await s3.list('originals/') + s3.call = small_pages + actual, pages = await s3.list_with_pages('originals/') + assert actual == expected and pages > 1 + print(f'PASS: S3 pagination: {len(actual)} keys across {pages} pages, no missing/duplicate keys', flush=True) + finally: + s3.client.close() + await db.close() + +asyncio.run(main()) +''' + + +def main(): + args = arguments(__doc__) + client = Client(args.base_url, args.ca_file, timeout=30) + alice, _, _, chat = client.bootstrap() + payload = {"chat_id": chat["id"], "user_id": alice["id"], "token": alice["token"], + "image": base64.b64encode((ROOT / "tests/fixtures/sample.png").read_bytes()).decode()} + subprocess.run(["docker", "compose", "--profile", "maintenance", "run", "--rm", "--no-deps", + "--entrypoint", "/app/.venv/bin/python", "reconcile", "-c", PROBE], + cwd=ROOT, input=json.dumps(payload), text=True, check=True) + + +if __name__ == "__main__": + run(main) diff --git a/student-tests/reconcile_benchmark.py b/student-tests/reconcile_benchmark.py new file mode 100644 index 00000000..358ba405 --- /dev/null +++ b/student-tests/reconcile_benchmark.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Measure repeatable reconciler inventory cost for two controlled S3 sizes.""" +import json +from pathlib import Path +import statistics +import subprocess +import sys +import uuid + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tests")) +from client import arguments, check, run + + +def compose(*args, input=None) -> str: + return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, text=True, input=input) + + +def reconcile() -> dict: + output = compose("--profile", "maintenance", "run", "--rm", "--no-deps", "reconcile", + "--grace-seconds", "3600") + return json.loads(next(line for line in reversed(output.splitlines()) if line.startswith("{"))) + + +def mutate(action: str, keys: list[str]) -> None: + code = """from src.storage.s3 import S3Settings, S3Store +import json, sys +s3 = S3Store(S3Settings.load()) +try: + for key in json.load(sys.stdin): + if sys.argv[1] == 'put': + s3.client.put_object(Bucket=s3.settings.bucket, Key=key, Body=b'track-2-benchmark') + else: + s3.client.delete_object(Bucket=s3.settings.bucket, Key=key) +finally: + s3.client.close() +""" + compose("run", "--rm", "--no-deps", "s3-init", "/app/.venv/bin/python", "-c", code, action, + input=json.dumps(keys)) + + +def main() -> None: + arguments(__doc__) # Keeps the common --base-url/--ca-file command interface. + results = [] + for requested in (32, 128): + prefix = f"originals/track2-benchmark-{uuid.uuid4().hex}-" + keys = [prefix + str(index) for index in range(requested)] + try: + mutate("put", keys) + reports = [reconcile() for _ in range(3)] + for report in reports: + check(set(keys).issubset(report["fresh_unreferenced"]), + "benchmark objects were not included in the inventory") + check(not set(keys) & set(report["orphan_candidates"]), + "grace period classified fresh benchmark objects as deletable") + seconds = [report["scan_metrics"]["total_seconds"] for report in reports] + results.append({ + "added_unreferenced_objects": requested, + "runs": 3, + "s3_objects_scanned": reports[-1]["s3_objects"], + "metadata_rows_scanned": reports[-1]["metadata_rows"], + "s3_list_pages": reports[-1]["scan_metrics"]["s3_list_pages"], + "seconds": seconds, + "median_seconds": round(statistics.median(seconds), 6), + }) + finally: + mutate("delete", keys) + print(json.dumps({"method": "three dry-run inventories per controlled size", "results": results}, + sort_keys=True)) + print("PASS: reconciler cost measured for 32 and 128 added S3 objects; fixtures removed") + + +if __name__ == "__main__": + run(main) diff --git a/student-tests/remaining_regressions.py b/student-tests/remaining_regressions.py new file mode 100644 index 00000000..af79f767 --- /dev/null +++ b/student-tests/remaining_regressions.py @@ -0,0 +1,127 @@ +"""Contract, isolated-render and benchmark regressions without Docker.""" +import asyncio +from contextlib import asynccontextmanager +from datetime import datetime, timedelta, timezone +import importlib.util +from io import BytesIO +import json +from pathlib import Path +import sys +import tempfile +import threading +import time +from types import SimpleNamespace +import unittest +from unittest.mock import AsyncMock, patch +from uuid import uuid4 + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) +from PIL import Image +from src.worker import variants, render_isolated, PermanentFailure +from src.storage.postgres import PostgresStore +from src.reconcile import scan +from src.errors import ApiError + + +def png(width, height): + buffer = BytesIO() + Image.new('RGBA', (width, height), (255, 0, 0, 0)).save(buffer, 'PNG') + return buffer.getvalue() + + +class Regressions(unittest.IsolatedAsyncioTestCase): + async def test_failed_attachment_is_rejected(self): + store, connection = PostgresStore('unused'), AsyncMock() + connection.fetchval.side_effect = [1, 1, 1] + @asynccontextmanager + async def context(): + yield connection + connection.transaction = context + store._pool = SimpleNamespace(acquire=context) + with self.assertRaises(ApiError) as error: + await store.create_message_idempotent(uuid4(), uuid4(), '', 'key', 86400, [uuid4()]) + self.assertEqual(error.exception.status_code, 409) + connection.fetchrow.assert_not_awaited() + + def test_floor_and_small_images(self): + self.assertEqual(variants(png(300, 200))[0][3:], (256, 170)) + self.assertEqual(variants(png(200, 300))[0][3:], (170, 256)) + self.assertEqual(variants(png(10, 8))[0][3:], (10, 8)) + for name, data, mime, width, height in variants(png(300, 200)): + with Image.open(BytesIO(data)) as image: + self.assertEqual(image.size, (width, height)) + self.assertTrue(all(value > 240 for value in image.convert('RGB').getpixel((0, 0)))) + + def test_renderer_success_and_permanent_error(self): + self.assertEqual(render_isolated(png(300, 200), 10)[0][3:], (256, 170)) + with self.assertRaises(PermanentFailure): + render_isolated(b'not an image', 10) + + def test_renderer_is_killed_on_deadline(self): + started = time.monotonic() + with self.assertRaises(TimeoutError): + render_isolated(png(300, 200), .0001) + self.assertLess(time.monotonic() - started, 2) + + async def test_cleanup_retains_active_and_failed_staging(self): + old = datetime.now(timezone.utc) - timedelta(days=2) + db, s3 = AsyncMock(), AsyncMock() + db.attachment_keys.return_value = {'staging/active': uuid4(), 'staging/failed': uuid4(), 'originals/ready': uuid4()} + s3.list_with_pages.side_effect = [({'staging/active': old, 'staging/failed': old, 'staging/orphan': old}, 1), + ({'originals/ready': old}, 1), ({'variants/orphan': old}, 1)] + report = await scan(SimpleNamespace(grace_seconds=60, apply=True, object_key=[]), db, s3) + self.assertEqual(report['deleted'], ['staging/orphan', 'variants/orphan']) + + async def test_permanent_failure_is_atomic_and_not_retried(self): + store = PostgresStore('unused') + connection = AsyncMock() + connection.fetchrow.return_value = {'attempt': 1} + @asynccontextmanager + async def context(): + yield connection + connection.transaction = context + store._pool = SimpleNamespace(acquire=context) + result = await store.retry_attachment_job(uuid4(), 5, 1, datetime.now(timezone.utc), 'invalid_image') + self.assertFalse(result) + self.assertEqual(connection.execute.await_count, 2) + for call in connection.execute.call_args_list: + self.assertIn("status='failed'", call.args[0]) + self.assertEqual(call.args[2], 'invalid_image') + + def test_benchmark_partial_batch_and_recovery(self): + spec = importlib.util.spec_from_file_location('track1', ROOT / 'scripts/track1.py') + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + jobs, lock = {}, threading.Lock() + class FakeClient: + def __init__(self, *args, **kwargs): pass + def bootstrap(self): return {}, {}, {}, {'id': str(uuid4())} + def call(self, method, path, *args, **kwargs): + if method == 'POST': + key = str(uuid4()) + with lock: jobs[key] = {'id': key, 'status': 'ready', 'queue_wait_seconds': .01} + return {'status': 202, 'json': {'id': key}} + return {'json': {'status': 'ready'}} + def attachment(self, *args): return {}, b'' + def wait_ready(self, *args, **kwargs): return {} + def metrics(_): + with lock: + return {'jobs': list(jobs.values()), 'pending': 0, 'rss_bytes': 1000, 'concurrency': 2, 'prefetch': 2} + with tempfile.TemporaryDirectory() as directory: + destination = str(Path(directory) / 'result.json') + args = SimpleNamespace(run_name='test', workers=2, prefetch=2, max_pending=20, images=10, + concurrency=8, size=8, max_ready_seconds=2, require_backpressure=False, + output=destination, base_url=None, ca_file=None) + info = {'Config': {'Env': ['MAX_PENDING_JOBS=20']}, 'HostConfig': {'Memory': 4096, 'NanoCpus': 2}} + with patch.object(module, 'arguments', return_value=args), patch.object(module, 'Client', FakeClient), \ + patch.object(module, 'metrics', side_effect=metrics), patch.object(module, 'container_info', return_value=info), \ + patch.object(module, 'broker_depth', return_value={'messages_ready': 0, 'messages_unacknowledged': 0}): + module.main() + report = json.loads(Path(destination).read_text()) + self.assertEqual(report['results']['ready'], 10) + self.assertTrue(report['results']['recovery_upload_ready']) + + +if __name__ == '__main__': + unittest.main() diff --git a/student-tests/review_regressions.py b/student-tests/review_regressions.py new file mode 100644 index 00000000..7e2bba78 --- /dev/null +++ b/student-tests/review_regressions.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +"""Регрессии ревью: не-ASCII Bearer и невалидная кодировка JSON через LB.""" +from pathlib import Path +import sys + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests")) +from client import Client, arguments, check, run + + +def main(): + args = arguments(__doc__) + client = Client(args.base_url, args.ca_file) + for path in ("/api/v1/users/me", "/api/v1/auth/sessions/current"): + method = "DELETE" if path.endswith("/current") else "GET" + result = client.call(method, path, 401, + headers={"Authorization": "Bearer é"}, schema="Error") + check(result["headers"].get("WWW-Authenticate", "").startswith("Bearer"), + "missing Bearer challenge") + check(result["json"]["error"]["code"] == "unauthorized", "wrong error code") + check(result["json"]["error"]["request_id"] == result["headers"]["X-Request-Id"], + "request IDs differ") + for raw in (b"\xff", b"{", b'{"username":'): + result = client.call("POST", "/api/v1/users", 400, raw=raw, + headers={"Content-Type": "application/json"}, schema="Error") + check(result["json"]["error"]["code"] == "invalid_request", "wrong JSON error code") + check(result["json"]["error"]["request_id"] == result["headers"]["X-Request-Id"], + "request IDs differ") + print("PASS: non-ASCII Bearer returns 401/challenge; malformed JSON returns contract Error/400") + + +if __name__ == "__main__": + run(main) diff --git a/student-tests/s3_permissions.py b/student-tests/s3_permissions.py new file mode 100644 index 00000000..d67e7be3 --- /dev/null +++ b/student-tests/s3_permissions.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""Prove that the API identity is bucket-scoped and lacks admin operations.""" +import subprocess +from pathlib import Path +import sys +import uuid + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tests")) +from client import Client, arguments, check, run + + +def compose(*args, input=None): + return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, text=True, + input=input, stderr=subprocess.DEVNULL) + + +def main(): + args = arguments(__doc__) + client = Client(args.base_url, args.ca_file) + alice, _bob, _eve, chat = client.bootstrap() + data = (ROOT / "tests/fixtures/sample.png").read_bytes() + client.call("POST", "/api/v1/chats/" + chat["id"] + "/attachments?filename=permission.png", 202, + actor=alice, raw=data, headers={"Content-Type": "image/png"}, schema="Attachment") + bucket = "permission-probe-" + uuid.uuid4().hex + code = """from botocore.exceptions import ClientError +from src.storage.s3 import S3Settings, S3Store +import sys +s3 = S3Store(S3Settings.load()) +try: + try: + s3.client.create_bucket(Bucket=sys.stdin.read().strip()) + except ClientError as error: + assert error.response['Error']['Code'] in {'AccessDenied', 'Unauthorized'}, error.response + else: + raise AssertionError('API S3 identity unexpectedly has admin bucket creation') +finally: + s3.client.close() +""" + compose("exec", "-T", "api-a", "/app/.venv/bin/python", "-c", code, input=bucket) + admin = """from src.storage.s3 import S3Settings, S3Store +import sys +s3 = S3Store(S3Settings.load()) +try: + s3.client.list_buckets() +finally: + s3.client.close() +""" + compose("run", "--rm", "--no-deps", "s3-init", "/app/.venv/bin/python", "-c", admin) + print("PASS: API identity cannot create buckets; maintenance identity has separate admin access") + + +if __name__ == "__main__": + run(main) diff --git a/student-tests/session_expiry.py b/student-tests/session_expiry.py new file mode 100644 index 00000000..37e29fb8 --- /dev/null +++ b/student-tests/session_expiry.py @@ -0,0 +1,20 @@ +#!/usr/bin/env python3 +"""Проверка expiry; запускается при SESSION_TTL_SECONDS=2.""" +from pathlib import Path +import sys +import time + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests")) +from client import Client, check, run # noqa: E402 + + +def main() -> None: + client = Client() + actor = client.user("expiry") + time.sleep(3) + client.call("GET", "/api/v1/users/me", 401, actor=actor) + print("PASS: expired Bearer session was rejected") + + +if __name__ == "__main__": + run(main) diff --git a/student-tests/upload_memory.py b/student-tests/upload_memory.py new file mode 100644 index 00000000..1075a4eb --- /dev/null +++ b/student-tests/upload_memory.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""Ten concurrent bounded uploads, with API-container peak memory sampling.""" +from concurrent.futures import ThreadPoolExecutor +import json +import http.client +from pathlib import Path +import subprocess +import sys +import threading +import time +from urllib.parse import quote +from urllib.parse import urlsplit + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tests")) +from client import Client, arguments, check, run + + +def api_memory() -> dict[str, int]: + ids = subprocess.check_output(["docker", "compose", "ps", "-q", "api-a", "api-b"], cwd=ROOT, text=True).split() + return {container[:12]: int(subprocess.check_output( + ["docker", "exec", container, "cat", "/sys/fs/cgroup/memory.current"], text=True)) + for container in ids} + + +def main(): + def configure(parser): + parser.add_argument("--output", default="evidence/upload-memory.json") + parser.add_argument("--size-mib", type=int, default=5) + args = arguments(__doc__, configure) + check(1 <= args.size_mib <= 10, "--size-mib must be 1..10") + client = Client(args.base_url, args.ca_file, timeout=60) + alice, _bob, _eve, chat = client.bootstrap() + seed = (ROOT / "tests/fixtures/sample.png").read_bytes() + payload = seed + b"\0" * (args.size_mib * 1024 * 1024 - len(seed)) + baseline, peak, samples, stop = api_memory(), {}, [], threading.Event() + check(len(baseline) == 2, "memory experiment requires two running APIs") + sampling_errors = [] + timestamps = [] + barrier = threading.Barrier(10) + + def sampler(): + while not stop.is_set(): + try: + values = api_memory() + check(set(values) == set(baseline), "API containers changed during measurement") + except Exception as error: + sampling_errors.append(type(error).__name__) + return + timestamps.append(time.monotonic()) + samples.append(values) + for name, value in values.items(): + peak[name] = max(peak.get(name, 0), value) + time.sleep(.1) + + def upload(index): + barrier.wait(timeout=15) + parsed = urlsplit(client.base) + connection = http.client.HTTPConnection(parsed.hostname, parsed.port, timeout=60) + + def chunks(): + for offset in range(0, len(payload), 512 * 1024): + yield payload[offset:offset + 512 * 1024] + # Keep all ten client streams open long enough for multiple memory samples. + time.sleep(.5) + + connection.request("POST", "/api/v1/chats/" + chat["id"] + "/attachments?filename=" + quote(f"load-{index}.png"), + body=chunks(), encode_chunked=True, + headers={"Authorization": "Bearer " + alice["token"], "Content-Type": "image/png"}) + response = connection.getresponse() + body = response.read() + connection.close() + check(response.status == 201, "upload %s returned %s: %s" % (index, response.status, body[:200])) + return json.loads(body) + + started = time.monotonic() + thread = threading.Thread(target=sampler, daemon=True) + thread.start() + try: + with ThreadPoolExecutor(max_workers=10) as pool: + attachments = list(pool.map(upload, range(10))) + finally: + stop.set() + thread.join(timeout=5) + elapsed = time.monotonic() - started + check(len({item["id"] for item in attachments}) == 10, "uploads were not distinct") + check(not sampling_errors and len(samples) >= 2 and not thread.is_alive(), + "memory sampler failed or did not collect enough samples: " + repr(sampling_errors)) + report = { + "uploads": 10, "file_bytes": len(payload), "concurrency": 10, + "api_upload_concurrency_limit": 1, "elapsed_seconds": round(elapsed, 3), + "baseline_memory_bytes": baseline, "peak_memory_bytes": peak, + "peak_growth_bytes": {name: peak.get(name, 0) - baseline.get(name, 0) for name in baseline}, + "samples": len(samples), + "sample_intervals_seconds": [round(b-a, 3) for a, b in zip(timestamps, timestamps[1:])], + "method": "cgroup memory.current via Docker; 0.1s pause AFTER each poll, actual intervals include Docker latency; ten clients start at a barrier and send chunked 512 KiB blocks every 0.5s. Peak is sampled, not continuous.", + } + destination = ROOT / args.output + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n") + print(json.dumps(report, ensure_ascii=False)) + check(all(value <= 512 * 1024 * 1024 for value in peak.values()), "API exceeded configured memory limit") + + +if __name__ == "__main__": + run(main) diff --git a/uv.lock b/uv.lock new file mode 100644 index 00000000..6f5df98f --- /dev/null +++ b/uv.lock @@ -0,0 +1,596 @@ +version = 1 +revision = 3 +requires-python = ">=3.12" +resolution-markers = [ + "python_full_version >= '3.14'", + "python_full_version < '3.14'", +] + +[[package]] +name = "annotated-doc" +version = "0.0.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/8e/38aa427ed5402449e226975b649c5dc73ccadfefeb95e6aecb8f8ea4b6b6/annotated_doc-0.0.5.tar.gz", hash = "sha256:c7e58ce09192557605d8bbd92836d7e1d520ac9580096042c0bfd197efacf1bb", size = 10758, upload-time = "2026-07-28T13:50:58.129Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3e/30/e900b21425a860e195f32e37657aa1f7c7f2b1bfb26f03ca209b90933c06/annotated_doc-0.0.5-py3-none-any.whl", hash = "sha256:117bac03a25ede5df5440e855b32d556049ca169ead221505badf432fed4b101", size = 5302, upload-time = "2026-07-28T13:50:57.239Z" }, +] + +[[package]] +name = "annotated-types" +version = "0.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5f/56/a8120250d128bed162cd73c76d45f6ef9991f3e068f62a8ee060afa3104a/annotated_types-0.8.0.tar.gz", hash = "sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7", size = 15893, upload-time = "2026-07-23T20:16:13.995Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/99/91/8acff4f5e50511b911bbccb72b8628a49c68ce14148cd9f6431094859a90/annotated_types-0.8.0-py3-none-any.whl", hash = "sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0", size = 13427, upload-time = "2026-07-23T20:16:12.938Z" }, +] + +[[package]] +name = "anyio" +version = "4.15.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions", marker = "python_full_version < '3.15'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a9/d2/f4d173e22df740bc37b1db102b386ba719b66e95b0f0d751f556b387e6d2/anyio-4.15.1.tar.gz", hash = "sha256:9f28306018cbd6d329e64a36d58256edff76dd996fe423bc957326e578b82a94", size = 276966, upload-time = "2026-09-05T10:42:39.44Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/b8/4bd346e22b28902df4d651910f5242c28d84e4a5c2435ca5c3f797ed7e2e/anyio-4.15.1-py3-none-any.whl", hash = "sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7ed101", size = 132079, upload-time = "2026-09-05T10:42:37.923Z" }, +] + +[[package]] +name = "argon2-cffi" +version = "25.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "argon2-cffi-bindings" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0e/89/ce5af8a7d472a67cc819d5d998aa8c82c5d860608c4db9f46f1162d7dab9/argon2_cffi-25.1.0.tar.gz", hash = "sha256:694ae5cc8a42f4c4e2bf2ca0e64e51e23a040c6a517a85074683d3959e1346c1", size = 45706, upload-time = "2025-06-03T06:55:32.073Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4f/d3/a8b22fa575b297cd6e3e3b0155c7e25db170edf1c74783d6a31a2490b8d9/argon2_cffi-25.1.0-py3-none-any.whl", hash = "sha256:fdc8b074db390fccb6eb4a3604ae7231f219aa669a2652e0f20e16ba513d5741", size = 14657, upload-time = "2025-06-03T06:55:30.804Z" }, +] + +[[package]] +name = "argon2-cffi-bindings" +version = "26.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cffi" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0b/43/bb8b6e8708d49a5ab36781333af092d9f483b198a2710d01281204640055/argon2_cffi_bindings-26.1.0.tar.gz", hash = "sha256:63505c71542a44b68b1e38060450fb006404170da375feb31af153e7f9c6205d", size = 1790807, upload-time = "2026-08-20T07:44:22.492Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/d2/0ae991f1b2181e5be49007c574710a800ad36c2978683addb3e67c474e55/argon2_cffi_bindings-26.1.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:21ca0396fe5ec995dd54431c32698189666f9224810acfa752e50d2bd94d9df2", size = 25521, upload-time = "2026-08-20T07:32:43.019Z" }, + { url = "https://files.pythonhosted.org/packages/7e/e4/ad91d8297638aa2258aad4501c306aca99480dfe76ccd638173fa3702db9/argon2_cffi_bindings-26.1.0-cp310-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:78de2d65e0b9ea7ce9d1b1c3e87297b2d7305a02c266ee2a2d6910daddd7ee69", size = 27177, upload-time = "2026-08-20T07:32:44.158Z" }, + { url = "https://files.pythonhosted.org/packages/6f/86/5363df11b86d02cf3662208e7406496327649cc90eb365bf6f4e8a54a41f/argon2_cffi_bindings-26.1.0-cp310-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:27f1821903e2ceadcb88ec2b45ef190897b7682449c772f4d9b53e42c520cf29", size = 26597, upload-time = "2026-08-20T07:32:45.172Z" }, + { url = "https://files.pythonhosted.org/packages/f4/b5/a14dcc592652347dad23ee93b278a4da5d2a25c9ed3ebd10d68eea823a4f/argon2_cffi_bindings-26.1.0-cp310-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d88e5f7e60f28ae0b0cc6b2f16c43e87cd642a196a86f85e0d8bb6fe016fc16d", size = 27403, upload-time = "2026-08-20T07:32:46.13Z" }, + { url = "https://files.pythonhosted.org/packages/b3/81/b4a20d4902af7f796390bf9245ff83c5217dfa7367efa1d14986956c482b/argon2_cffi_bindings-26.1.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:34b7d9c24a4165a2c61cc8ae11d44d48c9ce2830fb536cb7914e11fdd9962728", size = 27132, upload-time = "2026-08-20T07:32:47.13Z" }, + { url = "https://files.pythonhosted.org/packages/7e/1b/c8de358af07b1c490e0fcb863ef98e46ddb486e45567aca5a60bd68d9daa/argon2_cffi_bindings-26.1.0-cp310-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:224865cbbcb7a2bd1356741dff12b0134df726b6d44bb7b500df8e303cbd9e81", size = 27588, upload-time = "2026-08-20T07:32:48.087Z" }, + { url = "https://files.pythonhosted.org/packages/48/2f/7ee62a6e79f9309f9d9982d301b22a00010adb580c05c8109b94d7b33de0/argon2_cffi_bindings-26.1.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:ffff613aaa9ce6236766e2fc6dc560bb5abde7a2e2416e3db1f9ae395a2b4dd4", size = 26785, upload-time = "2026-08-20T07:32:48.977Z" }, + { url = "https://files.pythonhosted.org/packages/e9/10/960d0ee93d4897741bcaf4799c697dae2d81499f66fd1ed042a7dd54c1f4/argon2_cffi_bindings-26.1.0-cp310-abi3-win32.whl", hash = "sha256:a86c069c91a747a2c4e5c51473590aeb48172fff9b2130d23729a42d98665ecb", size = 23898, upload-time = "2026-08-20T07:32:50.114Z" }, + { url = "https://files.pythonhosted.org/packages/6d/3a/0cc14a05810e6add9bce5e87693334baa2222de5f647fa31781885b6573f/argon2_cffi_bindings-26.1.0-cp310-abi3-win_amd64.whl", hash = "sha256:2c36ff87b5dfaa477d0bd51e9d7f6abdae7c8955d2983c97419085d842154b3e", size = 25730, upload-time = "2026-08-20T07:32:51.091Z" }, + { url = "https://files.pythonhosted.org/packages/4e/db/d83cf2af140547f0b9cdaece05b2dc2dcbf991be4667331d073eff771435/argon2_cffi_bindings-26.1.0-cp310-abi3-win_arm64.whl", hash = "sha256:f9c4420a7a864fe1b86ce35befc95b8e39fb852493b81cf798671ddc265de638", size = 24478, upload-time = "2026-08-20T07:32:52.111Z" }, + { url = "https://files.pythonhosted.org/packages/bb/5f/f652055e18d2627e2eed94c7f31a792127cfe38df786635395d742321674/argon2_cffi_bindings-26.1.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:af11ac37a7c53dc16cb7950a6190851b0870fe218b6c60c0bb7ac355234e3083", size = 15434, upload-time = "2026-08-20T07:32:53.143Z" }, + { url = "https://files.pythonhosted.org/packages/76/38/de696045960f5b846d428c0fb6c130ed3da87aac2af209b05c193815404c/argon2_cffi_bindings-26.1.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:db0fcd827ca61622a01b220aadfbece01939acf53888f2cb98cd93e9b1e2c97e", size = 15449, upload-time = "2026-08-20T07:32:54.075Z" }, + { url = "https://files.pythonhosted.org/packages/91/0a/c25af768f6b75a5a71e31207f87c540656b2808c015260444a22763221ad/argon2_cffi_bindings-26.1.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:28524438cd3e723f25412f63d4fd516ff5bae9ae5aa56acbe2a1404398a0cf31", size = 25683, upload-time = "2026-08-20T07:32:55.05Z" }, + { url = "https://files.pythonhosted.org/packages/a8/7e/be212c751ab0bcea7f646615f933bf262e8e50b3f7bef32f861d0a2d066b/argon2_cffi_bindings-26.1.0-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ac82fc756a446b6ccd7139ce70efa9d8bbe541e7ad579a12dcb52764b7175c5f", size = 27311, upload-time = "2026-08-20T07:32:56.166Z" }, + { url = "https://files.pythonhosted.org/packages/a6/ee/f84b28e4afd13d3cac36c1d8fa8c239d2dc2c51cd978d02ee5d5ad98d9bb/argon2_cffi_bindings-26.1.0-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6a4e68eed961a8de6928d1c17ff3dc2a547e0e923c17f8f1cd79fb7bc9502f98", size = 26771, upload-time = "2026-08-20T07:32:57.206Z" }, + { url = "https://files.pythonhosted.org/packages/21/c3/95c07a023691ecd529da9cb6a8f0779e13ebc1bdfaa86d145fdc1c6e7e79/argon2_cffi_bindings-26.1.0-cp314-cp314t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:151dfaad9de753f4af2a7854e707e4784f2acc434340ade64239c5b104b2d605", size = 27568, upload-time = "2026-08-20T07:32:58.361Z" }, + { url = "https://files.pythonhosted.org/packages/e6/31/3a18e31406d8694b4d6a31573c3e572fff6bed318bb744453eb653766d22/argon2_cffi_bindings-26.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:061a6919145bbf282ebf1f9c59d3135d4833c25313c8595c0d68cf7712ddfce2", size = 27280, upload-time = "2026-08-20T07:32:59.343Z" }, + { url = "https://files.pythonhosted.org/packages/0b/39/d4be4577e178b2397aa5b5575c8a309bf0da2afe05fe0c72c8f398662d63/argon2_cffi_bindings-26.1.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:62ff20cd130c956c7c9144d5fe35228f98b51c579b2439e988b27ef93e16c02a", size = 27776, upload-time = "2026-08-20T07:33:00.325Z" }, + { url = "https://files.pythonhosted.org/packages/71/47/78f4dd96f7411339f723b96fe24039c1bd5835102b8a5ba71ac4ec712ac7/argon2_cffi_bindings-26.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:19423e5d7ac1cc354baab59eaabf18db2ec04ef6593b5abe5a34f323c4a8f87a", size = 26932, upload-time = "2026-08-20T07:33:01.272Z" }, + { url = "https://files.pythonhosted.org/packages/3b/cd/96bfd37434cc0a848a9066c291d84b28846c4c9ea289ed9866b1164d622b/argon2_cffi_bindings-26.1.0-cp314-cp314t-win32.whl", hash = "sha256:4f84cdd868978d7b7350a566c254042d44216d9e37f241f3a6d3b1dfebeede35", size = 24878, upload-time = "2026-08-20T07:33:02.189Z" }, + { url = "https://files.pythonhosted.org/packages/f1/42/d8b6810abd9b1bd2f47ebbccf460da59c9f32e94888bea4f7b137d998797/argon2_cffi_bindings-26.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:2b741888c93147444fdfc851abd81cc207f37f7f7da42062a00deb3888e57da8", size = 26656, upload-time = "2026-08-20T07:33:03.222Z" }, + { url = "https://files.pythonhosted.org/packages/a9/d1/095d95eaf2ed1d9f77268cf3291bde148c6cd56121f8db2c74c1ba618a0e/argon2_cffi_bindings-26.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:6ab674f668d5962a3a4136ae0812519b0f1586874263723a32181d60d64137e1", size = 25378, upload-time = "2026-08-20T07:33:04.332Z" }, + { url = "https://files.pythonhosted.org/packages/66/cb/214092c39c4dbcb72cf98b12234ddac2221f8fe2c0acf29c6a70fa83be53/argon2_cffi_bindings-26.1.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:1d98e33bd8bd67d7206c124e200bf2229c4cfa8c9c19f7b44a897f0fc71837eb", size = 25683, upload-time = "2026-08-20T07:33:05.337Z" }, + { url = "https://files.pythonhosted.org/packages/83/e5/02015b83e9b05ccb85ff2ced424cf6e83a12d3810bc7f66d679a92b69ffb/argon2_cffi_bindings-26.1.0-cp315-cp315t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ccaf0a46cbb380f1fd102a874e32aa629fd3cb0c0e94f4943fa1f6d5edc5dac6", size = 27310, upload-time = "2026-08-20T07:33:06.344Z" }, + { url = "https://files.pythonhosted.org/packages/c3/4a/85e612787d0796878b3b4f6bd53dcd5484b6fe7b64cc6fc7b6e6a04cf835/argon2_cffi_bindings-26.1.0-cp315-cp315t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f0c3103fcff20183e593459cfea6e012281c0e76ae3ed8b5565ad1b92eac3990", size = 26771, upload-time = "2026-08-20T07:33:07.429Z" }, + { url = "https://files.pythonhosted.org/packages/f6/84/ccb003b6f9969820e87656398f4d49c857def71a85ca1588a0e809afd7ce/argon2_cffi_bindings-26.1.0-cp315-cp315t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c49e853a3bef9dd10329f31f702e7fa9b5c58229ff9c2ff6d069efaf09177c08", size = 27569, upload-time = "2026-08-20T07:33:08.598Z" }, + { url = "https://files.pythonhosted.org/packages/88/07/c26b76debf0998ee08fbe947ab2058ac5de37d4b9d46b06c17abaa6c4ce9/argon2_cffi_bindings-26.1.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:6376d4b3aca039375ca8bf92f770da0ec424a1ce3a37077a8d3c557411aa56ca", size = 27279, upload-time = "2026-08-20T07:33:09.518Z" }, + { url = "https://files.pythonhosted.org/packages/ee/0d/ead6ddc029f91bc9b9390686dad3c808ab08100d348f6266b5f93f8970ee/argon2_cffi_bindings-26.1.0-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:9bacedc04b0402837586a17f0919e3dfdd95291f441f1f56bd80ec274c2840a1", size = 27774, upload-time = "2026-08-20T07:33:10.728Z" }, + { url = "https://files.pythonhosted.org/packages/7d/47/c108530d9eb86036b78d3af4de28b83b4a2d9a70512bd10ff8e59966aab4/argon2_cffi_bindings-26.1.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:76ae29acace5d33355344612844d588e19deaaba4639d8bb01601e4b1418ef36", size = 26933, upload-time = "2026-08-20T07:33:11.661Z" }, + { url = "https://files.pythonhosted.org/packages/a9/02/0bfc59e781c89acf64c31c388aade9d9d1c1ea38aa1ba1292fe07f607fe9/argon2_cffi_bindings-26.1.0-cp315-cp315t-win32.whl", hash = "sha256:df612391feca41c44d20118f3b88d1b86419465cd1f5496859f715ca60ec2210", size = 24875, upload-time = "2026-08-20T07:33:12.616Z" }, + { url = "https://files.pythonhosted.org/packages/61/c7/c3e46068cddffccecb8ad94d71135e9bf62bbc789589e7dfadc7c6f59214/argon2_cffi_bindings-26.1.0-cp315-cp315t-win_amd64.whl", hash = "sha256:1a0a29ed86960e44eaace7e081bdfab4f08b012fd96ec8edba71e2ad020939e4", size = 26655, upload-time = "2026-08-20T07:33:13.521Z" }, + { url = "https://files.pythonhosted.org/packages/f4/ca/18b9c8c45fecf34b9100ec6d7946057f14a158f2eaa20ea123a3e82351cb/argon2_cffi_bindings-26.1.0-cp315-cp315t-win_arm64.whl", hash = "sha256:d157ddfab1e8b21f2f1dedda9c09645d98b5ed0b667b0626be600a345d426440", size = 25376, upload-time = "2026-08-20T07:33:14.491Z" }, +] + +[[package]] +name = "asyncpg" +version = "0.31.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fe/cc/d18065ce2380d80b1bcce927c24a2642efd38918e33fd724bc4bca904877/asyncpg-0.31.0.tar.gz", hash = "sha256:c989386c83940bfbd787180f2b1519415e2d3d6277a70d9d0f0145ac73500735", size = 993667, upload-time = "2025-11-24T23:27:00.812Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/a6/59d0a146e61d20e18db7396583242e32e0f120693b67a8de43f1557033e2/asyncpg-0.31.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b44c31e1efc1c15188ef183f287c728e2046abb1d26af4d20858215d50d91fad", size = 662042, upload-time = "2025-11-24T23:25:49.578Z" }, + { url = "https://files.pythonhosted.org/packages/36/01/ffaa189dcb63a2471720615e60185c3f6327716fdc0fc04334436fbb7c65/asyncpg-0.31.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0c89ccf741c067614c9b5fc7f1fc6f3b61ab05ae4aaa966e6fd6b93097c7d20d", size = 638504, upload-time = "2025-11-24T23:25:51.501Z" }, + { url = "https://files.pythonhosted.org/packages/9f/62/3f699ba45d8bd24c5d65392190d19656d74ff0185f42e19d0bbd973bb371/asyncpg-0.31.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:12b3b2e39dc5470abd5e98c8d3373e4b1d1234d9fbdedf538798b2c13c64460a", size = 3426241, upload-time = "2025-11-24T23:25:53.278Z" }, + { url = "https://files.pythonhosted.org/packages/8c/d1/a867c2150f9c6e7af6462637f613ba67f78a314b00db220cd26ff559d532/asyncpg-0.31.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:aad7a33913fb8bcb5454313377cc330fbb19a0cd5faa7272407d8a0c4257b671", size = 3520321, upload-time = "2025-11-24T23:25:54.982Z" }, + { url = "https://files.pythonhosted.org/packages/7a/1a/cce4c3f246805ecd285a3591222a2611141f1669d002163abef999b60f98/asyncpg-0.31.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3df118d94f46d85b2e434fd62c84cb66d5834d5a890725fe625f498e72e4d5ec", size = 3316685, upload-time = "2025-11-24T23:25:57.43Z" }, + { url = "https://files.pythonhosted.org/packages/40/ae/0fc961179e78cc579e138fad6eb580448ecae64908f95b8cb8ee2f241f67/asyncpg-0.31.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bd5b6efff3c17c3202d4b37189969acf8927438a238c6257f66be3c426beba20", size = 3471858, upload-time = "2025-11-24T23:25:59.636Z" }, + { url = "https://files.pythonhosted.org/packages/52/b2/b20e09670be031afa4cbfabd645caece7f85ec62d69c312239de568e058e/asyncpg-0.31.0-cp312-cp312-win32.whl", hash = "sha256:027eaa61361ec735926566f995d959ade4796f6a49d3bde17e5134b9964f9ba8", size = 527852, upload-time = "2025-11-24T23:26:01.084Z" }, + { url = "https://files.pythonhosted.org/packages/b5/f0/f2ed1de154e15b107dc692262395b3c17fc34eafe2a78fc2115931561730/asyncpg-0.31.0-cp312-cp312-win_amd64.whl", hash = "sha256:72d6bdcbc93d608a1158f17932de2321f68b1a967a13e014998db87a72ed3186", size = 597175, upload-time = "2025-11-24T23:26:02.564Z" }, + { url = "https://files.pythonhosted.org/packages/95/11/97b5c2af72a5d0b9bc3fa30cd4b9ce22284a9a943a150fdc768763caf035/asyncpg-0.31.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c204fab1b91e08b0f47e90a75d1b3c62174dab21f670ad6c5d0f243a228f015b", size = 661111, upload-time = "2025-11-24T23:26:04.467Z" }, + { url = "https://files.pythonhosted.org/packages/1b/71/157d611c791a5e2d0423f09f027bd499935f0906e0c2a416ce712ba51ef3/asyncpg-0.31.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:54a64f91839ba59008eccf7aad2e93d6e3de688d796f35803235ea1c4898ae1e", size = 636928, upload-time = "2025-11-24T23:26:05.944Z" }, + { url = "https://files.pythonhosted.org/packages/2e/fc/9e3486fb2bbe69d4a867c0b76d68542650a7ff1574ca40e84c3111bb0c6e/asyncpg-0.31.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c0e0822b1038dc7253b337b0f3f676cadc4ac31b126c5d42691c39691962e403", size = 3424067, upload-time = "2025-11-24T23:26:07.957Z" }, + { url = "https://files.pythonhosted.org/packages/12/c6/8c9d076f73f07f995013c791e018a1cd5f31823c2a3187fc8581706aa00f/asyncpg-0.31.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bef056aa502ee34204c161c72ca1f3c274917596877f825968368b2c33f585f4", size = 3518156, upload-time = "2025-11-24T23:26:09.591Z" }, + { url = "https://files.pythonhosted.org/packages/ae/3b/60683a0baf50fbc546499cfb53132cb6835b92b529a05f6a81471ab60d0c/asyncpg-0.31.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0bfbcc5b7ffcd9b75ab1558f00db2ae07db9c80637ad1b2469c43df79d7a5ae2", size = 3319636, upload-time = "2025-11-24T23:26:11.168Z" }, + { url = "https://files.pythonhosted.org/packages/50/dc/8487df0f69bd398a61e1792b3cba0e47477f214eff085ba0efa7eac9ce87/asyncpg-0.31.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:22bc525ebbdc24d1261ecbf6f504998244d4e3be1721784b5f64664d61fbe602", size = 3472079, upload-time = "2025-11-24T23:26:13.164Z" }, + { url = "https://files.pythonhosted.org/packages/13/a1/c5bbeeb8531c05c89135cb8b28575ac2fac618bcb60119ee9696c3faf71c/asyncpg-0.31.0-cp313-cp313-win32.whl", hash = "sha256:f890de5e1e4f7e14023619399a471ce4b71f5418cd67a51853b9910fdfa73696", size = 527606, upload-time = "2025-11-24T23:26:14.78Z" }, + { url = "https://files.pythonhosted.org/packages/91/66/b25ccb84a246b470eb943b0107c07edcae51804912b824054b3413995a10/asyncpg-0.31.0-cp313-cp313-win_amd64.whl", hash = "sha256:dc5f2fa9916f292e5c5c8b2ac2813763bcd7f58e130055b4ad8a0531314201ab", size = 596569, upload-time = "2025-11-24T23:26:16.189Z" }, + { url = "https://files.pythonhosted.org/packages/3c/36/e9450d62e84a13aea6580c83a47a437f26c7ca6fa0f0fd40b6670793ea30/asyncpg-0.31.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:f6b56b91bb0ffc328c4e3ed113136cddd9deefdf5f79ab448598b9772831df44", size = 660867, upload-time = "2025-11-24T23:26:17.631Z" }, + { url = "https://files.pythonhosted.org/packages/82/4b/1d0a2b33b3102d210439338e1beea616a6122267c0df459ff0265cd5807a/asyncpg-0.31.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:334dec28cf20d7f5bb9e45b39546ddf247f8042a690bff9b9573d00086e69cb5", size = 638349, upload-time = "2025-11-24T23:26:19.689Z" }, + { url = "https://files.pythonhosted.org/packages/41/aa/e7f7ac9a7974f08eff9183e392b2d62516f90412686532d27e196c0f0eeb/asyncpg-0.31.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:98cc158c53f46de7bb677fd20c417e264fc02b36d901cc2a43bd6cb0dc6dbfd2", size = 3410428, upload-time = "2025-11-24T23:26:21.275Z" }, + { url = "https://files.pythonhosted.org/packages/6f/de/bf1b60de3dede5c2731e6788617a512bc0ebd9693eac297ee74086f101d7/asyncpg-0.31.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9322b563e2661a52e3cdbc93eed3be7748b289f792e0011cb2720d278b366ce2", size = 3471678, upload-time = "2025-11-24T23:26:23.627Z" }, + { url = "https://files.pythonhosted.org/packages/46/78/fc3ade003e22d8bd53aaf8f75f4be48f0b460fa73738f0391b9c856a9147/asyncpg-0.31.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19857a358fc811d82227449b7ca40afb46e75b33eb8897240c3839dd8b744218", size = 3313505, upload-time = "2025-11-24T23:26:25.235Z" }, + { url = "https://files.pythonhosted.org/packages/bf/e9/73eb8a6789e927816f4705291be21f2225687bfa97321e40cd23055e903a/asyncpg-0.31.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ba5f8886e850882ff2c2ace5732300e99193823e8107e2c53ef01c1ebfa1e85d", size = 3434744, upload-time = "2025-11-24T23:26:26.944Z" }, + { url = "https://files.pythonhosted.org/packages/08/4b/f10b880534413c65c5b5862f79b8e81553a8f364e5238832ad4c0af71b7f/asyncpg-0.31.0-cp314-cp314-win32.whl", hash = "sha256:cea3a0b2a14f95834cee29432e4ddc399b95700eb1d51bbc5bfee8f31fa07b2b", size = 532251, upload-time = "2025-11-24T23:26:28.404Z" }, + { url = "https://files.pythonhosted.org/packages/d3/2d/7aa40750b7a19efa5d66e67fc06008ca0f27ba1bd082e457ad82f59aba49/asyncpg-0.31.0-cp314-cp314-win_amd64.whl", hash = "sha256:04d19392716af6b029411a0264d92093b6e5e8285ae97a39957b9a9c14ea72be", size = 604901, upload-time = "2025-11-24T23:26:30.34Z" }, + { url = "https://files.pythonhosted.org/packages/ce/fe/b9dfe349b83b9dee28cc42360d2c86b2cdce4cb551a2c2d27e156bcac84d/asyncpg-0.31.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:bdb957706da132e982cc6856bb2f7b740603472b54c3ebc77fe60ea3e57e1bd2", size = 702280, upload-time = "2025-11-24T23:26:32Z" }, + { url = "https://files.pythonhosted.org/packages/6a/81/e6be6e37e560bd91e6c23ea8a6138a04fd057b08cf63d3c5055c98e81c1d/asyncpg-0.31.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6d11b198111a72f47154fa03b85799f9be63701e068b43f84ac25da0bda9cb31", size = 682931, upload-time = "2025-11-24T23:26:33.572Z" }, + { url = "https://files.pythonhosted.org/packages/a6/45/6009040da85a1648dd5bc75b3b0a062081c483e75a1a29041ae63a0bf0dc/asyncpg-0.31.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:18c83b03bc0d1b23e6230f5bf8d4f217dc9bc08644ce0502a9d91dc9e634a9c7", size = 3581608, upload-time = "2025-11-24T23:26:35.638Z" }, + { url = "https://files.pythonhosted.org/packages/7e/06/2e3d4d7608b0b2b3adbee0d0bd6a2d29ca0fc4d8a78f8277df04e2d1fd7b/asyncpg-0.31.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e009abc333464ff18b8f6fd146addffd9aaf63e79aa3bb40ab7a4c332d0c5e9e", size = 3498738, upload-time = "2025-11-24T23:26:37.275Z" }, + { url = "https://files.pythonhosted.org/packages/7d/aa/7d75ede780033141c51d83577ea23236ba7d3a23593929b32b49db8ed36e/asyncpg-0.31.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3b1fbcb0e396a5ca435a8826a87e5c2c2cc0c8c68eb6fadf82168056b0e53a8c", size = 3401026, upload-time = "2025-11-24T23:26:39.423Z" }, + { url = "https://files.pythonhosted.org/packages/ba/7a/15e37d45e7f7c94facc1e9148c0e455e8f33c08f0b8a0b1deb2c5171771b/asyncpg-0.31.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:8df714dba348efcc162d2adf02d213e5fab1bd9f557e1305633e851a61814a7a", size = 3429426, upload-time = "2025-11-24T23:26:41.032Z" }, + { url = "https://files.pythonhosted.org/packages/13/d5/71437c5f6ae5f307828710efbe62163974e71237d5d46ebd2869ea052d10/asyncpg-0.31.0-cp314-cp314t-win32.whl", hash = "sha256:1b41f1afb1033f2b44f3234993b15096ddc9cd71b21a42dbd87fc6a57b43d65d", size = 614495, upload-time = "2025-11-24T23:26:42.659Z" }, + { url = "https://files.pythonhosted.org/packages/3c/d7/8fb3044eaef08a310acfe23dae9a8e2e07d305edc29a53497e52bc76eca7/asyncpg-0.31.0-cp314-cp314t-win_amd64.whl", hash = "sha256:bd4107bb7cdd0e9e65fae66a62afd3a249663b844fa34d479f6d5b3bef9c04c3", size = 706062, upload-time = "2025-11-24T23:26:44.086Z" }, +] + +[[package]] +name = "boto3" +version = "1.43.115" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "botocore" }, + { name = "jmespath" }, + { name = "s3transfer" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0e/4a/b0ce191492a1724be73d8aacaf70d508bc8cbb9674c046d2ff7181dc51fb/boto3-1.43.115.tar.gz", hash = "sha256:c0f35e5f9cdf0edb365195e306d5586d3970b224660a11a87a4ec361ec3d7168", size = 112677, upload-time = "2026-10-19T19:19:53.093Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/be/bc/40dffae3238ef8d3415393132751fe2c88e8663ef84a31f0950e439ed2d5/boto3-1.43.115-py3-none-any.whl", hash = "sha256:6667a0c21d48dc125dbba88444bc8a601ce881a99341a0e6169f9333898e620d", size = 140044, upload-time = "2026-10-19T19:19:51.444Z" }, +] + +[[package]] +name = "botocore" +version = "1.43.115" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "jmespath" }, + { name = "python-dateutil" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/6f/10/ca525e88db50f5153ddad2a3d55908b5ace915d10f443d5e5135ba808d8d/botocore-1.43.115.tar.gz", hash = "sha256:aa4d9c4e7ef99988b7347e0cd276ba84e1eb1625510557b0e36cd57ecb19bbf2", size = 16383796, upload-time = "2026-10-19T19:19:45.382Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/77/2d/4b6b6f5704d1ec9b412c1dd9700bda0c315619dbac4a21418180a81a2780/botocore-1.43.115-py3-none-any.whl", hash = "sha256:9ceda640c868bc15f5399129e8cfdef8cbb4b350e8a99547c9b5f477cfb9619a", size = 16085101, upload-time = "2026-10-19T19:19:42.203Z" }, +] + +[[package]] +name = "cffi" +version = "2.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pycparser", marker = "implementation_name != 'PyPy'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9e/ef/008a1939e372c06329a3fce4279c02f328488f3526744906eeec3da7ad5f/cffi-2.1.1.tar.gz", hash = "sha256:dd31f52ea1086513bb9df30f8fcee9b8918323ae067a3d5b78bc826a000712be", size = 530807, upload-time = "2026-08-03T21:21:18.939Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/69/43965eccfdead3b9220015fd1320e117be8c6ed01a62ffab76eeb752f5d5/cffi-2.1.1-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:c8c69575568085ba0b1b10c0249d779a214aea6f6522e949a0fc9fb0fcb449d0", size = 184821, upload-time = "2026-08-03T21:19:44.887Z" }, + { url = "https://files.pythonhosted.org/packages/54/7d/16e5a096677b5e313ca80cd5e5170efa3ea44624a82bb111925522da64b1/cffi-2.1.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f81b3b8f3d4e343550fa4baa0e479bba9f2d29ce9c2e9b51d1ce1718d7442fcf", size = 184719, upload-time = "2026-08-03T21:19:46.129Z" }, + { url = "https://files.pythonhosted.org/packages/56/e6/8941622732edec876dd17d0453dce07317ae96db34f2ec1436c9d3785986/cffi-2.1.1-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:811bd1e21d32de12efca32393a0ab3f5133b54fce9bd44b8bd77ab07da14bf6a", size = 214799, upload-time = "2026-08-03T21:19:47.218Z" }, + { url = "https://files.pythonhosted.org/packages/44/de/f98430906df1545ffde0d543dd124a7a439bc2cd32b36b9c53f805df7333/cffi-2.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:68e62fe11f30d5ca8289242866f0a5291402d8529ca2178ab8afc5c9694ae890", size = 222389, upload-time = "2026-08-03T21:19:48.331Z" }, + { url = "https://files.pythonhosted.org/packages/6a/5b/717f1526b9957b34456313c31645c5b82b8fb5c3fe9e4752999be7128bfc/cffi-2.1.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:4a7c934f7360e8cd64fe9efadcbd10c7c6364f531e432b9a4bf5ccbc9e0e8b50", size = 210249, upload-time = "2026-08-03T21:19:49.543Z" }, + { url = "https://files.pythonhosted.org/packages/64/b3/f8aa4f3e34986c7e4ec45072d1b1b9dd295b6b18007b45518d79726dd725/cffi-2.1.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:3143d81e29e1e20a9ce10901ec369012947876596f75a222235965f2b7ae832e", size = 208775, upload-time = "2026-08-03T21:19:50.918Z" }, + { url = "https://files.pythonhosted.org/packages/b1/db/dceb9dd5b231e1da801793f8acc9f3c52a7e1afe40bb1aae37e02b0faad5/cffi-2.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c1453022f490d2459a11819d83ad1d586e9ff65a12ac3e705ffebd46d3685dcf", size = 221822, upload-time = "2026-08-03T21:19:52.054Z" }, + { url = "https://files.pythonhosted.org/packages/a0/d2/6cd24ae3be000a634109c247d1475d62e5616d0dc78c82770942ec384248/cffi-2.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:208f941bb9d18e768138677f0a6d2ce01f590df56043dda1df1535ac57c88517", size = 225232, upload-time = "2026-08-03T21:19:53.109Z" }, + { url = "https://files.pythonhosted.org/packages/cb/52/3fa190537004dd7f0ab860a6dc7c0175b8667f68d1e618a46f5498d30250/cffi-2.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:210019b6c7cf07f081b4c54635c8cf744377001350e29cc0f81c4377b4797735", size = 223597, upload-time = "2026-08-03T21:19:54.515Z" }, + { url = "https://files.pythonhosted.org/packages/80/fb/0bb75b7039588c074b37ae99f40d9bfddf990ecb2fbc346ebccd2e56b9be/cffi-2.1.1-cp312-cp312-win32.whl", hash = "sha256:046bfc24911b37851ee1b51aab8bffe713d89c68c6a057b09484ce9fd5f69b4e", size = 175292, upload-time = "2026-08-03T21:19:55.566Z" }, + { url = "https://files.pythonhosted.org/packages/d9/79/615cc094e2fb508cade7de88d3b4f6c4ec2bab695c97bce9153dc65aadf5/cffi-2.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:f53e442b08449d42821fa4a4fba000095af9f62742a500f978a9f557ec44339a", size = 185919, upload-time = "2026-08-03T21:19:56.89Z" }, + { url = "https://files.pythonhosted.org/packages/70/c6/d0ea84713fe46b243a436a18fcd47d639732747e21635c8a27191b06dc30/cffi-2.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:7bde5e4cc5c10140859842b9d383af292b22639a4dffb725314baf45968cef80", size = 180093, upload-time = "2026-08-03T21:19:58.155Z" }, + { url = "https://files.pythonhosted.org/packages/9d/f4/035513d4117049066b4779dc3b7c0c0fdad175fa13731c9f4003f1cd1478/cffi-2.1.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:b5bdfd1c873d4e093aabc0ca84c4ca6dbc4f752afb5c86f146d9742580c9da2e", size = 194248, upload-time = "2026-08-03T21:19:59.399Z" }, + { url = "https://files.pythonhosted.org/packages/76/af/2aeb4dbb5fc41a04161ae9ff1518de7cec08e164f44a8ce6a4cf7fd2cd1d/cffi-2.1.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:31348097ff5bbe827ccc41795d4dd099d9f0625e7def00ee653c137a490c2a6c", size = 196908, upload-time = "2026-08-03T21:20:00.746Z" }, + { url = "https://files.pythonhosted.org/packages/a7/46/2e5fdde8555706dd98139a910ca11be02809f3f605ce956f655d0214e100/cffi-2.1.1-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:9d2055050ea716bd38b7f7f1579c275386646b4894c155a3e2f3cd62ed41b7c6", size = 184805, upload-time = "2026-08-03T21:20:02.02Z" }, + { url = "https://files.pythonhosted.org/packages/55/41/4c7042f317b9217502988f0873af87e16ad606dc20f84e546e3e6ce9764c/cffi-2.1.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:19ee6127ee34de7d83ce3d371ebc5ed91addbdcc39f9ab15ce4eb35a4e534971", size = 184764, upload-time = "2026-08-03T21:20:03.141Z" }, + { url = "https://files.pythonhosted.org/packages/43/1f/1c3d90d91811c8f86ced9ed637956c54bfe5b79ca98fe976d7f8c8979f6b/cffi-2.1.1-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:6a8dddef476fab96d066d578fc88526767b836ab5ab21754e1d5bf3879c31c7c", size = 214722, upload-time = "2026-08-03T21:20:04.377Z" }, + { url = "https://files.pythonhosted.org/packages/37/6f/3b5ce4c3b2192d250f04908f2bfd91ef34552ec8f7716a5d4abdb8d67bb2/cffi-2.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f16c709686a78c727bbbf059f92b0bf41c6fc60deec706d2dc19f529175a6125", size = 222369, upload-time = "2026-08-03T21:20:05.544Z" }, + { url = "https://files.pythonhosted.org/packages/02/10/4b3c75dde3d9663c9e02ba05c2668b954f671d4bbe346413ca8c696b295a/cffi-2.1.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:fcd22650c908d7b7da162bbfaab594a1227a15d1643a98c68b122ac642fa2264", size = 210175, upload-time = "2026-08-03T21:20:06.75Z" }, + { url = "https://files.pythonhosted.org/packages/df/62/14f74b9543e605d17701dc797b815958b8bb70b7624ce1b832ddad48ed6c/cffi-2.1.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:aa9511c62d14da7aacc9b4bf51f3f697a621e83b2d6919008243c3aad168eea3", size = 208670, upload-time = "2026-08-03T21:20:08.04Z" }, + { url = "https://files.pythonhosted.org/packages/95/95/86342356ff5953b3fb06f7ef7c5bee212d45e770abc7218d451b9148313c/cffi-2.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a931079504ecc49efed7744c476a5c343a92fabf66dec2db95edb1b2fdc770e2", size = 221824, upload-time = "2026-08-03T21:20:09.274Z" }, + { url = "https://files.pythonhosted.org/packages/eb/ff/7b3429ff53aafe931ed8a5fc69f481bbef7ba6de87ddcbb63d08f483f613/cffi-2.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a2d7755bef5a12ed488f4ef1f1b69ee9191d7396083b755a5d2295f6edb4768b", size = 225148, upload-time = "2026-08-03T21:20:10.7Z" }, + { url = "https://files.pythonhosted.org/packages/34/34/a95870b9221e09cf4f2ce3178b1a210abdfe63a1bd357da940418d7b8d15/cffi-2.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e0bcb7e0f677f543555d2adff3bf19c05f66cdb4796e5ff602442ab2fe3c4ef7", size = 223564, upload-time = "2026-08-03T21:20:12.165Z" }, + { url = "https://files.pythonhosted.org/packages/70/ea/839b50531021a647fb5e929f72cf97bc1ff702b5472166164b5b6e76b851/cffi-2.1.1-cp313-cp313-win32.whl", hash = "sha256:334644fbac4eff73d985a17a91226df55d0f394160c4cfb880e084c8f7161cac", size = 175263, upload-time = "2026-08-03T21:20:13.559Z" }, + { url = "https://files.pythonhosted.org/packages/60/a6/8b149b2c3f2e11aaa1618ef64500b45f50f22c57a977a4dff1aff1f91042/cffi-2.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:1aa5645c30469b09530c4ebca77ebf8f17618293c58f8549cb1a543a50236e7d", size = 185688, upload-time = "2026-08-03T21:20:14.69Z" }, + { url = "https://files.pythonhosted.org/packages/01/9a/11f687cb39d6a3504060d5242f04f48c735afb4d3d533958a20594890cb2/cffi-2.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:63bbfd5ded17c4840ac07cd8f1c21ba9d9708141f840b324f422f41b207e3973", size = 180078, upload-time = "2026-08-03T21:20:15.917Z" }, + { url = "https://files.pythonhosted.org/packages/d3/7b/d6bbf82b8b96e7391438898c42f5bd96dd02030fd5b64937d248220003e2/cffi-2.1.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7dbb61fe3a7699468030f71bbe5f8a0e326a151daa91beb11a6fc1f980c55e1c", size = 194064, upload-time = "2026-08-03T21:20:17.148Z" }, + { url = "https://files.pythonhosted.org/packages/94/e6/bcc91b283be94735e268487a054004f0aa19947b6348fa367db53230abc8/cffi-2.1.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:f24fb43132a4c6b4cb4eb029492919b2db645be6808d738f244fd146c03c32cb", size = 196720, upload-time = "2026-08-03T21:20:18.268Z" }, + { url = "https://files.pythonhosted.org/packages/d9/99/c4b0c17cacdc9c3b8f280026286a9826d6a208c0f047591a3c3ce99b91fd/cffi-2.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d28630f5854ab07ab1fd4aba756de52326c82e6be15d414b12793f1975048b54", size = 184964, upload-time = "2026-08-03T21:20:19.708Z" }, + { url = "https://files.pythonhosted.org/packages/b3/a9/9db617d05d7367c1ad0ab00b3aa6e6f9281edd689b4ee9ea0e5a84e89c97/cffi-2.1.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:661c298b4821edebead0c91edd2b00374d67ad7c5a1f7a91d4442633b79d6a72", size = 184962, upload-time = "2026-08-03T21:20:20.833Z" }, + { url = "https://files.pythonhosted.org/packages/67/b8/b42132ca113dc567d37684437b46ca1dafc885902b02a110a02d5b511857/cffi-2.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:58acb8ab8e295e6c5ea12f888cbb13cf21511ef2a3303a23f4325c29d17fe5c1", size = 222328, upload-time = "2026-08-03T21:20:22.118Z" }, + { url = "https://files.pythonhosted.org/packages/80/10/c5c0cbf0a657aecf59ef511409734230bf556f05a0d6c9eed7aa5c0a0166/cffi-2.1.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:456a61fa52d579ebf9df2e9552ead5129855dbaff6c1e5a9b1bc408809bdc062", size = 209985, upload-time = "2026-08-03T21:20:23.401Z" }, + { url = "https://files.pythonhosted.org/packages/d5/6c/bfa0b87b03b9238148beca990292843c9396ba069b54496596594173de7b/cffi-2.1.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a4f00aa42f75d6e4595e8866e748cc1705adc0cddfeb2ca86d0d03993d63ba03", size = 208530, upload-time = "2026-08-03T21:20:24.628Z" }, + { url = "https://files.pythonhosted.org/packages/e9/02/4e7d553a7ac4b4238b38b3c1b80d486e9d4436f8d2acbf87a0997fe3f402/cffi-2.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b0431303acaea1089ad4b3e9ce4e6518193def1118d4073ca848635ee4ea2e96", size = 221525, upload-time = "2026-08-03T21:20:25.758Z" }, + { url = "https://files.pythonhosted.org/packages/82/1d/a4aaf9babd75acb4d5f223bff71533bee748dd770a382619a798960ee9ba/cffi-2.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:64faea20f4e2613363a1a9b9c7dd73058f3ecd00133a511e72ad7c511658f527", size = 225053, upload-time = "2026-08-03T21:20:26.985Z" }, + { url = "https://files.pythonhosted.org/packages/81/10/5dc0e7bdd18e22107054288283380fc97a06ae3f1656a106908d666a3c88/cffi-2.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5c58fe613dc5e5336357eff555824a314d8e43282600435c8d1cb6a7a2fedd13", size = 223213, upload-time = "2026-08-03T21:20:28.277Z" }, + { url = "https://files.pythonhosted.org/packages/0b/e9/d0061c364cde06ee43168a0d076ac1da512cbc380d44767b844ba34fe2b6/cffi-2.1.1-cp314-cp314-win32.whl", hash = "sha256:1a18a57b58cfb21fc28d72e876acf10eaed67a1ed96226f92af4df681d571c4c", size = 177682, upload-time = "2026-08-03T21:20:44.288Z" }, + { url = "https://files.pythonhosted.org/packages/a7/06/1c3e01e3ba14c39f6d10bfbac52753b7e22259e38088e5cfe1d704918690/cffi-2.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:3222ba5d678f80a030e6afbcc33dc1ae5cb45facabb61cee2c7016b8432fde48", size = 187949, upload-time = "2026-08-03T21:20:45.623Z" }, + { url = "https://files.pythonhosted.org/packages/87/5b/da4e39efe18eeb89cf580ea9cfc66b6a7c3eadb808fc0cc1d3a295cb5a5d/cffi-2.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:ab36d55f9ed2d067327667c2fea18dda018eb628dd6347aa01dda6cf1f5d3836", size = 182947, upload-time = "2026-08-03T21:20:46.955Z" }, + { url = "https://files.pythonhosted.org/packages/23/59/40338bf421c5accea1d45158170c87006ef1cd371b05c077e76476949728/cffi-2.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7750c6449dff7864bb9bb27ddfb0267756189201a3afc911d82b3caacd70dfc3", size = 188504, upload-time = "2026-08-03T21:20:29.495Z" }, + { url = "https://files.pythonhosted.org/packages/7d/47/5ecf1023850036e674c77ec4de86182d309ae344e39e7cba984b7df5d647/cffi-2.1.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:0beceaabe56af686895136a2de78db54ecd8e4046b236b8fd6d6cb61389e9bf2", size = 188259, upload-time = "2026-08-03T21:20:31.291Z" }, + { url = "https://files.pythonhosted.org/packages/2a/9c/92934c3bea9f785b23eba304538c0b4d37a2a96d2431eb3a1bc87a11aa19/cffi-2.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:49cbc70e6542d4ccccb936558d1064a8012541e78f821f955cff24e357776c94", size = 223864, upload-time = "2026-08-03T21:20:32.571Z" }, + { url = "https://files.pythonhosted.org/packages/4d/45/ba4c93527bc38616a8bd36488acb69a2212d60486794f0c1f318949bbb76/cffi-2.1.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:e2d65b31f36619cda3999b78b2aa9632e76b78448e7a56fc4240824200e7c4fc", size = 211538, upload-time = "2026-08-03T21:20:33.808Z" }, + { url = "https://files.pythonhosted.org/packages/80/e9/b6ef565e452acb932fb0cb5443f44a78efbd1233e566f02b5a83855e9115/cffi-2.1.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:28907ab9bfb6aa13184cfc17c6b8e1023c5ab6fd7076d8c20a35e59fe04f8f29", size = 210688, upload-time = "2026-08-03T21:20:34.974Z" }, + { url = "https://files.pythonhosted.org/packages/9a/95/eff5f0cee78d2eabc7eebffec40d3fc1876b5f3c95582e018bb4b99601f2/cffi-2.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:51b31d1c98274844cfd7838ce00bfc27c7423a4dc00fc0772fc3331c2cc90676", size = 223803, upload-time = "2026-08-03T21:20:36.564Z" }, + { url = "https://files.pythonhosted.org/packages/fa/01/579d39fb8bef00a335a23d83757b44feb24cd6345a2c451b64cb67b9c362/cffi-2.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5e7cecbaadb83884793e05828cee59b210b24583b9c7425d0ba6a754fe22eb4e", size = 226763, upload-time = "2026-08-03T21:20:37.816Z" }, + { url = "https://files.pythonhosted.org/packages/8d/b0/0b44f47c60b01b57b6e2bbd92343f13a85a1d93bc46ccf6e47e244acd99c/cffi-2.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:25792eac27877609e7bb06d42ff88278a6624fff2ba9bbb523c09616b117e80f", size = 225688, upload-time = "2026-08-03T21:20:38.959Z" }, + { url = "https://files.pythonhosted.org/packages/eb/d2/3b7176cb570a1d3e27faf67b72f591af508036e0d8b2be2ef9af9e8c84bb/cffi-2.1.1-cp314-cp314t-win32.whl", hash = "sha256:8ef53b2de9bcb9197d31854256575d59dbac0cba72ac627bb291ef5eceb74be4", size = 182868, upload-time = "2026-08-03T21:20:40.388Z" }, + { url = "https://files.pythonhosted.org/packages/56/78/31f00c1bcd97c9bbf55f1bfdf5bc809a5de8887473e90bb9960dca825e80/cffi-2.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:616f097f2fe415bc92a247f02e11f634e1f9e9a83d327e3c915c15089c87869e", size = 194104, upload-time = "2026-08-03T21:20:41.725Z" }, + { url = "https://files.pythonhosted.org/packages/7b/1b/58496f2ed0a35de575250c02a43ab3cc2c04d494a88fed31c1cabc0fd176/cffi-2.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:ad2c86c495b899d862ea0f4b42891b8713a3bd45dd4105c7fd51c2a72f39f3a5", size = 186402, upload-time = "2026-08-03T21:20:43.042Z" }, + { url = "https://files.pythonhosted.org/packages/c1/8f/9ebe220eab48a093d1a5a5e339ab0dc7316eef3bb04d63c42f0251b61f50/cffi-2.1.1-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:dddad92b554513a31f272570678ba307fb9f618f05e3d4a5eacafff9eae03e1d", size = 194043, upload-time = "2026-08-03T21:20:48.179Z" }, + { url = "https://files.pythonhosted.org/packages/ff/69/844bad3ece306c4782c2ecb93597035b6690d48704b803914c199da1e8b3/cffi-2.1.1-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:da0e573f9f97159390c89d9f1a9e41908b66d408cc5b58d08cf3847d844c531b", size = 196737, upload-time = "2026-08-03T21:20:49.457Z" }, + { url = "https://files.pythonhosted.org/packages/1b/8a/af668013284634733f02d683458a0728739c7d6ddb5e14cb0c20832266fe/cffi-2.1.1-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:fb92203a88b3d3053034db775110081c49d28be6551923805e039924093761e4", size = 184933, upload-time = "2026-08-03T21:20:50.639Z" }, + { url = "https://files.pythonhosted.org/packages/0c/75/2f5207ff6d1a613133b23a5203cc0c2a628313b5eb3974d7956ae3c57950/cffi-2.1.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:2ae64be792b8966f2c69538199728b290e34726562896df1e5dc8ffd8d8188e8", size = 185002, upload-time = "2026-08-03T21:20:52.173Z" }, + { url = "https://files.pythonhosted.org/packages/e2/31/9e1313b0a6e30e91b3b3d3fff51ae99c857c07738e3afcce1f7334e1b7ab/cffi-2.1.1-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:507a24c282e0f42f8ed737cf048572cbf580468da5555764a8331735e9c736b6", size = 222271, upload-time = "2026-08-03T21:20:53.462Z" }, + { url = "https://files.pythonhosted.org/packages/50/e3/f6234a833e6e08c7007003074723c406559eecf9b48dfc97471e5a8eb7a0/cffi-2.1.1-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:246fa40ce8645a614ff682e0b70f37134e460eaf93a775e0cbe3cca585a67a80", size = 209919, upload-time = "2026-08-03T21:20:54.783Z" }, + { url = "https://files.pythonhosted.org/packages/0d/fc/5f74e293fced6edb51af3a46c4ccf6c23c9943774ecb375ddbd522c76add/cffi-2.1.1-cp315-cp315-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:471cee653ae88de62096552e6d24ccb4a5adb8c8c9f10b5054d0122c15bf2779", size = 208529, upload-time = "2026-08-03T21:20:56.066Z" }, + { url = "https://files.pythonhosted.org/packages/44/16/29e6d01b388bef055ecd6ca8244b3f4d336bd09e92d5d892187b9601084e/cffi-2.1.1-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aeae0e330c9f6acd681f647d46cefd30c29f93e3392882e792e82080c9691399", size = 221630, upload-time = "2026-08-03T21:20:57.336Z" }, + { url = "https://files.pythonhosted.org/packages/a4/18/fa7f1f6857d5eb88a4ca99ffcbfb7c387a287ccc154c64a73e86314745d7/cffi-2.1.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:42a494cee34437f05546455144f2b5d9ac09b1face62bcfce597d2e521066688", size = 225134, upload-time = "2026-08-03T21:20:58.675Z" }, + { url = "https://files.pythonhosted.org/packages/e0/9f/e8e3dfa04a1b4c241f8c91faacad872b4d4efd051d49764ad4e2fd4b9fea/cffi-2.1.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:cc572dace3f60ef98d7b12ff411d20f5362feb31a0439eab0085bbfd349982d7", size = 223197, upload-time = "2026-08-03T21:20:59.968Z" }, + { url = "https://files.pythonhosted.org/packages/f8/7e/8debeb04f1ab9fe2a6963964cd6f1aaf7192627b83926586a6a4e089c9fa/cffi-2.1.1-cp315-cp315-win32.whl", hash = "sha256:4f42141fc14250de6dde5ee7ea4432be017252d91f19c5ad043c084cea629cac", size = 177683, upload-time = "2026-08-03T21:21:14.901Z" }, + { url = "https://files.pythonhosted.org/packages/e0/31/5158704cc474ab65c1647932e88be78dc0873f47130e253be38bcaf13d01/cffi-2.1.1-cp315-cp315-win_amd64.whl", hash = "sha256:e6e8cff14d6fb0be70a09c0bdc58096f501952d04624ebf867e0e56da2df8960", size = 187897, upload-time = "2026-08-03T21:21:16.108Z" }, + { url = "https://files.pythonhosted.org/packages/cc/4b/b3a2da8570c704ffc0f9762cdc3ec0f02c8573798e0b5cf7f11c82bbb70f/cffi-2.1.1-cp315-cp315-win_arm64.whl", hash = "sha256:27350daa11d4f10c540e6e89dada4c54feb7256ad03e9a4dc075ebad7ba360d1", size = 182935, upload-time = "2026-08-03T21:21:17.271Z" }, + { url = "https://files.pythonhosted.org/packages/d0/ef/5443574510a1207e6f6bc38ba6e1f1de36cb48fef07b2728bb896a21f430/cffi-2.1.1-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:c26608d2222fb1e94487e4a387d85f13eb55d5ed725cb25a0c589ac4ee60e7bc", size = 188464, upload-time = "2026-08-03T21:21:01.163Z" }, + { url = "https://files.pythonhosted.org/packages/7e/ae/a56fa8c4686ad50e148fcbc8d3ae0d03915ff5c30d795058988c24118cef/cffi-2.1.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:4be96343e422f2dfcd12ab5c9f5aebe03f82f737c6bffeca6830b3875cb44aab", size = 188262, upload-time = "2026-08-03T21:21:02.382Z" }, + { url = "https://files.pythonhosted.org/packages/53/b2/6187f46f2912276a3ae284076109cc5c8680482f11f766ccf26db4a86427/cffi-2.1.1-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:937c0052c05a31ca1daf18de3158eed4dbfcb9cc107adbea227728d647be701e", size = 223779, upload-time = "2026-08-03T21:21:03.553Z" }, + { url = "https://files.pythonhosted.org/packages/8a/f6/c3ad28bd19f77047a03084424fbd4cbe997303267c14423737324be0385d/cffi-2.1.1-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:df423d40ee8654634421812bc3b196da3f9bd7d32929da813f8394c4348a5358", size = 211520, upload-time = "2026-08-03T21:21:04.863Z" }, + { url = "https://files.pythonhosted.org/packages/a0/cd/ccac9013a5bd9fd764de118674ab9c805b5ca10c19270d90ee273f8b2240/cffi-2.1.1-cp315-cp315t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:a730a083190634c65cca36ba5f489531576ebd79bcd5c8e172130f6453127231", size = 210673, upload-time = "2026-08-03T21:21:06.223Z" }, + { url = "https://files.pythonhosted.org/packages/52/86/2976131c639aead931c5bee5aba67e4b09fbeb8018b6f282f70803f923a7/cffi-2.1.1-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:363e05fa78e15116c3c32c210ee36884fd6b9afa6d440e47112c3bd511d64cb6", size = 223835, upload-time = "2026-08-03T21:21:07.539Z" }, + { url = "https://files.pythonhosted.org/packages/ac/0c/33a7aeab2f9c76918c52e084beb39c570db3588133412929e8ec06fab90b/cffi-2.1.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:770de9db11e84213beec501cfcaa013b019820ca881e03344dea5844f7876d94", size = 226705, upload-time = "2026-08-03T21:21:08.774Z" }, + { url = "https://files.pythonhosted.org/packages/e3/26/2cde30fdde421130bfc18f70395731a6e6b2053c6a1978a5258ff04e72fa/cffi-2.1.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:7da0c5eff80f0197f3b3d1232ec5a682a9325f4ae9016a78f5f5ca35f9ced1f5", size = 225539, upload-time = "2026-08-03T21:21:09.911Z" }, + { url = "https://files.pythonhosted.org/packages/6d/cd/a361394c94b2129d604bb846f624a8e88255a3ee33129c434a00d715e64f/cffi-2.1.1-cp315-cp315t-win32.whl", hash = "sha256:06c72bb76605a4b0cd0aad6930b69d4baf7dd5d806cfc409b824191099700e66", size = 182707, upload-time = "2026-08-03T21:21:11.226Z" }, + { url = "https://files.pythonhosted.org/packages/9b/b5/ba2b299993c26577d529b6ae29841f9e15b9fcf004d65f423f4fcf94ade9/cffi-2.1.1-cp315-cp315t-win_amd64.whl", hash = "sha256:d9c275eaacd24aa73f94ffd6de08fc3f932424d8b6c376f4bed7cde376fe7bc3", size = 193772, upload-time = "2026-08-03T21:21:12.39Z" }, + { url = "https://files.pythonhosted.org/packages/aa/29/35e016098c814cd93de9cd320c66b5bfba14dc6ecedd3cb518fa7c408c69/cffi-2.1.1-cp315-cp315t-win_arm64.whl", hash = "sha256:d18e5ac0f2f03f4f518d3e23db0f0cad7faa1da8620e9c09461d443bbf6e6692", size = 186360, upload-time = "2026-08-03T21:21:13.636Z" }, +] + +[[package]] +name = "click" +version = "8.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/0e/7fa0ef50764b67090eca4114772a2abf8b6148198475e54c660b97caeee6/click-8.5.0.tar.gz", hash = "sha256:ba0d2089de75ea0310e2dde03160e6ca10009947fb95a182f9b54021bb272e34", size = 382235, upload-time = "2026-08-26T13:33:14.56Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/58/50/6c0d534c5f134586a8e1ba4e330569e32f057e33372ae556463212fb4cd3/click-8.5.0-py3-none-any.whl", hash = "sha256:255bc9599cf7748b4b1a446ccc735421bd08a2ae529a8b88597d3de5664ee360", size = 125251, upload-time = "2026-08-26T13:33:12.928Z" }, +] + +[[package]] +name = "fastapi" +version = "0.141.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "pydantic" }, + { name = "starlette" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8a/02/91e3416a8fdd715abb903a952a6bec7cdd8d14eed55d415fc8595524c319/fastapi-0.141.1.tar.gz", hash = "sha256:e8822fc40db1e1858054d7a949a888695bc9bdce70139178e33bd2871a453ca1", size = 425799, upload-time = "2026-07-29T17:18:05.568Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/03/10388a42375ee7e4ac9b94eb2c5c569c8b5795e377e701c9ac3ad63de890/fastapi-0.141.1-py3-none-any.whl", hash = "sha256:bfb91aa2d334c61cb35ba9a116fc123b3d3df31640b801cf57a7a78ec3f603b3", size = 131954, upload-time = "2026-07-29T17:18:04.364Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "idna" +version = "3.20" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f5/08/8eea9d4b8302028f3abb2c0813953f7aec26d33b7a8960ed760e65ff29fa/idna-3.20.tar.gz", hash = "sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44", size = 216463, upload-time = "2026-09-17T14:11:04.752Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/58/a2/bb081bab032533a855d44de1d56f8e8426114ff1ba5d1f07a438a0a654f8/idna-3.20-py3-none-any.whl", hash = "sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c", size = 69583, upload-time = "2026-09-17T14:11:03.168Z" }, +] + +[[package]] +name = "jmespath" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d3/59/322338183ecda247fb5d1763a6cbe46eff7222eaeebafd9fa65d4bf5cb11/jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d", size = 27377, upload-time = "2026-01-22T16:35:26.279Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/14/2f/967ba146e6d58cf6a652da73885f52fc68001525b4197effc174321d70b4/jmespath-1.1.0-py3-none-any.whl", hash = "sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64", size = 20419, upload-time = "2026-01-22T16:35:24.919Z" }, +] + +[[package]] +name = "mtusi2026lab4" +version = "0.1.0" +source = { virtual = "." } +dependencies = [ + { name = "argon2-cffi" }, + { name = "asyncpg" }, + { name = "boto3" }, + { name = "fastapi" }, + { name = "pika" }, + { name = "pillow" }, + { name = "uvicorn" }, +] + +[package.metadata] +requires-dist = [ + { name = "argon2-cffi", specifier = "==25.1.0" }, + { name = "asyncpg", specifier = "==0.31.0" }, + { name = "boto3", specifier = "==1.43.115" }, + { name = "fastapi", specifier = "==0.141.1" }, + { name = "pika", specifier = "==1.3.2" }, + { name = "pillow", specifier = "==13.0.0" }, + { name = "uvicorn", specifier = "==0.40.0" }, +] + +[[package]] +name = "pika" +version = "1.3.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/db/db/d4102f356af18f316c67f2cead8ece307f731dd63140e2c71f170ddacf9b/pika-1.3.2.tar.gz", hash = "sha256:b2a327ddddf8570b4965b3576ac77091b850262d34ce8c1d8cb4e4146aa4145f", size = 145029, upload-time = "2023-05-05T14:25:43.368Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f9/f3/f412836ec714d36f0f4ab581b84c491e3f42c6b5b97a6c6ed1817f3c16d0/pika-1.3.2-py3-none-any.whl", hash = "sha256:0779a7c1fafd805672796085560d290213a465e4f6f76a6fb19e378d8041a14f", size = 155415, upload-time = "2023-05-05T14:25:41.484Z" }, +] + +[[package]] +name = "pillow" +version = "13.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ce/a4/9cc30690a394a8305e8be4c9036aa16d0dfddadd09079860b582f562496d/pillow-13.0.0.tar.gz", hash = "sha256:cdcd1ba258178f7356d74170c9a81ba4eab6a5d55748bc402a5a2a6fe6be3510", size = 47058631, upload-time = "2026-10-16T22:18:09.718Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7a/b8/58d79332229482eb0d27c82973850808c19d37e5bdc14443d3417bd373f5/pillow-13.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:67c8d44ad89fa7eb88de5d6ef6199b1e03dd24974f85a67639fc8badb801fe79", size = 5568643, upload-time = "2026-10-16T22:15:44.474Z" }, + { url = "https://files.pythonhosted.org/packages/39/4b/f2ed53d83aa44ac19599ac0a54897f867fafb95f406980d1e24083f8cbe3/pillow-13.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc481f742ccda20e9ed760b8f978dff50f2a4e9bc253ac167fb0a05b0783ef52", size = 4852329, upload-time = "2026-10-16T22:15:46.284Z" }, + { url = "https://files.pythonhosted.org/packages/3b/bb/9e30e8ed18c20800ee327a12107d5249ad8dc444867b3adfcc21654e3c71/pillow-13.0.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8bf295fe459343bc6093c8e5e0df60bfa189e0fcb775a22e2c2488fdeccc6c0d", size = 6519309, upload-time = "2026-10-16T22:15:48.14Z" }, + { url = "https://files.pythonhosted.org/packages/e6/99/fdf65cc539e266ab93546c261bdf98411fe165c6e4e13f9dbf597097192b/pillow-13.0.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:42d63be3af4f9b951b199b0a9d23365d37ab14687e6d77b74940432c8515c4ca", size = 7144542, upload-time = "2026-10-16T22:15:50.093Z" }, + { url = "https://files.pythonhosted.org/packages/26/9b/f38b7c4d8f599537e2d1a6d546cec9beef7536ff8f5c49a3f9c78179d8ac/pillow-13.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4bd024d567eca931b668978d672dba914d4b8e2628e2c3073a2ad178de8a2620", size = 6618042, upload-time = "2026-10-16T22:15:52.054Z" }, + { url = "https://files.pythonhosted.org/packages/a8/7c/45ee28531f89ee4582967d8a17f7a26d71ca64cd902b93c269590cd60a00/pillow-13.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:23edf4ec4b6f4f240e8df6e4985e9ee2ced86474632433e55fb1e2548744bf40", size = 7228137, upload-time = "2026-10-16T22:15:54.009Z" }, + { url = "https://files.pythonhosted.org/packages/01/bc/9c2a969cb1b70201d04fa17fd5aedf02a732870b8da86bceaaa30b0ca385/pillow-13.0.0-cp312-cp312-win32.whl", hash = "sha256:02464da86f561f410e13b061ec37c3068a9cb557a095494157742d07b3c24a27", size = 6515423, upload-time = "2026-10-16T22:15:56.25Z" }, + { url = "https://files.pythonhosted.org/packages/ff/cd/73a79ecdc822076002eb24b5e1e4a530b0bb5310fcb6d239abbec877110b/pillow-13.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:ff3fe9eb621fbb9db4c4be33ab946f6bac5f98ab50ca8d3260503653656c961b", size = 7272166, upload-time = "2026-10-16T22:15:58.597Z" }, + { url = "https://files.pythonhosted.org/packages/4a/a3/be5a1236efd085b3984e4305da8f179ae9dab80c991d2ad78f2bd5c4c843/pillow-13.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:c3bc8ee0271387e0b1e3ad0ff893645990a656e79bf9c42f4f734a3642b11d51", size = 2698943, upload-time = "2026-10-16T22:16:00.637Z" }, + { url = "https://files.pythonhosted.org/packages/82/e2/9132d38df320e8099e2642d9fa538f18a7dfebf68a875a31725dc2a115aa/pillow-13.0.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:e98b4f91f99b24b1da72e10621ba181f7281922384a7403094debb4199de6c00", size = 4190620, upload-time = "2026-10-16T22:16:02.527Z" }, + { url = "https://files.pythonhosted.org/packages/cb/be/436dac0996876f65ef1916ffb711c5ef4af2ebc57414b6f16642e07f6b47/pillow-13.0.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:34fb3db1149c879343861831728c243c432d4a5afc8f1878ca7612ef8de2a033", size = 4290286, upload-time = "2026-10-16T22:16:04.861Z" }, + { url = "https://files.pythonhosted.org/packages/60/16/7c1007162e4169ea0d5bec47e2e0b2174f56c50cf5b8c02a1dc23e41037e/pillow-13.0.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:2d50500a4a3bc6469adcdb319cf05d688b08ca3bf0f22a05339f3b25b433a599", size = 3723169, upload-time = "2026-10-16T22:16:06.956Z" }, + { url = "https://files.pythonhosted.org/packages/08/bc/578ab2eaa04705a8978bd9be0fe15c3d43a8fc95097c4f7e39540e79e4fa/pillow-13.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ef5fe6b7db3f501c56000ba8fe7c7e1a35ca1d45824fbeb95a67daa85cff4dbe", size = 5568469, upload-time = "2026-10-16T22:16:08.664Z" }, + { url = "https://files.pythonhosted.org/packages/98/c8/ccede3dcc4f274b80fb49199eed1ef34ba05566c8b71976b9d1f111da2e3/pillow-13.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f11a3c1358bb3c68012a7b1f41b223c3db9f7bfde90b481df028cfe1c3363313", size = 4852187, upload-time = "2026-10-16T22:16:10.693Z" }, + { url = "https://files.pythonhosted.org/packages/fb/3c/cd0770e96d20bfd990db024bcbd43897ff3e88da1f4782cf90ea231b1d4f/pillow-13.0.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:19b66ca0a4d25cc0200b381ce824c7ea090db61427315e73cfb05dbf1b35c369", size = 6515908, upload-time = "2026-10-16T22:16:12.896Z" }, + { url = "https://files.pythonhosted.org/packages/ef/fe/f5cdaa2d1c4086e757430f368c271a58ee29cfe7378e7ae7b9d964015c25/pillow-13.0.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5f56499c1a74d2b862beb0c33b8016f99d095858455df9d21583bbbc1c15b582", size = 7142544, upload-time = "2026-10-16T22:16:14.844Z" }, + { url = "https://files.pythonhosted.org/packages/fb/b5/a54f6d44adbcd23737c8c12953f42be107e4cca64e247c0ca3a5b563061c/pillow-13.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a2a14cd73d5e6c927f644b0110ee81f59c4ef506259ec9183904747f2caacc81", size = 6613097, upload-time = "2026-10-16T22:16:16.893Z" }, + { url = "https://files.pythonhosted.org/packages/e6/b4/9e6c1207c21b2e91db0d15c16d30f3701ce5add1bd4da0e787761e66cb6e/pillow-13.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:08b826a9bf1265c76de04a42061d61ea6b109418b7826f28ff1ecdfaa26c5502", size = 7224846, upload-time = "2026-10-16T22:16:19.195Z" }, + { url = "https://files.pythonhosted.org/packages/78/9f/d3c3cb0518f3dd1eee0ef501535f53d15597a53c5c4118d178086143397a/pillow-13.0.0-cp313-cp313-win32.whl", hash = "sha256:12b0fa15879fe5a9d9eebe66e4e2d74128a773b1a4448010c9c714f45e04b9d8", size = 6516013, upload-time = "2026-10-16T22:16:21.389Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ed/1b16fb66a37e1b9e41370b93b2c614c7da761a5b2a41dc333f7af4639962/pillow-13.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:9dda0788a9932776a75d8bcc020076edc308ce2508b51f6739a45e3158b3fc42", size = 7284777, upload-time = "2026-10-16T22:16:23.393Z" }, + { url = "https://files.pythonhosted.org/packages/87/7d/8d02eba9f7ecc87f8e90f731e032cc4773802f2a843763afd0c66a535ff4/pillow-13.0.0-cp313-cp313-win_arm64.whl", hash = "sha256:40a266371ece581e5de13b6635ef78d7487824af80561b6ca27694a88c19bc0f", size = 2698826, upload-time = "2026-10-16T22:16:25.382Z" }, + { url = "https://files.pythonhosted.org/packages/8e/45/fbad9c7f5e2f1745c373a434d46072bc84eddfefaa28b852425ac957fda6/pillow-13.0.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:44629d4498c9cb6b2051806155f22141df0d8a929c56134b166655232dfb2f8c", size = 4190721, upload-time = "2026-10-16T22:16:27.524Z" }, + { url = "https://files.pythonhosted.org/packages/a2/bf/2074902a259a90692c5dbf132885d21671644cbbd3f690047516243a55db/pillow-13.0.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:bf97ba1b5542050e9b0c74e632323b6d8136cc99d64e193abebee22fe6013c76", size = 4290236, upload-time = "2026-10-16T22:16:29.562Z" }, + { url = "https://files.pythonhosted.org/packages/ff/a5/3d1bb93b8ea38d15d21356c1b8e9f50971ac592aaa248f7fb71739094c0e/pillow-13.0.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:aa694c49787040943570d61c9b8fc01271bd2a2cfdd55f0d896f8aecdce1a981", size = 3723093, upload-time = "2026-10-16T22:16:31.249Z" }, + { url = "https://files.pythonhosted.org/packages/84/a1/be060135d85187b25fa8b2573786c0e45c62ae27527fff5257675eb97f4a/pillow-13.0.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:49c12af515073e9bf511e4ef736743bf70d531a643668722279314b937742ba4", size = 5572253, upload-time = "2026-10-16T22:16:33.103Z" }, + { url = "https://files.pythonhosted.org/packages/b5/f4/0bf6f1ac9ba7ee2ffe9914aeb410a1035a0c1a3c42f122388d7fc950bc4b/pillow-13.0.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:d9b0a2bee3f3e03e9f29057aedb807dbf1e1d96ee1b9d0f422d9db6c8583271d", size = 4852269, upload-time = "2026-10-16T22:16:35.102Z" }, + { url = "https://files.pythonhosted.org/packages/83/16/b603274617b3e6728da3a0133d4fba1ea8d7fbe0c4ea5fef9cbffa398b8b/pillow-13.0.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5933fcea5a5ba64c686f55c886a115cbfd92f2900460784a1e79909fdbafd7b6", size = 6516058, upload-time = "2026-10-16T22:16:37.228Z" }, + { url = "https://files.pythonhosted.org/packages/1d/f2/1371707ce438ba9a43f9d24089828494ec6f9396e28cd7903aa704dd6306/pillow-13.0.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e91c2b1cbeae768c98752e67e717e283d68dae45717b41f2d91eaee648e37b7d", size = 7142574, upload-time = "2026-10-16T22:16:39.172Z" }, + { url = "https://files.pythonhosted.org/packages/b9/a9/62cf68587f910bc87c5951bcb6e9430c7643a65076adf5f18b4bf350e2bd/pillow-13.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7f6299e494e46253d40c34653cac369412f44ca53b5f85d848d3f95b1ed0546c", size = 6614295, upload-time = "2026-10-16T22:16:41.117Z" }, + { url = "https://files.pythonhosted.org/packages/6c/c9/9e9b48178649730299be0bafbf5f810a5b6977538f5a92d47e4ba5f5d608/pillow-13.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f9bd7d260d986f3553518c5d723544346fa48031f8aef3265d786aeb59845bc0", size = 7224841, upload-time = "2026-10-16T22:16:43.118Z" }, + { url = "https://files.pythonhosted.org/packages/8f/48/c44e91384b1b6c072a714a63d00608acc16a6a968735e478e4be4384e5ac/pillow-13.0.0-cp314-cp314-win32.whl", hash = "sha256:2e31c4e68cffd7b1f116127626813b82fcba19edf6b940a7b0c266d6484d0055", size = 6508694, upload-time = "2026-10-16T22:16:45.883Z" }, + { url = "https://files.pythonhosted.org/packages/50/e2/7ad17747517e4c813dca78257d6bb593466f8231baf1384bc98d92745edb/pillow-13.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:f7cbb7a64e7ec664775195e4f81923bc8a93d9451b963f844e59118f63147a7a", size = 7281830, upload-time = "2026-10-16T22:16:47.725Z" }, + { url = "https://files.pythonhosted.org/packages/41/9f/e5b51f2ce8ef4563215334035403ce61a244f93a2e65c24a940131618f87/pillow-13.0.0-cp314-cp314-win_arm64.whl", hash = "sha256:a6aaa7871aebca081eb5c45f83e845bbd9712ae99fa9286dbfdcee835e63f4b4", size = 2695468, upload-time = "2026-10-16T22:16:49.515Z" }, + { url = "https://files.pythonhosted.org/packages/97/73/a53502f731f6c84805a4ce87a8881e86d07a65aee64728b900d112dc703e/pillow-13.0.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7b999816b5850a35ada08443e6e03945f1f6488144a83afe2d37b240a552aafa", size = 5578196, upload-time = "2026-10-16T22:16:51.291Z" }, + { url = "https://files.pythonhosted.org/packages/21/50/7e949959a56b5559c78ef172b70fc6025706acb289f3f648a15329e4fa6f/pillow-13.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:67871a13cf47275201553b536c025da777f997fa562a19eb798af4ad4b97c18f", size = 4857276, upload-time = "2026-10-16T22:16:53.399Z" }, + { url = "https://files.pythonhosted.org/packages/c9/52/43ded0ac08e77e39d17bd6d894162572201b5e8fdc4a022b14c55a1960a4/pillow-13.0.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e8ca8f69b7c77fa96dad99f8638c1fc4f1c0bed4acaa1baf7a3c31bdb80a0ab1", size = 6607008, upload-time = "2026-10-16T22:16:55.525Z" }, + { url = "https://files.pythonhosted.org/packages/0d/2e/4a57ed216d5bf877387cba4d44c02aa87abe51ad2e06404aa6b597c4da39/pillow-13.0.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6bce6a29c3898191f4c9998e4d4e8f69790caeabde46e54da4bc235ce5bde683", size = 7224281, upload-time = "2026-10-16T22:16:57.738Z" }, + { url = "https://files.pythonhosted.org/packages/5c/3d/2929b248e135752d63aaeabf53f6de797e905b7b6263f9c1e2adade20aba/pillow-13.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:816752a5ef76a80484c8c441c85593b7ea9304f6231fc5776ad1fbff7e773058", size = 6695600, upload-time = "2026-10-16T22:16:59.842Z" }, + { url = "https://files.pythonhosted.org/packages/bc/33/fb0da992cf084ed1ee6ee05e96b6191e35c351eb707ca2df874feb24ce47/pillow-13.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5bdb38ba220b61b4ec31dd1046dea6b016d3c6850a23f43ed8b58332b2fc668e", size = 7300589, upload-time = "2026-10-16T22:17:01.686Z" }, + { url = "https://files.pythonhosted.org/packages/bb/af/c30dd9ef6a3dce84ecb1b2393cac629047a32e66279056c6cb46c3ffd65c/pillow-13.0.0-cp314-cp314t-win32.whl", hash = "sha256:06903aac074d00c988e77a1fc435286dc498506d4ca8d6f69df9babddf4ecc15", size = 6517204, upload-time = "2026-10-16T22:17:03.722Z" }, + { url = "https://files.pythonhosted.org/packages/85/d5/ef3a8ae7609c8ab26570abf18194882534b480d245a9148b895a2262c004/pillow-13.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:9a635068d9793c7f0d929f39f18b58f1bd86c236e8e7b64167b86af9232ebca0", size = 7286695, upload-time = "2026-10-16T22:17:05.543Z" }, + { url = "https://files.pythonhosted.org/packages/84/3c/0861db0c69a03689ebe3e7599d35668a414deb5aaafd8255215e4e62e3d3/pillow-13.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0d1a9cf7bc6917538b20f8d06b9ab2147e542f68849e96fc58eb66bdf6acc692", size = 2698098, upload-time = "2026-10-16T22:17:07.674Z" }, + { url = "https://files.pythonhosted.org/packages/6e/eb/e86a41962a3a31ad495a3d6a912d9002b01bba4a6fa71c28703abdca7f69/pillow-13.0.0-cp315-cp315-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7d5e097eabd5d4915c27ac9f6e9325bb1e01a7c94d7348b5c4451d3fc30e8521", size = 4191213, upload-time = "2026-10-16T22:17:09.449Z" }, + { url = "https://files.pythonhosted.org/packages/dd/04/7e7445e706afec84ef06dc25b2c77f9cecff242c23730c62928ad1089d4a/pillow-13.0.0-cp315-cp315-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:51ebb529fc5e8abdd55302f6c3f8943037348ec2384329a3e55677f69ae2fdfc", size = 4290650, upload-time = "2026-10-16T22:17:11.196Z" }, + { url = "https://files.pythonhosted.org/packages/46/85/d534089068ac40c0f61f650389a296c812eab745a8a733f81feec39ae955/pillow-13.0.0-cp315-cp315-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:c87a04e7f152ee4db5dbe15f0bfa3fe628a1aeedbe66b81c43b87f52c813f9ac", size = 3723078, upload-time = "2026-10-16T22:17:12.772Z" }, + { url = "https://files.pythonhosted.org/packages/ce/2b/00b5c3b401df538b5dbe311d3899730fae5ba1e21cb8b80a128887ac6747/pillow-13.0.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:20722cd5375533ad754053363a80b8fb30c79d37930a6f7228030e237ce3bd5d", size = 5572292, upload-time = "2026-10-16T22:17:14.539Z" }, + { url = "https://files.pythonhosted.org/packages/72/f5/a6b0610f946e9b54ae888ba7ad29d6e2f8d9b11f69e212d70416b7186c09/pillow-13.0.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:04a63720f79e22fd54d897c233a68314c50ca0603360f54ce2bd5db8f0f43782", size = 4852222, upload-time = "2026-10-16T22:17:16.521Z" }, + { url = "https://files.pythonhosted.org/packages/1e/d0/c0ff25bb0427ec70f32aac995453321fc3c4494a46c7292ff02d9d260b04/pillow-13.0.0-cp315-cp315-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4f8db54177035a2b5f717f94475b74907e12a96dc74f6d487f3966b0a4694071", size = 6527349, upload-time = "2026-10-16T22:17:18.534Z" }, + { url = "https://files.pythonhosted.org/packages/85/11/e46fa0bcc0fc117f1eacc206b3d51f04d583fab328b4b740672d0438d83f/pillow-13.0.0-cp315-cp315-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8f36aa77f74b14a2103622b196402c71330e4e7cf4122ee45c854e9a01e0eecf", size = 7152873, upload-time = "2026-10-16T22:17:20.415Z" }, + { url = "https://files.pythonhosted.org/packages/7c/53/2bdeca705f3a58679fa9201233193c86a30238340ca29b3a152c3fcc719b/pillow-13.0.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:149a1c323ba366f90b48e6f1a6db6582eb50512df374588de56fc909abc0f0f3", size = 6626871, upload-time = "2026-10-16T22:17:22.361Z" }, + { url = "https://files.pythonhosted.org/packages/14/de/3829030f209722ade3753bb456b805a0e7f885cf885b483407365c82e9ca/pillow-13.0.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e892354a424b1e82a78ba16785822bdd40de06f19446a3d4a28047df1a4f8932", size = 7236439, upload-time = "2026-10-16T22:17:24.352Z" }, + { url = "https://files.pythonhosted.org/packages/44/b1/67fc37388382bc036a74f04da5bb6fcbf90ec3ac1f45893e06176c373fd3/pillow-13.0.0-cp315-cp315-win32.whl", hash = "sha256:8e459b613cf31f8980f584387b4fe47bf08e076e288d61aa8eba1f7fd4ddbb81", size = 6508688, upload-time = "2026-10-16T22:17:26.314Z" }, + { url = "https://files.pythonhosted.org/packages/44/05/fdb6a79aee10861165526b41b7b440e36ee1ff58613b2fbefe3acbf06d4f/pillow-13.0.0-cp315-cp315-win_amd64.whl", hash = "sha256:d4737f50d23ddd11d6685faacbb9252887dde0bc7997127757247983d74399d6", size = 7281107, upload-time = "2026-10-16T22:17:28.198Z" }, + { url = "https://files.pythonhosted.org/packages/e6/9c/6c6f7dc2a566467baf121394e83de25ac287cffe34cc030726beb4be0224/pillow-13.0.0-cp315-cp315-win_arm64.whl", hash = "sha256:2310e2adfc13c6d457d7d0f2ffdd5c56a74a317650cf765887df8d515a647ac0", size = 2695513, upload-time = "2026-10-16T22:17:30.013Z" }, + { url = "https://files.pythonhosted.org/packages/23/16/e9120defdb8bff693b582d035312acc6f530cdc2bc5238a87f410efccd82/pillow-13.0.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:21c32061a5512dc13f87cc7303e120e4d77c2f70df89a35af0e3ce246e5ac57f", size = 5578129, upload-time = "2026-10-16T22:17:31.642Z" }, + { url = "https://files.pythonhosted.org/packages/47/22/76faf2bb2448f48a84cf894c1e2bb1090d7d2b1aa521ece131cb2f05d631/pillow-13.0.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:1c774b9c0ba0d0b356f71530677b39028878a2cae5a93e0a13779cfd17120f88", size = 4857428, upload-time = "2026-10-16T22:17:33.43Z" }, + { url = "https://files.pythonhosted.org/packages/b3/64/0861ea4e03c50b14af9ed9dd8b9f388ab42f445c623a9fcbab19aef7aa55/pillow-13.0.0-cp315-cp315t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:20822140c9a2ba9c5e42df342b0a892ab2e102fc0e3db967a133bb9ce3f898bb", size = 6612582, upload-time = "2026-10-16T22:17:35.415Z" }, + { url = "https://files.pythonhosted.org/packages/5c/20/cd1e9b3f60a9f3d5ebe46da36955c26004092a501f7f2c42f9fcc26611bc/pillow-13.0.0-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ffea6b2986a797c84b6d1d27d4f43e0b83c73fc16281a930891cd237cf0d08fe", size = 7230176, upload-time = "2026-10-16T22:17:37.362Z" }, + { url = "https://files.pythonhosted.org/packages/b6/ef/ffb0a2fc85690714fb7ac662956949645d57f15bf4426f520da361e7b6e3/pillow-13.0.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:05cd2cd39055d0b9650060e4513ef4a9a0aea870083950bd046a0d6490f93a22", size = 6703640, upload-time = "2026-10-16T22:17:39.313Z" }, + { url = "https://files.pythonhosted.org/packages/e0/c2/6402ee3ccb3363aeed8fdb3f9fe56703ac5377a539f0b81b53999d7e121f/pillow-13.0.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:c6fd031333dedcad7ce3e03d244093f43504f209e86925ebdb2b868fa4037630", size = 7309231, upload-time = "2026-10-16T22:17:41.021Z" }, + { url = "https://files.pythonhosted.org/packages/fc/96/d6b82e8b8c06ad143b66b12e0b31051e4bf489b4fabca1f390a14fe1ebde/pillow-13.0.0-cp315-cp315t-win32.whl", hash = "sha256:c50c80e3591e5cf9a8dfa6007d4f10732f02e32263850999286e56ccd650cfb2", size = 6517200, upload-time = "2026-10-16T22:17:42.886Z" }, + { url = "https://files.pythonhosted.org/packages/60/d4/0062e6005e9b270b165015aafd343364ac378ec5a685431ea42baad33357/pillow-13.0.0-cp315-cp315t-win_amd64.whl", hash = "sha256:adb3c2124c68e480f4632d06cb7caa3b48a51678475f483b4edb1c6dee29ea43", size = 7287488, upload-time = "2026-10-16T22:17:45.56Z" }, + { url = "https://files.pythonhosted.org/packages/e4/81/b8eb349746be7bce47dd95337c6634e84910af91b59aaa9159c034c614fa/pillow-13.0.0-cp315-cp315t-win_arm64.whl", hash = "sha256:4a25756244bdcce3b50580baadf320fa48297f0eeb8e114ba64139a40d4d0fc8", size = 2698073, upload-time = "2026-10-16T22:17:47.556Z" }, + { url = "https://files.pythonhosted.org/packages/60/da/f6d644e631c5b370c5d0b899f3ccadd2780aab070b29a110527cf4e1fd2c/pillow-13.0.0-pp312-pypy312_pp80-macosx_10_15_x86_64.whl", hash = "sha256:8619685cad8efc4c38c66d3b36aad854523297c43d748b3b6f0cc03c90ba3463", size = 5523188, upload-time = "2026-10-16T22:17:59.334Z" }, + { url = "https://files.pythonhosted.org/packages/e4/b0/2594f202d743a35d4fe80682bfd96f277e5ca7aecb27cf64ef5bfc1d000b/pillow-13.0.0-pp312-pypy312_pp80-macosx_11_0_arm64.whl", hash = "sha256:7e91e97a929232cdb828889ee6ad0e48d7f40f4ec38818f4e4a8968b2b1af22e", size = 4806808, upload-time = "2026-10-16T22:18:01.188Z" }, + { url = "https://files.pythonhosted.org/packages/8e/ed/d504165ef79af6fafe25c76726cf30f544cc236002f7ed054f1d51adc4ff/pillow-13.0.0-pp312-pypy312_pp80-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:93ff9c9c3663ce23aaa5d0b6b6abac55359e2a0c06c7e30f3771829af816be63", size = 5464358, upload-time = "2026-10-16T22:18:02.779Z" }, + { url = "https://files.pythonhosted.org/packages/7b/fe/a10aba5503ca13e9e009b8bd0a1a4ce5584bdce246245fd6f3a83701369c/pillow-13.0.0-pp312-pypy312_pp80-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e65dd6e75de67b2362a4fa3e16aa627595f65054dc4ec73d64680c84d592fa61", size = 6043514, upload-time = "2026-10-16T22:18:04.538Z" }, + { url = "https://files.pythonhosted.org/packages/c0/37/66b12dc476041fe523bca29c9d58fbf1ea92a4eaf73acfb9e993948f1191/pillow-13.0.0-pp312-pypy312_pp80-win_amd64.whl", hash = "sha256:567f20f1c75897d7663830da6ea8af719e03d712b1d9ee7f813319f9ece42e5a", size = 7366030, upload-time = "2026-10-16T22:18:06.638Z" }, +] + +[[package]] +name = "pycparser" +version = "3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ac/d3/eb1d3bc30dda12f7e69640ae2ac8cb10240b71fb73024ad528b7d2ae73da/pycparser-3.1.tar.gz", hash = "sha256:b3fc6dec06a8b2fefa0ed4ff92285306a5e3be9987bc5603c9edbdc4e492418f", size = 105946, upload-time = "2026-10-08T12:26:34.349Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/99/ce/b3ae9ee0324d991c860187be2a6ee436d27a3d02397eaddbb101ec901f3d/pycparser-3.1-py3-none-any.whl", hash = "sha256:f09d358c840bd147b79e55f2bc494f18ea869dc897f5852a8f5766b74f787882", size = 48709, upload-time = "2026-10-08T12:26:32.97Z" }, +] + +[[package]] +name = "pydantic" +version = "2.13.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-types" }, + { name = "pydantic-core" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/53/ef/fc4f868f4e2cee79f863883abffceff107875f569b848507319842d2a681/pydantic-2.13.5.tar.gz", hash = "sha256:51a9c5f7b2f8e636f04c6cada605d9b6a3bf1348fdf945a3d8869b19bba0ee08", size = 845750, upload-time = "2026-08-28T14:04:00.916Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/eb/47/c95ffc2009878c7aac0c5e08528022dcb885933252a88b5f170058014464/pydantic-2.13.5-py3-none-any.whl", hash = "sha256:346a034f080da3755d8e9cb5e00e8b07de1d39e4f6e2c87d8ab7cafa0b269a73", size = 472589, upload-time = "2026-08-28T14:03:59.136Z" }, +] + +[[package]] +name = "pydantic-core" +version = "2.46.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/af/f9/8a06bea35ef8daf588f707784c973a7046e0034c8d8cfb08828eeffb8b75/pydantic_core-2.46.5.tar.gz", hash = "sha256:10416c15b8839ecc4ef4d0885da76da6fd0f67333a0eb8aff6d93c4b8f2910fc", size = 472262, upload-time = "2026-08-28T10:01:31.677Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/3f/76358795aa7a8c6d4f36e2cb828ad1c90ee118e1393a9281664f5aade9d4/pydantic_core-2.46.5-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:b9fe6fb92520e3fd61f2e49000b6911b188824f089b75973ea06d6267f0b476d", size = 2076516, upload-time = "2026-08-28T09:58:21.576Z" }, + { url = "https://files.pythonhosted.org/packages/db/50/26b091836076ce4cb2fac264186936acc069e0595772cfd02a563bc4761a/pydantic_core-2.46.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a39ac25a9a2fa4072efdb429833c4a4c8009a51ff9eea3eeae131713cd27991e", size = 1922874, upload-time = "2026-08-28T09:58:23.766Z" }, + { url = "https://files.pythonhosted.org/packages/09/f0/2a8ce3849e299d44e2d2c196b6082643a3235565a735cb51db7a6261f614/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4fdc8b93a41521988916eeaa271173fcca7fa0803d62f87675aac8dcec1c8e29", size = 1951772, upload-time = "2026-08-28T09:58:25.435Z" }, + { url = "https://files.pythonhosted.org/packages/87/46/ac0dc8bdd9e6048183a14eb127764e7ad9240021c17513074a4711b0e31e/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b98134087d9de723658d17a42c7d0da8d6e2ef08015dee7dc93889047315f5e4", size = 2031832, upload-time = "2026-08-28T09:58:27.102Z" }, + { url = "https://files.pythonhosted.org/packages/c4/c2/339de5bef7be36301a2231eaa52e62163742c2281f11b5f4892bc79785cd/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e652ab17569c94bff5475520f907b7148b8c24036a8ebbe5cf7cf7493d28579a", size = 2208645, upload-time = "2026-08-28T09:58:28.948Z" }, + { url = "https://files.pythonhosted.org/packages/7b/a0/9ff22b797724262da14427abaed4dd1d864a139693fc5e7809114376a716/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d925f3d9afd05a8c0fb3a1031463a8d59ebe5e2afad297e29c78be19e13b4e62", size = 2265935, upload-time = "2026-08-28T09:58:30.625Z" }, + { url = "https://files.pythonhosted.org/packages/c0/a4/eb9409ec0736e50aa70a412f16c204ed149516846912f7e6724d4c73ee53/pydantic_core-2.46.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0fc5be0abd4a407e200d844b404e33639a554e7bd0d448e7b9ae181be4789ac2", size = 2066284, upload-time = "2026-08-28T09:58:32.289Z" }, + { url = "https://files.pythonhosted.org/packages/c0/02/7f6156ffc926857f1c37c07d9a388682865a81830ab6a1b637082c25e399/pydantic_core-2.46.5-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:816ff0a6550ffc06c098ccd2e0698600f9aa7da192a79eaa6f9af504a35db869", size = 2105889, upload-time = "2026-08-28T09:58:33.986Z" }, + { url = "https://files.pythonhosted.org/packages/92/b1/e781d357ebe09fc929f995700f1b3503e8897f1cece183ecb1300d4d67e9/pydantic_core-2.46.5-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c7ea57fc63aa7da93a1bd2d644e6577befae10c52c4e36377635eea1056a74f5", size = 2158006, upload-time = "2026-08-28T09:58:35.647Z" }, + { url = "https://files.pythonhosted.org/packages/70/0a/644597d84ab400e50609c192120b85c9681c22d3a20461b9060a79be0a7a/pydantic_core-2.46.5-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:efd62a42486f1bda5d24cb4f63d15a3c7768375fe83d36f9417b4ad7a2fb20b3", size = 2158408, upload-time = "2026-08-28T09:58:37.38Z" }, + { url = "https://files.pythonhosted.org/packages/1e/ee/ca3b7b3a4b3769ffe9ce9432a7c9be755de9593a46d3b0d54d0409323e44/pydantic_core-2.46.5-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:2bc9419666990c06d7397831f2126a1ecc3594aaa3ff7de5bf2d066802f4e07b", size = 2309609, upload-time = "2026-08-28T09:58:39.22Z" }, + { url = "https://files.pythonhosted.org/packages/ce/52/39fa1f451486019524ca685020390e7ca351832fd874530ba30c8628e6dc/pydantic_core-2.46.5-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:18a09e1e1011b462f2e32774f25859ef1223d5c2b0546a633cf56654710721e0", size = 2342618, upload-time = "2026-08-28T09:58:40.89Z" }, + { url = "https://files.pythonhosted.org/packages/81/5e/468fc630568c61dcef3cd47ad32ffbeed9af643f49208d1ea86ab4f890c4/pydantic_core-2.46.5-cp312-cp312-win32.whl", hash = "sha256:5cb482e9e84c851f4e623fe4acc1ced89168cf1fe18f7089db4548c8f5bbb65b", size = 1939475, upload-time = "2026-08-28T09:58:42.591Z" }, + { url = "https://files.pythonhosted.org/packages/cf/c9/4c19f41b84cf6b622a72fbeed7665b25d47a187d68d47d0d430c07f23268/pydantic_core-2.46.5-cp312-cp312-win_amd64.whl", hash = "sha256:5e81740c09e310f5aa5cbd3e434a01c154d4bef93241c7877b39f211d2b78ba8", size = 2043140, upload-time = "2026-08-28T09:58:44.272Z" }, + { url = "https://files.pythonhosted.org/packages/af/dd/0c1a050299147c746e5256db16d645ab5efd4f78c59937d581a0524e74a2/pydantic_core-2.46.5-cp312-cp312-win_arm64.whl", hash = "sha256:f7b0ec93a2893de856652154d73b7ba622f26fa97726487dcac373de5f4c6084", size = 1997729, upload-time = "2026-08-28T09:58:46.13Z" }, + { url = "https://files.pythonhosted.org/packages/f5/37/5abe39a8372a61d3dc3c1338fc504281c01b32fdb3169cd7187153b56d3e/pydantic_core-2.46.5-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:b7ca9034437b6022f941f4857459562ee00a560b97e7cce8a0ec5a74fc6766e0", size = 2075885, upload-time = "2026-08-28T09:58:47.856Z" }, + { url = "https://files.pythonhosted.org/packages/21/43/6323b1f8b217780454c61304bcd2b38ae4762f50754414124603ccc90bb2/pydantic_core-2.46.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f332f0e72a5a0400141f830744e141bf9f97917878dbe968669e8a7fefea78ff", size = 1922768, upload-time = "2026-08-28T09:58:49.58Z" }, + { url = "https://files.pythonhosted.org/packages/0f/a3/c05ca796e1197618a774b01e596aeedfefc2f7d8c01ae3054e910b120e8a/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:193375f3548919d3f0b60936ca113ada3e38f264f91b9b8e0508efaad57be931", size = 1951241, upload-time = "2026-08-28T09:58:51.511Z" }, + { url = "https://files.pythonhosted.org/packages/68/32/33bc39ac705c52cffc908e8389f9754fdb208aea5c69cceddf4eb3ce99af/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:79bdfa52f843137045b2d081cc05c120ba6665d29b7559c2c47690906f39279f", size = 2031975, upload-time = "2026-08-28T09:58:53.166Z" }, + { url = "https://files.pythonhosted.org/packages/b0/70/2333e885c0f6a67bc105c5916965dac9b57f2718ee20d81d1a06a4ebdc13/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:24922243639cbdac66c75fcb6fd6495a9cb52b213d62f9a0d16f0310b1ff8038", size = 2208542, upload-time = "2026-08-28T09:58:55.017Z" }, + { url = "https://files.pythonhosted.org/packages/f7/ea/296debfb4264207bbda5936133892e027c0a58875ad53ebd512fba8ec3a2/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c76fe65e607be28c7fd4d56fc3c42b1583aa058ce3408b7ad0fd540171d31f9f", size = 2264692, upload-time = "2026-08-28T09:58:56.767Z" }, + { url = "https://files.pythonhosted.org/packages/d3/f2/9e4de77a6271e07a76d2d58b11c091a979c191ed2939bf80067568b369d2/pydantic_core-2.46.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6f7b393a8b3da82f5c1fc0751e6d01ac6c55b93c18226a60bdfba4a724efafd1", size = 2066633, upload-time = "2026-08-28T09:58:58.531Z" }, + { url = "https://files.pythonhosted.org/packages/8d/db/f9e9d0c97445987b2084823d5c240de88087338f04fc2cfaa2df186b8049/pydantic_core-2.46.5-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:7ac031912d54f3d83ef3b3eb98dfabc1608802e2202263d25957eeed40b94761", size = 2105235, upload-time = "2026-08-28T09:59:00.421Z" }, + { url = "https://files.pythonhosted.org/packages/07/c5/79169b047b3b2c3e99e04bc76372af9637e0bf6db638274fa927df96369e/pydantic_core-2.46.5-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:837b396ca3d7b74091ca623f6cbd8351bd42d670a79c2683e79fb089f06a2de5", size = 2157367, upload-time = "2026-08-28T09:59:02.442Z" }, + { url = "https://files.pythonhosted.org/packages/26/b5/ba6057afb7c291bd449f51b867f95aef2072941c4ce4e5c31d6ffd132d3b/pydantic_core-2.46.5-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:5ee239d575f80b08eca11f6e20f90c4c695de7825c67eefe6091fbf20dda648e", size = 2158420, upload-time = "2026-08-28T09:59:04.2Z" }, + { url = "https://files.pythonhosted.org/packages/6e/28/2057abecaafdc22912afa819603a51f0a62d40643b7c4871c51721fea9be/pydantic_core-2.46.5-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:e80675d75ae2cd14372cb65cad5400d9347a3d3f6c13000183f22dfd027283ed", size = 2309588, upload-time = "2026-08-28T09:59:06.048Z" }, + { url = "https://files.pythonhosted.org/packages/71/9d/881156dc404e27479c4246128d73538464cab4a239bec61995e227644c30/pydantic_core-2.46.5-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:9c4b71f10dd532fb7a5cbc8f58707779e64f03a258c2bf8bfbaecfcd9970b519", size = 2341866, upload-time = "2026-08-28T09:59:08.539Z" }, + { url = "https://files.pythonhosted.org/packages/5a/38/d66f443a259f84d13babdceae568e572b0ed26da17ca5d0a649ebb110a67/pydantic_core-2.46.5-cp313-cp313-win32.whl", hash = "sha256:97bf8de4d541598c94a59344eeb988a94c08ff76b5723c41f6567ec18c7892ea", size = 1938580, upload-time = "2026-08-28T09:59:10.402Z" }, + { url = "https://files.pythonhosted.org/packages/2c/1e/1d5371213f4cc9a7ed70c0bfcc7911de22311ee99a662a56077d7292d2ac/pydantic_core-2.46.5-cp313-cp313-win_amd64.whl", hash = "sha256:15f4a94963c95accac15b7b657bb177d3ad82bb90b0d0526d9a9b85079925db5", size = 2041980, upload-time = "2026-08-28T09:59:12.396Z" }, + { url = "https://files.pythonhosted.org/packages/5a/48/4222d90b1c67568bace4dec6dca6271449c66de3595d72b6d098f5fde597/pydantic_core-2.46.5-cp313-cp313-win_arm64.whl", hash = "sha256:d22a945598fb91236b4dd793a6e42e4f3dd7740bb5aace5ebd7d4c08d13bb575", size = 1997213, upload-time = "2026-08-28T09:59:14.245Z" }, + { url = "https://files.pythonhosted.org/packages/8e/8a/14596f2a8367da50cf7cbac48169ee5d9c8e11d486a3b527082384630c72/pydantic_core-2.46.5-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:c1c43ad4339643d70ebb8124e1305a7dab423001eff58bb41a0f731adbc98355", size = 2074081, upload-time = "2026-08-28T09:59:16.141Z" }, + { url = "https://files.pythonhosted.org/packages/ae/d5/d8a4eb6d6c7f66b91dd37c576d76e9e60fba900caf5372c17bcf949febc2/pydantic_core-2.46.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1a353f84de772f423b5ffb11d7ae352fbbef0f446f3c0b0af0f8236d7233606e", size = 1920497, upload-time = "2026-08-28T09:59:18.065Z" }, + { url = "https://files.pythonhosted.org/packages/8e/26/092079428f86e927e030b2c0ced87df69dbb1c875cdeaa67bf42ea2be746/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5086029a57366b8cf81b130a43908738095c270c21a8d7f0e8bdfdb89718e2f3", size = 1952130, upload-time = "2026-08-28T09:59:20.476Z" }, + { url = "https://files.pythonhosted.org/packages/08/c3/8ec0e290a9ebaebd64047bf5fda94be835c6b1551b02437e4b76778fbcd7/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:46c25dda9d092a06c08db76ffe0a197107904d0dfac653f7d5306bbcd6d6119c", size = 2026371, upload-time = "2026-08-28T09:59:22.227Z" }, + { url = "https://files.pythonhosted.org/packages/01/72/4fd20ad520fb8da0157f95b27a7eb05a72790ef08138e7701ac972c342ea/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:37ea7b83c935e5b0d68c9449b82651accf78a10828b2c02b2f2d9e9496446c21", size = 2202822, upload-time = "2026-08-28T09:59:24.277Z" }, + { url = "https://files.pythonhosted.org/packages/31/b0/d16e0771206b29314f0d52198b720be21e8a99ab2bf11e3bc0d7c9cebdff/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e64e88d5585bea9ce95861079de72006c7fa6d3df4e3a3b65ba31eb979c15c9f", size = 2262756, upload-time = "2026-08-28T09:59:26.608Z" }, + { url = "https://files.pythonhosted.org/packages/2c/9b/59634b7ac631c63b2a37760eb6943af3e29573d6b59a4abc5e7f019d4cee/pydantic_core-2.46.5-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:54d510bac3ee52247af28ed4bb18a1e799f040ac60fd2bf5ccd4c92f1fbe786f", size = 2068352, upload-time = "2026-08-28T09:59:29.044Z" }, + { url = "https://files.pythonhosted.org/packages/08/7c/570abb1ad2155348dc754ea91be22e5aaa18eb6d69a6068f7c6f2679a6ed/pydantic_core-2.46.5-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:a2a5e1d0ff29adddc9f6d6821a66302e4493f8ca898b715b6b1182c2c201ea0a", size = 2104777, upload-time = "2026-08-28T09:59:30.95Z" }, + { url = "https://files.pythonhosted.org/packages/8e/25/5bf74adc65a1ac5b7be3f6cb0bcb5433615c1598a801c19d830d84c98ded/pydantic_core-2.46.5-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:03b9666e41e35d8909852ba191a0607520f81b74eaf12ccf8737005dbb313821", size = 2156312, upload-time = "2026-08-28T09:59:32.604Z" }, + { url = "https://files.pythonhosted.org/packages/90/6a/2ef38830675e050121040618135564ed56b860b45433b02d9b4ebece46f3/pydantic_core-2.46.5-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:a91c17edf6eea2402cb5457b4c89e99bc5ed1004aa34c4adf1d4258c1a5c22c2", size = 2150067, upload-time = "2026-08-28T09:59:34.453Z" }, + { url = "https://files.pythonhosted.org/packages/90/ef/a7dbb03a14a64c2a4621f989c615ed9a892535a6cad938fc27079f919d80/pydantic_core-2.46.5-cp314-cp314-musllinux_1_1_armv7l.whl", hash = "sha256:b49924c73a235e969511bf2aabdff3beebf9820931f646c80274d5d780010c47", size = 2304516, upload-time = "2026-08-28T09:59:36.194Z" }, + { url = "https://files.pythonhosted.org/packages/68/f8/6bb4c4b80e8a6fde1904c64a51c62a1d04fcdfa3ea521a66b2ddefa1d885/pydantic_core-2.46.5-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:2cbd9a5eff05e51c447c34dfa4632145b26b09120cf04bd0c871e44c1a5e1c9a", size = 2335223, upload-time = "2026-08-28T09:59:37.931Z" }, + { url = "https://files.pythonhosted.org/packages/2a/80/f46b8c681195190b2c1f1c7c0a81abce60663e987613e09ef64d433dd96b/pydantic_core-2.46.5-cp314-cp314-win32.whl", hash = "sha256:2d5d76654becf5efd62c9e51c3756c67b49498b0c9a40884934c40807adbd074", size = 1934827, upload-time = "2026-08-28T09:59:39.836Z" }, + { url = "https://files.pythonhosted.org/packages/f7/3c/60674207246bc0a4009d2391b7c7251c7159f279c8d2ab8aae8ef46f3dee/pydantic_core-2.46.5-cp314-cp314-win_amd64.whl", hash = "sha256:fa10ef4112775900e7a0661068635eb67b2ab824fbde764de6e0e21982a93db0", size = 2042648, upload-time = "2026-08-28T09:59:41.792Z" }, + { url = "https://files.pythonhosted.org/packages/69/0c/117c562c7c1babdf44576b72a5e496906506c93690387ecfbca7c729ae2e/pydantic_core-2.46.5-cp314-cp314-win_arm64.whl", hash = "sha256:045ab3b6d308439e32b81cc173bba5b9018bc6ed896afd0c65b3b009b1699af5", size = 1989652, upload-time = "2026-08-28T09:59:43.702Z" }, + { url = "https://files.pythonhosted.org/packages/e8/66/9336ae58f9eb68c41d121894e52c4c89eccb07eb8f602a04ee9c3f37736a/pydantic_core-2.46.5-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:8816f3d218beb4b787de5c9759c259b8fa61f9dec42dc7811f320a33771778b7", size = 2065829, upload-time = "2026-08-28T09:59:45.364Z" }, + { url = "https://files.pythonhosted.org/packages/c5/02/bc19b47a96c2d3109760711acf22369e56bd7e405ca52f7ade164d2ead57/pydantic_core-2.46.5-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:bce57638e08ac148e5778cce7feb968307a727d66f8e2274a543d0cf0c9ad6a3", size = 1905716, upload-time = "2026-08-28T09:59:47.18Z" }, + { url = "https://files.pythonhosted.org/packages/52/a4/70b47c0509923dd98ccfed04fb3e32ea3849c82a0ff2205bb41009b43c00/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:976e1128455aa595ea04c79ccfedff1aaeab96ee013fcc916bed120c4f0ad94f", size = 1934216, upload-time = "2026-08-28T09:59:49.241Z" }, + { url = "https://files.pythonhosted.org/packages/52/ab/aa03b65f7bb198585edf806b906c3223ecf1795543e39e23aec4cce27ad2/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e7b891faeedeafba41b2983e5001a81b6a915b69544c7e7570d1989ce1c36ac7", size = 2010635, upload-time = "2026-08-28T09:59:51.692Z" }, + { url = "https://files.pythonhosted.org/packages/3c/8b/0da06343f30b84ec549aafd309c6456223d5dc8bd36af504c573faad561d/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5f194189415698233dd1114a093a9b56e61e2c57e11b469be3b0506f46f0771c", size = 2209369, upload-time = "2026-08-28T09:59:53.582Z" }, + { url = "https://files.pythonhosted.org/packages/d6/5b/844c4defaa34a3df66eb9257087d121d70c201298b96abdf9f492fc2f1bf/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:82a36973cf8a2ef5406f4fe2edbf8ed0c99629535d959e0b100c76a32535a111", size = 2253238, upload-time = "2026-08-28T09:59:55.484Z" }, + { url = "https://files.pythonhosted.org/packages/f4/64/a4e536cb16d7f61a7fd3120b46c577fc7fa7325992f69c4f52bc786d77d8/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:cdbb78909f52b981d3b2d56b97328d71eb0b974c36bd77c920123a7ebb192829", size = 2065740, upload-time = "2026-08-28T09:59:58.038Z" }, + { url = "https://files.pythonhosted.org/packages/5f/75/aaa38c6bc2d085f6605b34eabdc6a8a4e0b2e61fc9c8e6e52b28e97b3125/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:52e24eacdb536cade636aa90fb851835222becff8484b7001fdc78cb0290f2aa", size = 2087425, upload-time = "2026-08-28T09:59:59.898Z" }, + { url = "https://files.pythonhosted.org/packages/55/ae/fcab4cfc39aba3689e1d20c8b5250ad280957022c09af2ed9cd585602a5e/pydantic_core-2.46.5-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:37ae34309d7bd8c0d61ab839668058f2a7962ea1fc51d105d2db228fe0618034", size = 2139306, upload-time = "2026-08-28T10:00:03.057Z" }, + { url = "https://files.pythonhosted.org/packages/2d/f4/f1d03a4bc9d9acbc62f4d742b8a319af52f71885079868b2ff8e48a651ee/pydantic_core-2.46.5-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:0cdbada856a1c69a7624a64d3d9aefe79300bd6ef827b43a4f265010b9b55184", size = 2144589, upload-time = "2026-08-28T10:00:05.645Z" }, + { url = "https://files.pythonhosted.org/packages/83/f3/7a53bb1356de514a4cd295f25b6ac39237895620c0462d2592b76c16e114/pydantic_core-2.46.5-cp314-cp314t-musllinux_1_1_armv7l.whl", hash = "sha256:545f26c504b27c3758439a5e6d9349931f0a04f855668d5fe323c89e82300a38", size = 2288882, upload-time = "2026-08-28T10:00:07.931Z" }, + { url = "https://files.pythonhosted.org/packages/cd/94/5a81583660c175c59d49ffb09f4b3a44debeaf86a19fca664ae1cdd9ee32/pydantic_core-2.46.5-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:ff218293c9c806138dca139765e3b067621be52bcd93cdc14c7711be7ddc90a9", size = 2335210, upload-time = "2026-08-28T10:00:10.177Z" }, + { url = "https://files.pythonhosted.org/packages/5a/9f/5d685c2693b972d1a59c998586e8823712b66603aeff47ee60a4bdaafd37/pydantic_core-2.46.5-cp314-cp314t-win32.whl", hash = "sha256:97cf3eb53a8cccacf9d46686a0926186c9bfb5574f2ed66d3639d5fe117cd3a9", size = 1921180, upload-time = "2026-08-28T10:00:12.35Z" }, + { url = "https://files.pythonhosted.org/packages/70/12/5c94ee16d65a37a15f9e869f5e6256df111154491173801a4c5e800ab548/pydantic_core-2.46.5-cp314-cp314t-win_amd64.whl", hash = "sha256:d2f9fc07a8042a8f95925b35c4f04f469707c981fc33245b6ca187cf5d2dd290", size = 2020515, upload-time = "2026-08-28T10:00:14.774Z" }, + { url = "https://files.pythonhosted.org/packages/63/19/67830dda664e6bdf9285ee2e40f355d0d7d6b92aa0c42e8d217bb8d33d36/pydantic_core-2.46.5-cp314-cp314t-win_arm64.whl", hash = "sha256:acf8a67ba51f4ca9ddbd0e6b3000a65ac51ab734661778b3e7ba64d99a710f2f", size = 1989276, upload-time = "2026-08-28T10:00:16.984Z" }, + { url = "https://files.pythonhosted.org/packages/df/dd/053c2e4303f791f3b8f8a14ab0b22008e8eb21d868c0c90b4f9be705b76a/pydantic_core-2.46.5-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:013d6f3483d81e02e7c328831808f336c8596ee33b4bd4026b9ffb1e960b8942", size = 2062540, upload-time = "2026-08-28T10:01:00.318Z" }, + { url = "https://files.pythonhosted.org/packages/d7/dd/a18df751a5e37dd51bfad7f68e766999125bebe68c9e1d10a493ad01bd63/pydantic_core-2.46.5-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:e9c134bb666dd54b778b9fc0d2b50cbb7f979b9e3716f26a88c9ab3b6fc1dd0f", size = 1902040, upload-time = "2026-08-28T10:01:02.529Z" }, + { url = "https://files.pythonhosted.org/packages/b7/13/01d40f9d07ce8a779fd6e0bd8ad4fba91309500dd67b869e2e219d261a6d/pydantic_core-2.46.5-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:347ec774390c87326a2e4929d58d3f7e8763a104d5d35f4cd595a4c952366433", size = 1967479, upload-time = "2026-08-28T10:01:05.004Z" }, + { url = "https://files.pythonhosted.org/packages/fa/04/c81d4841331c2178b6fb09ae225425e110ed72d990c9fe556c4ec03d1013/pydantic_core-2.46.5-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8e24d8f05fa2d28513d94e877e9c75ad66175376209b3977f916e240e623193c", size = 2111034, upload-time = "2026-08-28T10:01:07.345Z" }, +] + +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + +[[package]] +name = "s3transfer" +version = "0.19.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "botocore" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/49/c5/0a5ce655109cb96b8dfad36908a4e108d15f5dc2eff8fbc3146c4312e704/s3transfer-0.19.3.tar.gz", hash = "sha256:8326acc3d056d7f7b8a37110c7d2aea41f67e83f6ae79cd3b5fbcca34c726406", size = 166087, upload-time = "2026-10-19T19:19:49.751Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e2/a4/88f9e10498759121060b8050a1554753dfc04df39bbbcc7aad9ecc8d9440/s3transfer-0.19.3-py3-none-any.whl", hash = "sha256:24eb7535804306355e31c952a90022bf5b2f9d760f9a11062c7493a217c08553", size = 90252, upload-time = "2026-10-19T19:19:48.309Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "starlette" +version = "1.7.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7b/2b/3850dc6bf7ef71b088962eba31dafc6cffd2f96e577ebb0bb316df96da3e/starlette-1.7.0.tar.gz", hash = "sha256:c79f74ea63cff761804fbbfb182f1e0b440c2d07b164d24700c5a1bab5d6ff5d", size = 2736246, upload-time = "2026-09-23T07:30:26.35Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4e/d6/1ec1b290f9e0fb067899b61e1d37a30c923068bad260b216dbe37a7d2967/starlette-1.7.0-py3-none-any.whl", hash = "sha256:67f8e99895493dd2911a03f11314af6ceebeae4e704bb9f43dfc6a9db151c93e", size = 78980, upload-time = "2026-09-23T07:30:24.567Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "typing-inspection" +version = "0.4.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a3/26/b09b8010994eccc3c09092e6b34058f36a460eea2d4c3e8b910c695975a0/typing_inspection-0.4.4.tar.gz", hash = "sha256:547274fa6b0a561ccf549cc9524b999a578e737d015d8709d021f9d0d13bea47", size = 76928, upload-time = "2026-08-12T12:37:25.997Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/67/81/4add07e5172b7ac40d8ed5ff580409a7801a4fe26d529bdd915401dabfbe/typing_inspection-0.4.4-py3-none-any.whl", hash = "sha256:65b8397ba37ccbce054456aaccddfc91e6e3083c92824df348d96ca832f3f147", size = 14750, upload-time = "2026-08-12T12:37:24.648Z" }, +] + +[[package]] +name = "urllib3" +version = "2.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e3/05/b17359e1cefb4f909b5e40b1b90a496d987258916dbbf88e842c729f510e/urllib3-2.8.0.tar.gz", hash = "sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63", size = 458972, upload-time = "2026-09-15T19:29:36.253Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/92/9d/c4e665119135114480843e7ab388fa94d8480650450e6f8e26b70d323a4c/urllib3-2.8.0-py3-none-any.whl", hash = "sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3", size = 135717, upload-time = "2026-09-15T19:29:34.577Z" }, +] + +[[package]] +name = "uvicorn" +version = "0.40.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c3/d1/8f3c683c9561a4e6689dd3b1d345c815f10f86acd044ee1fb9a4dcd0b8c5/uvicorn-0.40.0.tar.gz", hash = "sha256:839676675e87e73694518b5574fd0f24c9d97b46bea16df7b8c05ea1a51071ea", size = 81761, upload-time = "2025-12-21T14:16:22.45Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3d/d8/2083a1daa7439a66f3a48589a57d576aa117726762618f6bb09fe3798796/uvicorn-0.40.0-py3-none-any.whl", hash = "sha256:c6c8f55bc8bf13eb6fa9ff87ad62308bbbc33d0b67f84293151efe87e0d5f2ee", size = 68502, upload-time = "2025-12-21T14:16:21.041Z" }, +]