ЛР5— Павлова/Орлова— БПИ2301 — оценка 5 (трек 2) #53
@@ -0,0 +1,4 @@
|
||||
*.sh text eol=lf
|
||||
*.cfg text eol=lf
|
||||
Dockerfile text eol=lf
|
||||
*.sql text eol=lf
|
||||
@@ -1,18 +1,18 @@
|
||||
## Лабораторная 5
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Автор / группа: Орлова Владислава, БПИ2301
|
||||
Целевая оценка (3/4/5): 5
|
||||
Трек 5, если заявлен: Трек 2, идемпотентность pipeline
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
Что реализовано и что перенесено: Стенд лабораторной 4 сохранён: FastAPI, PostgreSQL, Redis, HAProxy, две реплики API, MinIO, миграции отдельным контейнером. Добавлены RabbitMQ, отдельный worker, outbox, состояния queued/processing/ready/failed, thumbnail, crop и optimized WebP. Трек 2 проверяет дубль, чужой порядок, падение до и после эффекта и старую версию pipeline.
|
||||
Команды запуска из чистого clone: `python -X utf8 scripts/bootstrap.py`, затем `docker compose --env-file .env up -d --build --scale api=2`. Программы make нет, команды Makefile запущены через `python -X utf8`. Публичные `.sh` на Windows запущены через WSL. Подробности в REPORT.md.
|
||||
Результаты проверок и ссылка на REPORT.md: smoke, persistence, failover, негативные проверки и сценарии трека 2 прошли. Числа и команды лежат в REPORT.md и evidence/.
|
||||
Ограничения / известные ошибки: по одному экземпляру Postgres, Redis, MinIO, RabbitMQ и HAProxy. При отказе обеих реплик клиент через HAProxy видит его HTML 503. Exactly-once нет. Прямой загрузки по временной ссылке нет. Нагрузку трека 1 я не измеряла.
|
||||
Источники и использование ИИ: задание, контракт, документация RabbitMQ, стенд лабораторной 4. Код и отчёт я собирала в Cursor.
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [x] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод. Программы make нет. Те же команды выполнены через `python -X utf8`, публичные `.sh` через WSL. Вывод записан в REPORT.md.
|
||||
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
FROM python:3.12.10-slim-bookworm@sha256:fd95fa221297a88e1cf49c55ec1828edd7c5a428187e67b5d1805692d11588db
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PYTHONPATH=/app/src \
|
||||
APP_HOST=0.0.0.0 \
|
||||
APP_PORT=8080
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin app
|
||||
|
||||
COPY requirements.txt /app/requirements.txt
|
||||
RUN pip install --no-cache-dir --upgrade pip==25.0.1 \
|
||||
&& pip install --no-cache-dir -r /app/requirements.txt
|
||||
|
||||
COPY migrations /app/migrations
|
||||
COPY src /app/src
|
||||
|
||||
USER 10001
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=5s --timeout=3s --start-period=25s --retries=8 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"
|
||||
|
||||
CMD ["python", "/app/src/main.py"]
|
||||
@@ -1,37 +1,125 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Орлова Владислава, БПИ2301
|
||||
- Целевая оценка: 5. Трек 2, идемпотентность pipeline
|
||||
- Commit/PR: ещё не публиковала
|
||||
- Источники и помощь: задание и контракт этой лабораторной, рабочий стенд лабораторной 4, документация RabbitMQ про publisher confirm и consumer ack. Код и отчёт я собирала в Cursor. Числа ниже сняты с моего запуска 12 ноября 2026, я их не округляла
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker Engine и Python 3.9+. У меня Docker Engine 26.1.4 и Python на Windows. Программы `make` в системе нет, поэтому команды из Makefile я запускала через `python -X utf8`. Флаг нужен, чтобы Windows не читала JSON в кодировке cp1251. Публичные `.sh` Windows не запускает как программу, их я запускала через WSL.
|
||||
|
||||
```sh
|
||||
python -X utf8 scripts/bootstrap.py
|
||||
docker compose --env-file .env up -d --build --scale api=2
|
||||
python -X utf8 scripts/wait_ready.py
|
||||
python -X utf8 scripts/check_contract.py
|
||||
python -X utf8 tests/smoke.py --base-url http://127.0.0.1:8080
|
||||
```
|
||||
|
||||
Проверки сохранности и остановки реплики:
|
||||
|
||||
```sh
|
||||
wsl -e bash -lc "cd '/mnt/c/Git/высоконагруженные приложения/Lab5' && python3 -X utf8 tests/resilience.py --mode persistence --action-script scripts/restart.sh --base-url http://127.0.0.1:8080"
|
||||
wsl -e bash -lc "cd '/mnt/c/Git/высоконагруженные приложения/Lab5' && python3 -X utf8 tests/resilience.py --mode failover --action-script scripts/stop-one.sh --base-url http://127.0.0.1:8080"
|
||||
```
|
||||
|
||||
`bootstrap.py` один раз создаёт `secrets/` и `.env`. Пароли в консоль не печатаются. Повторный запуск уже созданные файлы не перезаписывает. В git эти файлы не входят.
|
||||
|
||||
Остановить стенд, не удаляя данные: `docker compose --env-file .env stop`. Удалить учебные данные вместе с volume: `docker compose --env-file .env down -v`. Вторую команду я использую только когда хочу чистую базу.
|
||||
|
||||
Имя проекта Compose — `mtusi-lab5`. Соседняя папка лабораторной 4 из-за этого не пересекается со стендом. Наружу открыт только `127.0.0.1:8080` у балансировщика.
|
||||
|
||||
Образы, которые реально поднимаются:
|
||||
|
||||
| Роль | Образ |
|
||||
| --- | --- |
|
||||
| API и worker | свой Dockerfile, база `python:3.12.10-slim-bookworm@sha256:fd95fa221297a88e1cf49c55ec1828edd7c5a428187e67b5d1805692d11588db` |
|
||||
| Postgres | `postgres:16.10-bookworm@sha256:38471f330eb885e04de130b768d6db4e10469e2311879c7e5c699f6d2d8a1c74` |
|
||||
| Redis | `redis:7.4.2-bookworm@sha256:fbdbaea47b9ae4ecc2082ecdb4e1cea81e32176ffb1dcf643d422ad07427e5d9` |
|
||||
| RabbitMQ | `rabbitmq:3.13.7-alpine@sha256:d7af1c87c5f1eda13fcfca06db452bf3aeab6619fc3358b68535c0c02c4e52bc` |
|
||||
| Балансировщик | `haproxy:2.8.14@sha256:2344d1efbf7a932c7a3d49b60748c28af81c70fd11fcadd983e2be1f0376a69d` |
|
||||
| MinIO | `pgsty/minio@sha256:b6bfe7239bfc83fb90d31612d9704d86039dd714f7904b3f1ad68f211e602372` |
|
||||
|
||||
Сервер внутри образа MinIO отвечает `minio version RELEASE.2026-08-04T00-00-00Z`. Официальный `minio/minio` с Docker Hub больше не скачивается, а `quay.io/minio/minio` без логина отвечает `unauthorized`. Поэтому стенд закреплён на публичном образе `pgsty/minio` с digest. Это сервер MinIO с тем же протоколом S3.
|
||||
|
||||
У образа RabbitMQ 3.13.7 входной скрипт отвергает переменные `RABBITMQ_DEFAULT_USER_FILE` и `RABBITMQ_DEFAULT_PASS_FILE`. `scripts/rabbit-entry.sh` читает секреты и пишет `rabbitmq.conf` с `default_user` и `default_pass`. Имя узла зафиксировано как `rabbit@rabbitmq`: каталог mnesia переживает пересоздание контейнера. При первой загрузке имя узла было случайным, брокер не стал healthy, и я один раз удалила volume RabbitMQ. Это было до измеренного прогона сохранности. Сам прогон volume не удалял.
|
||||
|
||||
Лимиты: две реплики API по 0,50 CPU и 512 МиБ, worker 1,00 CPU и 768 МиБ, Postgres 1,00 CPU и 512 МиБ, Redis 0,25 CPU и 128 МиБ, RabbitMQ 1,00 CPU и 1024 МиБ, MinIO 0,50 CPU и 512 МиБ, HAProxy 0,25 CPU и 128 МиБ. Память API я подняла с 256 МиБ лабораторной 4, потому что приём файла всё ещё проверяет картинку до ответа 202.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Клиент ходит в HAProxy. HAProxy по кругу отправляет запрос на одну из двух реплик API. `X-Instance-Id` — hostname контейнера, поэтому `scripts/stop-one.sh` останавливает именно ту реплику, которую назвал тест. `restart: unless-stopped`: явный `docker stop` оставляет контейнер остановленным.
|
||||
|
||||
Сессии лежат в Redis с AOF. Сообщения, пользователи, вложения и задания лежат в PostgreSQL. Это источник истины. Сами байты лежат в приватном bucket `messenger-attachments`. Миграции применяет одноразовый контейнер `migrate`. Реплики API схему не накатывают. Схема ждёт версию `004_jobs`.
|
||||
|
||||
Из лабораторной 4 я перенесла регистрацию, сессии, чаты, ленту, идемпотентность POST сообщения, две реплики, приватный bucket и сверку объектов. Чужой объект по-прежнему даёт 404. UUID в разном регистре — один идентификатор, в базе он в нижнем регистре. Пропущенное поле `attachment_ids` и пустой список дают одно и то же тело идемпотентности. Ключ идемпотентности — это пользователь, метод, путь и заголовок `Idempotency-Key`. Это не id сообщения. Тот же ключ с другим телом даёт 409, вторая строка не создаётся. Тот же ключ в другом чате создаёт новое сообщение, потому что путь другой. Ошибка пользователя остаётся 4xx.
|
||||
|
||||
Загрузка теперь отвечает 202 и `status=queued`. Ответ уходит после двух устойчивых записей: байты уже лежат в `staging/<id>/original`, а в одной транзакции записаны строка вложения, задание и строка outbox. Байтов и паролей в сообщении брокера нет. В сообщении только поля `contracts/jobs.schema.json`: `job_id`, `attachment_id`, `pipeline_version`, `attempt` и время. `job_id` не меняется при повторной доставке. Повторная доставка сама по себе не увеличивает `attempt`.
|
||||
|
||||
Публикация в RabbitMQ идёт после COMMIT. У канала включён publisher confirm, сообщение persistent. Если брокер в этот момент недоступен, API всё равно возвращает 202: строка outbox остаётся с пустым `published_at`, worker позже её публикует. Непрерывного публикатора внутри API нет. Цикл чтения outbox крутится в worker раз в секунду, и API один раз пытается опустошить outbox при своём старте.
|
||||
|
||||
Worker — отдельный процесс. Пока он остановлен, API не уменьшает картинку. Он пишет оригинал байт в байт и три производных файла: JPEG thumbnail вписывается в 256×256 без увеличения, JPEG crop — центр 128×128, для маленького кадра увеличение разрешено, WebP optimized вписывается в 1280×1280 без увеличения. JPEG пишется с quality 85 и `optimize=True`. WebP пишется с quality 80 и method 4, без EXIF. Перед декодированием проверяются сигнатура, размер заголовка и предел 40 миллионов пикселей. Ориентация EXIF применяется к кадру до уменьшения.
|
||||
|
||||
Попыток по умолчанию 5. Временная ошибка ставит следующую попытку с паузой: база `min(30, 0.5 * 2^(attempt-1))` секунд плюс случайная добавка до 25 процентов. Постоянная ошибка, например битый файл, сразу даёт `failed` и `invalid_image`, без серии повторов. Зависший `processing` с истёкшим `lease_until` sweeper возвращает в `queued` и не увеличивает `attempt`. Живой worker продлевает аренду. Чужой владелец аренды получает отложенную доставку на 2 секунды и не начинает вторую копию. Подтверждение потребителя уходит после COMMIT эффекта. Для уже готового или уже проваленного задания подтверждение тоже уходит, но файлы заново не пишутся. Это at-least-once. Exactly-once доставки здесь нет. Ключ эффекта — `attachment_id`, `pipeline_version` и имя варианта.
|
||||
|
||||
`/health/ready` проверяет миграцию `004_jobs`, Redis и bucket. Остановленный брокер или worker сам по себе готовность API не роняет: иначе демонстрация «загрузить файл при остановленном worker» была бы невозможна. Текст 503 у API фиксированный: «Сервис временно недоступен».
|
||||
|
||||
Переменные из `.env.example` попали в стенд так. `SESSION_STORE` в примере равен `postgres`, в `compose.yaml` у API стоит `redis`. `BROKER_URL_FILE`, `WORKER_CONCURRENCY=2`, `JOB_MAX_ATTEMPTS=5`, `JOB_TIMEOUT_SECONDS=60`, `STAGING_RETENTION_SECONDS=86400`, `MAX_UPLOAD_BYTES=10485760`, `MAX_IMAGE_PIXELS=40000000` читаются так, как названы. Имена файлов ключей S3 в примере длиннее, контейнер читает `S3_ACCESS_KEY_FILE` и `S3_SECRET_KEY_FILE`.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Машина: Intel Core i5-10300H, 8 логических процессоров, 17008717824 байт памяти. Дата прогона 12 ноября 2026.
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Критерий | Команда и что я увидела | Файл |
|
||||
| --- | --- | --- |
|
||||
| Контракт на диске | `python -X utf8 scripts/check_contract.py` напечатал `PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.` | вывод в этом разделе |
|
||||
| Публичный smoke | `python -X utf8 tests/smoke.py --base-url http://127.0.0.1:8080` напечатал `PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.` Ответ загрузки 202, затем ready, оригинал совпал, thumbnail 256×160 | — |
|
||||
| Сохранность после рестарта хранилищ | `tests/resilience.py --mode persistence`, hook `scripts/restart.sh`, через WSL, код выхода 0. PASS. StartedAt изменился у postgres, redis, minio, rabbitmq и обеих API. У балансировщика и worker не изменился. Volume не удалялись | `evidence/persistence.txt` |
|
||||
| Остановка одной реплики | Два прогона `tests/resilience.py --mode failover` через WSL. До остановки были видны 2 реплики. Ошибок за прогон: 1 и 1. Успешных записей: 97 и 98. Восстановление: 1,469 с и 0,975 с. Оба раза остановился api-2 с кодом 143. Отдельно `TARGET_INSTANCE_ID=2e03f20dd3b5` остановил api-1, тоже код 143, api-2 остался работать | `evidence/failover.txt` |
|
||||
| Старые текстовые правила и границы файла | `python -X utf8 student-tests/run_negative.py`. Пустой текст, битый JSON и короткий пароль — 400. Пропущенный список и `[]` — одно сообщение. Другое тело при том же ключе — 409, второй строки нет. Тот же ключ в другом чате — новое сообщение. Чужой чат и неизвестный UUID вложения — 404. Без токена — 401. Имя с `/` — 400. Заголовок 8000×8000 и тело больше 10 МиБ — 413. Битый PNG и PNG с типом image/jpeg — 415 | `evidence/negative.json` |
|
||||
| Файл при остановленном worker | `python -X utf8 student-tests/run_pipeline.py`. Загрузка 202, скачивание 409, текстовое сообщение 201, вложение можно прикрепить к сообщению. В очереди `attachments.process` было 1 сообщение до рестарта API, 1 после рестарта API и 1 после рестарта RabbitMQ. Затем worker довёл файл до ready. SHA-256 оригинала `99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75`. Thumbnail 256×160, crop 128×128, optimized 320×200, строк вариантов 4 | `evidence/pipeline.json`, сценарий `durability` |
|
||||
| Разрыв COMMIT и publish | Тот же скрипт, сценарий `outbox_gap`. Worker остановлен, публикация из API пропущена специально. У outbox `published_at` пустой, глубина очереди 0. После старта worker файл стал ready, строк вариантов 4 | `evidence/pipeline.json` |
|
||||
| Повтор, чужой порядок, старая версия | Дубль того же задания оставил 4 строки вариантов. Сообщение с `attempt=2` при задании на попытке 1 не сменило попытку, итог ready. Новая `pipeline_version` 2 дошла до ready, старое задание стало `superseded`, повтор старого задания байты thumbnail не изменил | `evidence/pipeline.json` |
|
||||
| Падение до ack | `crash_before_effect`: RestartCount worker с 0 стал 1, файл ready. `crash_after_objects`: StartedAt сменился с `2026-11-12T09:24:05.644199607Z` на `2026-11-12T09:24:09.042866995Z`. `crash_after_commit`: с `2026-11-12T09:24:21.448659497Z` на `2026-11-12T09:24:24.489665631Z`. Во всех трёх случаях строк вариантов 4 | `evidence/pipeline.json` |
|
||||
| Временная ошибка и битый файл | `transient_once`: события `leased`, `retry`, `leased`, `ready`, попытка стала 2, строк вариантов 4. Битые байты в staging дали `invalid_image` на попытке 1. После возврата байтов replay вернул `queued`, повторный replay вернул `already_ready`, строк вариантов 4 | `evidence/pipeline.json` |
|
||||
| Истёкшая аренда | Заданию вручную поставлены `lease_owner=dead` и `lease_until` на две минуты в прошлом, неопубликованный outbox удалён. Worker сам вернул его в работу. События: `lease_expired`, `leased`, `ready`. Попытка осталась 1. До ready прошло 17,078 с от старта worker | `evidence/pipeline.json`, сценарий `lease` |
|
||||
| Текст 503 | После остановки Redis балансировщик ответил 503 страницей HAProxy: `503 Service Unavailable` и `No server is available to handle this request.` Прямой запрос в контейнер API ответил 503 и «Сервис временно недоступен». В обоих текстах не было слов redis, postgres, minio, password, select, rabbit, amqp, secret, database, bucket, messenger-attachments. После healthy Redis первый запрос через балансировщик получил 200 за 0,063 с | `evidence/pipeline.json`, сценарий `ready_503` |
|
||||
| EXIF | JPEG 200×100 с Orientation 6 загружен через API. Ответ 202, затем ready. Thumbnail 100×200, crop 128×128, optimized 100×200. Отдельный вызов `render` в контейнере worker сохранил исходные байты | `evidence/exif.json` |
|
||||
| Очередь failed в конце | `rabbitmqctl list_queues`: `attachments.failed` — 1, `attachments.process` — 0, `attachments.retry` — 0. Единица осталась от намеренного `invalid_image`. Replay пишет новую рабочую доставку и старое сообщение DLQ не удаляет | вывод этой команды |
|
||||
|
||||
## Эксперименты и отказы
|
||||
Публичные файлы в `contracts/` и `tests/` я не меняла. После всех сценариев smoke я запустила ещё раз, он снова напечатал PASS.
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
## Трек 2
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Я выбрала трек 2, а не трек про очередь под перегрузкой. Трек 2 стоит на тех же механизмах, которые и так нужны для оценки 4: outbox, повтор с паузой, падение после записи объекта и до ack, явный replay. Их можно показать на одном файле и конкретных идентификаторах. Трек 1 требовал бы отдельный нагрузочный опыт. Таких чисел у меня нет, и я их не придумывала.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
Гипотеза: повторная доставка, чужой порядок и старое задание после новой версии не создают второй набор файлов и не затирают более новый результат. Доставка при этом остаётся at-least-once.
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Как я ломала worker. В таблице `fault_injections` есть строка с видом сбоя. Пока строки нет, код её не включает. Виды: пропуск публикации из API, выход процесса до эффекта, после записи объектов и после COMMIT, одна временная ошибка. Выход — `os._exit(75)`. Политика `unless-stopped` поднимает контейнер снова.
|
||||
|
||||
Что наблюдали. Все семь сценариев трека в `evidence/pipeline.json` закончились `ok: true`. Новый `pipeline_version` остался 2. Старое задание получило `error_code=superseded` только на строке задания, у вложения `error_code` пустой. Повтор успешного задания вернул `already_ready` и не создал пятый вариант.
|
||||
|
||||
Это не exactly-once. Брокер может принести сообщение дважды. Второй раз worker видит уже записанный эффект или уже закрытое задание и только подтверждает доставку.
|
||||
|
||||
## Ограничения
|
||||
|
||||
Стенд переживает остановку одной реплики API, рестарт Postgres, Redis, MinIO и RabbitMQ, если volume на месте, и остановку worker на время приёма файла. Выключение компьютера и поломка диска данные не сохраняют. Postgres, Redis, MinIO, RabbitMQ и HAProxy в стенде по одному экземпляру. Их отказ останавливает соответствующую часть сервиса.
|
||||
|
||||
Прямую загрузку по временной ссылке я не добавляла.
|
||||
|
||||
Если обе реплики API не проходят проверку готовности, клиент через HAProxy видит английскую HTML-страницу 503, а не JSON API. Сам API при этом отвечает фиксированной русской фразой. Оба текста я проверила: имён хранилищ, SQL и секретов там нет.
|
||||
|
||||
Непрерывная публикация outbox живёт в worker. Если worker остановлен и разовая публикация из API пропущена, задание ждёт старта worker. Это я показала. Отдельного второго публикатора нет.
|
||||
|
||||
Аренду я проверила записью в SQL уже истёкшего срока, а не убийством процесса на середине Pillow. Сработал путь sweeper: `lease_expired`, затем обычная обработка, попытка не выросла.
|
||||
|
||||
Если декодирование упирается в срок аренды, рабочий поток Pillow остаётся демоном и может досчитать кадр уже после того, как попытка брошена. Такой случай я не запускала.
|
||||
|
||||
Строки `fault_injections` — учебный переключатель. В обычной работе таблица пустая, и эти ветки не выполняются.
|
||||
|
||||
Повтор успешного задания не исполняет его заново, но старое сообщение в `attachments.failed` не стирает. После прогона там осталось 1 сообщение.
|
||||
|
||||
Байты JPEG и WebP зависят от этой сборки Pillow. Другая версия библиотеки может дать другой файл при тех же правилах размера.
|
||||
|
||||
Нагрузку, throughput и p95 я не измеряла. Трек 1 я не заявляю.
|
||||
|
||||
+290
@@ -0,0 +1,290 @@
|
||||
name: mtusi-lab5
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16.10-bookworm@sha256:38471f330eb885e04de130b768d6db4e10469e2311879c7e5c699f6d2d8a1c74
|
||||
environment:
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
command:
|
||||
- postgres
|
||||
- -c
|
||||
- fsync=on
|
||||
- -c
|
||||
- synchronous_commit=on
|
||||
- -c
|
||||
- full_page_writes=on
|
||||
- -c
|
||||
- max_connections=80
|
||||
- -c
|
||||
- shared_buffers=64MB
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
secrets:
|
||||
- postgres_password
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
start_period: 10s
|
||||
cpus: "1.00"
|
||||
mem_limit: 512m
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- lab
|
||||
|
||||
redis:
|
||||
image: redis:7.4.2-bookworm@sha256:fbdbaea47b9ae4ecc2082ecdb4e1cea81e32176ffb1dcf643d422ad07427e5d9
|
||||
command:
|
||||
- redis-server
|
||||
- --appendonly
|
||||
- "yes"
|
||||
- --appendfsync
|
||||
- everysec
|
||||
- --dir
|
||||
- /data
|
||||
- --save
|
||||
- "60"
|
||||
- "1"
|
||||
- --bind
|
||||
- 0.0.0.0
|
||||
- --protected-mode
|
||||
- "no"
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
start_period: 5s
|
||||
cpus: "0.25"
|
||||
mem_limit: 128m
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- lab
|
||||
|
||||
rabbitmq:
|
||||
image: rabbitmq:3.13.7-alpine@sha256:d7af1c87c5f1eda13fcfca06db452bf3aeab6619fc3358b68535c0c02c4e52bc
|
||||
hostname: rabbitmq
|
||||
entrypoint: ["/bin/sh", "/init/rabbit-entry.sh"]
|
||||
volumes:
|
||||
- rabbitmq-data:/var/lib/rabbitmq
|
||||
- ./scripts/rabbit-entry.sh:/init/rabbit-entry.sh:ro
|
||||
secrets:
|
||||
- rabbit_user
|
||||
- rabbit_password
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
|
||||
interval: 5s
|
||||
timeout: 10s
|
||||
retries: 12
|
||||
start_period: 180s
|
||||
cpus: "1.00"
|
||||
mem_limit: 1024m
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- lab
|
||||
|
||||
minio:
|
||||
image: pgsty/minio@sha256:b6bfe7239bfc83fb90d31612d9704d86039dd714f7904b3f1ad68f211e602372
|
||||
command: ["server", "/data", "--console-address", "127.0.0.1:9001"]
|
||||
environment:
|
||||
MINIO_ROOT_USER_FILE: /run/secrets/minio_root_user
|
||||
MINIO_ROOT_PASSWORD_FILE: /run/secrets/minio_root_password
|
||||
volumes:
|
||||
- minio-data:/data
|
||||
secrets:
|
||||
- minio_root_user
|
||||
- minio_root_password
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1:9000/minio/health/live"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
start_period: 8s
|
||||
cpus: "0.50"
|
||||
mem_limit: 512m
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- lab
|
||||
|
||||
minio-init:
|
||||
image: pgsty/minio@sha256:b6bfe7239bfc83fb90d31612d9704d86039dd714f7904b3f1ad68f211e602372
|
||||
entrypoint: ["/bin/sh", "/init/minio-init.sh"]
|
||||
volumes:
|
||||
- ./scripts/minio-init.sh:/init/minio-init.sh:ro
|
||||
- ./infra/minio-api-policy.json:/init/policy.json:ro
|
||||
secrets:
|
||||
- minio_root_user
|
||||
- minio_root_password
|
||||
- s3_access_key
|
||||
- s3_secret_key
|
||||
depends_on:
|
||||
minio:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
networks:
|
||||
- lab
|
||||
|
||||
migrate:
|
||||
build: .
|
||||
image: mtusi-lab5-api:local
|
||||
command: ["python", "/app/src/migrate.py"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
secrets:
|
||||
- database_url
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
disable: true
|
||||
cpus: "0.50"
|
||||
mem_limit: 256m
|
||||
restart: "no"
|
||||
networks:
|
||||
- lab
|
||||
|
||||
api:
|
||||
build: .
|
||||
image: mtusi-lab5-api:local
|
||||
environment:
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: "8080"
|
||||
APP_ENV: ${APP_ENV:-lab}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
SESSION_STORE: redis
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
POOL_MAX: ${POOL_MAX:-10}
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
S3_ENDPOINT: minio:9000
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_SECURE: "false"
|
||||
S3_ACCESS_KEY_FILE: /run/secrets/s3_access_key
|
||||
S3_SECRET_KEY_FILE: /run/secrets/s3_secret_key
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
RECONCILE_GRACE_SECONDS: ${RECONCILE_GRACE_SECONDS:-120}
|
||||
secrets:
|
||||
- database_url
|
||||
- cache_url
|
||||
- broker_url
|
||||
- s3_access_key
|
||||
- s3_secret_key
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
redis:
|
||||
condition: service_healthy
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
minio:
|
||||
condition: service_healthy
|
||||
minio-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 25s
|
||||
init: true
|
||||
stop_grace_period: 10s
|
||||
cpus: "0.50"
|
||||
mem_limit: 512m
|
||||
deploy:
|
||||
replicas: 2
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- lab
|
||||
|
||||
worker:
|
||||
build: .
|
||||
image: mtusi-lab5-api:local
|
||||
command: ["python", "/app/src/worker.py"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
S3_ENDPOINT: minio:9000
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_SECURE: "false"
|
||||
S3_ACCESS_KEY_FILE: /run/secrets/s3_access_key
|
||||
S3_SECRET_KEY_FILE: /run/secrets/s3_secret_key
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||
secrets:
|
||||
- database_url
|
||||
- broker_url
|
||||
- s3_access_key
|
||||
- s3_secret_key
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
minio:
|
||||
condition: service_healthy
|
||||
minio-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
disable: true
|
||||
init: true
|
||||
stop_grace_period: 20s
|
||||
cpus: "1.00"
|
||||
mem_limit: 768m
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- lab
|
||||
|
||||
lb:
|
||||
image: haproxy:2.8.14@sha256:2344d1efbf7a932c7a3d49b60748c28af81c70fd11fcadd983e2be1f0376a69d
|
||||
volumes:
|
||||
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
ports:
|
||||
- "127.0.0.1:${APP_PORT:-8080}:8080"
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_healthy
|
||||
cpus: "0.25"
|
||||
mem_limit: 128m
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- lab
|
||||
|
||||
networks:
|
||||
lab:
|
||||
name: mtusi-lab5-net
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
redis-data:
|
||||
minio-data:
|
||||
rabbitmq-data:
|
||||
|
||||
secrets:
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
cache_url:
|
||||
file: ./secrets/cache_url
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
minio_root_user:
|
||||
file: ./secrets/minio_root_user
|
||||
minio_root_password:
|
||||
file: ./secrets/minio_root_password
|
||||
s3_access_key:
|
||||
file: ./secrets/s3_access_key
|
||||
s3_secret_key:
|
||||
file: ./secrets/s3_secret_key
|
||||
rabbit_user:
|
||||
file: ./secrets/rabbit_user
|
||||
rabbit_password:
|
||||
file: ./secrets/rabbit_password
|
||||
broker_url:
|
||||
file: ./secrets/broker_url
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"file": "JPEG 200x100, EXIF Orientation 6, red pixels",
|
||||
"in_process_render": {
|
||||
"where": "docker exec mtusi-lab5-worker-1 python, function images.render",
|
||||
"original_unchanged": true,
|
||||
"thumbnail": [100, 200],
|
||||
"crop": [128, 128],
|
||||
"optimized": [100, 200]
|
||||
},
|
||||
"http_upload": {
|
||||
"command": "POST /api/v1/chats/{id}/attachments?filename=orient6.jpg",
|
||||
"upload_status": "queued",
|
||||
"final_status": "ready",
|
||||
"variants": {
|
||||
"thumbnail": [100, 200],
|
||||
"crop": [128, 128],
|
||||
"optimized": [100, 200]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
Команда (через WSL):
|
||||
python3 -X utf8 tests/resilience.py --mode failover --action-script scripts/stop-one.sh --base-url http://127.0.0.1:8080
|
||||
|
||||
Прогон 1, код выхода 0:
|
||||
{"observed_instances_before": 2, "successful_writes": 97, "failed_attempts": 1, "first_recovery_seconds": 1.469}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Остановлен mtusi-lab5-api-2, exit 143, FinishedAt 2026-11-12T09:00:09.569371803Z.
|
||||
mtusi-lab5-api-1 остался running.
|
||||
|
||||
Между прогонами реплика поднята командой:
|
||||
docker compose --env-file .env up -d --no-deps --scale api=2 api
|
||||
|
||||
Прогон 2, код выхода 0:
|
||||
{"observed_instances_before": 2, "successful_writes": 98, "failed_attempts": 1, "first_recovery_seconds": 0.975}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Снова остановлен mtusi-lab5-api-2, exit 143. api-1 остался running.
|
||||
Публичный тест оба раза выбрал ту реплику, которая последней ответила. Это был api-2.
|
||||
|
||||
Отдельная проверка второй реплики, тот же scripts/stop-one.sh:
|
||||
TARGET_INSTANCE_ID=2e03f20dd3b5
|
||||
Остановлен mtusi-lab5-api-1, hostname 2e03f20dd3b5, exit 143.
|
||||
mtusi-lab5-api-2 остался running.
|
||||
|
||||
После проверки реплика снова запущена. Дата: 12 ноября 2026.
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"command": "python -X utf8 student-tests/run_negative.py http://127.0.0.1:8080",
|
||||
"result": "PASS negative",
|
||||
"results": [
|
||||
{"name": "chat_uuid_case", "status": 200},
|
||||
{"name": "blank_text", "status": 400},
|
||||
{"name": "bad_json", "status": 400},
|
||||
{"name": "short_password", "status": 400},
|
||||
{"name": "omitted_equals_empty", "status": 201},
|
||||
{"name": "same_key_other_body", "status": 409},
|
||||
{"name": "no_second_row", "status": 200},
|
||||
{"name": "same_key_other_chat", "status": 201},
|
||||
{"name": "foreign_chat", "status": 404},
|
||||
{"name": "bad_cursor", "status": 400},
|
||||
{"name": "slash_name", "status": 400},
|
||||
{"name": "huge_pixels", "status": 413},
|
||||
{"name": "bad_png", "status": 415},
|
||||
{"name": "type_mismatch", "status": 415},
|
||||
{"name": "oversize", "status": 413},
|
||||
{"name": "unknown_attachment", "status": 404},
|
||||
{"name": "no_token", "status": 401}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
Команда (через WSL, потому что Windows не запускает .sh):
|
||||
python3 -X utf8 tests/resilience.py --mode persistence --action-script scripts/restart.sh --base-url http://127.0.0.1:8080
|
||||
|
||||
Вывод:
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
Код выхода: 0. Дата: 12 ноября 2026.
|
||||
|
||||
StartedAt после hook:
|
||||
postgres 2026-11-12T08:58:51.311769003Z
|
||||
redis 2026-11-12T08:58:50.947096648Z
|
||||
minio 2026-11-12T08:58:51.30945974Z
|
||||
rabbitmq 2026-11-12T08:59:03.356512147Z
|
||||
api-1 2026-11-12T08:59:15.032550589Z
|
||||
api-2 2026-11-12T08:59:14.961115981Z
|
||||
lb 2026-11-12T08:56:49.466505107Z (не перезапускался)
|
||||
worker 2026-11-12T08:56:32.031493932Z (скрипт его не трогает)
|
||||
|
||||
Hook перезапускает postgres, redis, minio, rabbitmq и api. Volume не удалялись.
|
||||
@@ -0,0 +1,318 @@
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"name": "durability",
|
||||
"ok": true,
|
||||
"seconds": 122.813,
|
||||
"detail": {
|
||||
"upload_status": "queued",
|
||||
"content_while_queued": 409,
|
||||
"text_status": 201,
|
||||
"text_id": "0cb05d10-70d9-46a2-9fad-3272a92af1b3",
|
||||
"linked_id": "5ffaee48-8e7c-4742-a5f1-b2991fed1a4b",
|
||||
"queue_before_api_restart": 1,
|
||||
"queue_after_api_restart": 1,
|
||||
"queue_after_rabbit_restart": 1,
|
||||
"picture": {
|
||||
"pipeline_version": 1,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "outbox_gap",
|
||||
"ok": true,
|
||||
"seconds": 26.781,
|
||||
"detail": {
|
||||
"unpublished_before_worker": 1,
|
||||
"queue_depth": 0,
|
||||
"picture": {
|
||||
"pipeline_version": 1,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "duplicate",
|
||||
"ok": true,
|
||||
"seconds": 19.203,
|
||||
"detail": {
|
||||
"republish": true,
|
||||
"picture": {
|
||||
"pipeline_version": 1,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "reorder",
|
||||
"ok": true,
|
||||
"seconds": 19.688,
|
||||
"detail": {
|
||||
"attempt": 1,
|
||||
"picture": {
|
||||
"pipeline_version": 1,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "stale_version",
|
||||
"ok": true,
|
||||
"seconds": 26.906,
|
||||
"detail": {
|
||||
"bump": {
|
||||
"job_id": "a7af6898-08db-4a00-aff6-a2b654e3e0da",
|
||||
"pipeline_version": 2
|
||||
},
|
||||
"old_job_status": "superseded",
|
||||
"old_error_code": "superseded",
|
||||
"attachment_error_code": null,
|
||||
"picture": {
|
||||
"pipeline_version": 2,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
},
|
||||
"thumbnail_unchanged": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "crash_before_effect",
|
||||
"ok": true,
|
||||
"seconds": 19.281,
|
||||
"detail": {
|
||||
"restart_count_before": 0,
|
||||
"restart_count_after": 1,
|
||||
"picture": {
|
||||
"pipeline_version": 1,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "crash_after_objects",
|
||||
"ok": true,
|
||||
"seconds": 14.968,
|
||||
"detail": {
|
||||
"started_at": "2026-11-12T09:24:05.644199607Z",
|
||||
"restarted_at": "2026-11-12T09:24:09.042866995Z",
|
||||
"picture": {
|
||||
"pipeline_version": 1,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "crash_after_commit",
|
||||
"ok": true,
|
||||
"seconds": 11.563,
|
||||
"detail": {
|
||||
"started_at": "2026-11-12T09:24:21.448659497Z",
|
||||
"restarted_at": "2026-11-12T09:24:24.489665631Z",
|
||||
"picture": {
|
||||
"pipeline_version": 1,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "transient_once",
|
||||
"ok": true,
|
||||
"seconds": 13.187,
|
||||
"detail": {
|
||||
"attempt": 2,
|
||||
"events": [
|
||||
"leased",
|
||||
"retry",
|
||||
"leased",
|
||||
"ready"
|
||||
],
|
||||
"variant_rows": 4,
|
||||
"status": "ready"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "permanent_replay",
|
||||
"ok": true,
|
||||
"seconds": 28.407,
|
||||
"detail": {
|
||||
"corrupt_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"failed_code": "invalid_image",
|
||||
"attempt_at_fail": 1,
|
||||
"restored_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"replay": {
|
||||
"result": "queued"
|
||||
},
|
||||
"second_replay": {
|
||||
"result": "already_ready"
|
||||
},
|
||||
"variant_rows": 4,
|
||||
"picture": {
|
||||
"pipeline_version": 1,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "lease",
|
||||
"ok": true,
|
||||
"seconds": 41.14,
|
||||
"detail": {
|
||||
"seconds_until_ready": 17.078,
|
||||
"events": [
|
||||
"lease_expired",
|
||||
"leased",
|
||||
"ready"
|
||||
],
|
||||
"attempt": 1,
|
||||
"picture": {
|
||||
"pipeline_version": 1,
|
||||
"variant_rows": 4,
|
||||
"thumbnail": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"crop": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"optimized": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "ready_503",
|
||||
"ok": true,
|
||||
"seconds": 13.625,
|
||||
"detail": {
|
||||
"lb_status": 503,
|
||||
"lb_body": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n",
|
||||
"api_status": 503,
|
||||
"api_message": "Сервис временно недоступен",
|
||||
"forbidden": [],
|
||||
"recovered": 200,
|
||||
"recovery_seconds": 0.063
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
global
|
||||
maxconn 256
|
||||
tune.bufsize 65536
|
||||
tune.maxrewrite 8192
|
||||
|
||||
defaults
|
||||
mode http
|
||||
timeout connect 500ms
|
||||
timeout client 60s
|
||||
timeout server 60s
|
||||
timeout http-request 60s
|
||||
timeout http-keep-alive 2s
|
||||
timeout queue 2s
|
||||
timeout check 800ms
|
||||
option redispatch
|
||||
option dontlognull
|
||||
retries 2
|
||||
retry-on conn-failure empty-response 502 503 504
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
hold other 1s
|
||||
hold refused 1s
|
||||
hold nx 1s
|
||||
hold timeout 1s
|
||||
hold obsolete 1s
|
||||
accepted_payload_size 8192
|
||||
|
||||
frontend web
|
||||
bind *:8080
|
||||
default_backend api
|
||||
|
||||
backend api
|
||||
balance roundrobin
|
||||
# Короткий JSON сообщения можно повторить. Загрузку файла не повторяем:
|
||||
# повтор создал бы второй объект. Тело загрузки не буферизуется целиком.
|
||||
acl upload method POST path_reg -i ^/api/v1/chats/[0-9a-fA-F-]+/attachments$
|
||||
http-request wait-for-body time 2s if METH_POST !upload
|
||||
http-request disable-l7-retry if upload
|
||||
option httpchk
|
||||
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host api
|
||||
http-check expect status 200
|
||||
server-template api 8 api:8080 check inter 500ms fall 2 rise 2 resolvers docker init-addr none resolve-prefer ipv4 on-marked-down shutdown-sessions
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:DeleteObject"
|
||||
],
|
||||
"Resource": ["arn:aws:s3:::messenger-attachments/*"]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetBucketLocation", "s3:ListBucket"],
|
||||
"Resource": ["arn:aws:s3:::messenger-attachments"]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
-- Пользователи, чаты, участники и сообщения.
|
||||
-- Создание чата в приложении идёт одной транзакцией: строка chats и строки chat_members.
|
||||
-- Индекс messages_history_idx совпадает с keyset-запросом истории:
|
||||
-- WHERE chat_id = $1 AND (created_at, id) > ($2, $3)
|
||||
-- ORDER BY created_at, id
|
||||
-- LIMIT $4
|
||||
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL,
|
||||
display_name text NOT NULL,
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
CONSTRAINT users_username_unique UNIQUE (username),
|
||||
CONSTRAINT users_username_format CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
CONSTRAINT users_display_name_nonblank CHECK (btrim(display_name) <> ''),
|
||||
CONSTRAINT users_display_name_len CHECK (char_length(display_name) BETWEEN 1 AND 100)
|
||||
);
|
||||
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
CONSTRAINT chats_title_nonblank CHECK (btrim(title) <> ''),
|
||||
CONSTRAINT chats_title_len CHECK (char_length(title) BETWEEN 1 AND 100)
|
||||
);
|
||||
|
||||
CREATE TABLE chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats (id),
|
||||
user_id uuid NOT NULL REFERENCES users (id),
|
||||
ordinal smallint NOT NULL,
|
||||
PRIMARY KEY (chat_id, user_id),
|
||||
CONSTRAINT chat_members_ordinal_nonneg CHECK (ordinal >= 0)
|
||||
);
|
||||
|
||||
CREATE INDEX chat_members_user_idx ON chat_members (user_id, chat_id);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats (id),
|
||||
sender_id uuid NOT NULL REFERENCES users (id),
|
||||
body text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
CONSTRAINT messages_body_nonblank CHECK (btrim(body) <> ''),
|
||||
CONSTRAINT messages_body_len CHECK (char_length(body) BETWEEN 1 AND 4000)
|
||||
);
|
||||
|
||||
CREATE INDEX messages_history_idx ON messages (chat_id, created_at, id);
|
||||
@@ -0,0 +1,19 @@
|
||||
-- Результат успешного POST сообщения.
|
||||
-- Область ключа: пользователь + метод + путь чата + Idempotency-Key.
|
||||
-- Строка пишется в той же транзакции, что и messages, поэтому 5xx
|
||||
-- не оставляет «занятый» ключ. Другое тело при живом ключе — отказ
|
||||
-- на уровне приложения (хеш не совпал), без второй строки messages.
|
||||
|
||||
CREATE TABLE idempotency_keys (
|
||||
user_id uuid NOT NULL REFERENCES users (id),
|
||||
http_method text NOT NULL,
|
||||
request_path text NOT NULL,
|
||||
idem_key text NOT NULL,
|
||||
request_hash text NOT NULL,
|
||||
message_id uuid NOT NULL REFERENCES messages (id),
|
||||
created_at timestamptz NOT NULL,
|
||||
PRIMARY KEY (user_id, http_method, request_path, idem_key),
|
||||
CONSTRAINT idempotency_method_post CHECK (http_method = 'POST'),
|
||||
CONSTRAINT idempotency_key_format CHECK (idem_key ~ '^[A-Za-z0-9._:-]{1,128}$'),
|
||||
CONSTRAINT idempotency_hash_len CHECK (char_length(request_hash) = 64)
|
||||
);
|
||||
@@ -0,0 +1,42 @@
|
||||
-- Метаданные вложения. Байты лежат в приватном bucket, не в этой таблице.
|
||||
-- Пустой текст допустим, если к сообщению прикреплено вложение. Пробелы по-прежнему нельзя.
|
||||
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_body_nonblank;
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_body_len;
|
||||
ALTER TABLE messages
|
||||
ADD CONSTRAINT messages_body_len CHECK (char_length(body) <= 4000);
|
||||
ALTER TABLE messages
|
||||
ADD CONSTRAINT messages_body_nonblank CHECK (body = '' OR btrim(body) <> '');
|
||||
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats (id),
|
||||
owner_id uuid NOT NULL REFERENCES users (id),
|
||||
filename text NOT NULL,
|
||||
content_type text NOT NULL,
|
||||
size_bytes bigint NOT NULL,
|
||||
sha256 text NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
status text NOT NULL DEFAULT 'ready',
|
||||
created_at timestamptz NOT NULL,
|
||||
updated_at timestamptz NOT NULL,
|
||||
CONSTRAINT attachments_filename_len CHECK (char_length(filename) BETWEEN 1 AND 255),
|
||||
CONSTRAINT attachments_content_type_chk CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||
CONSTRAINT attachments_size_chk CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
CONSTRAINT attachments_sha_chk CHECK (char_length(sha256) = 64),
|
||||
CONSTRAINT attachments_status_chk CHECK (status = 'ready'),
|
||||
CONSTRAINT attachments_key_chk CHECK (char_length(object_key) BETWEEN 1 AND 512)
|
||||
);
|
||||
|
||||
CREATE INDEX attachments_chat_idx ON attachments (chat_id, created_at);
|
||||
|
||||
CREATE TABLE message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages (id),
|
||||
attachment_id uuid NOT NULL REFERENCES attachments (id),
|
||||
ordinal integer NOT NULL,
|
||||
PRIMARY KEY (message_id, attachment_id),
|
||||
UNIQUE (message_id, ordinal),
|
||||
CONSTRAINT message_attachments_ordinal_chk CHECK (ordinal >= 0 AND ordinal < 10)
|
||||
);
|
||||
|
||||
CREATE INDEX message_attachments_attachment_idx ON message_attachments (attachment_id);
|
||||
@@ -0,0 +1,108 @@
|
||||
-- Задание на обработку, варианты и outbox.
|
||||
-- Сообщение брокера не содержит байты и пароли: в outbox лежит только JSON задания.
|
||||
-- Ключ эффекта — (attachment_id, pipeline_version, name). Повтор не создаёт вторую строку.
|
||||
|
||||
ALTER TABLE attachments DROP CONSTRAINT attachments_status_chk;
|
||||
|
||||
ALTER TABLE attachments ALTER COLUMN status SET DEFAULT 'queued';
|
||||
|
||||
ALTER TABLE attachments ADD COLUMN error_code text;
|
||||
|
||||
ALTER TABLE attachments ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1;
|
||||
|
||||
ALTER TABLE attachments ADD COLUMN staging_key text;
|
||||
|
||||
ALTER TABLE attachments ADD COLUMN staging_deleted_at timestamptz;
|
||||
|
||||
ALTER TABLE attachments ADD CONSTRAINT attachments_status_chk CHECK (status IN ('queued', 'processing', 'ready', 'failed'));
|
||||
|
||||
ALTER TABLE attachments ADD CONSTRAINT attachments_pipeline_chk CHECK (pipeline_version >= 1);
|
||||
|
||||
ALTER TABLE attachments ADD CONSTRAINT attachments_error_chk CHECK (
|
||||
(status = 'failed' AND error_code IN ('invalid_image', 'processing_failed', 'retry_exhausted', 'processing_timeout'))
|
||||
OR (status <> 'failed' AND error_code IS NULL)
|
||||
);
|
||||
|
||||
CREATE TABLE attachment_jobs (
|
||||
id uuid PRIMARY KEY,
|
||||
attachment_id uuid NOT NULL REFERENCES attachments (id),
|
||||
pipeline_version integer NOT NULL,
|
||||
status text NOT NULL,
|
||||
attempt integer NOT NULL,
|
||||
max_attempts integer NOT NULL,
|
||||
lease_owner text,
|
||||
lease_until timestamptz,
|
||||
next_attempt_at timestamptz,
|
||||
error_code text,
|
||||
created_at timestamptz NOT NULL,
|
||||
updated_at timestamptz NOT NULL,
|
||||
CONSTRAINT attachment_jobs_status_chk CHECK (status IN ('queued', 'processing', 'ready', 'failed', 'superseded')),
|
||||
CONSTRAINT attachment_jobs_attempt_chk CHECK (attempt >= 1 AND attempt <= max_attempts),
|
||||
CONSTRAINT attachment_jobs_max_chk CHECK (max_attempts BETWEEN 1 AND 20),
|
||||
CONSTRAINT attachment_jobs_version_chk CHECK (pipeline_version >= 1),
|
||||
CONSTRAINT attachment_jobs_error_chk CHECK (
|
||||
error_code IS NULL OR error_code IN (
|
||||
'invalid_image', 'processing_failed', 'retry_exhausted', 'processing_timeout', 'superseded'
|
||||
)
|
||||
),
|
||||
CONSTRAINT attachment_jobs_one_version UNIQUE (attachment_id, pipeline_version)
|
||||
);
|
||||
|
||||
CREATE INDEX attachment_jobs_due_idx ON attachment_jobs (status, next_attempt_at);
|
||||
|
||||
CREATE TABLE attachment_variants (
|
||||
attachment_id uuid NOT NULL REFERENCES attachments (id),
|
||||
pipeline_version integer NOT NULL,
|
||||
name text NOT NULL,
|
||||
content_type text NOT NULL,
|
||||
size_bytes bigint NOT NULL,
|
||||
width integer NOT NULL,
|
||||
height integer NOT NULL,
|
||||
sha256 text NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
created_at timestamptz NOT NULL,
|
||||
PRIMARY KEY (attachment_id, pipeline_version, name),
|
||||
CONSTRAINT attachment_variants_name_chk CHECK (name IN ('original', 'thumbnail', 'crop', 'optimized')),
|
||||
CONSTRAINT attachment_variants_type_chk CHECK (content_type IN ('image/png', 'image/jpeg', 'image/webp')),
|
||||
CONSTRAINT attachment_variants_size_chk CHECK (size_bytes >= 1),
|
||||
CONSTRAINT attachment_variants_dim_chk CHECK (width >= 1 AND height >= 1),
|
||||
CONSTRAINT attachment_variants_sha_chk CHECK (sha256 ~ '^[a-f0-9]{64}$')
|
||||
);
|
||||
|
||||
CREATE TABLE outbox (
|
||||
id uuid PRIMARY KEY,
|
||||
job_id uuid NOT NULL REFERENCES attachment_jobs (id),
|
||||
attempt integer NOT NULL,
|
||||
payload jsonb NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
published_at timestamptz,
|
||||
CONSTRAINT outbox_attempt_chk CHECK (attempt >= 1)
|
||||
);
|
||||
|
||||
CREATE INDEX outbox_pending_idx ON outbox (created_at) WHERE published_at IS NULL;
|
||||
|
||||
CREATE TABLE job_events (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
job_id uuid NOT NULL,
|
||||
attachment_id uuid NOT NULL,
|
||||
pipeline_version integer NOT NULL,
|
||||
kind text NOT NULL,
|
||||
detail text,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
CONSTRAINT job_events_kind_chk CHECK (char_length(kind) BETWEEN 1 AND 40),
|
||||
CONSTRAINT job_events_detail_chk CHECK (detail IS NULL OR char_length(detail) <= 64)
|
||||
);
|
||||
|
||||
CREATE TABLE fault_injections (
|
||||
id uuid PRIMARY KEY,
|
||||
kind text NOT NULL,
|
||||
attachment_id uuid,
|
||||
armed boolean NOT NULL DEFAULT true,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
CONSTRAINT fault_injections_kind_chk CHECK (kind IN (
|
||||
'skip_inline_publish', 'crash_before_effect', 'crash_after_objects',
|
||||
'crash_after_commit', 'transient_once'
|
||||
))
|
||||
);
|
||||
|
||||
CREATE INDEX fault_injections_armed_idx ON fault_injections (kind, created_at) WHERE armed;
|
||||
@@ -0,0 +1,10 @@
|
||||
fastapi==0.115.12
|
||||
uvicorn[standard]==0.34.2
|
||||
pydantic==2.11.4
|
||||
psycopg[binary]==3.2.6
|
||||
psycopg-pool==3.2.4
|
||||
redis==5.2.1
|
||||
argon2-cffi==23.1.0
|
||||
minio==7.2.15
|
||||
pillow==11.1.0
|
||||
pika==1.3.2
|
||||
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Создать локальные .env и secrets для стенда. Пароль в консоль не печатается."""
|
||||
import secrets
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SECRET_DIR = ROOT / "secrets"
|
||||
PASSWORD_FILE = SECRET_DIR / "postgres_password"
|
||||
DATABASE_FILE = SECRET_DIR / "database_url"
|
||||
CACHE_FILE = SECRET_DIR / "cache_url"
|
||||
MINIO_USER_FILE = SECRET_DIR / "minio_root_user"
|
||||
MINIO_PASSWORD_FILE = SECRET_DIR / "minio_root_password"
|
||||
S3_ACCESS_FILE = SECRET_DIR / "s3_access_key"
|
||||
S3_SECRET_FILE = SECRET_DIR / "s3_secret_key"
|
||||
RABBIT_USER_FILE = SECRET_DIR / "rabbit_user"
|
||||
RABBIT_PASSWORD_FILE = SECRET_DIR / "rabbit_password"
|
||||
BROKER_FILE = SECRET_DIR / "broker_url"
|
||||
ENV_FILE = ROOT / ".env"
|
||||
ENV_EXAMPLE = ROOT / ".env.example"
|
||||
|
||||
|
||||
def main():
|
||||
files = (PASSWORD_FILE, DATABASE_FILE, CACHE_FILE)
|
||||
if any(path.exists() for path in files) and not all(path.exists() for path in files):
|
||||
print("secrets/ заполнены не полностью, ничего не перезаписываю", file=sys.stderr)
|
||||
return 1
|
||||
SECRET_DIR.mkdir(exist_ok=True)
|
||||
if all(path.exists() for path in files):
|
||||
print("secrets already exist")
|
||||
else:
|
||||
password = secrets.token_hex(24)
|
||||
PASSWORD_FILE.write_text(password + "\n", encoding="utf-8")
|
||||
DATABASE_FILE.write_text(
|
||||
"postgresql://messenger:{}@postgres:5432/messenger\n".format(password),
|
||||
encoding="utf-8",
|
||||
)
|
||||
CACHE_FILE.write_text("redis://redis:6379/0\n", encoding="utf-8")
|
||||
print("secrets created")
|
||||
s3_files = (MINIO_USER_FILE, MINIO_PASSWORD_FILE, S3_ACCESS_FILE, S3_SECRET_FILE)
|
||||
if any(path.exists() for path in s3_files) and not all(path.exists() for path in s3_files):
|
||||
print("secrets/ для MinIO заполнены не полностью, ничего не перезаписываю", file=sys.stderr)
|
||||
return 1
|
||||
if all(path.exists() for path in s3_files):
|
||||
print("s3 secrets already exist")
|
||||
else:
|
||||
MINIO_USER_FILE.write_text("labroot", encoding="utf-8")
|
||||
MINIO_PASSWORD_FILE.write_text(secrets.token_hex(24), encoding="utf-8")
|
||||
S3_ACCESS_FILE.write_text("api" + secrets.token_hex(8), encoding="utf-8")
|
||||
S3_SECRET_FILE.write_text(secrets.token_hex(24), encoding="utf-8")
|
||||
print("s3 secrets created")
|
||||
broker_files = (RABBIT_USER_FILE, RABBIT_PASSWORD_FILE, BROKER_FILE)
|
||||
if any(path.exists() for path in broker_files) and not all(path.exists() for path in broker_files):
|
||||
print("secrets/ для брокера заполнены не полностью, ничего не перезаписываю", file=sys.stderr)
|
||||
return 1
|
||||
if all(path.exists() for path in broker_files):
|
||||
print("broker secrets already exist")
|
||||
else:
|
||||
user = "lab"
|
||||
password = secrets.token_hex(24)
|
||||
RABBIT_USER_FILE.write_text(user, encoding="utf-8")
|
||||
RABBIT_PASSWORD_FILE.write_text(password, encoding="utf-8")
|
||||
BROKER_FILE.write_text(
|
||||
"amqp://{}:{}@rabbitmq:5672/%2F\n".format(user, password),
|
||||
encoding="utf-8",
|
||||
)
|
||||
print("broker secrets created")
|
||||
if ENV_FILE.exists():
|
||||
print(".env already exists")
|
||||
else:
|
||||
ENV_FILE.write_text(ENV_EXAMPLE.read_text(encoding="utf-8"), encoding="utf-8")
|
||||
print(".env created")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+30
@@ -0,0 +1,30 @@
|
||||
#!/bin/sh
|
||||
# Идемпотентно создаёт приватный bucket и отдельного пользователя API.
|
||||
# Корневой пароль остаётся только у этого контейнера и у MinIO.
|
||||
set -eu
|
||||
ROOT_USER=$(tr -d '\r\n' < /run/secrets/minio_root_user)
|
||||
ROOT_PW=$(tr -d '\r\n' < /run/secrets/minio_root_password)
|
||||
ACCESS=$(tr -d '\r\n' < /run/secrets/s3_access_key)
|
||||
SECRET=$(tr -d '\r\n' < /run/secrets/s3_secret_key)
|
||||
|
||||
attempt=0
|
||||
until mc alias set local http://minio:9000 "$ROOT_USER" "$ROOT_PW" >/dev/null; do
|
||||
attempt=$((attempt + 1))
|
||||
if [ "$attempt" -gt 30 ]; then
|
||||
echo "minio alias failed" >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
mc mb --ignore-existing local/messenger-attachments >/dev/null
|
||||
mc anonymous set none local/messenger-attachments >/dev/null
|
||||
|
||||
if ! mc admin user info local "$ACCESS" >/dev/null 2>&1; then
|
||||
mc admin user add local "$ACCESS" "$SECRET" >/dev/null
|
||||
fi
|
||||
mc admin policy detach local messenger-api --user "$ACCESS" >/dev/null 2>&1 || true
|
||||
mc admin policy rm local messenger-api >/dev/null 2>&1 || true
|
||||
mc admin policy create local messenger-api /init/policy.json >/dev/null
|
||||
mc admin policy attach local messenger-api --user "$ACCESS" >/dev/null
|
||||
echo "minio bucket ready"
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
# Пароль читается из secret и попадает только в файл конфигурации внутри контейнера.
|
||||
set -eu
|
||||
user=$(tr -d '\r\n' < /run/secrets/rabbit_user)
|
||||
pass=$(tr -d '\r\n' < /run/secrets/rabbit_password)
|
||||
mkdir -p /tmp/rabbit-conf
|
||||
printf 'default_user = %s\ndefault_pass = %s\nvm_memory_high_watermark.absolute = 768MB\n' "$user" "$pass" > /tmp/rabbit-conf/rabbitmq.conf
|
||||
chmod 644 /tmp/rabbit-conf/rabbitmq.conf
|
||||
export RABBITMQ_CONFIG_FILE=/tmp/rabbit-conf/rabbitmq
|
||||
exec /usr/local/bin/docker-entrypoint.sh rabbitmq-server
|
||||
Executable
+8
@@ -0,0 +1,8 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
# Рестарт API и всех хранилищ со состоянием. Volume не удаляем, миграции заново не заливаем.
|
||||
# Балансировщик без своего состояния не трогаем: он продолжает слушать тот же порт.
|
||||
docker compose --env-file .env restart postgres redis minio rabbitmq
|
||||
docker compose --env-file .env up -d --wait --no-deps postgres redis minio rabbitmq
|
||||
docker compose --env-file .env restart api
|
||||
Executable
+23
@@ -0,0 +1,23 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
: "${TARGET_INSTANCE_ID:?}"
|
||||
|
||||
found=""
|
||||
for id in $(docker ps -q \
|
||||
--filter label=com.docker.compose.project=mtusi-lab5 \
|
||||
--filter label=com.docker.compose.service=api); do
|
||||
host=$(docker inspect -f '{{.Config.Hostname}}' "$id")
|
||||
short=$(printf %s "$id" | cut -c1-12)
|
||||
if [ "$host" = "$TARGET_INSTANCE_ID" ] || [ "$short" = "$TARGET_INSTANCE_ID" ]; then
|
||||
found=$id
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$found" ]; then
|
||||
echo "api replica not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# SIGTERM, затем процесс остаётся остановленным (restart: unless-stopped).
|
||||
docker stop --time 5 "$found" >/dev/null
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Ждать GET /health/ready через балансировщик. Без токенов и тел ошибок в выводе."""
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
URL = os.environ.get("BASE_URL", "http://127.0.0.1:8080").rstrip("/") + "/health/ready"
|
||||
|
||||
|
||||
def main():
|
||||
deadline = time.monotonic() + 90
|
||||
last = "not-started"
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
response = urllib.request.urlopen(URL, timeout=2)
|
||||
if response.status == 200 and response.headers.get("X-Instance-Id"):
|
||||
print("ready")
|
||||
return 0
|
||||
last = str(response.status)
|
||||
except (urllib.error.URLError, TimeoutError, OSError) as exc:
|
||||
last = type(exc).__name__
|
||||
time.sleep(1)
|
||||
print("not ready: " + last, file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+228
@@ -0,0 +1,228 @@
|
||||
"""Проверка PNG/JPEG и доступ к приватному bucket. Ключ объекта назначает сервер."""
|
||||
import io
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from minio import Minio
|
||||
from minio.error import S3Error
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
from PIL.Image import DecompressionBombError
|
||||
|
||||
from errors import ApiError, DependencyDown
|
||||
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
MAX_PIXELS = 40_000_000
|
||||
BUCKET = os.environ.get("S3_BUCKET", "messenger-attachments")
|
||||
|
||||
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
|
||||
|
||||
_client = None
|
||||
|
||||
|
||||
def read_secret(path):
|
||||
with open(path, encoding="utf-8") as handle:
|
||||
return handle.read().strip()
|
||||
|
||||
|
||||
def client():
|
||||
global _client
|
||||
if _client is None:
|
||||
endpoint = os.environ.get("S3_ENDPOINT", "minio:9000").strip()
|
||||
access = read_secret(os.environ["S3_ACCESS_KEY_FILE"])
|
||||
secret = read_secret(os.environ["S3_SECRET_KEY_FILE"])
|
||||
secure = os.environ.get("S3_SECURE", "false").strip().lower() == "true"
|
||||
_client = Minio(endpoint, access_key=access, secret_key=secret, secure=secure)
|
||||
return _client
|
||||
|
||||
|
||||
def bucket_ready():
|
||||
try:
|
||||
return client().bucket_exists(BUCKET)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def put_bytes(key, data, content_type, created_at):
|
||||
stamp = created_at.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
try:
|
||||
client().put_object(
|
||||
BUCKET,
|
||||
key,
|
||||
io.BytesIO(data),
|
||||
length=len(data),
|
||||
content_type=content_type,
|
||||
metadata={"created-at": stamp},
|
||||
)
|
||||
except S3Error as exc:
|
||||
raise DependencyDown() from exc
|
||||
|
||||
|
||||
def delete_quiet(key):
|
||||
try:
|
||||
client().remove_object(BUCKET, key)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def open_object(key):
|
||||
try:
|
||||
return client().get_object(BUCKET, key)
|
||||
except S3Error as exc:
|
||||
code = getattr(exc, "code", "")
|
||||
if code in ("NoSuchKey", "NoSuchBucket"):
|
||||
return None
|
||||
raise DependencyDown() from exc
|
||||
|
||||
|
||||
def object_key(chat_id, attachment_id):
|
||||
return "chats/{}/attachments/{}/original".format(chat_id, attachment_id)
|
||||
|
||||
|
||||
def staging_object_key(attachment_id):
|
||||
return "staging/{}/original".format(attachment_id)
|
||||
|
||||
|
||||
def variant_object_key(attachment_id, pipeline_version, name):
|
||||
return "ready/{}/v{}/{}".format(attachment_id, int(pipeline_version), name)
|
||||
|
||||
|
||||
def header_size(data, declared):
|
||||
if declared == "image/png":
|
||||
return _png_size(data)
|
||||
if declared == "image/jpeg":
|
||||
return _jpeg_size(data)
|
||||
return None
|
||||
|
||||
|
||||
def read_bytes(key):
|
||||
body = open_object(key)
|
||||
if body is None:
|
||||
return None
|
||||
try:
|
||||
return body.read()
|
||||
finally:
|
||||
body.close()
|
||||
body.release_conn()
|
||||
|
||||
|
||||
def declared_type(header):
|
||||
if not header:
|
||||
return None
|
||||
media = header.split(";", 1)[0].strip().lower()
|
||||
if media in ("image/png", "image/jpeg"):
|
||||
return media
|
||||
return None
|
||||
|
||||
|
||||
def _png_size(data):
|
||||
if len(data) < 24 or data[:8] != b"\x89PNG\r\n\x1a\n" or data[12:16] != b"IHDR":
|
||||
return None
|
||||
width = int.from_bytes(data[16:20], "big")
|
||||
height = int.from_bytes(data[20:24], "big")
|
||||
return width, height
|
||||
|
||||
|
||||
def _jpeg_size(data):
|
||||
if len(data) < 4 or data[:2] != b"\xff\xd8":
|
||||
return None
|
||||
index = 2
|
||||
limit = len(data)
|
||||
while index + 8 < limit:
|
||||
if data[index] != 0xFF:
|
||||
return None
|
||||
while index < limit and data[index] == 0xFF:
|
||||
index += 1
|
||||
if index >= limit:
|
||||
return None
|
||||
marker = data[index]
|
||||
index += 1
|
||||
if marker in (0xD8, 0xD9):
|
||||
continue
|
||||
if index + 2 > limit:
|
||||
return None
|
||||
segment = int.from_bytes(data[index:index + 2], "big")
|
||||
if segment < 2 or index + segment > limit:
|
||||
return None
|
||||
if marker in (0xC0, 0xC1, 0xC2):
|
||||
if segment < 7:
|
||||
return None
|
||||
height = int.from_bytes(data[index + 3:index + 5], "big")
|
||||
width = int.from_bytes(data[index + 5:index + 7], "big")
|
||||
return width, height
|
||||
index += segment
|
||||
return None
|
||||
|
||||
|
||||
def _too_many_pixels(width, height):
|
||||
if width <= 0 or height <= 0:
|
||||
raise ApiError(415, "unsupported_media_type", "Файл не является PNG или JPEG")
|
||||
if width > MAX_PIXELS or height > MAX_PIXELS or width * height > MAX_PIXELS:
|
||||
raise ApiError(413, "payload_too_large", "В изображении больше 40 миллионов пикселей")
|
||||
|
||||
|
||||
def inspect_image(data, declared):
|
||||
if declared == "image/png":
|
||||
size = _png_size(data)
|
||||
elif declared == "image/jpeg":
|
||||
size = _jpeg_size(data)
|
||||
else:
|
||||
size = None
|
||||
detected = None
|
||||
if data.startswith(b"\x89PNG\r\n\x1a\n"):
|
||||
detected = "image/png"
|
||||
elif data.startswith(b"\xff\xd8\xff"):
|
||||
detected = "image/jpeg"
|
||||
if size is None or detected != declared:
|
||||
raise ApiError(415, "unsupported_media_type", "Файл не является PNG или JPEG")
|
||||
_too_many_pixels(size[0], size[1])
|
||||
try:
|
||||
with Image.open(io.BytesIO(data)) as image:
|
||||
image.load()
|
||||
except DecompressionBombError as exc:
|
||||
raise ApiError(413, "payload_too_large", "В изображении больше 40 миллионов пикселей") from exc
|
||||
except (UnidentifiedImageError, OSError, SyntaxError, ValueError) as exc:
|
||||
raise ApiError(415, "unsupported_media_type", "Файл не является PNG или JPEG") from exc
|
||||
return declared
|
||||
|
||||
|
||||
def list_objects(prefix=""):
|
||||
found = []
|
||||
prefixes = (prefix,) if prefix else ("chats/", "staging/", "ready/")
|
||||
try:
|
||||
for one in prefixes:
|
||||
found.extend(_list_prefix(one))
|
||||
except S3Error as exc:
|
||||
raise DependencyDown() from exc
|
||||
return found
|
||||
|
||||
|
||||
def _list_prefix(prefix):
|
||||
found = []
|
||||
try:
|
||||
for item in client().list_objects(BUCKET, prefix=prefix, recursive=True):
|
||||
if item.is_dir:
|
||||
continue
|
||||
stat = client().stat_object(BUCKET, item.object_name)
|
||||
raw = None
|
||||
if stat.metadata:
|
||||
meta = {str(key).lower(): value for key, value in stat.metadata.items()}
|
||||
raw = meta.get("x-amz-meta-created-at") or meta.get("created-at")
|
||||
created = _parse_stamp(raw) if raw else None
|
||||
if created is None and item.last_modified is not None:
|
||||
created = item.last_modified
|
||||
found.append({"key": item.object_name, "created_at": created})
|
||||
except S3Error as exc:
|
||||
raise DependencyDown() from exc
|
||||
return found
|
||||
|
||||
|
||||
def _parse_stamp(value):
|
||||
text = value.strip().replace("Z", "+00:00")
|
||||
try:
|
||||
parsed = datetime.fromisoformat(text)
|
||||
except ValueError:
|
||||
return None
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return parsed
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
"""RabbitMQ: publisher confirm при публикации, consumer ack отдельно в worker."""
|
||||
import json
|
||||
import os
|
||||
from contextlib import contextmanager
|
||||
|
||||
import pika
|
||||
from psycopg.types.json import Json
|
||||
|
||||
from blobs import read_secret
|
||||
|
||||
PROCESS_QUEUE = "attachments.process"
|
||||
RETRY_QUEUE = "attachments.retry"
|
||||
FAILED_QUEUE = "attachments.failed"
|
||||
|
||||
|
||||
def broker_url():
|
||||
return read_secret(os.environ["BROKER_URL_FILE"])
|
||||
|
||||
|
||||
def _parameters():
|
||||
params = pika.URLParameters(broker_url())
|
||||
params.heartbeat = 60
|
||||
params.blocked_connection_timeout = 10
|
||||
params.socket_timeout = 10
|
||||
params.connection_attempts = 1
|
||||
return params
|
||||
|
||||
|
||||
def ensure_topology(channel):
|
||||
channel.queue_declare(queue=PROCESS_QUEUE, durable=True)
|
||||
channel.queue_declare(
|
||||
queue=RETRY_QUEUE,
|
||||
durable=True,
|
||||
arguments={
|
||||
"x-dead-letter-exchange": "",
|
||||
"x-dead-letter-routing-key": PROCESS_QUEUE,
|
||||
},
|
||||
)
|
||||
channel.queue_declare(queue=FAILED_QUEUE, durable=True)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def broker_channel():
|
||||
connection = pika.BlockingConnection(_parameters())
|
||||
channel = connection.channel()
|
||||
channel.confirm_delivery()
|
||||
ensure_topology(channel)
|
||||
try:
|
||||
yield channel
|
||||
finally:
|
||||
try:
|
||||
connection.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def publish_raw(channel, queue, body, expiration_ms=None):
|
||||
fields = {
|
||||
"delivery_mode": 2,
|
||||
"content_type": "application/json",
|
||||
}
|
||||
if expiration_ms is not None:
|
||||
fields["expiration"] = str(int(expiration_ms))
|
||||
channel.basic_publish(
|
||||
exchange="",
|
||||
routing_key=queue,
|
||||
body=body,
|
||||
properties=pika.BasicProperties(**fields),
|
||||
mandatory=True,
|
||||
)
|
||||
|
||||
|
||||
def publish_pending(conn, only_job_id=None):
|
||||
published = 0
|
||||
while published < 20:
|
||||
with conn.transaction():
|
||||
conn.execute("SET LOCAL statement_timeout = '20s'")
|
||||
params = []
|
||||
sql = """
|
||||
SELECT o.id, o.payload
|
||||
FROM outbox o
|
||||
JOIN attachment_jobs j ON j.id = o.job_id
|
||||
WHERE o.published_at IS NULL
|
||||
AND (j.next_attempt_at IS NULL OR j.next_attempt_at <= now())
|
||||
"""
|
||||
if only_job_id is not None:
|
||||
sql += " AND o.job_id = %s"
|
||||
params.append(only_job_id)
|
||||
sql += """
|
||||
ORDER BY o.created_at
|
||||
FOR UPDATE OF o SKIP LOCKED
|
||||
LIMIT 1
|
||||
"""
|
||||
row = conn.execute(sql, params).fetchone()
|
||||
if row is None:
|
||||
return published
|
||||
body = json.dumps(
|
||||
row["payload"],
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
with broker_channel() as channel:
|
||||
publish_raw(channel, PROCESS_QUEUE, body)
|
||||
conn.execute(
|
||||
"UPDATE outbox SET published_at = now() WHERE id = %s AND published_at IS NULL",
|
||||
(row["id"],),
|
||||
)
|
||||
published += 1
|
||||
if only_job_id is not None:
|
||||
return published
|
||||
return published
|
||||
|
||||
|
||||
def publish_delay(body, delay_ms):
|
||||
with broker_channel() as channel:
|
||||
publish_raw(channel, RETRY_QUEUE, body, expiration_ms=max(1000, int(delay_ms)))
|
||||
|
||||
|
||||
def publish_failed(body):
|
||||
with broker_channel() as channel:
|
||||
publish_raw(channel, FAILED_QUEUE, body)
|
||||
|
||||
|
||||
def queue_depth(queue):
|
||||
with broker_channel() as channel:
|
||||
declared = channel.queue_declare(queue=queue, passive=True)
|
||||
return int(declared.method.message_count)
|
||||
|
||||
|
||||
def dump_payload(payload):
|
||||
return Json(payload)
|
||||
@@ -0,0 +1,32 @@
|
||||
class ApiError(Exception):
|
||||
def __init__(self, status, code, message, headers=None):
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.headers = headers or {}
|
||||
|
||||
|
||||
class DependencyDown(Exception):
|
||||
"""Postgres или Redis не ответили. Наружу это 503 без имени хранилища."""
|
||||
|
||||
|
||||
class IdempotencyConflict(Exception):
|
||||
"""Тот же Idempotency-Key уже зафиксирован с другим телом."""
|
||||
|
||||
|
||||
class AttachmentRejected(Exception):
|
||||
"""Вложение есть, но прикрепить его к сообщению нельзя."""
|
||||
|
||||
|
||||
class AttachmentNotReady(Exception):
|
||||
"""Байты ещё нельзя отдать: queued, processing или failed."""
|
||||
|
||||
|
||||
def error_body(code, message, request_id):
|
||||
return {
|
||||
"error": {
|
||||
"code": code,
|
||||
"message": message,
|
||||
"request_id": request_id,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Точечные сбои для проверки. Без строки в fault_injections код их не включает."""
|
||||
import uuid
|
||||
|
||||
from psycopg.pq import TransactionStatus
|
||||
|
||||
|
||||
def _idle(conn):
|
||||
# Чтение выше могло открыть транзакцию. Её откат не теряет запись:
|
||||
# сама запись начинается следующим transaction() и делает COMMIT.
|
||||
if conn.info.transaction_status != TransactionStatus.IDLE:
|
||||
conn.rollback()
|
||||
|
||||
|
||||
def take_fault(conn, kind, attachment_id=None):
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
row = conn.execute(
|
||||
"""
|
||||
UPDATE fault_injections AS target
|
||||
SET armed = false
|
||||
WHERE target.id = (
|
||||
SELECT id
|
||||
FROM fault_injections
|
||||
WHERE armed
|
||||
AND kind = %s
|
||||
AND (attachment_id IS NULL OR attachment_id = %s)
|
||||
ORDER BY created_at
|
||||
FOR UPDATE SKIP LOCKED
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING id
|
||||
""",
|
||||
(kind, attachment_id),
|
||||
).fetchone()
|
||||
return row is not None
|
||||
|
||||
|
||||
def arm_fault(conn, kind, attachment_id=None):
|
||||
fault_id = uuid.uuid4()
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO fault_injections (id, kind, attachment_id, armed)
|
||||
VALUES (%s, %s, %s, true)
|
||||
""",
|
||||
(fault_id, kind, attachment_id),
|
||||
)
|
||||
return str(fault_id)
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
"""Варианты изображения. Оригинал не перекодируется."""
|
||||
import io
|
||||
|
||||
from PIL import Image, ImageOps, UnidentifiedImageError
|
||||
from PIL.Image import DecompressionBombError
|
||||
|
||||
from blobs import MAX_PIXELS, header_size
|
||||
|
||||
THUMB_BOX = 256
|
||||
CROP_SIZE = 128
|
||||
OPT_BOX = 1280
|
||||
JPEG_QUALITY = 85
|
||||
WEBP_QUALITY = 80
|
||||
WEBP_METHOD = 4
|
||||
|
||||
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
|
||||
|
||||
|
||||
class PermanentJobError(Exception):
|
||||
def __init__(self, code):
|
||||
super().__init__(code)
|
||||
self.code = code
|
||||
|
||||
|
||||
class TransientJobError(Exception):
|
||||
def __init__(self, code):
|
||||
super().__init__(code)
|
||||
self.code = code
|
||||
|
||||
|
||||
def _rgb(image):
|
||||
if image.mode == "RGBA" or image.mode == "LA" or (image.mode == "P" and "transparency" in image.info):
|
||||
rgba = image.convert("RGBA")
|
||||
background = Image.new("RGB", rgba.size, (255, 255, 255))
|
||||
background.paste(rgba, mask=rgba.getchannel("A"))
|
||||
return background
|
||||
if image.mode == "RGB":
|
||||
return image
|
||||
return image.convert("RGB")
|
||||
|
||||
|
||||
def load_rgb(data, declared):
|
||||
size = header_size(data, declared)
|
||||
detected = None
|
||||
if data.startswith(b"\x89PNG\r\n\x1a\n"):
|
||||
detected = "image/png"
|
||||
elif data.startswith(b"\xff\xd8\xff"):
|
||||
detected = "image/jpeg"
|
||||
if size is None or detected != declared:
|
||||
raise PermanentJobError("invalid_image")
|
||||
width, height = size
|
||||
if width <= 0 or height <= 0 or width > MAX_PIXELS or height > MAX_PIXELS or width * height > MAX_PIXELS:
|
||||
raise PermanentJobError("invalid_image")
|
||||
try:
|
||||
with Image.open(io.BytesIO(data)) as image:
|
||||
image.load()
|
||||
oriented = ImageOps.exif_transpose(image)
|
||||
rgb = _rgb(oriented)
|
||||
rgb.load()
|
||||
return rgb
|
||||
except DecompressionBombError as exc:
|
||||
raise PermanentJobError("invalid_image") from exc
|
||||
except (UnidentifiedImageError, OSError, SyntaxError, ValueError) as exc:
|
||||
raise PermanentJobError("invalid_image") from exc
|
||||
|
||||
|
||||
def fit_inside(image, box):
|
||||
width, height = image.size
|
||||
if width <= box and height <= box:
|
||||
return image.copy()
|
||||
if width >= height:
|
||||
new_w = box
|
||||
new_h = max(1, (height * box) // width)
|
||||
else:
|
||||
new_h = box
|
||||
new_w = max(1, (width * box) // height)
|
||||
return image.resize((new_w, new_h), Image.Resampling.LANCZOS)
|
||||
|
||||
|
||||
def center_square(image, size):
|
||||
width, height = image.size
|
||||
side = min(width, height)
|
||||
left = (width - side) // 2
|
||||
top = (height - side) // 2
|
||||
cropped = image.crop((left, top, left + side, top + side))
|
||||
if cropped.size != (size, size):
|
||||
cropped = cropped.resize((size, size), Image.Resampling.LANCZOS)
|
||||
return cropped
|
||||
|
||||
|
||||
def _jpeg(image):
|
||||
buffer = io.BytesIO()
|
||||
image.save(buffer, format="JPEG", quality=JPEG_QUALITY, optimize=True)
|
||||
return {
|
||||
"bytes": buffer.getvalue(),
|
||||
"content_type": "image/jpeg",
|
||||
"width": image.size[0],
|
||||
"height": image.size[1],
|
||||
}
|
||||
|
||||
|
||||
def _webp(image):
|
||||
buffer = io.BytesIO()
|
||||
image.save(buffer, format="WEBP", quality=WEBP_QUALITY, method=WEBP_METHOD)
|
||||
return {
|
||||
"bytes": buffer.getvalue(),
|
||||
"content_type": "image/webp",
|
||||
"width": image.size[0],
|
||||
"height": image.size[1],
|
||||
}
|
||||
|
||||
|
||||
def render(data, declared):
|
||||
rgb = load_rgb(data, declared)
|
||||
size = header_size(data, declared)
|
||||
thumb = fit_inside(rgb, THUMB_BOX)
|
||||
crop = center_square(rgb, CROP_SIZE)
|
||||
optimized = fit_inside(rgb, OPT_BOX)
|
||||
return {
|
||||
"original": {
|
||||
"bytes": data,
|
||||
"content_type": declared,
|
||||
"width": size[0],
|
||||
"height": size[1],
|
||||
},
|
||||
"thumbnail": _jpeg(thumb),
|
||||
"crop": _jpeg(crop),
|
||||
"optimized": _webp(optimized),
|
||||
}
|
||||
+831
@@ -0,0 +1,831 @@
|
||||
"""Состояния задания. Эффект один на (attachment_id, pipeline_version, вариант)."""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import threading
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from psycopg.pq import TransactionStatus
|
||||
from psycopg.types.json import Json
|
||||
|
||||
from blobs import delete_quiet, put_bytes, read_bytes, variant_object_key
|
||||
from errors import DependencyDown
|
||||
from broker import publish_delay, publish_failed, publish_raw, broker_channel, PROCESS_QUEUE
|
||||
from faults import take_fault
|
||||
from images import PermanentJobError, TransientJobError, render
|
||||
|
||||
REQUIRED = ("original", "thumbnail", "crop", "optimized")
|
||||
PUBLIC_CODES = ("invalid_image", "processing_failed", "retry_exhausted", "processing_timeout")
|
||||
|
||||
|
||||
class LostLease(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def _idle(conn):
|
||||
if conn.info.transaction_status != TransactionStatus.IDLE:
|
||||
conn.rollback()
|
||||
|
||||
|
||||
def max_attempts():
|
||||
raw = os.environ.get("JOB_MAX_ATTEMPTS", "5")
|
||||
value = int(raw)
|
||||
if not 1 <= value <= 20:
|
||||
raise RuntimeError("JOB_MAX_ATTEMPTS")
|
||||
return value
|
||||
|
||||
|
||||
def lease_seconds():
|
||||
raw = os.environ.get("JOB_TIMEOUT_SECONDS", "60")
|
||||
value = int(raw)
|
||||
if value < 5:
|
||||
raise RuntimeError("JOB_TIMEOUT_SECONDS")
|
||||
return value
|
||||
|
||||
|
||||
def retention_seconds():
|
||||
raw = os.environ.get("STAGING_RETENTION_SECONDS", "86400")
|
||||
value = int(raw)
|
||||
if value < 60:
|
||||
raise RuntimeError("STAGING_RETENTION_SECONDS")
|
||||
return value
|
||||
|
||||
|
||||
def utc_now():
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def dump_time(value):
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
text = value.astimezone(timezone.utc).isoformat(timespec="microseconds")
|
||||
if text.endswith("+00:00"):
|
||||
text = text[:-6] + "Z"
|
||||
return text
|
||||
|
||||
|
||||
def backoff_seconds(failed_attempt):
|
||||
base = min(30.0, 0.5 * (2 ** (failed_attempt - 1)))
|
||||
return base + random.uniform(0, base * 0.25)
|
||||
|
||||
|
||||
def job_message(job_id, attachment_id, pipeline_version, attempt, created_at):
|
||||
if not isinstance(created_at, str):
|
||||
created_at = dump_time(created_at)
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"job_id": str(job_id),
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": str(attachment_id),
|
||||
"pipeline_version": int(pipeline_version),
|
||||
"attempt": int(attempt),
|
||||
"created_at": created_at,
|
||||
}
|
||||
|
||||
|
||||
def _uuid(value):
|
||||
try:
|
||||
return str(uuid.UUID(str(value)))
|
||||
except (ValueError, AttributeError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _stamp(value):
|
||||
if not isinstance(value, str) or not value.endswith("Z"):
|
||||
return None
|
||||
try:
|
||||
datetime.fromisoformat(value[:-1] + "+00:00")
|
||||
except ValueError:
|
||||
return None
|
||||
return value
|
||||
|
||||
|
||||
def parse_job(body):
|
||||
try:
|
||||
payload = json.loads(body.decode("utf-8"))
|
||||
except (UnicodeError, ValueError, AttributeError):
|
||||
return None
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
allowed = {
|
||||
"schema_version",
|
||||
"job_id",
|
||||
"type",
|
||||
"attachment_id",
|
||||
"pipeline_version",
|
||||
"attempt",
|
||||
"created_at",
|
||||
"traceparent",
|
||||
"tracestate",
|
||||
}
|
||||
if set(payload) - allowed:
|
||||
return None
|
||||
required = allowed - {"traceparent", "tracestate"}
|
||||
if not required <= set(payload):
|
||||
return None
|
||||
if payload.get("schema_version") != 1 or payload.get("type") != "attachment.process.v1":
|
||||
return None
|
||||
job_id = _uuid(payload.get("job_id"))
|
||||
attachment_id = _uuid(payload.get("attachment_id"))
|
||||
created_at = _stamp(payload.get("created_at"))
|
||||
version = payload.get("pipeline_version")
|
||||
attempt = payload.get("attempt")
|
||||
if job_id is None or attachment_id is None or created_at is None:
|
||||
return None
|
||||
if type(version) is not int or type(attempt) is not int:
|
||||
return None
|
||||
if version < 1 or attempt < 1:
|
||||
return None
|
||||
if "traceparent" in payload:
|
||||
traceparent = payload["traceparent"]
|
||||
if not isinstance(traceparent, str) or len(traceparent) != 55:
|
||||
return None
|
||||
if "tracestate" in payload:
|
||||
tracestate = payload["tracestate"]
|
||||
if not isinstance(tracestate, str) or len(tracestate) > 512:
|
||||
return None
|
||||
payload["job_id"] = job_id
|
||||
payload["attachment_id"] = attachment_id
|
||||
return payload
|
||||
|
||||
|
||||
def _event(conn, job, kind, detail=None):
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO job_events (job_id, attachment_id, pipeline_version, kind, detail)
|
||||
VALUES (%s, %s, %s, %s, %s)
|
||||
""",
|
||||
(job["id"], job["attachment_id"], job["pipeline_version"], kind, detail),
|
||||
)
|
||||
|
||||
|
||||
def _message_body(job, attempt, created_at):
|
||||
payload = job_message(job["id"], job["attachment_id"], job["pipeline_version"], attempt, created_at)
|
||||
return json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8")
|
||||
|
||||
|
||||
def fail_job(conn, job, code):
|
||||
if code not in PUBLIC_CODES and code != "superseded":
|
||||
code = "processing_failed"
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'failed', error_code = %s, lease_owner = NULL,
|
||||
lease_until = NULL, next_attempt_at = NULL, updated_at = now()
|
||||
WHERE id = %s AND status IN ('queued', 'processing')
|
||||
""",
|
||||
(code, job["id"]),
|
||||
)
|
||||
if code in PUBLIC_CODES:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'failed', error_code = %s, updated_at = now()
|
||||
WHERE id = %s AND pipeline_version = %s
|
||||
AND status IN ('queued', 'processing')
|
||||
""",
|
||||
(code, job["attachment_id"], job["pipeline_version"]),
|
||||
)
|
||||
_event(conn, job, "failed", code)
|
||||
try:
|
||||
publish_failed(_message_body(job, job["attempt"], job["created_at"]))
|
||||
except Exception:
|
||||
# Реестр failed уже в базе. Повторная доставка увидит failed и подтвердит сообщение.
|
||||
return
|
||||
|
||||
|
||||
def schedule_retry(conn, job, code, owner):
|
||||
attempt = int(job["attempt"])
|
||||
if attempt >= int(job["max_attempts"]):
|
||||
fail_job(conn, job, "retry_exhausted")
|
||||
return
|
||||
delay = backoff_seconds(attempt)
|
||||
new_attempt = attempt + 1
|
||||
created = utc_now()
|
||||
payload = job_message(job["id"], job["attachment_id"], job["pipeline_version"], new_attempt, created)
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
updated = conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued', attempt = %s, lease_owner = NULL, lease_until = NULL,
|
||||
next_attempt_at = now() + make_interval(secs => %s),
|
||||
error_code = NULL, updated_at = now()
|
||||
WHERE id = %s AND status = 'processing' AND lease_owner = %s
|
||||
RETURNING id
|
||||
""",
|
||||
(new_attempt, delay, job["id"], owner),
|
||||
).fetchone()
|
||||
if updated is None:
|
||||
return
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'queued', error_code = NULL, updated_at = now()
|
||||
WHERE id = %s AND pipeline_version = %s AND status = 'processing'
|
||||
""",
|
||||
(job["attachment_id"], job["pipeline_version"]),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO outbox (id, job_id, attempt, payload, created_at)
|
||||
VALUES (%s, %s, %s, %s, %s)
|
||||
""",
|
||||
(uuid.uuid4(), job["id"], new_attempt, Json(payload), created),
|
||||
)
|
||||
_event(conn, job, "retry", code)
|
||||
|
||||
|
||||
def release_expired(conn):
|
||||
released = 0
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
rows = conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued', lease_owner = NULL, lease_until = NULL, updated_at = now()
|
||||
WHERE status = 'processing' AND lease_until < now()
|
||||
RETURNING id, attachment_id, pipeline_version, attempt, created_at
|
||||
"""
|
||||
).fetchall()
|
||||
for row in rows:
|
||||
payload = job_message(row["id"], row["attachment_id"], row["pipeline_version"], row["attempt"], utc_now())
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO outbox (id, job_id, attempt, payload, created_at)
|
||||
SELECT %s, %s, %s, %s, now()
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM outbox WHERE job_id = %s AND published_at IS NULL
|
||||
)
|
||||
""",
|
||||
(uuid.uuid4(), row["id"], row["attempt"], Json(payload), row["id"]),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'queued', updated_at = now()
|
||||
WHERE id = %s AND pipeline_version = %s AND status = 'processing'
|
||||
""",
|
||||
(row["attachment_id"], row["pipeline_version"]),
|
||||
)
|
||||
_event(conn, row, "lease_expired", None)
|
||||
released += 1
|
||||
return released
|
||||
|
||||
|
||||
def cleanup_staging(conn):
|
||||
removed = 0
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.staging_key
|
||||
FROM attachments a
|
||||
WHERE a.staging_key IS NOT NULL
|
||||
AND a.staging_deleted_at IS NULL
|
||||
AND a.status IN ('ready', 'failed')
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM attachment_jobs j
|
||||
WHERE j.attachment_id = a.id
|
||||
AND j.status IN ('queued', 'processing')
|
||||
)
|
||||
"""
|
||||
).fetchall()
|
||||
for row in rows:
|
||||
if delete_quiet(row["staging_key"]):
|
||||
with conn.transaction():
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET staging_deleted_at = now()
|
||||
WHERE id = %s AND staging_deleted_at IS NULL
|
||||
""",
|
||||
(row["id"],),
|
||||
)
|
||||
removed += 1
|
||||
return removed
|
||||
|
||||
|
||||
def _load_job(conn, job_id):
|
||||
return conn.execute(
|
||||
"""
|
||||
SELECT id, attachment_id, pipeline_version, status, attempt, max_attempts,
|
||||
lease_owner, lease_until, next_attempt_at, error_code, created_at
|
||||
FROM attachment_jobs
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job_id,),
|
||||
).fetchone()
|
||||
|
||||
|
||||
def _load_attachment(conn, attachment_id):
|
||||
return conn.execute(
|
||||
"""
|
||||
SELECT id, content_type, sha256, object_key, staging_key, staging_deleted_at,
|
||||
status, pipeline_version
|
||||
FROM attachments
|
||||
WHERE id = %s
|
||||
""",
|
||||
(attachment_id,),
|
||||
).fetchone()
|
||||
|
||||
|
||||
def _acquire(conn, job, owner, seconds):
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
locked = conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'processing', lease_owner = %s,
|
||||
lease_until = now() + make_interval(secs => %s),
|
||||
updated_at = now()
|
||||
WHERE id = %s
|
||||
AND status IN ('queued', 'processing')
|
||||
AND attempt = %s
|
||||
AND pipeline_version = %s
|
||||
AND (lease_owner IS NULL OR lease_owner = %s OR lease_until < now())
|
||||
RETURNING id
|
||||
""",
|
||||
(owner, seconds, job["id"], job["attempt"], job["pipeline_version"], owner),
|
||||
).fetchone()
|
||||
if locked is None:
|
||||
return False
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'processing', updated_at = now()
|
||||
WHERE id = %s AND pipeline_version = %s AND status IN ('queued', 'processing')
|
||||
""",
|
||||
(job["attachment_id"], job["pipeline_version"]),
|
||||
)
|
||||
_event(conn, job, "leased", None)
|
||||
return True
|
||||
|
||||
|
||||
def _variants(conn, attachment_id, version):
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT name, object_key, sha256
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = %s AND pipeline_version = %s
|
||||
""",
|
||||
(attachment_id, version),
|
||||
).fetchall()
|
||||
return {row["name"]: row for row in rows}
|
||||
|
||||
|
||||
def _source(conn, attachment):
|
||||
if attachment["staging_key"] and attachment["staging_deleted_at"] is None:
|
||||
data = read_bytes(attachment["staging_key"])
|
||||
if data:
|
||||
return data
|
||||
data = read_bytes(attachment["object_key"])
|
||||
if data:
|
||||
return data
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT object_key
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = %s AND name = 'original'
|
||||
ORDER BY pipeline_version DESC
|
||||
LIMIT 1
|
||||
""",
|
||||
(attachment["id"],),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise TransientJobError("processing_failed")
|
||||
data = read_bytes(row["object_key"])
|
||||
if not data:
|
||||
raise TransientJobError("processing_failed")
|
||||
return data
|
||||
|
||||
|
||||
def _render_limited(data, declared, seconds):
|
||||
budget = max(5, seconds - 5)
|
||||
box = []
|
||||
errors = []
|
||||
|
||||
def run():
|
||||
try:
|
||||
box.append(render(data, declared))
|
||||
except Exception as exc:
|
||||
errors.append(exc)
|
||||
|
||||
thread = threading.Thread(target=run, daemon=True)
|
||||
thread.start()
|
||||
thread.join(budget)
|
||||
if thread.is_alive():
|
||||
raise TransientJobError("processing_timeout")
|
||||
if errors:
|
||||
exc = errors[0]
|
||||
if isinstance(exc, (PermanentJobError, TransientJobError)):
|
||||
raise exc
|
||||
raise PermanentJobError("invalid_image")
|
||||
return box[0]
|
||||
|
||||
|
||||
def _beat(pool, job_id, owner, seconds, stop):
|
||||
pause = max(2, seconds // 3)
|
||||
while not stop.wait(pause):
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
with conn.transaction():
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET lease_until = now() + make_interval(secs => %s), updated_at = now()
|
||||
WHERE id = %s AND lease_owner = %s AND status = 'processing'
|
||||
""",
|
||||
(seconds, job_id, owner),
|
||||
)
|
||||
except Exception:
|
||||
return
|
||||
|
||||
|
||||
def _supersede(conn, job):
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'superseded', error_code = 'superseded', lease_owner = NULL,
|
||||
lease_until = NULL, next_attempt_at = NULL, updated_at = now()
|
||||
WHERE id = %s AND status IN ('queued', 'processing')
|
||||
""",
|
||||
(job["id"],),
|
||||
)
|
||||
_event(conn, job, "superseded", None)
|
||||
|
||||
|
||||
def _finalize(conn, job, owner, rendered):
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
current = conn.execute(
|
||||
"""
|
||||
SELECT lease_owner, status, pipeline_version
|
||||
FROM attachment_jobs
|
||||
WHERE id = %s
|
||||
FOR UPDATE
|
||||
""",
|
||||
(job["id"],),
|
||||
).fetchone()
|
||||
if current is None or current["lease_owner"] != owner or current["status"] != "processing":
|
||||
raise LostLease()
|
||||
attachment = conn.execute(
|
||||
"""
|
||||
SELECT pipeline_version, status
|
||||
FROM attachments
|
||||
WHERE id = %s
|
||||
FOR UPDATE
|
||||
""",
|
||||
(job["attachment_id"],),
|
||||
).fetchone()
|
||||
if attachment is None or int(attachment["pipeline_version"]) > int(job["pipeline_version"]):
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'superseded', error_code = 'superseded', lease_owner = NULL,
|
||||
lease_until = NULL, updated_at = now()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job["id"],),
|
||||
)
|
||||
_event(conn, job, "superseded", None)
|
||||
return "superseded"
|
||||
now = utc_now()
|
||||
for name in REQUIRED:
|
||||
item = rendered[name]
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO attachment_variants (
|
||||
attachment_id, pipeline_version, name, content_type, size_bytes,
|
||||
width, height, sha256, object_key, created_at
|
||||
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON CONFLICT (attachment_id, pipeline_version, name) DO NOTHING
|
||||
""",
|
||||
(
|
||||
job["attachment_id"],
|
||||
job["pipeline_version"],
|
||||
name,
|
||||
item["content_type"],
|
||||
len(item["bytes"]),
|
||||
item["width"],
|
||||
item["height"],
|
||||
item["sha256"],
|
||||
item["object_key"],
|
||||
now,
|
||||
),
|
||||
)
|
||||
stored = conn.execute(
|
||||
"""
|
||||
SELECT count(*) AS n
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = %s AND pipeline_version = %s
|
||||
AND name IN ('original', 'thumbnail', 'crop', 'optimized')
|
||||
""",
|
||||
(job["attachment_id"], job["pipeline_version"]),
|
||||
).fetchone()
|
||||
if int(stored["n"]) < len(REQUIRED):
|
||||
raise TransientJobError("processing_failed")
|
||||
original = rendered["original"]
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'ready', error_code = NULL, pipeline_version = %s,
|
||||
object_key = %s, updated_at = %s
|
||||
WHERE id = %s AND pipeline_version <= %s
|
||||
""",
|
||||
(
|
||||
job["pipeline_version"],
|
||||
original["object_key"],
|
||||
now,
|
||||
job["attachment_id"],
|
||||
job["pipeline_version"],
|
||||
),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'ready', error_code = NULL, lease_owner = NULL,
|
||||
lease_until = NULL, next_attempt_at = NULL, updated_at = %s
|
||||
WHERE id = %s AND lease_owner = %s
|
||||
""",
|
||||
(now, job["id"], owner),
|
||||
)
|
||||
_event(conn, job, "ready", None)
|
||||
return "ready"
|
||||
|
||||
|
||||
def _drop_staging(conn, attachment):
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
active = conn.execute(
|
||||
"""
|
||||
SELECT count(*) AS n
|
||||
FROM attachment_jobs
|
||||
WHERE attachment_id = %s AND status IN ('queued', 'processing')
|
||||
""",
|
||||
(attachment["id"],),
|
||||
).fetchone()
|
||||
if int(active["n"]) > 0 or not attachment["staging_key"]:
|
||||
return
|
||||
if delete_quiet(attachment["staging_key"]):
|
||||
with conn.transaction():
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET staging_deleted_at = now()
|
||||
WHERE id = %s AND staging_deleted_at IS NULL
|
||||
""",
|
||||
(attachment["id"],),
|
||||
)
|
||||
|
||||
|
||||
def process_delivery(pool, body, owner):
|
||||
payload = parse_job(body)
|
||||
if payload is None:
|
||||
return "ack"
|
||||
with pool.connection() as conn:
|
||||
job = _load_job(conn, payload["job_id"])
|
||||
if job is None:
|
||||
return "ack"
|
||||
if job["status"] in ("ready", "failed", "superseded"):
|
||||
return "ack"
|
||||
if int(payload["attempt"]) != int(job["attempt"]) or int(payload["pipeline_version"]) != int(job["pipeline_version"]):
|
||||
return "ack"
|
||||
if job["next_attempt_at"] is not None and job["next_attempt_at"] > utc_now():
|
||||
delay = (job["next_attempt_at"] - utc_now()).total_seconds()
|
||||
publish_delay(body, int(delay * 1000))
|
||||
return "ack"
|
||||
with pool.connection() as conn:
|
||||
attachment = _load_attachment(conn, job["attachment_id"])
|
||||
if attachment is None:
|
||||
fail_job(conn, job, "invalid_image")
|
||||
return "ack"
|
||||
if int(attachment["pipeline_version"]) > int(job["pipeline_version"]):
|
||||
_supersede(conn, job)
|
||||
return "ack"
|
||||
if not _acquire(conn, job, owner, lease_seconds()):
|
||||
publish_delay(body, 2000)
|
||||
return "ack"
|
||||
stop = threading.Event()
|
||||
beater = threading.Thread(
|
||||
target=_beat,
|
||||
args=(pool, job["id"], owner, lease_seconds(), stop),
|
||||
daemon=True,
|
||||
)
|
||||
beater.start()
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
if take_fault(conn, "crash_before_effect", job["attachment_id"]):
|
||||
os_exit()
|
||||
if take_fault(conn, "transient_once", job["attachment_id"]):
|
||||
raise TransientJobError("processing_failed")
|
||||
present = _variants(conn, job["attachment_id"], job["pipeline_version"])
|
||||
source = None if set(REQUIRED) <= set(present) else _source(conn, attachment)
|
||||
if source is None:
|
||||
rendered = _rendered_from_rows(conn, job, present)
|
||||
if source is not None:
|
||||
if hashlib.sha256(source).hexdigest() != attachment["sha256"]:
|
||||
raise PermanentJobError("invalid_image")
|
||||
rendered = _render_limited(source, attachment["content_type"], lease_seconds())
|
||||
for name in REQUIRED:
|
||||
item = rendered[name]
|
||||
item["sha256"] = hashlib.sha256(item["bytes"]).hexdigest()
|
||||
item["object_key"] = variant_object_key(job["attachment_id"], job["pipeline_version"], name)
|
||||
if name in present:
|
||||
continue
|
||||
put_bytes(item["object_key"], item["bytes"], item["content_type"], utc_now())
|
||||
with pool.connection() as conn:
|
||||
if take_fault(conn, "crash_after_objects", job["attachment_id"]):
|
||||
os_exit()
|
||||
with pool.connection() as conn:
|
||||
_finalize(conn, job, owner, rendered)
|
||||
with pool.connection() as conn:
|
||||
if take_fault(conn, "crash_after_commit", job["attachment_id"]):
|
||||
os_exit()
|
||||
with pool.connection() as conn:
|
||||
_drop_staging(conn, attachment)
|
||||
return "ack"
|
||||
except PermanentJobError as exc:
|
||||
with pool.connection() as conn:
|
||||
fail_job(conn, job, exc.code)
|
||||
return "ack"
|
||||
except LostLease:
|
||||
return "ack"
|
||||
except (TransientJobError, DependencyDown) as exc:
|
||||
code = exc.code if isinstance(exc, TransientJobError) else "processing_failed"
|
||||
with pool.connection() as conn:
|
||||
schedule_retry(conn, job, code, owner)
|
||||
return "ack"
|
||||
finally:
|
||||
stop.set()
|
||||
|
||||
|
||||
def os_exit():
|
||||
os._exit(75)
|
||||
|
||||
|
||||
def _rendered_from_rows(conn, job, present):
|
||||
# Повтор после уже записанного эффекта не перекодирует файлы.
|
||||
rendered = {}
|
||||
for name in REQUIRED:
|
||||
row = present[name]
|
||||
data = read_bytes(row["object_key"])
|
||||
if data is None:
|
||||
raise TransientJobError("processing_failed")
|
||||
meta = conn.execute(
|
||||
"""
|
||||
SELECT content_type, width, height, sha256, object_key
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = %s AND pipeline_version = %s AND name = %s
|
||||
""",
|
||||
(job["attachment_id"], job["pipeline_version"], name),
|
||||
).fetchone()
|
||||
rendered[name] = {
|
||||
"bytes": data,
|
||||
"content_type": meta["content_type"],
|
||||
"width": int(meta["width"]),
|
||||
"height": int(meta["height"]),
|
||||
"sha256": meta["sha256"],
|
||||
"object_key": meta["object_key"],
|
||||
}
|
||||
return rendered
|
||||
|
||||
|
||||
def bump_pipeline(conn, attachment_id):
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
row = conn.execute(
|
||||
"SELECT id, pipeline_version FROM attachments WHERE id = %s FOR UPDATE",
|
||||
(attachment_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
version = int(row["pipeline_version"]) + 1
|
||||
job_id = uuid.uuid4()
|
||||
created = utc_now()
|
||||
final_key = variant_object_key(attachment_id, version, "original")
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET pipeline_version = %s, status = 'queued', error_code = NULL,
|
||||
object_key = %s, updated_at = %s
|
||||
WHERE id = %s
|
||||
""",
|
||||
(version, final_key, created, attachment_id),
|
||||
)
|
||||
attempts = max_attempts()
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO attachment_jobs (
|
||||
id, attachment_id, pipeline_version, status, attempt, max_attempts, created_at, updated_at
|
||||
) VALUES (%s, %s, %s, 'queued', 1, %s, %s, %s)
|
||||
""",
|
||||
(job_id, attachment_id, version, attempts, created, created),
|
||||
)
|
||||
payload = job_message(job_id, attachment_id, version, 1, created)
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO outbox (id, job_id, attempt, payload, created_at)
|
||||
VALUES (%s, %s, 1, %s, %s)
|
||||
""",
|
||||
(uuid.uuid4(), job_id, Json(payload), created),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO job_events (job_id, attachment_id, pipeline_version, kind, detail)
|
||||
VALUES (%s, %s, %s, 'version', %s)
|
||||
""",
|
||||
(job_id, attachment_id, version, str(version)),
|
||||
)
|
||||
return {"job_id": str(job_id), "pipeline_version": version}
|
||||
|
||||
|
||||
def replay_job(conn, job_id):
|
||||
_idle(conn)
|
||||
with conn.transaction():
|
||||
job = conn.execute(
|
||||
"""
|
||||
SELECT id, attachment_id, pipeline_version, status, attempt, max_attempts, created_at
|
||||
FROM attachment_jobs
|
||||
WHERE id = %s
|
||||
FOR UPDATE
|
||||
""",
|
||||
(job_id,),
|
||||
).fetchone()
|
||||
if job is None:
|
||||
return {"result": "missing"}
|
||||
if job["status"] == "ready":
|
||||
return {"result": "already_ready"}
|
||||
names = conn.execute(
|
||||
"""
|
||||
SELECT name FROM attachment_variants
|
||||
WHERE attachment_id = %s AND pipeline_version = %s
|
||||
""",
|
||||
(job["attachment_id"], job["pipeline_version"]),
|
||||
).fetchall()
|
||||
have = {row["name"] for row in names}
|
||||
if set(REQUIRED) <= have:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'ready', error_code = NULL, lease_owner = NULL,
|
||||
lease_until = NULL, next_attempt_at = NULL, updated_at = now()
|
||||
WHERE id = %s AND status <> 'ready'
|
||||
""",
|
||||
(job["id"],),
|
||||
)
|
||||
return {"result": "already_done"}
|
||||
if job["status"] != "failed":
|
||||
return {"result": "not_failed"}
|
||||
created = utc_now()
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued', attempt = 1, error_code = NULL, lease_owner = NULL,
|
||||
lease_until = NULL, next_attempt_at = NULL, updated_at = %s
|
||||
WHERE id = %s
|
||||
""",
|
||||
(created, job["id"]),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'queued', error_code = NULL, updated_at = %s
|
||||
WHERE id = %s AND pipeline_version = %s AND status = 'failed'
|
||||
""",
|
||||
(created, job["attachment_id"], job["pipeline_version"]),
|
||||
)
|
||||
payload = job_message(job["id"], job["attachment_id"], job["pipeline_version"], 1, created)
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO outbox (id, job_id, attempt, payload, created_at)
|
||||
VALUES (%s, %s, 1, %s, %s)
|
||||
""",
|
||||
(uuid.uuid4(), job["id"], Json(payload), created),
|
||||
)
|
||||
_event(conn, job, "replay", None)
|
||||
return {"result": "queued"}
|
||||
|
||||
|
||||
def republish(conn, job_id, attempt=None):
|
||||
job = conn.execute(
|
||||
"""
|
||||
SELECT id, attachment_id, pipeline_version, attempt, created_at
|
||||
FROM attachment_jobs
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job_id,),
|
||||
).fetchone()
|
||||
if job is None:
|
||||
return False
|
||||
chosen = int(job["attempt"] if attempt is None else attempt)
|
||||
body = _message_body(job, chosen, utc_now())
|
||||
with broker_channel() as channel:
|
||||
publish_raw(channel, PROCESS_QUEUE, body)
|
||||
return True
|
||||
+475
@@ -0,0 +1,475 @@
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
import anyio
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse, Response, StreamingResponse
|
||||
from pydantic import ValidationError
|
||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||
|
||||
from blobs import MAX_BYTES, declared_type, inspect_image, open_object
|
||||
from errors import ApiError, AttachmentNotReady, AttachmentRejected, DependencyDown, IdempotencyConflict, error_body
|
||||
from schemas import CreateChat, CreateMessage, CreateUser
|
||||
from store import Store, instance_id
|
||||
|
||||
LOGGER = logging.getLogger("messenger")
|
||||
store = Store()
|
||||
INSTANCE_ID = instance_id()
|
||||
|
||||
BASIC_CHALLENGE = 'Basic realm="messenger", charset="UTF-8"'
|
||||
BEARER_CHALLENGE = 'Bearer realm="messenger"'
|
||||
IDEMPOTENCY_RE = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
|
||||
MAX_JSON_BYTES = 65536
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(_app):
|
||||
await anyio.to_thread.run_sync(store.startup)
|
||||
yield
|
||||
store.shutting_down = True
|
||||
await anyio.to_thread.run_sync(store.shutdown)
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="mtusi-messenger",
|
||||
version="lab5",
|
||||
lifespan=lifespan,
|
||||
redirect_slashes=False,
|
||||
docs_url="/docs",
|
||||
redoc_url=None,
|
||||
openapi_url="/openapi.json",
|
||||
)
|
||||
|
||||
|
||||
def request_id_of(request: Request):
|
||||
rid = getattr(request.state, "request_id", None)
|
||||
if rid:
|
||||
return rid
|
||||
return str(uuid.uuid4())
|
||||
|
||||
|
||||
def stamp(request: Request, response):
|
||||
response.headers["X-Request-Id"] = request_id_of(request)
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
return response
|
||||
|
||||
|
||||
def json_ok(request: Request, payload, status_code=200):
|
||||
return stamp(request, JSONResponse(payload, status_code=status_code))
|
||||
|
||||
|
||||
def json_error(request: Request, status, code, message, extra_headers=None):
|
||||
headers = {"X-Request-Id": request_id_of(request), "X-Instance-Id": INSTANCE_ID}
|
||||
if extra_headers:
|
||||
headers.update(extra_headers)
|
||||
return JSONResponse(error_body(code, message, request_id_of(request)), status_code=status, headers=headers)
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def wrap_request(request: Request, call_next):
|
||||
incoming = request.headers.get("X-Request-Id", "")
|
||||
if incoming and len(incoming) <= 200 and incoming.isprintable() and incoming.strip():
|
||||
request.state.request_id = incoming.strip()
|
||||
else:
|
||||
request.state.request_id = str(uuid.uuid4())
|
||||
if store.shutting_down and request.url.path != "/health/live":
|
||||
return json_error(request, 503, "unavailable", "Сервис временно недоступен")
|
||||
response = await call_next(request)
|
||||
return stamp(request, response)
|
||||
|
||||
|
||||
@app.exception_handler(ApiError)
|
||||
async def api_error_handler(request: Request, exc: ApiError):
|
||||
return json_error(request, exc.status, exc.code, exc.message, exc.headers)
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def request_validation_handler(request: Request, _exc: RequestValidationError):
|
||||
return json_error(request, 400, "invalid_request", "Некорректный запрос")
|
||||
|
||||
|
||||
@app.exception_handler(ValidationError)
|
||||
async def pydantic_handler(request: Request, _exc: ValidationError):
|
||||
return json_error(request, 400, "invalid_request", "Некорректный запрос")
|
||||
|
||||
|
||||
@app.exception_handler(StarletteHTTPException)
|
||||
async def http_exception_handler(request: Request, exc: StarletteHTTPException):
|
||||
if exc.status_code == 404:
|
||||
return json_error(request, 404, "not_found", "Объект не найден")
|
||||
return json_error(request, exc.status_code if exc.status_code < 500 else 500, "invalid_request", "Некорректный запрос")
|
||||
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def unexpected_handler(request: Request, exc: Exception):
|
||||
if isinstance(exc, ApiError):
|
||||
return json_error(request, exc.status, exc.code, exc.message, exc.headers)
|
||||
LOGGER.error("unhandled %s", type(exc).__name__)
|
||||
return json_error(request, 500, "internal_error", "Внутренняя ошибка")
|
||||
|
||||
|
||||
def unavailable():
|
||||
raise ApiError(503, "unavailable", "Сервис временно недоступен")
|
||||
|
||||
|
||||
def bearer_unauthorized():
|
||||
raise ApiError(401, "unauthorized", "Требуется авторизация", {"WWW-Authenticate": BEARER_CHALLENGE})
|
||||
|
||||
|
||||
def basic_unauthorized():
|
||||
raise ApiError(401, "unauthorized", "Неверные учётные данные", {"WWW-Authenticate": BASIC_CHALLENGE})
|
||||
|
||||
|
||||
def invalid():
|
||||
raise ApiError(400, "invalid_request", "Некорректный запрос")
|
||||
|
||||
|
||||
async def call_store(func, *args):
|
||||
try:
|
||||
return await anyio.to_thread.run_sync(func, *args)
|
||||
except DependencyDown:
|
||||
unavailable()
|
||||
except IdempotencyConflict:
|
||||
raise ApiError(409, "conflict", "Ключ уже использован с другим телом")
|
||||
except AttachmentRejected:
|
||||
raise ApiError(409, "conflict", "Вложение нельзя прикрепить")
|
||||
except AttachmentNotReady:
|
||||
raise ApiError(409, "conflict", "Вложение недоступно")
|
||||
|
||||
|
||||
def parse_basic(request: Request):
|
||||
header = request.headers.get("Authorization", "")
|
||||
if not header.lower().startswith("basic "):
|
||||
basic_unauthorized()
|
||||
raw = header.split(" ", 1)[1].strip()
|
||||
try:
|
||||
import base64
|
||||
|
||||
decoded = base64.b64decode(raw, validate=True).decode("utf-8")
|
||||
except (ValueError, UnicodeError):
|
||||
basic_unauthorized()
|
||||
if ":" not in decoded:
|
||||
basic_unauthorized()
|
||||
username, password = decoded.split(":", 1)
|
||||
return username, password
|
||||
|
||||
|
||||
def parse_bearer(request: Request):
|
||||
header = request.headers.get("Authorization", "")
|
||||
# Basic на бизнес-методе не открывает сессию. X-User-Id не смотрим.
|
||||
if not header.lower().startswith("bearer "):
|
||||
bearer_unauthorized()
|
||||
token = header.split(" ", 1)[1].strip()
|
||||
if len(token) < 32 or len(token) > 512 or any(ch.isspace() for ch in token):
|
||||
bearer_unauthorized()
|
||||
return token
|
||||
|
||||
|
||||
async def current_user(request: Request):
|
||||
token = parse_bearer(request)
|
||||
user = await call_store(store.user_for_token, token)
|
||||
if user is None:
|
||||
bearer_unauthorized()
|
||||
request.state.user_id = user["id"]
|
||||
request.state.token = token
|
||||
return user
|
||||
|
||||
|
||||
def parse_chat_id(chat_id):
|
||||
try:
|
||||
return str(uuid.UUID(chat_id))
|
||||
except ValueError:
|
||||
invalid()
|
||||
|
||||
|
||||
def parse_limit(raw):
|
||||
if raw is None:
|
||||
return 50
|
||||
if not isinstance(raw, str) or len(raw) > 3 or not raw.isdigit():
|
||||
invalid()
|
||||
value = int(raw)
|
||||
if value < 1 or value > 100:
|
||||
invalid()
|
||||
return value
|
||||
|
||||
|
||||
async def read_json(request: Request):
|
||||
ctype = request.headers.get("content-type", "")
|
||||
if "application/json" not in ctype.lower():
|
||||
raise ApiError(415, "unsupported_media_type", "Нужен application/json")
|
||||
length = request.headers.get("content-length")
|
||||
if length is not None and (not length.isdigit() or int(length) > MAX_JSON_BYTES):
|
||||
raise ApiError(413, "payload_too_large", "Слишком большое тело")
|
||||
raw = await request.body()
|
||||
if len(raw) > MAX_JSON_BYTES:
|
||||
raise ApiError(413, "payload_too_large", "Слишком большое тело")
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
except (ValueError, UnicodeError):
|
||||
invalid()
|
||||
if not isinstance(payload, dict):
|
||||
invalid()
|
||||
return payload
|
||||
|
||||
|
||||
def parse_idempotency_key(request: Request):
|
||||
key = request.headers.get("Idempotency-Key")
|
||||
if key is None or not IDEMPOTENCY_RE.fullmatch(key):
|
||||
invalid()
|
||||
return key
|
||||
|
||||
|
||||
@app.get("/health/live")
|
||||
async def liveness(request: Request):
|
||||
return json_ok(request, {"status": "ok"})
|
||||
|
||||
|
||||
@app.get("/health/ready")
|
||||
async def readiness(request: Request):
|
||||
if store.shutting_down:
|
||||
raise ApiError(503, "unavailable", "Сервис временно недоступен")
|
||||
ok = await call_store(store.ready)
|
||||
if not ok:
|
||||
unavailable()
|
||||
return json_ok(request, {"status": "ok"})
|
||||
|
||||
|
||||
@app.post("/api/v1/users", status_code=201)
|
||||
async def register_user(request: Request, body: CreateUser):
|
||||
user = await call_store(store.create_user, body.username, body.display_name, body.password)
|
||||
if user is None:
|
||||
raise ApiError(409, "conflict", "Имя пользователя занято")
|
||||
return json_ok(request, user, 201)
|
||||
|
||||
|
||||
@app.post("/api/v1/auth/sessions", status_code=201)
|
||||
async def create_session(request: Request):
|
||||
username, password = parse_basic(request)
|
||||
session = await call_store(store.login, username, password)
|
||||
if session is None:
|
||||
basic_unauthorized()
|
||||
request.state.user_id = session["user"]["id"]
|
||||
return json_ok(request, session, 201)
|
||||
|
||||
|
||||
@app.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||
async def revoke_session(request: Request):
|
||||
user = await current_user(request)
|
||||
request.state.user_id = user["id"]
|
||||
removed = await call_store(store.revoke, request.state.token)
|
||||
if not removed:
|
||||
bearer_unauthorized()
|
||||
return stamp(request, Response(status_code=204))
|
||||
|
||||
|
||||
@app.get("/api/v1/users/me")
|
||||
async def me(request: Request):
|
||||
user = await current_user(request)
|
||||
return json_ok(request, user)
|
||||
|
||||
|
||||
@app.post("/api/v1/chats", status_code=201)
|
||||
async def create_chat(request: Request):
|
||||
user = await current_user(request)
|
||||
payload = await read_json(request)
|
||||
try:
|
||||
body = CreateChat.model_validate(payload)
|
||||
except ValidationError:
|
||||
invalid()
|
||||
try:
|
||||
chat = await call_store(store.create_chat, body.title, body.member_ids, user["id"])
|
||||
except ValueError:
|
||||
invalid()
|
||||
return json_ok(request, chat, 201)
|
||||
|
||||
|
||||
@app.get("/api/v1/chats")
|
||||
async def list_chats(request: Request):
|
||||
user = await current_user(request)
|
||||
limit = parse_limit(request.query_params.get("limit"))
|
||||
cursor = request.query_params.get("cursor")
|
||||
if cursor is not None and (not cursor or len(cursor) > 512):
|
||||
invalid()
|
||||
try:
|
||||
items, next_cursor = await call_store(store.list_chats, user["id"], limit, cursor)
|
||||
except ValueError:
|
||||
invalid()
|
||||
return json_ok(request, {"items": items, "next_cursor": next_cursor})
|
||||
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}")
|
||||
async def get_chat(request: Request, chat_id: str):
|
||||
user = await current_user(request)
|
||||
chat = await call_store(store.get_chat, parse_chat_id(chat_id), user["id"])
|
||||
if chat is None:
|
||||
raise ApiError(404, "not_found", "Объект не найден")
|
||||
return json_ok(request, chat)
|
||||
|
||||
|
||||
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201)
|
||||
async def create_message(request: Request, chat_id: str):
|
||||
user = await current_user(request)
|
||||
canonical = parse_chat_id(chat_id)
|
||||
payload = await read_json(request)
|
||||
key = parse_idempotency_key(request)
|
||||
try:
|
||||
body = CreateMessage.model_validate(payload)
|
||||
except ValidationError:
|
||||
invalid()
|
||||
try:
|
||||
message = await call_store(
|
||||
store.add_message, canonical, user["id"], body.text, key, body.attachment_ids
|
||||
)
|
||||
except ValueError:
|
||||
invalid()
|
||||
if message is None:
|
||||
raise ApiError(404, "not_found", "Объект не найден")
|
||||
return json_ok(request, message, 201)
|
||||
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}/messages")
|
||||
async def list_messages(request: Request, chat_id: str):
|
||||
user = await current_user(request)
|
||||
canonical = parse_chat_id(chat_id)
|
||||
limit = parse_limit(request.query_params.get("limit"))
|
||||
cursor = request.query_params.get("cursor")
|
||||
if cursor is not None and (not cursor or len(cursor) > 512):
|
||||
invalid()
|
||||
try:
|
||||
result = await call_store(store.list_messages, canonical, user["id"], limit, cursor)
|
||||
except ValueError:
|
||||
invalid()
|
||||
if result is None:
|
||||
raise ApiError(404, "not_found", "Объект не найден")
|
||||
items, next_cursor = result
|
||||
return json_ok(request, {"items": items, "next_cursor": next_cursor})
|
||||
|
||||
|
||||
def check_filename(value):
|
||||
if not isinstance(value, str) or not value or len(value) > 255:
|
||||
invalid()
|
||||
if "/" in value or "\\" in value or "\x00" in value:
|
||||
invalid()
|
||||
if any(ord(char) < 32 or ord(char) == 127 for char in value):
|
||||
invalid()
|
||||
return value
|
||||
|
||||
|
||||
def parse_attachment_id(value):
|
||||
try:
|
||||
return str(uuid.UUID(value))
|
||||
except (TypeError, ValueError):
|
||||
invalid()
|
||||
|
||||
|
||||
async def read_upload(request: Request):
|
||||
header = request.headers.get("content-length")
|
||||
if header is not None:
|
||||
try:
|
||||
announced = int(header)
|
||||
except ValueError:
|
||||
invalid()
|
||||
if announced > MAX_BYTES:
|
||||
raise ApiError(413, "payload_too_large", "Файл больше 10 МиБ")
|
||||
if announced < 1:
|
||||
raise ApiError(415, "unsupported_media_type", "Нужен PNG или JPEG")
|
||||
chunks = []
|
||||
total = 0
|
||||
async for chunk in request.stream():
|
||||
total += len(chunk)
|
||||
if total > MAX_BYTES:
|
||||
raise ApiError(413, "payload_too_large", "Файл больше 10 МиБ")
|
||||
chunks.append(chunk)
|
||||
if total < 1:
|
||||
raise ApiError(415, "unsupported_media_type", "Нужен PNG или JPEG")
|
||||
return b"".join(chunks)
|
||||
|
||||
|
||||
@app.post("/api/v1/chats/{chat_id}/attachments", status_code=202)
|
||||
async def upload_attachment(request: Request, chat_id: str):
|
||||
user = await current_user(request)
|
||||
canonical = parse_chat_id(chat_id)
|
||||
member = await call_store(store.is_member, canonical, user["id"])
|
||||
if not member:
|
||||
raise ApiError(404, "not_found", "Объект не найден")
|
||||
filename = check_filename(request.query_params.get("filename"))
|
||||
media = declared_type(request.headers.get("content-type"))
|
||||
if media is None:
|
||||
raise ApiError(415, "unsupported_media_type", "Нужен PNG или JPEG")
|
||||
data = await read_upload(request)
|
||||
content_type = await anyio.to_thread.run_sync(inspect_image, data, media)
|
||||
saved = await call_store(
|
||||
store.save_attachment, canonical, user["id"], filename, content_type, data
|
||||
)
|
||||
if saved is None:
|
||||
raise ApiError(404, "not_found", "Объект не найден")
|
||||
return json_ok(request, saved, 202)
|
||||
|
||||
|
||||
@app.get("/api/v1/attachments/{attachment_id}")
|
||||
async def get_attachment(request: Request, attachment_id: str):
|
||||
user = await current_user(request)
|
||||
canonical = parse_attachment_id(attachment_id)
|
||||
found = await call_store(store.get_attachment, canonical, user["id"])
|
||||
if found is None:
|
||||
raise ApiError(404, "not_found", "Объект не найден")
|
||||
return json_ok(request, found)
|
||||
|
||||
|
||||
@app.get("/api/v1/attachments/{attachment_id}/content")
|
||||
async def download_attachment(request: Request, attachment_id: str):
|
||||
user = await current_user(request)
|
||||
canonical = parse_attachment_id(attachment_id)
|
||||
variant = request.query_params.get("variant")
|
||||
if variant is None:
|
||||
variant = "original"
|
||||
if variant not in ("original", "thumbnail", "crop", "optimized"):
|
||||
invalid()
|
||||
opened = await call_store(store.open_attachment, canonical, user["id"], variant)
|
||||
if opened is None:
|
||||
raise ApiError(404, "not_found", "Объект не найден")
|
||||
|
||||
def chunks():
|
||||
body = open_object(opened["object_key"])
|
||||
if body is None:
|
||||
return
|
||||
try:
|
||||
for piece in body.stream(64 * 1024):
|
||||
yield piece
|
||||
finally:
|
||||
body.close()
|
||||
body.release_conn()
|
||||
|
||||
probe = await anyio.to_thread.run_sync(open_object, opened["object_key"])
|
||||
if probe is None:
|
||||
raise ApiError(404, "not_found", "Объект не найден")
|
||||
probe.close()
|
||||
probe.release_conn()
|
||||
response = StreamingResponse(chunks(), media_type=opened["content_type"])
|
||||
response.headers["Content-Length"] = str(opened["size_bytes"])
|
||||
return stamp(request, response)
|
||||
|
||||
|
||||
def setup_logging():
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
setup_logging()
|
||||
host = os.environ.get("APP_HOST", "0.0.0.0")
|
||||
port = int(os.environ.get("APP_PORT", "8080"))
|
||||
uvicorn.run(
|
||||
app,
|
||||
host=host,
|
||||
port=port,
|
||||
timeout_graceful_shutdown=5,
|
||||
timeout_keep_alive=2,
|
||||
access_log=False,
|
||||
)
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
"""Однократные миграции. Запускается отдельным контейнером, не каждой репликой API."""
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
LOGGER = logging.getLogger("migrate")
|
||||
MIGRATION_LOCK = 421003
|
||||
CONNECT_TIMEOUT = 3
|
||||
|
||||
|
||||
def read_secret_file(path):
|
||||
with open(path, encoding="utf-8") as handle:
|
||||
return handle.read().strip()
|
||||
|
||||
|
||||
def database_url():
|
||||
path = os.environ.get("DATABASE_URL_FILE")
|
||||
if path:
|
||||
return read_secret_file(path)
|
||||
url = os.environ.get("DATABASE_URL", "").strip()
|
||||
if not url:
|
||||
raise RuntimeError("DATABASE_URL_FILE")
|
||||
return url
|
||||
|
||||
|
||||
def split_sql(script):
|
||||
parts = []
|
||||
buf = []
|
||||
for line in script.splitlines():
|
||||
if line.strip().startswith("--"):
|
||||
continue
|
||||
buf.append(line)
|
||||
if line.strip().endswith(";"):
|
||||
statement = "\n".join(buf).strip()
|
||||
if statement.endswith(";"):
|
||||
statement = statement[:-1].strip()
|
||||
if statement:
|
||||
parts.append(statement)
|
||||
buf = []
|
||||
tail = "\n".join(buf).strip().rstrip(";").strip()
|
||||
if tail:
|
||||
parts.append(tail)
|
||||
return parts
|
||||
|
||||
|
||||
def wait_postgres(conninfo):
|
||||
last = None
|
||||
for _ in range(60):
|
||||
try:
|
||||
with psycopg.connect(conninfo, connect_timeout=CONNECT_TIMEOUT) as conn:
|
||||
conn.execute("SELECT 1")
|
||||
return
|
||||
except psycopg.OperationalError as exc:
|
||||
last = type(exc).__name__
|
||||
time.sleep(1)
|
||||
LOGGER.error("postgres not ready: %s", last)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def apply(conninfo):
|
||||
directory = Path(__file__).resolve().parents[1] / "migrations"
|
||||
with psycopg.connect(conninfo, connect_timeout=CONNECT_TIMEOUT, row_factory=dict_row) as conn:
|
||||
conn.execute("SELECT pg_advisory_lock(%s)", (MIGRATION_LOCK,))
|
||||
conn.commit()
|
||||
try:
|
||||
conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version text PRIMARY KEY,
|
||||
applied_at timestamptz NOT NULL DEFAULT now()
|
||||
)
|
||||
"""
|
||||
)
|
||||
conn.commit()
|
||||
for path in sorted(directory.glob("*.sql")):
|
||||
version = path.stem
|
||||
found = conn.execute(
|
||||
"SELECT 1 FROM schema_migrations WHERE version = %s",
|
||||
(version,),
|
||||
).fetchone()
|
||||
conn.rollback()
|
||||
if found:
|
||||
LOGGER.info("migration %s already applied", version)
|
||||
continue
|
||||
with conn.transaction():
|
||||
for statement in split_sql(path.read_text(encoding="utf-8")):
|
||||
conn.execute(statement)
|
||||
conn.execute(
|
||||
"INSERT INTO schema_migrations (version) VALUES (%s)",
|
||||
(version,),
|
||||
)
|
||||
LOGGER.info("migration %s applied", version)
|
||||
finally:
|
||||
conn.execute("SELECT pg_advisory_unlock(%s)", (MIGRATION_LOCK,))
|
||||
conn.commit()
|
||||
|
||||
|
||||
def main():
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
conninfo = database_url()
|
||||
wait_postgres(conninfo)
|
||||
apply(conninfo)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,26 @@
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHash, VerificationError, VerifyMismatchError
|
||||
|
||||
# Параметры ближе к учебному стенду, чем к максимальным: time=2, 19 MiB, Argon2id.
|
||||
# Хеш всё равно медленный password KDF, не SHA-256 и не открытый текст.
|
||||
hasher = PasswordHasher(
|
||||
time_cost=2,
|
||||
memory_cost=19456,
|
||||
parallelism=1,
|
||||
hash_len=32,
|
||||
salt_len=16,
|
||||
)
|
||||
|
||||
# Чтобы неизвестный username не отвечал заметно быстрее известного.
|
||||
DUMMY_HASH = hasher.hash("dummy-password-not-a-user")
|
||||
|
||||
|
||||
def hash_password(password):
|
||||
return hasher.hash(password)
|
||||
|
||||
|
||||
def verify_password(password, encoded):
|
||||
try:
|
||||
return hasher.verify(encoded, password)
|
||||
except (VerifyMismatchError, VerificationError, InvalidHash):
|
||||
return False
|
||||
@@ -0,0 +1,174 @@
|
||||
"""Сверка строк attachments и объектов bucket.
|
||||
|
||||
Свежий объект моложе grace не удаляется: загрузка могла записать файл
|
||||
раньше, чем строка в базе. Строка, на которую ссылается сообщение, тоже
|
||||
не удаляется, даже если файла уже нет. По умолчанию только план, без удалений.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
from blobs import BUCKET, delete_quiet, list_objects
|
||||
|
||||
|
||||
def grace_seconds():
|
||||
raw = os.environ.get("RECONCILE_GRACE_SECONDS", "120")
|
||||
value = int(raw)
|
||||
if value < 1:
|
||||
raise RuntimeError("RECONCILE_GRACE_SECONDS")
|
||||
return value
|
||||
|
||||
|
||||
def database_url():
|
||||
path = os.environ.get("DATABASE_URL_FILE")
|
||||
if not path:
|
||||
raise RuntimeError("DATABASE_URL_FILE")
|
||||
with open(path, encoding="utf-8") as handle:
|
||||
return handle.read().strip()
|
||||
|
||||
|
||||
def aware(value):
|
||||
if value is None:
|
||||
return None
|
||||
if value.tzinfo is None:
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
return value.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def plan(grace):
|
||||
started = time.perf_counter()
|
||||
objects = list_objects()
|
||||
listed = time.perf_counter()
|
||||
now = datetime.now(timezone.utc)
|
||||
with psycopg.connect(database_url(), row_factory=dict_row) as conn:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.object_key, a.staging_key, a.created_at, a.status,
|
||||
EXISTS (
|
||||
SELECT 1 FROM message_attachments m
|
||||
WHERE m.attachment_id = a.id
|
||||
) AS in_use
|
||||
FROM attachments a
|
||||
"""
|
||||
).fetchall()
|
||||
variant_rows = conn.execute("SELECT object_key FROM attachment_variants").fetchall()
|
||||
by_key = {}
|
||||
for row in rows:
|
||||
by_key[row["object_key"]] = row
|
||||
if row["staging_key"]:
|
||||
by_key[row["staging_key"]] = row
|
||||
for item in variant_rows:
|
||||
by_key.setdefault(item["object_key"], {"status": "variant"})
|
||||
object_keys = {item["key"] for item in objects}
|
||||
actions = []
|
||||
preserved = []
|
||||
for item in objects:
|
||||
row = by_key.get(item["key"])
|
||||
if row is not None:
|
||||
continue
|
||||
created = aware(item["created_at"]) or now
|
||||
age = (now - created).total_seconds()
|
||||
entry = {"key": item["key"], "age_seconds": round(age, 3)}
|
||||
if age < grace:
|
||||
entry["reason"] = "fresh_object"
|
||||
preserved.append(entry)
|
||||
continue
|
||||
entry["action"] = "delete_orphan_object"
|
||||
actions.append(entry)
|
||||
for row in rows:
|
||||
if row["object_key"] in object_keys:
|
||||
continue
|
||||
created = aware(row["created_at"]) or now
|
||||
age = (now - created).total_seconds()
|
||||
entry = {
|
||||
"attachment_id": str(row["id"]),
|
||||
"key": row["object_key"],
|
||||
"age_seconds": round(age, 3),
|
||||
}
|
||||
if row["status"] != "ready":
|
||||
entry["reason"] = "not_ready"
|
||||
preserved.append(entry)
|
||||
continue
|
||||
if row["in_use"]:
|
||||
entry["reason"] = "missing_but_in_use"
|
||||
preserved.append(entry)
|
||||
continue
|
||||
if age < grace:
|
||||
entry["reason"] = "fresh_row"
|
||||
preserved.append(entry)
|
||||
continue
|
||||
entry["action"] = "delete_orphan_row"
|
||||
actions.append(entry)
|
||||
return {
|
||||
"bucket": BUCKET,
|
||||
"grace_seconds": grace,
|
||||
"objects": len(objects),
|
||||
"rows": len(rows),
|
||||
"list_seconds": round(listed - started, 6),
|
||||
"plan_seconds": round(time.perf_counter() - started, 6),
|
||||
"actions": actions,
|
||||
"preserved": preserved,
|
||||
}
|
||||
|
||||
|
||||
def apply_plan(report):
|
||||
deleted_objects = 0
|
||||
deleted_rows = 0
|
||||
with psycopg.connect(database_url(), row_factory=dict_row) as conn:
|
||||
for action in report["actions"]:
|
||||
if action["action"] == "delete_orphan_object":
|
||||
if delete_quiet(action["key"]):
|
||||
deleted_objects += 1
|
||||
elif action["action"] == "delete_orphan_row":
|
||||
with conn.transaction():
|
||||
result = conn.execute(
|
||||
"""
|
||||
DELETE FROM attachments
|
||||
WHERE id = %s
|
||||
AND status = 'ready'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM message_attachments m
|
||||
WHERE m.attachment_id = attachments.id
|
||||
)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM attachment_jobs j
|
||||
WHERE j.attachment_id = attachments.id
|
||||
)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM attachment_variants v
|
||||
WHERE v.attachment_id = attachments.id
|
||||
)
|
||||
""",
|
||||
(action["attachment_id"],),
|
||||
)
|
||||
deleted_rows += result.rowcount
|
||||
report["deleted_objects"] = deleted_objects
|
||||
report["deleted_rows"] = deleted_rows
|
||||
return report
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Сверить вложения в базе и в bucket")
|
||||
parser.add_argument("--apply", action="store_true", help="выполнить удаления; без флага только план")
|
||||
parser.add_argument("--grace", type=int, default=None)
|
||||
args = parser.parse_args()
|
||||
grace = args.grace if args.grace is not None else grace_seconds()
|
||||
if grace < 1:
|
||||
print("grace должен быть не меньше 1 секунды", file=sys.stderr)
|
||||
return 2
|
||||
report = plan(grace)
|
||||
report["apply"] = bool(args.apply)
|
||||
if args.apply:
|
||||
apply_plan(report)
|
||||
print(json.dumps(report, ensure_ascii=False, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Явный повтор failed-задания. Уже готовый эффект второй раз не исполняется."""
|
||||
import json
|
||||
import sys
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
from blobs import read_secret
|
||||
from jobs import replay_job
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 2:
|
||||
print("usage: replay.py JOB_ID", file=sys.stderr)
|
||||
return 2
|
||||
url = read_secret(__import__("os").environ["DATABASE_URL_FILE"])
|
||||
with psycopg.connect(url, row_factory=dict_row) as conn:
|
||||
result = replay_job(conn, sys.argv[1])
|
||||
print(json.dumps(result, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, field_validator, model_validator
|
||||
|
||||
USERNAME_RE = re.compile(r"^[a-z0-9_]{3,32}$")
|
||||
|
||||
|
||||
class StrictModel(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
|
||||
def _visible(value, limit):
|
||||
if not isinstance(value, str) or not value.strip() or len(value) > limit:
|
||||
raise ValueError("text")
|
||||
return value
|
||||
|
||||
|
||||
class CreateUser(StrictModel):
|
||||
username: str
|
||||
display_name: str
|
||||
password: str
|
||||
|
||||
@field_validator("username")
|
||||
@classmethod
|
||||
def username_ok(cls, value):
|
||||
if not isinstance(value, str) or not USERNAME_RE.fullmatch(value):
|
||||
raise ValueError("username")
|
||||
return value
|
||||
|
||||
@field_validator("display_name")
|
||||
@classmethod
|
||||
def display_name_ok(cls, value):
|
||||
return _visible(value, 100)
|
||||
|
||||
@field_validator("password")
|
||||
@classmethod
|
||||
def password_ok(cls, value):
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise ValueError("password")
|
||||
# Длина в code points, не в байтах: bcrypt тут не подходит из-за усечения UTF-8.
|
||||
if len(value) < 12 or len(value) > 128:
|
||||
raise ValueError("password")
|
||||
return value
|
||||
|
||||
|
||||
class CreateChat(StrictModel):
|
||||
title: str
|
||||
member_ids: list
|
||||
|
||||
@field_validator("title")
|
||||
@classmethod
|
||||
def title_ok(cls, value):
|
||||
return _visible(value, 100)
|
||||
|
||||
@field_validator("member_ids")
|
||||
@classmethod
|
||||
def member_ids_ok(cls, value):
|
||||
if not isinstance(value, list) or not (1 <= len(value) <= 100):
|
||||
raise ValueError("member_ids")
|
||||
# Канонический UUID (нижний регистр). Иначе один и тот же id
|
||||
# в разном регистре выглядит как два участника и упирается в PK.
|
||||
normalized = []
|
||||
seen = set()
|
||||
for item in value:
|
||||
if not isinstance(item, str):
|
||||
raise ValueError("member_ids")
|
||||
try:
|
||||
canonical = str(uuid.UUID(item))
|
||||
except ValueError as exc:
|
||||
raise ValueError("member_ids") from exc
|
||||
if canonical in seen:
|
||||
raise ValueError("member_ids")
|
||||
seen.add(canonical)
|
||||
normalized.append(canonical)
|
||||
return normalized
|
||||
|
||||
|
||||
class CreateMessage(StrictModel):
|
||||
text: str
|
||||
attachment_ids: list = []
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def fill_attachments(cls, value):
|
||||
# Нет поля и явный [] — одно и то же тело для Idempotency-Key.
|
||||
if not isinstance(value, dict):
|
||||
return value
|
||||
data = dict(value)
|
||||
data.setdefault("attachment_ids", [])
|
||||
return data
|
||||
|
||||
@field_validator("text")
|
||||
@classmethod
|
||||
def text_ok(cls, value):
|
||||
if not isinstance(value, str) or len(value) > 4000:
|
||||
raise ValueError("text")
|
||||
if value != "" and not value.strip():
|
||||
raise ValueError("text")
|
||||
return value
|
||||
|
||||
@field_validator("attachment_ids")
|
||||
@classmethod
|
||||
def attachments_ok(cls, value):
|
||||
if not isinstance(value, list) or len(value) > 10:
|
||||
raise ValueError("attachment_ids")
|
||||
normalized = []
|
||||
seen = set()
|
||||
for item in value:
|
||||
if not isinstance(item, str):
|
||||
raise ValueError("attachment_ids")
|
||||
try:
|
||||
canonical = str(uuid.UUID(item))
|
||||
except ValueError as exc:
|
||||
raise ValueError("attachment_ids") from exc
|
||||
if canonical in seen:
|
||||
raise ValueError("attachment_ids")
|
||||
seen.add(canonical)
|
||||
normalized.append(canonical)
|
||||
return normalized
|
||||
|
||||
@model_validator(mode="after")
|
||||
def text_or_file(self):
|
||||
if self.text == "" and not self.attachment_ids:
|
||||
raise ValueError("text")
|
||||
if self.text and not self.text.strip():
|
||||
raise ValueError("text")
|
||||
return self
|
||||
+868
@@ -0,0 +1,868 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
import socket
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import psycopg
|
||||
import redis
|
||||
from psycopg.errors import CheckViolation, ForeignKeyViolation, UniqueViolation
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import ConnectionPool, PoolTimeout
|
||||
|
||||
from errors import AttachmentNotReady, AttachmentRejected, DependencyDown, IdempotencyConflict
|
||||
from passwords import DUMMY_HASH, hash_password, verify_password
|
||||
|
||||
LOGGER = logging.getLogger("messenger")
|
||||
|
||||
POOL_TIMEOUT_SECONDS = 3
|
||||
STATEMENT_TIMEOUT_MS = 5000
|
||||
LOCK_TIMEOUT_MS = 3000
|
||||
REDIS_MAX_CONNECTIONS = 16
|
||||
REDIS_SOCKET_TIMEOUT = 2
|
||||
SCHEMA_VERSION = "004_jobs"
|
||||
|
||||
SQL_MESSAGES_FIRST = """
|
||||
SELECT id, chat_id, sender_id, body AS text, created_at
|
||||
FROM messages
|
||||
WHERE chat_id = %s
|
||||
ORDER BY created_at, id
|
||||
LIMIT %s
|
||||
"""
|
||||
|
||||
SQL_MESSAGES_AFTER = """
|
||||
SELECT id, chat_id, sender_id, body AS text, created_at
|
||||
FROM messages
|
||||
WHERE chat_id = %s
|
||||
AND (created_at, id) > (%s, %s)
|
||||
ORDER BY created_at, id
|
||||
LIMIT %s
|
||||
"""
|
||||
|
||||
SQL_MESSAGE_BY_ID = """
|
||||
SELECT id, chat_id, sender_id, body AS text, created_at
|
||||
FROM messages
|
||||
WHERE id = %s
|
||||
"""
|
||||
|
||||
|
||||
def instance_id():
|
||||
# Docker без hostname: в контейнере это id контейнера, он не меняется,
|
||||
# пока живёт этот контейнер. stop-one.sh находит контейнер по этому значению.
|
||||
return socket.gethostname()
|
||||
|
||||
|
||||
def pool_max():
|
||||
raw = os.environ.get("POOL_MAX", "10")
|
||||
try:
|
||||
value = int(raw)
|
||||
except ValueError as exc:
|
||||
raise RuntimeError("POOL_MAX") from exc
|
||||
if not 1 <= value <= 32:
|
||||
raise RuntimeError("POOL_MAX")
|
||||
return value
|
||||
|
||||
|
||||
def utc_now():
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def dump_time(value):
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
text = value.astimezone(timezone.utc).isoformat(timespec="microseconds")
|
||||
if text.endswith("+00:00"):
|
||||
text = text[:-6] + "Z"
|
||||
return text
|
||||
|
||||
|
||||
def parse_time(value):
|
||||
if not isinstance(value, str) or not value.endswith("Z"):
|
||||
raise ValueError("time")
|
||||
return datetime.fromisoformat(value[:-1] + "+00:00")
|
||||
|
||||
|
||||
def encode_cursor(scope, created_at, item_id):
|
||||
raw = "{}|{}|{}".format(scope, created_at, item_id).encode("utf-8")
|
||||
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
|
||||
|
||||
|
||||
def decode_cursor(cursor, expected_scope):
|
||||
if not isinstance(cursor, str) or not cursor or len(cursor) > 512:
|
||||
raise ValueError("cursor")
|
||||
pad = "=" * (-len(cursor) % 4)
|
||||
try:
|
||||
decoded = base64.urlsafe_b64decode(cursor + pad).decode("utf-8")
|
||||
except (ValueError, UnicodeError) as exc:
|
||||
raise ValueError("cursor") from exc
|
||||
parts = decoded.split("|")
|
||||
if len(parts) != 3:
|
||||
raise ValueError("cursor")
|
||||
scope, created_at, item_id = parts
|
||||
if scope != expected_scope:
|
||||
raise ValueError("cursor")
|
||||
parse_time(created_at)
|
||||
try:
|
||||
uuid.UUID(item_id)
|
||||
except ValueError as exc:
|
||||
raise ValueError("cursor") from exc
|
||||
return created_at, item_id
|
||||
|
||||
|
||||
def session_key(token):
|
||||
digest = hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||
return "sess:" + digest
|
||||
|
||||
|
||||
def message_fingerprint(text, attachment_ids):
|
||||
# Канонический JSON: порядок ключей не важен, пропущенный attachment_ids равен [].
|
||||
canonical = json.dumps(
|
||||
{"attachment_ids": list(attachment_ids), "text": text},
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
)
|
||||
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def advisory_key(user_id, path, idem_key):
|
||||
raw = "\0".join((user_id, "POST", path, idem_key)).encode("utf-8")
|
||||
digest = hashlib.sha256(raw).digest()
|
||||
return int.from_bytes(digest[:8], "big", signed=True)
|
||||
|
||||
|
||||
def chat_path(chat_id):
|
||||
return "/api/v1/chats/" + chat_id + "/messages"
|
||||
|
||||
|
||||
def public_user(row):
|
||||
return {
|
||||
"id": str(row["id"]),
|
||||
"username": row["username"],
|
||||
"display_name": row["display_name"],
|
||||
"created_at": dump_time(row["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def public_message(row, attachment_ids=None):
|
||||
return {
|
||||
"id": str(row["id"]),
|
||||
"chat_id": str(row["chat_id"]),
|
||||
"sender_id": str(row["sender_id"]),
|
||||
"text": row["text"],
|
||||
"attachment_ids": list(attachment_ids or []),
|
||||
"created_at": dump_time(row["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def public_variant(row):
|
||||
return {
|
||||
"name": row["name"],
|
||||
"content_type": row["content_type"],
|
||||
"size_bytes": int(row["size_bytes"]),
|
||||
"width": int(row["width"]),
|
||||
"height": int(row["height"]),
|
||||
"sha256": row["sha256"],
|
||||
}
|
||||
|
||||
|
||||
def public_attachment(row, variants=None):
|
||||
return {
|
||||
"id": str(row["id"]),
|
||||
"chat_id": str(row["chat_id"]),
|
||||
"owner_id": str(row["owner_id"]),
|
||||
"filename": row["filename"],
|
||||
"content_type": row["content_type"],
|
||||
"size_bytes": int(row["size_bytes"]),
|
||||
"sha256": row["sha256"],
|
||||
"status": row["status"],
|
||||
"variants": [public_variant(item) for item in (variants or [])],
|
||||
"error_code": row["error_code"],
|
||||
"created_at": dump_time(row["created_at"]),
|
||||
"updated_at": dump_time(row["updated_at"]),
|
||||
}
|
||||
|
||||
|
||||
def public_chat(row, member_ids=None):
|
||||
members = member_ids if member_ids is not None else row["member_ids"]
|
||||
return {
|
||||
"id": str(row["id"]),
|
||||
"title": row["title"],
|
||||
"member_ids": [str(item) for item in members],
|
||||
"created_at": dump_time(row["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def read_secret_file(path):
|
||||
with open(path, encoding="utf-8") as handle:
|
||||
return handle.read().strip()
|
||||
|
||||
|
||||
def _db_down(exc):
|
||||
raise DependencyDown() from exc
|
||||
|
||||
|
||||
class Store:
|
||||
def __init__(self):
|
||||
self.pool = None
|
||||
self.redis = None
|
||||
self.ttl = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
|
||||
self.idem_ttl = int(os.environ.get("IDEMPOTENCY_TTL_SECONDS", "86400"))
|
||||
self.pool_size = pool_max()
|
||||
self.shutting_down = False
|
||||
if self.ttl < 1 or self.idem_ttl < 1:
|
||||
raise RuntimeError("TTL")
|
||||
kind = os.environ.get("SESSION_STORE", "redis").strip().lower()
|
||||
if kind != "redis":
|
||||
raise RuntimeError("SESSION_STORE")
|
||||
|
||||
def database_url(self):
|
||||
path = os.environ.get("DATABASE_URL_FILE")
|
||||
if path:
|
||||
return read_secret_file(path)
|
||||
url = os.environ.get("DATABASE_URL", "").strip()
|
||||
if not url:
|
||||
raise RuntimeError("DATABASE_URL_FILE")
|
||||
return url
|
||||
|
||||
def redis_url(self):
|
||||
path = os.environ.get("CACHE_URL_FILE")
|
||||
if path:
|
||||
return read_secret_file(path)
|
||||
url = os.environ.get("CACHE_URL", "").strip()
|
||||
if not url:
|
||||
raise RuntimeError("CACHE_URL_FILE")
|
||||
return url
|
||||
|
||||
def startup(self):
|
||||
conninfo = self.database_url()
|
||||
self._wait_schema(conninfo)
|
||||
self.pool = ConnectionPool(
|
||||
conninfo,
|
||||
min_size=1,
|
||||
max_size=self.pool_size,
|
||||
timeout=POOL_TIMEOUT_SECONDS,
|
||||
max_waiting=64,
|
||||
max_lifetime=300,
|
||||
kwargs={
|
||||
"connect_timeout": POOL_TIMEOUT_SECONDS,
|
||||
"row_factory": dict_row,
|
||||
"application_name": "mtusi-lab5-" + instance_id(),
|
||||
"options": "-c statement_timeout={} -c lock_timeout={} -c idle_in_transaction_session_timeout=15000".format(
|
||||
STATEMENT_TIMEOUT_MS,
|
||||
LOCK_TIMEOUT_MS,
|
||||
),
|
||||
},
|
||||
open=True,
|
||||
)
|
||||
pool = redis.ConnectionPool.from_url(
|
||||
self.redis_url(),
|
||||
max_connections=REDIS_MAX_CONNECTIONS,
|
||||
socket_timeout=REDIS_SOCKET_TIMEOUT,
|
||||
socket_connect_timeout=REDIS_SOCKET_TIMEOUT,
|
||||
health_check_interval=30,
|
||||
decode_responses=True,
|
||||
)
|
||||
self.redis = redis.Redis(connection_pool=pool)
|
||||
self._wait_redis()
|
||||
self._wait_bucket()
|
||||
try:
|
||||
from broker import broker_channel, publish_pending
|
||||
|
||||
with broker_channel():
|
||||
pass
|
||||
with self._connect() as conn:
|
||||
publish_pending(conn)
|
||||
except Exception as exc:
|
||||
LOGGER.warning("outbox startup deferred %s", type(exc).__name__)
|
||||
LOGGER.info("ready instance=%s ttl=%s pool_max=%s", instance_id(), self.ttl, self.pool_size)
|
||||
|
||||
def shutdown(self):
|
||||
self.shutting_down = True
|
||||
if self.pool is not None:
|
||||
self.pool.close()
|
||||
if self.redis is not None:
|
||||
self.redis.close()
|
||||
|
||||
def ready(self):
|
||||
if self.shutting_down or self.pool is None or self.redis is None:
|
||||
return False
|
||||
try:
|
||||
with self.pool.connection(timeout=POOL_TIMEOUT_SECONDS) as conn:
|
||||
row = conn.execute(
|
||||
"SELECT 1 FROM schema_migrations WHERE version = %s",
|
||||
(SCHEMA_VERSION,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
self.redis.ping()
|
||||
from blobs import bucket_ready
|
||||
|
||||
if not bucket_ready():
|
||||
return False
|
||||
except Exception:
|
||||
LOGGER.warning("ready check failed")
|
||||
return False
|
||||
return True
|
||||
|
||||
def _wait_redis(self):
|
||||
last = None
|
||||
for _ in range(60):
|
||||
try:
|
||||
self.redis.ping()
|
||||
return
|
||||
except redis.RedisError as exc:
|
||||
last = type(exc).__name__
|
||||
time.sleep(1)
|
||||
LOGGER.error("redis not ready: %s", last)
|
||||
raise SystemExit(1)
|
||||
|
||||
def _wait_schema(self, conninfo):
|
||||
# Схему накатывает сервис migrate. Реплики только ждут её, не применяют сами.
|
||||
last = None
|
||||
for _ in range(90):
|
||||
try:
|
||||
with psycopg.connect(conninfo, connect_timeout=POOL_TIMEOUT_SECONDS, row_factory=dict_row) as conn:
|
||||
row = conn.execute(
|
||||
"SELECT 1 FROM schema_migrations WHERE version = %s",
|
||||
(SCHEMA_VERSION,),
|
||||
).fetchone()
|
||||
if row is not None:
|
||||
return
|
||||
last = "migration missing"
|
||||
except psycopg.Error as exc:
|
||||
last = type(exc).__name__
|
||||
time.sleep(1)
|
||||
LOGGER.error("schema not ready: %s", last)
|
||||
raise SystemExit(1)
|
||||
|
||||
def _wait_bucket(self):
|
||||
from blobs import bucket_ready
|
||||
|
||||
for _ in range(60):
|
||||
if bucket_ready():
|
||||
return
|
||||
time.sleep(1)
|
||||
LOGGER.error("object storage not ready")
|
||||
raise SystemExit(1)
|
||||
|
||||
def _connect(self):
|
||||
try:
|
||||
return self.pool.connection(timeout=POOL_TIMEOUT_SECONDS)
|
||||
except PoolTimeout as exc:
|
||||
_db_down(exc)
|
||||
|
||||
def create_user(self, username, display_name, password):
|
||||
password_hash = hash_password(password)
|
||||
user_id = uuid.uuid4()
|
||||
created = utc_now()
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
with conn.transaction():
|
||||
row = conn.execute(
|
||||
"""
|
||||
INSERT INTO users (id, username, display_name, password_hash, created_at)
|
||||
VALUES (%s, %s, %s, %s, %s)
|
||||
ON CONFLICT (username) DO NOTHING
|
||||
RETURNING id, username, display_name, created_at
|
||||
""",
|
||||
(user_id, username, display_name, password_hash, created),
|
||||
).fetchone()
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
if row is None:
|
||||
return None
|
||||
return public_user(row)
|
||||
|
||||
def login(self, username, password):
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT id, username, display_name, password_hash, created_at
|
||||
FROM users
|
||||
WHERE username = %s
|
||||
""",
|
||||
(username,),
|
||||
).fetchone()
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
if row is None:
|
||||
verify_password(password, DUMMY_HASH)
|
||||
return None
|
||||
if not verify_password(password, row["password_hash"]):
|
||||
return None
|
||||
return self._issue(public_user(row))
|
||||
|
||||
def _issue(self, user):
|
||||
token = secrets.token_urlsafe(32)
|
||||
expires = utc_now() + timedelta(seconds=self.ttl)
|
||||
payload = json.dumps(
|
||||
{"user_id": user["id"], "expires_at": dump_time(expires)},
|
||||
separators=(",", ":"),
|
||||
)
|
||||
# pxat — абсолютное время. Рестарт Redis не сдвигает срок.
|
||||
# В значении тоже лежит expires_at: даже если TTL ключа разъедется, сессия кончится вовремя.
|
||||
try:
|
||||
self.redis.set(session_key(token), payload, pxat=int(expires.timestamp() * 1000))
|
||||
except redis.RedisError as exc:
|
||||
raise DependencyDown() from exc
|
||||
return {
|
||||
"token": token,
|
||||
"token_type": "Bearer",
|
||||
"expires_at": dump_time(expires),
|
||||
"user": user,
|
||||
}
|
||||
|
||||
def user_for_token(self, token):
|
||||
key = session_key(token)
|
||||
try:
|
||||
raw = self.redis.get(key)
|
||||
except redis.RedisError as exc:
|
||||
raise DependencyDown() from exc
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
expires = parse_time(payload["expires_at"])
|
||||
user_id = payload["user_id"]
|
||||
uuid.UUID(user_id)
|
||||
except (ValueError, KeyError, TypeError):
|
||||
return None
|
||||
if utc_now() >= expires:
|
||||
try:
|
||||
self.redis.delete(key)
|
||||
except redis.RedisError as exc:
|
||||
raise DependencyDown() from exc
|
||||
return None
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT id, username, display_name, created_at
|
||||
FROM users
|
||||
WHERE id = %s
|
||||
""",
|
||||
(user_id,),
|
||||
).fetchone()
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
if row is None:
|
||||
return None
|
||||
return public_user(row)
|
||||
|
||||
def revoke(self, token):
|
||||
try:
|
||||
deleted = self.redis.delete(session_key(token))
|
||||
except redis.RedisError as exc:
|
||||
raise DependencyDown() from exc
|
||||
return deleted == 1
|
||||
|
||||
def create_chat(self, title, member_ids, caller_id):
|
||||
try:
|
||||
caller_uuid = uuid.UUID(str(caller_id))
|
||||
member_uuids = []
|
||||
seen = set()
|
||||
for item in member_ids:
|
||||
member_uuid = uuid.UUID(str(item))
|
||||
if member_uuid in seen:
|
||||
raise ValueError("members")
|
||||
seen.add(member_uuid)
|
||||
member_uuids.append(member_uuid)
|
||||
except ValueError as exc:
|
||||
raise ValueError("members") from exc
|
||||
if caller_uuid not in seen:
|
||||
raise ValueError("members")
|
||||
chat_id = uuid.uuid4()
|
||||
created = utc_now()
|
||||
canonical_members = [str(item) for item in member_uuids]
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
try:
|
||||
with conn.transaction():
|
||||
conn.execute(
|
||||
"INSERT INTO chats (id, title, created_at) VALUES (%s, %s, %s)",
|
||||
(chat_id, title, created),
|
||||
)
|
||||
for ordinal, member_id in enumerate(member_uuids):
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO chat_members (chat_id, user_id, ordinal)
|
||||
VALUES (%s, %s, %s)
|
||||
""",
|
||||
(chat_id, member_id, ordinal),
|
||||
)
|
||||
row = conn.execute(
|
||||
"SELECT id, title, created_at FROM chats WHERE id = %s",
|
||||
(chat_id,),
|
||||
).fetchone()
|
||||
except ForeignKeyViolation as exc:
|
||||
raise ValueError("members") from exc
|
||||
except UniqueViolation as exc:
|
||||
raise ValueError("members") from exc
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
return public_chat(row, canonical_members)
|
||||
|
||||
def _member(self, conn, chat_id, user_id):
|
||||
return conn.execute(
|
||||
"SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s",
|
||||
(chat_id, user_id),
|
||||
).fetchone()
|
||||
|
||||
def get_chat(self, chat_id, user_id):
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
if self._member(conn, chat_id, user_id) is None:
|
||||
return None
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT c.id, c.title, c.created_at,
|
||||
array_agg(m.user_id ORDER BY m.ordinal) AS member_ids
|
||||
FROM chats c
|
||||
JOIN chat_members m ON m.chat_id = c.id
|
||||
WHERE c.id = %s
|
||||
GROUP BY c.id
|
||||
""",
|
||||
(chat_id,),
|
||||
).fetchone()
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
if row is None:
|
||||
return None
|
||||
return public_chat(row)
|
||||
|
||||
def list_chats(self, user_id, limit, cursor):
|
||||
after = None
|
||||
if cursor is not None:
|
||||
created_at, item_id = decode_cursor(cursor, "chats:" + user_id)
|
||||
after = (parse_time(created_at), uuid.UUID(item_id))
|
||||
sql = """
|
||||
SELECT c.id, c.title, c.created_at,
|
||||
array_agg(m.user_id ORDER BY m.ordinal) AS member_ids
|
||||
FROM chats c
|
||||
JOIN chat_members mine ON mine.chat_id = c.id AND mine.user_id = %s
|
||||
JOIN chat_members m ON m.chat_id = c.id
|
||||
"""
|
||||
params = [user_id]
|
||||
if after is not None:
|
||||
sql += " WHERE (c.created_at, c.id) > (%s, %s)"
|
||||
params.extend(after)
|
||||
sql += " GROUP BY c.id ORDER BY c.created_at, c.id LIMIT %s"
|
||||
params.append(limit + 1)
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
rows = conn.execute(sql, params).fetchall()
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
page = [public_chat(row) for row in rows[:limit]]
|
||||
if len(rows) > limit and page:
|
||||
last = page[-1]
|
||||
return page, encode_cursor("chats:" + user_id, last["created_at"], last["id"])
|
||||
return page, None
|
||||
|
||||
def _attachment_ids(self, conn, message_ids):
|
||||
if not message_ids:
|
||||
return {}
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT message_id, attachment_id
|
||||
FROM message_attachments
|
||||
WHERE message_id = ANY(%s::uuid[])
|
||||
ORDER BY ordinal
|
||||
""",
|
||||
(list(message_ids),),
|
||||
).fetchall()
|
||||
found = {}
|
||||
for row in rows:
|
||||
found.setdefault(str(row["message_id"]), []).append(str(row["attachment_id"]))
|
||||
return found
|
||||
|
||||
def is_member(self, chat_id, user_id):
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
return self._member(conn, chat_id, user_id) is not None
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
|
||||
def _variants(self, conn, attachment_id, pipeline_version):
|
||||
return conn.execute(
|
||||
"""
|
||||
SELECT name, content_type, size_bytes, width, height, sha256
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = %s AND pipeline_version = %s AND name <> 'original'
|
||||
ORDER BY CASE name
|
||||
WHEN 'thumbnail' THEN 1
|
||||
WHEN 'crop' THEN 2
|
||||
WHEN 'optimized' THEN 3
|
||||
ELSE 4
|
||||
END
|
||||
""",
|
||||
(attachment_id, pipeline_version),
|
||||
).fetchall()
|
||||
|
||||
def save_attachment(self, chat_id, owner_id, filename, content_type, data):
|
||||
from psycopg.types.json import Json
|
||||
|
||||
from blobs import delete_quiet, put_bytes, staging_object_key, variant_object_key
|
||||
from broker import publish_pending
|
||||
from faults import take_fault
|
||||
from jobs import job_message, max_attempts
|
||||
|
||||
attachment_id = uuid.uuid4()
|
||||
job_id = uuid.uuid4()
|
||||
created = utc_now()
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
staging = staging_object_key(attachment_id)
|
||||
final_key = variant_object_key(attachment_id, 1, "original")
|
||||
payload = job_message(job_id, attachment_id, 1, 1, created)
|
||||
attempts = max_attempts()
|
||||
put_bytes(staging, data, content_type, created)
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
with conn.transaction():
|
||||
if self._member(conn, chat_id, owner_id) is None:
|
||||
raise LookupError("chat")
|
||||
row = conn.execute(
|
||||
"""
|
||||
INSERT INTO attachments (
|
||||
id, chat_id, owner_id, filename, content_type,
|
||||
size_bytes, sha256, object_key, staging_key, status,
|
||||
pipeline_version, error_code, created_at, updated_at
|
||||
)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, 'queued', 1, NULL, %s, %s)
|
||||
RETURNING id, chat_id, owner_id, filename, content_type,
|
||||
size_bytes, sha256, status, error_code, created_at, updated_at
|
||||
""",
|
||||
(
|
||||
attachment_id,
|
||||
chat_id,
|
||||
owner_id,
|
||||
filename,
|
||||
content_type,
|
||||
len(data),
|
||||
digest,
|
||||
final_key,
|
||||
staging,
|
||||
created,
|
||||
created,
|
||||
),
|
||||
).fetchone()
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO attachment_jobs (
|
||||
id, attachment_id, pipeline_version, status, attempt,
|
||||
max_attempts, created_at, updated_at
|
||||
)
|
||||
VALUES (%s, %s, 1, 'queued', 1, %s, %s, %s)
|
||||
""",
|
||||
(job_id, attachment_id, attempts, created, created),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO outbox (id, job_id, attempt, payload, created_at)
|
||||
VALUES (%s, %s, 1, %s, %s)
|
||||
""",
|
||||
(uuid.uuid4(), job_id, Json(payload), created),
|
||||
)
|
||||
except (LookupError, ForeignKeyViolation):
|
||||
delete_quiet(staging)
|
||||
return None
|
||||
except psycopg.OperationalError as exc:
|
||||
delete_quiet(staging)
|
||||
_db_down(exc)
|
||||
except Exception:
|
||||
delete_quiet(staging)
|
||||
raise
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
skipped = take_fault(conn, "skip_inline_publish", attachment_id)
|
||||
if not skipped:
|
||||
with self._connect() as conn:
|
||||
publish_pending(conn, only_job_id=job_id)
|
||||
except Exception as exc:
|
||||
LOGGER.warning("publish deferred %s", type(exc).__name__)
|
||||
return public_attachment(row, [])
|
||||
|
||||
def get_attachment(self, attachment_id, user_id):
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.chat_id, a.owner_id, a.filename, a.content_type,
|
||||
a.size_bytes, a.sha256, a.status, a.error_code, a.pipeline_version,
|
||||
a.created_at, a.updated_at
|
||||
FROM attachments a
|
||||
JOIN chat_members m ON m.chat_id = a.chat_id AND m.user_id = %s
|
||||
WHERE a.id = %s
|
||||
""",
|
||||
(user_id, attachment_id),
|
||||
).fetchone()
|
||||
variants = []
|
||||
if row is not None and row["status"] == "ready":
|
||||
variants = self._variants(conn, row["id"], row["pipeline_version"])
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
if row is None:
|
||||
return None
|
||||
return public_attachment(row, variants)
|
||||
|
||||
def open_attachment(self, attachment_id, user_id, variant):
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT a.object_key, a.content_type, a.size_bytes, a.status, a.pipeline_version
|
||||
FROM attachments a
|
||||
JOIN chat_members m ON m.chat_id = a.chat_id AND m.user_id = %s
|
||||
WHERE a.id = %s
|
||||
""",
|
||||
(user_id, attachment_id),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
if row["status"] != "ready":
|
||||
raise AttachmentNotReady()
|
||||
if variant == "original":
|
||||
return {
|
||||
"object_key": row["object_key"],
|
||||
"content_type": row["content_type"],
|
||||
"size_bytes": int(row["size_bytes"]),
|
||||
}
|
||||
found = conn.execute(
|
||||
"""
|
||||
SELECT object_key, content_type, size_bytes
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = %s AND pipeline_version = %s AND name = %s
|
||||
""",
|
||||
(attachment_id, row["pipeline_version"], variant),
|
||||
).fetchone()
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
if found is None:
|
||||
return None
|
||||
return {
|
||||
"object_key": found["object_key"],
|
||||
"content_type": found["content_type"],
|
||||
"size_bytes": int(found["size_bytes"]),
|
||||
}
|
||||
|
||||
def add_message(self, chat_id, sender_id, text, idem_key, attachment_ids):
|
||||
# Один ключ сериализуется между репликами через advisory lock в той же
|
||||
# транзакции, где INSERT сообщения. Откат не оставляет ключ занятым.
|
||||
path = chat_path(chat_id)
|
||||
digest = message_fingerprint(text, attachment_ids)
|
||||
lock_key = advisory_key(sender_id, path, idem_key)
|
||||
message_id = uuid.uuid4()
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
with conn.transaction():
|
||||
# Иначе lock_timeout сессии (3 с) оборвёт очередь одинаковых ключей.
|
||||
conn.execute("SET LOCAL lock_timeout = '15s'")
|
||||
conn.execute("SELECT pg_advisory_xact_lock(%s)", (lock_key,))
|
||||
if self._member(conn, chat_id, sender_id) is None:
|
||||
return None
|
||||
existing = conn.execute(
|
||||
"""
|
||||
SELECT request_hash, message_id,
|
||||
created_at >= now() - make_interval(secs => %s) AS fresh
|
||||
FROM idempotency_keys
|
||||
WHERE user_id = %s AND http_method = 'POST'
|
||||
AND request_path = %s AND idem_key = %s
|
||||
""",
|
||||
(self.idem_ttl, sender_id, path, idem_key),
|
||||
).fetchone()
|
||||
if existing is not None and existing["fresh"]:
|
||||
if existing["request_hash"] != digest:
|
||||
raise IdempotencyConflict()
|
||||
row = conn.execute(SQL_MESSAGE_BY_ID, (existing["message_id"],)).fetchone()
|
||||
if row is None:
|
||||
raise RuntimeError("idempotency target missing")
|
||||
linked = self._attachment_ids(conn, [row["id"]]).get(str(row["id"]), [])
|
||||
return public_message(row, linked)
|
||||
if existing is not None:
|
||||
conn.execute(
|
||||
"""
|
||||
DELETE FROM idempotency_keys
|
||||
WHERE user_id = %s AND http_method = 'POST'
|
||||
AND request_path = %s AND idem_key = %s
|
||||
""",
|
||||
(sender_id, path, idem_key),
|
||||
)
|
||||
if attachment_ids:
|
||||
found = conn.execute(
|
||||
"""
|
||||
SELECT id, chat_id, status
|
||||
FROM attachments
|
||||
WHERE id = ANY(%s::uuid[])
|
||||
""",
|
||||
(list(attachment_ids),),
|
||||
).fetchall()
|
||||
by_id = {str(item["id"]): item for item in found}
|
||||
for attachment_id in attachment_ids:
|
||||
item = by_id.get(attachment_id)
|
||||
if item is None or str(item["chat_id"]) != chat_id:
|
||||
return None
|
||||
if item["status"] == "failed" or item["status"] not in ("queued", "processing", "ready"):
|
||||
raise AttachmentRejected()
|
||||
row = conn.execute(
|
||||
"""
|
||||
INSERT INTO messages (id, chat_id, sender_id, body, created_at)
|
||||
VALUES (%s, %s, %s, %s, clock_timestamp())
|
||||
RETURNING id, chat_id, sender_id, body AS text, created_at
|
||||
""",
|
||||
(message_id, chat_id, sender_id, text),
|
||||
).fetchone()
|
||||
for ordinal, attachment_id in enumerate(attachment_ids):
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO message_attachments (message_id, attachment_id, ordinal)
|
||||
VALUES (%s, %s, %s)
|
||||
""",
|
||||
(row["id"], attachment_id, ordinal),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO idempotency_keys (
|
||||
user_id, http_method, request_path, idem_key,
|
||||
request_hash, message_id, created_at
|
||||
)
|
||||
VALUES (%s, 'POST', %s, %s, %s, %s, clock_timestamp())
|
||||
""",
|
||||
(sender_id, path, idem_key, digest, row["id"]),
|
||||
)
|
||||
return public_message(row, attachment_ids)
|
||||
except CheckViolation as exc:
|
||||
raise ValueError("message") from exc
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
|
||||
def list_messages(self, chat_id, user_id, limit, cursor):
|
||||
after = None
|
||||
if cursor is not None:
|
||||
created_at, item_id = decode_cursor(cursor, chat_id)
|
||||
after = (parse_time(created_at), uuid.UUID(item_id))
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
if self._member(conn, chat_id, user_id) is None:
|
||||
return None
|
||||
if after is None:
|
||||
rows = conn.execute(SQL_MESSAGES_FIRST, (chat_id, limit + 1)).fetchall()
|
||||
else:
|
||||
rows = conn.execute(
|
||||
SQL_MESSAGES_AFTER,
|
||||
(chat_id, after[0], after[1], limit + 1),
|
||||
).fetchall()
|
||||
linked = self._attachment_ids(conn, [row["id"] for row in rows[:limit]])
|
||||
except psycopg.OperationalError as exc:
|
||||
_db_down(exc)
|
||||
page = [public_message(row, linked.get(str(row["id"]), [])) for row in rows[:limit]]
|
||||
if len(rows) > limit and page:
|
||||
last = page[-1]
|
||||
return page, encode_cursor(chat_id, last["created_at"], last["id"])
|
||||
return page, None
|
||||
+182
@@ -0,0 +1,182 @@
|
||||
"""Отдельный процесс. API не уменьшает картинку, даже если worker остановлен."""
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
import pika
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import ConnectionPool
|
||||
|
||||
from blobs import bucket_ready, read_secret
|
||||
from broker import PROCESS_QUEUE, _parameters, broker_channel, ensure_topology, publish_pending
|
||||
from jobs import cleanup_staging, lease_seconds, process_delivery, release_expired
|
||||
|
||||
LOGGER = logging.getLogger("worker")
|
||||
SCHEMA_VERSION = "004_jobs"
|
||||
|
||||
|
||||
def database_url():
|
||||
return read_secret(os.environ["DATABASE_URL_FILE"])
|
||||
|
||||
|
||||
def concurrency():
|
||||
value = int(os.environ.get("WORKER_CONCURRENCY", "2"))
|
||||
if not 1 <= value <= 8:
|
||||
raise RuntimeError("WORKER_CONCURRENCY")
|
||||
return value
|
||||
|
||||
|
||||
def wait_schema(conninfo):
|
||||
last = "error"
|
||||
for _ in range(90):
|
||||
try:
|
||||
with psycopg.connect(conninfo, connect_timeout=3, row_factory=dict_row) as conn:
|
||||
row = conn.execute(
|
||||
"SELECT 1 FROM schema_migrations WHERE version = %s",
|
||||
(SCHEMA_VERSION,),
|
||||
).fetchone()
|
||||
if row is not None:
|
||||
return
|
||||
last = "migration missing"
|
||||
except psycopg.Error as exc:
|
||||
last = type(exc).__name__
|
||||
time.sleep(1)
|
||||
LOGGER.error("schema not ready: %s", last)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def wait_bucket():
|
||||
for _ in range(60):
|
||||
if bucket_ready():
|
||||
return
|
||||
time.sleep(1)
|
||||
LOGGER.error("object storage not ready")
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def wait_broker():
|
||||
last = "error"
|
||||
for _ in range(60):
|
||||
try:
|
||||
with broker_channel():
|
||||
return
|
||||
except Exception as exc:
|
||||
last = type(exc).__name__
|
||||
time.sleep(1)
|
||||
LOGGER.error("broker not ready: %s", last)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def make_pool(conninfo, slots):
|
||||
return ConnectionPool(
|
||||
conninfo,
|
||||
min_size=1,
|
||||
max_size=slots * 2 + 2,
|
||||
timeout=3,
|
||||
kwargs={
|
||||
"connect_timeout": 3,
|
||||
"row_factory": dict_row,
|
||||
"application_name": "mtusi-lab5-worker",
|
||||
"options": "-c statement_timeout=5000 -c lock_timeout=3000 -c idle_in_transaction_session_timeout=30000",
|
||||
},
|
||||
open=True,
|
||||
)
|
||||
|
||||
|
||||
def consume(pool, owner, slots):
|
||||
executor = ThreadPoolExecutor(max_workers=slots, thread_name_prefix="job")
|
||||
while True:
|
||||
connection = None
|
||||
try:
|
||||
connection = pika.BlockingConnection(_parameters())
|
||||
channel = connection.channel()
|
||||
ensure_topology(channel)
|
||||
channel.basic_qos(prefetch_count=slots)
|
||||
|
||||
def on_message(ch, method, _props, body):
|
||||
executor.submit(run_one, pool, owner, ch, method.delivery_tag, body)
|
||||
|
||||
channel.basic_consume(PROCESS_QUEUE, on_message, auto_ack=False)
|
||||
channel.start_consuming()
|
||||
except Exception as exc:
|
||||
LOGGER.warning("consumer %s", type(exc).__name__)
|
||||
time.sleep(1)
|
||||
finally:
|
||||
if connection is not None:
|
||||
try:
|
||||
connection.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def run_one(pool, owner, channel, tag, body):
|
||||
try:
|
||||
process_delivery(pool, body, owner)
|
||||
action = "ack"
|
||||
except Exception as exc:
|
||||
LOGGER.warning("delivery %s", type(exc).__name__)
|
||||
action = "nack"
|
||||
|
||||
def finish():
|
||||
try:
|
||||
if action == "ack":
|
||||
channel.basic_ack(tag)
|
||||
else:
|
||||
channel.basic_nack(tag, requeue=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
channel.connection.add_callback_threadsafe(finish)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def main():
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
conninfo = database_url()
|
||||
wait_schema(conninfo)
|
||||
wait_bucket()
|
||||
wait_broker()
|
||||
slots = concurrency()
|
||||
pool = make_pool(conninfo, slots)
|
||||
owner = socket.gethostname()
|
||||
stop = threading.Event()
|
||||
|
||||
def drain():
|
||||
while not stop.wait(1):
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
publish_pending(conn)
|
||||
except Exception as exc:
|
||||
LOGGER.warning("drain %s", type(exc).__name__)
|
||||
|
||||
def sweep():
|
||||
ticks = 0
|
||||
while not stop.wait(5):
|
||||
ticks += 1
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
release_expired(conn)
|
||||
if ticks % 12 == 0:
|
||||
cleanup_staging(conn)
|
||||
except Exception as exc:
|
||||
LOGGER.warning("sweep %s", type(exc).__name__)
|
||||
|
||||
threading.Thread(target=drain, name="drain", daemon=True).start()
|
||||
threading.Thread(target=sweep, name="sweep", daemon=True).start()
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
publish_pending(conn)
|
||||
except Exception as exc:
|
||||
LOGGER.warning("startup drain %s", type(exc).__name__)
|
||||
LOGGER.info("worker ready owner=%s concurrency=%s lease=%s", owner, slots, lease_seconds())
|
||||
consume(pool, owner, slots)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Негативные проверки API. Токены и пароли не печатаются."""
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
import uuid
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure # noqa: E402
|
||||
|
||||
|
||||
def png(width, height, payload=b""):
|
||||
def chunk(tag, data):
|
||||
return struct.pack(">I", len(data)) + tag + data + struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF)
|
||||
|
||||
ihdr = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)
|
||||
return b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr) + payload + chunk(b"IEND", b"")
|
||||
|
||||
|
||||
def main():
|
||||
base = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8080"
|
||||
client = Client(base)
|
||||
alice = client.user("n1")
|
||||
bob = client.user("n2")
|
||||
eve = client.user("n3")
|
||||
chat = client.call(
|
||||
"POST",
|
||||
"/api/v1/chats",
|
||||
201,
|
||||
actor=alice,
|
||||
body={"title": "Негативные", "member_ids": [alice["id"].upper(), bob["id"]]},
|
||||
schema="Chat",
|
||||
)["json"]
|
||||
other = client.call(
|
||||
"POST",
|
||||
"/api/v1/chats",
|
||||
201,
|
||||
actor=alice,
|
||||
body={"title": "Другой", "member_ids": [alice["id"], bob["id"]]},
|
||||
schema="Chat",
|
||||
)["json"]
|
||||
upper = client.call("GET", "/api/v1/chats/" + chat["id"].upper(), actor=bob, schema="Chat")["json"]
|
||||
results = []
|
||||
|
||||
def record(name, status):
|
||||
results.append({"name": name, "status": status})
|
||||
|
||||
record("chat_uuid_case", 200 if upper["id"] == chat["id"] else 0)
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages"
|
||||
record("blank_text", client.call("POST", path, 400, actor=alice, body={"text": " "}, headers={"Idempotency-Key": "blank-1"})["status"])
|
||||
record("bad_json", client.call("POST", path, 400, actor=alice, raw=b"{", headers={"Content-Type": "application/json", "Idempotency-Key": "bad-json"})["status"])
|
||||
record("short_password", client.call("POST", "/api/v1/users", 400, body={"username": "abc", "display_name": "Имя", "password": "short"})["status"])
|
||||
key = uuid.uuid4().hex
|
||||
first = client.call("POST", path, 201, actor=alice, body={"text": "одно"}, headers={"Idempotency-Key": key}, schema="Message")["json"]
|
||||
second = client.call(
|
||||
"POST",
|
||||
path,
|
||||
201,
|
||||
actor=alice,
|
||||
body={"text": "одно", "attachment_ids": []},
|
||||
headers={"Idempotency-Key": key},
|
||||
schema="Message",
|
||||
)["json"]
|
||||
record("omitted_equals_empty", 201 if first == second else 0)
|
||||
record("same_key_other_body", client.call("POST", path, 409, actor=alice, body={"text": "другое"}, headers={"Idempotency-Key": key})["status"])
|
||||
history = client.call("GET", path.replace("/messages", "/messages?limit=100"), actor=bob, schema="MessagePage")["json"]
|
||||
record("no_second_row", 200 if sum(1 for item in history["items"] if item["text"] in ("одно", "другое")) == 1 else 0)
|
||||
other_path = "/api/v1/chats/" + other["id"] + "/messages"
|
||||
other_message = client.call(
|
||||
"POST",
|
||||
other_path,
|
||||
201,
|
||||
actor=alice,
|
||||
body={"text": "одно"},
|
||||
headers={"Idempotency-Key": key},
|
||||
schema="Message",
|
||||
)["json"]
|
||||
record("same_key_other_chat", 201 if other_message["id"] != first["id"] else 0)
|
||||
record("foreign_chat", client.call("GET", "/api/v1/chats/" + chat["id"], 404, actor=eve)["status"])
|
||||
record("bad_cursor", client.call("GET", "/api/v1/chats/" + chat["id"] + "/messages?cursor=not-a-cursor", 400, actor=alice)["status"])
|
||||
upload = "/api/v1/chats/" + chat["id"] + "/attachments"
|
||||
record("slash_name", client.call("POST", upload + "?filename=a/b.png", 400, actor=alice, raw=png(1, 1), headers={"Content-Type": "image/png"})["status"])
|
||||
record("huge_pixels", client.call("POST", upload + "?filename=big.png", 413, actor=alice, raw=png(8000, 8000), headers={"Content-Type": "image/png"})["status"])
|
||||
record("bad_png", client.call("POST", upload + "?filename=bad.png", 415, actor=alice, raw=b"\x89PNG\r\n\x1a\nbroken", headers={"Content-Type": "image/png"})["status"])
|
||||
record("type_mismatch", client.call("POST", upload + "?filename=no.jpg", 415, actor=alice, raw=png(1, 1), headers={"Content-Type": "image/jpeg"})["status"])
|
||||
record("oversize", client.call("POST", upload + "?filename=over.png", 413, actor=alice, raw=b"x" * (10 * 1024 * 1024 + 1), headers={"Content-Type": "image/png"})["status"])
|
||||
missing = str(uuid.uuid4())
|
||||
record("unknown_attachment", client.call("GET", "/api/v1/attachments/" + missing.upper(), 404, actor=alice)["status"])
|
||||
record("no_token", client.call("GET", "/api/v1/users/me", 401)["status"])
|
||||
body = json.dumps({"results": results}, ensure_ascii=False, indent=2)
|
||||
print(body)
|
||||
failed = [item for item in results if item["status"] not in (200, 201, 400, 401, 404, 409, 413, 415)]
|
||||
if failed:
|
||||
return 1
|
||||
expected = {
|
||||
"chat_uuid_case": 200,
|
||||
"blank_text": 400,
|
||||
"bad_json": 400,
|
||||
"short_password": 400,
|
||||
"omitted_equals_empty": 201,
|
||||
"same_key_other_body": 409,
|
||||
"no_second_row": 200,
|
||||
"same_key_other_chat": 201,
|
||||
"foreign_chat": 404,
|
||||
"bad_cursor": 400,
|
||||
"slash_name": 400,
|
||||
"huge_pixels": 413,
|
||||
"bad_png": 415,
|
||||
"type_mismatch": 415,
|
||||
"oversize": 413,
|
||||
"unknown_attachment": 404,
|
||||
"no_token": 401,
|
||||
}
|
||||
for name, status in expected.items():
|
||||
got = next(item["status"] for item in results if item["name"] == name)
|
||||
if got != status:
|
||||
print("mismatch " + name, file=sys.stderr)
|
||||
return 1
|
||||
print("PASS negative")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except Failure as exc:
|
||||
print("FAIL " + str(exc), file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,653 @@
|
||||
"""Проверки очереди и трека 2. Токены и пароли не печатаются."""
|
||||
import hashlib
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure # noqa: E402
|
||||
from image_probe import jpeg_size # noqa: E402
|
||||
|
||||
BASE = "http://127.0.0.1:8080"
|
||||
EVIDENCE = ROOT / "evidence" / "pipeline.json"
|
||||
INSIDE = r"""
|
||||
import json, logging, os, hashlib
|
||||
from datetime import datetime, timezone
|
||||
logging.disable(logging.CRITICAL)
|
||||
req = json.loads(REQUEST_JSON)
|
||||
from blobs import read_secret, read_bytes, put_bytes, delete_quiet
|
||||
from jobs import bump_pipeline, replay_job, republish
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
url = read_secret(os.environ["DATABASE_URL_FILE"])
|
||||
action = req["action"]
|
||||
out = {"action": action}
|
||||
with psycopg.connect(url, row_factory=dict_row) as conn:
|
||||
if action == "republish":
|
||||
out["ok"] = bool(republish(conn, req["job_id"], req.get("attempt")))
|
||||
elif action == "bump":
|
||||
out["result"] = bump_pipeline(conn, req["attachment_id"])
|
||||
elif action == "replay":
|
||||
out["result"] = replay_job(conn, req["job_id"])
|
||||
elif action == "corrupt":
|
||||
row = conn.execute(
|
||||
"SELECT staging_key FROM attachments WHERE id = %s",
|
||||
(req["attachment_id"],),
|
||||
).fetchone()
|
||||
key = row["staging_key"]
|
||||
data = read_bytes(key)
|
||||
stamp = datetime.now(timezone.utc)
|
||||
put_bytes(key + ".held", data, "application/octet-stream", stamp)
|
||||
put_bytes(key, b"not-an-image", "image/png", stamp)
|
||||
out["original_sha256"] = hashlib.sha256(data).hexdigest()
|
||||
out["bytes"] = len(data)
|
||||
elif action == "restore":
|
||||
row = conn.execute(
|
||||
"SELECT staging_key FROM attachments WHERE id = %s",
|
||||
(req["attachment_id"],),
|
||||
).fetchone()
|
||||
key = row["staging_key"]
|
||||
data = read_bytes(key + ".held")
|
||||
put_bytes(key, data, "image/png", datetime.now(timezone.utc))
|
||||
delete_quiet(key + ".held")
|
||||
out["restored_sha256"] = hashlib.sha256(data).hexdigest()
|
||||
out["bytes"] = len(data)
|
||||
else:
|
||||
raise SystemExit("unknown")
|
||||
print(json.dumps(out))
|
||||
"""
|
||||
|
||||
|
||||
def run(args, timeout=180, check=True):
|
||||
proc = subprocess.run(
|
||||
args,
|
||||
cwd=str(ROOT),
|
||||
capture_output=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
if check and proc.returncode != 0:
|
||||
err = proc.stderr.decode("utf-8", "replace")[-1200:]
|
||||
raise RuntimeError(args[0] + " failed: " + err)
|
||||
return proc
|
||||
|
||||
|
||||
def compose(*args, timeout=180):
|
||||
run(["docker", "compose", "--env-file", ".env", *args], timeout=timeout)
|
||||
|
||||
|
||||
def inspect(name, template):
|
||||
proc = run(["docker", "inspect", "-f", template, name], timeout=30)
|
||||
return proc.stdout.decode().strip()
|
||||
|
||||
|
||||
def wait_health(name, seconds=180):
|
||||
deadline = time.monotonic() + seconds
|
||||
last = ""
|
||||
while time.monotonic() < deadline:
|
||||
last = inspect(name, "{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}}")
|
||||
if last.startswith("running healthy"):
|
||||
return last
|
||||
time.sleep(1)
|
||||
raise RuntimeError(name + " not healthy: " + last)
|
||||
|
||||
|
||||
def restart_service(service, containers, seconds=180):
|
||||
if isinstance(containers, str):
|
||||
containers = [containers]
|
||||
before = {name: inspect(name, "{{.State.StartedAt}}") for name in containers}
|
||||
compose("restart", service)
|
||||
deadline = time.monotonic() + seconds
|
||||
last = ""
|
||||
while time.monotonic() < deadline:
|
||||
ready = True
|
||||
for name in containers:
|
||||
started = inspect(name, "{{.State.StartedAt}}")
|
||||
last = inspect(name, "{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}}")
|
||||
if started == before[name] or not last.startswith("running healthy"):
|
||||
ready = False
|
||||
if ready:
|
||||
return
|
||||
time.sleep(1)
|
||||
raise RuntimeError(service + " did not restart: " + last)
|
||||
|
||||
|
||||
def psql(sql):
|
||||
proc = run(
|
||||
[
|
||||
"docker", "compose", "--env-file", ".env", "exec", "-T", "postgres",
|
||||
"psql", "-U", "messenger", "-d", "messenger",
|
||||
"-v", "ON_ERROR_STOP=1", "-q", "-t", "-A", "-c", sql,
|
||||
],
|
||||
timeout=30,
|
||||
)
|
||||
return proc.stdout.decode("utf-8").strip()
|
||||
|
||||
|
||||
def state(attachment_id):
|
||||
sql = """
|
||||
SELECT json_build_object(
|
||||
'status', a.status,
|
||||
'error_code', a.error_code,
|
||||
'pipeline_version', a.pipeline_version,
|
||||
'jobs', (
|
||||
SELECT coalesce(json_agg(json_build_object(
|
||||
'id', j.id, 'status', j.status, 'attempt', j.attempt,
|
||||
'pipeline_version', j.pipeline_version, 'error_code', j.error_code
|
||||
) ORDER BY j.pipeline_version), '[]'::json)
|
||||
FROM attachment_jobs j WHERE j.attachment_id = a.id
|
||||
),
|
||||
'variants', (
|
||||
SELECT coalesce(json_agg(json_build_object(
|
||||
'name', v.name, 'pipeline_version', v.pipeline_version,
|
||||
'width', v.width, 'height', v.height, 'content_type', v.content_type
|
||||
) ORDER BY v.pipeline_version, v.name), '[]'::json)
|
||||
FROM attachment_variants v WHERE v.attachment_id = a.id
|
||||
),
|
||||
'unpublished', (
|
||||
SELECT count(*) FROM outbox o
|
||||
JOIN attachment_jobs j ON j.id = o.job_id
|
||||
WHERE j.attachment_id = a.id AND o.published_at IS NULL
|
||||
),
|
||||
'events', (
|
||||
SELECT coalesce(json_agg(e.kind ORDER BY e.created_at), '[]'::json)
|
||||
FROM job_events e WHERE e.attachment_id = a.id
|
||||
)
|
||||
)::text
|
||||
FROM attachments a WHERE a.id = '{id}'::uuid
|
||||
""".format(id=attachment_id)
|
||||
return json.loads(psql(sql))
|
||||
|
||||
|
||||
def arm(kind, attachment_id):
|
||||
target = "NULL" if attachment_id is None else "'" + attachment_id + "'::uuid"
|
||||
psql(
|
||||
"INSERT INTO fault_injections (id, kind, attachment_id, armed) "
|
||||
"VALUES (gen_random_uuid(), '{kind}', {target}, true)".format(kind=kind, target=target)
|
||||
)
|
||||
|
||||
|
||||
def queues():
|
||||
proc = run(
|
||||
[
|
||||
"docker", "compose", "--env-file", ".env", "exec", "-T", "rabbitmq",
|
||||
"rabbitmqctl", "list_queues", "name", "messages", "--quiet",
|
||||
],
|
||||
timeout=30,
|
||||
)
|
||||
found = {}
|
||||
for line in proc.stdout.decode().splitlines():
|
||||
parts = line.split()
|
||||
if len(parts) == 2 and parts[1].isdigit():
|
||||
found[parts[0]] = int(parts[1])
|
||||
return found
|
||||
|
||||
|
||||
def depth():
|
||||
last = ""
|
||||
for _ in range(20):
|
||||
try:
|
||||
found = queues()
|
||||
return found.get("attachments.process", -1)
|
||||
except RuntimeError as exc:
|
||||
last = str(exc)
|
||||
time.sleep(1)
|
||||
raise RuntimeError(last)
|
||||
|
||||
|
||||
def inside(req, timeout=90):
|
||||
code = INSIDE.replace("REQUEST_JSON", json.dumps(json.dumps(req)))
|
||||
proc = run(
|
||||
[
|
||||
"docker", "compose", "--env-file", ".env", "run", "--rm", "--no-deps", "-T",
|
||||
"--entrypoint", "python", "worker", "-c", code,
|
||||
],
|
||||
timeout=timeout,
|
||||
)
|
||||
lines = [line for line in proc.stdout.decode("utf-8").splitlines() if line.startswith("{")]
|
||||
if not lines:
|
||||
raise RuntimeError("no json from worker helper")
|
||||
return json.loads(lines[-1])
|
||||
|
||||
|
||||
def stop_worker():
|
||||
compose("stop", "worker")
|
||||
|
||||
|
||||
def start_worker():
|
||||
compose("start", "worker")
|
||||
deadline = time.monotonic() + 60
|
||||
while time.monotonic() < deadline:
|
||||
if inspect("mtusi-lab5-worker-1", "{{.State.Status}}") == "running":
|
||||
return
|
||||
time.sleep(0.5)
|
||||
raise RuntimeError("worker did not start")
|
||||
|
||||
|
||||
def restart_count():
|
||||
return int(inspect("mtusi-lab5-worker-1", "{{.RestartCount}}"))
|
||||
|
||||
|
||||
def wait_status(actor, attachment_id, want, seconds=90):
|
||||
deadline = time.monotonic() + seconds
|
||||
last = None
|
||||
while time.monotonic() < deadline:
|
||||
body = client.call(
|
||||
"GET", "/api/v1/attachments/" + attachment_id, actor=actor, schema="Attachment"
|
||||
)["json"]
|
||||
last = body["status"]
|
||||
if last == want:
|
||||
return body
|
||||
if want != "failed" and last == "failed":
|
||||
raise Failure("attachment failed " + attachment_id)
|
||||
time.sleep(0.4)
|
||||
raise Failure("timeout status=" + str(last) + " want=" + want)
|
||||
|
||||
|
||||
def check_picture(actor, ready, original):
|
||||
download = client.call("GET", "/api/v1/attachments/" + ready["id"] + "/content", actor=actor)
|
||||
if download["bytes"] != original:
|
||||
raise Failure("original bytes differ")
|
||||
names = {item["name"]: item for item in ready["variants"]}
|
||||
thumb = client.call(
|
||||
"GET", "/api/v1/attachments/" + ready["id"] + "/content?variant=thumbnail", actor=actor
|
||||
)
|
||||
crop = client.call(
|
||||
"GET", "/api/v1/attachments/" + ready["id"] + "/content?variant=crop", actor=actor
|
||||
)
|
||||
optimized = client.call(
|
||||
"GET", "/api/v1/attachments/" + ready["id"] + "/content?variant=optimized", actor=actor
|
||||
)
|
||||
if jpeg_size(thumb["bytes"]) != (256, 160):
|
||||
raise Failure("thumbnail size")
|
||||
if jpeg_size(crop["bytes"]) != (128, 128):
|
||||
raise Failure("crop size")
|
||||
if optimized["bytes"][:4] != b"RIFF" or optimized["bytes"][8:12] != b"WEBP":
|
||||
raise Failure("optimized is not webp")
|
||||
if names["thumbnail"]["width"] != 256 or names["thumbnail"]["height"] != 160:
|
||||
raise Failure("thumbnail metadata")
|
||||
if names["crop"]["width"] != 128 or names["crop"]["height"] != 128:
|
||||
raise Failure("crop metadata")
|
||||
if names["optimized"]["width"] != 320 or names["optimized"]["height"] != 200:
|
||||
raise Failure("optimized metadata")
|
||||
if hashlib.sha256(thumb["bytes"]).hexdigest() != names["thumbnail"]["sha256"]:
|
||||
raise Failure("thumbnail sha")
|
||||
rows = state(ready["id"])
|
||||
version_rows = [row for row in rows["variants"] if row["pipeline_version"] == rows["pipeline_version"]]
|
||||
if len(version_rows) != 4:
|
||||
raise Failure("variant rows " + str(len(version_rows)))
|
||||
return {
|
||||
"pipeline_version": rows["pipeline_version"],
|
||||
"variant_rows": len(version_rows),
|
||||
"thumbnail": [names["thumbnail"]["width"], names["thumbnail"]["height"]],
|
||||
"crop": [names["crop"]["width"], names["crop"]["height"]],
|
||||
"optimized": [names["optimized"]["width"], names["optimized"]["height"]],
|
||||
"original_sha256": hashlib.sha256(original).hexdigest(),
|
||||
}
|
||||
|
||||
|
||||
def upload(actor, chat):
|
||||
att, content = client.attachment(actor, chat)
|
||||
return att, content
|
||||
|
||||
|
||||
results = []
|
||||
|
||||
|
||||
def scenario(name, fn):
|
||||
started = time.monotonic()
|
||||
try:
|
||||
detail = fn()
|
||||
results.append({"name": name, "ok": True, "seconds": round(time.monotonic() - started, 3), "detail": detail})
|
||||
except Exception as exc:
|
||||
results.append({"name": name, "ok": False, "seconds": round(time.monotonic() - started, 3), "error": type(exc).__name__ + ": " + str(exc)})
|
||||
raise
|
||||
|
||||
|
||||
client = Client(BASE, timeout=20)
|
||||
|
||||
|
||||
def durability():
|
||||
stop_worker()
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
att, content = upload(alice, chat)
|
||||
blocked = client.call("GET", "/api/v1/attachments/" + att["id"] + "/content", 409, actor=bob)
|
||||
text = client.message(alice, chat, "текст пока файл в очереди")["json"]
|
||||
linked = client.message(alice, chat, "", attachments=[att["id"]])["json"]
|
||||
before_api = depth()
|
||||
if before_api < 1:
|
||||
raise Failure("job was not in the durable queue")
|
||||
restart_service("api", ["mtusi-lab5-api-1", "mtusi-lab5-api-2"])
|
||||
after_api = depth()
|
||||
restart_service("rabbitmq", "mtusi-lab5-rabbitmq-1", 180)
|
||||
after_rabbit = depth()
|
||||
if after_api < 1 or after_rabbit < 1:
|
||||
raise Failure("queue lost the job")
|
||||
start_worker()
|
||||
ready = wait_status(bob, att["id"], "ready", 90)
|
||||
picture = check_picture(bob, ready, content)
|
||||
return {
|
||||
"upload_status": att["status"],
|
||||
"content_while_queued": blocked["status"],
|
||||
"text_status": 201,
|
||||
"text_id": text["id"],
|
||||
"linked_id": linked["id"],
|
||||
"queue_before_api_restart": before_api,
|
||||
"queue_after_api_restart": after_api,
|
||||
"queue_after_rabbit_restart": after_rabbit,
|
||||
"picture": picture,
|
||||
}
|
||||
|
||||
|
||||
def outbox_gap():
|
||||
stop_worker()
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
arm("skip_inline_publish", None)
|
||||
before = depth()
|
||||
att, content = upload(alice, chat)
|
||||
info = state(att["id"])
|
||||
queued = depth()
|
||||
if info["unpublished"] != 1 or queued != before:
|
||||
raise Failure("outbox was published early")
|
||||
start_worker()
|
||||
ready = wait_status(bob, att["id"], "ready", 90)
|
||||
picture = check_picture(bob, ready, content)
|
||||
return {"unpublished_before_worker": info["unpublished"], "queue_depth": queued, "picture": picture}
|
||||
|
||||
|
||||
def duplicate():
|
||||
stop_worker()
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
att, content = upload(alice, chat)
|
||||
info = state(att["id"])
|
||||
job_id = info["jobs"][0]["id"]
|
||||
published = inside({"action": "republish", "job_id": job_id})
|
||||
start_worker()
|
||||
ready = wait_status(bob, att["id"], "ready", 90)
|
||||
picture = check_picture(bob, ready, content)
|
||||
return {"republish": published["ok"], "picture": picture}
|
||||
|
||||
|
||||
def reorder():
|
||||
stop_worker()
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
att, content = upload(alice, chat)
|
||||
info = state(att["id"])
|
||||
job_id = info["jobs"][0]["id"]
|
||||
inside({"action": "republish", "job_id": job_id, "attempt": 2})
|
||||
start_worker()
|
||||
ready = wait_status(bob, att["id"], "ready", 90)
|
||||
picture = check_picture(bob, ready, content)
|
||||
after = state(att["id"])
|
||||
if after["jobs"][0]["attempt"] != 1 or after["status"] != "ready":
|
||||
raise Failure("reorder changed the job")
|
||||
return {"attempt": after["jobs"][0]["attempt"], "picture": picture}
|
||||
|
||||
|
||||
def stale_version():
|
||||
stop_worker()
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
att, content = upload(alice, chat)
|
||||
info = state(att["id"])
|
||||
old_job = info["jobs"][0]["id"]
|
||||
bumped = inside({"action": "bump", "attachment_id": att["id"]})
|
||||
start_worker()
|
||||
ready = wait_status(bob, att["id"], "ready", 90)
|
||||
before = check_picture(bob, ready, content)
|
||||
thumb_before = client.call(
|
||||
"GET", "/api/v1/attachments/" + att["id"] + "/content?variant=thumbnail", actor=bob
|
||||
)["bytes"]
|
||||
inside({"action": "republish", "job_id": old_job, "attempt": 1})
|
||||
time.sleep(2)
|
||||
ready2 = client.call("GET", "/api/v1/attachments/" + att["id"], actor=bob, schema="Attachment")["json"]
|
||||
thumb_after = client.call(
|
||||
"GET", "/api/v1/attachments/" + att["id"] + "/content?variant=thumbnail", actor=bob
|
||||
)["bytes"]
|
||||
after = state(att["id"])
|
||||
old = [job for job in after["jobs"] if job["id"] == old_job][0]
|
||||
if ready2["status"] != "ready" or after["pipeline_version"] != 2:
|
||||
raise Failure("new version was overwritten")
|
||||
if old["status"] != "superseded":
|
||||
raise Failure("old job was not superseded")
|
||||
if thumb_before != thumb_after:
|
||||
raise Failure("old job changed thumbnail")
|
||||
return {
|
||||
"bump": bumped["result"],
|
||||
"old_job_status": old["status"],
|
||||
"old_error_code": old["error_code"],
|
||||
"attachment_error_code": after["error_code"],
|
||||
"picture": before,
|
||||
"thumbnail_unchanged": True,
|
||||
}
|
||||
|
||||
|
||||
def crash(kind):
|
||||
stop_worker()
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
att, content = upload(alice, chat)
|
||||
arm(kind, att["id"])
|
||||
start_worker()
|
||||
born = inspect("mtusi-lab5-worker-1", "{{.State.StartedAt}}")
|
||||
ready = wait_status(bob, att["id"], "ready", 120)
|
||||
picture = check_picture(bob, ready, content)
|
||||
restarted = inspect("mtusi-lab5-worker-1", "{{.State.StartedAt}}")
|
||||
if born == restarted:
|
||||
raise Failure("worker did not exit")
|
||||
return {"started_at": born, "restarted_at": restarted, "picture": picture}
|
||||
|
||||
|
||||
def transient():
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
stop_worker()
|
||||
att, _content = upload(alice, chat)
|
||||
arm("transient_once", att["id"])
|
||||
start_worker()
|
||||
wait_status(bob, att["id"], "ready", 90)
|
||||
info = state(att["id"])
|
||||
version_rows = [row for row in info["variants"] if row["pipeline_version"] == info["pipeline_version"]]
|
||||
if info["jobs"][0]["attempt"] != 2 or "retry" not in info["events"] or len(version_rows) != 4:
|
||||
raise Failure("transient retry did not keep one result")
|
||||
return {
|
||||
"attempt": info["jobs"][0]["attempt"],
|
||||
"events": info["events"],
|
||||
"variant_rows": len(version_rows),
|
||||
"status": info["status"],
|
||||
}
|
||||
|
||||
|
||||
def permanent_and_replay():
|
||||
stop_worker()
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
att, content = upload(alice, chat)
|
||||
corrupted = inside({"action": "corrupt", "attachment_id": att["id"]})
|
||||
start_worker()
|
||||
failed = wait_status(bob, att["id"], "failed", 90)
|
||||
info = state(att["id"])
|
||||
restored = inside({"action": "restore", "attachment_id": att["id"]})
|
||||
replayed = inside({"action": "replay", "job_id": info["jobs"][0]["id"]})
|
||||
ready = wait_status(bob, att["id"], "ready", 90)
|
||||
picture = check_picture(bob, ready, content)
|
||||
again = inside({"action": "replay", "job_id": info["jobs"][0]["id"]})
|
||||
after = state(att["id"])
|
||||
version_rows = [row for row in after["variants"] if row["pipeline_version"] == after["pipeline_version"]]
|
||||
if failed["error_code"] != "invalid_image" or info["jobs"][0]["attempt"] != 1:
|
||||
raise Failure("permanent error was retried")
|
||||
if replayed["result"].get("result") != "queued" or again["result"].get("result") != "already_ready":
|
||||
raise Failure("replay result")
|
||||
if len(version_rows) != 4 or restored["restored_sha256"] != corrupted["original_sha256"]:
|
||||
raise Failure("replay did not keep one result")
|
||||
return {
|
||||
"corrupt_sha256": corrupted["original_sha256"],
|
||||
"failed_code": failed["error_code"],
|
||||
"attempt_at_fail": info["jobs"][0]["attempt"],
|
||||
"restored_sha256": restored["restored_sha256"],
|
||||
"replay": replayed["result"],
|
||||
"second_replay": again["result"],
|
||||
"variant_rows": len(version_rows),
|
||||
"picture": picture,
|
||||
}
|
||||
|
||||
|
||||
def lease():
|
||||
stop_worker()
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
arm("skip_inline_publish", None)
|
||||
before = depth()
|
||||
att, content = upload(alice, chat)
|
||||
info = state(att["id"])
|
||||
if info["unpublished"] != 1 or depth() != before:
|
||||
raise Failure("lease setup published the job")
|
||||
job_id = info["jobs"][0]["id"]
|
||||
psql(
|
||||
"UPDATE attachment_jobs SET status = 'processing', lease_owner = 'dead', "
|
||||
"lease_until = now() - interval '2 minutes', updated_at = now() "
|
||||
"WHERE id = '{id}'::uuid".format(id=job_id)
|
||||
)
|
||||
psql(
|
||||
"UPDATE attachments SET status = 'processing', updated_at = now() WHERE id = '{id}'::uuid".format(
|
||||
id=att["id"]
|
||||
)
|
||||
)
|
||||
psql("DELETE FROM outbox WHERE job_id = '{id}'::uuid AND published_at IS NULL".format(id=job_id))
|
||||
started = time.monotonic()
|
||||
start_worker()
|
||||
ready = wait_status(bob, att["id"], "ready", 90)
|
||||
elapsed = round(time.monotonic() - started, 3)
|
||||
picture = check_picture(bob, ready, content)
|
||||
info = state(att["id"])
|
||||
if "lease_expired" not in info["events"] or info["jobs"][0]["attempt"] != 1:
|
||||
raise Failure("expired lease was not resumed")
|
||||
return {
|
||||
"seconds_until_ready": elapsed,
|
||||
"events": info["events"],
|
||||
"attempt": info["jobs"][0]["attempt"],
|
||||
"picture": picture,
|
||||
}
|
||||
|
||||
|
||||
def http_status(url):
|
||||
req = urllib.request.Request(url, headers={"Accept": "application/json"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as response:
|
||||
return response.status, response.read(4096).decode("utf-8", "replace")
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read(4096).decode("utf-8", "replace")
|
||||
|
||||
|
||||
def leaked(text):
|
||||
lowered = text.lower()
|
||||
return [
|
||||
word for word in (
|
||||
"redis", "postgres", "minio", "password", "select", "rabbit", "amqp",
|
||||
"secret", "database", "bucket", "messenger-attachments",
|
||||
)
|
||||
if word in lowered
|
||||
]
|
||||
|
||||
|
||||
def ready_503():
|
||||
compose("stop", "redis")
|
||||
time.sleep(2)
|
||||
lb_status, lb_body = http_status(BASE + "/health/ready")
|
||||
proc = run(
|
||||
[
|
||||
"docker", "exec", "mtusi-lab5-api-1", "python", "-c",
|
||||
"import urllib.request,urllib.error\n"
|
||||
"req=urllib.request.Request('http://127.0.0.1:8080/health/ready')\n"
|
||||
"try:\n"
|
||||
" r=urllib.request.urlopen(req, timeout=5)\n"
|
||||
" print(r.status)\n"
|
||||
" print(r.read(4096).decode())\n"
|
||||
"except urllib.error.HTTPError as e:\n"
|
||||
" print(e.code)\n"
|
||||
" print(e.read(4096).decode())\n",
|
||||
],
|
||||
timeout=20,
|
||||
)
|
||||
lines = proc.stdout.decode("utf-8", "replace").splitlines()
|
||||
api_status = int(lines[0])
|
||||
api_body = "\n".join(lines[1:])
|
||||
compose("start", "redis")
|
||||
wait_health("mtusi-lab5-redis-1")
|
||||
started = time.monotonic()
|
||||
deadline = started + 40
|
||||
recovered = None
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
recovered, _body = http_status(BASE + "/health/ready")
|
||||
except (urllib.error.URLError, TimeoutError, OSError):
|
||||
recovered = 0
|
||||
if recovered == 200:
|
||||
break
|
||||
time.sleep(0.5)
|
||||
recovery_seconds = round(time.monotonic() - started, 3)
|
||||
if lb_status != 503 or api_status != 503 or leaked(lb_body) or leaked(api_body):
|
||||
raise Failure("ready 503 text")
|
||||
if "Сервис временно недоступен" not in api_body or recovered != 200:
|
||||
raise Failure("ready 503 text")
|
||||
return {
|
||||
"lb_status": lb_status,
|
||||
"lb_body": lb_body,
|
||||
"api_status": api_status,
|
||||
"api_message": "Сервис временно недоступен",
|
||||
"forbidden": [],
|
||||
"recovered": recovered,
|
||||
"recovery_seconds": recovery_seconds,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
wait_health("mtusi-lab5-api-1")
|
||||
wait_health("mtusi-lab5-api-2")
|
||||
psql("UPDATE fault_injections SET armed = false WHERE armed")
|
||||
start_worker()
|
||||
deadline = time.monotonic() + 60
|
||||
while time.monotonic() < deadline:
|
||||
if depth() == 0 and queues().get("attachments.retry", 0) == 0:
|
||||
break
|
||||
time.sleep(0.5)
|
||||
steps = [
|
||||
("durability", durability),
|
||||
("outbox_gap", outbox_gap),
|
||||
("duplicate", duplicate),
|
||||
("reorder", reorder),
|
||||
("stale_version", stale_version),
|
||||
("crash_before_effect", lambda: crash("crash_before_effect")),
|
||||
("crash_after_objects", lambda: crash("crash_after_objects")),
|
||||
("crash_after_commit", lambda: crash("crash_after_commit")),
|
||||
("transient_once", transient),
|
||||
("permanent_replay", permanent_and_replay),
|
||||
("lease", lease),
|
||||
("ready_503", ready_503),
|
||||
]
|
||||
if len(sys.argv) > 1 and sys.argv[1] == "resume" and EVIDENCE.is_file():
|
||||
done = [
|
||||
item for item in json.loads(EVIDENCE.read_text(encoding="utf-8"))["results"] if item.get("ok")
|
||||
]
|
||||
results.extend(done)
|
||||
steps = [item for item in steps if item[0] not in {row["name"] for row in done}]
|
||||
failed = None
|
||||
for name, fn in steps:
|
||||
try:
|
||||
scenario(name, fn)
|
||||
except Exception as exc:
|
||||
failed = exc
|
||||
break
|
||||
EVIDENCE.write_text(json.dumps({"results": results}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
if failed is not None:
|
||||
print("FAIL " + type(failed).__name__ + ": " + str(failed), file=sys.stderr)
|
||||
return 1
|
||||
print("PASS pipeline")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except Failure as exc:
|
||||
print("FAIL " + str(exc), file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
Reference in New Issue
Block a user