forked from mtusi2026/mtusi2026lab4
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cced20387d | ||
|
|
fc22dcc4a4 |
@@ -18,3 +18,4 @@ __pycache__
|
|||||||
node_modules
|
node_modules
|
||||||
evidence
|
evidence
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
results
|
||||||
|
|||||||
@@ -31,3 +31,10 @@ test-persistence:
|
|||||||
test-failover:
|
test-failover:
|
||||||
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
|
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
|
||||||
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||||
|
|
||||||
|
test-attachments:
|
||||||
|
$(PYTHON) -X utf8 tests/attachments_negative.py --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||||
|
|
||||||
|
|
||||||
|
test-upload-abort:
|
||||||
|
$(PYTHON) -X utf8 tests/upload_abort_check.py --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||||
@@ -1,37 +1,437 @@
|
|||||||
# Отчёт — лабораторная 4
|
# Отчёт — лабораторная 4
|
||||||
|
|
||||||
- Автор, группа: TODO
|
- Автор, группа: TODO: Хуцаева Айса, БПИ2302.
|
||||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
- Целевая оценка: 5.
|
||||||
- Commit/PR: TODO после публикации
|
- Выбранный трек: трек 2 — согласованность двух хранилищ.
|
||||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
- Commit/PR: solution/khutsaeva-bpi2302.
|
||||||
|
|
||||||
## Запуск из чистого clone
|
|
||||||
|
|
||||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
|
||||||
|
|
||||||
## Архитектура и решения
|
## Архитектура и решения
|
||||||
|
|
||||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
### Процессы и хранилища
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
Client["Клиент"] --> LB["HAProxy"]
|
||||||
|
LB --> API["API × 2"]
|
||||||
|
API --> PG["PostgreSQL: метаданные"]
|
||||||
|
API --> Redis["Redis: сессии"]
|
||||||
|
API --> S3["MinIO: оригиналы"]
|
||||||
|
R["Служебный reconciler"] --> PG
|
||||||
|
R --> S3
|
||||||
|
Backup["Проверенные резервные копии"] --> R
|
||||||
|
```
|
||||||
|
|
||||||
|
### Что перенесено из lab3
|
||||||
|
|
||||||
|
- Две API-реплики и HAProxy.
|
||||||
|
- PostgreSQL и Redis с persistent volumes.
|
||||||
|
- Авторизация и серверные сессии.
|
||||||
|
- Проверка участия пользователя в чате.
|
||||||
|
- Идемпотентная отправка сообщений.
|
||||||
|
- Health/readiness endpoints.
|
||||||
|
- Миграции Flyway.
|
||||||
|
- Проверки перезапуска и отказа API-реплики.
|
||||||
|
|
||||||
|
### Что добавлено в lab4
|
||||||
|
|
||||||
|
- Приватное S3-хранилище MinIO.
|
||||||
|
- Метаданные Attachment в PostgreSQL.
|
||||||
|
- Связь сообщения с `attachment_ids`.
|
||||||
|
- Синхронный binary upload PNG/JPEG.
|
||||||
|
- Авторизованное скачивание оригиналов.
|
||||||
|
- Проверки размера, формата, filename и числа пикселей.
|
||||||
|
- Ограничение числа одновременно принимаемых загрузок.
|
||||||
|
- Очистка временных файлов и orphan-объектов.
|
||||||
|
- Отдельные ограниченные S3 credentials для API.
|
||||||
|
- Служебный reconciler с dry-run и восстановлением из проверенной копии.
|
||||||
|
- Эксперименты согласованности и стоимости обхода.
|
||||||
|
|
||||||
|
### Загрузка и ограничения
|
||||||
|
|
||||||
|
API принимает сырое тело PNG/JPEG, а не multipart-запрос.
|
||||||
|
|
||||||
|
Ограничения:
|
||||||
|
|
||||||
|
- размер оригинала — не более 10 MiB;
|
||||||
|
- число декодированных пикселей — не более 40 миллионов;
|
||||||
|
- MIME проверяется по содержимому;
|
||||||
|
- filename используется только для отображения;
|
||||||
|
- запрещаются разделители пути и управляющие символы в filename;
|
||||||
|
- object key назначает сервер;
|
||||||
|
- лишние параллельные загрузки получают `429 rate_limited`.
|
||||||
|
|
||||||
|
Тело запроса читается ограниченными порциями во временный файл.
|
||||||
|
Размер контролируется при чтении, в том числе при неизвестном
|
||||||
|
Content-Length. Размеры изображения проверяются до полного декодирования.
|
||||||
|
|
||||||
|
Локальный временный файл API используется только во время обработки.
|
||||||
|
После успешной загрузки оригинал хранится в S3, а не в каталоге API.
|
||||||
|
|
||||||
|
Ответ `201 ready` возвращается после сохранения оригинала и метаданных.
|
||||||
|
|
||||||
|
### Транзакционные границы
|
||||||
|
|
||||||
|
Общей атомарной транзакции между PostgreSQL и S3 нет.
|
||||||
|
|
||||||
|
При сохранении вложения:
|
||||||
|
|
||||||
|
1. Принимается и проверяется изображение.
|
||||||
|
2. Открывается транзакция PostgreSQL.
|
||||||
|
3. Захватывается advisory lock для object key.
|
||||||
|
4. Оригинал записывается в S3.
|
||||||
|
5. Метаданные записываются в PostgreSQL.
|
||||||
|
6. Транзакция фиксируется.
|
||||||
|
7. API возвращает `201`.
|
||||||
|
|
||||||
|
Если S3-запись прошла, а транзакция PostgreSQL не зафиксировалась,
|
||||||
|
возможно появление orphan-объекта. Он удаляется последующей сверкой
|
||||||
|
после grace period.
|
||||||
|
|
||||||
|
Очистка и reconciler используют тот же ключ блокировки, что и upload.
|
||||||
|
Перед удалением повторно проверяются наличие метаданных и возраст
|
||||||
|
объекта. Это защищает от удаления объекта незавершённой штатной загрузки.
|
||||||
|
|
||||||
|
### Поведение reconciler
|
||||||
|
|
||||||
|
| Состояние | Dry-run | Apply |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Старый объект без метаданных | `would_delete_orphans` | Удаление после повторной проверки |
|
||||||
|
| Свежий объект без метаданных | Пропуск | Пропуск |
|
||||||
|
| Метаданные и оригинал существуют | Проверка наличия и размера | То же |
|
||||||
|
| Оригинал отсутствует, проверенная копия есть | `would_restore` | Восстановление и проверка SHA-256 |
|
||||||
|
| Оригинал отсутствует, копии нет | `unresolved_missing` | Метаданные сохраняются |
|
||||||
|
| БД или S3 недоступны | Проход прерывается | Проход прерывается |
|
||||||
|
| Объект занят другим процессом | `busy` | Пропуск до следующего запуска |
|
||||||
|
|
||||||
|
Копия для восстановления имеет имя `<sha256>.bin`.
|
||||||
|
Перед восстановлением проверяются её размер и SHA-256.
|
||||||
|
После записи проверяются скачанные из S3 байты.
|
||||||
|
|
||||||
|
Reconciler не создаёт резервные копии автоматически.
|
||||||
|
В эксперименте копия сохранялась до восстановления намеренно
|
||||||
|
удалённого тестового оригинала.
|
||||||
|
|
||||||
|
Публичный контракт lab4 не расширялся статусом `missing`.
|
||||||
|
При отсутствии оригинала сохраняются сообщение и метаданные,
|
||||||
|
а download возвращает `503 unavailable`.
|
||||||
|
|
||||||
|
### Конфигурация
|
||||||
|
|
||||||
|
| Переменная | Назначение |
|
||||||
|
| --- | --- |
|
||||||
|
| `POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD` | Инициализация PostgreSQL и подключение приложения через Compose |
|
||||||
|
| `DB_URL`, `DB_USERNAME`, `DB_PASSWORD` | Настройки PostgreSQL внутри API |
|
||||||
|
| `REDIS_HOST`, `REDIS_PASSWORD` | Подключение к Redis |
|
||||||
|
| `S3_ENDPOINT` | Внутренний адрес `http://minio:9000` |
|
||||||
|
| `S3_BUCKET` | Bucket `messenger-attachments` |
|
||||||
|
| `S3_ACCESS_KEY`, `S3_SECRET_KEY` | Ограниченные credentials API |
|
||||||
|
| `MINIO_ROOT_USER`, `MINIO_ROOT_PASSWORD` | Административная инициализация MinIO |
|
||||||
|
| `API_REPLICAS` | Число API-реплик |
|
||||||
|
| `API_CPUS`, `API_MEMORY` | Лимиты API-контейнера |
|
||||||
|
| `DB_POOL_SIZE` | Размер пула соединений API |
|
||||||
|
| `APP_CLEANUP_ENABLED` | Временное отключение фоновой очистки в эксперименте |
|
||||||
|
|
||||||
|
В Java S3-параметры используются через `app.s3.*`.
|
||||||
|
Reconciler получает настройки из контейнеров и использует S3 credentials
|
||||||
|
API, а не root-пользователя MinIO.
|
||||||
|
|
||||||
|
В текущей реализации регион S3 задан как `us-east-1`.
|
||||||
|
Изменение одной переменной `S3_REGION` не переключает регион автоматически.
|
||||||
|
|
||||||
|
Предложенные skeleton переменные `DATABASE_URL_FILE`,
|
||||||
|
`S3_ACCESS_KEY_ID_FILE` и `S3_SECRET_ACCESS_KEY_FILE` не следует считать
|
||||||
|
реализованными автоматически: текущий вариант использует переменные
|
||||||
|
окружения.
|
||||||
|
|
||||||
|
TODO перед публикацией: сверить актуальный `.env.example` с этой таблицей,
|
||||||
|
удалить неподдерживаемые настройки либо явно пометить их как примеры.
|
||||||
|
Лимиты 10 MiB и 40 миллионов пикселей реализованы в валидаторе;
|
||||||
|
наличие одноимённых переменных в skeleton само по себе их не меняет.
|
||||||
|
|
||||||
## Доказательства критериев
|
## Доказательства критериев
|
||||||
|
|
||||||
|
JSON-отчёты экспериментов сохранены в `results/`.
|
||||||
|
Обозначения каталогов:
|
||||||
|
|
||||||
|
- `results/reconcile-<chat_id>/` — эксперимент исправления расхождений;
|
||||||
|
- `results/reconcile-failure-<chat_id>/` — отказы зависимостей;
|
||||||
|
- `results/reconcile-cost-<run_id>/` — измерения стоимости обхода.
|
||||||
|
|
||||||
|
Для проверок, выполненных ранее только с выводом в терминал,
|
||||||
|
ниже явно указана необходимость сохранить файл доказательства.
|
||||||
|
|
||||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| Публичный smoke | make test | TODO | TODO |
|
| Валидность контракта | `make validate` | PASS: OpenAPI и JSON Schema валидны; приложение этим не тестируется | TODO: сохранить вывод |
|
||||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
| Структура skeleton | `make check` | PASS: структура, ссылки и синтаксис Python; приложение этим не тестируется | TODO: сохранить вывод |
|
||||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
| Публичный smoke | `make test` | PASS: публичный сценарий lab4 | TODO: сохранить вывод |
|
||||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
| Сохранение возможностей lab3 | `make test`, persistence и failover | Сохраняются сообщения и действующая сессия | TODO: логи smoke и resilience |
|
||||||
|
| Метаданные в PostgreSQL, оригиналы в S3 | Миграция Attachment, код сохранения, smoke | Оригинал скачивается через API; метаданные отделены от байтов | Миграции, `AttachmentPersistenceService.java`, `S3StorageService.java`; TODO: лог smoke |
|
||||||
|
| Сообщения с вложениями | Smoke и `tests/reconcile_check.py` | Сообщение и ссылки на вложения сохраняются после сверки и восстановления | `results/reconcile-<chat_id>/`; TODO: итоговый stdout теста |
|
||||||
|
| Ограничения изображения | `make test-attachments` | Негативный suite прошёл: превышение размера/пикселей, повреждение, MIME и filename | TODO: сохранить stdout negative suite |
|
||||||
|
| Запрет чужого доступа | Smoke и negative suite | Посторонний не получает чужое вложение; чужой attachment и другой чат отклоняются | TODO: логи smoke и negative suite |
|
||||||
|
| Переключение API-реплики | `make test-failover ACTION_SCRIPT=./scripts/stop-one.sh` | 2 реплики до воздействия, 74 успешные записи, 2 неудачные попытки; первая успешная запись через 1,833 с | TODO: лог теста и состояние контейнеров |
|
||||||
|
| Остановка именно загрузившей реплики | Upload → фиксация `X-Instance-Id` → остановка этой реплики → download | Отдельная привязка остановленной реплики к успешному upload пока не зафиксирована в представленном доказательстве | TODO: добавить адресное доказательство |
|
||||||
|
| Сохранение после restart | `make test-persistence ACTION_SCRIPT=./scripts/restart.sh` | PASS: подтверждённые данные и действующая сессия сохранены | TODO: stdout теста и трасса hook |
|
||||||
|
| Недоступность S3 | `tests/reconcile_failure_check.py` | При остановке MinIO reconciler завершился без изменения объектов; после восстановления оригинал доступен через API | `02-minio-unavailable.json`, `summary.json` в каталоге failure |
|
||||||
|
| Пользовательские ошибки не превращаются в 500 | `make test-attachments` | Негативные сценарии завершились ожидаемыми клиентскими ошибками | TODO: stdout negative suite |
|
||||||
|
| Прерывание HTTP upload | `make test-upload-abort` | Временный файл удалён; обе реплики принимают следующие загрузки; оригинал цел | TODO: сохранить stdout |
|
||||||
|
| Аварийное завершение API | `python3 -X utf8 tests/upload_crash_check.py` | После SIGKILL файл сохранился; после запуска того же контейнера просроченный файл удалён | TODO: сохранить stdout |
|
||||||
|
| Отказ БД после S3-записи | `tests/upload_db_failure_check.py` | После исправления кода ошибки на `unavailable` сценарий прошёл; orphan удаляется очисткой | TODO: сохранить stdout |
|
||||||
|
| Безопасная очистка orphan | `tests/cleanup_check.py --bucket messenger-attachments` | Старый orphan удалён; свежий объект и зарегистрированное вложение сохранены | TODO: сохранить stdout |
|
||||||
|
| Ограниченная память при ≥10 клиентах | `tests/upload_memory_check.py` | 10 клиентов, 81 успешная загрузка, 661 ответ 429, 0 неожиданных ошибок и OOM | `results/upload-memory.json` |
|
||||||
|
| Идемпотентный bootstrap | Два запуска `docker compose run --rm --no-deps --pull never minio-init` | Оба завершились успешно; повторное назначение политики обработано | TODO: сохранить вывод обоих запусков |
|
||||||
|
| Отдельные ограниченные S3 credentials | `infra/minio/check-access.sh` через `minio-init` | API может записывать и читать разрешённые объекты; запрещённые операции отклоняются | TODO: сохранить итоговый stdout проверки |
|
||||||
|
| Private bucket вне API | Та же проверка доступа | Анонимное скачивание существующего объекта запрещено | TODO: сохранить stdout |
|
||||||
|
| Изоляция административной консоли | `docker compose port minio 9001` и проверка Compose | Требуется подтвердить отсутствие публикации на общем интерфейсе; при локальной публикации ожидается `127.0.0.1` | TODO: сохранить вывод и фрагмент конфигурации |
|
||||||
|
| Трек 5: обнаружение orphan и missing | `tests/reconcile_check.py` | Dry-run обнаружил по одному orphan и missing без изменений | `01-dry-run.json` |
|
||||||
|
| Трек 5: безопасное исправление | Тот же тест, apply | Старый orphan удалён; missing без копии не уничтожает метаданные | `02-apply-no-backup.json` |
|
||||||
|
| Трек 5: восстановление | Тот же тест с резервной копией | Dry-run запланировал восстановление; apply восстановил оригинал с проверкой байтов | `03-dry-run-with-backup.json`, `04-restore.json` |
|
||||||
|
| Трек 5: повторный запуск | Повторный apply | Нет повторных удалений или восстановлений | `05-repeat.json` |
|
||||||
|
| Трек 5: отказы обеих зависимостей | `tests/reconcile_failure_check.py` | MinIO и PostgreSQL останавливались по очереди; ложных missing и изменений данных нет | `results/reconcile-failure-<chat_id>/summary.json` |
|
||||||
|
| Трек 5: измерение стоимости обхода | `tests/reconcile_cost_check.py --sizes 10 100 1000 --repeats 3` | Все проходы завершены; количество SQL/S3-вызовов совпало с расчётом | `summary.json`, `samples.csv`, `n*-run-*.json` в каталоге cost |
|
||||||
|
|
||||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
### Подтверждение инфраструктурных воздействий
|
||||||
|
|
||||||
|
Для persistence недостаточно одного PASS.
|
||||||
|
|
||||||
|
В показанной трассе restart-hook выполнялись:
|
||||||
|
|
||||||
|
- остановка API и LB;
|
||||||
|
- перезапуск PostgreSQL, Redis и MinIO;
|
||||||
|
- ожидание готовности хранилищ;
|
||||||
|
- запуск API;
|
||||||
|
- запуск LB.
|
||||||
|
|
||||||
|
Volumes при этом сохранялись.
|
||||||
|
|
||||||
|
Для сдачи необходимо приложить трассу фактической версии hook и состояние
|
||||||
|
контейнеров после выполнения. Аналогично для failover необходимо сохранить
|
||||||
|
идентификатор остановленной API-реплики.
|
||||||
|
|
||||||
|
В тесте отказов reconciler использовались реальные `docker stop`
|
||||||
|
и `docker start` для контейнеров MinIO и PostgreSQL. Настройки подключений
|
||||||
|
фиксировались до остановки, поэтому проверялось поведение worker при
|
||||||
|
недоступной зависимости, а не только ошибка обнаружения контейнера.
|
||||||
|
|
||||||
## Эксперименты и отказы
|
## Эксперименты и отказы
|
||||||
|
|
||||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
### Гипотеза
|
||||||
|
|
||||||
|
Повторяемая сверка PostgreSQL и S3 позволяет обнаруживать и безопасно
|
||||||
|
исправлять расхождения при следующих условиях:
|
||||||
|
|
||||||
|
- свежие объекты защищены grace period;
|
||||||
|
- удаление проверяет отсутствие метаданных под общей с upload блокировкой;
|
||||||
|
- отказ хранилища не считается отсутствием объекта;
|
||||||
|
- потерянный оригинал восстанавливается только из проверенной копии;
|
||||||
|
- повторный запуск не создаёт дополнительных изменений.
|
||||||
|
|
||||||
|
Для исправного набора данных ожидается примерно линейный рост числа
|
||||||
|
SQL-вызовов и операций S3 относительно количества вложений.
|
||||||
|
Время полного запуска дополнительно включает постоянные накладные расходы.
|
||||||
|
|
||||||
|
### Окружение и лимиты
|
||||||
|
|
||||||
|
Эксперименты выполнялись в Windows с Docker Desktop и WSL.
|
||||||
|
|
||||||
|
| Ресурс | Значение |
|
||||||
|
| --- | --- |
|
||||||
|
| CPU, доступные Docker | 12 |
|
||||||
|
| RAM, доступная Docker | 8 171 163 648 байт, около 7,61 GiB |
|
||||||
|
| API | 2 реплики, каждая 1 CPU и 768 MiB |
|
||||||
|
| PostgreSQL | 1 CPU, 512 MiB |
|
||||||
|
| MinIO | 1 CPU, 768 MiB |
|
||||||
|
| Reconciler | 1 CPU, 256 MiB |
|
||||||
|
|
||||||
|
Это лимиты контейнеров, а не измеренное потребление CPU/RAM.
|
||||||
|
Фактическая загрузка CPU в эксперименте сверки отдельно не измерялась.
|
||||||
|
|
||||||
|
Image ID reconciler в эксперименте:
|
||||||
|
|
||||||
|
`sha256:1e2c36d99dfe28d7f9eb8f57dc8b5273ec75e20ae88e8427e560b72f4594bbb4`
|
||||||
|
|
||||||
|
Image ID остальных сервисов сохранены в `environment` итогового JSON.
|
||||||
|
|
||||||
|
### Проверка согласованности
|
||||||
|
|
||||||
|
Созданы:
|
||||||
|
|
||||||
|
- корректное вложение, используемое сообщением;
|
||||||
|
- вложение, у которого намеренно удалён оригинал;
|
||||||
|
- старый S3-объект без метаданных;
|
||||||
|
- свежий S3-объект без метаданных.
|
||||||
|
|
||||||
|
Для старого orphan выполнено реальное ожидание 65 секунд при тестовом
|
||||||
|
grace period 60 секунд.
|
||||||
|
|
||||||
|
Перед каждым этапом создавался отдельный свежий orphan. Это исключило
|
||||||
|
накопление времени предыдущих запусков Docker в возрасте проверяемого
|
||||||
|
свежего объекта.
|
||||||
|
|
||||||
|
| Этап | Результат | Время worker, с | SQL | S3 |
|
||||||
|
| --- | --- | ---: | ---: | ---: |
|
||||||
|
| Dry-run без копии | Обнаружены orphan и missing; изменений нет | 0,490 | 13 | 6 |
|
||||||
|
| Apply без копии | Orphan удалён; missing остаётся нерешённым | 0,600 | 13 | 7 |
|
||||||
|
| Dry-run с копией | Запланировано восстановление | 0,262 | 11 | 5 |
|
||||||
|
| Apply с копией | Оригинал восстановлен и проверен | 0,368 | 11 | 7 |
|
||||||
|
| Повторный apply | Оба оригинала существуют; изменений нет | 0,190 | 11 | 4 |
|
||||||
|
|
||||||
|
`needs_attention` без резервной копии является ожидаемым результатом.
|
||||||
|
Метаданные не удаляются для сокрытия проблемы.
|
||||||
|
|
||||||
|
### Отказы и восстановление
|
||||||
|
|
||||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| TODO | TODO | TODO | TODO |
|
| Остановка API-реплики | Другая реплика продолжает обслуживание | Failover PASS; 74 успешные записи, 2 неудачные попытки | Первая успешная запись через 1,833 с; подтверждённые данные сохранены |
|
||||||
|
| Restart стенда с volumes | Сохраняются данные и сессия | Persistence PASS | Полное время восстановления отдельно не зафиксировано; потери проверяемых данных нет |
|
||||||
|
| Обрыв HTTP upload | Удаляется временный файл | Upload-abort PASS; последующие загрузки работают | Точное время отдельно не сохранено |
|
||||||
|
| SIGKILL принимающего API | Просроченный временный файл удаляется после запуска | Crash-сценарий PASS | Использовано искусственное старое mtime; полное ожидание штатного grace не измерялось |
|
||||||
|
| Ошибка БД после S3-записи | Orphan удаляется последующей сверкой | Проверка прошла | Интервал очистки тестовый; точное время не сохранено |
|
||||||
|
| MinIO остановлен перед запуском reconciler | Проход завершается без изменений | `EndpointConnectionError`; только попытка `head_bucket` | Ошибка обнаружена за 0,343 с; данные и сессия сохранены после запуска MinIO |
|
||||||
|
| PostgreSQL остановлен перед запуском reconciler | S3 не изменяется | `OperationalError`; операций S3 нет | Ошибка обнаружена за 0,160 с; данные и сессия сохранены после запуска PostgreSQL |
|
||||||
|
| Зависимости восстановлены | Сверка снова выполняется | Удалён один старый orphan | Проход занял 0,196 с |
|
||||||
|
| Повторная сверка после восстановления | Нет дополнительных изменений | Чистый повторный apply | Проход занял 0,375 с |
|
||||||
|
|
||||||
|
Время обнаружения ошибки worker не является временем полного
|
||||||
|
восстановления инфраструктуры.
|
||||||
|
|
||||||
|
### Память при конкурентных загрузках
|
||||||
|
|
||||||
|
Профиль: closed-loop, 10 клиентов, без автоматических повторов POST.
|
||||||
|
Продолжительность — 30,344 с.
|
||||||
|
|
||||||
|
Изображение: PNG 6000×6000, 36 миллионов пикселей,
|
||||||
|
157 192 байта в сжатом виде.
|
||||||
|
|
||||||
|
| Метрика | Результат |
|
||||||
|
| --- | ---: |
|
||||||
|
| Успешные загрузки | 81 |
|
||||||
|
| Ответы 429 | 661 |
|
||||||
|
| Неожиданные ошибки | 0 |
|
||||||
|
| Успешные RPS | 2,669 |
|
||||||
|
| p50 успешных запросов | 601,77 мс |
|
||||||
|
| p95 успешных запросов | 2084,50 мс |
|
||||||
|
| p99 успешных запросов | 4082,42 мс |
|
||||||
|
| OOM / OOM kill | 0 / 0 |
|
||||||
|
|
||||||
|
| Реплика | Память до теста, MiB | Наблюдаемый пик, MiB | Пик cgroup за жизнь контейнера после теста, MiB |
|
||||||
|
| --- | ---: | ---: | ---: |
|
||||||
|
| API 1 | 356,10 | 375,53 | 377,39 |
|
||||||
|
| API 2 | 361,88 | 378,82 | 379,07 |
|
||||||
|
|
||||||
|
Примерно 89% запросов отклонены механизмом admission control.
|
||||||
|
Это не успешные загрузки и не скрытые серверные ошибки.
|
||||||
|
|
||||||
|
Десять клиентов не означают десять одновременно работающих декодеров:
|
||||||
|
число активных загрузок ограничено приложением.
|
||||||
|
|
||||||
|
Память измерялась по cgroup контейнера, а не только Java heap.
|
||||||
|
Выборочный мониторинг мог пропустить короткие пики.
|
||||||
|
Показатель lifetime peak включает запуск и предыдущую работу контейнера.
|
||||||
|
|
||||||
|
### Нагрузка текстовых сообщений
|
||||||
|
|
||||||
|
Публичный `make load` выполнял closed-loop POST сообщений без автоматических
|
||||||
|
повторов:
|
||||||
|
|
||||||
|
| Метрика | Результат |
|
||||||
|
| --- | ---: |
|
||||||
|
| Concurrency | 4 |
|
||||||
|
| Продолжительность | 30,102 с |
|
||||||
|
| Успешные запросы | 841 |
|
||||||
|
| Ошибки | 0 |
|
||||||
|
| Успешные RPS | 27,938 |
|
||||||
|
| p50 | 92,299 мс |
|
||||||
|
| p95 | 408,735 мс |
|
||||||
|
| p99 | 615,139 мс |
|
||||||
|
|
||||||
|
Это измерение текстовых сообщений, а не пропускной способности upload.
|
||||||
|
Задержки рассчитаны для успешных запросов.
|
||||||
|
Closed-loop профиль не задаёт фиксированный внешний arrival rate
|
||||||
|
и имеет ограничение coordinated omission.
|
||||||
|
|
||||||
|
### Методика измерения стоимости обхода
|
||||||
|
|
||||||
|
Для каждого размера создан отдельный чат с 10, 100 или 1000 вложений.
|
||||||
|
|
||||||
|
Использован публичный fixture:
|
||||||
|
|
||||||
|
- RGB PNG 320×200;
|
||||||
|
- размер 111 817 байт;
|
||||||
|
- SHA-256:
|
||||||
|
`99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75`.
|
||||||
|
|
||||||
|
Всего загружено 1110 файлов, около 118,37 MiB оригиналов.
|
||||||
|
Подготовка данных заняла 95,224 с и не включалась во время обхода.
|
||||||
|
|
||||||
|
После последней загрузки выполнено ожидание 65 секунд.
|
||||||
|
Фоновая очистка API была отключена.
|
||||||
|
Для каждого объёма выполнен один прогревочный и три измеряемых прохода.
|
||||||
|
Порядок объёмов перемешивался с фиксированным seed 1729.
|
||||||
|
|
||||||
|
Все проходы выполнены в dry-run.
|
||||||
|
В каждом подтверждено:
|
||||||
|
|
||||||
|
- просмотрено ровно N объектов;
|
||||||
|
- просмотрено ровно N записей метаданных;
|
||||||
|
- все N объектов зарегистрированы;
|
||||||
|
- все N оригиналов найдены при обходе метаданных;
|
||||||
|
- свежих пропущенных объектов и занятых блокировок нет;
|
||||||
|
- расхождений и операций записи/удаления нет.
|
||||||
|
|
||||||
|
После измерений проверены неизменность метаданных и сообщений,
|
||||||
|
а также байты первого и последнего вложения каждого набора.
|
||||||
|
|
||||||
|
### Результаты стоимости обхода
|
||||||
|
|
||||||
|
| Вложения | Worker min, с | Worker median, с | Worker max, с | Медиана полного запуска, с | SQL | S3 |
|
||||||
|
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||||
|
| 10 | 0,381 | 0,413 | 0,604 | 9,580 | 53 | 12 |
|
||||||
|
| 100 | 1,484 | 1,949 | 2,003 | 10,885 | 503 | 102 |
|
||||||
|
| 1000 | 11,090 | 14,097 | 14,241 | 23,009 | 5004 | 1003 |
|
||||||
|
|
||||||
|
Во всех трёх повторах одного объёма количество вызовов было одинаковым.
|
||||||
|
|
||||||
|
При размере страницы 500 для текущего алгоритма и исправных старых
|
||||||
|
вложений получены формулы:
|
||||||
|
|
||||||
|
- SQL-вызовы: `5N + ceil(N / 500) + 2`;
|
||||||
|
- операции S3: `N + ceil(N / 500) + 1`.
|
||||||
|
|
||||||
|
Измерения полностью совпали с этим расчётом.
|
||||||
|
|
||||||
|
На каждый зарегистрированный старый объект приходятся два SQL-вызова
|
||||||
|
в проходе S3: блокировка и проверка метаданных.
|
||||||
|
|
||||||
|
На каждую запись метаданных приходятся три SQL-вызова:
|
||||||
|
чтение ключа, блокировка и повторное чтение строки.
|
||||||
|
Дополнительно учитываются запросы страниц и глобальная блокировка.
|
||||||
|
|
||||||
|
В S3 выполняются проверка bucket, получение страниц списка и один HEAD
|
||||||
|
на каждое вложение.
|
||||||
|
|
||||||
|
SQL-счётчик не включает служебные BEGIN/COMMIT.
|
||||||
|
Счётчик S3 отражает вызовы операций SDK, а не измеренный сетевой трафик.
|
||||||
|
|
||||||
|
### Вывод по гипотезе
|
||||||
|
|
||||||
|
Гипотеза о линейном росте количества обращений к хранилищам подтверждена
|
||||||
|
для исследованного диапазона: результаты соответствуют формулам алгоритма.
|
||||||
|
|
||||||
|
Время worker увеличивается с объёмом данных, но не строго пропорционально:
|
||||||
|
на него влияют постоянные расходы, кэширование и состояние локального
|
||||||
|
стенда. Три повтора недостаточны для универсальной модели задержек.
|
||||||
|
|
||||||
|
Дополнительные расходы полного запуска составили примерно
|
||||||
|
8,882–9,224 с на проход. Они включают запуск Python-скрипта,
|
||||||
|
обнаружение контейнеров, Docker и запуск worker.
|
||||||
|
|
||||||
|
Для небольших наборов эти расходы превышают время самой сверки.
|
||||||
|
Для 1000 вложений медиана работы worker составила 14,097 с,
|
||||||
|
а полного запуска — 23,009 с.
|
||||||
|
|
||||||
|
|
||||||
## Ограничения и следующие шаги
|
|
||||||
|
|
||||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
|
||||||
|
|||||||
@@ -0,0 +1,4 @@
|
|||||||
|
services:
|
||||||
|
api:
|
||||||
|
environment:
|
||||||
|
APP_CLEANUP_ENABLED: "false"
|
||||||
+129
-9
@@ -5,35 +5,131 @@ services:
|
|||||||
POSTGRES_DB: ${POSTGRES_DB:?}
|
POSTGRES_DB: ${POSTGRES_DB:?}
|
||||||
POSTGRES_USER: ${POSTGRES_USER:?}
|
POSTGRES_USER: ${POSTGRES_USER:?}
|
||||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?}
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?}
|
||||||
command: ["postgres", "-c", "max_connections=100", "-c", "shared_buffers=128MB"]
|
command:
|
||||||
|
- postgres
|
||||||
|
- -c
|
||||||
|
- max_connections=100
|
||||||
|
- -c
|
||||||
|
- shared_buffers=128MB
|
||||||
volumes:
|
volumes:
|
||||||
- postgres_data:/var/lib/postgresql/data
|
- postgres_data:/var/lib/postgresql/data
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
|
test:
|
||||||
|
- CMD-SHELL
|
||||||
|
- pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
|
||||||
interval: 2s
|
interval: 2s
|
||||||
timeout: 1s
|
timeout: 1s
|
||||||
retries: 30
|
retries: 30
|
||||||
|
|
||||||
cpus: 1.0
|
cpus: 1.0
|
||||||
mem_limit: 512m
|
mem_limit: 512m
|
||||||
networks: [data]
|
networks: [data]
|
||||||
|
|
||||||
redis:
|
redis:
|
||||||
image: redis:7.4.1-alpine
|
image: redis:7.4.1-alpine
|
||||||
environment:
|
environment:
|
||||||
REDIS_PASSWORD: ${REDIS_PASSWORD:?}
|
REDIS_PASSWORD: ${REDIS_PASSWORD:?}
|
||||||
command: ['sh', '-c', 'exec redis-server --requirepass "$$REDIS_PASSWORD" --appendonly yes --appendfsync always --maxmemory 128mb --maxmemory-policy noeviction']
|
command:
|
||||||
|
- sh
|
||||||
|
- -c
|
||||||
|
- |
|
||||||
|
exec redis-server \
|
||||||
|
--requirepass "$$REDIS_PASSWORD" \
|
||||||
|
--appendonly yes \
|
||||||
|
--appendfsync always \
|
||||||
|
--maxmemory 128mb \
|
||||||
|
--maxmemory-policy noeviction
|
||||||
volumes:
|
volumes:
|
||||||
- redis_data:/data
|
- redis_data:/data
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "REDISCLI_AUTH=$$REDIS_PASSWORD redis-cli ping | grep -q PONG"]
|
test:
|
||||||
|
- CMD-SHELL
|
||||||
|
- REDISCLI_AUTH="$$REDIS_PASSWORD" redis-cli ping | grep -q PONG
|
||||||
interval: 2s
|
interval: 2s
|
||||||
timeout: 1s
|
timeout: 1s
|
||||||
retries: 30
|
retries: 30
|
||||||
cpus: 0.5
|
cpus: 0.5
|
||||||
mem_limit: 192m
|
mem_limit: 192m
|
||||||
networks: [data]
|
networks: [data]
|
||||||
|
|
||||||
|
minio:
|
||||||
|
build:
|
||||||
|
context: ./infra/minio
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
|
||||||
|
image: mtusi4-minio:2025-04-22-local
|
||||||
|
pull_policy: build
|
||||||
|
|
||||||
|
command:
|
||||||
|
- server
|
||||||
|
- /data
|
||||||
|
- --console-address
|
||||||
|
- ":9001"
|
||||||
|
|
||||||
|
environment:
|
||||||
|
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?}
|
||||||
|
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?}
|
||||||
|
MINIO_API_STALE_UPLOADS_EXPIRY: "24h"
|
||||||
|
MINIO_API_STALE_UPLOADS_CLEANUP_INTERVAL: "1h"
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
- minio_data:/data
|
||||||
|
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:9001:9001"
|
||||||
|
|
||||||
|
healthcheck:
|
||||||
|
test:
|
||||||
|
- CMD
|
||||||
|
- curl
|
||||||
|
- -f
|
||||||
|
- http://localhost:9000/minio/health/live
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 20
|
||||||
|
start_period: 10s
|
||||||
|
|
||||||
|
cpus: 1.0
|
||||||
|
mem_limit: 768m
|
||||||
|
restart: unless-stopped
|
||||||
|
networks: [data]
|
||||||
|
|
||||||
|
minio-init:
|
||||||
|
build:
|
||||||
|
context: ./infra/minio
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
|
||||||
|
image: mtusi4-minio:2025-04-22-local
|
||||||
|
pull_policy: build
|
||||||
|
|
||||||
|
depends_on:
|
||||||
|
minio:
|
||||||
|
condition: service_healthy
|
||||||
|
|
||||||
|
environment:
|
||||||
|
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?}
|
||||||
|
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?}
|
||||||
|
|
||||||
|
S3_BUCKET: ${S3_BUCKET:?}
|
||||||
|
S3_ACCESS_KEY: ${S3_ACCESS_KEY:?}
|
||||||
|
S3_SECRET_KEY: ${S3_SECRET_KEY:?}
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
- ./infra/minio/bootstrap.sh:/bootstrap/bootstrap.sh:ro
|
||||||
|
- ./infra/minio/check-access.sh:/bootstrap/check-access.sh:ro
|
||||||
|
|
||||||
|
entrypoint:
|
||||||
|
- /bin/sh
|
||||||
|
- /bootstrap/bootstrap.sh
|
||||||
|
|
||||||
|
restart: "no"
|
||||||
|
networks: [data]
|
||||||
|
|
||||||
migrate:
|
migrate:
|
||||||
image: flyway/flyway:10.20.1-alpine
|
image: flyway/flyway:10.20.1-alpine
|
||||||
command: ["-connectRetries=30", "migrate"]
|
command:
|
||||||
|
- -connectRetries=30
|
||||||
|
- migrate
|
||||||
environment:
|
environment:
|
||||||
FLYWAY_URL: jdbc:postgresql://postgres:5432/${POSTGRES_DB:?}
|
FLYWAY_URL: jdbc:postgresql://postgres:5432/${POSTGRES_DB:?}
|
||||||
FLYWAY_USER: ${POSTGRES_USER:?}
|
FLYWAY_USER: ${POSTGRES_USER:?}
|
||||||
@@ -47,9 +143,10 @@ services:
|
|||||||
mem_limit: 384m
|
mem_limit: 384m
|
||||||
restart: "no"
|
restart: "no"
|
||||||
networks: [data]
|
networks: [data]
|
||||||
|
|
||||||
api:
|
api:
|
||||||
build: .
|
build: .
|
||||||
image: ${APP_IMAGE:-mtusi3-api:local}
|
image: ${APP_IMAGE:-mtusi4-api:local}
|
||||||
environment:
|
environment:
|
||||||
DB_URL: jdbc:postgresql://postgres:5432/${POSTGRES_DB:?}
|
DB_URL: jdbc:postgresql://postgres:5432/${POSTGRES_DB:?}
|
||||||
DB_USERNAME: ${POSTGRES_USER:?}
|
DB_USERNAME: ${POSTGRES_USER:?}
|
||||||
@@ -58,23 +155,42 @@ services:
|
|||||||
REDIS_PASSWORD: ${REDIS_PASSWORD:?}
|
REDIS_PASSWORD: ${REDIS_PASSWORD:?}
|
||||||
DB_POOL_SIZE: ${DB_POOL_SIZE:-8}
|
DB_POOL_SIZE: ${DB_POOL_SIZE:-8}
|
||||||
FLYWAY_ENABLED: "false"
|
FLYWAY_ENABLED: "false"
|
||||||
|
S3_ENDPOINT: http://minio:9000
|
||||||
|
S3_ACCESS_KEY: ${S3_ACCESS_KEY:?}
|
||||||
|
S3_SECRET_KEY: ${S3_SECRET_KEY:?}
|
||||||
|
S3_BUCKET: ${S3_BUCKET:?}
|
||||||
|
UPLOAD_MAX_CONCURRENT: ${UPLOAD_MAX_CONCURRENT:-1}
|
||||||
|
UPLOAD_TEMP_DIR: /tmp/messenger-uploads
|
||||||
|
ATTACHMENT_CLEANUP_ENABLED: "true"
|
||||||
|
ATTACHMENT_CLEANUP_GRACE_SECONDS: ${ATTACHMENT_CLEANUP_GRACE_SECONDS:-3600}
|
||||||
|
ATTACHMENT_CLEANUP_INTERVAL_MS: ${ATTACHMENT_CLEANUP_INTERVAL_MS:-300000}
|
||||||
|
ATTACHMENT_CLEANUP_INITIAL_DELAY_MS: ${ATTACHMENT_CLEANUP_INITIAL_DELAY_MS:-60000}
|
||||||
|
|
||||||
depends_on:
|
depends_on:
|
||||||
migrate:
|
migrate:
|
||||||
condition: service_completed_successfully
|
condition: service_completed_successfully
|
||||||
redis:
|
redis:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
minio-init:
|
||||||
|
condition: service_completed_successfully
|
||||||
deploy:
|
deploy:
|
||||||
replicas: ${API_REPLICAS:-2}
|
replicas: ${API_REPLICAS:-2}
|
||||||
cpus: ${API_CPUS:-1.0}
|
cpus: ${API_CPUS:-1.0}
|
||||||
mem_limit: ${API_MEMORY:-768m}
|
mem_limit: ${API_MEMORY:-768m}
|
||||||
stop_grace_period: 20s
|
stop_grace_period: 20s
|
||||||
|
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "wget", "-qO-", "http://localhost:8080/health/ready"]
|
test:
|
||||||
interval: 2s
|
- CMD
|
||||||
timeout: 2s
|
- wget
|
||||||
|
- -qO-
|
||||||
|
- http://localhost:8080/health/ready
|
||||||
|
interval: 5s
|
||||||
|
timeout: 15s
|
||||||
start_period: 60s
|
start_period: 60s
|
||||||
retries: 3
|
retries: 3
|
||||||
networks: [edge, data]
|
networks: [edge, data]
|
||||||
|
|
||||||
lb:
|
lb:
|
||||||
image: haproxy:3.0.5-alpine
|
image: haproxy:3.0.5-alpine
|
||||||
ports:
|
ports:
|
||||||
@@ -84,12 +200,16 @@ services:
|
|||||||
depends_on:
|
depends_on:
|
||||||
api:
|
api:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|
||||||
cpus: 0.5
|
cpus: 0.5
|
||||||
mem_limit: 128m
|
mem_limit: 128m
|
||||||
networks: [edge]
|
networks: [edge]
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
postgres_data:
|
postgres_data:
|
||||||
redis_data:
|
redis_data:
|
||||||
|
minio_data:
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
edge:
|
edge:
|
||||||
data:
|
data:
|
||||||
|
|||||||
@@ -0,0 +1,41 @@
|
|||||||
|
FROM golang:1.24.2-bookworm AS builder
|
||||||
|
|
||||||
|
ENV CGO_ENABLED=0
|
||||||
|
|
||||||
|
WORKDIR /src
|
||||||
|
|
||||||
|
RUN git clone --depth 1 \
|
||||||
|
--branch RELEASE.2025-04-22T22-12-26Z \
|
||||||
|
https://github.com/minio/minio.git /src/minio
|
||||||
|
|
||||||
|
WORKDIR /src/minio
|
||||||
|
|
||||||
|
RUN go build -trimpath -o /out/minio .
|
||||||
|
|
||||||
|
WORKDIR /src
|
||||||
|
|
||||||
|
RUN git clone --depth 1 \
|
||||||
|
--branch RELEASE.2025-04-16T18-13-26Z \
|
||||||
|
https://github.com/minio/mc.git /src/mc
|
||||||
|
|
||||||
|
WORKDIR /src/mc
|
||||||
|
|
||||||
|
RUN go build -trimpath -o /out/mc .
|
||||||
|
|
||||||
|
|
||||||
|
FROM debian:bookworm-slim
|
||||||
|
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends \
|
||||||
|
ca-certificates curl \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
COPY --from=builder /out/minio /usr/local/bin/minio
|
||||||
|
COPY --from=builder /out/mc /usr/local/bin/mc
|
||||||
|
|
||||||
|
RUN mkdir -p /data
|
||||||
|
|
||||||
|
EXPOSE 9000 9001
|
||||||
|
|
||||||
|
ENTRYPOINT ["minio"]
|
||||||
|
CMD ["server", "/data", "--console-address", ":9001"]
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
: "${MINIO_ROOT_USER:?}"
|
||||||
|
: "${MINIO_ROOT_PASSWORD:?}"
|
||||||
|
: "${S3_BUCKET:?}"
|
||||||
|
: "${S3_ACCESS_KEY:?}"
|
||||||
|
: "${S3_SECRET_KEY:?}"
|
||||||
|
|
||||||
|
if [ "$S3_ACCESS_KEY" = "$MINIO_ROOT_USER" ]; then
|
||||||
|
echo "API must use a separate MinIO user" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$S3_SECRET_KEY" = "$MINIO_ROOT_PASSWORD" ]; then
|
||||||
|
echo "API and administrator passwords must differ" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
case "$S3_BUCKET" in
|
||||||
|
""|*[!a-z0-9.-]*)
|
||||||
|
echo "Invalid bucket name" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
mc alias set storage http://minio:9000 \
|
||||||
|
"$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
|
||||||
|
|
||||||
|
mc mb --ignore-existing "storage/$S3_BUCKET"
|
||||||
|
|
||||||
|
mc anonymous set none "storage/$S3_BUCKET"
|
||||||
|
|
||||||
|
cat > /tmp/attachments-policy.json <<EOF
|
||||||
|
{
|
||||||
|
"Version": "2012-10-17",
|
||||||
|
"Statement": [
|
||||||
|
{
|
||||||
|
"Sid": "InspectAttachmentBucket",
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Action": [
|
||||||
|
"s3:GetBucketLocation",
|
||||||
|
"s3:ListBucket"
|
||||||
|
],
|
||||||
|
"Resource": [
|
||||||
|
"arn:aws:s3:::$S3_BUCKET"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Sid": "ManageOriginalAttachments",
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Action": [
|
||||||
|
"s3:GetObject",
|
||||||
|
"s3:PutObject",
|
||||||
|
"s3:DeleteObject",
|
||||||
|
"s3:AbortMultipartUpload",
|
||||||
|
"s3:ListMultipartUploadParts"
|
||||||
|
],
|
||||||
|
"Resource": [
|
||||||
|
"arn:aws:s3:::$S3_BUCKET/originals/*"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
|
||||||
|
POLICY_NAME=messenger-attachments-api
|
||||||
|
|
||||||
|
mc admin policy create \
|
||||||
|
storage "$POLICY_NAME" /tmp/attachments-policy.json
|
||||||
|
|
||||||
|
mc admin user add \
|
||||||
|
storage "$S3_ACCESS_KEY" "$S3_SECRET_KEY" >/dev/null
|
||||||
|
|
||||||
|
mc admin user info --json \
|
||||||
|
storage "$S3_ACCESS_KEY" > /tmp/api-user.json
|
||||||
|
|
||||||
|
if grep -Eq \
|
||||||
|
'"policyName"[[:space:]]*:[[:space:]]*"messenger-attachments-api"' \
|
||||||
|
/tmp/api-user.json
|
||||||
|
then
|
||||||
|
echo "API policy is already attached"
|
||||||
|
else
|
||||||
|
mc admin policy attach \
|
||||||
|
storage "$POLICY_NAME" --user "$S3_ACCESS_KEY"
|
||||||
|
fi
|
||||||
|
|
||||||
|
mc alias set app http://minio:9000 \
|
||||||
|
"$S3_ACCESS_KEY" "$S3_SECRET_KEY" >/dev/null
|
||||||
|
|
||||||
|
mc stat "app/$S3_BUCKET" >/dev/null
|
||||||
|
|
||||||
|
echo "Bootstrap completed: private bucket and API user are ready"
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
: "${S3_BUCKET:?}"
|
||||||
|
: "${S3_ACCESS_KEY:?}"
|
||||||
|
: "${S3_SECRET_KEY:?}"
|
||||||
|
|
||||||
|
mc alias set app http://minio:9000 \
|
||||||
|
"$S3_ACCESS_KEY" "$S3_SECRET_KEY" >/dev/null
|
||||||
|
|
||||||
|
TEST_ID="$(cat /proc/sys/kernel/random/uuid)"
|
||||||
|
OBJECT_KEY="originals/access-check-$TEST_ID.txt"
|
||||||
|
OUTSIDE_KEY="outside-access-check-$TEST_ID.txt"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
mc rm --force \
|
||||||
|
"app/$S3_BUCKET/$OBJECT_KEY" >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
mc rm --force \
|
||||||
|
"app/$S3_BUCKET/$OUTSIDE_KEY" >/dev/null 2>&1 || true
|
||||||
|
}
|
||||||
|
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
printf 'private attachment test' > /tmp/expected.txt
|
||||||
|
|
||||||
|
mc cp /tmp/expected.txt \
|
||||||
|
"app/$S3_BUCKET/$OBJECT_KEY" >/dev/null
|
||||||
|
|
||||||
|
mc cat "app/$S3_BUCKET/$OBJECT_KEY" > /tmp/actual.txt
|
||||||
|
|
||||||
|
cmp /tmp/expected.txt /tmp/actual.txt
|
||||||
|
|
||||||
|
echo "PASS: API credentials can upload and download originals"
|
||||||
|
|
||||||
|
HTTP_CODE="$(
|
||||||
|
curl --silent --show-error \
|
||||||
|
--connect-timeout 3 \
|
||||||
|
--max-time 10 \
|
||||||
|
--output /tmp/anonymous-response.xml \
|
||||||
|
--write-out '%{http_code}' \
|
||||||
|
"http://minio:9000/$S3_BUCKET/$OBJECT_KEY"
|
||||||
|
)"
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" != "403" ]; then
|
||||||
|
echo "FAIL: anonymous GET returned HTTP $HTTP_CODE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! grep -q '<Code>AccessDenied</Code>' \
|
||||||
|
/tmp/anonymous-response.xml
|
||||||
|
then
|
||||||
|
echo "FAIL: expected S3 AccessDenied response" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "PASS: anonymous download of an existing object is denied"
|
||||||
|
|
||||||
|
if mc cp /tmp/expected.txt \
|
||||||
|
"app/$S3_BUCKET/$OUTSIDE_KEY" \
|
||||||
|
> /tmp/outside-result.txt 2>&1
|
||||||
|
then
|
||||||
|
echo "FAIL: API credentials can write outside originals/" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! grep -Eiq \
|
||||||
|
'AccessDenied|Access Denied|Insufficient permissions|not authorized|not allowed' \
|
||||||
|
/tmp/outside-result.txt
|
||||||
|
then
|
||||||
|
echo "FAIL: write failed for a reason other than access denial" >&2
|
||||||
|
cat /tmp/outside-result.txt >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "PASS: writing outside originals/ is denied"
|
||||||
|
|
||||||
|
if mc admin info app > /tmp/admin-result.txt 2>&1
|
||||||
|
then
|
||||||
|
echo "FAIL: API credentials have administrative access" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! grep -Eiq \
|
||||||
|
'AccessDenied|Access Denied|Insufficient permissions|not authorized|not allowed' \
|
||||||
|
/tmp/admin-result.txt
|
||||||
|
then
|
||||||
|
echo "FAIL: admin command failed for an unexpected reason" >&2
|
||||||
|
cat /tmp/admin-result.txt >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "PASS: administrative access is denied"
|
||||||
|
|
||||||
|
mc rm "app/$S3_BUCKET/$OBJECT_KEY" >/dev/null
|
||||||
|
|
||||||
|
echo "PASS: API credentials can delete originals for cleanup"
|
||||||
|
echo "PASS: S3 access checks completed"
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
PYTHONUNBUFFERED=1 \
|
||||||
|
PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
RUN pip install --no-cache-dir \
|
||||||
|
boto3==1.35.99 \
|
||||||
|
"psycopg[binary]==3.2.4"
|
||||||
|
|
||||||
|
COPY reconcile_worker.py /app/reconcile_worker.py
|
||||||
|
|
||||||
|
USER 65534:65534
|
||||||
|
|
||||||
|
ENTRYPOINT ["python", "/app/reconcile_worker.py"]
|
||||||
@@ -0,0 +1,454 @@
|
|||||||
|
import base64
|
||||||
|
from collections import Counter
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
import boto3
|
||||||
|
from botocore.config import Config
|
||||||
|
from botocore.exceptions import ClientError
|
||||||
|
import psycopg
|
||||||
|
from psycopg.rows import dict_row
|
||||||
|
|
||||||
|
|
||||||
|
MAX_BYTES = 10 * 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
|
class Reconciler:
|
||||||
|
def __init__(self, config):
|
||||||
|
self.config = config
|
||||||
|
self.started = time.monotonic()
|
||||||
|
|
||||||
|
self.cutoff = (
|
||||||
|
datetime.now(timezone.utc)
|
||||||
|
- timedelta(seconds=config["grace_seconds"])
|
||||||
|
)
|
||||||
|
|
||||||
|
self.counts = Counter()
|
||||||
|
self.sql_calls = 0
|
||||||
|
self.s3_calls = Counter()
|
||||||
|
self.examples = []
|
||||||
|
self.db = None
|
||||||
|
|
||||||
|
self.s3 = boto3.client(
|
||||||
|
"s3",
|
||||||
|
**config["s3"],
|
||||||
|
config=Config(
|
||||||
|
signature_version="s3v4",
|
||||||
|
s3={"addressing_style": "path"},
|
||||||
|
connect_timeout=3,
|
||||||
|
read_timeout=15,
|
||||||
|
retries={"total_max_attempts": 1},
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def sql(self, statement, params=()):
|
||||||
|
self.sql_calls += 1
|
||||||
|
return self.db.execute(statement, params)
|
||||||
|
|
||||||
|
def call(self, operation, **kwargs):
|
||||||
|
self.s3_calls[operation] += 1
|
||||||
|
|
||||||
|
return getattr(self.s3, operation)(
|
||||||
|
Bucket=self.config["bucket"],
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
def event(self, name, key=None):
|
||||||
|
self.counts[name] += 1
|
||||||
|
|
||||||
|
# Ограничиваем размер подробной части отчёта.
|
||||||
|
if key is not None and len(self.examples) < 100:
|
||||||
|
self.examples.append({
|
||||||
|
"outcome": name,
|
||||||
|
"object_key": key,
|
||||||
|
})
|
||||||
|
|
||||||
|
def deadline(self):
|
||||||
|
elapsed = time.monotonic() - self.started
|
||||||
|
|
||||||
|
if elapsed > self.config["max_seconds"]:
|
||||||
|
raise TimeoutError("Scan deadline reached")
|
||||||
|
|
||||||
|
def lock(self, key):
|
||||||
|
# Тот же ключ блокировки, что используется
|
||||||
|
# AttachmentRepository в Java.
|
||||||
|
row = self.sql(
|
||||||
|
"""
|
||||||
|
SELECT pg_try_advisory_xact_lock(
|
||||||
|
hashtextextended(%s, 0)
|
||||||
|
) AS ok
|
||||||
|
""",
|
||||||
|
("attachment-object:" + key,),
|
||||||
|
).fetchone()
|
||||||
|
|
||||||
|
if not row["ok"]:
|
||||||
|
self.event("busy", key)
|
||||||
|
|
||||||
|
return row["ok"]
|
||||||
|
|
||||||
|
def head(self, key):
|
||||||
|
try:
|
||||||
|
return self.call("head_object", Key=key)
|
||||||
|
|
||||||
|
except ClientError as error:
|
||||||
|
code = error.response.get("Error", {}).get("Code", "")
|
||||||
|
|
||||||
|
if code not in (
|
||||||
|
"404",
|
||||||
|
"NoSuchKey",
|
||||||
|
"NoSuchObject",
|
||||||
|
"NotFound",
|
||||||
|
):
|
||||||
|
raise
|
||||||
|
|
||||||
|
# Проверяем, что доступен сам bucket.
|
||||||
|
# Исчезновение bucket не считаем потерей
|
||||||
|
# отдельного оригинала.
|
||||||
|
self.call("head_bucket")
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
def scan_objects(self):
|
||||||
|
token = None
|
||||||
|
chat_id = self.config.get("chat_id")
|
||||||
|
|
||||||
|
prefix = (
|
||||||
|
f"originals/{chat_id}/"
|
||||||
|
if chat_id
|
||||||
|
else "originals/"
|
||||||
|
)
|
||||||
|
|
||||||
|
while True:
|
||||||
|
self.deadline()
|
||||||
|
|
||||||
|
args = {
|
||||||
|
"Prefix": prefix,
|
||||||
|
"MaxKeys": 500,
|
||||||
|
}
|
||||||
|
|
||||||
|
if token:
|
||||||
|
args["ContinuationToken"] = token
|
||||||
|
|
||||||
|
page = self.call("list_objects_v2", **args)
|
||||||
|
|
||||||
|
for item in page.get("Contents", []):
|
||||||
|
self.deadline()
|
||||||
|
self.event("objects_scanned")
|
||||||
|
|
||||||
|
key = item["Key"]
|
||||||
|
|
||||||
|
if item["LastModified"] >= self.cutoff:
|
||||||
|
self.event("fresh_objects")
|
||||||
|
continue
|
||||||
|
|
||||||
|
with self.db.transaction():
|
||||||
|
if not self.lock(key):
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Проверяем всю таблицу, даже если проход
|
||||||
|
# ограничен одним тестовым чатом.
|
||||||
|
row = self.sql(
|
||||||
|
"""
|
||||||
|
SELECT id
|
||||||
|
FROM attachments
|
||||||
|
WHERE object_key = %s
|
||||||
|
""",
|
||||||
|
(key,),
|
||||||
|
).fetchone()
|
||||||
|
|
||||||
|
if row:
|
||||||
|
self.event("referenced_objects")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Повторная проверка после блокировки.
|
||||||
|
current = self.head(key)
|
||||||
|
|
||||||
|
if current is None:
|
||||||
|
self.event("already_absent")
|
||||||
|
|
||||||
|
elif current["LastModified"] >= self.cutoff:
|
||||||
|
self.event("fresh_objects")
|
||||||
|
|
||||||
|
elif self.config["apply"]:
|
||||||
|
self.call("delete_object", Key=key)
|
||||||
|
self.event("deleted_orphans", key)
|
||||||
|
|
||||||
|
else:
|
||||||
|
self.event("would_delete_orphans", key)
|
||||||
|
|
||||||
|
if not page.get("IsTruncated"):
|
||||||
|
break
|
||||||
|
|
||||||
|
token = page["NextContinuationToken"]
|
||||||
|
|
||||||
|
def backup(self, row):
|
||||||
|
checksum = row["sha256"].strip()
|
||||||
|
|
||||||
|
if not re.fullmatch("[a-f0-9]{64}", checksum):
|
||||||
|
raise ValueError("Invalid metadata checksum")
|
||||||
|
|
||||||
|
# Имя копии определяется SHA-256, а не filename.
|
||||||
|
path = Path("/backups") / (checksum + ".bin")
|
||||||
|
|
||||||
|
if not path.exists():
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Ограничиваем чтение даже для ошибочной копии.
|
||||||
|
with path.open("rb") as stream:
|
||||||
|
data = stream.read(MAX_BYTES + 1)
|
||||||
|
|
||||||
|
valid = (
|
||||||
|
len(data) == row["size_bytes"]
|
||||||
|
and hashlib.sha256(data).hexdigest() == checksum
|
||||||
|
)
|
||||||
|
|
||||||
|
if not valid:
|
||||||
|
self.event("invalid_backups", row["object_key"])
|
||||||
|
return None
|
||||||
|
|
||||||
|
return data
|
||||||
|
|
||||||
|
def check_attachment(self, attachment_id):
|
||||||
|
# В текущем протоколе object_key неизменяем.
|
||||||
|
initial = self.sql(
|
||||||
|
"""
|
||||||
|
SELECT object_key
|
||||||
|
FROM attachments
|
||||||
|
WHERE id = %s
|
||||||
|
""",
|
||||||
|
(attachment_id,),
|
||||||
|
).fetchone()
|
||||||
|
|
||||||
|
if not initial:
|
||||||
|
return
|
||||||
|
|
||||||
|
key = initial["object_key"]
|
||||||
|
|
||||||
|
with self.db.transaction():
|
||||||
|
if not self.lock(key):
|
||||||
|
return
|
||||||
|
|
||||||
|
row = self.sql(
|
||||||
|
"""
|
||||||
|
SELECT *
|
||||||
|
FROM attachments
|
||||||
|
WHERE id = %s
|
||||||
|
FOR SHARE
|
||||||
|
""",
|
||||||
|
(attachment_id,),
|
||||||
|
).fetchone()
|
||||||
|
|
||||||
|
if not row:
|
||||||
|
return
|
||||||
|
|
||||||
|
if (
|
||||||
|
row["object_key"] != key
|
||||||
|
or not key.startswith("originals/")
|
||||||
|
):
|
||||||
|
raise ValueError("Unsupported object key")
|
||||||
|
|
||||||
|
if row["created_at"] >= self.cutoff:
|
||||||
|
self.event("fresh_metadata")
|
||||||
|
return
|
||||||
|
|
||||||
|
current = self.head(key)
|
||||||
|
|
||||||
|
if current is not None:
|
||||||
|
if current["ContentLength"] == row["size_bytes"]:
|
||||||
|
self.event("present_metadata")
|
||||||
|
else:
|
||||||
|
self.event("size_mismatches", key)
|
||||||
|
|
||||||
|
return
|
||||||
|
|
||||||
|
self.event("missing_objects", key)
|
||||||
|
|
||||||
|
data = self.backup(row)
|
||||||
|
|
||||||
|
if data is None:
|
||||||
|
# Метаданные и сообщения сохраняем.
|
||||||
|
self.event("unresolved_missing", key)
|
||||||
|
return
|
||||||
|
|
||||||
|
if not self.config["apply"]:
|
||||||
|
self.event("would_restore", key)
|
||||||
|
return
|
||||||
|
|
||||||
|
# Восстанавливаем отсутствующий объект,
|
||||||
|
# удерживая ту же блокировку, что и API.
|
||||||
|
self.call(
|
||||||
|
"put_object",
|
||||||
|
Key=key,
|
||||||
|
Body=data,
|
||||||
|
ContentType=row["content_type"],
|
||||||
|
ContentMD5=base64.b64encode(
|
||||||
|
hashlib.md5(data).digest()
|
||||||
|
).decode("ascii"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Проверяем фактические байты после записи.
|
||||||
|
response = self.call("get_object", Key=key)
|
||||||
|
|
||||||
|
with response["Body"] as stream:
|
||||||
|
actual = stream.read(MAX_BYTES + 1)
|
||||||
|
|
||||||
|
valid = (
|
||||||
|
len(actual) == row["size_bytes"]
|
||||||
|
and hashlib.sha256(actual).hexdigest()
|
||||||
|
== row["sha256"].strip()
|
||||||
|
)
|
||||||
|
|
||||||
|
if not valid:
|
||||||
|
raise ValueError(
|
||||||
|
"Restored object verification failed"
|
||||||
|
)
|
||||||
|
|
||||||
|
self.event("restored", key)
|
||||||
|
|
||||||
|
def scan_metadata(self):
|
||||||
|
cursor = None
|
||||||
|
chat_id = self.config.get("chat_id")
|
||||||
|
|
||||||
|
while True:
|
||||||
|
self.deadline()
|
||||||
|
|
||||||
|
rows = self.sql(
|
||||||
|
"""
|
||||||
|
SELECT id
|
||||||
|
FROM attachments
|
||||||
|
WHERE (
|
||||||
|
%s::uuid IS NULL
|
||||||
|
OR chat_id = %s::uuid
|
||||||
|
)
|
||||||
|
AND (
|
||||||
|
%s::uuid IS NULL
|
||||||
|
OR id > %s::uuid
|
||||||
|
)
|
||||||
|
ORDER BY id
|
||||||
|
LIMIT 500
|
||||||
|
""",
|
||||||
|
(chat_id, chat_id, cursor, cursor),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
if not rows:
|
||||||
|
return
|
||||||
|
|
||||||
|
for row in rows:
|
||||||
|
self.deadline()
|
||||||
|
self.event("metadata_scanned")
|
||||||
|
self.check_attachment(row["id"])
|
||||||
|
|
||||||
|
cursor = rows[-1]["id"]
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
status = "completed"
|
||||||
|
error_type = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
with psycopg.connect(
|
||||||
|
**self.config["db"],
|
||||||
|
autocommit=True,
|
||||||
|
row_factory=dict_row,
|
||||||
|
connect_timeout=5,
|
||||||
|
options=(
|
||||||
|
"-c statement_timeout=10000 "
|
||||||
|
"-c idle_in_transaction_session_timeout=120000"
|
||||||
|
),
|
||||||
|
) as db:
|
||||||
|
self.db = db
|
||||||
|
|
||||||
|
# Только один полный проход этой утилиты
|
||||||
|
# одновременно в данной БД.
|
||||||
|
locked = self.sql(
|
||||||
|
"""
|
||||||
|
SELECT pg_try_advisory_lock(
|
||||||
|
hashtextextended(
|
||||||
|
'attachment-reconciler:v1',
|
||||||
|
0
|
||||||
|
)
|
||||||
|
) AS ok
|
||||||
|
"""
|
||||||
|
).fetchone()["ok"]
|
||||||
|
|
||||||
|
if not locked:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Another reconciler is running"
|
||||||
|
)
|
||||||
|
|
||||||
|
self.call("head_bucket")
|
||||||
|
|
||||||
|
self.scan_objects()
|
||||||
|
self.scan_metadata()
|
||||||
|
|
||||||
|
# Закрытие соединения освобождает session lock.
|
||||||
|
# Блокировки объектов освобождаются
|
||||||
|
# при завершении отдельных транзакций.
|
||||||
|
|
||||||
|
except Exception as error:
|
||||||
|
status = "failed"
|
||||||
|
error_type = type(error).__name__
|
||||||
|
|
||||||
|
finally:
|
||||||
|
self.s3.close()
|
||||||
|
|
||||||
|
attention_counters = (
|
||||||
|
"unresolved_missing",
|
||||||
|
"size_mismatches",
|
||||||
|
"invalid_backups",
|
||||||
|
"busy",
|
||||||
|
)
|
||||||
|
|
||||||
|
if status == "completed" and any(
|
||||||
|
self.counts[name] for name in attention_counters
|
||||||
|
):
|
||||||
|
status = "needs_attention"
|
||||||
|
|
||||||
|
report = {
|
||||||
|
"status": status,
|
||||||
|
"mode": (
|
||||||
|
"apply"
|
||||||
|
if self.config["apply"]
|
||||||
|
else "dry-run"
|
||||||
|
),
|
||||||
|
"chat_id": self.config.get("chat_id"),
|
||||||
|
"cutoff": self.cutoff.isoformat(),
|
||||||
|
"elapsed_seconds": round(
|
||||||
|
time.monotonic() - self.started,
|
||||||
|
3,
|
||||||
|
),
|
||||||
|
"counts": dict(self.counts),
|
||||||
|
"sql_statement_calls": self.sql_calls,
|
||||||
|
"s3_operation_calls": dict(self.s3_calls),
|
||||||
|
"examples": self.examples,
|
||||||
|
"examples_limit": 100,
|
||||||
|
"error_type": error_type,
|
||||||
|
}
|
||||||
|
|
||||||
|
print(json.dumps(
|
||||||
|
report,
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
))
|
||||||
|
|
||||||
|
return {
|
||||||
|
"completed": 0,
|
||||||
|
"needs_attention": 2,
|
||||||
|
"failed": 1,
|
||||||
|
}[status]
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
config = json.load(sys.stdin)
|
||||||
|
|
||||||
|
if (
|
||||||
|
config["grace_seconds"] < 60
|
||||||
|
or config["max_seconds"] < 1
|
||||||
|
):
|
||||||
|
raise SystemExit("Invalid scan limits")
|
||||||
|
|
||||||
|
raise SystemExit(Reconciler(config).run())
|
||||||
+6
-1
@@ -1,4 +1,3 @@
|
|||||||
|
|
||||||
<?xml version="1.0" encoding="UTF-8"?>
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
@@ -98,6 +97,12 @@
|
|||||||
<scope>test</scope>
|
<scope>test</scope>
|
||||||
</dependency>
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>io.minio</groupId>
|
||||||
|
<artifactId>minio</artifactId>
|
||||||
|
<version>8.5.17</version>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
</dependencies>
|
</dependencies>
|
||||||
|
|
||||||
<build>
|
<build>
|
||||||
|
|||||||
@@ -39,6 +39,10 @@ public class GlobalExceptionHandler {
|
|||||||
ResponseEntity.BodyBuilder builder =
|
ResponseEntity.BodyBuilder builder =
|
||||||
ResponseEntity.status(e.getStatus());
|
ResponseEntity.status(e.getStatus());
|
||||||
|
|
||||||
|
if (e.getStatus() == HttpStatus.TOO_MANY_REQUESTS) {
|
||||||
|
builder.header("Retry-After", "1");
|
||||||
|
}
|
||||||
|
|
||||||
if (e.getStatus() == HttpStatus.UNAUTHORIZED) {
|
if (e.getStatus() == HttpStatus.UNAUTHORIZED) {
|
||||||
String path = request.getRequestURI();
|
String path = request.getRequestURI();
|
||||||
|
|
||||||
@@ -111,7 +115,7 @@ public class GlobalExceptionHandler {
|
|||||||
return ResponseEntity
|
return ResponseEntity
|
||||||
.status(HttpStatus.SERVICE_UNAVAILABLE)
|
.status(HttpStatus.SERVICE_UNAVAILABLE)
|
||||||
.body(build(
|
.body(build(
|
||||||
"service_unavailable",
|
"unavailable",
|
||||||
"Сервис временно недоступен"
|
"Сервис временно недоступен"
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
@@ -123,7 +127,7 @@ public class GlobalExceptionHandler {
|
|||||||
int status = e.getStatusCode().value();
|
int status = e.getStatusCode().value();
|
||||||
|
|
||||||
String code = status == 503
|
String code = status == 503
|
||||||
? "service_unavailable"
|
? "unavailable"
|
||||||
: "request_failed";
|
: "request_failed";
|
||||||
|
|
||||||
String message = status == 503
|
String message = status == 503
|
||||||
|
|||||||
@@ -0,0 +1,9 @@
|
|||||||
|
package com.vnp.demo.config;
|
||||||
|
|
||||||
|
import org.springframework.context.annotation.Configuration;
|
||||||
|
import org.springframework.scheduling.annotation.EnableScheduling;
|
||||||
|
|
||||||
|
@Configuration
|
||||||
|
@EnableScheduling
|
||||||
|
public class SchedulingConfig {
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
package com.vnp.demo.config;
|
||||||
|
|
||||||
|
import org.apache.coyote.http11.AbstractHttp11Protocol;
|
||||||
|
import org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory;
|
||||||
|
import org.springframework.boot.web.server.WebServerFactoryCustomizer;
|
||||||
|
import org.springframework.context.annotation.Bean;
|
||||||
|
import org.springframework.context.annotation.Configuration;
|
||||||
|
|
||||||
|
@Configuration
|
||||||
|
public class UploadTimeoutConfig {
|
||||||
|
|
||||||
|
@Bean
|
||||||
|
public WebServerFactoryCustomizer<TomcatServletWebServerFactory>
|
||||||
|
uploadTimeoutCustomizer() {
|
||||||
|
|
||||||
|
return factory -> factory.addConnectorCustomizers(connector -> {
|
||||||
|
if (connector.getProtocolHandler()
|
||||||
|
instanceof AbstractHttp11Protocol<?> protocol) {
|
||||||
|
|
||||||
|
protocol.setDisableUploadTimeout(false);
|
||||||
|
protocol.setConnectionUploadTimeout(15_000);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
package com.vnp.demo.controllers;
|
||||||
|
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.util.UUID;
|
||||||
|
|
||||||
|
import org.springframework.http.ContentDisposition;
|
||||||
|
import org.springframework.http.HttpHeaders;
|
||||||
|
import org.springframework.http.HttpStatus;
|
||||||
|
import org.springframework.http.MediaType;
|
||||||
|
import org.springframework.http.ResponseEntity;
|
||||||
|
import org.springframework.security.core.annotation.AuthenticationPrincipal;
|
||||||
|
import org.springframework.web.bind.annotation.GetMapping;
|
||||||
|
import org.springframework.web.bind.annotation.PathVariable;
|
||||||
|
import org.springframework.web.bind.annotation.PostMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestParam;
|
||||||
|
import org.springframework.web.bind.annotation.ResponseStatus;
|
||||||
|
import org.springframework.web.bind.annotation.RestController;
|
||||||
|
|
||||||
|
import com.vnp.demo.BusinessException;
|
||||||
|
import com.vnp.demo.dto.AttachmentResponse;
|
||||||
|
import com.vnp.demo.service.AttachmentService;
|
||||||
|
|
||||||
|
import jakarta.servlet.http.HttpServletRequest;
|
||||||
|
|
||||||
|
@RestController
|
||||||
|
@RequestMapping("/api/v1")
|
||||||
|
public class AttachmentController {
|
||||||
|
|
||||||
|
private final AttachmentService attachmentService;
|
||||||
|
|
||||||
|
public AttachmentController(
|
||||||
|
AttachmentService attachmentService
|
||||||
|
) {
|
||||||
|
this.attachmentService = attachmentService;
|
||||||
|
}
|
||||||
|
|
||||||
|
@PostMapping("/chats/{chat_id}/attachments")
|
||||||
|
@ResponseStatus(HttpStatus.CREATED)
|
||||||
|
public AttachmentResponse upload(
|
||||||
|
@AuthenticationPrincipal UUID userId,
|
||||||
|
@PathVariable("chat_id") UUID chatId,
|
||||||
|
@RequestParam(value = "filename", required = false)
|
||||||
|
String filename,
|
||||||
|
HttpServletRequest request
|
||||||
|
) {
|
||||||
|
try {
|
||||||
|
return attachmentService.upload(
|
||||||
|
userId,
|
||||||
|
chatId,
|
||||||
|
filename,
|
||||||
|
request.getContentType(),
|
||||||
|
request.getContentLengthLong(),
|
||||||
|
request.getInputStream()
|
||||||
|
);
|
||||||
|
} catch (IOException e) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Не удалось прочитать тело запроса"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@GetMapping("/attachments/{id}")
|
||||||
|
public AttachmentResponse metadata(
|
||||||
|
@AuthenticationPrincipal UUID userId,
|
||||||
|
@PathVariable("id") UUID id
|
||||||
|
) {
|
||||||
|
return attachmentService.getMetadata(userId, id);
|
||||||
|
}
|
||||||
|
|
||||||
|
@GetMapping("/attachments/{id}/content")
|
||||||
|
public ResponseEntity<byte[]> content(
|
||||||
|
@AuthenticationPrincipal UUID userId,
|
||||||
|
@PathVariable("id") UUID id,
|
||||||
|
@RequestParam(defaultValue = "original")
|
||||||
|
String variant
|
||||||
|
) {
|
||||||
|
AttachmentService.Download download =
|
||||||
|
attachmentService.download(userId, id, variant);
|
||||||
|
|
||||||
|
String disposition = ContentDisposition.attachment()
|
||||||
|
.filename(
|
||||||
|
download.filename(),
|
||||||
|
StandardCharsets.UTF_8
|
||||||
|
)
|
||||||
|
.build()
|
||||||
|
.toString();
|
||||||
|
|
||||||
|
return ResponseEntity.ok()
|
||||||
|
.contentType(
|
||||||
|
MediaType.parseMediaType(download.contentType())
|
||||||
|
)
|
||||||
|
.contentLength(download.bytes().length)
|
||||||
|
.header(HttpHeaders.CONTENT_DISPOSITION, disposition)
|
||||||
|
.header(HttpHeaders.CACHE_CONTROL, "private, no-store")
|
||||||
|
.header("X-Content-Type-Options", "nosniff")
|
||||||
|
.body(download.bytes());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,6 +8,7 @@ import org.springframework.jdbc.core.JdbcTemplate;
|
|||||||
import org.springframework.web.bind.annotation.GetMapping;
|
import org.springframework.web.bind.annotation.GetMapping;
|
||||||
import org.springframework.web.bind.annotation.RequestMapping;
|
import org.springframework.web.bind.annotation.RequestMapping;
|
||||||
import org.springframework.web.bind.annotation.RestController;
|
import org.springframework.web.bind.annotation.RestController;
|
||||||
|
import com.vnp.demo.service.S3StorageService;
|
||||||
|
|
||||||
import com.vnp.demo.BusinessException;
|
import com.vnp.demo.BusinessException;
|
||||||
import com.vnp.demo.dto.HealthResponse;
|
import com.vnp.demo.dto.HealthResponse;
|
||||||
@@ -18,8 +19,17 @@ public class HealthController {
|
|||||||
private final JdbcTemplate jdbc;
|
private final JdbcTemplate jdbc;
|
||||||
private final StringRedisTemplate redis;
|
private final StringRedisTemplate redis;
|
||||||
private final ApplicationAvailability availability;
|
private final ApplicationAvailability availability;
|
||||||
public HealthController(JdbcTemplate jdbc, StringRedisTemplate redis, ApplicationAvailability availability) {
|
private final S3StorageService storage;
|
||||||
this.jdbc = jdbc; this.redis = redis; this.availability = availability;
|
public HealthController(
|
||||||
|
JdbcTemplate jdbc,
|
||||||
|
StringRedisTemplate redis,
|
||||||
|
ApplicationAvailability availability,
|
||||||
|
S3StorageService storage
|
||||||
|
) {
|
||||||
|
this.jdbc = jdbc;
|
||||||
|
this.redis = redis;
|
||||||
|
this.availability = availability;
|
||||||
|
this.storage = storage;
|
||||||
}
|
}
|
||||||
@GetMapping("/live")
|
@GetMapping("/live")
|
||||||
public HealthResponse live() { return HealthResponse.ok(); }
|
public HealthResponse live() { return HealthResponse.ok(); }
|
||||||
@@ -30,9 +40,10 @@ public class HealthController {
|
|||||||
if (!Integer.valueOf(1).equals(jdbc.queryForObject("SELECT 1", Integer.class))) throw new IllegalStateException();
|
if (!Integer.valueOf(1).equals(jdbc.queryForObject("SELECT 1", Integer.class))) throw new IllegalStateException();
|
||||||
String pong = redis.execute((org.springframework.data.redis.core.RedisCallback<String>) c -> c.ping());
|
String pong = redis.execute((org.springframework.data.redis.core.RedisCallback<String>) c -> c.ping());
|
||||||
if (!"PONG".equals(pong)) throw new IllegalStateException();
|
if (!"PONG".equals(pong)) throw new IllegalStateException();
|
||||||
|
storage.checkReady();
|
||||||
return HealthResponse.ok();
|
return HealthResponse.ok();
|
||||||
} catch (Exception ex) {
|
} catch (Exception ex) {
|
||||||
throw new BusinessException("unavailable", "Сервис не готов", HttpStatus.SERVICE_UNAVAILABLE);
|
throw new BusinessException("service_unavailable", "Сервис не готов", HttpStatus.SERVICE_UNAVAILABLE);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
package com.vnp.demo.dto;
|
||||||
|
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.UUID;
|
||||||
|
|
||||||
|
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||||
|
import com.vnp.demo.models.Attachment;
|
||||||
|
|
||||||
|
public record AttachmentResponse(
|
||||||
|
UUID id,
|
||||||
|
|
||||||
|
@JsonProperty("chat_id")
|
||||||
|
UUID chatId,
|
||||||
|
|
||||||
|
@JsonProperty("owner_id")
|
||||||
|
UUID ownerId,
|
||||||
|
|
||||||
|
String filename,
|
||||||
|
|
||||||
|
@JsonProperty("content_type")
|
||||||
|
String contentType,
|
||||||
|
|
||||||
|
@JsonProperty("size_bytes")
|
||||||
|
long sizeBytes,
|
||||||
|
|
||||||
|
String sha256,
|
||||||
|
String status,
|
||||||
|
List<Object> variants,
|
||||||
|
|
||||||
|
@JsonProperty("error_code")
|
||||||
|
String errorCode,
|
||||||
|
|
||||||
|
@JsonProperty("created_at")
|
||||||
|
Instant createdAt,
|
||||||
|
|
||||||
|
@JsonProperty("updated_at")
|
||||||
|
Instant updatedAt
|
||||||
|
) {
|
||||||
|
public static AttachmentResponse from(Attachment attachment) {
|
||||||
|
return new AttachmentResponse(
|
||||||
|
attachment.id(),
|
||||||
|
attachment.chatId(),
|
||||||
|
attachment.ownerId(),
|
||||||
|
attachment.filename(),
|
||||||
|
attachment.contentType(),
|
||||||
|
attachment.sizeBytes(),
|
||||||
|
attachment.sha256(),
|
||||||
|
attachment.status(),
|
||||||
|
List.of(),
|
||||||
|
null,
|
||||||
|
attachment.createdAt(),
|
||||||
|
attachment.updatedAt()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,8 +1,53 @@
|
|||||||
package com.vnp.demo.dto;
|
package com.vnp.demo.dto;
|
||||||
|
|
||||||
import jakarta.validation.constraints.NotBlank;
|
import java.util.List;
|
||||||
|
import java.util.UUID;
|
||||||
|
|
||||||
public record MessageCreateRequest(
|
import com.fasterxml.jackson.annotation.JsonSetter;
|
||||||
@NotBlank String text
|
import com.fasterxml.jackson.annotation.Nulls;
|
||||||
) {
|
|
||||||
|
public class MessageCreateRequest {
|
||||||
|
|
||||||
|
private String text;
|
||||||
|
private List<UUID> attachmentIds = List.of();
|
||||||
|
|
||||||
|
public MessageCreateRequest() {
|
||||||
|
}
|
||||||
|
|
||||||
|
public MessageCreateRequest(String text) {
|
||||||
|
this.text = text;
|
||||||
|
}
|
||||||
|
|
||||||
|
public MessageCreateRequest(
|
||||||
|
String text,
|
||||||
|
List<UUID> attachmentIds
|
||||||
|
) {
|
||||||
|
this.text = text;
|
||||||
|
this.attachmentIds = attachmentIds;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getText() {
|
||||||
|
return text;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setText(String text) {
|
||||||
|
this.text = text;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String text() {
|
||||||
|
return text;
|
||||||
|
}
|
||||||
|
|
||||||
|
public List<UUID> attachmentIds() {
|
||||||
|
return attachmentIds;
|
||||||
|
}
|
||||||
|
|
||||||
|
@JsonSetter(
|
||||||
|
value = "attachment_ids",
|
||||||
|
nulls = Nulls.FAIL,
|
||||||
|
contentNulls = Nulls.FAIL
|
||||||
|
)
|
||||||
|
public void setAttachmentIds(List<UUID> attachmentIds) {
|
||||||
|
this.attachmentIds = attachmentIds;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
@@ -1,21 +1,26 @@
|
|||||||
package com.vnp.demo.dto;
|
package com.vnp.demo.dto;
|
||||||
|
|
||||||
import java.time.Instant;
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
import java.util.UUID;
|
import java.util.UUID;
|
||||||
|
|
||||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||||
|
|
||||||
public record MessageResponse(
|
public record MessageResponse(
|
||||||
UUID id,
|
UUID id,
|
||||||
|
|
||||||
@JsonProperty("chat_id")
|
@JsonProperty("chat_id")
|
||||||
UUID chat_id,
|
UUID chatId,
|
||||||
|
|
||||||
@JsonProperty("sender_id")
|
@JsonProperty("sender_id")
|
||||||
UUID sender_id,
|
UUID senderId,
|
||||||
|
|
||||||
String text,
|
String text,
|
||||||
|
|
||||||
@JsonProperty("created_at")
|
@JsonProperty("created_at")
|
||||||
Instant createdAt
|
Instant createdAt,
|
||||||
){}
|
|
||||||
|
@JsonProperty("attachment_ids")
|
||||||
|
List<UUID> attachmentIds
|
||||||
|
) {
|
||||||
|
}
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
package com.vnp.demo.models;
|
||||||
|
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.UUID;
|
||||||
|
|
||||||
|
public record Attachment(
|
||||||
|
UUID id,
|
||||||
|
UUID chatId,
|
||||||
|
UUID ownerId,
|
||||||
|
String filename,
|
||||||
|
String contentType,
|
||||||
|
long sizeBytes,
|
||||||
|
String sha256,
|
||||||
|
String objectKey,
|
||||||
|
String status,
|
||||||
|
Instant createdAt,
|
||||||
|
Instant updatedAt
|
||||||
|
) {
|
||||||
|
}
|
||||||
@@ -1,14 +1,19 @@
|
|||||||
package com.vnp.demo.models;
|
package com.vnp.demo.models;
|
||||||
|
|
||||||
import java.time.Instant;
|
import java.time.Instant;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
import java.util.UUID;
|
import java.util.UUID;
|
||||||
|
|
||||||
|
import jakarta.persistence.CollectionTable;
|
||||||
import jakarta.persistence.Column;
|
import jakarta.persistence.Column;
|
||||||
|
import jakarta.persistence.ElementCollection;
|
||||||
import jakarta.persistence.Entity;
|
import jakarta.persistence.Entity;
|
||||||
import jakarta.persistence.FetchType;
|
import jakarta.persistence.FetchType;
|
||||||
import jakarta.persistence.Id;
|
import jakarta.persistence.Id;
|
||||||
import jakarta.persistence.JoinColumn;
|
import jakarta.persistence.JoinColumn;
|
||||||
import jakarta.persistence.ManyToOne;
|
import jakarta.persistence.ManyToOne;
|
||||||
|
import jakarta.persistence.OrderColumn;
|
||||||
import jakarta.persistence.Table;
|
import jakarta.persistence.Table;
|
||||||
|
|
||||||
@Entity
|
@Entity
|
||||||
@@ -32,13 +37,32 @@ public class Message {
|
|||||||
@Column(name = "created_at", nullable = false)
|
@Column(name = "created_at", nullable = false)
|
||||||
private Instant createdAt = Instant.now();
|
private Instant createdAt = Instant.now();
|
||||||
|
|
||||||
|
@ElementCollection(fetch = FetchType.LAZY)
|
||||||
|
@CollectionTable(
|
||||||
|
name = "message_attachments",
|
||||||
|
joinColumns = @JoinColumn(name = "message_id")
|
||||||
|
)
|
||||||
|
@Column(name = "attachment_id", nullable = false)
|
||||||
|
@OrderColumn(name = "position")
|
||||||
|
private List<UUID> attachmentIds = new ArrayList<>();
|
||||||
|
|
||||||
protected Message() {
|
protected Message() {
|
||||||
}
|
}
|
||||||
|
|
||||||
public Message(Chat chat, User sender, String text) {
|
public Message(Chat chat, User sender, String text) {
|
||||||
|
this(chat, sender, text, List.of());
|
||||||
|
}
|
||||||
|
|
||||||
|
public Message(
|
||||||
|
Chat chat,
|
||||||
|
User sender,
|
||||||
|
String text,
|
||||||
|
List<UUID> attachmentIds
|
||||||
|
) {
|
||||||
this.chat = chat;
|
this.chat = chat;
|
||||||
this.sender = sender;
|
this.sender = sender;
|
||||||
this.text = text;
|
this.text = text;
|
||||||
|
this.attachmentIds = new ArrayList<>(attachmentIds);
|
||||||
}
|
}
|
||||||
|
|
||||||
public UUID getId() {
|
public UUID getId() {
|
||||||
@@ -60,4 +84,8 @@ public class Message {
|
|||||||
public Instant getCreatedAt() {
|
public Instant getCreatedAt() {
|
||||||
return createdAt;
|
return createdAt;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public List<UUID> getAttachmentIds() {
|
||||||
|
return attachmentIds;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
package com.vnp.demo.repositories;
|
||||||
|
|
||||||
|
import java.sql.ResultSet;
|
||||||
|
import java.sql.SQLException;
|
||||||
|
import java.sql.Timestamp;
|
||||||
|
import java.util.Optional;
|
||||||
|
import java.util.UUID;
|
||||||
|
|
||||||
|
import org.springframework.jdbc.core.JdbcTemplate;
|
||||||
|
import org.springframework.stereotype.Repository;
|
||||||
|
|
||||||
|
import com.vnp.demo.models.Attachment;
|
||||||
|
|
||||||
|
@Repository
|
||||||
|
public class AttachmentRepository {
|
||||||
|
|
||||||
|
private final JdbcTemplate jdbc;
|
||||||
|
|
||||||
|
public AttachmentRepository(JdbcTemplate jdbc) {
|
||||||
|
this.jdbc = jdbc;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void save(Attachment attachment) {
|
||||||
|
jdbc.update(
|
||||||
|
"""
|
||||||
|
INSERT INTO attachments (
|
||||||
|
id,
|
||||||
|
chat_id,
|
||||||
|
owner_id,
|
||||||
|
filename,
|
||||||
|
content_type,
|
||||||
|
size_bytes,
|
||||||
|
sha256,
|
||||||
|
object_key,
|
||||||
|
status,
|
||||||
|
created_at,
|
||||||
|
updated_at
|
||||||
|
)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||||
|
""",
|
||||||
|
attachment.id(),
|
||||||
|
attachment.chatId(),
|
||||||
|
attachment.ownerId(),
|
||||||
|
attachment.filename(),
|
||||||
|
attachment.contentType(),
|
||||||
|
attachment.sizeBytes(),
|
||||||
|
attachment.sha256(),
|
||||||
|
attachment.objectKey(),
|
||||||
|
attachment.status(),
|
||||||
|
Timestamp.from(attachment.createdAt()),
|
||||||
|
Timestamp.from(attachment.updatedAt())
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Optional<Attachment> findById(UUID id) {
|
||||||
|
return jdbc.query(
|
||||||
|
"SELECT * FROM attachments WHERE id = ?",
|
||||||
|
this::mapRow,
|
||||||
|
id
|
||||||
|
).stream().findFirst();
|
||||||
|
}
|
||||||
|
|
||||||
|
private Attachment mapRow(ResultSet rs, int rowNum)
|
||||||
|
throws SQLException {
|
||||||
|
|
||||||
|
return new Attachment(
|
||||||
|
rs.getObject("id", UUID.class),
|
||||||
|
rs.getObject("chat_id", UUID.class),
|
||||||
|
rs.getObject("owner_id", UUID.class),
|
||||||
|
rs.getString("filename"),
|
||||||
|
rs.getString("content_type"),
|
||||||
|
rs.getLong("size_bytes"),
|
||||||
|
rs.getString("sha256"),
|
||||||
|
rs.getString("object_key"),
|
||||||
|
rs.getString("status"),
|
||||||
|
rs.getTimestamp("created_at").toInstant(),
|
||||||
|
rs.getTimestamp("updated_at").toInstant()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
public boolean existsByObjectKey(String objectKey) {
|
||||||
|
Boolean exists = jdbc.queryForObject(
|
||||||
|
"""
|
||||||
|
SELECT EXISTS (
|
||||||
|
SELECT 1
|
||||||
|
FROM attachments
|
||||||
|
WHERE object_key = ?
|
||||||
|
)
|
||||||
|
""",
|
||||||
|
Boolean.class,
|
||||||
|
objectKey
|
||||||
|
);
|
||||||
|
|
||||||
|
return Boolean.TRUE.equals(exists);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void lockObject(String objectKey) {
|
||||||
|
jdbc.query(
|
||||||
|
"""
|
||||||
|
SELECT pg_advisory_xact_lock(
|
||||||
|
hashtextextended(?, 0)
|
||||||
|
)
|
||||||
|
""",
|
||||||
|
(rs, rowNum) -> Boolean.TRUE,
|
||||||
|
"attachment-object:" + objectKey
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
public boolean tryLockObject(String objectKey) {
|
||||||
|
Boolean acquired = jdbc.queryForObject(
|
||||||
|
"""
|
||||||
|
SELECT pg_try_advisory_xact_lock(
|
||||||
|
hashtextextended(?, 0)
|
||||||
|
)
|
||||||
|
""",
|
||||||
|
Boolean.class,
|
||||||
|
"attachment-object:" + objectKey
|
||||||
|
);
|
||||||
|
|
||||||
|
return Boolean.TRUE.equals(acquired);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
package com.vnp.demo.service;
|
||||||
|
|
||||||
|
import java.time.Instant;
|
||||||
|
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
|
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||||
|
import org.springframework.scheduling.annotation.Scheduled;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import com.vnp.demo.service.AttachmentPersistenceService.CleanupResult;
|
||||||
|
|
||||||
|
import io.minio.Result;
|
||||||
|
import io.minio.messages.Item;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@ConditionalOnProperty(
|
||||||
|
name = "app.cleanup.enabled",
|
||||||
|
havingValue = "true",
|
||||||
|
matchIfMissing = true
|
||||||
|
)
|
||||||
|
public class AttachmentCleanupJob {
|
||||||
|
|
||||||
|
private static final Logger log =
|
||||||
|
LoggerFactory.getLogger(AttachmentCleanupJob.class);
|
||||||
|
|
||||||
|
private final S3StorageService storage;
|
||||||
|
private final AttachmentPersistenceService persistence;
|
||||||
|
private final long graceSeconds;
|
||||||
|
|
||||||
|
public AttachmentCleanupJob(
|
||||||
|
S3StorageService storage,
|
||||||
|
AttachmentPersistenceService persistence,
|
||||||
|
@Value("${app.cleanup.grace-seconds:3600}")
|
||||||
|
long graceSeconds
|
||||||
|
) {
|
||||||
|
if (graceSeconds < 60) {
|
||||||
|
throw new IllegalArgumentException(
|
||||||
|
"Grace period должен быть не меньше 60 секунд"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
this.storage = storage;
|
||||||
|
this.persistence = persistence;
|
||||||
|
this.graceSeconds = graceSeconds;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Scheduled(
|
||||||
|
initialDelayString =
|
||||||
|
"${app.cleanup.initial-delay-ms:60000}",
|
||||||
|
fixedDelayString =
|
||||||
|
"${app.cleanup.interval-ms:300000}"
|
||||||
|
)
|
||||||
|
public void cleanup() {
|
||||||
|
Instant cutoff = Instant.now()
|
||||||
|
.minusSeconds(graceSeconds);
|
||||||
|
|
||||||
|
long scanned = 0;
|
||||||
|
long candidates = 0;
|
||||||
|
long deleted = 0;
|
||||||
|
|
||||||
|
try {
|
||||||
|
for (Result<Item> result : storage.listOriginals()) {
|
||||||
|
Item item = result.get();
|
||||||
|
|
||||||
|
if (item.isDir()) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
scanned++;
|
||||||
|
|
||||||
|
Instant lastModified = item.lastModified()
|
||||||
|
.toInstant();
|
||||||
|
|
||||||
|
if (!lastModified.isBefore(cutoff)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
candidates++;
|
||||||
|
|
||||||
|
CleanupResult outcome =
|
||||||
|
persistence.removeIfOrphan(
|
||||||
|
item.objectName(),
|
||||||
|
cutoff
|
||||||
|
);
|
||||||
|
|
||||||
|
if (outcome == CleanupResult.DELETED) {
|
||||||
|
deleted++;
|
||||||
|
|
||||||
|
log.info(
|
||||||
|
"Attachment cleanup removed orphan: {}",
|
||||||
|
item.objectName()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
log.info(
|
||||||
|
"Attachment cleanup completed: "
|
||||||
|
+ "scanned={}, candidates={}, deleted={}",
|
||||||
|
scanned,
|
||||||
|
candidates,
|
||||||
|
deleted
|
||||||
|
);
|
||||||
|
} catch (Exception e) {
|
||||||
|
// Прерываем текущий проход при ошибке БД или S3.
|
||||||
|
// Следующий запуск повторит проверку.
|
||||||
|
// Не выводим credentials или содержимое ответа S3.
|
||||||
|
log.warn(
|
||||||
|
"Attachment cleanup interrupted: "
|
||||||
|
+ "scanned={}, candidates={}, deleted={}, errorType={}",
|
||||||
|
scanned,
|
||||||
|
candidates,
|
||||||
|
deleted,
|
||||||
|
e.getClass().getSimpleName()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
package com.vnp.demo.service;
|
||||||
|
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
|
||||||
|
import org.springframework.stereotype.Service;
|
||||||
|
import org.springframework.transaction.annotation.Isolation;
|
||||||
|
import org.springframework.transaction.annotation.Transactional;
|
||||||
|
|
||||||
|
import com.vnp.demo.models.Attachment;
|
||||||
|
import com.vnp.demo.repositories.AttachmentRepository;
|
||||||
|
|
||||||
|
@Service
|
||||||
|
public class AttachmentPersistenceService {
|
||||||
|
|
||||||
|
private final AttachmentRepository attachmentRepository;
|
||||||
|
private final S3StorageService storage;
|
||||||
|
|
||||||
|
public AttachmentPersistenceService(
|
||||||
|
AttachmentRepository attachmentRepository,
|
||||||
|
S3StorageService storage
|
||||||
|
) {
|
||||||
|
this.attachmentRepository = attachmentRepository;
|
||||||
|
this.storage = storage;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Transactional(
|
||||||
|
isolation = Isolation.READ_COMMITTED,
|
||||||
|
timeout = 60
|
||||||
|
)
|
||||||
|
public void save(
|
||||||
|
Attachment attachment,
|
||||||
|
Path temporaryFile
|
||||||
|
) {
|
||||||
|
attachmentRepository.lockObject(
|
||||||
|
attachment.objectKey()
|
||||||
|
);
|
||||||
|
|
||||||
|
storage.put(
|
||||||
|
attachment.objectKey(),
|
||||||
|
temporaryFile,
|
||||||
|
attachment.sizeBytes(),
|
||||||
|
attachment.contentType()
|
||||||
|
);
|
||||||
|
|
||||||
|
attachmentRepository.save(attachment);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Transactional(
|
||||||
|
isolation = Isolation.READ_COMMITTED,
|
||||||
|
timeout = 30
|
||||||
|
)
|
||||||
|
public CleanupResult removeIfOrphan(
|
||||||
|
String objectKey,
|
||||||
|
Instant cutoff
|
||||||
|
) {
|
||||||
|
if (!objectKey.startsWith("originals/")) {
|
||||||
|
return CleanupResult.SKIPPED;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!attachmentRepository.tryLockObject(objectKey)) {
|
||||||
|
return CleanupResult.BUSY;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (attachmentRepository.existsByObjectKey(objectKey)) {
|
||||||
|
return CleanupResult.REFERENCED;
|
||||||
|
}
|
||||||
|
|
||||||
|
Instant lastModified = storage.lastModifiedOrNull(
|
||||||
|
objectKey
|
||||||
|
);
|
||||||
|
|
||||||
|
if (lastModified == null) {
|
||||||
|
return CleanupResult.MISSING;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!lastModified.isBefore(cutoff)) {
|
||||||
|
return CleanupResult.RECENT;
|
||||||
|
}
|
||||||
|
|
||||||
|
storage.remove(objectKey);
|
||||||
|
|
||||||
|
return CleanupResult.DELETED;
|
||||||
|
}
|
||||||
|
|
||||||
|
public enum CleanupResult {
|
||||||
|
DELETED,
|
||||||
|
REFERENCED,
|
||||||
|
RECENT,
|
||||||
|
BUSY,
|
||||||
|
MISSING,
|
||||||
|
SKIPPED
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,163 @@
|
|||||||
|
package com.vnp.demo.service;
|
||||||
|
|
||||||
|
import java.io.InputStream;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.time.temporal.ChronoUnit;
|
||||||
|
import java.util.UUID;
|
||||||
|
|
||||||
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
|
import com.vnp.demo.BusinessException;
|
||||||
|
import com.vnp.demo.dto.AttachmentResponse;
|
||||||
|
import com.vnp.demo.models.Attachment;
|
||||||
|
import com.vnp.demo.repositories.AttachmentRepository;
|
||||||
|
import com.vnp.demo.repositories.ChatMemberRepository;
|
||||||
|
import com.vnp.demo.validation.ImageValidator;
|
||||||
|
import com.vnp.demo.validation.ImageValidator.ValidatedImage;
|
||||||
|
|
||||||
|
@Service
|
||||||
|
public class AttachmentService {
|
||||||
|
|
||||||
|
private final AttachmentRepository attachmentRepository;
|
||||||
|
private final ChatMemberRepository chatMemberRepository;
|
||||||
|
private final ImageValidator imageValidator;
|
||||||
|
private final S3StorageService storage;
|
||||||
|
private final UploadLimiter uploadLimiter;
|
||||||
|
private final AttachmentPersistenceService persistence;
|
||||||
|
|
||||||
|
public AttachmentService(
|
||||||
|
AttachmentRepository attachmentRepository,
|
||||||
|
ChatMemberRepository chatMemberRepository,
|
||||||
|
ImageValidator imageValidator,
|
||||||
|
S3StorageService storage,
|
||||||
|
UploadLimiter uploadLimiter,
|
||||||
|
AttachmentPersistenceService persistence
|
||||||
|
) {
|
||||||
|
this.attachmentRepository = attachmentRepository;
|
||||||
|
this.chatMemberRepository = chatMemberRepository;
|
||||||
|
this.imageValidator = imageValidator;
|
||||||
|
this.storage = storage;
|
||||||
|
this.uploadLimiter = uploadLimiter;
|
||||||
|
this.persistence = persistence;
|
||||||
|
}
|
||||||
|
|
||||||
|
public AttachmentResponse upload(
|
||||||
|
UUID userId,
|
||||||
|
UUID chatId,
|
||||||
|
String filename,
|
||||||
|
String contentType,
|
||||||
|
long contentLength,
|
||||||
|
InputStream input
|
||||||
|
) {
|
||||||
|
checkChatAccess(userId, chatId);
|
||||||
|
|
||||||
|
imageValidator.validateFilename(filename);
|
||||||
|
|
||||||
|
try (UploadLimiter.Permit permit = uploadLimiter.acquire();
|
||||||
|
ValidatedImage image = imageValidator.validate(
|
||||||
|
input,
|
||||||
|
contentLength,
|
||||||
|
contentType
|
||||||
|
)) {
|
||||||
|
|
||||||
|
UUID attachmentId = UUID.randomUUID();
|
||||||
|
|
||||||
|
String objectKey = "originals/"
|
||||||
|
+ chatId
|
||||||
|
+ "/"
|
||||||
|
+ attachmentId;
|
||||||
|
|
||||||
|
Instant now = Instant.now()
|
||||||
|
.truncatedTo(ChronoUnit.MICROS);
|
||||||
|
|
||||||
|
Attachment attachment = new Attachment(
|
||||||
|
attachmentId,
|
||||||
|
chatId,
|
||||||
|
userId,
|
||||||
|
filename,
|
||||||
|
image.contentType(),
|
||||||
|
image.sizeBytes(),
|
||||||
|
image.sha256(),
|
||||||
|
objectKey,
|
||||||
|
"ready",
|
||||||
|
now,
|
||||||
|
now
|
||||||
|
);
|
||||||
|
|
||||||
|
persistence.save(attachment, image.path());
|
||||||
|
|
||||||
|
return AttachmentResponse.from(attachment);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public AttachmentResponse getMetadata(
|
||||||
|
UUID userId,
|
||||||
|
UUID attachmentId
|
||||||
|
) {
|
||||||
|
return AttachmentResponse.from(
|
||||||
|
getAccessibleAttachment(userId, attachmentId)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Download download(
|
||||||
|
UUID userId,
|
||||||
|
UUID attachmentId,
|
||||||
|
String variant
|
||||||
|
) {
|
||||||
|
Attachment attachment = getAccessibleAttachment(
|
||||||
|
userId,
|
||||||
|
attachmentId
|
||||||
|
);
|
||||||
|
|
||||||
|
if (!"ready".equals(attachment.status())) {
|
||||||
|
throw BusinessException.conflict(
|
||||||
|
"Вложение ещё не готово"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!"original".equals(variant)) {
|
||||||
|
throw BusinessException.notFound();
|
||||||
|
}
|
||||||
|
|
||||||
|
byte[] bytes = storage.get(
|
||||||
|
attachment.objectKey(),
|
||||||
|
attachment.sizeBytes(),
|
||||||
|
attachment.sha256()
|
||||||
|
);
|
||||||
|
|
||||||
|
return new Download(
|
||||||
|
attachment.filename(),
|
||||||
|
attachment.contentType(),
|
||||||
|
bytes
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private Attachment getAccessibleAttachment(
|
||||||
|
UUID userId,
|
||||||
|
UUID attachmentId
|
||||||
|
) {
|
||||||
|
Attachment attachment = attachmentRepository
|
||||||
|
.findById(attachmentId)
|
||||||
|
.orElseThrow(BusinessException::notFound);
|
||||||
|
|
||||||
|
checkChatAccess(userId, attachment.chatId());
|
||||||
|
|
||||||
|
return attachment;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void checkChatAccess(UUID userId, UUID chatId) {
|
||||||
|
boolean member = chatMemberRepository
|
||||||
|
.existsByIdChatIdAndIdUserId(chatId, userId);
|
||||||
|
|
||||||
|
if (!member) {
|
||||||
|
throw BusinessException.notFound();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public record Download(
|
||||||
|
String filename,
|
||||||
|
String contentType,
|
||||||
|
byte[] bytes
|
||||||
|
) {
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,7 +1,9 @@
|
|||||||
package com.vnp.demo.service;
|
package com.vnp.demo.service;
|
||||||
|
|
||||||
|
import java.util.HashSet;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.UUID;
|
import java.util.UUID;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
import org.springframework.data.domain.PageRequest;
|
import org.springframework.data.domain.PageRequest;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
@@ -12,9 +14,11 @@ import com.vnp.demo.CursorCodec;
|
|||||||
import com.vnp.demo.dto.MessageCreateRequest;
|
import com.vnp.demo.dto.MessageCreateRequest;
|
||||||
import com.vnp.demo.dto.MessagePageResponse;
|
import com.vnp.demo.dto.MessagePageResponse;
|
||||||
import com.vnp.demo.dto.MessageResponse;
|
import com.vnp.demo.dto.MessageResponse;
|
||||||
|
import com.vnp.demo.models.Attachment;
|
||||||
import com.vnp.demo.models.Chat;
|
import com.vnp.demo.models.Chat;
|
||||||
import com.vnp.demo.models.Message;
|
import com.vnp.demo.models.Message;
|
||||||
import com.vnp.demo.models.User;
|
import com.vnp.demo.models.User;
|
||||||
|
import com.vnp.demo.repositories.AttachmentRepository;
|
||||||
import com.vnp.demo.repositories.ChatMemberRepository;
|
import com.vnp.demo.repositories.ChatMemberRepository;
|
||||||
import com.vnp.demo.repositories.ChatRepository;
|
import com.vnp.demo.repositories.ChatRepository;
|
||||||
import com.vnp.demo.repositories.MessageRepository;
|
import com.vnp.demo.repositories.MessageRepository;
|
||||||
@@ -30,6 +34,7 @@ public class MessageService {
|
|||||||
private final ChatRepository chatRepository;
|
private final ChatRepository chatRepository;
|
||||||
private final ChatMemberRepository chatMemberRepository;
|
private final ChatMemberRepository chatMemberRepository;
|
||||||
private final UserRepository userRepository;
|
private final UserRepository userRepository;
|
||||||
|
private final AttachmentRepository attachmentRepository;
|
||||||
private final CursorCodec cursorCodec;
|
private final CursorCodec cursorCodec;
|
||||||
|
|
||||||
@PersistenceContext
|
@PersistenceContext
|
||||||
@@ -40,12 +45,14 @@ public class MessageService {
|
|||||||
ChatRepository chatRepository,
|
ChatRepository chatRepository,
|
||||||
ChatMemberRepository chatMemberRepository,
|
ChatMemberRepository chatMemberRepository,
|
||||||
UserRepository userRepository,
|
UserRepository userRepository,
|
||||||
|
AttachmentRepository attachmentRepository,
|
||||||
CursorCodec cursorCodec
|
CursorCodec cursorCodec
|
||||||
) {
|
) {
|
||||||
this.messageRepository = messageRepository;
|
this.messageRepository = messageRepository;
|
||||||
this.chatRepository = chatRepository;
|
this.chatRepository = chatRepository;
|
||||||
this.chatMemberRepository = chatMemberRepository;
|
this.chatMemberRepository = chatMemberRepository;
|
||||||
this.userRepository = userRepository;
|
this.userRepository = userRepository;
|
||||||
|
this.attachmentRepository = attachmentRepository;
|
||||||
this.cursorCodec = cursorCodec;
|
this.cursorCodec = cursorCodec;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -53,43 +60,72 @@ public class MessageService {
|
|||||||
public MessageResponse create(
|
public MessageResponse create(
|
||||||
UUID userId,
|
UUID userId,
|
||||||
UUID chatId,
|
UUID chatId,
|
||||||
MessageCreateRequest request, String key
|
MessageCreateRequest request,
|
||||||
|
String key
|
||||||
) {
|
) {
|
||||||
if (key == null || !key.matches("[A-Za-z0-9._:-]{1,128}")) {
|
if (key == null
|
||||||
throw BusinessException.invalid("Некорректный Idempotency-Key");
|
|| !key.matches("[A-Za-z0-9._:-]{1,128}")) {
|
||||||
}
|
|
||||||
checkAccess(userId, chatId);
|
|
||||||
|
|
||||||
if (request == null
|
|
||||||
|| request.text() == null
|
|
||||||
|| request.text().isBlank()
|
|
||||||
|| request.text().codePointCount(
|
|
||||||
0,
|
|
||||||
request.text().length()
|
|
||||||
) > 4000) {
|
|
||||||
|
|
||||||
throw BusinessException.invalid(
|
throw BusinessException.invalid(
|
||||||
"Текст сообщения должен содержать от 1 до 4000 символов"
|
"Некорректный Idempotency-Key"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
checkAccess(userId, chatId);
|
||||||
|
|
||||||
|
List<UUID> attachmentIds = validateRequest(request);
|
||||||
|
|
||||||
|
String attachmentSignature = attachmentIds.stream()
|
||||||
|
.map(UUID::toString)
|
||||||
|
.collect(Collectors.joining(","));
|
||||||
|
|
||||||
String scope = userId + ":" + chatId + ":" + key;
|
String scope = userId + ":" + chatId + ":" + key;
|
||||||
entityManager.createNativeQuery("SELECT 1 FROM pg_advisory_xact_lock(hashtextextended(:scope, 0))")
|
|
||||||
.setParameter("scope", scope).getResultList();
|
entityManager.createNativeQuery(
|
||||||
|
"""
|
||||||
|
SELECT 1
|
||||||
|
FROM pg_advisory_xact_lock(
|
||||||
|
hashtextextended(:scope, 0)
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
).setParameter("scope", scope).getResultList();
|
||||||
|
|
||||||
@SuppressWarnings("unchecked")
|
@SuppressWarnings("unchecked")
|
||||||
List<Object[]> existing = entityManager.createNativeQuery(
|
List<Object[]> existing = entityManager.createNativeQuery(
|
||||||
"SELECT message_id, request_text FROM message_idempotency " +
|
"""
|
||||||
"WHERE user_id = :user AND chat_id = :chat AND key = :key")
|
SELECT message_id, request_text, request_attachment_ids
|
||||||
.setParameter("user", userId).setParameter("chat", chatId)
|
FROM message_idempotency
|
||||||
.setParameter("key", key).getResultList();
|
WHERE user_id = :user
|
||||||
|
AND chat_id = :chat
|
||||||
|
AND key = :key
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
.setParameter("user", userId)
|
||||||
|
.setParameter("chat", chatId)
|
||||||
|
.setParameter("key", key)
|
||||||
|
.getResultList();
|
||||||
|
|
||||||
if (!existing.isEmpty()) {
|
if (!existing.isEmpty()) {
|
||||||
Object[] row = existing.get(0);
|
Object[] row = existing.get(0);
|
||||||
if (!request.text().equals(row[1])) {
|
|
||||||
throw BusinessException.conflict("Ключ уже использован с другим сообщением");
|
boolean sameText = request.text().equals(row[1]);
|
||||||
|
boolean sameAttachments =
|
||||||
|
attachmentSignature.equals(row[2]);
|
||||||
|
|
||||||
|
if (!sameText || !sameAttachments) {
|
||||||
|
throw BusinessException.conflict(
|
||||||
|
"Ключ уже использован с другим сообщением"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
return toResponse(messageRepository.findById((UUID) row[0]).orElseThrow());
|
|
||||||
|
Message original = messageRepository
|
||||||
|
.findById((UUID) row[0])
|
||||||
|
.orElseThrow(BusinessException::notFound);
|
||||||
|
|
||||||
|
return toResponse(original);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
validateAttachments(chatId, attachmentIds);
|
||||||
|
|
||||||
Chat chat = chatRepository.findById(chatId)
|
Chat chat = chatRepository.findById(chatId)
|
||||||
.orElseThrow(BusinessException::notFound);
|
.orElseThrow(BusinessException::notFound);
|
||||||
|
|
||||||
@@ -99,23 +135,110 @@ public class MessageService {
|
|||||||
Message message = new Message(
|
Message message = new Message(
|
||||||
chat,
|
chat,
|
||||||
sender,
|
sender,
|
||||||
request.text()
|
request.text(),
|
||||||
|
attachmentIds
|
||||||
);
|
);
|
||||||
|
|
||||||
Message saved = messageRepository.saveAndFlush(message);
|
Message saved = messageRepository.saveAndFlush(message);
|
||||||
|
|
||||||
// Перечитываем значения, фактически сохранённые PostgreSQL.
|
|
||||||
entityManager.refresh(saved);
|
entityManager.refresh(saved);
|
||||||
|
|
||||||
entityManager.createNativeQuery(
|
entityManager.createNativeQuery(
|
||||||
"INSERT INTO message_idempotency(user_id, chat_id, key, request_text, message_id) " +
|
"""
|
||||||
"VALUES (:user, :chat, :key, :text, :message)")
|
INSERT INTO message_idempotency (
|
||||||
.setParameter("user", userId).setParameter("chat", chatId)
|
user_id,
|
||||||
.setParameter("key", key).setParameter("text", request.text())
|
chat_id,
|
||||||
.setParameter("message", saved.getId()).executeUpdate();
|
key,
|
||||||
|
request_text,
|
||||||
|
request_attachment_ids,
|
||||||
|
message_id
|
||||||
|
)
|
||||||
|
VALUES (
|
||||||
|
:user,
|
||||||
|
:chat,
|
||||||
|
:key,
|
||||||
|
:text,
|
||||||
|
:attachments,
|
||||||
|
:message
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
.setParameter("user", userId)
|
||||||
|
.setParameter("chat", chatId)
|
||||||
|
.setParameter("key", key)
|
||||||
|
.setParameter("text", request.text())
|
||||||
|
.setParameter("attachments", attachmentSignature)
|
||||||
|
.setParameter("message", saved.getId())
|
||||||
|
.executeUpdate();
|
||||||
|
|
||||||
return toResponse(saved);
|
return toResponse(saved);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private List<UUID> validateRequest(
|
||||||
|
MessageCreateRequest request
|
||||||
|
) {
|
||||||
|
if (request == null || request.text() == null) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Поле text обязательно"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (request.text().codePointCount(
|
||||||
|
0,
|
||||||
|
request.text().length()
|
||||||
|
) > 4000) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Текст не должен превышать 4000 символов"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
List<UUID> attachmentIds = request.attachmentIds();
|
||||||
|
|
||||||
|
if (attachmentIds == null
|
||||||
|
|| attachmentIds.size() > 10
|
||||||
|
|| attachmentIds.stream().anyMatch(id -> id == null)) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"attachment_ids должен содержать от 0 до 10 UUID"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (new HashSet<>(attachmentIds).size()
|
||||||
|
!= attachmentIds.size()) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Повторяющиеся вложения недопустимы"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (request.text().isBlank() && attachmentIds.isEmpty()) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Сообщение должно содержать текст или вложение"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return List.copyOf(attachmentIds);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void validateAttachments(
|
||||||
|
UUID chatId,
|
||||||
|
List<UUID> attachmentIds
|
||||||
|
) {
|
||||||
|
for (UUID attachmentId : attachmentIds) {
|
||||||
|
Attachment attachment = attachmentRepository
|
||||||
|
.findById(attachmentId)
|
||||||
|
.orElseThrow(BusinessException::notFound);
|
||||||
|
|
||||||
|
if (!chatId.equals(attachment.chatId())) {
|
||||||
|
throw BusinessException.notFound();
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!"ready".equals(attachment.status())) {
|
||||||
|
throw BusinessException.conflict(
|
||||||
|
"Вложение не готово"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
@Transactional(readOnly = true)
|
@Transactional(readOnly = true)
|
||||||
public MessagePageResponse list(
|
public MessagePageResponse list(
|
||||||
UUID userId,
|
UUID userId,
|
||||||
@@ -140,14 +263,11 @@ public class MessageService {
|
|||||||
List<Message> messages;
|
List<Message> messages;
|
||||||
|
|
||||||
if (position == null) {
|
if (position == null) {
|
||||||
|
|
||||||
messages = messageRepository.findFirstPage(
|
messages = messageRepository.findFirstPage(
|
||||||
chatId,
|
chatId,
|
||||||
PageRequest.of(0, limit + 1)
|
PageRequest.of(0, limit + 1)
|
||||||
);
|
);
|
||||||
|
|
||||||
} else {
|
} else {
|
||||||
|
|
||||||
messages = messageRepository.findNextPage(
|
messages = messageRepository.findNextPage(
|
||||||
chatId,
|
chatId,
|
||||||
position.createdAt(),
|
position.createdAt(),
|
||||||
@@ -169,7 +289,6 @@ public class MessageService {
|
|||||||
String nextCursor = null;
|
String nextCursor = null;
|
||||||
|
|
||||||
if (hasNext && !page.isEmpty()) {
|
if (hasNext && !page.isEmpty()) {
|
||||||
|
|
||||||
Message last = page.get(page.size() - 1);
|
Message last = page.get(page.size() - 1);
|
||||||
|
|
||||||
nextCursor = cursorCodec.encodeMessage(
|
nextCursor = cursorCodec.encodeMessage(
|
||||||
@@ -179,23 +298,14 @@ public class MessageService {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
return new MessagePageResponse(
|
return new MessagePageResponse(items, nextCursor);
|
||||||
items,
|
|
||||||
nextCursor
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private void checkAccess(
|
private void checkAccess(UUID userId, UUID chatId) {
|
||||||
UUID userId,
|
boolean member = chatMemberRepository
|
||||||
UUID chatId
|
.existsByIdChatIdAndIdUserId(chatId, userId);
|
||||||
) {
|
|
||||||
boolean isMember = chatMemberRepository
|
|
||||||
.existsByIdChatIdAndIdUserId(
|
|
||||||
chatId,
|
|
||||||
userId
|
|
||||||
);
|
|
||||||
|
|
||||||
if (!isMember) {
|
if (!member) {
|
||||||
throw BusinessException.notFound();
|
throw BusinessException.notFound();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -206,7 +316,8 @@ public class MessageService {
|
|||||||
message.getChat().getId(),
|
message.getChat().getId(),
|
||||||
message.getSender().getId(),
|
message.getSender().getId(),
|
||||||
message.getText(),
|
message.getText(),
|
||||||
message.getCreatedAt()
|
message.getCreatedAt(),
|
||||||
|
List.copyOf(message.getAttachmentIds())
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
package com.vnp.demo.service;
|
||||||
|
|
||||||
|
import java.io.InputStream;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.concurrent.TimeUnit;
|
||||||
|
|
||||||
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
|
import org.springframework.http.HttpStatus;
|
||||||
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
|
import com.vnp.demo.BusinessException;
|
||||||
|
import com.vnp.demo.validation.ImageValidator;
|
||||||
|
|
||||||
|
import io.minio.BucketExistsArgs;
|
||||||
|
import io.minio.GetObjectArgs;
|
||||||
|
import io.minio.ListObjectsArgs;
|
||||||
|
import io.minio.MinioClient;
|
||||||
|
import io.minio.PutObjectArgs;
|
||||||
|
import io.minio.RemoveObjectArgs;
|
||||||
|
import io.minio.Result;
|
||||||
|
import io.minio.StatObjectArgs;
|
||||||
|
import io.minio.StatObjectResponse;
|
||||||
|
import io.minio.errors.ErrorResponseException;
|
||||||
|
import io.minio.messages.Item;
|
||||||
|
import okhttp3.OkHttpClient;
|
||||||
|
|
||||||
|
@Service
|
||||||
|
public class S3StorageService {
|
||||||
|
|
||||||
|
private final MinioClient client;
|
||||||
|
private final String bucket;
|
||||||
|
|
||||||
|
public S3StorageService(
|
||||||
|
@Value("${app.s3.endpoint}") String endpoint,
|
||||||
|
@Value("${app.s3.access-key}") String accessKey,
|
||||||
|
@Value("${app.s3.secret-key}") String secretKey,
|
||||||
|
@Value("${app.s3.bucket}") String bucket
|
||||||
|
) {
|
||||||
|
this.bucket = bucket;
|
||||||
|
|
||||||
|
OkHttpClient httpClient = new OkHttpClient.Builder()
|
||||||
|
.connectTimeout(2, TimeUnit.SECONDS)
|
||||||
|
.writeTimeout(10, TimeUnit.SECONDS)
|
||||||
|
.readTimeout(10, TimeUnit.SECONDS)
|
||||||
|
.callTimeout(30, TimeUnit.SECONDS)
|
||||||
|
.retryOnConnectionFailure(false)
|
||||||
|
.build();
|
||||||
|
|
||||||
|
this.client = MinioClient.builder()
|
||||||
|
.endpoint(endpoint)
|
||||||
|
.credentials(accessKey, secretKey)
|
||||||
|
.region("us-east-1")
|
||||||
|
.httpClient(httpClient)
|
||||||
|
.build();
|
||||||
|
}
|
||||||
|
|
||||||
|
public void put(
|
||||||
|
String objectKey,
|
||||||
|
Path file,
|
||||||
|
long sizeBytes,
|
||||||
|
String contentType
|
||||||
|
) {
|
||||||
|
if (sizeBytes < 1
|
||||||
|
|| sizeBytes > ImageValidator.MAX_BYTES) {
|
||||||
|
throw new BusinessException(
|
||||||
|
"payload_too_large",
|
||||||
|
"Некорректный размер загружаемого файла",
|
||||||
|
HttpStatus.PAYLOAD_TOO_LARGE
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
try (InputStream input = Files.newInputStream(file)) {
|
||||||
|
client.putObject(
|
||||||
|
PutObjectArgs.builder()
|
||||||
|
.bucket(bucket)
|
||||||
|
.object(objectKey)
|
||||||
|
.contentType(contentType)
|
||||||
|
.stream(input, sizeBytes, ImageValidator.MAX_BYTES)
|
||||||
|
.build()
|
||||||
|
);
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public byte[] get(
|
||||||
|
String objectKey,
|
||||||
|
long expectedSize,
|
||||||
|
String expectedSha256
|
||||||
|
) {
|
||||||
|
if (expectedSize < 1
|
||||||
|
|| expectedSize > ImageValidator.MAX_BYTES) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
|
||||||
|
try (InputStream input = client.getObject(
|
||||||
|
GetObjectArgs.builder()
|
||||||
|
.bucket(bucket)
|
||||||
|
.object(objectKey)
|
||||||
|
.build()
|
||||||
|
)) {
|
||||||
|
byte[] bytes = input.readNBytes(
|
||||||
|
Math.toIntExact(expectedSize) + 1
|
||||||
|
);
|
||||||
|
|
||||||
|
if (bytes.length != expectedSize
|
||||||
|
|| !ImageValidator.sha256(bytes)
|
||||||
|
.equals(expectedSha256)) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
|
||||||
|
return bytes;
|
||||||
|
} catch (BusinessException e) {
|
||||||
|
throw e;
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public Iterable<Result<Item>> listOriginals() {
|
||||||
|
return client.listObjects(
|
||||||
|
ListObjectsArgs.builder()
|
||||||
|
.bucket(bucket)
|
||||||
|
.prefix("originals/")
|
||||||
|
.recursive(true)
|
||||||
|
.build()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Instant lastModifiedOrNull(String objectKey) {
|
||||||
|
try {
|
||||||
|
StatObjectResponse response = client.statObject(
|
||||||
|
StatObjectArgs.builder()
|
||||||
|
.bucket(bucket)
|
||||||
|
.object(objectKey)
|
||||||
|
.build()
|
||||||
|
);
|
||||||
|
|
||||||
|
return response.lastModified().toInstant();
|
||||||
|
} catch (ErrorResponseException e) {
|
||||||
|
String code = e.errorResponse().code();
|
||||||
|
|
||||||
|
if ("NoSuchKey".equals(code)
|
||||||
|
|| "NoSuchObject".equals(code)) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
throw unavailable();
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public void remove(String objectKey) {
|
||||||
|
try {
|
||||||
|
client.removeObject(
|
||||||
|
RemoveObjectArgs.builder()
|
||||||
|
.bucket(bucket)
|
||||||
|
.object(objectKey)
|
||||||
|
.build()
|
||||||
|
);
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public void checkReady() {
|
||||||
|
try {
|
||||||
|
boolean exists = client.bucketExists(
|
||||||
|
BucketExistsArgs.builder()
|
||||||
|
.bucket(bucket)
|
||||||
|
.build()
|
||||||
|
);
|
||||||
|
|
||||||
|
if (!exists) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
} catch (BusinessException e) {
|
||||||
|
throw e;
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw unavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static BusinessException unavailable() {
|
||||||
|
return new BusinessException(
|
||||||
|
"unavailable",
|
||||||
|
"Хранилище файлов временно недоступно",
|
||||||
|
HttpStatus.SERVICE_UNAVAILABLE
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
package com.vnp.demo.service;
|
||||||
|
|
||||||
|
import java.util.concurrent.Semaphore;
|
||||||
|
import java.util.concurrent.atomic.AtomicBoolean;
|
||||||
|
|
||||||
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
|
import org.springframework.http.HttpStatus;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import com.vnp.demo.BusinessException;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
public class UploadLimiter {
|
||||||
|
|
||||||
|
private final Semaphore semaphore;
|
||||||
|
|
||||||
|
public UploadLimiter(
|
||||||
|
@Value("${app.uploads.max-concurrent:1}")
|
||||||
|
int maxConcurrent
|
||||||
|
) {
|
||||||
|
if (maxConcurrent < 1) {
|
||||||
|
throw new IllegalArgumentException(
|
||||||
|
"max-concurrent должен быть больше нуля"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
this.semaphore = new Semaphore(maxConcurrent);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Permit acquire() {
|
||||||
|
if (!semaphore.tryAcquire()) {
|
||||||
|
throw new BusinessException(
|
||||||
|
"rate_limited",
|
||||||
|
"Все обработчики загрузки заняты. Повторите запрос позже",
|
||||||
|
HttpStatus.TOO_MANY_REQUESTS
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return new Permit(semaphore);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static final class Permit implements AutoCloseable {
|
||||||
|
|
||||||
|
private final Semaphore semaphore;
|
||||||
|
private final AtomicBoolean released = new AtomicBoolean(false);
|
||||||
|
|
||||||
|
private Permit(Semaphore semaphore) {
|
||||||
|
this.semaphore = semaphore;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void close() {
|
||||||
|
if (released.compareAndSet(false, true)) {
|
||||||
|
semaphore.release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,481 @@
|
|||||||
|
package com.vnp.demo.validation;
|
||||||
|
|
||||||
|
import java.awt.image.BufferedImage;
|
||||||
|
import java.io.IOException;
|
||||||
|
import java.io.InputStream;
|
||||||
|
import java.io.OutputStream;
|
||||||
|
import java.nio.file.DirectoryStream;
|
||||||
|
import java.nio.file.Files;
|
||||||
|
import java.nio.file.LinkOption;
|
||||||
|
import java.nio.file.Path;
|
||||||
|
import java.security.MessageDigest;
|
||||||
|
import java.security.NoSuchAlgorithmException;
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.HexFormat;
|
||||||
|
import java.util.Iterator;
|
||||||
|
import java.util.Set;
|
||||||
|
import java.util.concurrent.ConcurrentHashMap;
|
||||||
|
import java.util.concurrent.TimeUnit;
|
||||||
|
import java.util.concurrent.atomic.AtomicBoolean;
|
||||||
|
|
||||||
|
import javax.imageio.ImageIO;
|
||||||
|
import javax.imageio.ImageReader;
|
||||||
|
import javax.imageio.stream.FileImageInputStream;
|
||||||
|
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
|
import org.springframework.http.HttpStatus;
|
||||||
|
import org.springframework.http.MediaType;
|
||||||
|
import org.springframework.scheduling.annotation.Scheduled;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
import com.vnp.demo.BusinessException;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
public class ImageValidator {
|
||||||
|
|
||||||
|
public static final int MAX_BYTES = 10 * 1024 * 1024;
|
||||||
|
public static final long MAX_PIXELS = 40_000_000L;
|
||||||
|
|
||||||
|
private static final int BUFFER_SIZE = 64 * 1024;
|
||||||
|
|
||||||
|
private static final long MAX_RECEIVE_NANOS =
|
||||||
|
TimeUnit.SECONDS.toNanos(60);
|
||||||
|
|
||||||
|
private static final long TEMP_FILE_GRACE_SECONDS = 3600;
|
||||||
|
|
||||||
|
private static final Logger log =
|
||||||
|
LoggerFactory.getLogger(ImageValidator.class);
|
||||||
|
|
||||||
|
private static final Set<Path> ACTIVE_FILES =
|
||||||
|
ConcurrentHashMap.newKeySet();
|
||||||
|
|
||||||
|
private final Path tempDirectory;
|
||||||
|
|
||||||
|
public ImageValidator(
|
||||||
|
@Value("${app.uploads.temp-dir:/tmp/messenger-uploads}")
|
||||||
|
String tempDirectory
|
||||||
|
) {
|
||||||
|
this.tempDirectory = Path.of(tempDirectory)
|
||||||
|
.toAbsolutePath()
|
||||||
|
.normalize();
|
||||||
|
|
||||||
|
try {
|
||||||
|
Files.createDirectories(this.tempDirectory);
|
||||||
|
} catch (IOException e) {
|
||||||
|
throw new IllegalStateException(
|
||||||
|
"Не удалось создать каталог временных загрузок",
|
||||||
|
e
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public record ValidatedImage(
|
||||||
|
Path path,
|
||||||
|
long sizeBytes,
|
||||||
|
String contentType,
|
||||||
|
String sha256
|
||||||
|
) implements AutoCloseable {
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void close() {
|
||||||
|
deleteTemporaryFile(path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public void validateFilename(String filename) {
|
||||||
|
if (filename == null) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Не указано имя файла"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
int length = filename.codePointCount(
|
||||||
|
0,
|
||||||
|
filename.length()
|
||||||
|
);
|
||||||
|
|
||||||
|
if (length < 1 || length > 255) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Имя файла должно содержать от 1 до 255 символов"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
boolean unsafe = filename.codePoints().anyMatch(
|
||||||
|
codePoint -> codePoint == '/'
|
||||||
|
|| codePoint == '\\'
|
||||||
|
|| Character.isISOControl(codePoint)
|
||||||
|
);
|
||||||
|
|
||||||
|
if (unsafe) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Имя файла содержит недопустимые символы"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public ValidatedImage validate(
|
||||||
|
InputStream input,
|
||||||
|
long contentLength,
|
||||||
|
String declaredContentType
|
||||||
|
) {
|
||||||
|
if (contentLength > MAX_BYTES) {
|
||||||
|
throw tooLarge(
|
||||||
|
"Размер файла превышает 10 MiB"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
String declaredMime = normalizeMime(
|
||||||
|
declaredContentType
|
||||||
|
);
|
||||||
|
|
||||||
|
Path temporaryFile = createTemporaryFile();
|
||||||
|
|
||||||
|
boolean success = false;
|
||||||
|
|
||||||
|
try {
|
||||||
|
MessageDigest digest = newSha256();
|
||||||
|
|
||||||
|
long size = copyLimited(
|
||||||
|
input,
|
||||||
|
temporaryFile,
|
||||||
|
digest
|
||||||
|
);
|
||||||
|
|
||||||
|
if (size == 0) {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
|
||||||
|
if (contentLength >= 0 && size != contentLength) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Тело запроса получено не полностью"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
String actualMime = validateImageContent(
|
||||||
|
temporaryFile
|
||||||
|
);
|
||||||
|
|
||||||
|
if (!actualMime.equals(declaredMime)) {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
|
||||||
|
String checksum = HexFormat.of()
|
||||||
|
.formatHex(digest.digest());
|
||||||
|
|
||||||
|
ValidatedImage result = new ValidatedImage(
|
||||||
|
temporaryFile,
|
||||||
|
size,
|
||||||
|
actualMime,
|
||||||
|
checksum
|
||||||
|
);
|
||||||
|
|
||||||
|
success = true;
|
||||||
|
|
||||||
|
return result;
|
||||||
|
} finally {
|
||||||
|
if (!success) {
|
||||||
|
deleteTemporaryFile(temporaryFile);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private Path createTemporaryFile() {
|
||||||
|
try {
|
||||||
|
Path file = Files.createTempFile(
|
||||||
|
tempDirectory,
|
||||||
|
"upload-",
|
||||||
|
".tmp"
|
||||||
|
);
|
||||||
|
|
||||||
|
ACTIVE_FILES.add(file);
|
||||||
|
|
||||||
|
return file;
|
||||||
|
} catch (IOException e) {
|
||||||
|
throw temporaryStorageUnavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private long copyLimited(
|
||||||
|
InputStream input,
|
||||||
|
Path destination,
|
||||||
|
MessageDigest digest
|
||||||
|
) {
|
||||||
|
byte[] buffer = new byte[BUFFER_SIZE];
|
||||||
|
long total = 0;
|
||||||
|
long started = System.nanoTime();
|
||||||
|
|
||||||
|
try (OutputStream output =
|
||||||
|
Files.newOutputStream(destination)) {
|
||||||
|
|
||||||
|
while (true) {
|
||||||
|
checkReceiveDeadline(started);
|
||||||
|
|
||||||
|
int requested = (int) Math.min(
|
||||||
|
buffer.length,
|
||||||
|
(long) MAX_BYTES - total + 1
|
||||||
|
);
|
||||||
|
|
||||||
|
int count;
|
||||||
|
|
||||||
|
try {
|
||||||
|
count = input.read(buffer, 0, requested);
|
||||||
|
} catch (IOException e) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Передача файла прервана или превышено время ожидания"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
checkReceiveDeadline(started);
|
||||||
|
|
||||||
|
if (count == -1) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (count == 0) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
total += count;
|
||||||
|
|
||||||
|
if (total > MAX_BYTES) {
|
||||||
|
throw tooLarge(
|
||||||
|
"Размер файла превышает 10 MiB"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
output.write(buffer, 0, count);
|
||||||
|
digest.update(buffer, 0, count);
|
||||||
|
}
|
||||||
|
|
||||||
|
return total;
|
||||||
|
} catch (IOException e) {
|
||||||
|
throw temporaryStorageUnavailable();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void checkReceiveDeadline(long started) {
|
||||||
|
if (System.nanoTime() - started >= MAX_RECEIVE_NANOS) {
|
||||||
|
throw BusinessException.invalid(
|
||||||
|
"Превышено допустимое время приёма файла"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private String normalizeMime(String value) {
|
||||||
|
if (value == null) {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
MediaType type = MediaType.parseMediaType(value);
|
||||||
|
|
||||||
|
if (type.getType().equalsIgnoreCase("image")
|
||||||
|
&& type.getSubtype().equalsIgnoreCase("png")) {
|
||||||
|
return "image/png";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (type.getType().equalsIgnoreCase("image")
|
||||||
|
&& type.getSubtype().equalsIgnoreCase("jpeg")) {
|
||||||
|
return "image/jpeg";
|
||||||
|
}
|
||||||
|
} catch (IllegalArgumentException e) {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
|
||||||
|
private String validateImageContent(Path file) {
|
||||||
|
try (FileImageInputStream imageInput =
|
||||||
|
new FileImageInputStream(file.toFile())) {
|
||||||
|
|
||||||
|
Iterator<ImageReader> readers =
|
||||||
|
ImageIO.getImageReaders(imageInput);
|
||||||
|
|
||||||
|
if (!readers.hasNext()) {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
|
||||||
|
ImageReader reader = readers.next();
|
||||||
|
|
||||||
|
try {
|
||||||
|
reader.setInput(imageInput, true, true);
|
||||||
|
|
||||||
|
String format = reader.getFormatName();
|
||||||
|
String mime;
|
||||||
|
|
||||||
|
if ("png".equalsIgnoreCase(format)) {
|
||||||
|
mime = "image/png";
|
||||||
|
} else if ("jpeg".equalsIgnoreCase(format)
|
||||||
|
|| "jpg".equalsIgnoreCase(format)) {
|
||||||
|
mime = "image/jpeg";
|
||||||
|
} else {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
|
||||||
|
int width = reader.getWidth(0);
|
||||||
|
int height = reader.getHeight(0);
|
||||||
|
|
||||||
|
if (width <= 0 || height <= 0) {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
|
||||||
|
long pixels = (long) width * height;
|
||||||
|
|
||||||
|
if (pixels > MAX_PIXELS) {
|
||||||
|
throw tooLarge(
|
||||||
|
"Изображение содержит более 40 миллионов пикселей"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
AtomicBoolean warning = new AtomicBoolean(false);
|
||||||
|
|
||||||
|
reader.addIIOReadWarningListener(
|
||||||
|
(source, message) -> warning.set(true)
|
||||||
|
);
|
||||||
|
|
||||||
|
BufferedImage image = reader.read(0);
|
||||||
|
|
||||||
|
if (image == null) {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
if (warning.get()) {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
image.flush();
|
||||||
|
}
|
||||||
|
|
||||||
|
return mime;
|
||||||
|
} finally {
|
||||||
|
reader.dispose();
|
||||||
|
}
|
||||||
|
} catch (BusinessException e) {
|
||||||
|
throw e;
|
||||||
|
} catch (IOException | RuntimeException e) {
|
||||||
|
throw unsupported();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Scheduled(
|
||||||
|
initialDelay = 60_000,
|
||||||
|
fixedDelay = 300_000
|
||||||
|
)
|
||||||
|
public void cleanupTemporaryFiles() {
|
||||||
|
Instant cutoff = Instant.now()
|
||||||
|
.minusSeconds(TEMP_FILE_GRACE_SECONDS);
|
||||||
|
|
||||||
|
long deleted = 0;
|
||||||
|
|
||||||
|
try (DirectoryStream<Path> files =
|
||||||
|
Files.newDirectoryStream(
|
||||||
|
tempDirectory,
|
||||||
|
"upload-*.tmp"
|
||||||
|
)) {
|
||||||
|
|
||||||
|
for (Path file : files) {
|
||||||
|
if (ACTIVE_FILES.contains(file)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!Files.isRegularFile(
|
||||||
|
file,
|
||||||
|
LinkOption.NOFOLLOW_LINKS
|
||||||
|
)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
Instant modified = Files.getLastModifiedTime(
|
||||||
|
file,
|
||||||
|
LinkOption.NOFOLLOW_LINKS
|
||||||
|
).toInstant();
|
||||||
|
|
||||||
|
if (!modified.isBefore(cutoff)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (Files.deleteIfExists(file)) {
|
||||||
|
deleted++;
|
||||||
|
}
|
||||||
|
} catch (IOException e) {
|
||||||
|
log.warn(
|
||||||
|
"Не удалось очистить временный файл {}",
|
||||||
|
file.getFileName(),
|
||||||
|
e
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
log.info(
|
||||||
|
"Temporary upload cleanup completed: deleted={}",
|
||||||
|
deleted
|
||||||
|
);
|
||||||
|
} catch (IOException e) {
|
||||||
|
log.warn(
|
||||||
|
"Не удалось просмотреть каталог временных загрузок",
|
||||||
|
e
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static MessageDigest newSha256() {
|
||||||
|
try {
|
||||||
|
return MessageDigest.getInstance("SHA-256");
|
||||||
|
} catch (NoSuchAlgorithmException e) {
|
||||||
|
throw new IllegalStateException(
|
||||||
|
"SHA-256 недоступен",
|
||||||
|
e
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static String sha256(byte[] bytes) {
|
||||||
|
return HexFormat.of().formatHex(
|
||||||
|
newSha256().digest(bytes)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void deleteTemporaryFile(Path file) {
|
||||||
|
try {
|
||||||
|
Files.deleteIfExists(file);
|
||||||
|
} catch (IOException e) {
|
||||||
|
log.warn(
|
||||||
|
"Не удалось удалить временный файл {}",
|
||||||
|
file.getFileName(),
|
||||||
|
e
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
|
||||||
|
ACTIVE_FILES.remove(file);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static BusinessException unsupported() {
|
||||||
|
return new BusinessException(
|
||||||
|
"unsupported_media_type",
|
||||||
|
"Ожидается корректное изображение PNG или JPEG",
|
||||||
|
HttpStatus.UNSUPPORTED_MEDIA_TYPE
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static BusinessException tooLarge(String message) {
|
||||||
|
return new BusinessException(
|
||||||
|
"payload_too_large",
|
||||||
|
message,
|
||||||
|
HttpStatus.PAYLOAD_TOO_LARGE
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static BusinessException temporaryStorageUnavailable() {
|
||||||
|
return new BusinessException(
|
||||||
|
"unavailable",
|
||||||
|
"Временное хранилище загрузок недоступно",
|
||||||
|
HttpStatus.SERVICE_UNAVAILABLE
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -47,3 +47,19 @@ spring:
|
|||||||
app:
|
app:
|
||||||
session:
|
session:
|
||||||
ttl-seconds: 86400
|
ttl-seconds: 86400
|
||||||
|
|
||||||
|
s3:
|
||||||
|
endpoint: ${S3_ENDPOINT:http://localhost:9000}
|
||||||
|
access-key: ${S3_ACCESS_KEY:}
|
||||||
|
secret-key: ${S3_SECRET_KEY:}
|
||||||
|
bucket: ${S3_BUCKET:messenger-attachments}
|
||||||
|
|
||||||
|
uploads:
|
||||||
|
max-concurrent: ${UPLOAD_MAX_CONCURRENT:1}
|
||||||
|
temp-dir: ${UPLOAD_TEMP_DIR:/tmp/messenger-uploads}
|
||||||
|
|
||||||
|
cleanup:
|
||||||
|
enabled: ${ATTACHMENT_CLEANUP_ENABLED:true}
|
||||||
|
grace-seconds: ${ATTACHMENT_CLEANUP_GRACE_SECONDS:3600}
|
||||||
|
interval-ms: ${ATTACHMENT_CLEANUP_INTERVAL_MS:300000}
|
||||||
|
initial-delay-ms: ${ATTACHMENT_CLEANUP_INITIAL_DELAY_MS:60000}
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
CREATE TABLE attachments (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
chat_id UUID NOT NULL REFERENCES chats(id),
|
||||||
|
owner_id UUID NOT NULL REFERENCES users(id),
|
||||||
|
|
||||||
|
filename TEXT NOT NULL,
|
||||||
|
content_type VARCHAR(32) NOT NULL,
|
||||||
|
size_bytes BIGINT NOT NULL,
|
||||||
|
sha256 CHAR(64) NOT NULL,
|
||||||
|
|
||||||
|
object_key TEXT NOT NULL UNIQUE,
|
||||||
|
status VARCHAR(16) NOT NULL DEFAULT 'ready',
|
||||||
|
|
||||||
|
created_at TIMESTAMPTZ NOT NULL,
|
||||||
|
updated_at TIMESTAMPTZ NOT NULL,
|
||||||
|
|
||||||
|
CONSTRAINT chk_attachment_filename
|
||||||
|
CHECK (char_length(filename) BETWEEN 1 AND 255),
|
||||||
|
|
||||||
|
CONSTRAINT chk_attachment_content_type
|
||||||
|
CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||||
|
|
||||||
|
CONSTRAINT chk_attachment_size
|
||||||
|
CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||||
|
|
||||||
|
CONSTRAINT chk_attachment_sha256
|
||||||
|
CHECK (sha256 ~ '^[a-f0-9]{64}$'),
|
||||||
|
|
||||||
|
CONSTRAINT chk_attachment_status
|
||||||
|
CHECK (status = 'ready'),
|
||||||
|
|
||||||
|
CONSTRAINT fk_attachment_membership
|
||||||
|
FOREIGN KEY (chat_id, owner_id)
|
||||||
|
REFERENCES chat_members(chat_id, user_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_attachments_chat
|
||||||
|
ON attachments(chat_id, created_at, id);
|
||||||
|
|
||||||
|
CREATE TABLE message_attachments (
|
||||||
|
message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||||
|
attachment_id UUID NOT NULL REFERENCES attachments(id),
|
||||||
|
position INTEGER NOT NULL,
|
||||||
|
|
||||||
|
PRIMARY KEY (message_id, position),
|
||||||
|
|
||||||
|
CONSTRAINT uq_message_attachment
|
||||||
|
UNIQUE (message_id, attachment_id),
|
||||||
|
|
||||||
|
CONSTRAINT chk_attachment_position
|
||||||
|
CHECK (position BETWEEN 0 AND 9)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_message_attachments_attachment
|
||||||
|
ON message_attachments(attachment_id);
|
||||||
|
|
||||||
|
|
||||||
|
ALTER TABLE messages
|
||||||
|
DROP CONSTRAINT chk_message_text;
|
||||||
|
|
||||||
|
ALTER TABLE messages
|
||||||
|
ADD CONSTRAINT chk_message_text
|
||||||
|
CHECK (char_length(text) BETWEEN 0 AND 4000);
|
||||||
|
|
||||||
|
-- Старые запросы не содержали вложений, поэтому для них значение пустое.
|
||||||
|
ALTER TABLE message_idempotency
|
||||||
|
ADD COLUMN request_attachment_ids TEXT NOT NULL DEFAULT '';
|
||||||
@@ -0,0 +1,293 @@
|
|||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from urllib.parse import unquote, urlsplit
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
|
|
||||||
|
def output(args):
|
||||||
|
result = subprocess.run(
|
||||||
|
args,
|
||||||
|
cwd=ROOT,
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
if result.returncode:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Docker command failed; "
|
||||||
|
"check Docker and running services"
|
||||||
|
)
|
||||||
|
|
||||||
|
return result.stdout.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def service(name):
|
||||||
|
ids = output([
|
||||||
|
"docker", "compose", "ps", "-q", name,
|
||||||
|
]).splitlines()
|
||||||
|
|
||||||
|
if not ids:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Service is not running: " + name
|
||||||
|
)
|
||||||
|
|
||||||
|
return json.loads(output([
|
||||||
|
"docker", "inspect", ids[0],
|
||||||
|
]))[0]
|
||||||
|
|
||||||
|
|
||||||
|
def environment(container):
|
||||||
|
return dict(
|
||||||
|
item.split("=", 1)
|
||||||
|
for item in container["Config"]["Env"]
|
||||||
|
if "=" in item
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Reconcile PostgreSQL attachment metadata and S3"
|
||||||
|
)
|
||||||
|
|
||||||
|
modes = parser.add_mutually_exclusive_group()
|
||||||
|
modes.add_argument("--apply", action="store_true")
|
||||||
|
modes.add_argument("--dry-run", action="store_true")
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--chat-id",
|
||||||
|
type=lambda value: str(uuid.UUID(value)),
|
||||||
|
default=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--grace-seconds",
|
||||||
|
type=int,
|
||||||
|
default=3600,
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--max-seconds",
|
||||||
|
type=int,
|
||||||
|
default=600,
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--backup-dir",
|
||||||
|
default="backups/attachments",
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--output",
|
||||||
|
default="results/reconcile.json",
|
||||||
|
)
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if args.grace_seconds < 60 or args.max_seconds < 1:
|
||||||
|
parser.error(
|
||||||
|
"grace-seconds >= 60; max-seconds >= 1"
|
||||||
|
)
|
||||||
|
|
||||||
|
api = service("api")
|
||||||
|
postgres = service("postgres")
|
||||||
|
minio = service("minio")
|
||||||
|
|
||||||
|
env = environment(api)
|
||||||
|
pg = environment(postgres)
|
||||||
|
|
||||||
|
common = (
|
||||||
|
set(api["NetworkSettings"]["Networks"])
|
||||||
|
& set(postgres["NetworkSettings"]["Networks"])
|
||||||
|
& set(minio["NetworkSettings"]["Networks"])
|
||||||
|
)
|
||||||
|
|
||||||
|
if len(common) != 1:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Expected exactly one common Docker network "
|
||||||
|
"for API, PostgreSQL and MinIO"
|
||||||
|
)
|
||||||
|
|
||||||
|
network = next(iter(common))
|
||||||
|
|
||||||
|
database_url = (
|
||||||
|
env.get("SPRING_DATASOURCE_URL")
|
||||||
|
or env.get("DB_URL")
|
||||||
|
or env.get("DATABASE_URL")
|
||||||
|
or ""
|
||||||
|
)
|
||||||
|
|
||||||
|
if database_url:
|
||||||
|
address = urlsplit(
|
||||||
|
database_url.removeprefix("jdbc:")
|
||||||
|
)
|
||||||
|
|
||||||
|
if address.scheme not in ("postgresql", "postgres"):
|
||||||
|
raise RuntimeError(
|
||||||
|
"Unsupported database URL scheme"
|
||||||
|
)
|
||||||
|
|
||||||
|
db_host = address.hostname
|
||||||
|
db_port = address.port or 5432
|
||||||
|
db_name = unquote(address.path.lstrip("/"))
|
||||||
|
|
||||||
|
url_user = (
|
||||||
|
unquote(address.username)
|
||||||
|
if address.username else None
|
||||||
|
)
|
||||||
|
|
||||||
|
url_password = (
|
||||||
|
unquote(address.password)
|
||||||
|
if address.password else None
|
||||||
|
)
|
||||||
|
|
||||||
|
else:
|
||||||
|
db_host = env.get("DB_HOST") or "postgres"
|
||||||
|
db_port = int(env.get("DB_PORT") or "5432")
|
||||||
|
|
||||||
|
db_name = (
|
||||||
|
env.get("DB_NAME")
|
||||||
|
or pg.get("POSTGRES_DB")
|
||||||
|
or pg.get("POSTGRES_USER")
|
||||||
|
or "postgres"
|
||||||
|
)
|
||||||
|
|
||||||
|
url_user = None
|
||||||
|
url_password = None
|
||||||
|
|
||||||
|
db_user = (
|
||||||
|
env.get("SPRING_DATASOURCE_USERNAME")
|
||||||
|
or env.get("DB_USERNAME")
|
||||||
|
or env.get("DB_USER")
|
||||||
|
or url_user
|
||||||
|
or pg.get("POSTGRES_USER")
|
||||||
|
)
|
||||||
|
|
||||||
|
db_password = (
|
||||||
|
env.get("SPRING_DATASOURCE_PASSWORD")
|
||||||
|
or env.get("DB_PASSWORD")
|
||||||
|
or url_password
|
||||||
|
or pg.get("POSTGRES_PASSWORD")
|
||||||
|
)
|
||||||
|
|
||||||
|
if not db_host or not db_name:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Cannot determine PostgreSQL host or database"
|
||||||
|
)
|
||||||
|
|
||||||
|
if not db_user or not db_password:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Database credentials are not available "
|
||||||
|
"in container environment"
|
||||||
|
)
|
||||||
|
|
||||||
|
required = [
|
||||||
|
"S3_ENDPOINT",
|
||||||
|
"S3_ACCESS_KEY",
|
||||||
|
"S3_SECRET_KEY",
|
||||||
|
"S3_BUCKET",
|
||||||
|
]
|
||||||
|
|
||||||
|
if any(not env.get(key) for key in required):
|
||||||
|
raise RuntimeError(
|
||||||
|
"Missing S3 settings in API container environment"
|
||||||
|
)
|
||||||
|
|
||||||
|
config = {
|
||||||
|
"apply": args.apply,
|
||||||
|
"chat_id": args.chat_id,
|
||||||
|
"grace_seconds": args.grace_seconds,
|
||||||
|
"max_seconds": args.max_seconds,
|
||||||
|
"bucket": env["S3_BUCKET"],
|
||||||
|
"db": {
|
||||||
|
"host": db_host,
|
||||||
|
"port": db_port,
|
||||||
|
"dbname": db_name,
|
||||||
|
"user": db_user,
|
||||||
|
"password": db_password,
|
||||||
|
},
|
||||||
|
"s3": {
|
||||||
|
"endpoint_url": env["S3_ENDPOINT"],
|
||||||
|
"aws_access_key_id": env["S3_ACCESS_KEY"],
|
||||||
|
"aws_secret_access_key": env["S3_SECRET_KEY"],
|
||||||
|
"region_name": "us-east-1",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
backups = (ROOT / args.backup_dir).resolve()
|
||||||
|
backups.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
command = [
|
||||||
|
"docker", "run",
|
||||||
|
"--rm",
|
||||||
|
"-i",
|
||||||
|
"--network", network,
|
||||||
|
"--read-only",
|
||||||
|
"--tmpfs", "/tmp:rw,noexec,nosuid,size=16m",
|
||||||
|
"--memory", "256m",
|
||||||
|
"--cpus", "1",
|
||||||
|
"--cap-drop", "ALL",
|
||||||
|
"--security-opt", "no-new-privileges",
|
||||||
|
"--mount",
|
||||||
|
(
|
||||||
|
f"type=bind,source={backups},"
|
||||||
|
"target=/backups,readonly"
|
||||||
|
),
|
||||||
|
"mtusi4-reconciler:local",
|
||||||
|
]
|
||||||
|
|
||||||
|
result = subprocess.run(
|
||||||
|
command,
|
||||||
|
cwd=ROOT,
|
||||||
|
input=json.dumps(config),
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
report = json.loads(result.stdout)
|
||||||
|
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Worker did not return JSON; "
|
||||||
|
"check image build and backup directory permissions"
|
||||||
|
) from None
|
||||||
|
|
||||||
|
target = ROOT / args.output
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
rendered = json.dumps(
|
||||||
|
report,
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
)
|
||||||
|
|
||||||
|
target.write_text(
|
||||||
|
rendered + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(rendered)
|
||||||
|
print("Report saved:", target, file=sys.stderr)
|
||||||
|
|
||||||
|
return result.returncode
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
||||||
|
except (
|
||||||
|
RuntimeError,
|
||||||
|
ValueError,
|
||||||
|
OSError,
|
||||||
|
subprocess.TimeoutExpired,
|
||||||
|
) as error:
|
||||||
|
print("FAIL:", str(error), file=sys.stderr)
|
||||||
|
raise SystemExit(1)
|
||||||
+40
-5
@@ -3,11 +3,46 @@ set -eu
|
|||||||
|
|
||||||
cd "$(dirname "$0")/.."
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
docker compose stop -t 20 api lb
|
mkdir -p .local-logs
|
||||||
|
exec > .local-logs/restart.log 2>&1
|
||||||
|
|
||||||
docker compose restart postgres redis
|
set -x
|
||||||
|
|
||||||
docker compose up -d --no-deps --no-recreate \
|
date -u
|
||||||
--wait --wait-timeout 30 postgres redis
|
|
||||||
|
|
||||||
docker compose start api lb
|
echo "Stopping API and load balancer"
|
||||||
|
docker compose stop -t 10 api lb
|
||||||
|
|
||||||
|
echo "Restarting persistent services"
|
||||||
|
docker compose restart -t 10 postgres redis minio
|
||||||
|
|
||||||
|
echo "Waiting for persistent services"
|
||||||
|
docker compose up -d \
|
||||||
|
--no-build \
|
||||||
|
--pull never \
|
||||||
|
--no-deps \
|
||||||
|
--no-recreate \
|
||||||
|
--wait \
|
||||||
|
--wait-timeout 25 \
|
||||||
|
postgres redis minio
|
||||||
|
|
||||||
|
echo "Starting API"
|
||||||
|
docker compose up -d \
|
||||||
|
--no-build \
|
||||||
|
--pull never \
|
||||||
|
--no-deps \
|
||||||
|
--no-recreate \
|
||||||
|
--wait \
|
||||||
|
--wait-timeout 45 \
|
||||||
|
api
|
||||||
|
|
||||||
|
echo "Starting load balancer without dependencies"
|
||||||
|
docker compose up -d \
|
||||||
|
--no-build \
|
||||||
|
--pull never \
|
||||||
|
--no-deps \
|
||||||
|
--no-recreate \
|
||||||
|
lb
|
||||||
|
|
||||||
|
echo "Restart completed"
|
||||||
|
date -u
|
||||||
@@ -0,0 +1,321 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import os
|
||||||
|
import struct
|
||||||
|
import uuid
|
||||||
|
import zlib
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
from client import Client, arguments, check, run
|
||||||
|
|
||||||
|
|
||||||
|
MAX_BYTES = 10 * 1024 * 1024
|
||||||
|
|
||||||
|
FIXTURE = (
|
||||||
|
Path(__file__).resolve().parent
|
||||||
|
/ "fixtures"
|
||||||
|
/ "sample.png"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def make_png(width, height, noisy=False):
|
||||||
|
"""Отдельное изображение для проверки лимита байтов или пикселей."""
|
||||||
|
|
||||||
|
def chunk(kind, data):
|
||||||
|
return (
|
||||||
|
struct.pack(">I", len(data))
|
||||||
|
+ kind
|
||||||
|
+ data
|
||||||
|
+ struct.pack(
|
||||||
|
">I",
|
||||||
|
zlib.crc32(kind + data) & 0xFFFFFFFF,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
compressor = zlib.compressobj(level=1)
|
||||||
|
compressed = bytearray()
|
||||||
|
|
||||||
|
for _ in range(height):
|
||||||
|
row = os.urandom(width) if noisy else bytes(width)
|
||||||
|
compressed.extend(compressor.compress(b"\x00" + row))
|
||||||
|
|
||||||
|
compressed.extend(compressor.flush())
|
||||||
|
|
||||||
|
header = struct.pack(
|
||||||
|
">IIBBBBB",
|
||||||
|
width, height,
|
||||||
|
8, 0, 0, 0, 0,
|
||||||
|
)
|
||||||
|
|
||||||
|
return (
|
||||||
|
b"\x89PNG\r\n\x1a\n"
|
||||||
|
+ chunk(b"IHDR", header)
|
||||||
|
+ chunk(b"IDAT", bytes(compressed))
|
||||||
|
+ chunk(b"IEND", b"")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
args = arguments("Attachment negative tests")
|
||||||
|
c = Client(args.base_url, args.ca_file, timeout=30)
|
||||||
|
|
||||||
|
alice, bob, eve, chat = c.bootstrap()
|
||||||
|
chat_path = "/api/v1/chats/" + chat["id"]
|
||||||
|
|
||||||
|
original = FIXTURE.read_bytes()
|
||||||
|
|
||||||
|
def expect_error(label, method, path, status, code, **kwargs):
|
||||||
|
response = c.call(method, path, status, **kwargs)
|
||||||
|
|
||||||
|
check(
|
||||||
|
response["json"]["error"]["code"] == code,
|
||||||
|
label + ": incorrect error code",
|
||||||
|
)
|
||||||
|
|
||||||
|
print("PASS:", label, flush=True)
|
||||||
|
|
||||||
|
def reject_upload(
|
||||||
|
label,
|
||||||
|
content,
|
||||||
|
status,
|
||||||
|
code,
|
||||||
|
mime="image/png",
|
||||||
|
filename="sample.png",
|
||||||
|
actor=None,
|
||||||
|
):
|
||||||
|
expect_error(
|
||||||
|
label,
|
||||||
|
"POST",
|
||||||
|
chat_path + "/attachments?filename="
|
||||||
|
+ quote(filename, safe=""),
|
||||||
|
status,
|
||||||
|
code,
|
||||||
|
actor=alice if actor is None else actor,
|
||||||
|
raw=content,
|
||||||
|
headers={"Content-Type": mime},
|
||||||
|
)
|
||||||
|
|
||||||
|
def reject_message(label, target_chat, ids, status, code):
|
||||||
|
expect_error(
|
||||||
|
label,
|
||||||
|
"POST",
|
||||||
|
"/api/v1/chats/" + target_chat["id"] + "/messages",
|
||||||
|
status,
|
||||||
|
code,
|
||||||
|
actor=alice,
|
||||||
|
body={"text": "", "attachment_ids": ids},
|
||||||
|
headers={"Idempotency-Key": uuid.uuid4().hex},
|
||||||
|
)
|
||||||
|
|
||||||
|
# 1. Контрольная загрузка исходного sample.png.
|
||||||
|
attachment, uploaded = c.attachment(alice, chat)
|
||||||
|
|
||||||
|
check(uploaded == original, "Unexpected fixture content")
|
||||||
|
|
||||||
|
downloaded = c.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/" + attachment["id"] + "/content",
|
||||||
|
actor=bob,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(downloaded["bytes"] == original, "Original bytes changed")
|
||||||
|
check(
|
||||||
|
attachment["sha256"] == hashlib.sha256(original).hexdigest(),
|
||||||
|
"SHA-256 mismatch",
|
||||||
|
)
|
||||||
|
|
||||||
|
print("PASS: original PNG upload and download", flush=True)
|
||||||
|
|
||||||
|
# 2. Настоящий PNG больше 10 MiB, но меньше 40 млн пикселей.
|
||||||
|
large = make_png(4096, 2700, noisy=True)
|
||||||
|
check(len(large) > MAX_BYTES, "Fixture must exceed 10 MiB")
|
||||||
|
|
||||||
|
reject_upload(
|
||||||
|
"file larger than 10 MiB",
|
||||||
|
large,
|
||||||
|
413,
|
||||||
|
"payload_too_large",
|
||||||
|
)
|
||||||
|
del large
|
||||||
|
|
||||||
|
# 3. Небольшой по байтам PNG с 40 960 000 пикселями.
|
||||||
|
many_pixels = make_png(6400, 6400)
|
||||||
|
check(len(many_pixels) < MAX_BYTES, "Fixture exceeds byte limit")
|
||||||
|
|
||||||
|
reject_upload(
|
||||||
|
"image larger than 40 million pixels",
|
||||||
|
many_pixels,
|
||||||
|
413,
|
||||||
|
"payload_too_large",
|
||||||
|
)
|
||||||
|
del many_pixels
|
||||||
|
|
||||||
|
# 4. Повреждение создаётся в памяти, sample.png не меняется.
|
||||||
|
reject_upload(
|
||||||
|
"truncated PNG",
|
||||||
|
original[:len(original) // 2],
|
||||||
|
415,
|
||||||
|
"unsupported_media_type",
|
||||||
|
)
|
||||||
|
|
||||||
|
reject_upload(
|
||||||
|
"text disguised as PNG",
|
||||||
|
b"This is not an image",
|
||||||
|
415,
|
||||||
|
"unsupported_media_type",
|
||||||
|
)
|
||||||
|
|
||||||
|
# 5. Реальный PNG объявлен как JPEG.
|
||||||
|
reject_upload(
|
||||||
|
"MIME spoofing",
|
||||||
|
original,
|
||||||
|
415,
|
||||||
|
"unsupported_media_type",
|
||||||
|
mime="image/jpeg",
|
||||||
|
filename="fake.jpg",
|
||||||
|
)
|
||||||
|
|
||||||
|
reject_upload(
|
||||||
|
"empty file",
|
||||||
|
b"",
|
||||||
|
415,
|
||||||
|
"unsupported_media_type",
|
||||||
|
)
|
||||||
|
|
||||||
|
# 6. Опасные и недопустимые имена файлов.
|
||||||
|
for filename in (
|
||||||
|
"../sample.png",
|
||||||
|
"folder\\sample.png",
|
||||||
|
"bad\x00name.png",
|
||||||
|
"bad\nname.png",
|
||||||
|
"",
|
||||||
|
"x" * 256,
|
||||||
|
):
|
||||||
|
reject_upload(
|
||||||
|
"invalid filename " + repr(filename),
|
||||||
|
original,
|
||||||
|
400,
|
||||||
|
"invalid_request",
|
||||||
|
filename=filename,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 7. Посторонний не может читать метаданные и оригинал.
|
||||||
|
for suffix in ("", "/content"):
|
||||||
|
expect_error(
|
||||||
|
"outsider GET " + (suffix or "metadata"),
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/" + attachment["id"] + suffix,
|
||||||
|
404,
|
||||||
|
"not_found",
|
||||||
|
actor=eve,
|
||||||
|
)
|
||||||
|
|
||||||
|
reject_upload(
|
||||||
|
"outsider upload",
|
||||||
|
original,
|
||||||
|
404,
|
||||||
|
"not_found",
|
||||||
|
actor=eve,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 8. Участник чата может прикрепить файл другого участника.
|
||||||
|
message = c.message(
|
||||||
|
bob,
|
||||||
|
chat,
|
||||||
|
"",
|
||||||
|
attachments=[attachment["id"]],
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
check(
|
||||||
|
message["attachment_ids"] == [attachment["id"]],
|
||||||
|
"Chat member could not attach the file",
|
||||||
|
)
|
||||||
|
print("PASS: chat member can attach the file", flush=True)
|
||||||
|
|
||||||
|
# 9. Alice состоит в обоих чатах, но перенос вложения запрещён.
|
||||||
|
second_chat = c.call(
|
||||||
|
"POST",
|
||||||
|
"/api/v1/chats",
|
||||||
|
201,
|
||||||
|
actor=alice,
|
||||||
|
body={
|
||||||
|
"title": "Second test chat",
|
||||||
|
"member_ids": [alice["id"], bob["id"]],
|
||||||
|
},
|
||||||
|
schema="Chat",
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
reject_message(
|
||||||
|
"attachment from another chat",
|
||||||
|
second_chat,
|
||||||
|
[attachment["id"]],
|
||||||
|
404,
|
||||||
|
"not_found",
|
||||||
|
)
|
||||||
|
|
||||||
|
reject_message(
|
||||||
|
"unknown attachment",
|
||||||
|
chat,
|
||||||
|
[str(uuid.uuid4())],
|
||||||
|
404,
|
||||||
|
"not_found",
|
||||||
|
)
|
||||||
|
|
||||||
|
reject_message(
|
||||||
|
"duplicate attachment IDs",
|
||||||
|
chat,
|
||||||
|
[attachment["id"], attachment["id"]],
|
||||||
|
400,
|
||||||
|
"invalid_request",
|
||||||
|
)
|
||||||
|
|
||||||
|
# 10. Идемпотентность учитывает вложения.
|
||||||
|
key = uuid.uuid4().hex
|
||||||
|
|
||||||
|
first = c.message(
|
||||||
|
alice, chat, "Photo",
|
||||||
|
key=key,
|
||||||
|
attachments=[attachment["id"]],
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
replay = c.message(
|
||||||
|
alice, chat, "Photo",
|
||||||
|
key=key,
|
||||||
|
attachments=[attachment["id"]],
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
check(first == replay, "Idempotent replay differs")
|
||||||
|
|
||||||
|
expect_error(
|
||||||
|
"same key with different attachments",
|
||||||
|
"POST",
|
||||||
|
chat_path + "/messages",
|
||||||
|
409,
|
||||||
|
"conflict",
|
||||||
|
actor=alice,
|
||||||
|
body={"text": "Photo", "attachment_ids": []},
|
||||||
|
headers={"Idempotency-Key": key},
|
||||||
|
)
|
||||||
|
|
||||||
|
text_key = uuid.uuid4().hex
|
||||||
|
|
||||||
|
first = c.message(
|
||||||
|
alice, chat, "Text only", key=text_key,
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
replay = c.message(
|
||||||
|
alice, chat, "Text only",
|
||||||
|
key=text_key,
|
||||||
|
attachments=[],
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
check(first == replay, "Missing attachment_ids differs from []")
|
||||||
|
print("PASS: missing attachment_ids equals []", flush=True)
|
||||||
|
|
||||||
|
print("PASS: attachment negative tests completed", flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run(main)
|
||||||
@@ -0,0 +1,240 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from client import Client
|
||||||
|
|
||||||
|
|
||||||
|
def check(condition, message):
|
||||||
|
if not condition:
|
||||||
|
raise RuntimeError(message)
|
||||||
|
|
||||||
|
|
||||||
|
def docker_script(script, *arguments, input_bytes=None):
|
||||||
|
command = [
|
||||||
|
"docker", "compose", "exec", "-T",
|
||||||
|
"minio",
|
||||||
|
"sh", "-ec", script,
|
||||||
|
"cleanup-check",
|
||||||
|
*arguments,
|
||||||
|
]
|
||||||
|
|
||||||
|
result = subprocess.run(
|
||||||
|
command,
|
||||||
|
input=input_bytes,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Команда MinIO завершилась ошибкой. "
|
||||||
|
"Проверь доступность minio и название bucket."
|
||||||
|
)
|
||||||
|
|
||||||
|
return result.stdout.decode("utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def configure_minio():
|
||||||
|
docker_script(
|
||||||
|
'''
|
||||||
|
mc alias set cleanupcheck http://localhost:9000 \
|
||||||
|
"$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
|
||||||
|
'''
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def put_orphan(bucket, key):
|
||||||
|
docker_script(
|
||||||
|
'mc pipe "cleanupcheck/$1/$2" >/dev/null',
|
||||||
|
bucket,
|
||||||
|
key,
|
||||||
|
input_bytes=b"synthetic cleanup test object\n",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def list_test_objects(bucket, prefix):
|
||||||
|
output = docker_script(
|
||||||
|
'mc ls --json "cleanupcheck/$1/$2"',
|
||||||
|
bucket,
|
||||||
|
prefix,
|
||||||
|
)
|
||||||
|
|
||||||
|
names = set()
|
||||||
|
|
||||||
|
for line in output.splitlines():
|
||||||
|
if not line.strip():
|
||||||
|
continue
|
||||||
|
|
||||||
|
item = json.loads(line)
|
||||||
|
|
||||||
|
check(
|
||||||
|
item.get("status") != "error",
|
||||||
|
"MinIO вернул ошибку при получении списка объектов",
|
||||||
|
)
|
||||||
|
|
||||||
|
key = item.get("key")
|
||||||
|
|
||||||
|
if key:
|
||||||
|
names.add(key.rstrip("/").split("/")[-1])
|
||||||
|
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def remove_test_object(bucket, key):
|
||||||
|
docker_script(
|
||||||
|
'mc rm --force "cleanupcheck/$1/$2" >/dev/null',
|
||||||
|
bucket,
|
||||||
|
key,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def check_download(client, actor, attachment, original):
|
||||||
|
result = client.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/" + attachment["id"] + "/content",
|
||||||
|
actor=actor,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
result["bytes"] == original,
|
||||||
|
"Корректное вложение потеряно или изменилось",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
hashlib.sha256(result["bytes"]).hexdigest()
|
||||||
|
== attachment["sha256"],
|
||||||
|
"SHA-256 корректного вложения не совпадает",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def wait_until(deadline):
|
||||||
|
remaining = deadline - time.monotonic()
|
||||||
|
|
||||||
|
if remaining > 0:
|
||||||
|
time.sleep(remaining)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument(
|
||||||
|
"--base-url",
|
||||||
|
default="http://localhost:8080",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--bucket",
|
||||||
|
default="messenger-attachments",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
check(
|
||||||
|
re.fullmatch(r"[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]", args.bucket),
|
||||||
|
"Некорректное название bucket",
|
||||||
|
)
|
||||||
|
|
||||||
|
client = Client(args.base_url, None)
|
||||||
|
|
||||||
|
alice, bob, eve, chat = client.bootstrap()
|
||||||
|
|
||||||
|
# Создаётся через API: в S3 и PostgreSQL.
|
||||||
|
attachment, original = client.attachment(alice, chat)
|
||||||
|
|
||||||
|
check_download(client, bob, attachment, original)
|
||||||
|
|
||||||
|
configure_minio()
|
||||||
|
|
||||||
|
prefix = "originals/cleanup-test-" + uuid.uuid4().hex + "/"
|
||||||
|
old_key = prefix + "old.bin"
|
||||||
|
fresh_key = prefix + "fresh.bin"
|
||||||
|
|
||||||
|
try:
|
||||||
|
put_orphan(args.bucket, old_key)
|
||||||
|
old_created = time.monotonic()
|
||||||
|
|
||||||
|
names = list_test_objects(args.bucket, prefix)
|
||||||
|
|
||||||
|
check(
|
||||||
|
"old.bin" in names,
|
||||||
|
"Тестовый объект не появился в S3",
|
||||||
|
)
|
||||||
|
|
||||||
|
print("Created API attachment and an untracked S3 object", flush=True)
|
||||||
|
|
||||||
|
wait_until(old_created + 25)
|
||||||
|
|
||||||
|
names = list_test_objects(args.bucket, prefix)
|
||||||
|
|
||||||
|
check(
|
||||||
|
"old.bin" in names,
|
||||||
|
"Объект удалён раньше grace period",
|
||||||
|
)
|
||||||
|
|
||||||
|
print("PASS: object younger than grace period is preserved", flush=True)
|
||||||
|
|
||||||
|
wait_until(old_created + 65)
|
||||||
|
|
||||||
|
put_orphan(args.bucket, fresh_key)
|
||||||
|
fresh_created = time.monotonic()
|
||||||
|
|
||||||
|
print("Waiting for cleanup of the old object...", flush=True)
|
||||||
|
|
||||||
|
deadline = fresh_created + 30
|
||||||
|
|
||||||
|
while True:
|
||||||
|
names = list_test_objects(args.bucket, prefix)
|
||||||
|
|
||||||
|
check(
|
||||||
|
"fresh.bin" in names,
|
||||||
|
"Свежий объект удалён раньше grace period",
|
||||||
|
)
|
||||||
|
|
||||||
|
if "old.bin" not in names:
|
||||||
|
break
|
||||||
|
|
||||||
|
check(
|
||||||
|
time.monotonic() < deadline,
|
||||||
|
"Старый объект не удалён. Проверь настройки и логи очистки",
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
print("PASS: old object without metadata is deleted", flush=True)
|
||||||
|
print("PASS: fresh object without metadata is preserved", flush=True)
|
||||||
|
|
||||||
|
client.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/" + attachment["id"],
|
||||||
|
actor=bob,
|
||||||
|
schema="Attachment",
|
||||||
|
)
|
||||||
|
|
||||||
|
check_download(client, bob, attachment, original)
|
||||||
|
|
||||||
|
print("PASS: attachment with metadata and its bytes are preserved", flush=True)
|
||||||
|
print("PASS: cleanup scenario completed", flush=True)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
|
||||||
|
for key in (old_key, fresh_key):
|
||||||
|
try:
|
||||||
|
remove_test_object(args.bucket, key)
|
||||||
|
except Exception as error:
|
||||||
|
print(
|
||||||
|
f"WARNING: final cleanup failed for {key}: {error}",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
try:
|
||||||
|
main()
|
||||||
|
except Exception as error:
|
||||||
|
print("FAIL:", error, flush=True)
|
||||||
|
raise SystemExit(1)
|
||||||
@@ -0,0 +1,686 @@
|
|||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from client import Client, arguments, check, run
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
BUCKET = None
|
||||||
|
|
||||||
|
|
||||||
|
def command(args, data=None, timeout=30):
|
||||||
|
result = subprocess.run(
|
||||||
|
args,
|
||||||
|
cwd=ROOT,
|
||||||
|
input=data,
|
||||||
|
capture_output=True,
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
result.returncode == 0,
|
||||||
|
"Command failed: " + " ".join(args[:3]),
|
||||||
|
)
|
||||||
|
|
||||||
|
return result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def sql(statement):
|
||||||
|
result = command(
|
||||||
|
[
|
||||||
|
"docker",
|
||||||
|
"compose",
|
||||||
|
"exec",
|
||||||
|
"-T",
|
||||||
|
"postgres",
|
||||||
|
"sh",
|
||||||
|
"-ec",
|
||||||
|
(
|
||||||
|
'exec psql -X -A -t -v ON_ERROR_STOP=1 '
|
||||||
|
'-U "$POSTGRES_USER" '
|
||||||
|
'-d "${POSTGRES_DB:-$POSTGRES_USER}"'
|
||||||
|
),
|
||||||
|
],
|
||||||
|
statement.encode(),
|
||||||
|
)
|
||||||
|
|
||||||
|
return result.decode().strip()
|
||||||
|
|
||||||
|
|
||||||
|
def mc(*args, data=None):
|
||||||
|
return command(
|
||||||
|
[
|
||||||
|
"docker",
|
||||||
|
"compose",
|
||||||
|
"exec",
|
||||||
|
"-T",
|
||||||
|
"minio",
|
||||||
|
"sh",
|
||||||
|
"-ec",
|
||||||
|
(
|
||||||
|
"mc alias set reconcile_test "
|
||||||
|
"http://localhost:9000 "
|
||||||
|
'"$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" '
|
||||||
|
'>/dev/null; exec mc "$@"'
|
||||||
|
),
|
||||||
|
"sh",
|
||||||
|
*args,
|
||||||
|
],
|
||||||
|
data,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def target(key):
|
||||||
|
return "reconcile_test/" + BUCKET + "/" + key
|
||||||
|
|
||||||
|
|
||||||
|
def object_names(chat_id):
|
||||||
|
raw = mc(
|
||||||
|
"ls",
|
||||||
|
"--recursive",
|
||||||
|
"--json",
|
||||||
|
target("originals/" + chat_id + "/"),
|
||||||
|
)
|
||||||
|
|
||||||
|
names = set()
|
||||||
|
|
||||||
|
for line in raw.decode().splitlines():
|
||||||
|
if not line.strip():
|
||||||
|
continue
|
||||||
|
|
||||||
|
item = json.loads(line)
|
||||||
|
|
||||||
|
check(
|
||||||
|
item.get("status") == "success",
|
||||||
|
"Cannot list test S3 prefix",
|
||||||
|
)
|
||||||
|
|
||||||
|
if item.get("type") == "file":
|
||||||
|
names.add(
|
||||||
|
item["key"].rstrip("/").rsplit("/", 1)[-1]
|
||||||
|
)
|
||||||
|
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def inspect_api():
|
||||||
|
ids = command([
|
||||||
|
"docker",
|
||||||
|
"compose",
|
||||||
|
"ps",
|
||||||
|
"-q",
|
||||||
|
"api",
|
||||||
|
]).decode().split()
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(ids) == 2,
|
||||||
|
"Start exactly two API replicas",
|
||||||
|
)
|
||||||
|
|
||||||
|
containers = json.loads(command([
|
||||||
|
"docker",
|
||||||
|
"inspect",
|
||||||
|
*ids,
|
||||||
|
]))
|
||||||
|
|
||||||
|
buckets = set()
|
||||||
|
|
||||||
|
for container in containers:
|
||||||
|
env = dict(
|
||||||
|
value.split("=", 1)
|
||||||
|
for value in container["Config"]["Env"]
|
||||||
|
if "=" in value
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
env.get("APP_CLEANUP_ENABLED", "").lower()
|
||||||
|
== "false",
|
||||||
|
"Disable AttachmentCleanupJob "
|
||||||
|
"on both replicas first",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
bool(env.get("S3_BUCKET")),
|
||||||
|
"S3_BUCKET is missing",
|
||||||
|
)
|
||||||
|
|
||||||
|
buckets.add(env["S3_BUCKET"])
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(buckets) == 1,
|
||||||
|
"API replicas use different buckets",
|
||||||
|
)
|
||||||
|
|
||||||
|
return buckets.pop()
|
||||||
|
|
||||||
|
|
||||||
|
def snapshot(chat_id):
|
||||||
|
# chat_id предварительно проверен через uuid.UUID.
|
||||||
|
return sql(
|
||||||
|
"SELECT coalesce("
|
||||||
|
"json_agg(a ORDER BY id), '[]'::json"
|
||||||
|
") FROM attachments a "
|
||||||
|
"WHERE chat_id='" + chat_id + "'::uuid;"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def reconcile(
|
||||||
|
chat_id,
|
||||||
|
backups,
|
||||||
|
directory,
|
||||||
|
label,
|
||||||
|
apply=False,
|
||||||
|
):
|
||||||
|
path = directory / (label + ".json")
|
||||||
|
|
||||||
|
result = subprocess.run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
"-X",
|
||||||
|
"utf8",
|
||||||
|
str(ROOT / "scripts/reconcile.py"),
|
||||||
|
"--apply" if apply else "--dry-run",
|
||||||
|
"--chat-id",
|
||||||
|
chat_id,
|
||||||
|
"--grace-seconds",
|
||||||
|
"60",
|
||||||
|
"--max-seconds",
|
||||||
|
"120",
|
||||||
|
"--backup-dir",
|
||||||
|
str(backups),
|
||||||
|
"--output",
|
||||||
|
str(path),
|
||||||
|
],
|
||||||
|
cwd=ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=180,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
result.returncode in (0, 2),
|
||||||
|
"Reconciler failed at "
|
||||||
|
+ label
|
||||||
|
+ "; inspect "
|
||||||
|
+ str(path),
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
path.exists(),
|
||||||
|
"Reconciler did not save report",
|
||||||
|
)
|
||||||
|
|
||||||
|
report = json.loads(
|
||||||
|
path.read_text(encoding="utf-8")
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
report["status"] in (
|
||||||
|
"completed",
|
||||||
|
"needs_attention",
|
||||||
|
),
|
||||||
|
"Scan failed",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
report["counts"].get("busy", 0) == 0,
|
||||||
|
"Object lock was busy; repeat experiment",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
report["counts"].get("size_mismatches", 0) == 0,
|
||||||
|
"Unexpected size mismatch",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
label + ": " + json.dumps(report["counts"]),
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def count(report, name):
|
||||||
|
return report["counts"].get(name, 0)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
global BUCKET
|
||||||
|
|
||||||
|
args = arguments(
|
||||||
|
"Lab4 reconciler: dry-run, repair "
|
||||||
|
"and repeated execution"
|
||||||
|
)
|
||||||
|
|
||||||
|
BUCKET = inspect_api()
|
||||||
|
|
||||||
|
c = Client(
|
||||||
|
args.base_url,
|
||||||
|
args.ca_file,
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
alice, bob, _, chat = c.bootstrap()
|
||||||
|
chat_id = str(uuid.UUID(chat["id"]))
|
||||||
|
|
||||||
|
stable, content = c.attachment(alice, chat)
|
||||||
|
missing, missing_bytes = c.attachment(alice, chat)
|
||||||
|
|
||||||
|
message = c.message(
|
||||||
|
alice,
|
||||||
|
chat,
|
||||||
|
"Reconciler experiment",
|
||||||
|
attachments=[
|
||||||
|
stable["id"],
|
||||||
|
missing["id"],
|
||||||
|
],
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
missing_id = str(uuid.UUID(missing["id"]))
|
||||||
|
|
||||||
|
key = sql(
|
||||||
|
"SELECT object_key FROM attachments "
|
||||||
|
"WHERE id='" + missing_id + "'::uuid;"
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
key.startswith("originals/" + chat_id + "/"),
|
||||||
|
"Unexpected original key",
|
||||||
|
)
|
||||||
|
|
||||||
|
prefix = "originals/" + chat_id + "/"
|
||||||
|
|
||||||
|
old_key = (
|
||||||
|
prefix + "orphan-old-" + uuid.uuid4().hex
|
||||||
|
)
|
||||||
|
|
||||||
|
fresh_key = None
|
||||||
|
fresh_started = None
|
||||||
|
|
||||||
|
# Запоминаем все созданные orphan-ключи,
|
||||||
|
# чтобы попытаться убрать их при исключении.
|
||||||
|
synthetic_keys = {old_key}
|
||||||
|
|
||||||
|
backups = (
|
||||||
|
ROOT / "backups" / ("reconcile-" + chat_id)
|
||||||
|
)
|
||||||
|
|
||||||
|
directory = (
|
||||||
|
ROOT / "results" / ("reconcile-" + chat_id)
|
||||||
|
)
|
||||||
|
|
||||||
|
backups.mkdir(parents=True, exist_ok=False)
|
||||||
|
directory.mkdir(parents=True, exist_ok=False)
|
||||||
|
|
||||||
|
metadata_before = snapshot(chat_id)
|
||||||
|
|
||||||
|
needs_restore = False
|
||||||
|
completed = False
|
||||||
|
|
||||||
|
def verify_metadata():
|
||||||
|
check(
|
||||||
|
snapshot(chat_id) == metadata_before,
|
||||||
|
"Attachment metadata changed",
|
||||||
|
)
|
||||||
|
|
||||||
|
page = c.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/chats/"
|
||||||
|
+ chat_id
|
||||||
|
+ "/messages?limit=100",
|
||||||
|
actor=bob,
|
||||||
|
schema="MessagePage",
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
found = next(
|
||||||
|
(
|
||||||
|
item
|
||||||
|
for item in page["items"]
|
||||||
|
if item["id"] == message["id"]
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
found == message,
|
||||||
|
"Message or attachment references changed",
|
||||||
|
)
|
||||||
|
|
||||||
|
def prepare_fresh():
|
||||||
|
nonlocal fresh_key, fresh_started
|
||||||
|
|
||||||
|
# Предыдущий свежий объект больше не нужен.
|
||||||
|
# Его сохранность проверена предыдущим этапом.
|
||||||
|
if fresh_key is not None:
|
||||||
|
previous_name = fresh_key.rsplit("/", 1)[-1]
|
||||||
|
|
||||||
|
if previous_name in object_names(chat_id):
|
||||||
|
mc("rm", target(fresh_key))
|
||||||
|
|
||||||
|
fresh_key = (
|
||||||
|
prefix + "orphan-fresh-" + uuid.uuid4().hex
|
||||||
|
)
|
||||||
|
|
||||||
|
synthetic_keys.add(fresh_key)
|
||||||
|
|
||||||
|
# Начинаем отсчёт до PUT: это консервативная
|
||||||
|
# верхняя оценка возраста объекта.
|
||||||
|
fresh_started = time.monotonic()
|
||||||
|
|
||||||
|
mc(
|
||||||
|
"pipe",
|
||||||
|
target(fresh_key),
|
||||||
|
data=content,
|
||||||
|
)
|
||||||
|
|
||||||
|
def verify_fresh():
|
||||||
|
check(
|
||||||
|
fresh_key is not None
|
||||||
|
and fresh_started is not None,
|
||||||
|
"Fresh test object was not created",
|
||||||
|
)
|
||||||
|
|
||||||
|
names = object_names(chat_id)
|
||||||
|
elapsed = time.monotonic() - fresh_started
|
||||||
|
|
||||||
|
check(
|
||||||
|
elapsed < 55,
|
||||||
|
"A single reconciliation stage exceeded "
|
||||||
|
"the fresh-object timing budget; "
|
||||||
|
"repeat on an idle stand",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
fresh_key.rsplit("/", 1)[-1] in names,
|
||||||
|
"Fresh orphan was removed",
|
||||||
|
)
|
||||||
|
|
||||||
|
def download(attachment, expected):
|
||||||
|
result = c.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/"
|
||||||
|
+ attachment["id"]
|
||||||
|
+ "/content",
|
||||||
|
actor=bob,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
result["bytes"] == expected,
|
||||||
|
"Original bytes changed",
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
mc(
|
||||||
|
"pipe",
|
||||||
|
target(old_key),
|
||||||
|
data=content,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Old orphan created. Waiting 65 seconds "
|
||||||
|
"for the real grace period...",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(65)
|
||||||
|
|
||||||
|
prepare_fresh()
|
||||||
|
|
||||||
|
# Байты пока доступны только тесту.
|
||||||
|
# Reconciler ещё не получает резервную копию.
|
||||||
|
needs_restore = True
|
||||||
|
mc("rm", target(key))
|
||||||
|
|
||||||
|
before = object_names(chat_id)
|
||||||
|
|
||||||
|
dry = reconcile(
|
||||||
|
chat_id,
|
||||||
|
backups,
|
||||||
|
directory,
|
||||||
|
"01-dry-run",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
count(dry, "would_delete_orphans") == 1,
|
||||||
|
"Old orphan not detected",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
count(dry, "unresolved_missing") == 1,
|
||||||
|
"Missing original not detected",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
object_names(chat_id) == before,
|
||||||
|
"Dry-run changed S3 objects",
|
||||||
|
)
|
||||||
|
|
||||||
|
verify_metadata()
|
||||||
|
verify_fresh()
|
||||||
|
download(stable, content)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: dry-run found both faults "
|
||||||
|
"and changed no test data",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Для следующего этапа создаём новый свежий объект.
|
||||||
|
prepare_fresh()
|
||||||
|
|
||||||
|
applied = reconcile(
|
||||||
|
chat_id,
|
||||||
|
backups,
|
||||||
|
directory,
|
||||||
|
"02-apply-no-backup",
|
||||||
|
apply=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
count(applied, "deleted_orphans") == 1,
|
||||||
|
"Old orphan not deleted",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
count(applied, "unresolved_missing") == 1,
|
||||||
|
"Missing original incorrectly repaired",
|
||||||
|
)
|
||||||
|
|
||||||
|
names = object_names(chat_id)
|
||||||
|
|
||||||
|
check(
|
||||||
|
old_key.rsplit("/", 1)[-1] not in names,
|
||||||
|
"Old orphan still exists",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
key.rsplit("/", 1)[-1] not in names,
|
||||||
|
"Missing original unexpectedly exists",
|
||||||
|
)
|
||||||
|
|
||||||
|
verify_metadata()
|
||||||
|
verify_fresh()
|
||||||
|
|
||||||
|
c.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/"
|
||||||
|
+ missing_id
|
||||||
|
+ "/content",
|
||||||
|
expected=503,
|
||||||
|
actor=bob,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: old orphan deleted; missing original "
|
||||||
|
"reported without metadata loss",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Предоставляем reconciler достоверную копию.
|
||||||
|
backup_file = (
|
||||||
|
backups / (missing["sha256"] + ".bin")
|
||||||
|
)
|
||||||
|
|
||||||
|
backup_file.write_bytes(missing_bytes)
|
||||||
|
|
||||||
|
prepare_fresh()
|
||||||
|
before = object_names(chat_id)
|
||||||
|
|
||||||
|
planned = reconcile(
|
||||||
|
chat_id,
|
||||||
|
backups,
|
||||||
|
directory,
|
||||||
|
"03-dry-run-with-backup",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
count(planned, "would_restore") == 1,
|
||||||
|
"Valid backup not recognized",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
object_names(chat_id) == before,
|
||||||
|
"Dry-run restored an object",
|
||||||
|
)
|
||||||
|
|
||||||
|
verify_metadata()
|
||||||
|
verify_fresh()
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: dry-run plans restoration "
|
||||||
|
"without changing objects",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
prepare_fresh()
|
||||||
|
|
||||||
|
restored = reconcile(
|
||||||
|
chat_id,
|
||||||
|
backups,
|
||||||
|
directory,
|
||||||
|
"04-restore",
|
||||||
|
apply=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
count(restored, "restored") == 1,
|
||||||
|
"Original not restored",
|
||||||
|
)
|
||||||
|
|
||||||
|
download(missing, missing_bytes)
|
||||||
|
needs_restore = False
|
||||||
|
|
||||||
|
download(stable, content)
|
||||||
|
verify_metadata()
|
||||||
|
verify_fresh()
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: original restored byte-for-byte; "
|
||||||
|
"used attachment and message preserved",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
prepare_fresh()
|
||||||
|
|
||||||
|
repeated = reconcile(
|
||||||
|
chat_id,
|
||||||
|
backups,
|
||||||
|
directory,
|
||||||
|
"05-repeat",
|
||||||
|
apply=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
for name in (
|
||||||
|
"deleted_orphans",
|
||||||
|
"restored",
|
||||||
|
"unresolved_missing",
|
||||||
|
"missing_objects",
|
||||||
|
):
|
||||||
|
check(
|
||||||
|
count(repeated, name) == 0,
|
||||||
|
"Repeated run is not clean: " + name,
|
||||||
|
)
|
||||||
|
|
||||||
|
verify_metadata()
|
||||||
|
verify_fresh()
|
||||||
|
|
||||||
|
download(stable, content)
|
||||||
|
download(missing, missing_bytes)
|
||||||
|
|
||||||
|
completed = True
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: repeated apply makes "
|
||||||
|
"no additional changes",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Reports: " + str(directory),
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
# При исключении пытаемся восстановить оригинал,
|
||||||
|
# намеренно удалённый этим тестом.
|
||||||
|
if needs_restore:
|
||||||
|
try:
|
||||||
|
mc(
|
||||||
|
"pipe",
|
||||||
|
target(key),
|
||||||
|
data=missing_bytes,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Test original restored "
|
||||||
|
"during final cleanup",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
print(
|
||||||
|
"WARNING: restore test original manually "
|
||||||
|
"from backup; key=" + key,
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
(
|
||||||
|
backups / (missing["sha256"] + ".bin")
|
||||||
|
).write_bytes(missing_bytes)
|
||||||
|
|
||||||
|
cleanup_ok = True
|
||||||
|
|
||||||
|
# Удаляем только существующие синтетические объекты.
|
||||||
|
# Уже удалённый reconciler старый orphan — не ошибка.
|
||||||
|
for orphan in sorted(synthetic_keys):
|
||||||
|
try:
|
||||||
|
name = orphan.rsplit("/", 1)[-1]
|
||||||
|
|
||||||
|
if name in object_names(chat_id):
|
||||||
|
mc("rm", target(orphan))
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
cleanup_ok = False
|
||||||
|
|
||||||
|
print(
|
||||||
|
"WARNING: cannot remove synthetic orphan: "
|
||||||
|
+ orphan,
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
if completed:
|
||||||
|
check(
|
||||||
|
cleanup_ok,
|
||||||
|
"Scenario passed but final "
|
||||||
|
"test-object cleanup failed",
|
||||||
|
)
|
||||||
|
|
||||||
|
print("PASS: reconciler experiment completed")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run(main)
|
||||||
@@ -0,0 +1,561 @@
|
|||||||
|
import csv
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
from pathlib import Path
|
||||||
|
import random
|
||||||
|
import statistics
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from client import Client, arguments, check, run
|
||||||
|
import reconcile_check as helpers
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
|
|
||||||
|
def options(parser):
|
||||||
|
parser.add_argument(
|
||||||
|
"--sizes",
|
||||||
|
type=int,
|
||||||
|
nargs="+",
|
||||||
|
default=[10, 100, 1000],
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--repeats",
|
||||||
|
type=int,
|
||||||
|
default=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"--max-seconds",
|
||||||
|
type=int,
|
||||||
|
default=600,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def digest_metadata(chat_id):
|
||||||
|
snapshot = helpers.snapshot(chat_id)
|
||||||
|
|
||||||
|
return hashlib.sha256(
|
||||||
|
snapshot.encode()
|
||||||
|
).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def environment_info():
|
||||||
|
ids = helpers.command([
|
||||||
|
"docker",
|
||||||
|
"compose",
|
||||||
|
"ps",
|
||||||
|
"-q",
|
||||||
|
"api",
|
||||||
|
"postgres",
|
||||||
|
"minio",
|
||||||
|
]).decode().split()
|
||||||
|
|
||||||
|
containers = json.loads(
|
||||||
|
helpers.command([
|
||||||
|
"docker",
|
||||||
|
"inspect",
|
||||||
|
*ids,
|
||||||
|
])
|
||||||
|
)
|
||||||
|
|
||||||
|
cpu_count = int(helpers.command([
|
||||||
|
"docker",
|
||||||
|
"info",
|
||||||
|
"--format",
|
||||||
|
"{{.NCPU}}",
|
||||||
|
]))
|
||||||
|
|
||||||
|
memory_bytes = int(helpers.command([
|
||||||
|
"docker",
|
||||||
|
"info",
|
||||||
|
"--format",
|
||||||
|
"{{.MemTotal}}",
|
||||||
|
]))
|
||||||
|
|
||||||
|
worker_image = helpers.command([
|
||||||
|
"docker",
|
||||||
|
"image",
|
||||||
|
"inspect",
|
||||||
|
"--format",
|
||||||
|
"{{.Id}}",
|
||||||
|
"mtusi4-reconciler:local",
|
||||||
|
]).decode().strip()
|
||||||
|
|
||||||
|
# Сохраняем только параметры окружения,
|
||||||
|
# необходимые для воспроизводимости.
|
||||||
|
# Credentials и полный docker inspect не записываем.
|
||||||
|
services = []
|
||||||
|
|
||||||
|
for container in containers:
|
||||||
|
services.append({
|
||||||
|
"service": container["Config"]["Labels"].get(
|
||||||
|
"com.docker.compose.service"
|
||||||
|
),
|
||||||
|
"image_id": container["Image"],
|
||||||
|
"memory_limit_bytes": (
|
||||||
|
container["HostConfig"]["Memory"]
|
||||||
|
),
|
||||||
|
"nano_cpus": (
|
||||||
|
container["HostConfig"]["NanoCpus"]
|
||||||
|
),
|
||||||
|
})
|
||||||
|
|
||||||
|
return {
|
||||||
|
"docker_cpus": cpu_count,
|
||||||
|
"docker_memory_bytes": memory_bytes,
|
||||||
|
"worker_image_id": worker_image,
|
||||||
|
"worker_memory_limit_bytes": 256 * 1024 * 1024,
|
||||||
|
"worker_cpu_limit": 1,
|
||||||
|
"services": services,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
args = arguments(
|
||||||
|
"Measure reconciler traversal cost",
|
||||||
|
options,
|
||||||
|
)
|
||||||
|
|
||||||
|
sizes = sorted(set(args.sizes))
|
||||||
|
|
||||||
|
check(
|
||||||
|
all(1 <= size <= 5000 for size in sizes),
|
||||||
|
"Sizes must be between 1 and 5000",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
1 <= args.repeats <= 10,
|
||||||
|
"Repeats must be between 1 and 10",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
args.max_seconds >= 1,
|
||||||
|
"max-seconds must be positive",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Проверяет две API-реплики и отключённую очистку.
|
||||||
|
helpers.BUCKET = helpers.inspect_api()
|
||||||
|
|
||||||
|
fixture = (
|
||||||
|
ROOT / "tests/fixtures/sample.png"
|
||||||
|
).read_bytes()
|
||||||
|
|
||||||
|
fixture_sha = hashlib.sha256(fixture).hexdigest()
|
||||||
|
|
||||||
|
directory = (
|
||||||
|
ROOT
|
||||||
|
/ "results"
|
||||||
|
/ ("reconcile-cost-" + uuid.uuid4().hex)
|
||||||
|
)
|
||||||
|
|
||||||
|
backups = directory / "empty-backups"
|
||||||
|
backups.mkdir(parents=True, exist_ok=False)
|
||||||
|
|
||||||
|
report = {
|
||||||
|
"passed": False,
|
||||||
|
"hypothesis": (
|
||||||
|
"Traversal calls grow approximately linearly "
|
||||||
|
"with attachment count; wall time also "
|
||||||
|
"contains fixed overhead."
|
||||||
|
),
|
||||||
|
"sizes": sizes,
|
||||||
|
"repeats": args.repeats,
|
||||||
|
"grace_seconds": 60,
|
||||||
|
"fixture_size_bytes": len(fixture),
|
||||||
|
"fixture_sha256": fixture_sha,
|
||||||
|
"environment": environment_info(),
|
||||||
|
"groups": [],
|
||||||
|
"samples": [],
|
||||||
|
"summary": [],
|
||||||
|
"method": (
|
||||||
|
"Separate chats; API uploads; real grace wait; "
|
||||||
|
"one warm-up per size; shuffled measured rounds; "
|
||||||
|
"dry-run only."
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
groups = []
|
||||||
|
|
||||||
|
client = Client(
|
||||||
|
args.base_url,
|
||||||
|
args.ca_file,
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
def save():
|
||||||
|
(directory / "summary.json").write_text(
|
||||||
|
json.dumps(
|
||||||
|
report,
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
fields = [
|
||||||
|
"attachments",
|
||||||
|
"trial",
|
||||||
|
"worker_seconds",
|
||||||
|
"launcher_seconds",
|
||||||
|
"sql_calls",
|
||||||
|
"s3_calls",
|
||||||
|
"s3_list_calls",
|
||||||
|
"s3_head_calls",
|
||||||
|
]
|
||||||
|
|
||||||
|
with (directory / "samples.csv").open(
|
||||||
|
"w",
|
||||||
|
encoding="utf-8",
|
||||||
|
newline="",
|
||||||
|
) as stream:
|
||||||
|
writer = csv.DictWriter(
|
||||||
|
stream,
|
||||||
|
fieldnames=fields,
|
||||||
|
)
|
||||||
|
|
||||||
|
writer.writeheader()
|
||||||
|
writer.writerows(report["samples"])
|
||||||
|
|
||||||
|
def scan(group, label):
|
||||||
|
path = directory / (
|
||||||
|
f'n{group["n"]}-{label}.json'
|
||||||
|
)
|
||||||
|
|
||||||
|
started = time.monotonic()
|
||||||
|
|
||||||
|
result = subprocess.run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
"-X",
|
||||||
|
"utf8",
|
||||||
|
str(ROOT / "scripts/reconcile.py"),
|
||||||
|
"--dry-run",
|
||||||
|
"--chat-id",
|
||||||
|
group["chat_id"],
|
||||||
|
"--grace-seconds",
|
||||||
|
"60",
|
||||||
|
"--max-seconds",
|
||||||
|
str(args.max_seconds),
|
||||||
|
"--backup-dir",
|
||||||
|
str(backups),
|
||||||
|
"--output",
|
||||||
|
str(path),
|
||||||
|
],
|
||||||
|
cwd=ROOT,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=args.max_seconds + 180,
|
||||||
|
)
|
||||||
|
|
||||||
|
wall = time.monotonic() - started
|
||||||
|
|
||||||
|
check(
|
||||||
|
result.returncode == 0,
|
||||||
|
"Scan failed: " + str(path),
|
||||||
|
)
|
||||||
|
|
||||||
|
data = json.loads(
|
||||||
|
path.read_text(encoding="utf-8")
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
data["status"] == "completed"
|
||||||
|
and data["mode"] == "dry-run",
|
||||||
|
"Unexpected scan status",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
data.get("chat_id") == group["chat_id"],
|
||||||
|
"Wrong scan scope",
|
||||||
|
)
|
||||||
|
|
||||||
|
counts = data["counts"]
|
||||||
|
|
||||||
|
# Каждый объект и каждая запись должны пройти
|
||||||
|
# полноценную проверку после grace period.
|
||||||
|
for field in (
|
||||||
|
"objects_scanned",
|
||||||
|
"referenced_objects",
|
||||||
|
"metadata_scanned",
|
||||||
|
"present_metadata",
|
||||||
|
):
|
||||||
|
check(
|
||||||
|
counts.get(field, 0) == group["n"],
|
||||||
|
"Incomplete scan: " + field,
|
||||||
|
)
|
||||||
|
|
||||||
|
for field in (
|
||||||
|
"fresh_objects",
|
||||||
|
"fresh_metadata",
|
||||||
|
"busy",
|
||||||
|
"missing_objects",
|
||||||
|
"unresolved_missing",
|
||||||
|
"size_mismatches",
|
||||||
|
"would_delete_orphans",
|
||||||
|
"deleted_orphans",
|
||||||
|
"restored",
|
||||||
|
):
|
||||||
|
check(
|
||||||
|
counts.get(field, 0) == 0,
|
||||||
|
"Unexpected outcome: " + field,
|
||||||
|
)
|
||||||
|
|
||||||
|
calls = data["s3_operation_calls"]
|
||||||
|
|
||||||
|
check(
|
||||||
|
calls.get("put_object", 0) == 0
|
||||||
|
and calls.get("delete_object", 0) == 0,
|
||||||
|
"Dry-run attempted mutations",
|
||||||
|
)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"attachments": group["n"],
|
||||||
|
"trial": label,
|
||||||
|
"worker_seconds": data["elapsed_seconds"],
|
||||||
|
"launcher_seconds": round(wall, 3),
|
||||||
|
"sql_calls": data["sql_statement_calls"],
|
||||||
|
"s3_calls": sum(calls.values()),
|
||||||
|
"s3_list_calls": calls.get(
|
||||||
|
"list_objects_v2", 0
|
||||||
|
),
|
||||||
|
"s3_head_calls": calls.get(
|
||||||
|
"head_object", 0
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
save()
|
||||||
|
|
||||||
|
try:
|
||||||
|
setup_started = time.monotonic()
|
||||||
|
|
||||||
|
# Каждый размер получает отдельный чат.
|
||||||
|
for size in sizes:
|
||||||
|
alice, bob, _, chat = client.bootstrap()
|
||||||
|
chat_id = str(uuid.UUID(chat["id"]))
|
||||||
|
|
||||||
|
entry = {
|
||||||
|
"attachments": size,
|
||||||
|
"chat_id": chat_id,
|
||||||
|
"uploaded": 0,
|
||||||
|
}
|
||||||
|
|
||||||
|
report["groups"].append(entry)
|
||||||
|
save()
|
||||||
|
|
||||||
|
sample_attachments = []
|
||||||
|
|
||||||
|
# Последовательные загрузки.
|
||||||
|
# Автоматических повторов POST нет.
|
||||||
|
for index in range(size):
|
||||||
|
attachment, content = client.attachment(
|
||||||
|
alice,
|
||||||
|
chat,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
content == fixture,
|
||||||
|
"Fixture changed during experiment",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Для контрольного скачивания сохраняем
|
||||||
|
# первое и последнее вложение.
|
||||||
|
if index in {0, size - 1}:
|
||||||
|
sample_attachments.append(attachment)
|
||||||
|
|
||||||
|
entry["uploaded"] = index + 1
|
||||||
|
|
||||||
|
if (
|
||||||
|
(index + 1) % 100 == 0
|
||||||
|
or index + 1 == size
|
||||||
|
):
|
||||||
|
print(
|
||||||
|
f"Preparing N={size}: "
|
||||||
|
f"{index + 1}/{size}",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
save()
|
||||||
|
|
||||||
|
message = client.message(
|
||||||
|
alice,
|
||||||
|
chat,
|
||||||
|
"Reconciler cost experiment",
|
||||||
|
attachments=[
|
||||||
|
attachment["id"]
|
||||||
|
for attachment in sample_attachments
|
||||||
|
],
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
groups.append({
|
||||||
|
"n": size,
|
||||||
|
"chat_id": chat_id,
|
||||||
|
"bob": bob,
|
||||||
|
"samples": sample_attachments,
|
||||||
|
"message": message,
|
||||||
|
"metadata_digest": digest_metadata(chat_id),
|
||||||
|
})
|
||||||
|
|
||||||
|
report["setup_seconds"] = round(
|
||||||
|
time.monotonic() - setup_started,
|
||||||
|
3,
|
||||||
|
)
|
||||||
|
|
||||||
|
save()
|
||||||
|
|
||||||
|
print(
|
||||||
|
"All uploads completed. Waiting 65 seconds "
|
||||||
|
"for the real grace period...",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(65)
|
||||||
|
|
||||||
|
# Прогрев сохраняется отдельными JSON,
|
||||||
|
# но не включается в итоговую статистику.
|
||||||
|
for group in groups:
|
||||||
|
scan(group, "warmup")
|
||||||
|
|
||||||
|
print(
|
||||||
|
f'Warm-up N={group["n"]}: complete',
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# В каждом раунде меняем порядок объёмов.
|
||||||
|
# Фиксированный seed делает порядок повторяемым.
|
||||||
|
rng = random.Random(1729)
|
||||||
|
|
||||||
|
for trial in range(1, args.repeats + 1):
|
||||||
|
order = list(groups)
|
||||||
|
rng.shuffle(order)
|
||||||
|
|
||||||
|
for group in order:
|
||||||
|
sample = scan(
|
||||||
|
group,
|
||||||
|
f"run-{trial}",
|
||||||
|
)
|
||||||
|
|
||||||
|
report["samples"].append(sample)
|
||||||
|
save()
|
||||||
|
|
||||||
|
print(
|
||||||
|
f'N={group["n"]}, run={trial}: '
|
||||||
|
f'worker={sample["worker_seconds"]}s, '
|
||||||
|
f'launcher={sample["launcher_seconds"]}s, '
|
||||||
|
f'SQL={sample["sql_calls"]}, '
|
||||||
|
f'S3={sample["s3_calls"]}',
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Контроль после измерений.
|
||||||
|
for group in groups:
|
||||||
|
check(
|
||||||
|
digest_metadata(group["chat_id"])
|
||||||
|
== group["metadata_digest"],
|
||||||
|
"Metadata changed during benchmark",
|
||||||
|
)
|
||||||
|
|
||||||
|
for attachment in group["samples"]:
|
||||||
|
downloaded = client.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/"
|
||||||
|
+ attachment["id"]
|
||||||
|
+ "/content",
|
||||||
|
actor=group["bob"],
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
downloaded["bytes"] == fixture,
|
||||||
|
"Sample original changed",
|
||||||
|
)
|
||||||
|
|
||||||
|
page = client.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/chats/"
|
||||||
|
+ group["chat_id"]
|
||||||
|
+ "/messages?limit=100",
|
||||||
|
actor=group["bob"],
|
||||||
|
schema="MessagePage",
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
check(
|
||||||
|
group["message"] in page["items"],
|
||||||
|
"Sample message changed",
|
||||||
|
)
|
||||||
|
|
||||||
|
for size in sizes:
|
||||||
|
samples = [
|
||||||
|
sample
|
||||||
|
for sample in report["samples"]
|
||||||
|
if sample["attachments"] == size
|
||||||
|
]
|
||||||
|
|
||||||
|
worker_times = [
|
||||||
|
sample["worker_seconds"]
|
||||||
|
for sample in samples
|
||||||
|
]
|
||||||
|
|
||||||
|
launcher_times = [
|
||||||
|
sample["launcher_seconds"]
|
||||||
|
for sample in samples
|
||||||
|
]
|
||||||
|
|
||||||
|
report["summary"].append({
|
||||||
|
"attachments": size,
|
||||||
|
"worker_seconds_min": min(worker_times),
|
||||||
|
"worker_seconds_median": round(
|
||||||
|
statistics.median(worker_times),
|
||||||
|
3,
|
||||||
|
),
|
||||||
|
"worker_seconds_max": max(worker_times),
|
||||||
|
"launcher_seconds_median": round(
|
||||||
|
statistics.median(launcher_times),
|
||||||
|
3,
|
||||||
|
),
|
||||||
|
"sql_calls_observed": sorted({
|
||||||
|
sample["sql_calls"]
|
||||||
|
for sample in samples
|
||||||
|
}),
|
||||||
|
"s3_calls_observed": sorted({
|
||||||
|
sample["s3_calls"]
|
||||||
|
for sample in samples
|
||||||
|
}),
|
||||||
|
"sql_calls_expected_for_current_algorithm": (
|
||||||
|
5 * size
|
||||||
|
+ math.ceil(size / 500)
|
||||||
|
+ 2
|
||||||
|
),
|
||||||
|
"s3_calls_expected_for_current_algorithm": (
|
||||||
|
size
|
||||||
|
+ math.ceil(size / 500)
|
||||||
|
+ 1
|
||||||
|
),
|
||||||
|
})
|
||||||
|
|
||||||
|
report["passed"] = True
|
||||||
|
|
||||||
|
finally:
|
||||||
|
save()
|
||||||
|
|
||||||
|
print(json.dumps(
|
||||||
|
report["summary"],
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
))
|
||||||
|
|
||||||
|
print("Reports: " + str(directory))
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: reconciler traversal cost "
|
||||||
|
"experiment completed"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run(main)
|
||||||
@@ -0,0 +1,492 @@
|
|||||||
|
import contextlib
|
||||||
|
import importlib.util
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from client import Client, Failure, arguments, check, run
|
||||||
|
import reconcile_check as helpers
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
|
|
||||||
|
def load_launcher():
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"reconciler_launcher",
|
||||||
|
ROOT / "scripts/reconcile.py",
|
||||||
|
)
|
||||||
|
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def state(container_id):
|
||||||
|
raw = helpers.command([
|
||||||
|
"docker",
|
||||||
|
"inspect",
|
||||||
|
"--format",
|
||||||
|
"{{json .State}}",
|
||||||
|
container_id,
|
||||||
|
])
|
||||||
|
|
||||||
|
return json.loads(raw)
|
||||||
|
|
||||||
|
|
||||||
|
def wait_container(container_id, seconds=90):
|
||||||
|
deadline = time.monotonic() + seconds
|
||||||
|
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
current = state(container_id)
|
||||||
|
|
||||||
|
healthy = (
|
||||||
|
current.get("Running")
|
||||||
|
and current.get("Health", {}).get("Status")
|
||||||
|
== "healthy"
|
||||||
|
)
|
||||||
|
|
||||||
|
if healthy:
|
||||||
|
return
|
||||||
|
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
raise Failure(
|
||||||
|
"Container did not become healthy: "
|
||||||
|
+ container_id[:12]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def wait_api(client, seconds=90):
|
||||||
|
deadline = time.monotonic() + seconds
|
||||||
|
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
try:
|
||||||
|
client.call(
|
||||||
|
"GET",
|
||||||
|
"/health/ready",
|
||||||
|
schema="Health",
|
||||||
|
)
|
||||||
|
|
||||||
|
return
|
||||||
|
|
||||||
|
except Failure:
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
raise Failure(
|
||||||
|
"API did not recover after dependency restart"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
args = arguments(
|
||||||
|
"Reconciler failure checks: "
|
||||||
|
"MinIO and PostgreSQL outage"
|
||||||
|
)
|
||||||
|
|
||||||
|
helpers.BUCKET = helpers.inspect_api()
|
||||||
|
|
||||||
|
launcher = load_launcher()
|
||||||
|
|
||||||
|
saved_services = {
|
||||||
|
name: launcher.service(name)
|
||||||
|
for name in ("api", "postgres", "minio")
|
||||||
|
}
|
||||||
|
|
||||||
|
launcher.service = lambda name: saved_services[name]
|
||||||
|
|
||||||
|
c = Client(
|
||||||
|
args.base_url,
|
||||||
|
args.ca_file,
|
||||||
|
timeout=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
alice, bob, _, chat = c.bootstrap()
|
||||||
|
chat_id = str(uuid.UUID(chat["id"]))
|
||||||
|
|
||||||
|
attachment, content = c.attachment(alice, chat)
|
||||||
|
|
||||||
|
message = c.message(
|
||||||
|
alice,
|
||||||
|
chat,
|
||||||
|
"Reconciler dependency failure",
|
||||||
|
attachments=[attachment["id"]],
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
original_name = str(uuid.UUID(attachment["id"]))
|
||||||
|
|
||||||
|
orphan_key = (
|
||||||
|
"originals/"
|
||||||
|
+ chat_id
|
||||||
|
+ "/orphan-failure-"
|
||||||
|
+ uuid.uuid4().hex
|
||||||
|
)
|
||||||
|
|
||||||
|
orphan_name = orphan_key.rsplit("/", 1)[-1]
|
||||||
|
|
||||||
|
metadata_before = helpers.snapshot(chat_id)
|
||||||
|
|
||||||
|
directory = (
|
||||||
|
ROOT
|
||||||
|
/ "results"
|
||||||
|
/ ("reconcile-failure-" + chat_id)
|
||||||
|
)
|
||||||
|
|
||||||
|
backups = (
|
||||||
|
ROOT
|
||||||
|
/ "backups"
|
||||||
|
/ ("reconcile-failure-" + chat_id)
|
||||||
|
)
|
||||||
|
|
||||||
|
directory.mkdir(parents=True, exist_ok=False)
|
||||||
|
backups.mkdir(parents=True, exist_ok=False)
|
||||||
|
|
||||||
|
stopped = set()
|
||||||
|
summary = []
|
||||||
|
completed = False
|
||||||
|
|
||||||
|
def worker(label, apply, expected_code):
|
||||||
|
path = directory / (label + ".json")
|
||||||
|
previous_argv = sys.argv
|
||||||
|
|
||||||
|
try:
|
||||||
|
sys.argv = [
|
||||||
|
str(ROOT / "scripts/reconcile.py"),
|
||||||
|
"--apply" if apply else "--dry-run",
|
||||||
|
"--chat-id",
|
||||||
|
chat_id,
|
||||||
|
"--grace-seconds",
|
||||||
|
"60",
|
||||||
|
"--max-seconds",
|
||||||
|
"120",
|
||||||
|
"--backup-dir",
|
||||||
|
str(backups),
|
||||||
|
"--output",
|
||||||
|
str(path),
|
||||||
|
]
|
||||||
|
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
code = launcher.main()
|
||||||
|
|
||||||
|
finally:
|
||||||
|
sys.argv = previous_argv
|
||||||
|
|
||||||
|
check(
|
||||||
|
code == expected_code,
|
||||||
|
"Unexpected return code at " + label,
|
||||||
|
)
|
||||||
|
|
||||||
|
report = json.loads(
|
||||||
|
path.read_text(encoding="utf-8")
|
||||||
|
)
|
||||||
|
|
||||||
|
summary.append({
|
||||||
|
"stage": label,
|
||||||
|
"status": report["status"],
|
||||||
|
"elapsed_seconds": report["elapsed_seconds"],
|
||||||
|
"error_type": report["error_type"],
|
||||||
|
})
|
||||||
|
|
||||||
|
print(
|
||||||
|
label
|
||||||
|
+ ": status="
|
||||||
|
+ report["status"]
|
||||||
|
+ ", error_type="
|
||||||
|
+ str(report["error_type"]),
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
return report
|
||||||
|
|
||||||
|
def verify_data(expect_orphan):
|
||||||
|
check(
|
||||||
|
helpers.snapshot(chat_id) == metadata_before,
|
||||||
|
"Attachment metadata changed",
|
||||||
|
)
|
||||||
|
|
||||||
|
names = helpers.object_names(chat_id)
|
||||||
|
|
||||||
|
expected = {original_name}
|
||||||
|
|
||||||
|
if expect_orphan:
|
||||||
|
expected.add(orphan_name)
|
||||||
|
|
||||||
|
check(
|
||||||
|
names == expected,
|
||||||
|
"Unexpected S3 object set",
|
||||||
|
)
|
||||||
|
|
||||||
|
result = c.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/"
|
||||||
|
+ attachment["id"]
|
||||||
|
+ "/content",
|
||||||
|
actor=bob,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
result["bytes"] == content,
|
||||||
|
"Confirmed original changed",
|
||||||
|
)
|
||||||
|
|
||||||
|
page = c.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/chats/"
|
||||||
|
+ chat_id
|
||||||
|
+ "/messages?limit=100",
|
||||||
|
actor=bob,
|
||||||
|
schema="MessagePage",
|
||||||
|
)["json"]
|
||||||
|
|
||||||
|
check(
|
||||||
|
message in page["items"],
|
||||||
|
"Message or attachment references changed",
|
||||||
|
)
|
||||||
|
|
||||||
|
def restore(name):
|
||||||
|
container_id = saved_services[name]["Id"]
|
||||||
|
|
||||||
|
helpers.command(
|
||||||
|
[
|
||||||
|
"docker",
|
||||||
|
"start",
|
||||||
|
container_id,
|
||||||
|
],
|
||||||
|
timeout=60,
|
||||||
|
)
|
||||||
|
|
||||||
|
wait_container(container_id)
|
||||||
|
stopped.discard(name)
|
||||||
|
|
||||||
|
wait_api(c)
|
||||||
|
|
||||||
|
try:
|
||||||
|
helpers.mc(
|
||||||
|
"pipe",
|
||||||
|
helpers.target(orphan_key),
|
||||||
|
data=content,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Waiting 65 seconds "
|
||||||
|
"for the real orphan grace period...",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(65)
|
||||||
|
|
||||||
|
baseline = worker(
|
||||||
|
"01-baseline",
|
||||||
|
apply=False,
|
||||||
|
expected_code=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
baseline["counts"].get(
|
||||||
|
"would_delete_orphans", 0
|
||||||
|
) == 1,
|
||||||
|
"No deletion candidate in baseline",
|
||||||
|
)
|
||||||
|
|
||||||
|
verify_data(expect_orphan=True)
|
||||||
|
|
||||||
|
for number, name in enumerate(
|
||||||
|
("minio", "postgres"),
|
||||||
|
start=2,
|
||||||
|
):
|
||||||
|
container_id = saved_services[name]["Id"]
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Stopping "
|
||||||
|
+ name
|
||||||
|
+ " for the experiment",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
stopped.add(name)
|
||||||
|
|
||||||
|
helpers.command(
|
||||||
|
[
|
||||||
|
"docker",
|
||||||
|
"stop",
|
||||||
|
"-t",
|
||||||
|
"10",
|
||||||
|
container_id,
|
||||||
|
],
|
||||||
|
timeout=60,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
not state(container_id).get("Running"),
|
||||||
|
"Dependency did not stop",
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
report = worker(
|
||||||
|
f"{number:02d}-{name}-unavailable",
|
||||||
|
apply=True,
|
||||||
|
expected_code=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
report["status"] == "failed",
|
||||||
|
"Outage was not reported as failed",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
report["error_type"] is not None,
|
||||||
|
"Missing error type",
|
||||||
|
)
|
||||||
|
|
||||||
|
for operation in (
|
||||||
|
"delete_object",
|
||||||
|
"put_object",
|
||||||
|
):
|
||||||
|
check(
|
||||||
|
report["s3_operation_calls"].get(
|
||||||
|
operation, 0
|
||||||
|
) == 0,
|
||||||
|
"Mutation attempted "
|
||||||
|
"during dependency outage",
|
||||||
|
)
|
||||||
|
|
||||||
|
for counter in (
|
||||||
|
"missing_objects",
|
||||||
|
"unresolved_missing",
|
||||||
|
"deleted_orphans",
|
||||||
|
"restored",
|
||||||
|
):
|
||||||
|
check(
|
||||||
|
report["counts"].get(
|
||||||
|
counter, 0
|
||||||
|
) == 0,
|
||||||
|
"Outage was confused "
|
||||||
|
"with a data inconsistency",
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
restore(name)
|
||||||
|
|
||||||
|
verify_data(expect_orphan=True)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: "
|
||||||
|
+ name
|
||||||
|
+ " outage aborted reconciliation; "
|
||||||
|
"data and session survived",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
repaired = worker(
|
||||||
|
"04-recovery-apply",
|
||||||
|
apply=True,
|
||||||
|
expected_code=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
repaired["counts"].get(
|
||||||
|
"deleted_orphans", 0
|
||||||
|
) == 1,
|
||||||
|
"Orphan was not removed after recovery",
|
||||||
|
)
|
||||||
|
|
||||||
|
verify_data(expect_orphan=False)
|
||||||
|
|
||||||
|
repeated = worker(
|
||||||
|
"05-recovery-repeat",
|
||||||
|
apply=True,
|
||||||
|
expected_code=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
for counter in (
|
||||||
|
"deleted_orphans",
|
||||||
|
"restored",
|
||||||
|
"missing_objects",
|
||||||
|
"unresolved_missing",
|
||||||
|
):
|
||||||
|
check(
|
||||||
|
repeated["counts"].get(
|
||||||
|
counter, 0
|
||||||
|
) == 0,
|
||||||
|
"Repeated run is not clean",
|
||||||
|
)
|
||||||
|
|
||||||
|
verify_data(expect_orphan=False)
|
||||||
|
completed = True
|
||||||
|
|
||||||
|
finally:
|
||||||
|
recovery_errors = []
|
||||||
|
|
||||||
|
for name in tuple(stopped):
|
||||||
|
try:
|
||||||
|
restore(name)
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
recovery_errors.append(name)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"WARNING: could not restore service: "
|
||||||
|
+ name,
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
if not recovery_errors:
|
||||||
|
try:
|
||||||
|
if orphan_name in helpers.object_names(chat_id):
|
||||||
|
helpers.mc(
|
||||||
|
"rm",
|
||||||
|
helpers.target(orphan_key),
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
recovery_errors.append(
|
||||||
|
"test orphan cleanup"
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"WARNING: cannot remove synthetic orphan: "
|
||||||
|
+ orphan_key,
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
not recovery_errors,
|
||||||
|
"Recovery incomplete: "
|
||||||
|
+ ", ".join(recovery_errors),
|
||||||
|
)
|
||||||
|
|
||||||
|
if completed:
|
||||||
|
report = {
|
||||||
|
"passed": True,
|
||||||
|
"chat_id": chat_id,
|
||||||
|
"stages": summary,
|
||||||
|
}
|
||||||
|
|
||||||
|
(directory / "summary.json").write_text(
|
||||||
|
json.dumps(
|
||||||
|
report,
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: reconciliation resumes after recovery "
|
||||||
|
"and repeated apply changes nothing"
|
||||||
|
)
|
||||||
|
|
||||||
|
print("Reports: " + str(directory))
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: reconciler dependency failure "
|
||||||
|
"scenario completed"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run(main)
|
||||||
@@ -0,0 +1,273 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import re
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from client import Client, arguments, check, run
|
||||||
|
|
||||||
|
|
||||||
|
FIXTURE = (
|
||||||
|
Path(__file__).resolve().parent
|
||||||
|
/ "fixtures"
|
||||||
|
/ "sample.png"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def command_output(command):
|
||||||
|
result = subprocess.run(
|
||||||
|
command,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
text=True,
|
||||||
|
timeout=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
result.returncode == 0,
|
||||||
|
"Docker command failed; check that API containers are running",
|
||||||
|
)
|
||||||
|
|
||||||
|
return result.stdout.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def api_containers():
|
||||||
|
output = command_output([
|
||||||
|
"docker", "compose", "ps", "-q", "api",
|
||||||
|
])
|
||||||
|
|
||||||
|
containers = output.splitlines()
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(containers) == 2,
|
||||||
|
"This test requires exactly two running API replicas",
|
||||||
|
)
|
||||||
|
|
||||||
|
for container in containers:
|
||||||
|
check(
|
||||||
|
re.fullmatch(r"[a-f0-9]{12,64}", container) is not None,
|
||||||
|
"Unexpected container ID",
|
||||||
|
)
|
||||||
|
|
||||||
|
return containers
|
||||||
|
|
||||||
|
|
||||||
|
def temporary_files(containers):
|
||||||
|
result = set()
|
||||||
|
|
||||||
|
for container in containers:
|
||||||
|
output = command_output([
|
||||||
|
"docker", "exec", container,
|
||||||
|
"sh", "-ec",
|
||||||
|
'''
|
||||||
|
directory="${UPLOAD_TEMP_DIR:-/tmp/messenger-uploads}"
|
||||||
|
|
||||||
|
if [ -d "$directory" ]; then
|
||||||
|
find "$directory" \
|
||||||
|
-maxdepth 1 \
|
||||||
|
-type f \
|
||||||
|
-name 'upload-*.tmp' \
|
||||||
|
-print
|
||||||
|
fi
|
||||||
|
''',
|
||||||
|
])
|
||||||
|
|
||||||
|
for filename in output.splitlines():
|
||||||
|
if filename:
|
||||||
|
result.add((container, filename))
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def wait_for_empty_directory(containers, seconds=25):
|
||||||
|
deadline = time.monotonic() + seconds
|
||||||
|
|
||||||
|
while True:
|
||||||
|
remaining = temporary_files(containers)
|
||||||
|
|
||||||
|
if not remaining:
|
||||||
|
return
|
||||||
|
|
||||||
|
check(
|
||||||
|
time.monotonic() < deadline,
|
||||||
|
"Temporary upload files were not removed",
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(0.5)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
args = arguments("Interrupted upload cleanup check")
|
||||||
|
|
||||||
|
client = Client(
|
||||||
|
args.base_url,
|
||||||
|
args.ca_file,
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
address = urlsplit(client.base)
|
||||||
|
|
||||||
|
check(
|
||||||
|
address.scheme == "http",
|
||||||
|
"This local test expects an HTTP endpoint",
|
||||||
|
)
|
||||||
|
|
||||||
|
containers = api_containers()
|
||||||
|
|
||||||
|
check(
|
||||||
|
not temporary_files(containers),
|
||||||
|
"Temporary directory is not empty. "
|
||||||
|
"Finish other uploads and inspect leftover files first",
|
||||||
|
)
|
||||||
|
|
||||||
|
alice, bob, eve, chat = client.bootstrap()
|
||||||
|
|
||||||
|
original = FIXTURE.read_bytes()
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(original) > 32768,
|
||||||
|
"Fixture must be larger than the partial upload",
|
||||||
|
)
|
||||||
|
|
||||||
|
path = (
|
||||||
|
address.path.rstrip("/")
|
||||||
|
+ "/api/v1/chats/"
|
||||||
|
+ chat["id"]
|
||||||
|
+ "/attachments?filename=interrupted.png"
|
||||||
|
)
|
||||||
|
|
||||||
|
headers = (
|
||||||
|
f"POST {path} HTTP/1.1\r\n"
|
||||||
|
f"Host: {address.netloc}\r\n"
|
||||||
|
f"Authorization: Bearer {alice['token']}\r\n"
|
||||||
|
"Content-Type: image/png\r\n"
|
||||||
|
f"Content-Length: {len(original)}\r\n"
|
||||||
|
"Connection: close\r\n"
|
||||||
|
"\r\n"
|
||||||
|
).encode("ascii")
|
||||||
|
|
||||||
|
connection = socket.create_connection(
|
||||||
|
(address.hostname, address.port or 80),
|
||||||
|
timeout=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
observed = False
|
||||||
|
|
||||||
|
try:
|
||||||
|
connection.sendall(headers)
|
||||||
|
connection.sendall(original[:32768])
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Partial body sent; waiting for a temporary file",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
deadline = time.monotonic() + 5
|
||||||
|
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
if temporary_files(containers):
|
||||||
|
observed = True
|
||||||
|
break
|
||||||
|
|
||||||
|
time.sleep(0.2)
|
||||||
|
|
||||||
|
check(
|
||||||
|
observed,
|
||||||
|
"Temporary file was not observed. "
|
||||||
|
"Check API logs and proxy timeouts",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: API started receiving the incomplete upload",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
# Закрываем соединение до передачи остальных байтов.
|
||||||
|
try:
|
||||||
|
connection.shutdown(socket.SHUT_RDWR)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
print("Client connection closed before upload completion", flush=True)
|
||||||
|
|
||||||
|
wait_for_empty_directory(containers)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: temporary file removed after interrupted upload",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Проверяем, что разрешение UploadLimiter не осталось занятым.
|
||||||
|
# Последовательными запросами через LB наблюдаем обе реплики.
|
||||||
|
instances = set()
|
||||||
|
last_attachment = None
|
||||||
|
|
||||||
|
for _ in range(20):
|
||||||
|
response = client.call(
|
||||||
|
"POST",
|
||||||
|
"/api/v1/chats/" + chat["id"]
|
||||||
|
+ "/attachments?filename=after-abort.png",
|
||||||
|
201,
|
||||||
|
actor=alice,
|
||||||
|
raw=original,
|
||||||
|
headers={"Content-Type": "image/png"},
|
||||||
|
schema="Attachment",
|
||||||
|
)
|
||||||
|
|
||||||
|
attachment = response["json"]
|
||||||
|
|
||||||
|
check(
|
||||||
|
attachment["status"] == "ready",
|
||||||
|
"Upload did not become ready",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
attachment["sha256"]
|
||||||
|
== hashlib.sha256(original).hexdigest(),
|
||||||
|
"Uploaded checksum mismatch",
|
||||||
|
)
|
||||||
|
|
||||||
|
instances.add(response["headers"]["X-Instance-Id"])
|
||||||
|
last_attachment = attachment
|
||||||
|
|
||||||
|
if len(instances) >= 2:
|
||||||
|
break
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(instances) == 2,
|
||||||
|
"Successful uploads were not observed on both API replicas",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: both API replicas accept uploads after cancellation",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
downloaded = client.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/"
|
||||||
|
+ last_attachment["id"]
|
||||||
|
+ "/content",
|
||||||
|
actor=bob,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
downloaded["bytes"] == original,
|
||||||
|
"Downloaded file differs from the original",
|
||||||
|
)
|
||||||
|
|
||||||
|
wait_for_empty_directory(containers)
|
||||||
|
|
||||||
|
print("PASS: subsequent upload is intact", flush=True)
|
||||||
|
print("PASS: interrupted upload scenario completed", flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run(main)
|
||||||
@@ -0,0 +1,249 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from client import Client, arguments, check, run
|
||||||
|
from upload_abort_check import (
|
||||||
|
FIXTURE,
|
||||||
|
api_containers,
|
||||||
|
command_output,
|
||||||
|
temporary_files,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def container_running(container):
|
||||||
|
value = command_output([
|
||||||
|
"docker", "inspect",
|
||||||
|
"--format", "{{.State.Running}}",
|
||||||
|
container,
|
||||||
|
])
|
||||||
|
|
||||||
|
return value == "true"
|
||||||
|
|
||||||
|
|
||||||
|
def container_healthy(container):
|
||||||
|
value = command_output([
|
||||||
|
"docker", "inspect",
|
||||||
|
"--format", "{{.State.Health.Status}}",
|
||||||
|
container,
|
||||||
|
])
|
||||||
|
|
||||||
|
return value == "healthy"
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
args = arguments("API crash during an incomplete upload")
|
||||||
|
|
||||||
|
client = Client(
|
||||||
|
args.base_url,
|
||||||
|
args.ca_file,
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
address = urlsplit(client.base)
|
||||||
|
|
||||||
|
check(
|
||||||
|
address.scheme == "http",
|
||||||
|
"This local test expects HTTP",
|
||||||
|
)
|
||||||
|
|
||||||
|
containers = api_containers()
|
||||||
|
|
||||||
|
check(
|
||||||
|
not temporary_files(containers),
|
||||||
|
"Temporary directories must be empty before the test",
|
||||||
|
)
|
||||||
|
|
||||||
|
alice, bob, eve, chat = client.bootstrap()
|
||||||
|
|
||||||
|
good_attachment, original = client.attachment(alice, chat)
|
||||||
|
|
||||||
|
content = FIXTURE.read_bytes()
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(content) > 32768,
|
||||||
|
"Fixture is too small",
|
||||||
|
)
|
||||||
|
|
||||||
|
path = (
|
||||||
|
address.path.rstrip("/")
|
||||||
|
+ "/api/v1/chats/"
|
||||||
|
+ chat["id"]
|
||||||
|
+ "/attachments?filename=crash-test.png"
|
||||||
|
)
|
||||||
|
|
||||||
|
headers = (
|
||||||
|
f"POST {path} HTTP/1.1\r\n"
|
||||||
|
f"Host: {address.netloc}\r\n"
|
||||||
|
f"Authorization: Bearer {alice['token']}\r\n"
|
||||||
|
"Content-Type: image/png\r\n"
|
||||||
|
f"Content-Length: {len(content)}\r\n"
|
||||||
|
"Connection: close\r\n"
|
||||||
|
"\r\n"
|
||||||
|
).encode("ascii")
|
||||||
|
|
||||||
|
connection = None
|
||||||
|
target = None
|
||||||
|
temporary_path = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
connection = socket.create_connection(
|
||||||
|
(address.hostname, address.port or 80),
|
||||||
|
timeout=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
connection.sendall(headers)
|
||||||
|
connection.sendall(content[:32768])
|
||||||
|
|
||||||
|
print("Partial upload started", flush=True)
|
||||||
|
|
||||||
|
deadline = time.monotonic() + 5
|
||||||
|
|
||||||
|
while True:
|
||||||
|
found = temporary_files(containers)
|
||||||
|
|
||||||
|
if found:
|
||||||
|
check(
|
||||||
|
len(found) == 1,
|
||||||
|
"Expected exactly one active temporary file",
|
||||||
|
)
|
||||||
|
|
||||||
|
target, temporary_path = next(iter(found))
|
||||||
|
break
|
||||||
|
|
||||||
|
check(
|
||||||
|
time.monotonic() < deadline,
|
||||||
|
"Temporary file was not observed",
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(0.2)
|
||||||
|
|
||||||
|
|
||||||
|
command_output([
|
||||||
|
"docker", "exec", target,
|
||||||
|
"touch", "-t", "200001010000",
|
||||||
|
temporary_path,
|
||||||
|
])
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Temporary file found; test timestamp set to the past",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
command_output([
|
||||||
|
"docker", "kill",
|
||||||
|
"--signal", "KILL",
|
||||||
|
target,
|
||||||
|
])
|
||||||
|
|
||||||
|
check(
|
||||||
|
not container_running(target),
|
||||||
|
"Target API container did not stop",
|
||||||
|
)
|
||||||
|
|
||||||
|
connection.close()
|
||||||
|
connection = None
|
||||||
|
|
||||||
|
copied = subprocess.run(
|
||||||
|
[
|
||||||
|
"docker", "cp",
|
||||||
|
target + ":" + temporary_path,
|
||||||
|
"-",
|
||||||
|
],
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
timeout=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
copied.returncode == 0,
|
||||||
|
"Temporary file did not survive the process crash",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: temporary file remains after SIGKILL",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
command_output([
|
||||||
|
"docker", "start", target,
|
||||||
|
])
|
||||||
|
|
||||||
|
ready_deadline = time.monotonic() + 120
|
||||||
|
|
||||||
|
while not container_healthy(target):
|
||||||
|
check(
|
||||||
|
container_running(target),
|
||||||
|
"API exited after restart",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
time.monotonic() < ready_deadline,
|
||||||
|
"API did not become healthy after restart",
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: the same API container recovered",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Waiting for temporary-file cleanup...",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
cleanup_deadline = time.monotonic() + 120
|
||||||
|
|
||||||
|
while (target, temporary_path) in temporary_files([target]):
|
||||||
|
check(
|
||||||
|
time.monotonic() < cleanup_deadline,
|
||||||
|
"Old temporary file was not removed",
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: stale temporary file removed after API restart",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
downloaded = client.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/"
|
||||||
|
+ good_attachment["id"]
|
||||||
|
+ "/content",
|
||||||
|
actor=bob,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
downloaded["bytes"] == original,
|
||||||
|
"Confirmed attachment was lost or changed",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: existing session and confirmed attachment survived",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
client.attachment(alice, chat)
|
||||||
|
|
||||||
|
print("PASS: subsequent upload succeeds", flush=True)
|
||||||
|
print("PASS: API crash scenario completed", flush=True)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
if connection is not None:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
if target is not None and not container_running(target):
|
||||||
|
command_output([
|
||||||
|
"docker", "start", target,
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run(main)
|
||||||
@@ -0,0 +1,278 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
from client import Client, arguments, check, run
|
||||||
|
|
||||||
|
|
||||||
|
def execute(command, input_text=None):
|
||||||
|
result = subprocess.run(
|
||||||
|
command,
|
||||||
|
input=input_text,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
result.returncode == 0,
|
||||||
|
"Infrastructure command failed. "
|
||||||
|
"Check PostgreSQL, MinIO and the configured bucket",
|
||||||
|
)
|
||||||
|
|
||||||
|
return result.stdout.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def sql(query):
|
||||||
|
return execute(
|
||||||
|
[
|
||||||
|
"docker", "compose", "exec", "-T",
|
||||||
|
"postgres", "sh", "-ec",
|
||||||
|
'''
|
||||||
|
exec psql -X \
|
||||||
|
-v ON_ERROR_STOP=1 \
|
||||||
|
-U "$POSTGRES_USER" \
|
||||||
|
-d "$POSTGRES_DB" \
|
||||||
|
-At
|
||||||
|
''',
|
||||||
|
],
|
||||||
|
input_text=query,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def minio(script, *arguments):
|
||||||
|
return execute([
|
||||||
|
"docker", "compose", "exec", "-T",
|
||||||
|
"minio", "sh", "-ec",
|
||||||
|
script,
|
||||||
|
"failure-check",
|
||||||
|
*arguments,
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def configure_minio():
|
||||||
|
minio(
|
||||||
|
'''
|
||||||
|
mc alias set failurecheck http://localhost:9000 \
|
||||||
|
"$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
|
||||||
|
'''
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def list_objects(bucket, prefix):
|
||||||
|
output = minio(
|
||||||
|
'mc ls --json "failurecheck/$1/$2"',
|
||||||
|
bucket,
|
||||||
|
prefix,
|
||||||
|
)
|
||||||
|
|
||||||
|
objects = set()
|
||||||
|
|
||||||
|
for line in output.splitlines():
|
||||||
|
if not line.strip():
|
||||||
|
continue
|
||||||
|
|
||||||
|
item = json.loads(line)
|
||||||
|
|
||||||
|
check(
|
||||||
|
item.get("status") != "error",
|
||||||
|
"MinIO failed to list test objects",
|
||||||
|
)
|
||||||
|
|
||||||
|
if item.get("key"):
|
||||||
|
objects.add(item["key"])
|
||||||
|
|
||||||
|
return objects
|
||||||
|
|
||||||
|
|
||||||
|
def wait_until(deadline):
|
||||||
|
remaining = deadline - time.monotonic()
|
||||||
|
|
||||||
|
if remaining > 0:
|
||||||
|
time.sleep(remaining)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
def configure(parser):
|
||||||
|
parser.add_argument(
|
||||||
|
"--bucket",
|
||||||
|
default="messenger-attachments",
|
||||||
|
)
|
||||||
|
|
||||||
|
args = arguments(
|
||||||
|
"Failure between S3 upload and metadata INSERT",
|
||||||
|
configure,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
re.fullmatch(
|
||||||
|
r"[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]",
|
||||||
|
args.bucket,
|
||||||
|
) is not None,
|
||||||
|
"Invalid bucket name",
|
||||||
|
)
|
||||||
|
|
||||||
|
client = Client(
|
||||||
|
args.base_url,
|
||||||
|
args.ca_file,
|
||||||
|
timeout=40,
|
||||||
|
)
|
||||||
|
|
||||||
|
alice, bob, eve, chat = client.bootstrap()
|
||||||
|
|
||||||
|
chat_id = str(uuid.UUID(chat["id"]))
|
||||||
|
|
||||||
|
configure_minio()
|
||||||
|
|
||||||
|
good_attachment, original = client.attachment(alice, chat)
|
||||||
|
|
||||||
|
prefix = f"originals/{chat_id}/"
|
||||||
|
|
||||||
|
before = list_objects(args.bucket, prefix)
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(before) == 1,
|
||||||
|
"Expected one control object in the new test chat",
|
||||||
|
)
|
||||||
|
|
||||||
|
marker = uuid.uuid4().hex
|
||||||
|
filename = f"db-failure-{marker}.png"
|
||||||
|
|
||||||
|
function_name = f"test_attachment_failure_{marker}"
|
||||||
|
trigger_name = f"test_attachment_failure_{marker}"
|
||||||
|
|
||||||
|
cleanup_sql = f"""
|
||||||
|
DROP TRIGGER IF EXISTS {trigger_name}
|
||||||
|
ON public.attachments;
|
||||||
|
|
||||||
|
DROP FUNCTION IF EXISTS public.{function_name}();
|
||||||
|
"""
|
||||||
|
|
||||||
|
try:
|
||||||
|
sql(f"""
|
||||||
|
CREATE FUNCTION public.{function_name}()
|
||||||
|
RETURNS trigger
|
||||||
|
LANGUAGE plpgsql
|
||||||
|
AS $$
|
||||||
|
BEGIN
|
||||||
|
RAISE EXCEPTION
|
||||||
|
'Synthetic attachment metadata failure';
|
||||||
|
END;
|
||||||
|
$$;
|
||||||
|
|
||||||
|
CREATE TRIGGER {trigger_name}
|
||||||
|
BEFORE INSERT ON public.attachments
|
||||||
|
FOR EACH ROW
|
||||||
|
WHEN (NEW.filename = '{filename}')
|
||||||
|
EXECUTE FUNCTION public.{function_name}();
|
||||||
|
""")
|
||||||
|
|
||||||
|
print(
|
||||||
|
"Installed a DB failure for one synthetic filename",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
response = client.call(
|
||||||
|
"POST",
|
||||||
|
"/api/v1/chats/" + chat_id
|
||||||
|
+ "/attachments?filename="
|
||||||
|
+ quote(filename, safe=""),
|
||||||
|
503,
|
||||||
|
actor=alice,
|
||||||
|
raw=original,
|
||||||
|
headers={"Content-Type": "image/png"},
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
response["json"]["error"]["code"] == "unavailable",
|
||||||
|
"Expected unavailable error code",
|
||||||
|
)
|
||||||
|
|
||||||
|
print("PASS: failed metadata INSERT returns 503", flush=True)
|
||||||
|
|
||||||
|
sql(cleanup_sql)
|
||||||
|
|
||||||
|
count = sql(f"""
|
||||||
|
SELECT count(*)
|
||||||
|
FROM public.attachments
|
||||||
|
WHERE chat_id = '{chat_id}'::uuid
|
||||||
|
AND filename = '{filename}';
|
||||||
|
""")
|
||||||
|
|
||||||
|
check(
|
||||||
|
count == "0",
|
||||||
|
"Metadata was unexpectedly committed",
|
||||||
|
)
|
||||||
|
|
||||||
|
after = list_objects(args.bucket, prefix)
|
||||||
|
orphan_objects = after - before
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(orphan_objects) == 1,
|
||||||
|
"Expected exactly one S3 object without metadata",
|
||||||
|
)
|
||||||
|
|
||||||
|
orphan = next(iter(orphan_objects))
|
||||||
|
observed_at = time.monotonic()
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: S3 object exists, but metadata was rolled back",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
wait_until(observed_at + 25)
|
||||||
|
|
||||||
|
check(
|
||||||
|
orphan in list_objects(args.bucket, prefix),
|
||||||
|
"Orphan was deleted before the grace period",
|
||||||
|
)
|
||||||
|
|
||||||
|
print("PASS: grace period protects the recent object", flush=True)
|
||||||
|
print("Waiting for scheduled cleanup...", flush=True)
|
||||||
|
|
||||||
|
deadline = observed_at + 100
|
||||||
|
|
||||||
|
while orphan in list_objects(args.bucket, prefix):
|
||||||
|
check(
|
||||||
|
time.monotonic() < deadline,
|
||||||
|
"Orphan was not deleted. Check cleanup settings and logs",
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
print(
|
||||||
|
"PASS: orphan created by a real failed INSERT was deleted",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
downloaded = client.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/"
|
||||||
|
+ good_attachment["id"]
|
||||||
|
+ "/content",
|
||||||
|
actor=bob,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
downloaded["bytes"] == original,
|
||||||
|
"Control attachment was lost or modified",
|
||||||
|
)
|
||||||
|
|
||||||
|
print("PASS: control attachment is intact", flush=True)
|
||||||
|
|
||||||
|
client.attachment(alice, chat)
|
||||||
|
|
||||||
|
print("PASS: normal uploads work after the failure", flush=True)
|
||||||
|
print("PASS: S3/DB failure scenario completed", flush=True)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
sql(cleanup_sql)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run(main)
|
||||||
@@ -0,0 +1,495 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from collections import Counter
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from attachments_negative import make_png
|
||||||
|
from client import Client, arguments, check, run
|
||||||
|
|
||||||
|
|
||||||
|
def docker_output(command):
|
||||||
|
result = subprocess.run(
|
||||||
|
command,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
result.returncode == 0,
|
||||||
|
"Docker command failed while measuring memory",
|
||||||
|
)
|
||||||
|
|
||||||
|
return result.stdout.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def get_api_containers():
|
||||||
|
output = docker_output([
|
||||||
|
"docker", "compose", "ps", "-q", "api",
|
||||||
|
])
|
||||||
|
|
||||||
|
containers = output.splitlines()
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(containers) == 2,
|
||||||
|
"Expected exactly two running API replicas",
|
||||||
|
)
|
||||||
|
|
||||||
|
return containers
|
||||||
|
|
||||||
|
|
||||||
|
def memory_snapshot(container):
|
||||||
|
# Docker Desktop с cgroup v2.
|
||||||
|
# memory.current включает память всего контейнера,
|
||||||
|
# а не только Java heap.
|
||||||
|
output = docker_output([
|
||||||
|
"docker", "exec", container,
|
||||||
|
"sh", "-ec",
|
||||||
|
'''
|
||||||
|
cat /sys/fs/cgroup/memory.current
|
||||||
|
|
||||||
|
if [ -r /sys/fs/cgroup/memory.peak ]; then
|
||||||
|
cat /sys/fs/cgroup/memory.peak
|
||||||
|
else
|
||||||
|
echo unavailable
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat /sys/fs/cgroup/memory.events
|
||||||
|
''',
|
||||||
|
])
|
||||||
|
|
||||||
|
lines = output.splitlines()
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(lines) >= 2,
|
||||||
|
"Cannot read cgroup v2 memory statistics",
|
||||||
|
)
|
||||||
|
|
||||||
|
events = {}
|
||||||
|
|
||||||
|
for line in lines[2:]:
|
||||||
|
key, value = line.split()
|
||||||
|
events[key] = int(value)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"current_bytes": int(lines[0]),
|
||||||
|
"lifetime_peak_bytes": (
|
||||||
|
int(lines[1])
|
||||||
|
if lines[1].isdigit()
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
"oom": events.get("oom", 0),
|
||||||
|
"oom_kill": events.get("oom_kill", 0),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def mib(value):
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
return round(value / 1024 / 1024, 2)
|
||||||
|
|
||||||
|
|
||||||
|
def percentile(values, fraction):
|
||||||
|
if not values:
|
||||||
|
return None
|
||||||
|
|
||||||
|
ordered = sorted(values)
|
||||||
|
index = int((len(ordered) - 1) * fraction)
|
||||||
|
|
||||||
|
return round(ordered[index], 2)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
def configure(parser):
|
||||||
|
parser.add_argument("--concurrency", type=int, default=10)
|
||||||
|
parser.add_argument("--duration", type=int, default=30)
|
||||||
|
parser.add_argument(
|
||||||
|
"--output",
|
||||||
|
default="results/upload-memory.json",
|
||||||
|
)
|
||||||
|
|
||||||
|
args = arguments(
|
||||||
|
"Concurrent image uploads and API memory measurement",
|
||||||
|
configure,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
args.concurrency >= 10,
|
||||||
|
"Use at least 10 concurrent clients",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
args.duration >= 10,
|
||||||
|
"Duration must be at least 10 seconds",
|
||||||
|
)
|
||||||
|
|
||||||
|
controller = Client(
|
||||||
|
args.base_url,
|
||||||
|
args.ca_file,
|
||||||
|
timeout=40,
|
||||||
|
)
|
||||||
|
|
||||||
|
alice, bob, eve, chat = controller.bootstrap()
|
||||||
|
containers = get_api_containers()
|
||||||
|
|
||||||
|
print("Generating PNG 6000x6000...", flush=True)
|
||||||
|
|
||||||
|
image = make_png(6000, 6000)
|
||||||
|
checksum = hashlib.sha256(image).hexdigest()
|
||||||
|
|
||||||
|
check(
|
||||||
|
len(image) <= 10 * 1024 * 1024,
|
||||||
|
"Generated image exceeds upload byte limit",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"Image: {len(image)} bytes, 36000000 pixels",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
path = (
|
||||||
|
"/api/v1/chats/"
|
||||||
|
+ chat["id"]
|
||||||
|
+ "/attachments?filename=memory-test.png"
|
||||||
|
)
|
||||||
|
|
||||||
|
clients = [
|
||||||
|
Client(args.base_url, args.ca_file, timeout=40)
|
||||||
|
for _ in range(args.concurrency)
|
||||||
|
]
|
||||||
|
|
||||||
|
baseline = {
|
||||||
|
container: memory_snapshot(container)
|
||||||
|
for container in containers
|
||||||
|
}
|
||||||
|
|
||||||
|
samples = {container: [] for container in containers}
|
||||||
|
monitor_errors = []
|
||||||
|
stop_monitor = threading.Event()
|
||||||
|
|
||||||
|
started = time.monotonic()
|
||||||
|
|
||||||
|
def monitor():
|
||||||
|
while not stop_monitor.is_set():
|
||||||
|
for container in containers:
|
||||||
|
try:
|
||||||
|
snapshot = memory_snapshot(container)
|
||||||
|
|
||||||
|
samples[container].append({
|
||||||
|
"elapsed_seconds": round(
|
||||||
|
time.monotonic() - started,
|
||||||
|
3,
|
||||||
|
),
|
||||||
|
**snapshot,
|
||||||
|
})
|
||||||
|
|
||||||
|
except Exception as error:
|
||||||
|
monitor_errors.append(
|
||||||
|
type(error).__name__
|
||||||
|
)
|
||||||
|
stop_monitor.set()
|
||||||
|
return
|
||||||
|
|
||||||
|
stop_monitor.wait(0.5)
|
||||||
|
|
||||||
|
barrier = threading.Barrier(args.concurrency + 1)
|
||||||
|
|
||||||
|
def worker(index):
|
||||||
|
client = clients[index]
|
||||||
|
|
||||||
|
counters = Counter()
|
||||||
|
latencies = []
|
||||||
|
errors = []
|
||||||
|
instances = set()
|
||||||
|
successful_instances = set()
|
||||||
|
first_success = None
|
||||||
|
|
||||||
|
barrier.wait()
|
||||||
|
|
||||||
|
while time.monotonic() - started < args.duration:
|
||||||
|
request_started = time.monotonic()
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = client.call(
|
||||||
|
"POST",
|
||||||
|
path,
|
||||||
|
expected=[201, 429],
|
||||||
|
actor=alice,
|
||||||
|
raw=image,
|
||||||
|
headers={"Content-Type": "image/png"},
|
||||||
|
schema="Attachment",
|
||||||
|
)
|
||||||
|
|
||||||
|
instance = response["headers"]["X-Instance-Id"]
|
||||||
|
instances.add(instance)
|
||||||
|
|
||||||
|
if response["status"] == 429:
|
||||||
|
check(
|
||||||
|
response["json"]["error"]["code"]
|
||||||
|
== "rate_limited",
|
||||||
|
"Unexpected error code for 429",
|
||||||
|
)
|
||||||
|
|
||||||
|
counters["rate_limited"] += 1
|
||||||
|
|
||||||
|
else:
|
||||||
|
attachment = response["json"]
|
||||||
|
|
||||||
|
check(
|
||||||
|
attachment["status"] == "ready",
|
||||||
|
"Successful upload is not ready",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
attachment["sha256"] == checksum,
|
||||||
|
"Uploaded SHA-256 mismatch",
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
attachment["size_bytes"] == len(image),
|
||||||
|
"Uploaded size mismatch",
|
||||||
|
)
|
||||||
|
|
||||||
|
counters["successful"] += 1
|
||||||
|
successful_instances.add(instance)
|
||||||
|
|
||||||
|
latencies.append(
|
||||||
|
(time.monotonic() - request_started) * 1000
|
||||||
|
)
|
||||||
|
|
||||||
|
if first_success is None:
|
||||||
|
first_success = attachment
|
||||||
|
|
||||||
|
except Exception as error:
|
||||||
|
counters["unexpected_errors"] += 1
|
||||||
|
|
||||||
|
if len(errors) < 3:
|
||||||
|
errors.append(str(error))
|
||||||
|
|
||||||
|
time.sleep(0.1)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"counters": dict(counters),
|
||||||
|
"latencies": latencies,
|
||||||
|
"errors": errors,
|
||||||
|
"instances": instances,
|
||||||
|
"successful_instances": successful_instances,
|
||||||
|
"first_success": first_success,
|
||||||
|
}
|
||||||
|
|
||||||
|
monitor_thread = threading.Thread(
|
||||||
|
target=monitor,
|
||||||
|
daemon=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
with ThreadPoolExecutor(
|
||||||
|
max_workers=args.concurrency
|
||||||
|
) as executor:
|
||||||
|
|
||||||
|
futures = [
|
||||||
|
executor.submit(worker, index)
|
||||||
|
for index in range(args.concurrency)
|
||||||
|
]
|
||||||
|
|
||||||
|
started = time.monotonic()
|
||||||
|
monitor_thread.start()
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"Starting {args.concurrency} clients "
|
||||||
|
f"for {args.duration} seconds...",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
barrier.wait()
|
||||||
|
|
||||||
|
worker_results = [
|
||||||
|
future.result()
|
||||||
|
for future in futures
|
||||||
|
]
|
||||||
|
|
||||||
|
elapsed = time.monotonic() - started
|
||||||
|
|
||||||
|
stop_monitor.set()
|
||||||
|
monitor_thread.join(timeout=25)
|
||||||
|
|
||||||
|
check(
|
||||||
|
not monitor_thread.is_alive(),
|
||||||
|
"Memory monitor did not stop",
|
||||||
|
)
|
||||||
|
|
||||||
|
final = {
|
||||||
|
container: memory_snapshot(container)
|
||||||
|
for container in containers
|
||||||
|
}
|
||||||
|
|
||||||
|
totals = Counter()
|
||||||
|
latencies = []
|
||||||
|
errors = []
|
||||||
|
instances = set()
|
||||||
|
successful_instances = set()
|
||||||
|
|
||||||
|
for result in worker_results:
|
||||||
|
totals.update(result["counters"])
|
||||||
|
latencies.extend(result["latencies"])
|
||||||
|
instances.update(result["instances"])
|
||||||
|
successful_instances.update(
|
||||||
|
result["successful_instances"]
|
||||||
|
)
|
||||||
|
|
||||||
|
errors.extend(result["errors"])
|
||||||
|
|
||||||
|
download_errors = []
|
||||||
|
|
||||||
|
for result in worker_results:
|
||||||
|
attachment = result["first_success"]
|
||||||
|
|
||||||
|
if attachment is None:
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = controller.call(
|
||||||
|
"GET",
|
||||||
|
"/api/v1/attachments/"
|
||||||
|
+ attachment["id"]
|
||||||
|
+ "/content",
|
||||||
|
actor=bob,
|
||||||
|
)
|
||||||
|
|
||||||
|
check(
|
||||||
|
response["bytes"] == image,
|
||||||
|
"Downloaded image differs from the original",
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as error:
|
||||||
|
download_errors.append(str(error))
|
||||||
|
|
||||||
|
memory_report = {}
|
||||||
|
|
||||||
|
for container in containers:
|
||||||
|
measurements = [
|
||||||
|
baseline[container]["current_bytes"],
|
||||||
|
final[container]["current_bytes"],
|
||||||
|
]
|
||||||
|
|
||||||
|
measurements.extend(
|
||||||
|
sample["current_bytes"]
|
||||||
|
for sample in samples[container]
|
||||||
|
)
|
||||||
|
|
||||||
|
memory_report[container[:12]] = {
|
||||||
|
"baseline_mib": mib(
|
||||||
|
baseline[container]["current_bytes"]
|
||||||
|
),
|
||||||
|
"observed_peak_mib": mib(max(measurements)),
|
||||||
|
"final_mib": mib(
|
||||||
|
final[container]["current_bytes"]
|
||||||
|
),
|
||||||
|
"lifetime_peak_before_mib": mib(
|
||||||
|
baseline[container]["lifetime_peak_bytes"]
|
||||||
|
),
|
||||||
|
"lifetime_peak_after_mib": mib(
|
||||||
|
final[container]["lifetime_peak_bytes"]
|
||||||
|
),
|
||||||
|
"oom_delta": (
|
||||||
|
final[container]["oom"]
|
||||||
|
- baseline[container]["oom"]
|
||||||
|
),
|
||||||
|
"oom_kill_delta": (
|
||||||
|
final[container]["oom_kill"]
|
||||||
|
- baseline[container]["oom_kill"]
|
||||||
|
),
|
||||||
|
"sample_count": len(samples[container]),
|
||||||
|
}
|
||||||
|
|
||||||
|
success = totals["successful"]
|
||||||
|
limited = totals["rate_limited"]
|
||||||
|
unexpected = totals["unexpected_errors"]
|
||||||
|
|
||||||
|
passed = (
|
||||||
|
success > 0
|
||||||
|
and unexpected == 0
|
||||||
|
and not monitor_errors
|
||||||
|
and not download_errors
|
||||||
|
and len(successful_instances) >= 2
|
||||||
|
and all(
|
||||||
|
item["sample_count"] > 0
|
||||||
|
and item["oom_delta"] == 0
|
||||||
|
and item["oom_kill_delta"] == 0
|
||||||
|
for item in memory_report.values()
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
report = {
|
||||||
|
"passed": passed,
|
||||||
|
"workload": "closed-loop image uploads; no automatic retries",
|
||||||
|
"concurrency": args.concurrency,
|
||||||
|
"requested_duration_seconds": args.duration,
|
||||||
|
"elapsed_seconds": round(elapsed, 3),
|
||||||
|
"image": {
|
||||||
|
"width": 6000,
|
||||||
|
"height": 6000,
|
||||||
|
"pixels": 36_000_000,
|
||||||
|
"size_bytes": len(image),
|
||||||
|
"sha256": checksum,
|
||||||
|
},
|
||||||
|
"successful_uploads": success,
|
||||||
|
"rate_limited_requests": limited,
|
||||||
|
"unexpected_errors": unexpected,
|
||||||
|
"successful_rps": round(success / elapsed, 3),
|
||||||
|
"observed_instances": sorted(instances),
|
||||||
|
"successful_instances": sorted(successful_instances),
|
||||||
|
"successful_latency_ms": {
|
||||||
|
"p50": percentile(latencies, 0.50),
|
||||||
|
"p95": percentile(latencies, 0.95),
|
||||||
|
"p99": percentile(latencies, 0.99),
|
||||||
|
},
|
||||||
|
"memory": memory_report,
|
||||||
|
"error_examples": errors[:10],
|
||||||
|
"monitor_errors": monitor_errors,
|
||||||
|
"download_errors": download_errors,
|
||||||
|
"notes": [
|
||||||
|
"429 responses are admission-control rejections, not successful uploads.",
|
||||||
|
"Observed memory peak is sampled and may miss short spikes.",
|
||||||
|
"Lifetime cgroup peak includes earlier container activity and startup.",
|
||||||
|
"Container memory includes more than Java heap.",
|
||||||
|
"Concurrency means concurrent clients, not 10 simultaneous decoders.",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
output = Path(args.output)
|
||||||
|
output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
saved_report = {
|
||||||
|
**report,
|
||||||
|
"memory_samples": samples,
|
||||||
|
}
|
||||||
|
|
||||||
|
output.write_text(
|
||||||
|
json.dumps(saved_report, ensure_ascii=False, indent=2),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(
|
||||||
|
json.dumps(report, ensure_ascii=False, indent=2),
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
print("Report saved to:", output, flush=True)
|
||||||
|
|
||||||
|
check(
|
||||||
|
passed,
|
||||||
|
"Experiment detected errors; inspect the saved report",
|
||||||
|
)
|
||||||
|
|
||||||
|
print("PASS: concurrent upload experiment completed", flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run(main)
|
||||||
Reference in New Issue
Block a user