Completed for a grade of three
This commit is contained in:
@@ -1,41 +1,100 @@
|
||||
# Отчёт — лабораторная 4
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Кучерявенко Максим, БВТ2303
|
||||
- Целевая оценка: **3**. Критерии 4/5 не заявлены.
|
||||
- Commit/PR: после просмотра и публикации.
|
||||
- Основа: собственный lab3; структура конфигураций, controllers → services → repositories. Для S3 используется AWS SDK for Kotlin 1.6.41; изображения проверяет JDK ImageIO. ИИ использован как парный разработчик.
|
||||
- Источники: [AWS SDK for Kotlin](https://github.com/awslabs/aws-sdk-kotlin), [релиз MinIO](https://github.com/minio/minio/releases/tag/RELEASE.2025-09-07T16-13-09Z), [ImageReader](https://docs.oracle.com/en/java/javase/21/docs/api/java.desktop/javax/imageio/ImageReader.html), [S3 uploads](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html).
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker с Compose v2, Python 3.9+, openssl. Для JVM-тестов нужны JDK 21+ и Docker (Testcontainers самостоятельно поднимает PostgreSQL, Redis и MinIO).
|
||||
|
||||
```sh
|
||||
./scripts/init-secrets.sh
|
||||
|
||||
# Если 8080 занят lab3, как при проверке этой работы:
|
||||
export LAB_HTTP_PORT=8084
|
||||
export BASE_URL=http://localhost:8084
|
||||
# Для свободного стандартного порта можно использовать 8080 в обеих переменных.
|
||||
|
||||
docker compose config --quiet
|
||||
docker compose up -d --build --wait
|
||||
make check
|
||||
make test
|
||||
./gradlew test
|
||||
|
||||
# Остановка именно реплики, принявшей upload; скрипт сам возвращает её в работу.
|
||||
python3 scripts/attachment-failover.py
|
||||
|
||||
# Проверка lab3: запись сообщений во время отказа, идемпотентность, чтение файла.
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
docker compose start api
|
||||
|
||||
# Перезапуск LB, обеих API-реплик, PostgreSQL, Redis и MinIO без удаления volumes.
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
|
||||
# Безопасная остановка, данные сохраняются:
|
||||
docker compose down
|
||||
# Только для намеренного удаления всех учебных данных:
|
||||
# docker compose down -v
|
||||
```
|
||||
|
||||
Compose собирает общий образ `mtusi-messenger:lab4` через `api`; одноразовый `migrate` использует тот же образ и применяет Flyway V1–V4 до старта реплик. `s3-init` ждёт MinIO, создаёт bucket через `mc mb --ignore-existing` и закрывает анонимный доступ через `mc anonymous set none`. Повторный запуск сохраняет bucket и объекты. Секреты создаются локально, существующие файлы не перезаписываются; `.env`, `secrets/` и `run/development/` не входят в Git.
|
||||
|
||||
MinIO RELEASE.2025-09-07T16-13-09Z и mc RELEASE.2025-08-13T08-35-41Z закреплены digest в Compose. PostgreSQL 17.6, Redis 8.4.0, HAProxy 3.2.4 — из lab3. API использует Kotlin 2.3.20, Ktor 3.3.0, JDK 21. Проверено на macOS arm64 с Docker 28.4.0. MinIO S3 и console не публикуются на host; наружу доступен только LB на loopback.
|
||||
|
||||
### Настройки
|
||||
|
||||
`.env` необязателен. В Compose есть значения по умолчанию; `docker compose --env-file .env.example config --quiet` также проходит. Переменные из skeleton сопоставлены так:
|
||||
|
||||
- `API_REPLICAS`, `SESSION_TTL_SECONDS`, `IDEMPOTENCY_TTL_SECONDS`, `S3_BUCKET` используются в Compose. Дополнительно `LAB_HTTP_PORT` задаёт опубликованный порт LB.
|
||||
- `S3_ENDPOINT=http://s3:9000`, `S3_ACCESS_KEY_ID_FILE`, `S3_SECRET_ACCESS_KEY_FILE` явно передаются API в Compose; в локальном JVM-процессе читаются из environment или HOCON `storage`.
|
||||
- `S3_REGION=us-east-1` закреплён в SDK. `MAX_UPLOAD_BYTES=10485760` и `MAX_IMAGE_PIXELS=40000000` закреплены в валидаторе по обязательному контракту; переменными окружения не увеличиваются.
|
||||
- Вместо предложенных skeleton `DATABASE_URL_FILE` и `CACHE_URL_FILE` используются `DB_HOST/PORT/NAME/USER/PASSWORD_FILE` и `REDIS_HOST/PORT/PASSWORD_FILE`. PostgreSQL всегда источник истины для сессий; `SESSION_STORE` не переключает реализацию. `APP_ENV` не используется.
|
||||
- `APP_HOST/PORT` задаются внутри Compose, порт приложения — 8080. `LAB_HTTP_PORT` меняет только внешний порт.
|
||||
|
||||
Для отдельного `./gradlew run` задача создаёт недостающие файлы `run/development/config` из `_template`, сохраняя существующие настройки. Нужно настроить доступные локальному JVM PostgreSQL, Redis и S3; изолированные адреса Compose с host не доступны. Полностью воспроизводимый путь проверки выше использует Compose.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
Перенесена реализация lab3: Kotlin/Ktor, PostgreSQL и миграции Flyway, Redis для кэша сессий, две API-реплики за HAProxy, идемпотентность сообщений, graceful shutdown, тесты и эксплуатационные скрипты. Сохранена структура проекта, ориентированная на pixefy.
|
||||
Клиент → HAProxy → две Ktor API-реплики → PostgreSQL (метаданные, связи, сессии) и MinIO (байты). Redis остаётся кэшем сессий с fallback в PostgreSQL. API-реплики взаимозаменяемы, sticky sessions нет.
|
||||
|
||||
[Отчёт lab3 с инструкциями запуска](archive/lab3/REPORT.md) и [исторические доказательства](archive/lab3/evidence/) сохранены отдельно; они не подтверждают выполнение lab4. Контракты, задание и публичные тесты lab4 сохранены. S3 и вложения пока не реализованы; работа над критериями оценок ещё не начата.
|
||||
`attachments` содержит ID, chat_id, owner_id, отображаемое имя, MIME, размер, SHA-256 и время создания. `message_attachments` хранит порядок вложений сообщения, уникальность и внешние ключи. S3 key — `originals/<server-generated UUID>`; filename никогда не используется как путь. В API нет ключей S3 и credentials. В lab4 все опубликованные метаданные уже ready, updated_at равен created_at, variants=[], error_code=null.
|
||||
|
||||
TODO: описать изменения lab4 и соответствие переменных из .env.example реальной конфигурации.
|
||||
Upload: Bearer-аутентификация → проверка членства → filename/MIME/размер → ограниченное чтение тела → проверка изображения → S3 PUT → INSERT/COMMIT метаданных → 201. Нет долгой DB-транзакции во время HTTP upload или S3 PUT. Оригинальные bytes не перекодируются. Download: аутентификация → доступ через chat_id в PostgreSQL → S3 GET → bytes с истинным MIME через API без redirect. Посторонний и неизвестный ID получают одинаковый 404.
|
||||
|
||||
Сообщение допускает пустой текст с 1–10 уникальными вложениями этого чата. Член чата может прикрепить загруженный другим участником файл. Создание сообщения, связей и результата Idempotency-Key выполняется в одной транзакции. Отсутствующий attachment_ids и [] равнозначны; повтор возвращает тот же Message. Hash дополнен вложениями, при replay дополнительно сравниваются исходный текст и список ID. Старые текстовые ключи lab3 остаются пригодны для replay. Сохраняются keyset-пагинация, отзыв сессий между репликами и graceful shutdown.
|
||||
|
||||
### Ограничения входных данных
|
||||
|
||||
- Тело читается из канала порциями до 8192 bytes, с проверкой лимита до добавления в буфер. Превышение Content-Length отсекается сразу; при отсутствии длины применяется тот же счётчик bytes. Максимум — 10 MiB, превышение — 413; приём тела ограничен 15 секундами.
|
||||
- ImageIO определяет реальный формат, допускаются PNG/JPEG и совпадение MIME. Размеры считываются из заголовка до `reader.read`; произведение считается в Long, более 40 млн пикселей — 413.
|
||||
- Затем выполняется полное декодирование, чтобы одного корректного заголовка было недостаточно. Ошибки/предупреждения декодера — 415. Для PNG дополнительно проверяются границы chunks, CRC и IEND, для JPEG — завершающий EOI.
|
||||
- Filename: 1–255 Unicode code points; `/`, `\`, NUL и управляющие символы — 400.
|
||||
- Временные файлы в API не создаются: ограниченный массив оригинала и декодированный raster находятся в RAM. ImageIO использует MemoryCacheImageInputStream. У download также есть предел 10 MiB.
|
||||
- S3-вызов ограничен coroutine timeout в 8 с, включая получение тела ответа. HAProxy server timeout увеличен до 30 с для синхронной загрузки. Автоматическое повторение POST после передачи запроса не включается.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
Проверки выполнены 05.11.2026, через `http://localhost:8084`. Ниже актуальные результаты lab4; материалы в `archive/lab3` исторические.
|
||||
|
||||
| Критерий | Команда/сценарий | Наблюдение | Доказательство |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
| Контракт, 201 ready, checksum, сообщение с вложением, ACL | `make test` | PASS публичного сценария lab4 | [smoke.txt](evidence/smoke.txt) |
|
||||
| Возможности lab3 и новые проверки | `./gradlew test` | 75 тестов, 0 failures/errors/skipped | [gradle-tests.txt](evidence/gradle-tests.txt) |
|
||||
| PNG/JPEG, MIME, опасное имя, лимиты bytes/pixels до декодирования | AttachmentControllerTest, ImageValidatorTest | Оба формата скачиваются побайтно; 400/413/415 на неверный ввод | [gradle-tests.txt](evidence/gradle-tests.txt) |
|
||||
| Приватное S3, чужой чат, отказ именно загрузившей реплики | `python3 scripts/attachment-failover.py` | 403 для anonymous S3 GET известного объекта; 404 постороннему; иной X-Instance-Id и те же bytes после отказа | [attachment-failover.txt](evidence/attachment-failover.txt) |
|
||||
| Сохранность после restart всех компонентов | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS, прежняя сессия, метаданные и оригинал доступны; StartedAt всех процессов изменился | [persistence.txt](evidence/persistence.txt) |
|
||||
| Запись сообщений во время отказа API | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | Проверка восстановления записи, отсутствия дублей и сохранности данных | [failover.txt](evidence/failover.txt) |
|
||||
| Недоступность S3 | AttachmentControllerTest: pause MinIO | Upload/content/ready → 503, live → 200; после unpause исходные bytes доступны | [gradle-tests.txt](evidence/gradle-tests.txt) |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
## Согласованность и ограничения уровня 3
|
||||
|
||||
## Эксперименты и отказы
|
||||
201 подтверждается только после успешных S3 PUT и COMMIT PostgreSQL. Между этими хранилищами нет общей транзакции. При падении после PUT, до COMMIT, возможен объект без метаданных. Автоматическая сверка/очистка таких объектов и безопасная компенсация пока не реализованы — это следующий этап на 4. При потере ответа upload его повтор может создать ещё одно вложение, как разрешено контрактом.
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
На 3 API использует те же локальные S3 credentials, что и bootstrap; отдельная учётная запись с ограниченной policy ещё не реализована. Bucket приватен, порты S3/console наружу не опубликованы. Это локальный учебный стенд, не production hardening.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Память одной загрузки ограничена размером bytes и raster до 40 млн пикселей; потоковой обработки до S3, ограничения числа одновременно декодируемых изображений и эксперимента на ≥10 параллельных загрузках пока нет. Такие нагрузки могут исчерпать память реплики. Они относятся к следующему этапу, уровень 4 не заявляется.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Readiness требует PostgreSQL и S3, Redis допускает fallback. При отказе S3 HAProxy выведет обе реплики из ротации; LB может вернуть собственный 503 вместо JSON API. Прямые обращения к API при отказе S3 проверены тестом. Одноузловые PostgreSQL, MinIO и LB остаются точками отказа: volumes переживают restart, но не заменяют репликацию и backup. Сохраняются ограничения конкурентной keyset-пагинации и часов из [отчёта lab3](archive/lab3/REPORT.md).
|
||||
|
||||
@@ -46,6 +46,7 @@ dependencies {
|
||||
implementation(libs.flyway.postgresql)
|
||||
|
||||
implementation(libs.lettuce)
|
||||
implementation(awssdk.services.s3)
|
||||
|
||||
implementation(libs.bouncycastle)
|
||||
|
||||
|
||||
+55
-2
@@ -55,10 +55,49 @@ services:
|
||||
memory: 128m
|
||||
restart: unless-stopped
|
||||
|
||||
s3:
|
||||
# MinIO RELEASE.2025-09-07T16-13-09Z, pinned multi-platform image.
|
||||
image: quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
|
||||
command: ["server", "/data", "--console-address", ":9001"]
|
||||
environment:
|
||||
MINIO_ROOT_USER_FILE: /run/secrets/s3_access_key_id
|
||||
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_secret_access_key
|
||||
secrets:
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
volumes:
|
||||
- s3-data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512m
|
||||
restart: unless-stopped
|
||||
|
||||
s3-init:
|
||||
# mc RELEASE.2025-08-13T08-35-41Z
|
||||
image: quay.io/minio/mc@sha256:a7fe349ef4bd8521fb8497f55c6042871b2ae640607cf99d9bede5e9bdf11727
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
entrypoint: ["/bin/sh", "/bootstrap.sh"]
|
||||
environment:
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
secrets:
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
volumes:
|
||||
- ./infra/s3/bootstrap.sh:/bootstrap.sh:ro
|
||||
restart: "no"
|
||||
|
||||
# One-shot: applies Flyway migrations once and exits. Replicas start only after it succeeded,
|
||||
# so they never race each other on the schema.
|
||||
migrate:
|
||||
build: .
|
||||
image: mtusi-messenger:lab4
|
||||
command: ["migrate"]
|
||||
healthcheck:
|
||||
@@ -98,6 +137,8 @@ services:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
environment:
|
||||
APP_HOST: "0.0.0.0"
|
||||
APP_PORT: "8080"
|
||||
@@ -114,9 +155,15 @@ services:
|
||||
SESSION_CACHE_SINGLE_FLIGHT: ${SESSION_CACHE_SINGLE_FLIGHT:-true}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
SHUTDOWN_DRAIN_DELAY_MS: "3000"
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
secrets:
|
||||
- db_password
|
||||
- redis_password
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
# SIGTERM: readiness 503 and 3 s drain (HAProxy needs <= 2 s to notice), then Ktor closes the socket and
|
||||
# finishes requests in flight within 5 s. 3 + 5 = 8 s < 10 s before Docker sends SIGKILL.
|
||||
stop_grace_period: 10s
|
||||
@@ -130,7 +177,7 @@ services:
|
||||
api:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "127.0.0.1:8080:8080"
|
||||
- "127.0.0.1:${LAB_HTTP_PORT:-8080}:8080"
|
||||
# Port 8404 (health and stats) stays inside the compose network.
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8404/healthz"]
|
||||
@@ -150,5 +197,11 @@ secrets:
|
||||
redis_password:
|
||||
file: ./secrets/redis_password
|
||||
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
s3-data:
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
Upload: 201 ready; instance=ecbe49d85c82
|
||||
Anonymous S3 GET of known object: 403
|
||||
Outsider metadata/content: 404
|
||||
Download: 200; instance=9440528ba703
|
||||
SHA-256: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
PASS: uploading replica stopped; original served by another replica; stopped replica restored
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8084"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 123, "failed_attempts": 0, "first_recovery_seconds": 5.537}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,22 @@
|
||||
Command: ./gradlew test --rerun-tasks
|
||||
AWS SDK for Kotlin 1.6.41; BUILD SUCCESSFUL
|
||||
Tests: 75; failures: 0; errors: 0; skipped: 0
|
||||
com.mtusi.messenger.main.infrastructure.controllers.attachment.AttachmentControllerTest: tests=3 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.controllers.authentication.AuthenticationControllerTest: tests=9 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.controllers.chat.ChatControllerTest: tests=6 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.controllers.health.HealthControllerTest: tests=2 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.controllers.message.MessageConcurrencyTest: tests=1 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.controllers.message.MessageControllerTest: tests=7 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.controllers.message.MessageIdempotencyTest: tests=7 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.controllers.user.UserControllerTest: tests=11 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepositoryTest: tests=3 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.repositories.chat.ExposedChatRepositoryTest: tests=2 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.services.attachment.ImageValidatorTest: tests=2 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.services.authentication.CrossReplicaRevocationTest: tests=2 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.services.authentication.SessionLookupSingleFlightTest: tests=4 failures=0
|
||||
com.mtusi.messenger.main.infrastructure.services.hashing.DefaultHashingServiceTest: tests=4 failures=0
|
||||
com.mtusi.messenger.main.plugins.CorsTest: tests=2 failures=0
|
||||
com.mtusi.messenger.main.plugins.DatabaseMigrationTest: tests=1 failures=0
|
||||
com.mtusi.messenger.main.plugins.DependencyFailureTest: tests=5 failures=0
|
||||
com.mtusi.messenger.main.plugins.InstanceIdTest: tests=1 failures=0
|
||||
com.mtusi.messenger.main.plugins.SessionCacheTest: tests=3 failures=0
|
||||
@@ -0,0 +1,17 @@
|
||||
Before restart:
|
||||
/mtusi-messenger-lab4-api-1 started=2026-11-04T23:04:21.983430629Z
|
||||
/mtusi-messenger-lab4-api-2 started=2026-11-04T23:03:16.431601751Z
|
||||
/mtusi-messenger-lab4-lb-1 started=2026-11-04T23:03:19.024302711Z
|
||||
/mtusi-messenger-lab4-postgres-1 started=2026-11-04T23:03:06.139259385Z
|
||||
/mtusi-messenger-lab4-redis-1 started=2026-11-04T23:03:06.148613844Z
|
||||
/mtusi-messenger-lab4-s3-1 started=2026-11-04T23:03:06.148211344Z
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8084"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
After restart:
|
||||
/mtusi-messenger-lab4-api-1 started=2026-11-04T23:04:56.021297464Z
|
||||
/mtusi-messenger-lab4-api-2 started=2026-11-04T23:04:56.113348339Z
|
||||
/mtusi-messenger-lab4-lb-1 started=2026-11-04T23:04:59.215310882Z
|
||||
/mtusi-messenger-lab4-postgres-1 started=2026-11-04T23:04:47.265519668Z
|
||||
/mtusi-messenger-lab4-redis-1 started=2026-11-04T23:04:47.259546585Z
|
||||
/mtusi-messenger-lab4-s3-1 started=2026-11-04T23:04:47.267464335Z
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://localhost:8084"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -14,7 +14,7 @@ defaults
|
||||
option forwardfor
|
||||
timeout connect 1s
|
||||
timeout client 30s
|
||||
timeout server 10s
|
||||
timeout server 30s
|
||||
timeout http-request 10s
|
||||
timeout http-keep-alive 5s
|
||||
# Retried only when the connection to a replica fails, i.e. before the request was sent:
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
# No tracing: credentials must not appear in logs.
|
||||
mc alias set local http://s3:9000 "$(cat /run/secrets/s3_access_key_id)" "$(cat /run/secrets/s3_secret_access_key)" >/dev/null
|
||||
mc mb --ignore-existing "local/$S3_BUCKET"
|
||||
mc anonymous set none "local/$S3_BUCKET"
|
||||
@@ -0,0 +1,6 @@
|
||||
storage {
|
||||
endpoint = "http://localhost:9000"
|
||||
bucket = "messenger-attachments"
|
||||
accessKeyFile = "../../secrets/s3_access_key_id"
|
||||
secretKeyFile = "../../secrets/s3_secret_access_key"
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stop the exact API instance which accepted the upload, then verify the original.
|
||||
Creates synthetic data; restores the stopped replica in finally. No credentials in output.
|
||||
"""
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / 'tests'))
|
||||
from client import Client, Failure, check
|
||||
|
||||
|
||||
def main():
|
||||
c = Client(timeout=10)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
content = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
result = c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=sample.png',
|
||||
201, actor=alice, raw=content, headers={'Content-Type': 'image/png'}, schema='Attachment')
|
||||
attachment = result['json']
|
||||
target = result['headers']['X-Instance-Id']
|
||||
c.message(alice, chat, '', attachments=[attachment['id']])
|
||||
print('Upload: 201 ready; instance=' + target, flush=True)
|
||||
# Probe the real known object anonymously from the internal network, not an unknown key.
|
||||
probe = subprocess.run(['docker', 'compose', 'exec', '-T', 'lb', 'wget', '-S', '-O', '/dev/null',
|
||||
'http://s3:9000/' + os.environ.get('S3_BUCKET', 'messenger-attachments') + '/originals/' + attachment['id']],
|
||||
cwd=ROOT, capture_output=True, text=True)
|
||||
check(probe.returncode != 0 and '403' in probe.stderr, 'private bucket did not reject anonymous GET')
|
||||
print('Anonymous S3 GET of known object: 403', flush=True)
|
||||
path = '/api/v1/attachments/' + attachment['id']
|
||||
c.call('GET', path, 404, actor=eve)
|
||||
c.call('GET', path + '/content', 404, actor=eve)
|
||||
print('Outsider metadata/content: 404', flush=True)
|
||||
try:
|
||||
subprocess.run([str(ROOT / 'scripts/stop-one.sh')], cwd=ROOT,
|
||||
env=dict(os.environ, TARGET_INSTANCE_ID=target), check=True, capture_output=True)
|
||||
deadline = time.monotonic() + 15
|
||||
while True:
|
||||
try:
|
||||
download = c.call('GET', path + '/content', actor=bob)
|
||||
check(download['headers']['X-Instance-Id'] != target, 'stopped instance returned content')
|
||||
check(download['bytes'] == content, 'download differs from original')
|
||||
print('Download: 200; instance=' + download['headers']['X-Instance-Id'], flush=True)
|
||||
print('SHA-256: ' + hashlib.sha256(content).hexdigest(), flush=True)
|
||||
break
|
||||
except Failure:
|
||||
if time.monotonic() >= deadline:
|
||||
raise
|
||||
time.sleep(0.2)
|
||||
finally:
|
||||
subprocess.run(['docker', 'compose', 'start', 'api'], cwd=ROOT, check=True, capture_output=True)
|
||||
print('PASS: uploading replica stopped; original served by another replica; stopped replica restored')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -9,7 +9,7 @@ mkdir -p secrets
|
||||
# Readable by the non-root users inside the containers (bind-mounted as is).
|
||||
umask 022
|
||||
|
||||
for name in db_password redis_password; do
|
||||
for name in db_password redis_password s3_access_key_id s3_secret_access_key; do
|
||||
if [ ! -s "secrets/$name" ]; then
|
||||
openssl rand -hex 24 > "secrets/$name"
|
||||
echo "created secrets/$name"
|
||||
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
#!/bin/sh
|
||||
# Restarts the balancer, every API replica, PostgreSQL and Redis, keeping the named volume (no `down -v`, no re-seed).
|
||||
# Restarts the balancer, every API replica, PostgreSQL, Redis and S3, keeping the named volume (no `down -v`, no re-seed).
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
docker compose stop lb api redis postgres
|
||||
docker compose stop lb api redis postgres s3
|
||||
docker compose up -d --wait
|
||||
|
||||
@@ -4,6 +4,10 @@ dependencyResolutionManagement {
|
||||
}
|
||||
|
||||
versionCatalogs {
|
||||
create("awssdk") {
|
||||
from("aws.sdk.kotlin:version-catalog:1.6.41")
|
||||
}
|
||||
|
||||
create("ktorLibs") {
|
||||
from("io.ktor:ktor-version-catalog:3.3.0")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
package com.mtusi.messenger.main.config.storage
|
||||
|
||||
import io.ktor.server.application.Application
|
||||
import io.ktor.server.config.getAs
|
||||
import org.koin.core.annotation.Provided
|
||||
import org.koin.core.annotation.Single
|
||||
|
||||
@Single
|
||||
class DefaultStorageConfig(
|
||||
@Provided app: Application
|
||||
) : StorageConfig by
|
||||
(app.environment.config.propertyOrNull("storage")?.getAs<SerialStorageConfig>() ?: SerialStorageConfig())
|
||||
@@ -0,0 +1,13 @@
|
||||
package com.mtusi.messenger.main.config.storage
|
||||
|
||||
import kotlinx.serialization.Serializable
|
||||
|
||||
@Serializable
|
||||
data class SerialStorageConfig(
|
||||
override val endpoint: String = "http://localhost:9000",
|
||||
override val bucket: String = "attachments",
|
||||
override val accessKey: String? = null,
|
||||
override val secretKey: String? = null,
|
||||
override val accessKeyFile: String? = null,
|
||||
override val secretKeyFile: String? = null,
|
||||
) : StorageConfig
|
||||
@@ -0,0 +1,10 @@
|
||||
package com.mtusi.messenger.main.config.storage
|
||||
|
||||
interface StorageConfig {
|
||||
val endpoint: String
|
||||
val bucket: String
|
||||
val accessKey: String?
|
||||
val secretKey: String?
|
||||
val accessKeyFile: String?
|
||||
val secretKeyFile: String?
|
||||
}
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
package com.mtusi.messenger.main.infrastructure.controllers.attachment
|
||||
|
||||
import com.mtusi.messenger.main.extensions.currentUser
|
||||
import com.mtusi.messenger.main.infrastructure.services.attachment.AttachmentService
|
||||
import com.mtusi.messenger.main.infrastructure.services.attachment.MAX_ATTACHMENT_BYTES
|
||||
import com.mtusi.messenger.main.infrastructure.types.ApiException
|
||||
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import com.mtusi.messenger.main.plugins.authentication.SESSION_AUTH
|
||||
import com.mtusi.messenger.main.utils.requireUuid
|
||||
import io.ktor.http.*
|
||||
import io.ktor.server.auth.authenticate
|
||||
import io.ktor.server.request.*
|
||||
import io.ktor.server.response.*
|
||||
import io.ktor.server.routing.*
|
||||
import io.ktor.utils.io.readAvailable
|
||||
import kotlinx.coroutines.withTimeoutOrNull
|
||||
import org.koin.ktor.ext.inject
|
||||
import java.io.ByteArrayOutputStream
|
||||
import kotlin.time.Duration.Companion.milliseconds
|
||||
|
||||
fun Routing.configureAttachmentRouting() {
|
||||
|
||||
val service by inject<AttachmentService>()
|
||||
|
||||
authenticate(SESSION_AUTH) {
|
||||
|
||||
post("/api/v1/chats/{chat_id}/attachments") {
|
||||
|
||||
val actor = call.currentUser
|
||||
val chatId = requireUuid(call.parameters["chat_id"], "chat_id")
|
||||
|
||||
service.requireMember(chatId, actor.id)
|
||||
|
||||
val filename = call.request.queryParameters["filename"]
|
||||
?: throw ApiException(ErrorCode.INVALID_REQUEST, "filename is required")
|
||||
|
||||
if (filename.codePointCount(0, filename.length) !in 1..255 || filename.any { it == '/' || it == '\\' || it.isISOControl() }) {
|
||||
throw ApiException(ErrorCode.INVALID_REQUEST, "invalid filename")
|
||||
}
|
||||
|
||||
val type = call.request.contentType().withoutParameters().toString()
|
||||
if (type !in setOf("image/png", "image/jpeg")) {
|
||||
throw ApiException(ErrorCode.UNSUPPORTED_MEDIA_TYPE, "expected PNG or JPEG")
|
||||
}
|
||||
|
||||
if ((call.request.contentLength() ?: 0) > MAX_ATTACHMENT_BYTES) {
|
||||
throw ApiException(ErrorCode.PAYLOAD_TOO_LARGE, "file exceeds 10 MiB")
|
||||
}
|
||||
|
||||
val bytes = withTimeoutOrNull(15_000.milliseconds) {
|
||||
|
||||
val input = call.receiveChannel()
|
||||
val output = ByteArrayOutputStream()
|
||||
val buffer = ByteArray(8192)
|
||||
|
||||
while (true) {
|
||||
val count = input.readAvailable(buffer, 0, minOf(buffer.size, MAX_ATTACHMENT_BYTES + 1 - output.size()))
|
||||
if (count == -1) break
|
||||
if (output.size() + count > MAX_ATTACHMENT_BYTES) {
|
||||
throw ApiException(ErrorCode.PAYLOAD_TOO_LARGE, "file exceeds 10 MiB")
|
||||
}
|
||||
output.write(buffer, 0, count)
|
||||
}
|
||||
|
||||
output.toByteArray()
|
||||
} ?: throw ApiException(ErrorCode.INVALID_REQUEST, "upload timed out")
|
||||
|
||||
call.respond(HttpStatusCode.Created, service.upload(chatId, actor.id, filename, type, bytes))
|
||||
}
|
||||
|
||||
get("/api/v1/attachments/{id}") {
|
||||
call.respond(service.find(requireUuid(call.parameters["id"], "id"), call.currentUser.id))
|
||||
}
|
||||
|
||||
get("/api/v1/attachments/{id}/content") {
|
||||
|
||||
val attachment = service.find(requireUuid(call.parameters["id"], "id"), call.currentUser.id)
|
||||
|
||||
if ((call.request.queryParameters["variant"] ?: "original") != "original") {
|
||||
throw ApiException(ErrorCode.NOT_FOUND, "variant not found")
|
||||
}
|
||||
|
||||
call.response.header(HttpHeaders.CacheControl, "private, no-store")
|
||||
call.response.header("X-Content-Type-Options", "nosniff")
|
||||
|
||||
call.respondBytes(service.content(attachment), ContentType.parse(attachment.contentType))
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
+20
-1
@@ -5,12 +5,19 @@ import com.mtusi.messenger.main.infrastructure.controllers.health.responses.Heal
|
||||
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import com.mtusi.messenger.main.plugins.isDraining
|
||||
import com.mtusi.messenger.main.plugins.suspendedTransaction
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.attachment.storage.ObjectStorageRepository
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
import org.koin.ktor.ext.inject
|
||||
import io.ktor.http.HttpStatusCode
|
||||
import io.ktor.server.application.log
|
||||
import io.ktor.server.response.respond
|
||||
import io.ktor.server.routing.Routing
|
||||
import io.ktor.server.routing.get
|
||||
import io.ktor.server.routing.route
|
||||
import kotlinx.coroutines.CancellationException
|
||||
import kotlinx.coroutines.currentCoroutineContext
|
||||
import kotlinx.coroutines.ensureActive
|
||||
|
||||
fun Routing.configureHealthRouting() = route("/health") {
|
||||
|
||||
@@ -18,7 +25,9 @@ fun Routing.configureHealthRouting() = route("/health") {
|
||||
call.respond(HttpStatusCode.OK, HealthResponse())
|
||||
}
|
||||
|
||||
// PostgreSQL is the only dependency of the synchronous path: without it nothing can be served.
|
||||
val storage by inject<ObjectStorageRepository>()
|
||||
|
||||
// PostgreSQL and S3 are dependencies of the synchronous path; Redis has a DB fallback.
|
||||
get("ready") {
|
||||
|
||||
// SIGTERM received: leave the balancer's rotation while requests in flight are still being finished.
|
||||
@@ -33,6 +42,11 @@ fun Routing.configureHealthRouting() = route("/health") {
|
||||
}
|
||||
true
|
||||
} catch (cause: Exception) {
|
||||
|
||||
if (cause is CancellationException) {
|
||||
currentCoroutineContext().ensureActive()
|
||||
}
|
||||
|
||||
call.application.log.warn("Readiness check failed: ${cause.message}")
|
||||
false
|
||||
}
|
||||
@@ -42,6 +56,11 @@ fun Routing.configureHealthRouting() = route("/health") {
|
||||
return@get
|
||||
}
|
||||
|
||||
if (!withContext(Dispatchers.IO) { storage.ready() }) {
|
||||
call.respondError(ErrorCode.UNAVAILABLE, "object storage unavailable")
|
||||
return@get
|
||||
}
|
||||
|
||||
call.respond(HttpStatusCode.OK, HealthResponse())
|
||||
}
|
||||
|
||||
|
||||
+9
-8
@@ -1,5 +1,6 @@
|
||||
package com.mtusi.messenger.main.infrastructure.controllers.message
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.types.ApiException
|
||||
import com.mtusi.messenger.main.extensions.currentUser
|
||||
import com.mtusi.messenger.main.extensions.respondError
|
||||
import com.mtusi.messenger.main.infrastructure.controllers._common.parameters.pageParameters
|
||||
@@ -12,7 +13,6 @@ import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import com.mtusi.messenger.main.plugins.authentication.SESSION_AUTH
|
||||
import com.mtusi.messenger.main.plugins.suspendedTransaction
|
||||
import com.mtusi.messenger.main.utils.requireIdempotencyKey
|
||||
import com.mtusi.messenger.main.utils.requireNonBlankBounded
|
||||
import com.mtusi.messenger.main.utils.requireUuid
|
||||
import io.ktor.http.HttpStatusCode
|
||||
import io.ktor.server.auth.authenticate
|
||||
@@ -37,19 +37,20 @@ fun Routing.configureMessageRouting() = route("/api/v1/chats/{chat_id}/messages"
|
||||
val idempotencyKey = requireIdempotencyKey(call.request.headers["Idempotency-Key"])
|
||||
val body = call.receive<CreateMessageRequest>()
|
||||
|
||||
requireNonBlankBounded(
|
||||
body.text,
|
||||
min = 1,
|
||||
max = MAX_MESSAGE_TEXT_LENGTH,
|
||||
field = "text"
|
||||
)
|
||||
val attachmentIds = body.attachmentIds.map { requireUuid(it, "attachment_ids") }
|
||||
if (attachmentIds.size > 10 || attachmentIds.distinct().size != attachmentIds.size ||
|
||||
body.text.codePointCount(0, body.text.length) > MAX_MESSAGE_TEXT_LENGTH ||
|
||||
(body.text.isBlank() && attachmentIds.isEmpty())) {
|
||||
throw ApiException(ErrorCode.INVALID_REQUEST, "invalid message")
|
||||
}
|
||||
|
||||
val result = suspendedTransaction {
|
||||
messageService.send(
|
||||
chatId = chatId,
|
||||
senderId = actor.id,
|
||||
text = body.text,
|
||||
idempotencyKey = idempotencyKey
|
||||
idempotencyKey = idempotencyKey,
|
||||
attachmentIds = attachmentIds
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
+1
@@ -5,4 +5,5 @@ import kotlinx.serialization.Serializable
|
||||
@Serializable
|
||||
data class CreateMessageRequest(
|
||||
val text: String,
|
||||
val attachmentIds: List<String> = emptyList(),
|
||||
)
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
package com.mtusi.messenger.main.infrastructure.dto
|
||||
|
||||
import com.mtusi.messenger.main.compat.serializers.SerialUUID
|
||||
import kotlinx.serialization.Serializable
|
||||
import kotlin.time.Instant
|
||||
|
||||
@Serializable
|
||||
data class Attachment(
|
||||
val id: SerialUUID,
|
||||
val chatId: SerialUUID,
|
||||
val ownerId: SerialUUID,
|
||||
val filename: String,
|
||||
val contentType: String,
|
||||
val sizeBytes: Long,
|
||||
val sha256: String,
|
||||
val createdAt: Instant,
|
||||
val updatedAt: Instant = createdAt,
|
||||
val status: String = "ready",
|
||||
val variants: List<String> = emptyList(),
|
||||
val errorCode: String? = null,
|
||||
)
|
||||
@@ -11,4 +11,5 @@ data class Message(
|
||||
val senderId: SerialUUID,
|
||||
val text: String,
|
||||
val createdAt: Instant,
|
||||
val attachmentIds: List<SerialUUID> = emptyList(),
|
||||
)
|
||||
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
package com.mtusi.messenger.main.infrastructure.models.attachment
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.models.chat.ChatModel
|
||||
import com.mtusi.messenger.main.infrastructure.models.user.UserModel
|
||||
import org.jetbrains.exposed.v1.core.dao.id.java.UUIDTable
|
||||
import org.jetbrains.exposed.v1.datetime.timestamp
|
||||
|
||||
object AttachmentModel : UUIDTable("attachments") {
|
||||
val chatId = reference("chat_id", ChatModel)
|
||||
val ownerId = reference("owner_id", UserModel)
|
||||
val filename = varchar("filename", 255)
|
||||
val contentType = varchar("content_type", 32)
|
||||
val sizeBytes = long("size_bytes")
|
||||
val sha256 = varchar("sha256", 64)
|
||||
val createdAt = timestamp("created_at")
|
||||
}
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
package com.mtusi.messenger.main.infrastructure.models.attachment
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.models.message.MessageModel
|
||||
import org.jetbrains.exposed.v1.core.Table
|
||||
|
||||
object MessageAttachmentModel : Table("message_attachments") {
|
||||
val messageId = reference("message_id", MessageModel)
|
||||
val attachmentId = reference("attachment_id", AttachmentModel)
|
||||
val position = integer("position")
|
||||
override val primaryKey = PrimaryKey(messageId, attachmentId)
|
||||
}
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
package com.mtusi.messenger.main.infrastructure.repositories.attachment
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.dto.Attachment
|
||||
import java.util.UUID
|
||||
|
||||
interface AttachmentRepository {
|
||||
|
||||
fun create(attachment: Attachment)
|
||||
|
||||
fun findById(id: UUID): Attachment?
|
||||
|
||||
fun attach(
|
||||
messageId: UUID,
|
||||
ids: List<UUID>
|
||||
)
|
||||
|
||||
fun idsForMessage(messageId: UUID): List<UUID>
|
||||
}
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
package com.mtusi.messenger.main.infrastructure.repositories.attachment
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.dto.Attachment
|
||||
import com.mtusi.messenger.main.infrastructure.models.attachment.AttachmentModel
|
||||
import com.mtusi.messenger.main.infrastructure.models.attachment.MessageAttachmentModel
|
||||
import org.jetbrains.exposed.v1.jdbc.insert
|
||||
import org.jetbrains.exposed.v1.jdbc.selectAll
|
||||
import org.jetbrains.exposed.v1.core.ResultRow
|
||||
import org.jetbrains.exposed.v1.core.eq
|
||||
import org.jetbrains.exposed.v1.core.SortOrder
|
||||
import org.koin.core.annotation.Single
|
||||
import java.util.UUID
|
||||
|
||||
@Single
|
||||
class ExposedAttachmentRepository : AttachmentRepository {
|
||||
|
||||
override fun create(attachment: Attachment) {
|
||||
AttachmentModel.insert {
|
||||
it[id] = attachment.id
|
||||
it[chatId] = attachment.chatId
|
||||
it[ownerId] = attachment.ownerId
|
||||
it[filename] = attachment.filename
|
||||
it[contentType] = attachment.contentType
|
||||
it[sizeBytes] = attachment.sizeBytes
|
||||
it[sha256] = attachment.sha256
|
||||
it[createdAt] = attachment.createdAt
|
||||
}
|
||||
}
|
||||
|
||||
override fun findById(
|
||||
id: UUID
|
||||
): Attachment? {
|
||||
return AttachmentModel
|
||||
.selectAll()
|
||||
.where(AttachmentModel.id eq id)
|
||||
.firstOrNull()
|
||||
?.toAttachment()
|
||||
}
|
||||
|
||||
override fun attach(
|
||||
messageId: UUID,
|
||||
ids: List<UUID>
|
||||
) {
|
||||
ids.forEachIndexed { index, id ->
|
||||
MessageAttachmentModel.insert {
|
||||
it[this.messageId] = messageId
|
||||
it[attachmentId] = id
|
||||
it[position] = index
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
override fun idsForMessage(
|
||||
messageId: UUID
|
||||
): List<UUID> {
|
||||
return MessageAttachmentModel
|
||||
.selectAll()
|
||||
.where(MessageAttachmentModel.messageId eq messageId)
|
||||
.orderBy(MessageAttachmentModel.position to SortOrder.ASC)
|
||||
.map {
|
||||
it[MessageAttachmentModel.attachmentId].value
|
||||
}
|
||||
}
|
||||
|
||||
private fun ResultRow.toAttachment() = Attachment(
|
||||
id = this[AttachmentModel.id].value,
|
||||
chatId = this[AttachmentModel.chatId].value,
|
||||
ownerId = this[AttachmentModel.ownerId].value,
|
||||
filename = this[AttachmentModel.filename],
|
||||
contentType = this[AttachmentModel.contentType],
|
||||
sizeBytes = this[AttachmentModel.sizeBytes],
|
||||
sha256 = this[AttachmentModel.sha256],
|
||||
createdAt = this[AttachmentModel.createdAt]
|
||||
)
|
||||
|
||||
}
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
package com.mtusi.messenger.main.infrastructure.repositories.attachment.storage
|
||||
|
||||
import java.util.UUID
|
||||
|
||||
interface ObjectStorageRepository {
|
||||
|
||||
suspend fun put(
|
||||
id: UUID,
|
||||
bytes: ByteArray,
|
||||
contentType: String
|
||||
)
|
||||
|
||||
suspend fun get(id: UUID): ByteArray
|
||||
|
||||
suspend fun delete(id: UUID)
|
||||
|
||||
suspend fun ready(): Boolean
|
||||
}
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
package com.mtusi.messenger.main.infrastructure.repositories.attachment.storage
|
||||
|
||||
import aws.sdk.kotlin.runtime.auth.credentials.StaticCredentialsProvider
|
||||
import aws.sdk.kotlin.services.s3.S3Client
|
||||
import aws.sdk.kotlin.services.s3.deleteObject
|
||||
import aws.sdk.kotlin.services.s3.model.GetObjectRequest
|
||||
import aws.sdk.kotlin.services.s3.headBucket
|
||||
import aws.sdk.kotlin.services.s3.putObject
|
||||
import aws.smithy.kotlin.runtime.content.ByteStream
|
||||
import aws.smithy.kotlin.runtime.content.toByteArray
|
||||
import aws.smithy.kotlin.runtime.net.url.Url
|
||||
import com.mtusi.messenger.main.config.storage.StorageConfig
|
||||
import com.mtusi.messenger.main.infrastructure.services.attachment.MAX_ATTACHMENT_BYTES
|
||||
import com.mtusi.messenger.main.infrastructure.types.ApiException
|
||||
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import com.mtusi.messenger.main.utils.resolveSecret
|
||||
import kotlinx.coroutines.CancellationException
|
||||
import kotlinx.coroutines.withTimeoutOrNull
|
||||
import org.koin.core.annotation.Single
|
||||
import java.util.UUID
|
||||
import kotlin.time.Duration.Companion.milliseconds
|
||||
|
||||
@Single
|
||||
class S3ObjectStorageRepository(
|
||||
private val config: StorageConfig
|
||||
) : ObjectStorageRepository {
|
||||
|
||||
private val client by lazy {
|
||||
S3Client {
|
||||
endpointUrl = Url.parse(config.endpoint)
|
||||
region = "us-east-1"
|
||||
forcePathStyle = true
|
||||
|
||||
credentialsProvider = StaticCredentialsProvider {
|
||||
accessKeyId = requireNotNull(resolveSecret(config.accessKey, config.accessKeyFile))
|
||||
secretAccessKey = requireNotNull(resolveSecret(config.secretKey, config.secretKeyFile))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
override suspend fun put(
|
||||
id: UUID,
|
||||
bytes: ByteArray,
|
||||
contentType: String
|
||||
) {
|
||||
request {
|
||||
client.putObject {
|
||||
bucket = config.bucket
|
||||
key = getObjectKey(id)
|
||||
body = ByteStream.fromBytes(bytes)
|
||||
this.contentType = contentType
|
||||
contentLength = bytes.size.toLong()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
override suspend fun get(
|
||||
id: UUID
|
||||
): ByteArray = request {
|
||||
client.getObject(GetObjectRequest {
|
||||
bucket = config.bucket
|
||||
key = getObjectKey(id)
|
||||
}) { response ->
|
||||
|
||||
val contentLength = requireNotNull(response.contentLength)
|
||||
|
||||
check(contentLength >= 1L && contentLength <= MAX_ATTACHMENT_BYTES.toLong())
|
||||
|
||||
val bytes = requireNotNull(response.body?.toByteArray())
|
||||
|
||||
check(bytes.size <= MAX_ATTACHMENT_BYTES)
|
||||
|
||||
bytes
|
||||
}
|
||||
}
|
||||
|
||||
override suspend fun delete(
|
||||
id: UUID
|
||||
) {
|
||||
request {
|
||||
client.deleteObject {
|
||||
bucket = config.bucket
|
||||
key = getObjectKey(id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
override suspend fun ready(): Boolean {
|
||||
return try {
|
||||
request {
|
||||
client.headBucket {
|
||||
bucket = config.bucket
|
||||
}
|
||||
}
|
||||
true
|
||||
} catch (_: ApiException) {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
private fun getObjectKey(id: UUID) = "originals/$id"
|
||||
|
||||
private suspend fun <T : Any> request(block: suspend () -> T): T {
|
||||
return try {
|
||||
withTimeoutOrNull(8000.milliseconds) {
|
||||
block()
|
||||
} ?: throw ApiException(ErrorCode.UNAVAILABLE, "object storage unavailable")
|
||||
} catch (cause: CancellationException) {
|
||||
throw cause
|
||||
} catch (_: Exception) {
|
||||
throw ApiException(ErrorCode.UNAVAILABLE, "object storage unavailable")
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
+5
-1
@@ -1,5 +1,6 @@
|
||||
package com.mtusi.messenger.main.infrastructure.repositories.message
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.attachment.AttachmentRepository
|
||||
import com.mtusi.messenger.main.infrastructure.dto.Message
|
||||
import com.mtusi.messenger.main.infrastructure.models.message.MessageModel
|
||||
import com.mtusi.messenger.main.utils.PageKey
|
||||
@@ -18,7 +19,9 @@ import java.util.UUID
|
||||
import kotlin.time.Clock
|
||||
|
||||
@Single
|
||||
class ExposedMessageRepository : MessageRepository {
|
||||
class ExposedMessageRepository(
|
||||
private val attachments: AttachmentRepository
|
||||
) : MessageRepository {
|
||||
|
||||
override fun create(
|
||||
chatId: UUID,
|
||||
@@ -85,6 +88,7 @@ class ExposedMessageRepository : MessageRepository {
|
||||
chatId = this[MessageModel.chatId].value,
|
||||
senderId = this[MessageModel.senderId].value,
|
||||
text = this[MessageModel.text],
|
||||
attachmentIds = attachments.idsForMessage(this[MessageModel.id].value),
|
||||
createdAt = this[MessageModel.createdAt]
|
||||
)
|
||||
|
||||
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
package com.mtusi.messenger.main.infrastructure.services.attachment
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.dto.Attachment
|
||||
import java.util.UUID
|
||||
|
||||
interface AttachmentService {
|
||||
|
||||
suspend fun requireMember(
|
||||
chatId: UUID,
|
||||
actorId: UUID
|
||||
)
|
||||
|
||||
suspend fun upload(
|
||||
chatId: UUID,
|
||||
actorId: UUID,
|
||||
filename: String,
|
||||
contentType: String,
|
||||
bytes: ByteArray
|
||||
): Attachment
|
||||
|
||||
suspend fun find(
|
||||
id: UUID,
|
||||
actorId: UUID
|
||||
): Attachment
|
||||
|
||||
suspend fun content(attachment: Attachment): ByteArray
|
||||
}
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
package com.mtusi.messenger.main.infrastructure.services.attachment
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.dto.Attachment
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.attachment.AttachmentRepository
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.attachment.storage.ObjectStorageRepository
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.chat.ChatRepository
|
||||
import com.mtusi.messenger.main.infrastructure.types.ApiException
|
||||
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import com.mtusi.messenger.main.plugins.suspendedTransaction
|
||||
import com.mtusi.messenger.main.utils.computeSha256
|
||||
import com.mtusi.messenger.main.utils.nowMicros
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
import org.koin.core.annotation.Single
|
||||
import java.util.UUID
|
||||
import kotlin.time.Clock
|
||||
|
||||
@Single
|
||||
class DefaultAttachmentService(
|
||||
private val attachmentRepository: AttachmentRepository,
|
||||
private val chatRepository: ChatRepository,
|
||||
private val objectStorageRepository: ObjectStorageRepository,
|
||||
) : AttachmentService {
|
||||
|
||||
override suspend fun requireMember(
|
||||
chatId: UUID,
|
||||
actorId: UUID
|
||||
) {
|
||||
if (!suspendedTransaction { chatRepository.isMember(chatId, actorId) }) {
|
||||
notFound()
|
||||
}
|
||||
}
|
||||
|
||||
override suspend fun upload(
|
||||
chatId: UUID,
|
||||
actorId: UUID,
|
||||
filename: String,
|
||||
contentType: String,
|
||||
bytes: ByteArray
|
||||
): Attachment {
|
||||
|
||||
requireMember(chatId, actorId)
|
||||
|
||||
val type = withContext(Dispatchers.IO) { ImageValidator.validate(bytes, contentType) }
|
||||
val attachment = Attachment(
|
||||
id = UUID.randomUUID(),
|
||||
chatId = chatId,
|
||||
ownerId = actorId,
|
||||
filename = filename,
|
||||
contentType = type,
|
||||
sizeBytes = bytes.size.toLong(),
|
||||
sha256 = bytes.computeSha256(),
|
||||
createdAt = Clock.System.nowMicros()
|
||||
)
|
||||
|
||||
objectStorageRepository.put(attachment.id, bytes, type)
|
||||
|
||||
suspendedTransaction {
|
||||
attachmentRepository.create(attachment)
|
||||
}
|
||||
return attachment
|
||||
}
|
||||
|
||||
override suspend fun find(
|
||||
id: UUID,
|
||||
actorId: UUID
|
||||
): Attachment = suspendedTransaction {
|
||||
val attachment = attachmentRepository.findById(id) ?: notFound()
|
||||
if (!chatRepository.isMember(attachment.chatId, actorId)) {
|
||||
notFound()
|
||||
}
|
||||
attachment
|
||||
}
|
||||
|
||||
override suspend fun content(attachment: Attachment): ByteArray = withContext(Dispatchers.IO) {
|
||||
objectStorageRepository.get(attachment.id)
|
||||
}
|
||||
|
||||
private fun notFound(): Nothing = throw ApiException(ErrorCode.NOT_FOUND, "attachment or chat not found")
|
||||
}
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
package com.mtusi.messenger.main.infrastructure.services.attachment
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.types.ApiException
|
||||
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import java.nio.ByteBuffer
|
||||
import java.util.zip.CRC32
|
||||
import java.io.ByteArrayInputStream
|
||||
import javax.imageio.ImageIO
|
||||
import javax.imageio.stream.MemoryCacheImageInputStream
|
||||
|
||||
const val MAX_ATTACHMENT_BYTES = 10 * 1024 * 1024
|
||||
private const val MAX_PIXELS = 40_000_000L
|
||||
|
||||
object ImageValidator {
|
||||
|
||||
fun validate(
|
||||
bytes: ByteArray,
|
||||
declaredType: String
|
||||
): String {
|
||||
|
||||
if (bytes.size > MAX_ATTACHMENT_BYTES) {
|
||||
throw ApiException(ErrorCode.PAYLOAD_TOO_LARGE, "file exceeds 10 MiB")
|
||||
}
|
||||
|
||||
try {
|
||||
MemoryCacheImageInputStream(ByteArrayInputStream(bytes)).use { input ->
|
||||
val readers = ImageIO.getImageReaders(input)
|
||||
if (!readers.hasNext()) {
|
||||
invalid()
|
||||
}
|
||||
|
||||
val reader = readers.next()
|
||||
|
||||
try {
|
||||
reader.input = input
|
||||
|
||||
val type = when (reader.formatName.lowercase()) {
|
||||
"png" -> "image/png"
|
||||
"jpeg", "jpg" -> "image/jpeg"
|
||||
else -> invalid()
|
||||
}
|
||||
|
||||
if (declaredType != type) {
|
||||
invalid()
|
||||
}
|
||||
|
||||
if (type == "image/png") {
|
||||
validatePngChunks(bytes)
|
||||
}
|
||||
|
||||
if (type == "image/jpeg" && (
|
||||
bytes.size < 4 ||
|
||||
bytes[bytes.lastIndex - 1] != 0xff.toByte() ||
|
||||
bytes.last() != 0xd9.toByte()
|
||||
)) {
|
||||
invalid()
|
||||
}
|
||||
|
||||
val width = reader.getWidth(0)
|
||||
val height = reader.getHeight(0)
|
||||
|
||||
if (width <= 0 || height <= 0) {
|
||||
invalid()
|
||||
}
|
||||
|
||||
if (width.toLong() * height > MAX_PIXELS) {
|
||||
throw ApiException(ErrorCode.PAYLOAD_TOO_LARGE, "image exceeds 40 million pixels")
|
||||
}
|
||||
|
||||
var warning = false
|
||||
reader.addIIOReadWarningListener { _, _ -> warning = true }
|
||||
|
||||
val image = reader.read(0) ?: invalid()
|
||||
image.flush()
|
||||
|
||||
if (warning) {
|
||||
invalid()
|
||||
}
|
||||
|
||||
return type
|
||||
} finally {
|
||||
reader.dispose()
|
||||
}
|
||||
}
|
||||
} catch (cause: ApiException) {
|
||||
throw cause
|
||||
} catch (_: Exception) {
|
||||
invalid()
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
private fun validatePngChunks(bytes: ByteArray) {
|
||||
|
||||
var offset = 8
|
||||
|
||||
while (offset + 12 <= bytes.size) {
|
||||
val length = ByteBuffer.wrap(bytes, offset, 4).int
|
||||
|
||||
if (length < 0 || length.toLong() + offset + 12 > bytes.size) {
|
||||
invalid()
|
||||
}
|
||||
|
||||
val type = String(bytes, offset + 4, 4, Charsets.US_ASCII)
|
||||
|
||||
val crc = CRC32().apply { update(bytes, offset + 4, length + 4) }.value
|
||||
val expected = ByteBuffer.wrap(bytes, offset + 8 + length, 4).int.toLong() and 0xffffffffL
|
||||
if (crc != expected) {
|
||||
invalid()
|
||||
}
|
||||
|
||||
offset += length + 12
|
||||
if (type == "IEND") {
|
||||
if (length != 0 || offset != bytes.size) {
|
||||
invalid()
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
invalid()
|
||||
}
|
||||
|
||||
private fun invalid(): Nothing = throw ApiException(
|
||||
ErrorCode.UNSUPPORTED_MEDIA_TYPE, "invalid PNG/JPEG or MIME mismatch"
|
||||
)
|
||||
}
|
||||
+29
-8
@@ -1,5 +1,8 @@
|
||||
package com.mtusi.messenger.main.infrastructure.services.message
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import com.mtusi.messenger.main.infrastructure.types.ApiException
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.attachment.AttachmentRepository
|
||||
import com.mtusi.messenger.main.config.message.MessageConfig
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.chat.ChatRepository
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.message.MessageRepository
|
||||
@@ -24,6 +27,7 @@ class DefaultMessageService(
|
||||
private val chatRepository: ChatRepository,
|
||||
private val messageRepository: MessageRepository,
|
||||
private val idempotencyRepository: MessageIdempotencyRepository,
|
||||
private val attachments: AttachmentRepository,
|
||||
private val config: MessageConfig
|
||||
) : MessageService {
|
||||
|
||||
@@ -31,7 +35,8 @@ class DefaultMessageService(
|
||||
chatId: UUID,
|
||||
senderId: UUID,
|
||||
text: String,
|
||||
idempotencyKey: String
|
||||
idempotencyKey: String,
|
||||
attachmentIds: List<UUID>
|
||||
): MessageCreationResult {
|
||||
|
||||
// Access is checked on every retry too: a replay must not outlive the membership.
|
||||
@@ -39,8 +44,15 @@ class DefaultMessageService(
|
||||
return MessageCreationResult.ChatNotFound
|
||||
}
|
||||
|
||||
// The body has a single field, so its hash is the hash of the text.
|
||||
val requestHash = text.toByteArray().computeSha256()
|
||||
attachmentIds.forEach { id ->
|
||||
if (attachments.findById(id)?.chatId != chatId)
|
||||
throw ApiException(
|
||||
ErrorCode.NOT_FOUND, "attachment not found")
|
||||
}
|
||||
|
||||
val requestHash = (text + if (attachmentIds.isEmpty()) "" else "\u0000attachments:" + attachmentIds.joinToString(","))
|
||||
.toByteArray().computeSha256()
|
||||
|
||||
val now = Clock.System.nowMicros()
|
||||
|
||||
repeat(CLAIM_ATTEMPTS) {
|
||||
@@ -52,9 +64,10 @@ class DefaultMessageService(
|
||||
senderId = senderId,
|
||||
text = text
|
||||
)
|
||||
attachments.attach(message.id, attachmentIds)
|
||||
idempotencyRepository.attachMessage(senderId, chatId, idempotencyKey, message.id)
|
||||
|
||||
return MessageCreationResult.Success(message)
|
||||
return MessageCreationResult.Success(message.copy(attachmentIds = attachmentIds))
|
||||
}
|
||||
|
||||
// The key is taken by a committed transaction (claim waits for an uncommitted one).
|
||||
@@ -72,11 +85,19 @@ class DefaultMessageService(
|
||||
return MessageCreationResult.IdempotencyKeyReused
|
||||
}
|
||||
|
||||
val messageId = checkNotNull(record.messageId) { "committed idempotency key without a message" }
|
||||
val messageId = checkNotNull(record.messageId) {
|
||||
"committed idempotency key without a message"
|
||||
}
|
||||
|
||||
return MessageCreationResult.Success(
|
||||
checkNotNull(messageRepository.findById(messageId)) { "message of an idempotency key is missing" }
|
||||
)
|
||||
val original = checkNotNull(messageRepository.findById(messageId)) {
|
||||
"message of an idempotency key is missing"
|
||||
}
|
||||
|
||||
if (original.text != text || original.attachmentIds != attachmentIds) {
|
||||
return MessageCreationResult.IdempotencyKeyReused
|
||||
}
|
||||
|
||||
return MessageCreationResult.Success(original)
|
||||
}
|
||||
|
||||
error("Idempotency key could not be claimed")
|
||||
|
||||
+2
-1
@@ -10,7 +10,8 @@ interface MessageService {
|
||||
chatId: UUID,
|
||||
senderId: UUID,
|
||||
text: String,
|
||||
idempotencyKey: String
|
||||
idempotencyKey: String,
|
||||
attachmentIds: List<UUID> = emptyList()
|
||||
): MessageCreationResult
|
||||
|
||||
fun findPageByChatId(
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
package com.mtusi.messenger.main.plugins
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.models.attachment.MessageAttachmentModel
|
||||
import com.mtusi.messenger.main.infrastructure.models.attachment.AttachmentModel
|
||||
import com.mtusi.messenger.main.config.database.DatabaseConfig
|
||||
import com.mtusi.messenger.main.config.database.SerialDatabaseConfig
|
||||
import com.mtusi.messenger.main.infrastructure.models.authentication.SessionModel
|
||||
@@ -117,6 +119,8 @@ private fun generateMigration(description: String) {
|
||||
require(description.isNotEmpty()) { "Description cannot be empty" }
|
||||
|
||||
val tables: Array<Table> = arrayOf(
|
||||
AttachmentModel,
|
||||
MessageAttachmentModel,
|
||||
UserModel,
|
||||
ChatModel,
|
||||
ChatMemberModel,
|
||||
|
||||
@@ -5,6 +5,7 @@ import com.mtusi.messenger.main.infrastructure.controllers.chat.configureChatRou
|
||||
import com.mtusi.messenger.main.infrastructure.controllers.health.configureHealthRouting
|
||||
import com.mtusi.messenger.main.infrastructure.controllers.message.configureMessageRouting
|
||||
import com.mtusi.messenger.main.infrastructure.controllers.user.configureUserRouting
|
||||
import com.mtusi.messenger.main.infrastructure.controllers.attachment.configureAttachmentRouting
|
||||
import io.ktor.server.application.Application
|
||||
import io.ktor.server.routing.routing
|
||||
|
||||
@@ -15,4 +16,5 @@ fun Application.configureRouting() = routing {
|
||||
configureAuthenticationRouting()
|
||||
configureChatRouting()
|
||||
configureMessageRouting()
|
||||
configureAttachmentRouting()
|
||||
}
|
||||
|
||||
@@ -98,3 +98,15 @@ authentication {
|
||||
sessionRevokeTimeoutMs = 8000
|
||||
sessionRevokeTimeoutMs = ${?SESSION_REVOKE_TIMEOUT_MS}
|
||||
}
|
||||
|
||||
# S3 credentials are local files, never source-controlled values.
|
||||
storage {
|
||||
endpoint = "http://localhost:9000"
|
||||
endpoint = ${?S3_ENDPOINT}
|
||||
bucket = "messenger-attachments"
|
||||
bucket = ${?S3_BUCKET}
|
||||
accessKey = ${?S3_ACCESS_KEY_ID}
|
||||
secretKey = ${?S3_SECRET_ACCESS_KEY}
|
||||
accessKeyFile = ${?S3_ACCESS_KEY_ID_FILE}
|
||||
secretKeyFile = ${?S3_SECRET_ACCESS_KEY_FILE}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
CREATE TABLE attachments (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID NOT NULL REFERENCES chats(id),
|
||||
owner_id UUID NOT NULL REFERENCES users(id),
|
||||
filename VARCHAR(255) NOT NULL,
|
||||
content_type VARCHAR(32) NOT NULL CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||
size_bytes BIGINT NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 VARCHAR(64) NOT NULL,
|
||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL
|
||||
);
|
||||
CREATE INDEX attachments_chat_id_idx ON attachments(chat_id);
|
||||
CREATE TABLE message_attachments (
|
||||
message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id),
|
||||
position INTEGER NOT NULL CHECK (position BETWEEN 0 AND 9),
|
||||
PRIMARY KEY (message_id, attachment_id),
|
||||
UNIQUE (message_id, position)
|
||||
);
|
||||
CREATE INDEX message_attachments_attachment_id_idx ON message_attachments(attachment_id);
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
package com.mtusi.messenger.main.infrastructure.controllers.attachment
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.dto.Attachment
|
||||
import com.mtusi.messenger.main.infrastructure.dto.Message
|
||||
import com.mtusi.messenger.main.infrastructure.controllers.message.requests.CreateMessageRequest
|
||||
import com.mtusi.messenger.main.support.*
|
||||
import io.ktor.client.call.body
|
||||
import io.ktor.client.request.*
|
||||
import io.ktor.http.*
|
||||
import java.awt.image.BufferedImage
|
||||
import java.io.ByteArrayOutputStream
|
||||
import javax.imageio.ImageIO
|
||||
import kotlin.test.*
|
||||
|
||||
class AttachmentControllerTest {
|
||||
@Test
|
||||
|
||||
fun `upload originals ACL message links and idempotency`() = runApp { client ->
|
||||
val alice = client.register("alice")
|
||||
val bob = client.register("bob")
|
||||
val outsider = client.register("outsider")
|
||||
val chat = client.createChat(alice, "files", listOf(alice.id, bob.id))
|
||||
for (format in listOf("png", "jpeg")) {
|
||||
val bytes = image(format)
|
||||
val upload = client.post("/api/v1/chats/${chat.id}/attachments?filename=photo.$format") {
|
||||
auth(alice)
|
||||
contentType(ContentType.parse("image/$format"))
|
||||
setBody(bytes)
|
||||
}
|
||||
assertEquals(HttpStatusCode.Created, upload.status)
|
||||
val attachment = upload.body<Attachment>()
|
||||
assertEquals("ready", attachment.status)
|
||||
assertEquals(bytes.size.toLong(), attachment.sizeBytes)
|
||||
val path = "/api/v1/attachments/${attachment.id}"
|
||||
val download = client.get("$path/content") { auth(bob) }
|
||||
assertEquals(HttpStatusCode.OK, download.status)
|
||||
assertContentEquals(bytes, download.body<ByteArray>())
|
||||
assertEquals(HttpStatusCode.NotFound, client.get(path) { auth(outsider) }.status)
|
||||
assertEquals(HttpStatusCode.NotFound, client.get("$path/content") { auth(outsider) }.status)
|
||||
assertEquals(HttpStatusCode.Unauthorized, client.get("$path/content").status)
|
||||
assertEquals(HttpStatusCode.NotFound, client.get("$path/content?variant=thumbnail") { auth(bob) }.status)
|
||||
val key = "file-$format"
|
||||
suspend fun send(ids: List<String>) = client.post("/api/v1/chats/${chat.id}/messages") {
|
||||
auth(bob)
|
||||
idempotencyKey(key)
|
||||
contentType(ContentType.Application.Json)
|
||||
setBody(CreateMessageRequest("", ids))
|
||||
}
|
||||
val first = send(listOf(attachment.id.toString()))
|
||||
assertEquals(HttpStatusCode.Created, first.status)
|
||||
val message = first.body<Message>()
|
||||
assertEquals(listOf(attachment.id), message.attachmentIds)
|
||||
assertEquals(message, send(listOf(attachment.id.toString())).body<Message>())
|
||||
val otherChat = client.createChat(bob, "other", listOf(bob.id))
|
||||
assertEquals(HttpStatusCode.NotFound, client.post("/api/v1/chats/${otherChat.id}/messages") {
|
||||
auth(bob)
|
||||
idempotencyKey()
|
||||
contentType(ContentType.Application.Json)
|
||||
setBody(CreateMessageRequest("", listOf(attachment.id.toString())))
|
||||
}.status)
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
fun `invalid images filenames and byte limit return client errors`() = runApp { client ->
|
||||
val user = client.register("upload")
|
||||
val chat = client.createChat(user, "files", listOf(user.id))
|
||||
suspend fun upload(
|
||||
bytes: ByteArray,
|
||||
type: String = "image/png",
|
||||
name: String = "photo.png"
|
||||
) =
|
||||
client.post("/api/v1/chats/${chat.id}/attachments") {
|
||||
auth(user)
|
||||
parameter("filename", name)
|
||||
contentType(ContentType.parse(type))
|
||||
setBody(bytes)
|
||||
}
|
||||
assertEquals(HttpStatusCode.UnsupportedMediaType, upload("not an image".toByteArray()).status)
|
||||
assertEquals(HttpStatusCode.UnsupportedMediaType, upload(image("png"), "image/jpeg").status)
|
||||
assertEquals(HttpStatusCode.UnsupportedMediaType, upload(image("png").copyOf(40)).status)
|
||||
for (name in listOf("../x.png", "a\\b.png", "a\n.png", "a\u0000.png", "x".repeat(256)))
|
||||
assertEquals(HttpStatusCode.BadRequest, upload(image("png"), name = name).status)
|
||||
assertEquals(HttpStatusCode.PayloadTooLarge, upload(ByteArray(10 * 1024 * 1024 + 1)).status)
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
fun `S3 outage fails upload download and readiness without losing the original`() = runApp { client ->
|
||||
val user = client.register("outage")
|
||||
val chat = client.createChat(user, "files", listOf(user.id))
|
||||
val bytes = image("png")
|
||||
suspend fun upload() = client.post("/api/v1/chats/${chat.id}/attachments?filename=photo.png") {
|
||||
auth(user)
|
||||
contentType(ContentType.Image.PNG)
|
||||
setBody(bytes)
|
||||
}
|
||||
val attachment = upload().body<Attachment>()
|
||||
val path = "/api/v1/attachments/${attachment.id}/content"
|
||||
val container = TestStorage.container
|
||||
container.dockerClient.pauseContainerCmd(container.containerId).exec()
|
||||
try {
|
||||
assertEquals(HttpStatusCode.ServiceUnavailable, upload().status)
|
||||
assertEquals(HttpStatusCode.ServiceUnavailable, client.get(path) { auth(user) }.status)
|
||||
assertEquals(HttpStatusCode.ServiceUnavailable, client.get("/health/ready").status)
|
||||
assertEquals(HttpStatusCode.OK, client.get("/health/live").status)
|
||||
} finally {
|
||||
container.dockerClient.unpauseContainerCmd(container.containerId).exec()
|
||||
}
|
||||
assertContentEquals(bytes, client.get(path) { auth(user) }.body<ByteArray>())
|
||||
assertEquals(HttpStatusCode.OK, client.get("/health/ready").status)
|
||||
}
|
||||
|
||||
private fun image(format: String): ByteArray = ByteArrayOutputStream().use {
|
||||
ImageIO.write(BufferedImage(32, 20, BufferedImage.TYPE_INT_RGB), format, it)
|
||||
it.toByteArray()
|
||||
}
|
||||
}
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
package com.mtusi.messenger.main.infrastructure.services.attachment
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.types.ApiException
|
||||
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import java.awt.image.BufferedImage
|
||||
import java.io.ByteArrayOutputStream
|
||||
import java.nio.ByteBuffer
|
||||
import java.util.zip.CRC32
|
||||
import javax.imageio.ImageIO
|
||||
import kotlin.test.*
|
||||
|
||||
class ImageValidatorTest {
|
||||
|
||||
private fun png() = ByteArrayOutputStream().also {
|
||||
ImageIO.write(BufferedImage(2, 2, BufferedImage.TYPE_INT_RGB), "png", it)
|
||||
}.toByteArray()
|
||||
|
||||
@Test
|
||||
fun `pixel limit is enforced on header before decoding`() {
|
||||
val bytes = png()
|
||||
ByteBuffer.wrap(bytes, 16, 8).putInt(10000).putInt(4001)
|
||||
val crc = CRC32().apply { update(bytes, 12, 17) }.value
|
||||
ByteBuffer.wrap(bytes, 29, 4).putInt(crc.toInt())
|
||||
assertEquals(ErrorCode.PAYLOAD_TOO_LARGE, assertFailsWith<ApiException> {
|
||||
ImageValidator.validate(bytes, "image/png")
|
||||
}.code)
|
||||
}
|
||||
@Test
|
||||
fun `corrupted CRC and missing end chunk are rejected`() {
|
||||
val good = png()
|
||||
for (bytes in listOf(good.copyOf(good.size - 12), good.clone().apply { this[29] = (this[29].toInt() xor 1).toByte() })) {
|
||||
assertEquals(ErrorCode.UNSUPPORTED_MEDIA_TYPE, assertFailsWith<ApiException> {
|
||||
ImageValidator.validate(bytes, "image/png")
|
||||
}.code)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -62,6 +62,10 @@ private fun testConfig(
|
||||
) = HoconApplicationConfig(
|
||||
ConfigFactory.parseMap(
|
||||
mapOf(
|
||||
"storage.endpoint" to TestStorage.endpoint,
|
||||
"storage.bucket" to TestStorage.BUCKET,
|
||||
"storage.accessKey" to TestStorage.ACCESS_KEY,
|
||||
"storage.secretKey" to TestStorage.SECRET_KEY,
|
||||
"database.host" to database.host,
|
||||
"database.port" to database.firstMappedPort,
|
||||
"database.name" to database.databaseName,
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
package com.mtusi.messenger.main.support
|
||||
|
||||
import aws.sdk.kotlin.runtime.auth.credentials.StaticCredentialsProvider
|
||||
import aws.sdk.kotlin.services.s3.S3Client
|
||||
import aws.sdk.kotlin.services.s3.createBucket
|
||||
import aws.smithy.kotlin.runtime.net.url.Url
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import org.testcontainers.containers.GenericContainer
|
||||
import org.testcontainers.containers.wait.strategy.Wait
|
||||
|
||||
object TestStorage {
|
||||
const val ACCESS_KEY = "test-access-key"
|
||||
const val SECRET_KEY = "test-secret-key-for-local-tests"
|
||||
const val BUCKET = "test-attachments"
|
||||
val container by lazy {
|
||||
GenericContainer("quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e")
|
||||
.withEnv("MINIO_ROOT_USER", ACCESS_KEY)
|
||||
.withEnv("MINIO_ROOT_PASSWORD", SECRET_KEY)
|
||||
.withCommand("server", "/data")
|
||||
.withExposedPorts(9000)
|
||||
.waitingFor(Wait.forHttp("/minio/health/live").forPort(9000))
|
||||
.also {
|
||||
it.start()
|
||||
}
|
||||
}
|
||||
val endpoint: String by lazy {
|
||||
val url = "http://${container.host}:${container.getMappedPort(9000)}"
|
||||
runBlocking {
|
||||
S3Client {
|
||||
endpointUrl = Url.parse(url)
|
||||
region = "us-east-1"
|
||||
forcePathStyle = true
|
||||
credentialsProvider = StaticCredentialsProvider {
|
||||
accessKeyId = ACCESS_KEY
|
||||
secretAccessKey = SECRET_KEY
|
||||
}
|
||||
}.use { client ->
|
||||
client.createBucket {
|
||||
bucket = BUCKET
|
||||
}
|
||||
}
|
||||
}
|
||||
url
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user