feat: harden attachment storage for level 4
This commit is contained in:
@@ -20,3 +20,10 @@ MINIO_ROOT_USER=minio-root
|
||||
MINIO_ROOT_PASSWORD=replace-with-a-different-long-random-root-secret
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
MAX_CONCURRENT_UPLOADS=3
|
||||
UPLOAD_PERMIT_WAIT_MILLIS=10000
|
||||
# Orphans younger than this are never deleted: protects the S3 PUT -> DB COMMIT window.
|
||||
ATTACHMENT_CLEANUP_GRACE_SECONDS=3600
|
||||
ATTACHMENT_CLEANUP_DRY_RUN=false
|
||||
ATTACHMENT_RECONCILE_INTERVAL_MILLIS=600000
|
||||
ATTACHMENT_RECONCILE_INITIAL_DELAY_MILLIS=60000
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Отчёт — лабораторная 4
|
||||
|
||||
- Автор, группа: Егор Антипин, БПИ2301
|
||||
- Целевая оценка; выбранный трек 5, если нужен: 3 (обязательный lab4)
|
||||
- Целевая оценка; выбранный трек 5, если нужен: 4
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: контракт репозитория; MinIO Java SDK 8.6.0; документация MinIO/S3; помощь Codex при реализации и проверке.
|
||||
|
||||
@@ -17,6 +17,9 @@ docker compose --env-file .env up -d --build --scale api=2
|
||||
docker compose --env-file .env ps
|
||||
python3 tests/smoke.py --base-url http://localhost:8080
|
||||
python3 student_tests/lab4_negative.py --base-url http://localhost:8080
|
||||
ENV_FILE=.env ./scripts/check-private-bucket.sh
|
||||
python3 student_tests/concurrent_upload_memory.py --base-url http://localhost:8080 \
|
||||
--compose-env-file .env --concurrency 12 --output evidence/upload-memory.json
|
||||
```
|
||||
|
||||
`migrate` применяет Flyway V1–V4 один раз. `minio-init` идемпотентно создаёт приватный bucket и ограниченного API-user. Обычная безопасная остановка: `docker compose --env-file .env down` — named volumes сохраняются. Намеренное удаление только учебных данных: `docker compose --env-file .env down --volumes`.
|
||||
@@ -27,7 +30,11 @@ python3 student_tests/lab4_negative.py --base-url http://localhost:8080
|
||||
|
||||
`attachments` содержит chat/owner, display filename, истинный MIME, size, SHA-256 и непрозрачный object key вида `attachments/<server-uuid>/original`. `message_attachments` использует составные FK `(message_id, chat_id)` и `(attachment_id, chat_id)`, поэтому база не позволяет связать сообщение с вложением другого чата.
|
||||
|
||||
Upload читает raw body порциями с жёсткой границей 10 MiB, проверяет filename, реальный PNG/JPEG через ImageIO, совпадение MIME и ≤40 млн пикселей. После успешного S3 PUT метаданные фиксируются отдельной DB-транзакцией. При ошибке DB выполняется компенсирующий DELETE объекта. Ответ 201 возвращается только после обеих записей. Ошибка MinIO преобразуется в JSON 503 без внутренних деталей.
|
||||
Upload читает raw body порциями по 8 KiB с жёсткой границей 10 MiB, проверяет filename, реальный PNG/JPEG через ImageIO, совпадение MIME и ≤40 млн пикселей. До чтения body захватывается один из трёх fair semaphore permits на реплику; ожидание ограничено 10 секундами и возвращает 429 с `Retry-After`. Nginx ограничивает body 11 MiB, ожидание тела 10 секундами, запрос к API 20 секундами; MinIO client имеет connect 2 с и общий call timeout 15 с. Поэтому число одновременно удерживаемых byte/decode buffers ограничено.
|
||||
|
||||
После успешного S3 PUT метаданные фиксируются отдельной DB-транзакцией. При ошибке DB выполняется компенсирующий DELETE объекта. Обрыв входного потока происходит до PUT и не создаёт object. Ответ 201 возвращается только после обеих записей. Ошибка MinIO преобразуется в JSON 503 без внутренних деталей.
|
||||
|
||||
Для аварии процесса между S3 PUT и DB COMMIT работает `AttachmentReconciler`: каждые 10 минут он сравнивает `attachments.object_key` с prefix `attachments/` в MinIO. Orphan удаляется только если `lastModified` старше grace period 3600 секунд; свежий объект не трогается, что защищает нормальный commit window. Missing object не удаляет метаданные автоматически, а пишет ERROR для ручного расследования. Есть `ATTACHMENT_CLEANUP_DRY_RUN=true`. Unit-тесты доказывают old/fresh orphan, dry-run и missing-object поведение.
|
||||
|
||||
Download сначала загружает метаданные и проверяет членство пользователя в чате, затем читает object через API. Redirect/presigned URL не используется. Filename участвует только в `Content-Disposition`.
|
||||
|
||||
@@ -38,24 +45,26 @@ Compose и Swarm используют отдельный persistent volume `mini
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | `python -X utf8 tests/smoke.py --base-url http://localhost:18084` | PASS, включая upload/message/download/ACL | `evidence/lab4-2026-11-11.md` |
|
||||
| Private bucket/bootstrap | `docker compose logs minio-init` | bucket private, API policy attached | `evidence/lab4-2026-11-11.md` |
|
||||
| Private bucket/bootstrap | повторный `minio-init`; `scripts/check-private-bucket.cmd` | bootstrap повторяем; anonymous list запрещён; API credentials non-admin | `evidence/lab4-2026-11-11.md` |
|
||||
| Негативные файлы и чужой чат | `python -X utf8 student_tests/lab4_negative.py --base-url http://localhost:18084` | PASS | `evidence/lab4-2026-11-11.md` |
|
||||
| Обрыв/DB failure/reconciler | `AttachmentServiceTest`, `AttachmentReconcilerTest` | PASS: нет object до PUT; compensating DELETE; grace/dry-run/missing | `evidence/lab4-2026-11-11.md` |
|
||||
| 12 конкурентных upload + RAM | `student_tests/concurrent_upload_memory.py ... --concurrency 12` | 12/12; peak 839909375 B aggregate; +171442175 B | `evidence/upload-memory-2026-11-11.json` |
|
||||
| Полный restart | `tests/resilience.py --mode persistence ...` | PASS; session/message/original preserved | `evidence/lab4-2026-11-11.md` |
|
||||
| Отказ API-реплики | `tests/resilience.py --mode failover ...` | PASS; download через другую replica | `evidence/lab4-2026-11-11.md` |
|
||||
| Недоступность S3 | stop MinIO + `lab4_negative.py --expect-s3-unavailable` | JSON 503 unavailable; после start smoke PASS | `evidence/lab4-2026-11-11.md` |
|
||||
|
||||
Unit-набор `AttachmentServiceTest` отдельно проверяет server-generated key, MIME spoofing, ранний byte limit, ACL до download, чужой chat attachment и отказ ObjectStorage до записи метаданных.
|
||||
Unit-набор `AttachmentServiceTest` отдельно проверяет server-generated key, MIME spoofing, ранний byte limit, ACL до download, чужой chat attachment, обрыв request body, отказ ObjectStorage и compensating DELETE после DB failure. `AttachmentReconcilerTest` проверяет grace period, dry-run и missing objects. Полный Gradle suite внутри Compose network: `BUILD SUCCESSFUL`.
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
Проверялась функциональная устойчивость, не максимальная производительность. Стенд: Docker Desktop, 2 API replicas по 768 MiB, MinIO 512 MiB, PostgreSQL 768 MiB. Fixture — PNG 111817 bytes. Нагрузочный эксперимент уровня 4 не заявлен.
|
||||
Проверялась функциональная устойчивость и ограниченность памяти, не максимальная производительность. Стенд: Docker Desktop, 2 API replicas по 768 MiB, MinIO 512 MiB, PostgreSQL 768 MiB. Конкурентный эксперимент сгенерировал один валидный несжимаемый PNG 1800×1200, 6835833 bytes, и одновременно отправил его 12 раз. Все 12 upload завершились 201 за 10.156 с. По пяти `docker stats` samples aggregate API baseline был 668467200 B, peak 839909375 B, рост 171442175 B. Результат повторяемый, но sampling может пропустить краткий пик и не является профилированием JVM heap.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| Stop обслужившей API replica | Запросы продолжает вторая replica; общий MinIO сохраняет original | 2 instance до отказа, 105 успешных записей, 0 ошибок | 1.636 с до подтверждённого восстановления |
|
||||
| Stop обслужившей API replica | Запросы продолжает вторая replica; общий MinIO сохраняет original | 2 instance до отказа, 102 успешные записи, 0 ошибок | 1.386 с до подтверждённого восстановления |
|
||||
| Restart nginx/API/Redis/PostgreSQL/MinIO без удаления volumes | Сессия, message, attachment metadata и bytes сохранятся | PASS persistence | <90 с, потерь нет |
|
||||
| Stop MinIO | Upload/download не превращаются в 500 | JSON 503 `unavailable`; после start публичный smoke снова PASS | Восстановление после healthcheck MinIO |
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
Стенд выдерживает отказ одной API-реплики и перезапуск всех сервисов при сохранённых named volumes. MinIO развёрнут одной репликой: во время его отказа attachment upload/download недоступны с 503, но PostgreSQL API остаётся работоспособным. HA самого S3, multipart upload, presigned flow, thumbnail pipeline и автоматический reconciler orphan/missing objects не входят в заявленный уровень. Компенсирующий DELETE закрывает обычную ошибку DB после S3 PUT, но аварийное завершение процесса между двумя действиями потребовало бы отдельного reconciler для уровня 4.
|
||||
Стенд выдерживает отказ одной API-реплики и перезапуск всех сервисов при сохранённых named volumes. MinIO развёрнут одной репликой: во время его отказа attachment upload/download недоступны с 503, но PostgreSQL API остаётся работоспособным. HA самого S3, multipart upload, presigned flow и thumbnail pipeline не реализованы. Reconciler автоматически удаляет только старые orphan objects; missing objects намеренно не чинятся автоматически, чтобы не скрывать потерю данных.
|
||||
|
||||
@@ -28,6 +28,12 @@ x-api-environment: &api-environment
|
||||
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
|
||||
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
|
||||
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||
MAX_CONCURRENT_UPLOADS: ${MAX_CONCURRENT_UPLOADS:-3}
|
||||
UPLOAD_PERMIT_WAIT_MILLIS: ${UPLOAD_PERMIT_WAIT_MILLIS:-10000}
|
||||
ATTACHMENT_CLEANUP_GRACE_SECONDS: ${ATTACHMENT_CLEANUP_GRACE_SECONDS:-3600}
|
||||
ATTACHMENT_CLEANUP_DRY_RUN: ${ATTACHMENT_CLEANUP_DRY_RUN:-false}
|
||||
ATTACHMENT_RECONCILE_INTERVAL_MILLIS: ${ATTACHMENT_RECONCILE_INTERVAL_MILLIS:-600000}
|
||||
ATTACHMENT_RECONCILE_INITIAL_DELAY_MILLIS: ${ATTACHMENT_RECONCILE_INITIAL_DELAY_MILLIS:-60000}
|
||||
|
||||
services:
|
||||
postgres:
|
||||
@@ -107,6 +113,35 @@ services:
|
||||
restart: "no"
|
||||
networks: [backend]
|
||||
|
||||
private-bucket-check:
|
||||
profiles: ["verification"]
|
||||
image: bitnamilegacy/minio-client:2025.4.16-debian-12-r0@sha256:28e38f52642c06a2344fa974cf55786b4c52601b6b9b0676c68d40ee0876a5bc
|
||||
depends_on:
|
||||
minio-init:
|
||||
condition: service_completed_successfully
|
||||
environment:
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command:
|
||||
- |
|
||||
set -eu
|
||||
mc alias set anonymous http://minio:9000
|
||||
if mc ls "anonymous/$$S3_BUCKET" >/dev/null 2>&1; then
|
||||
echo "FAIL: anonymous bucket listing succeeded" >&2
|
||||
exit 1
|
||||
fi
|
||||
mc alias set restricted http://minio:9000 "$$S3_ACCESS_KEY_ID" "$$S3_SECRET_ACCESS_KEY"
|
||||
mc ls "restricted/$$S3_BUCKET" >/dev/null
|
||||
if mc admin info restricted >/dev/null 2>&1; then
|
||||
echo "FAIL: API credentials have administrative access" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "PASS: bucket is private and API credentials are non-admin"
|
||||
restart: "no"
|
||||
networks: [backend]
|
||||
|
||||
# One-shot Flyway runner. It is invoked explicitly before API replicas.
|
||||
migrate:
|
||||
image: hla-lab4-api:local
|
||||
|
||||
@@ -12,10 +12,11 @@
|
||||
## Проверки
|
||||
|
||||
```text
|
||||
AttachmentServiceTest: BUILD SUCCESSFUL, 6 tests.
|
||||
AttachmentServiceTest + AttachmentReconcilerTest: BUILD SUCCESSFUL, 10 tests.
|
||||
tests/smoke.py: PASS: lab4 public API scenario.
|
||||
student_tests/lab4_negative.py:
|
||||
PASS: foreign chat/user, MIME spoof, corruption, byte limit and filename checks.
|
||||
PASS: unknown/foreign attachment, foreign chat/user, MIME spoof, corruption,
|
||||
byte limit and filename checks.
|
||||
```
|
||||
|
||||
Persistence hook останавливал nginx, обе API, Redis, PostgreSQL и MinIO, затем запускал те же named volumes и повторял idempotent bucket bootstrap:
|
||||
@@ -27,7 +28,7 @@ PASS: persistence preserved confirmed data and existing session.
|
||||
Failover hook остановил replica из `X-Instance-Id`, обслужившую тестовый запрос:
|
||||
|
||||
```json
|
||||
{"observed_instances_before":2,"successful_writes":105,"failed_attempts":0,"first_recovery_seconds":1.636}
|
||||
{"observed_instances_before":2,"successful_writes":102,"failed_attempts":0,"first_recovery_seconds":1.386}
|
||||
```
|
||||
|
||||
```text
|
||||
@@ -41,3 +42,38 @@ PASS: unavailable S3 returns contract JSON 503
|
||||
```
|
||||
|
||||
После запуска того же MinIO volume публичный smoke повторно завершился PASS, checksum скачанного original совпал.
|
||||
|
||||
## Level 4: consistency, memory and private S3
|
||||
|
||||
Unit tests inject an interrupted request stream before S3 PUT and a repository
|
||||
failure after S3 PUT. The first leaves storage empty; the second verifies the
|
||||
compensating DELETE. Reconciler tests prove that an orphan younger than the
|
||||
one-hour grace period survives, an older orphan is deleted, dry-run never
|
||||
deletes, and a missing object is reported.
|
||||
|
||||
Bootstrap was executed again against the existing bucket and completed with
|
||||
exit code 0. The direct S3 probe (without application API) produced:
|
||||
|
||||
```text
|
||||
PASS: bucket is private and API credentials are non-admin
|
||||
```
|
||||
|
||||
Full Gradle suite using the live Compose PostgreSQL/Redis/MinIO network:
|
||||
|
||||
```text
|
||||
BUILD SUCCESSFUL in 11m 47s
|
||||
```
|
||||
|
||||
Concurrent memory experiment (`evidence/upload-memory-2026-11-11.json`):
|
||||
|
||||
```json
|
||||
{"concurrency":12,"successful_uploads":12,"payload_bytes_each":6835833,
|
||||
"duration_seconds":10.156,"api_replicas":2,"memory_samples":5,
|
||||
"baseline_api_memory_bytes":668467200,"peak_api_memory_bytes":839909375,
|
||||
"peak_increase_bytes":171442175,"configured_max_concurrent_uploads_per_replica":3}
|
||||
```
|
||||
|
||||
The memory values are the aggregate of both API containers from `docker stats`.
|
||||
Sampling can miss a shorter peak, so this is an operational bound observation,
|
||||
not a JVM heap proof. The code-level bounds are 10 MiB/request, 40M pixels and
|
||||
three active uploads per replica.
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"concurrency": 12,
|
||||
"successful_uploads": 12,
|
||||
"payload_bytes_each": 6835833,
|
||||
"duration_seconds": 10.156,
|
||||
"api_replicas": 2,
|
||||
"memory_samples": 5,
|
||||
"baseline_api_memory_bytes": 668467200,
|
||||
"peak_api_memory_bytes": 839909375,
|
||||
"peak_increase_bytes": 171442175,
|
||||
"configured_max_concurrent_uploads_per_replica": 3,
|
||||
"configured_max_upload_bytes": 10485760,
|
||||
"configured_max_image_pixels": 40000000
|
||||
}
|
||||
@@ -25,3 +25,24 @@
|
||||
Политика содержит placeholder `__S3_BUCKET__`; bootstrap подставляет фактическое
|
||||
значение `S3_BUCKET` перед созданием policy. Повторный запуск безопасен: bucket,
|
||||
user и policy переиспользуются, а private anonymous policy устанавливается снова.
|
||||
|
||||
Проверка вне API:
|
||||
|
||||
```sh
|
||||
ENV_FILE=.env ./scripts/check-private-bucket.sh
|
||||
```
|
||||
|
||||
Одноразовый `private-bucket-check` подключается непосредственно к S3 внутри
|
||||
backend network без HTTP API приложения. Он требует, чтобы anonymous list bucket
|
||||
завершился отказом, проверяет list с ограниченными API credentials и требует,
|
||||
чтобы `mc admin info` с этими credentials также завершился отказом. Сервис имеет
|
||||
Compose profile `verification` и не работает постоянно.
|
||||
|
||||
## Сверка PostgreSQL и MinIO
|
||||
|
||||
API запускает `AttachmentReconciler` с prefix `attachments/`. Метаданные из
|
||||
PostgreSQL сравниваются с объектами MinIO. Старый orphan удаляется после
|
||||
`ATTACHMENT_CLEANUP_GRACE_SECONDS` (по умолчанию 3600), свежий orphan защищён от
|
||||
гонки PUT/COMMIT. `ATTACHMENT_CLEANUP_DRY_RUN=true` только сообщает кандидатов.
|
||||
Missing object логируется как ERROR, но metadata автоматически не удаляется.
|
||||
Интервал и initial delay задаются через `.env.example`.
|
||||
|
||||
@@ -8,6 +8,7 @@ events {
|
||||
http {
|
||||
# Let the application enforce the exact 10 MiB contract and return JSON.
|
||||
client_max_body_size 11m;
|
||||
client_body_timeout 10s;
|
||||
access_log /var/log/nginx/access.log;
|
||||
error_log /var/log/nginx/error.log warn;
|
||||
|
||||
@@ -66,6 +67,22 @@ http {
|
||||
proxy_next_upstream_tries 2;
|
||||
}
|
||||
|
||||
location ~ ^/api/v1/chats/[0-9a-fA-F-]+/attachments$ {
|
||||
proxy_pass http://api_backend;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_request_buffering on;
|
||||
|
||||
proxy_connect_timeout 500ms;
|
||||
proxy_send_timeout 20s;
|
||||
proxy_read_timeout 20s;
|
||||
# Upload is not replayed: it has no idempotency key. The client may retry explicitly.
|
||||
proxy_next_upstream off;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass http://api_backend;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
@echo off
|
||||
setlocal EnableExtensions
|
||||
cd /d "%~dp0\.."
|
||||
if not defined ENV_FILE set "ENV_FILE=.env"
|
||||
|
||||
docker compose --env-file "%ENV_FILE%" --profile verification run --rm private-bucket-check
|
||||
exit /b %ERRORLEVEL%
|
||||
@@ -0,0 +1,9 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
PROJECT_DIR=$(dirname -- "$SCRIPT_DIR")
|
||||
cd "$PROJECT_DIR"
|
||||
: "${ENV_FILE:=.env}"
|
||||
|
||||
docker compose --env-file "$ENV_FILE" --profile verification run --rm private-bucket-check
|
||||
@@ -3,8 +3,10 @@ package com.example.mtusi_messenger;
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
import org.springframework.context.ConfigurableApplicationContext;
|
||||
import org.springframework.scheduling.annotation.EnableScheduling;
|
||||
|
||||
@SpringBootApplication
|
||||
@EnableScheduling
|
||||
public class MtusiMessengerApplication {
|
||||
|
||||
public static void main(String[] args) {
|
||||
|
||||
@@ -23,7 +23,7 @@ public record CreateMessageRequest(
|
||||
}
|
||||
|
||||
@AssertTrue(message = "attachment_ids must be unique and non-null")
|
||||
public boolean areAttachmentIdsValid() {
|
||||
public boolean isAttachmentIdsValid() {
|
||||
return attachment_ids != null
|
||||
&& attachment_ids.stream().noneMatch(java.util.Objects::isNull)
|
||||
&& new HashSet<>(attachment_ids).size() == attachment_ids.size();
|
||||
|
||||
@@ -12,5 +12,6 @@ public final class ApiExceptions {
|
||||
public static ApiException conflict(String m) { return new ApiException("conflict", HttpStatus.CONFLICT, m); }
|
||||
public static ApiException payloadTooLarge(String m) { return new ApiException("payload_too_large", HttpStatus.PAYLOAD_TOO_LARGE, m); }
|
||||
public static ApiException unsupportedMediaType(String m) { return new ApiException("unsupported_media_type", HttpStatus.UNSUPPORTED_MEDIA_TYPE, m); }
|
||||
public static ApiException tooManyRequests(String m) { return new ApiException("too_many_requests", HttpStatus.TOO_MANY_REQUESTS, m); }
|
||||
public static ApiException unavailable(String m) { return new ApiException("unavailable", HttpStatus.SERVICE_UNAVAILABLE, m); }
|
||||
}
|
||||
|
||||
@@ -5,9 +5,11 @@ import com.example.mtusi_messenger.model.Attachment;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
import java.util.Set;
|
||||
|
||||
public interface AttachmentRepository {
|
||||
Attachment save(Attachment attachment);
|
||||
Optional<Attachment> findById(UUID id);
|
||||
List<Attachment> findAllByIdsInChat(UUID chatId, List<UUID> ids);
|
||||
Set<String> findAllObjectKeys();
|
||||
}
|
||||
|
||||
@@ -9,6 +9,8 @@ import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
import java.util.Set;
|
||||
import java.util.HashSet;
|
||||
|
||||
@Repository
|
||||
public class PostgresAttachmentRepository implements AttachmentRepository {
|
||||
@@ -58,6 +60,12 @@ public class PostgresAttachmentRepository implements AttachmentRepository {
|
||||
(rs, rowNum) -> map(rs), args);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Set<String> findAllObjectKeys() {
|
||||
return new HashSet<>(jdbc.query("SELECT object_key FROM attachments",
|
||||
(rs, rowNum) -> rs.getString("object_key")));
|
||||
}
|
||||
|
||||
private static Attachment map(java.sql.ResultSet rs) throws java.sql.SQLException {
|
||||
return new Attachment(
|
||||
rs.getObject("id", UUID.class),
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
package com.example.mtusi_messenger.service;
|
||||
|
||||
import com.example.mtusi_messenger.repository.AttachmentRepository;
|
||||
import com.example.mtusi_messenger.storage.ObjectStorage;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.scheduling.annotation.Scheduled;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.HashSet;
|
||||
import java.util.Set;
|
||||
|
||||
@Component
|
||||
public class AttachmentReconciler {
|
||||
private static final Logger log = LoggerFactory.getLogger(AttachmentReconciler.class);
|
||||
private static final String PREFIX = "attachments/";
|
||||
private final AttachmentRepository attachments;
|
||||
private final ObjectStorage storage;
|
||||
private final Duration gracePeriod;
|
||||
private final boolean dryRun;
|
||||
private final Clock clock;
|
||||
|
||||
@Autowired
|
||||
public AttachmentReconciler(
|
||||
AttachmentRepository attachments,
|
||||
ObjectStorage storage,
|
||||
@Value("${messenger.attachments.cleanup-grace-seconds:3600}") long graceSeconds,
|
||||
@Value("${messenger.attachments.cleanup-dry-run:false}") boolean dryRun
|
||||
) {
|
||||
this(attachments, storage, Duration.ofSeconds(graceSeconds), dryRun, Clock.systemUTC());
|
||||
}
|
||||
|
||||
AttachmentReconciler(AttachmentRepository attachments,
|
||||
ObjectStorage storage,
|
||||
Duration gracePeriod,
|
||||
boolean dryRun,
|
||||
Clock clock) {
|
||||
if (gracePeriod.isNegative() || gracePeriod.isZero()) {
|
||||
throw new IllegalArgumentException("Attachment cleanup grace period must be positive");
|
||||
}
|
||||
this.attachments = attachments;
|
||||
this.storage = storage;
|
||||
this.gracePeriod = gracePeriod;
|
||||
this.dryRun = dryRun;
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
@Scheduled(
|
||||
fixedDelayString = "${messenger.attachments.reconcile-interval-millis:600000}",
|
||||
initialDelayString = "${messenger.attachments.reconcile-initial-delay-millis:60000}")
|
||||
public void scheduledReconcile() {
|
||||
try {
|
||||
Report report = reconcile();
|
||||
if (report.orphanObjects() > 0 || report.missingObjects() > 0) {
|
||||
log.warn("Attachment reconciliation: {} orphan, {} deleted, {} missing, dryRun={}",
|
||||
report.orphanObjects(), report.deletedObjects(), report.missingObjects(), dryRun);
|
||||
}
|
||||
} catch (RuntimeException error) {
|
||||
log.warn("Attachment reconciliation postponed: {}", error.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
public Report reconcile() {
|
||||
Set<String> databaseKeys = attachments.findAllObjectKeys();
|
||||
var storedObjects = storage.list(PREFIX);
|
||||
Set<String> storageKeys = new HashSet<>();
|
||||
Instant deleteBefore = clock.instant().minus(gracePeriod);
|
||||
int orphans = 0;
|
||||
int deleted = 0;
|
||||
|
||||
for (ObjectStorage.StoredObject object : storedObjects) {
|
||||
storageKeys.add(object.objectKey());
|
||||
if (!databaseKeys.contains(object.objectKey()) && object.lastModified().isBefore(deleteBefore)) {
|
||||
orphans++;
|
||||
if (!dryRun) {
|
||||
storage.delete(object.objectKey());
|
||||
deleted++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
int missing = 0;
|
||||
for (String databaseKey : databaseKeys) {
|
||||
if (!storageKeys.contains(databaseKey)) {
|
||||
missing++;
|
||||
log.error("Attachment metadata references missing object {}", databaseKey);
|
||||
}
|
||||
}
|
||||
return new Report(databaseKeys.size(), storedObjects.size(), orphans, deleted, missing, dryRun);
|
||||
}
|
||||
|
||||
public record Report(int metadataObjects,
|
||||
int storageObjects,
|
||||
int orphanObjects,
|
||||
int deletedObjects,
|
||||
int missingObjects,
|
||||
boolean dryRun) {
|
||||
}
|
||||
}
|
||||
@@ -25,6 +25,8 @@ import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.Semaphore;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
@Service
|
||||
public class AttachmentService {
|
||||
@@ -35,6 +37,8 @@ public class AttachmentService {
|
||||
private final TransactionTemplate transaction;
|
||||
private final long maxUploadBytes;
|
||||
private final long maxImagePixels;
|
||||
private final Semaphore uploadPermits;
|
||||
private final long uploadPermitWaitMillis;
|
||||
|
||||
public AttachmentService(
|
||||
AttachmentRepository attachments,
|
||||
@@ -42,7 +46,9 @@ public class AttachmentService {
|
||||
ObjectStorage storage,
|
||||
TransactionTemplate transaction,
|
||||
@Value("${messenger.attachments.max-upload-bytes:10485760}") long maxUploadBytes,
|
||||
@Value("${messenger.attachments.max-image-pixels:40000000}") long maxImagePixels
|
||||
@Value("${messenger.attachments.max-image-pixels:40000000}") long maxImagePixels,
|
||||
@Value("${messenger.attachments.max-concurrent-uploads:3}") int maxConcurrentUploads,
|
||||
@Value("${messenger.attachments.upload-permit-wait-millis:10000}") long uploadPermitWaitMillis
|
||||
) {
|
||||
this.attachments = attachments;
|
||||
this.chats = chats;
|
||||
@@ -50,6 +56,11 @@ public class AttachmentService {
|
||||
this.transaction = transaction;
|
||||
this.maxUploadBytes = maxUploadBytes;
|
||||
this.maxImagePixels = maxImagePixels;
|
||||
if (maxConcurrentUploads < 1 || uploadPermitWaitMillis < 1) {
|
||||
throw new IllegalArgumentException("Upload concurrency limits must be positive");
|
||||
}
|
||||
this.uploadPermits = new Semaphore(maxConcurrentUploads, true);
|
||||
this.uploadPermitWaitMillis = uploadPermitWaitMillis;
|
||||
}
|
||||
|
||||
public Attachment upload(UUID chatId,
|
||||
@@ -65,6 +76,30 @@ public class AttachmentService {
|
||||
throw ApiExceptions.payloadTooLarge("Attachment exceeds 10 MiB");
|
||||
}
|
||||
|
||||
try {
|
||||
try {
|
||||
if (!uploadPermits.tryAcquire(uploadPermitWaitMillis, TimeUnit.MILLISECONDS)) {
|
||||
throw ApiExceptions.tooManyRequests("Upload capacity is temporarily exhausted");
|
||||
}
|
||||
} catch (InterruptedException interrupted) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw ApiExceptions.unavailable("Upload was interrupted");
|
||||
}
|
||||
try {
|
||||
return storeValidated(chatId, ownerId, safeFilename, contentType, input);
|
||||
} finally {
|
||||
uploadPermits.release();
|
||||
}
|
||||
} catch (com.example.mtusi_messenger.exception.ApiException error) {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private Attachment storeValidated(UUID chatId,
|
||||
UUID ownerId,
|
||||
String safeFilename,
|
||||
String contentType,
|
||||
InputStream input) {
|
||||
byte[] bytes = readBounded(input);
|
||||
if (bytes.length == 0) {
|
||||
throw ApiExceptions.unsupportedMediaType("Image is empty");
|
||||
|
||||
@@ -2,6 +2,7 @@ package com.example.mtusi_messenger.storage;
|
||||
|
||||
import io.minio.GetObjectArgs;
|
||||
import io.minio.MinioClient;
|
||||
import io.minio.ListObjectsArgs;
|
||||
import io.minio.PutObjectArgs;
|
||||
import io.minio.RemoveObjectArgs;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
@@ -10,6 +11,8 @@ import okhttp3.OkHttpClient;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
@Component
|
||||
public class MinioObjectStorage implements ObjectStorage {
|
||||
@@ -71,4 +74,19 @@ public class MinioObjectStorage implements ObjectStorage {
|
||||
throw new ObjectStorageException("Cannot remove attachment object", error);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<StoredObject> list(String prefix) {
|
||||
try {
|
||||
List<StoredObject> objects = new ArrayList<>();
|
||||
for (var result : client.listObjects(ListObjectsArgs.builder()
|
||||
.bucket(bucket).prefix(prefix).recursive(true).build())) {
|
||||
var item = result.get();
|
||||
objects.add(new StoredObject(item.objectName(), item.lastModified().toInstant(), item.size()));
|
||||
}
|
||||
return List.copyOf(objects);
|
||||
} catch (Exception error) {
|
||||
throw new ObjectStorageException("Cannot list attachment objects", error);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,7 +1,14 @@
|
||||
package com.example.mtusi_messenger.storage;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
|
||||
public interface ObjectStorage {
|
||||
void put(String objectKey, byte[] bytes, String contentType);
|
||||
byte[] get(String objectKey);
|
||||
void delete(String objectKey);
|
||||
List<StoredObject> list(String prefix);
|
||||
|
||||
record StoredObject(String objectKey, Instant lastModified, long sizeBytes) {
|
||||
}
|
||||
}
|
||||
|
||||
@@ -23,6 +23,9 @@ public class GlobalExceptionHandler {
|
||||
if (e.status() == HttpStatus.UNAUTHORIZED && e.authenticate() != null) {
|
||||
builder.header("WWW-Authenticate", e.authenticate());
|
||||
}
|
||||
if (e.status() == HttpStatus.TOO_MANY_REQUESTS) {
|
||||
builder.header("Retry-After", "1");
|
||||
}
|
||||
return builder.body(body);
|
||||
}
|
||||
@ExceptionHandler({MethodArgumentNotValidException.class, MethodArgumentTypeMismatchException.class, HttpMessageNotReadableException.class})
|
||||
|
||||
@@ -69,3 +69,9 @@ messenger:
|
||||
attachments:
|
||||
max-upload-bytes: ${MAX_UPLOAD_BYTES:10485760}
|
||||
max-image-pixels: ${MAX_IMAGE_PIXELS:40000000}
|
||||
max-concurrent-uploads: ${MAX_CONCURRENT_UPLOADS:3}
|
||||
upload-permit-wait-millis: ${UPLOAD_PERMIT_WAIT_MILLIS:10000}
|
||||
cleanup-grace-seconds: ${ATTACHMENT_CLEANUP_GRACE_SECONDS:3600}
|
||||
cleanup-dry-run: ${ATTACHMENT_CLEANUP_DRY_RUN:false}
|
||||
reconcile-interval-millis: ${ATTACHMENT_RECONCILE_INTERVAL_MILLIS:600000}
|
||||
reconcile-initial-delay-millis: ${ATTACHMENT_RECONCILE_INITIAL_DELAY_MILLIS:60000}
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
package com.example.mtusi_messenger.service;
|
||||
|
||||
import com.example.mtusi_messenger.repository.AttachmentRepository;
|
||||
import com.example.mtusi_messenger.storage.ObjectStorage;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
class AttachmentReconcilerTest {
|
||||
private static final Instant NOW = Instant.parse("2026-11-11T12:00:00Z");
|
||||
|
||||
@Test
|
||||
void deletesOnlyOrphansOlderThanGracePeriodAndReportsMissingObjects() {
|
||||
AttachmentRepository repository = mock(AttachmentRepository.class);
|
||||
when(repository.findAllObjectKeys()).thenReturn(Set.of(
|
||||
"attachments/present/original", "attachments/missing/original"));
|
||||
FakeStorage storage = new FakeStorage(List.of(
|
||||
object("attachments/present/original", NOW.minusSeconds(7200)),
|
||||
object("attachments/old-orphan/original", NOW.minusSeconds(7200)),
|
||||
object("attachments/fresh-orphan/original", NOW.minusSeconds(30))));
|
||||
AttachmentReconciler reconciler = new AttachmentReconciler(repository, storage,
|
||||
Duration.ofHours(1), false, Clock.fixed(NOW, ZoneOffset.UTC));
|
||||
|
||||
AttachmentReconciler.Report report = reconciler.reconcile();
|
||||
|
||||
assertEquals(List.of("attachments/old-orphan/original"), storage.deleted);
|
||||
assertEquals(1, report.orphanObjects());
|
||||
assertEquals(1, report.deletedObjects());
|
||||
assertEquals(1, report.missingObjects());
|
||||
}
|
||||
|
||||
@Test
|
||||
void dryRunNeverDeletesOldOrphan() {
|
||||
AttachmentRepository repository = mock(AttachmentRepository.class);
|
||||
when(repository.findAllObjectKeys()).thenReturn(Set.of());
|
||||
FakeStorage storage = new FakeStorage(List.of(
|
||||
object("attachments/old-orphan/original", NOW.minusSeconds(7200))));
|
||||
AttachmentReconciler reconciler = new AttachmentReconciler(repository, storage,
|
||||
Duration.ofHours(1), true, Clock.fixed(NOW, ZoneOffset.UTC));
|
||||
|
||||
AttachmentReconciler.Report report = reconciler.reconcile();
|
||||
|
||||
assertEquals(List.of(), storage.deleted);
|
||||
assertEquals(1, report.orphanObjects());
|
||||
assertEquals(0, report.deletedObjects());
|
||||
}
|
||||
|
||||
private static ObjectStorage.StoredObject object(String key, Instant modified) {
|
||||
return new ObjectStorage.StoredObject(key, modified, 100);
|
||||
}
|
||||
|
||||
private static final class FakeStorage implements ObjectStorage {
|
||||
private final List<StoredObject> objects;
|
||||
private final List<String> deleted = new ArrayList<>();
|
||||
|
||||
private FakeStorage(List<StoredObject> objects) {
|
||||
this.objects = objects;
|
||||
}
|
||||
|
||||
@Override public void put(String objectKey, byte[] bytes, String contentType) { }
|
||||
@Override public byte[] get(String objectKey) { return new byte[0]; }
|
||||
@Override public void delete(String objectKey) { deleted.add(objectKey); }
|
||||
@Override public List<StoredObject> list(String prefix) { return objects; }
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,7 @@ import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
import java.io.IOException;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
@@ -51,7 +52,7 @@ class AttachmentServiceTest {
|
||||
new Chat(chatId, "test", List.of(ownerId), Instant.now()));
|
||||
when(repository.save(any())).thenAnswer(invocation -> invocation.getArgument(0));
|
||||
service = new AttachmentService(repository, chats, storage, transaction,
|
||||
10_485_760, 40_000_000);
|
||||
10_485_760, 40_000_000, 3, 10_000);
|
||||
png = Files.readAllBytes(Path.of("tests", "fixtures", "sample.png"));
|
||||
}
|
||||
|
||||
@@ -110,6 +111,39 @@ class AttachmentServiceTest {
|
||||
verify(repository, never()).save(any());
|
||||
}
|
||||
|
||||
@Test
|
||||
void interruptedRequestDoesNotCreateS3Object() {
|
||||
java.io.InputStream source = new java.io.InputStream() {
|
||||
private int position;
|
||||
|
||||
@Override
|
||||
public int read() throws IOException {
|
||||
if (position >= 16_384) {
|
||||
throw new IOException("client disconnected");
|
||||
}
|
||||
return png[position++];
|
||||
}
|
||||
};
|
||||
|
||||
ApiException error = assertThrows(ApiException.class, () -> service.upload(
|
||||
chatId, ownerId, "photo.png", "image/png", -1, source));
|
||||
|
||||
assertEquals(HttpStatus.BAD_REQUEST, error.status());
|
||||
assertTrue(storage.objects.isEmpty());
|
||||
verify(repository, never()).save(any());
|
||||
}
|
||||
|
||||
@Test
|
||||
void databaseFailureCompensatesAlreadyWrittenS3Object() {
|
||||
when(repository.save(any())).thenThrow(new IllegalStateException("database commit failed"));
|
||||
|
||||
assertThrows(IllegalStateException.class, () -> service.upload(
|
||||
chatId, ownerId, "photo.png", "image/png", png.length,
|
||||
new ByteArrayInputStream(png)));
|
||||
|
||||
assertTrue(storage.objects.isEmpty(), "compensating delete must remove the orphan");
|
||||
}
|
||||
|
||||
@Test
|
||||
void downloadsOnlyAfterChatAuthorization() {
|
||||
Attachment attachment = new Attachment(UUID.randomUUID(), chatId, ownerId, "photo.png",
|
||||
@@ -144,8 +178,17 @@ class AttachmentServiceTest {
|
||||
public void delete(String objectKey) {
|
||||
objects.remove(objectKey);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<StoredObject> list(String prefix) {
|
||||
return objects.entrySet().stream()
|
||||
.filter(entry -> entry.getKey().startsWith(prefix))
|
||||
.map(entry -> new StoredObject(entry.getKey(), Instant.now(), entry.getValue().length))
|
||||
.toList();
|
||||
}
|
||||
}
|
||||
|
||||
private static final class IOExceptionForTest extends RuntimeException {
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -12,3 +12,7 @@ The course-provided black-box suite remains in `tests/`; Spring tests remain in
|
||||
- `rolling_update.py` — sequential replacement under write load.
|
||||
- `node_failover.py` — hard loss of one worker in the disposable three-node
|
||||
Swarm lab, task relocation and confirmed-data verification.
|
||||
- `lab4_negative.py` — oversized/corrupt/MIME-spoofed uploads, unsafe filename,
|
||||
unknown attachment, foreign user and cross-chat binding; every user error is 4xx.
|
||||
- `concurrent_upload_memory.py` — 12 simultaneous ~6.5 MiB valid PNG uploads and
|
||||
aggregate API container memory sampling through `docker stats`.
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run 10+ simultaneous valid PNG uploads and sample aggregate API container RAM."""
|
||||
import argparse
|
||||
import binascii
|
||||
import concurrent.futures
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / 'tests'))
|
||||
from client import Client, Failure, check, run # noqa: E402
|
||||
|
||||
|
||||
def png_chunk(kind, payload):
|
||||
return (struct.pack('>I', len(payload)) + kind + payload
|
||||
+ struct.pack('>I', binascii.crc32(kind + payload) & 0xffffffff))
|
||||
|
||||
|
||||
def make_png(width=1800, height=1200):
|
||||
rng = random.Random(20261111)
|
||||
rows = bytearray()
|
||||
for _ in range(height):
|
||||
rows.append(0)
|
||||
rows.extend(rng.randbytes(width * 3))
|
||||
return (b'\x89PNG\r\n\x1a\n'
|
||||
+ png_chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
|
||||
+ png_chunk(b'IDAT', zlib.compress(bytes(rows), 1))
|
||||
+ png_chunk(b'IEND', b''))
|
||||
|
||||
|
||||
def memory_bytes(value):
|
||||
match = re.fullmatch(r'([0-9]+(?:\.[0-9]+)?)\s*([A-Za-z]+)', value.strip())
|
||||
if not match:
|
||||
raise ValueError('unrecognized Docker memory value: ' + value)
|
||||
number, unit = match.groups()
|
||||
factors = {'B': 1, 'kB': 1000, 'KiB': 1024, 'MB': 1000**2,
|
||||
'MiB': 1024**2, 'GB': 1000**3, 'GiB': 1024**3}
|
||||
return int(float(number) * factors[unit])
|
||||
|
||||
|
||||
def api_containers(env_file):
|
||||
result = subprocess.run(
|
||||
['docker', 'compose', '--env-file', env_file, 'ps', '-q', 'api'],
|
||||
cwd=ROOT, check=True, capture_output=True, text=True)
|
||||
ids = [line for line in result.stdout.splitlines() if line]
|
||||
check(len(ids) >= 2, 'memory experiment requires at least two API containers')
|
||||
return ids
|
||||
|
||||
|
||||
def sample_total_memory(container_ids):
|
||||
result = subprocess.run(
|
||||
['docker', 'stats', '--no-stream', '--format', '{{.MemUsage}}', *container_ids],
|
||||
check=True, capture_output=True, text=True)
|
||||
usages = []
|
||||
for line in result.stdout.splitlines():
|
||||
if line.strip():
|
||||
usages.append(memory_bytes(line.split('/', 1)[0].strip()))
|
||||
check(len(usages) == len(container_ids), 'docker stats did not return every API container')
|
||||
return sum(usages)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--base-url', default='http://localhost:8080')
|
||||
parser.add_argument('--compose-env-file', default='.env')
|
||||
parser.add_argument('--concurrency', type=int, default=12)
|
||||
parser.add_argument('--output', default='evidence/upload-memory.json')
|
||||
args = parser.parse_args()
|
||||
check(args.concurrency >= 10, 'experiment requires concurrency >= 10')
|
||||
|
||||
bootstrap = Client(args.base_url, timeout=30)
|
||||
alice, bob, _, chat = bootstrap.bootstrap()
|
||||
payload = make_png()
|
||||
check(len(payload) <= 10_485_760, 'generated PNG exceeds upload limit')
|
||||
containers = api_containers(args.compose_env_file)
|
||||
baseline = sample_total_memory(containers)
|
||||
peak = baseline
|
||||
samples = []
|
||||
stop = threading.Event()
|
||||
|
||||
def monitor():
|
||||
nonlocal peak
|
||||
while not stop.is_set():
|
||||
try:
|
||||
current = sample_total_memory(containers)
|
||||
samples.append(current)
|
||||
peak = max(peak, current)
|
||||
except (subprocess.SubprocessError, Failure, ValueError):
|
||||
pass
|
||||
stop.wait(0.1)
|
||||
|
||||
barrier = threading.Barrier(args.concurrency)
|
||||
|
||||
def upload(index):
|
||||
client = Client(args.base_url, timeout=30)
|
||||
barrier.wait(timeout=30)
|
||||
path = (f"/api/v1/chats/{chat['id']}/attachments"
|
||||
f"?filename=memory-{index}.png")
|
||||
return client.call('POST', path, 201, actor=alice, raw=payload,
|
||||
headers={'Content-Type': 'image/png'}, schema='Attachment')['json']['id']
|
||||
|
||||
monitor_thread = threading.Thread(target=monitor, daemon=True)
|
||||
monitor_thread.start()
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as pool:
|
||||
ids = list(pool.map(upload, range(args.concurrency)))
|
||||
finally:
|
||||
stop.set()
|
||||
monitor_thread.join(timeout=5)
|
||||
duration = time.monotonic() - started
|
||||
check(len(ids) == args.concurrency and len(set(ids)) == len(ids),
|
||||
'not every concurrent upload completed exactly once')
|
||||
|
||||
result = {
|
||||
'concurrency': args.concurrency,
|
||||
'successful_uploads': len(ids),
|
||||
'payload_bytes_each': len(payload),
|
||||
'duration_seconds': round(duration, 3),
|
||||
'api_replicas': len(containers),
|
||||
'memory_samples': len(samples),
|
||||
'baseline_api_memory_bytes': baseline,
|
||||
'peak_api_memory_bytes': peak,
|
||||
'peak_increase_bytes': max(0, peak - baseline),
|
||||
'configured_max_concurrent_uploads_per_replica': 3,
|
||||
'configured_max_upload_bytes': 10_485_760,
|
||||
'configured_max_image_pixels': 40_000_000,
|
||||
}
|
||||
output = ROOT / args.output
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(json.dumps(result, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
|
||||
print(json.dumps(result, ensure_ascii=False))
|
||||
print('PASS: concurrent upload memory experiment')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -39,6 +39,12 @@ def main():
|
||||
client.call('POST', '/api/v1/chats/' + other_chat['id'] + '/messages', 404,
|
||||
actor=alice, headers={'Idempotency-Key': uuid.uuid4().hex},
|
||||
body={'text': '', 'attachment_ids': [attachment['id']]})
|
||||
client.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 404,
|
||||
actor=alice, headers={'Idempotency-Key': uuid.uuid4().hex},
|
||||
body={'text': '', 'attachment_ids': [str(uuid.uuid4())]})
|
||||
client.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 400,
|
||||
actor=alice, headers={'Idempotency-Key': uuid.uuid4().hex},
|
||||
body={'text': '', 'attachment_ids': [attachment['id'], attachment['id']]})
|
||||
|
||||
client.call('POST', upload_path, 415, actor=alice, raw=png,
|
||||
headers={'Content-Type': 'image/jpeg'})
|
||||
@@ -52,7 +58,7 @@ def main():
|
||||
downloaded = client.call('GET', '/api/v1/attachments/' + attachment['id'] + '/content',
|
||||
actor=bob)
|
||||
check(downloaded['bytes'] == original, 'authorized download changed bytes')
|
||||
print('PASS: foreign chat/user, MIME spoof, corruption, byte limit and filename checks')
|
||||
print('PASS: unknown/foreign attachment, foreign chat/user, MIME spoof, corruption, byte limit and filename checks')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -6,6 +6,8 @@ events {
|
||||
}
|
||||
|
||||
http {
|
||||
client_max_body_size 11m;
|
||||
client_body_timeout 10s;
|
||||
access_log /var/log/nginx/access.log;
|
||||
error_log /var/log/nginx/error.log warn;
|
||||
resolver 127.0.0.11 valid=2s ipv6=off;
|
||||
@@ -53,6 +55,20 @@ http {
|
||||
proxy_next_upstream_tries 2;
|
||||
}
|
||||
|
||||
location ~ ^/api/v1/chats/[0-9a-fA-F-]+/attachments$ {
|
||||
proxy_pass http://api_backend;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_request_buffering on;
|
||||
proxy_connect_timeout 500ms;
|
||||
proxy_send_timeout 20s;
|
||||
proxy_read_timeout 20s;
|
||||
proxy_next_upstream off;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass http://api_backend;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
@@ -27,6 +27,12 @@ services:
|
||||
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?S3_SECRET_ACCESS_KEY is required}
|
||||
MAX_UPLOAD_BYTES: 10485760
|
||||
MAX_IMAGE_PIXELS: 40000000
|
||||
MAX_CONCURRENT_UPLOADS: 3
|
||||
UPLOAD_PERMIT_WAIT_MILLIS: 10000
|
||||
ATTACHMENT_CLEANUP_GRACE_SECONDS: 3600
|
||||
ATTACHMENT_CLEANUP_DRY_RUN: "false"
|
||||
ATTACHMENT_RECONCILE_INTERVAL_MILLIS: 600000
|
||||
ATTACHMENT_RECONCILE_INITIAL_DELAY_MILLIS: 60000
|
||||
FLYWAY_ENABLED: "false"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "--fail", "--silent", "http://localhost:8080/health/ready"]
|
||||
|
||||
Reference in New Issue
Block a user