fit: complete lab4, fix P1.1-P1.3, add Track 1 presigned

This commit is contained in:
2026-11-06 04:23:59 +03:00
parent 3223dd469a
commit 7741b66fd2
16 changed files with 577 additions and 219 deletions
+29 -11
View File
@@ -1,7 +1,8 @@
# Отчёт — лабораторная 4
- Автор, группа: Никитин Андрей Александрович, группа БВТ2301
- Целевая оценка: 5 (трек 1 — прямой/presigned upload)
- Целевая оценка: 5
- Трек 5: Track 1 (Presigned upload)
- Commit/PR: ветка `solution/nikitin-bvt2301` → MR в `main` (шаблон `.github/pull_request_template.md`); remote `gitea.privcloud.tech/AndrewNikkk/mtusi2026lab4`
- Источники и помощь: контракт и публичные тесты курса; PostgreSQL 16, Valkey, Nginx, MinIO, libpqxx 7.10.0, libsodium, hiredis, nlohmann/json, minio-cpp v0.3.0 (официальный SigV4), stb_image. Помощь ИИ: Cursor.
@@ -60,7 +61,7 @@ docker compose --env-file .env stop
# docker compose --env-file .env down -v
```
Хосты в секретах — имена сервисов Compose (`db`, `cache`, `minio`), не loopback. Порты MinIO **не** публикуются по умолчанию; снаружи только LB `:8080`. Для host-side PUT к presigned URL — опционально `compose.presign.yaml` + `S3_PRESIGN_ENDPOINT=http://127.0.0.1:9000` (loopback). Иначе PUT идёт из контейнера на сети compose (`http://minio:9000/...`). Миграции 001–006 применяет API при старте под `pg_advisory_lock`.
Хосты в секретах — имена сервисов Compose (`db`, `cache`, `minio`), не loopback. Порты MinIO **не** публикуются по умолчанию; снаружи только LB `:8080`. Для host-side PUT к presigned URL — опционально `compose.presign.yaml` + `S3_PRESIGN_ENDPOINT=http://127.0.0.1:9000` (loopback). Иначе PUT идёт из контейнера на сети compose (`http://minio:9000/...`). Миграции 001–007 применяет API при старте под `pg_advisory_lock`.
Соответствие `.env.example` → compose: `S3_ENDPOINT`/`S3_BUCKET`/`S3_REGION`, `S3_*_FILE` → docker secrets, `MAX_UPLOAD_BYTES`/`MAX_IMAGE_PIXELS`, `DIRECT_UPLOAD_TTL_SECONDS`, `DRAIN_SECONDS`, `API_REPLICAS`, `SESSION_STORE=redis`. Root MinIO только у `minio` / init-job'ов; API видит лишь limited `*_FILE`.
@@ -68,16 +69,22 @@ docker compose --env-file .env stop
```
Proxy: Клиент → lb → api (buffer body) → MinIO PutObject → PG attachments
Direct: Клиент → lb → api (JSON init) → presigned URL
Клиент ─────────────── PUT ──────→ MinIO
Клиент → lb → api complete (Stat+Get+sha256+image) → PG attachments
Direct: Клиент → lb → api init → presigned PUT URL (staging/{chat}/{upload})
Клиент ─────────────── PUT ──────→ MinIO staging
Клиент → lb → api complete (HEAD+Get+sha256+image+CopyObject)
staging → attachments/{chat}/{attachment}
→ PG attachment + completed session
```
**Источник истины:** метаданные вложений и связь message↔attachment — PostgreSQL; байты — объект в MinIO с ключом `attachments/{chat_id}/{uuid}` (filename только для отображения). Pending direct sessions — таблица `direct_uploads` (миграция 006).
**Источник истины:** метаданные вложений и связь message↔attachment — PostgreSQL; оригинал — объект MinIO `attachments/{chat_id}/{attachment_id}` (filename только для отображения). Presigned PUT пишет в отдельный `staging/{chat_id}/{upload_id}`. Direct sessions хранят оба ключа в `direct_uploads` (миграции 006–007).
**Порядок записи (proxy, компенсация):** S3 `PutObject` → `INSERT attachments` → COMMIT. Если INSERT/COMMIT упал — best-effort `DeleteObject`, ответ 500. Строка в PG без объекта недопустима. Orphan'ы: `tools/gc_orphans.sh` (учитывает `direct_uploads.status=pending`).
**Трек 1 (direct):** `POST /api/v1/chats/{id}/direct-uploads` → короткая SigV4 PUT URL (`minio-cpp GetPresignedObjectUrl` / PresignV4, TTL=`DIRECT_UPLOAD_TTL_SECONDS`; в ответе рекомендованный `Content-Type` для метаданных объекта) → клиентский PUT → `POST .../complete` (владелец, Stat size, sha256, magic/image) → `attachments` ready с тем же id. `DELETE .../direct-uploads/{id}` — abort + DeleteObject. Просроченные pending: `tools/gc_direct_uploads.sh`. Обязательный proxy `POST .../attachments` **сохранён**.
**Track 1 (presigned):** `POST /api/v1/chats/{id}/direct-uploads` создаёт `pending` сессию и выдаёт SigV4 PUT URL (`minio-cpp GetPresignedObjectUrl`, TTL по `DIRECT_UPLOAD_TTL_SECONDS`, 30–3600 секунд; default 300). URL сам по себе повторно используем до истечения TTL, поэтому он не считается одноразовым токеном. Одноразово публикуется именно сессия complete: её статус переходит `pending → completed`.
`POST /api/v1/direct-uploads/{id}/complete` в одной PostgreSQL-транзакции делает `SELECT ... FOR UPDATE`, проверяет существование/размер/hash staging-объекта и изображение, затем выполняет серверный `CopyObject` в final prefix, вставляет `attachments.status=ready`, помечает сессию `completed` и коммитит. SHA-256 берётся из `expected_sha256`, заявленного при init. Повторный complete возвращает сохранённый Attachment без повторного S3 copy; конкурентные complete одного id сериализуются блокировкой строки. Удаление staging после публикации best-effort.
Повторный PUT по ещё действующей старой ссылке может создать staging-orphan, но не перезаписывает final object. `tools/gc_orphans.sh` сканирует `attachments/` и `staging/`, сохраняет pending staging keys и удаляет неизвестные объекты только после `GRACE_SECONDS`. Abort удаляет staging best-effort; стандартный proxy `POST /api/v1/chats/{id}/attachments` **сохранён**.
**ACL:** метаданные и `/content` только участнику чата; чужой → 404; без Bearer → 401. Unsigned GET на `http://minio:9000/$BUCKET/$KEY` → 403.
@@ -108,6 +115,9 @@ Direct: Клиент → lb → api (JSON init) → presigned URL
| Graceful drain | `sh student-tests/test_graceful_drain.sh` | SIGTERM → ready 503/stop; стенд восстановлен | вывод теста |
| Идемпотентный bootstrap bucket/user | `minio-init`, `minio-user-init` | повторный up без ошибки; `mc mb --ignore-existing`; anonymous none | логи init |
| Трек 1: init→PUT→complete; proxy жив | `sh student-tests/run_grade5_track1.sh` | PASS (ok + negatives + gc dry-run) | `student-tests/` |
| Track 1: concurrent complete | `sh student-tests/test_presigned_complete_concurrent.sh` | 4 complete: ровно 1×201, остальные 200/409, ни одного 500; metadata и `/content` SHA-256 совпали с исходником | `student-tests/test_presigned_complete_concurrent.sh` |
| Track 1: replay safety | `sh student-tests/test_direct_upload_ok.sh` | upload → complete → повторный PUT 20 B по старой ссылке → download SHA-256 исходного PNG; final не изменён | `student-tests/test_direct_upload_ok.sh` |
| Download при отсутствующем S3-объекте | `sh student-tests/test_download_missing_object.sh` | после `mc rm` content отвечает 404/5xx, не 200; объект восстанавливается тестом | `student-tests/test_download_missing_object.sh` |
| Трек 1: RAM/трафик ≥2 размеров | `sh tools/exp_direct_vs_proxy.sh` | transfer: direct ≪ proxy по RSS и api_body | `artifacts/track1/results.tsv` |
Перезапускаемые процессы при persistence hook: **db, cache, minio, api (все реплики), lb**. Volumes `pgdata`, `valkeydata`, `miniodata` **не** удаляются. Failover hook останавливает **только** одну API-реплику по `X-Instance-Id` ≡ `Config.Hostname`.
@@ -157,9 +167,15 @@ Direct: Клиент → lb → api (JSON init) → presigned URL
**Подтверждено измерением (grade 4):** 10 параллельных ~5.6 MiB proxy-загрузок → 201; delta RSS < naive; private bucket; persistence/failover.
## Трек 1 — прямой upload (оценка 5)
## Track 1: Presigned upload
**Гипотеза:** при одинаковом размере файла пиковый RSS API и объём трафика через LB/API на этапе передачи байт ниже для direct (клиент→MinIO), чем для proxy (клиент→API→MinIO); complete добавляет короткий GetObject на API.
**Архитектура и семантика:** presigned PUT направлен только на `staging/{chat_id}/{upload_id}`. После HEAD/stat, проверки `expected_size`, SHA-256 из init и image validation API копирует staging в `attachments/{chat_id}/{attachment_id}`. В `attachments.s3_key` хранится только final key. TTL ссылки задаётся `DIRECT_UPLOAD_TTL_SECONDS` (30–3600 секунд, default 300); пока TTL не истёк, URL можно повторно вызвать, но replay меняет только staging. Complete логически single-use: `pending → completed`; `SELECT FOR UPDATE` делает повторы и конкурентные запросы идемпотентными, они возвращают существующий Attachment без S3-публикации.
Если старый PUT выполняется после complete, он оставляет staging-orphan; final-объект остаётся прежним. `tools/gc_orphans.sh` очищает orphan-объекты обоих prefix по `GRACE_SECONDS`, не удаляя staging активных `pending` сессий. Staging cleanup после успешной публикации и abort — best-effort.
**Ограничения:** стандартная загрузка ограничена 10 MiB. Multipart-upload для файлов больше 100 MiB не реализован; presigned GET для скачивания тоже не реализован. Download остаётся авторизованным через API. Контракт `contracts/openapi.json` не менялся: Track 1 — расширение, обязательный proxy upload сохранён.
**Proxy vs presigned:** proxy передаёт тело через LB/API и буферизует его перед MinIO, поэтому upload потребляет память API и дополнительный сетевой hop. При presigned передаче байты идут клиент→MinIO напрямую, API получает только init/complete и не буферизует тело PUT. Complete добавляет HEAD/stat, чтение для SHA/image validation и server-side copy, поэтому его latency выше собственно PUT и измеряется отдельно.
**API (расширение, openapi не менялся):**
@@ -171,9 +187,9 @@ Direct: Клиент → lb → api (JSON init) → presigned URL
| DELETE | `/api/v1/direct-uploads/{id}` | abort + DeleteObject |
| POST | `/api/v1/chats/{id}/attachments` | обязательный proxy — без изменений |
**Негативные:** complete без PUT → 400; checksum mismatch → 400 + abort; чужой complete → 404; abort → 204, повторный complete → 409; expired → 410 (`gc_direct_uploads` / expire_stale).
**Состояния и ошибки:** complete без объекта → `409 not_ready`; размер или checksum mismatch → `409 hash_mismatch`; чужой upload → 404; abort → 204; повторный complete после успеха → 200 с тем же Attachment; aborted/expired session не публикуется; просроченный pending → 410.
**Проверка:** `sh student-tests/run_grade5_track1.sh` → PASS. Эксперимент: `sh tools/exp_direct_vs_proxy.sh` → `artifacts/track1/results.tsv`.
**Проверка:** `sh student-tests/run_grade5_track1.sh` → PASS; `sh student-tests/run_grade3.sh` → PASS; `sh student-tests/run_grade4.sh` → PASS. Четыре одновременных complete дали ровно один 201, остальные ответы были 200/409, 500 не было; metadata и content SHA-256 совпали. Replay-проверка повторно PUT-ит 20-byte текст по старому URL и подтверждает SHA-256 исходного PNG при download. Эксперимент proxy vs direct: `sh tools/exp_direct_vs_proxy.sh` → `artifacts/track1/results.tsv`.
Методика: closed-loop; concurrency 3 (1 MiB) / 4 (4 MiB); RSS = сумма `VmRSS` процессов `messenger` по репликам api; фаза `transfer` — только передача байт (для direct — параллельные PUT на MinIO); `complete` — отдельно (GetObject+sha256+image на API). `api_body_bytes_approx` — оценка объёма тел через API (proxy = N×size; direct transfer = 0).
@@ -189,3 +205,5 @@ Direct: Клиент → lb → api (JSON init) → presigned URL
**Вывод:** на этапе передачи байт direct почти не двигает RSS API и не гонит payload через LB/API; proxy платит и трафиком, и пиком RSS. Complete у direct кратковременно читает объект (учебный verify) — это отдельная фаза, не отменяет выигрыш на upload path.
**Ограничение:** без `compose.presign.yaml` PUT выполняется с контейнера на сети compose (порты MinIO на хост не открыты). Подпись — `minio-cpp` PresignV4 (не самописный SigV4).
Track 5 (reconciler) не выполнялся; Track 1 (presigned) выполнен, см. раздел выше.
+1 -1
View File
@@ -50,7 +50,7 @@ Basic — это кодирование, не шифрование. HTTP lab1–
- Файл заранее привязан к чату. В POST сообщения добавляется `attachment_ids` (0–10 уникальных ID, по умолчанию []). Можно отправить `text=""` с хотя бы одним вложением. Все вложения должны принадлежать этому чату; чужое/неизвестное — 404. Любой участник может ссылаться на доступное вложение чата. Failed-вложение прикреплять нельзя (409); queued/processing с lab5 можно.
- `GET /attachments/{id}` и `/content` доступны только участникам чата. Download возвращает байты с истинным Content-Type, через авторизованный API, **без redirect** в обязательном контракте. Приватный bucket сам по себе не заменяет эту проверку. `variant` по умолчанию original. Для queued/processing content — 409; для failed — 409. Отсутствующий вариант ready-объекта — 404.
- В ready-метаданных `error_code=null`. В failed — краткий машинный код (например `invalid_image`, `processing_failed`, `retry_exhausted`), без stack trace. `size_bytes`, `sha256`, `content_type` всегда описывают оригинал; у вариантов отдельные поля. Публичный API не раскрывает ключи бакета, внутренние адреса и credentials.
- Прямой upload/download по временной подписи — факультативное расширение. Его дополняют ограничение срока/размера, уникальный ключ и подтверждение реально загруженного объекта; он не заменяет стандартный endpoint. Особенности подписанных URL: [S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html).
- Прямой upload/download по временной подписи — факультативное расширение. Его дополняют ограничение срока/размера, уникальный ключ и подтверждение реально загруженного объекта; он не заменяет стандартный endpoint. Особенности подписанных URL: [S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html). В этом решении checksum на complete — SHA-256, заявленный при init (`expected_sha256`); тело complete пустое, отдельный hash в complete не принимается. Presigned PUT пишет только в `staging/{chat_id}/{upload_id}`. Complete копирует объект в `attachments/{chat_id}/{id}` и именно этот ключ сохраняет в `attachments.s3_key`. Повторный PUT той же ссылкой после complete не затирает оригинал. Схема расширения: `contracts/extensions.openapi.json`.
## Фоновая обработка (с lab5)
+10 -10
View File
@@ -5,18 +5,18 @@
"201": 10
},
"success": 10,
"wall_ms": 160918.8,
"wall_ms": 167347.8,
"latency_ms": {
"p50": 398.4,
"p95": 669.1,
"p99": 669.1,
"max": 669.1
"p50": 1125.9,
"p95": 1630.0,
"p99": 1630.0,
"max": 1630.0
},
"approx_success_rps": 0.062,
"api_rss_baseline_mib": 51.96,
"api_rss_peak_mib": 68.4,
"api_rss_delta_mib": 16.44,
"api_cpu_peak_percent": 4.0,
"approx_success_rps": 0.06,
"api_rss_baseline_mib": 0.0,
"api_rss_peak_mib": 0.0,
"api_rss_delta_mib": 0.0,
"api_cpu_peak_percent": 0.0,
"naive_full_buffer_mib": 56.09,
"max_concurrent_uploads": 3,
"note": "RSS capped by MAX_CONCURRENT_UPLOADS (default 3) per API process; slot taken before buffering body >256KiB; malloc_trim after each upload. rss_peak \u2264 baseline + MAX_CONCURRENT_UPLOADS \u00d7 (body + decoded_raster). Closed-loop client; RPS is throughput of confirmed 201s."
+12
View File
@@ -0,0 +1,12 @@
-- Track 1: presigned PUT lands in staging/; complete copies to attachments/.
-- Existing rows keep the old key as staging so in-flight sessions still resolve.
ALTER TABLE direct_uploads ADD COLUMN staging_key text;
UPDATE direct_uploads
SET staging_key = s3_key
WHERE staging_key IS NULL;
ALTER TABLE direct_uploads ALTER COLUMN staging_key SET NOT NULL;
ALTER TABLE direct_uploads
ADD CONSTRAINT direct_uploads_staging_key_key UNIQUE (staging_key);
+123 -8
View File
@@ -36,7 +36,9 @@ DirectUpload row_to(const pqxx::row& r) {
d.expected_size = r["expected_size"].as<int>();
d.expected_sha256 = r["expected_sha256"].as<std::string>();
d.s3_key = r["s3_key"].as<std::string>();
d.staging_key = r["staging_key"].as<std::string>();
d.status = r["status"].as<std::string>();
d.overdue = r["overdue"].as<bool>();
if (r["attachment_id"].is_null()) {
d.attachment_id = std::nullopt;
} else {
@@ -50,11 +52,40 @@ DirectUpload row_to(const pqxx::row& r) {
constexpr const char* kCols =
"id, chat_id, owner_id, filename, content_type, expected_size, "
"expected_sha256, s3_key, status, attachment_id, "
"expected_sha256, s3_key, staging_key, status, attachment_id, "
"(status = 'pending' AND expires_at < now()) AS overdue, "
"EXTRACT(EPOCH FROM expires_at)::bigint AS expires_at_epoch, "
"EXTRACT(EPOCH FROM created_at)::bigint AS created_at_epoch, "
"EXTRACT(EPOCH FROM updated_at)::bigint AS updated_at_epoch";
constexpr const char* kAttachmentColumns =
"id, chat_id, owner_id, filename, content_type, size_bytes, sha256, "
"status, s3_key, error_code, "
"EXTRACT(EPOCH FROM created_at)::bigint AS created_at_epoch, "
"EXTRACT(EPOCH FROM updated_at)::bigint AS updated_at_epoch";
Attachment attachment_from_row(const pqxx::row& r) {
Attachment a;
a.id = r["id"].as<std::string>();
a.chat_id = r["chat_id"].as<std::string>();
a.owner_id = r["owner_id"].as<std::string>();
a.filename = r["filename"].as<std::string>();
a.content_type = r["content_type"].as<std::string>();
a.size_bytes = r["size_bytes"].as<int>();
a.sha256 = r["sha256"].as<std::string>();
a.status = r["status"].as<std::string>();
a.s3_key = r["s3_key"].as<std::string>();
if (r["error_code"].is_null()) {
a.error_code = std::nullopt;
} else {
a.error_code = r["error_code"].as<std::string>();
}
a.created_at = epoch_to_rfc3339(r["created_at_epoch"].as<long long>());
a.updated_at = epoch_to_rfc3339(r["updated_at_epoch"].as<long long>());
a.variants.clear();
return a;
}
} // namespace
DirectUpload DirectUploadRepo::insert(const DirectUpload& in) {
@@ -65,17 +96,18 @@ DirectUpload DirectUploadRepo::insert(const DirectUpload& in) {
auto r = tx.exec(
"INSERT INTO direct_uploads ("
" id, chat_id, owner_id, filename, content_type, expected_size, "
" expected_sha256, s3_key, status, attachment_id, "
" expected_sha256, s3_key, staging_key, status, attachment_id, "
" expires_at, created_at, updated_at"
") VALUES ("
" $1::uuid, $2::uuid, $3::uuid, $4, $5, $6, $7, $8, $9, NULL, "
" to_timestamp($10::bigint), to_timestamp($11::bigint), "
" to_timestamp($12::bigint)"
" $1::uuid, $2::uuid, $3::uuid, $4, $5, $6, $7, $8, $9, $10, NULL, "
" to_timestamp($11::bigint), to_timestamp($12::bigint), "
" to_timestamp($13::bigint)"
") RETURNING " +
std::string(kCols),
pqxx::params{in.id, in.chat_id, in.owner_id, in.filename,
in.content_type, in.expected_size, in.expected_sha256,
in.s3_key, in.status, in.expires_at_epoch, ts, ts});
in.s3_key, in.staging_key, in.status, in.expires_at_epoch,
ts, ts});
tx.commit();
return row_to(r[0]);
}
@@ -157,11 +189,94 @@ std::vector<std::string> DirectUploadRepo::pending_s3_keys() {
auto lease = pool_.acquire();
pqxx::work tx(*lease);
auto r = tx.exec(
"SELECT s3_key FROM direct_uploads WHERE status = 'pending'");
"SELECT staging_key FROM direct_uploads WHERE status = 'pending'");
std::vector<std::string> out;
out.reserve(r.size());
for (const auto& row : r) {
out.push_back(row["s3_key"].as<std::string>());
out.push_back(row["staging_key"].as<std::string>());
}
return out;
}
void DirectUploadRepo::with_lock(const std::function<void(UploadLock&)>& fn) {
auto lease = pool_.acquire();
pqxx::work tx(*lease);
UploadLock lock(tx);
fn(lock);
}
std::optional<DirectUpload> UploadLock::lock(const std::string& id) {
auto r = tx_.exec(
"SELECT " + std::string(kCols) +
" FROM direct_uploads WHERE id = $1::uuid FOR UPDATE",
pqxx::params{id});
if (r.empty()) {
return std::nullopt;
}
return row_to(r[0]);
}
std::optional<Attachment> UploadLock::find_attachment(const std::string& id) {
auto r = tx_.exec(
"SELECT " + std::string(kAttachmentColumns) +
" FROM attachments WHERE id = $1::uuid",
pqxx::params{id});
if (r.empty()) {
return std::nullopt;
}
return attachment_from_row(r[0]);
}
std::optional<Attachment> UploadLock::insert_ready(const Attachment& in,
bool& inserted) {
inserted = false;
const std::string ts = now_epoch_str();
auto r = tx_.exec(
"INSERT INTO attachments ("
" id, chat_id, owner_id, filename, content_type, size_bytes, sha256, "
" status, s3_key, error_code, created_at, updated_at"
") VALUES ("
" $1::uuid, $2::uuid, $3::uuid, $4, $5, $6, $7, $8, $9, NULL, "
" to_timestamp($10::bigint), to_timestamp($11::bigint)"
") ON CONFLICT (id) DO NOTHING RETURNING " +
std::string(kAttachmentColumns),
pqxx::params{in.id, in.chat_id, in.owner_id, in.filename,
in.content_type, in.size_bytes, in.sha256, in.status,
in.s3_key, ts, ts});
if (r.empty()) {
return std::nullopt;
}
inserted = true;
return attachment_from_row(r[0]);
}
void UploadLock::mark_completed(const std::string& id,
const std::string& attachment_id) {
const std::string ts = now_epoch_str();
auto r = tx_.exec(
"UPDATE direct_uploads SET status = 'completed', "
" attachment_id = $2::uuid, "
" updated_at = to_timestamp($3::bigint) "
"WHERE id = $1::uuid AND status = 'pending'",
pqxx::params{id, attachment_id, ts});
if (r.affected_rows() != 1) {
throw std::runtime_error("direct upload complete lost the pending row");
}
}
void UploadLock::mark_expired(const std::string& id) {
const std::string ts = now_epoch_str();
tx_.exec(
"UPDATE direct_uploads SET status = 'expired', "
" updated_at = to_timestamp($2::bigint) "
"WHERE id = $1::uuid AND status = 'pending'",
pqxx::params{id, ts});
}
void UploadLock::commit() {
if (committed_) {
return;
}
tx_.commit();
committed_ = true;
}
+31 -2
View File
@@ -3,10 +3,13 @@
#include "db.hpp"
#include "models.hpp"
#include <functional>
#include <optional>
#include <string>
#include <vector>
class UploadLock;
class DirectUploadRepo {
public:
explicit DirectUploadRepo(PgPool& pool) : pool_(pool) {}
@@ -22,12 +25,38 @@ public:
std::optional<DirectUpload> mark_aborted(const std::string& id);
// pending → expired where expires_at < now; returns expired rows (s3_key).
// pending → expired where expires_at < now; returns expired rows.
std::vector<DirectUpload> expire_stale();
// Active pending keys (for orphan GC exclusion).
// Staging keys of still-pending sessions (orphan GC must keep these).
std::vector<std::string> pending_s3_keys();
// One PG transaction. Callback may call S3 while the row lock is held.
// Commit explicitly via UploadLock::commit(); otherwise the work rolls back.
void with_lock(const std::function<void(UploadLock&)>& fn);
private:
PgPool& pool_;
};
// Row lock for a single complete(). Not safe to use after with_lock returns.
class UploadLock {
public:
std::optional<DirectUpload> lock(const std::string& id);
std::optional<Attachment> find_attachment(const std::string& id);
// ON CONFLICT (id) DO NOTHING. inserted=false and nullopt → row already exists.
std::optional<Attachment> insert_ready(const Attachment& in, bool& inserted);
void mark_completed(const std::string& id, const std::string& attachment_id);
void mark_expired(const std::string& id);
void commit();
private:
friend class DirectUploadRepo;
explicit UploadLock(pqxx::work& tx) : tx_(tx) {}
pqxx::work& tx_;
bool committed_ = false;
};
+175 -158
View File
@@ -1686,6 +1686,7 @@ static void handle_client(int client,
send_all(client, error_response(503, "unavailable",
"service unavailable", request_id));
} else {
bool response_headers_sent = false;
try {
const auto att = attachments.find_by_id(attachment_id);
if (!att) {
@@ -1705,36 +1706,51 @@ static void handle_client(int client,
409, "conflict",
"attachment not ready", request_id));
} else {
std::string head =
"HTTP/1.1 200 OK\r\n"
"Content-Type: " + att->content_type + "\r\n"
"X-Request-Id: " + request_id + "\r\n"
"X-Instance-Id: " + g_instance_id + "\r\n"
"Content-Length: " +
std::to_string(att->size_bytes) + "\r\n"
"Connection: close\r\n"
"\r\n";
send_all(client, head);
s3->get_object(att->s3_key, [&](const char* data,
std::size_t n) {
size_t sent = 0;
while (sent < n) {
const ssize_t w = ::write(
client, data + sent, n - sent);
if (w <= 0) {
throw std::runtime_error(
"content write failed");
std::int64_t object_size = 0;
std::string etag;
if (!s3->stat_object(att->s3_key, object_size, etag)) {
send_all(client, error_response(
404, "not_found", "not found", request_id));
} else if (object_size != att->size_bytes) {
send_all(client, error_response(
500, "internal_error", "internal error",
request_id));
} else {
std::string head =
"HTTP/1.1 200 OK\r\n"
"Content-Type: " + att->content_type + "\r\n"
"X-Request-Id: " + request_id + "\r\n"
"X-Instance-Id: " + g_instance_id + "\r\n"
"Content-Length: " +
std::to_string(att->size_bytes) + "\r\n"
"Connection: close\r\n"
"\r\n";
response_headers_sent = true;
send_all(client, head);
s3->get_object(att->s3_key, [&](const char* data,
std::size_t n) {
size_t sent = 0;
while (sent < n) {
const ssize_t w = ::write(
client, data + sent, n - sent);
if (w <= 0) {
throw std::runtime_error(
"content write failed");
}
sent += static_cast<size_t>(w);
}
sent += static_cast<size_t>(w);
}
});
});
}
}
}
} catch (const std::exception& e) {
// If headers were not sent yet, return JSON error; otherwise
// the connection is already mid-body — just close.
send_all(client, failure_response(e, "download_attachment",
request_id));
if (response_headers_sent) {
std::clog << "download_attachment stream failed: "
<< e.what() << "\n";
} else {
send_all(client, failure_response(e, "download_attachment",
request_id));
}
}
}
}
@@ -1845,6 +1861,7 @@ static void handle_client(int client,
d.expected_size = static_cast<int>(*size_b);
d.expected_sha256 = *sha;
d.s3_key = "attachments/" + chat_id + "/" + d.id;
d.staging_key = "staging/" + chat_id + "/" + d.id;
d.status = "pending";
const auto now = std::chrono::system_clock::now();
const auto secs =
@@ -1854,7 +1871,7 @@ static void handle_client(int client,
d.expires_at_epoch = secs + ttl;
const DirectUpload saved = direct_uploads.insert(d);
const std::string url = s3->presign_put(
saved.s3_key, static_cast<unsigned>(ttl),
saved.staging_key, static_cast<unsigned>(ttl),
saved.content_type);
nlohmann::json out;
out["id"] = saved.id;
@@ -1890,137 +1907,137 @@ static void handle_client(int client,
"service unavailable", request_id));
} else {
try {
auto sess = direct_uploads.find_by_id(direct_upload_id);
if (!sess || sess->owner_id != user_id) {
send_all(client, error_response(404, "not_found",
"not found", request_id));
} else if (sess->status != "pending") {
send_all(client, error_response(
409, "conflict",
"upload session not pending", request_id));
} else {
// Flip overdue pending → expired, then re-read.
direct_uploads.expire_stale();
sess = direct_uploads.find_by_id(direct_upload_id);
if (!sess || sess->status != "pending") {
send_all(client, error_response(
410, "gone", "upload session expired", request_id));
} else {
std::int64_t obj_size = 0;
std::string etag;
if (!s3->stat_object(sess->s3_key, obj_size, etag)) {
send_all(client, error_response(
400, "invalid_request",
"object not uploaded", request_id));
} else if (obj_size != sess->expected_size) {
try {
s3->delete_object(sess->s3_key);
} catch (...) {
}
direct_uploads.mark_aborted(sess->id);
send_all(client, error_response(
400, "invalid_request",
"size mismatch", request_id));
} else {
std::vector<std::uint8_t> buf;
buf.reserve(static_cast<std::size_t>(obj_size));
s3->get_object(
sess->s3_key,
[&](const char* data, std::size_t n) {
buf.insert(buf.end(),
reinterpret_cast<const std::uint8_t*>(
data),
reinterpret_cast<const std::uint8_t*>(
data) +
n);
});
if (static_cast<std::int64_t>(buf.size()) !=
sess->expected_size) {
send_all(client, error_response(
400, "invalid_request",
"size mismatch", request_id));
} else {
const std::string got_sha = sha256_hex_bytes(
buf.data(), buf.size());
if (got_sha != sess->expected_sha256) {
try {
s3->delete_object(sess->s3_key);
} catch (...) {
}
direct_uploads.mark_aborted(sess->id);
send_all(client, error_response(
400, "invalid_request",
"checksum mismatch", request_id));
} else {
ImageLimits lim;
if (const char* env =
std::getenv("MAX_UPLOAD_BYTES")) {
try {
lim.max_bytes = std::stoll(env);
} catch (...) {
}
}
if (const char* env =
std::getenv("MAX_IMAGE_PIXELS")) {
try {
lim.max_pixels = std::stoll(env);
} catch (...) {
}
}
ImageInfo info;
auto rej = validate_image(
buf.data(), buf.size(), lim, info);
if (rej ||
info.content_type != sess->content_type) {
try {
s3->delete_object(sess->s3_key);
} catch (...) {
}
direct_uploads.mark_aborted(sess->id);
send_all(client, error_response(
415, "unsupported_media_type",
"unsupported media type",
request_id));
} else {
Attachment a;
a.id = sess->id; // stable id = session id
a.chat_id = sess->chat_id;
a.owner_id = sess->owner_id;
a.filename = sess->filename;
a.content_type = sess->content_type;
a.size_bytes = sess->expected_size;
a.sha256 = sess->expected_sha256;
a.status = "ready";
a.s3_key = sess->s3_key;
bool inserted = false;
try {
const Attachment saved =
attachments.insert(a);
inserted = true;
direct_uploads.mark_completed(
sess->id, saved.id);
send_all(client, http_response(
201, "Created", request_id,
attachment_json(saved)));
} catch (const std::exception& e) {
if (!inserted) {
try {
s3->delete_object(a.s3_key);
} catch (...) {
}
direct_uploads.mark_aborted(
sess->id);
}
send_all(client, failure_response(
e, "direct_upload_complete",
request_id));
}
}
}
}
std::string response;
direct_uploads.with_lock([&](UploadLock& upload) {
const auto sess = upload.lock(direct_upload_id);
if (!sess || sess->owner_id != user_id) {
response = error_response(404, "not_found", "not found",
request_id);
return;
}
if (sess->status == "completed") {
const auto saved = upload.find_attachment(
sess->attachment_id.value_or(sess->id));
if (!saved) {
throw std::runtime_error(
"completed upload has no attachment");
}
upload.commit();
response = http_response(200, "OK", request_id,
attachment_json(*saved));
return;
}
if (sess->status != "pending") {
response = error_response(
409, "conflict", "upload session not pending",
request_id);
return;
}
if (sess->overdue) {
upload.mark_expired(sess->id);
upload.commit();
response = error_response(410, "gone",
"upload session expired",
request_id);
return;
}
std::int64_t obj_size = 0;
std::string etag;
if (!s3->stat_object(sess->staging_key, obj_size, etag)) {
upload.commit();
response = error_response(409, "not_ready",
"upload object not ready",
request_id);
return;
}
if (obj_size != sess->expected_size) {
upload.commit();
response = error_response(409, "hash_mismatch",
"uploaded object mismatch",
request_id);
return;
}
std::vector<std::uint8_t> buf;
buf.reserve(static_cast<std::size_t>(obj_size));
s3->get_object(
sess->staging_key,
[&](const char* data, std::size_t n) {
buf.insert(buf.end(),
reinterpret_cast<const std::uint8_t*>(data),
reinterpret_cast<const std::uint8_t*>(data) + n);
});
if (static_cast<std::int64_t>(buf.size()) !=
sess->expected_size ||
sha256_hex_bytes(buf.data(), buf.size()) !=
sess->expected_sha256) {
upload.commit();
response = error_response(409, "hash_mismatch",
"uploaded object mismatch",
request_id);
return;
}
ImageLimits lim;
if (const char* env = std::getenv("MAX_UPLOAD_BYTES")) {
try {
lim.max_bytes = std::stoll(env);
} catch (...) {
}
}
}
if (const char* env = std::getenv("MAX_IMAGE_PIXELS")) {
try {
lim.max_pixels = std::stoll(env);
} catch (...) {
}
}
ImageInfo info;
const auto rejection =
validate_image(buf.data(), buf.size(), lim, info);
if (rejection || info.content_type != sess->content_type) {
upload.commit();
response = error_response(415, "unsupported_media_type",
"unsupported media type",
request_id);
return;
}
Attachment a;
a.id = sess->id;
a.chat_id = sess->chat_id;
a.owner_id = sess->owner_id;
a.filename = sess->filename;
a.content_type = sess->content_type;
a.size_bytes = sess->expected_size;
a.sha256 = sess->expected_sha256;
a.status = "ready";
a.s3_key = sess->s3_key;
s3->copy_object(sess->staging_key, a.s3_key);
bool inserted = false;
auto saved = upload.insert_ready(a, inserted);
if (!saved) {
saved = upload.find_attachment(a.id);
}
if (!saved) {
throw std::runtime_error(
"attachment insert conflict without existing row");
}
try {
s3->delete_object(sess->staging_key);
} catch (const std::exception& e) {
std::clog << "direct upload staging cleanup failed: "
<< e.what() << "\n";
}
upload.mark_completed(sess->id, saved->id);
upload.commit();
response = http_response(inserted ? 201 : 200,
inserted ? "Created" : "OK",
request_id,
attachment_json(*saved));
});
send_all(client, response);
} catch (const std::exception& e) {
send_all(client, failure_response(e, "direct_upload_complete",
request_id));
@@ -2050,8 +2067,8 @@ static void handle_client(int client,
try {
std::int64_t sz = 0;
std::string et;
if (s3->stat_object(sess->s3_key, sz, et)) {
s3->delete_object(sess->s3_key);
if (s3->stat_object(sess->staging_key, sz, et)) {
s3->delete_object(sess->staging_key);
}
} catch (const std::exception& de) {
std::clog << "abort DeleteObject failed: "
+3 -1
View File
@@ -71,8 +71,10 @@ struct DirectUpload {
std::string content_type;
int expected_size = 0;
std::string expected_sha256;
std::string s3_key; // internal
std::string s3_key; // final key: attachments/{chat}/{id}
std::string staging_key; // presigned PUT target: staging/{chat}/{id}
std::string status; // pending|completed|aborted|expired
bool overdue = false; // pending && expires_at < now(); not a column
std::optional<std::string> attachment_id;
std::string expires_at; // RFC3339 for JSON
long long expires_at_epoch = 0; // for INSERT
+16
View File
@@ -181,6 +181,22 @@ std::int64_t S3Client::get_object(
return total;
}
void S3Client::copy_object(const std::string& src_key,
const std::string& dst_key) {
if (src_key.empty() || dst_key.empty() || src_key == dst_key) {
throw std::invalid_argument("copy_object: bad keys");
}
minio::s3::CopyObjectArgs args;
args.bucket = cfg_.bucket;
args.object = dst_key;
args.source.bucket = cfg_.bucket;
args.source.object = src_key;
minio::s3::CopyObjectResponse resp = impl_->client->CopyObject(args);
if (!resp) {
throw std::runtime_error("S3 CopyObject failed: " + resp.Error().String());
}
}
void S3Client::delete_object(const std::string& key) {
minio::s3::RemoveObjectArgs args;
args.bucket = cfg_.bucket;
+3
View File
@@ -45,6 +45,9 @@ public:
void delete_object(const std::string& key);
// Server-side copy inside the bucket (complete: staging → final).
void copy_object(const std::string& src_key, const std::string& dst_key);
// HeadObject-style metadata. Returns false if the object does not exist.
bool stat_object(const std::string& key, std::int64_t& size_bytes,
std::string& etag);
+3
View File
@@ -6,6 +6,9 @@ DIR=$(CDPATH= cd -- "$(dirname "$0")" && pwd)
echo "=== test_direct_upload_ok.sh ==="
sh "$DIR/test_direct_upload_ok.sh"
echo "=== test_presigned_complete_concurrent.sh ==="
sh "$DIR/test_presigned_complete_concurrent.sh"
echo "=== test_direct_upload_negatives.sh ==="
sh "$DIR/test_direct_upload_negatives.sh"
@@ -43,7 +43,8 @@ expect_code 201
ID1=$(printf '%s' "$HTTP_BODY" | jq -r .id)
http_json POST "/api/v1/direct-uploads/$ID1/complete" \
-H "Authorization: Bearer $ALICE_TOKEN"
expect_code 400
expect_code 409
assert_eq "$(printf '%s' "$HTTP_BODY" | jq -r .error.code)" not_ready
# --- abort ---
http_json POST "/api/v1/chats/$CHAT_ID/direct-uploads" \
@@ -91,7 +92,8 @@ put_code=$(put_via_net "$URL4" "$CT4" "$png")
[ "$put_code" = "200" ] || die "PUT checksum-case got $put_code"
http_json POST "/api/v1/direct-uploads/$ID4/complete" \
-H "Authorization: Bearer $ALICE_TOKEN"
expect_code 400
expect_code 409
assert_eq "$(printf '%s' "$HTTP_BODY" | jq -r .error.code)" hash_mismatch
# Proxy path still works
upload_file "$CHAT_ID" "$png" "proxy.png" "image/png"
+9 -4
View File
@@ -27,14 +27,12 @@ assert_eq "$CT" "image/png" content_type_header
# PUT bytes on the compose network (MinIO ports stay unpublished by default).
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
COMPOSE="docker compose --env-file $ROOT/.env -f $ROOT/compose.yaml"
$COMPOSE run --rm --no-deps \
put_code=$($COMPOSE run --rm --no-deps \
-v "$png:/tmp/up.bin:ro" \
--entrypoint curl api \
-sS -m 60 -o /dev/null -w '%{http_code}' \
-X PUT -H "Content-Type: $CT" --data-binary @/tmp/up.bin \
"$UPLOAD_URL" > /tmp/du_put_code.$$
put_code=$(cat /tmp/du_put_code.$$)
rm -f /tmp/du_put_code.$$
"$UPLOAD_URL")
[ "$put_code" = "200" ] || die "presigned PUT expected 200 got $put_code"
http_json POST "/api/v1/direct-uploads/$UPLOAD_ID/complete" \
@@ -45,6 +43,13 @@ assert_eq "$(printf '%s' "$HTTP_BODY" | jq -r .sha256)" "$want_sha" sha256
assert_eq "$(printf '%s' "$HTTP_BODY" | jq -r .id)" "$UPLOAD_ID" attachment_id
out=$(mktemp)
replay_code=$($COMPOSE run --rm --no-deps \
--entrypoint curl api \
-sS -m 60 -o /dev/null -w '%{http_code}' \
-X PUT -H "Content-Type: $CT" --data-binary 'replay-content-12345' \
"$UPLOAD_URL")
[ "$replay_code" = "200" ] || die "replay PUT expected 200 got $replay_code"
curl -sS -m 60 -o "$out" -H "Authorization: Bearer $ALICE_TOKEN" \
"$BASE_URL/api/v1/attachments/$UPLOAD_ID/content"
assert_eq "$(sha256_file "$out")" "$want_sha" download_sha256
@@ -0,0 +1,59 @@
#!/bin/sh
# A missing S3 object must fail before the content response headers are sent.
set -eu
. "$(dirname "$0")/lib.sh"
png="$FIXTURES/320x200.png"
[ -f "$png" ] || die "missing fixture $png"
COMPOSE="docker compose --env-file $ROOT/.env -f $ROOT/compose.yaml"
response_body=$(mktemp)
object_deleted=0
restore_object() {
action=$1
$COMPOSE run --rm --no-deps -T \
-e "CHAT_ID=$CHAT_ID" -e "ATTACHMENT_ID=$AID" \
-e "OBJECT_ACTION=$action" \
-v "$png:/tmp/restore.png:ro" \
--entrypoint /bin/sh minio-init -c '
set -eu
mc alias set local "$S3_ENDPOINT" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
object="local/$S3_BUCKET/attachments/$CHAT_ID/$ATTACHMENT_ID"
if [ "$OBJECT_ACTION" = delete ]; then
mc rm --force "$object"
else
mc cp /tmp/restore.png "$object"
fi
'
}
cleanup() {
if [ "$object_deleted" = 1 ]; then
restore_object restore || printf 'WARNING: failed to restore test object\n' >&2
fi
rm -f "$response_body"
}
trap cleanup 0 HUP INT TERM
want_sha=$(sha256_file "$png")
TOKEN=""
register_user alice
create_chat 'missing-object' "[\"$USER_ID\"]"
upload_file "$CHAT_ID" "$png" 'sample.png' 'image/png'
expect_code 201
AID=$(printf '%s' "$HTTP_BODY" | jq -r .id)
assert_eq "$(printf '%s' "$HTTP_BODY" | jq -r .sha256)" "$want_sha" metadata_sha
restore_object delete
object_deleted=1
code=$(curl -sS -m 60 -o "$response_body" -w '%{http_code}' \
-H "Authorization: Bearer $TOKEN" \
"$BASE_URL/api/v1/attachments/$AID/content")
case "$code" in
404|5??) ;;
*) die "expected 404/5xx for missing object, got HTTP $code body=$(cat "$response_body")" ;;
esac
restore_object restore
object_deleted=0
printf 'PASS test_download_missing_object\n'
@@ -0,0 +1,73 @@
#!/bin/sh
# Track 1: concurrent complete calls publish one intact attachment.
. "$(dirname "$0")/lib.sh"
png="$FIXTURES/320x200.png"
[ -f "$png" ] || die "missing fixture $png"
want_sha=$(sha256_file "$png")
want_size=$(wc -c < "$png" | tr -d ' ')
tmpdir=$(mktemp -d)
trap 'rm -rf "$tmpdir"' 0 HUP INT TERM
register_user alice
ALICE_TOKEN=$TOKEN
ALICE_ID=$USER_ID
create_chat "du-concurrent" "[\"$ALICE_ID\"]"
http_json POST "/api/v1/chats/$CHAT_ID/direct-uploads" \
-H "Authorization: Bearer $ALICE_TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"filename\":\"concurrent.png\",\"content_type\":\"image/png\",\"size_bytes\":$want_size,\"sha256\":\"$want_sha\"}"
expect_code 201
UPLOAD_ID=$(printf '%s' "$HTTP_BODY" | jq -r .id)
UPLOAD_URL=$(printf '%s' "$HTTP_BODY" | jq -r .upload_url)
CT=$(printf '%s' "$HTTP_BODY" | jq -r '.headers["Content-Type"]')
[ -n "$UPLOAD_ID" ] && [ "$UPLOAD_ID" != null ] || die "empty upload id"
[ -n "$UPLOAD_URL" ] && [ "$UPLOAD_URL" != null ] || die "empty upload_url"
COMPOSE="docker compose --env-file $ROOT/.env -f $ROOT/compose.yaml"
put_code=$($COMPOSE run --rm --no-deps \
-v "$png:/tmp/up.bin:ro" \
--entrypoint curl api \
-sS -m 60 -o /dev/null -w '%{http_code}' \
-X PUT -H "Content-Type: $CT" --data-binary @/tmp/up.bin \
"$UPLOAD_URL")
[ "$put_code" = "200" ] || die "presigned PUT expected 200 got $put_code"
pids=
for i in 1 2 3 4; do
curl -sS -m 60 -o "$tmpdir/body.$i" -w '%{http_code}' \
-X POST \
-H "Authorization: Bearer $ALICE_TOKEN" \
"$BASE_URL/api/v1/direct-uploads/$UPLOAD_ID/complete" \
> "$tmpdir/code.$i" &
pids="$pids $!"
done
for pid in $pids; do
wait "$pid" || die "complete request failed"
done
count_201=0
for i in 1 2 3 4; do
code=$(cat "$tmpdir/code.$i")
case "$code" in
201) count_201=$((count_201 + 1)) ;;
200|409) ;;
*) die "unexpected concurrent complete HTTP $code: $(cat "$tmpdir/body.$i")" ;;
esac
done
assert_eq "$count_201" 1 created_count
http_json GET "/api/v1/attachments/$UPLOAD_ID" \
-H "Authorization: Bearer $ALICE_TOKEN"
expect_code 200
assert_eq "$(printf '%s' "$HTTP_BODY" | jq -r .status)" ready metadata_status
assert_eq "$(printf '%s' "$HTTP_BODY" | jq -r .sha256)" "$want_sha" metadata_sha
code=$(curl -sS -m 60 -o "$tmpdir/content" -w '%{http_code}' \
-H "Authorization: Bearer $ALICE_TOKEN" \
"$BASE_URL/api/v1/attachments/$UPLOAD_ID/content")
assert_eq "$code" 200 content_status
assert_eq "$(sha256_file "$tmpdir/content")" "$want_sha" content_sha
printf 'PASS test_presigned_complete_concurrent\n'
+26 -22
View File
@@ -15,7 +15,7 @@ ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
COMPOSE="docker compose --env-file $ROOT/.env -f $ROOT/compose.yaml"
GRACE_SECONDS="${GRACE_SECONDS:-3600}"
APPLY="${APPLY:-0}"
PREFIX="${ORPHAN_PREFIX:-attachments/}"
PREFIX="${ORPHAN_PREFIX:-}"
S3_BUCKET=$($COMPOSE exec -T api sh -c 'printf %s "$S3_BUCKET"')
[ -n "$S3_BUCKET" ] || { echo "S3_BUCKET empty" >&2; exit 1; }
@@ -25,26 +25,27 @@ trap 'rm -rf "$TMP"' EXIT
echo "gc_orphans: bucket=$S3_BUCKET grace=${GRACE_SECONDS}s apply=$APPLY prefix=$PREFIX"
$COMPOSE run --rm --no-deps --entrypoint /bin/sh minio-init -c "
s3_listing=$($COMPOSE run --rm --no-deps -T --entrypoint /bin/sh minio-init -c "
set -eu
mc alias set local \"\$S3_ENDPOINT\" \"\$MINIO_ROOT_USER\" \"\$MINIO_ROOT_PASSWORD\" >/dev/null
mc ls --recursive --json \"local/\$S3_BUCKET/$PREFIX\" || true
" > "$TMP/s3.json"
")
printf '%s\n' "$s3_listing" > "$TMP/s3.json"
# Known keys: ready attachments + still-pending direct uploads (Track 1).
$COMPOSE exec -T db sh -c \
# Keep published attachments and staging objects for active direct uploads.
pg_keys=$($COMPOSE exec -T db sh -c \
'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atc "
SELECT s3_key FROM attachments
UNION
SELECT s3_key FROM direct_uploads WHERE status = '\''pending'\''
"' \
> "$TMP/pg.keys"
SELECT staging_key FROM direct_uploads WHERE status = '\''pending'\''
"')
printf '%s\n' "$pg_keys" > "$TMP/pg.keys"
python3 - "$TMP/s3.json" "$TMP/pg.keys" "$GRACE_SECONDS" "$TMP/del.keys" <<'PY'
python3 - "$TMP/s3.json" "$TMP/pg.keys" "$GRACE_SECONDS" "$TMP/del.keys" "$PREFIX" <<'PY'
import json, sys, time
from datetime import datetime
s3_path, pg_path, grace_s, out_path = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
s3_path, pg_path, grace_s, out_path, prefix = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4], sys.argv[5]
now = time.time()
known = {line.strip() for line in open(pg_path) if line.strip()}
orphans = []
@@ -59,9 +60,11 @@ for line in open(s3_path):
key = obj.get("key") or obj.get("Key")
if not key:
continue
# mc ls under prefix returns keys relative to that prefix.
if not key.startswith("attachments/"):
key = "attachments/" + key.lstrip("/")
key = key.lstrip("/")
if prefix and not key.startswith(prefix):
key = prefix.rstrip("/") + "/" + key
if not key.startswith(("attachments/", "staging/")):
continue
if key in known:
continue
age = None
@@ -98,14 +101,15 @@ if [ ! -s "$TMP/del.keys" ]; then
exit 0
fi
$COMPOSE run --rm --no-deps --entrypoint /bin/sh minio-init -c "
set -eu
mc alias set local \"\$S3_ENDPOINT\" \"\$MINIO_ROOT_USER\" \"\$MINIO_ROOT_PASSWORD\" >/dev/null
while IFS= read -r key; do
[ -n \"\$key\" ] || continue
echo \"deleting \$S3_BUCKET/\$key\"
mc rm --force \"local/\$S3_BUCKET/\$key\"
done
" < "$TMP/del.keys"
while IFS= read -r key; do
[ -n "$key" ] || continue
echo "deleting $S3_BUCKET/$key"
$COMPOSE run --rm --no-deps -T -e "GC_OBJECT_KEY=$key" \
--entrypoint /bin/sh minio-init -c '
set -eu
mc alias set local "$S3_ENDPOINT" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
mc rm --force "local/$S3_BUCKET/$GC_OBJECT_KEY"
'
done < "$TMP/del.keys"
echo "gc_orphans: delete done"