ЛР 4 — БВТ2303 — Меркулова — на 4 #6
+5
-1
@@ -7,7 +7,7 @@ SESSION_TTL_SECONDS=3600
|
||||
# Choose credentials locally. File paths contain no secret values.
|
||||
DATABASE_URL_FILE=/run/secrets/database_url
|
||||
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
|
||||
SESSION_STORE=postgres
|
||||
SESSION_STORE=redis
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
@@ -18,3 +18,7 @@ S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
|
||||
DB_TIMEOUT_MS=3000
|
||||
DB_MAX_CONNS=10
|
||||
CLEANUP_GRACE_SECONDS=3600
|
||||
|
||||
@@ -1,18 +1,18 @@
|
||||
## Лабораторная 4
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Автор / группа: Меркулова М.А. / БВТ2303
|
||||
Целевая оценка: 4
|
||||
Трек 5: не заявлен
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
Реализованы приватный MinIO, PNG/JPEG-вложения, авторизованное скачивание, связь с сообщениями и очистка незавершённых upload с grace. Из lab3 перенесены две Go API-реплики, Nginx, PostgreSQL, Redis-сессии и идемпотентность. MinIO собирается через `infra/Dockerfile.minio` из официальных исходников закреплённого commit с проверкой SHA-256 архива.
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
Запуск: `./scripts/init-local.sh`, затем `docker compose --env-file .env up -d --build --wait`, `make test`.
|
||||
Результаты: `REPORT.md`, `evidence/lab4-results.md` и JSON в `evidence/`.
|
||||
Ограничения: один host и единичные LB/БД/Redis/MinIO; без треков 5. Секреты генерируются локально вне Git.
|
||||
Источники и ИИ: материалы преподавателя, официальные MinIO/Go/Docker SDK; ИИ использован для реализации и проверок.
|
||||
|
||||
- [x] Контракт и публичные тесты сохранены.
|
||||
- [x] Предусмотрен запуск из отдельного clone с новыми секретами.
|
||||
- [x] Реализованы ограничения файлов, ACL, приватность и отдельные S3 credentials.
|
||||
- [x] Есть очистка после прерываний и ошибок с защитным grace.
|
||||
- [x] Проверки lab4 завершены и реальные результаты приложены.
|
||||
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
FROM golang:1.26.8-alpine3.23@sha256:a8fa79c5bd40d880b52bd3b6d7669ecdcfd00e85facdd427d279efb5ddd79cb1 AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY cmd/ ./cmd/
|
||||
COPY internal/ ./internal/
|
||||
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /messenger ./cmd/messenger
|
||||
|
||||
FROM scratch
|
||||
COPY --from=build /messenger /messenger
|
||||
USER 65532:65532
|
||||
ENV APP_HOST=0.0.0.0 APP_PORT=8080
|
||||
EXPOSE 8080
|
||||
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 CMD ["/messenger", "healthcheck"]
|
||||
ENTRYPOINT ["/messenger"]
|
||||
@@ -1,37 +1,144 @@
|
||||
# Отчёт — лабораторная 4
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Меркулова М.А., БВТ2303
|
||||
- Целевая оценка: 4, трек 5 не заявлен
|
||||
- Ветка: `solution/BVT2303-Merkulova`, создана от исходного `main`
|
||||
- API: Go 1.26.8, PostgreSQL 17.6, Redis 8.2.1, Nginx 1.29.1.
|
||||
- S3: MinIO `RELEASE.2025-10-15T17-29-55Z`, официальный commit `9e49d5e7a648f00e26f2246f4dc28e6b07f8c84a`.
|
||||
- Библиотеки: pgx/v5, go-redis/v9, scrypt, minio-go/v7 7.0.95 и madmin-go/v3 3.0.95.
|
||||
- ИИ использован для проектирования, кода и проверок. Публичный контракт и тесты не изменены; приведены результаты локальных запусков.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker Desktop с Compose/BuildKit, Python 3.9+, make, OpenSSL и доступ к Docker Hub, GitHub codeload и Go module proxy. Локальный Go нужен только для отдельных go test/go vet.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
./scripts/init-local.sh
|
||||
docker compose --env-file .env config --quiet
|
||||
docker compose --env-file .env up -d --build --wait
|
||||
make check
|
||||
make test
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
`Dockerfile` собирает образ Go API, а `infra/Dockerfile.minio` — MinIO из официальных исходников закреплённого commit. SHA-256 архива проверяется при сборке. Docker Compose автоматически собирает оба образа и запускает сервисы; отдельная установка MinIO не требуется.
|
||||
|
||||
## Доказательства критериев
|
||||
`init-local.sh` генерирует случайные локальные secrets и `.env`, которые игнорируются Git. Миграции выполняет одноразовый `migrate`; приватный bucket и ограниченного пользователя создаёт `s3-init`. Он повторяем и не выдаёт root credentials API-копиям. Единственный публичный адрес — `http://127.0.0.1:8080`; API и S3 порты наружу не публикуются. Консоль MinIO выключена (`MINIO_BROWSER=off`), адрес дополнительно ограничен loopback контейнера.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Остановка с сохранением данных: `docker compose --env-file .env down`. Намеренный сброс только учебных данных: `docker compose --env-file .env down -v` — удаляет PostgreSQL/Redis/MinIO volumes.
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
## Архитектура и изменения относительно lab3
|
||||
|
||||
## Эксперименты и отказы
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C[HTTP-клиент] --> L[Nginx]
|
||||
L --> A1[API api-1]
|
||||
L --> A2[API api-2]
|
||||
A1 --> P[(PostgreSQL: метаданные)]
|
||||
A2 --> P
|
||||
A1 --> R[(Redis: сессии)]
|
||||
A2 --> R
|
||||
A1 --> S[(MinIO: приватные оригиналы)]
|
||||
A2 --> S
|
||||
G[Очистка незавершённых загрузок] --> P
|
||||
G --> S
|
||||
```
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Перенесены регистрация, scrypt-пароли, Basic/Bearer, TTL/AOF, чаты, keyset-пагинация, две API-копии, общий Idempotency-Key, readiness-aware Nginx и graceful shutdown. Добавлены `attachments.go`, `objectstore/`, cleanup, миграция 003 и `attachment_ids` сообщений. Отсутствующее поле и `[]` эквивалентны; null и неверные типы отклоняются. Прежняя идемпотентность текстовых сообщений совместима с lab3.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
PostgreSQL — источник истины для пользователей, чатов, сообщений, метаданных и журнала незавершённых upload. Оригиналы — только в приватном bucket `messenger-attachments` с ключами `originals/<UUID>`, которые назначает сервер. Filename — только отображаемое имя. Публичный ответ не содержит object key или credentials.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
История сортируется по `(created_at, id)`; время назначает PostgreSQL через `clock_timestamp()`. Каждый запрос страницы читает актуальные подтверждённые записи, общей snapshot между страницами нет. В неизменной истории курсор не повторяет последнюю запись. При конкурентных транзакциях запись с ключом раньше уже выданного курсора может подтвердиться позднее и не попасть в продолжение обхода; гарантия полного snapshot при одновременных вставках не заявляется.
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Участие в чате проверяется до доступа к вложению; посторонние и attachment другого чата получают 404. Любой участник может прикрепить доступный файл своего чата. Download идёт через API без redirect и возвращает исходные bytes/Content-Type. Ready-метаданные: SHA-256 оригинала, размер, `variants=[]`, `error_code=null`.
|
||||
|
||||
## Лимиты и ресурсы
|
||||
|
||||
Лимиты оригинала жёсткие: 10 485 760 байт, 40 млн пикселей, PNG/JPEG. Content-Type сверяется с содержимым. Размеры проверяются через DecodeConfig до полной распаковки; полное декодирование отбрасывает повреждённые изображения. Filename — 1–255 Unicode code points без `/`, `\`, NUL и управляющих символов.
|
||||
|
||||
Upload копируется через ограниченный reader и буфер 32 KiB с вычислением SHA-256 во временный файл, затем передаётся в S3. Нет неограниченного ReadAll. На реплику один активный upload/decoder, временный файл <=10 MiB + 1 проверочный байт, tmpfs <=20 MiB. Файл удаляется при завершении запроса; API не имеет persistent volume с оригиналами, download от него не зависит. Максимальный 16-bit RGBA bitmap ограничен примерно 320 MB; декодер PNG также выделяет два буфера строк. Для крайнего широкого изображения нужен запас до примерно 960 MB живых буферов. Поэтому лимит API 1536 MiB, а мягкий Go GOMEMLIMIT — 1200 MiB. Потоковый приём не отменяет памяти декодера.
|
||||
|
||||
HTTP: header 5 s, read 45 s, write 60 s. Upload/download context 45 s, S3 PUT 20 s, соединение S3 2 s, SQL обычно 3 s. Nginx не буферизует request/response, upstream timeout 65 s. Байтный лимит применяет API для JSON 413. `MAX_UPLOAD_BYTES`/`MAX_IMAGE_PIXELS` в примере описывают фиксированные пределы, а не увеличение разрешённых значений.
|
||||
|
||||
Ресурсы Docker: API по 0.5 CPU/1536 MiB; PostgreSQL 1 CPU/512 MiB; MinIO 1 CPU/512 MiB; Redis 0.25 CPU/128 MiB; Nginx 0.25 CPU/64 MiB; cleanup 0.25 CPU/128 MiB. Проверки проведены на macOS arm64 с Docker VM 10 vCPU и 8 217 059 328 байт доступной памяти (около 7.65 GiB).
|
||||
|
||||
SIGTERM: Go прекращает новые соединения и ждёт текущие до 8 s; Compose даёт API 10 s. Hook намеренного отказа использует stop timeout 3 s и может оборвать запрос, который клиент повторяет по правилам контракта.
|
||||
|
||||
## Согласованность БД и S3
|
||||
|
||||
1. Неверный или прерванный HTTP upload не создаёт ready-метаданные. Файл полностью принят с лимитами и проверен до S3 PUT.
|
||||
2. Перед PUT отдельно фиксируется durable intent в `attachment_uploads`.
|
||||
3. Транзакция берёт advisory lock по object key, выполняет PUT, создаёт ready-метаданные, удаляет intent и делает COMMIT. Только после этого возвращается 201.
|
||||
4. Ошибка/падение оставляет intent. При неопределённом COMMIT объект немедленно не удаляется: COMMIT мог состояться.
|
||||
5. Cleanup раз в 60 s просматривает старые intents и S3 originals. Grace по умолчанию 3600 s, минимум 120 s — больше бюджета обычного upload. Свежий intent/объект сохраняется. Объект с ready-метаданными сохраняется при любом возрасте, даже если ещё не привязан к сообщению.
|
||||
6. Cleanup использует тот же advisory lock, пропускает активную запись и повторно проверяет БД. Сначала удаляется неучтённый объект, потом intent. Повтор после ошибки/падения безопасен. S3-обход покрывает и объект без intent, например поздний PUT после разрыва связи.
|
||||
|
||||
Ручной проход: `docker compose --env-file .env exec -T cleanup /messenger cleanup`. Это механизм очистки для уровня 4; ремонта missing-объектов и исследовательского трека 5 нет. Время отдельного прохода ограничено; полный обход рассчитан на учебный объём.
|
||||
|
||||
Root credentials есть только у MinIO и s3-init. API/cleanup имеют отдельного пользователя: GetBucketLocation/ListBucket только нашего bucket, GetObject/PutObject/DeleteObject только `originals/*`; создание других bucket запрещено. Nginx не получает секреты БД/S3.
|
||||
|
||||
## Проверки критериев
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go vet ./...
|
||||
python3 student-tests/verify_attachments.py
|
||||
python3 student-tests/verify_upload_recovery.py
|
||||
python3 student-tests/upload_experiment.py
|
||||
ACTION_SCRIPT=./scripts/restart.sh make test-persistence
|
||||
ACTION_SCRIPT=./scripts/stop-one.sh make test-failover
|
||||
./scripts/start-api.sh
|
||||
```
|
||||
|
||||
В `student-tests/` находятся три проверки lab4: вложения и права доступа, восстановление после ошибок загрузки, конкурентные загрузки с измерением памяти. Публичные `tests/` проверяют общий API, сохранность данных и отказ API-реплики. `start-api.sh` восстанавливает реплики после failover; `rolling-update.sh` остаётся служебной командой последовательного обновления API.
|
||||
|
||||
После изменений сервиса дополнительные проверки из `student-tests/regression/` запускаются по отдельности:
|
||||
|
||||
```sh
|
||||
python3 student-tests/regression/verify_idempotency.py
|
||||
python3 student-tests/regression/verify_expiry.py
|
||||
python3 student-tests/regression/verify_dependencies.py
|
||||
python3 student-tests/regression/verify_unready_backend.py
|
||||
```
|
||||
|
||||
| Файл | Что проверяет |
|
||||
| --- | --- |
|
||||
| verify_idempotency.py | 12 одновременных повторов через две API создают одно сообщение; изменение тела с прежним ключом даёт 409 |
|
||||
| verify_expiry.py | Истечение сессии с TTL 2 секунды и возврат обычных настроек |
|
||||
| verify_dependencies.py | Ответы 503 при отказе Redis/PostgreSQL и восстановление доступа |
|
||||
| verify_unready_backend.py | Неготовая api1 исключается из балансировки, api2 продолжает чтение и запись |
|
||||
|
||||
Нужен запущенный локальный Compose-стенд. Тесты выполняются последовательно: они временно меняют TTL, останавливают хранилища или отключают сеть api1 и восстанавливают настройки в `finally`. Старые нагрузочные эксперименты и SQL-демонстрация в этот набор не входят.
|
||||
|
||||
| Критерий | Сценарий | Доказательство |
|
||||
| --- | --- | --- |
|
||||
| Публичный API, checksum, сообщения с файлом, ACL | make test | evidence/lab4-results.md |
|
||||
| Размер, повреждение, MIME, pixel bomb, filename | verify_attachments.py | evidence/attachments-grade4.json |
|
||||
| Чужой attachment_id и привязка другого чата | verify_attachments.py | evidence/attachments-grade4.json |
|
||||
| Upload → остановка загрузившей API → download на другой | verify_attachments.py | evidence/attachments-grade4.json |
|
||||
| Приватный bucket, отдельные ограниченные credentials | anonymous GET=403 и запрещённое CreateBucket | evidence/attachments-grade4.json |
|
||||
| Прерванная передача, отказ S3, ошибка после PUT до INSERT | verify_upload_recovery.py | evidence/upload-recovery.json |
|
||||
| Grace, сохранность fresh/ready, повторная очистка | verify_upload_recovery.py | evidence/upload-recovery.json |
|
||||
| >=10 одновременных upload, измеренный пик памяти | upload_experiment.py | evidence/upload-memory.json |
|
||||
| Restart всех хранилищ с прежним токеном и оригиналом | restart.sh + официальный persistence | evidence/persistence.json |
|
||||
| Отказ API с сохранностью подтверждений и оригинала | официальный failover | evidence/failover.json |
|
||||
| Повтор bootstrap и запуск на новых volumes | s3-init и отдельный тестовый проект | evidence/lab4-results.md |
|
||||
|
||||
Recovery-тест обрывает HTTP-передачу, останавливает S3 и временно добавляет отдельный CHECK к attachments: S3 PUT проходит, а INSERT метаданных отклоняется. CHECK снимается в finally. Выдерживается настоящий grace 120 s. Затем проверяется удаление двух orphan (одного даже без intent), сохранение fresh/ready и повторяемость очистки. Это проверка на учебном стенде.
|
||||
|
||||
## Эксперимент памяти
|
||||
|
||||
Гипотеза: один decoder и ограниченный spool на реплику ограничивают память при конкурентных upload. Генерируется PNG 3000×3000 (9 млн пикселей); 10 клиентов начинают по одному upload через общий барьер. API пересоздаются до опыта для сброса cgroup `memory.peak`. Прогрев — readiness, отдельного upload-прогрева нет. Фиксируются время, успехи/ошибки, RPS, p50/p95/p99 и kernel memory.peak каждой API, включая её старт и tmpfs. Каждый оригинал затем скачивается и сравнивается побайтно. Реальные значения — в `evidence/upload-memory.json`.
|
||||
|
||||
Это один локальный повтор короткой пачки, а не поиск предельного RPS. RSS не обязан мгновенно падать после запроса из-за Go GC. Чтение memory.peak самим бинарным файлом также входит в cgroup; публичного debug endpoint нет.
|
||||
|
||||
Результаты серии 10 upload: 10/10 успехов, 0 ошибок; peak API 88.72/88.86 MiB при лимите 1536 MiB. Отдельно корректная PNG 40 млн пикселей с крайним соотношением сторон принята, пик обслужившей копии 665.25 MiB. Это разные опыты; полные результаты приведены в `evidence/lab4-results.md`.
|
||||
|
||||
## Ограничения
|
||||
|
||||
Все компоненты на одном Docker-host. Отказ одной API переносится, но MinIO/Redis/PostgreSQL/Nginx и host остаются единичными точками отказа. При отказе S3 — 503 и отрицательная readiness. Не реализованы прямой presigned-upload, worker и варианты изображений, репликация хранилищ и треки на 5.
|
||||
|
||||
LB опрашивает readiness с паузой 1 s и таймаутом каждой пробы 2 s: расчётно до 5 s на обнаружение в цикле плюс reload; healthstatus Docker независим и может измениться позже. Краткие ошибки POST до обнаружения допустимы и измеряются; клиент сообщения повторяет тот же ключ.
|
||||
|
||||
## Источники
|
||||
|
||||
Контракт текущего skeleton; [официальный MinIO commit](https://github.com/minio/minio/tree/9e49d5e7a648f00e26f2246f4dc28e6b07f8c84a), [MinIO Go SDK](https://pkg.go.dev/github.com/minio/minio-go/v7@v7.0.95), [Admin SDK](https://pkg.go.dev/github.com/minio/madmin-go/v3@v3.0.95), [Go image](https://pkg.go.dev/image), [Docker Compose](https://docs.docker.com/reference/compose-file/).
|
||||
|
||||
@@ -0,0 +1,289 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"mtusi-messenger/internal/database"
|
||||
"mtusi-messenger/internal/messenger"
|
||||
"mtusi-messenger/internal/objectstore"
|
||||
)
|
||||
|
||||
func env(name, fallback string) string {
|
||||
if value := os.Getenv(name); value != "" {
|
||||
return value
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
func secret(envName, fileEnv string) (string, error) {
|
||||
if value := os.Getenv(envName); value != "" {
|
||||
return value, nil
|
||||
}
|
||||
path := os.Getenv(fileEnv)
|
||||
if path == "" {
|
||||
return "", fmt.Errorf("set %s or %s", envName, fileEnv)
|
||||
}
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
value := strings.TrimSpace(string(b))
|
||||
if value == "" {
|
||||
return "", fmt.Errorf("%s is empty", path)
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
func positiveInt(name string, fallback int) (int, error) {
|
||||
raw := env(name, strconv.Itoa(fallback))
|
||||
n, err := strconv.Atoi(raw)
|
||||
if err != nil || n < 1 {
|
||||
return 0, fmt.Errorf("invalid %s", name)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
func main() {
|
||||
if len(os.Args) > 1 && os.Args[1] == "memory" {
|
||||
values := map[string]int64{}
|
||||
for _, name := range []string{"memory.current", "memory.peak", "memory.max"} {
|
||||
data, err := os.ReadFile("/sys/fs/cgroup/" + name)
|
||||
if err != nil {
|
||||
log.Fatal("cgroup memory metrics unavailable")
|
||||
}
|
||||
n, err := strconv.ParseInt(strings.TrimSpace(string(data)), 10, 64)
|
||||
if err != nil {
|
||||
log.Fatal("invalid cgroup metric")
|
||||
}
|
||||
values[name] = n
|
||||
}
|
||||
_ = json.NewEncoder(os.Stdout).Encode(values)
|
||||
return
|
||||
}
|
||||
port := env("APP_PORT", "8080")
|
||||
if len(os.Args) > 1 && os.Args[1] == "healthcheck" {
|
||||
client := &http.Client{Timeout: 2 * time.Second}
|
||||
resp, err := client.Get("http://127.0.0.1:" + port + "/health/ready")
|
||||
if err != nil || resp.StatusCode != http.StatusOK {
|
||||
os.Exit(1)
|
||||
}
|
||||
_ = resp.Body.Close()
|
||||
return
|
||||
}
|
||||
host := env("APP_HOST", "0.0.0.0")
|
||||
if _, err := strconv.Atoi(port); err != nil {
|
||||
log.Fatal("invalid APP_PORT")
|
||||
}
|
||||
ttlSeconds, err := positiveInt("SESSION_TTL_SECONDS", 3600)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
idempotencyTTLSeconds, err := positiveInt("IDEMPOTENCY_TTL_SECONDS", 86400)
|
||||
if err != nil || idempotencyTTLSeconds < 86400 {
|
||||
log.Fatal("IDEMPOTENCY_TTL_SECONDS must be at least 86400")
|
||||
}
|
||||
dbTimeoutMS, err := positiveInt("DB_TIMEOUT_MS", 3000)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
maxConns, err := positiveInt("DB_MAX_CONNS", 10)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
databaseURL, err := secret("DATABASE_URL", "DATABASE_URL_FILE")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
logger := log.New(os.Stdout, "", log.LstdFlags|log.LUTC)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
config, err := pgxpool.ParseConfig(databaseURL)
|
||||
if err != nil {
|
||||
log.Fatal("invalid database configuration")
|
||||
}
|
||||
config.MaxConns = int32(maxConns)
|
||||
config.ConnConfig.RuntimeParams["application_name"] = env("INSTANCE_ID", "maintenance")
|
||||
config.MinConns = 1
|
||||
config.MaxConnLifetime = 30 * time.Minute
|
||||
var pool *pgxpool.Pool
|
||||
for {
|
||||
pool, err = pgxpool.NewWithConfig(ctx, config)
|
||||
if err == nil {
|
||||
err = pool.Ping(ctx)
|
||||
}
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if pool != nil {
|
||||
pool.Close()
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
log.Fatal("database unavailable during startup")
|
||||
}
|
||||
time.Sleep(time.Second)
|
||||
}
|
||||
defer pool.Close()
|
||||
if len(os.Args) > 1 && os.Args[1] == "migrate" {
|
||||
if err = database.Migrate(ctx, pool); err != nil {
|
||||
log.Fatalf("migration failed: %v", err)
|
||||
}
|
||||
logger.Print("migrations complete")
|
||||
return
|
||||
}
|
||||
access, err := secret("S3_ACCESS_KEY_ID", "S3_ACCESS_KEY_ID_FILE")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
secretKey, err := secret("S3_SECRET_ACCESS_KEY", "S3_SECRET_ACCESS_KEY_FILE")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
objects, err := objectstore.New(env("S3_ENDPOINT", "http://s3:9000"), env("S3_REGION", "us-east-1"), env("S3_BUCKET", "messenger-attachments"), access, secretKey)
|
||||
if err != nil {
|
||||
log.Fatal("invalid S3 configuration")
|
||||
}
|
||||
if len(os.Args) > 1 && os.Args[1] == "s3-bootstrap" {
|
||||
for {
|
||||
err = objects.Client.MakeBucket(ctx, objects.Bucket, minio.MakeBucketOptions{Region: env("S3_REGION", "us-east-1")})
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if exists, e := objects.Client.BucketExists(ctx, objects.Bucket); e == nil && exists {
|
||||
break
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
log.Fatal("S3 bootstrap unavailable")
|
||||
}
|
||||
time.Sleep(time.Second)
|
||||
}
|
||||
if err = objects.Client.SetBucketPolicy(ctx, objects.Bucket, ""); err != nil {
|
||||
log.Fatal("cannot enforce private bucket")
|
||||
}
|
||||
apiAccess, err := secret("S3_API_ACCESS_KEY_ID", "S3_API_ACCESS_KEY_ID_FILE")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
apiSecret, err := secret("S3_API_SECRET_ACCESS_KEY", "S3_API_SECRET_ACCESS_KEY_FILE")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if err = objectstore.Bootstrap(ctx, env("S3_ENDPOINT", "http://s3:9000"), objects.Bucket, access, secretKey, apiAccess, apiSecret); err != nil {
|
||||
log.Fatalf("S3 policy bootstrap: %v", err)
|
||||
}
|
||||
logger.Print("private MinIO bucket and limited API user ready")
|
||||
return
|
||||
}
|
||||
if len(os.Args) > 1 && (os.Args[1] == "cleanup" || os.Args[1] == "cleanup-loop") {
|
||||
grace, err := positiveInt("CLEANUP_GRACE_SECONDS", 3600)
|
||||
if err != nil || grace < 120 {
|
||||
log.Fatal("CLEANUP_GRACE_SECONDS must be at least 120")
|
||||
}
|
||||
loopctx, stopCleanup := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
|
||||
defer stopCleanup()
|
||||
for {
|
||||
cleanupctx, cancelCleanup := context.WithTimeout(loopctx, 60*time.Second)
|
||||
result, cleanupErr := messenger.Cleanup(cleanupctx, pool, objects, time.Duration(grace)*time.Second)
|
||||
cancelCleanup()
|
||||
if cleanupErr != nil {
|
||||
logger.Print("cleanup failed; retained objects for next sweep")
|
||||
} else {
|
||||
_ = json.NewEncoder(os.Stdout).Encode(result)
|
||||
}
|
||||
if os.Args[1] == "cleanup" {
|
||||
if cleanupErr != nil {
|
||||
os.Exit(1)
|
||||
}
|
||||
return
|
||||
}
|
||||
select {
|
||||
case <-loopctx.Done():
|
||||
return
|
||||
case <-time.After(60 * time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(os.Args) > 2 && os.Args[1] == "s3-audit" {
|
||||
if err = objects.Audit(ctx, os.Args[2]); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Println("PASS: anonymous S3 GET=403; API credentials cannot create buckets")
|
||||
return
|
||||
}
|
||||
if len(os.Args) > 2 && os.Args[1] == "s3-stat" {
|
||||
info, err := objects.Client.StatObject(ctx, objects.Bucket, os.Args[2], minio.StatObjectOptions{})
|
||||
if err != nil {
|
||||
if minio.ToErrorResponse(err).Code != "NoSuchKey" {
|
||||
log.Fatal("S3 stat failed")
|
||||
}
|
||||
fmt.Println(`{"exists":false}`)
|
||||
return
|
||||
}
|
||||
_ = json.NewEncoder(os.Stdout).Encode(map[string]any{"exists": true, "size_bytes": info.Size})
|
||||
return
|
||||
}
|
||||
instanceID := env("INSTANCE_ID", "")
|
||||
if instanceID == "" {
|
||||
log.Fatal("INSTANCE_ID is required")
|
||||
}
|
||||
cacheURL, err := secret("CACHE_URL", "CACHE_URL_FILE")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
redisOptions, err := redis.ParseURL(cacheURL)
|
||||
if err != nil {
|
||||
log.Fatal("invalid cache configuration")
|
||||
}
|
||||
sessions := redis.NewClient(redisOptions)
|
||||
defer sessions.Close()
|
||||
for {
|
||||
if err = sessions.Ping(ctx).Err(); err == nil {
|
||||
break
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
log.Fatal("session store unavailable during startup")
|
||||
}
|
||||
time.Sleep(time.Second)
|
||||
}
|
||||
for {
|
||||
if err = objects.Ready(ctx); err == nil {
|
||||
break
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
log.Fatal("S3 unavailable during startup")
|
||||
}
|
||||
time.Sleep(time.Second)
|
||||
}
|
||||
dbTimeout := time.Duration(dbTimeoutMS) * time.Millisecond
|
||||
srv := &http.Server{Addr: host + ":" + port, Handler: messenger.NewHandler(pool, sessions, objects, time.Duration(ttlSeconds)*time.Second, time.Duration(idempotencyTTLSeconds)*time.Second, dbTimeout, instanceID, logger), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 45 * time.Second, WriteTimeout: 60 * time.Second, IdleTimeout: 30 * time.Second, MaxHeaderBytes: 32 * 1024}
|
||||
stop := make(chan os.Signal, 1)
|
||||
signal.Notify(stop, syscall.SIGTERM, os.Interrupt)
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
<-stop
|
||||
logger.Print("shutdown started")
|
||||
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 8*time.Second)
|
||||
defer shutdownCancel()
|
||||
if shutdownErr := srv.Shutdown(shutdownCtx); shutdownErr != nil {
|
||||
logger.Printf("shutdown error: %v", shutdownErr)
|
||||
}
|
||||
logger.Print("shutdown complete")
|
||||
}()
|
||||
logger.Printf("listening on %s", srv.Addr)
|
||||
if err = srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
logger.Fatal(err)
|
||||
}
|
||||
<-done
|
||||
}
|
||||
+286
@@ -0,0 +1,286 @@
|
||||
name: mtusi-lab4
|
||||
|
||||
x-api-common: &api-common
|
||||
image: mtusi-lab4-api:local
|
||||
environment: &api-environment
|
||||
APP_HOST: ${APP_HOST:-0.0.0.0}
|
||||
APP_PORT: 8080
|
||||
APP_ENV: ${APP_ENV:-lab}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
DB_TIMEOUT_MS: ${DB_TIMEOUT_MS:-3000}
|
||||
DB_MAX_CONNS: ${DB_MAX_CONNS:-10}
|
||||
GOMEMLIMIT: 1200MiB
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: us-east-1
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
CLEANUP_GRACE_SECONDS: ${CLEANUP_GRACE_SECONDS:-3600}
|
||||
DATABASE_URL_FILE: ${DATABASE_URL_FILE:-/run/secrets/database_url}
|
||||
CACHE_URL_FILE: ${CACHE_URL_FILE:-/run/secrets/cache_url}
|
||||
secrets:
|
||||
- database_url
|
||||
- cache_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
tmpfs:
|
||||
- /tmp:size=20m,mode=1777
|
||||
networks:
|
||||
- frontend
|
||||
- backend
|
||||
healthcheck:
|
||||
test: ["CMD", "/messenger", "healthcheck"]
|
||||
interval: 3s
|
||||
timeout: 2s
|
||||
retries: 20
|
||||
start_period: 3s
|
||||
restart: unless-stopped
|
||||
init: true
|
||||
stop_grace_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.50"
|
||||
memory: 1536M
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17.6-alpine3.22
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets:
|
||||
- postgres_password
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
networks:
|
||||
- backend
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.00"
|
||||
memory: 512M
|
||||
|
||||
redis:
|
||||
image: redis:8.2.1-alpine3.22
|
||||
command:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- exec redis-server --appendonly yes --appendfsync always --requirepass "$$(cat /run/secrets/redis_password)"
|
||||
secrets:
|
||||
- redis_password
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
networks:
|
||||
- backend
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "redis-cli --no-auth-warning -a \"$$(cat /run/secrets/redis_password)\" ping | grep PONG"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 128M
|
||||
|
||||
migrate:
|
||||
image: mtusi-lab4-api:local
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
healthcheck:
|
||||
disable: true
|
||||
command: ["migrate"]
|
||||
environment:
|
||||
<<: *api-environment
|
||||
secrets:
|
||||
- database_url
|
||||
networks:
|
||||
- backend
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 128M
|
||||
|
||||
api1:
|
||||
<<: *api-common
|
||||
environment:
|
||||
<<: *api-environment
|
||||
INSTANCE_ID: api-1
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
redis:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
|
||||
api2:
|
||||
<<: *api-common
|
||||
environment:
|
||||
<<: *api-environment
|
||||
INSTANCE_ID: api-2
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
redis:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
|
||||
s3:
|
||||
image: mtusi-lab4-minio:RELEASE.2025-10-15T17-29-55Z
|
||||
build:
|
||||
context: .
|
||||
dockerfile: infra/Dockerfile.minio
|
||||
command: ["server", "/data", "--address", ":9000", "--console-address", "127.0.0.1:9001"]
|
||||
environment:
|
||||
MINIO_ROOT_USER_FILE: /run/secrets/s3_admin_access
|
||||
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_admin_secret
|
||||
MINIO_BROWSER: "off"
|
||||
MINIO_UPDATE: "off"
|
||||
volumes:
|
||||
- minio_data:/data
|
||||
secrets:
|
||||
- s3_admin_access
|
||||
- s3_admin_secret
|
||||
networks:
|
||||
- backend
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-T", "1", "-O", "/dev/null", "http://127.0.0.1:9000/minio/health/ready"]
|
||||
interval: 3s
|
||||
timeout: 2s
|
||||
retries: 20
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.00"
|
||||
memory: 512M
|
||||
|
||||
s3-init:
|
||||
image: mtusi-lab4-api:local
|
||||
healthcheck:
|
||||
disable: true
|
||||
command: ["s3-bootstrap"]
|
||||
environment:
|
||||
<<: *api-environment
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_admin_access
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_admin_secret
|
||||
S3_API_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_API_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
secrets:
|
||||
- database_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
- s3_admin_access
|
||||
- s3_admin_secret
|
||||
networks:
|
||||
- backend
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
s3:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 128M
|
||||
|
||||
cleanup:
|
||||
image: mtusi-lab4-api:local
|
||||
healthcheck:
|
||||
disable: true
|
||||
command: ["cleanup-loop"]
|
||||
environment:
|
||||
<<: *api-environment
|
||||
secrets:
|
||||
- database_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
networks:
|
||||
- backend
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 128M
|
||||
|
||||
nginx:
|
||||
image: nginx:1.29.1-alpine3.22
|
||||
command: ["/bin/sh", "/health-aware.sh"]
|
||||
volumes:
|
||||
- ./infra/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./infra/health-aware.sh:/health-aware.sh:ro
|
||||
networks:
|
||||
- frontend
|
||||
depends_on:
|
||||
api1:
|
||||
condition: service_healthy
|
||||
api2:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "127.0.0.1:${APP_PORT:-8080}:8080"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:8080/health/ready"]
|
||||
interval: 3s
|
||||
timeout: 2s
|
||||
retries: 20
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 64M
|
||||
|
||||
secrets:
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
redis_password:
|
||||
file: ./secrets/redis_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
cache_url:
|
||||
file: ./secrets/cache_url
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
s3_admin_access:
|
||||
file: ./secrets/s3_admin_access
|
||||
s3_admin_secret:
|
||||
file: ./secrets/s3_admin_secret
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
minio_data:
|
||||
|
||||
networks:
|
||||
frontend:
|
||||
backend:
|
||||
internal: true
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"png_jpeg": "PASS",
|
||||
"size_corruption_mime_pixels_filename": "PASS",
|
||||
"exact_size_uploads_through_lb": [
|
||||
6294458,
|
||||
10485760
|
||||
],
|
||||
"corrupt_idat_and_missing_iend": "PASS",
|
||||
"pixel_limit_40005000_and_200000000": "PASS",
|
||||
"cross_chat_and_outsider": "PASS",
|
||||
"message_idempotency": "PASS",
|
||||
"wide_png_40000000_pixels": "PASS",
|
||||
"wide_memory_bytes": {
|
||||
"api1": {
|
||||
"memory.current": 85532672,
|
||||
"memory.max": 1610612736,
|
||||
"memory.peak": 85532672
|
||||
},
|
||||
"api2": {
|
||||
"memory.current": 697561088,
|
||||
"memory.max": 1610612736,
|
||||
"memory.peak": 697561088
|
||||
}
|
||||
},
|
||||
"private_bucket_and_limited_credentials": "PASS",
|
||||
"upload_replica_stopped_download_equal": true
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"status": "PASS",
|
||||
"metrics": {
|
||||
"observed_instances_before": 2,
|
||||
"successful_writes": 129,
|
||||
"failed_attempts": 0,
|
||||
"first_recovery_seconds": 0.831
|
||||
},
|
||||
"stopped_services": [
|
||||
"api2"
|
||||
],
|
||||
"confirmed_data_session_and_original_preserved": true,
|
||||
"replicas_restored": true
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
# Результаты лабораторной 4
|
||||
|
||||
Проверено 20 октября 2026 года. Ветка `solution/BVT2303-Merkulova`. Среда: macOS arm64, Docker VM 10 vCPU, 8 217 059 328 байт памяти. Данные синтетические; пароли, токены и ключи в результатах не сохранялись.
|
||||
|
||||
## MinIO и запуск с пустыми хранилищами
|
||||
|
||||
MinIO `RELEASE.2025-10-15T17-29-55Z` собирается через `infra/Dockerfile.minio` из официального архива. Commit: `9e49d5e7a648f00e26f2246f4dc28e6b07f8c84a`. Проверяемый SHA-256 архива: `be6d0bd3696c3a13a35f02d3a0280b64319c67918b4501c5c3d87f96d000085c`.
|
||||
|
||||
```text
|
||||
$ docker compose --env-file .env exec -T s3 /minio --version
|
||||
minio version RELEASE.2025-10-15T17-29-55Z (commit-id=9e49d5e7a648f00e26f2246f4dc28e6b07f8c84a)
|
||||
Runtime: go1.26.8 linux/arm64
|
||||
```
|
||||
|
||||
Отдельная копия файлов проекта без `.env`, secrets и прежних volumes запущена на порту 18084. Созданы новые секреты и пустые PostgreSQL/Redis/MinIO volumes. Кэш сборки Docker использовался; данные хранилищ были новыми. Это локальная проверка изолированной копии рабочего дерева; commit/push и clone удалённой ветки ещё не выполнены.
|
||||
|
||||
```text
|
||||
$ ./scripts/init-local.sh
|
||||
$ APP_PORT=18084 docker compose --project-name mtusi-lab4-latest-clean --env-file .env up -d --build --wait
|
||||
PASS: MinIO healthy; migrate and s3-init completed; both API replicas and Nginx healthy
|
||||
$ make test BASE_URL=http://localhost:18084
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
$ APP_PORT=18084 docker compose --project-name mtusi-lab4-latest-clean --env-file .env run --rm --no-deps s3-init
|
||||
private MinIO bucket and limited API user ready
|
||||
```
|
||||
|
||||
Временный стенд после проверки удалён. Основной стенд и его volumes сохранены. Проверки вложений, очистки, нагрузки, persistence и failover ниже повторены на указанном октябрьском релизе MinIO.
|
||||
|
||||
## Исходники и контракт
|
||||
|
||||
```text
|
||||
$ CGO_ENABLED=0 go test ./...
|
||||
PASS: all packages; password hash tests passed
|
||||
$ CGO_ENABLED=0 go vet ./...
|
||||
exit 0, no diagnostics
|
||||
$ make check
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
```
|
||||
|
||||
Проверки Go выполнены с отдельным временным GOCACHE. `git diff` исходных README/COURSE/CONTRIBUTING/MAINTAINERS/Makefile/lab.json/contracts/tests, преподавательских check/validate scripts и contracts workflow пустой. Секреты и `.env` игнорируются Git. Python-синтаксис собственных тестов, executable bits shell-скриптов и Compose config проверены. Полная OpenAPI-валидация требует дополнительных зависимостей и в этом прогоне не запускалась; контракт не изменялся.
|
||||
|
||||
## Файлы, ограничения и права
|
||||
|
||||
```text
|
||||
$ python3 student-tests/verify_attachments.py
|
||||
PASS: anonymous S3 GET=403; API credentials cannot create buckets
|
||||
PASS: attachment validation, authorization, idempotency and cross-replica download
|
||||
```
|
||||
|
||||
Результаты: [attachments-grade4.json](attachments-grade4.json).
|
||||
|
||||
- PNG и JPEG приняты; размер, lowercase SHA-256 и скачанные байты совпали с оригиналами.
|
||||
- Через LB приняты корректные PNG размером 6 294 458 и ровно 10 485 760 байт. 10 485 761 байт отклонён с 413.
|
||||
- Повреждённый IDAT, PNG без IEND, обрезанный файл, не-изображение и подмена MIME отклонены с 415.
|
||||
- Корректный PNG ровно 40 млн пикселей принят; заголовки 40 005 000, 100 000 000 и 200 000 000 пикселей отклонены с 413 до полной распаковки.
|
||||
- Опасные filename, повторяющиеся/слишком длинные списки ID, null, отсутствие text и неверные типы полей отклонены с 400.
|
||||
- Посторонний, неизвестный attachment_id и файл другого чата получают 404. Другой участник того же чата может прикрепить доступное вложение.
|
||||
- Повтор сообщения возвращает тот же результат; другое тело с прежним ключом даёт 409. Отсутствующий attachment_ids эквивалентен [].
|
||||
- Анонимный GET объекта напрямую из S3 вернул 403; API credentials не разрешают создать другой bucket.
|
||||
- Реальная API-реплика из ответа upload остановлена. После её исключения из upstream в пределах 10 секунд другая реплика скачала оригинал без изменений. Сам download выполняется без повторов.
|
||||
|
||||
Проверка Docker inspect обеих API подтвердила отсутствие root-переменных и admin secret mounts, опубликованных портов и persistent volume оригиналов. Секретные значения не выводились. Консоль MinIO выключена и привязана к loopback контейнера.
|
||||
|
||||
В отдельном опыте с PNG 40 000 000×1, 16-bit RGBA пик API составил 697561088 байт (665.25 MiB), ниже hard limit 1536 MiB. Это измерение отдельного запуска, не универсальный пик для всех изображений.
|
||||
|
||||
## Ошибки и очистка
|
||||
|
||||
```text
|
||||
$ python3 student-tests/verify_upload_recovery.py
|
||||
PASS: interrupted upload; S3 outage=503 and recovery; failed metadata after successful PUT
|
||||
PASS: grace=120 s; orphan objects removed; fresh/ready preserved; repeat removed 0
|
||||
```
|
||||
|
||||
Результаты: [upload-recovery.json](upload-recovery.json). Реальная оборванная HTTP-передача не создала ready-метаданные. После остановки MinIO readiness и upload вернули JSON 503, а анонимный бизнес-запрос остался 401. После запуска MinIO исходный файл скачан без изменений.
|
||||
|
||||
Временный CHECK в PostgreSQL отклонил INSERT метаданных после успешного S3 PUT; CHECK снят в finally. Созданы два orphan, у одного намеренно удалён intent. До окончания grace объекты сохранены. После ожидания 122 секунд удалено 2 объекта и 2 intents; ready-объект сохранён. Повторный проход удалил 0 объектов и 0 intents. Эти проверки используют минимальный тестовый grace 120 s; обычное значение — 3600 s.
|
||||
|
||||
## Десять одновременных загрузок
|
||||
|
||||
```text
|
||||
$ python3 student-tests/upload_experiment.py
|
||||
PASS: 10 successful uploads; 0 failures; all originals downloaded unchanged
|
||||
```
|
||||
|
||||
PNG 3000×3000, 9 млн пикселей, 6930898 байт. Десять клиентов стартуют через общий барьер. До опыта API пересозданы для сброса cgroup memory.peak. Отдельного upload-прогрева нет. Один upload/decoder на реплику, tmpfs 20 MiB, hard limit 1536 MiB, Go GOMEMLIMIT 1200 MiB.
|
||||
|
||||
Успехи: 10; ошибки: 0; длительность 0.822 s; 12.171 successful requests/s. p50=517.029 ms; p95=820.418 ms; p99=820.418 ms. При 10 измерениях p95/p99 совпадают с максимумом.
|
||||
|
||||
| API | Peak bytes | Peak MiB |
|
||||
| --- | ---: | ---: |
|
||||
| api1 | 93032448 | 88.72 |
|
||||
| api2 | 93179904 | 88.86 |
|
||||
|
||||
Источник: Linux cgroup v2 memory.peak, включая старт API, tmpfs и диагностический процесс. Это одна локальная серия, не поиск предельного RPS. Полные данные: [upload-memory.json](upload-memory.json).
|
||||
|
||||
## Сохранность и отказ API
|
||||
|
||||
```text
|
||||
$ ACTION_SCRIPT=./scripts/restart.sh make test-persistence
|
||||
PASS: persistence preserved confirmed data and existing session.
|
||||
$ ACTION_SCRIPT=./scripts/stop-one.sh make test-failover
|
||||
PASS: failover preserved confirmed data and existing session.
|
||||
```
|
||||
|
||||
[persistence.json](persistence.json) содержит времена запуска до и после: PostgreSQL, Redis, MinIO, api1, api2, cleanup и Nginx действительно перезапущены. Прежний токен читает подтверждённые данные и тот же оригинал. Volumes не удалялись.
|
||||
|
||||
[failover.json](failover.json): до отказа наблюдались две реплики; успешных записей 129, неудачных попыток 0, первый успешный ответ после остановки за 0.831 s (бюджет ≤10 s). Подтверждённые сообщения и файл сохранились, дублей нет. Одна реплика действительно остановлена; после проверки обе запущены обратно.
|
||||
|
||||
## Дополнительные проверки сохранённых возможностей
|
||||
|
||||
В `student-tests/regression/` находятся четыре проверки. Команды отдельного запуска:
|
||||
|
||||
```sh
|
||||
python3 student-tests/regression/verify_idempotency.py
|
||||
python3 student-tests/regression/verify_expiry.py
|
||||
python3 student-tests/regression/verify_dependencies.py
|
||||
python3 student-tests/regression/verify_unready_backend.py
|
||||
```
|
||||
|
||||
Все четыре проверки прошли; результаты текущего прогона — [regression.json](regression.json). Двенадцать одновременных запросов через обе API создали одно сообщение; конфликтующий повтор получил 409. Сессия с TTL 2 s истекла. Отказы Redis/PostgreSQL дали 503, доступ после запуска восстановился. После отключения backend-сети api1 здоровая api2 обслужила 40 GET и 10 POST. Обычный TTL, запущенные хранилища и backend-сеть api1 восстановлены.
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"status": "PASS",
|
||||
"restarted_services": {
|
||||
"postgres": true,
|
||||
"redis": true,
|
||||
"s3": true,
|
||||
"api1": true,
|
||||
"api2": true,
|
||||
"cleanup": true,
|
||||
"nginx": true
|
||||
},
|
||||
"before_started_at": {
|
||||
"postgres": "2026-10-20T17:57:17.622217678Z",
|
||||
"redis": "2026-10-20T17:57:17.599509803Z",
|
||||
"s3": "2026-10-20T18:01:54.288039876Z",
|
||||
"api1": "2026-10-20T18:01:10.796988342Z",
|
||||
"api2": "2026-10-20T18:01:10.785085842Z",
|
||||
"cleanup": "2026-10-20T17:57:21.31433068Z",
|
||||
"nginx": "2026-10-20T17:57:27.117387044Z"
|
||||
},
|
||||
"after_started_at": {
|
||||
"postgres": "2026-10-20T18:04:39.232420924Z",
|
||||
"redis": "2026-10-20T18:04:39.269826383Z",
|
||||
"s3": "2026-10-20T18:04:39.205629091Z",
|
||||
"api1": "2026-10-20T18:04:43.077954926Z",
|
||||
"api2": "2026-10-20T18:04:43.017981093Z",
|
||||
"cleanup": "2026-10-20T18:04:42.913498926Z",
|
||||
"nginx": "2026-10-20T18:04:48.819410929Z"
|
||||
},
|
||||
"confirmed_data_session_and_original_preserved": true
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"verify_idempotency.py": {
|
||||
"status": "PASS",
|
||||
"seconds": 0.376
|
||||
},
|
||||
"verify_expiry.py": {
|
||||
"status": "PASS",
|
||||
"seconds": 15.661
|
||||
},
|
||||
"verify_dependencies.py": {
|
||||
"status": "PASS",
|
||||
"seconds": 1.811
|
||||
},
|
||||
"verify_unready_backend.py": {
|
||||
"status": "PASS",
|
||||
"seconds": 63.148
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"concurrent_uploads": 10,
|
||||
"file_size_bytes": 6930898,
|
||||
"pixels": 9000000,
|
||||
"successful_uploads": 10,
|
||||
"failed_uploads": 0,
|
||||
"elapsed_seconds": 0.822,
|
||||
"successful_rps": 12.171,
|
||||
"latency_ms": {
|
||||
"p50": 517.029,
|
||||
"p95": 820.418,
|
||||
"p99": 820.418
|
||||
},
|
||||
"memory_before_bytes": {
|
||||
"api1": {
|
||||
"memory.current": 13914112,
|
||||
"memory.max": 1610612736,
|
||||
"memory.peak": 14172160
|
||||
},
|
||||
"api2": {
|
||||
"memory.current": 14135296,
|
||||
"memory.max": 1610612736,
|
||||
"memory.peak": 15749120
|
||||
}
|
||||
},
|
||||
"memory_after_bytes": {
|
||||
"api1": {
|
||||
"memory.current": 91078656,
|
||||
"memory.max": 1610612736,
|
||||
"memory.peak": 93032448
|
||||
},
|
||||
"api2": {
|
||||
"memory.current": 89001984,
|
||||
"memory.max": 1610612736,
|
||||
"memory.peak": 93179904
|
||||
}
|
||||
},
|
||||
"peak_source": "Linux cgroup v2 memory.peak; fresh API containers including startup",
|
||||
"decoder_slots_per_replica": 1,
|
||||
"instances": [
|
||||
"api-1",
|
||||
"api-2"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"partial_upload": "PASS",
|
||||
"s3_outage_503_and_recovery": "PASS",
|
||||
"metadata_failure_after_put": "PASS",
|
||||
"orphan_without_intent": "PASS",
|
||||
"grace_seconds": 120,
|
||||
"fresh_preserved": true,
|
||||
"ready_preserved": true,
|
||||
"fresh_sweep": {
|
||||
"removed_objects": 0,
|
||||
"removed_intents": 0,
|
||||
"kept_ready": 60,
|
||||
"skipped_recent": 0,
|
||||
"skipped_busy": 0
|
||||
},
|
||||
"cleanup": {
|
||||
"removed_objects": 2,
|
||||
"removed_intents": 2,
|
||||
"kept_ready": 78,
|
||||
"skipped_recent": 0,
|
||||
"skipped_busy": 0
|
||||
},
|
||||
"repeat": {
|
||||
"removed_objects": 0,
|
||||
"removed_intents": 0,
|
||||
"kept_ready": 78,
|
||||
"skipped_recent": 0,
|
||||
"skipped_busy": 0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
module mtusi-messenger
|
||||
|
||||
go 1.26.0
|
||||
|
||||
require (
|
||||
github.com/jackc/pgx/v5 v5.7.6
|
||||
github.com/minio/madmin-go/v3 v3.0.95
|
||||
github.com/minio/minio-go/v7 v7.0.95
|
||||
github.com/redis/go-redis/v9 v9.14.0
|
||||
golang.org/x/crypto v0.42.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/go-ini/ini v1.67.0 // indirect
|
||||
github.com/go-ole/go-ole v1.3.0 // indirect
|
||||
github.com/goccy/go-json v0.10.5 // indirect
|
||||
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
|
||||
github.com/golang/protobuf v1.5.4 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/klauspost/compress v1.18.0 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
|
||||
github.com/lufia/plan9stats v0.0.0-20240909124753-873cd0166683 // indirect
|
||||
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
|
||||
github.com/minio/crc64nvme v1.0.2 // indirect
|
||||
github.com/minio/md5-simd v1.1.2 // indirect
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
||||
github.com/philhofer/fwd v1.2.0 // indirect
|
||||
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
|
||||
github.com/prometheus/client_model v0.6.1 // indirect
|
||||
github.com/prometheus/common v0.59.1 // indirect
|
||||
github.com/prometheus/procfs v0.15.1 // indirect
|
||||
github.com/prometheus/prom2json v1.4.0 // indirect
|
||||
github.com/prometheus/prometheus v0.54.1 // indirect
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
github.com/safchain/ethtool v0.4.1 // indirect
|
||||
github.com/secure-io/sio-go v0.3.1 // indirect
|
||||
github.com/shirou/gopsutil/v3 v3.24.5 // indirect
|
||||
github.com/shoenig/go-m1cpu v0.1.6 // indirect
|
||||
github.com/tinylib/msgp v1.3.0 // indirect
|
||||
github.com/tklauser/go-sysconf v0.3.14 // indirect
|
||||
github.com/tklauser/numcpus v0.8.0 // indirect
|
||||
github.com/yusufpapurcu/wmi v1.2.4 // indirect
|
||||
golang.org/x/net v0.43.0 // indirect
|
||||
golang.org/x/sync v0.17.0 // indirect
|
||||
golang.org/x/sys v0.36.0 // indirect
|
||||
golang.org/x/text v0.29.0 // indirect
|
||||
google.golang.org/protobuf v1.34.2 // indirect
|
||||
)
|
||||
@@ -0,0 +1,128 @@
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
|
||||
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
|
||||
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
|
||||
github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE=
|
||||
github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78=
|
||||
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
|
||||
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg=
|
||||
github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
|
||||
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.7.6 h1:rWQc5FwZSPX58r1OQmkuaNicxdmExaEz5A2DO2hUuTk=
|
||||
github.com/jackc/pgx/v5 v5.7.6/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
|
||||
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
|
||||
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/lufia/plan9stats v0.0.0-20240909124753-873cd0166683 h1:7UMa6KCCMjZEMDtTVdcGu0B1GmmC7QJKiCCjyTAWQy0=
|
||||
github.com/lufia/plan9stats v0.0.0-20240909124753-873cd0166683/go.mod h1:ilwx/Dta8jXAgpFYFvSWEMwxmbWXyiUHkd5FwyKhb5k=
|
||||
github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo=
|
||||
github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
|
||||
github.com/minio/crc64nvme v1.0.2 h1:6uO1UxGAD+kwqWWp7mBFsi5gAse66C4NXO8cmcVculg=
|
||||
github.com/minio/crc64nvme v1.0.2/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
|
||||
github.com/minio/madmin-go/v3 v3.0.95 h1:KRy4NIpTLKrHcuTQbKrwKs3SvGoe+ZAHT2Pbl1zJxDc=
|
||||
github.com/minio/madmin-go/v3 v3.0.95/go.mod h1:pMLdj9OtN0CANNs5tdm6opvOlDFfj0WhbztboZAjRWE=
|
||||
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
|
||||
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
|
||||
github.com/minio/minio-go/v7 v7.0.95 h1:ywOUPg+PebTMTzn9VDsoFJy32ZuARN9zhB+K3IYEvYU=
|
||||
github.com/minio/minio-go/v7 v7.0.95/go.mod h1:wOOX3uxS334vImCNRVyIDdXX9OsXDm89ToynKgqUKlo=
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
|
||||
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
|
||||
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU=
|
||||
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
|
||||
github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
|
||||
github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
|
||||
github.com/prometheus/common v0.59.1 h1:LXb1quJHWm1P6wq/U824uxYi4Sg0oGvNeUm1z5dJoX0=
|
||||
github.com/prometheus/common v0.59.1/go.mod h1:GpWM7dewqmVYcd7SmRaiWVe9SSqjf0UrwnYnpEZNuT0=
|
||||
github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
|
||||
github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
|
||||
github.com/prometheus/prom2json v1.4.0 h1:2AEOsd1ebqql/p9u0IWgCpUAteAAf9Lnf/SVyieqer4=
|
||||
github.com/prometheus/prom2json v1.4.0/go.mod h1:DmcIMPspQD/fMyFCYti5qJJbuEnqDh3DGoooO0sgr4w=
|
||||
github.com/prometheus/prometheus v0.54.1 h1:vKuwQNjnYN2/mDoWfHXDhAsz/68q/dQDb+YbcEqU7MQ=
|
||||
github.com/prometheus/prometheus v0.54.1/go.mod h1:xlLByHhk2g3ycakQGrMaU8K7OySZx98BzeCR99991NY=
|
||||
github.com/redis/go-redis/v9 v9.14.0 h1:u4tNCjXOyzfgeLN+vAZaW1xUooqWDqVEsZN0U01jfAE=
|
||||
github.com/redis/go-redis/v9 v9.14.0/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw=
|
||||
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/safchain/ethtool v0.4.1 h1:S6mEleTADqgynileXoiapt/nKnatyR6bmIHoF+h2ADo=
|
||||
github.com/safchain/ethtool v0.4.1/go.mod h1:XLLnZmy4OCRTkksP/UiMjij96YmIsBfmBQcs7H6tA48=
|
||||
github.com/secure-io/sio-go v0.3.1 h1:dNvY9awjabXTYGsTF1PiCySl9Ltofk9GA3VdWlo7rRc=
|
||||
github.com/secure-io/sio-go v0.3.1/go.mod h1:+xbkjDzPjwh4Axd07pRKSNriS9SCiYksWnZqdnfpQxs=
|
||||
github.com/shirou/gopsutil/v3 v3.24.5 h1:i0t8kL+kQTvpAYToeuiVk3TgDeKOFioZO3Ztz/iZ9pI=
|
||||
github.com/shirou/gopsutil/v3 v3.24.5/go.mod h1:bsoOS1aStSs9ErQ1WWfxllSeS1K5D+U30r2NfcubMVk=
|
||||
github.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFtM=
|
||||
github.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ=
|
||||
github.com/shoenig/test v0.6.4 h1:kVTaSd7WLz5WZ2IaoM0RSzRsUD+m8wRR+5qvntpn4LU=
|
||||
github.com/shoenig/test v0.6.4/go.mod h1:byHiCGXqrVaflBLAMq/srcZIHynQPQgeyvkvXnjqq0k=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
|
||||
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||
github.com/tinylib/msgp v1.3.0 h1:ULuf7GPooDaIlbyvgAxBV/FI7ynli6LZ1/nVUNu+0ww=
|
||||
github.com/tinylib/msgp v1.3.0/go.mod h1:ykjzy2wzgrlvpDCRc4LA8UXy6D8bzMSuAF3WD57Gok0=
|
||||
github.com/tklauser/go-sysconf v0.3.14 h1:g5vzr9iPFFz24v2KZXs/pvpvh8/V9Fw6vQK5ZZb78yU=
|
||||
github.com/tklauser/go-sysconf v0.3.14/go.mod h1:1ym4lWMLUOhuBOPGtRcJm7tEGX4SCYNEEEtghGG/8uY=
|
||||
github.com/tklauser/numcpus v0.8.0 h1:Mx4Wwe/FjZLeQsK/6kt2EOepwwSl7SmJrK5bV/dXYgY=
|
||||
github.com/tklauser/numcpus v0.8.0/go.mod h1:ZJZlAY+dmR4eut8epnzf0u/VwodKmryxR8txiloSqBE=
|
||||
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
|
||||
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.42.0 h1:chiH31gIWm57EkTXpwnqf8qeuMUi0yekh6mT2AvFlqI=
|
||||
golang.org/x/crypto v0.42.0/go.mod h1:4+rDnOTJhQCx2q7/j6rAN5XDw8kPjeaXEUR2eL94ix8=
|
||||
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||
golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
|
||||
golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
|
||||
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
|
||||
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k=
|
||||
golang.org/x/sys v0.36.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
|
||||
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
|
||||
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
|
||||
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,16 @@
|
||||
# Build MinIO from a pinned commit of the official upstream source.
|
||||
FROM golang:1.26.8-alpine3.23@sha256:a8fa79c5bd40d880b52bd3b6d7669ecdcfd00e85facdd427d279efb5ddd79cb1 AS build
|
||||
ADD --checksum=sha256:be6d0bd3696c3a13a35f02d3a0280b64319c67918b4501c5c3d87f96d000085c https://codeload.github.com/minio/minio/tar.gz/refs/tags/RELEASE.2025-10-15T17-29-55Z /source.tar.gz
|
||||
WORKDIR /src
|
||||
RUN tar -xzf /source.tar.gz --strip-components=1 -C /src
|
||||
RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build \
|
||||
CGO_ENABLED=0 GOMAXPROCS=2 go build -p 2 -trimpath -buildvcs=false \
|
||||
-ldflags='-s -w -X github.com/minio/minio/cmd.Version=2025-10-15T17:29:55Z -X github.com/minio/minio/cmd.ReleaseTag=RELEASE.2025-10-15T17-29-55Z -X github.com/minio/minio/cmd.CommitID=9e49d5e7a648f00e26f2246f4dc28e6b07f8c84a -X github.com/minio/minio/cmd.CopyrightYear=2025' \
|
||||
-o /minio .
|
||||
|
||||
FROM busybox:1.37.0-musl
|
||||
COPY --from=build /minio /minio
|
||||
COPY --from=build /src/LICENSE /licenses/MinIO-LICENSE
|
||||
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
|
||||
RUN mkdir -p /data
|
||||
ENTRYPOINT ["/minio"]
|
||||
@@ -0,0 +1,43 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
upstream=/run/nginx-upstream.conf
|
||||
|
||||
write_servers() {
|
||||
if [ "$1" = 1 ]; then
|
||||
printf '%s\n' 'server api1:8080 resolve max_fails=0;'
|
||||
fi
|
||||
if [ "$2" = 1 ]; then
|
||||
printf '%s\n' 'server api2:8080 resolve max_fails=0;'
|
||||
fi
|
||||
}
|
||||
|
||||
# With no healthy replica, keep both reachable so API can return its normal
|
||||
# 401/503 JSON responses instead of Nginx reporting "no live upstreams".
|
||||
write_servers 1 1 > "$upstream"
|
||||
|
||||
monitor() {
|
||||
sleep 2
|
||||
while :; do
|
||||
ready1=0
|
||||
ready2=0
|
||||
wget -q -T 2 -O /dev/null http://api1:8080/health/ready && ready1=1 || true
|
||||
wget -q -T 2 -O /dev/null http://api2:8080/health/ready && ready2=1 || true
|
||||
if [ "$ready1" = 0 ] && [ "$ready2" = 0 ]; then
|
||||
ready1=1
|
||||
ready2=1
|
||||
fi
|
||||
next=$(mktemp /run/nginx-upstream.XXXXXX)
|
||||
write_servers "$ready1" "$ready2" > "$next"
|
||||
if cmp -s "$next" "$upstream"; then
|
||||
rm -f "$next"
|
||||
else
|
||||
mv "$next" "$upstream"
|
||||
nginx -t >/dev/null 2>&1 && nginx -s reload
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
monitor &
|
||||
exec nginx -g 'daemon off;'
|
||||
@@ -0,0 +1,35 @@
|
||||
events {}
|
||||
|
||||
http {
|
||||
access_log /dev/stdout;
|
||||
error_log /dev/stderr notice;
|
||||
|
||||
resolver 127.0.0.11 valid=2s ipv6=off;
|
||||
|
||||
upstream messenger_api {
|
||||
zone messenger_api 64k;
|
||||
include /run/nginx-upstream.conf;
|
||||
keepalive 16;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
client_max_body_size 0;
|
||||
client_body_timeout 20s;
|
||||
|
||||
location / {
|
||||
proxy_request_buffering off;
|
||||
proxy_buffering off;
|
||||
proxy_pass http://messenger_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_connect_timeout 500ms;
|
||||
proxy_send_timeout 65s;
|
||||
proxy_read_timeout 65s;
|
||||
proxy_next_upstream error timeout http_502 http_503 http_504;
|
||||
proxy_next_upstream_tries 2;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"embed"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
//go:embed migrations/*.sql
|
||||
var migrations embed.FS
|
||||
|
||||
func Migrate(ctx context.Context, pool *pgxpool.Pool) error {
|
||||
tx, err := pool.BeginTx(ctx, pgx.TxOptions{})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
if _, err = tx.Exec(ctx, `SELECT pg_advisory_xact_lock(202602)`); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = tx.Exec(ctx, `CREATE TABLE IF NOT EXISTS schema_migrations (version integer PRIMARY KEY, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())`); err != nil {
|
||||
return err
|
||||
}
|
||||
entries, err := fs.ReadDir(migrations, "migrations")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sort.Slice(entries, func(i, j int) bool { return entries[i].Name() < entries[j].Name() })
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".sql") {
|
||||
continue
|
||||
}
|
||||
part := strings.SplitN(entry.Name(), "_", 2)[0]
|
||||
version, parseErr := strconv.Atoi(part)
|
||||
if parseErr != nil {
|
||||
return fmt.Errorf("invalid migration name %s", entry.Name())
|
||||
}
|
||||
var applied bool
|
||||
if err = tx.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM schema_migrations WHERE version=$1)`, version).Scan(&applied); err != nil {
|
||||
return err
|
||||
}
|
||||
if applied {
|
||||
continue
|
||||
}
|
||||
sqlBytes, readErr := migrations.ReadFile("migrations/" + entry.Name())
|
||||
if readErr != nil {
|
||||
return readErr
|
||||
}
|
||||
if _, err = tx.Exec(ctx, string(sqlBytes)); err != nil {
|
||||
return fmt.Errorf("migration %s: %w", entry.Name(), err)
|
||||
}
|
||||
if _, err = tx.Exec(ctx, `INSERT INTO schema_migrations(version) VALUES($1)`, version); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name text NOT NULL CHECK (char_length(btrim(display_name)) > 0 AND char_length(display_name) <= 100),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL CHECK (char_length(btrim(title)) > 0 AND char_length(title) <= 100),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE INDEX chat_members_user_chat_idx ON chat_members (user_id, chat_id);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
|
||||
text text NOT NULL CHECK (char_length(btrim(text)) > 0 AND char_length(text) <= 4000),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE INDEX messages_chat_history_idx ON messages (chat_id, created_at, id);
|
||||
@@ -0,0 +1,13 @@
|
||||
CREATE TABLE message_idempotency (
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
method text NOT NULL,
|
||||
path text NOT NULL,
|
||||
idempotency_key text NOT NULL,
|
||||
request_hash bytea NOT NULL,
|
||||
message_id uuid REFERENCES messages(id) ON DELETE CASCADE,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
PRIMARY KEY (user_id, method, path, idempotency_key)
|
||||
);
|
||||
|
||||
CREATE INDEX message_idempotency_created_at_idx
|
||||
ON message_idempotency (created_at);
|
||||
@@ -0,0 +1,30 @@
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
owner_id uuid NOT NULL REFERENCES users(id),
|
||||
filename text NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
content_type text NOT NULL CHECK (content_type IN ('image/png','image/jpeg')),
|
||||
size_bytes bigint NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 text NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
|
||||
status text NOT NULL DEFAULT 'ready' CHECK (status='ready'),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX attachments_chat_idx ON attachments(chat_id,created_at,id);
|
||||
|
||||
-- Durable intent is committed BEFORE any S3 PUT. A crashed upload remains
|
||||
-- visible to the cleanup process even if its ready metadata never committed.
|
||||
CREATE TABLE attachment_uploads (
|
||||
id uuid PRIMARY KEY,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX attachment_uploads_created_idx ON attachment_uploads(created_at);
|
||||
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_text_check CHECK (char_length(text)<=4000);
|
||||
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_body_check
|
||||
CHECK (char_length(btrim(text))>0 OR cardinality(attachment_ids)>0);
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_attachments_limit CHECK (cardinality(attachment_ids)<=10);
|
||||
@@ -0,0 +1,223 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"image"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
"io"
|
||||
"mime"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
const MaxUploadBytes int64 = 10 * 1024 * 1024
|
||||
const MaxImagePixels int64 = 40000000
|
||||
|
||||
type Attachment struct {
|
||||
ID string `json:"id"`
|
||||
ChatID string `json:"chat_id"`
|
||||
OwnerID string `json:"owner_id"`
|
||||
Filename string `json:"filename"`
|
||||
ContentType string `json:"content_type"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Status string `json:"status"`
|
||||
Variants []any `json:"variants"`
|
||||
ErrorCode *string `json:"error_code"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
}
|
||||
|
||||
func tooLarge() *apiError {
|
||||
return &apiError{status: 413, code: "payload_too_large", message: "Превышен лимит изображения"}
|
||||
}
|
||||
func invalidImage() *apiError {
|
||||
return &apiError{status: 415, code: "unsupported_media_type", message: "Требуется корректный PNG или JPEG"}
|
||||
}
|
||||
|
||||
func rollbackUpload(tx pgx.Tx) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
_ = tx.Rollback(ctx)
|
||||
}
|
||||
|
||||
func validFilename(s string) bool {
|
||||
if !utf8.ValidString(s) || strings.TrimSpace(s) == "" || utf8.RuneCountInString(s) > 255 {
|
||||
return false
|
||||
}
|
||||
for _, c := range s {
|
||||
if c == '/' || c == '\\' || unicode.IsControl(c) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (a *API) isMember(r *http.Request, uid, chatID string) *apiError {
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
var member bool
|
||||
if err := a.db.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM chat_members WHERE chat_id=$1 AND user_id=$2)`, chatID, uid).Scan(&member); err != nil {
|
||||
return unavailable()
|
||||
}
|
||||
if !member {
|
||||
return missing()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *API) uploadAttachment(r *http.Request, uid, chatID string) (response, *apiError) {
|
||||
if e := a.isMember(r, uid, chatID); e != nil {
|
||||
return response{}, e
|
||||
}
|
||||
query, err := url.ParseQuery(r.URL.RawQuery)
|
||||
if err != nil || len(query) != 1 || len(query["filename"]) != 1 || !validFilename(query.Get("filename")) {
|
||||
return response{}, bad("Неверное имя файла")
|
||||
}
|
||||
kind, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
|
||||
if err != nil || (kind != "image/png" && kind != "image/jpeg") {
|
||||
return response{}, invalidImage()
|
||||
}
|
||||
if r.ContentLength > MaxUploadBytes {
|
||||
return response{}, tooLarge()
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 45*time.Second)
|
||||
defer cancel()
|
||||
// One decoder/temporary original per replica bounds RAM and tmpfs usage.
|
||||
select {
|
||||
case a.uploadSlots <- struct{}{}:
|
||||
defer func() { <-a.uploadSlots }()
|
||||
case <-ctx.Done():
|
||||
return response{}, unavailable()
|
||||
}
|
||||
file, err := os.CreateTemp("", "upload-*")
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
defer os.Remove(file.Name())
|
||||
defer file.Close()
|
||||
hash := sha256.New()
|
||||
size, err := io.CopyBuffer(io.MultiWriter(file, hash), io.LimitReader(r.Body, MaxUploadBytes+1), make([]byte, 32*1024))
|
||||
if size > MaxUploadBytes {
|
||||
return response{}, tooLarge()
|
||||
}
|
||||
if err != nil {
|
||||
return response{}, bad("Загрузка прервана")
|
||||
}
|
||||
if size == 0 {
|
||||
return response{}, invalidImage()
|
||||
}
|
||||
file.Seek(0, io.SeekStart)
|
||||
config, format, err := image.DecodeConfig(file)
|
||||
if err != nil || (format != "png" && format != "jpeg") || "image/"+format != kind {
|
||||
return response{}, invalidImage()
|
||||
}
|
||||
if config.Width < 1 || config.Height < 1 || int64(config.Width) > MaxImagePixels/int64(config.Height) {
|
||||
return response{}, tooLarge()
|
||||
}
|
||||
file.Seek(0, io.SeekStart)
|
||||
// Decode after the dimension guard to reject corrupt payloads, not just
|
||||
// apparently valid headers. The decoded bitmap is never persisted.
|
||||
if e := validateImage(file); e != nil {
|
||||
return response{}, e
|
||||
}
|
||||
file.Seek(0, io.SeekStart)
|
||||
id := newID()
|
||||
key := "originals/" + id
|
||||
dbctx, dbcancel := a.dbContext(ctx)
|
||||
_, err = a.db.Exec(dbctx, `INSERT INTO attachment_uploads(id,object_key) VALUES($1,$2)`, id, key)
|
||||
dbcancel()
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
tx, err := a.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
defer rollbackUpload(tx)
|
||||
// Cleanup and upload use the same lock. Cleanup cannot delete an object
|
||||
// while its upload/metadata transaction is in progress.
|
||||
if _, err = tx.Exec(ctx, `SELECT pg_advisory_xact_lock(hashtextextended($1,4))`, key); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
var pending bool
|
||||
if err = tx.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM attachment_uploads WHERE id=$1)`, id).Scan(&pending); err != nil || !pending {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
putctx, putcancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
err = a.objects.Put(putctx, key, file, size, kind)
|
||||
putcancel()
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
att := Attachment{ID: id, ChatID: chatID, OwnerID: uid, Filename: query.Get("filename"), ContentType: kind, SizeBytes: size, SHA256: hex.EncodeToString(hash.Sum(nil)), Status: "ready", Variants: []any{}}
|
||||
var created, updated time.Time
|
||||
if err = tx.QueryRow(ctx, `INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256) VALUES($1,$2,$3,$4,$5,$6,$7,$8) RETURNING created_at,updated_at`, id, chatID, uid, att.Filename, key, kind, size, att.SHA256).Scan(&created, &updated); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if _, err = tx.Exec(ctx, `DELETE FROM attachment_uploads WHERE id=$1`, id); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
att.CreatedAt = formatTime(created)
|
||||
att.UpdatedAt = formatTime(updated)
|
||||
return response{status: 201, body: att}, nil
|
||||
}
|
||||
|
||||
func validateImage(r io.Reader) *apiError {
|
||||
_, _, err := image.Decode(r)
|
||||
if err != nil {
|
||||
return invalidImage()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *API) getAttachment(r *http.Request, uid, id string, content bool) (response, *apiError) {
|
||||
query, err := url.ParseQuery(r.URL.RawQuery)
|
||||
if err != nil {
|
||||
return response{}, bad("Неверные параметры вложения")
|
||||
}
|
||||
if (!content && len(query) > 0) || (content && (len(query) > 1 || (len(query) > 0 && len(query["variant"]) != 1))) {
|
||||
return response{}, bad("Неверные параметры вложения")
|
||||
}
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
att := Attachment{Variants: []any{}}
|
||||
var key string
|
||||
var created, updated time.Time
|
||||
err = a.db.QueryRow(ctx, `SELECT a.id::text,a.chat_id::text,a.owner_id::text,a.filename,a.object_key,a.content_type,a.size_bytes,a.sha256,a.status,a.created_at,a.updated_at FROM attachments a JOIN chat_members cm ON cm.chat_id=a.chat_id AND cm.user_id=$2 WHERE a.id=$1`, id, uid).Scan(&att.ID, &att.ChatID, &att.OwnerID, &att.Filename, &key, &att.ContentType, &att.SizeBytes, &att.SHA256, &att.Status, &created, &updated)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return response{}, missing()
|
||||
}
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
att.CreatedAt = formatTime(created)
|
||||
att.UpdatedAt = formatTime(updated)
|
||||
if !content {
|
||||
return response{status: 200, body: att}, nil
|
||||
}
|
||||
if variant, ok := query["variant"]; ok && variant[0] != "original" {
|
||||
return response{}, missing()
|
||||
}
|
||||
downloadCtx, downloadCancel := context.WithTimeout(r.Context(), 45*time.Second)
|
||||
object, err := a.objects.Get(downloadCtx, key)
|
||||
if err != nil {
|
||||
downloadCancel()
|
||||
return response{}, unavailable()
|
||||
}
|
||||
return response{status: 200, content: object, contentType: att.ContentType, length: att.SizeBytes, filename: att.Filename, cancel: downloadCancel}, nil
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/minio/minio-go/v7"
|
||||
"mtusi-messenger/internal/objectstore"
|
||||
)
|
||||
|
||||
type CleanupResult struct {
|
||||
RemovedObjects int `json:"removed_objects"`
|
||||
RemovedIntents int64 `json:"removed_intents"`
|
||||
KeptReady int `json:"kept_ready"`
|
||||
SkippedRecent int `json:"skipped_recent"`
|
||||
SkippedBusy int `json:"skipped_busy"`
|
||||
}
|
||||
|
||||
// Cleanup is deliberately small: original objects only, no metadata repair or
|
||||
// lab5 worker. A durable intent and an S3 sweep also cover late PUT completion.
|
||||
func Cleanup(ctx context.Context, db *pgxpool.Pool, store *objectstore.Store, grace time.Duration) (CleanupResult, error) {
|
||||
result := CleanupResult{}
|
||||
cutoff := time.Now().Add(-grace)
|
||||
clean := func(key string) error {
|
||||
if !strings.HasPrefix(key, "originals/") {
|
||||
return nil
|
||||
}
|
||||
tx, err := db.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer rollbackUpload(tx)
|
||||
var locked bool
|
||||
if err = tx.QueryRow(ctx, `SELECT pg_try_advisory_xact_lock(hashtextextended($1,4))`, key).Scan(&locked); err != nil {
|
||||
return err
|
||||
}
|
||||
if !locked {
|
||||
result.SkippedBusy++
|
||||
return nil
|
||||
}
|
||||
var ready, recent bool
|
||||
if err = tx.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM attachments WHERE object_key=$1),EXISTS(SELECT 1 FROM attachment_uploads WHERE object_key=$1 AND created_at>$2)`, key, cutoff).Scan(&ready, &recent); err != nil {
|
||||
return err
|
||||
}
|
||||
if ready {
|
||||
result.KeptReady++
|
||||
return nil
|
||||
}
|
||||
if recent {
|
||||
result.SkippedRecent++
|
||||
return nil
|
||||
}
|
||||
info, err := store.Client.StatObject(ctx, store.Bucket, key, minio.StatObjectOptions{})
|
||||
exists := err == nil
|
||||
if err != nil && minio.ToErrorResponse(err).Code != "NoSuchKey" {
|
||||
return err
|
||||
}
|
||||
if exists && info.LastModified.After(cutoff) {
|
||||
result.SkippedRecent++
|
||||
return nil
|
||||
}
|
||||
if exists {
|
||||
if err = store.Delete(ctx, key); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
command, err := tx.Exec(ctx, `DELETE FROM attachment_uploads WHERE object_key=$1`, key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
result.RemovedIntents += command.RowsAffected()
|
||||
if exists {
|
||||
result.RemovedObjects++
|
||||
}
|
||||
return nil
|
||||
}
|
||||
rows, err := db.Query(ctx, `SELECT object_key FROM attachment_uploads WHERE created_at<$1 ORDER BY created_at LIMIT 1000`, cutoff)
|
||||
if err != nil {
|
||||
return result, err
|
||||
}
|
||||
keys := []string{}
|
||||
for rows.Next() {
|
||||
var key string
|
||||
if err = rows.Scan(&key); err != nil {
|
||||
rows.Close()
|
||||
return result, err
|
||||
}
|
||||
keys = append(keys, key)
|
||||
}
|
||||
err = rows.Err()
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
return result, err
|
||||
}
|
||||
for _, key := range keys {
|
||||
if err = clean(key); err != nil {
|
||||
return result, err
|
||||
}
|
||||
}
|
||||
listctx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
for object := range store.Client.ListObjects(listctx, store.Bucket, minio.ListObjectsOptions{Prefix: "originals/", Recursive: true}) {
|
||||
if object.Err != nil {
|
||||
return result, fmt.Errorf("S3 cleanup listing: %w", object.Err)
|
||||
}
|
||||
if object.LastModified.Before(cutoff) {
|
||||
if err = clean(object.Key); err != nil {
|
||||
return result, err
|
||||
}
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/crypto/scrypt"
|
||||
)
|
||||
|
||||
const scryptN, scryptR, scryptP = 32768, 8, 1
|
||||
|
||||
func hashPassword(password string) (string, error) {
|
||||
salt := make([]byte, 16)
|
||||
if _, err := rand.Read(salt); err != nil {
|
||||
return "", err
|
||||
}
|
||||
hash, err := scrypt.Key([]byte(password), salt, scryptN, scryptR, scryptP, 32)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
b64 := base64.RawStdEncoding
|
||||
return fmt.Sprintf("scrypt$%d$%d$%d$%s$%s", scryptN, scryptR, scryptP, b64.EncodeToString(salt), b64.EncodeToString(hash)), nil
|
||||
}
|
||||
|
||||
func verifyPassword(encoded, password string) bool {
|
||||
parts := strings.Split(encoded, "$")
|
||||
if len(parts) != 6 || parts[0] != "scrypt" {
|
||||
return false
|
||||
}
|
||||
n, e1 := strconv.Atoi(parts[1])
|
||||
r, e2 := strconv.Atoi(parts[2])
|
||||
p, e3 := strconv.Atoi(parts[3])
|
||||
salt, e4 := base64.RawStdEncoding.DecodeString(parts[4])
|
||||
expected, e5 := base64.RawStdEncoding.DecodeString(parts[5])
|
||||
if e1 != nil || e2 != nil || e3 != nil || e4 != nil || e5 != nil || len(expected) != 32 {
|
||||
return false
|
||||
}
|
||||
actual, err := scrypt.Key([]byte(password), salt, n, r, p, len(expected))
|
||||
return err == nil && subtle.ConstantTimeCompare(actual, expected) == 1
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPasswordHash(t *testing.T) {
|
||||
const password = "long-test-password"
|
||||
first, err := hashPassword(password)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second, err := hashPassword(password)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if first == password || strings.Contains(first, password) {
|
||||
t.Fatal("hash contains the original password")
|
||||
}
|
||||
if first == second {
|
||||
t.Fatal("independent hashes must use different salts")
|
||||
}
|
||||
if !verifyPassword(first, password) {
|
||||
t.Fatal("correct password was rejected")
|
||||
}
|
||||
if verifyPassword(first, "wrong-test-password") {
|
||||
t.Fatal("wrong password was accepted")
|
||||
}
|
||||
if verifyPassword("damaged-hash", password) {
|
||||
t.Fatal("malformed hash was accepted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,754 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"mime"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"mtusi-messenger/internal/objectstore"
|
||||
)
|
||||
|
||||
type User struct {
|
||||
ID string `json:"id"`
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
type Chat struct {
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
MemberIDs []string `json:"member_ids"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
type Message struct {
|
||||
ID string `json:"id"`
|
||||
ChatID string `json:"chat_id"`
|
||||
SenderID string `json:"sender_id"`
|
||||
Text string `json:"text"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
AttachmentIDs []string `json:"attachment_ids"`
|
||||
}
|
||||
type Session struct {
|
||||
Token string `json:"token"`
|
||||
TokenType string `json:"token_type"`
|
||||
ExpiresAt string `json:"expires_at"`
|
||||
User User `json:"user"`
|
||||
}
|
||||
|
||||
type API struct {
|
||||
objects *objectstore.Store
|
||||
uploadSlots chan struct{}
|
||||
db *pgxpool.Pool
|
||||
sessions *redis.Client
|
||||
ttl time.Duration
|
||||
idempotencyTTL time.Duration
|
||||
dbTimeout time.Duration
|
||||
instanceID string
|
||||
logger *log.Logger
|
||||
}
|
||||
|
||||
func NewHandler(db *pgxpool.Pool, sessions *redis.Client, objects *objectstore.Store, ttl, idempotencyTTL, dbTimeout time.Duration, instanceID string, logger *log.Logger) http.Handler {
|
||||
return &API{db: db, sessions: sessions, objects: objects, uploadSlots: make(chan struct{}, 1), ttl: ttl, idempotencyTTL: idempotencyTTL, dbTimeout: dbTimeout, instanceID: instanceID, logger: logger}
|
||||
}
|
||||
|
||||
var uuidRE = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||
var usernameRE = regexp.MustCompile(`^[a-z0-9_]{3,32}$`)
|
||||
var idempotencyKeyRE = regexp.MustCompile(`^[A-Za-z0-9._:-]{1,128}$`)
|
||||
|
||||
type apiError struct {
|
||||
status int
|
||||
code, message, challenge string
|
||||
}
|
||||
type response struct {
|
||||
status int
|
||||
body any
|
||||
noBody bool
|
||||
content io.ReadCloser
|
||||
contentType string
|
||||
length int64
|
||||
filename string
|
||||
cancel context.CancelFunc
|
||||
}
|
||||
|
||||
func bad(msg string) *apiError { return &apiError{status: 400, code: "invalid_request", message: msg} }
|
||||
func missing() *apiError {
|
||||
return &apiError{status: 404, code: "not_found", message: "Объект не найден"}
|
||||
}
|
||||
func unavailable() *apiError {
|
||||
return &apiError{status: 503, code: "unavailable", message: "Сервис временно недоступен"}
|
||||
}
|
||||
func unauthorized(scheme string) *apiError {
|
||||
challenge := `Bearer realm="messenger"`
|
||||
if scheme == "Basic" {
|
||||
challenge = `Basic realm="messenger", charset="UTF-8"`
|
||||
}
|
||||
return &apiError{status: 401, code: "unauthorized", message: "Неверные учётные данные", challenge: challenge}
|
||||
}
|
||||
|
||||
func (a *API) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
id := newID()
|
||||
w.Header().Set("X-Request-Id", id)
|
||||
w.Header().Set("X-Instance-Id", a.instanceID)
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
start, status := time.Now(), 500
|
||||
defer func() {
|
||||
if x := recover(); x != nil {
|
||||
status = 500
|
||||
a.logger.Printf("request_id=%s panic=%T", id, x)
|
||||
writeError(w, id, &apiError{status: 500, code: "internal_error", message: "Внутренняя ошибка"})
|
||||
}
|
||||
a.logger.Printf("request_id=%s method=%s path=%s status=%d duration_ms=%d", id, r.Method, r.URL.Path, status, time.Since(start).Milliseconds())
|
||||
}()
|
||||
res, apiErr := a.route(r)
|
||||
if apiErr != nil {
|
||||
status = apiErr.status
|
||||
writeError(w, id, apiErr)
|
||||
return
|
||||
}
|
||||
status = res.status
|
||||
if res.content != nil {
|
||||
if res.cancel != nil {
|
||||
defer res.cancel()
|
||||
}
|
||||
defer res.content.Close()
|
||||
w.Header().Set("Content-Type", res.contentType)
|
||||
w.Header().Set("Content-Length", strconv.FormatInt(res.length, 10))
|
||||
w.Header().Set("Content-Disposition", mime.FormatMediaType("inline", map[string]string{"filename": res.filename}))
|
||||
w.Header().Set("Cache-Control", "private, no-store")
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.WriteHeader(status)
|
||||
if _, err := io.Copy(w, res.content); err != nil {
|
||||
a.logger.Printf("request_id=%s download interrupted", id)
|
||||
}
|
||||
return
|
||||
}
|
||||
w.WriteHeader(status)
|
||||
if !res.noBody {
|
||||
_ = json.NewEncoder(w).Encode(res.body)
|
||||
}
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, requestID string, e *apiError) {
|
||||
if e.challenge != "" {
|
||||
w.Header().Set("WWW-Authenticate", e.challenge)
|
||||
}
|
||||
w.WriteHeader(e.status)
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"error": map[string]string{"code": e.code, "message": e.message, "request_id": requestID}})
|
||||
}
|
||||
|
||||
func decode(r *http.Request, dst any) *apiError {
|
||||
if strings.ToLower(strings.TrimSpace(strings.Split(r.Header.Get("Content-Type"), ";")[0])) != "application/json" {
|
||||
return &apiError{status: 415, code: "unsupported_media_type", message: "Требуется application/json"}
|
||||
}
|
||||
if r.ContentLength > 1<<20 {
|
||||
return &apiError{status: 413, code: "payload_too_large", message: "Слишком большой запрос"}
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(r.Body, (1<<20)+1))
|
||||
if err != nil {
|
||||
return bad("Не удалось прочитать JSON")
|
||||
}
|
||||
if len(data) > 1<<20 {
|
||||
return &apiError{status: 413, code: "payload_too_large", message: "Слишком большой запрос"}
|
||||
}
|
||||
if !utf8.Valid(data) {
|
||||
return bad("JSON должен быть в UTF-8")
|
||||
}
|
||||
var fields map[string]json.RawMessage
|
||||
if json.Unmarshal(data, &fields) == nil {
|
||||
for _, value := range fields {
|
||||
if bytes.Equal(bytes.TrimSpace(value), []byte("null")) {
|
||||
return bad("Поля запроса не должны быть null")
|
||||
}
|
||||
}
|
||||
}
|
||||
d := json.NewDecoder(bytes.NewReader(data))
|
||||
d.DisallowUnknownFields()
|
||||
if err = d.Decode(dst); err != nil {
|
||||
return bad("Неверный JSON или поля")
|
||||
}
|
||||
var rest any
|
||||
if !errors.Is(d.Decode(&rest), io.EOF) {
|
||||
return bad("Ожидался один JSON-объект")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validText(s string, max int) bool {
|
||||
return utf8.ValidString(s) && strings.TrimSpace(s) != "" && utf8.RuneCountInString(s) <= max
|
||||
}
|
||||
func validPassword(s string) bool {
|
||||
n := utf8.RuneCountInString(s)
|
||||
return utf8.ValidString(s) && n >= 12 && n <= 128
|
||||
}
|
||||
func formatTime(t time.Time) string { return t.UTC().Format(time.RFC3339Nano) }
|
||||
func newID() string {
|
||||
var b [16]byte
|
||||
if _, err := rand.Read(b[:]); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
b[6] = (b[6] & 0x0f) | 0x40
|
||||
b[8] = (b[8] & 0x3f) | 0x80
|
||||
return fmt.Sprintf("%x-%x-%x-%x-%x", b[:4], b[4:6], b[6:8], b[8:10], b[10:])
|
||||
}
|
||||
func newToken() string {
|
||||
b := make([]byte, 32)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return base64.RawURLEncoding.EncodeToString(b)
|
||||
}
|
||||
func tokenKey(token string) string {
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
return "session:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
func cursorFor(resource string, created time.Time, id string) string {
|
||||
b, _ := json.Marshal([]string{resource, formatTime(created), id})
|
||||
return base64.RawURLEncoding.EncodeToString(b)
|
||||
}
|
||||
func parsePage(r *http.Request, resource string) (int, *time.Time, string, *apiError) {
|
||||
q, err := url.ParseQuery(r.URL.RawQuery)
|
||||
if err != nil {
|
||||
return 0, nil, "", bad("Неверные параметры страницы")
|
||||
}
|
||||
for k, v := range q {
|
||||
if (k != "limit" && k != "cursor") || len(v) != 1 {
|
||||
return 0, nil, "", bad("Неверные параметры страницы")
|
||||
}
|
||||
}
|
||||
limit := 50
|
||||
if raw := q.Get("limit"); raw != "" {
|
||||
n, e := strconv.Atoi(raw)
|
||||
if e != nil || n < 1 || n > 100 {
|
||||
return 0, nil, "", bad("limit должен быть от 1 до 100")
|
||||
}
|
||||
limit = n
|
||||
} else if _, ok := q["limit"]; ok {
|
||||
return 0, nil, "", bad("Пустой limit")
|
||||
}
|
||||
if _, ok := q["cursor"]; !ok {
|
||||
return limit, nil, "", nil
|
||||
}
|
||||
raw := q.Get("cursor")
|
||||
if raw == "" || len(raw) > 1024 {
|
||||
return 0, nil, "", bad("Неверный cursor")
|
||||
}
|
||||
b, e := base64.RawURLEncoding.DecodeString(raw)
|
||||
if e != nil {
|
||||
return 0, nil, "", bad("Неверный cursor")
|
||||
}
|
||||
var p []string
|
||||
if json.Unmarshal(b, &p) != nil || len(p) != 3 || p[0] != resource || !uuidRE.MatchString(p[2]) {
|
||||
return 0, nil, "", bad("Неверный cursor")
|
||||
}
|
||||
t, e := time.Parse(time.RFC3339Nano, p[1])
|
||||
if e != nil {
|
||||
return 0, nil, "", bad("Неверный cursor")
|
||||
}
|
||||
return limit, &t, p[2], nil
|
||||
}
|
||||
func (a *API) dbContext(parent context.Context) (context.Context, context.CancelFunc) {
|
||||
return context.WithTimeout(parent, a.dbTimeout)
|
||||
}
|
||||
|
||||
func (a *API) authenticate(r *http.Request) (string, string, *apiError) {
|
||||
h := r.Header.Get("Authorization")
|
||||
if !strings.HasPrefix(h, "Bearer ") || strings.TrimSpace(strings.TrimPrefix(h, "Bearer ")) == "" {
|
||||
return "", "", unauthorized("Bearer")
|
||||
}
|
||||
token := strings.TrimSpace(strings.TrimPrefix(h, "Bearer "))
|
||||
ctx, cancel := context.WithTimeout(r.Context(), a.dbTimeout)
|
||||
defer cancel()
|
||||
uid, err := a.sessions.Get(ctx, tokenKey(token)).Result()
|
||||
if errors.Is(err, redis.Nil) {
|
||||
return "", "", unauthorized("Bearer")
|
||||
}
|
||||
if err != nil {
|
||||
a.logger.Printf("session store unavailable: %v", err)
|
||||
return "", "", unavailable()
|
||||
}
|
||||
if !uuidRE.MatchString(uid) {
|
||||
return "", "", unauthorized("Bearer")
|
||||
}
|
||||
return uid, token, nil
|
||||
}
|
||||
|
||||
func scanUser(row pgx.Row) (User, string, error) {
|
||||
var u User
|
||||
var created time.Time
|
||||
var password string
|
||||
err := row.Scan(&u.ID, &u.Username, &u.DisplayName, &created, &password)
|
||||
u.CreatedAt = formatTime(created)
|
||||
return u, password, err
|
||||
}
|
||||
|
||||
func (a *API) route(r *http.Request) (response, *apiError) {
|
||||
p := r.URL.Path
|
||||
if r.Method == "GET" && p == "/health/live" {
|
||||
return response{status: 200, body: map[string]string{"status": "ok"}}, nil
|
||||
}
|
||||
if r.Method == "GET" && p == "/health/ready" {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), time.Second)
|
||||
defer cancel()
|
||||
if a.db.Ping(ctx) != nil || a.sessions.Ping(ctx).Err() != nil || a.objects.Ready(ctx) != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
return response{status: 200, body: map[string]string{"status": "ok"}}, nil
|
||||
}
|
||||
if r.Method == "POST" && p == "/api/v1/users" {
|
||||
return a.createUser(r)
|
||||
}
|
||||
if r.Method == "POST" && p == "/api/v1/auth/sessions" {
|
||||
return a.createSession(r)
|
||||
}
|
||||
if !strings.HasPrefix(p, "/api/v1/") {
|
||||
return response{}, missing()
|
||||
}
|
||||
uid, token, authErr := a.authenticate(r)
|
||||
if authErr != nil {
|
||||
return response{}, authErr
|
||||
}
|
||||
if r.Method == "DELETE" && p == "/api/v1/auth/sessions/current" {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), a.dbTimeout)
|
||||
defer cancel()
|
||||
if err := a.sessions.Del(ctx, tokenKey(token)).Err(); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
return response{status: 204, noBody: true}, nil
|
||||
}
|
||||
if r.Method == "GET" && p == "/api/v1/users/me" {
|
||||
return a.currentUser(r, uid)
|
||||
}
|
||||
if strings.HasPrefix(p, "/api/v1/attachments/") {
|
||||
parts := strings.Split(strings.TrimPrefix(p, "/api/v1/attachments/"), "/")
|
||||
if !uuidRE.MatchString(parts[0]) {
|
||||
return response{}, bad("Неверный attachment_id")
|
||||
}
|
||||
if r.Method == "GET" && (len(parts) == 1 || (len(parts) == 2 && parts[1] == "content")) {
|
||||
return a.getAttachment(r, uid, parts[0], len(parts) == 2)
|
||||
}
|
||||
return response{}, missing()
|
||||
}
|
||||
if p == "/api/v1/chats" {
|
||||
if r.Method == "POST" {
|
||||
return a.createChat(r, uid)
|
||||
}
|
||||
if r.Method == "GET" {
|
||||
return a.listChats(r, uid)
|
||||
}
|
||||
}
|
||||
if !strings.HasPrefix(p, "/api/v1/chats/") {
|
||||
return response{}, missing()
|
||||
}
|
||||
parts := strings.Split(strings.TrimPrefix(p, "/api/v1/chats/"), "/")
|
||||
if len(parts) < 1 || !uuidRE.MatchString(parts[0]) {
|
||||
return response{}, bad("Неверный chat_id")
|
||||
}
|
||||
chatID := parts[0]
|
||||
if len(parts) == 2 && parts[1] == "attachments" && r.Method == "POST" {
|
||||
return a.uploadAttachment(r, uid, chatID)
|
||||
}
|
||||
if len(parts) == 1 && r.Method == "GET" {
|
||||
return a.getChat(r, uid, chatID)
|
||||
}
|
||||
if len(parts) == 2 && parts[1] == "messages" {
|
||||
if r.Method == "POST" {
|
||||
return a.createMessage(r, uid, chatID)
|
||||
}
|
||||
if r.Method == "GET" {
|
||||
return a.listMessages(r, uid, chatID)
|
||||
}
|
||||
}
|
||||
return response{}, missing()
|
||||
}
|
||||
|
||||
func (a *API) createUser(r *http.Request) (response, *apiError) {
|
||||
var in struct {
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
Password string `json:"password"`
|
||||
}
|
||||
if e := decode(r, &in); e != nil {
|
||||
return response{}, e
|
||||
}
|
||||
if !usernameRE.MatchString(in.Username) || !validText(in.DisplayName, 100) || !validPassword(in.Password) {
|
||||
return response{}, bad("Неверные данные пользователя")
|
||||
}
|
||||
passwordHash, err := hashPassword(in.Password)
|
||||
if err != nil {
|
||||
return response{}, &apiError{status: 500, code: "internal_error", message: "Внутренняя ошибка"}
|
||||
}
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
var u User
|
||||
var created time.Time
|
||||
err = a.db.QueryRow(ctx, `INSERT INTO users(id,username,display_name,password_hash) VALUES($1,$2,$3,$4) RETURNING id::text,username,display_name,created_at`, newID(), in.Username, in.DisplayName, passwordHash).Scan(&u.ID, &u.Username, &u.DisplayName, &created)
|
||||
if err != nil {
|
||||
var pgErr *pgconn.PgError
|
||||
if errors.As(err, &pgErr) && pgErr.Code == "23505" {
|
||||
return response{}, &apiError{status: 409, code: "conflict", message: "Имя пользователя занято"}
|
||||
}
|
||||
return response{}, unavailable()
|
||||
}
|
||||
u.CreatedAt = formatTime(created)
|
||||
return response{status: 201, body: u}, nil
|
||||
}
|
||||
|
||||
func (a *API) createSession(r *http.Request) (response, *apiError) {
|
||||
if r.ContentLength > 0 {
|
||||
return response{}, bad("Тело запроса не допускается")
|
||||
}
|
||||
h := r.Header.Get("Authorization")
|
||||
if !strings.HasPrefix(h, "Basic ") {
|
||||
return response{}, unauthorized("Basic")
|
||||
}
|
||||
raw, err := base64.StdEncoding.DecodeString(strings.TrimSpace(strings.TrimPrefix(h, "Basic ")))
|
||||
if err != nil {
|
||||
return response{}, unauthorized("Basic")
|
||||
}
|
||||
parts := strings.SplitN(string(raw), ":", 2)
|
||||
if len(parts) != 2 || !usernameRE.MatchString(parts[0]) {
|
||||
return response{}, unauthorized("Basic")
|
||||
}
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
u, passwordHash, err := scanUser(a.db.QueryRow(ctx, `SELECT id::text,username,display_name,created_at,password_hash FROM users WHERE username=$1`, parts[0]))
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return response{}, unauthorized("Basic")
|
||||
}
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if !verifyPassword(passwordHash, parts[1]) {
|
||||
return response{}, unauthorized("Basic")
|
||||
}
|
||||
token := newToken()
|
||||
expires := time.Now().UTC().Add(a.ttl)
|
||||
if err = a.sessions.Set(ctx, tokenKey(token), u.ID, a.ttl).Err(); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
return response{status: 201, body: Session{Token: token, TokenType: "Bearer", ExpiresAt: formatTime(expires), User: u}}, nil
|
||||
}
|
||||
|
||||
func (a *API) currentUser(r *http.Request, uid string) (response, *apiError) {
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
u, _, err := scanUser(a.db.QueryRow(ctx, `SELECT id::text,username,display_name,created_at,password_hash FROM users WHERE id=$1`, uid))
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return response{}, unauthorized("Bearer")
|
||||
}
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
return response{status: 200, body: u}, nil
|
||||
}
|
||||
|
||||
func (a *API) createChat(r *http.Request, uid string) (response, *apiError) {
|
||||
var in struct {
|
||||
Title string `json:"title"`
|
||||
MemberIDs []string `json:"member_ids"`
|
||||
}
|
||||
if e := decode(r, &in); e != nil {
|
||||
return response{}, e
|
||||
}
|
||||
if !validText(in.Title, 100) || len(in.MemberIDs) < 1 || len(in.MemberIDs) > 100 {
|
||||
return response{}, bad("Неверные данные чата")
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, id := range in.MemberIDs {
|
||||
if !uuidRE.MatchString(id) || seen[id] {
|
||||
return response{}, bad("Неверные участники чата")
|
||||
}
|
||||
seen[id] = true
|
||||
}
|
||||
if !seen[uid] {
|
||||
return response{}, bad("Создатель должен быть участником")
|
||||
}
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
tx, err := a.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
c := Chat{ID: newID(), Title: in.Title, MemberIDs: append([]string{}, in.MemberIDs...)}
|
||||
var created time.Time
|
||||
if err = tx.QueryRow(ctx, `INSERT INTO chats(id,title) VALUES($1,$2) RETURNING created_at`, c.ID, c.Title).Scan(&created); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
for _, id := range c.MemberIDs {
|
||||
if _, err = tx.Exec(ctx, `INSERT INTO chat_members(chat_id,user_id) VALUES($1,$2)`, c.ID, id); err != nil {
|
||||
var pgErr *pgconn.PgError
|
||||
if errors.As(err, &pgErr) && pgErr.Code == "23503" {
|
||||
return response{}, bad("Неверные участники чата")
|
||||
}
|
||||
return response{}, unavailable()
|
||||
}
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
c.CreatedAt = formatTime(created)
|
||||
return response{status: 201, body: c}, nil
|
||||
}
|
||||
|
||||
func scanChat(row pgx.Row) (Chat, time.Time, error) {
|
||||
var c Chat
|
||||
var t time.Time
|
||||
err := row.Scan(&c.ID, &c.Title, &c.MemberIDs, &t)
|
||||
c.CreatedAt = formatTime(t)
|
||||
return c, t, err
|
||||
}
|
||||
func chatSelect() string {
|
||||
return `SELECT c.id::text,c.title,ARRAY(SELECT cm2.user_id::text FROM chat_members cm2 WHERE cm2.chat_id=c.id ORDER BY cm2.user_id),c.created_at FROM chats c`
|
||||
}
|
||||
|
||||
func (a *API) getChat(r *http.Request, uid, chatID string) (response, *apiError) {
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
c, _, err := scanChat(a.db.QueryRow(ctx, chatSelect()+` WHERE c.id=$1 AND EXISTS(SELECT 1 FROM chat_members cm WHERE cm.chat_id=c.id AND cm.user_id=$2)`, chatID, uid))
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return response{}, missing()
|
||||
}
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
return response{status: 200, body: c}, nil
|
||||
}
|
||||
|
||||
func (a *API) listChats(r *http.Request, uid string) (response, *apiError) {
|
||||
limit, afterTime, afterID, e := parsePage(r, "chats:"+uid)
|
||||
if e != nil {
|
||||
return response{}, e
|
||||
}
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
if afterTime != nil {
|
||||
var ok bool
|
||||
err := a.db.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM chats c JOIN chat_members cm ON cm.chat_id=c.id WHERE cm.user_id=$1 AND c.id=$2 AND c.created_at=$3)`, uid, afterID, *afterTime).Scan(&ok)
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if !ok {
|
||||
return response{}, bad("Неверный cursor")
|
||||
}
|
||||
}
|
||||
var rows pgx.Rows
|
||||
var err error
|
||||
base := chatSelect() + ` JOIN chat_members mine ON mine.chat_id=c.id WHERE mine.user_id=$1`
|
||||
if afterTime == nil {
|
||||
rows, err = a.db.Query(ctx, base+` ORDER BY c.created_at,c.id LIMIT $2`, uid, limit+1)
|
||||
} else {
|
||||
rows, err = a.db.Query(ctx, base+` AND (c.created_at,c.id)>($2,$3) ORDER BY c.created_at,c.id LIMIT $4`, uid, *afterTime, afterID, limit+1)
|
||||
}
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
defer rows.Close()
|
||||
items := make([]Chat, 0, limit+1)
|
||||
times := make([]time.Time, 0, limit+1)
|
||||
for rows.Next() {
|
||||
c, t, scanErr := scanChat(rows)
|
||||
if scanErr != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
items = append(items, c)
|
||||
times = append(times, t)
|
||||
}
|
||||
if rows.Err() != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
var next any
|
||||
if len(items) > limit {
|
||||
last := items[limit-1]
|
||||
next = cursorFor("chats:"+uid, times[limit-1], last.ID)
|
||||
items = items[:limit]
|
||||
}
|
||||
return response{status: 200, body: map[string]any{"items": items, "next_cursor": next}}, nil
|
||||
}
|
||||
|
||||
func (a *API) createMessage(r *http.Request, uid, chatID string) (response, *apiError) {
|
||||
key := r.Header.Get("Idempotency-Key")
|
||||
if !idempotencyKeyRE.MatchString(key) {
|
||||
return response{}, bad("Требуется корректный Idempotency-Key")
|
||||
}
|
||||
var in struct {
|
||||
Text *string `json:"text"`
|
||||
AttachmentIDs []string `json:"attachment_ids"`
|
||||
}
|
||||
if e := decode(r, &in); e != nil {
|
||||
return response{}, e
|
||||
}
|
||||
if in.Text == nil {
|
||||
return response{}, bad("Требуется поле text")
|
||||
}
|
||||
text := *in.Text
|
||||
if !utf8.ValidString(text) || utf8.RuneCountInString(text) > 4000 || len(in.AttachmentIDs) > 10 || (strings.TrimSpace(text) == "" && len(in.AttachmentIDs) == 0) {
|
||||
return response{}, bad("Текст сообщения должен содержать от 1 до 4000 символов")
|
||||
}
|
||||
if in.AttachmentIDs == nil {
|
||||
in.AttachmentIDs = []string{}
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for i, id := range in.AttachmentIDs {
|
||||
id = strings.ToLower(id)
|
||||
if !uuidRE.MatchString(id) || seen[id] {
|
||||
return response{}, bad("Неверные attachment_ids")
|
||||
}
|
||||
seen[id] = true
|
||||
in.AttachmentIDs[i] = id
|
||||
}
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
tx, err := a.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
var member bool
|
||||
if err = tx.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM chat_members WHERE chat_id=$1 AND user_id=$2)`, chatID, uid).Scan(&member); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if !member {
|
||||
return response{}, missing()
|
||||
}
|
||||
requestHash := sha256.Sum256([]byte(text))
|
||||
if len(in.AttachmentIDs) > 0 {
|
||||
canonical, _ := json.Marshal(in)
|
||||
requestHash = sha256.Sum256(canonical)
|
||||
}
|
||||
for _, id := range in.AttachmentIDs {
|
||||
var state string
|
||||
err = tx.QueryRow(ctx, `SELECT status FROM attachments WHERE id=$1 AND chat_id=$2`, id, chatID).Scan(&state)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return response{}, missing()
|
||||
}
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if state != "ready" {
|
||||
return response{}, &apiError{status: 409, code: "conflict", message: "Вложение не готово"}
|
||||
}
|
||||
}
|
||||
if _, err = tx.Exec(ctx, `DELETE FROM message_idempotency WHERE user_id=$1 AND method=$2 AND path=$3 AND idempotency_key=$4 AND created_at < clock_timestamp() - ($5 * interval '1 second')`, uid, r.Method, r.URL.Path, key, int64(a.idempotencyTTL/time.Second)); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
result, err := tx.Exec(ctx, `INSERT INTO message_idempotency(user_id,method,path,idempotency_key,request_hash) VALUES($1,$2,$3,$4,$5) ON CONFLICT DO NOTHING`, uid, r.Method, r.URL.Path, key, requestHash[:])
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
inserted := result.RowsAffected() == 1
|
||||
if !inserted {
|
||||
var storedHash []byte
|
||||
var messageID string
|
||||
if err = tx.QueryRow(ctx, `SELECT request_hash,message_id::text FROM message_idempotency WHERE user_id=$1 AND method=$2 AND path=$3 AND idempotency_key=$4 FOR UPDATE`, uid, r.Method, r.URL.Path, key).Scan(&storedHash, &messageID); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if !bytes.Equal(storedHash, requestHash[:]) {
|
||||
return response{}, &apiError{status: 409, code: "conflict", message: "Idempotency-Key уже использован с другим запросом"}
|
||||
}
|
||||
var m Message
|
||||
var created time.Time
|
||||
if err = tx.QueryRow(ctx, `SELECT id::text,chat_id::text,sender_id::text,text,created_at,attachment_ids::text[] FROM messages WHERE id=$1`, messageID).Scan(&m.ID, &m.ChatID, &m.SenderID, &m.Text, &created, &m.AttachmentIDs); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
m.CreatedAt = formatTime(created)
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
return response{status: 201, body: m}, nil
|
||||
}
|
||||
m := Message{ID: newID(), ChatID: chatID, SenderID: uid, Text: text, AttachmentIDs: in.AttachmentIDs}
|
||||
var created time.Time
|
||||
if err = tx.QueryRow(ctx, `INSERT INTO messages(id,chat_id,sender_id,text,attachment_ids) VALUES($1,$2,$3,$4,$5) RETURNING created_at`, m.ID, m.ChatID, m.SenderID, m.Text, m.AttachmentIDs).Scan(&created); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if _, err = tx.Exec(ctx, `UPDATE message_idempotency SET message_id=$1 WHERE user_id=$2 AND method=$3 AND path=$4 AND idempotency_key=$5`, m.ID, uid, r.Method, r.URL.Path, key); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
m.CreatedAt = formatTime(created)
|
||||
return response{status: 201, body: m}, nil
|
||||
}
|
||||
|
||||
func (a *API) listMessages(r *http.Request, uid, chatID string) (response, *apiError) {
|
||||
limit, afterTime, afterID, e := parsePage(r, "messages:"+chatID)
|
||||
if e != nil {
|
||||
return response{}, e
|
||||
}
|
||||
ctx, cancel := a.dbContext(r.Context())
|
||||
defer cancel()
|
||||
var member bool
|
||||
if err := a.db.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM chat_members WHERE chat_id=$1 AND user_id=$2)`, chatID, uid).Scan(&member); err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if !member {
|
||||
return response{}, missing()
|
||||
}
|
||||
if afterTime != nil {
|
||||
var ok bool
|
||||
err := a.db.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM messages WHERE chat_id=$1 AND id=$2 AND created_at=$3)`, chatID, afterID, *afterTime).Scan(&ok)
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
if !ok {
|
||||
return response{}, bad("Неверный cursor")
|
||||
}
|
||||
}
|
||||
var rows pgx.Rows
|
||||
var err error
|
||||
base := `SELECT id::text,chat_id::text,sender_id::text,text,created_at,attachment_ids::text[] FROM messages WHERE chat_id=$1`
|
||||
if afterTime == nil {
|
||||
rows, err = a.db.Query(ctx, base+` ORDER BY created_at,id LIMIT $2`, chatID, limit+1)
|
||||
} else {
|
||||
rows, err = a.db.Query(ctx, base+` AND (created_at,id)>($2,$3) ORDER BY created_at,id LIMIT $4`, chatID, *afterTime, afterID, limit+1)
|
||||
}
|
||||
if err != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
defer rows.Close()
|
||||
items := make([]Message, 0, limit+1)
|
||||
times := make([]time.Time, 0, limit+1)
|
||||
for rows.Next() {
|
||||
var m Message
|
||||
var t time.Time
|
||||
if rows.Scan(&m.ID, &m.ChatID, &m.SenderID, &m.Text, &t, &m.AttachmentIDs) != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
m.CreatedAt = formatTime(t)
|
||||
items = append(items, m)
|
||||
times = append(times, t)
|
||||
}
|
||||
if rows.Err() != nil {
|
||||
return response{}, unavailable()
|
||||
}
|
||||
var next any
|
||||
if len(items) > limit {
|
||||
last := items[limit-1]
|
||||
next = cursorFor("messages:"+chatID, times[limit-1], last.ID)
|
||||
items = items[:limit]
|
||||
}
|
||||
return response{status: 200, body: map[string]any{"items": items, "next_cursor": next}}, nil
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package objectstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/url"
|
||||
|
||||
madmin "github.com/minio/madmin-go/v3"
|
||||
)
|
||||
|
||||
// Bootstrap receives root credentials only in the one-shot s3-init process.
|
||||
// The API and cleaner receive a separate user limited to this bucket/prefix.
|
||||
func Bootstrap(ctx context.Context, endpoint, bucket, rootAccess, rootSecret, apiAccess, apiSecret string) error {
|
||||
u, err := url.Parse(endpoint)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
admin, err := madmin.New(u.Host, rootAccess, rootSecret, u.Scheme == "https")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = admin.AddUser(ctx, apiAccess, apiSecret); err != nil {
|
||||
return err
|
||||
}
|
||||
policy := map[string]any{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": []any{
|
||||
map[string]any{"Effect": "Allow", "Action": []string{"s3:GetBucketLocation", "s3:ListBucket"}, "Resource": []string{"arn:aws:s3:::" + bucket}},
|
||||
map[string]any{"Effect": "Allow", "Action": []string{"s3:GetObject", "s3:PutObject", "s3:DeleteObject"}, "Resource": []string{"arn:aws:s3:::" + bucket + "/originals/*"}},
|
||||
},
|
||||
}
|
||||
encoded, err := json.Marshal(policy)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err = admin.AddCannedPolicy(ctx, "messenger-api", encoded); err != nil {
|
||||
return err
|
||||
}
|
||||
return admin.SetPolicy(ctx, "messenger-api", apiAccess, false)
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
package objectstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
type Store struct {
|
||||
Client *minio.Client
|
||||
Bucket string
|
||||
endpoint string
|
||||
}
|
||||
|
||||
func New(endpoint, region, bucket, access, secret string) (*Store, error) {
|
||||
u, err := url.Parse(endpoint)
|
||||
if err != nil || u.Host == "" || (u.Scheme != "http" && u.Scheme != "https") || u.Path != "" {
|
||||
return nil, fmt.Errorf("invalid S3 endpoint")
|
||||
}
|
||||
transport := &http.Transport{
|
||||
DialContext: (&net.Dialer{Timeout: 2 * time.Second, KeepAlive: 30 * time.Second}).DialContext,
|
||||
MaxIdleConns: 16, MaxIdleConnsPerHost: 8, MaxConnsPerHost: 8,
|
||||
ResponseHeaderTimeout: 10 * time.Second, IdleConnTimeout: 30 * time.Second,
|
||||
TLSHandshakeTimeout: 3 * time.Second,
|
||||
}
|
||||
client, err := minio.New(u.Host, &minio.Options{
|
||||
Creds: credentials.NewStaticV4(access, secret, ""), Secure: u.Scheme == "https",
|
||||
Region: region, Transport: transport, MaxRetries: 1,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Store{Client: client, Bucket: bucket, endpoint: endpoint}, nil
|
||||
}
|
||||
|
||||
func (s *Store) Ready(ctx context.Context) error {
|
||||
ok, err := s.Client.BucketExists(ctx, s.Bucket)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !ok {
|
||||
return fmt.Errorf("S3 bucket missing")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Store) Put(ctx context.Context, key string, r io.Reader, size int64, kind string) error {
|
||||
_, err := s.Client.PutObject(ctx, s.Bucket, key, r, size, minio.PutObjectOptions{
|
||||
ContentType: kind, DisableMultipart: true,
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) Get(ctx context.Context, key string) (*minio.Object, error) {
|
||||
object, err := s.Client.GetObject(ctx, s.Bucket, key, minio.GetObjectOptions{})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err = object.Stat(); err != nil {
|
||||
object.Close()
|
||||
return nil, err
|
||||
}
|
||||
return object, nil
|
||||
}
|
||||
|
||||
func (s *Store) Delete(ctx context.Context, key string) error {
|
||||
return s.Client.RemoveObject(ctx, s.Bucket, key, minio.RemoveObjectOptions{})
|
||||
}
|
||||
|
||||
// Audit performs an actual unauthenticated GET and an out-of-scope operation
|
||||
// with API credentials. Neither check prints credentials or object URLs.
|
||||
func (s *Store) Audit(ctx context.Context, key string) error {
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", s.endpoint+"/"+s.Bucket+"/"+key, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
response, err := (&http.Client{Timeout: 5 * time.Second}).Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
response.Body.Close()
|
||||
if response.StatusCode != 403 {
|
||||
return fmt.Errorf("anonymous GET returned %d", response.StatusCode)
|
||||
}
|
||||
err = s.Client.MakeBucket(ctx, "forbidden-audit-bucket", minio.MakeBucketOptions{})
|
||||
if minio.ToErrorResponse(err).Code != "AccessDenied" {
|
||||
return fmt.Errorf("API credentials are not restricted: %v", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
Executable
+21
@@ -0,0 +1,21 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
mkdir -p secrets
|
||||
if [ ! -f secrets/postgres_password ]; then
|
||||
umask 077
|
||||
openssl rand -hex 24 > secrets/postgres_password
|
||||
fi
|
||||
if [ ! -f secrets/redis_password ]; then
|
||||
umask 077
|
||||
openssl rand -hex 24 > secrets/redis_password
|
||||
fi
|
||||
postgres_password=$(cat secrets/postgres_password)
|
||||
redis_password=$(cat secrets/redis_password)
|
||||
umask 077
|
||||
printf 'postgres://messenger:%s@postgres:5432/messenger?sslmode=disable\n' "$postgres_password" > secrets/database_url
|
||||
printf 'redis://:%s@redis:6379/0\n' "$redis_password" > secrets/cache_url
|
||||
if [ ! -f .env ]; then cp .env.example .env; fi
|
||||
for secret_name in s3_admin_access s3_admin_secret s3_access_key_id s3_secret_access_key; do
|
||||
if [ ! -f "secrets/$secret_name" ]; then openssl rand -hex 24 > "secrets/$secret_name"; fi
|
||||
done
|
||||
printf '%s\n' 'S3 credentials ready; bucket and policy are bootstrapped by s3-init.'
|
||||
Executable
+27
@@ -0,0 +1,27 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
wait_healthy() {
|
||||
service=$1
|
||||
container_id=$(docker compose --env-file .env ps -q "$service")
|
||||
attempts=0
|
||||
until [ "$(docker inspect --format '{{.State.Health.Status}}' "$container_id" 2>/dev/null || true)" = healthy ]; do
|
||||
attempts=$((attempts + 1))
|
||||
if [ "$attempts" -ge 60 ]; then
|
||||
printf '%s\n' "$service did not become healthy" >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
docker compose --env-file .env restart postgres redis s3 >/dev/null
|
||||
wait_healthy postgres
|
||||
wait_healthy redis
|
||||
wait_healthy s3
|
||||
docker compose --env-file .env restart api1 api2 cleanup >/dev/null
|
||||
wait_healthy api1
|
||||
wait_healthy api2
|
||||
docker compose --env-file .env restart nginx >/dev/null
|
||||
wait_healthy nginx
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
wait_healthy() {
|
||||
service=$1
|
||||
container_id=$(docker compose --env-file .env ps -q "$service")
|
||||
attempts=0
|
||||
until [ "$(docker inspect --format '{{.State.Health.Status}}' "$container_id" 2>/dev/null || true)" = healthy ]; do
|
||||
attempts=$((attempts + 1))
|
||||
if [ "$attempts" -ge 60 ]; then
|
||||
printf '%s\n' "$service did not become healthy" >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
docker compose --env-file .env build migrate >/dev/null
|
||||
docker compose --env-file .env run --rm --no-deps migrate >/dev/null
|
||||
for service in api1 api2; do
|
||||
docker compose --env-file .env up -d --no-deps --force-recreate "$service" >/dev/null
|
||||
wait_healthy "$service"
|
||||
docker compose --env-file .env exec -T nginx nginx -s reload >/dev/null
|
||||
done
|
||||
Executable
+17
@@ -0,0 +1,17 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
docker compose --env-file .env start api1 api2 >/dev/null
|
||||
for service in api1 api2; do
|
||||
container_id=$(docker compose --env-file .env ps -q "$service")
|
||||
attempts=0
|
||||
until [ "$(docker inspect --format '{{.State.Health.Status}}' "$container_id" 2>/dev/null || true)" = healthy ]; do
|
||||
attempts=$((attempts + 1))
|
||||
if [ "$attempts" -ge 60 ]; then
|
||||
printf '%s\n' "$service did not become healthy" >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
done
|
||||
Executable
+12
@@ -0,0 +1,12 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
: "${TARGET_INSTANCE_ID:?test provides target API instance alias}"
|
||||
|
||||
case "$TARGET_INSTANCE_ID" in
|
||||
api-1) service=api1 ;;
|
||||
api-2) service=api2 ;;
|
||||
*) printf '%s\n' "Unknown API instance alias" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
docker compose --env-file .env stop --timeout 3 "$service" >/dev/null
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 6.0 KiB |
@@ -0,0 +1,58 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stop Redis and PostgreSQL separately and check bounded API failures."""
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure, check # noqa: E402
|
||||
|
||||
|
||||
def compose(*args):
|
||||
subprocess.run(["docker", "compose", "--env-file", ".env", *args], cwd=ROOT,
|
||||
check=True, stdout=subprocess.DEVNULL)
|
||||
|
||||
|
||||
def wait_ready(client, timeout=60):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
client.call("GET", "/health/ready", schema="Health")
|
||||
return
|
||||
except Failure:
|
||||
time.sleep(0.5)
|
||||
raise Failure("stack did not recover")
|
||||
|
||||
|
||||
def main():
|
||||
client = Client(timeout=5)
|
||||
actor = client.user("dependency")
|
||||
try:
|
||||
compose("stop", "redis")
|
||||
client.call("GET", "/health/ready", 503)
|
||||
client.call("GET", "/api/v1/users/me", 503, actor=actor)
|
||||
client.call("GET", "/api/v1/users/me", 401)
|
||||
finally:
|
||||
compose("start", "redis")
|
||||
wait_ready(client)
|
||||
client.call("GET", "/api/v1/users/me", actor=actor, schema="User")
|
||||
|
||||
try:
|
||||
compose("stop", "postgres")
|
||||
client.call("GET", "/health/ready", 503)
|
||||
client.call("GET", "/api/v1/users/me", 503, actor=actor)
|
||||
finally:
|
||||
compose("start", "postgres")
|
||||
wait_ready(client)
|
||||
client.call("GET", "/api/v1/users/me", actor=actor, schema="User")
|
||||
print("PASS: Redis/PostgreSQL failures returned 503; anonymous access stayed 401; state recovered")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (Failure, OSError, subprocess.CalledProcessError) as exc:
|
||||
print("FAIL:", exc)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Recreate the API with a two-second session TTL and prove expiry."""
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure, check # noqa: E402
|
||||
|
||||
|
||||
def compose(env, *args):
|
||||
subprocess.run(["docker", "compose", "--env-file", ".env", *args], cwd=ROOT,
|
||||
env=env, check=True, stdout=subprocess.DEVNULL)
|
||||
|
||||
|
||||
def wait_ready(timeout=60):
|
||||
client = Client(timeout=2)
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
client.call("GET", "/health/ready", schema="Health")
|
||||
return
|
||||
except Failure:
|
||||
time.sleep(0.5)
|
||||
raise Failure("API did not become ready")
|
||||
|
||||
|
||||
def main():
|
||||
normal_env = dict(os.environ)
|
||||
normal_env.pop("SESSION_TTL_SECONDS", None)
|
||||
short_env = dict(normal_env)
|
||||
short_env["SESSION_TTL_SECONDS"] = "2"
|
||||
try:
|
||||
compose(short_env, "up", "-d", "--force-recreate", "--no-deps", "--wait", "api1", "api2")
|
||||
compose(short_env, "exec", "-T", "nginx", "nginx", "-s", "reload")
|
||||
wait_ready()
|
||||
client = Client()
|
||||
actor = client.user("expiry")
|
||||
time.sleep(3)
|
||||
client.call("GET", "/api/v1/users/me", 401, actor=actor)
|
||||
print("PASS: two-second Redis session expired and was rejected")
|
||||
finally:
|
||||
compose(normal_env, "up", "-d", "--force-recreate", "--no-deps", "--wait", "api1", "api2")
|
||||
compose(normal_env, "exec", "-T", "nginx", "nginx", "-s", "reload")
|
||||
wait_ready()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (Failure, OSError, subprocess.CalledProcessError) as exc:
|
||||
print("FAIL:", exc)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Prove that one idempotency key creates one message across both replicas."""
|
||||
import concurrent.futures
|
||||
import json
|
||||
import sys
|
||||
import threading
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, arguments, check, run # noqa: E402
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments("Concurrent cross-replica idempotency check")
|
||||
base = Client(args.base_url, args.ca_file)
|
||||
alice, _, _, chat = base.bootstrap()
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages"
|
||||
key = "parallel-" + uuid.uuid4().hex
|
||||
body = {"text": "one logical concurrent message"}
|
||||
workers = 12
|
||||
barrier = threading.Barrier(workers)
|
||||
|
||||
def send(_):
|
||||
client = Client(args.base_url, args.ca_file)
|
||||
barrier.wait()
|
||||
result = client.call("POST", path, 201, actor=alice, body=body,
|
||||
headers={"Idempotency-Key": key}, schema="Message")
|
||||
return result["json"], result["headers"]["X-Instance-Id"]
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
|
||||
responses = list(pool.map(send, range(workers)))
|
||||
|
||||
messages = [item[0] for item in responses]
|
||||
instances = sorted(set(item[1] for item in responses))
|
||||
ids = sorted(set(item["id"] for item in messages))
|
||||
check(len(ids) == 1, "one key created more than one message")
|
||||
check(all(item == messages[0] for item in messages),
|
||||
"idempotent responses differ")
|
||||
check(len(instances) >= 2, "requests did not cross two API replicas")
|
||||
|
||||
base.call("POST", path, 409, actor=alice, body={"text": "different body"},
|
||||
headers={"Idempotency-Key": key})
|
||||
page = base.call("GET", path + "?limit=100", actor=alice,
|
||||
schema="MessagePage")["json"]
|
||||
check(sum(item["id"] == ids[0] for item in page["items"]) == 1,
|
||||
"history contains a duplicate message")
|
||||
|
||||
result = {"parallel_requests": workers, "instances": instances,
|
||||
"unique_message_ids": len(ids), "conflicting_reuse_status": 409}
|
||||
print("PASS:", json.dumps(result, ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,71 @@
|
||||
"""Check LB failover when api1 still runs but cannot reach its stores."""
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure, check, run # noqa: E402
|
||||
|
||||
|
||||
def docker(*args):
|
||||
return subprocess.run(["docker", *args], cwd=ROOT, check=True,
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
|
||||
|
||||
def compose(*args):
|
||||
return docker("compose", "--env-file", ".env", *args)
|
||||
|
||||
|
||||
def details(container):
|
||||
return json.loads(docker("inspect", container))[0]
|
||||
|
||||
|
||||
def wait_health(container, expected, timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if details(container)["State"]["Health"]["Status"] == expected:
|
||||
return
|
||||
time.sleep(1)
|
||||
raise Failure("api1 did not become " + expected)
|
||||
|
||||
|
||||
def main():
|
||||
client = Client(timeout=8)
|
||||
seen = set()
|
||||
for _ in range(20):
|
||||
seen.add(client.call("GET", "/health/ready", schema="Health")
|
||||
["headers"]["X-Instance-Id"])
|
||||
check(seen == {"api-1", "api-2"}, "both instances must be ready initially")
|
||||
actor, _, _, chat = client.bootstrap()
|
||||
container = compose("ps", "-q", "api1")
|
||||
check(bool(container), "api1 container is missing")
|
||||
network = next((name for name in details(container)["NetworkSettings"]["Networks"]
|
||||
if name.endswith("_backend")), None)
|
||||
check(network is not None, "api1 backend network is missing")
|
||||
disconnected = False
|
||||
try:
|
||||
docker("network", "disconnect", network, container)
|
||||
disconnected = True
|
||||
wait_health(container, "unhealthy")
|
||||
for _ in range(20):
|
||||
for path, schema in (("/api/v1/users/me", "User"),
|
||||
("/health/ready", "Health")):
|
||||
response = client.call("GET", path, actor=actor, schema=schema)
|
||||
check(response["headers"]["X-Instance-Id"] == "api-2",
|
||||
path + " was not served by healthy api2")
|
||||
for number in range(10):
|
||||
response = client.message(actor, chat, "unready check " + str(number))
|
||||
check(response["headers"]["X-Instance-Id"] == "api-2",
|
||||
"message was not accepted by healthy api2")
|
||||
print("PASS: api1 unhealthy with frontend connection; 40 GETs and 10 POSTs served by api2")
|
||||
finally:
|
||||
if disconnected:
|
||||
docker("network", "connect", "--alias", "api1", network, container)
|
||||
wait_health(container, "healthy")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Ten simultaneous uploads with kernel cgroup peak-memory measurements."""
|
||||
import concurrent.futures
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import random
|
||||
import struct
|
||||
import threading
|
||||
import time
|
||||
|
||||
from verify_attachments import ROOT, Client, check, compose, png_chunk, run, upload, wait_ready
|
||||
|
||||
|
||||
def fixture():
|
||||
# 9M pixels; random rows make a ~6.8 MiB valid PNG without another library.
|
||||
rng = random.Random(2026)
|
||||
import zlib
|
||||
compressor = zlib.compressobj(1)
|
||||
parts = []
|
||||
for row in range(3000):
|
||||
pixels = rng.randbytes(9000) if row % 4 == 0 else bytes(9000)
|
||||
parts.append(compressor.compress(b"\x00" + pixels))
|
||||
parts.append(compressor.flush())
|
||||
return (b"\x89PNG\r\n\x1a\n" + png_chunk(b"IHDR", struct.pack(">IIBBBBB", 3000, 3000, 8, 2, 0, 0, 0))
|
||||
+ png_chunk(b"IDAT", b"".join(parts)) + png_chunk(b"IEND", b""))
|
||||
|
||||
|
||||
def memory():
|
||||
return {service: json.loads(compose("exec", "-T", service, "/messenger", "memory"))
|
||||
for service in ("api1", "api2")}
|
||||
|
||||
|
||||
def main():
|
||||
client = Client(timeout=60)
|
||||
actor, _, _, chat = client.bootstrap()
|
||||
# A new container cgroup resets memory.peak for this experiment.
|
||||
compose("up", "-d", "--no-deps", "--force-recreate", "--wait", "api1", "api2")
|
||||
compose("exec", "-T", "nginx", "nginx", "-s", "reload")
|
||||
wait_ready(client)
|
||||
raw = fixture()
|
||||
digest = hashlib.sha256(raw).hexdigest()
|
||||
before = memory()
|
||||
barrier = threading.Barrier(11)
|
||||
|
||||
def send(number):
|
||||
c = Client(timeout=60)
|
||||
barrier.wait()
|
||||
started = time.monotonic()
|
||||
response = upload(c, actor, chat, raw, filename="load-%d.png" % number)
|
||||
check(response["json"]["sha256"] == digest, "concurrent upload checksum mismatch")
|
||||
return response, time.monotonic() - started
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
|
||||
futures = [pool.submit(send, number) for number in range(10)]
|
||||
started = time.monotonic()
|
||||
barrier.wait()
|
||||
responses = [future.result() for future in futures]
|
||||
elapsed = time.monotonic() - started
|
||||
after = memory()
|
||||
for metrics in after.values():
|
||||
check(metrics["memory.peak"] < metrics["memory.max"], "upload memory reached container limit")
|
||||
for response, _ in responses:
|
||||
downloaded = client.call("GET", "/api/v1/attachments/" + response["json"]["id"] + "/content", actor=actor)
|
||||
check(downloaded["bytes"] == raw, "concurrent original changed")
|
||||
times = sorted(duration * 1000 for _, duration in responses)
|
||||
output = {"concurrent_uploads": 10, "file_size_bytes": len(raw), "pixels": 9000000,
|
||||
"successful_uploads": 10, "failed_uploads": 0,
|
||||
"elapsed_seconds": round(elapsed, 3), "successful_rps": round(10 / elapsed, 3),
|
||||
"latency_ms": {"p50": round(times[4], 3), "p95": round(times[math.ceil(.95 * 10) - 1], 3), "p99": round(times[9], 3)},
|
||||
"memory_before_bytes": before, "memory_after_bytes": after,
|
||||
"peak_source": "Linux cgroup v2 memory.peak; fresh API containers including startup",
|
||||
"decoder_slots_per_replica": 1,
|
||||
"instances": sorted({response["headers"]["X-Instance-Id"] for response, _ in responses})}
|
||||
(ROOT / "evidence/upload-memory.json").write_text(json.dumps(output, indent=2) + "\n")
|
||||
print("PASS:", json.dumps(output))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,169 @@
|
||||
"""Grade 4: image validation, attachment ACLs and cross-replica persistence."""
|
||||
import hashlib
|
||||
import json
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure, check, run # noqa: E402
|
||||
|
||||
|
||||
def compose(*args, data=None):
|
||||
result = subprocess.run(["docker", "compose", "--env-file", ".env", *args],
|
||||
cwd=ROOT, input=data, capture_output=True, text=True, check=True)
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def psql(sql):
|
||||
return compose("exec", "-T", "postgres", "psql", "-X", "-q", "-A", "-t",
|
||||
"-v", "ON_ERROR_STOP=1", "-U", "messenger", "-d", "messenger", data=sql)
|
||||
|
||||
|
||||
def png_chunk(kind, data):
|
||||
return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data))
|
||||
|
||||
|
||||
def bomb_header(width=10000, height=10000):
|
||||
# A valid PNG header must be rejected BEFORE decoding its huge bitmap.
|
||||
return b"\x89PNG\r\n\x1a\n" + png_chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0))
|
||||
|
||||
|
||||
def wide_png():
|
||||
# Exactly 40M pixels, 16-bit RGBA, one very wide scanline. The compressed
|
||||
# original is small but the decoder needs both bitmap and scanline buffers.
|
||||
compressor = zlib.compressobj(1)
|
||||
parts = [compressor.compress(b"\x00")]
|
||||
remaining = 40000000 * 8
|
||||
block = bytes(1024 * 1024)
|
||||
while remaining:
|
||||
n = min(len(block), remaining)
|
||||
parts.append(compressor.compress(block[:n]))
|
||||
remaining -= n
|
||||
parts.append(compressor.flush())
|
||||
return (b"\x89PNG\r\n\x1a\n" + png_chunk(b"IHDR", struct.pack(">IIBBBBB", 40000000, 1, 16, 6, 0, 0, 0))
|
||||
+ png_chunk(b"IDAT", b"".join(parts)) + png_chunk(b"IEND", b""))
|
||||
|
||||
|
||||
def upload(client, actor, chat, data, kind="image/png", filename="sample.png", expected=201):
|
||||
return client.call("POST", "/api/v1/chats/" + chat["id"] + "/attachments?filename=" + quote(filename, safe=""),
|
||||
expected, actor=actor, raw=data, headers={"Content-Type": kind},
|
||||
schema="Attachment" if expected == 201 else None)
|
||||
|
||||
|
||||
def wait_ready(client, timeout=60):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
client.call("GET", "/health/ready", schema="Health")
|
||||
return
|
||||
except Failure:
|
||||
time.sleep(.5)
|
||||
raise Failure("stack did not recover")
|
||||
|
||||
|
||||
def main():
|
||||
client = Client(timeout=30)
|
||||
alice, bob, eve, chat = client.bootstrap()
|
||||
png = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
jpeg = (ROOT / "student-tests/fixtures/sample.jpg").read_bytes()
|
||||
att = upload(client, alice, chat, png)["json"]
|
||||
jpg = upload(client, alice, chat, jpeg, "image/jpeg", "photo.jpg")["json"]
|
||||
for item, raw in ((att, png), (jpg, jpeg)):
|
||||
check(item["size_bytes"] == len(raw) and item["sha256"] == hashlib.sha256(raw).hexdigest(), "original metadata mismatch")
|
||||
check(client.call("GET", "/api/v1/attachments/" + item["id"] + "/content", actor=bob)["bytes"] == raw, "original bytes changed")
|
||||
# A private ancillary chunk before IEND creates valid exact-size PNGs.
|
||||
for size in (6294458, 10485760):
|
||||
sized = png[:-12] + png_chunk(b"vpAg", bytes(size - len(png) - 12)) + png[-12:]
|
||||
large = upload(client, alice, chat, sized, filename="boundary.png")["json"]
|
||||
check(large["size_bytes"] == size, "boundary upload size mismatch")
|
||||
check(client.call("GET", "/api/v1/attachments/" + large["id"] + "/content", actor=bob)["bytes"] == sized,
|
||||
"boundary original bytes changed")
|
||||
upload(client, alice, chat, b"x" * (10485760 + 1), expected=413)
|
||||
upload(client, alice, chat, png[:40], expected=415)
|
||||
damaged = bytearray(png)
|
||||
damaged[len(damaged) // 2] ^= 255
|
||||
upload(client, alice, chat, bytes(damaged), expected=415)
|
||||
upload(client, alice, chat, png[:-12], expected=415)
|
||||
corrupt = png[:33] + png_chunk(b"IDAT", b"invalid compressed data") + png_chunk(b"IEND", b"")
|
||||
upload(client, alice, chat, corrupt, expected=415)
|
||||
upload(client, alice, chat, png, "image/jpeg", expected=415)
|
||||
upload(client, alice, chat, b"not an image", expected=415)
|
||||
upload(client, alice, chat, bomb_header(), expected=413)
|
||||
upload(client, alice, chat, bomb_header(20000, 10000), expected=413)
|
||||
upload(client, alice, chat, bomb_header(8001, 5000), expected=413)
|
||||
wide = wide_png()
|
||||
wide_response = upload(client, alice, chat, wide, filename="wide.png")
|
||||
check(wide_response["json"]["sha256"] == hashlib.sha256(wide).hexdigest(), "wide PNG changed")
|
||||
wide_metrics = {service: json.loads(compose("exec", "-T", service, "/messenger", "memory")) for service in ("api1", "api2")}
|
||||
check(all(m["memory.peak"] < m["memory.max"] for m in wide_metrics.values()), "wide PNG exceeded memory limit")
|
||||
for filename in ("../image.png", "a\\b.png", "bad\x00.png", "bad\n.png", " " , "x" * 256):
|
||||
upload(client, alice, chat, png, filename=filename, expected=400)
|
||||
for suffix in ("", "/content"):
|
||||
client.call("GET", "/api/v1/attachments/" + att["id"] + suffix, 404, actor=eve)
|
||||
upload(client, eve, chat, png, expected=404)
|
||||
other = client.call("POST", "/api/v1/chats", 201, actor=alice,
|
||||
body={"title": "Other", "member_ids": [alice["id"]]}, schema="Chat")["json"]
|
||||
other_att = upload(client, alice, other, png)["json"]
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages"
|
||||
for ids in ([other_att["id"]], [str(uuid.uuid4())]):
|
||||
client.call("POST", path, 404, actor=alice, body={"text": "", "attachment_ids": ids},
|
||||
headers={"Idempotency-Key": uuid.uuid4().hex})
|
||||
for body in ({"text": "x", "attachment_ids": [att["id"], att["id"]]},
|
||||
{"text": "x", "attachment_ids": None}, {"text": None, "attachment_ids": [att["id"]]},
|
||||
{"attachment_ids": [att["id"]]}, {"text": 42}, {"text": "x", "attachment_ids": {}},
|
||||
{"text": "x", "attachment_ids": [str(uuid.uuid4()) for _ in range(11)]}):
|
||||
client.call("POST", path, 400, actor=alice, body=body, headers={"Idempotency-Key": uuid.uuid4().hex})
|
||||
key = uuid.uuid4().hex
|
||||
first = client.message(bob, chat, "", key=key, attachments=[att["id"]])["json"]
|
||||
check(client.message(bob, chat, "", key=key, attachments=[att["id"]])["json"] == first, "attachment replay changed")
|
||||
client.call("POST", path, 409, actor=bob, body={"text": "", "attachment_ids": [jpg["id"]]}, headers={"Idempotency-Key": key})
|
||||
text_key = uuid.uuid4().hex
|
||||
text = client.message(alice, chat, "text", key=text_key)["json"]
|
||||
replay = client.call("POST", path, 201, actor=alice, body={"attachment_ids": [], "text": "text"}, headers={"Idempotency-Key": text_key}, schema="Message")["json"]
|
||||
check(text == replay, "omitted attachment_ids and [] are not equivalent")
|
||||
object_key = psql("SELECT object_key FROM attachments WHERE id='" + att["id"] + "';")
|
||||
audit = compose("exec", "-T", "api1", "/messenger", "s3-audit", object_key)
|
||||
# Explicitly stop the replica that accepted this particular upload.
|
||||
uploaded = upload(client, alice, chat, png)
|
||||
service = {"api-1": "api1", "api-2": "api2"}[uploaded["headers"]["X-Instance-Id"]]
|
||||
try:
|
||||
compose("stop", "--timeout", "3", service)
|
||||
wait_ready(client)
|
||||
# One ready response can arrive before the monitor removes the stopped
|
||||
# replica. Allow the documented ten-second failover budget, then check
|
||||
# the download without retries so storage/content failures stay visible.
|
||||
deadline = time.monotonic() + 10
|
||||
while time.monotonic() < deadline:
|
||||
upstream = compose("exec", "-T", "nginx", "cat", "/run/nginx-upstream.conf")
|
||||
if "server " + service + ":8080" not in upstream:
|
||||
break
|
||||
time.sleep(.2)
|
||||
else:
|
||||
raise Failure("stopped API was not excluded within ten seconds")
|
||||
downloaded = client.call("GET", "/api/v1/attachments/" + uploaded["json"]["id"] + "/content", actor=bob)
|
||||
check(downloaded["bytes"] == png, "other replica lost uploaded file")
|
||||
check(downloaded["headers"]["X-Instance-Id"] != uploaded["headers"]["X-Instance-Id"], "upload replica was not stopped")
|
||||
finally:
|
||||
compose("start", service)
|
||||
wait_ready(client)
|
||||
result = {"png_jpeg": "PASS", "size_corruption_mime_pixels_filename": "PASS",
|
||||
"exact_size_uploads_through_lb": [6294458, 10485760],
|
||||
"corrupt_idat_and_missing_iend": "PASS", "pixel_limit_40005000_and_200000000": "PASS",
|
||||
"cross_chat_and_outsider": "PASS", "message_idempotency": "PASS",
|
||||
"wide_png_40000000_pixels": "PASS", "wide_memory_bytes": wide_metrics,
|
||||
"private_bucket_and_limited_credentials": "PASS", "upload_replica_stopped_download_equal": True}
|
||||
(ROOT / "evidence").mkdir(exist_ok=True)
|
||||
(ROOT / "evidence/attachments-grade4.json").write_text(json.dumps(result, indent=2) + "\n")
|
||||
print(audit)
|
||||
print("PASS: attachment validation, authorization, idempotency and cross-replica download")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Interrupt upload, fail S3 and metadata writes, then verify grace cleanup."""
|
||||
import json
|
||||
import socket
|
||||
import time
|
||||
import uuid
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from verify_attachments import ROOT, Client, check, compose, psql, run, upload, wait_ready
|
||||
|
||||
|
||||
def stat(key):
|
||||
return json.loads(compose("exec", "-T", "api1", "/messenger", "s3-stat", key))
|
||||
|
||||
|
||||
def cleanup():
|
||||
return json.loads(compose("exec", "-T", "-e", "CLEANUP_GRACE_SECONDS=120", "cleanup", "/messenger", "cleanup"))
|
||||
|
||||
|
||||
def main():
|
||||
client = Client(timeout=30)
|
||||
actor, _, _, chat = client.bootstrap()
|
||||
raw = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
good = upload(client, actor, chat, raw)["json"]
|
||||
ready_key = psql("SELECT object_key FROM attachments WHERE id='" + good["id"] + "';")
|
||||
count_before = psql("SELECT count(*) FROM attachments;")
|
||||
# Abort a real HTTP upload with a body shorter than its Content-Length.
|
||||
address = urlsplit(client.base)
|
||||
with socket.create_connection((address.hostname, address.port or 80), timeout=5) as connection:
|
||||
headers = ("POST /api/v1/chats/" + chat["id"] + "/attachments?filename=interrupted.png HTTP/1.1\r\n"
|
||||
"Host: localhost\r\nAuthorization: Bearer " + actor["token"] + "\r\n"
|
||||
"Content-Type: image/png\r\nContent-Length: " + str(len(raw) + 1000) + "\r\nConnection: close\r\n\r\n")
|
||||
connection.sendall(headers.encode() + raw[:40])
|
||||
time.sleep(1)
|
||||
check(psql("SELECT count(*) FROM attachments;") == count_before, "interrupted upload left ready metadata")
|
||||
try:
|
||||
compose("stop", "s3")
|
||||
client.call("GET", "/health/ready", 503)
|
||||
upload(client, actor, chat, raw, expected=503)
|
||||
client.call("GET", "/api/v1/users/me", 401)
|
||||
finally:
|
||||
compose("start", "s3")
|
||||
wait_ready(client)
|
||||
check(client.call("GET", "/api/v1/attachments/" + good["id"] + "/content", actor=actor)["bytes"] == raw,
|
||||
"S3 restart lost original")
|
||||
constraint = "injected_" + uuid.uuid4().hex
|
||||
failed_keys = []
|
||||
try:
|
||||
psql("ALTER TABLE attachments ADD CONSTRAINT " + constraint + " CHECK (filename NOT LIKE 'failure-%') NOT VALID;")
|
||||
for number in range(2):
|
||||
before = set(psql("SELECT object_key FROM attachment_uploads;").splitlines())
|
||||
upload(client, actor, chat, raw, filename="failure-%d.png" % number, expected=503)
|
||||
after = set(psql("SELECT object_key FROM attachment_uploads;").splitlines())
|
||||
new = after - before
|
||||
check(len(new) == 1, "failed metadata write did not leave one durable intent")
|
||||
key = new.pop()
|
||||
check(stat(key)["exists"], "failure was not injected after successful S3 PUT")
|
||||
failed_keys.append(key)
|
||||
finally:
|
||||
psql("ALTER TABLE attachments DROP CONSTRAINT IF EXISTS " + constraint + ";")
|
||||
# Also cover an object whose intent is missing, as with late S3 completion.
|
||||
psql("DELETE FROM attachment_uploads WHERE object_key='" + failed_keys[1] + "';")
|
||||
fresh = cleanup()
|
||||
check(all(stat(key)["exists"] for key in failed_keys), "fresh objects removed before safe grace")
|
||||
print("Waiting 122 seconds for safe cleanup grace; fresh objects were preserved.", flush=True)
|
||||
deadline = time.monotonic() + 122
|
||||
while time.monotonic() < deadline:
|
||||
time.sleep(max(0, min(1, deadline - time.monotonic())))
|
||||
cleaned = cleanup()
|
||||
check(all(not stat(key)["exists"] for key in failed_keys), "orphan objects survived cleanup")
|
||||
check(stat(ready_key)["exists"], "cleanup deleted ready referenced data")
|
||||
repeat = cleanup()
|
||||
check(repeat["removed_objects"] == 0 and repeat["removed_intents"] == 0, "cleanup repeat is not idempotent")
|
||||
result = {"partial_upload": "PASS", "s3_outage_503_and_recovery": "PASS",
|
||||
"metadata_failure_after_put": "PASS", "orphan_without_intent": "PASS",
|
||||
"grace_seconds": 120, "fresh_preserved": True, "ready_preserved": True,
|
||||
"fresh_sweep": fresh, "cleanup": cleaned, "repeat": repeat}
|
||||
(ROOT / "evidence/upload-recovery.json").write_text(json.dumps(result, indent=2) + "\n")
|
||||
print("PASS:", json.dumps(result))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
Reference in New Issue
Block a user