forked from mtusi2026/mtusi2026lab4
Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b158520730 | ||
|
|
1bede8e5c0 | ||
|
|
e721df0248 | ||
|
|
1a03cc1a3d |
+3
-13
@@ -1,20 +1,10 @@
|
||||
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
|
||||
# These are logical interface suggestions; map actual framework names in REPORT.md.
|
||||
APP_HOST=0.0.0.0
|
||||
APP_PORT=8080
|
||||
APP_ENV=lab
|
||||
SESSION_TTL_SECONDS=3600
|
||||
# Choose credentials locally. File paths contain no secret values.
|
||||
DATABASE_URL_FILE=/run/secrets/database_url
|
||||
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
|
||||
SESSION_STORE=postgres
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
S3_ENDPOINT=http://s3:9000
|
||||
API_REPLICAS=2
|
||||
S3_REGION=us-east-1
|
||||
S3_BUCKET=messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
CLEANUP_GRACE_SECONDS=600
|
||||
CLEANUP_INTERVAL_SECONDS=300
|
||||
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
FROM golang:1.26-alpine@sha256:c95332c2af86b6d89b91bd0500f4b9529ccbd090a0d1855c6d1ceaa142ae8615 AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY *.go ./
|
||||
COPY migrations ./migrations
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/messenger .
|
||||
|
||||
FROM alpine:3.21@sha256:ce64758a109eb420d874a118f87920e625e12d3634e03b4a5573fd9f6e5d3507
|
||||
RUN adduser -D -u 10001 app
|
||||
WORKDIR /app
|
||||
COPY --from=build /out/messenger /app/messenger
|
||||
USER 10001
|
||||
EXPOSE 8080
|
||||
ENV APP_HOST=0.0.0.0 APP_PORT=8080 APP_ENV=lab
|
||||
STOPSIGNAL SIGTERM
|
||||
HEALTHCHECK --interval=5s --timeout=3s --start-period=10s --retries=3 \
|
||||
CMD wget -qO- http://127.0.0.1:8080/health/ready >/dev/null || exit 1
|
||||
ENTRYPOINT ["/app/messenger"]
|
||||
@@ -1,37 +1,112 @@
|
||||
# Отчёт — лабораторная 4
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Березин Герман / БВТ2304
|
||||
- Целевая оценка: 4
|
||||
- Commit/PR: после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: Go 1.26, pgx, go-redis, minio-go v7 (S3-клиент), HAProxy 3.0, PostgreSQL 17, Valkey 8.1, SeaweedFS (S3); код lab3 перенесён за основу; ИИ-ассистент использован при разработке
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Требуются Docker 29 с Compose v2 и Python 3.9+.
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
sh scripts/mksecrets.sh
|
||||
docker compose --env-file .env up -d --build --wait
|
||||
make check && make test
|
||||
```
|
||||
|
||||
Порядок старта: postgres, valkey, s3 → `migrate` (однократно) → `s3-init` (идемпотентное создание приватного bucket) → `api` × 2 → `lb`; рядом работает `janitor`. Публичный адрес: `http://localhost:8080`.
|
||||
|
||||
Остановка без потери данных: `docker compose --env-file .env stop`. Намеренное удаление учебных данных: `docker compose --env-file .env down -v`.
|
||||
|
||||
Проверки:
|
||||
|
||||
```sh
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
python3 tests/attachments.py
|
||||
python3 tests/s3_checks.py
|
||||
python3 tests/upload_memory.py --output evidence/upload-memory.json
|
||||
python3 tests/idempotency.py
|
||||
python3 tests/scenario.py --mode failover-each --output evidence/failover-each.json
|
||||
python3 tests/scenario.py --mode rolling --output evidence/rolling.json
|
||||
```
|
||||
|
||||
После `make test-failover` остановленную реплику восстанавливают: `docker compose --env-file .env up -d --wait`.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Клиент → HAProxy → `api` × 2 → PostgreSQL (метаданные) и SeaweedFS S3 (байты). Valkey остаётся общим кэшем сессий, источник истины сессий — PostgreSQL.
|
||||
|
||||
- **Хранилище.** S3-сервер — SeaweedFS (образ закреплён по digest). Образы MinIO недоступны для скачивания, поэтому выбран SeaweedFS. Bucket `messenger-attachments` приватный, данные лежат в volume `s3_data`. Порты S3 наружу не публикуются, в хост-сеть выходит только `lb`.
|
||||
- **Модель.** Таблица `attachments` (id, chat_id, owner_id, filename, content_type, size_bytes, sha256, object_key, created_at) и `message_attachments` (message_id, attachment_id, position). Object key — `attachments/<uuid>`, задаётся сервером; `filename` только отображается и проверяется (1–255 символов, без `/`, `\`, NUL и управляющих символов).
|
||||
- **Загрузка.** Сырое тело PNG/JPEG идёт потоком в S3 без накопления в памяти API и без локальных файлов. Проверки: Content-Type → 415; `Content-Length` и `MaxBytesReader` до 10 MiB → 413; magic bytes → 415 (подмена MIME); после записи — заголовок изображения (`image.DecodeConfig`, без декодирования пикселей), совпадение формата с Content-Type, хвост файла (IEND/EOI) → 415 для повреждённых и обрезанных файлов; ≤ 40 млн пикселей → 413. SHA-256 и размер считаются во время потока. При любой ошибке объект удаляется. 201 `ready` возвращается только после записи объекта и строки в PostgreSQL.
|
||||
- **Скачивание.** Доступ только участникам чата (JOIN с `chat_members`), иначе 404. Первый блок читается до отправки заголовков, поэтому сбой S3 даёт 503, а не оборванный ответ 200. Redirect нет.
|
||||
- **Сообщения.** `attachment_ids` (до 10, уникальные); вложение должно принадлежать этому чату, иначе 404; пустой `text` допустим при наличии вложений. Хеш тела идемпотентности включает `attachment_ids`, для сообщений без вложений он совпадает с lab3.
|
||||
- **Права S3.** API и janitor используют отдельные ключи (`Read/Write/List` только на свой bucket). Административные ключи есть только у `s3-init`, который создаёт bucket (повторный запуск ничего не меняет). Секреты генерирует `scripts/mksecrets.sh`, в Git их нет.
|
||||
- **Сети.** `s3` находится в сети `storage` вместе с `api`, `janitor`, `s3-init`; `postgres`, `valkey`, `lb` — в `backend`. Админ-интерфейсы SeaweedFS (master, filer) не публикуются.
|
||||
- Остальное перенесено из lab3: HAProxy с проверками готовности, идемпотентность, отдельные миграции, graceful shutdown.
|
||||
|
||||
### Согласованность БД и S3
|
||||
|
||||
Порядок записи: объект в S3 → строка в PostgreSQL.
|
||||
|
||||
| Сбой | Результат | Компенсация |
|
||||
| --- | --- | --- |
|
||||
| Клиент оборвал загрузку | запись в S3 не завершается, ответа нет | объект удаляется в обработчике |
|
||||
| Невалидное изображение, превышение лимитов | объект записан, но не принят | объект удаляется до ответа |
|
||||
| Ошибка PostgreSQL после S3 PUT | 503 | объект удаляется в обработчике |
|
||||
| Падение процесса между PUT и INSERT | остаётся объект без строки | `janitor` удаляет такие объекты |
|
||||
|
||||
`janitor` (`messenger janitor`) раз в `CLEANUP_INTERVAL_SECONDS` (300 с) перебирает объекты под префиксом `attachments/`, проверяет их по `attachments.object_key` и удаляет объекты без записи, старше `CLEANUP_GRACE_SECONDS` (600 с). Grace period защищает объекты текущих загрузок. Разовый запуск: `docker compose run --rm janitor cleanup`. Строки без объекта автоматически не чистятся: файл читается как 404/503, строка сохраняется.
|
||||
|
||||
### Лимиты памяти и потоковая обработка
|
||||
|
||||
| Параметр | Значение |
|
||||
| --- | --- |
|
||||
| Размер файла | не более 10 MiB (`MAX_UPLOAD_BYTES`), проверяется до чтения тела |
|
||||
| Пиксели | не более 40 млн (`MAX_IMAGE_PIXELS`), декодируется только заголовок |
|
||||
| Параллельные загрузки на реплику | не более 10 (`UPLOAD_CONCURRENCY`), иначе 429 с `Retry-After` |
|
||||
| Буфер на загрузку | не более одной части multipart S3 (5 MiB) и 256 KiB для проверки заголовка |
|
||||
| Таймауты API | ReadHeader 5 s, Read 10 s, Write 15 s, запись в S3 до 60 s, чтение из S3 до 30 s |
|
||||
| Лимиты контейнеров | api 192 MiB и `GOMEMLIMIT=150MiB`; s3 512 MiB; postgres 512 MiB; остальные 64–128 MiB |
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Публичный smoke | `make test` | PASS | evidence/smoke.txt |
|
||||
| Сохранность после restart (метаданные и оригинал) | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS; перезапускаются postgres, valkey, s3, api, lb, janitor | evidence/persistence.txt |
|
||||
| Загрузка → остановка реплики → скачивание через другую | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` (обе реплики) | PASS, 0 ошибок записи, восстановление около 3.5 s, оригинал читается | evidence/failover-1.txt, evidence/failover-2.txt |
|
||||
| Большой файл, повреждённое изображение, подмена MIME, чужое и чужого чата attachment_id | `python3 tests/attachments.py` | 413 / 415 / 415 / 404 / 404, ни одного 500 | evidence/attachments.txt |
|
||||
| Приватный bucket вне API, права ключей | `python3 tests/s3_checks.py` | анонимный GET/PUT/list и неверные ключи → 403; ключ API не создаёт bucket; наружу опубликован только `lb` | evidence/s3-checks.txt |
|
||||
| Недоступность S3 | `python3 tests/s3_checks.py` | при остановленном S3 загрузка и скачивание → 503, метаданные и текстовые сообщения работают, после запуска всё восстанавливается | evidence/s3-checks.txt |
|
||||
| Очистка неучтённых объектов | `python3 tests/s3_checks.py` | свежий объект сохраняется, объект старше grace удаляется, используемый объект остаётся | evidence/s3-checks.txt |
|
||||
| ≥10 одновременных загрузок, пик памяти | `python3 tests/upload_memory.py` | 12 потоков × 3 файла по 9.3 MiB, 36 из 36 успешно, пик api 103–120 MiB при лимите 192 MiB | evidence/upload-memory.json |
|
||||
| Идемпотентность из lab3 | `python3 tests/idempotency.py` | 32 параллельных запроса → 1 сообщение | evidence/idempotency.txt |
|
||||
| Отказы каждой реплики и rolling update под нагрузкой | `tests/scenario.py` | RPS, ошибки, восстановление | evidence/failover-each.json, evidence/rolling.json |
|
||||
| Идемпотентный bootstrap bucket | повторный `docker compose up` | `s3-init` завершается с кодом 0, bucket не пересоздаётся | `docker compose logs s3-init` |
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Стенд: один хост; лимиты контейнеров как выше. Нагрузка сообщений — 8 потоков, closed-loop, при ошибке повтор с тем же ключом. Загрузки — 12 потоков по 3 файла PNG 9 722 608 байт (несжимаемый шум).
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
| Загрузка 12×3 файла параллельно | память API ограничена | пик api 120 / 103 MiB (в покое около 62 MiB), ошибок нет | потерь нет |
|
||||
| Остановка S3 | 503 на файлах, текст работает | как ожидалось | после запуска S3 около 20–30 s до готовности записи, файлы целы |
|
||||
| Остановка каждой API-реплики (под нагрузкой сообщений) | продолжение чтения и записи | 0 неуспешных попыток, RPS около 440 против 520 до отказа | возврат реплики в ротацию около 6.5–7 s после старта |
|
||||
| Rolling update | без простоя | 4 неуспешные попытки, RPS около 517 | старые реплики заменены и не отвечают, дубликатов нет |
|
||||
| Restart всего стенда | данные и оригинал сохраняются | PASS persistence | прежний токен действителен |
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
- Целостность изображения проверяется по заголовку и хвосту файла, но не полным декодированием: повреждение в середине файла с корректными заголовком и концом не обнаруживается. Полное декодирование потребовало бы памяти, пропорциональной размеру изображения.
|
||||
- Падение процесса между S3 PUT и INSERT оставляет объект до очистки `janitor`; задержка до удаления — grace period плюс интервал (до 15 минут).
|
||||
- Если S3 перезапускается, запись становится доступной через 20–30 s после старта; healthcheck S3 учитывает это.
|
||||
- Filer SeaweedFS слушает внутри сети `storage` без аутентификации. Доступ к нему имеют только контейнеры `api`, `janitor`, `s3-init`; наружу порт не публикуется. S3-аутентификация действует на порту 9000.
|
||||
- HAProxy, PostgreSQL, Valkey и S3 — единые точки отказа; репликации нет, выживание отказа этих узлов не заявляется.
|
||||
- Измерение памяти — выборка `docker stats` раз в секунду, это оценка, а не точный пик.
|
||||
- Нагрузочные генераторы Python (closed-loop) ограничены клиентом, RPS — ориентир.
|
||||
- Секреты лежат в `./secrets` вне Git и читаются всеми локально, чтобы их читал процесс с UID 10001; это допустимо только для локального стенда.
|
||||
|
||||
+430
@@ -0,0 +1,430 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"hash"
|
||||
"image"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
const (
|
||||
objectPrefix = "attachments/"
|
||||
probeHead = 256 << 10
|
||||
tailLen = 12
|
||||
)
|
||||
|
||||
var (
|
||||
pngMagic = []byte{0x89, 'P', 'N', 'G', '\r', '\n', 0x1a, '\n'}
|
||||
pngTail = []byte{0, 0, 0, 0, 'I', 'E', 'N', 'D', 0xae, 0x42, 0x60, 0x82}
|
||||
jpegMagic = []byte{0xff, 0xd8, 0xff}
|
||||
jpegTail = []byte{0xff, 0xd9}
|
||||
)
|
||||
|
||||
type limits struct {
|
||||
maxBytes int64
|
||||
maxPixels int64
|
||||
uploadSlots int
|
||||
}
|
||||
|
||||
type attachment struct {
|
||||
ID, ChatID, OwnerID, Filename, ContentType, SHA256, Key string
|
||||
Size int64
|
||||
Created time.Time
|
||||
}
|
||||
|
||||
func (x *attachment) json() map[string]any {
|
||||
return map[string]any{
|
||||
"id": x.ID,
|
||||
"chat_id": x.ChatID,
|
||||
"owner_id": x.OwnerID,
|
||||
"filename": x.Filename,
|
||||
"content_type": x.ContentType,
|
||||
"size_bytes": x.Size,
|
||||
"sha256": x.SHA256,
|
||||
"status": "ready",
|
||||
"variants": []any{},
|
||||
"error_code": nil,
|
||||
"created_at": formatTime(x.Created),
|
||||
"updated_at": formatTime(x.Created),
|
||||
}
|
||||
}
|
||||
|
||||
type probe struct {
|
||||
sum hash.Hash
|
||||
size int64
|
||||
head []byte
|
||||
tail []byte
|
||||
}
|
||||
|
||||
func newProbe() *probe {
|
||||
return &probe{sum: sha256.New()}
|
||||
}
|
||||
|
||||
func (p *probe) Write(b []byte) (int, error) {
|
||||
p.sum.Write(b)
|
||||
p.size += int64(len(b))
|
||||
if room := probeHead - len(p.head); room > 0 {
|
||||
if room > len(b) {
|
||||
room = len(b)
|
||||
}
|
||||
p.head = append(p.head, b[:room]...)
|
||||
}
|
||||
p.tail = append(p.tail, b...)
|
||||
if len(p.tail) > tailLen {
|
||||
p.tail = append(p.tail[:0], p.tail[len(p.tail)-tailLen:]...)
|
||||
}
|
||||
return len(b), nil
|
||||
}
|
||||
|
||||
func (p *probe) digest() string {
|
||||
return hex.EncodeToString(p.sum.Sum(nil))
|
||||
}
|
||||
|
||||
var (
|
||||
errNotImage = errors.New("not an image")
|
||||
errTooBig = errors.New("too large")
|
||||
)
|
||||
|
||||
func (p *probe) verify(contentType string, maxPixels int64) error {
|
||||
cfg, format, err := image.DecodeConfig(bytes.NewReader(p.head))
|
||||
if err != nil {
|
||||
return errNotImage
|
||||
}
|
||||
var tail []byte
|
||||
switch contentType {
|
||||
case "image/png":
|
||||
if format != "png" {
|
||||
return errNotImage
|
||||
}
|
||||
tail = pngTail
|
||||
case "image/jpeg":
|
||||
if format != "jpeg" {
|
||||
return errNotImage
|
||||
}
|
||||
tail = jpegTail
|
||||
}
|
||||
if !bytes.HasSuffix(p.tail, tail) {
|
||||
return errNotImage
|
||||
}
|
||||
if int64(cfg.Width)*int64(cfg.Height) > maxPixels {
|
||||
return errTooBig
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func magicMatches(contentType string, head []byte) bool {
|
||||
switch contentType {
|
||||
case "image/png":
|
||||
return bytes.HasPrefix(head, pngMagic)
|
||||
case "image/jpeg":
|
||||
return bytes.HasPrefix(head, jpegMagic)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func validFilename(name string) bool {
|
||||
n := utf8.RuneCountInString(name)
|
||||
if n < 1 || n > 255 || !utf8.ValidString(name) {
|
||||
return false
|
||||
}
|
||||
for _, r := range name {
|
||||
if r == '/' || r == '\\' || r == 0 || unicode.IsControl(r) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
type trackedReader struct {
|
||||
r io.Reader
|
||||
err error
|
||||
}
|
||||
|
||||
func (t *trackedReader) Read(b []byte) (int, error) {
|
||||
n, err := t.r.Read(b)
|
||||
if err != nil && err != io.EOF && t.err == nil {
|
||||
t.err = err
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
func isTooLarge(err error) bool {
|
||||
var mbe *http.MaxBytesError
|
||||
return errors.As(err, &mbe)
|
||||
}
|
||||
|
||||
func (a *App) dropObject(key string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := a.store.Remove(ctx, key); err != nil {
|
||||
a.log.Warn("object cleanup failed", "key", key, "err", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) handleUploadAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
reqID := reqIDFrom(r)
|
||||
u, _ := a.authUser(w, r)
|
||||
if u == nil {
|
||||
return
|
||||
}
|
||||
chatID := r.PathValue("chat_id")
|
||||
if !isUUID(chatID) {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "invalid chat_id")
|
||||
return
|
||||
}
|
||||
ok, err := a.isMember(r.Context(), chatID, u.ID)
|
||||
if err != nil {
|
||||
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
|
||||
return
|
||||
}
|
||||
if !ok {
|
||||
a.writeErr(w, reqID, 404, "not_found", "chat not found")
|
||||
return
|
||||
}
|
||||
select {
|
||||
case a.uploads <- struct{}{}:
|
||||
defer func() { <-a.uploads }()
|
||||
default:
|
||||
w.Header().Set("Retry-After", "1")
|
||||
a.writeErr(w, reqID, 429, "rate_limited", "too many concurrent uploads")
|
||||
return
|
||||
}
|
||||
filename := r.URL.Query().Get("filename")
|
||||
if !validFilename(filename) {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "invalid filename")
|
||||
return
|
||||
}
|
||||
contentType := strings.ToLower(strings.TrimSpace(strings.Split(r.Header.Get("Content-Type"), ";")[0]))
|
||||
if contentType != "image/png" && contentType != "image/jpeg" {
|
||||
a.writeErr(w, reqID, 415, "unsupported_media_type", "Content-Type must be image/png or image/jpeg")
|
||||
return
|
||||
}
|
||||
if r.ContentLength > a.limits.maxBytes {
|
||||
a.writeErr(w, reqID, 413, "payload_too_large", "file too large")
|
||||
return
|
||||
}
|
||||
body := &trackedReader{r: http.MaxBytesReader(w, r.Body, a.limits.maxBytes)}
|
||||
head := make([]byte, 16)
|
||||
n, _ := io.ReadFull(body, head)
|
||||
if body.err != nil {
|
||||
if isTooLarge(body.err) {
|
||||
a.writeErr(w, reqID, 413, "payload_too_large", "file too large")
|
||||
} else {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "upload interrupted")
|
||||
}
|
||||
return
|
||||
}
|
||||
head = head[:n]
|
||||
if !magicMatches(contentType, head) {
|
||||
a.writeErr(w, reqID, 415, "unsupported_media_type", "content does not match image type")
|
||||
return
|
||||
}
|
||||
|
||||
id := newUUID()
|
||||
key := objectPrefix + id
|
||||
pr := newProbe()
|
||||
size := r.ContentLength
|
||||
if size < 0 {
|
||||
size = -1
|
||||
}
|
||||
src := io.TeeReader(io.MultiReader(bytes.NewReader(head), body), pr)
|
||||
putCtx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
if err := a.store.Put(putCtx, key, src, size, contentType); err != nil {
|
||||
a.dropObject(key)
|
||||
switch {
|
||||
case isTooLarge(body.err):
|
||||
a.writeErr(w, reqID, 413, "payload_too_large", "file too large")
|
||||
case body.err != nil:
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "upload interrupted")
|
||||
default:
|
||||
a.log.Error("s3 put failed", "request_id", reqID, "err", err.Error())
|
||||
a.writeErr(w, reqID, 503, "unavailable", "storage unavailable")
|
||||
}
|
||||
return
|
||||
}
|
||||
if err := pr.verify(contentType, a.limits.maxPixels); err != nil {
|
||||
a.dropObject(key)
|
||||
if err == errTooBig {
|
||||
a.writeErr(w, reqID, 413, "payload_too_large", "image too large")
|
||||
} else {
|
||||
a.writeErr(w, reqID, 415, "unsupported_media_type", "invalid image")
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
att := &attachment{
|
||||
ID: id, ChatID: chatID, OwnerID: u.ID, Filename: filename, ContentType: contentType,
|
||||
SHA256: pr.digest(), Key: key, Size: pr.size, Created: nowUTC(),
|
||||
}
|
||||
ctx, cancelDB := dbCtx(r.Context())
|
||||
defer cancelDB()
|
||||
_, err = a.db.Exec(ctx, `
|
||||
insert into attachments(id, chat_id, owner_id, filename, content_type, size_bytes, sha256, object_key, created_at)
|
||||
values($1::uuid, $2::uuid, $3::uuid, $4, $5, $6, $7, $8, $9)`,
|
||||
att.ID, att.ChatID, att.OwnerID, att.Filename, att.ContentType, att.Size, att.SHA256, att.Key, att.Created)
|
||||
if err != nil {
|
||||
a.dropObject(key)
|
||||
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
|
||||
return
|
||||
}
|
||||
a.writeJSON(w, 201, att.json())
|
||||
}
|
||||
|
||||
func (a *App) loadAttachment(ctx context.Context, id, userID string) (*attachment, error) {
|
||||
var x attachment
|
||||
err := a.db.QueryRow(ctx, `
|
||||
select a.id::text, a.chat_id::text, a.owner_id::text, a.filename, a.content_type,
|
||||
a.size_bytes, a.sha256, a.object_key, a.created_at
|
||||
from attachments a
|
||||
join chat_members m on m.chat_id = a.chat_id and m.user_id = $2::uuid
|
||||
where a.id = $1::uuid`, id, userID).
|
||||
Scan(&x.ID, &x.ChatID, &x.OwnerID, &x.Filename, &x.ContentType, &x.Size, &x.SHA256, &x.Key, &x.Created)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &x, nil
|
||||
}
|
||||
|
||||
func (a *App) attachmentFor(w http.ResponseWriter, r *http.Request) *attachment {
|
||||
reqID := reqIDFrom(r)
|
||||
u, _ := a.authUser(w, r)
|
||||
if u == nil {
|
||||
return nil
|
||||
}
|
||||
id := r.PathValue("attachment_id")
|
||||
if !isUUID(id) {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "invalid attachment_id")
|
||||
return nil
|
||||
}
|
||||
ctx, cancel := dbCtx(r.Context())
|
||||
defer cancel()
|
||||
att, err := a.loadAttachment(ctx, id, u.ID)
|
||||
if err != nil {
|
||||
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
|
||||
return nil
|
||||
}
|
||||
if att == nil {
|
||||
a.writeErr(w, reqID, 404, "not_found", "attachment not found")
|
||||
return nil
|
||||
}
|
||||
return att
|
||||
}
|
||||
|
||||
func (a *App) handleGetAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
if att := a.attachmentFor(w, r); att != nil {
|
||||
a.writeJSON(w, 200, att.json())
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) handleDownloadAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
reqID := reqIDFrom(r)
|
||||
if v := r.URL.Query().Get("variant"); v != "" && v != "original" {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "invalid variant")
|
||||
return
|
||||
}
|
||||
att := a.attachmentFor(w, r)
|
||||
if att == nil {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
|
||||
defer cancel()
|
||||
obj, info, err := a.store.Get(ctx, att.Key)
|
||||
if err != nil {
|
||||
if isNoSuchKey(err) {
|
||||
a.writeErr(w, reqID, 404, "not_found", "attachment not found")
|
||||
return
|
||||
}
|
||||
a.log.Error("s3 get failed", "request_id", reqID, "err", err.Error())
|
||||
a.writeErr(w, reqID, 503, "unavailable", "storage unavailable")
|
||||
return
|
||||
}
|
||||
defer obj.Close()
|
||||
first := make([]byte, 32<<10)
|
||||
n, err := io.ReadFull(obj, first)
|
||||
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF {
|
||||
a.log.Error("s3 read failed", "request_id", reqID, "err", err.Error())
|
||||
a.writeErr(w, reqID, 503, "unavailable", "storage unavailable")
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", att.ContentType)
|
||||
w.Header().Set("Content-Length", strconv.FormatInt(info.Size, 10))
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.WriteHeader(200)
|
||||
_, _ = w.Write(first[:n])
|
||||
_, _ = io.Copy(w, obj)
|
||||
}
|
||||
|
||||
func cleanupOrphans(ctx context.Context, db *pgxpool.Pool, store *Store, grace time.Duration, log *slog.Logger) (int, error) {
|
||||
cutoff := time.Now().Add(-grace)
|
||||
removed := 0
|
||||
flush := func(keys []string) error {
|
||||
if len(keys) == 0 {
|
||||
return nil
|
||||
}
|
||||
rows, err := db.Query(ctx, `select object_key from attachments where object_key = any($1)`, keys)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
known := map[string]bool{}
|
||||
for rows.Next() {
|
||||
var k string
|
||||
if err := rows.Scan(&k); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
known[k] = true
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, k := range keys {
|
||||
if known[k] {
|
||||
continue
|
||||
}
|
||||
if err := store.Remove(ctx, k); err != nil {
|
||||
return err
|
||||
}
|
||||
log.Info("orphan object removed", "key", k)
|
||||
removed++
|
||||
}
|
||||
return nil
|
||||
}
|
||||
var batch []string
|
||||
for obj := range store.List(ctx) {
|
||||
if obj.Err != nil {
|
||||
return removed, obj.Err
|
||||
}
|
||||
if obj.LastModified.After(cutoff) {
|
||||
continue
|
||||
}
|
||||
batch = append(batch, obj.Key)
|
||||
if len(batch) == 500 {
|
||||
if err := flush(batch); err != nil {
|
||||
return removed, err
|
||||
}
|
||||
batch = batch[:0]
|
||||
}
|
||||
}
|
||||
return removed, flush(batch)
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"golang.org/x/crypto/argon2"
|
||||
)
|
||||
|
||||
const (
|
||||
argonTime = 2
|
||||
argonMemory = 19456
|
||||
argonThreads = 1
|
||||
argonKeyLen = 32
|
||||
argonSaltLen = 16
|
||||
)
|
||||
|
||||
func hashPassword(password string) (string, error) {
|
||||
salt := make([]byte, argonSaltLen)
|
||||
if _, err := rand.Read(salt); err != nil {
|
||||
return "", err
|
||||
}
|
||||
hash := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
|
||||
b64Salt := base64.RawStdEncoding.EncodeToString(salt)
|
||||
b64Hash := base64.RawStdEncoding.EncodeToString(hash)
|
||||
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
|
||||
argon2.Version, argonMemory, argonTime, argonThreads, b64Salt, b64Hash), nil
|
||||
}
|
||||
|
||||
func checkPassword(encoded, password string) (bool, error) {
|
||||
parts := strings.Split(encoded, "$")
|
||||
if len(parts) != 6 || parts[1] != "argon2id" {
|
||||
return false, errors.New("bad hash")
|
||||
}
|
||||
var version int
|
||||
if _, err := fmt.Sscanf(parts[2], "v=%d", &version); err != nil {
|
||||
return false, err
|
||||
}
|
||||
var memory, timeCost, threads int
|
||||
for _, p := range strings.Split(parts[3], ",") {
|
||||
kv := strings.SplitN(p, "=", 2)
|
||||
if len(kv) != 2 {
|
||||
return false, errors.New("bad params")
|
||||
}
|
||||
n, err := strconv.Atoi(kv[1])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
switch kv[0] {
|
||||
case "m":
|
||||
memory = n
|
||||
case "t":
|
||||
timeCost = n
|
||||
case "p":
|
||||
threads = n
|
||||
}
|
||||
}
|
||||
salt, err := base64.RawStdEncoding.DecodeString(parts[4])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
want, err := base64.RawStdEncoding.DecodeString(parts[5])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
got := argon2.IDKey([]byte(password), salt, uint32(timeCost), uint32(memory), uint8(threads), uint32(len(want)))
|
||||
return subtle.ConstantTimeCompare(got, want) == 1, nil
|
||||
}
|
||||
|
||||
func newToken() (string, []byte, string, error) {
|
||||
raw := make([]byte, 32)
|
||||
if _, err := rand.Read(raw); err != nil {
|
||||
return "", nil, "", err
|
||||
}
|
||||
token := base64.RawURLEncoding.EncodeToString(raw)
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
hash := make([]byte, len(sum))
|
||||
copy(hash, sum[:])
|
||||
return token, hash, hex.EncodeToString(hash), nil
|
||||
}
|
||||
|
||||
func tokenHash(token string) ([]byte, string) {
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
hash := make([]byte, len(sum))
|
||||
copy(hash, sum[:])
|
||||
return hash, hex.EncodeToString(hash)
|
||||
}
|
||||
|
||||
func parseBasic(header string) (username, password string, ok bool) {
|
||||
const prefix = "Basic "
|
||||
if !strings.HasPrefix(header, prefix) {
|
||||
return "", "", false
|
||||
}
|
||||
decoded, err := base64.StdEncoding.DecodeString(strings.TrimSpace(header[len(prefix):]))
|
||||
if err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
s := string(decoded)
|
||||
i := strings.IndexByte(s, ':')
|
||||
if i < 0 {
|
||||
return "", "", false
|
||||
}
|
||||
username = s[:i]
|
||||
password = s[i+1:]
|
||||
return username, password, true
|
||||
}
|
||||
|
||||
func parseBearer(header string) (string, bool) {
|
||||
const prefix = "Bearer "
|
||||
if !strings.HasPrefix(header, prefix) {
|
||||
return "", false
|
||||
}
|
||||
tok := strings.TrimSpace(header[len(prefix):])
|
||||
if tok == "" {
|
||||
return "", false
|
||||
}
|
||||
return tok, true
|
||||
}
|
||||
|
||||
func validPassword(p string) bool {
|
||||
n := utf8.RuneCountInString(p)
|
||||
return n >= 12 && n <= 128
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
type cachedSession struct {
|
||||
UserID string `json:"user_id"`
|
||||
ExpiresAt time.Time `json:"expires_at"`
|
||||
}
|
||||
|
||||
type Cache struct {
|
||||
rdb *redis.Client
|
||||
log *slog.Logger
|
||||
}
|
||||
|
||||
func openCache(log *slog.Logger) *Cache {
|
||||
url, err := readSecret("CACHE_URL", "CACHE_URL_FILE")
|
||||
if err != nil {
|
||||
log.Warn("cache disabled", "err", err.Error())
|
||||
return &Cache{log: log}
|
||||
}
|
||||
opt, err := redis.ParseURL(url)
|
||||
if err != nil {
|
||||
log.Warn("cache parse failed", "err", err.Error())
|
||||
return &Cache{log: log}
|
||||
}
|
||||
opt.DialTimeout = 200 * time.Millisecond
|
||||
opt.ReadTimeout = 200 * time.Millisecond
|
||||
opt.WriteTimeout = 200 * time.Millisecond
|
||||
opt.PoolTimeout = 200 * time.Millisecond
|
||||
rdb := redis.NewClient(opt)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
|
||||
defer cancel()
|
||||
if err := rdb.Ping(ctx).Err(); err != nil {
|
||||
log.Warn("cache ping failed, will bypass", "err", err.Error())
|
||||
}
|
||||
return &Cache{rdb: rdb, log: log}
|
||||
}
|
||||
|
||||
func (c *Cache) Close() {
|
||||
if c.rdb != nil {
|
||||
_ = c.rdb.Close()
|
||||
}
|
||||
}
|
||||
|
||||
func sessKey(hashHex string) string {
|
||||
return "sess:" + hashHex
|
||||
}
|
||||
|
||||
func (c *Cache) GetSession(ctx context.Context, hashHex string) (*cachedSession, bool) {
|
||||
if c.rdb == nil {
|
||||
return nil, false
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
|
||||
defer cancel()
|
||||
val, err := c.rdb.Get(ctx, sessKey(hashHex)).Result()
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
var s cachedSession
|
||||
if err := json.Unmarshal([]byte(val), &s); err != nil {
|
||||
return nil, false
|
||||
}
|
||||
if time.Now().UTC().After(s.ExpiresAt) {
|
||||
return nil, false
|
||||
}
|
||||
return &s, true
|
||||
}
|
||||
|
||||
func (c *Cache) SetSession(ctx context.Context, hashHex string, userID string, expiresAt time.Time) {
|
||||
if c.rdb == nil {
|
||||
return
|
||||
}
|
||||
ttl := time.Until(expiresAt)
|
||||
if ttl <= 0 {
|
||||
return
|
||||
}
|
||||
body, err := json.Marshal(cachedSession{UserID: userID, ExpiresAt: expiresAt.UTC()})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
|
||||
defer cancel()
|
||||
if err := c.rdb.Set(ctx, sessKey(hashHex), body, ttl).Err(); err != nil {
|
||||
c.log.Warn("cache set failed", "err", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Cache) DelSession(ctx context.Context, hashHex string) {
|
||||
if c.rdb == nil {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
|
||||
defer cancel()
|
||||
if err := c.rdb.Del(ctx, sessKey(hashHex)).Err(); err != nil {
|
||||
c.log.Warn("cache del failed", "err", err.Error())
|
||||
}
|
||||
}
|
||||
+244
@@ -0,0 +1,244 @@
|
||||
x-api-env: &api-env
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: 8080
|
||||
APP_ENV: ${APP_ENV:-lab}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
SESSION_STORE: postgres
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
|
||||
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||
UPLOAD_CONCURRENCY: ${UPLOAD_CONCURRENCY:-10}
|
||||
GOMEMLIMIT: 150MiB
|
||||
SHUTDOWN_TIMEOUT_SECONDS: 10
|
||||
DRAIN_SECONDS: 3
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17-alpine@sha256:b0f9560a2de083e2cc7382e75f808c7381a32852a7ec49117deedb300e552b24
|
||||
environment:
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
POSTGRES_DB: messenger
|
||||
secrets:
|
||||
- postgres_password
|
||||
volumes:
|
||||
- pg_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
networks:
|
||||
- backend
|
||||
restart: unless-stopped
|
||||
|
||||
valkey:
|
||||
image: valkey/valkey:8.1-alpine@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
|
||||
command: ["valkey-server", "--appendonly", "yes", "--save", "60", "1"]
|
||||
volumes:
|
||||
- valkey_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "valkey-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
networks:
|
||||
- backend
|
||||
restart: unless-stopped
|
||||
|
||||
s3:
|
||||
image: chrislusf/seaweedfs@sha256:c2658820e763294a9e9de7b36c81e59b879e3459148351aee4e31a126ca64860
|
||||
command:
|
||||
- server
|
||||
- -dir=/data
|
||||
- -ip=s3
|
||||
- -master.volumeSizeLimitMB=128
|
||||
- -volume.max=30
|
||||
- -s3
|
||||
- -s3.port=9000
|
||||
- -s3.config=/run/secrets/s3_config
|
||||
secrets:
|
||||
- s3_config
|
||||
volumes:
|
||||
- s3_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9333/dir/assign | grep -q fid"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
networks:
|
||||
- storage
|
||||
restart: unless-stopped
|
||||
|
||||
s3-init:
|
||||
image: mtusi-messenger:lab4
|
||||
command: ["s3-init"]
|
||||
environment:
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_admin_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_admin_secret_access_key
|
||||
secrets:
|
||||
- s3_admin_access_key_id
|
||||
- s3_admin_secret_access_key
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
networks:
|
||||
- storage
|
||||
restart: "no"
|
||||
|
||||
janitor:
|
||||
image: mtusi-messenger:lab4
|
||||
command: ["janitor"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
CLEANUP_GRACE_SECONDS: ${CLEANUP_GRACE_SECONDS:-600}
|
||||
CLEANUP_INTERVAL_SECONDS: ${CLEANUP_INTERVAL_SECONDS:-300}
|
||||
secrets:
|
||||
- database_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
disable: true
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 64M
|
||||
networks:
|
||||
- backend
|
||||
- storage
|
||||
restart: unless-stopped
|
||||
|
||||
migrate:
|
||||
build: .
|
||||
image: mtusi-messenger:lab4
|
||||
command: ["migrate"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
secrets:
|
||||
- database_url
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
networks:
|
||||
- backend
|
||||
restart: "no"
|
||||
|
||||
api:
|
||||
image: mtusi-messenger:lab4
|
||||
environment: *api-env
|
||||
secrets:
|
||||
- database_url
|
||||
- cache_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
valkey:
|
||||
condition: service_healthy
|
||||
stop_grace_period: 15s
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 192M
|
||||
networks:
|
||||
- backend
|
||||
- storage
|
||||
restart: unless-stopped
|
||||
|
||||
lb:
|
||||
image: haproxy:3.0-alpine@sha256:cf3af0facf7eb22b9fee463daf0fbc07ead740b8150696dce724264faf7b4318
|
||||
ports:
|
||||
- "127.0.0.1:8080:8080"
|
||||
volumes:
|
||||
- ./lb/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
depends_on:
|
||||
- api
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
networks:
|
||||
- backend
|
||||
restart: unless-stopped
|
||||
|
||||
secrets:
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
cache_url:
|
||||
file: ./secrets/cache_url
|
||||
s3_config:
|
||||
file: ./secrets/s3_config.json
|
||||
s3_admin_access_key_id:
|
||||
file: ./secrets/s3_admin_access_key_id
|
||||
s3_admin_secret_access_key:
|
||||
file: ./secrets/s3_admin_secret_access_key
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
|
||||
volumes:
|
||||
pg_data:
|
||||
valkey_data:
|
||||
s3_data:
|
||||
|
||||
networks:
|
||||
backend:
|
||||
driver: bridge
|
||||
storage:
|
||||
driver: bridge
|
||||
@@ -0,0 +1,57 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func readSecret(envKey, fileKey string) (string, error) {
|
||||
if v := os.Getenv(envKey); v != "" {
|
||||
return strings.TrimSpace(v), nil
|
||||
}
|
||||
path := os.Getenv(fileKey)
|
||||
if path == "" {
|
||||
return "", fmt.Errorf("%s or %s required", envKey, fileKey)
|
||||
}
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return strings.TrimSpace(string(b)), nil
|
||||
}
|
||||
|
||||
func openDB(ctx context.Context) (*pgxpool.Pool, error) {
|
||||
dsn, err := readSecret("DATABASE_URL", "DATABASE_URL_FILE")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg.MaxConns = 10
|
||||
cfg.MinConns = 2
|
||||
cfg.MaxConnLifetime = 5 * time.Minute
|
||||
cfg.ConnConfig.ConnectTimeout = 3 * time.Second
|
||||
|
||||
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pingCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
if err := pool.Ping(pingCtx); err != nil {
|
||||
pool.Close()
|
||||
return nil, err
|
||||
}
|
||||
return pool, nil
|
||||
}
|
||||
|
||||
func dbCtx(parent context.Context) (context.Context, context.CancelFunc) {
|
||||
return context.WithTimeout(parent, 3*time.Second)
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: attachment validation, limits and access control
|
||||
@@ -0,0 +1,2 @@
|
||||
{"observed_instances_before": 2, "successful_writes": 140, "failed_attempts": 0, "first_recovery_seconds": 3.46}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,2 @@
|
||||
{"observed_instances_before": 2, "successful_writes": 139, "failed_attempts": 0, "first_recovery_seconds": 3.453}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,71 @@
|
||||
{
|
||||
"baseline": {
|
||||
"window_seconds": 6,
|
||||
"successful_requests": 3132,
|
||||
"failed_attempts": 0,
|
||||
"successful_rps": 522.0,
|
||||
"last_error_offset_seconds": null,
|
||||
"instances": [
|
||||
"2eb97a202e04",
|
||||
"e1e2d306d0d7"
|
||||
]
|
||||
},
|
||||
"replicas": [
|
||||
{
|
||||
"stopped_replica": "2eb97a202e04",
|
||||
"during_failure": {
|
||||
"window_seconds": 15.34,
|
||||
"successful_requests": 6776,
|
||||
"failed_attempts": 0,
|
||||
"successful_rps": 441.8,
|
||||
"last_error_offset_seconds": null,
|
||||
"instances": [
|
||||
"2eb97a202e04",
|
||||
"e1e2d306d0d7"
|
||||
]
|
||||
},
|
||||
"seen_stopped_replica_after_stop": false,
|
||||
"stop_command_seconds": 3.34,
|
||||
"restore_seconds": 6.48,
|
||||
"after_restore": {
|
||||
"window_seconds": 7.0,
|
||||
"successful_requests": 3639,
|
||||
"failed_attempts": 0,
|
||||
"successful_rps": 519.9,
|
||||
"last_error_offset_seconds": null,
|
||||
"instances": [
|
||||
"2eb97a202e04",
|
||||
"e1e2d306d0d7"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"stopped_replica": "e1e2d306d0d7",
|
||||
"during_failure": {
|
||||
"window_seconds": 15.49,
|
||||
"successful_requests": 6805,
|
||||
"failed_attempts": 0,
|
||||
"successful_rps": 439.4,
|
||||
"last_error_offset_seconds": null,
|
||||
"instances": [
|
||||
"2eb97a202e04",
|
||||
"e1e2d306d0d7"
|
||||
]
|
||||
},
|
||||
"seen_stopped_replica_after_stop": false,
|
||||
"stop_command_seconds": 3.49,
|
||||
"restore_seconds": 6.93,
|
||||
"after_restore": {
|
||||
"window_seconds": 7.0,
|
||||
"successful_requests": 3618,
|
||||
"failed_attempts": 0,
|
||||
"successful_rps": 516.9,
|
||||
"last_error_offset_seconds": null,
|
||||
"instances": [
|
||||
"2eb97a202e04",
|
||||
"e1e2d306d0d7"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: 32 parallel requests, 1 message, 2 instances
|
||||
@@ -0,0 +1 @@
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"old_replicas": [
|
||||
"2eb97a202e04",
|
||||
"e1e2d306d0d7"
|
||||
],
|
||||
"new_replicas": [
|
||||
"525377e6c5c3",
|
||||
"ffb74e9857ae"
|
||||
],
|
||||
"update_window": {
|
||||
"window_seconds": 16.54,
|
||||
"successful_requests": 8547,
|
||||
"failed_attempts": 4,
|
||||
"successful_rps": 516.8,
|
||||
"last_error_offset_seconds": 13.5,
|
||||
"instances": [
|
||||
"2eb97a202e04",
|
||||
"525377e6c5c3",
|
||||
"e1e2d306d0d7",
|
||||
"ffb74e9857ae"
|
||||
]
|
||||
},
|
||||
"old_replicas_served_after_update": []
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: private bucket, limited credentials, s3 outage, orphan cleanup
|
||||
@@ -0,0 +1 @@
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"file_bytes": 9722608,
|
||||
"concurrent_uploads": 12,
|
||||
"rounds": 3,
|
||||
"uploads_ok": 36,
|
||||
"uploads_failed": 0,
|
||||
"elapsed_seconds": 12.49,
|
||||
"api_memory_limit_mib": 192,
|
||||
"idle_memory_mib": {
|
||||
"mtusi2026lab4-api-5": 62.2,
|
||||
"mtusi2026lab4-api-6": 64.0
|
||||
},
|
||||
"peak_memory_mib": {
|
||||
"mtusi2026lab4-api-5": 120.1,
|
||||
"mtusi2026lab4-api-6": 103.4
|
||||
},
|
||||
"buffering_all_files_would_need_mib": 111.3
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
module mtusi-messenger
|
||||
|
||||
go 1.26.0
|
||||
|
||||
require (
|
||||
github.com/jackc/pgx/v5 v5.11.0
|
||||
github.com/minio/minio-go/v7 v7.4.1
|
||||
github.com/redis/go-redis/v9 v9.23.0
|
||||
golang.org/x/crypto v0.58.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/klauspost/compress v1.19.2 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.4.0 // indirect
|
||||
github.com/klauspost/crc32 v1.3.0 // indirect
|
||||
github.com/minio/crc64nvme v1.1.1 // indirect
|
||||
github.com/minio/md5-simd v1.1.2 // indirect
|
||||
github.com/philhofer/fwd v1.2.0 // indirect
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
github.com/tinylib/msgp v1.6.4 // indirect
|
||||
github.com/zeebo/xxh3 v1.1.0 // indirect
|
||||
go.uber.org/atomic v1.12.0 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||
golang.org/x/net v0.60.0 // indirect
|
||||
golang.org/x/sync v0.24.0 // indirect
|
||||
golang.org/x/sys v0.49.0 // indirect
|
||||
golang.org/x/text v0.43.0 // indirect
|
||||
gopkg.in/ini.v1 v1.67.3 // indirect
|
||||
)
|
||||
@@ -0,0 +1,77 @@
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
|
||||
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8=
|
||||
github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
|
||||
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
|
||||
github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM=
|
||||
github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw=
|
||||
github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
|
||||
github.com/minio/crc64nvme v1.1.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
|
||||
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
|
||||
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
|
||||
github.com/minio/minio-go/v7 v7.4.1 h1:PtU/zIOfQRWZ+QydXAZysc+xZYHdG+BBR4Z6TUvy73w=
|
||||
github.com/minio/minio-go/v7 v7.4.1/go.mod h1:nQnQi7zTfXR4lREtYXZQWqo5aw0Rqv+ZqCGrTORygRk=
|
||||
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
|
||||
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/redis/go-redis/v9 v9.23.0 h1:/TTVdJa8BRNsybwwGGnmiYktHNlUXtbOQxlWpHBPL3s=
|
||||
github.com/redis/go-redis/v9 v9.23.0/go.mod h1:EWP2UUk+XDaI1s5nQB5mablZN6vccXlmBQTyfnF87D0=
|
||||
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
|
||||
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||
github.com/tinylib/msgp v1.6.4 h1:mOwYbyYDLPj35mkA2BjjYejgJk9BuHxDdvRnb6v2ZcQ=
|
||||
github.com/tinylib/msgp v1.6.4/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
|
||||
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
|
||||
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE=
|
||||
go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
|
||||
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
|
||||
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
|
||||
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
|
||||
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
||||
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
||||
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
||||
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/ini.v1 v1.67.3 h1:iM9Lhz5MRSGhHVGGwCuzG9KO8PoirCXj/m/qTmOJJQw=
|
||||
gopkg.in/ini.v1 v1.67.3/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
+1085
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,38 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 2000
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
accepted_payload_size 8192
|
||||
hold valid 1s
|
||||
hold nx 1s
|
||||
hold obsolete 1s
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
option httplog
|
||||
option redispatch 1
|
||||
retries 3
|
||||
retry-on conn-failure empty-response
|
||||
timeout connect 1s
|
||||
timeout client 20s
|
||||
timeout server 15s
|
||||
timeout http-request 10s
|
||||
timeout queue 5s
|
||||
|
||||
frontend public
|
||||
bind :8080
|
||||
default_backend api
|
||||
|
||||
backend api
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
http-check send meth GET uri /health/ready
|
||||
http-check expect status 200
|
||||
default-server check inter 1s fastinter 500ms downinter 1s fall 2 rise 2 resolvers docker init-addr none on-marked-down shutdown-sessions
|
||||
server-template api 1-8 api:8080
|
||||
@@ -0,0 +1,136 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func main() {
|
||||
log := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
|
||||
|
||||
host := envOr("APP_HOST", "0.0.0.0")
|
||||
port := envOr("APP_PORT", "8080")
|
||||
shutdownSec, _ := strconv.Atoi(envOr("SHUTDOWN_TIMEOUT_SECONDS", "5"))
|
||||
if shutdownSec < 1 {
|
||||
shutdownSec = 5
|
||||
}
|
||||
ttlSec, _ := strconv.Atoi(envOr("SESSION_TTL_SECONDS", "3600"))
|
||||
if ttlSec < 1 {
|
||||
ttlSec = 3600
|
||||
}
|
||||
|
||||
idemSec, _ := strconv.Atoi(envOr("IDEMPOTENCY_TTL_SECONDS", "86400"))
|
||||
if idemSec < 1 {
|
||||
idemSec = 86400
|
||||
}
|
||||
drainSec, _ := strconv.Atoi(envOr("DRAIN_SECONDS", "3"))
|
||||
|
||||
ctx := context.Background()
|
||||
if len(os.Args) > 1 && os.Args[1] == "s3-init" {
|
||||
if err := s3Init(ctx); err != nil {
|
||||
log.Error("s3 init failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
log.Info("bucket ready")
|
||||
return
|
||||
}
|
||||
db, err := openDB(ctx)
|
||||
if err != nil {
|
||||
log.Error("database open failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
if len(os.Args) > 1 && os.Args[1] == "migrate" {
|
||||
if err := migrate(ctx, db); err != nil {
|
||||
log.Error("migrate failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
log.Info("migrations applied")
|
||||
return
|
||||
}
|
||||
|
||||
store, err := openStore()
|
||||
if err != nil {
|
||||
log.Error("s3 open failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if len(os.Args) > 1 && (os.Args[1] == "cleanup" || os.Args[1] == "janitor") {
|
||||
runCleanup(ctx, db, store, log, os.Args[1] == "janitor")
|
||||
return
|
||||
}
|
||||
|
||||
cache := openCache(log)
|
||||
defer cache.Close()
|
||||
|
||||
maxBytes, _ := strconv.ParseInt(envOr("MAX_UPLOAD_BYTES", "10485760"), 10, 64)
|
||||
maxPixels, _ := strconv.ParseInt(envOr("MAX_IMAGE_PIXELS", "40000000"), 10, 64)
|
||||
slots, _ := strconv.Atoi(envOr("UPLOAD_CONCURRENCY", "10"))
|
||||
if slots < 1 {
|
||||
slots = 10
|
||||
}
|
||||
app := NewApp(db, cache, log, time.Duration(ttlSec)*time.Second, time.Duration(idemSec)*time.Second, store, limits{maxBytes: maxBytes, maxPixels: maxPixels, uploadSlots: slots})
|
||||
|
||||
addr := net.JoinHostPort(host, port)
|
||||
srv := &http.Server{
|
||||
Addr: addr,
|
||||
Handler: app.Routes(),
|
||||
ReadHeaderTimeout: 5 * time.Second,
|
||||
ReadTimeout: 10 * time.Second,
|
||||
WriteTimeout: 15 * time.Second,
|
||||
IdleTimeout: 30 * time.Second,
|
||||
}
|
||||
|
||||
go func() {
|
||||
log.Info("listening", "addr", addr, "env", envOr("APP_ENV", "lab"), "session_ttl_s", ttlSec)
|
||||
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||
log.Error("server error", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
}()
|
||||
|
||||
stop := make(chan os.Signal, 1)
|
||||
signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)
|
||||
sig := <-stop
|
||||
log.Info("shutdown started", "signal", sig.String())
|
||||
app.SetReady(false)
|
||||
time.Sleep(time.Duration(drainSec) * time.Second)
|
||||
|
||||
shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(shutdownSec)*time.Second)
|
||||
defer cancel()
|
||||
if err := srv.Shutdown(shCtx); err != nil {
|
||||
log.Error("shutdown error", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
log.Info("shutdown finished")
|
||||
}
|
||||
|
||||
func runCleanup(ctx context.Context, db *pgxpool.Pool, store *Store, log *slog.Logger, loop bool) {
|
||||
graceSec, _ := strconv.Atoi(envOr("CLEANUP_GRACE_SECONDS", "600"))
|
||||
intervalSec, _ := strconv.Atoi(envOr("CLEANUP_INTERVAL_SECONDS", "300"))
|
||||
for {
|
||||
removed, err := cleanupOrphans(ctx, db, store, time.Duration(graceSec)*time.Second, log)
|
||||
if err != nil {
|
||||
log.Error("cleanup failed", "err", err.Error())
|
||||
if !loop {
|
||||
os.Exit(1)
|
||||
}
|
||||
} else {
|
||||
log.Info("cleanup finished", "removed", removed)
|
||||
}
|
||||
if !loop {
|
||||
return
|
||||
}
|
||||
time.Sleep(time.Duration(intervalSec) * time.Second)
|
||||
}
|
||||
}
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"embed"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
//go:embed migrations/*.sql
|
||||
var migrationFS embed.FS
|
||||
|
||||
func migrate(ctx context.Context, pool *pgxpool.Pool) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
conn, err := pool.Acquire(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer conn.Release()
|
||||
if _, err := conn.Exec(ctx, `select pg_advisory_lock(727001)`); err != nil {
|
||||
return err
|
||||
}
|
||||
defer conn.Exec(context.Background(), `select pg_advisory_unlock(727001)`)
|
||||
|
||||
_, err = conn.Exec(ctx, `
|
||||
create table if not exists schema_migrations (
|
||||
version text primary key,
|
||||
applied_at timestamptz not null
|
||||
)`)
|
||||
if err != nil {
|
||||
return fmt.Errorf("schema_migrations: %w", err)
|
||||
}
|
||||
|
||||
entries, err := migrationFS.ReadDir("migrations")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() && strings.HasSuffix(e.Name(), ".sql") {
|
||||
names = append(names, e.Name())
|
||||
}
|
||||
}
|
||||
sort.Strings(names)
|
||||
|
||||
for _, name := range names {
|
||||
var exists bool
|
||||
err := conn.QueryRow(ctx, `select exists(select 1 from schema_migrations where version=$1)`, name).Scan(&exists)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if exists {
|
||||
continue
|
||||
}
|
||||
|
||||
body, err := migrationFS.ReadFile("migrations/" + name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tx, err := conn.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, string(body)); err != nil {
|
||||
_ = tx.Rollback(ctx)
|
||||
return fmt.Errorf("migration %s: %w", name, err)
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `insert into schema_migrations(version, applied_at) values($1, now())`, name); err != nil {
|
||||
_ = tx.Rollback(ctx)
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
create table users (
|
||||
id uuid primary key,
|
||||
username text not null unique,
|
||||
display_name text not null,
|
||||
password_hash text not null,
|
||||
created_at timestamptz not null
|
||||
);
|
||||
|
||||
create table chats (
|
||||
id uuid primary key,
|
||||
title text not null,
|
||||
created_at timestamptz not null
|
||||
);
|
||||
|
||||
create table chat_members (
|
||||
chat_id uuid not null references chats(id),
|
||||
user_id uuid not null references users(id),
|
||||
primary key (chat_id, user_id)
|
||||
);
|
||||
|
||||
create index chat_members_user_idx on chat_members (user_id);
|
||||
|
||||
create table messages (
|
||||
id uuid primary key,
|
||||
chat_id uuid not null references chats(id),
|
||||
sender_id uuid not null references users(id),
|
||||
text text not null,
|
||||
created_at timestamptz not null
|
||||
);
|
||||
|
||||
create index messages_history_idx on messages (chat_id, created_at, id);
|
||||
|
||||
create table sessions (
|
||||
token_hash bytea primary key,
|
||||
user_id uuid not null references users(id) on delete cascade,
|
||||
created_at timestamptz not null,
|
||||
expires_at timestamptz not null
|
||||
);
|
||||
@@ -0,0 +1,10 @@
|
||||
create table idempotency_keys (
|
||||
user_id uuid not null references users(id) on delete cascade,
|
||||
method text not null,
|
||||
path text not null,
|
||||
key text not null,
|
||||
body_hash text not null,
|
||||
response jsonb not null,
|
||||
created_at timestamptz not null,
|
||||
primary key (user_id, method, path, key)
|
||||
);
|
||||
@@ -0,0 +1,22 @@
|
||||
create table attachments (
|
||||
id uuid primary key,
|
||||
chat_id uuid not null references chats(id),
|
||||
owner_id uuid not null references users(id),
|
||||
filename text not null,
|
||||
content_type text not null,
|
||||
size_bytes bigint not null,
|
||||
sha256 text not null,
|
||||
object_key text not null unique,
|
||||
created_at timestamptz not null
|
||||
);
|
||||
|
||||
create index attachments_chat_idx on attachments (chat_id);
|
||||
|
||||
create table message_attachments (
|
||||
message_id uuid not null references messages(id),
|
||||
attachment_id uuid not null references attachments(id),
|
||||
position int not null,
|
||||
primary key (message_id, attachment_id)
|
||||
);
|
||||
|
||||
create index message_attachments_attachment_idx on message_attachments (attachment_id);
|
||||
@@ -0,0 +1,33 @@
|
||||
package main
|
||||
|
||||
import "time"
|
||||
|
||||
type User struct {
|
||||
ID string
|
||||
Username string
|
||||
DisplayName string
|
||||
PasswordHash string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
type Chat struct {
|
||||
ID string
|
||||
Title string
|
||||
MemberIDs []string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
type Message struct {
|
||||
ID string
|
||||
ChatID string
|
||||
SenderID string
|
||||
Text string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
type Session struct {
|
||||
TokenHash []byte
|
||||
UserID string
|
||||
CreatedAt time.Time
|
||||
ExpiresAt time.Time
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
type Store struct {
|
||||
client *minio.Client
|
||||
bucket string
|
||||
}
|
||||
|
||||
func openStore() (*Store, error) {
|
||||
endpoint, err := url.Parse(envOr("S3_ENDPOINT", "http://s3:9000"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
keyID, err := readSecret("S3_ACCESS_KEY_ID", "S3_ACCESS_KEY_ID_FILE")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
secret, err := readSecret("S3_SECRET_ACCESS_KEY", "S3_SECRET_ACCESS_KEY_FILE")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
transport := &http.Transport{
|
||||
DialContext: (&net.Dialer{Timeout: 2 * time.Second}).DialContext,
|
||||
ResponseHeaderTimeout: 10 * time.Second,
|
||||
MaxIdleConnsPerHost: 16,
|
||||
IdleConnTimeout: 30 * time.Second,
|
||||
}
|
||||
client, err := minio.New(endpoint.Host, &minio.Options{
|
||||
Creds: credentials.NewStaticV4(keyID, secret, ""),
|
||||
Secure: endpoint.Scheme == "https",
|
||||
Region: envOr("S3_REGION", "us-east-1"),
|
||||
Transport: transport,
|
||||
MaxRetries: 1,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Store{client: client, bucket: envOr("S3_BUCKET", "messenger-attachments")}, nil
|
||||
}
|
||||
|
||||
func (s *Store) Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) error {
|
||||
_, err := s.client.PutObject(ctx, s.bucket, key, r, size, minio.PutObjectOptions{
|
||||
ContentType: contentType,
|
||||
PartSize: 5 << 20,
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) Get(ctx context.Context, key string) (*minio.Object, minio.ObjectInfo, error) {
|
||||
obj, err := s.client.GetObject(ctx, s.bucket, key, minio.GetObjectOptions{})
|
||||
if err != nil {
|
||||
return nil, minio.ObjectInfo{}, err
|
||||
}
|
||||
info, err := obj.Stat()
|
||||
if err != nil {
|
||||
obj.Close()
|
||||
return nil, minio.ObjectInfo{}, err
|
||||
}
|
||||
return obj, info, nil
|
||||
}
|
||||
|
||||
func (s *Store) Remove(ctx context.Context, key string) error {
|
||||
return s.client.RemoveObject(ctx, s.bucket, key, minio.RemoveObjectOptions{})
|
||||
}
|
||||
|
||||
func (s *Store) List(ctx context.Context) <-chan minio.ObjectInfo {
|
||||
return s.client.ListObjects(ctx, s.bucket, minio.ListObjectsOptions{Prefix: objectPrefix, Recursive: true})
|
||||
}
|
||||
|
||||
func isNoSuchKey(err error) bool {
|
||||
return minio.ToErrorResponse(err).Code == "NoSuchKey"
|
||||
}
|
||||
|
||||
func s3Init(ctx context.Context) error {
|
||||
s, err := openStore()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
|
||||
defer cancel()
|
||||
for {
|
||||
exists, err := s.client.BucketExists(ctx, s.bucket)
|
||||
if err == nil {
|
||||
if exists {
|
||||
return nil
|
||||
}
|
||||
err = s.client.MakeBucket(ctx, s.bucket, minio.MakeBucketOptions{Region: envOr("S3_REGION", "us-east-1")})
|
||||
if err == nil || minio.ToErrorResponse(err).Code == "BucketAlreadyOwnedByYou" {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return err
|
||||
case <-time.After(2 * time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
|
||||
mkdir -p "$ROOT/secrets"
|
||||
cd "$ROOT/secrets"
|
||||
gen() {
|
||||
[ -f "$1" ] || openssl rand -hex "$2" | tr -d '\n' > "$1"
|
||||
}
|
||||
if [ ! -f postgres_password ]; then
|
||||
openssl rand -base64 24 | tr -d '\n' > postgres_password
|
||||
fi
|
||||
PASS=$(cat postgres_password)
|
||||
ENC=$(printf '%s' "$PASS" | python3 -c 'import sys,urllib.parse; print(urllib.parse.quote(sys.stdin.read(), safe=""))')
|
||||
printf 'postgres://messenger:%s@postgres:5432/messenger?sslmode=disable\n' "$ENC" > database_url
|
||||
printf 'redis://valkey:6379/0\n' > cache_url
|
||||
gen s3_admin_access_key_id 8
|
||||
gen s3_admin_secret_access_key 20
|
||||
gen s3_access_key_id 8
|
||||
gen s3_secret_access_key 20
|
||||
BUCKET=$(grep -E '^S3_BUCKET=' "$ROOT/.env" 2>/dev/null | cut -d= -f2 || true)
|
||||
BUCKET=${BUCKET:-messenger-attachments}
|
||||
printf '{"identities":[{"name":"admin","credentials":[{"accessKey":"%s","secretKey":"%s"}],"actions":["Admin","Read","Write","List","Tagging"]},{"name":"api","credentials":[{"accessKey":"%s","secretKey":"%s"}],"actions":["Read:%s","Write:%s","List:%s"]}]}\n' \
|
||||
"$(cat s3_admin_access_key_id)" "$(cat s3_admin_secret_access_key)" "$(cat s3_access_key_id)" "$(cat s3_secret_access_key)" "$BUCKET" "$BUCKET" "$BUCKET" > s3_config.json
|
||||
chmod 644 ./*
|
||||
echo "secrets ready in $ROOT/secrets"
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
docker compose --env-file .env restart postgres valkey s3
|
||||
docker compose --env-file .env up -d --wait postgres valkey s3
|
||||
docker compose --env-file .env restart api lb janitor
|
||||
Executable
+16
@@ -0,0 +1,16 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
dc="docker compose --env-file .env"
|
||||
old=$($dc ps -q api)
|
||||
n=$(echo "$old" | wc -w)
|
||||
$dc build migrate
|
||||
$dc up -d --no-deps migrate janitor
|
||||
$dc up -d --no-deps --no-recreate --scale api=$((n * 2)) api
|
||||
$dc up -d --wait --no-deps --no-recreate --scale api=$((n * 2)) api
|
||||
for c in $old; do
|
||||
docker stop -t 10 "$c" >/dev/null
|
||||
docker rm "$c" >/dev/null
|
||||
done
|
||||
$dc up -d --wait --no-deps --no-recreate --scale api=$n api
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
: "${TARGET_INSTANCE_ID:?}"
|
||||
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
id=$(docker compose --env-file .env ps -q api | while read -r c; do
|
||||
case "$c" in "$TARGET_INSTANCE_ID"*) echo "$c";; esac
|
||||
done)
|
||||
[ -n "$id" ] || exit 1
|
||||
docker stop -t 10 "$id" >/dev/null
|
||||
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from client import Client, arguments, check, run
|
||||
from image_gen import png, png_header_only
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
SAMPLE = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
|
||||
|
||||
def upload(c, actor, chat, data, expected, content_type='image/png', filename='file.png'):
|
||||
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=' + filename, expected,
|
||||
actor=actor, raw=data, headers={'Content-Type': content_type},
|
||||
schema='Attachment' if expected == 201 else None)
|
||||
|
||||
|
||||
def post_message(c, actor, chat, ids, expected, text=''):
|
||||
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', expected, actor=actor,
|
||||
body={'text': text, 'attachment_ids': ids}, headers={'Idempotency-Key': uuid.uuid4().hex}, schema='Message' if expected == 201 else None)
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Attachment validation and access checks')
|
||||
c = Client(args.base_url, args.ca_file)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
other = c.call('POST', '/api/v1/chats', 201, actor=alice, body={'title': 'second', 'member_ids': [alice['id'], bob['id']]}, schema='Chat')['json']
|
||||
eve_chat = c.call('POST', '/api/v1/chats', 201, actor=eve, body={'title': 'eve', 'member_ids': [eve['id'], bob['id']]}, schema='Chat')['json']
|
||||
|
||||
upload(c, alice, chat, b'x' * (MAX_BYTES + 1), 413)
|
||||
upload(c, alice, chat, SAMPLE + b'\0' * MAX_BYTES, 413)
|
||||
upload(c, alice, chat, png_header_only(8000, 8000), 413)
|
||||
upload(c, alice, chat, SAMPLE[:len(SAMPLE) // 2], 415)
|
||||
upload(c, alice, chat, SAMPLE[:-1], 415)
|
||||
upload(c, alice, chat, b'\x89PNG\r\n\x1a\n' + b'garbage' * 100, 415)
|
||||
upload(c, alice, chat, b'', 415)
|
||||
upload(c, alice, chat, b'GIF89a' + b'\0' * 64, 415)
|
||||
upload(c, alice, chat, SAMPLE, 415, content_type='image/jpeg')
|
||||
upload(c, alice, chat, SAMPLE, 415, content_type='text/plain')
|
||||
upload(c, alice, chat, SAMPLE, 415, content_type='application/octet-stream')
|
||||
|
||||
for name in ('..%2Fx.png', 'a%5Cb.png', 'a%00b.png', 'a%0Ab.png', 'a' * 256, ''):
|
||||
upload(c, alice, chat, SAMPLE, 400, filename=name)
|
||||
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments', 400, actor=alice, raw=SAMPLE, headers={'Content-Type': 'image/png'})
|
||||
upload(c, eve, chat, SAMPLE, 404)
|
||||
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=a.png', 401, raw=SAMPLE, headers={'Content-Type': 'image/png'})
|
||||
|
||||
ok = upload(c, alice, chat, SAMPLE, 201)['json']
|
||||
check(ok['sha256'] == hashlib.sha256(SAMPLE).hexdigest(), 'checksum mismatch')
|
||||
check(ok['filename'] == 'file.png' and ok['status'] == 'ready' and ok['variants'] == [], 'attachment fields')
|
||||
unicode_name = upload(c, alice, chat, SAMPLE, 201, filename='%D1%84%D0%BE%D1%82%D0%BE.png')['json']
|
||||
check(unicode_name['filename'] == 'фото.png', 'unicode filename changed')
|
||||
|
||||
post_message(c, alice, other, [ok['id']], 404)
|
||||
post_message(c, alice, chat, [str(uuid.uuid4())], 404)
|
||||
post_message(c, alice, chat, [ok['id'], ok['id']], 400)
|
||||
post_message(c, alice, chat, ['not-a-uuid'], 400)
|
||||
post_message(c, alice, chat, [str(uuid.uuid4()) for _ in range(11)], 400)
|
||||
post_message(c, alice, chat, [], 400)
|
||||
post_message(c, eve, chat, [ok['id']], 404)
|
||||
post_message(c, eve, eve_chat, [ok['id']], 404)
|
||||
|
||||
shared = post_message(c, bob, chat, [ok['id'], unicode_name['id']], 201, text='from bob')['json']
|
||||
check(shared['attachment_ids'] == [ok['id'], unicode_name['id']], 'attachment order changed')
|
||||
history = c.call('GET', '/api/v1/chats/' + chat['id'] + '/messages?limit=100', actor=alice, schema='MessagePage')['json']['items']
|
||||
check(any(m['id'] == shared['id'] and m['attachment_ids'] == shared['attachment_ids'] for m in history), 'history lost attachment_ids')
|
||||
|
||||
for suffix in ('', '/content'):
|
||||
c.call('GET', '/api/v1/attachments/' + ok['id'] + suffix, 404, actor=eve)
|
||||
c.call('GET', '/api/v1/attachments/' + str(uuid.uuid4()) + suffix, 404, actor=alice)
|
||||
c.call('GET', '/api/v1/attachments/bad' + suffix, 400, actor=alice)
|
||||
c.call('GET', '/api/v1/attachments/' + ok['id'] + suffix, 401)
|
||||
c.call('GET', '/api/v1/attachments/' + ok['id'] + '/content?variant=thumbnail', 400, actor=bob)
|
||||
download = c.call('GET', '/api/v1/attachments/' + ok['id'] + '/content', actor=bob)
|
||||
check(download['bytes'] == SAMPLE, 'download differs')
|
||||
|
||||
big = png(1800, 1800, noise=True)
|
||||
check(len(big) < MAX_BYTES, 'generated image exceeds limit')
|
||||
large = upload(c, alice, chat, big, 201)['json']
|
||||
check(c.call('GET', '/api/v1/attachments/' + large['id'] + '/content', actor=bob)['bytes'] == big, 'large download differs')
|
||||
print('PASS: attachment validation, limits and access control')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env python3
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import threading
|
||||
import uuid
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Concurrent Idempotency-Key check through the load balancer')
|
||||
c = Client(args.base_url, args.ca_file)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
path = '/api/v1/chats/' + chat['id'] + '/messages'
|
||||
parallel = 32
|
||||
key = 'idem-' + uuid.uuid4().hex
|
||||
barrier = threading.Barrier(parallel)
|
||||
|
||||
def send(_):
|
||||
local = Client(args.base_url, args.ca_file)
|
||||
barrier.wait()
|
||||
return local.call('POST', path, 201, actor=alice, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')
|
||||
|
||||
with ThreadPoolExecutor(max_workers=parallel) as pool:
|
||||
results = list(pool.map(send, range(parallel)))
|
||||
ids = {r['json']['id'] for r in results}
|
||||
instances = {r['headers']['X-Instance-Id'] for r in results}
|
||||
check(len(ids) == 1, 'parallel requests produced %d messages' % len(ids))
|
||||
check(len(instances) >= 2, 'requests were served by one instance only')
|
||||
check(len({r['json']['created_at'] for r in results}) == 1, 'created_at differs between replays')
|
||||
|
||||
listing = c.call('GET', path, actor=alice, schema='MessagePage')['json']['items']
|
||||
check(sum(1 for m in listing if m['text'] == 'same') == 1, 'duplicate message in history')
|
||||
|
||||
c.call('POST', path, 409, actor=alice, body={'text': 'other'}, headers={'Idempotency-Key': key})
|
||||
c.call('POST', path, 400, actor=alice, body={'text': 'x'})
|
||||
c.call('POST', path, 400, actor=alice, body={'text': 'x'}, headers={'Idempotency-Key': 'bad key!'})
|
||||
other = c.call('POST', path, 201, actor=bob, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')['json']
|
||||
check(other['id'] not in ids, 'key must be scoped per user')
|
||||
print('PASS: %d parallel requests, 1 message, %d instances' % (parallel, len(instances)))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,23 @@
|
||||
import os
|
||||
import struct
|
||||
import zlib
|
||||
|
||||
|
||||
def chunk(kind, data):
|
||||
body = kind + data
|
||||
return struct.pack('>I', len(data)) + body + struct.pack('>I', zlib.crc32(body) & 0xffffffff)
|
||||
|
||||
|
||||
def png(width, height, noise=False):
|
||||
header = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
|
||||
raw = bytearray()
|
||||
row = width * 3 + 1
|
||||
for _ in range(height):
|
||||
raw += b'\x00' + (os.urandom(row - 1) if noise else bytes(row - 1))
|
||||
data = zlib.compress(bytes(raw), 0 if noise else 6)
|
||||
return header + chunk(b'IDAT', data) + chunk(b'IEND', b'')
|
||||
|
||||
|
||||
def png_header_only(width, height):
|
||||
header = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
|
||||
return header + chunk(b'IDAT', zlib.compress(b'\x00')) + chunk(b'IEND', b'')
|
||||
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
SAMPLE = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
BUCKET = 'messenger-attachments'
|
||||
|
||||
|
||||
def compose(*args, check_rc=True, stdin=None):
|
||||
return subprocess.run(['docker', 'compose', '--env-file', '.env', *args], cwd=ROOT, check=check_rc,
|
||||
capture_output=True, text=True, input=stdin)
|
||||
|
||||
|
||||
def secret(name):
|
||||
return (ROOT / 'secrets' / name).read_text().strip()
|
||||
|
||||
|
||||
def s3_status(method, path, user=None, data=None):
|
||||
cmd = ['exec', '-T', 's3', 'curl', '-s', '-o', '/dev/null', '-w', '%{http_code}', '-X', method]
|
||||
if user:
|
||||
cmd += ['--aws-sigv4', 'aws:amz:us-east-1:s3', '--user', user[0] + ':' + user[1]]
|
||||
if data is not None:
|
||||
cmd += ['--data-binary', data]
|
||||
cmd.append('http://127.0.0.1:9000' + path)
|
||||
return compose(*cmd).stdout.strip()
|
||||
|
||||
|
||||
def upload(c, actor, chat, expected):
|
||||
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=a.png', expected, actor=actor,
|
||||
raw=SAMPLE, headers={'Content-Type': 'image/png'}, schema='Attachment' if expected == 201 else None)
|
||||
|
||||
|
||||
def wait_api(c):
|
||||
deadline = time.monotonic() + 60
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
c.call('GET', '/health/ready')
|
||||
return
|
||||
except Failure:
|
||||
time.sleep(1)
|
||||
raise Failure('api did not recover')
|
||||
|
||||
|
||||
def wait_s3(timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if compose('ps', 's3', '--format', '{{.Status}}').stdout.count('(healthy)'):
|
||||
return
|
||||
time.sleep(1)
|
||||
raise Failure('s3 did not recover')
|
||||
|
||||
|
||||
def object_key(attachment_id):
|
||||
out = compose('exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-Atc',
|
||||
"select object_key from attachments where id='%s'" % attachment_id).stdout.strip()
|
||||
check(out.startswith('attachments/'), 'object key not found')
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('S3 privacy, outage and orphan cleanup checks')
|
||||
c = Client(args.base_url, args.ca_file, timeout=15)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
att = upload(c, alice, chat, 201)['json']
|
||||
key = object_key(att['id'])
|
||||
check(key != 'attachments/' + att['filename'] and att['filename'] not in key, 'object key derived from filename')
|
||||
|
||||
api = (secret('s3_access_key_id'), secret('s3_secret_access_key'))
|
||||
admin = (secret('s3_admin_access_key_id'), secret('s3_admin_secret_access_key'))
|
||||
path = '/' + BUCKET + '/' + key
|
||||
check(s3_status('GET', path) == '403', 'anonymous GET must be denied')
|
||||
check(s3_status('GET', '/' + BUCKET + '/') == '403', 'anonymous listing must be denied')
|
||||
check(s3_status('PUT', path + '-anon', data='x') == '403', 'anonymous PUT must be denied')
|
||||
check(s3_status('GET', path, ('wrong', 'wrong')) == '403', 'wrong credentials must be denied')
|
||||
check(s3_status('GET', path, api) == '200', 'api credentials cannot read')
|
||||
check(s3_status('PUT', '/other-bucket-' + uuid.uuid4().hex[:8], api) == '403', 'api credentials can create buckets')
|
||||
ports = compose('ps', '--format', '{{.Service}} {{.Ports}}').stdout
|
||||
published = [line.split()[0] for line in ports.splitlines() if '->' in line]
|
||||
check(published == ['lb'], 'only lb may publish ports, found %s' % published)
|
||||
|
||||
compose('stop', 's3')
|
||||
try:
|
||||
upload(c, alice, chat, 503)
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', 503, actor=bob)
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'], 200, actor=bob, schema='Attachment')
|
||||
c.message(alice, chat, 'text works while s3 is down')
|
||||
finally:
|
||||
compose('start', 's3')
|
||||
wait_s3()
|
||||
wait_api(c)
|
||||
deadline = time.monotonic() + 60
|
||||
while True:
|
||||
try:
|
||||
body = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes']
|
||||
check(body == SAMPLE, 'content differs after s3 restart')
|
||||
break
|
||||
except Failure:
|
||||
check(time.monotonic() < deadline, 'download did not recover')
|
||||
time.sleep(1)
|
||||
upload(c, alice, chat, 201)
|
||||
|
||||
orphan = '/' + BUCKET + '/attachments/' + uuid.uuid4().hex
|
||||
check(s3_status('PUT', orphan, admin, 'orphan') == '200', 'cannot create orphan')
|
||||
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=3600', 'janitor', 'cleanup')
|
||||
check(s3_status('GET', orphan, admin) == '200', 'fresh object was removed inside grace period')
|
||||
time.sleep(1)
|
||||
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=0', 'janitor', 'cleanup')
|
||||
check(s3_status('GET', orphan, admin) == '404', 'orphan was not removed')
|
||||
check(c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes'] == SAMPLE, 'referenced object was removed')
|
||||
print('PASS: private bucket, limited credentials, s3 outage, orphan cleanup')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import json
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def sh(*cmd):
|
||||
return subprocess.run(cmd, cwd=ROOT, check=True, capture_output=True, text=True).stdout.strip()
|
||||
|
||||
|
||||
def api_containers():
|
||||
out = sh('docker', 'compose', '--env-file', '.env', 'ps', '-q', 'api')
|
||||
return [x[:12] for x in out.split()]
|
||||
|
||||
|
||||
def wait_healthy(count, timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
out = sh('docker', 'compose', '--env-file', '.env', 'ps', 'api', '--format', '{{.Status}}')
|
||||
if out.count('(healthy)') >= count:
|
||||
return
|
||||
time.sleep(1)
|
||||
raise Failure('replicas did not become healthy')
|
||||
|
||||
|
||||
class Load:
|
||||
def __init__(self, base, ca, concurrency):
|
||||
c = Client(base, ca)
|
||||
self.alice, _, _, self.chat = c.bootstrap()
|
||||
self.base, self.ca, self.n = base, ca, concurrency
|
||||
self.events = []
|
||||
self.lock = threading.Lock()
|
||||
self.stop = threading.Event()
|
||||
self.t0 = time.monotonic()
|
||||
|
||||
def worker(self):
|
||||
c = Client(self.base, self.ca, timeout=3)
|
||||
while not self.stop.is_set():
|
||||
key = uuid.uuid4().hex
|
||||
first_error = None
|
||||
while not self.stop.is_set():
|
||||
try:
|
||||
r = c.message(self.alice, self.chat, 'k-' + key, key=key)
|
||||
t = time.monotonic() - self.t0
|
||||
with self.lock:
|
||||
self.events.append((t, True, r['headers']['X-Instance-Id'], first_error))
|
||||
break
|
||||
except Failure:
|
||||
t = time.monotonic() - self.t0
|
||||
first_error = first_error or t
|
||||
with self.lock:
|
||||
self.events.append((t, False, None, None))
|
||||
time.sleep(0.05)
|
||||
|
||||
def start(self):
|
||||
self.pool = ThreadPoolExecutor(max_workers=self.n)
|
||||
self.futures = [self.pool.submit(self.worker) for _ in range(self.n)]
|
||||
|
||||
def finish(self):
|
||||
self.stop.set()
|
||||
self.pool.shutdown(wait=True)
|
||||
|
||||
def summary(self, begin, end):
|
||||
ev = [e for e in self.events if begin <= e[0] < end]
|
||||
ok = [e for e in ev if e[1]]
|
||||
bad = [e for e in ev if not e[1]]
|
||||
span = max(end - begin, 1e-9)
|
||||
return {
|
||||
'window_seconds': round(span, 2),
|
||||
'successful_requests': len(ok),
|
||||
'failed_attempts': len(bad),
|
||||
'successful_rps': round(len(ok) / span, 1),
|
||||
'last_error_offset_seconds': round(max(e[0] for e in bad) - begin, 2) if bad else None,
|
||||
'instances': sorted({e[2] for e in ok}),
|
||||
}
|
||||
|
||||
def instances_between(self, begin, end=float('inf')):
|
||||
return sorted({e[2] for e in self.events if e[1] and begin <= e[0] < end})
|
||||
|
||||
|
||||
def failover_each(args):
|
||||
ids = api_containers()
|
||||
check(len(ids) >= 2, 'need at least two replicas')
|
||||
load = Load(args.base_url, args.ca_file, args.concurrency)
|
||||
load.start()
|
||||
time.sleep(args.warmup)
|
||||
results = []
|
||||
base = load.summary(2, args.warmup)
|
||||
for target in ids:
|
||||
before = time.monotonic() - load.t0
|
||||
sh('docker', 'stop', '-t', '10', target)
|
||||
stopped = time.monotonic() - load.t0
|
||||
time.sleep(args.hold)
|
||||
after_stop = time.monotonic() - load.t0
|
||||
window = load.summary(before, after_stop)
|
||||
sh('docker', 'start', target)
|
||||
wait_healthy(len(ids))
|
||||
restored = time.monotonic() - load.t0
|
||||
time.sleep(args.warmup)
|
||||
results.append({
|
||||
'stopped_replica': target,
|
||||
'during_failure': window,
|
||||
'seen_stopped_replica_after_stop': target in load.instances_between(stopped + 0.5, after_stop),
|
||||
'stop_command_seconds': round(stopped - before, 2),
|
||||
'restore_seconds': round(restored - after_stop, 2),
|
||||
'after_restore': load.summary(restored + 1, restored + args.warmup),
|
||||
})
|
||||
load.finish()
|
||||
return {'baseline': base, 'replicas': results}
|
||||
|
||||
|
||||
def rolling(args):
|
||||
old = api_containers()
|
||||
check(len(old) >= 2, 'need at least two replicas')
|
||||
n = len(old)
|
||||
load = Load(args.base_url, args.ca_file, args.concurrency)
|
||||
load.start()
|
||||
time.sleep(args.warmup)
|
||||
begin = time.monotonic() - load.t0
|
||||
sh('scripts/rolling-update.sh')
|
||||
end = time.monotonic() - load.t0
|
||||
time.sleep(args.warmup)
|
||||
load.finish()
|
||||
return {
|
||||
'old_replicas': old,
|
||||
'new_replicas': api_containers(),
|
||||
'update_window': load.summary(begin, end),
|
||||
'old_replicas_served_after_update': [i for i in load.instances_between(end) if i in old],
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
def configure(p):
|
||||
p.add_argument('--mode', choices=['failover-each', 'rolling'], required=True)
|
||||
p.add_argument('--concurrency', type=int, default=8)
|
||||
p.add_argument('--warmup', type=float, default=8)
|
||||
p.add_argument('--hold', type=float, default=12)
|
||||
p.add_argument('--output')
|
||||
args = arguments('Failure and rolling update under load', configure)
|
||||
report = failover_each(args) if args.mode == 'failover-each' else rolling(args)
|
||||
text = json.dumps(report, indent=2)
|
||||
if args.output:
|
||||
Path(args.output).write_text(text + '\n')
|
||||
print(text)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env python3
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from client import Client, Failure, arguments, check, run
|
||||
from image_gen import png
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
UNITS = {'B': 1, 'KiB': 1024, 'MiB': 1024 ** 2, 'GiB': 1024 ** 3}
|
||||
|
||||
|
||||
def to_mib(text):
|
||||
value, unit = re.match(r'([\d.]+)\s*([A-Za-z]+)', text).groups()
|
||||
return float(value) * UNITS[unit] / 1024 ** 2
|
||||
|
||||
|
||||
def sample(stop, peaks):
|
||||
proc = subprocess.Popen(['docker', 'stats', '--format', '{{.Name}} {{.MemUsage}}'], cwd=ROOT,
|
||||
stdout=subprocess.PIPE, text=True)
|
||||
try:
|
||||
for line in proc.stdout:
|
||||
line = re.sub(r'\x1b\[[0-9;]*[A-Za-z]', '', line).strip()
|
||||
m = re.match(r'(\S+-api-\d+)\s+(\S+ ?\S+) /', line)
|
||||
if m:
|
||||
peaks[m.group(1)] = max(peaks.get(m.group(1), 0), to_mib(m.group(2)))
|
||||
if stop.is_set():
|
||||
break
|
||||
finally:
|
||||
proc.terminate()
|
||||
|
||||
|
||||
def main():
|
||||
def configure(p):
|
||||
p.add_argument('--concurrency', type=int, default=12)
|
||||
p.add_argument('--rounds', type=int, default=3)
|
||||
p.add_argument('--output')
|
||||
args = arguments('Concurrent upload memory experiment', configure)
|
||||
check(args.concurrency >= 10, 'use at least 10 concurrent uploads')
|
||||
c = Client(args.base_url, args.ca_file, timeout=60)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
data = png(1800, 1800, noise=True)
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
path = '/api/v1/chats/' + chat['id'] + '/attachments?filename=load.png'
|
||||
|
||||
idle, stop = {}, threading.Event()
|
||||
t = threading.Thread(target=sample, args=(stop, idle))
|
||||
t.start()
|
||||
time.sleep(4)
|
||||
stop.set()
|
||||
t.join()
|
||||
|
||||
peaks, stop = {}, threading.Event()
|
||||
t = threading.Thread(target=sample, args=(stop, peaks))
|
||||
t.start()
|
||||
time.sleep(2)
|
||||
barrier = threading.Barrier(args.concurrency)
|
||||
results = []
|
||||
|
||||
def worker(_):
|
||||
local = Client(args.base_url, args.ca_file, timeout=60)
|
||||
barrier.wait()
|
||||
out = []
|
||||
for _ in range(args.rounds):
|
||||
try:
|
||||
r = local.call('POST', path, 201, actor=alice, raw=data, headers={'Content-Type': 'image/png'}, schema='Attachment')
|
||||
out.append(r['json']['sha256'] == digest)
|
||||
except Failure:
|
||||
out.append(False)
|
||||
return out
|
||||
|
||||
started = time.monotonic()
|
||||
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
|
||||
for out in pool.map(worker, range(args.concurrency)):
|
||||
results.extend(out)
|
||||
elapsed = time.monotonic() - started
|
||||
time.sleep(2)
|
||||
stop.set()
|
||||
t.join()
|
||||
limit = 192
|
||||
report = {
|
||||
'file_bytes': len(data),
|
||||
'concurrent_uploads': args.concurrency,
|
||||
'rounds': args.rounds,
|
||||
'uploads_ok': sum(results),
|
||||
'uploads_failed': len(results) - sum(results),
|
||||
'elapsed_seconds': round(elapsed, 2),
|
||||
'api_memory_limit_mib': limit,
|
||||
'idle_memory_mib': {k: round(v, 1) for k, v in sorted(idle.items())},
|
||||
'peak_memory_mib': {k: round(v, 1) for k, v in sorted(peaks.items())},
|
||||
'buffering_all_files_would_need_mib': round(args.concurrency * len(data) / 1024 ** 2, 1),
|
||||
}
|
||||
text = json.dumps(report, indent=2)
|
||||
if args.output:
|
||||
Path(args.output).write_text(text + '\n')
|
||||
print(text)
|
||||
check(report['uploads_failed'] == 0, 'some uploads failed')
|
||||
check(max(peaks.values()) < limit, 'peak memory reached the limit')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
Reference in New Issue
Block a user