ЛР 4 — БВТ 2304 — Березин — на 4 #30

Closed
yagera wants to merge 4 commits from yagera/mtusi2026lab4-fork:solution/bvt2304-berezin into main
39 changed files with 3456 additions and 28 deletions
+3 -13
View File
@@ -1,20 +1,10 @@
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
# These are logical interface suggestions; map actual framework names in REPORT.md.
APP_HOST=0.0.0.0
APP_PORT=8080
APP_ENV=lab
SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
S3_ENDPOINT=http://s3:9000
API_REPLICAS=2
S3_REGION=us-east-1
S3_BUCKET=messenger-attachments
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
CLEANUP_GRACE_SECONDS=600
CLEANUP_INTERVAL_SECONDS=300
+19
View File
@@ -0,0 +1,19 @@
FROM golang:1.26-alpine@sha256:c95332c2af86b6d89b91bd0500f4b9529ccbd090a0d1855c6d1ceaa142ae8615 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY *.go ./
COPY migrations ./migrations
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/messenger .
FROM alpine:3.21@sha256:ce64758a109eb420d874a118f87920e625e12d3634e03b4a5573fd9f6e5d3507
RUN adduser -D -u 10001 app
WORKDIR /app
COPY --from=build /out/messenger /app/messenger
USER 10001
EXPOSE 8080
ENV APP_HOST=0.0.0.0 APP_PORT=8080 APP_ENV=lab
STOPSIGNAL SIGTERM
HEALTHCHECK --interval=5s --timeout=3s --start-period=10s --retries=3 \
CMD wget -qO- http://127.0.0.1:8080/health/ready >/dev/null || exit 1
ENTRYPOINT ["/app/messenger"]
+90 -15
View File
@@ -1,37 +1,112 @@
# Отчёт — лабораторная 4
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Березин Герман / БВТ2304
- Целевая оценка: 4
- Commit/PR: после публикации
- Использованные источники, библиотеки и помощь ИИ: Go 1.26, pgx, go-redis, minio-go v7 (S3-клиент), HAProxy 3.0, PostgreSQL 17, Valkey 8.1, SeaweedFS (S3); код lab3 перенесён за основу; ИИ-ассистент использован при разработке
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Требуются Docker 29 с Compose v2 и Python 3.9+.
```sh
cp .env.example .env
sh scripts/mksecrets.sh
docker compose --env-file .env up -d --build --wait
make check && make test
```
Порядок старта: postgres, valkey, s3 → `migrate` (однократно) → `s3-init` (идемпотентное создание приватного bucket) → `api` × 2 → `lb`; рядом работает `janitor`. Публичный адрес: `http://localhost:8080`.
Остановка без потери данных: `docker compose --env-file .env stop`. Намеренное удаление учебных данных: `docker compose --env-file .env down -v`.
Проверки:
```sh
make test-persistence ACTION_SCRIPT=scripts/restart.sh
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/attachments.py
python3 tests/s3_checks.py
python3 tests/upload_memory.py --output evidence/upload-memory.json
python3 tests/idempotency.py
python3 tests/scenario.py --mode failover-each --output evidence/failover-each.json
python3 tests/scenario.py --mode rolling --output evidence/rolling.json
```
После `make test-failover` остановленную реплику восстанавливают: `docker compose --env-file .env up -d --wait`.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Клиент → HAProxy → `api` × 2 → PostgreSQL (метаданные) и SeaweedFS S3 (байты). Valkey остаётся общим кэшем сессий, источник истины сессий — PostgreSQL.
- **Хранилище.** S3-сервер — SeaweedFS (образ закреплён по digest). Образы MinIO недоступны для скачивания, поэтому выбран SeaweedFS. Bucket `messenger-attachments` приватный, данные лежат в volume `s3_data`. Порты S3 наружу не публикуются, в хост-сеть выходит только `lb`.
- **Модель.** Таблица `attachments` (id, chat_id, owner_id, filename, content_type, size_bytes, sha256, object_key, created_at) и `message_attachments` (message_id, attachment_id, position). Object key — `attachments/<uuid>`, задаётся сервером; `filename` только отображается и проверяется (1–255 символов, без `/`, `\`, NUL и управляющих символов).
- **Загрузка.** Сырое тело PNG/JPEG идёт потоком в S3 без накопления в памяти API и без локальных файлов. Проверки: Content-Type → 415; `Content-Length` и `MaxBytesReader` до 10 MiB → 413; magic bytes → 415 (подмена MIME); после записи — заголовок изображения (`image.DecodeConfig`, без декодирования пикселей), совпадение формата с Content-Type, хвост файла (IEND/EOI) → 415 для повреждённых и обрезанных файлов; ≤ 40 млн пикселей → 413. SHA-256 и размер считаются во время потока. При любой ошибке объект удаляется. 201 `ready` возвращается только после записи объекта и строки в PostgreSQL.
- **Скачивание.** Доступ только участникам чата (JOIN с `chat_members`), иначе 404. Первый блок читается до отправки заголовков, поэтому сбой S3 даёт 503, а не оборванный ответ 200. Redirect нет.
- **Сообщения.** `attachment_ids` (до 10, уникальные); вложение должно принадлежать этому чату, иначе 404; пустой `text` допустим при наличии вложений. Хеш тела идемпотентности включает `attachment_ids`, для сообщений без вложений он совпадает с lab3.
- **Права S3.** API и janitor используют отдельные ключи (`Read/Write/List` только на свой bucket). Административные ключи есть только у `s3-init`, который создаёт bucket (повторный запуск ничего не меняет). Секреты генерирует `scripts/mksecrets.sh`, в Git их нет.
- **Сети.** `s3` находится в сети `storage` вместе с `api`, `janitor`, `s3-init`; `postgres`, `valkey`, `lb` — в `backend`. Админ-интерфейсы SeaweedFS (master, filer) не публикуются.
- Остальное перенесено из lab3: HAProxy с проверками готовности, идемпотентность, отдельные миграции, graceful shutdown.
### Согласованность БД и S3
Порядок записи: объект в S3 → строка в PostgreSQL.
| Сбой | Результат | Компенсация |
| --- | --- | --- |
| Клиент оборвал загрузку | запись в S3 не завершается, ответа нет | объект удаляется в обработчике |
| Невалидное изображение, превышение лимитов | объект записан, но не принят | объект удаляется до ответа |
| Ошибка PostgreSQL после S3 PUT | 503 | объект удаляется в обработчике |
| Падение процесса между PUT и INSERT | остаётся объект без строки | `janitor` удаляет такие объекты |
`janitor` (`messenger janitor`) раз в `CLEANUP_INTERVAL_SECONDS` (300 с) перебирает объекты под префиксом `attachments/`, проверяет их по `attachments.object_key` и удаляет объекты без записи, старше `CLEANUP_GRACE_SECONDS` (600 с). Grace period защищает объекты текущих загрузок. Разовый запуск: `docker compose run --rm janitor cleanup`. Строки без объекта автоматически не чистятся: файл читается как 404/503, строка сохраняется.
### Лимиты памяти и потоковая обработка
| Параметр | Значение |
| --- | --- |
| Размер файла | не более 10 MiB (`MAX_UPLOAD_BYTES`), проверяется до чтения тела |
| Пиксели | не более 40 млн (`MAX_IMAGE_PIXELS`), декодируется только заголовок |
| Параллельные загрузки на реплику | не более 10 (`UPLOAD_CONCURRENCY`), иначе 429 с `Retry-After` |
| Буфер на загрузку | не более одной части multipart S3 (5 MiB) и 256 KiB для проверки заголовка |
| Таймауты API | ReadHeader 5 s, Read 10 s, Write 15 s, запись в S3 до 60 s, чтение из S3 до 30 s |
| Лимиты контейнеров | api 192 MiB и `GOMEMLIMIT=150MiB`; s3 512 MiB; postgres 512 MiB; остальные 64–128 MiB |
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Публичный smoke | `make test` | PASS | evidence/smoke.txt |
| Сохранность после restart (метаданные и оригинал) | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS; перезапускаются postgres, valkey, s3, api, lb, janitor | evidence/persistence.txt |
| Загрузка → остановка реплики → скачивание через другую | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` (обе реплики) | PASS, 0 ошибок записи, восстановление около 3.5 s, оригинал читается | evidence/failover-1.txt, evidence/failover-2.txt |
| Большой файл, повреждённое изображение, подмена MIME, чужое и чужого чата attachment_id | `python3 tests/attachments.py` | 413 / 415 / 415 / 404 / 404, ни одного 500 | evidence/attachments.txt |
| Приватный bucket вне API, права ключей | `python3 tests/s3_checks.py` | анонимный GET/PUT/list и неверные ключи → 403; ключ API не создаёт bucket; наружу опубликован только `lb` | evidence/s3-checks.txt |
| Недоступность S3 | `python3 tests/s3_checks.py` | при остановленном S3 загрузка и скачивание → 503, метаданные и текстовые сообщения работают, после запуска всё восстанавливается | evidence/s3-checks.txt |
| Очистка неучтённых объектов | `python3 tests/s3_checks.py` | свежий объект сохраняется, объект старше grace удаляется, используемый объект остаётся | evidence/s3-checks.txt |
| ≥10 одновременных загрузок, пик памяти | `python3 tests/upload_memory.py` | 12 потоков × 3 файла по 9.3 MiB, 36 из 36 успешно, пик api 103–120 MiB при лимите 192 MiB | evidence/upload-memory.json |
| Идемпотентность из lab3 | `python3 tests/idempotency.py` | 32 параллельных запроса → 1 сообщение | evidence/idempotency.txt |
| Отказы каждой реплики и rolling update под нагрузкой | `tests/scenario.py` | RPS, ошибки, восстановление | evidence/failover-each.json, evidence/rolling.json |
| Идемпотентный bootstrap bucket | повторный `docker compose up` | `s3-init` завершается с кодом 0, bucket не пересоздаётся | `docker compose logs s3-init` |
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Стенд: один хост; лимиты контейнеров как выше. Нагрузка сообщений — 8 потоков, closed-loop, при ошибке повтор с тем же ключом. Загрузки — 12 потоков по 3 файла PNG 9 722 608 байт (несжимаемый шум).
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
| Загрузка 12×3 файла параллельно | память API ограничена | пик api 120 / 103 MiB (в покое около 62 MiB), ошибок нет | потерь нет |
| Остановка S3 | 503 на файлах, текст работает | как ожидалось | после запуска S3 около 20–30 s до готовности записи, файлы целы |
| Остановка каждой API-реплики (под нагрузкой сообщений) | продолжение чтения и записи | 0 неуспешных попыток, RPS около 440 против 520 до отказа | возврат реплики в ротацию около 6.5–7 s после старта |
| Rolling update | без простоя | 4 неуспешные попытки, RPS около 517 | старые реплики заменены и не отвечают, дубликатов нет |
| Restart всего стенда | данные и оригинал сохраняются | PASS persistence | прежний токен действителен |
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
- Целостность изображения проверяется по заголовку и хвосту файла, но не полным декодированием: повреждение в середине файла с корректными заголовком и концом не обнаруживается. Полное декодирование потребовало бы памяти, пропорциональной размеру изображения.
- Падение процесса между S3 PUT и INSERT оставляет объект до очистки `janitor`; задержка до удаления — grace period плюс интервал (до 15 минут).
- Если S3 перезапускается, запись становится доступной через 20–30 s после старта; healthcheck S3 учитывает это.
- Filer SeaweedFS слушает внутри сети `storage` без аутентификации. Доступ к нему имеют только контейнеры `api`, `janitor`, `s3-init`; наружу порт не публикуется. S3-аутентификация действует на порту 9000.
- HAProxy, PostgreSQL, Valkey и S3 — единые точки отказа; репликации нет, выживание отказа этих узлов не заявляется.
- Измерение памяти — выборка `docker stats` раз в секунду, это оценка, а не точный пик.
- Нагрузочные генераторы Python (closed-loop) ограничены клиентом, RPS — ориентир.
- Секреты лежат в `./secrets` вне Git и читаются всеми локально, чтобы их читал процесс с UID 10001; это допустимо только для локального стенда.
+430
View File
@@ -0,0 +1,430 @@
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"hash"
"image"
_ "image/jpeg"
_ "image/png"
"io"
"log/slog"
"net/http"
"strconv"
"strings"
"time"
"unicode"
"unicode/utf8"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
const (
objectPrefix = "attachments/"
probeHead = 256 << 10
tailLen = 12
)
var (
pngMagic = []byte{0x89, 'P', 'N', 'G', '\r', '\n', 0x1a, '\n'}
pngTail = []byte{0, 0, 0, 0, 'I', 'E', 'N', 'D', 0xae, 0x42, 0x60, 0x82}
jpegMagic = []byte{0xff, 0xd8, 0xff}
jpegTail = []byte{0xff, 0xd9}
)
type limits struct {
maxBytes int64
maxPixels int64
uploadSlots int
}
type attachment struct {
ID, ChatID, OwnerID, Filename, ContentType, SHA256, Key string
Size int64
Created time.Time
}
func (x *attachment) json() map[string]any {
return map[string]any{
"id": x.ID,
"chat_id": x.ChatID,
"owner_id": x.OwnerID,
"filename": x.Filename,
"content_type": x.ContentType,
"size_bytes": x.Size,
"sha256": x.SHA256,
"status": "ready",
"variants": []any{},
"error_code": nil,
"created_at": formatTime(x.Created),
"updated_at": formatTime(x.Created),
}
}
type probe struct {
sum hash.Hash
size int64
head []byte
tail []byte
}
func newProbe() *probe {
return &probe{sum: sha256.New()}
}
func (p *probe) Write(b []byte) (int, error) {
p.sum.Write(b)
p.size += int64(len(b))
if room := probeHead - len(p.head); room > 0 {
if room > len(b) {
room = len(b)
}
p.head = append(p.head, b[:room]...)
}
p.tail = append(p.tail, b...)
if len(p.tail) > tailLen {
p.tail = append(p.tail[:0], p.tail[len(p.tail)-tailLen:]...)
}
return len(b), nil
}
func (p *probe) digest() string {
return hex.EncodeToString(p.sum.Sum(nil))
}
var (
errNotImage = errors.New("not an image")
errTooBig = errors.New("too large")
)
func (p *probe) verify(contentType string, maxPixels int64) error {
cfg, format, err := image.DecodeConfig(bytes.NewReader(p.head))
if err != nil {
return errNotImage
}
var tail []byte
switch contentType {
case "image/png":
if format != "png" {
return errNotImage
}
tail = pngTail
case "image/jpeg":
if format != "jpeg" {
return errNotImage
}
tail = jpegTail
}
if !bytes.HasSuffix(p.tail, tail) {
return errNotImage
}
if int64(cfg.Width)*int64(cfg.Height) > maxPixels {
return errTooBig
}
return nil
}
func magicMatches(contentType string, head []byte) bool {
switch contentType {
case "image/png":
return bytes.HasPrefix(head, pngMagic)
case "image/jpeg":
return bytes.HasPrefix(head, jpegMagic)
}
return false
}
func validFilename(name string) bool {
n := utf8.RuneCountInString(name)
if n < 1 || n > 255 || !utf8.ValidString(name) {
return false
}
for _, r := range name {
if r == '/' || r == '\\' || r == 0 || unicode.IsControl(r) {
return false
}
}
return true
}
type trackedReader struct {
r io.Reader
err error
}
func (t *trackedReader) Read(b []byte) (int, error) {
n, err := t.r.Read(b)
if err != nil && err != io.EOF && t.err == nil {
t.err = err
}
return n, err
}
func isTooLarge(err error) bool {
var mbe *http.MaxBytesError
return errors.As(err, &mbe)
}
func (a *App) dropObject(key string) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := a.store.Remove(ctx, key); err != nil {
a.log.Warn("object cleanup failed", "key", key, "err", err.Error())
}
}
func (a *App) handleUploadAttachment(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
u, _ := a.authUser(w, r)
if u == nil {
return
}
chatID := r.PathValue("chat_id")
if !isUUID(chatID) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid chat_id")
return
}
ok, err := a.isMember(r.Context(), chatID, u.ID)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
if !ok {
a.writeErr(w, reqID, 404, "not_found", "chat not found")
return
}
select {
case a.uploads <- struct{}{}:
defer func() { <-a.uploads }()
default:
w.Header().Set("Retry-After", "1")
a.writeErr(w, reqID, 429, "rate_limited", "too many concurrent uploads")
return
}
filename := r.URL.Query().Get("filename")
if !validFilename(filename) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid filename")
return
}
contentType := strings.ToLower(strings.TrimSpace(strings.Split(r.Header.Get("Content-Type"), ";")[0]))
if contentType != "image/png" && contentType != "image/jpeg" {
a.writeErr(w, reqID, 415, "unsupported_media_type", "Content-Type must be image/png or image/jpeg")
return
}
if r.ContentLength > a.limits.maxBytes {
a.writeErr(w, reqID, 413, "payload_too_large", "file too large")
return
}
body := &trackedReader{r: http.MaxBytesReader(w, r.Body, a.limits.maxBytes)}
head := make([]byte, 16)
n, _ := io.ReadFull(body, head)
if body.err != nil {
if isTooLarge(body.err) {
a.writeErr(w, reqID, 413, "payload_too_large", "file too large")
} else {
a.writeErr(w, reqID, 400, "invalid_request", "upload interrupted")
}
return
}
head = head[:n]
if !magicMatches(contentType, head) {
a.writeErr(w, reqID, 415, "unsupported_media_type", "content does not match image type")
return
}
id := newUUID()
key := objectPrefix + id
pr := newProbe()
size := r.ContentLength
if size < 0 {
size = -1
}
src := io.TeeReader(io.MultiReader(bytes.NewReader(head), body), pr)
putCtx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
defer cancel()
if err := a.store.Put(putCtx, key, src, size, contentType); err != nil {
a.dropObject(key)
switch {
case isTooLarge(body.err):
a.writeErr(w, reqID, 413, "payload_too_large", "file too large")
case body.err != nil:
a.writeErr(w, reqID, 400, "invalid_request", "upload interrupted")
default:
a.log.Error("s3 put failed", "request_id", reqID, "err", err.Error())
a.writeErr(w, reqID, 503, "unavailable", "storage unavailable")
}
return
}
if err := pr.verify(contentType, a.limits.maxPixels); err != nil {
a.dropObject(key)
if err == errTooBig {
a.writeErr(w, reqID, 413, "payload_too_large", "image too large")
} else {
a.writeErr(w, reqID, 415, "unsupported_media_type", "invalid image")
}
return
}
att := &attachment{
ID: id, ChatID: chatID, OwnerID: u.ID, Filename: filename, ContentType: contentType,
SHA256: pr.digest(), Key: key, Size: pr.size, Created: nowUTC(),
}
ctx, cancelDB := dbCtx(r.Context())
defer cancelDB()
_, err = a.db.Exec(ctx, `
insert into attachments(id, chat_id, owner_id, filename, content_type, size_bytes, sha256, object_key, created_at)
values($1::uuid, $2::uuid, $3::uuid, $4, $5, $6, $7, $8, $9)`,
att.ID, att.ChatID, att.OwnerID, att.Filename, att.ContentType, att.Size, att.SHA256, att.Key, att.Created)
if err != nil {
a.dropObject(key)
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.writeJSON(w, 201, att.json())
}
func (a *App) loadAttachment(ctx context.Context, id, userID string) (*attachment, error) {
var x attachment
err := a.db.QueryRow(ctx, `
select a.id::text, a.chat_id::text, a.owner_id::text, a.filename, a.content_type,
a.size_bytes, a.sha256, a.object_key, a.created_at
from attachments a
join chat_members m on m.chat_id = a.chat_id and m.user_id = $2::uuid
where a.id = $1::uuid`, id, userID).
Scan(&x.ID, &x.ChatID, &x.OwnerID, &x.Filename, &x.ContentType, &x.Size, &x.SHA256, &x.Key, &x.Created)
if errors.Is(err, pgx.ErrNoRows) {
return nil, nil
}
if err != nil {
return nil, err
}
return &x, nil
}
func (a *App) attachmentFor(w http.ResponseWriter, r *http.Request) *attachment {
reqID := reqIDFrom(r)
u, _ := a.authUser(w, r)
if u == nil {
return nil
}
id := r.PathValue("attachment_id")
if !isUUID(id) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid attachment_id")
return nil
}
ctx, cancel := dbCtx(r.Context())
defer cancel()
att, err := a.loadAttachment(ctx, id, u.ID)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return nil
}
if att == nil {
a.writeErr(w, reqID, 404, "not_found", "attachment not found")
return nil
}
return att
}
func (a *App) handleGetAttachment(w http.ResponseWriter, r *http.Request) {
if att := a.attachmentFor(w, r); att != nil {
a.writeJSON(w, 200, att.json())
}
}
func (a *App) handleDownloadAttachment(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
if v := r.URL.Query().Get("variant"); v != "" && v != "original" {
a.writeErr(w, reqID, 400, "invalid_request", "invalid variant")
return
}
att := a.attachmentFor(w, r)
if att == nil {
return
}
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
defer cancel()
obj, info, err := a.store.Get(ctx, att.Key)
if err != nil {
if isNoSuchKey(err) {
a.writeErr(w, reqID, 404, "not_found", "attachment not found")
return
}
a.log.Error("s3 get failed", "request_id", reqID, "err", err.Error())
a.writeErr(w, reqID, 503, "unavailable", "storage unavailable")
return
}
defer obj.Close()
first := make([]byte, 32<<10)
n, err := io.ReadFull(obj, first)
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF {
a.log.Error("s3 read failed", "request_id", reqID, "err", err.Error())
a.writeErr(w, reqID, 503, "unavailable", "storage unavailable")
return
}
w.Header().Set("Content-Type", att.ContentType)
w.Header().Set("Content-Length", strconv.FormatInt(info.Size, 10))
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(200)
_, _ = w.Write(first[:n])
_, _ = io.Copy(w, obj)
}
func cleanupOrphans(ctx context.Context, db *pgxpool.Pool, store *Store, grace time.Duration, log *slog.Logger) (int, error) {
cutoff := time.Now().Add(-grace)
removed := 0
flush := func(keys []string) error {
if len(keys) == 0 {
return nil
}
rows, err := db.Query(ctx, `select object_key from attachments where object_key = any($1)`, keys)
if err != nil {
return err
}
known := map[string]bool{}
for rows.Next() {
var k string
if err := rows.Scan(&k); err != nil {
rows.Close()
return err
}
known[k] = true
}
rows.Close()
if err := rows.Err(); err != nil {
return err
}
for _, k := range keys {
if known[k] {
continue
}
if err := store.Remove(ctx, k); err != nil {
return err
}
log.Info("orphan object removed", "key", k)
removed++
}
return nil
}
var batch []string
for obj := range store.List(ctx) {
if obj.Err != nil {
return removed, obj.Err
}
if obj.LastModified.After(cutoff) {
continue
}
batch = append(batch, obj.Key)
if len(batch) == 500 {
if err := flush(batch); err != nil {
return removed, err
}
batch = batch[:0]
}
}
return removed, flush(batch)
}
+131
View File
@@ -0,0 +1,131 @@
package main
import (
"crypto/rand"
"crypto/sha256"
"crypto/subtle"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"strconv"
"strings"
"unicode/utf8"
"golang.org/x/crypto/argon2"
)
const (
argonTime = 2
argonMemory = 19456
argonThreads = 1
argonKeyLen = 32
argonSaltLen = 16
)
func hashPassword(password string) (string, error) {
salt := make([]byte, argonSaltLen)
if _, err := rand.Read(salt); err != nil {
return "", err
}
hash := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
b64Salt := base64.RawStdEncoding.EncodeToString(salt)
b64Hash := base64.RawStdEncoding.EncodeToString(hash)
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
argon2.Version, argonMemory, argonTime, argonThreads, b64Salt, b64Hash), nil
}
func checkPassword(encoded, password string) (bool, error) {
parts := strings.Split(encoded, "$")
if len(parts) != 6 || parts[1] != "argon2id" {
return false, errors.New("bad hash")
}
var version int
if _, err := fmt.Sscanf(parts[2], "v=%d", &version); err != nil {
return false, err
}
var memory, timeCost, threads int
for _, p := range strings.Split(parts[3], ",") {
kv := strings.SplitN(p, "=", 2)
if len(kv) != 2 {
return false, errors.New("bad params")
}
n, err := strconv.Atoi(kv[1])
if err != nil {
return false, err
}
switch kv[0] {
case "m":
memory = n
case "t":
timeCost = n
case "p":
threads = n
}
}
salt, err := base64.RawStdEncoding.DecodeString(parts[4])
if err != nil {
return false, err
}
want, err := base64.RawStdEncoding.DecodeString(parts[5])
if err != nil {
return false, err
}
got := argon2.IDKey([]byte(password), salt, uint32(timeCost), uint32(memory), uint8(threads), uint32(len(want)))
return subtle.ConstantTimeCompare(got, want) == 1, nil
}
func newToken() (string, []byte, string, error) {
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
return "", nil, "", err
}
token := base64.RawURLEncoding.EncodeToString(raw)
sum := sha256.Sum256([]byte(token))
hash := make([]byte, len(sum))
copy(hash, sum[:])
return token, hash, hex.EncodeToString(hash), nil
}
func tokenHash(token string) ([]byte, string) {
sum := sha256.Sum256([]byte(token))
hash := make([]byte, len(sum))
copy(hash, sum[:])
return hash, hex.EncodeToString(hash)
}
func parseBasic(header string) (username, password string, ok bool) {
const prefix = "Basic "
if !strings.HasPrefix(header, prefix) {
return "", "", false
}
decoded, err := base64.StdEncoding.DecodeString(strings.TrimSpace(header[len(prefix):]))
if err != nil {
return "", "", false
}
s := string(decoded)
i := strings.IndexByte(s, ':')
if i < 0 {
return "", "", false
}
username = s[:i]
password = s[i+1:]
return username, password, true
}
func parseBearer(header string) (string, bool) {
const prefix = "Bearer "
if !strings.HasPrefix(header, prefix) {
return "", false
}
tok := strings.TrimSpace(header[len(prefix):])
if tok == "" {
return "", false
}
return tok, true
}
func validPassword(p string) bool {
n := utf8.RuneCountInString(p)
return n >= 12 && n <= 128
}
+104
View File
@@ -0,0 +1,104 @@
package main
import (
"context"
"encoding/json"
"log/slog"
"time"
"github.com/redis/go-redis/v9"
)
type cachedSession struct {
UserID string `json:"user_id"`
ExpiresAt time.Time `json:"expires_at"`
}
type Cache struct {
rdb *redis.Client
log *slog.Logger
}
func openCache(log *slog.Logger) *Cache {
url, err := readSecret("CACHE_URL", "CACHE_URL_FILE")
if err != nil {
log.Warn("cache disabled", "err", err.Error())
return &Cache{log: log}
}
opt, err := redis.ParseURL(url)
if err != nil {
log.Warn("cache parse failed", "err", err.Error())
return &Cache{log: log}
}
opt.DialTimeout = 200 * time.Millisecond
opt.ReadTimeout = 200 * time.Millisecond
opt.WriteTimeout = 200 * time.Millisecond
opt.PoolTimeout = 200 * time.Millisecond
rdb := redis.NewClient(opt)
ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
defer cancel()
if err := rdb.Ping(ctx).Err(); err != nil {
log.Warn("cache ping failed, will bypass", "err", err.Error())
}
return &Cache{rdb: rdb, log: log}
}
func (c *Cache) Close() {
if c.rdb != nil {
_ = c.rdb.Close()
}
}
func sessKey(hashHex string) string {
return "sess:" + hashHex
}
func (c *Cache) GetSession(ctx context.Context, hashHex string) (*cachedSession, bool) {
if c.rdb == nil {
return nil, false
}
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
val, err := c.rdb.Get(ctx, sessKey(hashHex)).Result()
if err != nil {
return nil, false
}
var s cachedSession
if err := json.Unmarshal([]byte(val), &s); err != nil {
return nil, false
}
if time.Now().UTC().After(s.ExpiresAt) {
return nil, false
}
return &s, true
}
func (c *Cache) SetSession(ctx context.Context, hashHex string, userID string, expiresAt time.Time) {
if c.rdb == nil {
return
}
ttl := time.Until(expiresAt)
if ttl <= 0 {
return
}
body, err := json.Marshal(cachedSession{UserID: userID, ExpiresAt: expiresAt.UTC()})
if err != nil {
return
}
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
if err := c.rdb.Set(ctx, sessKey(hashHex), body, ttl).Err(); err != nil {
c.log.Warn("cache set failed", "err", err.Error())
}
}
func (c *Cache) DelSession(ctx context.Context, hashHex string) {
if c.rdb == nil {
return
}
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
if err := c.rdb.Del(ctx, sessKey(hashHex)).Err(); err != nil {
c.log.Warn("cache del failed", "err", err.Error())
}
}
+244
View File
@@ -0,0 +1,244 @@
x-api-env: &api-env
APP_HOST: 0.0.0.0
APP_PORT: 8080
APP_ENV: ${APP_ENV:-lab}
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
SESSION_STORE: postgres
S3_ENDPOINT: http://s3:9000
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
UPLOAD_CONCURRENCY: ${UPLOAD_CONCURRENCY:-10}
GOMEMLIMIT: 150MiB
SHUTDOWN_TIMEOUT_SECONDS: 10
DRAIN_SECONDS: 3
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
services:
postgres:
image: postgres:17-alpine@sha256:b0f9560a2de083e2cc7382e75f808c7381a32852a7ec49117deedb300e552b24
environment:
POSTGRES_USER: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
POSTGRES_DB: messenger
secrets:
- postgres_password
volumes:
- pg_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
interval: 5s
timeout: 3s
retries: 10
start_period: 5s
deploy:
resources:
limits:
cpus: "1.0"
memory: 512M
networks:
- backend
restart: unless-stopped
valkey:
image: valkey/valkey:8.1-alpine@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
command: ["valkey-server", "--appendonly", "yes", "--save", "60", "1"]
volumes:
- valkey_data:/data
healthcheck:
test: ["CMD", "valkey-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
start_period: 5s
deploy:
resources:
limits:
cpus: "0.5"
memory: 128M
networks:
- backend
restart: unless-stopped
s3:
image: chrislusf/seaweedfs@sha256:c2658820e763294a9e9de7b36c81e59b879e3459148351aee4e31a126ca64860
command:
- server
- -dir=/data
- -ip=s3
- -master.volumeSizeLimitMB=128
- -volume.max=30
- -s3
- -s3.port=9000
- -s3.config=/run/secrets/s3_config
secrets:
- s3_config
volumes:
- s3_data:/data
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9333/dir/assign | grep -q fid"]
interval: 5s
timeout: 3s
retries: 12
start_period: 10s
deploy:
resources:
limits:
cpus: "1.0"
memory: 512M
networks:
- storage
restart: unless-stopped
s3-init:
image: mtusi-messenger:lab4
command: ["s3-init"]
environment:
S3_ENDPOINT: http://s3:9000
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_admin_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_admin_secret_access_key
secrets:
- s3_admin_access_key_id
- s3_admin_secret_access_key
depends_on:
s3:
condition: service_healthy
migrate:
condition: service_completed_successfully
deploy:
resources:
limits:
cpus: "0.5"
memory: 128M
networks:
- storage
restart: "no"
janitor:
image: mtusi-messenger:lab4
command: ["janitor"]
environment:
DATABASE_URL_FILE: /run/secrets/database_url
S3_ENDPOINT: http://s3:9000
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
CLEANUP_GRACE_SECONDS: ${CLEANUP_GRACE_SECONDS:-600}
CLEANUP_INTERVAL_SECONDS: ${CLEANUP_INTERVAL_SECONDS:-300}
secrets:
- database_url
- s3_access_key_id
- s3_secret_access_key
depends_on:
s3-init:
condition: service_completed_successfully
healthcheck:
disable: true
deploy:
resources:
limits:
cpus: "0.25"
memory: 64M
networks:
- backend
- storage
restart: unless-stopped
migrate:
build: .
image: mtusi-messenger:lab4
command: ["migrate"]
environment:
DATABASE_URL_FILE: /run/secrets/database_url
secrets:
- database_url
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "0.5"
memory: 128M
networks:
- backend
restart: "no"
api:
image: mtusi-messenger:lab4
environment: *api-env
secrets:
- database_url
- cache_url
- s3_access_key_id
- s3_secret_access_key
depends_on:
s3-init:
condition: service_completed_successfully
valkey:
condition: service_healthy
stop_grace_period: 15s
deploy:
replicas: ${API_REPLICAS:-2}
resources:
limits:
cpus: "0.5"
memory: 192M
networks:
- backend
- storage
restart: unless-stopped
lb:
image: haproxy:3.0-alpine@sha256:cf3af0facf7eb22b9fee463daf0fbc07ead740b8150696dce724264faf7b4318
ports:
- "127.0.0.1:8080:8080"
volumes:
- ./lb/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
depends_on:
- api
deploy:
resources:
limits:
cpus: "0.5"
memory: 128M
networks:
- backend
restart: unless-stopped
secrets:
postgres_password:
file: ./secrets/postgres_password
database_url:
file: ./secrets/database_url
cache_url:
file: ./secrets/cache_url
s3_config:
file: ./secrets/s3_config.json
s3_admin_access_key_id:
file: ./secrets/s3_admin_access_key_id
s3_admin_secret_access_key:
file: ./secrets/s3_admin_secret_access_key
s3_access_key_id:
file: ./secrets/s3_access_key_id
s3_secret_access_key:
file: ./secrets/s3_secret_access_key
volumes:
pg_data:
valkey_data:
s3_data:
networks:
backend:
driver: bridge
storage:
driver: bridge
+57
View File
@@ -0,0 +1,57 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
func readSecret(envKey, fileKey string) (string, error) {
if v := os.Getenv(envKey); v != "" {
return strings.TrimSpace(v), nil
}
path := os.Getenv(fileKey)
if path == "" {
return "", fmt.Errorf("%s or %s required", envKey, fileKey)
}
b, err := os.ReadFile(path)
if err != nil {
return "", err
}
return strings.TrimSpace(string(b)), nil
}
func openDB(ctx context.Context) (*pgxpool.Pool, error) {
dsn, err := readSecret("DATABASE_URL", "DATABASE_URL_FILE")
if err != nil {
return nil, err
}
cfg, err := pgxpool.ParseConfig(dsn)
if err != nil {
return nil, err
}
cfg.MaxConns = 10
cfg.MinConns = 2
cfg.MaxConnLifetime = 5 * time.Minute
cfg.ConnConfig.ConnectTimeout = 3 * time.Second
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
return nil, err
}
pingCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
if err := pool.Ping(pingCtx); err != nil {
pool.Close()
return nil, err
}
return pool, nil
}
func dbCtx(parent context.Context) (context.Context, context.CancelFunc) {
return context.WithTimeout(parent, 3*time.Second)
}
+1
View File
@@ -0,0 +1 @@
PASS: attachment validation, limits and access control
+2
View File
@@ -0,0 +1,2 @@
{"observed_instances_before": 2, "successful_writes": 140, "failed_attempts": 0, "first_recovery_seconds": 3.46}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+2
View File
@@ -0,0 +1,2 @@
{"observed_instances_before": 2, "successful_writes": 139, "failed_attempts": 0, "first_recovery_seconds": 3.453}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+71
View File
@@ -0,0 +1,71 @@
{
"baseline": {
"window_seconds": 6,
"successful_requests": 3132,
"failed_attempts": 0,
"successful_rps": 522.0,
"last_error_offset_seconds": null,
"instances": [
"2eb97a202e04",
"e1e2d306d0d7"
]
},
"replicas": [
{
"stopped_replica": "2eb97a202e04",
"during_failure": {
"window_seconds": 15.34,
"successful_requests": 6776,
"failed_attempts": 0,
"successful_rps": 441.8,
"last_error_offset_seconds": null,
"instances": [
"2eb97a202e04",
"e1e2d306d0d7"
]
},
"seen_stopped_replica_after_stop": false,
"stop_command_seconds": 3.34,
"restore_seconds": 6.48,
"after_restore": {
"window_seconds": 7.0,
"successful_requests": 3639,
"failed_attempts": 0,
"successful_rps": 519.9,
"last_error_offset_seconds": null,
"instances": [
"2eb97a202e04",
"e1e2d306d0d7"
]
}
},
{
"stopped_replica": "e1e2d306d0d7",
"during_failure": {
"window_seconds": 15.49,
"successful_requests": 6805,
"failed_attempts": 0,
"successful_rps": 439.4,
"last_error_offset_seconds": null,
"instances": [
"2eb97a202e04",
"e1e2d306d0d7"
]
},
"seen_stopped_replica_after_stop": false,
"stop_command_seconds": 3.49,
"restore_seconds": 6.93,
"after_restore": {
"window_seconds": 7.0,
"successful_requests": 3618,
"failed_attempts": 0,
"successful_rps": 516.9,
"last_error_offset_seconds": null,
"instances": [
"2eb97a202e04",
"e1e2d306d0d7"
]
}
}
]
}
+1
View File
@@ -0,0 +1 @@
PASS: 32 parallel requests, 1 message, 2 instances
+1
View File
@@ -0,0 +1 @@
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+24
View File
@@ -0,0 +1,24 @@
{
"old_replicas": [
"2eb97a202e04",
"e1e2d306d0d7"
],
"new_replicas": [
"525377e6c5c3",
"ffb74e9857ae"
],
"update_window": {
"window_seconds": 16.54,
"successful_requests": 8547,
"failed_attempts": 4,
"successful_rps": 516.8,
"last_error_offset_seconds": 13.5,
"instances": [
"2eb97a202e04",
"525377e6c5c3",
"e1e2d306d0d7",
"ffb74e9857ae"
]
},
"old_replicas_served_after_update": []
}
+1
View File
@@ -0,0 +1 @@
PASS: private bucket, limited credentials, s3 outage, orphan cleanup
+1
View File
@@ -0,0 +1 @@
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
+18
View File
@@ -0,0 +1,18 @@
{
"file_bytes": 9722608,
"concurrent_uploads": 12,
"rounds": 3,
"uploads_ok": 36,
"uploads_failed": 0,
"elapsed_seconds": 12.49,
"api_memory_limit_mib": 192,
"idle_memory_mib": {
"mtusi2026lab4-api-5": 62.2,
"mtusi2026lab4-api-6": 64.0
},
"peak_memory_mib": {
"mtusi2026lab4-api-5": 120.1,
"mtusi2026lab4-api-6": 103.4
},
"buffering_all_files_would_need_mib": 111.3
}
+35
View File
@@ -0,0 +1,35 @@
module mtusi-messenger
go 1.26.0
require (
github.com/jackc/pgx/v5 v5.11.0
github.com/minio/minio-go/v7 v7.4.1
github.com/redis/go-redis/v9 v9.23.0
golang.org/x/crypto v0.58.0
)
require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/compress v1.19.2 // indirect
github.com/klauspost/cpuid/v2 v2.4.0 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/tinylib/msgp v1.6.4 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
go.uber.org/atomic v1.12.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/net v0.60.0 // indirect
golang.org/x/sync v0.24.0 // indirect
golang.org/x/sys v0.49.0 // indirect
golang.org/x/text v0.43.0 // indirect
gopkg.in/ini.v1 v1.67.3 // indirect
)
+77
View File
@@ -0,0 +1,77 @@
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8=
github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM=
github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw=
github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
github.com/minio/crc64nvme v1.1.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
github.com/minio/minio-go/v7 v7.4.1 h1:PtU/zIOfQRWZ+QydXAZysc+xZYHdG+BBR4Z6TUvy73w=
github.com/minio/minio-go/v7 v7.4.1/go.mod h1:nQnQi7zTfXR4lREtYXZQWqo5aw0Rqv+ZqCGrTORygRk=
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/redis/go-redis/v9 v9.23.0 h1:/TTVdJa8BRNsybwwGGnmiYktHNlUXtbOQxlWpHBPL3s=
github.com/redis/go-redis/v9 v9.23.0/go.mod h1:EWP2UUk+XDaI1s5nQB5mablZN6vccXlmBQTyfnF87D0=
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/tinylib/msgp v1.6.4 h1:mOwYbyYDLPj35mkA2BjjYejgJk9BuHxDdvRnb6v2ZcQ=
github.com/tinylib/msgp v1.6.4/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE=
go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/ini.v1 v1.67.3 h1:iM9Lhz5MRSGhHVGGwCuzG9KO8PoirCXj/m/qTmOJJQw=
gopkg.in/ini.v1 v1.67.3/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+1085
View File
File diff suppressed because it is too large Load Diff
+38
View File
@@ -0,0 +1,38 @@
global
log stdout format raw local0
maxconn 2000
resolvers docker
nameserver dns 127.0.0.11:53
accepted_payload_size 8192
hold valid 1s
hold nx 1s
hold obsolete 1s
resolve_retries 3
timeout resolve 1s
timeout retry 1s
defaults
mode http
log global
option httplog
option redispatch 1
retries 3
retry-on conn-failure empty-response
timeout connect 1s
timeout client 20s
timeout server 15s
timeout http-request 10s
timeout queue 5s
frontend public
bind :8080
default_backend api
backend api
balance roundrobin
option httpchk
http-check send meth GET uri /health/ready
http-check expect status 200
default-server check inter 1s fastinter 500ms downinter 1s fall 2 rise 2 resolvers docker init-addr none on-marked-down shutdown-sessions
server-template api 1-8 api:8080
+136
View File
@@ -0,0 +1,136 @@
package main
import (
"context"
"log/slog"
"net"
"net/http"
"os"
"os/signal"
"strconv"
"syscall"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
func main() {
log := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
host := envOr("APP_HOST", "0.0.0.0")
port := envOr("APP_PORT", "8080")
shutdownSec, _ := strconv.Atoi(envOr("SHUTDOWN_TIMEOUT_SECONDS", "5"))
if shutdownSec < 1 {
shutdownSec = 5
}
ttlSec, _ := strconv.Atoi(envOr("SESSION_TTL_SECONDS", "3600"))
if ttlSec < 1 {
ttlSec = 3600
}
idemSec, _ := strconv.Atoi(envOr("IDEMPOTENCY_TTL_SECONDS", "86400"))
if idemSec < 1 {
idemSec = 86400
}
drainSec, _ := strconv.Atoi(envOr("DRAIN_SECONDS", "3"))
ctx := context.Background()
if len(os.Args) > 1 && os.Args[1] == "s3-init" {
if err := s3Init(ctx); err != nil {
log.Error("s3 init failed", "err", err.Error())
os.Exit(1)
}
log.Info("bucket ready")
return
}
db, err := openDB(ctx)
if err != nil {
log.Error("database open failed", "err", err.Error())
os.Exit(1)
}
defer db.Close()
if len(os.Args) > 1 && os.Args[1] == "migrate" {
if err := migrate(ctx, db); err != nil {
log.Error("migrate failed", "err", err.Error())
os.Exit(1)
}
log.Info("migrations applied")
return
}
store, err := openStore()
if err != nil {
log.Error("s3 open failed", "err", err.Error())
os.Exit(1)
}
if len(os.Args) > 1 && (os.Args[1] == "cleanup" || os.Args[1] == "janitor") {
runCleanup(ctx, db, store, log, os.Args[1] == "janitor")
return
}
cache := openCache(log)
defer cache.Close()
maxBytes, _ := strconv.ParseInt(envOr("MAX_UPLOAD_BYTES", "10485760"), 10, 64)
maxPixels, _ := strconv.ParseInt(envOr("MAX_IMAGE_PIXELS", "40000000"), 10, 64)
slots, _ := strconv.Atoi(envOr("UPLOAD_CONCURRENCY", "10"))
if slots < 1 {
slots = 10
}
app := NewApp(db, cache, log, time.Duration(ttlSec)*time.Second, time.Duration(idemSec)*time.Second, store, limits{maxBytes: maxBytes, maxPixels: maxPixels, uploadSlots: slots})
addr := net.JoinHostPort(host, port)
srv := &http.Server{
Addr: addr,
Handler: app.Routes(),
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 10 * time.Second,
WriteTimeout: 15 * time.Second,
IdleTimeout: 30 * time.Second,
}
go func() {
log.Info("listening", "addr", addr, "env", envOr("APP_ENV", "lab"), "session_ttl_s", ttlSec)
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Error("server error", "err", err.Error())
os.Exit(1)
}
}()
stop := make(chan os.Signal, 1)
signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)
sig := <-stop
log.Info("shutdown started", "signal", sig.String())
app.SetReady(false)
time.Sleep(time.Duration(drainSec) * time.Second)
shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(shutdownSec)*time.Second)
defer cancel()
if err := srv.Shutdown(shCtx); err != nil {
log.Error("shutdown error", "err", err.Error())
os.Exit(1)
}
log.Info("shutdown finished")
}
func runCleanup(ctx context.Context, db *pgxpool.Pool, store *Store, log *slog.Logger, loop bool) {
graceSec, _ := strconv.Atoi(envOr("CLEANUP_GRACE_SECONDS", "600"))
intervalSec, _ := strconv.Atoi(envOr("CLEANUP_INTERVAL_SECONDS", "300"))
for {
removed, err := cleanupOrphans(ctx, db, store, time.Duration(graceSec)*time.Second, log)
if err != nil {
log.Error("cleanup failed", "err", err.Error())
if !loop {
os.Exit(1)
}
} else {
log.Info("cleanup finished", "removed", removed)
}
if !loop {
return
}
time.Sleep(time.Duration(intervalSec) * time.Second)
}
}
+84
View File
@@ -0,0 +1,84 @@
package main
import (
"context"
"embed"
"fmt"
"sort"
"strings"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
//go:embed migrations/*.sql
var migrationFS embed.FS
func migrate(ctx context.Context, pool *pgxpool.Pool) error {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
conn, err := pool.Acquire(ctx)
if err != nil {
return err
}
defer conn.Release()
if _, err := conn.Exec(ctx, `select pg_advisory_lock(727001)`); err != nil {
return err
}
defer conn.Exec(context.Background(), `select pg_advisory_unlock(727001)`)
_, err = conn.Exec(ctx, `
create table if not exists schema_migrations (
version text primary key,
applied_at timestamptz not null
)`)
if err != nil {
return fmt.Errorf("schema_migrations: %w", err)
}
entries, err := migrationFS.ReadDir("migrations")
if err != nil {
return err
}
var names []string
for _, e := range entries {
if !e.IsDir() && strings.HasSuffix(e.Name(), ".sql") {
names = append(names, e.Name())
}
}
sort.Strings(names)
for _, name := range names {
var exists bool
err := conn.QueryRow(ctx, `select exists(select 1 from schema_migrations where version=$1)`, name).Scan(&exists)
if err != nil {
return err
}
if exists {
continue
}
body, err := migrationFS.ReadFile("migrations/" + name)
if err != nil {
return err
}
tx, err := conn.Begin(ctx)
if err != nil {
return err
}
if _, err := tx.Exec(ctx, string(body)); err != nil {
_ = tx.Rollback(ctx)
return fmt.Errorf("migration %s: %w", name, err)
}
if _, err := tx.Exec(ctx, `insert into schema_migrations(version, applied_at) values($1, now())`, name); err != nil {
_ = tx.Rollback(ctx)
return err
}
if err := tx.Commit(ctx); err != nil {
return err
}
}
return nil
}
+38
View File
@@ -0,0 +1,38 @@
create table users (
id uuid primary key,
username text not null unique,
display_name text not null,
password_hash text not null,
created_at timestamptz not null
);
create table chats (
id uuid primary key,
title text not null,
created_at timestamptz not null
);
create table chat_members (
chat_id uuid not null references chats(id),
user_id uuid not null references users(id),
primary key (chat_id, user_id)
);
create index chat_members_user_idx on chat_members (user_id);
create table messages (
id uuid primary key,
chat_id uuid not null references chats(id),
sender_id uuid not null references users(id),
text text not null,
created_at timestamptz not null
);
create index messages_history_idx on messages (chat_id, created_at, id);
create table sessions (
token_hash bytea primary key,
user_id uuid not null references users(id) on delete cascade,
created_at timestamptz not null,
expires_at timestamptz not null
);
+10
View File
@@ -0,0 +1,10 @@
create table idempotency_keys (
user_id uuid not null references users(id) on delete cascade,
method text not null,
path text not null,
key text not null,
body_hash text not null,
response jsonb not null,
created_at timestamptz not null,
primary key (user_id, method, path, key)
);
+22
View File
@@ -0,0 +1,22 @@
create table attachments (
id uuid primary key,
chat_id uuid not null references chats(id),
owner_id uuid not null references users(id),
filename text not null,
content_type text not null,
size_bytes bigint not null,
sha256 text not null,
object_key text not null unique,
created_at timestamptz not null
);
create index attachments_chat_idx on attachments (chat_id);
create table message_attachments (
message_id uuid not null references messages(id),
attachment_id uuid not null references attachments(id),
position int not null,
primary key (message_id, attachment_id)
);
create index message_attachments_attachment_idx on message_attachments (attachment_id);
+33
View File
@@ -0,0 +1,33 @@
package main
import "time"
type User struct {
ID string
Username string
DisplayName string
PasswordHash string
CreatedAt time.Time
}
type Chat struct {
ID string
Title string
MemberIDs []string
CreatedAt time.Time
}
type Message struct {
ID string
ChatID string
SenderID string
Text string
CreatedAt time.Time
}
type Session struct {
TokenHash []byte
UserID string
CreatedAt time.Time
ExpiresAt time.Time
}
+109
View File
@@ -0,0 +1,109 @@
package main
import (
"context"
"io"
"net"
"net/http"
"net/url"
"time"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
type Store struct {
client *minio.Client
bucket string
}
func openStore() (*Store, error) {
endpoint, err := url.Parse(envOr("S3_ENDPOINT", "http://s3:9000"))
if err != nil {
return nil, err
}
keyID, err := readSecret("S3_ACCESS_KEY_ID", "S3_ACCESS_KEY_ID_FILE")
if err != nil {
return nil, err
}
secret, err := readSecret("S3_SECRET_ACCESS_KEY", "S3_SECRET_ACCESS_KEY_FILE")
if err != nil {
return nil, err
}
transport := &http.Transport{
DialContext: (&net.Dialer{Timeout: 2 * time.Second}).DialContext,
ResponseHeaderTimeout: 10 * time.Second,
MaxIdleConnsPerHost: 16,
IdleConnTimeout: 30 * time.Second,
}
client, err := minio.New(endpoint.Host, &minio.Options{
Creds: credentials.NewStaticV4(keyID, secret, ""),
Secure: endpoint.Scheme == "https",
Region: envOr("S3_REGION", "us-east-1"),
Transport: transport,
MaxRetries: 1,
})
if err != nil {
return nil, err
}
return &Store{client: client, bucket: envOr("S3_BUCKET", "messenger-attachments")}, nil
}
func (s *Store) Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) error {
_, err := s.client.PutObject(ctx, s.bucket, key, r, size, minio.PutObjectOptions{
ContentType: contentType,
PartSize: 5 << 20,
})
return err
}
func (s *Store) Get(ctx context.Context, key string) (*minio.Object, minio.ObjectInfo, error) {
obj, err := s.client.GetObject(ctx, s.bucket, key, minio.GetObjectOptions{})
if err != nil {
return nil, minio.ObjectInfo{}, err
}
info, err := obj.Stat()
if err != nil {
obj.Close()
return nil, minio.ObjectInfo{}, err
}
return obj, info, nil
}
func (s *Store) Remove(ctx context.Context, key string) error {
return s.client.RemoveObject(ctx, s.bucket, key, minio.RemoveObjectOptions{})
}
func (s *Store) List(ctx context.Context) <-chan minio.ObjectInfo {
return s.client.ListObjects(ctx, s.bucket, minio.ListObjectsOptions{Prefix: objectPrefix, Recursive: true})
}
func isNoSuchKey(err error) bool {
return minio.ToErrorResponse(err).Code == "NoSuchKey"
}
func s3Init(ctx context.Context) error {
s, err := openStore()
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
for {
exists, err := s.client.BucketExists(ctx, s.bucket)
if err == nil {
if exists {
return nil
}
err = s.client.MakeBucket(ctx, s.bucket, minio.MakeBucketOptions{Region: envOr("S3_REGION", "us-east-1")})
if err == nil || minio.ToErrorResponse(err).Code == "BucketAlreadyOwnedByYou" {
return nil
}
}
select {
case <-ctx.Done():
return err
case <-time.After(2 * time.Second):
}
}
}
+25
View File
@@ -0,0 +1,25 @@
#!/bin/sh
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
mkdir -p "$ROOT/secrets"
cd "$ROOT/secrets"
gen() {
[ -f "$1" ] || openssl rand -hex "$2" | tr -d '\n' > "$1"
}
if [ ! -f postgres_password ]; then
openssl rand -base64 24 | tr -d '\n' > postgres_password
fi
PASS=$(cat postgres_password)
ENC=$(printf '%s' "$PASS" | python3 -c 'import sys,urllib.parse; print(urllib.parse.quote(sys.stdin.read(), safe=""))')
printf 'postgres://messenger:%s@postgres:5432/messenger?sslmode=disable\n' "$ENC" > database_url
printf 'redis://valkey:6379/0\n' > cache_url
gen s3_admin_access_key_id 8
gen s3_admin_secret_access_key 20
gen s3_access_key_id 8
gen s3_secret_access_key 20
BUCKET=$(grep -E '^S3_BUCKET=' "$ROOT/.env" 2>/dev/null | cut -d= -f2 || true)
BUCKET=${BUCKET:-messenger-attachments}
printf '{"identities":[{"name":"admin","credentials":[{"accessKey":"%s","secretKey":"%s"}],"actions":["Admin","Read","Write","List","Tagging"]},{"name":"api","credentials":[{"accessKey":"%s","secretKey":"%s"}],"actions":["Read:%s","Write:%s","List:%s"]}]}\n' \
"$(cat s3_admin_access_key_id)" "$(cat s3_admin_secret_access_key)" "$(cat s3_access_key_id)" "$(cat s3_secret_access_key)" "$BUCKET" "$BUCKET" "$BUCKET" > s3_config.json
chmod 644 ./*
echo "secrets ready in $ROOT/secrets"
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
docker compose --env-file .env restart postgres valkey s3
docker compose --env-file .env up -d --wait postgres valkey s3
docker compose --env-file .env restart api lb janitor
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
dc="docker compose --env-file .env"
old=$($dc ps -q api)
n=$(echo "$old" | wc -w)
$dc build migrate
$dc up -d --no-deps migrate janitor
$dc up -d --no-deps --no-recreate --scale api=$((n * 2)) api
$dc up -d --wait --no-deps --no-recreate --scale api=$((n * 2)) api
for c in $old; do
docker stop -t 10 "$c" >/dev/null
docker rm "$c" >/dev/null
done
$dc up -d --wait --no-deps --no-recreate --scale api=$n api
+10
View File
@@ -0,0 +1,10 @@
#!/bin/sh
set -eu
: "${TARGET_INSTANCE_ID:?}"
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
id=$(docker compose --env-file .env ps -q api | while read -r c; do
case "$c" in "$TARGET_INSTANCE_ID"*) echo "$c";; esac
done)
[ -n "$id" ] || exit 1
docker stop -t 10 "$id" >/dev/null
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env python3
import hashlib
import uuid
from pathlib import Path
from client import Client, arguments, check, run
from image_gen import png, png_header_only
ROOT = Path(__file__).resolve().parent.parent
SAMPLE = (ROOT / 'tests/fixtures/sample.png').read_bytes()
MAX_BYTES = 10 * 1024 * 1024
def upload(c, actor, chat, data, expected, content_type='image/png', filename='file.png'):
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=' + filename, expected,
actor=actor, raw=data, headers={'Content-Type': content_type},
schema='Attachment' if expected == 201 else None)
def post_message(c, actor, chat, ids, expected, text=''):
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', expected, actor=actor,
body={'text': text, 'attachment_ids': ids}, headers={'Idempotency-Key': uuid.uuid4().hex}, schema='Message' if expected == 201 else None)
def main():
args = arguments('Attachment validation and access checks')
c = Client(args.base_url, args.ca_file)
alice, bob, eve, chat = c.bootstrap()
other = c.call('POST', '/api/v1/chats', 201, actor=alice, body={'title': 'second', 'member_ids': [alice['id'], bob['id']]}, schema='Chat')['json']
eve_chat = c.call('POST', '/api/v1/chats', 201, actor=eve, body={'title': 'eve', 'member_ids': [eve['id'], bob['id']]}, schema='Chat')['json']
upload(c, alice, chat, b'x' * (MAX_BYTES + 1), 413)
upload(c, alice, chat, SAMPLE + b'\0' * MAX_BYTES, 413)
upload(c, alice, chat, png_header_only(8000, 8000), 413)
upload(c, alice, chat, SAMPLE[:len(SAMPLE) // 2], 415)
upload(c, alice, chat, SAMPLE[:-1], 415)
upload(c, alice, chat, b'\x89PNG\r\n\x1a\n' + b'garbage' * 100, 415)
upload(c, alice, chat, b'', 415)
upload(c, alice, chat, b'GIF89a' + b'\0' * 64, 415)
upload(c, alice, chat, SAMPLE, 415, content_type='image/jpeg')
upload(c, alice, chat, SAMPLE, 415, content_type='text/plain')
upload(c, alice, chat, SAMPLE, 415, content_type='application/octet-stream')
for name in ('..%2Fx.png', 'a%5Cb.png', 'a%00b.png', 'a%0Ab.png', 'a' * 256, ''):
upload(c, alice, chat, SAMPLE, 400, filename=name)
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments', 400, actor=alice, raw=SAMPLE, headers={'Content-Type': 'image/png'})
upload(c, eve, chat, SAMPLE, 404)
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=a.png', 401, raw=SAMPLE, headers={'Content-Type': 'image/png'})
ok = upload(c, alice, chat, SAMPLE, 201)['json']
check(ok['sha256'] == hashlib.sha256(SAMPLE).hexdigest(), 'checksum mismatch')
check(ok['filename'] == 'file.png' and ok['status'] == 'ready' and ok['variants'] == [], 'attachment fields')
unicode_name = upload(c, alice, chat, SAMPLE, 201, filename='%D1%84%D0%BE%D1%82%D0%BE.png')['json']
check(unicode_name['filename'] == 'фото.png', 'unicode filename changed')
post_message(c, alice, other, [ok['id']], 404)
post_message(c, alice, chat, [str(uuid.uuid4())], 404)
post_message(c, alice, chat, [ok['id'], ok['id']], 400)
post_message(c, alice, chat, ['not-a-uuid'], 400)
post_message(c, alice, chat, [str(uuid.uuid4()) for _ in range(11)], 400)
post_message(c, alice, chat, [], 400)
post_message(c, eve, chat, [ok['id']], 404)
post_message(c, eve, eve_chat, [ok['id']], 404)
shared = post_message(c, bob, chat, [ok['id'], unicode_name['id']], 201, text='from bob')['json']
check(shared['attachment_ids'] == [ok['id'], unicode_name['id']], 'attachment order changed')
history = c.call('GET', '/api/v1/chats/' + chat['id'] + '/messages?limit=100', actor=alice, schema='MessagePage')['json']['items']
check(any(m['id'] == shared['id'] and m['attachment_ids'] == shared['attachment_ids'] for m in history), 'history lost attachment_ids')
for suffix in ('', '/content'):
c.call('GET', '/api/v1/attachments/' + ok['id'] + suffix, 404, actor=eve)
c.call('GET', '/api/v1/attachments/' + str(uuid.uuid4()) + suffix, 404, actor=alice)
c.call('GET', '/api/v1/attachments/bad' + suffix, 400, actor=alice)
c.call('GET', '/api/v1/attachments/' + ok['id'] + suffix, 401)
c.call('GET', '/api/v1/attachments/' + ok['id'] + '/content?variant=thumbnail', 400, actor=bob)
download = c.call('GET', '/api/v1/attachments/' + ok['id'] + '/content', actor=bob)
check(download['bytes'] == SAMPLE, 'download differs')
big = png(1800, 1800, noise=True)
check(len(big) < MAX_BYTES, 'generated image exceeds limit')
large = upload(c, alice, chat, big, 201)['json']
check(c.call('GET', '/api/v1/attachments/' + large['id'] + '/content', actor=bob)['bytes'] == big, 'large download differs')
print('PASS: attachment validation, limits and access control')
if __name__ == '__main__':
run(main)
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env python3
from concurrent.futures import ThreadPoolExecutor
import threading
import uuid
from client import Client, Failure, arguments, check, run
def main():
args = arguments('Concurrent Idempotency-Key check through the load balancer')
c = Client(args.base_url, args.ca_file)
alice, bob, eve, chat = c.bootstrap()
path = '/api/v1/chats/' + chat['id'] + '/messages'
parallel = 32
key = 'idem-' + uuid.uuid4().hex
barrier = threading.Barrier(parallel)
def send(_):
local = Client(args.base_url, args.ca_file)
barrier.wait()
return local.call('POST', path, 201, actor=alice, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')
with ThreadPoolExecutor(max_workers=parallel) as pool:
results = list(pool.map(send, range(parallel)))
ids = {r['json']['id'] for r in results}
instances = {r['headers']['X-Instance-Id'] for r in results}
check(len(ids) == 1, 'parallel requests produced %d messages' % len(ids))
check(len(instances) >= 2, 'requests were served by one instance only')
check(len({r['json']['created_at'] for r in results}) == 1, 'created_at differs between replays')
listing = c.call('GET', path, actor=alice, schema='MessagePage')['json']['items']
check(sum(1 for m in listing if m['text'] == 'same') == 1, 'duplicate message in history')
c.call('POST', path, 409, actor=alice, body={'text': 'other'}, headers={'Idempotency-Key': key})
c.call('POST', path, 400, actor=alice, body={'text': 'x'})
c.call('POST', path, 400, actor=alice, body={'text': 'x'}, headers={'Idempotency-Key': 'bad key!'})
other = c.call('POST', path, 201, actor=bob, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')['json']
check(other['id'] not in ids, 'key must be scoped per user')
print('PASS: %d parallel requests, 1 message, %d instances' % (parallel, len(instances)))
if __name__ == '__main__':
run(main)
+23
View File
@@ -0,0 +1,23 @@
import os
import struct
import zlib
def chunk(kind, data):
body = kind + data
return struct.pack('>I', len(data)) + body + struct.pack('>I', zlib.crc32(body) & 0xffffffff)
def png(width, height, noise=False):
header = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
raw = bytearray()
row = width * 3 + 1
for _ in range(height):
raw += b'\x00' + (os.urandom(row - 1) if noise else bytes(row - 1))
data = zlib.compress(bytes(raw), 0 if noise else 6)
return header + chunk(b'IDAT', data) + chunk(b'IEND', b'')
def png_header_only(width, height):
header = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
return header + chunk(b'IDAT', zlib.compress(b'\x00')) + chunk(b'IEND', b'')
+119
View File
@@ -0,0 +1,119 @@
#!/usr/bin/env python3
import hashlib
import subprocess
import time
import uuid
from pathlib import Path
from client import Client, Failure, arguments, check, run
ROOT = Path(__file__).resolve().parent.parent
SAMPLE = (ROOT / 'tests/fixtures/sample.png').read_bytes()
BUCKET = 'messenger-attachments'
def compose(*args, check_rc=True, stdin=None):
return subprocess.run(['docker', 'compose', '--env-file', '.env', *args], cwd=ROOT, check=check_rc,
capture_output=True, text=True, input=stdin)
def secret(name):
return (ROOT / 'secrets' / name).read_text().strip()
def s3_status(method, path, user=None, data=None):
cmd = ['exec', '-T', 's3', 'curl', '-s', '-o', '/dev/null', '-w', '%{http_code}', '-X', method]
if user:
cmd += ['--aws-sigv4', 'aws:amz:us-east-1:s3', '--user', user[0] + ':' + user[1]]
if data is not None:
cmd += ['--data-binary', data]
cmd.append('http://127.0.0.1:9000' + path)
return compose(*cmd).stdout.strip()
def upload(c, actor, chat, expected):
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=a.png', expected, actor=actor,
raw=SAMPLE, headers={'Content-Type': 'image/png'}, schema='Attachment' if expected == 201 else None)
def wait_api(c):
deadline = time.monotonic() + 60
while time.monotonic() < deadline:
try:
c.call('GET', '/health/ready')
return
except Failure:
time.sleep(1)
raise Failure('api did not recover')
def wait_s3(timeout=90):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if compose('ps', 's3', '--format', '{{.Status}}').stdout.count('(healthy)'):
return
time.sleep(1)
raise Failure('s3 did not recover')
def object_key(attachment_id):
out = compose('exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-Atc',
"select object_key from attachments where id='%s'" % attachment_id).stdout.strip()
check(out.startswith('attachments/'), 'object key not found')
return out
def main():
args = arguments('S3 privacy, outage and orphan cleanup checks')
c = Client(args.base_url, args.ca_file, timeout=15)
alice, bob, eve, chat = c.bootstrap()
att = upload(c, alice, chat, 201)['json']
key = object_key(att['id'])
check(key != 'attachments/' + att['filename'] and att['filename'] not in key, 'object key derived from filename')
api = (secret('s3_access_key_id'), secret('s3_secret_access_key'))
admin = (secret('s3_admin_access_key_id'), secret('s3_admin_secret_access_key'))
path = '/' + BUCKET + '/' + key
check(s3_status('GET', path) == '403', 'anonymous GET must be denied')
check(s3_status('GET', '/' + BUCKET + '/') == '403', 'anonymous listing must be denied')
check(s3_status('PUT', path + '-anon', data='x') == '403', 'anonymous PUT must be denied')
check(s3_status('GET', path, ('wrong', 'wrong')) == '403', 'wrong credentials must be denied')
check(s3_status('GET', path, api) == '200', 'api credentials cannot read')
check(s3_status('PUT', '/other-bucket-' + uuid.uuid4().hex[:8], api) == '403', 'api credentials can create buckets')
ports = compose('ps', '--format', '{{.Service}} {{.Ports}}').stdout
published = [line.split()[0] for line in ports.splitlines() if '->' in line]
check(published == ['lb'], 'only lb may publish ports, found %s' % published)
compose('stop', 's3')
try:
upload(c, alice, chat, 503)
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', 503, actor=bob)
c.call('GET', '/api/v1/attachments/' + att['id'], 200, actor=bob, schema='Attachment')
c.message(alice, chat, 'text works while s3 is down')
finally:
compose('start', 's3')
wait_s3()
wait_api(c)
deadline = time.monotonic() + 60
while True:
try:
body = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes']
check(body == SAMPLE, 'content differs after s3 restart')
break
except Failure:
check(time.monotonic() < deadline, 'download did not recover')
time.sleep(1)
upload(c, alice, chat, 201)
orphan = '/' + BUCKET + '/attachments/' + uuid.uuid4().hex
check(s3_status('PUT', orphan, admin, 'orphan') == '200', 'cannot create orphan')
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=3600', 'janitor', 'cleanup')
check(s3_status('GET', orphan, admin) == '200', 'fresh object was removed inside grace period')
time.sleep(1)
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=0', 'janitor', 'cleanup')
check(s3_status('GET', orphan, admin) == '404', 'orphan was not removed')
check(c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes'] == SAMPLE, 'referenced object was removed')
print('PASS: private bucket, limited credentials, s3 outage, orphan cleanup')
if __name__ == '__main__':
run(main)
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
from concurrent.futures import ThreadPoolExecutor
import json
import subprocess
import threading
import time
import uuid
from pathlib import Path
from client import Client, Failure, arguments, check, run
ROOT = Path(__file__).resolve().parent.parent
def sh(*cmd):
return subprocess.run(cmd, cwd=ROOT, check=True, capture_output=True, text=True).stdout.strip()
def api_containers():
out = sh('docker', 'compose', '--env-file', '.env', 'ps', '-q', 'api')
return [x[:12] for x in out.split()]
def wait_healthy(count, timeout=90):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
out = sh('docker', 'compose', '--env-file', '.env', 'ps', 'api', '--format', '{{.Status}}')
if out.count('(healthy)') >= count:
return
time.sleep(1)
raise Failure('replicas did not become healthy')
class Load:
def __init__(self, base, ca, concurrency):
c = Client(base, ca)
self.alice, _, _, self.chat = c.bootstrap()
self.base, self.ca, self.n = base, ca, concurrency
self.events = []
self.lock = threading.Lock()
self.stop = threading.Event()
self.t0 = time.monotonic()
def worker(self):
c = Client(self.base, self.ca, timeout=3)
while not self.stop.is_set():
key = uuid.uuid4().hex
first_error = None
while not self.stop.is_set():
try:
r = c.message(self.alice, self.chat, 'k-' + key, key=key)
t = time.monotonic() - self.t0
with self.lock:
self.events.append((t, True, r['headers']['X-Instance-Id'], first_error))
break
except Failure:
t = time.monotonic() - self.t0
first_error = first_error or t
with self.lock:
self.events.append((t, False, None, None))
time.sleep(0.05)
def start(self):
self.pool = ThreadPoolExecutor(max_workers=self.n)
self.futures = [self.pool.submit(self.worker) for _ in range(self.n)]
def finish(self):
self.stop.set()
self.pool.shutdown(wait=True)
def summary(self, begin, end):
ev = [e for e in self.events if begin <= e[0] < end]
ok = [e for e in ev if e[1]]
bad = [e for e in ev if not e[1]]
span = max(end - begin, 1e-9)
return {
'window_seconds': round(span, 2),
'successful_requests': len(ok),
'failed_attempts': len(bad),
'successful_rps': round(len(ok) / span, 1),
'last_error_offset_seconds': round(max(e[0] for e in bad) - begin, 2) if bad else None,
'instances': sorted({e[2] for e in ok}),
}
def instances_between(self, begin, end=float('inf')):
return sorted({e[2] for e in self.events if e[1] and begin <= e[0] < end})
def failover_each(args):
ids = api_containers()
check(len(ids) >= 2, 'need at least two replicas')
load = Load(args.base_url, args.ca_file, args.concurrency)
load.start()
time.sleep(args.warmup)
results = []
base = load.summary(2, args.warmup)
for target in ids:
before = time.monotonic() - load.t0
sh('docker', 'stop', '-t', '10', target)
stopped = time.monotonic() - load.t0
time.sleep(args.hold)
after_stop = time.monotonic() - load.t0
window = load.summary(before, after_stop)
sh('docker', 'start', target)
wait_healthy(len(ids))
restored = time.monotonic() - load.t0
time.sleep(args.warmup)
results.append({
'stopped_replica': target,
'during_failure': window,
'seen_stopped_replica_after_stop': target in load.instances_between(stopped + 0.5, after_stop),
'stop_command_seconds': round(stopped - before, 2),
'restore_seconds': round(restored - after_stop, 2),
'after_restore': load.summary(restored + 1, restored + args.warmup),
})
load.finish()
return {'baseline': base, 'replicas': results}
def rolling(args):
old = api_containers()
check(len(old) >= 2, 'need at least two replicas')
n = len(old)
load = Load(args.base_url, args.ca_file, args.concurrency)
load.start()
time.sleep(args.warmup)
begin = time.monotonic() - load.t0
sh('scripts/rolling-update.sh')
end = time.monotonic() - load.t0
time.sleep(args.warmup)
load.finish()
return {
'old_replicas': old,
'new_replicas': api_containers(),
'update_window': load.summary(begin, end),
'old_replicas_served_after_update': [i for i in load.instances_between(end) if i in old],
}
def main():
def configure(p):
p.add_argument('--mode', choices=['failover-each', 'rolling'], required=True)
p.add_argument('--concurrency', type=int, default=8)
p.add_argument('--warmup', type=float, default=8)
p.add_argument('--hold', type=float, default=12)
p.add_argument('--output')
args = arguments('Failure and rolling update under load', configure)
report = failover_each(args) if args.mode == 'failover-each' else rolling(args)
text = json.dumps(report, indent=2)
if args.output:
Path(args.output).write_text(text + '\n')
print(text)
if __name__ == '__main__':
run(main)
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
from concurrent.futures import ThreadPoolExecutor
import hashlib
import json
import re
import subprocess
import threading
import time
from pathlib import Path
from client import Client, Failure, arguments, check, run
from image_gen import png
ROOT = Path(__file__).resolve().parent.parent
UNITS = {'B': 1, 'KiB': 1024, 'MiB': 1024 ** 2, 'GiB': 1024 ** 3}
def to_mib(text):
value, unit = re.match(r'([\d.]+)\s*([A-Za-z]+)', text).groups()
return float(value) * UNITS[unit] / 1024 ** 2
def sample(stop, peaks):
proc = subprocess.Popen(['docker', 'stats', '--format', '{{.Name}} {{.MemUsage}}'], cwd=ROOT,
stdout=subprocess.PIPE, text=True)
try:
for line in proc.stdout:
line = re.sub(r'\x1b\[[0-9;]*[A-Za-z]', '', line).strip()
m = re.match(r'(\S+-api-\d+)\s+(\S+ ?\S+) /', line)
if m:
peaks[m.group(1)] = max(peaks.get(m.group(1), 0), to_mib(m.group(2)))
if stop.is_set():
break
finally:
proc.terminate()
def main():
def configure(p):
p.add_argument('--concurrency', type=int, default=12)
p.add_argument('--rounds', type=int, default=3)
p.add_argument('--output')
args = arguments('Concurrent upload memory experiment', configure)
check(args.concurrency >= 10, 'use at least 10 concurrent uploads')
c = Client(args.base_url, args.ca_file, timeout=60)
alice, bob, eve, chat = c.bootstrap()
data = png(1800, 1800, noise=True)
digest = hashlib.sha256(data).hexdigest()
path = '/api/v1/chats/' + chat['id'] + '/attachments?filename=load.png'
idle, stop = {}, threading.Event()
t = threading.Thread(target=sample, args=(stop, idle))
t.start()
time.sleep(4)
stop.set()
t.join()
peaks, stop = {}, threading.Event()
t = threading.Thread(target=sample, args=(stop, peaks))
t.start()
time.sleep(2)
barrier = threading.Barrier(args.concurrency)
results = []
def worker(_):
local = Client(args.base_url, args.ca_file, timeout=60)
barrier.wait()
out = []
for _ in range(args.rounds):
try:
r = local.call('POST', path, 201, actor=alice, raw=data, headers={'Content-Type': 'image/png'}, schema='Attachment')
out.append(r['json']['sha256'] == digest)
except Failure:
out.append(False)
return out
started = time.monotonic()
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
for out in pool.map(worker, range(args.concurrency)):
results.extend(out)
elapsed = time.monotonic() - started
time.sleep(2)
stop.set()
t.join()
limit = 192
report = {
'file_bytes': len(data),
'concurrent_uploads': args.concurrency,
'rounds': args.rounds,
'uploads_ok': sum(results),
'uploads_failed': len(results) - sum(results),
'elapsed_seconds': round(elapsed, 2),
'api_memory_limit_mib': limit,
'idle_memory_mib': {k: round(v, 1) for k, v in sorted(idle.items())},
'peak_memory_mib': {k: round(v, 1) for k, v in sorted(peaks.items())},
'buffering_all_files_would_need_mib': round(args.concurrency * len(data) / 1024 ** 2, 1),
}
text = json.dumps(report, indent=2)
if args.output:
Path(args.output).write_text(text + '\n')
print(text)
check(report['uploads_failed'] == 0, 'some uploads failed')
check(max(peaks.values()) < limit, 'peak memory reached the limit')
if __name__ == '__main__':
run(main)