ЛР 3 — БВТ2304 — Березин — на 4 #35

Closed
yagera wants to merge 5 commits from yagera/mtusi2026lab3:solution/bvt2304-berezin into main
30 changed files with 2221 additions and 25 deletions
+1 -10
View File
@@ -1,13 +1,4 @@
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
# These are logical interface suggestions; map actual framework names in REPORT.md.
APP_HOST=0.0.0.0
APP_PORT=8080
APP_ENV=lab
SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
API_REPLICAS=2
+19
View File
@@ -0,0 +1,19 @@
FROM golang:1.26-alpine@sha256:c95332c2af86b6d89b91bd0500f4b9529ccbd090a0d1855c6d1ceaa142ae8615 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY *.go ./
COPY migrations ./migrations
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/messenger .
FROM alpine:3.21@sha256:ce64758a109eb420d874a118f87920e625e12d3634e03b4a5573fd9f6e5d3507
RUN adduser -D -u 10001 app
WORKDIR /app
COPY --from=build /out/messenger /app/messenger
USER 10001
EXPOSE 8080
ENV APP_HOST=0.0.0.0 APP_PORT=8080 APP_ENV=lab
STOPSIGNAL SIGTERM
HEALTHCHECK --interval=5s --timeout=3s --start-period=10s --retries=3 \
CMD wget -qO- http://127.0.0.1:8080/health/ready >/dev/null || exit 1
ENTRYPOINT ["/app/messenger"]
+71 -15
View File
@@ -1,37 +1,93 @@
# Отчёт — лабораторная 3
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Березин Герман / БВТ2304
- Целевая оценка: 4
- Commit/PR: после публикации
- Использованные источники, библиотеки и помощь ИИ: Go 1.26, pgx, go-redis, HAProxy 3.0, PostgreSQL 17, Valkey 8.1; код lab2 перенесён за основу; ИИ-ассистент использован при разработке
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Требуются Docker 29 с Compose v2 и Python 3.9+.
```sh
sh scripts/mksecrets.sh
cp .env.example .env
docker compose --env-file .env up -d --build --wait
make test
```
Порядок старта: postgres и valkey → одноразовый сервис `migrate` → `api` (2 реплики) → `lb`. Публичный адрес: `http://localhost:8080`.
Остановка без потери данных: `docker compose --env-file .env stop`. Намеренное удаление данных: `docker compose --env-file .env down -v`.
Проверки:
```sh
make test-persistence ACTION_SCRIPT=scripts/restart.sh
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/idempotency.py
python3 tests/scenario.py --mode failover-each --output evidence/failover-each.json
python3 tests/scenario.py --mode rolling --output evidence/rolling.json
```
После `make test-failover` остановленная реплика восстанавливается вручную: `docker compose --env-file .env up -d --wait`.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab2, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Клиент → HAProxy (`lb`, единственный опубликованный порт, 127.0.0.1:8080) → `api` × 2 → PostgreSQL и Valkey.
- Источник истины — PostgreSQL: пользователи, чаты, сообщения, сессии, ключи идемпотентности. Valkey используется только как общий кэш сессий. Состояния в памяти API нет, sticky session не используется.
- `X-Instance-Id` — hostname контейнера (12 символов id), по нему `stop-one.sh` находит контейнер.
- Миграции вынесены из API в сервис `migrate` (`messenger migrate`), который выполняется один раз до запуска `api` (`service_completed_successfully`), под `pg_advisory_lock`. Миграция 0002 только добавляет таблицу `idempotency_keys`, поэтому совместима с работающей предыдущей версией.
- Идемпотентность: таблица `idempotency_keys` с первичным ключом (user_id, method, path, key). Вставка ключа и сообщения выполняется в одной транзакции; `insert ... on conflict do nothing` заставляет конкурентный запрос ждать коммита первого. При повторе сверяется SHA-256 текста: совпал — возвращается сохранённый ответ 201, не совпал — 409. Ключи старше `IDEMPOTENCY_TTL_SECONDS` (24 ч) заменяются при следующем использовании. Сессия и членство в чате проверяются заново при каждом повторе.
- Переменные `.env.example` соответствуют конфигурации: `APP_ENV`, `SESSION_TTL_SECONDS`, `IDEMPOTENCY_TTL_SECONDS`, `API_REPLICAS`.
### Проверки готовности, таймауты, остановка
| Параметр | Значение |
| --- | --- |
| Периодичность health-check HAProxy (`GET /health/ready`) | inter 1 s, fastinter/downinter 0.5/1 s |
| Порог отказа / восстановления | fall 2 / rise 2 |
| Максимальное время обнаружения отказа | около 2–3 s (fall 2 × inter 1 s + timeout) |
| Повтор при ошибке соединения | `retries 3`, `option redispatch`, `retry-on conn-failure empty-response` |
| Таймауты HAProxy | connect 1 s, server 15 s, client 20 s |
| Таймауты API | ReadHeader 5 s, Read 10 s, Write 15 s, Idle 30 s; запросы к БД 3 s |
| SIGTERM | `/health/ready` сразу отвечает 503, затем пауза `DRAIN_SECONDS=3`, затем `Shutdown` до 10 s; `stop_grace_period: 15s` |
| Обновление адресов backend | `server-template api 1-8` + резолвер Docker DNS (127.0.0.11), `hold valid 1s` |
Лимиты: api, valkey, migrate, lb — 0.5 CPU / 128 MiB; postgres — 1 CPU / 512 MiB. Порты API наружу не публикуются, `container_name` не задан, число реплик меняется `API_REPLICAS` или `--scale`.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Публичный smoke | `make test` | PASS | evidence/smoke.txt |
| Сохранность после restart | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS, прежний токен действителен; перезапускаются postgres, valkey, api, lb | evidence/persistence.txt |
| Отказ одной реплики | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | PASS, 2 реплики до отказа, 0 ошибок, восстановление записи 3.4 s; повтор для второй реплики — тоже PASS | evidence/failover-1.txt, evidence/failover-2.txt |
| Идемпотентность, параллельно | `python3 tests/idempotency.py` | 32 параллельных запроса с одним ключом через 2 реплики → 1 сообщение; другой body → 409; без ключа → 400 | evidence/idempotency.txt |
| Последовательная остановка каждой реплики | `tests/scenario.py --mode failover-each` | RPS, ошибки и восстановление по каждой реплике | evidence/failover-each.json |
| Лимиты ресурсов | `docker stats` | api ≤ 128 MiB, postgres ≤ 512 MiB | evidence/limits.txt |
| Rolling update под нагрузкой | `scripts/rolling-update.sh` через `tests/scenario.py --mode rolling` | обе старые реплики заменены новыми, ответы старых прекратились | evidence/rolling.json |
| Миграции однократно | сервис `migrate` | запускается один раз, выход 0, API стартует после него | `docker compose logs migrate` |
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Стенд: один хост, 8 клиентских потоков, closed-loop POST сообщений с уникальным ключом; при ошибке запрос повторяется с тем же ключом. Лимиты ресурсов как выше. Базовая нагрузка около 1000 успешных RPS.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
| `docker stop` реплики 1 (под нагрузкой) | единицы ошибок, затем работа на второй реплике | 0–3 неуспешные попытки за 15 s, последняя ошибка в первые 0.7 s; остановленная реплика после остановки не отвечает; RPS ≈ 970 | запись не прерывалась дольше 1 s; подтверждённые сообщения целы |
| то же для реплики 2 | то же | 0–3 ошибки, RPS ≈ 960 | то же; возврат реплики в ротацию около 6 s после старта |
| Rolling update (4 контейнера, замена старых) | почти без ошибок | 1–4 неуспешные попытки за ≈ 13.5 s, RPS ≈ 975 | старые реплики перестают отвечать; дубликатов нет (повтор с тем же ключом) |
| Restart postgres/valkey/api/lb | данные и сессия сохраняются | PASS persistence | прежний токен действителен |
Точные значения лежат в `evidence/*.json`; от запуска к запуску число ошибок плавает (0–4), поэтому приведены диапазоны.
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
- Выдерживается отказ одной API-реплики. HAProxy, PostgreSQL и Valkey — единые точки отказа, репликации БД нет и выживание её отказа не заявляется.
- Ошибки во время отказа возможны на запросах, уже принятых остановленной репликой; клиент повторяет их с тем же ключом.
- Нагрузочный генератор на Python (closed-loop) ограничен клиентом и подвержен coordinated omission; RPS — ориентир, а не предел системы.
- HAProxy выполняет только проверки готовности по HTTP и не знает о версии приложения; rolling update построен скриптом (scale ×2 → остановка старых → возврат к исходному числу), а не оркестратором.
- Миграции совместимы только вперёд-аддитивно (новые таблицы); несовместимые изменения схемы потребуют схемы expand/contract.
- Секреты лежат в `./secrets` вне Git и доступны на чтение всем локально, чтобы их читал процесс с UID 10001; это допустимо только для локального стенда.
+131
View File
@@ -0,0 +1,131 @@
package main
import (
"crypto/rand"
"crypto/sha256"
"crypto/subtle"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"strconv"
"strings"
"unicode/utf8"
"golang.org/x/crypto/argon2"
)
const (
argonTime = 2
argonMemory = 19456
argonThreads = 1
argonKeyLen = 32
argonSaltLen = 16
)
func hashPassword(password string) (string, error) {
salt := make([]byte, argonSaltLen)
if _, err := rand.Read(salt); err != nil {
return "", err
}
hash := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
b64Salt := base64.RawStdEncoding.EncodeToString(salt)
b64Hash := base64.RawStdEncoding.EncodeToString(hash)
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
argon2.Version, argonMemory, argonTime, argonThreads, b64Salt, b64Hash), nil
}
func checkPassword(encoded, password string) (bool, error) {
parts := strings.Split(encoded, "$")
if len(parts) != 6 || parts[1] != "argon2id" {
return false, errors.New("bad hash")
}
var version int
if _, err := fmt.Sscanf(parts[2], "v=%d", &version); err != nil {
return false, err
}
var memory, timeCost, threads int
for _, p := range strings.Split(parts[3], ",") {
kv := strings.SplitN(p, "=", 2)
if len(kv) != 2 {
return false, errors.New("bad params")
}
n, err := strconv.Atoi(kv[1])
if err != nil {
return false, err
}
switch kv[0] {
case "m":
memory = n
case "t":
timeCost = n
case "p":
threads = n
}
}
salt, err := base64.RawStdEncoding.DecodeString(parts[4])
if err != nil {
return false, err
}
want, err := base64.RawStdEncoding.DecodeString(parts[5])
if err != nil {
return false, err
}
got := argon2.IDKey([]byte(password), salt, uint32(timeCost), uint32(memory), uint8(threads), uint32(len(want)))
return subtle.ConstantTimeCompare(got, want) == 1, nil
}
func newToken() (string, []byte, string, error) {
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
return "", nil, "", err
}
token := base64.RawURLEncoding.EncodeToString(raw)
sum := sha256.Sum256([]byte(token))
hash := make([]byte, len(sum))
copy(hash, sum[:])
return token, hash, hex.EncodeToString(hash), nil
}
func tokenHash(token string) ([]byte, string) {
sum := sha256.Sum256([]byte(token))
hash := make([]byte, len(sum))
copy(hash, sum[:])
return hash, hex.EncodeToString(hash)
}
func parseBasic(header string) (username, password string, ok bool) {
const prefix = "Basic "
if !strings.HasPrefix(header, prefix) {
return "", "", false
}
decoded, err := base64.StdEncoding.DecodeString(strings.TrimSpace(header[len(prefix):]))
if err != nil {
return "", "", false
}
s := string(decoded)
i := strings.IndexByte(s, ':')
if i < 0 {
return "", "", false
}
username = s[:i]
password = s[i+1:]
return username, password, true
}
func parseBearer(header string) (string, bool) {
const prefix = "Bearer "
if !strings.HasPrefix(header, prefix) {
return "", false
}
tok := strings.TrimSpace(header[len(prefix):])
if tok == "" {
return "", false
}
return tok, true
}
func validPassword(p string) bool {
n := utf8.RuneCountInString(p)
return n >= 12 && n <= 128
}
+104
View File
@@ -0,0 +1,104 @@
package main
import (
"context"
"encoding/json"
"log/slog"
"time"
"github.com/redis/go-redis/v9"
)
type cachedSession struct {
UserID string `json:"user_id"`
ExpiresAt time.Time `json:"expires_at"`
}
type Cache struct {
rdb *redis.Client
log *slog.Logger
}
func openCache(log *slog.Logger) *Cache {
url, err := readSecret("CACHE_URL", "CACHE_URL_FILE")
if err != nil {
log.Warn("cache disabled", "err", err.Error())
return &Cache{log: log}
}
opt, err := redis.ParseURL(url)
if err != nil {
log.Warn("cache parse failed", "err", err.Error())
return &Cache{log: log}
}
opt.DialTimeout = 200 * time.Millisecond
opt.ReadTimeout = 200 * time.Millisecond
opt.WriteTimeout = 200 * time.Millisecond
opt.PoolTimeout = 200 * time.Millisecond
rdb := redis.NewClient(opt)
ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
defer cancel()
if err := rdb.Ping(ctx).Err(); err != nil {
log.Warn("cache ping failed, will bypass", "err", err.Error())
}
return &Cache{rdb: rdb, log: log}
}
func (c *Cache) Close() {
if c.rdb != nil {
_ = c.rdb.Close()
}
}
func sessKey(hashHex string) string {
return "sess:" + hashHex
}
func (c *Cache) GetSession(ctx context.Context, hashHex string) (*cachedSession, bool) {
if c.rdb == nil {
return nil, false
}
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
val, err := c.rdb.Get(ctx, sessKey(hashHex)).Result()
if err != nil {
return nil, false
}
var s cachedSession
if err := json.Unmarshal([]byte(val), &s); err != nil {
return nil, false
}
if time.Now().UTC().After(s.ExpiresAt) {
return nil, false
}
return &s, true
}
func (c *Cache) SetSession(ctx context.Context, hashHex string, userID string, expiresAt time.Time) {
if c.rdb == nil {
return
}
ttl := time.Until(expiresAt)
if ttl <= 0 {
return
}
body, err := json.Marshal(cachedSession{UserID: userID, ExpiresAt: expiresAt.UTC()})
if err != nil {
return
}
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
if err := c.rdb.Set(ctx, sessKey(hashHex), body, ttl).Err(); err != nil {
c.log.Warn("cache set failed", "err", err.Error())
}
}
func (c *Cache) DelSession(ctx context.Context, hashHex string) {
if c.rdb == nil {
return
}
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
if err := c.rdb.Del(ctx, sessKey(hashHex)).Err(); err != nil {
c.log.Warn("cache del failed", "err", err.Error())
}
}
+132
View File
@@ -0,0 +1,132 @@
x-api-env: &api-env
APP_HOST: 0.0.0.0
APP_PORT: 8080
APP_ENV: ${APP_ENV:-lab}
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
SESSION_STORE: postgres
SHUTDOWN_TIMEOUT_SECONDS: 10
DRAIN_SECONDS: 3
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
services:
postgres:
image: postgres:17-alpine@sha256:b0f9560a2de083e2cc7382e75f808c7381a32852a7ec49117deedb300e552b24
environment:
POSTGRES_USER: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
POSTGRES_DB: messenger
secrets:
- postgres_password
volumes:
- pg_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
interval: 5s
timeout: 3s
retries: 10
start_period: 5s
deploy:
resources:
limits:
cpus: "1.0"
memory: 512M
networks:
- backend
restart: unless-stopped
valkey:
image: valkey/valkey:8.1-alpine@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
command: ["valkey-server", "--appendonly", "yes", "--save", "60", "1"]
volumes:
- valkey_data:/data
healthcheck:
test: ["CMD", "valkey-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
start_period: 5s
deploy:
resources:
limits:
cpus: "0.5"
memory: 128M
networks:
- backend
restart: unless-stopped
migrate:
build: .
image: mtusi-messenger:lab3
command: ["migrate"]
environment:
DATABASE_URL_FILE: /run/secrets/database_url
secrets:
- database_url
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "0.5"
memory: 128M
networks:
- backend
restart: "no"
api:
image: mtusi-messenger:lab3
environment: *api-env
secrets:
- database_url
- cache_url
depends_on:
migrate:
condition: service_completed_successfully
valkey:
condition: service_healthy
stop_grace_period: 15s
deploy:
replicas: ${API_REPLICAS:-2}
resources:
limits:
cpus: "0.5"
memory: 128M
networks:
- backend
restart: unless-stopped
lb:
image: haproxy:3.0-alpine@sha256:cf3af0facf7eb22b9fee463daf0fbc07ead740b8150696dce724264faf7b4318
ports:
- "127.0.0.1:8080:8080"
volumes:
- ./lb/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
depends_on:
- api
deploy:
resources:
limits:
cpus: "0.5"
memory: 128M
networks:
- backend
restart: unless-stopped
secrets:
postgres_password:
file: ./secrets/postgres_password
database_url:
file: ./secrets/database_url
cache_url:
file: ./secrets/cache_url
volumes:
pg_data:
valkey_data:
networks:
backend:
driver: bridge
+57
View File
@@ -0,0 +1,57 @@
package main
import (
"context"
"fmt"
"os"
"strings"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
func readSecret(envKey, fileKey string) (string, error) {
if v := os.Getenv(envKey); v != "" {
return strings.TrimSpace(v), nil
}
path := os.Getenv(fileKey)
if path == "" {
return "", fmt.Errorf("%s or %s required", envKey, fileKey)
}
b, err := os.ReadFile(path)
if err != nil {
return "", err
}
return strings.TrimSpace(string(b)), nil
}
func openDB(ctx context.Context) (*pgxpool.Pool, error) {
dsn, err := readSecret("DATABASE_URL", "DATABASE_URL_FILE")
if err != nil {
return nil, err
}
cfg, err := pgxpool.ParseConfig(dsn)
if err != nil {
return nil, err
}
cfg.MaxConns = 10
cfg.MinConns = 2
cfg.MaxConnLifetime = 5 * time.Minute
cfg.ConnConfig.ConnectTimeout = 3 * time.Second
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
return nil, err
}
pingCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
if err := pool.Ping(pingCtx); err != nil {
pool.Close()
return nil, err
}
return pool, nil
}
func dbCtx(parent context.Context) (context.Context, context.CancelFunc) {
return context.WithTimeout(parent, 3*time.Second)
}
+2
View File
@@ -0,0 +1,2 @@
{"observed_instances_before": 2, "successful_writes": 145, "failed_attempts": 0, "first_recovery_seconds": 3.416}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+5
View File
@@ -0,0 +1,5 @@
{"observed_instances_before": 2, "successful_writes": 145, "failed_attempts": 0, "first_recovery_seconds": 3.415}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
mtusi2026lab3-api-5 Up About a minute (healthy)
mtusi2026lab3-api-6 Exited (0) 11 seconds ago
mtusi2026lab2-api-1 Exited (128) 2 days ago
+71
View File
@@ -0,0 +1,71 @@
{
"baseline": {
"window_seconds": 6,
"successful_requests": 5971,
"failed_attempts": 0,
"successful_rps": 995.2,
"last_error_offset_seconds": null,
"instances": [
"10d7197fe55d",
"f6ed4d4ee4a9"
]
},
"replicas": [
{
"stopped_replica": "10d7197fe55d",
"during_failure": {
"window_seconds": 15.29,
"successful_requests": 14488,
"failed_attempts": 0,
"successful_rps": 947.6,
"last_error_offset_seconds": null,
"instances": [
"10d7197fe55d",
"f6ed4d4ee4a9"
]
},
"seen_stopped_replica_after_stop": false,
"stop_command_seconds": 3.29,
"restore_seconds": 5.74,
"after_restore": {
"window_seconds": 7.0,
"successful_requests": 6921,
"failed_attempts": 0,
"successful_rps": 988.7,
"last_error_offset_seconds": null,
"instances": [
"10d7197fe55d",
"f6ed4d4ee4a9"
]
}
},
{
"stopped_replica": "f6ed4d4ee4a9",
"during_failure": {
"window_seconds": 15.31,
"successful_requests": 14713,
"failed_attempts": 0,
"successful_rps": 961.2,
"last_error_offset_seconds": null,
"instances": [
"10d7197fe55d",
"f6ed4d4ee4a9"
]
},
"seen_stopped_replica_after_stop": false,
"stop_command_seconds": 3.31,
"restore_seconds": 5.72,
"after_restore": {
"window_seconds": 7.0,
"successful_requests": 6900,
"failed_attempts": 0,
"successful_rps": 985.7,
"last_error_offset_seconds": null,
"instances": [
"10d7197fe55d",
"f6ed4d4ee4a9"
]
}
}
]
}
+1
View File
@@ -0,0 +1 @@
PASS: 32 parallel requests, 1 message, 2 instances
+5
View File
@@ -0,0 +1,5 @@
mtusi2026lab3-api-5 65.68MiB / 128MiB 0.07%
mtusi2026lab3-api-6 3.09MiB / 128MiB 0.08%
mtusi2026lab3-lb-1 19.93MiB / 128MiB 0.09%
mtusi2026lab3-valkey-1 4.043MiB / 128MiB 0.21%
mtusi2026lab3-postgres-1 179.9MiB / 512MiB 0.15%
+2
View File
@@ -0,0 +1,2 @@
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+24
View File
@@ -0,0 +1,24 @@
{
"old_replicas": [
"226c5e9378fa",
"abdb789868c1"
],
"new_replicas": [
"e3aafe78ed80",
"400d26a30e0c"
],
"update_window": {
"window_seconds": 13.7,
"successful_requests": 13478,
"failed_attempts": 4,
"successful_rps": 983.8,
"last_error_offset_seconds": 11.23,
"instances": [
"226c5e9378fa",
"400d26a30e0c",
"abdb789868c1",
"e3aafe78ed80"
]
},
"old_replicas_served_after_update": []
}
+2
View File
@@ -0,0 +1,2 @@
python3 tests/smoke.py --base-url "http://localhost:8080"
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
+20
View File
@@ -0,0 +1,20 @@
module mtusi-messenger
go 1.26.0
require (
github.com/jackc/pgx/v5 v5.11.0
github.com/redis/go-redis/v9 v9.23.0
golang.org/x/crypto v0.58.0
)
require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
go.uber.org/atomic v1.12.0 // indirect
golang.org/x/sync v0.24.0 // indirect
golang.org/x/sys v0.49.0 // indirect
golang.org/x/text v0.43.0 // indirect
)
+41
View File
@@ -0,0 +1,41 @@
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/redis/go-redis/v9 v9.23.0 h1:/TTVdJa8BRNsybwwGGnmiYktHNlUXtbOQxlWpHBPL3s=
github.com/redis/go-redis/v9 v9.23.0/go.mod h1:EWP2UUk+XDaI1s5nQB5mablZN6vccXlmBQTyfnF87D0=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE=
go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+998
View File
@@ -0,0 +1,998 @@
package main
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"os"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"unicode/utf8"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgxpool"
)
var idempotencyKeyRe = regexp.MustCompile(`^[A-Za-z0-9._:-]{1,128}$`)
var instanceID = func() string {
h, err := os.Hostname()
if err != nil || h == "" {
return newUUID()
}
return h
}()
var usernameRe = regexp.MustCompile(`^[a-z0-9_]{3,32}$`)
type apiError struct {
Code string `json:"code"`
Message string `json:"message"`
ReqID string `json:"request_id"`
}
type errorBody struct {
Error apiError `json:"error"`
}
type App struct {
db *pgxpool.Pool
cache *Cache
log *slog.Logger
ready atomic.Bool
sessionTTL time.Duration
idempotencyTTL time.Duration
}
func NewApp(db *pgxpool.Pool, cache *Cache, log *slog.Logger, sessionTTL, idempotencyTTL time.Duration) *App {
a := &App{db: db, cache: cache, log: log, sessionTTL: sessionTTL, idempotencyTTL: idempotencyTTL}
a.ready.Store(true)
return a
}
func (a *App) SetReady(v bool) {
a.ready.Store(v)
}
func newUUID() string {
var b [16]byte
_, _ = rand.Read(b[:])
b[6] = (b[6] & 0x0f) | 0x40
b[8] = (b[8] & 0x3f) | 0x80
return fmt.Sprintf("%x-%x-%x-%x-%x", b[0:4], b[4:6], b[6:8], b[8:10], b[10:16])
}
func nowUTC() time.Time {
return time.Now().UTC().Truncate(time.Microsecond)
}
func formatTime(t time.Time) string {
return t.UTC().Format("2006-01-02T15:04:05.000000Z")
}
func parseTime(s string) (time.Time, error) {
return time.Parse("2006-01-02T15:04:05.000000Z", s)
}
type statusWriter struct {
http.ResponseWriter
status int
}
func (w *statusWriter) WriteHeader(code int) {
w.status = code
w.ResponseWriter.WriteHeader(code)
}
type ctxKey int
const reqIDKey ctxKey = 1
func (a *App) Middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
reqID := newUUID()
w.Header().Set("X-Request-Id", reqID)
w.Header().Set("X-Instance-Id", instanceID)
sw := &statusWriter{ResponseWriter: w, status: 200}
defer func() {
if rec := recover(); rec != nil {
a.writeErr(sw, reqID, 500, "internal_error", "internal error")
a.log.Error("panic", "request_id", reqID, "err", fmt.Sprint(rec))
}
a.log.Info("request",
"method", r.Method,
"path", r.URL.Path,
"status", sw.status,
"duration_ms", time.Since(start).Milliseconds(),
"request_id", reqID,
)
}()
next.ServeHTTP(sw, r.WithContext(context.WithValue(r.Context(), reqIDKey, reqID)))
})
}
func reqIDFrom(r *http.Request) string {
if v, ok := r.Context().Value(reqIDKey).(string); ok {
return v
}
return ""
}
func (a *App) writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
enc := json.NewEncoder(w)
enc.SetEscapeHTML(false)
_ = enc.Encode(v)
}
func (a *App) writeErr(w http.ResponseWriter, reqID string, status int, code, msg string) {
a.writeJSON(w, status, errorBody{Error: apiError{Code: code, Message: msg, ReqID: reqID}})
}
func (a *App) writeUnauthorized(w http.ResponseWriter, r *http.Request, realm string) {
reqID := reqIDFrom(r)
if realm == "Basic" {
w.Header().Set("WWW-Authenticate", `Basic realm="messenger", charset="UTF-8"`)
} else {
w.Header().Set("WWW-Authenticate", `Bearer realm="messenger"`)
}
a.writeErr(w, reqID, 401, "unauthorized", "unauthorized")
}
func (a *App) Routes() http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("GET /health/live", a.handleLive)
mux.HandleFunc("GET /health/ready", a.handleReady)
mux.HandleFunc("POST /api/v1/users", a.handleCreateUser)
mux.HandleFunc("GET /api/v1/users/me", a.handleMe)
mux.HandleFunc("POST /api/v1/chats", a.handleCreateChat)
mux.HandleFunc("GET /api/v1/chats", a.handleListChats)
mux.HandleFunc("GET /api/v1/chats/{chat_id}", a.handleGetChat)
mux.HandleFunc("POST /api/v1/chats/{chat_id}/messages", a.handleCreateMessage)
mux.HandleFunc("GET /api/v1/chats/{chat_id}/messages", a.handleListMessages)
mux.HandleFunc("POST /api/v1/auth/sessions", a.handleCreateSession)
mux.HandleFunc("DELETE /api/v1/auth/sessions/current", a.handleRevokeSession)
return a.Middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, pattern := mux.Handler(r)
if pattern == "" {
a.writeErr(w, reqIDFrom(r), 404, "not_found", "not found")
return
}
mux.ServeHTTP(w, r)
}))
}
func (a *App) handleLive(w http.ResponseWriter, r *http.Request) {
a.writeJSON(w, 200, map[string]string{"status": "ok"})
}
func (a *App) handleReady(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
if !a.ready.Load() {
a.writeErr(w, reqID, 503, "unavailable", "shutting down")
return
}
ctx, cancel := dbCtx(r.Context())
defer cancel()
if err := a.db.Ping(ctx); err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.writeJSON(w, 200, map[string]string{"status": "ok"})
}
func (a *App) readJSON(w http.ResponseWriter, r *http.Request, dst any) bool {
reqID := reqIDFrom(r)
ct := r.Header.Get("Content-Type")
media := strings.ToLower(strings.TrimSpace(strings.Split(ct, ";")[0]))
if media != "application/json" {
a.writeErr(w, reqID, 415, "unsupported_media_type", "Content-Type must be application/json")
return false
}
r.Body = http.MaxBytesReader(w, r.Body, 1<<20)
dec := json.NewDecoder(r.Body)
dec.DisallowUnknownFields()
if err := dec.Decode(dst); err != nil {
if _, ok := err.(*http.MaxBytesError); ok {
a.writeErr(w, reqID, 413, "payload_too_large", "body too large")
return false
}
a.writeErr(w, reqID, 400, "invalid_request", "invalid json")
return false
}
var extra json.RawMessage
if err := dec.Decode(&extra); err != io.EOF {
a.writeErr(w, reqID, 400, "invalid_request", "invalid json")
return false
}
return true
}
func isBlank(s string) bool {
return strings.TrimSpace(s) == ""
}
func isUUID(s string) bool {
if len(s) != 36 {
return false
}
for i, c := range s {
switch i {
case 8, 13, 18, 23:
if c != '-' {
return false
}
default:
if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F')) {
return false
}
}
}
raw := strings.ReplaceAll(s, "-", "")
_, err := hex.DecodeString(raw)
return err == nil
}
func isPgCode(err error, code string) bool {
var pgErr *pgconn.PgError
return errors.As(err, &pgErr) && pgErr.Code == code
}
func userJSON(u *User) map[string]any {
return map[string]any{
"id": u.ID,
"username": u.Username,
"display_name": u.DisplayName,
"created_at": formatTime(u.CreatedAt),
}
}
func (a *App) getUserByID(ctx context.Context, id string) (*User, error) {
ctx, cancel := dbCtx(ctx)
defer cancel()
var u User
err := a.db.QueryRow(ctx, `
select id::text, username, display_name, password_hash, created_at
from users where id = $1::uuid`, id).Scan(
&u.ID, &u.Username, &u.DisplayName, &u.PasswordHash, &u.CreatedAt)
if errors.Is(err, pgx.ErrNoRows) {
return nil, nil
}
if err != nil {
return nil, err
}
return &u, nil
}
func (a *App) authUser(w http.ResponseWriter, r *http.Request) (*User, string) {
token, ok := parseBearer(r.Header.Get("Authorization"))
if !ok {
a.writeUnauthorized(w, r, "Bearer")
return nil, ""
}
hash, hashHex := tokenHash(token)
if cs, hit := a.cache.GetSession(r.Context(), hashHex); hit {
u, err := a.getUserByID(r.Context(), cs.UserID)
if err != nil {
a.writeErr(w, reqIDFrom(r), 503, "unavailable", "database unavailable")
return nil, ""
}
if u == nil {
a.writeUnauthorized(w, r, "Bearer")
return nil, ""
}
return u, hashHex
}
ctx, cancel := dbCtx(r.Context())
defer cancel()
var userID string
var expiresAt time.Time
err := a.db.QueryRow(ctx, `
select user_id::text, expires_at from sessions
where token_hash = $1 and expires_at > now()`, hash).Scan(&userID, &expiresAt)
if errors.Is(err, pgx.ErrNoRows) {
a.writeUnauthorized(w, r, "Bearer")
return nil, ""
}
if err != nil {
a.writeErr(w, reqIDFrom(r), 503, "unavailable", "database unavailable")
return nil, ""
}
u, err := a.getUserByID(r.Context(), userID)
if err != nil {
a.writeErr(w, reqIDFrom(r), 503, "unavailable", "database unavailable")
return nil, ""
}
if u == nil {
a.writeUnauthorized(w, r, "Bearer")
return nil, ""
}
a.cache.SetSession(r.Context(), hashHex, userID, expiresAt)
return u, hashHex
}
func (a *App) handleCreateUser(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
var body struct {
Username *string `json:"username"`
DisplayName *string `json:"display_name"`
Password *string `json:"password"`
}
if !a.readJSON(w, r, &body) {
return
}
if body.Username == nil || body.DisplayName == nil || body.Password == nil {
a.writeErr(w, reqID, 400, "invalid_request", "username, display_name and password required")
return
}
username := *body.Username
display := *body.DisplayName
password := *body.Password
if !usernameRe.MatchString(username) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid username")
return
}
n := utf8.RuneCountInString(display)
if n < 1 || n > 100 || isBlank(display) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid display_name")
return
}
if !validPassword(password) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid password")
return
}
ph, err := hashPassword(password)
if err != nil {
a.writeErr(w, reqID, 500, "internal_error", "internal error")
return
}
id := newUUID()
createdAt := nowUTC()
ctx, cancel := dbCtx(r.Context())
defer cancel()
_, err = a.db.Exec(ctx, `
insert into users(id, username, display_name, password_hash, created_at)
values($1::uuid, $2, $3, $4, $5)`, id, username, display, ph, createdAt)
if isPgCode(err, "23505") {
a.writeErr(w, reqID, 409, "conflict", "username taken")
return
}
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.writeJSON(w, 201, map[string]any{
"id": id,
"username": username,
"display_name": display,
"created_at": formatTime(createdAt),
})
}
func (a *App) handleCreateSession(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
username, password, ok := parseBasic(r.Header.Get("Authorization"))
if !ok {
a.writeUnauthorized(w, r, "Basic")
return
}
if !usernameRe.MatchString(username) || !validPassword(password) {
a.writeUnauthorized(w, r, "Basic")
return
}
ctx, cancel := dbCtx(r.Context())
defer cancel()
var u User
err := a.db.QueryRow(ctx, `
select id::text, username, display_name, password_hash, created_at
from users where username = $1`, username).Scan(
&u.ID, &u.Username, &u.DisplayName, &u.PasswordHash, &u.CreatedAt)
if errors.Is(err, pgx.ErrNoRows) {
a.writeUnauthorized(w, r, "Basic")
return
}
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
match, err := checkPassword(u.PasswordHash, password)
if err != nil || !match {
a.writeUnauthorized(w, r, "Basic")
return
}
token, hash, hashHex, err := newToken()
if err != nil {
a.writeErr(w, reqID, 500, "internal_error", "internal error")
return
}
createdAt := nowUTC()
expiresAt := createdAt.Add(a.sessionTTL)
_, err = a.db.Exec(ctx, `
insert into sessions(token_hash, user_id, created_at, expires_at)
values($1, $2::uuid, $3, $4)`, hash, u.ID, createdAt, expiresAt)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.cache.SetSession(r.Context(), hashHex, u.ID, expiresAt)
a.writeJSON(w, 201, map[string]any{
"token": token,
"token_type": "Bearer",
"expires_at": formatTime(expiresAt),
"user": userJSON(&u),
})
}
func (a *App) handleRevokeSession(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
token, ok := parseBearer(r.Header.Get("Authorization"))
if !ok {
a.writeUnauthorized(w, r, "Bearer")
return
}
hash, hashHex := tokenHash(token)
ctx, cancel := dbCtx(r.Context())
defer cancel()
tag, err := a.db.Exec(ctx, `
delete from sessions where token_hash = $1 and expires_at > now()`, hash)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.cache.DelSession(r.Context(), hashHex)
if tag.RowsAffected() == 0 {
a.writeUnauthorized(w, r, "Bearer")
return
}
w.Header().Set("X-Request-Id", reqID)
w.WriteHeader(204)
}
func (a *App) handleMe(w http.ResponseWriter, r *http.Request) {
u, _ := a.authUser(w, r)
if u == nil {
return
}
a.writeJSON(w, 200, userJSON(u))
}
func (a *App) handleCreateChat(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
u, _ := a.authUser(w, r)
if u == nil {
return
}
var body struct {
Title *string `json:"title"`
MemberIDs *[]string `json:"member_ids"`
}
if !a.readJSON(w, r, &body) {
return
}
if body.Title == nil || body.MemberIDs == nil {
a.writeErr(w, reqID, 400, "invalid_request", "title and member_ids required")
return
}
title := *body.Title
n := utf8.RuneCountInString(title)
if n < 1 || n > 100 || isBlank(title) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid title")
return
}
members := *body.MemberIDs
if len(members) < 1 || len(members) > 100 {
a.writeErr(w, reqID, 400, "invalid_request", "invalid member_ids")
return
}
seen := make(map[string]struct{}, len(members))
hasCaller := false
for _, id := range members {
if !isUUID(id) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid member id")
return
}
if _, dup := seen[id]; dup {
a.writeErr(w, reqID, 400, "invalid_request", "duplicate member")
return
}
seen[id] = struct{}{}
if id == u.ID {
hasCaller = true
}
}
if !hasCaller {
a.writeErr(w, reqID, 400, "invalid_request", "caller must be member")
return
}
chatID := newUUID()
createdAt := nowUTC()
ctx, cancel := dbCtx(r.Context())
defer cancel()
tx, err := a.db.Begin(ctx)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
defer tx.Rollback(ctx)
_, err = tx.Exec(ctx, `insert into chats(id, title, created_at) values($1::uuid, $2, $3)`,
chatID, title, createdAt)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
for _, mid := range members {
_, err = tx.Exec(ctx, `insert into chat_members(chat_id, user_id) values($1::uuid, $2::uuid)`,
chatID, mid)
if isPgCode(err, "23503") {
a.writeErr(w, reqID, 400, "invalid_request", "unknown member")
return
}
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
}
if err := tx.Commit(ctx); err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.writeJSON(w, 201, map[string]any{
"id": chatID,
"title": title,
"member_ids": members,
"created_at": formatTime(createdAt),
})
}
func encodeCursor(parts ...string) string {
return base64.RawURLEncoding.EncodeToString([]byte(strings.Join(parts, "|")))
}
func decodeCursor(s string) ([]string, error) {
b, err := base64.RawURLEncoding.DecodeString(s)
if err != nil {
return nil, err
}
return strings.Split(string(b), "|"), nil
}
func parseLimit(r *http.Request) (int, error) {
q := r.URL.Query().Get("limit")
if q == "" {
return 50, nil
}
n, err := strconv.Atoi(q)
if err != nil || n < 1 || n > 100 {
return 0, fmt.Errorf("bad limit")
}
return n, nil
}
func (a *App) loadChatMembers(ctx context.Context, chatID string) ([]string, error) {
ctx, cancel := dbCtx(ctx)
defer cancel()
rows, err := a.db.Query(ctx, `
select user_id::text from chat_members where chat_id = $1::uuid order by user_id`, chatID)
if err != nil {
return nil, err
}
defer rows.Close()
var ids []string
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
return nil, err
}
ids = append(ids, id)
}
return ids, rows.Err()
}
func (a *App) isMember(ctx context.Context, chatID, userID string) (bool, error) {
ctx, cancel := dbCtx(ctx)
defer cancel()
var ok bool
err := a.db.QueryRow(ctx, `
select exists(
select 1 from chat_members where chat_id = $1::uuid and user_id = $2::uuid
)`, chatID, userID).Scan(&ok)
return ok, err
}
func (a *App) handleListChats(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
u, _ := a.authUser(w, r)
if u == nil {
return
}
limit, err := parseLimit(r)
if err != nil {
a.writeErr(w, reqID, 400, "invalid_request", "invalid limit")
return
}
var afterTS *time.Time
var afterID string
if cur := r.URL.Query().Get("cursor"); cur != "" {
parts, err := decodeCursor(cur)
if err != nil || len(parts) != 3 {
a.writeErr(w, reqID, 400, "invalid_request", "invalid cursor")
return
}
if parts[0] != u.ID {
a.writeErr(w, reqID, 400, "invalid_request", "invalid cursor")
return
}
ts, err := parseTime(parts[1])
if err != nil || !isUUID(parts[2]) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid cursor")
return
}
afterTS = &ts
afterID = parts[2]
}
ctx, cancel := dbCtx(r.Context())
defer cancel()
var rows pgx.Rows
if afterTS == nil {
rows, err = a.db.Query(ctx, `
select c.id::text, c.title, c.created_at
from chats c
join chat_members m on m.chat_id = c.id
where m.user_id = $1::uuid
order by c.created_at, c.id
limit $2`, u.ID, limit+1)
} else {
rows, err = a.db.Query(ctx, `
select c.id::text, c.title, c.created_at
from chats c
join chat_members m on m.chat_id = c.id
where m.user_id = $1::uuid
and (c.created_at, c.id) > ($2, $3::uuid)
order by c.created_at, c.id
limit $4`, u.ID, *afterTS, afterID, limit+1)
}
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
defer rows.Close()
type row struct {
id, title string
created time.Time
}
var list []row
for rows.Next() {
var x row
if err := rows.Scan(&x.id, &x.title, &x.created); err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
list = append(list, x)
}
if err := rows.Err(); err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
hasMore := len(list) > limit
if hasMore {
list = list[:limit]
}
out := make([]map[string]any, 0, len(list))
for _, c := range list {
members, err := a.loadChatMembers(r.Context(), c.id)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
out = append(out, map[string]any{
"id": c.id,
"title": c.title,
"member_ids": members,
"created_at": formatTime(c.created),
})
}
var next any = nil
if hasMore && len(list) > 0 {
last := list[len(list)-1]
next = encodeCursor(u.ID, formatTime(last.created), last.id)
}
a.writeJSON(w, 200, map[string]any{"items": out, "next_cursor": next})
}
func (a *App) handleGetChat(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
u, _ := a.authUser(w, r)
if u == nil {
return
}
chatID := r.PathValue("chat_id")
if !isUUID(chatID) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid chat_id")
return
}
ok, err := a.isMember(r.Context(), chatID, u.ID)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
if !ok {
a.writeErr(w, reqID, 404, "not_found", "chat not found")
return
}
ctx, cancel := dbCtx(r.Context())
defer cancel()
var title string
var createdAt time.Time
err = a.db.QueryRow(ctx, `select title, created_at from chats where id = $1::uuid`, chatID).
Scan(&title, &createdAt)
if errors.Is(err, pgx.ErrNoRows) {
a.writeErr(w, reqID, 404, "not_found", "chat not found")
return
}
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
members, err := a.loadChatMembers(r.Context(), chatID)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.writeJSON(w, 200, map[string]any{
"id": chatID,
"title": title,
"member_ids": members,
"created_at": formatTime(createdAt),
})
}
func (a *App) handleCreateMessage(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
u, _ := a.authUser(w, r)
if u == nil {
return
}
chatID := r.PathValue("chat_id")
if !isUUID(chatID) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid chat_id")
return
}
key := r.Header.Get("Idempotency-Key")
if !idempotencyKeyRe.MatchString(key) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid Idempotency-Key")
return
}
ok, err := a.isMember(r.Context(), chatID, u.ID)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
if !ok {
a.writeErr(w, reqID, 404, "not_found", "chat not found")
return
}
var body struct {
Text *string `json:"text"`
}
if !a.readJSON(w, r, &body) {
return
}
if body.Text == nil {
a.writeErr(w, reqID, 400, "invalid_request", "text required")
return
}
text := *body.Text
n := utf8.RuneCountInString(text)
if n < 1 || n > 4000 || isBlank(text) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid text")
return
}
sum := sha256.Sum256([]byte(text))
bodyHash := hex.EncodeToString(sum[:])
path := r.URL.Path
ctx, cancel := dbCtx(r.Context())
defer cancel()
tx, err := a.db.Begin(ctx)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
defer tx.Rollback(ctx)
if _, err := tx.Exec(ctx, `
delete from idempotency_keys
where user_id=$1::uuid and method='POST' and path=$2 and key=$3
and created_at < $4`,
u.ID, path, key, time.Now().Add(-a.idempotencyTTL)); err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
msgID := newUUID()
createdAt := nowUTC()
resp := map[string]any{
"id": msgID,
"chat_id": chatID,
"sender_id": u.ID,
"text": text,
"created_at": formatTime(createdAt),
}
respJSON, _ := json.Marshal(resp)
tag, err := tx.Exec(ctx, `
insert into idempotency_keys(user_id, method, path, key, body_hash, response, created_at)
values($1::uuid, 'POST', $2, $3, $4, $5, $6)
on conflict do nothing`,
u.ID, path, key, bodyHash, respJSON, createdAt)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
if tag.RowsAffected() == 0 {
var storedHash string
var stored []byte
err := tx.QueryRow(ctx, `
select body_hash, response from idempotency_keys
where user_id=$1::uuid and method='POST' and path=$2 and key=$3`,
u.ID, path, key).Scan(&storedHash, &stored)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
if storedHash != bodyHash {
a.writeErr(w, reqID, 409, "conflict", "idempotency key reused with different body")
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(201)
_, _ = w.Write(append(stored, '\n'))
return
}
if _, err := tx.Exec(ctx, `
insert into messages(id, chat_id, sender_id, text, created_at)
values($1::uuid, $2::uuid, $3::uuid, $4, $5)`,
msgID, chatID, u.ID, text, createdAt); err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
if err := tx.Commit(ctx); err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.writeJSON(w, 201, resp)
}
func (a *App) handleListMessages(w http.ResponseWriter, r *http.Request) {
reqID := reqIDFrom(r)
u, _ := a.authUser(w, r)
if u == nil {
return
}
chatID := r.PathValue("chat_id")
if !isUUID(chatID) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid chat_id")
return
}
ok, err := a.isMember(r.Context(), chatID, u.ID)
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
if !ok {
a.writeErr(w, reqID, 404, "not_found", "chat not found")
return
}
limit, err := parseLimit(r)
if err != nil {
a.writeErr(w, reqID, 400, "invalid_request", "invalid limit")
return
}
var afterTS *time.Time
var afterID string
if cur := r.URL.Query().Get("cursor"); cur != "" {
parts, err := decodeCursor(cur)
if err != nil || len(parts) != 3 {
a.writeErr(w, reqID, 400, "invalid_request", "invalid cursor")
return
}
if parts[0] != chatID {
a.writeErr(w, reqID, 400, "invalid_request", "invalid cursor")
return
}
ts, err := parseTime(parts[1])
if err != nil || !isUUID(parts[2]) {
a.writeErr(w, reqID, 400, "invalid_request", "invalid cursor")
return
}
afterTS = &ts
afterID = parts[2]
}
ctx, cancel := dbCtx(r.Context())
defer cancel()
var rows pgx.Rows
if afterTS == nil {
rows, err = a.db.Query(ctx, `
select id::text, chat_id::text, sender_id::text, text, created_at
from messages
where chat_id = $1::uuid
order by created_at, id
limit $2`, chatID, limit+1)
} else {
rows, err = a.db.Query(ctx, `
select id::text, chat_id::text, sender_id::text, text, created_at
from messages
where chat_id = $1::uuid
and (created_at, id) > ($2, $3::uuid)
order by created_at, id
limit $4`, chatID, *afterTS, afterID, limit+1)
}
if err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
defer rows.Close()
type row struct {
id, chatID, senderID, text string
created time.Time
}
var list []row
for rows.Next() {
var x row
if err := rows.Scan(&x.id, &x.chatID, &x.senderID, &x.text, &x.created); err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
list = append(list, x)
}
if err := rows.Err(); err != nil {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
hasMore := len(list) > limit
if hasMore {
list = list[:limit]
}
out := make([]map[string]any, 0, len(list))
for _, m := range list {
out = append(out, map[string]any{
"id": m.id,
"chat_id": m.chatID,
"sender_id": m.senderID,
"text": m.text,
"created_at": formatTime(m.created),
})
}
var next any = nil
if hasMore && len(list) > 0 {
last := list[len(list)-1]
next = encodeCursor(chatID, formatTime(last.created), last.id)
}
a.writeJSON(w, 200, map[string]any{"items": out, "next_cursor": next})
}
func envOr(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
+38
View File
@@ -0,0 +1,38 @@
global
log stdout format raw local0
maxconn 2000
resolvers docker
nameserver dns 127.0.0.11:53
accepted_payload_size 8192
hold valid 1s
hold nx 1s
hold obsolete 1s
resolve_retries 3
timeout resolve 1s
timeout retry 1s
defaults
mode http
log global
option httplog
option redispatch 1
retries 3
retry-on conn-failure empty-response
timeout connect 1s
timeout client 20s
timeout server 15s
timeout http-request 10s
timeout queue 5s
frontend public
bind :8080
default_backend api
backend api
balance roundrobin
option httpchk
http-check send meth GET uri /health/ready
http-check expect status 200
default-server check inter 1s fastinter 500ms downinter 1s fall 2 rise 2 resolvers docker init-addr none on-marked-down shutdown-sessions
server-template api 1-8 api:8080
+89
View File
@@ -0,0 +1,89 @@
package main
import (
"context"
"log/slog"
"net"
"net/http"
"os"
"os/signal"
"strconv"
"syscall"
"time"
)
func main() {
log := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
host := envOr("APP_HOST", "0.0.0.0")
port := envOr("APP_PORT", "8080")
shutdownSec, _ := strconv.Atoi(envOr("SHUTDOWN_TIMEOUT_SECONDS", "5"))
if shutdownSec < 1 {
shutdownSec = 5
}
ttlSec, _ := strconv.Atoi(envOr("SESSION_TTL_SECONDS", "3600"))
if ttlSec < 1 {
ttlSec = 3600
}
idemSec, _ := strconv.Atoi(envOr("IDEMPOTENCY_TTL_SECONDS", "86400"))
if idemSec < 1 {
idemSec = 86400
}
drainSec, _ := strconv.Atoi(envOr("DRAIN_SECONDS", "3"))
ctx := context.Background()
db, err := openDB(ctx)
if err != nil {
log.Error("database open failed", "err", err.Error())
os.Exit(1)
}
defer db.Close()
if len(os.Args) > 1 && os.Args[1] == "migrate" {
if err := migrate(ctx, db); err != nil {
log.Error("migrate failed", "err", err.Error())
os.Exit(1)
}
log.Info("migrations applied")
return
}
cache := openCache(log)
defer cache.Close()
app := NewApp(db, cache, log, time.Duration(ttlSec)*time.Second, time.Duration(idemSec)*time.Second)
addr := net.JoinHostPort(host, port)
srv := &http.Server{
Addr: addr,
Handler: app.Routes(),
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 10 * time.Second,
WriteTimeout: 15 * time.Second,
IdleTimeout: 30 * time.Second,
}
go func() {
log.Info("listening", "addr", addr, "env", envOr("APP_ENV", "lab"), "session_ttl_s", ttlSec)
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Error("server error", "err", err.Error())
os.Exit(1)
}
}()
stop := make(chan os.Signal, 1)
signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)
sig := <-stop
log.Info("shutdown started", "signal", sig.String())
app.SetReady(false)
time.Sleep(time.Duration(drainSec) * time.Second)
shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(shutdownSec)*time.Second)
defer cancel()
if err := srv.Shutdown(shCtx); err != nil {
log.Error("shutdown error", "err", err.Error())
os.Exit(1)
}
log.Info("shutdown finished")
}
+84
View File
@@ -0,0 +1,84 @@
package main
import (
"context"
"embed"
"fmt"
"sort"
"strings"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
//go:embed migrations/*.sql
var migrationFS embed.FS
func migrate(ctx context.Context, pool *pgxpool.Pool) error {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
conn, err := pool.Acquire(ctx)
if err != nil {
return err
}
defer conn.Release()
if _, err := conn.Exec(ctx, `select pg_advisory_lock(727001)`); err != nil {
return err
}
defer conn.Exec(context.Background(), `select pg_advisory_unlock(727001)`)
_, err = conn.Exec(ctx, `
create table if not exists schema_migrations (
version text primary key,
applied_at timestamptz not null
)`)
if err != nil {
return fmt.Errorf("schema_migrations: %w", err)
}
entries, err := migrationFS.ReadDir("migrations")
if err != nil {
return err
}
var names []string
for _, e := range entries {
if !e.IsDir() && strings.HasSuffix(e.Name(), ".sql") {
names = append(names, e.Name())
}
}
sort.Strings(names)
for _, name := range names {
var exists bool
err := conn.QueryRow(ctx, `select exists(select 1 from schema_migrations where version=$1)`, name).Scan(&exists)
if err != nil {
return err
}
if exists {
continue
}
body, err := migrationFS.ReadFile("migrations/" + name)
if err != nil {
return err
}
tx, err := conn.Begin(ctx)
if err != nil {
return err
}
if _, err := tx.Exec(ctx, string(body)); err != nil {
_ = tx.Rollback(ctx)
return fmt.Errorf("migration %s: %w", name, err)
}
if _, err := tx.Exec(ctx, `insert into schema_migrations(version, applied_at) values($1, now())`, name); err != nil {
_ = tx.Rollback(ctx)
return err
}
if err := tx.Commit(ctx); err != nil {
return err
}
}
return nil
}
+38
View File
@@ -0,0 +1,38 @@
create table users (
id uuid primary key,
username text not null unique,
display_name text not null,
password_hash text not null,
created_at timestamptz not null
);
create table chats (
id uuid primary key,
title text not null,
created_at timestamptz not null
);
create table chat_members (
chat_id uuid not null references chats(id),
user_id uuid not null references users(id),
primary key (chat_id, user_id)
);
create index chat_members_user_idx on chat_members (user_id);
create table messages (
id uuid primary key,
chat_id uuid not null references chats(id),
sender_id uuid not null references users(id),
text text not null,
created_at timestamptz not null
);
create index messages_history_idx on messages (chat_id, created_at, id);
create table sessions (
token_hash bytea primary key,
user_id uuid not null references users(id) on delete cascade,
created_at timestamptz not null,
expires_at timestamptz not null
);
+10
View File
@@ -0,0 +1,10 @@
create table idempotency_keys (
user_id uuid not null references users(id) on delete cascade,
method text not null,
path text not null,
key text not null,
body_hash text not null,
response jsonb not null,
created_at timestamptz not null,
primary key (user_id, method, path, key)
);
+33
View File
@@ -0,0 +1,33 @@
package main
import "time"
type User struct {
ID string
Username string
DisplayName string
PasswordHash string
CreatedAt time.Time
}
type Chat struct {
ID string
Title string
MemberIDs []string
CreatedAt time.Time
}
type Message struct {
ID string
ChatID string
SenderID string
Text string
CreatedAt time.Time
}
type Session struct {
TokenHash []byte
UserID string
CreatedAt time.Time
ExpiresAt time.Time
}
+13
View File
@@ -0,0 +1,13 @@
#!/bin/sh
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
mkdir -p "$ROOT/secrets"
if [ ! -f "$ROOT/secrets/postgres_password" ]; then
openssl rand -base64 24 | tr -d '\n' > "$ROOT/secrets/postgres_password"
fi
PASS=$(cat "$ROOT/secrets/postgres_password")
ENC=$(printf '%s' "$PASS" | python3 -c 'import sys,urllib.parse; print(urllib.parse.quote(sys.stdin.read(), safe=""))')
printf 'postgres://messenger:%s@postgres:5432/messenger?sslmode=disable\n' "$ENC" > "$ROOT/secrets/database_url"
printf 'redis://valkey:6379/0\n' > "$ROOT/secrets/cache_url"
chmod 644 "$ROOT/secrets/"*
echo "secrets ready in $ROOT/secrets"
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
docker compose --env-file .env restart postgres valkey
docker compose --env-file .env up -d --wait postgres valkey
docker compose --env-file .env restart api lb
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
set -eu
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
dc="docker compose --env-file .env"
old=$($dc ps -q api)
n=$(echo "$old" | wc -w)
$dc build migrate
$dc up -d --no-deps --no-recreate migrate
$dc up -d --no-deps --no-recreate --scale api=$((n * 2)) api
$dc up -d --wait --no-deps --no-recreate --scale api=$((n * 2)) api
for c in $old; do
docker stop -t 10 "$c" >/dev/null
docker rm "$c" >/dev/null
done
$dc up -d --wait --no-deps --no-recreate --scale api=$n api
+10
View File
@@ -0,0 +1,10 @@
#!/bin/sh
set -eu
: "${TARGET_INSTANCE_ID:?}"
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
id=$(docker compose --env-file .env ps -q api | while read -r c; do
case "$c" in "$TARGET_INSTANCE_ID"*) echo "$c";; esac
done)
[ -n "$id" ] || exit 1
docker stop -t 10 "$id" >/dev/null
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env python3
from concurrent.futures import ThreadPoolExecutor
import threading
import uuid
from client import Client, Failure, arguments, check, run
def main():
args = arguments('Concurrent Idempotency-Key check through the load balancer')
c = Client(args.base_url, args.ca_file)
alice, bob, eve, chat = c.bootstrap()
path = '/api/v1/chats/' + chat['id'] + '/messages'
parallel = 32
key = 'idem-' + uuid.uuid4().hex
barrier = threading.Barrier(parallel)
def send(_):
local = Client(args.base_url, args.ca_file)
barrier.wait()
return local.call('POST', path, 201, actor=alice, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')
with ThreadPoolExecutor(max_workers=parallel) as pool:
results = list(pool.map(send, range(parallel)))
ids = {r['json']['id'] for r in results}
instances = {r['headers']['X-Instance-Id'] for r in results}
check(len(ids) == 1, 'parallel requests produced %d messages' % len(ids))
check(len(instances) >= 2, 'requests were served by one instance only')
check(len({r['json']['created_at'] for r in results}) == 1, 'created_at differs between replays')
listing = c.call('GET', path, actor=alice, schema='MessagePage')['json']['items']
check(sum(1 for m in listing if m['text'] == 'same') == 1, 'duplicate message in history')
c.call('POST', path, 409, actor=alice, body={'text': 'other'}, headers={'Idempotency-Key': key})
c.call('POST', path, 400, actor=alice, body={'text': 'x'})
c.call('POST', path, 400, actor=alice, body={'text': 'x'}, headers={'Idempotency-Key': 'bad key!'})
other = c.call('POST', path, 201, actor=bob, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')['json']
check(other['id'] not in ids, 'key must be scoped per user')
print('PASS: %d parallel requests, 1 message, %d instances' % (parallel, len(instances)))
if __name__ == '__main__':
run(main)
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
from concurrent.futures import ThreadPoolExecutor
import json
import subprocess
import threading
import time
import uuid
from pathlib import Path
from client import Client, Failure, arguments, check, run
ROOT = Path(__file__).resolve().parent.parent
def sh(*cmd):
return subprocess.run(cmd, cwd=ROOT, check=True, capture_output=True, text=True).stdout.strip()
def api_containers():
out = sh('docker', 'compose', '--env-file', '.env', 'ps', '-q', 'api')
return [x[:12] for x in out.split()]
def wait_healthy(count, timeout=90):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
out = sh('docker', 'compose', '--env-file', '.env', 'ps', 'api', '--format', '{{.Status}}')
if out.count('(healthy)') >= count:
return
time.sleep(1)
raise Failure('replicas did not become healthy')
class Load:
def __init__(self, base, ca, concurrency):
c = Client(base, ca)
self.alice, _, _, self.chat = c.bootstrap()
self.base, self.ca, self.n = base, ca, concurrency
self.events = []
self.lock = threading.Lock()
self.stop = threading.Event()
self.t0 = time.monotonic()
def worker(self):
c = Client(self.base, self.ca, timeout=3)
while not self.stop.is_set():
key = uuid.uuid4().hex
first_error = None
while not self.stop.is_set():
try:
r = c.message(self.alice, self.chat, 'k-' + key, key=key)
t = time.monotonic() - self.t0
with self.lock:
self.events.append((t, True, r['headers']['X-Instance-Id'], first_error))
break
except Failure:
t = time.monotonic() - self.t0
first_error = first_error or t
with self.lock:
self.events.append((t, False, None, None))
time.sleep(0.05)
def start(self):
self.pool = ThreadPoolExecutor(max_workers=self.n)
self.futures = [self.pool.submit(self.worker) for _ in range(self.n)]
def finish(self):
self.stop.set()
self.pool.shutdown(wait=True)
def summary(self, begin, end):
ev = [e for e in self.events if begin <= e[0] < end]
ok = [e for e in ev if e[1]]
bad = [e for e in ev if not e[1]]
span = max(end - begin, 1e-9)
return {
'window_seconds': round(span, 2),
'successful_requests': len(ok),
'failed_attempts': len(bad),
'successful_rps': round(len(ok) / span, 1),
'last_error_offset_seconds': round(max(e[0] for e in bad) - begin, 2) if bad else None,
'instances': sorted({e[2] for e in ok}),
}
def instances_between(self, begin, end=float('inf')):
return sorted({e[2] for e in self.events if e[1] and begin <= e[0] < end})
def failover_each(args):
ids = api_containers()
check(len(ids) >= 2, 'need at least two replicas')
load = Load(args.base_url, args.ca_file, args.concurrency)
load.start()
time.sleep(args.warmup)
results = []
base = load.summary(2, args.warmup)
for target in ids:
before = time.monotonic() - load.t0
sh('docker', 'stop', '-t', '10', target)
stopped = time.monotonic() - load.t0
time.sleep(args.hold)
after_stop = time.monotonic() - load.t0
window = load.summary(before, after_stop)
sh('docker', 'start', target)
wait_healthy(len(ids))
restored = time.monotonic() - load.t0
time.sleep(args.warmup)
results.append({
'stopped_replica': target,
'during_failure': window,
'seen_stopped_replica_after_stop': target in load.instances_between(stopped + 0.5, after_stop),
'stop_command_seconds': round(stopped - before, 2),
'restore_seconds': round(restored - after_stop, 2),
'after_restore': load.summary(restored + 1, restored + args.warmup),
})
load.finish()
return {'baseline': base, 'replicas': results}
def rolling(args):
old = api_containers()
check(len(old) >= 2, 'need at least two replicas')
n = len(old)
load = Load(args.base_url, args.ca_file, args.concurrency)
load.start()
time.sleep(args.warmup)
begin = time.monotonic() - load.t0
sh('scripts/rolling-update.sh')
end = time.monotonic() - load.t0
time.sleep(args.warmup)
load.finish()
return {
'old_replicas': old,
'new_replicas': api_containers(),
'update_window': load.summary(begin, end),
'old_replicas_served_after_update': [i for i in load.instances_between(end) if i in old],
}
def main():
def configure(p):
p.add_argument('--mode', choices=['failover-each', 'rolling'], required=True)
p.add_argument('--concurrency', type=int, default=8)
p.add_argument('--warmup', type=float, default=8)
p.add_argument('--hold', type=float, default=12)
p.add_argument('--output')
args = arguments('Failure and rolling update under load', configure)
report = failover_each(args) if args.mode == 'failover-each' else rolling(args)
text = json.dumps(report, indent=2)
if args.output:
Path(args.output).write_text(text + '\n')
print(text)
if __name__ == '__main__':
run(main)