Исправил отзыв сессии без доверия к кэшу и добавил недостающие тесты

This commit is contained in:
gera
2026-11-19 21:38:31 +03:00
parent 27fafc6725
commit 583013fa06
12 changed files with 597 additions and 160 deletions
+91
View File
@@ -0,0 +1,91 @@
# Изменения lab5 относительно lab4
## API
- `POST /chats/{chat_id}/attachments` возвращает `202` со статусом `queued` вместо `201 ready`. Проверки при приёме те же, что в lab4.
- Оригинал сначала сохраняется в приватный S3-prefix `staging/<id>`, а не сразу в конечный `attachments/<id>`.
- `GET /attachments/{id}` возвращает статусы `queued`, `processing`, `ready`, `failed`, а также `error_code` и `variants`.
- `GET /attachments/{id}/content`:
- принимает `variant=original|thumbnail|crop|optimized`, неизвестное имя → 400;
- для не-ready вложения → 409;
- отсутствующий вариант ready-вложения → 404.
- `POST /chats/{chat_id}/messages`: прикрепить `failed`-вложение нельзя (409), `queued`/`processing` можно.
## База данных
Новая миграция `migrations/0004_jobs.sql`:
- В `attachments` добавлены `status`, `error_code`, `staging_key`, `updated_at`. `object_key` теперь заполняет worker.
- `attachment_variants`: варианты с первичным ключом `(attachment_id, pipeline_version, name)`.
- `jobs`: задача обработки (статус, попытка, лимит попыток, время следующей попытки, lease, последняя ошибка).
- `outbox`: задания на публикацию в брокер.
- `job_replays`: аудит операторских повторов.
## Новые компоненты
- `broker`: Valkey Streams, отдельный сервис с volume, `appendfsync always`, `noeviction`, паролем из secrets. Stream `attachments.jobs`, группа `workers`.
- `worker`: отдельный процесс (`messenger worker`), 2 потребителя, prefetch 1, лимит 768 MiB.
- `broker.go`: формат сообщения по `jobs.schema.json`, публикация, outbox relay (работает в каждой API-реплике).
- `worker.go`:
- захват задачи с lease и fencing по `lease_owner` и `attempt`;
- обработка и фиксация результата;
- retry с backoff и jitter;
- reaper для задач с истёкшим lease;
- повторное чтение своих pending и `XAUTOCLAIM`;
- команды `failed` и `replay`.
- `imaging.go`:
- thumbnail JPEG 256, crop JPEG 128×128, optimized WebP quality 80 (вписан в 1280);
- применение EXIF orientation, alpha на белом;
- предварительное уменьшение больших изображений для экономии памяти.
## Изменения существующего кода
- `attachments.go`: приём в staging, транзакция `attachments` + `jobs` + `outbox`, ответ 202, загрузка вариантов, скачивание по `variant`. Janitor дополнительно чистит `staging/`.
- `handlers.go`: проверка failed-вложений при создании сообщения, точка внедрения отказа.
- `main.go`: подкоманды `worker`, `failed`, `replay`; запуск relay в API; новые переменные окружения.
- `s3.go`: listing по произвольному prefix.
- `Dockerfile`: сборка с тегом `nodynamic` (WebP без системных библиотек).
- `go.mod`: добавлены `golang.org/x/image`, `github.com/gen2brain/webp`, `github.com/rwcarlsen/goexif`.
## Инфраструктура и скрипты
- `compose.yaml`: сервисы `broker` и `worker`, volume `broker_data`, secrets `broker_conf` и `broker_url`, образ `mtusi-messenger:lab5`, переменные для точек внедрения отказа (по умолчанию пусты).
- `scripts/mksecrets.sh`: генерирует пароль брокера, `broker.conf` и `broker_url`.
- `scripts/restart.sh`: перезапускает также `broker` и `worker`.
- `scripts/rolling-update.sh`: обновляет также `worker`.
- `.env.example`: `WORKER_CONCURRENCY`, `JOB_MAX_ATTEMPTS`, `JOB_TIMEOUT_SECONDS`, `JOB_BACKOFF_BASE_MS`, `JOB_BACKOFF_MAX_MS`, `STAGING_RETENTION_SECONDS`.
## Надёжность
- Работа не теряется после ответа 202: оригинал в S3, задача и outbox в одной транзакции PostgreSQL.
- Разрыв COMMIT/publish закрыт transactional outbox.
- Повторная доставка и повторы не создают второго набора вариантов: детерминированные S3-ключи и upsert по ключу эффекта.
- Зависшие задачи возвращаются в очередь по истечении lease.
- Постоянные ошибки (`invalid_image`, `processing_failed`) уходят в failed сразу, временные повторяются до `retry_exhausted`.
- Failed-задачи можно повторить через `replay`. Повтор уже готовой задачи ничего не делает.
- Повреждение внутри PNG, которое lab4 пропускал (проверялись только заголовок и конец), теперь ловит worker при полном декодировании.
## Тесты
Новые:
- `tests/stack.py`: общие хелперы.
- `tests/variants.py`: размеры, форматы, sha256, без увеличения, alpha, EXIF.
- `tests/queue_recovery.py`: сценарий на 3 и повторная доставка.
- `tests/faults.py`: 5 сценариев сбоев на 4.
- `tests/worker_load.py`: конкуренция, память и timeout worker.
Адаптированы под 202: `tests/attachments.py`, `tests/s3_checks.py` (плюс проверка очистки staging), `tests/upload_memory.py`.
Без изменений: `tests/idempotency.py`, `tests/scenario.py`, `tests/image_gen.py`.
## Отчёт и доказательства
- `REPORT.md` переписан под lab5.
- Новые файлы в `evidence/`: `variants.txt`, `queue-recovery.txt`, `faults.txt`, `worker-load.json`.
- Остальные файлы evidence получены заново на стенде lab5.
## Исправления после ревью
- Авторизация проверяет сессию только в PostgreSQL одним запросом (`sessions` ⋈ `users`, `expires_at > now()`); запись кэша не принимается за доказательство действительности токена. Код кэша сессий (`cache.go`) удалён.
- Добавлен `tests/session_revocation.py`: Valkey остановлен → logout 204 → Valkey запущен → прежний токен 401.
- Тесты `image_gen.py`, `idempotency.py`, `s3_checks.py`, `scenario.py`, `upload_memory.py` были не добавлены в git, из-за чего `attachments.py` и `worker_load.py` падали с `ModuleNotFoundError: image_gen`.
+3 -1
View File
@@ -42,6 +42,7 @@ python3 tests/attachments.py
python3 tests/s3_checks.py
python3 tests/upload_memory.py --output evidence/upload-memory.json
python3 tests/idempotency.py
python3 tests/session_revocation.py
python3 tests/scenario.py --mode failover-each --output evidence/failover-each.json
python3 tests/scenario.py --mode rolling --output evidence/rolling.json
```
@@ -65,7 +66,7 @@ worker ──XREADGROUP──→ claim (lease) → S3 attachments/<id>/... → C
| broker | `broker` | valkey/valkey:8.1-alpine@sha256:081c2f5c… | `valkey-cli ping` с паролем | broker_data | backend | broker.conf (`requirepass`) |
| s3 | `s3` | chrislusf/seaweedfs@sha256:c2658820… | `/dir/assign` выдаёт fid | s3_data | storage | s3_config.json |
| load_balancer | `lb` | haproxy:3.0-alpine@sha256:cf3af0fa… | проверки `/health/ready` бэкендов | нет | backend, 127.0.0.1:8080 | нет |
| кэш сессий | `valkey` | valkey/valkey:8.1-alpine | `valkey-cli ping` | valkey_data | backend | нет |
| Valkey (сессии проверяются только в PostgreSQL) | `valkey` | valkey/valkey:8.1-alpine | `valkey-cli ping` | valkey_data | backend | нет |
Bootstrap: `migrate` применяет `migrations/*.sql` под advisory lock, повторный запуск пропускает применённые. `s3-init` создаёт bucket, если его нет. Consumer group `workers` создают API (перед первой публикацией) и worker при старте (`XGROUP CREATE … 0 MKSTREAM`, ошибка BUSYGROUP игнорируется).
@@ -128,6 +129,7 @@ Bootstrap: `migrate` применяет `migrations/*.sql` под advisory lock,
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke (202 → ready, оригинал, настоящий JPEG 256×160) | `make check && make test` | PASS | evidence/smoke.txt |
| Отозванная сессия не воскресает после перезапуска Valkey | `tests/session_revocation.py` | Valkey остановлен → logout 204 → Valkey запущен → прежний токен 401 | evidence/session-revocation.txt |
| Минимум lab4 сохранён (валидация, доступ, приватный bucket, отказ S3, идемпотентность) | `tests/attachments.py`, `tests/s3_checks.py`, `tests/idempotency.py` | PASS: 413/415/400/404, анонимный доступ 403, 503 при остановленном S3 | evidence/attachments.txt, evidence/s3-checks.txt, evidence/idempotency.txt |
| Restart всех stores с volumes | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS; перезапущены postgres, valkey, broker, s3, api, lb, janitor, worker (состояния до/после в файле) | evidence/persistence.txt |
| Failover каждой API-реплики | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | PASS для api-1 и api-2, 0–1 неуспешная попытка, восстановление 3.5 s | evidence/failover-1.txt, evidence/failover-2.txt |
-104
View File
@@ -1,104 +0,0 @@
package main
import (
"context"
"encoding/json"
"log/slog"
"time"
"github.com/redis/go-redis/v9"
)
type cachedSession struct {
UserID string `json:"user_id"`
ExpiresAt time.Time `json:"expires_at"`
}
type Cache struct {
rdb *redis.Client
log *slog.Logger
}
func openCache(log *slog.Logger) *Cache {
url, err := readSecret("CACHE_URL", "CACHE_URL_FILE")
if err != nil {
log.Warn("cache disabled", "err", err.Error())
return &Cache{log: log}
}
opt, err := redis.ParseURL(url)
if err != nil {
log.Warn("cache parse failed", "err", err.Error())
return &Cache{log: log}
}
opt.DialTimeout = 200 * time.Millisecond
opt.ReadTimeout = 200 * time.Millisecond
opt.WriteTimeout = 200 * time.Millisecond
opt.PoolTimeout = 200 * time.Millisecond
rdb := redis.NewClient(opt)
ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
defer cancel()
if err := rdb.Ping(ctx).Err(); err != nil {
log.Warn("cache ping failed, will bypass", "err", err.Error())
}
return &Cache{rdb: rdb, log: log}
}
func (c *Cache) Close() {
if c.rdb != nil {
_ = c.rdb.Close()
}
}
func sessKey(hashHex string) string {
return "sess:" + hashHex
}
func (c *Cache) GetSession(ctx context.Context, hashHex string) (*cachedSession, bool) {
if c.rdb == nil {
return nil, false
}
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
val, err := c.rdb.Get(ctx, sessKey(hashHex)).Result()
if err != nil {
return nil, false
}
var s cachedSession
if err := json.Unmarshal([]byte(val), &s); err != nil {
return nil, false
}
if time.Now().UTC().After(s.ExpiresAt) {
return nil, false
}
return &s, true
}
func (c *Cache) SetSession(ctx context.Context, hashHex string, userID string, expiresAt time.Time) {
if c.rdb == nil {
return
}
ttl := time.Until(expiresAt)
if ttl <= 0 {
return
}
body, err := json.Marshal(cachedSession{UserID: userID, ExpiresAt: expiresAt.UTC()})
if err != nil {
return
}
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
if err := c.rdb.Set(ctx, sessKey(hashHex), body, ttl).Err(); err != nil {
c.log.Warn("cache set failed", "err", err.Error())
}
}
func (c *Cache) DelSession(ctx context.Context, hashHex string) {
if c.rdb == nil {
return
}
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
if err := c.rdb.Del(ctx, sessKey(hashHex)).Err(); err != nil {
c.log.Warn("cache del failed", "err", err.Error())
}
}
+1
View File
@@ -0,0 +1 @@
PASS: token revoked while valkey was stopped stays revoked after valkey restart (401)
+10 -51
View File
@@ -49,7 +49,6 @@ type errorBody struct {
type App struct {
db *pgxpool.Pool
cache *Cache
log *slog.Logger
ready atomic.Bool
sessionTTL time.Duration
@@ -60,8 +59,8 @@ type App struct {
fault string
}
func NewApp(db *pgxpool.Pool, cache *Cache, log *slog.Logger, sessionTTL, idempotencyTTL time.Duration, store *Store, lim limits) *App {
a := &App{db: db, cache: cache, log: log, sessionTTL: sessionTTL, idempotencyTTL: idempotencyTTL, store: store, limits: lim, uploads: make(chan struct{}, lim.uploadSlots), fault: os.Getenv("FAULT_POINT")}
func NewApp(db *pgxpool.Pool, log *slog.Logger, sessionTTL, idempotencyTTL time.Duration, store *Store, lim limits) *App {
a := &App{db: db, log: log, sessionTTL: sessionTTL, idempotencyTTL: idempotencyTTL, store: store, limits: lim, uploads: make(chan struct{}, lim.uploadSlots), fault: os.Getenv("FAULT_POINT")}
a.ready.Store(true)
return a
}
@@ -268,23 +267,6 @@ func userJSON(u *User) map[string]any {
}
}
func (a *App) getUserByID(ctx context.Context, id string) (*User, error) {
ctx, cancel := dbCtx(ctx)
defer cancel()
var u User
err := a.db.QueryRow(ctx, `
select id::text, username, display_name, password_hash, created_at
from users where id = $1::uuid`, id).Scan(
&u.ID, &u.Username, &u.DisplayName, &u.PasswordHash, &u.CreatedAt)
if errors.Is(err, pgx.ErrNoRows) {
return nil, nil
}
if err != nil {
return nil, err
}
return &u, nil
}
func (a *App) authUser(w http.ResponseWriter, r *http.Request) (*User, string) {
token, ok := parseBearer(r.Header.Get("Authorization"))
if !ok {
@@ -293,26 +275,15 @@ func (a *App) authUser(w http.ResponseWriter, r *http.Request) (*User, string) {
}
hash, hashHex := tokenHash(token)
if cs, hit := a.cache.GetSession(r.Context(), hashHex); hit {
u, err := a.getUserByID(r.Context(), cs.UserID)
if err != nil {
a.writeErr(w, reqIDFrom(r), 503, "unavailable", "database unavailable")
return nil, ""
}
if u == nil {
a.writeUnauthorized(w, r, "Bearer")
return nil, ""
}
return u, hashHex
}
ctx, cancel := dbCtx(r.Context())
defer cancel()
var userID string
var u User
var expiresAt time.Time
err := a.db.QueryRow(ctx, `
select user_id::text, expires_at from sessions
where token_hash = $1 and expires_at > now()`, hash).Scan(&userID, &expiresAt)
select u.id::text, u.username, u.display_name, u.password_hash, u.created_at, s.expires_at
from sessions s join users u on u.id = s.user_id
where s.token_hash = $1 and s.expires_at > now()`, hash).Scan(
&u.ID, &u.Username, &u.DisplayName, &u.PasswordHash, &u.CreatedAt, &expiresAt)
if errors.Is(err, pgx.ErrNoRows) {
a.writeUnauthorized(w, r, "Bearer")
return nil, ""
@@ -321,17 +292,7 @@ func (a *App) authUser(w http.ResponseWriter, r *http.Request) (*User, string) {
a.writeErr(w, reqIDFrom(r), 503, "unavailable", "database unavailable")
return nil, ""
}
u, err := a.getUserByID(r.Context(), userID)
if err != nil {
a.writeErr(w, reqIDFrom(r), 503, "unavailable", "database unavailable")
return nil, ""
}
if u == nil {
a.writeUnauthorized(w, r, "Bearer")
return nil, ""
}
a.cache.SetSession(r.Context(), hashHex, userID, expiresAt)
return u, hashHex
return &u, hashHex
}
func (a *App) handleCreateUser(w http.ResponseWriter, r *http.Request) {
@@ -423,7 +384,7 @@ func (a *App) handleCreateSession(w http.ResponseWriter, r *http.Request) {
a.writeUnauthorized(w, r, "Basic")
return
}
token, hash, hashHex, err := newToken()
token, hash, _, err := newToken()
if err != nil {
a.writeErr(w, reqID, 500, "internal_error", "internal error")
return
@@ -437,7 +398,6 @@ func (a *App) handleCreateSession(w http.ResponseWriter, r *http.Request) {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.cache.SetSession(r.Context(), hashHex, u.ID, expiresAt)
a.writeJSON(w, 201, map[string]any{
"token": token,
"token_type": "Bearer",
@@ -453,7 +413,7 @@ func (a *App) handleRevokeSession(w http.ResponseWriter, r *http.Request) {
a.writeUnauthorized(w, r, "Bearer")
return
}
hash, hashHex := tokenHash(token)
hash, _ := tokenHash(token)
ctx, cancel := dbCtx(r.Context())
defer cancel()
tag, err := a.db.Exec(ctx, `
@@ -462,7 +422,6 @@ func (a *App) handleRevokeSession(w http.ResponseWriter, r *http.Request) {
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
return
}
a.cache.DelSession(r.Context(), hashHex)
if tag.RowsAffected() == 0 {
a.writeUnauthorized(w, r, "Bearer")
return
+1 -4
View File
@@ -125,11 +125,8 @@ func main() {
return
}
cache := openCache(log)
defer cache.Close()
slots := envInt("UPLOAD_CONCURRENCY", 10)
app := NewApp(db, cache, log, time.Duration(ttlSec)*time.Second, time.Duration(idemSec)*time.Second, store,
app := NewApp(db, log, time.Duration(ttlSec)*time.Second, time.Duration(idemSec)*time.Second, store,
limits{maxBytes: maxBytes, maxPixels: maxPixels, uploadSlots: slots, maxAttempts: maxAttempts})
relayCtx, stopRelay := context.WithCancel(ctx)
defer stopRelay()
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env python3
from concurrent.futures import ThreadPoolExecutor
import threading
import uuid
from client import Client, Failure, arguments, check, run
def main():
args = arguments('Concurrent Idempotency-Key check through the load balancer')
c = Client(args.base_url, args.ca_file)
alice, bob, eve, chat = c.bootstrap()
path = '/api/v1/chats/' + chat['id'] + '/messages'
parallel = 32
key = 'idem-' + uuid.uuid4().hex
barrier = threading.Barrier(parallel)
def send(_):
local = Client(args.base_url, args.ca_file)
barrier.wait()
return local.call('POST', path, 201, actor=alice, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')
with ThreadPoolExecutor(max_workers=parallel) as pool:
results = list(pool.map(send, range(parallel)))
ids = {r['json']['id'] for r in results}
instances = {r['headers']['X-Instance-Id'] for r in results}
check(len(ids) == 1, 'parallel requests produced %d messages' % len(ids))
check(len(instances) >= 2, 'requests were served by one instance only')
check(len({r['json']['created_at'] for r in results}) == 1, 'created_at differs between replays')
listing = c.call('GET', path, actor=alice, schema='MessagePage')['json']['items']
check(sum(1 for m in listing if m['text'] == 'same') == 1, 'duplicate message in history')
c.call('POST', path, 409, actor=alice, body={'text': 'other'}, headers={'Idempotency-Key': key})
c.call('POST', path, 400, actor=alice, body={'text': 'x'})
c.call('POST', path, 400, actor=alice, body={'text': 'x'}, headers={'Idempotency-Key': 'bad key!'})
other = c.call('POST', path, 201, actor=bob, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')['json']
check(other['id'] not in ids, 'key must be scoped per user')
print('PASS: %d parallel requests, 1 message, %d instances' % (parallel, len(instances)))
if __name__ == '__main__':
run(main)
+23
View File
@@ -0,0 +1,23 @@
import os
import struct
import zlib
def chunk(kind, data):
body = kind + data
return struct.pack('>I', len(data)) + body + struct.pack('>I', zlib.crc32(body) & 0xffffffff)
def png(width, height, noise=False):
header = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
raw = bytearray()
row = width * 3 + 1
for _ in range(height):
raw += b'\x00' + (os.urandom(row - 1) if noise else bytes(row - 1))
data = zlib.compress(bytes(raw), 0 if noise else 6)
return header + chunk(b'IDAT', data) + chunk(b'IEND', b'')
def png_header_only(width, height):
header = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
return header + chunk(b'IDAT', zlib.compress(b'\x00')) + chunk(b'IEND', b'')
+134
View File
@@ -0,0 +1,134 @@
#!/usr/bin/env python3
import hashlib
import subprocess
import time
import uuid
from pathlib import Path
from client import Client, Failure, arguments, check, run
from stack import wait_status
ROOT = Path(__file__).resolve().parent.parent
SAMPLE = (ROOT / 'tests/fixtures/sample.png').read_bytes()
BUCKET = 'messenger-attachments'
def compose(*args, check_rc=True, stdin=None):
return subprocess.run(['docker', 'compose', '--env-file', '.env', *args], cwd=ROOT, check=check_rc,
capture_output=True, text=True, input=stdin)
def secret(name):
return (ROOT / 'secrets' / name).read_text().strip()
def s3_status(method, path, user=None, data=None):
cmd = ['exec', '-T', 's3', 'curl', '-s', '-o', '/dev/null', '-w', '%{http_code}', '-X', method]
if user:
cmd += ['--aws-sigv4', 'aws:amz:us-east-1:s3', '--user', user[0] + ':' + user[1]]
if data is not None:
cmd += ['--data-binary', data]
cmd.append('http://127.0.0.1:9000' + path)
return compose(*cmd).stdout.strip()
def upload(c, actor, chat, expected):
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=a.png', expected, actor=actor,
raw=SAMPLE, headers={'Content-Type': 'image/png'}, schema='Attachment' if expected == 202 else None)
def wait_api(c):
deadline = time.monotonic() + 60
while time.monotonic() < deadline:
try:
c.call('GET', '/health/ready')
return
except Failure:
time.sleep(1)
raise Failure('api did not recover')
def wait_s3(timeout=90):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if compose('ps', 's3', '--format', '{{.Status}}').stdout.count('(healthy)'):
return
time.sleep(1)
raise Failure('s3 did not recover')
def object_key(attachment_id):
out = compose('exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-Atc',
"select object_key from attachments where id='%s'" % attachment_id).stdout.strip()
check(out.startswith('attachments/'), 'object key not found')
return out
def main():
args = arguments('S3 privacy, outage and orphan cleanup checks')
c = Client(args.base_url, args.ca_file, timeout=15)
alice, bob, eve, chat = c.bootstrap()
att = upload(c, alice, chat, 202)['json']
wait_status(c, bob, att['id'], ('ready',), 60)
key = object_key(att['id'])
check(key != 'attachments/' + att['filename'] and att['filename'] not in key, 'object key derived from filename')
api = (secret('s3_access_key_id'), secret('s3_secret_access_key'))
admin = (secret('s3_admin_access_key_id'), secret('s3_admin_secret_access_key'))
path = '/' + BUCKET + '/' + key
check(s3_status('GET', path) == '403', 'anonymous GET must be denied')
check(s3_status('GET', '/' + BUCKET + '/') == '403', 'anonymous listing must be denied')
check(s3_status('PUT', path + '-anon', data='x') == '403', 'anonymous PUT must be denied')
check(s3_status('GET', path, ('wrong', 'wrong')) == '403', 'wrong credentials must be denied')
check(s3_status('GET', path, api) == '200', 'api credentials cannot read')
check(s3_status('PUT', '/other-bucket-' + uuid.uuid4().hex[:8], api) == '403', 'api credentials can create buckets')
ports = compose('ps', '--format', '{{.Service}} {{.Ports}}').stdout
published = [line.split()[0] for line in ports.splitlines() if '->' in line]
check(published == ['lb'], 'only lb may publish ports, found %s' % published)
compose('stop', 's3')
try:
upload(c, alice, chat, 503)
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', 503, actor=bob)
c.call('GET', '/api/v1/attachments/' + att['id'], 200, actor=bob, schema='Attachment')
c.message(alice, chat, 'text works while s3 is down')
finally:
compose('start', 's3')
wait_s3()
wait_api(c)
deadline = time.monotonic() + 60
while True:
try:
body = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes']
check(body == SAMPLE, 'content differs after s3 restart')
break
except Failure:
check(time.monotonic() < deadline, 'download did not recover')
time.sleep(1)
upload(c, alice, chat, 202)
orphan = '/' + BUCKET + '/attachments/' + uuid.uuid4().hex
check(s3_status('PUT', orphan, admin, 'orphan') == '200', 'cannot create orphan')
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=3600', 'janitor', 'cleanup')
check(s3_status('GET', orphan, admin) == '200', 'fresh object was removed inside grace period')
time.sleep(1)
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=0', 'janitor', 'cleanup')
check(s3_status('GET', orphan, admin) == '404', 'orphan was not removed')
check(c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes'] == SAMPLE, 'referenced object was removed')
check(s3_status('GET', '/' + BUCKET + '/staging/' + att['id'], admin) == '404', 'staging kept after ready')
stale = '/' + BUCKET + '/staging/' + str(uuid.uuid4())
check(s3_status('PUT', stale, admin, 'stale') == '200', 'cannot create stale staging object')
compose('stop', 'worker')
try:
queued = upload(c, alice, chat, 202)['json']
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=0', 'janitor', 'cleanup')
check(s3_status('GET', '/' + BUCKET + '/staging/' + queued['id'], admin) == '200', 'staging of queued job was removed')
check(s3_status('GET', stale, admin) == '404', 'unreferenced staging object was not removed')
finally:
compose('up', '-d', '--no-deps', 'worker')
check(wait_status(c, bob, queued['id'], ('ready', 'failed'), 60)['status'] == 'ready', 'queued job lost staging')
print('PASS: private bucket, limited credentials, s3 outage, orphan and staging cleanup')
if __name__ == '__main__':
run(main)
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
from concurrent.futures import ThreadPoolExecutor
import json
import subprocess
import threading
import time
import uuid
from pathlib import Path
from client import Client, Failure, arguments, check, run
ROOT = Path(__file__).resolve().parent.parent
def sh(*cmd):
return subprocess.run(cmd, cwd=ROOT, check=True, capture_output=True, text=True).stdout.strip()
def api_containers():
out = sh('docker', 'compose', '--env-file', '.env', 'ps', '-q', 'api')
return [x[:12] for x in out.split()]
def wait_healthy(count, timeout=90):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
out = sh('docker', 'compose', '--env-file', '.env', 'ps', 'api', '--format', '{{.Status}}')
if out.count('(healthy)') >= count:
return
time.sleep(1)
raise Failure('replicas did not become healthy')
class Load:
def __init__(self, base, ca, concurrency):
c = Client(base, ca)
self.alice, _, _, self.chat = c.bootstrap()
self.base, self.ca, self.n = base, ca, concurrency
self.events = []
self.lock = threading.Lock()
self.stop = threading.Event()
self.t0 = time.monotonic()
def worker(self):
c = Client(self.base, self.ca, timeout=3)
while not self.stop.is_set():
key = uuid.uuid4().hex
first_error = None
while not self.stop.is_set():
try:
r = c.message(self.alice, self.chat, 'k-' + key, key=key)
t = time.monotonic() - self.t0
with self.lock:
self.events.append((t, True, r['headers']['X-Instance-Id'], first_error))
break
except Failure:
t = time.monotonic() - self.t0
first_error = first_error or t
with self.lock:
self.events.append((t, False, None, None))
time.sleep(0.05)
def start(self):
self.pool = ThreadPoolExecutor(max_workers=self.n)
self.futures = [self.pool.submit(self.worker) for _ in range(self.n)]
def finish(self):
self.stop.set()
self.pool.shutdown(wait=True)
def summary(self, begin, end):
ev = [e for e in self.events if begin <= e[0] < end]
ok = [e for e in ev if e[1]]
bad = [e for e in ev if not e[1]]
span = max(end - begin, 1e-9)
return {
'window_seconds': round(span, 2),
'successful_requests': len(ok),
'failed_attempts': len(bad),
'successful_rps': round(len(ok) / span, 1),
'last_error_offset_seconds': round(max(e[0] for e in bad) - begin, 2) if bad else None,
'instances': sorted({e[2] for e in ok}),
}
def instances_between(self, begin, end=float('inf')):
return sorted({e[2] for e in self.events if e[1] and begin <= e[0] < end})
def failover_each(args):
ids = api_containers()
check(len(ids) >= 2, 'need at least two replicas')
load = Load(args.base_url, args.ca_file, args.concurrency)
load.start()
time.sleep(args.warmup)
results = []
base = load.summary(2, args.warmup)
for target in ids:
before = time.monotonic() - load.t0
sh('docker', 'stop', '-t', '10', target)
stopped = time.monotonic() - load.t0
time.sleep(args.hold)
after_stop = time.monotonic() - load.t0
window = load.summary(before, after_stop)
sh('docker', 'start', target)
wait_healthy(len(ids))
restored = time.monotonic() - load.t0
time.sleep(args.warmup)
results.append({
'stopped_replica': target,
'during_failure': window,
'seen_stopped_replica_after_stop': target in load.instances_between(stopped + 0.5, after_stop),
'stop_command_seconds': round(stopped - before, 2),
'restore_seconds': round(restored - after_stop, 2),
'after_restore': load.summary(restored + 1, restored + args.warmup),
})
load.finish()
return {'baseline': base, 'replicas': results}
def rolling(args):
old = api_containers()
check(len(old) >= 2, 'need at least two replicas')
n = len(old)
load = Load(args.base_url, args.ca_file, args.concurrency)
load.start()
time.sleep(args.warmup)
begin = time.monotonic() - load.t0
sh('scripts/rolling-update.sh')
end = time.monotonic() - load.t0
time.sleep(args.warmup)
load.finish()
return {
'old_replicas': old,
'new_replicas': api_containers(),
'update_window': load.summary(begin, end),
'old_replicas_served_after_update': [i for i in load.instances_between(end) if i in old],
}
def main():
def configure(p):
p.add_argument('--mode', choices=['failover-each', 'rolling'], required=True)
p.add_argument('--concurrency', type=int, default=8)
p.add_argument('--warmup', type=float, default=8)
p.add_argument('--hold', type=float, default=12)
p.add_argument('--output')
args = arguments('Failure and rolling update under load', configure)
report = failover_each(args) if args.mode == 'failover-each' else rolling(args)
text = json.dumps(report, indent=2)
if args.output:
Path(args.output).write_text(text + '\n')
print(text)
if __name__ == '__main__':
run(main)
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env python3
import time
from client import Client, arguments, check, run
from stack import compose
def wait_valkey(timeout=60):
deadline = time.monotonic() + timeout
while '(healthy)' not in compose('ps', 'valkey', '--format', '{{.Status}}').stdout:
check(time.monotonic() < deadline, 'valkey did not recover')
time.sleep(1)
def main():
args = arguments('Session revocation must not depend on the cache')
c = Client(args.base_url, args.ca_file, timeout=15)
alice, bob, eve, chat = c.bootstrap()
c.call('GET', '/api/v1/chats', 200, actor=alice)
compose('stop', 'valkey')
try:
c.call('DELETE', '/api/v1/auth/sessions/current', 204, actor=alice)
finally:
compose('start', 'valkey')
wait_valkey()
c.call('GET', '/api/v1/chats', 401, actor=alice)
c.call('GET', '/api/v1/chats', 200, actor=bob)
print('PASS: token revoked while valkey was stopped stays revoked after valkey restart (401)')
if __name__ == '__main__':
run(main)
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
from concurrent.futures import ThreadPoolExecutor
import hashlib
import json
import re
import subprocess
import threading
import time
from pathlib import Path
from client import Client, Failure, arguments, check, run
from image_gen import png
ROOT = Path(__file__).resolve().parent.parent
UNITS = {'B': 1, 'KiB': 1024, 'MiB': 1024 ** 2, 'GiB': 1024 ** 3}
def to_mib(text):
value, unit = re.match(r'([\d.]+)\s*([A-Za-z]+)', text).groups()
return float(value) * UNITS[unit] / 1024 ** 2
def sample(stop, peaks):
proc = subprocess.Popen(['docker', 'stats', '--format', '{{.Name}} {{.MemUsage}}'], cwd=ROOT,
stdout=subprocess.PIPE, text=True)
try:
for line in proc.stdout:
line = re.sub(r'\x1b\[[0-9;]*[A-Za-z]', '', line).strip()
m = re.match(r'(\S+-api-\d+)\s+(\S+ ?\S+) /', line)
if m:
peaks[m.group(1)] = max(peaks.get(m.group(1), 0), to_mib(m.group(2)))
if stop.is_set():
break
finally:
proc.terminate()
def main():
def configure(p):
p.add_argument('--concurrency', type=int, default=12)
p.add_argument('--rounds', type=int, default=3)
p.add_argument('--output')
args = arguments('Concurrent upload memory experiment', configure)
check(args.concurrency >= 10, 'use at least 10 concurrent uploads')
c = Client(args.base_url, args.ca_file, timeout=60)
alice, bob, eve, chat = c.bootstrap()
data = png(1800, 1800, noise=True)
digest = hashlib.sha256(data).hexdigest()
path = '/api/v1/chats/' + chat['id'] + '/attachments?filename=load.png'
idle, stop = {}, threading.Event()
t = threading.Thread(target=sample, args=(stop, idle))
t.start()
time.sleep(4)
stop.set()
t.join()
peaks, stop = {}, threading.Event()
t = threading.Thread(target=sample, args=(stop, peaks))
t.start()
time.sleep(2)
barrier = threading.Barrier(args.concurrency)
results = []
def worker(_):
local = Client(args.base_url, args.ca_file, timeout=60)
barrier.wait()
out = []
for _ in range(args.rounds):
try:
r = local.call('POST', path, 202, actor=alice, raw=data, headers={'Content-Type': 'image/png'}, schema='Attachment')
out.append(r['json']['sha256'] == digest)
except Failure:
out.append(False)
return out
started = time.monotonic()
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
for out in pool.map(worker, range(args.concurrency)):
results.extend(out)
elapsed = time.monotonic() - started
time.sleep(2)
stop.set()
t.join()
limit = 192
report = {
'file_bytes': len(data),
'concurrent_uploads': args.concurrency,
'rounds': args.rounds,
'uploads_ok': sum(results),
'uploads_failed': len(results) - sum(results),
'elapsed_seconds': round(elapsed, 2),
'api_memory_limit_mib': limit,
'idle_memory_mib': {k: round(v, 1) for k, v in sorted(idle.items())},
'peak_memory_mib': {k: round(v, 1) for k, v in sorted(peaks.items())},
'buffering_all_files_would_need_mib': round(args.concurrency * len(data) / 1024 ** 2, 1),
}
text = json.dumps(report, indent=2)
if args.output:
Path(args.output).write_text(text + '\n')
print(text)
check(report['uploads_failed'] == 0, 'some uploads failed')
check(max(peaks.values()) < limit, 'peak memory reached the limit')
if __name__ == '__main__':
run(main)