ЛР7 - БВТ2303 - Кунецкий - на 5 #10

Closed
vladislavKunetsky wants to merge 3 commits from vladislavKunetsky/mtusi2026lab7:lab7 into main
570 changed files with 298308 additions and 137 deletions
+7
View File
@@ -18,3 +18,10 @@ __pycache__
node_modules
evidence
.DS_Store
messenger
s3init
bin
coverage.out
.cache
coverage.out.tmp
*.log
+87 -36
View File
@@ -1,41 +1,92 @@
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
# These are logical interface suggestions; map actual framework names in REPORT.md.
APP_HOST=0.0.0.0
APP_PORT=8080
APP_ENV=lab
# Local synthetic stand. Copy to .env; this file contains NO credential values.
# make prepare generates role keys/backend secrets; operator root/unseal state
# stays outside the checkout under ~/.local/share/mtusi-lab7/<project> (0700).
COMPOSE_PROJECT_NAME=mtusi-messenger-lab6
LAB7_SECRETS_DIR=secrets/lab7
BASE_URL=https://localhost:8443
CA_FILE=secrets/lab7/pki/operator/current/ca.crt
PUBLIC_PORT=8443
S3_PORT=8333
BAO_PORT=8200
GRAFANA_PORT=3000
PROMETHEUS_PORT=9090
LOKI_PORT=3100
TEMPO_PORT=3200
BROKER_MANAGEMENT_PORT=15671
LB_STATS_PORT=8404
# Operator-local integration ports (only docker-compose.local.yaml publishes them).
TEST_POSTGRES_PORT=15432
TEST_REDIS_PORT=16379
# Service identities/CA/CRL are per-role mounts, set explicitly in compose.yaml:
# TLS_CA_FILE=/run/identity/current/ca.crt
# TLS_CERT_FILE=/run/identity/current/tls.crt
# TLS_KEY_FILE=/run/identity/current/tls.key
# TLS_CRL_FILE=/run/identity/current/ca.crl
# TLS_VERSION_FILE=/run/identity/current-version (atomic immutable-version selector)
# BAO_ADDR=https://openbao:8200; BAO_ROLE=api/worker/relay;
# BAO_KV_PATH=secret/data/<role>; credentials fetched by cert-auth at startup.
# Password PHC parameters: Argon2id v19 m=19456 KiB,t=2,p=1 (measured in REPORT).
# Shared login allowance: account8/minute, sourceIP120/minute, global16/second.
# Hashing concurrency: at most2 derivations per API replica; no unbounded queue.
APP_ENV=dev
SERVER_MAX_BODY_BYTES=262144
SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
S3_ENDPOINT=https://s3:9000
S3_REGION=us-east-1
API_REPLICAS=2
API_CPU_LIMIT=0.50
API_MEM_LIMIT=256M
POSTGRES_CPU_LIMIT=0.50
POSTGRES_MEM_LIMIT=320M
POSTGRES_MAX_CONNS=10
S3_BUCKET=messenger-attachments
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
BROKER_URL_FILE=/run/secrets/broker_url
WORKER_CONCURRENCY=2
JOB_MAX_ATTEMPTS=5
JOB_TIMEOUT_SECONDS=60
STAGING_RETENTION_SECONDS=86400
OTEL_SERVICE_NAME=messenger-api
# Worker must set a distinct service.name.
OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
# TLS and OpenBao: actual paths/ports depend on your implemented trust matrix.
PUBLIC_BASE_URL=https://localhost:8443
TLS_CA_FILE=/run/secrets/ca.crt
TLS_CERT_FILE=/run/secrets/service.crt
TLS_KEY_FILE=/run/secrets/service.key
BAO_ADDR=https://openbao:8200
BAO_CACERT=/run/secrets/bao-ca.crt
# No root token here. Bootstrap operator -> short-lived service auth -> scoped token.
ARGON2_MEMORY_KIB=19456
ARGON2_ITERATIONS=2
ARGON2_PARALLELISM=1
S3_PRESIGN_TTL=5m
S3_SWEEP_INTERVAL=1m
S3_SWEEP_GRACE=10m
WORKER_CONCURRENCY=1
WORKER_PREFETCH=1
WORKER_CPU_LIMIT=1.00
WORKER_MEM_LIMIT=1G
BROKER_CPU_LIMIT=0.50
BROKER_MEM_LIMIT=384M
PROCESSING_PIPELINE_VERSION=1
PROCESSING_MAX_ATTEMPTS=5
PROCESSING_JOB_TIMEOUT=60s
PROCESSING_LEASE_TTL=90s
PROCESSING_HEARTBEAT=15s
PROCESSING_RETRY_BASE=2s
PROCESSING_RETRY_CAP=60s
OUTBOX_POLL_INTERVAL=1s
OUTBOX_CLAIM_TTL=30s
OUTBOX_BATCH=20
OUTBOX_REAPER_INTERVAL=15s
OUTBOX_RECONCILE_AFTER=5m
OUTBOX_RETENTION=168h
RELAY_CPU_LIMIT=0.25
RELAY_MEM_LIMIT=128M
STAGING_ORPHAN_GRACE=1h
STAGING_FAILED_RETENTION=168h
ADMISSION_MAX_PENDING_PER_USER=20
ADMISSION_MAX_PENDING=200
ADMISSION_MAX_STAGED_BYTES=2147483648
ADMISSION_MAX_CONCURRENT_UPLOADS=8
# Telemetry is part of the full stand, not a plaintext optional profile.
TELEMETRY_ENABLED=1
OTEL_TRACES_SAMPLER=always_on
OTEL_METRIC_EXPORT_INTERVAL=5000
OTEL_COLLECTOR_CPU_LIMIT=0.25
OTEL_COLLECTOR_MEM_LIMIT=192M
PROMETHEUS_CPU_LIMIT=0.50
PROMETHEUS_MEM_LIMIT=320M
LOKI_CPU_LIMIT=0.50
LOKI_MEM_LIMIT=320M
TEMPO_CPU_LIMIT=0.25
TEMPO_MEM_LIMIT=768M
GRAFANA_CPU_LIMIT=0.25
GRAFANA_MEM_LIMIT=512M
+60
View File
@@ -0,0 +1,60 @@
name: Application
on: [push, pull_request]
permissions:
contents: read
env:
GO_VERSION: "1.27.3"
GOLANGCI_LINT_VERSION: v2.15.0
jobs:
quality:
name: Go invariants and static checks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}
cache: true
- run: make sqlc generate-mocks
- run: make lint unit-test
stand:
name: Lab7 verified TLS and retained features
needs: quality
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}
cache: true
- uses: actions/setup-python@v5
with:
python-version: "3.14"
- run: cp .env.example .env
- name: Build, initialize, unseal and start isolated stand
run: make COMPOSE_PROJECT_NAME=mtusi-lab7-ci up
- name: Public HTTPS and trust
run: make COMPOSE_PROJECT_NAME=mtusi-lab7-ci test test-security
- name: Internal identities and permissions
run: make COMPOSE_PROJECT_NAME=mtusi-lab7-ci test-lab7
- name: Preserved asynchronous attachment features
run: make COMPOSE_PROJECT_NAME=mtusi-lab7-ci test-attachments test-presign test-queue test-idempotency
- name: Telemetry configuration and all three signals
run: |
docker compose --project-name mtusi-lab7-ci --env-file .env exec -T prometheus promtool check config /etc/prometheus/prometheus.yml
make COMPOSE_PROJECT_NAME=mtusi-lab7-ci test-observability
- name: Persist data through operator restart
run: make COMPOSE_PROJECT_NAME=mtusi-lab7-ci test-persistence ACTION_SCRIPT=scripts/restart.sh
- name: Publish sanitized evidence
if: always()
uses: actions/upload-artifact@v4
with:
name: lab7-evidence
path: evidence/lab7/
- name: Isolated runner cleanup
if: always()
run: docker compose --project-name mtusi-lab7-ci --env-file .env down -v
+8
View File
@@ -22,8 +22,16 @@ dist/
build/
target/
bin/
/messenger
/s3init
/worker
/relay
/jobctl
obj/
coverage/
coverage.out
coverage.out.tmp
.cache/
*.log
.DS_Store
.idea/
+139
View File
@@ -0,0 +1,139 @@
# yaml-language-server: $schema=https://golangci-lint.run/jsonschema/golangci.v2.13.jsonschema.json
version: "2"
run:
timeout: 5m
tests: true
build-tags:
- integration
linters:
default: standard
enable:
- bodyclose
- contextcheck
- containedctx
- noctx
- rowserrcheck
- sqlclosecheck
- fatcontext
- errorlint
- errname
- nilerr
- nilnil
- asasalint
- durationcheck
- exhaustive
- forcetypeassert
- gocheckcompilerdirectives
- gocritic
- makezero
- musttag
- reassign
- recvcheck
- unparam
- wastedassign
- gosec
- canonicalheader
- copyloopvar
- exptostd
- intrange
- perfsprint
- unconvert
- usestdlibvars
- usetesting
- prealloc
- revive
- misspell
- nolintlint
- gomoddirectives
- sloglint
- depguard
- forbidigo
- testifylint
- thelper
- tparallel
settings:
govet:
enable-all: true
disable:
- fieldalignment
- shadow
gosec:
excludes:
- G404
exhaustive:
default-signifies-exhaustive: true
nolintlint:
require-explanation: true
require-specific: true
sloglint:
attr-only: true
no-global: default
context: scope
testifylint:
enable-all: true
depguard:
rules:
main:
deny:
- pkg: math/rand$
desc: use math/rand/v2
- pkg: github.com/pkg/errors
desc: use stdlib errors and fmt.Errorf with %w
- pkg: io/ioutil
desc: deprecated
domain:
files:
- "**/internal/domain/**"
deny:
- pkg: database/sql
desc: domain must not depend on infrastructure
- pkg: net/http
desc: domain must not depend on transport
forbidigo:
forbid:
- pattern: ^fmt\.Print(f|ln)?$
msg: use slog
- pattern: ^log\.(Print|Fatal|Panic).*$
msg: use slog
exclusions:
generated: lax
presets:
- comments
- std-error-handling
- common-false-positives
rules:
- path: _test\.go
linters:
- gosec
- prealloc
- forcetypeassert
- unparam
formatters:
enable:
- gofumpt
- goimports
settings:
gofumpt:
module-path: mtusi-messenger
goimports:
local-prefixes:
- mtusi-messenger
exclusions:
generated: lax
issues:
max-issues-per-linter: 0
max-same-issues: 0
+38
View File
@@ -0,0 +1,38 @@
#https://vektra.github.io/mockery/latest/configuration/
all: false
dir: "{{.InterfaceDir}}/mocks"
pkgname: mocks
filename: "{{.InterfaceName}}.go"
structname: "Mock{{.InterfaceName}}"
template: testify
formatter: goimports
force-file-write: true
include-auto-generated: false
log-level: info
packages:
mtusi-messenger/internal/platform/cache:
config:
all: true
mtusi-messenger/internal/storage/redis:
config:
all: true
mtusi-messenger/internal/service/user:
config:
all: true
mtusi-messenger/internal/service/chat:
config:
all: true
mtusi-messenger/internal/service/message:
config:
all: true
mtusi-messenger/internal/service/attachment:
config:
all: true
mtusi-messenger/internal/service/session:
config:
all: true
mtusi-messenger/internal/transport/rest:
config:
all: true
+10
View File
@@ -0,0 +1,10 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/ogen-go/ogen/v1.24.0/schemas/ogen.jsonschema.json
# https://ogen.dev/docs/config/
generator:
features:
disable:
- paths/client
- webhooks/client
- webhooks/server
- ogen/otel
+79
View File
@@ -0,0 +1,79 @@
# syntax=docker/dockerfile:1.7
ARG GO_VERSION=1.27.3
ARG ALPINE_VERSION=3.22
ARG GO_IMAGE_DIGEST=sha256:fa3d596898d9f126af7c01d14ee7597bab3f39cccf392ef36b69fe77cd681f18
ARG ALPINE_IMAGE_DIGEST=sha256:49340b346798875b5592e5d420241f2f9cfba5f47e62ebb1446a1b34734e898f
FROM golang:${GO_VERSION}-alpine@${GO_IMAGE_DIGEST} AS build
# Platform of the produced binaries. Left empty on purpose: the classic builder
# does not supply these args, and the previous hardcoded `amd64` default silently
# produced x86-64 binaries inside an arm64 image. Those ran under emulation and
# the Go runtime crashed under load (fatals like "found pointer to free object").
# Empty means "build for the builder's own platform"; cross-compile explicitly:
# docker build --build-arg TARGETARCH=amd64 ...
ARG TARGETOS
ARG TARGETARCH
ARG VERSION=dev
ENV GOTOOLCHAIN=local
WORKDIR /opt
COPY go.mod go.sum ./
RUN go mod download
# The Go compiler spikes well above 1 GiB per package on large dependencies
# (AWS SDK + pgx + Redis), so an unbounded or even -p=2 build OOMs Docker
# Desktop's default VM. Build one package at a time; override with
# --build-arg GOFLAGS=-p=$(nproc) on a bigger host.
ARG GOFLAGS="-p=1"
ENV GOFLAGS=$GOFLAGS
# Keep the compiler well below Docker Desktop's default VM limit.
ENV GOGC=50
ENV GOMEMLIMIT=1GiB
COPY . .
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -trimpath -buildvcs=false -tags timetzdata,nodynamic \
-ldflags="-s -w -X main.version=${VERSION}" \
-o /out/messenger ./cmd/messenger \
&& CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -trimpath -buildvcs=false -tags timetzdata,nodynamic \
-ldflags="-s -w -X main.version=${VERSION}" \
-o /out/migrate ./cmd/migrate \
&& CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -trimpath -buildvcs=false -tags timetzdata,nodynamic \
-ldflags="-s -w -X main.version=${VERSION}" \
-o /out/s3init ./cmd/s3init \
&& CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -trimpath -buildvcs=false -tags timetzdata,nodynamic \
-ldflags="-s -w -X main.version=${VERSION}" \
-o /out/worker ./cmd/worker \
&& CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -trimpath -buildvcs=false -tags timetzdata,nodynamic \
-ldflags="-s -w -X main.version=${VERSION}" \
-o /out/relay ./cmd/relay \
&& CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -trimpath -buildvcs=false -tags timetzdata,nodynamic \
-ldflags="-s -w -X main.version=${VERSION}" \
-o /out/jobctl ./cmd/jobctl
FROM alpine:${ALPINE_VERSION}@${ALPINE_IMAGE_DIGEST} AS runtime
RUN apk add --no-cache ca-certificates wget \
&& addgroup -S -g 10001 app \
&& adduser -S -D -H -u 10001 -G app app
COPY --from=build --chown=10001:10001 /out/messenger /usr/local/bin/messenger
COPY --from=build --chown=10001:10001 /out/migrate /usr/local/bin/migrate
COPY --from=build --chown=10001:10001 /out/s3init /usr/local/bin/s3init
COPY --from=build --chown=10001:10001 /out/worker /usr/local/bin/worker
COPY --from=build --chown=10001:10001 /out/relay /usr/local/bin/relay
COPY --from=build --chown=10001:10001 /out/jobctl /usr/local/bin/jobctl
COPY --chown=10001:10001 config/dev/config.yaml /etc/messenger/dev/config.yaml
USER 10001:10001
EXPOSE 8080
STOPSIGNAL SIGTERM
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
CMD wget -q -O /dev/null http://127.0.0.1:8084/health/live || exit 1
ENTRYPOINT ["/usr/local/bin/messenger"]
+179 -17
View File
@@ -1,13 +1,39 @@
PYTHON ?= python3
GO ?= go
VERSION ?= $(shell git describe --always --dirty 2>/dev/null || echo dev)
BASE_URL ?= https://localhost:8443
CA_FILE ?=
LAB7_SECRETS_DIR ?= secrets/lab7
CA_FILE ?= $(LAB7_SECRETS_DIR)/pki/operator/current/ca.crt
ACTION_SCRIPT ?=
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
PIPELINE ?= 2
ROLE ?= api
CERT_TTL ?= 1h
TLS_ARGS = --ca-file "$(CA_FILE)"
COMPOSE_BIN ?= docker compose
COMPOSE_FILE ?= compose.yaml
ENV_FILE ?= .env
COMPOSE = $(COMPOSE_BIN) --env-file $(ENV_FILE) -f $(COMPOSE_FILE)
PYTHON_LOCAL = env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 $(PYTHON)
LOCAL_BIN := $(CURDIR)/bin
GOLANGCI_LINT_VERSION ?= v2.15.0
GOLANGCI_LINT := $(LOCAL_BIN)/golangci-lint
OPENAPI_SPEC := contracts/openapi.json
OGEN_CONFIG := .ogen.yml
OAS_DIR := internal/transport/rest/internal/oas
.PHONY: help check validate test load test-persistence test-failover test-security
-include .env
export
ENV_KEYS := $(shell sed -n 's/^\([A-Za-z_][A-Za-z0-9_]*\)=.*/\1/p' .env 2>/dev/null)
CLEAN_ENV = env $(addprefix -u ,$(ENV_KEYS))
.DEFAULT_GOAL := help
.PHONY: help check validate prepare build bootstrap up down ps logs test test-security test-lab7 test-pki auth-budget pki-issue pki-renew renew-watch bao-unseal test-attachments test-presign test-queue test-faults test-track2 test-observability test-slo test-idempotency test-lb-readiness load test-persistence test-failover jobs-failed replay reprocess unit-test integration-test test-db-up test-db-down dev-up dev-down dev-reload migrate generate generate-api generate-mocks sqlc cover lint
help:
@echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes"
@echo "prepare/build/up: local PKI, image, operator Bao init/unseal and full verified-TLS stand"
@echo "test/test-security/test-lab7: public HTTPS, external trust, complete internal security probes"
@echo "test-pki: 3 isolated lifecycle experiments; auth-budget: measured Argon2/login budget"
@echo "pki-issue/pki-renew/renew-watch ROLE=api CERT_TTL=1h; bao-unseal: operator recovery"
check:
$(PYTHON) scripts/check_contract.py
@@ -15,19 +41,155 @@ check:
validate:
$(PYTHON) scripts/validate_openapi.py
prepare:
$(PYTHON_LOCAL) scripts/securityctl.py prepare
build: prepare
$(COMPOSE) build api
bootstrap:
$(PYTHON_LOCAL) scripts/securityctl.py bootstrap
up: build
$(PYTHON_LOCAL) scripts/securityctl.py up
down:
$(COMPOSE) down
ps:
$(COMPOSE) ps
logs:
$(COMPOSE) logs -f --tail 100
bao-unseal:
$(PYTHON_LOCAL) scripts/securityctl.py unseal
pki-issue:
$(PYTHON_LOCAL) scripts/securityctl.py issue "$(ROLE)" --ttl "$(CERT_TTL)"
pki-renew:
$(PYTHON_LOCAL) scripts/securityctl.py renew "$(ROLE)" --ttl "$(CERT_TTL)"
renew-watch:
$(PYTHON_LOCAL) scripts/securityctl.py renew "$(ROLE)" --ttl "$(CERT_TTL)" --watch --interval 1
renewer:
$(PYTHON_LOCAL) scripts/lab7_renewer.py --interval 1800 --ttl "$(CERT_TTL)"
renew-once:
$(PYTHON_LOCAL) scripts/lab7_renewer.py --once --ttl "$(CERT_TTL)" --output evidence/lab7/renewer.json
test:
$(PYTHON) tests/smoke.py --base-url "$(BASE_URL)" $(TLS_ARGS)
load:
$(PYTHON) tests/load.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-persistence:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable restart hook"; exit 2)
$(PYTHON) tests/resilience.py --mode persistence --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
test-failover:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
$(PYTHON_LOCAL) tests/smoke.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-security:
$(PYTHON) tests/security.py --base-url "$(BASE_URL)" $(TLS_ARGS)
$(PYTHON_LOCAL) tests/security.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-lab7:
$(PYTHON_LOCAL) student-tests/lab7_security.py --base-url "$(BASE_URL)" $(TLS_ARGS) --output evidence/lab7/security.json
test-pki:
$(PYTHON_LOCAL) scripts/lab7_track2.py --reps 3 --duration 30 --output evidence/lab7/track2.json
auth-budget:
$(PYTHON_LOCAL) scripts/lab7_auth_benchmark.py --api $(TLS_ARGS) --output evidence/lab7/auth.json
test-attachments:
$(PYTHON_LOCAL) student-tests/attachments_negative.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-presign:
$(PYTHON_LOCAL) student-tests/presign_flow.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-queue:
$(PYTHON_LOCAL) student-tests/lab5_async.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-faults:
$(PYTHON_LOCAL) student-tests/lab5_faults.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-track2:
$(PYTHON_LOCAL) scripts/lab5_track2.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-observability:
$(PYTHON_LOCAL) scripts/check_evidence.py
$(PYTHON_LOCAL) scripts/lab6_verify.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-slo:
$(PYTHON_LOCAL) scripts/lab6_track2.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-idempotency:
$(PYTHON_LOCAL) student-tests/idempotency_parallel.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-lb-readiness:
scripts/lb_readiness_check.sh
load:
$(PYTHON_LOCAL) tests/load.py --base-url "$(BASE_URL)" $(TLS_ARGS)
test-persistence:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT=scripts/restart.sh"; exit 2)
$(PYTHON_LOCAL) tests/resilience.py --mode persistence --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
test-failover:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT=scripts/stop-one.sh"; exit 2)
$(PYTHON_LOCAL) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
jobs-failed:
$(PYTHON_LOCAL) scripts/securityctl.py jobctl failed
replay:
@test -n "$(JOB)" || (echo "Set JOB=<job_id>"; exit 2)
$(PYTHON_LOCAL) scripts/securityctl.py jobctl replay -job "$(JOB)"
reprocess:
@test -n "$(ATTACHMENT)" || (echo "Set ATTACHMENT=<attachment_id>"; exit 2)
$(PYTHON_LOCAL) scripts/securityctl.py jobctl reprocess -attachment "$(ATTACHMENT)" -pipeline "$(PIPELINE)"
unit-test:
$(CLEAN_ENV) $(GO) test -race -shuffle=on -count=1 -timeout=5m ./...
integration-test:
$(PYTHON_LOCAL) scripts/lab7_integration.py
test-db-up:
$(COMPOSE_BIN) --env-file $(ENV_FILE) -f docker-compose.local.yaml up -d --no-build postgres redis
test-db-down:
$(COMPOSE_BIN) --env-file $(ENV_FILE) -f docker-compose.local.yaml down
dev-up: build
COMPOSE_FILE=docker-compose.local.yaml $(PYTHON_LOCAL) scripts/securityctl.py up
dev-down:
$(COMPOSE_BIN) --env-file $(ENV_FILE) -f docker-compose.local.yaml down
dev-reload:
COMPOSE_FILE=docker-compose.local.yaml $(PYTHON_LOCAL) scripts/securityctl.py restart
migrate:
$(COMPOSE) run --rm --no-deps migrate
generate: generate-api generate-mocks sqlc
generate-api:
$(GO) run ./tools/specgen
$(GO) tool ogen --config $(OGEN_CONFIG) --target $(OAS_DIR) --package oas --clean .cache/openapi.codegen.json
generate-mocks:
$(GO) tool mockery
sqlc:
$(GO) tool sqlc generate
COVER_EXCLUDE ?= /(mocks|oas)/|/cmd/
cover:
$(CLEAN_ENV) $(GO) test -race -shuffle=on -count=1 -timeout=5m -covermode=atomic -coverprofile=coverage.out.tmp ./...
@grep -Ev '$(COVER_EXCLUDE)' coverage.out.tmp > coverage.out
@rm coverage.out.tmp
$(GO) tool cover -func=coverage.out
$(GOLANGCI_LINT): go.mod Makefile
GOBIN=$(LOCAL_BIN) $(GO) install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@$(GOLANGCI_LINT_VERSION)
lint: $(GOLANGCI_LINT)
$(GOLANGCI_LINT) run -v
+81 -16
View File
@@ -1,37 +1,102 @@
# Отчёт — лабораторная 7
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Кунецкий Владислав, bvt2303
- Целевая оценка: 5; выбран трек 2 (жизненный цикл PKI/secrets)
- Commit/PR: ветка `lab7` в origin; реализация, отчёт и evidence — commit `1b6e96c` поверх skeleton `20afc2c` (ранние `3441f22`, `d923140`, `30b6672` объединены squash при публикации); PR: https://gitea.privcloud.tech/mtusi2026/mtusi2026lab7/pulls/10.
- Использованные источники: README lab7 и `contracts/trust-matrix.md`; OWASP Password Storage и File Upload Cheat Sheets; документация OpenBao (AppRole/cert auth, policies, PKI, production hardening); документация Go `crypto/tls`, `crypto/x509`, `os.Root`; документация HAProxy, RabbitMQ, SeaweedFS, Grafana/Loki/Tempo/Prometheus, OpenTelemetry Collector. ИИ-ассистент использовался для генерации и ревью кода, подготовки проверок и текста отчёта; все команды и наблюдения перепроверены на локальном стенде, секреты в отчёт не включались.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Требования: Docker Desktop/Compose v2, Go 1.27.3, Python 3.9+, `make`, `openssl` в PATH. Внешние сервисы не нужны: все зависимости поднимаются из `compose.yaml`, версии образов закреплены digest.
```sh
cp .env.example .env
make up # prepare -> build -> bootstrap -> up; создаёт локальные PKI/секреты
make test # публичный HTTPS smoke
make test-security # внешний TLS trust + отрицательные проверки
```
`make up` создаёт PKI, backend credentials, init/unseal OpenBao, применяет миграции, инициализирует S3 и поднимает стенд. Секреты и приватные ключи создаются локально; в Git, образы и telemetry они не попадают. CA для тестов: `secrets/lab7/pki/operator/current/ca.crt` (публичный сертификат; приватный ключ остаётся у оператора). Готовность — `GET /health/ready` по `https://localhost:8443` с проверкой CA и hostname.
Дополнительные проверки:
```sh
make test-lab7 # security.json: полный набор отрицательных проверок
make test-pki # 3 изолированных прогона PKI lifecycle (трек 2)
make auth-budget # измеренный бюджет Argon2/login
make renewer # foreground renewal/CRL loop; под supervisor — постоянный
make renew-once # один цикл renew всех ролей + sync-crls (evidence/lab7/renewer.json)
make validate PYTHON=.venv/bin/python
make unit-test integration-test
python3 scripts/lab7_scan.py --output evidence/lab7/scans
```
Остановка: `make down` (volumes сохраняются). Намеренное удаление учебных данных: `make down`, затем `docker compose --env-file .env down --volumes --remove-orphans`, удалить `secrets/lab7` и `~/.local/share/mtusi-lab7/<project>` (root/unseal state). Команда удаляет все данные стенда и необратима.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab6, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Процессы: HAProxy LB (TLS ingress + mTLS к API), две Go API-реплики, worker, relay, PostgreSQL, Redis, RabbitMQ, SeaweedFS, OpenBao, OTel Collector с TLS-gateway, Prometheus, Loki, Tempo, Grafana. Одноразовые `migrate`, `s3init`, `volume-init`, `jobctl` — явные operator jobs.
Клиент проходит TLS с проверкой CA и DNS SAN до LB; LB требует mTLS к каждой API-реплике, Go server требует точный URI `spiffe://mtusi/lb`. Для остальных связей — отдельные identities и проверка SAN/ACL по матрице: PostgreSQL native TLS + SCRAM и раздельные роли, S3 native TLS + SigV4 и role prefixes, RabbitMQ native mTLS с EXTERNAL-пользователями и ACL, Collector mTLS gateway с CN allowlist, Grafana→data sources mTLS с query-only allowlist, Redis TLS + ACL. OpenBao — не dev-mode, persistent raft; приложение аутентифицируется сертификатом, получает собственный KV и собственный PKI issue path; токены 60 s (max 300 s), KV freshness 60 s, admission закрывается за 1 s до ближайшего lease.
Секреты версионируются: immutable `versions/<name>` + атомарный regular-file `current-version`; Go и HAProxy перечитывают leaf/CRL на новых соединениях, trust changes требуют рестарта. `securityctl revoke` после `pki/revoke` явно вызывает OpenBao `GET /pki/crl/rotate` и публикует свежий CRL: `auto_rebuild` сам по себе обновляет только истёкший CRL, поэтому без rotate отозванный serial оставался бы вне bundle. Отдельный offline root/unseal state — у оператора (`~/.local/share/mtusi-lab7/<project>`, 0700/0600). `make renewer` — host-side supervisor loop: каждые 30 минут выпускает новые leaf для всех ролей (service cert auth; operator fallback только при классифицированном expiry, иначе fail loudly), перезагружает native owners и вызывает `sync-crls`; CRL обновляются чаще срока действия, offline root CRL — за сутки до NextUpdate. Если стенд был остановлен дольше TTL, `bootstrap`/`make up` вызывает `recover_transport` и перевыпускает истёкшие role-leaf до старта зависимостей (`expired_identities_renewed`). Loop не монтируется в app-контейнеры, потому что ему нужен Docker CLI для reload native services. Контейнеры: non-root, read_only где совместимо, `no-new-privileges`, drop ALL capabilities (кроме объяснённых), resource limits, отдельные сети `default`/`observability`/`edge`; наружу опубликованы только нужный TLS ingress и локальные admin-интерфейсы.
Из lab6 перенесены API/worker/relay, асинхронные вложения с outbox, идемпотентность, кэш/лимиты и наблюдаемость. В lab7 добавлены: TLS/mTLS на всех границах, OpenBao с cert auth и scoped policies, PKI lifecycle (renewal/expiry/revocation/CA rollover), Argon2id, общий login rate limit, hardening контейнеров, модель угроз и scan triage. Контракт OpenAPI 1.7.0 не менялся; публичные тесты не ослаблялись.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
| Публичный smoke, HTTPS CA/hostname | `make test`; `make test-security` | trusted HTTPS проходит; недоверенный CA отклоняется ошибкой верификации | `evidence/lab7/security.json` (`public_ca_hostname`), `final-verification.txt` |
| Argon2id salt/параметры | `make auth-budget`; `internal/platform/password` | m=19 MiB, t=2, p=1; host-бенчмарк Argon2 p50 14.7 ms, p95 17.0 ms; API login: 120 запросов, p50 5.2 ms, p95 57.6 ms, 429 при исчерпании бюджета; cgroup peak 76–92 MB (after) | `evidence/lab7/auth.json` |
| LB→API обязательный mTLS + identity | `make test-lab7`; `make test-pki` | lb 200; нет cert/чужой CA/неверный URI — TLS rejection; отзыв старого LB отклоняется на свежем соединении | `security.json` (`mtls_and_telemetry_acl`), `track2.json` |
| Backend TLS и минимальные права | `make test-lab7` | PostgreSQL SQLSTATE 42501 для чужих таблиц; S3 GET/PUT/DELETE/COPY по чужому prefix — 403 (worker bucket-level LIST возвращает metadata, задокументировано); Redis foreign key NOPERM; broker EXTERNAL ACL; Bao foreign KV/issuer 403 | `security.json` (`database_privileges`, `s3_prefix_acl`, `redis_tls_cache_acl`, `broker_external_acl`, `bao_api_policy`) |
| OpenBao non-dev, scoped, persistent | `securityctl.py up`, `audit-summary`; `make test-lab7` | root token отозван; app использует cert auth; unseal — у оператора; KV/issuer только свои; audit пишет login/issue/revoke/renew | `security.json` (`bao_api_policy`), `track2.json` (`bootstrap.root_token_revoked`, `actions.bao_audit`), `bao-audit.txt` |
| Контейнеры без privileged/host/socket | `runtime_container_network_isolation` | apps non-root/read_only; из соседнего namespace connection refused; лишние capabilities сняты | `security.json`, `compose.yaml` |
| Rate limit/body limit/ACL/logout/expiry | `make test-lab7`; `make test` | общий 429 + `Retry-After`; чужой чат/вложение недоступны; logout и DB expiry закрывают доступ; forged `X-User-Id` не меняет identity | `security.json` (`shared_rate_limit`, `http_acl_logout_expiry_bounds`, `real_async_pipeline`) |
| Уровень 4: broker и Collector mTLS | `broker_external_acl`, `mtls_and_telemetry_acl` | разные identities; разрешённая роль 200/публикация, запрещённая роль 403, нет cert/чужой CA — TLS rejection | `security.json` |
| Отказ plaintext/чужому CA на backends и telemetry | `make test-lab7` | PostgreSQL/Redis/S3 plaintext-канал отвергнут; S3 с чужим CA — ca_rejected; Loki не содержит sentinel-секретов | `security.json` (`plaintext_refusal`, `credential_log_sentinels`) |
| Уровень 4: rehash, отсутствие утечек, login DoS | `auth-budget`, `credential_log_sentinels`; unit `TestAuthenticateUpgradesHashesWithCAS` | rehash старого hash реализован и покрыт unit-тестом (CAS), runtime-вход с legacy-PHC отдельно не снимался; пароли/токены не в логах; бюджет измерен | `auth.json`, `security.json`, `internal/service/session/service_test.go` |
| Уровень 4: ротация secret/leaf, отзыв, expiry/sealed | `make test-pki`; `make test-lab7` | leaf renewal без пересборки; отзыв старого leaf; sealed Bao закрывает admission; CA prepare→issue→retire с проверкой старой identity | `track2.json` |
| Уровень 4: hardening + scans + threat model | `runtime_container_network_isolation`; `scans/summary.json`; `security/THREAT_MODEL.md` | 13 образов просканированы, все scanner exits 0; достижимых findings в коде приложения нет; остаточные CVE разобраны по применимости | `scans/summary.json`, `security/SCAN_TRIAGE.md`, `THREAT_MODEL.md` |
| Уровень 5, трек 2 | `make test-pki` | 3 изолированных прогона: renewal/expiry/sealed/revocation/CA rollover под нагрузкой; измерены ошибки и восстановление; fallback на plaintext/verify-off отсутствует | `evidence/lab7/track2.json`, `renewer.json`, `bao-audit.txt` |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
Примечание: `native-revocation.json`, `root-crl.json`, `verification.json`, `restart-scope.json`, `failover-scope.json`, `sequential-failover.json`, `track2-ca-before.json` и `track2-renewal-before.json` — pre-final fixtures предыдущих прогонов. Финальные заявления подкреплены `track2.json`, `security.json`, `auth.json`, `renewer.json`, `bao-audit.txt`, `resilience-final.txt` и `final-verification.txt`.
**Гипотеза трека 2:** при короткоживущих leaf и трёхфазном CA rollover стенд не имеет fallback на plaintext/verify-off, а окно отказа ограничено рестартом/сменой trust и восстанавливается без потери сообщений, вложений и sessions. Опровержение — любой успешный запрос без проверки TLS, сохранение старой identity после retire или потеря данных после recovery.
### Наблюдаемые отказы и восстановление
| Воздействие | Ожидание | Наблюдение | Время восстановления / потеря |
| --- | --- | --- | --- |
| Рестарт стенда (`make test-persistence ACTION_SCRIPT=scripts/restart.sh`) | сообщение, полные байты вложения и сессия сохраняются | Full attachment bytes, confirmed message and existing session preserved; имена volumes не изменились | данные не потеряны |
| Остановка одной API-реплики (`make test-failover ACTION_SCRIPT=scripts/stop-one.sh`) | вторая реплика продолжает сервис | 84 успешные записи, 0 failed attempts; message/attachment/session сохранены; stateful-сервисы не перезапускались | first recovery 1.365 s (`evidence/lab7/resilience-final.txt`) |
| Последовательная остановка обеих реплик (`student-tests/replica_failover.py`) | сервис восстанавливается после возврата реплик | 91 и 100 успешных записей, 0 failed; история 191 сообщение, дублей/потерь нет | 1.786 s / 0.889 s |
| Истечение API leaf под нагрузкой | admission закрывается, процесс завершается в bound, затем recovery | ready закрыт; bounded process exit; после issue 1h API восстановлен; сообщение/вложение сохранены | recovery 1.05–2.75 s в прогонах (`track2.json`) |
| Sealed OpenBao | admission закрывается не позже 60 s freshness; после unseal — recovery | ready закрыт; наблюдаемое закрытие 60.5–61.7 s (включая 1 s cadence watchdog); после unseal API/worker/relay восстановлены; сообщение/вложение сохранены | recovery 2.25–6.41 s (`track2.json`) |
| Отзыв старого leaf LB/Collector | свежие и resumption-попытки отклоняются | `tls_certificate_rejected`; новый leaf работает | без потери данных |
| CA rollover prepare→issue→retire | overlap старого trust, затем отказ старой identity и работа нового pipeline | старый leaf проходит в overlap; после retire отклоняется; новый upload/readiness, Bao/Redis/telemetry пути и retained bytes проходят | см. `track2.json` |
| Истечение offline root CRL | CRL обновляется оператором до final 24h | refreshed signed CRL, NextUpdate > 24h; unexpired CRL не перегенерируется; Docker targets не мутируются | без простоя |
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Методика трека 2: `scripts/lab7_track2.py` создаёт отдельный compose-проект на каждый повтор (свои порты и volumes), поднимает полный стенд, гоняет замкнутый HTTPS-клиент 2 rps и выполняет фазы: baseline → leaf renewal (20 s TTL, watcher) → intentional expiry → recovery → sealed Bao → recovery → LB leaf revoke → worker S3 secret rotation → CA prepare/issue/retire. Для каждой фазы фиксируются HTTP-статусы, ошибки по классам, latency, окна отказа и восстановление. После каждой фазы CA rollover сначала прогоняется полный публичный upload pipeline (retry с 15 s timeout, deadline 180 s), затем проверяются контрактные детали. Ограничения: один Docker Desktop host, single-node PostgreSQL/Redis/RabbitMQ/SeaweedFS/OpenBao, замкнутый клиент — это lifecycle-проба, а не capacity benchmark; оба API leaf намеренно истекают (total outage, не A/B failover); TLS tickets/cache отключены.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
Результат трека: `evidence/lab7/track2.json` — 3/3 повторений, `passed: true`. Leaf renewal: по 139–140 запросов на прогон, 0 ошибок, p95 12.9–18.2 ms. Intentional expiry: recovery 1.05–2.75 s; sealed Bao: recovery 2.25–6.41 s; отзыв LB leaf: окна ≈ 0.5 s; CA prepare/issue/retire: 179.4–203.2 s на фазу, 114–153 ошибок на 355–402 запросов (1–2 connection/TLS, 6–8 HTTP 502, 107–144 HTTP 503 при rolling restart/store reload), затем `final_recovery_control` — 60/60 успешных, 0 ошибок. `warm_pipeline` после каждой фазы: 1 попытка, 0 ошибок; retire перепубликовал 19 identities, старый LB отклонён с `TLSV1_ALERT_UNKNOWN_CA`. Ошибки ограничены окнами рестартов и перечислены в `load.observed_outage_windows`; fallback на plaintext/verify-off отсутствует.
Argon2/login: `auth.json` содержит два измерения. Host-бенчмарк `hash_verify`: m=19 MiB, t=2, p=1, p50 14.7 ms, p95 17.0 ms, peak RSS 48.0 MB; это верхняя оценка стоимости Argon2, не API-контейнер. API-проба `api_auth`: 120 запросов, p50 5.2 ms, p95 57.6 ms, 429 при превышении account/source/global лимитов, cgroup peak 76–92 MB (after). Бюджет дериваций ограничен per-process, `ErrBusy` отдаётся как 429 с `Retry-After`, неизвестный пользователь тратит тот же Argon2-бюджет (dummy verify), ошибка необязательного rehash не блокирует успешный вход. Ограничение: host-бенчмарк не воспроизводит CPU-квоту API; API-проба измеряет реальный контейнер, но включает сеть и rate limiter.
Сканирование: `scans/summary.json` — 13 образов, все scanner exits 0; `govulncheck` не нашёл достижимых symbol/package findings в коде приложения (module-only GO-2026-5932 разобран в triage). Остаточные HIGH/MEDIUM относятся к инфраструктурным образам (Grafana/Loki и др.) и ограничены сетевыми границами, non-root, resource limits и отсутствием публикации; детали и residual risk — в `security/SCAN_TRIAGE.md`.
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
- Стенд single-node: нет HA для PostgreSQL/Redis/RabbitMQ/SeaweedFS/OpenBao; проверены отказ API-реплик и рестарты, но не отказ stateful-узла.
- Замкнутый клиент 2 rps и 30 s control-фазы — измерение lifecycle, не пропускная способность; p95 включает окна рестартов.
- Оба API leaf истекают намеренно, поэтому expiry-фаза измеряет fail-closed total outage и recovery, а не переключение между валидными leaf.
- TLS tickets/ClientSessionCache отключены: проверяется fresh connection и явная resumption-попытка, положительный resumption не заявляется.
- CA private key/unseal shares — offline у оператора; автоматизация их хранения не реализована и не заявляется.
- Модель оператора учебная: root key и 2-of-3 unseal shares лежат в одном operator-only каталоге на том же хосте (0700/0600), а operator-сертификат по политике эквивалентен root. Это не HSM и не раздельное хранение долей; отзыв root token — defense-in-depth, а не граница против скомпрометированного оператора.
- `jobctl`/миграции используют DB-роль оператора; отдельная least-privilege migration role — production-улучшение, не заявленное здесь.
- TLS-аудит покрывает заявленные mTLS/ACL-границы, plaintext-refusal для PostgreSQL/Redis/S3 и sentinel-скан Loki; Tempo-атрибуты и historic deleted logs не text-сканируются. Это остаточный пробел, перечисленный в threat model.
- `make renewer` — host-side loop под supervisor; в production его место занимает оркестратор/короткоживущие workload identities.
- Следующие шаги: HA stateful-слоя, динамические DB credentials, автоматическая ротация backend secrets по TTL, более строгий контроль первого bootstrap root of trust.
+237
View File
@@ -0,0 +1,237 @@
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"log/slog"
"os"
"os/signal"
"syscall"
"time"
"github.com/google/uuid"
"mtusi-messenger/internal/config"
"mtusi-messenger/internal/imageproc"
"mtusi-messenger/internal/logger"
"mtusi-messenger/internal/storage/postgres"
)
func main() {
if len(os.Args) < 2 {
usage()
os.Exit(2)
}
switch os.Args[1] {
case "failed", "replay", "reprocess", "duplicate", "inspect":
default:
usage()
os.Exit(2)
}
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
}
func run() error {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
cfg, err := config.LoadOperator()
if err != nil {
return fmt.Errorf("load config: %w", err)
}
log := logger.New(logger.WithLevel(cfg.SlogLevel()), logger.WithFormat(logger.FormatConsole))
slog.SetDefault(log)
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
store, err := postgres.New(ctx, cfg.Postgres, time.Duration(cfg.Idempotency.TTLSeconds)*time.Second)
if err != nil {
return fmt.Errorf("connect database: %w", err)
}
defer store.Close()
repo := store.Pipeline()
switch os.Args[1] {
case "failed":
return runFailed(ctx, repo)
case "replay":
return runReplay(ctx, repo, cfg)
case "reprocess":
return runReprocess(ctx, repo, cfg)
case "duplicate":
return runDuplicate(ctx, repo)
case "inspect":
return runInspect(ctx, repo)
default:
return fmt.Errorf("unknown command %q", os.Args[1])
}
}
func usage() {
fmt.Fprintln(os.Stderr, `usage: jobctl <command> [flags]
failed -limit N
replay -job JOB_ID [-attempts N]
reprocess -attachment ATTACHMENT_ID -pipeline N
duplicate -attachment ATTACHMENT_ID | -job JOB_ID
inspect -attachment ATTACHMENT_ID`)
}
func runFailed(ctx context.Context, repo *postgres.PipelineRepository) error {
flags := flag.NewFlagSet("failed", flag.ContinueOnError)
limit := flags.Int("limit", 50, "maximum jobs")
if err := flags.Parse(os.Args[2:]); err != nil {
return err
}
jobs, err := repo.ListFailedJobs(ctx, *limit)
if err != nil {
return err
}
return printJSON(jobs)
}
func runReplay(ctx context.Context, repo *postgres.PipelineRepository, cfg config.Config) error {
flags := flag.NewFlagSet("replay", flag.ContinueOnError)
jobID := flags.String("job", "", "job id")
attempts := flags.Int("attempts", cfg.Processing.MaxAttempts, "additional attempts")
if err := flags.Parse(os.Args[2:]); err != nil {
return err
}
id, err := uuid.Parse(*jobID)
if err != nil {
return fmt.Errorf("invalid job id: %w", err)
}
job, err := repo.ReplayJob(ctx, id, *attempts)
if err != nil {
return err
}
return printJSON(job)
}
func runReprocess(ctx context.Context, repo *postgres.PipelineRepository, cfg config.Config) error {
flags := flag.NewFlagSet("reprocess", flag.ContinueOnError)
attachmentID := flags.String("attachment", "", "attachment id")
version := flags.Int("pipeline", 2, "target pipeline version")
if err := flags.Parse(os.Args[2:]); err != nil {
return err
}
if _, ok := imageproc.Pipeline(*version); !ok {
return fmt.Errorf("unsupported pipeline version %d", *version)
}
id, err := uuid.Parse(*attachmentID)
if err != nil {
return fmt.Errorf("invalid attachment id: %w", err)
}
jobID, err := uuid.NewV7()
if err != nil {
return err
}
job, err := repo.Reprocess(
ctx,
id,
*version,
cfg.Processing.MaxAttempts,
jobID,
)
if err != nil {
return err
}
return printJSON(job)
}
func runDuplicate(ctx context.Context, repo *postgres.PipelineRepository) error {
flags := flag.NewFlagSet("duplicate", flag.ContinueOnError)
attachmentID := flags.String("attachment", "", "attachment id")
jobID := flags.String("job", "", "specific job id (old pipeline version)")
if err := flags.Parse(os.Args[2:]); err != nil {
return err
}
if *jobID != "" {
id, err := uuid.Parse(*jobID)
if err != nil {
return fmt.Errorf("invalid job id: %w", err)
}
message, err := repo.DuplicateJob(ctx, id)
if err != nil {
return err
}
return printJSON(message)
}
id, err := uuid.Parse(*attachmentID)
if err != nil {
return fmt.Errorf("attachment or job id is required: %w", err)
}
message, err := repo.Duplicate(ctx, id)
if err != nil {
return err
}
return printJSON(message)
}
func runInspect(ctx context.Context, repo *postgres.PipelineRepository) error {
flags := flag.NewFlagSet("inspect", flag.ContinueOnError)
attachmentID := flags.String("attachment", "", "attachment id")
if err := flags.Parse(os.Args[2:]); err != nil {
return err
}
id, err := uuid.Parse(*attachmentID)
if err != nil {
return fmt.Errorf("invalid attachment id: %w", err)
}
attachment, err := repo.GetForID(ctx, id)
if err != nil {
return err
}
variants, err := repo.ListVariants(ctx, id, attachment.PipelineVersion)
if err != nil {
return err
}
var jobOutput any
job, jobErr := repo.GetJobByAttachment(ctx, id, attachment.PipelineVersion)
if jobErr == nil {
jobOutput = job
}
return printJSON(map[string]any{
"attachment": attachment,
"variants": variants,
"job": jobOutput,
})
}
func printJSON(value any) error {
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
if err := encoder.Encode(value); err != nil {
return fmt.Errorf("write output: %w", err)
}
return nil
}
+337
View File
@@ -0,0 +1,337 @@
package main
import (
"context"
"errors"
"fmt"
"log/slog"
"net"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"mtusi-messenger/internal/config"
"mtusi-messenger/internal/health"
"mtusi-messenger/internal/logger"
"mtusi-messenger/internal/platform/bao"
"mtusi-messenger/internal/platform/cache"
"mtusi-messenger/internal/platform/objectstore"
"mtusi-messenger/internal/platform/tlsconfig"
"mtusi-messenger/internal/platform/token"
"mtusi-messenger/internal/service/attachment"
"mtusi-messenger/internal/service/chat"
"mtusi-messenger/internal/service/message"
"mtusi-messenger/internal/service/session"
"mtusi-messenger/internal/service/user"
"mtusi-messenger/internal/storage/postgres"
storageredis "mtusi-messenger/internal/storage/redis"
"mtusi-messenger/internal/telemetry"
"mtusi-messenger/internal/transport/rest"
)
var version = "dev"
func main() {
if err := execute(); err != nil {
logger.New().Error("api stopped with error", slog.Any("error", err))
os.Exit(1)
}
}
func execute() (err error) {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
serviceCtx, manager, err := bao.Start(ctx, "api")
if err != nil {
return fmt.Errorf("start secret manager: %w", err)
}
var shutdown func(context.Context) error
defer func() {
manager.Stop()
shutdownCtx, cancel := manager.ShutdownContext(serviceCtx)
defer cancel()
if shutdown != nil {
err = errors.Join(err, shutdown(shutdownCtx))
}
err = errors.Join(err, manager.Close(serviceCtx))
if errors.Is(context.Cause(serviceCtx), bao.ErrExpired) {
err = errors.Join(err, bao.ErrExpired)
}
}()
cfg, err := config.Load()
if err != nil {
return fmt.Errorf("load config: %w", err)
}
log := logger.New(
logger.WithLevel(cfg.SlogLevel()),
logger.WithFormat(cfg.LoggerFormat()),
)
shutdown, err = telemetry.Setup(serviceCtx, "messenger-api", version)
if err != nil {
return fmt.Errorf("setup telemetry: %w", err)
}
log = telemetry.WrapLogger(log, "messenger-api", cfg.SlogLevel())
slog.SetDefault(log)
return run(
serviceCtx,
cfg,
log,
manager,
)
}
func run(
ctx context.Context,
cfg config.Config,
log *slog.Logger,
manager *bao.Manager,
) (err error) {
store, err := postgres.New(
ctx,
cfg.Postgres,
time.Duration(cfg.Idempotency.TTLSeconds)*time.Second,
postgres.WithQueryTracer(telemetry.NewPGXTracer()),
)
if err != nil {
return fmt.Errorf("connect database: %w", err)
}
defer func() {
err = errors.Join(err, finish(ctx, manager, func(context.Context) error {
store.Close()
return nil
}))
}()
if err := telemetry.RegisterDBPool(store.PoolStats); err != nil {
log.WarnContext(ctx, "register db pool metrics", slog.Any("error", err))
}
if err := telemetry.RegisterProcessMetrics(); err != nil {
log.WarnContext(ctx, "register process metrics", slog.Any("error", err))
}
objectStoreConcrete, err := objectstore.New(objectstore.Config{
Endpoint: cfg.S3.Endpoint,
PublicEndpoint: cfg.S3.PublicEndpoint,
Region: cfg.S3.Region,
Bucket: cfg.S3.Bucket,
AccessKeyID: cfg.S3.AccessKeyID,
SecretAccessKey: cfg.S3.SecretAccessKey,
UsePathStyle: cfg.S3.UsePathStyle,
Timeout: cfg.S3.Timeout,
})
if err != nil {
return fmt.Errorf("connect object store: %w", err)
}
defer objectStoreConcrete.Close()
objectStore := telemetry.WrapStore(objectStoreConcrete)
pipelineRepository := store.Pipeline()
attachmentService := attachment.NewService(
pipelineRepository,
objectStore,
attachment.Config{
MaxUploadBytes: cfg.S3.MaxUploadBytes,
MaxImagePixels: cfg.S3.MaxImagePixels,
PresignTTL: cfg.S3.PresignTTL,
PipelineVersion: cfg.Processing.PipelineVersion,
MaxAttempts: cfg.Processing.MaxAttempts,
MaxPendingPerUser: cfg.Admission.MaxPendingPerUser,
MaxPending: cfg.Admission.MaxPending,
MaxStagedBytes: cfg.Admission.MaxStagedBytes,
MaxConcurrent: cfg.Admission.MaxConcurrent,
},
log,
)
redisCache, err := connectRedis(ctx, cfg.Redis)
if err != nil {
return err
}
defer func() {
err = errors.Join(err, finish(ctx, manager, func(context.Context) error {
return redisCache.Close()
}))
}()
sessionRepo := session.Repository(store.Sessions())
chatRepo := chat.Repository(store.Chats())
pingers := []health.Pinger{store}
if cfg.Redis.Enabled {
if cfg.Cache.Enabled {
distributed, err := cache.NewDistributed(redisCache, cache.DistributedConfig{
LockTTL: cfg.Cache.LockTTL,
WaitTimeout: cfg.Cache.LockWaitTimeout,
}, log)
if err != nil {
return fmt.Errorf("build distributed cache: %w", err)
}
chatRepo = storageredis.NewChatRepository(
store.Chats(),
distributed,
cfg.Cache.ChatListTTL,
log,
)
}
log.InfoContext(ctx, "redis cache enabled", slog.String("host", cfg.Redis.Host), slog.Int("port", cfg.Redis.Port))
}
ttl := time.Duration(cfg.Session.TTLSeconds) * time.Second
sessionService := session.NewService(sessionRepo, token.New(), ttl)
router, err := rest.NewRouter(
rest.NewHandler(
user.NewService(store.Users()),
chat.NewService(chatRepo),
message.NewService(store.Messages(), pipelineRepository),
attachmentService,
health.New(pingers...),
sessionService,
),
rest.NewSecurityHandler(sessionService),
rest.NewExtensions(attachmentService, sessionService, log),
log,
cfg.Server.MaxBodyBytes,
cfg.S3.MaxUploadBytes,
)
if err != nil {
return fmt.Errorf("build router: %w", err)
}
router = rest.WithLoginLimits(router, store, rest.DefaultLoginLimits(), log)
loader, err := tlsconfig.New(tlsconfig.FilesFromEnv())
if err != nil {
return fmt.Errorf("load server identity: %w", err)
}
server := &http.Server{
Addr: net.JoinHostPort(cfg.Server.Host, "8080"),
Handler: admission(ctx, router),
TLSConfig: loader.Server("spiffe://mtusi/lb"),
ReadHeaderTimeout: cfg.Server.ReadHeaderTimeout,
ReadTimeout: cfg.Server.ReadTimeout,
BaseContext: func(net.Listener) context.Context { return ctx },
}
healthServer := localHealthServer(ctx, health.New(pingers...), loader.VerifyTrustExact)
defer func() {
err = errors.Join(err, finish(ctx, manager, func(shutdownCtx context.Context) error {
results := make(chan error, 2)
for _, srv := range []*http.Server{server, healthServer} {
go func() {
shutdownErr := srv.Shutdown(shutdownCtx)
if shutdownErr != nil {
shutdownErr = errors.Join(shutdownErr, srv.Close())
}
results <- shutdownErr
}()
}
return errors.Join(<-results, <-results)
}))
}()
errc := make(chan error, 2)
go func() { errc <- server.ListenAndServeTLS("", "") }()
go func() { errc <- healthServer.ListenAndServe() }()
log.InfoContext(ctx, "server started", slog.String("addr", server.Addr))
select {
case listenErr := <-errc:
return fmt.Errorf("listen: %w", listenErr)
case <-ctx.Done():
return nil
}
}
func connectRedis(ctx context.Context, cfg config.Redis) (cache.LockStore, error) {
if !cfg.Enabled {
return cache.NoOpCache{}, nil
}
client, err := cache.NewRedis(ctx, cache.RedisConfig{
Host: cfg.Host,
Port: cfg.Port,
Username: cfg.Username,
Password: cfg.Password,
DB: cfg.DB,
MaxRetries: cfg.MaxRetries,
DialTimeout: cfg.DialTimeout,
ReadTimeout: cfg.ReadTimeout,
WriteTimeout: cfg.WriteTimeout,
PoolSize: cfg.PoolSize,
})
if err != nil {
return nil, fmt.Errorf("connect redis: %w", err)
}
return client, nil
}
func finish(parent context.Context, manager *bao.Manager, cleanup func(context.Context) error) error {
manager.Stop()
ctx, cancel := manager.ShutdownContext(parent)
defer cancel()
done := make(chan error, 1)
go func() { done <- cleanup(ctx) }()
select {
case err := <-done:
return err
case <-ctx.Done():
return fmt.Errorf("shutdown deadline: %w", ctx.Err())
}
}
func admission(ctx context.Context, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if ctx.Err() != nil && r.URL.Path != "/health/live" {
http.Error(w, "unavailable", http.StatusServiceUnavailable)
return
}
next.ServeHTTP(w, r)
})
}
func localHealthServer(ctx context.Context, checker *health.Checker, trust func() error) *http.Server {
mux := http.NewServeMux()
mux.HandleFunc("GET /health/live", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
fmt.Fprintln(w, `{"status":"ok"}`)
})
mux.HandleFunc("GET /health/ready", func(w http.ResponseWriter, _ *http.Request) {
if ctx.Err() != nil {
http.Error(w, "unavailable", http.StatusServiceUnavailable)
return
}
probeCtx, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
if err := checker.Ping(probeCtx); err != nil || ctx.Err() != nil {
http.Error(w, "unavailable", http.StatusServiceUnavailable)
return
}
if trust != nil {
if err := trust(); err != nil {
http.Error(w, "unavailable", http.StatusServiceUnavailable)
return
}
}
w.Header().Set("Content-Type", "application/json")
fmt.Fprintln(w, `{"status":"ok"}`)
})
return &http.Server{
Addr: "127.0.0.1:8084",
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 5 * time.Second,
WriteTimeout: 5 * time.Second,
BaseContext: func(net.Listener) context.Context { return ctx },
}
}
+93
View File
@@ -0,0 +1,93 @@
package main
import (
"context"
"database/sql"
"errors"
"fmt"
"log/slog"
"os"
"os/signal"
"syscall"
"github.com/jackc/pgx/v5/stdlib"
"github.com/pressly/goose/v3"
"mtusi-messenger/internal/config"
"mtusi-messenger/internal/logger"
"mtusi-messenger/internal/storage/postgres"
"mtusi-messenger/internal/storage/postgres/migrations"
)
func main() {
log := logger.New()
if err := run(os.Args[1:]); err != nil {
log.Error("migrate failed", slog.Any("error", err))
os.Exit(1)
}
}
func run(args []string) (err error) {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
cfg, err := config.LoadOperator()
if err != nil {
return fmt.Errorf("load config: %w", err)
}
slog.SetDefault(logger.New(
logger.WithLevel(cfg.SlogLevel()),
logger.WithFormat(cfg.LoggerFormat()),
))
command := "up"
if len(args) > 0 {
command = args[0]
}
poolCfg, err := postgres.PoolConfig(cfg.Postgres)
if err != nil {
return fmt.Errorf("database config: %w", err)
}
db := stdlib.OpenDB(*poolCfg.ConnConfig)
defer func() { err = errors.Join(err, db.Close()) }()
pingCtx, cancel := context.WithTimeout(ctx, cfg.Postgres.ConnectTimeout)
defer cancel()
if err := db.PingContext(pingCtx); err != nil {
return fmt.Errorf("ping database: %w", err)
}
goose.SetBaseFS(migrations.FS)
if err := goose.SetDialect("postgres"); err != nil {
return fmt.Errorf("set dialect: %w", err)
}
if err := goCommand(ctx, db, command); err != nil {
return err
}
return nil
}
func goCommand(ctx context.Context, db *sql.DB, command string) error {
switch command {
case "up":
return goose.UpContext(ctx, db, ".")
case "up-by-one":
return goose.UpByOneContext(ctx, db, ".")
case "down":
return goose.DownContext(ctx, db, ".")
case "status":
return goose.StatusContext(ctx, db, ".")
case "version":
return goose.VersionContext(ctx, db, ".")
case "redo":
return goose.RedoContext(ctx, db, ".")
default:
return errors.New("unknown command: " + command + " (up, up-by-one, down, status, version, redo)")
}
}
+377
View File
@@ -0,0 +1,377 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net"
"net/http"
"os"
"os/signal"
"sync"
"syscall"
"time"
"mtusi-messenger/internal/broker"
"mtusi-messenger/internal/config"
"mtusi-messenger/internal/domain"
"mtusi-messenger/internal/logger"
"mtusi-messenger/internal/outbox"
"mtusi-messenger/internal/platform/bao"
"mtusi-messenger/internal/storage/postgres"
"mtusi-messenger/internal/telemetry"
)
const maxUnpublishedAge = 10 * time.Minute
var version = "dev"
func main() {
if err := execute(); err != nil {
logger.New().Error("relay stopped with error", slog.Any("error", err))
os.Exit(1)
}
}
func execute() (err error) {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
serviceCtx, manager, err := bao.Start(ctx, "relay")
if err != nil {
return fmt.Errorf("start secret manager: %w", err)
}
var shutdown func(context.Context) error
defer func() {
manager.Stop()
shutdownCtx, cancel := manager.ShutdownContext(serviceCtx)
defer cancel()
if shutdown != nil {
err = errors.Join(err, shutdown(shutdownCtx))
}
err = errors.Join(err, manager.Close(serviceCtx))
if errors.Is(context.Cause(serviceCtx), bao.ErrExpired) {
err = errors.Join(err, bao.ErrExpired)
}
}()
cfg, err := config.Load()
if err != nil {
return fmt.Errorf("load config: %w", err)
}
log := logger.New(
logger.WithLevel(cfg.SlogLevel()),
logger.WithFormat(cfg.LoggerFormat()),
)
shutdown, err = telemetry.Setup(serviceCtx, "messenger-relay", version)
if err != nil {
return fmt.Errorf("setup telemetry: %w", err)
}
log = telemetry.WrapLogger(log, "messenger-relay", cfg.SlogLevel())
slog.SetDefault(log)
return run(
serviceCtx,
cfg,
log,
manager,
)
}
type publisher struct {
cfg broker.Config
mu sync.Mutex
conn *broker.Client
isClosed bool
}
func (p *publisher) client(ctx context.Context) (*broker.Client, error) {
p.mu.Lock()
defer p.mu.Unlock()
if p.isClosed || ctx.Err() != nil {
return nil, context.Canceled
}
if p.conn != nil && p.conn.Connected() {
return p.conn, nil
}
if p.conn != nil {
if err := p.conn.Close(); err != nil {
return nil, fmt.Errorf("close disconnected broker: %w", err)
}
p.conn = nil
}
conn, err := broker.Dial(p.cfg)
if err != nil {
return nil, err
}
if ctx.Err() != nil {
return nil, errors.Join(context.Canceled, conn.Close())
}
p.conn = conn
return conn, nil
}
func (p *publisher) PublishConfirmed(
ctx context.Context,
routingKey string,
message domain.JobMessage,
) error {
conn, err := p.client(ctx)
if err != nil {
return err
}
if err := conn.PublishConfirmed(ctx, routingKey, message); err != nil {
p.mu.Lock()
if p.conn == conn {
p.conn = nil
}
p.mu.Unlock()
return errors.Join(err, conn.Close())
}
return nil
}
func (p *publisher) Connect(ctx context.Context) error {
_, err := p.client(ctx)
return err
}
func (p *publisher) Connected() bool {
p.mu.Lock()
defer p.mu.Unlock()
return !p.isClosed && p.conn != nil && p.conn.Connected()
}
func (p *publisher) Close() error {
p.mu.Lock()
p.isClosed = true
conn := p.conn
p.conn = nil
p.mu.Unlock()
if conn != nil {
return conn.Close()
}
return nil
}
func brokerConfig(cfg config.Config) broker.Config {
return broker.Config{
URL: cfg.Broker.URL,
Exchange: cfg.Broker.Exchange,
Queue: cfg.Broker.Queue,
ConnectTimeout: cfg.Broker.ConnectTimeout,
ConfirmTimeout: cfg.Broker.ConfirmTimeout,
Heartbeat: cfg.Broker.Heartbeat,
Prefetch: cfg.Broker.Prefetch,
Role: "relay",
Producer: true,
}
}
func run(
ctx context.Context,
cfg config.Config,
log *slog.Logger,
manager *bao.Manager,
) (err error) {
if cfg.Broker.URL == "" {
return errors.New("broker url is required for relay")
}
if err := requireLoopback(cfg.Relay.HealthAddr); err != nil {
return err
}
store, err := postgres.New(
ctx,
cfg.Postgres,
time.Duration(cfg.Idempotency.TTLSeconds)*time.Second,
postgres.WithQueryTracer(telemetry.NewPGXTracer()),
)
if err != nil {
return fmt.Errorf("connect database: %w", err)
}
defer func() {
err = errors.Join(err, finish(ctx, manager, func(context.Context) error {
store.Close()
return nil
}))
}()
if err := telemetry.RegisterDBPool(store.PoolStats); err != nil {
log.WarnContext(ctx, "register db pool metrics", slog.Any("error", err))
}
if err := telemetry.RegisterProcessMetrics(); err != nil {
log.WarnContext(ctx, "register process metrics", slog.Any("error", err))
}
pub := &publisher{cfg: brokerConfig(cfg)}
closeDone := make(chan error, 1)
stopClose := context.AfterFunc(ctx, func() { closeDone <- pub.Close() })
defer stopClose()
defer func() {
err = errors.Join(err, finish(ctx, manager, func(shutdownCtx context.Context) error {
select {
case closeErr := <-closeDone:
return closeErr
case <-shutdownCtx.Done():
return shutdownCtx.Err()
}
}))
}()
if err := pub.Connect(ctx); err != nil {
log.WarnContext(ctx, "broker is not reachable yet; relay will retry", slog.Any("error", err))
}
relay := outbox.New(
store.Pipeline(),
pub,
outbox.Config{
Owner: "relay-" + cfg.Broker.Queue,
PollInterval: cfg.Outbox.PollInterval,
ClaimTTL: cfg.Outbox.ClaimTTL,
Batch: cfg.Outbox.Batch,
ReaperInterval: cfg.Outbox.ReaperInterval,
ReconcileAfter: cfg.Outbox.ReconcileAfter,
Retention: cfg.Outbox.Retention,
},
log,
)
if err := telemetry.RegisterRelayMetrics(store.Pipeline().OldestActiveJobAge, relay.Health, func(ctx context.Context) (int64, error) {
return store.Pipeline().CountOverdueJobs(ctx, 30*time.Second)
}); err != nil {
log.WarnContext(ctx, "register relay metrics", slog.Any("error", err))
}
relayDone := make(chan struct{})
go func() {
defer close(relayDone)
relay.Run(ctx)
}()
server := healthServer(
ctx,
cfg.Relay.HealthAddr,
store,
pub,
relay,
)
serverDone := make(chan error, 1)
go func() { serverDone <- server.ListenAndServe() }()
defer func() {
err = errors.Join(err, finish(ctx, manager, func(shutdownCtx context.Context) error {
serverErr := server.Shutdown(shutdownCtx)
if serverErr != nil {
serverErr = errors.Join(serverErr, server.Close())
}
select {
case <-relayDone:
return serverErr
case <-shutdownCtx.Done():
return errors.Join(serverErr, shutdownCtx.Err())
}
}))
}()
select {
case <-ctx.Done():
return nil
case serverErr := <-serverDone:
return fmt.Errorf("health listener: %w", serverErr)
}
}
func healthServer(
serviceCtx context.Context,
addr string,
store *postgres.Store,
pub *publisher,
relay *outbox.Relay,
) *http.Server {
mux := http.NewServeMux()
mux.HandleFunc("GET /health/live", func(w http.ResponseWriter, _ *http.Request) {
writeHealth(w, http.StatusOK, map[string]any{"status": "ok"})
})
mux.HandleFunc("GET /health/ready", func(w http.ResponseWriter, _ *http.Request) {
if serviceCtx.Err() != nil {
writeHealth(w, http.StatusServiceUnavailable, map[string]any{"status": "unavailable"})
return
}
ctx, cancel := context.WithTimeout(serviceCtx, 3*time.Second)
defer cancel()
if err := store.Ping(ctx); err != nil {
writeHealth(w, http.StatusServiceUnavailable, map[string]any{"status": "unavailable"})
return
}
if !pub.Connected() {
writeHealth(w, http.StatusServiceUnavailable, map[string]any{"status": "unavailable"})
return
}
oldest, err := relay.Health(ctx)
if err != nil {
writeHealth(w, http.StatusServiceUnavailable, map[string]any{"status": "unavailable"})
return
}
if oldest > maxUnpublishedAge || serviceCtx.Err() != nil {
writeHealth(w, http.StatusServiceUnavailable, map[string]any{
"status": "unavailable",
"outbox_oldest_seconds": int(oldest.Seconds()),
})
return
}
writeHealth(w, http.StatusOK, map[string]any{
"status": "ok",
"outbox_oldest_seconds": int(oldest.Seconds()),
})
})
return &http.Server{
Addr: addr,
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 5 * time.Second,
WriteTimeout: 5 * time.Second,
BaseContext: func(net.Listener) context.Context { return serviceCtx },
}
}
func writeHealth(w http.ResponseWriter, status int, body map[string]any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(body)
}
func requireLoopback(addr string) error {
host, _, err := net.SplitHostPort(addr)
if err != nil {
return fmt.Errorf("health listener address: %w", err)
}
ip := net.ParseIP(host)
if ip == nil || !ip.IsLoopback() {
return errors.New("health listener must bind a loopback ip")
}
return nil
}
func finish(parent context.Context, manager *bao.Manager, cleanup func(context.Context) error) error {
manager.Stop()
ctx, cancel := manager.ShutdownContext(parent)
defer cancel()
done := make(chan error, 1)
go func() { done <- cleanup(ctx) }()
select {
case err := <-done:
return err
case <-ctx.Done():
return fmt.Errorf("shutdown deadline: %w", ctx.Err())
}
}
+197
View File
@@ -0,0 +1,197 @@
// Command s3init idempotently creates the private attachment bucket using
// operator-only file credentials over the shared TLS trust boundary.
package main
import (
"context"
"errors"
"fmt"
"log/slog"
"net"
"net/http"
"net/url"
"os"
"os/signal"
"strings"
"syscall"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/smithy-go"
"mtusi-messenger/internal/platform/secretfile"
"mtusi-messenger/internal/platform/tlsconfig"
)
const bootstrapTimeout = 120 * time.Second
type settings struct {
Endpoint string
Region string
Bucket string
AccessKey string
SecretKey string
}
func main() {
log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
if err := run(log); err != nil {
log.Error("s3 bootstrap failed", slog.Any("error", err))
os.Exit(1)
}
}
func run(log *slog.Logger) error {
cfg, err := loadSettings()
if err != nil {
return err
}
signalCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
ctx, cancel := context.WithTimeout(signalCtx, bootstrapTimeout)
defer cancel()
client, transport, err := newClient(cfg)
if err != nil {
return err
}
defer transport.CloseIdleConnections()
if err := waitForBucket(ctx, client, cfg.Bucket); err != nil {
return err
}
log.Info("s3 bucket ready",
slog.String("endpoint", cfg.Endpoint),
slog.String("bucket", cfg.Bucket),
)
return nil
}
func loadSettings() (settings, error) {
cfg := settings{
Endpoint: strings.TrimSpace(os.Getenv("S3_ENDPOINT")),
Region: strings.TrimSpace(os.Getenv("S3_REGION")),
Bucket: strings.TrimSpace(os.Getenv("S3_BUCKET")),
}
endpoint, err := url.Parse(cfg.Endpoint)
if err != nil || endpoint.Scheme != "https" || endpoint.Hostname() == "" || endpoint.User != nil {
return settings{}, errors.New("s3 endpoint must be an https url without credentials")
}
if endpoint.RawQuery != "" || endpoint.Fragment != "" {
return settings{}, errors.New("s3 endpoint must not contain a query or fragment")
}
if cfg.Region == "" || cfg.Bucket == "" {
return settings{}, errors.New("s3 region and bucket are required")
}
cfg.AccessKey, err = operatorSecret("S3_ACCESS_KEY_ID_FILE")
if err != nil {
return settings{}, err
}
cfg.SecretKey, err = operatorSecret("S3_SECRET_ACCESS_KEY_FILE")
if err != nil {
return settings{}, err
}
return cfg, nil
}
func newClient(cfg settings) (*s3.Client, *http.Transport, error) {
clientTLS, err := tlsconfig.NewClientFromEnv("")
if err != nil {
return nil, nil, fmt.Errorf("configure object store tls: %w", err)
}
transport := &http.Transport{
DialContext: (&net.Dialer{
Timeout: 3 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
TLSClientConfig: clientTLS,
TLSHandshakeTimeout: 3 * time.Second,
ResponseHeaderTimeout: 10 * time.Second,
IdleConnTimeout: 30 * time.Second,
}
client := s3.New(s3.Options{
Region: cfg.Region,
BaseEndpoint: aws.String(cfg.Endpoint),
UsePathStyle: true,
Credentials: credentials.NewStaticCredentialsProvider(cfg.AccessKey, cfg.SecretKey, ""),
HTTPClient: &http.Client{
Transport: transport,
Timeout: 15 * time.Second,
CheckRedirect: func(*http.Request, []*http.Request) error {
return errors.New("object store redirects are not permitted")
},
},
})
return client, transport, nil
}
func waitForBucket(ctx context.Context, client *s3.Client, bucket string) error {
for {
_, err := client.HeadBucket(ctx, &s3.HeadBucketInput{Bucket: aws.String(bucket)})
if err == nil {
return nil
}
if !isNotFound(err) {
return fmt.Errorf("head bucket: %w", err)
}
if _, err := client.CreateBucket(ctx, &s3.CreateBucketInput{Bucket: aws.String(bucket)}); err == nil {
return nil
} else if isBucketExists(err) {
return nil
} else if !isNotFound(err) {
return fmt.Errorf("create bucket: %w", err)
}
select {
case <-ctx.Done():
return fmt.Errorf("wait for object store: %w", ctx.Err())
case <-time.After(time.Second):
}
}
}
func isNotFound(err error) bool {
var apiErr smithy.APIError
if errors.As(err, &apiErr) {
switch apiErr.ErrorCode() {
case "NotFound", "NoSuchBucket", "NoSuchKey":
return true
}
}
return false
}
func isBucketExists(err error) bool {
var apiErr smithy.APIError
if errors.As(err, &apiErr) {
switch apiErr.ErrorCode() {
case "BucketAlreadyExists", "BucketAlreadyOwnedByYou":
return true
}
}
return false
}
func operatorSecret(env string) (string, error) {
path := strings.TrimSpace(os.Getenv(env))
if path == "" {
return "", fmt.Errorf("%s is required", env)
}
value, err := secretfile.Read(path)
if err != nil {
return "", fmt.Errorf("read %s: %w", env, err)
}
secret := strings.TrimSpace(string(value))
if secret == "" {
return "", fmt.Errorf("%s is empty", env)
}
return secret, nil
}
+366
View File
@@ -0,0 +1,366 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net"
"net/http"
"os"
"os/signal"
"sync"
"syscall"
"time"
"mtusi-messenger/internal/broker"
"mtusi-messenger/internal/config"
"mtusi-messenger/internal/logger"
"mtusi-messenger/internal/platform/bao"
"mtusi-messenger/internal/platform/objectstore"
"mtusi-messenger/internal/service/attachment"
"mtusi-messenger/internal/storage/postgres"
"mtusi-messenger/internal/telemetry"
"mtusi-messenger/internal/worker"
)
var version = "dev"
func main() {
if err := execute(); err != nil {
logger.New().Error("worker stopped with error", slog.Any("error", err))
os.Exit(1)
}
}
func execute() (err error) {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
serviceCtx, manager, err := bao.Start(ctx, "worker")
if err != nil {
return fmt.Errorf("start secret manager: %w", err)
}
var shutdown func(context.Context) error
defer func() {
manager.Stop()
shutdownCtx, cancel := manager.ShutdownContext(serviceCtx)
defer cancel()
if shutdown != nil {
err = errors.Join(err, shutdown(shutdownCtx))
}
err = errors.Join(err, manager.Close(serviceCtx))
if errors.Is(context.Cause(serviceCtx), bao.ErrExpired) {
err = errors.Join(err, bao.ErrExpired)
}
}()
cfg, err := config.Load()
if err != nil {
return fmt.Errorf("load config: %w", err)
}
log := logger.New(
logger.WithLevel(cfg.SlogLevel()),
logger.WithFormat(cfg.LoggerFormat()),
)
shutdown, err = telemetry.Setup(serviceCtx, "messenger-worker", version)
if err != nil {
return fmt.Errorf("setup telemetry: %w", err)
}
log = telemetry.WrapLogger(log, "messenger-worker", cfg.SlogLevel())
slog.SetDefault(log)
return run(
serviceCtx,
cfg,
log,
manager,
)
}
func run(
ctx context.Context,
cfg config.Config,
log *slog.Logger,
manager *bao.Manager,
) (err error) {
if cfg.Broker.URL == "" {
return errors.New("broker url is required for worker")
}
if err := requireLoopback(cfg.Worker.HealthAddr); err != nil {
return err
}
store, err := postgres.New(
ctx,
cfg.Postgres,
time.Duration(cfg.Idempotency.TTLSeconds)*time.Second,
postgres.WithQueryTracer(telemetry.NewPGXTracer()),
)
if err != nil {
return fmt.Errorf("connect database: %w", err)
}
defer func() {
err = errors.Join(err, finish(ctx, manager, func(context.Context) error {
store.Close()
return nil
}))
}()
if err := telemetry.RegisterDBPool(store.PoolStats); err != nil {
log.WarnContext(ctx, "register db pool metrics", slog.Any("error", err))
}
if err := telemetry.RegisterProcessMetrics(); err != nil {
log.WarnContext(ctx, "register process metrics", slog.Any("error", err))
}
objectStoreConcrete, err := objectstore.New(objectstore.Config{
Endpoint: cfg.S3.Endpoint,
PublicEndpoint: cfg.S3.PublicEndpoint,
Region: cfg.S3.Region,
Bucket: cfg.S3.Bucket,
AccessKeyID: cfg.S3.AccessKeyID,
SecretAccessKey: cfg.S3.SecretAccessKey,
UsePathStyle: cfg.S3.UsePathStyle,
Timeout: cfg.S3.Timeout,
})
if err != nil {
return fmt.Errorf("connect object store: %w", err)
}
defer objectStoreConcrete.Close()
objectStore := telemetry.WrapStore(objectStoreConcrete)
workerInstance := worker.New(
store.Pipeline(),
objectStore,
worker.Config{
Concurrency: cfg.Worker.Concurrency,
LeaseTTL: cfg.Processing.LeaseTTL,
Heartbeat: cfg.Processing.Heartbeat,
JobTimeout: cfg.Processing.JobTimeout,
RetryBase: cfg.Processing.RetryBase,
RetryCap: cfg.Processing.RetryCap,
MaxUploadBytes: cfg.S3.MaxUploadBytes,
MaxImagePixels: cfg.S3.MaxImagePixels,
},
log,
)
if err := telemetry.RegisterWorkerMetrics(workerInstance.Busy, int64(cfg.Worker.Concurrency)); err != nil {
log.WarnContext(ctx, "register worker metrics", slog.Any("error", err))
}
sweeper := attachment.NewSweeper(
store.Pipeline(),
objectStore,
store,
cfg.S3.SweepGrace,
cfg.Staging.OrphanGrace,
cfg.Staging.FailedRetention,
cfg.S3.SweepInterval,
log,
)
sweepDone := make(chan struct{})
go func() {
defer close(sweepDone)
sweeper.Run(ctx)
}()
consumerDone := make(chan error, 1)
go func() {
consumerDone <- consumeWithReconnect(
ctx,
cfg,
workerInstance,
log,
)
}()
server := healthServer(
ctx,
cfg.Worker.HealthAddr,
store,
objectStore,
workerInstance,
)
serverDone := make(chan error, 1)
go func() { serverDone <- server.ListenAndServe() }()
defer func() {
err = errors.Join(err, finish(ctx, manager, func(shutdownCtx context.Context) error {
serverErr := server.Shutdown(shutdownCtx)
if serverErr != nil {
serverErr = errors.Join(serverErr, server.Close())
}
select {
case <-sweepDone:
case <-shutdownCtx.Done():
return errors.Join(serverErr, shutdownCtx.Err())
}
select {
case consumerErr := <-consumerDone:
return errors.Join(serverErr, consumerErr)
case <-shutdownCtx.Done():
return errors.Join(serverErr, shutdownCtx.Err())
}
}))
}()
select {
case <-ctx.Done():
return nil
case serverErr := <-serverDone:
return fmt.Errorf("health listener: %w", serverErr)
}
}
func consumeWithReconnect(
ctx context.Context,
cfg config.Config,
workerInstance *worker.Worker,
log *slog.Logger,
) error {
for ctx.Err() == nil {
client, err := broker.Dial(brokerConfig(cfg))
if err != nil {
log.WarnContext(ctx, "broker dial failed, retrying", slog.Any("error", err))
if !sleepCtx(ctx, 2*time.Second) {
return nil
}
continue
}
var closeOnce sync.Once
var closeErr error
closeClient := func() {
closeOnce.Do(func() { closeErr = client.Close() })
}
stopClose := context.AfterFunc(ctx, closeClient)
err = workerInstance.Run(ctx, client)
stopClose()
closeClient()
if closeErr != nil && ctx.Err() == nil {
log.WarnContext(ctx, "close broker failed", slog.Any("error", closeErr))
}
if ctx.Err() != nil {
return closeErr
}
if err != nil {
log.WarnContext(ctx, "consumer stopped, reconnecting", slog.Any("error", err))
}
if !sleepCtx(ctx, 2*time.Second) {
return nil
}
}
return nil
}
func sleepCtx(ctx context.Context, d time.Duration) bool {
timer := time.NewTimer(d)
defer timer.Stop()
select {
case <-ctx.Done():
return false
case <-timer.C:
return true
}
}
func brokerConfig(cfg config.Config) broker.Config {
return broker.Config{
URL: cfg.Broker.URL,
Exchange: cfg.Broker.Exchange,
Queue: cfg.Broker.Queue,
ConnectTimeout: cfg.Broker.ConnectTimeout,
ConfirmTimeout: cfg.Broker.ConfirmTimeout,
Heartbeat: cfg.Broker.Heartbeat,
Prefetch: cfg.Worker.Prefetch,
Role: "worker",
Producer: false,
}
}
func healthServer(
serviceCtx context.Context,
addr string,
store *postgres.Store,
objectStore objectstore.Store,
workerInstance *worker.Worker,
) *http.Server {
mux := http.NewServeMux()
mux.HandleFunc("GET /health/live", func(w http.ResponseWriter, _ *http.Request) {
writeHealth(w, http.StatusOK, map[string]any{"status": "ok"})
})
mux.HandleFunc("GET /health/ready", func(w http.ResponseWriter, _ *http.Request) {
if serviceCtx.Err() != nil {
writeHealth(w, http.StatusServiceUnavailable, map[string]any{"status": "unavailable"})
return
}
ctx, cancel := context.WithTimeout(serviceCtx, 5*time.Second)
defer cancel()
if err := store.Ping(ctx); err != nil {
writeHealth(w, http.StatusServiceUnavailable, map[string]any{"status": "unavailable"})
return
}
if err := objectStore.Ping(ctx); err != nil {
writeHealth(w, http.StatusServiceUnavailable, map[string]any{"status": "unavailable"})
return
}
running, connected := workerInstance.Ready()
if !running || !connected || serviceCtx.Err() != nil {
writeHealth(w, http.StatusServiceUnavailable, map[string]any{"status": "unavailable"})
return
}
writeHealth(w, http.StatusOK, map[string]any{"status": "ok"})
})
return &http.Server{
Addr: addr,
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 5 * time.Second,
WriteTimeout: 6 * time.Second,
BaseContext: func(net.Listener) context.Context { return serviceCtx },
}
}
func writeHealth(w http.ResponseWriter, status int, body map[string]any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(body)
}
func requireLoopback(addr string) error {
host, _, err := net.SplitHostPort(addr)
if err != nil {
return fmt.Errorf("health listener address: %w", err)
}
ip := net.ParseIP(host)
if ip == nil || !ip.IsLoopback() {
return errors.New("health listener must bind a loopback ip")
}
return nil
}
func finish(parent context.Context, manager *bao.Manager, cleanup func(context.Context) error) error {
manager.Stop()
ctx, cancel := manager.ShutdownContext(parent)
defer cancel()
done := make(chan error, 1)
go func() { done <- cleanup(ctx) }()
select {
case err := <-done:
return err
case <-ctx.Done():
return fmt.Errorf("shutdown deadline: %w", ctx.Err())
}
}
+867
View File
@@ -0,0 +1,867 @@
# Secure lab7 cutover. The default project name retains the lab6 data volumes.
# Fresh verification overrides COMPOSE_PROJECT_NAME and LAB7_SECRETS_DIR.
name: mtusi-messenger-lab6
x-log: &log
driver: json-file
options:
max-size: "10m"
max-file: "3"
x-hardened: &hardened
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
read_only: true
group_add: ["10001"]
tmpfs:
- /tmp:rw,noexec,nosuid,size=128m,mode=1777
logging: *log
stop_grace_period: 40s
restart: unless-stopped
x-app: &app
<<: *hardened
image: mtusi-messenger:lab7
user: "10001:10001"
x-tls-environment: &tls-environment
TLS_CA_FILE: /run/identity/current/ca.crt
TLS_CERT_FILE: /run/identity/current/tls.crt
TLS_KEY_FILE: /run/identity/current/tls.key
TLS_CRL_FILE: /run/identity/current/ca.crl
TLS_VERSION_FILE: /run/identity/current-version
x-app-environment: &app-environment
<<: *tls-environment
APP_CONFIG: /etc/messenger/dev/config.yaml
BAO_ADDR: https://openbao:8200
POSTGRES_HOST: postgres
POSTGRES_PORT: "5432"
POSTGRES_DATABASE: messenger
POSTGRES_SSL_MODE: verify-full
POSTGRES_MAX_CONNS: "${POSTGRES_MAX_CONNS:-10}"
POSTGRES_MAX_CONN_LIFETIME: 30s
POSTGRES_MAX_CONN_IDLE_TIME: 10s
POSTGRES_CONNECT_TIMEOUT: 5s
POSTGRES_QUERY_TIMEOUT: 5s
S3_ENDPOINT: https://seaweedfs:8333
S3_PUBLIC_ENDPOINT: "https://localhost:${S3_PORT:-8333}"
S3_REGION: us-east-1
S3_BUCKET: "${S3_BUCKET:-messenger-attachments}"
S3_USE_PATH_STYLE: "true"
S3_TIMEOUT: 30s
S3_PRESIGN_TTL: "${S3_PRESIGN_TTL:-5m}"
S3_SWEEP_INTERVAL: "${S3_SWEEP_INTERVAL:-1m}"
S3_SWEEP_GRACE: "${S3_SWEEP_GRACE:-10m}"
MAX_UPLOAD_BYTES: "${MAX_UPLOAD_BYTES:-10485760}"
MAX_IMAGE_PIXELS: "${MAX_IMAGE_PIXELS:-40000000}"
SESSION_TTL_SECONDS: "${SESSION_TTL_SECONDS:-3600}"
IDEMPOTENCY_TTL_SECONDS: "${IDEMPOTENCY_TTL_SECONDS:-86400}"
BROKER_URL: amqps://broker:5671/messenger
BROKER_EXCHANGE: attachments.jobs
BROKER_QUEUE: attachment.process.v1
BROKER_CONNECT_TIMEOUT: 5s
BROKER_CONFIRM_TIMEOUT: 5s
BROKER_HEARTBEAT: 10s
PROCESSING_PIPELINE_VERSION: "${PROCESSING_PIPELINE_VERSION:-1}"
PROCESSING_MAX_ATTEMPTS: "${PROCESSING_MAX_ATTEMPTS:-5}"
PROCESSING_JOB_TIMEOUT: "${PROCESSING_JOB_TIMEOUT:-60s}"
PROCESSING_LEASE_TTL: "${PROCESSING_LEASE_TTL:-90s}"
PROCESSING_HEARTBEAT: "${PROCESSING_HEARTBEAT:-15s}"
PROCESSING_RETRY_BASE: "${PROCESSING_RETRY_BASE:-2s}"
PROCESSING_RETRY_CAP: "${PROCESSING_RETRY_CAP:-60s}"
STAGING_ORPHAN_GRACE: "${STAGING_ORPHAN_GRACE:-1h}"
STAGING_FAILED_RETENTION: "${STAGING_FAILED_RETENTION:-168h}"
ADMISSION_MAX_PENDING_PER_USER: "${ADMISSION_MAX_PENDING_PER_USER:-20}"
ADMISSION_MAX_PENDING: "${ADMISSION_MAX_PENDING:-200}"
ADMISSION_MAX_STAGED_BYTES: "${ADMISSION_MAX_STAGED_BYTES:-2147483648}"
ADMISSION_MAX_CONCURRENT_UPLOADS: "${ADMISSION_MAX_CONCURRENT_UPLOADS:-8}"
OTEL_EXPORTER_OTLP_ENDPOINT: https://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=lab
OTEL_METRIC_EXPORT_INTERVAL: "${OTEL_METRIC_EXPORT_INTERVAL:-5000}"
OTEL_TRACES_SAMPLER: "${OTEL_TRACES_SAMPLER:-always_on}"
TELEMETRY_ENABLED: "${TELEMETRY_ENABLED:-1}"
LOG_LEVEL: info
LOG_FORMAT: json
x-gateway: &gateway
<<: *hardened
image: haproxy:3.2-alpine@sha256:96ec3c75a9264ba94e733eeacb44051ed6e7b79bb77a73c2c36f236b6423f1f0
user: "10001:10001"
deploy:
resources:
limits:
cpus: "0.10"
memory: 64M
services:
# This operator-only job prepares private data volume ownership. It has no
# network, no socket, and only the capabilities required for existing volumes.
volume-init:
image: alpine:3.22@sha256:49340b346798875b5592e5d420241f2f9cfba5f47e62ebb1446a1b34734e898f
user: "0:0"
network_mode: none
cap_drop: [ALL]
cap_add: [CHOWN, DAC_OVERRIDE]
security_opt: [no-new-privileges:true]
read_only: true
restart: "no"
logging: *log
command:
- sh
- -ec
- |
chown -R 70:70 /state/postgres
chown -R 999:1000 /state/redis
chown -R 10001:10001 /state/s3 /state/loki /state/tempo
chown -R 100:101 /state/rabbit
chown -R 65534:65534 /state/prometheus
chown -R 472:472 /state/grafana
chown -R 100:1000 /state/bao /state/audit
volumes:
- postgres18-data:/state/postgres
- redis7-data:/state/redis
- seaweedfs-data:/state/s3
- rabbitmq-data:/state/rabbit
- prometheus-data:/state/prometheus
- loki-data:/state/loki
- tempo-data:/state/tempo
- grafana-data:/state/grafana
- openbao-raft-data:/state/bao
- openbao-audit:/state/audit
deploy:
resources:
limits: {cpus: "0.25", memory: 64M}
openbao:
<<: *hardened
image: openbao/openbao:2.7.2@sha256:5046911075e137e154ce49ab051502f4ec7cf367d484ae4a4dce665d9b90561f
user: "100:1000"
networks: [default, edge]
command: [server, -config=/etc/openbao/server.hcl]
environment:
SKIP_CHOWN: "true"
SKIP_SETCAP: "true"
volumes:
- openbao-raft-data:/openbao/file
- openbao-audit:/openbao/audit
- type: bind
source: ./infra/openbao/server.hcl
target: /etc/openbao/server.hcl
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/openbao
target: /run/identity
read_only: true
bind: {create_host_path: false}
ports: ["127.0.0.1:${BAO_PORT:-8200}:8200"]
depends_on:
volume-init: {condition: service_completed_successfully}
healthcheck: {disable: true}
deploy:
resources:
limits: {cpus: "0.50", memory: 256M}
postgres:
<<: *hardened
image: postgres:18-alpine@sha256:123b17998ccaf47e53b502e8c6a1e79afd8f579c6f3eec83c563da026f957395
user: "70:70"
entrypoint: [/bin/sh, /security/entrypoint.sh]
environment:
POSTGRES_USER: lab_operator
POSTGRES_PASSWORD_FILE: /run/backend/operator-password
POSTGRES_DB: messenger
volumes:
- postgres18-data:/var/lib/postgresql
- type: bind
source: ./infra/postgres/entrypoint.sh
target: /security/entrypoint.sh
read_only: true
bind: {create_host_path: false}
- type: bind
source: ./infra/postgres/pg_hba.conf
target: /etc/postgresql/pg_hba.conf
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/postgres
target: /run/identity
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/backends/postgres
target: /run/backend
read_only: true
bind: {create_host_path: false}
tmpfs:
- /tmp:rw,noexec,nosuid,size=64m,mode=1777
- /var/run/postgresql:rw,noexec,nosuid,size=8m,uid=70,gid=70,mode=0750
depends_on:
volume-init: {condition: service_completed_successfully}
healthcheck:
test: [CMD, pg_isready, -U, lab_operator, -d, messenger]
interval: 5s
timeout: 5s
retries: 20
start_period: 10s
deploy:
resources:
limits:
cpus: "${POSTGRES_CPU_LIMIT:-0.50}"
memory: ${POSTGRES_MEM_LIMIT:-320M}
redis:
<<: *hardened
image: redis:7.4-alpine@sha256:c5d34a98396a4775e223c74e929162c35c5a2182901ba26b50ade939941ce14b
user: "999:1000"
command: [redis-server, /etc/redis/redis.conf]
volumes:
- redis7-data:/data
- type: bind
source: ./infra/redis/redis.conf
target: /etc/redis/redis.conf
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/redis
target: /run/identity
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/backends/redis
target: /run/backend
read_only: true
bind: {create_host_path: false}
depends_on:
volume-init: {condition: service_completed_successfully}
healthcheck:
test: [CMD-SHELL, 'REDISCLI_AUTH=$$(cat /run/backend/health-password) redis-cli --tls --cacert /run/identity/current/ca.crt --user health ping']
interval: 5s
timeout: 5s
retries: 20
deploy:
resources:
limits: {cpus: "0.25", memory: 128M}
# Namespace owner: all master/filer/volume transports are loopback-only.
seaweedfs:
<<: *hardened
image: chrislusf/seaweedfs:4.51@sha256:f7a49e3f3972ebed5be2ed4befa4a0c68eb7bdddba6e692d42729ceb545134b8
user: "10001:10001"
networks: [default, edge]
command: [server, -filer, -ip=127.0.0.1, -ip.bind=127.0.0.1, -dir=/data, -master.volumeSizeLimitMB=64, -filer.exposeDirectoryData=false]
environment: {GOGC: "50", GOMEMLIMIT: 512MiB}
volumes:
- seaweedfs-data:/data
- type: bind
source: ./infra/s3/security.toml
target: /etc/seaweedfs/security.toml
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/s3
target: /run/identity
read_only: true
bind: {create_host_path: false}
ports: ["127.0.0.1:${S3_PORT:-8333}:8333"]
depends_on:
volume-init: {condition: service_completed_successfully}
healthcheck:
test: [CMD-SHELL, 'wget -q -O /dev/null http://127.0.0.1:8888/healthz']
interval: 5s
timeout: 5s
retries: 24
start_period: 30s
deploy:
resources:
limits: {cpus: "1.00", memory: 640M}
# Native HTTPS S3 and its auxiliary mTLS gRPC share the closed storage namespace.
s3:
<<: *hardened
image: chrislusf/seaweedfs:4.51@sha256:f7a49e3f3972ebed5be2ed4befa4a0c68eb7bdddba6e692d42729ceb545134b8
user: "10001:10001"
network_mode: service:seaweedfs
command: [s3, -filer=127.0.0.1:8888.18888, -ip.bind=0.0.0.0, -port=8333, -port.iceberg=0, -port.lance=0, -config=/run/backend/s3.json, -cert.file=/run/identity/current/tls.crt, -key.file=/run/identity/current/tls.key, -cacert.file=/run/identity/current/ca.crt]
volumes:
- type: bind
source: ./infra/s3/security.toml
target: /etc/seaweedfs/security.toml
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/s3
target: /run/identity
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/backends/s3
target: /run/backend
read_only: true
bind: {create_host_path: false}
depends_on:
seaweedfs: {condition: service_healthy}
healthcheck: {disable: true}
deploy:
resources:
limits: {cpus: "0.50", memory: 256M}
broker:
<<: *hardened
image: rabbitmq:4.3.7-management-alpine@sha256:1528a88eebf2bc4d0ab7e9d5d9bc542e9bdf5e1b105e65848fcf8de5c2186528
user: "100:101"
hostname: broker
extra_hosts: ["broker:127.0.0.1"]
environment:
RABBITMQ_NODENAME: rabbit@broker
ERL_EPMD_ADDRESS: 127.0.0.1
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: '-kernel inet_dist_use_interface {127,0,0,1}'
networks: [default, observability, edge]
volumes:
- rabbitmq-data:/var/lib/rabbitmq
- type: bind
source: ./infra/rabbitmq/rabbitmq.conf
target: /etc/rabbitmq/rabbitmq.conf
read_only: true
bind: {create_host_path: false}
- type: bind
source: ./infra/rabbitmq/enabled_plugins
target: /etc/rabbitmq/enabled_plugins
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/broker
target: /run/identity
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/backends/broker
target: /run/backend
read_only: true
bind: {create_host_path: false}
ports: ["127.0.0.1:${BROKER_MANAGEMENT_PORT:-15671}:15671"]
depends_on:
volume-init: {condition: service_completed_successfully}
healthcheck:
test: [CMD, rabbitmq-diagnostics, -q, ping]
interval: 10s
timeout: 5s
retries: 20
start_period: 30s
deploy:
resources:
limits:
cpus: "${BROKER_CPU_LIMIT:-0.50}"
memory: ${BROKER_MEM_LIMIT:-384M}
broker-admin-tls:
<<: *gateway
network_mode: service:broker
volumes:
- type: bind
source: ./infra/rabbitmq/admin-tls.cfg
target: /usr/local/etc/haproxy/haproxy.cfg
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/broker
target: /run/identity
read_only: true
bind: {create_host_path: false}
depends_on:
broker: {condition: service_started}
healthcheck: {disable: true}
migrate:
<<: *app
entrypoint: [/usr/local/bin/migrate]
command: [up]
environment:
<<: *tls-environment
APP_CONFIG: /etc/messenger/dev/config.yaml
POSTGRES_HOST: postgres
POSTGRES_PORT: "5432"
POSTGRES_DATABASE: messenger
POSTGRES_USER: lab_operator
POSTGRES_PASSWORD_FILE: /run/bootstrap/postgres-password
POSTGRES_SSL_MODE: verify-full
volumes:
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/operator
target: /run/identity
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/bootstrap/migrate
target: /run/bootstrap
read_only: true
bind: {create_host_path: false}
depends_on:
postgres: {condition: service_healthy}
profiles: [operator]
healthcheck: {disable: true}
restart: "no"
deploy:
resources:
limits: {cpus: "0.50", memory: 256M}
s3-init:
<<: *app
entrypoint: [/usr/local/bin/s3init]
environment:
<<: *tls-environment
S3_ENDPOINT: https://seaweedfs:8333
S3_REGION: us-east-1
S3_BUCKET: "${S3_BUCKET:-messenger-attachments}"
S3_ACCESS_KEY_ID_FILE: /run/bootstrap/s3-admin-access-key
S3_SECRET_ACCESS_KEY_FILE: /run/bootstrap/s3-admin-secret-key
volumes:
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/operator
target: /run/identity
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/bootstrap/s3
target: /run/bootstrap
read_only: true
bind: {create_host_path: false}
depends_on:
s3: {condition: service_started}
profiles: [operator]
healthcheck: {disable: true}
restart: "no"
deploy:
resources:
limits: {cpus: "0.25", memory: 128M}
jobctl:
extends: {service: migrate}
entrypoint: [/usr/local/bin/jobctl]
command: [failed]
api:
<<: *app
build:
context: .
dockerfile: Dockerfile
args: {VERSION: "${VERSION:-dev}"}
environment:
<<: *app-environment
BAO_ROLE: api
BAO_KV_PATH: secret/data/api
OTEL_SERVICE_NAME: messenger-api
POSTGRES_USER: messenger_api
REDIS_ENABLED: "true"
REDIS_HOST: redis
REDIS_PORT: "6379"
REDIS_USERNAME: cache
REDIS_DB: "0"
CACHE_ENABLED: "true"
SERVER_HOST: 0.0.0.0
SERVER_PORT: "8080"
SERVER_MAX_BODY_BYTES: "${SERVER_MAX_BODY_BYTES:-262144}"
SERVER_READ_HEADER_TIMEOUT: 10s
SERVER_READ_TIMEOUT: 60s
SERVER_SHUTDOWN_TIMEOUT: 30s
GOMEMLIMIT: 192MiB
networks:
default: {aliases: [api-replica]}
volumes:
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/api
target: /run/identity
read_only: true
bind: {create_host_path: false}
depends_on:
postgres: {condition: service_healthy}
redis: {condition: service_healthy}
openbao: {condition: service_started}
healthcheck:
test: [CMD, wget, -q, -O, /dev/null, http://127.0.0.1:8084/health/ready]
interval: 5s
timeout: 5s
retries: 20
start_period: 5s
deploy:
replicas: ${API_REPLICAS:-2}
resources:
limits:
cpus: "${API_CPU_LIMIT:-0.50}"
memory: ${API_MEM_LIMIT:-256M}
worker:
<<: *app
entrypoint: [/usr/local/bin/worker]
environment:
<<: *app-environment
BAO_ROLE: worker
BAO_KV_PATH: secret/data/worker
OTEL_SERVICE_NAME: messenger-worker
POSTGRES_USER: messenger_worker
POSTGRES_MAX_CONNS: "4"
REDIS_ENABLED: "false"
CACHE_ENABLED: "false"
WORKER_CONCURRENCY: "${WORKER_CONCURRENCY:-1}"
WORKER_PREFETCH: "${WORKER_PREFETCH:-1}"
WORKER_SHUTDOWN_TIMEOUT: 30s
WORKER_HEALTH_ADDR: 127.0.0.1:8081
GOMEMLIMIT: 768MiB
FAULTS_ENABLED: "${FAULTS_ENABLED:-0}"
FAULT_TARGET_JOB: "${FAULT_TARGET_JOB:-}"
FAULT_WORKER_AFTER_CLAIM: "${FAULT_WORKER_AFTER_CLAIM:-}"
FAULT_WORKER_AFTER_FIRST_VARIANT: "${FAULT_WORKER_AFTER_FIRST_VARIANT:-}"
FAULT_WORKER_AFTER_PUT: "${FAULT_WORKER_AFTER_PUT:-}"
FAULT_WORKER_AFTER_COMMIT: "${FAULT_WORKER_AFTER_COMMIT:-}"
FAULT_WORKER_FAIL: "${FAULT_WORKER_FAIL:-}"
FAULT_WORKER_SLEEP_AFTER_CLAIM: "${FAULT_WORKER_SLEEP_AFTER_CLAIM:-}"
volumes:
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/worker
target: /run/identity
read_only: true
bind: {create_host_path: false}
depends_on:
postgres: {condition: service_healthy}
broker: {condition: service_healthy}
openbao: {condition: service_started}
healthcheck:
test: [CMD, wget, -q, -O, /dev/null, http://127.0.0.1:8081/health/ready]
interval: 5s
timeout: 5s
retries: 20
start_period: 10s
deploy:
resources:
limits:
cpus: "${WORKER_CPU_LIMIT:-1.00}"
memory: ${WORKER_MEM_LIMIT:-1G}
relay:
<<: *app
entrypoint: [/usr/local/bin/relay]
environment:
<<: *app-environment
BAO_ROLE: relay
BAO_KV_PATH: secret/data/relay
OTEL_SERVICE_NAME: messenger-relay
POSTGRES_USER: messenger_relay
POSTGRES_MAX_CONNS: "3"
REDIS_ENABLED: "false"
CACHE_ENABLED: "false"
RELAY_HEALTH_ADDR: 127.0.0.1:8082
OUTBOX_POLL_INTERVAL: "${OUTBOX_POLL_INTERVAL:-1s}"
OUTBOX_CLAIM_TTL: "${OUTBOX_CLAIM_TTL:-30s}"
OUTBOX_BATCH: "${OUTBOX_BATCH:-20}"
OUTBOX_REAPER_INTERVAL: "${OUTBOX_REAPER_INTERVAL:-15s}"
OUTBOX_RECONCILE_AFTER: "${OUTBOX_RECONCILE_AFTER:-5m}"
OUTBOX_RETENTION: "${OUTBOX_RETENTION:-168h}"
GOMEMLIMIT: 96MiB
FAULTS_ENABLED: "${FAULTS_ENABLED:-0}"
FAULT_TARGET_JOB: "${FAULT_TARGET_JOB:-}"
FAULT_OUTBOX_AFTER_COMMIT: "${FAULT_OUTBOX_AFTER_COMMIT:-}"
volumes:
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/relay
target: /run/identity
read_only: true
bind: {create_host_path: false}
depends_on:
postgres: {condition: service_healthy}
broker: {condition: service_healthy}
openbao: {condition: service_started}
healthcheck:
test: [CMD, wget, -q, -O, /dev/null, http://127.0.0.1:8082/health/ready]
interval: 5s
timeout: 5s
retries: 20
deploy:
resources:
limits:
cpus: "${RELAY_CPU_LIMIT:-0.25}"
memory: ${RELAY_MEM_LIMIT:-128M}
lb:
<<: *gateway
networks: [default, observability, edge]
volumes:
- type: bind
source: ./infra/lb/haproxy.cfg
target: /usr/local/etc/haproxy/haproxy.cfg
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/lb
target: /run/identity
read_only: true
bind: {create_host_path: false}
ports:
- "127.0.0.1:${PUBLIC_PORT:-8443}:8443"
- "127.0.0.1:${LB_STATS_PORT:-8404}:8404"
depends_on:
api: {condition: service_healthy}
healthcheck: {disable: true}
otel-collector:
<<: *hardened
image: otel/opentelemetry-collector-contrib:0.164.0@sha256:340c9ca6a36617084c6661bdc4f0fe92b7cda5a805d61673ec8db375fddc522b
user: "10001:10001"
command: [--config=/etc/otel/collector.yaml]
environment: {GOMEMLIMIT: 140MiB}
networks:
default: {aliases: [collector]}
observability: {aliases: [collector]}
volumes:
- type: bind
source: ./infra/otel/collector.yaml
target: /etc/otel/collector.yaml
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/collector
target: /run/identity
read_only: true
bind: {create_host_path: false}
healthcheck: {disable: true}
deploy:
resources:
limits:
cpus: "${OTEL_COLLECTOR_CPU_LIMIT:-0.25}"
memory: ${OTEL_COLLECTOR_MEM_LIMIT:-192M}
collector-tls:
<<: *gateway
network_mode: service:otel-collector
volumes:
- type: bind
source: ./infra/telemetry-gateway/collector.cfg
target: /usr/local/etc/haproxy/haproxy.cfg
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/collector
target: /run/identity
read_only: true
bind: {create_host_path: false}
depends_on:
otel-collector: {condition: service_started}
healthcheck: {disable: true}
prometheus:
<<: *hardened
image: prom/prometheus:v3.15.1@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
user: "65534:65534"
command: [--config.file=/etc/prometheus/prometheus.yml, --web.listen-address=127.0.0.1:19090, --storage.tsdb.path=/prometheus, --storage.tsdb.retention.time=10d, --storage.tsdb.retention.size=2GB]
networks: [observability, edge]
environment: {GOMEMLIMIT: 256MiB}
volumes:
- prometheus-data:/prometheus
- type: bind
source: ./infra/prometheus
target: /etc/prometheus
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/prometheus
target: /run/identity
read_only: true
bind: {create_host_path: false}
ports: ["127.0.0.1:${PROMETHEUS_PORT:-9090}:9090"]
depends_on:
volume-init: {condition: service_completed_successfully}
healthcheck:
test: [CMD, wget, -q, -O, /dev/null, http://127.0.0.1:19090/-/ready]
interval: 5s
timeout: 5s
retries: 20
deploy:
resources:
limits:
cpus: "${PROMETHEUS_CPU_LIMIT:-0.50}"
memory: ${PROMETHEUS_MEM_LIMIT:-320M}
prometheus-tls:
<<: *gateway
network_mode: service:prometheus
volumes:
- type: bind
source: ./infra/telemetry-gateway/prometheus.cfg
target: /usr/local/etc/haproxy/haproxy.cfg
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/prometheus
target: /run/identity
read_only: true
bind: {create_host_path: false}
depends_on:
prometheus: {condition: service_started}
healthcheck: {disable: true}
loki:
<<: *hardened
image: grafana/loki:3.6.4@sha256:be31579ac047e9f78b81e48f3b69d3af709e7299b431d5aa78bcda43382f9511
user: "10001:10001"
command: [-config.file=/etc/loki/loki.yaml]
networks: [observability, edge]
environment: {GOMEMLIMIT: 256MiB}
volumes:
- loki-data:/loki
- type: bind
source: ./infra/loki/loki.yaml
target: /etc/loki/loki.yaml
read_only: true
bind: {create_host_path: false}
ports: ["127.0.0.1:${LOKI_PORT:-3100}:3100"]
depends_on:
volume-init: {condition: service_completed_successfully}
healthcheck: {disable: true}
deploy:
resources:
limits:
cpus: "${LOKI_CPU_LIMIT:-0.50}"
memory: ${LOKI_MEM_LIMIT:-320M}
loki-tls:
<<: *gateway
network_mode: service:loki
volumes:
- type: bind
source: ./infra/telemetry-gateway/loki.cfg
target: /usr/local/etc/haproxy/haproxy.cfg
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/loki
target: /run/identity
read_only: true
bind: {create_host_path: false}
depends_on:
loki: {condition: service_started}
healthcheck: {disable: true}
tempo:
<<: *hardened
image: grafana/tempo:2.10.10@sha256:12b0820f13ffa10ce956b6e9acc3b03f3cffded99109be6cc09d106fe56e5d38
user: "10001:10001"
command: [-config.file=/etc/tempo/tempo.yaml]
networks: [observability, edge]
environment: {GOMEMLIMIT: 640MiB}
volumes:
- tempo-data:/var/tempo
- type: bind
source: ./infra/tempo/tempo.yaml
target: /etc/tempo/tempo.yaml
read_only: true
bind: {create_host_path: false}
ports: ["127.0.0.1:${TEMPO_PORT:-3200}:3200"]
depends_on:
volume-init: {condition: service_completed_successfully}
healthcheck: {disable: true}
deploy:
resources:
limits:
cpus: "${TEMPO_CPU_LIMIT:-0.25}"
memory: ${TEMPO_MEM_LIMIT:-768M}
tempo-tls:
<<: *gateway
network_mode: service:tempo
volumes:
- type: bind
source: ./infra/telemetry-gateway/tempo.cfg
target: /usr/local/etc/haproxy/haproxy.cfg
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/tempo
target: /run/identity
read_only: true
bind: {create_host_path: false}
depends_on:
tempo: {condition: service_started}
healthcheck: {disable: true}
grafana:
<<: *hardened
image: grafana/grafana:13.2.5@sha256:9f7aa1e1e3246aade5792d35a738049f7f32f89cd51d1956cf426a4e86268a84
user: "472:472"
networks: [observability, edge]
environment:
GF_SERVER_HTTP_ADDR: 127.0.0.1
GF_SERVER_HTTP_PORT: "13000"
GF_SERVER_PROTOCOL: http
GF_SERVER_ROOT_URL: "https://localhost:${GRAFANA_PORT:-3000}/"
GF_SECURITY_ADMIN_PASSWORD__FILE: /etc/grafana/provisioning/admin-password
GF_AUTH_ANONYMOUS_ENABLED: "false"
GF_AUTH_PROXY_ENABLED: "false"
GF_USERS_ALLOW_SIGN_UP: "false"
volumes:
- grafana-data:/var/lib/grafana
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/backends/grafana
target: /etc/grafana/provisioning
read_only: true
bind: {create_host_path: false}
- type: bind
source: ./infra/grafana/dashboards
target: /var/lib/grafana/dashboards
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/grafana
target: /run/identity
read_only: true
bind: {create_host_path: false}
ports: ["127.0.0.1:${GRAFANA_PORT:-3000}:3000"]
depends_on:
volume-init: {condition: service_completed_successfully}
healthcheck:
test: [CMD, wget, -q, -O, /dev/null, http://127.0.0.1:13000/api/health]
interval: 5s
timeout: 5s
retries: 20
deploy:
resources:
limits:
cpus: "${GRAFANA_CPU_LIMIT:-0.25}"
memory: ${GRAFANA_MEM_LIMIT:-512M}
grafana-tls:
<<: *gateway
network_mode: service:grafana
volumes:
- type: bind
source: ./infra/telemetry-gateway/grafana.cfg
target: /usr/local/etc/haproxy/haproxy.cfg
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${LAB7_SECRETS_DIR:-./secrets/lab7}/pki/grafana
target: /run/identity
read_only: true
bind: {create_host_path: false}
depends_on:
grafana: {condition: service_started}
healthcheck: {disable: true}
networks:
default: {internal: true}
observability: {internal: true}
# Docker Desktop does not publish ports for internal-only namespaces.
# Only CA-verified TLS owners join edge; API replicas still have no host ports.
edge:
volumes:
postgres18-data:
redis7-data:
seaweedfs-data:
rabbitmq-data:
prometheus-data:
loki-data:
tempo-data:
grafana-data:
openbao-raft-data:
openbao-audit:
+53
View File
@@ -0,0 +1,53 @@
env: dev
server:
host: 0.0.0.0
port: 8080
read_header_timeout: 10s
shutdown_timeout: 10s
# Предел размера тела запроса; JSON API ЛР3 укладывается в сотни килобайт,
# а загрузка вложений ЛР4 (до 10 MiB) поднимает это значение явно.
max_body_bytes: 262144
log:
level: info
format: json
postgres:
host: localhost
port: 5432
database: messenger
user: ""
password: ""
ssl_mode: verify-full
max_conns: 10
min_conns: 1
max_conn_lifetime: 1h
max_conn_idle_time: 30m
connect_timeout: 5s
query_timeout: 5s
redis:
enabled: true
host: localhost
port: 6379
db: 0
# -1 disables go-redis retries; 0 keeps the library default (3 retries).
max_retries: -1
dial_timeout: 300ms
read_timeout: 200ms
write_timeout: 200ms
pool_size: 10
cache:
enabled: true
chat_list_ttl: 5m
lock_ttl: 10s
lock_wait_timeout: 2s
session:
ttl_seconds: 3600
# Retention of a message Idempotency-Key; contract 1.3.0 requires ≥ 24 h.
idempotency:
ttl_seconds: 86400
+90 -28
View File
@@ -1,37 +1,99 @@
# Матрица доверия — заполнить в решении lab7
# Матрица доверия — реализованный стенд lab7
Это обязательная матрица покрытия. Для каждой строки укажите реальный порт, endpoint, CA, service identity, проверку SAN/ACL, срок жизни credentials и команду положительного/отрицательного теста. TLS шифрует и подтверждает сервер; mTLS также проверяет клиента. При доверенном CA всё равно нужно ограничивать разрешённую identity. Plaintext hop допустим только внутри закрытого локального transport/proxy boundary, без опубликованного обхода; границу и rationale нужно доказать.
## Identities, CA и leases
| Связь | Минимум на 3 | Дополнительно на 4 | Доказательство / конфиг |
| --- | --- | --- | --- |
| Клиент → ingress/LB | TLS, CA + hostname, Bearer | Проверка expiry/revocation сессии и общего login rate limit | TODO |
| LB → API (все реплики) | mTLS + allowlist LB identity | Ротация leaf без обхода проверки | TODO |
| API/worker → PostgreSQL | TLS verify-full + раздельные ограниченные DB roles | Ротация credentials; mTLS — допустимое усиление | TODO |
| API/worker → S3 | TLS + разные scoped credentials/prefix rights | Проверка denied prefix и ротация | TODO |
| API/worker → broker | TLS + разные producer/consumer identities и ACL | mTLS + ограничение допустимых SAN/ролей | TODO |
| API/worker → Collector | TLS; сеть/аутентификация запрещает произвольный ingest | mTLS с раздельными identities | TODO |
| Collector → telemetry backends | TLS + scoped backend credentials или mTLS | Ротация и проверка least privilege | TODO |
| Grafana → data sources | TLS с проверкой сервера, ограниченная identity | Отрицательные проверки доступа | TODO |
| API/worker/agent → OpenBao | TLS + сервисный auth + scoped policy | Renewal/expiry/revocation | TODO |
| Оператор → OpenBao/Grafana/S3 admin | TLS + authentication, доступ локальный/через защищённый канал | Разделение operator/application roles | TODO |
| API → Redis/Valkey, если используется | TLS + ACL/пароль с минимальными правами | Ротация; mTLS по возможности | TODO |
| Replication/кластерные каналы, если добавлены | TLS/mTLS либо подтверждённый шифрованный transport | Проверка заявленной модели отказа | TODO |
Все TLS endpoints проверяются по CA и DNS SAN; TLS ≥1.2. Offline root находится
у оператора вне checkout, OpenBao выдаёт EC P-256 leaf через внутренний intermediate.
URI service identity — `spiffe://mtusi/<role>`; DNS SANs задаёт
`scripts/lab7_environment.py:ROLES`. Доверенный CA сам по себе не является ACL.
Нет выбранного компонента — отметьте N/A с причиной, а не оставляйте пустую строку. Внутренний публичный health listener, если нужен, минимален и изолирован; через него нельзя обращаться к бизнес-API без mTLS. Отдельно задокументируйте scrape/health/bootstrap каналы: они тоже не должны создавать обход.
Default leaf TTL — 1h, максимум PKI role — 1h; host-side `make renewer`
выпускает новые leaf каждые 30 минут и синхронизирует CRL; исследовательский renewal/expiry
использует 20s. Cert-auth tokens: 60s, max/explicit max300s. KV freshness у Go
приложения — 60s, независимо от срока leaf и token; admission закрывается за1s
до ближайшего deadline (наблюдаемое закрытие при sealed Bao: 60.5–61.7s с учётом
1s cadence watchdog). Backend KV passwords/SigV4 keys статические и живут до
явной ротации — это не динамические60s credentials. Session TTL3600s;
server хранит только SHA-256 token hash.
## Обязательные отрицательные проверки
| Связь | Endpoint / протокол | Проверка identity и права | Lease / reload | Положительная и отрицательная проверка |
| --- | --- | --- | --- | --- |
| Клиент → ingress/LB | `https://localhost:8443`, TLS, Bearer | DNS SANlocalhost; без clientcert. LB заменяет forwarding headers и удаляет X-User-Id | LB leaf; пользовательская session3600s/logout/expiry | `make test test-security`; `public_ca_hostname`, `http_acl_logout_expiry_bounds`, `shared_rate_limit` |
| LB → все API-реплики | `api-replica:8080`, native mTLS | LB проверяет SANapi-replica/CA/CRL; Go server требует точный URI`spiffe://mtusi/lb` | Atomic leaf/CRL reload Go; trust changes restart; tickets/cache off | `mtls_and_telemetry_acl`: lb200; absent/wrongCA/wrongURI TLS拒绝; `make test-pki` oldLB revoke/retire |
| API/worker/relay → PostgreSQL | `postgres:5432`, native TLS+SCRAM, verify-full | Раздельные messenger_api/worker/relay roles; no superuser/DDL. Worker/relay не читают users/sessions/messages | Статический password в ownKV; connection lifetime30s; credentials change → bounded restart | `database_privileges`: SQLSTATE42501; `make integration-test` actual nativeTLS DB suites |
| API/worker → S3 | `https://seaweedfs:8333`, native TLS+SigV4 | API stagingRead/Write, finalRead; worker managed-prefixRead/Write+bucketList; foreign GET/PUT/DELETE/COPY/bucket 403, worker bucket-level LIST returns managed metadata (documented limitation) | Static keys until explicitrotation; role-specific ownKV. No claim of required clientcert on S3HTTP | `s3_prefix_acl`: GET/PUT/DELETE/COPY+fullreadback; `make test-presign test-queue`; workerkeyrotation inPKIstudy |
| Relay/API outbox/worker → broker | `broker:5671`, native mTLS/EXTERNAL-only | API commits DBoutbox, relay publishes onlyattachments.jobs; worker consumes onlyattachment.process.v1. First DNS SAN principal, anchored broker ACL | Leaf+broker user permissions; nativeprincipal revoke distinct from serial CRL | `broker_external_acl`: positive relaypublish/workerconsume; opposite/topology/wrongDNS403, no-cert/wrongCA TLS rejection |
| API/worker/relay → Collector | `otel-collector:4318`, mTLS gateway → loopback14318 | Exact CNapi/worker/relay and POST/v1/{traces,metrics,logs}; unknownCN/verb/path denied | Gateway loads CA/CRL atstartup; revoke/rollover restarts gateways; app pools are not claimed drained | `mtls_and_telemetry_acl`: api200, lb403, missing/wrongCA rejection; full `make test-observability` |
| Collector → telemetry backends | Loki`https://loki:3100`; TempoGRPCS`tempo:4437` | CollectorCN onlylogpush/OTLP; no dataquery. Native hops127.0.0.1:13100/14317 | Distinct collectorleaf; fixedsingletenantfake | `mtls_and_telemetry_acl` collectorquery403; actual traces/logs in `make test-observability` |
| Grafana → data sources | Prometheus9090/Loki3100/Tempo3200, HTTPS/mTLS | GrafanaCN query allowlist; push/delete/admin/reload forbidden; CA+hostname verified | Ownleaf in secureJsonData rendered outsideGit; rotation rerenders/restarts | `mtls_and_telemetry_acl` forbiddenwrite403; Grafana/dataSources/TraceQL correlation intelemetrygate |
| API/worker/relay → OpenBao | `https://openbao:8200`, verifiedTLS+cert-auth | Authrole checks CN+URI; ownsecret/data/<role> and ownpki/issue/<role> only; no root/defaultpolicy | Token60s/max300s; cert reauth; KVfresh60s; independentleafdeadline | `bao_api_policy`: ownKV200, foreignKV/issuer/policyadmin403; `make test-pki` renewal/expiry/recovery |
| Оператор → OpenBao/Grafana/S3 admin | LoopbackHTTPS8200/3000/8333 | Separate operatorcertificate+scopedBaopolicy; Grafana localpassword; SigV4S3operator. No anonymousadmin | Root token revoked afterbootstrap; unseal2-of-3 hoststate; operatorleafrotation | `securityctl.py up`, `audit-summary`, S3readinessfullbytes; noappoperatorcredentialmounts inisolation |
| API → Redis | `redis:6379`, nativeTLS+ACLpassword | `cache` user: chat_list:* / lock:chat_list:* onlyGET/SET/DEL/PING/EVAL/scripts; default useroff | StaticownKVpassword; RedisAOF persists; no requiredclientcert claimed | `redis_tls_cache_acl`: PONG/allowedcache, foreignkeyNOPERM; nativeTLS integrationDB7 |
| Replication/cluster channels | ExternalreplicationN/A: allstatefulstores single-node | BaoRaftcluster127.0.0.1:8201nativeTLS; S3master/filer/volume oneclosednamespace, internalgRPCmTLSCNs3; telemetrygossiploopback | NoHAclaim forDB/S3/broker/Bao/host | `runtime_container_network_isolation`; `seaweed_auxiliary_mtls`; actualpersistence/bothAPIreplicafailures |
1. HTTPS с правильным CA/hostname проходит; с пустым/чужим CA и неправильным hostname — ошибка верификации, не «сервер недоступен».
2. LB→API: без client cert, с cert чужого CA, с запрещённой identity от доверенного CA — отказ; с разрешённой LB identity — успех.
3. На 4 повторите набор для broker и Collector; ограничение на network policy не заменяет проверку identity.
4. Пользователь вне чата не читает metadata/content и не прикрепляет чужой файл; поддельный X-User-Id не меняет Bearer identity.
5. OpenBao app policy читает только свой secret/path; запись policy и чтение чужого path запрещены. Worker не получает admin credentials API/БД/S3.
6. На 4 отзовите secret/cert и дождитесь контролируемого эффекта с учётом действующих соединений, TLS resumption и cache/TTL. Покажите новое подключение, иначе тест отзыва может ничего не проверить.
## Служебные, scrape и health каналы
## Bootstrap и эксплуатация
| Канал | Boundary / identity | Проверка |
| --- | --- | --- |
| Local API/worker/relay health8084/8081/8082 | Namespace loopback, no business API; no hostpublication | Из соседнего defaultnamespaceconnection_refused; Gohealthcloses admission onauthorizationexpiry |
| HAProxy stats8404 | mTLS, CNoperator/prometheus; monitoronly/metrics | Separate healthbackend excluded fromAPI SLI; Prometheus scrape overTLS |
| Collector metrics8888/8889, health13134 | mTLS/CNprometheus oroperator; localhops18888/18889/13133 | Privateportprobes, fulltelemetryscrapes |
| Prometheus/Loki/Tempo selfmetrics | HTTPS/mTLS/CNprometheus GETmetrics/ready only | Wrongqueryrole403; native19090/13100/13200 andgRPC/gossip notnetwork-accessible |
| RabbitMQ management15671, scrape15693 | mTLS HAProxy → loopback15672/15692; operatoradmin Basic onlymanagement, Prometheus onlyscrape | NoAMQP5672 orplaintextmanagement published; brokerACL/scapedata |
| S3 auxiliary18333 | Native mTLS, onlyCNs3; HTTP2 ALPNh2 | api/worker/operator/no-cert denied, s3TLSchannelaccepted; not anadminmethodproof |
| S3 master/filer/volume | HTTP9333/8888/8080 andgRPC19333/18888/18080 onnamespace loopback; gRPC additionallyCNs3 | Neighbor namespaceconnection_refused; Iceberg8181/Lance9101 disabled, not excluded |
| Migrations/S3-init/volume-init | Explicitoperatorjobs; no rootCA/unseal mount | Migrate DBoperator only; S3init operator only; volume-init has no network/socket, CHOWN/DAC_OVERRIDE only |
В репозитории только открытые конфиги, policy templates и scripts. CA private key, leaf keys, root token, unseal shares и SecretID не коммитятся, не запекаются в образ и не печатаются в отчёте. Для локального стенда оператор создаёт их вне Git и передаёт через files/secrets с ограниченными правами. Опишите, кто имеет доступ к CA/unseal, что нужно после рестарта OpenBao и как восстановить работу.
Native plaintext hops exist **only within the shared Linux network namespace**
of a backend+its TLS gateway, or S3co-located internal HTTP components. A helper
compares actual `/proc/self/ns/net`, not just containerIDs. None of these listeners
is a published bypass; only TLS owner joins routable networks. `default` is used
by apps, `observability` by telemetry, `edge` by their TLS owners.
OpenBao не использует dev-mode. Выберите persistent storage и auth метод (например AppRole с короткоживущим SecretID/response wrapping, cert auth или иная обоснованная identity). Не подменяйте root token постоянной переменной BAO_TOKEN в приложении. Рабочие tokens имеют policy, TTL и renewal/re-auth сценарий. Статические KV secrets на 3 допустимы при реальном чтении через сервисную identity; динамические DB credentials — хорошее расширение, но не обязательное название технологии.
## Bootstrap, current-version и отзыв
Разрешённый компромисс учебного стенда: ручной operator init/unseal, если он безопасно документирован и не нужен при каждом обычном рестарте API. Автоматическое хранение всех unseal shares рядом с OpenBao без защиты не считается решённым bootstrap.
1. `cp .env.example .env; make up`: offlinekeys/backendcredentials/complete role
versions are generated locally; init/unseal/provision/migrations follow.
2. RootCA/privateoperator/unseal state is in `~/.local/share/mtusi-lab7/<project>`
with0700/0600. Only public CA/intermediate/CRLs and each role's own leaf/key
enter role mounts. Operator recovery material never enters app/images/Git.
3. Native consumers use `current` alias. Every Go snapshot reads one bounded
regular `current-version` selector and immutable `versions/<name>`; cert/key/CA/CRL
cannot be mixed across a swap. Selector is owner/group-readable before atomic
publication. No last-known-invalid fallback, arbitrary path or symlink retry.
4. `make pki-renew ROLE=api CERT_TTL=1h` and `renew-watch` use owncert-auth. Token
renewal cannot renew theleaf. Missing/expired/revoked material fails closed.
5. `securityctl.py revoke ROLE --serial SERIAL`: publishes issuer-signed CRLs,
updates BaoauthCRLs, then restarts native CRL-consuming HAProxy gateways.
Existing application TLS/DB/S3/broker pools are not claimed to be drained;
only fresh/resumption attempts are measured.
Go verifies currentCRLs onfreshconnections. Principalcompromise also requires
backendcredential/token/authrole disable; RabbitMQ EXTERNAL user/connection
revocation is separate and not falsely represented by CRLfileupdate.
6. Offline rootCRLs are regenerated before their final24h and onoperator sync;
nativegateways are reconciled on `up`. StaleCRL verification is never disabled.
7. CArollover uses prepare(overlapCA/CRLs+authtrust) → issue(newintermediate/leaves)
→ retire(oldCA/issuer/authCRLs). Retire re-publishes every running role identity
with a new-only trust bundle (the leaf is unchanged), so an old certificate is
rejected by trust, not by a missing CRL; a process whose mounted trust differs
from its startup roots fails closed (`restart required`). Prepare/retire reload
api/worker/relay one replica at a time; issue only reloads native owners and
Go re-reads the new leaf on fresh handshakes. Readiness closes on root removal
so LB retires a not-yet-restarted replica instead of routing into TLS failures.
Study checks oldtrustedleaf during overlap, fresh/resumption attempts after
retirement, newpipeline/Redis/Bao/telemetry paths and retained message/filebytes;
each phase waits for the full
public upload pipeline before asserting contract details.
## Обязательные отрицательные наборы
- CA/hostname errors are checked as verification errors, not TCPtimeouts.
- API/broker/Collector: allowedidentitypositive, absentcert, foreignCA andsameCA
forbiddenrole negatives. HTTP403 aftertrustedmTLS is distinguished from TLSdenial.
- Foreignchat/attachment access, forgedX-User-Id, logout/DBexpiry, uploadbounds,
shared429/Retry-After, nativeDB/S3/broker/Baopolicy allow/deny are runtime-tested.
- Leaf/CArevocation checks freshconnections and explicit resumptionattempts;
tickets/cacheoff means no claim of a positively resumedsession.
- `evidence/lab7/security.json`, `verification.json`, `restart-scope.json`,
`failover-scope.json`, `sequential-failover.json` andPKIstudy contain sanitized
observations; [scantriage](../security/SCAN_TRIAGE.md) names residualrisk.
+12
View File
@@ -0,0 +1,12 @@
# Secure local/operator integration ports; never a published API bypass.
include:
- path: compose.yaml
services:
postgres:
networks: [default, edge]
ports:
- "127.0.0.1:${TEST_POSTGRES_PORT:-15432}:5432"
redis:
networks: [default, edge]
ports:
- "127.0.0.1:${TEST_REDIS_PORT:-16379}:6379"
+7
View File
@@ -0,0 +1,7 @@
# Architecture of the app binary inside the image
# generated: 2026-10-15T19:03:01Z
## emulated build (--build-arg TARGETARCH=amd64): e_machine 3e = x86-64
02 00 3e 00
## fixed build (no TARGETARCH default): e_machine b7 = aarch64
02 00 b7 00
+3
View File
@@ -0,0 +1,3 @@
# body limit check — 2026-10-16T14:02:25Z (limit 256 KiB from SERVER_MAX_BODY_BYTES)
тело 400 КиБ (сверх лимита): 413
тело 100 КиБ (в пределах): в„– 400
+7
View File
@@ -0,0 +1,7 @@
# Отказ PostgreSQL на пути отправки сообщения — 2026-10-16T14:10:06Z
до остановки БД, через LB : 201 (instance=828f6451261b)
до остановки БД, напрямую : 201
при остановленной БД, напрямую (ожидается 503): 503
при остановленной БД, через LB : 503 (instance=828f6451261b)
готовность после возврата БД: True
после возврата БД, через LB : 201 (instance=828f6451261b)
+5
View File
@@ -0,0 +1,5 @@
# make test-failover (прогон 1) — 2026-10-16T13:59:43Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 127, "failed_attempts": 0, "first_recovery_seconds": 0.214}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+5
View File
@@ -0,0 +1,5 @@
# make test-failover (прогон 2) — 2026-10-16T13:59:43Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 100, "failed_attempts": 0, "first_recovery_seconds": 2.237}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+54
View File
@@ -0,0 +1,54 @@
# student-tests/replica_failover.py — 2026-10-16T13:59:43Z
{
"before": {
"instances": [
"52861386fddb",
"8587b4251b72"
]
},
"replicas": [
{
"stopped": "52861386fddb",
"stopped_container": "stop-one.sh: stopped api replica(s): mtusi-messenger-lab3-api-5",
"hook_returncode": 0,
"hook_seconds": 0.151,
"hook_error": "",
"successful_writes": 91,
"successful_rps": 6.06,
"failed_attempts": 0,
"uncertain_retries": 0,
"first_recovery_seconds": 1.659,
"instances_after_recovery": [
"8587b4251b72"
],
"messages_in_history": 91,
"restored_seconds": 0.201,
"instances_restored": [
"52861386fddb",
"8587b4251b72"
]
},
{
"stopped": "8587b4251b72",
"stopped_container": "stop-one.sh: stopped api replica(s): mtusi-messenger-lab3-api-6",
"hook_returncode": 0,
"hook_seconds": 0.154,
"hook_error": "",
"successful_writes": 100,
"successful_rps": 6.66,
"failed_attempts": 0,
"uncertain_retries": 0,
"first_recovery_seconds": 1.656,
"instances_after_recovery": [
"52861386fddb"
],
"messages_in_history": 191,
"restored_seconds": 0.182,
"instances_restored": [
"52861386fddb",
"8587b4251b72"
]
}
]
}
PASS: каждая реплика остановлена по очереди; записи продолжились, дублей и потерь нет
+4
View File
@@ -0,0 +1,4 @@
# make test-idempotency — 2026-10-16T13:59:43Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 student-tests/idempotency_parallel.py --base-url "http://localhost:8080"
{"replay": {"workers": 12, "instances": ["52861386fddb", "8587b4251b72"], "message_id": "01a14502-f747-7fa8-bbad-84fa27f0ce80"}, "conflict": {"workers": 12, "accepted": 1, "conflicts": 11}, "stable_body": {"message_id": "01a14502-f771-7ea3-a03a-4d559e466ac8"}, "key_required": {"status": 400}}
PASS: параллельный повтор с одним Idempotency-Key создаёт ровно одно сообщение
+17
View File
@@ -0,0 +1,17 @@
# Артефакты предыдущих лабораторных (ЛР1 и ЛР2)
Файлы перенесены сюда из `evidence/`, потому что относятся к **прежним** конфигурациям
стенда и не являются доказательствами ЛР3: они сняты в ЛР1/ЛР2 в репозитории
`mtusi-messenger` и попали сюда вместе с переносом кода. Часть из них ссылается на
образ `mtusi2026lab1-messenger` и на in-memory хранилище, которое удалено ещё в ЛР2,
поэтому их нельзя читать как проверки текущего решения.
- `check.txt`, `quality.txt`, `concurrency.txt`, `docker.txt`, `health.txt`, `restart.txt`, `sigterm.txt` — ЛР1: офлайн-проверка структуры, линт и тесты, конкурентные чтения, запуск контейнера и обработка SIGTERM.
- `memory-experiment.*` — ЛР1: сравнение in-memory хранилища и PostgreSQL.
- `track1-benchmark/` — ЛР1, трек 1: бенчмарк append-оптимизации in-memory хранилища (скрипт `scripts/track1_benchmark.py` удалён в ЛР3 как осиротевший: он ссылался на удалённый файл хранилища).
- `cache/` — ЛР2: эксперименты с кэшем списка чатов, согласованность и поведение при отказе Redis.
- `postgres/explain-analyze.txt` — ЛР2: план запроса истории сообщений.
Доказательства ЛР3 лежат рядом, в `evidence/` (`smoke.txt`, `idempotency.txt`,
`persistence.txt`, `failover/`, `lb-readiness.txt`, `rolling-update/`,
`stand.txt`, `arch.txt`, `track1/`), и собраны в `REPORT.md`.
+11
View File
@@ -0,0 +1,11 @@
# API cache performance experiment
# generated: 2026-10-09T21:41:31Z
running trial 1, cache=False, requests=1000
{"requests": 1000, "workers": 8, "duration_seconds": 0.455918375053443, "rps": 2193.375074831716, "p50_ms": 3.3378750667907298, "p95_ms": 5.237665958702564, "p99_ms": 7.377958972938359, "trial": 1, "cache_enabled": false}
running trial 1, cache=True, requests=1000
{"requests": 1000, "workers": 8, "duration_seconds": 0.424924207967706, "rps": 2353.360861182094, "p50_ms": 3.075250016991049, "p95_ms": 5.355541012249887, "p99_ms": 6.619499996304512, "trial": 1, "cache_enabled": true}
running trial 2, cache=False, requests=1000
{"requests": 1000, "workers": 8, "duration_seconds": 0.5072490000165999, "rps": 1971.4183763147382, "p50_ms": 3.680104040540755, "p95_ms": 6.2174160266295075, "p99_ms": 8.554207975976169, "trial": 2, "cache_enabled": false}
running trial 2, cache=True, requests=1000
{"requests": 1000, "workers": 8, "duration_seconds": 0.4116047500865534, "rps": 2429.51520795063, "p50_ms": 2.9188335174694657, "p95_ms": 5.480541964061558, "p99_ms": 7.245125016197562, "trial": 2, "cache_enabled": true}
{"results": [{"requests": 1000, "workers": 8, "duration_seconds": 0.455918375053443, "rps": 2193.375074831716, "p50_ms": 3.3378750667907298, "p95_ms": 5.237665958702564, "p99_ms": 7.377958972938359, "trial": 1, "cache_enabled": false}, {"requests": 1000, "workers": 8, "duration_seconds": 0.424924207967706, "rps": 2353.360861182094, "p50_ms": 3.075250016991049, "p95_ms": 5.355541012249887, "p99_ms": 6.619499996304512, "trial": 1, "cache_enabled": true}, {"requests": 1000, "workers": 8, "duration_seconds": 0.5072490000165999, "rps": 1971.4183763147382, "p50_ms": 3.680104040540755, "p95_ms": 6.2174160266295075, "p99_ms": 8.554207975976169, "trial": 2, "cache_enabled": false}, {"requests": 1000, "workers": 8, "duration_seconds": 0.4116047500865534, "rps": 2429.51520795063, "p50_ms": 2.9188335174694657, "p95_ms": 5.480541964061558, "p99_ms": 7.245125016197562, "trial": 2, "cache_enabled": true}], "users": 4, "chats": 60, "sessions_cached_in_both_modes": true}
+36
View File
@@ -0,0 +1,36 @@
# Cache consistency evidence (track 1)
# generated: 2026-10-09T21:38:43Z
# redis: localhost:6379 db=1
=== RUN TestChatRepositorySuite
=== RUN TestChatRepositorySuite/TestBrokenCacheEntryIsDroppedAndRepopulated
=== RUN TestChatRepositorySuite/TestCacheUnavailableFallsBackToSource
=== RUN TestChatRepositorySuite/TestConcurrentColdReadsAreDeduplicated
chat_repository_test.go:226: distributed lock: 100 concurrent cold reads -> 1 source load
=== RUN TestChatRepositorySuite/TestConcurrentColdReadsWithoutProtectionStampede
chat_repository_test.go:427: stampede demonstration: 100 concurrent reads -> 100 source loads
=== RUN TestChatRepositorySuite/TestCreateInvalidatesMemberCaches
=== RUN TestChatRepositorySuite/TestDeeperPagesBypassCache
=== RUN TestChatRepositorySuite/TestEvictedCacheEntryReloadsFromSource
=== RUN TestChatRepositorySuite/TestFirstPageIsCached
=== RUN TestChatRepositorySuite/TestLimitIsPartOfCacheKey
=== RUN TestChatRepositorySuite/TestMultipleInstancesShareDistributedLock
chat_repository_test.go:276: multi-instance distributed lock: 3 instances x 30 reads -> 1 source load
=== RUN TestChatRepositorySuite/TestSlowLoaderBeyondCacheBudgetReturnsData
=== RUN TestChatRepositorySuite/TestSourceErrorIsNotRetried
--- PASS: TestChatRepositorySuite (0.83s)
--- PASS: TestChatRepositorySuite/TestBrokenCacheEntryIsDroppedAndRepopulated (0.00s)
--- PASS: TestChatRepositorySuite/TestCacheUnavailableFallsBackToSource (0.00s)
--- PASS: TestChatRepositorySuite/TestConcurrentColdReadsAreDeduplicated (0.03s)
--- PASS: TestChatRepositorySuite/TestConcurrentColdReadsWithoutProtectionStampede (0.02s)
--- PASS: TestChatRepositorySuite/TestCreateInvalidatesMemberCaches (0.01s)
--- PASS: TestChatRepositorySuite/TestDeeperPagesBypassCache (0.00s)
--- PASS: TestChatRepositorySuite/TestEvictedCacheEntryReloadsFromSource (0.00s)
--- PASS: TestChatRepositorySuite/TestFirstPageIsCached (0.00s)
--- PASS: TestChatRepositorySuite/TestLimitIsPartOfCacheKey (0.00s)
--- PASS: TestChatRepositorySuite/TestMultipleInstancesShareDistributedLock (0.05s)
--- PASS: TestChatRepositorySuite/TestSlowLoaderBeyondCacheBudgetReturnsData (0.70s)
--- PASS: TestChatRepositorySuite/TestSourceErrorIsNotRetried (0.00s)
PASS
ok mtusi-messenger/internal/storage/redis 1.338s
? mtusi-messenger/internal/storage/redis/mocks [no test files]
+22
View File
@@ -0,0 +1,22 @@
# Cache latency experiment
# generated: 2026-10-06T15:53:09Z
go version go1.27.1 darwin/arm64
Darwin arm64
2026/10/06 18:53:10 OK 00001_init.sql (10.27ms)
2026/10/06 18:53:10 goose: successfully migrated database to version: 1
=== RUN TestCacheLatencyExperiment
cache_latency_experiment_test.go:83: baseline: requests=1000 source_loads=1000 p50=317.209µs p95=343.542µs p99=377.625µs
cache_latency_experiment_test.go:84: redis cache: requests=1000 source_loads=1 p50=411.791µs p95=527.709µs p99=565.625µs
--- PASS: TestCacheLatencyExperiment (0.97s)
=== RUN TestCacheLatencyExperiment
cache_latency_experiment_test.go:83: baseline: requests=1000 source_loads=1000 p50=271.125µs p95=330.584µs p99=389.209µs
cache_latency_experiment_test.go:84: redis cache: requests=1000 source_loads=1 p50=388µs p95=453.959µs p99=512.625µs
--- PASS: TestCacheLatencyExperiment (0.86s)
=== RUN TestCacheLatencyExperiment
cache_latency_experiment_test.go:83: baseline: requests=1000 source_loads=1000 p50=268.541µs p95=318.417µs p99=367.083µs
cache_latency_experiment_test.go:84: redis cache: requests=1000 source_loads=1 p50=408.833µs p95=473.125µs p99=541.042µs
--- PASS: TestCacheLatencyExperiment (0.88s)
PASS
ok mtusi-messenger/internal/storage/postgres 3.097s
? mtusi-messenger/internal/storage/postgres/internal/sqlcgen [no test files]
? mtusi-messenger/internal/storage/postgres/migrations [no test files]
+26
View File
@@ -0,0 +1,26 @@
# Cache performance experiment
# generated: 2026-10-09T21:41:14Z
go version go1.27.1 darwin/arm64
Darwin arm64
2026/10/10 00:41:15 OK 00001_init.sql (10.03ms)
2026/10/10 00:41:15 OK 00002_user_chat_versions.sql (2.17ms)
2026/10/10 00:41:15 goose: successfully migrated database to version: 2
=== RUN TestCachePerformanceExperiment
cache_performance_experiment_test.go:115: baseline (no cache): requests=10000 concurrency=50 sql=10000 duration=1.528s rps=6542 p50=7.490875ms p95=9.302792ms p99=10.779ms
cache_performance_experiment_test.go:116: redis cache (cold): requests=10000 concurrency=50 sql=1 duration=851ms rps=11749 p50=4.192709ms p95=5.318916ms p99=5.905625ms
cache_performance_experiment_test.go:117: redis cache (hot): requests=10000 concurrency=50 sql=0 duration=751ms rps=13320 p50=3.670958ms p95=4.644958ms p99=5.21475ms
--- PASS: TestCachePerformanceExperiment (3.50s)
=== RUN TestCachePerformanceExperiment
cache_performance_experiment_test.go:115: baseline (no cache): requests=10000 concurrency=50 sql=10000 duration=1.415s rps=7066 p50=6.901958ms p95=8.697666ms p99=10.560416ms
cache_performance_experiment_test.go:116: redis cache (cold): requests=10000 concurrency=50 sql=1 duration=846ms rps=11815 p50=4.033083ms p95=6.0305ms p99=6.963417ms
cache_performance_experiment_test.go:117: redis cache (hot): requests=10000 concurrency=50 sql=0 duration=817ms rps=12234 p50=3.999959ms p95=5.060959ms p99=5.802042ms
--- PASS: TestCachePerformanceExperiment (3.39s)
=== RUN TestCachePerformanceExperiment
cache_performance_experiment_test.go:115: baseline (no cache): requests=10000 concurrency=50 sql=10000 duration=1.511s rps=6618 p50=7.302667ms p95=9.420708ms p99=12.437917ms
cache_performance_experiment_test.go:116: redis cache (cold): requests=10000 concurrency=50 sql=1 duration=827ms rps=12093 p50=3.984125ms p95=5.655167ms p99=6.8085ms
cache_performance_experiment_test.go:117: redis cache (hot): requests=10000 concurrency=50 sql=0 duration=817ms rps=12247 p50=3.946959ms p95=5.363208ms p99=6.508458ms
--- PASS: TestCachePerformanceExperiment (3.55s)
PASS
ok mtusi-messenger/internal/storage/postgres 10.978s
? mtusi-messenger/internal/storage/postgres/internal/sqlcgen [no test files]
? mtusi-messenger/internal/storage/postgres/migrations [no test files]
+8
View File
@@ -0,0 +1,8 @@
baseline: 6ms
after stop: 476ms
stop: n=1 max=476ms p95=476ms budget=2000ms
pause parallel: n=30 max=215ms p95=214ms budget=2000ms
create during pause: 201
after recovery: new chat visible, 5ms
lock keys without TTL: 0
PASS: redis outage handled within budget
+3
View File
@@ -0,0 +1,3 @@
$ make check
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
+2
View File
@@ -0,0 +1,2 @@
command: docker run + Python ThreadPoolExecutor(10), 100 POST messages
{"sent": 100, "read": 100, "unique": 100}
+11
View File
@@ -0,0 +1,11 @@
$ docker inspect --format ...
status=running health=healthy user=10001:10001
$ curl -i http://127.0.0.1:8080/health/live
HTTP/1.1 200 OK
Access-Control-Expose-Headers: X-Request-Id
Content-Type: application/json; charset=utf-8
X-Request-Id: 3d145e7d-423a-4510-9d6b-6f45b7cf552d
Date: Thu, 24 Sep 2026 20:13:14 GMT
Content-Length: 15
{"status":"ok"}
+1
View File
@@ -0,0 +1 @@
{"status":"ok"}
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 102 KiB

+8
View File
@@ -0,0 +1,8 @@
$ python3 scripts/memory_experiment.py --image mtusi2026lab1-messenger:arm64 --memory 64m --max-messages 20000 --output evidence/memory-experiment.json --plot evidence/memory-experiment.png
memory_limit=64m
successful_messages=8610
failed_requests=0
OOMKilled=true
exit_code=137
failure_time_seconds=5.9315
plot=evidence/memory-experiment.png
@@ -0,0 +1,64 @@
# EXPLAIN (ANALYZE, BUFFERS)
## history_first_page
Limit (cost=0.29..5.40 rows=51 width=73) (actual time=0.007..0.012 rows=51.00 loops=1)
Buffers: shared hit=4
-> Index Scan using messages_chat_id_created_at_id_idx on messages (cost=0.29..1002.85 rows=10000 width=73) (actual time=0.006..0.009 rows=51.00 loops=1)
Index Cond: (chat_id = '01a0f77e-9381-736f-9a2b-af3b1e1e87fd'::uuid)
Index Searches: 1
Buffers: shared hit=4
Planning:
Buffers: shared hit=44
Planning Time: 0.147 ms
Execution Time: 0.019 ms
## history_next_page
Limit (cost=0.29..7.71 rows=51 width=73) (actual time=0.004..0.010 rows=51.00 loops=1)
Buffers: shared hit=4
-> Index Scan using messages_chat_id_created_at_id_idx on messages (cost=0.29..524.01 rows=3599 width=73) (actual time=0.004..0.008 rows=51.00 loops=1)
Index Cond: ((chat_id = '01a0f77e-9381-736f-9a2b-af3b1e1e87fd'::uuid) AND (ROW(created_at, id) > ROW('2026-10-01 11:44:23.146682+00'::timestamp with time zone, '00000000-0000-0000-0000-000000000000'::uuid)))
Index Searches: 1
Buffers: shared hit=4
Planning Time: 0.021 ms
Execution Time: 0.027 ms
## chats_list_for_member
Limit (cost=0.57..446.14 rows=51 width=70) (actual time=0.018..0.081 rows=51.00 loops=1)
Buffers: shared hit=309
-> Nested Loop (cost=0.57..87375.78 rows=10001 width=70) (actual time=0.018..0.079 rows=51.00 loops=1)
Buffers: shared hit=309
-> Index Scan using chats_created_at_id_idx on chats c (cost=0.29..554.24 rows=10001 width=38) (actual time=0.004..0.007 rows=51.00 loops=1)
Index Searches: 1
Buffers: shared hit=3
-> Index Only Scan using chat_members_user_id_chat_id_idx on chat_members visible (cost=0.29..0.36 rows=1 width=16) (actual time=0.001..0.001 rows=1.00 loops=51)
Index Cond: ((user_id = '01a0f77e-937a-7d1e-aacb-8909b9a65779'::uuid) AND (chat_id = c.id))
Heap Fetches: 51
Index Searches: 51
Buffers: shared hit=153
SubPlan 1
-> Aggregate (cost=8.31..8.32 rows=1 width=32) (actual time=0.001..0.001 rows=1.00 loops=51)
Buffers: shared hit=153
-> Index Only Scan using chat_members_pkey on chat_members m (cost=0.29..8.30 rows=1 width=16) (actual time=0.000..0.000 rows=1.00 loops=51)
Index Cond: (chat_id = c.id)
Heap Fetches: 51
Index Searches: 51
Buffers: shared hit=153
Planning:
Buffers: shared hit=95
Planning Time: 0.250 ms
Execution Time: 0.098 ms
## user_by_username
Index Scan using users_username_key on users (cost=0.29..8.30 rows=1 width=42) (actual time=0.005..0.005 rows=1.00 loops=1)
Index Cond: (username = 'bulk_5000'::text)
Index Searches: 1
Buffers: shared hit=3
Planning:
Buffers: shared hit=43
Planning Time: 0.085 ms
Execution Time: 0.008 ms
+43
View File
@@ -0,0 +1,43 @@
$ make check
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
$ go vet ./...
$ make lint
/Users/vladislav/Projects/mtusi2026lab1-fork/bin/golangci-lint run -v
level=info msg="golangci-lint has version 2.13.2 built with go1.27.0 from (unknown, modified: ?, mod sum: \"h1:bCyq3E4vo9qwzifjpzJqYndEt7Ncva80qkk8G2B4TXU=\") on (unknown)"
level=info msg="[config_reader] Config search paths: [./ /Users/vladislav/Projects/mtusi2026lab1-fork /Users/vladislav/Projects /Users/vladislav /Users /]"
level=info msg="[config_reader] Used config file .golangci.yml"
level=info msg="[config_reader] Module name \"mtusi-messenger\""
level=info msg="[goenv] Read go env for 5.97475ms: map[string]string{\"GOCACHE\":\"/Users/vladislav/Library/Caches/go-build\", \"GOROOT\":\"/opt/homebrew/Cellar/go/1.27.1/libexec\"}"
level=info msg="[lintersdb] Active 50 linters: [asasalint bodyclose canonicalheader containedctx contextcheck copyloopvar depguard durationcheck errcheck errname errorlint exhaustive exptostd fatcontext forbidigo forcetypeassert gocheckcompilerdirectives gocritic gofumpt goimports gomoddirectives gosec govet ineffassign intrange makezero misspell musttag nilerr nilnil noctx nolintlint perfsprint prealloc reassign recvcheck revive rowserrcheck sloglint sqlclosecheck staticcheck testifylint thelper tparallel unconvert unparam unused usestdlibvars usetesting wastedassign]"
level=info msg="[loader] Using build tags: [integration]"
level=info msg="[loader] Go packages loading at mode 8767 (compiled_files|files|name|types_sizes|deps|exports_file|imports) took 122.006042ms"
level=info msg="[runner/filename_unadjuster] Pre-built 0 adjustments in 4.792166ms"
level=info msg="[linters_context/goanalysis] analyzers took 0s with no stages"
level=info msg="[runner/exclusion_rules] Skipped 2 issues by rules: [Path: \"_test\\\\.go\", Linters: \"gosec, prealloc, forcetypeassert, unparam\"]"
level=info msg="[runner] Issues before processing: 280, after processing: 0"
level=info msg="[runner] Processors filtering stat (in/out): exclusion_paths: 280/280, invalid_issue: 280/280, path_absoluter: 280/280, filename_unadjuster: 280/280, generated_file_filter: 280/100, exclusion_rules: 100/0, cgo: 280/280, path_relativity: 280/280"
level=info msg="[runner] processing took 607.916µs with stages: generated_file_filter: 420.959µs, exclusion_rules: 86.124µs, path_relativity: 83.917µs, sort_results: 4.458µs, invalid_issue: 4.292µs, cgo: 3.959µs, path_absoluter: 1.417µs, filename_unadjuster: 792ns, max_same_issues: 582ns, nolint_filter: 457ns, fixer: 292ns, exclusion_paths: 167ns, path_prettifier: 125ns, source_code: 125ns, path_shortener: 124ns, uniq_by_line: 42ns, severity-rules: 42ns, max_per_file_from_linter: 42ns, max_from_linter: 0s, diff: 0s"
level=info msg="[runner] linters took 422.737625ms with stages: goanalysis_metalinter: 422.101875ms"
0 issues.
level=info msg="File cache stats: 0 entries of total size 0B"
level=info msg="Memory: 7 samples, avg is 87.0MB, max is 111.2MB"
level=info msg="Execution took 555.972916ms"
$ make unit-test
go test -race -shuffle=on -count=1 -timeout=5m ./...
? mtusi-messenger/cmd/messenger [no test files]
ok mtusi-messenger/internal/config 1.838s
ok mtusi-messenger/internal/domain 1.498s
ok mtusi-messenger/internal/logger 2.150s
ok mtusi-messenger/internal/service/chat 2.470s
? mtusi-messenger/internal/service/chat/mocks [no test files]
ok mtusi-messenger/internal/service/message 2.797s
? mtusi-messenger/internal/service/message/mocks [no test files]
ok mtusi-messenger/internal/service/user 3.474s
? mtusi-messenger/internal/service/user/mocks [no test files]
ok mtusi-messenger/internal/storage/memory 3.159s
ok mtusi-messenger/internal/transport/rest 3.863s
? mtusi-messenger/internal/transport/rest/oas [no test files]
+10
View File
@@ -0,0 +1,10 @@
$ docker stop mtusi2026lab1-messenger-lab1
$ docker run -d --name mtusi2026lab1-messenger-lab1 -p 127.0.0.1:8080:8080 mtusi2026lab1-messenger:verify
$ curl -i -H "X-User-Id: <created-before-restart>" http://127.0.0.1:8080/api/v1/users/me
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
X-Request-Id: eadf9c21-d861-4c2c-8e76-46f086d27a11
Date: Thu, 24 Sep 2026 20:13:02 GMT
Content-Length: 110
{"error":{"code":"unauthorized","message":"Unauthorized","request_id":"eadf9c21-d861-4c2c-8e76-46f086d27a11"}}
+7
View File
@@ -0,0 +1,7 @@
$ docker kill --signal=TERM mtusi2026lab1-messenger-lab1
$ docker wait mtusi2026lab1-messenger-lab1
0
elapsed_ms=114
{"level":"info","time":"2026-09-24T20:14:19.192628563Z","msg":"server started","addr":"0.0.0.0:8080"}
{"level":"info","time":"2026-09-24T20:14:20.186231067Z","msg":"shutdown signal received","timeout":"10s"}
{"level":"info","time":"2026-09-24T20:14:20.188996621Z","msg":"server stopped"}
@@ -0,0 +1,21 @@
goos: darwin
goarch: arm64
pkg: mtusi-messenger/internal/storage/memory
cpu: Apple M5 Pro
BenchmarkMessageCreate/history=1000-18 3822346 309.5 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=1000-18 3904586 307.6 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=1000-18 3889698 308.1 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=1000-18 3892981 308.2 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=1000-18 3888345 308.3 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 3578911 335.4 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 3570348 335.9 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 3581305 335.0 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 3573933 336.0 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 3563053 336.4 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 3425475 349.8 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 3435330 349.1 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 3438290 349.3 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 3420394 349.8 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 3420736 350.1 ns/op 16 B/op 1 allocs/op
PASS
ok mtusi-messenger/internal/storage/memory 18.237s
@@ -0,0 +1,128 @@
[
{
"history": 100,
"concurrency": 1,
"duration_seconds": 2.002,
"successful_requests": 526,
"failed_requests": 0,
"rps": 262.791,
"p50_ms": 3.378,
"p95_ms": 4.12,
"p99_ms": 16.979,
"rss_min_kib": 18880,
"rss_max_kib": 22816,
"rss_delta_kib": 3936
},
{
"history": 100,
"concurrency": 4,
"duration_seconds": 2.002,
"successful_requests": 1450,
"failed_requests": 0,
"rps": 724.134,
"p50_ms": 4.18,
"p95_ms": 16.695,
"p99_ms": 21.604,
"rss_min_kib": 20512,
"rss_max_kib": 24208,
"rss_delta_kib": 3696
},
{
"history": 100,
"concurrency": 16,
"duration_seconds": 2.079,
"successful_requests": 2315,
"failed_requests": 0,
"rps": 1113.309,
"p50_ms": 8.563,
"p95_ms": 29.584,
"p99_ms": 93.023,
"rss_min_kib": 18880,
"rss_max_kib": 25296,
"rss_delta_kib": 6416
},
{
"history": 1000,
"concurrency": 1,
"duration_seconds": 2.007,
"successful_requests": 478,
"failed_requests": 0,
"rps": 238.14,
"p50_ms": 3.477,
"p95_ms": 10.892,
"p99_ms": 20.404,
"rss_min_kib": 23120,
"rss_max_kib": 23584,
"rss_delta_kib": 464
},
{
"history": 1000,
"concurrency": 4,
"duration_seconds": 2.003,
"successful_requests": 1380,
"failed_requests": 0,
"rps": 688.971,
"p50_ms": 4.197,
"p95_ms": 17.189,
"p99_ms": 28.458,
"rss_min_kib": 23344,
"rss_max_kib": 24592,
"rss_delta_kib": 1248
},
{
"history": 1000,
"concurrency": 16,
"duration_seconds": 2.005,
"successful_requests": 2298,
"failed_requests": 0,
"rps": 1146.063,
"p50_ms": 8.474,
"p95_ms": 29.384,
"p99_ms": 92.528,
"rss_min_kib": 23232,
"rss_max_kib": 26272,
"rss_delta_kib": 3040
},
{
"history": 5000,
"concurrency": 1,
"duration_seconds": 2.001,
"successful_requests": 494,
"failed_requests": 0,
"rps": 246.902,
"p50_ms": 3.414,
"p95_ms": 10.767,
"p99_ms": 18.996,
"rss_min_kib": 25248,
"rss_max_kib": 25296,
"rss_delta_kib": 48
},
{
"history": 5000,
"concurrency": 4,
"duration_seconds": 2.004,
"successful_requests": 1398,
"failed_requests": 0,
"rps": 697.712,
"p50_ms": 4.147,
"p95_ms": 17.142,
"p99_ms": 26.446,
"rss_min_kib": 24816,
"rss_max_kib": 25104,
"rss_delta_kib": 288
},
{
"history": 5000,
"concurrency": 16,
"duration_seconds": 2.006,
"successful_requests": 2261,
"failed_requests": 0,
"rps": 1126.862,
"p50_ms": 8.635,
"p95_ms": 24.133,
"p99_ms": 93.225,
"rss_min_kib": 25200,
"rss_max_kib": 26752,
"rss_delta_kib": 1552
}
]
@@ -0,0 +1,110 @@
[
{
"history": 100,
"concurrency": 1,
"rps_before": 262.791,
"p95_ms_before": 4.12,
"p99_ms_before": 16.979,
"rss_max_kib_before": 22816,
"rps_after": 265.189,
"p95_ms_after": 4.276,
"p99_ms_after": 17.123,
"rss_max_kib_after": 22736
},
{
"history": 100,
"concurrency": 4,
"rps_before": 724.134,
"p95_ms_before": 16.695,
"p99_ms_before": 21.604,
"rss_max_kib_before": 24208,
"rps_after": 720.55,
"p95_ms_after": 16.868,
"p99_ms_after": 21.043,
"rss_max_kib_after": 24128
},
{
"history": 100,
"concurrency": 16,
"rps_before": 1113.309,
"p95_ms_before": 29.584,
"p99_ms_before": 93.023,
"rss_max_kib_before": 25296,
"rps_after": 1125.923,
"p95_ms_after": 29.142,
"p99_ms_after": 91.221,
"rss_max_kib_after": 25216
},
{
"history": 1000,
"concurrency": 1,
"rps_before": 238.14,
"p95_ms_before": 10.892,
"p99_ms_before": 20.404,
"rss_max_kib_before": 23584,
"rps_after": 237.68,
"p95_ms_after": 11.279,
"p99_ms_after": 19.675,
"rss_max_kib_after": 23504
},
{
"history": 1000,
"concurrency": 4,
"rps_before": 688.971,
"p95_ms_before": 17.189,
"p99_ms_before": 28.458,
"rss_max_kib_before": 24592,
"rps_after": 667.05,
"p95_ms_after": 17.243,
"p99_ms_after": 29.938,
"rss_max_kib_after": 25280
},
{
"history": 1000,
"concurrency": 16,
"rps_before": 1146.063,
"p95_ms_before": 29.384,
"p99_ms_before": 92.528,
"rss_max_kib_before": 26272,
"rps_after": 1102.581,
"p95_ms_after": 30.252,
"p99_ms_after": 93.912,
"rss_max_kib_after": 25200
},
{
"history": 5000,
"concurrency": 1,
"rps_before": 246.902,
"p95_ms_before": 10.767,
"p99_ms_before": 18.996,
"rss_max_kib_before": 25296,
"rps_after": 241.241,
"p95_ms_after": 10.897,
"p99_ms_after": 18.681,
"rss_max_kib_after": 25456
},
{
"history": 5000,
"concurrency": 4,
"rps_before": 697.712,
"p95_ms_before": 17.142,
"p99_ms_before": 26.446,
"rss_max_kib_before": 25104,
"rps_after": 685.839,
"p95_ms_after": 17.418,
"p99_ms_after": 26.618,
"rss_max_kib_after": 25376
},
{
"history": 5000,
"concurrency": 16,
"rps_before": 1126.862,
"p95_ms_before": 24.133,
"p99_ms_before": 93.225,
"rss_max_kib_before": 26752,
"rps_after": 1079.866,
"p95_ms_after": 31.525,
"p99_ms_after": 94.709,
"rss_max_kib_after": 26640
}
]
@@ -0,0 +1,211 @@
{
"mode": "baseline",
"command": [
"go",
"run",
"-overlay",
"/var/folders/jd/nsgkfx9s75xb84909646j0zm0000gn/T/messenger-track1-58qai1az/overlay.json",
"./cmd/messenger"
],
"go_version": "go version go1.27.1 darwin/arm64",
"os_arch": [
"darwin",
"arm64"
],
"requests_per_case": 100000,
"rss_source": "ps RSS of go run child executable, KiB; sampled every 100 ms",
"rows": [
{
"history": 1000,
"concurrency": 1,
"duration_seconds": 13.634,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7334.409,
"p50_ms": 0.13,
"p95_ms": 0.146,
"p99_ms": 0.186,
"rss_min_kib": 23616,
"rss_max_kib": 59968,
"rss_delta_kib": 36352
},
{
"history": 1000,
"concurrency": 4,
"duration_seconds": 11.333,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 8823.924,
"p50_ms": 0.42,
"p95_ms": 0.654,
"p99_ms": 0.869,
"rss_min_kib": 23424,
"rss_max_kib": 58464,
"rss_delta_kib": 35040
},
{
"history": 1000,
"concurrency": 16,
"duration_seconds": 13.63,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7336.995,
"p50_ms": 2.021,
"p95_ms": 3.514,
"p99_ms": 4.631,
"rss_min_kib": 23520,
"rss_max_kib": 66544,
"rss_delta_kib": 43024
},
{
"history": 1000,
"concurrency": 32,
"duration_seconds": 13.538,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7386.656,
"p50_ms": 4.022,
"p95_ms": 7.026,
"p99_ms": 8.953,
"rss_min_kib": 23856,
"rss_max_kib": 65616,
"rss_delta_kib": 41760
},
{
"history": 10000,
"concurrency": 1,
"duration_seconds": 14.008,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7138.887,
"p50_ms": 0.13,
"p95_ms": 0.149,
"p99_ms": 0.21,
"rss_min_kib": 25984,
"rss_max_kib": 70416,
"rss_delta_kib": 44432
},
{
"history": 10000,
"concurrency": 4,
"duration_seconds": 11.057,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 9043.732,
"p50_ms": 0.406,
"p95_ms": 0.626,
"p99_ms": 0.841,
"rss_min_kib": 26272,
"rss_max_kib": 67376,
"rss_delta_kib": 41104
},
{
"history": 10000,
"concurrency": 16,
"duration_seconds": 13.436,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7442.895,
"p50_ms": 1.977,
"p95_ms": 3.485,
"p99_ms": 4.703,
"rss_min_kib": 26576,
"rss_max_kib": 75616,
"rss_delta_kib": 49040
},
{
"history": 10000,
"concurrency": 32,
"duration_seconds": 13.546,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7382.485,
"p50_ms": 4.01,
"p95_ms": 7.051,
"p99_ms": 9.096,
"rss_min_kib": 25456,
"rss_max_kib": 77792,
"rss_delta_kib": 52336
},
{
"history": 50000,
"concurrency": 1,
"duration_seconds": 13.969,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7158.653,
"p50_ms": 0.13,
"p95_ms": 0.149,
"p99_ms": 0.211,
"rss_min_kib": 40080,
"rss_max_kib": 81072,
"rss_delta_kib": 40992
},
{
"history": 50000,
"concurrency": 4,
"duration_seconds": 11.014,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 9079.212,
"p50_ms": 0.404,
"p95_ms": 0.623,
"p99_ms": 0.823,
"rss_min_kib": 40448,
"rss_max_kib": 77088,
"rss_delta_kib": 36640
},
{
"history": 50000,
"concurrency": 16,
"duration_seconds": 13.479,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7419.068,
"p50_ms": 1.99,
"p95_ms": 3.471,
"p99_ms": 4.694,
"rss_min_kib": 42336,
"rss_max_kib": 79552,
"rss_delta_kib": 37216
},
{
"history": 50000,
"concurrency": 32,
"duration_seconds": 13.464,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7427.333,
"p50_ms": 3.99,
"p95_ms": 7.049,
"p99_ms": 9.007,
"rss_min_kib": 40336,
"rss_max_kib": 89552,
"rss_delta_kib": 49216
}
]
}
@@ -0,0 +1,266 @@
[
{
"history": 1000,
"concurrency": 1,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 13.634,
"duration_seconds_after": 13.919,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 7334.409,
"rps_after": 7184.594,
"p50_ms_before": 0.13,
"p50_ms_after": 0.129,
"p95_ms_before": 0.146,
"p95_ms_after": 0.15,
"p99_ms_before": 0.186,
"p99_ms_after": 0.231,
"rss_max_kib_before": 59968,
"rss_max_kib_after": 62048
},
{
"history": 1000,
"concurrency": 4,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 11.333,
"duration_seconds_after": 11.115,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 8823.924,
"rps_after": 8996.523,
"p50_ms_before": 0.42,
"p50_ms_after": 0.407,
"p95_ms_before": 0.654,
"p95_ms_after": 0.632,
"p99_ms_before": 0.869,
"p99_ms_after": 0.867,
"rss_max_kib_before": 58464,
"rss_max_kib_after": 61040
},
{
"history": 1000,
"concurrency": 16,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 13.63,
"duration_seconds_after": 13.364,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 7336.995,
"rps_after": 7482.719,
"p50_ms_before": 2.021,
"p50_ms_after": 1.977,
"p95_ms_before": 3.514,
"p95_ms_after": 3.445,
"p99_ms_before": 4.631,
"p99_ms_after": 4.577,
"rss_max_kib_before": 66544,
"rss_max_kib_after": 61328
},
{
"history": 1000,
"concurrency": 32,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 13.538,
"duration_seconds_after": 13.547,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 7386.656,
"rps_after": 7381.5,
"p50_ms_before": 4.022,
"p50_ms_after": 4.016,
"p95_ms_before": 7.026,
"p95_ms_after": 7.058,
"p99_ms_before": 8.953,
"p99_ms_after": 9.054,
"rss_max_kib_before": 65616,
"rss_max_kib_after": 68768
},
{
"history": 10000,
"concurrency": 1,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 14.008,
"duration_seconds_after": 13.984,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 7138.887,
"rps_after": 7150.931,
"p50_ms_before": 0.13,
"p50_ms_after": 0.131,
"p95_ms_before": 0.149,
"p95_ms_after": 0.146,
"p99_ms_before": 0.21,
"p99_ms_after": 0.187,
"rss_max_kib_before": 70416,
"rss_max_kib_after": 68128
},
{
"history": 10000,
"concurrency": 4,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 11.057,
"duration_seconds_after": 11.046,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 9043.732,
"rps_after": 9052.965,
"p50_ms_before": 0.406,
"p50_ms_after": 0.404,
"p95_ms_before": 0.626,
"p95_ms_after": 0.628,
"p99_ms_before": 0.841,
"p99_ms_after": 0.859,
"rss_max_kib_before": 67376,
"rss_max_kib_after": 71632
},
{
"history": 10000,
"concurrency": 16,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 13.436,
"duration_seconds_after": 13.439,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 7442.895,
"rps_after": 7440.817,
"p50_ms_before": 1.977,
"p50_ms_after": 1.989,
"p95_ms_before": 3.485,
"p95_ms_after": 3.468,
"p99_ms_before": 4.703,
"p99_ms_after": 4.701,
"rss_max_kib_before": 75616,
"rss_max_kib_after": 73712
},
{
"history": 10000,
"concurrency": 32,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 13.546,
"duration_seconds_after": 13.478,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 7382.485,
"rps_after": 7419.326,
"p50_ms_before": 4.01,
"p50_ms_after": 3.984,
"p95_ms_before": 7.051,
"p95_ms_after": 7.049,
"p99_ms_before": 9.096,
"p99_ms_after": 9.096,
"rss_max_kib_before": 77792,
"rss_max_kib_after": 70976
},
{
"history": 50000,
"concurrency": 1,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 13.969,
"duration_seconds_after": 13.863,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 7158.653,
"rps_after": 7213.61,
"p50_ms_before": 0.13,
"p50_ms_after": 0.129,
"p95_ms_before": 0.149,
"p95_ms_after": 0.145,
"p99_ms_before": 0.211,
"p99_ms_after": 0.191,
"rss_max_kib_before": 81072,
"rss_max_kib_after": 81824
},
{
"history": 50000,
"concurrency": 4,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 11.014,
"duration_seconds_after": 11.066,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 9079.212,
"rps_after": 9036.335,
"p50_ms_before": 0.404,
"p50_ms_after": 0.405,
"p95_ms_before": 0.623,
"p95_ms_after": 0.63,
"p99_ms_before": 0.823,
"p99_ms_after": 0.842,
"rss_max_kib_before": 77088,
"rss_max_kib_after": 86832
},
{
"history": 50000,
"concurrency": 16,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 13.479,
"duration_seconds_after": 13.416,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 7419.068,
"rps_after": 7453.53,
"p50_ms_before": 1.99,
"p50_ms_after": 1.978,
"p95_ms_before": 3.471,
"p95_ms_after": 3.455,
"p99_ms_before": 4.694,
"p99_ms_after": 4.688,
"rss_max_kib_before": 79552,
"rss_max_kib_after": 89824
},
{
"history": 50000,
"concurrency": 32,
"requested_requests": 100000,
"warmup_requests": 100,
"duration_seconds_before": 13.464,
"duration_seconds_after": 13.542,
"successful_requests_before": 100000,
"successful_requests_after": 100000,
"failed_requests_before": 0,
"failed_requests_after": 0,
"rps_before": 7427.333,
"rps_after": 7384.536,
"p50_ms_before": 3.99,
"p50_ms_after": 4.005,
"p95_ms_before": 7.049,
"p95_ms_after": 7.084,
"p99_ms_before": 9.007,
"p99_ms_after": 9.203,
"rss_max_kib_before": 89552,
"rss_max_kib_after": 80128
}
]
@@ -0,0 +1,209 @@
{
"mode": "optimized",
"command": [
"go",
"run",
"./cmd/messenger"
],
"go_version": "go version go1.27.1 darwin/arm64",
"os_arch": [
"darwin",
"arm64"
],
"requests_per_case": 100000,
"rss_source": "ps RSS of go run child executable, KiB; sampled every 100 ms",
"rows": [
{
"history": 1000,
"concurrency": 1,
"duration_seconds": 13.919,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7184.594,
"p50_ms": 0.129,
"p95_ms": 0.15,
"p99_ms": 0.231,
"rss_min_kib": 23504,
"rss_max_kib": 62048,
"rss_delta_kib": 38544
},
{
"history": 1000,
"concurrency": 4,
"duration_seconds": 11.115,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 8996.523,
"p50_ms": 0.407,
"p95_ms": 0.632,
"p99_ms": 0.867,
"rss_min_kib": 23744,
"rss_max_kib": 61040,
"rss_delta_kib": 37296
},
{
"history": 1000,
"concurrency": 16,
"duration_seconds": 13.364,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7482.719,
"p50_ms": 1.977,
"p95_ms": 3.445,
"p99_ms": 4.577,
"rss_min_kib": 22480,
"rss_max_kib": 61328,
"rss_delta_kib": 38848
},
{
"history": 1000,
"concurrency": 32,
"duration_seconds": 13.547,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7381.5,
"p50_ms": 4.016,
"p95_ms": 7.058,
"p99_ms": 9.054,
"rss_min_kib": 23152,
"rss_max_kib": 68768,
"rss_delta_kib": 45616
},
{
"history": 10000,
"concurrency": 1,
"duration_seconds": 13.984,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7150.931,
"p50_ms": 0.131,
"p95_ms": 0.146,
"p99_ms": 0.187,
"rss_min_kib": 25936,
"rss_max_kib": 68128,
"rss_delta_kib": 42192
},
{
"history": 10000,
"concurrency": 4,
"duration_seconds": 11.046,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 9052.965,
"p50_ms": 0.404,
"p95_ms": 0.628,
"p99_ms": 0.859,
"rss_min_kib": 26448,
"rss_max_kib": 71632,
"rss_delta_kib": 45184
},
{
"history": 10000,
"concurrency": 16,
"duration_seconds": 13.439,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7440.817,
"p50_ms": 1.989,
"p95_ms": 3.468,
"p99_ms": 4.701,
"rss_min_kib": 26480,
"rss_max_kib": 73712,
"rss_delta_kib": 47232
},
{
"history": 10000,
"concurrency": 32,
"duration_seconds": 13.478,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7419.326,
"p50_ms": 3.984,
"p95_ms": 7.049,
"p99_ms": 9.096,
"rss_min_kib": 26176,
"rss_max_kib": 70976,
"rss_delta_kib": 44800
},
{
"history": 50000,
"concurrency": 1,
"duration_seconds": 13.863,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7213.61,
"p50_ms": 0.129,
"p95_ms": 0.145,
"p99_ms": 0.191,
"rss_min_kib": 40592,
"rss_max_kib": 81824,
"rss_delta_kib": 41232
},
{
"history": 50000,
"concurrency": 4,
"duration_seconds": 11.066,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 9036.335,
"p50_ms": 0.405,
"p95_ms": 0.63,
"p99_ms": 0.842,
"rss_min_kib": 42944,
"rss_max_kib": 86832,
"rss_delta_kib": 43888
},
{
"history": 50000,
"concurrency": 16,
"duration_seconds": 13.416,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7453.53,
"p50_ms": 1.978,
"p95_ms": 3.455,
"p99_ms": 4.688,
"rss_min_kib": 40768,
"rss_max_kib": 89824,
"rss_delta_kib": 49056
},
{
"history": 50000,
"concurrency": 32,
"duration_seconds": 13.542,
"requested_requests": 100000,
"warmup_requests": 100,
"successful_requests": 100000,
"failed_requests": 0,
"rps": 7384.536,
"p50_ms": 4.005,
"p95_ms": 7.084,
"p99_ms": 9.203,
"rss_min_kib": 42848,
"rss_max_kib": 80128,
"rss_delta_kib": 37280
}
]
}
@@ -0,0 +1,21 @@
goos: darwin
goarch: arm64
pkg: mtusi-messenger/internal/storage/memory
cpu: Apple M5 Pro
BenchmarkMessageCreate/history=1000-18 4830075 243.3 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=1000-18 4958697 241.4 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=1000-18 4984602 241.8 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=1000-18 4976372 241.8 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=1000-18 4962843 241.7 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 4967149 241.8 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 4946976 242.1 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 4950400 242.3 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 4938321 242.4 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=10000-18 4955269 242.2 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 4943572 242.0 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 4969927 242.4 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 4931466 242.5 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 4948990 242.8 ns/op 16 B/op 1 allocs/op
BenchmarkMessageCreate/history=50000-18 4953249 242.6 ns/op 16 B/op 1 allocs/op
PASS
ok mtusi-messenger/internal/storage/memory 18.240s
@@ -0,0 +1,128 @@
[
{
"history": 100,
"concurrency": 1,
"duration_seconds": 2.002,
"successful_requests": 531,
"failed_requests": 0,
"rps": 265.189,
"p50_ms": 3.355,
"p95_ms": 4.276,
"p99_ms": 17.123,
"rss_min_kib": 18800,
"rss_max_kib": 22736,
"rss_delta_kib": 3936
},
{
"history": 100,
"concurrency": 4,
"duration_seconds": 2.005,
"successful_requests": 1445,
"failed_requests": 0,
"rps": 720.55,
"p50_ms": 4.197,
"p95_ms": 16.868,
"p99_ms": 21.043,
"rss_min_kib": 20768,
"rss_max_kib": 24128,
"rss_delta_kib": 3360
},
{
"history": 100,
"concurrency": 16,
"duration_seconds": 2.012,
"successful_requests": 2265,
"failed_requests": 0,
"rps": 1125.923,
"p50_ms": 8.851,
"p95_ms": 29.142,
"p99_ms": 91.221,
"rss_min_kib": 20672,
"rss_max_kib": 25216,
"rss_delta_kib": 4544
},
{
"history": 1000,
"concurrency": 1,
"duration_seconds": 2.011,
"successful_requests": 478,
"failed_requests": 0,
"rps": 237.68,
"p50_ms": 3.467,
"p95_ms": 11.279,
"p99_ms": 19.675,
"rss_min_kib": 23264,
"rss_max_kib": 23504,
"rss_delta_kib": 240
},
{
"history": 1000,
"concurrency": 4,
"duration_seconds": 2.004,
"successful_requests": 1337,
"failed_requests": 0,
"rps": 667.05,
"p50_ms": 4.249,
"p95_ms": 17.243,
"p99_ms": 29.938,
"rss_min_kib": 23264,
"rss_max_kib": 25280,
"rss_delta_kib": 2016
},
{
"history": 1000,
"concurrency": 16,
"duration_seconds": 2.005,
"successful_requests": 2211,
"failed_requests": 0,
"rps": 1102.581,
"p50_ms": 8.818,
"p95_ms": 30.252,
"p99_ms": 93.912,
"rss_min_kib": 23424,
"rss_max_kib": 25200,
"rss_delta_kib": 1776
},
{
"history": 5000,
"concurrency": 1,
"duration_seconds": 2.002,
"successful_requests": 483,
"failed_requests": 0,
"rps": 241.241,
"p50_ms": 3.466,
"p95_ms": 10.897,
"p99_ms": 18.681,
"rss_min_kib": 25456,
"rss_max_kib": 25456,
"rss_delta_kib": 0
},
{
"history": 5000,
"concurrency": 4,
"duration_seconds": 2.003,
"successful_requests": 1374,
"failed_requests": 0,
"rps": 685.839,
"p50_ms": 4.207,
"p95_ms": 17.418,
"p99_ms": 26.618,
"rss_min_kib": 24304,
"rss_max_kib": 25376,
"rss_delta_kib": 1072
},
{
"history": 5000,
"concurrency": 16,
"duration_seconds": 2.002,
"successful_requests": 2162,
"failed_requests": 0,
"rps": 1079.866,
"p50_ms": 8.939,
"p95_ms": 31.525,
"p99_ms": 94.709,
"rss_min_kib": 24848,
"rss_max_kib": 26640,
"rss_delta_kib": 1792
}
]
+30
View File
@@ -0,0 +1,30 @@
# Lab4 evidence
All files were produced on the local Compose stand (`docker compose up -d --build`)
with SeaweedFS 3.80, two API replicas and HAProxy.
| File | Scenario | Command |
|---|---|---|
| `smoke.txt` | Public cumulative smoke (LAB=4) | `make test` |
| `negative.txt` | Size/format/filename/authz/limits | `make test-attachments` |
| `presign.txt` | Presigned upload happy path and negatives | `make test-presign` |
| `expiry.txt` | Expired presigned URL (TTL=3s) | `PRESIGN_EXPIRY_WAIT=4 make test-presign` |
| `failover.txt` | Upload, stop the serving replica, download via another | `python3 student-tests/attachment_failover.py` |
| `persistence.txt` | Restart the stand with volumes | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` |
| `s3-init.txt` | Idempotent bucket bootstrap, app/forbidden prefix probes | `docker compose run --rm s3-init` (twice) |
| `private-bucket.txt` | Anonymous S3 GET and published ports | `docker compose exec seaweedfs wget ...` |
| `s3-outage.txt` | S3 down: attachments 503, other endpoints 200 | `docker compose stop seaweedfs` |
| `sweep.txt` | Abandoned presign cleanup after grace; ready survives | `S3_SWEEP_GRACE=2s S3_SWEEP_INTERVAL=1s S3_PRESIGN_TTL=1s` |
| `memory/` | 10 concurrent 9.4 MiB uploads, cgroup memory samples | `python3 scripts/lab4_memory_experiment.py` |
| `track1/` | Proxy vs presigned upload: traffic, latency, memory | `python3 scripts/lab4_track1.py` |
Notes:
- Credentials are synthetic local values from `infra/s3/s3.json`; `.env` is
ignored by git.
- `memory.current` includes page cache from temporary spool files; `anon` is
the primary RAM metric.
- Presigned completion re-reads the object once to verify SHA-256, so API RX
stays close to the file size. The client-facing LB leg is the actual saving.
- Presigned URLs sign `Content-Length`: an oversized PUT is rejected by S3 with 403
before anything is stored.
+3
View File
@@ -0,0 +1,3 @@
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 student-tests/presign_flow.py --base-url "http://localhost:8080"
EXPIRED: presigned URL rejected after 4.0 seconds
PASS: lab4 presigned upload flow
+1
View File
@@ -0,0 +1 @@
PASS: lab4 attachment failover; served by 7fd3e5df907d
+7
View File
@@ -0,0 +1,7 @@
elapsed,phase,anon,current
0.378,during,281362432,400949248
0.804,during,281423872,381423616
1.395,during,281542656,403361792
1.827,during,281542656,371871744
2.376,during,281632768,411824128
2.81,during,281653248,381726720
1 elapsed phase anon current
2 0.378 during 281362432 400949248
3 0.804 during 281423872 381423616
4 1.395 during 281542656 403361792
5 1.827 during 281542656 371871744
6 2.376 during 281632768 411824128
7 2.81 during 281653248 381726720
+11
View File
@@ -0,0 +1,11 @@
Lab4 memory experiment
concurrency: 10
file_size_bytes: 9454448
uploads: 30
rounds: 3
elapsed_seconds: 2.980
baseline_anon_bytes: 280317952
peak_anon_bytes: 281653248
anon_delta_bytes: 1335296
peak_memory_current_bytes: 411824128
note: memory.current includes page cache from temporary spool files; anon is the primary metric
+14
View File
@@ -0,0 +1,14 @@
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 student-tests/attachments_negative.py --base-url "http://localhost:8080"
10 MiB + 1 body: 413
empty body: 415
corrupt PNG (IDAT CRC): 415
MIME spoof (PNG declared as JPEG): 415
dangerous filenames: 400
foreign attachment metadata/content: 404
empty text without attachments: 400
attachment from another chat: 404
11 attachment ids: 400
duplicate attachment ids: 400
attachment-only message: 201
anonymous direct S3 GET: 403
PASS: lab4 attachment negative scenarios
+3
View File
@@ -0,0 +1,3 @@
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+10
View File
@@ -0,0 +1,10 @@
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 student-tests/presign_flow.py --base-url "http://localhost:8080"
happy path: direct PUT 200, complete ready, checksum/size match
message with attachment downloaded as the second chat member
repeated complete is idempotent
foreign complete: 404
complete before PUT: 409
wrong completion checksum: 409
oversized direct PUT: 403 from S3; complete: 409 (nothing stored)
same-size overwrite: last PUT wins, checksum verified
PASS: lab4 presigned upload flow
+10
View File
@@ -0,0 +1,10 @@
=== anonymous GET ===
wget: server returned error: HTTP/1.1 403 Forbidden
exit=1
=== published ports ===
api 8080/tcp
api 8080/tcp
lb 127.0.0.1:8404->8404/tcp, 127.0.0.1:8080->80/tcp
postgres 5432/tcp
redis 6379/tcp
seaweedfs 7333/tcp, 8080/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp, 127.0.0.1:8333->8333/tcp
+14
View File
@@ -0,0 +1,14 @@
=== run 1 ===
Container mtusi-messenger-lab4-seaweedfs-1 Running
Container mtusi-messenger-lab4-seaweedfs-1 Waiting
Container mtusi-messenger-lab4-seaweedfs-1 Healthy
Container mtusi-messenger-lab4-s3-init-run-77217672cba1 Creating
Container mtusi-messenger-lab4-s3-init-run-77217672cba1 Created
{"time":"2026-10-21T20:24:44.714991263Z","level":"INFO","msg":"s3 bucket ready","endpoint":"http://seaweedfs:8333","bucket":"messenger-attachments"}
=== run 2 (idempotent) ===
Container mtusi-messenger-lab4-seaweedfs-1 Running
Container mtusi-messenger-lab4-seaweedfs-1 Waiting
Container mtusi-messenger-lab4-seaweedfs-1 Healthy
Container mtusi-messenger-lab4-s3-init-run-33ccde88f9aa Creating
Container mtusi-messenger-lab4-s3-init-run-33ccde88f9aa Created
{"time":"2026-10-21T20:24:45.578062247Z","level":"INFO","msg":"s3 bucket ready","endpoint":"http://seaweedfs:8333","bucket":"messenger-attachments"}
+8
View File
@@ -0,0 +1,8 @@
created attachment before outage: 01a1601e-9947-753c-8085-65db29e8a909
seaweedfs stopped
health/ready during outage: {'status': 'ok'}
chat list during outage: 200
download during outage: 503
upload during outage: 503
seaweedfs restarted
upload after recovery: ready 01a1601f-1c45-793b-bb59-281781ea8482
+2
View File
@@ -0,0 +1,2 @@
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 tests/smoke.py --base-url "http://localhost:8080"
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
+4
View File
@@ -0,0 +1,4 @@
ready attachment created: 01a16022-78dc-7349-b3da-2937b2faf1f0
abandoned presign attachment: 01a16022-78e3-733f-861b-0443011dcfc8 (no PUT)
abandoned pending row removed after grace
ready attachment survived sweep: ready 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
+21
View File
@@ -0,0 +1,21 @@
size,bytes,flow,iteration,latency_s,lb_rx_bytes,lb_tx_bytes,api_rx_bytes,api_tx_bytes,api_anon_peak
~1MiB,1050888,proxy,0,0.2291,1058727,1075375,1074306,1059556,284934144
~1MiB,1050888,proxy,1,0.2253,1058659,1058991,1060120,1061998,285270016
~1MiB,1050888,proxy,2,0.2328,1058925,1059123,1058798,1059912,285433856
~1MiB,1050888,proxy,3,0.227,1058485,1058633,1058297,1059591,285433856
~1MiB,1050888,proxy,4,0.2284,1057165,1057115,1057126,1058104,285638656
~1MiB,1050888,presign,0,0.2249,4098,3850,1058691,10486,285917184
~1MiB,1050888,presign,1,0.2262,4163,3981,1058060,9274,286142464
~1MiB,1050888,presign,2,0.2356,3548,3680,1057677,9240,286138368
~1MiB,1050888,presign,3,0.2297,3482,3614,1058397,10146,286183424
~1MiB,1050888,presign,4,0.2388,3482,3680,1057479,8382,286216192
~9MiB,9454448,proxy,0,0.2374,9491811,9511561,9503858,9488878,286212096
~9MiB,9454448,proxy,1,0.2397,9498015,9511259,9498331,9490528,286228480
~9MiB,9454448,proxy,2,0.2247,9492207,9511759,9493573,9486286,286261248
~9MiB,9454448,proxy,3,0.2273,9486399,9496645,9483662,9481323,286285824
~9MiB,9454448,proxy,4,0.237,9483275,9494891,9493892,9482709,286285824
~9MiB,9454448,presign,0,0.2431,4098,3916,9473045,12060,286306304
~9MiB,9454448,presign,1,0.246,3552,3680,9472208,9451,286318592
~9MiB,9454448,presign,2,0.2347,4164,3982,9473447,11088,286326784
~9MiB,9454448,presign,3,0.2346,3482,3614,9471731,9174,286343168
~9MiB,9454448,presign,4,0.2389,4098,3916,9473771,14040,286367744
1 size bytes flow iteration latency_s lb_rx_bytes lb_tx_bytes api_rx_bytes api_tx_bytes api_anon_peak
2 ~1MiB 1050888 proxy 0 0.2291 1058727 1075375 1074306 1059556 284934144
3 ~1MiB 1050888 proxy 1 0.2253 1058659 1058991 1060120 1061998 285270016
4 ~1MiB 1050888 proxy 2 0.2328 1058925 1059123 1058798 1059912 285433856
5 ~1MiB 1050888 proxy 3 0.227 1058485 1058633 1058297 1059591 285433856
6 ~1MiB 1050888 proxy 4 0.2284 1057165 1057115 1057126 1058104 285638656
7 ~1MiB 1050888 presign 0 0.2249 4098 3850 1058691 10486 285917184
8 ~1MiB 1050888 presign 1 0.2262 4163 3981 1058060 9274 286142464
9 ~1MiB 1050888 presign 2 0.2356 3548 3680 1057677 9240 286138368
10 ~1MiB 1050888 presign 3 0.2297 3482 3614 1058397 10146 286183424
11 ~1MiB 1050888 presign 4 0.2388 3482 3680 1057479 8382 286216192
12 ~9MiB 9454448 proxy 0 0.2374 9491811 9511561 9503858 9488878 286212096
13 ~9MiB 9454448 proxy 1 0.2397 9498015 9511259 9498331 9490528 286228480
14 ~9MiB 9454448 proxy 2 0.2247 9492207 9511759 9493573 9486286 286261248
15 ~9MiB 9454448 proxy 3 0.2273 9486399 9496645 9483662 9481323 286285824
16 ~9MiB 9454448 proxy 4 0.237 9483275 9494891 9493892 9482709 286285824
17 ~9MiB 9454448 presign 0 0.2431 4098 3916 9473045 12060 286306304
18 ~9MiB 9454448 presign 1 0.246 3552 3680 9472208 9451 286318592
19 ~9MiB 9454448 presign 2 0.2347 4164 3982 9473447 11088 286326784
20 ~9MiB 9454448 presign 3 0.2346 3482 3614 9471731 9174 286343168
21 ~9MiB 9454448 presign 4 0.2389 4098 3916 9473771 14040 286367744
+12
View File
@@ -0,0 +1,12 @@
Lab4 track 1: proxy vs presigned upload
| size | flow | median latency, s | p95 latency, s | LB rx, bytes | API rx, bytes | API anon peak, bytes |
|---|---|---:|---:|---:|---:|---:|
| ~1MiB | proxy | 0.228 | 0.233 | 1058392 | 1061729 | 285638656 |
| ~1MiB | presign | 0.230 | 0.239 | 3755 | 1058061 | 286216192 |
| ~9MiB | proxy | 0.237 | 0.240 | 9490341 | 9494663 | 286285824 |
| ~9MiB | presign | 0.239 | 0.246 | 3879 | 9472840 | 286367744 |
Note: presigned upload keeps the file body off the API request path and off the load balancer.
Completion still streams the object from S3 once to verify its SHA-256, so api_rx stays close to
the file size; the saving is in the client-facing LB leg and in not accepting/buffering the body.
+20
View File
@@ -0,0 +1,20 @@
# Lab5 evidence
All artifacts were produced against the local Compose stand on 2026-10-30
(2 vCPU, 2 GiB Docker VM, RabbitMQ 4.3.7, PostgreSQL 18, SeaweedFS 3.80).
- `stand.txt` — containers, image versions, VM limits.
- `smoke.txt` — public `make test`.
- `async.txt` — worker down/restart, broker restart with volume, stack restart.
- `variants.txt` — thumbnail/crop/optimized metadata, bytes, dimensions.
- `faults.txt` — permanent/transient retry, DLQ/replay, crash before/after
side effect, crash before/after ack, outbox gap, lease takeover.
- `persistence.txt`, `failover.txt`, `idempotency.txt`, `lb-readiness.txt` —
cumulative lab2–lab4 regressions.
- `track2/run.txt`, `track2/runs.csv`, `track2/summary.md` — track 2
duplicate/stale/version harness (3 repetitions).
- `unit-test.txt`, `integration-test.txt`, `vet.txt`, `lint.txt`,
`validate.txt` — build/quality gates.
No `.env`, tokens, credentials or presigned URLs are stored here. Raw SQL rows
contain only synthetic IDs and metadata.
+12
View File
@@ -0,0 +1,12 @@
# make test-queue 2026-10-31T11:26:40Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 student-tests/lab5_async.py --base-url "http://localhost:8080"
worker down: 202 queued, content 409, queued attachment linked, API restarted
worker restarted: ready, thumbnail and original downloadable, staging cleared
broker down: upload accepted into outbox
broker + relay recovered: queued job reached ready
stack restart: ready attachment and original preserved
worker down: reprocess pipeline 2 accepted, v1 audit event recorded, upload still queued
worker up: reprocessed upload reached ready at pipeline 2, v1 superseded
worker down: reprocess pipeline 3 refused, v1 job untouched
worker up: preserved upload reached ready at pipeline 1
PASS: lab5 async worker/broker durability
+25
View File
@@ -0,0 +1,25 @@
# make test-attachments 2026-10-31T11:26:14Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 student-tests/attachments_negative.py --base-url "http://localhost:8080"
10 MiB + 1 body: 413
empty body: 415
corrupt PNG (IDAT CRC): 415
MIME spoof (PNG declared as JPEG): 415
structurally valid PNG with non-zlib IDAT: 415
valid zlib, empty PNG raster: 415
valid zlib, invalid PNG scanline filter: 415
invalid PNG color type/bit depth pair: 415
duplicate PNG IHDR: 415
trailing bytes after PNG zlib stream: 415
tRNS on RGBA PNG: 415
short tRNS on RGB PNG: 415
PLTE entries exceed palette bit depth: 415
palette tRNS exceeds PLTE entries: 415
dangerous filenames: 400
foreign attachment metadata/content: 404
empty text without attachments: 400
attachment from another chat: 404
11 attachment ids: 400
duplicate attachment ids: 400
attachment-only message: 201
anonymous direct S3 GET: 403
PASS: lab4 attachment negative scenarios
+29
View File
@@ -0,0 +1,29 @@
# db state 2026-10-30T20:16:08Z
status | count
---------+-------
failed | 2
pending | 4
ready | 31
(3 rows)
status | count
--------+-------
failed | 2
ready | 34
(2 rows)
variants
----------
102
(1 row)
unpublished | outbox
-------------+--------
0 | 58
(1 row)
Timeout: 60.0 seconds ...
Listing queues for vhost messenger ...
name messages messages_ready messages_unacknowledged consumers
attachment.process.dead 4 4 0 0
attachment.process.v1 0 0 0 0
+5
View File
@@ -0,0 +1,5 @@
# failover 2026-10-30T20:11:43Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 99, "failed_attempts": 0, "first_recovery_seconds": 0.221}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+15
View File
@@ -0,0 +1,15 @@
# make test-faults 2026-10-31T11:27:03Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 student-tests/lab5_faults.py --base-url "http://localhost:8080"
permanent failure: failed/injected_failure, content 409, failed attachment not linkable, registry visible
replay: failed -> queued -> ready, one thumbnail
replay ready job: refused
replay older failed version while v2 queued: refused, state intact
v2 published after older replay refusal
replay of failed reprocess: queued projection restored, then ready at pipeline 2
transient exhaustion: attempts=5, retry_exhausted, four retries, DLQ message present
crash after first variant: redelivered, same three variants, attempts=2
crash after commit before ack: ready remains, redelivery is a no-op
crash after claim before side effect: requeued, attempts=2, ready
outbox gap: crash before publish recovered through the relay, outbox published, ready
lease expiry: paused worker, reaper requeued, stale attempt fenced, ready
PASS: lab5 level 4 fault injection
+4
View File
@@ -0,0 +1,4 @@
# idempotency 2026-10-30T20:11:43Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 student-tests/idempotency_parallel.py --base-url "http://localhost:8080"
{"replay": {"workers": 12, "instances": ["1c67613c89f2", "ee96975adb44"], "message_id": "01a18e70-8fcd-7519-91ab-d9c43b6a5f37"}, "conflict": {"workers": 12, "accepted": 1, "conflicts": 11}, "stable_body": {"message_id": "01a18e70-8ff9-759f-ab87-abd58f58c59d"}, "key_required": {"status": 400}}
PASS: параллельный повтор с одним Idempotency-Key создаёт ровно одно сообщение
+2
View File
@@ -0,0 +1,2 @@
ok mtusi-messenger/internal/storage/postgres 9.448s
ok mtusi-messenger/internal/storage/redis 1.238s
+31
View File
@@ -0,0 +1,31 @@
# LB excludes a not-ready but reachable replica (lab5)
# generated: 2026-10-30T20:18:32Z
# lb: haproxy:3.2-alpine@sha256:254e6370f01cd654e0744edf607c67f33777c1b6a9dc0d4b28bcba7171460122
## 1. baseline: requests through LB by replica
22 23d07355bd77
18 56bd7451c245
stats: api1=UP api2=MAINT (resolution) api3=UP api4=MAINT (resolution) api5=MAINT (resolution) api6=MAINT (resolution) api7=MAINT (resolution) api8=MAINT (resolution) BACKEND=UP
## 2. start a temporary TCP proxy to postgres and a candidate replica behind the same network alias
instance-id: 42c73eb4c0a5, readiness while its database is reachable: 200
load balancer slot for the candidate: api2, stats: api1=UP api2=UP 1/2 api3=UP api4=MAINT (resolution) api5=MAINT (resolution) api6=MAINT (resolution) api7=MAINT (resolution) api8=MAINT (resolution) BACKEND=UP
## 3. cut the candidate's database access (stop only the TCP proxy)
candidate is still running: running, readiness: 503
waiting for the load balancer to mark api2 DOWN
api2 went DOWN 1s after its database was cut (check: inter 2s fall 2)
stats after failure: api1=UP api2=DOWN api3=UP api4=MAINT (resolution) api5=MAINT (resolution) api6=MAINT (resolution) api7=MAINT (resolution) api8=MAINT (resolution) BACKEND=UP
## 4. 40 requests through LB while the candidate is not ready
22 23d07355bd77
18 56bd7451c245
(the candidate must not appear above)
## 5. business writes through LB during the exclusion
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 816.58,
## 6. cleanup
stats after cleanup: api1=UP api2=MAINT (resolution) api3=UP api4=MAINT (resolution) api5=MAINT (resolution) api6=MAINT (resolution) api7=MAINT (resolution) api8=MAINT (resolution) BACKEND=UP
+34
View File
@@ -0,0 +1,34 @@
# lb readiness 2026-10-30T20:18:31Z
scripts/lb_readiness_check.sh
# LB excludes a not-ready but reachable replica (lab5)
# generated: 2026-10-30T20:18:32Z
# lb: haproxy:3.2-alpine@sha256:254e6370f01cd654e0744edf607c67f33777c1b6a9dc0d4b28bcba7171460122
## 1. baseline: requests through LB by replica
22 23d07355bd77
18 56bd7451c245
stats: api1=UP api2=MAINT (resolution) api3=UP api4=MAINT (resolution) api5=MAINT (resolution) api6=MAINT (resolution) api7=MAINT (resolution) api8=MAINT (resolution) BACKEND=UP
## 2. start a temporary TCP proxy to postgres and a candidate replica behind the same network alias
instance-id: 42c73eb4c0a5, readiness while its database is reachable: 200
load balancer slot for the candidate: api2, stats: api1=UP api2=UP 1/2 api3=UP api4=MAINT (resolution) api5=MAINT (resolution) api6=MAINT (resolution) api7=MAINT (resolution) api8=MAINT (resolution) BACKEND=UP
## 3. cut the candidate's database access (stop only the TCP proxy)
candidate is still running: running, readiness: 503
waiting for the load balancer to mark api2 DOWN
api2 went DOWN 1s after its database was cut (check: inter 2s fall 2)
stats after failure: api1=UP api2=DOWN api3=UP api4=MAINT (resolution) api5=MAINT (resolution) api6=MAINT (resolution) api7=MAINT (resolution) api8=MAINT (resolution) BACKEND=UP
## 4. 40 requests through LB while the candidate is not ready
22 23d07355bd77
18 56bd7451c245
(the candidate must not appear above)
## 5. business writes through LB during the exclusion
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 816.58,
## 6. cleanup
stats after cleanup: api1=UP api2=MAINT (resolution) api3=UP api4=MAINT (resolution) api5=MAINT (resolution) api6=MAINT (resolution) api7=MAINT (resolution) api8=MAINT (resolution) BACKEND=UP
lb_readiness_check.sh: evidence in evidence/lab5/lb-readiness-detail.txt
+20
View File
@@ -0,0 +1,20 @@
/Users/vladislav/Projects/mtusi2026lab5/bin/golangci-lint run -v
level=info msg="golangci-lint has version 2.13.2 built with go1.27.1 from (unknown, modified: ?, mod sum: \"h1:bCyq3E4vo9qwzifjpzJqYndEt7Ncva80qkk8G2B4TXU=\") on (unknown)"
level=info msg="[config_reader] Config search paths: [./ /Users/vladislav/Projects/mtusi2026lab5 /Users/vladislav/Projects /Users/vladislav /Users /]"
level=info msg="[config_reader] Used config file .golangci.yml"
level=info msg="[config_reader] Module name \"mtusi-messenger\""
level=info msg="[goenv] Read go env for 6.264542ms: map[string]string{\"GOCACHE\":\"/Users/vladislav/Library/Caches/go-build\", \"GOROOT\":\"/opt/homebrew/Cellar/go/1.27.1/libexec\"}"
level=info msg="[lintersdb] Active 50 linters: [asasalint bodyclose canonicalheader containedctx contextcheck copyloopvar depguard durationcheck errcheck errname errorlint exhaustive exptostd fatcontext forbidigo forcetypeassert gocheckcompilerdirectives gocritic gofumpt goimports gomoddirectives gosec govet ineffassign intrange makezero misspell musttag nilerr nilnil noctx nolintlint perfsprint prealloc reassign recvcheck revive rowserrcheck sloglint sqlclosecheck staticcheck testifylint thelper tparallel unconvert unparam unused usestdlibvars usetesting wastedassign]"
level=info msg="[loader] Using build tags: [integration]"
level=info msg="[loader] Go packages loading at mode 8767 (compiled_files|deps|exports_file|files|types_sizes|imports|name) took 193.808375ms"
level=info msg="[runner/filename_unadjuster] Pre-built 0 adjustments in 11.667ms"
level=info msg="[linters_context/goanalysis] analyzers took 0s with no stages"
level=info msg="[runner/exclusion_rules] Skipped 14 issues by rules: [Path: \"_test\\\\.go\", Linters: \"gosec, prealloc, forcetypeassert, unparam\"]"
level=info msg="[runner] Issues before processing: 887, after processing: 0"
level=info msg="[runner] Processors filtering stat (in/out): path_absoluter: 887/887, cgo: 887/887, filename_unadjuster: 887/887, nolint_filter: 6/0, invalid_issue: 887/887, exclusion_paths: 887/887, path_relativity: 887/887, generated_file_filter: 887/298, exclusion_rules: 298/6"
level=info msg="[runner] processing took 2.828168ms with stages: generated_file_filter: 1.490334ms, nolint_filter: 764.209µs, path_relativity: 284.917µs, exclusion_rules: 251.001µs, invalid_issue: 13.417µs, cgo: 12.25µs, path_absoluter: 4.666µs, sort_results: 2.916µs, filename_unadjuster: 2.542µs, max_same_issues: 874ns, diff: 167ns, fixer: 166ns, path_prettifier: 125ns, max_per_file_from_linter: 125ns, exclusion_paths: 125ns, path_shortener: 124ns, uniq_by_line: 84ns, severity-rules: 42ns, max_from_linter: 42ns, source_code: 42ns"
level=info msg="[runner] linters took 629.148708ms with stages: goanalysis_metalinter: 626.277292ms"
0 issues.
level=info msg="File cache stats: 0 entries of total size 0B"
level=info msg="Memory: 10 samples, avg is 87.2MB, max is 111.5MB"
level=info msg="Execution took 841.622625ms"
+4
View File
@@ -0,0 +1,4 @@
# persistence 2026-10-30T20:12:05Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+11
View File
@@ -0,0 +1,11 @@
# presign 2026-10-30T20:11:41Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 student-tests/presign_flow.py --base-url "http://localhost:8080"
happy path: direct PUT 200, complete queued, ready after worker
message with attachment downloaded as the second chat member
repeated complete is idempotent
foreign complete: 404
complete before PUT: 409
wrong completion checksum: 409
oversized direct PUT: 403 from S3; complete: 409 (nothing stored)
same-size overwrite: last PUT wins, checksum verified
PASS: lab4 presigned upload flow
+3
View File
@@ -0,0 +1,3 @@
# make test 2026-10-31T11:26:59Z
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 tests/smoke.py --base-url "http://localhost:8080"
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
+20
View File
@@ -0,0 +1,20 @@
# stand 2026-10-30T20:16:07Z
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi-messenger-lab5-api-1 mtusi-messenger:lab5 "/usr/local/bin/mess…" api 42 seconds ago Up Less than a second (health: starting) 8080/tcp
mtusi-messenger-lab5-api-2 mtusi-messenger:lab5 "/usr/local/bin/mess…" api 42 seconds ago Up Less than a second (health: starting) 8080/tcp
mtusi-messenger-lab5-broker-1 rabbitmq:4.3.7-management-alpine@sha256:1528a88eebf2bc4d0ab7e9d5d9bc542e9bdf5e1b105e65848fcf8de5c2186528 "docker-entrypoint.s…" broker 8 minutes ago Up 8 seconds (healthy) 4369/tcp, 5671-5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 127.0.0.1:15672->15672/tcp
mtusi-messenger-lab5-lb-1 haproxy:3.2-alpine@sha256:254e6370f01cd654e0744edf607c67f33777c1b6a9dc0d4b28bcba7171460122 "docker-entrypoint.s…" lb 8 minutes ago Up 3 minutes 127.0.0.1:8404->8404/tcp, 127.0.0.1:8080->80/tcp
mtusi-messenger-lab5-postgres-1 postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 "docker-entrypoint.s…" postgres 8 minutes ago Up 3 minutes (healthy) 5432/tcp
mtusi-messenger-lab5-redis-1 redis:7.4-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 "docker-entrypoint.s…" redis 8 minutes ago Up 4 minutes (healthy) 6379/tcp
mtusi-messenger-lab5-relay-1 mtusi-messenger:lab5 "/usr/local/bin/relay" relay 42 seconds ago Up Less than a second (health: starting) 8080/tcp, 127.0.0.1:8082->8082/tcp
mtusi-messenger-lab5-seaweedfs-1 chrislusf/seaweedfs:3.80@sha256:1055999e08eed1789b0ae45d235126e4495e23d3fb9d6396293fd42539b1ae6a "/entrypoint.sh serv…" seaweedfs 8 minutes ago Up 3 minutes (healthy) 7333/tcp, 8080/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp, 127.0.0.1:8333->8333/tcp
mtusi-messenger-lab5-worker-1 mtusi-messenger:lab5 "/usr/local/bin/work…" worker 42 seconds ago Up Less than a second (health: starting) 8080/tcp, 127.0.0.1:8081->8081/tcp
mtusi-messenger:lab5 10b83ff52c7d
rabbitmq:4.3.7-management-alpine 1528a88eebf2
haproxy:3.2-alpine 254e6370f01c
redis:7.4-alpine 858f009f9709
postgres:18-alpine 77f585114c32
chrislusf/seaweedfs:3.80 1055999e08ee
CPUs=2 MemTotal=2053644288
+7
View File
@@ -0,0 +1,7 @@
Dduplicate_before_effect rep=1: PASS
Dstale_version_after_v2 rep=1: PASS
Dduplicate_before_effect rep=2: PASS
Dstale_version_after_v2 rep=2: PASS
Dduplicate_before_effect rep=3: PASS
Dstale_version_after_v2 rep=3: PASS
PASS: track2 6/6 scenarios
+7
View File
@@ -0,0 +1,7 @@
scenario,rep,pass,detail
duplicate_before_effect,1,1,attempts=1
stale_version_after_v2,1,1,v1_job=01a18e6e-17b0-763b-b330-7adbcb43a700
duplicate_before_effect,2,1,attempts=1
stale_version_after_v2,2,1,v1_job=01a18e6e-455e-75f3-857a-750ed660378b
duplicate_before_effect,3,1,attempts=1
stale_version_after_v2,3,1,v1_job=01a18e6e-74f2-7b4a-842b-c014c626413c
1 scenario rep pass detail
2 duplicate_before_effect 1 1 attempts=1
3 stale_version_after_v2 1 1 v1_job=01a18e6e-17b0-763b-b330-7adbcb43a700
4 duplicate_before_effect 2 1 attempts=1
5 stale_version_after_v2 2 1 v1_job=01a18e6e-455e-75f3-857a-750ed660378b
6 duplicate_before_effect 3 1 attempts=1
7 stale_version_after_v2 3 1 v1_job=01a18e6e-74f2-7b4a-842b-c014c626413c
+21
View File
@@ -0,0 +1,21 @@
# Track 2 — versioned pipeline and duplicate/stale delivery
Hypothesis: with the effect key `(attachment_id, pipeline_version, variant)`,
deterministic object keys, a serving version that changes only when all v2
variants are committed, and stale-version acknowledgements, an at-least-once
pipeline loses no accepted work and never overwrites a newer result with an
older duplicate.
Method: N=3 repetitions. Each repetition creates fresh users/chat and runs:
- duplicate pipeline publication while the worker is stopped, followed by recovery;
- v1 ready -> explicit reprocess to v2 -> old v1 message delivered after v2 ready.
Invariants checked: unique variant rows per effect key; ready remains ready;
serving pipeline version never regresses; original bytes/checksum unchanged;
stale job state is not rewritten.
Limitations: this harness covers duplicate and stale/reorder delivery. Crash
windows (before/after side effect, before/after ack, outbox gap) and lease
takeover are covered by student-tests/lab5_faults.py; both are single-node
Compose evidence, not a production availability claim. Delivery is
at-least-once; exactly-once is not claimed.
+41
View File
@@ -0,0 +1,41 @@
env -u APP_ENV -u SERVER_HOST -u SERVER_PORT -u SERVER_READ_HEADER_TIMEOUT -u SERVER_READ_TIMEOUT -u SERVER_SHUTDOWN_TIMEOUT -u SERVER_MAX_BODY_BYTES -u LOG_LEVEL -u LOG_FORMAT -u POSTGRES_HOST -u POSTGRES_PORT -u POSTGRES_DATABASE -u POSTGRES_USER -u POSTGRES_PASSWORD -u POSTGRES_SSL_MODE -u POSTGRES_MAX_CONNS -u POSTGRES_MIN_CONNS -u POSTGRES_MAX_CONN_LIFETIME -u POSTGRES_MAX_CONN_IDLE_TIME -u POSTGRES_CONNECT_TIMEOUT -u POSTGRES_QUERY_TIMEOUT -u REDIS_ENABLED -u REDIS_HOST -u REDIS_PORT -u REDIS_PASSWORD -u REDIS_DB -u REDIS_MAX_RETRIES -u REDIS_DIAL_TIMEOUT -u REDIS_READ_TIMEOUT -u REDIS_WRITE_TIMEOUT -u REDIS_POOL_SIZE -u CACHE_ENABLED -u CACHE_CHAT_LIST_TTL -u CACHE_LOCK_TTL -u CACHE_LOCK_WAIT_TIMEOUT -u SESSION_TTL_SECONDS -u IDEMPOTENCY_TTL_SECONDS -u API_REPLICAS -u API_CPU_LIMIT -u API_MEM_LIMIT -u POSTGRES_MAX_CONNS -u POSTGRES_CPU_LIMIT -u POSTGRES_MEM_LIMIT -u LB_STATS_PORT -u S3_ENDPOINT -u S3_PUBLIC_ENDPOINT -u S3_PORT -u S3_REGION -u S3_BUCKET -u S3_ACCESS_KEY_ID -u S3_SECRET_ACCESS_KEY -u MAX_UPLOAD_BYTES -u MAX_IMAGE_PIXELS -u S3_PRESIGN_TTL -u S3_SWEEP_INTERVAL -u S3_SWEEP_GRACE -u BROKER_USER -u BROKER_PASSWORD -u BROKER_VHOST -u BROKER_URL -u BROKER_EXCHANGE -u BROKER_QUEUE -u BROKER_DLX -u BROKER_DLQ -u BROKER_MANAGEMENT_PORT -u BROKER_CPU_LIMIT -u BROKER_MEM_LIMIT -u WORKER_CONCURRENCY -u WORKER_PREFETCH -u WORKER_CPU_LIMIT -u WORKER_MEM_LIMIT -u WORKER_SHUTDOWN_TIMEOUT -u PROCESSING_PIPELINE_VERSION -u PROCESSING_MAX_ATTEMPTS -u PROCESSING_JOB_TIMEOUT -u PROCESSING_LEASE_TTL -u PROCESSING_HEARTBEAT -u PROCESSING_RETRY_BASE -u PROCESSING_RETRY_CAP -u OUTBOX_POLL_INTERVAL -u OUTBOX_CLAIM_TTL -u OUTBOX_BATCH -u OUTBOX_REAPER_INTERVAL -u OUTBOX_RETENTION -u RELAY_CPU_LIMIT -u RELAY_MEM_LIMIT -u STAGING_ORPHAN_GRACE -u STAGING_FAILED_RETENTION -u ADMISSION_MAX_PENDING_PER_USER -u ADMISSION_MAX_PENDING -u ADMISSION_MAX_STAGED_BYTES go test -race -shuffle=on -count=1 -timeout=5m ./...
? mtusi-messenger/cmd/jobctl [no test files]
? mtusi-messenger/cmd/messenger [no test files]
? mtusi-messenger/cmd/migrate [no test files]
? mtusi-messenger/cmd/relay [no test files]
? mtusi-messenger/cmd/s3init [no test files]
? mtusi-messenger/cmd/worker [no test files]
? mtusi-messenger/internal/broker [no test files]
ok mtusi-messenger/internal/config 1.717s
ok mtusi-messenger/internal/domain 2.647s
? mtusi-messenger/internal/failpoint [no test files]
ok mtusi-messenger/internal/health 1.315s
ok mtusi-messenger/internal/imagecheck 3.766s
ok mtusi-messenger/internal/imageproc 2.400s
ok mtusi-messenger/internal/logger 2.338s
? mtusi-messenger/internal/outbox [no test files]
ok mtusi-messenger/internal/platform/cache 4.909s
? mtusi-messenger/internal/platform/cache/mocks [no test files]
ok mtusi-messenger/internal/platform/objectstore 6.507s
ok mtusi-messenger/internal/platform/password 3.349s
ok mtusi-messenger/internal/platform/token 3.304s
ok mtusi-messenger/internal/service/attachment 5.105s
? mtusi-messenger/internal/service/attachment/mocks [no test files]
ok mtusi-messenger/internal/service/chat 4.460s
? mtusi-messenger/internal/service/chat/mocks [no test files]
ok mtusi-messenger/internal/service/message 4.119s
? mtusi-messenger/internal/service/message/mocks [no test files]
ok mtusi-messenger/internal/service/session 5.771s
? mtusi-messenger/internal/service/session/mocks [no test files]
ok mtusi-messenger/internal/service/user 6.251s
? mtusi-messenger/internal/service/user/mocks [no test files]
ok mtusi-messenger/internal/storage/postgres 6.484s
? mtusi-messenger/internal/storage/postgres/internal/sqlcgen [no test files]
? mtusi-messenger/internal/storage/postgres/migrations [no test files]
? mtusi-messenger/internal/storage/redis [no test files]
? mtusi-messenger/internal/storage/redis/mocks [no test files]
ok mtusi-messenger/internal/transport/rest 6.893s
? mtusi-messenger/internal/transport/rest/internal/oas [no test files]
? mtusi-messenger/internal/transport/rest/mocks [no test files]
? mtusi-messenger/internal/worker [no test files]
? mtusi-messenger/tools/specgen [no test files]
+3
View File
@@ -0,0 +1,3 @@
# make validate PYTHON=.venv/bin/python 2026-10-30T22:22:13Z
.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
+3
View File
@@ -0,0 +1,3 @@
# lab5_variants 2026-10-30T20:08:36Z
variants: thumbnail 256x160, crop 128x128, optimized WebP 320x200; metadata == bytes
PASS: lab5 variant contract
+30
View File
@@ -0,0 +1,30 @@
# Lab6 evidence
- `smoke.txt` — `make test` with the telemetry profile running.
- `telemetry.txt` — `make test-observability`: Prometheus/Loki/Tempo/Grafana
queries, route-label validation, forbidden-sentinel search.
- `unit-test.txt` — `make unit-test` (`go test -race -shuffle=on`).
- `stand.txt` — `docker compose --profile telemetry ps`.
- `metrics.txt` — actual sample counts from Prometheus for the required signal set.
- `queries/promql.txt` — PromQL used by dashboards and alerts.
- `tempo-search.json`, `trace-pipeline.json` — raw Tempo responses for an
upload trace; `trace-retry.txt` — attempt=2 trace after a transient fault;
traces include API, relay `job.publish`, worker `job.process`, S3 and DB
child spans.
- Track 2 evidence is generated by `scripts/lab6_track2_full.py`; raw CSV/JSON
belong in `track2/`. `/health/*` is routed to a separate HAProxy backend and
is excluded from `api_backend` SLI. TTR SLI uses two sources: good events
from the time-to-ready histogram bucket `le="30.0"` and bad events from the
separate `messenger_jobs_ttr_missed_total` counter.
- Полный live-прогон Track 2 выполнен 2026-11-06; результаты — в
`track2/full/` и `track2/ttd-ttr.md`. Старые короткие прогоны не
используются как доказательства текущего результата.
- `storage-down.txt` / `storage-down-series.json` — telemetry-storage outage
observations over time (API continues; bounded Collector queues).
- `db-down.txt` — PostgreSQL outage: API/LB responds 5xx, never 401.
- `regression/` is for the lab5 regression commands (`test-queue`,
`test-faults`, `test-attachments`, `test-presign`, `test-idempotency`,
`test-lb-readiness`, persistence/failover).
No tokens, signed URLs, message bodies or images are stored here. The
`scripts/check_evidence.py` scanner is part of `make test-observability`.
+77
View File
@@ -0,0 +1,77 @@
[
{
"phase": "warmup",
"started_utc": "2026-11-06T08:37:06Z",
"finished_utc": "2026-11-06T09:42:56Z",
"firing": {},
"pending_not_fired": {
"APISLIMissing": 1
},
"page": 0,
"ticket": 0
},
{
"phase": "baseline",
"started_utc": "2026-11-06T09:42:56Z",
"finished_utc": "2026-11-06T10:02:51Z",
"firing": {},
"pending_not_fired": {},
"page": 0,
"ticket": 0
},
{
"phase": "burst",
"started_utc": "2026-11-06T10:02:51Z",
"finished_utc": "2026-11-06T11:29:41Z",
"firing": {},
"pending_not_fired": {
"APISLIMissing": 2
},
"page": 0,
"ticket": 0
},
{
"phase": "sustained",
"started_utc": "2026-11-06T11:29:41Z",
"finished_utc": "2026-11-06T11:49:53Z",
"firing": {
"APIErrorBudgetFastBurn": 1,
"APIErrorBudgetSlowBurn": 1,
"QueueMetricsStale": 1
},
"pending_not_fired": {},
"page": 1,
"ticket": 1
},
{
"phase": "resolution_wait",
"started_utc": "2026-11-06T11:49:53Z",
"finished_utc": "2026-11-06T12:52:06Z",
"firing": {},
"pending_not_fired": {},
"page": 0,
"ticket": 0
},
{
"phase": "ttr",
"started_utc": "2026-11-06T12:52:06Z",
"finished_utc": "2026-11-06T13:07:47Z",
"firing": {
"JobsOverdue": 1,
"QueueOldestJobAgeCritical": 1,
"QueueOldestJobAgeHigh": 1
},
"pending_not_fired": {},
"page": 1,
"ticket": 0
},
{
"phase": "false-positive",
"started_utc": "2026-11-06T13:07:47Z",
"finished_utc": "2026-11-06T13:30:24Z",
"firing": {},
"pending_not_fired": {},
"page": 0,
"ticket": 0
}
]
+11
View File
@@ -0,0 +1,11 @@
# Alert episodes by phase
| phase | page | ticket | firing | pending-not-fired |
|---|---:|---:|---|---|
| warmup | 0 | 0 | {} | {'APISLIMissing': 1} |
| baseline | 0 | 0 | {} | {} |
| burst | 0 | 0 | {} | {'APISLIMissing': 2} |
| sustained | 1 | 1 | {'APIErrorBudgetFastBurn': 1, 'APIErrorBudgetSlowBurn': 1, 'QueueMetricsStale': 1} | {} |
| resolution_wait | 0 | 0 | {} | {} |
| ttr | 1 | 0 | {'JobsOverdue': 1, 'QueueOldestJobAgeCritical': 1, 'QueueOldestJobAgeHigh': 1} | {} |
| false-positive | 0 | 0 | {} | {} |
+12
View File
@@ -0,0 +1,12 @@
# Alert episodes by phase
| phase | page | ticket | firing | pending-not-fired |
|---|---:|---:|---|---|
| warmup | 0 | 0 | {} | {'APISLIMissing': 1} |
| baseline | 0 | 0 | {} | {} |
| burst | 0 | 0 | {} | {'APISLIMissing': 2} |
| sustained | 1 | 1 | {'APIErrorBudgetFastBurn': 1, 'APIErrorBudgetSlowBurn': 1, 'QueueMetricsStale': 1} | {} |
| resolution_wait | 0 | 0 | {} | {} |
| ttr | 1 | 0 | {'JobsOverdue': 1, 'QueueOldestJobAgeCritical': 1, 'QueueOldestJobAgeHigh': 1} | {} |
| false-positive | 0 | 0 | {} | {} |
+57
View File
@@ -0,0 +1,57 @@
UTC 2026-11-06T13:34:14Z
HEAD 77fe98d775fdc07d5f9bfe04f7b4133c96c0849c
$ make test
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 tests/smoke.py --base-url "http://localhost:8080"
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
$ make test-observability
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 scripts/check_evidence.py
PASS: evidence has no obvious secrets, signed URLs or bearer tokens
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 scripts/lab6_verify.py --base-url "http://localhost:8080"
NOTE: created request_id=lab6-verify-015163e6e039495c91258218e57f1a94
PASS: lab6 strict telemetry gate (signals, trace/log correlation, labels, Grafana, rules, privacy sentinels)
$ make test-slo
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 python3 scripts/lab6_track2.py --base-url "http://localhost:8080"
Container mtusi-messenger-lab6-postgres-1 Paused
Container mtusi-messenger-lab6-postgres-1 Unpaused
{
"base_url": "http://localhost:8080",
"rate": 0.5,
"duration_s": 138.01,
"scheduled": 70,
"counters": {
"accepted": 3,
"ready": 3,
"failed": 61,
"messages": 56,
"uploads": 3,
"dropped": 0
},
"failures": 61,
"failures_by_reason": {
"http_503": 61
},
"failure_samples": [
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/messages: expected [201], received 503",
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/attachments: expected [202], received 503",
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/messages: expected [201], received 503",
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/messages: expected [201], received 503",
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/messages: expected [201], received 503",
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/messages: expected [201], received 503",
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/messages: expected [201], received 503",
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/messages: expected [201], received 503",
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/attachments: expected [202], received 503",
"Failure: POST /api/v1/chats/01a1b111-5050-7eb1-b4d5-4a156e2010f4/messages: expected [201], received 503"
],
"latency_ms": {
"p50": 6.7,
"p95": 10.8,
"p99": 10.8
},
"ttr_ms": {
"p50": 633.4,
"p95": 635.1,
"p99": 635.1
}
}
alert timeline: /Users/vladislav/Projects/mtusi2026lab6/evidence/lab6/track2/burst-alerts.csv
track2 evidence: /Users/vladislav/Projects/mtusi2026lab6/evidence/lab6/track2 load_rc= 1 alerts_rc= 0
@@ -0,0 +1,27 @@
{
"base_url": "http://localhost:8080",
"rate": 0.2,
"duration_s": 655.01,
"scheduled": 132,
"counters": {
"accepted": 27,
"ready": 27,
"failed": 0,
"messages": 105,
"uploads": 27,
"dropped": 0
},
"failures": 0,
"failures_by_reason": {},
"failure_samples": [],
"latency_ms": {
"p50": 6.2,
"p95": 9.2,
"p99": 11.9
},
"ttr_ms": {
"p50": 646.2,
"p95": 1265.8,
"p99": 1268.0
}
}
@@ -0,0 +1,29 @@
UTC 2026-11-05T22:39:30Z
HEAD 639582280f0cc2236dee3140207168df23cc8bf1
$ docker compose exec prometheus wget -qO- http://otel-collector:8888/metrics
# HELP otelcol_exporter_queue_capacity Fixed capacity of the retry queue (in batches). [Alpha]
# TYPE otelcol_exporter_queue_capacity gauge
otelcol_exporter_queue_capacity{data_type="logs",exporter="otlp_http/loki"} 1.048576e+07
otelcol_exporter_queue_capacity{data_type="traces",exporter="otlp_grpc/tempo"} 1.048576e+07
# HELP otelcol_exporter_queue_size Current size of the retry queue (in batches). [Alpha]
# TYPE otelcol_exporter_queue_size gauge
otelcol_exporter_queue_size{data_type="logs",exporter="otlp_http/loki"} 0
otelcol_exporter_queue_size{data_type="traces",exporter="otlp_grpc/tempo"} 0
# HELP otelcol_receiver_failed_log_records The number of log records that failed to be processed by the receiver due to internal errors. [Alpha]
# TYPE otelcol_receiver_failed_log_records counter
otelcol_receiver_failed_log_records{receiver="otlp",transport="http"} 0
# HELP otelcol_receiver_failed_metric_points The number of metric points that failed to be processed by the receiver due to internal errors. [Alpha]
# TYPE otelcol_receiver_failed_metric_points counter
otelcol_receiver_failed_metric_points{receiver="otlp",transport="http"} 0
# HELP otelcol_receiver_failed_spans The number of spans that failed to be processed by the receiver due to internal errors. [Alpha]
# TYPE otelcol_receiver_failed_spans counter
otelcol_receiver_failed_spans{receiver="otlp",transport="http"} 0
# HELP otelcol_receiver_refused_log_records Number of log records that could not be pushed into the pipeline. [Alpha]
# TYPE otelcol_receiver_refused_log_records counter
otelcol_receiver_refused_log_records{receiver="otlp",transport="http"} 0
# HELP otelcol_receiver_refused_metric_points Number of metric points that could not be pushed into the pipeline. [Alpha]
# TYPE otelcol_receiver_refused_metric_points counter
otelcol_receiver_refused_metric_points{receiver="otlp",transport="http"} 0
# HELP otelcol_receiver_refused_spans Number of spans that could not be pushed into the pipeline. [Alpha]
# TYPE otelcol_receiver_refused_spans counter
otelcol_receiver_refused_spans{receiver="otlp",transport="http"} 0
+6
View File
@@ -0,0 +1,6 @@
UTC 2026-11-06T13:33:57Z
HEAD 77fe98d775fdc07d5f9bfe04f7b4133c96c0849c
$ make integration-test
go test -tags=integration -p 1 -count=1 -timeout=15m ./internal/storage/postgres ./internal/storage/redis
ok mtusi-messenger/internal/storage/postgres 8.234s
ok mtusi-messenger/internal/storage/redis 1.292s
+84
View File
@@ -0,0 +1,84 @@
UTC 2026-11-06T13:33:34Z
HEAD 77fe98d775fdc07d5f9bfe04f7b4133c96c0849c
$ make check
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
python3 scripts/check_s3_creds.py
PASS: S3 app identity matches .env; admin stays in s3.json/s3-init
$ make unit-test
env -u APP_ENV -u SERVER_HOST -u SERVER_PORT -u SERVER_READ_HEADER_TIMEOUT -u SERVER_READ_TIMEOUT -u SERVER_SHUTDOWN_TIMEOUT -u SERVER_MAX_BODY_BYTES -u LOG_LEVEL -u LOG_FORMAT -u POSTGRES_HOST -u POSTGRES_PORT -u POSTGRES_DATABASE -u POSTGRES_USER -u POSTGRES_PASSWORD -u POSTGRES_SSL_MODE -u POSTGRES_MAX_CONNS -u POSTGRES_MIN_CONNS -u POSTGRES_MAX_CONN_LIFETIME -u POSTGRES_MAX_CONN_IDLE_TIME -u POSTGRES_CONNECT_TIMEOUT -u POSTGRES_QUERY_TIMEOUT -u REDIS_ENABLED -u REDIS_HOST -u REDIS_PORT -u REDIS_PASSWORD -u REDIS_DB -u REDIS_MAX_RETRIES -u REDIS_DIAL_TIMEOUT -u REDIS_READ_TIMEOUT -u REDIS_WRITE_TIMEOUT -u REDIS_POOL_SIZE -u CACHE_ENABLED -u CACHE_CHAT_LIST_TTL -u CACHE_LOCK_TTL -u CACHE_LOCK_WAIT_TIMEOUT -u SESSION_TTL_SECONDS -u IDEMPOTENCY_TTL_SECONDS -u API_REPLICAS -u API_CPU_LIMIT -u API_MEM_LIMIT -u POSTGRES_MAX_CONNS -u POSTGRES_CPU_LIMIT -u POSTGRES_MEM_LIMIT -u LB_STATS_PORT -u S3_ENDPOINT -u S3_PUBLIC_ENDPOINT -u S3_PORT -u S3_REGION -u S3_BUCKET -u S3_ACCESS_KEY_ID -u S3_SECRET_ACCESS_KEY -u MAX_UPLOAD_BYTES -u MAX_IMAGE_PIXELS -u S3_PRESIGN_TTL -u S3_SWEEP_INTERVAL -u S3_SWEEP_GRACE -u BROKER_USER -u BROKER_PASSWORD -u BROKER_VHOST -u BROKER_URL -u BROKER_EXCHANGE -u BROKER_QUEUE -u BROKER_DLX -u BROKER_DLQ -u BROKER_MANAGEMENT_PORT -u BROKER_CPU_LIMIT -u BROKER_MEM_LIMIT -u WORKER_CONCURRENCY -u WORKER_PREFETCH -u WORKER_CPU_LIMIT -u WORKER_MEM_LIMIT -u WORKER_SHUTDOWN_TIMEOUT -u WORKER_HEALTH_PORT -u PROCESSING_PIPELINE_VERSION -u PROCESSING_MAX_ATTEMPTS -u PROCESSING_JOB_TIMEOUT -u PROCESSING_LEASE_TTL -u PROCESSING_HEARTBEAT -u PROCESSING_RETRY_BASE -u PROCESSING_RETRY_CAP -u OUTBOX_POLL_INTERVAL -u OUTBOX_CLAIM_TTL -u OUTBOX_BATCH -u OUTBOX_REAPER_INTERVAL -u OUTBOX_RECONCILE_AFTER -u OUTBOX_RETENTION -u RELAY_CPU_LIMIT -u RELAY_MEM_LIMIT -u RELAY_HEALTH_PORT -u STAGING_ORPHAN_GRACE -u STAGING_FAILED_RETENTION -u ADMISSION_MAX_PENDING_PER_USER -u ADMISSION_MAX_PENDING -u ADMISSION_MAX_STAGED_BYTES -u ADMISSION_MAX_CONCURRENT_UPLOADS -u OTEL_EXPORTER_OTLP_ENDPOINT -u OTEL_EXPORTER_OTLP_PROTOCOL -u OTEL_RESOURCE_ATTRIBUTES -u OTEL_METRIC_EXPORT_INTERVAL -u TELEMETRY_ENABLED -u PROMETHEUS_PORT -u LOKI_PORT -u TEMPO_PORT -u GRAFANA_PORT -u OTEL_COLLECTOR_CPU_LIMIT -u OTEL_COLLECTOR_MEM_LIMIT -u PROMETHEUS_CPU_LIMIT -u PROMETHEUS_MEM_LIMIT -u LOKI_CPU_LIMIT -u LOKI_MEM_LIMIT -u TEMPO_CPU_LIMIT -u TEMPO_MEM_LIMIT -u GRAFANA_CPU_LIMIT -u GRAFANA_MEM_LIMIT go test -race -shuffle=on -count=1 -timeout=5m ./...
? mtusi-messenger/cmd/jobctl [no test files]
? mtusi-messenger/cmd/messenger [no test files]
? mtusi-messenger/cmd/migrate [no test files]
? mtusi-messenger/cmd/relay [no test files]
? mtusi-messenger/cmd/s3init [no test files]
? mtusi-messenger/cmd/worker [no test files]
? mtusi-messenger/internal/broker [no test files]
ok mtusi-messenger/internal/config 1.747s
ok mtusi-messenger/internal/domain 2.759s
? mtusi-messenger/internal/failpoint [no test files]
ok mtusi-messenger/internal/health 2.400s
ok mtusi-messenger/internal/imagecheck 2.085s
ok mtusi-messenger/internal/imageproc 3.626s
ok mtusi-messenger/internal/logger 3.554s
? mtusi-messenger/internal/outbox [no test files]
ok mtusi-messenger/internal/platform/cache 4.833s
? mtusi-messenger/internal/platform/cache/mocks [no test files]
? mtusi-messenger/internal/platform/instanceid [no test files]
ok mtusi-messenger/internal/platform/objectstore 3.918s
ok mtusi-messenger/internal/platform/password 5.632s
ok mtusi-messenger/internal/platform/token 4.941s
ok mtusi-messenger/internal/service/attachment 5.544s
? mtusi-messenger/internal/service/attachment/mocks [no test files]
ok mtusi-messenger/internal/service/chat 5.919s
? mtusi-messenger/internal/service/chat/mocks [no test files]
ok mtusi-messenger/internal/service/message 6.281s
? mtusi-messenger/internal/service/message/mocks [no test files]
ok mtusi-messenger/internal/service/session 6.928s
? mtusi-messenger/internal/service/session/mocks [no test files]
ok mtusi-messenger/internal/service/user 7.266s
? mtusi-messenger/internal/service/user/mocks [no test files]
ok mtusi-messenger/internal/storage/postgres 7.689s
? mtusi-messenger/internal/storage/postgres/internal/sqlcgen [no test files]
? mtusi-messenger/internal/storage/postgres/migrations [no test files]
? mtusi-messenger/internal/storage/redis [no test files]
? mtusi-messenger/internal/storage/redis/mocks [no test files]
ok mtusi-messenger/internal/telemetry 8.058s
ok mtusi-messenger/internal/transport/rest 8.529s
? mtusi-messenger/internal/transport/rest/internal/oas [no test files]
? mtusi-messenger/internal/transport/rest/mocks [no test files]
ok mtusi-messenger/internal/worker 7.565s
? mtusi-messenger/tools/specgen [no test files]
$ make lint
/Users/vladislav/Projects/mtusi2026lab6/bin/golangci-lint run -v
level=info msg="golangci-lint has version 2.13.2 built with go1.27.1 from (unknown, modified: ?, mod sum: \"h1:bCyq3E4vo9qwzifjpzJqYndEt7Ncva80qkk8G2B4TXU=\") on (unknown)"
level=info msg="[config_reader] Config search paths: [./ /Users/vladislav/Projects/mtusi2026lab6 /Users/vladislav/Projects /Users/vladislav /Users /]"
level=info msg="[config_reader] Used config file .golangci.yml"
level=info msg="[config_reader] Module name \"mtusi-messenger\""
level=info msg="[goenv] Read go env for 5.452167ms: map[string]string{\"GOCACHE\":\"/Users/vladislav/Library/Caches/go-build\", \"GOROOT\":\"/opt/homebrew/Cellar/go/1.27.1/libexec\"}"
level=info msg="[lintersdb] Active 50 linters: [asasalint bodyclose canonicalheader containedctx contextcheck copyloopvar depguard durationcheck errcheck errname errorlint exhaustive exptostd fatcontext forbidigo forcetypeassert gocheckcompilerdirectives gocritic gofumpt goimports gomoddirectives gosec govet ineffassign intrange makezero misspell musttag nilerr nilnil noctx nolintlint perfsprint prealloc reassign recvcheck revive rowserrcheck sloglint sqlclosecheck staticcheck testifylint thelper tparallel unconvert unparam unused usestdlibvars usetesting wastedassign]"
level=info msg="[loader] Using build tags: [integration]"
level=info msg="[loader] Go packages loading at mode 8767 (exports_file|types_sizes|compiled_files|deps|files|imports|name) took 328.172459ms"
level=info msg="[runner/filename_unadjuster] Pre-built 0 adjustments in 8.257709ms"
level=info msg="[linters_context/goanalysis] analyzers took 0s with no stages"
level=info msg="[runner/exclusion_rules] Skipped 15 issues by rules: [Path: \"_test\\\\.go\", Linters: \"gosec, prealloc, forcetypeassert, unparam\"]"
level=info msg="[runner] Issues before processing: 894, after processing: 0"
level=info msg="[runner] Processors filtering stat (in/out): path_absoluter: 894/894, cgo: 894/894, exclusion_paths: 894/894, exclusion_rules: 305/6, filename_unadjuster: 894/894, invalid_issue: 894/894, nolint_filter: 6/0, path_relativity: 894/894, generated_file_filter: 894/305"
level=info msg="[runner] processing took 3.308334ms with stages: generated_file_filter: 1.592083ms, nolint_filter: 799.625µs, path_relativity: 567.333µs, exclusion_rules: 281.584µs, cgo: 27.917µs, invalid_issue: 16.916µs, path_absoluter: 10.542µs, filename_unadjuster: 6.208µs, sort_results: 3.876µs, max_same_issues: 625ns, exclusion_paths: 417ns, fixer: 292ns, max_from_linter: 208ns, source_code: 125ns, path_prettifier: 125ns, path_shortener: 124ns, max_per_file_from_linter: 84ns, severity-rules: 84ns, uniq_by_line: 83ns, diff: 83ns"
level=info msg="[runner] linters took 715.361458ms with stages: goanalysis_metalinter: 712.008375ms"
0 issues.
level=info msg="File cache stats: 0 entries of total size 0B"
level=info msg="Memory: 12 samples, avg is 83.2MB, max is 111.1MB"
level=info msg="Execution took 1.057716875s"
$ go vet ./...
$ promtool check rules
Checking /etc/prometheus/rules/alerts.yml
SUCCESS: 15 rules found
Checking /etc/prometheus/rules/slo.yml
SUCCESS: 10 rules found
$ promtool test rules
SUCCESS
$ docker compose config -q
@@ -0,0 +1,27 @@
{
"base_url": "http://localhost:8080",
"rate": 0.2,
"duration_s": 775.03,
"scheduled": 156,
"counters": {
"accepted": 31,
"ready": 31,
"failed": 0,
"messages": 125,
"uploads": 31,
"dropped": 0
},
"failures": 0,
"failures_by_reason": {},
"failure_samples": [],
"latency_ms": {
"p50": 7.7,
"p95": 10.5,
"p99": 12.4
},
"ttr_ms": {
"p50": 955.9,
"p95": 1265.6,
"p99": 1266.7
}
}
File diff suppressed because it is too large Load Diff
+16
View File
@@ -0,0 +1,16 @@
UTC 2026-11-05T22:44:18Z
HEAD 639582280f0cc2236dee3140207168df23cc8bf1
$ student-tests/lab5_faults.py
permanent failure: failed/injected_failure, content 409, failed attachment not linkable, registry visible
replay: failed -> queued -> ready, one thumbnail
replay ready job: refused
replay older failed version while v2 queued: refused, state intact
v2 published after older replay refusal
replay of failed reprocess: queued projection restored, then ready at pipeline 2
transient exhaustion: attempts=5, retry_exhausted, four retries, DLQ message present
crash after first variant: redelivered, same three variants, attempts=2
crash after commit before ack: ready remains, redelivery is a no-op
crash after claim before side effect: requeued, attempts=2, ready
outbox gap: crash before publish recovered through the relay, outbox published, ready
lease expiry: paused worker, reaper requeued, stale attempt fenced, ready
PASS: lab5 level 4 fault injection

Some files were not shown because too many files have changed in this diff Show More