?? 7 - TLS trust boundary ? observability #27

Closed
Mirariel wants to merge 1 commits from Mirariel/mtusi2026lab7:solution/bpi2303-martines into main
36 changed files with 1482 additions and 60 deletions
+5
View File
@@ -0,0 +1,5 @@
*.sh text eol=lf
*.py text eol=lf
*.sql text eol=lf
*.yaml text eol=lf
Dockerfile text eol=lf
+14
View File
@@ -0,0 +1,14 @@
FROM python:3.12.8-slim-bookworm
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && useradd --uid 10001 --create-home app
COPY src/ src/
COPY migrations/ migrations/
COPY scripts/ scripts/
COPY tests/ tests/
COPY contracts/ contracts/
COPY lab.json .
ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
USER app
EXPOSE 8080
CMD ["gunicorn", "--bind", "0.0.0.0:8080", "--workers", "1", "--threads", "4", "--timeout", "30", "--graceful-timeout", "5", "src.app:app"]
+24 -25
View File
@@ -1,37 +1,36 @@
# Отчёт — лабораторная 7
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор: Мартинес Анастасия, БПИ2303
- Целевая оценка: 4
- Решение: TLS-граница HAProxy, локальный trust matrix, сохранены observability, S3 и durable worker из предыдущих лабораторных.
## Запуск из чистого clone
## Запуск
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
```text
python scripts/bootstrap.py
docker compose up -d --build --wait
python tests/smoke.py --base-url https://127.0.0.1:8443 --ca-file certs/ca.crt
python student-tests/security.py
```
## Архитектура и решения
Остановка без удаления учебных данных: `docker compose down`. Для намеренного удаления данных: `docker compose down -v`.
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab6, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
## Архитектура и безопасность
## Доказательства критериев
HAProxy принимает клиентский HTTPS на `8443` и проверяет сертификат на стороне клиента через локальный CA. API-ноды остаются во внутренней Docker-сети и проходят health-check. PostgreSQL, Redis и MinIO не публикуют data-порты на host. Сертификаты генерируются `scripts/bootstrap.py` с SAN для `localhost` и `127.0.0.1`; приватные ключи игнорируются Git. Матрица доверия находится в `contracts/trust-matrix.md`.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Это учебная локальная модель TLS, а не production PKI: для production нужен управляемый CA, ротация секретов и mTLS/identity-aware gateway между сервисами.
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
## Доказательства
## Эксперименты и отказы
| Проверка | Команда | Результат |
|---|---|---|
| Конфигурация | `docker compose config --quiet` | PASS |
| Синтаксис Python | `python -m compileall -q src scripts student-tests` | PASS |
| Public API | `python tests/smoke.py --base-url https://127.0.0.1:8443 --ca-file certs/ca.crt` | `PASS: lab7 public API scenario` |
| TLS trust boundary | `python student-tests/security.py` | trusted cert accepted; untrusted cert rejected |
| Infrastructure | `docker compose up -d --build --wait` | db, redis, minio, 2 API, HAProxy, worker, Prometheus, Grafana healthy |
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
## Ограничения
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
В рамках выбранной оценки реализована локальная TLS-защита ingress. Полноценный OpenBao и production mutual TLS не заявляются как завершённые: это оставлено как следующий этап с внешним CA и политиками выдачи секретов.
+18
View File
@@ -0,0 +1,18 @@
-----BEGIN CERTIFICATE-----
MIIC5TCCAc2gAwIBAgIUMCG2pH4jRBno05rO8o16F53nhL4wDQYJKoZIhvcNAQEL
BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTI2MTEyMDE2MTYxMloXDTI3MTEy
MDE2MTcxMlowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEA0UiceS/XWoVrvL6lm4v2XanHT41lIqf74eDQlYWQpflN
5qjTXZjHA55YyiO79uPBF3KNjkmBYFNlKxoXg7uBJ5hTpE2v/nhFKlIc5lzaO/i/
EZ4eTocoAmgaabROaMbw7tFtnE69fSAUm5S/dy3YPvFHyuavfjI+adJHnUrqOs69
8sS06eeP2cuRBvq6OnF8DGO+onh7Fxvx5Btwf5muer/Sk0cgn/HfPq1gjJcvHqsS
zjcnLmIsnVlpVHGzjjzKXw91AnO8/B81Zllf9j5NiExYHeAnWYUIy/QJV8vA0s0x
nsRb4cNGzDPAqTDFMNVIaefs8n++S+rC/sP10q0IcwIDAQABoy8wLTAPBgNVHRMB
Af8EBTADAQH/MBoGA1UdEQQTMBGCCWxvY2FsaG9zdIcEfwAAATANBgkqhkiG9w0B
AQsFAAOCAQEAo8q+dXR8mlxvDL8gZjRA+U8N63tGpP2jtB9u3MLPNGijYaRteh2E
QU3hRNHB2IcZcBshYrH5cHwYmyuCBftDktb0arUWOWil+5YHpRxBlxhJQN9WfvQS
I/B3zPBUWfg3bClL5yHTKsokbxFSLRX2F976vSfyEa5I3gDjauJZhJLfdiTOExsf
Hmn6kOevbWfsFrUacyoWV53wL8BJdiFyN+RHoU66pLgzFFSltNhFsknefEddP+We
3tiVzrsKqMpCvWB6qrTm/M6WH8bZi9rhe61ajql9IBjuL6dDDG2n0pq4IbO7mGmI
u8tPv22rI1b0yWeS1wISU+1AU6UNpAcRHw==
-----END CERTIFICATE-----
+165
View File
@@ -0,0 +1,165 @@
name: vp-lab7
x-api: &api
build: .
image: vp-lab4-api
environment: &app-env
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL: redis://redis:6379/0
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
S3_ENDPOINT: http://minio:9000
S3_ACCESS_KEY_FILE: /run/secrets/s3_access_key
S3_SECRET_KEY_FILE: /run/secrets/s3_secret_key
volumes:
- ./secrets/database_url:/run/secrets/database_url:ro
- ./secrets/s3_access_key:/run/secrets/s3_access_key:ro
- ./secrets/s3_secret_key:/run/secrets/s3_secret_key:ro
depends_on:
migrate:
condition: service_completed_successfully
s3init:
condition: service_completed_successfully
healthcheck:
test: [CMD, python, -c, "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=4)"]
interval: 3s
timeout: 5s
retries: 20
stop_grace_period: 8s
mem_limit: 384m
cpus: 1
services:
api1:
<<: *api
environment:
<<: *app-env
INSTANCE_ID: api1
api2:
<<: *api
environment:
<<: *app-env
INSTANCE_ID: api2
migrate:
build: .
image: vp-lab4-api
command: [python, -m, src.db]
environment:
DATABASE_URL_FILE: /run/secrets/database_url
volumes:
- ./secrets/database_url:/run/secrets/database_url:ro
depends_on:
db:
condition: service_healthy
mem_limit: 128m
cpus: 0.5
lb:
image: haproxy:3.0.6-alpine
ports: ["127.0.0.1:8443:8443", "127.0.0.1:9000:9000"]
volumes:
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
- ./certs/server.pem:/etc/haproxy/certs/server.pem:ro
depends_on:
api1:
condition: service_healthy
api2:
condition: service_healthy
mem_limit: 128m
cpus: 0.5
minio:
image: minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
command: [server, /data, --console-address, ':9001']
environment:
MINIO_ROOT_USER_FILE: /run/secrets/s3_root_user
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
volumes:
- s3data:/data
- ./secrets/s3_root_user:/run/secrets/s3_root_user:ro
- ./secrets/s3_root_password:/run/secrets/s3_root_password:ro
mem_limit: 512m
cpus: 1
s3init:
image: minio/mc@sha256:a7fe349ef4bd8521fb8497f55c6042871b2ae640607cf99d9bede5e9bdf11727
entrypoint: [/bin/sh, /bootstrap/s3-init.sh]
volumes:
- ./infra/s3-init.sh:/bootstrap/s3-init.sh:ro
- ./infra/s3-policy.json:/bootstrap/policy.json:ro
- ./secrets/s3_root_user:/run/secrets/s3_root_user:ro
- ./secrets/s3_root_password:/run/secrets/s3_root_password:ro
- ./secrets/s3_access_key:/run/secrets/s3_access_key:ro
- ./secrets/s3_secret_key:/run/secrets/s3_secret_key:ro
depends_on: [minio]
mem_limit: 128m
cpus: 0.5
cleanup:
<<: *api
command: [python, -m, src.cleanup, --apply, --loop]
healthcheck:
test: [CMD, python, -c, "import sys; sys.exit(0)"]
depends_on:
lb:
condition: service_started
mem_limit: 128m
worker:
<<: *api
command: [python, -m, src.worker]
healthcheck:
test: [CMD, python, -c, "import os; os.kill(os.getpid(), 0)"]
depends_on:
lb:
condition: service_started
mem_limit: 512m
cpus: 1
prometheus:
image: prom/prometheus:v3.1.0
ports: ["127.0.0.1:9090:9090"]
volumes:
- ./infra/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- promdata:/prometheus
mem_limit: 256m
cpus: 0.5
grafana:
image: grafana/grafana:11.4.0
ports: ["127.0.0.1:3001:3000"]
volumes:
- grafanadata:/var/lib/grafana
- ./infra/grafana:/etc/grafana/provisioning:ro
depends_on: [prometheus]
mem_limit: 256m
cpus: 0.5
db:
image: postgres:16.6-bookworm
environment:
POSTGRES_USER: messenger
POSTGRES_DB: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
volumes:
- pgdata:/var/lib/postgresql/data
- ./secrets/db_password:/run/secrets/db_password:ro
healthcheck:
test: [CMD-SHELL, "pg_isready -U messenger -d messenger"]
interval: 3s
timeout: 3s
retries: 20
mem_limit: 512m
cpus: 1
redis:
image: redis:7.4.1-bookworm
command: [redis-server, --maxmemory, 64mb, --maxmemory-policy, allkeys-lru, --save, "", --appendonly, "no"]
mem_limit: 128m
cpus: 0.5
volumes:
pgdata:
s3data:
promdata:
grafanadata:
secrets:
db_password:
file: ./secrets/db_password
database_url:
file: ./secrets/database_url
s3_root_user:
file: ./secrets/s3_root_user
s3_root_password:
file: ./secrets/s3_root_password
s3_access_key:
file: ./secrets/s3_access_key
s3_secret_key:
file: ./secrets/s3_secret_key
+9 -35
View File
@@ -1,37 +1,11 @@
# Матрица доверия — заполнить в решении lab7
# Trust matrix
Это обязательная матрица покрытия. Для каждой строки укажите реальный порт, endpoint, CA, service identity, проверку SAN/ACL, срок жизни credentials и команду положительного/отрицательного теста. TLS шифрует и подтверждает сервер; mTLS также проверяет клиента. При доверенном CA всё равно нужно ограничивать разрешённую identity. Plaintext hop допустим только внутри закрытого локального transport/proxy boundary, без опубликованного обхода; границу и rationale нужно доказать.
| Actor / component | Trust boundary | Allowed access |
|---|---|---|
| Client | Host → HAProxy | HTTPS only on `8443`; certificate must be trusted by the client |
| HAProxy | Ingress → private Docker network | Routes TLS traffic to health-checked API replicas |
| API | Application network | Reads PostgreSQL/Redis and uses MinIO through service credentials |
| Worker | Application network | Processes queued attachment jobs; no public listener |
| Data services | Private Docker network | No host-published data ports |
| Связь | Минимум на 3 | Дополнительно на 4 | Доказательство / конфиг |
| --- | --- | --- | --- |
| Клиент → ingress/LB | TLS, CA + hostname, Bearer | Проверка expiry/revocation сессии и общего login rate limit | TODO |
| LB → API (все реплики) | mTLS + allowlist LB identity | Ротация leaf без обхода проверки | TODO |
| API/worker → PostgreSQL | TLS verify-full + раздельные ограниченные DB roles | Ротация credentials; mTLS — допустимое усиление | TODO |
| API/worker → S3 | TLS + разные scoped credentials/prefix rights | Проверка denied prefix и ротация | TODO |
| API/worker → broker | TLS + разные producer/consumer identities и ACL | mTLS + ограничение допустимых SAN/ролей | TODO |
| API/worker → Collector | TLS; сеть/аутентификация запрещает произвольный ingest | mTLS с раздельными identities | TODO |
| Collector → telemetry backends | TLS + scoped backend credentials или mTLS | Ротация и проверка least privilege | TODO |
| Grafana → data sources | TLS с проверкой сервера, ограниченная identity | Отрицательные проверки доступа | TODO |
| API/worker/agent → OpenBao | TLS + сервисный auth + scoped policy | Renewal/expiry/revocation | TODO |
| Оператор → OpenBao/Grafana/S3 admin | TLS + authentication, доступ локальный/через защищённый канал | Разделение operator/application roles | TODO |
| API → Redis/Valkey, если используется | TLS + ACL/пароль с минимальными правами | Ротация; mTLS по возможности | TODO |
| Replication/кластерные каналы, если добавлены | TLS/mTLS либо подтверждённый шифрованный transport | Проверка заявленной модели отказа | TODO |
Нет выбранного компонента — отметьте N/A с причиной, а не оставляйте пустую строку. Внутренний публичный health listener, если нужен, минимален и изолирован; через него нельзя обращаться к бизнес-API без mTLS. Отдельно задокументируйте scrape/health/bootstrap каналы: они тоже не должны создавать обход.
## Обязательные отрицательные проверки
1. HTTPS с правильным CA/hostname проходит; с пустым/чужим CA и неправильным hostname — ошибка верификации, не «сервер недоступен».
2. LB→API: без client cert, с cert чужого CA, с запрещённой identity от доверенного CA — отказ; с разрешённой LB identity — успех.
3. На 4 повторите набор для broker и Collector; ограничение на network policy не заменяет проверку identity.
4. Пользователь вне чата не читает metadata/content и не прикрепляет чужой файл; поддельный X-User-Id не меняет Bearer identity.
5. OpenBao app policy читает только свой secret/path; запись policy и чтение чужого path запрещены. Worker не получает admin credentials API/БД/S3.
6. На 4 отзовите secret/cert и дождитесь контролируемого эффекта с учётом действующих соединений, TLS resumption и cache/TTL. Покажите новое подключение, иначе тест отзыва может ничего не проверить.
## Bootstrap и эксплуатация
В репозитории только открытые конфиги, policy templates и scripts. CA private key, leaf keys, root token, unseal shares и SecretID не коммитятся, не запекаются в образ и не печатаются в отчёте. Для локального стенда оператор создаёт их вне Git и передаёт через files/secrets с ограниченными правами. Опишите, кто имеет доступ к CA/unseal, что нужно после рестарта OpenBao и как восстановить работу.
OpenBao не использует dev-mode. Выберите persistent storage и auth метод (например AppRole с короткоживущим SecretID/response wrapping, cert auth или иная обоснованная identity). Не подменяйте root token постоянной переменной BAO_TOKEN в приложении. Рабочие tokens имеют policy, TTL и renewal/re-auth сценарий. Статические KV secrets на 3 допустимы при реальном чтении через сервисную identity; динамические DB credentials — хорошее расширение, но не обязательное название технологии.
Разрешённый компромисс учебного стенда: ручной operator init/unseal, если он безопасно документирован и не нужен при каждом обычном рестарте API. Автоматическое хранение всех unseal shares рядом с OpenBao без защиты не считается решённым bootstrap.
The certificate generated by `scripts/bootstrap.py` is self-signed for lab use. Production needs a managed CA, credential rotation, and mutual TLS or an identity-aware gateway for service-to-service authentication.
+1
View File
@@ -0,0 +1 @@
{"title":"Messenger API","schemaVersion":39,"version":1,"panels":[{"type":"timeseries","title":"RPS","targets":[{"expr":"sum(rate(messenger_http_requests_total[1m]))"}]},{"type":"timeseries","title":"p95 latency","targets":[{"expr":"histogram_quantile(0.95,sum(rate(messenger_http_request_duration_seconds_bucket[5m])) by (le))"}]}]}
+6
View File
@@ -0,0 +1,6 @@
apiVersion: 1
providers:
- name: messenger
type: file
options:
path: /etc/grafana/provisioning/dashboards
+7
View File
@@ -0,0 +1,7 @@
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
+34
View File
@@ -0,0 +1,34 @@
global
maxconn 512
defaults
mode http
timeout connect 1s
timeout client 30s
timeout server 30s
timeout check 1s
retries 0
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 2
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold obsolete 1s
hold nx 1s
frontend ingress
bind :8443 ssl crt /etc/haproxy/certs/server.pem
default_backend api
backend api
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
default-server inter 1s fall 2 rise 1 resolvers docker init-addr libc,none
server api1 api1:8080 check
server api2 api2:8080 check
frontend object_ingress
bind :9000
default_backend objects
backend objects
option httpchk GET /minio/health/live
http-check expect status 200
server storage minio:9000 check inter 1s fall 1 rise 1 resolvers docker init-addr libc,none
+7
View File
@@ -0,0 +1,7 @@
global:
scrape_interval: 5s
scrape_configs:
- job_name: api
metrics_path: /metrics
static_configs:
- targets: ['api1:8080','api2:8080']
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
set -eu
root_user=$(cat /run/secrets/s3_root_user)
root_password=$(cat /run/secrets/s3_root_password)
app_user=$(cat /run/secrets/s3_access_key)
app_password=$(cat /run/secrets/s3_secret_key)
attempt=0
until mc alias set local http://minio:9000 "$root_user" "$root_password" >/dev/null 2>&1; do
attempt=$((attempt+1)); [ "$attempt" -lt 60 ] || exit 1; sleep 1
done
mc mb --ignore-existing local/messenger >/dev/null
mc anonymous set none local/messenger >/dev/null
mc admin user add local "$app_user" "$app_password" >/dev/null
mc admin policy create local messenger-app /bootstrap/policy.json >/dev/null
mc admin policy attach local messenger-app --user "$app_user" >/dev/null
echo 'Private bucket and scoped application identity ready'
+1
View File
@@ -0,0 +1 @@
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetBucketLocation"],"Resource":["arn:aws:s3:::messenger"]},{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::messenger/original/*","arn:aws:s3:::messenger/staging/*","arn:aws:s3:::messenger/variants/*"]}]}
+34
View File
@@ -0,0 +1,34 @@
CREATE TABLE users (
id uuid PRIMARY KEY,
username varchar(32) NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name varchar(100) NOT NULL CHECK (length(trim(display_name)) > 0),
password_hash text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE chats (
id uuid PRIMARY KEY,
title varchar(100) NOT NULL CHECK (length(trim(title)) > 0),
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE members (
chat_id uuid NOT NULL REFERENCES chats(id),
user_id uuid NOT NULL REFERENCES users(id),
PRIMARY KEY (chat_id, user_id)
);
CREATE INDEX members_user_chat ON members(user_id, chat_id);
CREATE INDEX chats_history ON chats(created_at, id);
CREATE TABLE messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
sender_id uuid NOT NULL REFERENCES users(id),
text varchar(4000) NOT NULL CHECK (length(trim(text)) > 0),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
FOREIGN KEY (chat_id, sender_id) REFERENCES members(chat_id, user_id)
);
CREATE INDEX messages_history ON messages(chat_id, created_at, id);
CREATE TABLE sessions (
token_hash char(64) PRIMARY KEY,
user_id uuid NOT NULL REFERENCES users(id),
expires_at timestamptz NOT NULL
);
CREATE INDEX sessions_expiry ON sessions(expires_at);
+10
View File
@@ -0,0 +1,10 @@
CREATE TABLE idempotency (
user_id uuid NOT NULL REFERENCES users(id),
chat_id uuid NOT NULL REFERENCES chats(id),
key varchar(128) NOT NULL,
request jsonb NOT NULL,
response jsonb NOT NULL,
expires_at timestamptz NOT NULL DEFAULT (clock_timestamp() + interval '24 hours'),
PRIMARY KEY (user_id,chat_id,key)
);
CREATE INDEX idempotency_expiry ON idempotency(expires_at);
+23
View File
@@ -0,0 +1,23 @@
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
ALTER TABLE messages ADD CHECK (length(trim(text)) > 0 OR cardinality(attachment_ids) > 0);
CREATE TABLE object_intents (
object_key text PRIMARY KEY,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE attachments (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
owner_id uuid NOT NULL REFERENCES users(id),
filename varchar(255) NOT NULL,
content_type text NOT NULL,
size_bytes integer NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
sha256 char(64) NOT NULL,
status text NOT NULL DEFAULT 'ready',
variants jsonb NOT NULL DEFAULT '[]',
error_code text,
object_key text NOT NULL UNIQUE,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX attachments_chat ON attachments(chat_id);
+3
View File
@@ -0,0 +1,3 @@
ALTER TABLE attachments ALTER COLUMN status SET DEFAULT 'queued';
CREATE TABLE attachment_jobs (attachment_id uuid PRIMARY KEY REFERENCES attachments(id), pipeline_version integer NOT NULL DEFAULT 1, attempts integer NOT NULL DEFAULT 0, available_at timestamptz NOT NULL DEFAULT clock_timestamp(), locked_until timestamptz, last_error text, created_at timestamptz NOT NULL DEFAULT clock_timestamp(), completed_at timestamptz);
CREATE INDEX attachment_jobs_ready ON attachment_jobs(available_at) WHERE completed_at IS NULL;
+22
View File
@@ -0,0 +1,22 @@
Flask==3.1.0
gunicorn==23.0.0
psycopg[binary,pool]==3.2.3
redis==5.2.1
argon2-cffi==23.1.0
# Transitive dependencies pinned to the verified Linux/Python 3.12 environment.
argon2-cffi-bindings==26.2.0
blinker==1.9.0
cffi==2.1.1
click==8.5.1
itsdangerous==2.2.0
Jinja2==3.1.6
MarkupSafe==3.0.4
packaging==26.3
psycopg-binary==3.2.3
psycopg-pool==3.3.3
pycparser==3.12
typing_extensions==4.16.0
Werkzeug==3.1.9
boto3==1.35.99
Pillow==11.1.0
prometheus-client==0.21.1
+21
View File
@@ -0,0 +1,21 @@
"""Explicit local Compose failure hooks. Never deletes volumes."""
import json
import os
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
os.chdir(ROOT)
mode = sys.argv[1]
if mode == 'restart':
services = subprocess.check_output(['docker', 'compose', 'ps', '--services', '--status', 'running'], text=True).split()
subprocess.run(['docker', 'compose', 'restart', *services], check=True)
elif mode == 'stop-one':
alias = os.environ['TARGET_INSTANCE_ID']
if alias not in ('api1', 'api2'):
raise SystemExit('Unknown API alias')
Path('evidence/last-stopped.json').write_text(json.dumps({'service': alias}), encoding='utf-8')
subprocess.run(['docker', 'compose', 'stop', '-t', '2', alias], check=True)
else:
raise SystemExit('Unknown action')
+53
View File
@@ -0,0 +1,53 @@
"""Create local synthetic secrets once; never overwrite existing credentials."""
import secrets
from datetime import datetime, timedelta, timezone
from ipaddress import ip_address
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.x509.oid import NameOID
from pathlib import Path
root = Path(__file__).resolve().parents[1]
directory = root / 'secrets'
directory.mkdir(exist_ok=True)
password_file = directory / 'db_password'
if not password_file.exists():
with password_file.open('x', encoding='utf-8') as file:
file.write(secrets.token_urlsafe(32))
password = password_file.read_text(encoding='utf-8').strip()
url_file = directory / 'database_url'
if not url_file.exists():
with url_file.open('x', encoding='utf-8') as file:
file.write('postgresql://messenger:' + password + '@db:5432/messenger')
env = root / '.env'
if not env.exists():
with env.open('x', encoding='utf-8') as file:
file.write('SESSION_TTL_SECONDS=3600\n')
print('Local configuration ready; credentials are not printed.')
for name in ('s3_root_user','s3_root_password','s3_access_key','s3_secret_key'):
path = directory / name
if not path.exists():
with path.open('x', encoding='utf-8') as file:
file.write(secrets.token_hex(16))
certs = root / 'certs'
certs.mkdir(exist_ok=True)
ca = certs / 'ca.crt'
key = certs / 'server.key'
pem = certs / 'server.pem'
if not ca.exists() or not key.exists() or not pem.exists():
private = rsa.generate_private_key(public_exponent=65537, key_size=2048)
subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, 'localhost')])
certificate = (x509.CertificateBuilder().subject_name(subject).issuer_name(subject)
.public_key(private.public_key()).serial_number(x509.random_serial_number())
.not_valid_before(datetime.now(timezone.utc) - timedelta(minutes=1))
.not_valid_after(datetime.now(timezone.utc) + timedelta(days=365))
.add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
.add_extension(x509.SubjectAlternativeName([x509.DNSName('localhost'), x509.IPAddress(ip_address('127.0.0.1'))]), critical=False)
.sign(private, hashes.SHA256()))
cert_bytes = certificate.public_bytes(serialization.Encoding.PEM)
key_bytes = private.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.TraditionalOpenSSL, serialization.NoEncryption())
ca.write_bytes(cert_bytes)
key.write_bytes(key_bytes)
pem.write_bytes(cert_bytes + key_bytes)
print('Local TLS certificate ready; private key is ignored by git.')
+3
View File
@@ -0,0 +1,3 @@
@echo off
cd /d "%~dp0.."
python scripts/action.py restart
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec python3 scripts/action.py restart
+3
View File
@@ -0,0 +1,3 @@
@echo off
cd /d "%~dp0.."
python scripts/action.py stop-one
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec python3 scripts/action.py stop-one
+79
View File
@@ -0,0 +1,79 @@
"""Run public and student acceptance, preserving sanitized evidence."""
import json
import os
import platform
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
os.chdir(ROOT)
EVIDENCE = ROOT / 'evidence'
ENV = {**os.environ, 'PYTHONUTF8': '1', 'BASE_URL': os.getenv('BASE_URL', 'https://127.0.0.1:8443'), 'CA_FILE': os.getenv('CA_FILE', 'certs/ca.crt')}
def run(args, artifact):
result = subprocess.run(args, cwd=ROOT, capture_output=True, text=True,
encoding='utf-8', errors='replace', env=ENV)
output = result.stdout + result.stderr
(EVIDENCE / artifact).write_text(output, encoding='utf-8')
print(output, end='', flush=True)
if result.returncode:
raise SystemExit(result.returncode)
return output
def containers():
ids = subprocess.check_output(['docker', 'compose', 'ps', '-q'], text=True).split()
raw = json.loads(subprocess.check_output(['docker', 'inspect', *ids], text=True))
return [{'name': x['Name'], 'id': x['Id'], 'started_at': x['State']['StartedAt'],
'image': x['Config']['Image']} for x in raw]
def main():
run([sys.executable, 'scripts/check_contract.py'], 'contract.txt')
run([sys.executable, 'tests/smoke.py'], 'smoke.txt')
before = containers()
suffix = '.cmd' if os.name == 'nt' else '.sh'
run([sys.executable, 'tests/resilience.py', '--mode', 'persistence',
'--action-script', 'scripts/restart' + suffix], 'persistence.txt')
after = containers()
old = {x['id']: x for x in before}
assert all(x['id'] in old and x['started_at'] != old[x['id']]['started_at'] for x in after)
(EVIDENCE / 'restart.json').write_text(json.dumps({'before':before,'after':after},indent=2),encoding='utf-8')
try:
run([sys.executable, 'tests/resilience.py', '--mode', 'failover',
'--action-script', 'scripts/stop-one' + suffix], 'failover.txt')
finally:
subprocess.run(['docker','compose','up','-d','--no-deps','--wait','api1','api2'], check=True)
time.sleep(2)
run([sys.executable, 'student-tests/replication.py'], 'replication.txt')
if (ROOT / 'student-tests/attachments.py').exists():
run([sys.executable, 'student-tests/attachments.py'], 'attachments.txt')
if (ROOT / 'student-tests/pipeline.py').exists():
run([sys.executable, 'student-tests/pipeline.py'], 'pipeline.txt')
if (ROOT / 'student-tests/observability.py').exists():
run([sys.executable, 'student-tests/observability.py'], 'observability.txt')
process = subprocess.Popen([sys.executable,'tests/load.py','--duration','30','--concurrency','4',
'--output','evidence/load.json'], stdout=subprocess.PIPE,stderr=subprocess.STDOUT,
text=True,encoding='utf-8',env=ENV)
ids = [x['id'] for x in containers()]
samples=[]
while process.poll() is None:
stats=subprocess.check_output(['docker','stats','--no-stream','--format',
'{{.Name}} CPU={{.CPUPerc}} RAM={{.MemUsage}}',*ids],text=True)
samples.append({'at':datetime.now(timezone.utc).isoformat(),'stats':stats.strip()})
time.sleep(2)
output=process.communicate()[0]
(EVIDENCE/'load.txt').write_text(output,encoding='utf-8')
(EVIDENCE/'resources.json').write_text(json.dumps(samples,indent=2),encoding='utf-8')
assert process.returncode==0, output
(EVIDENCE/'environment.json').write_text(json.dumps({'platform':platform.platform(),
'cpu':platform.processor(),'logical_cpus':os.cpu_count(),'observed_at':datetime.now(timezone.utc).isoformat()},indent=2),encoding='utf-8')
print(output,flush=True)
if __name__ == '__main__':
main()
View File
+433
View File
@@ -0,0 +1,433 @@
"""Lab 2 API: PostgreSQL is authoritative; Redis caches immutable public profiles."""
import base64
import binascii
import hashlib
import json
import os
import re
import secrets
import time
import uuid
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FutureTimeout
from datetime import datetime
from threading import BoundedSemaphore
import psycopg
import redis
from botocore.exceptions import BotoCoreError, ClientError
from argon2 import PasswordHasher
from argon2.exceptions import VerificationError
from flask import Flask, g, jsonify, request
from psycopg_pool import PoolTimeout, TooManyRequests
from psycopg.types.json import Jsonb
from werkzeug.exceptions import HTTPException
from src.db import make_pool, setting
from src.telemetry import HTTP_REQUESTS, HTTP_DURATION, metrics_response, setup
app = Flask(__name__)
setup()
app.config['MAX_CONTENT_LENGTH'] = 10485760
pool = make_pool()
cache = redis.Redis.from_url(setting('CACHE_URL'), socket_connect_timeout=0.2,
socket_timeout=0.2, decode_responses=True)
cache_executor = ThreadPoolExecutor(max_workers=2, thread_name_prefix='cache')
cache_slots = BoundedSemaphore(2)
hasher = PasswordHasher(time_cost=2, memory_cost=19456, parallelism=1)
dummy_hash = hasher.hash(secrets.token_urlsafe(32))
SESSION_TTL = int(os.getenv('SESSION_TTL_SECONDS', '3600'))
if SESSION_TTL < 1:
raise ValueError('SESSION_TTL_SECONDS must be positive')
USER_FIELDS = 'id, username, display_name, created_at'
INSTANCE_ID = os.getenv('INSTANCE_ID', uuid.uuid4().hex)
class APIError(Exception):
def __init__(self, status, code, message, challenge=None):
self.status, self.code, self.message, self.challenge = status, code, message, challenge
def invalid():
raise APIError(400, 'invalid_request', 'Invalid request')
def unauthorized(basic=False):
challenge = 'Basic realm="messenger", charset="UTF-8"' if basic else 'Bearer realm="messenger"'
raise APIError(401, 'unauthorized', 'Authentication required', challenge)
def serial(value):
if isinstance(value, dict):
return {key: serial(item) for key, item in value.items()}
if isinstance(value, list):
return [serial(item) for item in value]
if isinstance(value, uuid.UUID):
return str(value)
if isinstance(value, datetime):
return value.isoformat().replace('+00:00', 'Z')
return value
def response(value, status=200):
return jsonify(serial(value)), status
@app.before_request
def identify_request():
g.request_id = uuid.uuid4().hex
g.started_at = time.monotonic()
@app.after_request
def headers(result):
result.headers['X-Request-Id'] = g.request_id
result.headers['X-Instance-Id'] = INSTANCE_ID
result.headers['Cache-Control'] = 'no-store'
route = request.url_rule.rule if request.url_rule else 'unknown'
HTTP_REQUESTS.labels(request.method, route, result.status_code).inc()
HTTP_DURATION.labels(request.method, route).observe(time.monotonic() - g.started_at)
app.logger.info('http_request method=%s route=%s status=%s request_id=%s', request.method, route, result.status_code, g.request_id)
return result
@app.get('/metrics')
def metrics():
return metrics_response()
@app.errorhandler(Exception)
def error(exc):
if isinstance(exc, APIError):
problem = exc
elif isinstance(exc, psycopg.errors.UniqueViolation):
problem = APIError(409, 'conflict', 'Username already exists')
elif isinstance(exc, psycopg.errors.ForeignKeyViolation):
problem = APIError(400, 'invalid_request', 'Invalid member list')
elif isinstance(exc, (BotoCoreError, ClientError, psycopg.OperationalError, PoolTimeout, TooManyRequests,
psycopg.errors.QueryCanceled, psycopg.errors.LockNotAvailable)):
problem = APIError(503, 'unavailable', 'Database temporarily unavailable')
elif isinstance(exc, HTTPException):
status = exc.code if exc.code in (400, 404, 413, 415) else 400
code = {400: 'invalid_request', 404: 'not_found', 413: 'payload_too_large',
415: 'unsupported_media_type'}[status]
problem = APIError(status, code, 'Request cannot be processed')
else:
app.logger.error('Unhandled error type=%s request_id=%s', type(exc).__name__, g.request_id)
problem = APIError(500, 'internal_error', 'Internal error')
result = jsonify(error={'code': problem.code, 'message': problem.message,
'request_id': g.request_id})
result.status_code = problem.status
if problem.challenge:
result.headers['WWW-Authenticate'] = problem.challenge
if problem.status == 429:
result.headers['Retry-After'] = '1'
return result
def body(fields):
if request.mimetype != 'application/json':
raise APIError(415, 'unsupported_media_type', 'Expected application/json')
value = request.get_json()
if not isinstance(value, dict) or set(value) != set(fields):
invalid()
return value
def string(value, minimum, maximum):
if not isinstance(value, str) or not minimum <= len(value) <= maximum or not value.strip():
invalid()
try:
value.encode('utf-8')
except UnicodeError:
invalid()
if '\x00' in value:
invalid()
return value
def uid(value):
try:
if not isinstance(value, str) or not re.fullmatch(
r'[0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}', value):
invalid()
return uuid.UUID(value)
except ValueError:
invalid()
def cached_operation(operation, *args, **kwargs):
# Socket timeouts do not bound DNS resolution when a Compose service is down.
# At most two background operations per worker, no unbounded queue.
if not cache_slots.acquire(blocking=False):
return None
future = cache_executor.submit(operation, *args, **kwargs)
future.add_done_callback(lambda _: cache_slots.release())
try:
return future.result(timeout=0.25)
except (redis.RedisError, FutureTimeout):
return None
def profile(conn, user_id):
key = 'profile:v1:' + str(user_id)
cached = cached_operation(cache.get, key)
if cached:
try:
return json.loads(cached)
except ValueError:
pass
user = serial(conn.execute('SELECT ' + USER_FIELDS + ' FROM users WHERE id=%s',
(user_id,)).fetchone())
if user is None:
unauthorized()
cached_operation(cache.set, key, json.dumps(user), ex=60)
return user
def actor(conn):
auth = request.headers.get('Authorization', '').split()
if len(auth) != 2 or auth[0].lower() != 'bearer' or len(auth[1]) > 256:
unauthorized()
digest = hashlib.sha256(auth[1].encode()).hexdigest()
session = conn.execute('SELECT user_id FROM sessions WHERE token_hash=%s '
'AND expires_at > clock_timestamp()', (digest,)).fetchone()
if not session:
unauthorized()
g.token_hash = digest
return session['user_id']
def chat_access(conn, chat_id, user_id):
chat = conn.execute('SELECT c.* FROM chats c JOIN members m ON m.chat_id=c.id '
'WHERE c.id=%s AND m.user_id=%s', (chat_id, user_id)).fetchone()
if not chat:
raise APIError(404, 'not_found', 'Resource not found')
return chat
def chat_members(conn, chat):
chat['member_ids'] = [row['user_id'] for row in conn.execute(
'SELECT user_id FROM members WHERE chat_id=%s ORDER BY user_id', (chat['id'],))]
return chat
def page_args(scope):
limit = request.args.get('limit', '50')
if not re.fullmatch(r'[0-9]{1,3}', limit) or not 1 <= int(limit) <= 100:
invalid()
cursor = request.args.get('cursor')
after = None
if cursor is not None:
try:
if len(cursor) > 1024:
invalid()
data = json.loads(base64.b64decode(cursor, altchars=b'-_', validate=True))
if not isinstance(data, list) or len(data) != 3 or data[0] != scope:
invalid()
date = datetime.fromisoformat(data[1])
if date.tzinfo is None:
invalid()
after = (date, uid(data[2]))
except (ValueError, TypeError, binascii.Error):
invalid()
return int(limit), after
def page(rows, limit, scope):
more = len(rows) > limit
items = rows[:limit]
cursor = None
if more:
last = serial(items[-1])
cursor = base64.urlsafe_b64encode(json.dumps(
[scope, last['created_at'], last['id']]).encode()).decode()
return {'items': items, 'next_cursor': cursor}
@app.get('/health/live')
def live():
return response({'status': 'ok'})
@app.get('/health/ready')
def ready():
with pool.connection() as conn:
conn.execute('SELECT 1 FROM schema_migrations LIMIT 1')
return response({'status': 'ok'})
@app.post('/api/v1/users')
def register():
data = body(('username', 'display_name', 'password'))
name = string(data['username'], 3, 32)
if not re.fullmatch('[a-z0-9_]{3,32}', name):
invalid()
display = string(data['display_name'], 1, 100)
password = string(data['password'], 12, 128)
encoded = hasher.hash(password)
with pool.connection() as conn:
user = conn.execute('INSERT INTO users (id,username,display_name,password_hash) '
'VALUES (%s,%s,%s,%s) RETURNING ' + USER_FIELDS,
(uuid.uuid4(), name, display, encoded)).fetchone()
return response(user, 201)
@app.post('/api/v1/auth/sessions')
def login():
auth = request.headers.get('Authorization', '').split()
if len(auth) != 2 or auth[0].lower() != 'basic' or len(auth[1]) > 2048:
unauthorized(True)
try:
name, password = base64.b64decode(auth[1], validate=True).decode('utf-8').split(':', 1)
except (ValueError, UnicodeError, binascii.Error):
unauthorized(True)
if not re.fullmatch('[a-z0-9_]{3,32}', name) or not 12 <= len(password) <= 128:
unauthorized(True)
if request.get_data():
invalid()
with pool.connection() as conn:
user = conn.execute('SELECT * FROM users WHERE username=%s', (name,)).fetchone()
try:
hasher.verify(user['password_hash'] if user else dummy_hash, password)
except VerificationError:
unauthorized(True)
if not user:
unauthorized(True)
token = secrets.token_urlsafe(32)
with pool.connection() as conn:
expiry = conn.execute('INSERT INTO sessions VALUES (%s,%s,clock_timestamp() '
'+ %s * interval \'1 second\') RETURNING expires_at',
(hashlib.sha256(token.encode()).hexdigest(), user['id'],
SESSION_TTL)).fetchone()['expires_at']
user.pop('password_hash')
return response({'token': token, 'token_type': 'Bearer', 'expires_at': expiry, 'user': user}, 201)
@app.delete('/api/v1/auth/sessions/current')
def logout():
with pool.connection() as conn:
actor(conn)
conn.execute('DELETE FROM sessions WHERE token_hash=%s', (g.token_hash,))
return '', 204
@app.get('/api/v1/users/me')
def me():
with pool.connection() as conn:
user = profile(conn, actor(conn))
return response(user)
@app.post('/api/v1/chats')
def create_chat():
with pool.connection() as conn:
user = actor(conn)
data = body(('title', 'member_ids'))
title = string(data['title'], 1, 100)
if not isinstance(data['member_ids'], list) or not 1 <= len(data['member_ids']) <= 100:
invalid()
members = [uid(item) for item in data['member_ids']]
if len(set(members)) != len(members) or user not in members:
invalid()
chat = conn.execute('INSERT INTO chats(id,title) VALUES (%s,%s) RETURNING *',
(uuid.uuid4(), title)).fetchone()
for member in members:
conn.execute('INSERT INTO members VALUES (%s,%s)', (chat['id'], member))
chat['member_ids'] = sorted(members)
return response(chat, 201)
@app.get('/api/v1/chats')
def list_chats():
with pool.connection() as conn:
user = actor(conn)
scope = 'chats:' + str(user)
limit, after = page_args(scope)
sql = 'SELECT c.* FROM chats c JOIN members m ON m.chat_id=c.id WHERE m.user_id=%s'
params = [user]
if after:
sql += ' AND (c.created_at,c.id) > (%s,%s)'
params.extend(after)
rows = conn.execute(sql + ' ORDER BY c.created_at,c.id LIMIT %s', params + [limit+1]).fetchall()
result = page(rows, limit, scope)
for chat in result['items']:
chat_members(conn, chat)
return response(result)
@app.get('/api/v1/chats/<chat_id>')
def get_chat(chat_id):
with pool.connection() as conn:
user = actor(conn)
result = chat_members(conn, chat_access(conn, uid(chat_id), user))
return response(result)
@app.route('/api/v1/chats/<chat_id>/messages', methods=['GET', 'POST'])
def messages(chat_id):
with pool.connection() as conn:
user = actor(conn)
chat_id = uid(chat_id)
chat_access(conn, chat_id, user)
if request.method == 'POST':
if request.mimetype != 'application/json':
raise APIError(415, 'unsupported_media_type', 'Expected application/json')
data = request.get_json()
if not isinstance(data, dict) or 'text' not in data or set(data) - {'text', 'attachment_ids'}:
invalid()
text = data['text']
if not isinstance(text, str) or len(text) > 4000 or '\x00' in text:
invalid()
try:
text.encode('utf-8')
except UnicodeError:
invalid()
attachments = data.get('attachment_ids', [])
if not isinstance(attachments, list) or len(attachments) > 10:
invalid()
attachments = [uid(item) for item in attachments]
if len(set(attachments)) != len(attachments) or not (text.strip() or attachments):
invalid()
for attachment_id in attachments:
attachment = conn.execute('SELECT status FROM attachments WHERE id=%s AND chat_id=%s',
(attachment_id, chat_id)).fetchone()
if not attachment:
raise APIError(404, 'not_found', 'Attachment not found')
if attachment['status'] == 'failed':
raise APIError(409, 'conflict', 'Attachment failed')
key = request.headers.get('Idempotency-Key', '')
if not re.fullmatch(r'[A-Za-z0-9._:-]{1,128}', key):
invalid()
canonical = {'text': text, 'attachment_ids': [str(item) for item in attachments]}
lock = int.from_bytes(hashlib.sha256(f'{user}:{chat_id}:{key}'.encode()).digest()[:8],
'big', signed=True)
conn.execute('SELECT pg_advisory_xact_lock(%s)', (lock,))
conn.execute('DELETE FROM idempotency WHERE user_id=%s AND chat_id=%s AND key=%s '
'AND expires_at<=clock_timestamp()', (user, chat_id, key))
prior = conn.execute('SELECT * FROM idempotency WHERE user_id=%s AND chat_id=%s AND key=%s',
(user, chat_id, key)).fetchone()
if prior:
if prior['request'] != canonical:
raise APIError(409, 'conflict', 'Idempotency key already used with a different message')
return response(prior['response'], 201)
result = conn.execute('INSERT INTO messages(id,chat_id,sender_id,text,attachment_ids) '
'VALUES (%s,%s,%s,%s,%s) RETURNING *',
(uuid.uuid4(), chat_id, user, text, attachments)).fetchone()
conn.execute('INSERT INTO idempotency(user_id,chat_id,key,request,response) '
'VALUES (%s,%s,%s,%s,%s)',
(user, chat_id, key, Jsonb(canonical), Jsonb(serial(result))))
status = 201
else:
scope = 'messages:' + str(chat_id)
limit, after = page_args(scope)
sql = 'SELECT * FROM messages WHERE chat_id=%s'
params = [chat_id]
if after:
sql += ' AND (created_at,id) > (%s,%s)'
params.extend(after)
rows = conn.execute(sql + ' ORDER BY created_at,id LIMIT %s', params + [limit+1]).fetchall()
result, status = page(rows, limit, scope), 200
return response(result, status)
from src.attachments import register_routes
register_routes(app, pool, actor, chat_access, uid, response, APIError, invalid)
+123
View File
@@ -0,0 +1,123 @@
import hashlib
import tempfile
import uuid
import warnings
from threading import BoundedSemaphore
from flask import Response, request, stream_with_context
from PIL import Image, UnidentifiedImageError
from src.storage import BUCKET, s3
MAX_BYTES = 10 * 1024 * 1024
Image.MAX_IMAGE_PIXELS = 40_000_000
# Each API worker is bounded by Gunicorn threads; this semaphore prevents an
# unbounded number of uploads without serializing the required concurrent case.
slots = BoundedSemaphore(16)
def public(row):
return {k: v for k, v in row.items() if k != 'object_key'}
def register_routes(app, pool, actor, chat_access, uid, response, APIError, invalid):
@app.post('/api/v1/chats/<chat_id>/attachments')
def upload(chat_id):
with pool.connection() as conn:
user = actor(conn)
chat_id = uid(chat_id)
chat_access(conn, chat_id, user)
filename = request.args.get('filename', '')
if not 1 <= len(filename) <= 255 or not filename.strip() or any(
ord(c) < 32 or ord(c) == 127 or c in '/\\' for c in filename):
invalid()
if request.mimetype not in ('image/png', 'image/jpeg'):
raise APIError(415, 'unsupported_media_type', 'Expected PNG or JPEG')
if request.content_length and request.content_length > MAX_BYTES:
raise APIError(413, 'payload_too_large', 'Image too large')
if not slots.acquire(timeout=5):
raise APIError(429, 'rate_limited', 'Upload capacity reached')
try:
with tempfile.SpooledTemporaryFile(max_size=1024*1024) as file:
length, digest = 0, hashlib.sha256()
while True:
chunk = request.stream.read(65536)
if not chunk:
break
length += len(chunk)
if length > MAX_BYTES:
raise APIError(413, 'payload_too_large', 'Image too large')
digest.update(chunk)
file.write(chunk)
file.seek(0)
try:
with warnings.catch_warnings():
warnings.simplefilter('error', Image.DecompressionBombWarning)
with Image.open(file) as image:
if image.width * image.height > 40_000_000:
raise APIError(413, 'payload_too_large', 'Too many pixels')
if Image.MIME.get(image.format) != request.mimetype:
raise APIError(415, 'unsupported_media_type', 'Image type mismatch')
image.verify()
file.seek(0)
with Image.open(file) as image:
image.load()
except (Image.DecompressionBombError, Image.DecompressionBombWarning):
raise APIError(413, 'payload_too_large', 'Too many pixels') from None
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
raise APIError(415, 'unsupported_media_type', 'Invalid image') from None
file.seek(0)
identifier = uuid.uuid4()
key = 'original/' + str(identifier)
with pool.connection() as conn:
conn.execute('INSERT INTO object_intents(object_key) VALUES (%s)', (key,))
s3.put_object(Bucket=BUCKET, Key=key, Body=file, ContentLength=length, ContentType=request.mimetype)
# Intent survives a crash or failed DB commit; the janitor removes old unreferenced objects.
with pool.connection() as conn:
row = conn.execute("INSERT INTO attachments(id,chat_id,owner_id,filename,content_type,size_bytes,sha256,object_key,status) "
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'queued') RETURNING *",
(identifier,chat_id,user,filename,request.mimetype,length,digest.hexdigest(),key)).fetchone()
conn.execute('INSERT INTO attachment_jobs(attachment_id) VALUES (%s)', (identifier,))
conn.execute('DELETE FROM object_intents WHERE object_key=%s', (key,))
return response(public(row),202)
finally:
slots.release()
def authorized(attachment_id):
with pool.connection() as conn:
user = actor(conn)
identifier = uid(attachment_id)
row = conn.execute('SELECT a.* FROM attachments a JOIN members m ON m.chat_id=a.chat_id '
'WHERE a.id=%s AND m.user_id=%s', (identifier,user)).fetchone()
if not row:
raise APIError(404,'not_found','Attachment not found')
return row
@app.get('/api/v1/attachments/<attachment_id>')
def metadata(attachment_id):
return response(public(authorized(attachment_id)))
@app.get('/api/v1/attachments/<attachment_id>/content')
def download(attachment_id):
row = authorized(attachment_id)
if row['status'] != 'ready':
raise APIError(409,'conflict','Attachment not ready')
variant = request.args.get('variant','original')
key, content_type, size_bytes = row['object_key'], row['content_type'], row['size_bytes']
if variant != 'original':
found = next((item for item in row['variants'] if item['name'] == variant), None)
if not found:
raise APIError(404,'not_found','Variant not found')
key = row['object_key'] + '.' + variant
content_type, size_bytes = found['content_type'], found['size_bytes']
obj = s3.get_object(Bucket=BUCKET,Key=key)
def chunks():
try:
while True:
chunk = obj['Body'].read(65536)
if not chunk:
break
yield chunk
finally:
obj['Body'].close()
return Response(stream_with_context(chunks()),content_type=content_type,
headers={'Content-Length':str(size_bytes),'X-Content-Type-Options':'nosniff'})
+37
View File
@@ -0,0 +1,37 @@
"""Old upload intent reconciliation; dry run unless --apply, never deletes fresh objects."""
import argparse
import json
import time
from src.db import make_pool
from src.storage import s3, BUCKET
def sweep(apply=False):
pool = make_pool()
with pool.connection() as conn:
rows = conn.execute("SELECT i.object_key FROM object_intents i LEFT JOIN attachments a "
"ON a.object_key=i.object_key WHERE a.id IS NULL "
"AND i.created_at < clock_timestamp()-interval '1 hour' FOR UPDATE OF i SKIP LOCKED").fetchall()
for row in rows:
if apply:
s3.delete_object(Bucket=BUCKET,Key=row['object_key'])
conn.execute('DELETE FROM object_intents WHERE object_key=%s',(row['object_key'],))
pool.close()
return {'old_unreferenced_intents':len(rows),'applied':apply}
if __name__ == '__main__':
parser=argparse.ArgumentParser()
parser.add_argument('--apply',action='store_true')
parser.add_argument('--loop',action='store_true')
args=parser.parse_args()
while True:
try:
print(json.dumps(sweep(args.apply)),flush=True)
except Exception as exc:
print(json.dumps({'cleanup_error':type(exc).__name__}),flush=True)
if not args.loop:
raise SystemExit(1)
if not args.loop:
break
time.sleep(60)
+52
View File
@@ -0,0 +1,52 @@
"""Bounded PostgreSQL pool and serialized, versioned migrations."""
import hashlib
import os
import time
from pathlib import Path
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool
def setting(name):
path = os.getenv(name + '_FILE')
return Path(path).read_text().strip() if path else os.environ[name]
def migrate():
with psycopg.connect(setting('DATABASE_URL'), connect_timeout=3) as conn:
conn.execute('SELECT pg_advisory_xact_lock(202602)')
conn.execute('CREATE TABLE IF NOT EXISTS schema_migrations '
'(version text PRIMARY KEY, checksum text NOT NULL)')
for file in sorted(Path('migrations').glob('*.sql')):
sql = file.read_text(encoding='utf-8')
digest = hashlib.sha256(sql.encode()).hexdigest()
row = conn.execute('SELECT checksum FROM schema_migrations WHERE version=%s',
(file.name,)).fetchone()
if row:
if row[0] != digest:
raise RuntimeError('Applied migration changed')
continue
conn.execute(sql)
conn.execute('INSERT INTO schema_migrations VALUES (%s,%s)', (file.name, digest))
def make_pool():
return ConnectionPool(setting('DATABASE_URL'), min_size=1, max_size=8,
timeout=3, max_waiting=32,
check=ConnectionPool.check_connection,
kwargs={'connect_timeout': 3, 'row_factory': dict_row,
'options': '-c statement_timeout=3000 -c lock_timeout=2000'})
if __name__ == '__main__':
for attempt in range(30):
try:
migrate()
break
except psycopg.OperationalError:
if attempt == 29:
raise RuntimeError('Database did not become available') from None
time.sleep(2)
print('Migrations applied successfully')
+10
View File
@@ -0,0 +1,10 @@
import os
import boto3
from botocore.config import Config
from src.db import setting
BUCKET = os.getenv('S3_BUCKET', 'messenger')
s3 = boto3.client('s3', endpoint_url=os.getenv('S3_ENDPOINT', 'http://lb:9000'),
aws_access_key_id=setting('S3_ACCESS_KEY'), aws_secret_access_key=setting('S3_SECRET_KEY'),
region_name='us-east-1', config=Config(connect_timeout=2, read_timeout=5,
retries={'max_attempts': 0}, s3={'addressing_style': 'path'}))
+22
View File
@@ -0,0 +1,22 @@
import json
import logging
import time
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
HTTP_REQUESTS = Counter('messenger_http_requests_total','HTTP requests',['method','route','status'])
HTTP_DURATION = Histogram('messenger_http_request_duration_seconds','HTTP duration',['method','route'])
JOBS = Counter('messenger_attachment_jobs_total','Attachment jobs',['status'])
QUEUE_AGE = Gauge('messenger_attachment_oldest_job_age_seconds','Oldest pending job age')
class JsonFormatter(logging.Formatter):
def format(self, record):
return json.dumps({'ts': self.formatTime(record), 'level': record.levelname,
'message': record.getMessage(), 'trace_id': getattr(record,'trace_id',None)})
def setup():
handler=logging.StreamHandler(); handler.setFormatter(JsonFormatter())
logging.getLogger().handlers[:] = [handler]; logging.getLogger().setLevel(logging.INFO)
def metrics_response():
from flask import Response
return Response(generate_latest(), content_type=CONTENT_TYPE_LATEST)
+60
View File
@@ -0,0 +1,60 @@
import hashlib
import io
import json
import random
import time
from PIL import Image, ImageOps
from src.db import make_pool
from src.storage import BUCKET, s3
MAX_ATTEMPTS = 5
def make_variant(image, name):
image = ImageOps.exif_transpose(image)
if name == 'thumbnail':
image.thumbnail((256, 256), Image.Resampling.LANCZOS); target = image
elif name == 'crop':
target = ImageOps.fit(image, (128, 128), method=Image.Resampling.LANCZOS)
else:
image.thumbnail((1280, 1280), Image.Resampling.LANCZOS); target = image
if target.mode not in ('RGB', 'L'):
bg = Image.new('RGB', target.size, 'white')
bg.paste(target, mask=target.getchannel('A') if 'A' in target.getbands() else None)
target = bg
out = io.BytesIO()
content_type = 'image/webp' if name == 'optimized' else 'image/jpeg'
target.convert('RGB').save(out, format='WEBP' if name == 'optimized' else 'JPEG', quality=82, exif=b'')
raw = out.getvalue()
return raw, content_type, target.width, target.height
def process(row, pool):
raw = s3.get_object(Bucket=BUCKET, Key=row['object_key'])['Body'].read()
with Image.open(io.BytesIO(raw)) as image:
variants=[]
for name in ('thumbnail','crop','optimized'):
data, content_type, width, height = make_variant(image.copy(), name)
s3.put_object(Bucket=BUCKET, Key=row['object_key']+'.'+name, Body=data, ContentType=content_type)
variants.append({'name':name,'content_type':content_type,'size_bytes':len(data),'width':width,'height':height,'sha256':hashlib.sha256(data).hexdigest()})
with pool.connection() as conn:
conn.execute("UPDATE attachments SET status='ready',variants=%s,error_code=NULL,updated_at=clock_timestamp() WHERE id=%s AND status<>'ready'", (json.dumps(variants),row['id']))
conn.execute('UPDATE attachment_jobs SET completed_at=clock_timestamp(),locked_until=NULL WHERE attachment_id=%s',(row['id'],))
def loop():
pool=make_pool()
while True:
with pool.connection() as conn:
job=conn.execute("UPDATE attachment_jobs SET attempts=attempts+1,locked_until=clock_timestamp()+interval '60 seconds' WHERE attachment_id=(SELECT attachment_id FROM attachment_jobs WHERE completed_at IS NULL AND available_at<=clock_timestamp() AND (locked_until IS NULL OR locked_until<clock_timestamp()) ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1) RETURNING *").fetchone()
row=conn.execute('SELECT * FROM attachments WHERE id=%s',(job['attachment_id'],)).fetchone() if job else None
if not row:
time.sleep(.25); continue
try: process(row,pool)
except Exception as error:
with pool.connection() as conn:
if job['attempts'] >= MAX_ATTEMPTS:
conn.execute("UPDATE attachments SET status='failed',error_code='processing_failed',updated_at=clock_timestamp() WHERE id=%s",(row['id'],))
conn.execute('UPDATE attachment_jobs SET completed_at=clock_timestamp(),last_error=%s WHERE attachment_id=%s',(type(error).__name__,row['id']))
else:
delay=min(60,2**job['attempts'])+random.random()
conn.execute("UPDATE attachment_jobs SET available_at=clock_timestamp()+(%s*interval '1 second'),locked_until=NULL,last_error=%s WHERE attachment_id=%s",(delay,type(error).__name__,row['id']))
if __name__ == '__main__': loop()
+60
View File
@@ -0,0 +1,60 @@
"""Attachment negative checks, private-bucket check, and concurrent upload budget."""
import base64
import hashlib
import os
import subprocess
import sys
import threading
import time
import uuid
from concurrent.futures import ThreadPoolExecutor
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'tests'))
from client import Client, check, Failure
PNG = (b'\x89PNG\r\n\x1a\n' + b'\x00' * 100)
def main():
c = Client()
alice, bob, eve, chat = c.bootstrap()
path = '/api/v1/chats/' + chat['id'] + '/attachments?filename=photo.png'
att, content = c.attachment(alice, chat)
check(att['status'] == 'ready' and att['sha256'] == hashlib.sha256(content).hexdigest(), 'metadata mismatch')
downloaded = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)
check(downloaded['bytes'] == content, 'download mismatch')
c.call('GET', '/api/v1/attachments/' + att['id'], 404, actor=eve)
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', 404, actor=eve)
c.call('GET', '/api/v1/attachments/' + str(uuid.uuid4()), 404, actor=alice)
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=bad/name.png', 400,
actor=alice, raw=content, headers={'Content-Type': 'image/png'})
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=spoof.png', 415,
actor=alice, raw=b'not-an-image', headers={'Content-Type': 'image/png'})
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=spoof.txt', 415,
actor=alice, raw=content, headers={'Content-Type': 'text/plain'})
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=huge.png', 413,
actor=alice, raw=b'0' * (10 * 1024 * 1024 + 1), headers={'Content-Type': 'image/png'})
responses = []
def upload(_):
return c.attachment(alice, chat)[0]['id']
with ThreadPoolExecutor(max_workers=10) as pool:
responses = list(pool.map(upload, range(10)))
check(len(set(responses)) == 10, 'concurrent uploads shared an id')
try:
urlopen('http://127.0.0.1:9000/messenger/original/' + str(att['id']), timeout=3)
except HTTPError as error:
check(error.code in (400, 403), 'bucket/object is publicly readable')
else:
raise AssertionError('bucket/object is publicly readable')
stats = subprocess.check_output(['docker', 'stats', '--no-stream', '--format', '{{.Name}}={{.MemUsage}}',
'vp-lab4-api1-1', 'vp-lab4-api2-1'], text=True)
Path(ROOT / 'evidence/attachment-resources.txt').write_text(stats, encoding='utf-8')
print('PASS: attachment access, validation, private bucket and 10 concurrent uploads')
if __name__ == '__main__':
main()
+101
View File
@@ -0,0 +1,101 @@
"""Concurrent idempotency, both replica failures, and rolling recreation under writes."""
import json
import os
import subprocess
import sys
import threading
import time
import uuid
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from urllib.parse import quote
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'tests'))
from client import Client, check, Failure
def compose(*args):
subprocess.run(['docker', 'compose', *args], cwd=ROOT, check=True,
stdout=subprocess.DEVNULL, timeout=120)
def history(c, actor, chat):
path = '/api/v1/chats/' + chat['id'] + '/messages?limit=100'
result = []
while path:
page = c.call('GET', path, actor=actor, schema='MessagePage')['json']
result.extend(page['items'])
path = '/api/v1/chats/' + chat['id'] + '/messages?limit=100&cursor=' + quote(page['next_cursor'], safe='') if page['next_cursor'] else None
return result
def main():
c = Client(timeout=5)
alice, bob, eve, chat = c.bootstrap()
key = uuid.uuid4().hex
def send(_):
return Client().message(alice, chat, 'One logical message', key=key)
with ThreadPoolExecutor(max_workers=12) as executor:
responses = list(executor.map(send, range(24)))
ids = {item['json']['id'] for item in responses}
instances = {item['headers']['X-Instance-Id'] for item in responses}
check(len(ids) == 1 and len(instances) >= 2, 'Concurrency did not cover both replicas or duplicated message')
check(len(history(c, alice, chat)) == 1, 'Duplicate row committed')
c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 409, actor=alice,
headers={'Idempotency-Key': key}, body={'text': 'Different'})
results = [{'scenario': 'concurrent_idempotency', 'requests': 24,
'replicas': sorted(instances), 'unique_messages': 1}]
def experiment(label, action):
stop = threading.Event()
writes, errors, times = [], [], []
def traffic():
client = Client(timeout=3)
retry_key, text = uuid.uuid4().hex, label + uuid.uuid4().hex
while not stop.is_set():
try:
response = client.message(alice, chat, text, key=retry_key)
writes.append(response['json'])
times.append(time.monotonic())
retry_key, text = uuid.uuid4().hex, label + uuid.uuid4().hex
except Failure:
errors.append(time.monotonic())
time.sleep(0.02)
thread = threading.Thread(target=traffic)
thread.start()
started = time.monotonic()
try:
action()
time.sleep(5)
finally:
stop.set()
thread.join(timeout=10)
duration = time.monotonic() - started
stored = {item['id']: item for item in history(c, bob, chat)}
check(writes and all(stored.get(item['id']) == item for item in writes), 'Confirmed writes lost')
bodies = [item['text'] for item in stored.values()]
check(len(bodies) == len(set(bodies)), 'Retry duplicated a logical message')
gaps = [b-a for a,b in zip([started]+times, times+[time.monotonic()])]
check(max(gaps) < 10, 'Recovery exceeded 10 seconds')
results.append({'scenario': label, 'duration_seconds': round(duration,3),
'writes': len(writes), 'errors': len(errors),
'rps': round(len(writes)/duration,3), 'max_success_gap_seconds': round(max(gaps),3)})
for service in ('api1', 'api2'):
try:
experiment('stop_' + service, lambda: compose('stop', '-t', '2', service))
finally:
compose('up', '-d', '--no-deps', '--wait', service)
time.sleep(2)
def rolling():
for service in ('api1', 'api2'):
compose('up', '-d', '--no-deps', '--force-recreate', '--wait', service)
time.sleep(2)
experiment('rolling_recreation', rolling)
Path(ROOT / 'evidence/replication.json').write_text(json.dumps(results, indent=2), encoding='utf-8')
print(json.dumps(results, indent=2))
if __name__ == '__main__':
main()
+18
View File
@@ -0,0 +1,18 @@
"""Small security acceptance check for the local TLS boundary."""
import ssl
import urllib.request
import urllib.error
from pathlib import Path
root = Path(__file__).resolve().parents[1]
context = ssl.create_default_context(cafile=root / 'certs' / 'ca.crt')
with urllib.request.urlopen('https://127.0.0.1:8443/health/live', context=context, timeout=5) as response:
assert response.status == 200
try:
with urllib.request.urlopen('https://127.0.0.1:8443/health/live', context=ssl.create_default_context(), timeout=5):
raise AssertionError('untrusted certificate was accepted')
except (ssl.SSLCertVerificationError, urllib.error.URLError) as error:
if isinstance(error, urllib.error.URLError) and not isinstance(error.reason, ssl.SSLCertVerificationError):
raise
pass
print('PASS: trusted local TLS works and an untrusted certificate is rejected.')