forked from mtusi2026/mtusi2026lab4
Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1873c4c798 | ||
|
|
463333351f | ||
|
|
6039524214 |
@@ -0,0 +1,34 @@
|
||||
name: solution
|
||||
on: [push, pull_request]
|
||||
permissions:
|
||||
contents: read
|
||||
jobs:
|
||||
application:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Fresh stack, contract and public API
|
||||
run: |
|
||||
make up
|
||||
make check
|
||||
make test
|
||||
- name: Real PostgreSQL and race detector
|
||||
run: make test-race
|
||||
- name: Student HTTP cases
|
||||
run: python3 student-tests/attachments.py --base-url http://localhost:8084
|
||||
- name: Persistence of existing session and data
|
||||
run: make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
- name: API failover through LB
|
||||
run: make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
- name: Safe diagnostics
|
||||
if: always()
|
||||
run: docker compose logs --tail 100 > evidence/ci-containers.txt
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: solution-evidence
|
||||
path: evidence/
|
||||
- name: Dispose this CI-only stack
|
||||
if: always()
|
||||
run: docker compose --profile verify down -v --remove-orphans
|
||||
+202
@@ -0,0 +1,202 @@
|
||||
# Защита ЛР 4: файл, метаданные и два независимых хранилища
|
||||
|
||||
Автор: Крайкин Глеб, БВТ 2303. Базовые вопросы о балансировке и идемпотентности подробно разобраны в `lab3/DEFENSE.md`; каждое решение ЛР 4 при этом запускается самостоятельно. Здесь сосредоточься на новой части. Команды и реальные результаты находятся в `QUICKSTART.md`, `REPORT.md`, `evidence/`.
|
||||
|
||||
## Рассказ на 90 секунд
|
||||
|
||||
«Я сохранил два API за HAProxy, общие сессии и идемпотентность сообщений из третьей работы. Добавил синхронную загрузку PNG/JPEG. Байты оригинала хранятся в приватном MinIO, метаданные — в PostgreSQL. Имя файла только отображается; object key назначает сервер. Вложение сразу привязано к чату. Для метаданных, скачивания и добавления в сообщение проверяются сессия и членство в чате.
|
||||
|
||||
До декодирования ограничены байты и число пикселей. Сначала файл потоково записывается во временный ограниченный spool, затем проверяется конфигурация и полное декодирование. Один API одновременно валидирует одну загрузку; скачивание идёт потоком. 201 возвращается только после PUT оригинала и COMMIT метаданных.
|
||||
|
||||
У PostgreSQL и S3 нет общей SQL-транзакции. Поэтому до PUT сохраняется журнал намерения, а reconciler регулярно сверяет хранилища. Он имеет dry-run и grace period, повторно проверяет объект под тем же lock, что загрузка, удаляет старые неучтённые объекты и безопасно исправляет пропавшие неиспользуемые записи. Используемые ссылки не удаляются: потерянные байты для них нужно восстанавливать из исходника или backup. Я внедрил сбои обеих сторон, проверил повторный запуск очистки и защиту свежих и используемых вложений».
|
||||
|
||||
## Архитектура
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C[Клиент] --> LB[HAProxy :8084]
|
||||
LB --> A[API A]
|
||||
LB --> B[API B]
|
||||
A --> PG[(PostgreSQL: metadata, ACL, sessions)]
|
||||
B --> PG
|
||||
A --> S3[(Приватный MinIO: originals)]
|
||||
B --> S3
|
||||
R[Reconciler раз в 60 s] --> PG
|
||||
R --> S3
|
||||
```
|
||||
|
||||
Только LB публикует порт host. S3 API доступен внутри Compose-сети, консоль MinIO слушает loopback внутри S3-контейнера. У API отдельный application access key, которому разрешены ListBucket/GetBucketLocation и GetObject/PutObject/DeleteObject только в `originals/*` одного bucket. API не получает root credentials. Bootstrap MinIO использует root только в отдельном одноразовом контейнере.
|
||||
|
||||
**S3-совместимый** означает API для bucket/object, а не обещание всех гарантий Amazon S3 или распределённой отказоустойчивости. В нашем MinIO один контейнер и один persistent volume: отказ host/disk не переживается автоматически.
|
||||
|
||||
## Что где хранится
|
||||
|
||||
| Объект | Хранилище | Почему |
|
||||
|---|---|---|
|
||||
| Оригинальные PNG/JPEG bytes | MinIO, приватный bucket | Общие бинарные объекты для всех API |
|
||||
| Имя, MIME, size, SHA-256, chat/owner, object_key | `attachments` PostgreSQL | Поиск, связи, права, согласованность |
|
||||
| Связь сообщения и вложения, порядок | `message_attachments` | 0–10 вложений; FK; стабильный порядок |
|
||||
| Намерение незавершённой загрузки | `upload_intents` PostgreSQL | После crash остаётся понятное происхождение объекта |
|
||||
| Сессии, участники, сообщения, idempotency result | PostgreSQL | Перенесённая общая модель |
|
||||
| Временный spool до PUT | `/tmp` конкретного API | Ограниченный временный ресурс, удаляется при выходе; не источник оригинала |
|
||||
| Профиль пользователя | Valkey TTL 300 | Необязательный восстановимый кэш |
|
||||
|
||||
У `attachments.object_key` UNIQUE. `message_attachments` имеет FK к сообщению и вложению, уникальные `(message_id,attachment_id)` и `(message_id,position)`. Связь с другим чатом проверяется в одной транзакции handler; FK подтверждают существование записей. Обязательный API не меняет chat_id вложения.
|
||||
|
||||
Object key имеет вид `originals/<случайный UUID>`. Filename `кот.png` никак не участвует в пути. Key не возвращается в публичной Attachment-схеме. Секретность UUID не является проверкой доступа.
|
||||
|
||||
## Путь upload по коду
|
||||
|
||||
Открой `internal/messenger/attachments.go`, функцию `uploadAttachment`, и одновременно `server.go`.
|
||||
|
||||
1. `ServeHTTP`: аутентификация Bearer, `getChat` проверяет членство. Чужой чат даёт 404 до обработки файла.
|
||||
2. Query `filename` должен быть единственным, непустым, ≤255 code points, без `/`, `\`, NUL и управляющих символов.
|
||||
3. Допустимы только заявленные `image/png` и `image/jpeg`. MIME из заголовка не считается доказательством формата.
|
||||
4. Проверяется размер тела. Для слишком большого Content-Length читается максимум limit+1 bytes в discard, чтобы обычный клиент смог получить 413; не читается произвольное тело целиком.
|
||||
5. Semaphore допускает только одну активную загрузку на API. Остальные ждут в пределах deadline; до получения слота их сжатые файлы не читаются в большие буферы.
|
||||
6. `io.Copy` записывает ограниченное тело во временный файл и одновременно вычисляет SHA-256. Для неизвестного Content-Length действует `http.MaxBytesReader`.
|
||||
7. `image.DecodeConfig` определяет реальный PNG/JPEG и ширину/высоту. До полной распаковки проверяется ≤40 000 000 pixels.
|
||||
8. Полное `image.Decode` выявляет повреждённые данные, которые одного правильного заголовка не исключают. Возвращать ошибку Go/stack trace пользователю нельзя: 415.
|
||||
9. Сформировать UUID/key; отдельно закоммитить `upload_intents`.
|
||||
10. Открыть transaction, взять per-object advisory lock с отдельным hash namespace `4`.
|
||||
11. PUT оригинала из spool в MinIO. Multipart отключён: ограниченный объект ≤10 MiB отправляется одним PUT.
|
||||
12. INSERT `attachments`, DELETE intent, COMMIT одной SQL-транзакцией.
|
||||
13. Только теперь `201 ready`; `variants=[]`, `error_code=null`. Временный файл закрывается и удаляется через defer.
|
||||
|
||||
**Полезный вопрос:** что случится, если клиент закрыл соединение на шаге 6? `io.Copy` вернёт ошибку, временный файл удалится, объект и метаданные ещё не созданы. Если закрытие/сбой после PUT — возможен orphan, который найдёт сверка.
|
||||
|
||||
201 здесь означает, что на момент ответа PUT и COMMIT успешно завершились. Это не вечная гарантия, что оператор завтра не удалит объект вручную.
|
||||
|
||||
## Почему bytes и pixels — разные ограничения
|
||||
|
||||
Сжатый PNG небольшого размера может описывать огромное изображение. Ограничение 10 MiB относится к передаче/хранению сжатого файла. Ограничение 40M pixels относится к памяти и работе декодера. Проверять пиксели после полной распаковки поздно.
|
||||
|
||||
Условная оценка RGBA8: 40M × 4 ≈160 MB. Для NRGBA64 возможны 8 bytes/pixel ≈320 MB плюс служебная память. Поэтому per-API semaphore=1, memory limit 768 MiB, GOMEMLIMIT=600 MiB и ограничение HTTP-времени работают вместе. GOMEMLIMIT — мягкая цель GC, а не замена Docker memory limit.
|
||||
|
||||
Файл не держится целиком в RAM при spool/PUT/download, но **декодированное изображение занимает память**. Не называй весь upload «потоковым декодированием без памяти». Здесь потоковые I/O и отдельно ограниченный полный decode. Временный файл имеет максимум 10 MiB; один активный upload ограничивает временный disk footprint одного API. 10 клиентов — это 10 одновременных HTTP-запросов, а не 10 одновременных полных распаковок на одной реплике.
|
||||
|
||||
Реальные пики cgroup memory (включают память процесса и page cache, а не только Go heap) находятся в `evidence/attachments.json`. Они относятся к синтетическому PNG 1024×768, не доказывают такой же пик для предельных 40M pixels. Пик cgroup считается с запуска контейнера; в отчёте отдельно указаны значения до/после окна и sampled current.
|
||||
|
||||
## Путь download
|
||||
|
||||
`serveAttachment` принимает только GET. Через JOIN с `chat_members` ищет доступное вложение. После проверки принадлежности:
|
||||
|
||||
1. Для метаданных вернуть публичную Attachment без внутренних ключей и адресов.
|
||||
2. Для `/content` проверить variant: только отсутствующий/`original` существует в ЛР 4.
|
||||
3. Выполнить GetObject и Stat до отправки HTTP 200: отказ/отсутствие → 503, произвольный XML S3 не уходит клиенту.
|
||||
4. Задать истинный Content-Type, Content-Length, `Content-Disposition: attachment` с безопасным форматированием, `X-Content-Type-Options: nosniff`.
|
||||
5. `io.Copy` передаёт байты оригинала, без redirect и преобразования изображения.
|
||||
|
||||
Если сеть/S3 оборвутся после начала ответа, уже отправленный 200 нельзя превратить в JSON 503. Content-Length позволяет клиенту заметить неполную передачу; клиент сверяет SHA-256 метаданных и может повторить GET.
|
||||
|
||||
Любой участник чата может читать и прикреплять доступное вложение, независимо от загрузившего пользователя. Посторонний получает 404 и для метаданных, и для bytes. Доступ к bucket без подписи даёт 403, но приложение всё равно обязано проверять ACL.
|
||||
|
||||
## Сообщение с вложениями и идемпотентность
|
||||
|
||||
Текст обязателен как поле, но может быть `""`, если есть вложения. Без вложений пустой/пробельный текст запрещён. `attachment_ids` опционален и по умолчанию `[]`; null, дубли, неверные UUID и >10 ID запрещены. Сравнение отсутствующего списка и `[]` нормализовано до hash body, значит повтор возвращает то же сообщение.
|
||||
|
||||
В `createMessage`: сначала idempotency lock/result, затем SELECT всех доступных вложений того же чата `FOR KEY SHARE`, UPDATE чата, INSERT message, INSERT links, INSERT result, COMMIT. Неизвестное вложение/другой чат →404; в результате транзакция не оставит частичное сообщение.
|
||||
|
||||
`FOR KEY SHARE` согласован с `FOR UPDATE` у очистки метаданных. Если сообщение уже использует вложение, очистка увидит ссылку и не удалит его. Если очистка первой безопасно удалила missing unused metadata, отправка увидит отсутствие и вернёт 404. Проверка членства immutable в обязательном API; изменения участников потребовали бы пересмотра транзакционной модели.
|
||||
|
||||
Сам upload не имеет обязательного Idempotency-Key. Повтор upload после неопределённого ответа может создать второе вложение. Это не нарушает контракт; нельзя выдавать гарантию message POST за гарантию всех POST.
|
||||
|
||||
## Почему транзакция PostgreSQL не охватывает S3
|
||||
|
||||
`BEGIN` в PostgreSQL управляет её собственными строками. MinIO выполняет отдельный HTTP PUT и не участвует в этом COMMIT. У нас нет распределённого двухфазного протокола между ними.
|
||||
|
||||
| Точка сбоя | Что может остаться | Как восстанавливаем |
|
||||
|---|---|---|
|
||||
| До intent | Только временный spool | defer / удаление временного container filesystem |
|
||||
| Intent записан, PUT не завершился | Старый intent, без объекта | Reconciler после grace удалит intent |
|
||||
| PUT завершён, INSERT/COMMIT БД не прошёл | Объект + intent, без metadata | Удалить старый orphan и intent |
|
||||
| PUT и COMMIT прошли, ответ потерян | Полное вложение, клиент не уверен | Не обещаем идемпотентный upload; повтор создаст другой UUID |
|
||||
| Оператор удалил объект после успеха | Metadata есть, bytes нет | Unused old metadata можно удалить; referenced metadata сохраняется для восстановления |
|
||||
| Оператор удалил unused metadata | Объект без metadata | Старый orphan удаляется после grace |
|
||||
|
||||
Нельзя просто вызвать DeleteObject при любой ошибке COMMIT: COMMIT мог пройти, а ответ БД потеряться. Тогда немедленная компенсация удалила бы объект подтверждённой метадатой. Именно поэтому безопасная повторная сверка важнее «попробовать удалить в catch».
|
||||
|
||||
## Алгоритм reconciler на пять
|
||||
|
||||
Открой `internal/messenger/reconcile.go`. Параметры: по умолчанию dry-run; `--apply` разрешает изменения; grace по умолчанию 3600 s, меньше 60 s запрещён. Upload deadline 30 s, минимальный grace больше него.
|
||||
|
||||
1. Получить inventory `originals/*` в S3 и union ключей из attachments/upload_intents.
|
||||
2. Для каждого key открыть SQL-транзакцию и получить тот же per-key advisory lock, который держит upload.
|
||||
3. Заново прочитать metadata `FOR UPDATE`, проверить свежесть intent, сделать актуальный HEAD объекта. Старый snapshot inventory — только список кандидатов.
|
||||
4. Любая свежая metadata/intent/object защищает объект. Неизвестная ошибка HEAD (сеть, permission, 5xx) прерывает проход; она не приравнивается к NoSuchKey.
|
||||
5. Старый объект без metadata — orphan: в dry-run посчитать, в apply удалить объект.
|
||||
6. Старые metadata без объекта — missing: если есть message link, сохранить; иначе безопасно удалить metadata.
|
||||
7. Старый завершённый/заброшенный intent очистить. Зафиксировать транзакцию. Повторный проход не должен повторно создавать побочные эффекты.
|
||||
8. Выдать JSON с counts/duration без credentials или списка приватных URL.
|
||||
|
||||
Автоматический Compose-сервис `reconciler` выполняет проход раз в 60 s с default grace 1 h. Следовательно, обнаружение/очистка старого orphan — не мгновенное, а примерно grace + интервал успешного прохода. При отказе зависимости проход безопасно завершается ошибкой и повторяется. Healthcheck сервиса проверяет жизнь loop-процесса; успешность сверок проверяется по JSON-логам. Ручной dry-run не останавливает автоматическую защиту свежих объектов.
|
||||
|
||||
**Что защищает от гонки:** единый per-key lock upload/reconcile, повторное чтение под lock, row lock metadata против attachment linkage, grace по текущим часам и времени объекта. Просто LIST → DELETE без повторного чтения опасен: между этими шагами upload мог успеть сделать COMMIT.
|
||||
|
||||
**Что не умеет reconciler:** создавать исходное изображение из SHA-256, чинить произвольную порчу bytes с прежним размером, восстанавливать backup, сохранять данные при потере всего disk. Missing referenced — явный сигнал оператору, а не разрешение уничтожить существующую ссылку. В исследовании его байты возвращены из сохранённого синтетического оригинала, отдельно от автоматической очистки.
|
||||
|
||||
Обход — O(number of known keys), с HEAD и несколькими SQL-запросами на ключ. Текущий inventory хранится map в памяти: для миллионов объектов нужны paging, persistent cursor, batches и лимит скорости. Дополнительный dry-run:19/119/1019 ключей за 21/73/498 ms, без удалений; raw в `evidence/inventory-cost.json`. Это небольшой локальный опыт, не оценка petabyte bucket.
|
||||
|
||||
## Как действительно внедрялись ошибки
|
||||
|
||||
`student-tests/consistency.py` создаёт только синтетические данные своего стенда:
|
||||
|
||||
- Реальная SQL trigger отказывает INSERT metadata для специального имени после успешного PUT. В работающем приложении нет публичного fault endpoint.
|
||||
- Реальный объект удаляется через admin mc; отдельно удаляются unused metadata.
|
||||
- Клиент закрывает настоящий TCP socket посреди upload.
|
||||
- MinIO останавливается; upload должен дать 503 в ограниченный срок.
|
||||
- Проверяется прямой anonymous HTTP GET известного object key: 403.
|
||||
- Выдерживается настоящий grace ≥60 s; даты объектов не подделываются.
|
||||
- Dry-run обнаруживает проблемы, apply исправляет допустимое, rerun не делает повторных удалений.
|
||||
- Свежий orphan и свежее вложение сохраняются. Используемое существующее вложение скачивается побайтно; metadata используемого missing не удаляется.
|
||||
|
||||
Числа и длительность — в `evidence/consistency.json`. Публичный persistence проверяет byte identity после полного restart, а `attachment_failover.py` останавливает именно загрузивший API, сверяет Docker inspect и скачивает через другой.
|
||||
|
||||
## Вопросы преподавателя
|
||||
|
||||
1. **Почему не положить файл в PostgreSQL?** Можно хранить bytea/large objects, но выбранное разделение убирает бинарный объём из основной БД и делает общий object API. Цена — согласованность двух хранилищ и отдельный backup.
|
||||
2. **Почему Docker volume API недостаточно?** Он локален конкретной реплике/host и не задаёт общий объектный интерфейс/авторизацию. Временный spool не является постоянным оригиналом.
|
||||
3. **Что такое bucket/key?** Bucket — логический контейнер объектов; key — имя объекта внутри bucket, не путь ОС. Здесь key генерируется сервером.
|
||||
4. **Достаточно ли UUID против утечки?** Нет: знание ID не должно давать право читать чужой объект.
|
||||
5. **Почему filename не является key?** Защита от traversal, коллизий, перезаписи и смешения пользовательского имени с внутренним пространством ключей.
|
||||
6. **Почему недостаточно Content-Type?** Его выбирает клиент. Нужны сигнатура/декодер, config и полная проверка изображения.
|
||||
7. **Почему не просто DecodeConfig?** Он может прочитать правильный header повреждённого файла. Полный decode обнаруживает больше ошибок.
|
||||
8. **Что ограничивает RAM?** Bytes limit, pixels limit до full decode, semaphore, stream spool/PUT/download, cgroup memory limit. Каждая мера решает свою часть задачи.
|
||||
9. **Можно ли назвать upload асинхронным?** Нет, в ЛР 4 он синхронный и 201 ready. В ЛР 5 контракт меняется на 202 и фоновые состояния.
|
||||
10. **Когда отвечать 201?** Только после сохранения оригинала и COMMIT метаданных. После одного PUT этого недостаточно.
|
||||
11. **PUT прошёл, БД не закоммитила — что будет?** Intent и orphan останутся до безопасного reconciler. Автоматический retry INSERT не должен генерировать случайный новый key без учёта состояния.
|
||||
12. **Почему нельзя удалять сразу в defer?** Ошибка COMMIT может быть неопределённой: БД могла зафиксировать metadata. Сверка проверяет актуальное состояние.
|
||||
13. **Почему grace?** Чтобы незавершённая/только завершённая загрузка не выглядела мусором; дополнительная защита от гонок и времени обхода.
|
||||
14. **Зачем lock поверх grace?** Временной запас сам по себе не атомарен с проверкой/удалением. Lock согласует uploader и cleanup на одном ключе.
|
||||
15. **Как отличаешь missing от отказа сети?** Только конкретный S3 NoSuchKey/NotFound означает отсутствие. Другие HEAD ошибки запрещают repair.
|
||||
16. **Как восстановить missing used?** Из оригинала/backup. Hash не содержит файл. Ссылка сохраняется и проблема остаётся наблюдаемой.
|
||||
17. **Что будет при повторном запуске reconciler?** Уже удалённые orphan не появятся; исправленные unused metadata исчезнут; protected referenced missing останется сигналом до restore.
|
||||
18. **Где проверяется attachment другого чата?** В транзакции createMessage по `id AND chat_id`, до INSERT сообщения.
|
||||
19. **Почему [] и отсутствие — один body?** Это объявленная семантика API, нормализуется до hash идемпотентности.
|
||||
20. **Почему GET content без redirect?** Обязательный контракт требует авторизованный API и побайтный ответ. Presigned URLs — другой факультативный поток.
|
||||
21. **У приложения root key MinIO?** Нет. Root только bootstrap; API имеет отдельный user и policy одного bucket/prefix.
|
||||
22. **Есть HA S3?** Нет, один MinIO/volume на том же host. API failover не равен storage/host failover.
|
||||
23. **Что такое ETag?** Нельзя в общем случае считать его SHA-256 оригинала. Здесь SHA-256 вычисляет приложение по входным bytes и отдаёт в metadata.
|
||||
24. **Если SHA совпал после скачивания?** Подтверждено совпадение исходного и скачанного файла с криптографически малой вероятностью коллизии. Это не доказывает правильность ACL и backup.
|
||||
25. **Если S3 пропал, работают ли текстовые сообщения?** Ready в этой реализации считает S3 критичным для всего обязательного синхронного пути; HAProxy исключает API. Это сознательное грубое ограничение, можно развить отдельную деградацию маршрутов.
|
||||
26. **Есть ли автоматическая проверка порчи одинакового размера?** В download проверяются доступность и size; исследовательская проверка сверяет SHA. Reconciler сейчас проверяет наличие, не перечитывает каждый объект ради hash.
|
||||
|
||||
## Самопроверка: объяснить на бумаге
|
||||
|
||||
- Проведи границу SQL-транзакции на схеме upload. Покажи, почему PUT находится вне атомарности PostgreSQL.
|
||||
- Разбери crash до PUT, после PUT, после COMMIT и после потери ответа.
|
||||
- Покажи точную строчку, где filename мог бы стать опасным, если бы его использовали как путь.
|
||||
- Оцени память изображения 10 000×10 000 RGBA и скажи, на каком шаге оно отклоняется.
|
||||
- Предскажи HTTP-коды: неверный токен, чужой файл, другой чат, битый PNG, >10 MiB, несуществующий variant.
|
||||
- Сравни 10 одновременных клиентов и 10 одновременно декодируемых изображений.
|
||||
- Объясни, почему dry-run нужен перед ручной очисткой и почему он не заменяет locks.
|
||||
- Покажи, как сообщение и reconciler конкурируют за metadata: FOR KEY SHARE против FOR UPDATE.
|
||||
- Прочитай `consistency.json`: сколько найдено, сколько исправлено, что намеренно осталось и почему.
|
||||
- Объясни преподавателю, какую гарантию ты НЕ можешь получить из одного Docker-host и одного MinIO volume.
|
||||
|
||||
## Первичные материалы
|
||||
|
||||
- [MinIO Go SDK](https://pkg.go.dev/github.com/minio/minio-go/v7@v7.0.95)
|
||||
- [S3 PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html)
|
||||
- [Go image: DecodeConfig и Decode](https://pkg.go.dev/image)
|
||||
- [PostgreSQL locks](https://www.postgresql.org/docs/17/explicit-locking.html)
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
FROM golang:1.24.4-alpine3.22 AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY cmd/ ./cmd/
|
||||
COPY internal/ ./internal/
|
||||
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
|
||||
FROM build AS tests
|
||||
RUN apk add --no-cache gcc musl-dev
|
||||
ENV CGO_ENABLED=1
|
||||
CMD ["go", "test", "-race", "-count=1", "-v", "./..."]
|
||||
|
||||
FROM alpine:3.22.1
|
||||
RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
USER 10001:10001
|
||||
ENV APP_HOST=0.0.0.0 APP_PORT=8080 APP_ENV=lab
|
||||
EXPOSE 8080
|
||||
HEALTHCHECK --interval=5s --timeout=4s --start-period=3s --retries=3 CMD ["/usr/local/bin/messenger", "healthcheck"]
|
||||
STOPSIGNAL SIGTERM
|
||||
ENTRYPOINT ["/usr/local/bin/messenger"]
|
||||
@@ -0,0 +1,7 @@
|
||||
# ЛР 4 — БВТ 2303 — Крайкин — на 5
|
||||
|
||||
Добавлены приватное MinIO-хранилище, PostgreSQL metadata, авторизованный PNG/JPEG upload/download и message attachment_ids. Ограничены bytes/pixels/RAM; 201 ready возвращается после S3 PUT и COMMIT. Прежний text-only API, балансировка, общие сессии и идемпотентность сохранены.
|
||||
|
||||
Цель:5, трек 2 (согласованность двух хранилищ). Persistent upload journal и периодический reconciler поддерживают dry-run/apply/grace, повторно проверяют состояние под общим lock, защищают свежие и используемые вложения. Реальные сбои PUT/metadata, missing/orphan и обрыв передачи проверены; повторный repair не создаёт новых эффектов. Used missing требует восстановления оригинала отдельно, ссылка не удаляется.
|
||||
|
||||
Проверки: smoke, negative files/ACL, 10 concurrent uploads с cgroup memory, private bucket/limited app identity, полный restart, остановка именно uploader и download другой API, оба API/rolling, race tests, fault/reconcile experiment и чистый clone. Команды/raw results/границы — REPORT.md и evidence/. Контракт и публичные tests сохранены, помощь Codex раскрыта. DEFENSE.md/QUICKSTART.md/Postman приложены.
|
||||
@@ -1,5 +1,5 @@
|
||||
PYTHON ?= python3
|
||||
BASE_URL ?= http://localhost:8080
|
||||
BASE_URL ?= http://localhost:8084
|
||||
CA_FILE ?=
|
||||
ACTION_SCRIPT ?=
|
||||
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
|
||||
@@ -28,3 +28,30 @@ test-persistence:
|
||||
test-failover:
|
||||
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
|
||||
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||
|
||||
.PHONY: bootstrap up down test-student experiment reconcile
|
||||
bootstrap:
|
||||
$(PYTHON) scripts/bootstrap.py
|
||||
up: bootstrap
|
||||
docker compose --env-file .env up -d --build --wait
|
||||
down:
|
||||
docker compose --env-file .env down
|
||||
test-student:
|
||||
go test -race -count=1 ./...
|
||||
$(PYTHON) student-tests/attachments.py --base-url "$(BASE_URL)"
|
||||
experiment:
|
||||
$(PYTHON) student-tests/consistency.py
|
||||
reconcile:
|
||||
docker compose run --rm --no-deps api reconcile
|
||||
|
||||
test-race:
|
||||
docker compose --profile verify run --build --rm tests
|
||||
|
||||
clean-clone:
|
||||
$(PYTHON) student-tests/clean_clone.py
|
||||
snapshot:
|
||||
$(PYTHON) student-tests/snapshot.py
|
||||
test-permissions:
|
||||
$(PYTHON) student-tests/permissions.py
|
||||
inventory-cost:
|
||||
$(PYTHON) student-tests/inventory_cost.py
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# Запуск и демонстрация ЛР 4
|
||||
|
||||
Открой Docker Desktop. Нужны Docker Compose v2 с `--wait`, Python 3.9+ и make. Системный Go не нужен для обычного контейнерного запуска.
|
||||
|
||||
```sh
|
||||
cd /Users/glebkraykin/Projects/highload-apps-labs/lab4
|
||||
make up
|
||||
curl http://localhost:8084/health/ready
|
||||
make test
|
||||
python3 student-tests/attachments.py --base-url http://localhost:8084
|
||||
```
|
||||
|
||||
`make up` создаёт локальные secrets/.env, PostgreSQL, приватный MinIO, ограниченного S3 user, migrations, два API, HAProxy и периодическую сверку. Публичный адрес API — http://localhost:8084. Консоль MinIO/БД/API-реплики наружу не опубликованы. Для другого порта поменяй APP_PORT и передавай соответствующий BASE_URL тестам/Postman.
|
||||
|
||||
```sh
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
docker compose up -d --wait
|
||||
python3 student-tests/attachment_failover.py
|
||||
python3 student-tests/operations.py
|
||||
make test-race
|
||||
make experiment
|
||||
```
|
||||
|
||||
Эксперимент создаёт синтетические файлы, останавливает S3, намеренно портит только свои синтетические данные, выдерживает grace, проверяет repair и восстанавливает used missing оригинал из fixture. На время запускать отдельно от других тестов. Проход может занимать несколько минут.
|
||||
|
||||
Ручная безопасная сверка:
|
||||
|
||||
```sh
|
||||
make reconcile
|
||||
# Явное применение с обычным grace 1 час:
|
||||
docker compose run --rm --no-deps api reconcile --apply
|
||||
```
|
||||
|
||||
Без `--apply` ничего не удаляется. Автоматический reconciler работает раз в минуту с grace 1 час. Для учебного fault-experiment используется `--grace-seconds 60`; меньше 60 сервер не принимает. Содержимое используемых missing объектов нужно восстанавливать отдельно, JSON counts показывают проблему.
|
||||
|
||||
```sh
|
||||
docker compose logs --tail 10 reconciler
|
||||
docker compose logs --tail 20 api
|
||||
make down
|
||||
```
|
||||
|
||||
Остановка сохраняет volumes PostgreSQL/MinIO. `down -v` удаляет данные намеренно; рабочие проверки так не делают.
|
||||
|
||||
Postman: импортируй `postman/lab4.postman_collection.json`. Для binary upload выбери локальный `tests/fixtures/sample.png` в Body → binary. Токены/ID сохраняются автоматически; после upload будут доступны metadata, content и message только с файлом. Перед экспортом коллекции очисти живые токены.
|
||||
|
||||
Подготовка к защите — `DEFENSE.md`; реальные результаты — `REPORT.md` и `evidence/`.
|
||||
@@ -1,37 +1,116 @@
|
||||
# Отчёт — лабораторная 4
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор: **Крайкин Глеб Алексеевич, БВТ 2303**.
|
||||
- Целевая оценка: **5**, трек 2 — согласованность двух хранилищ.
|
||||
- Skeleton: `ec540a1a725862a8619fcb758daec4ad197f0dd5`.
|
||||
- Ветка: `solution/bvt2303-kraykin`, commit: `git rev-parse HEAD`.
|
||||
- Проверки: 29.10.2026, Docker Desktop Linux/arm64, macOS arm64. Реальные контейнеры/ресурсы — `evidence/runtime.json`.
|
||||
- Перенесены только собственные исходники/миграции/конфигурации из текущего решения ЛР 3: Go API, PostgreSQL sessions/ACL/history/idempotency, Valkey cache, HAProxy и несколько API. Публичные tests/contract/README/COURSE текущей ЛР 4 сохранены.
|
||||
- ИИ: Codex помог с проектированием, реализацией, тестами, исследованием и подготовкой защиты. S3 — официальный MinIO Go SDK v7.0.95, image/png и image/jpeg — Go standard library. Чужие студенческие решения не использовались.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Docker/Compose v2 с --wait, Python 3.9+ и make. Go 1.24+ нужен только для локальных Go-команд; контейнерный make test-race не требует host Go.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
make up
|
||||
make check
|
||||
make test
|
||||
python3 student-tests/attachments.py --base-url http://localhost:8084
|
||||
make test-race
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
docker compose up -d --wait
|
||||
python3 student-tests/attachment_failover.py
|
||||
python3 student-tests/operations.py
|
||||
make experiment
|
||||
make reconcile
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Bootstrap генерирует .env и локальные секреты, не перезаписывает существующие. Создаются приватный bucket и отдельный app user/policy, затем отдельные миграции и API/LB/reconciler. Зависимости не требуются заранее. Только host port `127.0.0.1:8084` у HAProxy. API/PostgreSQL/Valkey/S3 не публикуют host ports, консоль MinIO bind 127.0.0.1 внутри S3-контейнера.
|
||||
|
||||
## Доказательства критериев
|
||||
Pinned images: Go 1.24.4-alpine3.22 build, Alpine 3.22.1 runtime, PostgreSQL 17.6-alpine, Valkey 8.1.3-alpine, HAProxy 3.2.6-alpine, MinIO `RELEASE.2025-09-07T16-13-09Z` (quay.io), mc `RELEASE.2025-08-13T08-35-41Z`. Runtime API UID10001, PID1 messenger; deps/go.sum закреплены.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Лимиты по умолчанию: API ×2 — по 1 CPU/768 MiB, GOMEMLIMIT600MiB, один active upload на API; PostgreSQL 2 CPU/512 MiB; S3 1 CPU/512 MiB; Valkey0.5 CPU/128 MiB; LB1 CPU/128 MiB; reconciler0.25 CPU/128 MiB, pool2. Image IDs/StartedAt/cgroup limits — runtime.json.
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
Настройки .env.example сохранены. APP_PORT — host port; SESSION_TTL_SECONDS — абсолютная SQL session expiry; DATABASE/CACHE/CURSOR secret paths — как ЛР 3. S3_ENDPOINT/BUCKET и файлы app credentials используются API; region фиксирован us-east-1. MAX_UPLOAD_BYTES/MAX_IMAGE_PIXELS и IDEMPOTENCY_TTL_SECONDS в skeleton — описательные интерфейсные предложения: runtime использует фиксированные контрактные 10 MiB/40M/24h. Увеличить их переменной окружения нельзя. API_REPLICAS/API_CPUS/API_MEMORY/DB_POOL_MAX — дополнительные настройки стенда.
|
||||
|
||||
## Эксперименты и отказы
|
||||
```sh
|
||||
make down
|
||||
make up
|
||||
# Только намеренный сброс ненужных учебных данных:
|
||||
# docker compose down -v
|
||||
```
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
PostgreSQL и MinIO имеют именованные volumes. Обычные тесты их не удаляют; clean-clone работает с собственным новым project. Secrets excluded Git/build context, root S3 credentials не монтируются в API. Существующий пароль PostgreSQL меняется отдельно в БД.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Migration job получает только database_url. Reconciler получает только database_url и app S3 credentials; cache URL/cursor key нужны только HTTP API. Команды CLI завершают нужный путь до чтения остальных секретов.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
## Архитектура и модель
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Клиент → HAProxy → API ×2. PostgreSQL — metadata, users/chats/membership/messages/sessions/idempotency. MinIO — неизменяемые оригиналы `originals/<UUID>`. Valkey — необязательный public profile cache. Новая migration003 добавляет attachments, message_attachments, upload_intents и разрешает пустой текст только на уровне business-validation при наличии attachments. Старые migrations не переписаны.
|
||||
|
||||
Upload: session/ACL → filename/MIME/byte limits → semaphore → capped temporary spool + SHA256 → DecodeConfig/pixel limit → full Decode → durable intent COMMIT → BEGIN + per-key advisory lock → S3 PUT → INSERT metadata + DELETE intent → COMMIT →201 ready. PUT и metadata не являются одной распределённой транзакцией. Persistent intent и периодический reconciler компенсируют неопределённые/незавершённые операции без опасного немедленного DELETE после неоднозначного COMMIT.
|
||||
|
||||
Spool ограничен 10 MiB, удаляется через defer и не используется как обязательный оригинал. Размер/пиксели ограничиваются до неограниченного чтения/декодирования. Полный decode всё же выделяет память изображения (для 40M NRGBA64 около 320 MB), поэтому одновременно один upload/API. HTTP upload/download budgets30s, read/write35s, SQL statement2s/lock1s. Скачивание io.Copy, metadata/Stat до начала 200, оригинальные bytes без redirect/перекодирования. Readiness требует PostgreSQL и доступного bucket; это грубая общая деградация при отказе S3, включая текстовые маршруты через LB.
|
||||
|
||||
Message normalizes missing attachment_ids to [], rejects null/duplicate/>10, verifies every attachment in this chat under FOR KEY SHARE; message+links+idempotency result commit together. Сессия и ACL заново проверяются при replay. Text-only прежние POST сохранены, attachment-only text="" разрешён. S3 key скрыт из public metadata; display filename не является путём.
|
||||
|
||||
MinIO app policy: ListBucket/GetBucketLocation одного bucket; GetObject/PutObject/DeleteObject только originals/*; нет bucket policy/admin/user management. Bootstrap mc с root credentials отдельно создаёт bucket, запрещает anonymous, создаёт/назначает app policy повторяемо. Проверен anonymous GET известного существующего объекта →403.
|
||||
|
||||
## Reconciler и трек на пять
|
||||
|
||||
Гипотеза: durable upload journal + повторная сверка под тем же per-key lock позволяют обнаружить и безопасно исправить рассогласование S3/БД, не удаляя свежие объекты или metadata используемых вложений.
|
||||
|
||||
`reconcile` по умолчанию dry-run. `--apply` применяет удаления, grace default3600s/minimum60s >upload deadline30s. Compose loop запускает проход раз в 60s; при ошибке зависимости проход завершается и повторяется позднее. Health loop проверяет живой процесс, успешность — JSON логов. Дополнительный ручной dry-run:
|
||||
|
||||
```sh
|
||||
docker compose run --rm --no-deps api reconcile
|
||||
# Только на синтетическом стенде: применить с grace60s
|
||||
docker compose run --rm --no-deps api reconcile --grace-seconds 60 --apply
|
||||
```
|
||||
|
||||
Алгоритм: LIST prefix + union SQL keys → на ключ BEGIN/advisory lock → metadata FOR UPDATE + актуальные intents + S3 HEAD → защитить свежие → удалить старый orphan → для missing удалить только unused old metadata → удалить abandoned intent → COMMIT. Ошибка сети/5xx/permission не трактуется как отсутствие объекта. Защита используемых metadata согласована с FOR KEY SHARE сообщения. Файлы used missing не реконструируются из hash: сохраняем ссылку и сигнализируем для restore из оригинала/backup.
|
||||
|
||||
Метод `student-tests/consistency.py`: реальные synthetic uploads; SQL trigger отклоняет metadata INSERT после PUT; отдельно удаляются объект и unused metadata; настоящий TCP socket прерывает передачу; MinIO останавливается. Выдерживается реальный grace62s, затем создаются fresh object/metadata. Выполняются dry-run, apply, rerun; valid used original скачивается и сверяется, referenced missing metadata сохраняется и затем файл восстанавливается из fixture. Нет публичного fault endpoint.
|
||||
|
||||
Результат первого полного прохода: после DB fault durable intents1/metadata0; interrupted metadata0; S3 outage503 за 0.721s; anonymous403. Dry-run на 11 ключах обнаружил orphan2/missing2/stale intents2, fresh protected2, referenced protected1; duration15ms. Apply удалил 2 объекта/1 unused metadata/2 intents за 18 ms. Rerun7 ключей за 13 ms, повторных удалений 0; referenced missing оставался защищённым до отдельного restore оригинала. Полный raw — consistency.json. В следующем повторе counts могут расти из-за синтетических fixture предыдущих запусков; инварианты проверяются assertions.
|
||||
|
||||
Дополнительно `student-tests/inventory_cost.py` измерил dry-run на 19/119/1019 ключах:21/73/498 ms, все свежие объекты защищены, удалений 0. Синтетический отдельный prefix после проверки удалён; baseline данные не менялись. Raw: `evidence/inventory-cost.json`. Стоимость обхода O(N), HEAD + несколько SQL roundtrips на ключ; это локальные малые наборы, не оценка petabyte bucket. Inventory map O(N) RAM; для больших bucket нужны bounded batches/cursor/rate limiting и отдельный durable reconciliation ledger. SHA-сверка всех stored bytes не входит в текущий reconciler.
|
||||
|
||||
## Доказательства каждого критерия
|
||||
|
||||
| Критерий | Команда/наблюдение | Evidence |
|
||||
|---|---|---|
|
||||
| Прежний обязательный API и новый file API | make check/test | check.txt, smoke.txt |
|
||||
| Минимум lab3: несколько API/общие sessions/idempotency | public failover/persistence, race HTTP, operations | failover.txt, persistence.txt, race.txt, operations.json |
|
||||
| Байты S3, metadata PostgreSQL | Docker/SQL inspection, image download | runtime.json, migration003, consistency.json |
|
||||
| 201 ready только после PUT+COMMIT | код + after-PUT DB failure503 | attachments.go, consistency.json |
|
||||
| SHA/byte identity участнику, запрет постороннему | public smoke +10downloads | smoke.txt, attachments.json |
|
||||
| Byte/pixels/filename/MIME/corrupt limits | own negative tests | attachments.txt, attachments.json |
|
||||
| Cross-chat/unknown attachment IDs | 404; normalization []/omission same result | attachments.txt |
|
||||
| Остановлен именно uploader →другой API скачал | alias → real container inspect, byte comparison | attachment-failover.json |
|
||||
| Полный restart сохраняет file/session/metadata | same original и прежний Bearer | persistence.txt, persistence-processes.json |
|
||||
| Обрыв передачи/ошибка после PUT не оставляют вечный мусор | durable intents, automatic loop, grace+repair | consistency.json, reconcile.go |
|
||||
| ≥10 simultaneous uploads, измерение memory peak |10HTTP clients, capped decode semaphore | attachments.json |
|
||||
| Повторяемый private bucket/bootstrap, app credentials | repeated startup, anonymous403, policy; app admin/bucket-policy changes denied | startup-final.txt, consistency.json, permissions.json, s3-policy.json |
|
||||
| Трек 5: fault обеих сторон/dry-run/apply/rerun | реальный trigger, missing/orphan, safe repair | consistency.txt/json |
|
||||
| Защита fresh/used и стоимость обхода | protected counts, valid byte identity, duration | consistency.json |
|
||||
| Чистый clone, новые secrets/volumes | smoke/persistence с пустого стенда | clean-clone.txt |
|
||||
|
||||
## Память и нагрузка
|
||||
|
||||
10 одновременных HTTP upload синтетического PNG1024×768, 786432 pixels, 2360847 bytes. В первом полном прогоне upload+проверка скачиваний заняли 0.776s; все SHA256 совпали. До окна cgroup memory current около 137–138 MiB/API, peak около 137–138 MiB; после окна peak около 141–142 MiB/API. Exact before/after и sampled current/peak — attachments.json. Это memory cgroup (включая cache), не только heap; peak с начала жизни контейнера. Одновременно один декодер/API, остальные запросы ждут. Не заявляется такой же пик для 40M pixels, универсальный RPS upload или модель unlimited concurrency.
|
||||
|
||||
## Ограничения
|
||||
|
||||
Нет HA PostgreSQL/MinIO/LB/host, удалённого backup или гарантии против disk loss. Приватный bucket не заменяет ACL; случайный key не секрет. Нет presigned flow, thumbnail/worker (это ЛР 5), редактирования/удаления пользовательских файлов, полного hash-scrub хранилища, quota/rate limiting и разделения DB runtime/migration roles. Удаление bytes referenced объекта требует внешнего restore; reconciler безопасно сообщает проблему и не уничтожает ссылку. Local HTTP только synthetic; для удалённой демонстрации нужен TLS.
|
||||
|
||||
Материалы: DEFENSE.md, QUICKSTART.md, Postman, исходники и все raw evidence. Оценку выставляет преподаватель после демонстрации и объяснения, PASS сам по себе её не гарантирует.
|
||||
|
||||
## Первичные источники
|
||||
|
||||
- [MinIO Go SDK v7.0.95](https://pkg.go.dev/github.com/minio/minio-go/v7@v7.0.95)
|
||||
- [S3 PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html)
|
||||
- [Go image](https://pkg.go.dev/image)
|
||||
- [PostgreSQL locks](https://www.postgresql.org/docs/17/explicit-locking.html)
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"log/slog"
|
||||
"mtusi/messenger/internal/database"
|
||||
"mtusi/messenger/internal/messenger"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
func env(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
slog.Error("server failed", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
func run() error {
|
||||
host := env("APP_HOST", "0.0.0.0")
|
||||
port := env("APP_PORT", "8080")
|
||||
p, err := strconv.Atoi(port)
|
||||
if err != nil || p < 1 || p > 65535 {
|
||||
return fmt.Errorf("invalid APP_PORT")
|
||||
}
|
||||
if len(os.Args) == 2 && os.Args[1] == "healthcheck" {
|
||||
c := http.Client{Timeout: time.Second}
|
||||
r, e := c.Get("http://127.0.0.1:" + port + "/health/ready")
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
defer r.Body.Close()
|
||||
if r.StatusCode != 200 {
|
||||
return fmt.Errorf("not ready: %d", r.StatusCode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
|
||||
ttl, err := strconv.Atoi(env("SESSION_TTL_SECONDS", "3600"))
|
||||
if err != nil || ttl < 1 || ttl > 86400 {
|
||||
return fmt.Errorf("SESSION_TTL_SECONDS must be 1..86400")
|
||||
}
|
||||
databaseURL, err := secret("DATABASE_URL_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
startup, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
pool, err := database.Open(startup, databaseURL)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer pool.Close()
|
||||
if len(os.Args) == 2 && os.Args[1] == "migrate" {
|
||||
return database.Migrate(startup, pool)
|
||||
}
|
||||
storeAccess, err := secret("S3_ACCESS_KEY_ID_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
storeSecret, err := secret("S3_SECRET_ACCESS_KEY_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
objects, err := messenger.NewObjectStore(env("S3_ENDPOINT", "http://s3:9000"), env("S3_BUCKET", "messenger-attachments"), storeAccess, storeSecret)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(os.Args) > 1 && os.Args[1] == "reconcile" {
|
||||
return messenger.Reconcile(pool, objects, os.Args[2:])
|
||||
}
|
||||
cacheURL, err := secret("CACHE_URL_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cursorKey, err := secret("CURSOR_KEY_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
key, err := base64.RawURLEncoding.DecodeString(cursorKey)
|
||||
if err != nil || len(key) != 32 {
|
||||
return fmt.Errorf("invalid cursor key")
|
||||
}
|
||||
opts, err := redis.ParseURL(cacheURL)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid cache configuration")
|
||||
}
|
||||
opts.DialTimeout = 60 * time.Millisecond
|
||||
opts.ReadTimeout = 60 * time.Millisecond
|
||||
opts.WriteTimeout = 60 * time.Millisecond
|
||||
opts.PoolTimeout = 60 * time.Millisecond
|
||||
opts.ContextTimeoutEnabled = true
|
||||
opts.MaxRetries = -1
|
||||
opts.PoolSize = 10
|
||||
cache := redis.NewClient(opts)
|
||||
defer cache.Close()
|
||||
api := messenger.New(pool, cache, key, ttl, log)
|
||||
api.SetObjectStore(objects)
|
||||
srv := &http.Server{Addr: net.JoinHostPort(host, port), Handler: api, ReadHeaderTimeout: 3 * time.Second, ReadTimeout: 35 * time.Second, WriteTimeout: 35 * time.Second, IdleTimeout: 30 * time.Second, MaxHeaderBytes: 16 << 10}
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
log.Info("listening", "address", srv.Addr, "session_store", "postgres", "profile_cache", "valkey", "environment", env("APP_ENV", "lab"))
|
||||
done <- srv.ListenAndServe()
|
||||
}()
|
||||
select {
|
||||
case err := <-done:
|
||||
return err
|
||||
case <-ctx.Done():
|
||||
}
|
||||
start := time.Now()
|
||||
api.Stop()
|
||||
log.Info("shutdown started")
|
||||
shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := srv.Shutdown(shutdown); err != nil {
|
||||
_ = srv.Close()
|
||||
return err
|
||||
}
|
||||
err = <-done
|
||||
if !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
log.Info("shutdown completed", "duration_ms", time.Since(start).Milliseconds())
|
||||
return nil
|
||||
}
|
||||
|
||||
func secret(name string) (string, error) {
|
||||
path := os.Getenv(name)
|
||||
if path == "" {
|
||||
return "", fmt.Errorf("missing %s", name)
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("cannot read %s", name)
|
||||
}
|
||||
return strings.TrimSpace(string(raw)), nil
|
||||
}
|
||||
+140
@@ -0,0 +1,140 @@
|
||||
name: mtusi-lab4-kraykin
|
||||
x-api: &api
|
||||
image: mtusi-messenger:lab4
|
||||
build: .
|
||||
environment: &api-env
|
||||
APP_PORT: 8080
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
CURSOR_KEY_FILE: /run/secrets/cursor_key
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
GOMEMLIMIT: 600MiB
|
||||
DB_POOL_MAX: ${DB_POOL_MAX:-10}
|
||||
secrets: [database_url, cache_url, cursor_key, s3_access_key_id, s3_secret_access_key]
|
||||
cpus: ${API_CPUS:-1}
|
||||
mem_limit: ${API_MEMORY:-768m}
|
||||
stop_grace_period: 8s
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17.6-alpine
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets: [postgres_password]
|
||||
volumes: [postgres_data:/var/lib/postgresql/data]
|
||||
cpus: 2
|
||||
mem_limit: 512m
|
||||
shm_size: 128m
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'pg_isready -U messenger -d messenger']
|
||||
interval: 1s
|
||||
timeout: 2s
|
||||
retries: 30
|
||||
valkey:
|
||||
image: valkey/valkey:8.1.3-alpine
|
||||
command: [valkey-server, /run/secrets/valkey_config]
|
||||
secrets: [valkey_config]
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
s3:
|
||||
image: quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z
|
||||
command: [server, /data, --console-address, '127.0.0.1:9001']
|
||||
environment:
|
||||
MINIO_ROOT_USER_FILE: /run/secrets/s3_root_user
|
||||
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
|
||||
secrets: [s3_root_user, s3_root_password]
|
||||
volumes: [s3_data:/data]
|
||||
cpus: 1
|
||||
mem_limit: 512m
|
||||
healthcheck:
|
||||
test: [CMD, curl, -f, http://localhost:9000/minio/health/ready]
|
||||
interval: 1s
|
||||
timeout: 2s
|
||||
retries: 30
|
||||
s3-init:
|
||||
image: minio/mc:RELEASE.2025-08-13T08-35-41Z
|
||||
entrypoint: [/bin/sh, /config/s3-init.sh]
|
||||
volumes: ['./infra:/config:ro']
|
||||
secrets: [s3_root_user, s3_root_password, s3_access_key_id, s3_secret_access_key]
|
||||
depends_on:
|
||||
s3: {condition: service_healthy}
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
migrate:
|
||||
<<: *api
|
||||
secrets: [database_url]
|
||||
command: [migrate]
|
||||
depends_on:
|
||||
postgres: {condition: service_healthy}
|
||||
restart: 'no'
|
||||
api:
|
||||
<<: *api
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
depends_on:
|
||||
s3-init: {condition: service_completed_successfully}
|
||||
migrate: {condition: service_completed_successfully}
|
||||
valkey: {condition: service_started}
|
||||
reconciler:
|
||||
<<: *api
|
||||
secrets: [database_url, s3_access_key_id, s3_secret_access_key]
|
||||
entrypoint: [/bin/sh, /config/reconcile-loop.sh]
|
||||
command: []
|
||||
volumes: ['./infra/reconcile-loop.sh:/config/reconcile-loop.sh:ro']
|
||||
environment:
|
||||
<<: *api-env
|
||||
DB_POOL_MAX: 2
|
||||
cpus: 0.25
|
||||
mem_limit: 128m
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'kill -0 1']
|
||||
interval: 5s
|
||||
timeout: 1s
|
||||
retries: 3
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
s3-init: {condition: service_completed_successfully}
|
||||
tests:
|
||||
image: mtusi-messenger-tests:lab4
|
||||
cpus: 2
|
||||
mem_limit: 1024m
|
||||
profiles: [verify]
|
||||
build:
|
||||
context: .
|
||||
target: tests
|
||||
environment:
|
||||
DATABASE_TEST_URL_FILE: /run/secrets/database_url
|
||||
secrets: [database_url]
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
lb:
|
||||
image: haproxy:3.2.6-alpine
|
||||
volumes: ['./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro']
|
||||
ports: ['127.0.0.1:${APP_PORT:-8084}:8080']
|
||||
cpus: 1
|
||||
mem_limit: 128m
|
||||
depends_on:
|
||||
api: {condition: service_healthy}
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'wget -q -O /dev/null http://127.0.0.1:8080/health/ready']
|
||||
interval: 2s
|
||||
timeout: 4s
|
||||
retries: 20
|
||||
secrets:
|
||||
s3_root_user: {file: ./secrets/s3_root_user}
|
||||
s3_root_password: {file: ./secrets/s3_root_password}
|
||||
s3_access_key_id: {file: ./secrets/s3_access_key_id}
|
||||
s3_secret_access_key: {file: ./secrets/s3_secret_access_key}
|
||||
postgres_password: {file: ./secrets/postgres_password}
|
||||
database_url: {file: ./secrets/database_url}
|
||||
valkey_config: {file: ./secrets/valkey_config}
|
||||
cache_url: {file: ./secrets/cache_url}
|
||||
cursor_key: {file: ./secrets/cursor_key}
|
||||
volumes:
|
||||
s3_data:
|
||||
postgres_data:
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"upload_instance": "740499345a61",
|
||||
"stopped_container": "/mtusi-lab4-kraykin-api-1",
|
||||
"download_instance": "d270450ddb48",
|
||||
"byte_equal": true,
|
||||
"first_recovery_seconds": 0.2448889999999999,
|
||||
"errors": 0
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: exact uploader stopped; another API returned identical original
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"negative_cases": 15,
|
||||
"simultaneous_clients": 10,
|
||||
"fixture_bytes": 2360847,
|
||||
"fixture_pixels": 786432,
|
||||
"duration_seconds": 0.7762856660000002,
|
||||
"before_memory_current_peak_bytes": {
|
||||
"740499345a61": [
|
||||
144093184,
|
||||
144244736
|
||||
],
|
||||
"d270450ddb48": [
|
||||
144650240,
|
||||
144732160
|
||||
]
|
||||
},
|
||||
"after_memory_current_peak_bytes": {
|
||||
"740499345a61": [
|
||||
144769024,
|
||||
148447232
|
||||
],
|
||||
"d270450ddb48": [
|
||||
144351232,
|
||||
148041728
|
||||
]
|
||||
},
|
||||
"sampled_memory_current_peak_bytes": [
|
||||
{
|
||||
"740499345a61": [
|
||||
147968000,
|
||||
148447232
|
||||
],
|
||||
"d270450ddb48": [
|
||||
144375808,
|
||||
148041728
|
||||
]
|
||||
}
|
||||
],
|
||||
"all_sha256_match": true,
|
||||
"per_api_upload_slots": 1
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: file boundaries, ACL, normalization and 10 simultaneous uploads
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
@@ -0,0 +1,297 @@
|
||||
Cloning into '/var/folders/hx/czbzpyk92ds_mnf7n9xrzn_80000gn/T/verify-lab4-abb798930c-q4q2sj3b/repo'...
|
||||
done.
|
||||
python3 scripts/bootstrap.py
|
||||
Local secrets ready (values hidden); .env and secrets/ are ignored by Git.
|
||||
docker compose --env-file .env up -d --build --wait
|
||||
Compose can now delegate builds to bake for better performance.
|
||||
To do so, set COMPOSE_BAKE=true.
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [migrate internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 782B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 ...
|
||||
|
||||
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 0.3s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 1.0s
|
||||
|
||||
#4 [migrate internal] load .dockerignore
|
||||
#4 transferring context: 196B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [migrate stage-2 1/3] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate internal] load build context
|
||||
#7 transferring context: 59.95kB done
|
||||
#7 DONE 0.0s
|
||||
|
||||
#8 [migrate build 2/7] WORKDIR /src
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate build 3/7] COPY go.mod go.sum ./
|
||||
#9 CACHED
|
||||
|
||||
#10 [migrate build 4/7] RUN go mod download
|
||||
#10 CACHED
|
||||
|
||||
#11 [migrate build 5/7] COPY cmd/ ./cmd/
|
||||
#11 DONE 0.0s
|
||||
|
||||
#12 [migrate build 6/7] COPY internal/ ./internal/
|
||||
#12 DONE 0.0s
|
||||
|
||||
#13 [migrate build 7/7] RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#13 ...
|
||||
|
||||
#14 [migrate stage-2 2/3] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#14 CACHED
|
||||
|
||||
#13 [migrate build 7/7] RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#13 DONE 9.3s
|
||||
|
||||
#15 [migrate stage-2 3/3] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#15 DONE 0.0s
|
||||
|
||||
#16 [migrate] exporting to image
|
||||
#16 exporting layers
|
||||
#16 exporting layers 0.2s done
|
||||
#16 exporting manifest sha256:7b3e3366e0849ae3a26482892f53318f1f9a607b3c968c6ce7f0cf45ec703d0d
|
||||
#16 exporting manifest sha256:7b3e3366e0849ae3a26482892f53318f1f9a607b3c968c6ce7f0cf45ec703d0d done
|
||||
#16 exporting config sha256:0078b0a280b4e1b8b93f252249b55b93929e1b271a343898b14583db3bd9386a done
|
||||
#16 exporting attestation manifest sha256:66becfa734707d5690b7bea61a39ff86c74c707ef17205d36b2fe32ef8d6f1b7 done
|
||||
#16 exporting manifest list sha256:7a15bd364baad85889c9443d61b06b8aa4864619c66f970031369accf9218711 done
|
||||
#16 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#16 unpacking to docker.io/library/mtusi-messenger:lab4 0.0s done
|
||||
#16 DONE 0.3s
|
||||
|
||||
#17 [migrate] resolving provenance for metadata file
|
||||
#17 DONE 0.0s
|
||||
|
||||
#18 [reconciler internal] load build definition from Dockerfile
|
||||
#18 transferring dockerfile: 782B done
|
||||
#18 DONE 0.0s
|
||||
|
||||
#19 [api internal] load build definition from Dockerfile
|
||||
#19 transferring dockerfile: 782B done
|
||||
#19 DONE 0.0s
|
||||
|
||||
#2 [reconciler internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 1.3s
|
||||
|
||||
#3 [reconciler internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 0.6s
|
||||
|
||||
#20 [api internal] load .dockerignore
|
||||
#20 transferring context: 196B done
|
||||
#20 DONE 0.0s
|
||||
|
||||
#21 [reconciler internal] load .dockerignore
|
||||
#21 transferring context: 196B done
|
||||
#21 DONE 0.0s
|
||||
|
||||
#6 [api build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#5 [api stage-2 1/3] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#22 [api internal] load build context
|
||||
#22 transferring context: 858B done
|
||||
#22 DONE 0.0s
|
||||
|
||||
#23 [reconciler internal] load build context
|
||||
#23 transferring context: 59.95kB done
|
||||
#23 DONE 0.0s
|
||||
|
||||
#24 [api build 6/7] COPY internal/ ./internal/
|
||||
#24 CACHED
|
||||
|
||||
#8 [api build 2/7] WORKDIR /src
|
||||
#8 CACHED
|
||||
|
||||
#25 [api build 4/7] RUN go mod download
|
||||
#25 CACHED
|
||||
|
||||
#26 [api build 5/7] COPY cmd/ ./cmd/
|
||||
#26 CACHED
|
||||
|
||||
#27 [api build 3/7] COPY go.mod go.sum ./
|
||||
#27 CACHED
|
||||
|
||||
#28 [api build 7/7] RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#28 CACHED
|
||||
|
||||
#14 [reconciler stage-2 2/3] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#14 CACHED
|
||||
|
||||
#29 [reconciler stage-2 3/3] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#29 CACHED
|
||||
|
||||
#30 [reconciler] exporting to image
|
||||
#30 exporting layers done
|
||||
#30 exporting manifest sha256:03f22e0e818b670fdafac423d3fdee196cec87c07b55b7e5dd36fcbd9e2ca96b done
|
||||
#30 exporting config sha256:b4d3337e5c4beeac6b2f65b911db99a3ffad41d02104e823fea2cd54d1f350ea 0.0s done
|
||||
#30 exporting attestation manifest sha256:c309fb37ba5fab930bdf5b472b7bfa14432aec4b6064c3e2e21e3d75aa9e493f 0.0s done
|
||||
#30 exporting manifest list sha256:c40901c24384b907f171a82ac026288112c35a98ecd7ec0b11393dbe81085971
|
||||
#30 exporting manifest list sha256:c40901c24384b907f171a82ac026288112c35a98ecd7ec0b11393dbe81085971 done
|
||||
#30 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#30 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#30 DONE 0.1s
|
||||
|
||||
#31 [api] exporting to image
|
||||
#31 exporting layers done
|
||||
#31 exporting manifest sha256:8b94d76b1c982e37d2ca4558591d582fcbc5edcc685e1083b849b61e1940f275 done
|
||||
#31 exporting config sha256:14b65e422341bd6614639d320241d96e708294d83ed9b8162ffbe8c0a4dfac37 0.0s done
|
||||
#31 exporting attestation manifest sha256:95d4568214b8c8430951166ae1332c7214b91068d01c28965d2eff4e0ffd131e 0.0s done
|
||||
#31 exporting manifest list sha256:3b2a53cc95740a2ec3a837c5b54a4ee6b1dd3348fd184b5ad8a6c145da0066f3 done
|
||||
#31 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#31 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#31 DONE 0.1s
|
||||
|
||||
#32 [reconciler] resolving provenance for metadata file
|
||||
#32 DONE 0.0s
|
||||
|
||||
#33 [api] resolving provenance for metadata file
|
||||
#33 DONE 0.0s
|
||||
api Built
|
||||
migrate Built
|
||||
reconciler Built
|
||||
Network verify-lab4-abb798930c_default Creating
|
||||
Network verify-lab4-abb798930c_default Created
|
||||
Volume "verify-lab4-abb798930c_s3_data" Creating
|
||||
Volume "verify-lab4-abb798930c_s3_data" Created
|
||||
Volume "verify-lab4-abb798930c_postgres_data" Creating
|
||||
Volume "verify-lab4-abb798930c_postgres_data" Created
|
||||
Container verify-lab4-abb798930c-valkey-1 Creating
|
||||
Container verify-lab4-abb798930c-postgres-1 Creating
|
||||
Container verify-lab4-abb798930c-s3-1 Creating
|
||||
Container verify-lab4-abb798930c-valkey-1 Created
|
||||
Container verify-lab4-abb798930c-s3-1 Created
|
||||
Container verify-lab4-abb798930c-s3-init-1 Creating
|
||||
Container verify-lab4-abb798930c-postgres-1 Created
|
||||
Container verify-lab4-abb798930c-migrate-1 Creating
|
||||
Container verify-lab4-abb798930c-s3-init-1 Created
|
||||
Container verify-lab4-abb798930c-migrate-1 Created
|
||||
Container verify-lab4-abb798930c-api-2 Creating
|
||||
Container verify-lab4-abb798930c-reconciler-1 Creating
|
||||
Container verify-lab4-abb798930c-api-1 Creating
|
||||
Container verify-lab4-abb798930c-reconciler-1 Created
|
||||
Container verify-lab4-abb798930c-api-1 Created
|
||||
Container verify-lab4-abb798930c-api-2 Created
|
||||
Container verify-lab4-abb798930c-lb-1 Creating
|
||||
Container verify-lab4-abb798930c-lb-1 Created
|
||||
Container verify-lab4-abb798930c-s3-1 Starting
|
||||
Container verify-lab4-abb798930c-postgres-1 Starting
|
||||
Container verify-lab4-abb798930c-valkey-1 Starting
|
||||
Container verify-lab4-abb798930c-s3-1 Started
|
||||
Container verify-lab4-abb798930c-s3-1 Waiting
|
||||
Container verify-lab4-abb798930c-postgres-1 Started
|
||||
Container verify-lab4-abb798930c-postgres-1 Waiting
|
||||
Container verify-lab4-abb798930c-valkey-1 Started
|
||||
Container verify-lab4-abb798930c-s3-1 Healthy
|
||||
Container verify-lab4-abb798930c-s3-init-1 Starting
|
||||
Container verify-lab4-abb798930c-postgres-1 Healthy
|
||||
Container verify-lab4-abb798930c-migrate-1 Starting
|
||||
Container verify-lab4-abb798930c-migrate-1 Started
|
||||
Container verify-lab4-abb798930c-s3-init-1 Started
|
||||
Container verify-lab4-abb798930c-migrate-1 Waiting
|
||||
Container verify-lab4-abb798930c-s3-init-1 Waiting
|
||||
Container verify-lab4-abb798930c-migrate-1 Waiting
|
||||
Container verify-lab4-abb798930c-s3-init-1 Waiting
|
||||
Container verify-lab4-abb798930c-migrate-1 Exited
|
||||
Container verify-lab4-abb798930c-migrate-1 Exited
|
||||
Container verify-lab4-abb798930c-s3-init-1 Exited
|
||||
Container verify-lab4-abb798930c-api-2 Starting
|
||||
Container verify-lab4-abb798930c-s3-init-1 Exited
|
||||
Container verify-lab4-abb798930c-reconciler-1 Starting
|
||||
Container verify-lab4-abb798930c-reconciler-1 Started
|
||||
Container verify-lab4-abb798930c-api-2 Started
|
||||
Container verify-lab4-abb798930c-api-1 Starting
|
||||
Container verify-lab4-abb798930c-api-1 Started
|
||||
Container verify-lab4-abb798930c-api-2 Waiting
|
||||
Container verify-lab4-abb798930c-api-1 Waiting
|
||||
Container verify-lab4-abb798930c-api-2 Healthy
|
||||
Container verify-lab4-abb798930c-api-1 Healthy
|
||||
Container verify-lab4-abb798930c-lb-1 Starting
|
||||
Container verify-lab4-abb798930c-lb-1 Started
|
||||
Container verify-lab4-abb798930c-migrate-1 Waiting
|
||||
Container verify-lab4-abb798930c-postgres-1 Waiting
|
||||
Container verify-lab4-abb798930c-s3-init-1 Waiting
|
||||
Container verify-lab4-abb798930c-lb-1 Waiting
|
||||
Container verify-lab4-abb798930c-valkey-1 Waiting
|
||||
Container verify-lab4-abb798930c-api-2 Waiting
|
||||
Container verify-lab4-abb798930c-api-1 Waiting
|
||||
Container verify-lab4-abb798930c-reconciler-1 Waiting
|
||||
Container verify-lab4-abb798930c-s3-1 Waiting
|
||||
Container verify-lab4-abb798930c-valkey-1 Healthy
|
||||
Container verify-lab4-abb798930c-postgres-1 Healthy
|
||||
Container verify-lab4-abb798930c-migrate-1 Exited
|
||||
Container verify-lab4-abb798930c-s3-init-1 Exited
|
||||
Container verify-lab4-abb798930c-reconciler-1 Healthy
|
||||
Container verify-lab4-abb798930c-s3-1 Healthy
|
||||
Container verify-lab4-abb798930c-api-2 Healthy
|
||||
Container verify-lab4-abb798930c-api-1 Healthy
|
||||
Container verify-lab4-abb798930c-lb-1 Healthy
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
python3 tests/smoke.py --base-url "http://localhost:49851"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:49851"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Container verify-lab4-abb798930c-reconciler-1 Stopping
|
||||
Container verify-lab4-abb798930c-lb-1 Stopping
|
||||
Container verify-lab4-abb798930c-reconciler-1 Stopped
|
||||
Container verify-lab4-abb798930c-reconciler-1 Removing
|
||||
Container verify-lab4-abb798930c-reconciler-1 Removed
|
||||
Container verify-lab4-abb798930c-lb-1 Stopped
|
||||
Container verify-lab4-abb798930c-lb-1 Removing
|
||||
Container verify-lab4-abb798930c-lb-1 Removed
|
||||
Container verify-lab4-abb798930c-api-2 Stopping
|
||||
Container verify-lab4-abb798930c-api-1 Stopping
|
||||
Container verify-lab4-abb798930c-api-2 Stopped
|
||||
Container verify-lab4-abb798930c-api-2 Removing
|
||||
Container verify-lab4-abb798930c-api-2 Removed
|
||||
Container verify-lab4-abb798930c-api-1 Stopped
|
||||
Container verify-lab4-abb798930c-api-1 Removing
|
||||
Container verify-lab4-abb798930c-api-1 Removed
|
||||
Container verify-lab4-abb798930c-migrate-1 Stopping
|
||||
Container verify-lab4-abb798930c-s3-init-1 Stopping
|
||||
Container verify-lab4-abb798930c-valkey-1 Stopping
|
||||
Container verify-lab4-abb798930c-s3-init-1 Stopped
|
||||
Container verify-lab4-abb798930c-s3-init-1 Removing
|
||||
Container verify-lab4-abb798930c-migrate-1 Stopped
|
||||
Container verify-lab4-abb798930c-migrate-1 Removing
|
||||
Container verify-lab4-abb798930c-migrate-1 Removed
|
||||
Container verify-lab4-abb798930c-postgres-1 Stopping
|
||||
Container verify-lab4-abb798930c-s3-init-1 Removed
|
||||
Container verify-lab4-abb798930c-s3-1 Stopping
|
||||
Container verify-lab4-abb798930c-postgres-1 Stopped
|
||||
Container verify-lab4-abb798930c-postgres-1 Removing
|
||||
Container verify-lab4-abb798930c-postgres-1 Removed
|
||||
Container verify-lab4-abb798930c-s3-1 Stopped
|
||||
Container verify-lab4-abb798930c-s3-1 Removing
|
||||
Container verify-lab4-abb798930c-valkey-1 Stopped
|
||||
Container verify-lab4-abb798930c-valkey-1 Removing
|
||||
Container verify-lab4-abb798930c-s3-1 Removed
|
||||
Container verify-lab4-abb798930c-valkey-1 Removed
|
||||
Volume verify-lab4-abb798930c_postgres_data Removing
|
||||
Volume verify-lab4-abb798930c_s3_data Removing
|
||||
Network verify-lab4-abb798930c_default Removing
|
||||
Volume verify-lab4-abb798930c_postgres_data Removed
|
||||
Volume verify-lab4-abb798930c_s3_data Removed
|
||||
Network verify-lab4-abb798930c_default Removed
|
||||
PASS: real clean clone, fresh secrets/volumes, smoke, same-session persistence, actual store/API restart, non-root PID1
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"after_pg_insert_fault": {
|
||||
"intents": 1,
|
||||
"metadata_committed": 0
|
||||
},
|
||||
"interrupted_upload_metadata": 0,
|
||||
"private_bucket_anonymous_http_403": true,
|
||||
"s3_outage": {
|
||||
"status": 503,
|
||||
"seconds": 0.7208712079999997
|
||||
},
|
||||
"dry_run": {
|
||||
"dry_run": true,
|
||||
"grace_seconds": 60,
|
||||
"scanned": 11,
|
||||
"orphans": 2,
|
||||
"missing": 2,
|
||||
"stale_intents": 2,
|
||||
"fresh_protected": 2,
|
||||
"referenced_protected": 1,
|
||||
"deleted_objects": 0,
|
||||
"deleted_metadata": 0,
|
||||
"cleared_intents": 0,
|
||||
"duration_ms": 15
|
||||
},
|
||||
"apply": {
|
||||
"dry_run": false,
|
||||
"grace_seconds": 60,
|
||||
"scanned": 11,
|
||||
"orphans": 2,
|
||||
"missing": 2,
|
||||
"stale_intents": 2,
|
||||
"fresh_protected": 2,
|
||||
"referenced_protected": 1,
|
||||
"deleted_objects": 2,
|
||||
"deleted_metadata": 1,
|
||||
"cleared_intents": 2,
|
||||
"duration_ms": 18
|
||||
},
|
||||
"rerun": {
|
||||
"dry_run": false,
|
||||
"grace_seconds": 60,
|
||||
"scanned": 7,
|
||||
"orphans": 0,
|
||||
"missing": 1,
|
||||
"stale_intents": 0,
|
||||
"fresh_protected": 2,
|
||||
"referenced_protected": 1,
|
||||
"deleted_objects": 0,
|
||||
"deleted_metadata": 0,
|
||||
"cleared_intents": 0,
|
||||
"duration_ms": 13
|
||||
},
|
||||
"used_missing_restored_from_original": true,
|
||||
"remaining_fresh_orphan_protected": "originals/8c85afd7-1d51-4d15-9861-98a87ad6afb9"
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
Fault fixtures created; waiting for 60s grace without overriding timestamps
|
||||
PASS: faults, private bucket, dry-run, safe repair, repeated repair and preservation
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8084"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 126, "failed_attempts": 1, "first_recovery_seconds": 0.855}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,57 @@
|
||||
Container mtusi-lab4-kraykin-valkey-1 Running
|
||||
Container mtusi-lab4-kraykin-s3-1 Running
|
||||
Container mtusi-lab4-kraykin-postgres-1 Running
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Created
|
||||
Container mtusi-lab4-kraykin-migrate-1 Recreate
|
||||
Container mtusi-lab4-kraykin-migrate-1 Recreated
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Recreate
|
||||
Container mtusi-lab4-kraykin-api-3 Recreate
|
||||
Container mtusi-lab4-kraykin-api-4 Recreate
|
||||
Container mtusi-lab4-kraykin-api-3 Recreated
|
||||
Container mtusi-lab4-kraykin-api-4 Recreated
|
||||
Container mtusi-lab4-kraykin-lb-1 Running
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Recreated
|
||||
Container mtusi-lab4-kraykin-s3-1 Waiting
|
||||
Container mtusi-lab4-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab4-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab4-kraykin-s3-1 Healthy
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Starting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Starting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Started
|
||||
Container mtusi-lab4-kraykin-migrate-1 Started
|
||||
Container mtusi-lab4-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab4-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Starting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab4-kraykin-api-4 Starting
|
||||
Container mtusi-lab4-kraykin-api-4 Started
|
||||
Container mtusi-lab4-kraykin-api-3 Starting
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Started
|
||||
Container mtusi-lab4-kraykin-api-3 Started
|
||||
Container mtusi-lab4-kraykin-api-4 Waiting
|
||||
Container mtusi-lab4-kraykin-api-3 Waiting
|
||||
Container mtusi-lab4-kraykin-api-4 Healthy
|
||||
Container mtusi-lab4-kraykin-api-3 Healthy
|
||||
Container mtusi-lab4-kraykin-lb-1 Waiting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab4-kraykin-api-4 Waiting
|
||||
Container mtusi-lab4-kraykin-api-3 Waiting
|
||||
Container mtusi-lab4-kraykin-valkey-1 Waiting
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab4-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-1 Waiting
|
||||
Container mtusi-lab4-kraykin-valkey-1 Healthy
|
||||
Container mtusi-lab4-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Healthy
|
||||
Container mtusi-lab4-kraykin-lb-1 Healthy
|
||||
Container mtusi-lab4-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab4-kraykin-s3-1 Healthy
|
||||
Container mtusi-lab4-kraykin-api-4 Healthy
|
||||
Container mtusi-lab4-kraykin-api-3 Healthy
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"method": "same existing dataset plus 0/100/1000 fresh synthetic keys; dry-run; CLI API limits; no metadata mutation; temporary prefix removed",
|
||||
"runs": [
|
||||
{
|
||||
"dry_run": true,
|
||||
"grace_seconds": 3600,
|
||||
"scanned": 19,
|
||||
"orphans": 0,
|
||||
"missing": 0,
|
||||
"stale_intents": 0,
|
||||
"fresh_protected": 19,
|
||||
"referenced_protected": 0,
|
||||
"deleted_objects": 0,
|
||||
"deleted_metadata": 0,
|
||||
"cleared_intents": 0,
|
||||
"duration_ms": 21,
|
||||
"extra_synthetic_objects": 0
|
||||
},
|
||||
{
|
||||
"dry_run": true,
|
||||
"grace_seconds": 3600,
|
||||
"scanned": 119,
|
||||
"orphans": 0,
|
||||
"missing": 0,
|
||||
"stale_intents": 0,
|
||||
"fresh_protected": 119,
|
||||
"referenced_protected": 0,
|
||||
"deleted_objects": 0,
|
||||
"deleted_metadata": 0,
|
||||
"cleared_intents": 0,
|
||||
"duration_ms": 73,
|
||||
"extra_synthetic_objects": 100
|
||||
},
|
||||
{
|
||||
"dry_run": true,
|
||||
"grace_seconds": 3600,
|
||||
"scanned": 1019,
|
||||
"orphans": 0,
|
||||
"missing": 0,
|
||||
"stale_intents": 0,
|
||||
"fresh_protected": 1019,
|
||||
"referenced_protected": 0,
|
||||
"deleted_objects": 0,
|
||||
"deleted_metadata": 0,
|
||||
"cleared_intents": 0,
|
||||
"duration_ms": 498,
|
||||
"extra_synthetic_objects": 1000
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: dry-run costs measured at +0/+100/+1000 keys, probe prefix removed
|
||||
@@ -0,0 +1,3 @@
|
||||
/private/tmp/highload-plot-venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
|
||||
warnings.warn(
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"failures": [
|
||||
{
|
||||
"successful_writes": 281,
|
||||
"errors": 2,
|
||||
"successful_rps": 34.10685167722262,
|
||||
"first_recovery_seconds": 0.685,
|
||||
"action_seconds": 0.222,
|
||||
"instances": [
|
||||
"27a974159396",
|
||||
"760e23615fd3"
|
||||
],
|
||||
"confirmed_preserved": true,
|
||||
"stopped_alias": "760e23615fd3"
|
||||
},
|
||||
{
|
||||
"successful_writes": 258,
|
||||
"errors": 3,
|
||||
"successful_rps": 31.17221074863829,
|
||||
"first_recovery_seconds": 0.723,
|
||||
"action_seconds": 0.206,
|
||||
"instances": [
|
||||
"27a974159396",
|
||||
"760e23615fd3"
|
||||
],
|
||||
"confirmed_preserved": true,
|
||||
"stopped_alias": "27a974159396"
|
||||
}
|
||||
],
|
||||
"rolling": {
|
||||
"successful_writes": 639,
|
||||
"errors": 4,
|
||||
"successful_rps": 33.450115883772696,
|
||||
"first_recovery_seconds": 16.032,
|
||||
"action_seconds": 16.008,
|
||||
"instances": [
|
||||
"27a974159396",
|
||||
"760e23615fd3",
|
||||
"872505db3153",
|
||||
"dd2369f59dcb"
|
||||
],
|
||||
"confirmed_preserved": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
Container mtusi-lab4-kraykin-api-2 Running
|
||||
Container mtusi-lab4-kraykin-api-3 Creating
|
||||
Container mtusi-lab4-kraykin-api-3 Created
|
||||
Container mtusi-lab4-kraykin-api-3 Starting
|
||||
Container mtusi-lab4-kraykin-api-3 Started
|
||||
Container mtusi-lab4-kraykin-api-3 Waiting
|
||||
Container mtusi-lab4-kraykin-api-2 Waiting
|
||||
Container mtusi-lab4-kraykin-api-3 Healthy
|
||||
Container mtusi-lab4-kraykin-api-2 Healthy
|
||||
Container mtusi-lab4-kraykin-api-3 Running
|
||||
Container mtusi-lab4-kraykin-api-4 Creating
|
||||
Container mtusi-lab4-kraykin-api-4 Created
|
||||
Container mtusi-lab4-kraykin-api-4 Starting
|
||||
Container mtusi-lab4-kraykin-api-4 Started
|
||||
Container mtusi-lab4-kraykin-api-4 Waiting
|
||||
Container mtusi-lab4-kraykin-api-3 Waiting
|
||||
Container mtusi-lab4-kraykin-api-4 Healthy
|
||||
Container mtusi-lab4-kraykin-api-3 Healthy
|
||||
PASS: each API stopped and rolling replacement completed; confirmed writes preserved
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"admin_denied": true,
|
||||
"policy_change_denied": true
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: S3 application identity denied admin and bucket-policy changes
|
||||
@@ -0,0 +1,77 @@
|
||||
{
|
||||
"scope": "disposable real clean clone",
|
||||
"before": {
|
||||
"/verify-lab4-abb798930c-api-1": {
|
||||
"service": "api",
|
||||
"started_at": "2026-10-29T18:43:00.717213968Z",
|
||||
"image_id": "sha256:3b2a53cc95740a2ec3a837c5b54a4ee6b1dd3348fd184b5ad8a6c145da0066f3",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab4-abb798930c-api-2": {
|
||||
"service": "api",
|
||||
"started_at": "2026-10-29T18:43:00.656641468Z",
|
||||
"image_id": "sha256:3b2a53cc95740a2ec3a837c5b54a4ee6b1dd3348fd184b5ad8a6c145da0066f3",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab4-abb798930c-postgres-1": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-10-29T18:42:57.938151342Z",
|
||||
"image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab4-abb798930c-s3-1": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-10-29T18:42:57.938943675Z",
|
||||
"image_id": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
|
||||
"user": "",
|
||||
"pid1": "/usr/bin/docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab4-abb798930c-valkey-1": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-10-29T18:42:57.926019258Z",
|
||||
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"/verify-lab4-abb798930c-api-1": {
|
||||
"service": "api",
|
||||
"started_at": "2026-10-29T18:43:13.07148921Z",
|
||||
"image_id": "sha256:3b2a53cc95740a2ec3a837c5b54a4ee6b1dd3348fd184b5ad8a6c145da0066f3",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab4-abb798930c-api-2": {
|
||||
"service": "api",
|
||||
"started_at": "2026-10-29T18:43:13.149502126Z",
|
||||
"image_id": "sha256:3b2a53cc95740a2ec3a837c5b54a4ee6b1dd3348fd184b5ad8a6c145da0066f3",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab4-abb798930c-postgres-1": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-10-29T18:43:11.290679417Z",
|
||||
"image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab4-abb798930c-s3-1": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-10-29T18:43:11.254019375Z",
|
||||
"image_id": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
|
||||
"user": "",
|
||||
"pid1": "/usr/bin/docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab4-abb798930c-valkey-1": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-10-29T18:43:11.314105625Z",
|
||||
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8084"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,161 @@
|
||||
docker compose --profile verify run --build --rm tests
|
||||
Compose can now delegate builds to bake for better performance.
|
||||
To do so, set COMPOSE_BAKE=true.
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [migrate internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 782B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#2 ...
|
||||
|
||||
#3 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#3 DONE 1.1s
|
||||
|
||||
#4 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#4 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#2 DONE 1.1s
|
||||
|
||||
#5 [migrate internal] load .dockerignore
|
||||
#5 transferring context: 196B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate internal] load build context
|
||||
#6 transferring context: 858B done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate stage-2 1/3] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#7 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 done
|
||||
#7 DONE 0.0s
|
||||
|
||||
#8 [migrate build 6/7] COPY internal/ ./internal/
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate build 4/7] RUN go mod download
|
||||
#9 CACHED
|
||||
|
||||
#10 [migrate build 2/7] WORKDIR /src
|
||||
#10 CACHED
|
||||
|
||||
#11 [migrate build 5/7] COPY cmd/ ./cmd/
|
||||
#11 CACHED
|
||||
|
||||
#12 [migrate stage-2 2/3] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#12 CACHED
|
||||
|
||||
#13 [migrate build 3/7] COPY go.mod go.sum ./
|
||||
#13 CACHED
|
||||
|
||||
#14 [migrate build 7/7] RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#14 CACHED
|
||||
|
||||
#15 [migrate stage-2 3/3] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#15 CACHED
|
||||
|
||||
#16 [migrate] exporting to image
|
||||
#16 exporting layers done
|
||||
#16 exporting manifest sha256:9441f50744152286872ca8cceaf0a27c06a3c85573c4fcf057a7d4834e9b1515 done
|
||||
#16 exporting config sha256:60009a73534b7f3d5bd356bb88e566d8cd42100a9c0f8dfbec37f0eaf7c9f840 done
|
||||
#16 exporting attestation manifest sha256:d966b72a6abda792377395e6129bc456095f8d3cf9825da77d8fdce3bb12314e done
|
||||
#16 exporting manifest list sha256:4966ff86d344cf88c53ef4858b0fdfc56f870ca653c3754c44d0d8db911f1e37 done
|
||||
#16 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#16 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#16 DONE 0.0s
|
||||
|
||||
#17 [migrate] resolving provenance for metadata file
|
||||
#17 DONE 0.0s
|
||||
migrate Built
|
||||
Container mtusi-lab4-kraykin-postgres-1 Running
|
||||
Container mtusi-lab4-kraykin-migrate-1 Recreate
|
||||
Container mtusi-lab4-kraykin-migrate-1 Recreated
|
||||
Container mtusi-lab4-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab4-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab4-kraykin-migrate-1 Starting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Started
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [tests internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 782B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [tests internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.3s
|
||||
|
||||
#3 [tests build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#3 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [tests internal] load .dockerignore
|
||||
#4 transferring context: 196B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [tests internal] load build context
|
||||
#5 transferring context: 858B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [tests build 3/7] COPY go.mod go.sum ./
|
||||
#6 CACHED
|
||||
|
||||
#7 [tests build 4/7] RUN go mod download
|
||||
#7 CACHED
|
||||
|
||||
#8 [tests build 5/7] COPY cmd/ ./cmd/
|
||||
#8 CACHED
|
||||
|
||||
#9 [tests build 2/7] WORKDIR /src
|
||||
#9 CACHED
|
||||
|
||||
#10 [tests build 6/7] COPY internal/ ./internal/
|
||||
#10 CACHED
|
||||
|
||||
#11 [tests build 7/7] RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#11 CACHED
|
||||
|
||||
#12 [tests tests 1/1] RUN apk add --no-cache gcc musl-dev
|
||||
#12 0.089 fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz
|
||||
#12 2.184 fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/community/aarch64/APKINDEX.tar.gz
|
||||
#12 4.712 (1/14) Upgrading musl (1.2.5-r10 -> 1.2.5-r13)
|
||||
#12 5.376 (2/14) Installing libgcc (14.2.0-r6)
|
||||
#12 5.600 (3/14) Installing jansson (2.14.1-r0)
|
||||
#12 5.689 (4/14) Installing libstdc++ (14.2.0-r6)
|
||||
#12 6.804 (5/14) Installing zstd-libs (1.5.7-r0)
|
||||
#12 7.153 (6/14) Installing binutils (2.44-r3)
|
||||
#12 11.35 (7/14) Installing libgomp (14.2.0-r6)
|
||||
#12 11.66 (8/14) Installing libatomic (14.2.0-r6)
|
||||
#12 11.75 (9/14) Installing gmp (6.3.0-r3)
|
||||
#12 12.26 (10/14) Installing isl26 (0.26-r1)
|
||||
#12 13.56 (11/14) Installing mpfr4 (4.2.1_p1-r0)
|
||||
#12 14.05 (12/14) Installing mpc1 (1.3.1-r1)
|
||||
#12 14.18 (13/14) Installing gcc (14.2.0-r6)
|
||||
#12 115.6 (14/14) Installing musl-dev (1.2.5-r13)
|
||||
#12 120.9 Executing busybox-1.37.0-r18.trigger
|
||||
#12 121.0 OK: 163 MiB in 30 packages
|
||||
#12 DONE 121.0s
|
||||
|
||||
#13 [tests] exporting to image
|
||||
#13 exporting layers
|
||||
#13 exporting layers 4.4s done
|
||||
#13 exporting manifest sha256:5d9e23a1457ff785391addc78d5bd05a53f323f7f824c64ddb0ee3e6a5b3cd58 done
|
||||
#13 exporting config sha256:f426b704be961867d1d868a7623951dd5c2b9c4c38175c4af103bb30bc841c6f done
|
||||
#13 exporting attestation manifest sha256:8e54edcc6fcc3f61c2d11af782c42eceb41bac0debd0aedf6ecda240f24932f0 0.0s done
|
||||
#13 exporting manifest list sha256:e1892344d0ddf2e325e2d94a0a308a8b345d5e05901fd4f04de91c72b6e86032 0.0s done
|
||||
#13 naming to docker.io/library/mtusi-lab4-kraykin-tests:latest done
|
||||
#13 unpacking to docker.io/library/mtusi-lab4-kraykin-tests:latest
|
||||
#13 unpacking to docker.io/library/mtusi-lab4-kraykin-tests:latest 2.0s done
|
||||
#13 DONE 6.4s
|
||||
|
||||
#14 [tests] resolving provenance for metadata file
|
||||
#14 DONE 0.0s
|
||||
? mtusi/messenger/cmd/server [no test files]
|
||||
? mtusi/messenger/internal/database [no test files]
|
||||
=== RUN TestDatabaseConcurrentHTTP
|
||||
database_test.go:162: 100 confirmed messages from 10 clients through real HTTP and PostgreSQL; race detector enabled
|
||||
--- PASS: TestDatabaseConcurrentHTTP (1.94s)
|
||||
=== RUN TestPasswordFullUnicodeRange
|
||||
--- PASS: TestPasswordFullUnicodeRange (2.33s)
|
||||
PASS
|
||||
ok mtusi/messenger/internal/messenger 5.285s
|
||||
@@ -0,0 +1 @@
|
||||
{"dry_run":false,"grace_seconds":3600,"scanned":19,"orphans":0,"missing":0,"stale_intents":0,"fresh_protected":19,"referenced_protected":0,"deleted_objects":0,"deleted_metadata":0,"cleared_intents":0,"duration_ms":26}
|
||||
@@ -0,0 +1,164 @@
|
||||
{
|
||||
"platform": "macOS-26.6.2-arm64-arm-64bit",
|
||||
"containers": [
|
||||
{
|
||||
"service": "api",
|
||||
"name": "/mtusi-lab4-kraykin-api-3",
|
||||
"alias": "bc92a45b401d",
|
||||
"image": "mtusi-messenger:lab4",
|
||||
"image_id": "sha256:3b2a53cc95740a2ec3a837c5b54a4ee6b1dd3348fd184b5ad8a6c145da0066f3",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/usr/local/bin/messenger",
|
||||
"started_at": "2026-10-29T18:45:00.690863676Z",
|
||||
"memory_limit_bytes": 805306368,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/run/secrets/cache_url",
|
||||
"/run/secrets/cursor_key",
|
||||
"/run/secrets/s3_access_key_id",
|
||||
"/run/secrets/s3_secret_access_key",
|
||||
"/run/secrets/database_url"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "api",
|
||||
"name": "/mtusi-lab4-kraykin-api-4",
|
||||
"alias": "2c790877b903",
|
||||
"image": "mtusi-messenger:lab4",
|
||||
"image_id": "sha256:3b2a53cc95740a2ec3a837c5b54a4ee6b1dd3348fd184b5ad8a6c145da0066f3",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/usr/local/bin/messenger",
|
||||
"started_at": "2026-10-29T18:45:00.625505093Z",
|
||||
"memory_limit_bytes": 805306368,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/run/secrets/database_url",
|
||||
"/run/secrets/cache_url",
|
||||
"/run/secrets/cursor_key",
|
||||
"/run/secrets/s3_access_key_id",
|
||||
"/run/secrets/s3_secret_access_key"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "lb",
|
||||
"name": "/mtusi-lab4-kraykin-lb-1",
|
||||
"alias": "5f33b84e5d4b",
|
||||
"image": "haproxy:3.2.6-alpine",
|
||||
"image_id": "sha256:3a819392f5a6af2a9203d1c63ff0feba74f0c940623a558511be06d7d85e361c",
|
||||
"user": "haproxy",
|
||||
"pid1_path": "docker-entrypoint.sh",
|
||||
"started_at": "2026-10-29T18:18:52.320016589Z",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8084"
|
||||
}
|
||||
]
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/usr/local/etc/haproxy/haproxy.cfg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "postgres",
|
||||
"name": "/mtusi-lab4-kraykin-postgres-1",
|
||||
"alias": "f8d8ea93ff11",
|
||||
"image": "postgres:17.6-alpine",
|
||||
"image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94",
|
||||
"user": "",
|
||||
"pid1_path": "docker-entrypoint.sh",
|
||||
"started_at": "2026-10-29T18:18:44.742051585Z",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus_limit": 2000000000,
|
||||
"published_ports": {
|
||||
"5432/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/var/lib/postgresql/data",
|
||||
"/run/secrets/postgres_password"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "reconciler",
|
||||
"name": "/mtusi-lab4-kraykin-reconciler-1",
|
||||
"alias": "71dec82b97b3",
|
||||
"image": "mtusi-messenger:lab4",
|
||||
"image_id": "sha256:3b2a53cc95740a2ec3a837c5b54a4ee6b1dd3348fd184b5ad8a6c145da0066f3",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/bin/sh",
|
||||
"started_at": "2026-10-29T18:45:00.626424343Z",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus_limit": 250000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/run/secrets/database_url",
|
||||
"/run/secrets/s3_access_key_id",
|
||||
"/config/reconcile-loop.sh",
|
||||
"/run/secrets/s3_secret_access_key"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "s3",
|
||||
"name": "/mtusi-lab4-kraykin-s3-1",
|
||||
"alias": "b65b44d90df0",
|
||||
"image": "quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z",
|
||||
"image_id": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
|
||||
"user": "",
|
||||
"pid1_path": "/usr/bin/docker-entrypoint.sh",
|
||||
"started_at": "2026-10-29T18:19:22.429228797Z",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"9000/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/data",
|
||||
"/run/secrets/s3_root_user",
|
||||
"/run/secrets/s3_root_password"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "valkey",
|
||||
"name": "/mtusi-lab4-kraykin-valkey-1",
|
||||
"alias": "3b389e192453",
|
||||
"image": "valkey/valkey:8.1.3-alpine",
|
||||
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
|
||||
"user": "",
|
||||
"pid1_path": "docker-entrypoint.sh",
|
||||
"started_at": "2026-10-29T18:18:44.817039961Z",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus_limit": 500000000,
|
||||
"published_ports": {
|
||||
"6379/tcp": null
|
||||
},
|
||||
"health": null,
|
||||
"mount_targets": [
|
||||
"/run/secrets/valkey_config",
|
||||
"/data"
|
||||
]
|
||||
}
|
||||
],
|
||||
"postgres_durability": [
|
||||
"fsync=on",
|
||||
"full_page_writes=on",
|
||||
"max_connections=100",
|
||||
"synchronous_commit=on"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://localhost:8084"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://localhost:8084"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,217 @@
|
||||
python3 scripts/bootstrap.py
|
||||
Local secrets ready (values hidden); .env and secrets/ are ignored by Git.
|
||||
docker compose --env-file .env up -d --build --wait
|
||||
Compose can now delegate builds to bake for better performance.
|
||||
To do so, set COMPOSE_BAKE=true.
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [migrate internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 782B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.3s
|
||||
|
||||
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 0.3s
|
||||
|
||||
#4 [migrate internal] load .dockerignore
|
||||
#4 transferring context: 196B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [migrate stage-2 1/3] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate internal] load build context
|
||||
#7 transferring context: 5.45kB done
|
||||
#7 DONE 0.0s
|
||||
|
||||
#8 [migrate build 3/7] COPY go.mod go.sum ./
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate build 4/7] RUN go mod download
|
||||
#9 CACHED
|
||||
|
||||
#10 [migrate build 2/7] WORKDIR /src
|
||||
#10 CACHED
|
||||
|
||||
#11 [migrate build 5/7] COPY cmd/ ./cmd/
|
||||
#11 CACHED
|
||||
|
||||
#12 [migrate build 6/7] COPY internal/ ./internal/
|
||||
#12 DONE 0.0s
|
||||
|
||||
#13 [migrate build 7/7] RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#13 DONE 8.1s
|
||||
|
||||
#14 [migrate stage-2 2/3] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#14 CACHED
|
||||
|
||||
#15 [migrate stage-2 3/3] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#15 CACHED
|
||||
|
||||
#16 [migrate] exporting to image
|
||||
#16 exporting layers done
|
||||
#16 exporting manifest sha256:9441f50744152286872ca8cceaf0a27c06a3c85573c4fcf057a7d4834e9b1515 done
|
||||
#16 exporting config sha256:60009a73534b7f3d5bd356bb88e566d8cd42100a9c0f8dfbec37f0eaf7c9f840 done
|
||||
#16 exporting attestation manifest sha256:76c6ee04c86aab36a2867a17fcfc2b699b60fea67bc5a95af81f53804a9784b3 done
|
||||
#16 exporting manifest list sha256:b0eac6a3dc5162e762256cebf10b2f441e8770d7b5982ab0e62f7026bb89462e done
|
||||
#16 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#16 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#16 DONE 0.0s
|
||||
|
||||
#17 [migrate] resolving provenance for metadata file
|
||||
#17 DONE 0.0s
|
||||
|
||||
#18 [api internal] load build definition from Dockerfile
|
||||
#18 transferring dockerfile: 782B done
|
||||
#18 DONE 0.0s
|
||||
|
||||
#19 [reconciler internal] load build definition from Dockerfile
|
||||
#19 transferring dockerfile: 782B done
|
||||
#19 DONE 0.0s
|
||||
|
||||
#3 [reconciler internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 0.6s
|
||||
|
||||
#2 [reconciler internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.6s
|
||||
|
||||
#20 [api internal] load .dockerignore
|
||||
#20 transferring context: 196B done
|
||||
#20 DONE 0.0s
|
||||
|
||||
#21 [reconciler internal] load .dockerignore
|
||||
#21 transferring context: 196B done
|
||||
#21 DONE 0.0s
|
||||
|
||||
#5 [api stage-2 1/3] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [reconciler build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#22 [reconciler internal] load build context
|
||||
#22 transferring context: 858B done
|
||||
#22 DONE 0.0s
|
||||
|
||||
#23 [api internal] load build context
|
||||
#23 transferring context: 5.45kB done
|
||||
#23 DONE 0.0s
|
||||
|
||||
#24 [reconciler build 5/7] COPY cmd/ ./cmd/
|
||||
#24 CACHED
|
||||
|
||||
#25 [reconciler build 3/7] COPY go.mod go.sum ./
|
||||
#25 CACHED
|
||||
|
||||
#26 [reconciler build 7/7] RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#26 CACHED
|
||||
|
||||
#27 [reconciler build 4/7] RUN go mod download
|
||||
#27 CACHED
|
||||
|
||||
#14 [reconciler stage-2 2/3] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#14 CACHED
|
||||
|
||||
#10 [reconciler build 2/7] WORKDIR /src
|
||||
#10 CACHED
|
||||
|
||||
#28 [reconciler build 6/7] COPY internal/ ./internal/
|
||||
#28 CACHED
|
||||
|
||||
#29 [reconciler stage-2 3/3] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#29 CACHED
|
||||
|
||||
#30 [reconciler] exporting to image
|
||||
#30 exporting layers done
|
||||
#30 exporting manifest sha256:ec1c8251ab9399b52f7d9541c89bbafbf6060d1cb5e3478744091ef7fdf7384e done
|
||||
#30 exporting config sha256:c3c0eed0d886cba165d1bfa0500f22a1af0d8af52e6f48f30c9055155998e9cd done
|
||||
#30 exporting attestation manifest sha256:49bbd4e0347549aafe8a9007ec174051789e3bbccb182338db36a57d505f6d22 0.0s done
|
||||
#30 exporting manifest list sha256:b3556b43f86c97ab3e226771680299b3c6e7ce36335e5a4ed135cd89afcc3590 done
|
||||
#30 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#30 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#30 DONE 0.0s
|
||||
|
||||
#31 [api] exporting to image
|
||||
#31 exporting layers done
|
||||
#31 exporting manifest sha256:97674ea7eb530a98dac589b5b4a1cce37c4297ab16b79a734254f4e69a7d62b7 done
|
||||
#31 exporting config sha256:54975c5a5b51c1520cd3e633f1fae90adeabddbc39fe33c44620e0ecd753a838 done
|
||||
#31 exporting attestation manifest sha256:86f66017f2dddcab957737ac7e7e9b44c2b396ebe520a23d9a75971dff92ff09 0.0s done
|
||||
#31 exporting manifest list sha256:74515ae47b7868f01cc66402b83634e784fcd53df1775b0633776a126bae2446 done
|
||||
#31 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#31 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#31 DONE 0.1s
|
||||
|
||||
#32 [reconciler] resolving provenance for metadata file
|
||||
#32 DONE 0.0s
|
||||
|
||||
#33 [api] resolving provenance for metadata file
|
||||
#33 DONE 0.0s
|
||||
api Built
|
||||
migrate Built
|
||||
reconciler Built
|
||||
Container mtusi-lab4-kraykin-valkey-1 Running
|
||||
Container mtusi-lab4-kraykin-postgres-1 Running
|
||||
Container mtusi-lab4-kraykin-s3-1 Running
|
||||
Container mtusi-lab4-kraykin-migrate-1 Recreate
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Created
|
||||
Container mtusi-lab4-kraykin-migrate-1 Recreated
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Recreate
|
||||
Container mtusi-lab4-kraykin-api-1 Recreate
|
||||
Container mtusi-lab4-kraykin-api-2 Recreate
|
||||
Container mtusi-lab4-kraykin-api-2 Recreated
|
||||
Container mtusi-lab4-kraykin-api-1 Recreated
|
||||
Container mtusi-lab4-kraykin-lb-1 Running
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Recreated
|
||||
Container mtusi-lab4-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-1 Waiting
|
||||
Container mtusi-lab4-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab4-kraykin-migrate-1 Starting
|
||||
Container mtusi-lab4-kraykin-s3-1 Healthy
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Starting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Started
|
||||
Container mtusi-lab4-kraykin-migrate-1 Started
|
||||
Container mtusi-lab4-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab4-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab4-kraykin-api-2 Starting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Starting
|
||||
Container mtusi-lab4-kraykin-api-2 Started
|
||||
Container mtusi-lab4-kraykin-api-1 Starting
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Started
|
||||
Container mtusi-lab4-kraykin-api-1 Started
|
||||
Container mtusi-lab4-kraykin-api-2 Waiting
|
||||
Container mtusi-lab4-kraykin-api-1 Waiting
|
||||
Container mtusi-lab4-kraykin-api-2 Healthy
|
||||
Container mtusi-lab4-kraykin-api-1 Healthy
|
||||
Container mtusi-lab4-kraykin-api-2 Waiting
|
||||
Container mtusi-lab4-kraykin-api-1 Waiting
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Waiting
|
||||
Container mtusi-lab4-kraykin-valkey-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab4-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-1 Waiting
|
||||
Container mtusi-lab4-kraykin-lb-1 Waiting
|
||||
Container mtusi-lab4-kraykin-lb-1 Healthy
|
||||
Container mtusi-lab4-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab4-kraykin-valkey-1 Healthy
|
||||
Container mtusi-lab4-kraykin-s3-1 Healthy
|
||||
Container mtusi-lab4-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab4-kraykin-reconciler-1 Healthy
|
||||
Container mtusi-lab4-kraykin-api-2 Healthy
|
||||
Container mtusi-lab4-kraykin-api-1 Healthy
|
||||
@@ -0,0 +1,201 @@
|
||||
python3 scripts/bootstrap.py
|
||||
Local secrets ready (values hidden); .env and secrets/ are ignored by Git.
|
||||
docker compose --env-file .env up -d --build --wait
|
||||
Compose can now delegate builds to bake for better performance.
|
||||
To do so, set COMPOSE_BAKE=true.
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [migrate internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 782B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 1.0s
|
||||
|
||||
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 2.8s
|
||||
|
||||
#4 [migrate internal] load .dockerignore
|
||||
#4 transferring context: 196B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [migrate stage-2 1/3] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate internal] load build context
|
||||
#7 transferring context: 58.80kB done
|
||||
#7 DONE 0.0s
|
||||
|
||||
#8 [migrate build 2/7] WORKDIR /src
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate build 3/7] COPY go.mod go.sum ./
|
||||
#9 DONE 0.0s
|
||||
|
||||
#10 [migrate build 4/7] RUN go mod download
|
||||
#10 DONE 90.3s
|
||||
|
||||
#11 [migrate build 5/7] COPY cmd/ ./cmd/
|
||||
#11 DONE 0.0s
|
||||
|
||||
#12 [migrate build 6/7] COPY internal/ ./internal/
|
||||
#12 DONE 0.0s
|
||||
|
||||
#13 [migrate build 7/7] RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#13 DONE 7.0s
|
||||
|
||||
#14 [migrate stage-2 2/3] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#14 CACHED
|
||||
|
||||
#15 [migrate stage-2 3/3] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#15 DONE 0.0s
|
||||
|
||||
#16 [migrate] exporting to image
|
||||
#16 exporting layers
|
||||
#16 exporting layers 0.2s done
|
||||
#16 exporting manifest sha256:6ce80109faba977d714b5da035f1931095bb47d92fffdab4ef825045a0be6730 done
|
||||
#16 exporting config sha256:104d808cb89e9f12da0a4c2da903632b70915950850d32b2a7afe4cba6276984 done
|
||||
#16 exporting attestation manifest sha256:b2313dd3ddcf63146300a0f3a48916ff46d6bd44e6f5c9109ba5e94d85a0abec done
|
||||
#16 exporting manifest list sha256:7a8da84af468b11d13c460e215778949438bcc0a87288350307c24ee4e403448 done
|
||||
#16 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#16 unpacking to docker.io/library/mtusi-messenger:lab4 0.0s done
|
||||
#16 DONE 0.3s
|
||||
|
||||
#17 [migrate] resolving provenance for metadata file
|
||||
#17 DONE 0.0s
|
||||
|
||||
#18 [api internal] load build definition from Dockerfile
|
||||
#18 transferring dockerfile: 782B done
|
||||
#18 DONE 0.0s
|
||||
|
||||
#2 [api internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 3.0s
|
||||
|
||||
#3 [api internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 5.3s
|
||||
|
||||
#19 [api internal] load .dockerignore
|
||||
#19 transferring context: 196B done
|
||||
#19 DONE 0.0s
|
||||
|
||||
#6 [api build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#5 [api stage-2 1/3] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#20 [api internal] load build context
|
||||
#20 transferring context: 858B done
|
||||
#20 DONE 0.0s
|
||||
|
||||
#8 [api build 2/7] WORKDIR /src
|
||||
#8 CACHED
|
||||
|
||||
#21 [api build 6/7] COPY internal/ ./internal/
|
||||
#21 CACHED
|
||||
|
||||
#22 [api build 4/7] RUN go mod download
|
||||
#22 CACHED
|
||||
|
||||
#23 [api build 5/7] COPY cmd/ ./cmd/
|
||||
#23 CACHED
|
||||
|
||||
#24 [api build 7/7] RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#24 CACHED
|
||||
|
||||
#25 [api build 3/7] COPY go.mod go.sum ./
|
||||
#25 CACHED
|
||||
|
||||
#14 [api stage-2 2/3] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#14 CACHED
|
||||
|
||||
#26 [api stage-2 3/3] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#26 CACHED
|
||||
|
||||
#27 [api] exporting to image
|
||||
#27 exporting layers done
|
||||
#27 exporting manifest sha256:5a9bcfc0f45ce663eec7973d20ae788d1fd9c95c85bcd98af4ebb604aad1fced done
|
||||
#27 exporting config sha256:436613bb794e0473f4425fad59bdea2567dd4e6b180fb8d4148509d7a1c294d8 done
|
||||
#27 exporting attestation manifest sha256:8f17ac63b7f3db7c99283019ff763711f7e4da5e9f466094d1accf597000c355 done
|
||||
#27 exporting manifest list sha256:eba13534488fb4d7501efbe70117b5f64dbb25fd7d7a5b451b724a08c65993c4 done
|
||||
#27 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#27 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#27 DONE 0.0s
|
||||
|
||||
#28 [api] resolving provenance for metadata file
|
||||
#28 DONE 0.0s
|
||||
api Built
|
||||
migrate Built
|
||||
Network mtusi-lab4-kraykin_default Creating
|
||||
Network mtusi-lab4-kraykin_default Created
|
||||
Volume "mtusi-lab4-kraykin_postgres_data" Creating
|
||||
Volume "mtusi-lab4-kraykin_postgres_data" Created
|
||||
Volume "mtusi-lab4-kraykin_s3_data" Creating
|
||||
Volume "mtusi-lab4-kraykin_s3_data" Created
|
||||
Container mtusi-lab4-kraykin-valkey-1 Creating
|
||||
Container mtusi-lab4-kraykin-postgres-1 Creating
|
||||
Container mtusi-lab4-kraykin-s3-1 Creating
|
||||
Container mtusi-lab4-kraykin-valkey-1 Created
|
||||
Container mtusi-lab4-kraykin-s3-1 Created
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Creating
|
||||
Container mtusi-lab4-kraykin-postgres-1 Created
|
||||
Container mtusi-lab4-kraykin-migrate-1 Creating
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Created
|
||||
Container mtusi-lab4-kraykin-migrate-1 Created
|
||||
Container mtusi-lab4-kraykin-api-2 Creating
|
||||
Container mtusi-lab4-kraykin-api-1 Creating
|
||||
Container mtusi-lab4-kraykin-api-1 Created
|
||||
Container mtusi-lab4-kraykin-api-2 Created
|
||||
Container mtusi-lab4-kraykin-lb-1 Creating
|
||||
Container mtusi-lab4-kraykin-lb-1 Created
|
||||
Container mtusi-lab4-kraykin-postgres-1 Starting
|
||||
Container mtusi-lab4-kraykin-s3-1 Starting
|
||||
Container mtusi-lab4-kraykin-valkey-1 Starting
|
||||
Container mtusi-lab4-kraykin-postgres-1 Started
|
||||
Container mtusi-lab4-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-1 Started
|
||||
Container mtusi-lab4-kraykin-s3-1 Waiting
|
||||
Container mtusi-lab4-kraykin-valkey-1 Started
|
||||
Container mtusi-lab4-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab4-kraykin-migrate-1 Starting
|
||||
Container mtusi-lab4-kraykin-s3-1 Healthy
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Starting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Started
|
||||
Container mtusi-lab4-kraykin-migrate-1 Started
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab4-kraykin-api-2 Starting
|
||||
Container mtusi-lab4-kraykin-api-2 Started
|
||||
Container mtusi-lab4-kraykin-api-1 Starting
|
||||
Container mtusi-lab4-kraykin-api-1 Started
|
||||
Container mtusi-lab4-kraykin-api-2 Waiting
|
||||
Container mtusi-lab4-kraykin-api-1 Waiting
|
||||
Container mtusi-lab4-kraykin-api-2 Healthy
|
||||
Container mtusi-lab4-kraykin-api-1 Healthy
|
||||
Container mtusi-lab4-kraykin-lb-1 Starting
|
||||
Container mtusi-lab4-kraykin-lb-1 Started
|
||||
Container mtusi-lab4-kraykin-s3-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab4-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab4-kraykin-api-2 Waiting
|
||||
Container mtusi-lab4-kraykin-api-1 Waiting
|
||||
Container mtusi-lab4-kraykin-lb-1 Waiting
|
||||
Container mtusi-lab4-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab4-kraykin-valkey-1 Waiting
|
||||
Container mtusi-lab4-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab4-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab4-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab4-kraykin-valkey-1 Healthy
|
||||
Container mtusi-lab4-kraykin-s3-1 Healthy
|
||||
Container mtusi-lab4-kraykin-api-2 Healthy
|
||||
Container mtusi-lab4-kraykin-api-1 Healthy
|
||||
Container mtusi-lab4-kraykin-lb-1 Healthy
|
||||
@@ -0,0 +1,3 @@
|
||||
? mtusi/messenger/cmd/server [no test files]
|
||||
? mtusi/messenger/internal/database [no test files]
|
||||
ok mtusi/messenger/internal/messenger 3.202s
|
||||
@@ -0,0 +1,33 @@
|
||||
module mtusi/messenger
|
||||
|
||||
go 1.24.0
|
||||
|
||||
require (
|
||||
github.com/jackc/pgx/v5 v5.7.6
|
||||
github.com/minio/minio-go/v7 v7.0.95
|
||||
github.com/redis/go-redis/v9 v9.12.1
|
||||
golang.org/x/crypto v0.41.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/go-ini/ini v1.67.0 // indirect
|
||||
github.com/goccy/go-json v0.10.5 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/klauspost/compress v1.18.0 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
|
||||
github.com/minio/crc64nvme v1.0.2 // indirect
|
||||
github.com/minio/md5-simd v1.1.2 // indirect
|
||||
github.com/philhofer/fwd v1.2.0 // indirect
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
github.com/tinylib/msgp v1.3.0 // indirect
|
||||
golang.org/x/net v0.42.0 // indirect
|
||||
golang.org/x/sync v0.16.0 // indirect
|
||||
golang.org/x/sys v0.35.0 // indirect
|
||||
golang.org/x/text v0.28.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,67 @@
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
|
||||
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
|
||||
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
|
||||
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.7.6 h1:rWQc5FwZSPX58r1OQmkuaNicxdmExaEz5A2DO2hUuTk=
|
||||
github.com/jackc/pgx/v5 v5.7.6/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
|
||||
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
|
||||
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/minio/crc64nvme v1.0.2 h1:6uO1UxGAD+kwqWWp7mBFsi5gAse66C4NXO8cmcVculg=
|
||||
github.com/minio/crc64nvme v1.0.2/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
|
||||
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
|
||||
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
|
||||
github.com/minio/minio-go/v7 v7.0.95 h1:ywOUPg+PebTMTzn9VDsoFJy32ZuARN9zhB+K3IYEvYU=
|
||||
github.com/minio/minio-go/v7 v7.0.95/go.mod h1:wOOX3uxS334vImCNRVyIDdXX9OsXDm89ToynKgqUKlo=
|
||||
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
|
||||
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/redis/go-redis/v9 v9.12.1 h1:k5iquqv27aBtnTm2tIkROUDp8JBXhXZIVu1InSgvovg=
|
||||
github.com/redis/go-redis/v9 v9.12.1/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw=
|
||||
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
|
||||
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||
github.com/tinylib/msgp v1.3.0 h1:ULuf7GPooDaIlbyvgAxBV/FI7ynli6LZ1/nVUNu+0ww=
|
||||
github.com/tinylib/msgp v1.3.0/go.mod h1:ykjzy2wzgrlvpDCRc4LA8UXy6D8bzMSuAF3WD57Gok0=
|
||||
golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
|
||||
golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
|
||||
golang.org/x/net v0.42.0 h1:jzkYrhi3YQWD6MLBJcsklgQsoAcw89EcZbJw8Z614hs=
|
||||
golang.org/x/net v0.42.0/go.mod h1:FF1RA5d3u7nAYA4z2TkclSCKh68eSXtiFwcWQpPXdt8=
|
||||
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
|
||||
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
|
||||
golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
|
||||
golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
|
||||
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,27 @@
|
||||
global
|
||||
maxconn 512
|
||||
log stdout format raw local0
|
||||
defaults
|
||||
log global
|
||||
mode http
|
||||
timeout connect 500ms
|
||||
timeout client 35s
|
||||
timeout server 35s
|
||||
timeout check 800ms
|
||||
retries 0
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
resolve_retries 2
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
hold obsolete 1s
|
||||
frontend public
|
||||
bind :8080
|
||||
default_backend replicas
|
||||
backend replicas
|
||||
balance roundrobin
|
||||
option httpchk GET /health/ready
|
||||
http-check expect status 200
|
||||
default-server inter 1s fall 2 rise 1
|
||||
server-template api 4 api:8080 check resolvers docker init-addr libc,none resolve-prefer ipv4
|
||||
@@ -0,0 +1,9 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
trap 'exit 0' TERM INT
|
||||
while :; do
|
||||
# A dependency outage fails this pass without permitting destructive repair.
|
||||
messenger reconcile --apply || printf '%s\n' 'Reconcile pass failed; retry next minute' >&2
|
||||
sleep 60 &
|
||||
wait $! || true
|
||||
done
|
||||
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
# Do not enable shell tracing: mc arguments contain local credentials.
|
||||
mc alias set local http://s3:9000 "$(cat /run/secrets/s3_root_user)" "$(cat /run/secrets/s3_root_password)" >/dev/null
|
||||
mc mb --ignore-existing local/messenger-attachments >/dev/null
|
||||
mc anonymous set none local/messenger-attachments >/dev/null
|
||||
mc admin policy create local messenger-app /config/s3-policy.json >/dev/null
|
||||
mc admin user add local "$(cat /run/secrets/s3_access_key_id)" "$(cat /run/secrets/s3_secret_access_key)" >/dev/null
|
||||
mc admin policy attach local messenger-app --user "$(cat /run/secrets/s3_access_key_id)" >/dev/null
|
||||
printf '%s\n' 'Private bucket and restricted application user ready'
|
||||
@@ -0,0 +1 @@
|
||||
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:ListBucket","s3:GetBucketLocation"],"Resource":["arn:aws:s3:::messenger-attachments"]},{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::messenger-attachments/originals/*"]}]}
|
||||
@@ -0,0 +1,103 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"embed"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
//go:embed migrations/*.sql
|
||||
var migrations embed.FS
|
||||
|
||||
func Open(ctx context.Context, url string) (*pgxpool.Pool, error) {
|
||||
cfg, err := pgxpool.ParseConfig(url)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid database configuration")
|
||||
}
|
||||
cfg.MaxConns = 10
|
||||
if v := os.Getenv("DB_POOL_MAX"); v != "" {
|
||||
n, e := strconv.Atoi(v)
|
||||
if e != nil || n < 1 || n > 64 {
|
||||
return nil, fmt.Errorf("DB_POOL_MAX must be 1..64")
|
||||
}
|
||||
cfg.MaxConns = int32(n)
|
||||
}
|
||||
cfg.MinConns = 0
|
||||
cfg.MaxConnLifetime = 30 * time.Minute
|
||||
cfg.MaxConnIdleTime = 5 * time.Minute
|
||||
cfg.ConnConfig.ConnectTimeout = 2 * time.Second
|
||||
cfg.ConnConfig.RuntimeParams["statement_timeout"] = "2000"
|
||||
cfg.ConnConfig.RuntimeParams["lock_timeout"] = "1000"
|
||||
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot create database pool")
|
||||
}
|
||||
if err = pool.Ping(ctx); err != nil {
|
||||
pool.Close()
|
||||
return nil, fmt.Errorf("database unavailable")
|
||||
}
|
||||
return pool, nil
|
||||
}
|
||||
|
||||
// All migrations and the version ledger commit together. An advisory lock also
|
||||
// serializes bootstrap across API instances; edited applied migrations fail.
|
||||
func Migrate(ctx context.Context, pool *pgxpool.Pool) error {
|
||||
tx, err := pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
if _, err = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(2026002)"); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = tx.Exec(ctx, `CREATE TABLE IF NOT EXISTS schema_migrations(version text PRIMARY KEY, checksum text NOT NULL, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())`); err != nil {
|
||||
return err
|
||||
}
|
||||
entries, err := migrations.ReadDir("migrations")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sort.Slice(entries, func(i, j int) bool { return entries[i].Name() < entries[j].Name() })
|
||||
for _, entry := range entries {
|
||||
body, err := migrations.ReadFile("migrations/" + entry.Name())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
checksum := fmt.Sprintf("%x", sha256.Sum256(body))
|
||||
var stored string
|
||||
rows, err := tx.Query(ctx, "SELECT checksum FROM schema_migrations WHERE version=$1", entry.Name())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
found := rows.Next()
|
||||
if found {
|
||||
err = rows.Scan(&stored)
|
||||
}
|
||||
if err == nil {
|
||||
err = rows.Err()
|
||||
}
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if found {
|
||||
if stored != checksum {
|
||||
return fmt.Errorf("applied migration changed: %s", entry.Name())
|
||||
}
|
||||
continue
|
||||
}
|
||||
if _, err = tx.Exec(ctx, string(body)); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = tx.Exec(ctx, "INSERT INTO schema_migrations(version,checksum) VALUES($1,$2)", entry.Name(), checksum); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
last_message_at timestamptz NOT NULL DEFAULT '-infinity'
|
||||
);
|
||||
CREATE TABLE chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX chat_members_user_idx ON chat_members(user_id, chat_id);
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
sender_id uuid NOT NULL,
|
||||
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
|
||||
created_at timestamptz NOT NULL,
|
||||
FOREIGN KEY (chat_id, sender_id) REFERENCES chat_members(chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX messages_history_idx ON messages(chat_id, created_at, id);
|
||||
CREATE INDEX chats_order_idx ON chats(created_at, id);
|
||||
CREATE TABLE sessions (
|
||||
token_hash bytea PRIMARY KEY CHECK (octet_length(token_hash) = 32),
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
expires_at timestamptz NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX sessions_expiry_idx ON sessions(expires_at);
|
||||
@@ -0,0 +1,10 @@
|
||||
CREATE TABLE message_requests (
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
key text NOT NULL,
|
||||
body_hash bytea NOT NULL,
|
||||
response jsonb NOT NULL,
|
||||
expires_at timestamptz NOT NULL,
|
||||
PRIMARY KEY(user_id,chat_id,key)
|
||||
);
|
||||
CREATE INDEX message_requests_expiry_idx ON message_requests(expires_at);
|
||||
@@ -0,0 +1,29 @@
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ADD CHECK(char_length(text) BETWEEN 0 AND 4000);
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
owner_id uuid NOT NULL REFERENCES users(id),
|
||||
filename text NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
content_type text NOT NULL CHECK(content_type IN ('image/png','image/jpeg')),
|
||||
size_bytes bigint NOT NULL CHECK(size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 text NOT NULL CHECK(sha256 ~ '^[a-f0-9]{64}$'),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages(id),
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
position integer NOT NULL CHECK(position BETWEEN 0 AND 9),
|
||||
PRIMARY KEY(message_id,attachment_id),
|
||||
UNIQUE(message_id,position)
|
||||
);
|
||||
-- Durable upload journal is committed BEFORE PUT, so incomplete work remains
|
||||
-- attributable after process termination. Final metadata + journal deletion
|
||||
-- share one PG transaction, under the same per-key lock as the reconciler.
|
||||
CREATE TABLE upload_intents (
|
||||
object_key text PRIMARY KEY,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX attachments_chat_idx ON attachments(chat_id);
|
||||
@@ -0,0 +1,273 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"image"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
"io"
|
||||
"mime"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
const maxUpload = 10 << 20
|
||||
const maxPixels = 40000000
|
||||
|
||||
type ObjectStore struct {
|
||||
Client *minio.Client
|
||||
Bucket string
|
||||
}
|
||||
|
||||
func NewObjectStore(endpoint, bucket, access, secret string) (*ObjectStore, error) {
|
||||
u, e := url.Parse(endpoint)
|
||||
if e != nil || u.Host == "" || (u.Scheme != "http" && u.Scheme != "https") {
|
||||
return nil, fmt.Errorf("invalid S3 endpoint")
|
||||
}
|
||||
transport := http.DefaultTransport.(*http.Transport).Clone()
|
||||
transport.MaxConnsPerHost = 8
|
||||
transport.ResponseHeaderTimeout = 5 * time.Second
|
||||
client, e := minio.New(u.Host, &minio.Options{Creds: credentials.NewStaticV4(access, secret, ""), Secure: u.Scheme == "https", Region: "us-east-1", Transport: transport, MaxRetries: 1})
|
||||
if e != nil {
|
||||
return nil, fmt.Errorf("invalid S3 configuration")
|
||||
}
|
||||
return &ObjectStore{client, bucket}, nil
|
||||
}
|
||||
func (o *ObjectStore) Ready(ctx context.Context) bool {
|
||||
ok, e := o.Client.BucketExists(ctx, o.Bucket)
|
||||
return e == nil && ok
|
||||
}
|
||||
func (s *Server) SetObjectStore(o *ObjectStore) { s.objects = o; s.uploads = make(chan struct{}, 1) }
|
||||
|
||||
type Attachment struct {
|
||||
ID string `json:"id"`
|
||||
ChatID string `json:"chat_id"`
|
||||
OwnerID string `json:"owner_id"`
|
||||
Filename string `json:"filename"`
|
||||
ContentType string `json:"content_type"`
|
||||
Size int64 `json:"size_bytes"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Status string `json:"status"`
|
||||
Variants []any `json:"variants"`
|
||||
ErrorCode *string `json:"error_code"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
}
|
||||
|
||||
const attachmentColumns = `a.id::text,a.chat_id::text,a.owner_id::text,a.filename,a.content_type,a.size_bytes,a.sha256,a.created_at,a.updated_at,a.object_key`
|
||||
|
||||
func scanAttachment(row scanner) (Attachment, string, error) {
|
||||
a := Attachment{Status: "ready", Variants: []any{}}
|
||||
var c, u time.Time
|
||||
var key string
|
||||
e := row.Scan(&a.ID, &a.ChatID, &a.OwnerID, &a.Filename, &a.ContentType, &a.Size, &a.SHA256, &c, &u, &key)
|
||||
a.CreatedAt = stamp(c)
|
||||
a.UpdatedAt = stamp(u)
|
||||
return a, key, e
|
||||
}
|
||||
func validFilename(name string) bool {
|
||||
if !validText(name, 255) {
|
||||
return false
|
||||
}
|
||||
for _, c := range name {
|
||||
if c == '/' || c == '\\' || unicode.IsControl(c) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// A temporary capped spool avoids holding a compressed file in RAM; it is
|
||||
// removed on every exit. DecodeConfig bounds pixels before full validation.
|
||||
func (s *Server) uploadAttachment(w http.ResponseWriter, r *http.Request, uid, cid string) {
|
||||
q, e := url.ParseQuery(r.URL.RawQuery)
|
||||
if e != nil || len(q["filename"]) != 1 || !validFilename(q.Get("filename")) {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
contentType, _, e := mime.ParseMediaType(r.Header.Get("Content-Type"))
|
||||
if e != nil || (contentType != "image/png" && contentType != "image/jpeg") {
|
||||
fail(w, 415, "unsupported_media_type", "Expected PNG or JPEG")
|
||||
return
|
||||
}
|
||||
if r.ContentLength > maxUpload {
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(r.Body, maxUpload+1))
|
||||
fail(w, 413, "payload_too_large", "Maximum upload is 10 MiB")
|
||||
return
|
||||
}
|
||||
select {
|
||||
case s.uploads <- struct{}{}:
|
||||
defer func() { <-s.uploads }()
|
||||
case <-r.Context().Done():
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
f, e := os.CreateTemp("", "messenger-upload-*")
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer os.Remove(f.Name())
|
||||
defer f.Close()
|
||||
hash := sha256.New()
|
||||
n, e := io.Copy(io.MultiWriter(f, hash), http.MaxBytesReader(w, r.Body, maxUpload))
|
||||
if e != nil {
|
||||
var over *http.MaxBytesError
|
||||
if errors.As(e, &over) {
|
||||
fail(w, 413, "payload_too_large", "Maximum upload is 10 MiB")
|
||||
} else {
|
||||
bad(w)
|
||||
}
|
||||
return
|
||||
}
|
||||
_, _ = f.Seek(0, io.SeekStart)
|
||||
config, format, e := image.DecodeConfig(f)
|
||||
if e != nil || "image/"+format != contentType {
|
||||
fail(w, 415, "unsupported_media_type", "Invalid image or MIME mismatch")
|
||||
return
|
||||
}
|
||||
if config.Width < 1 || config.Height < 1 || int64(config.Width)*int64(config.Height) > maxPixels {
|
||||
fail(w, 413, "payload_too_large", "Maximum image is 40 million pixels")
|
||||
return
|
||||
}
|
||||
_, _ = f.Seek(0, io.SeekStart)
|
||||
if !validateImage(f) {
|
||||
fail(w, 415, "unsupported_media_type", "Invalid image")
|
||||
return
|
||||
}
|
||||
_, _ = f.Seek(0, io.SeekStart)
|
||||
id := uuid()
|
||||
key := "originals/" + id
|
||||
ctx := r.Context()
|
||||
// Journal survives a crash between PUT and metadata COMMIT.
|
||||
if _, e = s.db.Exec(ctx, "INSERT INTO upload_intents(object_key) VALUES($1)", key); e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
tx, e := s.db.Begin(ctx)
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
if _, e = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,4))", key); e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
// Disable multipart: the bounded spool is <=10MiB, hence one atomic object PUT.
|
||||
_, e = s.objects.Client.PutObject(ctx, s.objects.Bucket, key, f, n, minio.PutObjectOptions{ContentType: contentType, DisableMultipart: true})
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
a, _, e := scanAttachment(tx.QueryRow(ctx, `INSERT INTO attachments AS a(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256) VALUES($1,$2,$3,$4,$5,$6,$7,$8) RETURNING `+attachmentColumns, id, cid, uid, q.Get("filename"), key, contentType, n, hex.EncodeToString(hash.Sum(nil))))
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if _, e = tx.Exec(ctx, "DELETE FROM upload_intents WHERE object_key=$1", key); e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if tx.Commit(ctx) != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
reply(w, 201, a)
|
||||
}
|
||||
func validateImage(r io.Reader) bool { _, _, e := image.Decode(r); return e == nil }
|
||||
|
||||
func (s *Server) serveAttachment(w http.ResponseWriter, r *http.Request, uid string, parts []string) {
|
||||
if r.Method != "GET" || (len(parts) != 4 && len(parts) != 5) || (len(parts) == 5 && parts[4] != "content") {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
id := strings.ToLower(parts[3])
|
||||
if !uuidRE.MatchString(id) {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
a, key, e := scanAttachment(s.db.QueryRow(r.Context(), `SELECT `+attachmentColumns+` FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id AND m.user_id=$2 WHERE a.id=$1`, id, uid))
|
||||
if errors.Is(e, pgx.ErrNoRows) {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if len(parts) == 4 {
|
||||
reply(w, 200, a)
|
||||
return
|
||||
}
|
||||
q, e := url.ParseQuery(r.URL.RawQuery)
|
||||
if e != nil || len(q["variant"]) > 1 {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
variant := q.Get("variant")
|
||||
if variant != "" && variant != "original" {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
object, e := s.objects.Client.GetObject(r.Context(), s.objects.Bucket, key, minio.GetObjectOptions{})
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer object.Close()
|
||||
info, e := object.Stat()
|
||||
if e != nil || info.Size != a.Size {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", a.ContentType)
|
||||
w.Header().Set("Content-Length", fmt.Sprint(a.Size))
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": a.Filename}))
|
||||
w.WriteHeader(200)
|
||||
_, _ = io.Copy(w, object)
|
||||
}
|
||||
|
||||
// Normalize optional attachments before hashing; JSON key ordering is irrelevant.
|
||||
func parseMessage(w http.ResponseWriter, r *http.Request) (string, []string, bool) {
|
||||
var in struct {
|
||||
Text *string `json:"text"`
|
||||
Attachments []string `json:"attachment_ids"`
|
||||
}
|
||||
// strict decoder accepts an explicitly optional field (suffix '?').
|
||||
if !decode(w, r, &in, "text", "attachment_ids?") {
|
||||
return "", nil, false
|
||||
}
|
||||
if in.Text == nil || !utf8.ValidString(*in.Text) || utf8.RuneCountInString(*in.Text) > 4000 || len(in.Attachments) > 10 || (strings.TrimSpace(*in.Text) == "" && len(in.Attachments) == 0) {
|
||||
bad(w)
|
||||
return "", nil, false
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for i, id := range in.Attachments {
|
||||
id = strings.ToLower(id)
|
||||
if !uuidRE.MatchString(id) || seen[id] {
|
||||
bad(w)
|
||||
return "", nil, false
|
||||
}
|
||||
seen[id] = true
|
||||
in.Attachments[i] = id
|
||||
}
|
||||
if in.Attachments == nil {
|
||||
in.Attachments = []string{}
|
||||
}
|
||||
return *in.Text, in.Attachments, true
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"mtusi/messenger/internal/database"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Real PostgreSQL + HTTP + the race detector, in the optional Compose tests
|
||||
// service. Local go test skips this test unless the DB secret path is supplied.
|
||||
func TestDatabaseConcurrentHTTP(t *testing.T) {
|
||||
path := os.Getenv("DATABASE_TEST_URL_FILE")
|
||||
if path == "" {
|
||||
t.Skip("run make test-race for real PostgreSQL HTTP integration")
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
pool, err := database.Open(ctx, strings.TrimSpace(string(raw)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer pool.Close()
|
||||
if err = database.Migrate(ctx, pool); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
api := New(pool, nil, make([]byte, 32), 3600, nil)
|
||||
ts := httptest.NewServer(api)
|
||||
defer ts.Close()
|
||||
call := func(method, path, body, auth string, status int) map[string]any {
|
||||
t.Helper()
|
||||
r, err := http.NewRequest(method, ts.URL+path, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return nil
|
||||
}
|
||||
if body != "" {
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
r.Header.Set("Authorization", auth)
|
||||
if method == "POST" && strings.HasSuffix(path, "/messages") {
|
||||
r.Header.Set("Idempotency-Key", uuid())
|
||||
}
|
||||
response, err := ts.Client().Do(r)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return nil
|
||||
}
|
||||
defer response.Body.Close()
|
||||
b, err := io.ReadAll(response.Body)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return nil
|
||||
}
|
||||
if response.StatusCode != status {
|
||||
t.Errorf("%s %s: want %d got %d", method, path, status, response.StatusCode)
|
||||
return nil
|
||||
}
|
||||
var v map[string]any
|
||||
if err = json.Unmarshal(b, &v); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
username := "go_" + strings.ReplaceAll(uuid(), "-", "")[:20]
|
||||
user := call("POST", "/api/v1/users", fmt.Sprintf(`{"username":%q,"display_name":"Race","password":"Synthetic-Go-Test-123"}`, username), "", 201)
|
||||
if user == nil {
|
||||
t.Fatal("create failed")
|
||||
}
|
||||
basic := "Basic " + base64.StdEncoding.EncodeToString([]byte(username+":Synthetic-Go-Test-123"))
|
||||
session := call("POST", "/api/v1/auth/sessions", "", basic, 201)
|
||||
if session == nil {
|
||||
t.Fatal("login failed")
|
||||
}
|
||||
bearer := "Bearer " + session["token"].(string)
|
||||
chat := call("POST", "/api/v1/chats", fmt.Sprintf(`{"title":"Race","member_ids":[%q]}`, user["id"]), bearer, 201)
|
||||
if chat == nil {
|
||||
t.Fatal("chat failed")
|
||||
}
|
||||
messages := "/api/v1/chats/" + chat["id"].(string) + "/messages"
|
||||
var wg sync.WaitGroup
|
||||
for worker := 0; worker < 10; worker++ {
|
||||
wg.Add(1)
|
||||
go func(worker int) {
|
||||
defer wg.Done()
|
||||
for j := 0; j < 10; j++ {
|
||||
call("POST", messages, fmt.Sprintf(`{"text":"worker-%d-message-%d"}`, worker, j), bearer, 201)
|
||||
}
|
||||
}(worker)
|
||||
}
|
||||
wg.Wait()
|
||||
page := call("GET", messages+"?limit=100", "", bearer, 200)
|
||||
if page == nil {
|
||||
t.Fatal("history failed")
|
||||
}
|
||||
items := page["items"].([]any)
|
||||
if len(items) != 100 {
|
||||
t.Fatalf("expected 100 messages, got %d", len(items))
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, v := range items {
|
||||
m := v.(map[string]any)
|
||||
id := m["id"].(string)
|
||||
if seen[id] {
|
||||
t.Fatal("duplicate ID")
|
||||
}
|
||||
seen[id] = true
|
||||
}
|
||||
// Race a single key through the real server, then verify only one side effect.
|
||||
var repeatWG sync.WaitGroup
|
||||
ids := make(chan string, 12)
|
||||
for i := 0; i < 12; i++ {
|
||||
repeatWG.Add(1)
|
||||
go func() {
|
||||
defer repeatWG.Done()
|
||||
req, _ := http.NewRequest("POST", ts.URL+messages, strings.NewReader(`{"text":"one key"}`))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", bearer)
|
||||
req.Header.Set("Idempotency-Key", "same-concurrent-key")
|
||||
res, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return
|
||||
}
|
||||
defer res.Body.Close()
|
||||
var m Message
|
||||
if res.StatusCode != 201 || json.NewDecoder(res.Body).Decode(&m) != nil {
|
||||
t.Error("repeat failed")
|
||||
return
|
||||
}
|
||||
ids <- m.ID
|
||||
}()
|
||||
}
|
||||
repeatWG.Wait()
|
||||
close(ids)
|
||||
var expected string
|
||||
for id := range ids {
|
||||
if expected == "" {
|
||||
expected = id
|
||||
}
|
||||
if id != expected {
|
||||
t.Error("idempotency duplicate")
|
||||
}
|
||||
}
|
||||
var count int
|
||||
if err := pool.QueryRow(context.Background(), "SELECT count(*) FROM messages WHERE chat_id=$1", chat["id"]).Scan(&count); err != nil || count != 101 {
|
||||
t.Errorf("want 101 persisted messages, got %d: %v", count, err)
|
||||
}
|
||||
t.Log("100 confirmed messages from 10 clients through real HTTP and PostgreSQL; race detector enabled")
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"golang.org/x/crypto/argon2"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Fixed, bounded parameters: 64 MiB, 3 passes, 2 lanes, 16-byte salt,
|
||||
// 32-byte result. The HTTP layer bounds simultaneous KDF work to 2 calls.
|
||||
func hashPassword(password string) string {
|
||||
salt := make([]byte, 16)
|
||||
if _, err := rand.Read(salt); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
hash := argon2.IDKey([]byte(password), salt, 3, 64*1024, 2, 32)
|
||||
return "$argon2id$v=19$m=65536,t=3,p=2$" + base64.RawStdEncoding.EncodeToString(salt) + "$" + base64.RawStdEncoding.EncodeToString(hash)
|
||||
}
|
||||
func verifyPassword(password, encoded string) bool {
|
||||
parts := strings.Split(encoded, "$")
|
||||
if len(parts) != 6 || strings.Join(parts[:4], "$") != "$argon2id$v=19$m=65536,t=3,p=2" {
|
||||
return false
|
||||
}
|
||||
salt, e1 := base64.RawStdEncoding.DecodeString(parts[4])
|
||||
want, e2 := base64.RawStdEncoding.DecodeString(parts[5])
|
||||
if e1 != nil || e2 != nil || len(salt) != 16 || len(want) != 32 {
|
||||
return false
|
||||
}
|
||||
got := argon2.IDKey([]byte(password), salt, 3, 64*1024, 2, 32)
|
||||
return subtle.ConstantTimeCompare(got, want) == 1
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPasswordFullUnicodeRange(t *testing.T) {
|
||||
password := strings.Repeat("🙂", 128) // 512 bytes: exceeds bcrypt's 72-byte limit.
|
||||
hash := hashPassword(password)
|
||||
if !verifyPassword(password, hash) || verifyPassword(password+"x", hash) {
|
||||
t.Fatal("password mismatch")
|
||||
}
|
||||
if hash == hashPassword(password) {
|
||||
t.Fatal("salt reused")
|
||||
}
|
||||
if verifyPassword(password, "$argon2id$v=19$m=999999999,t=3,p=2$abc$def") {
|
||||
t.Fatal("unbounded parameters accepted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/minio/minio-go/v7"
|
||||
)
|
||||
|
||||
type ReconcileResult struct {
|
||||
DryRun bool `json:"dry_run"`
|
||||
GraceSeconds int `json:"grace_seconds"`
|
||||
Scanned int `json:"scanned"`
|
||||
Orphans int `json:"orphans"`
|
||||
Missing int `json:"missing"`
|
||||
StaleIntents int `json:"stale_intents"`
|
||||
Fresh int `json:"fresh_protected"`
|
||||
Referenced int `json:"referenced_protected"`
|
||||
DeletedObjects int `json:"deleted_objects"`
|
||||
DeletedMetadata int `json:"deleted_metadata"`
|
||||
ClearedIntents int `json:"cleared_intents"`
|
||||
DurationMS int64 `json:"duration_ms"`
|
||||
}
|
||||
|
||||
func Reconcile(db *pgxpool.Pool, o *ObjectStore, args []string) error {
|
||||
flags := flag.NewFlagSet("reconcile", flag.ContinueOnError)
|
||||
apply := flags.Bool("apply", false, "apply safe deletions; default dry-run")
|
||||
grace := flags.Int("grace-seconds", 3600, "minimum age (>=60s, greater than upload deadline)")
|
||||
if e := flags.Parse(args); e != nil {
|
||||
return e
|
||||
}
|
||||
if *grace < 60 {
|
||||
return fmt.Errorf("grace must be at least 60 seconds")
|
||||
}
|
||||
start := time.Now()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||
defer cancel()
|
||||
result := ReconcileResult{DryRun: !*apply, GraceSeconds: *grace}
|
||||
// Snapshot is only an inventory. Every candidate is rechecked under the SAME
|
||||
// per-object transaction lock as upload; current rows and HEAD decide removal.
|
||||
keys := map[string]bool{}
|
||||
for object := range o.Client.ListObjects(ctx, o.Bucket, minio.ListObjectsOptions{Prefix: "originals/", Recursive: true}) {
|
||||
if object.Err != nil {
|
||||
return fmt.Errorf("S3 list failed")
|
||||
}
|
||||
keys[object.Key] = true
|
||||
}
|
||||
rows, e := db.Query(ctx, "SELECT object_key FROM attachments UNION SELECT object_key FROM upload_intents")
|
||||
if e != nil {
|
||||
return fmt.Errorf("inventory failed")
|
||||
}
|
||||
for rows.Next() {
|
||||
var key string
|
||||
if e = rows.Scan(&key); e != nil {
|
||||
rows.Close()
|
||||
return e
|
||||
}
|
||||
keys[key] = true
|
||||
}
|
||||
e = rows.Err()
|
||||
rows.Close()
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
for key := range keys {
|
||||
result.Scanned++
|
||||
tx, e := db.Begin(ctx)
|
||||
if e != nil {
|
||||
return fmt.Errorf("reconcile transaction failed")
|
||||
}
|
||||
err := func() error {
|
||||
defer tx.Rollback(context.Background())
|
||||
if _, e = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,4))", key); e != nil {
|
||||
return e
|
||||
}
|
||||
var id string
|
||||
var metadataTime time.Time
|
||||
// Lock against concurrent message attachment linkage/deletion.
|
||||
rows, e := tx.Query(ctx, "SELECT id::text,created_at FROM attachments WHERE object_key=$1 FOR UPDATE", key)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
hasMetadata := rows.Next()
|
||||
if hasMetadata {
|
||||
e = rows.Scan(&id, &metadataTime)
|
||||
}
|
||||
rows.Close()
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
var intentAge bool
|
||||
if e = tx.QueryRow(ctx, "SELECT EXISTS(SELECT 1 FROM upload_intents WHERE object_key=$1 AND created_at<=clock_timestamp()-make_interval(secs=>$2))", key, *grace).Scan(&intentAge); e != nil {
|
||||
return e
|
||||
}
|
||||
var freshIntent bool
|
||||
if e = tx.QueryRow(ctx, "SELECT EXISTS(SELECT 1 FROM upload_intents WHERE object_key=$1 AND created_at>clock_timestamp()-make_interval(secs=>$2))", key, *grace).Scan(&freshIntent); e != nil {
|
||||
return e
|
||||
}
|
||||
info, headErr := o.Client.StatObject(ctx, o.Bucket, key, minio.StatObjectOptions{})
|
||||
exists := headErr == nil
|
||||
if headErr != nil {
|
||||
code := minio.ToErrorResponse(headErr).Code
|
||||
if code != "NoSuchKey" && code != "NoSuchObject" && code != "NotFound" {
|
||||
return fmt.Errorf("HEAD failed; no repair attempted")
|
||||
}
|
||||
}
|
||||
fresh := freshIntent || (exists && time.Since(info.LastModified) < time.Duration(*grace)*time.Second) || (hasMetadata && time.Since(metadataTime) < time.Duration(*grace)*time.Second)
|
||||
if fresh {
|
||||
result.Fresh++
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
if !hasMetadata && exists {
|
||||
result.Orphans++
|
||||
if *apply {
|
||||
if e = o.Client.RemoveObject(ctx, o.Bucket, key, minio.RemoveObjectOptions{}); e != nil {
|
||||
return e
|
||||
}
|
||||
result.DeletedObjects++
|
||||
}
|
||||
}
|
||||
if hasMetadata && !exists {
|
||||
result.Missing++
|
||||
var referenced bool
|
||||
if e = tx.QueryRow(ctx, "SELECT EXISTS(SELECT 1 FROM message_attachments WHERE attachment_id=$1)", id).Scan(&referenced); e != nil {
|
||||
return e
|
||||
}
|
||||
if referenced {
|
||||
result.Referenced++
|
||||
} else if *apply {
|
||||
if _, e = tx.Exec(ctx, "DELETE FROM attachments WHERE id=$1", id); e != nil {
|
||||
return e
|
||||
}
|
||||
result.DeletedMetadata++
|
||||
}
|
||||
}
|
||||
if intentAge {
|
||||
result.StaleIntents++
|
||||
if *apply {
|
||||
if _, e = tx.Exec(ctx, "DELETE FROM upload_intents WHERE object_key=$1", key); e != nil {
|
||||
return e
|
||||
}
|
||||
result.ClearedIntents++
|
||||
}
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}()
|
||||
if err != nil {
|
||||
return fmt.Errorf("reconcile failed safely: %w", err)
|
||||
}
|
||||
}
|
||||
result.DurationMS = time.Since(start).Milliseconds()
|
||||
return json.NewEncoder(os.Stdout).Encode(result)
|
||||
}
|
||||
@@ -0,0 +1,783 @@
|
||||
// Package messenger implements the lab4 contract. PostgreSQL is authoritative.
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/hmac"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"io"
|
||||
"log/slog"
|
||||
"mime"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
type User struct {
|
||||
ID string `json:"id"`
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
type Chat struct {
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
MemberIDs []string `json:"member_ids"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
type Message struct {
|
||||
AttachmentIDs []string `json:"attachment_ids"`
|
||||
ID string `json:"id"`
|
||||
ChatID string `json:"chat_id"`
|
||||
SenderID string `json:"sender_id"`
|
||||
Text string `json:"text"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
type Page[T any] struct {
|
||||
Items []T `json:"items"`
|
||||
NextCursor *string `json:"next_cursor"`
|
||||
}
|
||||
|
||||
func uuid() string {
|
||||
b := make([]byte, 16)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
b[6] = (b[6] & 15) | 64
|
||||
b[8] = (b[8] & 63) | 128
|
||||
return fmt.Sprintf("%x-%x-%x-%x-%x", b[:4], b[4:6], b[6:8], b[8:10], b[10:])
|
||||
}
|
||||
|
||||
var uuidRE = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||
var usernameRE = regexp.MustCompile(`^[a-z0-9_]{3,32}$`)
|
||||
|
||||
func validText(v string, max int) bool {
|
||||
return utf8.ValidString(v) && strings.TrimSpace(v) != "" && utf8.RuneCountInString(v) <= max
|
||||
}
|
||||
func reply(w http.ResponseWriter, status int, v any) {
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
func fail(w http.ResponseWriter, status int, code, msg string) {
|
||||
reply(w, status, map[string]any{"error": map[string]string{"code": code, "message": msg, "request_id": w.Header().Get("X-Request-Id")}})
|
||||
}
|
||||
func bad(w http.ResponseWriter) { fail(w, 400, "invalid_request", "Invalid request") }
|
||||
func missing(w http.ResponseWriter) { fail(w, 404, "not_found", "Object not found") }
|
||||
|
||||
// Decode exact object keys (encoding/json structs alone accept case variants).
|
||||
// Reject null, duplicate keys, trailing documents and invalid UTF-8 as well.
|
||||
func decode(w http.ResponseWriter, r *http.Request, dst any, fields ...string) bool {
|
||||
media, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
|
||||
if err != nil || media != "application/json" {
|
||||
fail(w, 415, "unsupported_media_type", "Expected application/json")
|
||||
return false
|
||||
}
|
||||
b, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20))
|
||||
if err != nil {
|
||||
if _, ok := err.(*http.MaxBytesError); ok {
|
||||
fail(w, 413, "payload_too_large", "Body exceeds 1 MiB")
|
||||
} else {
|
||||
bad(w)
|
||||
}
|
||||
return false
|
||||
}
|
||||
if !utf8.Valid(b) {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
d := json.NewDecoder(bytes.NewReader(b))
|
||||
token, err := d.Token()
|
||||
if err != nil || token != json.Delim('{') {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
allowed := map[string]bool{}
|
||||
for _, f := range fields {
|
||||
allowed[strings.TrimSuffix(f, "?")] = true
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for d.More() {
|
||||
t, e := d.Token()
|
||||
k, ok := t.(string)
|
||||
if e != nil || !ok || !allowed[k] || seen[k] {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
seen[k] = true
|
||||
var raw json.RawMessage
|
||||
if d.Decode(&raw) != nil || bytes.Equal(raw, []byte("null")) {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
}
|
||||
if _, err = d.Token(); err != nil {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
if _, err = d.Token(); err != io.EOF {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
for _, f := range fields {
|
||||
if !strings.HasSuffix(f, "?") && !seen[f] {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
}
|
||||
if json.Unmarshal(b, dst) != nil {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
type cursor struct {
|
||||
Scope string `json:"scope"`
|
||||
Time string `json:"time"`
|
||||
ID string `json:"id"`
|
||||
}
|
||||
|
||||
func (s *Server) encodeCursor(c cursor) *string {
|
||||
b, _ := json.Marshal(c)
|
||||
mac := hmac.New(sha256.New, s.secret)
|
||||
mac.Write(b)
|
||||
v := base64.RawURLEncoding.EncodeToString(b) + "." + base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
|
||||
return &v
|
||||
}
|
||||
func (s *Server) pagination(w http.ResponseWriter, r *http.Request, scope string) (int, cursor, bool) {
|
||||
var c cursor
|
||||
limit := 50
|
||||
// ParseQuery explicitly reports malformed escaping, unlike URL.Query.
|
||||
values, e := url.ParseQuery(r.URL.RawQuery)
|
||||
if e != nil {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
if v, ok := values["limit"]; ok {
|
||||
if len(v) != 1 || v[0] == "" || strings.Trim(v[0], "0123456789") != "" {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
limit, e = strconv.Atoi(v[0])
|
||||
if e != nil || limit < 1 || limit > 100 {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
}
|
||||
if v, ok := values["cursor"]; ok {
|
||||
if len(v) != 1 || len(v[0]) > 2048 {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
parts := strings.Split(v[0], ".")
|
||||
if len(parts) != 2 {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
b, e1 := base64.RawURLEncoding.DecodeString(parts[0])
|
||||
sig, e2 := base64.RawURLEncoding.DecodeString(parts[1])
|
||||
mac := hmac.New(sha256.New, s.secret)
|
||||
mac.Write(b)
|
||||
if e1 != nil || e2 != nil || !hmac.Equal(sig, mac.Sum(nil)) || json.Unmarshal(b, &c) != nil || c.Scope != scope || !uuidRE.MatchString(c.ID) {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
}
|
||||
if c.ID != "" {
|
||||
if _, err := time.Parse(time.RFC3339Nano, c.Time); err != nil {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
}
|
||||
return limit, c, true
|
||||
}
|
||||
|
||||
type Server struct {
|
||||
db *pgxpool.Pool
|
||||
cache *redis.Client
|
||||
secret []byte
|
||||
ttl int
|
||||
log *slog.Logger
|
||||
stopping atomic.Bool
|
||||
kdf chan struct{}
|
||||
objects *ObjectStore
|
||||
uploads chan struct{}
|
||||
instance string
|
||||
dummyHash string
|
||||
}
|
||||
|
||||
func New(db *pgxpool.Pool, cache *redis.Client, secret []byte, ttl int, logger *slog.Logger) *Server {
|
||||
hostname, _ := os.Hostname()
|
||||
return &Server{instance: hostname, db: db, cache: cache, secret: secret, ttl: ttl, log: logger, kdf: make(chan struct{}, 2), dummyHash: hashPassword("dummy-not-a-real-password")}
|
||||
}
|
||||
func (s *Server) Stop() { s.stopping.Store(true) }
|
||||
func (s *Server) acquireKDF(ctx context.Context, w http.ResponseWriter) bool {
|
||||
select {
|
||||
case s.kdf <- struct{}{}:
|
||||
return true
|
||||
case <-ctx.Done():
|
||||
unavailable(w)
|
||||
return false
|
||||
}
|
||||
}
|
||||
func unavailable(w http.ResponseWriter) {
|
||||
fail(w, 503, "unavailable", "Dependency temporarily unavailable")
|
||||
}
|
||||
func unauthorized(w http.ResponseWriter, scheme string) {
|
||||
challenge := scheme + ` realm="messenger"`
|
||||
if scheme == "Basic" {
|
||||
challenge += `, charset="UTF-8"`
|
||||
}
|
||||
w.Header().Set("WWW-Authenticate", challenge)
|
||||
fail(w, 401, "unauthorized", "Invalid or missing credentials")
|
||||
}
|
||||
func stamp(t time.Time) string { return t.UTC().Format("2006-01-02T15:04:05.000000Z") }
|
||||
|
||||
const userColumns = "id::text, username, display_name, created_at"
|
||||
|
||||
type scanner interface{ Scan(...any) error }
|
||||
|
||||
func scanUser(row scanner) (User, error) {
|
||||
var u User
|
||||
var t time.Time
|
||||
err := row.Scan(&u.ID, &u.Username, &u.DisplayName, &t)
|
||||
u.CreatedAt = stamp(t)
|
||||
return u, err
|
||||
}
|
||||
func scanMessage(row scanner) (Message, error) {
|
||||
var m Message
|
||||
var t time.Time
|
||||
err := row.Scan(&m.ID, &m.ChatID, &m.SenderID, &m.Text, &t)
|
||||
m.CreatedAt = stamp(t)
|
||||
m.AttachmentIDs = []string{}
|
||||
return m, err
|
||||
}
|
||||
|
||||
func (s *Server) profile(ctx context.Context, uid string) (User, error) {
|
||||
// Valkey caches ONLY immutable User JSON. Session validity is always read
|
||||
// from PostgreSQL first; cache loss never permits anonymous access.
|
||||
key := "profile:v1:" + uid
|
||||
if s.cache != nil {
|
||||
c, cancel := context.WithTimeout(ctx, 80*time.Millisecond)
|
||||
raw, err := s.cache.Get(c, key).Bytes()
|
||||
cancel()
|
||||
var u User
|
||||
if err == nil && json.Unmarshal(raw, &u) == nil && u.ID == uid {
|
||||
return u, nil
|
||||
}
|
||||
}
|
||||
u, err := scanUser(s.db.QueryRow(ctx, "SELECT "+userColumns+" FROM users WHERE id=$1", uid))
|
||||
if err != nil {
|
||||
return u, err
|
||||
}
|
||||
if s.cache != nil {
|
||||
raw, _ := json.Marshal(u)
|
||||
c, cancel := context.WithTimeout(ctx, 80*time.Millisecond)
|
||||
_ = s.cache.Set(c, key, raw, 300*time.Second).Err()
|
||||
cancel()
|
||||
}
|
||||
return u, nil
|
||||
}
|
||||
func (s *Server) authenticate(w http.ResponseWriter, r *http.Request) (User, []byte, bool) {
|
||||
parts := strings.Fields(r.Header.Get("Authorization"))
|
||||
if len(parts) != 2 || !strings.EqualFold(parts[0], "Bearer") {
|
||||
unauthorized(w, "Bearer")
|
||||
return User{}, nil, false
|
||||
}
|
||||
raw, err := base64.RawURLEncoding.DecodeString(parts[1])
|
||||
if err != nil || len(raw) != 32 {
|
||||
unauthorized(w, "Bearer")
|
||||
return User{}, nil, false
|
||||
}
|
||||
digest := sha256.Sum256([]byte(parts[1]))
|
||||
var uid string
|
||||
err = s.db.QueryRow(r.Context(), "SELECT user_id::text FROM sessions WHERE token_hash=$1 AND expires_at>clock_timestamp()", digest[:]).Scan(&uid)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
unauthorized(w, "Bearer")
|
||||
return User{}, nil, false
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return User{}, nil, false
|
||||
}
|
||||
u, err := s.profile(r.Context(), uid)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return User{}, nil, false
|
||||
}
|
||||
return u, digest[:], true
|
||||
}
|
||||
|
||||
type recorder struct {
|
||||
http.ResponseWriter
|
||||
status int
|
||||
}
|
||||
|
||||
func (w *recorder) WriteHeader(status int) { w.status = status; w.ResponseWriter.WriteHeader(status) }
|
||||
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
start := time.Now()
|
||||
out := &recorder{w, 200}
|
||||
w = out
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.Header().Set("X-Request-Id", uuid())
|
||||
w.Header().Set("X-Instance-Id", s.instance)
|
||||
budget := 3 * time.Second
|
||||
if strings.Contains(r.URL.Path, "attachments") {
|
||||
budget = 30 * time.Second
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), budget)
|
||||
defer cancel()
|
||||
r = r.WithContext(ctx)
|
||||
route := "unknown"
|
||||
defer func() {
|
||||
if recover() != nil {
|
||||
fail(w, 500, "internal_error", "Internal server error")
|
||||
}
|
||||
if s.log != nil {
|
||||
s.log.Info("request", "method", r.Method, "route", route, "status", out.status, "duration_us", time.Since(start).Microseconds(), "request_id", w.Header().Get("X-Request-Id"))
|
||||
}
|
||||
}()
|
||||
if r.Method == "GET" && (r.URL.Path == "/health/live" || r.URL.Path == "/health/ready") {
|
||||
route = r.URL.Path
|
||||
if r.URL.Path == "/health/ready" && (s.stopping.Load() || s.db.Ping(ctx) != nil || (s.objects != nil && !s.objects.Ready(ctx))) {
|
||||
unavailable(w)
|
||||
} else {
|
||||
reply(w, 200, map[string]string{"status": "ok"})
|
||||
}
|
||||
return
|
||||
}
|
||||
if s.stopping.Load() {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if r.Method == "POST" && r.URL.Path == "/api/v1/users" {
|
||||
route = "/users"
|
||||
s.createUser(w, r)
|
||||
return
|
||||
}
|
||||
if r.Method == "POST" && r.URL.Path == "/api/v1/auth/sessions" {
|
||||
route = "/auth/sessions"
|
||||
s.login(w, r)
|
||||
return
|
||||
}
|
||||
user, digest, ok := s.authenticate(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if r.Method == "DELETE" && r.URL.Path == "/api/v1/auth/sessions/current" {
|
||||
route = "/auth/sessions/current"
|
||||
tag, err := s.db.Exec(ctx, "DELETE FROM sessions WHERE token_hash=$1", digest)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
} else if tag.RowsAffected() == 0 {
|
||||
unauthorized(w, "Bearer")
|
||||
} else {
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
return
|
||||
}
|
||||
if r.Method == "GET" && r.URL.Path == "/api/v1/users/me" {
|
||||
route = "/users/me"
|
||||
reply(w, 200, user)
|
||||
return
|
||||
}
|
||||
if r.URL.Path == "/api/v1/chats" {
|
||||
route = "/chats"
|
||||
switch r.Method {
|
||||
case "POST":
|
||||
s.createChat(w, r, user.ID)
|
||||
case "GET":
|
||||
s.listChats(w, r, user.ID)
|
||||
default:
|
||||
missing(w)
|
||||
}
|
||||
return
|
||||
}
|
||||
parts := strings.Split(strings.TrimPrefix(r.URL.Path, "/"), "/")
|
||||
if len(parts) >= 4 && parts[0] == "api" && parts[1] == "v1" && parts[2] == "attachments" && s.objects != nil {
|
||||
route = "/attachments/{id}"
|
||||
s.serveAttachment(w, r, user.ID, parts)
|
||||
return
|
||||
}
|
||||
if len(parts) < 4 || len(parts) > 5 || parts[0] != "api" || parts[1] != "v1" || parts[2] != "chats" {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
cid := strings.ToLower(parts[3])
|
||||
route = "/chats/{id}"
|
||||
if !uuidRE.MatchString(cid) {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
chat, err := s.getChat(ctx, cid, user.ID)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if len(parts) == 4 && r.Method == "GET" {
|
||||
reply(w, 200, chat)
|
||||
return
|
||||
}
|
||||
if len(parts) == 5 && parts[4] == "attachments" && r.Method == "POST" && s.objects != nil {
|
||||
route = "/chats/{id}/attachments"
|
||||
s.uploadAttachment(w, r, user.ID, cid)
|
||||
return
|
||||
}
|
||||
if len(parts) == 5 && parts[4] == "messages" {
|
||||
route = "/chats/{id}/messages"
|
||||
switch r.Method {
|
||||
case "POST":
|
||||
s.createMessage(w, r, user.ID, cid)
|
||||
case "GET":
|
||||
s.listMessages(w, r, cid)
|
||||
default:
|
||||
missing(w)
|
||||
}
|
||||
return
|
||||
}
|
||||
missing(w)
|
||||
}
|
||||
func (s *Server) createUser(w http.ResponseWriter, r *http.Request) {
|
||||
var input struct {
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
Password string `json:"password"`
|
||||
}
|
||||
if !decode(w, r, &input, "username", "display_name", "password") {
|
||||
return
|
||||
}
|
||||
if !usernameRE.MatchString(input.Username) || !validText(input.DisplayName, 100) || !validText(input.Password, 128) || utf8.RuneCountInString(input.Password) < 12 {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
if !s.acquireKDF(r.Context(), w) {
|
||||
return
|
||||
}
|
||||
hash := hashPassword(input.Password)
|
||||
<-s.kdf
|
||||
u, err := scanUser(s.db.QueryRow(r.Context(), "INSERT INTO users(id,username,display_name,password_hash) VALUES($1,$2,$3,$4) RETURNING "+userColumns, uuid(), input.Username, input.DisplayName, hash))
|
||||
var pgerr *pgconn.PgError
|
||||
if errors.As(err, &pgerr) && pgerr.Code == "23505" {
|
||||
fail(w, 409, "conflict", "Username already exists")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
reply(w, 201, u)
|
||||
}
|
||||
func (s *Server) login(w http.ResponseWriter, r *http.Request) {
|
||||
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1))
|
||||
if err != nil || len(body) != 0 {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
username, password, ok := r.BasicAuth()
|
||||
if !ok || !usernameRE.MatchString(username) || !validText(password, 128) || utf8.RuneCountInString(password) < 12 {
|
||||
unauthorized(w, "Basic")
|
||||
return
|
||||
}
|
||||
var u User
|
||||
var hash string
|
||||
var t time.Time
|
||||
err = s.db.QueryRow(r.Context(), "SELECT "+userColumns+", password_hash FROM users WHERE username=$1", username).Scan(&u.ID, &u.Username, &u.DisplayName, &t, &hash)
|
||||
absent := errors.Is(err, pgx.ErrNoRows)
|
||||
if err != nil && !absent {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if absent {
|
||||
hash = s.dummyHash
|
||||
}
|
||||
if !s.acquireKDF(r.Context(), w) {
|
||||
return
|
||||
}
|
||||
valid := verifyPassword(password, hash)
|
||||
<-s.kdf
|
||||
if !valid || absent {
|
||||
unauthorized(w, "Basic")
|
||||
return
|
||||
}
|
||||
raw := make([]byte, 32)
|
||||
if _, err = rand.Read(raw); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
token := base64.RawURLEncoding.EncodeToString(raw)
|
||||
digest := sha256.Sum256([]byte(token))
|
||||
var expires time.Time
|
||||
err = s.db.QueryRow(r.Context(), "INSERT INTO sessions(token_hash,user_id,expires_at) VALUES($1,$2,clock_timestamp()+make_interval(secs=>$3)) RETURNING expires_at", digest[:], u.ID, s.ttl).Scan(&expires)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
u.CreatedAt = stamp(t)
|
||||
reply(w, 201, map[string]any{"token": token, "token_type": "Bearer", "expires_at": stamp(expires), "user": u})
|
||||
}
|
||||
func (s *Server) createChat(w http.ResponseWriter, r *http.Request, uid string) {
|
||||
var input struct {
|
||||
Title string `json:"title"`
|
||||
MemberIDs []string `json:"member_ids"`
|
||||
}
|
||||
if !decode(w, r, &input, "title", "member_ids") {
|
||||
return
|
||||
}
|
||||
if !validText(input.Title, 100) || len(input.MemberIDs) < 1 || len(input.MemberIDs) > 100 {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for i, id := range input.MemberIDs {
|
||||
id = strings.ToLower(id)
|
||||
if !uuidRE.MatchString(id) || seen[id] {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
seen[id] = true
|
||||
input.MemberIDs[i] = id
|
||||
}
|
||||
if !seen[uid] {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
tx, err := s.db.Begin(r.Context())
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
chat := Chat{ID: uuid(), Title: input.Title, MemberIDs: input.MemberIDs}
|
||||
var t time.Time
|
||||
err = tx.QueryRow(r.Context(), "INSERT INTO chats(id,title) VALUES($1,$2) RETURNING created_at", chat.ID, chat.Title).Scan(&t)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
for _, id := range input.MemberIDs {
|
||||
_, err = tx.Exec(r.Context(), "INSERT INTO chat_members(chat_id,user_id) VALUES($1,$2)", chat.ID, id)
|
||||
var pgerr *pgconn.PgError
|
||||
if errors.As(err, &pgerr) && pgerr.Code == "23503" {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
}
|
||||
if tx.Commit(r.Context()) != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
chat.CreatedAt = stamp(t)
|
||||
reply(w, 201, chat)
|
||||
}
|
||||
|
||||
const chatColumns = `c.id::text,c.title,c.created_at,ARRAY(SELECT m.user_id::text FROM chat_members m WHERE m.chat_id=c.id ORDER BY m.user_id)`
|
||||
|
||||
func scanChat(row scanner) (Chat, error) {
|
||||
var c Chat
|
||||
var t time.Time
|
||||
err := row.Scan(&c.ID, &c.Title, &t, &c.MemberIDs)
|
||||
c.CreatedAt = stamp(t)
|
||||
return c, err
|
||||
}
|
||||
func (s *Server) getChat(ctx context.Context, cid, uid string) (Chat, error) {
|
||||
return scanChat(s.db.QueryRow(ctx, "SELECT "+chatColumns+" FROM chats c JOIN chat_members a ON a.chat_id=c.id AND a.user_id=$2 WHERE c.id=$1", cid, uid))
|
||||
}
|
||||
|
||||
var idempotencyRE = regexp.MustCompile(`^[A-Za-z0-9._:-]{1,128}$`)
|
||||
|
||||
func (s *Server) createMessage(w http.ResponseWriter, r *http.Request, uid, cid string) {
|
||||
text, attachments, ok := parseMessage(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
input := struct {
|
||||
Text string `json:"text"`
|
||||
Attachments []string `json:"attachment_ids"`
|
||||
}{text, attachments}
|
||||
keys := r.Header.Values("Idempotency-Key")
|
||||
if len(keys) != 1 || !idempotencyRE.MatchString(keys[0]) {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(input)
|
||||
digest := sha256.Sum256(body)
|
||||
ctx := r.Context()
|
||||
tx, err := s.db.Begin(ctx)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
// Serialize only the same scope/key, across processes. Hash collisions only
|
||||
// serialize unrelated requests; the full unique key still decides identity.
|
||||
_, err = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,0))", uid+":"+cid+":"+keys[0])
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
var previous, saved []byte
|
||||
err = tx.QueryRow(ctx, "SELECT body_hash,response FROM message_requests WHERE user_id=$1 AND chat_id=$2 AND key=$3 AND expires_at>clock_timestamp()", uid, cid, keys[0]).Scan(&previous, &saved)
|
||||
if err == nil {
|
||||
if !bytes.Equal(previous, digest[:]) {
|
||||
fail(w, 409, "conflict", "Key already used with another message")
|
||||
return
|
||||
}
|
||||
if tx.Commit(ctx) != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(201)
|
||||
_, _ = w.Write(saved)
|
||||
return
|
||||
}
|
||||
if !errors.Is(err, pgx.ErrNoRows) {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
// A row lock prevents cleanup of an attachment while a message links it.
|
||||
for _, id := range attachments {
|
||||
var found string
|
||||
err = tx.QueryRow(ctx, "SELECT id::text FROM attachments WHERE id=$1 AND chat_id=$2 FOR KEY SHARE", id, cid).Scan(&found)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
}
|
||||
var t time.Time
|
||||
err = tx.QueryRow(ctx, "UPDATE chats SET last_message_at=GREATEST(clock_timestamp(),last_message_at+interval '1 microsecond') WHERE id=$1 RETURNING last_message_at", cid).Scan(&t)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
m, err := scanMessage(tx.QueryRow(ctx, "INSERT INTO messages(id,chat_id,sender_id,text,created_at) VALUES($1,$2,$3,$4,$5) RETURNING id::text,chat_id::text,sender_id::text,text,created_at", uuid(), cid, uid, input.Text, t))
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
m.AttachmentIDs = attachments
|
||||
for i, id := range attachments {
|
||||
if _, err = tx.Exec(ctx, "INSERT INTO message_attachments(message_id,attachment_id,position) VALUES($1,$2,$3)", m.ID, id, i); err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
}
|
||||
response, _ := json.Marshal(m)
|
||||
_, err = tx.Exec(ctx, `INSERT INTO message_requests(user_id,chat_id,key,body_hash,response,expires_at) VALUES($1,$2,$3,$4,$5,clock_timestamp()+interval '24 hours') ON CONFLICT(user_id,chat_id,key) DO UPDATE SET body_hash=EXCLUDED.body_hash,response=EXCLUDED.response,expires_at=EXCLUDED.expires_at`, uid, cid, keys[0], digest[:], response)
|
||||
if err != nil || tx.Commit(ctx) != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
reply(w, 201, m)
|
||||
}
|
||||
func (s *Server) listMessages(w http.ResponseWriter, r *http.Request, cid string) {
|
||||
scope := "messages:" + cid
|
||||
limit, c, ok := s.pagination(w, r, scope)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
query := `SELECT id::text,chat_id::text,sender_id::text,text,created_at,ARRAY(SELECT attachment_id::text FROM message_attachments ma WHERE ma.message_id=messages.id ORDER BY position) FROM messages WHERE chat_id=$1`
|
||||
args := []any{cid}
|
||||
if c.ID != "" {
|
||||
query += ` AND (created_at,id)>($2::timestamptz,$3::uuid)`
|
||||
args = append(args, c.Time, c.ID)
|
||||
}
|
||||
query += fmt.Sprintf(" ORDER BY messages.created_at,messages.id LIMIT $%d", len(args)+1)
|
||||
args = append(args, limit+1)
|
||||
rows, err := s.db.Query(r.Context(), query, args...)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
items := []Message{}
|
||||
for rows.Next() {
|
||||
var m Message
|
||||
var created time.Time
|
||||
err := rows.Scan(&m.ID, &m.ChatID, &m.SenderID, &m.Text, &created, &m.AttachmentIDs)
|
||||
m.CreatedAt = stamp(created)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
items = append(items, m)
|
||||
}
|
||||
if rows.Err() != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
var next *string
|
||||
if len(items) > limit {
|
||||
items = items[:limit]
|
||||
last := items[limit-1]
|
||||
next = s.encodeCursor(cursor{scope, last.CreatedAt, last.ID})
|
||||
}
|
||||
reply(w, 200, Page[Message]{items, next})
|
||||
}
|
||||
func (s *Server) listChats(w http.ResponseWriter, r *http.Request, uid string) {
|
||||
scope := "chats:" + uid
|
||||
limit, c, ok := s.pagination(w, r, scope)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
query := "SELECT " + chatColumns + " FROM chats c JOIN chat_members a ON a.chat_id=c.id AND a.user_id=$1"
|
||||
args := []any{uid}
|
||||
if c.ID != "" {
|
||||
query += " WHERE (c.created_at,c.id)>($2::timestamptz,$3::uuid)"
|
||||
args = append(args, c.Time, c.ID)
|
||||
}
|
||||
query += fmt.Sprintf(" ORDER BY c.created_at,c.id LIMIT $%d", len(args)+1)
|
||||
args = append(args, limit+1)
|
||||
rows, err := s.db.Query(r.Context(), query, args...)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
items := []Chat{}
|
||||
for rows.Next() {
|
||||
chat, err := scanChat(rows)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
items = append(items, chat)
|
||||
}
|
||||
if rows.Err() != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
var next *string
|
||||
if len(items) > limit {
|
||||
items = items[:limit]
|
||||
last := items[limit-1]
|
||||
next = s.encodeCursor(cursor{scope, last.CreatedAt, last.ID})
|
||||
}
|
||||
reply(w, 200, Page[Chat]{items, next})
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,33 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate local-only credentials. Existing credentials are never overwritten."""
|
||||
import base64
|
||||
import os
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
root=Path(__file__).resolve().parents[1]
|
||||
directory=root/'secrets'
|
||||
directory.mkdir(mode=0o700,exist_ok=True)
|
||||
password_file=directory/'postgres_password'
|
||||
if not password_file.exists():
|
||||
password=secrets.token_hex(24)
|
||||
password_file.write_text(password+'\n')
|
||||
else:
|
||||
password=password_file.read_text().strip()
|
||||
cache_file=directory/'valkey_config'
|
||||
if not cache_file.exists():
|
||||
cache_password=secrets.token_hex(24)
|
||||
cache_file.write_text('bind 0.0.0.0\nprotected-mode yes\nport 6379\ndir /data\nsave ""\nappendonly no\nrequirepass '+cache_password+'\n')
|
||||
else:
|
||||
cache_password=next(line.split()[1] for line in cache_file.read_text().splitlines() if line.startswith('requirepass '))
|
||||
values={'database_url':f'postgres://messenger:{password}@postgres:5432/messenger?sslmode=disable',
|
||||
'cache_url':f'redis://:{cache_password}@valkey:6379/0',
|
||||
'cursor_key':base64.urlsafe_b64encode(secrets.token_bytes(32)).decode().rstrip('=')}
|
||||
values.update({'s3_root_user':'root_'+secrets.token_hex(8), 's3_root_password':secrets.token_hex(24), 's3_access_key_id':'app_'+secrets.token_hex(8), 's3_secret_access_key':secrets.token_hex(24)})
|
||||
for name,value in values.items():
|
||||
p=directory/name
|
||||
if not p.exists():p.write_text(value+'\n')
|
||||
# Compose mounts individual files. Readable by container UID 10001; host
|
||||
# directory 0700 prevents traversal by other local users.
|
||||
for p in directory.iterdir():os.chmod(p,0o444)
|
||||
if not (root/'.env').exists():(root/'.env').write_text('APP_PORT=8084\nSESSION_TTL_SECONDS=3600\n')
|
||||
print('Local secrets ready (values hidden); .env and secrets/ are ignored by Git.')
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
docker compose stop lb api reconciler
|
||||
docker compose restart postgres valkey s3
|
||||
docker compose up -d --wait
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
ids=$(docker compose ps -q api)
|
||||
count=$(printf '%s\n' "$ids" | wc -l | tr -d ' ')
|
||||
for id in $ids; do
|
||||
docker stop --time 6 "$id" >/dev/null
|
||||
docker rm "$id" >/dev/null
|
||||
docker compose up -d --no-deps --scale api="$count" --wait api >/dev/null
|
||||
sleep 2
|
||||
done
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
: "${TARGET_INSTANCE_ID:?required API alias}"
|
||||
for id in $(docker compose ps -q api); do
|
||||
alias=$(docker inspect --format '{{.Config.Hostname}}' "$id")
|
||||
if [ "$alias" = "$TARGET_INSTANCE_ID" ]; then
|
||||
docker stop --time 6 "$id" >/dev/null
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
printf '%s\n' 'Target API instance is not running' >&2
|
||||
exit 1
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env python3
|
||||
import json,os,subprocess,sys,time
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1];sys.path.insert(0,str(ROOT/'tests'))
|
||||
from client import Client,Failure,check
|
||||
c=Client('http://localhost:8084');a,b,e,chat=c.bootstrap();original=(ROOT/'tests/fixtures/sample.png').read_bytes()
|
||||
upload=c.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=failover.png',201,actor=a,raw=original,headers={'Content-Type':'image/png'},schema='Attachment')
|
||||
alias=upload['headers']['X-Instance-Id'];ids=subprocess.check_output(['docker','compose','ps','-q','api'],cwd=ROOT,text=True).splitlines()
|
||||
matched=[id for id in ids if json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]['Config']['Hostname']==alias];check(len(matched)==1,'alias mapping')
|
||||
started=time.monotonic();failures=0
|
||||
try:
|
||||
subprocess.run([str(ROOT/'scripts/stop-one.sh')],cwd=ROOT,env=dict(os.environ,TARGET_INSTANCE_ID=alias),check=True,stdout=subprocess.DEVNULL)
|
||||
while True:
|
||||
try:r=c.call('GET','/api/v1/attachments/'+upload['json']['id']+'/content',actor=b);break
|
||||
except Failure:failures+=1;check(time.monotonic()-started<10,'download recovery >10s');time.sleep(.1)
|
||||
check(r['bytes']==original and r['headers']['X-Instance-Id']!=alias,'download not preserved through another process')
|
||||
stopped=json.loads(subprocess.check_output(['docker','inspect',matched[0]],text=True))[0];check(not stopped['State']['Running'],'target not stopped')
|
||||
result={'upload_instance':alias,'stopped_container':stopped['Name'],'download_instance':r['headers']['X-Instance-Id'],'byte_equal':True,'first_recovery_seconds':time.monotonic()-started,'errors':failures}
|
||||
(ROOT/'evidence/attachment-failover.json').write_text(json.dumps(result,indent=2)+'\n');print('PASS: exact uploader stopped; another API returned identical original')
|
||||
finally:subprocess.run(['docker','compose','up','-d','--wait'],cwd=ROOT,check=True,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
|
||||
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
import concurrent.futures as futures
|
||||
import hashlib,json,os,struct,subprocess,sys,time,urllib.parse,uuid,zlib
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1];sys.path.insert(0,str(ROOT/'tests'))
|
||||
from client import Client,arguments,check
|
||||
|
||||
def png(width,height):
|
||||
def chunk(tag,data):return struct.pack('!I',len(data))+tag+data+struct.pack('!I',zlib.crc32(tag+data)&0xffffffff)
|
||||
# Synthetic noise compresses poorly and makes >=2MiB fixtures without Pillow.
|
||||
pixels=b''.join(b'\0'+os.urandom(width*3) for _ in range(height))
|
||||
return b'\x89PNG\r\n\x1a\n'+chunk(b'IHDR',struct.pack('!IIBBBBB',width,height,8,2,0,0,0))+chunk(b'IDAT',zlib.compress(pixels,1))+chunk(b'IEND',b'')
|
||||
def dc(*a):return subprocess.check_output(['docker','compose',*a],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
|
||||
def stats():
|
||||
result={}
|
||||
for id in dc('ps','-q','api').splitlines():
|
||||
raw=dc('exec','-T','--index',str(json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]['Config']['Labels']['com.docker.compose.container-number']),'api','sh','-c','cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.peak')
|
||||
result[id[:12]]=[int(x) for x in raw.split()]
|
||||
return result
|
||||
|
||||
def main():
|
||||
args=arguments('Attachment boundaries and concurrent memory experiment');c=Client(args.base_url or 'http://localhost:8084');a,b,e,chat=c.bootstrap();path='/api/v1/chats/'+chat['id']+'/attachments'
|
||||
original=(ROOT/'tests/fixtures/sample.png').read_bytes()
|
||||
def upload(raw=original,name='valid.png',mime='image/png',expected=201,actor=a):
|
||||
return c.call('POST',path+'?filename='+urllib.parse.quote(name,safe=''),expected,actor=actor,raw=raw,headers={'Content-Type':mime},schema='Attachment' if expected==201 else None)
|
||||
for name in ['../x.png','a\\b.png','bad\x00.png','bad\n.png',''] :upload(name=name,expected=400)
|
||||
upload(raw=b'not image',expected=415);upload(mime='image/jpeg',expected=415);upload(raw=original[:40],expected=415)
|
||||
upload(raw=b'x'*(10*1024*1024+1),expected=413)
|
||||
# Valid PNG header + CRC, too many pixels; rejection precedes decompression.
|
||||
def chunk(tag,data):return struct.pack('!I',len(data))+tag+data+struct.pack('!I',zlib.crc32(tag+data)&0xffffffff)
|
||||
huge=b'\x89PNG\r\n\x1a\n'+chunk(b'IHDR',struct.pack('!IIBBBBB',10000,10000,8,2,0,0,0))+chunk(b'IDAT',b'')
|
||||
upload(raw=huge,expected=413)
|
||||
att=upload()['json'];key=uuid.uuid4().hex
|
||||
c.message(a,chat,'',key=key,attachments=[att['id']])
|
||||
other=c.call('POST','/api/v1/chats',201,actor=a,body={'title':'Other','member_ids':[a['id']]},schema='Chat')['json']
|
||||
c.call('POST','/api/v1/chats/'+other['id']+'/messages',404,actor=a,body={'text':'wrong chat','attachment_ids':[att['id']]},headers={'Idempotency-Key':uuid.uuid4().hex})
|
||||
c.call('POST','/api/v1/chats/'+chat['id']+'/messages',404,actor=a,body={'text':'unknown','attachment_ids':[str(uuid.uuid4())]},headers={'Idempotency-Key':uuid.uuid4().hex})
|
||||
for suffix in ['', '/content']:c.call('GET','/api/v1/attachments/'+att['id']+suffix,404,actor=e)
|
||||
# Same semantic body, reordered keys and optional [] omitted.
|
||||
first=c.message(a,chat,'normalized',key='normalize-'+key)['json']
|
||||
second=c.message(a,chat,'normalized',key='normalize-'+key,attachments=[])['json'];check(first==second,'optional [] changed idempotency')
|
||||
fixture=png(1024,768);before=stats();start=time.monotonic();snapshots=[]
|
||||
with futures.ThreadPoolExecutor(max_workers=10) as pool:
|
||||
pending=[pool.submit(upload,fixture,'concurrent.png') for _ in range(10)]
|
||||
while not all(f.done() for f in pending):snapshots.append(stats());time.sleep(.1)
|
||||
responses=[f.result() for f in pending]
|
||||
for r in responses:
|
||||
download=c.call('GET','/api/v1/attachments/'+r['json']['id']+'/content',actor=b)
|
||||
check(hashlib.sha256(download['bytes']).hexdigest()==r['json']['sha256'],'checksum mismatch')
|
||||
after=stats()
|
||||
result={'negative_cases':15,'simultaneous_clients':10,'fixture_bytes':len(fixture),'fixture_pixels':1024*768,'duration_seconds':time.monotonic()-start,'before_memory_current_peak_bytes':before,'after_memory_current_peak_bytes':after,'sampled_memory_current_peak_bytes':snapshots,'all_sha256_match':True,'per_api_upload_slots':1}
|
||||
(ROOT/'evidence/attachments.json').write_text(json.dumps(result,indent=2)+'\n');print('PASS: file boundaries, ACL, normalization and 10 simultaneous uploads')
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run committed solution from a real local clone, new credentials and volumes."""
|
||||
import json,os,socket,subprocess,tempfile,uuid
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
lab=json.loads((ROOT/'lab.json').read_text())['number']
|
||||
project='verify-lab'+str(lab)+'-'+uuid.uuid4().hex[:10]
|
||||
with socket.socket() as sock:sock.bind(('127.0.0.1',0));port=sock.getsockname()[1]
|
||||
with tempfile.TemporaryDirectory(prefix=project+'-') as directory:
|
||||
clone=Path(directory)/'repo';subprocess.run(['git','clone','--no-hardlinks',str(ROOT),str(clone)],check=True)
|
||||
env=dict(os.environ,COMPOSE_PROJECT_NAME=project,APP_PORT=str(port),API_REPLICAS='2',SESSION_TTL_SECONDS='3600')
|
||||
def run(*args):subprocess.run(args,cwd=clone,env=env,check=True)
|
||||
def snapshots():
|
||||
ids=subprocess.check_output(['docker','compose','ps','-q'],cwd=clone,env=env,text=True).splitlines()
|
||||
result={}
|
||||
for id in ids:
|
||||
item=json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]
|
||||
service=item['Config']['Labels']['com.docker.compose.service']
|
||||
if service in ['api','postgres','valkey','s3']:result[item['Name']]={'service':service,'started_at':item['State']['StartedAt'],'image_id':item['Image'],'user':item['Config'].get('User',''),'pid1':item['Path']}
|
||||
return result
|
||||
try:
|
||||
run('make','up');run('make','check');run('make','test','BASE_URL=http://localhost:'+str(port))
|
||||
before=snapshots();run('make','test-persistence','ACTION_SCRIPT=scripts/restart.sh','BASE_URL=http://localhost:'+str(port));after=snapshots()
|
||||
for name,item in before.items():assert after[name]['started_at']!=item['started_at'],name+' was not restarted'
|
||||
for item in after.values():
|
||||
if item['service']=='api':assert item['user']=='10001:10001' and item['pid1']=='/usr/local/bin/messenger'
|
||||
(ROOT/'evidence/persistence-processes.json').write_text(json.dumps({'scope':'disposable real clean clone','before':before,'after':after},indent=2)+'\n')
|
||||
print('PASS: real clean clone, fresh secrets/volumes, smoke, same-session persistence, actual store/API restart, non-root PID1')
|
||||
finally:
|
||||
# Only this unique synthetic project, never the working lab volumes.
|
||||
run('docker','compose','down','-v','--remove-orphans')
|
||||
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic storage faults on synthetic attachments; repeatable reconciliation."""
|
||||
import hashlib,json,os,subprocess,sys,time,uuid,urllib.request,urllib.error,socket
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1];sys.path.insert(0,str(ROOT/'tests'))
|
||||
from client import Client,check
|
||||
BASE='http://localhost:8084'
|
||||
def dc(*a,input=None):return subprocess.check_output(['docker','compose',*a],cwd=ROOT,input=input,text=True,stderr=subprocess.DEVNULL).strip()
|
||||
def sql(query):return dc('exec','-T','postgres','psql','-v','ON_ERROR_STOP=1','-U','messenger','-d','messenger','-Atqc',query)
|
||||
def mc(command):
|
||||
# Credentials are consumed inside the one-shot admin container; never output.
|
||||
prefix='mc alias set local http://s3:9000 "$(cat /run/secrets/s3_root_user)" "$(cat /run/secrets/s3_root_password)" >/dev/null; '
|
||||
return dc('run','--rm','--no-deps','--entrypoint','/bin/sh','s3-init','-c',prefix+command)
|
||||
def reconcile(apply=False):
|
||||
raw=dc('run','--rm','--no-deps','api','reconcile','--grace-seconds','60',*(['--apply'] if apply else []))
|
||||
return json.loads(raw.splitlines()[-1])
|
||||
def main():
|
||||
c=Client(BASE);a,b,e,chat=c.bootstrap();original=(ROOT/'tests/fixtures/sample.png').read_bytes();report={}
|
||||
def upload(name):return c.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename='+name,201,actor=a,raw=original,headers={'Content-Type':'image/png'},schema='Attachment')['json']
|
||||
def key(att):return sql("SELECT object_key FROM attachments WHERE id='"+att['id']+"'")
|
||||
used=upload('used.png');c.message(a,chat,'Used and preserved',attachments=[used['id']])
|
||||
missing=upload('missing-unused.png');missing_used=upload('missing-used.png');c.message(a,chat,'Keep reference',attachments=[missing_used['id']])
|
||||
orphan=upload('orphan.png');orphan_key=key(orphan)
|
||||
sql("DELETE FROM attachments WHERE id='"+orphan['id']+"'")
|
||||
for att in [missing,missing_used]:mc('mc rm local/messenger-attachments/'+key(att))
|
||||
# Fail exactly AFTER S3 PUT, at the metadata INSERT. The persistent intent
|
||||
# and real orphan are evidence, without a runtime fault-injection endpoint.
|
||||
sql("CREATE FUNCTION research_reject_attachment() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.filename='commit_fail.png' THEN RAISE EXCEPTION 'synthetic metadata failure'; END IF; RETURN NEW; END $$; CREATE TRIGGER research_fail BEFORE INSERT ON attachments FOR EACH ROW EXECUTE FUNCTION research_reject_attachment()")
|
||||
try:c.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=commit_fail.png',503,actor=a,raw=original,headers={'Content-Type':'image/png'})
|
||||
finally:sql('DROP TRIGGER research_fail ON attachments; DROP FUNCTION research_reject_attachment()')
|
||||
report['after_pg_insert_fault']={'intents':int(sql('SELECT count(*) FROM upload_intents')),'metadata_committed':int(sql("SELECT count(*) FROM attachments WHERE filename='commit_fail.png'"))}
|
||||
check(report['after_pg_insert_fault']['intents']>=1 and report['after_pg_insert_fault']['metadata_committed']==0,'journal missing after DB fault')
|
||||
# Interrupted client body: close a real LB socket after part of the PNG.
|
||||
from urllib.parse import urlsplit
|
||||
target=urlsplit(BASE);s=socket.create_connection((target.hostname,target.port));head=('POST /api/v1/chats/'+chat['id']+'/attachments?filename=interrupted.png HTTP/1.1\r\nHost: localhost\r\nAuthorization: Bearer '+a['token']+'\r\nContent-Type: image/png\r\nContent-Length: '+str(len(original))+'\r\n\r\n').encode();s.sendall(head+original[:30]);s.close();time.sleep(1)
|
||||
report['interrupted_upload_metadata']=int(sql("SELECT count(*) FROM attachments WHERE filename='interrupted.png'"));check(report['interrupted_upload_metadata']==0,'interrupted metadata')
|
||||
# Test anonymous access to the exact existing private object, from the network.
|
||||
cmd='wget -S -O /dev/null http://s3:9000/messenger-attachments/'+key(used)
|
||||
denied=subprocess.run(['docker','compose','exec','-T','api','sh','-c',cmd],cwd=ROOT,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,text=True)
|
||||
report['private_bucket_anonymous_http_403']='403 Forbidden' in denied.stdout;check(report['private_bucket_anonymous_http_403'],'bucket is public')
|
||||
# S3 outage cannot produce 201. LB can synthesize its own 503 while all
|
||||
# readiness probes fail, so inspect raw status instead of API-only headers.
|
||||
dc('stop','s3');start=time.monotonic()
|
||||
try:
|
||||
req=urllib.request.Request(BASE+'/api/v1/chats/'+chat['id']+'/attachments?filename=down.png',data=original,method='POST',headers={'Authorization':'Bearer '+a['token'],'Content-Type':'image/png'})
|
||||
try:status=urllib.request.urlopen(req,timeout=5).status
|
||||
except urllib.error.HTTPError as ex:status=ex.code
|
||||
report['s3_outage']={'status':status,'seconds':time.monotonic()-start};check(status==503,'S3 down accepted upload')
|
||||
finally:dc('start','s3');dc('up','-d','--wait');time.sleep(2)
|
||||
print('Fault fixtures created; waiting for 60s grace without overriding timestamps',flush=True)
|
||||
# All old fixtures predate this wait. Fresh objects are created afterwards.
|
||||
time.sleep(62)
|
||||
fresh=upload('fresh.png');fresh_orphan='originals/'+str(uuid.uuid4());mc('mc cp /config/s3-policy.json local/messenger-attachments/'+fresh_orphan+' >/dev/null')
|
||||
report['dry_run']=reconcile();report['apply']=reconcile(True);report['rerun']=reconcile(True)
|
||||
check(report['dry_run']['orphans']>=2 and report['dry_run']['missing']>=2,'fault detection incomplete')
|
||||
check(report['apply']['deleted_objects']>=2 and report['apply']['deleted_metadata']>=1,'repair incomplete')
|
||||
check(report['apply']['referenced_protected']>=1 and report['apply']['fresh_protected']>=2,'unsafe repair')
|
||||
check(report['rerun']['deleted_objects']==0 and report['rerun']['deleted_metadata']==0 and report['rerun']['cleared_intents']==0,'not idempotent')
|
||||
for att in [used,fresh]:
|
||||
data=c.call('GET','/api/v1/attachments/'+att['id']+'/content',actor=b)['bytes'];check(hashlib.sha256(data).hexdigest()==att['sha256'],'valid original removed')
|
||||
check(sql("SELECT count(*) FROM attachments WHERE id='"+missing_used['id']+"'")=='1','used metadata deleted')
|
||||
# Bytes cannot be reconstructed from a checksum: restoring the missing used
|
||||
# object requires the original, kept here only as the synthetic fixture.
|
||||
restore_key=key(missing_used)
|
||||
fixture_hex=original.hex()
|
||||
# Pass original through stdin to mc pipe (no auth in stdin or output).
|
||||
prefix='mc alias set local http://s3:9000 "$(cat /run/secrets/s3_root_user)" "$(cat /run/secrets/s3_root_password)" >/dev/null; mc pipe local/messenger-attachments/'+restore_key+' >/dev/null'
|
||||
subprocess.run(['docker','compose','run','--rm','--no-deps','-T','--entrypoint','/bin/sh','s3-init','-c',prefix],cwd=ROOT,input=original,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL,check=True)
|
||||
report['used_missing_restored_from_original']=c.call('GET','/api/v1/attachments/'+missing_used['id']+'/content',actor=b)['bytes']==original
|
||||
report['remaining_fresh_orphan_protected']=fresh_orphan
|
||||
(ROOT/'evidence/consistency.json').write_text(json.dumps(report,indent=2)+'\n');print('PASS: faults, private bucket, dry-run, safe repair, repeated repair and preservation')
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Measure dry-run inventory cost at +0/+100/+1000 synthetic private keys."""
|
||||
import json,subprocess,tempfile,uuid
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
|
||||
prefix='originals/cost-'+uuid.uuid4().hex
|
||||
setup='mc alias set local http://s3:9000 "$(cat /run/secrets/s3_root_user)" "$(cat /run/secrets/s3_root_password)" >/dev/null; '
|
||||
results=[]
|
||||
def measure(extra):
|
||||
raw=dc('run','--rm','--no-deps','api','reconcile')
|
||||
r=json.loads(raw.splitlines()[-1]);r['extra_synthetic_objects']=extra;results.append(r)
|
||||
with tempfile.TemporaryDirectory(prefix='lab4-inventory-') as folder:
|
||||
try:
|
||||
measure(0)
|
||||
for count in [100,1000]:
|
||||
for i in range(count):(Path(folder)/str(i)).write_text('synthetic-inventory-only\n')
|
||||
dc('run','--rm','--no-deps','--volume',folder+':/probe:ro','--entrypoint','/bin/sh','s3-init','-c',setup+'mc mirror --quiet /probe local/messenger-attachments/'+prefix+' >/dev/null')
|
||||
measure(count)
|
||||
finally:
|
||||
dc('run','--rm','--no-deps','--entrypoint','/bin/sh','s3-init','-c',setup+'mc rm --recursive --force local/messenger-attachments/'+prefix+' >/dev/null')
|
||||
assert all(r['deleted_objects']==0 and r['deleted_metadata']==0 for r in results)
|
||||
(ROOT/'evidence/inventory-cost.json').write_text(json.dumps({'method':'same existing dataset plus 0/100/1000 fresh synthetic keys; dry-run; CLI API limits; no metadata mutation; temporary prefix removed','runs':results},indent=2)+'\n')
|
||||
print('PASS: dry-run costs measured at +0/+100/+1000 keys, probe prefix removed')
|
||||
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stop each real container, then replace them in sequence under writes."""
|
||||
import json,os,subprocess,sys,threading,time,uuid
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0,str(ROOT/'tests'))
|
||||
from client import Client,Failure,check
|
||||
BASE=os.environ.get('BASE_URL','http://localhost:8084')
|
||||
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
|
||||
def inspect(id):return json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]
|
||||
def ready():
|
||||
deadline=time.monotonic()+45
|
||||
while time.monotonic()<deadline:
|
||||
try:
|
||||
dc('up','-d','--no-deps','--wait','api');time.sleep(2);return
|
||||
except subprocess.CalledProcessError:time.sleep(1)
|
||||
raise AssertionError('API did not recover')
|
||||
def run_phase(c,actor,chat,action,seconds=7):
|
||||
results=[];failed=0;stop=threading.Event();events=[];seen=set()
|
||||
def write():
|
||||
nonlocal failed
|
||||
client=Client(BASE,timeout=2);key=uuid.uuid4().hex
|
||||
while not stop.is_set():
|
||||
try:
|
||||
response=client.message(actor,chat,'operation probe '+key,key=key)
|
||||
results.append((time.monotonic(),response['json']['id']))
|
||||
seen.add(response['headers']['X-Instance-Id']);key=uuid.uuid4().hex
|
||||
except Failure:failed+=1
|
||||
time.sleep(.02)
|
||||
worker=threading.Thread(target=write);worker.start();time.sleep(1)
|
||||
started=time.monotonic();action();action_done=time.monotonic();time.sleep(seconds);stop.set();worker.join()
|
||||
history=[];path='/api/v1/chats/'+chat['id']+'/messages?limit=100'
|
||||
from urllib.parse import quote
|
||||
while True:
|
||||
page=c.call('GET',path,actor=actor,schema='MessagePage')['json'];history+=page['items']
|
||||
if page['next_cursor'] is None:break
|
||||
path='/api/v1/chats/'+chat['id']+'/messages?limit=100&cursor='+quote(page['next_cursor'],safe='')
|
||||
ids=[m['id'] for m in history];check(len(ids)==len(set(ids)),'duplicate IDs')
|
||||
check({i for _,i in results}.issubset(set(ids)),'lost confirmed message')
|
||||
successes_after=[t for t,_ in results if t>=action_done]
|
||||
check(successes_after,'no successful write after action')
|
||||
recovery=successes_after[0]-started
|
||||
return {'successful_writes':len(results),'errors':failed,'successful_rps':len(results)/(time.monotonic()-started+1),'first_recovery_seconds':round(recovery,3),'action_seconds':round(action_done-started,3),'instances':sorted(seen),'confirmed_preserved':True}
|
||||
def main():
|
||||
ready();c=Client(BASE);a,b,e,chat=c.bootstrap();results={'failures':[]}
|
||||
ids=dc('ps','-q','api').splitlines();check(len(ids)==2,'requires two API containers')
|
||||
for id in ids:
|
||||
alias=inspect(id)['Config']['Hostname']
|
||||
env=dict(os.environ,TARGET_INSTANCE_ID=alias)
|
||||
def action():subprocess.run([str(ROOT/'scripts/stop-one.sh')],env=env,check=True,stdout=subprocess.DEVNULL)
|
||||
try:
|
||||
result=run_phase(c,a,chat,action);result['stopped_alias']=alias
|
||||
check(not inspect(id)['State']['Running'],'target is still running')
|
||||
check(result['first_recovery_seconds']<10,'recovery >10s')
|
||||
results['failures'].append(result)
|
||||
finally:ready()
|
||||
results['rolling']=run_phase(c,a,chat,lambda:subprocess.run([str(ROOT/'scripts/rolling.sh')],check=True,stdout=subprocess.DEVNULL),seconds=2)
|
||||
check(len(dc('ps','-q','api').splitlines())==2,'rolling lost replicas')
|
||||
(ROOT/'evidence/operations.json').write_text(json.dumps(results,indent=2)+'\n')
|
||||
print('PASS: each API stopped and rolling replacement completed; confirmed writes preserved')
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,16 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prove the app S3 identity cannot manage users or modify bucket policy."""
|
||||
import json,subprocess
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
prefix='mc alias set app http://s3:9000 "$(cat /run/secrets/s3_access_key_id)" "$(cat /run/secrets/s3_secret_access_key)" >/dev/null; '
|
||||
checks={}
|
||||
try:
|
||||
for name,command in [('admin_denied','mc admin info app'),('policy_change_denied','mc anonymous set download app/messenger-attachments')]:
|
||||
r=subprocess.run(['docker','compose','run','--rm','--no-deps','--entrypoint','/bin/sh','s3-init','-c',prefix+command],cwd=ROOT,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,text=True)
|
||||
checks[name]=r.returncode!=0 and ('denied' in r.stdout.lower() or 'not allowed' in r.stdout.lower())
|
||||
assert checks[name],name+' was unexpectedly permitted'
|
||||
finally:
|
||||
# Always re-apply private policy with the bootstrap identity, including failure.
|
||||
subprocess.run(['docker','compose','run','--rm','--no-deps','s3-init'],cwd=ROOT,check=True,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
|
||||
(ROOT/'evidence/permissions.json').write_text(json.dumps(checks,indent=2)+'\n');print('PASS: S3 application identity denied admin and bucket-policy changes')
|
||||
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Small sanitized runtime manifest; no environment, passwords or tokens."""
|
||||
import json,platform,subprocess
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
ids=subprocess.check_output(['docker','compose','ps','-q'],cwd=ROOT,text=True).splitlines()
|
||||
result={'platform':platform.platform(),'containers':[]}
|
||||
for id in ids:
|
||||
item=json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]
|
||||
result['containers'].append({'service':item['Config']['Labels']['com.docker.compose.service'],'name':item['Name'],'alias':item['Config']['Hostname'],'image':item['Config']['Image'],'image_id':item['Image'],'user':item['Config']['User'],'pid1_path':item['Path'],'started_at':item['State']['StartedAt'],'memory_limit_bytes':item['HostConfig']['Memory'],'nano_cpus_limit':item['HostConfig']['NanoCpus'],'published_ports':item['NetworkSettings']['Ports'],'health':item['State'].get('Health',{}).get('Status'),'mount_targets':[m['Destination'] for m in item['Mounts']]})
|
||||
result['postgres_durability']=subprocess.check_output(['docker','compose','exec','-T','postgres','psql','-U','messenger','-d','messenger','-Atqc',"SELECT name||'='||setting FROM pg_settings WHERE name IN ('fsync','synchronous_commit','full_page_writes','max_connections') ORDER BY name"],cwd=ROOT,text=True).splitlines()
|
||||
(ROOT/'evidence/runtime.json').write_text(json.dumps(result,indent=2)+'\n')
|
||||
print('Saved sanitized runtime manifest: resources, images, PID1, ports, durability')
|
||||
Reference in New Issue
Block a user