forked from mtusi2026/mtusi2026lab4
4: исправление по ревью — проверка смысла чанков PNG (палитра, tRNS, порядок, текст), регрессионные тесты, evidence
This commit is contained in:
@@ -91,7 +91,7 @@ s3-init: одноразово создаёт приватный bucket и пр
|
||||
3. Тип по сигнатуре (`\x89PNG...`, `FF D8 FF`), не по заголовку. Не PNG/JPEG -> 415; заявленный тип не совпадает с содержимым -> 415.
|
||||
4. Размер в пикселях читается из заголовка (IHDR / SOF) и проверяется ДО распаковки: больше 40 000 000 -> 413.
|
||||
5. Проверка целостности с ограниченной памятью:
|
||||
- PNG — своя потоковая проверка (`uploads.check_png`): CRC каждого чанка, порядок IHDR/PLTE/IDAT/IEND, допустимые тип цвета и глубина, ничего после IEND; IDAT распаковывается zlib кусками по 64 KiB (вывод не хранится), считается точный объём данных по IHDR (с учётом Adam7) и проверяется байт фильтра каждой строки. «Zip-бомба» не раздувает память: распаковка идёт порциями и обрывается, как только данных больше, чем обещал заголовок.
|
||||
- PNG — своя потоковая проверка (`uploads.check_png`): CRC каждого чанка и смысл критичных чанков по спецификации PNG (размер палитры 1–256 записей и не больше 2^глубина, размер tRNS по типу цвета, пустой IEND, порядок чанков: всё, что обязано идти до IDAT, — до него, IDAT подряд, неизвестный критичный чанк запрещён); IDAT распаковывается кусками по 64 KiB с подсчётом точного объёма и проверкой байта фильтра каждой строки. Затем Pillow разбирает метаданные до IDAT (Image.open, без verify: verify читает каждый чанк, включая IDAT до 10 MiB, в память). После IDAT допускаются только tIME, tEXt, zTXt, iTXt; текстовые чанки проверяются потоково (ключ 1–79 байт, корректный zlib, не больше 1 MiB распакованного текста); анимированные PNG (acTL/fcTL/fdAT) отклоняются; чанк метаданных до IDAT — не больше 2 MiB. Исправлено по замечанию: раньше проверялись только CRC, и PNG с палитрой из 257 записей получал 201.
|
||||
- JPEG — Pillow декодирует весь поток, но в масштабе 1/8 (`draft`): 40 Мпикс -> ~0,6 Мпикс, около 2 MiB.
|
||||
- Одновременно в реплике проверяется не больше `DECODE_CONCURRENCY=2` файлов (CPU).
|
||||
Битый файл -> 415. Любая ошибка разбора ловится и превращается в 415, а не 500. Проверка PNG сверена с Pillow: 400 случайных корректных PNG разных типов приняты, 400 испорченных (обрезка, перевёрнутый бит) отклонены.
|
||||
@@ -187,36 +187,36 @@ S3-клиент: connect timeout 1 с, read timeout 5 с, всего 2 попы
|
||||
Методика памяти (`student-tests/upload_memory.py`): три сценария — 10 x PNG из случайного шума ~9,5 MiB (почти несжимаемый, максимум трафика); 10 x PNG 8000x5000 RGB (40 Мпикс, максимум распаковки); 20 смешанных. Перед каждым сценарием `docker compose restart api` (сбрасывает пик), прогрев 8 загрузками, затем все потоки стартуют одновременно (barrier) через балансировщик. Пик RSS — `VmHWM` из `/proc/1/status` процесса API (максимум за жизнь процесса по данным ядра, короткие всплески не теряются), дополнительно пик памяти cgroup контейнера и опрос раз в ~0,1–0,5 с. Проверяются коды 201 и SHA-256 скачанных файлов. Клиент — тот же компьютер, поэтому время загрузки ограничено и клиентом; это замер памяти, а не пропускной способности.
|
||||
|
||||
<!-- measurements:begin -->
|
||||
Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-21, commit 2dbed80 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB.
|
||||
Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-22, commit c942c61 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB.
|
||||
|
||||
Публичный тест отказа (`make test-failover`, 15 с записи во время остановки реплики):
|
||||
|
||||
| Прогон | Остановлена реплика | Реплик до отказа | Успешных записей | Неудачных попыток (повторены с тем же ключом) | Первая запись после hook, с |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | 11a93acecaca | 2 | 126 | 0 | 4.39 |
|
||||
| 2 | 6128eb268c44 | 2 | 124 | 0 | 4.328 |
|
||||
| 1 | 343119df2488 | 2 | 123 | 0 | 4.422 |
|
||||
| 2 | d590adc21943 | 2 | 124 | 0 | 4.531 |
|
||||
|
||||
Одновременные загрузки (`student-tests/upload_memory.py`, перед каждым сценарием реплики перезапускаются, лимит api 512 MiB / 1 CPU). Пик RSS — VmHWM процесса API; cgroup — память контейнера, включая page cache временных файлов:
|
||||
|
||||
| Сценарий | Одновременно | Объём, MiB | Время, с (p50 / max) | Реплика: загрузок | RSS до, MiB | Пик RSS, MiB | Рост, MiB | Пик cgroup, MiB |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 1.38 / 1.52 | 11a93acecaca: 5 | 93.5 | 106.2 | 12.6 | 134.1 |
|
||||
| | | | | 6128eb268c44: 5 | 94.8 | 105.3 | 10.5 | 135.2 |
|
||||
| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 11a93acecaca: 5 | 93.2 | 96.6 | 3.4 | 84.3 |
|
||||
| | | | | 6128eb268c44: 5 | 93.1 | 94.7 | 1.6 | 86.8 |
|
||||
| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.37 / 1.77 | 11a93acecaca: 10 | 94.8 | 121.7 | 26.9 | 181.9 |
|
||||
| | | | | 6128eb268c44: 10 | 95.0 | 116.1 | 21.0 | 155.6 |
|
||||
| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 2.87 / 3.52 | 343119df2488: 5 | 94.8 | 107.6 | 12.8 | 136.0 |
|
||||
| | | | | d590adc21943: 5 | 91.2 | 104.5 | 13.3 | 136.2 |
|
||||
| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 343119df2488: 5 | 95.1 | 96.6 | 1.5 | 86.5 |
|
||||
| | | | | d590adc21943: 5 | 92.1 | 96.6 | 4.6 | 83.5 |
|
||||
| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.5 / 1.78 | 343119df2488: 10 | 93.2 | 116.2 | 23.0 | 158.2 |
|
||||
| | | | | d590adc21943: 10 | 95.0 | 118.6 | 23.6 | 167.8 |
|
||||
|
||||
Один PNG 8000x5000 RGB внутри контейнера api, дополнительный пик RSS: наивно (тело целиком в памяти + полное декодирование Pillow) 161 MiB, потоковая проверка 0 MiB. 10 таких наивных загрузок одновременно не поместились бы в 512 MiB.
|
||||
|
||||
Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 245.844 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс.
|
||||
Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 239.325 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.39, 4.328 с (hook ждёт остановки ~4 с); потерь нет |
|
||||
| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 39.2 с до готовности; потерь нет |
|
||||
| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 20141 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет |
|
||||
| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14766 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет |
|
||||
| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.422, 4.531 с (hook ждёт остановки ~4 с); потерь нет |
|
||||
| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 47.0 с до готовности; потерь нет |
|
||||
| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 19453 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет |
|
||||
| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14891 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет |
|
||||
| падение между PUT в S3 и COMMIT в БД (код API в контейнере, finish_upload падает) | вложение не видно, janitor не трогает его до конца grace period, потом удаляет объект и строку | GET/прикрепление -> 404; janitor: свежая строка сохранена, dry-run только отчёт, затем удалено | мусора не осталось; повторный проход ничего не делает |
|
||||
| обрыв загрузки клиентом (9 МБ заявлено, отправлено 0,2 и 3 МБ) | ничего не сохраняется | строк и объектов не прибавилось, временных файлов 0 | мгновенно |
|
||||
<!-- measurements:end -->
|
||||
@@ -231,5 +231,6 @@ S3-клиент: connect timeout 1 с, read timeout 5 с, всего 2 попы
|
||||
- Обход bucket janitor’ом держит в памяти список ключей (для учебного объёма — мегабайты); для миллионов объектов нужна сверка по диапазонам ключей или S3 Inventory.
|
||||
- Временный файл загрузки (часть тела сверх 256 KiB) лежит на диске контейнера. В Linux `tempfile.TemporaryFile` создаётся уже без имени в каталоге (O_TMPFILE), поэтому даже при SIGKILL реплики место освобождается ядром, «забытых» файлов не бывает. Одновременно на диске может быть до 10 MiB на каждую идущую загрузку; лимита на число одновременных загрузок, кроме лимитов HAProxy/uvicorn, нет.
|
||||
- Проверяется структура изображения, а не его безопасность для всех декодеров (например, JPEG с некорректными, но декодируемыми данными пройдёт). Перекодирование оригинала — задача lab5.
|
||||
- PNG с частными служебными чанками после IDAT и анимированные PNG отклоняются (строже спецификации, на практике не встречаются в фотографиях).
|
||||
- MinIO собирался быть первым выбором, но его образы больше не доступны в публичных реестрах: закреплённый тег защищает от неожиданного обновления, но не от исчезновения образа. Для эксплуатации нужно зеркалить образы в свой реестр. SeaweedFS запущен одним процессом `weed server` на одном диске — это SPOF и не защита от потери диска; веб-интерфейсы master/filer без авторизации, закрыты только сетью.
|
||||
- Скрипты `failover_metrics.py`, `rolling_update.py`, `test_expiry.py` из lab3 остались в `student-tests/` и работают со стендом, но в общий прогон lab4 не входят (это уровень 4 lab3, а не lab4).
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# attachments: lab4: files, limits, damaged images, MIME, filenames, access, attachment_ids
|
||||
# date: 2026-10-21 21:03:05, commit: 2dbed80
|
||||
# date: 2026-10-22 14:19:26, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/test_attachments.py
|
||||
# exit code: 0, 4.0 s
|
||||
# exit code: 0, 5.4 s
|
||||
|
||||
sample.png: 111817 bytes, sha256 99ea7533c441..., downloaded by the other member, same bytes
|
||||
sample.jpg: 1492 bytes, sha256 c310b9a1c208..., downloaded by the other member, same bytes
|
||||
@@ -14,7 +14,7 @@ exactly 10485760 bytes (10 MiB) -> 201
|
||||
PASS: the 10 MiB limit is exact
|
||||
12 MiB without Content-Length (chunked) -> 413 as soon as the 10 MiB limit is passed
|
||||
PASS: oversized streamed body is cut off, the API keeps working
|
||||
8000x5000 = 40 000 000 pixels (38957 bytes compressed) -> 201 5ffe6847
|
||||
8000x5000 = 40 000 000 pixels (38957 bytes compressed) -> 201 af872a80
|
||||
8001x5000 = 40 005 000 pixels (38962 bytes) -> 413
|
||||
header claims 100000x100000 in 65 bytes -> 413 from the header, nothing is decoded
|
||||
PASS: pixel limit is checked before decompression
|
||||
@@ -30,6 +30,35 @@ random bytes (1600 bytes, image/png) -> 415
|
||||
plain text (28 bytes, image/jpeg) -> 415
|
||||
empty body -> 400
|
||||
PASS: damaged and foreign files are rejected with 415, never 500
|
||||
PLTE 771 bytes = 257 entries (review case) (850 bytes, valid CRCs) -> 415
|
||||
PLTE length not a multiple of 3 (83 bytes, valid CRCs) -> 415
|
||||
empty PLTE (79 bytes, valid CRCs) -> 415
|
||||
palette larger than bit depth allows (depth 1, 3 entries) (88 bytes, valid CRCs) -> 415
|
||||
palette image without PLTE (67 bytes, valid CRCs) -> 415
|
||||
PLTE in a grayscale image (85 bytes, valid CRCs) -> 415
|
||||
PLTE after IDAT (85 bytes, valid CRCs) -> 415
|
||||
two PLTE chunks (103 bytes, valid CRCs) -> 415
|
||||
tRNS longer than the palette (100 bytes, valid CRCs) -> 415
|
||||
tRNS before PLTE (98 bytes, valid CRCs) -> 415
|
||||
tRNS of wrong size for RGB (4 bytes instead of 6) (85 bytes, valid CRCs) -> 415
|
||||
tRNS in an image with alpha channel (82 bytes, valid CRCs) -> 415
|
||||
IEND with data (70 bytes, valid CRCs) -> 415
|
||||
unknown critical chunk (81 bytes, valid CRCs) -> 415
|
||||
IDAT chunks not consecutive (95 bytes, valid CRCs) -> 415
|
||||
gAMA after IDAT (85 bytes, valid CRCs) -> 415
|
||||
gAMA of 2 bytes instead of 4 (83 bytes, valid CRCs) -> 415
|
||||
3 MiB metadata chunk before IDAT (3145809 bytes, valid CRCs) -> 415
|
||||
private chunk after IDAT (181 bytes, valid CRCs) -> 415
|
||||
animated PNG frame (fdAT) (91 bytes, valid CRCs) -> 415
|
||||
tIME of 2 bytes instead of 7 (83 bytes, valid CRCs) -> 415
|
||||
tEXt without keyword separator (86 bytes, valid CRCs) -> 415
|
||||
zTXt with broken zlib after IDAT (91 bytes, valid CRCs) -> 415
|
||||
zTXt text bomb: 50 MB of text in ~50 KB after IDAT (48694 bytes, valid CRCs) -> 415
|
||||
palette 256 entries + tRNS + tEXt and tIME after IDAT -> 201, downloaded byte for byte
|
||||
palette depth 2 with 4 entries -> 201, downloaded byte for byte
|
||||
RGB with 6-byte tRNS and gAMA -> 201, downloaded byte for byte
|
||||
compressed zTXt (100 KB of text) and iTXt after IDAT -> 201, downloaded byte for byte
|
||||
PASS: chunk contents are checked, not only CRCs; valid PNGs with metadata are still accepted
|
||||
PNG bytes sent as image/jpeg -> 415
|
||||
JPEG bytes sent as image/png -> 415
|
||||
GIF bytes sent as image/png -> 415
|
||||
@@ -64,4 +93,4 @@ PASS: Idempotency-Key covers attachment_ids
|
||||
sent 1.1 MB of a declared 5 MB body and closed the connection
|
||||
the API keeps serving uploads; nothing of the aborted request is visible (DB/S3 side is checked by test_consistency.py)
|
||||
PASS: an interrupted upload is dropped
|
||||
11 passed, 0 failed
|
||||
12 passed, 0 failed
|
||||
|
||||
@@ -1,18 +1,18 @@
|
||||
# balancing: balancing, shared sessions/data/files, X-Instance-Id
|
||||
# date: 2026-10-21 21:08:50, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:22:27, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/test_balancing.py
|
||||
# exit code: 0, 1.9 s
|
||||
# exit code: 0, 2.0 s
|
||||
|
||||
20 x GET /users/me with one token -> replicas: {'6128eb268c44': 10, '11a93acecaca': 10}
|
||||
20 x GET /users/me with one token -> replicas: {'d590adc21943': 10, '343119df2488': 10}
|
||||
PASS: one session works on every replica, no second login (sessions are shared in Redis)
|
||||
message written by 11a93acecaca; read back by ['11a93acecaca', '6128eb268c44']
|
||||
message written by 343119df2488; read back by ['343119df2488', 'd590adc21943']
|
||||
PASS: a message written via one replica is read right away via another (shared PostgreSQL)
|
||||
logout on 6128eb268c44; old token then rejected by ['11a93acecaca', '6128eb268c44']
|
||||
logout on d590adc21943; old token then rejected by ['343119df2488', 'd590adc21943']
|
||||
PASS: logout on one replica revokes the session on all of them
|
||||
first POST on 6128eb268c44 -> 201 296cb1bf-8c1c-46f2-942a-4865d1b7d6af; same key on 11a93acecaca -> 201 296cb1bf-8c1c-46f2-942a-4865d1b7d6af
|
||||
first POST on d590adc21943 -> 201 6c2f624d-c728-4738-b486-b17ab735c0ae; same key on 343119df2488 -> 201 6c2f624d-c728-4738-b486-b17ab735c0ae
|
||||
PASS: Idempotency-Key is shared: a retry through another replica returns the same message
|
||||
file uploaded via 6128eb268c44; downloaded byte for byte via ['11a93acecaca', '6128eb268c44']
|
||||
file uploaded via d590adc21943; downloaded byte for byte via ['343119df2488', 'd590adc21943']
|
||||
PASS: a file uploaded through one replica is downloaded through another (shared S3, nothing local)
|
||||
statuses [200, 200, 201, 400, 401, 404, 409]: every answer has X-Instance-Id of a running container ['11a93acecaca', '6128eb268c44']; api containers: ['11a93acecaca', '6128eb268c44']
|
||||
statuses [200, 200, 201, 400, 401, 404, 409]: every answer has X-Instance-Id of a running container ['343119df2488', 'd590adc21943']; api containers: ['343119df2488', 'd590adc21943']
|
||||
PASS: X-Instance-Id is in every response and points to a real container
|
||||
6 passed, 0 failed
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
# check: offline contract check (make check)
|
||||
# date: 2026-10-21 21:02:57, commit: 2dbed80
|
||||
# date: 2026-10-22 14:19:16, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe scripts/check_contract.py
|
||||
# exit code: 0, 0.1 s
|
||||
|
||||
|
||||
+18
-18
@@ -1,47 +1,47 @@
|
||||
# consistency: lab4: aborted upload, crash between S3 and DB, janitor, orphans, S3 outage
|
||||
# date: 2026-10-21 21:08:14, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:21:48, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/test_consistency.py
|
||||
# exit code: 0, 148.7 s
|
||||
# exit code: 0, 133.0 s
|
||||
|
||||
declared 9 000 000 bytes, sent 211817, closed the connection
|
||||
declared 9 000 000 bytes, sent 3111817, closed the connection
|
||||
rows in attachments: 32 -> 32; objects in bucket: 31 -> 31
|
||||
temporary upload files in api containers: {'11a93acecaca': 0, '6128eb268c44': 0} (removed when the request ends)
|
||||
rows in attachments: 448 -> 448; objects in bucket: 444 -> 444
|
||||
temporary upload files in api containers: {'343119df2488': 0, 'd590adc21943': 0} (removed when the request ends)
|
||||
api containers have no volumes: the original is never stored locally
|
||||
PASS: an interrupted upload leaves no row, no object and no local file
|
||||
crash after S3 PUT: row 04aacbf6 state=uploading, object originals/219175ec1d5647f0aeaec84ee8243cb3 exists in S3
|
||||
crash after S3 PUT: row 9aa0bc8a state=uploading, object originals/8ae03455b55a4f798b7bc6248b450ef9 exists in S3
|
||||
the half-written attachment is invisible: GET -> 404, attach to a message -> 404
|
||||
janitor with default grace (3600 s) right away: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 33, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.101} -> fresh row and object are kept (the upload might still be running)
|
||||
row made 2 h old; janitor --dry-run: {"dry_run": true, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 0, "objects_scanned": 33, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.092} -> nothing deleted
|
||||
janitor: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 32, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.096} -> row and object deleted
|
||||
janitor again: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 32, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.103} -> idempotent
|
||||
janitor with default grace (3600 s) right away: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 446, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.133} -> fresh row and object are kept (the upload might still be running)
|
||||
row made 2 h old; janitor --dry-run: {"dry_run": true, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 0, "objects_scanned": 446, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.184} -> nothing deleted
|
||||
janitor: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 445, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.132} -> row and object deleted
|
||||
janitor again: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 445, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.179} -> idempotent
|
||||
the ready attachment of the same chat is untouched
|
||||
PASS: S3 written but DB not committed -> invisible, reclaimed by janitor only after the grace period
|
||||
object originals/867f5c5d42bf4bf9973b60ef8856d88f put into the bucket without a row in PostgreSQL
|
||||
janitor --grace-seconds 3 immediately: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 34, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.102} -> fresh object kept
|
||||
after 4 s, --dry-run: {"dry_run": true, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 34, "orphan_objects": 1, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.098} -> reported, kept
|
||||
janitor: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 34, "orphan_objects": 1, "orphan_objects_deleted": 1, "missing_objects": 1, "seconds": 0.1} -> orphan deleted
|
||||
object originals/101748e3a911439a9c1e11e768c99822 put into the bucket without a row in PostgreSQL
|
||||
janitor --grace-seconds 3 immediately: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 447, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.191} -> fresh object kept
|
||||
after 4 s, --dry-run: {"dry_run": true, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 447, "orphan_objects": 1, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.188} -> reported, kept
|
||||
janitor: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 447, "orphan_objects": 1, "orphan_objects_deleted": 1, "missing_objects": 4, "seconds": 0.1} -> orphan deleted
|
||||
ready attachments older than the grace period are untouched (they have rows)
|
||||
object of a ready attachment deleted behind the API's back: janitor reports missing_objects=2 and keeps the row; download -> 503 (not 500)
|
||||
object of a ready attachment deleted behind the API's back: janitor reports missing_objects=5 and keeps the row; download -> 503 (not 500)
|
||||
PASS: orphans are found and deleted only after the grace period; missing objects are reported
|
||||
--- S3 stopped (connection refused / name not resolved): docker compose stop s3
|
||||
upload -> 503 unavailable in 20141 ms, no storage details in the body
|
||||
upload -> 503 unavailable in 19453 ms, no storage details in the body
|
||||
download of an existing file -> 503
|
||||
metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes
|
||||
uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API
|
||||
docker compose start s3
|
||||
after S3 recovery the old file downloads byte for byte
|
||||
new upload -> 201
|
||||
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 35, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.099}
|
||||
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 448, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 5, "seconds": 0.191}
|
||||
PASS: S3 stopped (connection refused / name not resolved) -> 503 for file operations only, no visible half-uploads, everything cleaned up
|
||||
--- S3 frozen (no answer, timeouts): docker compose pause s3
|
||||
upload -> 503 unavailable in 14766 ms, no storage details in the body
|
||||
upload -> 503 unavailable in 14891 ms, no storage details in the body
|
||||
download of an existing file -> 503
|
||||
metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes
|
||||
uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API
|
||||
docker compose unpause s3
|
||||
after S3 recovery the old file downloads byte for byte
|
||||
new upload -> 201
|
||||
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 37, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.094}
|
||||
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 450, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 5, "seconds": 0.115}
|
||||
PASS: S3 frozen (no answer, timeouts) -> 503 for file operations only, no visible half-uploads, everything cleaned up
|
||||
5 passed, 0 failed
|
||||
|
||||
+4
-4
@@ -1,9 +1,9 @@
|
||||
# db: PostgreSQL: rollback, FK, Argon2id, unique keys, migrations
|
||||
# date: 2026-10-21 21:09:10, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:22:40, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/test_db.py
|
||||
# exit code: 0, 14.2 s
|
||||
# exit code: 0, 6.2 s
|
||||
|
||||
chat with unknown member -> 400; rows with this title: 0; chats without members: 0; chats insert counter: 57 -> 58
|
||||
chat with unknown member -> 400; rows with this title: 0; chats without members: 0; chats insert counter: 335 -> 336
|
||||
PASS: chat row was inserted and then rolled back together with members
|
||||
direct INSERT of a message from a non-member -> exit code 1: ERROR: insert or update on table "messages" violates foreign key constraint "messages_chat_id_sender_id_fkey"
|
||||
PASS: the database itself rejects a message from a non-member
|
||||
@@ -14,7 +14,7 @@ PASS: username is unique in the database
|
||||
applied migrations: 001_init.sql,002_text_as_bytes.sql,003_idempotency.sql,004_attachments.sql; running the migrate service once more
|
||||
after the second run: 001_init.sql,002_text_as_bytes.sql,003_idempotency.sql,004_attachments.sql; log: Migrations applied: none (schema is up to date)
|
||||
PASS: migrations are applied once, a repeated run does nothing
|
||||
api replicas that checked the schema at start: 8; api log lines 'Migrations applied': 0; migrate service: Exited (0) 6 minutes ago
|
||||
api replicas that checked the schema at start: 14; api log lines 'Migrations applied': 0; migrate service: Exited (0) 4 minutes ago
|
||||
PASS: only the migrate service applies migrations, replicas wait for the schema
|
||||
unique constraint for Idempotency-Key: UNIQUE (chat_id, sender_id, idempotency_key)
|
||||
PASS: one message per (chat, sender, Idempotency-Key) is enforced by the database
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
=== Data volume
|
||||
messages_total | messages_in_big_chat
|
||||
----------------+----------------------
|
||||
20146 | 15000
|
||||
35618 | 15000
|
||||
|
||||
|
||||
=== History index
|
||||
@@ -13,39 +13,39 @@
|
||||
=== 1. First page: limit=50, the app asks for 51 rows to know if there is a next page
|
||||
QUERY PLAN
|
||||
------------------------------------------------------------------------------------------------
|
||||
Limit (actual time=0.005..0.011 rows=51 loops=1)
|
||||
Buffers: shared hit=5
|
||||
Limit (actual time=0.006..0.010 rows=51 loops=1)
|
||||
Buffers: shared hit=4
|
||||
-> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1)
|
||||
Index Cond: (chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid)
|
||||
Buffers: shared hit=5
|
||||
Index Cond: (chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid)
|
||||
Buffers: shared hit=4
|
||||
Planning:
|
||||
Buffers: shared hit=6
|
||||
Planning Time: 0.060 ms
|
||||
Buffers: shared hit=8
|
||||
Planning Time: 0.071 ms
|
||||
Execution Time: 0.020 ms
|
||||
|
||||
|
||||
=== 2. Page after the 10000th message by keyset cursor (this is what the app does)
|
||||
QUERY PLAN
|
||||
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
||||
Limit (actual time=0.005..0.012 rows=51 loops=1)
|
||||
Buffers: shared hit=6
|
||||
-> Index Scan using messages_history on messages (actual time=0.005..0.009 rows=51 loops=1)
|
||||
Index Cond: ((chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '170ea586-3f7e-4314-a9bf-8e4f394941ac'::uuid)))
|
||||
Buffers: shared hit=6
|
||||
Planning Time: 0.044 ms
|
||||
Execution Time: 0.031 ms
|
||||
Limit (actual time=0.006..0.011 rows=51 loops=1)
|
||||
Buffers: shared hit=5
|
||||
-> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1)
|
||||
Index Cond: ((chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '4bd22867-547f-4ecf-8182-443fbb004188'::uuid)))
|
||||
Buffers: shared hit=5
|
||||
Planning Time: 0.040 ms
|
||||
Execution Time: 0.029 ms
|
||||
|
||||
|
||||
=== 2b. The same page exactly as the app sends it since lab4: with attachment_ids of every message
|
||||
QUERY PLAN
|
||||
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
||||
Limit (actual time=0.020..0.225 rows=51 loops=1)
|
||||
Buffers: shared hit=60
|
||||
-> Index Scan using messages_history on messages m (actual time=0.019..0.222 rows=51 loops=1)
|
||||
Index Cond: ((chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '170ea586-3f7e-4314-a9bf-8e4f394941ac'::uuid)))
|
||||
Buffers: shared hit=60
|
||||
Limit (actual time=0.019..0.208 rows=51 loops=1)
|
||||
Buffers: shared hit=59
|
||||
-> Index Scan using messages_history on messages m (actual time=0.019..0.204 rows=51 loops=1)
|
||||
Index Cond: ((chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '4bd22867-547f-4ecf-8182-443fbb004188'::uuid)))
|
||||
Buffers: shared hit=59
|
||||
SubPlan 1
|
||||
-> Sort (actual time=0.003..0.003 rows=0 loops=51)
|
||||
-> Sort (actual time=0.002..0.002 rows=0 loops=51)
|
||||
Sort Key: a."position"
|
||||
Sort Method: quicksort Memory: 25kB
|
||||
Buffers: shared hit=54
|
||||
@@ -57,39 +57,44 @@
|
||||
Buffers: shared hit=51
|
||||
Planning:
|
||||
Buffers: shared hit=63
|
||||
Planning Time: 0.157 ms
|
||||
Execution Time: 0.242 ms
|
||||
Planning Time: 0.156 ms
|
||||
Execution Time: 0.225 ms
|
||||
|
||||
|
||||
=== 3. The same page with OFFSET 10000 (the app does NOT do this)
|
||||
QUERY PLAN
|
||||
---------------------------------------------------------------------------------------------------
|
||||
Limit (actual time=0.878..0.884 rows=51 loops=1)
|
||||
Buffers: shared hit=258
|
||||
-> Index Scan using messages_history on messages (actual time=0.007..0.669 rows=10051 loops=1)
|
||||
Index Cond: (chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid)
|
||||
Buffers: shared hit=258
|
||||
Planning Time: 0.037 ms
|
||||
Execution Time: 0.897 ms
|
||||
QUERY PLAN
|
||||
--------------------------------------------------------------------------------
|
||||
Limit (actual time=3.056..3.060 rows=51 loops=1)
|
||||
Buffers: shared hit=585
|
||||
-> Sort (actual time=2.647..2.857 rows=10051 loops=1)
|
||||
Sort Key: created_at, id
|
||||
Sort Method: quicksort Memory: 2377kB
|
||||
Buffers: shared hit=585
|
||||
-> Seq Scan on messages (actual time=0.005..1.278 rows=15000 loops=1)
|
||||
Filter: (chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid)
|
||||
Rows Removed by Filter: 20618
|
||||
Buffers: shared hit=585
|
||||
Planning Time: 0.033 ms
|
||||
Execution Time: 3.107 ms
|
||||
|
||||
|
||||
=== 4. Query 2 without the index (index is dropped only inside this transaction)
|
||||
QUERY PLAN
|
||||
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
||||
Limit (actual time=1.097..1.101 rows=51 loops=1)
|
||||
Buffers: shared hit=261
|
||||
-> Sort (actual time=1.097..1.098 rows=51 loops=1)
|
||||
Limit (actual time=1.432..1.436 rows=51 loops=1)
|
||||
Buffers: shared hit=585
|
||||
-> Sort (actual time=1.432..1.433 rows=51 loops=1)
|
||||
Sort Key: created_at, id
|
||||
Sort Method: top-N heapsort Memory: 31kB
|
||||
Buffers: shared hit=261
|
||||
-> Seq Scan on messages (actual time=0.307..0.737 rows=5000 loops=1)
|
||||
Filter: ((chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '170ea586-3f7e-4314-a9bf-8e4f394941ac'::uuid)))
|
||||
Rows Removed by Filter: 15146
|
||||
Buffers: shared hit=261
|
||||
Buffers: shared hit=585
|
||||
-> Seq Scan on messages (actual time=0.678..1.116 rows=5000 loops=1)
|
||||
Filter: ((chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '4bd22867-547f-4ecf-8182-443fbb004188'::uuid)))
|
||||
Rows Removed by Filter: 30618
|
||||
Buffers: shared hit=585
|
||||
Planning:
|
||||
Buffers: shared hit=4
|
||||
Planning Time: 0.051 ms
|
||||
Execution Time: 1.115 ms
|
||||
Planning Time: 0.049 ms
|
||||
Execution Time: 1.450 ms
|
||||
|
||||
|
||||
=== ROLLBACK: test data removed, index is back
|
||||
|
||||
+22
-21
@@ -1,43 +1,44 @@
|
||||
# explain: EXPLAIN of the history page (full plans: explain-history.txt)
|
||||
# date: 2026-10-21 21:09:14, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:22:45, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/explain.py
|
||||
# exit code: 0, 1.1 s
|
||||
|
||||
=== Data volume
|
||||
=== History index
|
||||
=== 1. First page: limit=50, the app asks for 51 rows to know if there is a next page
|
||||
Buffers: shared hit=4
|
||||
-> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1)
|
||||
Buffers: shared hit=4
|
||||
Buffers: shared hit=8
|
||||
Execution Time: 0.020 ms
|
||||
=== 2. Page after the 10000th message by keyset cursor (this is what the app does)
|
||||
Buffers: shared hit=5
|
||||
-> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1)
|
||||
Buffers: shared hit=5
|
||||
Buffers: shared hit=6
|
||||
Execution Time: 0.020 ms
|
||||
=== 2. Page after the 10000th message by keyset cursor (this is what the app does)
|
||||
Buffers: shared hit=6
|
||||
-> Index Scan using messages_history on messages (actual time=0.005..0.009 rows=51 loops=1)
|
||||
Buffers: shared hit=6
|
||||
Execution Time: 0.031 ms
|
||||
Execution Time: 0.029 ms
|
||||
=== 2b. The same page exactly as the app sends it since lab4: with attachment_ids of every message
|
||||
Buffers: shared hit=60
|
||||
-> Index Scan using messages_history on messages m (actual time=0.019..0.222 rows=51 loops=1)
|
||||
Buffers: shared hit=60
|
||||
Buffers: shared hit=59
|
||||
-> Index Scan using messages_history on messages m (actual time=0.019..0.204 rows=51 loops=1)
|
||||
Buffers: shared hit=59
|
||||
Buffers: shared hit=54
|
||||
-> Bitmap Heap Scan on message_attachments a (actual time=0.000..0.000 rows=0 loops=51)
|
||||
Buffers: shared hit=51
|
||||
-> Bitmap Index Scan on message_attachments_message_id_attachment_id_key (actual time=0.000..0.000 rows=0 loops=51)
|
||||
Buffers: shared hit=51
|
||||
Buffers: shared hit=63
|
||||
Execution Time: 0.242 ms
|
||||
Execution Time: 0.225 ms
|
||||
=== 3. The same page with OFFSET 10000 (the app does NOT do this)
|
||||
Buffers: shared hit=258
|
||||
-> Index Scan using messages_history on messages (actual time=0.007..0.669 rows=10051 loops=1)
|
||||
Buffers: shared hit=258
|
||||
Execution Time: 0.897 ms
|
||||
Buffers: shared hit=585
|
||||
Buffers: shared hit=585
|
||||
-> Seq Scan on messages (actual time=0.005..1.278 rows=15000 loops=1)
|
||||
Buffers: shared hit=585
|
||||
Execution Time: 3.107 ms
|
||||
=== 4. Query 2 without the index (index is dropped only inside this transaction)
|
||||
Buffers: shared hit=261
|
||||
Buffers: shared hit=261
|
||||
-> Seq Scan on messages (actual time=0.307..0.737 rows=5000 loops=1)
|
||||
Buffers: shared hit=261
|
||||
Buffers: shared hit=585
|
||||
Buffers: shared hit=585
|
||||
-> Seq Scan on messages (actual time=0.678..1.116 rows=5000 loops=1)
|
||||
Buffers: shared hit=585
|
||||
Buffers: shared hit=4
|
||||
Execution Time: 1.115 ms
|
||||
Execution Time: 1.450 ms
|
||||
=== ROLLBACK: test data removed, index is back
|
||||
PASS: full plans saved to evidence\explain-history.txt
|
||||
|
||||
+17
-17
@@ -1,33 +1,33 @@
|
||||
# failover: public failover test for every replica, file downloaded via the other one (make test-failover)
|
||||
# date: 2026-10-21 21:09:54, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:23:28, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/run_failover.py
|
||||
# exit code: 0, 39.7 s
|
||||
# exit code: 0, 43.2 s
|
||||
|
||||
hook: scripts\stop-one.cmd; api replicas: ['11a93acecaca', '6128eb268c44']
|
||||
hook: scripts\stop-one.cmd; api replicas: ['343119df2488', 'd590adc21943']
|
||||
=== run 1: api containers before
|
||||
SERVICE CONTAINER ID STATUS
|
||||
api 11a93acecaca Up 36 seconds (healthy)
|
||||
api 6128eb268c44 Up 36 seconds (healthy)
|
||||
api 343119df2488 Up 27 seconds (healthy)
|
||||
api d590adc21943 Up 27 seconds (healthy)
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.␍
|
||||
{"observed_instances_before": 2, "successful_writes": 126, "failed_attempts": 0, "first_recovery_seconds": 4.39}␍
|
||||
{"observed_instances_before": 2, "successful_writes": 123, "failed_attempts": 0, "first_recovery_seconds": 4.422}␍
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
api containers after
|
||||
SERVICE CONTAINER ID STATUS
|
||||
api 11a93acecaca Exited (0) 11 seconds ago
|
||||
api 6128eb268c44 Up 53 seconds (healthy)
|
||||
the hook stopped 11a93acecaca (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched
|
||||
restored: docker start 11a93acecaca -> healthy and back in HAProxy
|
||||
api 343119df2488 Exited (0) 11 seconds ago
|
||||
api d590adc21943 Up 43 seconds (healthy)
|
||||
the hook stopped 343119df2488 (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched
|
||||
restored: docker start 343119df2488 -> healthy and back in HAProxy
|
||||
=== run 2: api containers before
|
||||
SERVICE CONTAINER ID STATUS
|
||||
api 11a93acecaca Up 2 seconds (healthy)
|
||||
api 6128eb268c44 Up 56 seconds (healthy)
|
||||
api 343119df2488 Up 4 seconds (healthy)
|
||||
api d590adc21943 Up 48 seconds (healthy)
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.␍
|
||||
{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 4.328}␍
|
||||
{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 4.531}␍
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
api containers after
|
||||
SERVICE CONTAINER ID STATUS
|
||||
api 11a93acecaca Up 18 seconds (healthy)
|
||||
api 6128eb268c44 Exited (0) 11 seconds ago
|
||||
the hook stopped 6128eb268c44 (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched
|
||||
restored: docker start 6128eb268c44 -> healthy and back in HAProxy
|
||||
api 343119df2488 Up 20 seconds (healthy)
|
||||
api d590adc21943 Exited (0) 11 seconds ago
|
||||
the hook stopped d590adc21943 (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched
|
||||
restored: docker start d590adc21943 -> healthy and back in HAProxy
|
||||
PASS: public failover test passed 2 times; every api replica was stopped once and restored
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# failures: Redis and PostgreSQL outages via the balancer
|
||||
# date: 2026-10-21 21:11:20, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:24:54, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/test_failures.py
|
||||
# exit code: 0, 55.6 s
|
||||
|
||||
stopping redis
|
||||
/health/live -> 200, /health/ready -> 503 (8016 ms)
|
||||
valid token: GET /users/me -> 503 unavailable (8000 ms), session cannot be checked
|
||||
/health/live -> 200, /health/ready -> 503 (8000 ms)
|
||||
valid token: GET /users/me -> 503 unavailable (8031 ms), session cannot be checked
|
||||
unknown token -> 503, not 200: nobody is trusted without a check
|
||||
no token -> 401
|
||||
login -> 503: there is nowhere to store the session
|
||||
@@ -19,7 +19,7 @@ starting redis
|
||||
after start: the session issued before the crash still works (AOF, appendfsync always)
|
||||
PASS: sessions survive a Redis crash, not only a clean restart
|
||||
stopping postgres
|
||||
/health/live -> 200, /health/ready -> 503 (1016 ms)
|
||||
/health/live -> 200, /health/ready -> 503 (1031 ms)
|
||||
valid token: users/me, chats, history, new message -> 503 unavailable, no details
|
||||
no token -> 401, unknown token -> 401: anonymous users are not let in
|
||||
login -> 503, registration -> 503
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# idempotency: Idempotency-Key, parallel same key via replicas
|
||||
# date: 2026-10-21 21:08:52, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:22:30, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/test_idempotency.py
|
||||
# exit code: 0, 2.4 s
|
||||
# exit code: 0, 2.5 s
|
||||
|
||||
first -> 201; same JSON written differently -> 201; plain retry -> 201
|
||||
PASS: a retry with the same key returns the original 201 and message, no duplicate
|
||||
20 parallel POSTs with one key: statuses {201: 20}, replicas {'6128eb268c44': 10, '11a93acecaca': 10}, different bodies in answers: 1, messages in history: 1
|
||||
20 parallel POSTs with one key: statuses {201: 20}, replicas {'343119df2488': 10, 'd590adc21943': 10}, different bodies in answers: 1, messages in history: 1
|
||||
PASS: parallel requests with one key through different replicas create exactly one message
|
||||
20 parallel POSTs, one key, two different bodies: {('body A', 201): 10, ('body B', 409): 10}; saved: ['body A']
|
||||
PASS: the first body wins, the same body gets 201, the other body gets 409
|
||||
|
||||
+3
-3
@@ -2,12 +2,12 @@
|
||||
"lab": 4,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 30.0,
|
||||
"elapsed_seconds": 30.015,
|
||||
"elapsed_seconds": 30.047,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 7379,
|
||||
"successful_requests": 7191,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 245.844,
|
||||
"successful_rps": 239.325,
|
||||
"successful_latency_ms": {
|
||||
"p50": 16.0,
|
||||
"p95": 32.0,
|
||||
|
||||
+5
-5
@@ -1,18 +1,18 @@
|
||||
# load: public load generator, 30 s, concurrency 4
|
||||
# date: 2026-10-21 21:10:24, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:23:58, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe tests/load.py --base-url http://127.0.0.1:8080 --duration 30 --concurrency 4 --output evidence/load.json
|
||||
# exit code: 0, 30.4 s
|
||||
# exit code: 0, 30.3 s
|
||||
|
||||
{
|
||||
"lab": 4,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 30.0,
|
||||
"elapsed_seconds": 30.015,
|
||||
"elapsed_seconds": 30.047,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 7379,
|
||||
"successful_requests": 7191,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 245.844,
|
||||
"successful_rps": 239.325,
|
||||
"successful_latency_ms": {
|
||||
"p50": 16.0,
|
||||
"p95": 32.0,
|
||||
|
||||
+17
-17
@@ -1,24 +1,24 @@
|
||||
# persistence: public persistence test incl. S3 + proof of real restarts (make test-persistence)
|
||||
# date: 2026-10-21 21:12:49, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:25:42, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/run_persistence.py
|
||||
# exit code: 0, 40.2 s
|
||||
# exit code: 0, 47.9 s
|
||||
|
||||
before: container, ID, started at
|
||||
messenger-lab4-api-1 11a93acecaca 2026-10-21T18:11:37.067348364Z
|
||||
messenger-lab4-api-2 6128eb268c44 2026-10-21T18:11:36.916459116Z
|
||||
messenger-lab4-janitor-1 baefc0e97dff 2026-10-21T18:11:33.313531682Z
|
||||
messenger-lab4-lb-1 03697e5fe34c 2026-10-21T18:11:40.981519931Z
|
||||
messenger-lab4-postgres-1 f7a6a1ce25a7 2026-10-21T18:11:22.482229396Z
|
||||
messenger-lab4-redis-1 b4523b39a0fe 2026-10-21T18:11:22.626999402Z
|
||||
messenger-lab4-s3-1 e7cbe6ddfa4a 2026-10-21T18:11:32.264154477Z
|
||||
messenger-lab4-api-1 343119df2488 2026-10-22T11:23:03.387600262Z
|
||||
messenger-lab4-api-2 d590adc21943 2026-10-22T11:23:24.689368662Z
|
||||
messenger-lab4-janitor-1 d50ac08018c1 2026-10-22T11:18:17.423165415Z
|
||||
messenger-lab4-lb-1 0d5b262b09b3 2026-10-22T11:03:30.971491955Z
|
||||
messenger-lab4-postgres-1 84794c3ac2c5 2026-10-22T11:24:52.672662576Z
|
||||
messenger-lab4-redis-1 3130fd5532b0 2026-10-22T11:24:32.055032931Z
|
||||
messenger-lab4-s3-1 122ad603e3be 2026-10-22T11:20:52.20174809Z
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.␍
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
after (hook + waiting for readiness took 39.2 s): container, ID, started at, restarted?
|
||||
messenger-lab4-api-1 11a93acecaca 2026-10-21T18:12:46.288933033Z same container: True, restarted: True
|
||||
messenger-lab4-api-2 6128eb268c44 2026-10-21T18:12:46.174283308Z same container: True, restarted: True
|
||||
messenger-lab4-janitor-1 baefc0e97dff 2026-10-21T18:12:42.503993973Z same container: True, restarted: True
|
||||
messenger-lab4-lb-1 03697e5fe34c 2026-10-21T18:12:47.684809415Z same container: True, restarted: True
|
||||
messenger-lab4-postgres-1 f7a6a1ce25a7 2026-10-21T18:12:11.527270274Z same container: True, restarted: True
|
||||
messenger-lab4-redis-1 b4523b39a0fe 2026-10-21T18:12:11.368416107Z same container: True, restarted: True
|
||||
messenger-lab4-s3-1 e7cbe6ddfa4a 2026-10-21T18:12:21.264135454Z same container: True, restarted: True
|
||||
after (hook + waiting for readiness took 47.0 s): container, ID, started at, restarted?
|
||||
messenger-lab4-api-1 343119df2488 2026-10-22T11:25:36.201352227Z same container: True, restarted: True
|
||||
messenger-lab4-api-2 d590adc21943 2026-10-22T11:25:36.073058122Z same container: True, restarted: True
|
||||
messenger-lab4-janitor-1 d50ac08018c1 2026-10-22T11:25:32.252887377Z same container: True, restarted: True
|
||||
messenger-lab4-lb-1 0d5b262b09b3 2026-10-22T11:25:40.174401907Z same container: True, restarted: True
|
||||
messenger-lab4-postgres-1 84794c3ac2c5 2026-10-22T11:24:56.311470336Z same container: True, restarted: True
|
||||
messenger-lab4-redis-1 3130fd5532b0 2026-10-22T11:24:56.150164333Z same container: True, restarted: True
|
||||
messenger-lab4-s3-1 122ad603e3be 2026-10-22T11:25:06.0234901Z same container: True, restarted: True
|
||||
PASS: the hook restarted postgres, redis, s3 (SeaweedFS), all api replicas, janitor and lb (same containers, new start time); data, the attachment bytes and the old token survived
|
||||
|
||||
+3
-3
@@ -1,9 +1,9 @@
|
||||
# race: lab2 races via the balancer
|
||||
# date: 2026-10-21 21:08:56, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:22:34, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/test_race.py
|
||||
# exit code: 0, 1.2 s
|
||||
# exit code: 0, 1.1 s
|
||||
|
||||
sent 100 messages from 10 clients in 0.22 s; read 100 in 4 pages, unique ids: 100
|
||||
sent 100 messages from 10 clients in 0.19 s; read 100 in 4 pages, unique ids: 100
|
||||
PASS: 100 concurrent messages from 10 clients, no losses and no duplicates
|
||||
20 parallel registrations of one username: 201 x1, 409 x19
|
||||
PASS: concurrent registration created exactly one user
|
||||
|
||||
+4
-4
@@ -1,11 +1,11 @@
|
||||
# redis: Redis as the session store
|
||||
# date: 2026-10-21 21:09:13, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:22:43, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/redis_info.py
|
||||
# exit code: 0, 2.8 s
|
||||
# exit code: 0, 2.9 s
|
||||
|
||||
session key: session:f4f1fe0235d139de...; value is the user id: True
|
||||
session key: session:e57bdd083b4cb87e...; value is the user id: True
|
||||
TTL of the key, seconds: 3599
|
||||
sessions in Redis: 126
|
||||
sessions in Redis: 330
|
||||
config appendonly = yes
|
||||
config appendfsync = always
|
||||
config maxmemory-policy = noeviction
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# s3-security: lab4: private bucket outside API, limited API key, no console/ports, idempotent bootstrap
|
||||
# date: 2026-10-21 21:03:14, commit: 2dbed80
|
||||
# date: 2026-10-22 14:19:35, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/test_s3_security.py
|
||||
# exit code: 0, 9.5 s
|
||||
# exit code: 0, 9.6 s
|
||||
|
||||
anonymous get_object directly to S3 (from the data network) -> 403 AccessDenied
|
||||
anonymous head_object directly to S3 (from the data network) -> 403
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
# smoke: public smoke via the balancer (make test), with attachment
|
||||
# date: 2026-10-21 21:03:01, commit: 2dbed80
|
||||
# date: 2026-10-22 14:19:20, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe tests/smoke.py --base-url http://127.0.0.1:8080
|
||||
# exit code: 0, 0.6 s
|
||||
# exit code: 0, 0.8 s
|
||||
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
|
||||
+14
-14
@@ -1,21 +1,21 @@
|
||||
# stack: stand layout: replicas, ports, limits, networks, secrets, S3
|
||||
# date: 2026-10-21 21:03:00, commit: 2dbed80
|
||||
# date: 2026-10-22 14:19:19, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/check_stack.py
|
||||
# exit code: 0, 3.0 s
|
||||
# exit code: 0, 2.9 s
|
||||
|
||||
host: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; client Python 3.11.1
|
||||
Docker 28.1.1, Docker Compose version v2.35.1-desktop.1; Docker VM: 16 CPU, 15.2 GiB RAM, Docker Desktop
|
||||
|
||||
SERVICE CONTAINER ID STATUS PORTS
|
||||
api 11a93acecaca Up 20 seconds (healthy) 8080/tcp
|
||||
api 6128eb268c44 Up 20 seconds (healthy) 8080/tcp
|
||||
janitor baefc0e97dff Up 20 seconds
|
||||
lb 03697e5fe34c Up 18 seconds (healthy) 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
|
||||
migrate aca090330db4 Exited (0) 22 seconds ago
|
||||
postgres f7a6a1ce25a7 Up 2 minutes (healthy)
|
||||
redis b4523b39a0fe Up 2 minutes (healthy)
|
||||
s3 e7cbe6ddfa4a Up 2 minutes (healthy)
|
||||
s3-init 60719e674e18 Exited (0) 21 seconds ago
|
||||
SERVICE CONTAINER ID STATUS PORTS
|
||||
api 343119df2488 Up 17 seconds (healthy) 8080/tcp
|
||||
api d590adc21943 Up 18 seconds (healthy) 8080/tcp
|
||||
janitor d50ac08018c1 Up About a minute
|
||||
lb 0d5b262b09b3 Up 15 minutes (healthy) 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
|
||||
migrate 6c3123ccf237 Exited (0) About a minute ago
|
||||
postgres 84794c3ac2c5 Up 16 minutes (healthy)
|
||||
redis 3130fd5532b0 Up 16 minutes (healthy)
|
||||
s3 122ad603e3be Up 16 minutes (healthy)
|
||||
s3-init 47c2d7d47ed2 Exited (0) About a minute ago
|
||||
|
||||
container image cpus memory host ports networks
|
||||
messenger-lab4-api-1 messenger-lab4-api 1.00 512 MiB - data,edge
|
||||
@@ -35,6 +35,6 @@ secrets of redis: ['redis_auth']
|
||||
secrets of s3: ['s3_config']
|
||||
secrets of s3-init: ['s3_access_key_id', 's3_root_password', 's3_root_user', 's3_secret_access_key']
|
||||
|
||||
through http://127.0.0.1:8080 one session is served by 2 replicas: ['11a93acecaca', '6128eb268c44']
|
||||
/health/ready -> 200 from 11a93acecaca
|
||||
through http://127.0.0.1:8080 one session is served by 2 replicas: ['343119df2488', 'd590adc21943']
|
||||
/health/ready -> 200 from d590adc21943
|
||||
PASS: 2 api replicas behind one entry point, no direct API ports, limits on every container, postgres/redis/s3/janitor only in the internal data network, S3 admin keys only in s3 and s3-init
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# student-tests: lab1/lab2 API tests via the balancer
|
||||
# date: 2026-10-21 21:08:55, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:22:33, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/test_api.py
|
||||
# exit code: 0, 2.7 s
|
||||
# exit code: 0, 2.9 s
|
||||
|
||||
PASS: duplicate username returns 409
|
||||
PASS: invalid JSON returns 400
|
||||
|
||||
+32
-15
@@ -2,39 +2,56 @@
|
||||
|
||||
| Шаг | Что проверяет | Результат | Время, с | Файл |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| persistence | public persistence test incl. S3 + proof of real restarts (make test-persistence) | PASS | 40 | evidence/persistence.txt |
|
||||
| check | offline contract check (make check) | PASS | 0 | evidence/check.txt |
|
||||
| stack | stand layout: replicas, ports, limits, networks, secrets, S3 | PASS | 3 | evidence/stack.txt |
|
||||
| smoke | public smoke via the balancer (make test), with attachment | PASS | 1 | evidence/smoke.txt |
|
||||
| attachments | lab4: files, limits, damaged images, MIME, filenames, access, attachment_ids | PASS | 5 | evidence/attachments.txt |
|
||||
| s3-security | lab4: private bucket outside API, limited API key, no console/ports, idempotent bootstrap | PASS | 10 | evidence/s3-security.txt |
|
||||
| consistency | lab4: aborted upload, crash between S3 and DB, janitor, orphans, S3 outage | PASS | 133 | evidence/consistency.txt |
|
||||
| upload-memory | lab4: >=10 concurrent uploads, peak memory of api replicas | PASS | 37 | evidence/upload-memory.txt |
|
||||
| balancing | balancing, shared sessions/data/files, X-Instance-Id | PASS | 2 | evidence/balancing.txt |
|
||||
| idempotency | Idempotency-Key, parallel same key via replicas | PASS | 2 | evidence/idempotency.txt |
|
||||
| student-tests | lab1/lab2 API tests via the balancer | PASS | 3 | evidence/student-tests.txt |
|
||||
| race | lab2 races via the balancer | PASS | 1 | evidence/race.txt |
|
||||
| db | PostgreSQL: rollback, FK, Argon2id, unique keys, migrations | PASS | 6 | evidence/db.txt |
|
||||
| redis | Redis as the session store | PASS | 3 | evidence/redis.txt |
|
||||
| explain | EXPLAIN of the history page (full plans: explain-history.txt) | PASS | 1 | evidence/explain.txt |
|
||||
| failover | public failover test for every replica, file downloaded via the other one (make test-failover) | PASS | 43 | evidence/failover.txt |
|
||||
| load | public load generator, 30 s, concurrency 4 | PASS | 30 | evidence/load.txt |
|
||||
| failures | Redis and PostgreSQL outages via the balancer | PASS | 56 | evidence/failures.txt |
|
||||
| persistence | public persistence test incl. S3 + proof of real restarts (make test-persistence) | PASS | 48 | evidence/persistence.txt |
|
||||
|
||||
# Замеры
|
||||
|
||||
Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-21, commit 2dbed80 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB.
|
||||
Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-22, commit c942c61 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB.
|
||||
|
||||
Публичный тест отказа (`make test-failover`, 15 с записи во время остановки реплики):
|
||||
|
||||
| Прогон | Остановлена реплика | Реплик до отказа | Успешных записей | Неудачных попыток (повторены с тем же ключом) | Первая запись после hook, с |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | 11a93acecaca | 2 | 126 | 0 | 4.39 |
|
||||
| 2 | 6128eb268c44 | 2 | 124 | 0 | 4.328 |
|
||||
| 1 | 343119df2488 | 2 | 123 | 0 | 4.422 |
|
||||
| 2 | d590adc21943 | 2 | 124 | 0 | 4.531 |
|
||||
|
||||
Одновременные загрузки (`student-tests/upload_memory.py`, перед каждым сценарием реплики перезапускаются, лимит api 512 MiB / 1 CPU). Пик RSS — VmHWM процесса API; cgroup — память контейнера, включая page cache временных файлов:
|
||||
|
||||
| Сценарий | Одновременно | Объём, MiB | Время, с (p50 / max) | Реплика: загрузок | RSS до, MiB | Пик RSS, MiB | Рост, MiB | Пик cgroup, MiB |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 1.38 / 1.52 | 11a93acecaca: 5 | 93.5 | 106.2 | 12.6 | 134.1 |
|
||||
| | | | | 6128eb268c44: 5 | 94.8 | 105.3 | 10.5 | 135.2 |
|
||||
| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 11a93acecaca: 5 | 93.2 | 96.6 | 3.4 | 84.3 |
|
||||
| | | | | 6128eb268c44: 5 | 93.1 | 94.7 | 1.6 | 86.8 |
|
||||
| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.37 / 1.77 | 11a93acecaca: 10 | 94.8 | 121.7 | 26.9 | 181.9 |
|
||||
| | | | | 6128eb268c44: 10 | 95.0 | 116.1 | 21.0 | 155.6 |
|
||||
| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 2.87 / 3.52 | 343119df2488: 5 | 94.8 | 107.6 | 12.8 | 136.0 |
|
||||
| | | | | d590adc21943: 5 | 91.2 | 104.5 | 13.3 | 136.2 |
|
||||
| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 343119df2488: 5 | 95.1 | 96.6 | 1.5 | 86.5 |
|
||||
| | | | | d590adc21943: 5 | 92.1 | 96.6 | 4.6 | 83.5 |
|
||||
| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.5 / 1.78 | 343119df2488: 10 | 93.2 | 116.2 | 23.0 | 158.2 |
|
||||
| | | | | d590adc21943: 10 | 95.0 | 118.6 | 23.6 | 167.8 |
|
||||
|
||||
Один PNG 8000x5000 RGB внутри контейнера api, дополнительный пик RSS: наивно (тело целиком в памяти + полное декодирование Pillow) 161 MiB, потоковая проверка 0 MiB. 10 таких наивных загрузок одновременно не поместились бы в 512 MiB.
|
||||
|
||||
Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 245.844 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс.
|
||||
Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 239.325 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.39, 4.328 с (hook ждёт остановки ~4 с); потерь нет |
|
||||
| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 39.2 с до готовности; потерь нет |
|
||||
| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 20141 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет |
|
||||
| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14766 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет |
|
||||
| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.422, 4.531 с (hook ждёт остановки ~4 с); потерь нет |
|
||||
| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 47.0 с до готовности; потерь нет |
|
||||
| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 19453 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет |
|
||||
| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14891 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет |
|
||||
| падение между PUT в S3 и COMMIT в БД (код API в контейнере, finish_upload падает) | вложение не видно, janitor не трогает его до конца grace period, потом удаляет объект и строку | GET/прикрепление -> 404; janitor: свежая строка сохранена, dry-run только отчёт, затем удалено | мусора не осталось; повторный проход ничего не делает |
|
||||
| обрыв загрузки клиентом (9 МБ заявлено, отправлено 0,2 и 3 МБ) | ничего не сохраняется | строк и объектов не прибавилось, временных файлов 0 | мгновенно |
|
||||
|
||||
+51
-51
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"date": "2026-10-21 21:08",
|
||||
"date": "2026-10-22 14:22",
|
||||
"api_limit_mib": 512,
|
||||
"api_limit_cpus": 1.0,
|
||||
"runs": [
|
||||
@@ -11,32 +11,32 @@
|
||||
],
|
||||
"total_mib": 94.9,
|
||||
"all_201": true,
|
||||
"wall_seconds": 1.52,
|
||||
"wall_seconds": 3.52,
|
||||
"latency_s": {
|
||||
"p50": 1.38,
|
||||
"max": 1.52
|
||||
"p50": 2.87,
|
||||
"max": 3.52
|
||||
},
|
||||
"sampler_rounds": 9,
|
||||
"sampler_rounds": 18,
|
||||
"replicas": [
|
||||
{
|
||||
"replica": "11a93acecaca",
|
||||
"uploads_served": 5,
|
||||
"rss_before_mib": 93.5,
|
||||
"rss_peak_mib": 106.2,
|
||||
"rss_growth_mib": 12.6,
|
||||
"rss_peak_sampled_mib": 106.2,
|
||||
"cgroup_peak_mib": 134.1,
|
||||
"cgroup_peak_sampled_mib": 127.7
|
||||
},
|
||||
{
|
||||
"replica": "6128eb268c44",
|
||||
"replica": "343119df2488",
|
||||
"uploads_served": 5,
|
||||
"rss_before_mib": 94.8,
|
||||
"rss_peak_mib": 105.3,
|
||||
"rss_growth_mib": 10.5,
|
||||
"rss_peak_sampled_mib": 105.3,
|
||||
"cgroup_peak_mib": 135.2,
|
||||
"cgroup_peak_sampled_mib": 131.4
|
||||
"rss_peak_mib": 107.6,
|
||||
"rss_growth_mib": 12.8,
|
||||
"rss_peak_sampled_mib": 107.6,
|
||||
"cgroup_peak_mib": 136.0,
|
||||
"cgroup_peak_sampled_mib": 134.5
|
||||
},
|
||||
{
|
||||
"replica": "d590adc21943",
|
||||
"uploads_served": 5,
|
||||
"rss_before_mib": 91.2,
|
||||
"rss_peak_mib": 104.5,
|
||||
"rss_growth_mib": 13.3,
|
||||
"rss_peak_sampled_mib": 104.5,
|
||||
"cgroup_peak_mib": 136.2,
|
||||
"cgroup_peak_sampled_mib": 133.3
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -53,27 +53,27 @@
|
||||
"p50": 0.85,
|
||||
"max": 1.09
|
||||
},
|
||||
"sampler_rounds": 8,
|
||||
"sampler_rounds": 12,
|
||||
"replicas": [
|
||||
{
|
||||
"replica": "11a93acecaca",
|
||||
"replica": "343119df2488",
|
||||
"uploads_served": 5,
|
||||
"rss_before_mib": 93.2,
|
||||
"rss_before_mib": 95.1,
|
||||
"rss_peak_mib": 96.6,
|
||||
"rss_growth_mib": 3.4,
|
||||
"rss_growth_mib": 1.5,
|
||||
"rss_peak_sampled_mib": 96.6,
|
||||
"cgroup_peak_mib": 84.3,
|
||||
"cgroup_peak_sampled_mib": 77.3
|
||||
"cgroup_peak_mib": 86.5,
|
||||
"cgroup_peak_sampled_mib": 77.6
|
||||
},
|
||||
{
|
||||
"replica": "6128eb268c44",
|
||||
"replica": "d590adc21943",
|
||||
"uploads_served": 5,
|
||||
"rss_before_mib": 93.1,
|
||||
"rss_peak_mib": 94.7,
|
||||
"rss_growth_mib": 1.6,
|
||||
"rss_peak_sampled_mib": 94.6,
|
||||
"cgroup_peak_mib": 86.8,
|
||||
"cgroup_peak_sampled_mib": 77.3
|
||||
"rss_before_mib": 92.1,
|
||||
"rss_peak_mib": 96.6,
|
||||
"rss_growth_mib": 4.6,
|
||||
"rss_peak_sampled_mib": 96.6,
|
||||
"cgroup_peak_mib": 83.5,
|
||||
"cgroup_peak_sampled_mib": 77.8
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -87,32 +87,32 @@
|
||||
],
|
||||
"total_mib": 133.2,
|
||||
"all_201": true,
|
||||
"wall_seconds": 1.77,
|
||||
"wall_seconds": 1.78,
|
||||
"latency_s": {
|
||||
"p50": 1.37,
|
||||
"max": 1.77
|
||||
"p50": 1.5,
|
||||
"max": 1.78
|
||||
},
|
||||
"sampler_rounds": 10,
|
||||
"sampler_rounds": 12,
|
||||
"replicas": [
|
||||
{
|
||||
"replica": "11a93acecaca",
|
||||
"replica": "343119df2488",
|
||||
"uploads_served": 10,
|
||||
"rss_before_mib": 94.8,
|
||||
"rss_peak_mib": 121.7,
|
||||
"rss_growth_mib": 26.9,
|
||||
"rss_peak_sampled_mib": 121.7,
|
||||
"cgroup_peak_mib": 181.9,
|
||||
"cgroup_peak_sampled_mib": 180.6
|
||||
"rss_before_mib": 93.2,
|
||||
"rss_peak_mib": 116.2,
|
||||
"rss_growth_mib": 23.0,
|
||||
"rss_peak_sampled_mib": 116.2,
|
||||
"cgroup_peak_mib": 158.2,
|
||||
"cgroup_peak_sampled_mib": 146.7
|
||||
},
|
||||
{
|
||||
"replica": "6128eb268c44",
|
||||
"replica": "d590adc21943",
|
||||
"uploads_served": 10,
|
||||
"rss_before_mib": 95.0,
|
||||
"rss_peak_mib": 116.1,
|
||||
"rss_growth_mib": 21.0,
|
||||
"rss_peak_sampled_mib": 116.1,
|
||||
"cgroup_peak_mib": 155.6,
|
||||
"cgroup_peak_sampled_mib": 155.6
|
||||
"rss_peak_mib": 118.6,
|
||||
"rss_growth_mib": 23.6,
|
||||
"rss_peak_sampled_mib": 118.6,
|
||||
"cgroup_peak_mib": 167.8,
|
||||
"cgroup_peak_sampled_mib": 167.7
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
# upload-memory: lab4: >=10 concurrent uploads, peak memory of api replicas
|
||||
# date: 2026-10-21 21:08:48, commit: 2dbed80 + uncommitted changes
|
||||
# date: 2026-10-22 14:22:25, commit: c942c61 + uncommitted changes
|
||||
# command: python.exe student-tests/upload_memory.py --output evidence/upload-memory.json
|
||||
# exit code: 0, 34.2 s
|
||||
# exit code: 0, 37.0 s
|
||||
|
||||
client: Python 3.11.1 on Windows 10
|
||||
preparing synthetic images (random-noise PNG ~9.5 MiB each, 40 Mpx PNG)...
|
||||
large: 9953047 bytes; 40 Mpx: 116714 bytes
|
||||
{"scenario": "10 x ~9.5 MiB PNG (noise)", "concurrency": 10, "file_bytes": [9953047], "total_mib": 94.9, "all_201": true, "wall_seconds": 1.52, "latency_s": {"p50": 1.38, "max": 1.52}, "sampler_rounds": 9, "replicas": [{"replica": "11a93acecaca", "uploads_served": 5, "rss_before_mib": 93.5, "rss_peak_mib": 106.2, "rss_growth_mib": 12.6, "rss_peak_sampled_mib": 106.2, "cgroup_peak_mib": 134.1, "cgroup_peak_sampled_mib": 127.7}, {"replica": "6128eb268c44", "uploads_served": 5, "rss_before_mib": 94.8, "rss_peak_mib": 105.3, "rss_growth_mib": 10.5, "rss_peak_sampled_mib": 105.3, "cgroup_peak_mib": 135.2, "cgroup_peak_sampled_mib": 131.4}]}
|
||||
{"scenario": "10 x 40 Mpx PNG (8000x5000 RGB)", "concurrency": 10, "file_bytes": [116714], "total_mib": 1.1, "all_201": true, "wall_seconds": 1.09, "latency_s": {"p50": 0.85, "max": 1.09}, "sampler_rounds": 8, "replicas": [{"replica": "11a93acecaca", "uploads_served": 5, "rss_before_mib": 93.2, "rss_peak_mib": 96.6, "rss_growth_mib": 3.4, "rss_peak_sampled_mib": 96.6, "cgroup_peak_mib": 84.3, "cgroup_peak_sampled_mib": 77.3}, {"replica": "6128eb268c44", "uploads_served": 5, "rss_before_mib": 93.1, "rss_peak_mib": 94.7, "rss_growth_mib": 1.6, "rss_peak_sampled_mib": 94.6, "cgroup_peak_mib": 86.8, "cgroup_peak_sampled_mib": 77.3}]}
|
||||
{"scenario": "20 mixed: large + 40 Mpx + JPEG", "concurrency": 20, "file_bytes": [1492, 116714, 9953047], "total_mib": 133.2, "all_201": true, "wall_seconds": 1.77, "latency_s": {"p50": 1.37, "max": 1.77}, "sampler_rounds": 10, "replicas": [{"replica": "11a93acecaca", "uploads_served": 10, "rss_before_mib": 94.8, "rss_peak_mib": 121.7, "rss_growth_mib": 26.9, "rss_peak_sampled_mib": 121.7, "cgroup_peak_mib": 181.9, "cgroup_peak_sampled_mib": 180.6}, {"replica": "6128eb268c44", "uploads_served": 10, "rss_before_mib": 95.0, "rss_peak_mib": 116.1, "rss_growth_mib": 21.0, "rss_peak_sampled_mib": 116.1, "cgroup_peak_mib": 155.6, "cgroup_peak_sampled_mib": 155.6}]}
|
||||
{"scenario": "10 x ~9.5 MiB PNG (noise)", "concurrency": 10, "file_bytes": [9953047], "total_mib": 94.9, "all_201": true, "wall_seconds": 3.52, "latency_s": {"p50": 2.87, "max": 3.52}, "sampler_rounds": 18, "replicas": [{"replica": "343119df2488", "uploads_served": 5, "rss_before_mib": 94.8, "rss_peak_mib": 107.6, "rss_growth_mib": 12.8, "rss_peak_sampled_mib": 107.6, "cgroup_peak_mib": 136.0, "cgroup_peak_sampled_mib": 134.5}, {"replica": "d590adc21943", "uploads_served": 5, "rss_before_mib": 91.2, "rss_peak_mib": 104.5, "rss_growth_mib": 13.3, "rss_peak_sampled_mib": 104.5, "cgroup_peak_mib": 136.2, "cgroup_peak_sampled_mib": 133.3}]}
|
||||
{"scenario": "10 x 40 Mpx PNG (8000x5000 RGB)", "concurrency": 10, "file_bytes": [116714], "total_mib": 1.1, "all_201": true, "wall_seconds": 1.09, "latency_s": {"p50": 0.85, "max": 1.09}, "sampler_rounds": 12, "replicas": [{"replica": "343119df2488", "uploads_served": 5, "rss_before_mib": 95.1, "rss_peak_mib": 96.6, "rss_growth_mib": 1.5, "rss_peak_sampled_mib": 96.6, "cgroup_peak_mib": 86.5, "cgroup_peak_sampled_mib": 77.6}, {"replica": "d590adc21943", "uploads_served": 5, "rss_before_mib": 92.1, "rss_peak_mib": 96.6, "rss_growth_mib": 4.6, "rss_peak_sampled_mib": 96.6, "cgroup_peak_mib": 83.5, "cgroup_peak_sampled_mib": 77.8}]}
|
||||
{"scenario": "20 mixed: large + 40 Mpx + JPEG", "concurrency": 20, "file_bytes": [1492, 116714, 9953047], "total_mib": 133.2, "all_201": true, "wall_seconds": 1.78, "latency_s": {"p50": 1.5, "max": 1.78}, "sampler_rounds": 12, "replicas": [{"replica": "343119df2488", "uploads_served": 10, "rss_before_mib": 93.2, "rss_peak_mib": 116.2, "rss_growth_mib": 23.0, "rss_peak_sampled_mib": 116.2, "cgroup_peak_mib": 158.2, "cgroup_peak_sampled_mib": 146.7}, {"replica": "d590adc21943", "uploads_served": 10, "rss_before_mib": 95.0, "rss_peak_mib": 118.6, "rss_growth_mib": 23.6, "rss_peak_sampled_mib": 118.6, "cgroup_peak_mib": 167.8, "cgroup_peak_sampled_mib": 167.7}]}
|
||||
one 8000x5000 RGB PNG inside the api container, extra peak RSS: naive (body in RAM + full decode) 161 MiB, streaming check 0 MiB
|
||||
PASS: all uploads succeeded; largest RSS growth of one replica: 26.9 MiB (limit 512 MiB); results in evidence/upload-memory.json
|
||||
PASS: all uploads succeeded; largest RSS growth of one replica: 23.6 MiB (limit 512 MiB); results in evidence/upload-memory.json
|
||||
|
||||
+22
-3
@@ -22,7 +22,7 @@ def chunk(kind: bytes, data: bytes) -> bytes:
|
||||
def png(width, height, *, gray=False, noise=False, interlace=0, pad_to=None, idat=None) -> bytes:
|
||||
"""PNG width x height. noise=True — случайные пиксели (почти не сжимаются, файл большой).
|
||||
|
||||
pad_to — дополнить файл служебным чанком prVt до точного размера в байтах.
|
||||
pad_to — дополнить файл текстовым чанком tEXt после IDAT до точного размера в байтах.
|
||||
idat — готовые сжатые данные (для «бомбы»: заголовок обещает одно, данных другое).
|
||||
"""
|
||||
channels = 1 if gray else 3
|
||||
@@ -43,9 +43,9 @@ def png(width, height, *, gray=False, noise=False, interlace=0, pad_to=None, ida
|
||||
idat = b"".join(parts)
|
||||
body = b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header) + chunk(b"IDAT", idat)
|
||||
if pad_to is not None:
|
||||
need = pad_to - len(body) - 12 - 12 # чанк-заполнитель + IEND
|
||||
need = pad_to - len(body) - 12 - 12 - 8 # заголовок чанка-заполнителя, IEND, ключ "Comment\0"
|
||||
assert need >= 0, "image is already larger than pad_to"
|
||||
body += chunk(b"prVt", bytes(need))
|
||||
body += chunk(b"tEXt", b"Comment\x00" + b"a" * need)
|
||||
return body + chunk(b"IEND", b"")
|
||||
|
||||
|
||||
@@ -72,3 +72,22 @@ def flip_byte(data: bytes, position: int) -> bytes:
|
||||
|
||||
|
||||
GIF = b"GIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\xff\xff\xff!\xf9\x04\x01\x00\x00\x00\x00,\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02D\x01\x00;"
|
||||
|
||||
|
||||
# --- сборка PNG из произвольных чанков (для проверок смысла чанков при верных CRC) ---
|
||||
|
||||
def ihdr(width, height, depth=8, color=2, interlace=0):
|
||||
return b"IHDR", struct.pack(">IIBBBBB", width, height, depth, color, 0, 0, interlace)
|
||||
|
||||
|
||||
def zero_idat(width, height, depth=8, channels=3):
|
||||
"""Сжатые пиксели: каждая строка = байт фильтра 0 + нули (индекс палитры 0 / чёрный)."""
|
||||
row_bytes = (width * depth * channels + 7) // 8
|
||||
return b"IDAT", zlib.compress((b"\x00" + bytes(row_bytes)) * height)
|
||||
|
||||
|
||||
def assemble(*chunks):
|
||||
"""PNG из чанков (kind, data) с ВЕРНЫМИ CRC; IEND добавляется, если не передан."""
|
||||
if not chunks or chunks[-1][0] != b"IEND":
|
||||
chunks = chunks + ((b"IEND", b""),)
|
||||
return b"\x89PNG\r\n\x1a\n" + b"".join(chunk(kind, data) for kind, data in chunks)
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
import hashlib
|
||||
import http.client
|
||||
import socket
|
||||
import struct
|
||||
import urllib.parse
|
||||
import uuid
|
||||
import zlib
|
||||
@@ -135,6 +136,73 @@ def test_damaged_images():
|
||||
print("PASS: damaged and foreign files are rejected with 415, never 500")
|
||||
|
||||
|
||||
def test_png_chunk_semantics():
|
||||
"""CRC у всех чанков верные, но содержимое нарушает спецификацию PNG -> 415, а не 201.
|
||||
|
||||
Регрессия: PNG 1x1, глубина 8, палитровый, PLTE из 771 байта (257 записей) раньше получал 201 ready,
|
||||
хотя Pillow его отклоняет (invalid palette size).
|
||||
"""
|
||||
alice, bob, _, chat = setup()
|
||||
from images import assemble, ihdr, zero_idat
|
||||
pal = lambda entries: (b"PLTE", bytes(3 * entries))
|
||||
idat_p = zero_idat(1, 1, 8, 1)
|
||||
bad = [
|
||||
("PLTE 771 bytes = 257 entries (review case)",
|
||||
assemble(ihdr(1, 1, 8, 3), (b"PLTE", bytes(771)), (b"IDAT", zlib.compress(b"\0\0")))),
|
||||
("PLTE length not a multiple of 3", assemble(ihdr(1, 1, 8, 3), (b"PLTE", bytes(4)), idat_p)),
|
||||
("empty PLTE", assemble(ihdr(1, 1, 8, 3), (b"PLTE", b""), idat_p)),
|
||||
("palette larger than bit depth allows (depth 1, 3 entries)",
|
||||
assemble(ihdr(1, 1, 1, 3), pal(3), zero_idat(1, 1, 1, 1))),
|
||||
("palette image without PLTE", assemble(ihdr(1, 1, 8, 3), idat_p)),
|
||||
("PLTE in a grayscale image", assemble(ihdr(1, 1, 8, 0), pal(2), zero_idat(1, 1, 8, 1))),
|
||||
("PLTE after IDAT", assemble(ihdr(1, 1, 8, 3), idat_p, pal(2))),
|
||||
("two PLTE chunks", assemble(ihdr(1, 1, 8, 3), pal(2), pal(2), idat_p)),
|
||||
("tRNS longer than the palette", assemble(ihdr(1, 1, 8, 3), pal(2), (b"tRNS", bytes(3)), idat_p)),
|
||||
("tRNS before PLTE", assemble(ihdr(1, 1, 8, 3), (b"tRNS", bytes(1)), pal(2), idat_p)),
|
||||
("tRNS of wrong size for RGB (4 bytes instead of 6)",
|
||||
assemble(ihdr(1, 1, 8, 2), (b"tRNS", bytes(4)), zero_idat(1, 1, 8, 3))),
|
||||
("tRNS in an image with alpha channel",
|
||||
assemble(ihdr(1, 1, 8, 6), (b"tRNS", bytes(2)), zero_idat(1, 1, 8, 4))),
|
||||
("IEND with data", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"IEND", b"x"))),
|
||||
("unknown critical chunk", assemble(ihdr(1, 1, 8, 2), (b"ABCD", b""), zero_idat(1, 1, 8, 3))),
|
||||
("IDAT chunks not consecutive",
|
||||
assemble(ihdr(1, 2, 8, 2), (b"IDAT", zero_idat(1, 2, 8, 3)[1][:5]), (b"tEXt", b"k\0v"),
|
||||
(b"IDAT", zero_idat(1, 2, 8, 3)[1][5:]))),
|
||||
("gAMA after IDAT", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"gAMA", bytes(4)))),
|
||||
("gAMA of 2 bytes instead of 4", assemble(ihdr(1, 1, 8, 2), (b"gAMA", bytes(2)), zero_idat(1, 1, 8, 3))),
|
||||
("3 MiB metadata chunk before IDAT", assemble(ihdr(1, 1, 8, 2), (b"prVt", bytes(3 * 1024 * 1024)),
|
||||
zero_idat(1, 1, 8, 3))),
|
||||
("private chunk after IDAT", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"prVt", bytes(100)))),
|
||||
("animated PNG frame (fdAT)", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"fdAT", bytes(10)))),
|
||||
("tIME of 2 bytes instead of 7", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"tIME", bytes(2)))),
|
||||
("tEXt without keyword separator", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"tEXt", b"nokey"))),
|
||||
("zTXt with broken zlib after IDAT",
|
||||
assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"zTXt", b"k\0\0garbage"))),
|
||||
("zTXt text bomb: 50 MB of text in ~50 KB after IDAT",
|
||||
assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"zTXt", b"k\0\0" + zlib.compress(bytes(50_000_000))))),
|
||||
]
|
||||
for name, data in bad:
|
||||
check_error(upload(alice, chat["id"], data, filename="bad.png", expected=415), "unsupported_media_type")
|
||||
print(f"{name} ({len(data)} bytes, valid CRCs) -> 415")
|
||||
good = [
|
||||
("palette 256 entries + tRNS + tEXt and tIME after IDAT",
|
||||
assemble(ihdr(2, 2, 8, 3), pal(256), (b"tRNS", bytes(256)), zero_idat(2, 2, 8, 1),
|
||||
(b"tEXt", b"Comment\0ok"), (b"tIME", struct.pack(">HBBBBB", 2026, 10, 22, 12, 0, 0)))),
|
||||
("palette depth 2 with 4 entries", assemble(ihdr(3, 1, 2, 3), pal(4), zero_idat(3, 1, 2, 1))),
|
||||
("RGB with 6-byte tRNS and gAMA", assemble(ihdr(1, 1, 8, 2), (b"gAMA", struct.pack(">I", 45455)),
|
||||
(b"tRNS", bytes(6)), zero_idat(1, 1, 8, 3))),
|
||||
("compressed zTXt (100 KB of text) and iTXt after IDAT",
|
||||
assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"zTXt", b"Comment\0\0" + zlib.compress(b"a" * 100_000)),
|
||||
(b"iTXt", b"Title\0\1\0ru\0\0" + zlib.compress("Привет".encode())))),
|
||||
]
|
||||
for name, data in good:
|
||||
att = upload(alice, chat["id"], data, filename="good.png").json
|
||||
got = call("GET", f"/api/v1/attachments/{att['id']}/content", 200, token=bob["token"])
|
||||
check(got.data == data, "valid PNG changed")
|
||||
print(f"{name} -> 201, downloaded byte for byte")
|
||||
print("PASS: chunk contents are checked, not only CRCs; valid PNGs with metadata are still accepted")
|
||||
|
||||
|
||||
def test_mime_substitution():
|
||||
alice, _, _, chat = setup()
|
||||
cases = [
|
||||
@@ -264,5 +332,5 @@ def test_aborted_upload():
|
||||
|
||||
if __name__ == "__main__":
|
||||
run_tests([test_png_and_jpeg_roundtrip, test_size_limit_exact_boundary, test_size_limit_streaming,
|
||||
test_pixel_limit, test_damaged_images, test_mime_substitution, test_filenames, test_access_control,
|
||||
test_pixel_limit, test_damaged_images, test_png_chunk_semantics, test_mime_substitution, test_filenames, test_access_control,
|
||||
test_message_attachments, test_idempotency_with_attachments, test_aborted_upload])
|
||||
|
||||
+183
-15
@@ -183,7 +183,12 @@ def inspect(upload: Upload, declared: str) -> ImageInfo:
|
||||
upload.file.seek(0)
|
||||
try:
|
||||
with _decode_slots:
|
||||
width, height = check_png(upload.file) if detected == PNG else check_jpeg(upload.file)
|
||||
if detected == PNG:
|
||||
width, height = check_png(upload.file)
|
||||
upload.file.seek(0)
|
||||
check_png_metadata(upload.file)
|
||||
else:
|
||||
width, height = check_jpeg(upload.file)
|
||||
except TooManyPixels:
|
||||
raise payload_too_large(f"Image has more than {settings.MAX_IMAGE_PIXELS} pixels") from None
|
||||
except Damaged as exc:
|
||||
@@ -262,13 +267,40 @@ class PngRows:
|
||||
return self.total == self.expected
|
||||
|
||||
|
||||
# Чанки, содержимое которых проверяется (держим в памяти; размер ограничен спецификацией):
|
||||
# IHDR — 13 байт, PLTE — до 256 x 3 = 768, tRNS — до 256 (палитра) / 6 (RGB).
|
||||
SMALL_CHUNK_LIMIT = {b"IHDR": 13, b"PLTE": 768, b"tRNS": 256}
|
||||
CRITICAL = (b"IHDR", b"PLTE", b"IDAT", b"IEND")
|
||||
# После данных изображения допустимы только время и текст (строгий список).
|
||||
AFTER_IDAT = {b"IEND", b"tIME", b"tEXt", b"zTXt", b"iTXt"}
|
||||
TEXT_CHUNKS = (b"tEXt", b"zTXt", b"iTXt")
|
||||
# Анимированные PNG (APNG) не поддерживаются: кадры — отдельная сущность, превью строится по одному кадру.
|
||||
APNG_CHUNKS = (b"acTL", b"fcTL", b"fdAT")
|
||||
# Как PngImagePlugin.MAX_TEXT_CHUNK в Pillow: больше 1 MiB распакованного текста в одном чанке — отказ.
|
||||
TEXT_LIMIT = 1024 * 1024
|
||||
# Чанки метаданных до IDAT Pillow (Image.open) читает целиком: ограничиваем, чтобы память не росла.
|
||||
# ICC-профили и EXIF реальных картинок — единицы-сотни KiB.
|
||||
PRE_IDAT_CHUNK_LIMIT = 2 * 1024 * 1024
|
||||
# Чанки, которые по спецификации обязаны стоять ДО первого IDAT (PNG 3rd ed., 5.6 «Chunk ordering»).
|
||||
# После IDAT допустимы только tIME, tEXt, zTXt, iTXt, кадры APNG и частные чанки.
|
||||
BEFORE_IDAT = {b"PLTE", b"tRNS", b"cHRM", b"cICP", b"gAMA", b"iCCP", b"mDCV", b"cLLI", b"sBIT", b"sRGB",
|
||||
b"bKGD", b"hIST", b"pHYs", b"sPLT", b"eXIf"}
|
||||
|
||||
|
||||
def check_png(file: BinaryIO) -> tuple[int, int]:
|
||||
"""Структура PNG по спецификации (PNG 3rd ed., гл. 5 и 11) без декодирования пикселей в память.
|
||||
|
||||
Кроме CRC каждого чанка проверяется СМЫСЛ критичных чанков и tRNS: CRC защищает только от
|
||||
случайной порчи, а намеренно собранный файл с верными CRC может быть некорректным.
|
||||
"""
|
||||
if file.read(8) != PNG_SIGNATURE:
|
||||
raise Damaged("signature")
|
||||
rows = inflater = None
|
||||
width = height = 0
|
||||
seen_idat = idat_finished = have_plte = False
|
||||
width = height = depth = 0
|
||||
color = None
|
||||
palette_entries = 0
|
||||
seen = set() # какие чанки уже встречались
|
||||
idat_started = idat_finished = False
|
||||
first = True
|
||||
while True:
|
||||
header = file.read(8)
|
||||
@@ -277,11 +309,37 @@ def check_png(file: BinaryIO) -> tuple[int, int]:
|
||||
length, kind = struct.unpack(">I4s", header)
|
||||
if length > 0x7FFFFFFF:
|
||||
raise Damaged("chunk length")
|
||||
if not kind.isalpha() or not kind.isascii():
|
||||
raise Damaged("chunk type")
|
||||
if first and kind != b"IHDR":
|
||||
raise Damaged("IHDR must be first")
|
||||
if kind[:1].isupper() and kind not in CRITICAL:
|
||||
raise Damaged(f"unknown critical chunk {kind!r}") # такой чанк декодер обязан отвергнуть
|
||||
if kind in SMALL_CHUNK_LIMIT and length > SMALL_CHUNK_LIMIT[kind]:
|
||||
raise Damaged(f"{kind!r} is too long")
|
||||
if kind == b"IDAT":
|
||||
if rows is None or idat_finished:
|
||||
raise Damaged("IDAT chunks must be consecutive")
|
||||
idat_started = True
|
||||
elif idat_started:
|
||||
idat_finished = True
|
||||
if kind in APNG_CHUNKS:
|
||||
raise Damaged("animated PNG is not supported")
|
||||
if kind in BEFORE_IDAT and idat_started:
|
||||
raise Damaged(f"{kind!r} after IDAT")
|
||||
if idat_finished and kind not in AFTER_IDAT:
|
||||
raise Damaged(f"{kind!r} after IDAT")
|
||||
if not idat_started and kind != b"IDAT" and length > PRE_IDAT_CHUNK_LIMIT:
|
||||
raise Damaged(f"{kind!r} metadata chunk is too large")
|
||||
if kind == b"tIME" and length != 7:
|
||||
raise Damaged("tIME size")
|
||||
if kind in (b"IHDR", b"PLTE", b"tRNS", b"IEND") and kind in seen:
|
||||
raise Damaged(f"duplicate {kind!r}")
|
||||
seen.add(kind)
|
||||
|
||||
crc = zlib.crc32(kind)
|
||||
keep = kind in (b"IHDR",)
|
||||
kept = b""
|
||||
text = TextChunk(kind) if kind in TEXT_CHUNKS else None
|
||||
left = length
|
||||
while left:
|
||||
piece = file.read(min(CRC_CHUNK, left))
|
||||
@@ -289,18 +347,20 @@ def check_png(file: BinaryIO) -> tuple[int, int]:
|
||||
raise Damaged("truncated chunk")
|
||||
left -= len(piece)
|
||||
crc = zlib.crc32(piece, crc)
|
||||
if keep:
|
||||
if kind in SMALL_CHUNK_LIMIT:
|
||||
kept += piece
|
||||
elif kind == b"IDAT":
|
||||
if rows is None or idat_finished:
|
||||
raise Damaged("IDAT out of order")
|
||||
feed_inflate(inflater, rows, piece)
|
||||
elif text is not None:
|
||||
text.feed(piece)
|
||||
if text is not None:
|
||||
text.finish()
|
||||
stored = file.read(4)
|
||||
if len(stored) < 4 or struct.unpack(">I", stored)[0] != crc & 0xFFFFFFFF:
|
||||
raise Damaged(f"CRC mismatch in {kind!r}")
|
||||
|
||||
if kind == b"IHDR":
|
||||
if not first or length != 13:
|
||||
if length != 13:
|
||||
raise Damaged("IHDR")
|
||||
width, height, depth, color, compression, filter_method, interlace = struct.unpack(">IIBBBBB", kept)
|
||||
if width > 0x7FFFFFFF or height > 0x7FFFFFFF:
|
||||
@@ -313,20 +373,34 @@ def check_png(file: BinaryIO) -> tuple[int, int]:
|
||||
rows = PngRows(width, height, depth, color, interlace)
|
||||
inflater = zlib.decompressobj()
|
||||
elif kind == b"PLTE":
|
||||
have_plte = True
|
||||
elif kind == b"IDAT":
|
||||
seen_idat = True
|
||||
# 1-256 записей по 3 байта; для палитровых — не больше 2^глубина; у серых палитры быть не может.
|
||||
if color in (0, 4):
|
||||
raise Damaged("PLTE in a grayscale image")
|
||||
if length == 0 or length % 3:
|
||||
raise Damaged("palette size")
|
||||
palette_entries = length // 3
|
||||
if color == 3 and palette_entries > 1 << depth:
|
||||
raise Damaged("palette larger than the bit depth allows")
|
||||
elif kind == b"tRNS":
|
||||
# Прозрачность: серое — 2 байта, RGB — 6, палитра — не больше записей PLTE; при альфа-канале запрещён.
|
||||
if color in (4, 6):
|
||||
raise Damaged("tRNS with an alpha channel")
|
||||
if color == 3 and not palette_entries:
|
||||
raise Damaged("tRNS before PLTE")
|
||||
expected_ok = {0: length == 2, 2: length == 6, 3: 1 <= length <= palette_entries}[color]
|
||||
if not expected_ok:
|
||||
raise Damaged("tRNS size")
|
||||
elif kind == b"IEND":
|
||||
if length != 0:
|
||||
raise Damaged("IEND with data")
|
||||
break
|
||||
if seen_idat and kind != b"IDAT":
|
||||
idat_finished = True
|
||||
first = False
|
||||
|
||||
if file.read(1):
|
||||
raise Damaged("data after IEND")
|
||||
if color == 3 and not have_plte:
|
||||
if color == 3 and not palette_entries:
|
||||
raise Damaged("palette image without PLTE")
|
||||
if not seen_idat:
|
||||
if not idat_started:
|
||||
raise Damaged("no IDAT")
|
||||
tail = inflater.flush(INFLATE_CHUNK)
|
||||
if tail:
|
||||
@@ -336,6 +410,100 @@ def check_png(file: BinaryIO) -> tuple[int, int]:
|
||||
return width, height
|
||||
|
||||
|
||||
def check_png_metadata(file: BinaryIO) -> None:
|
||||
"""Второе мнение: стандартный парсер Pillow разбирает метаданные до IDAT (gAMA, iCCP, pHYs, tEXt…),
|
||||
НЕ читая пиксели. Только open, без verify(): verify читает каждый чанк, включая IDAT до 10 MiB,
|
||||
целиком в память, а CRC и данные после IDAT уже потоково проверил check_png.
|
||||
Нужен для вспомогательных чанков: по спецификации их можно игнорировать, но если Pillow
|
||||
не может их разобрать, файл нельзя будет обработать (превью в lab5) — лучше отклонить сразу.
|
||||
"""
|
||||
with Image.open(file, formats=["PNG"]):
|
||||
pass # open разбирает все чанки до первого IDAT; после IDAT check_png пускает только время и текст
|
||||
|
||||
|
||||
class TextChunk:
|
||||
"""Потоковая проверка tEXt / zTXt / iTXt: ключ 1-79 байт, метод сжатия 0, сжатый текст —
|
||||
корректный zlib не больше TEXT_LIMIT после распаковки (распаковка кусками, текст не хранится)."""
|
||||
|
||||
def __init__(self, kind: bytes):
|
||||
self.kind = kind
|
||||
self.head = b""
|
||||
self.done_header = False
|
||||
self.inflater = None
|
||||
self.size = 0
|
||||
|
||||
def feed(self, piece: bytes) -> None:
|
||||
if not self.done_header:
|
||||
self.head += piece
|
||||
rest = self._header()
|
||||
if rest is None:
|
||||
if len(self.head) > 4096:
|
||||
raise Damaged("text chunk header")
|
||||
return
|
||||
piece = rest
|
||||
if self.inflater is not None and piece:
|
||||
self._inflate(piece)
|
||||
|
||||
def _keyword_end(self) -> int | None:
|
||||
end = self.head.find(b"\x00")
|
||||
if end == -1:
|
||||
return None
|
||||
if not 1 <= end <= 79:
|
||||
raise Damaged("text keyword")
|
||||
return end
|
||||
|
||||
def _header(self) -> bytes | None:
|
||||
end = self._keyword_end()
|
||||
if end is None:
|
||||
return None
|
||||
h = self.head
|
||||
if self.kind == b"tEXt":
|
||||
self.done_header = True
|
||||
return b""
|
||||
if self.kind == b"zTXt":
|
||||
if len(h) < end + 2:
|
||||
return None
|
||||
if h[end + 1] != 0:
|
||||
raise Damaged("zTXt compression method")
|
||||
self.done_header, self.inflater = True, zlib.decompressobj()
|
||||
return h[end + 2:]
|
||||
# iTXt: keyword \0 flag method language \0 translated-keyword \0 text
|
||||
if len(h) < end + 3:
|
||||
return None
|
||||
flag, method = h[end + 1], h[end + 2]
|
||||
if flag not in (0, 1) or (flag and method != 0):
|
||||
raise Damaged("iTXt compression")
|
||||
lang_end = h.find(b"\x00", end + 3)
|
||||
tkey_end = h.find(b"\x00", lang_end + 1) if lang_end != -1 else -1
|
||||
if tkey_end == -1:
|
||||
return None
|
||||
self.done_header = True
|
||||
if flag:
|
||||
self.inflater = zlib.decompressobj()
|
||||
return h[tkey_end + 1:]
|
||||
|
||||
def _inflate(self, data: bytes) -> None:
|
||||
if self.inflater.eof:
|
||||
raise Damaged("data after compressed text")
|
||||
out = self.inflater.decompress(data, INFLATE_CHUNK)
|
||||
self.size += len(out)
|
||||
while self.inflater.unconsumed_tail and not self.inflater.eof and self.size <= TEXT_LIMIT:
|
||||
out = self.inflater.decompress(self.inflater.unconsumed_tail, INFLATE_CHUNK)
|
||||
self.size += len(out)
|
||||
if self.size > TEXT_LIMIT:
|
||||
raise Damaged("compressed text is too large")
|
||||
if self.inflater.eof and self.inflater.unused_data:
|
||||
raise Damaged("data after compressed text")
|
||||
|
||||
def finish(self) -> None:
|
||||
if not self.done_header:
|
||||
raise Damaged("text chunk header")
|
||||
if self.inflater is not None:
|
||||
self.size += len(self.inflater.flush(INFLATE_CHUNK))
|
||||
if not self.inflater.eof or self.size > TEXT_LIMIT:
|
||||
raise Damaged("compressed text")
|
||||
|
||||
|
||||
def feed_inflate(inflater, rows: PngRows, data: bytes) -> None:
|
||||
"""Распаковка кусками не больше 64 KiB: «zip-бомба» не раздует память."""
|
||||
if inflater.eof:
|
||||
|
||||
Reference in New Issue
Block a user