4: исправление по ревью — проверка смысла чанков PNG (палитра, tRNS, порядок, текст), регрессионные тесты, evidence

This commit is contained in:
kalibr456
2026-10-22 14:30:53 +03:00
parent 5c3a31db2e
commit 4d4b914676
26 changed files with 587 additions and 279 deletions
+16 -15
View File
@@ -91,7 +91,7 @@ s3-init: одноразово создаёт приватный bucket и пр
3. Тип по сигнатуре (`\x89PNG...`, `FF D8 FF`), не по заголовку. Не PNG/JPEG -> 415; заявленный тип не совпадает с содержимым -> 415.
4. Размер в пикселях читается из заголовка (IHDR / SOF) и проверяется ДО распаковки: больше 40 000 000 -> 413.
5. Проверка целостности с ограниченной памятью:
- PNG — своя потоковая проверка (`uploads.check_png`): CRC каждого чанка, порядок IHDR/PLTE/IDAT/IEND, допустимые тип цвета и глубина, ничего после IEND; IDAT распаковывается zlib кусками по 64 KiB (вывод не хранится), считается точный объём данных по IHDR (с учётом Adam7) и проверяется байт фильтра каждой строки. «Zip-бомба» не раздувает память: распаковка идёт порциями и обрывается, как только данных больше, чем обещал заголовок.
- PNG — своя потоковая проверка (`uploads.check_png`): CRC каждого чанка и смысл критичных чанков по спецификации PNG (размер палитры 1–256 записей и не больше 2^глубина, размер tRNS по типу цвета, пустой IEND, порядок чанков: всё, что обязано идти до IDAT, — до него, IDAT подряд, неизвестный критичный чанк запрещён); IDAT распаковывается кусками по 64 KiB с подсчётом точного объёма и проверкой байта фильтра каждой строки. Затем Pillow разбирает метаданные до IDAT (Image.open, без verify: verify читает каждый чанк, включая IDAT до 10 MiB, в память). После IDAT допускаются только tIME, tEXt, zTXt, iTXt; текстовые чанки проверяются потоково (ключ 1–79 байт, корректный zlib, не больше 1 MiB распакованного текста); анимированные PNG (acTL/fcTL/fdAT) отклоняются; чанк метаданных до IDAT — не больше 2 MiB. Исправлено по замечанию: раньше проверялись только CRC, и PNG с палитрой из 257 записей получал 201.
- JPEG — Pillow декодирует весь поток, но в масштабе 1/8 (`draft`): 40 Мпикс -> ~0,6 Мпикс, около 2 MiB.
- Одновременно в реплике проверяется не больше `DECODE_CONCURRENCY=2` файлов (CPU).
Битый файл -> 415. Любая ошибка разбора ловится и превращается в 415, а не 500. Проверка PNG сверена с Pillow: 400 случайных корректных PNG разных типов приняты, 400 испорченных (обрезка, перевёрнутый бит) отклонены.
@@ -187,36 +187,36 @@ S3-клиент: connect timeout 1 с, read timeout 5 с, всего 2 попы
Методика памяти (`student-tests/upload_memory.py`): три сценария — 10 x PNG из случайного шума ~9,5 MiB (почти несжимаемый, максимум трафика); 10 x PNG 8000x5000 RGB (40 Мпикс, максимум распаковки); 20 смешанных. Перед каждым сценарием `docker compose restart api` (сбрасывает пик), прогрев 8 загрузками, затем все потоки стартуют одновременно (barrier) через балансировщик. Пик RSS — `VmHWM` из `/proc/1/status` процесса API (максимум за жизнь процесса по данным ядра, короткие всплески не теряются), дополнительно пик памяти cgroup контейнера и опрос раз в ~0,1–0,5 с. Проверяются коды 201 и SHA-256 скачанных файлов. Клиент — тот же компьютер, поэтому время загрузки ограничено и клиентом; это замер памяти, а не пропускной способности.
<!-- measurements:begin -->
Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-21, commit 2dbed80 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB.
Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-22, commit c942c61 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB.
Публичный тест отказа (`make test-failover`, 15 с записи во время остановки реплики):
| Прогон | Остановлена реплика | Реплик до отказа | Успешных записей | Неудачных попыток (повторены с тем же ключом) | Первая запись после hook, с |
| --- | --- | --- | --- | --- | --- |
| 1 | 11a93acecaca | 2 | 126 | 0 | 4.39 |
| 2 | 6128eb268c44 | 2 | 124 | 0 | 4.328 |
| 1 | 343119df2488 | 2 | 123 | 0 | 4.422 |
| 2 | d590adc21943 | 2 | 124 | 0 | 4.531 |
Одновременные загрузки (`student-tests/upload_memory.py`, перед каждым сценарием реплики перезапускаются, лимит api 512 MiB / 1 CPU). Пик RSS — VmHWM процесса API; cgroup — память контейнера, включая page cache временных файлов:
| Сценарий | Одновременно | Объём, MiB | Время, с (p50 / max) | Реплика: загрузок | RSS до, MiB | Пик RSS, MiB | Рост, MiB | Пик cgroup, MiB |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 1.38 / 1.52 | 11a93acecaca: 5 | 93.5 | 106.2 | 12.6 | 134.1 |
| | | | | 6128eb268c44: 5 | 94.8 | 105.3 | 10.5 | 135.2 |
| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 11a93acecaca: 5 | 93.2 | 96.6 | 3.4 | 84.3 |
| | | | | 6128eb268c44: 5 | 93.1 | 94.7 | 1.6 | 86.8 |
| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.37 / 1.77 | 11a93acecaca: 10 | 94.8 | 121.7 | 26.9 | 181.9 |
| | | | | 6128eb268c44: 10 | 95.0 | 116.1 | 21.0 | 155.6 |
| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 2.87 / 3.52 | 343119df2488: 5 | 94.8 | 107.6 | 12.8 | 136.0 |
| | | | | d590adc21943: 5 | 91.2 | 104.5 | 13.3 | 136.2 |
| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 343119df2488: 5 | 95.1 | 96.6 | 1.5 | 86.5 |
| | | | | d590adc21943: 5 | 92.1 | 96.6 | 4.6 | 83.5 |
| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.5 / 1.78 | 343119df2488: 10 | 93.2 | 116.2 | 23.0 | 158.2 |
| | | | | d590adc21943: 10 | 95.0 | 118.6 | 23.6 | 167.8 |
Один PNG 8000x5000 RGB внутри контейнера api, дополнительный пик RSS: наивно (тело целиком в памяти + полное декодирование Pillow) 161 MiB, потоковая проверка 0 MiB. 10 таких наивных загрузок одновременно не поместились бы в 512 MiB.
Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 245.844 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс.
Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 239.325 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.39, 4.328 с (hook ждёт остановки ~4 с); потерь нет |
| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 39.2 с до готовности; потерь нет |
| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 20141 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет |
| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14766 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет |
| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.422, 4.531 с (hook ждёт остановки ~4 с); потерь нет |
| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 47.0 с до готовности; потерь нет |
| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 19453 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет |
| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14891 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет |
| падение между PUT в S3 и COMMIT в БД (код API в контейнере, finish_upload падает) | вложение не видно, janitor не трогает его до конца grace period, потом удаляет объект и строку | GET/прикрепление -> 404; janitor: свежая строка сохранена, dry-run только отчёт, затем удалено | мусора не осталось; повторный проход ничего не делает |
| обрыв загрузки клиентом (9 МБ заявлено, отправлено 0,2 и 3 МБ) | ничего не сохраняется | строк и объектов не прибавилось, временных файлов 0 | мгновенно |
<!-- measurements:end -->
@@ -231,5 +231,6 @@ S3-клиент: connect timeout 1 с, read timeout 5 с, всего 2 попы
- Обход bucket janitor’ом держит в памяти список ключей (для учебного объёма — мегабайты); для миллионов объектов нужна сверка по диапазонам ключей или S3 Inventory.
- Временный файл загрузки (часть тела сверх 256 KiB) лежит на диске контейнера. В Linux `tempfile.TemporaryFile` создаётся уже без имени в каталоге (O_TMPFILE), поэтому даже при SIGKILL реплики место освобождается ядром, «забытых» файлов не бывает. Одновременно на диске может быть до 10 MiB на каждую идущую загрузку; лимита на число одновременных загрузок, кроме лимитов HAProxy/uvicorn, нет.
- Проверяется структура изображения, а не его безопасность для всех декодеров (например, JPEG с некорректными, но декодируемыми данными пройдёт). Перекодирование оригинала — задача lab5.
- PNG с частными служебными чанками после IDAT и анимированные PNG отклоняются (строже спецификации, на практике не встречаются в фотографиях).
- MinIO собирался быть первым выбором, но его образы больше не доступны в публичных реестрах: закреплённый тег защищает от неожиданного обновления, но не от исчезновения образа. Для эксплуатации нужно зеркалить образы в свой реестр. SeaweedFS запущен одним процессом `weed server` на одном диске — это SPOF и не защита от потери диска; веб-интерфейсы master/filer без авторизации, закрыты только сетью.
- Скрипты `failover_metrics.py`, `rolling_update.py`, `test_expiry.py` из lab3 остались в `student-tests/` и работают со стендом, но в общий прогон lab4 не входят (это уровень 4 lab3, а не lab4).
+33 -4
View File
@@ -1,7 +1,7 @@
# attachments: lab4: files, limits, damaged images, MIME, filenames, access, attachment_ids
# date: 2026-10-21 21:03:05, commit: 2dbed80
# date: 2026-10-22 14:19:26, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/test_attachments.py
# exit code: 0, 4.0 s
# exit code: 0, 5.4 s
sample.png: 111817 bytes, sha256 99ea7533c441..., downloaded by the other member, same bytes
sample.jpg: 1492 bytes, sha256 c310b9a1c208..., downloaded by the other member, same bytes
@@ -14,7 +14,7 @@ exactly 10485760 bytes (10 MiB) -> 201
PASS: the 10 MiB limit is exact
12 MiB without Content-Length (chunked) -> 413 as soon as the 10 MiB limit is passed
PASS: oversized streamed body is cut off, the API keeps working
8000x5000 = 40 000 000 pixels (38957 bytes compressed) -> 201 5ffe6847
8000x5000 = 40 000 000 pixels (38957 bytes compressed) -> 201 af872a80
8001x5000 = 40 005 000 pixels (38962 bytes) -> 413
header claims 100000x100000 in 65 bytes -> 413 from the header, nothing is decoded
PASS: pixel limit is checked before decompression
@@ -30,6 +30,35 @@ random bytes (1600 bytes, image/png) -> 415
plain text (28 bytes, image/jpeg) -> 415
empty body -> 400
PASS: damaged and foreign files are rejected with 415, never 500
PLTE 771 bytes = 257 entries (review case) (850 bytes, valid CRCs) -> 415
PLTE length not a multiple of 3 (83 bytes, valid CRCs) -> 415
empty PLTE (79 bytes, valid CRCs) -> 415
palette larger than bit depth allows (depth 1, 3 entries) (88 bytes, valid CRCs) -> 415
palette image without PLTE (67 bytes, valid CRCs) -> 415
PLTE in a grayscale image (85 bytes, valid CRCs) -> 415
PLTE after IDAT (85 bytes, valid CRCs) -> 415
two PLTE chunks (103 bytes, valid CRCs) -> 415
tRNS longer than the palette (100 bytes, valid CRCs) -> 415
tRNS before PLTE (98 bytes, valid CRCs) -> 415
tRNS of wrong size for RGB (4 bytes instead of 6) (85 bytes, valid CRCs) -> 415
tRNS in an image with alpha channel (82 bytes, valid CRCs) -> 415
IEND with data (70 bytes, valid CRCs) -> 415
unknown critical chunk (81 bytes, valid CRCs) -> 415
IDAT chunks not consecutive (95 bytes, valid CRCs) -> 415
gAMA after IDAT (85 bytes, valid CRCs) -> 415
gAMA of 2 bytes instead of 4 (83 bytes, valid CRCs) -> 415
3 MiB metadata chunk before IDAT (3145809 bytes, valid CRCs) -> 415
private chunk after IDAT (181 bytes, valid CRCs) -> 415
animated PNG frame (fdAT) (91 bytes, valid CRCs) -> 415
tIME of 2 bytes instead of 7 (83 bytes, valid CRCs) -> 415
tEXt without keyword separator (86 bytes, valid CRCs) -> 415
zTXt with broken zlib after IDAT (91 bytes, valid CRCs) -> 415
zTXt text bomb: 50 MB of text in ~50 KB after IDAT (48694 bytes, valid CRCs) -> 415
palette 256 entries + tRNS + tEXt and tIME after IDAT -> 201, downloaded byte for byte
palette depth 2 with 4 entries -> 201, downloaded byte for byte
RGB with 6-byte tRNS and gAMA -> 201, downloaded byte for byte
compressed zTXt (100 KB of text) and iTXt after IDAT -> 201, downloaded byte for byte
PASS: chunk contents are checked, not only CRCs; valid PNGs with metadata are still accepted
PNG bytes sent as image/jpeg -> 415
JPEG bytes sent as image/png -> 415
GIF bytes sent as image/png -> 415
@@ -64,4 +93,4 @@ PASS: Idempotency-Key covers attachment_ids
sent 1.1 MB of a declared 5 MB body and closed the connection
the API keeps serving uploads; nothing of the aborted request is visible (DB/S3 side is checked by test_consistency.py)
PASS: an interrupted upload is dropped
11 passed, 0 failed
12 passed, 0 failed
+8 -8
View File
@@ -1,18 +1,18 @@
# balancing: balancing, shared sessions/data/files, X-Instance-Id
# date: 2026-10-21 21:08:50, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:22:27, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/test_balancing.py
# exit code: 0, 1.9 s
# exit code: 0, 2.0 s
20 x GET /users/me with one token -> replicas: {'6128eb268c44': 10, '11a93acecaca': 10}
20 x GET /users/me with one token -> replicas: {'d590adc21943': 10, '343119df2488': 10}
PASS: one session works on every replica, no second login (sessions are shared in Redis)
message written by 11a93acecaca; read back by ['11a93acecaca', '6128eb268c44']
message written by 343119df2488; read back by ['343119df2488', 'd590adc21943']
PASS: a message written via one replica is read right away via another (shared PostgreSQL)
logout on 6128eb268c44; old token then rejected by ['11a93acecaca', '6128eb268c44']
logout on d590adc21943; old token then rejected by ['343119df2488', 'd590adc21943']
PASS: logout on one replica revokes the session on all of them
first POST on 6128eb268c44 -> 201 296cb1bf-8c1c-46f2-942a-4865d1b7d6af; same key on 11a93acecaca -> 201 296cb1bf-8c1c-46f2-942a-4865d1b7d6af
first POST on d590adc21943 -> 201 6c2f624d-c728-4738-b486-b17ab735c0ae; same key on 343119df2488 -> 201 6c2f624d-c728-4738-b486-b17ab735c0ae
PASS: Idempotency-Key is shared: a retry through another replica returns the same message
file uploaded via 6128eb268c44; downloaded byte for byte via ['11a93acecaca', '6128eb268c44']
file uploaded via d590adc21943; downloaded byte for byte via ['343119df2488', 'd590adc21943']
PASS: a file uploaded through one replica is downloaded through another (shared S3, nothing local)
statuses [200, 200, 201, 400, 401, 404, 409]: every answer has X-Instance-Id of a running container ['11a93acecaca', '6128eb268c44']; api containers: ['11a93acecaca', '6128eb268c44']
statuses [200, 200, 201, 400, 401, 404, 409]: every answer has X-Instance-Id of a running container ['343119df2488', 'd590adc21943']; api containers: ['343119df2488', 'd590adc21943']
PASS: X-Instance-Id is in every response and points to a real container
6 passed, 0 failed
+1 -1
View File
@@ -1,5 +1,5 @@
# check: offline contract check (make check)
# date: 2026-10-21 21:02:57, commit: 2dbed80
# date: 2026-10-22 14:19:16, commit: c942c61 + uncommitted changes
# command: python.exe scripts/check_contract.py
# exit code: 0, 0.1 s
+18 -18
View File
@@ -1,47 +1,47 @@
# consistency: lab4: aborted upload, crash between S3 and DB, janitor, orphans, S3 outage
# date: 2026-10-21 21:08:14, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:21:48, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/test_consistency.py
# exit code: 0, 148.7 s
# exit code: 0, 133.0 s
declared 9 000 000 bytes, sent 211817, closed the connection
declared 9 000 000 bytes, sent 3111817, closed the connection
rows in attachments: 32 -> 32; objects in bucket: 31 -> 31
temporary upload files in api containers: {'11a93acecaca': 0, '6128eb268c44': 0} (removed when the request ends)
rows in attachments: 448 -> 448; objects in bucket: 444 -> 444
temporary upload files in api containers: {'343119df2488': 0, 'd590adc21943': 0} (removed when the request ends)
api containers have no volumes: the original is never stored locally
PASS: an interrupted upload leaves no row, no object and no local file
crash after S3 PUT: row 04aacbf6 state=uploading, object originals/219175ec1d5647f0aeaec84ee8243cb3 exists in S3
crash after S3 PUT: row 9aa0bc8a state=uploading, object originals/8ae03455b55a4f798b7bc6248b450ef9 exists in S3
the half-written attachment is invisible: GET -> 404, attach to a message -> 404
janitor with default grace (3600 s) right away: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 33, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.101} -> fresh row and object are kept (the upload might still be running)
row made 2 h old; janitor --dry-run: {"dry_run": true, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 0, "objects_scanned": 33, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.092} -> nothing deleted
janitor: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 32, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.096} -> row and object deleted
janitor again: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 32, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.103} -> idempotent
janitor with default grace (3600 s) right away: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 446, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.133} -> fresh row and object are kept (the upload might still be running)
row made 2 h old; janitor --dry-run: {"dry_run": true, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 0, "objects_scanned": 446, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.184} -> nothing deleted
janitor: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 445, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.132} -> row and object deleted
janitor again: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 445, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.179} -> idempotent
the ready attachment of the same chat is untouched
PASS: S3 written but DB not committed -> invisible, reclaimed by janitor only after the grace period
object originals/867f5c5d42bf4bf9973b60ef8856d88f put into the bucket without a row in PostgreSQL
janitor --grace-seconds 3 immediately: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 34, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.102} -> fresh object kept
after 4 s, --dry-run: {"dry_run": true, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 34, "orphan_objects": 1, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.098} -> reported, kept
janitor: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 34, "orphan_objects": 1, "orphan_objects_deleted": 1, "missing_objects": 1, "seconds": 0.1} -> orphan deleted
object originals/101748e3a911439a9c1e11e768c99822 put into the bucket without a row in PostgreSQL
janitor --grace-seconds 3 immediately: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 447, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.191} -> fresh object kept
after 4 s, --dry-run: {"dry_run": true, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 447, "orphan_objects": 1, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.188} -> reported, kept
janitor: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 447, "orphan_objects": 1, "orphan_objects_deleted": 1, "missing_objects": 4, "seconds": 0.1} -> orphan deleted
ready attachments older than the grace period are untouched (they have rows)
object of a ready attachment deleted behind the API's back: janitor reports missing_objects=2 and keeps the row; download -> 503 (not 500)
object of a ready attachment deleted behind the API's back: janitor reports missing_objects=5 and keeps the row; download -> 503 (not 500)
PASS: orphans are found and deleted only after the grace period; missing objects are reported
--- S3 stopped (connection refused / name not resolved): docker compose stop s3
upload -> 503 unavailable in 20141 ms, no storage details in the body
upload -> 503 unavailable in 19453 ms, no storage details in the body
download of an existing file -> 503
metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes
uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API
docker compose start s3
after S3 recovery the old file downloads byte for byte
new upload -> 201
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 35, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.099}
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 448, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 5, "seconds": 0.191}
PASS: S3 stopped (connection refused / name not resolved) -> 503 for file operations only, no visible half-uploads, everything cleaned up
--- S3 frozen (no answer, timeouts): docker compose pause s3
upload -> 503 unavailable in 14766 ms, no storage details in the body
upload -> 503 unavailable in 14891 ms, no storage details in the body
download of an existing file -> 503
metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes
uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API
docker compose unpause s3
after S3 recovery the old file downloads byte for byte
new upload -> 201
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 37, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.094}
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 450, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 5, "seconds": 0.115}
PASS: S3 frozen (no answer, timeouts) -> 503 for file operations only, no visible half-uploads, everything cleaned up
5 passed, 0 failed
+4 -4
View File
@@ -1,9 +1,9 @@
# db: PostgreSQL: rollback, FK, Argon2id, unique keys, migrations
# date: 2026-10-21 21:09:10, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:22:40, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/test_db.py
# exit code: 0, 14.2 s
# exit code: 0, 6.2 s
chat with unknown member -> 400; rows with this title: 0; chats without members: 0; chats insert counter: 57 -> 58
chat with unknown member -> 400; rows with this title: 0; chats without members: 0; chats insert counter: 335 -> 336
PASS: chat row was inserted and then rolled back together with members
direct INSERT of a message from a non-member -> exit code 1: ERROR: insert or update on table "messages" violates foreign key constraint "messages_chat_id_sender_id_fkey"
PASS: the database itself rejects a message from a non-member
@@ -14,7 +14,7 @@ PASS: username is unique in the database
applied migrations: 001_init.sql,002_text_as_bytes.sql,003_idempotency.sql,004_attachments.sql; running the migrate service once more
after the second run: 001_init.sql,002_text_as_bytes.sql,003_idempotency.sql,004_attachments.sql; log: Migrations applied: none (schema is up to date)
PASS: migrations are applied once, a repeated run does nothing
api replicas that checked the schema at start: 8; api log lines 'Migrations applied': 0; migrate service: Exited (0) 6 minutes ago
api replicas that checked the schema at start: 14; api log lines 'Migrations applied': 0; migrate service: Exited (0) 4 minutes ago
PASS: only the migrate service applies migrations, replicas wait for the schema
unique constraint for Idempotency-Key: UNIQUE (chat_id, sender_id, idempotency_key)
PASS: one message per (chat, sender, Idempotency-Key) is enforced by the database
+46 -41
View File
@@ -1,7 +1,7 @@
=== Data volume
messages_total | messages_in_big_chat
----------------+----------------------
20146 | 15000
35618 | 15000
=== History index
@@ -13,39 +13,39 @@
=== 1. First page: limit=50, the app asks for 51 rows to know if there is a next page
QUERY PLAN
------------------------------------------------------------------------------------------------
Limit (actual time=0.005..0.011 rows=51 loops=1)
Buffers: shared hit=5
Limit (actual time=0.006..0.010 rows=51 loops=1)
Buffers: shared hit=4
-> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1)
Index Cond: (chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid)
Buffers: shared hit=5
Index Cond: (chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid)
Buffers: shared hit=4
Planning:
Buffers: shared hit=6
Planning Time: 0.060 ms
Buffers: shared hit=8
Planning Time: 0.071 ms
Execution Time: 0.020 ms
=== 2. Page after the 10000th message by keyset cursor (this is what the app does)
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (actual time=0.005..0.012 rows=51 loops=1)
Buffers: shared hit=6
-> Index Scan using messages_history on messages (actual time=0.005..0.009 rows=51 loops=1)
Index Cond: ((chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '170ea586-3f7e-4314-a9bf-8e4f394941ac'::uuid)))
Buffers: shared hit=6
Planning Time: 0.044 ms
Execution Time: 0.031 ms
Limit (actual time=0.006..0.011 rows=51 loops=1)
Buffers: shared hit=5
-> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1)
Index Cond: ((chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '4bd22867-547f-4ecf-8182-443fbb004188'::uuid)))
Buffers: shared hit=5
Planning Time: 0.040 ms
Execution Time: 0.029 ms
=== 2b. The same page exactly as the app sends it since lab4: with attachment_ids of every message
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (actual time=0.020..0.225 rows=51 loops=1)
Buffers: shared hit=60
-> Index Scan using messages_history on messages m (actual time=0.019..0.222 rows=51 loops=1)
Index Cond: ((chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '170ea586-3f7e-4314-a9bf-8e4f394941ac'::uuid)))
Buffers: shared hit=60
Limit (actual time=0.019..0.208 rows=51 loops=1)
Buffers: shared hit=59
-> Index Scan using messages_history on messages m (actual time=0.019..0.204 rows=51 loops=1)
Index Cond: ((chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '4bd22867-547f-4ecf-8182-443fbb004188'::uuid)))
Buffers: shared hit=59
SubPlan 1
-> Sort (actual time=0.003..0.003 rows=0 loops=51)
-> Sort (actual time=0.002..0.002 rows=0 loops=51)
Sort Key: a."position"
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=54
@@ -57,39 +57,44 @@
Buffers: shared hit=51
Planning:
Buffers: shared hit=63
Planning Time: 0.157 ms
Execution Time: 0.242 ms
Planning Time: 0.156 ms
Execution Time: 0.225 ms
=== 3. The same page with OFFSET 10000 (the app does NOT do this)
QUERY PLAN
---------------------------------------------------------------------------------------------------
Limit (actual time=0.878..0.884 rows=51 loops=1)
Buffers: shared hit=258
-> Index Scan using messages_history on messages (actual time=0.007..0.669 rows=10051 loops=1)
Index Cond: (chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid)
Buffers: shared hit=258
Planning Time: 0.037 ms
Execution Time: 0.897 ms
QUERY PLAN
--------------------------------------------------------------------------------
Limit (actual time=3.056..3.060 rows=51 loops=1)
Buffers: shared hit=585
-> Sort (actual time=2.647..2.857 rows=10051 loops=1)
Sort Key: created_at, id
Sort Method: quicksort Memory: 2377kB
Buffers: shared hit=585
-> Seq Scan on messages (actual time=0.005..1.278 rows=15000 loops=1)
Filter: (chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid)
Rows Removed by Filter: 20618
Buffers: shared hit=585
Planning Time: 0.033 ms
Execution Time: 3.107 ms
=== 4. Query 2 without the index (index is dropped only inside this transaction)
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (actual time=1.097..1.101 rows=51 loops=1)
Buffers: shared hit=261
-> Sort (actual time=1.097..1.098 rows=51 loops=1)
Limit (actual time=1.432..1.436 rows=51 loops=1)
Buffers: shared hit=585
-> Sort (actual time=1.432..1.433 rows=51 loops=1)
Sort Key: created_at, id
Sort Method: top-N heapsort Memory: 31kB
Buffers: shared hit=261
-> Seq Scan on messages (actual time=0.307..0.737 rows=5000 loops=1)
Filter: ((chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '170ea586-3f7e-4314-a9bf-8e4f394941ac'::uuid)))
Rows Removed by Filter: 15146
Buffers: shared hit=261
Buffers: shared hit=585
-> Seq Scan on messages (actual time=0.678..1.116 rows=5000 loops=1)
Filter: ((chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '4bd22867-547f-4ecf-8182-443fbb004188'::uuid)))
Rows Removed by Filter: 30618
Buffers: shared hit=585
Planning:
Buffers: shared hit=4
Planning Time: 0.051 ms
Execution Time: 1.115 ms
Planning Time: 0.049 ms
Execution Time: 1.450 ms
=== ROLLBACK: test data removed, index is back
+22 -21
View File
@@ -1,43 +1,44 @@
# explain: EXPLAIN of the history page (full plans: explain-history.txt)
# date: 2026-10-21 21:09:14, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:22:45, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/explain.py
# exit code: 0, 1.1 s
=== Data volume
=== History index
=== 1. First page: limit=50, the app asks for 51 rows to know if there is a next page
Buffers: shared hit=4
-> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1)
Buffers: shared hit=4
Buffers: shared hit=8
Execution Time: 0.020 ms
=== 2. Page after the 10000th message by keyset cursor (this is what the app does)
Buffers: shared hit=5
-> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1)
Buffers: shared hit=5
Buffers: shared hit=6
Execution Time: 0.020 ms
=== 2. Page after the 10000th message by keyset cursor (this is what the app does)
Buffers: shared hit=6
-> Index Scan using messages_history on messages (actual time=0.005..0.009 rows=51 loops=1)
Buffers: shared hit=6
Execution Time: 0.031 ms
Execution Time: 0.029 ms
=== 2b. The same page exactly as the app sends it since lab4: with attachment_ids of every message
Buffers: shared hit=60
-> Index Scan using messages_history on messages m (actual time=0.019..0.222 rows=51 loops=1)
Buffers: shared hit=60
Buffers: shared hit=59
-> Index Scan using messages_history on messages m (actual time=0.019..0.204 rows=51 loops=1)
Buffers: shared hit=59
Buffers: shared hit=54
-> Bitmap Heap Scan on message_attachments a (actual time=0.000..0.000 rows=0 loops=51)
Buffers: shared hit=51
-> Bitmap Index Scan on message_attachments_message_id_attachment_id_key (actual time=0.000..0.000 rows=0 loops=51)
Buffers: shared hit=51
Buffers: shared hit=63
Execution Time: 0.242 ms
Execution Time: 0.225 ms
=== 3. The same page with OFFSET 10000 (the app does NOT do this)
Buffers: shared hit=258
-> Index Scan using messages_history on messages (actual time=0.007..0.669 rows=10051 loops=1)
Buffers: shared hit=258
Execution Time: 0.897 ms
Buffers: shared hit=585
Buffers: shared hit=585
-> Seq Scan on messages (actual time=0.005..1.278 rows=15000 loops=1)
Buffers: shared hit=585
Execution Time: 3.107 ms
=== 4. Query 2 without the index (index is dropped only inside this transaction)
Buffers: shared hit=261
Buffers: shared hit=261
-> Seq Scan on messages (actual time=0.307..0.737 rows=5000 loops=1)
Buffers: shared hit=261
Buffers: shared hit=585
Buffers: shared hit=585
-> Seq Scan on messages (actual time=0.678..1.116 rows=5000 loops=1)
Buffers: shared hit=585
Buffers: shared hit=4
Execution Time: 1.115 ms
Execution Time: 1.450 ms
=== ROLLBACK: test data removed, index is back
PASS: full plans saved to evidence\explain-history.txt
+17 -17
View File
@@ -1,33 +1,33 @@
# failover: public failover test for every replica, file downloaded via the other one (make test-failover)
# date: 2026-10-21 21:09:54, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:23:28, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/run_failover.py
# exit code: 0, 39.7 s
# exit code: 0, 43.2 s
hook: scripts\stop-one.cmd; api replicas: ['11a93acecaca', '6128eb268c44']
hook: scripts\stop-one.cmd; api replicas: ['343119df2488', 'd590adc21943']
=== run 1: api containers before
SERVICE CONTAINER ID STATUS
api 11a93acecaca Up 36 seconds (healthy)
api 6128eb268c44 Up 36 seconds (healthy)
api 343119df2488 Up 27 seconds (healthy)
api d590adc21943 Up 27 seconds (healthy)
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.␍
{"observed_instances_before": 2, "successful_writes": 126, "failed_attempts": 0, "first_recovery_seconds": 4.39}␍
{"observed_instances_before": 2, "successful_writes": 123, "failed_attempts": 0, "first_recovery_seconds": 4.422}␍
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
api containers after
SERVICE CONTAINER ID STATUS
api 11a93acecaca Exited (0) 11 seconds ago
api 6128eb268c44 Up 53 seconds (healthy)
the hook stopped 11a93acecaca (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched
restored: docker start 11a93acecaca -> healthy and back in HAProxy
api 343119df2488 Exited (0) 11 seconds ago
api d590adc21943 Up 43 seconds (healthy)
the hook stopped 343119df2488 (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched
restored: docker start 343119df2488 -> healthy and back in HAProxy
=== run 2: api containers before
SERVICE CONTAINER ID STATUS
api 11a93acecaca Up 2 seconds (healthy)
api 6128eb268c44 Up 56 seconds (healthy)
api 343119df2488 Up 4 seconds (healthy)
api d590adc21943 Up 48 seconds (healthy)
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.␍
{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 4.328}␍
{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 4.531}␍
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
api containers after
SERVICE CONTAINER ID STATUS
api 11a93acecaca Up 18 seconds (healthy)
api 6128eb268c44 Exited (0) 11 seconds ago
the hook stopped 6128eb268c44 (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched
restored: docker start 6128eb268c44 -> healthy and back in HAProxy
api 343119df2488 Up 20 seconds (healthy)
api d590adc21943 Exited (0) 11 seconds ago
the hook stopped d590adc21943 (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched
restored: docker start d590adc21943 -> healthy and back in HAProxy
PASS: public failover test passed 2 times; every api replica was stopped once and restored
+4 -4
View File
@@ -1,11 +1,11 @@
# failures: Redis and PostgreSQL outages via the balancer
# date: 2026-10-21 21:11:20, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:24:54, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/test_failures.py
# exit code: 0, 55.6 s
stopping redis
/health/live -> 200, /health/ready -> 503 (8016 ms)
valid token: GET /users/me -> 503 unavailable (8000 ms), session cannot be checked
/health/live -> 200, /health/ready -> 503 (8000 ms)
valid token: GET /users/me -> 503 unavailable (8031 ms), session cannot be checked
unknown token -> 503, not 200: nobody is trusted without a check
no token -> 401
login -> 503: there is nowhere to store the session
@@ -19,7 +19,7 @@ starting redis
after start: the session issued before the crash still works (AOF, appendfsync always)
PASS: sessions survive a Redis crash, not only a clean restart
stopping postgres
/health/live -> 200, /health/ready -> 503 (1016 ms)
/health/live -> 200, /health/ready -> 503 (1031 ms)
valid token: users/me, chats, history, new message -> 503 unavailable, no details
no token -> 401, unknown token -> 401: anonymous users are not let in
login -> 503, registration -> 503
+3 -3
View File
@@ -1,11 +1,11 @@
# idempotency: Idempotency-Key, parallel same key via replicas
# date: 2026-10-21 21:08:52, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:22:30, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/test_idempotency.py
# exit code: 0, 2.4 s
# exit code: 0, 2.5 s
first -> 201; same JSON written differently -> 201; plain retry -> 201
PASS: a retry with the same key returns the original 201 and message, no duplicate
20 parallel POSTs with one key: statuses {201: 20}, replicas {'6128eb268c44': 10, '11a93acecaca': 10}, different bodies in answers: 1, messages in history: 1
20 parallel POSTs with one key: statuses {201: 20}, replicas {'343119df2488': 10, 'd590adc21943': 10}, different bodies in answers: 1, messages in history: 1
PASS: parallel requests with one key through different replicas create exactly one message
20 parallel POSTs, one key, two different bodies: {('body A', 201): 10, ('body B', 409): 10}; saved: ['body A']
PASS: the first body wins, the same body gets 201, the other body gets 409
+3 -3
View File
@@ -2,12 +2,12 @@
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 30.0,
"elapsed_seconds": 30.015,
"elapsed_seconds": 30.047,
"concurrency": 4,
"successful_requests": 7379,
"successful_requests": 7191,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 245.844,
"successful_rps": 239.325,
"successful_latency_ms": {
"p50": 16.0,
"p95": 32.0,
+5 -5
View File
@@ -1,18 +1,18 @@
# load: public load generator, 30 s, concurrency 4
# date: 2026-10-21 21:10:24, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:23:58, commit: c942c61 + uncommitted changes
# command: python.exe tests/load.py --base-url http://127.0.0.1:8080 --duration 30 --concurrency 4 --output evidence/load.json
# exit code: 0, 30.4 s
# exit code: 0, 30.3 s
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 30.0,
"elapsed_seconds": 30.015,
"elapsed_seconds": 30.047,
"concurrency": 4,
"successful_requests": 7379,
"successful_requests": 7191,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 245.844,
"successful_rps": 239.325,
"successful_latency_ms": {
"p50": 16.0,
"p95": 32.0,
+17 -17
View File
@@ -1,24 +1,24 @@
# persistence: public persistence test incl. S3 + proof of real restarts (make test-persistence)
# date: 2026-10-21 21:12:49, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:25:42, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/run_persistence.py
# exit code: 0, 40.2 s
# exit code: 0, 47.9 s
before: container, ID, started at
messenger-lab4-api-1 11a93acecaca 2026-10-21T18:11:37.067348364Z
messenger-lab4-api-2 6128eb268c44 2026-10-21T18:11:36.916459116Z
messenger-lab4-janitor-1 baefc0e97dff 2026-10-21T18:11:33.313531682Z
messenger-lab4-lb-1 03697e5fe34c 2026-10-21T18:11:40.981519931Z
messenger-lab4-postgres-1 f7a6a1ce25a7 2026-10-21T18:11:22.482229396Z
messenger-lab4-redis-1 b4523b39a0fe 2026-10-21T18:11:22.626999402Z
messenger-lab4-s3-1 e7cbe6ddfa4a 2026-10-21T18:11:32.264154477Z
messenger-lab4-api-1 343119df2488 2026-10-22T11:23:03.387600262Z
messenger-lab4-api-2 d590adc21943 2026-10-22T11:23:24.689368662Z
messenger-lab4-janitor-1 d50ac08018c1 2026-10-22T11:18:17.423165415Z
messenger-lab4-lb-1 0d5b262b09b3 2026-10-22T11:03:30.971491955Z
messenger-lab4-postgres-1 84794c3ac2c5 2026-10-22T11:24:52.672662576Z
messenger-lab4-redis-1 3130fd5532b0 2026-10-22T11:24:32.055032931Z
messenger-lab4-s3-1 122ad603e3be 2026-10-22T11:20:52.20174809Z
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.␍
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
after (hook + waiting for readiness took 39.2 s): container, ID, started at, restarted?
messenger-lab4-api-1 11a93acecaca 2026-10-21T18:12:46.288933033Z same container: True, restarted: True
messenger-lab4-api-2 6128eb268c44 2026-10-21T18:12:46.174283308Z same container: True, restarted: True
messenger-lab4-janitor-1 baefc0e97dff 2026-10-21T18:12:42.503993973Z same container: True, restarted: True
messenger-lab4-lb-1 03697e5fe34c 2026-10-21T18:12:47.684809415Z same container: True, restarted: True
messenger-lab4-postgres-1 f7a6a1ce25a7 2026-10-21T18:12:11.527270274Z same container: True, restarted: True
messenger-lab4-redis-1 b4523b39a0fe 2026-10-21T18:12:11.368416107Z same container: True, restarted: True
messenger-lab4-s3-1 e7cbe6ddfa4a 2026-10-21T18:12:21.264135454Z same container: True, restarted: True
after (hook + waiting for readiness took 47.0 s): container, ID, started at, restarted?
messenger-lab4-api-1 343119df2488 2026-10-22T11:25:36.201352227Z same container: True, restarted: True
messenger-lab4-api-2 d590adc21943 2026-10-22T11:25:36.073058122Z same container: True, restarted: True
messenger-lab4-janitor-1 d50ac08018c1 2026-10-22T11:25:32.252887377Z same container: True, restarted: True
messenger-lab4-lb-1 0d5b262b09b3 2026-10-22T11:25:40.174401907Z same container: True, restarted: True
messenger-lab4-postgres-1 84794c3ac2c5 2026-10-22T11:24:56.311470336Z same container: True, restarted: True
messenger-lab4-redis-1 3130fd5532b0 2026-10-22T11:24:56.150164333Z same container: True, restarted: True
messenger-lab4-s3-1 122ad603e3be 2026-10-22T11:25:06.0234901Z same container: True, restarted: True
PASS: the hook restarted postgres, redis, s3 (SeaweedFS), all api replicas, janitor and lb (same containers, new start time); data, the attachment bytes and the old token survived
+3 -3
View File
@@ -1,9 +1,9 @@
# race: lab2 races via the balancer
# date: 2026-10-21 21:08:56, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:22:34, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/test_race.py
# exit code: 0, 1.2 s
# exit code: 0, 1.1 s
sent 100 messages from 10 clients in 0.22 s; read 100 in 4 pages, unique ids: 100
sent 100 messages from 10 clients in 0.19 s; read 100 in 4 pages, unique ids: 100
PASS: 100 concurrent messages from 10 clients, no losses and no duplicates
20 parallel registrations of one username: 201 x1, 409 x19
PASS: concurrent registration created exactly one user
+4 -4
View File
@@ -1,11 +1,11 @@
# redis: Redis as the session store
# date: 2026-10-21 21:09:13, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:22:43, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/redis_info.py
# exit code: 0, 2.8 s
# exit code: 0, 2.9 s
session key: session:f4f1fe0235d139de...; value is the user id: True
session key: session:e57bdd083b4cb87e...; value is the user id: True
TTL of the key, seconds: 3599
sessions in Redis: 126
sessions in Redis: 330
config appendonly = yes
config appendfsync = always
config maxmemory-policy = noeviction
+2 -2
View File
@@ -1,7 +1,7 @@
# s3-security: lab4: private bucket outside API, limited API key, no console/ports, idempotent bootstrap
# date: 2026-10-21 21:03:14, commit: 2dbed80
# date: 2026-10-22 14:19:35, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/test_s3_security.py
# exit code: 0, 9.5 s
# exit code: 0, 9.6 s
anonymous get_object directly to S3 (from the data network) -> 403 AccessDenied
anonymous head_object directly to S3 (from the data network) -> 403
+2 -2
View File
@@ -1,6 +1,6 @@
# smoke: public smoke via the balancer (make test), with attachment
# date: 2026-10-21 21:03:01, commit: 2dbed80
# date: 2026-10-22 14:19:20, commit: c942c61 + uncommitted changes
# command: python.exe tests/smoke.py --base-url http://127.0.0.1:8080
# exit code: 0, 0.6 s
# exit code: 0, 0.8 s
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
+14 -14
View File
@@ -1,21 +1,21 @@
# stack: stand layout: replicas, ports, limits, networks, secrets, S3
# date: 2026-10-21 21:03:00, commit: 2dbed80
# date: 2026-10-22 14:19:19, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/check_stack.py
# exit code: 0, 3.0 s
# exit code: 0, 2.9 s
host: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; client Python 3.11.1
Docker 28.1.1, Docker Compose version v2.35.1-desktop.1; Docker VM: 16 CPU, 15.2 GiB RAM, Docker Desktop
SERVICE CONTAINER ID STATUS PORTS
api 11a93acecaca Up 20 seconds (healthy) 8080/tcp
api 6128eb268c44 Up 20 seconds (healthy) 8080/tcp
janitor baefc0e97dff Up 20 seconds
lb 03697e5fe34c Up 18 seconds (healthy) 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
migrate aca090330db4 Exited (0) 22 seconds ago
postgres f7a6a1ce25a7 Up 2 minutes (healthy)
redis b4523b39a0fe Up 2 minutes (healthy)
s3 e7cbe6ddfa4a Up 2 minutes (healthy)
s3-init 60719e674e18 Exited (0) 21 seconds ago
SERVICE CONTAINER ID STATUS PORTS
api 343119df2488 Up 17 seconds (healthy) 8080/tcp
api d590adc21943 Up 18 seconds (healthy) 8080/tcp
janitor d50ac08018c1 Up About a minute
lb 0d5b262b09b3 Up 15 minutes (healthy) 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
migrate 6c3123ccf237 Exited (0) About a minute ago
postgres 84794c3ac2c5 Up 16 minutes (healthy)
redis 3130fd5532b0 Up 16 minutes (healthy)
s3 122ad603e3be Up 16 minutes (healthy)
s3-init 47c2d7d47ed2 Exited (0) About a minute ago
container image cpus memory host ports networks
messenger-lab4-api-1 messenger-lab4-api 1.00 512 MiB - data,edge
@@ -35,6 +35,6 @@ secrets of redis: ['redis_auth']
secrets of s3: ['s3_config']
secrets of s3-init: ['s3_access_key_id', 's3_root_password', 's3_root_user', 's3_secret_access_key']
through http://127.0.0.1:8080 one session is served by 2 replicas: ['11a93acecaca', '6128eb268c44']
/health/ready -> 200 from 11a93acecaca
through http://127.0.0.1:8080 one session is served by 2 replicas: ['343119df2488', 'd590adc21943']
/health/ready -> 200 from d590adc21943
PASS: 2 api replicas behind one entry point, no direct API ports, limits on every container, postgres/redis/s3/janitor only in the internal data network, S3 admin keys only in s3 and s3-init
+2 -2
View File
@@ -1,7 +1,7 @@
# student-tests: lab1/lab2 API tests via the balancer
# date: 2026-10-21 21:08:55, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:22:33, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/test_api.py
# exit code: 0, 2.7 s
# exit code: 0, 2.9 s
PASS: duplicate username returns 409
PASS: invalid JSON returns 400
+32 -15
View File
@@ -2,39 +2,56 @@
| Шаг | Что проверяет | Результат | Время, с | Файл |
| --- | --- | --- | --- | --- |
| persistence | public persistence test incl. S3 + proof of real restarts (make test-persistence) | PASS | 40 | evidence/persistence.txt |
| check | offline contract check (make check) | PASS | 0 | evidence/check.txt |
| stack | stand layout: replicas, ports, limits, networks, secrets, S3 | PASS | 3 | evidence/stack.txt |
| smoke | public smoke via the balancer (make test), with attachment | PASS | 1 | evidence/smoke.txt |
| attachments | lab4: files, limits, damaged images, MIME, filenames, access, attachment_ids | PASS | 5 | evidence/attachments.txt |
| s3-security | lab4: private bucket outside API, limited API key, no console/ports, idempotent bootstrap | PASS | 10 | evidence/s3-security.txt |
| consistency | lab4: aborted upload, crash between S3 and DB, janitor, orphans, S3 outage | PASS | 133 | evidence/consistency.txt |
| upload-memory | lab4: >=10 concurrent uploads, peak memory of api replicas | PASS | 37 | evidence/upload-memory.txt |
| balancing | balancing, shared sessions/data/files, X-Instance-Id | PASS | 2 | evidence/balancing.txt |
| idempotency | Idempotency-Key, parallel same key via replicas | PASS | 2 | evidence/idempotency.txt |
| student-tests | lab1/lab2 API tests via the balancer | PASS | 3 | evidence/student-tests.txt |
| race | lab2 races via the balancer | PASS | 1 | evidence/race.txt |
| db | PostgreSQL: rollback, FK, Argon2id, unique keys, migrations | PASS | 6 | evidence/db.txt |
| redis | Redis as the session store | PASS | 3 | evidence/redis.txt |
| explain | EXPLAIN of the history page (full plans: explain-history.txt) | PASS | 1 | evidence/explain.txt |
| failover | public failover test for every replica, file downloaded via the other one (make test-failover) | PASS | 43 | evidence/failover.txt |
| load | public load generator, 30 s, concurrency 4 | PASS | 30 | evidence/load.txt |
| failures | Redis and PostgreSQL outages via the balancer | PASS | 56 | evidence/failures.txt |
| persistence | public persistence test incl. S3 + proof of real restarts (make test-persistence) | PASS | 48 | evidence/persistence.txt |
# Замеры
Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-21, commit 2dbed80 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB.
Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-22, commit c942c61 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB.
Публичный тест отказа (`make test-failover`, 15 с записи во время остановки реплики):
| Прогон | Остановлена реплика | Реплик до отказа | Успешных записей | Неудачных попыток (повторены с тем же ключом) | Первая запись после hook, с |
| --- | --- | --- | --- | --- | --- |
| 1 | 11a93acecaca | 2 | 126 | 0 | 4.39 |
| 2 | 6128eb268c44 | 2 | 124 | 0 | 4.328 |
| 1 | 343119df2488 | 2 | 123 | 0 | 4.422 |
| 2 | d590adc21943 | 2 | 124 | 0 | 4.531 |
Одновременные загрузки (`student-tests/upload_memory.py`, перед каждым сценарием реплики перезапускаются, лимит api 512 MiB / 1 CPU). Пик RSS — VmHWM процесса API; cgroup — память контейнера, включая page cache временных файлов:
| Сценарий | Одновременно | Объём, MiB | Время, с (p50 / max) | Реплика: загрузок | RSS до, MiB | Пик RSS, MiB | Рост, MiB | Пик cgroup, MiB |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 1.38 / 1.52 | 11a93acecaca: 5 | 93.5 | 106.2 | 12.6 | 134.1 |
| | | | | 6128eb268c44: 5 | 94.8 | 105.3 | 10.5 | 135.2 |
| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 11a93acecaca: 5 | 93.2 | 96.6 | 3.4 | 84.3 |
| | | | | 6128eb268c44: 5 | 93.1 | 94.7 | 1.6 | 86.8 |
| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.37 / 1.77 | 11a93acecaca: 10 | 94.8 | 121.7 | 26.9 | 181.9 |
| | | | | 6128eb268c44: 10 | 95.0 | 116.1 | 21.0 | 155.6 |
| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 2.87 / 3.52 | 343119df2488: 5 | 94.8 | 107.6 | 12.8 | 136.0 |
| | | | | d590adc21943: 5 | 91.2 | 104.5 | 13.3 | 136.2 |
| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 343119df2488: 5 | 95.1 | 96.6 | 1.5 | 86.5 |
| | | | | d590adc21943: 5 | 92.1 | 96.6 | 4.6 | 83.5 |
| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.5 / 1.78 | 343119df2488: 10 | 93.2 | 116.2 | 23.0 | 158.2 |
| | | | | d590adc21943: 10 | 95.0 | 118.6 | 23.6 | 167.8 |
Один PNG 8000x5000 RGB внутри контейнера api, дополнительный пик RSS: наивно (тело целиком в памяти + полное декодирование Pillow) 161 MiB, потоковая проверка 0 MiB. 10 таких наивных загрузок одновременно не поместились бы в 512 MiB.
Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 245.844 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс.
Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 239.325 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.39, 4.328 с (hook ждёт остановки ~4 с); потерь нет |
| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 39.2 с до готовности; потерь нет |
| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 20141 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет |
| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14766 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет |
| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.422, 4.531 с (hook ждёт остановки ~4 с); потерь нет |
| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 47.0 с до готовности; потерь нет |
| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 19453 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет |
| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14891 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет |
| падение между PUT в S3 и COMMIT в БД (код API в контейнере, finish_upload падает) | вложение не видно, janitor не трогает его до конца grace period, потом удаляет объект и строку | GET/прикрепление -> 404; janitor: свежая строка сохранена, dry-run только отчёт, затем удалено | мусора не осталось; повторный проход ничего не делает |
| обрыв загрузки клиентом (9 МБ заявлено, отправлено 0,2 и 3 МБ) | ничего не сохраняется | строк и объектов не прибавилось, временных файлов 0 | мгновенно |
+51 -51
View File
@@ -1,5 +1,5 @@
{
"date": "2026-10-21 21:08",
"date": "2026-10-22 14:22",
"api_limit_mib": 512,
"api_limit_cpus": 1.0,
"runs": [
@@ -11,32 +11,32 @@
],
"total_mib": 94.9,
"all_201": true,
"wall_seconds": 1.52,
"wall_seconds": 3.52,
"latency_s": {
"p50": 1.38,
"max": 1.52
"p50": 2.87,
"max": 3.52
},
"sampler_rounds": 9,
"sampler_rounds": 18,
"replicas": [
{
"replica": "11a93acecaca",
"uploads_served": 5,
"rss_before_mib": 93.5,
"rss_peak_mib": 106.2,
"rss_growth_mib": 12.6,
"rss_peak_sampled_mib": 106.2,
"cgroup_peak_mib": 134.1,
"cgroup_peak_sampled_mib": 127.7
},
{
"replica": "6128eb268c44",
"replica": "343119df2488",
"uploads_served": 5,
"rss_before_mib": 94.8,
"rss_peak_mib": 105.3,
"rss_growth_mib": 10.5,
"rss_peak_sampled_mib": 105.3,
"cgroup_peak_mib": 135.2,
"cgroup_peak_sampled_mib": 131.4
"rss_peak_mib": 107.6,
"rss_growth_mib": 12.8,
"rss_peak_sampled_mib": 107.6,
"cgroup_peak_mib": 136.0,
"cgroup_peak_sampled_mib": 134.5
},
{
"replica": "d590adc21943",
"uploads_served": 5,
"rss_before_mib": 91.2,
"rss_peak_mib": 104.5,
"rss_growth_mib": 13.3,
"rss_peak_sampled_mib": 104.5,
"cgroup_peak_mib": 136.2,
"cgroup_peak_sampled_mib": 133.3
}
]
},
@@ -53,27 +53,27 @@
"p50": 0.85,
"max": 1.09
},
"sampler_rounds": 8,
"sampler_rounds": 12,
"replicas": [
{
"replica": "11a93acecaca",
"replica": "343119df2488",
"uploads_served": 5,
"rss_before_mib": 93.2,
"rss_before_mib": 95.1,
"rss_peak_mib": 96.6,
"rss_growth_mib": 3.4,
"rss_growth_mib": 1.5,
"rss_peak_sampled_mib": 96.6,
"cgroup_peak_mib": 84.3,
"cgroup_peak_sampled_mib": 77.3
"cgroup_peak_mib": 86.5,
"cgroup_peak_sampled_mib": 77.6
},
{
"replica": "6128eb268c44",
"replica": "d590adc21943",
"uploads_served": 5,
"rss_before_mib": 93.1,
"rss_peak_mib": 94.7,
"rss_growth_mib": 1.6,
"rss_peak_sampled_mib": 94.6,
"cgroup_peak_mib": 86.8,
"cgroup_peak_sampled_mib": 77.3
"rss_before_mib": 92.1,
"rss_peak_mib": 96.6,
"rss_growth_mib": 4.6,
"rss_peak_sampled_mib": 96.6,
"cgroup_peak_mib": 83.5,
"cgroup_peak_sampled_mib": 77.8
}
]
},
@@ -87,32 +87,32 @@
],
"total_mib": 133.2,
"all_201": true,
"wall_seconds": 1.77,
"wall_seconds": 1.78,
"latency_s": {
"p50": 1.37,
"max": 1.77
"p50": 1.5,
"max": 1.78
},
"sampler_rounds": 10,
"sampler_rounds": 12,
"replicas": [
{
"replica": "11a93acecaca",
"replica": "343119df2488",
"uploads_served": 10,
"rss_before_mib": 94.8,
"rss_peak_mib": 121.7,
"rss_growth_mib": 26.9,
"rss_peak_sampled_mib": 121.7,
"cgroup_peak_mib": 181.9,
"cgroup_peak_sampled_mib": 180.6
"rss_before_mib": 93.2,
"rss_peak_mib": 116.2,
"rss_growth_mib": 23.0,
"rss_peak_sampled_mib": 116.2,
"cgroup_peak_mib": 158.2,
"cgroup_peak_sampled_mib": 146.7
},
{
"replica": "6128eb268c44",
"replica": "d590adc21943",
"uploads_served": 10,
"rss_before_mib": 95.0,
"rss_peak_mib": 116.1,
"rss_growth_mib": 21.0,
"rss_peak_sampled_mib": 116.1,
"cgroup_peak_mib": 155.6,
"cgroup_peak_sampled_mib": 155.6
"rss_peak_mib": 118.6,
"rss_growth_mib": 23.6,
"rss_peak_sampled_mib": 118.6,
"cgroup_peak_mib": 167.8,
"cgroup_peak_sampled_mib": 167.7
}
]
}
+6 -6
View File
@@ -1,13 +1,13 @@
# upload-memory: lab4: >=10 concurrent uploads, peak memory of api replicas
# date: 2026-10-21 21:08:48, commit: 2dbed80 + uncommitted changes
# date: 2026-10-22 14:22:25, commit: c942c61 + uncommitted changes
# command: python.exe student-tests/upload_memory.py --output evidence/upload-memory.json
# exit code: 0, 34.2 s
# exit code: 0, 37.0 s
client: Python 3.11.1 on Windows 10
preparing synthetic images (random-noise PNG ~9.5 MiB each, 40 Mpx PNG)...
large: 9953047 bytes; 40 Mpx: 116714 bytes
{"scenario": "10 x ~9.5 MiB PNG (noise)", "concurrency": 10, "file_bytes": [9953047], "total_mib": 94.9, "all_201": true, "wall_seconds": 1.52, "latency_s": {"p50": 1.38, "max": 1.52}, "sampler_rounds": 9, "replicas": [{"replica": "11a93acecaca", "uploads_served": 5, "rss_before_mib": 93.5, "rss_peak_mib": 106.2, "rss_growth_mib": 12.6, "rss_peak_sampled_mib": 106.2, "cgroup_peak_mib": 134.1, "cgroup_peak_sampled_mib": 127.7}, {"replica": "6128eb268c44", "uploads_served": 5, "rss_before_mib": 94.8, "rss_peak_mib": 105.3, "rss_growth_mib": 10.5, "rss_peak_sampled_mib": 105.3, "cgroup_peak_mib": 135.2, "cgroup_peak_sampled_mib": 131.4}]}
{"scenario": "10 x 40 Mpx PNG (8000x5000 RGB)", "concurrency": 10, "file_bytes": [116714], "total_mib": 1.1, "all_201": true, "wall_seconds": 1.09, "latency_s": {"p50": 0.85, "max": 1.09}, "sampler_rounds": 8, "replicas": [{"replica": "11a93acecaca", "uploads_served": 5, "rss_before_mib": 93.2, "rss_peak_mib": 96.6, "rss_growth_mib": 3.4, "rss_peak_sampled_mib": 96.6, "cgroup_peak_mib": 84.3, "cgroup_peak_sampled_mib": 77.3}, {"replica": "6128eb268c44", "uploads_served": 5, "rss_before_mib": 93.1, "rss_peak_mib": 94.7, "rss_growth_mib": 1.6, "rss_peak_sampled_mib": 94.6, "cgroup_peak_mib": 86.8, "cgroup_peak_sampled_mib": 77.3}]}
{"scenario": "20 mixed: large + 40 Mpx + JPEG", "concurrency": 20, "file_bytes": [1492, 116714, 9953047], "total_mib": 133.2, "all_201": true, "wall_seconds": 1.77, "latency_s": {"p50": 1.37, "max": 1.77}, "sampler_rounds": 10, "replicas": [{"replica": "11a93acecaca", "uploads_served": 10, "rss_before_mib": 94.8, "rss_peak_mib": 121.7, "rss_growth_mib": 26.9, "rss_peak_sampled_mib": 121.7, "cgroup_peak_mib": 181.9, "cgroup_peak_sampled_mib": 180.6}, {"replica": "6128eb268c44", "uploads_served": 10, "rss_before_mib": 95.0, "rss_peak_mib": 116.1, "rss_growth_mib": 21.0, "rss_peak_sampled_mib": 116.1, "cgroup_peak_mib": 155.6, "cgroup_peak_sampled_mib": 155.6}]}
{"scenario": "10 x ~9.5 MiB PNG (noise)", "concurrency": 10, "file_bytes": [9953047], "total_mib": 94.9, "all_201": true, "wall_seconds": 3.52, "latency_s": {"p50": 2.87, "max": 3.52}, "sampler_rounds": 18, "replicas": [{"replica": "343119df2488", "uploads_served": 5, "rss_before_mib": 94.8, "rss_peak_mib": 107.6, "rss_growth_mib": 12.8, "rss_peak_sampled_mib": 107.6, "cgroup_peak_mib": 136.0, "cgroup_peak_sampled_mib": 134.5}, {"replica": "d590adc21943", "uploads_served": 5, "rss_before_mib": 91.2, "rss_peak_mib": 104.5, "rss_growth_mib": 13.3, "rss_peak_sampled_mib": 104.5, "cgroup_peak_mib": 136.2, "cgroup_peak_sampled_mib": 133.3}]}
{"scenario": "10 x 40 Mpx PNG (8000x5000 RGB)", "concurrency": 10, "file_bytes": [116714], "total_mib": 1.1, "all_201": true, "wall_seconds": 1.09, "latency_s": {"p50": 0.85, "max": 1.09}, "sampler_rounds": 12, "replicas": [{"replica": "343119df2488", "uploads_served": 5, "rss_before_mib": 95.1, "rss_peak_mib": 96.6, "rss_growth_mib": 1.5, "rss_peak_sampled_mib": 96.6, "cgroup_peak_mib": 86.5, "cgroup_peak_sampled_mib": 77.6}, {"replica": "d590adc21943", "uploads_served": 5, "rss_before_mib": 92.1, "rss_peak_mib": 96.6, "rss_growth_mib": 4.6, "rss_peak_sampled_mib": 96.6, "cgroup_peak_mib": 83.5, "cgroup_peak_sampled_mib": 77.8}]}
{"scenario": "20 mixed: large + 40 Mpx + JPEG", "concurrency": 20, "file_bytes": [1492, 116714, 9953047], "total_mib": 133.2, "all_201": true, "wall_seconds": 1.78, "latency_s": {"p50": 1.5, "max": 1.78}, "sampler_rounds": 12, "replicas": [{"replica": "343119df2488", "uploads_served": 10, "rss_before_mib": 93.2, "rss_peak_mib": 116.2, "rss_growth_mib": 23.0, "rss_peak_sampled_mib": 116.2, "cgroup_peak_mib": 158.2, "cgroup_peak_sampled_mib": 146.7}, {"replica": "d590adc21943", "uploads_served": 10, "rss_before_mib": 95.0, "rss_peak_mib": 118.6, "rss_growth_mib": 23.6, "rss_peak_sampled_mib": 118.6, "cgroup_peak_mib": 167.8, "cgroup_peak_sampled_mib": 167.7}]}
one 8000x5000 RGB PNG inside the api container, extra peak RSS: naive (body in RAM + full decode) 161 MiB, streaming check 0 MiB
PASS: all uploads succeeded; largest RSS growth of one replica: 26.9 MiB (limit 512 MiB); results in evidence/upload-memory.json
PASS: all uploads succeeded; largest RSS growth of one replica: 23.6 MiB (limit 512 MiB); results in evidence/upload-memory.json
+22 -3
View File
@@ -22,7 +22,7 @@ def chunk(kind: bytes, data: bytes) -> bytes:
def png(width, height, *, gray=False, noise=False, interlace=0, pad_to=None, idat=None) -> bytes:
"""PNG width x height. noise=True — случайные пиксели (почти не сжимаются, файл большой).
pad_to — дополнить файл служебным чанком prVt до точного размера в байтах.
pad_to — дополнить файл текстовым чанком tEXt после IDAT до точного размера в байтах.
idat — готовые сжатые данные (для «бомбы»: заголовок обещает одно, данных другое).
"""
channels = 1 if gray else 3
@@ -43,9 +43,9 @@ def png(width, height, *, gray=False, noise=False, interlace=0, pad_to=None, ida
idat = b"".join(parts)
body = b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header) + chunk(b"IDAT", idat)
if pad_to is not None:
need = pad_to - len(body) - 12 - 12 # чанк-заполнитель + IEND
need = pad_to - len(body) - 12 - 12 - 8 # заголовок чанка-заполнителя, IEND, ключ "Comment\0"
assert need >= 0, "image is already larger than pad_to"
body += chunk(b"prVt", bytes(need))
body += chunk(b"tEXt", b"Comment\x00" + b"a" * need)
return body + chunk(b"IEND", b"")
@@ -72,3 +72,22 @@ def flip_byte(data: bytes, position: int) -> bytes:
GIF = b"GIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\xff\xff\xff!\xf9\x04\x01\x00\x00\x00\x00,\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02D\x01\x00;"
# --- сборка PNG из произвольных чанков (для проверок смысла чанков при верных CRC) ---
def ihdr(width, height, depth=8, color=2, interlace=0):
return b"IHDR", struct.pack(">IIBBBBB", width, height, depth, color, 0, 0, interlace)
def zero_idat(width, height, depth=8, channels=3):
"""Сжатые пиксели: каждая строка = байт фильтра 0 + нули (индекс палитры 0 / чёрный)."""
row_bytes = (width * depth * channels + 7) // 8
return b"IDAT", zlib.compress((b"\x00" + bytes(row_bytes)) * height)
def assemble(*chunks):
"""PNG из чанков (kind, data) с ВЕРНЫМИ CRC; IEND добавляется, если не передан."""
if not chunks or chunks[-1][0] != b"IEND":
chunks = chunks + ((b"IEND", b""),)
return b"\x89PNG\r\n\x1a\n" + b"".join(chunk(kind, data) for kind, data in chunks)
+69 -1
View File
@@ -6,6 +6,7 @@
import hashlib
import http.client
import socket
import struct
import urllib.parse
import uuid
import zlib
@@ -135,6 +136,73 @@ def test_damaged_images():
print("PASS: damaged and foreign files are rejected with 415, never 500")
def test_png_chunk_semantics():
"""CRC у всех чанков верные, но содержимое нарушает спецификацию PNG -> 415, а не 201.
Регрессия: PNG 1x1, глубина 8, палитровый, PLTE из 771 байта (257 записей) раньше получал 201 ready,
хотя Pillow его отклоняет (invalid palette size).
"""
alice, bob, _, chat = setup()
from images import assemble, ihdr, zero_idat
pal = lambda entries: (b"PLTE", bytes(3 * entries))
idat_p = zero_idat(1, 1, 8, 1)
bad = [
("PLTE 771 bytes = 257 entries (review case)",
assemble(ihdr(1, 1, 8, 3), (b"PLTE", bytes(771)), (b"IDAT", zlib.compress(b"\0\0")))),
("PLTE length not a multiple of 3", assemble(ihdr(1, 1, 8, 3), (b"PLTE", bytes(4)), idat_p)),
("empty PLTE", assemble(ihdr(1, 1, 8, 3), (b"PLTE", b""), idat_p)),
("palette larger than bit depth allows (depth 1, 3 entries)",
assemble(ihdr(1, 1, 1, 3), pal(3), zero_idat(1, 1, 1, 1))),
("palette image without PLTE", assemble(ihdr(1, 1, 8, 3), idat_p)),
("PLTE in a grayscale image", assemble(ihdr(1, 1, 8, 0), pal(2), zero_idat(1, 1, 8, 1))),
("PLTE after IDAT", assemble(ihdr(1, 1, 8, 3), idat_p, pal(2))),
("two PLTE chunks", assemble(ihdr(1, 1, 8, 3), pal(2), pal(2), idat_p)),
("tRNS longer than the palette", assemble(ihdr(1, 1, 8, 3), pal(2), (b"tRNS", bytes(3)), idat_p)),
("tRNS before PLTE", assemble(ihdr(1, 1, 8, 3), (b"tRNS", bytes(1)), pal(2), idat_p)),
("tRNS of wrong size for RGB (4 bytes instead of 6)",
assemble(ihdr(1, 1, 8, 2), (b"tRNS", bytes(4)), zero_idat(1, 1, 8, 3))),
("tRNS in an image with alpha channel",
assemble(ihdr(1, 1, 8, 6), (b"tRNS", bytes(2)), zero_idat(1, 1, 8, 4))),
("IEND with data", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"IEND", b"x"))),
("unknown critical chunk", assemble(ihdr(1, 1, 8, 2), (b"ABCD", b""), zero_idat(1, 1, 8, 3))),
("IDAT chunks not consecutive",
assemble(ihdr(1, 2, 8, 2), (b"IDAT", zero_idat(1, 2, 8, 3)[1][:5]), (b"tEXt", b"k\0v"),
(b"IDAT", zero_idat(1, 2, 8, 3)[1][5:]))),
("gAMA after IDAT", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"gAMA", bytes(4)))),
("gAMA of 2 bytes instead of 4", assemble(ihdr(1, 1, 8, 2), (b"gAMA", bytes(2)), zero_idat(1, 1, 8, 3))),
("3 MiB metadata chunk before IDAT", assemble(ihdr(1, 1, 8, 2), (b"prVt", bytes(3 * 1024 * 1024)),
zero_idat(1, 1, 8, 3))),
("private chunk after IDAT", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"prVt", bytes(100)))),
("animated PNG frame (fdAT)", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"fdAT", bytes(10)))),
("tIME of 2 bytes instead of 7", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"tIME", bytes(2)))),
("tEXt without keyword separator", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"tEXt", b"nokey"))),
("zTXt with broken zlib after IDAT",
assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"zTXt", b"k\0\0garbage"))),
("zTXt text bomb: 50 MB of text in ~50 KB after IDAT",
assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"zTXt", b"k\0\0" + zlib.compress(bytes(50_000_000))))),
]
for name, data in bad:
check_error(upload(alice, chat["id"], data, filename="bad.png", expected=415), "unsupported_media_type")
print(f"{name} ({len(data)} bytes, valid CRCs) -> 415")
good = [
("palette 256 entries + tRNS + tEXt and tIME after IDAT",
assemble(ihdr(2, 2, 8, 3), pal(256), (b"tRNS", bytes(256)), zero_idat(2, 2, 8, 1),
(b"tEXt", b"Comment\0ok"), (b"tIME", struct.pack(">HBBBBB", 2026, 10, 22, 12, 0, 0)))),
("palette depth 2 with 4 entries", assemble(ihdr(3, 1, 2, 3), pal(4), zero_idat(3, 1, 2, 1))),
("RGB with 6-byte tRNS and gAMA", assemble(ihdr(1, 1, 8, 2), (b"gAMA", struct.pack(">I", 45455)),
(b"tRNS", bytes(6)), zero_idat(1, 1, 8, 3))),
("compressed zTXt (100 KB of text) and iTXt after IDAT",
assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"zTXt", b"Comment\0\0" + zlib.compress(b"a" * 100_000)),
(b"iTXt", b"Title\0\1\0ru\0\0" + zlib.compress("Привет".encode())))),
]
for name, data in good:
att = upload(alice, chat["id"], data, filename="good.png").json
got = call("GET", f"/api/v1/attachments/{att['id']}/content", 200, token=bob["token"])
check(got.data == data, "valid PNG changed")
print(f"{name} -> 201, downloaded byte for byte")
print("PASS: chunk contents are checked, not only CRCs; valid PNGs with metadata are still accepted")
def test_mime_substitution():
alice, _, _, chat = setup()
cases = [
@@ -264,5 +332,5 @@ def test_aborted_upload():
if __name__ == "__main__":
run_tests([test_png_and_jpeg_roundtrip, test_size_limit_exact_boundary, test_size_limit_streaming,
test_pixel_limit, test_damaged_images, test_mime_substitution, test_filenames, test_access_control,
test_pixel_limit, test_damaged_images, test_png_chunk_semantics, test_mime_substitution, test_filenames, test_access_control,
test_message_attachments, test_idempotency_with_attachments, test_aborted_upload])
+183 -15
View File
@@ -183,7 +183,12 @@ def inspect(upload: Upload, declared: str) -> ImageInfo:
upload.file.seek(0)
try:
with _decode_slots:
width, height = check_png(upload.file) if detected == PNG else check_jpeg(upload.file)
if detected == PNG:
width, height = check_png(upload.file)
upload.file.seek(0)
check_png_metadata(upload.file)
else:
width, height = check_jpeg(upload.file)
except TooManyPixels:
raise payload_too_large(f"Image has more than {settings.MAX_IMAGE_PIXELS} pixels") from None
except Damaged as exc:
@@ -262,13 +267,40 @@ class PngRows:
return self.total == self.expected
# Чанки, содержимое которых проверяется (держим в памяти; размер ограничен спецификацией):
# IHDR — 13 байт, PLTE — до 256 x 3 = 768, tRNS — до 256 (палитра) / 6 (RGB).
SMALL_CHUNK_LIMIT = {b"IHDR": 13, b"PLTE": 768, b"tRNS": 256}
CRITICAL = (b"IHDR", b"PLTE", b"IDAT", b"IEND")
# После данных изображения допустимы только время и текст (строгий список).
AFTER_IDAT = {b"IEND", b"tIME", b"tEXt", b"zTXt", b"iTXt"}
TEXT_CHUNKS = (b"tEXt", b"zTXt", b"iTXt")
# Анимированные PNG (APNG) не поддерживаются: кадры — отдельная сущность, превью строится по одному кадру.
APNG_CHUNKS = (b"acTL", b"fcTL", b"fdAT")
# Как PngImagePlugin.MAX_TEXT_CHUNK в Pillow: больше 1 MiB распакованного текста в одном чанке — отказ.
TEXT_LIMIT = 1024 * 1024
# Чанки метаданных до IDAT Pillow (Image.open) читает целиком: ограничиваем, чтобы память не росла.
# ICC-профили и EXIF реальных картинок — единицы-сотни KiB.
PRE_IDAT_CHUNK_LIMIT = 2 * 1024 * 1024
# Чанки, которые по спецификации обязаны стоять ДО первого IDAT (PNG 3rd ed., 5.6 «Chunk ordering»).
# После IDAT допустимы только tIME, tEXt, zTXt, iTXt, кадры APNG и частные чанки.
BEFORE_IDAT = {b"PLTE", b"tRNS", b"cHRM", b"cICP", b"gAMA", b"iCCP", b"mDCV", b"cLLI", b"sBIT", b"sRGB",
b"bKGD", b"hIST", b"pHYs", b"sPLT", b"eXIf"}
def check_png(file: BinaryIO) -> tuple[int, int]:
"""Структура PNG по спецификации (PNG 3rd ed., гл. 5 и 11) без декодирования пикселей в память.
Кроме CRC каждого чанка проверяется СМЫСЛ критичных чанков и tRNS: CRC защищает только от
случайной порчи, а намеренно собранный файл с верными CRC может быть некорректным.
"""
if file.read(8) != PNG_SIGNATURE:
raise Damaged("signature")
rows = inflater = None
width = height = 0
seen_idat = idat_finished = have_plte = False
width = height = depth = 0
color = None
palette_entries = 0
seen = set() # какие чанки уже встречались
idat_started = idat_finished = False
first = True
while True:
header = file.read(8)
@@ -277,11 +309,37 @@ def check_png(file: BinaryIO) -> tuple[int, int]:
length, kind = struct.unpack(">I4s", header)
if length > 0x7FFFFFFF:
raise Damaged("chunk length")
if not kind.isalpha() or not kind.isascii():
raise Damaged("chunk type")
if first and kind != b"IHDR":
raise Damaged("IHDR must be first")
if kind[:1].isupper() and kind not in CRITICAL:
raise Damaged(f"unknown critical chunk {kind!r}") # такой чанк декодер обязан отвергнуть
if kind in SMALL_CHUNK_LIMIT and length > SMALL_CHUNK_LIMIT[kind]:
raise Damaged(f"{kind!r} is too long")
if kind == b"IDAT":
if rows is None or idat_finished:
raise Damaged("IDAT chunks must be consecutive")
idat_started = True
elif idat_started:
idat_finished = True
if kind in APNG_CHUNKS:
raise Damaged("animated PNG is not supported")
if kind in BEFORE_IDAT and idat_started:
raise Damaged(f"{kind!r} after IDAT")
if idat_finished and kind not in AFTER_IDAT:
raise Damaged(f"{kind!r} after IDAT")
if not idat_started and kind != b"IDAT" and length > PRE_IDAT_CHUNK_LIMIT:
raise Damaged(f"{kind!r} metadata chunk is too large")
if kind == b"tIME" and length != 7:
raise Damaged("tIME size")
if kind in (b"IHDR", b"PLTE", b"tRNS", b"IEND") and kind in seen:
raise Damaged(f"duplicate {kind!r}")
seen.add(kind)
crc = zlib.crc32(kind)
keep = kind in (b"IHDR",)
kept = b""
text = TextChunk(kind) if kind in TEXT_CHUNKS else None
left = length
while left:
piece = file.read(min(CRC_CHUNK, left))
@@ -289,18 +347,20 @@ def check_png(file: BinaryIO) -> tuple[int, int]:
raise Damaged("truncated chunk")
left -= len(piece)
crc = zlib.crc32(piece, crc)
if keep:
if kind in SMALL_CHUNK_LIMIT:
kept += piece
elif kind == b"IDAT":
if rows is None or idat_finished:
raise Damaged("IDAT out of order")
feed_inflate(inflater, rows, piece)
elif text is not None:
text.feed(piece)
if text is not None:
text.finish()
stored = file.read(4)
if len(stored) < 4 or struct.unpack(">I", stored)[0] != crc & 0xFFFFFFFF:
raise Damaged(f"CRC mismatch in {kind!r}")
if kind == b"IHDR":
if not first or length != 13:
if length != 13:
raise Damaged("IHDR")
width, height, depth, color, compression, filter_method, interlace = struct.unpack(">IIBBBBB", kept)
if width > 0x7FFFFFFF or height > 0x7FFFFFFF:
@@ -313,20 +373,34 @@ def check_png(file: BinaryIO) -> tuple[int, int]:
rows = PngRows(width, height, depth, color, interlace)
inflater = zlib.decompressobj()
elif kind == b"PLTE":
have_plte = True
elif kind == b"IDAT":
seen_idat = True
# 1-256 записей по 3 байта; для палитровых — не больше 2^глубина; у серых палитры быть не может.
if color in (0, 4):
raise Damaged("PLTE in a grayscale image")
if length == 0 or length % 3:
raise Damaged("palette size")
palette_entries = length // 3
if color == 3 and palette_entries > 1 << depth:
raise Damaged("palette larger than the bit depth allows")
elif kind == b"tRNS":
# Прозрачность: серое — 2 байта, RGB — 6, палитра — не больше записей PLTE; при альфа-канале запрещён.
if color in (4, 6):
raise Damaged("tRNS with an alpha channel")
if color == 3 and not palette_entries:
raise Damaged("tRNS before PLTE")
expected_ok = {0: length == 2, 2: length == 6, 3: 1 <= length <= palette_entries}[color]
if not expected_ok:
raise Damaged("tRNS size")
elif kind == b"IEND":
if length != 0:
raise Damaged("IEND with data")
break
if seen_idat and kind != b"IDAT":
idat_finished = True
first = False
if file.read(1):
raise Damaged("data after IEND")
if color == 3 and not have_plte:
if color == 3 and not palette_entries:
raise Damaged("palette image without PLTE")
if not seen_idat:
if not idat_started:
raise Damaged("no IDAT")
tail = inflater.flush(INFLATE_CHUNK)
if tail:
@@ -336,6 +410,100 @@ def check_png(file: BinaryIO) -> tuple[int, int]:
return width, height
def check_png_metadata(file: BinaryIO) -> None:
"""Второе мнение: стандартный парсер Pillow разбирает метаданные до IDAT (gAMA, iCCP, pHYs, tEXt…),
НЕ читая пиксели. Только open, без verify(): verify читает каждый чанк, включая IDAT до 10 MiB,
целиком в память, а CRC и данные после IDAT уже потоково проверил check_png.
Нужен для вспомогательных чанков: по спецификации их можно игнорировать, но если Pillow
не может их разобрать, файл нельзя будет обработать (превью в lab5) — лучше отклонить сразу.
"""
with Image.open(file, formats=["PNG"]):
pass # open разбирает все чанки до первого IDAT; после IDAT check_png пускает только время и текст
class TextChunk:
"""Потоковая проверка tEXt / zTXt / iTXt: ключ 1-79 байт, метод сжатия 0, сжатый текст —
корректный zlib не больше TEXT_LIMIT после распаковки (распаковка кусками, текст не хранится)."""
def __init__(self, kind: bytes):
self.kind = kind
self.head = b""
self.done_header = False
self.inflater = None
self.size = 0
def feed(self, piece: bytes) -> None:
if not self.done_header:
self.head += piece
rest = self._header()
if rest is None:
if len(self.head) > 4096:
raise Damaged("text chunk header")
return
piece = rest
if self.inflater is not None and piece:
self._inflate(piece)
def _keyword_end(self) -> int | None:
end = self.head.find(b"\x00")
if end == -1:
return None
if not 1 <= end <= 79:
raise Damaged("text keyword")
return end
def _header(self) -> bytes | None:
end = self._keyword_end()
if end is None:
return None
h = self.head
if self.kind == b"tEXt":
self.done_header = True
return b""
if self.kind == b"zTXt":
if len(h) < end + 2:
return None
if h[end + 1] != 0:
raise Damaged("zTXt compression method")
self.done_header, self.inflater = True, zlib.decompressobj()
return h[end + 2:]
# iTXt: keyword \0 flag method language \0 translated-keyword \0 text
if len(h) < end + 3:
return None
flag, method = h[end + 1], h[end + 2]
if flag not in (0, 1) or (flag and method != 0):
raise Damaged("iTXt compression")
lang_end = h.find(b"\x00", end + 3)
tkey_end = h.find(b"\x00", lang_end + 1) if lang_end != -1 else -1
if tkey_end == -1:
return None
self.done_header = True
if flag:
self.inflater = zlib.decompressobj()
return h[tkey_end + 1:]
def _inflate(self, data: bytes) -> None:
if self.inflater.eof:
raise Damaged("data after compressed text")
out = self.inflater.decompress(data, INFLATE_CHUNK)
self.size += len(out)
while self.inflater.unconsumed_tail and not self.inflater.eof and self.size <= TEXT_LIMIT:
out = self.inflater.decompress(self.inflater.unconsumed_tail, INFLATE_CHUNK)
self.size += len(out)
if self.size > TEXT_LIMIT:
raise Damaged("compressed text is too large")
if self.inflater.eof and self.inflater.unused_data:
raise Damaged("data after compressed text")
def finish(self) -> None:
if not self.done_header:
raise Damaged("text chunk header")
if self.inflater is not None:
self.size += len(self.inflater.flush(INFLATE_CHUNK))
if not self.inflater.eof or self.size > TEXT_LIMIT:
raise Damaged("compressed text")
def feed_inflate(inflater, rows: PngRows, data: bytes) -> None:
"""Распаковка кусками не больше 64 KiB: «zip-бомба» не раздует память."""
if inflater.eof: