diff --git a/REPORT.md b/REPORT.md index 527b32b..f1dbf76 100644 --- a/REPORT.md +++ b/REPORT.md @@ -91,7 +91,7 @@ s3-init: одноразово создаёт приватный bucket и пр 3. Тип по сигнатуре (`\x89PNG...`, `FF D8 FF`), не по заголовку. Не PNG/JPEG -> 415; заявленный тип не совпадает с содержимым -> 415. 4. Размер в пикселях читается из заголовка (IHDR / SOF) и проверяется ДО распаковки: больше 40 000 000 -> 413. 5. Проверка целостности с ограниченной памятью: - - PNG — своя потоковая проверка (`uploads.check_png`): CRC каждого чанка, порядок IHDR/PLTE/IDAT/IEND, допустимые тип цвета и глубина, ничего после IEND; IDAT распаковывается zlib кусками по 64 KiB (вывод не хранится), считается точный объём данных по IHDR (с учётом Adam7) и проверяется байт фильтра каждой строки. «Zip-бомба» не раздувает память: распаковка идёт порциями и обрывается, как только данных больше, чем обещал заголовок. + - PNG — своя потоковая проверка (`uploads.check_png`): CRC каждого чанка и смысл критичных чанков по спецификации PNG (размер палитры 1–256 записей и не больше 2^глубина, размер tRNS по типу цвета, пустой IEND, порядок чанков: всё, что обязано идти до IDAT, — до него, IDAT подряд, неизвестный критичный чанк запрещён); IDAT распаковывается кусками по 64 KiB с подсчётом точного объёма и проверкой байта фильтра каждой строки. Затем Pillow разбирает метаданные до IDAT (Image.open, без verify: verify читает каждый чанк, включая IDAT до 10 MiB, в память). После IDAT допускаются только tIME, tEXt, zTXt, iTXt; текстовые чанки проверяются потоково (ключ 1–79 байт, корректный zlib, не больше 1 MiB распакованного текста); анимированные PNG (acTL/fcTL/fdAT) отклоняются; чанк метаданных до IDAT — не больше 2 MiB. Исправлено по замечанию: раньше проверялись только CRC, и PNG с палитрой из 257 записей получал 201. - JPEG — Pillow декодирует весь поток, но в масштабе 1/8 (`draft`): 40 Мпикс -> ~0,6 Мпикс, около 2 MiB. - Одновременно в реплике проверяется не больше `DECODE_CONCURRENCY=2` файлов (CPU). Битый файл -> 415. Любая ошибка разбора ловится и превращается в 415, а не 500. Проверка PNG сверена с Pillow: 400 случайных корректных PNG разных типов приняты, 400 испорченных (обрезка, перевёрнутый бит) отклонены. @@ -187,36 +187,36 @@ S3-клиент: connect timeout 1 с, read timeout 5 с, всего 2 попы Методика памяти (`student-tests/upload_memory.py`): три сценария — 10 x PNG из случайного шума ~9,5 MiB (почти несжимаемый, максимум трафика); 10 x PNG 8000x5000 RGB (40 Мпикс, максимум распаковки); 20 смешанных. Перед каждым сценарием `docker compose restart api` (сбрасывает пик), прогрев 8 загрузками, затем все потоки стартуют одновременно (barrier) через балансировщик. Пик RSS — `VmHWM` из `/proc/1/status` процесса API (максимум за жизнь процесса по данным ядра, короткие всплески не теряются), дополнительно пик памяти cgroup контейнера и опрос раз в ~0,1–0,5 с. Проверяются коды 201 и SHA-256 скачанных файлов. Клиент — тот же компьютер, поэтому время загрузки ограничено и клиентом; это замер памяти, а не пропускной способности. -Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-21, commit 2dbed80 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB. +Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-22, commit c942c61 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB. Публичный тест отказа (`make test-failover`, 15 с записи во время остановки реплики): | Прогон | Остановлена реплика | Реплик до отказа | Успешных записей | Неудачных попыток (повторены с тем же ключом) | Первая запись после hook, с | | --- | --- | --- | --- | --- | --- | -| 1 | 11a93acecaca | 2 | 126 | 0 | 4.39 | -| 2 | 6128eb268c44 | 2 | 124 | 0 | 4.328 | +| 1 | 343119df2488 | 2 | 123 | 0 | 4.422 | +| 2 | d590adc21943 | 2 | 124 | 0 | 4.531 | Одновременные загрузки (`student-tests/upload_memory.py`, перед каждым сценарием реплики перезапускаются, лимит api 512 MiB / 1 CPU). Пик RSS — VmHWM процесса API; cgroup — память контейнера, включая page cache временных файлов: | Сценарий | Одновременно | Объём, MiB | Время, с (p50 / max) | Реплика: загрузок | RSS до, MiB | Пик RSS, MiB | Рост, MiB | Пик cgroup, MiB | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 1.38 / 1.52 | 11a93acecaca: 5 | 93.5 | 106.2 | 12.6 | 134.1 | -| | | | | 6128eb268c44: 5 | 94.8 | 105.3 | 10.5 | 135.2 | -| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 11a93acecaca: 5 | 93.2 | 96.6 | 3.4 | 84.3 | -| | | | | 6128eb268c44: 5 | 93.1 | 94.7 | 1.6 | 86.8 | -| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.37 / 1.77 | 11a93acecaca: 10 | 94.8 | 121.7 | 26.9 | 181.9 | -| | | | | 6128eb268c44: 10 | 95.0 | 116.1 | 21.0 | 155.6 | +| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 2.87 / 3.52 | 343119df2488: 5 | 94.8 | 107.6 | 12.8 | 136.0 | +| | | | | d590adc21943: 5 | 91.2 | 104.5 | 13.3 | 136.2 | +| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 343119df2488: 5 | 95.1 | 96.6 | 1.5 | 86.5 | +| | | | | d590adc21943: 5 | 92.1 | 96.6 | 4.6 | 83.5 | +| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.5 / 1.78 | 343119df2488: 10 | 93.2 | 116.2 | 23.0 | 158.2 | +| | | | | d590adc21943: 10 | 95.0 | 118.6 | 23.6 | 167.8 | Один PNG 8000x5000 RGB внутри контейнера api, дополнительный пик RSS: наивно (тело целиком в памяти + полное декодирование Pillow) 161 MiB, потоковая проверка 0 MiB. 10 таких наивных загрузок одновременно не поместились бы в 512 MiB. -Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 245.844 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс. +Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 239.325 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс. | Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных | | --- | --- | --- | --- | -| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.39, 4.328 с (hook ждёт остановки ~4 с); потерь нет | -| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 39.2 с до готовности; потерь нет | -| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 20141 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет | -| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14766 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет | +| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.422, 4.531 с (hook ждёт остановки ~4 с); потерь нет | +| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 47.0 с до готовности; потерь нет | +| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 19453 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет | +| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14891 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет | | падение между PUT в S3 и COMMIT в БД (код API в контейнере, finish_upload падает) | вложение не видно, janitor не трогает его до конца grace period, потом удаляет объект и строку | GET/прикрепление -> 404; janitor: свежая строка сохранена, dry-run только отчёт, затем удалено | мусора не осталось; повторный проход ничего не делает | | обрыв загрузки клиентом (9 МБ заявлено, отправлено 0,2 и 3 МБ) | ничего не сохраняется | строк и объектов не прибавилось, временных файлов 0 | мгновенно | @@ -231,5 +231,6 @@ S3-клиент: connect timeout 1 с, read timeout 5 с, всего 2 попы - Обход bucket janitor’ом держит в памяти список ключей (для учебного объёма — мегабайты); для миллионов объектов нужна сверка по диапазонам ключей или S3 Inventory. - Временный файл загрузки (часть тела сверх 256 KiB) лежит на диске контейнера. В Linux `tempfile.TemporaryFile` создаётся уже без имени в каталоге (O_TMPFILE), поэтому даже при SIGKILL реплики место освобождается ядром, «забытых» файлов не бывает. Одновременно на диске может быть до 10 MiB на каждую идущую загрузку; лимита на число одновременных загрузок, кроме лимитов HAProxy/uvicorn, нет. - Проверяется структура изображения, а не его безопасность для всех декодеров (например, JPEG с некорректными, но декодируемыми данными пройдёт). Перекодирование оригинала — задача lab5. +- PNG с частными служебными чанками после IDAT и анимированные PNG отклоняются (строже спецификации, на практике не встречаются в фотографиях). - MinIO собирался быть первым выбором, но его образы больше не доступны в публичных реестрах: закреплённый тег защищает от неожиданного обновления, но не от исчезновения образа. Для эксплуатации нужно зеркалить образы в свой реестр. SeaweedFS запущен одним процессом `weed server` на одном диске — это SPOF и не защита от потери диска; веб-интерфейсы master/filer без авторизации, закрыты только сетью. - Скрипты `failover_metrics.py`, `rolling_update.py`, `test_expiry.py` из lab3 остались в `student-tests/` и работают со стендом, но в общий прогон lab4 не входят (это уровень 4 lab3, а не lab4). diff --git a/evidence/attachments.txt b/evidence/attachments.txt index dc0ed88..b9820b5 100644 --- a/evidence/attachments.txt +++ b/evidence/attachments.txt @@ -1,7 +1,7 @@ # attachments: lab4: files, limits, damaged images, MIME, filenames, access, attachment_ids -# date: 2026-10-21 21:03:05, commit: 2dbed80 +# date: 2026-10-22 14:19:26, commit: c942c61 + uncommitted changes # command: python.exe student-tests/test_attachments.py -# exit code: 0, 4.0 s +# exit code: 0, 5.4 s sample.png: 111817 bytes, sha256 99ea7533c441..., downloaded by the other member, same bytes sample.jpg: 1492 bytes, sha256 c310b9a1c208..., downloaded by the other member, same bytes @@ -14,7 +14,7 @@ exactly 10485760 bytes (10 MiB) -> 201 PASS: the 10 MiB limit is exact 12 MiB without Content-Length (chunked) -> 413 as soon as the 10 MiB limit is passed PASS: oversized streamed body is cut off, the API keeps working -8000x5000 = 40 000 000 pixels (38957 bytes compressed) -> 201 5ffe6847 +8000x5000 = 40 000 000 pixels (38957 bytes compressed) -> 201 af872a80 8001x5000 = 40 005 000 pixels (38962 bytes) -> 413 header claims 100000x100000 in 65 bytes -> 413 from the header, nothing is decoded PASS: pixel limit is checked before decompression @@ -30,6 +30,35 @@ random bytes (1600 bytes, image/png) -> 415 plain text (28 bytes, image/jpeg) -> 415 empty body -> 400 PASS: damaged and foreign files are rejected with 415, never 500 +PLTE 771 bytes = 257 entries (review case) (850 bytes, valid CRCs) -> 415 +PLTE length not a multiple of 3 (83 bytes, valid CRCs) -> 415 +empty PLTE (79 bytes, valid CRCs) -> 415 +palette larger than bit depth allows (depth 1, 3 entries) (88 bytes, valid CRCs) -> 415 +palette image without PLTE (67 bytes, valid CRCs) -> 415 +PLTE in a grayscale image (85 bytes, valid CRCs) -> 415 +PLTE after IDAT (85 bytes, valid CRCs) -> 415 +two PLTE chunks (103 bytes, valid CRCs) -> 415 +tRNS longer than the palette (100 bytes, valid CRCs) -> 415 +tRNS before PLTE (98 bytes, valid CRCs) -> 415 +tRNS of wrong size for RGB (4 bytes instead of 6) (85 bytes, valid CRCs) -> 415 +tRNS in an image with alpha channel (82 bytes, valid CRCs) -> 415 +IEND with data (70 bytes, valid CRCs) -> 415 +unknown critical chunk (81 bytes, valid CRCs) -> 415 +IDAT chunks not consecutive (95 bytes, valid CRCs) -> 415 +gAMA after IDAT (85 bytes, valid CRCs) -> 415 +gAMA of 2 bytes instead of 4 (83 bytes, valid CRCs) -> 415 +3 MiB metadata chunk before IDAT (3145809 bytes, valid CRCs) -> 415 +private chunk after IDAT (181 bytes, valid CRCs) -> 415 +animated PNG frame (fdAT) (91 bytes, valid CRCs) -> 415 +tIME of 2 bytes instead of 7 (83 bytes, valid CRCs) -> 415 +tEXt without keyword separator (86 bytes, valid CRCs) -> 415 +zTXt with broken zlib after IDAT (91 bytes, valid CRCs) -> 415 +zTXt text bomb: 50 MB of text in ~50 KB after IDAT (48694 bytes, valid CRCs) -> 415 +palette 256 entries + tRNS + tEXt and tIME after IDAT -> 201, downloaded byte for byte +palette depth 2 with 4 entries -> 201, downloaded byte for byte +RGB with 6-byte tRNS and gAMA -> 201, downloaded byte for byte +compressed zTXt (100 KB of text) and iTXt after IDAT -> 201, downloaded byte for byte +PASS: chunk contents are checked, not only CRCs; valid PNGs with metadata are still accepted PNG bytes sent as image/jpeg -> 415 JPEG bytes sent as image/png -> 415 GIF bytes sent as image/png -> 415 @@ -64,4 +93,4 @@ PASS: Idempotency-Key covers attachment_ids sent 1.1 MB of a declared 5 MB body and closed the connection the API keeps serving uploads; nothing of the aborted request is visible (DB/S3 side is checked by test_consistency.py) PASS: an interrupted upload is dropped -11 passed, 0 failed +12 passed, 0 failed diff --git a/evidence/balancing.txt b/evidence/balancing.txt index dbbf431..6038195 100644 --- a/evidence/balancing.txt +++ b/evidence/balancing.txt @@ -1,18 +1,18 @@ # balancing: balancing, shared sessions/data/files, X-Instance-Id -# date: 2026-10-21 21:08:50, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:22:27, commit: c942c61 + uncommitted changes # command: python.exe student-tests/test_balancing.py -# exit code: 0, 1.9 s +# exit code: 0, 2.0 s -20 x GET /users/me with one token -> replicas: {'6128eb268c44': 10, '11a93acecaca': 10} +20 x GET /users/me with one token -> replicas: {'d590adc21943': 10, '343119df2488': 10} PASS: one session works on every replica, no second login (sessions are shared in Redis) -message written by 11a93acecaca; read back by ['11a93acecaca', '6128eb268c44'] +message written by 343119df2488; read back by ['343119df2488', 'd590adc21943'] PASS: a message written via one replica is read right away via another (shared PostgreSQL) -logout on 6128eb268c44; old token then rejected by ['11a93acecaca', '6128eb268c44'] +logout on d590adc21943; old token then rejected by ['343119df2488', 'd590adc21943'] PASS: logout on one replica revokes the session on all of them -first POST on 6128eb268c44 -> 201 296cb1bf-8c1c-46f2-942a-4865d1b7d6af; same key on 11a93acecaca -> 201 296cb1bf-8c1c-46f2-942a-4865d1b7d6af +first POST on d590adc21943 -> 201 6c2f624d-c728-4738-b486-b17ab735c0ae; same key on 343119df2488 -> 201 6c2f624d-c728-4738-b486-b17ab735c0ae PASS: Idempotency-Key is shared: a retry through another replica returns the same message -file uploaded via 6128eb268c44; downloaded byte for byte via ['11a93acecaca', '6128eb268c44'] +file uploaded via d590adc21943; downloaded byte for byte via ['343119df2488', 'd590adc21943'] PASS: a file uploaded through one replica is downloaded through another (shared S3, nothing local) -statuses [200, 200, 201, 400, 401, 404, 409]: every answer has X-Instance-Id of a running container ['11a93acecaca', '6128eb268c44']; api containers: ['11a93acecaca', '6128eb268c44'] +statuses [200, 200, 201, 400, 401, 404, 409]: every answer has X-Instance-Id of a running container ['343119df2488', 'd590adc21943']; api containers: ['343119df2488', 'd590adc21943'] PASS: X-Instance-Id is in every response and points to a real container 6 passed, 0 failed diff --git a/evidence/check.txt b/evidence/check.txt index 61c52c6..8d74695 100644 --- a/evidence/check.txt +++ b/evidence/check.txt @@ -1,5 +1,5 @@ # check: offline contract check (make check) -# date: 2026-10-21 21:02:57, commit: 2dbed80 +# date: 2026-10-22 14:19:16, commit: c942c61 + uncommitted changes # command: python.exe scripts/check_contract.py # exit code: 0, 0.1 s diff --git a/evidence/consistency.txt b/evidence/consistency.txt index b345844..e4cb476 100644 --- a/evidence/consistency.txt +++ b/evidence/consistency.txt @@ -1,47 +1,47 @@ # consistency: lab4: aborted upload, crash between S3 and DB, janitor, orphans, S3 outage -# date: 2026-10-21 21:08:14, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:21:48, commit: c942c61 + uncommitted changes # command: python.exe student-tests/test_consistency.py -# exit code: 0, 148.7 s +# exit code: 0, 133.0 s declared 9 000 000 bytes, sent 211817, closed the connection declared 9 000 000 bytes, sent 3111817, closed the connection -rows in attachments: 32 -> 32; objects in bucket: 31 -> 31 -temporary upload files in api containers: {'11a93acecaca': 0, '6128eb268c44': 0} (removed when the request ends) +rows in attachments: 448 -> 448; objects in bucket: 444 -> 444 +temporary upload files in api containers: {'343119df2488': 0, 'd590adc21943': 0} (removed when the request ends) api containers have no volumes: the original is never stored locally PASS: an interrupted upload leaves no row, no object and no local file -crash after S3 PUT: row 04aacbf6 state=uploading, object originals/219175ec1d5647f0aeaec84ee8243cb3 exists in S3 +crash after S3 PUT: row 9aa0bc8a state=uploading, object originals/8ae03455b55a4f798b7bc6248b450ef9 exists in S3 the half-written attachment is invisible: GET -> 404, attach to a message -> 404 -janitor with default grace (3600 s) right away: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 33, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.101} -> fresh row and object are kept (the upload might still be running) -row made 2 h old; janitor --dry-run: {"dry_run": true, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 0, "objects_scanned": 33, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.092} -> nothing deleted -janitor: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 32, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.096} -> row and object deleted -janitor again: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 32, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.103} -> idempotent +janitor with default grace (3600 s) right away: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 446, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.133} -> fresh row and object are kept (the upload might still be running) +row made 2 h old; janitor --dry-run: {"dry_run": true, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 0, "objects_scanned": 446, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.184} -> nothing deleted +janitor: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 445, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.132} -> row and object deleted +janitor again: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 445, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.179} -> idempotent the ready attachment of the same chat is untouched PASS: S3 written but DB not committed -> invisible, reclaimed by janitor only after the grace period -object originals/867f5c5d42bf4bf9973b60ef8856d88f put into the bucket without a row in PostgreSQL -janitor --grace-seconds 3 immediately: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 34, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.102} -> fresh object kept -after 4 s, --dry-run: {"dry_run": true, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 34, "orphan_objects": 1, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.098} -> reported, kept -janitor: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 34, "orphan_objects": 1, "orphan_objects_deleted": 1, "missing_objects": 1, "seconds": 0.1} -> orphan deleted +object originals/101748e3a911439a9c1e11e768c99822 put into the bucket without a row in PostgreSQL +janitor --grace-seconds 3 immediately: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 447, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.191} -> fresh object kept +after 4 s, --dry-run: {"dry_run": true, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 447, "orphan_objects": 1, "orphan_objects_deleted": 0, "missing_objects": 4, "seconds": 0.188} -> reported, kept +janitor: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "objects_scanned": 447, "orphan_objects": 1, "orphan_objects_deleted": 1, "missing_objects": 4, "seconds": 0.1} -> orphan deleted ready attachments older than the grace period are untouched (they have rows) -object of a ready attachment deleted behind the API's back: janitor reports missing_objects=2 and keeps the row; download -> 503 (not 500) +object of a ready attachment deleted behind the API's back: janitor reports missing_objects=5 and keeps the row; download -> 503 (not 500) PASS: orphans are found and deleted only after the grace period; missing objects are reported --- S3 stopped (connection refused / name not resolved): docker compose stop s3 -upload -> 503 unavailable in 20141 ms, no storage details in the body +upload -> 503 unavailable in 19453 ms, no storage details in the body download of an existing file -> 503 metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API docker compose start s3 after S3 recovery the old file downloads byte for byte new upload -> 201 -janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 35, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.099} +janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 448, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 5, "seconds": 0.191} PASS: S3 stopped (connection refused / name not resolved) -> 503 for file operations only, no visible half-uploads, everything cleaned up --- S3 frozen (no answer, timeouts): docker compose pause s3 -upload -> 503 unavailable in 14766 ms, no storage details in the body +upload -> 503 unavailable in 14891 ms, no storage details in the body download of an existing file -> 503 metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API docker compose unpause s3 after S3 recovery the old file downloads byte for byte new upload -> 201 -janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 37, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.094} +janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "objects_scanned": 450, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 5, "seconds": 0.115} PASS: S3 frozen (no answer, timeouts) -> 503 for file operations only, no visible half-uploads, everything cleaned up 5 passed, 0 failed diff --git a/evidence/db.txt b/evidence/db.txt index 481e403..77cb2e7 100644 --- a/evidence/db.txt +++ b/evidence/db.txt @@ -1,9 +1,9 @@ # db: PostgreSQL: rollback, FK, Argon2id, unique keys, migrations -# date: 2026-10-21 21:09:10, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:22:40, commit: c942c61 + uncommitted changes # command: python.exe student-tests/test_db.py -# exit code: 0, 14.2 s +# exit code: 0, 6.2 s -chat with unknown member -> 400; rows with this title: 0; chats without members: 0; chats insert counter: 57 -> 58 +chat with unknown member -> 400; rows with this title: 0; chats without members: 0; chats insert counter: 335 -> 336 PASS: chat row was inserted and then rolled back together with members direct INSERT of a message from a non-member -> exit code 1: ERROR: insert or update on table "messages" violates foreign key constraint "messages_chat_id_sender_id_fkey" PASS: the database itself rejects a message from a non-member @@ -14,7 +14,7 @@ PASS: username is unique in the database applied migrations: 001_init.sql,002_text_as_bytes.sql,003_idempotency.sql,004_attachments.sql; running the migrate service once more after the second run: 001_init.sql,002_text_as_bytes.sql,003_idempotency.sql,004_attachments.sql; log: Migrations applied: none (schema is up to date) PASS: migrations are applied once, a repeated run does nothing -api replicas that checked the schema at start: 8; api log lines 'Migrations applied': 0; migrate service: Exited (0) 6 minutes ago +api replicas that checked the schema at start: 14; api log lines 'Migrations applied': 0; migrate service: Exited (0) 4 minutes ago PASS: only the migrate service applies migrations, replicas wait for the schema unique constraint for Idempotency-Key: UNIQUE (chat_id, sender_id, idempotency_key) PASS: one message per (chat, sender, Idempotency-Key) is enforced by the database diff --git a/evidence/explain-history.txt b/evidence/explain-history.txt index 678f473..46f4949 100644 --- a/evidence/explain-history.txt +++ b/evidence/explain-history.txt @@ -1,7 +1,7 @@ === Data volume messages_total | messages_in_big_chat ----------------+---------------------- - 20146 | 15000 + 35618 | 15000 === History index @@ -13,39 +13,39 @@ === 1. First page: limit=50, the app asks for 51 rows to know if there is a next page QUERY PLAN ------------------------------------------------------------------------------------------------ - Limit (actual time=0.005..0.011 rows=51 loops=1) - Buffers: shared hit=5 + Limit (actual time=0.006..0.010 rows=51 loops=1) + Buffers: shared hit=4 -> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1) - Index Cond: (chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) - Buffers: shared hit=5 + Index Cond: (chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) + Buffers: shared hit=4 Planning: - Buffers: shared hit=6 - Planning Time: 0.060 ms + Buffers: shared hit=8 + Planning Time: 0.071 ms Execution Time: 0.020 ms === 2. Page after the 10000th message by keyset cursor (this is what the app does) QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ - Limit (actual time=0.005..0.012 rows=51 loops=1) - Buffers: shared hit=6 - -> Index Scan using messages_history on messages (actual time=0.005..0.009 rows=51 loops=1) - Index Cond: ((chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '170ea586-3f7e-4314-a9bf-8e4f394941ac'::uuid))) - Buffers: shared hit=6 - Planning Time: 0.044 ms - Execution Time: 0.031 ms + Limit (actual time=0.006..0.011 rows=51 loops=1) + Buffers: shared hit=5 + -> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1) + Index Cond: ((chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '4bd22867-547f-4ecf-8182-443fbb004188'::uuid))) + Buffers: shared hit=5 + Planning Time: 0.040 ms + Execution Time: 0.029 ms === 2b. The same page exactly as the app sends it since lab4: with attachment_ids of every message QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ - Limit (actual time=0.020..0.225 rows=51 loops=1) - Buffers: shared hit=60 - -> Index Scan using messages_history on messages m (actual time=0.019..0.222 rows=51 loops=1) - Index Cond: ((chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '170ea586-3f7e-4314-a9bf-8e4f394941ac'::uuid))) - Buffers: shared hit=60 + Limit (actual time=0.019..0.208 rows=51 loops=1) + Buffers: shared hit=59 + -> Index Scan using messages_history on messages m (actual time=0.019..0.204 rows=51 loops=1) + Index Cond: ((chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '4bd22867-547f-4ecf-8182-443fbb004188'::uuid))) + Buffers: shared hit=59 SubPlan 1 - -> Sort (actual time=0.003..0.003 rows=0 loops=51) + -> Sort (actual time=0.002..0.002 rows=0 loops=51) Sort Key: a."position" Sort Method: quicksort Memory: 25kB Buffers: shared hit=54 @@ -57,39 +57,44 @@ Buffers: shared hit=51 Planning: Buffers: shared hit=63 - Planning Time: 0.157 ms - Execution Time: 0.242 ms + Planning Time: 0.156 ms + Execution Time: 0.225 ms === 3. The same page with OFFSET 10000 (the app does NOT do this) - QUERY PLAN ---------------------------------------------------------------------------------------------------- - Limit (actual time=0.878..0.884 rows=51 loops=1) - Buffers: shared hit=258 - -> Index Scan using messages_history on messages (actual time=0.007..0.669 rows=10051 loops=1) - Index Cond: (chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) - Buffers: shared hit=258 - Planning Time: 0.037 ms - Execution Time: 0.897 ms + QUERY PLAN +-------------------------------------------------------------------------------- + Limit (actual time=3.056..3.060 rows=51 loops=1) + Buffers: shared hit=585 + -> Sort (actual time=2.647..2.857 rows=10051 loops=1) + Sort Key: created_at, id + Sort Method: quicksort Memory: 2377kB + Buffers: shared hit=585 + -> Seq Scan on messages (actual time=0.005..1.278 rows=15000 loops=1) + Filter: (chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) + Rows Removed by Filter: 20618 + Buffers: shared hit=585 + Planning Time: 0.033 ms + Execution Time: 3.107 ms === 4. Query 2 without the index (index is dropped only inside this transaction) QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - Limit (actual time=1.097..1.101 rows=51 loops=1) - Buffers: shared hit=261 - -> Sort (actual time=1.097..1.098 rows=51 loops=1) + Limit (actual time=1.432..1.436 rows=51 loops=1) + Buffers: shared hit=585 + -> Sort (actual time=1.432..1.433 rows=51 loops=1) Sort Key: created_at, id Sort Method: top-N heapsort Memory: 31kB - Buffers: shared hit=261 - -> Seq Scan on messages (actual time=0.307..0.737 rows=5000 loops=1) - Filter: ((chat_id = '7c307498-c398-4a1e-be96-13556a559cbf'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '170ea586-3f7e-4314-a9bf-8e4f394941ac'::uuid))) - Rows Removed by Filter: 15146 - Buffers: shared hit=261 + Buffers: shared hit=585 + -> Seq Scan on messages (actual time=0.678..1.116 rows=5000 loops=1) + Filter: ((chat_id = '872415a3-3ab7-4e2b-bbc5-e5d03904fd1d'::uuid) AND (ROW(created_at, id) > ROW('2026-09-01 02:46:40+00'::timestamp with time zone, '4bd22867-547f-4ecf-8182-443fbb004188'::uuid))) + Rows Removed by Filter: 30618 + Buffers: shared hit=585 Planning: Buffers: shared hit=4 - Planning Time: 0.051 ms - Execution Time: 1.115 ms + Planning Time: 0.049 ms + Execution Time: 1.450 ms === ROLLBACK: test data removed, index is back diff --git a/evidence/explain.txt b/evidence/explain.txt index 597939b..ac52e43 100644 --- a/evidence/explain.txt +++ b/evidence/explain.txt @@ -1,43 +1,44 @@ # explain: EXPLAIN of the history page (full plans: explain-history.txt) -# date: 2026-10-21 21:09:14, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:22:45, commit: c942c61 + uncommitted changes # command: python.exe student-tests/explain.py # exit code: 0, 1.1 s === Data volume === History index === 1. First page: limit=50, the app asks for 51 rows to know if there is a next page + Buffers: shared hit=4 + -> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1) + Buffers: shared hit=4 + Buffers: shared hit=8 + Execution Time: 0.020 ms +=== 2. Page after the 10000th message by keyset cursor (this is what the app does) Buffers: shared hit=5 -> Index Scan using messages_history on messages (actual time=0.005..0.008 rows=51 loops=1) Buffers: shared hit=5 - Buffers: shared hit=6 - Execution Time: 0.020 ms -=== 2. Page after the 10000th message by keyset cursor (this is what the app does) - Buffers: shared hit=6 - -> Index Scan using messages_history on messages (actual time=0.005..0.009 rows=51 loops=1) - Buffers: shared hit=6 - Execution Time: 0.031 ms + Execution Time: 0.029 ms === 2b. The same page exactly as the app sends it since lab4: with attachment_ids of every message - Buffers: shared hit=60 - -> Index Scan using messages_history on messages m (actual time=0.019..0.222 rows=51 loops=1) - Buffers: shared hit=60 + Buffers: shared hit=59 + -> Index Scan using messages_history on messages m (actual time=0.019..0.204 rows=51 loops=1) + Buffers: shared hit=59 Buffers: shared hit=54 -> Bitmap Heap Scan on message_attachments a (actual time=0.000..0.000 rows=0 loops=51) Buffers: shared hit=51 -> Bitmap Index Scan on message_attachments_message_id_attachment_id_key (actual time=0.000..0.000 rows=0 loops=51) Buffers: shared hit=51 Buffers: shared hit=63 - Execution Time: 0.242 ms + Execution Time: 0.225 ms === 3. The same page with OFFSET 10000 (the app does NOT do this) - Buffers: shared hit=258 - -> Index Scan using messages_history on messages (actual time=0.007..0.669 rows=10051 loops=1) - Buffers: shared hit=258 - Execution Time: 0.897 ms + Buffers: shared hit=585 + Buffers: shared hit=585 + -> Seq Scan on messages (actual time=0.005..1.278 rows=15000 loops=1) + Buffers: shared hit=585 + Execution Time: 3.107 ms === 4. Query 2 without the index (index is dropped only inside this transaction) - Buffers: shared hit=261 - Buffers: shared hit=261 - -> Seq Scan on messages (actual time=0.307..0.737 rows=5000 loops=1) - Buffers: shared hit=261 + Buffers: shared hit=585 + Buffers: shared hit=585 + -> Seq Scan on messages (actual time=0.678..1.116 rows=5000 loops=1) + Buffers: shared hit=585 Buffers: shared hit=4 - Execution Time: 1.115 ms + Execution Time: 1.450 ms === ROLLBACK: test data removed, index is back PASS: full plans saved to evidence\explain-history.txt diff --git a/evidence/failover.txt b/evidence/failover.txt index 9bb22e8..26db5df 100644 --- a/evidence/failover.txt +++ b/evidence/failover.txt @@ -1,33 +1,33 @@ # failover: public failover test for every replica, file downloaded via the other one (make test-failover) -# date: 2026-10-21 21:09:54, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:23:28, commit: c942c61 + uncommitted changes # command: python.exe student-tests/run_failover.py -# exit code: 0, 39.7 s +# exit code: 0, 43.2 s -hook: scripts\stop-one.cmd; api replicas: ['11a93acecaca', '6128eb268c44'] +hook: scripts\stop-one.cmd; api replicas: ['343119df2488', 'd590adc21943'] === run 1: api containers before SERVICE CONTAINER ID STATUS -api 11a93acecaca Up 36 seconds (healthy) -api 6128eb268c44 Up 36 seconds (healthy) +api 343119df2488 Up 27 seconds (healthy) +api d590adc21943 Up 27 seconds (healthy) Executing explicit failover hook; hook output suppressed to avoid leaking secrets. -{"observed_instances_before": 2, "successful_writes": 126, "failed_attempts": 0, "first_recovery_seconds": 4.39} +{"observed_instances_before": 2, "successful_writes": 123, "failed_attempts": 0, "first_recovery_seconds": 4.422} PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. api containers after SERVICE CONTAINER ID STATUS -api 11a93acecaca Exited (0) 11 seconds ago -api 6128eb268c44 Up 53 seconds (healthy) -the hook stopped 11a93acecaca (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched -restored: docker start 11a93acecaca -> healthy and back in HAProxy +api 343119df2488 Exited (0) 11 seconds ago +api d590adc21943 Up 43 seconds (healthy) +the hook stopped 343119df2488 (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched +restored: docker start 343119df2488 -> healthy and back in HAProxy === run 2: api containers before SERVICE CONTAINER ID STATUS -api 11a93acecaca Up 2 seconds (healthy) -api 6128eb268c44 Up 56 seconds (healthy) +api 343119df2488 Up 4 seconds (healthy) +api d590adc21943 Up 48 seconds (healthy) Executing explicit failover hook; hook output suppressed to avoid leaking secrets. -{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 4.328} +{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 4.531} PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. api containers after SERVICE CONTAINER ID STATUS -api 11a93acecaca Up 18 seconds (healthy) -api 6128eb268c44 Exited (0) 11 seconds ago -the hook stopped 6128eb268c44 (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched -restored: docker start 6128eb268c44 -> healthy and back in HAProxy +api 343119df2488 Up 20 seconds (healthy) +api d590adc21943 Exited (0) 11 seconds ago +the hook stopped d590adc21943 (X-Instance-Id of that replica = container ID); other api containers kept running, postgres/redis/lb were not touched +restored: docker start d590adc21943 -> healthy and back in HAProxy PASS: public failover test passed 2 times; every api replica was stopped once and restored diff --git a/evidence/failures.txt b/evidence/failures.txt index 28d1d69..ce7d206 100644 --- a/evidence/failures.txt +++ b/evidence/failures.txt @@ -1,11 +1,11 @@ # failures: Redis and PostgreSQL outages via the balancer -# date: 2026-10-21 21:11:20, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:24:54, commit: c942c61 + uncommitted changes # command: python.exe student-tests/test_failures.py # exit code: 0, 55.6 s stopping redis -/health/live -> 200, /health/ready -> 503 (8016 ms) -valid token: GET /users/me -> 503 unavailable (8000 ms), session cannot be checked +/health/live -> 200, /health/ready -> 503 (8000 ms) +valid token: GET /users/me -> 503 unavailable (8031 ms), session cannot be checked unknown token -> 503, not 200: nobody is trusted without a check no token -> 401 login -> 503: there is nowhere to store the session @@ -19,7 +19,7 @@ starting redis after start: the session issued before the crash still works (AOF, appendfsync always) PASS: sessions survive a Redis crash, not only a clean restart stopping postgres -/health/live -> 200, /health/ready -> 503 (1016 ms) +/health/live -> 200, /health/ready -> 503 (1031 ms) valid token: users/me, chats, history, new message -> 503 unavailable, no details no token -> 401, unknown token -> 401: anonymous users are not let in login -> 503, registration -> 503 diff --git a/evidence/idempotency.txt b/evidence/idempotency.txt index 205a798..fce0432 100644 --- a/evidence/idempotency.txt +++ b/evidence/idempotency.txt @@ -1,11 +1,11 @@ # idempotency: Idempotency-Key, parallel same key via replicas -# date: 2026-10-21 21:08:52, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:22:30, commit: c942c61 + uncommitted changes # command: python.exe student-tests/test_idempotency.py -# exit code: 0, 2.4 s +# exit code: 0, 2.5 s first -> 201; same JSON written differently -> 201; plain retry -> 201 PASS: a retry with the same key returns the original 201 and message, no duplicate -20 parallel POSTs with one key: statuses {201: 20}, replicas {'6128eb268c44': 10, '11a93acecaca': 10}, different bodies in answers: 1, messages in history: 1 +20 parallel POSTs with one key: statuses {201: 20}, replicas {'343119df2488': 10, 'd590adc21943': 10}, different bodies in answers: 1, messages in history: 1 PASS: parallel requests with one key through different replicas create exactly one message 20 parallel POSTs, one key, two different bodies: {('body A', 201): 10, ('body B', 409): 10}; saved: ['body A'] PASS: the first body wins, the same body gets 201, the other body gets 409 diff --git a/evidence/load.json b/evidence/load.json index c4db3af..c41ff89 100644 --- a/evidence/load.json +++ b/evidence/load.json @@ -2,12 +2,12 @@ "lab": 4, "workload": "closed-loop POST messages; no automatic retries", "requested_duration_seconds": 30.0, - "elapsed_seconds": 30.015, + "elapsed_seconds": 30.047, "concurrency": 4, - "successful_requests": 7379, + "successful_requests": 7191, "failed_requests": 0, "error_ratio": 0.0, - "successful_rps": 245.844, + "successful_rps": 239.325, "successful_latency_ms": { "p50": 16.0, "p95": 32.0, diff --git a/evidence/load.txt b/evidence/load.txt index 8529586..2d10d4a 100644 --- a/evidence/load.txt +++ b/evidence/load.txt @@ -1,18 +1,18 @@ # load: public load generator, 30 s, concurrency 4 -# date: 2026-10-21 21:10:24, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:23:58, commit: c942c61 + uncommitted changes # command: python.exe tests/load.py --base-url http://127.0.0.1:8080 --duration 30 --concurrency 4 --output evidence/load.json -# exit code: 0, 30.4 s +# exit code: 0, 30.3 s { "lab": 4, "workload": "closed-loop POST messages; no automatic retries", "requested_duration_seconds": 30.0, - "elapsed_seconds": 30.015, + "elapsed_seconds": 30.047, "concurrency": 4, - "successful_requests": 7379, + "successful_requests": 7191, "failed_requests": 0, "error_ratio": 0.0, - "successful_rps": 245.844, + "successful_rps": 239.325, "successful_latency_ms": { "p50": 16.0, "p95": 32.0, diff --git a/evidence/persistence.txt b/evidence/persistence.txt index f23f416..b7334d5 100644 --- a/evidence/persistence.txt +++ b/evidence/persistence.txt @@ -1,24 +1,24 @@ # persistence: public persistence test incl. S3 + proof of real restarts (make test-persistence) -# date: 2026-10-21 21:12:49, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:25:42, commit: c942c61 + uncommitted changes # command: python.exe student-tests/run_persistence.py -# exit code: 0, 40.2 s +# exit code: 0, 47.9 s before: container, ID, started at - messenger-lab4-api-1 11a93acecaca 2026-10-21T18:11:37.067348364Z - messenger-lab4-api-2 6128eb268c44 2026-10-21T18:11:36.916459116Z - messenger-lab4-janitor-1 baefc0e97dff 2026-10-21T18:11:33.313531682Z - messenger-lab4-lb-1 03697e5fe34c 2026-10-21T18:11:40.981519931Z - messenger-lab4-postgres-1 f7a6a1ce25a7 2026-10-21T18:11:22.482229396Z - messenger-lab4-redis-1 b4523b39a0fe 2026-10-21T18:11:22.626999402Z - messenger-lab4-s3-1 e7cbe6ddfa4a 2026-10-21T18:11:32.264154477Z + messenger-lab4-api-1 343119df2488 2026-10-22T11:23:03.387600262Z + messenger-lab4-api-2 d590adc21943 2026-10-22T11:23:24.689368662Z + messenger-lab4-janitor-1 d50ac08018c1 2026-10-22T11:18:17.423165415Z + messenger-lab4-lb-1 0d5b262b09b3 2026-10-22T11:03:30.971491955Z + messenger-lab4-postgres-1 84794c3ac2c5 2026-10-22T11:24:52.672662576Z + messenger-lab4-redis-1 3130fd5532b0 2026-10-22T11:24:32.055032931Z + messenger-lab4-s3-1 122ad603e3be 2026-10-22T11:20:52.20174809Z Executing explicit persistence hook; hook output suppressed to avoid leaking secrets. PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence. -after (hook + waiting for readiness took 39.2 s): container, ID, started at, restarted? - messenger-lab4-api-1 11a93acecaca 2026-10-21T18:12:46.288933033Z same container: True, restarted: True - messenger-lab4-api-2 6128eb268c44 2026-10-21T18:12:46.174283308Z same container: True, restarted: True - messenger-lab4-janitor-1 baefc0e97dff 2026-10-21T18:12:42.503993973Z same container: True, restarted: True - messenger-lab4-lb-1 03697e5fe34c 2026-10-21T18:12:47.684809415Z same container: True, restarted: True - messenger-lab4-postgres-1 f7a6a1ce25a7 2026-10-21T18:12:11.527270274Z same container: True, restarted: True - messenger-lab4-redis-1 b4523b39a0fe 2026-10-21T18:12:11.368416107Z same container: True, restarted: True - messenger-lab4-s3-1 e7cbe6ddfa4a 2026-10-21T18:12:21.264135454Z same container: True, restarted: True +after (hook + waiting for readiness took 47.0 s): container, ID, started at, restarted? + messenger-lab4-api-1 343119df2488 2026-10-22T11:25:36.201352227Z same container: True, restarted: True + messenger-lab4-api-2 d590adc21943 2026-10-22T11:25:36.073058122Z same container: True, restarted: True + messenger-lab4-janitor-1 d50ac08018c1 2026-10-22T11:25:32.252887377Z same container: True, restarted: True + messenger-lab4-lb-1 0d5b262b09b3 2026-10-22T11:25:40.174401907Z same container: True, restarted: True + messenger-lab4-postgres-1 84794c3ac2c5 2026-10-22T11:24:56.311470336Z same container: True, restarted: True + messenger-lab4-redis-1 3130fd5532b0 2026-10-22T11:24:56.150164333Z same container: True, restarted: True + messenger-lab4-s3-1 122ad603e3be 2026-10-22T11:25:06.0234901Z same container: True, restarted: True PASS: the hook restarted postgres, redis, s3 (SeaweedFS), all api replicas, janitor and lb (same containers, new start time); data, the attachment bytes and the old token survived diff --git a/evidence/race.txt b/evidence/race.txt index 7e5c4df..b638d87 100644 --- a/evidence/race.txt +++ b/evidence/race.txt @@ -1,9 +1,9 @@ # race: lab2 races via the balancer -# date: 2026-10-21 21:08:56, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:22:34, commit: c942c61 + uncommitted changes # command: python.exe student-tests/test_race.py -# exit code: 0, 1.2 s +# exit code: 0, 1.1 s -sent 100 messages from 10 clients in 0.22 s; read 100 in 4 pages, unique ids: 100 +sent 100 messages from 10 clients in 0.19 s; read 100 in 4 pages, unique ids: 100 PASS: 100 concurrent messages from 10 clients, no losses and no duplicates 20 parallel registrations of one username: 201 x1, 409 x19 PASS: concurrent registration created exactly one user diff --git a/evidence/redis.txt b/evidence/redis.txt index b118ac4..c3f0159 100644 --- a/evidence/redis.txt +++ b/evidence/redis.txt @@ -1,11 +1,11 @@ # redis: Redis as the session store -# date: 2026-10-21 21:09:13, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:22:43, commit: c942c61 + uncommitted changes # command: python.exe student-tests/redis_info.py -# exit code: 0, 2.8 s +# exit code: 0, 2.9 s -session key: session:f4f1fe0235d139de...; value is the user id: True +session key: session:e57bdd083b4cb87e...; value is the user id: True TTL of the key, seconds: 3599 -sessions in Redis: 126 +sessions in Redis: 330 config appendonly = yes config appendfsync = always config maxmemory-policy = noeviction diff --git a/evidence/s3-security.txt b/evidence/s3-security.txt index d118d61..d509931 100644 --- a/evidence/s3-security.txt +++ b/evidence/s3-security.txt @@ -1,7 +1,7 @@ # s3-security: lab4: private bucket outside API, limited API key, no console/ports, idempotent bootstrap -# date: 2026-10-21 21:03:14, commit: 2dbed80 +# date: 2026-10-22 14:19:35, commit: c942c61 + uncommitted changes # command: python.exe student-tests/test_s3_security.py -# exit code: 0, 9.5 s +# exit code: 0, 9.6 s anonymous get_object directly to S3 (from the data network) -> 403 AccessDenied anonymous head_object directly to S3 (from the data network) -> 403 diff --git a/evidence/smoke.txt b/evidence/smoke.txt index 95abed9..9da4055 100644 --- a/evidence/smoke.txt +++ b/evidence/smoke.txt @@ -1,6 +1,6 @@ # smoke: public smoke via the balancer (make test), with attachment -# date: 2026-10-21 21:03:01, commit: 2dbed80 +# date: 2026-10-22 14:19:20, commit: c942c61 + uncommitted changes # command: python.exe tests/smoke.py --base-url http://127.0.0.1:8080 -# exit code: 0, 0.6 s +# exit code: 0, 0.8 s PASS: lab4 public API scenario; infrastructure and grading evidence remain separate. diff --git a/evidence/stack.txt b/evidence/stack.txt index 9b6415d..ef8ddb1 100644 --- a/evidence/stack.txt +++ b/evidence/stack.txt @@ -1,21 +1,21 @@ # stack: stand layout: replicas, ports, limits, networks, secrets, S3 -# date: 2026-10-21 21:03:00, commit: 2dbed80 +# date: 2026-10-22 14:19:19, commit: c942c61 + uncommitted changes # command: python.exe student-tests/check_stack.py -# exit code: 0, 3.0 s +# exit code: 0, 2.9 s host: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; client Python 3.11.1 Docker 28.1.1, Docker Compose version v2.35.1-desktop.1; Docker VM: 16 CPU, 15.2 GiB RAM, Docker Desktop -SERVICE CONTAINER ID STATUS PORTS -api 11a93acecaca Up 20 seconds (healthy) 8080/tcp -api 6128eb268c44 Up 20 seconds (healthy) 8080/tcp -janitor baefc0e97dff Up 20 seconds -lb 03697e5fe34c Up 18 seconds (healthy) 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp -migrate aca090330db4 Exited (0) 22 seconds ago -postgres f7a6a1ce25a7 Up 2 minutes (healthy) -redis b4523b39a0fe Up 2 minutes (healthy) -s3 e7cbe6ddfa4a Up 2 minutes (healthy) -s3-init 60719e674e18 Exited (0) 21 seconds ago +SERVICE CONTAINER ID STATUS PORTS +api 343119df2488 Up 17 seconds (healthy) 8080/tcp +api d590adc21943 Up 18 seconds (healthy) 8080/tcp +janitor d50ac08018c1 Up About a minute +lb 0d5b262b09b3 Up 15 minutes (healthy) 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp +migrate 6c3123ccf237 Exited (0) About a minute ago +postgres 84794c3ac2c5 Up 16 minutes (healthy) +redis 3130fd5532b0 Up 16 minutes (healthy) +s3 122ad603e3be Up 16 minutes (healthy) +s3-init 47c2d7d47ed2 Exited (0) About a minute ago container image cpus memory host ports networks messenger-lab4-api-1 messenger-lab4-api 1.00 512 MiB - data,edge @@ -35,6 +35,6 @@ secrets of redis: ['redis_auth'] secrets of s3: ['s3_config'] secrets of s3-init: ['s3_access_key_id', 's3_root_password', 's3_root_user', 's3_secret_access_key'] -through http://127.0.0.1:8080 one session is served by 2 replicas: ['11a93acecaca', '6128eb268c44'] -/health/ready -> 200 from 11a93acecaca +through http://127.0.0.1:8080 one session is served by 2 replicas: ['343119df2488', 'd590adc21943'] +/health/ready -> 200 from d590adc21943 PASS: 2 api replicas behind one entry point, no direct API ports, limits on every container, postgres/redis/s3/janitor only in the internal data network, S3 admin keys only in s3 and s3-init diff --git a/evidence/student-tests.txt b/evidence/student-tests.txt index 9ac2c0f..980ab73 100644 --- a/evidence/student-tests.txt +++ b/evidence/student-tests.txt @@ -1,7 +1,7 @@ # student-tests: lab1/lab2 API tests via the balancer -# date: 2026-10-21 21:08:55, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:22:33, commit: c942c61 + uncommitted changes # command: python.exe student-tests/test_api.py -# exit code: 0, 2.7 s +# exit code: 0, 2.9 s PASS: duplicate username returns 409 PASS: invalid JSON returns 400 diff --git a/evidence/summary.md b/evidence/summary.md index 5df443a..465f181 100644 --- a/evidence/summary.md +++ b/evidence/summary.md @@ -2,39 +2,56 @@ | Шаг | Что проверяет | Результат | Время, с | Файл | | --- | --- | --- | --- | --- | -| persistence | public persistence test incl. S3 + proof of real restarts (make test-persistence) | PASS | 40 | evidence/persistence.txt | +| check | offline contract check (make check) | PASS | 0 | evidence/check.txt | +| stack | stand layout: replicas, ports, limits, networks, secrets, S3 | PASS | 3 | evidence/stack.txt | +| smoke | public smoke via the balancer (make test), with attachment | PASS | 1 | evidence/smoke.txt | +| attachments | lab4: files, limits, damaged images, MIME, filenames, access, attachment_ids | PASS | 5 | evidence/attachments.txt | +| s3-security | lab4: private bucket outside API, limited API key, no console/ports, idempotent bootstrap | PASS | 10 | evidence/s3-security.txt | +| consistency | lab4: aborted upload, crash between S3 and DB, janitor, orphans, S3 outage | PASS | 133 | evidence/consistency.txt | +| upload-memory | lab4: >=10 concurrent uploads, peak memory of api replicas | PASS | 37 | evidence/upload-memory.txt | +| balancing | balancing, shared sessions/data/files, X-Instance-Id | PASS | 2 | evidence/balancing.txt | +| idempotency | Idempotency-Key, parallel same key via replicas | PASS | 2 | evidence/idempotency.txt | +| student-tests | lab1/lab2 API tests via the balancer | PASS | 3 | evidence/student-tests.txt | +| race | lab2 races via the balancer | PASS | 1 | evidence/race.txt | +| db | PostgreSQL: rollback, FK, Argon2id, unique keys, migrations | PASS | 6 | evidence/db.txt | +| redis | Redis as the session store | PASS | 3 | evidence/redis.txt | +| explain | EXPLAIN of the history page (full plans: explain-history.txt) | PASS | 1 | evidence/explain.txt | +| failover | public failover test for every replica, file downloaded via the other one (make test-failover) | PASS | 43 | evidence/failover.txt | +| load | public load generator, 30 s, concurrency 4 | PASS | 30 | evidence/load.txt | +| failures | Redis and PostgreSQL outages via the balancer | PASS | 56 | evidence/failures.txt | +| persistence | public persistence test incl. S3 + proof of real restarts (make test-persistence) | PASS | 48 | evidence/persistence.txt | # Замеры -Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-21, commit 2dbed80 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB. +Машина: Windows 10, AMD64 Family 26 Model 68 Stepping 0, AuthenticAMD; Docker VM: 16 CPU, 15.2 GiB RAM; клиент Python 3.11.1; дата прогона 2026-10-22, commit c942c61 + uncommitted changes. Лимиты контейнеров: api 1 CPU / 512 MiB каждая, postgres и s3 (SeaweedFS) по 1 CPU / 512 MiB, redis и lb по 0,5 CPU / 128 MiB, janitor 0,25 CPU / 192 MiB. Публичный тест отказа (`make test-failover`, 15 с записи во время остановки реплики): | Прогон | Остановлена реплика | Реплик до отказа | Успешных записей | Неудачных попыток (повторены с тем же ключом) | Первая запись после hook, с | | --- | --- | --- | --- | --- | --- | -| 1 | 11a93acecaca | 2 | 126 | 0 | 4.39 | -| 2 | 6128eb268c44 | 2 | 124 | 0 | 4.328 | +| 1 | 343119df2488 | 2 | 123 | 0 | 4.422 | +| 2 | d590adc21943 | 2 | 124 | 0 | 4.531 | Одновременные загрузки (`student-tests/upload_memory.py`, перед каждым сценарием реплики перезапускаются, лимит api 512 MiB / 1 CPU). Пик RSS — VmHWM процесса API; cgroup — память контейнера, включая page cache временных файлов: | Сценарий | Одновременно | Объём, MiB | Время, с (p50 / max) | Реплика: загрузок | RSS до, MiB | Пик RSS, MiB | Рост, MiB | Пик cgroup, MiB | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 1.38 / 1.52 | 11a93acecaca: 5 | 93.5 | 106.2 | 12.6 | 134.1 | -| | | | | 6128eb268c44: 5 | 94.8 | 105.3 | 10.5 | 135.2 | -| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 11a93acecaca: 5 | 93.2 | 96.6 | 3.4 | 84.3 | -| | | | | 6128eb268c44: 5 | 93.1 | 94.7 | 1.6 | 86.8 | -| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.37 / 1.77 | 11a93acecaca: 10 | 94.8 | 121.7 | 26.9 | 181.9 | -| | | | | 6128eb268c44: 10 | 95.0 | 116.1 | 21.0 | 155.6 | +| 10 x ~9.5 MiB PNG (noise) | 10 | 94.9 | 2.87 / 3.52 | 343119df2488: 5 | 94.8 | 107.6 | 12.8 | 136.0 | +| | | | | d590adc21943: 5 | 91.2 | 104.5 | 13.3 | 136.2 | +| 10 x 40 Mpx PNG (8000x5000 RGB) | 10 | 1.1 | 0.85 / 1.09 | 343119df2488: 5 | 95.1 | 96.6 | 1.5 | 86.5 | +| | | | | d590adc21943: 5 | 92.1 | 96.6 | 4.6 | 83.5 | +| 20 mixed: large + 40 Mpx + JPEG | 20 | 133.2 | 1.5 / 1.78 | 343119df2488: 10 | 93.2 | 116.2 | 23.0 | 158.2 | +| | | | | d590adc21943: 10 | 95.0 | 118.6 | 23.6 | 167.8 | Один PNG 8000x5000 RGB внутри контейнера api, дополнительный пик RSS: наивно (тело целиком в памяти + полное декодирование Pillow) 161 MiB, потоковая проверка 0 MiB. 10 таких наивных загрузок одновременно не поместились бы в 512 MiB. -Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 245.844 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс. +Публичный генератор нагрузки (`tests/load.py`, closed loop, concurrency 4, 30 с, 2 реплики): 239.325 успешных POST/с, ошибок 0 (доля 0.0), задержка успешных p50 16.0 мс, p95 32.0 мс, p99 32.0 мс. | Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных | | --- | --- | --- | --- | -| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.39, 4.328 с (hook ждёт остановки ~4 с); потерь нет | -| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 39.2 с до готовности; потерь нет | -| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 20141 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет | -| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14766 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет | +| публичный `make test-failover` для каждой реплики | запись продолжается не позже 10 с, подтверждённые сообщения на месте, дублей нет | PASS 2 раз(а), неудачных попыток: 0, 0 | первая запись после hook через 4.422, 4.531 с (hook ждёт остановки ~4 с); потерь нет | +| перезапуск всего стенда (`restart` hook: postgres, redis, s3, api, janitor, lb) | данные, файл и прежний токен живы | PASS, те же контейнеры с новым временем старта | 47.0 с до готовности; потерь нет | +| `docker compose stop s3` (SeaweedFS остановлен) | загрузка и скачивание -> 503, метаданные, сообщения и /health/ready работают | upload 503 за 19453 мс, download 503, остальное 200/201 | после start файлы на месте; uploading-строку убрал janitor; потерь нет | +| `docker compose pause s3` (SeaweedFS завис) | то же, но через таймауты S3-клиента | upload 503 за 14891 мс (2 попытки x 5 с + откат 2 с) | после unpause файлы на месте; потерь нет | | падение между PUT в S3 и COMMIT в БД (код API в контейнере, finish_upload падает) | вложение не видно, janitor не трогает его до конца grace period, потом удаляет объект и строку | GET/прикрепление -> 404; janitor: свежая строка сохранена, dry-run только отчёт, затем удалено | мусора не осталось; повторный проход ничего не делает | | обрыв загрузки клиентом (9 МБ заявлено, отправлено 0,2 и 3 МБ) | ничего не сохраняется | строк и объектов не прибавилось, временных файлов 0 | мгновенно | diff --git a/evidence/upload-memory.json b/evidence/upload-memory.json index 3050178..c9eed9e 100644 --- a/evidence/upload-memory.json +++ b/evidence/upload-memory.json @@ -1,5 +1,5 @@ { - "date": "2026-10-21 21:08", + "date": "2026-10-22 14:22", "api_limit_mib": 512, "api_limit_cpus": 1.0, "runs": [ @@ -11,32 +11,32 @@ ], "total_mib": 94.9, "all_201": true, - "wall_seconds": 1.52, + "wall_seconds": 3.52, "latency_s": { - "p50": 1.38, - "max": 1.52 + "p50": 2.87, + "max": 3.52 }, - "sampler_rounds": 9, + "sampler_rounds": 18, "replicas": [ { - "replica": "11a93acecaca", - "uploads_served": 5, - "rss_before_mib": 93.5, - "rss_peak_mib": 106.2, - "rss_growth_mib": 12.6, - "rss_peak_sampled_mib": 106.2, - "cgroup_peak_mib": 134.1, - "cgroup_peak_sampled_mib": 127.7 - }, - { - "replica": "6128eb268c44", + "replica": "343119df2488", "uploads_served": 5, "rss_before_mib": 94.8, - "rss_peak_mib": 105.3, - "rss_growth_mib": 10.5, - "rss_peak_sampled_mib": 105.3, - "cgroup_peak_mib": 135.2, - "cgroup_peak_sampled_mib": 131.4 + "rss_peak_mib": 107.6, + "rss_growth_mib": 12.8, + "rss_peak_sampled_mib": 107.6, + "cgroup_peak_mib": 136.0, + "cgroup_peak_sampled_mib": 134.5 + }, + { + "replica": "d590adc21943", + "uploads_served": 5, + "rss_before_mib": 91.2, + "rss_peak_mib": 104.5, + "rss_growth_mib": 13.3, + "rss_peak_sampled_mib": 104.5, + "cgroup_peak_mib": 136.2, + "cgroup_peak_sampled_mib": 133.3 } ] }, @@ -53,27 +53,27 @@ "p50": 0.85, "max": 1.09 }, - "sampler_rounds": 8, + "sampler_rounds": 12, "replicas": [ { - "replica": "11a93acecaca", + "replica": "343119df2488", "uploads_served": 5, - "rss_before_mib": 93.2, + "rss_before_mib": 95.1, "rss_peak_mib": 96.6, - "rss_growth_mib": 3.4, + "rss_growth_mib": 1.5, "rss_peak_sampled_mib": 96.6, - "cgroup_peak_mib": 84.3, - "cgroup_peak_sampled_mib": 77.3 + "cgroup_peak_mib": 86.5, + "cgroup_peak_sampled_mib": 77.6 }, { - "replica": "6128eb268c44", + "replica": "d590adc21943", "uploads_served": 5, - "rss_before_mib": 93.1, - "rss_peak_mib": 94.7, - "rss_growth_mib": 1.6, - "rss_peak_sampled_mib": 94.6, - "cgroup_peak_mib": 86.8, - "cgroup_peak_sampled_mib": 77.3 + "rss_before_mib": 92.1, + "rss_peak_mib": 96.6, + "rss_growth_mib": 4.6, + "rss_peak_sampled_mib": 96.6, + "cgroup_peak_mib": 83.5, + "cgroup_peak_sampled_mib": 77.8 } ] }, @@ -87,32 +87,32 @@ ], "total_mib": 133.2, "all_201": true, - "wall_seconds": 1.77, + "wall_seconds": 1.78, "latency_s": { - "p50": 1.37, - "max": 1.77 + "p50": 1.5, + "max": 1.78 }, - "sampler_rounds": 10, + "sampler_rounds": 12, "replicas": [ { - "replica": "11a93acecaca", + "replica": "343119df2488", "uploads_served": 10, - "rss_before_mib": 94.8, - "rss_peak_mib": 121.7, - "rss_growth_mib": 26.9, - "rss_peak_sampled_mib": 121.7, - "cgroup_peak_mib": 181.9, - "cgroup_peak_sampled_mib": 180.6 + "rss_before_mib": 93.2, + "rss_peak_mib": 116.2, + "rss_growth_mib": 23.0, + "rss_peak_sampled_mib": 116.2, + "cgroup_peak_mib": 158.2, + "cgroup_peak_sampled_mib": 146.7 }, { - "replica": "6128eb268c44", + "replica": "d590adc21943", "uploads_served": 10, "rss_before_mib": 95.0, - "rss_peak_mib": 116.1, - "rss_growth_mib": 21.0, - "rss_peak_sampled_mib": 116.1, - "cgroup_peak_mib": 155.6, - "cgroup_peak_sampled_mib": 155.6 + "rss_peak_mib": 118.6, + "rss_growth_mib": 23.6, + "rss_peak_sampled_mib": 118.6, + "cgroup_peak_mib": 167.8, + "cgroup_peak_sampled_mib": 167.7 } ] } diff --git a/evidence/upload-memory.txt b/evidence/upload-memory.txt index cdcb73f..8a71056 100644 --- a/evidence/upload-memory.txt +++ b/evidence/upload-memory.txt @@ -1,13 +1,13 @@ # upload-memory: lab4: >=10 concurrent uploads, peak memory of api replicas -# date: 2026-10-21 21:08:48, commit: 2dbed80 + uncommitted changes +# date: 2026-10-22 14:22:25, commit: c942c61 + uncommitted changes # command: python.exe student-tests/upload_memory.py --output evidence/upload-memory.json -# exit code: 0, 34.2 s +# exit code: 0, 37.0 s client: Python 3.11.1 on Windows 10 preparing synthetic images (random-noise PNG ~9.5 MiB each, 40 Mpx PNG)... large: 9953047 bytes; 40 Mpx: 116714 bytes -{"scenario": "10 x ~9.5 MiB PNG (noise)", "concurrency": 10, "file_bytes": [9953047], "total_mib": 94.9, "all_201": true, "wall_seconds": 1.52, "latency_s": {"p50": 1.38, "max": 1.52}, "sampler_rounds": 9, "replicas": [{"replica": "11a93acecaca", "uploads_served": 5, "rss_before_mib": 93.5, "rss_peak_mib": 106.2, "rss_growth_mib": 12.6, "rss_peak_sampled_mib": 106.2, "cgroup_peak_mib": 134.1, "cgroup_peak_sampled_mib": 127.7}, {"replica": "6128eb268c44", "uploads_served": 5, "rss_before_mib": 94.8, "rss_peak_mib": 105.3, "rss_growth_mib": 10.5, "rss_peak_sampled_mib": 105.3, "cgroup_peak_mib": 135.2, "cgroup_peak_sampled_mib": 131.4}]} -{"scenario": "10 x 40 Mpx PNG (8000x5000 RGB)", "concurrency": 10, "file_bytes": [116714], "total_mib": 1.1, "all_201": true, "wall_seconds": 1.09, "latency_s": {"p50": 0.85, "max": 1.09}, "sampler_rounds": 8, "replicas": [{"replica": "11a93acecaca", "uploads_served": 5, "rss_before_mib": 93.2, "rss_peak_mib": 96.6, "rss_growth_mib": 3.4, "rss_peak_sampled_mib": 96.6, "cgroup_peak_mib": 84.3, "cgroup_peak_sampled_mib": 77.3}, {"replica": "6128eb268c44", "uploads_served": 5, "rss_before_mib": 93.1, "rss_peak_mib": 94.7, "rss_growth_mib": 1.6, "rss_peak_sampled_mib": 94.6, "cgroup_peak_mib": 86.8, "cgroup_peak_sampled_mib": 77.3}]} -{"scenario": "20 mixed: large + 40 Mpx + JPEG", "concurrency": 20, "file_bytes": [1492, 116714, 9953047], "total_mib": 133.2, "all_201": true, "wall_seconds": 1.77, "latency_s": {"p50": 1.37, "max": 1.77}, "sampler_rounds": 10, "replicas": [{"replica": "11a93acecaca", "uploads_served": 10, "rss_before_mib": 94.8, "rss_peak_mib": 121.7, "rss_growth_mib": 26.9, "rss_peak_sampled_mib": 121.7, "cgroup_peak_mib": 181.9, "cgroup_peak_sampled_mib": 180.6}, {"replica": "6128eb268c44", "uploads_served": 10, "rss_before_mib": 95.0, "rss_peak_mib": 116.1, "rss_growth_mib": 21.0, "rss_peak_sampled_mib": 116.1, "cgroup_peak_mib": 155.6, "cgroup_peak_sampled_mib": 155.6}]} +{"scenario": "10 x ~9.5 MiB PNG (noise)", "concurrency": 10, "file_bytes": [9953047], "total_mib": 94.9, "all_201": true, "wall_seconds": 3.52, "latency_s": {"p50": 2.87, "max": 3.52}, "sampler_rounds": 18, "replicas": [{"replica": "343119df2488", "uploads_served": 5, "rss_before_mib": 94.8, "rss_peak_mib": 107.6, "rss_growth_mib": 12.8, "rss_peak_sampled_mib": 107.6, "cgroup_peak_mib": 136.0, "cgroup_peak_sampled_mib": 134.5}, {"replica": "d590adc21943", "uploads_served": 5, "rss_before_mib": 91.2, "rss_peak_mib": 104.5, "rss_growth_mib": 13.3, "rss_peak_sampled_mib": 104.5, "cgroup_peak_mib": 136.2, "cgroup_peak_sampled_mib": 133.3}]} +{"scenario": "10 x 40 Mpx PNG (8000x5000 RGB)", "concurrency": 10, "file_bytes": [116714], "total_mib": 1.1, "all_201": true, "wall_seconds": 1.09, "latency_s": {"p50": 0.85, "max": 1.09}, "sampler_rounds": 12, "replicas": [{"replica": "343119df2488", "uploads_served": 5, "rss_before_mib": 95.1, "rss_peak_mib": 96.6, "rss_growth_mib": 1.5, "rss_peak_sampled_mib": 96.6, "cgroup_peak_mib": 86.5, "cgroup_peak_sampled_mib": 77.6}, {"replica": "d590adc21943", "uploads_served": 5, "rss_before_mib": 92.1, "rss_peak_mib": 96.6, "rss_growth_mib": 4.6, "rss_peak_sampled_mib": 96.6, "cgroup_peak_mib": 83.5, "cgroup_peak_sampled_mib": 77.8}]} +{"scenario": "20 mixed: large + 40 Mpx + JPEG", "concurrency": 20, "file_bytes": [1492, 116714, 9953047], "total_mib": 133.2, "all_201": true, "wall_seconds": 1.78, "latency_s": {"p50": 1.5, "max": 1.78}, "sampler_rounds": 12, "replicas": [{"replica": "343119df2488", "uploads_served": 10, "rss_before_mib": 93.2, "rss_peak_mib": 116.2, "rss_growth_mib": 23.0, "rss_peak_sampled_mib": 116.2, "cgroup_peak_mib": 158.2, "cgroup_peak_sampled_mib": 146.7}, {"replica": "d590adc21943", "uploads_served": 10, "rss_before_mib": 95.0, "rss_peak_mib": 118.6, "rss_growth_mib": 23.6, "rss_peak_sampled_mib": 118.6, "cgroup_peak_mib": 167.8, "cgroup_peak_sampled_mib": 167.7}]} one 8000x5000 RGB PNG inside the api container, extra peak RSS: naive (body in RAM + full decode) 161 MiB, streaming check 0 MiB -PASS: all uploads succeeded; largest RSS growth of one replica: 26.9 MiB (limit 512 MiB); results in evidence/upload-memory.json +PASS: all uploads succeeded; largest RSS growth of one replica: 23.6 MiB (limit 512 MiB); results in evidence/upload-memory.json diff --git a/student-tests/images.py b/student-tests/images.py index a83b5de..e49bfc6 100644 --- a/student-tests/images.py +++ b/student-tests/images.py @@ -22,7 +22,7 @@ def chunk(kind: bytes, data: bytes) -> bytes: def png(width, height, *, gray=False, noise=False, interlace=0, pad_to=None, idat=None) -> bytes: """PNG width x height. noise=True — случайные пиксели (почти не сжимаются, файл большой). - pad_to — дополнить файл служебным чанком prVt до точного размера в байтах. + pad_to — дополнить файл текстовым чанком tEXt после IDAT до точного размера в байтах. idat — готовые сжатые данные (для «бомбы»: заголовок обещает одно, данных другое). """ channels = 1 if gray else 3 @@ -43,9 +43,9 @@ def png(width, height, *, gray=False, noise=False, interlace=0, pad_to=None, ida idat = b"".join(parts) body = b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header) + chunk(b"IDAT", idat) if pad_to is not None: - need = pad_to - len(body) - 12 - 12 # чанк-заполнитель + IEND + need = pad_to - len(body) - 12 - 12 - 8 # заголовок чанка-заполнителя, IEND, ключ "Comment\0" assert need >= 0, "image is already larger than pad_to" - body += chunk(b"prVt", bytes(need)) + body += chunk(b"tEXt", b"Comment\x00" + b"a" * need) return body + chunk(b"IEND", b"") @@ -72,3 +72,22 @@ def flip_byte(data: bytes, position: int) -> bytes: GIF = b"GIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\xff\xff\xff!\xf9\x04\x01\x00\x00\x00\x00,\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02D\x01\x00;" + + +# --- сборка PNG из произвольных чанков (для проверок смысла чанков при верных CRC) --- + +def ihdr(width, height, depth=8, color=2, interlace=0): + return b"IHDR", struct.pack(">IIBBBBB", width, height, depth, color, 0, 0, interlace) + + +def zero_idat(width, height, depth=8, channels=3): + """Сжатые пиксели: каждая строка = байт фильтра 0 + нули (индекс палитры 0 / чёрный).""" + row_bytes = (width * depth * channels + 7) // 8 + return b"IDAT", zlib.compress((b"\x00" + bytes(row_bytes)) * height) + + +def assemble(*chunks): + """PNG из чанков (kind, data) с ВЕРНЫМИ CRC; IEND добавляется, если не передан.""" + if not chunks or chunks[-1][0] != b"IEND": + chunks = chunks + ((b"IEND", b""),) + return b"\x89PNG\r\n\x1a\n" + b"".join(chunk(kind, data) for kind, data in chunks) diff --git a/student-tests/test_attachments.py b/student-tests/test_attachments.py index e1178c1..4f88f7d 100644 --- a/student-tests/test_attachments.py +++ b/student-tests/test_attachments.py @@ -6,6 +6,7 @@ import hashlib import http.client import socket +import struct import urllib.parse import uuid import zlib @@ -135,6 +136,73 @@ def test_damaged_images(): print("PASS: damaged and foreign files are rejected with 415, never 500") +def test_png_chunk_semantics(): + """CRC у всех чанков верные, но содержимое нарушает спецификацию PNG -> 415, а не 201. + + Регрессия: PNG 1x1, глубина 8, палитровый, PLTE из 771 байта (257 записей) раньше получал 201 ready, + хотя Pillow его отклоняет (invalid palette size). + """ + alice, bob, _, chat = setup() + from images import assemble, ihdr, zero_idat + pal = lambda entries: (b"PLTE", bytes(3 * entries)) + idat_p = zero_idat(1, 1, 8, 1) + bad = [ + ("PLTE 771 bytes = 257 entries (review case)", + assemble(ihdr(1, 1, 8, 3), (b"PLTE", bytes(771)), (b"IDAT", zlib.compress(b"\0\0")))), + ("PLTE length not a multiple of 3", assemble(ihdr(1, 1, 8, 3), (b"PLTE", bytes(4)), idat_p)), + ("empty PLTE", assemble(ihdr(1, 1, 8, 3), (b"PLTE", b""), idat_p)), + ("palette larger than bit depth allows (depth 1, 3 entries)", + assemble(ihdr(1, 1, 1, 3), pal(3), zero_idat(1, 1, 1, 1))), + ("palette image without PLTE", assemble(ihdr(1, 1, 8, 3), idat_p)), + ("PLTE in a grayscale image", assemble(ihdr(1, 1, 8, 0), pal(2), zero_idat(1, 1, 8, 1))), + ("PLTE after IDAT", assemble(ihdr(1, 1, 8, 3), idat_p, pal(2))), + ("two PLTE chunks", assemble(ihdr(1, 1, 8, 3), pal(2), pal(2), idat_p)), + ("tRNS longer than the palette", assemble(ihdr(1, 1, 8, 3), pal(2), (b"tRNS", bytes(3)), idat_p)), + ("tRNS before PLTE", assemble(ihdr(1, 1, 8, 3), (b"tRNS", bytes(1)), pal(2), idat_p)), + ("tRNS of wrong size for RGB (4 bytes instead of 6)", + assemble(ihdr(1, 1, 8, 2), (b"tRNS", bytes(4)), zero_idat(1, 1, 8, 3))), + ("tRNS in an image with alpha channel", + assemble(ihdr(1, 1, 8, 6), (b"tRNS", bytes(2)), zero_idat(1, 1, 8, 4))), + ("IEND with data", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"IEND", b"x"))), + ("unknown critical chunk", assemble(ihdr(1, 1, 8, 2), (b"ABCD", b""), zero_idat(1, 1, 8, 3))), + ("IDAT chunks not consecutive", + assemble(ihdr(1, 2, 8, 2), (b"IDAT", zero_idat(1, 2, 8, 3)[1][:5]), (b"tEXt", b"k\0v"), + (b"IDAT", zero_idat(1, 2, 8, 3)[1][5:]))), + ("gAMA after IDAT", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"gAMA", bytes(4)))), + ("gAMA of 2 bytes instead of 4", assemble(ihdr(1, 1, 8, 2), (b"gAMA", bytes(2)), zero_idat(1, 1, 8, 3))), + ("3 MiB metadata chunk before IDAT", assemble(ihdr(1, 1, 8, 2), (b"prVt", bytes(3 * 1024 * 1024)), + zero_idat(1, 1, 8, 3))), + ("private chunk after IDAT", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"prVt", bytes(100)))), + ("animated PNG frame (fdAT)", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"fdAT", bytes(10)))), + ("tIME of 2 bytes instead of 7", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"tIME", bytes(2)))), + ("tEXt without keyword separator", assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"tEXt", b"nokey"))), + ("zTXt with broken zlib after IDAT", + assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"zTXt", b"k\0\0garbage"))), + ("zTXt text bomb: 50 MB of text in ~50 KB after IDAT", + assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"zTXt", b"k\0\0" + zlib.compress(bytes(50_000_000))))), + ] + for name, data in bad: + check_error(upload(alice, chat["id"], data, filename="bad.png", expected=415), "unsupported_media_type") + print(f"{name} ({len(data)} bytes, valid CRCs) -> 415") + good = [ + ("palette 256 entries + tRNS + tEXt and tIME after IDAT", + assemble(ihdr(2, 2, 8, 3), pal(256), (b"tRNS", bytes(256)), zero_idat(2, 2, 8, 1), + (b"tEXt", b"Comment\0ok"), (b"tIME", struct.pack(">HBBBBB", 2026, 10, 22, 12, 0, 0)))), + ("palette depth 2 with 4 entries", assemble(ihdr(3, 1, 2, 3), pal(4), zero_idat(3, 1, 2, 1))), + ("RGB with 6-byte tRNS and gAMA", assemble(ihdr(1, 1, 8, 2), (b"gAMA", struct.pack(">I", 45455)), + (b"tRNS", bytes(6)), zero_idat(1, 1, 8, 3))), + ("compressed zTXt (100 KB of text) and iTXt after IDAT", + assemble(ihdr(1, 1, 8, 2), zero_idat(1, 1, 8, 3), (b"zTXt", b"Comment\0\0" + zlib.compress(b"a" * 100_000)), + (b"iTXt", b"Title\0\1\0ru\0\0" + zlib.compress("Привет".encode())))), + ] + for name, data in good: + att = upload(alice, chat["id"], data, filename="good.png").json + got = call("GET", f"/api/v1/attachments/{att['id']}/content", 200, token=bob["token"]) + check(got.data == data, "valid PNG changed") + print(f"{name} -> 201, downloaded byte for byte") + print("PASS: chunk contents are checked, not only CRCs; valid PNGs with metadata are still accepted") + + def test_mime_substitution(): alice, _, _, chat = setup() cases = [ @@ -264,5 +332,5 @@ def test_aborted_upload(): if __name__ == "__main__": run_tests([test_png_and_jpeg_roundtrip, test_size_limit_exact_boundary, test_size_limit_streaming, - test_pixel_limit, test_damaged_images, test_mime_substitution, test_filenames, test_access_control, + test_pixel_limit, test_damaged_images, test_png_chunk_semantics, test_mime_substitution, test_filenames, test_access_control, test_message_attachments, test_idempotency_with_attachments, test_aborted_upload]) diff --git a/uploads.py b/uploads.py index 2452212..c01bfc1 100644 --- a/uploads.py +++ b/uploads.py @@ -183,7 +183,12 @@ def inspect(upload: Upload, declared: str) -> ImageInfo: upload.file.seek(0) try: with _decode_slots: - width, height = check_png(upload.file) if detected == PNG else check_jpeg(upload.file) + if detected == PNG: + width, height = check_png(upload.file) + upload.file.seek(0) + check_png_metadata(upload.file) + else: + width, height = check_jpeg(upload.file) except TooManyPixels: raise payload_too_large(f"Image has more than {settings.MAX_IMAGE_PIXELS} pixels") from None except Damaged as exc: @@ -262,13 +267,40 @@ class PngRows: return self.total == self.expected +# Чанки, содержимое которых проверяется (держим в памяти; размер ограничен спецификацией): +# IHDR — 13 байт, PLTE — до 256 x 3 = 768, tRNS — до 256 (палитра) / 6 (RGB). +SMALL_CHUNK_LIMIT = {b"IHDR": 13, b"PLTE": 768, b"tRNS": 256} +CRITICAL = (b"IHDR", b"PLTE", b"IDAT", b"IEND") +# После данных изображения допустимы только время и текст (строгий список). +AFTER_IDAT = {b"IEND", b"tIME", b"tEXt", b"zTXt", b"iTXt"} +TEXT_CHUNKS = (b"tEXt", b"zTXt", b"iTXt") +# Анимированные PNG (APNG) не поддерживаются: кадры — отдельная сущность, превью строится по одному кадру. +APNG_CHUNKS = (b"acTL", b"fcTL", b"fdAT") +# Как PngImagePlugin.MAX_TEXT_CHUNK в Pillow: больше 1 MiB распакованного текста в одном чанке — отказ. +TEXT_LIMIT = 1024 * 1024 +# Чанки метаданных до IDAT Pillow (Image.open) читает целиком: ограничиваем, чтобы память не росла. +# ICC-профили и EXIF реальных картинок — единицы-сотни KiB. +PRE_IDAT_CHUNK_LIMIT = 2 * 1024 * 1024 +# Чанки, которые по спецификации обязаны стоять ДО первого IDAT (PNG 3rd ed., 5.6 «Chunk ordering»). +# После IDAT допустимы только tIME, tEXt, zTXt, iTXt, кадры APNG и частные чанки. +BEFORE_IDAT = {b"PLTE", b"tRNS", b"cHRM", b"cICP", b"gAMA", b"iCCP", b"mDCV", b"cLLI", b"sBIT", b"sRGB", + b"bKGD", b"hIST", b"pHYs", b"sPLT", b"eXIf"} + + def check_png(file: BinaryIO) -> tuple[int, int]: + """Структура PNG по спецификации (PNG 3rd ed., гл. 5 и 11) без декодирования пикселей в память. + + Кроме CRC каждого чанка проверяется СМЫСЛ критичных чанков и tRNS: CRC защищает только от + случайной порчи, а намеренно собранный файл с верными CRC может быть некорректным. + """ if file.read(8) != PNG_SIGNATURE: raise Damaged("signature") rows = inflater = None - width = height = 0 - seen_idat = idat_finished = have_plte = False + width = height = depth = 0 color = None + palette_entries = 0 + seen = set() # какие чанки уже встречались + idat_started = idat_finished = False first = True while True: header = file.read(8) @@ -277,11 +309,37 @@ def check_png(file: BinaryIO) -> tuple[int, int]: length, kind = struct.unpack(">I4s", header) if length > 0x7FFFFFFF: raise Damaged("chunk length") + if not kind.isalpha() or not kind.isascii(): + raise Damaged("chunk type") if first and kind != b"IHDR": raise Damaged("IHDR must be first") + if kind[:1].isupper() and kind not in CRITICAL: + raise Damaged(f"unknown critical chunk {kind!r}") # такой чанк декодер обязан отвергнуть + if kind in SMALL_CHUNK_LIMIT and length > SMALL_CHUNK_LIMIT[kind]: + raise Damaged(f"{kind!r} is too long") + if kind == b"IDAT": + if rows is None or idat_finished: + raise Damaged("IDAT chunks must be consecutive") + idat_started = True + elif idat_started: + idat_finished = True + if kind in APNG_CHUNKS: + raise Damaged("animated PNG is not supported") + if kind in BEFORE_IDAT and idat_started: + raise Damaged(f"{kind!r} after IDAT") + if idat_finished and kind not in AFTER_IDAT: + raise Damaged(f"{kind!r} after IDAT") + if not idat_started and kind != b"IDAT" and length > PRE_IDAT_CHUNK_LIMIT: + raise Damaged(f"{kind!r} metadata chunk is too large") + if kind == b"tIME" and length != 7: + raise Damaged("tIME size") + if kind in (b"IHDR", b"PLTE", b"tRNS", b"IEND") and kind in seen: + raise Damaged(f"duplicate {kind!r}") + seen.add(kind) + crc = zlib.crc32(kind) - keep = kind in (b"IHDR",) kept = b"" + text = TextChunk(kind) if kind in TEXT_CHUNKS else None left = length while left: piece = file.read(min(CRC_CHUNK, left)) @@ -289,18 +347,20 @@ def check_png(file: BinaryIO) -> tuple[int, int]: raise Damaged("truncated chunk") left -= len(piece) crc = zlib.crc32(piece, crc) - if keep: + if kind in SMALL_CHUNK_LIMIT: kept += piece elif kind == b"IDAT": - if rows is None or idat_finished: - raise Damaged("IDAT out of order") feed_inflate(inflater, rows, piece) + elif text is not None: + text.feed(piece) + if text is not None: + text.finish() stored = file.read(4) if len(stored) < 4 or struct.unpack(">I", stored)[0] != crc & 0xFFFFFFFF: raise Damaged(f"CRC mismatch in {kind!r}") if kind == b"IHDR": - if not first or length != 13: + if length != 13: raise Damaged("IHDR") width, height, depth, color, compression, filter_method, interlace = struct.unpack(">IIBBBBB", kept) if width > 0x7FFFFFFF or height > 0x7FFFFFFF: @@ -313,20 +373,34 @@ def check_png(file: BinaryIO) -> tuple[int, int]: rows = PngRows(width, height, depth, color, interlace) inflater = zlib.decompressobj() elif kind == b"PLTE": - have_plte = True - elif kind == b"IDAT": - seen_idat = True + # 1-256 записей по 3 байта; для палитровых — не больше 2^глубина; у серых палитры быть не может. + if color in (0, 4): + raise Damaged("PLTE in a grayscale image") + if length == 0 or length % 3: + raise Damaged("palette size") + palette_entries = length // 3 + if color == 3 and palette_entries > 1 << depth: + raise Damaged("palette larger than the bit depth allows") + elif kind == b"tRNS": + # Прозрачность: серое — 2 байта, RGB — 6, палитра — не больше записей PLTE; при альфа-канале запрещён. + if color in (4, 6): + raise Damaged("tRNS with an alpha channel") + if color == 3 and not palette_entries: + raise Damaged("tRNS before PLTE") + expected_ok = {0: length == 2, 2: length == 6, 3: 1 <= length <= palette_entries}[color] + if not expected_ok: + raise Damaged("tRNS size") elif kind == b"IEND": + if length != 0: + raise Damaged("IEND with data") break - if seen_idat and kind != b"IDAT": - idat_finished = True first = False if file.read(1): raise Damaged("data after IEND") - if color == 3 and not have_plte: + if color == 3 and not palette_entries: raise Damaged("palette image without PLTE") - if not seen_idat: + if not idat_started: raise Damaged("no IDAT") tail = inflater.flush(INFLATE_CHUNK) if tail: @@ -336,6 +410,100 @@ def check_png(file: BinaryIO) -> tuple[int, int]: return width, height +def check_png_metadata(file: BinaryIO) -> None: + """Второе мнение: стандартный парсер Pillow разбирает метаданные до IDAT (gAMA, iCCP, pHYs, tEXt…), + НЕ читая пиксели. Только open, без verify(): verify читает каждый чанк, включая IDAT до 10 MiB, + целиком в память, а CRC и данные после IDAT уже потоково проверил check_png. + Нужен для вспомогательных чанков: по спецификации их можно игнорировать, но если Pillow + не может их разобрать, файл нельзя будет обработать (превью в lab5) — лучше отклонить сразу. + """ + with Image.open(file, formats=["PNG"]): + pass # open разбирает все чанки до первого IDAT; после IDAT check_png пускает только время и текст + + +class TextChunk: + """Потоковая проверка tEXt / zTXt / iTXt: ключ 1-79 байт, метод сжатия 0, сжатый текст — + корректный zlib не больше TEXT_LIMIT после распаковки (распаковка кусками, текст не хранится).""" + + def __init__(self, kind: bytes): + self.kind = kind + self.head = b"" + self.done_header = False + self.inflater = None + self.size = 0 + + def feed(self, piece: bytes) -> None: + if not self.done_header: + self.head += piece + rest = self._header() + if rest is None: + if len(self.head) > 4096: + raise Damaged("text chunk header") + return + piece = rest + if self.inflater is not None and piece: + self._inflate(piece) + + def _keyword_end(self) -> int | None: + end = self.head.find(b"\x00") + if end == -1: + return None + if not 1 <= end <= 79: + raise Damaged("text keyword") + return end + + def _header(self) -> bytes | None: + end = self._keyword_end() + if end is None: + return None + h = self.head + if self.kind == b"tEXt": + self.done_header = True + return b"" + if self.kind == b"zTXt": + if len(h) < end + 2: + return None + if h[end + 1] != 0: + raise Damaged("zTXt compression method") + self.done_header, self.inflater = True, zlib.decompressobj() + return h[end + 2:] + # iTXt: keyword \0 flag method language \0 translated-keyword \0 text + if len(h) < end + 3: + return None + flag, method = h[end + 1], h[end + 2] + if flag not in (0, 1) or (flag and method != 0): + raise Damaged("iTXt compression") + lang_end = h.find(b"\x00", end + 3) + tkey_end = h.find(b"\x00", lang_end + 1) if lang_end != -1 else -1 + if tkey_end == -1: + return None + self.done_header = True + if flag: + self.inflater = zlib.decompressobj() + return h[tkey_end + 1:] + + def _inflate(self, data: bytes) -> None: + if self.inflater.eof: + raise Damaged("data after compressed text") + out = self.inflater.decompress(data, INFLATE_CHUNK) + self.size += len(out) + while self.inflater.unconsumed_tail and not self.inflater.eof and self.size <= TEXT_LIMIT: + out = self.inflater.decompress(self.inflater.unconsumed_tail, INFLATE_CHUNK) + self.size += len(out) + if self.size > TEXT_LIMIT: + raise Damaged("compressed text is too large") + if self.inflater.eof and self.inflater.unused_data: + raise Damaged("data after compressed text") + + def finish(self) -> None: + if not self.done_header: + raise Damaged("text chunk header") + if self.inflater is not None: + self.size += len(self.inflater.flush(INFLATE_CHUNK)) + if not self.inflater.eof or self.size > TEXT_LIMIT: + raise Damaged("compressed text") + + def feed_inflate(inflater, rows: PngRows, data: bytes) -> None: """Распаковка кусками не больше 64 KiB: «zip-бомба» не раздует память.""" if inflater.eof: