forked from mtusi2026/mtusi2026lab5
7.3 KiB
7.3 KiB
Сводка проверок
| Шаг | Что проверяет | Результат | Время, с | Файл |
|---|---|---|---|---|
| check | offline contract check (make check) | PASS | 0 | evidence/native-sandbox/check.txt |
| smoke | public smoke via the balancer (make test), with attachment | PASS | 1 | evidence/native-sandbox/smoke.txt |
| jobs-variants | lab5: thumbnail/crop/optimized, EXIF orientation, alpha, no upscale, metadata stripped | PASS | 11 | evidence/native-sandbox/jobs-variants.txt |
| jobs-limits | lab5: worker limits - bytes, pixels, memory (RLIMIT_AS), time (JOB_TIMEOUT_SECONDS) | PASS | 69 | evidence/native-sandbox/jobs-limits.txt |
| jobs-recovery | lab5: worker down -> 202 -> restart api+broker -> ready; broker down -> outbox waits | PASS | 46 | evidence/native-sandbox/jobs-recovery.txt |
| jobs-outbox | lab5: dispatcher crash between COMMIT and publish / after confirm | PASS | 8 | evidence/native-sandbox/jobs-outbox.txt |
| jobs-crash | lab5: worker crash after S3 PUT / after COMMIT before ack, staging cleanup, frozen worker lease | PASS | 143 | evidence/native-sandbox/jobs-crash.txt |
| jobs-retry | lab5: backoff+jitter, retry_exhausted, replay, permanent errors, DLQ | PASS | 49 | evidence/native-sandbox/jobs-retry.txt |
| jobs-concurrency | lab5: WORKER_CONCURRENCY consumers, prefetch 1, bounded processing | PASS | 7 | evidence/native-sandbox/jobs-concurrency.txt |
| attachments | lab4: files, limits, damaged images, MIME, filenames, access, attachment_ids | PASS | 11 | evidence/native-sandbox/attachments.txt |
| student-tests | lab1/lab2 API tests via the balancer | PASS | 3 | evidence/native-sandbox/student-tests.txt |
| race | lab2 races via the balancer | PASS | 2 | evidence/native-sandbox/race.txt |
| persistence | public persistence test (make test-persistence) on the native stand; docker-inspect proof needs Docker | PASS | 40 | evidence/native-sandbox/persistence.txt |
Где прогнано
Нативный стенд без Docker (в среде сборки нет доступа к Docker Hub): PostgreSQL 16, RabbitMQ 3.12.1, SeaweedFS 3.80, Redis — процессами с той же конфигурацией и секретами, что в compose.yaml; 1 реплика API на 8080 без HAProxy; Python 3.11. Команды docker compose ... тестов выполнялись через шим, переводящий их на эти процессы. Для защиты повторить на Docker: python student-tests/collect_evidence.py (вывод в evidence/).
Отказы lab5
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|---|---|---|---|
| worker остановлен, upload, рестарт rabbitmq и api с volumes, start worker | 202 queued, сообщение с вложением 201, content 409; после рестартов job в очереди; тот же job_id -> ready | worker started -> same job d62da8f8-b127-4141-95c3-8f691c39bb72 ready, thumbnail downloadable; staging object deleted | потерь нет; staging удалён после COMMIT |
| rabbitmq остановлен во время upload | 202 и /health/ready 200 без брокера; строка outbox ждёт; после start публикуется | rabbitmq started -> dispatcher published the pending row -> ready | потерь нет |
| crash dispatcher между COMMIT задания и publish | строка outbox не отмечена -> публикуется после рестарта процесса | dispatcher killed between COMMIT and publish; restarted by policy, outbox re-published -> job 21f39e5f-676b-43a2-92ca-532e1a932fb0 ready, attempt 1, no manual action | без ручного действия |
| crash dispatcher после confirm, до отметки outbox | дубль сообщения, worker распознаёт по состоянию job | dispatcher killed after publisher confirm, before marking outbox -> message published twice; worker: ['created', 'claimed', 'ready', 'duplicate_ignored'] -> one result set | один набор вариантов |
| crash worker после PUT в S3, до COMMIT и ack | повтор после истечения аренды перезаписывает те же ключи | worker killed after S3 PUT (before COMMIT and ack): redelivery saw a live lease (no-op), lease expired after 15 s, attempt 2 rewrote the same keys -> one set; events ['created', 'claimed', 'duplicate_ignored', 'lease_expired', 'claimed', 'ready'] | один набор, 4 объекта |
| crash worker после COMMIT, до ack | redelivery ready-задачи -> no-op ack | worker killed after COMMIT, before ack: broker redelivered, worker acked as no-op (1:duplicate_ignored:status=ready); queue empty (0 ready, 0 unacked) | очередь пуста, дублей нет |
| worker завис (не продлевает аренду) | dispatcher возвращает job, другой поток завершает; поздний результат отклонён fencing | frozen attempt 1 stopped renewing; lease expired, recovered by dispatcher, attempt 2 ready after 17 s; frozen attempt woke up -> stale_result_rejected; events ['created', 'claimed', 'lease_expired', 'claimed', 'ready', 'stale_result_rejected'] | ready не откатан |
| зависание обработки изображения | процесс убит по JOB_TIMEOUT_SECONDS, временная ошибка, аренда продлевалась | imaging hang -> killed after 60 s (processing_timeout, temporary), lease renewed meanwhile (no lease_expired), attempt 2 ready after 63 s | повтор -> ready |
| нехватка памяти в процессе обработки (RLIMIT_AS) | failed/resource_limit, worker жив | memory bomb in the imaging process -> failed/resource_limit after attempt 1; worker itself survived | постоянная ошибка, без повторов |
| 2 временные ошибки S3 | backoff с jitter, затем ready | 2 x storage_unavailable -> retries after 1.40 s and 2.40 s (equal jitter: 1..2 s, 2..4 s), attempt 3 ready | - |
| 5 временных ошибок подряд | failed/retry_exhausted, реестр failed, content 409 | 5 temporary errors -> failed/retry_exhausted (public error_code), content 409, attach 409, listed by ops.py failed (replayable) | replay оператора |
| replay failed и повторный replay ready | тот же job_id -> ready; повтор отклонён | ops replay -> same job_id 43d09c9b-f72f-4918-91d1-3407fbb12848 attempt 6 ready; second replay refused: "job is ready, replay is only for failed jobs" | дубля нет |
| невалидное сообщение и неизвестный job_id | reject -> DLQ, dlq-replay не исполняет | schema-invalid message and unknown job_id -> reject -> DLQ (3 message(s)); dlq-replay kept them: ['unknown job', 'invalid message is kept for analysis'] | сообщения сохранены |
| 8 заданий в очереди, старт worker | WORKER_CONCURRENCY=2, prefetch 1 | 2 consumers, prefetch_count 1, manual ack; 8 queued 3000x2000 JPEG drained in 2.9 s after worker start, peak processing 2 of 8 (16 samples) | - |