PR checklist review, 2026-10-26

make check: PASS (offline skeleton check; application not tested).
python student-tests/pipeline_critical.py: 8 tests, OK.
python student-tests/remaining_regressions.py: 7 tests, OK.
docker compose config --quiet: exit 0.
git diff 276dbda -- README.md contracts tests: empty.
git ls-files secrets .env: empty.
git check-ignore .env secrets/postgres_password: both ignored.

Secret scan scope:
- 90 Git-tracked working-tree files.
- 128 blobs reachable through git rev-list --objects --all.
- Compared against 10 locally configured secret values, read without printing.
- Searched private-key headers, AWS keys, GitHub/OpenAI token formats,
  JWT-shaped tokens, and literal PostgreSQL/AMQP/Redis credential URLs.
- No findings. Template substitutions in credential URLs were excluded.
- This is a scoped heuristic check, not proof that every possible secret format is absent.

Mock benchmark numbers printed by the regression suite are not load-test evidence.

Integration attempt:
- Started installed Docker Desktop; Docker Engine 29.8.0 became available.
- make validate PYTHON=.venv/bin/python: PASS, full OpenAPI 3.1 / JSON Schema.
- docker compose up -d --build: images built, exit 1 during startup.
- migrate: exited 0.
- s3-init: PASS: private S3 bucket ready; exited 0.
- broker (rabbitmq:4.1.2-alpine): exited 1.
  error: RABBITMQ_DEFAULT_PASS_FILE is set but deprecated
  error: deprecated environment variables detected
  Please use a configuration file instead
- API, worker, outbox and load_balancer remained Created, not running.
- Public smoke and failure experiments cannot run until broker starts.
- Volumes were preserved. PostgreSQL, S3 and reconcile were left running.

Follow-up after RabbitMQ startup fix, 2026-10-26:
- Removed deprecated RABBITMQ_DEFAULT_PASS_FILE from Compose.
- Read password from Docker secret in infra/rabbitmq-secret-entrypoint.sh;
  export inside container and exec stock entrypoint with rabbitmq-server.
- docker compose up -d: exit 0.
- docker compose ps -a: broker, api-a, api-b, worker, load_balancer,
  postgres and s3 healthy; outbox/reconcile running; migrate/s3-init exited 0.

Actual check output (exit 0):
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
.venv/bin/python tests/smoke.py --base-url "http://localhost:8080"
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.

.venv/bin/python student-tests/attachments.py (exit 0):
PASS: PNG/JPEG originals, metadata, attachment-only messages and ACL
PASS: filename, MIME, malformed image, byte/pixel and chunked limits
PASS: attachment scope, duplicates and idempotency normalization
PASS: inclusive 10 MiB and 40 million pixel boundaries

.venv/bin/python student-tests/pipeline_critical.py (exit 0):
Ran 8 tests in 0.012s
OK
.venv/bin/python student-tests/remaining_regressions.py (exit 0):
Ran 7 tests in 0.576s
OK

Mock benchmark output is excluded: it is not real load evidence.
Failure/restart experiments and real load profiles remain unverified.
Stack left running; no volumes deleted.

Follow-up verification, 2026-10-26/27:
- Real fault suite: evidence/pipeline-integration.json.
- Real SQL, Linux, ACL and cleanup locking checks: evidence/pipeline-invariants.json.
- Public persistence and both API failovers with before/after container states:
  evidence/resilience.json. Initial isolated run passed; overlapping exploratory
  restart/failover attempts are not counted as successful evidence.
- S3 permissions: PASS: API identity cannot create buckets; maintenance identity has separate admin access.
- Idempotency: PASS: concurrent idempotency through load balancer.
- After Docker was stopped, 10 queued jobs had published outbox records but
  RabbitMQ reported 0 ready and 0 unacknowledged messages. Original disappearance
  cause is not established. Added 60s unclaimed-job publication watchdog.
- Fixed redelivery bypassing retry delay / expired lease attempt accounting.
- Current-worktree secret scan: 107 tracked/nonignored files, 9 local single-line
  secret values >=12 bytes and private-key/GitHub/AWS patterns; no findings.
  This complements the earlier history scan; neither is a universal guarantee.
- Test cleanup deleted four explicitly selected orphan objects from its own
  crashed attempt. Active result objects and all volumes were preserved.

Final real load series, 2026-10-27:
- Six runs: workers/prefetch 1/1, 2/2, 4/4; PNG 512 and 2400 square.
- Each run: 160 uploads, client concurrency 16, MAX_PENDING_JOBS=32.
- Total 960 requests = 220 accepted (all ready) + 740 expected HTTP 429.
- Each run: failed=0, timed_out=0, metric_errors=[], drained=true,
  recovery_upload_ready=true. Observed pending never exceeded 32.
- Source records: real-c*-p*-s*.json; table/formulas/limitations in REPORT.md.
- Earlier small runs were exploratory and replaced by this comparable series.
- Job schema: 425 real outbox payloads validated; persisted redelivery is no-op.
  Detailed result: job-contract.json.
- Offline check, full OpenAPI validation, 8 + 7 local regressions passed again
  after the final PostgreSQL watchdog change. Mock throughput is not evidence.
- Final public persistence: PASS, actual service StartedAt values changed.
- Final public failovers: PASS for both API replicas. First recovery times:
  2.436s (113 successful writes, 1 failed attempt) and
  4.675s (95 successful writes, 2 failed attempts).
- Final public smoke after those failures: PASS.

Final confirmation on the watchdog build, 2026-10-27:
- All seven pipeline integration stages passed again; pipeline-integration.json
  contains the final run, not the earlier exploratory attempts.
- Attachments, S3 permissions, concurrent idempotency and public smoke passed again.
- Worker metrics: pending=0, concurrency=2, prefetch=2.
- Working RabbitMQ queue: messages_ready=0, messages_unacknowledged=0.
- DLQ contains 3 deliberately malformed test messages (not pending image work).
- broker, worker, both APIs, PostgreSQL, S3 and LB healthy; outbox/reconcile running;
  migrate and s3-init exited 0. Normal configuration restored; volumes preserved.
- Final working-tree secret scan: 108 files, 9 local values plus key patterns,
  no findings. git diff --check passed; README/contracts/public tests unchanged.
