python3 scripts/lab5/worker_down_demo.py --base-url "http://localhost:8080"
[1] compose ps
SERVICE    STATUS
api        Up 41 seconds (healthy)
api        Up 2 seconds (health: starting)
broker     Up 44 seconds (healthy)
haproxy    Up 45 seconds
postgres   Up 45 seconds (healthy)
redis      Up 45 seconds (healthy)
s3         Up 46 seconds (healthy)
worker     Up 45 seconds (healthy)
[2] stop worker
[3] upload while worker is down (expect 202 queued)
attachment=ddc95667-82f9-49ed-a7d9-031ae46fb4f8 status=queued
[4] content and thumbnail must answer 409 before ready
[5] link the queued attachment to messages
message=13caa29b-26bd-49e6-8534-1a078df9a901
[6] wait for outbox publish and a message in the queue
queue={'messages': 1, 'type': 'quorum'}
[7] restart broker and API with volumes
[8] the quorum message survived the broker restart
queue={'messages': 1, 'type': 'quorum'}
[9] start worker
[10] wait ready and verify the thumbnail
ready attachment=ddc95667-82f9-49ed-a7d9-031ae46fb4f8 thumbnail=256x160 sha256=a18bfc29cd931180...
[11] compose ps
SERVICE    STATUS
api        Up 43 seconds (healthy)
api        Up 43 seconds (healthy)
broker     Up 45 seconds (healthy)
haproxy    Up About a minute
postgres   Up About a minute (healthy)
redis      Up About a minute (healthy)
s3         Up About a minute (healthy)
worker     Up 2 seconds (health: starting)
PASS: worker-down durability demo
