python3 scripts/lab5/worker_down_demo.py --base-url "http://localhost:8080" 
[1] compose ps
SERVICE          STATUS
api              Up Less than a second (health: starting)
api              Up 24 seconds (healthy)
broker           Up 27 seconds (healthy)
grafana          Up 28 seconds
haproxy          Up 28 seconds
loki             Up 27 seconds
otel-collector   Up 28 seconds
postgres         Up 28 seconds (healthy)
prometheus       Up 28 seconds
redis            Up 28 seconds (healthy)
s3               Up 28 seconds (healthy)
tempo            Up 28 seconds
worker           Up 28 seconds (healthy)
[2] stop worker
[3] upload while worker is down (expect 202 queued)
attachment=38d64985-29ef-4e80-a385-169bbcb65060 status=queued
[4] content and thumbnail must answer 409 before ready
[5] link the queued attachment to messages
message=38265a3d-2654-4c38-9615-19c32c013019
[6] wait for outbox publish and a message in the queue
queue={'messages': 1, 'type': 'quorum'}
[7] restart broker and API with volumes
[8] the quorum message survived the broker restart
queue={'messages': 1, 'type': 'quorum'}
[9] start worker
[10] wait ready and verify the thumbnail
ready attachment=38d64985-29ef-4e80-a385-169bbcb65060 thumbnail=256x160 sha256=a18bfc29cd931180...
[11] compose ps
SERVICE          STATUS
api              Up 22 seconds (healthy)
api              Up 22 seconds (healthy)
broker           Up 24 seconds (healthy)
grafana          Up About a minute
haproxy          Up About a minute
loki             Up About a minute
otel-collector   Up About a minute
postgres         Up About a minute (healthy)
prometheus       Up About a minute
redis            Up About a minute (healthy)
s3               Up About a minute (healthy)
tempo            Up About a minute
worker           Up 1 second (health: starting)
PASS: worker-down durability demo
