Failover profile label baseline from 459f84c on a checked host: stop and
kill of each replica under 200 req/s give 0 errors and LB exclusion in
0.79-1.81 s (bound 2.5 s), pause gives LB exclusion in 5.83-6.00 s
(bound 6.5 s) with 504s for half the requests and recovery in 6 s;
reconciliation of retried writes matches. Evidence for detection,
keep-alive (LB closes idle replica connections at 10.1 s) and the hung
replica key lock, regression checks, timeout chain and level 4 rows in
REPORT.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1.8 KiB
1.8 KiB
| 1 | phase | mode | replica | instance | slot | action_at | signal | lb_out_s | lb_out_state | lb_out_reason | bound_s | first_write_ok_s | first_read_ok_s | last_error_s | requests | errors_before | errors_down | codes_down | errors_after | codes_after | min_success | recovery_s | ok_rps_before | ok_rps_down_min | p95_before_ms | p95_down_max_ms | restore_at | lb_up_s | slot_back | instance_back_s | writes | writes_failed | retried_ok | write_chat_messages | distinct_texts | pg_idle_tx_terminated | dropped_iterations | flags |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 01-stop-r1 | stop | 1 | 8c238aef7975 | api-2 | 2026-10-10T20:42:40.136+00:00 | 15 | 1.06 | MAINT | No IP for server | 2.5 | 0.01 | 0.01 | 10002 | 0 | 0 | 0 | 1.0 | 0 | 200.0 | 200 | 5 | 505 | 2026-10-10T20:43:00.202+00:00 | 1.73 | api-2 | 1.73 | 5001 | 0 | 0 | 5001 | 5001 | 0 | 0 | ||||
| 3 | 02-stop-r2 | stop | 2 | a5d4fe1bb879 | api-1 | 2026-10-10T20:43:33.240+00:00 | 15 | 1.01 | MAINT | No IP for server | 2.5 | 0.0 | 0.0 | 10002 | 0 | 0 | 0 | 1.0 | 0 | 200.0 | 200 | 4 | 505 | 2026-10-10T20:43:53.297+00:00 | 1.21 | api-1 | 1.21 | 5001 | 0 | 0 | 10002 | 10002 | 0 | 0 | ||||
| 4 | 03-kill-r1 | kill | 1 | 8c238aef7975 | api-2 | 2026-10-10T20:44:26.487+00:00 | 9 | 1.81 | MAINT | No IP for server | 2.5 | 0.01 | 0.01 | 10000 | 0 | 0 | 0 | 1.0 | 0 | 200.1 | 199 | 4 | 505 | 2026-10-10T20:44:46.557+00:00 | 1.16 | api-2 | 1.17 | 5000 | 0 | 0 | 15002 | 15002 | 0 | 0 | ||||
| 5 | 04-kill-r2 | kill | 2 | a5d4fe1bb879 | api-1 | 2026-10-10T20:45:19.549+00:00 | 9 | 0.79 | MAINT | No IP for server | 2.5 | 0.0 | 0.0 | 10002 | 0 | 0 | 0 | 1.0 | 0 | 200.0 | 200 | 5 | 505 | 2026-10-10T20:45:39.619+00:00 | 1.75 | api-1 | 1.75 | 5001 | 0 | 0 | 20003 | 20003 | 0 | 0 | ||||
| 6 | 05-pause-r1 | pause | 1 | 8c238aef7975 | api-2 | 2026-10-10T20:46:12.647+00:00 | 6.0 | DOWN | Layer7 timeout | 6.5 | 0.03 | 0.01 | 6.0 | 10002 | 0 | 602 | 504:602 | 0 | 0.5 | 6 | 199.8 | 99 | 5 | 104 | 2026-10-10T20:46:32.647+00:00 | 1.53 | api-2 | 1.54 | 5001 | 240 | 240 | 25004 | 25004 | 0 | 0 | |||
| 7 | 06-pause-r2 | pause | 2 | a5d4fe1bb879 | api-1 | 2026-10-10T20:47:06.382+00:00 | 5.83 | DOWN | Layer7 timeout | 6.5 | 5.83 | 0.0 | 5.82 | 10002 | 0 | 584 | 504:584 | 0 | 0.5 | 6 | 199.9 | 100 | 4 | 150 | 2026-10-10T20:47:26.381+00:00 | 1.53 | api-1 | 1.54 | 5001 | 584 | 584 | 30005 | 30005 | 0 | 0 |