# 2026-10-07 22:25:25 UTC, python 3.14.7, win32
# docker 29.8.1
== Stack: replicas, pinned images, health, limits, published ports, secrets ==
$ docker compose ps -a --format table {{.Name}}	{{.Image}}	{{.Status}}	{{.Ports}}
NAME                    IMAGE                                                                                                STATUS                     PORTS
mtusi-lab3-api-3        mtusi-messenger:lab3                                                                                 Up 6 minutes (healthy)     8080/tcp
mtusi-lab3-api-4        mtusi-messenger:lab3                                                                                 Up 6 minutes (healthy)     8080/tcp
mtusi-lab3-lb-1         haproxy:3.4.5-trixie@sha256:76928c0d6b39bdd5f1c15d519cf48c47a6aa18c1dc552f7af1c146d2aa003c14         Up 6 minutes               127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
mtusi-lab3-migrate-1    sha256:c2a29f803684ad58f400a42d32e5c262fa44580101e65188ef92ca6015e17885                              Exited (0) 6 minutes ago   
mtusi-lab3-postgres-1   postgres:18.6-trixie@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336         Up 6 minutes (healthy)     5432/tcp
mtusi-lab3-valkey-1     valkey/valkey:9.0.6-trixie@sha256:63346cb24a61221e76bdf41acce99b3968a9fa83d8122144deab45394b27b4f2   Up 6 minutes (healthy)     6379/tcp
$ docker compose config --format json
=> PASS: third-party images are pinned by tag and digest (haproxy:3.4.5-trixie@sha256:76928c0d6b39bdd5f1c15d519cf48c47a6aa18c1dc552f7af1c146d2aa003c14, postgres:18.6-trixie@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336, valkey/valkey:9.0.6-trixie@sha256:63346cb24a61221e76bdf41acce99b3968a9fa83d8122144deab45394b27b4f2)

api replicas: mtusi-lab3-api-3 (68717763b8c6), mtusi-lab3-api-4 (7fc81857cf65)
=> PASS: at least two api replicas of one compose service, no container_name (mtusi-lab3-api-3, mtusi-lab3-api-4)

health: mtusi-lab3-api-3=healthy, mtusi-lab3-api-4=healthy, postgres=healthy, valkey=healthy
=> PASS: all api replicas, postgres and valkey are healthy

=> PASS: API processes do not run as root (mtusi-lab3-api-3 uid=10001, mtusi-lab3-api-4 uid=10001)

container                      cpus     memory   published ports
lb                             0.50      128 MB   127.0.0.1:8080->8080/tcp 127.0.0.1:8404->8404/tcp
mtusi-lab3-api-3               0.50      256 MB   -
mtusi-lab3-api-4               0.50      256 MB   -
postgres                       1.00      512 MB   -
valkey                         0.50      256 MB   -
=> PASS: every container has CPU and memory limits

=> PASS: only the load balancer publishes ports, and only on 127.0.0.1

secrets per service: api=cache_url,database_url; lb=-; migrate=database_url; postgres=postgres_password; valkey=valkey_auth
=> PASS: secrets are given only where needed

$ docker compose exec -T valkey sh -c valkey-cli --no-auth-warning -a "$(sed -n "s/^requirepass //p" /run/secrets/valkey_auth)" CONFIG GET appendonly
appendonly
yes
$ docker compose exec -T valkey sh -c valkey-cli --no-auth-warning -a "$(sed -n "s/^requirepass //p" /run/secrets/valkey_auth)" CONFIG GET appendfsync
appendfsync
always
$ docker compose exec -T valkey sh -c valkey-cli --no-auth-warning -a "$(sed -n "s/^requirepass //p" /run/secrets/valkey_auth)" CONFIG GET maxmemory-policy
maxmemory-policy
noeviction
=> PASS: Valkey persistence is AOF with fsync always, no eviction

== Migrations: run once by the migrate service, idempotent re-run ==
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c SELECT version || ' ' || applied_at FROM schema_migrations ORDER BY version
0001_schema 2026-10-07 22:03:12.021632+00
0002_messages_history_index 2026-10-07 22:03:12.034748+00
0003_idempotency_keys 2026-10-07 22:03:12.038316+00
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c SELECT format('%s %s %s', conrelid::regclass, contype, conname) FROM pg_constraint WHERE conrelid::regclass::text = 'idempotency_keys' ORDER BY 1
idempotency_keys c idempotency_keys_key_format
idempotency_keys f idempotency_keys_message_id_fkey
idempotency_keys f idempotency_keys_user_id_fkey
idempotency_keys n idempotency_keys_created_at_not_null
idempotency_keys n idempotency_keys_key_not_null
idempotency_keys n idempotency_keys_message_id_not_null
idempotency_keys n idempotency_keys_method_not_null
idempotency_keys n idempotency_keys_path_not_null
idempotency_keys n idempotency_keys_request_hash_not_null
idempotency_keys n idempotency_keys_user_id_not_null
idempotency_keys p idempotency_keys_pkey
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c SELECT (SELECT count(*) FROM schema_migrations) || ' ' || (SELECT count(*) FROM users)
3 205
$ docker compose run --rm --no-deps migrate
Container mtusi-lab3-migrate-run-ee50296698bb Creating 
 Container mtusi-lab3-migrate-run-ee50296698bb Created 
INFO event=migrations_done applied_now=0 total=3
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c SELECT (SELECT count(*) FROM schema_migrations) || ' ' || (SELECT count(*) FROM users)
3 205
schema_migrations and users rows before/after: 3 205 / 3 205
=> PASS: re-running migrations applies nothing and duplicates nothing

== Balancing: one session, many replicas ==
X-Instance-Id 68717763b8c6 (mtusi-lab3-api-3): 20 responses with status 200
X-Instance-Id 7fc81857cf65 (mtusi-lab3-api-4): 20 responses with status 200
=> PASS: 40 requests with one token are served by every replica without a new login

== Idempotency: 20 parallel POSTs with one key through the load balancer ==
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c SELECT count(*) FROM messages WHERE chat_id = '83406a9f-af29-4189-bd5a-80f48d1d064f'
1
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c SELECT count(*) FROM idempotency_keys WHERE key = '8eaaba5dfe6645a895251432314215c9'
1
statuses: [201]; distinct message ids: 1; replicas that answered: 68717763b8c6, 7fc81857cf65
messages in the chat: 1; idempotency_keys rows for the key: 1
same key, another body                                     -> 409 conflict in 0.01 s, X-Instance-Id 7fc81857cf65
=> PASS: one message for 20 concurrent requests with one key on several replicas

=> PASS: the same key with another body is rejected with 409

== Rollback: chat creation fails on the second INSERT ==
POST /api/v1/chats with one unknown member                 -> 400 invalid_request in 0.02 s, X-Instance-Id 68717763b8c6
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c SELECT count(*) FROM chats WHERE title = 'rollback-probe-197e464657'
0
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c SELECT count(*) FROM chats c WHERE NOT EXISTS (SELECT 1 FROM chat_members m WHERE m.chat_id = c.id)
0
rows in chats with this title: 0; chats without members: 0
=> PASS: composite chat creation is rolled back as a whole

== Registration race: 20 parallel requests with one username ==
statuses: 201 409 409 409 409 409 409 409 409 409 409 409 409 409 409 409 409 409 409 409; replicas: 68717763b8c6, 7fc81857cf65
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c SELECT count(*) FROM users WHERE username = 'race_9ef48474bc80'
1
rows in users with this username: 1
=> PASS: exactly one registration wins, the other 19 get 409

== Connection pools: 60 parallel clients, at most 10 connections per replica (2 replicas) ==
requests: 563, statuses: {200: 563}
API connections in pg_stat_activity, 9 samples: max 20
=> PASS: pools never open more than 10 connections per replica and all requests succeed

== Timeouts: chat row locked for 6 s by another session, API answers 503 after lock_timeout=3s ==
$ docker compose exec -T postgres psql -U messenger -d messenger -v ON_ERROR_STOP=1 -At -c "BEGIN; SELECT 1 FROM chats WHERE id = '6026c0e8-0d1a-4e86-ae54-da84dc874367' FOR UPDATE; SELECT pg_sleep(6); ROLLBACK;"   (in the background)
POST message while the chat row is locked                  -> 503 unavailable in 3.02 s, X-Instance-Id 68717763b8c6
POST message after the lock is released                    -> 201  in 0.02 s, X-Instance-Id 7fc81857cf65
=> PASS: waiting for a lock is bounded: 503 after about 3 s, then writes work again (3.02 s)

== Logs: useful fields, no message texts, passwords or tokens (all api replicas and lb) ==
$ docker compose logs --no-log-prefix api lb
$ docker compose logs --tail 2 api
api-3  | 2026-10-07T22:25:54.557Z INFO request_id=966a547d1ac94f6c82eff4c595ee9f9b method=POST path=/api/v1/chats status=201 duration_ms=5.8 user_id=44136183-c19b-4b5a-adbb-6858489b0330
api-4  | 2026-10-07T22:25:54.437Z INFO request_id=964da93e470f47caa94279cd2e9e7516 method=POST path=/api/v1/chats/6026c0e8-0d1a-4e86-ae54-da84dc874367/messages status=201 duration_ms=12.0 user_id=ac7c4c7a-41aa-401d-bdd5-39d27066e9b3
api-4  | 2026-10-07T22:25:54.543Z INFO request_id=23d18bd6174749a99f9fb1dd222bdfb7 method=POST path=/api/v1/auth/sessions status=201 duration_ms=39.0 user_id=44136183-c19b-4b5a-adbb-6858489b0330
api-3  | 2026-10-07T22:25:54.583Z INFO request_id=61b9245ffbf24e6ea1d5c0e46ddd1e80 method=POST path=/api/v1/chats/2bdbc8f5-9be8-449d-940b-5c97ef9a6943/messages status=201 duration_ms=10.7 user_id=44136183-c19b-4b5a-adbb-6858489b0330
$ docker compose logs --no-log-prefix --tail 2 lb
172.19.0.1:57694 [07/Oct/2026:22:25:54.549] public api/api1 0/0/1/6/7 201 373 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/chats HTTP/1.1"
172.19.0.1:57710 [07/Oct/2026:22:25:54.570] public api/api1 0/0/0/13/13 201 457 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/chats/2bdbc8f5-9be8-449d-940b-5c97ef9a6943/messages HTTP/1.1"
searched 41389 log lines; found: nothing
=> PASS: logs contain request_id/status/duration and no secrets

== Session expiry: api replicas recreated with SESSION_TTL_SECONDS=2, then restored ==
$ docker compose up -d --no-deps --wait api
GET /users/me right after login                            -> 200  in 0.01 s, X-Instance-Id 911800b97ae9
$ docker compose exec -T valkey sh -c valkey-cli --no-auth-warning -a "$(sed -n "s/^requirepass //p" /run/secrets/valkey_auth)" PTTL session:6b03ee75d9f79c514394462e72c39c21a3d11ace901c5a42c89b5ef93cd136fb
1694
PTTL of the session key in Valkey (ms): 1694
GET /users/me 3 s later                                    -> 401 unauthorized in 0.02 s, X-Instance-Id ba820ebd9c68
$ docker compose exec -T valkey sh -c valkey-cli --no-auth-warning -a "$(sed -n "s/^requirepass //p" /run/secrets/valkey_auth)" EXISTS session:6b03ee75d9f79c514394462e72c39c21a3d11ace901c5a42c89b5ef93cd136fb
0
session key still exists in Valkey: 0
=> PASS: session expires after SESSION_TTL_SECONDS and the key disappears from Valkey

$ docker compose up -d --no-deps --wait api
== Valkey is down ==
$ docker compose stop valkey
Container mtusi-lab3-valkey-1 Stopping 
 Container mtusi-lab3-valkey-1 Stopped
GET /users/me with a valid token (first seconds)           -> 503 unavailable in 1.02 s, X-Instance-Id 34f46eb4cd91
GET /users/me with a valid token (after 3 s)               -> 503 unavailable in 0.01 s, X-Instance-Id -
GET /users/me without a token                              -> 503 unavailable in 0.01 s, X-Instance-Id -
GET /health/ready                                          -> 503 unavailable in 0.00 s, X-Instance-Id -
slowest of 3 more authenticated requests: 0.01 s
$ docker compose start valkey
Container mtusi-lab3-valkey-1 Starting 
 Container mtusi-lab3-valkey-1 Started
same token after valkey is back: 200 after 1.5 s
=> PASS: without valkey: bounded 503, nothing is served as anonymous

=> PASS: the stand recovers by itself when valkey is back, the session is still valid

== PostgreSQL is down ==
$ docker compose stop postgres
Container mtusi-lab3-postgres-1 Stopping 
 Container mtusi-lab3-postgres-1 Stopped
GET /users/me with a valid token (first seconds)           -> 503 unavailable in 3.02 s, X-Instance-Id cc69293773ec
GET /users/me with a valid token (after 3 s)               -> 503 unavailable in 0.01 s, X-Instance-Id -
GET /users/me without a token                              -> 503 unavailable in 0.01 s, X-Instance-Id -
GET /health/ready                                          -> 503 unavailable in 0.01 s, X-Instance-Id -
slowest of 3 more authenticated requests: 0.01 s
$ docker compose start postgres
Container mtusi-lab3-postgres-1 Starting 
 Container mtusi-lab3-postgres-1 Started
same token after postgres is back: 200 after 5.6 s
=> PASS: without postgres: bounded 503, nothing is served as anonymous

=> PASS: the stand recovers by itself when postgres is back, the session is still valid

== Public failover test: tests/resilience.py --mode failover, repeated until every replica was the target ==
$ C:\Users\Dima Chernikov\AppData\Local\Programs\Python\Python314\python.exe -X utf8 tests/resilience.py --mode failover --action-script scripts/stop-one.cmd --base-url http://127.0.0.1:8080
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.

{"observed_instances_before": 2, "successful_writes": 112, "failed_attempts": 0, "first_recovery_seconds": 3.827}

PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
attempt 1: stopped container cc69293773ec (mtusi-lab3-api-4), State: exited, ExitCode 0
=> PASS: public failover test passes (attempt 1) (17.1 s)

$ docker compose up -d --wait
$ C:\Users\Dima Chernikov\AppData\Local\Programs\Python\Python314\python.exe -X utf8 tests/resilience.py --mode failover --action-script scripts/stop-one.cmd --base-url http://127.0.0.1:8080
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.

{"observed_instances_before": 2, "successful_writes": 106, "failed_attempts": 0, "first_recovery_seconds": 4.132}

PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
attempt 2: stopped container 34f46eb4cd91 (mtusi-lab3-api-3), State: exited, ExitCode 0
=> PASS: public failover test passes (attempt 2) (16.7 s)

$ docker compose up -d --wait
=> PASS: every replica was stopped by the hook at least once (mtusi-lab3-api-3, mtusi-lab3-api-4)

== Experiment: stop, then kill, each replica in turn under load; restore before the next one ==
$ docker stop 34f46eb4cd91
$ docker start 34f46eb4cd91
{"replica": "mtusi-lab3-api-3", "lb_slot": "api2", "action": "docker stop", "command_s": 3.93, "baseline_ok_rps": 148.2, "during_ok_rps": 81.3, "errors_during": 0, "requests_during": 970, "max_gap_s": 0.18, "lb_marked_down_s": 1.0, "last_served_s": 0.97, "back_in_rotation_s": 5.19, "after_ok_rps": 218.1, "errors_after": 0}
$ docker stop cc69293773ec
$ docker start cc69293773ec
{"replica": "mtusi-lab3-api-4", "lb_slot": "api1", "action": "docker stop", "command_s": 3.96, "baseline_ok_rps": 171.4, "during_ok_rps": 83.8, "errors_during": 0, "requests_during": 1002, "max_gap_s": 0.1, "lb_marked_down_s": 1.09, "last_served_s": 1.02, "back_in_rotation_s": 5.34, "after_ok_rps": 165.3, "errors_after": 0}
$ docker kill 34f46eb4cd91
$ docker start 34f46eb4cd91
{"replica": "mtusi-lab3-api-3", "lb_slot": "api2", "action": "docker kill", "command_s": 0.4, "baseline_ok_rps": 202.0, "during_ok_rps": 69.4, "errors_during": 2, "requests_during": 585, "max_gap_s": 0.93, "lb_marked_down_s": 0.47, "last_served_s": 0.1, "back_in_rotation_s": 5.47, "after_ok_rps": 123.8, "errors_after": 0}
$ docker kill cc69293773ec
$ docker start cc69293773ec
{"replica": "mtusi-lab3-api-4", "lb_slot": "api1", "action": "docker kill", "command_s": 0.43, "baseline_ok_rps": 151.4, "during_ok_rps": 67.6, "errors_during": 3, "requests_during": 573, "max_gap_s": 0.98, "lb_marked_down_s": 0.42, "last_served_s": 0.07, "back_in_rotation_s": 5.49, "after_ok_rps": 156.2, "errors_after": 0}

replica          action        cmd,s     rps0      rps  errors    gap,s   down,s   back,s     rps1
mtusi-lab3-api-3 docker stop    3.93    148.2     81.3       0     0.18      1.0     5.19    218.1
mtusi-lab3-api-4 docker stop    3.96    171.4     83.8       0      0.1     1.09     5.34    165.3
mtusi-lab3-api-3 docker kill     0.4    202.0     69.4       2     0.93     0.47     5.47    123.8
mtusi-lab3-api-4 docker kill    0.43    151.4     67.6       3     0.98     0.42     5.49    156.2
acknowledged writes: 4858, found in history: 4858, duplicates: 0
=> PASS: graceful stop of each replica: no client errors, LB removes it within 3 s

=> PASS: kill of each replica: LB removes it within 3 s, writes go on within 10 s

=> PASS: every replica comes back into rotation after docker start

=> PASS: no acknowledged write is lost or duplicated (retries reuse the key)

== Rolling update under load: one replica at a time, new image, migrations once ==
$ docker compose build api
image before 4f0ecfa64963, after e85c788ea2be, APP_VERSION=rolling-1791412151
$ docker compose run --rm --no-deps migrate
Container mtusi-lab3-migrate-run-132d5c300c9b Creating 
 Container mtusi-lab3-migrate-run-132d5c300c9b Created 
INFO event=migrations_done applied_now=0 total=3
$ docker compose up -d --no-deps --no-recreate --scale api=3 api
$ docker stop 34f46eb4cd91
$ docker rm 34f46eb4cd91
replaced mtusi-lab3-api-3 (34f46eb4cd91) -> 27757842cabd
$ docker compose up -d --no-deps --no-recreate --scale api=3 api
$ docker stop cc69293773ec
$ docker rm cc69293773ec
replaced mtusi-lab3-api-4 (cc69293773ec) -> 45ad15a9929a
during the update: {"requests": 5582, "ok": 5582, "errors": 0, "ok_rps": 190.1}
final replicas: mtusi-lab3-api-5 27757842cabd version=rolling-1791412151, mtusi-lab3-api-6 45ad15a9929a version=rolling-1791412151
acknowledged writes: 2791, found in history: 2791, duplicates: 0
=> PASS: migrations are applied once and the change is backward compatible

=> PASS: every replica was replaced one by one by the new image under load

=> PASS: no client errors and no lost or duplicated writes during the rolling update (5582 requests, 190.1 ok RPS)

== Full restart: public tests/resilience.py --mode persistence with the restart hook ==
$ docker volume inspect -f {{.Name}} {{.CreatedAt}} mtusi-lab3_pgdata mtusi-lab3_valkeydata
mtusi-lab3_pgdata 2026-10-07T22:03:02Z
mtusi-lab3_valkeydata 2026-10-07T22:03:02Z
$ C:\Users\Dima Chernikov\AppData\Local\Programs\Python\Python314\python.exe -X utf8 tests/resilience.py --mode persistence --action-script scripts/restart.cmd --base-url http://127.0.0.1:8080
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.

PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
$ docker volume inspect -f {{.Name}} {{.CreatedAt}} mtusi-lab3_pgdata mtusi-lab3_valkeydata
mtusi-lab3_pgdata 2026-10-07T22:03:02Z
mtusi-lab3_valkeydata 2026-10-07T22:03:02Z
lb                 StartedAt 22:19:18.913 -> 22:30:10.030
mtusi-lab3-api-5   StartedAt 22:29:22.591 -> 22:30:06.409
mtusi-lab3-api-6   StartedAt 22:29:34.442 -> 22:30:06.102
postgres           StartedAt 22:26:35.615 -> 22:29:56.621
valkey             StartedAt 22:26:26.521 -> 22:29:56.526
=> PASS: persistence test passes after restarting every replica, postgres, valkey and lb with the same volumes (hook scripts/restart.cmd, 19.9 s)

== Summary ==
PASS  third-party images are pinned by tag and digest
PASS  at least two api replicas of one compose service, no container_name
PASS  all api replicas, postgres and valkey are healthy
PASS  API processes do not run as root
PASS  every container has CPU and memory limits
PASS  only the load balancer publishes ports, and only on 127.0.0.1
PASS  secrets are given only where needed
PASS  Valkey persistence is AOF with fsync always, no eviction
PASS  re-running migrations applies nothing and duplicates nothing
PASS  40 requests with one token are served by every replica without a new login
PASS  one message for 20 concurrent requests with one key on several replicas
PASS  the same key with another body is rejected with 409
PASS  composite chat creation is rolled back as a whole
PASS  exactly one registration wins, the other 19 get 409
PASS  pools never open more than 10 connections per replica and all requests succeed
PASS  waiting for a lock is bounded: 503 after about 3 s, then writes work again
PASS  logs contain request_id/status/duration and no secrets
PASS  session expires after SESSION_TTL_SECONDS and the key disappears from Valkey
PASS  without valkey: bounded 503, nothing is served as anonymous
PASS  the stand recovers by itself when valkey is back, the session is still valid
PASS  without postgres: bounded 503, nothing is served as anonymous
PASS  the stand recovers by itself when postgres is back, the session is still valid
PASS  public failover test passes (attempt 1)
PASS  public failover test passes (attempt 2)
PASS  every replica was stopped by the hook at least once
PASS  graceful stop of each replica: no client errors, LB removes it within 3 s
PASS  kill of each replica: LB removes it within 3 s, writes go on within 10 s
PASS  every replica comes back into rotation after docker start
PASS  no acknowledged write is lost or duplicated (retries reuse the key)
PASS  migrations are applied once and the change is backward compatible
PASS  every replica was replaced one by one by the new image under load
PASS  no client errors and no lost or duplicated writes during the rolling update
PASS  persistence test passes after restarting every replica, postgres, valkey and lb with the same volumes
RESULT: PASS
