Commit Graph
6 Commits
Author SHA1 Message Date
Artyom 0a1f48bd73 Recover stale PostgreSQL connections before starting requests 2026-11-05 21:55:14 +03:00
Artyom 71c643c6d1 Adopt async SQLAlchemy storage and verified Alembic migrations
Keep the message and idempotency receipt in one Core CTE statement with chat-first locking. Preserve rollback, bounded dependency failures, and domain ports.

Replace the SQL runner with Alembic. Adopt legacy databases only after validating their journal and PostgreSQL schema, retain all data, and verify migrations on fresh and populated databases.
2026-11-05 21:38:12 +03:00
ArtyomandClaude Opus 5.5 d99a48a5db Put readiness behind a port and log the drain window through Logger
The /health/ready route imported the PostgreSQL adapter and kept the
drain flag in the HTTP layer, which gunicorn_conf imported to raise it.
Readiness is now a port of the shared application layer; its adapter in
shared/infrastructure checks the per-process drain flag and pings
PostgreSQL, and PlatformProvider assembles it. The route only knows the
port, and gunicorn_conf raises the flag through the infrastructure
module. Another critical dependency now changes the adapter and the
provider, not the HTTP layer.

The drain window was logged by a uvicorn logger with a printf string;
it is now the service.draining event of a StructlogLogger with the same
fields as the application, and failover-evidence looks for it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:40:16 +03:00
ArtyomandClaude Opus 5.5 aa1485bd86 Answer requests cancelled at shutdown with 503 in the contract format
After the drain window uvicorn waits for accepted requests for three
seconds and then cancels their tasks. CancelledError is a BaseException,
so neither the pipeline nor ServerErrorMiddleware saw it and uvicorn
answered with its own text/plain 500 without X-Request-Id and
X-Instance-Id. The pipeline now handles the cancellation next to the
request timeout: it logs request.cancelled, sends 503 unavailable in the
common error format when the response has not started and re-raises,
so uvicorn only closes the transport after the buffered answer.

The declared Content-Length check joins the guard as a rejection to keep
the call within the statement limit; behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:39:09 +03:00
ArtyomandClaude Opus 5.5 a59e3a39a8 Carry over the lab 2 fixes made after 8cddb09
Brings the system changes of lab 2 from 8cddb09..102579e onto the lab 3
code, resolving each conflict by meaning:

- the request pipeline counts the body bytes actually received, so a
  chunked body over the limit gets 413 instead of reaching the app;
- logout revokes through a SessionRevocations port and withdraws the
  mark when PostgreSQL rejects the delete, so 503 means the session
  still works; PostgresSessionDirectory loses its no-op invalidate;
- an empty cursor is 400 on history and on the chat list;
- EventBus.project and the test that allowed importing an event across
  zones are gone, the config rules speak of the schema foreign key;
- the worker waits one second less than the rest of the shutdown
  budget, so it cancels a stuck request with 500 and exits 0 instead
  of being killed by the gunicorn master. Lab 3 keeps 3 s after the
  drain window for the database timeouts, so SHUTDOWN_TIMEOUT_SECONDS
  becomes 6 s and the stop hooks wait 7 s, still under the 8 s grace;
- the lab 3 single statement post already locks the chat only for a
  member, so lab 2's LOCK_CHAT edit is not needed; its test is kept;
- lab 2 tests come along with Idempotency-Key where lab 3 needs it;
  the short TTL after restart runs without api-shortttl by shortening
  the session in the database; test_drain adds a stuck request;
- make validate uses .venv with openapi-spec-validator, PYTEST_ARGS,
  faults-evidence, down with the bench profile, the image is tagged by
  the compose project, start probes every 200 ms, cache evidence fails
  without a tombstone and checks the key expiry, kdf.txt shows the
  seed stub apart.

The track 2 tooling of lab 2 (track.py, postgres-restore and the
database argument of migrate) and its evidence scripts stay out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:32:49 +03:00
ArtyomandClaude Opus 5.5 6fb4587ef6 Leave the rotation before closing the listener on SIGTERM
On SIGTERM gunicorn closed the listener at once, so a request already
queued on the closing socket reached the balancer as a dropped
connection after sending: a retry for GET and keyed messages, a 502
for user, chat and login posts. Now each worker first raises a
draining flag and keeps serving for DRAIN_SECONDS (2 s by default,
above the 1.5 s worst detection of a 503): readiness answers 503
unavailable, liveness stays 200, and the balancer takes the replica
out while it is still healthy. Only then uvicorn closes the listener
and finishes accepted requests within SHUTDOWN_TIMEOUT_SECONDS minus
the window, so gunicorn graceful_timeout stays 5 s.

The worker runs a uvicorn Server subclass whose main loop adds the
window after the exit signal; UvicornWorker._serve is overridden to
use it. Tests check readiness under the flag and run real gunicorn on
a free port: during the window ready is 503 and live 200, the port
closes after the window, a registration accepted before SIGTERM and
finished after the port closed still gets 201, and the master exits
with code 0 before the shutdown limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:33:02 +03:00