forked from mtusi2026/mtusi2026lab4
fix: одна повторная попытка S3 в том же дедлайне (мёртвые keep-alive после рестарта S3)
Найдено прогоном из чистого clone: после stop/start S3 первое скачивание получало 503 на соединении из пула к старому процессу. Все операции идемпотентны по ключу. Тест TTL при рестарте: 30 с (рестарт Valkey + API с проверкой S3 дольше 12 с). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f9f5ab9e75
commit
046e257f90
@@ -7,8 +7,11 @@
|
||||
* Deadlines: botocore connect/read timeouts AND an asyncio deadline around every call, so a
|
||||
paused or unreachable S3 costs at most S3_TIMEOUT_SECONDS, then 503 `unavailable`. Unlike a
|
||||
libpq query (db.py), an aiohttp request can be cancelled safely, so the call is cancelled.
|
||||
* No retries inside botocore (max_attempts=1): the request handler decides; an upload is not
|
||||
retried behind the client's back (contract: a retry may create another object).
|
||||
* ONE retry inside botocore (total_max_attempts=2), within the same asyncio deadline: after S3
|
||||
restarts, pooled keep-alive connections to the old process are dead and the first request on
|
||||
such a connection fails although S3 is up again (found by test_s3_outage_..._stop on a clean
|
||||
clone). Every call here is idempotent PER KEY (PUT of the same bytes to the same server-chosen
|
||||
key, GET, HEAD, DELETE), so repeating it cannot create a second attachment.
|
||||
* Checksums: Content-MD5 is sent with every PUT, so S3 rejects bytes damaged on the way.
|
||||
botocore's newer default (aws-chunked CRC32 trailers) is switched off ("when_required"):
|
||||
plain, widely supported S3 requests.
|
||||
@@ -65,7 +68,7 @@ class Storage:
|
||||
self._config = AioConfig(
|
||||
connect_timeout=settings.s3_connect_timeout_seconds,
|
||||
read_timeout=settings.s3_timeout_seconds,
|
||||
retries={"total_max_attempts": 1, "mode": "standard"},
|
||||
retries={"total_max_attempts": 2, "mode": "standard"},
|
||||
s3={"addressing_style": "path"},
|
||||
signature_version="s3v4",
|
||||
request_checksum_calculation="when_required",
|
||||
|
||||
@@ -38,7 +38,8 @@ def test_short_ttl_expires(api, store):
|
||||
|
||||
|
||||
def test_restart_does_not_extend_ttl(api):
|
||||
with api_with_env(SESSION_TTL_SECONDS="12"):
|
||||
# 30 s: a restart of Valkey + both API replicas (readiness now also waits for S3) takes ~12-15 s.
|
||||
with api_with_env(SESSION_TTL_SECONDS="30"):
|
||||
actor = api.register("ttlrestart")
|
||||
session = login_session(api, actor)
|
||||
restart_services("valkey", "api")
|
||||
|
||||
Reference in New Issue
Block a user