forked from mtusi2026/mtusi2026lab4
When the ready commit fails after the object was written, the upload handler now locks the row, keeps it if it turned out ready, and otherwise deletes the object first and the pending row second before answering 503. A ready update that touches no row means the intent was already swept, so it fails as unavailable and the object goes too; a 201 without a row cannot happen. Every worker runs an UploadSweeper that every 30 s removes pending intents older than UPLOAD_GRACE_SECONDS (900 s, at least 60 s) one row at a time under FOR UPDATE SKIP LOCKED over the partial pending index, object first, and logs an audit event for each discard. Bodies are buffered only inside one of UPLOAD_CONCURRENCY slots per worker and released before the ready commit, and a client that disconnects in the middle of a body becomes a 400 instead of an unhandled 500. The tests run against a throwaway PostgreSQL: a trigger rejects the ready commit and the compensation leaves neither row nor object, a row removed during the write gives 503, a failed write stays until the grace ends, fresh intents and an aged ready attachment in a message survive a sweep, two concurrent sweeps share five intents exactly, the background loop runs by itself, a second upload waits for the slot without reading its body, and aborted and stalled bodies leave nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
69 lines
2.1 KiB
Python
69 lines
2.1 KiB
Python
from collections.abc import AsyncIterator, Awaitable, Callable
|
|
from datetime import datetime, timedelta
|
|
from typing import Any
|
|
|
|
from dishka import Provider, Scope, provide
|
|
|
|
from messenger.attachments.application.errors import ObjectMissingError
|
|
from messenger.attachments.application.ports import ObjectContent, ObjectStorage
|
|
from messenger.shared.application.errors import Unavailable
|
|
|
|
|
|
class NullLogger:
|
|
__slots__ = ()
|
|
|
|
def info(self, event: str, /, **fields: Any) -> None: ...
|
|
|
|
def warning(self, event: str, /, **fields: Any) -> None: ...
|
|
|
|
def error(self, event: str, /, **fields: Any) -> None: ...
|
|
|
|
|
|
class FrozenClock:
|
|
__slots__ = ("_current",)
|
|
|
|
def __init__(self, start: datetime) -> None:
|
|
self._current = start
|
|
|
|
def now(self) -> datetime:
|
|
self._current += timedelta(microseconds=1)
|
|
return self._current
|
|
|
|
|
|
async def pieces(data: bytes, size: int = 65_536) -> AsyncIterator[bytes]:
|
|
for start in range(0, len(data), size):
|
|
yield data[start : start + size]
|
|
|
|
|
|
class MemoryObjectStorage:
|
|
__slots__ = ("after_put", "failing", "objects")
|
|
|
|
def __init__(self) -> None:
|
|
self.objects: dict[str, tuple[bytes, str]] = {}
|
|
self.failing = False
|
|
self.after_put: Callable[[str], Awaitable[None]] | None = None
|
|
|
|
async def put(self, key: str, data: bytes, content_type: str) -> None:
|
|
if self.failing:
|
|
raise Unavailable("S3 недоступно")
|
|
self.objects[key] = (bytes(data), content_type)
|
|
if self.after_put is not None:
|
|
await self.after_put(key)
|
|
|
|
async def open(self, key: str) -> ObjectContent:
|
|
if self.failing:
|
|
raise Unavailable("S3 недоступно")
|
|
if key not in self.objects:
|
|
raise ObjectMissingError(key)
|
|
data = self.objects[key][0]
|
|
return ObjectContent(size=len(data), chunks=pieces(data))
|
|
|
|
async def delete(self, key: str) -> None:
|
|
self.objects.pop(key, None)
|
|
|
|
|
|
class MemoryStorageProvider(Provider):
|
|
@provide(scope=Scope.APP)
|
|
def storage(self) -> ObjectStorage:
|
|
return MemoryObjectStorage()
|