Добавить наблюдаемость OpenTelemetry и Grafana на оценку 3
This commit is contained in:
@@ -28,3 +28,12 @@ OTEL_SERVICE_NAME=messenger-api
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
|
||||
|
||||
# Actual Compose overrides (all other application settings have defaults).
|
||||
LB_PORT=8080
|
||||
LB_STATS_PORT=8404
|
||||
GRAFANA_PORT=3000
|
||||
PROMETHEUS_PORT=9090
|
||||
LOKI_PORT=3100
|
||||
TEMPO_PORT=3200
|
||||
# If another lab uses 8080/8404, choose e.g. 8086/8406 locally.
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
* text=auto
|
||||
*.sh text eol=lf
|
||||
*.py text eol=lf
|
||||
*.yaml text eol=lf
|
||||
Dockerfile* text eol=lf
|
||||
@@ -25,4 +25,43 @@ Smoke: `python -X utf8 tests/smoke.py`; дополнительные прове
|
||||
`python -X utf8 scripts/test_lab5.py --quick`.
|
||||
Остановка без потери данных: `docker compose down` (без `-v`).
|
||||
|
||||
Первый этап — только перенос; наблюдаемость будет добавлена следующими коммитами.
|
||||
## Наблюдаемость: уровень 3
|
||||
|
||||
API, worker и publisher используют OpenTelemetry Python SDK 1.28.2,
|
||||
OTLP HTTP/protobuf → Collector contrib 0.111.0 → Prometheus 2.55.0,
|
||||
Loki 3.2.1 (native OTLP), Tempo 2.6.1 → Grafana 11.2.2.
|
||||
Полные конфигурации и импортируемый dashboard находятся в `infra/`.
|
||||
Данные backend и состояние Grafana сохраняются в именованных volumes.
|
||||
|
||||
Метрики SDK `messenger.http.requests` и `messenger.jobs.processed` — counters;
|
||||
`messenger.http.duration` и `messenger.jobs.duration` — histograms в секундах.
|
||||
Prometheus добавляет `_total` к counters и `_seconds_bucket/sum/count` к
|
||||
histograms. Labels HTTP: method, route template, status_class; jobs:
|
||||
job_type и outcome. Resource attributes: service.name, service.version=1.6.0,
|
||||
service.instance.id, deployment.environment.name=lab. Сервисы называются
|
||||
messenger-api, messenger-worker и messenger-publisher.
|
||||
|
||||
API log `http.request.complete` связывается с ответом по X-Request-Id и
|
||||
X-Trace-Id. Worker log `job.complete` связывается с реальным CONSUMER span
|
||||
по trace_id/span_id. В логах есть timestamp/severity/event и service resource.
|
||||
Технические job/attachment ID разрешены только в traces/logs синтетического
|
||||
стенда; метрики не используют идентификаторы объектов.
|
||||
SDK экспортирует только явные allowlist-события, без текстов сообщений,
|
||||
паролей, токенов, SQL и exception text. Uvicorn access log выключен.
|
||||
|
||||
Grafana доступна только через loopback http://localhost:3000 (учебный
|
||||
anonymous Viewer); dashboard **Messenger — lab6**, три datasource заданы
|
||||
provisioning автоматически. Prometheus/Loki/Tempo на loopback
|
||||
9090/3100/3200 нужны также для воспроизводимых проверок.
|
||||
OTLP порты доступны только внутри сети Compose.
|
||||
|
||||
Collector применяет memory_limiter (192 MiB), batch (не больше 512 records),
|
||||
очереди exporters по 256 batches и retry до 30 секунд. У SDK фоновые очереди
|
||||
traces/logs по 512 records, batches по 64, HTTP timeout 2 секунды, период
|
||||
метрик 5 секунд. Ни API request, ни job не ожидают сетевой экспорт.
|
||||
Зависимости telemetry не входят в readiness приложения.
|
||||
|
||||
Источники: [OTel SDK](https://opentelemetry-python.readthedocs.io/en/stable/sdk/),
|
||||
[Loki OTLP](https://grafana.com/docs/loki/latest/send-data/otel/),
|
||||
[Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/).
|
||||
Реализация и документация подготовлены с помощью Codex.
|
||||
|
||||
+9
-1
@@ -14,6 +14,7 @@ from app.db import SessionLocal, engine
|
||||
from app.pipeline import (RETENTION_SECONDS, arm_from_env, broker_channel,
|
||||
crash_if_armed, envelope, publish, retry_or_fail)
|
||||
from app.storage import storage
|
||||
from app.telemetry import configure, emit, install_termination_handler, shutdown
|
||||
|
||||
logger = logging.getLogger("messenger.jobs")
|
||||
|
||||
@@ -86,10 +87,13 @@ async def relay_one(channel) -> bool:
|
||||
{"id": row["id"]})
|
||||
await db.commit()
|
||||
logger.info("published job=%s attempt=%s", row["job_id"], row["attempt"])
|
||||
emit("job.published", job_id=str(row["job_id"]), attempt=row["attempt"])
|
||||
return True
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
configure("messenger-publisher")
|
||||
install_termination_handler()
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
for name in ("aio_pika", "aiormq"):
|
||||
logging.getLogger(name).setLevel(logging.CRITICAL)
|
||||
@@ -128,7 +132,11 @@ async def main() -> None:
|
||||
await connection.close()
|
||||
await storage.close()
|
||||
await engine.dispose()
|
||||
await asyncio.to_thread(shutdown)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except asyncio.CancelledError:
|
||||
pass # SIGTERM has run finally/SDK flush; exit cleanly.
|
||||
|
||||
+18
-17
@@ -30,9 +30,10 @@ from app.errors import (
|
||||
Unavailable,
|
||||
api_error_handler,
|
||||
common_headers,
|
||||
incoming_request_id,
|
||||
)
|
||||
from app.lifecycle import draining
|
||||
from app.config import settings
|
||||
from app.telemetry import HttpTelemetryMiddleware, configure, shutdown
|
||||
from app.models import (
|
||||
Attachment,
|
||||
Chat,
|
||||
@@ -61,30 +62,26 @@ from app.services import (
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
|
||||
configure("messenger-api", settings.instance_id)
|
||||
await storage.start()
|
||||
janitor = asyncio.create_task(janitor_loop())
|
||||
yield
|
||||
janitor.cancel()
|
||||
await asyncio.gather(janitor, return_exceptions=True)
|
||||
await storage.close()
|
||||
await session_cache.close()
|
||||
await engine.dispose()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
janitor.cancel()
|
||||
await asyncio.gather(janitor, return_exceptions=True)
|
||||
await storage.close()
|
||||
await session_cache.close()
|
||||
await engine.dispose()
|
||||
await asyncio.to_thread(shutdown)
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="MTUSI Messenger — лабораторная 5",
|
||||
version="1.5.0",
|
||||
title="MTUSI Messenger — лабораторная 6",
|
||||
version="1.6.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
@app.middleware("http")
|
||||
async def add_request_id(request: Request, call_next):
|
||||
request.state.request_id = incoming_request_id(request)
|
||||
|
||||
response = await call_next(request)
|
||||
response.headers.update(common_headers(request))
|
||||
return response
|
||||
|
||||
@app.exception_handler(ApiError)
|
||||
async def handle_api_error(
|
||||
request: Request,
|
||||
@@ -458,3 +455,7 @@ async def download_attachment_route(
|
||||
media_type=headers.pop("Content-Type"),
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
# Wrap outside FastAPI ServerErrorMiddleware to observe its final 500 response.
|
||||
app = HttpTelemetryMiddleware(app)
|
||||
|
||||
@@ -62,6 +62,7 @@ def main() -> None:
|
||||
timeout_keep_alive=30,
|
||||
timeout_graceful_shutdown=settings.shutdown_timeout_seconds,
|
||||
log_level="info",
|
||||
access_log=False, # raw URL can contain filename; safe events are exported instead
|
||||
)
|
||||
DrainingServer(config).run()
|
||||
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
"""Explicit, bounded OTLP instrumentation; no body/headers/SQL auto-capture.
|
||||
|
||||
Only this module's safe event logger is exported. Existing application and
|
||||
third-party loggers are deliberately not attached to the OTLP handler.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
from opentelemetry import metrics, trace
|
||||
from opentelemetry._logs import set_logger_provider
|
||||
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
|
||||
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.propagate import extract
|
||||
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
|
||||
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor, LogExportResult
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import MetricExportResult, PeriodicExportingMetricReader
|
||||
from opentelemetry.sdk.metrics.view import ExplicitBucketHistogramAggregation, View
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExportResult
|
||||
from opentelemetry.trace import SpanKind, Status, StatusCode
|
||||
|
||||
meter = metrics.get_meter("messenger", "1.6.0")
|
||||
tracer = trace.get_tracer("messenger", "1.6.0")
|
||||
http_requests = meter.create_counter("messenger.http.requests")
|
||||
http_duration = meter.create_histogram("messenger.http.duration", unit="s")
|
||||
jobs_processed = meter.create_counter("messenger.jobs.processed")
|
||||
jobs_duration = meter.create_histogram("messenger.jobs.duration", unit="s")
|
||||
export_failures = meter.create_counter("messenger.telemetry.export.failures")
|
||||
dropped_records = meter.create_counter("messenger.telemetry.dropped.records")
|
||||
|
||||
_providers: list = []
|
||||
_service = "messenger"
|
||||
_event_logger = logging.getLogger("messenger.telemetry.events")
|
||||
_event_logger.propagate = False
|
||||
_event_logger.setLevel(logging.INFO)
|
||||
_ALLOWED_FIELDS = {
|
||||
"event", "request_id", "trace_id", "span_id", "method", "route", "status_code",
|
||||
"outcome", "duration_seconds", "job_type", "job_id", "attachment_id", "attempt",
|
||||
"exception_type", "operation", "queue", "count",
|
||||
}
|
||||
|
||||
|
||||
class _NoHttpRetry:
|
||||
# Verified against pinned OTel 1.28.2. Retry/backpressure is the Collector's
|
||||
# responsibility; a failing SDK export gets one HTTP attempt, <=2 s socket
|
||||
# timeout, then discards its bounded batch. No request waits on export.
|
||||
@staticmethod
|
||||
def _retryable(_response):
|
||||
return False
|
||||
|
||||
|
||||
class _SpanExporter(_NoHttpRetry, OTLPSpanExporter):
|
||||
def export(self, spans):
|
||||
try:
|
||||
result = super().export(spans)
|
||||
except Exception:
|
||||
result = SpanExportResult.FAILURE
|
||||
if result != SpanExportResult.SUCCESS:
|
||||
export_failures.add(1, {"signal": "traces"})
|
||||
dropped_records.add(len(spans), {"signal": "traces"})
|
||||
return result
|
||||
|
||||
|
||||
class _LogExporter(_NoHttpRetry, OTLPLogExporter):
|
||||
def export(self, batch):
|
||||
try:
|
||||
result = super().export(batch)
|
||||
except Exception:
|
||||
result = LogExportResult.FAILURE
|
||||
if result != LogExportResult.SUCCESS:
|
||||
export_failures.add(1, {"signal": "logs"})
|
||||
dropped_records.add(len(batch), {"signal": "logs"})
|
||||
return result
|
||||
|
||||
|
||||
class _MetricExporter(_NoHttpRetry, OTLPMetricExporter):
|
||||
def export(self, metrics_data, timeout_millis=10000, **kwargs):
|
||||
try:
|
||||
result = super().export(metrics_data, timeout_millis, **kwargs)
|
||||
except Exception:
|
||||
result = MetricExportResult.FAILURE
|
||||
if result != MetricExportResult.SUCCESS:
|
||||
export_failures.add(1, {"signal": "metrics"})
|
||||
return result
|
||||
|
||||
|
||||
class _JsonFormatter(logging.Formatter):
|
||||
def format(self, record):
|
||||
value = {
|
||||
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(record.created)),
|
||||
"severity": record.levelname, "service.name": _service,
|
||||
}
|
||||
value.update({key: getattr(record, key) for key in _ALLOWED_FIELDS
|
||||
if hasattr(record, key)})
|
||||
return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
|
||||
|
||||
|
||||
def configure(service: str, instance_id: str | None = None) -> None:
|
||||
"""One resource/provider set per process. OTEL_SDK_DISABLED supports off runs."""
|
||||
global _service
|
||||
if _providers:
|
||||
return
|
||||
_service = service
|
||||
console = logging.StreamHandler()
|
||||
console.setFormatter(_JsonFormatter())
|
||||
_event_logger.addHandler(console)
|
||||
if os.getenv("OTEL_SDK_DISABLED", "false").lower() == "true":
|
||||
return
|
||||
resource = Resource({
|
||||
"service.name": service,
|
||||
"service.version": os.getenv("APP_VERSION", "1.6.0"),
|
||||
"service.instance.id": instance_id or uuid4().hex,
|
||||
"deployment.environment.name": "lab",
|
||||
})
|
||||
endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4318").rstrip("/")
|
||||
trace_provider = TracerProvider(resource=resource)
|
||||
trace_provider.add_span_processor(BatchSpanProcessor(
|
||||
_SpanExporter(endpoint=endpoint + "/v1/traces", timeout=2),
|
||||
max_queue_size=512, max_export_batch_size=64,
|
||||
schedule_delay_millis=1000, export_timeout_millis=2000,
|
||||
))
|
||||
trace.set_tracer_provider(trace_provider)
|
||||
log_provider = LoggerProvider(resource=resource)
|
||||
log_provider.add_log_record_processor(BatchLogRecordProcessor(
|
||||
_LogExporter(endpoint=endpoint + "/v1/logs", timeout=2),
|
||||
max_queue_size=512, max_export_batch_size=64,
|
||||
schedule_delay_millis=1000, export_timeout_millis=2000,
|
||||
))
|
||||
set_logger_provider(log_provider)
|
||||
_event_logger.addHandler(LoggingHandler(level=logging.INFO, logger_provider=log_provider))
|
||||
reader = PeriodicExportingMetricReader(
|
||||
_MetricExporter(endpoint=endpoint + "/v1/metrics", timeout=2),
|
||||
export_interval_millis=5000, export_timeout_millis=2500,
|
||||
)
|
||||
# Boundaries are seconds. Quantiles are computed over merged replica buckets.
|
||||
boundaries = (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120)
|
||||
metric_provider = MeterProvider(resource=resource, metric_readers=[reader], views=[
|
||||
View(instrument_type=metrics.Histogram,
|
||||
aggregation=ExplicitBucketHistogramAggregation(boundaries)),
|
||||
])
|
||||
metrics.set_meter_provider(metric_provider)
|
||||
_providers.extend((metric_provider, log_provider, trace_provider))
|
||||
|
||||
|
||||
def shutdown() -> None:
|
||||
"""Flush bounded background buffers on normal termination (not SIGKILL)."""
|
||||
for provider in _providers:
|
||||
provider.shutdown()
|
||||
_providers.clear()
|
||||
|
||||
|
||||
def install_termination_handler() -> None:
|
||||
"""Turn worker/publisher SIGTERM into cancellation so their finally runs."""
|
||||
import asyncio
|
||||
loop = asyncio.get_running_loop()
|
||||
task = asyncio.current_task()
|
||||
if task is not None:
|
||||
try:
|
||||
loop.add_signal_handler(signal.SIGTERM, task.cancel)
|
||||
except NotImplementedError: # Windows host; Compose containers use POSIX.
|
||||
signal.signal(signal.SIGTERM, lambda *_: loop.call_soon_threadsafe(task.cancel))
|
||||
|
||||
|
||||
def emit(event: str, *, severity: int = logging.INFO, **fields) -> None:
|
||||
"""Call with a source-defined event and technical fields, never free text."""
|
||||
current = trace.get_current_span().get_span_context()
|
||||
attrs = {key: value for key, value in fields.items()
|
||||
if key in _ALLOWED_FIELDS and value is not None}
|
||||
attrs.update(event=event, trace_id=f"{current.trace_id:032x}", span_id=f"{current.span_id:016x}")
|
||||
_event_logger.log(severity, event, extra=attrs)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def span(name: str, attributes=None, kind=SpanKind.INTERNAL, parent=None):
|
||||
"""Safe span helper: never export exception messages or stack traces."""
|
||||
with tracer.start_as_current_span(
|
||||
name, attributes=attributes, kind=kind, context=parent,
|
||||
record_exception=False, set_status_on_exception=False,
|
||||
) as active:
|
||||
try:
|
||||
yield active
|
||||
except BaseException as exc:
|
||||
active.set_attribute("exception_type", type(exc).__name__)
|
||||
active.set_status(Status(StatusCode.ERROR))
|
||||
raise
|
||||
|
||||
|
||||
class HttpTelemetryMiddleware:
|
||||
"""Outer ASGI wrapper sees final status, streaming duration and 500 responses."""
|
||||
def __init__(self, app):
|
||||
self.app = app
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] != "http":
|
||||
return await self.app(scope, receive, send)
|
||||
from starlette.requests import Request
|
||||
from app.errors import common_headers, incoming_request_id
|
||||
|
||||
request = Request(scope)
|
||||
request.state.request_id = incoming_request_id(request)
|
||||
method = scope.get("method", "OTHER")
|
||||
if method not in {"GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"}:
|
||||
method = "OTHER"
|
||||
status = 500
|
||||
started = time.perf_counter()
|
||||
# Only W3C propagation fields are consumed; headers are never exported.
|
||||
parent = extract(dict(request.headers))
|
||||
with span("HTTP request", kind=SpanKind.SERVER, parent=parent) as active:
|
||||
async def observed_send(message):
|
||||
nonlocal status
|
||||
if message["type"] == "http.response.start":
|
||||
status = message["status"]
|
||||
headers = dict(message.get("headers", []))
|
||||
for key, value in common_headers(request).items():
|
||||
headers[key.lower().encode()] = value.encode()
|
||||
headers[b"x-trace-id"] = f"{active.get_span_context().trace_id:032x}".encode()
|
||||
message = {**message, "headers": list(headers.items())}
|
||||
await send(message)
|
||||
|
||||
try:
|
||||
await self.app(scope, receive, observed_send)
|
||||
finally:
|
||||
route = getattr(scope.get("route"), "path", "unmatched")
|
||||
duration = time.perf_counter() - started
|
||||
attrs = {"method": method, "route": route, "status_class": f"{status // 100}xx"}
|
||||
active.update_name(f"{method} {route}")
|
||||
active.set_attributes({**attrs, "http.response.status_code": status})
|
||||
if status >= 500:
|
||||
active.set_status(Status(StatusCode.ERROR))
|
||||
http_requests.add(1, attrs)
|
||||
http_duration.record(duration, attrs)
|
||||
emit("http.request.complete", severity=logging.ERROR if status >= 500 else logging.INFO,
|
||||
request_id=request.state.request_id, method=method, route=route,
|
||||
status_code=status, outcome="error" if status >= 500 else "success",
|
||||
duration_seconds=duration)
|
||||
+51
-4
@@ -8,6 +8,7 @@ import logging
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from sqlalchemy import text
|
||||
@@ -18,6 +19,9 @@ from app.pipeline import (IMAGE_TIMEOUT, LEASE_SECONDS, TOTAL_TIMEOUT,
|
||||
arm_from_env, broker_channel, crash_if_armed,
|
||||
event, parse_envelope, retry_or_fail)
|
||||
from app.storage import ObjectMissing, storage
|
||||
from app.telemetry import (configure, emit, install_termination_handler, jobs_duration,
|
||||
jobs_processed, shutdown, span)
|
||||
from opentelemetry.trace import SpanKind, Status, StatusCode
|
||||
|
||||
logger = logging.getLogger("messenger.worker")
|
||||
|
||||
@@ -145,20 +149,21 @@ async def finish(row: dict, *, variants=None, code: str | None = None,
|
||||
return True
|
||||
|
||||
|
||||
async def handle(message) -> None:
|
||||
async def _handle(message) -> str:
|
||||
try:
|
||||
job = parse_envelope(message.body)
|
||||
except (ValueError, TypeError, KeyError, AttributeError):
|
||||
logger.warning("invalid envelope sent to durable DLQ")
|
||||
await message.reject(requeue=False)
|
||||
return
|
||||
return "invalid"
|
||||
row = await claim(job)
|
||||
if row is None:
|
||||
# A ready/failed/stale attempt is a durable no-op. A duplicate of
|
||||
# active processing is covered by the DB lease recovery, so it too
|
||||
# can be acknowledged without running a second concurrent decoder.
|
||||
await message.ack()
|
||||
return
|
||||
return "duplicate"
|
||||
outcome = "error"
|
||||
try:
|
||||
async with asyncio.timeout(TOTAL_TIMEOUT):
|
||||
variants = await build_objects(row)
|
||||
@@ -172,12 +177,50 @@ async def handle(message) -> None:
|
||||
if await finish(row, variants=variants):
|
||||
await crash_if_armed("crash_after_commit_before_ack")
|
||||
logger.info("ready attachment=%s", row["id"])
|
||||
outcome = "success"
|
||||
else:
|
||||
outcome = "duplicate"
|
||||
# If finish failed, the exception propagates and the connection closes:
|
||||
# RabbitMQ redelivers while PG keeps the durable lease/retry intention.
|
||||
await message.ack()
|
||||
return outcome
|
||||
|
||||
|
||||
async def handle(message) -> None:
|
||||
# Technical IDs in logs/spans only; metric dimensions stay low-cardinality.
|
||||
job = {}
|
||||
try:
|
||||
job = parse_envelope(message.body)
|
||||
except (ValueError, TypeError, KeyError, AttributeError):
|
||||
pass
|
||||
started = time.perf_counter()
|
||||
outcome = "error"
|
||||
with span("attachments process", kind=SpanKind.CONSUMER, attributes={
|
||||
"messaging.system": "rabbitmq", "messaging.destination.name": "attachments",
|
||||
"messaging.operation": "process", "job_type": "attachment.process.v1",
|
||||
}) as active:
|
||||
try:
|
||||
outcome = await _handle(message)
|
||||
finally:
|
||||
duration = time.perf_counter() - started
|
||||
attrs = {"job_type": "attachment.process.v1", "outcome": outcome}
|
||||
jobs_processed.add(1, attrs)
|
||||
if outcome in {"success", "error"}:
|
||||
jobs_duration.record(duration, attrs)
|
||||
active.set_attribute("outcome", outcome)
|
||||
if outcome in {"error", "invalid"}:
|
||||
active.set_status(Status(StatusCode.ERROR))
|
||||
for name in ("job_id", "attachment_id", "attempt"):
|
||||
if name in job:
|
||||
active.set_attribute(name, job[name])
|
||||
emit("job.complete", severity=logging.WARNING if outcome == "error" else logging.INFO,
|
||||
**attrs, job_id=job.get("job_id"), attachment_id=job.get("attachment_id"),
|
||||
attempt=job.get("attempt"), duration_seconds=duration)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
configure("messenger-worker")
|
||||
install_termination_handler()
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
for name in ("aio_pika", "aiormq"):
|
||||
logging.getLogger(name).setLevel(logging.CRITICAL)
|
||||
@@ -204,7 +247,11 @@ async def main() -> None:
|
||||
await connection.close()
|
||||
await storage.close()
|
||||
await engine.dispose()
|
||||
await asyncio.to_thread(shutdown)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except asyncio.CancelledError:
|
||||
pass # SIGTERM has run finally/SDK flush; exit cleanly.
|
||||
|
||||
+92
-1
@@ -1,5 +1,5 @@
|
||||
#file: noinspection ComposeUnknownValues
|
||||
# Lab5: API x2 -> durable S3 staging + PostgreSQL outbox -> publisher
|
||||
# Lab6: API x2 -> durable S3 staging + PostgreSQL outbox -> publisher
|
||||
# -> RabbitMQ -> worker -> final private S3 objects + PostgreSQL metadata.
|
||||
# Start: see REPORT.md, "Запуск из чистого clone".
|
||||
name: messenger-lab6
|
||||
@@ -53,6 +53,12 @@ x-api-env: &api-env
|
||||
WORKER_FAULT: "${WORKER_FAULT:-}"
|
||||
# API fault-injection is disabled during normal operation.
|
||||
FAULT_INJECTION: "${FAULT_INJECTION:-}"
|
||||
OTEL_SDK_DISABLED: "${OTEL_SDK_DISABLED:-false}"
|
||||
OTEL_SERVICE_NAME: messenger-api
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES: service.version=1.6.0,deployment.environment.name=lab
|
||||
OTEL_METRIC_EXPORT_INTERVAL: "5000"
|
||||
|
||||
services:
|
||||
db:
|
||||
@@ -252,6 +258,7 @@ services:
|
||||
command: ["python", "-m", "app.jobs"]
|
||||
environment:
|
||||
<<: *api-env
|
||||
OTEL_SERVICE_NAME: messenger-publisher
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key, broker_url]
|
||||
networks: [backend]
|
||||
@@ -271,6 +278,7 @@ services:
|
||||
command: ["python", "-m", "app.worker"]
|
||||
environment:
|
||||
<<: *api-env
|
||||
OTEL_SERVICE_NAME: messenger-worker
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key, broker_url]
|
||||
networks: [backend]
|
||||
@@ -307,6 +315,84 @@ services:
|
||||
resources:
|
||||
limits: { cpus: "0.5", memory: 128M }
|
||||
|
||||
# Telemetry is deliberately absent from API depends_on/readiness.
|
||||
# No OTLP ports on the host: application traffic stays in backend.
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:0.111.0
|
||||
command: ["--config=/etc/otelcol/collector.yaml"]
|
||||
volumes:
|
||||
- ./infra/otel/collector.yaml:/etc/otelcol/collector.yaml:ro
|
||||
networks: [backend]
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits: { cpus: "0.5", memory: 256M }
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:v2.55.0
|
||||
command:
|
||||
- --config.file=/etc/prometheus/prometheus.yaml
|
||||
- --storage.tsdb.path=/prometheus
|
||||
- --storage.tsdb.retention.time=7d
|
||||
- --storage.tsdb.retention.size=512MB
|
||||
volumes:
|
||||
- ./infra/prometheus:/etc/prometheus:ro
|
||||
- prometheus_data:/prometheus
|
||||
ports: ["127.0.0.1:${PROMETHEUS_PORT:-9090}:9090"]
|
||||
networks: [backend, observability]
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits: { cpus: "0.5", memory: 512M }
|
||||
|
||||
loki:
|
||||
image: grafana/loki:3.2.1
|
||||
command: ["-config.file=/etc/loki/loki.yaml"]
|
||||
volumes:
|
||||
- ./infra/loki/loki.yaml:/etc/loki/loki.yaml:ro
|
||||
- loki_data:/loki
|
||||
ports: ["127.0.0.1:${LOKI_PORT:-3100}:3100"]
|
||||
networks: [backend, observability]
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits: { cpus: "0.5", memory: 384M }
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.6.1
|
||||
# A new named volume needs write access for the local evaluation backend.
|
||||
user: "0:0"
|
||||
command: ["-config.file=/etc/tempo/tempo.yaml"]
|
||||
volumes:
|
||||
- ./infra/tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
|
||||
- tempo_data:/var/tempo
|
||||
ports: ["127.0.0.1:${TEMPO_PORT:-3200}:3200"]
|
||||
networks: [backend, observability]
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits: { cpus: "0.5", memory: 384M }
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:11.2.2
|
||||
environment:
|
||||
GF_AUTH_ANONYMOUS_ENABLED: "true"
|
||||
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
|
||||
GF_AUTH_DISABLE_LOGIN_FORM: "true"
|
||||
GF_ANALYTICS_REPORTING_ENABLED: "false"
|
||||
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
|
||||
GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES: "false"
|
||||
volumes:
|
||||
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
- grafana_data:/var/lib/grafana
|
||||
ports: ["127.0.0.1:${GRAFANA_PORT:-3000}:3000"]
|
||||
networks: [backend, observability]
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits: { cpus: "0.5", memory: 256M }
|
||||
|
||||
# Runs the public and own checks in Linux (needed on Windows: the hooks are
|
||||
# POSIX .sh scripts that call the docker CLI). Profile "tools": not started
|
||||
# by a plain "up". The Docker socket grants control over the Docker host —
|
||||
@@ -328,6 +414,7 @@ services:
|
||||
networks: [edge, backend]
|
||||
|
||||
networks:
|
||||
observability: {}
|
||||
edge: {}
|
||||
# PostgreSQL, Valkey and S3: no route outside, the LB is not attached.
|
||||
backend:
|
||||
@@ -337,6 +424,10 @@ volumes:
|
||||
postgres_data: {}
|
||||
s3_data: {}
|
||||
broker_data: {}
|
||||
prometheus_data: {}
|
||||
loki_data: {}
|
||||
tempo_data: {}
|
||||
grafana_data: {}
|
||||
|
||||
secrets:
|
||||
postgres_password:
|
||||
|
||||
@@ -0,0 +1,328 @@
|
||||
{
|
||||
"uid": "messenger-lab6",
|
||||
"title": "Messenger lab6 - Observability",
|
||||
"description": "Application OTLP metrics, structured logs and traces. HTTP error ratio is business 5xx / all business responses; health excluded. Duration units are seconds. All API replicas are aggregated.",
|
||||
"tags": [
|
||||
"lab6",
|
||||
"opentelemetry"
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"editable": false,
|
||||
"timezone": "browser",
|
||||
"refresh": "5s",
|
||||
"time": {
|
||||
"from": "now-15m",
|
||||
"to": "now"
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "timeseries",
|
||||
"title": "Business API requests / second",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval]))",
|
||||
"legendFormat": ""
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "timeseries",
|
||||
"title": "Business API 5xx / all responses",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[$__rate_interval])) / clamp_min(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])), 1e-9)",
|
||||
"legendFormat": ""
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "timeseries",
|
||||
"title": "API mean duration (seconds)",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(messenger_http_duration_seconds_sum{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])) / clamp_min(sum(rate(messenger_http_duration_seconds_count{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])), 1e-9)",
|
||||
"legendFormat": ""
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "timeseries",
|
||||
"title": "Worker attempt outcomes / second",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (outcome) (rate(messenger_jobs_processed_total{service_name=\"messenger-worker\"}[$__rate_interval]))",
|
||||
"legendFormat": "{{outcome}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Worker mean attempt duration (seconds)",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(messenger_jobs_duration_seconds_sum{service_name=\"messenger-worker\"}[$__rate_interval])) / clamp_min(sum(rate(messenger_jobs_duration_seconds_count{service_name=\"messenger-worker\"}[$__rate_interval])), 1e-9)",
|
||||
"legendFormat": ""
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Worker completed attempts (counter)",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (outcome) (messenger_jobs_processed_total{service_name=\"messenger-worker\"})",
|
||||
"legendFormat": "{{outcome}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "logs",
|
||||
"title": "Structured application events (expand for trace_id / span_id / request_id)",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 24,
|
||||
"w": 24,
|
||||
"h": 9
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "{service_name=~\"messenger-api|messenger-worker|messenger-publisher\"}",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"enableLogDetails": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "table",
|
||||
"title": "Recent real API / worker traces (open TraceID)",
|
||||
"datasource": {
|
||||
"type": "tempo",
|
||||
"uid": "tempo"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 33,
|
||||
"w": 24,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"queryType": "traceql",
|
||||
"query": "{ resource.service.name = \"messenger-api\" || resource.service.name = \"messenger-worker\" }",
|
||||
"limit": 20,
|
||||
"tableType": "traces"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showHeader": true
|
||||
}
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"templating": {
|
||||
"list": []
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: messenger
|
||||
orgId: 1
|
||||
folder: Messenger
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 10
|
||||
allowUiUpdates: false
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,39 @@
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
uid: prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
editable: false
|
||||
jsonData:
|
||||
timeInterval: 5s
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
editable: false
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: TraceID
|
||||
matcherRegex: '"trace_id"\s*:\s*"([0-9a-f]{32})"'
|
||||
datasourceUid: tempo
|
||||
url: '$${__value.raw}'
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
editable: false
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: -5m
|
||||
spanEndTimeShift: 5m
|
||||
filterByTraceID: true
|
||||
filterBySpanID: false
|
||||
tags: [{ key: service.name, value: service_name }]
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
@@ -0,0 +1,39 @@
|
||||
auth_enabled: false
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
common:
|
||||
path_prefix: /loki
|
||||
replication_factor: 1
|
||||
ring:
|
||||
instance_addr: 127.0.0.1
|
||||
kvstore:
|
||||
store: inmemory
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2024-01-01
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
limits_config:
|
||||
allow_structured_metadata: true
|
||||
retention_period: 168h
|
||||
# Bound cardinality: request IDs / trace IDs remain structured metadata.
|
||||
otlp_config:
|
||||
resource_attributes:
|
||||
ignore_defaults: true
|
||||
attributes_config:
|
||||
- action: index_label
|
||||
attributes: [service.name, deployment.environment.name]
|
||||
compactor:
|
||||
working_directory: /loki/compactor
|
||||
retention_enabled: true
|
||||
delete_request_store: filesystem
|
||||
analytics:
|
||||
reporting_enabled: false
|
||||
@@ -0,0 +1,69 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 192
|
||||
spike_limit_mib: 48
|
||||
batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 256
|
||||
send_batch_max_size: 512
|
||||
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: 0.0.0.0:8889
|
||||
metric_expiration: 5m
|
||||
resource_to_telemetry_conversion:
|
||||
enabled: true
|
||||
otlphttp/tempo:
|
||||
endpoint: http://tempo:4318
|
||||
timeout: 3s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 256
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 30s
|
||||
otlphttp/loki:
|
||||
# otlphttp appends /v1/logs; Loki 3.2 accepts native OTLP here.
|
||||
endpoint: http://loki:3100/otlp
|
||||
timeout: 3s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 256
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 30s
|
||||
|
||||
service:
|
||||
telemetry:
|
||||
logs:
|
||||
level: warn
|
||||
metrics:
|
||||
address: 0.0.0.0:8888
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/tempo]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/loki]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [prometheus]
|
||||
@@ -0,0 +1,14 @@
|
||||
global:
|
||||
scrape_interval: 5s
|
||||
evaluation_interval: 5s
|
||||
|
||||
scrape_configs:
|
||||
- job_name: messenger
|
||||
static_configs:
|
||||
- targets: [otel-collector:8889]
|
||||
- job_name: collector
|
||||
static_configs:
|
||||
- targets: [otel-collector:8888]
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
- targets: [localhost:9090]
|
||||
@@ -0,0 +1,25 @@
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
ingester:
|
||||
trace_idle_period: 5s
|
||||
max_block_duration: 1m
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 168h
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
local:
|
||||
path: /var/tempo/blocks
|
||||
usage_report:
|
||||
reporting_enabled: false
|
||||
@@ -13,3 +13,8 @@ Pillow==11.0.0
|
||||
|
||||
# Lab5: durable RabbitMQ messages and publisher confirmations.
|
||||
aio-pika==9.4.3
|
||||
|
||||
# Explicit OTLP/HTTP SDK; semantic conventions bundled as 0.49b2.
|
||||
opentelemetry-api==1.28.2
|
||||
opentelemetry-sdk==1.28.2
|
||||
opentelemetry-exporter-otlp-proto-http==1.28.2
|
||||
|
||||
Reference in New Issue
Block a user