Добавить наблюдаемость OpenTelemetry и Grafana на оценку 3

This commit is contained in:
2026-11-21 12:11:34 +03:00
parent 36d84eaf44
commit d0a062e2d2
17 changed files with 1000 additions and 24 deletions
+9
View File
@@ -28,3 +28,12 @@ OTEL_SERVICE_NAME=messenger-api
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
# Actual Compose overrides (all other application settings have defaults).
LB_PORT=8080
LB_STATS_PORT=8404
GRAFANA_PORT=3000
PROMETHEUS_PORT=9090
LOKI_PORT=3100
TEMPO_PORT=3200
# If another lab uses 8080/8404, choose e.g. 8086/8406 locally.
+5
View File
@@ -0,0 +1,5 @@
* text=auto
*.sh text eol=lf
*.py text eol=lf
*.yaml text eol=lf
Dockerfile* text eol=lf
+40 -1
View File
@@ -25,4 +25,43 @@ Smoke: `python -X utf8 tests/smoke.py`; дополнительные прове
`python -X utf8 scripts/test_lab5.py --quick`.
Остановка без потери данных: `docker compose down` (без `-v`).
Первый этап — только перенос; наблюдаемость будет добавлена следующими коммитами.
## Наблюдаемость: уровень 3
API, worker и publisher используют OpenTelemetry Python SDK 1.28.2,
OTLP HTTP/protobuf → Collector contrib 0.111.0 → Prometheus 2.55.0,
Loki 3.2.1 (native OTLP), Tempo 2.6.1 → Grafana 11.2.2.
Полные конфигурации и импортируемый dashboard находятся в `infra/`.
Данные backend и состояние Grafana сохраняются в именованных volumes.
Метрики SDK `messenger.http.requests` и `messenger.jobs.processed` — counters;
`messenger.http.duration` и `messenger.jobs.duration` — histograms в секундах.
Prometheus добавляет `_total` к counters и `_seconds_bucket/sum/count` к
histograms. Labels HTTP: method, route template, status_class; jobs:
job_type и outcome. Resource attributes: service.name, service.version=1.6.0,
service.instance.id, deployment.environment.name=lab. Сервисы называются
messenger-api, messenger-worker и messenger-publisher.
API log `http.request.complete` связывается с ответом по X-Request-Id и
X-Trace-Id. Worker log `job.complete` связывается с реальным CONSUMER span
по trace_id/span_id. В логах есть timestamp/severity/event и service resource.
Технические job/attachment ID разрешены только в traces/logs синтетического
стенда; метрики не используют идентификаторы объектов.
SDK экспортирует только явные allowlist-события, без текстов сообщений,
паролей, токенов, SQL и exception text. Uvicorn access log выключен.
Grafana доступна только через loopback http://localhost:3000 (учебный
anonymous Viewer); dashboard **Messenger — lab6**, три datasource заданы
provisioning автоматически. Prometheus/Loki/Tempo на loopback
9090/3100/3200 нужны также для воспроизводимых проверок.
OTLP порты доступны только внутри сети Compose.
Collector применяет memory_limiter (192 MiB), batch (не больше 512 records),
очереди exporters по 256 batches и retry до 30 секунд. У SDK фоновые очереди
traces/logs по 512 records, batches по 64, HTTP timeout 2 секунды, период
метрик 5 секунд. Ни API request, ни job не ожидают сетевой экспорт.
Зависимости telemetry не входят в readiness приложения.
Источники: [OTel SDK](https://opentelemetry-python.readthedocs.io/en/stable/sdk/),
[Loki OTLP](https://grafana.com/docs/loki/latest/send-data/otel/),
[Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/).
Реализация и документация подготовлены с помощью Codex.
+9 -1
View File
@@ -14,6 +14,7 @@ from app.db import SessionLocal, engine
from app.pipeline import (RETENTION_SECONDS, arm_from_env, broker_channel,
crash_if_armed, envelope, publish, retry_or_fail)
from app.storage import storage
from app.telemetry import configure, emit, install_termination_handler, shutdown
logger = logging.getLogger("messenger.jobs")
@@ -86,10 +87,13 @@ async def relay_one(channel) -> bool:
{"id": row["id"]})
await db.commit()
logger.info("published job=%s attempt=%s", row["job_id"], row["attempt"])
emit("job.published", job_id=str(row["job_id"]), attempt=row["attempt"])
return True
async def main() -> None:
configure("messenger-publisher")
install_termination_handler()
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
for name in ("aio_pika", "aiormq"):
logging.getLogger(name).setLevel(logging.CRITICAL)
@@ -128,7 +132,11 @@ async def main() -> None:
await connection.close()
await storage.close()
await engine.dispose()
await asyncio.to_thread(shutdown)
if __name__ == "__main__":
asyncio.run(main())
try:
asyncio.run(main())
except asyncio.CancelledError:
pass # SIGTERM has run finally/SDK flush; exit cleanly.
+18 -17
View File
@@ -30,9 +30,10 @@ from app.errors import (
Unavailable,
api_error_handler,
common_headers,
incoming_request_id,
)
from app.lifecycle import draining
from app.config import settings
from app.telemetry import HttpTelemetryMiddleware, configure, shutdown
from app.models import (
Attachment,
Chat,
@@ -61,30 +62,26 @@ from app.services import (
@asynccontextmanager
async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
configure("messenger-api", settings.instance_id)
await storage.start()
janitor = asyncio.create_task(janitor_loop())
yield
janitor.cancel()
await asyncio.gather(janitor, return_exceptions=True)
await storage.close()
await session_cache.close()
await engine.dispose()
try:
yield
finally:
janitor.cancel()
await asyncio.gather(janitor, return_exceptions=True)
await storage.close()
await session_cache.close()
await engine.dispose()
await asyncio.to_thread(shutdown)
app = FastAPI(
title="MTUSI Messenger — лабораторная 5",
version="1.5.0",
title="MTUSI Messenger — лабораторная 6",
version="1.6.0",
lifespan=lifespan,
)
@app.middleware("http")
async def add_request_id(request: Request, call_next):
request.state.request_id = incoming_request_id(request)
response = await call_next(request)
response.headers.update(common_headers(request))
return response
@app.exception_handler(ApiError)
async def handle_api_error(
request: Request,
@@ -458,3 +455,7 @@ async def download_attachment_route(
media_type=headers.pop("Content-Type"),
headers=headers,
)
# Wrap outside FastAPI ServerErrorMiddleware to observe its final 500 response.
app = HttpTelemetryMiddleware(app)
+1
View File
@@ -62,6 +62,7 @@ def main() -> None:
timeout_keep_alive=30,
timeout_graceful_shutdown=settings.shutdown_timeout_seconds,
log_level="info",
access_log=False, # raw URL can contain filename; safe events are exported instead
)
DrainingServer(config).run()
+245
View File
@@ -0,0 +1,245 @@
"""Explicit, bounded OTLP instrumentation; no body/headers/SQL auto-capture.
Only this module's safe event logger is exported. Existing application and
third-party loggers are deliberately not attached to the OTLP handler.
"""
from __future__ import annotations
from contextlib import contextmanager
import json
import logging
import os
import signal
import time
from uuid import uuid4
from opentelemetry import metrics, trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.propagate import extract
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor, LogExportResult
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import MetricExportResult, PeriodicExportingMetricReader
from opentelemetry.sdk.metrics.view import ExplicitBucketHistogramAggregation, View
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExportResult
from opentelemetry.trace import SpanKind, Status, StatusCode
meter = metrics.get_meter("messenger", "1.6.0")
tracer = trace.get_tracer("messenger", "1.6.0")
http_requests = meter.create_counter("messenger.http.requests")
http_duration = meter.create_histogram("messenger.http.duration", unit="s")
jobs_processed = meter.create_counter("messenger.jobs.processed")
jobs_duration = meter.create_histogram("messenger.jobs.duration", unit="s")
export_failures = meter.create_counter("messenger.telemetry.export.failures")
dropped_records = meter.create_counter("messenger.telemetry.dropped.records")
_providers: list = []
_service = "messenger"
_event_logger = logging.getLogger("messenger.telemetry.events")
_event_logger.propagate = False
_event_logger.setLevel(logging.INFO)
_ALLOWED_FIELDS = {
"event", "request_id", "trace_id", "span_id", "method", "route", "status_code",
"outcome", "duration_seconds", "job_type", "job_id", "attachment_id", "attempt",
"exception_type", "operation", "queue", "count",
}
class _NoHttpRetry:
# Verified against pinned OTel 1.28.2. Retry/backpressure is the Collector's
# responsibility; a failing SDK export gets one HTTP attempt, <=2 s socket
# timeout, then discards its bounded batch. No request waits on export.
@staticmethod
def _retryable(_response):
return False
class _SpanExporter(_NoHttpRetry, OTLPSpanExporter):
def export(self, spans):
try:
result = super().export(spans)
except Exception:
result = SpanExportResult.FAILURE
if result != SpanExportResult.SUCCESS:
export_failures.add(1, {"signal": "traces"})
dropped_records.add(len(spans), {"signal": "traces"})
return result
class _LogExporter(_NoHttpRetry, OTLPLogExporter):
def export(self, batch):
try:
result = super().export(batch)
except Exception:
result = LogExportResult.FAILURE
if result != LogExportResult.SUCCESS:
export_failures.add(1, {"signal": "logs"})
dropped_records.add(len(batch), {"signal": "logs"})
return result
class _MetricExporter(_NoHttpRetry, OTLPMetricExporter):
def export(self, metrics_data, timeout_millis=10000, **kwargs):
try:
result = super().export(metrics_data, timeout_millis, **kwargs)
except Exception:
result = MetricExportResult.FAILURE
if result != MetricExportResult.SUCCESS:
export_failures.add(1, {"signal": "metrics"})
return result
class _JsonFormatter(logging.Formatter):
def format(self, record):
value = {
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(record.created)),
"severity": record.levelname, "service.name": _service,
}
value.update({key: getattr(record, key) for key in _ALLOWED_FIELDS
if hasattr(record, key)})
return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
def configure(service: str, instance_id: str | None = None) -> None:
"""One resource/provider set per process. OTEL_SDK_DISABLED supports off runs."""
global _service
if _providers:
return
_service = service
console = logging.StreamHandler()
console.setFormatter(_JsonFormatter())
_event_logger.addHandler(console)
if os.getenv("OTEL_SDK_DISABLED", "false").lower() == "true":
return
resource = Resource({
"service.name": service,
"service.version": os.getenv("APP_VERSION", "1.6.0"),
"service.instance.id": instance_id or uuid4().hex,
"deployment.environment.name": "lab",
})
endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4318").rstrip("/")
trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(BatchSpanProcessor(
_SpanExporter(endpoint=endpoint + "/v1/traces", timeout=2),
max_queue_size=512, max_export_batch_size=64,
schedule_delay_millis=1000, export_timeout_millis=2000,
))
trace.set_tracer_provider(trace_provider)
log_provider = LoggerProvider(resource=resource)
log_provider.add_log_record_processor(BatchLogRecordProcessor(
_LogExporter(endpoint=endpoint + "/v1/logs", timeout=2),
max_queue_size=512, max_export_batch_size=64,
schedule_delay_millis=1000, export_timeout_millis=2000,
))
set_logger_provider(log_provider)
_event_logger.addHandler(LoggingHandler(level=logging.INFO, logger_provider=log_provider))
reader = PeriodicExportingMetricReader(
_MetricExporter(endpoint=endpoint + "/v1/metrics", timeout=2),
export_interval_millis=5000, export_timeout_millis=2500,
)
# Boundaries are seconds. Quantiles are computed over merged replica buckets.
boundaries = (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120)
metric_provider = MeterProvider(resource=resource, metric_readers=[reader], views=[
View(instrument_type=metrics.Histogram,
aggregation=ExplicitBucketHistogramAggregation(boundaries)),
])
metrics.set_meter_provider(metric_provider)
_providers.extend((metric_provider, log_provider, trace_provider))
def shutdown() -> None:
"""Flush bounded background buffers on normal termination (not SIGKILL)."""
for provider in _providers:
provider.shutdown()
_providers.clear()
def install_termination_handler() -> None:
"""Turn worker/publisher SIGTERM into cancellation so their finally runs."""
import asyncio
loop = asyncio.get_running_loop()
task = asyncio.current_task()
if task is not None:
try:
loop.add_signal_handler(signal.SIGTERM, task.cancel)
except NotImplementedError: # Windows host; Compose containers use POSIX.
signal.signal(signal.SIGTERM, lambda *_: loop.call_soon_threadsafe(task.cancel))
def emit(event: str, *, severity: int = logging.INFO, **fields) -> None:
"""Call with a source-defined event and technical fields, never free text."""
current = trace.get_current_span().get_span_context()
attrs = {key: value for key, value in fields.items()
if key in _ALLOWED_FIELDS and value is not None}
attrs.update(event=event, trace_id=f"{current.trace_id:032x}", span_id=f"{current.span_id:016x}")
_event_logger.log(severity, event, extra=attrs)
@contextmanager
def span(name: str, attributes=None, kind=SpanKind.INTERNAL, parent=None):
"""Safe span helper: never export exception messages or stack traces."""
with tracer.start_as_current_span(
name, attributes=attributes, kind=kind, context=parent,
record_exception=False, set_status_on_exception=False,
) as active:
try:
yield active
except BaseException as exc:
active.set_attribute("exception_type", type(exc).__name__)
active.set_status(Status(StatusCode.ERROR))
raise
class HttpTelemetryMiddleware:
"""Outer ASGI wrapper sees final status, streaming duration and 500 responses."""
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] != "http":
return await self.app(scope, receive, send)
from starlette.requests import Request
from app.errors import common_headers, incoming_request_id
request = Request(scope)
request.state.request_id = incoming_request_id(request)
method = scope.get("method", "OTHER")
if method not in {"GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"}:
method = "OTHER"
status = 500
started = time.perf_counter()
# Only W3C propagation fields are consumed; headers are never exported.
parent = extract(dict(request.headers))
with span("HTTP request", kind=SpanKind.SERVER, parent=parent) as active:
async def observed_send(message):
nonlocal status
if message["type"] == "http.response.start":
status = message["status"]
headers = dict(message.get("headers", []))
for key, value in common_headers(request).items():
headers[key.lower().encode()] = value.encode()
headers[b"x-trace-id"] = f"{active.get_span_context().trace_id:032x}".encode()
message = {**message, "headers": list(headers.items())}
await send(message)
try:
await self.app(scope, receive, observed_send)
finally:
route = getattr(scope.get("route"), "path", "unmatched")
duration = time.perf_counter() - started
attrs = {"method": method, "route": route, "status_class": f"{status // 100}xx"}
active.update_name(f"{method} {route}")
active.set_attributes({**attrs, "http.response.status_code": status})
if status >= 500:
active.set_status(Status(StatusCode.ERROR))
http_requests.add(1, attrs)
http_duration.record(duration, attrs)
emit("http.request.complete", severity=logging.ERROR if status >= 500 else logging.INFO,
request_id=request.state.request_id, method=method, route=route,
status_code=status, outcome="error" if status >= 500 else "success",
duration_seconds=duration)
+51 -4
View File
@@ -8,6 +8,7 @@ import logging
from pathlib import Path
import sys
import tempfile
import time
from uuid import UUID, uuid4
from sqlalchemy import text
@@ -18,6 +19,9 @@ from app.pipeline import (IMAGE_TIMEOUT, LEASE_SECONDS, TOTAL_TIMEOUT,
arm_from_env, broker_channel, crash_if_armed,
event, parse_envelope, retry_or_fail)
from app.storage import ObjectMissing, storage
from app.telemetry import (configure, emit, install_termination_handler, jobs_duration,
jobs_processed, shutdown, span)
from opentelemetry.trace import SpanKind, Status, StatusCode
logger = logging.getLogger("messenger.worker")
@@ -145,20 +149,21 @@ async def finish(row: dict, *, variants=None, code: str | None = None,
return True
async def handle(message) -> None:
async def _handle(message) -> str:
try:
job = parse_envelope(message.body)
except (ValueError, TypeError, KeyError, AttributeError):
logger.warning("invalid envelope sent to durable DLQ")
await message.reject(requeue=False)
return
return "invalid"
row = await claim(job)
if row is None:
# A ready/failed/stale attempt is a durable no-op. A duplicate of
# active processing is covered by the DB lease recovery, so it too
# can be acknowledged without running a second concurrent decoder.
await message.ack()
return
return "duplicate"
outcome = "error"
try:
async with asyncio.timeout(TOTAL_TIMEOUT):
variants = await build_objects(row)
@@ -172,12 +177,50 @@ async def handle(message) -> None:
if await finish(row, variants=variants):
await crash_if_armed("crash_after_commit_before_ack")
logger.info("ready attachment=%s", row["id"])
outcome = "success"
else:
outcome = "duplicate"
# If finish failed, the exception propagates and the connection closes:
# RabbitMQ redelivers while PG keeps the durable lease/retry intention.
await message.ack()
return outcome
async def handle(message) -> None:
# Technical IDs in logs/spans only; metric dimensions stay low-cardinality.
job = {}
try:
job = parse_envelope(message.body)
except (ValueError, TypeError, KeyError, AttributeError):
pass
started = time.perf_counter()
outcome = "error"
with span("attachments process", kind=SpanKind.CONSUMER, attributes={
"messaging.system": "rabbitmq", "messaging.destination.name": "attachments",
"messaging.operation": "process", "job_type": "attachment.process.v1",
}) as active:
try:
outcome = await _handle(message)
finally:
duration = time.perf_counter() - started
attrs = {"job_type": "attachment.process.v1", "outcome": outcome}
jobs_processed.add(1, attrs)
if outcome in {"success", "error"}:
jobs_duration.record(duration, attrs)
active.set_attribute("outcome", outcome)
if outcome in {"error", "invalid"}:
active.set_status(Status(StatusCode.ERROR))
for name in ("job_id", "attachment_id", "attempt"):
if name in job:
active.set_attribute(name, job[name])
emit("job.complete", severity=logging.WARNING if outcome == "error" else logging.INFO,
**attrs, job_id=job.get("job_id"), attachment_id=job.get("attachment_id"),
attempt=job.get("attempt"), duration_seconds=duration)
async def main() -> None:
configure("messenger-worker")
install_termination_handler()
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
for name in ("aio_pika", "aiormq"):
logging.getLogger(name).setLevel(logging.CRITICAL)
@@ -204,7 +247,11 @@ async def main() -> None:
await connection.close()
await storage.close()
await engine.dispose()
await asyncio.to_thread(shutdown)
if __name__ == "__main__":
asyncio.run(main())
try:
asyncio.run(main())
except asyncio.CancelledError:
pass # SIGTERM has run finally/SDK flush; exit cleanly.
+92 -1
View File
@@ -1,5 +1,5 @@
#file: noinspection ComposeUnknownValues
# Lab5: API x2 -> durable S3 staging + PostgreSQL outbox -> publisher
# Lab6: API x2 -> durable S3 staging + PostgreSQL outbox -> publisher
# -> RabbitMQ -> worker -> final private S3 objects + PostgreSQL metadata.
# Start: see REPORT.md, "Запуск из чистого clone".
name: messenger-lab6
@@ -53,6 +53,12 @@ x-api-env: &api-env
WORKER_FAULT: "${WORKER_FAULT:-}"
# API fault-injection is disabled during normal operation.
FAULT_INJECTION: "${FAULT_INJECTION:-}"
OTEL_SDK_DISABLED: "${OTEL_SDK_DISABLED:-false}"
OTEL_SERVICE_NAME: messenger-api
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
OTEL_RESOURCE_ATTRIBUTES: service.version=1.6.0,deployment.environment.name=lab
OTEL_METRIC_EXPORT_INTERVAL: "5000"
services:
db:
@@ -252,6 +258,7 @@ services:
command: ["python", "-m", "app.jobs"]
environment:
<<: *api-env
OTEL_SERVICE_NAME: messenger-publisher
BROKER_URL_FILE: /run/secrets/broker_url
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key, broker_url]
networks: [backend]
@@ -271,6 +278,7 @@ services:
command: ["python", "-m", "app.worker"]
environment:
<<: *api-env
OTEL_SERVICE_NAME: messenger-worker
BROKER_URL_FILE: /run/secrets/broker_url
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key, broker_url]
networks: [backend]
@@ -307,6 +315,84 @@ services:
resources:
limits: { cpus: "0.5", memory: 128M }
# Telemetry is deliberately absent from API depends_on/readiness.
# No OTLP ports on the host: application traffic stays in backend.
otel-collector:
image: otel/opentelemetry-collector-contrib:0.111.0
command: ["--config=/etc/otelcol/collector.yaml"]
volumes:
- ./infra/otel/collector.yaml:/etc/otelcol/collector.yaml:ro
networks: [backend]
restart: unless-stopped
deploy:
resources:
limits: { cpus: "0.5", memory: 256M }
prometheus:
image: prom/prometheus:v2.55.0
command:
- --config.file=/etc/prometheus/prometheus.yaml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=7d
- --storage.tsdb.retention.size=512MB
volumes:
- ./infra/prometheus:/etc/prometheus:ro
- prometheus_data:/prometheus
ports: ["127.0.0.1:${PROMETHEUS_PORT:-9090}:9090"]
networks: [backend, observability]
restart: unless-stopped
deploy:
resources:
limits: { cpus: "0.5", memory: 512M }
loki:
image: grafana/loki:3.2.1
command: ["-config.file=/etc/loki/loki.yaml"]
volumes:
- ./infra/loki/loki.yaml:/etc/loki/loki.yaml:ro
- loki_data:/loki
ports: ["127.0.0.1:${LOKI_PORT:-3100}:3100"]
networks: [backend, observability]
restart: unless-stopped
deploy:
resources:
limits: { cpus: "0.5", memory: 384M }
tempo:
image: grafana/tempo:2.6.1
# A new named volume needs write access for the local evaluation backend.
user: "0:0"
command: ["-config.file=/etc/tempo/tempo.yaml"]
volumes:
- ./infra/tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
- tempo_data:/var/tempo
ports: ["127.0.0.1:${TEMPO_PORT:-3200}:3200"]
networks: [backend, observability]
restart: unless-stopped
deploy:
resources:
limits: { cpus: "0.5", memory: 384M }
grafana:
image: grafana/grafana:11.2.2
environment:
GF_AUTH_ANONYMOUS_ENABLED: "true"
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
GF_AUTH_DISABLE_LOGIN_FORM: "true"
GF_ANALYTICS_REPORTING_ENABLED: "false"
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES: "false"
volumes:
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
- grafana_data:/var/lib/grafana
ports: ["127.0.0.1:${GRAFANA_PORT:-3000}:3000"]
networks: [backend, observability]
restart: unless-stopped
deploy:
resources:
limits: { cpus: "0.5", memory: 256M }
# Runs the public and own checks in Linux (needed on Windows: the hooks are
# POSIX .sh scripts that call the docker CLI). Profile "tools": not started
# by a plain "up". The Docker socket grants control over the Docker host —
@@ -328,6 +414,7 @@ services:
networks: [edge, backend]
networks:
observability: {}
edge: {}
# PostgreSQL, Valkey and S3: no route outside, the LB is not attached.
backend:
@@ -337,6 +424,10 @@ volumes:
postgres_data: {}
s3_data: {}
broker_data: {}
prometheus_data: {}
loki_data: {}
tempo_data: {}
grafana_data: {}
secrets:
postgres_password:
+328
View File
@@ -0,0 +1,328 @@
{
"uid": "messenger-lab6",
"title": "Messenger lab6 - Observability",
"description": "Application OTLP metrics, structured logs and traces. HTTP error ratio is business 5xx / all business responses; health excluded. Duration units are seconds. All API replicas are aggregated.",
"tags": [
"lab6",
"opentelemetry"
],
"schemaVersion": 39,
"version": 1,
"editable": false,
"timezone": "browser",
"refresh": "5s",
"time": {
"from": "now-15m",
"to": "now"
},
"panels": [
{
"id": 1,
"type": "timeseries",
"title": "Business API requests / second",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval]))",
"legendFormat": ""
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 2,
"type": "timeseries",
"title": "Business API 5xx / all responses",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[$__rate_interval])) / clamp_min(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])), 1e-9)",
"legendFormat": ""
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 3,
"type": "timeseries",
"title": "API mean duration (seconds)",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(rate(messenger_http_duration_seconds_sum{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])) / clamp_min(sum(rate(messenger_http_duration_seconds_count{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])), 1e-9)",
"legendFormat": ""
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 4,
"type": "timeseries",
"title": "Worker attempt outcomes / second",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum by (outcome) (rate(messenger_jobs_processed_total{service_name=\"messenger-worker\"}[$__rate_interval]))",
"legendFormat": "{{outcome}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 5,
"type": "timeseries",
"title": "Worker mean attempt duration (seconds)",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(rate(messenger_jobs_duration_seconds_sum{service_name=\"messenger-worker\"}[$__rate_interval])) / clamp_min(sum(rate(messenger_jobs_duration_seconds_count{service_name=\"messenger-worker\"}[$__rate_interval])), 1e-9)",
"legendFormat": ""
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 6,
"type": "timeseries",
"title": "Worker completed attempts (counter)",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum by (outcome) (messenger_jobs_processed_total{service_name=\"messenger-worker\"})",
"legendFormat": "{{outcome}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 7,
"type": "logs",
"title": "Structured application events (expand for trace_id / span_id / request_id)",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"x": 0,
"y": 24,
"w": 24,
"h": 9
},
"targets": [
{
"refId": "A",
"expr": "{service_name=~\"messenger-api|messenger-worker|messenger-publisher\"}",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": false,
"wrapLogMessage": true,
"enableLogDetails": true,
"sortOrder": "Descending"
}
},
{
"id": 8,
"type": "table",
"title": "Recent real API / worker traces (open TraceID)",
"datasource": {
"type": "tempo",
"uid": "tempo"
},
"gridPos": {
"x": 0,
"y": 33,
"w": 24,
"h": 8
},
"targets": [
{
"refId": "A",
"queryType": "traceql",
"query": "{ resource.service.name = \"messenger-api\" || resource.service.name = \"messenger-worker\" }",
"limit": 20,
"tableType": "traces"
}
],
"options": {
"showHeader": true
}
}
],
"annotations": {
"list": []
},
"templating": {
"list": []
}
}
@@ -0,0 +1,11 @@
apiVersion: 1
providers:
- name: messenger
orgId: 1
folder: Messenger
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: false
options:
path: /var/lib/grafana/dashboards
@@ -0,0 +1,39 @@
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
jsonData:
timeInterval: 5s
- name: Loki
uid: loki
type: loki
access: proxy
url: http://loki:3100
editable: false
jsonData:
derivedFields:
- name: TraceID
matcherRegex: '"trace_id"\s*:\s*"([0-9a-f]{32})"'
datasourceUid: tempo
url: '$${__value.raw}'
- name: Tempo
uid: tempo
type: tempo
access: proxy
url: http://tempo:3200
editable: false
jsonData:
tracesToLogsV2:
datasourceUid: loki
spanStartTimeShift: -5m
spanEndTimeShift: 5m
filterByTraceID: true
filterBySpanID: false
tags: [{ key: service.name, value: service_name }]
nodeGraph:
enabled: true
+39
View File
@@ -0,0 +1,39 @@
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
allow_structured_metadata: true
retention_period: 168h
# Bound cardinality: request IDs / trace IDs remain structured metadata.
otlp_config:
resource_attributes:
ignore_defaults: true
attributes_config:
- action: index_label
attributes: [service.name, deployment.environment.name]
compactor:
working_directory: /loki/compactor
retention_enabled: true
delete_request_store: filesystem
analytics:
reporting_enabled: false
+69
View File
@@ -0,0 +1,69 @@
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 192
spike_limit_mib: 48
batch:
timeout: 1s
send_batch_size: 256
send_batch_max_size: 512
exporters:
prometheus:
endpoint: 0.0.0.0:8889
metric_expiration: 5m
resource_to_telemetry_conversion:
enabled: true
otlphttp/tempo:
endpoint: http://tempo:4318
timeout: 3s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 256
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 5s
max_elapsed_time: 30s
otlphttp/loki:
# otlphttp appends /v1/logs; Loki 3.2 accepts native OTLP here.
endpoint: http://loki:3100/otlp
timeout: 3s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 256
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 5s
max_elapsed_time: 30s
service:
telemetry:
logs:
level: warn
metrics:
address: 0.0.0.0:8888
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/tempo]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/loki]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
+14
View File
@@ -0,0 +1,14 @@
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: messenger
static_configs:
- targets: [otel-collector:8889]
- job_name: collector
static_configs:
- targets: [otel-collector:8888]
- job_name: prometheus
static_configs:
- targets: [localhost:9090]
+25
View File
@@ -0,0 +1,25 @@
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
ingester:
trace_idle_period: 5s
max_block_duration: 1m
compactor:
compaction:
block_retention: 168h
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
local:
path: /var/tempo/blocks
usage_report:
reporting_enabled: false
+5
View File
@@ -13,3 +13,8 @@ Pillow==11.0.0
# Lab5: durable RabbitMQ messages and publisher confirmations.
aio-pika==9.4.3
# Explicit OTLP/HTTP SDK; semantic conventions bundled as 0.49b2.
opentelemetry-api==1.28.2
opentelemetry-sdk==1.28.2
opentelemetry-exporter-otlp-proto-http==1.28.2