Commit Graph
14 Commits
Author SHA1 Message Date
Artyom 202b985dea Add reproducible observability cost modes and external measurements 2026-11-06 21:22:38 +03:00
Artyom 20e7759f87 Finalize grade 4 verification and record observed limitations 2026-11-06 21:18:21 +03:00
Artyom 0fc02496fe Archive grade 4 traces, alerts and bounded outage evidence 2026-11-06 21:17:45 +03:00
Artyom 44d81fd80e Preserve distributed job context and expose grade 4 operational signals 2026-11-06 12:57:10 +03:00
Artyom 1a171d0d0f Archive verified grade 3 observability requirements 2026-11-06 12:47:37 +03:00
Artyom 2903d9dc73 Verify the grade 3 telemetry stack and inherited API guarantees 2026-11-06 12:41:06 +03:00
Artyom 95511002b1 Archive the lab 5 foundation forwarding change 2026-11-06 12:30:02 +03:00
Artyom 12db77133f Forward the lab 5 foundation while preserving lab 6 telemetry
Source: immutable lab5 snapshot f88e9ba43cdfd7eb6a5a7ae5feb4247b4e820071; forward the allowed diff from 64d5c67. Carry SQLAlchemy, Alembic, reliability fixes, versioned jobs, infrastructure and student tests without changing protected coursework files.

Validation: make check, validate, lint; 515 student tests passed with one Linux-only skip; public smoke with telemetry enabled and repeat migration passed. Stop the owned stack and retain volumes. Preserve the existing OTel work in progress; grade implementation remains a separate phase.
2026-11-06 12:29:35 +03:00
Artyom dc9256baf9 Propose forwarding the lab 5 foundation as an openspec change 2026-11-06 12:17:32 +03:00
ArtyomandClaude Opus 5.5 a8836ce8b5 Instrument the API, the relay and the worker
Start the telemetry in create_app, which gunicorn runs in every worker
after fork because preload_app is off, and in the relay and worker
processes with their own service names, and stop it after the rest of
the shutdown within TELEMETRY_SHUTDOWN_SECONDS. The API gets server
spans and the http.server.request.duration histogram in seconds from
the FastAPI instrumentation with stable HTTP semantic conventions,
without /health probes and without the internal send and receive
spans, and the request pipeline puts X-Request-Id on the server span
as messenger.request_id. The worker broker gets the FastStream
RabbitTelemetryMiddleware, so every delivery runs inside a consumer
span.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 04:45:26 +03:00
ArtyomandClaude Opus 5.5 e2ec3fe9af Count and trace attachment jobs through a job observer port
Add the JobObserver port to the attachments zone and its adapter on the
OpenTelemetry API: every delivery adds one to messenger.jobs.processed
and records messenger.job.duration in seconds with buckets from 50 ms to
60 s, both labelled only by job.type and outcome, and puts the job id,
the attachment id, the attempt and the outcome on the current consumer
span, which turns to ERROR on retry_scheduled, failed and error. The
worker entrypoint measures each delivery, reports the outcome of the
handler or error when it raised, and writes a job.processed record.
Without the SDK the adapter runs on the no-op providers of the API.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 04:45:20 +03:00
ArtyomandClaude Opus 5.5 e60b4eac98 Start the OpenTelemetry SDK in every process
Add the OpenTelemetry API and SDK, the OTLP over HTTP exporter and the
FastAPI and asyncpg instrumentations to the lock, and a platform module
that builds one resource per process (service.name, service.version,
service.instance.id as hostname and pid, deployment.environment.name
from OTEL_RESOURCE_ATTRIBUTES), batch processors for traces and logs,
a periodic metric reader, PostgreSQL client spans and a bounded
shutdown in a daemon thread instead of the 30 s flush of the SDK. The
SDK stays off without a Collector address or with OTEL_SDK_DISABLED.

The sampler lets only server, consumer and producer spans open a
trace, so readiness probes, outbox polls and sweeps do not become
one-span traces. Log records carry trace_id and span_id in stdout and
go to OTLP through a handler that keeps only the allowlisted fields,
and deferred event subscribers now run in the context the event was
emitted in, so audit records stay linked to their request or job.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 04:45:14 +03:00
ArtyomandClaude Opus 5.5 a41d6d39e8 Propose the telemetry pipeline as an openspec change
Add the change add-lab6-telemetry-pipeline for level 3 of lab 6: every
API worker process, the relay and the worker start the OpenTelemetry
SDK after fork with service.name, service.version, a per-process
service.instance.id and deployment.environment.name, export OTLP over
HTTP with batch processors only, record HTTP server spans and the
seconds histogram by route template, consumer spans and the
jobs.processed counter and job.duration histogram by job type and
outcome, PostgreSQL client spans inside requests and jobs, and send
every allowlisted log record to Loki with trace_id and span_id. The
stand gains the Collector, Prometheus, Loki, Tempo and Grafana with
volumes and provisioning from Git, built from the verified .prep/lab6
configuration.

The specs add lab6-service-telemetry and lab6-telemetry-stack and
extend the unified log requirement of lab1-service-operability.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 04:37:41 +03:00
ArtyomandClaude Opus 5.5 a0d3947710 Carry over the agent context and openspec history from lab 5
Bring the Claude Code settings, the lint hook, the opsx commands and
skills, the architecture notes, the lab 1 to lab 5 specs and the
archived lab 1 to lab 5 changes into the lab 6 skeleton, so lab 6
continues the same spec history instead of starting from scratch. The
files come from the lab 5 solution commit 64d5c67.

Rewrite AGENTS.md and the openspec context and rules for lab 6: the
business API stays at contract 1.6.0, API, relay and worker start the
OpenTelemetry SDK in every process after fork and export OTLP over HTTP
to a separate Collector, Prometheus, Loki and Tempo store metrics, logs
and traces on volumes, Grafana only reads them through provisioning
kept in Git, telemetry never blocks a request and never feeds
/health/ready, metric labels stay low-cardinality and message bodies
and secrets never leave the process. Out of scope moves to lab 7,
track 2 and clustered telemetry backends.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 03:46:20 +03:00