Application / Стенд Compose с RabbitMQ, relay и worker за HAProxy, smoke, pipeline, отказ реплики, рестарт и отказы зависимостей (pull_request) Canceled after 0s
Document grade 3 and 4 evidence, the observability cost study, and review limitations. Separate historical measurements and record static report validation without rerunning Docker workloads.
Source: immutable lab5 snapshot f88e9ba43cdfd7eb6a5a7ae5feb4247b4e820071; forward the allowed diff from 64d5c67. Carry SQLAlchemy, Alembic, reliability fixes, versioned jobs, infrastructure and student tests without changing protected coursework files.
Validation: make check, validate, lint; 515 student tests passed with one Linux-only skip; public smoke with telemetry enabled and repeat migration passed. Stop the owned stack and retain volumes. Preserve the existing OTel work in progress; grade implementation remains a separate phase.
Start the telemetry in create_app, which gunicorn runs in every worker
after fork because preload_app is off, and in the relay and worker
processes with their own service names, and stop it after the rest of
the shutdown within TELEMETRY_SHUTDOWN_SECONDS. The API gets server
spans and the http.server.request.duration histogram in seconds from
the FastAPI instrumentation with stable HTTP semantic conventions,
without /health probes and without the internal send and receive
spans, and the request pipeline puts X-Request-Id on the server span
as messenger.request_id. The worker broker gets the FastStream
RabbitTelemetryMiddleware, so every delivery runs inside a consumer
span.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add the JobObserver port to the attachments zone and its adapter on the
OpenTelemetry API: every delivery adds one to messenger.jobs.processed
and records messenger.job.duration in seconds with buckets from 50 ms to
60 s, both labelled only by job.type and outcome, and puts the job id,
the attachment id, the attempt and the outcome on the current consumer
span, which turns to ERROR on retry_scheduled, failed and error. The
worker entrypoint measures each delivery, reports the outcome of the
handler or error when it raised, and writes a job.processed record.
Without the SDK the adapter runs on the no-op providers of the API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add the OpenTelemetry API and SDK, the OTLP over HTTP exporter and the
FastAPI and asyncpg instrumentations to the lock, and a platform module
that builds one resource per process (service.name, service.version,
service.instance.id as hostname and pid, deployment.environment.name
from OTEL_RESOURCE_ATTRIBUTES), batch processors for traces and logs,
a periodic metric reader, PostgreSQL client spans and a bounded
shutdown in a daemon thread instead of the 30 s flush of the SDK. The
SDK stays off without a Collector address or with OTEL_SDK_DISABLED.
The sampler lets only server, consumer and producer spans open a
trace, so readiness probes, outbox polls and sweeps do not become
one-span traces. Log records carry trace_id and span_id in stdout and
go to OTLP through a handler that keeps only the allowlisted fields,
and deferred event subscribers now run in the context the event was
emitted in, so audit records stay linked to their request or job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add the change add-lab6-telemetry-pipeline for level 3 of lab 6: every
API worker process, the relay and the worker start the OpenTelemetry
SDK after fork with service.name, service.version, a per-process
service.instance.id and deployment.environment.name, export OTLP over
HTTP with batch processors only, record HTTP server spans and the
seconds histogram by route template, consumer spans and the
jobs.processed counter and job.duration histogram by job type and
outcome, PostgreSQL client spans inside requests and jobs, and send
every allowlisted log record to Loki with trace_id and span_id. The
stand gains the Collector, Prometheus, Loki, Tempo and Grafana with
volumes and provisioning from Git, built from the verified .prep/lab6
configuration.
The specs add lab6-service-telemetry and lab6-telemetry-stack and
extend the unified log requirement of lab1-service-operability.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bring the application, own tests, Dockerfile, toolchain, compose stack,
HAProxy, k6, RustFS and RabbitMQ configuration, migrations, hooks,
scenario scripts and the CI job from the lab 5 solution commit 64d5c67,
keep the lab 5 Makefile targets on top of the unchanged skeleton
targets, and append the lab 5 solution variables to the skeleton
.env.example. The package and the application follow the contract
version 1.6.0, and the image, the throwaway test containers and the
broker vhost description are tagged lab6 so they do not collide with a
lab 5 stand on the same host.
This is the starting point of the lab 6 cycle: lint, own tests and the
offline skeleton check pass before any lab 6 change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bring the Claude Code settings, the lint hook, the opsx commands and
skills, the architecture notes, the lab 1 to lab 5 specs and the
archived lab 1 to lab 5 changes into the lab 6 skeleton, so lab 6
continues the same spec history instead of starting from scratch. The
files come from the lab 5 solution commit 64d5c67.
Rewrite AGENTS.md and the openspec context and rules for lab 6: the
business API stays at contract 1.6.0, API, relay and worker start the
OpenTelemetry SDK in every process after fork and export OTLP over HTTP
to a separate Collector, Prometheus, Loki and Tempo store metrics, logs
and traces on volumes, Grafana only reads them through provisioning
kept in Git, telemetry never blocks a request and never feeds
/health/ready, metric labels stay low-cardinality and message bodies
and secrets never leave the process. Out of scope moves to lab 7,
track 2 and clustered telemetry backends.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>