Commit Graph
28 Commits
Author SHA1 Message Date
ricoromaandClaude Opus 5.5 b3a5ab8c2b docs: archive track1-telemetry-cost and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:17:06 +03:00
ricoromaandClaude Opus 5.5 56132c77e9 fix: address track1-telemetry-cost verify findings
Base accepting the saturated full-r3 run on the user's 2026-10-31 host
warning rule instead of the swap decision and ask whether to re-shoot;
complete the report section with limits, metric freshness at 30 s and
per-class and per-component tables; correct derived numbers and the
quantile error wording; align H6 in design with the report; fix the
quantile_error docstring; park S1 and S4 as later proposals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:12:48 +03:00
ricoromaandClaude Opus 5.5 613cceb69d docs: report telemetry cost results from baseline2
Fill the track 1 section of REPORT.md from evidence/telemetry-cost/
baseline2/summary.md: latency and CPU per variant, ingest with per-day
and 7-day projections, losses by hop, what each variant keeps, quantile
accuracy, H1-H8 and B1-B5 verdicts (B4 and H7 fail; H3 and H5 fail on
one missing slow-write trace under typical tail sampling), the
saturated full-r3 run, Tempo memory at its limit, recommendation with
conditions and limits. Point the lab 6 defense step at
bench_telemetry_quick and close tasks 12.3-12.5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 05:52:37 +03:00
ricoromaandClaude Opus 5.5 0f5dbfcf3c bench: capture telemetry cost label baseline2
21/21 runs on 68e14b6. balanced passes B1-B3 and B5, so no
decision-12 fix is needed; B4 fails (replica CPU per request +27 %,
light p99 +30.7 ms) and H7 with it. H3 and H5 fail on one missing
slow-write trace of 75 under typical tail sampling. Host swap-outs
are ignored by the user's decision of 2026-11-15.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 05:48:28 +03:00
ricoromaandClaude Opus 5.5 68e14b6ddf fix(bench): clear stale k6 console files and check telemetry storages around lookup
k6 --console-output appends, so a run killed before cleanup left
<run>.warm.k6 in .work and the resumed warmup counted 24 of 12.
Remove the console file before each k6 run and a stale .work at
start; ignore evidence/**/.work/.

A telemetry storage that exits during the lookup (Tempo in
metrics-r3: name no longer resolvable) now fails the run with the
container state, OOMKilled flag and its log tail saved to raw/;
Tempo requests of the lookup are limited to 3 at a time and
memory peaks after the lookup are recorded.

Raise the light classes of the telemetry cost load by 15 % (read
126.5, write 23, me 34.5, card 34.5 per second): on a quiet host
`full` kept the replicas at 28-29.5 % CPU, below the 30 %
calibration bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 23:42:53 +03:00
ricoromaandClaude Opus 5.5 509afca89e feat: telemetry cost profile with head sampling, coarse buckets and balanced tail sampling
- [telemetry] trace_sample_ratio (ParentBased(TraceIdRatioBased), 1 keeps
  ParentBased(ALWAYS_ON)) and metric_buckets (default|coarse with SLO bounds),
  export interval <= 30 s; otel.sdk.span.started, span.live dropped; startup
  records name ratio, buckets and interval
- bench: telemetry services on mtusi-bench, TELEMETRY=on switch for uploads
  and queue, make bench_telemetry[_quick] with off/full/full-aa/head/tail/
  metrics/balanced, rare events, lookup from Tempo/Loki/Prometheus, cost
  sampler, verdicts H1-H8 and B1-B5; collector layers and lean Prometheus in
  bench/otel with preflight checks
- REPORT: telemetry cost hypotheses and criteria before the label

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:17:25 +03:00
ricoromaandClaude Opus 5.5 37f86801d6 docs: propose track1-telemetry-cost
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 09:23:41 +03:00
ricoromaandClaude Opus 5.5 0315ee13ac docs: archive slo-alerts and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:07:04 +03:00
ricoromaandClaude Opus 5.5 af4e47b55c docs: re-capture telemetry-check evidence on 4a2f9c9
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:03:22 +03:00
ricoromaandClaude Opus 5.5 4a2f9c9304 fix: wait for all relay pass records in telemetry-check Loki comparison
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 04:58:21 +03:00
ricoromaandClaude Opus 5.5 a92dff30fd docs: re-capture slo-alerts alert and backend-outage evidence on ef60b47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 04:54:28 +03:00
ricoromaandClaude Opus 5.5 ef60b47246 fix: address slo-alerts verify findings
Count the first SDK telemetry loss on the loss panel, print pending
moments on every alerts-check step, check the loss panel on a quiet
stand, align two scenarios and the SLI cross-check text with what the
driver verifies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 04:08:13 +03:00
ricoromaandClaude Opus 5.5 107fe7f033 docs: capture slo-alerts evidence on 15ea156
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:36:17 +03:00
ricoromaandClaude Opus 5.5 15ea156404 feat: add SLOs, Prometheus alert rules and telemetry loss accounting
HAProxy /metrics scraped by the Collector, recording rules and seven alerts with promtool tests and runbooks (make alerts_test, CI step), byte-sized Collector export queues with 30 s retries, SDK internal metrics, SLO/LB/pipeline dashboard rows, driver commands alerts-check and backend-outage (change slo-alerts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:35:01 +03:00
ricoromaandClaude Opus 5.5 839433c23b docs: propose slo-alerts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:07:18 +03:00
ricoromaandClaude Opus 5.5 5f562dc646 docs: archive job-trace-propagation and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:04:40 +03:00
ricoromaandClaude Opus 5.5 c1055b4d6c docs: re-capture trace-chain and collector-outage evidence on 72e2598
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:02:20 +03:00
ricoromaandClaude Opus 5.5 72e2598d28 fix: address job-trace-propagation verify findings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:53:13 +03:00
ricoromaandClaude Opus 5.5 eaa1161cad docs: capture job-trace-propagation evidence on 72cfa9a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:16:24 +03:00
ricoromaandClaude Opus 5.5 72cfa9aaf6 feat: propagate trace context through outbox and add pipeline saturation metrics
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:54:37 +03:00
ricoromaandClaude Opus 5.5 ff0073e8e3 docs: propose job-trace-propagation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:38:05 +03:00
ricoromaandClaude Opus 5.5 6b1ad631e9 docs: archive otel-telemetry and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:31:22 +03:00
ricoromaandClaude Opus 5.5 f9642864c4 docs: re-capture telemetry budget on 4178b3d
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:22:19 +03:00
ricoromaandClaude Opus 5.5 4178b3df3e fix: raise tempo and grafana memory limits to measured peaks
Peaks under make test + load + student_test left Tempo and Grafana below
1.5x headroom; re-capture telemetry-check and collector-outage evidence on
68f1e97.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:12:40 +03:00
ricoromaandClaude Opus 5.5 68f1e9794e fix: address otel-telemetry verify findings
Shutdown budget leaves room for SDK shutdown, stand-from-scratch accepts only
its throwaway project, OTLP exporters ignore proxy environment, telemetry-check
covers log/trace links, health series and worker instance id; spec deltas
match what the parser and SDK lifetime allow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:54:14 +03:00
ricoromaandClaude Opus 5.5 a4497e3f32 feat: add OpenTelemetry SDK, collector and telemetry stores
Collector (contrib), Prometheus, Loki, Tempo and Grafana with provisioning
from git; SDK in API and worker with level 3 metrics, traces and logs
carrying trace_id/span_id; telemetry off by default outside the stand;
broker healthcheck runs as the broker user so a fresh stand starts once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:19:50 +03:00
ricoromaandClaude Opus 5.5 9b35c256b7 docs: propose otel-telemetry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 05:51:59 +03:00
ricoromaandClaude Opus 5.5 9cc6c0c7a6 chore: port lab 5 solution into the lab 6 skeleton
Source: mtusi2026lab5-safronov, solution/bvt2301-safronov, 046ec9e.
Contracts, public tests and evidence of lab 5 are not carried over.
The compose project, image and CI prefix are renamed to lab6; the first
broker start defect is recorded in KNOWN_ISSUES.md; PLAN.md outlines the
OpenSpec changes of lab 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 04:46:56 +03:00