10 KiB
Checkpoint update: the approved559001001-byte Grafana/plugin download, native18-panel UI/42-span links, bounded response contract and real DB/worker alerts now have evidence in ../../REPORT.md. Historical preparation/gate wording below is retained as provenance; fresh-volume/source/package completion is tracked in ../../REPORT.md.
OpenObserve migration checkpoint
The owner approved replacing Prometheus, Loki and Tempo with OpenObserve. Grafana and the separate OTel Collector remain mandatory. The authoritative shared boundary is ../reference/OPENOBSERVE-BOUNDARY.md. Backend95 stays byte-identical; its native OTLP HTTP/protobuf interface does not depend on the former storage stack.
The implementation uses one isolated OpenObserve single-node instance with a persistent local data/metadata volume. This is an explicitly temporary engineering default. It does not reuse Azur credentials, S3, Litestream, Kubernetes resources or runtime metadata. The application database, broker and S3 remain separate failure domains. The existing core application and historical VP5 archive remain available.
Preserved baseline
Before changes, 49 configuration, helper, report and evidence files were copied and SHA-256 verified under .local/history/pre-openobserve-2026-11-15T224455797Z. This private snapshot is historical, not evidence for the new stack. Existing backend source95, VP5 ZIP and lab7 source112 are outside this migration's write boundary.
Case table before implementation
| Situation or input | Concrete example | Observable expected result | Level |
|---|---|---|---|
| Valid all-signal ingestion | Actual current API request and worker image processing | Collector delivers metrics, structured logs and related API/producer/consumer/worker spans to isolated OpenObserve; native query content verified | Integration |
| Metric naming and units | Request counter, duration histogram, immutable readiness SLO gauges | Measured native names and units; histogram percentiles agree with controlled observations; no assumed messenger_ exporter prefix |
Integration |
| Trace field compatibility | Actual accepted OTLP spans, including nonzero worker duration and parent | Native trace columns match plugin canonical nanosecond start/end/duration path; correct hierarchy and positive duration | Integration and browser |
| Grafana drill-down | Log containing trace_id and response request_id | Native metrics panel, structured log and actual waterfall render; log-to-trace and trace-to-log resolve the same trace | Browser |
| Query identity scope | Dedicated principal's required query and forbidden write/admin path | Intended queries succeed, write/admin/other organization paths denied; root credential never provisioned to Grafana | Integration |
| Bounded query | Oversized or long-range request, excessive result limit | Explicit query/time/result and data-proxy response limits enforced, ordinary bounded query succeeds | Integration |
| High error alert | Controlled application DB failure plus healthy positive control | Actual rule engine changes normal → firing → resolved and keeps observation-availability separate | Integration |
| Queue and readiness SLO | Short/sustained worker pause and matured initial-acceptance cohort | Queue-age and two SLO multiwindow rules use measured series, fire and resolve at their declared windows | Integration |
| Telemetry storage failure | Own OpenObserve unavailable while core requests continue | Application remains available, SDK/Collector retry queues and memory stay bounded, stop finishes within deadline | Fault integration |
| Collector recovery | Delivery resumes after bounded outage | New signals arrive after recovery; losses/retries reported from measured counters, no claim of unlimited durability | Fault integration |
| Persistence | Restart only owned telemetry services with named volume retained | Previously queried synthetic application signals survive restart | Integration |
| Both API failures | Stop each owned API replica separately while history is used | Other replica serves confirmed state through load balancer; bounded failover verified for both | Integration |
| Cache identity | Exact public OCI digest and size in stopped node | Regular blob, expected size and raw SHA-256; no runtime metadata read, source mutation or new pull | Operational integration |
| Native archive boundary | Final Compose bind files and new configuration | Complete canonical archive with hashes, no .local, credentials, cache or builds |
Artifact integration |
Failures include auth rejection, native schema mismatch, exporter retry/queue overflow, unavailable storage, absent metric names, trace query truncation, failed rule evaluation and restart loss. They are tested at the actual boundary, rather than by asserting configuration text. Config parsing/dry-run and source probes are recorded separately from runtime acceptance. Storage loss during an outage is not called zero-loss without measured delivery evidence.
Current evidence and gates
Registry metadata only: OpenObserve v1.1.2 pinned index sha256:4a154849432f75c4fb024de32fe34163c02406c905dab22bf8d213d2e3386392; linux/amd64 manifest sha256:26618f379c86b253c0d91624c487dd14b71293699845ecbc8457c94dca14819f. Config and compressed layers total 161,811,141 bytes. Collector-k8s0.163.2 totals51,500,993 bytes; Grafana13.3.1 totals506,207,527 bytes. Raw index/manifest SHA-256 matches the registry descriptors. This is not a layer download or runtime PASS.
A ready-made third-party Grafana plugin is being independently reviewed by root/SII. No authored plugin, protocol shim or upstream modification is authorized. Actual OpenObserve ingestion must establish the canonical timestamp path before Grafana acceptance. OpenObserve root/admin credentials must not enter Grafana. Large new layer downloads remain gated on owner approval; current work first inspects exact known public OCI content in the stopped node read-only.
Contract6 → 7
The backend OTLP interface and source bytes remain unchanged. Collector receives HTTP/protobuf on4318 and gRPC on4317 only where explicitly bound internally; all three signals forward through native OTLP HTTP to the isolated OpenObserve organization. The exact endpoint/auth header, stream names, native metric names/units and trace columns will be emitted as a narrow measured contract only after ingestion. Lab7 keeps its TLS/mTLS gateways and updates only its own infra/provisioning. No bulk backend or lab6 configuration synchronization is permitted.
Full readiness remains gated on the runtime, Grafana, alert, persistence/failover and independent review rows above. Historical source/unit/core-runtime checks do not satisfy the replacement stack's acceptance.
Targeted parser boundary cases
After the first21 native gateway cases, independent review requested explicit duplicate JSON/query parameters, encoded parameter names, fractional ranges, SQL LIMIT versus requested size, and incomplete/chunked bodies. The expected contract rejects ambiguous duplicate protected fields/parameters and ranges beyond the declared upper bound; it must keep complete legitimate chunked JSON usable and reject oversized/incomplete bodies within the transport deadline. A benign SQL LIMIT2/query.size1 probe establishes the native engine's precedence without a large response. These are additional cases, not already covered by the first21 receipt. The current Prometheus epoch converter floors fractional seconds, so its measured effective bound must be stated or tightened; a config text comparison cannot establish that behavior.
Repeated bootstrap/render is part of the start contract: existing byte-identical dedicated credential files must be preserved and reused, and a changed credential must fail closed instead of silently rotating it. The renderer's initial exclusive-create-only behavior is exercised as RED before adding this repeat path. Provisioning captures each one-time service-account response in its owned private operation directory before readback; partial creation remains distinguishable from a fresh start.
Fresh start must not require the not-yet-rendered Grafana query environment while its profile is disabled. Grafana's env_file is therefore optional at Compose parse time; up.sh renders its dedicated account before a later explicit Grafana start. PostgreSQL readiness uses TCP127.0.0.1 rather than the temporary initialization server's Unix socket, consistent with the already confirmed adjacent lab7 initialization race. The installed PostgreSQL18.6 TCP readiness invocation is verified; current-volume startup does not by itself prove every fresh-volume path.
Independent pre-freeze review found that a Grafana process on the shared private network could bypass the query gateway using its otherwise broad native SA. The corrective contract isolates Grafana on a query-front network; only the existing HAProxy joins query-front and telemetry-data, while raw OpenObserve and Collector never join query-front. Separate existing edge membership preserves host-loopback publications; Grafana joins only the dedicated noninternal query-front network. A cached unprivileged client with the actual dedicated query credential must demonstrate the old raw read/write route as RED, then gateway-positive/raw-IP+host-loopback-published-route+Collector-negative behavior on the new network. It is a network-principal equivalence check while the Grafana image remains gated, not a claim that Grafana rendered anything.
Account bootstrap must checkpoint each received credential before creating the next account. On a retained intent, it recovers only through captured native JSON and live credential/role checks; ambiguous partial bytes fail closed without repeating account POST. Existing tokens must prove expected search200/ingest200/admin-rotation403 before canonical/derived publication or a successful start. Required-port regressions cover a second-create failure, replay from one persisted account, captured-response recovery, uncertain attempt/no-retry, stale401, unavailable503 and unexpected administration status.
Native positive-query controls on the retained workload exposed different start_time labels per OTLP instrument. SLO arithmetic must use on(service_name,service_instance_id) for both subtraction and division as well as observer matching. The actual provisioned panel16 and ready-burn were empty in RED despite matured=timely=1; after the configuration fix all17 controls are finite/nonempty with healthy ratios0. No new smoke or producer was needed.