Files
mtusi2026lab6/docs/reference/OPENOBSERVE-BOUNDARY.md

9.4 KiB

OpenObserve migration boundary

User scope: replace Prometheus/Loki/Tempo storage in lab6 and lab7. Keep the original mandatory OTel Collector, Grafana, OpenBao, business API, PostgreSQL, Valkey, S3, worker and security criteria. Existing immutable archives/receipts remain historical; new configuration requires its own acceptance.

Ownership

  • Lab6 author: builtin VP, including its isolated remote6 and vp/STATUS.md.
  • Lab7 author: HC t67, infrastructure only; backend112 and protected19 remain byte-identical.
  • Root: this boundary, cross-subject plan and native Grafana integration acceptance.
  • Independent source reviewer: builtin SII; no implementation writes.

Shared configuration defaults

Use one isolated single-node OpenObserve instance per lab, persistent local data/metadata volume independent of application PostgreSQL. This is a temporary engineering default for the coursework, not the S3/Litestream topology of azur-games. Do not add NATS/etcd or copy Azur credentials/configuration blindly. No telemetry data migration is required from the not-yet-run old storage stack; retain historical source and manifests.

API and worker keep their current OTLP HTTP/protobuf interface and W3C propagation. The Collector forwards logs, metrics and traces through native OTLP to OpenObserve. Authentication is file/environment based outside Git, images and diagnostics. No OpenObserve secret belongs to application code or API/worker OpenBao KV. Start with the isolated native default organization; record exact stream names and native normalized metric names after live ingestion instead of assuming the old Prometheus exporter namespace/suffixes.

Native lab6 export confirmed that the Collector file provider preserves terminal newlines in a header scalar. Generated Authorization header files must contain no CR/LF; configuration validation alone did not detect the invalid wire header. Keep prior failed bytes private and validate authenticated positive export after writing the corrected file.

Lab6 implements the shared OTLP routes/field names and sends a narrow checked artifact contract to lab7. Lab7 retains its TLS/mTLS gateway architecture and updates the declared telemetry matrix, identity allow/deny tests and any actual provisioning consumer. Do not bulk-copy backend or all lab6 configs into lab7.

Grafana query path

Grafana remains in both labs. Metrics use its native Prometheus data source against the documented OpenObserve Prometheus-compatible organization endpoint. Logs and trace rendering use an existing OpenObserve data source plugin, not an authored protocol shim/plugin.

The official openobserve/openobserve-grafana-plugin current source confirms logs/metrics, not a native trace frame. The ready-made third-party Windforce17/openobserve-grafana-plugin v1.0.1 candidate is pinned to commit e10214739827dda9318705d49db262b28d67bcfe, Apache-2.0, requires Grafana>=12.4.0 and is unsigned. It is not an official OpenObserve release. Its actual frontend trace_id path creates a canonical Grafana trace frame and internal log/trace links; its generic backend table frames are not a substitute for that frontend route. Independent authentication/limits review and a native Grafana rendering check remain required before accepting it.

Independent source probes confirm the canonical nanosecond span path, but exposed conditional defects in the microsecond _timestamp fallback (factor1000) and ignored custom duration-field setting. Inspect actual native OpenObserve trace columns and enforce a correctly rendered canonical start/end/duration path; do not accept an empty or zero-duration waterfall and do not add a custom shim or silently patch the dependency. Bound Grafana data-proxy response size/time and actual query results as well as the Collector.

The pinned OpenObserve1.1.2 decoder (commit d16d693ba99bc20621a58b948ecbcad7ec7fa882) preserves native start_time/end_time in nanoseconds, _timestamp and duration in microseconds. The plugin's normal native start/end path is source-proven compatible; inspect real rows before acceptance. Its optional duration-search shortcut applies nanoseconds to the native microsecond duration column; do not use that shortcut for acceptance. Native SQL such as duration > 5000 means greater than5ms and can be used through the existing SQL mode when such a filter is needed. No upstream code patch is part of this migration.

Provisioning from the actual pinned plugin consumers: data source type: openobserve, top-level URL/access proxy, basicAuth: true, basicAuthUser and secureJsonData.basicAuthPassword. The declared TypeScript apiKey is not read by the backend. Actual jsonData keys include timestamp_column, default_organization, default_log_stream, default_trace_stream, log_message_field, trace_id_field, span_id_field, service_name_field, span_name_field and status_field; there are no configurable start/end keys. Set values from measured native schema, not invented aliases.

Trace lookup target uses queryType: trace_id, streamType: traces, actual stream/organization, both traceId and query equal to the actual trace ID, sqlMode: false, displayMode: trace. Optional explicit start/end bounds are microseconds; use Grafana's actual time range. Set a positive GF_DATAPROXY_RESPONSE_LIMIT in bytes (default/unset0 disables it). The plugin overrides backend total/search timeouts to210/180s; jsonData.timeout is not proof of a shorter bound. Lab7's native query gateway must enforce its documented transport budget; native engine/resource bounds remain explicit.

Native lab6 parser probes established that search query.size=1 with SQL LIMIT 2 returns two rows: a size-field guard is not a hard returned-row cap. Use explicit bounded provisioned SQL and verify the actual Grafana response-byte cap and native engine/resource budgets separately. Do not claim a universal1000-row limit from request validation. Native duplicate JSON keys are rejected422; duplicate or percent-encoded GET parameter names and fractional range bounds need their own native gateway controls and positive cases. The initial21-case receipt does not establish those later guards. Provision the native Prometheus data source HTTP method explicitly to match the gateway, rather than assuming Grafana's default.

Use a dedicated query principal restricted to the isolated lab organization and needed read paths. Never put OpenObserve root/admin credentials into Grafana. If native community roles cannot enforce the required read boundary, enforce exact actual query paths with the existing native gateway and verify allow/deny; do not claim a role restriction that the product does not implement. Limit unsigned loading to the exact checked plugin ID. Preserve standard TLS verification on each declared network hop.

Pinned1.1.2 OSS source forces ordinary POST /api/default/users creations to Admin even when viewer/user was requested. Do not use that route for the Grafana principal. The actual nonadmin initializer candidate is POST /api/{org}/service_accounts with email/first_name/last_name and a one-time token response; native auth and admin denial must be verified before final provisioning. Initial owner password policy is8..128 Unicode characters with lower/upper/digit ASCII and one other character. Capture issued service tokens privately in the same process; publish only safe shape/status evidence.

Native lab6 verification created two service accounts through that exact route: HTTP200 with code/message/token/user; both list as service_account. Both can search and ingest JSON, while both receive403 for admin token rotation. They are nonadmin identities, not native read-only or ingest-only roles. Enforce the query/ingest separation with the existing native HAProxy method/path boundary and closed raw listener; test the actual forbidden requests, not the account label. Never provision the initial owner credential to Collector or Grafana in the final configuration.

Acceptance gates

  1. Real current application emits all three signals; OpenObserve queries return their content and shared trace/span identifiers.
  2. Grafana displays metrics/histogram percentiles, structured logs and an actual trace waterfall including API, producer/consumer and worker spans; log/trace navigation resolves the same trace.
  3. Provisioned dashboards and alerts use the measured names/units and actual rule engine; high error rate, queue lag, SLO windows and firing/resolved transitions are tested with positive controls.
  4. Persistence/restart and storage outage preserve core API behavior with bounded SDK/Collector queues. Old on-disk evidence never becomes new PASS without execution.
  5. Lab7 additionally proves updated TLS/mTLS/identity/least-privilege and required native OpenBao lifecycle/rotation acceptance. Cached operator heartbeat evidence does not stand for OpenBao.
  6. Preserve initial source/config generation through build, runtime and packaging; exclude caches and verify the exact canonical archive boundary.

The user approved up to600 MB of new downloads on omarchy for the pinned Grafana13.3.1 image and existing OpenObserve plugin1.0.1. Lab6 performs one digest/SHA-verified download; lab7 reuses those exact cached artifacts without another download. The budget is our measured image/plugin size plus margin, not an assignment requirement. OpenBao image downloads and model downloads for SII remain separately pending and are not covered by this approval. Preserve earlier unapproved checkpoints as historical evidence.