14 KiB
Lab7 OpenObserve storage migration
Approved scope and preserved boundary
User authorized replacement of Prometheus/Loki/Tempo storage with OpenObserve. Grafana, OTel Collector, non-dev OpenBao, app PostgreSQL, S3, Valkey, both API replicas and worker remain mandatory. A byte snapshot of root's shared boundary is OPENOBSERVE-SHARED-BOUNDARY.md beside this document; it owns the exact shared native ingest/query contract and Grafana integration decision. Backend112 remains byte-identical to BACKEND-READY19:04:33.200Z / SHA09fe19253582282e489406600f3ee20c1e52d786778d6ce90f9d9da4146e5609.
The accepted ZIP8e15/361/ca313 and their source/runtime/review evidence stay historical. OpenObserve passed scoped native bootstrap/all3/persistence/mTLS/query checks; Grafana19panels/trace links/two original alert incidents/response boundaries have separate historical provider receipts in GRAFANA-NATIVE.md. Native OpenBao2.7.2/PebbleDB/Agent, rotation/expiry/seal/recovery and bounded sharedHTTP429 now have a separate accepted composite in ../evidence/openbao/NATIVE-COMPOSITE.json. No failed whole command or former operator provider was relabeled. Final delivery docs/archive review is separate; full security/whole telemetry/teacher grade remain unclaimed. Approved quotas:600000000bytes Grafana/plugin(actual559001001once) and85000000OpenBao(actual80099233); SII is not authorized by either. Azur services/credentials/cache ownership are unchanged.
The local shared-boundary and native-identity files are byte-identical historical contract snapshots. Their then-pending download/gateway labels preserve producer provenance and are not current permission/status instructions; use RUN.md/REPORT.md and the Lab7 composite for the current state. They do not require a sibling workspace at runtime.
Root approved one isolated single-node OpenObserve per lab using its own persistent local data/metadata volume and native default organization. This is a temporary coursework default. No Azur S3/Litestream/NATS/etcd topology or credentials are carried over. API/worker keep their existing OTLP HTTP/protobuf endpoint and mTLS roles; no OpenObserve secret is added to their OpenBao KV or app mounts.
Temporary internal names follow existing service conventions: openobserve and openobserve-gateway. Actual native ports, route allowlists, auth principal provisioning, Collector secret-file consumers and plugin provisioning must be verified before generated runtime configuration is accepted. Metrics use native Grafana Prometheus querying; normalized series/streams come from the checked live lab6 ingest contract, not the previous messenger Prometheus namespace.
The ready-made third-party plugin is Windforce17/openobserve-grafana-plugin v1.0.1 commit e10214739827dda9318705d49db262b28d67bcfe, Apache2, Grafana>=12.4, unsigned. Its unmodified frontend native trace frame and actual Grafana waterfall/links were checked; generic Go backend tables are not used as waterfall proof. Native auth and separate browser/backend response cases are in GRAFANA-NATIVE.md. It is not the official OpenObserve plugin. Unsigned loading is restricted to this exact checked plugin ID; no shim or dependency patch is introduced.
Write boundary
Permitted changes: own topology/config generation, telemetry gateways/PKI roles, Collector exporters, Grafana provisioning/dashboard/alerts, bootstrap only for actual new consumers, trust matrix, owned tests/docs/evidence and a newly verified source package. Preserve source leases, exclusive historical evidence and FIRST-snapshot archive guards. No backend source writes, no predecessor bulk sync, no original public contract/test changes, no production publish or external messages.
Collector and Grafana use distinct client identities. Collector may ingest only; Grafana may query only within the isolated organization. Native OpenObserve permissions must be measured, with exact native gateway paths enforcing restrictions if community roles are broader. No root/admin credential is placed in Grafana. The raw storage listener remains loopback-only in its gateway namespace, with no externally published port or plaintext cross-container fallback. Actual credential files and generated private provisioning remain outside Git/build/log/evidence payloads.
Cases before code
| Situation / input class | Example | Expected result | Level |
|---|---|---|---|
| Storage replacement | generated manifest | Exactly one OpenObserve; old three stores and their gateways absent | Pure generator regression |
| Mandatory neighbors | API/worker/PG/S3/Valkey/Collector/Grafana/Bao | All retained; unchanged app OTLP/file contracts and backend hashes | Generator + source lease |
| Persistent isolation | OpenObserve restart | Dedicated local volume, independent of appPG/Azur; retained own signals | Native restart/query |
| Allowed ingestion | controlled API upload and worker | All three signal payloads queryable with same trace/span context | Native OTLP/query |
| Allowed query | dedicated Grafana principal | Metrics/logs/real trace frame and log↔trace links resolve current data | Native Grafana |
| Untrusted transport | no cert, foreign CA, wrong SAN, expired leaf | Specific TLS/identity denial with positive controls; no timeout-as-denial | Native mTLS matrix |
| Trusted wrong role | app/worker on storage, Grafana ingest, Collector query/admin | Denied despite same CA; correct role succeeds | Native gateway/OO |
| Wrong credentials/org | rejected password, foreign organization | Explicit auth/authorization denial; no shared root/admin principal | Native auth |
| Closed local hop | storage raw port from another namespace | Refused; no app/storage port publication | Native network |
| Bounded outage | OpenObserve stopped | Core messaging works; SDK/Collector buffers/RAM bounded; recovery observed | Native failure/recovery |
| Alert transitions | worker backlog/API failure | Provisioned native engine firing then resolved using measured names/units | Native rule/state |
| Secret failure | credential read/render denied | Fail closed, fixed primary error/stage, no diagnostic leak | Fake-port + native |
| Regression boundary | backend112 / old ZIP8e15 | Exact hashes unchanged; new evidence never overwrites accepted history | Source/artifact guards |
Failures to cover: unavailable storage, startup/auth failure, partial credential provisioning, oversized/timeout query, retry duplication, missing signal fields, plugin/server incompatibility, old native metric-name assumptions, restart during ingest, stale exporter queues, wrong TLS identity, expired/revoked certificate and failed evidence I/O. Native Bao render/renew/seal/rotation remains required separately; operator heartbeat evidence is not its substitute.
Historical initial inventory and next gates
Read-only Docker image inventory under the current operator finds only Alertmanager among OpenObserve/Collector/Grafana/OpenBao/telemetry names. This does not prove absence in another containerd cache. Native ctr exists; root coordinates privileged cache inspection/digest discovery without starting Azur. No layers or plugin artifacts have been downloaded by this worker.
Implementation starts with metadata-only RED against the actual existing generator. Runtime routes, plugin/auth configuration and image pins follow the root checked boundary. Native images/UI, all three signals, alert transitions, complete mTLS/least privilege/Bao acceptance and independent final review remain pending.
Native account provisioning cases before code
The verified native identity contract is copied byte-identically as OPENOBSERVE-NATIVE-IDENTITIES.json beside this document, SHA a96ee905c87ca284a8c71f662e1d1034354c525e08e8215078bd63ea7da23a8d. OSS human-user creation forces Admin and is not used. Separate service accounts are nonadmin but both can read/write natively; the two certificate/method/path listeners provide the split. Tokens must be captured privately in the same process before another account request.
| Account situation | Expected outcome | Level |
|---|---|---|
| Valid native HTTP200/code200/user/token | Save exact own token before returning success | Required fake ports then native |
| HTTP401/403/500 or embedded non200 code | Static failure; no credential publication | Required fake ports |
| Missing/empty/oversized token or unexpected user | Fail closed; no invalid Grafana credential | Required fake ports |
| Token persistence failure | Preserve failure; do not provision next identity | Required fake ports |
| Existing saved account | Reuse verified own bytes; do not rotate implicitly | Native workflow |
| Query SA sends ingest / Collector SA sends query | TLS or explicit gateway denial, correct path200 around negative | Native gateway |
Native schema copied from checked lab6 ingestion: logs/traces messenger; trace parent reference_parent_span_id; start_time/end_time nanoseconds; duration/_timestamp microseconds. Histogram bases are http_duration/job_duration/time_to_ready and counter http_requests/jobs_processed, with no legacy exporter namespace or invented _total/unit suffix. Grafana dashboard expressions now follow these names. The optional plugin duration shortcut remains unused.
Query serialization regression
Current112 all3 session failed at native readback because the authored URI guard rejected plus in the native URLSearchParams serialization of a spaced PromQL query. The command exited1 and stopped its stand. This was a helper failure after app smoke, not a successful whole-session acceptance.
| Input class | Expected behavior | Level |
|---|---|---|
| URLSearchParams PromQL with spaces | Accept serialized plus before any credential read | Pure validator regression |
| Foreign absolute URL or organization | Reject fixed code | Pure validator regression |
| Raw CR/LF, fragment, unencoded path space | Reject fixed code | Pure validator regression |
| Persisted actualSDK trace/log/metric data after helper fix | Query existing native data; do not repeat successful smoke | Native readback/persistence |
Native readback then exposed HTTP422 with a plain deserialize error for the explicit instant-query time parameter. Preserve the HTTP outcome independently of JSON parsing: error-status text bodies retain status/data=null; successful JSON stays parsed; a successful non-JSON response fails with its HTTP code. These three classes are pure response-boundary regressions. The raw native response stays private; query-time grammar must follow the measured native consumer.
The second failure was isolated further: native error markers are floating-point/expected-i64, and the actual filesystem mtimeMs1794785761292.7983 produces fractional search start_time1794784861292798.2. Instant PromQL time grammar was not the cause established by that response and remains unchanged. Native search bounds need integer microseconds.
| Native search bounds case | Expected | Level |
|---|---|---|
| Actual fractional filesystem timestamp | Quantize to integer microseconds before native JSON serialization | Pure producer-boundary regression |
| NaN/infinity/negative/unsafe integer | Fixed rejection before request | Pure regression |
| Inverted/zero/over1hour window | Fixed rejection | Pure regression |
| Valid current window | Same integer microseconds on JSON roundtrip | Pure regression |
Historical storage-only phase closure
Native receipts and historical failed commands are described in STATUS.md.85TS,63unit tests/206expect and nonwriting formatter passed. Exact current gateway policy was parsed by native HAProxy and passed23 native query cases; TLS2listener identity/trust tests passed separately. No native Grafana rendering/alert transitions or Bao acceptance is implied. The old Prometheus/Loki/Tempo config files remain unmounted historical source; manifest/topology determine current consumers.
The canonical shared-decision origin is workspace vp/OPENOBSERVE-BOUNDARY.md. The package uses its local byte snapshot, OPENOBSERVE-SHARED-BOUNDARY.md; it does not require the sibling workspace at runtime. OPENOBSERVE-NATIVE-IDENTITIES.json preserves the public producer contract byte-for-byte, so its original evidence field is provenance for lab6. Local OPENOBSERVE-NATIVE-SCHEMA.json copies only the measured field names/version/schema consumed here and binds the exact original public contract SHA; authorization/credential fields are omitted. Own lab7 evidence links are resolved from STATUS.md.
Historical independent-review correction checkpoint
The initial65-path source review is retained unchanged in evidence/openobserve/review-final-source.json. Three bounded corrections had a separate delta: numerator-only error fallback (native healthy0/positive0.2/all-absentNoData), fixed historical SQL/PromQL snapshot end through required fake ports, and saved-account native startup guards before derived publication. That historical checkpoint had88TS and79unit tests/243assertions; exact failures/controls and native existing-account200/403 proof remain in docs/OPENOBSERVE-REVIEW-FIX-CASES.md and retained evidence. No new producer or service account was created for that correction. Its then-pending Grafana/Bao gates are superseded only by their separately scoped later receipts.
Native Grafana phase
The current storage stack is OpenObserve1.1.2 + Collector-k8s0.163.2 + Grafana13.3.1 with the unmodified ready-made OpenObserve1.0.1 plugin. Native19-panel content, trace42 waterfall/links, both original alert incidents and actual8MiB response consumers have separate measured receipts. See GRAFANA-NATIVE.md; earlier source/available-native/ZIP361 checkpoints above are historical. Backend112 remains unchanged. Later native Bao/Agent accepted phases and CLI portability are linked by the review wrapper; current source checks123TS/178tests420/fmt0 and all limitations are in REPORT.md. No whole-grade/full-security or fresh canonical build claim is added.