Joseph Doherty
3dd7aa40ff
Publishes the three ZB.MOM.WW.LocalDb packages to the Gitea feed (restore-verified from a scratch consumer) and adds the build/push.sh the other shared libs already have. The status prose across CLAUDE.md, README.md and components/*/GAPS.md had drifted from reality, so it was re-derived from the feed listing and the actual PackageReferences + registration calls on each consumer's default branch rather than from prior claims. Five claims were false: Health "not yet adopted" (all four apps wire MapZbHealth), GalaxyRepository's mxaccessgw adoption "a follow-on" (its Server wires AddZbGalaxyRepository), Configuration "not yet pushed", Secrets G-8 "not yet committed", and Theme pinned at 0.2.0 (all four are on 0.3.1). Every doc also said "three apps" while HistorianGateway is a fourth consumer of seven libs, and all eight libraries' test counts were stale (re-ran each suite; all green). Surfaces one previously unrecorded gap: Secrets source is at 0.1.3 with KEK rotation committed, but the feed tops out at 0.1.2, so no app can consume rotation until it is published. Health and observability divergence tables are labelled historical, not re-verified — the libraries are proven wired, but per-app probe coverage vs spec was not re-walked. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
252 lines
17 KiB
Markdown
252 lines
17 KiB
Markdown
# Observability — gaps & adoption backlog
|
||
|
||
Divergence of each project from [`spec/SPEC.md`](spec/SPEC.md), and the ordered backlog to
|
||
reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 partial · ✅ matches.
|
||
|
||
> **✅ ADOPTED — verified 2026-07-18.** `ZB.MOM.WW.Telemetry` + `.Serilog` `0.1.0` are on the Gitea
|
||
> feed and **all four apps reference both and call `AddZbTelemetry`** (Resource + standard
|
||
> instrumentation + Prometheus `/metrics`). Logging: OtOpcUa, MxAccessGateway and **HistorianGateway**
|
||
> (a fourth consumer this file predates) use `AddZbSerilog`; ScadaBridge deliberately keeps its own
|
||
> `LoggerConfigurationFactory` for min-level governance and adds only the shared `TraceContextEnricher`
|
||
> — that split is by design, not a gap. Library suite: **34 tests**, green 2026-07-18.
|
||
>
|
||
> **The `⛔ not set` divergence tables below are the pre-adoption 2026-06-01 snapshot** — the §1
|
||
> Resource-attribute gaps they describe are what `AddZbTelemetry` closed. Read them as the
|
||
> historical case for the library, not as current state. Deferred items noted at the time
|
||
> (MxGateway `ms`→`s` + Meter rename, ScadaBridge app instruments + Site-node HTTP/1.1 metrics
|
||
> listener, OTLP wiring) were **not** re-verified in this pass.
|
||
|
||
## Divergence vs spec
|
||
|
||
### §1 OTel Resource / `service.name` (P1 — nobody has it)
|
||
|
||
| Spec attribute | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||
|---|---|---|---|
|
||
| `service.name` | ⛔ not set | ⛔ not set | ⛔ not set |
|
||
| `service.namespace` | ⛔ not set | ⛔ not set | ⛔ not set |
|
||
| `service.version` | ⛔ not set | ⛔ not set | ⛔ not set |
|
||
| `site.id` | ⛔ not set | ⛔ not set | ⛔ not set |
|
||
| `node.role` | ⛔ not set | ⛔ not set | ⛔ not set |
|
||
| `host.name` | ⛔ not set | ⛔ not set | ⛔ not set |
|
||
|
||
No project configures a `ResourceBuilder` or equivalent. Every metric and span from every node is
|
||
**indistinguishable** in a backend — no service identity, no topology (site/role), no version label.
|
||
This is the single highest-value gap across the fleet; closing it requires only adding
|
||
`AddZbTelemetry` with options.
|
||
|
||
→ **Gap R1 (P1):** All three projects must call `AddZbTelemetry` with `ServiceName`, `SiteId`,
|
||
`NodeRole` options to populate the shared Resource. None may do so before the library is available.
|
||
|
||
### §2 Metrics export (P1 — MxGateway metrics are invisible)
|
||
|
||
| | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||
|---|---|---|---|
|
||
| OTel SDK present | ✅ (`AddOpenTelemetry`) | ⛔ none | ⛔ none (`OpenTelemetry.Api` only) |
|
||
| Meter registered | ✅ `ZB.MOM.WW.OtOpcUa` | 🟡 `MxGateway.Server` (not via OTel SDK) | ⛔ no Meter |
|
||
| Prometheus export | ✅ `/metrics` | ⛔ `GetSnapshot()` only | ⛔ none |
|
||
| OTLP export | ⛔ not available | ⛔ not available | ⛔ not available |
|
||
|
||
MxGateway has 20 production instruments (13 counters, 3 histograms, 4 gauges) recording real data
|
||
that is **never exported**. The data lives only in an in-memory `GetSnapshot()` read path. Adding
|
||
`AddZbTelemetry` with `Meters = ["MxGateway.Server"]` closes this gap without touching
|
||
`GatewayMetrics.cs`.
|
||
|
||
→ **Gap M1 (P1):** MxGateway: wire OTel SDK via `AddZbTelemetry` so `GatewayMetrics` exports.
|
||
This is one half of the in-pass adoption (logging migration is the other).
|
||
→ **Gap M2:** ScadaBridge: define a `ScadaBridgeTelemetry` class and first application instruments
|
||
(`scadabridge.*`); register via `AddZbTelemetry`. Currently zero instrumentation.
|
||
|
||
### §3 MxGateway logging: MEL → Serilog (P1 — in-pass adoption, done in this task)
|
||
|
||
| | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||
|---|---|---|---|
|
||
| Logging framework | ✅ Serilog | ⛔ MEL only | ✅ Serilog |
|
||
| Structural enrichers | 🟡 driver-scope only | ⛔ MEL scope (provider-dependent) | ✅ `SiteId`/`NodeRole`/`NodeHostname` |
|
||
| Correlation mechanism | `LogContextEnricher.Push` | `GatewayLogScope` + `BeginScope` | structural enrichers |
|
||
| Log redaction | ⛔ none | ✅ `GatewayLogRedactor` | ⛔ none |
|
||
| `ILogRedactor` seam | ⛔ none | ⛔ bespoke | ⛔ none |
|
||
|
||
**Status: done in this task (Task #9).** The MxGateway logging migration — MEL → `AddZbSerilog`,
|
||
`GatewayLogScope` → `LogContext.PushProperty`, `GatewayLogRedactor` → `ILogRedactor` seam — is
|
||
executed as part of the `ZB.MOM.WW.Telemetry` library build. See
|
||
[`current-state/mxaccessgw/CURRENT-STATE.md`](current-state/mxaccessgw/CURRENT-STATE.md) adoption
|
||
plan for the exact changes.
|
||
|
||
→ **Gap L1 (in-pass, done):** MxGateway MEL → `AddZbSerilog` + `LogContext` correlation + `ILogRedactor`.
|
||
|
||
### §4 Trace↔log correlation (nobody has it)
|
||
|
||
| | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||
|---|---|---|---|
|
||
| `trace_id`/`span_id` in logs | ⛔ absent | ⛔ absent (no spans) | ⛔ absent (no spans) |
|
||
| Mechanism | `TraceContextEnricher` not wired | n/a | n/a |
|
||
|
||
OtOpcUa creates spans (`otopcua.deploy.apply`, `otopcua.opcua.address_space_rebuild`) but never
|
||
pushes `Activity.Current`'s trace context onto Serilog's `LogContext`. A log emitted inside a span
|
||
cannot be joined to the span in a backend. MxGateway and ScadaBridge have no spans at all.
|
||
|
||
The shared `TraceContextEnricher` (part of `ZB.MOM.WW.Telemetry.Serilog`) closes this for all
|
||
three projects on `AddZbSerilog` adoption — it is wired automatically.
|
||
|
||
→ **Gap C1:** OtOpcUa: adopt `AddZbSerilog` to wire `TraceContextEnricher`; spans and logs become
|
||
joinable once the enricher is active.
|
||
→ **Gap C2:** ScadaBridge/MxGateway: enricher wired on adoption; effective once spans are created.
|
||
|
||
### §5 Duration unit: `ms` vs `s`
|
||
|
||
| | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||
|---|---|---|---|
|
||
| Histogram unit | ✅ `s` | ⛔ `ms` (`workers.startup.duration`, `commands.duration`, `events.stream_send.duration`) | n/a (no histograms) |
|
||
|
||
MxGateway's three histograms record in milliseconds. OTel semantic conventions (and OtOpcUa) use
|
||
seconds. Raw values differ by a factor of 1000 — dashboards and SLO alerts would need separate
|
||
scaling rules without normalization.
|
||
|
||
→ **Gap U1:** MxGateway: convert the three histogram call sites to record in seconds (multiply by
|
||
`0.001` or redefine the instruments). This is a **breaking change to existing dashboards** and is
|
||
tagged as a convergence item separate from the initial adoption.
|
||
|
||
### §6 Meter naming: `MxGateway.Server` vs namespace convention
|
||
|
||
| | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||
|---|---|---|---|
|
||
| Meter name | ✅ `ZB.MOM.WW.OtOpcUa` | ⛔ `MxGateway.Server` | n/a |
|
||
|
||
The fleet convention (per `spec/SPEC.md`) is `<project-namespace>` — OtOpcUa uses
|
||
`ZB.MOM.WW.OtOpcUa`; the gateway's meter should be `ZB.MOM.WW.MxGateway`. `MxGateway.Server` is
|
||
the assembly name, not the namespace, and does not carry the `ZB.MOM.WW` product prefix.
|
||
|
||
→ **Gap N1:** MxGateway: rename Meter from `"MxGateway.Server"` → `"ZB.MOM.WW.MxGateway"`.
|
||
This is a **Prometheus metric label change** that breaks dashboards/alerts and is tracked
|
||
separately from the initial adoption.
|
||
|
||
### §7 Standard instrumentation (nobody has the full set)
|
||
|
||
| Instrumentation | OtOpcUa | MxAccessGateway | ScadaBridge |
|
||
|---|---|---|---|
|
||
| ASP.NET Core request metrics | ⛔ not added | ⛔ not added | ⛔ not added |
|
||
| `HttpClient` metrics | ⛔ not added | ⛔ not added | ⛔ not added |
|
||
| Runtime / process metrics | ⛔ not added | ⛔ not added | ⛔ not added |
|
||
| gRPC client metrics | ⛔ not added | ⛔ not added | n/a |
|
||
|
||
`AddZbTelemetry` enables these via `AddAspNetCoreInstrumentation`, `AddHttpClientInstrumentation`,
|
||
`AddRuntimeInstrumentation`, and `AddProcessInstrumentation` by default — all three projects gain
|
||
them on `AddZbTelemetry` adoption without code changes.
|
||
|
||
→ **Gap S1:** all three projects lack standard instrumentation; closed automatically by
|
||
`AddZbTelemetry` adoption.
|
||
|
||
### §8 Serilog version split (`Serilog.AspNetCore` 8 vs 9)
|
||
|
||
OtOpcUa and ScadaBridge use `Serilog.AspNetCore` but may be on different versions due to their
|
||
independent csproj updates. The shared `ZB.MOM.WW.Telemetry.Serilog` package must declare a
|
||
`Serilog.AspNetCore` version floor that works for both. Verify version alignment on adoption.
|
||
|
||
→ **Gap V1:** confirm `Serilog.AspNetCore` version compatibility across all three projects on
|
||
`AddZbSerilog` adoption; align if diverged.
|
||
|
||
### §9 ScadaBridge: zero instrumentation
|
||
|
||
ScadaBridge has no application instruments today. The `OpenTelemetry.Api` ref is a CVE patch, not
|
||
instrumentation. No counter, histogram, gauge, or span exists anywhere in the solution.
|
||
|
||
→ **Gap I1:** ScadaBridge: define `ScadaBridgeTelemetry` with a `ZB.MOM.WW.ScadaBridge` Meter and
|
||
a first set of `scadabridge.*` instruments. Tracked as a follow-on (application-specific work,
|
||
not shared-library work).
|
||
|
||
## Adoption backlog (ordered)
|
||
|
||
| # | Item | Projects | Priority | Effort | Risk | Notes |
|
||
|---|---|---|---|---|---|---|
|
||
| 1 | MxGateway MEL → `AddZbSerilog`: `LogContext` correlation + `ILogRedactor` (Gap L1) | MxGateway | P1 | M | Low | **In-pass, done in Task #9** — unblocked by library build |
|
||
| 2 | MxGateway: wire OTel SDK via `AddZbTelemetry`; `GatewayMetrics` begins exporting (Gap M1) | MxGateway | P1 | S | Low | Bundled with #1 in Task #9 |
|
||
| 3 | All: `AddZbTelemetry` with `ServiceName`/`SiteId`/`NodeRole` → shared Resource (Gap R1) | OtOpcUa, ScadaBridge | P1 | S | Low | MxGateway covered by #2; others are follow-on |
|
||
| 4 | OtOpcUa: adopt `AddZbSerilog` + `TraceContextEnricher` (Gaps C1, V1) | OtOpcUa | P2 | S | Low | Keep `LogContextEnricher`; add shared enrichers alongside |
|
||
| 5 | ScadaBridge: adopt `AddZbSerilog`; replace `LoggerConfigurationFactory` (Gap C2, V1) | ScadaBridge | P2 | S | Low | Enricher names already match; `LoggerConfigurationFactory` deleted |
|
||
| 6 | MxGateway: histogram `ms` → `s` conversion (Gap U1) | MxGateway | P2 | S | Med | Breaking dashboard/alert change; coordinate with ops |
|
||
| 7 | MxGateway: rename Meter `"MxGateway.Server"` → `"ZB.MOM.WW.MxGateway"` (Gap N1) | MxGateway | P3 | XS | Med | Breaking Prometheus label change; coordinate with ops |
|
||
| 8 | All: standard instrumentation via `AddZbTelemetry` options (Gap S1) | all 3 | P2 | XS | Low | Automatic on `AddZbTelemetry` adoption; no extra code |
|
||
| 9 | ScadaBridge: define `ScadaBridgeTelemetry` + first `scadabridge.*` instruments (Gap I1) | ScadaBridge | P2 | M | Low | Application-specific work; tracked in ScadaBridge repo |
|
||
| 10 | OtOpcUa: wire OTLP exporter alongside Prometheus (Gap M2) | OtOpcUa | P3 | S | Low | Opt-in via `AddZbTelemetry` options; no code rewrite |
|
||
| 11 | All: fix the tracing no-op in OtOpcUa (spans recorded, no exporter) | OtOpcUa | P2 | S | Low | Wire OTLP or at minimum document the gap |
|
||
|
||
**Sequencing:** Items #1 and #2 are the in-pass adoption (Task #9, MxGateway only). Items #3–#5
|
||
are the follow-on OtOpcUa/ScadaBridge adoptions — they are independent of each other and can land
|
||
in either order. Items #6–#7 (unit/naming convergence) are breaking changes requiring ops
|
||
coordination; defer until dashboards can be updated. Items #8–#11 are cleanups that bundle
|
||
naturally with #3–#5.
|
||
|
||
This mirrors the Auth and UI-Theme pattern: the shared library is built first; adoption by each
|
||
app is opt-in and tracked here, not forced.
|
||
|
||
## Decisions still open
|
||
|
||
- Canonical `SiteId` and `NodeRole` config binding path — ScadaBridge reads from its own config
|
||
hierarchy; `AddZbSerilog` must accept the value directly (caller-supplied) rather than reading
|
||
from a fixed config section, to remain project-agnostic.
|
||
|
||
## Decisions settled (no longer open)
|
||
|
||
- **Prometheus vs OTLP default (SETTLED):** `AddZbTelemetry` defaults to Prometheus (matching
|
||
OtOpcUa's existing `/metrics` posture). OTLP is opt-in via `ZbTelemetryOptions.Exporter =
|
||
ZbExporter.Otlp`. See `spec/SPEC.md` §4 and shared-contract `ZbTelemetryOptions.Exporter`.
|
||
- **`ms`→`s` conversion and Meter rename bundling (SETTLED — deferred):** Both the histogram
|
||
unit migration (Gap U1) and the Meter rename (Gap N1) are deferred from the initial MxGateway
|
||
adoption (Task #9). They are breaking dashboard/alert changes requiring ops coordination and
|
||
are tracked as separate backlog items #6 and #7 in the adoption backlog above.
|
||
|
||
## Adoption status — 2026-06-01 (DONE)
|
||
|
||
`ZB.MOM.WW.Telemetry` + `ZB.MOM.WW.Telemetry.Serilog` (`0.1.0`) were adopted across **all three**
|
||
sister apps in one pass, behaviour-preserving. Each adoption landed on a per-repo branch
|
||
`feat/adopt-zb-telemetry` (one commit per task). Plan + design:
|
||
[`docs/plans/2026-06-01-telemetry-library-adoption.md`](../../docs/plans/2026-06-01-telemetry-library-adoption.md).
|
||
|
||
> **Correction:** the prior claim that *"MxAccessGateway logging was adopted (MEL → Serilog) on its
|
||
> own branch"* was **false on `main`** — MxGateway was still MEL-only, and its `MxGateway.Server`
|
||
> meter was never exported. The full MEL→Serilog migration **and** the metrics export both landed
|
||
> in this 2026-06-01 pass.
|
||
|
||
| Repo | `AddZbTelemetry` (Resource + std instrumentation + Prometheus) | `/metrics` | Logging | Meter (unchanged) |
|
||
|---|---|---|---|---|
|
||
| **OtOpcUa** | ✅ replaced hand-rolled `ObservabilityExtensions` | ✅ `/metrics` (path unchanged) | ✅ `AddZbSerilog` (sinks moved to `appsettings`; `LogContextEnricher` kept) | `ZB.MOM.WW.OtOpcUa` |
|
||
| **ScadaBridge** | ✅ added in `BindSharedOptions` (both Central + Site roots) | ✅ Central; mapped on Site too (see follow-on) | ⚠️ **kept `LoggerConfigurationFactory`** + added shared `TraceContextEnricher` — did **not** adopt `AddZbSerilog` | (none yet; #9) |
|
||
| **MxAccessGateway** | ✅ exports existing `GatewayMetrics` | ✅ new `/metrics` | ✅ MEL→`AddZbSerilog`; `GatewayLogRedactor` exposed via `ILogRedactor` seam (`GatewayLogRedactorSeam`); `GatewayLogScope`/middleware kept as-is | `MxGateway.Server` (name + `ms` units unchanged) |
|
||
|
||
### Accepted scope decisions (deviations from the original backlog)
|
||
|
||
- **ScadaBridge keeps `LoggerConfigurationFactory` (backlog #5 revised).** The factory implements a
|
||
documented governance contract (REQ-HOST-8 / Host-011/014/020/022): `ScadaBridge:Logging:MinimumLevel`
|
||
is the floor and **overrides** `Serilog:MinimumLevel`, with operator warnings. `AddZbSerilog`
|
||
hard-codes `MinimumLevel.Is(Information)` before `ReadFrom.Configuration`, which would invert that
|
||
precedence and silently drop the knob. So ScadaBridge keeps the factory and only **adds the shared
|
||
`TraceContextEnricher`** to it — gaining trace↔log correlation without regressing the contract. Full
|
||
`AddZbSerilog` adoption for ScadaBridge would first require teaching the shared bootstrap to accept a
|
||
caller-supplied minimum-level governance hook.
|
||
- **MxGateway keeps `GatewayLogScope` + request-logging middleware as-is.** The Serilog MEL provider
|
||
captures MEL `BeginScope` dictionaries as structured properties, so the scope/correlation code keeps
|
||
producing the same properties under Serilog. Only the provider swap + the `ILogRedactor` adapter were
|
||
needed.
|
||
|
||
## Follow-ons — DONE 2026-06-01
|
||
|
||
All the deferred follow-ons were then executed (branch `feat/telemetry-followons` per repo,
|
||
behaviour-preserving except the intentional, no-consumer-yet metric-shape change in #6/#7). Plan:
|
||
[`docs/plans/2026-06-01-telemetry-followons.md`](../../docs/plans/2026-06-01-telemetry-followons.md).
|
||
|
||
| Item | Status | What landed |
|
||
|---|---|---|
|
||
| **#6** MxGateway histogram `ms`→`s` | ✅ | 3 histograms record `.TotalSeconds`, unit `"s"`. Safe — never Prometheus-exported before, so no dashboards broke. |
|
||
| **#7** Meter rename → `ZB.MOM.WW.MxGateway` | ✅ | `GatewayMetrics.MeterName` renamed; `docs/Metrics.md` synced. |
|
||
| **#9** ScadaBridge app instruments | ✅ | `ScadaBridgeTelemetry` meter (`ZB.MOM.WW.ScadaBridge`) + first 4: `deployments.applied` (counter), `store_and_forward.queue.depth` (sync-safe cached gauge), `inbound_api.requests` (counter, bounded `method` tag), `site.connection.up` (balanced open/close gauge). |
|
||
| **#10/#11** OTLP opt-in | ✅ | All 3 apps read `<App>:Telemetry:Exporter` (`Prometheus`\|`Otlp`) + `:OtlpEndpoint`, default Prometheus. Setting OTLP also exports OtOpcUa's spans (resolves the trace no-op) — once a collector endpoint is configured. |
|
||
| **Site-node `/metrics` scrape** | ✅ | ScadaBridge `NodeOptions.MetricsPort` (default **8084**, avoids the site `RemotingPort=8082` collision) + a second `Http1AndHttp2` Kestrel listener on the Site role; `StartupValidator` enforces MetricsPort ≠ Remoting/Grpc. |
|
||
| Serilog version drift | ✅ | OtOpcUa `Serilog.AspNetCore`/`.Extensions.Hosting`/`.Settings.Configuration` aligned to `10.0.0` (family-consistent). |
|
||
|
||
**Still open (not code — operational/future):**
|
||
|
||
- **OTLP is opt-in but unexercised** until an OTel collector endpoint is deployed and the
|
||
`<App>:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` config is set. The wiring is in place; only a
|
||
collector is missing.
|
||
- **Further ScadaBridge instruments** beyond the first 4 are additive future work (not blocking).
|