scadaproj/components/observability/GAPS.md

Observability — gaps & adoption backlog

Divergence of each project from spec/SPEC.md, and the ordered backlog to reach the shared ZB.MOM.WW.Telemetry library. Status legend: ⛔ gap · 🟡 partial · ✅ matches.

✅ ADOPTED — verified 2026-07-18. ZB.MOM.WW.Telemetry + .Serilog 0.1.0 are on the Gitea feed and all four apps reference both and call AddZbTelemetry (Resource + standard instrumentation + Prometheus /metrics). Logging: OtOpcUa, MxAccessGateway and HistorianGateway (a fourth consumer this file predates) use AddZbSerilog; ScadaBridge deliberately keeps its own LoggerConfigurationFactory for min-level governance and adds only the shared TraceContextEnricher — that split is by design, not a gap. Library suite: 34 tests, green 2026-07-18.

The ⛔ not set divergence tables below are the pre-adoption 2026-06-01 snapshot — the §1 Resource-attribute gaps they describe are what AddZbTelemetry closed. Read them as the historical case for the library, not as current state. The items deferred at adoption time (MxGateway ms→s + Meter rename, ScadaBridge app instruments + Site-node :8084 metrics listener, OTLP opt-in wiring) were all subsequently executed the same day — see the "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there (deploying an OTel collector; further ScadaBridge instruments) remain.

Divergence vs spec

§1 OTel Resource / service.name (P1 — nobody has it)

Spec attribute	OtOpcUa	MxAccessGateway	ScadaBridge
service.name	⛔ not set	⛔ not set	⛔ not set
service.namespace	⛔ not set	⛔ not set	⛔ not set
service.version	⛔ not set	⛔ not set	⛔ not set
site.id	⛔ not set	⛔ not set	⛔ not set
node.role	⛔ not set	⛔ not set	⛔ not set
host.name	⛔ not set	⛔ not set	⛔ not set

No project configures a ResourceBuilder or equivalent. Every metric and span from every node is indistinguishable in a backend — no service identity, no topology (site/role), no version label. This is the single highest-value gap across the fleet; closing it requires only adding AddZbTelemetry with options.

→ Gap R1 (P1): All three projects must call AddZbTelemetry with ServiceName, SiteId, NodeRole options to populate the shared Resource. None may do so before the library is available.

§2 Metrics export (P1 — MxGateway metrics are invisible)

	OtOpcUa	MxAccessGateway	ScadaBridge
OTel SDK present	✅ (AddOpenTelemetry)	⛔ none	⛔ none (OpenTelemetry.Api only)
Meter registered	✅ ZB.MOM.WW.OtOpcUa	🟡 MxGateway.Server (not via OTel SDK)	⛔ no Meter
Prometheus export	✅ /metrics	⛔ GetSnapshot() only	⛔ none
OTLP export	⛔ not available	⛔ not available	⛔ not available

MxGateway has 20 production instruments (13 counters, 3 histograms, 4 gauges) recording real data that is never exported. The data lives only in an in-memory GetSnapshot() read path. Adding AddZbTelemetry with Meters = ["MxGateway.Server"] closes this gap without touching GatewayMetrics.cs.

→ Gap M1 (P1): MxGateway: wire OTel SDK via AddZbTelemetry so GatewayMetrics exports. This is one half of the in-pass adoption (logging migration is the other). → Gap M2: ScadaBridge: define a ScadaBridgeTelemetry class and first application instruments (scadabridge.*); register via AddZbTelemetry. Currently zero instrumentation.

§3 MxGateway logging: MEL → Serilog (P1 — in-pass adoption, done in this task)

	OtOpcUa	MxAccessGateway	ScadaBridge
Logging framework	✅ Serilog	⛔ MEL only	✅ Serilog
Structural enrichers	🟡 driver-scope only	⛔ MEL scope (provider-dependent)	✅ SiteId/NodeRole/NodeHostname
Correlation mechanism	LogContextEnricher.Push	GatewayLogScope + BeginScope	structural enrichers
Log redaction	⛔ none	✅ GatewayLogRedactor	⛔ none
ILogRedactor seam	⛔ none	⛔ bespoke	⛔ none

Status: done in this task (Task #9). The MxGateway logging migration — MEL → AddZbSerilog, GatewayLogScope → LogContext.PushProperty, GatewayLogRedactor → ILogRedactor seam — is executed as part of the ZB.MOM.WW.Telemetry library build. See current-state/mxaccessgw/CURRENT-STATE.md adoption plan for the exact changes.

→ Gap L1 (in-pass, done): MxGateway MEL → AddZbSerilog + LogContext correlation + ILogRedactor.

§4 Trace↔log correlation (nobody has it)

	OtOpcUa	MxAccessGateway	ScadaBridge
trace_id/span_id in logs	⛔ absent	⛔ absent (no spans)	⛔ absent (no spans)
Mechanism	TraceContextEnricher not wired	n/a	n/a

OtOpcUa creates spans (otopcua.deploy.apply, otopcua.opcua.address_space_rebuild) but never pushes Activity.Current's trace context onto Serilog's LogContext. A log emitted inside a span cannot be joined to the span in a backend. MxGateway and ScadaBridge have no spans at all.

The shared TraceContextEnricher (part of ZB.MOM.WW.Telemetry.Serilog) closes this for all three projects on AddZbSerilog adoption — it is wired automatically.

→ Gap C1: OtOpcUa: adopt AddZbSerilog to wire TraceContextEnricher; spans and logs become joinable once the enricher is active. → Gap C2: ScadaBridge/MxGateway: enricher wired on adoption; effective once spans are created.

§5 Duration unit: ms vs s

	OtOpcUa	MxAccessGateway	ScadaBridge
Histogram unit	✅ s	⛔ ms (workers.startup.duration, commands.duration, events.stream_send.duration)	n/a (no histograms)

MxGateway's three histograms record in milliseconds. OTel semantic conventions (and OtOpcUa) use seconds. Raw values differ by a factor of 1000 — dashboards and SLO alerts would need separate scaling rules without normalization.

→ Gap U1: MxGateway: convert the three histogram call sites to record in seconds (multiply by 0.001 or redefine the instruments). This is a breaking change to existing dashboards and is tagged as a convergence item separate from the initial adoption.

§6 Meter naming: MxGateway.Server vs namespace convention

	OtOpcUa	MxAccessGateway	ScadaBridge
Meter name	✅ ZB.MOM.WW.OtOpcUa	⛔ MxGateway.Server	n/a

The fleet convention (per spec/SPEC.md) is <project-namespace> — OtOpcUa uses ZB.MOM.WW.OtOpcUa; the gateway's meter should be ZB.MOM.WW.MxGateway. MxGateway.Server is the assembly name, not the namespace, and does not carry the ZB.MOM.WW product prefix.

→ Gap N1: MxGateway: rename Meter from "MxGateway.Server" → "ZB.MOM.WW.MxGateway". This is a Prometheus metric label change that breaks dashboards/alerts and is tracked separately from the initial adoption.

§7 Standard instrumentation (nobody has the full set)

Instrumentation	OtOpcUa	MxAccessGateway	ScadaBridge
ASP.NET Core request metrics	⛔ not added	⛔ not added	⛔ not added
HttpClient metrics	⛔ not added	⛔ not added	⛔ not added
Runtime / process metrics	⛔ not added	⛔ not added	⛔ not added
gRPC client metrics	⛔ not added	⛔ not added	n/a

AddZbTelemetry enables these via AddAspNetCoreInstrumentation, AddHttpClientInstrumentation, AddRuntimeInstrumentation, and AddProcessInstrumentation by default — all three projects gain them on AddZbTelemetry adoption without code changes.

→ Gap S1: all three projects lack standard instrumentation; closed automatically by AddZbTelemetry adoption.

§8 Serilog version split (Serilog.AspNetCore 8 vs 9)

OtOpcUa and ScadaBridge use Serilog.AspNetCore but may be on different versions due to their independent csproj updates. The shared ZB.MOM.WW.Telemetry.Serilog package must declare a Serilog.AspNetCore version floor that works for both. Verify version alignment on adoption.

→ Gap V1: confirm Serilog.AspNetCore version compatibility across all three projects on AddZbSerilog adoption; align if diverged.

§9 ScadaBridge: zero instrumentation

ScadaBridge has no application instruments today. The OpenTelemetry.Api ref is a CVE patch, not instrumentation. No counter, histogram, gauge, or span exists anywhere in the solution.

→ Gap I1: ScadaBridge: define ScadaBridgeTelemetry with a ZB.MOM.WW.ScadaBridge Meter and a first set of scadabridge.* instruments. Tracked as a follow-on (application-specific work, not shared-library work).

Adoption backlog (ordered)

#	Item	Projects	Priority	Effort	Risk	Notes
1	MxGateway MEL → AddZbSerilog: LogContext correlation + ILogRedactor (Gap L1)	MxGateway	P1	M	Low	In-pass, done in Task #9 — unblocked by library build
2	MxGateway: wire OTel SDK via AddZbTelemetry; GatewayMetrics begins exporting (Gap M1)	MxGateway	P1	S	Low	Bundled with #1 in Task #9
3	All: AddZbTelemetry with ServiceName/SiteId/NodeRole → shared Resource (Gap R1)	OtOpcUa, ScadaBridge	P1	S	Low	MxGateway covered by #2; others are follow-on
4	OtOpcUa: adopt AddZbSerilog + TraceContextEnricher (Gaps C1, V1)	OtOpcUa	P2	S	Low	Keep LogContextEnricher; add shared enrichers alongside
5	ScadaBridge: adopt AddZbSerilog; replace LoggerConfigurationFactory (Gap C2, V1)	ScadaBridge	P2	S	Low	Enricher names already match; LoggerConfigurationFactory deleted
6	MxGateway: histogram ms → s conversion (Gap U1)	MxGateway	P2	S	Med	Breaking dashboard/alert change; coordinate with ops
7	MxGateway: rename Meter "MxGateway.Server" → "ZB.MOM.WW.MxGateway" (Gap N1)	MxGateway	P3	XS	Med	Breaking Prometheus label change; coordinate with ops
8	All: standard instrumentation via AddZbTelemetry options (Gap S1)	all 3	P2	XS	Low	Automatic on AddZbTelemetry adoption; no extra code
9	ScadaBridge: define ScadaBridgeTelemetry + first scadabridge.* instruments (Gap I1)	ScadaBridge	P2	M	Low	Application-specific work; tracked in ScadaBridge repo
10	OtOpcUa: wire OTLP exporter alongside Prometheus (Gap M2)	OtOpcUa	P3	S	Low	Opt-in via AddZbTelemetry options; no code rewrite
11	All: fix the tracing no-op in OtOpcUa (spans recorded, no exporter)	OtOpcUa	P2	S	Low	Wire OTLP or at minimum document the gap

Sequencing: Items #1 and #2 are the in-pass adoption (Task #9, MxGateway only). Items #3–#5 are the follow-on OtOpcUa/ScadaBridge adoptions — they are independent of each other and can land in either order. Items #6–#7 (unit/naming convergence) are breaking changes requiring ops coordination; defer until dashboards can be updated. Items #8–#11 are cleanups that bundle naturally with #3–#5.

This mirrors the Auth and UI-Theme pattern: the shared library is built first; adoption by each app is opt-in and tracked here, not forced.

Decisions still open

• Canonical SiteId and NodeRole config binding path — ScadaBridge reads from its own config hierarchy; AddZbSerilog must accept the value directly (caller-supplied) rather than reading from a fixed config section, to remain project-agnostic.

Decisions settled (no longer open)

• Prometheus vs OTLP default (SETTLED): AddZbTelemetry defaults to Prometheus (matching OtOpcUa's existing /metrics posture). OTLP is opt-in via ZbTelemetryOptions.Exporter = ZbExporter.Otlp. See spec/SPEC.md §4 and shared-contract ZbTelemetryOptions.Exporter.
• ms→s conversion and Meter rename bundling (SETTLED — deferred): Both the histogram unit migration (Gap U1) and the Meter rename (Gap N1) are deferred from the initial MxGateway adoption (Task #9). They are breaking dashboard/alert changes requiring ops coordination and are tracked as separate backlog items #6 and #7 in the adoption backlog above.

Adoption status — 2026-06-01 (DONE)

ZB.MOM.WW.Telemetry + ZB.MOM.WW.Telemetry.Serilog (0.1.0) were adopted across all three sister apps in one pass, behaviour-preserving. Each adoption landed on a per-repo branch feat/adopt-zb-telemetry (one commit per task). Plan + design: docs/plans/2026-06-01-telemetry-library-adoption.md.

Correction: the prior claim that "MxAccessGateway logging was adopted (MEL → Serilog) on its own branch" was false on main — MxGateway was still MEL-only, and its MxGateway.Server meter was never exported. The full MEL→Serilog migration and the metrics export both landed in this 2026-06-01 pass.

Repo	AddZbTelemetry (Resource + std instrumentation + Prometheus)	/metrics	Logging	Meter (unchanged)
OtOpcUa	✅ replaced hand-rolled ObservabilityExtensions	✅ /metrics (path unchanged)	✅ AddZbSerilog (sinks moved to appsettings; LogContextEnricher kept)	ZB.MOM.WW.OtOpcUa
ScadaBridge	✅ added in BindSharedOptions (both Central + Site roots)	✅ Central; mapped on Site too (see follow-on)	⚠️ kept LoggerConfigurationFactory + added shared TraceContextEnricher — did not adopt AddZbSerilog	(none yet; #9)
MxAccessGateway	✅ exports existing GatewayMetrics	✅ new /metrics	✅ MEL→AddZbSerilog; GatewayLogRedactor exposed via ILogRedactor seam (GatewayLogRedactorSeam); GatewayLogScope/middleware kept as-is	MxGateway.Server (name + ms units unchanged)

Accepted scope decisions (deviations from the original backlog)

• ScadaBridge keeps LoggerConfigurationFactory (backlog #5 revised). The factory implements a documented governance contract (REQ-HOST-8 / Host-011/014/020/022): ScadaBridge:Logging:MinimumLevel is the floor and overrides Serilog:MinimumLevel, with operator warnings. AddZbSerilog hard-codes MinimumLevel.Is(Information) before ReadFrom.Configuration, which would invert that precedence and silently drop the knob. So ScadaBridge keeps the factory and only adds the shared TraceContextEnricher to it — gaining trace↔log correlation without regressing the contract. Full AddZbSerilog adoption for ScadaBridge would first require teaching the shared bootstrap to accept a caller-supplied minimum-level governance hook.
• MxGateway keeps GatewayLogScope + request-logging middleware as-is. The Serilog MEL provider captures MEL BeginScope dictionaries as structured properties, so the scope/correlation code keeps producing the same properties under Serilog. Only the provider swap + the ILogRedactor adapter were needed.

Follow-ons — DONE 2026-06-01

All the deferred follow-ons were then executed (branch feat/telemetry-followons per repo, behaviour-preserving except the intentional, no-consumer-yet metric-shape change in #6/#7). Plan: docs/plans/2026-06-01-telemetry-followons.md.

Item	Status	What landed
#6 MxGateway histogram ms→s	✅	3 histograms record .TotalSeconds, unit "s". Safe — never Prometheus-exported before, so no dashboards broke.
#7 Meter rename → ZB.MOM.WW.MxGateway	✅	GatewayMetrics.MeterName renamed; docs/Metrics.md synced.
#9 ScadaBridge app instruments	✅	ScadaBridgeTelemetry meter (ZB.MOM.WW.ScadaBridge) + first 4: deployments.applied (counter), store_and_forward.queue.depth (sync-safe cached gauge), inbound_api.requests (counter, bounded method tag), site.connection.up (balanced open/close gauge).
#10/#11 OTLP opt-in	✅	All 3 apps read <App>:Telemetry:Exporter (Prometheus|Otlp) + :OtlpEndpoint, default Prometheus. Setting OTLP also exports OtOpcUa's spans (resolves the trace no-op) — once a collector endpoint is configured.
Site-node /metrics scrape	✅	ScadaBridge NodeOptions.MetricsPort (default 8084, avoids the site RemotingPort=8082 collision) + a second Http1AndHttp2 Kestrel listener on the Site role; StartupValidator enforces MetricsPort ≠ Remoting/Grpc.
Serilog version drift	✅	OtOpcUa Serilog.AspNetCore/.Extensions.Hosting/.Settings.Configuration aligned to 10.0.0 (family-consistent).

Still open (not code — operational/future):

• OTLP is opt-in but unexercised until an OTel collector endpoint is deployed and the <App>:Telemetry:Exporter=Otlp + :OtlpEndpoint config is set. The wiring is in place; only a collector is missing.
• Further ScadaBridge instruments beyond the first 4 are additive future work (not blocking).