Files
scadaproj/ZB.MOM.WW.Telemetry/CLAUDE.md
T

12 KiB

ZB.MOM.WW.Telemetry

Observability libraries for the ZB.MOM.WW SCADA family (OtOpcUa, MxAccessGateway, ScadaBridge). These are libraries, not a service — each package is linked directly into the consuming application at build time. There is no central telemetry process; instrumentation runs in-process alongside the application.

Backlog / open work: lives in Gitea issues on dohertj2/scadaproj (tea issues list --repo dohertj2/scadaproj) — #13 (OTel collector / OTLP end to end) is closed — proven 2026-11-12 against zb-otel-collector (infra/otel-collector/, evidence components/observability/2026-11-12-otlp-end-to-end-evidence.md); #55 (service.version/site.id/node.role + logs-over-OTLP) is closed by 0.3.0 (2026-11-12). Open: #54 (wonder-tier collector). The history below is a record, not a status tracker; file new follow-ups as issues.

The library normalizes the three-project observability surface: a shared OpenTelemetry Resource driven by a single identity triple (service.name / site.id / node.role), standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and TraceContextEnricher for trace↔log correlation.

Built at 0.1.0, published to the Gitea NuGet feed, and adopted across all three apps on 2026-06-01 (branch feat/adopt-zb-telemetry per repo, behaviour-preserving). MxAccessGateway's MEL→Serilog migration + metrics export both landed in this pass — they were not actually done beforehand despite the earlier claim. ScadaBridge kept its own LoggerConfigurationFactory until 2026-11-12, when it switched to AddZbSerilog 0.2.0 (ScadaBridge#94). Per-repo result + deferred follow-ons tracked in ~/Desktop/scadaproj/components/observability/GAPS.md.


Packages

Package Responsibilities Key Dependencies
ZB.MOM.WW.Telemetry AddZbTelemetry — wires OTel SDK (metrics + tracing), populates shared Resource (service.name, service.namespace, service.version, site.id, node.role, host.name), registers caller-supplied Meters/ActivitySources, adds standard instrumentation (ASP.NET Core, HttpClient, gRPC client, runtime, process), Prometheus always-on exporter, OTLP additive overlay. app.MapZbMetrics() — mounts /metrics. ZbTelemetryOptions — the single options object shared by both packages. 0.3.0: service.version defaults from the entry assembly's informational version; ResourceAttributes; ApplyConfiguration(<App>:Telemetry section); publishes its options in DI for the Serilog log sink. Microsoft.AspNetCore.App (framework ref), OpenTelemetry.* stack
ZB.MOM.WW.Telemetry.Serilog AddZbSerilog — shared two-stage Serilog bootstrap: ReadFrom.Configuration-driven sinks, MinimumLevel.Is(Information) default (config-overridable), SiteId/NodeRole/NodeHostname enrichers from ZbTelemetryOptions, TraceContextEnricher (writes trace_id/span_id from Activity.Current), ILogRedactor seam (per-project sensitive-field redaction via RedactionEnricher). 0.2.0: optional ZbSerilogOptions overload — MinimumLevelGovernor (app key outranks Serilog:MinimumLevel) + ConfigureLogger (final filters/enrichers/sinks). 0.3.0: the OTLP log sink follows the AddZbTelemetry registration in the same container (ExportLogs=false opts out). Does NOT freeze Log.Logger — safe for multi-host/test scenarios. ZB.MOM.WW.Telemetry, Serilog.* stack

Consumer matrix

Consumer ZB.MOM.WW.Telemetry (core) ZB.MOM.WW.Telemetry.Serilog
OtOpcUa ✅ 0.3.0 (master 412da27c) — site.id = Cluster id, node.role = plain roles ✅ 0.3.0 (AddZbSerilog; logs follow AddZbTelemetry)
MxAccessGateway ✅ 0.3.0 (main d9d9ff48) — node.role=standalone, site.id from config ✅ 0.3.0 (logs follow AddZbTelemetry)
ScadaBridge ✅ 0.3.0 (main a633005f, both roots) ✅ 0.3.0 AddZbSerilog with MinimumLevelGovernor + ConfigureLogger on a pre-host builder; the exporter is passed explicitly via ScadaBridgeTelemetryConfig
HistorianGateway ✅ 0.3.0 (main ae1dbcc) — node.role=standalone, site.id from config ✅ 0.3.0 (logs follow AddZbTelemetry)

All four pin both packages at 0.3.0 on their origin default branches since 2026-11-12 (scadaproj#55; not deployed — the wonder boxes still run their earlier builds, which have no collector anyway, #54). See components/observability/GAPS.md for the full result.


Build, test, and pack commands

# From ZB.MOM.WW.Telemetry/

# Build
dotnet build ZB.MOM.WW.Telemetry.slnx
dotnet build ZB.MOM.WW.Telemetry.slnx -c Release

# Test (no external dependencies — no running OTel collector, no Serilog backend required)
dotnet test ZB.MOM.WW.Telemetry.slnx

# Pack (two .nupkg files land in artifacts/)
dotnet pack ZB.MOM.WW.Telemetry.slnx -c Release -o ./artifacts

All test assemblies run offline:

Assembly Tests
ZB.MOM.WW.Telemetry.Tests 38
ZB.MOM.WW.Telemetry.Serilog.Tests 39
Total 77

GeneratePackageOnBuild is off — pack explicitly with the command above.


Status

0.3.0 (2026-11-12, scadaproj#55) — PUBLISHED (both packages, packed from the tree of scadaproj main 2011c8d (committed as df9a328 before rebasing onto main; identical ZB.MOM.WW.Telemetry/ tree); 38 + 39 tests green in Release). SHA256 (feed download = local pack): ZB.MOM.WW.Telemetry.0.3.0.nupkg a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2, ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg 04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c. Adopted by all four apps the same day.

  • service.version: ServiceVersion = null (default) now resolves to the entry assembly's AssemblyInformationalVersion (else its assembly version); "" omits; any value overrides. Behaviour change from 0.2.x, where null omitted the attribute.
  • ResourceAttributes (extra Resource attributes on all signals; first-class keys reserved, rejected at startup).
  • ApplyConfiguration(IConfiguration section) — Exporter, OtlpEndpoint, ServiceVersion, SiteId, NodeRole, ExportLogs, ResourceAttributes:<key>; absent keys keep code defaults.
  • Logs over OTLP: AddZbTelemetry registers its options in DI. The AddZbSerilog OTLP log sink follows them when its own options do not select OTLP. Root cause of "no app ever exported a log": the sink keyed only on the AddZbSerilog options, which every app filled with the service name alone. ExportLogs = false keeps logs local.
  • Includes the never-published 0.2.1 fixes below, including the stricter OTLP endpoint check. That check was shipped deliberately: no checked-in config sets an endpoint, it only runs when Exporter=Otlp, and no wonder tier has a collector (#54).
  • Feed versions: 0.1.0, 0.2.0, 0.3.0 (0.2.1 was never published).

0.2.1 (2026-11-09, scadaproj#48) — never published; shipped inside 0.3.0. P3 review fixes: RedactionEnricher is now registered AFTER ConfigureLogger, so enrichers added in the hook are redacted (009); dictionary keys that print identically are all kept (later ones get a #n suffix in the redactor's view; the original key is restored on rebuild) (010); the OTLP endpoint must be http/https with a host (collector:4317 is now rejected) and the pre-built-options AddZbTelemetry(services, options) overload validates ServiceName + endpoint like the lambda overloads (011). Behaviour change: a scheme-less OTLP endpoint that previously passed validation now throws at startup.

0.2.0 (2026-11-09, scadaproj#12) — PUBLISHED 2026-11-12 (both packages, packed from 01c6dc1 — the 0.2.0 code plus the corrected RepositoryUrl, i.e. before the 0.2.1 commit 1786bab; 12 + 30 tests green in Release). SHA256: ZB.MOM.WW.Telemetry.0.2.0.nupkg 71749cb3c34a0d0aba3eca43a6c12398b427f494e8c4c6f216db0be7d3ab40dc, ZB.MOM.WW.Telemetry.Serilog.0.2.0.nupkg 2aef391e7ab28061c5b3365c62c73e5c972ba8bcecd88bd0e0471550cfcc40c0 (feed download matches the local pack). First consumer: ScadaBridge (#94). Additive: a new AddZbSerilog(builder, configure, configureSerilog) overload taking ZbSerilogOptions — MinimumLevelGovernor (Func<IConfiguration, LogEventLevel?>, applied with MinimumLevel.Is AFTER ReadFrom.Configuration, so an app-owned key outranks Serilog:MinimumLevel; null defers to config; namespace Override:* entries still apply) and ConfigureLogger (runs last, for filters such as ScadaBridge's standby active-node exclusion). The two-argument overload is unchanged. This is the library half of ScadaBridge's AddZbSerilog adoption; the app-side switch is a follow-on in that repo.

Built at 0.1.0, published to the Gitea NuGet feed, and adopted across all three apps on 2026-06-01 (branch feat/adopt-zb-telemetry per repo, behaviour-preserving). MxAccessGateway's MEL→Serilog migration and metrics export both landed in this pass (not beforehand, despite the earlier claim). Deferred follow-ons (MxGateway ms→s + Meter rename, ScadaBridge app instruments

  • Site-node HTTP/1.1 metrics listener, OTLP wiring) are tracked in the component backlog (remaining open items are Gitea issues: #12 min-level governance hook — closed; #13 OTLP collector — closed 2026-11-12):
  • ~/Desktop/scadaproj/components/observability/GAPS.md — adoption status + deferred follow-ons

Design documentation:

  • ~/Desktop/scadaproj/components/observability/spec/SPEC.md — normalized observability target
  • ~/Desktop/scadaproj/components/observability/spec/METRIC-CONVENTIONS.md — metric naming reference
  • ~/Desktop/scadaproj/components/observability/shared-contract/ZB.MOM.WW.Telemetry.md — proposed shared-library API
  • ~/Desktop/scadaproj/components/observability/current-state/ — per-project current state (code-verified)

Index history (migrated from the scadaproj root CLAUDE.md)

Moved verbatim out of the scadaproj root CLAUDE.md on 2026-09-03 so it loads only when needed. This file is now the authoritative place for this history — append new entries here and keep the root index row short.

Narrative

The observability component is fully populated: a normalized spec, a metric-conventions reference, a shared-contract, three current-state docs, and an adoption GAPS backlog. Shared = OTel Resource (service.name/site.id/node.role identity triple) + standard instrumentation (ASP.NET Core, HttpClient, gRPC client, runtime, process) + Prometheus always-on exporter + OTLP opt-in

  • Serilog two-stage bootstrap + SiteId/NodeRole/NodeHostname enrichers + TraceContextEnricher (trace_id/span_id)
  • ILogRedactor seam; left per-project = application Meters/ActivitySources, sink config, per-operation enrichers, and redaction policies.

The shared library is built and lives in this repo at ZB.MOM.WW.Telemetry/ (.NET 10; 2 packages — ZB.MOM.WW.Telemetry, ZB.MOM.WW.Telemetry.Serilog; 34 tests (verified 2026-07-18); dotnet pack → 2 nupkgs @ 0.1.0). Adopted across all three apps on 2026-06-01 (branch feat/adopt-zb-telemetry per repo, behaviour-preserving): AddZbTelemetry (Resource + standard instrumentation + Prometheus /metrics) everywhere; OtOpcUa + MxGateway on AddZbSerilog (MxGateway's MEL→Serilog migration + metrics export both landed in this pass — they were not actually done beforehand despite an earlier claim); ScadaBridge keeps its LoggerConfigurationFactory (min-level governance) and only adds the shared TraceContextEnricher. Deferred: MxGateway ms→s + Meter rename, ScadaBridge app instruments + Site-node HTTP/1.1 metrics listener, OTLP wiring. Per-repo result tracked in components/observability/GAPS.md. Build/test from ZB.MOM.WW.Telemetry/: dotnet test. Consumer matrix: all three apps consume both packages after adoption (OtOpcUa, MxGateway Server, ScadaBridge Host + any instrumented project).