Files
scadaproj/ZB.MOM.WW.Telemetry
Joseph Doherty 88c557dee8 fix(telemetry): identical resource across all 3 signals (symmetric OTLP trigger + deterministic service.instance.id)
Fix 1 — symmetric OTLP trigger: ZbSerilogConfig.ApplyOpenTelemetryExport now activates only
when options.Exporter == ZbExporter.Otlp, matching the core OTel metrics/traces path. The
previous fallback that also triggered on a bare OtlpEndpoint is removed; OtlpEndpoint is the
address to use when Otlp is selected, not an independent enable.

Fix 2 — deterministic service.instance.id: ZbResource.InstanceId (MachineName:ProcessId) is
a new public property that produces a stable, process-unique id without a random GUID.
ZbResource.Configure passes autoGenerateServiceInstanceId:false + serviceInstanceId:InstanceId
so metrics and traces never get a random auto-generated id. ZbSerilogConfig.BuildResourceAttributes
adds service.instance.id from ZbResource.InstanceId so the Serilog OTLP log sink carries the
exact same value — all three signals now share an identical resource for cross-signal joins.

Tests: +2 in ZbResourceTests (InstanceId determinism, no-GUID check), +2 in RedactionTests
(service.instance.id parity assertion in BuildResourceAttributes, symmetric OTLP trigger tests).
Total: 9 + 14 = 23 tests, all green.
2026-06-01 08:26:09 -04:00
..

ZB.MOM.WW.Telemetry

Observability libraries for the ZB.MOM.WW SCADA family (OtOpcUa, MxAccessGateway, ScadaBridge). These are libraries, not a service — each package is linked directly into the consuming application at build time. There is no central telemetry process; all instrumentation runs in-process alongside the application.

The library normalizes the three-project observability surface: a shared OpenTelemetry Resource identity, standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and trace↔log correlation — so metrics, traces, and log lines from the same node carry identical dimensions and can join up in any backend.


Packages

Package Description Key Dependencies
ZB.MOM.WW.Telemetry AddZbTelemetry extension, ZbTelemetryOptions, shared OTel Resource builder (ZbResource), standard instrumentation (ASP.NET Core, HttpClient, gRPC client, runtime, process), Prometheus always-on exporter + OTLP opt-in overlay, app.MapZbMetrics() endpoint extension. Microsoft.AspNetCore.App (framework ref), OpenTelemetry.* stack
ZB.MOM.WW.Telemetry.Serilog AddZbSerilog extension, shared enrichers (SiteId/NodeRole/NodeHostname), TraceContextEnricher (writes trace_id/span_id from Activity.Current into every log event), ILogRedactor seam (per-project sensitive-field redaction), RedactionEnricher. ZB.MOM.WW.Telemetry, Serilog.* stack

The unifying hinge

The single ZbTelemetryOptions object drives both packages. Its identity triple — ServiceName → OTel Resource service.name, SiteIdsite.id, NodeRolenode.role — is applied once and flows automatically to both the OpenTelemetry Resource (so every metric and span carries it) and the Serilog enrichers (so every log event carries it). A metric, a span, and a log line emitted by the same node share identical service.name, site.id, and node.role dimensions, enabling cross-signal correlation in any backend (Grafana, Jaeger, Seq, Loki, etc.) without per-project bookkeeping.


Consumer matrix

Consumer ZB.MOM.WW.Telemetry (core) ZB.MOM.WW.Telemetry.Serilog
OtOpcUa yes yes
MxAccessGateway yes yes (logging adopted — MEL → Serilog migration done)
ScadaBridge yes yes

All three apps consume both packages after adoption. MxAccessGateway's MEL→Serilog migration is the one in-pass adoption completed on its own branch; OtOpcUa and ScadaBridge adoption is follow-on (tracked in components/observability/GAPS.md).


OTel signals

AddZbTelemetry wires all three OpenTelemetry signals in a single call:

Signal What is wired
Metrics App Meters (via options.Meters[]) + standard: ASP.NET Core, HttpClient, .NET runtime, process. Exported via Prometheus (always on) with OTLP as an additive overlay.
Traces App ActivitySources (via options.ActivitySources[]) + standard: ASP.NET Core, HttpClient, gRPC client. Exported via OTLP when Exporter = ZbExporter.Otlp.
Logs Wired by AddZbSerilog (companion call). Serilog is used as the log sink; logs are bridged to OpenTelemetry via Serilog.Sinks.OpenTelemetry when configured.

Trace↔log correlation is automatic: TraceContextEnricher reads Activity.Current for each log event and attaches trace_id and span_id, so log events produced inside a traced request carry the same span identity as the trace backend.


Exporter options

Prometheus is always wired for metrics regardless of the Exporter setting. OTLP is an additive overlay — set Exporter = ZbExporter.Otlp and OtlpEndpoint to push to a collector in addition to the scrape endpoint.

// Prometheus only (default — scrape /metrics)
builder.AddZbTelemetry(o =>
{
    o.ServiceName = "mxgateway";
    o.SiteId      = config["Site:Id"];
    o.NodeRole    = "standalone";
    o.Meters      = ["ZB.MOM.WW.MxGateway"];
});

// OTLP overlay (metrics + traces pushed to collector; /metrics still active)
builder.AddZbTelemetry(o =>
{
    o.ServiceName  = "mxgateway";
    o.SiteId       = config["Site:Id"];
    o.NodeRole     = "standalone";
    o.Meters       = ["ZB.MOM.WW.MxGateway"];
    o.Exporter     = ZbExporter.Otlp;
    o.OtlpEndpoint = "http://collector:4317";
});

// Mount the Prometheus scrape endpoint (call after app.UseRouting())
app.MapZbMetrics();   // → /metrics
// Serilog bootstrap (same options object drives enrichers)
builder.AddZbSerilog(o =>
{
    o.ServiceName = "mxgateway";
    o.SiteId      = config["Site:Id"];
    o.NodeRole    = "standalone";
});

Building and testing

# from ZB.MOM.WW.Telemetry/
dotnet build ZB.MOM.WW.Telemetry.slnx
dotnet test  ZB.MOM.WW.Telemetry.slnx

All test assemblies run with no external dependencies (no running OTel collector, no Serilog backend):

Assembly Tests
ZB.MOM.WW.Telemetry.Tests 7
ZB.MOM.WW.Telemetry.Serilog.Tests 12
Total 19

Packing

dotnet pack ZB.MOM.WW.Telemetry.slnx -c Release -o ./artifacts

Produces two .nupkg files in artifacts/:

ZB.MOM.WW.Telemetry.0.1.0.nupkg
ZB.MOM.WW.Telemetry.Serilog.0.1.0.nupkg

GeneratePackageOnBuild is off — pack explicitly as above. Both packages are versioned lockstep from Directory.Build.props.


Status

Built at 0.1.0. MxAccessGateway logging adopted (MEL → Serilog migration, on its own branch). Broader OtOpcUa and ScadaBridge telemetry adoption deferred. Adoption is tracked in the component backlog:

  • ~/Desktop/scadaproj/components/observability/GAPS.md

Design documentation lives alongside that backlog:

  • ~/Desktop/scadaproj/components/observability/spec/SPEC.md — normalized observability target
  • ~/Desktop/scadaproj/components/observability/spec/METRIC-CONVENTIONS.md — metric naming reference
  • ~/Desktop/scadaproj/components/observability/shared-contract/ZB.MOM.WW.Telemetry.md — proposed API
  • ~/Desktop/scadaproj/components/observability/current-state/ — per-project current state (code-verified)