Files
scadaproj/ZB.MOM.WW.Telemetry
Joseph Doherty cfc7168ace feat(telemetry): 0.3.0 service.version default, ResourceAttributes, logs follow AddZbTelemetry OTLP (#55)
- service.version: null ServiceVersion now defaults to the entry assembly's
  AssemblyInformationalVersion (the family apps stamp the git SHA there); "" omits it.
- ResourceAttributes: extra Resource attributes on every signal; first-class keys
  (service.*, host.name, site.id, node.role) are reserved and rejected at startup.
- Logs over OTLP: AddZbTelemetry publishes its options in DI and AddZbSerilog's OTLP
  log sink follows them when its own options do not select Otlp, so
  <App>:Telemetry:Exporter=Otlp now switches logs along with metrics and traces, with
  the same endpoint and Resource. ExportLogs=false opts out.
- Ships the unpublished 0.2.1 review fixes, including the stricter OTLP endpoint check
  (scheme-less host:port rejected when Exporter=Otlp); no app config sets an endpoint.
- Tests 42 -> 74, incl. a loopback OTLP/gRPC receiver proving logs arrive with the
  Resource, and an ExportLogs=false case with a metrics positive control.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 21:28:50 -05:00
..

ZB.MOM.WW.Telemetry

Observability libraries for the ZB.MOM.WW SCADA family (OtOpcUa, MxAccessGateway, ScadaBridge). These are libraries, not a service — each package is linked directly into the consuming application at build time. There is no central telemetry process; all instrumentation runs in-process alongside the application.

The library normalizes the three-project observability surface: a shared OpenTelemetry Resource identity, standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and trace↔log correlation — so metrics, traces, and log lines from the same node carry identical dimensions and can join up in any backend.


Packages

Package Description Key Dependencies
ZB.MOM.WW.Telemetry AddZbTelemetry extension, ZbTelemetryOptions, shared OTel Resource builder (ZbResource), standard instrumentation (ASP.NET Core, HttpClient, gRPC client, runtime, process), Prometheus always-on exporter + OTLP opt-in overlay, app.MapZbMetrics() endpoint extension. Microsoft.AspNetCore.App (framework ref), OpenTelemetry.* stack
ZB.MOM.WW.Telemetry.Serilog AddZbSerilog extension, shared enrichers (SiteId/NodeRole/NodeHostname), TraceContextEnricher (writes trace_id/span_id from Activity.Current into every log event), ILogRedactor seam (per-project sensitive-field redaction), RedactionEnricher. ZB.MOM.WW.Telemetry, Serilog.* stack

The unifying hinge

The single ZbTelemetryOptions object drives both packages. Its identity triple — ServiceName → OTel Resource service.name, SiteId → site.id, NodeRole → node.role — is applied once and flows automatically to both the OpenTelemetry Resource (so every metric and span carries it) and the Serilog enrichers (so every log event carries it). A metric, a span, and a log line emitted by the same node share identical service.name, site.id, and node.role dimensions, enabling cross-signal correlation in any backend (Grafana, Jaeger, Seq, Loki, etc.) without per-project bookkeeping.


Consumer matrix

Consumer ZB.MOM.WW.Telemetry (core) ZB.MOM.WW.Telemetry.Serilog
OtOpcUa yes yes
MxAccessGateway yes yes (logging adopted — MEL → Serilog migration done)
ScadaBridge yes yes

All three apps consume both packages after adoption. MxAccessGateway's MEL→Serilog migration is the one in-pass adoption completed on its own branch; OtOpcUa and ScadaBridge adoption is follow-on (tracked in components/observability/GAPS.md).


OTel signals

AddZbTelemetry wires all three OpenTelemetry signals in a single call:

Signal What is wired
Metrics App Meters (via options.Meters[]) + standard: ASP.NET Core, HttpClient, .NET runtime, process. Exported via Prometheus (always on) with OTLP as an additive overlay.
Traces App ActivitySources (via options.ActivitySources[]) + standard: ASP.NET Core, HttpClient, gRPC client. Exported via OTLP when Exporter = ZbExporter.Otlp.
Logs Wired by AddZbSerilog (companion call). Serilog is used as the log sink; logs are bridged to OpenTelemetry via Serilog.Sinks.OpenTelemetry when OTLP is selected. 0.3.0+: the OTLP log sink follows the AddZbTelemetry registration in the same container, so Exporter = Otlp on AddZbTelemetry alone sends logs to the same endpoint with the same Resource (ExportLogs = false opts out). Options passed to AddZbSerilog that select OTLP themselves still win (for a logger built outside the host container, e.g. ScadaBridge).

Trace↔log correlation is automatic: TraceContextEnricher reads Activity.Current for each log event and attaches trace_id and span_id, so log events produced inside a traced request carry the same span identity as the trace backend.

Redaction reach. A registered ILogRedactor may remove or replace any value, and RedactionEnricher honours both (a removed key is dropped from the event). Scalar properties appear as their unwrapped CLR value; destructured properties are projected into mutable views the redactor can descend into — a {@Object} is an IDictionary<string, object?> of its fields, a logged collection an IList<object?>, a logged dictionary an IDictionary<string, object?> — all recursively, so a field nested inside a destructured object can be masked or removed:

if (properties["command"] is IDictionary<string, object?> command) command["apiKey"] = "***";

Structure type tags and dictionary keys are preserved on rebuild, and untouched properties are left intact (not reallocated). See the ILogRedactor XML doc for the full contract.


Resource identity (0.3.0+)

Attribute Source
service.name / service.namespace ServiceName (required) / ServiceNamespace (default ZB.MOM.WW)
service.version ServiceVersion; null (default) = the entry assembly's AssemblyInformationalVersion (the family apps stamp the git SHA there), "" = omit, any other value = override
site.id / node.role SiteId / NodeRole (omitted when null/empty)
service.instance.id / host.name computed: MachineName:ProcessId / MachineName
anything else ResourceAttributes (e.g. ["otopcua.cluster.id"] = "c1"); the keys above are reserved and rejected at startup

The same attribute set goes to metrics, traces and the OTLP log sink.

Exporter options

Prometheus is always wired for metrics regardless of the Exporter setting. OTLP is an additive overlay — set Exporter = ZbExporter.Otlp and OtlpEndpoint to push to a collector in addition to the scrape endpoint.

// Prometheus only (default — scrape /metrics)
builder.AddZbTelemetry(o =>
{
    o.ServiceName = "mxgateway";
    o.SiteId      = config["Site:Id"];
    o.NodeRole    = "standalone";
    o.Meters      = ["ZB.MOM.WW.MxGateway"];
});

// OTLP overlay (metrics + traces pushed to collector; /metrics still active)
builder.AddZbTelemetry(o =>
{
    o.ServiceName  = "mxgateway";
    o.SiteId       = config["Site:Id"];
    o.NodeRole     = "standalone";
    o.Meters       = ["ZB.MOM.WW.MxGateway"];
    o.Exporter     = ZbExporter.Otlp;
    o.OtlpEndpoint = "http://collector:4317";
});

// Mount the Prometheus scrape endpoint (call after app.UseRouting())
app.MapZbMetrics();   // → /metrics
// Serilog bootstrap (same options object drives enrichers)
builder.AddZbSerilog(o =>
{
    o.ServiceName = "mxgateway";
    o.SiteId      = config["Site:Id"];
    o.NodeRole    = "standalone";
});
// 0.2.0+: optional hooks for apps whose OWN key governs the minimum level (e.g. ScadaBridge's
// ScadaBridge:Logging:MinimumLevel, which must outrank Serilog:MinimumLevel). The governor runs
// AFTER ReadFrom.Configuration, so a non-null result overrides Serilog:MinimumLevel[:Default];
// null defers to config. Serilog:MinimumLevel:Override:* namespace entries still apply.
// ConfigureLogger runs last (filters, extra enrichers/sinks). Both default to null, in which
// case this is identical to the two-argument overload.
builder.AddZbSerilog(
    o => { o.ServiceName = "scadabridge"; o.SiteId = siteId; o.NodeRole = nodeRole; },
    s =>
    {
        s.MinimumLevelGovernor = cfg =>
            Enum.TryParse<LogEventLevel>(cfg["ScadaBridge:Logging:MinimumLevel"], true, out var l) ? l : null;
        s.ConfigureLogger = (lc, sp) => lc.Filter.ByExcluding(MyFilter);
    });

Building and testing

# from ZB.MOM.WW.Telemetry/
dotnet build ZB.MOM.WW.Telemetry.slnx
dotnet test  ZB.MOM.WW.Telemetry.slnx

All test assemblies run with no external dependencies (no running OTel collector, no Serilog backend):

Assembly Tests
ZB.MOM.WW.Telemetry.Tests 35
ZB.MOM.WW.Telemetry.Serilog.Tests 39
Total 74

The log-export tests start a loopback OTLP/gRPC receiver on Kestrel (no external collector).


Packing

dotnet pack ZB.MOM.WW.Telemetry.slnx -c Release -o ./artifacts

Produces two .nupkg files in artifacts/:

ZB.MOM.WW.Telemetry.0.3.0.nupkg
ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg

GeneratePackageOnBuild is off — pack explicitly as above. Both packages are versioned lockstep from Directory.Build.props.


Status

0.3.0 (scadaproj#55) is the current release on the Gitea feed: service.version defaults from the entry assembly, ResourceAttributes for extra Resource attributes, logs follow AddZbTelemetry's OTLP exporter, and the 0.2.1 review fixes (including the stricter OTLP endpoint check: a scheme-less host:port now fails at startup when Exporter = Otlp). Feed versions: 0.1.0, 0.2.0, 0.3.0 (0.2.1 was never published). Consumer pins: see CLAUDE.md. Adoption history is in the component backlog:

  • ~/Desktop/scadaproj/components/observability/GAPS.md

Design documentation lives alongside that backlog:

  • ~/Desktop/scadaproj/components/observability/spec/SPEC.md — normalized observability target
  • ~/Desktop/scadaproj/components/observability/spec/METRIC-CONVENTIONS.md — metric naming reference
  • ~/Desktop/scadaproj/components/observability/shared-contract/ZB.MOM.WW.Telemetry.md — proposed API
  • ~/Desktop/scadaproj/components/observability/current-state/ — per-project current state (code-verified)