scadaproj/components/observability/spec/SPEC.md

Observability — normalized target spec

Status: Draft. The single design the sister projects converge on. Derived from the three code-verified current-state docs (../current-state/). Goal is path to shared code (../shared-contract/ZB.MOM.WW.Telemetry.md), so each normalized section maps to a shared library seam.

0. Scope

Normalized here: one OpenTelemetry bootstrap across all three signals (metrics + traces + logs) via a single AddZbTelemetry extension; the shared Resource attribute set (service.name / service.namespace / service.version / site.id / node.role / host.name) that makes every node distinguishable in a collector; standard instrumentation everyone enables (ASP.NET Core, HttpClient, gRPC client, runtime, process meters); exporter conventions (Prometheus scrape endpoint default, OTLP opt-in); a shared Serilog bootstrap with identity enrichers (SiteId, NodeRole from ZbTelemetryOptions; NodeHostname auto from Environment.MachineName) matching the OTel Resource dimensions (metrics and logs therefore carry identical dimensions); a TraceContextEnricher that stamps trace_id/span_id from Activity.Current onto every Serilog event, enabling log↔trace correlation; an ILogRedactor redaction seam.

Explicitly NOT normalized (domain-specific — keep per project): each app's actual instruments — otopcua.* meters and spans, mxgateway.* counters/histograms/gauges — they are registered through the shared bootstrap but their names and semantics remain bespoke (see METRIC-CONVENTIONS.md §4); the redaction policy (which field names, which command types) — only the ILogRedactor seam is shared, each project supplies its own implementation; the MxGateway net48 x86 worker's IWorkerLogger (stderr key=value format, out-of-process, out of scope).

1. OpenTelemetry pipeline — AddZbTelemetry

A single IHostApplicationBuilder extension is the front door for all three OTel signals. It wires the shared Resource, registers standard instrumentation, and configures the selected exporter:

builder.AddZbTelemetry(o =>
{
    o.ServiceName      = "mxgateway";          // populates Resource service.name
    o.ServiceNamespace = "ZB.MOM.WW";          // constant across the fleet (default)
    o.ServiceVersion   = "1.0.0";              // populated from AssemblyInformationalVersion
    o.SiteId           = cfg.SiteId;           // Resource site.id + Serilog SiteId property
    o.NodeRole         = cfg.NodeRole;         // Resource node.role + Serilog NodeRole property
    o.Meters           = ["MxGateway.Server"]; // app's own Meter name(s)
    o.ActivitySources  = ["MxGateway.Server"]; // app's own ActivitySource name(s)
    o.Exporter         = ZbExporter.Prometheus; // default; ZbExporter.Otlp opt-in
    // o.OtlpEndpoint  = "http://collector:4317"; // required when Exporter = Otlp
});

app.MapZbMetrics(); // mounts Prometheus /metrics scrape endpoint

This is the headline fix: nobody in the fleet sets a Resource or service.name today, making every node indistinguishable in a collector. Every project must call AddZbTelemetry to be observable.

IServiceCollection overload: AddZbTelemetry also has an IServiceCollection-based overload for host configurations where IHostApplicationBuilder is not available (detailed in the shared-contract). The IHostApplicationBuilder overload is the primary path for all three apps on .NET 10.

2. Shared Resource

The OTel Resource attached to all three signals is built from ZbTelemetryOptions:

OTel attribute	Options property	Notes
service.name	ServiceName	Required. Lower-case short identifier (otopcua, mxgateway, scadabridge)
service.namespace	ServiceNamespace	Default "ZB.MOM.WW" — constant across the fleet
service.version	ServiceVersion	Optional; recommend populating from AssemblyInformationalVersion
service.instance.id	(auto)	Always populated from ZbResource.InstanceId — a deterministic MachineName:ProcessId. The OTel SDK's random-GUID default is disabled so every signal from the same process carries an identical, restart-stable instance id for cross-signal correlation
site.id	SiteId	Optional; identifies the physical/logical site
node.role	NodeRole	Optional; e.g. "central", "site", "hub"
host.name	(auto)	Always populated from Environment.MachineName

The same SiteId and NodeRole values are passed to the Serilog enrichers (§4) so a metric, a span, and a log line from the same node carry identical dimensions and join up in any OTel-compatible backend.

3. Standard instrumentation

AddZbTelemetry enables the following instrumentation for all projects. Any project that already enables a subset gets it consolidated; no project may skip this baseline:

Instrumentation	Package	Signal
ASP.NET Core	OpenTelemetry.Instrumentation.AspNetCore	Traces + Metrics
HttpClient	OpenTelemetry.Instrumentation.Http	Traces + Metrics
gRPC client	OpenTelemetry.Instrumentation.GrpcNetClient	Traces
.NET runtime	OpenTelemetry.Instrumentation.Runtime	Metrics
Process	OpenTelemetry.Instrumentation.Process	Metrics

App-specific Meter names and ActivitySource names are registered via o.Meters and o.ActivitySources. This is how MxGateway's hand-rolled GatewayMetrics finally gets an export path instead of dying in an in-memory GetSnapshot().

4. Exporter conventions

ZbTelemetryOptions.Exporter selects the export path:

Value	Behaviour
ZbExporter.Prometheus	Mounts a Prometheus /metrics scrape endpoint via app.MapZbMetrics(). Default for all three apps — consistent with OtOpcUa's existing /metrics.
ZbExporter.Otlp	Exports to an OTLP endpoint specified by o.OtlpEndpoint (gRPC, http://collector:4317). Opt-in path to a real OTel Collector; coexists with Prometheus.

Both exporters carry the shared Resource. OTLP is the path to a real backend (Tempo, Prometheus-remote-write, Loki); Prometheus covers the "scrape from the node" case that all three apps currently use or aspire to.

5. Serilog logging stack

AddZbSerilog is a companion extension in the .Serilog package. It registers the Serilog application logger in DI and applies a set of fixed enrichers:

No process-global state. AddZbSerilog does NOT set or freeze the static Log.Logger. It passes preserveStaticLogger: true to AddSerilog so the global logger is left entirely untouched. This keeps the library safe for multi-host processes (integration tests, Aspire, test suites) — calling AddZbSerilog twice in the same process must never throw "The logger is already frozen".

The optional Stage-1 bootstrap logger (CreateBootstrapLogger) — a minimal console-only Log.Logger for capturing startup exceptions before IConfiguration is available — is the application's responsibility, not the library's. Apps that need it should add this to Program.cs before calling AddZbSerilog:

Log.Logger = new LoggerConfiguration().WriteTo.Console().CreateBootstrapLogger();

Application logger (DI): reads sinks and overrides from IConfiguration (ReadFrom.Configuration) and applies a set of fixed enrichers:

Enricher	Property name	Source
ZbLogEnricherNames.SiteId	"SiteId"	ZbTelemetryOptions.SiteId
ZbLogEnricherNames.NodeRole	"NodeRole"	ZbTelemetryOptions.NodeRole
ZbLogEnricherNames.NodeHostname	"NodeHostname"	Environment.MachineName
TraceContextEnricher	trace_id, span_id	Activity.Current
RedactionEnricher	(project-defined fields)	ILogRedactor implementation

SiteId and NodeRole are bound from the same ZbTelemetryOptions object as the OTel Resource; NodeHostname is populated automatically from Environment.MachineName (not a caller-supplied option). All three identity properties appear on logs and metrics/traces alike, so signals from the same node carry identical dimensions. When no Activity.Current is present (e.g. background services, startup), TraceContextEnricher emits nothing — it does not inject empty or zero values.

MinimumLevel is set explicitly in code (default Information) and can be overridden via IConfiguration (Serilog:MinimumLevel). Sinks are fully config-driven: ReadFrom.Configuration reads Serilog:WriteTo from appsettings.json / environment.

Per-project config paths: AddZbSerilog reads Serilog:MinimumLevel from IConfiguration. Callers that bind MinimumLevel from a different key (e.g. ScadaBridge's ScadaBridge:Logging:MinimumLevel) apply that override themselves before or after calling AddZbSerilog. The config key for MinimumLevel remains per-project; AddZbSerilog is not parameterized on it.

OTel log export is wired in the same call: logs flow through the OTel pipeline with the same Resource attached, making all three signals (metrics / traces / logs) available in a single backend.

6. Redaction seam — ILogRedactor

ILogRedactor is a single-method interface that receives the mutable log-event property dictionary and scrubs any fields that must not leave the process:

public interface ILogRedactor
{
    void Redact(IDictionary<string, object?> properties);
}

RedactionEnricher applies a registered ILogRedactor on every log event. The seam is shared; the policy is per-project (which field names, which command types, which classification levels). MxGateway's existing GatewayLogRedactor is the reference implementation; it migrates to this seam during adoption. If no ILogRedactor is registered, RedactionEnricher is a no-op.

This preserves the operational property MxGateway already has (secrets never leave the process in log events) while making the plumbing reusable.

7. Per-project migration

Project	Current state	Primary gaps	What normalizes
OtOpcUa	Full OTel SDK (WithMetrics + WithTracing); Prometheus /metrics; Serilog bootstrap; 7 instruments + 2 spans.	No Resource / service.name anywhere; no trace↔log correlation; no SiteId/NodeRole enrichers.	Call AddZbTelemetry (adds Resource; consolidates standard instrumentation); call AddZbSerilog (adds TraceContextEnricher + identity enrichers); remove bespoke Serilog bootstrap (if a pre-Build bootstrap logger is wanted, add Log.Logger = new LoggerConfiguration()...CreateBootstrapLogger(); in Program.cs).
MxGateway	Hand-rolled GatewayMetrics (13 counters / 3 histograms ms / 4 gauges); in-memory snapshot only — no export; MEL logging with GatewayLogScope correlation + GatewayLogRedactor; no OTel SDK.	No OTel SDK; no export; ms histograms diverge from OTel semconv (s); MEL → Serilog migration; no Resource.	Call AddZbTelemetry (wires OTel SDK around existing GatewayMetrics — finally exports); call AddZbSerilog (replaces MEL; re-expresses GatewayLogScope as LogContext.PushProperty; moves GatewayLogRedactor behind ILogRedactor). Duration unit convergence (ms→s) tracked in GAPS. This is the one adoption done now.
ScadaBridge	OpenTelemetry.Api ref only (dangling — CVE-patch origin, zero usage); Serilog bootstrap (LoggerConfigurationFactory) with SiteId/NodeRole/NodeHostname enrichers.	No OTel SDK; no metrics; no tracing; no export; no trace↔log correlation. ScadaBridge's enricher property names are already the target names — migration is additive.	Call AddZbTelemetry (adds OTel SDK + metrics + traces + export); call AddZbSerilog (consolidates LoggerConfigurationFactory; adds TraceContextEnricher); if a pre-Build bootstrap logger is wanted, set Log.Logger = new LoggerConfiguration()...CreateBootstrapLogger(); in Program.cs.

The MxGateway logging migration (MEL → Serilog, re-expressing GatewayLogRedactor behind ILogRedactor) is the only sister-repo touch in scope for this release. OtOpcUa and ScadaBridge adoption is deferred to the follow-on tracked in ../GAPS.md.

8. Acceptance (what "converged" means)

A project is converged when: (a) it calls builder.AddZbTelemetry(o => ...) with all required Resource attributes populated; (b) it calls app.MapZbMetrics() (or configures OTLP); (c) it calls builder.AddZbSerilog(...) and the TraceContextEnricher stamps trace_id/span_id on every log event emitted under an active Activity; (d) its ILogRedactor implementation (if applicable) is registered and applied by RedactionEnricher; (e) every node in the fleet is distinguishable by service.name + site.id + node.role in a collector or log aggregator.