scadaproj/components/observability/current-state/otopcua/CURRENT-STATE.md

Observability — current state: OtOpcUa

Repo: ~/Desktop/OtOpcUa. Stack: .NET 10, Akka.NET, OPC UA; solution ZB.MOM.WW.OtOpcUa.slnx. Telemetry code lives in two places: src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ (host-side bootstrap) and src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/ (instruments + enricher). All paths relative to repo root. Verified 2026-06-01.

The most complete observability implementation in the family: OpenTelemetry SDK with both metrics and tracing signals, Prometheus export, Serilog structured logging with a per-session correlation enricher, and a dedicated instrument vocabulary. The one significant gap: no OTel Resource / service.name, so all signals are indistinguishable from one another and from other fleet members in a backend.

1. Metrics (OpenTelemetry SDK)

Bootstrap — ObservabilityExtensions.cs

src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ObservabilityExtensions.cs:

• :18 — AddOtOpcUaObservability(IServiceCollection) is the service-registration entry point.
• :20 — AddOpenTelemetry() wires the OTel SDK.
• :21–23 — .WithMetrics(b => b.AddMeter(OtOpcUaTelemetry.MeterName).AddPrometheusExporter()): registers the application meter and attaches the Prometheus scrape exporter.
• :24–25 — .WithTracing(b => b.AddSource(OtOpcUaTelemetry.ActivitySourceName)): registers the application activity source for trace data.
• No ResourceBuilder call anywhere — service.name, service.namespace, service.version, site.id, and node.role are not set. The OTel SDK defaults to an empty/SDK-default Resource.
• :36 — MapOtOpcUaMetrics(IEndpointRouteBuilder) maps the Prometheus endpoint.
• :38 — endpoint path is /metrics.

Program.cs:

• :138 — builder.Services.AddOtOpcUaObservability()
• :160 — app.MapOtOpcUaMetrics()

Package refs in csproj: OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.Prometheus.AspNetCore. No OpenTelemetry.Exporter.OpenTelemetryProtocol — OTLP is not available; Prometheus is the only export path.

Instruments — OtOpcUaTelemetry.cs

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs:

• :19 — MeterName = "ZB.MOM.WW.OtOpcUa" (the Meter the SDK will collect).
• :20 — ActivitySourceName = "ZB.MOM.WW.OtOpcUa" (the ActivitySource for spans).

Instruments defined (all static readonly on OtOpcUaTelemetry):

Instrument	Kind	Unit	Subsystem
otopcua.deploy.applied	Counter<long>	—	deploy
otopcua.deploy.apply.duration	Histogram<double>	s	deploy
otopcua.driver.lifecycle	Counter<long>	—	driver
otopcua.virtualtag.eval	Counter<long>	—	virtual-tag
otopcua.scriptedalarm.transition	Counter<long>	—	scripted-alarm
otopcua.opcua.sink.write	Counter<long>	—	opc-ua sink
otopcua.redundancy.service_level_change	Counter<long>	—	redundancy

Two activity spans: otopcua.deploy.apply, otopcua.opcua.address_space_rebuild.

Naming convention: otopcua.<subsystem>.<event>. Duration histogram correctly uses unit s (OTel semantic conventions). No standard instrumentation (ASP.NET Core, HttpClient, runtime, gRPC client meters) is wired — only the bespoke application instruments.

2. Logging (Serilog)

Bootstrap

Program.cs:

• :49–52 — two-stage Serilog bootstrap: initial logger for startup, then full UseSerilog(ReadFrom.Configuration). Sinks: Console + rolling file logs/otopcua-.log.
• :141 — UseSerilogRequestLogging() on the WebApplication.

Correlation enricher — LogContextEnricher.cs

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/LogContextEnricher.cs:

• :18–36 — Push(driverInstanceId, driverType, capability, correlationId) calls LogContext.PushProperty for four properties:
  • DriverInstanceId — Galaxy driver instance GUID.
  • DriverType — driver type discriminator.
  • CapabilityName — OPC UA capability being exercised.
  • CorrelationId — caller-supplied correlation token.

This enricher is driver-lifecycle-scoped, not request-scoped — it pushes when a driver operation begins and is disposable to pop on completion.

No trace_id / span_id enricher. Although OtOpcUa creates ActivitySource spans, the active Activity.Current trace context is never pushed onto Serilog's LogContext. A log line emitted during a span cannot be correlated to the span in a backend.

No structural enrichers for service.name / site.id / node.role — these dimensions are absent from every log line. ScadaBridge has these; OtOpcUa does not.

3. Signal summary

Signal	Provider	Export	Resource / service.name
Metrics	OTel SDK (Meter + WithMetrics)	Prometheus /metrics	⛔ none
Traces	OTel SDK (ActivitySource + WithTracing)	⛔ none (no exporter configured)	⛔ none
Logs	Serilog	Console + rolling file	⛔ none (no service.name property)
Trace↔log correlation	—	—	⛔ absent (trace_id/span_id not pushed)

Note: WithTracing registers the ActivitySource for collection, but no exporter (OTLP or otherwise) is attached to the tracing pipeline. Spans are created and recorded by the SDK but never shipped anywhere — effectively a no-op in production.

4. Notable design choices

• Instrument naming follows <meter>.<subsystem>.<event> cleanly and consistently — this is the pattern the shared spec codifies as the fleet convention.
• Duration unit correctly uses s on otopcua.deploy.apply.duration — no conversion needed on adoption; this contrasts with MxAccessGateway's ms histograms.
• LogContextEnricher is bespoke but valuable — the DriverInstanceId/DriverType/CapabilityName correlation is OtOpcUa-specific domain context; it should survive adoption behind the shared enricher layer.
• No OTLP path — with no OTLP exporter, OtOpcUa cannot send metrics or traces to a collector (Prometheus is scrape-pull only). This limits operational flexibility.

---

Adoption plan → ZB.MOM.WW.Telemetry

Replace with shared bootstrap:

• AddOtOpcUaObservability() → builder.AddZbTelemetry(o => { o.ServiceName = "otopcua"; o.SiteId = ...; o.NodeRole = ...; o.Meters = [OtOpcUaTelemetry.MeterName]; o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName]; }). This adds the missing Resource (gains service.name / service.namespace / service.version / site.id / node.role / host.name on every metric and span). Prometheus /metrics stays the default exporter; OTLP becomes opt-in via options.
• Add standard instrumentation through AddZbTelemetry options: ASP.NET Core meters, HttpClient, runtime + process meters — none wired today.
• Fix the tracing no-op: wire an OTLP exporter (or at minimum note that tracing is recorded but not exported); AddZbTelemetry provides OTLP as the opt-in path.
• MapOtOpcUaMetrics → app.MapZbMetrics() (same /metrics path; shared convention).

Replace with shared Serilog bootstrap:

• Serilog bootstrap in Program.cs:49–52 → builder.AddZbSerilog(o => { o.ServiceName = "otopcua"; o.SiteId = ...; o.NodeRole = ...; }). This adds structural SiteId / NodeRole / NodeHostname properties to every log line (currently absent) and wires the TraceContextEnricher so trace_id/span_id appear on log lines emitted during active spans.
• Console + file sinks continue via ReadFrom.Configuration in appsettings.json — no sink changes needed.
• UseSerilogRequestLogging() stays.

Keep bespoke:

• OtOpcUaTelemetry.cs — the application Meter, ActivitySource, and all instrument definitions (otopcua.* counters, histograms, spans). These are domain instruments; AddZbTelemetry registers them by name but does not own them.
• LogContextEnricher.cs — driver-lifecycle correlation properties (DriverInstanceId, DriverType, CapabilityName, CorrelationId) are OtOpcUa-specific. The enricher continues to push via LogContext.PushProperty alongside the shared enrichers.
• ObservabilityExtensions.cs itself can be simplified or removed — it becomes a thin wrapper that calls AddZbTelemetry with OtOpcUa-specific options. The per-project entry point remains; only the implementation body is delegated to the shared library.

Adoption is a follow-on task (tracked in GAPS.md), not part of the ZB.MOM.WW.Telemetry library build. The library build delivers the shared bootstrap and enrichers; adoption lands in the OtOpcUa repo as a separate commit once the nupkg is available.