Files
scadaproj/components/observability/current-state/otopcua/CURRENT-STATE.md
T
Joseph Doherty e1a2df83f7 feat(infra): shared dev OTel collector; OTLP proven end to end for all four apps (#13)
- infra/otel-collector/: zb-otel-collector (otelcol-contrib 0.162.0) on 10.100.0.35 —
  OTLP gRPC :4317 / HTTP :4318, debug + file exporters, Prometheus re-export :8889,
  health :13133; README runbook + otelsum.py summariser. Deployed.
- components/observability/2026-11-12-otlp-end-to-end-evidence.md: metrics + traces
  received from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway;
  OtOpcUa Galaxy driver allowlist defect fixed in lmxopcua#892 (db86b996).
- GAPS.md, current-state banners, index row, Telemetry CLAUDE.md: #13 closed; follow-ups
  #54 (wonder-tier collector), #55 (resource identity + logs over OTLP), ScadaBridge#133
  (Central role). Telemetry 0.2.1 deliberately left unpublished (no lib change needed).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:48:36 -05:00

9.3 KiB
Raw Blame History

Observability — current state: OtOpcUa

2026-11-12 update (scadaproj#13): this is the pre-adoption 2026-06-01 snapshot. OtOpcUa now calls AddZbTelemetry (Resource + OTLP opt-in), and OTLP export of metrics and traces is proven end to end against the dev collector (otopcua.deploy.apply, otopcua.opcua.address_space_rebuild, otopcua.deploy.* metrics arrived). That ends the "tracing no-op" below once OtOpcUa:Telemetry:Exporter=Otlp is set. The Galaxy driver's galaxy.* spans and galaxy.events.* counters were missing from the allowlists; fixed in lmxopcua#892 (master db86b996). Still absent from the Resource: service.version, site.id, node.role (scadaproj#55). Evidence: 2026-11-12-otlp-end-to-end-evidence.md.

Repo: ~/Desktop/OtOpcUa. Stack: .NET 10, Akka.NET, OPC UA; solution ZB.MOM.WW.OtOpcUa.slnx. Telemetry code lives in two places: src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ (host-side bootstrap) and src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/ (instruments + enricher). All paths relative to repo root. Verified 2026-06-01.

The most complete observability implementation in the family: OpenTelemetry SDK with both metrics and tracing signals, Prometheus export, Serilog structured logging with a per-session correlation enricher, and a dedicated instrument vocabulary. The one significant gap: no OTel Resource / service.name, so all signals are indistinguishable from one another and from other fleet members in a backend.

1. Metrics (OpenTelemetry SDK)

Bootstrap — ObservabilityExtensions.cs

src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ObservabilityExtensions.cs:

  • :18 — AddOtOpcUaObservability(IServiceCollection) is the service-registration entry point.
  • :20 — AddOpenTelemetry() wires the OTel SDK.
  • :21–23 — .WithMetrics(b => b.AddMeter(OtOpcUaTelemetry.MeterName).AddPrometheusExporter()): registers the application meter and attaches the Prometheus scrape exporter.
  • :24–25 — .WithTracing(b => b.AddSource(OtOpcUaTelemetry.ActivitySourceName)): registers the application activity source for trace data.
  • No ResourceBuilder call anywhere — service.name, service.namespace, service.version, site.id, and node.role are not set. The OTel SDK defaults to an empty/SDK-default Resource.
  • :36 — MapOtOpcUaMetrics(IEndpointRouteBuilder) maps the Prometheus endpoint.
  • :38 — endpoint path is /metrics.

Program.cs:

  • :138 — builder.Services.AddOtOpcUaObservability()
  • :160 — app.MapOtOpcUaMetrics()

Package refs in csproj: OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.Prometheus.AspNetCore. No OpenTelemetry.Exporter.OpenTelemetryProtocol — OTLP is not available; Prometheus is the only export path.

Instruments — OtOpcUaTelemetry.cs

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs:

  • :19 — MeterName = "ZB.MOM.WW.OtOpcUa" (the Meter the SDK will collect).
  • :20 — ActivitySourceName = "ZB.MOM.WW.OtOpcUa" (the ActivitySource for spans).

Instruments defined (all static readonly on OtOpcUaTelemetry):

Instrument Kind Unit Subsystem
otopcua.deploy.applied Counter<long> — deploy
otopcua.deploy.apply.duration Histogram<double> s deploy
otopcua.driver.lifecycle Counter<long> — driver
otopcua.virtualtag.eval Counter<long> — virtual-tag
otopcua.scriptedalarm.transition Counter<long> — scripted-alarm
otopcua.opcua.sink.write Counter<long> — opc-ua sink
otopcua.redundancy.service_level_change Counter<long> — redundancy

Two activity spans: otopcua.deploy.apply, otopcua.opcua.address_space_rebuild.

Naming convention: otopcua.<subsystem>.<event>. Duration histogram correctly uses unit s (OTel semantic conventions). No standard instrumentation (ASP.NET Core, HttpClient, runtime, gRPC client meters) is wired — only the bespoke application instruments.

2. Logging (Serilog)

Bootstrap

Program.cs:

  • :49–52 — two-stage Serilog bootstrap: initial logger for startup, then full UseSerilog(ReadFrom.Configuration). Sinks: Console + rolling file logs/otopcua-.log.
  • :141 — UseSerilogRequestLogging() on the WebApplication.

Correlation enricher — LogContextEnricher.cs

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/LogContextEnricher.cs:

  • :18–36 — Push(driverInstanceId, driverType, capability, correlationId) calls LogContext.PushProperty for four properties:
    • DriverInstanceId — Galaxy driver instance GUID.
    • DriverType — driver type discriminator.
    • CapabilityName — OPC UA capability being exercised.
    • CorrelationId — caller-supplied correlation token.

This enricher is driver-lifecycle-scoped, not request-scoped — it pushes when a driver operation begins and is disposable to pop on completion.

No trace_id / span_id enricher. Although OtOpcUa creates ActivitySource spans, the active Activity.Current trace context is never pushed onto Serilog's LogContext. A log line emitted during a span cannot be correlated to the span in a backend.

No structural enrichers for service.name / site.id / node.role — these dimensions are absent from every log line. ScadaBridge has these; OtOpcUa does not.

3. Signal summary

Signal Provider Export Resource / service.name
Metrics OTel SDK (Meter + WithMetrics) Prometheus /metrics ⛔ none
Traces OTel SDK (ActivitySource + WithTracing) ⛔ none (no exporter configured) ⛔ none
Logs Serilog Console + rolling file ⛔ none (no service.name property)
Trace↔log correlation — — ⛔ absent (trace_id/span_id not pushed)

Note: WithTracing registers the ActivitySource for collection, but no exporter (OTLP or otherwise) is attached to the tracing pipeline. Spans are created and recorded by the SDK but never shipped anywhere — effectively a no-op in production.

4. Notable design choices

  • Instrument naming follows <meter>.<subsystem>.<event> cleanly and consistently — this is the pattern the shared spec codifies as the fleet convention.
  • Duration unit correctly uses s on otopcua.deploy.apply.duration — no conversion needed on adoption; this contrasts with MxAccessGateway's ms histograms.
  • LogContextEnricher is bespoke but valuable — the DriverInstanceId/DriverType/CapabilityName correlation is OtOpcUa-specific domain context; it should survive adoption behind the shared enricher layer.
  • No OTLP path — with no OTLP exporter, OtOpcUa cannot send metrics or traces to a collector (Prometheus is scrape-pull only). This limits operational flexibility.

Adoption plan → ZB.MOM.WW.Telemetry

Replace with shared bootstrap:

  • AddOtOpcUaObservability() → builder.AddZbTelemetry(o => { o.ServiceName = "otopcua"; o.SiteId = ...; o.NodeRole = ...; o.Meters = [OtOpcUaTelemetry.MeterName]; o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName]; }). This adds the missing Resource (gains service.name / service.namespace / service.version / site.id / node.role / host.name on every metric and span). Prometheus /metrics stays the default exporter; OTLP becomes opt-in via options.
  • Add standard instrumentation through AddZbTelemetry options: ASP.NET Core meters, HttpClient, runtime + process meters — none wired today.
  • Fix the tracing no-op: wire an OTLP exporter (or at minimum note that tracing is recorded but not exported); AddZbTelemetry provides OTLP as the opt-in path.
  • MapOtOpcUaMetrics → app.MapZbMetrics() (same /metrics path; shared convention).

Replace with shared Serilog bootstrap:

  • Serilog bootstrap in Program.cs:49–52 → builder.AddZbSerilog(o => { o.ServiceName = "otopcua"; o.SiteId = ...; o.NodeRole = ...; }). This adds structural SiteId / NodeRole / NodeHostname properties to every log line (currently absent) and wires the TraceContextEnricher so trace_id/span_id appear on log lines emitted during active spans.
  • Console + file sinks continue via ReadFrom.Configuration in appsettings.json — no sink changes needed.
  • UseSerilogRequestLogging() stays.

Keep bespoke:

  • OtOpcUaTelemetry.cs — the application Meter, ActivitySource, and all instrument definitions (otopcua.* counters, histograms, spans). These are domain instruments; AddZbTelemetry registers them by name but does not own them.
  • LogContextEnricher.cs — driver-lifecycle correlation properties (DriverInstanceId, DriverType, CapabilityName, CorrelationId) are OtOpcUa-specific. The enricher continues to push via LogContext.PushProperty alongside the shared enrichers.
  • ObservabilityExtensions.cs itself can be simplified or removed — it becomes a thin wrapper that calls AddZbTelemetry with OtOpcUa-specific options. The per-project entry point remains; only the implementation body is delegated to the shared library.

Adoption is a follow-on task (tracked in GAPS.md), not part of the ZB.MOM.WW.Telemetry library build. The library build delivers the shared bootstrap and enrichers; adoption lands in the OtOpcUa repo as a separate commit once the nupkg is available.