Files
scadaproj/components/observability/current-state/otopcua/CURRENT-STATE.md
T
Joseph Doherty f05ce80269 docs(observability): Telemetry 0.3.0 published, adopted by all four apps, logs over OTLP proven (#55)
Evidence doc gains the #55 section (root cause of no logs, 0.3.0 contents + SHA256,
per-app commits, collector output per app: service.version, site.id, node.role and
log records on OtOpcUa, HistorianGateway, ScadaBridge Site and MxAccessGateway).
GAPS, current-state, SPEC, the Telemetry CLAUDE.md and the index row updated;
infra/otel-collector gains otelidentity.py (Resource per signal + log samples).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 21:28:50 -05:00

9.9 KiB
Raw Blame History

Observability — current state: OtOpcUa

2026-11-12 update (scadaproj#13): this is the pre-adoption 2026-06-01 snapshot. OtOpcUa now calls AddZbTelemetry (Resource + OTLP opt-in), and OTLP export of metrics and traces is proven end to end against the dev collector (otopcua.deploy.apply, otopcua.opcua.address_space_rebuild, otopcua.deploy.* metrics arrived). That ends the "tracing no-op" below once OtOpcUa:Telemetry:Exporter=Otlp is set. The Galaxy driver's galaxy.* spans and galaxy.events.* counters were missing from the allowlists; fixed in lmxopcua#892 (master db86b996). Still absent from the Resource: service.version, site.id, node.role (scadaproj#55). Evidence: 2026-11-12-otlp-end-to-end-evidence.md.

2026-11-12 update (scadaproj#55): closed on master 412da27c (Telemetry 0.3.0). The Resource now carries service.version (entry-assembly informational version, e.g. 1.0.0+412da27c…), site.id = the node's application Cluster id (suffix of its cluster-<ClusterId> role; omitted on a legacy node) and node.role = its plain roles (admin, driver, admin,driver). OtOpcUa:Telemetry:SiteId / :NodeRole / :ServiceVersion / :ResourceAttributes:* override them. OtOpcUa:Telemetry:Exporter=Otlp now also sends the Serilog logs to the collector (proven: 73 log records with the same Resource).

Repo: ~/Desktop/OtOpcUa. Stack: .NET 10, Akka.NET, OPC UA; solution ZB.MOM.WW.OtOpcUa.slnx. Telemetry code lives in two places: src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ (host-side bootstrap) and src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/ (instruments + enricher). All paths relative to repo root. Verified 2026-06-01.

The most complete observability implementation in the family: OpenTelemetry SDK with both metrics and tracing signals, Prometheus export, Serilog structured logging with a per-session correlation enricher, and a dedicated instrument vocabulary. The one significant gap: no OTel Resource / service.name, so all signals are indistinguishable from one another and from other fleet members in a backend.

1. Metrics (OpenTelemetry SDK)

Bootstrap — ObservabilityExtensions.cs

src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ObservabilityExtensions.cs:

  • :18 — AddOtOpcUaObservability(IServiceCollection) is the service-registration entry point.
  • :20 — AddOpenTelemetry() wires the OTel SDK.
  • :21–23 — .WithMetrics(b => b.AddMeter(OtOpcUaTelemetry.MeterName).AddPrometheusExporter()): registers the application meter and attaches the Prometheus scrape exporter.
  • :24–25 — .WithTracing(b => b.AddSource(OtOpcUaTelemetry.ActivitySourceName)): registers the application activity source for trace data.
  • No ResourceBuilder call anywhere — service.name, service.namespace, service.version, site.id, and node.role are not set. The OTel SDK defaults to an empty/SDK-default Resource.
  • :36 — MapOtOpcUaMetrics(IEndpointRouteBuilder) maps the Prometheus endpoint.
  • :38 — endpoint path is /metrics.

Program.cs:

  • :138 — builder.Services.AddOtOpcUaObservability()
  • :160 — app.MapOtOpcUaMetrics()

Package refs in csproj: OpenTelemetry.Extensions.Hosting, OpenTelemetry.Exporter.Prometheus.AspNetCore. No OpenTelemetry.Exporter.OpenTelemetryProtocol — OTLP is not available; Prometheus is the only export path.

Instruments — OtOpcUaTelemetry.cs

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs:

  • :19 — MeterName = "ZB.MOM.WW.OtOpcUa" (the Meter the SDK will collect).
  • :20 — ActivitySourceName = "ZB.MOM.WW.OtOpcUa" (the ActivitySource for spans).

Instruments defined (all static readonly on OtOpcUaTelemetry):

Instrument Kind Unit Subsystem
otopcua.deploy.applied Counter<long> — deploy
otopcua.deploy.apply.duration Histogram<double> s deploy
otopcua.driver.lifecycle Counter<long> — driver
otopcua.virtualtag.eval Counter<long> — virtual-tag
otopcua.scriptedalarm.transition Counter<long> — scripted-alarm
otopcua.opcua.sink.write Counter<long> — opc-ua sink
otopcua.redundancy.service_level_change Counter<long> — redundancy

Two activity spans: otopcua.deploy.apply, otopcua.opcua.address_space_rebuild.

Naming convention: otopcua.<subsystem>.<event>. Duration histogram correctly uses unit s (OTel semantic conventions). No standard instrumentation (ASP.NET Core, HttpClient, runtime, gRPC client meters) is wired — only the bespoke application instruments.

2. Logging (Serilog)

Bootstrap

Program.cs:

  • :49–52 — two-stage Serilog bootstrap: initial logger for startup, then full UseSerilog(ReadFrom.Configuration). Sinks: Console + rolling file logs/otopcua-.log.
  • :141 — UseSerilogRequestLogging() on the WebApplication.

Correlation enricher — LogContextEnricher.cs

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/LogContextEnricher.cs:

  • :18–36 — Push(driverInstanceId, driverType, capability, correlationId) calls LogContext.PushProperty for four properties:
    • DriverInstanceId — Galaxy driver instance GUID.
    • DriverType — driver type discriminator.
    • CapabilityName — OPC UA capability being exercised.
    • CorrelationId — caller-supplied correlation token.

This enricher is driver-lifecycle-scoped, not request-scoped — it pushes when a driver operation begins and is disposable to pop on completion.

No trace_id / span_id enricher. Although OtOpcUa creates ActivitySource spans, the active Activity.Current trace context is never pushed onto Serilog's LogContext. A log line emitted during a span cannot be correlated to the span in a backend.

No structural enrichers for service.name / site.id / node.role — these dimensions are absent from every log line. ScadaBridge has these; OtOpcUa does not.

3. Signal summary

Signal Provider Export Resource / service.name
Metrics OTel SDK (Meter + WithMetrics) Prometheus /metrics ⛔ none
Traces OTel SDK (ActivitySource + WithTracing) ⛔ none (no exporter configured) ⛔ none
Logs Serilog Console + rolling file ⛔ none (no service.name property)
Trace↔log correlation — — ⛔ absent (trace_id/span_id not pushed)

Note: WithTracing registers the ActivitySource for collection, but no exporter (OTLP or otherwise) is attached to the tracing pipeline. Spans are created and recorded by the SDK but never shipped anywhere — effectively a no-op in production.

4. Notable design choices

  • Instrument naming follows <meter>.<subsystem>.<event> cleanly and consistently — this is the pattern the shared spec codifies as the fleet convention.
  • Duration unit correctly uses s on otopcua.deploy.apply.duration — no conversion needed on adoption; this contrasts with MxAccessGateway's ms histograms.
  • LogContextEnricher is bespoke but valuable — the DriverInstanceId/DriverType/CapabilityName correlation is OtOpcUa-specific domain context; it should survive adoption behind the shared enricher layer.
  • No OTLP path — with no OTLP exporter, OtOpcUa cannot send metrics or traces to a collector (Prometheus is scrape-pull only). This limits operational flexibility.

Adoption plan → ZB.MOM.WW.Telemetry

Replace with shared bootstrap:

  • AddOtOpcUaObservability() → builder.AddZbTelemetry(o => { o.ServiceName = "otopcua"; o.SiteId = ...; o.NodeRole = ...; o.Meters = [OtOpcUaTelemetry.MeterName]; o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName]; }). This adds the missing Resource (gains service.name / service.namespace / service.version / site.id / node.role / host.name on every metric and span). Prometheus /metrics stays the default exporter; OTLP becomes opt-in via options.
  • Add standard instrumentation through AddZbTelemetry options: ASP.NET Core meters, HttpClient, runtime + process meters — none wired today.
  • Fix the tracing no-op: wire an OTLP exporter (or at minimum note that tracing is recorded but not exported); AddZbTelemetry provides OTLP as the opt-in path.
  • MapOtOpcUaMetrics → app.MapZbMetrics() (same /metrics path; shared convention).

Replace with shared Serilog bootstrap:

  • Serilog bootstrap in Program.cs:49–52 → builder.AddZbSerilog(o => { o.ServiceName = "otopcua"; o.SiteId = ...; o.NodeRole = ...; }). This adds structural SiteId / NodeRole / NodeHostname properties to every log line (currently absent) and wires the TraceContextEnricher so trace_id/span_id appear on log lines emitted during active spans.
  • Console + file sinks continue via ReadFrom.Configuration in appsettings.json — no sink changes needed.
  • UseSerilogRequestLogging() stays.

Keep bespoke:

  • OtOpcUaTelemetry.cs — the application Meter, ActivitySource, and all instrument definitions (otopcua.* counters, histograms, spans). These are domain instruments; AddZbTelemetry registers them by name but does not own them.
  • LogContextEnricher.cs — driver-lifecycle correlation properties (DriverInstanceId, DriverType, CapabilityName, CorrelationId) are OtOpcUa-specific. The enricher continues to push via LogContext.PushProperty alongside the shared enrichers.
  • ObservabilityExtensions.cs itself can be simplified or removed — it becomes a thin wrapper that calls AddZbTelemetry with OtOpcUa-specific options. The per-project entry point remains; only the implementation body is delegated to the shared library.

Adoption is a follow-on task (tracked in GAPS.md), not part of the ZB.MOM.WW.Telemetry library build. The library build delivers the shared bootstrap and enrichers; adoption lands in the OtOpcUa repo as a separate commit once the nupkg is available.