f05ce80269
Evidence doc gains the #55 section (root cause of no logs, 0.3.0 contents + SHA256, per-app commits, collector output per app: service.version, site.id, node.role and log records on OtOpcUa, HistorianGateway, ScadaBridge Site and MxAccessGateway). GAPS, current-state, SPEC, the Telemetry CLAUDE.md and the index row updated; infra/otel-collector gains otelidentity.py (Resource per signal + log samples). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
174 lines
9.9 KiB
Markdown
174 lines
9.9 KiB
Markdown
# Observability — current state: OtOpcUa
|
||
|
||
> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. OtOpcUa now calls
|
||
> `AddZbTelemetry` (Resource + OTLP opt-in), and OTLP export of **metrics and traces** is proven end to end
|
||
> against the dev collector (`otopcua.deploy.apply`, `otopcua.opcua.address_space_rebuild`, `otopcua.deploy.*`
|
||
> metrics arrived). That ends the "tracing no-op" below once `OtOpcUa:Telemetry:Exporter=Otlp` is set. The
|
||
> Galaxy driver's `galaxy.*` spans and `galaxy.events.*` counters were missing from the allowlists; fixed in
|
||
> lmxopcua#892 (`master` `db86b996`). Still absent from the Resource: `service.version`, `site.id`, `node.role`
|
||
> (scadaproj#55). Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
|
||
>
|
||
> **2026-11-12 update (scadaproj#55):** closed on `master` `412da27c` (Telemetry `0.3.0`). The Resource now
|
||
> carries `service.version` (entry-assembly informational version, e.g. `1.0.0+412da27c…`), `site.id` = the
|
||
> node's application Cluster id (suffix of its `cluster-<ClusterId>` role; omitted on a legacy node) and
|
||
> `node.role` = its plain roles (`admin`, `driver`, `admin,driver`). `OtOpcUa:Telemetry:SiteId` / `:NodeRole` /
|
||
> `:ServiceVersion` / `:ResourceAttributes:*` override them. `OtOpcUa:Telemetry:Exporter=Otlp` now also sends
|
||
> the Serilog logs to the collector (proven: 73 log records with the same Resource).
|
||
|
||
Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`.
|
||
Telemetry code lives in two places: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/` (host-side
|
||
bootstrap) and `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/` (instruments + enricher).
|
||
All paths relative to repo root. Verified 2026-06-01.
|
||
|
||
The most complete observability implementation in the family: OpenTelemetry SDK with both metrics and
|
||
tracing signals, Prometheus export, Serilog structured logging with a per-session correlation enricher,
|
||
and a dedicated instrument vocabulary. The one significant gap: **no OTel Resource / `service.name`**,
|
||
so all signals are indistinguishable from one another and from other fleet members in a backend.
|
||
|
||
## 1. Metrics (OpenTelemetry SDK)
|
||
|
||
### Bootstrap — `ObservabilityExtensions.cs`
|
||
|
||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ObservabilityExtensions.cs`:
|
||
|
||
- `:18` — `AddOtOpcUaObservability(IServiceCollection)` is the service-registration entry point.
|
||
- `:20` — `AddOpenTelemetry()` wires the OTel SDK.
|
||
- `:21–23` — `.WithMetrics(b => b.AddMeter(OtOpcUaTelemetry.MeterName).AddPrometheusExporter())`:
|
||
registers the application meter and attaches the Prometheus scrape exporter.
|
||
- `:24–25` — `.WithTracing(b => b.AddSource(OtOpcUaTelemetry.ActivitySourceName))`:
|
||
registers the application activity source for trace data.
|
||
- **No `ResourceBuilder` call anywhere** — `service.name`, `service.namespace`, `service.version`,
|
||
`site.id`, and `node.role` are not set. The OTel SDK defaults to an empty/SDK-default Resource.
|
||
- `:36` — `MapOtOpcUaMetrics(IEndpointRouteBuilder)` maps the Prometheus endpoint.
|
||
- `:38` — endpoint path is `/metrics`.
|
||
|
||
`Program.cs`:
|
||
- `:138` — `builder.Services.AddOtOpcUaObservability()`
|
||
- `:160` — `app.MapOtOpcUaMetrics()`
|
||
|
||
Package refs in csproj: `OpenTelemetry.Extensions.Hosting`, `OpenTelemetry.Exporter.Prometheus.AspNetCore`.
|
||
**No `OpenTelemetry.Exporter.OpenTelemetryProtocol`** — OTLP is not available; Prometheus is the
|
||
only export path.
|
||
|
||
### Instruments — `OtOpcUaTelemetry.cs`
|
||
|
||
`src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs`:
|
||
|
||
- `:19` — `MeterName = "ZB.MOM.WW.OtOpcUa"` (the `Meter` the SDK will collect).
|
||
- `:20` — `ActivitySourceName = "ZB.MOM.WW.OtOpcUa"` (the `ActivitySource` for spans).
|
||
|
||
Instruments defined (all `static readonly` on `OtOpcUaTelemetry`):
|
||
|
||
| Instrument | Kind | Unit | Subsystem |
|
||
|---|---|---|---|
|
||
| `otopcua.deploy.applied` | `Counter<long>` | — | deploy |
|
||
| `otopcua.deploy.apply.duration` | `Histogram<double>` | `s` | deploy |
|
||
| `otopcua.driver.lifecycle` | `Counter<long>` | — | driver |
|
||
| `otopcua.virtualtag.eval` | `Counter<long>` | — | virtual-tag |
|
||
| `otopcua.scriptedalarm.transition` | `Counter<long>` | — | scripted-alarm |
|
||
| `otopcua.opcua.sink.write` | `Counter<long>` | — | opc-ua sink |
|
||
| `otopcua.redundancy.service_level_change` | `Counter<long>` | — | redundancy |
|
||
|
||
Two activity spans: `otopcua.deploy.apply`, `otopcua.opcua.address_space_rebuild`.
|
||
|
||
Naming convention: `otopcua.<subsystem>.<event>`. Duration histogram correctly uses unit `s`
|
||
(OTel semantic conventions). **No standard instrumentation** (ASP.NET Core, HttpClient, runtime,
|
||
gRPC client meters) is wired — only the bespoke application instruments.
|
||
|
||
## 2. Logging (Serilog)
|
||
|
||
### Bootstrap
|
||
|
||
`Program.cs`:
|
||
- `:49–52` — two-stage Serilog bootstrap: initial logger for startup, then full
|
||
`UseSerilog(ReadFrom.Configuration)`. Sinks: Console + rolling file `logs/otopcua-.log`.
|
||
- `:141` — `UseSerilogRequestLogging()` on the `WebApplication`.
|
||
|
||
### Correlation enricher — `LogContextEnricher.cs`
|
||
|
||
`src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/LogContextEnricher.cs`:
|
||
|
||
- `:18–36` — `Push(driverInstanceId, driverType, capability, correlationId)` calls
|
||
`LogContext.PushProperty` for four properties:
|
||
- `DriverInstanceId` — Galaxy driver instance GUID.
|
||
- `DriverType` — driver type discriminator.
|
||
- `CapabilityName` — OPC UA capability being exercised.
|
||
- `CorrelationId` — caller-supplied correlation token.
|
||
|
||
This enricher is driver-lifecycle-scoped, not request-scoped — it pushes when a driver operation
|
||
begins and is disposable to pop on completion.
|
||
|
||
**No `trace_id` / `span_id` enricher.** Although OtOpcUa creates `ActivitySource` spans, the
|
||
active `Activity.Current` trace context is never pushed onto Serilog's `LogContext`. A log line
|
||
emitted during a span cannot be correlated to the span in a backend.
|
||
|
||
**No structural enrichers for `service.name` / `site.id` / `node.role`** — these dimensions are
|
||
absent from every log line. ScadaBridge has these; OtOpcUa does not.
|
||
|
||
## 3. Signal summary
|
||
|
||
| Signal | Provider | Export | Resource / service.name |
|
||
|---|---|---|---|
|
||
| Metrics | OTel SDK (`Meter` + `WithMetrics`) | Prometheus `/metrics` | ⛔ none |
|
||
| Traces | OTel SDK (`ActivitySource` + `WithTracing`) | ⛔ none (no exporter configured) | ⛔ none |
|
||
| Logs | Serilog | Console + rolling file | ⛔ none (no `service.name` property) |
|
||
| Trace↔log correlation | — | — | ⛔ absent (`trace_id`/`span_id` not pushed) |
|
||
|
||
Note: `WithTracing` registers the `ActivitySource` for collection, but no exporter (OTLP or
|
||
otherwise) is attached to the tracing pipeline. Spans are created and recorded by the SDK but never
|
||
shipped anywhere — effectively a no-op in production.
|
||
|
||
## 4. Notable design choices
|
||
|
||
- **Instrument naming** follows `<meter>.<subsystem>.<event>` cleanly and consistently — this is the
|
||
pattern the shared spec codifies as the fleet convention.
|
||
- **Duration unit** correctly uses `s` on `otopcua.deploy.apply.duration` — no conversion needed on
|
||
adoption; this contrasts with MxAccessGateway's `ms` histograms.
|
||
- **LogContextEnricher is bespoke but valuable** — the `DriverInstanceId`/`DriverType`/`CapabilityName`
|
||
correlation is OtOpcUa-specific domain context; it should survive adoption behind the shared
|
||
enricher layer.
|
||
- **No OTLP path** — with no OTLP exporter, OtOpcUa cannot send metrics or traces to a collector
|
||
(Prometheus is scrape-pull only). This limits operational flexibility.
|
||
|
||
---
|
||
|
||
## Adoption plan → `ZB.MOM.WW.Telemetry`
|
||
|
||
**Replace with shared bootstrap:**
|
||
|
||
- `AddOtOpcUaObservability()` → `builder.AddZbTelemetry(o => { o.ServiceName = "otopcua"; o.SiteId = ...; o.NodeRole = ...; o.Meters = [OtOpcUaTelemetry.MeterName]; o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName]; })`.
|
||
This adds the missing `Resource` (gains `service.name` / `service.namespace` / `service.version` /
|
||
`site.id` / `node.role` / `host.name` on every metric and span). Prometheus `/metrics` stays the
|
||
default exporter; OTLP becomes opt-in via options.
|
||
- Add standard instrumentation through `AddZbTelemetry` options: ASP.NET Core meters, HttpClient,
|
||
runtime + process meters — none wired today.
|
||
- Fix the tracing no-op: wire an OTLP exporter (or at minimum note that tracing is recorded but not
|
||
exported); `AddZbTelemetry` provides OTLP as the opt-in path.
|
||
- `MapOtOpcUaMetrics` → `app.MapZbMetrics()` (same `/metrics` path; shared convention).
|
||
|
||
**Replace with shared Serilog bootstrap:**
|
||
|
||
- Serilog bootstrap in `Program.cs:49–52` → `builder.AddZbSerilog(o => { o.ServiceName = "otopcua"; o.SiteId = ...; o.NodeRole = ...; })`.
|
||
This adds structural `SiteId` / `NodeRole` / `NodeHostname` properties to every log line
|
||
(currently absent) and wires the `TraceContextEnricher` so `trace_id`/`span_id` appear on log
|
||
lines emitted during active spans.
|
||
- Console + file sinks continue via `ReadFrom.Configuration` in `appsettings.json` — no sink changes
|
||
needed.
|
||
- `UseSerilogRequestLogging()` stays.
|
||
|
||
**Keep bespoke:**
|
||
|
||
- `OtOpcUaTelemetry.cs` — the application `Meter`, `ActivitySource`, and all instrument definitions
|
||
(`otopcua.*` counters, histograms, spans). These are domain instruments; `AddZbTelemetry` registers
|
||
them by name but does not own them.
|
||
- `LogContextEnricher.cs` — driver-lifecycle correlation properties (`DriverInstanceId`,
|
||
`DriverType`, `CapabilityName`, `CorrelationId`) are OtOpcUa-specific. The enricher continues to
|
||
push via `LogContext.PushProperty` alongside the shared enrichers.
|
||
- `ObservabilityExtensions.cs` itself can be simplified or removed — it becomes a thin wrapper that
|
||
calls `AddZbTelemetry` with OtOpcUa-specific options. The per-project entry point remains; only
|
||
the implementation body is delegated to the shared library.
|
||
|
||
**Adoption is a follow-on task** (tracked in `GAPS.md`), not part of the `ZB.MOM.WW.Telemetry`
|
||
library build. The library build delivers the shared bootstrap and enrichers; adoption lands in the
|
||
OtOpcUa repo as a separate commit once the nupkg is available.
|