Joseph Doherty
fba3d09eed
Complete the observability normalization component docs: - components/observability/current-state/otopcua/CURRENT-STATE.md — full OTel SDK (metrics + tracing) + Prometheus; 7 otopcua.* instruments + 2 spans; Serilog with driver-scope LogContextEnricher; no Resource/service.name anywhere; tracing pipeline wired but no exporter; adoption plan: AddZbTelemetry gains shared Resource + trace↔log correlation; LogContextEnricher kept bespoke. - components/observability/current-state/mxaccessgw/CURRENT-STATE.md — 20 hand-rolled instruments (13 counters, 3 histograms ms-unit, 4 gauges) in GatewayMetrics.cs; no OTel SDK → metrics never export; MEL logging with GatewayLogScope correlation and GatewayLogRedactor; adoption plan: in-pass MEL → AddZbSerilog migration (LogContext correlation, ILogRedactor seam) + AddZbTelemetry wires OTel SDK so GatewayMetrics finally exports. - components/observability/current-state/scadabridge/CURRENT-STATE.md — OpenTelemetry.Api is a CVE-patch override only (zero instrumentation); Serilog with SiteId/NodeRole/NodeHostname enrichers (strongest set in family); adoption plan: replace CVE ref with AddZbTelemetry; adopt AddZbSerilog (LoggerConfigurationFactory deleted); add first scadabridge.* instruments. - components/observability/GAPS.md — divergence table across §1 Resource (P1, nobody), §2 metrics export (P1, MxGateway invisible), §3 MxGateway MEL→Serilog (P1, in-pass done), §4 trace↔log correlation, §5 ms→s unit, §6 Meter naming, §7 standard instrumentation, §8 Serilog version, §9 ScadaBridge zero instrumentation; 11-item prioritized backlog. - components/observability/README.md — overview, per-project status table (OTel today / metrics / tracing / logging / enrichers / adoption status), normalized vs. left-per-project boundary, 2-package structure, component status.
107 lines
7.0 KiB
Markdown
107 lines
7.0 KiB
Markdown
# Observability (metrics / traces / logs)
|
|
|
|
Third normalized component under the operability cluster. **Goal: path to shared code** — converge
|
|
the three sister projects onto a common OpenTelemetry Resource, a shared Serilog bootstrap with
|
|
unified enrichers, and a trace↔log correlation bridge, proposed as the `ZB.MOM.WW.Telemetry`
|
|
library set (2 packages), while each project keeps its own application instruments and sink
|
|
configuration.
|
|
|
|
- The one target: [`spec/SPEC.md`](spec/SPEC.md)
|
|
- Metric naming reference: [`spec/METRIC-CONVENTIONS.md`](spec/METRIC-CONVENTIONS.md)
|
|
- The proposed shared library: [`shared-contract/ZB.MOM.WW.Telemetry.md`](shared-contract/ZB.MOM.WW.Telemetry.md)
|
|
- Divergences + backlog: [`GAPS.md`](GAPS.md)
|
|
- Current state, per project: [`current-state/`](current-state/)
|
|
|
|
## Why observability is a strong normalization candidate
|
|
|
|
All three projects instrument something — but in three completely different ways and at three very
|
|
different levels of completeness. The divergences are structural:
|
|
|
|
- **OtOpcUa** has the full OpenTelemetry SDK (metrics + tracing), Prometheus export, and a bespoke
|
|
Serilog enricher for driver-lifecycle correlation — but no Resource (`service.name` is never set)
|
|
and no trace↔log bridge.
|
|
- **MxAccessGateway** has 20 hand-rolled instruments (counters, histograms, gauges) recording real
|
|
production data — that never leave the process. No OTel SDK, no exporter, no tracing. Logging
|
|
uses Microsoft.Extensions.Logging rather than Serilog, with a bespoke correlation-scope and
|
|
redaction pipeline.
|
|
- **ScadaBridge** has zero application instruments. Its `OpenTelemetry.Api` reference is a CVE
|
|
patch, not instrumentation. It does have the cleanest structured logging enricher set
|
|
(`SiteId`/`NodeRole`/`NodeHostname`) — but those properties exist only in Serilog, not in the
|
|
OTel Resource, so logs and metrics cannot join in a backend.
|
|
|
|
Nobody sets a Resource. Nobody does trace↔log correlation. MxGateway's metrics are invisible.
|
|
ScadaBridge has no metrics at all.
|
|
|
|
The common fix is a single `AddZbTelemetry(options)` call that: creates a shared Resource from a
|
|
`service.name`/`site.id`/`node.role` options object; registers the project's own Meter/ActivitySource
|
|
names with the OTel SDK; and exposes Prometheus `/metrics`. A companion `AddZbSerilog(options)` wires
|
|
Serilog with the same options as enricher properties and adds `TraceContextEnricher` so logs carry
|
|
`trace_id`/`span_id`. The unifying hinge: the same identity triple (`service.name`/`site.id`/
|
|
`node.role`) populates both the OTel Resource and the Serilog enrichers, so a metric, a span, and
|
|
a log line from the same node carry identical dimensions and join up in a backend.
|
|
|
|
One adoption happens **in this task**: MxAccessGateway migrates off MEL onto `AddZbSerilog`. All
|
|
other app wiring is follow-on, consistent with how Auth and UI-Theme are structured.
|
|
|
|
## Status by project
|
|
|
|
| Project | OTel SDK today | Metrics today | Tracing today | Logging today | Enrichers today | Adoption status |
|
|
|---|---|---|---|---|---|---|
|
|
| **OtOpcUa** | ✅ full SDK (`WithMetrics`+`WithTracing`) | ✅ 7 instruments (`otopcua.*`); Prometheus `/metrics` | 🟡 2 spans defined; no exporter | Serilog (Console+File) | `DriverInstanceId`/`DriverType`/`CapabilityName`/`CorrelationId` (driver-scope) | Not started (follow-on) |
|
|
| **MxAccessGateway** | ⛔ none (hand-rolled `Meter`) | 🟡 20 instruments (`mxgateway.*`); **never exported** | ⛔ none | MEL → **migrating to Serilog in this task** | `SessionId`/`WorkerProcessId`/`CorrelationId`/`CommandMethod` (MEL scope) | **In progress (Task #9)** |
|
|
| **ScadaBridge** | ⛔ (`OpenTelemetry.Api` CVE-patch only) | ⛔ zero instruments | ⛔ none | Serilog (Console+File) | `SiteId`/`NodeRole`/`NodeHostname` (process-level; strongest set) | Not started (follow-on) |
|
|
|
|
See each project's [`current-state/<project>/CURRENT-STATE.md`](current-state/) for the
|
|
code-verified detail and its adoption plan.
|
|
|
|
## Normalized vs. left per-project
|
|
|
|
**Normalized (the shared target):**
|
|
|
|
- `AddZbTelemetry(ZbTelemetryOptions)` — front door for the OTel SDK. Populates the shared
|
|
Resource (`service.name`, `service.namespace`, `service.version`, `site.id`, `node.role`,
|
|
`host.name`). Registers the caller-supplied Meter and ActivitySource name(s). Wires standard
|
|
instrumentation (ASP.NET Core, HttpClient, runtime, process). Prometheus default; OTLP opt-in.
|
|
- `app.MapZbMetrics()` — maps the Prometheus `/metrics` endpoint (shared path + shared exporter).
|
|
- `AddZbSerilog(ZbTelemetryOptions)` — shared Serilog two-stage bootstrap generalizing
|
|
ScadaBridge's `LoggerConfigurationFactory`. Wires `SiteId`/`NodeRole`/`NodeHostname` enrichers
|
|
from the same options object as the OTel Resource. Wires `TraceContextEnricher`
|
|
(`trace_id`/`span_id` from `Activity.Current`). Preserves `ReadFrom.Configuration` for sinks
|
|
and explicit `MinimumLevel.Is` override.
|
|
- `ILogRedactor` seam — generalized from MxGateway's `GatewayLogRedactor`. The seam is shared;
|
|
the redaction policy (which fields/commands) stays per-project.
|
|
- Metric naming convention: `<meter>.<subsystem>.<event>`; Meter name = project namespace
|
|
(`ZB.MOM.WW.<ProjectName>`); duration unit = `s` (OTel semconv).
|
|
|
|
**Left per-project (not forced together):**
|
|
|
|
- Application `Meter`, `ActivitySource`, and all instrument definitions — `otopcua.*`,
|
|
`mxgateway.*`, `scadabridge.*` instruments are owned by each repo.
|
|
- Serilog sink configuration (`appsettings.json` Console/File templates, rolling intervals).
|
|
- Per-operation/per-session correlation enrichers (`LogContextEnricher` in OtOpcUa;
|
|
`LogContext.PushProperty` scope in MxGateway after migration).
|
|
- Redaction policies (`MxGatewayLogRedactor` implements `ILogRedactor` with gateway-specific
|
|
command/field rules).
|
|
- Config section paths for `SiteId`/`NodeRole`/`NodeHostname` — each project binds these from
|
|
its own config hierarchy and passes the resolved values to `AddZbTelemetry`/`AddZbSerilog`.
|
|
|
|
## Package structure
|
|
|
|
`ZB.MOM.WW.Telemetry` ships as two dependency-split packages:
|
|
|
|
| Package | Contents | Consumers |
|
|
|---|---|---|
|
|
| `ZB.MOM.WW.Telemetry` | `AddZbTelemetry`, `ZbTelemetryOptions`, Resource builder, standard instrumentation, Prometheus/OTLP exporters, `app.MapZbMetrics()` | All three |
|
|
| `ZB.MOM.WW.Telemetry.Serilog` | `AddZbSerilog`, shared enrichers (`SiteId`/`NodeRole`/`NodeHostname`/`TraceContextEnricher`), `ILogRedactor` seam | All three (Serilog users); MxGateway on migration |
|
|
|
|
Both packages share `ZbTelemetryOptions` as the single options object that drives Resource
|
|
attributes, Serilog enrichers, Meter/ActivitySource names, and exporter selection — the unifying
|
|
hinge that makes a metric, a span, and a log line from the same node carry identical dimensions.
|
|
|
|
## Component status
|
|
|
|
**Status: Draft.** Spec and shared-contract written; current-state docs verified; GAPS backlog
|
|
populated. Library implementation in progress (`ZB.MOM.WW.Telemetry` — Task #8). MxAccessGateway
|
|
MEL → Serilog migration in progress (Task #9, blocked by library build). Adoption by OtOpcUa and
|
|
ScadaBridge is follow-on, tracked in [`GAPS.md`](GAPS.md).
|