41a5e4d187
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
169 lines
14 KiB
Markdown
169 lines
14 KiB
Markdown
# ZB.MOM.WW.Telemetry
|
|
|
|
Observability libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central telemetry process; instrumentation runs in-process alongside the application.
|
|
|
|
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #13 (OTel collector / OTLP end to end) is **closed** — proven 2026-11-12 against `zb-otel-collector` (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`); #55 (`service.version`/`site.id`/`node.role` + logs-over-OTLP) is **closed** by `0.3.0` (2026-11-12); #56 (OTLP export calls traced as client spans) is **closed** by `0.3.1` (2026-11-12). Open: #54 (wonder-tier collector). The history below is a record, not a status tracker; file new follow-ups as issues.
|
|
|
|
The library normalizes the three-project observability surface: a shared OpenTelemetry Resource driven by a single identity triple (`service.name` / `site.id` / `node.role`), standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and `TraceContextEnricher` for trace↔log correlation.
|
|
|
|
**Built at 0.1.0, published to the Gitea NuGet feed, and adopted across all three apps on 2026-06-01** (branch `feat/adopt-zb-telemetry` per repo, behaviour-preserving). MxAccessGateway's MEL→Serilog migration + metrics export both landed in this pass — they were *not* actually done beforehand despite the earlier claim. ScadaBridge kept its own `LoggerConfigurationFactory` until **2026-11-12**, when it switched to `AddZbSerilog` 0.2.0 (ScadaBridge#94). Per-repo result + deferred follow-ons tracked in `~/Desktop/scadaproj/components/observability/GAPS.md`.
|
|
|
|
---
|
|
|
|
## Packages
|
|
|
|
| Package | Responsibilities | Key Dependencies |
|
|
|---|---|---|
|
|
| `ZB.MOM.WW.Telemetry` | `AddZbTelemetry` — wires OTel SDK (metrics + tracing), populates shared Resource (`service.name`, `service.namespace`, `service.version`, `site.id`, `node.role`, `host.name`), registers caller-supplied Meters/ActivitySources, adds standard instrumentation (ASP.NET Core, HttpClient, gRPC client, runtime, process), Prometheus always-on exporter, OTLP additive overlay. `app.MapZbMetrics()` — mounts `/metrics`. `ZbTelemetryOptions` — the single options object shared by both packages. **0.3.0:** `service.version` defaults from the entry assembly's informational version; `ResourceAttributes`; `ApplyConfiguration(<App>:Telemetry section)`; publishes its options in DI for the Serilog log sink. | `Microsoft.AspNetCore.App` (framework ref), `OpenTelemetry.*` stack |
|
|
| `ZB.MOM.WW.Telemetry.Serilog` | `AddZbSerilog` — shared two-stage Serilog bootstrap: `ReadFrom.Configuration`-driven sinks, `MinimumLevel.Is(Information)` default (config-overridable), `SiteId`/`NodeRole`/`NodeHostname` enrichers from `ZbTelemetryOptions`, `TraceContextEnricher` (writes `trace_id`/`span_id` from `Activity.Current`), `ILogRedactor` seam (per-project sensitive-field redaction via `RedactionEnricher`). **0.2.0:** optional `ZbSerilogOptions` overload — `MinimumLevelGovernor` (app key outranks `Serilog:MinimumLevel`) + `ConfigureLogger` (final filters/enrichers/sinks). **0.3.0:** the OTLP log sink follows the `AddZbTelemetry` registration in the same container (`ExportLogs=false` opts out). Does NOT freeze `Log.Logger` — safe for multi-host/test scenarios. | `ZB.MOM.WW.Telemetry`, `Serilog.*` stack |
|
|
|
|
---
|
|
|
|
## Consumer matrix
|
|
|
|
| Consumer | `ZB.MOM.WW.Telemetry` (core) | `ZB.MOM.WW.Telemetry.Serilog` |
|
|
|---|:---:|:---:|
|
|
| **OtOpcUa** | ✅ **0.3.1** (`master` `f3178c5f`) — `site.id` = Cluster id, `node.role` = plain roles | ✅ **0.3.0** (`AddZbSerilog`; logs follow `AddZbTelemetry`) |
|
|
| **MxAccessGateway** | ✅ **0.3.1** (`main` `453fe015`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) |
|
|
| **ScadaBridge** | ✅ **0.3.1** (`main` `95584e2f`, both roots) | ✅ **0.3.0** `AddZbSerilog` with `MinimumLevelGovernor` + `ConfigureLogger` on a pre-host builder; the exporter is passed explicitly via `ScadaBridgeTelemetryConfig` |
|
|
| **HistorianGateway** | ✅ **0.3.1** (`main` `dffb5e2`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) |
|
|
|
|
All four pin both packages at **`0.3.1`** on their origin default branches since 2026-11-12 (scadaproj#56; `0.3.0`
|
|
earlier the same day, #55; not deployed — the wonder boxes still run their earlier builds, which have no collector anyway, #54). See `components/observability/GAPS.md` for the full result.
|
|
|
|
---
|
|
|
|
## Build, test, and pack commands
|
|
|
|
```bash
|
|
# From ZB.MOM.WW.Telemetry/
|
|
|
|
# Build
|
|
dotnet build ZB.MOM.WW.Telemetry.slnx
|
|
dotnet build ZB.MOM.WW.Telemetry.slnx -c Release
|
|
|
|
# Test (no external dependencies — no running OTel collector, no Serilog backend required)
|
|
dotnet test ZB.MOM.WW.Telemetry.slnx
|
|
|
|
# Pack (two .nupkg files land in artifacts/)
|
|
dotnet pack ZB.MOM.WW.Telemetry.slnx -c Release -o ./artifacts
|
|
```
|
|
|
|
All test assemblies run offline:
|
|
|
|
| Assembly | Tests |
|
|
|---|---|
|
|
| `ZB.MOM.WW.Telemetry.Tests` | 54 |
|
|
| `ZB.MOM.WW.Telemetry.Serilog.Tests` | 40 |
|
|
| **Total** | **94** |
|
|
|
|
`GeneratePackageOnBuild` is off — pack explicitly with the command above.
|
|
|
|
---
|
|
|
|
## Status
|
|
|
|
**`0.3.1` (2026-11-12, scadaproj#56) — PUBLISHED** (both packages, packed from scadaproj `main` `85654d7`, the
|
|
commit recorded in the nuspec; 54 + 40 tests green in Release). SHA256 (feed download = local pack):
|
|
`ZB.MOM.WW.Telemetry.0.3.1.nupkg` `baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3`,
|
|
`ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg` `188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931`.
|
|
Adopted by all four apps the same day.
|
|
- **Telemetry's own OTLP transport is not traced.** The SDK's OTLP exporters run inside
|
|
`SuppressInstrumentationScope`; `Serilog.Sinks.OpenTelemetry` does not, so every log batch produced a root
|
|
`opentelemetry.proto.collector.logs.v1.LogsService/Export` gRPC client span (scadabridge 146, mxgateway 13,
|
|
historian-gateway 10, otopcua 6 at the collector). `ZbOtlpExportCallFilter.IsExportCall` matches any
|
|
`/opentelemetry.proto.collector.*` path (any host) and OTLP/HTTP `/v1/{traces,metrics,logs}` on the configured
|
|
endpoint's host:port. HttpClient instrumentation drops those via `FilterHttpRequestMessage`. The gRPC client
|
|
instrumentation has **no Filter option**, so its `EnrichWithHttpRequestMessage` callback, which runs at start
|
|
before the HTTP request, clears `Recorded` + `IsAllDataRequested`. That also leaves the child HTTP activity
|
|
unsampled. Every other outgoing call keeps its span.
|
|
- Tests: predicate theory (16) + an end-to-end run against the loopback OTLP receiver. Positive controls: the app's
|
|
gRPC and HttpClient calls are traced. Negatives: the real log-sink export, `/v1/logs` and a collector path over
|
|
HttpClient are not. The receiver's own ASP.NET Core server spans share the process, so assertions key on
|
|
client-span markers (span name / `url.full`). Mutation-checked: no gRPC un-record, no HttpClient filter, and
|
|
"everything is export" each fail it. `VersionOf_reads_the_informational_version` now derives the expected prefix
|
|
from the assembly version instead of a `0.3.0` literal.
|
|
|
|
**`0.3.0` (2026-11-12, scadaproj#55) — PUBLISHED** (both packages, packed from the tree of scadaproj `main` `2011c8d` (committed as `df9a328` before rebasing onto main; identical `ZB.MOM.WW.Telemetry/` tree); 38 + 39
|
|
tests green in Release). SHA256 (feed download = local pack): `ZB.MOM.WW.Telemetry.0.3.0.nupkg`
|
|
`a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2`, `ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg`
|
|
`04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c`. Adopted by all four apps the same day.
|
|
- `service.version`: `ServiceVersion = null` (default) now resolves to the entry assembly's
|
|
`AssemblyInformationalVersion` (else its assembly version); `""` omits; any value overrides. **Behaviour
|
|
change** from 0.2.x, where null omitted the attribute.
|
|
- `ResourceAttributes` (extra Resource attributes on all signals; first-class keys reserved, rejected at startup).
|
|
- `ApplyConfiguration(IConfiguration section)` — `Exporter`, `OtlpEndpoint`, `ServiceVersion`, `SiteId`,
|
|
`NodeRole`, `ExportLogs`, `ResourceAttributes:<key>`; absent keys keep code defaults.
|
|
- **Logs over OTLP**: `AddZbTelemetry` registers its options in DI. The `AddZbSerilog` OTLP log sink
|
|
follows them when its own options do not select OTLP. Root cause of "no app ever exported a log": the
|
|
sink keyed only on the `AddZbSerilog` options, which every app filled with the service name alone.
|
|
`ExportLogs = false` keeps logs local.
|
|
- Includes the never-published **0.2.1** fixes below, including the stricter OTLP endpoint check. That check
|
|
was shipped deliberately: no checked-in config sets an endpoint, it only runs when `Exporter=Otlp`, and no
|
|
wonder tier has a collector (#54).
|
|
- Feed versions: `0.1.0`, `0.2.0`, `0.3.0`, `0.3.1` (`0.2.1` was never published).
|
|
|
|
**`0.2.1` (2026-11-09, scadaproj#48) — never published; shipped inside `0.3.0`.** P3 review fixes: `RedactionEnricher` is now registered
|
|
AFTER `ConfigureLogger`, so enrichers added in the hook are redacted (009); dictionary keys that print identically are
|
|
all kept (later ones get a `#n` suffix in the redactor's view; the original key is restored on rebuild) (010); the OTLP
|
|
endpoint must be `http`/`https` with a host (`collector:4317` is now rejected) and the pre-built-options
|
|
`AddZbTelemetry(services, options)` overload validates `ServiceName` + endpoint like the lambda overloads (011).
|
|
**Behaviour change:** a scheme-less OTLP endpoint that previously passed validation now throws at startup.
|
|
|
|
**`0.2.0` (2026-11-09, scadaproj#12) — PUBLISHED 2026-11-12** (both packages, packed from `01c6dc1` — the
|
|
0.2.0 code plus the corrected RepositoryUrl, i.e. before the 0.2.1 commit `1786bab`; 12 + 30 tests green in Release).
|
|
SHA256: `ZB.MOM.WW.Telemetry.0.2.0.nupkg` `71749cb3c34a0d0aba3eca43a6c12398b427f494e8c4c6f216db0be7d3ab40dc`,
|
|
`ZB.MOM.WW.Telemetry.Serilog.0.2.0.nupkg` `2aef391e7ab28061c5b3365c62c73e5c972ba8bcecd88bd0e0471550cfcc40c0`
|
|
(feed download matches the local pack). First consumer: ScadaBridge (#94). Additive:
|
|
a new `AddZbSerilog(builder, configure, configureSerilog)` overload taking `ZbSerilogOptions` —
|
|
`MinimumLevelGovernor` (`Func<IConfiguration, LogEventLevel?>`, applied with `MinimumLevel.Is`
|
|
AFTER `ReadFrom.Configuration`, so an app-owned key outranks `Serilog:MinimumLevel`; `null` defers to
|
|
config; namespace `Override:*` entries still apply) and `ConfigureLogger` (runs last, for filters such
|
|
as ScadaBridge's standby `active-node` exclusion). The two-argument overload is unchanged. This is
|
|
the library half of ScadaBridge's `AddZbSerilog` adoption; the app-side switch is a follow-on in
|
|
that repo.
|
|
|
|
Built at **0.1.0**, published to the Gitea NuGet feed, and **adopted across all three apps on
|
|
2026-06-01** (branch `feat/adopt-zb-telemetry` per repo, behaviour-preserving). MxAccessGateway's
|
|
MEL→Serilog migration and metrics export both landed in this pass (not beforehand, despite the
|
|
earlier claim). Deferred follow-ons (MxGateway `ms`→`s` + Meter rename, ScadaBridge app instruments
|
|
+ Site-node HTTP/1.1 metrics listener, OTLP wiring) are tracked in the component backlog (remaining open items are Gitea issues: #12 min-level governance hook — closed; #13 OTLP collector — closed 2026-11-12):
|
|
|
|
- `~/Desktop/scadaproj/components/observability/GAPS.md` — adoption status + deferred follow-ons
|
|
|
|
Design documentation:
|
|
|
|
- `~/Desktop/scadaproj/components/observability/spec/SPEC.md` — normalized observability target
|
|
- `~/Desktop/scadaproj/components/observability/spec/METRIC-CONVENTIONS.md` — metric naming reference
|
|
- `~/Desktop/scadaproj/components/observability/shared-contract/ZB.MOM.WW.Telemetry.md` — proposed shared-library API
|
|
- `~/Desktop/scadaproj/components/observability/current-state/` — per-project current state (code-verified)
|
|
|
|
## Index history (migrated from the scadaproj root CLAUDE.md)
|
|
|
|
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
|
|
|
|
### Narrative
|
|
|
|
The observability component is fully populated: a normalized [`spec`](components/observability/spec/SPEC.md),
|
|
a [`metric-conventions`](components/observability/spec/METRIC-CONVENTIONS.md) reference, a
|
|
[`shared-contract`](components/observability/shared-contract/ZB.MOM.WW.Telemetry.md), three
|
|
[`current-state`](components/observability/current-state/) docs, and an adoption [`GAPS`](components/observability/GAPS.md)
|
|
backlog. Shared = OTel Resource (service.name/site.id/node.role identity triple) + standard instrumentation
|
|
(ASP.NET Core, HttpClient, gRPC client, runtime, process) + Prometheus always-on exporter + OTLP opt-in
|
|
+ Serilog two-stage bootstrap + SiteId/NodeRole/NodeHostname enrichers + TraceContextEnricher (trace_id/span_id)
|
|
+ ILogRedactor seam; left per-project = application Meters/ActivitySources, sink config, per-operation
|
|
enrichers, and redaction policies.
|
|
|
|
The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/)
|
|
(.NET 10; 2 packages — `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Telemetry.Serilog`; 34 tests (verified 2026-07-18);
|
|
`dotnet pack` → 2 nupkgs @ 0.1.0). **Adopted across all three apps on 2026-06-01** (branch
|
|
`feat/adopt-zb-telemetry` per repo, behaviour-preserving): `AddZbTelemetry` (Resource + standard
|
|
instrumentation + Prometheus `/metrics`) everywhere; OtOpcUa + MxGateway on `AddZbSerilog` (MxGateway's
|
|
MEL→Serilog migration + metrics export both landed in this pass — they were *not* actually done
|
|
beforehand despite an earlier claim); ScadaBridge keeps its `LoggerConfigurationFactory` (min-level
|
|
governance) and only adds the shared `TraceContextEnricher`. Deferred: MxGateway `ms`→`s` + Meter
|
|
rename, ScadaBridge app instruments + Site-node HTTP/1.1 metrics listener, OTLP wiring. Per-repo
|
|
result tracked in [`components/observability/GAPS.md`](components/observability/GAPS.md).
|
|
Build/test from `ZB.MOM.WW.Telemetry/`: `dotnet test`. Consumer matrix: all three apps consume both
|
|
packages after adoption (OtOpcUa, MxGateway Server, ScadaBridge Host + any instrumented project).
|