docs(observability): Telemetry 0.3.1 published, adopted by all four apps, export spans gone at the collector (#56)

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 22:21:07 -05:00
parent 71b8595742
commit 41a5e4d187
3 changed files with 103 additions and 13 deletions
+32 -11
View File
@@ -2,7 +2,7 @@
Observability libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central telemetry process; instrumentation runs in-process alongside the application.
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #13 (OTel collector / OTLP end to end) is **closed** — proven 2026-11-12 against `zb-otel-collector` (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`); #55 (`service.version`/`site.id`/`node.role` + logs-over-OTLP) is **closed** by `0.3.0` (2026-11-12). Open: #54 (wonder-tier collector). The history below is a record, not a status tracker; file new follow-ups as issues.
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #13 (OTel collector / OTLP end to end) is **closed** — proven 2026-11-12 against `zb-otel-collector` (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`); #55 (`service.version`/`site.id`/`node.role` + logs-over-OTLP) is **closed** by `0.3.0` (2026-11-12); #56 (OTLP export calls traced as client spans) is **closed** by `0.3.1` (2026-11-12). Open: #54 (wonder-tier collector). The history below is a record, not a status tracker; file new follow-ups as issues.
The library normalizes the three-project observability surface: a shared OpenTelemetry Resource driven by a single identity triple (`service.name` / `site.id` / `node.role`), standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and `TraceContextEnricher` for trace↔log correlation.
@@ -23,13 +23,13 @@ The library normalizes the three-project observability surface: a shared OpenTel
| Consumer | `ZB.MOM.WW.Telemetry` (core) | `ZB.MOM.WW.Telemetry.Serilog` |
|---|:---:|:---:|
| **OtOpcUa** | ✅ **0.3.0** (`master` `412da27c`) — `site.id` = Cluster id, `node.role` = plain roles | ✅ **0.3.0** (`AddZbSerilog`; logs follow `AddZbTelemetry`) |
| **MxAccessGateway** | ✅ **0.3.0** (`main` `d9d9ff48`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) |
| **ScadaBridge** | ✅ **0.3.0** (`main` `a633005f`, both roots) | ✅ **0.3.0** `AddZbSerilog` with `MinimumLevelGovernor` + `ConfigureLogger` on a pre-host builder; the exporter is passed explicitly via `ScadaBridgeTelemetryConfig` |
| **HistorianGateway** | ✅ **0.3.0** (`main` `ae1dbcc`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) |
| **OtOpcUa** | ✅ **0.3.1** (`master` `f3178c5f`) — `site.id` = Cluster id, `node.role` = plain roles | ✅ **0.3.0** (`AddZbSerilog`; logs follow `AddZbTelemetry`) |
| **MxAccessGateway** | ✅ **0.3.1** (`main` `453fe015`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) |
| **ScadaBridge** | ✅ **0.3.1** (`main` `95584e2f`, both roots) | ✅ **0.3.0** `AddZbSerilog` with `MinimumLevelGovernor` + `ConfigureLogger` on a pre-host builder; the exporter is passed explicitly via `ScadaBridgeTelemetryConfig` |
| **HistorianGateway** | ✅ **0.3.1** (`main` `dffb5e2`) — `node.role=standalone`, `site.id` from config | ✅ **0.3.0** (logs follow `AddZbTelemetry`) |
All four pin both packages at **`0.3.0`** on their origin default branches since 2026-11-12 (scadaproj#55; not
deployed — the wonder boxes still run their earlier builds, which have no collector anyway, #54). See `components/observability/GAPS.md` for the full result.
All four pin both packages at **`0.3.1`** on their origin default branches since 2026-11-12 (scadaproj#56; `0.3.0`
earlier the same day, #55; not deployed — the wonder boxes still run their earlier builds, which have no collector anyway, #54). See `components/observability/GAPS.md` for the full result.
---
@@ -53,9 +53,9 @@ All test assemblies run offline:
| Assembly | Tests |
|---|---|
| `ZB.MOM.WW.Telemetry.Tests` | 38 |
| `ZB.MOM.WW.Telemetry.Serilog.Tests` | 39 |
| **Total** | **77** |
| `ZB.MOM.WW.Telemetry.Tests` | 54 |
| `ZB.MOM.WW.Telemetry.Serilog.Tests` | 40 |
| **Total** | **94** |
`GeneratePackageOnBuild` is off — pack explicitly with the command above.
@@ -63,6 +63,27 @@ All test assemblies run offline:
## Status
**`0.3.1` (2026-11-12, scadaproj#56) — PUBLISHED** (both packages, packed from scadaproj `main` `85654d7`, the
commit recorded in the nuspec; 54 + 40 tests green in Release). SHA256 (feed download = local pack):
`ZB.MOM.WW.Telemetry.0.3.1.nupkg` `baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3`,
`ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg` `188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931`.
Adopted by all four apps the same day.
- **Telemetry's own OTLP transport is not traced.** The SDK's OTLP exporters run inside
`SuppressInstrumentationScope`; `Serilog.Sinks.OpenTelemetry` does not, so every log batch produced a root
`opentelemetry.proto.collector.logs.v1.LogsService/Export` gRPC client span (scadabridge 146, mxgateway 13,
historian-gateway 10, otopcua 6 at the collector). `ZbOtlpExportCallFilter.IsExportCall` matches any
`/opentelemetry.proto.collector.*` path (any host) and OTLP/HTTP `/v1/{traces,metrics,logs}` on the configured
endpoint's host:port. HttpClient instrumentation drops those via `FilterHttpRequestMessage`. The gRPC client
instrumentation has **no Filter option**, so its `EnrichWithHttpRequestMessage` callback, which runs at start
before the HTTP request, clears `Recorded` + `IsAllDataRequested`. That also leaves the child HTTP activity
unsampled. Every other outgoing call keeps its span.
- Tests: predicate theory (16) + an end-to-end run against the loopback OTLP receiver. Positive controls: the app's
gRPC and HttpClient calls are traced. Negatives: the real log-sink export, `/v1/logs` and a collector path over
HttpClient are not. The receiver's own ASP.NET Core server spans share the process, so assertions key on
client-span markers (span name / `url.full`). Mutation-checked: no gRPC un-record, no HttpClient filter, and
"everything is export" each fail it. `VersionOf_reads_the_informational_version` now derives the expected prefix
from the assembly version instead of a `0.3.0` literal.
**`0.3.0` (2026-11-12, scadaproj#55) — PUBLISHED** (both packages, packed from the tree of scadaproj `main` `2011c8d` (committed as `df9a328` before rebasing onto main; identical `ZB.MOM.WW.Telemetry/` tree); 38 + 39
tests green in Release). SHA256 (feed download = local pack): `ZB.MOM.WW.Telemetry.0.3.0.nupkg`
`a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2`, `ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg`
@@ -80,7 +101,7 @@ tests green in Release). SHA256 (feed download = local pack): `ZB.MOM.WW.Telemet
- Includes the never-published **0.2.1** fixes below, including the stricter OTLP endpoint check. That check
was shipped deliberately: no checked-in config sets an endpoint, it only runs when `Exporter=Otlp`, and no
wonder tier has a collector (#54).
- Feed versions: `0.1.0`, `0.2.0`, `0.3.0` (`0.2.1` was never published).
- Feed versions: `0.1.0`, `0.2.0`, `0.3.0`, `0.3.1` (`0.2.1` was never published).
**`0.2.1` (2026-11-09, scadaproj#48) — never published; shipped inside `0.3.0`.** P3 review fixes: `RedactionEnricher` is now registered
AFTER `ConfigureLogger`, so enrichers added in the hook are redacted (009); dictionary keys that print identically are