docs(observability): Telemetry 0.3.1 published, adopted by all four apps, export spans gone at the collector (#56)

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 22:21:07 -05:00
parent 71b8595742
commit 41a5e4d187
3 changed files with 103 additions and 13 deletions
@@ -384,7 +384,76 @@ The rig stays on the rebuilt `482ac665` image.
- **scadaproj#56:** the Serilog OTLP sink's own `LogsService/Export` calls are traced as gRPC client spans in
**all four apps** (scadabridge 127, mxgateway 13, historian-gateway 10, otopcua 6 at the collector). The SDK's
trace and metric exporters suppress their own instrumentation, and `Serilog.Sinks.OpenTelemetry` does not. This
is noise, not a loop (0 log records about it). Fix in `ZB.MOM.WW.Telemetry` with an instrumentation filter.
is noise, not a loop (0 log records about it). Fix in `ZB.MOM.WW.Telemetry` with an instrumentation filter. **Fixed in 0.3.1, same day — see below.**
- **ScadaBridge#134:** rig images report `service.version = "1.0.0"` with no `+<sha>`, because `.dockerignore`
excludes `.git/`. Fix by passing `SourceRevisionId` as a build arg. Wonder deploys, published from a checkout, are
unaffected.
---
# OTLP export calls no longer traced — scadaproj#56 (2026-11-12, same day)
## Fix: `ZB.MOM.WW.Telemetry(.Serilog)` 0.3.1 — published
The SDK's OTLP exporters suppress their own instrumentation. `Serilog.Sinks.OpenTelemetry` does not, so
`AddGrpcClientInstrumentation()` traced every log batch it sent. `AddZbTelemetry` now classifies an outgoing call
with `ZbOtlpExportCallFilter.IsExportCall`: any `/opentelemetry.proto.collector.*` path on any host, or
OTLP/HTTP `/v1/{traces,metrics,logs}` on the configured `OtlpEndpoint`'s host:port.
- **HttpClient:** `FilterHttpRequestMessage` returns `false` for an export call.
- **gRPC client:** `GrpcClientTraceInstrumentationOptions` has no `Filter` in `1.15.1-beta.1`. The
`EnrichWithHttpRequestMessage` callback runs synchronously in the instrumentation's start handler, before
`HttpClient.SendAsync` (checked in the contrib source of that tag). For an export call it clears
`ActivityTraceFlags.Recorded` and `IsAllDataRequested`, so the export processor skips the span and the child
HTTP activity is unsampled under the default parent-based sampler.
- A processor that drops spans in `OnEnd` was rejected. It only works when it is registered before every exporter,
and an app could add an exporter first. The instrumentation hooks do not depend on processor order.
Published from scadaproj `main` `85654d7` (the commit in the nuspec). SHA256, feed download = local pack:
`ZB.MOM.WW.Telemetry.0.3.1.nupkg` `baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3`,
`ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg` `188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931`.
Registration lists `0.1.0, 0.2.0, 0.3.0, 0.3.1` for both ids.
**Tests (54 + 40, Release):**
- `OtlpExportCallFilterTests`: 16 cases. The positives are the three collector services, a collector path on
another host, and `/v1/*` on the endpoint. The negatives are ScadaBridge→site, HG→historian and mxgw paths, REST,
a `collectorx` look-alike, `/v1/logs` on another host or port, and relative URIs.
- `Otlp_export_calls_are_not_traced_but_the_apps_own_calls_are`: a host runs against the loopback OTLP receiver.
The positive controls are an app gRPC call (`zb.test.v1.AppService/Ping`) and an HttpClient call, both traced. The
negatives are a real Serilog-sink log batch (the test asserts it arrived), an HttpClient `POST /v1/logs` to the
endpoint, and a collector path over HttpClient; none is traced. The receiver's own ASP.NET Core server spans land
in the same export because the instrumentation listens process-wide. So the assertions key on client-span markers:
the protobuf span name for gRPC and `url.full` for HttpClient.
- Mutations: removing the gRPC un-record fails the test (*"LogsService/Export call was traced"*). Removing the
HttpClient filter fails it (`url.full …/v1/logs` found). Making `IsExportCall` always true fails it too
(*"positive control: the app's gRPC call was not traced"*). It passed 5 of 5 repeated runs.
## Apps (pushed to their default branch; no wonder-box deploys)
| App | Commit | Pin | Tests (Debug, against 0.3.1 — `project.assets.json` confirms it on every project) |
|---|---|---|---|
| OtOpcUa | `master` `f3178c5f` | `Directory.Packages.props` | `Host.IntegrationTests` 325 passed / 2 skipped |
| MxAccessGateway | `main` `453fe015` | `Server.csproj` | NonWindows solution: 1446 + 5 passed (28 skipped) |
| HistorianGateway | `main` `dffb5e2` | `Server.csproj` | 186 + 206 + 1414 passed (76 skipped) |
| ScadaBridge | `main` `95584e2f` | `Directory.Packages.props` | `Host.Tests` 721 passed |
## Collector before/after
The collector-wide count of `LogsService/Export` spans (`zb-otel-collector`, all runs to date) was the same before
and after both probe runs: **scadabridge 146, mxgateway 13, historian-gateway 10, otopcua 6**. The issue reported
127 for scadabridge; the ScadaBridge#133 centrals added more before the rig was restored. Each probe ran as a
throwaway local process on 0.3.1 with `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317`,
`OTEL_METRIC_EXPORT_INTERVAL=10000` and `site.id=otel-probe-56`, counted from its own start time:
| Run | Log export batches / records arrived | `*Service/Export` spans | App client spans still arriving |
|---|---|---|---|
| HistorianGateway `Development`, `:15220`/`:15221`, scratch stores, health/metrics GETs + 3 `HistorianStatus/Probe` | 15 / 168 | **0** | `HistoryService/GetInterfaceVersion` 14 (GrpcNetClient), `historian.Connect` 14, server spans |
| ScadaBridge **Site** `95584e2f`, own site id, own 1-node mesh `localhost:18082`, gRPC `:18083`, metrics `:18084`, central `localhost:18999` unreachable, scratch LocalDb/audit/secrets | 27 / 224 | **0** | `CentralControlService/Heartbeat` 17, `ReconcileSite` 2, `ReportSiteHealth` 2 (GrpcNetClient), server spans |
So the log sink exported throughout, the gRPC client instrumentation stayed on, and no export call was traced.
mxgw is proven by the library tests and its suite only; no windev scratch run was made. The docker rigs were not
touched.
The wonder boxes (#54) still run their earlier builds. They will emit `LogsService/Export` spans once a 0.3.0 build
is deployed, unless 0.3.1 is deployed instead.