docs(observability): Telemetry 0.3.1 published, adopted by all four apps, export spans gone at the collector (#56)
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -384,7 +384,76 @@ The rig stays on the rebuilt `482ac665` image.
|
||||
- **scadaproj#56:** the Serilog OTLP sink's own `LogsService/Export` calls are traced as gRPC client spans in
|
||||
**all four apps** (scadabridge 127, mxgateway 13, historian-gateway 10, otopcua 6 at the collector). The SDK's
|
||||
trace and metric exporters suppress their own instrumentation, and `Serilog.Sinks.OpenTelemetry` does not. This
|
||||
is noise, not a loop (0 log records about it). Fix in `ZB.MOM.WW.Telemetry` with an instrumentation filter.
|
||||
is noise, not a loop (0 log records about it). Fix in `ZB.MOM.WW.Telemetry` with an instrumentation filter. **Fixed in 0.3.1, same day — see below.**
|
||||
- **ScadaBridge#134:** rig images report `service.version = "1.0.0"` with no `+<sha>`, because `.dockerignore`
|
||||
excludes `.git/`. Fix by passing `SourceRevisionId` as a build arg. Wonder deploys, published from a checkout, are
|
||||
unaffected.
|
||||
|
||||
---
|
||||
|
||||
# OTLP export calls no longer traced — scadaproj#56 (2026-11-12, same day)
|
||||
|
||||
## Fix: `ZB.MOM.WW.Telemetry(.Serilog)` 0.3.1 — published
|
||||
|
||||
The SDK's OTLP exporters suppress their own instrumentation. `Serilog.Sinks.OpenTelemetry` does not, so
|
||||
`AddGrpcClientInstrumentation()` traced every log batch it sent. `AddZbTelemetry` now classifies an outgoing call
|
||||
with `ZbOtlpExportCallFilter.IsExportCall`: any `/opentelemetry.proto.collector.*` path on any host, or
|
||||
OTLP/HTTP `/v1/{traces,metrics,logs}` on the configured `OtlpEndpoint`'s host:port.
|
||||
|
||||
- **HttpClient:** `FilterHttpRequestMessage` returns `false` for an export call.
|
||||
- **gRPC client:** `GrpcClientTraceInstrumentationOptions` has no `Filter` in `1.15.1-beta.1`. The
|
||||
`EnrichWithHttpRequestMessage` callback runs synchronously in the instrumentation's start handler, before
|
||||
`HttpClient.SendAsync` (checked in the contrib source of that tag). For an export call it clears
|
||||
`ActivityTraceFlags.Recorded` and `IsAllDataRequested`, so the export processor skips the span and the child
|
||||
HTTP activity is unsampled under the default parent-based sampler.
|
||||
- A processor that drops spans in `OnEnd` was rejected. It only works when it is registered before every exporter,
|
||||
and an app could add an exporter first. The instrumentation hooks do not depend on processor order.
|
||||
|
||||
Published from scadaproj `main` `85654d7` (the commit in the nuspec). SHA256, feed download = local pack:
|
||||
`ZB.MOM.WW.Telemetry.0.3.1.nupkg` `baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3`,
|
||||
`ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg` `188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931`.
|
||||
Registration lists `0.1.0, 0.2.0, 0.3.0, 0.3.1` for both ids.
|
||||
|
||||
**Tests (54 + 40, Release):**
|
||||
- `OtlpExportCallFilterTests`: 16 cases. The positives are the three collector services, a collector path on
|
||||
another host, and `/v1/*` on the endpoint. The negatives are ScadaBridge→site, HG→historian and mxgw paths, REST,
|
||||
a `collectorx` look-alike, `/v1/logs` on another host or port, and relative URIs.
|
||||
- `Otlp_export_calls_are_not_traced_but_the_apps_own_calls_are`: a host runs against the loopback OTLP receiver.
|
||||
The positive controls are an app gRPC call (`zb.test.v1.AppService/Ping`) and an HttpClient call, both traced. The
|
||||
negatives are a real Serilog-sink log batch (the test asserts it arrived), an HttpClient `POST /v1/logs` to the
|
||||
endpoint, and a collector path over HttpClient; none is traced. The receiver's own ASP.NET Core server spans land
|
||||
in the same export because the instrumentation listens process-wide. So the assertions key on client-span markers:
|
||||
the protobuf span name for gRPC and `url.full` for HttpClient.
|
||||
- Mutations: removing the gRPC un-record fails the test (*"LogsService/Export call was traced"*). Removing the
|
||||
HttpClient filter fails it (`url.full …/v1/logs` found). Making `IsExportCall` always true fails it too
|
||||
(*"positive control: the app's gRPC call was not traced"*). It passed 5 of 5 repeated runs.
|
||||
|
||||
## Apps (pushed to their default branch; no wonder-box deploys)
|
||||
|
||||
| App | Commit | Pin | Tests (Debug, against 0.3.1 — `project.assets.json` confirms it on every project) |
|
||||
|---|---|---|---|
|
||||
| OtOpcUa | `master` `f3178c5f` | `Directory.Packages.props` | `Host.IntegrationTests` 325 passed / 2 skipped |
|
||||
| MxAccessGateway | `main` `453fe015` | `Server.csproj` | NonWindows solution: 1446 + 5 passed (28 skipped) |
|
||||
| HistorianGateway | `main` `dffb5e2` | `Server.csproj` | 186 + 206 + 1414 passed (76 skipped) |
|
||||
| ScadaBridge | `main` `95584e2f` | `Directory.Packages.props` | `Host.Tests` 721 passed |
|
||||
|
||||
## Collector before/after
|
||||
|
||||
The collector-wide count of `LogsService/Export` spans (`zb-otel-collector`, all runs to date) was the same before
|
||||
and after both probe runs: **scadabridge 146, mxgateway 13, historian-gateway 10, otopcua 6**. The issue reported
|
||||
127 for scadabridge; the ScadaBridge#133 centrals added more before the rig was restored. Each probe ran as a
|
||||
throwaway local process on 0.3.1 with `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317`,
|
||||
`OTEL_METRIC_EXPORT_INTERVAL=10000` and `site.id=otel-probe-56`, counted from its own start time:
|
||||
|
||||
| Run | Log export batches / records arrived | `*Service/Export` spans | App client spans still arriving |
|
||||
|---|---|---|---|
|
||||
| HistorianGateway `Development`, `:15220`/`:15221`, scratch stores, health/metrics GETs + 3 `HistorianStatus/Probe` | 15 / 168 | **0** | `HistoryService/GetInterfaceVersion` 14 (GrpcNetClient), `historian.Connect` 14, server spans |
|
||||
| ScadaBridge **Site** `95584e2f`, own site id, own 1-node mesh `localhost:18082`, gRPC `:18083`, metrics `:18084`, central `localhost:18999` unreachable, scratch LocalDb/audit/secrets | 27 / 224 | **0** | `CentralControlService/Heartbeat` 17, `ReconcileSite` 2, `ReportSiteHealth` 2 (GrpcNetClient), server spans |
|
||||
|
||||
So the log sink exported throughout, the gRPC client instrumentation stayed on, and no export call was traced.
|
||||
mxgw is proven by the library tests and its suite only; no windev scratch run was made. The docker rigs were not
|
||||
touched.
|
||||
|
||||
The wonder boxes (#54) still run their earlier builds. They will emit `LogsService/Export` spans once a 0.3.0 build
|
||||
is deployed, unless 0.3.1 is deployed instead.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user