[Telemetry] Serilog OTLP log sink's own LogsService/Export calls are traced as gRPC client spans (all four apps) #56

Closed
opened 2026-11-12 21:49:06 -05:00 by dohertj2 · 1 comment
Owner

Found while proving ScadaBridge Central OTLP export (ScadaBridge#133, 2026-11-12).

With Exporter=Otlp (Telemetry 0.3.0), every batch the Serilog OTLP sink sends produces a root client span opentelemetry.proto.collector.logs.v1.LogsService/Export (scope OpenTelemetry.Instrumentation.GrpcNetClient). The trace and metric exports do not show up, because the OTel SDK's own OTLP exporter runs its calls inside SuppressInstrumentationScope. Serilog.Sinks.OpenTelemetry is not part of the SDK and does not, so AddGrpcClientInstrumentation() (ZbTelemetryExtensions.cs:83) traces it.

Collector count on zb-otel-collector (all runs to date):

service LogsService/Export spans
scadabridge 127 (93 from the two rig centrals in ~2 min of the #133 run, the next-largest span after Heartbeat on central-a)
mxgateway 13
historian-gateway 10
otopcua 6

There is no feedback loop into logs: 0 log records mention the export, because HttpClient/gRPC logging is not at Information. So this is noise, not a runaway, but it scales with log volume and pollutes every trace backend.

Fix options (library side, ZB.MOM.WW.Telemetry):

  • (recommended) AddGrpcClientInstrumentation(o => o.Filter...) / AddHttpClientInstrumentation(o => o.FilterHttpRequestMessage = ...): drop requests whose URI path starts with /opentelemetry.proto.collector.. This is cheap, independent of the endpoint and covers any OTLP sink.
  • Or filter on the configured OtlpEndpoint host:port (more exact, but misses a sink pointed elsewhere).

Either needs a test showing the span is dropped, with a positive control for an ordinary gRPC call, then a release and the four pin bumps.

Found while proving ScadaBridge Central OTLP export (ScadaBridge#133, 2026-11-12). With `Exporter=Otlp` (Telemetry 0.3.0), every batch the Serilog OTLP sink sends produces a root client span `opentelemetry.proto.collector.logs.v1.LogsService/Export` (scope `OpenTelemetry.Instrumentation.GrpcNetClient`). The trace and metric exports do **not** show up, because the OTel SDK's own OTLP exporter runs its calls inside `SuppressInstrumentationScope`. `Serilog.Sinks.OpenTelemetry` is not part of the SDK and does not, so `AddGrpcClientInstrumentation()` (`ZbTelemetryExtensions.cs:83`) traces it. Collector count on `zb-otel-collector` (all runs to date): | service | LogsService/Export spans | |---|---| | scadabridge | 127 (93 from the two rig centrals in ~2 min of the #133 run, the next-largest span after Heartbeat on central-a) | | mxgateway | 13 | | historian-gateway | 10 | | otopcua | 6 | There is no feedback loop into logs: 0 log records mention the export, because HttpClient/gRPC logging is not at Information. So this is noise, not a runaway, but it scales with log volume and pollutes every trace backend. **Fix options (library side, `ZB.MOM.WW.Telemetry`):** - *(recommended)* `AddGrpcClientInstrumentation(o => o.Filter...)` / `AddHttpClientInstrumentation(o => o.FilterHttpRequestMessage = ...)`: drop requests whose URI path starts with `/opentelemetry.proto.collector.`. This is cheap, independent of the endpoint and covers any OTLP sink. - Or filter on the configured `OtlpEndpoint` host:port (more exact, but misses a sink pointed elsewhere). Either needs a test showing the span is dropped, with a positive control for an ordinary gRPC call, then a release and the four pin bumps.
dohertj2 added the type:bugpriority:p3cross-repo labels 2026-11-12 21:49:06 -05:00
Author
Owner

Fixed in ZB.MOM.WW.Telemetry(.Serilog) 0.3.1. It is published and adopted by all four apps. No wonder deploys were made.

Fix (85654d7): ZbOtlpExportCallFilter.IsExportCall matches any /opentelemetry.proto.collector.* path on any host. It also matches OTLP/HTTP /v1/{traces,metrics,logs} on the configured endpoint's host:port.

  • HttpClient: FilterHttpRequestMessage drops these calls.
  • gRPC client: GrpcClientTraceInstrumentationOptions has no Filter in 1.15.1-beta.1. Instead, the start-time EnrichWithHttpRequestMessage callback clears Recorded + IsAllDataRequested. It runs before the HTTP request starts, so the child HTTP activity is unsampled too.
  • A processor that drops spans in OnEnd was rejected, because it only works if it is registered before every exporter.
  • Every other outgoing call keeps its span.

Tests (54 + 40 green):

  • A predicate theory with 16 cases.
  • An end-to-end test against the loopback OTLP receiver. The positive controls are an app gRPC call and an HttpClient call, both traced. The negatives are the real Serilog log-sink export, /v1/logs to the endpoint, and a collector path over HttpClient, none traced.
  • Three mutations each fail the test: no gRPC un-record, no HttpClient filter, and "everything is export".

Published: both packages are on the feed (registration: 0.1.0, 0.2.0, 0.3.0, 0.3.1), and the feed download matches the local pack.

  • ZB.MOM.WW.Telemetry.0.3.1.nupkg baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3
  • ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg 188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931

Adopted (pushed):

  • OtOpcUa master f3178c5f: Host.IntegrationTests 325 passed.
  • mxgw main 453fe015: NonWindows suite 1451 passed.
  • HistorianGateway main dffb5e2: 1806 passed.
  • ScadaBridge main 95584e2f: Host.Tests 721 passed.

Collector: the collector-wide LogsService/Export count was the same before and after the probe runs: scadabridge 146, mxgateway 13, historian-gateway 10, otopcua 6. Each probe was a throwaway local process on 0.3.1 sending to 10.100.0.35:4317, counted from its own start:

run log batches / records export spans app client spans
HistorianGateway 15 / 168 0 HistoryService/GetInterfaceVersion 14
ScadaBridge Site 27 / 224 0 CentralControlService/Heartbeat 17, ReconcileSite 2, ReportSiteHealth 2

mxgw is proven by tests only. The docker rigs were not touched.

Evidence: components/observability/2026-11-12-otlp-end-to-end-evidence.md (section "scadaproj#56"), records commit 41a5e4d.

Fixed in **ZB.MOM.WW.Telemetry(.Serilog) 0.3.1**. It is published and adopted by all four apps. No wonder deploys were made. **Fix (`85654d7`):** `ZbOtlpExportCallFilter.IsExportCall` matches any `/opentelemetry.proto.collector.*` path on any host. It also matches OTLP/HTTP `/v1/{traces,metrics,logs}` on the configured endpoint's host:port. - HttpClient: `FilterHttpRequestMessage` drops these calls. - gRPC client: `GrpcClientTraceInstrumentationOptions` has **no `Filter`** in 1.15.1-beta.1. Instead, the start-time `EnrichWithHttpRequestMessage` callback clears `Recorded` + `IsAllDataRequested`. It runs before the HTTP request starts, so the child HTTP activity is unsampled too. - A processor that drops spans in `OnEnd` was rejected, because it only works if it is registered before every exporter. - Every other outgoing call keeps its span. **Tests (54 + 40 green):** - A predicate theory with 16 cases. - An end-to-end test against the loopback OTLP receiver. The positive controls are an app gRPC call and an HttpClient call, both traced. The negatives are the real Serilog log-sink export, `/v1/logs` to the endpoint, and a collector path over HttpClient, none traced. - Three mutations each fail the test: no gRPC un-record, no HttpClient filter, and "everything is export". **Published:** both packages are on the feed (registration: 0.1.0, 0.2.0, 0.3.0, 0.3.1), and the feed download matches the local pack. - `ZB.MOM.WW.Telemetry.0.3.1.nupkg` `baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3` - `ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg` `188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931` **Adopted (pushed):** - OtOpcUa `master` `f3178c5f`: Host.IntegrationTests 325 passed. - mxgw `main` `453fe015`: NonWindows suite 1451 passed. - HistorianGateway `main` `dffb5e2`: 1806 passed. - ScadaBridge `main` `95584e2f`: Host.Tests 721 passed. **Collector:** the collector-wide `LogsService/Export` count was the same before and after the probe runs: scadabridge 146, mxgateway 13, historian-gateway 10, otopcua 6. Each probe was a throwaway local process on 0.3.1 sending to `10.100.0.35:4317`, counted from its own start: | run | log batches / records | export spans | app client spans | |---|---|---|---| | HistorianGateway | 15 / 168 | **0** | `HistoryService/GetInterfaceVersion` 14 | | ScadaBridge Site | 27 / 224 | **0** | `CentralControlService/Heartbeat` 17, `ReconcileSite` 2, `ReportSiteHealth` 2 | mxgw is proven by tests only. The docker rigs were not touched. Evidence: `components/observability/2026-11-12-otlp-end-to-end-evidence.md` (section "scadaproj#56"), records commit `41a5e4d`.
Sign in to join this conversation.