[Telemetry] Serilog OTLP log sink's own LogsService/Export calls are traced as gRPC client spans (all four apps) #56
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found while proving ScadaBridge Central OTLP export (ScadaBridge#133, 2026-11-12).
With
Exporter=Otlp(Telemetry 0.3.0), every batch the Serilog OTLP sink sends produces a root client spanopentelemetry.proto.collector.logs.v1.LogsService/Export(scopeOpenTelemetry.Instrumentation.GrpcNetClient). The trace and metric exports do not show up, because the OTel SDK's own OTLP exporter runs its calls insideSuppressInstrumentationScope.Serilog.Sinks.OpenTelemetryis not part of the SDK and does not, soAddGrpcClientInstrumentation()(ZbTelemetryExtensions.cs:83) traces it.Collector count on
zb-otel-collector(all runs to date):There is no feedback loop into logs: 0 log records mention the export, because HttpClient/gRPC logging is not at Information. So this is noise, not a runaway, but it scales with log volume and pollutes every trace backend.
Fix options (library side,
ZB.MOM.WW.Telemetry):AddGrpcClientInstrumentation(o => o.Filter...)/AddHttpClientInstrumentation(o => o.FilterHttpRequestMessage = ...): drop requests whose URI path starts with/opentelemetry.proto.collector.. This is cheap, independent of the endpoint and covers any OTLP sink.OtlpEndpointhost:port (more exact, but misses a sink pointed elsewhere).Either needs a test showing the span is dropped, with a positive control for an ordinary gRPC call, then a release and the four pin bumps.
Fixed in ZB.MOM.WW.Telemetry(.Serilog) 0.3.1. It is published and adopted by all four apps. No wonder deploys were made.
Fix (
85654d7):ZbOtlpExportCallFilter.IsExportCallmatches any/opentelemetry.proto.collector.*path on any host. It also matches OTLP/HTTP/v1/{traces,metrics,logs}on the configured endpoint's host:port.FilterHttpRequestMessagedrops these calls.GrpcClientTraceInstrumentationOptionshas noFilterin 1.15.1-beta.1. Instead, the start-timeEnrichWithHttpRequestMessagecallback clearsRecorded+IsAllDataRequested. It runs before the HTTP request starts, so the child HTTP activity is unsampled too.OnEndwas rejected, because it only works if it is registered before every exporter.Tests (54 + 40 green):
/v1/logsto the endpoint, and a collector path over HttpClient, none traced.Published: both packages are on the feed (registration: 0.1.0, 0.2.0, 0.3.0, 0.3.1), and the feed download matches the local pack.
ZB.MOM.WW.Telemetry.0.3.1.nupkgbaf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931Adopted (pushed):
masterf3178c5f: Host.IntegrationTests 325 passed.main453fe015: NonWindows suite 1451 passed.maindffb5e2: 1806 passed.main95584e2f: Host.Tests 721 passed.Collector: the collector-wide
LogsService/Exportcount was the same before and after the probe runs: scadabridge 146, mxgateway 13, historian-gateway 10, otopcua 6. Each probe was a throwaway local process on 0.3.1 sending to10.100.0.35:4317, counted from its own start:HistoryService/GetInterfaceVersion14CentralControlService/Heartbeat17,ReconcileSite2,ReportSiteHealth2mxgw is proven by tests only. The docker rigs were not touched.
Evidence:
components/observability/2026-11-12-otlp-end-to-end-evidence.md(section "scadaproj#56"), records commit41a5e4d.