Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
34 KiB
OTLP export, end to end — evidence (2026-11-12, scadaproj#13)
First run of the family's OTLP export path against a real collector. Before this, every app had
the <App>:Telemetry:Exporter=Otlp + :OtlpEndpoint wiring but nothing had ever received from
it, and OtOpcUa's spans had no exporter anywhere.
Result: OTLP metrics + traces proven for OtOpcUa, HistorianGateway, ScadaBridge (Site
role) and MxAccessGateway. One real defect found and fixed (OtOpcUa Galaxy driver telemetry was
never exported, lmxopcua#892). No Telemetry library change was needed; 0.2.1 stays unpublished.
Collector
zb-otel-collector on the shared Docker host 10.100.0.35 — otel/opentelemetry-collector-contrib:0.162.0,
OTLP gRPC :4317 + HTTP :4318, Prometheus exporter :8889, health :13133; debug + file
exporters for proof. Source and runbook: infra/otel-collector/. The
four ports were free on the host before deploy, and no other container was touched.
$ curl -s http://10.100.0.35:13133/
{"status":"Server available","upSince":"2026-11-13T01:24:13.274419432Z",...}
$ curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d '{}' http://10.100.0.35:4318/v1/traces
200
windev> Test-NetConnection 10.100.0.35 -Port 4317 -InformationLevel Quiet
True
How each app was run
No shared rig container was recreated or reconfigured. Each app ran as a throwaway process with
its own state, Exporter=Otlp, OtlpEndpoint=http://10.100.0.35:4317 (gRPC) and
OTEL_METRIC_EXPORT_INTERVAL=10000.
| App | Build | Telemetry pin | How |
|---|---|---|---|
| OtOpcUa | master aaaa55e2 + the #892 fix (0b90379f) |
0.1.0 | Local dotnet host on the Mac: fused admin,driver single node in its own Akka mesh (localhost:14053), its own throwaway ConfigDb OtOpcUaOtel13 on the docker-dev SQL (EF-migrated, one ServerCluster/ClusterNode row, dropped afterwards), HTTP :19100, OPC UA :14840. Activity: one POST /api/deployments + health/metrics GETs. |
| HistorianGateway | main b95bae3 |
0.1.0 | Local dotnet host, Development, HTTP :15220 / gRPC :15221, scratch SQLite stores. Activity: status RPCs via grpcurl (the historian itself is unreachable without the VPN, so they fail — which still produces spans) + dashboard/health GETs. |
| ScadaBridge | main 139a43fb |
0.2.0 | Local dotnet Site host, own site id otel-probe, own 1-node mesh (localhost:18082), scratch LocalDb/audit/secrets, metrics listener :18084, central endpoint deliberately unreachable. The docker rig was in active use (ScadaBridge#46/#132; containers recreated minutes earlier), so it was not touched. |
| MxAccessGateway | deployed windev Server 447070a5 (scratch copy) |
0.1.0 | Copy of C:\publish\mxaccessgw\Server-20261112-447070a5 run as a separate process on :15120/:15130, Staging, alarms off, scratch auth/secrets stores, Worker:ExecutablePath pointed at a non-existent OtelProbeNoWorker.exe so its OrphanWorkerTerminator (which matches by image name) could not match the live service's x86 workers. Live MxAccessGw stayed Running, :5130/health/ready = 200 throughout, worker PIDs 14700,22108 unchanged before/after; the scratch copy was deleted. |
What arrived (collector otelsum.py output, trimmed)
Standard instrumentation scopes that every app sent are listed once at the end.
OtOpcUa — metrics + traces
RESOURCE otopcua {"service.name": "otopcua", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:48096", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS otopcua ZB.MOM.WW.OtOpcUa 3 ['otopcua.deploy.applied', 'otopcua.deploy.apply.duration', 'otopcua.redundancy.service_level_change']
METRICS otopcua ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.deploy.apply')
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.opcua.address_space_rebuild')
SPANS 1 ('otopcua', 'Microsoft.AspNetCore', 'POST /api/deployments')
SPANS 5 ('otopcua', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 3 ('otopcua', 'Microsoft.AspNetCore', 'GET /metrics')
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.deploy.apply", "traceId": "e5a579168f8c4629b25af4d3e4f2cb07", "spanId": "4f12d04ada38ea41", "kind": 1,
"attributes": [{"key": "otopcua.deployment_id", "value": {"stringValue": "e8b8c010d50144ccb371a91defceb6d9"}}, {"key": "otopcua.node_id", "value": {"stringValue": "localhost:14053"}},
{"key": "otopcua.revision", "value": {"stringValue": "ed37612b…7bba"}}, {"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.opcua.address_space_rebuild", "traceId": "f6d8c238b2399cce0928de84691a0106", "spanId": "305ec5a9012a799d", "kind": 1,
"attributes": [{"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
OtOpcUa's tracing is no longer a no-op once Exporter=Otlp is set: its own spans reach a backend
with their attributes. Note that address_space_rebuild is a separate trace from deploy.apply
(no parent — the context does not cross the Akka actor hop); the two join on
otopcua.correlation_id.
HistorianGateway — metrics + traces
RESOURCE historian-gateway {"service.name": "historian-gateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:49942", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS historian-gateway ZB.MOM.WW.HistorianGateway 10 ['historian_gateway.grpc.request.duration', 'historian_gateway.grpc.requests', 'historian_gateway.historian.connection.up', 'historian_gateway.historian.event_session_pool.lease_waiters', 'historian_gateway.historian.event_session_pool.warm_sessions', 'historian_gateway.historian.session_pool.lease_waiters', 'historian_gateway.historian.session_pool.warm_sessions', 'historian_gateway.redundancy.members.healthy', 'historian_gateway.session_pool.lease.wait.duration', 'historian_gateway.store_forward.queue.depth']
SPANS 8 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.Connect')
SPANS 2 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.pool.lease')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/Probe')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/GetConnectionStatus')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'galaxy_repository.v1.GalaxyRepository/GetLastDeployTime')
SPANS 8 ('historian-gateway', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'HistoryService/GetInterfaceVersion')
SPANS 4 ('historian-gateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.HistorianGateway.Server.Dashboard.Components.Pages.Home')
historian.Connect spans carry server.address/server.port and error.type=…HistorianTransportException
(no VPN); the ones a request triggered are parented under its gRPC request span, the pool warm-up ones are roots. Prometheus exporter, same data:
historian_gateway_grpc_requests_total{job="ZB.MOM.WW/historian-gateway",method="Probe",outcome="ok",service="HistorianStatus",service_name="historian-gateway",...} 1
ScadaBridge (Site) — metrics + traces
RESOURCE scadabridge {"service.name": "scadabridge", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:50923", "host.name": "Josephs-MacBook-Pro", "site.id": "otel-probe", "node.role": "Site", "_sdk": "1.15.3"}
METRICS scadabridge ZB.MOM.WW.ScadaBridge 6 ['scadabridge.central.transport.import_sessions.open', 'scadabridge.site.alarm_cache.aggregators.active', 'scadabridge.site.data_connections.connected', 'scadabridge.site.data_connections.total', 'scadabridge.site.stream.subscribers.active', 'scadabridge.store_and_forward.queue.depth']
METRICS scadabridge ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 12 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/Heartbeat')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReportSiteHealth')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReconcileSite')
SPANS 4 ('scadabridge', 'Microsoft.AspNetCore', 'GET /health/ready')
The only app that sends site.id / node.role. ScadaBridge has no ActivitySource of its own,
so its spans are standard instrumentation only. Central role not run → ScadaBridge#133.
MxAccessGateway — metrics + traces
RESOURCE mxgateway {"service.name": "mxgateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "DESKTOP-6JL3KKO:8720", "host.name": "DESKTOP-6JL3KKO", "_sdk": "1.15.3"}
METRICS mxgateway ZB.MOM.WW.MxGateway 5 ['mxgateway.alarms.provider_mode', 'mxgateway.events.grpc_stream_queue.depth', 'mxgateway.events.worker_queue.depth', 'mxgateway.sessions.open', 'mxgateway.workers.running']
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /metrics')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.MxGateway.Server.Dashboard.Components.Pages.DashboardHome')
Only the observable gauges appear: the counters/histograms (commands.*, workers.startup.duration, …)
record nothing without a worker session, and the x86 worker was deliberately not run. The gateway
has no ActivitySource of its own (spans are standard instrumentation only). Its gRPC port is not
reachable from off-box (windev firewall), so the traffic was generated on-box.
Standard instrumentation (all four)
System.Runtime (19 dotnet.*), OpenTelemetry.Instrumentation.Process (5 process.*),
System.Net.Http (4 http.client.*), Microsoft.AspNetCore.Hosting (http.server.request.duration,
http.server.active_requests), Microsoft.AspNetCore.Server.Kestrel, .Routing, .MemoryPool;
plus .Authentication/.Authorization/.Components on the apps with a Blazor dashboard.
Defect found and fixed
lmxopcua#892 — the Galaxy driver's telemetry was never exported. ZbTelemetryOptions.Meters
and .ActivitySources are strict allowlists. OtOpcUa named only ZB.MOM.WW.OtOpcUa and the
LocalDb meter, while the in-process Galaxy driver opens its spans (galaxy.subscribe_bulk,
galaxy.unsubscribe_bulk, galaxy.stream_events, galaxy.write, galaxy.get_hierarchy) on
ActivitySource ZB.MOM.WW.OtOpcUa.Driver.Galaxy and records the event pump's galaxy.events.*
counters on a Meter of the same name. All of it was silently dropped from /metrics and OTLP.
Fix: OtOpcUa master db86b996 (commit 0b90379f) — a public GalaxyDriverTelemetryNames in
the driver; ObservabilityExtensions.ObservedMeters / ObservedActivitySources name it.
ObservabilityAllowlistTests assert on the listener side (span sampled, instrument enabled) with an
unregistered-name negative control. Dropping the Galaxy entries fails 3 of 8 tests, so the tests can
catch the regression. Galaxy driver suite 584/584. Not deployed. A live galaxy.* span was not
driven: that would need a Galaxy driver deployment against a gateway.
The allowlists of the other apps were checked against every new Meter( / new ActivitySource(
in their source and in the hosted libraries. ScadaBridge (ZB.MOM.WW.ScadaBridge, LocalDb),
HistorianGateway (ZB.MOM.WW.HistorianGateway meter + source) and MxAccessGateway
(ZB.MOM.WW.MxGateway) are complete. Of the hosted libraries, only LocalDb defines a meter.
Checked, no defect
- Protocol/endpoint: every app uses the library's
AddOtlpExporterdefault (gRPC).http://host:4317works for all four. No checked-in config carries a scheme-less endpoint. - Prometheus stays up under Otlp:
/metricsanswered 200 on every app withExporter=Otlp(http.response.status_code=200on each app's exportedGET /metricsspan) (the overlay is additive, as documented). - Resource:
service.name,service.namespace=ZB.MOM.WW,service.instance.id,host.namepresent on all four.
Gaps filed (not fixed here)
- scadaproj#55:
service.versionis absent on all four apps;site.id/node.roleare absent on OtOpcUa, MxAccessGateway and HistorianGateway. No app exports logs over OTLP, because everyAddZbSerilogcall passes onlyServiceName, so the Serilog OTLP sink never activates (logs.jsonlstayed empty). - ScadaBridge#133: OTLP from the Central role is not yet proven.
- scadaproj#54: collector on the wonder tiers. DONE 2026-11-12: one loopback-only
ZbOtelCollectorper box on vd03/vq03/vq04, every app with an exporter setting switched to OTLP, and data proven at each collector. On vd03 and QA this includes the ScadaBridge Central role (node.role=Central). Record:infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md.
Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)
No finding needed a library change. The fixes and gaps above are all on the app side. 0.2.1's
scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also
need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays
built-but-unpublished until a pin bump wants it.
Resource identity + logs over OTLP — scadaproj#55 (2026-11-12, same day)
Closes the three #55 gaps found above: no service.version anywhere, no site.id/node.role on OtOpcUa,
MxAccessGateway and HistorianGateway, and no logs over OTLP from any app.
Root cause of "no logs"
ZbSerilogConfig.ApplyOpenTelemetryExport adds the WriteTo.OpenTelemetry sink only when the options
passed to AddZbSerilog say Exporter=Otlp. Every app filled those options with the service name alone
and set the exporter only on AddZbTelemetry, a separate options object the Serilog package never saw. The
sink condition could therefore never be true.
Library: ZB.MOM.WW.Telemetry(.Serilog) 0.3.0 — published
scadaproj main cfc7168 + 2011c8d. Feed SHA256 (feed download = local pack):
| Package | SHA256 |
|---|---|
ZB.MOM.WW.Telemetry.0.3.0.nupkg |
a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2 |
ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg |
04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c |
-
service.version:ServiceVersion = null(the default) now means the entry assembly'sAssemblyInformationalVersion(the git SHA the apps stamp), falling back to its assembly version. Set a value to override it;""omits the attribute. -
ResourceAttributes: extra Resource attributes on all three signals. The first-class keys (service.*,host.name,site.id,node.role) are reserved and rejected at startup. -
ApplyConfiguration(section): readsExporter,OtlpEndpoint,ServiceVersion,SiteId,NodeRole,ExportLogs,ResourceAttributes:<key>from an app's own<App>:Telemetrysection. Absent keys keep the code defaults, so the four hand-written exporter parsers are gone. -
Logs follow
AddZbTelemetry:AddZbTelemetryregisters its options in DI.AddZbSerilogresolves them inside itsAddSerilogcallback (so call order does not matter), and when its own options do not select OTLP, the OTLP log sink uses that registration's endpoint and Resource.ExportLogs = falsekeeps logs local. Levels and the configuration-driven sinks are unchanged. -
0.2.1 shipped in the same release. It holds the P3 review fixes: redaction after
ConfigureLogger, dictionary-key collisions, and the stricter endpoint check (collector:4317without a scheme now fails at startup whenExporter=Otlp). It is a minor bump because it adds API and changes behaviour. No checked-in config in any of the four repos sets an OTLP endpoint. The check is a no-op unlessExporter=Otlp, and no wonder tier had a collector at the time (#54). Since 2026-11-12 the wonder boxes setOtlpEndpoint=http://127.0.0.1:4317, scheme included, so they pass the check when 0.3.0 is deployed. -
Tests: 42 → 77 in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on Kestrel that proves a log record arrives with
service.version/site.id/node.role/an extra attribute, anExportLogs=falsecase with a metrics positive control, and a Prometheus case.Mutation checks, each caught by the named tests:
- Drop the DI lookup → the e2e log test fails.
- Drop the version default → the pipeline-Resource and e2e tests fail.
- Drop
ResourceAttributes→ 2 attribute tests and the e2e test fail. - Ignore
ExportLogs→ the decision test and the e2e test fail. - Disable the reserved-key check → 7 theory cases fail.
- Drop
ApplyConfiguration's attribute orSiteIdread → its test fails.
Apps (all pushed to their default branch; no wonder-box deploys)
| App | Commit | Pin | Identity wiring |
|---|---|---|---|
| OtOpcUa | master 412da27c |
0.1.0 → 0.3.0 | site.id = the application Cluster id (the cluster-<ClusterId> role suffix; omitted on a legacy node); node.role = the plain roles, sorted (admin,driver); Program passes its EffectiveRoles. |
| MxAccessGateway | main d9d9ff48 |
0.1.0 → 0.3.0 (csproj) | node.role=standalone (one gateway per box); site.id from MxGateway:Telemetry:SiteId (there is no node identity in its config, so it is omitted unless set). |
| HistorianGateway | main ae1dbcc |
0.1.0 → 0.3.0 (csproj) | Same as mxgw, under the root-level Telemetry: section. |
| ScadaBridge | main a633005f |
0.2.0 → 0.3.0 | Unchanged site.id/node.role. The new ScadaBridgeTelemetryConfig fills both AddZbTelemetry and the pre-host AddZbSerilog logger. The logger is built on its own container with no AddZbTelemetry to follow, so the exporter is passed to it explicitly. |
Tests: OtOpcUa Host.Tests 288/288 + Host.IntegrationTests 325/327 (2 skipped); mxgw MxGateway.Tests 1446/1452
(6 skipped); HistorianGateway 1414 + 206 + 186 passed; ScadaBridge Host.Tests 699/699. Each app gained identity
tests (OtOpcUa ObservabilityResourceIdentityTests, HG GatewayTracingTests, ScadaBridge
ScadaBridgeTelemetryConfigTests).
What arrived (collector, otelidentity.py, one process each)
All runs: Exporter=Otlp, OtlpEndpoint=http://10.100.0.35:4317, OTEL_METRIC_EXPORT_INTERVAL=10000.
OtOpcUa — local Release host, fused admin,driver,cluster-OTEL55 single node. Its own Akka mesh on
localhost:14053. Throwaway ConfigDb OtOpcUaOtel55 on the docker-dev SQL: EF-migrated, one
ServerCluster OTEL55 + one ClusterNode, gRPC mesh/telemetry keys of its own. The DB was dropped
afterwards. Activity: one POST /api/deployments (202) plus health/metrics GETs.
RESOURCE resourceMetrics {"service.name": "otopcua", "service.version": "1.0.0+412da27c5fb7e440e4a75d4f8e82e0e5b9832411", "service.instance.id": "Josephs-MacBook-Pro:67675", "site.id": "OTEL55", "node.role": "admin,driver", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 427, 'spans': 29, 'log records': 73} SEVERITY {'Information': 61, 'Warning': 12}
LOG {"severity": "Information", "body": "OtOpcUa.Host starting with roles=[admin,driver,cluster-OTEL55] (admin=True, driver=True) source=Configured"}
LOG {"severity": "Information", "body": "HTTP POST /telemetry.v1.TelemetryStreamService/Subscribe responded 200 in 29942.6698 ms", "traceId": "f5f90813adac357cdce4c165ab9e1468"}
The otopcua.deploy.apply and otopcua.opcua.address_space_rebuild spans arrived again.
HistorianGateway — local Release host, Development, HTTP :15220 / gRPC :15221, scratch SQLite stores,
Telemetry__SiteId=otel-probe-55. Activity: HistorianStatus/Probe + health GETs.
RESOURCE resourceMetrics {"service.name": "historian-gateway", "service.version": "1.0.0+ae1dbccc201aeebbe030c407870879a7985002db", "service.instance.id": "Josephs-MacBook-Pro:72641", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 184, 'spans': 12, 'log records': 29} SEVERITY {'Information': 25, 'Warning': 2, 'Error': 2}
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:15220/health/ready - null null", "traceId": "8eca56a689d61f710bdbdaad32879a6b"}
ScadaBridge (Site) — local Release Site host, own site id otel-probe-55, own 1-node mesh (localhost:18082),
scratch LocalDb/audit/secrets, metrics listener :18084, central endpoint deliberately unreachable.
RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0+a633005f4f826e662d09fa47359bced17ccd6acf", "service.instance.id": "Josephs-MacBook-Pro:69217", "site.id": "otel-probe-55", "node.role": "Site", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 290, 'spans': 30, 'log records': 78} SEVERITY {'Information': 66, 'Warning': 10, 'Error': 2}
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Site on localhost"}
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:18084/health/ready - null null", "traceId": "0e976e6c0b8d7d3d0aaf85a99f5fbe21"}
The first line, Starting ScadaBridge host …, is written to the static logger before any web host exists, so
the explicit exporter path on the pre-host logger works. Central not run: the rig was in use by the
ScadaBridge#132 failover drills (sb132-p / sb132-probe running, centrals recreated 3–6 minutes earlier), so
ScadaBridge#133 stays open.
MxAccessGateway — Release main 54aa61a0, published framework-dependent win-x64 from the Mac. (It was
pushed as d9d9ff48 after rebasing over one docs-only commit.) Run as a scratch copy at C:\otel55\mxgw on
windev, on :15120/:15130, under Staging + DisableLogin, with alarms off, scratch auth/secrets/snapshot
stores and a throwaway pepper. Worker:ExecutablePath pointed at the non-existent
C:\otel55\OtelProbeNoWorker.exe. It was launched through Win32_Process.Create, because a process started
from the ssh session dies with it. MxGateway__Telemetry__SiteId=otel-probe-55.
RESOURCE resourceMetrics {"service.name": "mxgateway", "service.version": "0.1.2+54aa61a0", "service.instance.id": "DESKTOP-6JL3KKO:23580", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "DESKTOP-6JL3KKO"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 454, 'spans': 37, 'log records': 21} SEVERITY {'Warning': 3, 'Information': 6, 'Error': 12}
LOG {"severity": "Information", "body": "Gateway alarm monitor disabled (MxGateway:Alarms:Enabled is false)."}
LOG {"severity": "Error", "body": "Health check worker-executable with status Unhealthy completed after 4.6339ms with message 'Worker executable ...", "traceId": "b76b4059dd4f33c88e556ec5ef2e4083"}
The live MxAccessGw service stayed Running with :5130/health/ready = 200 throughout. Its gateway PID
(9148) and worker PIDs (14700,22108) were unchanged before and after. The scratch copy and C:\otel55 were
deleted.
On every app, log records carry traceId when written inside a request, so logs join their spans in the
backend. The Resource is byte-identical across the three signals of each process.
ScadaBridge Central role over OTLP — ScadaBridge#133 (2026-11-12, same day)
Closes the last gap from the two runs above: the Central role had never exported. This run used the real docker rig instead of a throwaway host, because a Central needs the config DB and LDAP.
How it was run
- Build: the rig image predated Telemetry 0.3.0 (its
ZB.MOM.WW.Telemetry.dllwas0.2.0+01c6dc1d). It was rebuilt withdocker/deploy.shfrom a cleanmainat482ac665(which includesa633005f, Telemetry 0.3.0), and all 8 nodes came back healthy. - Override: a temporary compose file (scratch, never committed) set on central-a and central-b only
ScadaBridge__Telemetry__Exporter=Otlp,ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317andOTEL_METRIC_EXPORT_INTERVAL=10000. Both centrals were recreated together (docker compose -f docker/docker-compose.yml -f <override> up -d --force-recreate --no-deps central-a central-b), per the rig's pair-restart rule. Sites, Traefik and every other container were left alone. - Activity (through Traefik
:9000):- 9 inbound calls rejected with 401: no key, or an empty key.
- 3 authorized
POST /api/SimpleAlarmStatus(200{"WasSuccessful":true,…}), made with a temporary method-scoped API key. All three temporary keys were deleted afterwards (api-key listshows 0otel133*). - One
deploy instance --id 1492(rig-probe-1, status Success). - CLI
template list/site listand UI/health/metrics GETs, on top of the health polling and the site heartbeats that already happen.
What arrived (collector, one process per central)
central-a (active) RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0", "service.instance.id": "980946b3915d:6", "site.id": "central", "node.role": "Central", "host.name": "980946b3915d"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 463, 'spans': 321, 'log records': 3305} SEVERITY {'Information': 3046, 'Warning': 259}
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Central on scadabridge-central-a"}
central-b (standby) RESOURCE ... "service.instance.id": "d4d8e49df76f:6", "site.id": "central", "node.role": "Central" ... (identical across the three signals)
COUNTS {'metric points': 421, 'spans': 77, 'log records': 331} SEVERITY {'Information': 311, 'Warning': 20}
service.namespace=ZB.MOM.WW is present on both. Logs carry traceId inside requests, as on the other apps.
Central instruments (meter ZB.MOM.WW.ScadaBridge, last cumulative value on central-a):
scadabridge.deployments.applied 1
scadabridge.inbound_api.requests {method=SimpleAlarmStatus} 3
scadabridge.inbound_api.requests {method=<unauthorized>} 9
scadabridge.central.transport.import_sessions.open 0
+ the shared gauges: site.stream.subscribers.active, site.data_connections.{connected,total},
store_and_forward.queue.depth, site.alarm_cache.aggregators.active (all 0 on a central)
The counts match the activity exactly. The standby (central-b) exports only the observable gauges, because the deploy and inbound singletons run on the active node.
Spans on central-a (standard instrumentation; ScadaBridge has no ActivitySource of its own):
POST /scadabridge.centralcontrol.v1.CentralControlService/{Heartbeat 133, SubmitNotification 94, IngestAuditEvents 24, ReportSiteHealth 10},
POST /management 16, POST /auth/token 16, POST /api/{methodName} 12, GET /api/internal/deployments/{deploymentId}/config 1,
the gRPC client spans SiteCommandService/ExecuteRoute 3 + ExecuteLifecycle 1 (the inbound route to the site
and the deploy), GET /health/{active,ready}, and the Blazor Route / -> …CentralUI…Dashboard.
Allowlist: SiteServiceRegistration.ObservedMeters names ZB.MOM.WW.ScadaBridge (which holds every central and
site instrument) and the LocalDb meter, and the repository creates no other Meter or ActivitySource. No defect.
Restored
Both centrals were recreated together again on the default compose with no override (0 telemetry env vars). Then:
- all 8 nodes
/health/ready200, one active per pair (central-a, site-{a,b,c}-a 200; their partners 503); - central
memberCount 2, unreachableCount 0; - Traefik
/health/active200; - negative control: the new central processes (
a596a23d9a12,1b6fd48c6785) have 0 lines inmetrics/traces/logs.jsonl.
The rig stays on the rebuilt 482ac665 image.
Follow-ups filed (not fixed here)
- scadaproj#56: the Serilog OTLP sink's own
LogsService/Exportcalls are traced as gRPC client spans in all four apps (scadabridge 127, mxgateway 13, historian-gateway 10, otopcua 6 at the collector). The SDK's trace and metric exporters suppress their own instrumentation, andSerilog.Sinks.OpenTelemetrydoes not. This is noise, not a loop (0 log records about it). Fix inZB.MOM.WW.Telemetrywith an instrumentation filter. Fixed in 0.3.1, same day — see below. - ScadaBridge#134: rig images report
service.version = "1.0.0"with no+<sha>, because.dockerignoreexcludes.git/. Fix by passingSourceRevisionIdas a build arg. Wonder deploys, published from a checkout, are unaffected.
OTLP export calls no longer traced — scadaproj#56 (2026-11-12, same day)
Fix: ZB.MOM.WW.Telemetry(.Serilog) 0.3.1 — published
The SDK's OTLP exporters suppress their own instrumentation. Serilog.Sinks.OpenTelemetry does not, so
AddGrpcClientInstrumentation() traced every log batch it sent. AddZbTelemetry now classifies an outgoing call
with ZbOtlpExportCallFilter.IsExportCall: any /opentelemetry.proto.collector.* path on any host, or
OTLP/HTTP /v1/{traces,metrics,logs} on the configured OtlpEndpoint's host:port.
- HttpClient:
FilterHttpRequestMessagereturnsfalsefor an export call. - gRPC client:
GrpcClientTraceInstrumentationOptionshas noFilterin1.15.1-beta.1. TheEnrichWithHttpRequestMessagecallback runs synchronously in the instrumentation's start handler, beforeHttpClient.SendAsync(checked in the contrib source of that tag). For an export call it clearsActivityTraceFlags.RecordedandIsAllDataRequested, so the export processor skips the span and the child HTTP activity is unsampled under the default parent-based sampler. - A processor that drops spans in
OnEndwas rejected. It only works when it is registered before every exporter, and an app could add an exporter first. The instrumentation hooks do not depend on processor order.
Published from scadaproj main 85654d7 (the commit in the nuspec). SHA256, feed download = local pack:
ZB.MOM.WW.Telemetry.0.3.1.nupkg baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3,
ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg 188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931.
Registration lists 0.1.0, 0.2.0, 0.3.0, 0.3.1 for both ids.
Tests (54 + 40, Release):
OtlpExportCallFilterTests: 16 cases. The positives are the three collector services, a collector path on another host, and/v1/*on the endpoint. The negatives are ScadaBridge→site, HG→historian and mxgw paths, REST, acollectorxlook-alike,/v1/logson another host or port, and relative URIs.Otlp_export_calls_are_not_traced_but_the_apps_own_calls_are: a host runs against the loopback OTLP receiver. The positive controls are an app gRPC call (zb.test.v1.AppService/Ping) and an HttpClient call, both traced. The negatives are a real Serilog-sink log batch (the test asserts it arrived), an HttpClientPOST /v1/logsto the endpoint, and a collector path over HttpClient; none is traced. The receiver's own ASP.NET Core server spans land in the same export because the instrumentation listens process-wide. So the assertions key on client-span markers: the protobuf span name for gRPC andurl.fullfor HttpClient.- Mutations: removing the gRPC un-record fails the test ("LogsService/Export call was traced"). Removing the
HttpClient filter fails it (
url.full …/v1/logsfound). MakingIsExportCallalways true fails it too ("positive control: the app's gRPC call was not traced"). It passed 5 of 5 repeated runs.
Apps (pushed to their default branch; no wonder-box deploys)
| App | Commit | Pin | Tests (Debug, against 0.3.1 — project.assets.json confirms it on every project) |
|---|---|---|---|
| OtOpcUa | master f3178c5f |
Directory.Packages.props |
Host.IntegrationTests 325 passed / 2 skipped |
| MxAccessGateway | main 453fe015 |
Server.csproj |
NonWindows solution: 1446 + 5 passed (28 skipped) |
| HistorianGateway | main dffb5e2 |
Server.csproj |
186 + 206 + 1414 passed (76 skipped) |
| ScadaBridge | main 95584e2f |
Directory.Packages.props |
Host.Tests 721 passed |
Collector before/after
The collector-wide count of LogsService/Export spans (zb-otel-collector, all runs to date) was the same before
and after both probe runs: scadabridge 146, mxgateway 13, historian-gateway 10, otopcua 6. The issue reported
127 for scadabridge; the ScadaBridge#133 centrals added more before the rig was restored. Each probe ran as a
throwaway local process on 0.3.1 with Exporter=Otlp, OtlpEndpoint=http://10.100.0.35:4317,
OTEL_METRIC_EXPORT_INTERVAL=10000 and site.id=otel-probe-56, counted from its own start time:
| Run | Log export batches / records arrived | *Service/Export spans |
App client spans still arriving |
|---|---|---|---|
HistorianGateway Development, :15220/:15221, scratch stores, health/metrics GETs + 3 HistorianStatus/Probe |
15 / 168 | 0 | HistoryService/GetInterfaceVersion 14 (GrpcNetClient), historian.Connect 14, server spans |
ScadaBridge Site 95584e2f, own site id, own 1-node mesh localhost:18082, gRPC :18083, metrics :18084, central localhost:18999 unreachable, scratch LocalDb/audit/secrets |
27 / 224 | 0 | CentralControlService/Heartbeat 17, ReconcileSite 2, ReportSiteHealth 2 (GrpcNetClient), server spans |
So the log sink exported throughout, the gRPC client instrumentation stayed on, and no export call was traced. mxgw is proven by the library tests and its suite only; no windev scratch run was made. The docker rigs were not touched.
The wonder boxes (#54) still run their earlier builds. They will emit LogsService/Export spans once a 0.3.0 build
is deployed, unless 0.3.1 is deployed instead.