Files
scadaproj/components/observability/2026-11-12-otlp-end-to-end-evidence.md
T

34 KiB
Raw Blame History

OTLP export, end to end — evidence (2026-11-12, scadaproj#13)

First run of the family's OTLP export path against a real collector. Before this, every app had the <App>:Telemetry:Exporter=Otlp + :OtlpEndpoint wiring but nothing had ever received from it, and OtOpcUa's spans had no exporter anywhere.

Result: OTLP metrics + traces proven for OtOpcUa, HistorianGateway, ScadaBridge (Site role) and MxAccessGateway. One real defect found and fixed (OtOpcUa Galaxy driver telemetry was never exported, lmxopcua#892). No Telemetry library change was needed; 0.2.1 stays unpublished.

Collector

zb-otel-collector on the shared Docker host 10.100.0.35 — otel/opentelemetry-collector-contrib:0.162.0, OTLP gRPC :4317 + HTTP :4318, Prometheus exporter :8889, health :13133; debug + file exporters for proof. Source and runbook: infra/otel-collector/. The four ports were free on the host before deploy, and no other container was touched.

$ curl -s http://10.100.0.35:13133/
{"status":"Server available","upSince":"2026-11-13T01:24:13.274419432Z",...}
$ curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d '{}' http://10.100.0.35:4318/v1/traces
200
windev> Test-NetConnection 10.100.0.35 -Port 4317 -InformationLevel Quiet
True

How each app was run

No shared rig container was recreated or reconfigured. Each app ran as a throwaway process with its own state, Exporter=Otlp, OtlpEndpoint=http://10.100.0.35:4317 (gRPC) and OTEL_METRIC_EXPORT_INTERVAL=10000.

App Build Telemetry pin How
OtOpcUa master aaaa55e2 + the #892 fix (0b90379f) 0.1.0 Local dotnet host on the Mac: fused admin,driver single node in its own Akka mesh (localhost:14053), its own throwaway ConfigDb OtOpcUaOtel13 on the docker-dev SQL (EF-migrated, one ServerCluster/ClusterNode row, dropped afterwards), HTTP :19100, OPC UA :14840. Activity: one POST /api/deployments + health/metrics GETs.
HistorianGateway main b95bae3 0.1.0 Local dotnet host, Development, HTTP :15220 / gRPC :15221, scratch SQLite stores. Activity: status RPCs via grpcurl (the historian itself is unreachable without the VPN, so they fail — which still produces spans) + dashboard/health GETs.
ScadaBridge main 139a43fb 0.2.0 Local dotnet Site host, own site id otel-probe, own 1-node mesh (localhost:18082), scratch LocalDb/audit/secrets, metrics listener :18084, central endpoint deliberately unreachable. The docker rig was in active use (ScadaBridge#46/#132; containers recreated minutes earlier), so it was not touched.
MxAccessGateway deployed windev Server 447070a5 (scratch copy) 0.1.0 Copy of C:\publish\mxaccessgw\Server-20261112-447070a5 run as a separate process on :15120/:15130, Staging, alarms off, scratch auth/secrets stores, Worker:ExecutablePath pointed at a non-existent OtelProbeNoWorker.exe so its OrphanWorkerTerminator (which matches by image name) could not match the live service's x86 workers. Live MxAccessGw stayed Running, :5130/health/ready = 200 throughout, worker PIDs 14700,22108 unchanged before/after; the scratch copy was deleted.

What arrived (collector otelsum.py output, trimmed)

Standard instrumentation scopes that every app sent are listed once at the end.

OtOpcUa — metrics + traces

RESOURCE otopcua {"service.name": "otopcua", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:48096", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS otopcua ZB.MOM.WW.OtOpcUa 3 ['otopcua.deploy.applied', 'otopcua.deploy.apply.duration', 'otopcua.redundancy.service_level_change']
METRICS otopcua ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.deploy.apply')
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.opcua.address_space_rebuild')
SPANS 1 ('otopcua', 'Microsoft.AspNetCore', 'POST /api/deployments')
SPANS 5 ('otopcua', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 3 ('otopcua', 'Microsoft.AspNetCore', 'GET /metrics')
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.deploy.apply", "traceId": "e5a579168f8c4629b25af4d3e4f2cb07", "spanId": "4f12d04ada38ea41", "kind": 1,
 "attributes": [{"key": "otopcua.deployment_id", "value": {"stringValue": "e8b8c010d50144ccb371a91defceb6d9"}}, {"key": "otopcua.node_id", "value": {"stringValue": "localhost:14053"}},
                {"key": "otopcua.revision", "value": {"stringValue": "ed37612b…7bba"}}, {"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.opcua.address_space_rebuild", "traceId": "f6d8c238b2399cce0928de84691a0106", "spanId": "305ec5a9012a799d", "kind": 1,
 "attributes": [{"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}

OtOpcUa's tracing is no longer a no-op once Exporter=Otlp is set: its own spans reach a backend with their attributes. Note that address_space_rebuild is a separate trace from deploy.apply (no parent — the context does not cross the Akka actor hop); the two join on otopcua.correlation_id.

HistorianGateway — metrics + traces

RESOURCE historian-gateway {"service.name": "historian-gateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:49942", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS historian-gateway ZB.MOM.WW.HistorianGateway 10 ['historian_gateway.grpc.request.duration', 'historian_gateway.grpc.requests', 'historian_gateway.historian.connection.up', 'historian_gateway.historian.event_session_pool.lease_waiters', 'historian_gateway.historian.event_session_pool.warm_sessions', 'historian_gateway.historian.session_pool.lease_waiters', 'historian_gateway.historian.session_pool.warm_sessions', 'historian_gateway.redundancy.members.healthy', 'historian_gateway.session_pool.lease.wait.duration', 'historian_gateway.store_forward.queue.depth']
SPANS 8 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.Connect')
SPANS 2 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.pool.lease')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/Probe')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/GetConnectionStatus')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'galaxy_repository.v1.GalaxyRepository/GetLastDeployTime')
SPANS 8 ('historian-gateway', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'HistoryService/GetInterfaceVersion')
SPANS 4 ('historian-gateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.HistorianGateway.Server.Dashboard.Components.Pages.Home')

historian.Connect spans carry server.address/server.port and error.type=…HistorianTransportException (no VPN); the ones a request triggered are parented under its gRPC request span, the pool warm-up ones are roots. Prometheus exporter, same data:

historian_gateway_grpc_requests_total{job="ZB.MOM.WW/historian-gateway",method="Probe",outcome="ok",service="HistorianStatus",service_name="historian-gateway",...} 1

ScadaBridge (Site) — metrics + traces

RESOURCE scadabridge {"service.name": "scadabridge", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:50923", "host.name": "Josephs-MacBook-Pro", "site.id": "otel-probe", "node.role": "Site", "_sdk": "1.15.3"}
METRICS scadabridge ZB.MOM.WW.ScadaBridge 6 ['scadabridge.central.transport.import_sessions.open', 'scadabridge.site.alarm_cache.aggregators.active', 'scadabridge.site.data_connections.connected', 'scadabridge.site.data_connections.total', 'scadabridge.site.stream.subscribers.active', 'scadabridge.store_and_forward.queue.depth']
METRICS scadabridge ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 12 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/Heartbeat')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReportSiteHealth')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReconcileSite')
SPANS 4 ('scadabridge', 'Microsoft.AspNetCore', 'GET /health/ready')

The only app that sends site.id / node.role. ScadaBridge has no ActivitySource of its own, so its spans are standard instrumentation only. Central role not run → ScadaBridge#133.

MxAccessGateway — metrics + traces

RESOURCE mxgateway {"service.name": "mxgateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "DESKTOP-6JL3KKO:8720", "host.name": "DESKTOP-6JL3KKO", "_sdk": "1.15.3"}
METRICS mxgateway ZB.MOM.WW.MxGateway 5 ['mxgateway.alarms.provider_mode', 'mxgateway.events.grpc_stream_queue.depth', 'mxgateway.events.worker_queue.depth', 'mxgateway.sessions.open', 'mxgateway.workers.running']
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /metrics')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.MxGateway.Server.Dashboard.Components.Pages.DashboardHome')

Only the observable gauges appear: the counters/histograms (commands.*, workers.startup.duration, …) record nothing without a worker session, and the x86 worker was deliberately not run. The gateway has no ActivitySource of its own (spans are standard instrumentation only). Its gRPC port is not reachable from off-box (windev firewall), so the traffic was generated on-box.

Standard instrumentation (all four)

System.Runtime (19 dotnet.*), OpenTelemetry.Instrumentation.Process (5 process.*), System.Net.Http (4 http.client.*), Microsoft.AspNetCore.Hosting (http.server.request.duration, http.server.active_requests), Microsoft.AspNetCore.Server.Kestrel, .Routing, .MemoryPool; plus .Authentication/.Authorization/.Components on the apps with a Blazor dashboard.

Defect found and fixed

lmxopcua#892 — the Galaxy driver's telemetry was never exported. ZbTelemetryOptions.Meters and .ActivitySources are strict allowlists. OtOpcUa named only ZB.MOM.WW.OtOpcUa and the LocalDb meter, while the in-process Galaxy driver opens its spans (galaxy.subscribe_bulk, galaxy.unsubscribe_bulk, galaxy.stream_events, galaxy.write, galaxy.get_hierarchy) on ActivitySource ZB.MOM.WW.OtOpcUa.Driver.Galaxy and records the event pump's galaxy.events.* counters on a Meter of the same name. All of it was silently dropped from /metrics and OTLP.

Fix: OtOpcUa master db86b996 (commit 0b90379f) — a public GalaxyDriverTelemetryNames in the driver; ObservabilityExtensions.ObservedMeters / ObservedActivitySources name it. ObservabilityAllowlistTests assert on the listener side (span sampled, instrument enabled) with an unregistered-name negative control. Dropping the Galaxy entries fails 3 of 8 tests, so the tests can catch the regression. Galaxy driver suite 584/584. Not deployed. A live galaxy.* span was not driven: that would need a Galaxy driver deployment against a gateway.

The allowlists of the other apps were checked against every new Meter( / new ActivitySource( in their source and in the hosted libraries. ScadaBridge (ZB.MOM.WW.ScadaBridge, LocalDb), HistorianGateway (ZB.MOM.WW.HistorianGateway meter + source) and MxAccessGateway (ZB.MOM.WW.MxGateway) are complete. Of the hosted libraries, only LocalDb defines a meter.

Checked, no defect

  • Protocol/endpoint: every app uses the library's AddOtlpExporter default (gRPC). http://host:4317 works for all four. No checked-in config carries a scheme-less endpoint.
  • Prometheus stays up under Otlp: /metrics answered 200 on every app with Exporter=Otlp (http.response.status_code=200 on each app's exported GET /metrics span) (the overlay is additive, as documented).
  • Resource: service.name, service.namespace=ZB.MOM.WW, service.instance.id, host.name present on all four.

Gaps filed (not fixed here)

  • scadaproj#55: service.version is absent on all four apps; site.id/node.role are absent on OtOpcUa, MxAccessGateway and HistorianGateway. No app exports logs over OTLP, because every AddZbSerilog call passes only ServiceName, so the Serilog OTLP sink never activates (logs.jsonl stayed empty).
  • ScadaBridge#133: OTLP from the Central role is not yet proven.
  • scadaproj#54: collector on the wonder tiers. DONE 2026-11-12: one loopback-only ZbOtelCollector per box on vd03/vq03/vq04, every app with an exporter setting switched to OTLP, and data proven at each collector. On vd03 and QA this includes the ScadaBridge Central role (node.role=Central). Record: infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md.

Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)

No finding needed a library change. The fixes and gaps above are all on the app side. 0.2.1's scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays built-but-unpublished until a pin bump wants it.


Resource identity + logs over OTLP — scadaproj#55 (2026-11-12, same day)

Closes the three #55 gaps found above: no service.version anywhere, no site.id/node.role on OtOpcUa, MxAccessGateway and HistorianGateway, and no logs over OTLP from any app.

Root cause of "no logs"

ZbSerilogConfig.ApplyOpenTelemetryExport adds the WriteTo.OpenTelemetry sink only when the options passed to AddZbSerilog say Exporter=Otlp. Every app filled those options with the service name alone and set the exporter only on AddZbTelemetry, a separate options object the Serilog package never saw. The sink condition could therefore never be true.

Library: ZB.MOM.WW.Telemetry(.Serilog) 0.3.0 — published

scadaproj main cfc7168 + 2011c8d. Feed SHA256 (feed download = local pack):

Package SHA256
ZB.MOM.WW.Telemetry.0.3.0.nupkg a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2
ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg 04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c
  • service.version: ServiceVersion = null (the default) now means the entry assembly's AssemblyInformationalVersion (the git SHA the apps stamp), falling back to its assembly version. Set a value to override it; "" omits the attribute.

  • ResourceAttributes: extra Resource attributes on all three signals. The first-class keys (service.*, host.name, site.id, node.role) are reserved and rejected at startup.

  • ApplyConfiguration(section): reads Exporter, OtlpEndpoint, ServiceVersion, SiteId, NodeRole, ExportLogs, ResourceAttributes:<key> from an app's own <App>:Telemetry section. Absent keys keep the code defaults, so the four hand-written exporter parsers are gone.

  • Logs follow AddZbTelemetry: AddZbTelemetry registers its options in DI. AddZbSerilog resolves them inside its AddSerilog callback (so call order does not matter), and when its own options do not select OTLP, the OTLP log sink uses that registration's endpoint and Resource. ExportLogs = false keeps logs local. Levels and the configuration-driven sinks are unchanged.

  • 0.2.1 shipped in the same release. It holds the P3 review fixes: redaction after ConfigureLogger, dictionary-key collisions, and the stricter endpoint check (collector:4317 without a scheme now fails at startup when Exporter=Otlp). It is a minor bump because it adds API and changes behaviour. No checked-in config in any of the four repos sets an OTLP endpoint. The check is a no-op unless Exporter=Otlp, and no wonder tier had a collector at the time (#54). Since 2026-11-12 the wonder boxes set OtlpEndpoint=http://127.0.0.1:4317, scheme included, so they pass the check when 0.3.0 is deployed.

  • Tests: 42 → 77 in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on Kestrel that proves a log record arrives with service.version/site.id/node.role/an extra attribute, an ExportLogs=false case with a metrics positive control, and a Prometheus case.

    Mutation checks, each caught by the named tests:

    • Drop the DI lookup → the e2e log test fails.
    • Drop the version default → the pipeline-Resource and e2e tests fail.
    • Drop ResourceAttributes → 2 attribute tests and the e2e test fail.
    • Ignore ExportLogs → the decision test and the e2e test fail.
    • Disable the reserved-key check → 7 theory cases fail.
    • Drop ApplyConfiguration's attribute or SiteId read → its test fails.

Apps (all pushed to their default branch; no wonder-box deploys)

App Commit Pin Identity wiring
OtOpcUa master 412da27c 0.1.0 → 0.3.0 site.id = the application Cluster id (the cluster-<ClusterId> role suffix; omitted on a legacy node); node.role = the plain roles, sorted (admin,driver); Program passes its EffectiveRoles.
MxAccessGateway main d9d9ff48 0.1.0 → 0.3.0 (csproj) node.role=standalone (one gateway per box); site.id from MxGateway:Telemetry:SiteId (there is no node identity in its config, so it is omitted unless set).
HistorianGateway main ae1dbcc 0.1.0 → 0.3.0 (csproj) Same as mxgw, under the root-level Telemetry: section.
ScadaBridge main a633005f 0.2.0 → 0.3.0 Unchanged site.id/node.role. The new ScadaBridgeTelemetryConfig fills both AddZbTelemetry and the pre-host AddZbSerilog logger. The logger is built on its own container with no AddZbTelemetry to follow, so the exporter is passed to it explicitly.

Tests: OtOpcUa Host.Tests 288/288 + Host.IntegrationTests 325/327 (2 skipped); mxgw MxGateway.Tests 1446/1452 (6 skipped); HistorianGateway 1414 + 206 + 186 passed; ScadaBridge Host.Tests 699/699. Each app gained identity tests (OtOpcUa ObservabilityResourceIdentityTests, HG GatewayTracingTests, ScadaBridge ScadaBridgeTelemetryConfigTests).

What arrived (collector, otelidentity.py, one process each)

All runs: Exporter=Otlp, OtlpEndpoint=http://10.100.0.35:4317, OTEL_METRIC_EXPORT_INTERVAL=10000.

OtOpcUa — local Release host, fused admin,driver,cluster-OTEL55 single node. Its own Akka mesh on localhost:14053. Throwaway ConfigDb OtOpcUaOtel55 on the docker-dev SQL: EF-migrated, one ServerCluster OTEL55 + one ClusterNode, gRPC mesh/telemetry keys of its own. The DB was dropped afterwards. Activity: one POST /api/deployments (202) plus health/metrics GETs.

RESOURCE resourceMetrics {"service.name": "otopcua", "service.version": "1.0.0+412da27c5fb7e440e4a75d4f8e82e0e5b9832411", "service.instance.id": "Josephs-MacBook-Pro:67675", "site.id": "OTEL55", "node.role": "admin,driver", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans   {... identical ...}
RESOURCE resourceLogs    {... identical ...}
COUNTS {'metric points': 427, 'spans': 29, 'log records': 73} SEVERITY {'Information': 61, 'Warning': 12}
LOG {"severity": "Information", "body": "OtOpcUa.Host starting with roles=[admin,driver,cluster-OTEL55] (admin=True, driver=True) source=Configured"}
LOG {"severity": "Information", "body": "HTTP POST /telemetry.v1.TelemetryStreamService/Subscribe responded 200 in 29942.6698 ms", "traceId": "f5f90813adac357cdce4c165ab9e1468"}

The otopcua.deploy.apply and otopcua.opcua.address_space_rebuild spans arrived again.

HistorianGateway — local Release host, Development, HTTP :15220 / gRPC :15221, scratch SQLite stores, Telemetry__SiteId=otel-probe-55. Activity: HistorianStatus/Probe + health GETs.

RESOURCE resourceMetrics {"service.name": "historian-gateway", "service.version": "1.0.0+ae1dbccc201aeebbe030c407870879a7985002db", "service.instance.id": "Josephs-MacBook-Pro:72641", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans   {... identical ...}
RESOURCE resourceLogs    {... identical ...}
COUNTS {'metric points': 184, 'spans': 12, 'log records': 29} SEVERITY {'Information': 25, 'Warning': 2, 'Error': 2}
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:15220/health/ready - null null", "traceId": "8eca56a689d61f710bdbdaad32879a6b"}

ScadaBridge (Site) — local Release Site host, own site id otel-probe-55, own 1-node mesh (localhost:18082), scratch LocalDb/audit/secrets, metrics listener :18084, central endpoint deliberately unreachable.

RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0+a633005f4f826e662d09fa47359bced17ccd6acf", "service.instance.id": "Josephs-MacBook-Pro:69217", "site.id": "otel-probe-55", "node.role": "Site", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans   {... identical ...}
RESOURCE resourceLogs    {... identical ...}
COUNTS {'metric points': 290, 'spans': 30, 'log records': 78} SEVERITY {'Information': 66, 'Warning': 10, 'Error': 2}
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Site on localhost"}
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:18084/health/ready - null null", "traceId": "0e976e6c0b8d7d3d0aaf85a99f5fbe21"}

The first line, Starting ScadaBridge host …, is written to the static logger before any web host exists, so the explicit exporter path on the pre-host logger works. Central not run: the rig was in use by the ScadaBridge#132 failover drills (sb132-p / sb132-probe running, centrals recreated 3–6 minutes earlier), so ScadaBridge#133 stays open.

MxAccessGateway — Release main 54aa61a0, published framework-dependent win-x64 from the Mac. (It was pushed as d9d9ff48 after rebasing over one docs-only commit.) Run as a scratch copy at C:\otel55\mxgw on windev, on :15120/:15130, under Staging + DisableLogin, with alarms off, scratch auth/secrets/snapshot stores and a throwaway pepper. Worker:ExecutablePath pointed at the non-existent C:\otel55\OtelProbeNoWorker.exe. It was launched through Win32_Process.Create, because a process started from the ssh session dies with it. MxGateway__Telemetry__SiteId=otel-probe-55.

RESOURCE resourceMetrics {"service.name": "mxgateway", "service.version": "0.1.2+54aa61a0", "service.instance.id": "DESKTOP-6JL3KKO:23580", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "DESKTOP-6JL3KKO"}
RESOURCE resourceSpans   {... identical ...}
RESOURCE resourceLogs    {... identical ...}
COUNTS {'metric points': 454, 'spans': 37, 'log records': 21} SEVERITY {'Warning': 3, 'Information': 6, 'Error': 12}
LOG {"severity": "Information", "body": "Gateway alarm monitor disabled (MxGateway:Alarms:Enabled is false)."}
LOG {"severity": "Error", "body": "Health check worker-executable with status Unhealthy completed after 4.6339ms with message 'Worker executable ...", "traceId": "b76b4059dd4f33c88e556ec5ef2e4083"}

The live MxAccessGw service stayed Running with :5130/health/ready = 200 throughout. Its gateway PID (9148) and worker PIDs (14700,22108) were unchanged before and after. The scratch copy and C:\otel55 were deleted.

On every app, log records carry traceId when written inside a request, so logs join their spans in the backend. The Resource is byte-identical across the three signals of each process.


ScadaBridge Central role over OTLP — ScadaBridge#133 (2026-11-12, same day)

Closes the last gap from the two runs above: the Central role had never exported. This run used the real docker rig instead of a throwaway host, because a Central needs the config DB and LDAP.

How it was run

  • Build: the rig image predated Telemetry 0.3.0 (its ZB.MOM.WW.Telemetry.dll was 0.2.0+01c6dc1d). It was rebuilt with docker/deploy.sh from a clean main at 482ac665 (which includes a633005f, Telemetry 0.3.0), and all 8 nodes came back healthy.
  • Override: a temporary compose file (scratch, never committed) set on central-a and central-b only ScadaBridge__Telemetry__Exporter=Otlp, ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317 and OTEL_METRIC_EXPORT_INTERVAL=10000. Both centrals were recreated together (docker compose -f docker/docker-compose.yml -f <override> up -d --force-recreate --no-deps central-a central-b), per the rig's pair-restart rule. Sites, Traefik and every other container were left alone.
  • Activity (through Traefik :9000):
    • 9 inbound calls rejected with 401: no key, or an empty key.
    • 3 authorized POST /api/SimpleAlarmStatus (200 {"WasSuccessful":true,…}), made with a temporary method-scoped API key. All three temporary keys were deleted afterwards (api-key list shows 0 otel133*).
    • One deploy instance --id 1492 (rig-probe-1, status Success).
    • CLI template list / site list and UI/health/metrics GETs, on top of the health polling and the site heartbeats that already happen.

What arrived (collector, one process per central)

central-a (active)  RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0", "service.instance.id": "980946b3915d:6", "site.id": "central", "node.role": "Central", "host.name": "980946b3915d"}
                    RESOURCE resourceSpans   {... identical ...}
                    RESOURCE resourceLogs    {... identical ...}
                    COUNTS {'metric points': 463, 'spans': 321, 'log records': 3305} SEVERITY {'Information': 3046, 'Warning': 259}
                    LOG {"severity": "Information", "body": "Starting ScadaBridge host as Central on scadabridge-central-a"}
central-b (standby) RESOURCE ... "service.instance.id": "d4d8e49df76f:6", "site.id": "central", "node.role": "Central" ... (identical across the three signals)
                    COUNTS {'metric points': 421, 'spans': 77, 'log records': 331} SEVERITY {'Information': 311, 'Warning': 20}

service.namespace=ZB.MOM.WW is present on both. Logs carry traceId inside requests, as on the other apps.

Central instruments (meter ZB.MOM.WW.ScadaBridge, last cumulative value on central-a):

scadabridge.deployments.applied                      1
scadabridge.inbound_api.requests {method=SimpleAlarmStatus}  3
scadabridge.inbound_api.requests {method=<unauthorized>}     9
scadabridge.central.transport.import_sessions.open   0
+ the shared gauges: site.stream.subscribers.active, site.data_connections.{connected,total},
  store_and_forward.queue.depth, site.alarm_cache.aggregators.active (all 0 on a central)

The counts match the activity exactly. The standby (central-b) exports only the observable gauges, because the deploy and inbound singletons run on the active node.

Spans on central-a (standard instrumentation; ScadaBridge has no ActivitySource of its own): POST /scadabridge.centralcontrol.v1.CentralControlService/{Heartbeat 133, SubmitNotification 94, IngestAuditEvents 24, ReportSiteHealth 10}, POST /management 16, POST /auth/token 16, POST /api/{methodName} 12, GET /api/internal/deployments/{deploymentId}/config 1, the gRPC client spans SiteCommandService/ExecuteRoute 3 + ExecuteLifecycle 1 (the inbound route to the site and the deploy), GET /health/{active,ready}, and the Blazor Route / -> …CentralUI…Dashboard.

Allowlist: SiteServiceRegistration.ObservedMeters names ZB.MOM.WW.ScadaBridge (which holds every central and site instrument) and the LocalDb meter, and the repository creates no other Meter or ActivitySource. No defect.

Restored

Both centrals were recreated together again on the default compose with no override (0 telemetry env vars). Then:

  • all 8 nodes /health/ready 200, one active per pair (central-a, site-{a,b,c}-a 200; their partners 503);
  • central memberCount 2, unreachableCount 0;
  • Traefik /health/active 200;
  • negative control: the new central processes (a596a23d9a12, 1b6fd48c6785) have 0 lines in metrics/traces/logs.jsonl.

The rig stays on the rebuilt 482ac665 image.

Follow-ups filed (not fixed here)

  • scadaproj#56: the Serilog OTLP sink's own LogsService/Export calls are traced as gRPC client spans in all four apps (scadabridge 127, mxgateway 13, historian-gateway 10, otopcua 6 at the collector). The SDK's trace and metric exporters suppress their own instrumentation, and Serilog.Sinks.OpenTelemetry does not. This is noise, not a loop (0 log records about it). Fix in ZB.MOM.WW.Telemetry with an instrumentation filter. Fixed in 0.3.1, same day — see below.
  • ScadaBridge#134: rig images report service.version = "1.0.0" with no +<sha>, because .dockerignore excludes .git/. Fix by passing SourceRevisionId as a build arg. Wonder deploys, published from a checkout, are unaffected.

OTLP export calls no longer traced — scadaproj#56 (2026-11-12, same day)

Fix: ZB.MOM.WW.Telemetry(.Serilog) 0.3.1 — published

The SDK's OTLP exporters suppress their own instrumentation. Serilog.Sinks.OpenTelemetry does not, so AddGrpcClientInstrumentation() traced every log batch it sent. AddZbTelemetry now classifies an outgoing call with ZbOtlpExportCallFilter.IsExportCall: any /opentelemetry.proto.collector.* path on any host, or OTLP/HTTP /v1/{traces,metrics,logs} on the configured OtlpEndpoint's host:port.

  • HttpClient: FilterHttpRequestMessage returns false for an export call.
  • gRPC client: GrpcClientTraceInstrumentationOptions has no Filter in 1.15.1-beta.1. The EnrichWithHttpRequestMessage callback runs synchronously in the instrumentation's start handler, before HttpClient.SendAsync (checked in the contrib source of that tag). For an export call it clears ActivityTraceFlags.Recorded and IsAllDataRequested, so the export processor skips the span and the child HTTP activity is unsampled under the default parent-based sampler.
  • A processor that drops spans in OnEnd was rejected. It only works when it is registered before every exporter, and an app could add an exporter first. The instrumentation hooks do not depend on processor order.

Published from scadaproj main 85654d7 (the commit in the nuspec). SHA256, feed download = local pack: ZB.MOM.WW.Telemetry.0.3.1.nupkg baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3, ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg 188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931. Registration lists 0.1.0, 0.2.0, 0.3.0, 0.3.1 for both ids.

Tests (54 + 40, Release):

  • OtlpExportCallFilterTests: 16 cases. The positives are the three collector services, a collector path on another host, and /v1/* on the endpoint. The negatives are ScadaBridge→site, HG→historian and mxgw paths, REST, a collectorx look-alike, /v1/logs on another host or port, and relative URIs.
  • Otlp_export_calls_are_not_traced_but_the_apps_own_calls_are: a host runs against the loopback OTLP receiver. The positive controls are an app gRPC call (zb.test.v1.AppService/Ping) and an HttpClient call, both traced. The negatives are a real Serilog-sink log batch (the test asserts it arrived), an HttpClient POST /v1/logs to the endpoint, and a collector path over HttpClient; none is traced. The receiver's own ASP.NET Core server spans land in the same export because the instrumentation listens process-wide. So the assertions key on client-span markers: the protobuf span name for gRPC and url.full for HttpClient.
  • Mutations: removing the gRPC un-record fails the test ("LogsService/Export call was traced"). Removing the HttpClient filter fails it (url.full …/v1/logs found). Making IsExportCall always true fails it too ("positive control: the app's gRPC call was not traced"). It passed 5 of 5 repeated runs.

Apps (pushed to their default branch; no wonder-box deploys)

App Commit Pin Tests (Debug, against 0.3.1 — project.assets.json confirms it on every project)
OtOpcUa master f3178c5f Directory.Packages.props Host.IntegrationTests 325 passed / 2 skipped
MxAccessGateway main 453fe015 Server.csproj NonWindows solution: 1446 + 5 passed (28 skipped)
HistorianGateway main dffb5e2 Server.csproj 186 + 206 + 1414 passed (76 skipped)
ScadaBridge main 95584e2f Directory.Packages.props Host.Tests 721 passed

Collector before/after

The collector-wide count of LogsService/Export spans (zb-otel-collector, all runs to date) was the same before and after both probe runs: scadabridge 146, mxgateway 13, historian-gateway 10, otopcua 6. The issue reported 127 for scadabridge; the ScadaBridge#133 centrals added more before the rig was restored. Each probe ran as a throwaway local process on 0.3.1 with Exporter=Otlp, OtlpEndpoint=http://10.100.0.35:4317, OTEL_METRIC_EXPORT_INTERVAL=10000 and site.id=otel-probe-56, counted from its own start time:

Run Log export batches / records arrived *Service/Export spans App client spans still arriving
HistorianGateway Development, :15220/:15221, scratch stores, health/metrics GETs + 3 HistorianStatus/Probe 15 / 168 0 HistoryService/GetInterfaceVersion 14 (GrpcNetClient), historian.Connect 14, server spans
ScadaBridge Site 95584e2f, own site id, own 1-node mesh localhost:18082, gRPC :18083, metrics :18084, central localhost:18999 unreachable, scratch LocalDb/audit/secrets 27 / 224 0 CentralControlService/Heartbeat 17, ReconcileSite 2, ReportSiteHealth 2 (GrpcNetClient), server spans

So the log sink exported throughout, the gRPC client instrumentation stayed on, and no export call was traced. mxgw is proven by the library tests and its suite only; no windev scratch run was made. The docker rigs were not touched.

The wonder boxes (#54) still run their earlier builds. They will emit LogsService/Export spans once a 0.3.0 build is deployed, unless 0.3.1 is deployed instead.