Files
scadaproj/components/observability/2026-11-12-otlp-end-to-end-evidence.md
T

460 lines
34 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# OTLP export, end to end — evidence (2026-11-12, scadaproj#13)
First run of the family's OTLP export path against a real collector. Before this, every app had
the `<App>:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` wiring but nothing had ever received from
it, and OtOpcUa's spans had no exporter anywhere.
**Result:** OTLP **metrics + traces** proven for **OtOpcUa, HistorianGateway, ScadaBridge (Site
role) and MxAccessGateway**. One real defect found and fixed (OtOpcUa Galaxy driver telemetry was
never exported, lmxopcua#892). No Telemetry library change was needed; `0.2.1` stays unpublished.
## Collector
`zb-otel-collector` on the shared Docker host `10.100.0.35` — `otel/opentelemetry-collector-contrib:0.162.0`,
OTLP gRPC `:4317` + HTTP `:4318`, Prometheus exporter `:8889`, health `:13133`; `debug` + `file`
exporters for proof. Source and runbook: [`infra/otel-collector/`](../../infra/otel-collector/). The
four ports were free on the host before deploy, and no other container was touched.
```
$ curl -s http://10.100.0.35:13133/
{"status":"Server available","upSince":"2026-11-13T01:24:13.274419432Z",...}
$ curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d '{}' http://10.100.0.35:4318/v1/traces
200
windev> Test-NetConnection 10.100.0.35 -Port 4317 -InformationLevel Quiet
True
```
## How each app was run
No shared rig container was recreated or reconfigured. Each app ran as a throwaway process with
its own state, `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317` (gRPC) and
`OTEL_METRIC_EXPORT_INTERVAL=10000`.
| App | Build | Telemetry pin | How |
|---|---|---|---|
| OtOpcUa | `master` `aaaa55e2` + the #892 fix (`0b90379f`) | 0.1.0 | Local `dotnet` host on the Mac: fused `admin,driver` single node in its own Akka mesh (`localhost:14053`), its own throwaway ConfigDb `OtOpcUaOtel13` on the docker-dev SQL (EF-migrated, one `ServerCluster`/`ClusterNode` row, dropped afterwards), HTTP `:19100`, OPC UA `:14840`. Activity: one `POST /api/deployments` + health/metrics GETs. |
| HistorianGateway | `main` `b95bae3` | 0.1.0 | Local `dotnet` host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores. Activity: status RPCs via `grpcurl` (the historian itself is unreachable without the VPN, so they fail — which still produces spans) + dashboard/health GETs. |
| ScadaBridge | `main` `139a43fb` | 0.2.0 | Local `dotnet` **Site** host, own site id `otel-probe`, own 1-node mesh (`localhost:18082`), scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable. The docker rig was in active use (ScadaBridge#46/#132; containers recreated minutes earlier), so it was not touched. |
| MxAccessGateway | deployed windev Server `447070a5` (scratch copy) | 0.1.0 | Copy of `C:\publish\mxaccessgw\Server-20261112-447070a5` run as a separate process on `:15120`/`:15130`, `Staging`, alarms off, scratch auth/secrets stores, `Worker:ExecutablePath` pointed at a non-existent `OtelProbeNoWorker.exe` so its `OrphanWorkerTerminator` (which matches by image name) could not match the live service's x86 workers. Live `MxAccessGw` stayed `Running`, `:5130/health/ready` = 200 throughout, worker PIDs `14700,22108` unchanged before/after; the scratch copy was deleted. |
## What arrived (collector `otelsum.py` output, trimmed)
Standard instrumentation scopes that every app sent are listed once at the end.
### OtOpcUa — metrics + traces
```
RESOURCE otopcua {"service.name": "otopcua", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:48096", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS otopcua ZB.MOM.WW.OtOpcUa 3 ['otopcua.deploy.applied', 'otopcua.deploy.apply.duration', 'otopcua.redundancy.service_level_change']
METRICS otopcua ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.deploy.apply')
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.opcua.address_space_rebuild')
SPANS 1 ('otopcua', 'Microsoft.AspNetCore', 'POST /api/deployments')
SPANS 5 ('otopcua', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 3 ('otopcua', 'Microsoft.AspNetCore', 'GET /metrics')
```
```json
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.deploy.apply", "traceId": "e5a579168f8c4629b25af4d3e4f2cb07", "spanId": "4f12d04ada38ea41", "kind": 1,
"attributes": [{"key": "otopcua.deployment_id", "value": {"stringValue": "e8b8c010d50144ccb371a91defceb6d9"}}, {"key": "otopcua.node_id", "value": {"stringValue": "localhost:14053"}},
{"key": "otopcua.revision", "value": {"stringValue": "ed37612b…7bba"}}, {"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.opcua.address_space_rebuild", "traceId": "f6d8c238b2399cce0928de84691a0106", "spanId": "305ec5a9012a799d", "kind": 1,
"attributes": [{"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
```
OtOpcUa's tracing is no longer a no-op once `Exporter=Otlp` is set: its own spans reach a backend
with their attributes. Note that `address_space_rebuild` is a separate trace from `deploy.apply`
(no parent — the context does not cross the Akka actor hop); the two join on
`otopcua.correlation_id`.
### HistorianGateway — metrics + traces
```
RESOURCE historian-gateway {"service.name": "historian-gateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:49942", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS historian-gateway ZB.MOM.WW.HistorianGateway 10 ['historian_gateway.grpc.request.duration', 'historian_gateway.grpc.requests', 'historian_gateway.historian.connection.up', 'historian_gateway.historian.event_session_pool.lease_waiters', 'historian_gateway.historian.event_session_pool.warm_sessions', 'historian_gateway.historian.session_pool.lease_waiters', 'historian_gateway.historian.session_pool.warm_sessions', 'historian_gateway.redundancy.members.healthy', 'historian_gateway.session_pool.lease.wait.duration', 'historian_gateway.store_forward.queue.depth']
SPANS 8 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.Connect')
SPANS 2 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.pool.lease')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/Probe')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/GetConnectionStatus')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'galaxy_repository.v1.GalaxyRepository/GetLastDeployTime')
SPANS 8 ('historian-gateway', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'HistoryService/GetInterfaceVersion')
SPANS 4 ('historian-gateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.HistorianGateway.Server.Dashboard.Components.Pages.Home')
```
`historian.Connect` spans carry `server.address`/`server.port` and `error.type=…HistorianTransportException`
(no VPN); the ones a request triggered are parented under its gRPC request span, the pool warm-up ones are roots. Prometheus exporter, same data:
```
historian_gateway_grpc_requests_total{job="ZB.MOM.WW/historian-gateway",method="Probe",outcome="ok",service="HistorianStatus",service_name="historian-gateway",...} 1
```
### ScadaBridge (Site) — metrics + traces
```
RESOURCE scadabridge {"service.name": "scadabridge", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:50923", "host.name": "Josephs-MacBook-Pro", "site.id": "otel-probe", "node.role": "Site", "_sdk": "1.15.3"}
METRICS scadabridge ZB.MOM.WW.ScadaBridge 6 ['scadabridge.central.transport.import_sessions.open', 'scadabridge.site.alarm_cache.aggregators.active', 'scadabridge.site.data_connections.connected', 'scadabridge.site.data_connections.total', 'scadabridge.site.stream.subscribers.active', 'scadabridge.store_and_forward.queue.depth']
METRICS scadabridge ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 12 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/Heartbeat')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReportSiteHealth')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReconcileSite')
SPANS 4 ('scadabridge', 'Microsoft.AspNetCore', 'GET /health/ready')
```
The only app that sends `site.id` / `node.role`. ScadaBridge has no `ActivitySource` of its own,
so its spans are standard instrumentation only. **Central role not run** → ScadaBridge#133.
### MxAccessGateway — metrics + traces
```
RESOURCE mxgateway {"service.name": "mxgateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "DESKTOP-6JL3KKO:8720", "host.name": "DESKTOP-6JL3KKO", "_sdk": "1.15.3"}
METRICS mxgateway ZB.MOM.WW.MxGateway 5 ['mxgateway.alarms.provider_mode', 'mxgateway.events.grpc_stream_queue.depth', 'mxgateway.events.worker_queue.depth', 'mxgateway.sessions.open', 'mxgateway.workers.running']
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /metrics')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.MxGateway.Server.Dashboard.Components.Pages.DashboardHome')
```
Only the observable gauges appear: the counters/histograms (`commands.*`, `workers.startup.duration`, …)
record nothing without a worker session, and the x86 worker was deliberately not run. The gateway
has no `ActivitySource` of its own (spans are standard instrumentation only). Its gRPC port is not
reachable from off-box (windev firewall), so the traffic was generated on-box.
### Standard instrumentation (all four)
`System.Runtime` (19 `dotnet.*`), `OpenTelemetry.Instrumentation.Process` (5 `process.*`),
`System.Net.Http` (4 `http.client.*`), `Microsoft.AspNetCore.Hosting` (`http.server.request.duration`,
`http.server.active_requests`), `Microsoft.AspNetCore.Server.Kestrel`, `.Routing`, `.MemoryPool`;
plus `.Authentication`/`.Authorization`/`.Components` on the apps with a Blazor dashboard.
## Defect found and fixed
**lmxopcua#892 — the Galaxy driver's telemetry was never exported.** `ZbTelemetryOptions.Meters`
and `.ActivitySources` are strict allowlists. OtOpcUa named only `ZB.MOM.WW.OtOpcUa` and the
LocalDb meter, while the in-process Galaxy driver opens its spans (`galaxy.subscribe_bulk`,
`galaxy.unsubscribe_bulk`, `galaxy.stream_events`, `galaxy.write`, `galaxy.get_hierarchy`) on
ActivitySource `ZB.MOM.WW.OtOpcUa.Driver.Galaxy` and records the event pump's `galaxy.events.*`
counters on a Meter of the same name. All of it was silently dropped from `/metrics` and OTLP.
Fix: OtOpcUa `master` `db86b996` (commit `0b90379f`) — a public `GalaxyDriverTelemetryNames` in
the driver; `ObservabilityExtensions.ObservedMeters` / `ObservedActivitySources` name it.
`ObservabilityAllowlistTests` assert on the listener side (span sampled, instrument enabled) with an
unregistered-name negative control. Dropping the Galaxy entries fails 3 of 8 tests, so the tests can
catch the regression. Galaxy driver suite 584/584. Not deployed. A live `galaxy.*` span was not
driven: that would need a Galaxy driver deployment against a gateway.
The allowlists of the other apps were checked against every `new Meter(` / `new ActivitySource(`
in their source and in the hosted libraries. ScadaBridge (`ZB.MOM.WW.ScadaBridge`, LocalDb),
HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGateway
(`ZB.MOM.WW.MxGateway`) are complete. Of the hosted libraries, only LocalDb defines a meter.
## Checked, no defect
- **Protocol/endpoint:** every app uses the library's `AddOtlpExporter` default (gRPC). `http://host:4317`
works for all four. No checked-in config carries a scheme-less endpoint.
- **Prometheus stays up under Otlp:** `/metrics` answered 200 on every app with `Exporter=Otlp` (`http.response.status_code=200` on each app's exported `GET /metrics` span)
(the overlay is additive, as documented).
- **Resource:** `service.name`, `service.namespace=ZB.MOM.WW`, `service.instance.id`, `host.name`
present on all four.
## Gaps filed (not fixed here)
- **scadaproj#55:** `service.version` is absent on all four apps; `site.id`/`node.role` are absent
on OtOpcUa, MxAccessGateway and HistorianGateway. No app exports logs over OTLP, because every
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
(`logs.jsonl` stayed empty).
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
- **scadaproj#54:** collector on the wonder tiers. **DONE 2026-11-12:** one loopback-only `ZbOtelCollector` per box on vd03/vq03/vq04, every app with an exporter setting switched to OTLP, and data proven at each collector. On vd03 and QA this includes the **ScadaBridge Central role** (`node.role=Central`). Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)
No finding needed a library change. The fixes and gaps above are all on the app side. `0.2.1`'s
scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also
need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays
built-but-unpublished until a pin bump wants it.
---
# Resource identity + logs over OTLP — scadaproj#55 (2026-11-12, same day)
Closes the three #55 gaps found above: no `service.version` anywhere, no `site.id`/`node.role` on OtOpcUa,
MxAccessGateway and HistorianGateway, and no logs over OTLP from any app.
## Root cause of "no logs"
`ZbSerilogConfig.ApplyOpenTelemetryExport` adds the `WriteTo.OpenTelemetry` sink only when the options
passed **to `AddZbSerilog`** say `Exporter=Otlp`. Every app filled those options with the service name alone
and set the exporter only on `AddZbTelemetry`, a separate options object the Serilog package never saw. The
sink condition could therefore never be true.
## Library: `ZB.MOM.WW.Telemetry(.Serilog)` 0.3.0 — published
scadaproj `main` `cfc7168` + `2011c8d`. Feed SHA256 (feed download = local pack):
| Package | SHA256 |
|---|---|
| `ZB.MOM.WW.Telemetry.0.3.0.nupkg` | `a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2` |
| `ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg` | `04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c` |
- **`service.version`**: `ServiceVersion = null` (the default) now means the entry assembly's
`AssemblyInformationalVersion` (the git SHA the apps stamp), falling back to its assembly version. Set a
value to override it; `""` omits the attribute.
- **`ResourceAttributes`**: extra Resource attributes on all three signals. The first-class keys
(`service.*`, `host.name`, `site.id`, `node.role`) are reserved and rejected at startup.
- **`ApplyConfiguration(section)`**: reads `Exporter`, `OtlpEndpoint`, `ServiceVersion`, `SiteId`, `NodeRole`,
`ExportLogs`, `ResourceAttributes:<key>` from an app's own `<App>:Telemetry` section. Absent keys keep the
code defaults, so the four hand-written exporter parsers are gone.
- **Logs follow `AddZbTelemetry`**: `AddZbTelemetry` registers its options in DI. `AddZbSerilog` resolves them
inside its `AddSerilog` callback (so call order does not matter), and when its own options do not select
OTLP, the OTLP log sink uses that registration's endpoint and Resource. `ExportLogs = false` keeps logs
local. Levels and the configuration-driven sinks are unchanged.
- **0.2.1 shipped in the same release.** It holds the P3 review fixes: redaction after `ConfigureLogger`,
dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at
startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in
config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no
wonder tier had a collector at the time (#54). Since 2026-11-12 the wonder boxes set `OtlpEndpoint=http://127.0.0.1:4317`, scheme included, so they pass the check when 0.3.0 is deployed.
- **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on
Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an
`ExportLogs=false` case with a metrics positive control, and a Prometheus case.
Mutation checks, each caught by the named tests:
- Drop the DI lookup → the e2e log test fails.
- Drop the version default → the pipeline-Resource and e2e tests fail.
- Drop `ResourceAttributes` → 2 attribute tests and the e2e test fail.
- Ignore `ExportLogs` → the decision test and the e2e test fail.
- Disable the reserved-key check → 7 theory cases fail.
- Drop `ApplyConfiguration`'s attribute or `SiteId` read → its test fails.
## Apps (all pushed to their default branch; no wonder-box deploys)
| App | Commit | Pin | Identity wiring |
|---|---|---|---|
| OtOpcUa | `master` `412da27c` | 0.1.0 → 0.3.0 | `site.id` = the application Cluster id (the `cluster-<ClusterId>` role suffix; omitted on a legacy node); `node.role` = the plain roles, sorted (`admin,driver`); Program passes its `EffectiveRoles`. |
| MxAccessGateway | `main` `d9d9ff48` | 0.1.0 → 0.3.0 (csproj) | `node.role=standalone` (one gateway per box); `site.id` from `MxGateway:Telemetry:SiteId` (there is no node identity in its config, so it is omitted unless set). |
| HistorianGateway | `main` `ae1dbcc` | 0.1.0 → 0.3.0 (csproj) | Same as mxgw, under the root-level `Telemetry:` section. |
| ScadaBridge | `main` `a633005f` | 0.2.0 → 0.3.0 | Unchanged `site.id`/`node.role`. The new `ScadaBridgeTelemetryConfig` fills both `AddZbTelemetry` and the pre-host `AddZbSerilog` logger. The logger is built on its own container with no `AddZbTelemetry` to follow, so the exporter is passed to it explicitly. |
Tests: OtOpcUa Host.Tests 288/288 + Host.IntegrationTests 325/327 (2 skipped); mxgw MxGateway.Tests 1446/1452
(6 skipped); HistorianGateway 1414 + 206 + 186 passed; ScadaBridge Host.Tests 699/699. Each app gained identity
tests (OtOpcUa `ObservabilityResourceIdentityTests`, HG `GatewayTracingTests`, ScadaBridge
`ScadaBridgeTelemetryConfigTests`).
## What arrived (collector, [`otelidentity.py`](../../infra/otel-collector/otelidentity.py), one process each)
All runs: `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317`, `OTEL_METRIC_EXPORT_INTERVAL=10000`.
**OtOpcUa** — local Release host, fused `admin,driver,cluster-OTEL55` single node. Its own Akka mesh on
`localhost:14053`. Throwaway ConfigDb `OtOpcUaOtel55` on the docker-dev SQL: EF-migrated, one
`ServerCluster` `OTEL55` + one `ClusterNode`, gRPC mesh/telemetry keys of its own. The DB was **dropped
afterwards**. Activity: one `POST /api/deployments` (202) plus health/metrics GETs.
```
RESOURCE resourceMetrics {"service.name": "otopcua", "service.version": "1.0.0+412da27c5fb7e440e4a75d4f8e82e0e5b9832411", "service.instance.id": "Josephs-MacBook-Pro:67675", "site.id": "OTEL55", "node.role": "admin,driver", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 427, 'spans': 29, 'log records': 73} SEVERITY {'Information': 61, 'Warning': 12}
LOG {"severity": "Information", "body": "OtOpcUa.Host starting with roles=[admin,driver,cluster-OTEL55] (admin=True, driver=True) source=Configured"}
LOG {"severity": "Information", "body": "HTTP POST /telemetry.v1.TelemetryStreamService/Subscribe responded 200 in 29942.6698 ms", "traceId": "f5f90813adac357cdce4c165ab9e1468"}
```
The `otopcua.deploy.apply` and `otopcua.opcua.address_space_rebuild` spans arrived again.
**HistorianGateway** — local Release host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores,
`Telemetry__SiteId=otel-probe-55`. Activity: `HistorianStatus/Probe` + health GETs.
```
RESOURCE resourceMetrics {"service.name": "historian-gateway", "service.version": "1.0.0+ae1dbccc201aeebbe030c407870879a7985002db", "service.instance.id": "Josephs-MacBook-Pro:72641", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 184, 'spans': 12, 'log records': 29} SEVERITY {'Information': 25, 'Warning': 2, 'Error': 2}
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:15220/health/ready - null null", "traceId": "8eca56a689d61f710bdbdaad32879a6b"}
```
**ScadaBridge (Site)** — local Release **Site** host, own site id `otel-probe-55`, own 1-node mesh (`localhost:18082`),
scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable.
```
RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0+a633005f4f826e662d09fa47359bced17ccd6acf", "service.instance.id": "Josephs-MacBook-Pro:69217", "site.id": "otel-probe-55", "node.role": "Site", "host.name": "Josephs-MacBook-Pro"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 290, 'spans': 30, 'log records': 78} SEVERITY {'Information': 66, 'Warning': 10, 'Error': 2}
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Site on localhost"}
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:18084/health/ready - null null", "traceId": "0e976e6c0b8d7d3d0aaf85a99f5fbe21"}
```
The first line, `Starting ScadaBridge host …`, is written to the static logger before any web host exists, so
the explicit exporter path on the pre-host logger works. **Central not run:** the rig was in use by the
ScadaBridge#132 failover drills (`sb132-p` / `sb132-probe` running, centrals recreated 3–6 minutes earlier), so
ScadaBridge#133 stays open.
**MxAccessGateway** — Release `main` `54aa61a0`, published framework-dependent win-x64 from the Mac. (It was
pushed as `d9d9ff48` after rebasing over one docs-only commit.) Run as a scratch copy at `C:\otel55\mxgw` on
windev, on `:15120`/`:15130`, under `Staging` + `DisableLogin`, with alarms off, scratch auth/secrets/snapshot
stores and a throwaway pepper. `Worker:ExecutablePath` pointed at the non-existent
`C:\otel55\OtelProbeNoWorker.exe`. It was launched through `Win32_Process.Create`, because a process started
from the ssh session dies with it. `MxGateway__Telemetry__SiteId=otel-probe-55`.
```
RESOURCE resourceMetrics {"service.name": "mxgateway", "service.version": "0.1.2+54aa61a0", "service.instance.id": "DESKTOP-6JL3KKO:23580", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "DESKTOP-6JL3KKO"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 454, 'spans': 37, 'log records': 21} SEVERITY {'Warning': 3, 'Information': 6, 'Error': 12}
LOG {"severity": "Information", "body": "Gateway alarm monitor disabled (MxGateway:Alarms:Enabled is false)."}
LOG {"severity": "Error", "body": "Health check worker-executable with status Unhealthy completed after 4.6339ms with message 'Worker executable ...", "traceId": "b76b4059dd4f33c88e556ec5ef2e4083"}
```
The live `MxAccessGw` service stayed `Running` with `:5130/health/ready` = 200 throughout. Its gateway PID
(9148) and worker PIDs (`14700,22108`) were unchanged before and after. The scratch copy and `C:\otel55` were
deleted.
On every app, log records carry `traceId` when written inside a request, so logs join their spans in the
backend. The Resource is byte-identical across the three signals of each process.
---
# ScadaBridge Central role over OTLP — ScadaBridge#133 (2026-11-12, same day)
Closes the last gap from the two runs above: the **Central** role had never exported. This run used the real
docker rig instead of a throwaway host, because a Central needs the config DB and LDAP.
## How it was run
- **Build:** the rig image predated Telemetry 0.3.0 (its `ZB.MOM.WW.Telemetry.dll` was `0.2.0+01c6dc1d`). It was
rebuilt with `docker/deploy.sh` from a clean `main` at `482ac665` (which includes `a633005f`, Telemetry 0.3.0),
and all 8 nodes came back healthy.
- **Override:** a temporary compose file (scratch, never committed) set on **central-a and central-b only**
`ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` and
`OTEL_METRIC_EXPORT_INTERVAL=10000`. Both centrals were recreated together
(`docker compose -f docker/docker-compose.yml -f <override> up -d --force-recreate --no-deps central-a central-b`),
per the rig's pair-restart rule. Sites, Traefik and every other container were left alone.
- **Activity (through Traefik `:9000`):**
- 9 inbound calls rejected with 401: no key, or an empty key.
- 3 authorized `POST /api/SimpleAlarmStatus` (200 `{"WasSuccessful":true,…}`), made with a temporary method-scoped
API key. All three temporary keys were deleted afterwards (`api-key list` shows 0 `otel133*`).
- One `deploy instance --id 1492` (`rig-probe-1`, status Success).
- CLI `template list` / `site list` and UI/health/metrics GETs, on top of the health polling and the site heartbeats
that already happen.
## What arrived (collector, one process per central)
```
central-a (active) RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0", "service.instance.id": "980946b3915d:6", "site.id": "central", "node.role": "Central", "host.name": "980946b3915d"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 463, 'spans': 321, 'log records': 3305} SEVERITY {'Information': 3046, 'Warning': 259}
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Central on scadabridge-central-a"}
central-b (standby) RESOURCE ... "service.instance.id": "d4d8e49df76f:6", "site.id": "central", "node.role": "Central" ... (identical across the three signals)
COUNTS {'metric points': 421, 'spans': 77, 'log records': 331} SEVERITY {'Information': 311, 'Warning': 20}
```
`service.namespace=ZB.MOM.WW` is present on both. Logs carry `traceId` inside requests, as on the other apps.
**Central instruments** (meter `ZB.MOM.WW.ScadaBridge`, last cumulative value on central-a):
```
scadabridge.deployments.applied 1
scadabridge.inbound_api.requests {method=SimpleAlarmStatus} 3
scadabridge.inbound_api.requests {method=<unauthorized>} 9
scadabridge.central.transport.import_sessions.open 0
+ the shared gauges: site.stream.subscribers.active, site.data_connections.{connected,total},
store_and_forward.queue.depth, site.alarm_cache.aggregators.active (all 0 on a central)
```
The counts match the activity exactly. The standby (central-b) exports only the observable gauges, because the
deploy and inbound singletons run on the active node.
**Spans on central-a** (standard instrumentation; ScadaBridge has no `ActivitySource` of its own):
`POST /scadabridge.centralcontrol.v1.CentralControlService/{Heartbeat 133, SubmitNotification 94, IngestAuditEvents 24, ReportSiteHealth 10}`,
`POST /management` 16, `POST /auth/token` 16, `POST /api/{methodName}` 12, `GET /api/internal/deployments/{deploymentId}/config` 1,
the gRPC client spans `SiteCommandService/ExecuteRoute` 3 + `ExecuteLifecycle` 1 (the inbound route to the site
and the deploy), `GET /health/{active,ready}`, and the Blazor `Route / -> …CentralUI…Dashboard`.
**Allowlist:** `SiteServiceRegistration.ObservedMeters` names `ZB.MOM.WW.ScadaBridge` (which holds every central and
site instrument) and the LocalDb meter, and the repository creates no other `Meter` or `ActivitySource`. **No defect.**
## Restored
Both centrals were recreated together again on the default compose with no override (0 telemetry env vars). Then:
- all 8 nodes `/health/ready` 200, one active per pair (central-a, site-{a,b,c}-a 200; their partners 503);
- central `memberCount 2, unreachableCount 0`;
- Traefik `/health/active` 200;
- negative control: the new central processes (`a596a23d9a12`, `1b6fd48c6785`) have **0** lines in
`metrics/traces/logs.jsonl`.
The rig stays on the rebuilt `482ac665` image.
## Follow-ups filed (not fixed here)
- **scadaproj#56:** the Serilog OTLP sink's own `LogsService/Export` calls are traced as gRPC client spans in
**all four apps** (scadabridge 127, mxgateway 13, historian-gateway 10, otopcua 6 at the collector). The SDK's
trace and metric exporters suppress their own instrumentation, and `Serilog.Sinks.OpenTelemetry` does not. This
is noise, not a loop (0 log records about it). Fix in `ZB.MOM.WW.Telemetry` with an instrumentation filter. **Fixed in 0.3.1, same day — see below.**
- **ScadaBridge#134:** rig images report `service.version = "1.0.0"` with no `+<sha>`, because `.dockerignore`
excludes `.git/`. Fix by passing `SourceRevisionId` as a build arg. Wonder deploys, published from a checkout, are
unaffected.
---
# OTLP export calls no longer traced — scadaproj#56 (2026-11-12, same day)
## Fix: `ZB.MOM.WW.Telemetry(.Serilog)` 0.3.1 — published
The SDK's OTLP exporters suppress their own instrumentation. `Serilog.Sinks.OpenTelemetry` does not, so
`AddGrpcClientInstrumentation()` traced every log batch it sent. `AddZbTelemetry` now classifies an outgoing call
with `ZbOtlpExportCallFilter.IsExportCall`: any `/opentelemetry.proto.collector.*` path on any host, or
OTLP/HTTP `/v1/{traces,metrics,logs}` on the configured `OtlpEndpoint`'s host:port.
- **HttpClient:** `FilterHttpRequestMessage` returns `false` for an export call.
- **gRPC client:** `GrpcClientTraceInstrumentationOptions` has no `Filter` in `1.15.1-beta.1`. The
`EnrichWithHttpRequestMessage` callback runs synchronously in the instrumentation's start handler, before
`HttpClient.SendAsync` (checked in the contrib source of that tag). For an export call it clears
`ActivityTraceFlags.Recorded` and `IsAllDataRequested`, so the export processor skips the span and the child
HTTP activity is unsampled under the default parent-based sampler.
- A processor that drops spans in `OnEnd` was rejected. It only works when it is registered before every exporter,
and an app could add an exporter first. The instrumentation hooks do not depend on processor order.
Published from scadaproj `main` `85654d7` (the commit in the nuspec). SHA256, feed download = local pack:
`ZB.MOM.WW.Telemetry.0.3.1.nupkg` `baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3`,
`ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg` `188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931`.
Registration lists `0.1.0, 0.2.0, 0.3.0, 0.3.1` for both ids.
**Tests (54 + 40, Release):**
- `OtlpExportCallFilterTests`: 16 cases. The positives are the three collector services, a collector path on
another host, and `/v1/*` on the endpoint. The negatives are ScadaBridge→site, HG→historian and mxgw paths, REST,
a `collectorx` look-alike, `/v1/logs` on another host or port, and relative URIs.
- `Otlp_export_calls_are_not_traced_but_the_apps_own_calls_are`: a host runs against the loopback OTLP receiver.
The positive controls are an app gRPC call (`zb.test.v1.AppService/Ping`) and an HttpClient call, both traced. The
negatives are a real Serilog-sink log batch (the test asserts it arrived), an HttpClient `POST /v1/logs` to the
endpoint, and a collector path over HttpClient; none is traced. The receiver's own ASP.NET Core server spans land
in the same export because the instrumentation listens process-wide. So the assertions key on client-span markers:
the protobuf span name for gRPC and `url.full` for HttpClient.
- Mutations: removing the gRPC un-record fails the test (*"LogsService/Export call was traced"*). Removing the
HttpClient filter fails it (`url.full …/v1/logs` found). Making `IsExportCall` always true fails it too
(*"positive control: the app's gRPC call was not traced"*). It passed 5 of 5 repeated runs.
## Apps (pushed to their default branch; no wonder-box deploys)
| App | Commit | Pin | Tests (Debug, against 0.3.1 — `project.assets.json` confirms it on every project) |
|---|---|---|---|
| OtOpcUa | `master` `f3178c5f` | `Directory.Packages.props` | `Host.IntegrationTests` 325 passed / 2 skipped |
| MxAccessGateway | `main` `453fe015` | `Server.csproj` | NonWindows solution: 1446 + 5 passed (28 skipped) |
| HistorianGateway | `main` `dffb5e2` | `Server.csproj` | 186 + 206 + 1414 passed (76 skipped) |
| ScadaBridge | `main` `95584e2f` | `Directory.Packages.props` | `Host.Tests` 721 passed |
## Collector before/after
The collector-wide count of `LogsService/Export` spans (`zb-otel-collector`, all runs to date) was the same before
and after both probe runs: **scadabridge 146, mxgateway 13, historian-gateway 10, otopcua 6**. The issue reported
127 for scadabridge; the ScadaBridge#133 centrals added more before the rig was restored. Each probe ran as a
throwaway local process on 0.3.1 with `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317`,
`OTEL_METRIC_EXPORT_INTERVAL=10000` and `site.id=otel-probe-56`, counted from its own start time:
| Run | Log export batches / records arrived | `*Service/Export` spans | App client spans still arriving |
|---|---|---|---|
| HistorianGateway `Development`, `:15220`/`:15221`, scratch stores, health/metrics GETs + 3 `HistorianStatus/Probe` | 15 / 168 | **0** | `HistoryService/GetInterfaceVersion` 14 (GrpcNetClient), `historian.Connect` 14, server spans |
| ScadaBridge **Site** `95584e2f`, own site id, own 1-node mesh `localhost:18082`, gRPC `:18083`, metrics `:18084`, central `localhost:18999` unreachable, scratch LocalDb/audit/secrets | 27 / 224 | **0** | `CentralControlService/Heartbeat` 17, `ReconcileSite` 2, `ReportSiteHealth` 2 (GrpcNetClient), server spans |
So the log sink exported throughout, the gRPC client instrumentation stayed on, and no export call was traced.
mxgw is proven by the library tests and its suite only; no windev scratch run was made. The docker rigs were not
touched.
The wonder boxes (#54) still run their earlier builds. They will emit `LogsService/Export` spans once a 0.3.0 build
is deployed, unless 0.3.1 is deployed instead.