41a5e4d187
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
460 lines
34 KiB
Markdown
460 lines
34 KiB
Markdown
# OTLP export, end to end — evidence (2026-11-12, scadaproj#13)
|
||
|
||
First run of the family's OTLP export path against a real collector. Before this, every app had
|
||
the `<App>:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` wiring but nothing had ever received from
|
||
it, and OtOpcUa's spans had no exporter anywhere.
|
||
|
||
**Result:** OTLP **metrics + traces** proven for **OtOpcUa, HistorianGateway, ScadaBridge (Site
|
||
role) and MxAccessGateway**. One real defect found and fixed (OtOpcUa Galaxy driver telemetry was
|
||
never exported, lmxopcua#892). No Telemetry library change was needed; `0.2.1` stays unpublished.
|
||
|
||
## Collector
|
||
|
||
`zb-otel-collector` on the shared Docker host `10.100.0.35` — `otel/opentelemetry-collector-contrib:0.162.0`,
|
||
OTLP gRPC `:4317` + HTTP `:4318`, Prometheus exporter `:8889`, health `:13133`; `debug` + `file`
|
||
exporters for proof. Source and runbook: [`infra/otel-collector/`](../../infra/otel-collector/). The
|
||
four ports were free on the host before deploy, and no other container was touched.
|
||
|
||
```
|
||
$ curl -s http://10.100.0.35:13133/
|
||
{"status":"Server available","upSince":"2026-11-13T01:24:13.274419432Z",...}
|
||
$ curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d '{}' http://10.100.0.35:4318/v1/traces
|
||
200
|
||
windev> Test-NetConnection 10.100.0.35 -Port 4317 -InformationLevel Quiet
|
||
True
|
||
```
|
||
|
||
## How each app was run
|
||
|
||
No shared rig container was recreated or reconfigured. Each app ran as a throwaway process with
|
||
its own state, `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317` (gRPC) and
|
||
`OTEL_METRIC_EXPORT_INTERVAL=10000`.
|
||
|
||
| App | Build | Telemetry pin | How |
|
||
|---|---|---|---|
|
||
| OtOpcUa | `master` `aaaa55e2` + the #892 fix (`0b90379f`) | 0.1.0 | Local `dotnet` host on the Mac: fused `admin,driver` single node in its own Akka mesh (`localhost:14053`), its own throwaway ConfigDb `OtOpcUaOtel13` on the docker-dev SQL (EF-migrated, one `ServerCluster`/`ClusterNode` row, dropped afterwards), HTTP `:19100`, OPC UA `:14840`. Activity: one `POST /api/deployments` + health/metrics GETs. |
|
||
| HistorianGateway | `main` `b95bae3` | 0.1.0 | Local `dotnet` host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores. Activity: status RPCs via `grpcurl` (the historian itself is unreachable without the VPN, so they fail — which still produces spans) + dashboard/health GETs. |
|
||
| ScadaBridge | `main` `139a43fb` | 0.2.0 | Local `dotnet` **Site** host, own site id `otel-probe`, own 1-node mesh (`localhost:18082`), scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable. The docker rig was in active use (ScadaBridge#46/#132; containers recreated minutes earlier), so it was not touched. |
|
||
| MxAccessGateway | deployed windev Server `447070a5` (scratch copy) | 0.1.0 | Copy of `C:\publish\mxaccessgw\Server-20261112-447070a5` run as a separate process on `:15120`/`:15130`, `Staging`, alarms off, scratch auth/secrets stores, `Worker:ExecutablePath` pointed at a non-existent `OtelProbeNoWorker.exe` so its `OrphanWorkerTerminator` (which matches by image name) could not match the live service's x86 workers. Live `MxAccessGw` stayed `Running`, `:5130/health/ready` = 200 throughout, worker PIDs `14700,22108` unchanged before/after; the scratch copy was deleted. |
|
||
|
||
## What arrived (collector `otelsum.py` output, trimmed)
|
||
|
||
Standard instrumentation scopes that every app sent are listed once at the end.
|
||
|
||
### OtOpcUa — metrics + traces
|
||
|
||
```
|
||
RESOURCE otopcua {"service.name": "otopcua", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:48096", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
|
||
METRICS otopcua ZB.MOM.WW.OtOpcUa 3 ['otopcua.deploy.applied', 'otopcua.deploy.apply.duration', 'otopcua.redundancy.service_level_change']
|
||
METRICS otopcua ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
|
||
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.deploy.apply')
|
||
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.opcua.address_space_rebuild')
|
||
SPANS 1 ('otopcua', 'Microsoft.AspNetCore', 'POST /api/deployments')
|
||
SPANS 5 ('otopcua', 'Microsoft.AspNetCore', 'GET /health/ready')
|
||
SPANS 3 ('otopcua', 'Microsoft.AspNetCore', 'GET /metrics')
|
||
```
|
||
|
||
```json
|
||
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.deploy.apply", "traceId": "e5a579168f8c4629b25af4d3e4f2cb07", "spanId": "4f12d04ada38ea41", "kind": 1,
|
||
"attributes": [{"key": "otopcua.deployment_id", "value": {"stringValue": "e8b8c010d50144ccb371a91defceb6d9"}}, {"key": "otopcua.node_id", "value": {"stringValue": "localhost:14053"}},
|
||
{"key": "otopcua.revision", "value": {"stringValue": "ed37612b…7bba"}}, {"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
|
||
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.opcua.address_space_rebuild", "traceId": "f6d8c238b2399cce0928de84691a0106", "spanId": "305ec5a9012a799d", "kind": 1,
|
||
"attributes": [{"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
|
||
```
|
||
|
||
OtOpcUa's tracing is no longer a no-op once `Exporter=Otlp` is set: its own spans reach a backend
|
||
with their attributes. Note that `address_space_rebuild` is a separate trace from `deploy.apply`
|
||
(no parent — the context does not cross the Akka actor hop); the two join on
|
||
`otopcua.correlation_id`.
|
||
|
||
### HistorianGateway — metrics + traces
|
||
|
||
```
|
||
RESOURCE historian-gateway {"service.name": "historian-gateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:49942", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
|
||
METRICS historian-gateway ZB.MOM.WW.HistorianGateway 10 ['historian_gateway.grpc.request.duration', 'historian_gateway.grpc.requests', 'historian_gateway.historian.connection.up', 'historian_gateway.historian.event_session_pool.lease_waiters', 'historian_gateway.historian.event_session_pool.warm_sessions', 'historian_gateway.historian.session_pool.lease_waiters', 'historian_gateway.historian.session_pool.warm_sessions', 'historian_gateway.redundancy.members.healthy', 'historian_gateway.session_pool.lease.wait.duration', 'historian_gateway.store_forward.queue.depth']
|
||
SPANS 8 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.Connect')
|
||
SPANS 2 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.pool.lease')
|
||
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/Probe')
|
||
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/GetConnectionStatus')
|
||
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'galaxy_repository.v1.GalaxyRepository/GetLastDeployTime')
|
||
SPANS 8 ('historian-gateway', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'HistoryService/GetInterfaceVersion')
|
||
SPANS 4 ('historian-gateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.HistorianGateway.Server.Dashboard.Components.Pages.Home')
|
||
```
|
||
|
||
`historian.Connect` spans carry `server.address`/`server.port` and `error.type=…HistorianTransportException`
|
||
(no VPN); the ones a request triggered are parented under its gRPC request span, the pool warm-up ones are roots. Prometheus exporter, same data:
|
||
|
||
```
|
||
historian_gateway_grpc_requests_total{job="ZB.MOM.WW/historian-gateway",method="Probe",outcome="ok",service="HistorianStatus",service_name="historian-gateway",...} 1
|
||
```
|
||
|
||
### ScadaBridge (Site) — metrics + traces
|
||
|
||
```
|
||
RESOURCE scadabridge {"service.name": "scadabridge", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:50923", "host.name": "Josephs-MacBook-Pro", "site.id": "otel-probe", "node.role": "Site", "_sdk": "1.15.3"}
|
||
METRICS scadabridge ZB.MOM.WW.ScadaBridge 6 ['scadabridge.central.transport.import_sessions.open', 'scadabridge.site.alarm_cache.aggregators.active', 'scadabridge.site.data_connections.connected', 'scadabridge.site.data_connections.total', 'scadabridge.site.stream.subscribers.active', 'scadabridge.store_and_forward.queue.depth']
|
||
METRICS scadabridge ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
|
||
SPANS 12 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/Heartbeat')
|
||
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReportSiteHealth')
|
||
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReconcileSite')
|
||
SPANS 4 ('scadabridge', 'Microsoft.AspNetCore', 'GET /health/ready')
|
||
```
|
||
|
||
The only app that sends `site.id` / `node.role`. ScadaBridge has no `ActivitySource` of its own,
|
||
so its spans are standard instrumentation only. **Central role not run** → ScadaBridge#133.
|
||
|
||
### MxAccessGateway — metrics + traces
|
||
|
||
```
|
||
RESOURCE mxgateway {"service.name": "mxgateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "DESKTOP-6JL3KKO:8720", "host.name": "DESKTOP-6JL3KKO", "_sdk": "1.15.3"}
|
||
METRICS mxgateway ZB.MOM.WW.MxGateway 5 ['mxgateway.alarms.provider_mode', 'mxgateway.events.grpc_stream_queue.depth', 'mxgateway.events.worker_queue.depth', 'mxgateway.sessions.open', 'mxgateway.workers.running']
|
||
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /')
|
||
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /health/ready')
|
||
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /metrics')
|
||
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.MxGateway.Server.Dashboard.Components.Pages.DashboardHome')
|
||
```
|
||
|
||
Only the observable gauges appear: the counters/histograms (`commands.*`, `workers.startup.duration`, …)
|
||
record nothing without a worker session, and the x86 worker was deliberately not run. The gateway
|
||
has no `ActivitySource` of its own (spans are standard instrumentation only). Its gRPC port is not
|
||
reachable from off-box (windev firewall), so the traffic was generated on-box.
|
||
|
||
### Standard instrumentation (all four)
|
||
|
||
`System.Runtime` (19 `dotnet.*`), `OpenTelemetry.Instrumentation.Process` (5 `process.*`),
|
||
`System.Net.Http` (4 `http.client.*`), `Microsoft.AspNetCore.Hosting` (`http.server.request.duration`,
|
||
`http.server.active_requests`), `Microsoft.AspNetCore.Server.Kestrel`, `.Routing`, `.MemoryPool`;
|
||
plus `.Authentication`/`.Authorization`/`.Components` on the apps with a Blazor dashboard.
|
||
|
||
## Defect found and fixed
|
||
|
||
**lmxopcua#892 — the Galaxy driver's telemetry was never exported.** `ZbTelemetryOptions.Meters`
|
||
and `.ActivitySources` are strict allowlists. OtOpcUa named only `ZB.MOM.WW.OtOpcUa` and the
|
||
LocalDb meter, while the in-process Galaxy driver opens its spans (`galaxy.subscribe_bulk`,
|
||
`galaxy.unsubscribe_bulk`, `galaxy.stream_events`, `galaxy.write`, `galaxy.get_hierarchy`) on
|
||
ActivitySource `ZB.MOM.WW.OtOpcUa.Driver.Galaxy` and records the event pump's `galaxy.events.*`
|
||
counters on a Meter of the same name. All of it was silently dropped from `/metrics` and OTLP.
|
||
|
||
Fix: OtOpcUa `master` `db86b996` (commit `0b90379f`) — a public `GalaxyDriverTelemetryNames` in
|
||
the driver; `ObservabilityExtensions.ObservedMeters` / `ObservedActivitySources` name it.
|
||
`ObservabilityAllowlistTests` assert on the listener side (span sampled, instrument enabled) with an
|
||
unregistered-name negative control. Dropping the Galaxy entries fails 3 of 8 tests, so the tests can
|
||
catch the regression. Galaxy driver suite 584/584. Not deployed. A live `galaxy.*` span was not
|
||
driven: that would need a Galaxy driver deployment against a gateway.
|
||
|
||
The allowlists of the other apps were checked against every `new Meter(` / `new ActivitySource(`
|
||
in their source and in the hosted libraries. ScadaBridge (`ZB.MOM.WW.ScadaBridge`, LocalDb),
|
||
HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGateway
|
||
(`ZB.MOM.WW.MxGateway`) are complete. Of the hosted libraries, only LocalDb defines a meter.
|
||
|
||
## Checked, no defect
|
||
|
||
- **Protocol/endpoint:** every app uses the library's `AddOtlpExporter` default (gRPC). `http://host:4317`
|
||
works for all four. No checked-in config carries a scheme-less endpoint.
|
||
- **Prometheus stays up under Otlp:** `/metrics` answered 200 on every app with `Exporter=Otlp` (`http.response.status_code=200` on each app's exported `GET /metrics` span)
|
||
(the overlay is additive, as documented).
|
||
- **Resource:** `service.name`, `service.namespace=ZB.MOM.WW`, `service.instance.id`, `host.name`
|
||
present on all four.
|
||
|
||
## Gaps filed (not fixed here)
|
||
|
||
- **scadaproj#55:** `service.version` is absent on all four apps; `site.id`/`node.role` are absent
|
||
on OtOpcUa, MxAccessGateway and HistorianGateway. No app exports logs over OTLP, because every
|
||
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
|
||
(`logs.jsonl` stayed empty).
|
||
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
|
||
- **scadaproj#54:** collector on the wonder tiers. **DONE 2026-11-12:** one loopback-only `ZbOtelCollector` per box on vd03/vq03/vq04, every app with an exporter setting switched to OTLP, and data proven at each collector. On vd03 and QA this includes the **ScadaBridge Central role** (`node.role=Central`). Record: [`infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md`](../../infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md).
|
||
|
||
## Telemetry 0.2.1: not published (superseded — shipped inside 0.3.0, see below)
|
||
|
||
No finding needed a library change. The fixes and gaps above are all on the app side. `0.2.1`'s
|
||
scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also
|
||
need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays
|
||
built-but-unpublished until a pin bump wants it.
|
||
|
||
---
|
||
|
||
# Resource identity + logs over OTLP — scadaproj#55 (2026-11-12, same day)
|
||
|
||
Closes the three #55 gaps found above: no `service.version` anywhere, no `site.id`/`node.role` on OtOpcUa,
|
||
MxAccessGateway and HistorianGateway, and no logs over OTLP from any app.
|
||
|
||
## Root cause of "no logs"
|
||
|
||
`ZbSerilogConfig.ApplyOpenTelemetryExport` adds the `WriteTo.OpenTelemetry` sink only when the options
|
||
passed **to `AddZbSerilog`** say `Exporter=Otlp`. Every app filled those options with the service name alone
|
||
and set the exporter only on `AddZbTelemetry`, a separate options object the Serilog package never saw. The
|
||
sink condition could therefore never be true.
|
||
|
||
## Library: `ZB.MOM.WW.Telemetry(.Serilog)` 0.3.0 — published
|
||
|
||
scadaproj `main` `cfc7168` + `2011c8d`. Feed SHA256 (feed download = local pack):
|
||
|
||
| Package | SHA256 |
|
||
|---|---|
|
||
| `ZB.MOM.WW.Telemetry.0.3.0.nupkg` | `a61492383bc854f6108f20da6ec55a16b2e0bf6121cd7ca13c86a951cb29f5e2` |
|
||
| `ZB.MOM.WW.Telemetry.Serilog.0.3.0.nupkg` | `04675cead32e2fb2c2cfbe88cdb4b7895b4b1601c53ba379b6215f53c731478c` |
|
||
|
||
- **`service.version`**: `ServiceVersion = null` (the default) now means the entry assembly's
|
||
`AssemblyInformationalVersion` (the git SHA the apps stamp), falling back to its assembly version. Set a
|
||
value to override it; `""` omits the attribute.
|
||
- **`ResourceAttributes`**: extra Resource attributes on all three signals. The first-class keys
|
||
(`service.*`, `host.name`, `site.id`, `node.role`) are reserved and rejected at startup.
|
||
- **`ApplyConfiguration(section)`**: reads `Exporter`, `OtlpEndpoint`, `ServiceVersion`, `SiteId`, `NodeRole`,
|
||
`ExportLogs`, `ResourceAttributes:<key>` from an app's own `<App>:Telemetry` section. Absent keys keep the
|
||
code defaults, so the four hand-written exporter parsers are gone.
|
||
- **Logs follow `AddZbTelemetry`**: `AddZbTelemetry` registers its options in DI. `AddZbSerilog` resolves them
|
||
inside its `AddSerilog` callback (so call order does not matter), and when its own options do not select
|
||
OTLP, the OTLP log sink uses that registration's endpoint and Resource. `ExportLogs = false` keeps logs
|
||
local. Levels and the configuration-driven sinks are unchanged.
|
||
- **0.2.1 shipped in the same release.** It holds the P3 review fixes: redaction after `ConfigureLogger`,
|
||
dictionary-key collisions, and the stricter endpoint check (`collector:4317` without a scheme now fails at
|
||
startup when `Exporter=Otlp`). It is a minor bump because it adds API and changes behaviour. No checked-in
|
||
config in any of the four repos sets an OTLP endpoint. The check is a no-op unless `Exporter=Otlp`, and no
|
||
wonder tier had a collector at the time (#54). Since 2026-11-12 the wonder boxes set `OtlpEndpoint=http://127.0.0.1:4317`, scheme included, so they pass the check when 0.3.0 is deployed.
|
||
- **Tests**: 42 → **77** in Release (Telemetry 38, Serilog 39). They include a loopback OTLP/gRPC receiver on
|
||
Kestrel that proves a log record arrives with `service.version`/`site.id`/`node.role`/an extra attribute, an
|
||
`ExportLogs=false` case with a metrics positive control, and a Prometheus case.
|
||
|
||
Mutation checks, each caught by the named tests:
|
||
- Drop the DI lookup → the e2e log test fails.
|
||
- Drop the version default → the pipeline-Resource and e2e tests fail.
|
||
- Drop `ResourceAttributes` → 2 attribute tests and the e2e test fail.
|
||
- Ignore `ExportLogs` → the decision test and the e2e test fail.
|
||
- Disable the reserved-key check → 7 theory cases fail.
|
||
- Drop `ApplyConfiguration`'s attribute or `SiteId` read → its test fails.
|
||
|
||
## Apps (all pushed to their default branch; no wonder-box deploys)
|
||
|
||
| App | Commit | Pin | Identity wiring |
|
||
|---|---|---|---|
|
||
| OtOpcUa | `master` `412da27c` | 0.1.0 → 0.3.0 | `site.id` = the application Cluster id (the `cluster-<ClusterId>` role suffix; omitted on a legacy node); `node.role` = the plain roles, sorted (`admin,driver`); Program passes its `EffectiveRoles`. |
|
||
| MxAccessGateway | `main` `d9d9ff48` | 0.1.0 → 0.3.0 (csproj) | `node.role=standalone` (one gateway per box); `site.id` from `MxGateway:Telemetry:SiteId` (there is no node identity in its config, so it is omitted unless set). |
|
||
| HistorianGateway | `main` `ae1dbcc` | 0.1.0 → 0.3.0 (csproj) | Same as mxgw, under the root-level `Telemetry:` section. |
|
||
| ScadaBridge | `main` `a633005f` | 0.2.0 → 0.3.0 | Unchanged `site.id`/`node.role`. The new `ScadaBridgeTelemetryConfig` fills both `AddZbTelemetry` and the pre-host `AddZbSerilog` logger. The logger is built on its own container with no `AddZbTelemetry` to follow, so the exporter is passed to it explicitly. |
|
||
|
||
Tests: OtOpcUa Host.Tests 288/288 + Host.IntegrationTests 325/327 (2 skipped); mxgw MxGateway.Tests 1446/1452
|
||
(6 skipped); HistorianGateway 1414 + 206 + 186 passed; ScadaBridge Host.Tests 699/699. Each app gained identity
|
||
tests (OtOpcUa `ObservabilityResourceIdentityTests`, HG `GatewayTracingTests`, ScadaBridge
|
||
`ScadaBridgeTelemetryConfigTests`).
|
||
|
||
## What arrived (collector, [`otelidentity.py`](../../infra/otel-collector/otelidentity.py), one process each)
|
||
|
||
All runs: `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317`, `OTEL_METRIC_EXPORT_INTERVAL=10000`.
|
||
|
||
**OtOpcUa** — local Release host, fused `admin,driver,cluster-OTEL55` single node. Its own Akka mesh on
|
||
`localhost:14053`. Throwaway ConfigDb `OtOpcUaOtel55` on the docker-dev SQL: EF-migrated, one
|
||
`ServerCluster` `OTEL55` + one `ClusterNode`, gRPC mesh/telemetry keys of its own. The DB was **dropped
|
||
afterwards**. Activity: one `POST /api/deployments` (202) plus health/metrics GETs.
|
||
|
||
```
|
||
RESOURCE resourceMetrics {"service.name": "otopcua", "service.version": "1.0.0+412da27c5fb7e440e4a75d4f8e82e0e5b9832411", "service.instance.id": "Josephs-MacBook-Pro:67675", "site.id": "OTEL55", "node.role": "admin,driver", "host.name": "Josephs-MacBook-Pro"}
|
||
RESOURCE resourceSpans {... identical ...}
|
||
RESOURCE resourceLogs {... identical ...}
|
||
COUNTS {'metric points': 427, 'spans': 29, 'log records': 73} SEVERITY {'Information': 61, 'Warning': 12}
|
||
LOG {"severity": "Information", "body": "OtOpcUa.Host starting with roles=[admin,driver,cluster-OTEL55] (admin=True, driver=True) source=Configured"}
|
||
LOG {"severity": "Information", "body": "HTTP POST /telemetry.v1.TelemetryStreamService/Subscribe responded 200 in 29942.6698 ms", "traceId": "f5f90813adac357cdce4c165ab9e1468"}
|
||
```
|
||
The `otopcua.deploy.apply` and `otopcua.opcua.address_space_rebuild` spans arrived again.
|
||
|
||
**HistorianGateway** — local Release host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores,
|
||
`Telemetry__SiteId=otel-probe-55`. Activity: `HistorianStatus/Probe` + health GETs.
|
||
|
||
```
|
||
RESOURCE resourceMetrics {"service.name": "historian-gateway", "service.version": "1.0.0+ae1dbccc201aeebbe030c407870879a7985002db", "service.instance.id": "Josephs-MacBook-Pro:72641", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "Josephs-MacBook-Pro"}
|
||
RESOURCE resourceSpans {... identical ...}
|
||
RESOURCE resourceLogs {... identical ...}
|
||
COUNTS {'metric points': 184, 'spans': 12, 'log records': 29} SEVERITY {'Information': 25, 'Warning': 2, 'Error': 2}
|
||
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:15220/health/ready - null null", "traceId": "8eca56a689d61f710bdbdaad32879a6b"}
|
||
```
|
||
|
||
**ScadaBridge (Site)** — local Release **Site** host, own site id `otel-probe-55`, own 1-node mesh (`localhost:18082`),
|
||
scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable.
|
||
|
||
```
|
||
RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0+a633005f4f826e662d09fa47359bced17ccd6acf", "service.instance.id": "Josephs-MacBook-Pro:69217", "site.id": "otel-probe-55", "node.role": "Site", "host.name": "Josephs-MacBook-Pro"}
|
||
RESOURCE resourceSpans {... identical ...}
|
||
RESOURCE resourceLogs {... identical ...}
|
||
COUNTS {'metric points': 290, 'spans': 30, 'log records': 78} SEVERITY {'Information': 66, 'Warning': 10, 'Error': 2}
|
||
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Site on localhost"}
|
||
LOG {"severity": "Information", "body": "Request starting HTTP/1.1 GET http://localhost:18084/health/ready - null null", "traceId": "0e976e6c0b8d7d3d0aaf85a99f5fbe21"}
|
||
```
|
||
The first line, `Starting ScadaBridge host …`, is written to the static logger before any web host exists, so
|
||
the explicit exporter path on the pre-host logger works. **Central not run:** the rig was in use by the
|
||
ScadaBridge#132 failover drills (`sb132-p` / `sb132-probe` running, centrals recreated 3–6 minutes earlier), so
|
||
ScadaBridge#133 stays open.
|
||
|
||
**MxAccessGateway** — Release `main` `54aa61a0`, published framework-dependent win-x64 from the Mac. (It was
|
||
pushed as `d9d9ff48` after rebasing over one docs-only commit.) Run as a scratch copy at `C:\otel55\mxgw` on
|
||
windev, on `:15120`/`:15130`, under `Staging` + `DisableLogin`, with alarms off, scratch auth/secrets/snapshot
|
||
stores and a throwaway pepper. `Worker:ExecutablePath` pointed at the non-existent
|
||
`C:\otel55\OtelProbeNoWorker.exe`. It was launched through `Win32_Process.Create`, because a process started
|
||
from the ssh session dies with it. `MxGateway__Telemetry__SiteId=otel-probe-55`.
|
||
|
||
```
|
||
RESOURCE resourceMetrics {"service.name": "mxgateway", "service.version": "0.1.2+54aa61a0", "service.instance.id": "DESKTOP-6JL3KKO:23580", "site.id": "otel-probe-55", "node.role": "standalone", "host.name": "DESKTOP-6JL3KKO"}
|
||
RESOURCE resourceSpans {... identical ...}
|
||
RESOURCE resourceLogs {... identical ...}
|
||
COUNTS {'metric points': 454, 'spans': 37, 'log records': 21} SEVERITY {'Warning': 3, 'Information': 6, 'Error': 12}
|
||
LOG {"severity": "Information", "body": "Gateway alarm monitor disabled (MxGateway:Alarms:Enabled is false)."}
|
||
LOG {"severity": "Error", "body": "Health check worker-executable with status Unhealthy completed after 4.6339ms with message 'Worker executable ...", "traceId": "b76b4059dd4f33c88e556ec5ef2e4083"}
|
||
```
|
||
The live `MxAccessGw` service stayed `Running` with `:5130/health/ready` = 200 throughout. Its gateway PID
|
||
(9148) and worker PIDs (`14700,22108`) were unchanged before and after. The scratch copy and `C:\otel55` were
|
||
deleted.
|
||
|
||
On every app, log records carry `traceId` when written inside a request, so logs join their spans in the
|
||
backend. The Resource is byte-identical across the three signals of each process.
|
||
|
||
---
|
||
|
||
# ScadaBridge Central role over OTLP — ScadaBridge#133 (2026-11-12, same day)
|
||
|
||
Closes the last gap from the two runs above: the **Central** role had never exported. This run used the real
|
||
docker rig instead of a throwaway host, because a Central needs the config DB and LDAP.
|
||
|
||
## How it was run
|
||
|
||
- **Build:** the rig image predated Telemetry 0.3.0 (its `ZB.MOM.WW.Telemetry.dll` was `0.2.0+01c6dc1d`). It was
|
||
rebuilt with `docker/deploy.sh` from a clean `main` at `482ac665` (which includes `a633005f`, Telemetry 0.3.0),
|
||
and all 8 nodes came back healthy.
|
||
- **Override:** a temporary compose file (scratch, never committed) set on **central-a and central-b only**
|
||
`ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` and
|
||
`OTEL_METRIC_EXPORT_INTERVAL=10000`. Both centrals were recreated together
|
||
(`docker compose -f docker/docker-compose.yml -f <override> up -d --force-recreate --no-deps central-a central-b`),
|
||
per the rig's pair-restart rule. Sites, Traefik and every other container were left alone.
|
||
- **Activity (through Traefik `:9000`):**
|
||
- 9 inbound calls rejected with 401: no key, or an empty key.
|
||
- 3 authorized `POST /api/SimpleAlarmStatus` (200 `{"WasSuccessful":true,…}`), made with a temporary method-scoped
|
||
API key. All three temporary keys were deleted afterwards (`api-key list` shows 0 `otel133*`).
|
||
- One `deploy instance --id 1492` (`rig-probe-1`, status Success).
|
||
- CLI `template list` / `site list` and UI/health/metrics GETs, on top of the health polling and the site heartbeats
|
||
that already happen.
|
||
|
||
## What arrived (collector, one process per central)
|
||
|
||
```
|
||
central-a (active) RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0", "service.instance.id": "980946b3915d:6", "site.id": "central", "node.role": "Central", "host.name": "980946b3915d"}
|
||
RESOURCE resourceSpans {... identical ...}
|
||
RESOURCE resourceLogs {... identical ...}
|
||
COUNTS {'metric points': 463, 'spans': 321, 'log records': 3305} SEVERITY {'Information': 3046, 'Warning': 259}
|
||
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Central on scadabridge-central-a"}
|
||
central-b (standby) RESOURCE ... "service.instance.id": "d4d8e49df76f:6", "site.id": "central", "node.role": "Central" ... (identical across the three signals)
|
||
COUNTS {'metric points': 421, 'spans': 77, 'log records': 331} SEVERITY {'Information': 311, 'Warning': 20}
|
||
```
|
||
|
||
`service.namespace=ZB.MOM.WW` is present on both. Logs carry `traceId` inside requests, as on the other apps.
|
||
|
||
**Central instruments** (meter `ZB.MOM.WW.ScadaBridge`, last cumulative value on central-a):
|
||
|
||
```
|
||
scadabridge.deployments.applied 1
|
||
scadabridge.inbound_api.requests {method=SimpleAlarmStatus} 3
|
||
scadabridge.inbound_api.requests {method=<unauthorized>} 9
|
||
scadabridge.central.transport.import_sessions.open 0
|
||
+ the shared gauges: site.stream.subscribers.active, site.data_connections.{connected,total},
|
||
store_and_forward.queue.depth, site.alarm_cache.aggregators.active (all 0 on a central)
|
||
```
|
||
|
||
The counts match the activity exactly. The standby (central-b) exports only the observable gauges, because the
|
||
deploy and inbound singletons run on the active node.
|
||
|
||
**Spans on central-a** (standard instrumentation; ScadaBridge has no `ActivitySource` of its own):
|
||
`POST /scadabridge.centralcontrol.v1.CentralControlService/{Heartbeat 133, SubmitNotification 94, IngestAuditEvents 24, ReportSiteHealth 10}`,
|
||
`POST /management` 16, `POST /auth/token` 16, `POST /api/{methodName}` 12, `GET /api/internal/deployments/{deploymentId}/config` 1,
|
||
the gRPC client spans `SiteCommandService/ExecuteRoute` 3 + `ExecuteLifecycle` 1 (the inbound route to the site
|
||
and the deploy), `GET /health/{active,ready}`, and the Blazor `Route / -> …CentralUI…Dashboard`.
|
||
|
||
**Allowlist:** `SiteServiceRegistration.ObservedMeters` names `ZB.MOM.WW.ScadaBridge` (which holds every central and
|
||
site instrument) and the LocalDb meter, and the repository creates no other `Meter` or `ActivitySource`. **No defect.**
|
||
|
||
## Restored
|
||
|
||
Both centrals were recreated together again on the default compose with no override (0 telemetry env vars). Then:
|
||
- all 8 nodes `/health/ready` 200, one active per pair (central-a, site-{a,b,c}-a 200; their partners 503);
|
||
- central `memberCount 2, unreachableCount 0`;
|
||
- Traefik `/health/active` 200;
|
||
- negative control: the new central processes (`a596a23d9a12`, `1b6fd48c6785`) have **0** lines in
|
||
`metrics/traces/logs.jsonl`.
|
||
|
||
The rig stays on the rebuilt `482ac665` image.
|
||
|
||
## Follow-ups filed (not fixed here)
|
||
|
||
- **scadaproj#56:** the Serilog OTLP sink's own `LogsService/Export` calls are traced as gRPC client spans in
|
||
**all four apps** (scadabridge 127, mxgateway 13, historian-gateway 10, otopcua 6 at the collector). The SDK's
|
||
trace and metric exporters suppress their own instrumentation, and `Serilog.Sinks.OpenTelemetry` does not. This
|
||
is noise, not a loop (0 log records about it). Fix in `ZB.MOM.WW.Telemetry` with an instrumentation filter. **Fixed in 0.3.1, same day — see below.**
|
||
- **ScadaBridge#134:** rig images report `service.version = "1.0.0"` with no `+<sha>`, because `.dockerignore`
|
||
excludes `.git/`. Fix by passing `SourceRevisionId` as a build arg. Wonder deploys, published from a checkout, are
|
||
unaffected.
|
||
|
||
---
|
||
|
||
# OTLP export calls no longer traced — scadaproj#56 (2026-11-12, same day)
|
||
|
||
## Fix: `ZB.MOM.WW.Telemetry(.Serilog)` 0.3.1 — published
|
||
|
||
The SDK's OTLP exporters suppress their own instrumentation. `Serilog.Sinks.OpenTelemetry` does not, so
|
||
`AddGrpcClientInstrumentation()` traced every log batch it sent. `AddZbTelemetry` now classifies an outgoing call
|
||
with `ZbOtlpExportCallFilter.IsExportCall`: any `/opentelemetry.proto.collector.*` path on any host, or
|
||
OTLP/HTTP `/v1/{traces,metrics,logs}` on the configured `OtlpEndpoint`'s host:port.
|
||
|
||
- **HttpClient:** `FilterHttpRequestMessage` returns `false` for an export call.
|
||
- **gRPC client:** `GrpcClientTraceInstrumentationOptions` has no `Filter` in `1.15.1-beta.1`. The
|
||
`EnrichWithHttpRequestMessage` callback runs synchronously in the instrumentation's start handler, before
|
||
`HttpClient.SendAsync` (checked in the contrib source of that tag). For an export call it clears
|
||
`ActivityTraceFlags.Recorded` and `IsAllDataRequested`, so the export processor skips the span and the child
|
||
HTTP activity is unsampled under the default parent-based sampler.
|
||
- A processor that drops spans in `OnEnd` was rejected. It only works when it is registered before every exporter,
|
||
and an app could add an exporter first. The instrumentation hooks do not depend on processor order.
|
||
|
||
Published from scadaproj `main` `85654d7` (the commit in the nuspec). SHA256, feed download = local pack:
|
||
`ZB.MOM.WW.Telemetry.0.3.1.nupkg` `baf6661d03802b3781ff9f95cb2a15477a2bcfda8c26b79c0c6e850eec1480a3`,
|
||
`ZB.MOM.WW.Telemetry.Serilog.0.3.1.nupkg` `188fb0d3cade77cb8f84458fb5c5770f88ad9b4a007b897f895ad19c476e4931`.
|
||
Registration lists `0.1.0, 0.2.0, 0.3.0, 0.3.1` for both ids.
|
||
|
||
**Tests (54 + 40, Release):**
|
||
- `OtlpExportCallFilterTests`: 16 cases. The positives are the three collector services, a collector path on
|
||
another host, and `/v1/*` on the endpoint. The negatives are ScadaBridge→site, HG→historian and mxgw paths, REST,
|
||
a `collectorx` look-alike, `/v1/logs` on another host or port, and relative URIs.
|
||
- `Otlp_export_calls_are_not_traced_but_the_apps_own_calls_are`: a host runs against the loopback OTLP receiver.
|
||
The positive controls are an app gRPC call (`zb.test.v1.AppService/Ping`) and an HttpClient call, both traced. The
|
||
negatives are a real Serilog-sink log batch (the test asserts it arrived), an HttpClient `POST /v1/logs` to the
|
||
endpoint, and a collector path over HttpClient; none is traced. The receiver's own ASP.NET Core server spans land
|
||
in the same export because the instrumentation listens process-wide. So the assertions key on client-span markers:
|
||
the protobuf span name for gRPC and `url.full` for HttpClient.
|
||
- Mutations: removing the gRPC un-record fails the test (*"LogsService/Export call was traced"*). Removing the
|
||
HttpClient filter fails it (`url.full …/v1/logs` found). Making `IsExportCall` always true fails it too
|
||
(*"positive control: the app's gRPC call was not traced"*). It passed 5 of 5 repeated runs.
|
||
|
||
## Apps (pushed to their default branch; no wonder-box deploys)
|
||
|
||
| App | Commit | Pin | Tests (Debug, against 0.3.1 — `project.assets.json` confirms it on every project) |
|
||
|---|---|---|---|
|
||
| OtOpcUa | `master` `f3178c5f` | `Directory.Packages.props` | `Host.IntegrationTests` 325 passed / 2 skipped |
|
||
| MxAccessGateway | `main` `453fe015` | `Server.csproj` | NonWindows solution: 1446 + 5 passed (28 skipped) |
|
||
| HistorianGateway | `main` `dffb5e2` | `Server.csproj` | 186 + 206 + 1414 passed (76 skipped) |
|
||
| ScadaBridge | `main` `95584e2f` | `Directory.Packages.props` | `Host.Tests` 721 passed |
|
||
|
||
## Collector before/after
|
||
|
||
The collector-wide count of `LogsService/Export` spans (`zb-otel-collector`, all runs to date) was the same before
|
||
and after both probe runs: **scadabridge 146, mxgateway 13, historian-gateway 10, otopcua 6**. The issue reported
|
||
127 for scadabridge; the ScadaBridge#133 centrals added more before the rig was restored. Each probe ran as a
|
||
throwaway local process on 0.3.1 with `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317`,
|
||
`OTEL_METRIC_EXPORT_INTERVAL=10000` and `site.id=otel-probe-56`, counted from its own start time:
|
||
|
||
| Run | Log export batches / records arrived | `*Service/Export` spans | App client spans still arriving |
|
||
|---|---|---|---|
|
||
| HistorianGateway `Development`, `:15220`/`:15221`, scratch stores, health/metrics GETs + 3 `HistorianStatus/Probe` | 15 / 168 | **0** | `HistoryService/GetInterfaceVersion` 14 (GrpcNetClient), `historian.Connect` 14, server spans |
|
||
| ScadaBridge **Site** `95584e2f`, own site id, own 1-node mesh `localhost:18082`, gRPC `:18083`, metrics `:18084`, central `localhost:18999` unreachable, scratch LocalDb/audit/secrets | 27 / 224 | **0** | `CentralControlService/Heartbeat` 17, `ReconcileSite` 2, `ReportSiteHealth` 2 (GrpcNetClient), server spans |
|
||
|
||
So the log sink exported throughout, the gRPC client instrumentation stayed on, and no export call was traced.
|
||
mxgw is proven by the library tests and its suite only; no windev scratch run was made. The docker rigs were not
|
||
touched.
|
||
|
||
The wonder boxes (#54) still run their earlier builds. They will emit `LogsService/Export` spans once a 0.3.0 build
|
||
is deployed, unless 0.3.1 is deployed instead.
|
||
|