|
|
|
@@ -0,0 +1,173 @@
|
|
|
|
|
# OTLP export, end to end — evidence (2026-11-12, scadaproj#13)
|
|
|
|
|
|
|
|
|
|
First run of the family's OTLP export path against a real collector. Before this, every app had
|
|
|
|
|
the `<App>:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` wiring but nothing had ever received from
|
|
|
|
|
it, and OtOpcUa's spans had no exporter anywhere.
|
|
|
|
|
|
|
|
|
|
**Result:** OTLP **metrics + traces** proven for **OtOpcUa, HistorianGateway, ScadaBridge (Site
|
|
|
|
|
role) and MxAccessGateway**. One real defect found and fixed (OtOpcUa Galaxy driver telemetry was
|
|
|
|
|
never exported, lmxopcua#892). No Telemetry library change was needed; `0.2.1` stays unpublished.
|
|
|
|
|
|
|
|
|
|
## Collector
|
|
|
|
|
|
|
|
|
|
`zb-otel-collector` on the shared Docker host `10.100.0.35` — `otel/opentelemetry-collector-contrib:0.162.0`,
|
|
|
|
|
OTLP gRPC `:4317` + HTTP `:4318`, Prometheus exporter `:8889`, health `:13133`; `debug` + `file`
|
|
|
|
|
exporters for proof. Source and runbook: [`infra/otel-collector/`](../../infra/otel-collector/). The
|
|
|
|
|
four ports were free on the host before deploy, and no other container was touched.
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
$ curl -s http://10.100.0.35:13133/
|
|
|
|
|
{"status":"Server available","upSince":"2026-11-13T01:24:13.274419432Z",...}
|
|
|
|
|
$ curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d '{}' http://10.100.0.35:4318/v1/traces
|
|
|
|
|
200
|
|
|
|
|
windev> Test-NetConnection 10.100.0.35 -Port 4317 -InformationLevel Quiet
|
|
|
|
|
True
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## How each app was run
|
|
|
|
|
|
|
|
|
|
No shared rig container was recreated or reconfigured. Each app ran as a throwaway process with
|
|
|
|
|
its own state, `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317` (gRPC) and
|
|
|
|
|
`OTEL_METRIC_EXPORT_INTERVAL=10000`.
|
|
|
|
|
|
|
|
|
|
| App | Build | Telemetry pin | How |
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
| OtOpcUa | `master` `aaaa55e2` + the #892 fix (`0b90379f`) | 0.1.0 | Local `dotnet` host on the Mac: fused `admin,driver` single node in its own Akka mesh (`localhost:14053`), its own throwaway ConfigDb `OtOpcUaOtel13` on the docker-dev SQL (EF-migrated, one `ServerCluster`/`ClusterNode` row, dropped afterwards), HTTP `:19100`, OPC UA `:14840`. Activity: one `POST /api/deployments` + health/metrics GETs. |
|
|
|
|
|
| HistorianGateway | `main` `b95bae3` | 0.1.0 | Local `dotnet` host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores. Activity: status RPCs via `grpcurl` (the historian itself is unreachable without the VPN, so they fail — which still produces spans) + dashboard/health GETs. |
|
|
|
|
|
| ScadaBridge | `main` `139a43fb` | 0.2.0 | Local `dotnet` **Site** host, own site id `otel-probe`, own 1-node mesh (`localhost:18082`), scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable. The docker rig was in active use (ScadaBridge#46/#132; containers recreated minutes earlier), so it was not touched. |
|
|
|
|
|
| MxAccessGateway | deployed windev Server `447070a5` (scratch copy) | 0.1.0 | Copy of `C:\publish\mxaccessgw\Server-20261112-447070a5` run as a separate process on `:15120`/`:15130`, `Staging`, alarms off, scratch auth/secrets stores, `Worker:ExecutablePath` pointed at a non-existent `OtelProbeNoWorker.exe` so its `OrphanWorkerTerminator` (which matches by image name) could not match the live service's x86 workers. Live `MxAccessGw` stayed `Running`, `:5130/health/ready` = 200 throughout, worker PIDs `14700,22108` unchanged before/after; the scratch copy was deleted. |
|
|
|
|
|
|
|
|
|
|
## What arrived (collector `otelsum.py` output, trimmed)
|
|
|
|
|
|
|
|
|
|
Standard instrumentation scopes that every app sent are listed once at the end.
|
|
|
|
|
|
|
|
|
|
### OtOpcUa — metrics + traces
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
RESOURCE otopcua {"service.name": "otopcua", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:48096", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
|
|
|
|
|
METRICS otopcua ZB.MOM.WW.OtOpcUa 3 ['otopcua.deploy.applied', 'otopcua.deploy.apply.duration', 'otopcua.redundancy.service_level_change']
|
|
|
|
|
METRICS otopcua ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
|
|
|
|
|
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.deploy.apply')
|
|
|
|
|
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.opcua.address_space_rebuild')
|
|
|
|
|
SPANS 1 ('otopcua', 'Microsoft.AspNetCore', 'POST /api/deployments')
|
|
|
|
|
SPANS 5 ('otopcua', 'Microsoft.AspNetCore', 'GET /health/ready')
|
|
|
|
|
SPANS 3 ('otopcua', 'Microsoft.AspNetCore', 'GET /metrics')
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.deploy.apply", "traceId": "e5a579168f8c4629b25af4d3e4f2cb07", "spanId": "4f12d04ada38ea41", "kind": 1,
|
|
|
|
|
"attributes": [{"key": "otopcua.deployment_id", "value": {"stringValue": "e8b8c010d50144ccb371a91defceb6d9"}}, {"key": "otopcua.node_id", "value": {"stringValue": "localhost:14053"}},
|
|
|
|
|
{"key": "otopcua.revision", "value": {"stringValue": "ed37612b…7bba"}}, {"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
|
|
|
|
|
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.opcua.address_space_rebuild", "traceId": "f6d8c238b2399cce0928de84691a0106", "spanId": "305ec5a9012a799d", "kind": 1,
|
|
|
|
|
"attributes": [{"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
OtOpcUa's tracing is no longer a no-op once `Exporter=Otlp` is set: its own spans reach a backend
|
|
|
|
|
with their attributes. Note that `address_space_rebuild` is a separate trace from `deploy.apply`
|
|
|
|
|
(no parent — the context does not cross the Akka actor hop); the two join on
|
|
|
|
|
`otopcua.correlation_id`.
|
|
|
|
|
|
|
|
|
|
### HistorianGateway — metrics + traces
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
RESOURCE historian-gateway {"service.name": "historian-gateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:49942", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
|
|
|
|
|
METRICS historian-gateway ZB.MOM.WW.HistorianGateway 10 ['historian_gateway.grpc.request.duration', 'historian_gateway.grpc.requests', 'historian_gateway.historian.connection.up', 'historian_gateway.historian.event_session_pool.lease_waiters', 'historian_gateway.historian.event_session_pool.warm_sessions', 'historian_gateway.historian.session_pool.lease_waiters', 'historian_gateway.historian.session_pool.warm_sessions', 'historian_gateway.redundancy.members.healthy', 'historian_gateway.session_pool.lease.wait.duration', 'historian_gateway.store_forward.queue.depth']
|
|
|
|
|
SPANS 8 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.Connect')
|
|
|
|
|
SPANS 2 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.pool.lease')
|
|
|
|
|
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/Probe')
|
|
|
|
|
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/GetConnectionStatus')
|
|
|
|
|
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'galaxy_repository.v1.GalaxyRepository/GetLastDeployTime')
|
|
|
|
|
SPANS 8 ('historian-gateway', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'HistoryService/GetInterfaceVersion')
|
|
|
|
|
SPANS 4 ('historian-gateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.HistorianGateway.Server.Dashboard.Components.Pages.Home')
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`historian.Connect` spans carry `server.address`/`server.port` and `error.type=…HistorianTransportException`
|
|
|
|
|
(no VPN); the ones a request triggered are parented under its gRPC request span, the pool warm-up ones are roots. Prometheus exporter, same data:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
historian_gateway_grpc_requests_total{job="ZB.MOM.WW/historian-gateway",method="Probe",outcome="ok",service="HistorianStatus",service_name="historian-gateway",...} 1
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### ScadaBridge (Site) — metrics + traces
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
RESOURCE scadabridge {"service.name": "scadabridge", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:50923", "host.name": "Josephs-MacBook-Pro", "site.id": "otel-probe", "node.role": "Site", "_sdk": "1.15.3"}
|
|
|
|
|
METRICS scadabridge ZB.MOM.WW.ScadaBridge 6 ['scadabridge.central.transport.import_sessions.open', 'scadabridge.site.alarm_cache.aggregators.active', 'scadabridge.site.data_connections.connected', 'scadabridge.site.data_connections.total', 'scadabridge.site.stream.subscribers.active', 'scadabridge.store_and_forward.queue.depth']
|
|
|
|
|
METRICS scadabridge ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
|
|
|
|
|
SPANS 12 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/Heartbeat')
|
|
|
|
|
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReportSiteHealth')
|
|
|
|
|
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReconcileSite')
|
|
|
|
|
SPANS 4 ('scadabridge', 'Microsoft.AspNetCore', 'GET /health/ready')
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The only app that sends `site.id` / `node.role`. ScadaBridge has no `ActivitySource` of its own,
|
|
|
|
|
so its spans are standard instrumentation only. **Central role not run** → ScadaBridge#133.
|
|
|
|
|
|
|
|
|
|
### MxAccessGateway — metrics + traces
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
RESOURCE mxgateway {"service.name": "mxgateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "DESKTOP-6JL3KKO:8720", "host.name": "DESKTOP-6JL3KKO", "_sdk": "1.15.3"}
|
|
|
|
|
METRICS mxgateway ZB.MOM.WW.MxGateway 5 ['mxgateway.alarms.provider_mode', 'mxgateway.events.grpc_stream_queue.depth', 'mxgateway.events.worker_queue.depth', 'mxgateway.sessions.open', 'mxgateway.workers.running']
|
|
|
|
|
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /')
|
|
|
|
|
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /health/ready')
|
|
|
|
|
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /metrics')
|
|
|
|
|
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.MxGateway.Server.Dashboard.Components.Pages.DashboardHome')
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Only the observable gauges appear: the counters/histograms (`commands.*`, `workers.startup.duration`, …)
|
|
|
|
|
record nothing without a worker session, and the x86 worker was deliberately not run. The gateway
|
|
|
|
|
has no `ActivitySource` of its own (spans are standard instrumentation only). Its gRPC port is not
|
|
|
|
|
reachable from off-box (windev firewall), so the traffic was generated on-box.
|
|
|
|
|
|
|
|
|
|
### Standard instrumentation (all four)
|
|
|
|
|
|
|
|
|
|
`System.Runtime` (19 `dotnet.*`), `OpenTelemetry.Instrumentation.Process` (5 `process.*`),
|
|
|
|
|
`System.Net.Http` (4 `http.client.*`), `Microsoft.AspNetCore.Hosting` (`http.server.request.duration`,
|
|
|
|
|
`http.server.active_requests`), `Microsoft.AspNetCore.Server.Kestrel`, `.Routing`, `.MemoryPool`;
|
|
|
|
|
plus `.Authentication`/`.Authorization`/`.Components` on the apps with a Blazor dashboard.
|
|
|
|
|
|
|
|
|
|
## Defect found and fixed
|
|
|
|
|
|
|
|
|
|
**lmxopcua#892 — the Galaxy driver's telemetry was never exported.** `ZbTelemetryOptions.Meters`
|
|
|
|
|
and `.ActivitySources` are strict allowlists. OtOpcUa named only `ZB.MOM.WW.OtOpcUa` and the
|
|
|
|
|
LocalDb meter, while the in-process Galaxy driver opens its spans (`galaxy.subscribe_bulk`,
|
|
|
|
|
`galaxy.unsubscribe_bulk`, `galaxy.stream_events`, `galaxy.write`, `galaxy.get_hierarchy`) on
|
|
|
|
|
ActivitySource `ZB.MOM.WW.OtOpcUa.Driver.Galaxy` and records the event pump's `galaxy.events.*`
|
|
|
|
|
counters on a Meter of the same name. All of it was silently dropped from `/metrics` and OTLP.
|
|
|
|
|
|
|
|
|
|
Fix: OtOpcUa `master` `db86b996` (commit `0b90379f`) — a public `GalaxyDriverTelemetryNames` in
|
|
|
|
|
the driver; `ObservabilityExtensions.ObservedMeters` / `ObservedActivitySources` name it.
|
|
|
|
|
`ObservabilityAllowlistTests` assert on the listener side (span sampled, instrument enabled) with an
|
|
|
|
|
unregistered-name negative control. Dropping the Galaxy entries fails 3 of 8 tests, so the tests can
|
|
|
|
|
catch the regression. Galaxy driver suite 584/584. Not deployed. A live `galaxy.*` span was not
|
|
|
|
|
driven: that would need a Galaxy driver deployment against a gateway.
|
|
|
|
|
|
|
|
|
|
The allowlists of the other apps were checked against every `new Meter(` / `new ActivitySource(`
|
|
|
|
|
in their source and in the hosted libraries. ScadaBridge (`ZB.MOM.WW.ScadaBridge`, LocalDb),
|
|
|
|
|
HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGateway
|
|
|
|
|
(`ZB.MOM.WW.MxGateway`) are complete. Of the hosted libraries, only LocalDb defines a meter.
|
|
|
|
|
|
|
|
|
|
## Checked, no defect
|
|
|
|
|
|
|
|
|
|
- **Protocol/endpoint:** every app uses the library's `AddOtlpExporter` default (gRPC). `http://host:4317`
|
|
|
|
|
works for all four. No checked-in config carries a scheme-less endpoint.
|
|
|
|
|
- **Prometheus stays up under Otlp:** `/metrics` answered 200 on every app with `Exporter=Otlp` (`http.response.status_code=200` on each app's exported `GET /metrics` span)
|
|
|
|
|
(the overlay is additive, as documented).
|
|
|
|
|
- **Resource:** `service.name`, `service.namespace=ZB.MOM.WW`, `service.instance.id`, `host.name`
|
|
|
|
|
present on all four.
|
|
|
|
|
|
|
|
|
|
## Gaps filed (not fixed here)
|
|
|
|
|
|
|
|
|
|
- **scadaproj#55:** `service.version` is absent on all four apps; `site.id`/`node.role` are absent
|
|
|
|
|
on OtOpcUa, MxAccessGateway and HistorianGateway. No app exports logs over OTLP, because every
|
|
|
|
|
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
|
|
|
|
|
(`logs.jsonl` stayed empty).
|
|
|
|
|
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
|
|
|
|
|
- **scadaproj#54:** collector on the wonder tiers.
|
|
|
|
|
|
|
|
|
|
## Telemetry 0.2.1: not published
|
|
|
|
|
|
|
|
|
|
No finding needed a library change. The fixes and gaps above are all on the app side. `0.2.1`'s
|
|
|
|
|
scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also
|
|
|
|
|
need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays
|
|
|
|
|
built-but-unpublished until a pin bump wants it.
|