feat(infra): shared dev OTel collector; OTLP proven end to end for all four apps (#13)

- infra/otel-collector/: zb-otel-collector (otelcol-contrib 0.162.0) on 10.100.0.35 —
  OTLP gRPC :4317 / HTTP :4318, debug + file exporters, Prometheus re-export :8889,
  health :13133; README runbook + otelsum.py summariser. Deployed.
- components/observability/2026-11-12-otlp-end-to-end-evidence.md: metrics + traces
  received from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway;
  OtOpcUa Galaxy driver allowlist defect fixed in lmxopcua#892 (db86b996).
- GAPS.md, current-state banners, index row, Telemetry CLAUDE.md: #13 closed; follow-ups
  #54 (wonder-tier collector), #55 (resource identity + logs over OTLP), ScadaBridge#133
  (Central role). Telemetry 0.2.1 deliberately left unpublished (no lib change needed).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 20:48:36 -05:00
parent f98ba08910
commit e1a2df83f7
12 changed files with 409 additions and 7 deletions
@@ -0,0 +1,173 @@
# OTLP export, end to end — evidence (2026-11-12, scadaproj#13)
First run of the family's OTLP export path against a real collector. Before this, every app had
the `<App>:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` wiring but nothing had ever received from
it, and OtOpcUa's spans had no exporter anywhere.
**Result:** OTLP **metrics + traces** proven for **OtOpcUa, HistorianGateway, ScadaBridge (Site
role) and MxAccessGateway**. One real defect found and fixed (OtOpcUa Galaxy driver telemetry was
never exported, lmxopcua#892). No Telemetry library change was needed; `0.2.1` stays unpublished.
## Collector
`zb-otel-collector` on the shared Docker host `10.100.0.35` — `otel/opentelemetry-collector-contrib:0.162.0`,
OTLP gRPC `:4317` + HTTP `:4318`, Prometheus exporter `:8889`, health `:13133`; `debug` + `file`
exporters for proof. Source and runbook: [`infra/otel-collector/`](../../infra/otel-collector/). The
four ports were free on the host before deploy, and no other container was touched.
```
$ curl -s http://10.100.0.35:13133/
{"status":"Server available","upSince":"2026-11-13T01:24:13.274419432Z",...}
$ curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d '{}' http://10.100.0.35:4318/v1/traces
200
windev> Test-NetConnection 10.100.0.35 -Port 4317 -InformationLevel Quiet
True
```
## How each app was run
No shared rig container was recreated or reconfigured. Each app ran as a throwaway process with
its own state, `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317` (gRPC) and
`OTEL_METRIC_EXPORT_INTERVAL=10000`.
| App | Build | Telemetry pin | How |
|---|---|---|---|
| OtOpcUa | `master` `aaaa55e2` + the #892 fix (`0b90379f`) | 0.1.0 | Local `dotnet` host on the Mac: fused `admin,driver` single node in its own Akka mesh (`localhost:14053`), its own throwaway ConfigDb `OtOpcUaOtel13` on the docker-dev SQL (EF-migrated, one `ServerCluster`/`ClusterNode` row, dropped afterwards), HTTP `:19100`, OPC UA `:14840`. Activity: one `POST /api/deployments` + health/metrics GETs. |
| HistorianGateway | `main` `b95bae3` | 0.1.0 | Local `dotnet` host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores. Activity: status RPCs via `grpcurl` (the historian itself is unreachable without the VPN, so they fail — which still produces spans) + dashboard/health GETs. |
| ScadaBridge | `main` `139a43fb` | 0.2.0 | Local `dotnet` **Site** host, own site id `otel-probe`, own 1-node mesh (`localhost:18082`), scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable. The docker rig was in active use (ScadaBridge#46/#132; containers recreated minutes earlier), so it was not touched. |
| MxAccessGateway | deployed windev Server `447070a5` (scratch copy) | 0.1.0 | Copy of `C:\publish\mxaccessgw\Server-20261112-447070a5` run as a separate process on `:15120`/`:15130`, `Staging`, alarms off, scratch auth/secrets stores, `Worker:ExecutablePath` pointed at a non-existent `OtelProbeNoWorker.exe` so its `OrphanWorkerTerminator` (which matches by image name) could not match the live service's x86 workers. Live `MxAccessGw` stayed `Running`, `:5130/health/ready` = 200 throughout, worker PIDs `14700,22108` unchanged before/after; the scratch copy was deleted. |
## What arrived (collector `otelsum.py` output, trimmed)
Standard instrumentation scopes that every app sent are listed once at the end.
### OtOpcUa — metrics + traces
```
RESOURCE otopcua {"service.name": "otopcua", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:48096", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS otopcua ZB.MOM.WW.OtOpcUa 3 ['otopcua.deploy.applied', 'otopcua.deploy.apply.duration', 'otopcua.redundancy.service_level_change']
METRICS otopcua ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.deploy.apply')
SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.opcua.address_space_rebuild')
SPANS 1 ('otopcua', 'Microsoft.AspNetCore', 'POST /api/deployments')
SPANS 5 ('otopcua', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 3 ('otopcua', 'Microsoft.AspNetCore', 'GET /metrics')
```
```json
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.deploy.apply", "traceId": "e5a579168f8c4629b25af4d3e4f2cb07", "spanId": "4f12d04ada38ea41", "kind": 1,
"attributes": [{"key": "otopcua.deployment_id", "value": {"stringValue": "e8b8c010d50144ccb371a91defceb6d9"}}, {"key": "otopcua.node_id", "value": {"stringValue": "localhost:14053"}},
{"key": "otopcua.revision", "value": {"stringValue": "ed37612b…7bba"}}, {"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.opcua.address_space_rebuild", "traceId": "f6d8c238b2399cce0928de84691a0106", "spanId": "305ec5a9012a799d", "kind": 1,
"attributes": [{"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]}
```
OtOpcUa's tracing is no longer a no-op once `Exporter=Otlp` is set: its own spans reach a backend
with their attributes. Note that `address_space_rebuild` is a separate trace from `deploy.apply`
(no parent — the context does not cross the Akka actor hop); the two join on
`otopcua.correlation_id`.
### HistorianGateway — metrics + traces
```
RESOURCE historian-gateway {"service.name": "historian-gateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:49942", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"}
METRICS historian-gateway ZB.MOM.WW.HistorianGateway 10 ['historian_gateway.grpc.request.duration', 'historian_gateway.grpc.requests', 'historian_gateway.historian.connection.up', 'historian_gateway.historian.event_session_pool.lease_waiters', 'historian_gateway.historian.event_session_pool.warm_sessions', 'historian_gateway.historian.session_pool.lease_waiters', 'historian_gateway.historian.session_pool.warm_sessions', 'historian_gateway.redundancy.members.healthy', 'historian_gateway.session_pool.lease.wait.duration', 'historian_gateway.store_forward.queue.depth']
SPANS 8 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.Connect')
SPANS 2 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.pool.lease')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/Probe')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/GetConnectionStatus')
SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'galaxy_repository.v1.GalaxyRepository/GetLastDeployTime')
SPANS 8 ('historian-gateway', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'HistoryService/GetInterfaceVersion')
SPANS 4 ('historian-gateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.HistorianGateway.Server.Dashboard.Components.Pages.Home')
```
`historian.Connect` spans carry `server.address`/`server.port` and `error.type=…HistorianTransportException`
(no VPN); the ones a request triggered are parented under its gRPC request span, the pool warm-up ones are roots. Prometheus exporter, same data:
```
historian_gateway_grpc_requests_total{job="ZB.MOM.WW/historian-gateway",method="Probe",outcome="ok",service="HistorianStatus",service_name="historian-gateway",...} 1
```
### ScadaBridge (Site) — metrics + traces
```
RESOURCE scadabridge {"service.name": "scadabridge", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:50923", "host.name": "Josephs-MacBook-Pro", "site.id": "otel-probe", "node.role": "Site", "_sdk": "1.15.3"}
METRICS scadabridge ZB.MOM.WW.ScadaBridge 6 ['scadabridge.central.transport.import_sessions.open', 'scadabridge.site.alarm_cache.aggregators.active', 'scadabridge.site.data_connections.connected', 'scadabridge.site.data_connections.total', 'scadabridge.site.stream.subscribers.active', 'scadabridge.store_and_forward.queue.depth']
METRICS scadabridge ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth']
SPANS 12 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/Heartbeat')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReportSiteHealth')
SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReconcileSite')
SPANS 4 ('scadabridge', 'Microsoft.AspNetCore', 'GET /health/ready')
```
The only app that sends `site.id` / `node.role`. ScadaBridge has no `ActivitySource` of its own,
so its spans are standard instrumentation only. **Central role not run** → ScadaBridge#133.
### MxAccessGateway — metrics + traces
```
RESOURCE mxgateway {"service.name": "mxgateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "DESKTOP-6JL3KKO:8720", "host.name": "DESKTOP-6JL3KKO", "_sdk": "1.15.3"}
METRICS mxgateway ZB.MOM.WW.MxGateway 5 ['mxgateway.alarms.provider_mode', 'mxgateway.events.grpc_stream_queue.depth', 'mxgateway.events.worker_queue.depth', 'mxgateway.sessions.open', 'mxgateway.workers.running']
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /health/ready')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /metrics')
SPANS 7 ('mxgateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.MxGateway.Server.Dashboard.Components.Pages.DashboardHome')
```
Only the observable gauges appear: the counters/histograms (`commands.*`, `workers.startup.duration`, …)
record nothing without a worker session, and the x86 worker was deliberately not run. The gateway
has no `ActivitySource` of its own (spans are standard instrumentation only). Its gRPC port is not
reachable from off-box (windev firewall), so the traffic was generated on-box.
### Standard instrumentation (all four)
`System.Runtime` (19 `dotnet.*`), `OpenTelemetry.Instrumentation.Process` (5 `process.*`),
`System.Net.Http` (4 `http.client.*`), `Microsoft.AspNetCore.Hosting` (`http.server.request.duration`,
`http.server.active_requests`), `Microsoft.AspNetCore.Server.Kestrel`, `.Routing`, `.MemoryPool`;
plus `.Authentication`/`.Authorization`/`.Components` on the apps with a Blazor dashboard.
## Defect found and fixed
**lmxopcua#892 — the Galaxy driver's telemetry was never exported.** `ZbTelemetryOptions.Meters`
and `.ActivitySources` are strict allowlists. OtOpcUa named only `ZB.MOM.WW.OtOpcUa` and the
LocalDb meter, while the in-process Galaxy driver opens its spans (`galaxy.subscribe_bulk`,
`galaxy.unsubscribe_bulk`, `galaxy.stream_events`, `galaxy.write`, `galaxy.get_hierarchy`) on
ActivitySource `ZB.MOM.WW.OtOpcUa.Driver.Galaxy` and records the event pump's `galaxy.events.*`
counters on a Meter of the same name. All of it was silently dropped from `/metrics` and OTLP.
Fix: OtOpcUa `master` `db86b996` (commit `0b90379f`) — a public `GalaxyDriverTelemetryNames` in
the driver; `ObservabilityExtensions.ObservedMeters` / `ObservedActivitySources` name it.
`ObservabilityAllowlistTests` assert on the listener side (span sampled, instrument enabled) with an
unregistered-name negative control. Dropping the Galaxy entries fails 3 of 8 tests, so the tests can
catch the regression. Galaxy driver suite 584/584. Not deployed. A live `galaxy.*` span was not
driven: that would need a Galaxy driver deployment against a gateway.
The allowlists of the other apps were checked against every `new Meter(` / `new ActivitySource(`
in their source and in the hosted libraries. ScadaBridge (`ZB.MOM.WW.ScadaBridge`, LocalDb),
HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGateway
(`ZB.MOM.WW.MxGateway`) are complete. Of the hosted libraries, only LocalDb defines a meter.
## Checked, no defect
- **Protocol/endpoint:** every app uses the library's `AddOtlpExporter` default (gRPC). `http://host:4317`
works for all four. No checked-in config carries a scheme-less endpoint.
- **Prometheus stays up under Otlp:** `/metrics` answered 200 on every app with `Exporter=Otlp` (`http.response.status_code=200` on each app's exported `GET /metrics` span)
(the overlay is additive, as documented).
- **Resource:** `service.name`, `service.namespace=ZB.MOM.WW`, `service.instance.id`, `host.name`
present on all four.
## Gaps filed (not fixed here)
- **scadaproj#55:** `service.version` is absent on all four apps; `site.id`/`node.role` are absent
on OtOpcUa, MxAccessGateway and HistorianGateway. No app exports logs over OTLP, because every
`AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates
(`logs.jsonl` stayed empty).
- **ScadaBridge#133:** OTLP from the Central role is not yet proven.
- **scadaproj#54:** collector on the wonder tiers.
## Telemetry 0.2.1: not published
No finding needed a library change. The fixes and gaps above are all on the app side. `0.2.1`'s
scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also
need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays
built-but-unpublished until a pin bump wants it.
+15 -4
View File
@@ -22,7 +22,13 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p
> listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the
> "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously
> contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there
> (deploying an OTel collector → #13; further ScadaBridge instruments → ScadaBridge#81) remain.
> (further ScadaBridge instruments → ScadaBridge#81; collector on the wonder tiers → #54) remain.
>
> **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector`
> on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived
> from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway. One defect fixed: OtOpcUa's
> Galaxy driver meter/source were missing from the allowlists (lmxopcua#892). Evidence:
> [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
## Divergence vs spec
@@ -256,7 +262,12 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change
**Still open (not code — operational/future):**
- **OTLP is opt-in but unexercised** (→ #13) until an OTel collector endpoint is deployed and the
`<App>:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` config is set. The wiring is in place; only a
collector is missing.
- ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against
`zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central →
ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md).
- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still
Prometheus-only. → #54
- **Resource identity + logs-over-OTLP gaps** found by #13: `service.version` absent on all four apps;
`site.id`/`node.role` absent on OtOpcUa, MxAccessGateway, HistorianGateway; no app passes the exporter
settings to `AddZbSerilog`, so logs never travel over OTLP. → #55
- **Further ScadaBridge instruments** beyond the first 4 are additive future work (not blocking). → ScadaBridge#81
@@ -1,5 +1,11 @@
# Observability — current state: MxAccessGateway
> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. Since then the gateway
> exports its metrics via `AddZbTelemetry`; the meter is `ZB.MOM.WW.MxGateway` (seconds), and logging is Serilog.
> OTLP export of **metrics and traces** is proven end to end (scratch copy of the deployed windev build; the
> `mxgateway.*` gauges and ASP.NET Core spans arrived). The gateway still has no `ActivitySource` of its own.
> Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
Repo: `~/Desktop/MxAccessGateway`. Stack: .NET 10 gateway (x64) + .NET 4.8 worker (**x86**);
solution `src/MxGateway.sln`. Telemetry code is concentrated in
`src/ZB.MOM.WW.MxGateway.Server/Metrics/` (instruments) and
@@ -1,5 +1,13 @@
# Observability — current state: OtOpcUa
> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. OtOpcUa now calls
> `AddZbTelemetry` (Resource + OTLP opt-in), and OTLP export of **metrics and traces** is proven end to end
> against the dev collector (`otopcua.deploy.apply`, `otopcua.opcua.address_space_rebuild`, `otopcua.deploy.*`
> metrics arrived). That ends the "tracing no-op" below once `OtOpcUa:Telemetry:Exporter=Otlp` is set. The
> Galaxy driver's `galaxy.*` spans and `galaxy.events.*` counters were missing from the allowlists; fixed in
> lmxopcua#892 (`master` `db86b996`). Still absent from the Resource: `service.version`, `site.id`, `node.role`
> (scadaproj#55). Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`.
Telemetry code lives in two places: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/` (host-side
bootstrap) and `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/` (instruments + enricher).
@@ -1,5 +1,11 @@
# Observability — current state: ScadaBridge
> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. ScadaBridge now
> has the `ZB.MOM.WW.ScadaBridge` meter and `AddZbSerilog`. OTLP export of **metrics and traces** is proven
> end to end for a **Site** node (6 `scadabridge.*` instruments, `localdb.oplog.depth`, gRPC-client spans,
> with `site.id`/`node.role` on the Resource); the Central role is not yet proven (ScadaBridge#133).
> Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md).
Repo: `~/Desktop/ScadaBridge`. Stack: .NET 10, Akka.NET, Docker; solution
`ZB.MOM.WW.ScadaBridge.slnx`. The telemetry posture is split across a dangling OTel package ref
(metrics/traces) and a substantive Serilog setup (logs). All paths relative to repo root.