From e1a2df83f7ee1a0f74203ba334eacc8d9b6192e2 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 12 Nov 2026 20:48:36 -0500 Subject: [PATCH] feat(infra): shared dev OTel collector; OTLP proven end to end for all four apps (#13) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - infra/otel-collector/: zb-otel-collector (otelcol-contrib 0.162.0) on 10.100.0.35 — OTLP gRPC :4317 / HTTP :4318, debug + file exporters, Prometheus re-export :8889, health :13133; README runbook + otelsum.py summariser. Deployed. - components/observability/2026-11-12-otlp-end-to-end-evidence.md: metrics + traces received from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway; OtOpcUa Galaxy driver allowlist defect fixed in lmxopcua#892 (db86b996). - GAPS.md, current-state banners, index row, Telemetry CLAUDE.md: #13 closed; follow-ups #54 (wonder-tier collector), #55 (resource identity + logs over OTLP), ScadaBridge#133 (Central role). Telemetry 0.2.1 deliberately left unpublished (no lib change needed). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- CLAUDE.md | 6 +- ZB.MOM.WW.Telemetry/CLAUDE.md | 4 +- .../2026-11-12-otlp-end-to-end-evidence.md | 173 ++++++++++++++++++ components/observability/GAPS.md | 19 +- .../current-state/mxaccessgw/CURRENT-STATE.md | 6 + .../current-state/otopcua/CURRENT-STATE.md | 8 + .../scadabridge/CURRENT-STATE.md | 6 + infra/otel-collector/.gitignore | 1 + infra/otel-collector/README.md | 69 +++++++ infra/otel-collector/config.yaml | 65 +++++++ infra/otel-collector/docker-compose.yml | 31 ++++ infra/otel-collector/otelsum.py | 28 +++ 12 files changed, 409 insertions(+), 7 deletions(-) create mode 100644 components/observability/2026-11-12-otlp-end-to-end-evidence.md create mode 100644 infra/otel-collector/.gitignore create mode 100644 infra/otel-collector/README.md create mode 100644 infra/otel-collector/config.yaml create mode 100644 infra/otel-collector/docker-compose.yml create mode 100644 infra/otel-collector/otelsum.py diff --git a/CLAUDE.md b/CLAUDE.md index 2f2f6a2..bbfbd16 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -162,7 +162,7 @@ each project's **code-verified current state**, and the **gaps** between. See | Auth (login / identity / authz) | **2026-11-12: `0.5.0` PUBLISHED** (5 pkgs, SHA-verified) — adds passwordless `ILdapGroupLookup.GetUserGroupsAsync` (service account; `Found`/`UserNotFound`/`UserDisabled`/`AmbiguousUser`/`DirectoryUnavailable`/`NotSupported`/`ProviderDisabled`; ScadaBridge#46) and is the first published build of 0.4.2–0.4.4 (typed `ApiKeyPepperUnavailableException`; Sqlite 10.0.13; user-bind identity proof (#40/#50); SQL Server key ids case-exact; `ApiKeyFailure.StoreUnavailable`). scadaproj#51 closed: live-verified vs NAM AD 24/24. **ScadaBridge adopts 0.5.0 (#46); OtOpcUa/mxgw/HistorianGateway/BTDB stay on their pins.** Adopted. Feed carries **`0.4.1`** (2026-10-01; 0.4.0 the same day; opt-in user bind) — **all four apps pin `0.4.1` on their default branches since 2026-11-09** (scadaproj#5: OtOpcUa `1ecd01d3`, ScadaBridge `37fcd296`, mxgw `ace12f81`, HistorianGateway `0d06483`; `BindMode` left at `ServiceAccount`; NOT deployed — wonder boxes still run `0.3.0` builds). History below refers to the earlier `0.3.0` pins. **All four apps pin `0.3.0`** (2026-09-03, all on their default branch and pushed — and all four are DEPLOYED at those tips on DEV vd03 and QA vq03/vq04 since 2026-09-10: OtOpcUa `b8b4c18f`, mxgw `4f931e4`, HistorianGateway `d8d890c`; ScadaBridge already had it on `main` via the `inbound-api-key-shared-store` work, `e45c4af9`). **0.4.0 (2026-10-01) — opt-in LDAP user bind:** adds `LdapOptions.BindMode` (`ServiceAccount` default | `User`), `UserBindTemplate` and `AcceptDomainQualifiedNames`; in `User` mode `LdapAuthService` binds as the signing-in user and reads their own groups on the same connection, so no service account is needed. Bind failures map to `BadCredentials` (bad password, unusable name, or bind result 49) or `GroupLookupFailed` (zero/multiple entries or no groups after a good bind), other bind/connect/search errors fail over to the next endpoint, and `ServiceAccount` stays the default so no existing consumer needs to change. **0.4.1** fixes the review findings: `Username` is canonicalised from the entry's `UserNameAttribute`, empty passwords are refused in both bind modes (a `ServiceAccount`-mode behaviour change), and the docs prefer the down-level `EXAMPLE\{0}` template over UPN. First consumer: BTDB Manager; ScadaBridge/HistorianGateway/OtOpcUa/MxAccessGateway stay on `0.3.0` until they move to AD. `0.2.1` is a functional fix, not housekeeping — `0.1.5` cannot log in against an AD domain root at all. `FallbackServers` (0.2.0) live-gated on the redundant local GLAuth pair. Version narrative, consumer matrix, `tools/LdapProbe`: [`ZB.MOM.WW.Auth/CLAUDE.md`](ZB.MOM.WW.Auth/CLAUDE.md). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) | | UI Theme (layout / tokens / components) | **2026-11-09: all five consumers pin `0.5.0` on their default branches (scadaproj#9; not deployed, no browser check yet); **`0.5.3` published 2026-11-12** (TechButton `disabled`, nav-state WeakSet, opt-in `ThemeShell ScrollNav`; 0.5.3 shell `100dvh` fill, #10; no consumer pins it yet — ScadaBridge#61 is first).** Adopted. Feed `0.2.0`–`0.5.0` + `0.5.3`; **five** consumers (the ignitionoee router joined), all on `0.4.1` and pushed (2026-08-11); Every deployed wonder box (DEV vd03 + QA vq03/vq04) runs its repo tip as of 2026-09-10, so the deployed Theme is 0.4.1 everywhere. **On bump, TRIM the local `.btn` block, don't delete it** — mxgw's carried more than sizing. Narrative: [`ZB.MOM.WW.Theme/CLAUDE.md`](ZB.MOM.WW.Theme/CLAUDE.md). | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) | | Health (readiness / liveness / active-node) | **Adopted** — lib `0.3.0`, all 4 apps reference it + wire `MapZbHealth`. `0.2.1`/`0.3.0` changed the active tier's behaviour (standby now answers **503**; active = oldest `Up` member via `ClusterActiveNode`, not `RoleLeader` — the two diverge after any restart), and both Akka apps' private copies were deleted in its favour (merged + pushed). Narrative: [`ZB.MOM.WW.Health/CLAUDE.md`](ZB.MOM.WW.Health/CLAUDE.md). | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | -| Observability (metrics / traces / logs) | **Adopted** — **`0.2.0` published 2026-11-12** (feed `0.1.0` + `0.2.0`; source at `0.2.1`, unpublished). **All 4 apps** call `AddZbTelemetry` **and** `AddZbSerilog`: ScadaBridge switched 2026-11-12 on `0.2.0` (#94, `main` `2d2e9f35`, governor + `ConfigureLogger` hooks); the other three stay on `0.1.0`, source-compatible. Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | +| Observability (metrics / traces / logs) | **Adopted** — **`0.2.0` published 2026-11-12** (feed `0.1.0` + `0.2.0`; source at `0.2.1`, unpublished). **All 4 apps** call `AddZbTelemetry` **and** `AddZbSerilog`: ScadaBridge switched 2026-11-12 on `0.2.0` (#94, `main` `2d2e9f35`, governor + `ConfigureLogger` hooks); the other three stay on `0.1.0`, source-compatible. **OTLP proven end to end 2026-11-12 (#13, closed):** shared dev collector `zb-otel-collector` on `10.100.0.35:4317` ([`infra/otel-collector/`](infra/otel-collector/)); metrics + traces from all four apps (ScadaBridge Site role; Central → ScadaBridge#133); OtOpcUa Galaxy driver telemetry was missing from the allowlists → fixed lmxopcua#892 (`master` `db86b996`, not deployed); no lib change, so `0.2.1` stays unpublished. Open: wonder-tier collector #54; `service.version`/`site.id`/`node.role` + logs-over-OTLP #55. Evidence: [`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](components/observability/2026-11-12-otlp-end-to-end-evidence.md). Narrative: [`ZB.MOM.WW.Telemetry/CLAUDE.md`](ZB.MOM.WW.Telemetry/CLAUDE.md). | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | @@ -249,6 +249,10 @@ msbuild Database\Database.sqlproj /p:Configuration=Release # SSDT → bin\Rel > single `zb-shared-glauth` container on the Linux fixture host **`10.100.0.35:3893`** > (`baseDN dc=zb,dc=local`, Transport=None). Source of truth and deploy runbook: > [`scadaproj/infra/glauth/`](infra/glauth/) (`config.toml` + `docker-compose.yml` + `README.md`). +> +> **Shared OTel collector (dev):** `zb-otel-collector` on the same host — OTLP gRPC `:4317` / HTTP `:4318`, +> Prometheus re-export `:8889`, health `:13133`. Point an app at it with `:Telemetry:Exporter=Otlp` + +> `:OtlpEndpoint=http://10.100.0.35:4317`. Runbook: [`scadaproj/infra/otel-collector/`](infra/otel-collector/). ## Backlog / open work (family standard, 2026-11-06) diff --git a/ZB.MOM.WW.Telemetry/CLAUDE.md b/ZB.MOM.WW.Telemetry/CLAUDE.md index f129e54..4d055db 100644 --- a/ZB.MOM.WW.Telemetry/CLAUDE.md +++ b/ZB.MOM.WW.Telemetry/CLAUDE.md @@ -2,7 +2,7 @@ Observability libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGateway, ScadaBridge). These are **libraries, not a service** — each package is linked directly into the consuming application at build time. There is no central telemetry process; instrumentation runs in-process alongside the application. -**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #13 (OTel collector / OTLP end to end). The history below is a record, not a status tracker; file new follow-ups as issues. +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #13 (OTel collector / OTLP end to end) is **closed** — proven 2026-11-12 against `zb-otel-collector` (`infra/otel-collector/`, evidence `components/observability/2026-11-12-otlp-end-to-end-evidence.md`); no library change was needed, so `0.2.1` stays unpublished. Open: #54 (wonder-tier collector), #55 (`service.version`/`site.id`/`node.role` + logs-over-OTLP). The history below is a record, not a status tracker; file new follow-ups as issues. The library normalizes the three-project observability surface: a shared OpenTelemetry Resource driven by a single identity triple (`service.name` / `site.id` / `node.role`), standard instrumentation wiring, Prometheus and OTLP export, and a Serilog bootstrap with enrichers and `TraceContextEnricher` for trace↔log correlation. @@ -88,7 +88,7 @@ Built at **0.1.0**, published to the Gitea NuGet feed, and **adopted across all 2026-06-01** (branch `feat/adopt-zb-telemetry` per repo, behaviour-preserving). MxAccessGateway's MEL→Serilog migration and metrics export both landed in this pass (not beforehand, despite the earlier claim). Deferred follow-ons (MxGateway `ms`→`s` + Meter rename, ScadaBridge app instruments -+ Site-node HTTP/1.1 metrics listener, OTLP wiring) are tracked in the component backlog (remaining open items are Gitea issues: #12 min-level governance hook, #13 OTLP collector): ++ Site-node HTTP/1.1 metrics listener, OTLP wiring) are tracked in the component backlog (remaining open items are Gitea issues: #12 min-level governance hook — closed; #13 OTLP collector — closed 2026-11-12): - `~/Desktop/scadaproj/components/observability/GAPS.md` — adoption status + deferred follow-ons diff --git a/components/observability/2026-11-12-otlp-end-to-end-evidence.md b/components/observability/2026-11-12-otlp-end-to-end-evidence.md new file mode 100644 index 0000000..99c8082 --- /dev/null +++ b/components/observability/2026-11-12-otlp-end-to-end-evidence.md @@ -0,0 +1,173 @@ +# OTLP export, end to end — evidence (2026-11-12, scadaproj#13) + +First run of the family's OTLP export path against a real collector. Before this, every app had +the `:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` wiring but nothing had ever received from +it, and OtOpcUa's spans had no exporter anywhere. + +**Result:** OTLP **metrics + traces** proven for **OtOpcUa, HistorianGateway, ScadaBridge (Site +role) and MxAccessGateway**. One real defect found and fixed (OtOpcUa Galaxy driver telemetry was +never exported, lmxopcua#892). No Telemetry library change was needed; `0.2.1` stays unpublished. + +## Collector + +`zb-otel-collector` on the shared Docker host `10.100.0.35` — `otel/opentelemetry-collector-contrib:0.162.0`, +OTLP gRPC `:4317` + HTTP `:4318`, Prometheus exporter `:8889`, health `:13133`; `debug` + `file` +exporters for proof. Source and runbook: [`infra/otel-collector/`](../../infra/otel-collector/). The +four ports were free on the host before deploy, and no other container was touched. + +``` +$ curl -s http://10.100.0.35:13133/ +{"status":"Server available","upSince":"2026-11-13T01:24:13.274419432Z",...} +$ curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' -d '{}' http://10.100.0.35:4318/v1/traces +200 +windev> Test-NetConnection 10.100.0.35 -Port 4317 -InformationLevel Quiet +True +``` + +## How each app was run + +No shared rig container was recreated or reconfigured. Each app ran as a throwaway process with +its own state, `Exporter=Otlp`, `OtlpEndpoint=http://10.100.0.35:4317` (gRPC) and +`OTEL_METRIC_EXPORT_INTERVAL=10000`. + +| App | Build | Telemetry pin | How | +|---|---|---|---| +| OtOpcUa | `master` `aaaa55e2` + the #892 fix (`0b90379f`) | 0.1.0 | Local `dotnet` host on the Mac: fused `admin,driver` single node in its own Akka mesh (`localhost:14053`), its own throwaway ConfigDb `OtOpcUaOtel13` on the docker-dev SQL (EF-migrated, one `ServerCluster`/`ClusterNode` row, dropped afterwards), HTTP `:19100`, OPC UA `:14840`. Activity: one `POST /api/deployments` + health/metrics GETs. | +| HistorianGateway | `main` `b95bae3` | 0.1.0 | Local `dotnet` host, `Development`, HTTP `:15220` / gRPC `:15221`, scratch SQLite stores. Activity: status RPCs via `grpcurl` (the historian itself is unreachable without the VPN, so they fail — which still produces spans) + dashboard/health GETs. | +| ScadaBridge | `main` `139a43fb` | 0.2.0 | Local `dotnet` **Site** host, own site id `otel-probe`, own 1-node mesh (`localhost:18082`), scratch LocalDb/audit/secrets, metrics listener `:18084`, central endpoint deliberately unreachable. The docker rig was in active use (ScadaBridge#46/#132; containers recreated minutes earlier), so it was not touched. | +| MxAccessGateway | deployed windev Server `447070a5` (scratch copy) | 0.1.0 | Copy of `C:\publish\mxaccessgw\Server-20261112-447070a5` run as a separate process on `:15120`/`:15130`, `Staging`, alarms off, scratch auth/secrets stores, `Worker:ExecutablePath` pointed at a non-existent `OtelProbeNoWorker.exe` so its `OrphanWorkerTerminator` (which matches by image name) could not match the live service's x86 workers. Live `MxAccessGw` stayed `Running`, `:5130/health/ready` = 200 throughout, worker PIDs `14700,22108` unchanged before/after; the scratch copy was deleted. | + +## What arrived (collector `otelsum.py` output, trimmed) + +Standard instrumentation scopes that every app sent are listed once at the end. + +### OtOpcUa — metrics + traces + +``` +RESOURCE otopcua {"service.name": "otopcua", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:48096", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"} +METRICS otopcua ZB.MOM.WW.OtOpcUa 3 ['otopcua.deploy.applied', 'otopcua.deploy.apply.duration', 'otopcua.redundancy.service_level_change'] +METRICS otopcua ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth'] +SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.deploy.apply') +SPANS 1 ('otopcua', 'ZB.MOM.WW.OtOpcUa', 'otopcua.opcua.address_space_rebuild') +SPANS 1 ('otopcua', 'Microsoft.AspNetCore', 'POST /api/deployments') +SPANS 5 ('otopcua', 'Microsoft.AspNetCore', 'GET /health/ready') +SPANS 3 ('otopcua', 'Microsoft.AspNetCore', 'GET /metrics') +``` + +```json +{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.deploy.apply", "traceId": "e5a579168f8c4629b25af4d3e4f2cb07", "spanId": "4f12d04ada38ea41", "kind": 1, + "attributes": [{"key": "otopcua.deployment_id", "value": {"stringValue": "e8b8c010d50144ccb371a91defceb6d9"}}, {"key": "otopcua.node_id", "value": {"stringValue": "localhost:14053"}}, + {"key": "otopcua.revision", "value": {"stringValue": "ed37612b…7bba"}}, {"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]} +{"scope": "ZB.MOM.WW.OtOpcUa", "name": "otopcua.opcua.address_space_rebuild", "traceId": "f6d8c238b2399cce0928de84691a0106", "spanId": "305ec5a9012a799d", "kind": 1, + "attributes": [{"key": "otopcua.correlation_id", "value": {"stringValue": "6aeae9fdaaff4b49bbb989162d16a335"}}]} +``` + +OtOpcUa's tracing is no longer a no-op once `Exporter=Otlp` is set: its own spans reach a backend +with their attributes. Note that `address_space_rebuild` is a separate trace from `deploy.apply` +(no parent — the context does not cross the Akka actor hop); the two join on +`otopcua.correlation_id`. + +### HistorianGateway — metrics + traces + +``` +RESOURCE historian-gateway {"service.name": "historian-gateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:49942", "host.name": "Josephs-MacBook-Pro", "_sdk": "1.15.3"} +METRICS historian-gateway ZB.MOM.WW.HistorianGateway 10 ['historian_gateway.grpc.request.duration', 'historian_gateway.grpc.requests', 'historian_gateway.historian.connection.up', 'historian_gateway.historian.event_session_pool.lease_waiters', 'historian_gateway.historian.event_session_pool.warm_sessions', 'historian_gateway.historian.session_pool.lease_waiters', 'historian_gateway.historian.session_pool.warm_sessions', 'historian_gateway.redundancy.members.healthy', 'historian_gateway.session_pool.lease.wait.duration', 'historian_gateway.store_forward.queue.depth'] +SPANS 8 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.Connect') +SPANS 2 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian.pool.lease') +SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/Probe') +SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'historian_gateway.v1.HistorianStatus/GetConnectionStatus') +SPANS 1 ('historian-gateway', 'ZB.MOM.WW.HistorianGateway', 'galaxy_repository.v1.GalaxyRepository/GetLastDeployTime') +SPANS 8 ('historian-gateway', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'HistoryService/GetInterfaceVersion') +SPANS 4 ('historian-gateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.HistorianGateway.Server.Dashboard.Components.Pages.Home') +``` + +`historian.Connect` spans carry `server.address`/`server.port` and `error.type=…HistorianTransportException` +(no VPN); the ones a request triggered are parented under its gRPC request span, the pool warm-up ones are roots. Prometheus exporter, same data: + +``` +historian_gateway_grpc_requests_total{job="ZB.MOM.WW/historian-gateway",method="Probe",outcome="ok",service="HistorianStatus",service_name="historian-gateway",...} 1 +``` + +### ScadaBridge (Site) — metrics + traces + +``` +RESOURCE scadabridge {"service.name": "scadabridge", "service.namespace": "ZB.MOM.WW", "service.instance.id": "Josephs-MacBook-Pro:50923", "host.name": "Josephs-MacBook-Pro", "site.id": "otel-probe", "node.role": "Site", "_sdk": "1.15.3"} +METRICS scadabridge ZB.MOM.WW.ScadaBridge 6 ['scadabridge.central.transport.import_sessions.open', 'scadabridge.site.alarm_cache.aggregators.active', 'scadabridge.site.data_connections.connected', 'scadabridge.site.data_connections.total', 'scadabridge.site.stream.subscribers.active', 'scadabridge.store_and_forward.queue.depth'] +METRICS scadabridge ZB.MOM.WW.LocalDb.Replication 1 ['localdb.oplog.depth'] +SPANS 12 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/Heartbeat') +SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReportSiteHealth') +SPANS 2 ('scadabridge', 'OpenTelemetry.Instrumentation.GrpcNetClient', 'scadabridge.centralcontrol.v1.CentralControlService/ReconcileSite') +SPANS 4 ('scadabridge', 'Microsoft.AspNetCore', 'GET /health/ready') +``` + +The only app that sends `site.id` / `node.role`. ScadaBridge has no `ActivitySource` of its own, +so its spans are standard instrumentation only. **Central role not run** → ScadaBridge#133. + +### MxAccessGateway — metrics + traces + +``` +RESOURCE mxgateway {"service.name": "mxgateway", "service.namespace": "ZB.MOM.WW", "service.instance.id": "DESKTOP-6JL3KKO:8720", "host.name": "DESKTOP-6JL3KKO", "_sdk": "1.15.3"} +METRICS mxgateway ZB.MOM.WW.MxGateway 5 ['mxgateway.alarms.provider_mode', 'mxgateway.events.grpc_stream_queue.depth', 'mxgateway.events.worker_queue.depth', 'mxgateway.sessions.open', 'mxgateway.workers.running'] +SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /') +SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /health/ready') +SPANS 7 ('mxgateway', 'Microsoft.AspNetCore', 'GET /metrics') +SPANS 7 ('mxgateway', 'Microsoft.AspNetCore.Components', 'Route / -> ZB.MOM.WW.MxGateway.Server.Dashboard.Components.Pages.DashboardHome') +``` + +Only the observable gauges appear: the counters/histograms (`commands.*`, `workers.startup.duration`, …) +record nothing without a worker session, and the x86 worker was deliberately not run. The gateway +has no `ActivitySource` of its own (spans are standard instrumentation only). Its gRPC port is not +reachable from off-box (windev firewall), so the traffic was generated on-box. + +### Standard instrumentation (all four) + +`System.Runtime` (19 `dotnet.*`), `OpenTelemetry.Instrumentation.Process` (5 `process.*`), +`System.Net.Http` (4 `http.client.*`), `Microsoft.AspNetCore.Hosting` (`http.server.request.duration`, +`http.server.active_requests`), `Microsoft.AspNetCore.Server.Kestrel`, `.Routing`, `.MemoryPool`; +plus `.Authentication`/`.Authorization`/`.Components` on the apps with a Blazor dashboard. + +## Defect found and fixed + +**lmxopcua#892 — the Galaxy driver's telemetry was never exported.** `ZbTelemetryOptions.Meters` +and `.ActivitySources` are strict allowlists. OtOpcUa named only `ZB.MOM.WW.OtOpcUa` and the +LocalDb meter, while the in-process Galaxy driver opens its spans (`galaxy.subscribe_bulk`, +`galaxy.unsubscribe_bulk`, `galaxy.stream_events`, `galaxy.write`, `galaxy.get_hierarchy`) on +ActivitySource `ZB.MOM.WW.OtOpcUa.Driver.Galaxy` and records the event pump's `galaxy.events.*` +counters on a Meter of the same name. All of it was silently dropped from `/metrics` and OTLP. + +Fix: OtOpcUa `master` `db86b996` (commit `0b90379f`) — a public `GalaxyDriverTelemetryNames` in +the driver; `ObservabilityExtensions.ObservedMeters` / `ObservedActivitySources` name it. +`ObservabilityAllowlistTests` assert on the listener side (span sampled, instrument enabled) with an +unregistered-name negative control. Dropping the Galaxy entries fails 3 of 8 tests, so the tests can +catch the regression. Galaxy driver suite 584/584. Not deployed. A live `galaxy.*` span was not +driven: that would need a Galaxy driver deployment against a gateway. + +The allowlists of the other apps were checked against every `new Meter(` / `new ActivitySource(` +in their source and in the hosted libraries. ScadaBridge (`ZB.MOM.WW.ScadaBridge`, LocalDb), +HistorianGateway (`ZB.MOM.WW.HistorianGateway` meter + source) and MxAccessGateway +(`ZB.MOM.WW.MxGateway`) are complete. Of the hosted libraries, only LocalDb defines a meter. + +## Checked, no defect + +- **Protocol/endpoint:** every app uses the library's `AddOtlpExporter` default (gRPC). `http://host:4317` + works for all four. No checked-in config carries a scheme-less endpoint. +- **Prometheus stays up under Otlp:** `/metrics` answered 200 on every app with `Exporter=Otlp` (`http.response.status_code=200` on each app's exported `GET /metrics` span) + (the overlay is additive, as documented). +- **Resource:** `service.name`, `service.namespace=ZB.MOM.WW`, `service.instance.id`, `host.name` + present on all four. + +## Gaps filed (not fixed here) + +- **scadaproj#55:** `service.version` is absent on all four apps; `site.id`/`node.role` are absent + on OtOpcUa, MxAccessGateway and HistorianGateway. No app exports logs over OTLP, because every + `AddZbSerilog` call passes only `ServiceName`, so the Serilog OTLP sink never activates + (`logs.jsonl` stayed empty). +- **ScadaBridge#133:** OTLP from the Central role is not yet proven. +- **scadaproj#54:** collector on the wonder tiers. + +## Telemetry 0.2.1: not published + +No finding needed a library change. The fixes and gaps above are all on the app side. `0.2.1`'s +scheme-less-endpoint rejection is a startup-breaking behaviour change. Publishing it would also +need a consumer pin-bump plan, and this run shows no consumer is exposed to that typo. It stays +built-but-unpublished until a pin bump wants it. diff --git a/components/observability/GAPS.md b/components/observability/GAPS.md index 8a75ac1..059cd1f 100644 --- a/components/observability/GAPS.md +++ b/components/observability/GAPS.md @@ -22,7 +22,13 @@ reach the shared `ZB.MOM.WW.Telemetry` library. Status legend: ⛔ gap · 🟡 p > listener, OTLP opt-in wiring) were **all subsequently executed the same day** — see the > "Follow-ons — DONE 2026-06-01" table at the bottom of this file, which this banner previously > contradicted (reconciled 2026-08-07). Only the operational items under "Still open" there -> (deploying an OTel collector → #13; further ScadaBridge instruments → ScadaBridge#81) remain. +> (further ScadaBridge instruments → ScadaBridge#81; collector on the wonder tiers → #54) remain. +> +> **✅ OTLP PROVEN END TO END — 2026-11-12 (#13, closed).** Shared dev collector `zb-otel-collector` +> on 10.100.0.35 ([`infra/otel-collector/`](../../infra/otel-collector/)); metrics **and** traces arrived +> from OtOpcUa, HistorianGateway, ScadaBridge (Site) and MxAccessGateway. One defect fixed: OtOpcUa's +> Galaxy driver meter/source were missing from the allowlists (lmxopcua#892). Evidence: +> [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md). ## Divergence vs spec @@ -256,7 +262,12 @@ behaviour-preserving except the intentional, no-consumer-yet metric-shape change **Still open (not code — operational/future):** -- **OTLP is opt-in but unexercised** (→ #13) until an OTel collector endpoint is deployed and the - `:Telemetry:Exporter=Otlp` + `:OtlpEndpoint` config is set. The wiring is in place; only a - collector is missing. +- ~~**OTLP is opt-in but unexercised** (→ #13)~~ — **DONE 2026-11-12**: exercised end to end against + `zb-otel-collector` (10.100.0.35:4317) for all four apps (ScadaBridge Site role only; Central → + ScadaBridge#133). See [`2026-11-12-otlp-end-to-end-evidence.md`](2026-11-12-otlp-end-to-end-evidence.md). +- **No collector on the wonder tiers** (DEV vd03, QA vq03/vq04) — every deployed box is still + Prometheus-only. → #54 +- **Resource identity + logs-over-OTLP gaps** found by #13: `service.version` absent on all four apps; + `site.id`/`node.role` absent on OtOpcUa, MxAccessGateway, HistorianGateway; no app passes the exporter + settings to `AddZbSerilog`, so logs never travel over OTLP. → #55 - **Further ScadaBridge instruments** beyond the first 4 are additive future work (not blocking). → ScadaBridge#81 diff --git a/components/observability/current-state/mxaccessgw/CURRENT-STATE.md b/components/observability/current-state/mxaccessgw/CURRENT-STATE.md index 5619239..afc88b9 100644 --- a/components/observability/current-state/mxaccessgw/CURRENT-STATE.md +++ b/components/observability/current-state/mxaccessgw/CURRENT-STATE.md @@ -1,5 +1,11 @@ # Observability — current state: MxAccessGateway +> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. Since then the gateway +> exports its metrics via `AddZbTelemetry`; the meter is `ZB.MOM.WW.MxGateway` (seconds), and logging is Serilog. +> OTLP export of **metrics and traces** is proven end to end (scratch copy of the deployed windev build; the +> `mxgateway.*` gauges and ASP.NET Core spans arrived). The gateway still has no `ActivitySource` of its own. +> Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md). + Repo: `~/Desktop/MxAccessGateway`. Stack: .NET 10 gateway (x64) + .NET 4.8 worker (**x86**); solution `src/MxGateway.sln`. Telemetry code is concentrated in `src/ZB.MOM.WW.MxGateway.Server/Metrics/` (instruments) and diff --git a/components/observability/current-state/otopcua/CURRENT-STATE.md b/components/observability/current-state/otopcua/CURRENT-STATE.md index df33fcc..b15cf4e 100644 --- a/components/observability/current-state/otopcua/CURRENT-STATE.md +++ b/components/observability/current-state/otopcua/CURRENT-STATE.md @@ -1,5 +1,13 @@ # Observability — current state: OtOpcUa +> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. OtOpcUa now calls +> `AddZbTelemetry` (Resource + OTLP opt-in), and OTLP export of **metrics and traces** is proven end to end +> against the dev collector (`otopcua.deploy.apply`, `otopcua.opcua.address_space_rebuild`, `otopcua.deploy.*` +> metrics arrived). That ends the "tracing no-op" below once `OtOpcUa:Telemetry:Exporter=Otlp` is set. The +> Galaxy driver's `galaxy.*` spans and `galaxy.events.*` counters were missing from the allowlists; fixed in +> lmxopcua#892 (`master` `db86b996`). Still absent from the Resource: `service.version`, `site.id`, `node.role` +> (scadaproj#55). Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md). + Repo: `~/Desktop/OtOpcUa`. Stack: .NET 10, Akka.NET, OPC UA; solution `ZB.MOM.WW.OtOpcUa.slnx`. Telemetry code lives in two places: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/` (host-side bootstrap) and `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/` (instruments + enricher). diff --git a/components/observability/current-state/scadabridge/CURRENT-STATE.md b/components/observability/current-state/scadabridge/CURRENT-STATE.md index 4da379b..ef77e43 100644 --- a/components/observability/current-state/scadabridge/CURRENT-STATE.md +++ b/components/observability/current-state/scadabridge/CURRENT-STATE.md @@ -1,5 +1,11 @@ # Observability — current state: ScadaBridge +> **2026-11-12 update (scadaproj#13):** this is the pre-adoption 2026-06-01 snapshot. ScadaBridge now +> has the `ZB.MOM.WW.ScadaBridge` meter and `AddZbSerilog`. OTLP export of **metrics and traces** is proven +> end to end for a **Site** node (6 `scadabridge.*` instruments, `localdb.oplog.depth`, gRPC-client spans, +> with `site.id`/`node.role` on the Resource); the Central role is not yet proven (ScadaBridge#133). +> Evidence: [`2026-11-12-otlp-end-to-end-evidence.md`](../../2026-11-12-otlp-end-to-end-evidence.md). + Repo: `~/Desktop/ScadaBridge`. Stack: .NET 10, Akka.NET, Docker; solution `ZB.MOM.WW.ScadaBridge.slnx`. The telemetry posture is split across a dangling OTel package ref (metrics/traces) and a substantive Serilog setup (logs). All paths relative to repo root. diff --git a/infra/otel-collector/.gitignore b/infra/otel-collector/.gitignore new file mode 100644 index 0000000..8fce603 --- /dev/null +++ b/infra/otel-collector/.gitignore @@ -0,0 +1 @@ +data/ diff --git a/infra/otel-collector/README.md b/infra/otel-collector/README.md new file mode 100644 index 0000000..388a2c2 --- /dev/null +++ b/infra/otel-collector/README.md @@ -0,0 +1,69 @@ +# Shared dev OpenTelemetry collector (`zb-otel-collector`) + +One [OpenTelemetry Collector (contrib)](https://github.com/open-telemetry/opentelemetry-collector-contrib) +that the family apps' OTLP exporters point at for dev/test. It runs as a single container on the +shared Docker host **`10.100.0.35`**, next to the other shared fixtures (`zb-shared-glauth`, the +OtOpcUa driver fixtures). Added by scadaproj#13; evidence of the first end-to-end run: +[`components/observability/2026-11-12-otlp-end-to-end-evidence.md`](../../components/observability/2026-11-12-otlp-end-to-end-evidence.md). + +> Scope: **dev/test only**. The receivers have no auth and no TLS. No wonder tier (DEV vd03, QA +> vq03/vq04) has a collector yet — that is scadaproj#54. + +## Endpoints + +| Port | What | Use | +|---|---|---| +| `4317` | OTLP **gRPC** receiver | The family default. Every app's `OtlpExporter` uses the gRPC protocol, so point apps here. | +| `4318` | OTLP **HTTP/protobuf** receiver | For clients that only speak OTLP/HTTP (`/v1/traces`, `/v1/metrics`, `/v1/logs`). | +| `8889` | Prometheus exporter | Every metric received over OTLP, re-exposed for scraping, with `job=/`, `instance=` and the identity labels (`service_name`, `site_id`, `node_role`, `host_name`). Series expire 15 min after the last update. | +| `13133` | `health_check` | `curl -s http://10.100.0.35:13133/` → `{"status":"Server available",...}` | + +Reachable from the Mac and from windev (`10.100.0.48`) — both verified. + +## Pointing an app at it + +OTLP is opt-in and **additive**: Prometheus `/metrics` keeps working when it is switched on. + +| App | Config keys (env form) | +|---|---| +| OtOpcUa | `OtOpcUa__Telemetry__Exporter=Otlp`, `OtOpcUa__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` | +| ScadaBridge | `ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` | +| MxAccessGateway | `MxGateway__Telemetry__Exporter=Otlp`, `MxGateway__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` | +| HistorianGateway | `Telemetry__Exporter=Otlp`, `Telemetry__OtlpEndpoint=http://10.100.0.35:4317` (root-level section) | + +- Always include the scheme: `http://10.100.0.35:4317`. Telemetry `0.2.1`+ rejects a scheme-less + `host:port` at startup; `0.1.0`/`0.2.0` (what the apps pin today) do not catch it there. +- The SDK's default metric export interval is 60 s. For a quick check set + `OTEL_METRIC_EXPORT_INTERVAL=10000` (ms) on the process. +- What travels: **metrics + traces**. Logs do not, in any app today — `AddZbSerilog` is called + without the exporter settings (scadaproj#55). + +## Reading what arrived + +```bash +ssh dohertj2@10.100.0.35 +docker logs --tail 20 zb-otel-collector # one debug line per received batch +cd ~/zb-otel-collector/data && python3 ../otelsum.py # per service: Resource, metric names, span names +python3 ../otelsum.py otopcua # one service +curl -s http://10.100.0.35:8889/metrics | grep 'job="ZB.MOM.WW/otopcua"' | head +``` + +`data/{metrics,traces,logs}.jsonl` are OTLP-JSON, one export request per line, rotated at 50 MB +(3 backups). [`otelsum.py`](otelsum.py) is the summariser used for the evidence doc. + +## Deploy / update + +```bash +# from this directory on the Mac +ssh dohertj2@10.100.0.35 'mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data' +scp docker-compose.yml config.yaml otelsum.py dohertj2@10.100.0.35:zb-otel-collector/ +ssh dohertj2@10.100.0.35 'cd ~/zb-otel-collector && docker compose up -d --force-recreate' +``` + +- The image runs as uid `10001`, so `data/` must be writable by it (`chmod 777` above, or + `sudo chown 10001:10001 data`). +- The image is pinned to an exact release (`0.162.0`). Bump deliberately, re-run the health check + and one app, and update this README. `resource_to_telemetry_conversion` is deprecated in this + release; the config uses its replacement `resource_constant_labels`. +- Host directory: `~/zb-otel-collector/` (compose project `zb-otel-collector`). It shares no + network or volume with any other container. Tear down with `docker compose down` there. diff --git a/infra/otel-collector/config.yaml b/infra/otel-collector/config.yaml new file mode 100644 index 0000000..af84869 --- /dev/null +++ b/infra/otel-collector/config.yaml @@ -0,0 +1,65 @@ +# zb-otel-collector — dev/test collector for the ZB.MOM.WW family (scadaproj#13). +# Receives OTLP (gRPC 4317 + HTTP 4318) from OtOpcUa, ScadaBridge, MxAccessGateway and +# HistorianGateway and proves receipt three ways: +# * debug exporter -> `docker logs zb-otel-collector` (one summary line per batch) +# * file exporters -> /data/{metrics,traces,logs}.jsonl (OTLP-JSON, rotated) +# * prometheus exporter-> :8889/metrics, every received metric re-exposed with +# job/instance + site_id/node_role/host_name labels (resource_constant_labels) +# No auth on the receivers: dev network only. Do not expose this config to a production tier. + +extensions: + health_check: + endpoint: 0.0.0.0:13133 + +receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + +processors: + memory_limiter: + check_interval: 5s + limit_mib: 512 + spike_limit_mib: 128 + batch: {} + +exporters: + debug: + verbosity: basic + file/metrics: + path: /data/metrics.jsonl + rotation: { max_megabytes: 50, max_backups: 3 } + file/traces: + path: /data/traces.jsonl + rotation: { max_megabytes: 50, max_backups: 3 } + file/logs: + path: /data/logs.jsonl + rotation: { max_megabytes: 50, max_backups: 3 } + prometheus: + endpoint: 0.0.0.0:8889 + # service.name/namespace become job/instance automatically; also lift the family identity triple. + resource_constant_labels: + included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name] + metric_expiration: 15m + +service: + extensions: [health_check] + telemetry: + logs: + level: info + pipelines: + metrics: + receivers: [otlp] + processors: [memory_limiter, batch] + exporters: [debug, file/metrics, prometheus] + traces: + receivers: [otlp] + processors: [memory_limiter, batch] + exporters: [debug, file/traces] + logs: + receivers: [otlp] + processors: [memory_limiter, batch] + exporters: [debug, file/logs] diff --git a/infra/otel-collector/docker-compose.yml b/infra/otel-collector/docker-compose.yml new file mode 100644 index 0000000..c3c5f8a --- /dev/null +++ b/infra/otel-collector/docker-compose.yml @@ -0,0 +1,31 @@ +# Shared dev OpenTelemetry collector for the ZB.MOM.WW family (scadaproj#13). +# Deploy on the shared Docker host 10.100.0.35: +# mkdir -p ~/zb-otel-collector/data && chmod 777 ~/zb-otel-collector/data, copy docker-compose.yml, +# config.yaml and otelsum.py into ~/zb-otel-collector/ (README "Deploy / update") +# cd ~/zb-otel-collector && docker compose up -d +# Verify: curl -s http://10.100.0.35:13133/ -> {"status":"Server available",...} +# curl -s http://10.100.0.35:8889/metrics -> received metrics, re-exposed for Prometheus +# tail -f ~/zb-otel-collector/data/traces.jsonl (also metrics.jsonl / logs.jsonl) +name: zb-otel-collector +services: + otel-collector: + # Pinned exact release (published 2026-10-05). Bump deliberately; never :latest. + image: otel/opentelemetry-collector-contrib:0.162.0 + container_name: zb-otel-collector + restart: unless-stopped + # The image runs as uid 10001; the bind-mounted data dir must be writable by it + # (README: `sudo chown 10001:10001 data` or `chmod 777 data`). + command: ["--config=/etc/otelcol-contrib/config.yaml"] + ports: + - "4317:4317" # OTLP gRPC (family default: :Telemetry:OtlpEndpoint=http://10.100.0.35:4317) + - "4318:4318" # OTLP HTTP/protobuf + - "8889:8889" # Prometheus exporter: everything received over OTLP, re-exposed for scraping + - "13133:13133" # health_check extension + volumes: + - ./config.yaml:/etc/otelcol-contrib/config.yaml:ro + - ./data:/data + logging: + driver: json-file + options: + max-size: "20m" + max-file: "3" diff --git a/infra/otel-collector/otelsum.py b/infra/otel-collector/otelsum.py new file mode 100644 index 0000000..efd5162 --- /dev/null +++ b/infra/otel-collector/otelsum.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python3 +# Summarise what zb-otel-collector received: per service.name, the Resource attributes, the +# metric names per instrumentation scope, and span counts per (scope, name). +# Usage on the collector host: cd ~/zb-otel-collector/data && python3 ../otelsum.py [service.name] +import json,sys,collections +svc_filter=sys.argv[1] if len(sys.argv)>1 else None +def attrs(r): return {a["key"]:list(a["value"].values())[0] if a["value"] else None for a in r.get("attributes",[])} +res={}; mets=collections.defaultdict(set); spans=collections.Counter() +for f,key in (("metrics.jsonl","resourceMetrics"),("traces.jsonl","resourceSpans"),("logs.jsonl","resourceLogs")): + try: lines=open(f) + except FileNotFoundError: continue + for l in lines: + d=json.loads(l) + for r in d.get(key,[]): + a=attrs(r["resource"]); s=a.get("service.name") + if svc_filter and s!=svc_filter: continue + res[s]={k:v for k,v in a.items() if not k.startswith("telemetry.sdk")} | {"_sdk":a.get("telemetry.sdk.version")} + if key=="resourceMetrics": + for sm in r["scopeMetrics"]: + for m in sm["metrics"]: mets[(s,sm["scope"]["name"])].add(m["name"]) + elif key=="resourceSpans": + for ss in r["scopeSpans"]: + for sp in ss["spans"]: spans[(s,ss["scope"]["name"],sp["name"])]+=1 + else: + for sl in r["scopeLogs"]: spans[(s,"",sl["scope"].get("name",""))]+=len(sl["logRecords"]) +for s,a in res.items(): print("RESOURCE",s,json.dumps(a)) +for (s,sc),names in sorted(mets.items()): print("METRICS",s,sc,len(names),sorted(names)) +for k,v in sorted(spans.items()): print("SPANS",v,k)