From d93ca7b8c29298ee95fbeface3e40eda13e2a796 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 12 Nov 2026 21:49:33 -0500 Subject: [PATCH] docs(observability): ScadaBridge Central role over OTLP proven on the rig (ScadaBridge#133) Metrics, traces and logs from both rig centrals with node.role=Central and site.id=central; deployments.applied and inbound_api.requests arrive. Follow-ups scadaproj#56 (log-export self-spans) and ScadaBridge#134 (rig image lacks git SHA). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- .../2026-11-12-otlp-end-to-end-evidence.md | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/components/observability/2026-11-12-otlp-end-to-end-evidence.md b/components/observability/2026-11-12-otlp-end-to-end-evidence.md index 0fa6358..ba2c28c 100644 --- a/components/observability/2026-11-12-otlp-end-to-end-evidence.md +++ b/components/observability/2026-11-12-otlp-end-to-end-evidence.md @@ -305,3 +305,86 @@ deleted. On every app, log records carry `traceId` when written inside a request, so logs join their spans in the backend. The Resource is byte-identical across the three signals of each process. + +--- + +# ScadaBridge Central role over OTLP — ScadaBridge#133 (2026-11-12, same day) + +Closes the last gap from the two runs above: the **Central** role had never exported. This run used the real +docker rig instead of a throwaway host, because a Central needs the config DB and LDAP. + +## How it was run + +- **Build:** the rig image predated Telemetry 0.3.0 (its `ZB.MOM.WW.Telemetry.dll` was `0.2.0+01c6dc1d`). It was + rebuilt with `docker/deploy.sh` from a clean `main` at `482ac665` (which includes `a633005f`, Telemetry 0.3.0), + and all 8 nodes came back healthy. +- **Override:** a temporary compose file (scratch, never committed) set on **central-a and central-b only** + `ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` and + `OTEL_METRIC_EXPORT_INTERVAL=10000`. Both centrals were recreated together + (`docker compose -f docker/docker-compose.yml -f up -d --force-recreate --no-deps central-a central-b`), + per the rig's pair-restart rule. Sites, Traefik and every other container were left alone. +- **Activity (through Traefik `:9000`):** + - 9 inbound calls rejected with 401: no key, or an empty key. + - 3 authorized `POST /api/SimpleAlarmStatus` (200 `{"WasSuccessful":true,…}`), made with a temporary method-scoped + API key. All three temporary keys were deleted afterwards (`api-key list` shows 0 `otel133*`). + - One `deploy instance --id 1492` (`rig-probe-1`, status Success). + - CLI `template list` / `site list` and UI/health/metrics GETs, on top of the health polling and the site heartbeats + that already happen. + +## What arrived (collector, one process per central) + +``` +central-a (active) RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0", "service.instance.id": "980946b3915d:6", "site.id": "central", "node.role": "Central", "host.name": "980946b3915d"} + RESOURCE resourceSpans {... identical ...} + RESOURCE resourceLogs {... identical ...} + COUNTS {'metric points': 463, 'spans': 321, 'log records': 3305} SEVERITY {'Information': 3046, 'Warning': 259} + LOG {"severity": "Information", "body": "Starting ScadaBridge host as Central on scadabridge-central-a"} +central-b (standby) RESOURCE ... "service.instance.id": "d4d8e49df76f:6", "site.id": "central", "node.role": "Central" ... (identical across the three signals) + COUNTS {'metric points': 421, 'spans': 77, 'log records': 331} SEVERITY {'Information': 311, 'Warning': 20} +``` + +`service.namespace=ZB.MOM.WW` is present on both. Logs carry `traceId` inside requests, as on the other apps. + +**Central instruments** (meter `ZB.MOM.WW.ScadaBridge`, last cumulative value on central-a): + +``` +scadabridge.deployments.applied 1 +scadabridge.inbound_api.requests {method=SimpleAlarmStatus} 3 +scadabridge.inbound_api.requests {method=} 9 +scadabridge.central.transport.import_sessions.open 0 ++ the shared gauges: site.stream.subscribers.active, site.data_connections.{connected,total}, + store_and_forward.queue.depth, site.alarm_cache.aggregators.active (all 0 on a central) +``` + +The counts match the activity exactly. The standby (central-b) exports only the observable gauges, because the +deploy and inbound singletons run on the active node. + +**Spans on central-a** (standard instrumentation; ScadaBridge has no `ActivitySource` of its own): +`POST /scadabridge.centralcontrol.v1.CentralControlService/{Heartbeat 133, SubmitNotification 94, IngestAuditEvents 24, ReportSiteHealth 10}`, +`POST /management` 16, `POST /auth/token` 16, `POST /api/{methodName}` 12, `GET /api/internal/deployments/{deploymentId}/config` 1, +the gRPC client spans `SiteCommandService/ExecuteRoute` 3 + `ExecuteLifecycle` 1 (the inbound route to the site +and the deploy), `GET /health/{active,ready}`, and the Blazor `Route / -> …CentralUI…Dashboard`. + +**Allowlist:** `SiteServiceRegistration.ObservedMeters` names `ZB.MOM.WW.ScadaBridge` (which holds every central and +site instrument) and the LocalDb meter, and the repository creates no other `Meter` or `ActivitySource`. **No defect.** + +## Restored + +Both centrals were recreated together again on the default compose with no override (0 telemetry env vars). Then: +- all 8 nodes `/health/ready` 200, one active per pair (central-a, site-{a,b,c}-a 200; their partners 503); +- central `memberCount 2, unreachableCount 0`; +- Traefik `/health/active` 200; +- negative control: the new central processes (`a596a23d9a12`, `1b6fd48c6785`) have **0** lines in + `metrics/traces/logs.jsonl`. + +The rig stays on the rebuilt `482ac665` image. + +## Follow-ups filed (not fixed here) + +- **scadaproj#56:** the Serilog OTLP sink's own `LogsService/Export` calls are traced as gRPC client spans in + **all four apps** (scadabridge 127, mxgateway 13, historian-gateway 10, otopcua 6 at the collector). The SDK's + trace and metric exporters suppress their own instrumentation, and `Serilog.Sinks.OpenTelemetry` does not. This + is noise, not a loop (0 log records about it). Fix in `ZB.MOM.WW.Telemetry` with an instrumentation filter. +- **ScadaBridge#134:** rig images report `service.version = "1.0.0"` with no `+`, because `.dockerignore` + excludes `.git/`. Fix by passing `SourceRevisionId` as a build arg. Wonder deploys, published from a checkout, are + unaffected.