docs(observability): ScadaBridge Central role over OTLP proven on the rig (ScadaBridge#133)

Metrics, traces and logs from both rig centrals with node.role=Central and site.id=central;
deployments.applied and inbound_api.requests arrive. Follow-ups scadaproj#56 (log-export
self-spans) and ScadaBridge#134 (rig image lacks git SHA).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 21:49:33 -05:00
parent ca81dcbdd2
commit d93ca7b8c2
@@ -305,3 +305,86 @@ deleted.
On every app, log records carry `traceId` when written inside a request, so logs join their spans in the
backend. The Resource is byte-identical across the three signals of each process.
---
# ScadaBridge Central role over OTLP — ScadaBridge#133 (2026-11-12, same day)
Closes the last gap from the two runs above: the **Central** role had never exported. This run used the real
docker rig instead of a throwaway host, because a Central needs the config DB and LDAP.
## How it was run
- **Build:** the rig image predated Telemetry 0.3.0 (its `ZB.MOM.WW.Telemetry.dll` was `0.2.0+01c6dc1d`). It was
rebuilt with `docker/deploy.sh` from a clean `main` at `482ac665` (which includes `a633005f`, Telemetry 0.3.0),
and all 8 nodes came back healthy.
- **Override:** a temporary compose file (scratch, never committed) set on **central-a and central-b only**
`ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` and
`OTEL_METRIC_EXPORT_INTERVAL=10000`. Both centrals were recreated together
(`docker compose -f docker/docker-compose.yml -f <override> up -d --force-recreate --no-deps central-a central-b`),
per the rig's pair-restart rule. Sites, Traefik and every other container were left alone.
- **Activity (through Traefik `:9000`):**
- 9 inbound calls rejected with 401: no key, or an empty key.
- 3 authorized `POST /api/SimpleAlarmStatus` (200 `{"WasSuccessful":true,…}`), made with a temporary method-scoped
API key. All three temporary keys were deleted afterwards (`api-key list` shows 0 `otel133*`).
- One `deploy instance --id 1492` (`rig-probe-1`, status Success).
- CLI `template list` / `site list` and UI/health/metrics GETs, on top of the health polling and the site heartbeats
that already happen.
## What arrived (collector, one process per central)
```
central-a (active) RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0", "service.instance.id": "980946b3915d:6", "site.id": "central", "node.role": "Central", "host.name": "980946b3915d"}
RESOURCE resourceSpans {... identical ...}
RESOURCE resourceLogs {... identical ...}
COUNTS {'metric points': 463, 'spans': 321, 'log records': 3305} SEVERITY {'Information': 3046, 'Warning': 259}
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Central on scadabridge-central-a"}
central-b (standby) RESOURCE ... "service.instance.id": "d4d8e49df76f:6", "site.id": "central", "node.role": "Central" ... (identical across the three signals)
COUNTS {'metric points': 421, 'spans': 77, 'log records': 331} SEVERITY {'Information': 311, 'Warning': 20}
```
`service.namespace=ZB.MOM.WW` is present on both. Logs carry `traceId` inside requests, as on the other apps.
**Central instruments** (meter `ZB.MOM.WW.ScadaBridge`, last cumulative value on central-a):
```
scadabridge.deployments.applied 1
scadabridge.inbound_api.requests {method=SimpleAlarmStatus} 3
scadabridge.inbound_api.requests {method=<unauthorized>} 9
scadabridge.central.transport.import_sessions.open 0
+ the shared gauges: site.stream.subscribers.active, site.data_connections.{connected,total},
store_and_forward.queue.depth, site.alarm_cache.aggregators.active (all 0 on a central)
```
The counts match the activity exactly. The standby (central-b) exports only the observable gauges, because the
deploy and inbound singletons run on the active node.
**Spans on central-a** (standard instrumentation; ScadaBridge has no `ActivitySource` of its own):
`POST /scadabridge.centralcontrol.v1.CentralControlService/{Heartbeat 133, SubmitNotification 94, IngestAuditEvents 24, ReportSiteHealth 10}`,
`POST /management` 16, `POST /auth/token` 16, `POST /api/{methodName}` 12, `GET /api/internal/deployments/{deploymentId}/config` 1,
the gRPC client spans `SiteCommandService/ExecuteRoute` 3 + `ExecuteLifecycle` 1 (the inbound route to the site
and the deploy), `GET /health/{active,ready}`, and the Blazor `Route / -> …CentralUI…Dashboard`.
**Allowlist:** `SiteServiceRegistration.ObservedMeters` names `ZB.MOM.WW.ScadaBridge` (which holds every central and
site instrument) and the LocalDb meter, and the repository creates no other `Meter` or `ActivitySource`. **No defect.**
## Restored
Both centrals were recreated together again on the default compose with no override (0 telemetry env vars). Then:
- all 8 nodes `/health/ready` 200, one active per pair (central-a, site-{a,b,c}-a 200; their partners 503);
- central `memberCount 2, unreachableCount 0`;
- Traefik `/health/active` 200;
- negative control: the new central processes (`a596a23d9a12`, `1b6fd48c6785`) have **0** lines in
`metrics/traces/logs.jsonl`.
The rig stays on the rebuilt `482ac665` image.
## Follow-ups filed (not fixed here)
- **scadaproj#56:** the Serilog OTLP sink's own `LogsService/Export` calls are traced as gRPC client spans in
**all four apps** (scadabridge 127, mxgateway 13, historian-gateway 10, otopcua 6 at the collector). The SDK's
trace and metric exporters suppress their own instrumentation, and `Serilog.Sinks.OpenTelemetry` does not. This
is noise, not a loop (0 log records about it). Fix in `ZB.MOM.WW.Telemetry` with an instrumentation filter.
- **ScadaBridge#134:** rig images report `service.version = "1.0.0"` with no `+<sha>`, because `.dockerignore`
excludes `.git/`. Fix by passing `SourceRevisionId` as a build arg. Wonder deploys, published from a checkout, are
unaffected.