docs(observability): ScadaBridge Central role over OTLP proven on the rig (ScadaBridge#133)
Metrics, traces and logs from both rig centrals with node.role=Central and site.id=central; deployments.applied and inbound_api.requests arrive. Follow-ups scadaproj#56 (log-export self-spans) and ScadaBridge#134 (rig image lacks git SHA). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -305,3 +305,86 @@ deleted.
|
|||||||
|
|
||||||
On every app, log records carry `traceId` when written inside a request, so logs join their spans in the
|
On every app, log records carry `traceId` when written inside a request, so logs join their spans in the
|
||||||
backend. The Resource is byte-identical across the three signals of each process.
|
backend. The Resource is byte-identical across the three signals of each process.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# ScadaBridge Central role over OTLP — ScadaBridge#133 (2026-11-12, same day)
|
||||||
|
|
||||||
|
Closes the last gap from the two runs above: the **Central** role had never exported. This run used the real
|
||||||
|
docker rig instead of a throwaway host, because a Central needs the config DB and LDAP.
|
||||||
|
|
||||||
|
## How it was run
|
||||||
|
|
||||||
|
- **Build:** the rig image predated Telemetry 0.3.0 (its `ZB.MOM.WW.Telemetry.dll` was `0.2.0+01c6dc1d`). It was
|
||||||
|
rebuilt with `docker/deploy.sh` from a clean `main` at `482ac665` (which includes `a633005f`, Telemetry 0.3.0),
|
||||||
|
and all 8 nodes came back healthy.
|
||||||
|
- **Override:** a temporary compose file (scratch, never committed) set on **central-a and central-b only**
|
||||||
|
`ScadaBridge__Telemetry__Exporter=Otlp`, `ScadaBridge__Telemetry__OtlpEndpoint=http://10.100.0.35:4317` and
|
||||||
|
`OTEL_METRIC_EXPORT_INTERVAL=10000`. Both centrals were recreated together
|
||||||
|
(`docker compose -f docker/docker-compose.yml -f <override> up -d --force-recreate --no-deps central-a central-b`),
|
||||||
|
per the rig's pair-restart rule. Sites, Traefik and every other container were left alone.
|
||||||
|
- **Activity (through Traefik `:9000`):**
|
||||||
|
- 9 inbound calls rejected with 401: no key, or an empty key.
|
||||||
|
- 3 authorized `POST /api/SimpleAlarmStatus` (200 `{"WasSuccessful":true,…}`), made with a temporary method-scoped
|
||||||
|
API key. All three temporary keys were deleted afterwards (`api-key list` shows 0 `otel133*`).
|
||||||
|
- One `deploy instance --id 1492` (`rig-probe-1`, status Success).
|
||||||
|
- CLI `template list` / `site list` and UI/health/metrics GETs, on top of the health polling and the site heartbeats
|
||||||
|
that already happen.
|
||||||
|
|
||||||
|
## What arrived (collector, one process per central)
|
||||||
|
|
||||||
|
```
|
||||||
|
central-a (active) RESOURCE resourceMetrics {"service.name": "scadabridge", "service.version": "1.0.0", "service.instance.id": "980946b3915d:6", "site.id": "central", "node.role": "Central", "host.name": "980946b3915d"}
|
||||||
|
RESOURCE resourceSpans {... identical ...}
|
||||||
|
RESOURCE resourceLogs {... identical ...}
|
||||||
|
COUNTS {'metric points': 463, 'spans': 321, 'log records': 3305} SEVERITY {'Information': 3046, 'Warning': 259}
|
||||||
|
LOG {"severity": "Information", "body": "Starting ScadaBridge host as Central on scadabridge-central-a"}
|
||||||
|
central-b (standby) RESOURCE ... "service.instance.id": "d4d8e49df76f:6", "site.id": "central", "node.role": "Central" ... (identical across the three signals)
|
||||||
|
COUNTS {'metric points': 421, 'spans': 77, 'log records': 331} SEVERITY {'Information': 311, 'Warning': 20}
|
||||||
|
```
|
||||||
|
|
||||||
|
`service.namespace=ZB.MOM.WW` is present on both. Logs carry `traceId` inside requests, as on the other apps.
|
||||||
|
|
||||||
|
**Central instruments** (meter `ZB.MOM.WW.ScadaBridge`, last cumulative value on central-a):
|
||||||
|
|
||||||
|
```
|
||||||
|
scadabridge.deployments.applied 1
|
||||||
|
scadabridge.inbound_api.requests {method=SimpleAlarmStatus} 3
|
||||||
|
scadabridge.inbound_api.requests {method=<unauthorized>} 9
|
||||||
|
scadabridge.central.transport.import_sessions.open 0
|
||||||
|
+ the shared gauges: site.stream.subscribers.active, site.data_connections.{connected,total},
|
||||||
|
store_and_forward.queue.depth, site.alarm_cache.aggregators.active (all 0 on a central)
|
||||||
|
```
|
||||||
|
|
||||||
|
The counts match the activity exactly. The standby (central-b) exports only the observable gauges, because the
|
||||||
|
deploy and inbound singletons run on the active node.
|
||||||
|
|
||||||
|
**Spans on central-a** (standard instrumentation; ScadaBridge has no `ActivitySource` of its own):
|
||||||
|
`POST /scadabridge.centralcontrol.v1.CentralControlService/{Heartbeat 133, SubmitNotification 94, IngestAuditEvents 24, ReportSiteHealth 10}`,
|
||||||
|
`POST /management` 16, `POST /auth/token` 16, `POST /api/{methodName}` 12, `GET /api/internal/deployments/{deploymentId}/config` 1,
|
||||||
|
the gRPC client spans `SiteCommandService/ExecuteRoute` 3 + `ExecuteLifecycle` 1 (the inbound route to the site
|
||||||
|
and the deploy), `GET /health/{active,ready}`, and the Blazor `Route / -> …CentralUI…Dashboard`.
|
||||||
|
|
||||||
|
**Allowlist:** `SiteServiceRegistration.ObservedMeters` names `ZB.MOM.WW.ScadaBridge` (which holds every central and
|
||||||
|
site instrument) and the LocalDb meter, and the repository creates no other `Meter` or `ActivitySource`. **No defect.**
|
||||||
|
|
||||||
|
## Restored
|
||||||
|
|
||||||
|
Both centrals were recreated together again on the default compose with no override (0 telemetry env vars). Then:
|
||||||
|
- all 8 nodes `/health/ready` 200, one active per pair (central-a, site-{a,b,c}-a 200; their partners 503);
|
||||||
|
- central `memberCount 2, unreachableCount 0`;
|
||||||
|
- Traefik `/health/active` 200;
|
||||||
|
- negative control: the new central processes (`a596a23d9a12`, `1b6fd48c6785`) have **0** lines in
|
||||||
|
`metrics/traces/logs.jsonl`.
|
||||||
|
|
||||||
|
The rig stays on the rebuilt `482ac665` image.
|
||||||
|
|
||||||
|
## Follow-ups filed (not fixed here)
|
||||||
|
|
||||||
|
- **scadaproj#56:** the Serilog OTLP sink's own `LogsService/Export` calls are traced as gRPC client spans in
|
||||||
|
**all four apps** (scadabridge 127, mxgateway 13, historian-gateway 10, otopcua 6 at the collector). The SDK's
|
||||||
|
trace and metric exporters suppress their own instrumentation, and `Serilog.Sinks.OpenTelemetry` does not. This
|
||||||
|
is noise, not a loop (0 log records about it). Fix in `ZB.MOM.WW.Telemetry` with an instrumentation filter.
|
||||||
|
- **ScadaBridge#134:** rig images report `service.version = "1.0.0"` with no `+<sha>`, because `.dockerignore`
|
||||||
|
excludes `.git/`. Fix by passing `SourceRevisionId` as a build arg. Wonder deploys, published from a checkout, are
|
||||||
|
unaffected.
|
||||||
|
|||||||
Reference in New Issue
Block a user