feat(host+rig): give central a dedicated Prometheus scrape port (register row 56)

The row read 'central nodes expose no Prometheus endpoint'. That is not quite
what was wrong, and the difference matters. Central already maps /metrics
(Program.cs central branch) and telemetry is registered for both roles through
the shared BindSharedOptions, so the endpoint rides endpoint routing and has
always been served on central's web port (:5000 on the rig, anonymous — there
is no FallbackPolicy). What central lacked was the site branch's dedicated
MetricsPort, which is the port the established live-scrape recipe probes:

    docker exec <node> bash -c 'exec 3<>/dev/tcp/localhost/8084 ...'

So a probe of central on 8084 failed and central-only series (KPI/outbox
counters, the bundle-import session gauge + its new rejection counter,
alarm-cache gauges) read as unscrapeable while they were in fact being
exported on a different port.

Central now binds a dedicated HTTP/1.1 listener on ScadaBridge:Node:MetricsPort,
so one probe recipe covers all eight rig nodes. HTTP/1.1-capable because a
standard Prometheus scraper cannot negotiate h2c, which is why this cannot
share the gRPC port.

Read as a PRESENCE check, not GetValue's defaulted read. NodeOptions defaults
MetricsPort to 8084, so a defaulted read would make every existing central node
start binding a new port on upgrade — and a port already in use is a hard boot
failure, not a degraded one. Central gains the listener only where an operator
asked for it; site nodes are unchanged and still bind theirs unconditionally.
The bind is also skipped when the port would collide with the gRPC port or an
ASPNETCORE_URLS port, since Kestrel throws on a duplicate ListenAnyIP and
/metrics is reachable on those ports anyway.

Config only — no host port mapping added, because the probe runs inside the
container. Applied to both docker/ centrals and both docker-env2/ centrals.

FILE EDITS ONLY: no docker command was run, and deploy.sh was not invoked.
This commit is contained in:
Joseph Doherty
2026-08-15 12:53:26 -04:00
parent b5f120c370
commit 103f2f1931
6 changed files with 86 additions and 13 deletions
+33 -6
View File
@@ -181,12 +181,28 @@ Two traps it exists to avoid:
Binding to them gives a `Connected` data connection with `Good` quality and *zero*
event traffic, which looks healthy and probes nothing. Use the built-in simulated
nodes (`SpikeData`, `RandomSignedInt32`, `StepUp`, `FastUInt1..5`, `SlowUInt1..5`).
- **`scadabridge_site_connection_up` does not mean what its name suggests.** It counts
open central→site gRPC *subscriber streams*, not OPC UA data connections, so it sits
at 0 on a perfectly healthy site with nobody watching. For data-connection health read
`dataConnectionEndpoints` / `dataConnectionTagQuality` off `scadabridge health site`.
- **`scadabridge_site_stream_subscribers_active` is not data-connection health.** It counts
open central→site gRPC *subscriber streams*, so it sits at 0 on a perfectly healthy site
with nobody watching. (Register row 54 renamed it from `scadabridge_site_connection_up`,
whose name invited exactly that misreading — it is what made row 48's "0 on all six site
nodes" look like a fault when it was healthy steady state.)
Verify (metrics are on container port 8084; the containers ship no curl/wget):
For real data-connection health, the same row added two honest gauges:
| Series | Meaning |
| --- | --- |
| `scadabridge_site_data_connections_connected` | Connections currently in the `Connected` state |
| `scadabridge_site_data_connections_total` | Connections known to this node, in any state |
`connected < total` is the alert condition — neither number alone separates "a device is
down" from "nothing is configured". Both read **0 on the standby node**, which is healthy:
connections are opened by the deployment path on the active node only, so alert on the
active node or on `max()` across the pair. The health report's `dataConnectionEndpoints` /
`dataConnectionTagQuality` (via `scadabridge health site`) remain the per-connection detail.
Verify (metrics are on container port 8084 — **all eight nodes** as of register row 56, which
gave the two central nodes the same dedicated scrape listener the six site nodes already had;
the containers ship no curl/wget):
```bash
scadabridge debug snapshot --id <instance-id>
@@ -195,9 +211,20 @@ scadabridge debug stream --id <instance-id> # open a stream, then in another s
docker exec scadabridge-site-a-a bash -c \
'exec 3<>/dev/tcp/localhost/8084 && printf "GET /metrics HTTP/1.0\r\nHost: localhost\r\n\r\n" >&3 && cat <&3' \
| grep -E 'site_connection_up|stream_batch_size'
| grep -E 'stream_subscribers_active|stream_batch_size|data_connections'
# Central-only series (KPI/outbox counters, bundle-import session gauge + rejection
# counter, alarm-cache gauges) — same recipe, central container:
docker exec scadabridge-central-a bash -c \
'exec 3<>/dev/tcp/localhost/8084 && printf "GET /metrics HTTP/1.0\r\nHost: localhost\r\n\r\n" >&3 && cat <&3' \
| grep -E 'import_sessions|alarm_cache'
```
> Central binds 8084 only because `docker/central-node-*/appsettings.Central.json` now sets
> `ScadaBridge:Node:MetricsPort`. Unlike a site node, central reads that key **only when
> present** — a central deployment without it is unchanged and still serves `/metrics` on its
> web port, since the endpoint rides endpoint routing and is served on every bound listener.
### After Code Changes
Rebuild and redeploy. The Docker build cache skips NuGet restore when only source files change:
@@ -4,7 +4,9 @@
"Role": "Central",
"NodeName": "central-a",
"NodeHostname": "scadabridge-central-a",
"RemotingPort": 8081
"RemotingPort": 8081,
"_metricsPort": "Register row 56. Central maps /metrics on every listener it binds, so it was always reachable on the web port (:5000) — but not on 8084, which is the port the live-scrape recipe probes (docker exec ... </dev/tcp/localhost/8084). Declaring it here binds the same dedicated HTTP/1.1 scrape listener the six site nodes use, so one probe recipe works on all eight nodes and central-only series (KPI/outbox counters, the import-session gauge + rejection counter, alarm-cache gauges) are scrapeable live. Central reads this key ONLY when present — unlike a site node, which always binds 8084 — so an existing central without it is unchanged. No host port mapping is needed: the probe runs inside the container.",
"MetricsPort": 8084
},
"Cluster": {
"SeedNodes": [
@@ -4,7 +4,9 @@
"Role": "Central",
"NodeName": "central-b",
"NodeHostname": "scadabridge-central-b",
"RemotingPort": 8081
"RemotingPort": 8081,
"_metricsPort": "Register row 56 — see central-node-a. Dedicated HTTP/1.1 Prometheus scrape listener, matching the six site nodes so one probe recipe covers all eight. Central binds this only when the key is present.",
"MetricsPort": 8084
},
"Cluster": {
"SeedNodes": [