From 8b7ea3213dd8a02a5c6810504d7d689d97007e1f Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 26 Jul 2026 10:29:51 -0400 Subject: [PATCH] fix(rig,docs): backfill ClusterNode.GrpcPort and document the upgrade step (#493) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `GrpcPort` is a nullable column added by Phase 5. Every insert in the docker-dev seed is INSERT-if-not-exists — the right shape for a re-runnable seed, but it means a new nullable column never reaches rows an earlier version of the file created. On a persisted volume all six rows kept NULL, central found no dial targets, and `TelemetryDial:Mode=Grpc` connected to nothing. (`AkkaPort` escaped this only because it is NOT NULL with a default of 4053.) Seed now backfills `GrpcPort IS NULL AND CreatedBy = 'docker-dev-seed'` to 4056. NULL only — it must not overwrite a port an operator set deliberately on a long-lived dev volume, and NULL is the unambiguous "predates the column" marker. Deliberately NOT doing the suggested EF data migration for real deployments. The correct value is each node's own `Telemetry:GrpcListenPort`, which is per-deployment configuration the database cannot derive; a migration could only invent one, and a WRONG dial target is worse than the NULL the dialer already skips explicitly with a throttled warning. The rig can be backfilled because there the port is fixed by docker-compose.yml and therefore actually knowable. `docs/Telemetry.md` gains an upgrade section stating the prerequisite, the symptoms (the throttled skip log, LISTEN with no ESTABLISHED, the pill never going live) and the UPDATE to run before flipping any node to Grpc — plus why there is no migration. --- docker-dev/seed/seed-clusters.sql | 22 +++++++++++++++++++++ docs/Telemetry.md | 33 +++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/docker-dev/seed/seed-clusters.sql b/docker-dev/seed/seed-clusters.sql index 62bfa596..97ceca09 100644 --- a/docker-dev/seed/seed-clusters.sql +++ b/docker-dev/seed/seed-clusters.sql @@ -106,6 +106,28 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053') (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed'); +------------------------------------------------------------------------------ +-- Backfill: GrpcPort on rows that predate the column (#493) +-- +-- Every insert above is INSERT-if-not-exists, which is the right shape for a re-runnable seed +-- but means a *new nullable* column never reaches rows created by an earlier version of this +-- file. On a persisted volume that is exactly what happened when Phase 5 added GrpcPort: the +-- six ClusterNode rows already existed, so they kept NULL, TelemetryNodeSource found no dial +-- targets, and TelemetryDial:Mode=Grpc silently connected to nothing (throttled Warning per +-- node — the code degrades gracefully, so nothing failed loudly). +-- +-- Backfills NULL only. It must not overwrite a port an operator set deliberately on a +-- long-lived dev volume; a NULL is the unambiguous "this row predates the column" marker. +-- +-- AkkaPort deliberately has no equivalent here: it is NOT NULL with a default of 4053, so +-- pre-existing rows already carry a usable value and there is nothing to backfill. +------------------------------------------------------------------------------ + +UPDATE dbo.ClusterNode + SET GrpcPort = 4056 -- matches Telemetry__GrpcListenPort on all six nodes in docker-compose.yml + WHERE GrpcPort IS NULL + AND CreatedBy = 'docker-dev-seed'; + ------------------------------------------------------------------------------ -- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15) -- diff --git a/docs/Telemetry.md b/docs/Telemetry.md index a24570f1..9f04a30d 100644 --- a/docs/Telemetry.md +++ b/docs/Telemetry.md @@ -128,6 +128,39 @@ transports. ScadaBridge itself has since closed that gap with this identical interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a later hardening pass. See the design doc's superseded note for the full history. +## Upgrading an existing deployment: populate `ClusterNode.GrpcPort` first (#493) + +`GrpcPort` is a **nullable column added by Phase 5**, and nothing backfills it onto rows that +already existed. On an upgraded deployment every `ClusterNode` row therefore carries `NULL`, +central finds **no dial targets**, and flipping `TelemetryDial:Mode` to `Grpc` connects to +nothing. Fresh installs are unaffected. `AkkaPort` did not have this problem only because it is +`NOT NULL` with a default of 4053. + +Nothing fails loudly, which is what makes this worth stating up front — the code degrades +gracefully, once per node, throttled: + +``` +ClusterNode has no GrpcPort; it exposes no telemetry stream and is skipped in this dial refresh +(further skips of this node are silent) +``` + +Other symptoms: no `ESTABLISHED` sockets on the telemetry port (a `LISTEN` but nothing else in +`docker exec cat /proc/net/tcp6`), and the `/alerts` / `/script-log` pill never turning +live in `Grpc` mode. + +**Before flipping any node to `Grpc`**, set each driver node's `GrpcPort` to that node's own +`Telemetry:GrpcListenPort` — via the AdminUI node editor, or directly: + +```sql +UPDATE dbo.ClusterNode SET GrpcPort = WHERE NodeId = ''; +``` + +There is deliberately **no EF data migration** backfilling this. The correct value is that node's +own listener port, which is per-deployment configuration the database cannot derive; a migration +could only invent one, and a *wrong* dial target is worse than the `NULL` the dialer already skips +explicitly. The docker-dev seed does backfill (`GrpcPort IS NULL AND CreatedBy = 'docker-dev-seed'` +→ 4056) because there the port is fixed by `docker-compose.yml` and therefore actually knowable. + ## Reconnect and reliability Central's `TelemetryDialSupervisor` (an admin-role actor) runs **one reconnecting dialer per