Phase 5 telemetry: existing deployments must populate ClusterNode.GrpcPort (seed only backfills fresh installs) #493

New Issue

2026-07-23T17:38:20-04:00

dohertj2 commented

2026-07-23 17:38:20 -04:00

Context

Per-cluster mesh Phase 5 (gRPC live-telemetry stream, merged 35552a96) added a nullable
ClusterNode.GrpcPort column. Central's TelemetryDialSupervisor reads it to discover each driver
node's telemetry endpoint (http://{Host}:{GrpcPort}) when TelemetryDial:Mode=Grpc.

The gotcha (surfaced by the Phase 5 live gate)

On an existing deployment, GrpcPort is NULL after upgrade, so central finds no dial targets
and the gRPC telemetry stream never connects — even with the dark switch flipped to Grpc. Fresh
installs are unaffected.

Root cause

The DB/rig seed is idempotent INSERT-if-not-exists, so it never backfills a new nullable column
onto pre-existing ClusterNode rows. (AkkaPort, added in an earlier phase, was unaffected only
because its column carries a NOT NULL default of 4053; GrpcPort is nullable with no default, so
pre-existing rows keep NULL.)

Symptom / how to spot it

Central logs, once per node (throttled — the code degrades gracefully, and last-known DPS panels are
unaffected since the flip is per-node):

ClusterNode <id> has no GrpcPort; it exposes no telemetry stream and is skipped in this dial refresh
(further skips of this node are silent)

No telemetry sockets on the telemetry port (e.g. 4056) — docker exec <node> cat /proc/net/tcp6
shows a LISTEN but no ESTABLISHED.
The /alerts and /script-log connection pill does not turn "live" in Grpc mode.

Fix / operator action

Populate ClusterNode.GrpcPort for every driver node before/while flipping to Grpc — via the AdminUI
node edit (Phase 1 surfaced the field) or a data update:

UPDATE dbo.ClusterNode SET GrpcPort = <telemetry port, e.g. 4056>;

Suggested improvements (pick one or more)

Make the docker-dev seed (docker-dev/seed/seed-clusters.sql) UPSERT the transport port columns
(AkkaPort/GrpcPort) so a persisted volume backfills on re-run instead of only fresh volumes.
Consider an EF data migration that backfills GrpcPort (default, or derived from each node's own
Cluster:PublicHostname/config) for existing deployments upgrading to Phase 5.
Add an explicit upgrade note to docs/Telemetry.md / release notes.

References

Merge 35552a96 (feat/mesh-phase5)
Gate record: docs/plans/2026-07-23-mesh-phase5-live-gate.md
Guide: docs/Telemetry.md

Note: the code behaved correctly throughout (graceful throttled skip); this is a rig/upgrade data gap,
not a Phase 5 code defect.

## Context Per-cluster mesh **Phase 5** (gRPC live-telemetry stream, merged `35552a96`) added a nullable `ClusterNode.GrpcPort` column. Central's `TelemetryDialSupervisor` reads it to discover each driver node's telemetry endpoint (`http://{Host}:{GrpcPort}`) when `TelemetryDial:Mode=Grpc`. ## The gotcha (surfaced by the Phase 5 live gate) On an **existing** deployment, `GrpcPort` is `NULL` after upgrade, so central finds **no dial targets** and the gRPC telemetry stream never connects — even with the dark switch flipped to `Grpc`. Fresh installs are unaffected. ## Root cause The DB/rig seed is idempotent **INSERT-if-not-exists**, so it never backfills a *new nullable* column onto pre-existing `ClusterNode` rows. (`AkkaPort`, added in an earlier phase, was unaffected only because its column carries a **NOT NULL default of 4053**; `GrpcPort` is nullable with no default, so pre-existing rows keep `NULL`.) ## Symptom / how to spot it Central logs, once per node (throttled — the code degrades gracefully, and last-known DPS panels are unaffected since the flip is per-node): ``` ClusterNode <id> has no GrpcPort; it exposes no telemetry stream and is skipped in this dial refresh (further skips of this node are silent) ``` - No telemetry sockets on the telemetry port (e.g. 4056) — `docker exec <node> cat /proc/net/tcp6` shows a `LISTEN` but no `ESTABLISHED`. - The `/alerts` and `/script-log` connection pill does **not** turn "live" in Grpc mode. ## Fix / operator action Populate `ClusterNode.GrpcPort` for every driver node before/while flipping to `Grpc` — via the AdminUI node edit (Phase 1 surfaced the field) or a data update: ```sql UPDATE dbo.ClusterNode SET GrpcPort = <telemetry port, e.g. 4056>; ``` ## Suggested improvements (pick one or more) - Make the docker-dev seed (`docker-dev/seed/seed-clusters.sql`) **UPSERT** the transport port columns (`AkkaPort`/`GrpcPort`) so a persisted volume backfills on re-run instead of only fresh volumes. - Consider an EF data migration that backfills `GrpcPort` (default, or derived from each node's own `Cluster:PublicHostname`/config) for existing deployments upgrading to Phase 5. - Add an explicit upgrade note to `docs/Telemetry.md` / release notes. ## References - Merge `35552a96` (`feat/mesh-phase5`) - Gate record: `docs/plans/2026-07-23-mesh-phase5-live-gate.md` - Guide: `docs/Telemetry.md` _Note: the code behaved correctly throughout (graceful throttled skip); this is a rig/upgrade data gap, not a Phase 5 code defect._

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#493