dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 5 telemetry: existing deployments must populate ClusterNode.GrpcPort (seed only backfills fresh installs) #493

New Issue
Closed
opened 2026-07-23 17:38:20 -04:00 by dohertj2 · 1 comment
dohertj2 commented 2026-07-23 17:38:20 -04:00
Owner
Copy Link
Copy Source

Context

Per-cluster mesh Phase 5 (gRPC live-telemetry stream, merged 35552a96) added a nullable
ClusterNode.GrpcPort column. Central's TelemetryDialSupervisor reads it to discover each driver
node's telemetry endpoint (http://{Host}:{GrpcPort}) when TelemetryDial:Mode=Grpc.

The gotcha (surfaced by the Phase 5 live gate)

On an existing deployment, GrpcPort is NULL after upgrade, so central finds no dial targets
and the gRPC telemetry stream never connects — even with the dark switch flipped to Grpc. Fresh
installs are unaffected.

Root cause

The DB/rig seed is idempotent INSERT-if-not-exists, so it never backfills a new nullable column
onto pre-existing ClusterNode rows. (AkkaPort, added in an earlier phase, was unaffected only
because its column carries a NOT NULL default of 4053; GrpcPort is nullable with no default, so
pre-existing rows keep NULL.)

Symptom / how to spot it

Central logs, once per node (throttled — the code degrades gracefully, and last-known DPS panels are
unaffected since the flip is per-node):

ClusterNode <id> has no GrpcPort; it exposes no telemetry stream and is skipped in this dial refresh
(further skips of this node are silent)
  • No telemetry sockets on the telemetry port (e.g. 4056) — docker exec <node> cat /proc/net/tcp6
    shows a LISTEN but no ESTABLISHED.
  • The /alerts and /script-log connection pill does not turn "live" in Grpc mode.

Fix / operator action

Populate ClusterNode.GrpcPort for every driver node before/while flipping to Grpc — via the AdminUI
node edit (Phase 1 surfaced the field) or a data update:

UPDATE dbo.ClusterNode SET GrpcPort = <telemetry port, e.g. 4056>;

Suggested improvements (pick one or more)

  • Make the docker-dev seed (docker-dev/seed/seed-clusters.sql) UPSERT the transport port columns
    (AkkaPort/GrpcPort) so a persisted volume backfills on re-run instead of only fresh volumes.
  • Consider an EF data migration that backfills GrpcPort (default, or derived from each node's own
    Cluster:PublicHostname/config) for existing deployments upgrading to Phase 5.
  • Add an explicit upgrade note to docs/Telemetry.md / release notes.

References

  • Merge 35552a96 (feat/mesh-phase5)
  • Gate record: docs/plans/2026-07-23-mesh-phase5-live-gate.md
  • Guide: docs/Telemetry.md

Note: the code behaved correctly throughout (graceful throttled skip); this is a rig/upgrade data gap,
not a Phase 5 code defect.

## Context Per-cluster mesh **Phase 5** (gRPC live-telemetry stream, merged `35552a96`) added a nullable `ClusterNode.GrpcPort` column. Central's `TelemetryDialSupervisor` reads it to discover each driver node's telemetry endpoint (`http://{Host}:{GrpcPort}`) when `TelemetryDial:Mode=Grpc`. ## The gotcha (surfaced by the Phase 5 live gate) On an **existing** deployment, `GrpcPort` is `NULL` after upgrade, so central finds **no dial targets** and the gRPC telemetry stream never connects — even with the dark switch flipped to `Grpc`. Fresh installs are unaffected. ## Root cause The DB/rig seed is idempotent **INSERT-if-not-exists**, so it never backfills a *new nullable* column onto pre-existing `ClusterNode` rows. (`AkkaPort`, added in an earlier phase, was unaffected only because its column carries a **NOT NULL default of 4053**; `GrpcPort` is nullable with no default, so pre-existing rows keep `NULL`.) ## Symptom / how to spot it Central logs, once per node (throttled — the code degrades gracefully, and last-known DPS panels are unaffected since the flip is per-node): ``` ClusterNode <id> has no GrpcPort; it exposes no telemetry stream and is skipped in this dial refresh (further skips of this node are silent) ``` - No telemetry sockets on the telemetry port (e.g. 4056) — `docker exec <node> cat /proc/net/tcp6` shows a `LISTEN` but no `ESTABLISHED`. - The `/alerts` and `/script-log` connection pill does **not** turn "live" in Grpc mode. ## Fix / operator action Populate `ClusterNode.GrpcPort` for every driver node before/while flipping to `Grpc` — via the AdminUI node edit (Phase 1 surfaced the field) or a data update: ```sql UPDATE dbo.ClusterNode SET GrpcPort = <telemetry port, e.g. 4056>; ``` ## Suggested improvements (pick one or more) - Make the docker-dev seed (`docker-dev/seed/seed-clusters.sql`) **UPSERT** the transport port columns (`AkkaPort`/`GrpcPort`) so a persisted volume backfills on re-run instead of only fresh volumes. - Consider an EF data migration that backfills `GrpcPort` (default, or derived from each node's own `Cluster:PublicHostname`/config) for existing deployments upgrading to Phase 5. - Add an explicit upgrade note to `docs/Telemetry.md` / release notes. ## References - Merge `35552a96` (`feat/mesh-phase5`) - Gate record: `docs/plans/2026-07-23-mesh-phase5-live-gate.md` - Guide: `docs/Telemetry.md` _Note: the code behaved correctly throughout (graceful throttled skip); this is a rig/upgrade data gap, not a Phase 5 code defect._
dohertj2 commented 2026-07-26 10:43:01 -04:00
Author
Owner
Copy Link
Copy Source

Fixed on fix/issues-503-501-493-499-488, commit 8b7ea321. Took suggestions 1 and 3; deliberately not 2.

Seed backfill. docker-dev/seed/seed-clusters.sql now backfills GrpcPort where it IS NULL AND CreatedBy = 'docker-dev-seed' → 4056. NULL only — it must not overwrite a port an operator set deliberately on a long-lived dev volume, and NULL is the unambiguous "this row predates the column" marker. No AkkaPort equivalent: it is NOT NULL with a default of 4053, so there is nothing to backfill and a no-op UPDATE would only mislead.

Docs. docs/Telemetry.md gains an upgrade section before "Reconnect and reliability": the prerequisite, the symptoms (the throttled skip log, LISTEN with no ESTABLISHED, the pill never going live), and the UPDATE to run before flipping any node to Grpc.

No EF data migration, on purpose. The correct value is each node's own Telemetry:GrpcListenPort — per-deployment configuration the database cannot derive. A migration could only invent a value, and a wrong dial target is worse than the NULL the dialer already skips explicitly and gracefully: it would connect to something unintended, or fail in a way that looks like a live-telemetry bug rather than a missing setting. The rig is different precisely because there the port is fixed by docker-compose.yml and therefore actually knowable. That reasoning is recorded in the doc so it is not re-litigated.

Fixed on `fix/issues-503-501-493-499-488`, commit `8b7ea321`. Took suggestions 1 and 3; deliberately **not** 2. **Seed backfill.** `docker-dev/seed/seed-clusters.sql` now backfills `GrpcPort` where it `IS NULL AND CreatedBy = 'docker-dev-seed'` → 4056. NULL **only** — it must not overwrite a port an operator set deliberately on a long-lived dev volume, and NULL is the unambiguous "this row predates the column" marker. No `AkkaPort` equivalent: it is `NOT NULL` with a default of 4053, so there is nothing to backfill and a no-op UPDATE would only mislead. **Docs.** `docs/Telemetry.md` gains an upgrade section before "Reconnect and reliability": the prerequisite, the symptoms (the throttled skip log, `LISTEN` with no `ESTABLISHED`, the pill never going live), and the `UPDATE` to run before flipping any node to `Grpc`. **No EF data migration, on purpose.** The correct value is each node's own `Telemetry:GrpcListenPort` — per-deployment configuration the database cannot derive. A migration could only *invent* a value, and a **wrong** dial target is worse than the NULL the dialer already skips explicitly and gracefully: it would connect to something unintended, or fail in a way that looks like a live-telemetry bug rather than a missing setting. The rig is different precisely because there the port is fixed by `docker-compose.yml` and therefore actually knowable. That reasoning is recorded in the doc so it is not re-litigated.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#493
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?