lmxopcua/docs/plans/2026-07-23-mesh-phase5-live-gate.md

Per-Cluster Mesh Phase 5 — Live Gate Record

Date: 2026-07-23 Branch: feat/mesh-phase5 (Phase-5 image built + rig recreated with Phase-5 code) Rig: local docker-dev — central-1/2 (admin,driver, fused), site-a-1/2, site-b-1/2 (driver). Telemetry served on :4056 (h2c), shared key telemetry-docker-dev-key.

Result: PASSED. Every load-bearing exit-gate leg proven live. In Telemetry:Mode=Grpc, central dials each driver node's telemetry gRPC server, the full stream mesh forms, real telemetry traverses node→hub→gRPC→central→sink (AdminUI pill goes live), and kill-and-reconnect of a node recovers its stream automatically. In the default Dps mode the node hosts the server but central does not dial.

Evidence was gathered primarily at the TCP-socket level (/proc/net/tcp6, port 4056 = hex 0FD8) because it is decisive and log-independent: an established stream is an established stream.

What was verified

Leg 0 — Dps baseline (headline: node always hosts, central does NOT dial)

Fresh rig in the committed default (Telemetry:Mode unset ⇒ Dps). site-a-1 log: Now listening on: http://[::]:4056 — the dedicated h2c telemetry listener binds in Dps mode too (the node always hosts the server). Socket check: site-a-1 has a LISTEN on [::]:4056 and zero inbound ESTABLISHED — central does not dial in Dps mode. Cluster formed clean (central-1 sees central-2 + all four site nodes at Up), OPC UA up, drivers subscribed. Baseline correct.

Leg 1 — Grpc flip → full stream mesh forms ✅

Flipped via a docker-compose.override.yml (gate artifact, since removed) setting TelemetryDial__Mode=Grpc on the central pair and Telemetry__Mode=Grpc on all six driver nodes, then recreated. After the central pair discovered the (populated) ClusterNode.GrpcPort rows and dialed:

Node	inbound telemetry streams on :4056
central-1	2
central-2	2
site-a-1	2
site-a-2	2
site-b-1	2
site-b-2	2

Both central nodes each held 6 outbound streams (one per driver node incl. self). 12 streams total — the exact designed topology (both admins dial every driver node). Zero telemetry failures in the steady window. Dial happens only in Grpc mode (Leg 0 had zero); discovery is DB-sourced from ClusterNode (Host + GrpcPort), matching Phase 1 + the rig seed.

Leg 2 — data path end-to-end (AdminUI pill live) ✅

http://localhost:9200/alerts connection pill read "live". This is the load-bearing data-path proof: the dial supervisor marks a node Connected — and flips both broadcasters' SetConnected(true) — only on its FIRST received telemetry event. So "live" means a real telemetry event (the 5 s-periodic driver-resilience-status) traversed the node's publish seam → node-local hub → gRPC stream → central dial supervisor → TelemetryReceived → the in-process sink the Blazor panel reads. Transport and payload proven, not just a connected socket. (The panel's static "…from the cluster's DPS topic" caption is now stale wording — cosmetic, the broadcaster seam is unchanged, only its upstream swapped; noted as a doc-copy follow-up, non-blocking.)

Leg 3 — kill-and-reconnect recovers every stream ✅

docker kill otopcua-dev-site-b-1-1:

• central-1 logged Telemetry stream to node site-b-1:4053 failed (failure streak 1); reconnecting.
• central-1 outbound streams dropped 6 → 5; the pill stayed live (the other 5 streams carried the panel — a single node loss does not blank the fleet view).

docker start otopcua-dev-site-b-1-1: within 5 s of the node rejoining, site-b-1 was back to 2 inbound and central-1 back to 6 outbound, with zero ongoing telemetry failures. The generation-stamped reconnect recovered the stream automatically. (The same recovery was independently observed as a transient at central startup, when central briefly dialed its own/peer server before it finished booting — failure-streak-1 → reconnect → 6 streams.)

Auth — covered by the Task 6 integration test (not re-probed live)

TelemetryListenerTests (Host.IntegrationTests) already proves the mapped, interceptor-gated endpoint with two real gRPC calls: no bearer ⇒ PermissionDenied (which can only originate in TelemetryStreamAuthInterceptor on a mapped service — an unmapped service returns Unimplemented), and correct bearer ⇒ a clean stream. The rig ran with matching keys throughout; a key mismatch would have surfaced as PermissionDenied reconnect-spam (it did not).

Finding — ClusterNode.GrpcPort must be populated on an EXISTING deployment (upgrade gotcha)

The first Grpc flip produced no dial connections. Root cause was not a code defect: all six ClusterNode rows had GrpcPort = NULL. The rig's SQL volume persisted from before this branch, and the seed is idempotent INSERT-if-not-exists — so it never backfilled the new nullable GrpcPort onto pre-existing rows (AkkaPort was non-null only because its Phase-1 column carries a NOT NULL default 4053; GrpcPort is nullable with no default). The dial supervisor behaved correctly: it spawned in Grpc mode, ran discovery, and skipped all six nodes with the throttled ClusterNode <id> has no GrpcPort … skipped … (further skips of this node are silent) warning (the exact throttle added in Task 8's review fix). Populating GrpcPort = 4056 and bouncing the central pair produced the full 12-stream mesh immediately.

Operational takeaway (record in the upgrade notes): a fresh install seeds GrpcPort correctly (the Phase-11 seed INSERT includes it), but an existing deployment upgrading to Phase 5 must populate ClusterNode.GrpcPort per node — via the AdminUI node edit (Phase 1 surfaced the field) or a data update — or the telemetry dial silently finds no targets (loud-but-throttled warning, last-known DPS panels unaffected since the flip is per-node). Optional rig follow-up: make the docker-dev seed UPSERT the port columns so a persisted volume backfills on re-run (not blocking; this volume is now populated).

Cleanup

docker-compose.override.yml removed; rig recreated to the Dps default. ClusterNode.GrpcPort left populated at 4056 (correct — matches the fresh-install seed intent).

Verdict

Phase 5 live gate PASSED. Driver nodes host the telemetry gRPC server; central dials in only in Grpc mode; the full stream mesh forms from DB-sourced discovery; real telemetry reaches the AdminUI panels; kill-and-reconnect recovers every stream. The one issue surfaced was a rig/upgrade data gap (GrpcPort backfill), not a Phase-5 code defect — and it validated the graceful null-GrpcPort handling live.