docs(mesh-phase5): live gate PASSED — record + program status DONE

Full 12-stream telemetry mesh formed in Grpc mode, AdminUI pill live (data path
proven), kill-and-reconnect recovered a node's stream in ~5s; Dps baseline dials
nothing. Surfaced+validated the ClusterNode.GrpcPort upgrade-backfill gotcha.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

docs/plans/2026-07-23-mesh-phase5-live-gate.md

@@ -0,0 +1,102 @@
+# Per-Cluster Mesh Phase 5 — Live Gate Record
+
+**Date:** 2026-07-23
+**Branch:** `feat/mesh-phase5` (Phase-5 image built + rig recreated with Phase-5 code)
+**Rig:** local `docker-dev` — central-1/2 (`admin,driver`, fused), site-a-1/2, site-b-1/2 (`driver`).
+Telemetry served on **:4056** (h2c), shared key `telemetry-docker-dev-key`.
+
+**Result: PASSED.** Every load-bearing exit-gate leg proven live. In `Telemetry:Mode=Grpc`, central
+dials each driver node's telemetry gRPC server, the full stream mesh forms, real telemetry traverses
+node→hub→gRPC→central→sink (AdminUI pill goes live), and kill-and-reconnect of a node recovers its
+stream automatically. In the default `Dps` mode the node hosts the server but central does not dial.
+
+Evidence was gathered primarily at the **TCP-socket level** (`/proc/net/tcp6`, port 4056 = hex
+`0FD8`) because it is decisive and log-independent: an established stream is an established stream.
+
+## What was verified
+
+### Leg 0 — Dps baseline (headline: node always hosts, central does NOT dial)
+Fresh rig in the committed default (`Telemetry:Mode` unset ⇒ `Dps`). site-a-1 log:
+`Now listening on: http://[::]:4056` — the dedicated h2c telemetry listener binds in Dps mode too
+(the node **always** hosts the server). Socket check: site-a-1 has a **LISTEN** on `[::]:4056` and
+**zero** inbound ESTABLISHED — central does not dial in Dps mode. Cluster formed clean (central-1 sees
+central-2 + all four site nodes at `Up`), OPC UA up, drivers subscribed. Baseline correct.
+
+### Leg 1 — Grpc flip → full stream mesh forms  ✅
+Flipped via a `docker-compose.override.yml` (gate artifact, since removed) setting
+`TelemetryDial__Mode=Grpc` on the central pair and `Telemetry__Mode=Grpc` on all six driver nodes,
+then recreated. After the central pair discovered the (populated) `ClusterNode.GrpcPort` rows and
+dialed:
+
+| Node | inbound telemetry streams on :4056 |
+|---|---|
+| central-1 | 2 |
+| central-2 | 2 |
+| site-a-1 | 2 |
+| site-a-2 | 2 |
+| site-b-1 | 2 |
+| site-b-2 | 2 |
+
+Both central nodes each held **6 outbound** streams (one per driver node incl. self). **12 streams
+total** — the exact designed topology (both admins dial every driver node). Zero telemetry failures in
+the steady window. Dial happens **only** in Grpc mode (Leg 0 had zero); discovery is DB-sourced from
+`ClusterNode` (Host + GrpcPort), matching Phase 1 + the rig seed.
+
+### Leg 2 — data path end-to-end (AdminUI pill live)  ✅
+`http://localhost:9200/alerts` connection pill read **"live"**. This is the load-bearing data-path
+proof: the dial supervisor marks a node **Connected — and flips both broadcasters' `SetConnected(true)`
+— only on its FIRST received telemetry event**. So "live" means a real telemetry event (the
+5 s-periodic `driver-resilience-status`) traversed the node's publish seam → node-local hub → gRPC
+stream → central dial supervisor → `TelemetryReceived` → the in-process sink the Blazor panel reads.
+Transport *and* payload proven, not just a connected socket. (The panel's static "…from the cluster's
+DPS topic" caption is now stale wording — cosmetic, the broadcaster seam is unchanged, only its
+upstream swapped; noted as a doc-copy follow-up, non-blocking.)
+
+### Leg 3 — kill-and-reconnect recovers every stream  ✅
+`docker kill otopcua-dev-site-b-1-1`:
+- central-1 logged `Telemetry stream to node site-b-1:4053 failed (failure streak 1); reconnecting`.
+- central-1 outbound streams dropped 6 → 5; the pill stayed **live** (the other 5 streams carried the
+  panel — a single node loss does not blank the fleet view).
+
+`docker start otopcua-dev-site-b-1-1`: within **5 s** of the node rejoining, site-b-1 was back to **2
+inbound** and central-1 back to **6 outbound**, with **zero** ongoing telemetry failures. The
+generation-stamped reconnect recovered the stream automatically. (The same recovery was independently
+observed as a transient at central startup, when central briefly dialed its own/peer server before it
+finished booting — failure-streak-1 → reconnect → 6 streams.)
+
+### Auth — covered by the Task 6 integration test (not re-probed live)
+`TelemetryListenerTests` (Host.IntegrationTests) already proves the mapped, interceptor-gated endpoint
+with two real gRPC calls: **no bearer ⇒ `PermissionDenied`** (which can only originate in
+`TelemetryStreamAuthInterceptor` on a *mapped* service — an unmapped service returns `Unimplemented`),
+and correct bearer ⇒ a clean stream. The rig ran with matching keys throughout; a key mismatch would
+have surfaced as `PermissionDenied` reconnect-spam (it did not).
+
+## Finding — `ClusterNode.GrpcPort` must be populated on an EXISTING deployment (upgrade gotcha)
+The first Grpc flip produced **no** dial connections. Root cause was **not** a code defect: all six
+`ClusterNode` rows had `GrpcPort = NULL`. The rig's SQL volume persisted from before this branch, and
+the seed is idempotent **INSERT-if-not-exists** — so it never backfilled the new nullable `GrpcPort`
+onto pre-existing rows (`AkkaPort` was non-null only because its Phase-1 column carries a NOT NULL
+default `4053`; `GrpcPort` is nullable with no default). The dial supervisor behaved **correctly**: it
+spawned in Grpc mode, ran discovery, and skipped all six nodes with the throttled
+`ClusterNode <id> has no GrpcPort … skipped … (further skips of this node are silent)` warning (the
+exact throttle added in Task 8's review fix). Populating `GrpcPort = 4056` and bouncing the central
+pair produced the full 12-stream mesh immediately.
+
+**Operational takeaway (record in the upgrade notes):** a **fresh** install seeds `GrpcPort` correctly
+(the Phase-11 seed INSERT includes it), but an **existing** deployment upgrading to Phase 5 must
+populate `ClusterNode.GrpcPort` per node — via the AdminUI node edit (Phase 1 surfaced the field) or a
+data update — or the telemetry dial silently finds no targets (loud-but-throttled warning, last-known
+DPS panels unaffected since the flip is per-node). Optional rig follow-up: make the docker-dev seed
+UPSERT the port columns so a persisted volume backfills on re-run (not blocking; this volume is now
+populated).
+
+## Cleanup
+`docker-compose.override.yml` removed; rig recreated to the `Dps` default. `ClusterNode.GrpcPort`
+left populated at 4056 (correct — matches the fresh-install seed intent).
+
+## Verdict
+**Phase 5 live gate PASSED.** Driver nodes host the telemetry gRPC server; central dials in only in
+Grpc mode; the full stream mesh forms from DB-sourced discovery; real telemetry reaches the AdminUI
+panels; kill-and-reconnect recovers every stream. The one issue surfaced was a rig/upgrade data gap
+(`GrpcPort` backfill), not a Phase-5 code defect — and it validated the graceful null-`GrpcPort`
+handling live.