# Per-Cluster Mesh Phase 5 — Live Gate Record

**Date:** 2026-07-23
**Branch:** `feat/mesh-phase5` (Phase-5 image built + rig recreated with Phase-5 code)
**Rig:** local `docker-dev` — central-1/2 (`admin,driver`, fused), site-a-1/2, site-b-1/2 (`driver`).
Telemetry served on **:4056** (h2c), shared key `telemetry-docker-dev-key`.

**Result: PASSED.** Every load-bearing exit-gate leg proven live. In `Telemetry:Mode=Grpc`, central
dials each driver node's telemetry gRPC server, the full stream mesh forms, real telemetry traverses
node→hub→gRPC→central→sink (AdminUI pill goes live), and kill-and-reconnect of a node recovers its
stream automatically. In the default `Dps` mode the node hosts the server but central does not dial.

Evidence was gathered primarily at the **TCP-socket level** (`/proc/net/tcp6`, port 4056 = hex
`0FD8`) because it is decisive and log-independent: an established stream is an established stream.

## What was verified

### Leg 0 — Dps baseline (headline: node always hosts, central does NOT dial)
Fresh rig in the committed default (`Telemetry:Mode` unset ⇒ `Dps`). site-a-1 log:
`Now listening on: http://[::]:4056` — the dedicated h2c telemetry listener binds in Dps mode too
(the node **always** hosts the server). Socket check: site-a-1 has a **LISTEN** on `[::]:4056` and
**zero** inbound ESTABLISHED — central does not dial in Dps mode. Cluster formed clean (central-1 sees
central-2 + all four site nodes at `Up`), OPC UA up, drivers subscribed. Baseline correct.

### Leg 1 — Grpc flip → full stream mesh forms  ✅
Flipped via a `docker-compose.override.yml` (gate artifact, since removed) setting
`TelemetryDial__Mode=Grpc` on the central pair and `Telemetry__Mode=Grpc` on all six driver nodes,
then recreated. After the central pair discovered the (populated) `ClusterNode.GrpcPort` rows and
dialed:

| Node | inbound telemetry streams on :4056 |
|---|---|
| central-1 | 2 |
| central-2 | 2 |
| site-a-1 | 2 |
| site-a-2 | 2 |
| site-b-1 | 2 |
| site-b-2 | 2 |

Both central nodes each held **6 outbound** streams (one per driver node incl. self). **12 streams
total** — the exact designed topology (both admins dial every driver node). Zero telemetry failures in
the steady window. Dial happens **only** in Grpc mode (Leg 0 had zero); discovery is DB-sourced from
`ClusterNode` (Host + GrpcPort), matching Phase 1 + the rig seed.

### Leg 2 — data path end-to-end (AdminUI pill live)  ✅
`http://localhost:9200/alerts` connection pill read **"live"**. This is the load-bearing data-path
proof: the dial supervisor marks a node **Connected — and flips both broadcasters' `SetConnected(true)`
— only on its FIRST received telemetry event**. So "live" means a real telemetry event (the
5 s-periodic `driver-resilience-status`) traversed the node's publish seam → node-local hub → gRPC
stream → central dial supervisor → `TelemetryReceived` → the in-process sink the Blazor panel reads.
Transport *and* payload proven, not just a connected socket. (The panel's static "…from the cluster's
DPS topic" caption is now stale wording — cosmetic, the broadcaster seam is unchanged, only its
upstream swapped; noted as a doc-copy follow-up, non-blocking.)

### Leg 3 — kill-and-reconnect recovers every stream  ✅
`docker kill otopcua-dev-site-b-1-1`:
- central-1 logged `Telemetry stream to node site-b-1:4053 failed (failure streak 1); reconnecting`.
- central-1 outbound streams dropped 6 → 5; the pill stayed **live** (the other 5 streams carried the
  panel — a single node loss does not blank the fleet view).

`docker start otopcua-dev-site-b-1-1`: within **5 s** of the node rejoining, site-b-1 was back to **2
inbound** and central-1 back to **6 outbound**, with **zero** ongoing telemetry failures. The
generation-stamped reconnect recovered the stream automatically. (The same recovery was independently
observed as a transient at central startup, when central briefly dialed its own/peer server before it
finished booting — failure-streak-1 → reconnect → 6 streams.)

### Auth — covered by the Task 6 integration test (not re-probed live)
`TelemetryListenerTests` (Host.IntegrationTests) already proves the mapped, interceptor-gated endpoint
with two real gRPC calls: **no bearer ⇒ `PermissionDenied`** (which can only originate in
`TelemetryStreamAuthInterceptor` on a *mapped* service — an unmapped service returns `Unimplemented`),
and correct bearer ⇒ a clean stream. The rig ran with matching keys throughout; a key mismatch would
have surfaced as `PermissionDenied` reconnect-spam (it did not).

## Finding — `ClusterNode.GrpcPort` must be populated on an EXISTING deployment (upgrade gotcha)
The first Grpc flip produced **no** dial connections. Root cause was **not** a code defect: all six
`ClusterNode` rows had `GrpcPort = NULL`. The rig's SQL volume persisted from before this branch, and
the seed is idempotent **INSERT-if-not-exists** — so it never backfilled the new nullable `GrpcPort`
onto pre-existing rows (`AkkaPort` was non-null only because its Phase-1 column carries a NOT NULL
default `4053`; `GrpcPort` is nullable with no default). The dial supervisor behaved **correctly**: it
spawned in Grpc mode, ran discovery, and skipped all six nodes with the throttled
`ClusterNode <id> has no GrpcPort … skipped … (further skips of this node are silent)` warning (the
exact throttle added in Task 8's review fix). Populating `GrpcPort = 4056` and bouncing the central
pair produced the full 12-stream mesh immediately.

**Operational takeaway (record in the upgrade notes):** a **fresh** install seeds `GrpcPort` correctly
(the Phase-11 seed INSERT includes it), but an **existing** deployment upgrading to Phase 5 must
populate `ClusterNode.GrpcPort` per node — via the AdminUI node edit (Phase 1 surfaced the field) or a
data update — or the telemetry dial silently finds no targets (loud-but-throttled warning, last-known
DPS panels unaffected since the flip is per-node). Optional rig follow-up: make the docker-dev seed
UPSERT the port columns so a persisted volume backfills on re-run (not blocking; this volume is now
populated).

## Cleanup
`docker-compose.override.yml` removed; rig recreated to the `Dps` default. `ClusterNode.GrpcPort`
left populated at 4056 (correct — matches the fresh-install seed intent).

## Verdict
**Phase 5 live gate PASSED.** Driver nodes host the telemetry gRPC server; central dials in only in
Grpc mode; the full stream mesh forms from DB-sourced discovery; real telemetry reaches the AdminUI
panels; kill-and-reconnect recovers every stream. The one issue surfaced was a rig/upgrade data gap
(`GrpcPort` backfill), not a Phase-5 code defect — and it validated the graceful null-`GrpcPort`
handling live.