Joseph Doherty
f134935f99
Full 12-stream telemetry mesh formed in Grpc mode, AdminUI pill live (data path proven), kill-and-reconnect recovered a node's stream in ~5s; Dps baseline dials nothing. Surfaced+validated the ClusterNode.GrpcPort upgrade-backfill gotcha. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
103 lines
6.6 KiB
Markdown
103 lines
6.6 KiB
Markdown
# Per-Cluster Mesh Phase 5 — Live Gate Record
|
|
|
|
**Date:** 2026-07-23
|
|
**Branch:** `feat/mesh-phase5` (Phase-5 image built + rig recreated with Phase-5 code)
|
|
**Rig:** local `docker-dev` — central-1/2 (`admin,driver`, fused), site-a-1/2, site-b-1/2 (`driver`).
|
|
Telemetry served on **:4056** (h2c), shared key `telemetry-docker-dev-key`.
|
|
|
|
**Result: PASSED.** Every load-bearing exit-gate leg proven live. In `Telemetry:Mode=Grpc`, central
|
|
dials each driver node's telemetry gRPC server, the full stream mesh forms, real telemetry traverses
|
|
node→hub→gRPC→central→sink (AdminUI pill goes live), and kill-and-reconnect of a node recovers its
|
|
stream automatically. In the default `Dps` mode the node hosts the server but central does not dial.
|
|
|
|
Evidence was gathered primarily at the **TCP-socket level** (`/proc/net/tcp6`, port 4056 = hex
|
|
`0FD8`) because it is decisive and log-independent: an established stream is an established stream.
|
|
|
|
## What was verified
|
|
|
|
### Leg 0 — Dps baseline (headline: node always hosts, central does NOT dial)
|
|
Fresh rig in the committed default (`Telemetry:Mode` unset ⇒ `Dps`). site-a-1 log:
|
|
`Now listening on: http://[::]:4056` — the dedicated h2c telemetry listener binds in Dps mode too
|
|
(the node **always** hosts the server). Socket check: site-a-1 has a **LISTEN** on `[::]:4056` and
|
|
**zero** inbound ESTABLISHED — central does not dial in Dps mode. Cluster formed clean (central-1 sees
|
|
central-2 + all four site nodes at `Up`), OPC UA up, drivers subscribed. Baseline correct.
|
|
|
|
### Leg 1 — Grpc flip → full stream mesh forms ✅
|
|
Flipped via a `docker-compose.override.yml` (gate artifact, since removed) setting
|
|
`TelemetryDial__Mode=Grpc` on the central pair and `Telemetry__Mode=Grpc` on all six driver nodes,
|
|
then recreated. After the central pair discovered the (populated) `ClusterNode.GrpcPort` rows and
|
|
dialed:
|
|
|
|
| Node | inbound telemetry streams on :4056 |
|
|
|---|---|
|
|
| central-1 | 2 |
|
|
| central-2 | 2 |
|
|
| site-a-1 | 2 |
|
|
| site-a-2 | 2 |
|
|
| site-b-1 | 2 |
|
|
| site-b-2 | 2 |
|
|
|
|
Both central nodes each held **6 outbound** streams (one per driver node incl. self). **12 streams
|
|
total** — the exact designed topology (both admins dial every driver node). Zero telemetry failures in
|
|
the steady window. Dial happens **only** in Grpc mode (Leg 0 had zero); discovery is DB-sourced from
|
|
`ClusterNode` (Host + GrpcPort), matching Phase 1 + the rig seed.
|
|
|
|
### Leg 2 — data path end-to-end (AdminUI pill live) ✅
|
|
`http://localhost:9200/alerts` connection pill read **"live"**. This is the load-bearing data-path
|
|
proof: the dial supervisor marks a node **Connected — and flips both broadcasters' `SetConnected(true)`
|
|
— only on its FIRST received telemetry event**. So "live" means a real telemetry event (the
|
|
5 s-periodic `driver-resilience-status`) traversed the node's publish seam → node-local hub → gRPC
|
|
stream → central dial supervisor → `TelemetryReceived` → the in-process sink the Blazor panel reads.
|
|
Transport *and* payload proven, not just a connected socket. (The panel's static "…from the cluster's
|
|
DPS topic" caption is now stale wording — cosmetic, the broadcaster seam is unchanged, only its
|
|
upstream swapped; noted as a doc-copy follow-up, non-blocking.)
|
|
|
|
### Leg 3 — kill-and-reconnect recovers every stream ✅
|
|
`docker kill otopcua-dev-site-b-1-1`:
|
|
- central-1 logged `Telemetry stream to node site-b-1:4053 failed (failure streak 1); reconnecting`.
|
|
- central-1 outbound streams dropped 6 → 5; the pill stayed **live** (the other 5 streams carried the
|
|
panel — a single node loss does not blank the fleet view).
|
|
|
|
`docker start otopcua-dev-site-b-1-1`: within **5 s** of the node rejoining, site-b-1 was back to **2
|
|
inbound** and central-1 back to **6 outbound**, with **zero** ongoing telemetry failures. The
|
|
generation-stamped reconnect recovered the stream automatically. (The same recovery was independently
|
|
observed as a transient at central startup, when central briefly dialed its own/peer server before it
|
|
finished booting — failure-streak-1 → reconnect → 6 streams.)
|
|
|
|
### Auth — covered by the Task 6 integration test (not re-probed live)
|
|
`TelemetryListenerTests` (Host.IntegrationTests) already proves the mapped, interceptor-gated endpoint
|
|
with two real gRPC calls: **no bearer ⇒ `PermissionDenied`** (which can only originate in
|
|
`TelemetryStreamAuthInterceptor` on a *mapped* service — an unmapped service returns `Unimplemented`),
|
|
and correct bearer ⇒ a clean stream. The rig ran with matching keys throughout; a key mismatch would
|
|
have surfaced as `PermissionDenied` reconnect-spam (it did not).
|
|
|
|
## Finding — `ClusterNode.GrpcPort` must be populated on an EXISTING deployment (upgrade gotcha)
|
|
The first Grpc flip produced **no** dial connections. Root cause was **not** a code defect: all six
|
|
`ClusterNode` rows had `GrpcPort = NULL`. The rig's SQL volume persisted from before this branch, and
|
|
the seed is idempotent **INSERT-if-not-exists** — so it never backfilled the new nullable `GrpcPort`
|
|
onto pre-existing rows (`AkkaPort` was non-null only because its Phase-1 column carries a NOT NULL
|
|
default `4053`; `GrpcPort` is nullable with no default). The dial supervisor behaved **correctly**: it
|
|
spawned in Grpc mode, ran discovery, and skipped all six nodes with the throttled
|
|
`ClusterNode <id> has no GrpcPort … skipped … (further skips of this node are silent)` warning (the
|
|
exact throttle added in Task 8's review fix). Populating `GrpcPort = 4056` and bouncing the central
|
|
pair produced the full 12-stream mesh immediately.
|
|
|
|
**Operational takeaway (record in the upgrade notes):** a **fresh** install seeds `GrpcPort` correctly
|
|
(the Phase-11 seed INSERT includes it), but an **existing** deployment upgrading to Phase 5 must
|
|
populate `ClusterNode.GrpcPort` per node — via the AdminUI node edit (Phase 1 surfaced the field) or a
|
|
data update — or the telemetry dial silently finds no targets (loud-but-throttled warning, last-known
|
|
DPS panels unaffected since the flip is per-node). Optional rig follow-up: make the docker-dev seed
|
|
UPSERT the port columns so a persisted volume backfills on re-run (not blocking; this volume is now
|
|
populated).
|
|
|
|
## Cleanup
|
|
`docker-compose.override.yml` removed; rig recreated to the `Dps` default. `ClusterNode.GrpcPort`
|
|
left populated at 4056 (correct — matches the fresh-install seed intent).
|
|
|
|
## Verdict
|
|
**Phase 5 live gate PASSED.** Driver nodes host the telemetry gRPC server; central dials in only in
|
|
Grpc mode; the full stream mesh forms from DB-sourced discovery; real telemetry reaches the AdminUI
|
|
panels; kill-and-reconnect recovers every stream. The one issue surfaced was a rig/upgrade data gap
|
|
(`GrpcPort` backfill), not a Phase-5 code defect — and it validated the graceful null-`GrpcPort`
|
|
handling live.
|