diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index b1beff58..15d5e7f6 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -202,7 +202,7 @@ Phase 4 as code-complete, not verified, until this lands.** **no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can resolve the ConfigDb context. -### Phase 5 — gRPC stream contract for live telemetry — **code-complete on `feat/mesh-phase5`, live gate pending** +### Phase 5 — gRPC stream contract for live telemetry — **DONE 2026-07-23 (live gate PASSED)** **Scope as shipped (narrowed from the program sketch's "seven observability topics," settled with the user 2026-07-23 — mirrors how Phases 1 and 3 also honestly scoped down):** one server-streaming gRPC contract carrying a `oneof` event with **four** kinds, **driver nodes host the gRPC server (dedicated @@ -317,6 +317,6 @@ resource sizing on the site VMs should be checked once both products run the ful | 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. | | 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | | 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. | -| 5 gRPC telemetry | **code-complete (live gate pending)** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); auth-from-day-one fail-closed bearer key, superseding design §6.3. See `docs/Telemetry.md` + `2026-07-23-mesh-phase5-grpc-telemetry-stream.md`. | +| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. | | 6 mesh partition + co-location | not started | | 7 drill + live gates | not started | diff --git a/docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md.tasks.json b/docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md.tasks.json index a07f1ee1..e5a5a79e 100644 --- a/docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md.tasks.json @@ -16,7 +16,7 @@ {"id": 9, "subject": "Task 9: dark-switch central telemetry ingest (Dps bridges | Grpc)", "classification": "standard", "status": "completed", "commit": "d15c613e", "blockedBy": [8]}, {"id": 10, "subject": "Task 10: docs — Telemetry.md, config, supersede §6.3, program status","classification": "small", "status": "completed", "commit": "e99ea40e", "parallelizableWith": [11], "blockedBy": [9]}, {"id": 11, "subject": "Task 11: rig telemetry ports + shared key + ClusterNode.GrpcPort seed","classification": "small", "status": "completed", "commit": "ffb75b78", "note": "port 4056, docker compose config validated", "parallelizableWith": [10], "blockedBy": [6, 9]}, - {"id": 12, "subject": "Task 12: live gate (panels green over gRPC + kill-and-reconnect)", "classification": "high-risk","status": "pending", "blockedBy": [10, 11]} + {"id": 12, "subject": "Task 12: live gate (panels green over gRPC + kill-and-reconnect)", "classification": "high-risk","status": "completed", "note": "PASSED — 12-stream mesh, pill live, kill/reconnect recovered; found GrpcPort upgrade gotcha. Record: 2026-07-23-mesh-phase5-live-gate.md", "blockedBy": [10, 11]} ], "lastUpdated": "2026-07-23T00:00:00Z" } diff --git a/docs/plans/2026-07-23-mesh-phase5-live-gate.md b/docs/plans/2026-07-23-mesh-phase5-live-gate.md new file mode 100644 index 00000000..63c8f190 --- /dev/null +++ b/docs/plans/2026-07-23-mesh-phase5-live-gate.md @@ -0,0 +1,102 @@ +# Per-Cluster Mesh Phase 5 — Live Gate Record + +**Date:** 2026-07-23 +**Branch:** `feat/mesh-phase5` (Phase-5 image built + rig recreated with Phase-5 code) +**Rig:** local `docker-dev` — central-1/2 (`admin,driver`, fused), site-a-1/2, site-b-1/2 (`driver`). +Telemetry served on **:4056** (h2c), shared key `telemetry-docker-dev-key`. + +**Result: PASSED.** Every load-bearing exit-gate leg proven live. In `Telemetry:Mode=Grpc`, central +dials each driver node's telemetry gRPC server, the full stream mesh forms, real telemetry traverses +node→hub→gRPC→central→sink (AdminUI pill goes live), and kill-and-reconnect of a node recovers its +stream automatically. In the default `Dps` mode the node hosts the server but central does not dial. + +Evidence was gathered primarily at the **TCP-socket level** (`/proc/net/tcp6`, port 4056 = hex +`0FD8`) because it is decisive and log-independent: an established stream is an established stream. + +## What was verified + +### Leg 0 — Dps baseline (headline: node always hosts, central does NOT dial) +Fresh rig in the committed default (`Telemetry:Mode` unset ⇒ `Dps`). site-a-1 log: +`Now listening on: http://[::]:4056` — the dedicated h2c telemetry listener binds in Dps mode too +(the node **always** hosts the server). Socket check: site-a-1 has a **LISTEN** on `[::]:4056` and +**zero** inbound ESTABLISHED — central does not dial in Dps mode. Cluster formed clean (central-1 sees +central-2 + all four site nodes at `Up`), OPC UA up, drivers subscribed. Baseline correct. + +### Leg 1 — Grpc flip → full stream mesh forms ✅ +Flipped via a `docker-compose.override.yml` (gate artifact, since removed) setting +`TelemetryDial__Mode=Grpc` on the central pair and `Telemetry__Mode=Grpc` on all six driver nodes, +then recreated. After the central pair discovered the (populated) `ClusterNode.GrpcPort` rows and +dialed: + +| Node | inbound telemetry streams on :4056 | +|---|---| +| central-1 | 2 | +| central-2 | 2 | +| site-a-1 | 2 | +| site-a-2 | 2 | +| site-b-1 | 2 | +| site-b-2 | 2 | + +Both central nodes each held **6 outbound** streams (one per driver node incl. self). **12 streams +total** — the exact designed topology (both admins dial every driver node). Zero telemetry failures in +the steady window. Dial happens **only** in Grpc mode (Leg 0 had zero); discovery is DB-sourced from +`ClusterNode` (Host + GrpcPort), matching Phase 1 + the rig seed. + +### Leg 2 — data path end-to-end (AdminUI pill live) ✅ +`http://localhost:9200/alerts` connection pill read **"live"**. This is the load-bearing data-path +proof: the dial supervisor marks a node **Connected — and flips both broadcasters' `SetConnected(true)` +— only on its FIRST received telemetry event**. So "live" means a real telemetry event (the +5 s-periodic `driver-resilience-status`) traversed the node's publish seam → node-local hub → gRPC +stream → central dial supervisor → `TelemetryReceived` → the in-process sink the Blazor panel reads. +Transport *and* payload proven, not just a connected socket. (The panel's static "…from the cluster's +DPS topic" caption is now stale wording — cosmetic, the broadcaster seam is unchanged, only its +upstream swapped; noted as a doc-copy follow-up, non-blocking.) + +### Leg 3 — kill-and-reconnect recovers every stream ✅ +`docker kill otopcua-dev-site-b-1-1`: +- central-1 logged `Telemetry stream to node site-b-1:4053 failed (failure streak 1); reconnecting`. +- central-1 outbound streams dropped 6 → 5; the pill stayed **live** (the other 5 streams carried the + panel — a single node loss does not blank the fleet view). + +`docker start otopcua-dev-site-b-1-1`: within **5 s** of the node rejoining, site-b-1 was back to **2 +inbound** and central-1 back to **6 outbound**, with **zero** ongoing telemetry failures. The +generation-stamped reconnect recovered the stream automatically. (The same recovery was independently +observed as a transient at central startup, when central briefly dialed its own/peer server before it +finished booting — failure-streak-1 → reconnect → 6 streams.) + +### Auth — covered by the Task 6 integration test (not re-probed live) +`TelemetryListenerTests` (Host.IntegrationTests) already proves the mapped, interceptor-gated endpoint +with two real gRPC calls: **no bearer ⇒ `PermissionDenied`** (which can only originate in +`TelemetryStreamAuthInterceptor` on a *mapped* service — an unmapped service returns `Unimplemented`), +and correct bearer ⇒ a clean stream. The rig ran with matching keys throughout; a key mismatch would +have surfaced as `PermissionDenied` reconnect-spam (it did not). + +## Finding — `ClusterNode.GrpcPort` must be populated on an EXISTING deployment (upgrade gotcha) +The first Grpc flip produced **no** dial connections. Root cause was **not** a code defect: all six +`ClusterNode` rows had `GrpcPort = NULL`. The rig's SQL volume persisted from before this branch, and +the seed is idempotent **INSERT-if-not-exists** — so it never backfilled the new nullable `GrpcPort` +onto pre-existing rows (`AkkaPort` was non-null only because its Phase-1 column carries a NOT NULL +default `4053`; `GrpcPort` is nullable with no default). The dial supervisor behaved **correctly**: it +spawned in Grpc mode, ran discovery, and skipped all six nodes with the throttled +`ClusterNode has no GrpcPort … skipped … (further skips of this node are silent)` warning (the +exact throttle added in Task 8's review fix). Populating `GrpcPort = 4056` and bouncing the central +pair produced the full 12-stream mesh immediately. + +**Operational takeaway (record in the upgrade notes):** a **fresh** install seeds `GrpcPort` correctly +(the Phase-11 seed INSERT includes it), but an **existing** deployment upgrading to Phase 5 must +populate `ClusterNode.GrpcPort` per node — via the AdminUI node edit (Phase 1 surfaced the field) or a +data update — or the telemetry dial silently finds no targets (loud-but-throttled warning, last-known +DPS panels unaffected since the flip is per-node). Optional rig follow-up: make the docker-dev seed +UPSERT the port columns so a persisted volume backfills on re-run (not blocking; this volume is now +populated). + +## Cleanup +`docker-compose.override.yml` removed; rig recreated to the `Dps` default. `ClusterNode.GrpcPort` +left populated at 4056 (correct — matches the fresh-install seed intent). + +## Verdict +**Phase 5 live gate PASSED.** Driver nodes host the telemetry gRPC server; central dials in only in +Grpc mode; the full stream mesh forms from DB-sourced discovery; real telemetry reaches the AdminUI +panels; kill-and-reconnect recovers every stream. The one issue surfaced was a rig/upgrade data gap +(`GrpcPort` backfill), not a Phase-5 code defect — and it validated the graceful null-`GrpcPort` +handling live.