docs(mesh-phase5): live gate PASSED — record + program status DONE
Full 12-stream telemetry mesh formed in Grpc mode, AdminUI pill live (data path proven), kill-and-reconnect recovered a node's stream in ~5s; Dps baseline dials nothing. Surfaced+validated the ClusterNode.GrpcPort upgrade-backfill gotcha. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -202,7 +202,7 @@ Phase 4 as code-complete, not verified, until this lands.**
|
||||
**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
|
||||
resolve the ConfigDb context.
|
||||
|
||||
### Phase 5 — gRPC stream contract for live telemetry — **code-complete on `feat/mesh-phase5`, live gate pending**
|
||||
### Phase 5 — gRPC stream contract for live telemetry — **DONE 2026-07-23 (live gate PASSED)**
|
||||
**Scope as shipped (narrowed from the program sketch's "seven observability topics," settled with the
|
||||
user 2026-07-23 — mirrors how Phases 1 and 3 also honestly scoped down):** one server-streaming gRPC
|
||||
contract carrying a `oneof` event with **four** kinds, **driver nodes host the gRPC server (dedicated
|
||||
@@ -317,6 +317,6 @@ resource sizing on the site VMs should be checked once both products run the ful
|
||||
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
|
||||
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
|
||||
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
|
||||
| 5 gRPC telemetry | **code-complete (live gate pending)** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); auth-from-day-one fail-closed bearer key, superseding design §6.3. See `docs/Telemetry.md` + `2026-07-23-mesh-phase5-grpc-telemetry-stream.md`. |
|
||||
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
|
||||
| 6 mesh partition + co-location | not started |
|
||||
| 7 drill + live gates | not started |
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
{"id": 9, "subject": "Task 9: dark-switch central telemetry ingest (Dps bridges | Grpc)", "classification": "standard", "status": "completed", "commit": "d15c613e", "blockedBy": [8]},
|
||||
{"id": 10, "subject": "Task 10: docs — Telemetry.md, config, supersede §6.3, program status","classification": "small", "status": "completed", "commit": "e99ea40e", "parallelizableWith": [11], "blockedBy": [9]},
|
||||
{"id": 11, "subject": "Task 11: rig telemetry ports + shared key + ClusterNode.GrpcPort seed","classification": "small", "status": "completed", "commit": "ffb75b78", "note": "port 4056, docker compose config validated", "parallelizableWith": [10], "blockedBy": [6, 9]},
|
||||
{"id": 12, "subject": "Task 12: live gate (panels green over gRPC + kill-and-reconnect)", "classification": "high-risk","status": "pending", "blockedBy": [10, 11]}
|
||||
{"id": 12, "subject": "Task 12: live gate (panels green over gRPC + kill-and-reconnect)", "classification": "high-risk","status": "completed", "note": "PASSED — 12-stream mesh, pill live, kill/reconnect recovered; found GrpcPort upgrade gotcha. Record: 2026-07-23-mesh-phase5-live-gate.md", "blockedBy": [10, 11]}
|
||||
],
|
||||
"lastUpdated": "2026-07-23T00:00:00Z"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
# Per-Cluster Mesh Phase 5 — Live Gate Record
|
||||
|
||||
**Date:** 2026-07-23
|
||||
**Branch:** `feat/mesh-phase5` (Phase-5 image built + rig recreated with Phase-5 code)
|
||||
**Rig:** local `docker-dev` — central-1/2 (`admin,driver`, fused), site-a-1/2, site-b-1/2 (`driver`).
|
||||
Telemetry served on **:4056** (h2c), shared key `telemetry-docker-dev-key`.
|
||||
|
||||
**Result: PASSED.** Every load-bearing exit-gate leg proven live. In `Telemetry:Mode=Grpc`, central
|
||||
dials each driver node's telemetry gRPC server, the full stream mesh forms, real telemetry traverses
|
||||
node→hub→gRPC→central→sink (AdminUI pill goes live), and kill-and-reconnect of a node recovers its
|
||||
stream automatically. In the default `Dps` mode the node hosts the server but central does not dial.
|
||||
|
||||
Evidence was gathered primarily at the **TCP-socket level** (`/proc/net/tcp6`, port 4056 = hex
|
||||
`0FD8`) because it is decisive and log-independent: an established stream is an established stream.
|
||||
|
||||
## What was verified
|
||||
|
||||
### Leg 0 — Dps baseline (headline: node always hosts, central does NOT dial)
|
||||
Fresh rig in the committed default (`Telemetry:Mode` unset ⇒ `Dps`). site-a-1 log:
|
||||
`Now listening on: http://[::]:4056` — the dedicated h2c telemetry listener binds in Dps mode too
|
||||
(the node **always** hosts the server). Socket check: site-a-1 has a **LISTEN** on `[::]:4056` and
|
||||
**zero** inbound ESTABLISHED — central does not dial in Dps mode. Cluster formed clean (central-1 sees
|
||||
central-2 + all four site nodes at `Up`), OPC UA up, drivers subscribed. Baseline correct.
|
||||
|
||||
### Leg 1 — Grpc flip → full stream mesh forms ✅
|
||||
Flipped via a `docker-compose.override.yml` (gate artifact, since removed) setting
|
||||
`TelemetryDial__Mode=Grpc` on the central pair and `Telemetry__Mode=Grpc` on all six driver nodes,
|
||||
then recreated. After the central pair discovered the (populated) `ClusterNode.GrpcPort` rows and
|
||||
dialed:
|
||||
|
||||
| Node | inbound telemetry streams on :4056 |
|
||||
|---|---|
|
||||
| central-1 | 2 |
|
||||
| central-2 | 2 |
|
||||
| site-a-1 | 2 |
|
||||
| site-a-2 | 2 |
|
||||
| site-b-1 | 2 |
|
||||
| site-b-2 | 2 |
|
||||
|
||||
Both central nodes each held **6 outbound** streams (one per driver node incl. self). **12 streams
|
||||
total** — the exact designed topology (both admins dial every driver node). Zero telemetry failures in
|
||||
the steady window. Dial happens **only** in Grpc mode (Leg 0 had zero); discovery is DB-sourced from
|
||||
`ClusterNode` (Host + GrpcPort), matching Phase 1 + the rig seed.
|
||||
|
||||
### Leg 2 — data path end-to-end (AdminUI pill live) ✅
|
||||
`http://localhost:9200/alerts` connection pill read **"live"**. This is the load-bearing data-path
|
||||
proof: the dial supervisor marks a node **Connected — and flips both broadcasters' `SetConnected(true)`
|
||||
— only on its FIRST received telemetry event**. So "live" means a real telemetry event (the
|
||||
5 s-periodic `driver-resilience-status`) traversed the node's publish seam → node-local hub → gRPC
|
||||
stream → central dial supervisor → `TelemetryReceived` → the in-process sink the Blazor panel reads.
|
||||
Transport *and* payload proven, not just a connected socket. (The panel's static "…from the cluster's
|
||||
DPS topic" caption is now stale wording — cosmetic, the broadcaster seam is unchanged, only its
|
||||
upstream swapped; noted as a doc-copy follow-up, non-blocking.)
|
||||
|
||||
### Leg 3 — kill-and-reconnect recovers every stream ✅
|
||||
`docker kill otopcua-dev-site-b-1-1`:
|
||||
- central-1 logged `Telemetry stream to node site-b-1:4053 failed (failure streak 1); reconnecting`.
|
||||
- central-1 outbound streams dropped 6 → 5; the pill stayed **live** (the other 5 streams carried the
|
||||
panel — a single node loss does not blank the fleet view).
|
||||
|
||||
`docker start otopcua-dev-site-b-1-1`: within **5 s** of the node rejoining, site-b-1 was back to **2
|
||||
inbound** and central-1 back to **6 outbound**, with **zero** ongoing telemetry failures. The
|
||||
generation-stamped reconnect recovered the stream automatically. (The same recovery was independently
|
||||
observed as a transient at central startup, when central briefly dialed its own/peer server before it
|
||||
finished booting — failure-streak-1 → reconnect → 6 streams.)
|
||||
|
||||
### Auth — covered by the Task 6 integration test (not re-probed live)
|
||||
`TelemetryListenerTests` (Host.IntegrationTests) already proves the mapped, interceptor-gated endpoint
|
||||
with two real gRPC calls: **no bearer ⇒ `PermissionDenied`** (which can only originate in
|
||||
`TelemetryStreamAuthInterceptor` on a *mapped* service — an unmapped service returns `Unimplemented`),
|
||||
and correct bearer ⇒ a clean stream. The rig ran with matching keys throughout; a key mismatch would
|
||||
have surfaced as `PermissionDenied` reconnect-spam (it did not).
|
||||
|
||||
## Finding — `ClusterNode.GrpcPort` must be populated on an EXISTING deployment (upgrade gotcha)
|
||||
The first Grpc flip produced **no** dial connections. Root cause was **not** a code defect: all six
|
||||
`ClusterNode` rows had `GrpcPort = NULL`. The rig's SQL volume persisted from before this branch, and
|
||||
the seed is idempotent **INSERT-if-not-exists** — so it never backfilled the new nullable `GrpcPort`
|
||||
onto pre-existing rows (`AkkaPort` was non-null only because its Phase-1 column carries a NOT NULL
|
||||
default `4053`; `GrpcPort` is nullable with no default). The dial supervisor behaved **correctly**: it
|
||||
spawned in Grpc mode, ran discovery, and skipped all six nodes with the throttled
|
||||
`ClusterNode <id> has no GrpcPort … skipped … (further skips of this node are silent)` warning (the
|
||||
exact throttle added in Task 8's review fix). Populating `GrpcPort = 4056` and bouncing the central
|
||||
pair produced the full 12-stream mesh immediately.
|
||||
|
||||
**Operational takeaway (record in the upgrade notes):** a **fresh** install seeds `GrpcPort` correctly
|
||||
(the Phase-11 seed INSERT includes it), but an **existing** deployment upgrading to Phase 5 must
|
||||
populate `ClusterNode.GrpcPort` per node — via the AdminUI node edit (Phase 1 surfaced the field) or a
|
||||
data update — or the telemetry dial silently finds no targets (loud-but-throttled warning, last-known
|
||||
DPS panels unaffected since the flip is per-node). Optional rig follow-up: make the docker-dev seed
|
||||
UPSERT the port columns so a persisted volume backfills on re-run (not blocking; this volume is now
|
||||
populated).
|
||||
|
||||
## Cleanup
|
||||
`docker-compose.override.yml` removed; rig recreated to the `Dps` default. `ClusterNode.GrpcPort`
|
||||
left populated at 4056 (correct — matches the fresh-install seed intent).
|
||||
|
||||
## Verdict
|
||||
**Phase 5 live gate PASSED.** Driver nodes host the telemetry gRPC server; central dials in only in
|
||||
Grpc mode; the full stream mesh forms from DB-sourced discovery; real telemetry reaches the AdminUI
|
||||
panels; kill-and-reconnect recovers every stream. The one issue surfaced was a rig/upgrade data gap
|
||||
(`GrpcPort` backfill), not a Phase-5 code defect — and it validated the graceful null-`GrpcPort`
|
||||
handling live.
|
||||
Reference in New Issue
Block a user