From 57e1d0176668d3adbad3e859525ee27969af9756 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 22 Jul 2026 08:38:45 -0400 Subject: [PATCH] docs(mesh): rig seed + Phase 1 documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docker-dev seed: AkkaPort = 4053 on all six ClusterNode rows, so a freshly seeded rig matches a migrated one. GrpcPort left null. - config-db-schema.md: both columns, why AkkaPort is NOT NULL/4053 and GrpcPort is nullable, the unenforced duplication + its reconciler, and Enabled's new second meaning as the deploy path's expected-ack set. - Configuration.md: Cluster:Port / PublicHostname now flag that they are stored twice, with the "update the row too" instruction and why the drift is silent. - design doc §7: Phase 1 marked done, plus a "Phase 1 as shipped" note recording both deviations rather than leaving the sketch reading as what happened. - program plan: Phase 1 marked done; AdminUI node edit explicitly deferred. - CLAUDE.md: the deploy-path behaviour change and its three consequences. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- CLAUDE.md | 9 +++++- docker-dev/seed/seed-clusters.sql | 24 ++++++++-------- docs/Configuration.md | 6 ++-- .../2026-07-21-per-cluster-mesh-design.md | 21 +++++++++++++- .../2026-07-22-per-cluster-mesh-program.md | 8 +++++- docs/v2/config-db-schema.md | 28 +++++++++++++++++++ 6 files changed, 79 insertions(+), 17 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f8974e6d..ce0fd80b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -228,7 +228,14 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. -**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b done; 1–7 not started, Phase 1 planned in `…-phase1.md`). +**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2–7 not started). + +**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys: + +- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. `ClusterNode.Enabled = 0` is the maintenance hatch; the deadline log names the silent nodes. +- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out. +- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist. +- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. ## LocalDb pair-local store (Phases 1 + 2) diff --git a/docker-dev/seed/seed-clusters.sql b/docker-dev/seed/seed-clusters.sql index 37c687ec..a2edf468 100644 --- a/docker-dev/seed/seed-clusters.sql +++ b/docker-dev/seed/seed-clusters.sql @@ -64,13 +64,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ServerCluster WHERE ClusterId = 'SITE-B') IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-1:4053') INSERT INTO dbo.ClusterNode - (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) - VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed'); + (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) + VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 4053, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed'); IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053') INSERT INTO dbo.ClusterNode - (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) - VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed'); + (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) + VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 4053, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed'); ------------------------------------------------------------------------------ -- ClusterNode — site A @@ -78,13 +78,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053') IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-1:4053') INSERT INTO dbo.ClusterNode - (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) - VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed'); + (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) + VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed'); IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053') INSERT INTO dbo.ClusterNode - (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) - VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed'); + (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) + VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed'); ------------------------------------------------------------------------------ -- ClusterNode — site B @@ -92,13 +92,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053') IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-1:4053') INSERT INTO dbo.ClusterNode - (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) - VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed'); + (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) + VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed'); IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053') INSERT INTO dbo.ClusterNode - (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) - VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed'); + (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) + VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed'); ------------------------------------------------------------------------------ -- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15) diff --git a/docs/Configuration.md b/docs/Configuration.md index 06ec59e6..6d7d249c 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -101,13 +101,15 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only |---|---|---|---| | `SystemName` | string | `otopcua` | Akka actor-system name. | | `Hostname` | string | `0.0.0.0` | Bind hostname. | -| `Port` | int | `4053` | Cluster transport port. | -| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. | +| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. | +| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. | | `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). | | `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. | > The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above. +> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode). + ### `ConnectionStrings` → `ConfigDb` - **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally. diff --git a/docs/plans/2026-07-21-per-cluster-mesh-design.md b/docs/plans/2026-07-21-per-cluster-mesh-design.md index 2de3ebd2..1acff34d 100644 --- a/docs/plans/2026-07-21-per-cluster-mesh-design.md +++ b/docs/plans/2026-07-21-per-cluster-mesh-design.md @@ -287,7 +287,7 @@ Deliberately not a task plan — per-phase plans follow, one at a time. |---|---|---| | 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 | | 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** | -| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes | +| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | **Yes** — **DONE 2026-07-22** (see below) | | 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No | | 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No | | 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No | @@ -298,6 +298,25 @@ Deliberately not a task plan — per-phase plans follow, one at a time. Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each deserves its own live gate. +### Phase 1 as shipped (2026-07-22) + +Plan: `2026-07-21-per-cluster-mesh-phase1.md`. Two deviations from the sketch above, both decided +before implementing: + +- **No cluster-scope filtering of the expected-ack set.** The plan called for it, but there is no + cluster-scoped deployment to filter on: `Deployment` has no `ClusterId`, + `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and + `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The + expected set is every enabled `ClusterNode` row, which is what the membership rule produced too. +- **No per-node role column.** The membership rule filtered on the `driver` role and the DB has none. + Rather than add one — a second declaration of node roles, free to drift from `Cluster:Roles` — every + `ClusterNode` row is now *defined* to be a driver node. An admin-only node must not be given one. + +Also shipped beyond the sketch: `ClusterNodeAddressReconciler` (Task 4), an admin singleton that +catches `AkkaPort` drifting from the node's own `Cluster:Port`. **Phase 2 must revisit it** — once the +meshes split, an admin node cannot see site members and every site row would report +`EnabledRowNotInCluster` forever. + ## 8. Risks - **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index 6cc7306e..fa26c599 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -82,7 +82,13 @@ per phase:** (1) invoke writing-plans in this repo to produce references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit gate; (4) update this file's status column and the design doc's §7 table. -### Phase 1 — `ClusterNode` address columns + DB-sourced ack set +### Phase 1 — `ClusterNode` address columns + DB-sourced ack set — **DONE 2026-07-22** +Shipped per `2026-07-21-per-cluster-mesh-phase1.md`; see that plan's Task 6 gate record and the +design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no +such deployment exists; no per-node role column) and the one addition (`ClusterNodeAddressReconciler`). +**AdminUI node edit was deferred to Phase 2** — nothing reads the columns until then, and the +migration default plus the rig seed cover every node today. + **Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site` entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from diff --git a/docs/v2/config-db-schema.md b/docs/v2/config-db-schema.md index c53b9d73..e931a08c 100644 --- a/docs/v2/config-db-schema.md +++ b/docs/v2/config-db-schema.md @@ -148,6 +148,8 @@ CREATE TABLE dbo.ClusterNode ( Host nvarchar(255) NOT NULL, OpcUaPort int NOT NULL DEFAULT 4840, DashboardPort int NOT NULL DEFAULT 8081, + AkkaPort int NOT NULL DEFAULT 4053, + GrpcPort int NULL, ApplicationUri nvarchar(256) NOT NULL, ServiceLevelBase tinyint NOT NULL DEFAULT 200, DriverConfigOverridesJson nvarchar(max) NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1), @@ -167,6 +169,32 @@ CREATE UNIQUE INDEX UX_ClusterNode_Primary_Per_Cluster WHERE RedundancyRole = 'Primary'; ``` +#### `AkkaPort` / `GrpcPort` — central's dial targets (per-cluster mesh Phase 1) + +These are **the addresses central dials**, not the node's own binding configuration. The node binds +from `Cluster:Port` in its own appsettings; these columns duplicate that value for a reader that +cannot see the node's config. Today central shares a gossip ring with every node and does not need +them — [per-cluster mesh Phase 2](../plans/2026-07-21-per-cluster-mesh-design.md) splits the fleet +into one mesh per `Cluster`, after which `Host` + `AkkaPort` is how central builds its ClusterClient +contact points, and `GrpcPort` is the Phase 5 telemetry stream. + +`AkkaPort` is `NOT NULL DEFAULT 4053` because every node listens on a remoting port — `0` is never a +truthful value, and rows predating the column must migrate to something real. `GrpcPort` is nullable +with **no** default because nothing listens on it until Phase 5, and a non-null default would assert +a port that does not exist. + +**The duplication against `Cluster:Port` is not enforced by the schema.** A node that binds 4054 while +its row says 4053 is unreachable from central in Phase 2, and the symptom there is a silent absence of +acks rather than an error. `ClusterNodeAddressReconcilerActor` (an admin-role singleton) compares the +rows against observed cluster membership and logs an Error on mismatch. It reads membership rather +than having each driver node assert its own row, because Phase 4 removes the driver nodes' ConfigDb +connection entirely. + +`Enabled` also gained a second meaning in Phase 1: it is the **expected-ack set** for a deployment. +`ConfigPublishCoordinator` no longer derives that set from cluster membership, so an enabled row whose +node is down now fails the deployment at the apply deadline instead of letting it seal green without +that node. Set `Enabled = 0` for a node taken down for maintenance. + `DriverConfigOverridesJson` shape: ```jsonc