diff --git a/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md b/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md new file mode 100644 index 00000000..e6efe04f --- /dev/null +++ b/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md @@ -0,0 +1,513 @@ +# Per-Cluster Mesh Phase 6 — Mesh Partition + Co-location Topology + +> **For Claude:** REQUIRED SUB-SKILL: execute this plan task-by-task with +> superpowers-extended-cc:subagent-driven-development (the established mode for this workstream). + +**Goal:** Split OtOpcUa's single fleet-wide Akka mesh into **one 2-node mesh per application +`Cluster`** (central pair + one pair per site), so every Primary-gated decision and every gated +resource shares one pair-local scope — the ScadaBridge shape. Same ActorSystem name (`otopcua`) +everywhere; separation is by **seed-node partitioning** only. + +**Architecture:** Each pair's two nodes list only each other as seeds (self-first), so Akka forms +three independent meshes even on a shared network. Driver nodes carry a new `cluster-{ClusterId}` +role; the one pair-local singleton (`RedundancyStateActor`) re-scopes to it and is spawned on every +driver node, so each pair elects its own Primary and publishes `redundancy-state` on its own mesh's +DPS. Central (admin) reaches each site mesh over the explicit Phase 2/3/5 transports (ClusterClient ++ ConfigServe gRPC + telemetry gRPC), now partitioned **one ClusterClient per cluster**. The five +admin singletons stay on the central pair. Two admin-side consumers that assumed fleet-wide gossip +visibility (`ClusterNodeAddressReconciler`, `CentralCommunicationActor.RebuildClient`) are +re-scoped. Secrets replication becomes **pair-local** automatically (Akka DPS within each 2-node +mesh; no fleet-wide sync, no central-SQL dependency — consistent with Phase 4). A startup validator +refuses to boot a split-configured node (one carrying a `cluster-` role) that is still pointed at a +DPS transport mode, and the compiled transport defaults flip to the split-correct values. + +**Tech Stack:** .NET 10, Akka.NET + Akka.Hosting (`WithSingleton`/`WithActors`), Akka.Cluster.Tools +(ClusterClient + ClusterSingleton + DistributedPubSub), Akka.Cluster.Hosting `ClusterSingletonOptions`, +xUnit + Shouldly, docker-dev compose rig. + +**Decisions settled with the user 2026-07-24 (recon-driven; the first contradicts the program doc's +own guess):** +1. **Secrets after split = pair-local Akka.** The program doc guessed "SQL-hub mode like ScadaBridge," + but recon showed sites have **no SQL Server** (the point of Phase 4), so a SQL hub would force site + nodes back onto central SQL. Instead: leave the `AddZbSecretsAkkaReplication` wiring untouched — it + becomes per-pair for free once each mesh is two nodes. Fleet-wide secret sync is dropped by design; + document it. No library re-wire. +2. **DPS branch disposition = flip + change defaults + validator, keep the code.** The rig flips to + `ClusterClient`/`Grpc`/`FetchAndCache`; the compiled defaults change so a fresh deploy is + split-correct; the DPS branch code stays as dark switches; a new startup validator makes a + split-configured node refuse to boot on a DPS transport mode (neutralizes the silent-break footgun + without a large irreversible deletion). `redundancy-state`/`fleet-status` stay on DPS either way. +3. **Rig scope = OtOpcUa-only per-cluster meshes.** Rewrite docker-dev into 3 independent 2-node + meshes on a single shared network (faithful to production, where separation is by seeds over a WAN, + not by network isolation). Physical co-location with ScadaBridge is documented as the production + topology, not literally run in docker-dev. + +**Authoritative context (read before implementing):** +- `docs/plans/2026-07-22-per-cluster-mesh-program.md` §"Phase 6" (lines 266–284) + the tracking table. +- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §4 (oldest-Up election, already fixed), §5 + (target architecture), §7 (sequencing). +- Recon findings (this session) — summarized inline per task below. + +--- + +## What Phase 6 does NOT change (scope guards) + +- **ActorSystem name stays `otopcua`.** Do not rename it or make it per-cluster. +- **`AkkaClusterOptions` gains no new property.** `SeedNodes` + `Roles` already carry per-node values + from config; the split is in *how they're populated per pair* (rig/appsettings) + the `RoleParser` + allow-list + singleton scoping. Do not add a `ClusterId` option — every `ClusterNode` row is already + defined to be a driver node (Phase 1 decision), and roles are the declaration of record. +- **`SelectDriverPrimary` election logic is unchanged.** It filters on the `driver` role over + `Cluster.State.Members`; after the split that set is pair-local, so it elects the oldest of two. + No edit to the algorithm. +- **The five admin singletons stay admin-scoped** (`config-publish`, `admin-operations`, + `audit-writer`, `fleet-status`, `cluster-node-address-reconciler`). Only `redundancy-state` + re-homes. +- **`DriverMemberCount()`** in `DriverHostActor`/`ScriptedAlarmHostActor` needs **no code change** — + it reads whole-mesh members filtered by `driver`, which becomes pair-local for free. (Note the + semantics change in docs; do not "fix" it in code.) +- **Do NOT delete the DPS command/telemetry branches** (decision 2). Do NOT switch the network to + per-site isolation or wire in ScadaBridge (decision 3). Do NOT re-wire secrets to SQL (decision 1). + +--- + +### Task 0: `RoleParser` accepts `cluster-{ClusterId}`; consolidate the `driver` role literal + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** none (blocks 1, 2, 5, 7) + +**Context:** `RoleParser.Allowed` is a fixed `HashSet {"admin","driver","dev"}` +(`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs:5-8`). A `cluster-{ClusterId}` role fails it, so +no node can carry one. Also, three independent `"driver"` string literals exist +(`RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`, +`Runtime/ServiceCollectionExtensions.DriverRole`) — consolidate to a single source now since every +Phase 6 task touches role code. + +**Files:** +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs` +- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleParserTests.cs` (create if absent) + +**Steps:** +1. Add a well-known-roles surface to `RoleParser` (or a new `ClusterRoles` static): constants + `Admin = "admin"`, `Driver = "driver"`, `Dev = "dev"`, `ClusterRolePrefix = "cluster-"`, and a + helper `bool IsClusterRole(string role) => role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length`. +2. Change `Parse` validation: a role is allowed if it is one of the three fixed roles **or** + `IsClusterRole(role)` is true. Keep the existing trim/lowercase/empty-filter behavior. A role of + exactly `"cluster-"` (empty ClusterId) is **rejected**. +3. Add `RoleParser.ClusterIdFromRole(string) => role.Substring(ClusterRolePrefix.Length)` returning + the ClusterId for a cluster role (used by later tasks). +4. Update `RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`, and + `Runtime/ServiceCollectionExtensions.DriverRole` to reference the single `RoleParser.Driver` (keep + the public `const`/`static readonly` symbols as aliases if any test references them by their old + name — grep first; do not break existing test references). +5. Tests: `admin`/`driver`/`dev`/`cluster-MAIN`/`cluster-SITE-A` accepted; `cluster-` rejected; + `bogus` rejected; case/whitespace normalization preserved; `ClusterIdFromRole("cluster-SITE-A") == "SITE-A"`. + +**Acceptance:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests` green; `cluster-{X}` roles +parse; the three literals resolve to one constant. + +--- + +### Task 1: `ClusterRoleInfo` exposes the node's own cluster-scoped role + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** none (needs Task 0; blocks 2, 5) + +**Context:** Singleton registration and the validator both need "what is *this* node's +`cluster-{ClusterId}` role." `ClusterRoleInfo` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`, +`IClusterRoleInfo`, registered in `AddOtOpcUaCluster`) is the existing role/leader authority — extend +it. + +**Files:** +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs` +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/IClusterRoleInfo.cs` (or wherever the interface lives) +- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ClusterRoleInfoTests.cs` + +**Steps:** +1. Add `string? ClusterRole { get; }` (e.g. `"cluster-SITE-A"`) and `string? ClusterId { get; }` (e.g. + `"SITE-A"`) to `IClusterRoleInfo`, derived from the configured `AkkaClusterOptions.Roles` via + `RoleParser.IsClusterRole` (first match; log a Warning and pick the first if >1 cluster role is + configured — that is a misconfiguration). Null when the node carries no cluster role (e.g. a + pre-split/admin-only-legacy node). +2. Do NOT read `Cluster.State` for this — it is the node's **own configured** identity, available + before the cluster forms (the singleton registration runs at host build time). +3. Tests: a node configured `["driver","cluster-SITE-A"]` reports `ClusterRole="cluster-SITE-A"`, + `ClusterId="SITE-A"`; a node `["admin","driver"]` reports both null; two cluster roles → first wins + + warning path exercised. + +**Acceptance:** `ClusterRoleInfo` reports the node's cluster role/id from config; tests green. + +--- + +### Task 2: Re-home `RedundancyStateActor` to a `cluster-{ClusterId}` singleton on driver nodes + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 3, Task 4 (disjoint files) + +**Context (the crux).** Today `RedundancyStateActor` is registered as an **admin** singleton +(`ControlPlane/ServiceCollectionExtensions.cs:133-136`, `singletonOptions.Role = "admin"`), electing a +**fleet-wide** Primary and publishing `redundancy-state` over DPS. After the split, central's mesh +sees only its own pair — so a site pair (driver-only, no admin node) would have **no one** to elect its +Primary. Fix: scope the singleton to the node's `cluster-{ClusterId}` role and spawn it on **every +driver node** (`hasDriver`), so each mesh (central pair = `cluster-MAIN`, each site pair = +`cluster-{SITE}`) has exactly one, electing its own pair-local Primary and publishing on its own mesh's +DPS. **De-risked:** `RedundancyStateActor` has zero injected deps (verified — it only uses +`Cluster.Get(Context.System)` + DPS), so it spawns cleanly on a driver-only node. + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs` + (remove `redundancy-state` from `WithOtOpcUaControlPlaneSingletons`; add a new + `WithOtOpcUaClusterRedundancySingleton(IClusterRoleInfo)` extension that registers it scoped to the + cluster role) +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (call the new extension on `hasDriver`, not + `hasAdmin`; a fused `admin,driver` node calls it once via the driver branch) +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/...` singleton-role test; and confirm the + admin path no longer registers `redundancy-state`. + +**Steps:** +1. In `ControlPlane/ServiceCollectionExtensions.cs`: delete the `WithSingleton` block for + `RedundancyStateActorKey`/`redundancy-state` from the admin method (lines ~133-136). Keep the + registry key type. +2. Add: + ```csharp + public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton( + this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo) + { + var clusterRole = roleInfo.ClusterRole + ?? throw new InvalidOperationException( + "A driver node must carry a cluster-{ClusterId} role to host the redundancy singleton."); + var opts = new ClusterSingletonOptions { Role = clusterRole }; + return builder.WithSingleton( + RedundancyStateActor.Topic, RedundancyStateActor.Props(), opts /* createProxyToo per existing pattern */); + } + ``` + Match the exact `WithSingleton` overload/signature the other five singletons use (check createProxy + arg + singleton name conventions). +3. In `Program.cs`: within the `if (hasDriver)` Akka-configurator block (near line 357-374, alongside + `WithOtOpcUaRuntimeActors`), call `ab.WithOtOpcUaClusterRedundancySingleton(clusterRoleInfo)`. + Resolve `IClusterRoleInfo` from `sp`. Ensure it runs on the fused central node too (central has + `hasDriver=true`). Do NOT also register it in the `hasAdmin` block. +4. **Guard: a driver node with no cluster role must fail fast** (the new extension throws) — this is + caught more helpfully by the Task 5 validator, but the throw is the backstop. +5. Tests: assert the admin singleton set no longer contains `redundancy-state`; assert the new + extension registers a singleton scoped to `Role == clusterRole`; a node with null `ClusterRole` + throws. + +**Acceptance:** each mesh hosts exactly one `RedundancyStateActor` scoped to its cluster role; a +driver-only site node elects its own Primary; admin path no longer owns it. Unit tests green. + +**Live-verified later** (Task 9 gate): two Primaries fleet-wide, one per pair. + +--- + +### Task 3: Re-scope `ClusterNodeAddressReconciler` to central-mesh-visible rows + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 2, Task 4 (disjoint files) + +**Context:** The admin-singleton reconciler compares `Cluster.State.Members` (driver role) against +**every** `ClusterNode` row fleet-wide (`ClusterNodeAddressReconciler.cs:123-131`, +`ClusterNodeAddressReconcilerActor.cs:76-96`). After the split, central sees only its own pair, so +every site row logs `EnabledRowNotInCluster` forever (its own doc comment, +`ClusterNodeAddressReconciler.cs:60-65`, flags this). Fix: only reconcile rows whose `ClusterId` +matches central's own cluster — central can no longer observe other clusters via gossip, so it must +not assert anything about their rows. + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconciler.cs` (filter + the row set by own-cluster) +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs` + (pass the admin node's own `ClusterId` in; source it from `IClusterRoleInfo.ClusterId`, or from the + set of `ClusterId`s present among its own observed driver members) +- Update the class doc comment (remove the "Phase 2 must revisit" note; state it is now own-cluster + scoped). +- Test: `tests/.../ClusterNodeAddressReconcilerTests.cs` — a row in another cluster is **ignored** (no + finding); an own-cluster enabled row with no member still yields `EnabledRowNotInCluster`; drift + detection within own cluster still works. + +**Steps:** +1. Thread the reconciler's own `ClusterId` (from `IClusterRoleInfo.ClusterId`, which for a fused + central node is `"MAIN"`) into `Reconcile`. Filter `rows` to `row.ClusterId == ownClusterId` before + any comparison. (`ClusterNode` has a `ClusterId` column — confirmed.) +2. Keep `RowDialTargetDisagrees` / `RunningNodeHasNoRow` / `EnabledRowNotInCluster` semantics for + own-cluster rows only. +3. If `IClusterRoleInfo.ClusterId` is null on an admin node (shouldn't happen post-Phase-6, but guard), + log one Warning and skip the sweep rather than reconciling the whole fleet. +4. Update the class doc: replace the deferred-to-Phase-6 note with the delivered behavior. +5. Tests as above. + +**Acceptance:** central reconciles only MAIN rows; site rows never produce false +`EnabledRowNotInCluster`; own-cluster drift still caught. + +--- + +### Task 4: One `ClusterClient` per application `Cluster` in `CentralCommunicationActor` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 2, Task 3 (disjoint files) + +**Context:** `RebuildClient` (`CentralCommunicationActor.cs:271-297`) creates **one fleet-wide** +`ClusterClient` dialing every `ClusterNode` row, and `HandleMeshCommand` (`:128-157`) does one +`ClusterClient.SendToAll(...)`. The `TODO(Phase 6)` (`:249-268`) states this is correct only on a +single mesh: a `ClusterClientReceptionist` serves its whole cluster, so `SendToAll` on one client +reaches everyone. After the split, `SendToAll` on one client reaches only the mesh whose receptionist +answered — so central needs **one client per cluster**, and dispatch fans `SendToAll` across all of +them. + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs` +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs` + (extend: two separate site meshes + central, a command reaches BOTH via per-cluster clients; and a + single-cluster client's `SendToAll` does NOT reach the other cluster) + +**Steps:** +1. `LoadContactsFromDb` (`:177-221`): return contacts **grouped by `ClusterId`** — an + `ImmutableDictionary>` (ClusterId → receptionist contact paths). + Keep the `Enabled && !MaintenanceMode` filter. +2. Replace the single `_client`/`_currentContacts` with a `Dictionary Contacts)>` keyed by ClusterId. `HandleContactsLoaded` diffs **per cluster** + and rebuilds only the clusters whose contact set changed; stops+removes clients for clusters that + vanished. +3. `RebuildClient` becomes `RebuildClient(string clusterId, ImmutableHashSet contacts)` — + builds one `ClusterClient` per cluster (name it by clusterId so the actor paths are distinct). +4. `HandleMeshCommand` fans out: for each cluster client, `client.Tell(new ClusterClient.SendToAll( + MeshPaths.NodeCommunication, cmd.Message))`. Preserve the sender-relay idiom where it exists. + **A command with no live clients (all clusters down) drops with a Warning**, matching today's + "no reachable contact" behavior — do not buffer. +5. Update the `TODO(Phase 6)` doc block to describe the delivered per-cluster-client shape (remove the + TODO marker). +6. **Frame-size note:** the deploy path stays payload-free (notify-and-fetch), so no new frame-size + exposure — do not add payload to these commands. +7. Tests: extend `MeshClusterClientBoundaryTests` to two site meshes + central; assert a dispatch + reaches a node in each site mesh, and that per-cluster `SendToAll` is correctly partitioned. + +**Acceptance:** central dispatches deploy-notify/driver-control/alarm-commands to every cluster via +one client each; the boundary test proves cross-mesh partitioned delivery; acks still return via the +site nodes' own `ClusterClient.Send` to central (unchanged). + +--- + +### Task 5: Split-topology startup validator (cluster-role ⇒ non-DPS transport) + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 6 (disjoint files) + +**Context (decision 2):** the DPS branches stay as dark switches, but a node configured for the split +topology (it carries a `cluster-{ClusterId}` role) must not silently run a transport still in DPS +mode, which cannot cross meshes. Fail host start instead. Marker of "split topology" = the presence of +a `cluster-` role (they exist only post-Phase-6). + +**Files:** +- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs` (an + `IValidateOptions<...>` or a startup validator wired via `AddValidatedOptions`/`ValidateOnStart`, + mirroring `AkkaClusterOptionsValidator`/`ConfigSourceOptionsValidator` structure) +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`AddOtOpcUaCluster` + registers it) +- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitTopologyTransportValidatorTests.cs` + +**Rule (only fires when the node carries a `cluster-` role):** +- `MeshTransport:Mode` must be `ClusterClient` (not `Dps`) — every node. +- If `hasDriver`: `Telemetry:Mode` must be `Grpc` (node hosts the telemetry server). +- If `hasAdmin`: `TelemetryDial:Mode` must be `Grpc` (central dials). +- `ConfigSource:Mode`: **do not** add a rule here — Phase 4's `ConfigSourceOptionsValidator` already + requires `FetchAndCache` on driver-only nodes; a fused node legitimately stays `Direct`. +- A node with **no** `cluster-` role (legacy/pre-split) is exempt (rule does not fire), so existing + single-mesh deployments and tests are unaffected. + +**Steps:** +1. Read `Cluster:Roles` (same direct-read pattern the other validators use — not `IClusterRoleInfo`, + to avoid a service-resolution ordering dependency at validation time), `MeshTransport:Mode`, + `Telemetry:Mode`, `TelemetryDial:Mode`. +2. Implement the rule above; each violation adds a precise message naming the offending key + required + value. +3. Register with `ValidateOnStart` so a misconfigured node refuses to boot. +4. Tests: cluster-role node on `MeshTransport:Mode=Dps` fails; on `ClusterClient` passes; driver + cluster node on `Telemetry:Mode=Dps` fails; admin cluster node on `TelemetryDial:Mode=Dps` fails; + a non-cluster-role node on all-Dps passes (exempt). + +**Acceptance:** a split-configured node on any DPS transport refuses to start with a clear message; +legacy nodes unaffected. + +--- + +### Task 6: Flip compiled transport-mode defaults to split-correct + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** Task 5 (disjoint files) + +**Context (decision 2):** so a fresh deploy is split-correct without per-node overrides, change the +compiled defaults. `redundancy-state`/`fleet-status` stay DPS regardless. + +**Files:** +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs` (default `Mode` → `ClusterClient`) +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/TelemetryOptions.cs` (`TelemetryOptions.Mode` and + `TelemetryDialOptions.Mode` defaults → `Grpc`) +- Leave `ConfigSourceOptions.Mode` default = `Direct` (fused central stays Direct; Phase 4 validator + forces driver-only to FetchAndCache). +- Update tests that assert the old defaults (grep for `ModeDps`/`Mode == "Dps"`/`.Mode.ShouldBe` in + the Cluster.Tests and any options tests; adjust to the new defaults). Update `appsettings.json` + comments/keys if they document the default. + +**Steps:** +1. Change the three default initializers. +2. Grep + update every test asserting the prior default (there are a handful — do not miss the + Phase 2/5 option tests). +3. Confirm the docker-dev rig (Task 7) does not *rely* on the old default anywhere (it will set modes + explicitly anyway). + +**Acceptance:** `new MeshTransportOptions().Mode == ClusterClient`; both telemetry defaults `Grpc`; +all options tests green. + +--- + +### Task 7: Rewrite the docker-dev rig into three 2-node meshes + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none (needs Tasks 0–6 landed to actually run; the file edit is independent +but sequence it after code so the rig is testable) + +**Context (decision 3):** one shared network, seed-partitioned into three meshes. Central pair = +`cluster-MAIN`; site-a pair = `cluster-SITE-A`; site-b pair = `cluster-SITE-B`. Each pair lists only +its own two nodes as seeds (self-first). Flip the transports on. Secrets stays enabled (now +pair-local). Central must remain network-reachable to site nodes (shared network — already true; do +**not** isolate networks). + +**Files:** +- Modify: `docker-dev/docker-compose.yml` +- Modify: `docker-dev/seed/seed-clusters.sql` if any seeded address/port changes (AkkaPort stays 4053; + GrpcPort stays 4056 — likely no seed change, but verify the ClusterId/host columns still match). + +**Per-node changes:** +| Node | `Cluster__Roles` | `Cluster__SeedNodes__0` (self) | `__1` (partner) | +|---|---|---|---| +| central-1 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-1:4053` | `...central-2:4053` | +| central-2 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-2:4053` | `...central-1:4053` | +| site-a-1 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-1:4053` | `...site-a-2:4053` | +| site-a-2 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-2:4053` | `...site-a-1:4053` | +| site-b-1 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-1:4053` | `...site-b-2:4053` | +| site-b-2 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-2:4053` | `...site-b-1:4053` | + +**Transport flips (all nodes unless noted):** +- `MeshTransport__Mode: ClusterClient` (remove the `${OTOPCUA_MESH_MODE:-Dps}` default or set the var + default to ClusterClient). +- `Telemetry__Mode: Grpc` (all); `TelemetryDial__Mode: Grpc` (central pair only — the dialers). +- `ConfigSource__Mode: FetchAndCache` already on sites; central stays `Direct`. +- `MeshTransport__CentralContactPoints__0/1` on **site** nodes point at central-1/central-2 (unchanged + — sites learn central from appsettings). Central nodes do not need CentralContactPoints for + themselves; central discovers *site* contacts from `ClusterNode` rows (per-cluster clients, Task 4). +- **Secrets** (`x-secrets-env` anchor): keep `Secrets__Replication__Enabled=true` — it becomes + pair-local automatically. Add a compose comment noting the new pair-local semantics. +- **Remove** the `central-1`-as-seed lines from every site node (the old cross-pair seed). Update the + header comment block (lines 1-23) to describe three meshes, not one. +- **LocalDb sync:** keep site-a's replication demo; optionally add a distinct sync port for site-b to + model per-site ports (nice-to-have, not required for the gate — if added, use a different host-safe + port and its own peer address). + +**Steps:** +1. Rewrite the six nodes' `Cluster__Roles__*` and `Cluster__SeedNodes__*` per the table. +2. Flip the transport-mode env vars. +3. Update the header comment + secrets comment. +4. `docker compose -f docker-dev/docker-compose.yml config` to validate YAML (no live bring-up in this + task — that is the Task 9 gate). + +**Acceptance:** `docker compose config` validates; the six nodes carry per-pair self-first seeds + +cluster roles + split-correct transport modes; header comment describes three meshes. + +--- + +### Task 8: Docs sweep — status tables, caveat removal, pair-local secrets note + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 7 (docs only; no build) + +**Files:** +- `docs/Redundancy.md` — new §"Per-cluster meshes (Phase 6)": one Primary per pair (**two+ Primaries + fleet-wide, by design**), redundancy-state singleton is now `cluster-{ClusterId}`-scoped and + spawned per driver node; remove any single-mesh caveat. Add §"Secrets replication is pair-local". +- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs:34-35` — remove the + "Mesh-scope caveat / until the per-cluster mesh work lands" doc comment (it landed). +- AdminUI ClusterRedundancy page — remove the mesh-scope caveat text (grep the Razor for the caveat + string). +- `docs/Configuration.md` — document the `cluster-{ClusterId}` role, per-pair `Cluster:SeedNodes` + ordering, the new transport-mode defaults, and the split-topology validator. +- `docs/plans/2026-07-22-per-cluster-mesh-program.md` — mark Phase 6 **DONE** in the section + the + tracking table; note the three decisions. +- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §7 — mark row 6 DONE. +- `CLAUDE.md` — a "Per-cluster mesh (Phase 6)" note under the Redundancy section: three meshes, cluster + roles, one Primary per pair, pair-local secrets, per-cluster ClusterClient, reconciler own-cluster + scoped, transport defaults flipped + split validator. + +**Steps:** +1. Write the Redundancy.md sections; remove the two caveats (IManualFailoverService + Razor). +2. Update Configuration.md. +3. Update the two plan status tables + CLAUDE.md. +4. Grep for any remaining "single mesh" / "fleet-wide Primary" / "until the per-cluster mesh work + lands" copy and reconcile. + +**Acceptance:** no stale single-mesh caveats remain; status tables show Phase 6 done; secrets +pair-local + cluster-role documented. + +--- + +### Task 9: Live gate — three meshes, per-pair redundancy, cross-mesh transports + +**Classification:** high-risk +**Estimated implement time:** manual (controller-run against docker-dev) +**Parallelizable with:** none (final) + +**Context:** the exit gate. Bring the rewritten rig up and prove every previously-live-gated behavior +per pair, plus the split itself. Gather TCP/log/CLI evidence; record in +`docs/plans/2026-07-24-mesh-phase6-live-gate.md`. + +**Legs:** +1. **Three meshes form.** Each node's `Cluster.State.Members` shows **only its own pair** (2 members). + Evidence: redundancy CLI / logs per node, or `/proc/net/tcp` showing 4053 associations only within + each pair. central sees central-1/2 only; site-a sees site-a-1/2 only; site-b sees site-b-1/2 only. +2. **Two+ Primaries, one per pair.** Each pair elects its own oldest-Up driver Primary; ServiceLevel + **250/240** within each pair (Primary/Secondary), independently. Client.CLI `redundancy` per + endpoint (4840/4841 MAIN, 4842/4843 SITE-A, 4844/4845 SITE-B). +3. **Deploy reaches all clusters over per-cluster ClusterClient.** `POST :9200/api/deployments` + (X-Api-Key) seals green with all six nodes acking; central's logs show one client per cluster; + verify a SITE-A node applies the config (its address space serves). +4. **Telemetry per cluster over gRPC.** AdminUI `/alerts` + `/hosts` pills live; central dials each + site mesh's telemetry servers (2 inbound per driver node on 4056; central holds outbound per cluster). + Kill-and-reconnect a site node → its stream recovers (~5s). +5. **Reconciler quiet.** central's `ClusterNodeAddressReconciler` logs **no** `EnabledRowNotInCluster` + for SITE-A/SITE-B rows (own-cluster scoped now); MAIN drift still caught (probe with a deliberate + mismatch if practical). +6. **Secrets pair-local.** With `Secrets:Replication:Enabled=true`, a secret written on site-a-1 + replicates to site-a-2 (same mesh) and does **not** reach central or site-b (separate meshes) — or, + if a live secret write is impractical, assert the DPS topic is mesh-scoped via the membership + evidence from leg 1 + a doc note. No errors from the secrets actor about unreachable peers. +7. **Split validator.** Flip one site node to `MeshTransport__Mode=Dps` → it refuses to boot with the + Task 5 message. Restore. +8. **Failover within a pair** (bridges toward Phase 7): kill a site pair's Primary → the Secondary + becomes Primary (250) and the pair survives alone (auto-down 1-vs-1 — the deferred Phase 0a gate is + now testable; run it if time permits and note the result for Phase 7). + +**Record:** `docs/plans/2026-07-24-mesh-phase6-live-gate.md` with per-leg evidence + any findings. +**Cleanup:** restore the rig to a coherent committed state. + +**Acceptance:** all legs pass (or any deviation is documented + justified as in prior phases). Phase 6 +exit gate MET. + +--- + +## Risk register (carry from design §8 + program) + +- LocalDb is load-bearing for config (unchanged; #485 class). +- **Two+ Primaries fleet-wide is now correct, by design** — anything asserting "one Primary" is stale. +- Partial rollout hazard: a node whose mesh is split has pair-local `DriverMemberCount`, while a + not-yet-split node counts fleet-wide — do not run the fleet half-split in production without the + drill (Phase 7). +- Co-located VM loss takes out one node of **both** products (Phase 7 drill item; out of scope here). +- Frame size: deploy path stays payload-free — keep it that way. diff --git a/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.tasks.json b/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.tasks.json new file mode 100644 index 00000000..afb2b443 --- /dev/null +++ b/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.tasks.json @@ -0,0 +1,24 @@ +{ + "planPath": "docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md", + "program": "per-cluster-mesh", + "phase": 6, + "branch": "feat/mesh-phase6", + "decisions": { + "secrets": "pair-local Akka (no SQL-hub; sites have no SQL)", + "dpsBranches": "flip rig + change compiled defaults + split-topology validator; keep DPS code", + "rigScope": "OtOpcUa-only three 2-node meshes on one shared network" + }, + "tasks": [ + {"id": 0, "subject": "Task 0: RoleParser accepts cluster-{ClusterId}; consolidate driver literal", "classification": "small", "status": "pending"}, + {"id": 1, "subject": "Task 1: ClusterRoleInfo exposes node's own ClusterRole/ClusterId", "classification": "small", "status": "pending", "blockedBy": [0]}, + {"id": 2, "subject": "Task 2: re-home RedundancyStateActor to cluster-role singleton on drivers", "classification": "high-risk", "status": "pending", "blockedBy": [1], "parallelizableWith": [3, 4]}, + {"id": 3, "subject": "Task 3: re-scope ClusterNodeAddressReconciler to own-cluster rows", "classification": "standard", "status": "pending", "blockedBy": [1], "parallelizableWith": [2, 4]}, + {"id": 4, "subject": "Task 4: one ClusterClient per cluster in CentralCommunicationActor", "classification": "high-risk", "status": "pending", "blockedBy": [], "parallelizableWith": [2, 3]}, + {"id": 5, "subject": "Task 5: split-topology startup validator (cluster-role => non-DPS)", "classification": "standard", "status": "pending", "blockedBy": [0, 1], "parallelizableWith": [6]}, + {"id": 6, "subject": "Task 6: flip compiled transport-mode defaults (ClusterClient/Grpc)", "classification": "small", "status": "pending", "blockedBy": [], "parallelizableWith": [5]}, + {"id": 7, "subject": "Task 7: rewrite docker-dev rig into three 2-node meshes", "classification": "standard", "status": "pending", "blockedBy": [0, 2, 3, 4, 5, 6]}, + {"id": 8, "subject": "Task 8: docs sweep — status tables, caveat removal, pair-local secrets note", "classification": "small", "status": "pending", "blockedBy": [2, 3, 4, 5, 6], "parallelizableWith": [7]}, + {"id": 9, "subject": "Task 9: live gate — three meshes, per-pair redundancy, cross-mesh transports","classification": "high-risk", "status": "pending", "blockedBy": [7, 8]} + ], + "lastUpdated": "2026-07-24T00:00:00Z" +}