lmxopcua/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md

# Per-Cluster Mesh Phase 6 — Mesh Partition + Co-location Topology

> **For Claude:** REQUIRED SUB-SKILL: execute this plan task-by-task with
> superpowers-extended-cc:subagent-driven-development (the established mode for this workstream).

**Goal:** Split OtOpcUa's single fleet-wide Akka mesh into **one 2-node mesh per application
`Cluster`** (central pair + one pair per site), so every Primary-gated decision and every gated
resource shares one pair-local scope — the ScadaBridge shape. Same ActorSystem name (`otopcua`)
everywhere; separation is by **seed-node partitioning** only.

**Architecture:** Each pair's two nodes list only each other as seeds (self-first), so Akka forms
three independent meshes even on a shared network. Driver nodes carry a new `cluster-{ClusterId}`
role; the one pair-local singleton (`RedundancyStateActor`) re-scopes to it and is spawned on every
driver node, so each pair elects its own Primary and publishes `redundancy-state` on its own mesh's
DPS. Central (admin) reaches each site mesh over the explicit Phase 2/3/5 transports (ClusterClient
+ ConfigServe gRPC + telemetry gRPC), now partitioned **one ClusterClient per cluster**. The five
admin singletons stay on the central pair. Two admin-side consumers that assumed fleet-wide gossip
visibility (`ClusterNodeAddressReconciler`, `CentralCommunicationActor.RebuildClient`) are
re-scoped. Secrets replication becomes **pair-local** automatically (Akka DPS within each 2-node
mesh; no fleet-wide sync, no central-SQL dependency — consistent with Phase 4). A startup validator
refuses to boot a split-configured node (one carrying a `cluster-` role) that is still pointed at a
DPS transport mode, and the compiled transport defaults flip to the split-correct values.

**Tech Stack:** .NET 10, Akka.NET + Akka.Hosting (`WithSingleton`/`WithActors`), Akka.Cluster.Tools
(ClusterClient + ClusterSingleton + DistributedPubSub), Akka.Cluster.Hosting `ClusterSingletonOptions`,
xUnit + Shouldly, docker-dev compose rig.

**Decisions settled with the user 2026-07-24 (recon-driven; the first contradicts the program doc's
own guess):**
1. **Secrets after split = pair-local Akka.** The program doc guessed "SQL-hub mode like ScadaBridge,"
   but recon showed sites have **no SQL Server** (the point of Phase 4), so a SQL hub would force site
   nodes back onto central SQL. Instead: leave the `AddZbSecretsAkkaReplication` wiring untouched — it
   becomes per-pair for free once each mesh is two nodes. Fleet-wide secret sync is dropped by design;
   document it. No library re-wire.
2. **DPS branch disposition = flip + change defaults + validator, keep the code.** The rig flips to
   `ClusterClient`/`Grpc`/`FetchAndCache`; the compiled defaults change so a fresh deploy is
   split-correct; the DPS branch code stays as dark switches; a new startup validator makes a
   split-configured node refuse to boot on a DPS transport mode (neutralizes the silent-break footgun
   without a large irreversible deletion). `redundancy-state`/`fleet-status` stay on DPS either way.
3. **Rig scope = OtOpcUa-only per-cluster meshes.** Rewrite docker-dev into 3 independent 2-node
   meshes on a single shared network (faithful to production, where separation is by seeds over a WAN,
   not by network isolation). Physical co-location with ScadaBridge is documented as the production
   topology, not literally run in docker-dev.

**Authoritative context (read before implementing):**
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` §"Phase 6" (lines 266–284) + the tracking table.
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §4 (oldest-Up election, already fixed), §5
  (target architecture), §7 (sequencing).
- Recon findings (this session) — summarized inline per task below.

---

## What Phase 6 does NOT change (scope guards)

- **ActorSystem name stays `otopcua`.** Do not rename it or make it per-cluster.
- **`AkkaClusterOptions` gains no new property.** `SeedNodes` + `Roles` already carry per-node values
  from config; the split is in *how they're populated per pair* (rig/appsettings) + the `RoleParser`
  allow-list + singleton scoping. Do not add a `ClusterId` option — every `ClusterNode` row is already
  defined to be a driver node (Phase 1 decision), and roles are the declaration of record.
- **`SelectDriverPrimary` election logic is unchanged.** It filters on the `driver` role over
  `Cluster.State.Members`; after the split that set is pair-local, so it elects the oldest of two.
  No edit to the algorithm.
- **The five admin singletons stay admin-scoped** (`config-publish`, `admin-operations`,
  `audit-writer`, `fleet-status`, `cluster-node-address-reconciler`). Only `redundancy-state`
  re-homes.
- **`DriverMemberCount()`** in `DriverHostActor`/`ScriptedAlarmHostActor` needs **no code change** —
  it reads whole-mesh members filtered by `driver`, which becomes pair-local for free. (Note the
  semantics change in docs; do not "fix" it in code.)
- **Do NOT delete the DPS command/telemetry branches** (decision 2). Do NOT switch the network to
  per-site isolation or wire in ScadaBridge (decision 3). Do NOT re-wire secrets to SQL (decision 1).

---

### Task 0: `RoleParser` accepts `cluster-{ClusterId}`; consolidate the `driver` role literal

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (blocks 1, 2, 5, 7)

**Context:** `RoleParser.Allowed` is a fixed `HashSet {"admin","driver","dev"}`
(`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs:5-8`). A `cluster-{ClusterId}` role fails it, so
no node can carry one. Also, three independent `"driver"` string literals exist
(`RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`,
`Runtime/ServiceCollectionExtensions.DriverRole`) — consolidate to a single source now since every
Phase 6 task touches role code.

**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleParserTests.cs` (create if absent)

**Steps:**
1. Add a well-known-roles surface to `RoleParser` (or a new `ClusterRoles` static): constants
   `Admin = "admin"`, `Driver = "driver"`, `Dev = "dev"`, `ClusterRolePrefix = "cluster-"`, and a
   helper `bool IsClusterRole(string role) => role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length`.
2. Change `Parse` validation: a role is allowed if it is one of the three fixed roles **or**
   `IsClusterRole(role)` is true. Keep the existing trim/lowercase/empty-filter behavior. A role of
   exactly `"cluster-"` (empty ClusterId) is **rejected**.
3. Add `RoleParser.ClusterIdFromRole(string) => role.Substring(ClusterRolePrefix.Length)` returning
   the ClusterId for a cluster role (used by later tasks).
4. Update `RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`, and
   `Runtime/ServiceCollectionExtensions.DriverRole` to reference the single `RoleParser.Driver` (keep
   the public `const`/`static readonly` symbols as aliases if any test references them by their old
   name — grep first; do not break existing test references).
5. Tests: `admin`/`driver`/`dev`/`cluster-MAIN`/`cluster-SITE-A` accepted; `cluster-` rejected;
   `bogus` rejected; case/whitespace normalization preserved; `ClusterIdFromRole("cluster-SITE-A") == "SITE-A"`.

**Acceptance:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests` green; `cluster-{X}` roles
parse; the three literals resolve to one constant.

---

### Task 1: `ClusterRoleInfo` exposes the node's own cluster-scoped role

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (needs Task 0; blocks 2, 5)

**Context:** Singleton registration and the validator both need "what is *this* node's
`cluster-{ClusterId}` role." `ClusterRoleInfo` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`,
`IClusterRoleInfo`, registered in `AddOtOpcUaCluster`) is the existing role/leader authority — extend
it.

**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/IClusterRoleInfo.cs` (or wherever the interface lives)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ClusterRoleInfoTests.cs`

**Steps:**
1. Add `string? ClusterRole { get; }` (e.g. `"cluster-SITE-A"`) and `string? ClusterId { get; }` (e.g.
   `"SITE-A"`) to `IClusterRoleInfo`, derived from the configured `AkkaClusterOptions.Roles` via
   `RoleParser.IsClusterRole` (first match; log a Warning and pick the first if >1 cluster role is
   configured — that is a misconfiguration). Null when the node carries no cluster role (e.g. a
   pre-split/admin-only-legacy node).
2. Do NOT read `Cluster.State` for this — it is the node's **own configured** identity, available
   before the cluster forms (the singleton registration runs at host build time).
3. Tests: a node configured `["driver","cluster-SITE-A"]` reports `ClusterRole="cluster-SITE-A"`,
   `ClusterId="SITE-A"`; a node `["admin","driver"]` reports both null; two cluster roles → first wins
   + warning path exercised.

**Acceptance:** `ClusterRoleInfo` reports the node's cluster role/id from config; tests green.

---

### Task 2: Re-home `RedundancyStateActor` to a `cluster-{ClusterId}` singleton on driver nodes

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3, Task 4 (disjoint files)

**Context (the crux).** Today `RedundancyStateActor` is registered as an **admin** singleton
(`ControlPlane/ServiceCollectionExtensions.cs:133-136`, `singletonOptions.Role = "admin"`), electing a
**fleet-wide** Primary and publishing `redundancy-state` over DPS. After the split, central's mesh
sees only its own pair — so a site pair (driver-only, no admin node) would have **no one** to elect its
Primary. Fix: scope the singleton to the node's `cluster-{ClusterId}` role and spawn it on **every
driver node** (`hasDriver`), so each mesh (central pair = `cluster-MAIN`, each site pair =
`cluster-{SITE}`) has exactly one, electing its own pair-local Primary and publishing on its own mesh's
DPS. **De-risked:** `RedundancyStateActor` has zero injected deps (verified — it only uses
`Cluster.Get(Context.System)` + DPS), so it spawns cleanly on a driver-only node.

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs`
  (remove `redundancy-state` from `WithOtOpcUaControlPlaneSingletons`; add a new
  `WithOtOpcUaClusterRedundancySingleton(IClusterRoleInfo)` extension that registers it scoped to the
  cluster role)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (call the new extension on `hasDriver`, not
  `hasAdmin`; a fused `admin,driver` node calls it once via the driver branch)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/...` singleton-role test; and confirm the
  admin path no longer registers `redundancy-state`.

**Steps:**
1. In `ControlPlane/ServiceCollectionExtensions.cs`: delete the `WithSingleton` block for
   `RedundancyStateActorKey`/`redundancy-state` from the admin method (lines ~133-136). Keep the
   registry key type.
2. Add:
   ```csharp
   public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
       this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo)
   {
       // Cluster-scoped when the node carries a cluster-{ClusterId} role (Phase-6 split: one
       // redundancy singleton per pair, robust even if two meshes accidentally merged). Falls
       // back to the driver role for a legacy single-mesh node or a test harness with no cluster
       // role — where driver-scope is the pre-Phase-6 fleet-wide behavior, and on a genuinely
       // split 2-node mesh the driver role is ALREADY pair-local because the mesh IS the pair.
       // No throw: decision 2 requires legacy/harness nodes to keep booting unchanged.
       var role = roleInfo.ClusterRole ?? RoleParser.Driver;
       var opts = new ClusterSingletonOptions { Role = role };
       return builder.WithSingleton<RedundancyStateActorKey>(
           RedundancyStateActor.Topic, RedundancyStateActor.Props(), opts /* createProxyToo per existing pattern */);
   }
   ```
   Match the exact `WithSingleton` overload/signature the other five singletons use (check createProxy
   arg + singleton name conventions).
3. In `Program.cs`: within the `if (hasDriver)` Akka-configurator block (near line 357-374, alongside
   `WithOtOpcUaRuntimeActors`), call `ab.WithOtOpcUaClusterRedundancySingleton(clusterRoleInfo)`.
   Resolve `IClusterRoleInfo` from `sp`. Ensure it runs on the fused central node too (central has
   `hasDriver=true`). Do NOT also register it in the `hasAdmin` block.
4. **No fail-fast on a missing cluster role** — the fallback keeps legacy/harness nodes booting
   (decision 2). A *misconfigured* Phase-6 node (split transports, no cluster role) is a non-issue for
   the redundancy singleton: driver-scope is pair-local on a split mesh anyway.
5. Tests: assert the admin singleton set no longer contains `redundancy-state`; assert the new
   extension registers a singleton scoped to `Role == "cluster-SITE-A"` when the roleInfo reports that;
   assert that a roleInfo with null `ClusterRole` falls back to `Role == "driver"` (no throw).

**Acceptance:** each mesh hosts exactly one `RedundancyStateActor` scoped to its cluster role; a
driver-only site node elects its own Primary; admin path no longer owns it. Unit tests green.

**Live-verified later** (Task 9 gate): two Primaries fleet-wide, one per pair.

---

### Task 3: Re-scope `ClusterNodeAddressReconciler` to central-mesh-visible rows

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 4 (disjoint files)

**Context:** The admin-singleton reconciler compares `Cluster.State.Members` (driver role) against
**every** `ClusterNode` row fleet-wide (`ClusterNodeAddressReconciler.cs:123-131`,
`ClusterNodeAddressReconcilerActor.cs:76-96`). After the split, central sees only its own pair, so
every site row logs `EnabledRowNotInCluster` forever (its own doc comment,
`ClusterNodeAddressReconciler.cs:60-65`, flags this). Fix: only reconcile rows whose `ClusterId`
matches central's own cluster — central can no longer observe other clusters via gossip, so it must
not assert anything about their rows.

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconciler.cs` (filter
  the row set by own-cluster)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs`
  (pass the admin node's own `ClusterId` in; source it from `IClusterRoleInfo.ClusterId`, or from the
  set of `ClusterId`s present among its own observed driver members)
- Update the class doc comment (remove the "Phase 2 must revisit" note; state it is now own-cluster
  scoped).
- Test: `tests/.../ClusterNodeAddressReconcilerTests.cs` — a row in another cluster is **ignored** (no
  finding); an own-cluster enabled row with no member still yields `EnabledRowNotInCluster`; drift
  detection within own cluster still works.

**The pure `ClusterNodeAddressReconciler.Reconcile` function is UNCHANGED** — the fix is entirely in
the actor's two DB queries (`ClusterNodeAddressReconcilerActor.Reconcile`, lines 89-97), which filter
rows by the admin node's own `ClusterId` before projecting. `ClusterNode` has a `ClusterId` column.

**Steps:**
1. Inject the admin node's own `ClusterId` into the actor: add `IClusterRoleInfo` (or just a
   `string? ownClusterId`) to `Props`/ctor, sourced from `IClusterRoleInfo.ClusterId` at registration
   (`ControlPlane/ServiceCollectionExtensions.cs` where the reconciler singleton is registered).
2. In the actor's `Reconcile`, add `.Where(n => n.ClusterId == ownClusterId)` to BOTH the `rows` and
   `enabled` queries — but ONLY when `ownClusterId` is non-null.
3. **Null `ownClusterId` (legacy admin node with no cluster role) ⇒ reconcile ALL rows** — this is the
   correct single-mesh behavior (a legacy admin node genuinely sees every driver member via gossip).
   Do NOT skip. This preserves backward-compat, mirroring the Task 2 fallback philosophy.
4. `RowDialTargetDisagrees` / `RunningNodeHasNoRow` / `EnabledRowNotInCluster` semantics are unchanged;
   they now simply operate on the own-cluster row subset (post-split) or the full set (legacy).
5. Update the class doc comment in `ClusterNodeAddressReconciler.cs:60-65`: replace the "Phase 2 must
   revisit" note with the delivered own-cluster-scoped behavior.
6. Tests (`ClusterNodeAddressReconcilerActorTests` or the existing reconciler tests — pick the level
   that can inject `ownClusterId`): a SITE-A row is **ignored** when `ownClusterId=="MAIN"` (no
   `EnabledRowNotInCluster`); an own-cluster (MAIN) enabled row with no member still yields
   `EnabledRowNotInCluster`; own-cluster drift still caught; `ownClusterId==null` reconciles the full
   set (legacy). If the actor is hard to unit-test directly, a focused query-filter test + the
   unchanged pure-function tests suffice — say which you chose.

**Acceptance:** central reconciles only MAIN rows; site rows never produce false
`EnabledRowNotInCluster`; own-cluster drift still caught.

---

### Task 4: One `ClusterClient` per application `Cluster` in `CentralCommunicationActor`

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 3 (disjoint files)

**Context:** `RebuildClient` (`CentralCommunicationActor.cs:271-297`) creates **one fleet-wide**
`ClusterClient` dialing every `ClusterNode` row, and `HandleMeshCommand` (`:128-157`) does one
`ClusterClient.SendToAll(...)`. The `TODO(Phase 6)` (`:249-268`) states this is correct only on a
single mesh: a `ClusterClientReceptionist` serves its whole cluster, so `SendToAll` on one client
reaches everyone. After the split, `SendToAll` on one client reaches only the mesh whose receptionist
answered — so central needs **one client per cluster**, and dispatch fans `SendToAll` across all of
them.

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs`
  (extend: two separate site meshes + central, a command reaches BOTH via per-cluster clients; and a
  single-cluster client's `SendToAll` does NOT reach the other cluster)

**Steps:**
1. `LoadContactsFromDb` (`:177-221`): return contacts **grouped by `ClusterId`** — an
   `ImmutableDictionary<string, ImmutableHashSet<string>>` (ClusterId → receptionist contact paths).
   Keep the `Enabled && !MaintenanceMode` filter.
2. Replace the single `_client`/`_currentContacts` with a `Dictionary<string, (IActorRef Client,
   ImmutableHashSet<string> Contacts)>` keyed by ClusterId. `HandleContactsLoaded` diffs **per cluster**
   and rebuilds only the clusters whose contact set changed; stops+removes clients for clusters that
   vanished.
3. `RebuildClient` becomes `RebuildClient(string clusterId, ImmutableHashSet<string> contacts)` —
   builds one `ClusterClient` per cluster (name it by clusterId so the actor paths are distinct).
4. `HandleMeshCommand` fans out: for each cluster client, `client.Tell(new ClusterClient.SendToAll(
   MeshPaths.NodeCommunication, cmd.Message))`. Preserve the sender-relay idiom where it exists.
   **A command with no live clients (all clusters down) drops with a Warning**, matching today's
   "no reachable contact" behavior — do not buffer.
5. Update the `TODO(Phase 6)` doc block to describe the delivered per-cluster-client shape (remove the
   TODO marker).
6. **Frame-size note:** the deploy path stays payload-free (notify-and-fetch), so no new frame-size
   exposure — do not add payload to these commands.
7. Tests: extend `MeshClusterClientBoundaryTests` to two site meshes + central; assert a dispatch
   reaches a node in each site mesh, and that per-cluster `SendToAll` is correctly partitioned.

**Acceptance:** central dispatches deploy-notify/driver-control/alarm-commands to every cluster via
one client each; the boundary test proves cross-mesh partitioned delivery; acks still return via the
site nodes' own `ClusterClient.Send` to central (unchanged).

---

### Task 5: Split-topology startup validator (cluster-role ⇒ non-DPS transport)

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 6 (disjoint files)

**Context (decision 2):** the DPS branches stay as dark switches, but a node configured for the split
topology (it carries a `cluster-{ClusterId}` role) must not silently run a transport still in DPS
mode, which cannot cross meshes. Fail host start instead. Marker of "split topology" = the presence of
a `cluster-` role (they exist only post-Phase-6).

**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs` (an
  `IValidateOptions<...>` or a startup validator wired via `AddValidatedOptions`/`ValidateOnStart`,
  mirroring `AkkaClusterOptionsValidator`/`ConfigSourceOptionsValidator` structure)
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`AddOtOpcUaCluster`
  registers it)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitTopologyTransportValidatorTests.cs`

**Rule (only fires when the node carries a `cluster-` role):**
- `MeshTransport:Mode` must be `ClusterClient` (not `Dps`) — every node.
- If `hasDriver`: `Telemetry:Mode` must be `Grpc` (node hosts the telemetry server).
- If `hasAdmin`: `TelemetryDial:Mode` must be `Grpc` (central dials).
- `ConfigSource:Mode`: **do not** add a rule here — Phase 4's `ConfigSourceOptionsValidator` already
  requires `FetchAndCache` on driver-only nodes; a fused node legitimately stays `Direct`.
- A node with **no** `cluster-` role (legacy/pre-split) is exempt (rule does not fire), so existing
  single-mesh deployments and tests are unaffected.

**Steps:**
1. Read `Cluster:Roles` (same direct-read pattern the other validators use — not `IClusterRoleInfo`,
   to avoid a service-resolution ordering dependency at validation time), `MeshTransport:Mode`,
   `Telemetry:Mode`, `TelemetryDial:Mode`.
2. Implement the rule above; each violation adds a precise message naming the offending key + required
   value.
3. Register with `ValidateOnStart` so a misconfigured node refuses to boot.
4. Tests: cluster-role node on `MeshTransport:Mode=Dps` fails; on `ClusterClient` passes; driver
   cluster node on `Telemetry:Mode=Dps` fails; admin cluster node on `TelemetryDial:Mode=Dps` fails;
   a non-cluster-role node on all-Dps passes (exempt).

**Acceptance:** a split-configured node on any DPS transport refuses to start with a clear message;
legacy nodes unaffected.

---

### Task 6: Flip compiled transport-mode defaults to split-correct

**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 5 (disjoint files)

**Context (decision 2):** so a fresh deploy is split-correct without per-node overrides, change the
compiled defaults. `redundancy-state`/`fleet-status` stay DPS regardless.

**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs` (default `Mode` → `ClusterClient`)
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/TelemetryOptions.cs` (`TelemetryOptions.Mode` and
  `TelemetryDialOptions.Mode` defaults → `Grpc`)
- Leave `ConfigSourceOptions.Mode` default = `Direct` (fused central stays Direct; Phase 4 validator
  forces driver-only to FetchAndCache).
- Update tests that assert the old defaults (grep for `ModeDps`/`Mode == "Dps"`/`.Mode.ShouldBe` in
  the Cluster.Tests and any options tests; adjust to the new defaults). Update `appsettings.json`
  comments/keys if they document the default.

**Steps:**
1. Change the three default initializers.
2. Grep + update every test asserting the prior default (there are a handful — do not miss the
   Phase 2/5 option tests).
3. Confirm the docker-dev rig (Task 7) does not *rely* on the old default anywhere (it will set modes
   explicitly anyway).

**Acceptance:** `new MeshTransportOptions().Mode == ClusterClient`; both telemetry defaults `Grpc`;
all options tests green.

---

### Task 7: Rewrite the docker-dev rig into three 2-node meshes

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (needs Tasks 0–6 landed to actually run; the file edit is independent
but sequence it after code so the rig is testable)

**Context (decision 3):** one shared network, seed-partitioned into three meshes. Central pair =
`cluster-MAIN`; site-a pair = `cluster-SITE-A`; site-b pair = `cluster-SITE-B`. Each pair lists only
its own two nodes as seeds (self-first). Flip the transports on. Secrets stays enabled (now
pair-local). Central must remain network-reachable to site nodes (shared network — already true; do
**not** isolate networks).

**Files:**
- Modify: `docker-dev/docker-compose.yml`
- Modify: `docker-dev/seed/seed-clusters.sql` if any seeded address/port changes (AkkaPort stays 4053;
  GrpcPort stays 4056 — likely no seed change, but verify the ClusterId/host columns still match).

**Per-node changes:**
| Node | `Cluster__Roles` | `Cluster__SeedNodes__0` (self) | `__1` (partner) |
|---|---|---|---|
| central-1 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-1:4053` | `...central-2:4053` |
| central-2 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-2:4053` | `...central-1:4053` |
| site-a-1 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-1:4053` | `...site-a-2:4053` |
| site-a-2 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-2:4053` | `...site-a-1:4053` |
| site-b-1 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-1:4053` | `...site-b-2:4053` |
| site-b-2 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-2:4053` | `...site-b-1:4053` |

**Transport flips (all nodes unless noted):**
- `MeshTransport__Mode: ClusterClient` (remove the `${OTOPCUA_MESH_MODE:-Dps}` default or set the var
  default to ClusterClient).
- `Telemetry__Mode: Grpc` (all); `TelemetryDial__Mode: Grpc` (central pair only — the dialers).
- `ConfigSource__Mode: FetchAndCache` already on sites; central stays `Direct`.
- `MeshTransport__CentralContactPoints__0/1` on **site** nodes point at central-1/central-2 (unchanged
  — sites learn central from appsettings). Central nodes do not need CentralContactPoints for
  themselves; central discovers *site* contacts from `ClusterNode` rows (per-cluster clients, Task 4).
- **Secrets** (`x-secrets-env` anchor): keep `Secrets__Replication__Enabled=true` — it becomes
  pair-local automatically. Add a compose comment noting the new pair-local semantics.
- **Remove** the `central-1`-as-seed lines from every site node (the old cross-pair seed). Update the
  header comment block (lines 1-23) to describe three meshes, not one.
- **LocalDb sync:** keep site-a's replication demo; optionally add a distinct sync port for site-b to
  model per-site ports (nice-to-have, not required for the gate — if added, use a different host-safe
  port and its own peer address).

**Steps:**
1. Rewrite the six nodes' `Cluster__Roles__*` and `Cluster__SeedNodes__*` per the table.
2. Flip the transport-mode env vars.
3. Update the header comment + secrets comment.
4. `docker compose -f docker-dev/docker-compose.yml config` to validate YAML (no live bring-up in this
   task — that is the Task 9 gate).

**Acceptance:** `docker compose config` validates; the six nodes carry per-pair self-first seeds +
cluster roles + split-correct transport modes; header comment describes three meshes.

---

### Task 8: Docs sweep — status tables, caveat removal, pair-local secrets note

**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 7 (docs only; no build)

**Files:**
- `docs/Redundancy.md` — new §"Per-cluster meshes (Phase 6)": one Primary per pair (**two+ Primaries
  fleet-wide, by design**), redundancy-state singleton is now `cluster-{ClusterId}`-scoped and
  spawned per driver node; remove any single-mesh caveat. Add §"Secrets replication is pair-local".
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs:34-35` — remove the
  "Mesh-scope caveat / until the per-cluster mesh work lands" doc comment (it landed).
- AdminUI ClusterRedundancy page — remove the mesh-scope caveat text (grep the Razor for the caveat
  string).
- `docs/Configuration.md` — document the `cluster-{ClusterId}` role, per-pair `Cluster:SeedNodes`
  ordering, the new transport-mode defaults, and the split-topology validator.
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` — mark Phase 6 **DONE** in the section + the
  tracking table; note the three decisions.
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §7 — mark row 6 DONE.
- `CLAUDE.md` — a "Per-cluster mesh (Phase 6)" note under the Redundancy section: three meshes, cluster
  roles, one Primary per pair, pair-local secrets, per-cluster ClusterClient, reconciler own-cluster
  scoped, transport defaults flipped + split validator.

**Steps:**
1. Write the Redundancy.md sections; remove the two caveats (IManualFailoverService + Razor).
2. Update Configuration.md.
3. Update the two plan status tables + CLAUDE.md.
4. Grep for any remaining "single mesh" / "fleet-wide Primary" / "until the per-cluster mesh work
   lands" copy and reconcile.

**Acceptance:** no stale single-mesh caveats remain; status tables show Phase 6 done; secrets
pair-local + cluster-role documented.

---

### Task 9: Live gate — three meshes, per-pair redundancy, cross-mesh transports

**Classification:** high-risk
**Estimated implement time:** manual (controller-run against docker-dev)
**Parallelizable with:** none (final)

**Context:** the exit gate. Bring the rewritten rig up and prove every previously-live-gated behavior
per pair, plus the split itself. Gather TCP/log/CLI evidence; record in
`docs/plans/2026-07-24-mesh-phase6-live-gate.md`.

**Legs:**
1. **Three meshes form.** Each node's `Cluster.State.Members` shows **only its own pair** (2 members).
   Evidence: redundancy CLI / logs per node, or `/proc/net/tcp` showing 4053 associations only within
   each pair. central sees central-1/2 only; site-a sees site-a-1/2 only; site-b sees site-b-1/2 only.
2. **Two+ Primaries, one per pair.** Each pair elects its own oldest-Up driver Primary; ServiceLevel
   **250/240** within each pair (Primary/Secondary), independently. Client.CLI `redundancy` per
   endpoint (4840/4841 MAIN, 4842/4843 SITE-A, 4844/4845 SITE-B).
3. **Deploy reaches all clusters over per-cluster ClusterClient.** `POST :9200/api/deployments`
   (X-Api-Key) seals green with all six nodes acking; central's logs show one client per cluster;
   verify a SITE-A node applies the config (its address space serves).
4. **Telemetry per cluster over gRPC.** AdminUI `/alerts` + `/hosts` pills live; central dials each
   site mesh's telemetry servers (2 inbound per driver node on 4056; central holds outbound per cluster).
   Kill-and-reconnect a site node → its stream recovers (~5s).
5. **Reconciler quiet.** central's `ClusterNodeAddressReconciler` logs **no** `EnabledRowNotInCluster`
   for SITE-A/SITE-B rows (own-cluster scoped now); MAIN drift still caught (probe with a deliberate
   mismatch if practical).
6. **Secrets pair-local.** With `Secrets:Replication:Enabled=true`, a secret written on site-a-1
   replicates to site-a-2 (same mesh) and does **not** reach central or site-b (separate meshes) — or,
   if a live secret write is impractical, assert the DPS topic is mesh-scoped via the membership
   evidence from leg 1 + a doc note. No errors from the secrets actor about unreachable peers.
7. **Split validator.** Flip one site node to `MeshTransport__Mode=Dps` → it refuses to boot with the
   Task 5 message. Restore.
8. **Failover within a pair** (bridges toward Phase 7): kill a site pair's Primary → the Secondary
   becomes Primary (250) and the pair survives alone (auto-down 1-vs-1 — the deferred Phase 0a gate is
   now testable; run it if time permits and note the result for Phase 7).

**Record:** `docs/plans/2026-07-24-mesh-phase6-live-gate.md` with per-leg evidence + any findings.
**Cleanup:** restore the rig to a coherent committed state.

**Acceptance:** all legs pass (or any deviation is documented + justified as in prior phases). Phase 6
exit gate MET.

---

## Risk register (carry from design §8 + program)

- LocalDb is load-bearing for config (unchanged; #485 class).
- **Two+ Primaries fleet-wide is now correct, by design** — anything asserting "one Primary" is stale.
- Partial rollout hazard: a node whose mesh is split has pair-local `DriverMemberCount`, while a
  not-yet-split node counts fleet-wide — do not run the fleet half-split in production without the
  drill (Phase 7).
- Co-located VM loss takes out one node of **both** products (Phase 7 drill item; out of scope here).
- Frame size: deploy path stays payload-free — keep it that way.