lmxopcua/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md

Per-Cluster Mesh Phase 6 — Mesh Partition + Co-location Topology

For Claude: REQUIRED SUB-SKILL: execute this plan task-by-task with superpowers-extended-cc:subagent-driven-development (the established mode for this workstream).

Goal: Split OtOpcUa's single fleet-wide Akka mesh into one 2-node mesh per application Cluster (central pair + one pair per site), so every Primary-gated decision and every gated resource shares one pair-local scope — the ScadaBridge shape. Same ActorSystem name (otopcua) everywhere; separation is by seed-node partitioning only.

Architecture: Each pair's two nodes list only each other as seeds (self-first), so Akka forms three independent meshes even on a shared network. Driver nodes carry a new cluster-{ClusterId} role; the one pair-local singleton (RedundancyStateActor) re-scopes to it and is spawned on every driver node, so each pair elects its own Primary and publishes redundancy-state on its own mesh's DPS. Central (admin) reaches each site mesh over the explicit Phase 2/3/5 transports (ClusterClient

• ConfigServe gRPC + telemetry gRPC), now partitioned one ClusterClient per cluster. The five admin singletons stay on the central pair. Two admin-side consumers that assumed fleet-wide gossip visibility (ClusterNodeAddressReconciler, CentralCommunicationActor.RebuildClient) are re-scoped. Secrets replication becomes pair-local automatically (Akka DPS within each 2-node mesh; no fleet-wide sync, no central-SQL dependency — consistent with Phase 4). A startup validator refuses to boot a split-configured node (one carrying a cluster- role) that is still pointed at a DPS transport mode, and the compiled transport defaults flip to the split-correct values.

Tech Stack: .NET 10, Akka.NET + Akka.Hosting (WithSingleton/WithActors), Akka.Cluster.Tools (ClusterClient + ClusterSingleton + DistributedPubSub), Akka.Cluster.Hosting ClusterSingletonOptions, xUnit + Shouldly, docker-dev compose rig.

Decisions settled with the user 2026-07-24 (recon-driven; the first contradicts the program doc's own guess):

1. Secrets after split = pair-local Akka. The program doc guessed "SQL-hub mode like ScadaBridge," but recon showed sites have no SQL Server (the point of Phase 4), so a SQL hub would force site nodes back onto central SQL. Instead: leave the AddZbSecretsAkkaReplication wiring untouched — it becomes per-pair for free once each mesh is two nodes. Fleet-wide secret sync is dropped by design; document it. No library re-wire.
2. DPS branch disposition = flip + change defaults + validator, keep the code. The rig flips to ClusterClient/Grpc/FetchAndCache; the compiled defaults change so a fresh deploy is split-correct; the DPS branch code stays as dark switches; a new startup validator makes a split-configured node refuse to boot on a DPS transport mode (neutralizes the silent-break footgun without a large irreversible deletion). redundancy-state/fleet-status stay on DPS either way.
3. Rig scope = OtOpcUa-only per-cluster meshes. Rewrite docker-dev into 3 independent 2-node meshes on a single shared network (faithful to production, where separation is by seeds over a WAN, not by network isolation). Physical co-location with ScadaBridge is documented as the production topology, not literally run in docker-dev.

Authoritative context (read before implementing):

• docs/plans/2026-07-22-per-cluster-mesh-program.md §"Phase 6" (lines 266–284) + the tracking table.
• docs/plans/2026-07-21-per-cluster-mesh-design.md §4 (oldest-Up election, already fixed), §5 (target architecture), §7 (sequencing).
• Recon findings (this session) — summarized inline per task below.

---

What Phase 6 does NOT change (scope guards)

• ActorSystem name stays otopcua. Do not rename it or make it per-cluster.
• AkkaClusterOptions gains no new property. SeedNodes + Roles already carry per-node values from config; the split is in how they're populated per pair (rig/appsettings) + the RoleParser allow-list + singleton scoping. Do not add a ClusterId option — every ClusterNode row is already defined to be a driver node (Phase 1 decision), and roles are the declaration of record.
• SelectDriverPrimary election logic is unchanged. It filters on the driver role over Cluster.State.Members; after the split that set is pair-local, so it elects the oldest of two. No edit to the algorithm.
• The five admin singletons stay admin-scoped (config-publish, admin-operations, audit-writer, fleet-status, cluster-node-address-reconciler). Only redundancy-state re-homes.
• DriverMemberCount() in DriverHostActor/ScriptedAlarmHostActor needs no code change — it reads whole-mesh members filtered by driver, which becomes pair-local for free. (Note the semantics change in docs; do not "fix" it in code.)
• Do NOT delete the DPS command/telemetry branches (decision 2). Do NOT switch the network to per-site isolation or wire in ScadaBridge (decision 3). Do NOT re-wire secrets to SQL (decision 1).

---

Task 0: RoleParser accepts cluster-{ClusterId}; consolidate the driver role literal

Classification: small Estimated implement time: ~4 min Parallelizable with: none (blocks 1, 2, 5, 7)

Context: RoleParser.Allowed is a fixed HashSet {"admin","driver","dev"} (src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs:5-8). A cluster-{ClusterId} role fails it, so no node can carry one. Also, three independent "driver" string literals exist (RedundancyStateActor.DriverRole, ClusterNodeAddressReconcilerActor.DriverRole, Runtime/ServiceCollectionExtensions.DriverRole) — consolidate to a single source now since every Phase 6 task touches role code.

Files:

• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleParserTests.cs (create if absent)

Steps:

1. Add a well-known-roles surface to RoleParser (or a new ClusterRoles static): constants Admin = "admin", Driver = "driver", Dev = "dev", ClusterRolePrefix = "cluster-", and a helper bool IsClusterRole(string role) => role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length.
2. Change Parse validation: a role is allowed if it is one of the three fixed roles or IsClusterRole(role) is true. Keep the existing trim/lowercase/empty-filter behavior. A role of exactly "cluster-" (empty ClusterId) is rejected.
3. Add RoleParser.ClusterIdFromRole(string) => role.Substring(ClusterRolePrefix.Length) returning the ClusterId for a cluster role (used by later tasks).
4. Update RedundancyStateActor.DriverRole, ClusterNodeAddressReconcilerActor.DriverRole, and Runtime/ServiceCollectionExtensions.DriverRole to reference the single RoleParser.Driver (keep the public const/static readonly symbols as aliases if any test references them by their old name — grep first; do not break existing test references).
5. Tests: admin/driver/dev/cluster-MAIN/cluster-SITE-A accepted; cluster- rejected; bogus rejected; case/whitespace normalization preserved; ClusterIdFromRole("cluster-SITE-A") == "SITE-A".

Acceptance: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests green; cluster-{X} roles parse; the three literals resolve to one constant.

---

Task 1: ClusterRoleInfo exposes the node's own cluster-scoped role

Classification: small Estimated implement time: ~4 min Parallelizable with: none (needs Task 0; blocks 2, 5)

Context: Singleton registration and the validator both need "what is this node's cluster-{ClusterId} role." ClusterRoleInfo (src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs, IClusterRoleInfo, registered in AddOtOpcUaCluster) is the existing role/leader authority — extend it.

Files:

• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/IClusterRoleInfo.cs (or wherever the interface lives)
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ClusterRoleInfoTests.cs

Steps:

1. Add string? ClusterRole { get; } (e.g. "cluster-SITE-A") and string? ClusterId { get; } (e.g. "SITE-A") to IClusterRoleInfo, derived from the configured AkkaClusterOptions.Roles via RoleParser.IsClusterRole (first match; log a Warning and pick the first if >1 cluster role is configured — that is a misconfiguration). Null when the node carries no cluster role (e.g. a pre-split/admin-only-legacy node).
2. Do NOT read Cluster.State for this — it is the node's own configured identity, available before the cluster forms (the singleton registration runs at host build time).
3. Tests: a node configured ["driver","cluster-SITE-A"] reports ClusterRole="cluster-SITE-A", ClusterId="SITE-A"; a node ["admin","driver"] reports both null; two cluster roles → first wins
   • warning path exercised.

Acceptance: ClusterRoleInfo reports the node's cluster role/id from config; tests green.

---

Task 2: Re-home RedundancyStateActor to a cluster-{ClusterId} singleton on driver nodes

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 3, Task 4 (disjoint files)

Context (the crux). Today RedundancyStateActor is registered as an admin singleton (ControlPlane/ServiceCollectionExtensions.cs:133-136, singletonOptions.Role = "admin"), electing a fleet-wide Primary and publishing redundancy-state over DPS. After the split, central's mesh sees only its own pair — so a site pair (driver-only, no admin node) would have no one to elect its Primary. Fix: scope the singleton to the node's cluster-{ClusterId} role and spawn it on every driver node (hasDriver), so each mesh (central pair = cluster-MAIN, each site pair = cluster-{SITE}) has exactly one, electing its own pair-local Primary and publishing on its own mesh's DPS. De-risked: RedundancyStateActor has zero injected deps (verified — it only uses Cluster.Get(Context.System) + DPS), so it spawns cleanly on a driver-only node.

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs (remove redundancy-state from WithOtOpcUaControlPlaneSingletons; add a new WithOtOpcUaClusterRedundancySingleton(IClusterRoleInfo) extension that registers it scoped to the cluster role)
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs (call the new extension on hasDriver, not hasAdmin; a fused admin,driver node calls it once via the driver branch)
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/... singleton-role test; and confirm the admin path no longer registers redundancy-state.

Steps:

1. In ControlPlane/ServiceCollectionExtensions.cs: delete the WithSingleton block for RedundancyStateActorKey/redundancy-state from the admin method (lines ~133-136). Keep the registry key type.
2. Add:

   public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
       this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo)
   {
       // Cluster-scoped when the node carries a cluster-{ClusterId} role (Phase-6 split: one
       // redundancy singleton per pair, robust even if two meshes accidentally merged). Falls
       // back to the driver role for a legacy single-mesh node or a test harness with no cluster
       // role — where driver-scope is the pre-Phase-6 fleet-wide behavior, and on a genuinely
       // split 2-node mesh the driver role is ALREADY pair-local because the mesh IS the pair.
       // No throw: decision 2 requires legacy/harness nodes to keep booting unchanged.
       var role = roleInfo.ClusterRole ?? RoleParser.Driver;
       var opts = new ClusterSingletonOptions { Role = role };
       return builder.WithSingleton<RedundancyStateActorKey>(
           RedundancyStateActor.Topic, RedundancyStateActor.Props(), opts /* createProxyToo per existing pattern */);
   }
   

   Match the exact WithSingleton overload/signature the other five singletons use (check createProxy arg + singleton name conventions).
3. In Program.cs: within the if (hasDriver) Akka-configurator block (near line 357-374, alongside WithOtOpcUaRuntimeActors), call ab.WithOtOpcUaClusterRedundancySingleton(clusterRoleInfo). Resolve IClusterRoleInfo from sp. Ensure it runs on the fused central node too (central has hasDriver=true). Do NOT also register it in the hasAdmin block.
4. No fail-fast on a missing cluster role — the fallback keeps legacy/harness nodes booting (decision 2). A misconfigured Phase-6 node (split transports, no cluster role) is a non-issue for the redundancy singleton: driver-scope is pair-local on a split mesh anyway.
5. Tests: assert the admin singleton set no longer contains redundancy-state; assert the new extension registers a singleton scoped to Role == "cluster-SITE-A" when the roleInfo reports that; assert that a roleInfo with null ClusterRole falls back to Role == "driver" (no throw).

Acceptance: each mesh hosts exactly one RedundancyStateActor scoped to its cluster role; a driver-only site node elects its own Primary; admin path no longer owns it. Unit tests green.

Live-verified later (Task 9 gate): two Primaries fleet-wide, one per pair.

---

Task 3: Re-scope ClusterNodeAddressReconciler to central-mesh-visible rows

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 2, Task 4 (disjoint files)

Context: The admin-singleton reconciler compares Cluster.State.Members (driver role) against every ClusterNode row fleet-wide (ClusterNodeAddressReconciler.cs:123-131, ClusterNodeAddressReconcilerActor.cs:76-96). After the split, central sees only its own pair, so every site row logs EnabledRowNotInCluster forever (its own doc comment, ClusterNodeAddressReconciler.cs:60-65, flags this). Fix: only reconcile rows whose ClusterId matches central's own cluster — central can no longer observe other clusters via gossip, so it must not assert anything about their rows.

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconciler.cs (filter the row set by own-cluster)
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs (pass the admin node's own ClusterId in; source it from IClusterRoleInfo.ClusterId, or from the set of ClusterIds present among its own observed driver members)
• Update the class doc comment (remove the "Phase 2 must revisit" note; state it is now own-cluster scoped).
• Test: tests/.../ClusterNodeAddressReconcilerTests.cs — a row in another cluster is ignored (no finding); an own-cluster enabled row with no member still yields EnabledRowNotInCluster; drift detection within own cluster still works.

The pure ClusterNodeAddressReconciler.Reconcile function is UNCHANGED — the fix is entirely in the actor's two DB queries (ClusterNodeAddressReconcilerActor.Reconcile, lines 89-97), which filter rows by the admin node's own ClusterId before projecting. ClusterNode has a ClusterId column.

Steps:

1. Inject the admin node's own ClusterId into the actor: add IClusterRoleInfo (or just a string? ownClusterId) to Props/ctor, sourced from IClusterRoleInfo.ClusterId at registration (ControlPlane/ServiceCollectionExtensions.cs where the reconciler singleton is registered).
2. In the actor's Reconcile, add .Where(n => n.ClusterId == ownClusterId) to BOTH the rows and enabled queries — but ONLY when ownClusterId is non-null.
3. Null ownClusterId (legacy admin node with no cluster role) ⇒ reconcile ALL rows — this is the correct single-mesh behavior (a legacy admin node genuinely sees every driver member via gossip). Do NOT skip. This preserves backward-compat, mirroring the Task 2 fallback philosophy.
4. RowDialTargetDisagrees / RunningNodeHasNoRow / EnabledRowNotInCluster semantics are unchanged; they now simply operate on the own-cluster row subset (post-split) or the full set (legacy).
5. Update the class doc comment in ClusterNodeAddressReconciler.cs:60-65: replace the "Phase 2 must revisit" note with the delivered own-cluster-scoped behavior.
6. Tests (ClusterNodeAddressReconcilerActorTests or the existing reconciler tests — pick the level that can inject ownClusterId): a SITE-A row is ignored when ownClusterId=="MAIN" (no EnabledRowNotInCluster); an own-cluster (MAIN) enabled row with no member still yields EnabledRowNotInCluster; own-cluster drift still caught; ownClusterId==null reconciles the full set (legacy). If the actor is hard to unit-test directly, a focused query-filter test + the unchanged pure-function tests suffice — say which you chose.

Acceptance: central reconciles only MAIN rows; site rows never produce false EnabledRowNotInCluster; own-cluster drift still caught.

---

Task 4: One ClusterClient per application Cluster in CentralCommunicationActor

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 2, Task 3 (disjoint files)

Context: RebuildClient (CentralCommunicationActor.cs:271-297) creates one fleet-wide ClusterClient dialing every ClusterNode row, and HandleMeshCommand (:128-157) does one ClusterClient.SendToAll(...). The TODO(Phase 6) (:249-268) states this is correct only on a single mesh: a ClusterClientReceptionist serves its whole cluster, so SendToAll on one client reaches everyone. After the split, SendToAll on one client reaches only the mesh whose receptionist answered — so central needs one client per cluster, and dispatch fans SendToAll across all of them.

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs (extend: two separate site meshes + central, a command reaches BOTH via per-cluster clients; and a single-cluster client's SendToAll does NOT reach the other cluster)

Steps:

1. LoadContactsFromDb (:177-221): return contacts grouped by ClusterId — an ImmutableDictionary<string, ImmutableHashSet<string>> (ClusterId → receptionist contact paths). Keep the Enabled && !MaintenanceMode filter.
2. Replace the single _client/_currentContacts with a Dictionary<string, (IActorRef Client, ImmutableHashSet<string> Contacts)> keyed by ClusterId. HandleContactsLoaded diffs per cluster and rebuilds only the clusters whose contact set changed; stops+removes clients for clusters that vanished.
3. RebuildClient becomes RebuildClient(string clusterId, ImmutableHashSet<string> contacts) — builds one ClusterClient per cluster (name it by clusterId so the actor paths are distinct).
4. HandleMeshCommand fans out: for each cluster client, client.Tell(new ClusterClient.SendToAll( MeshPaths.NodeCommunication, cmd.Message)). Preserve the sender-relay idiom where it exists. A command with no live clients (all clusters down) drops with a Warning, matching today's "no reachable contact" behavior — do not buffer.
5. Update the TODO(Phase 6) doc block to describe the delivered per-cluster-client shape (remove the TODO marker).
6. Frame-size note: the deploy path stays payload-free (notify-and-fetch), so no new frame-size exposure — do not add payload to these commands.
7. Tests: extend MeshClusterClientBoundaryTests to two site meshes + central; assert a dispatch reaches a node in each site mesh, and that per-cluster SendToAll is correctly partitioned.

Acceptance: central dispatches deploy-notify/driver-control/alarm-commands to every cluster via one client each; the boundary test proves cross-mesh partitioned delivery; acks still return via the site nodes' own ClusterClient.Send to central (unchanged).

---

Task 5: Split-topology startup validator (cluster-role ⇒ non-DPS transport)

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 6 (disjoint files)

Context (decision 2): the DPS branches stay as dark switches, but a node configured for the split topology (it carries a cluster-{ClusterId} role) must not silently run a transport still in DPS mode, which cannot cross meshes. Fail host start instead. Marker of "split topology" = the presence of a cluster- role (they exist only post-Phase-6).

Files:

• Create: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs (an IValidateOptions<...> or a startup validator wired via AddValidatedOptions/ValidateOnStart, mirroring AkkaClusterOptionsValidator/ConfigSourceOptionsValidator structure)
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs (AddOtOpcUaCluster registers it)
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitTopologyTransportValidatorTests.cs

Rule (only fires when the node carries a cluster- role):

• MeshTransport:Mode must be ClusterClient (not Dps) — every node.
• If hasDriver: Telemetry:Mode must be Grpc (node hosts the telemetry server).
• If hasAdmin: TelemetryDial:Mode must be Grpc (central dials).
• ConfigSource:Mode: do not add a rule here — Phase 4's ConfigSourceOptionsValidator already requires FetchAndCache on driver-only nodes; a fused node legitimately stays Direct.
• A node with no cluster- role (legacy/pre-split) is exempt (rule does not fire), so existing single-mesh deployments and tests are unaffected.

Steps:

1. Read Cluster:Roles (same direct-read pattern the other validators use — not IClusterRoleInfo, to avoid a service-resolution ordering dependency at validation time), MeshTransport:Mode, Telemetry:Mode, TelemetryDial:Mode.
2. Implement the rule above; each violation adds a precise message naming the offending key + required value.
3. Register with ValidateOnStart so a misconfigured node refuses to boot.
4. Tests: cluster-role node on MeshTransport:Mode=Dps fails; on ClusterClient passes; driver cluster node on Telemetry:Mode=Dps fails; admin cluster node on TelemetryDial:Mode=Dps fails; a non-cluster-role node on all-Dps passes (exempt).

Acceptance: a split-configured node on any DPS transport refuses to start with a clear message; legacy nodes unaffected.

---

Task 6: Compiled transport-mode defaults — DECIDED: do NOT flip (data-backed deviation)

Classification: small → resolved as a no-code decision 2026-07-24 Parallelizable with: Task 5

Original intent (decision 2): change the compiled MeshTransport:Mode/Telemetry:Mode/ TelemetryDial:Mode defaults to the split-correct values so a fresh deploy is split-correct without per-node overrides.

What the blast-radius assessment found (read-only, this session): flipping the compiled defaults is HIGH-RISK and low-value. Every one of the three transport-option validators fail-closes when its mode is the non-Dps value but the mode's prerequisites are unset — and nothing in the repo sets those sections:

• MeshTransport:Mode=ClusterClient requires CentralContactPoints — no appsettings*.json and no test harness sets it → ValidateOnStart throws at boot.
• Telemetry:Mode=Grpc requires GrpcListenPort+ApiKey on a driver node — same gap.
• TelemetryDial:Mode=Grpc requires ApiKey with no role gate at all — breaks even roleless fixtures and docker-dev's 4 site nodes.
• Concretely this hard-fails all 14 Host.IntegrationTests (via TwoNodeClusterHarness, which sets none of these), all 5 shipped appsettings profiles, and several unit tests asserting the Dps default.

Why not flipping is correct, not a compromise: the Task 5 SplitTopologyTransportValidator already forces ClusterClient/Grpc/Grpc onto exactly the nodes carrying a post-Phase-6 cluster-{ClusterId} role, fail-closed, with a precise message — which is "a fresh Phase-6 deploy is split-correct," achieved more robustly than a default flip (a split node cannot even boot on the wrong transport). Flipping the compiled defaults would merely duplicate that enforcement for split nodes while indiscriminately breaking every legacy/test/admin node that legitimately still runs Dps. The docker-dev rig (Task 7) sets the modes explicitly on its cluster-role nodes, so it never relies on the default anyway.

Decision: keep MeshTransportOptions.Mode/TelemetryOptions.Mode/TelemetryDialOptions.Mode defaults at Dps. No code change. Enforcement of split-correctness lives in Task 5's validator + the explicit Task 7 rig config. Recorded in docs/Configuration.md (Task 8) so the next reader knows the defaults are intentionally Dps and the validator is the guardrail.

Acceptance: MET by decision — the split validator (Task 5) is the guardrail; no default flip.

---

Task 7: Rewrite the docker-dev rig into three 2-node meshes

Classification: standard Estimated implement time: ~5 min Parallelizable with: none (needs Tasks 0–6 landed to actually run; the file edit is independent but sequence it after code so the rig is testable)

Context (decision 3): one shared network, seed-partitioned into three meshes. Central pair = cluster-MAIN; site-a pair = cluster-SITE-A; site-b pair = cluster-SITE-B. Each pair lists only its own two nodes as seeds (self-first). Flip the transports on. Secrets stays enabled (now pair-local). Central must remain network-reachable to site nodes (shared network — already true; do not isolate networks).

Files:

• Modify: docker-dev/docker-compose.yml
• Modify: docker-dev/seed/seed-clusters.sql if any seeded address/port changes (AkkaPort stays 4053; GrpcPort stays 4056 — likely no seed change, but verify the ClusterId/host columns still match).

Per-node changes:

Node	Cluster__Roles	Cluster__SeedNodes__0 (self)	__1 (partner)
central-1	admin,driver,cluster-MAIN	akka.tcp://otopcua@central-1:4053	...central-2:4053
central-2	admin,driver,cluster-MAIN	akka.tcp://otopcua@central-2:4053	...central-1:4053
site-a-1	driver,cluster-SITE-A	akka.tcp://otopcua@site-a-1:4053	...site-a-2:4053
site-a-2	driver,cluster-SITE-A	akka.tcp://otopcua@site-a-2:4053	...site-a-1:4053
site-b-1	driver,cluster-SITE-B	akka.tcp://otopcua@site-b-1:4053	...site-b-2:4053
site-b-2	driver,cluster-SITE-B	akka.tcp://otopcua@site-b-2:4053	...site-b-1:4053

Transport flips (all nodes unless noted):

• MeshTransport__Mode: ClusterClient (remove the ${OTOPCUA_MESH_MODE:-Dps} default or set the var default to ClusterClient).
• Telemetry__Mode: Grpc (all); TelemetryDial__Mode: Grpc (central pair only — the dialers).
• ConfigSource__Mode: FetchAndCache already on sites; central stays Direct.
• MeshTransport__CentralContactPoints__0/1 on site nodes point at central-1/central-2 (unchanged — sites learn central from appsettings). Central nodes do not need CentralContactPoints for themselves; central discovers site contacts from ClusterNode rows (per-cluster clients, Task 4).
• Secrets (x-secrets-env anchor): keep Secrets__Replication__Enabled=true — it becomes pair-local automatically. Add a compose comment noting the new pair-local semantics.
• Remove the central-1-as-seed lines from every site node (the old cross-pair seed). Update the header comment block (lines 1-23) to describe three meshes, not one.
• LocalDb sync: keep site-a's replication demo; optionally add a distinct sync port for site-b to model per-site ports (nice-to-have, not required for the gate — if added, use a different host-safe port and its own peer address).

Steps:

1. Rewrite the six nodes' Cluster__Roles__* and Cluster__SeedNodes__* per the table.
2. Flip the transport-mode env vars.
3. Update the header comment + secrets comment.
4. docker compose -f docker-dev/docker-compose.yml config to validate YAML (no live bring-up in this task — that is the Task 9 gate).

Acceptance: docker compose config validates; the six nodes carry per-pair self-first seeds + cluster roles + split-correct transport modes; header comment describes three meshes.

---

Task 8: Docs sweep — status tables, caveat removal, pair-local secrets note

Classification: small Estimated implement time: ~5 min Parallelizable with: Task 7 (docs only; no build)

Files:

• docs/Redundancy.md — new §"Per-cluster meshes (Phase 6)": one Primary per pair (two+ Primaries fleet-wide, by design), redundancy-state singleton is now cluster-{ClusterId}-scoped and spawned per driver node; remove any single-mesh caveat. Add §"Secrets replication is pair-local".
• src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs:34-35 — remove the "Mesh-scope caveat / until the per-cluster mesh work lands" doc comment (it landed).
• AdminUI ClusterRedundancy page — remove the mesh-scope caveat text (grep the Razor for the caveat string).
• docs/Configuration.md — document the cluster-{ClusterId} role, per-pair Cluster:SeedNodes ordering, the new transport-mode defaults, and the split-topology validator.
• docs/plans/2026-07-22-per-cluster-mesh-program.md — mark Phase 6 DONE in the section + the tracking table; note the three decisions.
• docs/plans/2026-07-21-per-cluster-mesh-design.md §7 — mark row 6 DONE.
• CLAUDE.md — a "Per-cluster mesh (Phase 6)" note under the Redundancy section: three meshes, cluster roles, one Primary per pair, pair-local secrets, per-cluster ClusterClient, reconciler own-cluster scoped, transport defaults flipped + split validator.

Steps:

1. Write the Redundancy.md sections; remove the two caveats (IManualFailoverService + Razor).
2. Update Configuration.md.
3. Update the two plan status tables + CLAUDE.md.
4. Grep for any remaining "single mesh" / "fleet-wide Primary" / "until the per-cluster mesh work lands" copy and reconcile.

Acceptance: no stale single-mesh caveats remain; status tables show Phase 6 done; secrets pair-local + cluster-role documented.

---

Task 9: Live gate — three meshes, per-pair redundancy, cross-mesh transports

Classification: high-risk Estimated implement time: manual (controller-run against docker-dev) Parallelizable with: none (final)

Context: the exit gate. Bring the rewritten rig up and prove every previously-live-gated behavior per pair, plus the split itself. Gather TCP/log/CLI evidence; record in docs/plans/2026-07-24-mesh-phase6-live-gate.md.

Legs:

1. Three meshes form. Each node's Cluster.State.Members shows only its own pair (2 members). Evidence: redundancy CLI / logs per node, or /proc/net/tcp showing 4053 associations only within each pair. central sees central-1/2 only; site-a sees site-a-1/2 only; site-b sees site-b-1/2 only.
2. Two+ Primaries, one per pair. Each pair elects its own oldest-Up driver Primary; ServiceLevel 250/240 within each pair (Primary/Secondary), independently. Client.CLI redundancy per endpoint (4840/4841 MAIN, 4842/4843 SITE-A, 4844/4845 SITE-B).
3. Deploy reaches all clusters over per-cluster ClusterClient. POST :9200/api/deployments (X-Api-Key) seals green with all six nodes acking; central's logs show one client per cluster; verify a SITE-A node applies the config (its address space serves).
4. Telemetry per cluster over gRPC. AdminUI /alerts + /hosts pills live; central dials each site mesh's telemetry servers (2 inbound per driver node on 4056; central holds outbound per cluster). Kill-and-reconnect a site node → its stream recovers (~5s).
5. Reconciler quiet. central's ClusterNodeAddressReconciler logs no EnabledRowNotInCluster for SITE-A/SITE-B rows (own-cluster scoped now); MAIN drift still caught (probe with a deliberate mismatch if practical).
6. Secrets pair-local. With Secrets:Replication:Enabled=true, a secret written on site-a-1 replicates to site-a-2 (same mesh) and does not reach central or site-b (separate meshes) — or, if a live secret write is impractical, assert the DPS topic is mesh-scoped via the membership evidence from leg 1 + a doc note. No errors from the secrets actor about unreachable peers.
7. Split validator. Flip one site node to MeshTransport__Mode=Dps → it refuses to boot with the Task 5 message. Restore.
8. Failover within a pair (bridges toward Phase 7): kill a site pair's Primary → the Secondary becomes Primary (250) and the pair survives alone (auto-down 1-vs-1 — the deferred Phase 0a gate is now testable; run it if time permits and note the result for Phase 7).

Record: docs/plans/2026-07-24-mesh-phase6-live-gate.md with per-leg evidence + any findings. Cleanup: restore the rig to a coherent committed state.

Acceptance: all legs pass (or any deviation is documented + justified as in prior phases). Phase 6 exit gate MET.

---

Risk register (carry from design §8 + program)

• LocalDb is load-bearing for config (unchanged; #485 class).
• Two+ Primaries fleet-wide is now correct, by design — anything asserting "one Primary" is stale.
• Partial rollout hazard: a node whose mesh is split has pair-local DriverMemberCount, while a not-yet-split node counts fleet-wide — do not run the fleet half-split in production without the drill (Phase 7).
• Co-located VM loss takes out one node of both products (Phase 7 drill item; out of scope here).
• Frame size: deploy path stays payload-free — keep it that way.