dohertj2/lmxopcua
1
0
Fork 0
Code Issues 12 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(mesh): Phase 6 plan — mesh partition + co-location topology

Browse Source 
Plan + task ledger for splitting the single fleet mesh into three 2-node
meshes (central + per-site pairs), cluster-scoped roles + redundancy
singleton, per-cluster ClusterClient, own-cluster reconciler, split-topology
transport validator + flipped defaults, pair-local secrets, and the docker-dev
rig rewrite. Decisions settled with the user 2026-07-24.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 01:07:54 -04:00
parent 35552a9608
commit 02177fec7c
2 changed files with 537 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md
+513
View File
@@ -0,0 +1,513 @@
# Per-Cluster Mesh Phase 6 — Mesh Partition + Co-location Topology
> **For Claude:** REQUIRED SUB-SKILL: execute this plan task-by-task with
> superpowers-extended-cc:subagent-driven-development (the established mode for this workstream).
**Goal:** Split OtOpcUa's single fleet-wide Akka mesh into **one 2-node mesh per application
`Cluster`** (central pair + one pair per site), so every Primary-gated decision and every gated
resource shares one pair-local scope — the ScadaBridge shape. Same ActorSystem name (`otopcua`)
everywhere; separation is by **seed-node partitioning** only.
**Architecture:** Each pair's two nodes list only each other as seeds (self-first), so Akka forms
three independent meshes even on a shared network. Driver nodes carry a new `cluster-{ClusterId}`
role; the one pair-local singleton (`RedundancyStateActor`) re-scopes to it and is spawned on every
driver node, so each pair elects its own Primary and publishes `redundancy-state` on its own mesh's
DPS. Central (admin) reaches each site mesh over the explicit Phase 2/3/5 transports (ClusterClient
+ ConfigServe gRPC + telemetry gRPC), now partitioned **one ClusterClient per cluster**. The five
admin singletons stay on the central pair. Two admin-side consumers that assumed fleet-wide gossip
visibility (`ClusterNodeAddressReconciler`, `CentralCommunicationActor.RebuildClient`) are
re-scoped. Secrets replication becomes **pair-local** automatically (Akka DPS within each 2-node
mesh; no fleet-wide sync, no central-SQL dependency — consistent with Phase 4). A startup validator
refuses to boot a split-configured node (one carrying a `cluster-` role) that is still pointed at a
DPS transport mode, and the compiled transport defaults flip to the split-correct values.
**Tech Stack:** .NET 10, Akka.NET + Akka.Hosting (`WithSingleton`/`WithActors`), Akka.Cluster.Tools
(ClusterClient + ClusterSingleton + DistributedPubSub), Akka.Cluster.Hosting `ClusterSingletonOptions`,
xUnit + Shouldly, docker-dev compose rig.
**Decisions settled with the user 2026-07-24 (recon-driven; the first contradicts the program doc's
own guess):**
1. **Secrets after split = pair-local Akka.** The program doc guessed "SQL-hub mode like ScadaBridge,"
but recon showed sites have **no SQL Server** (the point of Phase 4), so a SQL hub would force site
nodes back onto central SQL. Instead: leave the `AddZbSecretsAkkaReplication` wiring untouched — it
becomes per-pair for free once each mesh is two nodes. Fleet-wide secret sync is dropped by design;
document it. No library re-wire.
2. **DPS branch disposition = flip + change defaults + validator, keep the code.** The rig flips to
`ClusterClient`/`Grpc`/`FetchAndCache`; the compiled defaults change so a fresh deploy is
split-correct; the DPS branch code stays as dark switches; a new startup validator makes a
split-configured node refuse to boot on a DPS transport mode (neutralizes the silent-break footgun
without a large irreversible deletion). `redundancy-state`/`fleet-status` stay on DPS either way.
3. **Rig scope = OtOpcUa-only per-cluster meshes.** Rewrite docker-dev into 3 independent 2-node
meshes on a single shared network (faithful to production, where separation is by seeds over a WAN,
not by network isolation). Physical co-location with ScadaBridge is documented as the production
topology, not literally run in docker-dev.
**Authoritative context (read before implementing):**
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` §"Phase 6" (lines 266–284) + the tracking table.
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §4 (oldest-Up election, already fixed), §5
(target architecture), §7 (sequencing).
- Recon findings (this session) — summarized inline per task below.
---
## What Phase 6 does NOT change (scope guards)
- **ActorSystem name stays `otopcua`.** Do not rename it or make it per-cluster.
- **`AkkaClusterOptions` gains no new property.** `SeedNodes` + `Roles` already carry per-node values
from config; the split is in *how they're populated per pair* (rig/appsettings) + the `RoleParser`
allow-list + singleton scoping. Do not add a `ClusterId` option — every `ClusterNode` row is already
defined to be a driver node (Phase 1 decision), and roles are the declaration of record.
- **`SelectDriverPrimary` election logic is unchanged.** It filters on the `driver` role over
`Cluster.State.Members`; after the split that set is pair-local, so it elects the oldest of two.
No edit to the algorithm.
- **The five admin singletons stay admin-scoped** (`config-publish`, `admin-operations`,
`audit-writer`, `fleet-status`, `cluster-node-address-reconciler`). Only `redundancy-state`
re-homes.
- **`DriverMemberCount()`** in `DriverHostActor`/`ScriptedAlarmHostActor` needs **no code change** —
it reads whole-mesh members filtered by `driver`, which becomes pair-local for free. (Note the
semantics change in docs; do not "fix" it in code.)
- **Do NOT delete the DPS command/telemetry branches** (decision 2). Do NOT switch the network to
per-site isolation or wire in ScadaBridge (decision 3). Do NOT re-wire secrets to SQL (decision 1).
---
### Task 0: `RoleParser` accepts `cluster-{ClusterId}`; consolidate the `driver` role literal
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (blocks 1, 2, 5, 7)
**Context:** `RoleParser.Allowed` is a fixed `HashSet {"admin","driver","dev"}`
(`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs:5-8`). A `cluster-{ClusterId}` role fails it, so
no node can carry one. Also, three independent `"driver"` string literals exist
(`RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`,
`Runtime/ServiceCollectionExtensions.DriverRole`) — consolidate to a single source now since every
Phase 6 task touches role code.
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleParserTests.cs` (create if absent)
**Steps:**
1. Add a well-known-roles surface to `RoleParser` (or a new `ClusterRoles` static): constants
`Admin = "admin"`, `Driver = "driver"`, `Dev = "dev"`, `ClusterRolePrefix = "cluster-"`, and a
helper `bool IsClusterRole(string role) => role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length`.
2. Change `Parse` validation: a role is allowed if it is one of the three fixed roles **or**
`IsClusterRole(role)` is true. Keep the existing trim/lowercase/empty-filter behavior. A role of
exactly `"cluster-"` (empty ClusterId) is **rejected**.
3. Add `RoleParser.ClusterIdFromRole(string) => role.Substring(ClusterRolePrefix.Length)` returning
the ClusterId for a cluster role (used by later tasks).
4. Update `RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`, and
`Runtime/ServiceCollectionExtensions.DriverRole` to reference the single `RoleParser.Driver` (keep
the public `const`/`static readonly` symbols as aliases if any test references them by their old
name — grep first; do not break existing test references).
5. Tests: `admin`/`driver`/`dev`/`cluster-MAIN`/`cluster-SITE-A` accepted; `cluster-` rejected;
`bogus` rejected; case/whitespace normalization preserved; `ClusterIdFromRole("cluster-SITE-A") == "SITE-A"`.
**Acceptance:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests` green; `cluster-{X}` roles
parse; the three literals resolve to one constant.
---
### Task 1: `ClusterRoleInfo` exposes the node's own cluster-scoped role
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (needs Task 0; blocks 2, 5)
**Context:** Singleton registration and the validator both need "what is *this* node's
`cluster-{ClusterId}` role." `ClusterRoleInfo` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`,
`IClusterRoleInfo`, registered in `AddOtOpcUaCluster`) is the existing role/leader authority — extend
it.
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/IClusterRoleInfo.cs` (or wherever the interface lives)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ClusterRoleInfoTests.cs`
**Steps:**
1. Add `string? ClusterRole { get; }` (e.g. `"cluster-SITE-A"`) and `string? ClusterId { get; }` (e.g.
`"SITE-A"`) to `IClusterRoleInfo`, derived from the configured `AkkaClusterOptions.Roles` via
`RoleParser.IsClusterRole` (first match; log a Warning and pick the first if >1 cluster role is
configured — that is a misconfiguration). Null when the node carries no cluster role (e.g. a
pre-split/admin-only-legacy node).
2. Do NOT read `Cluster.State` for this — it is the node's **own configured** identity, available
before the cluster forms (the singleton registration runs at host build time).
3. Tests: a node configured `["driver","cluster-SITE-A"]` reports `ClusterRole="cluster-SITE-A"`,
`ClusterId="SITE-A"`; a node `["admin","driver"]` reports both null; two cluster roles → first wins
+ warning path exercised.
**Acceptance:** `ClusterRoleInfo` reports the node's cluster role/id from config; tests green.
---
### Task 2: Re-home `RedundancyStateActor` to a `cluster-{ClusterId}` singleton on driver nodes
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3, Task 4 (disjoint files)
**Context (the crux).** Today `RedundancyStateActor` is registered as an **admin** singleton
(`ControlPlane/ServiceCollectionExtensions.cs:133-136`, `singletonOptions.Role = "admin"`), electing a
**fleet-wide** Primary and publishing `redundancy-state` over DPS. After the split, central's mesh
sees only its own pair — so a site pair (driver-only, no admin node) would have **no one** to elect its
Primary. Fix: scope the singleton to the node's `cluster-{ClusterId}` role and spawn it on **every
driver node** (`hasDriver`), so each mesh (central pair = `cluster-MAIN`, each site pair =
`cluster-{SITE}`) has exactly one, electing its own pair-local Primary and publishing on its own mesh's
DPS. **De-risked:** `RedundancyStateActor` has zero injected deps (verified — it only uses
`Cluster.Get(Context.System)` + DPS), so it spawns cleanly on a driver-only node.
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs`
(remove `redundancy-state` from `WithOtOpcUaControlPlaneSingletons`; add a new
`WithOtOpcUaClusterRedundancySingleton(IClusterRoleInfo)` extension that registers it scoped to the
cluster role)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (call the new extension on `hasDriver`, not
`hasAdmin`; a fused `admin,driver` node calls it once via the driver branch)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/...` singleton-role test; and confirm the
admin path no longer registers `redundancy-state`.
**Steps:**
1. In `ControlPlane/ServiceCollectionExtensions.cs`: delete the `WithSingleton` block for
`RedundancyStateActorKey`/`redundancy-state` from the admin method (lines ~133-136). Keep the
registry key type.
2. Add:
```csharp
public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo)
{
var clusterRole = roleInfo.ClusterRole
?? throw new InvalidOperationException(
"A driver node must carry a cluster-{ClusterId} role to host the redundancy singleton.");
var opts = new ClusterSingletonOptions { Role = clusterRole };
return builder.WithSingleton<RedundancyStateActorKey>(
RedundancyStateActor.Topic, RedundancyStateActor.Props(), opts /* createProxyToo per existing pattern */);
}
```
Match the exact `WithSingleton` overload/signature the other five singletons use (check createProxy
arg + singleton name conventions).
3. In `Program.cs`: within the `if (hasDriver)` Akka-configurator block (near line 357-374, alongside
`WithOtOpcUaRuntimeActors`), call `ab.WithOtOpcUaClusterRedundancySingleton(clusterRoleInfo)`.
Resolve `IClusterRoleInfo` from `sp`. Ensure it runs on the fused central node too (central has
`hasDriver=true`). Do NOT also register it in the `hasAdmin` block.
4. **Guard: a driver node with no cluster role must fail fast** (the new extension throws) — this is
caught more helpfully by the Task 5 validator, but the throw is the backstop.
5. Tests: assert the admin singleton set no longer contains `redundancy-state`; assert the new
extension registers a singleton scoped to `Role == clusterRole`; a node with null `ClusterRole`
throws.
**Acceptance:** each mesh hosts exactly one `RedundancyStateActor` scoped to its cluster role; a
driver-only site node elects its own Primary; admin path no longer owns it. Unit tests green.
**Live-verified later** (Task 9 gate): two Primaries fleet-wide, one per pair.
---
### Task 3: Re-scope `ClusterNodeAddressReconciler` to central-mesh-visible rows
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 4 (disjoint files)
**Context:** The admin-singleton reconciler compares `Cluster.State.Members` (driver role) against
**every** `ClusterNode` row fleet-wide (`ClusterNodeAddressReconciler.cs:123-131`,
`ClusterNodeAddressReconcilerActor.cs:76-96`). After the split, central sees only its own pair, so
every site row logs `EnabledRowNotInCluster` forever (its own doc comment,
`ClusterNodeAddressReconciler.cs:60-65`, flags this). Fix: only reconcile rows whose `ClusterId`
matches central's own cluster — central can no longer observe other clusters via gossip, so it must
not assert anything about their rows.
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconciler.cs` (filter
the row set by own-cluster)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs`
(pass the admin node's own `ClusterId` in; source it from `IClusterRoleInfo.ClusterId`, or from the
set of `ClusterId`s present among its own observed driver members)
- Update the class doc comment (remove the "Phase 2 must revisit" note; state it is now own-cluster
scoped).
- Test: `tests/.../ClusterNodeAddressReconcilerTests.cs` — a row in another cluster is **ignored** (no
finding); an own-cluster enabled row with no member still yields `EnabledRowNotInCluster`; drift
detection within own cluster still works.
**Steps:**
1. Thread the reconciler's own `ClusterId` (from `IClusterRoleInfo.ClusterId`, which for a fused
central node is `"MAIN"`) into `Reconcile`. Filter `rows` to `row.ClusterId == ownClusterId` before
any comparison. (`ClusterNode` has a `ClusterId` column — confirmed.)
2. Keep `RowDialTargetDisagrees` / `RunningNodeHasNoRow` / `EnabledRowNotInCluster` semantics for
own-cluster rows only.
3. If `IClusterRoleInfo.ClusterId` is null on an admin node (shouldn't happen post-Phase-6, but guard),
log one Warning and skip the sweep rather than reconciling the whole fleet.
4. Update the class doc: replace the deferred-to-Phase-6 note with the delivered behavior.
5. Tests as above.
**Acceptance:** central reconciles only MAIN rows; site rows never produce false
`EnabledRowNotInCluster`; own-cluster drift still caught.
---
### Task 4: One `ClusterClient` per application `Cluster` in `CentralCommunicationActor`
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 3 (disjoint files)
**Context:** `RebuildClient` (`CentralCommunicationActor.cs:271-297`) creates **one fleet-wide**
`ClusterClient` dialing every `ClusterNode` row, and `HandleMeshCommand` (`:128-157`) does one
`ClusterClient.SendToAll(...)`. The `TODO(Phase 6)` (`:249-268`) states this is correct only on a
single mesh: a `ClusterClientReceptionist` serves its whole cluster, so `SendToAll` on one client
reaches everyone. After the split, `SendToAll` on one client reaches only the mesh whose receptionist
answered — so central needs **one client per cluster**, and dispatch fans `SendToAll` across all of
them.
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs`
(extend: two separate site meshes + central, a command reaches BOTH via per-cluster clients; and a
single-cluster client's `SendToAll` does NOT reach the other cluster)
**Steps:**
1. `LoadContactsFromDb` (`:177-221`): return contacts **grouped by `ClusterId`** — an
`ImmutableDictionary<string, ImmutableHashSet<string>>` (ClusterId → receptionist contact paths).
Keep the `Enabled && !MaintenanceMode` filter.
2. Replace the single `_client`/`_currentContacts` with a `Dictionary<string, (IActorRef Client,
ImmutableHashSet<string> Contacts)>` keyed by ClusterId. `HandleContactsLoaded` diffs **per cluster**
and rebuilds only the clusters whose contact set changed; stops+removes clients for clusters that
vanished.
3. `RebuildClient` becomes `RebuildClient(string clusterId, ImmutableHashSet<string> contacts)` —
builds one `ClusterClient` per cluster (name it by clusterId so the actor paths are distinct).
4. `HandleMeshCommand` fans out: for each cluster client, `client.Tell(new ClusterClient.SendToAll(
MeshPaths.NodeCommunication, cmd.Message))`. Preserve the sender-relay idiom where it exists.
**A command with no live clients (all clusters down) drops with a Warning**, matching today's
"no reachable contact" behavior — do not buffer.
5. Update the `TODO(Phase 6)` doc block to describe the delivered per-cluster-client shape (remove the
TODO marker).
6. **Frame-size note:** the deploy path stays payload-free (notify-and-fetch), so no new frame-size
exposure — do not add payload to these commands.
7. Tests: extend `MeshClusterClientBoundaryTests` to two site meshes + central; assert a dispatch
reaches a node in each site mesh, and that per-cluster `SendToAll` is correctly partitioned.
**Acceptance:** central dispatches deploy-notify/driver-control/alarm-commands to every cluster via
one client each; the boundary test proves cross-mesh partitioned delivery; acks still return via the
site nodes' own `ClusterClient.Send` to central (unchanged).
---
### Task 5: Split-topology startup validator (cluster-role ⇒ non-DPS transport)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 6 (disjoint files)
**Context (decision 2):** the DPS branches stay as dark switches, but a node configured for the split
topology (it carries a `cluster-{ClusterId}` role) must not silently run a transport still in DPS
mode, which cannot cross meshes. Fail host start instead. Marker of "split topology" = the presence of
a `cluster-` role (they exist only post-Phase-6).
**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs` (an
`IValidateOptions<...>` or a startup validator wired via `AddValidatedOptions`/`ValidateOnStart`,
mirroring `AkkaClusterOptionsValidator`/`ConfigSourceOptionsValidator` structure)
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`AddOtOpcUaCluster`
registers it)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitTopologyTransportValidatorTests.cs`
**Rule (only fires when the node carries a `cluster-` role):**
- `MeshTransport:Mode` must be `ClusterClient` (not `Dps`) — every node.
- If `hasDriver`: `Telemetry:Mode` must be `Grpc` (node hosts the telemetry server).
- If `hasAdmin`: `TelemetryDial:Mode` must be `Grpc` (central dials).
- `ConfigSource:Mode`: **do not** add a rule here — Phase 4's `ConfigSourceOptionsValidator` already
requires `FetchAndCache` on driver-only nodes; a fused node legitimately stays `Direct`.
- A node with **no** `cluster-` role (legacy/pre-split) is exempt (rule does not fire), so existing
single-mesh deployments and tests are unaffected.
**Steps:**
1. Read `Cluster:Roles` (same direct-read pattern the other validators use — not `IClusterRoleInfo`,
to avoid a service-resolution ordering dependency at validation time), `MeshTransport:Mode`,
`Telemetry:Mode`, `TelemetryDial:Mode`.
2. Implement the rule above; each violation adds a precise message naming the offending key + required
value.
3. Register with `ValidateOnStart` so a misconfigured node refuses to boot.
4. Tests: cluster-role node on `MeshTransport:Mode=Dps` fails; on `ClusterClient` passes; driver
cluster node on `Telemetry:Mode=Dps` fails; admin cluster node on `TelemetryDial:Mode=Dps` fails;
a non-cluster-role node on all-Dps passes (exempt).
**Acceptance:** a split-configured node on any DPS transport refuses to start with a clear message;
legacy nodes unaffected.
---
### Task 6: Flip compiled transport-mode defaults to split-correct
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 5 (disjoint files)
**Context (decision 2):** so a fresh deploy is split-correct without per-node overrides, change the
compiled defaults. `redundancy-state`/`fleet-status` stay DPS regardless.
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs` (default `Mode` → `ClusterClient`)
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/TelemetryOptions.cs` (`TelemetryOptions.Mode` and
`TelemetryDialOptions.Mode` defaults → `Grpc`)
- Leave `ConfigSourceOptions.Mode` default = `Direct` (fused central stays Direct; Phase 4 validator
forces driver-only to FetchAndCache).
- Update tests that assert the old defaults (grep for `ModeDps`/`Mode == "Dps"`/`.Mode.ShouldBe` in
the Cluster.Tests and any options tests; adjust to the new defaults). Update `appsettings.json`
comments/keys if they document the default.
**Steps:**
1. Change the three default initializers.
2. Grep + update every test asserting the prior default (there are a handful — do not miss the
Phase 2/5 option tests).
3. Confirm the docker-dev rig (Task 7) does not *rely* on the old default anywhere (it will set modes
explicitly anyway).
**Acceptance:** `new MeshTransportOptions().Mode == ClusterClient`; both telemetry defaults `Grpc`;
all options tests green.
---
### Task 7: Rewrite the docker-dev rig into three 2-node meshes
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (needs Tasks 0–6 landed to actually run; the file edit is independent
but sequence it after code so the rig is testable)
**Context (decision 3):** one shared network, seed-partitioned into three meshes. Central pair =
`cluster-MAIN`; site-a pair = `cluster-SITE-A`; site-b pair = `cluster-SITE-B`. Each pair lists only
its own two nodes as seeds (self-first). Flip the transports on. Secrets stays enabled (now
pair-local). Central must remain network-reachable to site nodes (shared network — already true; do
**not** isolate networks).
**Files:**
- Modify: `docker-dev/docker-compose.yml`
- Modify: `docker-dev/seed/seed-clusters.sql` if any seeded address/port changes (AkkaPort stays 4053;
GrpcPort stays 4056 — likely no seed change, but verify the ClusterId/host columns still match).
**Per-node changes:**
| Node | `Cluster__Roles` | `Cluster__SeedNodes__0` (self) | `__1` (partner) |
|---|---|---|---|
| central-1 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-1:4053` | `...central-2:4053` |
| central-2 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-2:4053` | `...central-1:4053` |
| site-a-1 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-1:4053` | `...site-a-2:4053` |
| site-a-2 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-2:4053` | `...site-a-1:4053` |
| site-b-1 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-1:4053` | `...site-b-2:4053` |
| site-b-2 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-2:4053` | `...site-b-1:4053` |
**Transport flips (all nodes unless noted):**
- `MeshTransport__Mode: ClusterClient` (remove the `${OTOPCUA_MESH_MODE:-Dps}` default or set the var
default to ClusterClient).
- `Telemetry__Mode: Grpc` (all); `TelemetryDial__Mode: Grpc` (central pair only — the dialers).
- `ConfigSource__Mode: FetchAndCache` already on sites; central stays `Direct`.
- `MeshTransport__CentralContactPoints__0/1` on **site** nodes point at central-1/central-2 (unchanged
— sites learn central from appsettings). Central nodes do not need CentralContactPoints for
themselves; central discovers *site* contacts from `ClusterNode` rows (per-cluster clients, Task 4).
- **Secrets** (`x-secrets-env` anchor): keep `Secrets__Replication__Enabled=true` — it becomes
pair-local automatically. Add a compose comment noting the new pair-local semantics.
- **Remove** the `central-1`-as-seed lines from every site node (the old cross-pair seed). Update the
header comment block (lines 1-23) to describe three meshes, not one.
- **LocalDb sync:** keep site-a's replication demo; optionally add a distinct sync port for site-b to
model per-site ports (nice-to-have, not required for the gate — if added, use a different host-safe
port and its own peer address).
**Steps:**
1. Rewrite the six nodes' `Cluster__Roles__*` and `Cluster__SeedNodes__*` per the table.
2. Flip the transport-mode env vars.
3. Update the header comment + secrets comment.
4. `docker compose -f docker-dev/docker-compose.yml config` to validate YAML (no live bring-up in this
task — that is the Task 9 gate).
**Acceptance:** `docker compose config` validates; the six nodes carry per-pair self-first seeds +
cluster roles + split-correct transport modes; header comment describes three meshes.
---
### Task 8: Docs sweep — status tables, caveat removal, pair-local secrets note
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 7 (docs only; no build)
**Files:**
- `docs/Redundancy.md` — new §"Per-cluster meshes (Phase 6)": one Primary per pair (**two+ Primaries
fleet-wide, by design**), redundancy-state singleton is now `cluster-{ClusterId}`-scoped and
spawned per driver node; remove any single-mesh caveat. Add §"Secrets replication is pair-local".
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs:34-35` — remove the
"Mesh-scope caveat / until the per-cluster mesh work lands" doc comment (it landed).
- AdminUI ClusterRedundancy page — remove the mesh-scope caveat text (grep the Razor for the caveat
string).
- `docs/Configuration.md` — document the `cluster-{ClusterId}` role, per-pair `Cluster:SeedNodes`
ordering, the new transport-mode defaults, and the split-topology validator.
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` — mark Phase 6 **DONE** in the section + the
tracking table; note the three decisions.
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §7 — mark row 6 DONE.
- `CLAUDE.md` — a "Per-cluster mesh (Phase 6)" note under the Redundancy section: three meshes, cluster
roles, one Primary per pair, pair-local secrets, per-cluster ClusterClient, reconciler own-cluster
scoped, transport defaults flipped + split validator.
**Steps:**
1. Write the Redundancy.md sections; remove the two caveats (IManualFailoverService + Razor).
2. Update Configuration.md.
3. Update the two plan status tables + CLAUDE.md.
4. Grep for any remaining "single mesh" / "fleet-wide Primary" / "until the per-cluster mesh work
lands" copy and reconcile.
**Acceptance:** no stale single-mesh caveats remain; status tables show Phase 6 done; secrets
pair-local + cluster-role documented.
---
### Task 9: Live gate — three meshes, per-pair redundancy, cross-mesh transports
**Classification:** high-risk
**Estimated implement time:** manual (controller-run against docker-dev)
**Parallelizable with:** none (final)
**Context:** the exit gate. Bring the rewritten rig up and prove every previously-live-gated behavior
per pair, plus the split itself. Gather TCP/log/CLI evidence; record in
`docs/plans/2026-07-24-mesh-phase6-live-gate.md`.
**Legs:**
1. **Three meshes form.** Each node's `Cluster.State.Members` shows **only its own pair** (2 members).
Evidence: redundancy CLI / logs per node, or `/proc/net/tcp` showing 4053 associations only within
each pair. central sees central-1/2 only; site-a sees site-a-1/2 only; site-b sees site-b-1/2 only.
2. **Two+ Primaries, one per pair.** Each pair elects its own oldest-Up driver Primary; ServiceLevel
**250/240** within each pair (Primary/Secondary), independently. Client.CLI `redundancy` per
endpoint (4840/4841 MAIN, 4842/4843 SITE-A, 4844/4845 SITE-B).
3. **Deploy reaches all clusters over per-cluster ClusterClient.** `POST :9200/api/deployments`
(X-Api-Key) seals green with all six nodes acking; central's logs show one client per cluster;
verify a SITE-A node applies the config (its address space serves).
4. **Telemetry per cluster over gRPC.** AdminUI `/alerts` + `/hosts` pills live; central dials each
site mesh's telemetry servers (2 inbound per driver node on 4056; central holds outbound per cluster).
Kill-and-reconnect a site node → its stream recovers (~5s).
5. **Reconciler quiet.** central's `ClusterNodeAddressReconciler` logs **no** `EnabledRowNotInCluster`
for SITE-A/SITE-B rows (own-cluster scoped now); MAIN drift still caught (probe with a deliberate
mismatch if practical).
6. **Secrets pair-local.** With `Secrets:Replication:Enabled=true`, a secret written on site-a-1
replicates to site-a-2 (same mesh) and does **not** reach central or site-b (separate meshes) — or,
if a live secret write is impractical, assert the DPS topic is mesh-scoped via the membership
evidence from leg 1 + a doc note. No errors from the secrets actor about unreachable peers.
7. **Split validator.** Flip one site node to `MeshTransport__Mode=Dps` → it refuses to boot with the
Task 5 message. Restore.
8. **Failover within a pair** (bridges toward Phase 7): kill a site pair's Primary → the Secondary
becomes Primary (250) and the pair survives alone (auto-down 1-vs-1 — the deferred Phase 0a gate is
now testable; run it if time permits and note the result for Phase 7).
**Record:** `docs/plans/2026-07-24-mesh-phase6-live-gate.md` with per-leg evidence + any findings.
**Cleanup:** restore the rig to a coherent committed state.
**Acceptance:** all legs pass (or any deviation is documented + justified as in prior phases). Phase 6
exit gate MET.
---
## Risk register (carry from design §8 + program)
- LocalDb is load-bearing for config (unchanged; #485 class).
- **Two+ Primaries fleet-wide is now correct, by design** — anything asserting "one Primary" is stale.
- Partial rollout hazard: a node whose mesh is split has pair-local `DriverMemberCount`, while a
not-yet-split node counts fleet-wide — do not run the fleet half-split in production without the
drill (Phase 7).
- Co-located VM loss takes out one node of **both** products (Phase 7 drill item; out of scope here).
- Frame size: deploy path stays payload-free — keep it that way.
docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.tasks.json
+24
View File
@@ -0,0 +1,24 @@
{
"planPath": "docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md",
"program": "per-cluster-mesh",
"phase": 6,
"branch": "feat/mesh-phase6",
"decisions": {
"secrets": "pair-local Akka (no SQL-hub; sites have no SQL)",
"dpsBranches": "flip rig + change compiled defaults + split-topology validator; keep DPS code",
"rigScope": "OtOpcUa-only three 2-node meshes on one shared network"
},
"tasks": [
{"id": 0, "subject": "Task 0: RoleParser accepts cluster-{ClusterId}; consolidate driver literal", "classification": "small", "status": "pending"},
{"id": 1, "subject": "Task 1: ClusterRoleInfo exposes node's own ClusterRole/ClusterId", "classification": "small", "status": "pending", "blockedBy": [0]},
{"id": 2, "subject": "Task 2: re-home RedundancyStateActor to cluster-role singleton on drivers", "classification": "high-risk", "status": "pending", "blockedBy": [1], "parallelizableWith": [3, 4]},
{"id": 3, "subject": "Task 3: re-scope ClusterNodeAddressReconciler to own-cluster rows", "classification": "standard", "status": "pending", "blockedBy": [1], "parallelizableWith": [2, 4]},
{"id": 4, "subject": "Task 4: one ClusterClient per cluster in CentralCommunicationActor", "classification": "high-risk", "status": "pending", "blockedBy": [], "parallelizableWith": [2, 3]},
{"id": 5, "subject": "Task 5: split-topology startup validator (cluster-role => non-DPS)", "classification": "standard", "status": "pending", "blockedBy": [0, 1], "parallelizableWith": [6]},
{"id": 6, "subject": "Task 6: flip compiled transport-mode defaults (ClusterClient/Grpc)", "classification": "small", "status": "pending", "blockedBy": [], "parallelizableWith": [5]},
{"id": 7, "subject": "Task 7: rewrite docker-dev rig into three 2-node meshes", "classification": "standard", "status": "pending", "blockedBy": [0, 2, 3, 4, 5, 6]},
{"id": 8, "subject": "Task 8: docs sweep — status tables, caveat removal, pair-local secrets note", "classification": "small", "status": "pending", "blockedBy": [2, 3, 4, 5, 6], "parallelizableWith": [7]},
{"id": 9, "subject": "Task 9: live gate — three meshes, per-pair redundancy, cross-mesh transports","classification": "high-risk", "status": "pending", "blockedBy": [7, 8]}
],
"lastUpdated": "2026-07-24T00:00:00Z"
}