fix(mesh): scope ClusterNode address reconcile to the admin node's own cluster

Phase 6 splits the fleet into one mesh per Cluster, so an admin (central)
node's Cluster.State.Members shows only its own pair — it no longer sees
site members via gossip. The ClusterNodeAddressReconciler singleton compared
live membership against EVERY ClusterNode row fleet-wide, so post-split every
foreign-cluster row would log EnabledRowNotInCluster forever.

Scope the actor's two DB queries to rows whose ClusterId matches the admin
node's own (IClusterRoleInfo.ClusterId, sourced at registration). A legacy
admin node with no cluster role (null ClusterId) still reconciles the whole
fleet — it genuinely sees every member via gossip. The pure Reconcile
function and AddressMismatchKind semantics are unchanged; scoping lives
entirely in the actor's queries.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 01:39:06 -04:00
parent a2c5d57fa0
commit 2535df019b
6 changed files with 149 additions and 40 deletions
@@ -176,10 +176,14 @@ DPS. **De-risked:** `RedundancyStateActor` has zero injected deps (verified —
public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo)
{
var clusterRole = roleInfo.ClusterRole
?? throw new InvalidOperationException(
"A driver node must carry a cluster-{ClusterId} role to host the redundancy singleton.");
var opts = new ClusterSingletonOptions { Role = clusterRole };
// Cluster-scoped when the node carries a cluster-{ClusterId} role (Phase-6 split: one
// redundancy singleton per pair, robust even if two meshes accidentally merged). Falls
// back to the driver role for a legacy single-mesh node or a test harness with no cluster
// role — where driver-scope is the pre-Phase-6 fleet-wide behavior, and on a genuinely
// split 2-node mesh the driver role is ALREADY pair-local because the mesh IS the pair.
// No throw: decision 2 requires legacy/harness nodes to keep booting unchanged.
var role = roleInfo.ClusterRole ?? RoleParser.Driver;
var opts = new ClusterSingletonOptions { Role = role };
return builder.WithSingleton<RedundancyStateActorKey>(
RedundancyStateActor.Topic, RedundancyStateActor.Props(), opts /* createProxyToo per existing pattern */);
}
@@ -190,11 +194,12 @@ DPS. **De-risked:** `RedundancyStateActor` has zero injected deps (verified —
`WithOtOpcUaRuntimeActors`), call `ab.WithOtOpcUaClusterRedundancySingleton(clusterRoleInfo)`.
Resolve `IClusterRoleInfo` from `sp`. Ensure it runs on the fused central node too (central has
`hasDriver=true`). Do NOT also register it in the `hasAdmin` block.
4. **Guard: a driver node with no cluster role must fail fast** (the new extension throws) — this is
caught more helpfully by the Task 5 validator, but the throw is the backstop.
4. **No fail-fast on a missing cluster role** — the fallback keeps legacy/harness nodes booting
(decision 2). A *misconfigured* Phase-6 node (split transports, no cluster role) is a non-issue for
the redundancy singleton: driver-scope is pair-local on a split mesh anyway.
5. Tests: assert the admin singleton set no longer contains `redundancy-state`; assert the new
extension registers a singleton scoped to `Role == clusterRole`; a node with null `ClusterRole`
throws.
extension registers a singleton scoped to `Role == "cluster-SITE-A"` when the roleInfo reports that;
assert that a roleInfo with null `ClusterRole` falls back to `Role == "driver"` (no throw).
**Acceptance:** each mesh hosts exactly one `RedundancyStateActor` scoped to its cluster role; a
driver-only site node elects its own Primary; admin path no longer owns it. Unit tests green.
@@ -229,16 +234,29 @@ not assert anything about their rows.
finding); an own-cluster enabled row with no member still yields `EnabledRowNotInCluster`; drift
detection within own cluster still works.
**The pure `ClusterNodeAddressReconciler.Reconcile` function is UNCHANGED** — the fix is entirely in
the actor's two DB queries (`ClusterNodeAddressReconcilerActor.Reconcile`, lines 89-97), which filter
rows by the admin node's own `ClusterId` before projecting. `ClusterNode` has a `ClusterId` column.
**Steps:**
1. Thread the reconciler's own `ClusterId` (from `IClusterRoleInfo.ClusterId`, which for a fused
central node is `"MAIN"`) into `Reconcile`. Filter `rows` to `row.ClusterId == ownClusterId` before
any comparison. (`ClusterNode` has a `ClusterId` column — confirmed.)
2. Keep `RowDialTargetDisagrees` / `RunningNodeHasNoRow` / `EnabledRowNotInCluster` semantics for
own-cluster rows only.
3. If `IClusterRoleInfo.ClusterId` is null on an admin node (shouldn't happen post-Phase-6, but guard),
log one Warning and skip the sweep rather than reconciling the whole fleet.
4. Update the class doc: replace the deferred-to-Phase-6 note with the delivered behavior.
5. Tests as above.
1. Inject the admin node's own `ClusterId` into the actor: add `IClusterRoleInfo` (or just a
`string? ownClusterId`) to `Props`/ctor, sourced from `IClusterRoleInfo.ClusterId` at registration
(`ControlPlane/ServiceCollectionExtensions.cs` where the reconciler singleton is registered).
2. In the actor's `Reconcile`, add `.Where(n => n.ClusterId == ownClusterId)` to BOTH the `rows` and
`enabled` queries — but ONLY when `ownClusterId` is non-null.
3. **Null `ownClusterId` (legacy admin node with no cluster role) ⇒ reconcile ALL rows** — this is the
correct single-mesh behavior (a legacy admin node genuinely sees every driver member via gossip).
Do NOT skip. This preserves backward-compat, mirroring the Task 2 fallback philosophy.
4. `RowDialTargetDisagrees` / `RunningNodeHasNoRow` / `EnabledRowNotInCluster` semantics are unchanged;
they now simply operate on the own-cluster row subset (post-split) or the full set (legacy).
5. Update the class doc comment in `ClusterNodeAddressReconciler.cs:60-65`: replace the "Phase 2 must
revisit" note with the delivered own-cluster-scoped behavior.
6. Tests (`ClusterNodeAddressReconcilerActorTests` or the existing reconciler tests — pick the level
that can inject `ownClusterId`): a SITE-A row is **ignored** when `ownClusterId=="MAIN"` (no
`EnabledRowNotInCluster`); an own-cluster (MAIN) enabled row with no member still yields
`EnabledRowNotInCluster`; own-cluster drift still caught; `ownClusterId==null` reconciles the full
set (legacy). If the actor is hard to unit-test directly, a focused query-filter test + the
unchanged pure-function tests suffice — say which you chose.
**Acceptance:** central reconciles only MAIN rows; site rows never produce false
`EnabledRowNotInCluster`; own-cluster drift still caught.
@@ -9,8 +9,8 @@
"rigScope": "OtOpcUa-only three 2-node meshes on one shared network"
},
"tasks": [
{"id": 0, "subject": "Task 0: RoleParser accepts cluster-{ClusterId}; consolidate driver literal", "classification": "small", "status": "pending"},
{"id": 1, "subject": "Task 1: ClusterRoleInfo exposes node's own ClusterRole/ClusterId", "classification": "small", "status": "pending", "blockedBy": [0]},
{"id": 0, "subject": "Task 0: RoleParser accepts cluster-{ClusterId}; consolidate driver literal", "classification": "small", "status": "completed", "commit": "87ff00fd"},
{"id": 1, "subject": "Task 1: ClusterRoleInfo exposes node's own ClusterRole/ClusterId", "classification": "small", "status": "completed", "commit": "2d2f1dec", "note": "interface in Commons/Interfaces/IClusterRoleInfo.cs; ctor (ActorSystem, IOptions<AkkaClusterOptions>, ILogger)"},
{"id": 2, "subject": "Task 2: re-home RedundancyStateActor to cluster-role singleton on drivers", "classification": "high-risk", "status": "pending", "blockedBy": [1], "parallelizableWith": [3, 4]},
{"id": 3, "subject": "Task 3: re-scope ClusterNodeAddressReconciler to own-cluster rows", "classification": "standard", "status": "pending", "blockedBy": [1], "parallelizableWith": [2, 4]},
{"id": 4, "subject": "Task 4: one ClusterClient per cluster in CentralCommunicationActor", "classification": "high-risk", "status": "pending", "blockedBy": [], "parallelizableWith": [2, 3]},