feat(mesh): re-home the redundancy singleton to a per-cluster driver singleton

Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.

Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.

The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.

The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 01:28:47 -04:00
parent 2d2f1decae
commit a2c5d57fa0
3 changed files with 225 additions and 4 deletions
@@ -12,6 +12,7 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Services;
using ZB.MOM.WW.OtOpcUa.ControlPlane;
@@ -371,7 +372,17 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
ab.WithOtOpcUaSignalRBridges();
}
if (hasDriver)
{
ab.WithOtOpcUaRuntimeActors();
// Per-cluster mesh Phase 6: re-home the redundancy-state singleton off the admin node onto
// EVERY driver node, scoped to the node's own cluster-{ClusterId} role, so each mesh elects
// its own pair-local Primary (a driver-only site pair has no admin node to do it). Runs on
// the fused central node too (it is hasDriver), where it scopes to cluster-MAIN — exactly one
// redundancy singleton per node, since the admin branch above no longer registers one.
// IClusterRoleInfo is registered by AddOtOpcUaCluster (resolved lazily here); it throws at
// host start if this driver node carries no cluster-{ClusterId} role.
ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>());
}
});
// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own