fix(redundancy): elect the Primary by cluster age, not by address

RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.

They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.

BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.

NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.

Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.

Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 18:39:07 -04:00
parent 2964361a6f
commit 50b55ee4b9
18 changed files with 314 additions and 64 deletions
+20 -8
View File
@@ -6,7 +6,7 @@ OtOpcUa supports OPC UA **non-transparent** warm/hot redundancy. Two or more `Ot
> **Discovery surface.** The `ServerArray` path on the `Server` object is what each node populates with self + peer `ApplicationUri`s — see `OpcUaApplicationHost.PopulateServerArray` and the per-node `PeerApplicationUris` option below. The redundancy-object-type `ServerUriArray` proper (a child of `Server.ServerRedundancy`) remains deferred pending an SDK object-type upgrade; clients should read `Server.ServerArray` for peer discovery today. > **Discovery surface.** The `ServerArray` path on the `Server` object is what each node populates with self + peer `ApplicationUri`s — see `OpcUaApplicationHost.PopulateServerArray` and the per-node `PeerApplicationUris` option below. The redundancy-object-type `ServerUriArray` proper (a child of `Server.ServerRedundancy`) remains deferred pending an SDK object-type upgrade; clients should read `Server.ServerArray` for peer discovery today.
> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster role-leader** for the `driver` role. The operator no longer writes a role into the DB; cluster topology (specifically the `driver` role-leader) drives ServiceLevel automatically. > **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster membership** — the oldest Up member carrying the `driver` role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.
The runtime pieces live in: The runtime pieces live in:
@@ -19,7 +19,7 @@ The runtime pieces live in:
| `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). | | `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). |
| `PeerProbeSupervisor` | `OtOpcUa.Runtime.Health` | Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers. | | `PeerProbeSupervisor` | `OtOpcUa.Runtime.Health` | Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers. |
| `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. | | `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. |
| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. | | `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. (Akka's role-leader is *not* what elects the redundancy Primary — see below.) |
## ServiceLevel tiers ## ServiceLevel tiers
@@ -38,7 +38,7 @@ The four inputs are sourced locally per driver node:
| `DbReachable` | Local `DbHealthProbeActor``OpcUaPublishActor` Asks it on each `HealthTick`; an Ask timeout is treated as `Reachable=false`. | | `DbReachable` | Local `DbHealthProbeActor``OpcUaPublishActor` Asks it on each `HealthTick`; an Ask timeout is treated as `Reachable=false`. |
| `OpcUaProbeOk` | Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes. | | `OpcUaProbeOk` | Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes. |
| `Stale` (derived) | `!DbReachable \|\| (now lastDbHealth.AsOfUtc) > 30 s \|\| (now snapshotEntry.AsOfUtc) > 30 s`. | | `Stale` (derived) | `!DbReachable \|\| (now lastDbHealth.AsOfUtc) > 30 s \|\| (now snapshotEntry.AsOfUtc) > 30 s`. |
| `IsDriverRoleLeader` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. | | `IsDriverPrimary` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |
The resulting truth table (all tiers are now reachable at runtime): The resulting truth table (all tiers are now reachable at runtime):
@@ -47,8 +47,8 @@ The resulting truth table (all tiers are now reachable at runtime):
| Down / Detached | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255. | | Down / Detached | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255. |
| Critically degraded | 100 | ConfigDb unreachable AND data is stale. | | Critically degraded | 100 | ConfigDb unreachable AND data is stale. |
| Stale | 200 | Data stale but ConfigDb reachable. | | Stale | 200 | Data stale but ConfigDb reachable. |
| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not role-leader. | | Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not the driver Primary. |
| Healthy leader | 250 | Same as healthy follower + this node is the `driver` role-leader (+10 bonus). | | Healthy primary | 250 | Same as healthy follower + this node is the driver Primary (+10 bonus). |
> **Secondary 100 → 240 (behavior change).** Previously a healthy Secondary > **Secondary 100 → 240 (behavior change).** Previously a healthy Secondary
> published 100 (coarse role-only mapping). It now publishes **240** — both > published 100 (coarse role-only mapping). It now publishes **240** — both
@@ -66,11 +66,23 @@ ServiceLevel (even 0) is always published so no node lingers at the SDK default
255. 255.
Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver` Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver`
role is `Primary` when it holds the `driver` role-leader lease, otherwise role is `Primary` when it is the **oldest Up member carrying the `driver` role**, otherwise
`Secondary`; a node without the `driver` role is `Detached`. `Secondary`; a node without the `driver` role is `Detached`.
**Oldest, not role leader (changed 2026-07-21).** The election previously used
`ClusterState.RoleLeader("driver")` — the *lowest-addressed* Up member with the role. Akka offers
both, and they name different nodes: role leader is ordered by address (host, then port), oldest by
up-number. They agree on a freshly-formed cluster, and diverge after any restart — a restarted node
re-joins as the *youngest* while keeping its address, so if it holds the lower address it becomes
role leader. Meanwhile `ClusterSingletonManager` places singletons on the **oldest**, so the old
derivation could name a Primary that was not hosting the singletons, enabling the Primary-gated data
plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wrong node. Pinned by
`RedundancyPrimaryElectionTests`, which forms a real two-node cluster where the oldest node holds the
*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.** > **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
> `RoleLeader("driver")` yields exactly **one** Primary across the whole Akka cluster. A fleet that > The election yields exactly **one** Primary across the whole Akka cluster. A fleet that
> runs several application clusters (each with its own redundant pair) inside one Akka cluster — > runs several application clusters (each with its own redundant pair) inside one Akka cluster —
> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has > which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits > one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
@@ -149,7 +161,7 @@ OTOPCUA_ROLES=admin,driver
Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list. Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list.
There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster role-leadership). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula). There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
> **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers). > **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).
@@ -7,7 +7,7 @@ public readonly record struct NodeHealthInputs(
bool DbReachable, bool DbReachable,
bool OpcUaProbeOk, bool OpcUaProbeOk,
bool Stale, bool Stale,
bool IsDriverRoleLeader); bool IsDriverPrimary);
/// <summary> /// <summary>
/// Pure ServiceLevel computation per design §6. Output range 0255, where higher = "more /// Pure ServiceLevel computation per design §6. Output range 0255, where higher = "more
@@ -18,7 +18,9 @@ public readonly record struct NodeHealthInputs(
/// - Member not Up/Joining: 0 (cluster cannot trust this node). /// - Member not Up/Joining: 0 (cluster cannot trust this node).
/// - DB reachable + OPC UA probe ok + not stale: 240 (full service). /// - DB reachable + OPC UA probe ok + not stale: 240 (full service).
/// - Stale config (DB reachable or not, OPC UA probe state ignored): 100 or 200 depending on DB. /// - Stale config (DB reachable or not, OPC UA probe state ignored): 100 or 200 depending on DB.
/// - +10 bonus when this node holds the role-leader lease for the "driver" role. /// - +10 bonus when this node is the driver Primary (the oldest Up member with the "driver" role
/// where the cluster singletons live). Previously this keyed off Akka's role *leader*, the
/// lowest-addressed member, which names a different node once any node has restarted.
/// </summary> /// </summary>
public static class ServiceLevelCalculator public static class ServiceLevelCalculator
{ {
@@ -38,6 +40,6 @@ public static class ServiceLevelCalculator
_ => 0, _ => 0,
}; };
return (byte)Math.Clamp(basis + (h.IsDriverRoleLeader ? 10 : 0), 0, 255); return (byte)Math.Clamp(basis + (h.IsDriverPrimary ? 10 : 0), 0, 255);
} }
} }
@@ -8,9 +8,20 @@ public enum RedundancyRole { Primary, Secondary, Detached }
/// Snapshot of a single node's redundancy state. Aggregated by <c>RedundancyStateActor</c> /// Snapshot of a single node's redundancy state. Aggregated by <c>RedundancyStateActor</c>
/// to compute fleet-wide ServiceLevel. /// to compute fleet-wide ServiceLevel.
/// </summary> /// </summary>
/// <param name="NodeId">Canonical <c>host:port</c> id of the node this entry describes.</param>
/// <param name="Role">The node's redundancy role in the current snapshot.</param>
/// <param name="IsClusterLeader">Whether the node is the Akka cluster leader.</param>
/// <param name="IsDriverPrimary">
/// Whether the node is the driver Primary — the <b>oldest</b> Up member carrying the <c>driver</c>
/// role, which is where <c>ClusterSingletonManager</c> places singletons. Renamed from
/// <c>IsRoleLeaderForDriver</c>: it was derived from <c>ClusterState.RoleLeader("driver")</c>, the
/// lowest-<i>addressed</i> member, which diverges from the oldest after any node restart. The name
/// now describes what the value means rather than how it used to be computed.
/// </param>
/// <param name="AsOfUtc">When the snapshot was computed.</param>
public sealed record NodeRedundancyState( public sealed record NodeRedundancyState(
NodeId NodeId, NodeId NodeId,
RedundancyRole Role, RedundancyRole Role,
bool IsClusterLeader, bool IsClusterLeader,
bool IsRoleLeaderForDriver, bool IsDriverPrimary,
DateTime AsOfUtc); DateTime AsOfUtc);
@@ -111,9 +111,49 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
_log.Debug("Published RedundancyStateChanged with {Count} nodes", snapshot.Count); _log.Debug("Published RedundancyStateChanged with {Count} nodes", snapshot.Count);
} }
/// <summary>The cluster role that marks a node as carrying the driver runtime.</summary>
public const string DriverRole = "driver";
/// <summary>
/// Selects the driver Primary: the <b>oldest</b> Up member carrying the <see cref="DriverRole"/>.
/// </summary>
/// <remarks>
/// <para>
/// This deliberately does <b>not</b> use <c>ClusterState.RoleLeader(role)</c>, which it
/// previously did. Role leader is the <i>lowest-addressed</i> Up member with the role —
/// address order (host, then port), with no relationship to time. Oldest is the member
/// with the lowest up-number, and it is what <c>ClusterSingletonManager</c> uses to place
/// singletons.
/// </para>
/// <para>
/// On a freshly-formed cluster the two agree, which is why the discrepancy was invisible.
/// They diverge after any restart: the restarted node re-joins as the youngest but keeps
/// its address, so if it holds the lower address it becomes role leader while the
/// singletons — and all the work they own — stay on the other node. Electing it Primary
/// would enable the Primary-gated data plane (inbound writes, alarm acks, the fleet-wide
/// alerts emit, the alarm-history drain) on the node that is not hosting the work.
/// </para>
/// <para>
/// Only <see cref="MemberStatus.Up"/> members are eligible, matching singleton placement:
/// a <c>Leaving</c> node is handing its singletons over and must not be named Primary.
/// </para>
/// </remarks>
/// <param name="members">The current cluster members, in any order.</param>
/// <returns>The oldest Up driver member's address, or <c>null</c> when there is none.</returns>
public static Address? SelectDriverPrimary(IEnumerable<Member> members)
{
ArgumentNullException.ThrowIfNull(members);
return members
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
.OrderBy(m => m, Member.AgeOrdering)
.FirstOrDefault()
?.Address;
}
private IReadOnlyList<NodeRedundancyState> BuildSnapshot() private IReadOnlyList<NodeRedundancyState> BuildSnapshot()
{ {
var driverLeader = _cluster.State.RoleLeader("driver"); var driverPrimary = SelectDriverPrimary(_cluster.State.Members);
var clusterLeader = _cluster.State.Leader; var clusterLeader = _cluster.State.Leader;
var now = DateTime.UtcNow; var now = DateTime.UtcNow;
@@ -123,15 +163,16 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
var host = member.Address.Host; var host = member.Address.Host;
if (string.IsNullOrWhiteSpace(host)) continue; if (string.IsNullOrWhiteSpace(host)) continue;
var role = member.Roles.Contains("driver") var isPrimary = driverPrimary is not null && driverPrimary == member.Address;
? (driverLeader == member.Address ? CommonsRedundancyRole.Primary : CommonsRedundancyRole.Secondary) var role = member.Roles.Contains(DriverRole)
? (isPrimary ? CommonsRedundancyRole.Primary : CommonsRedundancyRole.Secondary)
: CommonsRedundancyRole.Detached; : CommonsRedundancyRole.Detached;
list.Add(new NodeRedundancyState( list.Add(new NodeRedundancyState(
ToNodeId(member.Address), ToNodeId(member.Address),
role, role,
IsClusterLeader: clusterLeader == member.Address, IsClusterLeader: clusterLeader == member.Address,
IsRoleLeaderForDriver: driverLeader == member.Address, IsDriverPrimary: isPrimary,
AsOfUtc: now)); AsOfUtc: now));
} }
return list; return list;
@@ -652,7 +652,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
Stale: !_lastDbHealth.Reachable Stale: !_lastDbHealth.Reachable
|| (now - _lastDbHealth.AsOfUtc) > _staleWindow || (now - _lastDbHealth.AsOfUtc) > _staleWindow
|| (now - entry.AsOfUtc) > _staleWindow, || (now - entry.AsOfUtc) > _staleWindow,
IsDriverRoleLeader: entry.IsRoleLeaderForDriver); IsDriverPrimary: entry.IsDriverPrimary);
Self.Tell(new ServiceLevelChanged(ServiceLevelCalculator.Compute(inputs))); Self.Tell(new ServiceLevelChanged(ServiceLevelCalculator.Compute(inputs)));
} }
@@ -661,7 +661,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
/// secondary → 100, _ → 0). Preserved as the back-compat / bootstrap seam.</summary> /// secondary → 100, _ → 0). Preserved as the back-compat / bootstrap seam.</summary>
private static byte LegacyRoleOnly(NodeRedundancyState entry) => entry.Role switch private static byte LegacyRoleOnly(NodeRedundancyState entry) => entry.Role switch
{ {
RedundancyRole.Primary when entry.IsRoleLeaderForDriver => 240, RedundancyRole.Primary when entry.IsDriverPrimary => 240,
RedundancyRole.Primary => 200, RedundancyRole.Primary => 200,
RedundancyRole.Secondary => 100, RedundancyRole.Secondary => 100,
_ => 0, _ => 0,
@@ -21,7 +21,7 @@ public sealed class ServiceLevelCalculatorTests
public void NotUp_returns_zero(MemberStatus status) public void NotUp_returns_zero(MemberStatus status)
{ {
var sl = ServiceLevelCalculator.Compute(new(status, var sl = ServiceLevelCalculator.Compute(new(status,
DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverRoleLeader: true)); DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverPrimary: true));
sl.ShouldBe((byte)0); sl.ShouldBe((byte)0);
} }
@@ -30,7 +30,7 @@ public sealed class ServiceLevelCalculatorTests
public void Fully_healthy_non_leader_returns_240() public void Fully_healthy_non_leader_returns_240()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverRoleLeader: false)); DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverPrimary: false));
sl.ShouldBe((byte)240); sl.ShouldBe((byte)240);
} }
@@ -39,7 +39,7 @@ public sealed class ServiceLevelCalculatorTests
public void Fully_healthy_role_leader_returns_250() public void Fully_healthy_role_leader_returns_250()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverRoleLeader: true)); DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverPrimary: true));
sl.ShouldBe((byte)250); sl.ShouldBe((byte)250);
} }
@@ -48,7 +48,7 @@ public sealed class ServiceLevelCalculatorTests
public void Db_reachable_but_stale_returns_200() public void Db_reachable_but_stale_returns_200()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: true, OpcUaProbeOk: true, Stale: true, IsDriverRoleLeader: false)); DbReachable: true, OpcUaProbeOk: true, Stale: true, IsDriverPrimary: false));
sl.ShouldBe((byte)200); sl.ShouldBe((byte)200);
} }
@@ -57,7 +57,7 @@ public sealed class ServiceLevelCalculatorTests
public void Db_unreachable_and_stale_returns_100() public void Db_unreachable_and_stale_returns_100()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: false, OpcUaProbeOk: false, Stale: true, IsDriverRoleLeader: false)); DbReachable: false, OpcUaProbeOk: false, Stale: true, IsDriverPrimary: false));
sl.ShouldBe((byte)100); sl.ShouldBe((byte)100);
} }
@@ -66,7 +66,7 @@ public sealed class ServiceLevelCalculatorTests
public void Opcua_probe_fail_when_not_stale_returns_zero() public void Opcua_probe_fail_when_not_stale_returns_zero()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: true, OpcUaProbeOk: false, Stale: false, IsDriverRoleLeader: false)); DbReachable: true, OpcUaProbeOk: false, Stale: false, IsDriverPrimary: false));
sl.ShouldBe((byte)0); sl.ShouldBe((byte)0);
} }
@@ -75,7 +75,7 @@ public sealed class ServiceLevelCalculatorTests
public void Joining_member_is_treated_like_Up() public void Joining_member_is_treated_like_Up()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Joining, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Joining,
DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverRoleLeader: false)); DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverPrimary: false));
sl.ShouldBe((byte)240); sl.ShouldBe((byte)240);
} }
@@ -85,7 +85,7 @@ public sealed class ServiceLevelCalculatorTests
{ {
// basis 240 + 10 = 250, already within byte range; confirms Clamp is in path // basis 240 + 10 = 250, already within byte range; confirms Clamp is in path
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverRoleLeader: true)); DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverPrimary: true));
((int)sl).ShouldBeLessThanOrEqualTo(255); ((int)sl).ShouldBeLessThanOrEqualTo(255);
} }
} }
@@ -0,0 +1,184 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
/// <summary>
/// Pins <b>which</b> node <see cref="RedundancyStateActor"/> elects Primary, using a real two-node
/// cluster deliberately built so that the oldest member is <i>not</i> the lowest-addressed one.
/// </summary>
/// <remarks>
/// <para>
/// The distinction is the whole point. Akka offers two different "first" members and they
/// disagree after any restart:
/// </para>
/// <list type="bullet">
/// <item>
/// <description>
/// <c>ClusterState.RoleLeader(role)</c> — the <b>lowest-addressed</b> Up member with
/// that role. Address order is host, then port; it has nothing to do with time.
/// </description>
/// </item>
/// <item>
/// <description>
/// <b>Oldest</b> — the Up member with the lowest up-number, i.e. the one that has been
/// in the cluster longest. This is what <c>ClusterSingletonManager</c> uses to place
/// singletons, and (under <c>keep-oldest</c>) what the split-brain resolver keys on.
/// </description>
/// </item>
/// </list>
/// <para>
/// They coincide on a freshly-formed cluster, which is why single-node and
/// happy-path tests never caught the difference. They diverge the moment a node restarts: the
/// restarted node becomes the <i>youngest</i> while keeping its address, so if it happens to
/// hold the lower address it becomes role leader — and the actor would have named it Primary
/// while the cluster singletons, and every piece of work they own, stayed on the other node.
/// The Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) would
/// then be enabled on the node that is not hosting the work.
/// </para>
/// <para>
/// This test reproduces that divergence directly rather than waiting for a restart: node A
/// binds the <b>higher</b> port and joins first (so it is oldest), node B binds the
/// <b>lower</b> port and joins second (so it is role leader). The correct answer is A.
/// </para>
/// </remarks>
public sealed class RedundancyPrimaryElectionTests : IAsyncLifetime
{
// Fixed, unusual ports: the test needs a deterministic address ordering, which port 0 cannot give.
private const int OldestPort = 19_531; // joins FIRST -> oldest, but the HIGHER address
private const int YoungestPort = 19_530; // joins SECOND -> role leader, the LOWER address
private ActorSystem? _oldest;
private ActorSystem? _youngest;
private static Config NodeConfig(int port) => ConfigurationFactory.ParseString($$"""
akka {
loglevel = "WARNING"
actor.provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
remote.dot-netty.tcp {
hostname = "127.0.0.1"
port = {{port}}
}
cluster {
seed-nodes = ["akka.tcp://redundancy-election@127.0.0.1:{{OldestPort}}"]
roles = ["admin", "driver"]
min-nr-of-members = 1
run-coordinated-shutdown-when-down = off
downing-provider-class = ""
}
}
""");
public async Task InitializeAsync()
{
// Order matters: the seed forms the cluster and is therefore the oldest member.
_oldest = ActorSystem.Create("redundancy-election", NodeConfig(OldestPort));
await WaitForUpAsync(_oldest, expectedMembers: 1);
_youngest = ActorSystem.Create("redundancy-election", NodeConfig(YoungestPort));
await WaitForUpAsync(_oldest, expectedMembers: 2);
await WaitForUpAsync(_youngest, expectedMembers: 2);
}
public async Task DisposeAsync()
{
if (_youngest is not null) await _youngest.Terminate();
if (_oldest is not null) await _oldest.Terminate();
}
private static async Task WaitForUpAsync(ActorSystem system, int expectedMembers)
{
var cluster = Akka.Cluster.Cluster.Get(system);
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
while (DateTime.UtcNow < deadline)
{
var up = cluster.State.Members.Count(m => m.Status == MemberStatus.Up);
if (up >= expectedMembers) return;
await Task.Delay(100);
}
throw new TimeoutException(
$"cluster on port {cluster.SelfAddress.Port} never saw {expectedMembers} Up members "
+ $"(saw {cluster.State.Members.Count(m => m.Status == MemberStatus.Up)})");
}
/// <summary>
/// Sanity check on the fixture itself: if the two orderings did not actually diverge, the real
/// assertion below would pass for the wrong reason. This asserts the setup produced the
/// divergence it claims to — role leader is the younger node, not the oldest one.
/// </summary>
[Fact]
public void Fixture_actually_produces_divergent_age_and_address_ordering()
{
var state = Akka.Cluster.Cluster.Get(_oldest!).State;
state.RoleLeader("driver")!.Port.ShouldBe(
YoungestPort,
"the fixture is only meaningful if the lowest-addressed node is the YOUNGER one");
}
/// <summary>
/// The elected Primary is the oldest driver member — the node that hosts the cluster singletons —
/// not the lowest-addressed one.
/// </summary>
[Fact]
public async Task Primary_is_the_oldest_driver_member_not_the_role_leader()
{
var snapshot = await CaptureSnapshotAsync();
var primaries = snapshot.Where(n => n.Role == RedundancyRole.Primary).ToList();
primaries.Count.ShouldBe(1, "exactly one node may be Primary");
primaries[0].NodeId.Value.ShouldBe(
$"127.0.0.1:{OldestPort}",
"Primary must follow cluster-singleton placement (oldest Up member), or the Primary-gated data plane is enabled on a node that is not hosting the work");
}
/// <summary>The younger node is Secondary, not a second Primary and not Detached.</summary>
[Fact]
public async Task Younger_driver_member_is_secondary()
{
var snapshot = await CaptureSnapshotAsync();
snapshot.Single(n => n.NodeId.Value == $"127.0.0.1:{YoungestPort}")
.Role.ShouldBe(RedundancyRole.Secondary);
}
/// <summary>
/// <c>IsDriverPrimary</c> agrees with <c>Role</c>. They feed different consumers — the OPC UA
/// ServiceLevel calculation reads the flag while the data-plane gates read the role — so a
/// disagreement would advertise one node as authoritative while gating writes on the other.
/// </summary>
[Fact]
public async Task IsDriverPrimary_flag_agrees_with_the_role()
{
var snapshot = await CaptureSnapshotAsync();
foreach (var node in snapshot)
{
node.IsDriverPrimary.ShouldBe(
node.Role == RedundancyRole.Primary,
$"{node.NodeId.Value} disagrees between Role and IsDriverPrimary");
}
}
private async Task<IReadOnlyList<NodeRedundancyState>> CaptureSnapshotAsync()
{
var tcs = new TaskCompletionSource<RedundancyStateChanged>(
TaskCreationOptions.RunContinuationsAsynchronously);
_oldest!.ActorOf(
RedundancyStateActor.Props(broadcast: msg =>
{
if (msg is RedundancyStateChanged changed) tcs.TrySetResult(changed);
}),
$"redundancy-{Guid.NewGuid():N}");
var published = await tcs.Task.WaitAsync(TimeSpan.FromSeconds(15));
return published.Nodes;
}
}
@@ -17,7 +17,7 @@ public sealed class ServiceLevelCalculatorTests
public void NotUp_returns_zero(MemberStatus status) public void NotUp_returns_zero(MemberStatus status)
{ {
var sl = ServiceLevelCalculator.Compute(new(status, var sl = ServiceLevelCalculator.Compute(new(status,
DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverRoleLeader: true)); DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverPrimary: true));
sl.ShouldBe((byte)0); sl.ShouldBe((byte)0);
} }
@@ -26,7 +26,7 @@ public sealed class ServiceLevelCalculatorTests
public void Fully_healthy_non_leader_returns_240() public void Fully_healthy_non_leader_returns_240()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverRoleLeader: false)); DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverPrimary: false));
sl.ShouldBe((byte)240); sl.ShouldBe((byte)240);
} }
@@ -35,7 +35,7 @@ public sealed class ServiceLevelCalculatorTests
public void Fully_healthy_role_leader_returns_250() public void Fully_healthy_role_leader_returns_250()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverRoleLeader: true)); DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverPrimary: true));
sl.ShouldBe((byte)250); sl.ShouldBe((byte)250);
} }
@@ -44,7 +44,7 @@ public sealed class ServiceLevelCalculatorTests
public void Db_reachable_but_stale_returns_200() public void Db_reachable_but_stale_returns_200()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: true, OpcUaProbeOk: true, Stale: true, IsDriverRoleLeader: false)); DbReachable: true, OpcUaProbeOk: true, Stale: true, IsDriverPrimary: false));
sl.ShouldBe((byte)200); sl.ShouldBe((byte)200);
} }
@@ -53,7 +53,7 @@ public sealed class ServiceLevelCalculatorTests
public void Db_unreachable_and_stale_returns_100() public void Db_unreachable_and_stale_returns_100()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: false, OpcUaProbeOk: false, Stale: true, IsDriverRoleLeader: false)); DbReachable: false, OpcUaProbeOk: false, Stale: true, IsDriverPrimary: false));
sl.ShouldBe((byte)100); sl.ShouldBe((byte)100);
} }
@@ -63,7 +63,7 @@ public sealed class ServiceLevelCalculatorTests
{ {
// (DbReachable=true, OpcUaProbeOk=false, Stale=false) falls through to the catch-all 0. // (DbReachable=true, OpcUaProbeOk=false, Stale=false) falls through to the catch-all 0.
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Up,
DbReachable: true, OpcUaProbeOk: false, Stale: false, IsDriverRoleLeader: false)); DbReachable: true, OpcUaProbeOk: false, Stale: false, IsDriverPrimary: false));
sl.ShouldBe((byte)0); sl.ShouldBe((byte)0);
} }
@@ -72,7 +72,7 @@ public sealed class ServiceLevelCalculatorTests
public void Joining_member_is_treated_like_Up_for_grading() public void Joining_member_is_treated_like_Up_for_grading()
{ {
var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Joining, var sl = ServiceLevelCalculator.Compute(new(MemberStatus.Joining,
DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverRoleLeader: false)); DbReachable: true, OpcUaProbeOk: true, Stale: false, IsDriverPrimary: false));
sl.ShouldBe((byte)240); sl.ShouldBe((byte)240);
} }
} }
@@ -106,7 +106,7 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
new[] new[]
{ {
new NodeRedundancyState(TestNode, RedundancyRole.Secondary, new NodeRedundancyState(TestNode, RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, AsOfUtc: DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, AsOfUtc: DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -330,7 +330,7 @@ public sealed class DriverHostActorNativeAlarmTests : RuntimeActorTestBase
NodeId: TestNode, NodeId: TestNode,
Role: role, Role: role,
IsClusterLeader: role == RedundancyRole.Primary, IsClusterLeader: role == RedundancyRole.Primary,
IsRoleLeaderForDriver: role == RedundancyRole.Primary, IsDriverPrimary: role == RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow), AsOfUtc: DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -216,7 +216,7 @@ public sealed class DriverHostActorPrimaryGateTests : RuntimeActorTestBase
{ {
new NodeRedundancyState(TestNode, role, new NodeRedundancyState(TestNode, role,
IsClusterLeader: role == RedundancyRole.Primary, IsClusterLeader: role == RedundancyRole.Primary,
IsRoleLeaderForDriver: role == RedundancyRole.Primary, IsDriverPrimary: role == RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow), AsOfUtc: DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -66,10 +66,10 @@ public sealed class DriverHostActorRoleViewTests : RuntimeActorTestBase
SpawnHost(view, driverMemberCount: 4).Tell(new RedundancyStateChanged( SpawnHost(view, driverMemberCount: 4).Tell(new RedundancyStateChanged(
[ [
new NodeRedundancyState(TestNode, RedundancyRole.Secondary, new NodeRedundancyState(TestNode, RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, AsOfUtc: DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, AsOfUtc: DateTime.UtcNow),
// A Primary DOES exist — in another pair. It is not this node's replication peer. // A Primary DOES exist — in another pair. It is not this node's replication peer.
new NodeRedundancyState(NodeId.Parse("some-other-pair:4053"), RedundancyRole.Primary, new NodeRedundancyState(NodeId.Parse("some-other-pair:4053"), RedundancyRole.Primary,
IsClusterLeader: true, IsRoleLeaderForDriver: true, AsOfUtc: DateTime.UtcNow), IsClusterLeader: true, IsDriverPrimary: true, AsOfUtc: DateTime.UtcNow),
], ],
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -198,13 +198,13 @@ public sealed class DriverHostActorRoleViewTests : RuntimeActorTestBase
[ [
new NodeRedundancyState(node, role, new NodeRedundancyState(node, role,
IsClusterLeader: role == RedundancyRole.Primary, IsClusterLeader: role == RedundancyRole.Primary,
IsRoleLeaderForDriver: role == RedundancyRole.Primary, IsDriverPrimary: role == RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow), AsOfUtc: DateTime.UtcNow),
new NodeRedundancyState( new NodeRedundancyState(
NodeId.Parse($"{PeerHost}:4053"), NodeId.Parse($"{PeerHost}:4053"),
role == RedundancyRole.Primary ? RedundancyRole.Secondary : RedundancyRole.Primary, role == RedundancyRole.Primary ? RedundancyRole.Secondary : RedundancyRole.Primary,
IsClusterLeader: role != RedundancyRole.Primary, IsClusterLeader: role != RedundancyRole.Primary,
IsRoleLeaderForDriver: role != RedundancyRole.Primary, IsDriverPrimary: role != RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow), AsOfUtc: DateTime.UtcNow),
], ],
CorrelationId.NewId()); CorrelationId.NewId());
@@ -106,7 +106,7 @@ public sealed class DriverHostActorWriteRoutingTests : RuntimeActorTestBase
new[] new[]
{ {
new NodeRedundancyState(TestNode, RedundancyRole.Secondary, new NodeRedundancyState(TestNode, RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, AsOfUtc: DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, AsOfUtc: DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -38,7 +38,7 @@ public sealed class PeerProbeSupervisorTests : RuntimeActorTestBase
Akka.Actor.Props.Create(() => new RecordingNoopActor(spawned)); Akka.Actor.Props.Create(() => new RecordingNoopActor(spawned));
private static NodeRedundancyState State(NodeId id, RedundancyRole role) => private static NodeRedundancyState State(NodeId id, RedundancyRole role) =>
new(id, role, IsClusterLeader: false, IsRoleLeaderForDriver: false, DateTime.UtcNow); new(id, role, IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow);
private static RedundancyStateChanged Snapshot(params NodeRedundancyState[] nodes) => private static RedundancyStateChanged Snapshot(params NodeRedundancyState[] nodes) =>
new(nodes, CorrelationId.NewId()); new(nodes, CorrelationId.NewId());
@@ -85,7 +85,7 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
NodeId: LocalNode, NodeId: LocalNode,
Role: role, Role: role,
IsClusterLeader: role == RedundancyRole.Primary, IsClusterLeader: role == RedundancyRole.Primary,
IsRoleLeaderForDriver: role == RedundancyRole.Primary, IsDriverPrimary: role == RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow), AsOfUtc: DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -166,7 +166,7 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
NodeId: new NodeId("some-other-node"), NodeId: new NodeId("some-other-node"),
Role: RedundancyRole.Secondary, Role: RedundancyRole.Secondary,
IsClusterLeader: false, IsClusterLeader: false,
IsRoleLeaderForDriver: false, IsDriverPrimary: false,
AsOfUtc: DateTime.UtcNow), AsOfUtc: DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -181,7 +181,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Detached, new NodeRedundancyState(local, RedundancyRole.Detached,
IsClusterLeader: false, IsRoleLeaderForDriver: false, DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -202,9 +202,9 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Primary, new NodeRedundancyState(local, RedundancyRole.Primary,
IsClusterLeader: true, IsRoleLeaderForDriver: true, DateTime.UtcNow), IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow),
new NodeRedundancyState(NodeId.Parse("other-node"), RedundancyRole.Secondary, new NodeRedundancyState(NodeId.Parse("other-node"), RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow),
}, },
CorrelationId.NewId()); CorrelationId.NewId());
actor.Tell(snapshot); actor.Tell(snapshot);
@@ -225,7 +225,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Secondary, new NodeRedundancyState(local, RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow),
}, },
CorrelationId.NewId()); CorrelationId.NewId());
actor.Tell(snapshot); actor.Tell(snapshot);
@@ -251,7 +251,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Primary, new NodeRedundancyState(local, RedundancyRole.Primary,
IsClusterLeader: true, IsRoleLeaderForDriver: true, DateTime.UtcNow), IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -277,7 +277,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Secondary, new NodeRedundancyState(local, RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -302,7 +302,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Secondary, new NodeRedundancyState(local, RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -329,7 +329,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Secondary, new NodeRedundancyState(local, RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, IsClusterLeader: false, IsDriverPrimary: false,
DateTime.UtcNow - TimeSpan.FromMinutes(1)), DateTime.UtcNow - TimeSpan.FromMinutes(1)),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -358,7 +358,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Primary, new NodeRedundancyState(local, RedundancyRole.Primary,
IsClusterLeader: true, IsRoleLeaderForDriver: true, DateTime.UtcNow), IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
AwaitAssert(() => publisher.Levels.ShouldContain((byte)250), AwaitAssert(() => publisher.Levels.ShouldContain((byte)250),
@@ -369,7 +369,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Detached, new NodeRedundancyState(local, RedundancyRole.Detached,
IsClusterLeader: false, IsRoleLeaderForDriver: false, DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -398,7 +398,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Secondary, new NodeRedundancyState(local, RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -441,7 +441,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Primary, new NodeRedundancyState(local, RedundancyRole.Primary,
IsClusterLeader: true, IsRoleLeaderForDriver: true, DateTime.UtcNow), IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -469,7 +469,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Primary, new NodeRedundancyState(local, RedundancyRole.Primary,
IsClusterLeader: true, IsRoleLeaderForDriver: true, DateTime.UtcNow), IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -498,7 +498,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Secondary, new NodeRedundancyState(local, RedundancyRole.Secondary,
IsClusterLeader: false, IsRoleLeaderForDriver: false, DateTime.UtcNow), IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -526,7 +526,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Primary, new NodeRedundancyState(local, RedundancyRole.Primary,
IsClusterLeader: true, IsRoleLeaderForDriver: true, DateTime.UtcNow), IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -548,7 +548,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Primary, new NodeRedundancyState(local, RedundancyRole.Primary,
IsClusterLeader: true, IsRoleLeaderForDriver: true, DateTime.UtcNow), IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
@@ -574,7 +574,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
actor.Tell(new RedundancyStateChanged(new[] actor.Tell(new RedundancyStateChanged(new[]
{ {
new NodeRedundancyState(local, RedundancyRole.Primary, new NodeRedundancyState(local, RedundancyRole.Primary,
IsClusterLeader: true, IsRoleLeaderForDriver: true, DateTime.UtcNow), IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow),
}, CorrelationId.NewId())); }, CorrelationId.NewId()));
// No direct DbHealthStatus Tell — the periodic HealthTick Ask must populate it → 250. // No direct DbHealthStatus Tell — the periodic HealthTick Ask must populate it → 250.
AwaitAssert(() => publisher.Levels.ShouldContain((byte)250), TimeSpan.FromSeconds(2)); AwaitAssert(() => publisher.Levels.ShouldContain((byte)250), TimeSpan.FromSeconds(2));
@@ -51,7 +51,7 @@ public sealed class ServiceLevelEndToEndTests : RuntimeActorTestBase
actor.Tell(new RedundancyStateChanged( actor.Tell(new RedundancyStateChanged(
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(localNode, RedundancyRole.Primary, IsClusterLeader: true, IsRoleLeaderForDriver: true, AsOfUtc: DateTime.UtcNow), new NodeRedundancyState(localNode, RedundancyRole.Primary, IsClusterLeader: true, IsDriverPrimary: true, AsOfUtc: DateTime.UtcNow),
}, },
CorrelationId: CorrelationId.NewId())); CorrelationId: CorrelationId.NewId()));
@@ -91,7 +91,7 @@ public sealed class ServiceLevelEndToEndTests : RuntimeActorTestBase
actor.Tell(new RedundancyStateChanged( actor.Tell(new RedundancyStateChanged(
Nodes: new[] Nodes: new[]
{ {
new NodeRedundancyState(localNode, RedundancyRole.Secondary, IsClusterLeader: false, IsRoleLeaderForDriver: false, AsOfUtc: DateTime.UtcNow), new NodeRedundancyState(localNode, RedundancyRole.Secondary, IsClusterLeader: false, IsDriverPrimary: false, AsOfUtc: DateTime.UtcNow),
}, },
CorrelationId: CorrelationId.NewId())); CorrelationId: CorrelationId.NewId()));
@@ -102,7 +102,7 @@ public sealed class ScriptedAlarmHostActorTests : RuntimeActorTestBase
NodeId: id, NodeId: id,
Role: role, Role: role,
IsClusterLeader: role == RedundancyRole.Primary, IsClusterLeader: role == RedundancyRole.Primary,
IsRoleLeaderForDriver: role == RedundancyRole.Primary, IsDriverPrimary: role == RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow), AsOfUtc: DateTime.UtcNow),
}, },
CorrelationId.NewId())); CorrelationId.NewId()));