fix(redundancy): a failed OPC UA endpoint drops ServiceLevel to 0 and steps the Primary down (lmxopcua#762)
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
IOpcUaEndpointState (review-remediation lane C3) recorded whether the SDK started, but only the health check read it: a fused node whose endpoint failed (port 4840 taken, ...) still advertised 250/240 and stayed driver Primary, so the Primary-gated data plane ran on a node no client could reach while the reachable peer refused writes as Secondary. - Move IOpcUaEndpointState/OpcUaEndpointState/OpcUaEndpointStatus from Host to Commons.OpcUa so the Runtime can consume it. - ServiceLevelCalculator: new NodeHealthInputs.EndpointListening (default true). false -> 0, ahead of every other input including the +10 Primary bonus. - OpcUaPublishActor: takes the endpoint state, feeds EndpointListening (Failed only; NotStarted / Stopped are boot/shutdown windows) on the DB-backed and DB-less paths, and the legacy role-only seam returns 0 for a failed endpoint too. - New OpcUaEndpointStepDownActor (one per driver node): a Primary (ClusterActiveNode.SelfIsActive, the election's own primitive) with a Failed endpoint gracefully Leaves its mesh - the manual failover mechanism - so the peer becomes the oldest Up driver member, hosts the singletons and is Primary. The election itself is not bent. Guarded on a peer having recently reported its own endpoint listening (EndpointReport on the new opcua-endpoint-state DPS topic), so a pair with both endpoints failed does not bounce the role and restart nodes forever, and a lone node never shuts itself down. - Runtime now references ZB.MOM.WW.Health.Akka (ClusterActiveNode). - Docs: Redundancy.md tier table + new "Endpoint start failure: Primary step-down" section; cluster-topology rule; HealthEndpoints / OpcUaEndpointHealthCheck comments (F-C1 landed). Tests: ServiceLevelCalculatorTests (+5), OpcUaPublishActorEndpointStateTests (7: DB-less, DB-backed, legacy seam, NotStarted/Stopped do not demote, a later failure demotes on the next tick), OpcUaEndpointStepDownActorTests (3, real two-node mesh: holds while the peer is not listening then leaves and the peer becomes Primary; failed Secondary and healthy Primary never leave). Each was mutation-checked: removing the calculator/legacy guard, the Leave, the peer-listening guard or the Primary guard fails the corresponding tests.
This commit is contained in:
Joseph Doherty
+119
@@ -0,0 +1,119 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Akka.Actor;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Health;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.OpcUa;
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#762: a node whose own OPC UA SDK failed to start must not advertise 250/240. Each test pairs the
|
||||
/// failed endpoint with a positive control — the SAME actor wiring and snapshot with a started endpoint — so a
|
||||
/// 0 cannot come from some other demotion (Detached, a missing entry, a peer probe).
|
||||
/// </summary>
|
||||
public sealed class OpcUaPublishActorEndpointStateTests : RuntimeActorTestBase
|
||||
{
|
||||
/// <summary>DB-less driver node (the survive-alone posture): Primary would be 250, a failed endpoint is 0.</summary>
|
||||
[Theory]
|
||||
[InlineData(true, 250)]
|
||||
[InlineData(false, 240)]
|
||||
public void DbLess_failed_endpoint_publishes_zero_instead_of_healthy_level(bool primary, byte healthy)
|
||||
{
|
||||
PublishedLevelFor(OpcUaEndpointStatus.Started, primary, dbLess: true).ShouldBe(healthy, "positive control");
|
||||
PublishedLevelFor(OpcUaEndpointStatus.Failed, primary, dbLess: true).ShouldBe((byte)0);
|
||||
}
|
||||
|
||||
/// <summary>DB-backed node on the full calculator path (DB reachable, fresh): same rule.</summary>
|
||||
[Fact]
|
||||
public void DbBacked_failed_endpoint_publishes_zero_instead_of_250()
|
||||
{
|
||||
PublishedLevelFor(OpcUaEndpointStatus.Started, primary: true, dbLess: false, dbBacked: true).ShouldBe((byte)250, "positive control");
|
||||
PublishedLevelFor(OpcUaEndpointStatus.Failed, primary: true, dbLess: false, dbBacked: true).ShouldBe((byte)0);
|
||||
}
|
||||
|
||||
/// <summary>The legacy role-only seam (no DB probe wired yet) must not hand a failed endpoint 240 either.</summary>
|
||||
[Fact]
|
||||
public void Legacy_role_only_seam_failed_endpoint_publishes_zero_instead_of_240()
|
||||
{
|
||||
PublishedLevelFor(OpcUaEndpointStatus.Started, primary: true, dbLess: false).ShouldBe((byte)240, "positive control");
|
||||
PublishedLevelFor(OpcUaEndpointStatus.Failed, primary: true, dbLess: false).ShouldBe((byte)0);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Boot (<see cref="OpcUaEndpointStatus.NotStarted"/>) and shutdown (<see cref="OpcUaEndpointStatus.Stopped"/>)
|
||||
/// are not faults: only a recorded start FAILURE demotes.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(OpcUaEndpointStatus.NotStarted)]
|
||||
[InlineData(OpcUaEndpointStatus.Stopped)]
|
||||
public void Only_a_failed_start_demotes(OpcUaEndpointStatus status)
|
||||
{
|
||||
PublishedLevelFor(status, primary: true, dbLess: true).ShouldBe((byte)250);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The fact is read on every recompute, so an endpoint that fails after the first healthy publish demotes on
|
||||
/// the next health tick without any message being sent to the actor.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Endpoint_failing_later_demotes_on_the_next_tick()
|
||||
{
|
||||
var publisher = new RecordingPublisher();
|
||||
var state = new OpcUaEndpointState();
|
||||
state.MarkStarted();
|
||||
var local = NodeId.Parse("endpoint-later-node");
|
||||
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
||||
serviceLevel: publisher, localNode: local, dbLess: true,
|
||||
healthTickInterval: TimeSpan.FromMilliseconds(25), endpointState: state));
|
||||
actor.Tell(Snapshot(local, primary: true));
|
||||
AwaitAssert(() => publisher.Levels.ShouldBe(new byte[] { 250 }), PresenceBudget);
|
||||
|
||||
state.MarkFailed("port 4840 in use");
|
||||
AwaitAssert(() => publisher.Levels.ShouldBe(new byte[] { 250, 0 }), PresenceBudget);
|
||||
}
|
||||
|
||||
private byte PublishedLevelFor(OpcUaEndpointStatus status, bool primary, bool dbLess, bool dbBacked = false)
|
||||
{
|
||||
var publisher = new RecordingPublisher();
|
||||
var state = new OpcUaEndpointState();
|
||||
switch (status)
|
||||
{
|
||||
case OpcUaEndpointStatus.Started: state.MarkStarted(); break;
|
||||
case OpcUaEndpointStatus.Failed: state.MarkFailed("port 4840 in use"); break;
|
||||
case OpcUaEndpointStatus.Stopped: state.MarkStopped(); break;
|
||||
}
|
||||
|
||||
var local = NodeId.Parse($"endpoint-{Guid.NewGuid():N}");
|
||||
// A DB-backed actor needs a probe ref to leave the legacy seam; its Ask is never answered here (the 5-min
|
||||
// tick keeps it to the one start-up Ask), and a healthy sample is Told directly BEFORE the snapshot so the
|
||||
// snapshot's recompute — the first publish — runs the full calculator path.
|
||||
var db = dbBacked ? CreateTestProbe() : null;
|
||||
var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests(
|
||||
serviceLevel: publisher, localNode: local, dbLess: dbLess, dbHealthProbe: db?.Ref,
|
||||
healthTickInterval: TimeSpan.FromMinutes(5), endpointState: state));
|
||||
if (db is not null)
|
||||
actor.Tell(new DbHealthProbeActor.DbHealthStatus(true, DateTime.UtcNow, null));
|
||||
actor.Tell(Snapshot(local, primary));
|
||||
AwaitAssert(() => publisher.Levels.Length.ShouldBeGreaterThanOrEqualTo(1), PresenceBudget);
|
||||
|
||||
Sys.Stop(actor);
|
||||
return publisher.Levels[0];
|
||||
}
|
||||
|
||||
private static RedundancyStateChanged Snapshot(NodeId local, bool primary) =>
|
||||
new(new[] { new NodeRedundancyState(local,
|
||||
primary ? RedundancyRole.Primary : RedundancyRole.Secondary,
|
||||
IsClusterLeader: primary, IsDriverPrimary: primary, DateTime.UtcNow) }, CorrelationId.NewId());
|
||||
|
||||
private sealed class RecordingPublisher : IServiceLevelPublisher
|
||||
{
|
||||
private readonly ConcurrentQueue<byte> _levels = new();
|
||||
public byte[] Levels => _levels.ToArray();
|
||||
public void Publish(byte serviceLevel) => _levels.Enqueue(serviceLevel);
|
||||
}
|
||||
}
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Redundancy;
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#762 on a real two-node driver mesh: a Primary whose OPC UA endpoint failed to start hands the role to
|
||||
/// its peer by leaving the mesh — but only once the peer reports a listening endpoint of its own.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Node <c>A</c> is a second <see cref="TestKit"/> that forms the cluster first, so it is the OLDEST Up driver
|
||||
/// member — the node the election (<see cref="ClusterActiveNode.OldestUpMember(IEnumerable{Member}, string?)"/>)
|
||||
/// names Primary. This class's own <see cref="TestKit.Sys"/> is node <c>B</c>, which joins second; it survives
|
||||
/// every test, so the base class disposes it normally.
|
||||
/// </remarks>
|
||||
public sealed class OpcUaEndpointStepDownActorTests : TestKit
|
||||
{
|
||||
private const string Hocon = @"
|
||||
akka {
|
||||
loglevel = ""WARNING""
|
||||
test.single-expect-default = 15s
|
||||
extensions = [
|
||||
""Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools""
|
||||
]
|
||||
actor.provider = ""Akka.Cluster.ClusterActorRefProvider, Akka.Cluster""
|
||||
remote.dot-netty.tcp {
|
||||
hostname = ""127.0.0.1""
|
||||
port = 0
|
||||
}
|
||||
cluster {
|
||||
seed-nodes = []
|
||||
roles = [""driver""]
|
||||
min-nr-of-members = 1
|
||||
run-coordinated-shutdown-when-down = off
|
||||
pub-sub.gossip-interval = 200ms
|
||||
}
|
||||
}";
|
||||
|
||||
private static readonly TimeSpan Tick = TimeSpan.FromMilliseconds(100);
|
||||
|
||||
/// <summary>How long a "does NOT leave" assertion watches — many ticks, and well past pub-sub convergence.</summary>
|
||||
private static readonly TimeSpan AbsenceWindow = TimeSpan.FromSeconds(3);
|
||||
|
||||
private readonly TestKit _nodeA;
|
||||
|
||||
public OpcUaEndpointStepDownActorTests() : base(Hocon + TestTimeFactor.Hocon)
|
||||
{
|
||||
_nodeA = new TestKit(Hocon + TestTimeFactor.Hocon);
|
||||
var clusterA = Akka.Cluster.Cluster.Get(_nodeA.Sys);
|
||||
clusterA.Join(clusterA.SelfAddress);
|
||||
AwaitCondition(() => UpCount(clusterA) == 1, RuntimeActorTestBase.PresenceBudget);
|
||||
|
||||
var clusterB = Akka.Cluster.Cluster.Get(Sys);
|
||||
clusterB.Join(clusterA.SelfAddress);
|
||||
AwaitCondition(() => UpCount(clusterA) == 2 && UpCount(clusterB) == 2, RuntimeActorTestBase.PresenceBudget);
|
||||
|
||||
// Fixture sanity: A must be the Primary going in, or every assertion below is about the wrong node.
|
||||
ClusterActiveNode.SelfIsActive(clusterA, RoleParser.Driver).ShouldBeTrue("node A formed the cluster, so it is the oldest Up driver member");
|
||||
ClusterActiveNode.SelfIsActive(clusterB, RoleParser.Driver).ShouldBeFalse();
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Dispose(bool disposing)
|
||||
{
|
||||
if (disposing) _nodeA.Dispose();
|
||||
base.Dispose(disposing);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The issue's test: A's endpoint failed to start, so A steps down and B becomes Primary. Also pins the
|
||||
/// ping-pong guard first — while B's own endpoint is not listening, A holds the role (handing it over would
|
||||
/// only move the outage and restart A for nothing); the moment B reports listening, A leaves.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Failed_primary_hands_the_role_to_a_peer_once_the_peer_reports_listening()
|
||||
{
|
||||
var stateA = Failed();
|
||||
var stateB = new OpcUaEndpointState(); // NotStarted: not listening yet
|
||||
_nodeA.Sys.ActorOf(OpcUaEndpointStepDownActor.Props(stateA, Tick), "step-down");
|
||||
Sys.ActorOf(OpcUaEndpointStepDownActor.Props(stateB, Tick), "step-down");
|
||||
|
||||
// Guard: no listening peer -> A stays Up and Primary.
|
||||
await Task.Delay(AbsenceWindow);
|
||||
_nodeA.Sys.WhenTerminated.IsCompleted.ShouldBeFalse("A must not leave while no peer can serve either");
|
||||
ClusterActiveNode.SelfIsActive(Akka.Cluster.Cluster.Get(_nodeA.Sys), RoleParser.Driver).ShouldBeTrue();
|
||||
|
||||
// B's endpoint comes up -> A leaves (its ActorSystem terminates, which in production exits the process)...
|
||||
stateB.MarkStarted();
|
||||
await _nodeA.Sys.WhenTerminated.WaitAsync(RuntimeActorTestBase.PresenceBudget);
|
||||
|
||||
// ... and B is now the oldest Up driver member, i.e. what RedundancyStateActor elects Primary.
|
||||
var clusterB = Akka.Cluster.Cluster.Get(Sys);
|
||||
AwaitCondition(() => ClusterActiveNode.SelfIsActive(clusterB, RoleParser.Driver)
|
||||
&& clusterB.State.Members.Count(m => m.Status == MemberStatus.Up) == 1,
|
||||
RuntimeActorTestBase.PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>A failed endpoint on the SECONDARY changes nothing: there is no role to hand over.</summary>
|
||||
[Fact]
|
||||
public async Task Failed_secondary_does_not_leave()
|
||||
{
|
||||
var stateA = new OpcUaEndpointState();
|
||||
stateA.MarkStarted();
|
||||
_nodeA.Sys.ActorOf(OpcUaEndpointStepDownActor.Props(stateA, Tick), "step-down");
|
||||
Sys.ActorOf(OpcUaEndpointStepDownActor.Props(Failed(), Tick), "step-down");
|
||||
|
||||
await Task.Delay(AbsenceWindow);
|
||||
Sys.WhenTerminated.IsCompleted.ShouldBeFalse();
|
||||
_nodeA.Sys.WhenTerminated.IsCompleted.ShouldBeFalse();
|
||||
UpCount(Akka.Cluster.Cluster.Get(Sys)).ShouldBe(2);
|
||||
ClusterActiveNode.SelfIsActive(Akka.Cluster.Cluster.Get(_nodeA.Sys), RoleParser.Driver).ShouldBeTrue();
|
||||
}
|
||||
|
||||
/// <summary>A healthy Primary never leaves, even with a listening peer (positive control for the trigger).</summary>
|
||||
[Fact]
|
||||
public async Task Healthy_primary_does_not_leave()
|
||||
{
|
||||
var stateA = new OpcUaEndpointState();
|
||||
stateA.MarkStarted();
|
||||
var stateB = new OpcUaEndpointState();
|
||||
stateB.MarkStarted();
|
||||
_nodeA.Sys.ActorOf(OpcUaEndpointStepDownActor.Props(stateA, Tick), "step-down");
|
||||
Sys.ActorOf(OpcUaEndpointStepDownActor.Props(stateB, Tick), "step-down");
|
||||
|
||||
await Task.Delay(AbsenceWindow);
|
||||
_nodeA.Sys.WhenTerminated.IsCompleted.ShouldBeFalse();
|
||||
UpCount(Akka.Cluster.Cluster.Get(Sys)).ShouldBe(2);
|
||||
}
|
||||
|
||||
private static OpcUaEndpointState Failed()
|
||||
{
|
||||
var state = new OpcUaEndpointState();
|
||||
state.MarkFailed("port 4840 in use");
|
||||
return state;
|
||||
}
|
||||
|
||||
private static int UpCount(Akka.Cluster.Cluster cluster) =>
|
||||
cluster.State.Members.Count(m => m.Status == MemberStatus.Up);
|
||||
}
|
||||
Reference in New Issue
Block a user