Files
lmxopcua/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Health/PeerProbeSupervisorTests.cs
T
Joseph Doherty 154171f48c test(runtime): fix the intermittent Runtime.Tests failure — 4 ordering/logic defects + the presence-budget class (#500)
The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test
name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it
was never ONE flaky test: five failures across three distinct tests in that batch, and
five more distinct tests over the verification rounds that followed.

Two hypotheses were tested and DISPROVED by measurement before anything was changed:

- Cluster-formation timeout. Every test in this assembly forms a real single-node Akka
  cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up
  — 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero
  timeouts. A 3.4x margin; not the cause.
- Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout.
  Not the cause.

FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout):

1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place
   The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and
   then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register,
   Unregister] interleaving LastSender is the DYING child and the assertion fails. Now
   captures the sender of the Register message as it arrives. Fully deterministic; no
   timing involved. (Swept the other four LastSender sites: all drain the Unregister
   first, so their ordering is fixed. Only this one was unsafe.)

2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath
   ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is
   still in Connecting, which deliberately fast-fails writes ("driver not connected"),
   and the NodeId->driver reverse map has not been pushed. Now retries until accepted.
   This does not weaken the assertion — every rejection branch replies WITHOUT reaching
   the driver, so Writes.Count.ShouldBe(1) still means exactly what it did.

3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_...
   One 500 ms constant served both presence budgets (AwaitAssert) and absence windows
   (ExpectNoMsg). Different quantities that happened to share a number, so the presence
   budget could not be raised without slowing every absence check. Split into
   AssertTimeout (5 s) and Settle (500 ms).

4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks
   Waited on `outbox == 0`, which is ALSO true before the first append — so the poll
   could win the race against the recorder and return having observed the INITIAL state,
   after which the CallCount>=2 check outside the block failed against a recorder that
   had not run. Both conditions are now polled together with the retry count as the
   discriminator. (The preceding test in the same file documents this exact trap.)

THE PRESENCE-BUDGET CLASS:

After those four, three consecutive 30-run rounds each still failed once — on a
DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s,
OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a
distribution, not a defect, and fixing it one test at a time was the wrong method.

RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s
AwaitAssert/AwaitCondition durations across four files route through it.

Why this is safe rather than sloppy: a presence budget is an upper bound before giving
up, not a wait — these helpers poll and return the instant the condition holds. Raising
one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass;
it only changes how quickly a real breakage reports. Absence windows are the opposite
(the elapsed time IS the assertion) and were deliberately left untouched with their own
short, individually-calibrated literals.

ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its
8 s budget was sized for message passing but actually waits on a REAL Roslyn compile
(ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold
script compiles are the slowest and most variable thing in the assembly, so that class
gets 30 s with the reason recorded.

Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13%
baseline. Full solution builds; all 41 unit-test projects green.

NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use
AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They
are excluded from the budget change on purpose — a bigger number does not fix them, and
rewriting them to settle-then-assert may legitimately turn them red.
2026-07-25 20:20:06 -04:00

159 lines
6.9 KiB
C#

using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Runtime.Health;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Health;
/// <summary>
/// Tests for <see cref="PeerProbeSupervisor"/>: it maintains exactly one peer-OPC-UA-probe child
/// per OTHER, non-Detached driver node named in the latest <see cref="RedundancyStateChanged"/>
/// snapshot, spawning/stopping children as the topology changes.
/// </summary>
public sealed class PeerProbeSupervisorTests : RuntimeActorTestBase
{
private static readonly NodeId Local = NodeId.Parse("local:4053");
private static readonly NodeId Peer = NodeId.Parse("peer:4053");
private static readonly NodeId Adm = NodeId.Parse("adm:4053");
/// <summary>No-op child actor stub so we can count children without real TCP probes.</summary>
private sealed class NoopActor : ReceiveActor { }
private static Props NoopProps() => Akka.Actor.Props.Create(() => new NoopActor());
/// <summary>
/// No-op child stub that records its own <see cref="ActorBase.Self"/> into a shared list on
/// start, in spawn order — lets a test grab a specific (e.g. the FIRST, old-generation) child
/// ref so it can deliver a synthetic <see cref="Terminated"/> for it.
/// </summary>
private sealed class RecordingNoopActor : ReceiveActor
{
public RecordingNoopActor(List<IActorRef> spawned) => spawned.Add(Self);
}
private static Props RecordingProps(List<IActorRef> spawned) =>
Akka.Actor.Props.Create(() => new RecordingNoopActor(spawned));
private static NodeRedundancyState State(NodeId id, RedundancyRole role) =>
new(id, role, IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow);
private static RedundancyStateChanged Snapshot(params NodeRedundancyState[] nodes) =>
new(nodes, CorrelationId.NewId());
/// <summary>Verifies one child is spawned per non-self, non-Detached peer — self and Detached
/// nodes are excluded.</summary>
[Fact]
public void Spawns_one_child_per_non_self_non_detached_peer()
{
var sup = ActorOfAsTestActorRef<PeerProbeSupervisor>(
PeerProbeSupervisor.PropsForTests(Local, _ => NoopProps()));
sup.Tell(Snapshot(
State(Local, RedundancyRole.Primary),
State(Peer, RedundancyRole.Secondary),
State(Adm, RedundancyRole.Detached)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(1),
duration: PresenceBudget);
}
/// <summary>Verifies the child for a departed peer is stopped when the next snapshot omits it.</summary>
[Fact]
public void Stops_child_for_departed_peer()
{
var sup = ActorOfAsTestActorRef<PeerProbeSupervisor>(
PeerProbeSupervisor.PropsForTests(Local, _ => NoopProps()));
sup.Tell(Snapshot(
State(Local, RedundancyRole.Primary),
State(Peer, RedundancyRole.Secondary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(1),
duration: PresenceBudget);
sup.Tell(Snapshot(State(Local, RedundancyRole.Primary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(0),
duration: PresenceBudget);
}
/// <summary>Verifies a single-node snapshot (just the local node) spawns no children.</summary>
[Fact]
public void Single_node_snapshot_spawns_no_children()
{
var sup = ActorOfAsTestActorRef<PeerProbeSupervisor>(
PeerProbeSupervisor.PropsForTests(Local, _ => NoopProps()));
sup.Tell(Snapshot(State(Local, RedundancyRole.Primary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(0),
duration: PresenceBudget);
}
/// <summary>Verifies a previously-removed peer is respawned when it re-appears, without an
/// "actor name not unique" collision on the sanitized child name.</summary>
[Fact]
public void Re_adding_a_previously_removed_peer_respawns_it()
{
var sup = ActorOfAsTestActorRef<PeerProbeSupervisor>(
PeerProbeSupervisor.PropsForTests(Local, _ => NoopProps()));
sup.Tell(Snapshot(
State(Local, RedundancyRole.Primary),
State(Peer, RedundancyRole.Secondary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(1),
duration: PresenceBudget);
sup.Tell(Snapshot(State(Local, RedundancyRole.Primary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(0),
duration: PresenceBudget);
sup.Tell(Snapshot(
State(Local, RedundancyRole.Primary),
State(Peer, RedundancyRole.Secondary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(1),
duration: PresenceBudget);
}
/// <summary>Locks in the stale-Terminated guard: when an OLD (already-replaced) child's
/// <see cref="Terminated"/> for a peer arrives AFTER a NEW child for the SAME peer was spawned,
/// the fresh child must NOT be evicted (removal is keyed by current-child ref-equality, not by
/// peer key). Without this guard a late stale Terminated would silently drop a live probe.</summary>
[Fact]
public void Stale_terminated_for_old_child_does_not_evict_fresh_peer_child()
{
var spawned = new List<IActorRef>();
var sup = ActorOfAsTestActorRef<PeerProbeSupervisor>(
PeerProbeSupervisor.PropsForTests(Local, _ => RecordingProps(spawned)));
// First add of the peer -> child #0 (the OLD ref).
sup.Tell(Snapshot(
State(Local, RedundancyRole.Primary),
State(Peer, RedundancyRole.Secondary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(1),
duration: PresenceBudget);
AwaitAssert(() => spawned.Count.ShouldBe(1), duration: PresenceBudget);
var oldRef = spawned[0];
// Drop the peer -> child #0 stopped, ChildCount back to 0.
sup.Tell(Snapshot(State(Local, RedundancyRole.Primary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(0),
duration: PresenceBudget);
// Re-add the SAME peer -> a NEW child #1 (the FRESH ref) is spawned.
sup.Tell(Snapshot(
State(Local, RedundancyRole.Primary),
State(Peer, RedundancyRole.Secondary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(1),
duration: PresenceBudget);
AwaitAssert(() => spawned.Count.ShouldBe(2), duration: PresenceBudget);
// Now deliver a STALE Terminated for the OLD ref. The current child for Peer is the fresh
// child #1, so ref-equality finds no match and the supervisor must leave ChildCount at 1.
sup.Tell(new Terminated(oldRef, existenceConfirmed: true, addressTerminated: false));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(1),
duration: PresenceBudget);
}
}