test(runtime): fix the intermittent Runtime.Tests failure — 4 ordering/logic defects + the presence-budget class (#500)

The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test
name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it
was never ONE flaky test: five failures across three distinct tests in that batch, and
five more distinct tests over the verification rounds that followed.

Two hypotheses were tested and DISPROVED by measurement before anything was changed:

- Cluster-formation timeout. Every test in this assembly forms a real single-node Akka
  cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up
  — 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero
  timeouts. A 3.4x margin; not the cause.
- Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout.
  Not the cause.

FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout):

1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place
   The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and
   then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register,
   Unregister] interleaving LastSender is the DYING child and the assertion fails. Now
   captures the sender of the Register message as it arrives. Fully deterministic; no
   timing involved. (Swept the other four LastSender sites: all drain the Unregister
   first, so their ordering is fixed. Only this one was unsafe.)

2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath
   ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is
   still in Connecting, which deliberately fast-fails writes ("driver not connected"),
   and the NodeId->driver reverse map has not been pushed. Now retries until accepted.
   This does not weaken the assertion — every rejection branch replies WITHOUT reaching
   the driver, so Writes.Count.ShouldBe(1) still means exactly what it did.

3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_...
   One 500 ms constant served both presence budgets (AwaitAssert) and absence windows
   (ExpectNoMsg). Different quantities that happened to share a number, so the presence
   budget could not be raised without slowing every absence check. Split into
   AssertTimeout (5 s) and Settle (500 ms).

4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks
   Waited on `outbox == 0`, which is ALSO true before the first append — so the poll
   could win the race against the recorder and return having observed the INITIAL state,
   after which the CallCount>=2 check outside the block failed against a recorder that
   had not run. Both conditions are now polled together with the retry count as the
   discriminator. (The preceding test in the same file documents this exact trap.)

THE PRESENCE-BUDGET CLASS:

After those four, three consecutive 30-run rounds each still failed once — on a
DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s,
OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a
distribution, not a defect, and fixing it one test at a time was the wrong method.

RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s
AwaitAssert/AwaitCondition durations across four files route through it.

Why this is safe rather than sloppy: a presence budget is an upper bound before giving
up, not a wait — these helpers poll and return the instant the condition holds. Raising
one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass;
it only changes how quickly a real breakage reports. Absence windows are the opposite
(the elapsed time IS the assertion) and were deliberately left untouched with their own
short, individually-calibrated literals.

ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its
8 s budget was sized for message passing but actually waits on a REAL Roslyn compile
(ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold
script compiles are the slowest and most variable thing in the assembly, so that class
gets 30 s with the reason recorded.

Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13%
baseline. Full solution builds; all 41 unit-test projects green.

NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use
AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They
are excluded from the budget change on purpose — a bigger number does not fix them, and
rewriting them to settle-then-assert may legitimately turn them red.
This commit is contained in:
Joseph Doherty
2026-07-25 20:20:06 -04:00
parent 4dc2ad8084
commit 154171f48c
11 changed files with 199 additions and 79 deletions
@@ -11,6 +11,25 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
/// </summary>
public abstract class RuntimeActorTestBase : TestKit
{
/// <summary>
/// Shared upper bound for a <b>presence</b> wait (<c>AwaitAssert</c> / <c>AwaitCondition</c>).
/// </summary>
/// <remarks>
/// <para><b>A budget, not a delay.</b> These helpers poll and return the instant the condition holds,
/// so the value is how long to wait before giving up. Raising it costs nothing on the happy path and
/// <em>cannot</em> make a genuinely failing assertion pass — it only changes how quickly a real
/// breakage is reported. That is the opposite of an <b>absence</b> window (<c>ExpectNoMsg</c>,
/// settle-then-assert), where the elapsed time IS the assertion and every millisecond is spent on
/// every run. Absence windows keep their own short, individually-calibrated literals and must not be
/// routed through here.</para>
/// <para><b>Why it exists (Gitea #500).</b> This assembly runs 44 Akka test classes roughly 14-way
/// parallel, each with its own <see cref="TestKit"/> ActorSystem. Budgets of 300500 ms sized on an
/// idle machine are comfortable in isolation and marginal under that contention: three consecutive
/// 30-run verification rounds each surfaced a <em>different</em> test failing on one, in three
/// different files. Fixing them one at a time was chasing a distribution rather than a defect.</para>
/// </remarks>
protected static readonly TimeSpan PresenceBudget = TimeSpan.FromSeconds(15);
/// <summary>Gets the Akka test HOCON configuration string for single-node cluster setup.</summary>
protected static string AkkaTestHocon => @"
akka {