test(runtime): fix the intermittent Runtime.Tests failure — 4 ordering/logic defects + the presence-budget class (#500)

The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test
name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it
was never ONE flaky test: five failures across three distinct tests in that batch, and
five more distinct tests over the verification rounds that followed.

Two hypotheses were tested and DISPROVED by measurement before anything was changed:

- Cluster-formation timeout. Every test in this assembly forms a real single-node Akka
  cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up
  — 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero
  timeouts. A 3.4x margin; not the cause.
- Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout.
  Not the cause.

FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout):

1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place
   The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and
   then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register,
   Unregister] interleaving LastSender is the DYING child and the assertion fails. Now
   captures the sender of the Register message as it arrives. Fully deterministic; no
   timing involved. (Swept the other four LastSender sites: all drain the Unregister
   first, so their ordering is fixed. Only this one was unsafe.)

2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath
   ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is
   still in Connecting, which deliberately fast-fails writes ("driver not connected"),
   and the NodeId->driver reverse map has not been pushed. Now retries until accepted.
   This does not weaken the assertion — every rejection branch replies WITHOUT reaching
   the driver, so Writes.Count.ShouldBe(1) still means exactly what it did.

3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_...
   One 500 ms constant served both presence budgets (AwaitAssert) and absence windows
   (ExpectNoMsg). Different quantities that happened to share a number, so the presence
   budget could not be raised without slowing every absence check. Split into
   AssertTimeout (5 s) and Settle (500 ms).

4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks
   Waited on `outbox == 0`, which is ALSO true before the first append — so the poll
   could win the race against the recorder and return having observed the INITIAL state,
   after which the CallCount>=2 check outside the block failed against a recorder that
   had not run. Both conditions are now polled together with the retry count as the
   discriminator. (The preceding test in the same file documents this exact trap.)

THE PRESENCE-BUDGET CLASS:

After those four, three consecutive 30-run rounds each still failed once — on a
DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s,
OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a
distribution, not a defect, and fixing it one test at a time was the wrong method.

RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s
AwaitAssert/AwaitCondition durations across four files route through it.

Why this is safe rather than sloppy: a presence budget is an upper bound before giving
up, not a wait — these helpers poll and return the instant the condition holds. Raising
one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass;
it only changes how quickly a real breakage reports. Absence windows are the opposite
(the elapsed time IS the assertion) and were deliberately left untouched with their own
short, individually-calibrated literals.

ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its
8 s budget was sized for message passing but actually waits on a REAL Roslyn compile
(ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold
script compiles are the slowest and most variable thing in the assembly, so that class
gets 30 s with the reason recorded.

Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13%
baseline. Full solution builds; all 41 unit-test projects green.

NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use
AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They
are excluded from the budget change on purpose — a bigger number does not fix them, and
rewriting them to settle-then-assert may legitimately turn them red.

tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ContinuousHistorizationRecorderTests.cs

@@ -295,10 +295,24 @@ public sealed class ContinuousHistorizationRecorderTests : TestKit
 
         // The first drain returns false (entry retained); after the backoff the retry drain succeeds
         // and acks, truncating the outbox to 0.
-        await AwaitAssertAsync(async () =>
-            Assert.Equal(0, await outbox.CountAsync(default)), TimeSpan.FromSeconds(5));
-
-        Assert.True(writer.CallCount >= 2, "the writer must have been called at least twice (a retry happened)");
+        //
+        // BOTH conditions are polled together, and the retry count is the load-bearing one. An empty
+        // outbox is NOT a distinguishing observation: it is equally true BEFORE the append lands, so
+        // waiting on it alone can be satisfied by the initial state and return before the recorder has
+        // done anything at all. That is exactly what happened intermittently under a fully parallel
+        // assembly run — the poll won the race against the first append, the wait returned immediately,
+        // and the CallCount check that used to sit outside the block then failed against a recorder that
+        // had not yet run. (The same trap is called out on the preceding test, which guards against it by
+        // pairing its count with a writer observation.)
+        await AwaitAssertAsync(
+            async () =>
+            {
+                Assert.True(
+                    writer.CallCount >= 2,
+                    "the writer must have been called at least twice (a retry happened)");
+                Assert.Equal(0, await outbox.CountAsync(default));
+            },
+            TimeSpan.FromSeconds(5));
     }
 
     [Fact]

tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/HistorianAdapterActorTests.cs

@@ -23,9 +23,27 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
     /// <summary>The local node id the gating tests construct the adapter with.</summary>
     private static readonly NodeId LocalNode = new("node-A");
 
-    /// <summary>A short window we allow the fire-and-forget enqueue to land within.</summary>
+    /// <summary>
+    ///     The window an <b>absence</b> assertion waits before concluding nothing arrived
+    ///     (<c>ExpectNoMsg</c>). Its length is a calibration decision — long enough that a message which
+    ///     was going to arrive would have — so it is deliberately NOT generous.
+    /// </summary>
     private static readonly TimeSpan Settle = TimeSpan.FromMilliseconds(500);
 
+    /// <summary>
+    ///     The budget a <b>presence</b> assertion may take to become true (<c>AwaitAssert</c>).
+    /// </summary>
+    /// <remarks>
+    ///     Separate from <see cref="Settle"/> on purpose, though they once shared its 500 ms. The two are
+    ///     different quantities that merely had the same number: a presence budget is an upper bound before
+    ///     giving up, and <c>AwaitAssert</c> returns the instant the condition holds, so a generous value
+    ///     costs nothing in the passing case and can never make a genuinely failing assertion pass. An
+    ///     absence window is the opposite — every millisecond is spent on every run. Conflating them meant
+    ///     the enqueue assertions could only be given more headroom by slowing every <c>ExpectNoMsg</c> in
+    ///     the class, so they kept a 500 ms budget that a fully parallel assembly run occasionally missed.
+    /// </remarks>
+    private static readonly TimeSpan AssertTimeout = TimeSpan.FromSeconds(5);
+
     /// <summary>Thread-safe fake sink that records every <see cref="EnqueueAsync"/> call.</summary>
     private sealed class RecordingSink : IAlarmHistorianSink
     {
@@ -101,7 +119,7 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
 
         actor.Tell(SampleEvent());
 
-        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
+        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
     }
 
     /// <summary>Secondary suppression (T2): when the cached local role is Secondary, the adapter MUST
@@ -145,7 +163,7 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
         TellRedundancyRole(actor, RedundancyRole.Primary);
         actor.Tell(SampleEvent());
 
-        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
+        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
     }
 
     /// <summary>Absent-node default-historize (T5): a snapshot that mentions only a DIFFERENT node
@@ -174,7 +192,7 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
         actor.Tell(SampleEvent());
 
         // Local role is still unknown ⇒ default-historize path: sink must record exactly one enqueue.
-        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
+        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
     }
 
     /// <summary>Builds an <see cref="AlarmTransitionEvent"/> (the shape published on the <c>alerts</c>
@@ -222,7 +240,7 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
                 e.Severity.ShouldBe(AlarmSeverity.High);
                 e.Comment.ShouldBe("note");
             },
-            Settle);
+            AssertTimeout);
     }
 
     /// <summary>Secondary suppression for alerts (T7): a Secondary node must NOT historize a transition
@@ -252,7 +270,7 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
         TellRedundancyRole(actor, RedundancyRole.Primary);
         actor.Tell(SampleTransition());
 
-        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
+        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
     }
 
     /// <summary>Per-alarm opt-out (T8b): a Primary node must NOT historize a transition whose
@@ -286,7 +304,7 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
         TellRedundancyRole(actor, RedundancyRole.Primary);
         actor.Tell(SampleTransition(historizeToAveva: null));
 
-        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), Settle);
+        AwaitAssert(() => sink.EnqueueCount.ShouldBe(1), AssertTimeout);
     }
 
     /// <summary>Severity buckets (T9): the OPC UA 1–1000 numeric severity on the transition maps onto
@@ -308,7 +326,7 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
 
         AwaitAssert(
             () => sink.Events.ShouldHaveSingleItem().Severity.ShouldBe(expected),
-            Settle);
+            AssertTimeout);
     }
 
     /// <summary>Rolling-restart null default (T10): an old-format transition deserialized by Akka's JSON
@@ -326,6 +344,6 @@ public sealed class HistorianAdapterActorTests : RuntimeActorTestBase
 
         AwaitAssert(
             () => sink.Events.ShouldHaveSingleItem().AlarmTypeName.ShouldBe("AlarmCondition"),
-            Settle);
+            AssertTimeout);
     }
 }