test(runtime): fix the intermittent Runtime.Tests failure — 4 ordering/logic defects + the presence-budget class (#500)

The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it was never ONE flaky test: five failures across three distinct tests in that batch, and five more distinct tests over the verification rounds that followed. Two hypotheses were tested and DISPROVED by measurement before anything was changed: - Cluster-formation timeout. Every test in this assembly forms a real single-node Akka cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up — 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero timeouts. A 3.4x margin; not the cause. - Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout. Not the cause. FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout): 1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register, Unregister] interleaving LastSender is the DYING child and the assertion fails. Now captures the sender of the Register message as it arrives. Fully deterministic; no timing involved. (Swept the other four LastSender sites: all drain the Unregister first, so their ordering is fixed. Only this one was unsafe.) 2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is still in Connecting, which deliberately fast-fails writes ("driver not connected"), and the NodeId->driver reverse map has not been pushed. Now retries until accepted. This does not weaken the assertion — every rejection branch replies WITHOUT reaching the driver, so Writes.Count.ShouldBe(1) still means exactly what it did. 3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_... One 500 ms constant served both presence budgets (AwaitAssert) and absence windows (ExpectNoMsg). Different quantities that happened to share a number, so the presence budget could not be raised without slowing every absence check. Split into AssertTimeout (5 s) and Settle (500 ms). 4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks Waited on `outbox == 0`, which is ALSO true before the first append — so the poll could win the race against the recorder and return having observed the INITIAL state, after which the CallCount>=2 check outside the block failed against a recorder that had not run. Both conditions are now polled together with the retry count as the discriminator. (The preceding test in the same file documents this exact trap.) THE PRESENCE-BUDGET CLASS: After those four, three consecutive 30-run rounds each still failed once — on a DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s, OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a distribution, not a defect, and fixing it one test at a time was the wrong method. RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s AwaitAssert/AwaitCondition durations across four files route through it. Why this is safe rather than sloppy: a presence budget is an upper bound before giving up, not a wait — these helpers poll and return the instant the condition holds. Raising one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass; it only changes how quickly a real breakage reports. Absence windows are the opposite (the elapsed time IS the assertion) and were deliberately left untouched with their own short, individually-calibrated literals. ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its 8 s budget was sized for message passing but actually waits on a REAL Roslyn compile (ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold script compiles are the slowest and most variable thing in the assembly, so that class gets 30 s with the reason recorded. Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13% baseline. Full solution builds; all 41 unit-test projects green. NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They are excluded from the budget change on purpose — a bigger number does not fix them, and rewriting them to settle-then-assert may legitimately turn them red.
2026-07-25 20:20:06 -04:00
parent 4dc2ad8084
commit 154171f48c
11 changed files with 199 additions and 79 deletions
@@ -55,11 +55,9 @@ public sealed class DriverHostActorWriteRoutingTests : RuntimeActorTestBase

        var actor = SpawnHostAndApply(db, deploymentId, recorder);

-        var asker = CreateTestProbe();
        // The node manager passes the FULL ns-qualified id; the host must normalise + resolve it.
-        actor.Tell(new DriverHostActor.RouteNodeWrite($"ns=3;s={UnsNodeId}", 123.0, AddressSpaceRealm.Uns), asker.Ref);
+        RouteWriteUntilAccepted(actor, $"ns=3;s={UnsNodeId}", 123.0, AddressSpaceRealm.Uns);

-        asker.ExpectMsg<DriverHostActor.NodeWriteResult>(Timeout).Success.ShouldBeTrue();
        AwaitAssert(() =>
        {
            recorder.Writes.Count.ShouldBe(1);
@@ -79,10 +77,8 @@ public sealed class DriverHostActorWriteRoutingTests : RuntimeActorTestBase

        var actor = SpawnHostAndApply(db, deploymentId, recorder);

-        var asker = CreateTestProbe();
-        actor.Tell(new DriverHostActor.RouteNodeWrite($"ns=2;s={RawPath}", 456.0, AddressSpaceRealm.Raw), asker.Ref);
+        RouteWriteUntilAccepted(actor, $"ns=2;s={RawPath}", 456.0, AddressSpaceRealm.Raw);

-        asker.ExpectMsg<DriverHostActor.NodeWriteResult>(Timeout).Success.ShouldBeTrue();
        AwaitAssert(() =>
        {
            recorder.Writes.Count.ShouldBe(1);
@@ -91,6 +87,48 @@ public sealed class DriverHostActorWriteRoutingTests : RuntimeActorTestBase
        }, duration: Timeout);
    }

+    /// <summary>
+    ///     Routes a write, retrying until the host accepts it, and fails with the host's own rejection
+    ///     <c>Reason</c> if it never does.
+    /// </summary>
+    /// <remarks>
+    ///     <para><b>Why a retry rather than a single Tell.</b> <c>ApplyAck</c> — which
+    ///     <see cref="SpawnHostAndApply"/> waits for — marks the end of the <em>apply</em>, not the point at
+    ///     which a write can succeed. Several things still have to happen after it: the spawned child has to
+    ///     finish <c>InitializeAsync</c> and leave <c>Connecting</c> (that state deliberately fast-fails
+    ///     writes with "driver not connected"), and the host has to push the NodeId→driver reverse map that
+    ///     resolves the write. Telling the write immediately therefore races the setup, which is why this
+    ///     assertion failed roughly 1 run in 30 of the fully parallel assembly — and never once in 60
+    ///     consecutive runs of this class alone.</para>
+    ///     <para><b>Why retrying does not weaken the assertion.</b> Every rejection branch — the Primary
+    ///     gate, an unresolved reverse map, "driver not running", and the child's own pre-Connected
+    ///     fast-fail — replies <em>without</em> reaching the driver. A rejected attempt records nothing, so
+    ///     the caller's <c>Writes.Count.ShouldBe(1)</c> still means exactly what it did before: the accepted
+    ///     write reached the driver exactly once.</para>
+    ///     <para>The failure message carries the last <c>Reason</c> so a genuine regression is diagnosable
+    ///     rather than surfacing as a bare "expected True".</para>
+    /// </remarks>
+    /// <param name="actor">The driver-host actor.</param>
+    /// <param name="nodeId">The ns-qualified NodeId string, as the node manager passes it.</param>
+    /// <param name="value">The value to write.</param>
+    /// <param name="realm">The address-space realm the NodeId belongs to.</param>
+    private void RouteWriteUntilAccepted(
+        IActorRef actor, string nodeId, object value, AddressSpaceRealm realm)
+    {
+        var lastReason = "(no reply)";
+        AwaitAssert(
+            () =>
+            {
+                var asker = CreateTestProbe();
+                actor.Tell(new DriverHostActor.RouteNodeWrite(nodeId, value, realm), asker.Ref);
+                var result = asker.ExpectMsg<DriverHostActor.NodeWriteResult>(TimeSpan.FromSeconds(1));
+                lastReason = result.Reason ?? "(none)";
+                result.Success.ShouldBeTrue($"host kept rejecting the write; last reason: {lastReason}");
+            },
+            duration: Timeout,
+            interval: TimeSpan.FromMilliseconds(50));
+    }
+
    /// <summary>On a SECONDARY, RouteNodeWrite (via the UNS NodeId) replies "not primary" and the driver
    /// receives NO write — the primary gate fires before the reverse-map lookup.</summary>
    [Fact]