test(runtime): fix the intermittent Runtime.Tests failure — 4 ordering/logic defects + the presence-budget class (#500)
The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test
name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it
was never ONE flaky test: five failures across three distinct tests in that batch, and
five more distinct tests over the verification rounds that followed.
Two hypotheses were tested and DISPROVED by measurement before anything was changed:
- Cluster-formation timeout. Every test in this assembly forms a real single-node Akka
cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up
— 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero
timeouts. A 3.4x margin; not the cause.
- Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout.
Not the cause.
FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout):
1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place
The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and
then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register,
Unregister] interleaving LastSender is the DYING child and the assertion fails. Now
captures the sender of the Register message as it arrives. Fully deterministic; no
timing involved. (Swept the other four LastSender sites: all drain the Unregister
first, so their ordering is fixed. Only this one was unsafe.)
2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath
ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is
still in Connecting, which deliberately fast-fails writes ("driver not connected"),
and the NodeId->driver reverse map has not been pushed. Now retries until accepted.
This does not weaken the assertion — every rejection branch replies WITHOUT reaching
the driver, so Writes.Count.ShouldBe(1) still means exactly what it did.
3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_...
One 500 ms constant served both presence budgets (AwaitAssert) and absence windows
(ExpectNoMsg). Different quantities that happened to share a number, so the presence
budget could not be raised without slowing every absence check. Split into
AssertTimeout (5 s) and Settle (500 ms).
4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks
Waited on `outbox == 0`, which is ALSO true before the first append — so the poll
could win the race against the recorder and return having observed the INITIAL state,
after which the CallCount>=2 check outside the block failed against a recorder that
had not run. Both conditions are now polled together with the retry count as the
discriminator. (The preceding test in the same file documents this exact trap.)
THE PRESENCE-BUDGET CLASS:
After those four, three consecutive 30-run rounds each still failed once — on a
DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s,
OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a
distribution, not a defect, and fixing it one test at a time was the wrong method.
RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s
AwaitAssert/AwaitCondition durations across four files route through it.
Why this is safe rather than sloppy: a presence budget is an upper bound before giving
up, not a wait — these helpers poll and return the instant the condition holds. Raising
one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass;
it only changes how quickly a real breakage reports. Absence windows are the opposite
(the elapsed time IS the assertion) and were deliberately left untouched with their own
short, individually-calibrated literals.
ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its
8 s budget was sized for message passing but actually waits on a REAL Roslyn compile
(ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold
script compiles are the slowest and most variable thing in the assembly, so that class
gets 30 s with the reason recorded.
Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13%
baseline. Full solution builds; all 41 unit-test projects green.
NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use
AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They
are excluded from the budget change on purpose — a bigger number does not fix them, and
rewriting them to settle-then-assert may legitimately turn them red.
This commit is contained in:
Joseph Doherty
@@ -53,7 +53,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
sink.Values[0].Value.ShouldBe(3.14);
|
||||
sink.Values[0].Quality.ShouldBe(OpcUaQuality.Good);
|
||||
sink.Values[1].Quality.ShouldBe(OpcUaQuality.Uncertain);
|
||||
}, duration: TimeSpan.FromMilliseconds(500));
|
||||
}, duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that AlarmStateUpdate routes to sink WriteAlarmCondition with the full snapshot.</summary>
|
||||
@@ -73,7 +73,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
sink.Alarms[0].State.Active.ShouldBeTrue();
|
||||
sink.Alarms[0].State.Acknowledged.ShouldBeFalse();
|
||||
sink.Alarms[0].State.Severity.ShouldBe((ushort)700);
|
||||
}, duration: TimeSpan.FromMilliseconds(500));
|
||||
}, duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>#477 — AlarmQualityUpdate routes to sink.WriteAlarmQuality with the quality + realm.</summary>
|
||||
@@ -93,7 +93,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
q.AlarmNodeId.ShouldBe("Plant/Modbus/dev1/temp_hi");
|
||||
q.Quality.ShouldBe(OpcUaQuality.Bad);
|
||||
q.Realm.ShouldBe(AddressSpaceRealm.Raw);
|
||||
}, duration: TimeSpan.FromMilliseconds(500));
|
||||
}, duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Builds a test <see cref="AlarmConditionSnapshot"/> with sensible defaults so each test
|
||||
@@ -117,7 +117,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
|
||||
actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => sink.RebuildCalls.ShouldBe(1), duration: TimeSpan.FromMilliseconds(500));
|
||||
AwaitAssert(() => sink.RebuildCalls.ShouldBe(1), duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that <see cref="OpcUaPublishActor.MaterialiseDiscoveredNodes"/> forwards to the
|
||||
@@ -146,7 +146,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
sink.Folders.ShouldContain(("EQ-1/Axes", "EQ-1", "Axes"));
|
||||
sink.Variables.ShouldContain(("EQ-1/Axes/X", "EQ-1/Axes", "X", "Double", false));
|
||||
sink.ModelChanges.ShouldContain("EQ-1");
|
||||
}, duration: TimeSpan.FromMilliseconds(500));
|
||||
}, duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that ServiceLevelChanged publishes to IServiceLevelPublisher once per unique level.</summary>
|
||||
@@ -161,7 +161,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
actor.Tell(new OpcUaPublishActor.ServiceLevelChanged(100));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldBe(new byte[] { 240, 100 }),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that the very first computed ServiceLevel is always published even when it is
|
||||
@@ -186,7 +186,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldBe(new byte[] { 0 }),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that RedundancyStateChanged drives local ServiceLevel publish for primary leader.</summary>
|
||||
@@ -210,7 +210,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
actor.Tell(snapshot);
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldBe(new byte[] { 240 }),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that RedundancyStateChanged for secondary publishes 100.</summary>
|
||||
@@ -231,7 +231,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
actor.Tell(snapshot);
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldBe(new byte[] { 100 }),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that the calculator path computes 250 for a healthy primary role-leader
|
||||
@@ -256,7 +256,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)250),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that the calculator path computes 240 for a healthy non-leader secondary
|
||||
@@ -282,7 +282,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)240),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that the calculator path computes 100 when the DB is unreachable
|
||||
@@ -307,7 +307,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)100),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that the calculator path computes 200 for a stale snapshot when the DB is
|
||||
@@ -335,7 +335,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)200),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that a detached local node publishes 0 (the calculator does not model
|
||||
@@ -362,7 +362,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
},
|
||||
CorrelationId.NewId()));
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)250),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
|
||||
// Now detach — expect the guard to drive ServiceLevel down to 0.
|
||||
actor.Tell(new RedundancyStateChanged(
|
||||
@@ -374,7 +374,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)0),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that an actively-observed, recent peer probe of MY endpoint that came back
|
||||
@@ -403,7 +403,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)0),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies branch (3) of <c>OpcUaProbeOk()</c>: a peer's NEGATIVE verdict about this node
|
||||
@@ -447,7 +447,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
|
||||
// Healthy (250), NOT 0 — proves an aged negative verdict does not demote.
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)250),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that with no peer probe result ever received, <c>OpcUaProbeOk()</c> defaults
|
||||
@@ -474,7 +474,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)250),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that a later <c>Ok==true</c> peer probe supersedes an earlier <c>Ok==false</c>
|
||||
@@ -503,7 +503,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)240),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that a peer probe result about a DIFFERENT node is ignored — it does not
|
||||
@@ -531,7 +531,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)250),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies the legacy back-compat seam: with no DB-health probe wired, the handler
|
||||
@@ -553,7 +553,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)240),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that the periodic <c>HealthTick</c> Asks the local <see cref="DbHealthProbeActor"/>
|
||||
@@ -607,7 +607,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)240),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>DB-less, member Up, snapshot fresh, Primary → 250 (basis 240 + 10 Primary bonus).</summary>
|
||||
@@ -629,7 +629,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)250),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>DB-less, snapshot stale (entry.AsOfUtc older than the stale window) → 200. Staleness
|
||||
@@ -654,7 +654,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)200),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>DB-less, Detached → 0. The Detached / missing-entry guard runs ABOVE the DB-less
|
||||
@@ -677,7 +677,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
},
|
||||
CorrelationId.NewId()));
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)250),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
|
||||
actor.Tell(new RedundancyStateChanged(
|
||||
Nodes: new[]
|
||||
@@ -688,7 +688,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)0),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Regression: a DB-BACKED node (probe wired + DbHealthStatus reachable, dbLess default
|
||||
@@ -714,7 +714,7 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase
|
||||
CorrelationId.NewId()));
|
||||
|
||||
AwaitAssert(() => publisher.Levels.ShouldContain((byte)240),
|
||||
duration: TimeSpan.FromMilliseconds(500));
|
||||
duration: PresenceBudget);
|
||||
}
|
||||
|
||||
/// <summary>Stub DB-health probe actor that answers <see cref="DbHealthProbeActor.GetStatus"/>
|
||||
|
||||
Reference in New Issue
Block a user