test(deployment): de-flake unreachable-site reconcile breaker test; assert structurally, not by 2 s wall clock (ScadaBridge#92)
Under a loaded full-project run the serial prepare of 6 instances with a 300 ms reconcile deadline took 2.6 s and tripped the fixed 2 s cap (5/10 full runs failed under CPU load). Deadline raised to 2 s so it cannot expire before the query is sent; the elapsed bound is now derived from the deadline (must beat (n-1) deadlines) while the one-query assertion proves the breaker. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -434,7 +434,14 @@ public class DeploySiteAsyncTests : TestKit
|
|||||||
var commActor = Sys.ActorOf(Props.Create(() =>
|
var commActor = Sys.ActorOf(Props.Create(() =>
|
||||||
new SilentQuerySiteActor(() => Interlocked.Increment(ref queryCount))));
|
new SilentQuerySiteActor(() => Interlocked.Increment(ref queryCount))));
|
||||||
|
|
||||||
var reconcileTimeout = TimeSpan.FromMilliseconds(300);
|
// The deadline must dwarf scheduler/JIT jitter (#92): the reconcile deadline
|
||||||
|
// covers the repo reads and the Ask itself, so a deadline of a few hundred ms
|
||||||
|
// can expire before the query is even sent when the full suite runs in
|
||||||
|
// parallel, and the old fixed 2 s wall-clock cap was blown by ~2.3 s of
|
||||||
|
// starvation on top of 6 x 300 ms. The breaker is proven STRUCTURALLY by the
|
||||||
|
// query count below; the elapsed check is only a coarse "not six deadlines"
|
||||||
|
// bound whose threshold is derived from the deadline, not a magic number.
|
||||||
|
var reconcileTimeout = TimeSpan.FromSeconds(2);
|
||||||
var service = CreateService(commActor, maxParallelism: 3, reconcileTimeout: reconcileTimeout);
|
var service = CreateService(commActor, maxParallelism: 3, reconcileTimeout: reconcileTimeout);
|
||||||
|
|
||||||
var sw = Stopwatch.StartNew();
|
var sw = Stopwatch.StartNew();
|
||||||
@@ -446,8 +453,10 @@ public class DeploySiteAsyncTests : TestKit
|
|||||||
// The circuit-breaker opened after the first unanswered reconcile.
|
// The circuit-breaker opened after the first unanswered reconcile.
|
||||||
Assert.Equal(1, Volatile.Read(ref queryCount));
|
Assert.Equal(1, Volatile.Read(ref queryCount));
|
||||||
|
|
||||||
// Prepare therefore cost ONE reconcile deadline, not six.
|
// Prepare therefore cost ONE reconcile deadline, not six: without the breaker
|
||||||
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(2),
|
// the serial prepare alone would take >= instanceCount x deadline.
|
||||||
|
var withoutBreakerFloor = instanceCount * reconcileTimeout;
|
||||||
|
Assert.True(sw.Elapsed < withoutBreakerFloor - reconcileTimeout,
|
||||||
$"bulk prepare took {sw.ElapsedMilliseconds} ms — the per-batch reconcile bound is not holding");
|
$"bulk prepare took {sw.ElapsedMilliseconds} ms — the per-batch reconcile bound is not holding");
|
||||||
|
|
||||||
// A skipped reconcile is not a failed deploy: every instance still went out.
|
// A skipped reconcile is not a failed deploy: every instance still went out.
|
||||||
|
|||||||
Reference in New Issue
Block a user