fix(site-runtime): post-boot settle sweep so replicated rows start in seconds, not a sweep interval (#135)

When a replicated site pair cold-starts together, the node that wins the
Deployment Manager singleton loads its own (stale) store, and the peer's
rows arrive by LocalDb sync about a second later. The boot load has already
run and the startup reconcile rightly reports no gap (the rows are in the
store by the time it asks central), so only the instance sweep starts those
instances - at its plain 60 s cadence. Reproduced on the primary rig: boot
load without the instance at 05:03:16, sweep start at 05:04:16.

The env2 drill read this as "never started" because the operator
redeployed at +49 s, before the first sweep tick at +60 s.

Fix: when boot-time Instance Actor creation finishes, the singleton runs
one sweep at once, then sweeps every InstanceSweepSettleIntervalSeconds
(default 5) for InstanceSweepSettleWindowSeconds (default 120), then falls
back to InstanceSweepIntervalSeconds. Startup never waits for the peer, so
a site whose peer is down for good still starts alone from its own store.
LocalDb 0.3.7 exposes no "snapshot applied" signal, so the fix rides the
arrival-path-agnostic sweep with all its existing guards.

Tests: PostBootSettleSweepTests (4) + validator theory. The two positive
tests fail with the fix disabled; the window-close test fails when the
settle timer is never cancelled.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 00:10:06 -05:00
parent c3c1f6e870
commit d7fa31f0b8
8 changed files with 404 additions and 2 deletions
@@ -23,6 +23,35 @@ public class SiteRuntimeOptionsValidatorTests
Assert.True(result.Succeeded, result.FailureMessage);
}
/// <summary>
/// ScadaBridge#135: the post-boot settle sweep cadence and window feed Akka timers, so a
/// negative value must be rejected with a key-naming message; 0 (disable) is valid.
/// </summary>
/// <param name="interval">InstanceSweepSettleIntervalSeconds.</param>
/// <param name="window">InstanceSweepSettleWindowSeconds.</param>
/// <param name="expectedKey">The key the failure must name, or null when valid.</param>
[Theory]
[InlineData(-1, 120, "InstanceSweepSettleIntervalSeconds")]
[InlineData(5, -1, "InstanceSweepSettleWindowSeconds")]
[InlineData(0, 0, null)]
public void InstanceSweepSettleOptions_RejectNegativeValues(int interval, int window, string? expectedKey)
{
var result = Validate(new SiteRuntimeOptions
{
InstanceSweepSettleIntervalSeconds = interval,
InstanceSweepSettleWindowSeconds = window,
});
if (expectedKey == null)
{
Assert.True(result.Succeeded, result.FailureMessage);
return;
}
Assert.True(result.Failed);
Assert.Contains(expectedKey, result.FailureMessage);
}
/// <summary>Verifies that zero startup batch size is rejected.</summary>
[Fact]
public void ZeroStartupBatchSize_IsRejected()