fix(site-runtime): post-boot settle sweep so replicated rows start in seconds, not a sweep interval (#135)

When a replicated site pair cold-starts together, the node that wins the
Deployment Manager singleton loads its own (stale) store, and the peer's
rows arrive by LocalDb sync about a second later. The boot load has already
run and the startup reconcile rightly reports no gap (the rows are in the
store by the time it asks central), so only the instance sweep starts those
instances - at its plain 60 s cadence. Reproduced on the primary rig: boot
load without the instance at 05:03:16, sweep start at 05:04:16.

The env2 drill read this as "never started" because the operator
redeployed at +49 s, before the first sweep tick at +60 s.

Fix: when boot-time Instance Actor creation finishes, the singleton runs
one sweep at once, then sweeps every InstanceSweepSettleIntervalSeconds
(default 5) for InstanceSweepSettleWindowSeconds (default 120), then falls
back to InstanceSweepIntervalSeconds. Startup never waits for the peer, so
a site whose peer is down for good still starts alone from its own store.
LocalDb 0.3.7 exposes no "snapshot applied" signal, so the fix rides the
arrival-path-agnostic sweep with all its existing guards.

Tests: PostBootSettleSweepTests (4) + validator theory. The two positive
tests fail with the fix disabled; the window-close test fails when the
settle timer is never cancelled.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 00:10:06 -05:00
parent c3c1f6e870
commit d7fa31f0b8
8 changed files with 404 additions and 2 deletions
+4
View File
@@ -36,6 +36,8 @@ The staggered startup above reads storage exactly once. Anything that writes a `
`DeploymentManagerActor` therefore runs a periodic sweep (`InstanceSweepIntervalSeconds`, default 60; `0` disables) that re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config either starts a missing `InstanceActor` or — when the running one's revision no longer matches the stored one — re-applies the stored config **through the ordinary redeploy path**, so nothing about redeploy semantics is re-implemented. It defers to any in-flight deploy/warm/termination, never starts a disabled instance, stays inert until boot-time creation has finished, makes one repair attempt per (instance, revision) pair so it can never become a restart loop, and is completely silent when there is nothing to repair. Repairs emit Warning `deployment` site events naming the trigger (and, for a stale-revision restart, both revisions). Design note: `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
**Post-boot settle window (ScadaBridge#135).** On a replicated pair that cold-starts together, the node that wins the singleton loads its own store and the peer's rows arrive by LocalDb sync about a second later; the startup reconcile then rightly finds no gap, so only the sweep starts those instances — at the plain cadence, up to 60 s after a healthy-looking boot (reproduced on the rig 2026-11-13). So the moment boot-time creation finishes the singleton runs one sweep at once, then sweeps every `InstanceSweepSettleIntervalSeconds` (default 5) for `InstanceSweepSettleWindowSeconds` (default 120) before returning to the plain interval. Startup never waits for the peer — a node whose peer is down for good starts from its own store immediately, exactly as before.
Register row **85** was the same gap one layer down: the sweep started the instance, but the instance came up with **no DCL connection actors**, so every tag subscribe was answered `Unknown connection: <name>`. The connection push (`EnsureDclConnections`) was the *caller's* job and two of the four Instance-Actor start paths omitted it; it now lives inside `CreateInstanceActor`. See *Deployment flow* below and §7 of the design note.
### Actor supervision and lifetimes
@@ -282,6 +284,8 @@ All options live in the `ScadaBridge:SiteRuntime` section, bound to `SiteRuntime
| `MirroredAlarmCapPerSource` | `1000` | Maximum mirrored conditions per `NativeAlarmActor` source binding before oldest is dropped and logged |
| `NativeAlarmRetryIntervalMs` | `5000` | Milliseconds before retrying a failed native alarm subscription |
| `InstanceSweepIntervalSeconds` | `60` | Interval of the singleton's instance reconcile sweep (register rows 78 + 72); `0` disables it — see Instance reconcile sweep |
| `InstanceSweepSettleIntervalSeconds` | `5` | Fast sweep cadence for the post-boot settle window (#135); `0` disables the settle phase. Only used when faster than `InstanceSweepIntervalSeconds` |
| `InstanceSweepSettleWindowSeconds` | `120` | Length of the post-boot settle window, from the end of boot-time Instance Actor creation (#135); `0` disables it |
The SQLite connection string is passed directly to `AddSiteRuntime(connectionString)` in the host composition root and is not part of `SiteRuntimeOptions`.
@@ -165,6 +165,8 @@ produces two operator-visible `deployment` site events naming both revisions.
`EnsureInstanceStartedCommand` path is unaffected either way. No config change is required on any
node to get the fix — the default is live.
**Post-boot settle window (ScadaBridge#135, 2026-11-13).** `InstanceSweepSettleIntervalSeconds` (default **5**) and `InstanceSweepSettleWindowSeconds` (default **120**), both validated `>= 0`. When boot-time creation finishes the singleton runs one sweep at once, then sweeps at the settle cadence for the window, then returns to the plain interval. It exists because a replicated pair that cold-starts together hits row 78 on the active node whenever the two stores differ, and at the plain 60 s cadence the replicated instances sat stopped for up to one full interval. `0` in either option disables the settle phase; with the sweep disabled neither does anything.
## 5. Observability
| Condition | Log | Site event |
@@ -104,6 +104,8 @@ flowchart TD
>
> Both are closed by one **periodic sweep owned by the Deployment Manager singleton** (`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`, default 60, validated `>= 0`; `0` disables it). Each tick re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config: starts the Instance Actor if none is running (row 78), or — if one is running at a revision the store has since replaced — **re-applies the stored config through the exact per-instance apply/restart path a central redeploy uses** (row 72). A sweep is chosen over a LocalDb change hook deliberately: a hook would cover only the replication arrival path, only on replicated sites, and `ILocalDb` exposes no change-notification API at all, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot snapshot resync, a hand-restored file, any future path). See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
>
> **Post-boot settle window (ScadaBridge#135).** A replicated pair that cold-starts together hits row 78 on every boot where the stores differ (a first replication enable on diverged stores, or a node that was down while its peer took deploys): the node that wins the singleton loads its own store, and the peer's rows land about a second later. At the plain cadence those instances stayed stopped for up to one full sweep interval. So the moment boot-time creation finishes, the singleton runs one sweep at once, then sweeps every `ScadaBridge:SiteRuntime:InstanceSweepSettleIntervalSeconds` (default 5) for `InstanceSweepSettleWindowSeconds` (default 120), then falls back to the plain interval. Startup never waits for the peer, so a site still starts alone from its own store when its peer is permanently down. Both are validated `>= 0`, `0` disables the settle phase, and neither does anything when the sweep itself is disabled.
>
> **The sweep reads metadata, not configs (review 2026-10-21 01/SR-P1).** Each tick compares only name / enabled / revision, but it used to read every row's `config_json` (~70 KB worst case) — on a 500-instance site ~35 MB of large-object-heap strings a minute, forever, on a perfectly healthy site. The tick now reads the `GetAllDeployedConfigMetadataAsync` projection (no `config_json`), decides its repairs on the actor thread exactly as before, and only then fetches the full row — one single-row read per instance it is about to start or restart — off-thread. When those rows land every decision is **re-validated**: a fetch this node has since mutated past is dropped whole by the same store-write generation guard, a row that vanished, was disabled or now carries a different revision than the one claimed is skipped, and the deploy-pipeline / running-actor guards are re-checked; a repair that does not act gives back its once-per-revision claim so the next tick re-evaluates it. Ticks stay suppressed until the fetch lands. The boot load keeps the full read — it runs once and needs every blob.
> The **row-72 decision is to converge, not merely warn**: the stored row *is* what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing `HandleDeploy` means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy.
@@ -271,6 +271,19 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private readonly TimeSpan _instanceSweepInterval;
/// <summary>
/// Fast sweep cadence for the post-boot settle window (ScadaBridge#135);
/// <see cref="TimeSpan.Zero"/> when the settle phase is off. See
/// <see cref="SiteRuntimeOptions.InstanceSweepSettleIntervalSeconds"/>.
/// </summary>
private readonly TimeSpan _instanceSweepSettleInterval;
/// <summary>
/// Length of the post-boot settle window (ScadaBridge#135). See
/// <see cref="SiteRuntimeOptions.InstanceSweepSettleWindowSeconds"/>.
/// </summary>
private readonly TimeSpan _instanceSweepSettleWindow;
/// <summary>
/// True once boot-time Instance Actor creation has finished (every staggered batch created,
/// or there were no enabled configs to create). The sweep is inert until then: a tick landing
@@ -383,6 +396,12 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
_instanceSweepInterval = options.InstanceSweepIntervalSeconds > 0
? TimeSpan.FromSeconds(options.InstanceSweepIntervalSeconds)
: TimeSpan.Zero;
_instanceSweepSettleInterval = options.InstanceSweepSettleIntervalSeconds > 0
? TimeSpan.FromSeconds(options.InstanceSweepSettleIntervalSeconds)
: TimeSpan.Zero;
_instanceSweepSettleWindow = options.InstanceSweepSettleWindowSeconds > 0
? TimeSpan.FromSeconds(options.InstanceSweepSettleWindowSeconds)
: TimeSpan.Zero;
_configLoader = configLoader ?? (() => _storage.GetAllDeployedConfigsAsync());
_deployedConfigLoader = deployedConfigLoader ?? (name => _storage.GetDeployedConfigAsync(name));
_configMetadataLoader = configMetadataLoader ?? (() => _storage.GetAllDeployedConfigMetadataAsync());
@@ -430,6 +449,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// an enabled config with no running actor (row 78), and a running actor whose revision
// no longer matches the stored one (row 72).
Receive<InstanceSweepTick>(_ => HandleInstanceSweepTick());
Receive<InstanceSweepSettleEnded>(_ => Timers.Cancel(InstanceSweepSettleTimerKey));
Receive<InstanceSweepLoaded>(HandleInstanceSweepLoaded);
// 01/SR-P1: the full rows of the instances a sweep decided to start/restart.
Receive<InstanceSweepRepairsLoaded>(HandleInstanceSweepRepairsLoaded);
@@ -842,7 +862,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// reconcile sweep may run. This is exactly the live case: the node booted
// with an empty deployed_configurations table and CDC replication seeds it moments
// later, with no boot load left to notice.
_startupInstanceCreationComplete = true;
MarkStartupInstanceCreationComplete();
return;
}
@@ -998,7 +1018,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
"All {Count} Instance Actors created", state.Configs.Count);
// Boot-time creation is done — the instance reconcile sweep may now run without
// racing the staggered batches it would otherwise mistake for a divergence.
_startupInstanceCreationComplete = true;
MarkStartupInstanceCreationComplete();
}
}
@@ -2103,6 +2123,58 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
$"Instance {instanceName} started by {reason} self-heal", details);
}
/// <summary>Timer key of the fast post-boot settle sweep (ScadaBridge#135).</summary>
private const string InstanceSweepSettleTimerKey = "instance-sweep-settle";
/// <summary>
/// Ends boot-time Instance Actor creation and opens the post-boot settle window
/// (ScadaBridge#135).
///
/// <para>
/// <b>The race this closes.</b> On a replicated pair that cold-starts together, the node that
/// wins the singleton loads its OWN store, and the peer's rows arrive by LocalDb sync about a
/// second later — a first enable on diverged stores, or a node that was down while its peer
/// took deploys. The boot load has already run, and the startup reconcile reports no gap
/// because the rows ARE in the store by the time it asks central. Only the sweep starts those
/// instances, and at its plain cadence (60 s by default) the site collected nothing for up to
/// a full interval after a healthy-looking boot. LocalDb exposes no "snapshot applied" signal,
/// so the fix rides the arrival-path-agnostic sweep: one sweep NOW (rows that landed while the
/// boot batches were being created), then a fast cadence for a bounded window, then the plain
/// interval.
/// </para>
///
/// <para>
/// <b>Startup never waits.</b> The node has already started everything its own store holds, so
/// a peer that is down for good costs only a few cheap metadata reads. Every guard the sweep
/// already carries (deploy-pipeline deferral, per-revision claims, the store-generation check)
/// applies unchanged, and a sweep that finds nothing stays silent.
/// </para>
/// </summary>
private void MarkStartupInstanceCreationComplete()
{
if (_startupInstanceCreationComplete)
return;
_startupInstanceCreationComplete = true;
if (_instanceSweepInterval <= TimeSpan.Zero)
return;
Self.Tell(InstanceSweepTick.Instance);
if (_instanceSweepSettleInterval > TimeSpan.Zero &&
_instanceSweepSettleWindow > TimeSpan.Zero &&
_instanceSweepSettleInterval < _instanceSweepInterval)
{
Timers.StartPeriodicTimer(
InstanceSweepSettleTimerKey, InstanceSweepTick.Instance,
_instanceSweepSettleInterval, _instanceSweepSettleInterval);
Timers.StartSingleTimer(
"instance-sweep-settle-end", InstanceSweepSettleEnded.Instance,
_instanceSweepSettleWindow);
}
}
/// <summary>Provenance string carried on every repair the instance reconcile sweep makes.</summary>
private const string SweepReason = "deployment-manager sweep";
@@ -4304,6 +4376,17 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
private InstanceSweepTick() { }
}
/// <summary>
/// Closes the post-boot settle window (ScadaBridge#135): the fast settle sweep timer is
/// cancelled and the sweep continues at <see cref="SiteRuntimeOptions.InstanceSweepIntervalSeconds"/>.
/// </summary>
internal sealed record InstanceSweepSettleEnded
{
/// <summary>The singleton instance — the message carries no state.</summary>
public static readonly InstanceSweepSettleEnded Instance = new();
private InstanceSweepSettleEnded() { }
}
/// <summary>
/// Carries one sweep's off-thread read of the durable deployed-config rows back onto the
/// actor thread, where every comparison and repair happens.
@@ -242,6 +242,36 @@ public class SiteRuntimeOptions
/// </summary>
public int InstanceSweepIntervalSeconds { get; set; } = 60;
/// <summary>
/// Fast cadence (seconds) of the instance reconcile sweep during the <b>post-boot settle
/// window</b> (ScadaBridge#135). When the Deployment Manager singleton finishes its boot-time
/// Instance Actor creation it runs one sweep at once, then sweeps every
/// <see cref="InstanceSweepSettleIntervalSeconds"/> for
/// <see cref="InstanceSweepSettleWindowSeconds"/> before falling back to
/// <see cref="InstanceSweepIntervalSeconds"/>.
/// <para>
/// Why: on a replicated pair the node that boots active loads its OWN store, and the peer's
/// rows arrive by LocalDb sync a moment later (a first enable on diverged stores, or a node
/// that was down while its peer took deploys). Those rows are invisible to the boot load and to
/// the startup reconcile (central rightly sees no gap), so only the sweep starts them — and at
/// the plain 60 s cadence that left the site collecting nothing for up to a full minute after a
/// cold start. Startup is never delayed: the node starts from its own store at once and a peer
/// that never answers costs nothing but a few cheap metadata reads.
/// </para>
/// <c>0</c> disables the settle phase (the immediate post-boot sweep still runs). It only
/// applies when the sweep is enabled and is faster than <see cref="InstanceSweepIntervalSeconds"/>.
/// Default: 5.
/// </summary>
public int InstanceSweepSettleIntervalSeconds { get; set; } = 5;
/// <summary>
/// Length (seconds) of the post-boot settle window, measured from the end of boot-time
/// Instance Actor creation, during which the sweep runs at
/// <see cref="InstanceSweepSettleIntervalSeconds"/> (ScadaBridge#135). <c>0</c> disables
/// the settle phase. Default: 120.
/// </summary>
public int InstanceSweepSettleWindowSeconds { get; set; } = 120;
/// <summary>
/// Flow Debugger trace arm window (design 2026-09-05 §5.3). When a TRACED script write
/// targets a data-sourced attribute, the Instance Actor remembers
@@ -113,6 +113,18 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase<SiteRunti
"0 disables the sweep, which reopens the replication-seeded-config and stale-revision " +
"self-heals (deferred register rows 78 and 72).");
builder.RequireThat(options.InstanceSweepSettleIntervalSeconds >= 0,
$"ScadaBridge:SiteRuntime:InstanceSweepSettleIntervalSeconds must be >= 0 " +
$"(was {options.InstanceSweepSettleIntervalSeconds}); it is the instance sweep's fast " +
"post-boot cadence and a negative value throws inside the Akka timer. 0 disables the " +
"settle phase (ScadaBridge#135).");
builder.RequireThat(options.InstanceSweepSettleWindowSeconds >= 0,
$"ScadaBridge:SiteRuntime:InstanceSweepSettleWindowSeconds must be >= 0 " +
$"(was {options.InstanceSweepSettleWindowSeconds}); it is how long the instance sweep " +
"runs at its fast post-boot cadence and a negative value throws inside the Akka timer. " +
"0 disables the settle phase (ScadaBridge#135).");
builder.RequireThat(options.TraceArmWindowSeconds > 0,
$"ScadaBridge:SiteRuntime:TraceArmWindowSeconds must be greater than 0 " +
$"(was {options.TraceArmWindowSeconds}); it is how long the Instance Actor waits for the " +
@@ -0,0 +1,240 @@
using System.Text.Json;
using Akka.Actor;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
/// <summary>
/// Regression tests for <b>ScadaBridge#135</b>: a replicated site pair cold-starts together, the
/// node that wins the Deployment Manager singleton loads its OWN (stale) store, and the peer's
/// rows arrive by LocalDb sync about a second later. Those rows are invisible to the boot load
/// (already run) and to the startup reconcile (central sees no gap — the rows ARE in the store by
/// then), so only the instance sweep starts them. At its plain 60 s cadence the site ran none of
/// the replicated instances for a full interval after a healthy-looking boot — reproduced on the
/// primary rig 2026-11-13 (boot load 05:03:16 without the instance, sweep start at 05:04:16).
///
/// <para>
/// The fix opens a post-boot settle window: one sweep the moment boot-time creation finishes,
/// then a fast cadence for a bounded window, then the plain interval. Startup itself never waits
/// for the peer. Every test here pins the plain interval at 60 s or more, so a pass can only come
/// from the settle path — a plain-cadence sweep cannot land inside the assertion budgets.
/// </para>
/// </summary>
public class PostBootSettleSweepTests : TestKit, IDisposable
{
private readonly SiteStorageService _storage;
private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly TestLocalDb _localDb;
/// <summary>The plain sweep interval: far longer than any assertion budget below.</summary>
private const int PlainSweepSeconds = 60;
/// <summary>Well inside one plain interval, comfortably more than a few settle ticks.</summary>
private static readonly TimeSpan HealBudget = TimeSpan.FromSeconds(8);
/// <summary>Creates a temporary LocalDb-backed site store and the compilation services for each test.</summary>
public PostBootSettleSweepTests()
{
_localDb = TestLocalDb.CreateTemp("post-boot-settle-sweep-test");
_storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService(
NullLogger<ScriptCompilationService>.Instance);
_sharedScriptLibrary = new SharedScriptLibrary(
_compilationService, NullLogger<SharedScriptLibrary>.Instance);
}
/// <inheritdoc />
void IDisposable.Dispose()
{
// Shut the actors down BEFORE the ILocalDb they write through is disposed.
Shutdown();
var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
/// <summary>
/// Starts a Deployment Manager and blocks until its boot-time config load has PROVABLY read
/// storage (its first instance-count report), so any row written afterwards is invisible to
/// the boot load — the replication arrival this issue is about.
/// </summary>
private IActorRef CreateBootedDeploymentManager(
SiteRuntimeOptions options,
Func<Task<List<DeployedInstance>>>? configLoader = null)
{
var health = new DeploymentManagerStartupProbe();
var dm = ActorOf(Props.Create(() => new DeploymentManagerActor(
_storage,
_compilationService,
_sharedScriptLibrary,
null,
options,
NullLogger<DeploymentManagerActor>.Instance,
null,
health,
null,
null,
null,
null,
configLoader,
null,
null,
null,
null,
null,
null,
null)));
AwaitCondition(() => health.ReportCount > 0, TimeSpan.FromSeconds(20),
TimeSpan.FromMilliseconds(25));
return dm;
}
private static string MakeConfigJson(string instanceName) =>
JsonSerializer.Serialize(new FlattenedConfiguration
{
InstanceUniqueName = instanceName,
Attributes =
[
new ResolvedAttribute { CanonicalName = "TestAttr", Value = "42", DataType = "Int32" }
]
});
private async Task<bool> InstanceIsRunningAsync(IActorRef dm, string instanceName) =>
!(await dm.Ask<DebugViewSnapshot>(
new SubscribeDebugViewRequest(instanceName, Guid.NewGuid().ToString()),
TimeSpan.FromSeconds(5))).InstanceNotFound;
/// <summary>
/// THE #135 reproduction. The node boots on a stale store holding one instance; the peer's
/// snapshot then adds a second enabled instance as a bare storage write (the sync applier's
/// only side effect). The stale instance runs from the boot load, and the replicated one must
/// start within seconds — not one full plain sweep interval later.
///
/// <para><b>Fails before the fix</b>: with only the 60 s plain sweep, the replicated instance
/// is still not running when the budget expires.</para>
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ReplicatedRowArrivingJustAfterBoot_StartsWithinTheSettleWindow()
{
await _storage.StoreDeployedConfigAsync(
"StaleLocal", MakeConfigJson("StaleLocal"), "dep-local", "rev-local", true);
var dm = CreateBootedDeploymentManager(new SiteRuntimeOptions
{
InstanceSweepIntervalSeconds = PlainSweepSeconds,
InstanceSweepSettleIntervalSeconds = 1,
InstanceSweepSettleWindowSeconds = 30,
});
await AwaitAssertAsync(async () =>
Assert.True(await InstanceIsRunningAsync(dm, "StaleLocal")), TimeSpan.FromSeconds(10));
// The peer's snapshot lands about a second after the boot load.
await _storage.StoreDeployedConfigAsync(
"FromPeer", MakeConfigJson("FromPeer"), "dep-peer", "rev-peer", true);
await AwaitAssertAsync(async () =>
Assert.True(
await InstanceIsRunningAsync(dm, "FromPeer"),
"A replicated config that arrived just after boot was not started within the "
+ "settle window — the site would run without it for a full sweep interval (#135)"),
HealBudget);
}
/// <summary>
/// A snapshot that lands while the boot batches are still being created (after the boot read)
/// is caught by the one sweep run the moment boot-time creation finishes — even with the fast
/// settle cadence switched off. The config loader writes the peer's row right after it has
/// read the store, which pins the arrival deterministically into that window.
///
/// <para><b>Fails before the fix</b>: nothing sweeps until the 60 s plain tick.</para>
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task ReplicatedRowArrivingDuringBootCreation_IsStartedByTheImmediatePostBootSweep()
{
var dm = CreateBootedDeploymentManager(
new SiteRuntimeOptions
{
InstanceSweepIntervalSeconds = PlainSweepSeconds,
InstanceSweepSettleIntervalSeconds = 0,
},
configLoader: async () =>
{
var bootView = await _storage.GetAllDeployedConfigsAsync();
await _storage.StoreDeployedConfigAsync(
"MidBootPeer", MakeConfigJson("MidBootPeer"), "dep-mid", "rev-mid", true);
return bootView;
});
await AwaitAssertAsync(async () =>
Assert.True(
await InstanceIsRunningAsync(dm, "MidBootPeer"),
"A config replicated in while boot-time creation was running was not started "
+ "by the post-boot sweep (#135)"),
HealBudget);
}
/// <summary>
/// The settle window is bounded: once it closes the sweep is back on its plain interval, so a
/// row arriving afterwards is NOT picked up at the fast cadence. Guards against a settle timer
/// that is never cancelled (a permanent 1 s sweep on every site).
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task AfterTheSettleWindowCloses_TheSweepFallsBackToThePlainInterval()
{
var dm = CreateBootedDeploymentManager(new SiteRuntimeOptions
{
InstanceSweepIntervalSeconds = PlainSweepSeconds,
InstanceSweepSettleIntervalSeconds = 1,
InstanceSweepSettleWindowSeconds = 2,
});
// Let the window close.
await Task.Delay(TimeSpan.FromSeconds(4));
await _storage.StoreDeployedConfigAsync(
"LatePeer", MakeConfigJson("LatePeer"), "dep-late", "rev-late", true);
// Several fast-cadence ticks' worth of time.
await Task.Delay(TimeSpan.FromSeconds(4));
Assert.False(await InstanceIsRunningAsync(dm, "LatePeer"));
}
/// <summary>
/// <c>InstanceSweepIntervalSeconds = 0</c> keeps meaning "no sweep at all": the settle
/// window must not resurrect a sweep an operator switched off.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DisabledSweep_IsNotReenabledByTheSettleWindow()
{
var dm = CreateBootedDeploymentManager(new SiteRuntimeOptions
{
InstanceSweepIntervalSeconds = 0,
InstanceSweepSettleIntervalSeconds = 1,
InstanceSweepSettleWindowSeconds = 30,
});
await _storage.StoreDeployedConfigAsync(
"NoSweepPeer", MakeConfigJson("NoSweepPeer"), "dep-n", "rev-n", true);
await Task.Delay(TimeSpan.FromSeconds(4));
Assert.False(await InstanceIsRunningAsync(dm, "NoSweepPeer"));
}
}
@@ -23,6 +23,35 @@ public class SiteRuntimeOptionsValidatorTests
Assert.True(result.Succeeded, result.FailureMessage);
}
/// <summary>
/// ScadaBridge#135: the post-boot settle sweep cadence and window feed Akka timers, so a
/// negative value must be rejected with a key-naming message; 0 (disable) is valid.
/// </summary>
/// <param name="interval">InstanceSweepSettleIntervalSeconds.</param>
/// <param name="window">InstanceSweepSettleWindowSeconds.</param>
/// <param name="expectedKey">The key the failure must name, or null when valid.</param>
[Theory]
[InlineData(-1, 120, "InstanceSweepSettleIntervalSeconds")]
[InlineData(5, -1, "InstanceSweepSettleWindowSeconds")]
[InlineData(0, 0, null)]
public void InstanceSweepSettleOptions_RejectNegativeValues(int interval, int window, string? expectedKey)
{
var result = Validate(new SiteRuntimeOptions
{
InstanceSweepSettleIntervalSeconds = interval,
InstanceSweepSettleWindowSeconds = window,
});
if (expectedKey == null)
{
Assert.True(result.Succeeded, result.FailureMessage);
return;
}
Assert.True(result.Failed);
Assert.Contains(expectedKey, result.FailureMessage);
}
/// <summary>Verifies that zero startup batch size is rejected.</summary>
[Fact]
public void ZeroStartupBatchSize_IsRejected()