diff --git a/docs/components/SiteRuntime.md b/docs/components/SiteRuntime.md index fa75c1f5..1886d866 100644 --- a/docs/components/SiteRuntime.md +++ b/docs/components/SiteRuntime.md @@ -36,6 +36,8 @@ The staggered startup above reads storage exactly once. Anything that writes a ` `DeploymentManagerActor` therefore runs a periodic sweep (`InstanceSweepIntervalSeconds`, default 60; `0` disables) that re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config either starts a missing `InstanceActor` or — when the running one's revision no longer matches the stored one — re-applies the stored config **through the ordinary redeploy path**, so nothing about redeploy semantics is re-implemented. It defers to any in-flight deploy/warm/termination, never starts a disabled instance, stays inert until boot-time creation has finished, makes one repair attempt per (instance, revision) pair so it can never become a restart loop, and is completely silent when there is nothing to repair. Repairs emit Warning `deployment` site events naming the trigger (and, for a stale-revision restart, both revisions). Design note: `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`. +**Post-boot settle window (ScadaBridge#135).** On a replicated pair that cold-starts together, the node that wins the singleton loads its own store and the peer's rows arrive by LocalDb sync about a second later; the startup reconcile then rightly finds no gap, so only the sweep starts those instances — at the plain cadence, up to 60 s after a healthy-looking boot (reproduced on the rig 2026-11-13). So the moment boot-time creation finishes the singleton runs one sweep at once, then sweeps every `InstanceSweepSettleIntervalSeconds` (default 5) for `InstanceSweepSettleWindowSeconds` (default 120) before returning to the plain interval. Startup never waits for the peer — a node whose peer is down for good starts from its own store immediately, exactly as before. + Register row **85** was the same gap one layer down: the sweep started the instance, but the instance came up with **no DCL connection actors**, so every tag subscribe was answered `Unknown connection: `. The connection push (`EnsureDclConnections`) was the *caller's* job and two of the four Instance-Actor start paths omitted it; it now lives inside `CreateInstanceActor`. See *Deployment flow* below and §7 of the design note. ### Actor supervision and lifetimes @@ -282,6 +284,8 @@ All options live in the `ScadaBridge:SiteRuntime` section, bound to `SiteRuntime | `MirroredAlarmCapPerSource` | `1000` | Maximum mirrored conditions per `NativeAlarmActor` source binding before oldest is dropped and logged | | `NativeAlarmRetryIntervalMs` | `5000` | Milliseconds before retrying a failed native alarm subscription | | `InstanceSweepIntervalSeconds` | `60` | Interval of the singleton's instance reconcile sweep (register rows 78 + 72); `0` disables it — see Instance reconcile sweep | +| `InstanceSweepSettleIntervalSeconds` | `5` | Fast sweep cadence for the post-boot settle window (#135); `0` disables the settle phase. Only used when faster than `InstanceSweepIntervalSeconds` | +| `InstanceSweepSettleWindowSeconds` | `120` | Length of the post-boot settle window, from the end of boot-time Instance Actor creation (#135); `0` disables it | The SQLite connection string is passed directly to `AddSiteRuntime(connectionString)` in the host composition root and is not part of `SiteRuntimeOptions`. diff --git a/docs/plans/2026-08-15-instance-sweep-self-heal-design.md b/docs/plans/2026-08-15-instance-sweep-self-heal-design.md index 0afa00be..5272a61f 100644 --- a/docs/plans/2026-08-15-instance-sweep-self-heal-design.md +++ b/docs/plans/2026-08-15-instance-sweep-self-heal-design.md @@ -165,6 +165,8 @@ produces two operator-visible `deployment` site events naming both revisions. `EnsureInstanceStartedCommand` path is unaffected either way. No config change is required on any node to get the fix — the default is live. +**Post-boot settle window (ScadaBridge#135, 2026-11-13).** `InstanceSweepSettleIntervalSeconds` (default **5**) and `InstanceSweepSettleWindowSeconds` (default **120**), both validated `>= 0`. When boot-time creation finishes the singleton runs one sweep at once, then sweeps at the settle cadence for the window, then returns to the plain interval. It exists because a replicated pair that cold-starts together hits row 78 on the active node whenever the two stores differ, and at the plain 60 s cadence the replicated instances sat stopped for up to one full interval. `0` in either option disables the settle phase; with the sweep disabled neither does anything. + ## 5. Observability | Condition | Log | Site event | diff --git a/docs/requirements/Component-SiteRuntime.md b/docs/requirements/Component-SiteRuntime.md index 50c2efce..baad4b0a 100644 --- a/docs/requirements/Component-SiteRuntime.md +++ b/docs/requirements/Component-SiteRuntime.md @@ -104,6 +104,8 @@ flowchart TD > > Both are closed by one **periodic sweep owned by the Deployment Manager singleton** (`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`, default 60, validated `>= 0`; `0` disables it). Each tick re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config: starts the Instance Actor if none is running (row 78), or — if one is running at a revision the store has since replaced — **re-applies the stored config through the exact per-instance apply/restart path a central redeploy uses** (row 72). A sweep is chosen over a LocalDb change hook deliberately: a hook would cover only the replication arrival path, only on replicated sites, and `ILocalDb` exposes no change-notification API at all, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot snapshot resync, a hand-restored file, any future path). See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`. > +> **Post-boot settle window (ScadaBridge#135).** A replicated pair that cold-starts together hits row 78 on every boot where the stores differ (a first replication enable on diverged stores, or a node that was down while its peer took deploys): the node that wins the singleton loads its own store, and the peer's rows land about a second later. At the plain cadence those instances stayed stopped for up to one full sweep interval. So the moment boot-time creation finishes, the singleton runs one sweep at once, then sweeps every `ScadaBridge:SiteRuntime:InstanceSweepSettleIntervalSeconds` (default 5) for `InstanceSweepSettleWindowSeconds` (default 120), then falls back to the plain interval. Startup never waits for the peer, so a site still starts alone from its own store when its peer is permanently down. Both are validated `>= 0`, `0` disables the settle phase, and neither does anything when the sweep itself is disabled. +> > **The sweep reads metadata, not configs (review 2026-10-21 01/SR-P1).** Each tick compares only name / enabled / revision, but it used to read every row's `config_json` (~70 KB worst case) — on a 500-instance site ~35 MB of large-object-heap strings a minute, forever, on a perfectly healthy site. The tick now reads the `GetAllDeployedConfigMetadataAsync` projection (no `config_json`), decides its repairs on the actor thread exactly as before, and only then fetches the full row — one single-row read per instance it is about to start or restart — off-thread. When those rows land every decision is **re-validated**: a fetch this node has since mutated past is dropped whole by the same store-write generation guard, a row that vanished, was disabled or now carries a different revision than the one claimed is skipped, and the deploy-pipeline / running-actor guards are re-checked; a repair that does not act gives back its once-per-revision claim so the next tick re-evaluates it. Ticks stay suppressed until the fetch lands. The boot load keeps the full read — it runs once and needs every blob. > The **row-72 decision is to converge, not merely warn**: the stored row *is* what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing `HandleDeploy` means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy. diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs index ca1fe5ed..7fdb1b5b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs @@ -271,6 +271,19 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers /// private readonly TimeSpan _instanceSweepInterval; + /// + /// Fast sweep cadence for the post-boot settle window (ScadaBridge#135); + /// when the settle phase is off. See + /// . + /// + private readonly TimeSpan _instanceSweepSettleInterval; + + /// + /// Length of the post-boot settle window (ScadaBridge#135). See + /// . + /// + private readonly TimeSpan _instanceSweepSettleWindow; + /// /// True once boot-time Instance Actor creation has finished (every staggered batch created, /// or there were no enabled configs to create). The sweep is inert until then: a tick landing @@ -383,6 +396,12 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers _instanceSweepInterval = options.InstanceSweepIntervalSeconds > 0 ? TimeSpan.FromSeconds(options.InstanceSweepIntervalSeconds) : TimeSpan.Zero; + _instanceSweepSettleInterval = options.InstanceSweepSettleIntervalSeconds > 0 + ? TimeSpan.FromSeconds(options.InstanceSweepSettleIntervalSeconds) + : TimeSpan.Zero; + _instanceSweepSettleWindow = options.InstanceSweepSettleWindowSeconds > 0 + ? TimeSpan.FromSeconds(options.InstanceSweepSettleWindowSeconds) + : TimeSpan.Zero; _configLoader = configLoader ?? (() => _storage.GetAllDeployedConfigsAsync()); _deployedConfigLoader = deployedConfigLoader ?? (name => _storage.GetDeployedConfigAsync(name)); _configMetadataLoader = configMetadataLoader ?? (() => _storage.GetAllDeployedConfigMetadataAsync()); @@ -430,6 +449,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers // an enabled config with no running actor (row 78), and a running actor whose revision // no longer matches the stored one (row 72). Receive(_ => HandleInstanceSweepTick()); + Receive(_ => Timers.Cancel(InstanceSweepSettleTimerKey)); Receive(HandleInstanceSweepLoaded); // 01/SR-P1: the full rows of the instances a sweep decided to start/restart. Receive(HandleInstanceSweepRepairsLoaded); @@ -842,7 +862,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers // reconcile sweep may run. This is exactly the live case: the node booted // with an empty deployed_configurations table and CDC replication seeds it moments // later, with no boot load left to notice. - _startupInstanceCreationComplete = true; + MarkStartupInstanceCreationComplete(); return; } @@ -998,7 +1018,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers "All {Count} Instance Actors created", state.Configs.Count); // Boot-time creation is done — the instance reconcile sweep may now run without // racing the staggered batches it would otherwise mistake for a divergence. - _startupInstanceCreationComplete = true; + MarkStartupInstanceCreationComplete(); } } @@ -2103,6 +2123,58 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers $"Instance {instanceName} started by {reason} self-heal", details); } + /// Timer key of the fast post-boot settle sweep (ScadaBridge#135). + private const string InstanceSweepSettleTimerKey = "instance-sweep-settle"; + + /// + /// Ends boot-time Instance Actor creation and opens the post-boot settle window + /// (ScadaBridge#135). + /// + /// + /// The race this closes. On a replicated pair that cold-starts together, the node that + /// wins the singleton loads its OWN store, and the peer's rows arrive by LocalDb sync about a + /// second later — a first enable on diverged stores, or a node that was down while its peer + /// took deploys. The boot load has already run, and the startup reconcile reports no gap + /// because the rows ARE in the store by the time it asks central. Only the sweep starts those + /// instances, and at its plain cadence (60 s by default) the site collected nothing for up to + /// a full interval after a healthy-looking boot. LocalDb exposes no "snapshot applied" signal, + /// so the fix rides the arrival-path-agnostic sweep: one sweep NOW (rows that landed while the + /// boot batches were being created), then a fast cadence for a bounded window, then the plain + /// interval. + /// + /// + /// + /// Startup never waits. The node has already started everything its own store holds, so + /// a peer that is down for good costs only a few cheap metadata reads. Every guard the sweep + /// already carries (deploy-pipeline deferral, per-revision claims, the store-generation check) + /// applies unchanged, and a sweep that finds nothing stays silent. + /// + /// + private void MarkStartupInstanceCreationComplete() + { + if (_startupInstanceCreationComplete) + return; + + _startupInstanceCreationComplete = true; + + if (_instanceSweepInterval <= TimeSpan.Zero) + return; + + Self.Tell(InstanceSweepTick.Instance); + + if (_instanceSweepSettleInterval > TimeSpan.Zero && + _instanceSweepSettleWindow > TimeSpan.Zero && + _instanceSweepSettleInterval < _instanceSweepInterval) + { + Timers.StartPeriodicTimer( + InstanceSweepSettleTimerKey, InstanceSweepTick.Instance, + _instanceSweepSettleInterval, _instanceSweepSettleInterval); + Timers.StartSingleTimer( + "instance-sweep-settle-end", InstanceSweepSettleEnded.Instance, + _instanceSweepSettleWindow); + } + } + /// Provenance string carried on every repair the instance reconcile sweep makes. private const string SweepReason = "deployment-manager sweep"; @@ -4304,6 +4376,17 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers private InstanceSweepTick() { } } + /// + /// Closes the post-boot settle window (ScadaBridge#135): the fast settle sweep timer is + /// cancelled and the sweep continues at . + /// + internal sealed record InstanceSweepSettleEnded + { + /// The singleton instance — the message carries no state. + public static readonly InstanceSweepSettleEnded Instance = new(); + private InstanceSweepSettleEnded() { } + } + /// /// Carries one sweep's off-thread read of the durable deployed-config rows back onto the /// actor thread, where every comparison and repair happens. diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs index d220855d..0a338e92 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs @@ -242,6 +242,36 @@ public class SiteRuntimeOptions /// public int InstanceSweepIntervalSeconds { get; set; } = 60; + /// + /// Fast cadence (seconds) of the instance reconcile sweep during the post-boot settle + /// window (ScadaBridge#135). When the Deployment Manager singleton finishes its boot-time + /// Instance Actor creation it runs one sweep at once, then sweeps every + /// for + /// before falling back to + /// . + /// + /// Why: on a replicated pair the node that boots active loads its OWN store, and the peer's + /// rows arrive by LocalDb sync a moment later (a first enable on diverged stores, or a node + /// that was down while its peer took deploys). Those rows are invisible to the boot load and to + /// the startup reconcile (central rightly sees no gap), so only the sweep starts them — and at + /// the plain 60 s cadence that left the site collecting nothing for up to a full minute after a + /// cold start. Startup is never delayed: the node starts from its own store at once and a peer + /// that never answers costs nothing but a few cheap metadata reads. + /// + /// 0 disables the settle phase (the immediate post-boot sweep still runs). It only + /// applies when the sweep is enabled and is faster than . + /// Default: 5. + /// + public int InstanceSweepSettleIntervalSeconds { get; set; } = 5; + + /// + /// Length (seconds) of the post-boot settle window, measured from the end of boot-time + /// Instance Actor creation, during which the sweep runs at + /// (ScadaBridge#135). 0 disables + /// the settle phase. Default: 120. + /// + public int InstanceSweepSettleWindowSeconds { get; set; } = 120; + /// /// Flow Debugger trace arm window (design 2026-09-05 §5.3). When a TRACED script write /// targets a data-sourced attribute, the Instance Actor remembers diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs index 44c1d913..01a2c05a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs @@ -113,6 +113,18 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase= 0, + $"ScadaBridge:SiteRuntime:InstanceSweepSettleIntervalSeconds must be >= 0 " + + $"(was {options.InstanceSweepSettleIntervalSeconds}); it is the instance sweep's fast " + + "post-boot cadence and a negative value throws inside the Akka timer. 0 disables the " + + "settle phase (ScadaBridge#135)."); + + builder.RequireThat(options.InstanceSweepSettleWindowSeconds >= 0, + $"ScadaBridge:SiteRuntime:InstanceSweepSettleWindowSeconds must be >= 0 " + + $"(was {options.InstanceSweepSettleWindowSeconds}); it is how long the instance sweep " + + "runs at its fast post-boot cadence and a negative value throws inside the Akka timer. " + + "0 disables the settle phase (ScadaBridge#135)."); + builder.RequireThat(options.TraceArmWindowSeconds > 0, $"ScadaBridge:SiteRuntime:TraceArmWindowSeconds must be greater than 0 " + $"(was {options.TraceArmWindowSeconds}); it is how long the Instance Actor waits for the " + diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/PostBootSettleSweepTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/PostBootSettleSweepTests.cs new file mode 100644 index 00000000..25afa9d5 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/PostBootSettleSweepTests.cs @@ -0,0 +1,240 @@ +using System.Text.Json; +using Akka.Actor; +using Akka.TestKit.Xunit2; +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; +using ZB.MOM.WW.ScadaBridge.TestSupport; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; + +/// +/// Regression tests for ScadaBridge#135: a replicated site pair cold-starts together, the +/// node that wins the Deployment Manager singleton loads its OWN (stale) store, and the peer's +/// rows arrive by LocalDb sync about a second later. Those rows are invisible to the boot load +/// (already run) and to the startup reconcile (central sees no gap — the rows ARE in the store by +/// then), so only the instance sweep starts them. At its plain 60 s cadence the site ran none of +/// the replicated instances for a full interval after a healthy-looking boot — reproduced on the +/// primary rig 2026-11-13 (boot load 05:03:16 without the instance, sweep start at 05:04:16). +/// +/// +/// The fix opens a post-boot settle window: one sweep the moment boot-time creation finishes, +/// then a fast cadence for a bounded window, then the plain interval. Startup itself never waits +/// for the peer. Every test here pins the plain interval at 60 s or more, so a pass can only come +/// from the settle path — a plain-cadence sweep cannot land inside the assertion budgets. +/// +/// +public class PostBootSettleSweepTests : TestKit, IDisposable +{ + private readonly SiteStorageService _storage; + private readonly ScriptCompilationService _compilationService; + private readonly SharedScriptLibrary _sharedScriptLibrary; + private readonly TestLocalDb _localDb; + + /// The plain sweep interval: far longer than any assertion budget below. + private const int PlainSweepSeconds = 60; + + /// Well inside one plain interval, comfortably more than a few settle ticks. + private static readonly TimeSpan HealBudget = TimeSpan.FromSeconds(8); + + /// Creates a temporary LocalDb-backed site store and the compilation services for each test. + public PostBootSettleSweepTests() + { + _localDb = TestLocalDb.CreateTemp("post-boot-settle-sweep-test"); + _storage = new SiteStorageService(_localDb.Db, NullLogger.Instance); + _storage.InitializeAsync().GetAwaiter().GetResult(); + _compilationService = new ScriptCompilationService( + NullLogger.Instance); + _sharedScriptLibrary = new SharedScriptLibrary( + _compilationService, NullLogger.Instance); + } + + /// + void IDisposable.Dispose() + { + // Shut the actors down BEFORE the ILocalDb they write through is disposed. + Shutdown(); + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + + /// + /// Starts a Deployment Manager and blocks until its boot-time config load has PROVABLY read + /// storage (its first instance-count report), so any row written afterwards is invisible to + /// the boot load — the replication arrival this issue is about. + /// + private IActorRef CreateBootedDeploymentManager( + SiteRuntimeOptions options, + Func>>? configLoader = null) + { + var health = new DeploymentManagerStartupProbe(); + var dm = ActorOf(Props.Create(() => new DeploymentManagerActor( + _storage, + _compilationService, + _sharedScriptLibrary, + null, + options, + NullLogger.Instance, + null, + health, + null, + null, + null, + null, + configLoader, + null, + null, + null, + null, + null, + null, + null))); + + AwaitCondition(() => health.ReportCount > 0, TimeSpan.FromSeconds(20), + TimeSpan.FromMilliseconds(25)); + return dm; + } + + private static string MakeConfigJson(string instanceName) => + JsonSerializer.Serialize(new FlattenedConfiguration + { + InstanceUniqueName = instanceName, + Attributes = + [ + new ResolvedAttribute { CanonicalName = "TestAttr", Value = "42", DataType = "Int32" } + ] + }); + + private async Task InstanceIsRunningAsync(IActorRef dm, string instanceName) => + !(await dm.Ask( + new SubscribeDebugViewRequest(instanceName, Guid.NewGuid().ToString()), + TimeSpan.FromSeconds(5))).InstanceNotFound; + + /// + /// THE #135 reproduction. The node boots on a stale store holding one instance; the peer's + /// snapshot then adds a second enabled instance as a bare storage write (the sync applier's + /// only side effect). The stale instance runs from the boot load, and the replicated one must + /// start within seconds — not one full plain sweep interval later. + /// + /// Fails before the fix: with only the 60 s plain sweep, the replicated instance + /// is still not running when the budget expires. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task ReplicatedRowArrivingJustAfterBoot_StartsWithinTheSettleWindow() + { + await _storage.StoreDeployedConfigAsync( + "StaleLocal", MakeConfigJson("StaleLocal"), "dep-local", "rev-local", true); + + var dm = CreateBootedDeploymentManager(new SiteRuntimeOptions + { + InstanceSweepIntervalSeconds = PlainSweepSeconds, + InstanceSweepSettleIntervalSeconds = 1, + InstanceSweepSettleWindowSeconds = 30, + }); + + await AwaitAssertAsync(async () => + Assert.True(await InstanceIsRunningAsync(dm, "StaleLocal")), TimeSpan.FromSeconds(10)); + + // The peer's snapshot lands about a second after the boot load. + await _storage.StoreDeployedConfigAsync( + "FromPeer", MakeConfigJson("FromPeer"), "dep-peer", "rev-peer", true); + + await AwaitAssertAsync(async () => + Assert.True( + await InstanceIsRunningAsync(dm, "FromPeer"), + "A replicated config that arrived just after boot was not started within the " + + "settle window — the site would run without it for a full sweep interval (#135)"), + HealBudget); + } + + /// + /// A snapshot that lands while the boot batches are still being created (after the boot read) + /// is caught by the one sweep run the moment boot-time creation finishes — even with the fast + /// settle cadence switched off. The config loader writes the peer's row right after it has + /// read the store, which pins the arrival deterministically into that window. + /// + /// Fails before the fix: nothing sweeps until the 60 s plain tick. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task ReplicatedRowArrivingDuringBootCreation_IsStartedByTheImmediatePostBootSweep() + { + var dm = CreateBootedDeploymentManager( + new SiteRuntimeOptions + { + InstanceSweepIntervalSeconds = PlainSweepSeconds, + InstanceSweepSettleIntervalSeconds = 0, + }, + configLoader: async () => + { + var bootView = await _storage.GetAllDeployedConfigsAsync(); + await _storage.StoreDeployedConfigAsync( + "MidBootPeer", MakeConfigJson("MidBootPeer"), "dep-mid", "rev-mid", true); + return bootView; + }); + + await AwaitAssertAsync(async () => + Assert.True( + await InstanceIsRunningAsync(dm, "MidBootPeer"), + "A config replicated in while boot-time creation was running was not started " + + "by the post-boot sweep (#135)"), + HealBudget); + } + + /// + /// The settle window is bounded: once it closes the sweep is back on its plain interval, so a + /// row arriving afterwards is NOT picked up at the fast cadence. Guards against a settle timer + /// that is never cancelled (a permanent 1 s sweep on every site). + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task AfterTheSettleWindowCloses_TheSweepFallsBackToThePlainInterval() + { + var dm = CreateBootedDeploymentManager(new SiteRuntimeOptions + { + InstanceSweepIntervalSeconds = PlainSweepSeconds, + InstanceSweepSettleIntervalSeconds = 1, + InstanceSweepSettleWindowSeconds = 2, + }); + + // Let the window close. + await Task.Delay(TimeSpan.FromSeconds(4)); + + await _storage.StoreDeployedConfigAsync( + "LatePeer", MakeConfigJson("LatePeer"), "dep-late", "rev-late", true); + + // Several fast-cadence ticks' worth of time. + await Task.Delay(TimeSpan.FromSeconds(4)); + + Assert.False(await InstanceIsRunningAsync(dm, "LatePeer")); + } + + /// + /// InstanceSweepIntervalSeconds = 0 keeps meaning "no sweep at all": the settle + /// window must not resurrect a sweep an operator switched off. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task DisabledSweep_IsNotReenabledByTheSettleWindow() + { + var dm = CreateBootedDeploymentManager(new SiteRuntimeOptions + { + InstanceSweepIntervalSeconds = 0, + InstanceSweepSettleIntervalSeconds = 1, + InstanceSweepSettleWindowSeconds = 30, + }); + + await _storage.StoreDeployedConfigAsync( + "NoSweepPeer", MakeConfigJson("NoSweepPeer"), "dep-n", "rev-n", true); + + await Task.Delay(TimeSpan.FromSeconds(4)); + + Assert.False(await InstanceIsRunningAsync(dm, "NoSweepPeer")); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs index 0d58b82b..b0085a6d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs @@ -23,6 +23,35 @@ public class SiteRuntimeOptionsValidatorTests Assert.True(result.Succeeded, result.FailureMessage); } + /// + /// ScadaBridge#135: the post-boot settle sweep cadence and window feed Akka timers, so a + /// negative value must be rejected with a key-naming message; 0 (disable) is valid. + /// + /// InstanceSweepSettleIntervalSeconds. + /// InstanceSweepSettleWindowSeconds. + /// The key the failure must name, or null when valid. + [Theory] + [InlineData(-1, 120, "InstanceSweepSettleIntervalSeconds")] + [InlineData(5, -1, "InstanceSweepSettleWindowSeconds")] + [InlineData(0, 0, null)] + public void InstanceSweepSettleOptions_RejectNegativeValues(int interval, int window, string? expectedKey) + { + var result = Validate(new SiteRuntimeOptions + { + InstanceSweepSettleIntervalSeconds = interval, + InstanceSweepSettleWindowSeconds = window, + }); + + if (expectedKey == null) + { + Assert.True(result.Succeeded, result.FailureMessage); + return; + } + + Assert.True(result.Failed); + Assert.Contains(expectedKey, result.FailureMessage); + } + /// Verifies that zero startup batch size is rejected. [Fact] public void ZeroStartupBatchSize_IsRejected()