fix(site-runtime): post-boot settle sweep so replicated rows start in seconds, not a sweep interval (#135)
When a replicated site pair cold-starts together, the node that wins the Deployment Manager singleton loads its own (stale) store, and the peer's rows arrive by LocalDb sync about a second later. The boot load has already run and the startup reconcile rightly reports no gap (the rows are in the store by the time it asks central), so only the instance sweep starts those instances - at its plain 60 s cadence. Reproduced on the primary rig: boot load without the instance at 05:03:16, sweep start at 05:04:16. The env2 drill read this as "never started" because the operator redeployed at +49 s, before the first sweep tick at +60 s. Fix: when boot-time Instance Actor creation finishes, the singleton runs one sweep at once, then sweeps every InstanceSweepSettleIntervalSeconds (default 5) for InstanceSweepSettleWindowSeconds (default 120), then falls back to InstanceSweepIntervalSeconds. Startup never waits for the peer, so a site whose peer is down for good still starts alone from its own store. LocalDb 0.3.7 exposes no "snapshot applied" signal, so the fix rides the arrival-path-agnostic sweep with all its existing guards. Tests: PostBootSettleSweepTests (4) + validator theory. The two positive tests fail with the fix disabled; the window-close test fails when the settle timer is never cancelled. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -36,6 +36,8 @@ The staggered startup above reads storage exactly once. Anything that writes a `
|
||||
|
||||
`DeploymentManagerActor` therefore runs a periodic sweep (`InstanceSweepIntervalSeconds`, default 60; `0` disables) that re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config either starts a missing `InstanceActor` or — when the running one's revision no longer matches the stored one — re-applies the stored config **through the ordinary redeploy path**, so nothing about redeploy semantics is re-implemented. It defers to any in-flight deploy/warm/termination, never starts a disabled instance, stays inert until boot-time creation has finished, makes one repair attempt per (instance, revision) pair so it can never become a restart loop, and is completely silent when there is nothing to repair. Repairs emit Warning `deployment` site events naming the trigger (and, for a stale-revision restart, both revisions). Design note: `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
|
||||
|
||||
**Post-boot settle window (ScadaBridge#135).** On a replicated pair that cold-starts together, the node that wins the singleton loads its own store and the peer's rows arrive by LocalDb sync about a second later; the startup reconcile then rightly finds no gap, so only the sweep starts those instances — at the plain cadence, up to 60 s after a healthy-looking boot (reproduced on the rig 2026-11-13). So the moment boot-time creation finishes the singleton runs one sweep at once, then sweeps every `InstanceSweepSettleIntervalSeconds` (default 5) for `InstanceSweepSettleWindowSeconds` (default 120) before returning to the plain interval. Startup never waits for the peer — a node whose peer is down for good starts from its own store immediately, exactly as before.
|
||||
|
||||
Register row **85** was the same gap one layer down: the sweep started the instance, but the instance came up with **no DCL connection actors**, so every tag subscribe was answered `Unknown connection: <name>`. The connection push (`EnsureDclConnections`) was the *caller's* job and two of the four Instance-Actor start paths omitted it; it now lives inside `CreateInstanceActor`. See *Deployment flow* below and §7 of the design note.
|
||||
|
||||
### Actor supervision and lifetimes
|
||||
@@ -282,6 +284,8 @@ All options live in the `ScadaBridge:SiteRuntime` section, bound to `SiteRuntime
|
||||
| `MirroredAlarmCapPerSource` | `1000` | Maximum mirrored conditions per `NativeAlarmActor` source binding before oldest is dropped and logged |
|
||||
| `NativeAlarmRetryIntervalMs` | `5000` | Milliseconds before retrying a failed native alarm subscription |
|
||||
| `InstanceSweepIntervalSeconds` | `60` | Interval of the singleton's instance reconcile sweep (register rows 78 + 72); `0` disables it — see Instance reconcile sweep |
|
||||
| `InstanceSweepSettleIntervalSeconds` | `5` | Fast sweep cadence for the post-boot settle window (#135); `0` disables the settle phase. Only used when faster than `InstanceSweepIntervalSeconds` |
|
||||
| `InstanceSweepSettleWindowSeconds` | `120` | Length of the post-boot settle window, from the end of boot-time Instance Actor creation (#135); `0` disables it |
|
||||
|
||||
The SQLite connection string is passed directly to `AddSiteRuntime(connectionString)` in the host composition root and is not part of `SiteRuntimeOptions`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user