fix(site-runtime): post-boot settle sweep so replicated rows start in seconds, not a sweep interval (#135)

When a replicated site pair cold-starts together, the node that wins the
Deployment Manager singleton loads its own (stale) store, and the peer's
rows arrive by LocalDb sync about a second later. The boot load has already
run and the startup reconcile rightly reports no gap (the rows are in the
store by the time it asks central), so only the instance sweep starts those
instances - at its plain 60 s cadence. Reproduced on the primary rig: boot
load without the instance at 05:03:16, sweep start at 05:04:16.

The env2 drill read this as "never started" because the operator
redeployed at +49 s, before the first sweep tick at +60 s.

Fix: when boot-time Instance Actor creation finishes, the singleton runs
one sweep at once, then sweeps every InstanceSweepSettleIntervalSeconds
(default 5) for InstanceSweepSettleWindowSeconds (default 120), then falls
back to InstanceSweepIntervalSeconds. Startup never waits for the peer, so
a site whose peer is down for good still starts alone from its own store.
LocalDb 0.3.7 exposes no "snapshot applied" signal, so the fix rides the
arrival-path-agnostic sweep with all its existing guards.

Tests: PostBootSettleSweepTests (4) + validator theory. The two positive
tests fail with the fix disabled; the window-close test fails when the
settle timer is never cancelled.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 00:10:06 -05:00
parent c3c1f6e870
commit d7fa31f0b8
8 changed files with 404 additions and 2 deletions
@@ -165,6 +165,8 @@ produces two operator-visible `deployment` site events naming both revisions.
`EnsureInstanceStartedCommand` path is unaffected either way. No config change is required on any
node to get the fix — the default is live.
**Post-boot settle window (ScadaBridge#135, 2026-11-13).** `InstanceSweepSettleIntervalSeconds` (default **5**) and `InstanceSweepSettleWindowSeconds` (default **120**), both validated `>= 0`. When boot-time creation finishes the singleton runs one sweep at once, then sweeps at the settle cadence for the window, then returns to the plain interval. It exists because a replicated pair that cold-starts together hits row 78 on the active node whenever the two stores differ, and at the plain 60 s cadence the replicated instances sat stopped for up to one full interval. `0` in either option disables the settle phase; with the sweep disabled neither does anything.
## 5. Observability
| Condition | Log | Site event |