fix(site-runtime): post-boot settle sweep so replicated rows start in seconds, not a sweep interval (#135)

When a replicated site pair cold-starts together, the node that wins the
Deployment Manager singleton loads its own (stale) store, and the peer's
rows arrive by LocalDb sync about a second later. The boot load has already
run and the startup reconcile rightly reports no gap (the rows are in the
store by the time it asks central), so only the instance sweep starts those
instances - at its plain 60 s cadence. Reproduced on the primary rig: boot
load without the instance at 05:03:16, sweep start at 05:04:16.

The env2 drill read this as "never started" because the operator
redeployed at +49 s, before the first sweep tick at +60 s.

Fix: when boot-time Instance Actor creation finishes, the singleton runs
one sweep at once, then sweeps every InstanceSweepSettleIntervalSeconds
(default 5) for InstanceSweepSettleWindowSeconds (default 120), then falls
back to InstanceSweepIntervalSeconds. Startup never waits for the peer, so
a site whose peer is down for good still starts alone from its own store.
LocalDb 0.3.7 exposes no "snapshot applied" signal, so the fix rides the
arrival-path-agnostic sweep with all its existing guards.

Tests: PostBootSettleSweepTests (4) + validator theory. The two positive
tests fail with the fix disabled; the window-close test fails when the
settle timer is never cancelled.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 00:10:06 -05:00
parent c3c1f6e870
commit d7fa31f0b8
8 changed files with 404 additions and 2 deletions
+4
View File
@@ -36,6 +36,8 @@ The staggered startup above reads storage exactly once. Anything that writes a `
`DeploymentManagerActor` therefore runs a periodic sweep (`InstanceSweepIntervalSeconds`, default 60; `0` disables) that re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config either starts a missing `InstanceActor` or — when the running one's revision no longer matches the stored one — re-applies the stored config **through the ordinary redeploy path**, so nothing about redeploy semantics is re-implemented. It defers to any in-flight deploy/warm/termination, never starts a disabled instance, stays inert until boot-time creation has finished, makes one repair attempt per (instance, revision) pair so it can never become a restart loop, and is completely silent when there is nothing to repair. Repairs emit Warning `deployment` site events naming the trigger (and, for a stale-revision restart, both revisions). Design note: `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
**Post-boot settle window (ScadaBridge#135).** On a replicated pair that cold-starts together, the node that wins the singleton loads its own store and the peer's rows arrive by LocalDb sync about a second later; the startup reconcile then rightly finds no gap, so only the sweep starts those instances — at the plain cadence, up to 60 s after a healthy-looking boot (reproduced on the rig 2026-11-13). So the moment boot-time creation finishes the singleton runs one sweep at once, then sweeps every `InstanceSweepSettleIntervalSeconds` (default 5) for `InstanceSweepSettleWindowSeconds` (default 120) before returning to the plain interval. Startup never waits for the peer — a node whose peer is down for good starts from its own store immediately, exactly as before.
Register row **85** was the same gap one layer down: the sweep started the instance, but the instance came up with **no DCL connection actors**, so every tag subscribe was answered `Unknown connection: <name>`. The connection push (`EnsureDclConnections`) was the *caller's* job and two of the four Instance-Actor start paths omitted it; it now lives inside `CreateInstanceActor`. See *Deployment flow* below and §7 of the design note.
### Actor supervision and lifetimes
@@ -282,6 +284,8 @@ All options live in the `ScadaBridge:SiteRuntime` section, bound to `SiteRuntime
| `MirroredAlarmCapPerSource` | `1000` | Maximum mirrored conditions per `NativeAlarmActor` source binding before oldest is dropped and logged |
| `NativeAlarmRetryIntervalMs` | `5000` | Milliseconds before retrying a failed native alarm subscription |
| `InstanceSweepIntervalSeconds` | `60` | Interval of the singleton's instance reconcile sweep (register rows 78 + 72); `0` disables it — see Instance reconcile sweep |
| `InstanceSweepSettleIntervalSeconds` | `5` | Fast sweep cadence for the post-boot settle window (#135); `0` disables the settle phase. Only used when faster than `InstanceSweepIntervalSeconds` |
| `InstanceSweepSettleWindowSeconds` | `120` | Length of the post-boot settle window, from the end of boot-time Instance Actor creation (#135); `0` disables it |
The SQLite connection string is passed directly to `AddSiteRuntime(connectionString)` in the host composition root and is not part of `SiteRuntimeOptions`.
@@ -165,6 +165,8 @@ produces two operator-visible `deployment` site events naming both revisions.
`EnsureInstanceStartedCommand` path is unaffected either way. No config change is required on any
node to get the fix — the default is live.
**Post-boot settle window (ScadaBridge#135, 2026-11-13).** `InstanceSweepSettleIntervalSeconds` (default **5**) and `InstanceSweepSettleWindowSeconds` (default **120**), both validated `>= 0`. When boot-time creation finishes the singleton runs one sweep at once, then sweeps at the settle cadence for the window, then returns to the plain interval. It exists because a replicated pair that cold-starts together hits row 78 on the active node whenever the two stores differ, and at the plain 60 s cadence the replicated instances sat stopped for up to one full interval. `0` in either option disables the settle phase; with the sweep disabled neither does anything.
## 5. Observability
| Condition | Log | Site event |
@@ -104,6 +104,8 @@ flowchart TD
>
> Both are closed by one **periodic sweep owned by the Deployment Manager singleton** (`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`, default 60, validated `>= 0`; `0` disables it). Each tick re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config: starts the Instance Actor if none is running (row 78), or — if one is running at a revision the store has since replaced — **re-applies the stored config through the exact per-instance apply/restart path a central redeploy uses** (row 72). A sweep is chosen over a LocalDb change hook deliberately: a hook would cover only the replication arrival path, only on replicated sites, and `ILocalDb` exposes no change-notification API at all, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot snapshot resync, a hand-restored file, any future path). See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
>
> **Post-boot settle window (ScadaBridge#135).** A replicated pair that cold-starts together hits row 78 on every boot where the stores differ (a first replication enable on diverged stores, or a node that was down while its peer took deploys): the node that wins the singleton loads its own store, and the peer's rows land about a second later. At the plain cadence those instances stayed stopped for up to one full sweep interval. So the moment boot-time creation finishes, the singleton runs one sweep at once, then sweeps every `ScadaBridge:SiteRuntime:InstanceSweepSettleIntervalSeconds` (default 5) for `InstanceSweepSettleWindowSeconds` (default 120), then falls back to the plain interval. Startup never waits for the peer, so a site still starts alone from its own store when its peer is permanently down. Both are validated `>= 0`, `0` disables the settle phase, and neither does anything when the sweep itself is disabled.
>
> **The sweep reads metadata, not configs (review 2026-10-21 01/SR-P1).** Each tick compares only name / enabled / revision, but it used to read every row's `config_json` (~70 KB worst case) — on a 500-instance site ~35 MB of large-object-heap strings a minute, forever, on a perfectly healthy site. The tick now reads the `GetAllDeployedConfigMetadataAsync` projection (no `config_json`), decides its repairs on the actor thread exactly as before, and only then fetches the full row — one single-row read per instance it is about to start or restart — off-thread. When those rows land every decision is **re-validated**: a fetch this node has since mutated past is dropped whole by the same store-write generation guard, a row that vanished, was disabled or now carries a different revision than the one claimed is skipped, and the deploy-pipeline / running-actor guards are re-checked; a repair that does not act gives back its once-per-revision claim so the next tick re-evaluates it. Ticks stay suppressed until the fetch lands. The boot load keeps the full read — it runs once and needs every blob.
> The **row-72 decision is to converge, not merely warn**: the stored row *is* what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing `HandleDeploy` means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy.