fix(siteruntime): instance reconcile sweep closes config/runtime divergence (register rows 78, 72)

The site's deployed_configurations rows are the durable truth about what should be
running; the set of live Instance Actors is what IS running. Nothing kept the two in
agreement after boot, and nothing surfaced a disagreement.

Row 78 (availability, proven live 2026-08-15) -- deployed_configurations is one of
LocalDb's ten replicated tables, so on a replicated pair a config can appear in a
node's file with no local actor involved at all: the CDC sync applier writes the row
directly. Live sequence: site-a-b's reconcile fetched 4 configs at its solo boot; the
pair was then restarted with a fresh site-a-a file and snapshot replication seeded
site-a-a's table within ~5s -- AFTER its boot-time load had read an empty table, and
invisibly to its own reconcile (central rightly reported "0 fetched", because central's
view of the site was correct). Enabled instances sat dead with their configs present
until yet another restart. Row 67's EnsureInstanceStartedCommand only covers configs
the reconcile itself FETCHES; this is the same gap through a side door.

Row 72 (silent divergence) -- for a STALE ReconcileGapItem the pass writes the newer
config via StoreDeployedConfigIfNewerAsync, a pure file write, while the running
Instance Actor keeps executing the previously compiled config until the next restart
or redeploy. Nothing said "stored revision != running revision".

Both are closed by one periodic sweep owned by the Deployment Manager singleton
(ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds, default 60, validated >= 0;
0 disables). Each tick re-reads the durable rows off-thread and, on the actor thread,
for every ENABLED stored config either starts a missing Instance Actor (row 78) or --
when the running one's revision no longer matches the stored one -- re-applies the
stored config through the ordinary redeploy path (row 72).

A sweep was chosen over a LocalDb change hook deliberately: a hook covers only the
replication arrival path, and only on the half of the fleet running with replication
enabled, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot
snapshot resync, a hand-restored file, any future path). It is also not available --
ILocalDb 0.3.0 exposes no change-notification or applied-rows callback, so a hook would
mean a new upstream capability in scadaproj plus an in-process bus into this singleton,
a cross-repo change to close a site-local gap.

Row 72's decision is to CONVERGE, not merely warn. The stored row is what central
successfully deployed; an actor serving the config central believes was replaced is the
actual fault, and a surface-only warning leaves the operator to issue by hand exactly
the redeploy the sweep would have performed. Reusing HandleDeploy means the compile
gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm
reset and all-or-nothing-per-instance are the production ones, not a second
implementation -- and a stop still does NOT cancel an in-flight script run, exactly as
for a central redeploy.

Guards, shared with row 67 via the extracted InstanceIsOwnedByADeployPipeline predicate
and the shared StartStoredInstance helper:
- a DISABLED instance is never started (config row, no actor is its correct state);
- the sweep defers to any deploy pipeline owning the instance (compile warm in flight,
  predecessor still terminating, deployment awaiting its persistence/init join), so it
  can never interleave with a real redeploy -- that window has the config stored and no
  actor, which an unguarded sweep would answer by creating a child whose actor name is
  still registered;
- inert until boot-time Instance Actor creation has finished, so a tick between
  staggered batches does not "heal" instances the boot load is about to create;
- ONE repair attempt per (instance, revision) pair -- this is what stops a periodic
  self-heal becoming a restart loop for an actor that dies during init, or for a stored
  config the compile gate rejects (a rejection leaves the running instance untouched:
  the gate returns before anything is stopped);
- reads never overlap, so a slow read cannot stack ticks into duplicate repairs.

A sweep that finds nothing is completely silent -- no log above Trace, no site event.
A repair emits the same Warning `deployment` site-event pattern as row 67 with the
trigger named ("deployment-manager sweep self-heal") so it is distinguishable from a
startup-reconcile heal; a stale-revision restart names BOTH revisions and a rejected
one adds an Error event. The synthetic redeploy replies to the singleton itself rather
than NoSender, because dead letters are a monitored site health metric.

CreateInstanceActor gains a revisionHash parameter and is the single creation funnel
that records it, so the stale check covers every path that can start an Instance Actor;
a missing entry reads as "unknown", never "stale", so a future path bypassing the funnel
degrades to no-op rather than to spurious restarts.

Fail-before by behavioral revert (sweep timer left unarmed): 5 of 10 new tests fail,
including both live symptoms -- the row-78 repro ("A replication-seeded config for an
enabled instance left no running Instance Actor") and the row-72 repro, whose snapshot
still shows the rev-1 value 111 while the store holds rev-2's 222. The 5 negative
guards (disabled / in-flight redeploy / matching revisions / silent-when-healthy /
interval-zero) are green in both states, as they must be.

SiteRuntime 626/626, Host 505/505, AuditLog 383/383.

Design note: docs/plans/2026-08-15-instance-sweep-self-heal-design.md

docs/plans/2026-08-15-instance-sweep-self-heal-design.md

@@ -0,0 +1,203 @@
+# Instance reconcile sweep — closing the config/runtime divergence gaps
+
+**Date:** 2026-08-15
+**Scope:** Site Runtime (`DeploymentManagerActor`), deferred-work-register rows **78** and **72**
+**Status:** implemented
+
+---
+
+## 1. The problem
+
+The site's `deployed_configurations` rows are the durable truth about what should be running.
+The set of live Instance Actors is what *is* running. Nothing kept the two in agreement after
+boot, and there was no signal when they disagreed.
+
+Before this change, an Instance Actor could be created from exactly three places:
+
+1. the boot-time SQLite load in `DeploymentManagerActor.PreStart`;
+2. an explicit deploy / enable command from central;
+3. (register row 67, commit `482165cb`) `SiteReconciliationActor`'s `EnsureInstanceStartedCommand`,
+   raised for enabled instances **the reconcile pass itself fetched and stored**.
+
+### Row 78 — configs that arrive by replication start nothing
+
+*(Medium-Low, availability; proven live 2026-08-15.)*
+
+`deployed_configurations` is one of LocalDb's ten replicated tables. On a replicated site pair,
+a config can therefore appear in a node's file with **no local actor involved at all**: the CDC
+sync applier writes the row directly.
+
+The live sequence was:
+
+- site-a-b booted solo; its startup reconcile fetched 4 configs from central and stored them
+  (row 67's machinery started their actors on that node — correct).
+- The pair was then restarted with a **fresh** site-a-a file. CDC snapshot replication seeded
+  site-a-a's `deployed_configurations` within ~5 s of boot.
+- That was **after** site-a-a's boot-time load had already read an empty table, and **invisible**
+  to site-a-a's own reconcile pass: central rightly reported *no gap* ("0 fetched"), because
+  central's view of the site was correct — the site really did have the configs.
+- Result: enabled instances sat dead with their configs present (`debug snapshot` →
+  `instanceNotFound`) until yet another restart.
+
+Row 67 covers configs the reconcile *fetches*. This is the same gap through a side door.
+
+### Row 72 — a refreshed config does not reach the running actor
+
+*(Medium-Low, same silent-divergence class.)*
+
+`ReconcileGapItem` covers both *missing* and *stale* configs. For a **stale** item the reconcile
+writes the newer config via `StoreDeployedConfigIfNewerAsync` — a pure file write. The running
+Instance Actor keeps executing the **previously compiled** config until the next restart or
+redeploy, and nothing anywhere surfaced "stored revision ≠ running revision".
+
+Row 72 was deliberately deferred out of row 67's scope because restarting a *live* instance
+carries redeploy semantics that needed their own decision. That decision is section 3.
+
+---
+
+## 2. The mechanism: one periodic sweep in the Deployment Manager singleton
+
+A single periodic tick on the Deployment Manager singleton re-reads every durable
+`deployed_configurations` row and repairs both divergences.
+
+```
+InstanceSweepTick (every InstanceSweepIntervalSeconds, default 60)
+  └── off-thread read of all deployed configs
+        └── InstanceSweepLoaded → actor thread
+              for each ENABLED stored config, unless a deploy pipeline owns the instance:
+                 no running actor       → start it            (row 78)
+                 running, revision ≠ stored → re-apply stored (row 72)
+```
+
+**Ownership.** The Deployment Manager is a cluster singleton scoped to the site-specific role, so
+the sweep runs on exactly one node — the active one — and Instance Actors stay under the
+singleton, which is the same invariant row 67's singleton-proxy routing preserved.
+
+### 2.1 Why a sweep and not a LocalDb change hook
+
+The register suggested either. A sweep was chosen:
+
+- **Coverage.** A replication-apply hook covers exactly one arrival path, and only on the half of
+  the fleet that runs with replication enabled (rig posture: site-a replicated, site-b/site-c
+  deliberately not). The sweep is arrival-path-agnostic — replication seeding, a post-boot
+  snapshot resync, an operator restoring a file by hand, and any future path all reduce to the
+  same question: *does the store agree with the runtime?*
+- **Availability.** `ILocalDb` exposes **no** change-notification, applied-rows callback or
+  observer API — verified against `ZB.MOM.WW.LocalDb` 0.3.0. A hook would require a new upstream
+  capability in the `scadaproj` library plus an in-process bus from the sync applier into this
+  singleton, i.e. a cross-repo change to close a site-local gap.
+- **Reuse.** The sweep inherits row 67's guards verbatim; the shared `StartStoredInstance` helper
+  means the reconcile heal and the sweep heal produce identical bookkeeping and identical
+  operator-facing events, differing only in the trigger they name.
+
+Cost is one SQLite read per minute per site — the same read `HandleDeploymentStateQuery` already
+performs per central query.
+
+### 2.2 Guards (all inherited or shared with row 67)
+
+- **Disabled instances are never started.** A disabled instance legitimately has a config row and
+  no actor; that is its correct steady state.
+- **The sweep defers to any deploy pipeline that owns the instance** — a compile warm in flight
+  (`_deployWarms`), a predecessor still terminating (`_terminatingActorsByName`), or a deployment
+  awaiting its persistence/init join (`_inFlightDeployments`). This is the same
+  `InstanceIsOwnedByADeployPipeline` predicate row 67's `ShouldAttemptEnsureStart` is now built
+  from. It matters: a redeploy passes through a window where the predecessor has been stopped and
+  removed from routing but the replacement does not exist yet, which an unguarded sweep would read
+  as "enabled config, no actor" and answer by creating a child whose actor name is still
+  registered.
+- **Inert until boot-time creation finishes.** A tick landing between two staggered startup
+  batches would "heal" instances the boot load is already about to create, announcing a fault on
+  a node that is merely still starting.
+- **One repair attempt per (instance, revision) pair.** This is what keeps a *periodic* self-heal
+  from becoming a restart loop: an Instance Actor that dies during init leaves its config row in
+  place, and a stored config the site compile gate rejects can never apply. Without the claim,
+  either would be re-attempted — and re-announced — every tick forever. A genuinely new revision
+  is a different key and is always attempted.
+- **No overlapping reads.** A slow storage read cannot let ticks stack up into duplicate repairs.
+- **Silent when healthy.** A sweep that finds nothing emits no log line above Trace and no site
+  event. This runs on every tick of every healthy site, forever.
+
+---
+
+## 3. The row-72 decision: converge, do not merely warn
+
+**Decision.** When the sweep finds a running instance whose revision no longer matches the stored
+row, it **applies the stored config through the same per-instance apply/restart path a central
+redeploy uses** (`HandleDeploy` → compile gate → stop/watch/recreate → persist), and announces
+both revisions in a Warning `deployment` site event.
+
+**Rationale.**
+
+- The stored row **is** what central successfully deployed. The running actor is executing a
+  config central believes was replaced. Leaving it running is the actual fault — a site quietly
+  serving superseded logic is worse than a controlled restart, and the operator has no way to
+  discover it.
+- The alternative (surface-only) leaves an operator with a Warning they can only act on by
+  issuing a redeploy — the exact operation the sweep would have performed, minus the delay and
+  minus the chance they see it.
+- Reusing `HandleDeploy` rather than writing a second apply path means the compile gate, the
+  warm-then-gate ordering, the redeploy-supersede semantics, the static-override and
+  native-alarm reset, and all-or-nothing-per-instance are the ones already in production. No new
+  semantics were invented. In particular a stop still does **not** cancel an in-flight script
+  run, exactly as for a central redeploy.
+
+**Why this is safe against the ordering hazards the register worried about.**
+
+- The sweep never acts on an instance a deploy pipeline owns, so it cannot interleave with a real
+  redeploy; and if a deploy arrives *after* the sweep's own deploy is queued, it is ordinary
+  last-write-wins on the same mailbox, which is a case the deploy path already handles.
+- A rejected restart is inert, not partial: `RunDeployGateAndProceed` returns **before** anything
+  is stopped, so the running instance keeps serving its old config. It is announced once
+  (Warning for the attempt, Error for the failure) and never retried for that revision.
+- The synthetic `DeployInstanceCommand` replies to `Self`, not to `NoSender` — dead letters are a
+  monitored site health metric, and a self-heal must not manufacture them.
+
+**Row 72's "nothing surfaces it" is false either way.** Even in the failure path the divergence
+produces two operator-visible `deployment` site events naming both revisions.
+
+---
+
+## 4. Configuration
+
+`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds` — default **60**, validated `>= 0`
+(`SiteRuntimeOptionsValidator`). **`0` disables the sweep**, which reopens both gaps; the row-67
+`EnsureInstanceStartedCommand` path is unaffected either way. No config change is required on any
+node to get the fix — the default is live.
+
+## 5. Observability
+
+| Condition | Log | Site event |
+|---|---|---|
+| Nothing to repair | — | — |
+| Row 78 start | Warning naming the instance and trigger | `deployment` / **Warning** — "started by deployment-manager sweep self-heal" |
+| Row 72 restart attempt | Warning naming both revisions | `deployment` / **Warning** — "restarted by deployment-manager sweep self-heal — running revision X is stale, applying stored revision Y" |
+| Row 72 restart applied | Information | — (the deploy path's own Info event covers it) |
+| Row 72 restart rejected | Warning with the gate's errors | `deployment` / **Error** — "stale-revision deployment-manager sweep self-heal failed" |
+| Storage read failed | Warning | — |
+
+The trigger is named in the message so a sweep heal is distinguishable from row 67's
+"startup reconcile self-heal".
+
+## 6. Tests
+
+`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceSweepSelfHealTests.cs` — 10 tests.
+
+Fail-before evidence (behavioral revert: the periodic timer left unarmed) — **5 fail, 5 pass**:
+
+| Test | Pre-fix |
+|---|---|
+| `Sweep_StartsEnabledInstance_WhoseConfigArrivedWithoutADeployOrReconcile` | **FAIL** — the live row-78 symptom |
+| `SweepStartedInstance_EmitsAWarningSiteEvent_NamingTheSweepTrigger` | **FAIL** |
+| `Sweep_StartsTheInstanceExactlyOnce_AcrossManySweeps` | **FAIL** |
+| `Sweep_RestartsInstance_ServingAStaleRevision` | **FAIL** — the row-72 symptom |
+| `Sweep_StaleRestartRejectedByTheCompileGate_IsAnnouncedOnceAndNotRetried` | **FAIL** |
+| `Sweep_NeverStarts_ADisabledInstance` | pass (negative guard) |
+| `Sweep_DefersToAnInFlightRedeploy` | pass (negative guard) |
+| `Sweep_DoesNotRestart_WhenStoredAndRunningRevisionsMatch` | pass (negative guard) |
+| `Sweep_IsSilent_WhenThereIsNothingToHeal` | pass (negative guard) |
+| `Sweep_DoesNothing_WhenTheIntervalIsZero` | pass (negative guard) |
+
+Every test gates on the Deployment Manager's first instance-count report before writing a config
+row, because a row written before the boot-time load has read storage is picked up by that load —
+which is precisely the path row 78 is about *not* having. Without the gate a test would silently
+prove the boot load works.