ScadaBridge/docs/plans/2026-08-15-instance-sweep-self-heal-design.md

# Instance reconcile sweep — closing the config/runtime divergence gaps

**Date:** 2026-08-15 (§7 added 2026-08-17)
**Scope:** Site Runtime (`DeploymentManagerActor`), deferred-work-register rows **78**, **72**
and **85**
**Status:** implemented

---

## 1. The problem

The site's `deployed_configurations` rows are the durable truth about what should be running.
The set of live Instance Actors is what *is* running. Nothing kept the two in agreement after
boot, and there was no signal when they disagreed.

Before this change, an Instance Actor could be created from exactly three places:

1. the boot-time SQLite load in `DeploymentManagerActor.PreStart`;
2. an explicit deploy / enable command from central;
3. (register row 67, commit `482165cb`) `SiteReconciliationActor`'s `EnsureInstanceStartedCommand`,
   raised for enabled instances **the reconcile pass itself fetched and stored**.

### Row 78 — configs that arrive by replication start nothing

*(Medium-Low, availability; proven live 2026-08-15.)*

`deployed_configurations` is one of LocalDb's ten replicated tables. On a replicated site pair,
a config can therefore appear in a node's file with **no local actor involved at all**: the CDC
sync applier writes the row directly.

The live sequence was:

- site-a-b booted solo; its startup reconcile fetched 4 configs from central and stored them
  (row 67's machinery started their actors on that node — correct).
- The pair was then restarted with a **fresh** site-a-a file. CDC snapshot replication seeded
  site-a-a's `deployed_configurations` within ~5 s of boot.
- That was **after** site-a-a's boot-time load had already read an empty table, and **invisible**
  to site-a-a's own reconcile pass: central rightly reported *no gap* ("0 fetched"), because
  central's view of the site was correct — the site really did have the configs.
- Result: enabled instances sat dead with their configs present (`debug snapshot` →
  `instanceNotFound`) until yet another restart.

Row 67 covers configs the reconcile *fetches*. This is the same gap through a side door.

### Row 72 — a refreshed config does not reach the running actor

*(Medium-Low, same silent-divergence class.)*

`ReconcileGapItem` covers both *missing* and *stale* configs. For a **stale** item the reconcile
writes the newer config via `StoreDeployedConfigIfNewerAsync` — a pure file write. The running
Instance Actor keeps executing the **previously compiled** config until the next restart or
redeploy, and nothing anywhere surfaced "stored revision ≠ running revision".

Row 72 was deliberately deferred out of row 67's scope because restarting a *live* instance
carries redeploy semantics that needed their own decision. That decision is section 3.

---

## 2. The mechanism: one periodic sweep in the Deployment Manager singleton

A single periodic tick on the Deployment Manager singleton re-reads every durable
`deployed_configurations` row and repairs both divergences.

```
InstanceSweepTick (every InstanceSweepIntervalSeconds, default 60)
  └── off-thread read of all deployed configs
        └── InstanceSweepLoaded → actor thread
              for each ENABLED stored config, unless a deploy pipeline owns the instance:
                 no running actor       → start it            (row 78)
                 running, revision ≠ stored → re-apply stored (row 72)
```

**Ownership.** The Deployment Manager is a cluster singleton scoped to the site-specific role, so
the sweep runs on exactly one node — the active one — and Instance Actors stay under the
singleton, which is the same invariant row 67's singleton-proxy routing preserved.

### 2.1 Why a sweep and not a LocalDb change hook

The register suggested either. A sweep was chosen:

- **Coverage.** A replication-apply hook covers exactly one arrival path, and only on the half of
  the fleet that runs with replication enabled (rig posture: site-a replicated, site-b/site-c
  deliberately not). The sweep is arrival-path-agnostic — replication seeding, a post-boot
  snapshot resync, an operator restoring a file by hand, and any future path all reduce to the
  same question: *does the store agree with the runtime?*
- **Availability.** `ILocalDb` exposes **no** change-notification, applied-rows callback or
  observer API — verified against `ZB.MOM.WW.LocalDb` 0.3.0. A hook would require a new upstream
  capability in the `scadaproj` library plus an in-process bus from the sync applier into this
  singleton, i.e. a cross-repo change to close a site-local gap.
- **Reuse.** The sweep inherits row 67's guards verbatim; the shared `StartStoredInstance` helper
  means the reconcile heal and the sweep heal produce identical bookkeeping and identical
  operator-facing events, differing only in the trigger they name.

Cost is one SQLite read per minute per site — the same read `HandleDeploymentStateQuery` already
performs per central query.

### 2.2 Guards (all inherited or shared with row 67)

- **Disabled instances are never started.** A disabled instance legitimately has a config row and
  no actor; that is its correct steady state.
- **The sweep defers to any deploy pipeline that owns the instance** — a compile warm in flight
  (`_deployWarms`), a predecessor still terminating (`_terminatingActorsByName`), or a deployment
  awaiting its persistence/init join (`_inFlightDeployments`). This is the same
  `InstanceIsOwnedByADeployPipeline` predicate row 67's `ShouldAttemptEnsureStart` is now built
  from. It matters: a redeploy passes through a window where the predecessor has been stopped and
  removed from routing but the replacement does not exist yet, which an unguarded sweep would read
  as "enabled config, no actor" and answer by creating a child whose actor name is still
  registered.
- **Inert until boot-time creation finishes.** A tick landing between two staggered startup
  batches would "heal" instances the boot load is already about to create, announcing a fault on
  a node that is merely still starting.
- **One repair attempt per (instance, revision) pair.** This is what keeps a *periodic* self-heal
  from becoming a restart loop: an Instance Actor that dies during init leaves its config row in
  place, and a stored config the site compile gate rejects can never apply. Without the claim,
  either would be re-attempted — and re-announced — every tick forever. A genuinely new revision
  is a different key and is always attempted.
- **No overlapping reads.** A slow storage read cannot let ticks stack up into duplicate repairs.
- **Silent when healthy.** A sweep that finds nothing emits no log line above Trace and no site
  event. This runs on every tick of every healthy site, forever.

---

## 3. The row-72 decision: converge, do not merely warn

**Decision.** When the sweep finds a running instance whose revision no longer matches the stored
row, it **applies the stored config through the same per-instance apply/restart path a central
redeploy uses** (`HandleDeploy` → compile gate → stop/watch/recreate → persist), and announces
both revisions in a Warning `deployment` site event.

**Rationale.**

- The stored row **is** what central successfully deployed. The running actor is executing a
  config central believes was replaced. Leaving it running is the actual fault — a site quietly
  serving superseded logic is worse than a controlled restart, and the operator has no way to
  discover it.
- The alternative (surface-only) leaves an operator with a Warning they can only act on by
  issuing a redeploy — the exact operation the sweep would have performed, minus the delay and
  minus the chance they see it.
- Reusing `HandleDeploy` rather than writing a second apply path means the compile gate, the
  warm-then-gate ordering, the redeploy-supersede semantics, the static-override and
  native-alarm reset, and all-or-nothing-per-instance are the ones already in production. No new
  semantics were invented. In particular a stop still does **not** cancel an in-flight script
  run, exactly as for a central redeploy.

**Why this is safe against the ordering hazards the register worried about.**

- The sweep never acts on an instance a deploy pipeline owns, so it cannot interleave with a real
  redeploy; and if a deploy arrives *after* the sweep's own deploy is queued, it is ordinary
  last-write-wins on the same mailbox, which is a case the deploy path already handles.
- A rejected restart is inert, not partial: `RunDeployGateAndProceed` returns **before** anything
  is stopped, so the running instance keeps serving its old config. It is announced once
  (Warning for the attempt, Error for the failure) and never retried for that revision.
- The synthetic `DeployInstanceCommand` replies to `Self`, not to `NoSender` — dead letters are a
  monitored site health metric, and a self-heal must not manufacture them.

**Row 72's "nothing surfaces it" is false either way.** Even in the failure path the divergence
produces two operator-visible `deployment` site events naming both revisions.

---

## 4. Configuration

`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds` — default **60**, validated `>= 0`
(`SiteRuntimeOptionsValidator`). **`0` disables the sweep**, which reopens both gaps; the row-67
`EnsureInstanceStartedCommand` path is unaffected either way. No config change is required on any
node to get the fix — the default is live.

## 5. Observability

| Condition | Log | Site event |
|---|---|---|
| Nothing to repair | — | — |
| Row 78 start | Warning naming the instance and trigger | `deployment` / **Warning** — "started by deployment-manager sweep self-heal" |
| Row 72 restart attempt | Warning naming both revisions | `deployment` / **Warning** — "restarted by deployment-manager sweep self-heal — running revision X is stale, applying stored revision Y" |
| Row 72 restart applied | Information | — (the deploy path's own Info event covers it) |
| Row 72 restart rejected | Warning with the gate's errors | `deployment` / **Error** — "stale-revision deployment-manager sweep self-heal failed" |
| Storage read failed | Warning | — |

The trigger is named in the message so a sweep heal is distinguishable from row 67's
"startup reconcile self-heal".

## 6. Tests

`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceSweepSelfHealTests.cs` — 10 tests.

Fail-before evidence (behavioral revert: the periodic timer left unarmed) — **5 fail, 5 pass**:

| Test | Pre-fix |
|---|---|
| `Sweep_StartsEnabledInstance_WhoseConfigArrivedWithoutADeployOrReconcile` | **FAIL** — the live row-78 symptom |
| `SweepStartedInstance_EmitsAWarningSiteEvent_NamingTheSweepTrigger` | **FAIL** |
| `Sweep_StartsTheInstanceExactlyOnce_AcrossManySweeps` | **FAIL** |
| `Sweep_RestartsInstance_ServingAStaleRevision` | **FAIL** — the row-72 symptom |
| `Sweep_StaleRestartRejectedByTheCompileGate_IsAnnouncedOnceAndNotRetried` | **FAIL** |
| `Sweep_NeverStarts_ADisabledInstance` | pass (negative guard) |
| `Sweep_DefersToAnInFlightRedeploy` | pass (negative guard) |
| `Sweep_DoesNotRestart_WhenStoredAndRunningRevisionsMatch` | pass (negative guard) |
| `Sweep_IsSilent_WhenThereIsNothingToHeal` | pass (negative guard) |
| `Sweep_DoesNothing_WhenTheIntervalIsZero` | pass (negative guard) |

Every test gates on the Deployment Manager's first instance-count report before writing a config
row, because a row written before the boot-time load has read storage is picked up by that load —
which is precisely the path row 78 is about *not* having. Without the gate a test would silently
prove the boot load works.

That gate is now a shared helper,
`tests/…/TestSupport/DeploymentManagerStartupProbe.cs`, and **register row 83** applied it to
row 67's suite (`ReconcileStartsInstanceTests`), which had the same trap behind a bare
`Task.Delay(500)`. Those tests also pin `InstanceSweepIntervalSeconds = 0`: with the sweep armed
it would independently start the same instances, and the suite would stop distinguishing the
row-67 mechanism from the row-78 one. The remaining `Task.Delay`s in that file are deliberate —
they precede assertions that *nothing happened*, which cannot be awaited on.

---

## 7. Row 85 — a self-healed instance started with no DCL connection actors

*(Medium-Low, availability; proven live 2026-08-16 by the row-78 close-out probe. Added
2026-08-17.)*

Row 78 got the Instance Actor running. Row 85 is what was still missing underneath it: the
instance came up with **no data connection actors**, so every tag it subscribed was answered
`Unknown connection: OPC PLC Simulator` by the DCL manager — for 6+ minutes, until a second pair
restart. Same class as row 78, one layer down.

### 7.1 Where connection actors actually come from

The register's suggested fix was "ensure a connection actor exists for every stored site data
connection". Reading the code first changed the shape of the answer, because the premise is not
how the runtime works:

- **`data_connection_definitions` has no production reader** *(true when this was written; rows
  89/90 gave it one on 2026-08-17 — see the residual note at the end of this section)*.
  `SiteStorageService.GetAllDataConnectionDefinitionsAsync` is called from exactly one place in
  the repo, and it is a test. Nothing loads that table at boot.
- A DCL connection actor is created **only as a side effect of starting an instance**, from the
  `Connections` map inside that instance's *own flattened config*, via `EnsureDclConnections`
  (plus the separate system-wide artifact-deploy path, `HandleApplyArtifactDataConnectionsToDcl`).

So "the connection row sat in the store with no actor" was right about the symptom and wrong
about the mechanism: there is no store→actor path to repair. The real defect is that
`EnsureDclConnections` was **the caller's job, and only two of the four callers did it**:

| Path | Creates the Instance Actor | Ensured its connections (pre-fix) |
|---|---|---|
| Boot-time staggered batch | `CreateInstanceActorBatch` | ✅ |
| Central deploy / redeploy | `ApplyDeployment` | ✅ |
| **Self-heal start** (row 67 ensure-started **and** row 78 sweep) | `StartStoredInstance` | ❌ |
| **Enable** a disabled instance | `HandleEnableResult` | ❌ |

The two misses are exactly the two paths that were added *after* the original pair, which is the
signature of a convention that lives at call sites instead of in the funnel.

**The enable path proves this was never replication-specific.** The boot-time load reads only
ENABLED configs, so a node that boots with an instance disabled never creates that instance's
connections; enabling it afterwards hit the identical dead end on an ordinary, unreplicated
site. Row 85 was reachable with no LocalDb replication anywhere in the picture — the replication
sequence is just how it was found.

### 7.2 The fix: move the ensure into the single creation funnel

`CreateInstanceActor` is already documented as *"the single creation funnel, so recording it here
covers every path that can ever start an Instance Actor"* — it is where `_runningRevisions` is
stamped for exactly that reason. `EnsureDclConnections(configJson)` now sits at the top of it,
and the two caller-side calls are removed.

- **Placed BEFORE the already-exists guard**, so the two paths that already called it keep their
  exact previous semantics (both called it unconditionally, before `CreateInstanceActor`).
- **No new message, no new option, no DCL change.** It reuses `EnsureDclConnection`'s existing
  `_createdConnections` hash guard, so an unchanged connection is a no-op — no message, no log,
  nothing on the wire. Steady state is silent, which matters because this now runs on every
  instance start on every node.
- **Structural, not another guard.** A fifth start path cannot forget, because there is nothing
  left to remember.

### 7.3 Why the DM and not the DCL manager

The register offered either. The DCL manager was rejected: it has no storage handle and no notion
of "the connections this site ought to have" — it is a router over actors it was told to create.
Teaching it to read `data_connection_definitions` would give the DCL a **second, divergent config
source** alongside the flattened instance configs the DM already pushes, with no arbiter when the
two disagree. The DM owns `_createdConnections`, owns the flattened configs, and is already the
only sender of `CreateConnectionCommand`/`RemoveConnectionCommand` — it is a single funnel
already, and the fix keeps it that way.

### 7.4 Why NOT a store-level connection pass in the sweep

A sweep pass over `data_connection_definitions` was designed and **deliberately dropped**. It is
recorded here because it looks obviously right and is not:

- **It would not have fixed row 85.** That table is written only by the artifact-deploy path, so
  on a site that has only ever received instance deploys it is empty — while the connections that
  matter live in the instances' flattened configs.
- **The unguarded version is actively harmful.** `_createdConnections` holds one hash per name.
  The flattened-config hash and the artifact-definition hash for the same connection can
  legitimately differ (artifact newer than an instance's flattened config — the supported
  version-skew case). A sweep re-hashing from the definitions table would flip the slot every
  tick against the deploy path flipping it back, **recreating the connection actor and dropping
  its live subscriptions on a loop**.
- **The guarded version is near-empty.** Restricting it to names absent from `_createdConnections`
  removes the ping-pong but then covers only "a definition row exists that no instance
  references" — where no connection actor is wanted anyway.

**Residual gap, accepted and recorded:** `data_connection_definitions` *is* one of LocalDb's ten
replicated tables, so a **connection-only** artifact deploy replicates to the peer node, and only
the node that received the deploy runs `HandleApplyArtifactDataConnectionsToDcl` — the peer keeps
serving the old endpoint until something restarts it or an instance is redeployed. That is a real
sibling of row 85, it is **not** closed here, and closing it needs the arbitration decision above
(which source wins), not another sweep pass. Flagged for the deferred-work register.

**Removal side:** also unchanged and out of scope. `RemoveConnectionCommand` is sent only from the
artifact-deploy reconcile; a connection deleted on the peer via replication leaves a live actor on
this node. Same root cause, same arbitration question, same register entry.

> **CLOSED 2026-08-17 (rows 89 + 90) — and the residual above is worded on a wrong premise.**
> The arbitration decision was made and implemented: **the definitions store wins wherever it has
> an opinion; the flattened config is authoritative only for names the store has never owned.** The
> guarded-version-is-near-empty objection above does not apply to that rule, because the guard is
> not "skip names already in `_createdConnections`" but "resolve the WINNER before hashing", which
> leaves exactly one candidate per name and makes the flip unrepresentable rather than merely
> avoided. Two premise corrections came out of validating it: the peer's DCL is **empty**, not
> stale (only the singleton's node ever gets a `CreateConnectionCommand`), so the defect is
> create-time on the next singleton incarnation and is **not replication-specific**; and
> `HandleCreateConnection` is create-only, so a changed config never reached a *live* connection
> actor on any node — that one is still open and is this file's successor's §6.1. See
> `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.

### 7.5 Tests

`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SelfHealCreatesDclConnectionTests.cs` —
7 tests, asserting on the `CreateConnectionCommand` messages the DM sends to a recording stand-in
for the DCL manager (the message whose absence *was* row 85).

Fail-before evidence (behavioral revert: `EnsureDclConnections` removed from the funnel and
restored at the two original call sites — i.e. the exact pre-fix code) — **4 fail, 3 pass**:

| Test | Pre-fix |
|---|---|
| `SweepStartedInstance_AlsoCreatesItsDclConnection` | **FAIL** — the live row-85 symptom |
| `ReconcileEnsureStartedInstance_AlsoCreatesItsDclConnection` | **FAIL** — row-67 path |
| `EnabledInstance_DisabledAtBoot_AlsoCreatesItsDclConnection` | **FAIL** — no replication involved |
| `SweepStartedInstance_CreatesTheConnection_ExactlyOnce_AcrossManySweeps` | **FAIL** (0, not 1) |
| `Sweep_SendsNoConnectionCommands_WhenThereIsNothingToHeal` | pass (steady-state silence) |
| `OrdinaryDeploy_StillCreatesItsDclConnection` | pass (refactor guard) |
| `BootLoadedInstance_StillCreatesItsDclConnection` | pass (refactor guard) |

The two refactor guards are the point of passing both before and after: they pin that moving the
call out of `CreateInstanceActorBatch` and `ApplyDeployment` did not regress the paths that
already worked.