ScadaBridge/docs/plans/2026-08-15-instance-sweep-self-heal-design.md

Instance reconcile sweep — closing the config/runtime divergence gaps

Date: 2026-08-15 (§7 added 2026-08-17) Scope: Site Runtime (DeploymentManagerActor), deferred-work-register rows 78, 72 and 85 Status: implemented

---

1. The problem

The site's deployed_configurations rows are the durable truth about what should be running. The set of live Instance Actors is what is running. Nothing kept the two in agreement after boot, and there was no signal when they disagreed.

Before this change, an Instance Actor could be created from exactly three places:

1. the boot-time SQLite load in DeploymentManagerActor.PreStart;
2. an explicit deploy / enable command from central;
3. (register row 67, commit 482165cb) SiteReconciliationActor's EnsureInstanceStartedCommand, raised for enabled instances the reconcile pass itself fetched and stored.

Row 78 — configs that arrive by replication start nothing

(Medium-Low, availability; proven live 2026-08-15.)

deployed_configurations is one of LocalDb's ten replicated tables. On a replicated site pair, a config can therefore appear in a node's file with no local actor involved at all: the CDC sync applier writes the row directly.

The live sequence was:

• site-a-b booted solo; its startup reconcile fetched 4 configs from central and stored them (row 67's machinery started their actors on that node — correct).
• The pair was then restarted with a fresh site-a-a file. CDC snapshot replication seeded site-a-a's deployed_configurations within ~5 s of boot.
• That was after site-a-a's boot-time load had already read an empty table, and invisible to site-a-a's own reconcile pass: central rightly reported no gap ("0 fetched"), because central's view of the site was correct — the site really did have the configs.
• Result: enabled instances sat dead with their configs present (debug snapshot → instanceNotFound) until yet another restart.

Row 67 covers configs the reconcile fetches. This is the same gap through a side door.

Row 72 — a refreshed config does not reach the running actor

(Medium-Low, same silent-divergence class.)

ReconcileGapItem covers both missing and stale configs. For a stale item the reconcile writes the newer config via StoreDeployedConfigIfNewerAsync — a pure file write. The running Instance Actor keeps executing the previously compiled config until the next restart or redeploy, and nothing anywhere surfaced "stored revision ≠ running revision".

Row 72 was deliberately deferred out of row 67's scope because restarting a live instance carries redeploy semantics that needed their own decision. That decision is section 3.

---

2. The mechanism: one periodic sweep in the Deployment Manager singleton

A single periodic tick on the Deployment Manager singleton re-reads every durable deployed_configurations row and repairs both divergences.

InstanceSweepTick (every InstanceSweepIntervalSeconds, default 60)
  └── off-thread read of all deployed configs
        └── InstanceSweepLoaded → actor thread
              for each ENABLED stored config, unless a deploy pipeline owns the instance:
                 no running actor       → start it            (row 78)
                 running, revision ≠ stored → re-apply stored (row 72)

Ownership. The Deployment Manager is a cluster singleton scoped to the site-specific role, so the sweep runs on exactly one node — the active one — and Instance Actors stay under the singleton, which is the same invariant row 67's singleton-proxy routing preserved.

2.1 Why a sweep and not a LocalDb change hook

The register suggested either. A sweep was chosen:

• Coverage. A replication-apply hook covers exactly one arrival path, and only on the half of the fleet that runs with replication enabled (rig posture: site-a replicated, site-b/site-c deliberately not). The sweep is arrival-path-agnostic — replication seeding, a post-boot snapshot resync, an operator restoring a file by hand, and any future path all reduce to the same question: does the store agree with the runtime?
• Availability. ILocalDb exposes no change-notification, applied-rows callback or observer API — verified against ZB.MOM.WW.LocalDb 0.3.0. A hook would require a new upstream capability in the scadaproj library plus an in-process bus from the sync applier into this singleton, i.e. a cross-repo change to close a site-local gap.
• Reuse. The sweep inherits row 67's guards verbatim; the shared StartStoredInstance helper means the reconcile heal and the sweep heal produce identical bookkeeping and identical operator-facing events, differing only in the trigger they name.

Cost is one SQLite read per minute per site — the same read HandleDeploymentStateQuery already performs per central query.

2.2 Guards (all inherited or shared with row 67)

• Disabled instances are never started. A disabled instance legitimately has a config row and no actor; that is its correct steady state.
• The sweep defers to any deploy pipeline that owns the instance — a compile warm in flight (_deployWarms), a predecessor still terminating (_terminatingActorsByName), or a deployment awaiting its persistence/init join (_inFlightDeployments). This is the same InstanceIsOwnedByADeployPipeline predicate row 67's ShouldAttemptEnsureStart is now built from. It matters: a redeploy passes through a window where the predecessor has been stopped and removed from routing but the replacement does not exist yet, which an unguarded sweep would read as "enabled config, no actor" and answer by creating a child whose actor name is still registered.
• Inert until boot-time creation finishes. A tick landing between two staggered startup batches would "heal" instances the boot load is already about to create, announcing a fault on a node that is merely still starting.
• One repair attempt per (instance, revision) pair. This is what keeps a periodic self-heal from becoming a restart loop: an Instance Actor that dies during init leaves its config row in place, and a stored config the site compile gate rejects can never apply. Without the claim, either would be re-attempted — and re-announced — every tick forever. A genuinely new revision is a different key and is always attempted.
• No overlapping reads. A slow storage read cannot let ticks stack up into duplicate repairs.
• Silent when healthy. A sweep that finds nothing emits no log line above Trace and no site event. This runs on every tick of every healthy site, forever.

---

3. The row-72 decision: converge, do not merely warn

Decision. When the sweep finds a running instance whose revision no longer matches the stored row, it applies the stored config through the same per-instance apply/restart path a central redeploy uses (HandleDeploy → compile gate → stop/watch/recreate → persist), and announces both revisions in a Warning deployment site event.

Rationale.

• The stored row is what central successfully deployed. The running actor is executing a config central believes was replaced. Leaving it running is the actual fault — a site quietly serving superseded logic is worse than a controlled restart, and the operator has no way to discover it.
• The alternative (surface-only) leaves an operator with a Warning they can only act on by issuing a redeploy — the exact operation the sweep would have performed, minus the delay and minus the chance they see it.
• Reusing HandleDeploy rather than writing a second apply path means the compile gate, the warm-then-gate ordering, the redeploy-supersede semantics, the static-override and native-alarm reset, and all-or-nothing-per-instance are the ones already in production. No new semantics were invented. In particular a stop still does not cancel an in-flight script run, exactly as for a central redeploy.

Why this is safe against the ordering hazards the register worried about.

• The sweep never acts on an instance a deploy pipeline owns, so it cannot interleave with a real redeploy; and if a deploy arrives after the sweep's own deploy is queued, it is ordinary last-write-wins on the same mailbox, which is a case the deploy path already handles.
• A rejected restart is inert, not partial: RunDeployGateAndProceed returns before anything is stopped, so the running instance keeps serving its old config. It is announced once (Warning for the attempt, Error for the failure) and never retried for that revision.
• The synthetic DeployInstanceCommand replies to Self, not to NoSender — dead letters are a monitored site health metric, and a self-heal must not manufacture them.

Row 72's "nothing surfaces it" is false either way. Even in the failure path the divergence produces two operator-visible deployment site events naming both revisions.

---

4. Configuration

ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds — default 60, validated >= 0 (SiteRuntimeOptionsValidator). 0 disables the sweep, which reopens both gaps; the row-67 EnsureInstanceStartedCommand path is unaffected either way. No config change is required on any node to get the fix — the default is live.

5. Observability

Condition	Log	Site event
Nothing to repair	—	—
Row 78 start	Warning naming the instance and trigger	deployment / Warning — "started by deployment-manager sweep self-heal"
Row 72 restart attempt	Warning naming both revisions	deployment / Warning — "restarted by deployment-manager sweep self-heal — running revision X is stale, applying stored revision Y"
Row 72 restart applied	Information	— (the deploy path's own Info event covers it)
Row 72 restart rejected	Warning with the gate's errors	deployment / Error — "stale-revision deployment-manager sweep self-heal failed"
Storage read failed	Warning	—

The trigger is named in the message so a sweep heal is distinguishable from row 67's "startup reconcile self-heal".

6. Tests

tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceSweepSelfHealTests.cs — 10 tests.

Fail-before evidence (behavioral revert: the periodic timer left unarmed) — 5 fail, 5 pass:

Test	Pre-fix
Sweep_StartsEnabledInstance_WhoseConfigArrivedWithoutADeployOrReconcile	FAIL — the live row-78 symptom
SweepStartedInstance_EmitsAWarningSiteEvent_NamingTheSweepTrigger	FAIL
Sweep_StartsTheInstanceExactlyOnce_AcrossManySweeps	FAIL
Sweep_RestartsInstance_ServingAStaleRevision	FAIL — the row-72 symptom
Sweep_StaleRestartRejectedByTheCompileGate_IsAnnouncedOnceAndNotRetried	FAIL
Sweep_NeverStarts_ADisabledInstance	pass (negative guard)
Sweep_DefersToAnInFlightRedeploy	pass (negative guard)
Sweep_DoesNotRestart_WhenStoredAndRunningRevisionsMatch	pass (negative guard)
Sweep_IsSilent_WhenThereIsNothingToHeal	pass (negative guard)
Sweep_DoesNothing_WhenTheIntervalIsZero	pass (negative guard)

Every test gates on the Deployment Manager's first instance-count report before writing a config row, because a row written before the boot-time load has read storage is picked up by that load — which is precisely the path row 78 is about not having. Without the gate a test would silently prove the boot load works.

That gate is now a shared helper, tests/…/TestSupport/DeploymentManagerStartupProbe.cs, and register row 83 applied it to row 67's suite (ReconcileStartsInstanceTests), which had the same trap behind a bare Task.Delay(500). Those tests also pin InstanceSweepIntervalSeconds = 0: with the sweep armed it would independently start the same instances, and the suite would stop distinguishing the row-67 mechanism from the row-78 one. The remaining Task.Delays in that file are deliberate — they precede assertions that nothing happened, which cannot be awaited on.

---

7. Row 85 — a self-healed instance started with no DCL connection actors

(Medium-Low, availability; proven live 2026-08-16 by the row-78 close-out probe. Added 2026-08-17.)

Row 78 got the Instance Actor running. Row 85 is what was still missing underneath it: the instance came up with no data connection actors, so every tag it subscribed was answered Unknown connection: OPC PLC Simulator by the DCL manager — for 6+ minutes, until a second pair restart. Same class as row 78, one layer down.

7.1 Where connection actors actually come from

The register's suggested fix was "ensure a connection actor exists for every stored site data connection". Reading the code first changed the shape of the answer, because the premise is not how the runtime works:

• data_connection_definitions has no production reader (true when this was written; rows 89/90 gave it one on 2026-08-17 — see the residual note at the end of this section). SiteStorageService.GetAllDataConnectionDefinitionsAsync is called from exactly one place in the repo, and it is a test. Nothing loads that table at boot.
• A DCL connection actor is created only as a side effect of starting an instance, from the Connections map inside that instance's own flattened config, via EnsureDclConnections (plus the separate system-wide artifact-deploy path, HandleApplyArtifactDataConnectionsToDcl).

So "the connection row sat in the store with no actor" was right about the symptom and wrong about the mechanism: there is no store→actor path to repair. The real defect is that EnsureDclConnections was the caller's job, and only two of the four callers did it:

Path	Creates the Instance Actor	Ensured its connections (pre-fix)
Boot-time staggered batch	CreateInstanceActorBatch	✅
Central deploy / redeploy	ApplyDeployment	✅
Self-heal start (row 67 ensure-started and row 78 sweep)	StartStoredInstance	❌
Enable a disabled instance	HandleEnableResult	❌

The two misses are exactly the two paths that were added after the original pair, which is the signature of a convention that lives at call sites instead of in the funnel.

The enable path proves this was never replication-specific. The boot-time load reads only ENABLED configs, so a node that boots with an instance disabled never creates that instance's connections; enabling it afterwards hit the identical dead end on an ordinary, unreplicated site. Row 85 was reachable with no LocalDb replication anywhere in the picture — the replication sequence is just how it was found.

7.2 The fix: move the ensure into the single creation funnel

CreateInstanceActor is already documented as "the single creation funnel, so recording it here covers every path that can ever start an Instance Actor" — it is where _runningRevisions is stamped for exactly that reason. EnsureDclConnections(configJson) now sits at the top of it, and the two caller-side calls are removed.

• Placed BEFORE the already-exists guard, so the two paths that already called it keep their exact previous semantics (both called it unconditionally, before CreateInstanceActor).
• No new message, no new option, no DCL change. It reuses EnsureDclConnection's existing _createdConnections hash guard, so an unchanged connection is a no-op — no message, no log, nothing on the wire. Steady state is silent, which matters because this now runs on every instance start on every node.
• Structural, not another guard. A fifth start path cannot forget, because there is nothing left to remember.

7.3 Why the DM and not the DCL manager

The register offered either. The DCL manager was rejected: it has no storage handle and no notion of "the connections this site ought to have" — it is a router over actors it was told to create. Teaching it to read data_connection_definitions would give the DCL a second, divergent config source alongside the flattened instance configs the DM already pushes, with no arbiter when the two disagree. The DM owns _createdConnections, owns the flattened configs, and is already the only sender of CreateConnectionCommand/RemoveConnectionCommand — it is a single funnel already, and the fix keeps it that way.

7.4 Why NOT a store-level connection pass in the sweep

A sweep pass over data_connection_definitions was designed and deliberately dropped. It is recorded here because it looks obviously right and is not:

• It would not have fixed row 85. That table is written only by the artifact-deploy path, so on a site that has only ever received instance deploys it is empty — while the connections that matter live in the instances' flattened configs.
• The unguarded version is actively harmful. _createdConnections holds one hash per name. The flattened-config hash and the artifact-definition hash for the same connection can legitimately differ (artifact newer than an instance's flattened config — the supported version-skew case). A sweep re-hashing from the definitions table would flip the slot every tick against the deploy path flipping it back, recreating the connection actor and dropping its live subscriptions on a loop.
• The guarded version is near-empty. Restricting it to names absent from _createdConnections removes the ping-pong but then covers only "a definition row exists that no instance references" — where no connection actor is wanted anyway.

Residual gap, accepted and recorded: data_connection_definitions is one of LocalDb's ten replicated tables, so a connection-only artifact deploy replicates to the peer node, and only the node that received the deploy runs HandleApplyArtifactDataConnectionsToDcl — the peer keeps serving the old endpoint until something restarts it or an instance is redeployed. That is a real sibling of row 85, it is not closed here, and closing it needs the arbitration decision above (which source wins), not another sweep pass. Flagged for the deferred-work register.

Removal side: also unchanged and out of scope. RemoveConnectionCommand is sent only from the artifact-deploy reconcile; a connection deleted on the peer via replication leaves a live actor on this node. Same root cause, same arbitration question, same register entry.

CLOSED 2026-08-17 (rows 89 + 90) — and the residual above is worded on a wrong premise. The arbitration decision was made and implemented: the definitions store wins wherever it has an opinion; the flattened config is authoritative only for names the store has never owned. The guarded-version-is-near-empty objection above does not apply to that rule, because the guard is not "skip names already in _createdConnections" but "resolve the WINNER before hashing", which leaves exactly one candidate per name and makes the flip unrepresentable rather than merely avoided. Two premise corrections came out of validating it: the peer's DCL is empty, not stale (only the singleton's node ever gets a CreateConnectionCommand), so the defect is create-time on the next singleton incarnation and is not replication-specific; and HandleCreateConnection is create-only, so a changed config never reached a live connection actor on any node — that one is still open and is this file's successor's §6.1. See docs/plans/2026-08-17-dcl-connection-arbitration-decision.md.

7.5 Tests

tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SelfHealCreatesDclConnectionTests.cs — 7 tests, asserting on the CreateConnectionCommand messages the DM sends to a recording stand-in for the DCL manager (the message whose absence was row 85).

Fail-before evidence (behavioral revert: EnsureDclConnections removed from the funnel and restored at the two original call sites — i.e. the exact pre-fix code) — 4 fail, 3 pass:

Test	Pre-fix
SweepStartedInstance_AlsoCreatesItsDclConnection	FAIL — the live row-85 symptom
ReconcileEnsureStartedInstance_AlsoCreatesItsDclConnection	FAIL — row-67 path
EnabledInstance_DisabledAtBoot_AlsoCreatesItsDclConnection	FAIL — no replication involved
SweepStartedInstance_CreatesTheConnection_ExactlyOnce_AcrossManySweeps	FAIL (0, not 1)
Sweep_SendsNoConnectionCommands_WhenThereIsNothingToHeal	pass (steady-state silence)
OrdinaryDeploy_StillCreatesItsDclConnection	pass (refactor guard)
BootLoadedInstance_StillCreatesItsDclConnection	pass (refactor guard)

The two refactor guards are the point of passing both before and after: they pin that moving the call out of CreateInstanceActorBatch and ApplyDeployment did not regress the paths that already worked.