# Instance reconcile sweep — closing the config/runtime divergence gaps **Date:** 2026-08-15 (§7 added 2026-08-17) **Scope:** Site Runtime (`DeploymentManagerActor`), deferred-work-register rows **78**, **72** and **85** **Status:** implemented --- ## 1. The problem The site's `deployed_configurations` rows are the durable truth about what should be running. The set of live Instance Actors is what *is* running. Nothing kept the two in agreement after boot, and there was no signal when they disagreed. Before this change, an Instance Actor could be created from exactly three places: 1. the boot-time SQLite load in `DeploymentManagerActor.PreStart`; 2. an explicit deploy / enable command from central; 3. (register row 67, commit `482165cb`) `SiteReconciliationActor`'s `EnsureInstanceStartedCommand`, raised for enabled instances **the reconcile pass itself fetched and stored**. ### Row 78 — configs that arrive by replication start nothing *(Medium-Low, availability; proven live 2026-08-15.)* `deployed_configurations` is one of LocalDb's ten replicated tables. On a replicated site pair, a config can therefore appear in a node's file with **no local actor involved at all**: the CDC sync applier writes the row directly. The live sequence was: - site-a-b booted solo; its startup reconcile fetched 4 configs from central and stored them (row 67's machinery started their actors on that node — correct). - The pair was then restarted with a **fresh** site-a-a file. CDC snapshot replication seeded site-a-a's `deployed_configurations` within ~5 s of boot. - That was **after** site-a-a's boot-time load had already read an empty table, and **invisible** to site-a-a's own reconcile pass: central rightly reported *no gap* ("0 fetched"), because central's view of the site was correct — the site really did have the configs. - Result: enabled instances sat dead with their configs present (`debug snapshot` → `instanceNotFound`) until yet another restart. Row 67 covers configs the reconcile *fetches*. This is the same gap through a side door. ### Row 72 — a refreshed config does not reach the running actor *(Medium-Low, same silent-divergence class.)* `ReconcileGapItem` covers both *missing* and *stale* configs. For a **stale** item the reconcile writes the newer config via `StoreDeployedConfigIfNewerAsync` — a pure file write. The running Instance Actor keeps executing the **previously compiled** config until the next restart or redeploy, and nothing anywhere surfaced "stored revision ≠ running revision". Row 72 was deliberately deferred out of row 67's scope because restarting a *live* instance carries redeploy semantics that needed their own decision. That decision is section 3. --- ## 2. The mechanism: one periodic sweep in the Deployment Manager singleton A single periodic tick on the Deployment Manager singleton re-reads every durable `deployed_configurations` row and repairs both divergences. ``` InstanceSweepTick (every InstanceSweepIntervalSeconds, default 60) └── off-thread read of all deployed configs └── InstanceSweepLoaded → actor thread for each ENABLED stored config, unless a deploy pipeline owns the instance: no running actor → start it (row 78) running, revision ≠ stored → re-apply stored (row 72) ``` **Ownership.** The Deployment Manager is a cluster singleton scoped to the site-specific role, so the sweep runs on exactly one node — the active one — and Instance Actors stay under the singleton, which is the same invariant row 67's singleton-proxy routing preserved. ### 2.1 Why a sweep and not a LocalDb change hook The register suggested either. A sweep was chosen: - **Coverage.** A replication-apply hook covers exactly one arrival path, and only on the half of the fleet that runs with replication enabled (rig posture: site-a replicated, site-b/site-c deliberately not). The sweep is arrival-path-agnostic — replication seeding, a post-boot snapshot resync, an operator restoring a file by hand, and any future path all reduce to the same question: *does the store agree with the runtime?* - **Availability.** `ILocalDb` exposes **no** change-notification, applied-rows callback or observer API — verified against `ZB.MOM.WW.LocalDb` 0.3.0. A hook would require a new upstream capability in the `scadaproj` library plus an in-process bus from the sync applier into this singleton, i.e. a cross-repo change to close a site-local gap. - **Reuse.** The sweep inherits row 67's guards verbatim; the shared `StartStoredInstance` helper means the reconcile heal and the sweep heal produce identical bookkeeping and identical operator-facing events, differing only in the trigger they name. Cost is one SQLite read per minute per site — the same read `HandleDeploymentStateQuery` already performs per central query. ### 2.2 Guards (all inherited or shared with row 67) - **Disabled instances are never started.** A disabled instance legitimately has a config row and no actor; that is its correct steady state. - **The sweep defers to any deploy pipeline that owns the instance** — a compile warm in flight (`_deployWarms`), a predecessor still terminating (`_terminatingActorsByName`), or a deployment awaiting its persistence/init join (`_inFlightDeployments`). This is the same `InstanceIsOwnedByADeployPipeline` predicate row 67's `ShouldAttemptEnsureStart` is now built from. It matters: a redeploy passes through a window where the predecessor has been stopped and removed from routing but the replacement does not exist yet, which an unguarded sweep would read as "enabled config, no actor" and answer by creating a child whose actor name is still registered. - **Inert until boot-time creation finishes.** A tick landing between two staggered startup batches would "heal" instances the boot load is already about to create, announcing a fault on a node that is merely still starting. - **One repair attempt per (instance, revision) pair.** This is what keeps a *periodic* self-heal from becoming a restart loop: an Instance Actor that dies during init leaves its config row in place, and a stored config the site compile gate rejects can never apply. Without the claim, either would be re-attempted — and re-announced — every tick forever. A genuinely new revision is a different key and is always attempted. - **No overlapping reads.** A slow storage read cannot let ticks stack up into duplicate repairs. - **Silent when healthy.** A sweep that finds nothing emits no log line above Trace and no site event. This runs on every tick of every healthy site, forever. --- ## 3. The row-72 decision: converge, do not merely warn **Decision.** When the sweep finds a running instance whose revision no longer matches the stored row, it **applies the stored config through the same per-instance apply/restart path a central redeploy uses** (`HandleDeploy` → compile gate → stop/watch/recreate → persist), and announces both revisions in a Warning `deployment` site event. **Rationale.** - The stored row **is** what central successfully deployed. The running actor is executing a config central believes was replaced. Leaving it running is the actual fault — a site quietly serving superseded logic is worse than a controlled restart, and the operator has no way to discover it. - The alternative (surface-only) leaves an operator with a Warning they can only act on by issuing a redeploy — the exact operation the sweep would have performed, minus the delay and minus the chance they see it. - Reusing `HandleDeploy` rather than writing a second apply path means the compile gate, the warm-then-gate ordering, the redeploy-supersede semantics, the static-override and native-alarm reset, and all-or-nothing-per-instance are the ones already in production. No new semantics were invented. In particular a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy. **Why this is safe against the ordering hazards the register worried about.** - The sweep never acts on an instance a deploy pipeline owns, so it cannot interleave with a real redeploy; and if a deploy arrives *after* the sweep's own deploy is queued, it is ordinary last-write-wins on the same mailbox, which is a case the deploy path already handles. - A rejected restart is inert, not partial: `RunDeployGateAndProceed` returns **before** anything is stopped, so the running instance keeps serving its old config. It is announced once (Warning for the attempt, Error for the failure) and never retried for that revision. - The synthetic `DeployInstanceCommand` replies to `Self`, not to `NoSender` — dead letters are a monitored site health metric, and a self-heal must not manufacture them. **Row 72's "nothing surfaces it" is false either way.** Even in the failure path the divergence produces two operator-visible `deployment` site events naming both revisions. --- ## 4. Configuration `ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds` — default **60**, validated `>= 0` (`SiteRuntimeOptionsValidator`). **`0` disables the sweep**, which reopens both gaps; the row-67 `EnsureInstanceStartedCommand` path is unaffected either way. No config change is required on any node to get the fix — the default is live. ## 5. Observability | Condition | Log | Site event | |---|---|---| | Nothing to repair | — | — | | Row 78 start | Warning naming the instance and trigger | `deployment` / **Warning** — "started by deployment-manager sweep self-heal" | | Row 72 restart attempt | Warning naming both revisions | `deployment` / **Warning** — "restarted by deployment-manager sweep self-heal — running revision X is stale, applying stored revision Y" | | Row 72 restart applied | Information | — (the deploy path's own Info event covers it) | | Row 72 restart rejected | Warning with the gate's errors | `deployment` / **Error** — "stale-revision deployment-manager sweep self-heal failed" | | Storage read failed | Warning | — | The trigger is named in the message so a sweep heal is distinguishable from row 67's "startup reconcile self-heal". ## 6. Tests `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceSweepSelfHealTests.cs` — 10 tests. Fail-before evidence (behavioral revert: the periodic timer left unarmed) — **5 fail, 5 pass**: | Test | Pre-fix | |---|---| | `Sweep_StartsEnabledInstance_WhoseConfigArrivedWithoutADeployOrReconcile` | **FAIL** — the live row-78 symptom | | `SweepStartedInstance_EmitsAWarningSiteEvent_NamingTheSweepTrigger` | **FAIL** | | `Sweep_StartsTheInstanceExactlyOnce_AcrossManySweeps` | **FAIL** | | `Sweep_RestartsInstance_ServingAStaleRevision` | **FAIL** — the row-72 symptom | | `Sweep_StaleRestartRejectedByTheCompileGate_IsAnnouncedOnceAndNotRetried` | **FAIL** | | `Sweep_NeverStarts_ADisabledInstance` | pass (negative guard) | | `Sweep_DefersToAnInFlightRedeploy` | pass (negative guard) | | `Sweep_DoesNotRestart_WhenStoredAndRunningRevisionsMatch` | pass (negative guard) | | `Sweep_IsSilent_WhenThereIsNothingToHeal` | pass (negative guard) | | `Sweep_DoesNothing_WhenTheIntervalIsZero` | pass (negative guard) | Every test gates on the Deployment Manager's first instance-count report before writing a config row, because a row written before the boot-time load has read storage is picked up by that load — which is precisely the path row 78 is about *not* having. Without the gate a test would silently prove the boot load works. That gate is now a shared helper, `tests/…/TestSupport/DeploymentManagerStartupProbe.cs`, and **register row 83** applied it to row 67's suite (`ReconcileStartsInstanceTests`), which had the same trap behind a bare `Task.Delay(500)`. Those tests also pin `InstanceSweepIntervalSeconds = 0`: with the sweep armed it would independently start the same instances, and the suite would stop distinguishing the row-67 mechanism from the row-78 one. The remaining `Task.Delay`s in that file are deliberate — they precede assertions that *nothing happened*, which cannot be awaited on. --- ## 7. Row 85 — a self-healed instance started with no DCL connection actors *(Medium-Low, availability; proven live 2026-08-16 by the row-78 close-out probe. Added 2026-08-17.)* Row 78 got the Instance Actor running. Row 85 is what was still missing underneath it: the instance came up with **no data connection actors**, so every tag it subscribed was answered `Unknown connection: OPC PLC Simulator` by the DCL manager — for 6+ minutes, until a second pair restart. Same class as row 78, one layer down. ### 7.1 Where connection actors actually come from The register's suggested fix was "ensure a connection actor exists for every stored site data connection". Reading the code first changed the shape of the answer, because the premise is not how the runtime works: - **`data_connection_definitions` has no production reader** *(true when this was written; rows 89/90 gave it one on 2026-08-17 — see the residual note at the end of this section)*. `SiteStorageService.GetAllDataConnectionDefinitionsAsync` is called from exactly one place in the repo, and it is a test. Nothing loads that table at boot. - A DCL connection actor is created **only as a side effect of starting an instance**, from the `Connections` map inside that instance's *own flattened config*, via `EnsureDclConnections` (plus the separate system-wide artifact-deploy path, `HandleApplyArtifactDataConnectionsToDcl`). So "the connection row sat in the store with no actor" was right about the symptom and wrong about the mechanism: there is no store→actor path to repair. The real defect is that `EnsureDclConnections` was **the caller's job, and only two of the four callers did it**: | Path | Creates the Instance Actor | Ensured its connections (pre-fix) | |---|---|---| | Boot-time staggered batch | `CreateInstanceActorBatch` | ✅ | | Central deploy / redeploy | `ApplyDeployment` | ✅ | | **Self-heal start** (row 67 ensure-started **and** row 78 sweep) | `StartStoredInstance` | ❌ | | **Enable** a disabled instance | `HandleEnableResult` | ❌ | The two misses are exactly the two paths that were added *after* the original pair, which is the signature of a convention that lives at call sites instead of in the funnel. **The enable path proves this was never replication-specific.** The boot-time load reads only ENABLED configs, so a node that boots with an instance disabled never creates that instance's connections; enabling it afterwards hit the identical dead end on an ordinary, unreplicated site. Row 85 was reachable with no LocalDb replication anywhere in the picture — the replication sequence is just how it was found. ### 7.2 The fix: move the ensure into the single creation funnel `CreateInstanceActor` is already documented as *"the single creation funnel, so recording it here covers every path that can ever start an Instance Actor"* — it is where `_runningRevisions` is stamped for exactly that reason. `EnsureDclConnections(configJson)` now sits at the top of it, and the two caller-side calls are removed. - **Placed BEFORE the already-exists guard**, so the two paths that already called it keep their exact previous semantics (both called it unconditionally, before `CreateInstanceActor`). - **No new message, no new option, no DCL change.** It reuses `EnsureDclConnection`'s existing `_createdConnections` hash guard, so an unchanged connection is a no-op — no message, no log, nothing on the wire. Steady state is silent, which matters because this now runs on every instance start on every node. - **Structural, not another guard.** A fifth start path cannot forget, because there is nothing left to remember. ### 7.3 Why the DM and not the DCL manager The register offered either. The DCL manager was rejected: it has no storage handle and no notion of "the connections this site ought to have" — it is a router over actors it was told to create. Teaching it to read `data_connection_definitions` would give the DCL a **second, divergent config source** alongside the flattened instance configs the DM already pushes, with no arbiter when the two disagree. The DM owns `_createdConnections`, owns the flattened configs, and is already the only sender of `CreateConnectionCommand`/`RemoveConnectionCommand` — it is a single funnel already, and the fix keeps it that way. ### 7.4 Why NOT a store-level connection pass in the sweep A sweep pass over `data_connection_definitions` was designed and **deliberately dropped**. It is recorded here because it looks obviously right and is not: - **It would not have fixed row 85.** That table is written only by the artifact-deploy path, so on a site that has only ever received instance deploys it is empty — while the connections that matter live in the instances' flattened configs. - **The unguarded version is actively harmful.** `_createdConnections` holds one hash per name. The flattened-config hash and the artifact-definition hash for the same connection can legitimately differ (artifact newer than an instance's flattened config — the supported version-skew case). A sweep re-hashing from the definitions table would flip the slot every tick against the deploy path flipping it back, **recreating the connection actor and dropping its live subscriptions on a loop**. - **The guarded version is near-empty.** Restricting it to names absent from `_createdConnections` removes the ping-pong but then covers only "a definition row exists that no instance references" — where no connection actor is wanted anyway. **Residual gap, accepted and recorded:** `data_connection_definitions` *is* one of LocalDb's ten replicated tables, so a **connection-only** artifact deploy replicates to the peer node, and only the node that received the deploy runs `HandleApplyArtifactDataConnectionsToDcl` — the peer keeps serving the old endpoint until something restarts it or an instance is redeployed. That is a real sibling of row 85, it is **not** closed here, and closing it needs the arbitration decision above (which source wins), not another sweep pass. Flagged for the deferred-work register. **Removal side:** also unchanged and out of scope. `RemoveConnectionCommand` is sent only from the artifact-deploy reconcile; a connection deleted on the peer via replication leaves a live actor on this node. Same root cause, same arbitration question, same register entry. > **CLOSED 2026-08-17 (rows 89 + 90) — and the residual above is worded on a wrong premise.** > The arbitration decision was made and implemented: **the definitions store wins wherever it has > an opinion; the flattened config is authoritative only for names the store has never owned.** The > guarded-version-is-near-empty objection above does not apply to that rule, because the guard is > not "skip names already in `_createdConnections`" but "resolve the WINNER before hashing", which > leaves exactly one candidate per name and makes the flip unrepresentable rather than merely > avoided. Two premise corrections came out of validating it: the peer's DCL is **empty**, not > stale (only the singleton's node ever gets a `CreateConnectionCommand`), so the defect is > create-time on the next singleton incarnation and is **not replication-specific**; and > `HandleCreateConnection` is create-only, so a changed config never reached a *live* connection > actor on any node — that one is still open and is this file's successor's §6.1. See > `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`. ### 7.5 Tests `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SelfHealCreatesDclConnectionTests.cs` — 7 tests, asserting on the `CreateConnectionCommand` messages the DM sends to a recording stand-in for the DCL manager (the message whose absence *was* row 85). Fail-before evidence (behavioral revert: `EnsureDclConnections` removed from the funnel and restored at the two original call sites — i.e. the exact pre-fix code) — **4 fail, 3 pass**: | Test | Pre-fix | |---|---| | `SweepStartedInstance_AlsoCreatesItsDclConnection` | **FAIL** — the live row-85 symptom | | `ReconcileEnsureStartedInstance_AlsoCreatesItsDclConnection` | **FAIL** — row-67 path | | `EnabledInstance_DisabledAtBoot_AlsoCreatesItsDclConnection` | **FAIL** — no replication involved | | `SweepStartedInstance_CreatesTheConnection_ExactlyOnce_AcrossManySweeps` | **FAIL** (0, not 1) | | `Sweep_SendsNoConnectionCommands_WhenThereIsNothingToHeal` | pass (steady-state silence) | | `OrdinaryDeploy_StillCreatesItsDclConnection` | pass (refactor guard) | | `BootLoadedInstance_StillCreatesItsDclConnection` | pass (refactor guard) | The two refactor guards are the point of passing both before and after: they pin that moving the call out of `CreateInstanceActorBatch` and `ApplyDeployment` did not regress the paths that already worked.