Merge rows 78+72: DeploymentManager singleton instance sweep — starts replication-seeded enabled instances and converges stale revisions through the real HandleDeploy path; one repair attempt per (instance, revision)
This commit is contained in:
Joseph Doherty
@@ -27,6 +27,15 @@ The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`:
|
||||
|
||||
The singleton reads all deployed configurations from SQLite in `PreStart`, compiles shared scripts off-thread, and then creates `InstanceActor` children in batches. The default batch size is 20, with a 100 ms delay between batches (`StartupBatchSize` and `StartupBatchDelayMs` in `SiteRuntimeOptions`). Without staggering, 500 instances each subscribing to OPC UA tags simultaneously would produce a reconnection storm that overwhelms the OPC UA server.
|
||||
|
||||
### Instance reconcile sweep
|
||||
|
||||
The staggered startup above reads storage exactly once. Anything that writes a `deployed_configurations` row afterwards without going through a deploy command therefore changed the store but not the runtime — and nothing said so. Two real cases (deferred register rows **78** and **72**):
|
||||
|
||||
- a config seeded by **LocalDb CDC replication** (a fresh standby filled in from its peer, or a snapshot resync) landed after the boot load and was invisible to the startup reconcile, so its enabled instances never started;
|
||||
- a config the startup reconcile **refreshed** because it was stale left the running `InstanceActor` serving the previously compiled revision.
|
||||
|
||||
`DeploymentManagerActor` therefore runs a periodic sweep (`InstanceSweepIntervalSeconds`, default 60; `0` disables) that re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config either starts a missing `InstanceActor` or — when the running one's revision no longer matches the stored one — re-applies the stored config **through the ordinary redeploy path**, so nothing about redeploy semantics is re-implemented. It defers to any in-flight deploy/warm/termination, never starts a disabled instance, stays inert until boot-time creation has finished, makes one repair attempt per (instance, revision) pair so it can never become a restart loop, and is completely silent when there is nothing to repair. Repairs emit Warning `deployment` site events naming the trigger (and, for a stale-revision restart, both revisions). Design note: `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
|
||||
|
||||
### Actor supervision and lifetimes
|
||||
|
||||
| Actor | Kind | Supervises children with | On exception |
|
||||
@@ -265,6 +274,7 @@ All options live in the `ScadaBridge:SiteRuntime` section, bound to `SiteRuntime
|
||||
| `MaxConcurrentRunsPerScript` | `4` | Runs in flight (queued or executing) per script/alarm on-trigger script before the newest is shed (WP3.1) |
|
||||
| `MirroredAlarmCapPerSource` | `1000` | Maximum mirrored conditions per `NativeAlarmActor` source binding before oldest is dropped and logged |
|
||||
| `NativeAlarmRetryIntervalMs` | `5000` | Milliseconds before retrying a failed native alarm subscription |
|
||||
| `InstanceSweepIntervalSeconds` | `60` | Interval of the singleton's instance reconcile sweep (register rows 78 + 72); `0` disables it — see Instance reconcile sweep |
|
||||
|
||||
The SQLite connection string is passed directly to `AddSiteRuntime(connectionString)` in the host composition root and is not part of `SiteRuntimeOptions`.
|
||||
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
# Instance reconcile sweep — closing the config/runtime divergence gaps
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Scope:** Site Runtime (`DeploymentManagerActor`), deferred-work-register rows **78** and **72**
|
||||
**Status:** implemented
|
||||
|
||||
---
|
||||
|
||||
## 1. The problem
|
||||
|
||||
The site's `deployed_configurations` rows are the durable truth about what should be running.
|
||||
The set of live Instance Actors is what *is* running. Nothing kept the two in agreement after
|
||||
boot, and there was no signal when they disagreed.
|
||||
|
||||
Before this change, an Instance Actor could be created from exactly three places:
|
||||
|
||||
1. the boot-time SQLite load in `DeploymentManagerActor.PreStart`;
|
||||
2. an explicit deploy / enable command from central;
|
||||
3. (register row 67, commit `482165cb`) `SiteReconciliationActor`'s `EnsureInstanceStartedCommand`,
|
||||
raised for enabled instances **the reconcile pass itself fetched and stored**.
|
||||
|
||||
### Row 78 — configs that arrive by replication start nothing
|
||||
|
||||
*(Medium-Low, availability; proven live 2026-08-15.)*
|
||||
|
||||
`deployed_configurations` is one of LocalDb's ten replicated tables. On a replicated site pair,
|
||||
a config can therefore appear in a node's file with **no local actor involved at all**: the CDC
|
||||
sync applier writes the row directly.
|
||||
|
||||
The live sequence was:
|
||||
|
||||
- site-a-b booted solo; its startup reconcile fetched 4 configs from central and stored them
|
||||
(row 67's machinery started their actors on that node — correct).
|
||||
- The pair was then restarted with a **fresh** site-a-a file. CDC snapshot replication seeded
|
||||
site-a-a's `deployed_configurations` within ~5 s of boot.
|
||||
- That was **after** site-a-a's boot-time load had already read an empty table, and **invisible**
|
||||
to site-a-a's own reconcile pass: central rightly reported *no gap* ("0 fetched"), because
|
||||
central's view of the site was correct — the site really did have the configs.
|
||||
- Result: enabled instances sat dead with their configs present (`debug snapshot` →
|
||||
`instanceNotFound`) until yet another restart.
|
||||
|
||||
Row 67 covers configs the reconcile *fetches*. This is the same gap through a side door.
|
||||
|
||||
### Row 72 — a refreshed config does not reach the running actor
|
||||
|
||||
*(Medium-Low, same silent-divergence class.)*
|
||||
|
||||
`ReconcileGapItem` covers both *missing* and *stale* configs. For a **stale** item the reconcile
|
||||
writes the newer config via `StoreDeployedConfigIfNewerAsync` — a pure file write. The running
|
||||
Instance Actor keeps executing the **previously compiled** config until the next restart or
|
||||
redeploy, and nothing anywhere surfaced "stored revision ≠ running revision".
|
||||
|
||||
Row 72 was deliberately deferred out of row 67's scope because restarting a *live* instance
|
||||
carries redeploy semantics that needed their own decision. That decision is section 3.
|
||||
|
||||
---
|
||||
|
||||
## 2. The mechanism: one periodic sweep in the Deployment Manager singleton
|
||||
|
||||
A single periodic tick on the Deployment Manager singleton re-reads every durable
|
||||
`deployed_configurations` row and repairs both divergences.
|
||||
|
||||
```
|
||||
InstanceSweepTick (every InstanceSweepIntervalSeconds, default 60)
|
||||
└── off-thread read of all deployed configs
|
||||
└── InstanceSweepLoaded → actor thread
|
||||
for each ENABLED stored config, unless a deploy pipeline owns the instance:
|
||||
no running actor → start it (row 78)
|
||||
running, revision ≠ stored → re-apply stored (row 72)
|
||||
```
|
||||
|
||||
**Ownership.** The Deployment Manager is a cluster singleton scoped to the site-specific role, so
|
||||
the sweep runs on exactly one node — the active one — and Instance Actors stay under the
|
||||
singleton, which is the same invariant row 67's singleton-proxy routing preserved.
|
||||
|
||||
### 2.1 Why a sweep and not a LocalDb change hook
|
||||
|
||||
The register suggested either. A sweep was chosen:
|
||||
|
||||
- **Coverage.** A replication-apply hook covers exactly one arrival path, and only on the half of
|
||||
the fleet that runs with replication enabled (rig posture: site-a replicated, site-b/site-c
|
||||
deliberately not). The sweep is arrival-path-agnostic — replication seeding, a post-boot
|
||||
snapshot resync, an operator restoring a file by hand, and any future path all reduce to the
|
||||
same question: *does the store agree with the runtime?*
|
||||
- **Availability.** `ILocalDb` exposes **no** change-notification, applied-rows callback or
|
||||
observer API — verified against `ZB.MOM.WW.LocalDb` 0.3.0. A hook would require a new upstream
|
||||
capability in the `scadaproj` library plus an in-process bus from the sync applier into this
|
||||
singleton, i.e. a cross-repo change to close a site-local gap.
|
||||
- **Reuse.** The sweep inherits row 67's guards verbatim; the shared `StartStoredInstance` helper
|
||||
means the reconcile heal and the sweep heal produce identical bookkeeping and identical
|
||||
operator-facing events, differing only in the trigger they name.
|
||||
|
||||
Cost is one SQLite read per minute per site — the same read `HandleDeploymentStateQuery` already
|
||||
performs per central query.
|
||||
|
||||
### 2.2 Guards (all inherited or shared with row 67)
|
||||
|
||||
- **Disabled instances are never started.** A disabled instance legitimately has a config row and
|
||||
no actor; that is its correct steady state.
|
||||
- **The sweep defers to any deploy pipeline that owns the instance** — a compile warm in flight
|
||||
(`_deployWarms`), a predecessor still terminating (`_terminatingActorsByName`), or a deployment
|
||||
awaiting its persistence/init join (`_inFlightDeployments`). This is the same
|
||||
`InstanceIsOwnedByADeployPipeline` predicate row 67's `ShouldAttemptEnsureStart` is now built
|
||||
from. It matters: a redeploy passes through a window where the predecessor has been stopped and
|
||||
removed from routing but the replacement does not exist yet, which an unguarded sweep would read
|
||||
as "enabled config, no actor" and answer by creating a child whose actor name is still
|
||||
registered.
|
||||
- **Inert until boot-time creation finishes.** A tick landing between two staggered startup
|
||||
batches would "heal" instances the boot load is already about to create, announcing a fault on
|
||||
a node that is merely still starting.
|
||||
- **One repair attempt per (instance, revision) pair.** This is what keeps a *periodic* self-heal
|
||||
from becoming a restart loop: an Instance Actor that dies during init leaves its config row in
|
||||
place, and a stored config the site compile gate rejects can never apply. Without the claim,
|
||||
either would be re-attempted — and re-announced — every tick forever. A genuinely new revision
|
||||
is a different key and is always attempted.
|
||||
- **No overlapping reads.** A slow storage read cannot let ticks stack up into duplicate repairs.
|
||||
- **Silent when healthy.** A sweep that finds nothing emits no log line above Trace and no site
|
||||
event. This runs on every tick of every healthy site, forever.
|
||||
|
||||
---
|
||||
|
||||
## 3. The row-72 decision: converge, do not merely warn
|
||||
|
||||
**Decision.** When the sweep finds a running instance whose revision no longer matches the stored
|
||||
row, it **applies the stored config through the same per-instance apply/restart path a central
|
||||
redeploy uses** (`HandleDeploy` → compile gate → stop/watch/recreate → persist), and announces
|
||||
both revisions in a Warning `deployment` site event.
|
||||
|
||||
**Rationale.**
|
||||
|
||||
- The stored row **is** what central successfully deployed. The running actor is executing a
|
||||
config central believes was replaced. Leaving it running is the actual fault — a site quietly
|
||||
serving superseded logic is worse than a controlled restart, and the operator has no way to
|
||||
discover it.
|
||||
- The alternative (surface-only) leaves an operator with a Warning they can only act on by
|
||||
issuing a redeploy — the exact operation the sweep would have performed, minus the delay and
|
||||
minus the chance they see it.
|
||||
- Reusing `HandleDeploy` rather than writing a second apply path means the compile gate, the
|
||||
warm-then-gate ordering, the redeploy-supersede semantics, the static-override and
|
||||
native-alarm reset, and all-or-nothing-per-instance are the ones already in production. No new
|
||||
semantics were invented. In particular a stop still does **not** cancel an in-flight script
|
||||
run, exactly as for a central redeploy.
|
||||
|
||||
**Why this is safe against the ordering hazards the register worried about.**
|
||||
|
||||
- The sweep never acts on an instance a deploy pipeline owns, so it cannot interleave with a real
|
||||
redeploy; and if a deploy arrives *after* the sweep's own deploy is queued, it is ordinary
|
||||
last-write-wins on the same mailbox, which is a case the deploy path already handles.
|
||||
- A rejected restart is inert, not partial: `RunDeployGateAndProceed` returns **before** anything
|
||||
is stopped, so the running instance keeps serving its old config. It is announced once
|
||||
(Warning for the attempt, Error for the failure) and never retried for that revision.
|
||||
- The synthetic `DeployInstanceCommand` replies to `Self`, not to `NoSender` — dead letters are a
|
||||
monitored site health metric, and a self-heal must not manufacture them.
|
||||
|
||||
**Row 72's "nothing surfaces it" is false either way.** Even in the failure path the divergence
|
||||
produces two operator-visible `deployment` site events naming both revisions.
|
||||
|
||||
---
|
||||
|
||||
## 4. Configuration
|
||||
|
||||
`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds` — default **60**, validated `>= 0`
|
||||
(`SiteRuntimeOptionsValidator`). **`0` disables the sweep**, which reopens both gaps; the row-67
|
||||
`EnsureInstanceStartedCommand` path is unaffected either way. No config change is required on any
|
||||
node to get the fix — the default is live.
|
||||
|
||||
## 5. Observability
|
||||
|
||||
| Condition | Log | Site event |
|
||||
|---|---|---|
|
||||
| Nothing to repair | — | — |
|
||||
| Row 78 start | Warning naming the instance and trigger | `deployment` / **Warning** — "started by deployment-manager sweep self-heal" |
|
||||
| Row 72 restart attempt | Warning naming both revisions | `deployment` / **Warning** — "restarted by deployment-manager sweep self-heal — running revision X is stale, applying stored revision Y" |
|
||||
| Row 72 restart applied | Information | — (the deploy path's own Info event covers it) |
|
||||
| Row 72 restart rejected | Warning with the gate's errors | `deployment` / **Error** — "stale-revision deployment-manager sweep self-heal failed" |
|
||||
| Storage read failed | Warning | — |
|
||||
|
||||
The trigger is named in the message so a sweep heal is distinguishable from row 67's
|
||||
"startup reconcile self-heal".
|
||||
|
||||
## 6. Tests
|
||||
|
||||
`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceSweepSelfHealTests.cs` — 10 tests.
|
||||
|
||||
Fail-before evidence (behavioral revert: the periodic timer left unarmed) — **5 fail, 5 pass**:
|
||||
|
||||
| Test | Pre-fix |
|
||||
|---|---|
|
||||
| `Sweep_StartsEnabledInstance_WhoseConfigArrivedWithoutADeployOrReconcile` | **FAIL** — the live row-78 symptom |
|
||||
| `SweepStartedInstance_EmitsAWarningSiteEvent_NamingTheSweepTrigger` | **FAIL** |
|
||||
| `Sweep_StartsTheInstanceExactlyOnce_AcrossManySweeps` | **FAIL** |
|
||||
| `Sweep_RestartsInstance_ServingAStaleRevision` | **FAIL** — the row-72 symptom |
|
||||
| `Sweep_StaleRestartRejectedByTheCompileGate_IsAnnouncedOnceAndNotRetried` | **FAIL** |
|
||||
| `Sweep_NeverStarts_ADisabledInstance` | pass (negative guard) |
|
||||
| `Sweep_DefersToAnInFlightRedeploy` | pass (negative guard) |
|
||||
| `Sweep_DoesNotRestart_WhenStoredAndRunningRevisionsMatch` | pass (negative guard) |
|
||||
| `Sweep_IsSilent_WhenThereIsNothingToHeal` | pass (negative guard) |
|
||||
| `Sweep_DoesNothing_WhenTheIntervalIsZero` | pass (negative guard) |
|
||||
|
||||
Every test gates on the Deployment Manager's first instance-count report before writing a config
|
||||
row, because a row written before the boot-time load has read storage is picked up by that load —
|
||||
which is precisely the path row 78 is about *not* having. Without the gate a test would silently
|
||||
prove the boot load works.
|
||||
@@ -98,6 +98,16 @@ flowchart TD
|
||||
|
||||
> **Startup reconcile heals the RUNTIME, not just the config file (register row 67)**: `SiteReconciliationActor` runs on **every** site node (not a singleton) and performs one best-effort pass shortly after boot — it reports the node's local deployed inventory to central and fetches + stores any config central reports as a gap. That pass fires **after** step 1's deployed-config read has already run, so storing a config started nothing: Instance Actors are created only by the boot-time load above and by an explicit deploy/enable. An instance whose config was missing at boot (lost volume, first boot after a restore, rehomed data dir) therefore stayed **dead until a second restart**, whose load finally picked it up — silently, with nothing surfacing "config stored, actor not running". The pass now raises a fire-and-forget `EnsureInstanceStartedCommand` for every **enabled** instance it healed, routed through the **deployment-manager singleton proxy** so Instance Actors are only ever created under the singleton regardless of which node's pass found the gap. The handler is deliberately advisory: it **no-ops when the actor is already running** (both nodes legitimately raise it for the same instance), **re-reads the durable row** — which is the authority on both the config body and enabled/disabled state, so a disable landing after the fetch still wins and the instance stays stopped — **defers to any in-flight deploy/redeploy** for that instance, preserving the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule (it never applies a config, only starts one already fully persisted), and treats a missing row as a quiet no-op (the normal case on an **unreplicated** site whose peer node healed its own file). A successful reconcile start is **surfaced**, not silent: a Warning log line plus an operational `deployment` site event, because reaching it means the node booted without a config for an instance that should have been running.
|
||||
|
||||
> **Instance reconcile sweep — the store and the runtime are kept in agreement, and disagreement is surfaced (register rows 78 + 72)**: the reconcile self-heal above covers configs the reconcile pass *itself* fetched. Two sibling divergences were not covered, and neither surfaced anything.
|
||||
> - **Row 78 — a config that arrives by replication starts nothing.** `deployed_configurations` is one of LocalDb's ten replicated tables, so on a replicated pair a config can appear in a node's file with **no local actor involved at all** — the CDC sync applier writes the row directly. Proven live 2026-08-15: a fresh site-a-a was seeded from its peer by snapshot replication within ~5 s of boot, i.e. **after** its boot-time load had read an empty table and **invisibly** to its own reconcile (central rightly reported no gap, because central's view of the site was correct). Enabled instances sat dead with their configs present until another restart.
|
||||
> - **Row 72 — a refreshed config does not reach the running actor.** For a *stale* `ReconcileGapItem` the pass writes the newer config via `StoreDeployedConfigIfNewerAsync`, a pure file write; the running Instance Actor keeps executing the previously compiled config until the next restart or redeploy.
|
||||
>
|
||||
> Both are closed by one **periodic sweep owned by the Deployment Manager singleton** (`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`, default 60, validated `>= 0`; `0` disables it). Each tick re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config: starts the Instance Actor if none is running (row 78), or — if one is running at a revision the store has since replaced — **re-applies the stored config through the exact per-instance apply/restart path a central redeploy uses** (row 72). A sweep is chosen over a LocalDb change hook deliberately: a hook would cover only the replication arrival path, only on replicated sites, and `ILocalDb` exposes no change-notification API at all, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot snapshot resync, a hand-restored file, any future path). See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
|
||||
>
|
||||
> The **row-72 decision is to converge, not merely warn**: the stored row *is* what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing `HandleDeploy` means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy.
|
||||
>
|
||||
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
|
||||
|
||||
> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.
|
||||
|
||||
> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory.
|
||||
@@ -127,6 +137,7 @@ flowchart TD
|
||||
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
|
||||
- **Delete**: Stops the Instance Actor and its children, removes the deployed configuration from local SQLite. Does **not** clear store-and-forward messages.
|
||||
- **Ensure Started** (`EnsureInstanceStartedCommand`, site-internal — never sent by central): advisory, idempotent request to make sure an already-persisted, enabled instance is actually running. Raised by the startup reconcile self-heal; see the note under **Startup Behavior**. Unlike the three commands above it has **no reply** and applies no configuration — it only starts an Instance Actor for a config that is already fully stored, and no-ops when the actor is running, the stored row says disabled, the row is absent, or a deploy for that instance is in flight.
|
||||
- **Instance reconcile sweep** (internal timer, not a command): the periodic singleton-owned check that the running Instance Actors still agree with the durable config rows — starting an enabled instance that has none, and re-applying a stored config a running instance has fallen behind. See the note under **Startup Behavior**; the start half shares its implementation (and therefore its bookkeeping and site events) with **Ensure Started**, and the restart half goes through the ordinary redeploy path.
|
||||
|
||||
### Debug Snapshot Routing
|
||||
- Receives `DebugSnapshotRequest` from the Communication Layer and forwards to the Instance Actor by unique name (same lookup as `SubscribeDebugViewRequest`).
|
||||
|
||||
Reference in New Issue
Block a user