P1-1 — the row-67 ensure-started self-heal re-read the durable row off-thread but
captured no _storeWriteGeneration and never ran IsStoreSnapshotStale, unlike the boot
load and the reconcile sweep. A Disable or Delete committing inside that window left the
result carrying an enabled config that passed every guard (neither command sets any of
the three pipeline markers ShouldAttemptEnsureStart re-checks), so the heal started an
instance the operator had just stopped — PERMANENTLY, since the sweep iterates store rows
only and has no runtime→store stop leg. The generation is now captured at dispatch,
threaded on EnsureInstanceStartedResult and checked on landing, and the comment claiming
the stored row already covered this is corrected. The heal reads through _configLoader,
the same seam the boot load and sweep use (identical production call), so a test can hold
the read open across a committed disable/delete.
P1-2 — the "static overrides reset on redeployment" invariant was racy in both
directions. ApplyDeployment created the successor Instance Actor first and only then
dispatched the clear, queued behind a full config-JSON write, so the successor's PreStart
override load routinely won and applied the PREDECESSOR's overrides — keys absent from
the new config included — to in-memory state for the actor's lifetime while the disk
ended cleared; a later restart then silently flipped the values. The clear is now
dispatched BEFORE CreateInstanceActor and handed to the actor as a gate its override load
chains behind; the persistence task joins the same task, so a failed clear still fails the
deploy exactly as before. Leg 2: the predecessor's PostStop flush was fire-and-forget and
could commit up to one coalescing window AFTER the clear (the Deployment Manager gates the
clear on Terminated), durably resurrecting overrides — PostStop now waits for it, bounded
at 5s with a Warning on overrun.
Tests: new RedeployStaticOverrideResetTests (5) pins both legs plus the gate mechanism and
first-deploy behaviour; three new cases in ReconcileStartsInstanceTests pin the discarded
disable/delete heals and that an unraced heal still starts. InstanceActorTests'
StaticOverride_ResetOnRedeployment comment claimed to simulate an ordering the production
path did not have — corrected.
Row 67's suite still opened every test with a bare Task.Delay(500) after
constructing the Deployment Manager. Each test then writes its
deployed_configurations row -- and a row that lands before the boot-time load
reads storage is picked up by that load, which means the test passes while
proving the BOOT LOAD works rather than the startup-reconcile path it names. A
sleep only makes that race unlikely; on a loaded machine the load can still be
in flight when it expires and the test silently changes what it proves instead
of failing.
The row-78 fix session hit exactly this trap and closed it in
InstanceSweepSelfHealTests by gating on the first
ISiteHealthCollector.SetInstanceCounts report -- the edge the manager raises
unconditionally the moment its load completes, including for an empty site.
This applies the same gate here.
The probe is now a shared TestSupport helper (DeploymentManagerStartupProbe,
added with row 85) rather than a private nested class, so both suites use one
implementation.
Also pins InstanceSweepIntervalSeconds = 0 for this suite: with the sweep armed
its periodic self-heal would independently start the same instances and the
suite would stop distinguishing the row-67 mechanism from the row-78 one.
The remaining Task.Delay calls in the file are deliberate and left alone -- they
precede assertions that NOTHING happened (a disabled instance stayed stopped, a
redundant ensure-started changed nothing), which cannot be awaited on. The one
delay that was a readiness wait rather than an absence wait is now an
AwaitAssertAsync on the instance actually being routable.
SiteReconciliationActor's node-STARTUP fetch healed the config FILE but never the
RUNTIME. Instance Actors are created from exactly two paths -- the Deployment
Manager's boot-time SQLite load and an explicit deploy/enable command -- and the
reconcile pass runs AFTER that boot load has already read storage. So a config
reconcile restored started no actor: an enabled, successfully-deployed instance
whose config was missing at boot (lost volume, first boot after a restore,
rehomed data dir) stayed dead until the NEXT restart, whose load finally picked
it up. Proven live on site-a 2026-08-15: "Reconcile: fetched + stored config"
with no "InstanceActor started" after it, and a debug snapshot answering
instanceNotFound.
The pass now raises a fire-and-forget EnsureInstanceStartedCommand (new Commons
Deployment message) for every ENABLED instance it healed, routed through the
deployment-manager singleton PROXY so Instance Actors stay under the singleton no
matter which node's reconcile found the gap.
The handler is deliberately advisory:
- idempotent -- no-op when the actor is already running, which both nodes'
reconcile actors legitimately trigger for the same instance;
- enabled-state-aware -- it re-reads the durable row, which is the authority on
both the config body and enabled/disabled, so a disable landing after the
fetch still wins and the instance stays stopped;
- it defers to any in-flight deploy/redeploy/warm for the instance, which keeps
the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule
intact -- this never applies a config, it only starts one already persisted;
- a missing row is a quiet no-op, the normal case on an unreplicated site whose
peer node healed its own file.
The heal is no longer silent: a successful reconcile start emits a Warning log
line and an operational `deployment` site event, because reaching it means the
node booted without a config for an instance that should have been running.
Joseph Doherty