fix(siteruntime): startup reconcile starts the healed InstanceActor (register row 67)

SiteReconciliationActor's node-STARTUP fetch healed the config FILE but never the
RUNTIME. Instance Actors are created from exactly two paths -- the Deployment
Manager's boot-time SQLite load and an explicit deploy/enable command -- and the
reconcile pass runs AFTER that boot load has already read storage. So a config
reconcile restored started no actor: an enabled, successfully-deployed instance
whose config was missing at boot (lost volume, first boot after a restore,
rehomed data dir) stayed dead until the NEXT restart, whose load finally picked
it up. Proven live on site-a 2026-08-15: "Reconcile: fetched + stored config"
with no "InstanceActor started" after it, and a debug snapshot answering
instanceNotFound.

The pass now raises a fire-and-forget EnsureInstanceStartedCommand (new Commons
Deployment message) for every ENABLED instance it healed, routed through the
deployment-manager singleton PROXY so Instance Actors stay under the singleton no
matter which node's reconcile found the gap.

The handler is deliberately advisory:
- idempotent -- no-op when the actor is already running, which both nodes'
  reconcile actors legitimately trigger for the same instance;
- enabled-state-aware -- it re-reads the durable row, which is the authority on
  both the config body and enabled/disabled, so a disable landing after the
  fetch still wins and the instance stays stopped;
- it defers to any in-flight deploy/redeploy/warm for the instance, which keeps
  the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule
  intact -- this never applies a config, it only starts one already persisted;
- a missing row is a quiet no-op, the normal case on an unreplicated site whose
  peer node healed its own file.

The heal is no longer silent: a successful reconcile start emits a Warning log
line and an operational `deployment` site event, because reaching it means the
node booted without a config for an instance that should have been running.
This commit is contained in:
Joseph Doherty
2026-08-15 12:43:25 -04:00
parent ef876b38c5
commit 482165cbac
5 changed files with 626 additions and 9 deletions
@@ -1024,6 +1024,11 @@ akka {{
// guarded-writes it (orphans are logged, never deleted). Requires the HTTP
// config fetcher; if it is somehow absent the self-heal is skipped (best-effort —
// replication remains the primary path and the next restart retries).
// The dmProxy is passed so the pass heals the RUNTIME too (register row 67): each
// enabled instance it restores gets an EnsureInstanceStartedCommand routed to the
// singleton, which starts the Instance Actor if the boot-time load missed it. Going
// through the PROXY (not a local ActorOf) keeps Instance Actors under the singleton
// regardless of which node's reconcile pass found the gap.
if (deploymentConfigFetcher != null)
{
var reconcileLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
@@ -1037,7 +1042,8 @@ akka {{
_nodeOptions.NodeName,
reconcileLogger,
null,
null)),
null,
dmProxy)),
"site-reconciliation");
_logger.LogInformation(
"SiteReconciliationActor created (per-node startup self-heal) for site {SiteId} node {Node}",