fix(siteruntime): startup reconcile starts the healed InstanceActor (register row 67)

SiteReconciliationActor's node-STARTUP fetch healed the config FILE but never the
RUNTIME. Instance Actors are created from exactly two paths -- the Deployment
Manager's boot-time SQLite load and an explicit deploy/enable command -- and the
reconcile pass runs AFTER that boot load has already read storage. So a config
reconcile restored started no actor: an enabled, successfully-deployed instance
whose config was missing at boot (lost volume, first boot after a restore,
rehomed data dir) stayed dead until the NEXT restart, whose load finally picked
it up. Proven live on site-a 2026-08-15: "Reconcile: fetched + stored config"
with no "InstanceActor started" after it, and a debug snapshot answering
instanceNotFound.

The pass now raises a fire-and-forget EnsureInstanceStartedCommand (new Commons
Deployment message) for every ENABLED instance it healed, routed through the
deployment-manager singleton PROXY so Instance Actors stay under the singleton no
matter which node's reconcile found the gap.

The handler is deliberately advisory:
- idempotent -- no-op when the actor is already running, which both nodes'
  reconcile actors legitimately trigger for the same instance;
- enabled-state-aware -- it re-reads the durable row, which is the authority on
  both the config body and enabled/disabled, so a disable landing after the
  fetch still wins and the instance stays stopped;
- it defers to any in-flight deploy/redeploy/warm for the instance, which keeps
  the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule
  intact -- this never applies a config, it only starts one already persisted;
- a missing row is a quiet no-op, the normal case on an unreplicated site whose
  peer node healed its own file.

The heal is no longer silent: a successful reconcile start emits a Warning log
line and an operational `deployment` site event, because reaching it means the
node booted without a config for an instance that should have been running.
This commit is contained in:
Joseph Doherty
2026-08-15 12:43:25 -04:00
parent ef876b38c5
commit 482165cbac
5 changed files with 626 additions and 9 deletions
@@ -0,0 +1,29 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
/// <summary>
/// Site-internal request to make sure an already-persisted instance is actually RUNNING.
/// Routes via the deployment-manager singleton proxy so the Instance Actor is only ever
/// created under the singleton, whichever node raised the request.
/// <para>
/// Emitted by <c>SiteReconciliationActor</c> after its node-STARTUP pass fetches and stores
/// a config the node was missing. The reconcile heal path writes the FILE; without this the
/// runtime was left untouched, so an instance whose config was absent at boot (lost volume,
/// first boot after a restore, rehomed data dir) stayed dead until the NEXT restart, whose
/// SQLite load finally started it.
/// </para>
/// <para>
/// <b>Fire-and-forget and idempotent.</b> There is no reply: the handler no-ops when the
/// instance is already running, when an in-flight deploy will create it anyway, or when the
/// durable row says the instance is disabled. The stored row — not this message — is the
/// authority on both the config body and enabled/disabled state, so a disable that landed
/// after the reconcile fetch still wins.
/// </para>
/// </summary>
/// <param name="InstanceUniqueName">Unique name of the instance that should be running.</param>
/// <param name="Reason">
/// Short human-readable provenance for the request (e.g. <c>"startup reconcile"</c>), carried
/// verbatim into the log line and the operational site event so the heal is not silent.
/// </param>
public record EnsureInstanceStartedCommand(
string InstanceUniqueName,
string Reason);