dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(siteruntime): startup reconcile starts the healed InstanceActor (register row 67)

Browse Source 
SiteReconciliationActor's node-STARTUP fetch healed the config FILE but never the
RUNTIME. Instance Actors are created from exactly two paths -- the Deployment
Manager's boot-time SQLite load and an explicit deploy/enable command -- and the
reconcile pass runs AFTER that boot load has already read storage. So a config
reconcile restored started no actor: an enabled, successfully-deployed instance
whose config was missing at boot (lost volume, first boot after a restore,
rehomed data dir) stayed dead until the NEXT restart, whose load finally picked
it up. Proven live on site-a 2026-08-15: "Reconcile: fetched + stored config"
with no "InstanceActor started" after it, and a debug snapshot answering
instanceNotFound.

The pass now raises a fire-and-forget EnsureInstanceStartedCommand (new Commons
Deployment message) for every ENABLED instance it healed, routed through the
deployment-manager singleton PROXY so Instance Actors stay under the singleton no
matter which node's reconcile found the gap.

The handler is deliberately advisory:
- idempotent -- no-op when the actor is already running, which both nodes'
  reconcile actors legitimately trigger for the same instance;
- enabled-state-aware -- it re-reads the durable row, which is the authority on
  both the config body and enabled/disabled, so a disable landing after the
  fetch still wins and the instance stays stopped;
- it defers to any in-flight deploy/redeploy/warm for the instance, which keeps
  the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule
  intact -- this never applies a config, it only starts one already persisted;
- a missing row is a quiet no-op, the normal case on an unreplicated site whose
  peer node healed its own file.

The heal is no longer silent: a successful reconcile start emits a Warning log
line and an operational `deployment` site event, because reaching it means the
node booted without a config for an instance that should have been running.
This commit is contained in:
Joseph Doherty
2026-08-15 12:43:25 -04:00
parent ef876b38c5
commit 482165cbac
5 changed files with 626 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReconciliationActor.cs
+68 -8
View File
@@ -14,7 +14,9 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
/// <item>read the node's local deployed inventory from SQLite,</item>
/// <item>report it to central via the <c>SiteCommunicationActor</c> Ask
/// (<see cref="ReconcileSiteRequest"/> → <see cref="ReconcileSiteResponse"/>),</item>
/// <item>fetch each gap item's config over HTTP and guarded-write it, and</item>
/// <item>fetch each gap item's config over HTTP and guarded-write it,</item>
/// <item>ask the Deployment Manager singleton to START each enabled instance it just
/// healed, and</item>
/// <item>LOG (never delete) any orphan the node still has but central no longer deploys.</item>
/// </list>
/// </summary>
@@ -36,6 +38,17 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
/// The site does NOT carry the central fetch base URL in its own config — it uses
/// <see cref="ReconcileSiteResponse.CentralFetchBaseUrl"/> from central's reply.
/// </para>
/// <para>
/// <b>The pass heals the runtime, not just the file</b> (deferred register row 67). Storing a
/// config starts nothing by itself: the Deployment Manager creates Instance Actors from the
/// boot-time SQLite load, which has already run by the time this pass fires. So for every
/// ENABLED instance whose config this pass stored, an <see cref="EnsureInstanceStartedCommand"/>
/// is sent to the Deployment Manager singleton, which starts the actor if it is not already
/// running. Without it, an instance whose config was missing at boot (lost volume, first boot
/// after a restore, rehomed data dir) needed a SECOND restart before it ran again.
/// The command is advisory, idempotent, and deliberately fire-and-forget — a site with no
/// Deployment Manager ref configured simply keeps the old file-only behavior.
/// </para>
/// </remarks>
public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
{
@@ -44,6 +57,7 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
private readonly SiteStorageService _storage;
private readonly IDeploymentConfigFetcher _configFetcher;
private readonly IActorRef _siteCommunicationActor;
private readonly IActorRef? _deploymentManager;
private readonly string _siteIdentifier;
private readonly string _nodeId;
private readonly ILogger<SiteReconciliationActor> _logger;
@@ -71,6 +85,14 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
/// Defaults to 5 seconds.
/// </param>
/// <param name="askTimeout">Round-trip timeout for the reconcile Ask to central. Defaults to 30 seconds.</param>
/// <param name="deploymentManager">
/// The Deployment Manager singleton PROXY. Every enabled instance this pass heals is
/// followed by a fire-and-forget <see cref="EnsureInstanceStartedCommand"/> so the runtime
/// heals with the file (register row 67). Routing through the proxy — rather than starting
/// anything locally — is what keeps Instance Actors under the singleton no matter which
/// node's reconcile pass found the gap. Optional: when null the pass stores configs and
/// starts nothing, exactly as it did before.
/// </param>
public SiteReconciliationActor(
SiteStorageService storage,
IDeploymentConfigFetcher configFetcher,
@@ -79,11 +101,13 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
string nodeId,
ILogger<SiteReconciliationActor> logger,
TimeSpan? initialDelay = null,
TimeSpan? askTimeout = null)
TimeSpan? askTimeout = null,
IActorRef? deploymentManager = null)
{
_storage = storage;
_configFetcher = configFetcher;
_siteCommunicationActor = siteCommunicationActor;
_deploymentManager = deploymentManager;
_siteIdentifier = siteIdentifier;
_nodeId = nodeId;
_logger = logger;
@@ -153,6 +177,9 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
var fetched = 0;
var failed = 0;
// Enabled instances healed by this pass, to be started via the Deployment Manager once
// the pass reports back on the actor thread (row 67).
var healedEnabled = new List<string>();
// Fetch + guarded-write each gap item. Per-item failure must not abort the rest.
foreach (var item in response.Gap)
@@ -168,6 +195,12 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
.ConfigureAwait(false);
fetched++;
// Only ENABLED instances are candidates to start; a disabled one correctly
// keeps a config row with no running actor. The Deployment Manager re-checks
// this against the durable row anyway — this is just the cheap filter.
if (item.IsEnabled)
healedEnabled.Add(item.InstanceUniqueName);
_logger.LogInformation(
"Reconcile: fetched + stored config for {Instance} (deployment {DeploymentId}, rev {Revision})",
item.InstanceUniqueName, item.DeploymentId, item.RevisionHash);
@@ -198,7 +231,7 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
name);
}
return ReconcilePassResult.Completed(fetched, failed, response.OrphanNames.Count);
return ReconcilePassResult.Completed(fetched, failed, response.OrphanNames.Count, healedEnabled);
}
private void HandleReconcilePassResult(ReconcilePassResult result)
@@ -216,6 +249,23 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
_logger.LogInformation(
"Reconcile pass for site {Site} node {Node} complete: {Fetched} fetched, {Failed} failed, {Orphans} orphan(s)",
_siteIdentifier, _nodeId, result.Fetched, result.Failed, result.Orphans);
// Heal the RUNTIME, not just the file (row 67). Storing a config starts nothing: the
// Deployment Manager's boot-time SQLite load has already run by now, so without this
// an instance missing at boot would stay dead until the next restart. Fire-and-forget
// through the singleton proxy — the handler is idempotent, re-checks the durable row's
// enabled state, and defers to any in-flight deploy.
if (_deploymentManager == null || result.HealedEnabled.Count == 0)
return;
foreach (var instanceName in result.HealedEnabled)
{
_logger.LogInformation(
"Reconcile: requesting start of healed instance {Instance} on site {Site}",
instanceName, _siteIdentifier);
_deploymentManager.Tell(
new EnsureInstanceStartedCommand(instanceName, "startup reconcile"));
}
}
// ── Internal messages ──
@@ -227,21 +277,31 @@ public sealed class SiteReconciliationActor : ReceiveActor, IWithTimers
private RunReconcile() { }
}
/// <summary>Summary of one reconcile pass, piped to <c>Self</c> for logging.</summary>
private sealed record ReconcilePassResult(int Fetched, int Failed, int Orphans, Exception? Error)
/// <summary>
/// Summary of one reconcile pass, piped to <c>Self</c> for logging and for the
/// ensure-started dispatch. <see cref="HealedEnabled"/> is built off the actor thread but
/// only ever read on it, so no synchronization is needed.
/// </summary>
private sealed record ReconcilePassResult(
int Fetched, int Failed, int Orphans, IReadOnlyList<string> HealedEnabled, Exception? Error)
{
/// <summary>Creates a result for a reconcile pass that ran to completion.</summary>
/// <param name="fetched">Number of instances successfully fetched/reconciled.</param>
/// <param name="failed">Number of instances that failed to reconcile.</param>
/// <param name="orphans">Number of local instances no longer deployed at central.</param>
/// <param name="healedEnabled">
/// Names of the ENABLED instances whose config this pass stored — each gets an
/// <see cref="EnsureInstanceStartedCommand"/> so the runtime heals with the file.
/// </param>
/// <returns>A completed <see cref="ReconcilePassResult"/> with no error.</returns>
public static ReconcilePassResult Completed(int fetched, int failed, int orphans)
=> new(fetched, failed, orphans, null);
public static ReconcilePassResult Completed(
int fetched, int failed, int orphans, IReadOnlyList<string> healedEnabled)
=> new(fetched, failed, orphans, healedEnabled, null);
/// <summary>Creates a result for a reconcile pass that failed before completing.</summary>
/// <param name="error">The exception that caused the pass to fail.</param>
/// <returns>A faulted <see cref="ReconcilePassResult"/> carrying the error.</returns>
public static ReconcilePassResult Faulted(Exception error)
=> new(0, 0, 0, error);
=> new(0, 0, 0, [], error);
}
}