dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(siteruntime): stamp the ensure-started heal with the store-write generation + order the redeploy override reset ahead of the successor (arch-review 2026-08-19 P1-1 + P1-2)

Browse Source 
P1-1 — the row-67 ensure-started self-heal re-read the durable row off-thread but
captured no _storeWriteGeneration and never ran IsStoreSnapshotStale, unlike the boot
load and the reconcile sweep. A Disable or Delete committing inside that window left the
result carrying an enabled config that passed every guard (neither command sets any of
the three pipeline markers ShouldAttemptEnsureStart re-checks), so the heal started an
instance the operator had just stopped — PERMANENTLY, since the sweep iterates store rows
only and has no runtime→store stop leg. The generation is now captured at dispatch,
threaded on EnsureInstanceStartedResult and checked on landing, and the comment claiming
the stored row already covered this is corrected. The heal reads through _configLoader,
the same seam the boot load and sweep use (identical production call), so a test can hold
the read open across a committed disable/delete.

P1-2 — the "static overrides reset on redeployment" invariant was racy in both
directions. ApplyDeployment created the successor Instance Actor first and only then
dispatched the clear, queued behind a full config-JSON write, so the successor's PreStart
override load routinely won and applied the PREDECESSOR's overrides — keys absent from
the new config included — to in-memory state for the actor's lifetime while the disk
ended cleared; a later restart then silently flipped the values. The clear is now
dispatched BEFORE CreateInstanceActor and handed to the actor as a gate its override load
chains behind; the persistence task joins the same task, so a failed clear still fails the
deploy exactly as before. Leg 2: the predecessor's PostStop flush was fire-and-forget and
could commit up to one coalescing window AFTER the clear (the Deployment Manager gates the
clear on Terminated), durably resurrecting overrides — PostStop now waits for it, bounded
at 5s with a Warning on overrun.

Tests: new RedeployStaticOverrideResetTests (5) pins both legs plus the gate mechanism and
first-deploy behaviour; three new cases in ReconcileStartsInstanceTests pin the discarded
disable/delete heals and that an unraced heal still starts. InstanceActorTests'
StaticOverride_ResetOnRedeployment comment claimed to simulate an ordering the production
path did not have — corrected.
This commit is contained in:
Joseph Doherty
2026-08-19 14:47:34 -04:00
parent 64e6ba70e8
commit e63ffd90ee
5 changed files with 650 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs
+67 -13
View File
@@ -1197,9 +1197,27 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
{
var instanceName = command.InstanceUniqueName;
// Static overrides and mirrored native alarm state are reset on redeployment per
// design decision — the new config may bind different sources, and the source snapshot
// re-seeds the mirror on (re)subscribe. Dispatched HERE, BEFORE the successor exists,
// and handed to it as a gate (arch-review 2026-08-19 P1-2). It used to run inside the
// persistence task below, i.e. after the actor was already created AND queued behind a
// full config-JSON write, so the successor's PreStart override load routinely won the
// race and applied the PREDECESSOR's overrides — including keys absent from the new
// config — to in-memory state for the actor's whole lifetime, while the disk ended up
// cleared. That combination is worse than either half alone: the running instance
// silently disagreed with its own store, and a later restart flipped the values.
var staticStateResetGate = Task.Run(async () =>
{
await _storage.ClearStaticOverridesAsync(instanceName);
await _storage.ClearNativeAlarmsForInstanceAsync(instanceName);
});
// Create the Instance Actor immediately. CreateInstanceActor first ensures the DCL
// connections this config's data-sourced attributes subscribe through (register row 85).
CreateInstanceActor(instanceName, command.FlattenedConfigurationJson, command.RevisionHash);
CreateInstanceActor(
instanceName, command.FlattenedConfigurationJson, command.RevisionHash,
staticStateResetGate);
if (!isRedeploy)
_deployedInstanceNames.Add(instanceName);
UpdateInstanceCounts();
@@ -1228,11 +1246,9 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
command.RevisionHash,
isEnabled: true);
// Static overrides and mirrored native alarm state are reset on
// redeployment per design decision — the new config may bind different
// sources, and the source snapshot re-seeds the mirror on (re)subscribe.
await _storage.ClearStaticOverridesAsync(instanceName);
await _storage.ClearNativeAlarmsForInstanceAsync(instanceName);
// Join the reset dispatched ahead of actor creation, so a failed clear still
// fails the deployment exactly as it did when the clear ran inline here.
await staticStateResetGate;
// No explicit replication step: deployed_configurations is a replicated table,
// so the write above is captured and shipped to the peer like any other row.
@@ -1566,13 +1582,26 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// Re-read the durable row off the actor thread. The stored row — NOT the requesting
// message — is the authority on the config body AND on enabled/disabled state, so a
// disable that landed after the reconcile fetch still wins and the instance stays
// stopped.
// disable that landed before this read still wins and the instance stays stopped.
//
// Row 101: a disable/delete that lands WHILE the read is in flight is a different
// problem, and the row itself cannot catch it — the read already saw the enabled row,
// and neither Disable nor Delete leaves any of the three markers
// ShouldAttemptEnsureStart re-checks. So capture the generation on the actor thread
// BEFORE the read starts and let the landing handler discard a superseded snapshot,
// exactly as the boot load and the sweep do. Without it the heal could start an
// instance the operator had just disabled or deleted, PERMANENTLY: the sweep only ever
// walks store rows, so it has no runtime→store stop leg to undo it.
var generationAtRead = _storeWriteGeneration;
Task.Run(async () =>
{
var configs = await _storage.GetAllDeployedConfigsAsync();
// Read through the same seam the boot load and the sweep read through
// (_configLoader defaults to _storage.GetAllDeployedConfigsAsync, so this is the
// identical production call), which lets a test hold the read open and land the
// result behind a committed disable/delete.
var configs = await _configLoader();
var config = configs.FirstOrDefault(c => c.InstanceUniqueName == instanceName);
return new EnsureInstanceStartedResult(command, config, null);
return new EnsureInstanceStartedResult(command, config, null, generationAtRead);
}).ContinueWith(t => t.IsCompletedSuccessfully
? t.Result
: new EnsureInstanceStartedResult(command, null, t.Exception?.GetBaseException().Message))
@@ -1620,6 +1649,14 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
return;
}
// Row 101, ensure-started half: this node disabled, deleted or redeployed an instance
// while the read was in flight, so the snapshot predates state the actor has already
// advanced past. Discard it — starting from it would resurrect an instance the operator
// just stopped, and nothing would ever stop it again. Nothing is lost by dropping it:
// the heal is advisory, and the instance reconcile sweep re-reads the store every tick.
if (IsStoreSnapshotStale(result.StoreGenerationAtRead, "ensure-started"))
return;
// No stored row: nothing to start. Normal when the config was superseded/removed
// between the reconcile fetch and this read, and on an UNREPLICATED site whose peer
// node healed its own file — this singleton's storage legitimately has no such row.
@@ -3216,7 +3253,15 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// the single creation funnel, so recording it here covers every path that can ever start an
/// Instance Actor.
/// </param>
internal void CreateInstanceActor(string instanceName, string configJson, string revisionHash)
/// <param name="staticStateResetGate">
/// The deploy path's static-override / native-alarm reset, dispatched BEFORE this call so the
/// new actor's PreStart override load can be ordered after it (arch-review 2026-08-19 P1-2).
/// <see langword="null"/> on every non-deploy start (boot batch, enable, self-heal), where
/// nothing is being reset and the load runs immediately as before.
/// </param>
internal void CreateInstanceActor(
string instanceName, string configJson, string revisionHash,
Task? staticStateResetGate = null)
{
// Deferred register row 85 — an Instance Actor is useless without the DCL connection
// actors its data-sourced attributes subscribe through, so ensuring them belongs in the
@@ -3262,7 +3307,9 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
_loggerFactory.CreateLogger<InstanceActor>(),
_dclManager,
_healthCollector,
_serviceProvider));
_serviceProvider,
null,
staticStateResetGate));
var actorRef = Context.ActorOf(props, instanceName);
_instanceActors[instanceName] = actorRef;
@@ -3461,8 +3508,15 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// the actor thread. No original sender: the ensure-started path is fire-and-forget, so
/// there is nobody to answer.
/// </summary>
/// <param name="StoreGenerationAtRead">
/// Row 101: <see cref="_storeWriteGeneration"/> captured on the actor thread when the read
/// was dispatched. A mismatch on landing means a disable/delete/deploy committed while the
/// read was in flight, and the heal is dropped whole (see
/// <see cref="HandleEnsureInstanceStartedResult"/>).
/// </param>
internal record EnsureInstanceStartedResult(
EnsureInstanceStartedCommand Command, DeployedInstance? Config, string? Error);
EnsureInstanceStartedCommand Command, DeployedInstance? Config, string? Error,
long StoreGenerationAtRead = 0);
/// <summary>
/// Periodic instance reconcile-sweep tick (deferred register rows 78 + 72). Interval comes