dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(siteruntime): extend the row-101 stale-snapshot guard to the instance-config half

Browse Source 
ReadStoreSnapshotAsync's snapshot is read off the actor thread and consumed only
when its piped message lands. Row 101 guarded the DEFINITIONS half of that race;
the CONFIG half was left to machinery whose reach is time-limited, and both legs
of the instance reconcile sweep were reachable through it:

- Row-72 leg: a read dispatched before a deploy's persist, landing after the
  deploy completed AND after the pipeline released ownership, carries the
  revision the deploy replaced. TryRestartStaleInstance then "repairs" the
  instance BACKWARD, re-deploying the snapshot's own superseded ConfigJson.
  InstanceIsOwnedByADeployPipeline covers only in-flight deploys, and the
  per-revision claim has never seen the old revision, so nothing stopped it.
- Row-78 leg: a read dispatched before a delete/disable commits still shows an
  enabled row while the actor is already stopped, so StartStoredInstance
  resurrects it — and delete clears _sweepAttemptedRevisions, freeing the claim.

Both verified real by neutralising the freshness check and re-running the new
tests (the fail-before site event literally reads "running revision rev-2 is
stale, applying stored revision rev-1"; the deleted instance came back).

_definitionsWriteGeneration becomes _storeWriteGeneration, covering both tables
(the snapshot is one consistent read and is discarded whole, so one counter is
the right shape); DefinitionsGenerationAtRead -> StoreGenerationAtRead on both
internal records. The generation is now bumped at BOTH ends of every mutation —
when the off-thread write is dispatched and when its completion message lands —
because the config paths advance actor state before the write, so an
initiation-only bump would let an in-window read look fresh. Disable, delete and
the init-failure rollback have no actor-thread completion message of their own,
so a minimal StoreWriteCompleted self-message carries the bump for them.

CDC-replicated writes, snapshot resyncs, restored files and
SiteReconciliationActor's fetch deliberately do NOT bump: discovering exactly
those is the sweep's reason to exist.

A stale sweep result is now dropped WHOLE (no overlay apply, no start, no
restart). A stale boot load cannot be dropped — startup must finish — so it
re-dispatches LoadDeployedConfigs(); a fifth consecutive superseded read is
applied anyway with a Warning, and a late one arriving after the boot completed
is dropped rather than looping against the S5 retry timer.

Tests: three fail-before regressions (backward restart, resurrection, boot
re-read) plus two guard twins pinning that the legitimate row-72 restart and
row-78 start still fire on a current-generation snapshot. The row-101
definitions tests keep passing; FreshEmptySweepSnapshot's hardcoded generation
moves 1 -> 2 now that one artifact apply bumps at both ends.

Docs: Component-SiteRuntime.md's stale-snapshot paragraph now describes the
whole snapshot (both halves, both consumers, the boot re-dispatch, the
CDC-no-bump rule), the components mirror follows, and the arbitration memo gains
§11 extending §10 to the config half.
This commit is contained in:
Joseph Doherty
2026-08-18 05:38:00 -04:00
parent 4da5f7cd7a
commit f02e116688
6 changed files with 617 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs
+234 -54
View File
@@ -97,6 +97,19 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// stale <see cref="RetryStartupLoad"/> re-running the load after success.
/// </summary>
private bool _startupLoadCompleted;
/// <summary>
/// How many consecutive boot-time store reads may be re-dispatched for being superseded
/// mid-flight (row 101) before one is applied anyway. Purely a livelock backstop — see
/// <see cref="HandleStartupConfigsLoaded"/>.
/// </summary>
private const int MaxConsecutiveStaleStartupLoads = 5;
/// <summary>
/// Consecutive boot-time store reads discarded as superseded. Reset the moment a fresh one
/// lands, so the cap only ever counts an unbroken run.
/// </summary>
private int _consecutiveStaleStartupLoads;
private readonly Dictionary<string, IActorRef> _instanceActors = new();
/// <summary>
/// Tracks Instance Actors that are terminating as part of a redeployment, keyed by
@@ -441,6 +454,10 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// Internal deploy persistence result
Receive<DeployPersistenceResult>(HandleDeployPersistenceResult);
// Row 101: an off-thread mutation of a snapshot table has committed. The only work is
// the actor-thread generation bump — see _storeWriteGeneration.
Receive<StoreWriteCompleted>(msg => BumpStoreWriteGeneration(msg.Source));
// Instance Actor init confirmation — the second half of the deploy-success
// join (S6). Success is only reported once both this and persistence land.
Receive<InstanceActorInitialized>(HandleInstanceActorInitialized);
@@ -498,9 +515,10 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private void LoadDeployedConfigs()
{
// Row 101: capture on the actor thread BEFORE the off-thread read starts, so a landing
// handler can tell whether an artifact apply mutated the overlay while it was in flight.
var generationAtRead = _definitionsWriteGeneration;
// Row 101: capture on the actor thread BEFORE the off-thread read starts, so the landing
// handler can tell whether this actor mutated either snapshot table while it was in
// flight.
var generationAtRead = _storeWriteGeneration;
ReadStoreSnapshotAsync().ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
@@ -601,6 +619,39 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
return;
}
// Row 101, boot half: a snapshot invalidated while it was in flight cannot simply be
// DROPPED the way the sweep's is — the boot load has to complete or the node stays a
// silent zero-instance node — so it is re-read instead. The re-dispatch captures the
// current generation, so it can only be stale again if yet another mutation lands in
// ITS window; that requires a fresh deploy/enable/disable/delete per attempt, which is
// externally driven and finite in practice. The counter is the belt-and-braces cap
// against a pathological arrival rate: after MaxConsecutiveStaleStartupLoads attempts
// the snapshot is applied anyway with a Warning, because a boot that never finishes is
// strictly worse than one that starts from a slightly superseded view — the instance
// reconcile sweep re-reads the store every tick and converges it afterwards.
if (IsStoreSnapshotStale(msg.StoreGenerationAtRead, "boot load"))
{
// A completed boot load means this is a late duplicate (the S5 retry timer racing a
// success), not the boot the node is waiting on. Re-reading for it would loop
// pointlessly; the sweep owns convergence from here.
if (_startupLoadCompleted)
return;
if (++_consecutiveStaleStartupLoads <= MaxConsecutiveStaleStartupLoads)
{
LoadDeployedConfigs();
return;
}
_logger.LogWarning(
"The boot-time deployed-config read has come back superseded {Attempts} times in "
+ "a row — applying it anyway so startup completes; the instance reconcile sweep "
+ "converges any difference on its next tick",
_consecutiveStaleStartupLoads);
}
_consecutiveStaleStartupLoads = 0;
// Mark the startup load complete so a stale retry timer is ignored.
_startupLoadCompleted = true;
@@ -610,8 +661,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// on the peer node rebuilds the whole DCL from flattened snapshots, and without the
// overlay in place first it would rebuild them STALE and only correct itself a sweep
// tick later — a needless create-then-diverge on every failover.
ApplyConnectionDefinitionOverlayIfFresh(
msg.ConnectionDefinitions, msg.DefinitionsGenerationAtRead, "boot load");
ApplyConnectionDefinitionOverlay(msg.ConnectionDefinitions);
var enabledConfigs = msg.Configs.Where(c => c.IsEnabled).ToList();
_deployedInstanceNames.Clear();
@@ -1164,6 +1214,10 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
IsRedeploy = isRedeploy
};
// Row 101 (initiation half): a deployed_configurations write is about to start off the
// actor thread, so every store read currently in flight predates it.
BumpStoreWriteGeneration("deploy persist dispatch");
// Persist to SQLite and clear static overrides asynchronously
Task.Run(async () =>
{
@@ -1211,6 +1265,12 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private void HandleDeployPersistenceResult(DeployPersistenceResult result)
{
// Row 101 (completion half): this message is created only after the store write has
// finished, so bumping here is the point that provably happens-after the commit. Bumped
// unconditionally — a FAILED write may still have committed the config row before a
// later step threw, so "failed" is not "nothing changed".
BumpStoreWriteGeneration("deploy persist result");
// If the S6 Terminated fallback already failed this deployment (the Instance
// Actor died during init), the in-flight entry is gone — swallow the now-stale
// persistence result so the deployer is answered exactly once.
@@ -1311,13 +1371,22 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private void RollbackPersistedConfig(string instanceName)
{
Task.Run(() => _storage.RemoveDeployedConfigAsync(instanceName)).ContinueWith(t =>
// Row 101 (initiation half) — the row is about to disappear from under any in-flight read.
BumpStoreWriteGeneration("init-failure rollback dispatch");
var removal = Task.Run(() => _storage.RemoveDeployedConfigAsync(instanceName));
removal.ContinueWith(t =>
{
if (!t.IsCompletedSuccessfully)
_logger.LogError(t.Exception?.GetBaseException(),
"Failed to roll back the persisted config for {Instance} after an init failure",
instanceName);
});
// Row 101 (completion half): this path has no other actor-thread completion message, so
// one exists purely to bump the generation after the DELETE has committed. Without it a
// sweep read dispatched between the two points above would still see the rolled-back row
// and start an instance whose deployment had just failed.
removal.ContinueWith(_ => new StoreWriteCompleted("init-failure rollback")).PipeTo(Self);
}
/// <summary>
@@ -1361,7 +1430,19 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
UpdateInstanceCounts();
var sender = replyTo;
_storage.SetInstanceEnabledAsync(instanceName, false).ContinueWith(t =>
// Row 101 (initiation half): the actor has just stopped the Instance Actor while the
// durable row still reads enabled, which is exactly the shape the row-78 sweep repairs.
// Any store read in flight would therefore "prove" a divergence that does not exist and
// restart the instance the operator just disabled.
BumpStoreWriteGeneration("disable dispatch");
var disableWrite = _storage.SetInstanceEnabledAsync(instanceName, false);
// Row 101 (completion half) — see StoreWriteCompleted. The is_enabled UPDATE has landed
// by the time this message is created, closing the window between the bump above and the
// commit.
disableWrite.ContinueWith(_ => new StoreWriteCompleted("disable")).PipeTo(Self);
disableWrite.ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
{
@@ -1399,6 +1480,9 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
var sender = replyTo;
// Row 101 (initiation half) — an is_enabled UPDATE is about to start off-thread.
BumpStoreWriteGeneration("enable dispatch");
Task.Run(async () =>
{
await _storage.SetInstanceEnabledAsync(instanceName, true);
@@ -1418,6 +1502,10 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private void HandleEnableResult(EnableResult result)
{
// Row 101 (completion half): the is_enabled UPDATE has committed by the time this
// message exists. Unconditional for the same reason as the deploy result.
BumpStoreWriteGeneration("enable result");
var instanceName = result.Command.InstanceUniqueName;
if (result.Error != null || result.Config == null)
@@ -1635,7 +1723,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
_instanceSweepInFlight = true;
// Row 101: capture on the actor thread BEFORE the off-thread read starts.
var generationAtRead = _definitionsWriteGeneration;
var generationAtRead = _storeWriteGeneration;
ReadStoreSnapshotAsync().ContinueWith(t => t.IsCompletedSuccessfully
? new InstanceSweepLoaded(t.Result.Configs, null, t.Result.Definitions, generationAtRead)
: new InstanceSweepLoaded([], t.Exception?.GetBaseException().Message))
@@ -1662,14 +1750,23 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
return;
}
// Row 101, sweep half: this node mutated one of the two tables while the read was in
// flight, so the snapshot predates state the actor has already advanced past. Discard it
// WHOLE — no overlay apply, no row-78 start, no row-72 restart. Every repair this sweep
// makes is derived from "the store says X, the runtime says Y", and a snapshot older
// than the runtime inverts that comparison: it would restart an instance BACKWARD onto
// the revision a deploy just replaced, or resurrect one a disable/delete just stopped.
// Nothing is lost by waiting — the next tick's read starts after the write.
if (IsStoreSnapshotStale(msg.StoreGenerationAtRead, "instance sweep"))
return;
// Rows 89 + 90: refresh the DCL arbitration overlay FIRST, so any instance this same
// sweep pass goes on to start resolves its connections against the current definitions
// rather than against the snapshot frozen into its flattened config. This is the
// convergence path for a definition row that arrived by LocalDb CDC replication: no new
// cross-node message exists (or is needed) — whichever node runs the singleton reads the
// row out of its own file on the next tick.
ApplyConnectionDefinitionOverlayIfFresh(
msg.ConnectionDefinitions, msg.DefinitionsGenerationAtRead, "instance sweep");
ApplyConnectionDefinitionOverlay(msg.ConnectionDefinitions);
foreach (var config in msg.Configs)
{
@@ -1888,7 +1985,18 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
UpdateInstanceCounts();
var sender = replyTo;
_storage.RemoveDeployedConfigAsync(instanceName).ContinueWith(t =>
// Row 101 (initiation half): the actor has already forgotten the instance (actor stopped,
// _deployedInstanceNames/_runningRevisions/_sweepAttemptedRevisions cleared) while the
// durable row is still there — and clearing _sweepAttemptedRevisions specifically hands
// the sweep a fresh repair claim, so an in-flight read landing now would resurrect the
// instance the operator just deleted.
BumpStoreWriteGeneration("delete dispatch");
var deleteWrite = _storage.RemoveDeployedConfigAsync(instanceName);
// Row 101 (completion half) — see StoreWriteCompleted.
deleteWrite.ContinueWith(_ => new StoreWriteCompleted("delete")).PipeTo(Self);
deleteWrite.ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
{
@@ -2228,45 +2336,96 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
private readonly HashSet<string> _centrallyRemovedConnections = new(StringComparer.Ordinal);
/// <summary>
/// Monotonic count of actor-thread definition mutations from artifact applies (deferred
/// register row 101). A store snapshot is read OFF the actor thread and compared against
/// the overlay only when its piped message lands — so a read that started before an
/// artifact apply's store writes can land AFTER <see
/// cref="HandleApplyArtifactDataConnectionsToDcl"/> populated the overlay, and its stale
/// data would then "prove" the just-applied names disappeared, tombstoning and stopping
/// live connections that nothing re-creates until an instance redeploy. Each snapshot read
/// captures this value on the actor thread at dispatch; a result carrying a stale
/// generation skips the overlay apply entirely (additions too — a stale row pushed via
/// <see cref="EnsureDclConnection"/> would reconfigure a live connection BACKWARD and set
/// up a flip-flop with the next fresh read). The next tick reads fresh and converges.
/// Monotonic count of this actor's own mutations of the two tables
/// <see cref="ReadStoreSnapshotAsync"/> reads — <c>deployed_configurations</c> AND
/// <c>data_connection_definitions</c> (deferred register row 101, extended to the config
/// half). A store snapshot is read OFF the actor thread but consumed only when its piped
/// message lands, so a read dispatched before a mutation can land after the actor's own
/// state has already advanced past it. Every consumer therefore captures this value on the
/// actor thread at read dispatch and discards the WHOLE snapshot on a mismatch.
///
/// <para>
/// <b>One counter for both tables, deliberately.</b> The snapshot is one consistent read and
/// is discarded whole, so a per-table counter would buy nothing but the chance to apply half
/// a snapshot that the other half's mutation already invalidated.
/// </para>
///
/// <para>
/// <b>What a stale snapshot would actually do</b> — each of these is a real defect the guard
/// closes, not a theoretical one:
/// <list type="bullet">
/// <item>DEFINITIONS half: absences "prove" just-applied names disappeared, tombstoning and
/// STOPPING live connections that nothing re-creates until an instance redeploy; rows push
/// superseded configs onto live connections and flip-flop with the next fresh read.</item>
/// <item>CONFIGS half, row-72 leg: a snapshot carrying the revision a deploy has just
/// replaced makes <see cref="TryRestartStaleInstance"/> "repair" the instance BACKWARD —
/// it redeploys the snapshot's own (old) config JSON through the redeploy path.
/// <see cref="InstanceIsOwnedByADeployPipeline"/> does not cover this: it only holds while
/// the deploy is in flight, and the stale snapshot can land after the pipeline released
/// ownership.</item>
/// <item>CONFIGS half, row-78 leg: a snapshot read before an undeploy/delete or a disable
/// committed still shows an enabled row while the actor is already stopped, so
/// <see cref="StartStoredInstance"/> RESURRECTS a just-stopped instance.</item>
/// </list>
/// </para>
///
/// <para>
/// <b>Bumped twice per mutation — at initiation and at completion</b> (see
/// <see cref="BumpStoreWriteGeneration"/>), which is what makes the guard airtight rather
/// than merely usually-right. The store write runs off the actor thread, so a read
/// dispatched between "the actor decided to mutate" and "the write committed" would compare
/// EQUAL against an initiation-only bump and be applied even though it predates the write.
/// Bumping again on the completion message — which is created only after the write has
/// committed — makes any such in-window read compare stale. Consequently: a read dispatched
/// before the mutation is discarded, a read dispatched during it is discarded, and a read
/// dispatched after the completion bump necessarily sees the committed row.
/// </para>
///
/// <para>
/// <b>Writes that must NOT bump.</b> Rows arriving by LocalDb CDC replication, by a snapshot
/// resync, by an operator restoring the file, or by <c>SiteReconciliationActor</c>'s
/// startup fetch never pass through this actor at all — and discovering exactly those is the
/// instance reconcile sweep's whole reason to exist (register rows 72 + 78). They are not
/// "stale relative to actor state"; they are the state the actor has yet to learn about.
/// Bumping for them (were it even possible) would blind the sweep to its own use case.
/// </para>
/// </summary>
private long _definitionsWriteGeneration;
private long _storeWriteGeneration;
/// <summary>
/// Row 101 guard in front of <see cref="ApplyConnectionDefinitionOverlay"/>: applies the
/// snapshot only when no artifact apply mutated the definitions while the read was in
/// flight. A stale snapshot is discarded WHOLE — its absences would spuriously tombstone
/// and stop just-applied live connections (which nothing re-creates until an instance
/// redeploy), and its rows would push superseded configs onto live connections. Discarding
/// is safe because the artifact apply already updated the overlay itself and the next
/// tick's read starts after the write, so convergence is one tick away.
/// Records that this actor has mutated one of the two snapshot tables, invalidating every
/// store read currently in flight. Called on the actor thread at BOTH ends of a mutation —
/// when the off-thread write is dispatched and again when its completion message lands. See
/// <see cref="_storeWriteGeneration"/> for why both ends are required.
/// </summary>
private void ApplyConnectionDefinitionOverlayIfFresh(
IReadOnlyList<StoredDataConnectionDefinition>? definitions,
long generationAtRead,
string source)
/// <param name="source">Provenance for the Trace line; never affects behaviour.</param>
private void BumpStoreWriteGeneration(string source)
{
if (generationAtRead != _definitionsWriteGeneration)
{
_logger.LogDebug(
"Discarding the {Source} data-connection definitions snapshot — an artifact "
+ "apply changed the definitions while the read was in flight (generation "
+ "{ReadGeneration} vs {CurrentGeneration}); the next sweep tick reads fresh",
source, generationAtRead, _definitionsWriteGeneration);
return;
}
_storeWriteGeneration++;
_logger.LogTrace(
"Store write generation bumped to {Generation} by {Source}", _storeWriteGeneration, source);
}
ApplyConnectionDefinitionOverlay(definitions);
/// <summary>
/// The row-101 freshness test both snapshot consumers run BEFORE touching anything the
/// snapshot carries. <see langword="true"/> means this actor mutated one of the two tables
/// while the read was in flight, so the snapshot predates state the actor has already
/// advanced past and must be discarded whole (see <see cref="_storeWriteGeneration"/> for
/// what each half would otherwise do). Logged at Debug: on a healthy site this fires only
/// when a deploy/enable/disable/delete happens to overlap a sweep tick, which is normal and
/// self-correcting.
/// </summary>
private bool IsStoreSnapshotStale(long generationAtRead, string source)
{
if (generationAtRead == _storeWriteGeneration)
return false;
_logger.LogDebug(
"Discarding the {Source} store snapshot — this node mutated the deployed configs or "
+ "the connection definitions while the read was in flight (generation "
+ "{ReadGeneration} vs {CurrentGeneration})",
source, generationAtRead, _storeWriteGeneration);
return true;
}
/// <summary>
@@ -2920,6 +3079,12 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// data-connections branch below dispatches via this captured ref.
var self = Self;
// Row 101 (initiation half): the artifact apply rewrites data_connection_definitions
// off-thread. The completion half is the ApplyArtifactDataConnectionsToDcl handler; this
// bump covers the window before it, where a read would otherwise land carrying the
// pre-apply table and be treated as current.
BumpStoreWriteGeneration("artifact apply dispatch");
Task.Run(async () =>
{
try
@@ -3121,9 +3286,10 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private void HandleApplyArtifactDataConnectionsToDcl(ApplyArtifactDataConnectionsToDcl msg)
{
// Row 101: any snapshot read dispatched before this point carries data that predates
// these mutations — mark it stale so its overlay half is discarded when it lands.
_definitionsWriteGeneration++;
// Row 101 (completion half): the definitions rows are committed by the time this message
// is dispatched, so any snapshot read that started earlier carries data predating them —
// mark it stale so it is discarded when it lands.
BumpStoreWriteGeneration("artifact apply");
foreach (var dc in msg.DataConnections)
{
@@ -3216,16 +3382,18 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// is the deploy-only site shape (see <see cref="ApplyConnectionDefinitionOverlay"/>).
/// </para>
/// </summary>
/// <param name="DefinitionsGenerationAtRead">
/// Row 101: <see cref="_definitionsWriteGeneration"/> captured on the actor thread when the
/// read was dispatched. A mismatch on landing means an artifact apply mutated the overlay
/// while this read was in flight — the snapshot's definitions half is stale and is skipped.
/// <param name="StoreGenerationAtRead">
/// Row 101: <see cref="_storeWriteGeneration"/> captured on the actor thread when the read
/// was dispatched. A mismatch on landing means this actor mutated the deployed configs or
/// the connection definitions while the read was in flight, so the WHOLE snapshot predates
/// the actor's own state. The boot load cannot simply drop it (startup has to finish), so
/// <see cref="HandleStartupConfigsLoaded"/> re-reads instead.
/// </param>
internal record StartupConfigsLoaded(
List<DeployedInstance> Configs,
string? Error,
IReadOnlyList<StoredDataConnectionDefinition>? ConnectionDefinitions = null,
long DefinitionsGenerationAtRead = 0);
long StoreGenerationAtRead = 0);
/// <summary>
/// Internal singleton signal that re-attempts the startup deployed-config load
@@ -3318,14 +3486,26 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// contract as <see cref="StartupConfigsLoaded"/>: null = read failed, keep the overlay.
/// </para>
/// </summary>
/// <param name="DefinitionsGenerationAtRead">
/// <param name="StoreGenerationAtRead">
/// Row 101: same stale-snapshot guard as <see cref="StartupConfigsLoaded"/> — see there.
/// The sweep's copy is DROPPED on a mismatch rather than re-read: there is nothing to
/// complete, and the next tick is already a fresh read.
/// </param>
internal record InstanceSweepLoaded(
List<DeployedInstance> Configs,
string? Error,
IReadOnlyList<StoredDataConnectionDefinition>? ConnectionDefinitions = null,
long DefinitionsGenerationAtRead = 0);
long StoreGenerationAtRead = 0);
/// <summary>
/// Row 101: piped back to self once an off-thread mutation of one of the two snapshot tables
/// has COMMITTED, purely so <see cref="_storeWriteGeneration"/> can be bumped on the actor
/// thread at a point that provably happens-after the write. Used by the mutation paths that
/// have no other actor-thread completion message of their own (disable, delete, the
/// init-failure rollback); paths that already pipe a result bump in that handler instead.
/// Carries only a provenance string, for the Trace line.
/// </summary>
internal sealed record StoreWriteCompleted(string Source);
internal record DeployPersistenceResult(
string DeploymentId, string InstanceName, bool Success, string? Error,