fix(siteruntime): extend the row-101 stale-snapshot guard to the instance-config half
ReadStoreSnapshotAsync's snapshot is read off the actor thread and consumed only when its piped message lands. Row 101 guarded the DEFINITIONS half of that race; the CONFIG half was left to machinery whose reach is time-limited, and both legs of the instance reconcile sweep were reachable through it: - Row-72 leg: a read dispatched before a deploy's persist, landing after the deploy completed AND after the pipeline released ownership, carries the revision the deploy replaced. TryRestartStaleInstance then "repairs" the instance BACKWARD, re-deploying the snapshot's own superseded ConfigJson. InstanceIsOwnedByADeployPipeline covers only in-flight deploys, and the per-revision claim has never seen the old revision, so nothing stopped it. - Row-78 leg: a read dispatched before a delete/disable commits still shows an enabled row while the actor is already stopped, so StartStoredInstance resurrects it — and delete clears _sweepAttemptedRevisions, freeing the claim. Both verified real by neutralising the freshness check and re-running the new tests (the fail-before site event literally reads "running revision rev-2 is stale, applying stored revision rev-1"; the deleted instance came back). _definitionsWriteGeneration becomes _storeWriteGeneration, covering both tables (the snapshot is one consistent read and is discarded whole, so one counter is the right shape); DefinitionsGenerationAtRead -> StoreGenerationAtRead on both internal records. The generation is now bumped at BOTH ends of every mutation — when the off-thread write is dispatched and when its completion message lands — because the config paths advance actor state before the write, so an initiation-only bump would let an in-window read look fresh. Disable, delete and the init-failure rollback have no actor-thread completion message of their own, so a minimal StoreWriteCompleted self-message carries the bump for them. CDC-replicated writes, snapshot resyncs, restored files and SiteReconciliationActor's fetch deliberately do NOT bump: discovering exactly those is the sweep's reason to exist. A stale sweep result is now dropped WHOLE (no overlay apply, no start, no restart). A stale boot load cannot be dropped — startup must finish — so it re-dispatches LoadDeployedConfigs(); a fifth consecutive superseded read is applied anyway with a Warning, and a late one arriving after the boot completed is dropped rather than looping against the S5 retry timer. Tests: three fail-before regressions (backward restart, resurrection, boot re-read) plus two guard twins pinning that the legitimate row-72 restart and row-78 start still fire on a current-generation snapshot. The row-101 definitions tests keep passing; FreshEmptySweepSnapshot's hardcoded generation moves 1 -> 2 now that one artifact apply bumps at both ends. Docs: Component-SiteRuntime.md's stale-snapshot paragraph now describes the whole snapshot (both halves, both consumers, the boot re-dispatch, the CDC-no-bump rule), the components mirror follows, and the arbitration memo gains §11 extending §10 to the config half.
This commit is contained in:
@@ -91,7 +91,7 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration`
|
||||
|
||||
> **`CreateInstanceActor` is the single funnel for the connection push too (register row 85)**: the `EnsureDclConnections` call moved *into* `CreateInstanceActor` on 2026-08-17. It used to sit at each caller, and only two of the four callers made it — the boot-time batch and this deploy path. `StartStoredInstance` (row-67 ensure-started and the row-78 sweep) and `HandleEnableResult` did not, so a self-healed or newly-enabled instance came up with no connection actors and every tag subscribe was answered `Unknown connection: <name>` indefinitely. See the *Instance reconcile sweep* section and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
|
||||
|
||||
> **Which config source wins (register rows 89 + 90, 2026-08-17)**: `EnsureDclConnections` no longer trusts the flattened snapshot blindly. `data_connection_definitions` is read on the same off-thread task as the deployed configs, at boot and on every sweep tick, into an actor-thread-confined **arbitration overlay**: a connection name with a row there is configured from that row, and only a name with no row uses the instance's flattened `Connections` entry (which is the correct source on a deploy-only site, where that table is legitimately empty). The winner is resolved **before** the per-name hash is computed, which is what makes the two-source ping-pong — the reason a store-level sweep was rejected in §7.4 — unrepresentable rather than merely avoided. Removals additionally leave an in-memory tombstone so a centrally deleted connection is not resurrected by the next instance start. A monotonic write-generation guard (row 101) discards a snapshot whose off-thread read raced an artifact apply — its stale absences would otherwise tombstone and stop just-applied live connections that nothing re-creates until an instance redeploy. **The register's premise was wrong in two ways** (the standby's DCL is empty rather than stale, so the real defect is create-time on the next singleton incarnation and is not replication-specific; and a re-issued create never reconfigured a *live* connection actor at all). See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
|
||||
> **Which config source wins (register rows 89 + 90, 2026-08-17)**: `EnsureDclConnections` no longer trusts the flattened snapshot blindly. `data_connection_definitions` is read on the same off-thread task as the deployed configs, at boot and on every sweep tick, into an actor-thread-confined **arbitration overlay**: a connection name with a row there is configured from that row, and only a name with no row uses the instance's flattened `Connections` entry (which is the correct source on a deploy-only site, where that table is legitimately empty). The winner is resolved **before** the per-name hash is computed, which is what makes the two-source ping-pong — the reason a store-level sweep was rejected in §7.4 — unrepresentable rather than merely avoided. Removals additionally leave an in-memory tombstone so a centrally deleted connection is not resurrected by the next instance start. A monotonic store-write-generation guard (row 101) discards — **whole** — any snapshot whose off-thread read raced one of this node's own mutations of *either* table, bumping the generation both when a write is dispatched and when its completion message lands so no in-flight read can look fresh. It covers both halves: a stale definitions half would tombstone and stop just-applied live connections that nothing re-creates until an instance redeploy, and a stale configs half would restart an instance **backward** onto the revision a deploy just replaced or resurrect one a disable/delete just stopped (the deploy-pipeline-ownership check does not reach either, since it only holds while a deploy is in flight). Writes that never pass through the actor — CDC replication, snapshot resync, a restored file, the startup reconcile fetch — deliberately do **not** bump, because discovering those is the sweep's whole purpose. A stale sweep result is dropped; a stale boot load is **re-read** instead, since startup must complete. **The register's premise was wrong in two ways** (the standby's DCL is empty rather than stale, so the real defect is create-time on the next singleton incarnation and is not replication-specific; and a re-issued create never reconfigured a *live* connection actor at all). See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
|
||||
|
||||
> **A changed config reaches the LIVE connection (register row 97)**: resolving the winner decides *what* to push; a third command was needed to make a push against an already-live connection mean anything, because `DataConnectionManagerActor.HandleCreateConnection` warns and returns for a name it already holds. `EnsureDclConnection` now has three outcomes off the one per-name hash slot — **unchanged → nothing**, **new → `CreateConnectionCommand`**, **changed → `ReconfigureConnectionCommand`** — and the connection actor applies the last of these in place by swapping its adapter through the failover machinery and re-issuing its subscriptions. Log line: `Reconfigured DCL connection {Connection} (protocol={Protocol}) in place` (Information), replacing the "will not take effect" Warning rows 89/90 added. See `DataConnectionLayer.md` → *Reconfiguring a live connection*.
|
||||
|
||||
|
||||
@@ -412,3 +412,113 @@ Applies to both snapshot consumers (boot load and sweep). Deterministic reproduc
|
||||
test that the legitimate row-90 disappearance still fires on a FRESH empty snapshot:
|
||||
`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections` (fails before the
|
||||
fix) and `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections`.
|
||||
|
||||
## 11. Postscript — the same race on the CONFIG half of the snapshot (register row 101 follow-on, 2026-08-18)
|
||||
|
||||
§10 fixed the DEFINITIONS half of `ReadStoreSnapshotAsync`'s snapshot. The **instance-config**
|
||||
half of the same snapshot has the identical TOCTOU, and it was left protected only by machinery
|
||||
whose reach is time-limited. This section extends §10 rather than revising it; everything §10
|
||||
says about the definitions half still stands.
|
||||
|
||||
### 11.1 The two hazards, both verified real against the code
|
||||
|
||||
Both were reproduced by neutralising the freshness check and re-running the new regression tests
|
||||
(see §11.5), so neither is theoretical.
|
||||
|
||||
**Hazard 1 — backward restart (row-72 leg).** A read dispatched before a deploy persists its new
|
||||
row can land after the deploy has completed *and* after the pipeline released the instance. The
|
||||
snapshot then carries the revision the deploy replaced while `_runningRevisions` carries the new
|
||||
one — the exact shape `TryRestartStaleInstance` repairs, in the wrong direction. It is not a
|
||||
harmless restart: the repair passes **the snapshot's own `ConfigJson`** to `HandleDeploy`, so the
|
||||
superseded config is literally re-deployed. The pre-existing guards do not reach it:
|
||||
`InstanceIsOwnedByADeployPipeline` covers only `_deployWarms` / `_terminatingActorsByName` /
|
||||
`_inFlightDeployments`, all of which are cleared when the deploy finishes; and
|
||||
`TryClaimSweepAttempt` does bound restarts to one per (instance, revision), but the *old* revision
|
||||
has never been claimed, so the claim is available and the instance simply stays reverted.
|
||||
Observed fail-before site event: *"Instance BackwardRestartPump restarted by deployment-manager
|
||||
sweep self-heal — running revision rev-2 is stale, applying stored revision rev-1"*.
|
||||
|
||||
**Hazard 2 — resurrection (row-78 leg).** A read dispatched before a delete or a disable commits
|
||||
still shows an enabled row while the actor is already stopped (both handlers stop the actor
|
||||
synchronously and write off-thread). `StartStoredInstance` then starts an instance the operator
|
||||
just removed. Delete is the worse of the two because it explicitly clears
|
||||
`_sweepAttemptedRevisions` for the instance, handing the sweep a fresh repair claim. Fail-before:
|
||||
the deleted instance was running again after the stale snapshot landed.
|
||||
|
||||
**Hazard 3 — the boot load's other consumers.** `HandleStartupConfigsLoaded` also rebuilds
|
||||
`_deployedInstanceNames` from the snapshot and hands its enabled configs to the staggered startup
|
||||
batch. A deploy or delete processed while the boot read is in flight therefore skews the
|
||||
deployed-instance count and can hand the batch a superseded config; `CreateInstanceActor`'s
|
||||
already-exists guard blunts the second effect but not the first.
|
||||
|
||||
### 11.2 One generation for both tables
|
||||
|
||||
`_definitionsWriteGeneration` is renamed `_storeWriteGeneration` and now counts mutations of BOTH
|
||||
snapshot tables; the message parameter `DefinitionsGenerationAtRead` becomes
|
||||
`StoreGenerationAtRead` on both internal records (neither is a wire message). One counter is the
|
||||
right shape because the snapshot is a single consistent read and is discarded whole either way —
|
||||
a per-table counter would only buy the ability to apply half a snapshot that the other half's
|
||||
mutation already invalidated.
|
||||
|
||||
### 11.3 Bump at BOTH ends of every mutation
|
||||
|
||||
§10 bumped once, in the handler that runs after the write. That is sufficient when the actor's
|
||||
own state advances in that same handler, but the config mutations do not all work that way —
|
||||
`ApplyDeployment` creates the Instance Actor and stamps the running revision *before* the
|
||||
persist, and `HandleDisable`/`HandleDelete` stop the actor *before* the write. So the generation
|
||||
is now bumped twice per mutation: once when the off-thread write is **dispatched**, once when its
|
||||
**completion** message lands. The pair leaves no window — a read dispatched before the mutation is
|
||||
discarded by the first bump, a read dispatched inside the write window is discarded by the second,
|
||||
and a read dispatched after the second necessarily sees the committed row. Two paths had no
|
||||
actor-thread completion message at all (disable/delete pipe their reply to the *caller*, and the
|
||||
init-failure rollback pipes nothing), so a minimal internal `StoreWriteCompleted(Source)` message
|
||||
exists purely to carry the completion bump for them.
|
||||
|
||||
Bump sites: `ApplyDeployment` + `HandleDeployPersistenceResult`; `HandleEnable` +
|
||||
`HandleEnableResult`; `HandleDisable` + `StoreWriteCompleted`; `HandleDelete` +
|
||||
`StoreWriteCompleted`; `RollbackPersistedConfig` + `StoreWriteCompleted`; `HandleDeployArtifacts`
|
||||
+ `HandleApplyArtifactDataConnectionsToDcl` (the §10 bump, kept). The deploy and enable completion
|
||||
bumps are **unconditional on success** — a failed write may still have committed the row before a
|
||||
later step threw, so "failed" is not "nothing changed". Cost of over-bumping is at most one
|
||||
discarded sweep tick or one extra boot read.
|
||||
|
||||
**Deliberately NOT bumped:** `SiteReconciliationActor`'s `StoreDeployedConfigIfNewerAsync` (a
|
||||
different actor), LocalDb CDC replication applies, snapshot resyncs, and an operator restoring the
|
||||
file. None passes through this actor, and discovering exactly those divergences is the sweep's
|
||||
entire reason to exist (rows 72 + 78). Bumping for them — were it even reachable — would blind the
|
||||
sweep to its own use case.
|
||||
|
||||
### 11.4 Sweep drops, boot re-reads
|
||||
|
||||
The sweep consumer discards a stale result **whole** (definitions half and configs half, no
|
||||
overlay apply, no start, no restart) at Debug, because there is nothing to complete and the next
|
||||
tick is already a fresh read. The boot consumer cannot do that — a node whose startup read is
|
||||
dropped stays a silent zero-instance node — so it **re-dispatches `LoadDeployedConfigs()`**, which
|
||||
captures the current generation. Convergence argument: staleness requires a fresh mutation inside
|
||||
each attempt's window, which is externally driven and finite. The belt-and-braces cap is
|
||||
`MaxConsecutiveStaleStartupLoads` = 5, after which the snapshot is applied anyway with a Warning
|
||||
(a boot that never finishes is strictly worse than one starting from a slightly superseded view
|
||||
the sweep converges on its next tick). A superseded boot snapshot arriving *after*
|
||||
`_startupLoadCompleted` — the S5 `RetryStartupLoad` timer racing a success — is dropped rather
|
||||
than re-read, so the retry timer and the re-dispatch cannot feed each other.
|
||||
|
||||
### 11.5 Tests
|
||||
|
||||
In `InstanceSweepSelfHealTests`, all hand-delivering crafted `InstanceSweepLoaded` /
|
||||
`StartupConfigsLoaded` messages with an explicit generation:
|
||||
|
||||
- `StaleSweepSnapshot_CarryingASupersededRevision_DoesNotRestartBackward` — fails before.
|
||||
- `StaleSweepSnapshot_ShowingADeletedInstance_DoesNotResurrectIt` — fails before (full
|
||||
deploy-then-delete shape, not the weaker never-started equivalent).
|
||||
- `StaleBootSnapshot_IsReReadRatherThanApplied` — fails before (one read instead of two); a slow
|
||||
first read plus a lifecycle command processed during it makes the race deterministic, and the
|
||||
test also pins that a LATE superseded snapshot is dropped rather than looping.
|
||||
- `FreshSweepSnapshot_CarryingANewerRevision_StillRestartsTheInstance` and
|
||||
`FreshSweepSnapshot_ShowingAStoredEnabledInstance_StillStartsIt` — the guard twins, pinning that
|
||||
legitimate row-72 and row-78 repairs survive. Both start the instance from the BOOT LOAD rather
|
||||
than a deploy command, precisely so no mutation passes through the actor and generation `0` is
|
||||
genuinely current.
|
||||
|
||||
`FreshEmptySweepSnapshot_StillRemovesDisappearedConnections` (§10) still pins the definitions
|
||||
half; its hardcoded current generation moves from `1` to `2` because one artifact apply now bumps
|
||||
at both ends.
|
||||
|
||||
@@ -106,7 +106,7 @@ flowchart TD
|
||||
>
|
||||
> The **row-72 decision is to converge, not merely warn**: the stored row *is* what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing `HandleDeploy` means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy.
|
||||
>
|
||||
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
|
||||
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. One guard is **not** in that list because it needed adding later: deferring to an in-flight deploy pipeline says nothing about a snapshot that landed *after* the pipeline finished, which is the stale-snapshot race — a tick whose read raced this node's own deploy/enable/disable/delete is discarded whole by the store-write generation check (see **Stale-snapshot guard** below). A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
|
||||
|
||||
> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table — that stayed true when rows 89/90 gave the table a reader, because the overlay it feeds is an *arbiter*, not a creation source, and at the time of row 85 nothing read the table at all. A connection actor exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done at the time: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7. That pass is now present, in a form that resolves the conflict rather than creating it — see **DCL connection arbitration** immediately below.
|
||||
|
||||
@@ -118,7 +118,17 @@ flowchart TD
|
||||
>
|
||||
> **The ping-pong is killed by keying the hash on the winner.** The conflict §7.4 identified is a property of *where the hash is computed*, not of the sweep: every ensure path now resolves the winning configuration **before** hashing or pushing, so for any name only one configuration can ever occupy the hash slot regardless of which path calls. The flip is not guarded against — it is unrepresentable. `HandleApplyArtifactDataConnectionsToDcl` writes its rows into the overlay in the same pass it pushes them, closing the same flip arriving through the artifact-deploy door.
|
||||
>
|
||||
> **Stale-snapshot guard (row 101).** The snapshot is read off the actor thread but compared against the overlay only when its piped message lands — so a read that starts before an artifact apply's store writes can land *after* `HandleApplyArtifactDataConnectionsToDcl` populated the overlay, and its stale data would "prove" the just-applied names disappeared: tombstoned, live connection actors stopped, and — because the removal evicts the hash slot the overlay's push is gated on — **nothing re-creates them until an instance redeploy**. Each snapshot read therefore captures a monotonic definitions-write generation on the actor thread at dispatch (`_definitionsWriteGeneration`, bumped by every artifact apply), and a result carrying a stale generation is discarded **whole** — additions included, since a stale row pushed at a live connection would reconfigure it backward and flip-flop with the next fresh read. Discarding is safe: the artifact apply already updated the overlay itself, and the next tick reads fresh. Surfaced as a test flake (a raced sweep removed `PlcA` where the test expected `PlcGone`); the deterministic reproduction is `DclConnectionArbitrationTests.StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections`.
|
||||
> **Stale-snapshot guard — covers the WHOLE snapshot, both halves and both consumers (row 101).** `ReadStoreSnapshotAsync` reads `deployed_configurations` and `data_connection_definitions` off the actor thread but is consumed only when its piped message lands, so a read dispatched before a mutation can land after the actor's own state has already moved past it. Every read therefore captures a single monotonic **store-write generation** on the actor thread at dispatch (`_storeWriteGeneration`), and a snapshot carrying a stale generation is discarded **whole**. One counter covers both tables on purpose: the snapshot is one consistent read and is discarded whole anyway, so a per-table counter would only buy the chance to apply half a snapshot the other half's mutation already invalidated.
|
||||
>
|
||||
> Each half would do real damage. **Definitions half:** stale absences "prove" just-applied names disappeared — tombstoned, live connection actors stopped, and because the removal evicts the hash slot the overlay's push is gated on, **nothing re-creates them until an instance redeploy**; stale rows push superseded configs at live connections and flip-flop with the next fresh read. **Configs half, row-72 leg:** a snapshot carrying the revision a deploy just replaced makes the stale-revision repair run **backward**, re-applying the snapshot's own superseded config JSON through the redeploy path — and `InstanceIsOwnedByADeployPipeline` is no defence, because it only holds while a deploy is *in flight* and the stale snapshot can land after the pipeline released the instance. **Configs half, row-78 leg:** a snapshot read before an undeploy/delete or a disable committed still shows an enabled row while the actor is already stopped, so the start repair **resurrects a just-stopped instance** — and delete specifically clears `_sweepAttemptedRevisions`, so the once-per-revision claim is freshly available and does not stand in the way either. All three were reproduced by neutralising the guard and re-running the regression tests.
|
||||
>
|
||||
> **The generation is bumped twice per mutation — when the off-thread write is dispatched and again when its completion message lands on the actor thread.** The pair is what leaves no window: an initiation-only bump would let a read dispatched *between* the decision and the commit compare equal and be applied even though it predates the write, and a completion-only bump would miss a read dispatched before the handler ran. With both, a read dispatched before the mutation is discarded, a read dispatched during it is discarded, and a read dispatched after the completion bump necessarily sees the committed row. Bump sites are every actor-thread path that mutates either table: deploy persist (dispatch + `DeployPersistenceResult`), enable (dispatch + `EnableResult`), disable and delete and the init-failure rollback (dispatch + a `StoreWriteCompleted` self-message that exists purely to bump after the commit), and the artifact apply (dispatch + `ApplyArtifactDataConnectionsToDcl`).
|
||||
>
|
||||
> **Writes that must NOT bump**, and this is load-bearing: rows arriving by LocalDb CDC replication, by a snapshot resync, by an operator restoring the file, or by `SiteReconciliationActor`'s startup fetch never pass through this actor — and discovering exactly those is the sweep's entire reason to exist (rows 72 + 78). They are not stale relative to actor state; they are the state the actor has yet to learn about.
|
||||
>
|
||||
> **The two consumers differ.** A stale **sweep** result is dropped whole at Debug — no overlay apply, no start, no restart — because there is nothing to complete and the next tick is already a fresh read. A stale **boot** load cannot be dropped (a node whose startup read is discarded stays a silent zero-instance node), so it **re-dispatches `LoadDeployedConfigs()`**, capturing the current generation. Staleness requires a fresh mutation inside each attempt's window, so this converges in practice; as a livelock backstop the fifth consecutive superseded boot read is applied anyway with a Warning, on the grounds that a boot that never finishes is worse than one starting from a slightly superseded view that the sweep converges on its next tick. A superseded boot snapshot arriving *after* the boot has completed (the S5 retry timer racing a success) is dropped rather than re-read.
|
||||
>
|
||||
> Surfaced originally as a test flake (a raced sweep removed `PlcA` where the test expected `PlcGone`). Deterministic reproductions: `DclConnectionArbitrationTests.StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections` for the definitions half, and in `InstanceSweepSelfHealTests` — `StaleSweepSnapshot_CarryingASupersededRevision_DoesNotRestartBackward`, `StaleSweepSnapshot_ShowingADeletedInstance_DoesNotResurrectIt`, `StaleBootSnapshot_IsReReadRatherThanApplied`, each paired with a `Fresh…` twin pinning that the legitimate row-72 restart and row-78 start still happen.
|
||||
>
|
||||
> **Removal (row 90).** Both removal paths share one helper and now record a **tombstone** for the incarnation: the artifact reconcile's `RemovedConnectionNames`, and the sweep noticing that a name it had adopted no longer has a row (a removal it can *prove* — the store owned it a tick ago and does not now). `EnsureDclConnections` skips a tombstoned name, so a removal survives the next instance start instead of being undone by it — a pre-existing bug, since the removal evicted the hash slot but left no memory of why. Tombstones are **in-memory only** by decision: a durable marker would be a second divergent authority, and flattened configs already resurrect a removed connection across a restart. A definition row reappearing clears the tombstone.
|
||||
>
|
||||
|
||||
Reference in New Issue
Block a user