fix(siteruntime): extend the row-101 stale-snapshot guard to the instance-config half

ReadStoreSnapshotAsync's snapshot is read off the actor thread and consumed only
when its piped message lands. Row 101 guarded the DEFINITIONS half of that race;
the CONFIG half was left to machinery whose reach is time-limited, and both legs
of the instance reconcile sweep were reachable through it:

- Row-72 leg: a read dispatched before a deploy's persist, landing after the
  deploy completed AND after the pipeline released ownership, carries the
  revision the deploy replaced. TryRestartStaleInstance then "repairs" the
  instance BACKWARD, re-deploying the snapshot's own superseded ConfigJson.
  InstanceIsOwnedByADeployPipeline covers only in-flight deploys, and the
  per-revision claim has never seen the old revision, so nothing stopped it.
- Row-78 leg: a read dispatched before a delete/disable commits still shows an
  enabled row while the actor is already stopped, so StartStoredInstance
  resurrects it — and delete clears _sweepAttemptedRevisions, freeing the claim.

Both verified real by neutralising the freshness check and re-running the new
tests (the fail-before site event literally reads "running revision rev-2 is
stale, applying stored revision rev-1"; the deleted instance came back).

_definitionsWriteGeneration becomes _storeWriteGeneration, covering both tables
(the snapshot is one consistent read and is discarded whole, so one counter is
the right shape); DefinitionsGenerationAtRead -> StoreGenerationAtRead on both
internal records. The generation is now bumped at BOTH ends of every mutation —
when the off-thread write is dispatched and when its completion message lands —
because the config paths advance actor state before the write, so an
initiation-only bump would let an in-window read look fresh. Disable, delete and
the init-failure rollback have no actor-thread completion message of their own,
so a minimal StoreWriteCompleted self-message carries the bump for them.

CDC-replicated writes, snapshot resyncs, restored files and
SiteReconciliationActor's fetch deliberately do NOT bump: discovering exactly
those is the sweep's reason to exist.

A stale sweep result is now dropped WHOLE (no overlay apply, no start, no
restart). A stale boot load cannot be dropped — startup must finish — so it
re-dispatches LoadDeployedConfigs(); a fifth consecutive superseded read is
applied anyway with a Warning, and a late one arriving after the boot completed
is dropped rather than looping against the S5 retry timer.

Tests: three fail-before regressions (backward restart, resurrection, boot
re-read) plus two guard twins pinning that the legitimate row-72 restart and
row-78 start still fire on a current-generation snapshot. The row-101
definitions tests keep passing; FreshEmptySweepSnapshot's hardcoded generation
moves 1 -> 2 now that one artifact apply bumps at both ends.

Docs: Component-SiteRuntime.md's stale-snapshot paragraph now describes the
whole snapshot (both halves, both consumers, the boot re-dispatch, the
CDC-no-bump rule), the components mirror follows, and the arbitration memo gains
§11 extending §10 to the config half.
This commit is contained in:
Joseph Doherty
2026-08-18 05:38:00 -04:00
parent 4da5f7cd7a
commit f02e116688
6 changed files with 617 additions and 60 deletions
+1 -1
View File
@@ -91,7 +91,7 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration`
> **`CreateInstanceActor` is the single funnel for the connection push too (register row 85)**: the `EnsureDclConnections` call moved *into* `CreateInstanceActor` on 2026-08-17. It used to sit at each caller, and only two of the four callers made it — the boot-time batch and this deploy path. `StartStoredInstance` (row-67 ensure-started and the row-78 sweep) and `HandleEnableResult` did not, so a self-healed or newly-enabled instance came up with no connection actors and every tag subscribe was answered `Unknown connection: <name>` indefinitely. See the *Instance reconcile sweep* section and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
> **Which config source wins (register rows 89 + 90, 2026-08-17)**: `EnsureDclConnections` no longer trusts the flattened snapshot blindly. `data_connection_definitions` is read on the same off-thread task as the deployed configs, at boot and on every sweep tick, into an actor-thread-confined **arbitration overlay**: a connection name with a row there is configured from that row, and only a name with no row uses the instance's flattened `Connections` entry (which is the correct source on a deploy-only site, where that table is legitimately empty). The winner is resolved **before** the per-name hash is computed, which is what makes the two-source ping-pong — the reason a store-level sweep was rejected in §7.4 — unrepresentable rather than merely avoided. Removals additionally leave an in-memory tombstone so a centrally deleted connection is not resurrected by the next instance start. A monotonic write-generation guard (row 101) discards a snapshot whose off-thread read raced an artifact apply — its stale absences would otherwise tombstone and stop just-applied live connections that nothing re-creates until an instance redeploy. **The register's premise was wrong in two ways** (the standby's DCL is empty rather than stale, so the real defect is create-time on the next singleton incarnation and is not replication-specific; and a re-issued create never reconfigured a *live* connection actor at all). See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
> **Which config source wins (register rows 89 + 90, 2026-08-17)**: `EnsureDclConnections` no longer trusts the flattened snapshot blindly. `data_connection_definitions` is read on the same off-thread task as the deployed configs, at boot and on every sweep tick, into an actor-thread-confined **arbitration overlay**: a connection name with a row there is configured from that row, and only a name with no row uses the instance's flattened `Connections` entry (which is the correct source on a deploy-only site, where that table is legitimately empty). The winner is resolved **before** the per-name hash is computed, which is what makes the two-source ping-pong — the reason a store-level sweep was rejected in §7.4 — unrepresentable rather than merely avoided. Removals additionally leave an in-memory tombstone so a centrally deleted connection is not resurrected by the next instance start. A monotonic store-write-generation guard (row 101) discards — **whole** — any snapshot whose off-thread read raced one of this node's own mutations of *either* table, bumping the generation both when a write is dispatched and when its completion message lands so no in-flight read can look fresh. It covers both halves: a stale definitions half would tombstone and stop just-applied live connections that nothing re-creates until an instance redeploy, and a stale configs half would restart an instance **backward** onto the revision a deploy just replaced or resurrect one a disable/delete just stopped (the deploy-pipeline-ownership check does not reach either, since it only holds while a deploy is in flight). Writes that never pass through the actor — CDC replication, snapshot resync, a restored file, the startup reconcile fetch — deliberately do **not** bump, because discovering those is the sweep's whole purpose. A stale sweep result is dropped; a stale boot load is **re-read** instead, since startup must complete. **The register's premise was wrong in two ways** (the standby's DCL is empty rather than stale, so the real defect is create-time on the next singleton incarnation and is not replication-specific; and a re-issued create never reconfigured a *live* connection actor at all). See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
> **A changed config reaches the LIVE connection (register row 97)**: resolving the winner decides *what* to push; a third command was needed to make a push against an already-live connection mean anything, because `DataConnectionManagerActor.HandleCreateConnection` warns and returns for a name it already holds. `EnsureDclConnection` now has three outcomes off the one per-name hash slot — **unchanged → nothing**, **new → `CreateConnectionCommand`**, **changed → `ReconfigureConnectionCommand`** — and the connection actor applies the last of these in place by swapping its adapter through the failover machinery and re-issuing its subscriptions. Log line: `Reconfigured DCL connection {Connection} (protocol={Protocol}) in place` (Information), replacing the "will not take effect" Warning rows 89/90 added. See `DataConnectionLayer.md` → *Reconfiguring a live connection*.
@@ -412,3 +412,113 @@ Applies to both snapshot consumers (boot load and sweep). Deterministic reproduc
test that the legitimate row-90 disappearance still fires on a FRESH empty snapshot:
`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections` (fails before the
fix) and `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections`.
## 11. Postscript — the same race on the CONFIG half of the snapshot (register row 101 follow-on, 2026-08-18)
§10 fixed the DEFINITIONS half of `ReadStoreSnapshotAsync`'s snapshot. The **instance-config**
half of the same snapshot has the identical TOCTOU, and it was left protected only by machinery
whose reach is time-limited. This section extends §10 rather than revising it; everything §10
says about the definitions half still stands.
### 11.1 The two hazards, both verified real against the code
Both were reproduced by neutralising the freshness check and re-running the new regression tests
(see §11.5), so neither is theoretical.
**Hazard 1 — backward restart (row-72 leg).** A read dispatched before a deploy persists its new
row can land after the deploy has completed *and* after the pipeline released the instance. The
snapshot then carries the revision the deploy replaced while `_runningRevisions` carries the new
one — the exact shape `TryRestartStaleInstance` repairs, in the wrong direction. It is not a
harmless restart: the repair passes **the snapshot's own `ConfigJson`** to `HandleDeploy`, so the
superseded config is literally re-deployed. The pre-existing guards do not reach it:
`InstanceIsOwnedByADeployPipeline` covers only `_deployWarms` / `_terminatingActorsByName` /
`_inFlightDeployments`, all of which are cleared when the deploy finishes; and
`TryClaimSweepAttempt` does bound restarts to one per (instance, revision), but the *old* revision
has never been claimed, so the claim is available and the instance simply stays reverted.
Observed fail-before site event: *"Instance BackwardRestartPump restarted by deployment-manager
sweep self-heal — running revision rev-2 is stale, applying stored revision rev-1"*.
**Hazard 2 — resurrection (row-78 leg).** A read dispatched before a delete or a disable commits
still shows an enabled row while the actor is already stopped (both handlers stop the actor
synchronously and write off-thread). `StartStoredInstance` then starts an instance the operator
just removed. Delete is the worse of the two because it explicitly clears
`_sweepAttemptedRevisions` for the instance, handing the sweep a fresh repair claim. Fail-before:
the deleted instance was running again after the stale snapshot landed.
**Hazard 3 — the boot load's other consumers.** `HandleStartupConfigsLoaded` also rebuilds
`_deployedInstanceNames` from the snapshot and hands its enabled configs to the staggered startup
batch. A deploy or delete processed while the boot read is in flight therefore skews the
deployed-instance count and can hand the batch a superseded config; `CreateInstanceActor`'s
already-exists guard blunts the second effect but not the first.
### 11.2 One generation for both tables
`_definitionsWriteGeneration` is renamed `_storeWriteGeneration` and now counts mutations of BOTH
snapshot tables; the message parameter `DefinitionsGenerationAtRead` becomes
`StoreGenerationAtRead` on both internal records (neither is a wire message). One counter is the
right shape because the snapshot is a single consistent read and is discarded whole either way —
a per-table counter would only buy the ability to apply half a snapshot that the other half's
mutation already invalidated.
### 11.3 Bump at BOTH ends of every mutation
§10 bumped once, in the handler that runs after the write. That is sufficient when the actor's
own state advances in that same handler, but the config mutations do not all work that way —
`ApplyDeployment` creates the Instance Actor and stamps the running revision *before* the
persist, and `HandleDisable`/`HandleDelete` stop the actor *before* the write. So the generation
is now bumped twice per mutation: once when the off-thread write is **dispatched**, once when its
**completion** message lands. The pair leaves no window — a read dispatched before the mutation is
discarded by the first bump, a read dispatched inside the write window is discarded by the second,
and a read dispatched after the second necessarily sees the committed row. Two paths had no
actor-thread completion message at all (disable/delete pipe their reply to the *caller*, and the
init-failure rollback pipes nothing), so a minimal internal `StoreWriteCompleted(Source)` message
exists purely to carry the completion bump for them.
Bump sites: `ApplyDeployment` + `HandleDeployPersistenceResult`; `HandleEnable` +
`HandleEnableResult`; `HandleDisable` + `StoreWriteCompleted`; `HandleDelete` +
`StoreWriteCompleted`; `RollbackPersistedConfig` + `StoreWriteCompleted`; `HandleDeployArtifacts`
+ `HandleApplyArtifactDataConnectionsToDcl` (the §10 bump, kept). The deploy and enable completion
bumps are **unconditional on success** — a failed write may still have committed the row before a
later step threw, so "failed" is not "nothing changed". Cost of over-bumping is at most one
discarded sweep tick or one extra boot read.
**Deliberately NOT bumped:** `SiteReconciliationActor`'s `StoreDeployedConfigIfNewerAsync` (a
different actor), LocalDb CDC replication applies, snapshot resyncs, and an operator restoring the
file. None passes through this actor, and discovering exactly those divergences is the sweep's
entire reason to exist (rows 72 + 78). Bumping for them — were it even reachable — would blind the
sweep to its own use case.
### 11.4 Sweep drops, boot re-reads
The sweep consumer discards a stale result **whole** (definitions half and configs half, no
overlay apply, no start, no restart) at Debug, because there is nothing to complete and the next
tick is already a fresh read. The boot consumer cannot do that — a node whose startup read is
dropped stays a silent zero-instance node — so it **re-dispatches `LoadDeployedConfigs()`**, which
captures the current generation. Convergence argument: staleness requires a fresh mutation inside
each attempt's window, which is externally driven and finite. The belt-and-braces cap is
`MaxConsecutiveStaleStartupLoads` = 5, after which the snapshot is applied anyway with a Warning
(a boot that never finishes is strictly worse than one starting from a slightly superseded view
the sweep converges on its next tick). A superseded boot snapshot arriving *after*
`_startupLoadCompleted` — the S5 `RetryStartupLoad` timer racing a success — is dropped rather
than re-read, so the retry timer and the re-dispatch cannot feed each other.
### 11.5 Tests
In `InstanceSweepSelfHealTests`, all hand-delivering crafted `InstanceSweepLoaded` /
`StartupConfigsLoaded` messages with an explicit generation:
- `StaleSweepSnapshot_CarryingASupersededRevision_DoesNotRestartBackward` — fails before.
- `StaleSweepSnapshot_ShowingADeletedInstance_DoesNotResurrectIt` — fails before (full
deploy-then-delete shape, not the weaker never-started equivalent).
- `StaleBootSnapshot_IsReReadRatherThanApplied` — fails before (one read instead of two); a slow
first read plus a lifecycle command processed during it makes the race deterministic, and the
test also pins that a LATE superseded snapshot is dropped rather than looping.
- `FreshSweepSnapshot_CarryingANewerRevision_StillRestartsTheInstance` and
`FreshSweepSnapshot_ShowingAStoredEnabledInstance_StillStartsIt` — the guard twins, pinning that
legitimate row-72 and row-78 repairs survive. Both start the instance from the BOOT LOAD rather
than a deploy command, precisely so no mutation passes through the actor and generation `0` is
genuinely current.
`FreshEmptySweepSnapshot_StillRemovesDisappearedConnections` (§10) still pins the definitions
half; its hardcoded current generation moves from `1` to `2` because one artifact apply now bumps
at both ends.
+12 -2
View File
@@ -106,7 +106,7 @@ flowchart TD
>
> The **row-72 decision is to converge, not merely warn**: the stored row *is* what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing `HandleDeploy` means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy.
>
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. One guard is **not** in that list because it needed adding later: deferring to an in-flight deploy pipeline says nothing about a snapshot that landed *after* the pipeline finished, which is the stale-snapshot race — a tick whose read raced this node's own deploy/enable/disable/delete is discarded whole by the store-write generation check (see **Stale-snapshot guard** below). A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table — that stayed true when rows 89/90 gave the table a reader, because the overlay it feeds is an *arbiter*, not a creation source, and at the time of row 85 nothing read the table at all. A connection actor exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done at the time: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7. That pass is now present, in a form that resolves the conflict rather than creating it — see **DCL connection arbitration** immediately below.
@@ -118,7 +118,17 @@ flowchart TD
>
> **The ping-pong is killed by keying the hash on the winner.** The conflict §7.4 identified is a property of *where the hash is computed*, not of the sweep: every ensure path now resolves the winning configuration **before** hashing or pushing, so for any name only one configuration can ever occupy the hash slot regardless of which path calls. The flip is not guarded against — it is unrepresentable. `HandleApplyArtifactDataConnectionsToDcl` writes its rows into the overlay in the same pass it pushes them, closing the same flip arriving through the artifact-deploy door.
>
> **Stale-snapshot guard (row 101).** The snapshot is read off the actor thread but compared against the overlay only when its piped message lands — so a read that starts before an artifact apply's store writes can land *after* `HandleApplyArtifactDataConnectionsToDcl` populated the overlay, and its stale data would "prove" the just-applied names disappeared: tombstoned, live connection actors stopped, and — because the removal evicts the hash slot the overlay's push is gated on — **nothing re-creates them until an instance redeploy**. Each snapshot read therefore captures a monotonic definitions-write generation on the actor thread at dispatch (`_definitionsWriteGeneration`, bumped by every artifact apply), and a result carrying a stale generation is discarded **whole** — additions included, since a stale row pushed at a live connection would reconfigure it backward and flip-flop with the next fresh read. Discarding is safe: the artifact apply already updated the overlay itself, and the next tick reads fresh. Surfaced as a test flake (a raced sweep removed `PlcA` where the test expected `PlcGone`); the deterministic reproduction is `DclConnectionArbitrationTests.StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections`.
> **Stale-snapshot guard — covers the WHOLE snapshot, both halves and both consumers (row 101).** `ReadStoreSnapshotAsync` reads `deployed_configurations` and `data_connection_definitions` off the actor thread but is consumed only when its piped message lands, so a read dispatched before a mutation can land after the actor's own state has already moved past it. Every read therefore captures a single monotonic **store-write generation** on the actor thread at dispatch (`_storeWriteGeneration`), and a snapshot carrying a stale generation is discarded **whole**. One counter covers both tables on purpose: the snapshot is one consistent read and is discarded whole anyway, so a per-table counter would only buy the chance to apply half a snapshot the other half's mutation already invalidated.
>
> Each half would do real damage. **Definitions half:** stale absences "prove" just-applied names disappeared — tombstoned, live connection actors stopped, and because the removal evicts the hash slot the overlay's push is gated on, **nothing re-creates them until an instance redeploy**; stale rows push superseded configs at live connections and flip-flop with the next fresh read. **Configs half, row-72 leg:** a snapshot carrying the revision a deploy just replaced makes the stale-revision repair run **backward**, re-applying the snapshot's own superseded config JSON through the redeploy path — and `InstanceIsOwnedByADeployPipeline` is no defence, because it only holds while a deploy is *in flight* and the stale snapshot can land after the pipeline released the instance. **Configs half, row-78 leg:** a snapshot read before an undeploy/delete or a disable committed still shows an enabled row while the actor is already stopped, so the start repair **resurrects a just-stopped instance** — and delete specifically clears `_sweepAttemptedRevisions`, so the once-per-revision claim is freshly available and does not stand in the way either. All three were reproduced by neutralising the guard and re-running the regression tests.
>
> **The generation is bumped twice per mutation — when the off-thread write is dispatched and again when its completion message lands on the actor thread.** The pair is what leaves no window: an initiation-only bump would let a read dispatched *between* the decision and the commit compare equal and be applied even though it predates the write, and a completion-only bump would miss a read dispatched before the handler ran. With both, a read dispatched before the mutation is discarded, a read dispatched during it is discarded, and a read dispatched after the completion bump necessarily sees the committed row. Bump sites are every actor-thread path that mutates either table: deploy persist (dispatch + `DeployPersistenceResult`), enable (dispatch + `EnableResult`), disable and delete and the init-failure rollback (dispatch + a `StoreWriteCompleted` self-message that exists purely to bump after the commit), and the artifact apply (dispatch + `ApplyArtifactDataConnectionsToDcl`).
>
> **Writes that must NOT bump**, and this is load-bearing: rows arriving by LocalDb CDC replication, by a snapshot resync, by an operator restoring the file, or by `SiteReconciliationActor`'s startup fetch never pass through this actor — and discovering exactly those is the sweep's entire reason to exist (rows 72 + 78). They are not stale relative to actor state; they are the state the actor has yet to learn about.
>
> **The two consumers differ.** A stale **sweep** result is dropped whole at Debug — no overlay apply, no start, no restart — because there is nothing to complete and the next tick is already a fresh read. A stale **boot** load cannot be dropped (a node whose startup read is discarded stays a silent zero-instance node), so it **re-dispatches `LoadDeployedConfigs()`**, capturing the current generation. Staleness requires a fresh mutation inside each attempt's window, so this converges in practice; as a livelock backstop the fifth consecutive superseded boot read is applied anyway with a Warning, on the grounds that a boot that never finishes is worse than one starting from a slightly superseded view that the sweep converges on its next tick. A superseded boot snapshot arriving *after* the boot has completed (the S5 retry timer racing a success) is dropped rather than re-read.
>
> Surfaced originally as a test flake (a raced sweep removed `PlcA` where the test expected `PlcGone`). Deterministic reproductions: `DclConnectionArbitrationTests.StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections` for the definitions half, and in `InstanceSweepSelfHealTests` — `StaleSweepSnapshot_CarryingASupersededRevision_DoesNotRestartBackward`, `StaleSweepSnapshot_ShowingADeletedInstance_DoesNotResurrectIt`, `StaleBootSnapshot_IsReReadRatherThanApplied`, each paired with a `Fresh…` twin pinning that the legitimate row-72 restart and row-78 start still happen.
>
> **Removal (row 90).** Both removal paths share one helper and now record a **tombstone** for the incarnation: the artifact reconcile's `RemovedConnectionNames`, and the sweep noticing that a name it had adopted no longer has a row (a removal it can *prove* — the store owned it a tick ago and does not now). `EnsureDclConnections` skips a tombstoned name, so a removal survives the next instance start instead of being undone by it — a pre-existing bug, since the removal evicted the hash slot but left no memory of why. Tombstones are **in-memory only** by decision: a durable marker would be a second divergent authority, and flattened configs already resurrect a removed connection across a restart. A definition row reappearing clears the tombstone.
>