fix(siteruntime): guard the DCL arbitration overlay against stale store snapshots — register row 101

An off-thread snapshot read racing an artifact apply could land after the
apply populated the overlay; its stale absences then tombstoned and stopped
just-applied LIVE connections, which nothing re-creates until an instance
redeploy (the removal evicts the hash slot the overlay push is gated on).
Each read now captures a definitions-write generation at dispatch; a stale
result is discarded whole (additions too — a stale row would reconfigure a
live connection backward). Deterministic repro fails before the fix; a
fresh-generation guard test pins that row-90 disappearance still fires.
This commit is contained in:
Joseph Doherty
2026-08-17 07:03:26 -04:00
parent 63d7f273e6
commit 0d3a90a9fd
5 changed files with 166 additions and 7 deletions
@@ -387,3 +387,28 @@ tests plus SiteRuntime-010, renamed `…_ReissuesReconfigureCommand`).
§6 residuals **2** (a removal is not durable across a restart) and **3** (a delete is undetectable on
a node that never adopted the row) are unchanged and still open. §6 residual **4** was never a gap.
## 10. Postscript — the stale-snapshot race (register row 101, 2026-08-17)
The sweep design read the store off the actor thread (§5) and compared the result against the
overlay only when the piped message landed. That gap was a TOCTOU the original memo did not name:
a read starting before an artifact apply's store writes could land AFTER
`HandleApplyArtifactDataConnectionsToDcl` populated the overlay, and its stale snapshot would then
"prove" the just-applied names disappeared. The disappearance detection (§5) tombstoned them and
stopped the just-created live connection actors — and because the removal also evicts the hash
slot the overlay's push is gated on, the next fresh sweep clears the tombstone but re-creates
nothing: **the connection stays dead until an instance redeploy**. Surfaced as a one-off test flake
(`ApplyingArtifacts_RemovedDataConnection_StopsLiveDclConnection` received a removal for `PlcA`
where `PlcGone` was expected — overlay iteration order removing BOTH names off a stale empty
snapshot), first suspected to be test-fixture-local; verification showed the mechanism is real in
production whenever an artifact deploy lands while a sweep read is in flight.
**Fix: a definitions-write generation.** `_definitionsWriteGeneration` is bumped on the actor
thread by every artifact apply; each snapshot read captures it at dispatch and a result carrying a
stale generation is discarded WHOLE — additions included, since a stale row pushed at a live
connection would reconfigure it backward and flip-flop with the next fresh read. Discarding is
safe: the apply already updated the overlay in the same pass (§5), and the next tick reads fresh.
Applies to both snapshot consumers (boot load and sweep). Deterministic reproduction + a guard
test that the legitimate row-90 disappearance still fires on a FRESH empty snapshot:
`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections` (fails before the
fix) and `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections`.