fix(siteruntime): extend the row-101 stale-snapshot guard to the instance-config half

ReadStoreSnapshotAsync's snapshot is read off the actor thread and consumed only
when its piped message lands. Row 101 guarded the DEFINITIONS half of that race;
the CONFIG half was left to machinery whose reach is time-limited, and both legs
of the instance reconcile sweep were reachable through it:

- Row-72 leg: a read dispatched before a deploy's persist, landing after the
  deploy completed AND after the pipeline released ownership, carries the
  revision the deploy replaced. TryRestartStaleInstance then "repairs" the
  instance BACKWARD, re-deploying the snapshot's own superseded ConfigJson.
  InstanceIsOwnedByADeployPipeline covers only in-flight deploys, and the
  per-revision claim has never seen the old revision, so nothing stopped it.
- Row-78 leg: a read dispatched before a delete/disable commits still shows an
  enabled row while the actor is already stopped, so StartStoredInstance
  resurrects it — and delete clears _sweepAttemptedRevisions, freeing the claim.

Both verified real by neutralising the freshness check and re-running the new
tests (the fail-before site event literally reads "running revision rev-2 is
stale, applying stored revision rev-1"; the deleted instance came back).

_definitionsWriteGeneration becomes _storeWriteGeneration, covering both tables
(the snapshot is one consistent read and is discarded whole, so one counter is
the right shape); DefinitionsGenerationAtRead -> StoreGenerationAtRead on both
internal records. The generation is now bumped at BOTH ends of every mutation —
when the off-thread write is dispatched and when its completion message lands —
because the config paths advance actor state before the write, so an
initiation-only bump would let an in-window read look fresh. Disable, delete and
the init-failure rollback have no actor-thread completion message of their own,
so a minimal StoreWriteCompleted self-message carries the bump for them.

CDC-replicated writes, snapshot resyncs, restored files and
SiteReconciliationActor's fetch deliberately do NOT bump: discovering exactly
those is the sweep's reason to exist.

A stale sweep result is now dropped WHOLE (no overlay apply, no start, no
restart). A stale boot load cannot be dropped — startup must finish — so it
re-dispatches LoadDeployedConfigs(); a fifth consecutive superseded read is
applied anyway with a Warning, and a late one arriving after the boot completed
is dropped rather than looping against the S5 retry timer.

Tests: three fail-before regressions (backward restart, resurrection, boot
re-read) plus two guard twins pinning that the legitimate row-72 restart and
row-78 start still fire on a current-generation snapshot. The row-101
definitions tests keep passing; FreshEmptySweepSnapshot's hardcoded generation
moves 1 -> 2 now that one artifact apply bumps at both ends.

Docs: Component-SiteRuntime.md's stale-snapshot paragraph now describes the
whole snapshot (both halves, both consumers, the boot re-dispatch, the
CDC-no-bump rule), the components mirror follows, and the arbitration memo gains
§11 extending §10 to the config half.
This commit is contained in:
Joseph Doherty
2026-08-18 05:38:00 -04:00
parent 4da5f7cd7a
commit f02e116688
6 changed files with 617 additions and 60 deletions
@@ -412,3 +412,113 @@ Applies to both snapshot consumers (boot load and sweep). Deterministic reproduc
test that the legitimate row-90 disappearance still fires on a FRESH empty snapshot:
`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections` (fails before the
fix) and `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections`.
## 11. Postscript — the same race on the CONFIG half of the snapshot (register row 101 follow-on, 2026-08-18)
§10 fixed the DEFINITIONS half of `ReadStoreSnapshotAsync`'s snapshot. The **instance-config**
half of the same snapshot has the identical TOCTOU, and it was left protected only by machinery
whose reach is time-limited. This section extends §10 rather than revising it; everything §10
says about the definitions half still stands.
### 11.1 The two hazards, both verified real against the code
Both were reproduced by neutralising the freshness check and re-running the new regression tests
(see §11.5), so neither is theoretical.
**Hazard 1 — backward restart (row-72 leg).** A read dispatched before a deploy persists its new
row can land after the deploy has completed *and* after the pipeline released the instance. The
snapshot then carries the revision the deploy replaced while `_runningRevisions` carries the new
one — the exact shape `TryRestartStaleInstance` repairs, in the wrong direction. It is not a
harmless restart: the repair passes **the snapshot's own `ConfigJson`** to `HandleDeploy`, so the
superseded config is literally re-deployed. The pre-existing guards do not reach it:
`InstanceIsOwnedByADeployPipeline` covers only `_deployWarms` / `_terminatingActorsByName` /
`_inFlightDeployments`, all of which are cleared when the deploy finishes; and
`TryClaimSweepAttempt` does bound restarts to one per (instance, revision), but the *old* revision
has never been claimed, so the claim is available and the instance simply stays reverted.
Observed fail-before site event: *"Instance BackwardRestartPump restarted by deployment-manager
sweep self-heal — running revision rev-2 is stale, applying stored revision rev-1"*.
**Hazard 2 — resurrection (row-78 leg).** A read dispatched before a delete or a disable commits
still shows an enabled row while the actor is already stopped (both handlers stop the actor
synchronously and write off-thread). `StartStoredInstance` then starts an instance the operator
just removed. Delete is the worse of the two because it explicitly clears
`_sweepAttemptedRevisions` for the instance, handing the sweep a fresh repair claim. Fail-before:
the deleted instance was running again after the stale snapshot landed.
**Hazard 3 — the boot load's other consumers.** `HandleStartupConfigsLoaded` also rebuilds
`_deployedInstanceNames` from the snapshot and hands its enabled configs to the staggered startup
batch. A deploy or delete processed while the boot read is in flight therefore skews the
deployed-instance count and can hand the batch a superseded config; `CreateInstanceActor`'s
already-exists guard blunts the second effect but not the first.
### 11.2 One generation for both tables
`_definitionsWriteGeneration` is renamed `_storeWriteGeneration` and now counts mutations of BOTH
snapshot tables; the message parameter `DefinitionsGenerationAtRead` becomes
`StoreGenerationAtRead` on both internal records (neither is a wire message). One counter is the
right shape because the snapshot is a single consistent read and is discarded whole either way —
a per-table counter would only buy the ability to apply half a snapshot that the other half's
mutation already invalidated.
### 11.3 Bump at BOTH ends of every mutation
§10 bumped once, in the handler that runs after the write. That is sufficient when the actor's
own state advances in that same handler, but the config mutations do not all work that way —
`ApplyDeployment` creates the Instance Actor and stamps the running revision *before* the
persist, and `HandleDisable`/`HandleDelete` stop the actor *before* the write. So the generation
is now bumped twice per mutation: once when the off-thread write is **dispatched**, once when its
**completion** message lands. The pair leaves no window — a read dispatched before the mutation is
discarded by the first bump, a read dispatched inside the write window is discarded by the second,
and a read dispatched after the second necessarily sees the committed row. Two paths had no
actor-thread completion message at all (disable/delete pipe their reply to the *caller*, and the
init-failure rollback pipes nothing), so a minimal internal `StoreWriteCompleted(Source)` message
exists purely to carry the completion bump for them.
Bump sites: `ApplyDeployment` + `HandleDeployPersistenceResult`; `HandleEnable` +
`HandleEnableResult`; `HandleDisable` + `StoreWriteCompleted`; `HandleDelete` +
`StoreWriteCompleted`; `RollbackPersistedConfig` + `StoreWriteCompleted`; `HandleDeployArtifacts`
+ `HandleApplyArtifactDataConnectionsToDcl` (the §10 bump, kept). The deploy and enable completion
bumps are **unconditional on success** — a failed write may still have committed the row before a
later step threw, so "failed" is not "nothing changed". Cost of over-bumping is at most one
discarded sweep tick or one extra boot read.
**Deliberately NOT bumped:** `SiteReconciliationActor`'s `StoreDeployedConfigIfNewerAsync` (a
different actor), LocalDb CDC replication applies, snapshot resyncs, and an operator restoring the
file. None passes through this actor, and discovering exactly those divergences is the sweep's
entire reason to exist (rows 72 + 78). Bumping for them — were it even reachable — would blind the
sweep to its own use case.
### 11.4 Sweep drops, boot re-reads
The sweep consumer discards a stale result **whole** (definitions half and configs half, no
overlay apply, no start, no restart) at Debug, because there is nothing to complete and the next
tick is already a fresh read. The boot consumer cannot do that — a node whose startup read is
dropped stays a silent zero-instance node — so it **re-dispatches `LoadDeployedConfigs()`**, which
captures the current generation. Convergence argument: staleness requires a fresh mutation inside
each attempt's window, which is externally driven and finite. The belt-and-braces cap is
`MaxConsecutiveStaleStartupLoads` = 5, after which the snapshot is applied anyway with a Warning
(a boot that never finishes is strictly worse than one starting from a slightly superseded view
the sweep converges on its next tick). A superseded boot snapshot arriving *after*
`_startupLoadCompleted` — the S5 `RetryStartupLoad` timer racing a success — is dropped rather
than re-read, so the retry timer and the re-dispatch cannot feed each other.
### 11.5 Tests
In `InstanceSweepSelfHealTests`, all hand-delivering crafted `InstanceSweepLoaded` /
`StartupConfigsLoaded` messages with an explicit generation:
- `StaleSweepSnapshot_CarryingASupersededRevision_DoesNotRestartBackward` — fails before.
- `StaleSweepSnapshot_ShowingADeletedInstance_DoesNotResurrectIt` — fails before (full
deploy-then-delete shape, not the weaker never-started equivalent).
- `StaleBootSnapshot_IsReReadRatherThanApplied` — fails before (one read instead of two); a slow
first read plus a lifecycle command processed during it makes the race deterministic, and the
test also pins that a LATE superseded snapshot is dropped rather than looping.
- `FreshSweepSnapshot_CarryingANewerRevision_StillRestartsTheInstance` and
`FreshSweepSnapshot_ShowingAStoredEnabledInstance_StillStartsIt` — the guard twins, pinning that
legitimate row-72 and row-78 repairs survive. Both start the instance from the BOOT LOAD rather
than a deploy command, precisely so no mutation passes through the actor and generation `0` is
genuinely current.
`FreshEmptySweepSnapshot_StillRemovesDisappearedConnections` (§10) still pins the definitions
half; its hardcoded current generation moves from `1` to `2` because one artifact apply now bumps
at both ends.