fix(siteruntime): extend the row-101 stale-snapshot guard to the instance-config half
ReadStoreSnapshotAsync's snapshot is read off the actor thread and consumed only when its piped message lands. Row 101 guarded the DEFINITIONS half of that race; the CONFIG half was left to machinery whose reach is time-limited, and both legs of the instance reconcile sweep were reachable through it: - Row-72 leg: a read dispatched before a deploy's persist, landing after the deploy completed AND after the pipeline released ownership, carries the revision the deploy replaced. TryRestartStaleInstance then "repairs" the instance BACKWARD, re-deploying the snapshot's own superseded ConfigJson. InstanceIsOwnedByADeployPipeline covers only in-flight deploys, and the per-revision claim has never seen the old revision, so nothing stopped it. - Row-78 leg: a read dispatched before a delete/disable commits still shows an enabled row while the actor is already stopped, so StartStoredInstance resurrects it — and delete clears _sweepAttemptedRevisions, freeing the claim. Both verified real by neutralising the freshness check and re-running the new tests (the fail-before site event literally reads "running revision rev-2 is stale, applying stored revision rev-1"; the deleted instance came back). _definitionsWriteGeneration becomes _storeWriteGeneration, covering both tables (the snapshot is one consistent read and is discarded whole, so one counter is the right shape); DefinitionsGenerationAtRead -> StoreGenerationAtRead on both internal records. The generation is now bumped at BOTH ends of every mutation — when the off-thread write is dispatched and when its completion message lands — because the config paths advance actor state before the write, so an initiation-only bump would let an in-window read look fresh. Disable, delete and the init-failure rollback have no actor-thread completion message of their own, so a minimal StoreWriteCompleted self-message carries the bump for them. CDC-replicated writes, snapshot resyncs, restored files and SiteReconciliationActor's fetch deliberately do NOT bump: discovering exactly those is the sweep's reason to exist. A stale sweep result is now dropped WHOLE (no overlay apply, no start, no restart). A stale boot load cannot be dropped — startup must finish — so it re-dispatches LoadDeployedConfigs(); a fifth consecutive superseded read is applied anyway with a Warning, and a late one arriving after the boot completed is dropped rather than looping against the S5 retry timer. Tests: three fail-before regressions (backward restart, resurrection, boot re-read) plus two guard twins pinning that the legitimate row-72 restart and row-78 start still fire on a current-generation snapshot. The row-101 definitions tests keep passing; FreshEmptySweepSnapshot's hardcoded generation moves 1 -> 2 now that one artifact apply bumps at both ends. Docs: Component-SiteRuntime.md's stale-snapshot paragraph now describes the whole snapshot (both halves, both consumers, the boot re-dispatch, the CDC-no-bump rule), the components mirror follows, and the arbitration memo gains §11 extending §10 to the config half.
This commit is contained in:
@@ -412,3 +412,113 @@ Applies to both snapshot consumers (boot load and sweep). Deterministic reproduc
|
||||
test that the legitimate row-90 disappearance still fires on a FRESH empty snapshot:
|
||||
`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections` (fails before the
|
||||
fix) and `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections`.
|
||||
|
||||
## 11. Postscript — the same race on the CONFIG half of the snapshot (register row 101 follow-on, 2026-08-18)
|
||||
|
||||
§10 fixed the DEFINITIONS half of `ReadStoreSnapshotAsync`'s snapshot. The **instance-config**
|
||||
half of the same snapshot has the identical TOCTOU, and it was left protected only by machinery
|
||||
whose reach is time-limited. This section extends §10 rather than revising it; everything §10
|
||||
says about the definitions half still stands.
|
||||
|
||||
### 11.1 The two hazards, both verified real against the code
|
||||
|
||||
Both were reproduced by neutralising the freshness check and re-running the new regression tests
|
||||
(see §11.5), so neither is theoretical.
|
||||
|
||||
**Hazard 1 — backward restart (row-72 leg).** A read dispatched before a deploy persists its new
|
||||
row can land after the deploy has completed *and* after the pipeline released the instance. The
|
||||
snapshot then carries the revision the deploy replaced while `_runningRevisions` carries the new
|
||||
one — the exact shape `TryRestartStaleInstance` repairs, in the wrong direction. It is not a
|
||||
harmless restart: the repair passes **the snapshot's own `ConfigJson`** to `HandleDeploy`, so the
|
||||
superseded config is literally re-deployed. The pre-existing guards do not reach it:
|
||||
`InstanceIsOwnedByADeployPipeline` covers only `_deployWarms` / `_terminatingActorsByName` /
|
||||
`_inFlightDeployments`, all of which are cleared when the deploy finishes; and
|
||||
`TryClaimSweepAttempt` does bound restarts to one per (instance, revision), but the *old* revision
|
||||
has never been claimed, so the claim is available and the instance simply stays reverted.
|
||||
Observed fail-before site event: *"Instance BackwardRestartPump restarted by deployment-manager
|
||||
sweep self-heal — running revision rev-2 is stale, applying stored revision rev-1"*.
|
||||
|
||||
**Hazard 2 — resurrection (row-78 leg).** A read dispatched before a delete or a disable commits
|
||||
still shows an enabled row while the actor is already stopped (both handlers stop the actor
|
||||
synchronously and write off-thread). `StartStoredInstance` then starts an instance the operator
|
||||
just removed. Delete is the worse of the two because it explicitly clears
|
||||
`_sweepAttemptedRevisions` for the instance, handing the sweep a fresh repair claim. Fail-before:
|
||||
the deleted instance was running again after the stale snapshot landed.
|
||||
|
||||
**Hazard 3 — the boot load's other consumers.** `HandleStartupConfigsLoaded` also rebuilds
|
||||
`_deployedInstanceNames` from the snapshot and hands its enabled configs to the staggered startup
|
||||
batch. A deploy or delete processed while the boot read is in flight therefore skews the
|
||||
deployed-instance count and can hand the batch a superseded config; `CreateInstanceActor`'s
|
||||
already-exists guard blunts the second effect but not the first.
|
||||
|
||||
### 11.2 One generation for both tables
|
||||
|
||||
`_definitionsWriteGeneration` is renamed `_storeWriteGeneration` and now counts mutations of BOTH
|
||||
snapshot tables; the message parameter `DefinitionsGenerationAtRead` becomes
|
||||
`StoreGenerationAtRead` on both internal records (neither is a wire message). One counter is the
|
||||
right shape because the snapshot is a single consistent read and is discarded whole either way —
|
||||
a per-table counter would only buy the ability to apply half a snapshot that the other half's
|
||||
mutation already invalidated.
|
||||
|
||||
### 11.3 Bump at BOTH ends of every mutation
|
||||
|
||||
§10 bumped once, in the handler that runs after the write. That is sufficient when the actor's
|
||||
own state advances in that same handler, but the config mutations do not all work that way —
|
||||
`ApplyDeployment` creates the Instance Actor and stamps the running revision *before* the
|
||||
persist, and `HandleDisable`/`HandleDelete` stop the actor *before* the write. So the generation
|
||||
is now bumped twice per mutation: once when the off-thread write is **dispatched**, once when its
|
||||
**completion** message lands. The pair leaves no window — a read dispatched before the mutation is
|
||||
discarded by the first bump, a read dispatched inside the write window is discarded by the second,
|
||||
and a read dispatched after the second necessarily sees the committed row. Two paths had no
|
||||
actor-thread completion message at all (disable/delete pipe their reply to the *caller*, and the
|
||||
init-failure rollback pipes nothing), so a minimal internal `StoreWriteCompleted(Source)` message
|
||||
exists purely to carry the completion bump for them.
|
||||
|
||||
Bump sites: `ApplyDeployment` + `HandleDeployPersistenceResult`; `HandleEnable` +
|
||||
`HandleEnableResult`; `HandleDisable` + `StoreWriteCompleted`; `HandleDelete` +
|
||||
`StoreWriteCompleted`; `RollbackPersistedConfig` + `StoreWriteCompleted`; `HandleDeployArtifacts`
|
||||
+ `HandleApplyArtifactDataConnectionsToDcl` (the §10 bump, kept). The deploy and enable completion
|
||||
bumps are **unconditional on success** — a failed write may still have committed the row before a
|
||||
later step threw, so "failed" is not "nothing changed". Cost of over-bumping is at most one
|
||||
discarded sweep tick or one extra boot read.
|
||||
|
||||
**Deliberately NOT bumped:** `SiteReconciliationActor`'s `StoreDeployedConfigIfNewerAsync` (a
|
||||
different actor), LocalDb CDC replication applies, snapshot resyncs, and an operator restoring the
|
||||
file. None passes through this actor, and discovering exactly those divergences is the sweep's
|
||||
entire reason to exist (rows 72 + 78). Bumping for them — were it even reachable — would blind the
|
||||
sweep to its own use case.
|
||||
|
||||
### 11.4 Sweep drops, boot re-reads
|
||||
|
||||
The sweep consumer discards a stale result **whole** (definitions half and configs half, no
|
||||
overlay apply, no start, no restart) at Debug, because there is nothing to complete and the next
|
||||
tick is already a fresh read. The boot consumer cannot do that — a node whose startup read is
|
||||
dropped stays a silent zero-instance node — so it **re-dispatches `LoadDeployedConfigs()`**, which
|
||||
captures the current generation. Convergence argument: staleness requires a fresh mutation inside
|
||||
each attempt's window, which is externally driven and finite. The belt-and-braces cap is
|
||||
`MaxConsecutiveStaleStartupLoads` = 5, after which the snapshot is applied anyway with a Warning
|
||||
(a boot that never finishes is strictly worse than one starting from a slightly superseded view
|
||||
the sweep converges on its next tick). A superseded boot snapshot arriving *after*
|
||||
`_startupLoadCompleted` — the S5 `RetryStartupLoad` timer racing a success — is dropped rather
|
||||
than re-read, so the retry timer and the re-dispatch cannot feed each other.
|
||||
|
||||
### 11.5 Tests
|
||||
|
||||
In `InstanceSweepSelfHealTests`, all hand-delivering crafted `InstanceSweepLoaded` /
|
||||
`StartupConfigsLoaded` messages with an explicit generation:
|
||||
|
||||
- `StaleSweepSnapshot_CarryingASupersededRevision_DoesNotRestartBackward` — fails before.
|
||||
- `StaleSweepSnapshot_ShowingADeletedInstance_DoesNotResurrectIt` — fails before (full
|
||||
deploy-then-delete shape, not the weaker never-started equivalent).
|
||||
- `StaleBootSnapshot_IsReReadRatherThanApplied` — fails before (one read instead of two); a slow
|
||||
first read plus a lifecycle command processed during it makes the race deterministic, and the
|
||||
test also pins that a LATE superseded snapshot is dropped rather than looping.
|
||||
- `FreshSweepSnapshot_CarryingANewerRevision_StillRestartsTheInstance` and
|
||||
`FreshSweepSnapshot_ShowingAStoredEnabledInstance_StillStartsIt` — the guard twins, pinning that
|
||||
legitimate row-72 and row-78 repairs survive. Both start the instance from the BOOT LOAD rather
|
||||
than a deploy command, precisely so no mutation passes through the actor and generation `0` is
|
||||
genuinely current.
|
||||
|
||||
`FreshEmptySweepSnapshot_StillRemovesDisappearedConnections` (§10) still pins the definitions
|
||||
half; its hardcoded current generation moves from `1` to `2` because one artifact apply now bumps
|
||||
at both ends.
|
||||
|
||||
Reference in New Issue
Block a user