fix(siteruntime): guard the DCL arbitration overlay against stale store snapshots — register row 101

An off-thread snapshot read racing an artifact apply could land after the
apply populated the overlay; its stale absences then tombstoned and stopped
just-applied LIVE connections, which nothing re-creates until an instance
redeploy (the removal evicts the hash slot the overlay push is gated on).
Each read now captures a definitions-write generation at dispatch; a stale
result is discarded whole (additions too — a stale row would reconfigure a
live connection backward). Deterministic repro fails before the fix; a
fresh-generation guard test pins that row-90 disappearance still fires.
This commit is contained in:
Joseph Doherty
2026-08-17 07:03:26 -04:00
parent 63d7f273e6
commit 0d3a90a9fd
5 changed files with 166 additions and 7 deletions
@@ -567,6 +567,69 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
Assert.Equal(reAddedEndpoint, EndpointOf(CreatesFor(ConnectionName).Last()));
}
/// <summary>
/// Deferred register row 101 — the stale-snapshot race, made deterministic. A sweep's store
/// read runs OFF the actor thread; one that starts before an artifact apply's store writes
/// can land AFTER <c>HandleApplyArtifactDataConnectionsToDcl</c> populated the overlay. Its
/// snapshot then lacks the just-applied rows, and the disappearance detection would tombstone
/// and STOP the just-created live connections — which nothing re-creates until an instance
/// redeploy, because the removal also evicts the hash slot the overlay's push is gated on.
///
/// <para>
/// The crafted <c>InstanceSweepLoaded</c> below IS that race: an empty snapshot stamped with
/// the pre-apply write generation, delivered after the apply. <b>Fails before the fix</b>:
/// without the generation guard the actor removes the connection it created a moment ago.
/// </para>
/// </summary>
[Fact]
public async Task StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections()
{
// Sweep effectively disabled so the crafted message is the only sweep result in play.
var dm = CreateBootedDeploymentManager(sweepSeconds: 3600);
dm.Tell(new DeployArtifactsCommand(
"artifact-race-1", null, null, null, null,
[new DataConnectionArtifact(ConnectionName, "OpcUa", EndpointJson(DefinitionEndpoint), null)],
null, DateTimeOffset.UtcNow));
Assert.True(ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(20)).Success);
await AwaitCreateCountAsync(1, "the artifact apply never created the connection");
// Generation 0 = captured before the apply bumped it; empty list = the pre-write table.
dm.Tell(new DeploymentManagerActor.InstanceSweepLoaded(
[], null, ConnectionDefinitions: [], DefinitionsGenerationAtRead: 0));
await Task.Delay(TimeSpan.FromSeconds(2));
Assert.DoesNotContain(_removes, r => r.ConnectionName == ConnectionName);
}
/// <summary>
/// The row-101 guard must not blunt row 90's legitimate disappearance detection: the same
/// empty snapshot stamped with the CURRENT write generation is a real answer (the store
/// genuinely holds no rows any more) and still stops the connection.
/// </summary>
[Fact]
public async Task FreshEmptySweepSnapshot_StillRemovesDisappearedConnections()
{
var dm = CreateBootedDeploymentManager(sweepSeconds: 3600);
dm.Tell(new DeployArtifactsCommand(
"artifact-race-2", null, null, null, null,
[new DataConnectionArtifact(ConnectionName, "OpcUa", EndpointJson(DefinitionEndpoint), null)],
null, DateTimeOffset.UtcNow));
Assert.True(ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(20)).Success);
await AwaitCreateCountAsync(1, "the artifact apply never created the connection");
// Exactly one artifact apply has run, so the current write generation is 1.
dm.Tell(new DeploymentManagerActor.InstanceSweepLoaded(
[], null, ConnectionDefinitions: [], DefinitionsGenerationAtRead: 1));
await AwaitAssertAsync(() =>
{
Assert.Contains(_removes, r => r.ConnectionName == ConnectionName);
return Task.CompletedTask;
}, TimeSpan.FromSeconds(10));
}
/// <summary>
/// Stand-in for the DCL manager actor: records the two commands the Deployment Manager is the
/// sole sender of, and swallows everything else (Instance Actors send their tag subscribes