fix(siteruntime): guard the DCL arbitration overlay against stale store snapshots — register row 101
An off-thread snapshot read racing an artifact apply could land after the apply populated the overlay; its stale absences then tombstoned and stopped just-applied LIVE connections, which nothing re-creates until an instance redeploy (the removal evicts the hash slot the overlay push is gated on). Each read now captures a definitions-write generation at dispatch; a stale result is discarded whole (additions too — a stale row would reconfigure a live connection backward). Deterministic repro fails before the fix; a fresh-generation guard test pins that row-90 disappearance still fires.
This commit is contained in:
@@ -567,6 +567,69 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
Assert.Equal(reAddedEndpoint, EndpointOf(CreatesFor(ConnectionName).Last()));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Deferred register row 101 — the stale-snapshot race, made deterministic. A sweep's store
|
||||
/// read runs OFF the actor thread; one that starts before an artifact apply's store writes
|
||||
/// can land AFTER <c>HandleApplyArtifactDataConnectionsToDcl</c> populated the overlay. Its
|
||||
/// snapshot then lacks the just-applied rows, and the disappearance detection would tombstone
|
||||
/// and STOP the just-created live connections — which nothing re-creates until an instance
|
||||
/// redeploy, because the removal also evicts the hash slot the overlay's push is gated on.
|
||||
///
|
||||
/// <para>
|
||||
/// The crafted <c>InstanceSweepLoaded</c> below IS that race: an empty snapshot stamped with
|
||||
/// the pre-apply write generation, delivered after the apply. <b>Fails before the fix</b>:
|
||||
/// without the generation guard the actor removes the connection it created a moment ago.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections()
|
||||
{
|
||||
// Sweep effectively disabled so the crafted message is the only sweep result in play.
|
||||
var dm = CreateBootedDeploymentManager(sweepSeconds: 3600);
|
||||
|
||||
dm.Tell(new DeployArtifactsCommand(
|
||||
"artifact-race-1", null, null, null, null,
|
||||
[new DataConnectionArtifact(ConnectionName, "OpcUa", EndpointJson(DefinitionEndpoint), null)],
|
||||
null, DateTimeOffset.UtcNow));
|
||||
Assert.True(ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(20)).Success);
|
||||
await AwaitCreateCountAsync(1, "the artifact apply never created the connection");
|
||||
|
||||
// Generation 0 = captured before the apply bumped it; empty list = the pre-write table.
|
||||
dm.Tell(new DeploymentManagerActor.InstanceSweepLoaded(
|
||||
[], null, ConnectionDefinitions: [], DefinitionsGenerationAtRead: 0));
|
||||
|
||||
await Task.Delay(TimeSpan.FromSeconds(2));
|
||||
Assert.DoesNotContain(_removes, r => r.ConnectionName == ConnectionName);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The row-101 guard must not blunt row 90's legitimate disappearance detection: the same
|
||||
/// empty snapshot stamped with the CURRENT write generation is a real answer (the store
|
||||
/// genuinely holds no rows any more) and still stops the connection.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task FreshEmptySweepSnapshot_StillRemovesDisappearedConnections()
|
||||
{
|
||||
var dm = CreateBootedDeploymentManager(sweepSeconds: 3600);
|
||||
|
||||
dm.Tell(new DeployArtifactsCommand(
|
||||
"artifact-race-2", null, null, null, null,
|
||||
[new DataConnectionArtifact(ConnectionName, "OpcUa", EndpointJson(DefinitionEndpoint), null)],
|
||||
null, DateTimeOffset.UtcNow));
|
||||
Assert.True(ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(20)).Success);
|
||||
await AwaitCreateCountAsync(1, "the artifact apply never created the connection");
|
||||
|
||||
// Exactly one artifact apply has run, so the current write generation is 1.
|
||||
dm.Tell(new DeploymentManagerActor.InstanceSweepLoaded(
|
||||
[], null, ConnectionDefinitions: [], DefinitionsGenerationAtRead: 1));
|
||||
|
||||
await AwaitAssertAsync(() =>
|
||||
{
|
||||
Assert.Contains(_removes, r => r.ConnectionName == ConnectionName);
|
||||
return Task.CompletedTask;
|
||||
}, TimeSpan.FromSeconds(10));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Stand-in for the DCL manager actor: records the two commands the Deployment Manager is the
|
||||
/// sole sender of, and swallows everything else (Instance Actors send their tag subscribes
|
||||
|
||||
Reference in New Issue
Block a user