fix(s7): lifecycle gate + generation-stamped health and publishes (R07:H5)

Overlapping InitializeAsync calls (the actor's fire-and-forget init plus a
retry tick) could leak an opened PLC connection and orphan a probe loop: no
gate guarded the lifecycle, only _gate for I/O.

- _lifecycleGate (the Modbus pattern) serialises Initialize / Reinitialize /
  Shutdown / Dispose; teardown waits a bounded time, then tears down anyway.
- Every initialize and teardown starts a lifecycle generation. Health writes
  (init, read/write passes, poll errors, shutdown) and the init publish carry
  it and are dropped once it has moved on, so an abandoned init never
  publishes over a later shutdown and a shutdown's final write never clobbers
  a newer init. A superseded init disposes what it opened and throws.
- The init publish (options, tag tables, Plc, probe loop) happens under _gate;
  the tag tables are built fresh and swapped by one reference write instead of
  being cleared in place under live poll passes.
- Read/write passes re-check initialization under _gate, so a pass that waited
  out a teardown cannot reopen a connection on a shut-down driver.
- The semaphores are no longer disposed: Dispose can race a running shutdown.

Refs #742
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
Joseph Doherty
2026-10-22 15:10:21 -04:00
parent 87a7414df7
commit ecf1e86464
2 changed files with 467 additions and 115 deletions
@@ -0,0 +1,143 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.S7.Tests;
/// <summary>
/// R07:H5 — overlapping lifecycle calls. <c>DriverInstanceActor</c> dispatches initialize on a
/// fire-and-forget task and a reconnect retry can start a second attempt while the first is still in
/// <c>OpenAsync</c>; an abandoned initialize can also still be running when the next shutdown starts.
/// Every wait here is a condition wait on the fake's own signals, never a sleep.
/// </summary>
[Trait("Category", "Unit")]
public sealed class S7LifecycleGateTests
{
private static readonly TimeSpan Bound = TimeSpan.FromSeconds(10);
private static S7DriverOptions Options(bool probe = false) => new()
{
Host = "192.0.2.1",
Timeout = TimeSpan.FromSeconds(30), // the fake's gated open is released by the test, never timed out
Probe = new S7ProbeOptions { Enabled = probe, Interval = TimeSpan.FromSeconds(30) },
RawTags = S7RawTags.Entries(new S7TagDefinition("W0", "DB1.DBW0", S7DataType.UInt16, Writable: false)),
};
/// <summary>A factory whose FIRST open blocks until released; later opens complete immediately.</summary>
private sealed class FirstOpenBlocks
{
public readonly ScriptedS7PlcFactory Factory = new();
public readonly TaskCompletionSource FirstOpenEntered = new(TaskCreationOptions.RunContinuationsAsynchronously);
public readonly TaskCompletionSource ReleaseFirstOpen = new(TaskCreationOptions.RunContinuationsAsynchronously);
public FirstOpenBlocks()
{
Factory.Open = async (index, ct) =>
{
if (index != 0) return;
FirstOpenEntered.TrySetResult();
await ReleaseFirstOpen.Task.WaitAsync(ct).ConfigureAwait(false);
};
}
}
/// <summary>
/// A second initialize waits for the first instead of running beside it, then releases the first
/// attempt's connection: exactly one live connection remains. Before the gate, attempt B opened
/// its own connection while A was still in <c>OpenAsync</c>, and A's published connection (and
/// probe loop) was overwritten and never released.
/// </summary>
[Fact]
public async Task Overlapping_initializes_are_serialised_and_leave_one_live_connection()
{
var ct = TestContext.Current.CancellationToken;
var f = new FirstOpenBlocks();
using var drv = new S7Driver(Options(), "s7-overlap", f.Factory);
var a = drv.InitializeAsync("{}", ct);
await f.FirstOpenEntered.Task.WaitAsync(Bound, ct);
var b = drv.InitializeAsync("{}", ct);
// Positive control first: B is genuinely pending (not finished, not faulted)...
b.IsCompleted.ShouldBeFalse();
// ...and while A holds the gate B has not created a connection of its own. B can only reach
// its open after A completes, so this is not a race the test can lose by being slow.
f.Factory.Created.Count.ShouldBe(1, "a second initialize must not open beside the first");
f.ReleaseFirstOpen.SetResult();
await a.WaitAsync(Bound, ct);
await b.WaitAsync(Bound, ct);
f.Factory.Created.Count.ShouldBe(2);
f.Factory.Created[0].Disposed.ShouldBeTrue("B releases A's published connection before replacing it");
f.Factory.Created[1].Disposed.ShouldBeFalse();
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
(await drv.ReadAsync(["W0"], ct))[0].StatusCode.ShouldBe(0u);
}
/// <summary>
/// An initialize still running when a shutdown gives up waiting for it must not publish its
/// connection, its health or its probe loop over the shut-down driver; it fails and disposes what
/// it opened. Before the fix the late attempt published Healthy + a live connection after the
/// shutdown returned.
/// </summary>
[Fact]
public async Task An_initialize_superseded_by_shutdown_publishes_nothing()
{
var ct = TestContext.Current.CancellationToken;
var f = new FirstOpenBlocks();
using var drv = new S7Driver(Options(probe: true), "s7-superseded", f.Factory)
{
LifecycleGateTimeout = TimeSpan.FromSeconds(1),
};
var init = drv.InitializeAsync("{}", ct);
await f.FirstOpenEntered.Task.WaitAsync(Bound, ct);
// The shutdown cannot get the lifecycle gate (the attempt holds it), gives up after its bound
// and tears down anyway.
await drv.ShutdownAsync(ct).WaitAsync(Bound, ct);
drv.GetHealth().State.ShouldBe(DriverState.Unknown);
// The abandoned attempt now completes its open...
f.ReleaseFirstOpen.SetResult();
var ex = await Should.ThrowAsync<InvalidOperationException>(() => init.WaitAsync(Bound, ct));
ex.Message.ShouldContain("superseded");
// ...and published nothing: no connection kept, health untouched, the driver still shut down.
f.Factory.Created.ShouldHaveSingleItem().Disposed.ShouldBeTrue();
drv.GetHealth().State.ShouldBe(DriverState.Unknown, "a superseded attempt must not write Healthy or Faulted");
await Should.ThrowAsync<InvalidOperationException>(() => drv.ReadAsync(["W0"], ct));
// Positive control: the driver is not wedged — a fresh initialize publishes normally.
await drv.InitializeAsync("{}", ct).WaitAsync(Bound, ct);
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
(await drv.ReadAsync(["W0"], ct))[0].StatusCode.ShouldBe(0u);
}
/// <summary>
/// Dispose racing a shutdown that is still running is safe: neither throws, and the driver ends
/// torn down. The semaphores are no longer disposed, so the running shutdown cannot hit an
/// ObjectDisposedException.
/// </summary>
[Fact]
public async Task Dispose_while_a_shutdown_is_still_running_is_safe()
{
var ct = TestContext.Current.CancellationToken;
var f = new FirstOpenBlocks();
var drv = new S7Driver(Options(), "s7-dispose-race", f.Factory) { LifecycleGateTimeout = TimeSpan.FromSeconds(1) };
var init = drv.InitializeAsync("{}", ct);
await f.FirstOpenEntered.Task.WaitAsync(Bound, ct);
var shutdown = drv.ShutdownAsync(ct); // waits on the gate the blocked init holds
var dispose = drv.DisposeAsync().AsTask();
await shutdown.WaitAsync(Bound, ct);
await dispose.WaitAsync(Bound, ct);
f.ReleaseFirstOpen.SetResult();
await Should.ThrowAsync<InvalidOperationException>(() => init.WaitAsync(Bound, ct));
f.Factory.Created.ShouldHaveSingleItem().Disposed.ShouldBeTrue();
drv.GetHealth().State.ShouldBe(DriverState.Unknown);
}
}