Merge feat/676-opcuaclient-followups: sustained closed secure channel is a definitive session loss (#676 live-gate A1)
v2-ci / build (push) Successful in 4m53s
v2-ci / unit-tests (push) Failing after 24m38s

This commit is contained in:
Joseph Doherty
2026-10-11 12:35:09 -04:00
7 changed files with 218 additions and 17 deletions
+3 -2
View File
@@ -323,8 +323,9 @@ non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. T
reading. reading.
- **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises - **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises
`ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient: `ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient:
keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted` or a subscription the keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted`, a SUSTAINED closed
SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only connection (2nd `BadSecureChannelClosed`/`BadConnectionClosed` in one outage — what a real process stop produces; the
SDK never reopens it) or a subscription the SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only
while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with
the watchdog's conjunction. `docs/DriverLifecycle.md`. the watchdog's conjunction. `docs/DriverLifecycle.md`.
+7 -3
View File
@@ -458,11 +458,15 @@ alternatives:
communication" — must not raise: a stalled link that resumes keeps its session, subscriptions and in-place recovery, communication" — must not raise: a stalled link that resumes keeps its session, subscriptions and in-place recovery,
and tearing it down would cost all three. Those stay with the watchdog, whose conjunction is the honest test for them. and tearing it down would cost all three. Those stay with the watchdog, whose conjunction is the honest test for them.
- **Today only `OpcUaClientDriver` implements it:** keep-alive `BadSessionIdInvalid` / `BadSessionClosed` / - **Today only `OpcUaClientDriver` implements it:** keep-alive `BadSessionIdInvalid` / `BadSessionClosed` /
`BadSessionNotActivated` / `BadServerHalted`, or a subscription the server REPORTED dropped (not the driver's `BadSessionNotActivated` / `BadServerHalted`; a CLOSED connection (`BadSecureChannelClosed` /
`BadConnectionClosed`) once sustained — the second closed-channel keep-alive in one outage, which is what a real
process stop actually produces (live gate A1), and the SDK never reopens a closed connection by itself; or a
subscription the server REPORTED dropped (not the driver's
publishing-stall heuristic, which is an inference from silence and stays with the watchdog; see publishing-stall heuristic, which is an inference from silence and stays with the watchdog; see
[OpcUaClient.md](drivers/OpcUaClient.md) §"Upstream outage and recovery" for why `BadServerHalted` is on the list — [OpcUaClient.md](drivers/OpcUaClient.md) §"Upstream outage and recovery" for why `BadServerHalted` is on the list —
it was measured, the SDK never re-opens a halted channel). A gracefully restarted upstream therefore recovers in it was measured, the SDK never re-opens a halted channel). A restarted, stopped or killed upstream therefore
about one reconnect interval instead of ~100 s; a killed one still waits for the watchdog. recovers in about one keep-alive plus one reconnect interval instead of ~100 s; a stall/pause
(`BadRequestTimeout`/`BadNoCommunication`) still goes through the watchdog and keeps its in-place recovery.
- ⚠️ **Known gap:** "only while `Connected`" plus "once per connection" means a raise the host receives while not yet - ⚠️ **Known gap:** "only while `Connected`" plus "once per connection" means a raise the host receives while not yet
Connected (e.g. the protocol proves the fresh connection dead before `InitializeSucceeded` is processed) is swallowed, Connected (e.g. the protocol proves the fresh connection dead before `InitializeSucceeded` is processed) is swallowed,
and that connection falls back to the watchdog — the pre-#676 recovery time, in a narrow window. and that connection falls back to the watchdog — the pre-#676 recovery time, in a narrow window.
+19 -6
View File
@@ -137,11 +137,21 @@ The session keep-alive (`KeepAliveInterval`, default 5 s) is the liveness signal
because it was **measured**: on the restart wire test a gracefully stopped server reports `BadServerHalted` (and the because it was **measured**: on the restart wire test a gracefully stopped server reports `BadServerHalted` (and the
keep-alives alternate it with `BadNoCommunication`), and the SDK never re-opens a channel the server halted — 15 s keep-alives alternate it with `BadNoCommunication`), and the SDK never re-opens a channel the server halted — 15 s
after the server was back, the same session still failed every request with `BadServerHalted`; `BadSessionIdInvalid` after the server was back, the same session still failed every request with `BadServerHalted`; `BadSessionIdInvalid`
never arrived. Everything else — `BadNoCommunication`, `BadConnectionClosed`, `BadNotConnected`, never arrived.
`BadRequestTimeout`/`BadTimeout` (what a stalled link produces) — is **ambiguous** (the link may resume and keep its **A CLOSED connection is definitive too, once sustained (live gate A1).** Against a real opc-plc on docker-dev, none of
session, see the stall test) and stays with the watchdog. Consequence: a gracefully restarted upstream recovers in `docker restart`, a graceful SIGINT stop (opc-plc itself logged `BadServerHalted`) or `docker kill` delivered a
roughly one reconnect interval (≈10 s) instead of ~100 s; an upstream that vanishes without a halt (killed, cable session status to the client: the keep-alives reported `BadSecureChannelClosed` (0x80860000) once, then
pulled) still waits for the watchdog. ⚠️ **Known gap:** the host acts on the signal only while the driver is `BadConnectionClosed` (0x80AE0000) interleaved with `BadNoCommunication` for the whole outage, so the first cut of the
fast path never fired and recovery took 91–127 s. Reproduced in-process with `TcpStallProxy.HardClose` (every
connection reset, new ones refused, the server AND its session still alive behind it): the same sequence, and the SDK
1.5.378 session **never opens a new connection by itself** — 48 s after the proxy accepted connections again, not one
attempt. So a closed channel can never recover in place; on the **second** closed-channel keep-alive failure in one
outage (`IsClosedChannel`; a good keep-alive resets the count, so a single transient does not count) the driver raises
`ConnectionLost` — about one keep-alive interval after the connection dropped. Everything else —
`BadNoCommunication`, `BadNotConnected`, `BadRequestTimeout`/`BadTimeout` (what a stalled or paused link produces) —
is **ambiguous** (the link may resume and keep its session, see the stall test) and stays with the watchdog.
Consequence: a restarted, stopped or killed upstream is re-initialised within roughly one keep-alive plus one
reconnect interval instead of ~100 s; a pause (`docker pause`, a congested link) still recovers in place. ⚠️ **Known gap:** the host acts on the signal only while the driver is
`Connected`, and the driver raises it once per session — so a definitive loss raised while the host is not `Connected`, and the driver raises it once per session — so a definitive loss raised while the host is not
Connected (e.g. `BadServerHalted` on the fresh session before the host has processed the successful initialise) is Connected (e.g. `BadServerHalted` on the fresh session before the host has processed the successful initialise) is
swallowed and that session falls back to the watchdog. Re-raising on later keep-alives would close it but was left out swallowed and that session falls back to the watchdog. Re-raising on later keep-alives would close it but was left out
@@ -318,7 +328,10 @@ the reconnect handler, #670). They are not part of `DriverHealth` or the
used to be live-gate-only: the driver connects through `TcpStallProxy`, an in-test TCP forwarder whose pause holds used to be live-gate-only: the driver connects through `TcpStallProxy`, an in-test TCP forwarder whose pause holds
every byte without closing anything (which only works because of the endpoint rewrite above); it asserts one Bad, every byte without closing anything (which only works because of the endpoint rewrite above); it asserts one Bad,
a frozen read stamp, recovery on the SAME session, the identical value republished Good, a change published right a frozen read stamp, recovery on the SAME session, the identical value republished Good, a change published right
after recovery still arriving, and no source-time regression once the SDK's late backlog flush has arrived. after recovery still arriving, and no source-time regression once the SDK's late backlog flush has arrived. Its
second test uses `HardClose` (reset every connection, refuse new ones) — the shape of a real process stop — and
asserts the live-gate status sequence, that the SDK never reconnects on its own once the proxy accepts again,
`ConnectionLost` exactly once, and the re-initialise bringing the identical value back Good.
`OpcUaClientDriverReconnectWireTests` covers the server restart (incl. the `BadServerHalted` → `ConnectionLost` signal) `OpcUaClientDriverReconnectWireTests` covers the server restart (incl. the `BadServerHalted` → `ConnectionLost` signal)
and the subscription-lost case. and the subscription-lost case.
@@ -298,6 +298,14 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
/// cleared when the session is released, so a re-initialised driver reports its next loss.</summary> /// cleared when the session is released, so a re-initialised driver reports its next loss.</summary>
private int _connectionLostRaised; private int _connectionLostRaised;
/// <summary>#676 live gate A1 — closed-channel keep-alive failures (<see cref="IsClosedChannel"/>) in the current
/// outage; reset by a good keep-alive and when the session is released. Two make a closed channel "sustained".</summary>
private int _closedChannelFailures;
/// <summary>Closed-channel keep-alive failures in one outage that make the loss definitive (a single one may be a
/// transient that the next good keep-alive contradicts).</summary>
private const int SustainedClosedChannelFailures = 2;
/// <summary>#676 — the trailing reconcile read scheduled by the last in-place recovery. Test seam.</summary> /// <summary>#676 — the trailing reconcile read scheduled by the last in-place recovery. Test seam.</summary>
private Task _reconcileTask = Task.CompletedTask; private Task _reconcileTask = Task.CompletedTask;
@@ -1081,6 +1089,7 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
Interlocked.Exchange(ref _publishingStoppedSinceTicks, 0); Interlocked.Exchange(ref _publishingStoppedSinceTicks, 0);
// #676: re-arm the definitive-loss signal for the next session. // #676: re-arm the definitive-loss signal for the next session.
Interlocked.Exchange(ref _connectionLostRaised, 0); Interlocked.Exchange(ref _connectionLostRaised, 0);
Interlocked.Exchange(ref _closedChannelFailures, 0);
// Review of 3b88056f: the released session's post-recovery timers must not outlive it. // Review of 3b88056f: the released session's post-recovery timers must not outlive it.
CancelSessionWork(); CancelSessionWork();
} }
@@ -2869,6 +2878,7 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
if (!ServiceResult.IsBad(e.Status)) if (!ServiceResult.IsBad(e.Status))
{ {
Interlocked.Exchange(ref _closedChannelFailures, 0);
TransitionTo(HostState.Running); TransitionTo(HostState.Running);
// M1: the session answers but a subscription is dead — no stamp, no recovery; the watchdog reinit is the fix. // M1: the session answers but a subscription is dead — no stamp, no recovery; the watchdog reinit is the fix.
if (Volatile.Read(ref _subscriptionLost) == 1) if (Volatile.Read(ref _subscriptionLost) == 1)
@@ -2913,7 +2923,9 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
// #676: the first bad keep-alive is often ambiguous and a definitive one follows, so this is checked on EVERY // #676: the first bad keep-alive is often ambiguous and a definitive one follows, so this is checked on EVERY
// bad keep-alive — but only after the outage below has been published, so the Bad reaches subscribers first. // bad keep-alive — but only after the outage below has been published, so the Bad reaches subscribers first.
var keepAliveCode = e.Status?.StatusCode ?? StatusCodes.Good; var keepAliveCode = e.Status?.StatusCode ?? StatusCodes.Good;
var definitive = IsDefinitiveSessionLoss(keepAliveCode); var definitive = IsDefinitiveSessionLoss(keepAliveCode)
|| (IsClosedChannel(keepAliveCode)
&& Interlocked.Increment(ref _closedChannelFailures) >= SustainedClosedChannelFailures);
// A subscription loss already published Bad for this connection. // A subscription loss already published Bad for this connection.
if (Interlocked.Exchange(ref _sessionLost, 1) == 0 && Volatile.Read(ref _subscriptionLost) == 0) if (Interlocked.Exchange(ref _sessionLost, 1) == 0 && Volatile.Read(ref _subscriptionLost) == 0)
{ {
@@ -2927,6 +2939,23 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
RaiseConnectionLost($"upstream session keep-alive reported {StatusCodes.GetBrowseName(keepAliveCode.Code)}"); RaiseConnectionLost($"upstream session keep-alive reported {StatusCodes.GetBrowseName(keepAliveCode.Code)}");
} }
/// <summary>
/// #676 live gate A1 — the connection under the session is CLOSED (not merely silent). A real upstream process stop
/// (docker restart, SIGINT, docker kill) never produced a session status on the docker-dev gate: the keep-alives
/// reported <c>BadSecureChannelClosed</c> once, then <c>BadConnectionClosed</c> interleaved with
/// <c>BadNoCommunication</c> for the whole outage. Reproduced in-process (a proxy that resets every connection
/// while the server and its session stay alive): the SDK 1.5.378 session never opens a new connection on its
/// own — over 48 s with the server reachable again, not one connection attempt — so a closed channel can never
/// recover in place. It is treated as definitive once SUSTAINED (<see cref="SustainedClosedChannelFailures"/> in
/// one outage), so a single transient contradicted by the next good keep-alive does not count. A stall (a paused
/// upstream, a congested link) produces <c>BadRequestTimeout</c>/<c>BadNoCommunication</c> instead and keeps
/// its in-place recovery.
/// </summary>
/// <param name="status">A keep-alive status.</param>
/// <returns>True for a closed secure channel or connection.</returns>
internal static bool IsClosedChannel(StatusCode status) =>
status == StatusCodes.BadSecureChannelClosed || status == StatusCodes.BadConnectionClosed;
/// <summary> /// <summary>
/// #676 — keep-alive statuses that prove the session can NEVER recover in place, so waiting out the #527 /// #676 — keep-alive statuses that prove the session can NEVER recover in place, so waiting out the #527
/// watchdog would only delay the inevitable re-initialise. The server forgot the session /// watchdog would only delay the inevitable re-initialise. The server forgot the session
@@ -2934,9 +2963,10 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
/// (<c>BadSessionClosed</c>) or never activated it (<c>BadSessionNotActivated</c>); or the server HALTED /// (<c>BadSessionClosed</c>) or never activated it (<c>BadSessionNotActivated</c>); or the server HALTED
/// (<c>BadServerHalted</c>) — measured on the #676 restart wire test, the SDK never re-opens a channel the /// (<c>BadServerHalted</c>) — measured on the #676 restart wire test, the SDK never re-opens a channel the
/// server halted: 15 s after the server was back the same session still failed every request with /// server halted: 15 s after the server was back the same session still failed every request with
/// <c>BadServerHalted</c>, so a halted session is dead for good. Everything else — <c>BadNoCommunication</c>, /// <c>BadServerHalted</c>, so a halted session is dead for good. A closed connection is definitive too, but only
/// <c>BadConnectionClosed</c>, <c>BadNotConnected</c>, request timeouts — is AMBIGUOUS (a stalled link that /// once sustained — see <see cref="IsClosedChannel"/>. Everything else — <c>BadNoCommunication</c>,
/// resumes keeps its session, see the stall wire test) and stays with the watchdog. /// <c>BadNotConnected</c>, request timeouts — is AMBIGUOUS (a stalled link that resumes keeps its session, see the
/// stall wire test) and stays with the watchdog.
/// </summary> /// </summary>
/// <param name="status">A keep-alive status.</param> /// <param name="status">A keep-alive status.</param>
/// <returns>True when the session is definitively gone.</returns> /// <returns>True when the session is definitively gone.</returns>
@@ -105,11 +105,9 @@ public sealed class OpcUaClientConnectionLossSignalTests
public static TheoryData<uint> AmbiguousStatuses => public static TheoryData<uint> AmbiguousStatuses =>
[ [
StatusCodes.BadNoCommunication, StatusCodes.BadNoCommunication,
StatusCodes.BadConnectionClosed,
StatusCodes.BadNotConnected, StatusCodes.BadNotConnected,
StatusCodes.BadRequestTimeout, StatusCodes.BadRequestTimeout,
StatusCodes.BadTimeout, StatusCodes.BadTimeout,
StatusCodes.BadSecureChannelClosed,
]; ];
[Theory] [Theory]
@@ -153,6 +151,54 @@ public sealed class OpcUaClientConnectionLossSignalTests
raised.ShouldBe(0, "an ambiguous loss may yet resume in place — it belongs to the #527 watchdog"); raised.ShouldBe(0, "an ambiguous loss may yet resume in place — it belongs to the #527 watchdog");
} }
/// <summary>
/// #676 live gate A1: a real upstream process stop (docker restart / SIGINT / docker kill) never produced a
/// session status — the keep-alives carried <c>BadSecureChannelClosed</c> once, then <c>BadConnectionClosed</c>
/// (interleaved with <c>BadNoCommunication</c>) for the whole outage. Measured in-process (hard-closing proxy, the
/// server and its session still alive): the SDK 1.5.378 session NEVER opens a new connection on its own, so a
/// closed channel is dead for good. Sustained — a second closed-channel failure in the same outage — it raises.
/// </summary>
[Fact]
public void A_sustained_closed_channel_raises_ConnectionLost_once()
{
var (drv, _, _, _) = NewConnectedDriver();
using var _ = drv;
var raised = new List<string>();
drv.ConnectionLost += (_, e) => raised.Add(e.Reason);
drv.RaiseKeepAliveForTest(StatusCodes.BadSecureChannelClosed); // the live sequence …
raised.ShouldBeEmpty("one closed-channel failure may be a transient");
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
drv.RaiseKeepAliveForTest(StatusCodes.BadConnectionClosed); // … sustained
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
drv.RaiseKeepAliveForTest(StatusCodes.BadConnectionClosed);
raised.ShouldHaveSingleItem().ShouldContain("BadConnectionClosed");
}
[Theory]
[InlineData(0x80860000u)] // BadSecureChannelClosed
[InlineData(0x80AE0000u)] // BadConnectionClosed
public void A_single_closed_channel_blip_that_recovers_does_not_raise(uint closed)
{
var (drv, _, _, _) = NewConnectedDriver();
using var _ = drv;
var raised = 0;
drv.ConnectionLost += (_, _) => raised++;
drv.RaiseKeepAliveForTest(closed);
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
drv.RaiseKeepAliveForTest(StatusCodes.Good); // recovered — the count starts again
drv.RaiseKeepAliveForTest(closed);
drv.RaiseKeepAliveForTest(StatusCodes.Good);
raised.ShouldBe(0);
// Positive control: the same driver DOES raise once a closed channel is sustained.
drv.RaiseKeepAliveForTest(closed);
drv.RaiseKeepAliveForTest(closed);
raised.ShouldBe(1);
}
[Fact] [Fact]
public void An_ambiguous_outage_that_turns_definitive_raises_once() public void An_ambiguous_outage_that_turns_definitive_raises_once()
{ {
@@ -156,4 +156,76 @@ public sealed class OpcUaClientDriverStallWireTests
Volatile.Read(ref connectionLost).ShouldBe(0, Volatile.Read(ref connectionLost).ShouldBe(0,
"a stall is AMBIGUOUS (the session survived it) — it must go through the #527 watchdog, never the fast path"); "a stall is AMBIGUOUS (the session survived it) — it must go through the #527 watchdog, never the fast path");
} }
/// <summary>
/// #676 live gate A1 — a real upstream process stop is a CLOSED connection, not a stall: the keep-alives report
/// <c>BadSecureChannelClosed</c>, then <c>BadConnectionClosed</c> (with <c>BadNoCommunication</c>) for the rest of
/// the outage, and the SDK never opens a new connection on its own — not even when the server (here, with the
/// session still alive behind the proxy) is reachable again. The driver must therefore raise
/// <c>ConnectionLost</c> once the closed channel is sustained, so the host re-initialises in seconds instead of
/// waiting ~90 s for the watchdog; the re-initialise brings the identical value back Good.
/// </summary>
[Fact]
public async Task A_hard_closed_connection_raises_ConnectionLost_once_and_the_reinit_recovers_in_seconds()
{
var ct = TestContext.Current.CancellationToken;
await using var rig = await OpcUaClientDriverWireTests.StartRigAsync("drv-hardclose", ct);
await using var proxy = TcpStallProxy.Start(new Uri(rig.EndpointUrl).Port);
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 10.0f);
await using var drv = OpcUaClientDriverWireTests.NewDriver(
rig, "wire-hardclose", endpointUrl: proxy.Through(rig.EndpointUrl), sessionTimeout: TimeSpan.FromSeconds(120));
var lost = new List<string>();
drv.ConnectionLost += (_, e) => { lock (lost) lost.Add(e.Reason); };
await drv.InitializeAsync("{}", ct);
var seen = new List<Seen>();
drv.OnDataChange += (_, e) =>
{
if (e.FullReference == OpcUaClientDriverWireTests.SpeedRef)
lock (seen) seen.Add(new Seen(e.Snapshot.StatusCode, e.Snapshot.Value, e.Snapshot.SourceTimestampUtc));
};
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
await WaitAsync(() => { lock (seen) return seen.Any(s => s is { Status: 0u, Value: 10.0f }); }, "initial value");
var session = drv.Session.ShouldNotBeNull();
var keepAlives = new List<uint>();
session.KeepAlive += (_, e) => { lock (keepAlives) keepAlives.Add(e.Status?.StatusCode.Code ?? 0); };
// ── The upstream process goes away: every connection reset, new ones refused ──────────────────────────────
var killedAt = DateTime.UtcNow;
proxy.HardClose();
await WaitAsync(() => { lock (lost) return lost.Count > 0; },
"a sustained closed channel must raise ConnectionLost", TimeSpan.FromSeconds(15));
(DateTime.UtcNow - killedAt).ShouldBeLessThan(TimeSpan.FromSeconds(15), "seconds, not the watchdog's ~90 s");
lock (seen) seen.Count(s => s.Status == BadNoCommunication).ShouldBe(1, "the outage was published Bad first");
lock (keepAlives)
{
keepAlives.ShouldContain(0x80860000u, "BadSecureChannelClosed — the live-gate sequence");
keepAlives.ShouldContain(0x80AE0000u, "BadConnectionClosed — the live-gate sequence");
}
// The server is back — the session still exists on it — yet the SDK never reconnects by itself: no new
// connection reaches the proxy while further keep-alives fail. That is what makes the closed channel definitive.
proxy.Reopen();
var accepted = proxy.AcceptedConnections;
int kaAtReopen;
lock (keepAlives) kaAtReopen = keepAlives.Count;
await WaitAsync(() => { lock (keepAlives) return keepAlives.Count >= kaAtReopen + 4; }, "keep-alives keep failing");
proxy.AcceptedConnections.ShouldBe(accepted, "the SDK session never reopens a closed connection on its own");
drv.GetHealth().State.ShouldNotBe(DriverState.Healthy);
lock (lost) lost.Count.ShouldBe(1, "once per session, however many closed-channel keep-alives follow");
// ── The host's response (DriverInstanceActor: Reconnecting → ShutdownAsync + InitializeAsync → re-subscribe) ──
await drv.ShutdownAsync(ct);
await drv.InitializeAsync("{}", ct);
int before;
lock (seen) before = seen.Count;
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
await WaitAsync(() => { lock (seen) return seen.Skip(before).Any(s => s is { Status: 0u, Value: 10.0f }); },
"the identical value must come back Good after the re-initialise");
(DateTime.UtcNow - killedAt).ShouldBeLessThan(TimeSpan.FromSeconds(60), "well under the watchdog's 90 s");
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 77.0f);
await WaitAsync(() => { lock (seen) return seen.Any(s => s is { Status: 0u, Value: 77.0f }); }, "a later change arrives");
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
lock (lost) lost.Count.ShouldBe(1, "the healthy re-initialised session reports no loss");
}
} }
@@ -19,6 +19,7 @@ internal sealed class TcpStallProxy : IAsyncDisposable
private readonly List<TcpClient> _sockets = []; private readonly List<TcpClient> _sockets = [];
private readonly Lock _lock = new(); private readonly Lock _lock = new();
private TaskCompletionSource _gate = Open(); private TaskCompletionSource _gate = Open();
private volatile bool _refusing;
private Task _acceptLoop = Task.CompletedTask; private Task _acceptLoop = Task.CompletedTask;
private TcpStallProxy(int targetPort) => _targetPort = targetPort; private TcpStallProxy(int targetPort) => _targetPort = targetPort;
@@ -48,6 +49,34 @@ internal sealed class TcpStallProxy : IAsyncDisposable
if (_gate.Task.IsCompleted) _gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); if (_gate.Task.IsCompleted) _gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
} }
/// <summary>
/// The shape of a killed / stopped server process (#676 live gate): every open connection is RESET (linger 0 ⇒
/// RST, both sides) and new connections are refused (accepted and reset at once) until <see cref="Reopen"/>.
/// </summary>
public void HardClose()
{
List<TcpClient> victims;
lock (_lock)
{
_refusing = true;
victims = [.. _sockets];
_sockets.Clear();
}
foreach (var s in victims) Reset(s);
}
/// <summary>Accept and forward new connections again (the server is back); the reset ones stay dead.</summary>
public void Reopen() => _refusing = false;
/// <summary>Connections refused while <see cref="HardClose"/> was in force — a reconnect ATTEMPT by the client.</summary>
public int RefusedConnections { get; private set; }
private static void Reset(TcpClient s)
{
try { s.Client.LingerState = new LingerOption(true, 0); } catch { }
try { s.Dispose(); } catch { }
}
/// <summary>Releases everything held since <see cref="Pause"/>, in order.</summary> /// <summary>Releases everything held since <see cref="Pause"/>, in order.</summary>
public void Resume() public void Resume()
{ {
@@ -73,6 +102,12 @@ internal sealed class TcpStallProxy : IAsyncDisposable
TcpClient client; TcpClient client;
try { client = await _listener.AcceptTcpClientAsync(_cts.Token); } try { client = await _listener.AcceptTcpClientAsync(_cts.Token); }
catch { return; } catch { return; }
if (_refusing)
{
lock (_lock) RefusedConnections++;
Reset(client);
continue;
}
var upstream = new TcpClient(); var upstream = new TcpClient();
try { await upstream.ConnectAsync(IPAddress.Loopback, _targetPort, _cts.Token); } try { await upstream.ConnectAsync(IPAddress.Loopback, _targetPort, _cts.Token); }
catch catch