Merge feat/676-opcuaclient-followups: sustained closed secure channel is a definitive session loss (#676 live-gate A1)
v2-ci / build (push) Successful in 4m53s
v2-ci / unit-tests (push) Failing after 24m38s

This commit is contained in:
Joseph Doherty
2026-10-11 12:35:09 -04:00
7 changed files with 218 additions and 17 deletions
+3 -2
View File
@@ -323,8 +323,9 @@ non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. T
reading.
- **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises
`ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient:
keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted` or a subscription the
SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only
keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted`, a SUSTAINED closed
connection (2nd `BadSecureChannelClosed`/`BadConnectionClosed` in one outage — what a real process stop produces; the
SDK never reopens it) or a subscription the SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only
while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with
the watchdog's conjunction. `docs/DriverLifecycle.md`.
+7 -3
View File
@@ -458,11 +458,15 @@ alternatives:
communication" — must not raise: a stalled link that resumes keeps its session, subscriptions and in-place recovery,
and tearing it down would cost all three. Those stay with the watchdog, whose conjunction is the honest test for them.
- **Today only `OpcUaClientDriver` implements it:** keep-alive `BadSessionIdInvalid` / `BadSessionClosed` /
`BadSessionNotActivated` / `BadServerHalted`, or a subscription the server REPORTED dropped (not the driver's
`BadSessionNotActivated` / `BadServerHalted`; a CLOSED connection (`BadSecureChannelClosed` /
`BadConnectionClosed`) once sustained — the second closed-channel keep-alive in one outage, which is what a real
process stop actually produces (live gate A1), and the SDK never reopens a closed connection by itself; or a
subscription the server REPORTED dropped (not the driver's
publishing-stall heuristic, which is an inference from silence and stays with the watchdog; see
[OpcUaClient.md](drivers/OpcUaClient.md) §"Upstream outage and recovery" for why `BadServerHalted` is on the list —
it was measured, the SDK never re-opens a halted channel). A gracefully restarted upstream therefore recovers in
about one reconnect interval instead of ~100 s; a killed one still waits for the watchdog.
it was measured, the SDK never re-opens a halted channel). A restarted, stopped or killed upstream therefore
recovers in about one keep-alive plus one reconnect interval instead of ~100 s; a stall/pause
(`BadRequestTimeout`/`BadNoCommunication`) still goes through the watchdog and keeps its in-place recovery.
- ⚠️ **Known gap:** "only while `Connected`" plus "once per connection" means a raise the host receives while not yet
Connected (e.g. the protocol proves the fresh connection dead before `InitializeSucceeded` is processed) is swallowed,
and that connection falls back to the watchdog — the pre-#676 recovery time, in a narrow window.
+19 -6
View File
@@ -137,11 +137,21 @@ The session keep-alive (`KeepAliveInterval`, default 5 s) is the liveness signal
because it was **measured**: on the restart wire test a gracefully stopped server reports `BadServerHalted` (and the
keep-alives alternate it with `BadNoCommunication`), and the SDK never re-opens a channel the server halted — 15 s
after the server was back, the same session still failed every request with `BadServerHalted`; `BadSessionIdInvalid`
never arrived. Everything else — `BadNoCommunication`, `BadConnectionClosed`, `BadNotConnected`,
`BadRequestTimeout`/`BadTimeout` (what a stalled link produces) — is **ambiguous** (the link may resume and keep its
session, see the stall test) and stays with the watchdog. Consequence: a gracefully restarted upstream recovers in
roughly one reconnect interval (≈10 s) instead of ~100 s; an upstream that vanishes without a halt (killed, cable
pulled) still waits for the watchdog. ⚠️ **Known gap:** the host acts on the signal only while the driver is
never arrived.
**A CLOSED connection is definitive too, once sustained (live gate A1).** Against a real opc-plc on docker-dev, none of
`docker restart`, a graceful SIGINT stop (opc-plc itself logged `BadServerHalted`) or `docker kill` delivered a
session status to the client: the keep-alives reported `BadSecureChannelClosed` (0x80860000) once, then
`BadConnectionClosed` (0x80AE0000) interleaved with `BadNoCommunication` for the whole outage, so the first cut of the
fast path never fired and recovery took 91–127 s. Reproduced in-process with `TcpStallProxy.HardClose` (every
connection reset, new ones refused, the server AND its session still alive behind it): the same sequence, and the SDK
1.5.378 session **never opens a new connection by itself** — 48 s after the proxy accepted connections again, not one
attempt. So a closed channel can never recover in place; on the **second** closed-channel keep-alive failure in one
outage (`IsClosedChannel`; a good keep-alive resets the count, so a single transient does not count) the driver raises
`ConnectionLost` — about one keep-alive interval after the connection dropped. Everything else —
`BadNoCommunication`, `BadNotConnected`, `BadRequestTimeout`/`BadTimeout` (what a stalled or paused link produces) —
is **ambiguous** (the link may resume and keep its session, see the stall test) and stays with the watchdog.
Consequence: a restarted, stopped or killed upstream is re-initialised within roughly one keep-alive plus one
reconnect interval instead of ~100 s; a pause (`docker pause`, a congested link) still recovers in place. ⚠️ **Known gap:** the host acts on the signal only while the driver is
`Connected`, and the driver raises it once per session — so a definitive loss raised while the host is not
Connected (e.g. `BadServerHalted` on the fresh session before the host has processed the successful initialise) is
swallowed and that session falls back to the watchdog. Re-raising on later keep-alives would close it but was left out
@@ -318,7 +328,10 @@ the reconnect handler, #670). They are not part of `DriverHealth` or the
used to be live-gate-only: the driver connects through `TcpStallProxy`, an in-test TCP forwarder whose pause holds
every byte without closing anything (which only works because of the endpoint rewrite above); it asserts one Bad,
a frozen read stamp, recovery on the SAME session, the identical value republished Good, a change published right
after recovery still arriving, and no source-time regression once the SDK's late backlog flush has arrived.
after recovery still arriving, and no source-time regression once the SDK's late backlog flush has arrived. Its
second test uses `HardClose` (reset every connection, refuse new ones) — the shape of a real process stop — and
asserts the live-gate status sequence, that the SDK never reconnects on its own once the proxy accepts again,
`ConnectionLost` exactly once, and the re-initialise bringing the identical value back Good.
`OpcUaClientDriverReconnectWireTests` covers the server restart (incl. the `BadServerHalted` → `ConnectionLost` signal)
and the subscription-lost case.
@@ -298,6 +298,14 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
/// cleared when the session is released, so a re-initialised driver reports its next loss.</summary>
private int _connectionLostRaised;
/// <summary>#676 live gate A1 — closed-channel keep-alive failures (<see cref="IsClosedChannel"/>) in the current
/// outage; reset by a good keep-alive and when the session is released. Two make a closed channel "sustained".</summary>
private int _closedChannelFailures;
/// <summary>Closed-channel keep-alive failures in one outage that make the loss definitive (a single one may be a
/// transient that the next good keep-alive contradicts).</summary>
private const int SustainedClosedChannelFailures = 2;
/// <summary>#676 — the trailing reconcile read scheduled by the last in-place recovery. Test seam.</summary>
private Task _reconcileTask = Task.CompletedTask;
@@ -1081,6 +1089,7 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
Interlocked.Exchange(ref _publishingStoppedSinceTicks, 0);
// #676: re-arm the definitive-loss signal for the next session.
Interlocked.Exchange(ref _connectionLostRaised, 0);
Interlocked.Exchange(ref _closedChannelFailures, 0);
// Review of 3b88056f: the released session's post-recovery timers must not outlive it.
CancelSessionWork();
}
@@ -2869,6 +2878,7 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
if (!ServiceResult.IsBad(e.Status))
{
Interlocked.Exchange(ref _closedChannelFailures, 0);
TransitionTo(HostState.Running);
// M1: the session answers but a subscription is dead — no stamp, no recovery; the watchdog reinit is the fix.
if (Volatile.Read(ref _subscriptionLost) == 1)
@@ -2913,7 +2923,9 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
// #676: the first bad keep-alive is often ambiguous and a definitive one follows, so this is checked on EVERY
// bad keep-alive — but only after the outage below has been published, so the Bad reaches subscribers first.
var keepAliveCode = e.Status?.StatusCode ?? StatusCodes.Good;
var definitive = IsDefinitiveSessionLoss(keepAliveCode);
var definitive = IsDefinitiveSessionLoss(keepAliveCode)
|| (IsClosedChannel(keepAliveCode)
&& Interlocked.Increment(ref _closedChannelFailures) >= SustainedClosedChannelFailures);
// A subscription loss already published Bad for this connection.
if (Interlocked.Exchange(ref _sessionLost, 1) == 0 && Volatile.Read(ref _subscriptionLost) == 0)
{
@@ -2927,6 +2939,23 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
RaiseConnectionLost($"upstream session keep-alive reported {StatusCodes.GetBrowseName(keepAliveCode.Code)}");
}
/// <summary>
/// #676 live gate A1 — the connection under the session is CLOSED (not merely silent). A real upstream process stop
/// (docker restart, SIGINT, docker kill) never produced a session status on the docker-dev gate: the keep-alives
/// reported <c>BadSecureChannelClosed</c> once, then <c>BadConnectionClosed</c> interleaved with
/// <c>BadNoCommunication</c> for the whole outage. Reproduced in-process (a proxy that resets every connection
/// while the server and its session stay alive): the SDK 1.5.378 session never opens a new connection on its
/// own — over 48 s with the server reachable again, not one connection attempt — so a closed channel can never
/// recover in place. It is treated as definitive once SUSTAINED (<see cref="SustainedClosedChannelFailures"/> in
/// one outage), so a single transient contradicted by the next good keep-alive does not count. A stall (a paused
/// upstream, a congested link) produces <c>BadRequestTimeout</c>/<c>BadNoCommunication</c> instead and keeps
/// its in-place recovery.
/// </summary>
/// <param name="status">A keep-alive status.</param>
/// <returns>True for a closed secure channel or connection.</returns>
internal static bool IsClosedChannel(StatusCode status) =>
status == StatusCodes.BadSecureChannelClosed || status == StatusCodes.BadConnectionClosed;
/// <summary>
/// #676 — keep-alive statuses that prove the session can NEVER recover in place, so waiting out the #527
/// watchdog would only delay the inevitable re-initialise. The server forgot the session
@@ -2934,9 +2963,10 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
/// (<c>BadSessionClosed</c>) or never activated it (<c>BadSessionNotActivated</c>); or the server HALTED
/// (<c>BadServerHalted</c>) — measured on the #676 restart wire test, the SDK never re-opens a channel the
/// server halted: 15 s after the server was back the same session still failed every request with
/// <c>BadServerHalted</c>, so a halted session is dead for good. Everything else — <c>BadNoCommunication</c>,
/// <c>BadConnectionClosed</c>, <c>BadNotConnected</c>, request timeouts — is AMBIGUOUS (a stalled link that
/// resumes keeps its session, see the stall wire test) and stays with the watchdog.
/// <c>BadServerHalted</c>, so a halted session is dead for good. A closed connection is definitive too, but only
/// once sustained — see <see cref="IsClosedChannel"/>. Everything else — <c>BadNoCommunication</c>,
/// <c>BadNotConnected</c>, request timeouts — is AMBIGUOUS (a stalled link that resumes keeps its session, see the
/// stall wire test) and stays with the watchdog.
/// </summary>
/// <param name="status">A keep-alive status.</param>
/// <returns>True when the session is definitively gone.</returns>
@@ -105,11 +105,9 @@ public sealed class OpcUaClientConnectionLossSignalTests
public static TheoryData<uint> AmbiguousStatuses =>
[
StatusCodes.BadNoCommunication,
StatusCodes.BadConnectionClosed,
StatusCodes.BadNotConnected,
StatusCodes.BadRequestTimeout,
StatusCodes.BadTimeout,
StatusCodes.BadSecureChannelClosed,
];
[Theory]
@@ -153,6 +151,54 @@ public sealed class OpcUaClientConnectionLossSignalTests
raised.ShouldBe(0, "an ambiguous loss may yet resume in place — it belongs to the #527 watchdog");
}
/// <summary>
/// #676 live gate A1: a real upstream process stop (docker restart / SIGINT / docker kill) never produced a
/// session status — the keep-alives carried <c>BadSecureChannelClosed</c> once, then <c>BadConnectionClosed</c>
/// (interleaved with <c>BadNoCommunication</c>) for the whole outage. Measured in-process (hard-closing proxy, the
/// server and its session still alive): the SDK 1.5.378 session NEVER opens a new connection on its own, so a
/// closed channel is dead for good. Sustained — a second closed-channel failure in the same outage — it raises.
/// </summary>
[Fact]
public void A_sustained_closed_channel_raises_ConnectionLost_once()
{
var (drv, _, _, _) = NewConnectedDriver();
using var _ = drv;
var raised = new List<string>();
drv.ConnectionLost += (_, e) => raised.Add(e.Reason);
drv.RaiseKeepAliveForTest(StatusCodes.BadSecureChannelClosed); // the live sequence …
raised.ShouldBeEmpty("one closed-channel failure may be a transient");
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
drv.RaiseKeepAliveForTest(StatusCodes.BadConnectionClosed); // … sustained
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
drv.RaiseKeepAliveForTest(StatusCodes.BadConnectionClosed);
raised.ShouldHaveSingleItem().ShouldContain("BadConnectionClosed");
}
[Theory]
[InlineData(0x80860000u)] // BadSecureChannelClosed
[InlineData(0x80AE0000u)] // BadConnectionClosed
public void A_single_closed_channel_blip_that_recovers_does_not_raise(uint closed)
{
var (drv, _, _, _) = NewConnectedDriver();
using var _ = drv;
var raised = 0;
drv.ConnectionLost += (_, _) => raised++;
drv.RaiseKeepAliveForTest(closed);
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
drv.RaiseKeepAliveForTest(StatusCodes.Good); // recovered — the count starts again
drv.RaiseKeepAliveForTest(closed);
drv.RaiseKeepAliveForTest(StatusCodes.Good);
raised.ShouldBe(0);
// Positive control: the same driver DOES raise once a closed channel is sustained.
drv.RaiseKeepAliveForTest(closed);
drv.RaiseKeepAliveForTest(closed);
raised.ShouldBe(1);
}
[Fact]
public void An_ambiguous_outage_that_turns_definitive_raises_once()
{
@@ -156,4 +156,76 @@ public sealed class OpcUaClientDriverStallWireTests
Volatile.Read(ref connectionLost).ShouldBe(0,
"a stall is AMBIGUOUS (the session survived it) — it must go through the #527 watchdog, never the fast path");
}
/// <summary>
/// #676 live gate A1 — a real upstream process stop is a CLOSED connection, not a stall: the keep-alives report
/// <c>BadSecureChannelClosed</c>, then <c>BadConnectionClosed</c> (with <c>BadNoCommunication</c>) for the rest of
/// the outage, and the SDK never opens a new connection on its own — not even when the server (here, with the
/// session still alive behind the proxy) is reachable again. The driver must therefore raise
/// <c>ConnectionLost</c> once the closed channel is sustained, so the host re-initialises in seconds instead of
/// waiting ~90 s for the watchdog; the re-initialise brings the identical value back Good.
/// </summary>
[Fact]
public async Task A_hard_closed_connection_raises_ConnectionLost_once_and_the_reinit_recovers_in_seconds()
{
var ct = TestContext.Current.CancellationToken;
await using var rig = await OpcUaClientDriverWireTests.StartRigAsync("drv-hardclose", ct);
await using var proxy = TcpStallProxy.Start(new Uri(rig.EndpointUrl).Port);
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 10.0f);
await using var drv = OpcUaClientDriverWireTests.NewDriver(
rig, "wire-hardclose", endpointUrl: proxy.Through(rig.EndpointUrl), sessionTimeout: TimeSpan.FromSeconds(120));
var lost = new List<string>();
drv.ConnectionLost += (_, e) => { lock (lost) lost.Add(e.Reason); };
await drv.InitializeAsync("{}", ct);
var seen = new List<Seen>();
drv.OnDataChange += (_, e) =>
{
if (e.FullReference == OpcUaClientDriverWireTests.SpeedRef)
lock (seen) seen.Add(new Seen(e.Snapshot.StatusCode, e.Snapshot.Value, e.Snapshot.SourceTimestampUtc));
};
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
await WaitAsync(() => { lock (seen) return seen.Any(s => s is { Status: 0u, Value: 10.0f }); }, "initial value");
var session = drv.Session.ShouldNotBeNull();
var keepAlives = new List<uint>();
session.KeepAlive += (_, e) => { lock (keepAlives) keepAlives.Add(e.Status?.StatusCode.Code ?? 0); };
// ── The upstream process goes away: every connection reset, new ones refused ──────────────────────────────
var killedAt = DateTime.UtcNow;
proxy.HardClose();
await WaitAsync(() => { lock (lost) return lost.Count > 0; },
"a sustained closed channel must raise ConnectionLost", TimeSpan.FromSeconds(15));
(DateTime.UtcNow - killedAt).ShouldBeLessThan(TimeSpan.FromSeconds(15), "seconds, not the watchdog's ~90 s");
lock (seen) seen.Count(s => s.Status == BadNoCommunication).ShouldBe(1, "the outage was published Bad first");
lock (keepAlives)
{
keepAlives.ShouldContain(0x80860000u, "BadSecureChannelClosed — the live-gate sequence");
keepAlives.ShouldContain(0x80AE0000u, "BadConnectionClosed — the live-gate sequence");
}
// The server is back — the session still exists on it — yet the SDK never reconnects by itself: no new
// connection reaches the proxy while further keep-alives fail. That is what makes the closed channel definitive.
proxy.Reopen();
var accepted = proxy.AcceptedConnections;
int kaAtReopen;
lock (keepAlives) kaAtReopen = keepAlives.Count;
await WaitAsync(() => { lock (keepAlives) return keepAlives.Count >= kaAtReopen + 4; }, "keep-alives keep failing");
proxy.AcceptedConnections.ShouldBe(accepted, "the SDK session never reopens a closed connection on its own");
drv.GetHealth().State.ShouldNotBe(DriverState.Healthy);
lock (lost) lost.Count.ShouldBe(1, "once per session, however many closed-channel keep-alives follow");
// ── The host's response (DriverInstanceActor: Reconnecting → ShutdownAsync + InitializeAsync → re-subscribe) ──
await drv.ShutdownAsync(ct);
await drv.InitializeAsync("{}", ct);
int before;
lock (seen) before = seen.Count;
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
await WaitAsync(() => { lock (seen) return seen.Skip(before).Any(s => s is { Status: 0u, Value: 10.0f }); },
"the identical value must come back Good after the re-initialise");
(DateTime.UtcNow - killedAt).ShouldBeLessThan(TimeSpan.FromSeconds(60), "well under the watchdog's 90 s");
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 77.0f);
await WaitAsync(() => { lock (seen) return seen.Any(s => s is { Status: 0u, Value: 77.0f }); }, "a later change arrives");
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
lock (lost) lost.Count.ShouldBe(1, "the healthy re-initialised session reports no loss");
}
}
@@ -19,6 +19,7 @@ internal sealed class TcpStallProxy : IAsyncDisposable
private readonly List<TcpClient> _sockets = [];
private readonly Lock _lock = new();
private TaskCompletionSource _gate = Open();
private volatile bool _refusing;
private Task _acceptLoop = Task.CompletedTask;
private TcpStallProxy(int targetPort) => _targetPort = targetPort;
@@ -48,6 +49,34 @@ internal sealed class TcpStallProxy : IAsyncDisposable
if (_gate.Task.IsCompleted) _gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
}
/// <summary>
/// The shape of a killed / stopped server process (#676 live gate): every open connection is RESET (linger 0 ⇒
/// RST, both sides) and new connections are refused (accepted and reset at once) until <see cref="Reopen"/>.
/// </summary>
public void HardClose()
{
List<TcpClient> victims;
lock (_lock)
{
_refusing = true;
victims = [.. _sockets];
_sockets.Clear();
}
foreach (var s in victims) Reset(s);
}
/// <summary>Accept and forward new connections again (the server is back); the reset ones stay dead.</summary>
public void Reopen() => _refusing = false;
/// <summary>Connections refused while <see cref="HardClose"/> was in force — a reconnect ATTEMPT by the client.</summary>
public int RefusedConnections { get; private set; }
private static void Reset(TcpClient s)
{
try { s.Client.LingerState = new LingerOption(true, 0); } catch { }
try { s.Dispose(); } catch { }
}
/// <summary>Releases everything held since <see cref="Pause"/>, in order.</summary>
public void Resume()
{
@@ -73,6 +102,12 @@ internal sealed class TcpStallProxy : IAsyncDisposable
TcpClient client;
try { client = await _listener.AcceptTcpClientAsync(_cts.Token); }
catch { return; }
if (_refusing)
{
lock (_lock) RefusedConnections++;
Reset(client);
continue;
}
var upstream = new TcpClient();
try { await upstream.ConnectAsync(IPAddress.Loopback, _targetPort, _cts.Token); }
catch