fix(opcuaclient): #676 live gate A1 — a sustained closed connection is a definitive loss
Live gate A1 (docker-dev, opc-plc restart / SIGINT / kill): the fast path never fired — a real process stop delivers no session status; the keep-alives report BadSecureChannelClosed (0x80860000) once, then BadConnectionClosed (0x80AE0000) interleaved with BadNoCommunication for the whole outage. Reproduced in-process with TcpStallProxy.HardClose (reset every connection, refuse new ones; server and session alive behind it): the identical sequence, and the SDK 1.5.378 session never opens a new connection on its own (48 s with the proxy accepting again, zero attempts) — a closed channel cannot recover in place. The driver now treats BadSecureChannelClosed/BadConnectionClosed as definitive on the SECOND closed-channel keep-alive failure in one outage (a good keep-alive resets the count), raising ConnectionLost ~1 keep-alive after the drop. BadNoCommunication / BadRequestTimeout / BadNotConnected (stall/pause) stay with the watchdog; the stall test still sees no ConnectionLost. New wire test (hard close + reopen ⇒ raised once, SDK never reconnects, reinit brings the identical value back Good) and unit tests (sustained raises once; a single blip that recovers does not) — all red on the previous code. Docs updated.
This commit is contained in:
@@ -323,8 +323,9 @@ non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. T
|
|||||||
reading.
|
reading.
|
||||||
- **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises
|
- **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises
|
||||||
`ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient:
|
`ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient:
|
||||||
keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted` or a subscription the
|
keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted`, a SUSTAINED closed
|
||||||
SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only
|
connection (2nd `BadSecureChannelClosed`/`BadConnectionClosed` in one outage — what a real process stop produces; the
|
||||||
|
SDK never reopens it) or a subscription the SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only
|
||||||
while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with
|
while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with
|
||||||
the watchdog's conjunction. `docs/DriverLifecycle.md`.
|
the watchdog's conjunction. `docs/DriverLifecycle.md`.
|
||||||
|
|
||||||
|
|||||||
@@ -458,11 +458,15 @@ alternatives:
|
|||||||
communication" — must not raise: a stalled link that resumes keeps its session, subscriptions and in-place recovery,
|
communication" — must not raise: a stalled link that resumes keeps its session, subscriptions and in-place recovery,
|
||||||
and tearing it down would cost all three. Those stay with the watchdog, whose conjunction is the honest test for them.
|
and tearing it down would cost all three. Those stay with the watchdog, whose conjunction is the honest test for them.
|
||||||
- **Today only `OpcUaClientDriver` implements it:** keep-alive `BadSessionIdInvalid` / `BadSessionClosed` /
|
- **Today only `OpcUaClientDriver` implements it:** keep-alive `BadSessionIdInvalid` / `BadSessionClosed` /
|
||||||
`BadSessionNotActivated` / `BadServerHalted`, or a subscription the server REPORTED dropped (not the driver's
|
`BadSessionNotActivated` / `BadServerHalted`; a CLOSED connection (`BadSecureChannelClosed` /
|
||||||
|
`BadConnectionClosed`) once sustained — the second closed-channel keep-alive in one outage, which is what a real
|
||||||
|
process stop actually produces (live gate A1), and the SDK never reopens a closed connection by itself; or a
|
||||||
|
subscription the server REPORTED dropped (not the driver's
|
||||||
publishing-stall heuristic, which is an inference from silence and stays with the watchdog; see
|
publishing-stall heuristic, which is an inference from silence and stays with the watchdog; see
|
||||||
[OpcUaClient.md](drivers/OpcUaClient.md) §"Upstream outage and recovery" for why `BadServerHalted` is on the list —
|
[OpcUaClient.md](drivers/OpcUaClient.md) §"Upstream outage and recovery" for why `BadServerHalted` is on the list —
|
||||||
it was measured, the SDK never re-opens a halted channel). A gracefully restarted upstream therefore recovers in
|
it was measured, the SDK never re-opens a halted channel). A restarted, stopped or killed upstream therefore
|
||||||
about one reconnect interval instead of ~100 s; a killed one still waits for the watchdog.
|
recovers in about one keep-alive plus one reconnect interval instead of ~100 s; a stall/pause
|
||||||
|
(`BadRequestTimeout`/`BadNoCommunication`) still goes through the watchdog and keeps its in-place recovery.
|
||||||
- ⚠️ **Known gap:** "only while `Connected`" plus "once per connection" means a raise the host receives while not yet
|
- ⚠️ **Known gap:** "only while `Connected`" plus "once per connection" means a raise the host receives while not yet
|
||||||
Connected (e.g. the protocol proves the fresh connection dead before `InitializeSucceeded` is processed) is swallowed,
|
Connected (e.g. the protocol proves the fresh connection dead before `InitializeSucceeded` is processed) is swallowed,
|
||||||
and that connection falls back to the watchdog — the pre-#676 recovery time, in a narrow window.
|
and that connection falls back to the watchdog — the pre-#676 recovery time, in a narrow window.
|
||||||
|
|||||||
@@ -137,11 +137,21 @@ The session keep-alive (`KeepAliveInterval`, default 5 s) is the liveness signal
|
|||||||
because it was **measured**: on the restart wire test a gracefully stopped server reports `BadServerHalted` (and the
|
because it was **measured**: on the restart wire test a gracefully stopped server reports `BadServerHalted` (and the
|
||||||
keep-alives alternate it with `BadNoCommunication`), and the SDK never re-opens a channel the server halted — 15 s
|
keep-alives alternate it with `BadNoCommunication`), and the SDK never re-opens a channel the server halted — 15 s
|
||||||
after the server was back, the same session still failed every request with `BadServerHalted`; `BadSessionIdInvalid`
|
after the server was back, the same session still failed every request with `BadServerHalted`; `BadSessionIdInvalid`
|
||||||
never arrived. Everything else — `BadNoCommunication`, `BadConnectionClosed`, `BadNotConnected`,
|
never arrived.
|
||||||
`BadRequestTimeout`/`BadTimeout` (what a stalled link produces) — is **ambiguous** (the link may resume and keep its
|
**A CLOSED connection is definitive too, once sustained (live gate A1).** Against a real opc-plc on docker-dev, none of
|
||||||
session, see the stall test) and stays with the watchdog. Consequence: a gracefully restarted upstream recovers in
|
`docker restart`, a graceful SIGINT stop (opc-plc itself logged `BadServerHalted`) or `docker kill` delivered a
|
||||||
roughly one reconnect interval (≈10 s) instead of ~100 s; an upstream that vanishes without a halt (killed, cable
|
session status to the client: the keep-alives reported `BadSecureChannelClosed` (0x80860000) once, then
|
||||||
pulled) still waits for the watchdog. ⚠️ **Known gap:** the host acts on the signal only while the driver is
|
`BadConnectionClosed` (0x80AE0000) interleaved with `BadNoCommunication` for the whole outage, so the first cut of the
|
||||||
|
fast path never fired and recovery took 91–127 s. Reproduced in-process with `TcpStallProxy.HardClose` (every
|
||||||
|
connection reset, new ones refused, the server AND its session still alive behind it): the same sequence, and the SDK
|
||||||
|
1.5.378 session **never opens a new connection by itself** — 48 s after the proxy accepted connections again, not one
|
||||||
|
attempt. So a closed channel can never recover in place; on the **second** closed-channel keep-alive failure in one
|
||||||
|
outage (`IsClosedChannel`; a good keep-alive resets the count, so a single transient does not count) the driver raises
|
||||||
|
`ConnectionLost` — about one keep-alive interval after the connection dropped. Everything else —
|
||||||
|
`BadNoCommunication`, `BadNotConnected`, `BadRequestTimeout`/`BadTimeout` (what a stalled or paused link produces) —
|
||||||
|
is **ambiguous** (the link may resume and keep its session, see the stall test) and stays with the watchdog.
|
||||||
|
Consequence: a restarted, stopped or killed upstream is re-initialised within roughly one keep-alive plus one
|
||||||
|
reconnect interval instead of ~100 s; a pause (`docker pause`, a congested link) still recovers in place. ⚠️ **Known gap:** the host acts on the signal only while the driver is
|
||||||
`Connected`, and the driver raises it once per session — so a definitive loss raised while the host is not
|
`Connected`, and the driver raises it once per session — so a definitive loss raised while the host is not
|
||||||
Connected (e.g. `BadServerHalted` on the fresh session before the host has processed the successful initialise) is
|
Connected (e.g. `BadServerHalted` on the fresh session before the host has processed the successful initialise) is
|
||||||
swallowed and that session falls back to the watchdog. Re-raising on later keep-alives would close it but was left out
|
swallowed and that session falls back to the watchdog. Re-raising on later keep-alives would close it but was left out
|
||||||
@@ -318,7 +328,10 @@ the reconnect handler, #670). They are not part of `DriverHealth` or the
|
|||||||
used to be live-gate-only: the driver connects through `TcpStallProxy`, an in-test TCP forwarder whose pause holds
|
used to be live-gate-only: the driver connects through `TcpStallProxy`, an in-test TCP forwarder whose pause holds
|
||||||
every byte without closing anything (which only works because of the endpoint rewrite above); it asserts one Bad,
|
every byte without closing anything (which only works because of the endpoint rewrite above); it asserts one Bad,
|
||||||
a frozen read stamp, recovery on the SAME session, the identical value republished Good, a change published right
|
a frozen read stamp, recovery on the SAME session, the identical value republished Good, a change published right
|
||||||
after recovery still arriving, and no source-time regression once the SDK's late backlog flush has arrived.
|
after recovery still arriving, and no source-time regression once the SDK's late backlog flush has arrived. Its
|
||||||
|
second test uses `HardClose` (reset every connection, refuse new ones) — the shape of a real process stop — and
|
||||||
|
asserts the live-gate status sequence, that the SDK never reconnects on its own once the proxy accepts again,
|
||||||
|
`ConnectionLost` exactly once, and the re-initialise bringing the identical value back Good.
|
||||||
`OpcUaClientDriverReconnectWireTests` covers the server restart (incl. the `BadServerHalted` → `ConnectionLost` signal)
|
`OpcUaClientDriverReconnectWireTests` covers the server restart (incl. the `BadServerHalted` → `ConnectionLost` signal)
|
||||||
and the subscription-lost case.
|
and the subscription-lost case.
|
||||||
|
|
||||||
|
|||||||
@@ -298,6 +298,14 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
|
|||||||
/// cleared when the session is released, so a re-initialised driver reports its next loss.</summary>
|
/// cleared when the session is released, so a re-initialised driver reports its next loss.</summary>
|
||||||
private int _connectionLostRaised;
|
private int _connectionLostRaised;
|
||||||
|
|
||||||
|
/// <summary>#676 live gate A1 — closed-channel keep-alive failures (<see cref="IsClosedChannel"/>) in the current
|
||||||
|
/// outage; reset by a good keep-alive and when the session is released. Two make a closed channel "sustained".</summary>
|
||||||
|
private int _closedChannelFailures;
|
||||||
|
|
||||||
|
/// <summary>Closed-channel keep-alive failures in one outage that make the loss definitive (a single one may be a
|
||||||
|
/// transient that the next good keep-alive contradicts).</summary>
|
||||||
|
private const int SustainedClosedChannelFailures = 2;
|
||||||
|
|
||||||
/// <summary>#676 — the trailing reconcile read scheduled by the last in-place recovery. Test seam.</summary>
|
/// <summary>#676 — the trailing reconcile read scheduled by the last in-place recovery. Test seam.</summary>
|
||||||
private Task _reconcileTask = Task.CompletedTask;
|
private Task _reconcileTask = Task.CompletedTask;
|
||||||
|
|
||||||
@@ -1081,6 +1089,7 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
|
|||||||
Interlocked.Exchange(ref _publishingStoppedSinceTicks, 0);
|
Interlocked.Exchange(ref _publishingStoppedSinceTicks, 0);
|
||||||
// #676: re-arm the definitive-loss signal for the next session.
|
// #676: re-arm the definitive-loss signal for the next session.
|
||||||
Interlocked.Exchange(ref _connectionLostRaised, 0);
|
Interlocked.Exchange(ref _connectionLostRaised, 0);
|
||||||
|
Interlocked.Exchange(ref _closedChannelFailures, 0);
|
||||||
// Review of 3b88056f: the released session's post-recovery timers must not outlive it.
|
// Review of 3b88056f: the released session's post-recovery timers must not outlive it.
|
||||||
CancelSessionWork();
|
CancelSessionWork();
|
||||||
}
|
}
|
||||||
@@ -2869,6 +2878,7 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
|
|||||||
|
|
||||||
if (!ServiceResult.IsBad(e.Status))
|
if (!ServiceResult.IsBad(e.Status))
|
||||||
{
|
{
|
||||||
|
Interlocked.Exchange(ref _closedChannelFailures, 0);
|
||||||
TransitionTo(HostState.Running);
|
TransitionTo(HostState.Running);
|
||||||
// M1: the session answers but a subscription is dead — no stamp, no recovery; the watchdog reinit is the fix.
|
// M1: the session answers but a subscription is dead — no stamp, no recovery; the watchdog reinit is the fix.
|
||||||
if (Volatile.Read(ref _subscriptionLost) == 1)
|
if (Volatile.Read(ref _subscriptionLost) == 1)
|
||||||
@@ -2913,7 +2923,9 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
|
|||||||
// #676: the first bad keep-alive is often ambiguous and a definitive one follows, so this is checked on EVERY
|
// #676: the first bad keep-alive is often ambiguous and a definitive one follows, so this is checked on EVERY
|
||||||
// bad keep-alive — but only after the outage below has been published, so the Bad reaches subscribers first.
|
// bad keep-alive — but only after the outage below has been published, so the Bad reaches subscribers first.
|
||||||
var keepAliveCode = e.Status?.StatusCode ?? StatusCodes.Good;
|
var keepAliveCode = e.Status?.StatusCode ?? StatusCodes.Good;
|
||||||
var definitive = IsDefinitiveSessionLoss(keepAliveCode);
|
var definitive = IsDefinitiveSessionLoss(keepAliveCode)
|
||||||
|
|| (IsClosedChannel(keepAliveCode)
|
||||||
|
&& Interlocked.Increment(ref _closedChannelFailures) >= SustainedClosedChannelFailures);
|
||||||
// A subscription loss already published Bad for this connection.
|
// A subscription loss already published Bad for this connection.
|
||||||
if (Interlocked.Exchange(ref _sessionLost, 1) == 0 && Volatile.Read(ref _subscriptionLost) == 0)
|
if (Interlocked.Exchange(ref _sessionLost, 1) == 0 && Volatile.Read(ref _subscriptionLost) == 0)
|
||||||
{
|
{
|
||||||
@@ -2927,6 +2939,23 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
|
|||||||
RaiseConnectionLost($"upstream session keep-alive reported {StatusCodes.GetBrowseName(keepAliveCode.Code)}");
|
RaiseConnectionLost($"upstream session keep-alive reported {StatusCodes.GetBrowseName(keepAliveCode.Code)}");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// #676 live gate A1 — the connection under the session is CLOSED (not merely silent). A real upstream process stop
|
||||||
|
/// (docker restart, SIGINT, docker kill) never produced a session status on the docker-dev gate: the keep-alives
|
||||||
|
/// reported <c>BadSecureChannelClosed</c> once, then <c>BadConnectionClosed</c> interleaved with
|
||||||
|
/// <c>BadNoCommunication</c> for the whole outage. Reproduced in-process (a proxy that resets every connection
|
||||||
|
/// while the server and its session stay alive): the SDK 1.5.378 session never opens a new connection on its
|
||||||
|
/// own — over 48 s with the server reachable again, not one connection attempt — so a closed channel can never
|
||||||
|
/// recover in place. It is treated as definitive once SUSTAINED (<see cref="SustainedClosedChannelFailures"/> in
|
||||||
|
/// one outage), so a single transient contradicted by the next good keep-alive does not count. A stall (a paused
|
||||||
|
/// upstream, a congested link) produces <c>BadRequestTimeout</c>/<c>BadNoCommunication</c> instead and keeps
|
||||||
|
/// its in-place recovery.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="status">A keep-alive status.</param>
|
||||||
|
/// <returns>True for a closed secure channel or connection.</returns>
|
||||||
|
internal static bool IsClosedChannel(StatusCode status) =>
|
||||||
|
status == StatusCodes.BadSecureChannelClosed || status == StatusCodes.BadConnectionClosed;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// #676 — keep-alive statuses that prove the session can NEVER recover in place, so waiting out the #527
|
/// #676 — keep-alive statuses that prove the session can NEVER recover in place, so waiting out the #527
|
||||||
/// watchdog would only delay the inevitable re-initialise. The server forgot the session
|
/// watchdog would only delay the inevitable re-initialise. The server forgot the session
|
||||||
@@ -2934,9 +2963,10 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
|
|||||||
/// (<c>BadSessionClosed</c>) or never activated it (<c>BadSessionNotActivated</c>); or the server HALTED
|
/// (<c>BadSessionClosed</c>) or never activated it (<c>BadSessionNotActivated</c>); or the server HALTED
|
||||||
/// (<c>BadServerHalted</c>) — measured on the #676 restart wire test, the SDK never re-opens a channel the
|
/// (<c>BadServerHalted</c>) — measured on the #676 restart wire test, the SDK never re-opens a channel the
|
||||||
/// server halted: 15 s after the server was back the same session still failed every request with
|
/// server halted: 15 s after the server was back the same session still failed every request with
|
||||||
/// <c>BadServerHalted</c>, so a halted session is dead for good. Everything else — <c>BadNoCommunication</c>,
|
/// <c>BadServerHalted</c>, so a halted session is dead for good. A closed connection is definitive too, but only
|
||||||
/// <c>BadConnectionClosed</c>, <c>BadNotConnected</c>, request timeouts — is AMBIGUOUS (a stalled link that
|
/// once sustained — see <see cref="IsClosedChannel"/>. Everything else — <c>BadNoCommunication</c>,
|
||||||
/// resumes keeps its session, see the stall wire test) and stays with the watchdog.
|
/// <c>BadNotConnected</c>, request timeouts — is AMBIGUOUS (a stalled link that resumes keeps its session, see the
|
||||||
|
/// stall wire test) and stays with the watchdog.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <param name="status">A keep-alive status.</param>
|
/// <param name="status">A keep-alive status.</param>
|
||||||
/// <returns>True when the session is definitively gone.</returns>
|
/// <returns>True when the session is definitively gone.</returns>
|
||||||
|
|||||||
+48
-2
@@ -105,11 +105,9 @@ public sealed class OpcUaClientConnectionLossSignalTests
|
|||||||
public static TheoryData<uint> AmbiguousStatuses =>
|
public static TheoryData<uint> AmbiguousStatuses =>
|
||||||
[
|
[
|
||||||
StatusCodes.BadNoCommunication,
|
StatusCodes.BadNoCommunication,
|
||||||
StatusCodes.BadConnectionClosed,
|
|
||||||
StatusCodes.BadNotConnected,
|
StatusCodes.BadNotConnected,
|
||||||
StatusCodes.BadRequestTimeout,
|
StatusCodes.BadRequestTimeout,
|
||||||
StatusCodes.BadTimeout,
|
StatusCodes.BadTimeout,
|
||||||
StatusCodes.BadSecureChannelClosed,
|
|
||||||
];
|
];
|
||||||
|
|
||||||
[Theory]
|
[Theory]
|
||||||
@@ -153,6 +151,54 @@ public sealed class OpcUaClientConnectionLossSignalTests
|
|||||||
raised.ShouldBe(0, "an ambiguous loss may yet resume in place — it belongs to the #527 watchdog");
|
raised.ShouldBe(0, "an ambiguous loss may yet resume in place — it belongs to the #527 watchdog");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// #676 live gate A1: a real upstream process stop (docker restart / SIGINT / docker kill) never produced a
|
||||||
|
/// session status — the keep-alives carried <c>BadSecureChannelClosed</c> once, then <c>BadConnectionClosed</c>
|
||||||
|
/// (interleaved with <c>BadNoCommunication</c>) for the whole outage. Measured in-process (hard-closing proxy, the
|
||||||
|
/// server and its session still alive): the SDK 1.5.378 session NEVER opens a new connection on its own, so a
|
||||||
|
/// closed channel is dead for good. Sustained — a second closed-channel failure in the same outage — it raises.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void A_sustained_closed_channel_raises_ConnectionLost_once()
|
||||||
|
{
|
||||||
|
var (drv, _, _, _) = NewConnectedDriver();
|
||||||
|
using var _ = drv;
|
||||||
|
var raised = new List<string>();
|
||||||
|
drv.ConnectionLost += (_, e) => raised.Add(e.Reason);
|
||||||
|
|
||||||
|
drv.RaiseKeepAliveForTest(StatusCodes.BadSecureChannelClosed); // the live sequence …
|
||||||
|
raised.ShouldBeEmpty("one closed-channel failure may be a transient");
|
||||||
|
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||||
|
drv.RaiseKeepAliveForTest(StatusCodes.BadConnectionClosed); // … sustained
|
||||||
|
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||||
|
drv.RaiseKeepAliveForTest(StatusCodes.BadConnectionClosed);
|
||||||
|
|
||||||
|
raised.ShouldHaveSingleItem().ShouldContain("BadConnectionClosed");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Theory]
|
||||||
|
[InlineData(0x80860000u)] // BadSecureChannelClosed
|
||||||
|
[InlineData(0x80AE0000u)] // BadConnectionClosed
|
||||||
|
public void A_single_closed_channel_blip_that_recovers_does_not_raise(uint closed)
|
||||||
|
{
|
||||||
|
var (drv, _, _, _) = NewConnectedDriver();
|
||||||
|
using var _ = drv;
|
||||||
|
var raised = 0;
|
||||||
|
drv.ConnectionLost += (_, _) => raised++;
|
||||||
|
|
||||||
|
drv.RaiseKeepAliveForTest(closed);
|
||||||
|
drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication);
|
||||||
|
drv.RaiseKeepAliveForTest(StatusCodes.Good); // recovered — the count starts again
|
||||||
|
drv.RaiseKeepAliveForTest(closed);
|
||||||
|
drv.RaiseKeepAliveForTest(StatusCodes.Good);
|
||||||
|
|
||||||
|
raised.ShouldBe(0);
|
||||||
|
// Positive control: the same driver DOES raise once a closed channel is sustained.
|
||||||
|
drv.RaiseKeepAliveForTest(closed);
|
||||||
|
drv.RaiseKeepAliveForTest(closed);
|
||||||
|
raised.ShouldBe(1);
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void An_ambiguous_outage_that_turns_definitive_raises_once()
|
public void An_ambiguous_outage_that_turns_definitive_raises_once()
|
||||||
{
|
{
|
||||||
|
|||||||
+72
@@ -156,4 +156,76 @@ public sealed class OpcUaClientDriverStallWireTests
|
|||||||
Volatile.Read(ref connectionLost).ShouldBe(0,
|
Volatile.Read(ref connectionLost).ShouldBe(0,
|
||||||
"a stall is AMBIGUOUS (the session survived it) — it must go through the #527 watchdog, never the fast path");
|
"a stall is AMBIGUOUS (the session survived it) — it must go through the #527 watchdog, never the fast path");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// #676 live gate A1 — a real upstream process stop is a CLOSED connection, not a stall: the keep-alives report
|
||||||
|
/// <c>BadSecureChannelClosed</c>, then <c>BadConnectionClosed</c> (with <c>BadNoCommunication</c>) for the rest of
|
||||||
|
/// the outage, and the SDK never opens a new connection on its own — not even when the server (here, with the
|
||||||
|
/// session still alive behind the proxy) is reachable again. The driver must therefore raise
|
||||||
|
/// <c>ConnectionLost</c> once the closed channel is sustained, so the host re-initialises in seconds instead of
|
||||||
|
/// waiting ~90 s for the watchdog; the re-initialise brings the identical value back Good.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task A_hard_closed_connection_raises_ConnectionLost_once_and_the_reinit_recovers_in_seconds()
|
||||||
|
{
|
||||||
|
var ct = TestContext.Current.CancellationToken;
|
||||||
|
await using var rig = await OpcUaClientDriverWireTests.StartRigAsync("drv-hardclose", ct);
|
||||||
|
await using var proxy = TcpStallProxy.Start(new Uri(rig.EndpointUrl).Port);
|
||||||
|
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 10.0f);
|
||||||
|
|
||||||
|
await using var drv = OpcUaClientDriverWireTests.NewDriver(
|
||||||
|
rig, "wire-hardclose", endpointUrl: proxy.Through(rig.EndpointUrl), sessionTimeout: TimeSpan.FromSeconds(120));
|
||||||
|
var lost = new List<string>();
|
||||||
|
drv.ConnectionLost += (_, e) => { lock (lost) lost.Add(e.Reason); };
|
||||||
|
await drv.InitializeAsync("{}", ct);
|
||||||
|
var seen = new List<Seen>();
|
||||||
|
drv.OnDataChange += (_, e) =>
|
||||||
|
{
|
||||||
|
if (e.FullReference == OpcUaClientDriverWireTests.SpeedRef)
|
||||||
|
lock (seen) seen.Add(new Seen(e.Snapshot.StatusCode, e.Snapshot.Value, e.Snapshot.SourceTimestampUtc));
|
||||||
|
};
|
||||||
|
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
|
||||||
|
await WaitAsync(() => { lock (seen) return seen.Any(s => s is { Status: 0u, Value: 10.0f }); }, "initial value");
|
||||||
|
var session = drv.Session.ShouldNotBeNull();
|
||||||
|
var keepAlives = new List<uint>();
|
||||||
|
session.KeepAlive += (_, e) => { lock (keepAlives) keepAlives.Add(e.Status?.StatusCode.Code ?? 0); };
|
||||||
|
|
||||||
|
// ── The upstream process goes away: every connection reset, new ones refused ──────────────────────────────
|
||||||
|
var killedAt = DateTime.UtcNow;
|
||||||
|
proxy.HardClose();
|
||||||
|
await WaitAsync(() => { lock (lost) return lost.Count > 0; },
|
||||||
|
"a sustained closed channel must raise ConnectionLost", TimeSpan.FromSeconds(15));
|
||||||
|
(DateTime.UtcNow - killedAt).ShouldBeLessThan(TimeSpan.FromSeconds(15), "seconds, not the watchdog's ~90 s");
|
||||||
|
lock (seen) seen.Count(s => s.Status == BadNoCommunication).ShouldBe(1, "the outage was published Bad first");
|
||||||
|
lock (keepAlives)
|
||||||
|
{
|
||||||
|
keepAlives.ShouldContain(0x80860000u, "BadSecureChannelClosed — the live-gate sequence");
|
||||||
|
keepAlives.ShouldContain(0x80AE0000u, "BadConnectionClosed — the live-gate sequence");
|
||||||
|
}
|
||||||
|
|
||||||
|
// The server is back — the session still exists on it — yet the SDK never reconnects by itself: no new
|
||||||
|
// connection reaches the proxy while further keep-alives fail. That is what makes the closed channel definitive.
|
||||||
|
proxy.Reopen();
|
||||||
|
var accepted = proxy.AcceptedConnections;
|
||||||
|
int kaAtReopen;
|
||||||
|
lock (keepAlives) kaAtReopen = keepAlives.Count;
|
||||||
|
await WaitAsync(() => { lock (keepAlives) return keepAlives.Count >= kaAtReopen + 4; }, "keep-alives keep failing");
|
||||||
|
proxy.AcceptedConnections.ShouldBe(accepted, "the SDK session never reopens a closed connection on its own");
|
||||||
|
drv.GetHealth().State.ShouldNotBe(DriverState.Healthy);
|
||||||
|
lock (lost) lost.Count.ShouldBe(1, "once per session, however many closed-channel keep-alives follow");
|
||||||
|
|
||||||
|
// ── The host's response (DriverInstanceActor: Reconnecting → ShutdownAsync + InitializeAsync → re-subscribe) ──
|
||||||
|
await drv.ShutdownAsync(ct);
|
||||||
|
await drv.InitializeAsync("{}", ct);
|
||||||
|
int before;
|
||||||
|
lock (seen) before = seen.Count;
|
||||||
|
await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct);
|
||||||
|
await WaitAsync(() => { lock (seen) return seen.Skip(before).Any(s => s is { Status: 0u, Value: 10.0f }); },
|
||||||
|
"the identical value must come back Good after the re-initialise");
|
||||||
|
(DateTime.UtcNow - killedAt).ShouldBeLessThan(TimeSpan.FromSeconds(60), "well under the watchdog's 90 s");
|
||||||
|
OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 77.0f);
|
||||||
|
await WaitAsync(() => { lock (seen) return seen.Any(s => s is { Status: 0u, Value: 77.0f }); }, "a later change arrives");
|
||||||
|
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||||
|
lock (lost) lost.Count.ShouldBe(1, "the healthy re-initialised session reports no loss");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ internal sealed class TcpStallProxy : IAsyncDisposable
|
|||||||
private readonly List<TcpClient> _sockets = [];
|
private readonly List<TcpClient> _sockets = [];
|
||||||
private readonly Lock _lock = new();
|
private readonly Lock _lock = new();
|
||||||
private TaskCompletionSource _gate = Open();
|
private TaskCompletionSource _gate = Open();
|
||||||
|
private volatile bool _refusing;
|
||||||
private Task _acceptLoop = Task.CompletedTask;
|
private Task _acceptLoop = Task.CompletedTask;
|
||||||
|
|
||||||
private TcpStallProxy(int targetPort) => _targetPort = targetPort;
|
private TcpStallProxy(int targetPort) => _targetPort = targetPort;
|
||||||
@@ -48,6 +49,34 @@ internal sealed class TcpStallProxy : IAsyncDisposable
|
|||||||
if (_gate.Task.IsCompleted) _gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
if (_gate.Task.IsCompleted) _gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The shape of a killed / stopped server process (#676 live gate): every open connection is RESET (linger 0 ⇒
|
||||||
|
/// RST, both sides) and new connections are refused (accepted and reset at once) until <see cref="Reopen"/>.
|
||||||
|
/// </summary>
|
||||||
|
public void HardClose()
|
||||||
|
{
|
||||||
|
List<TcpClient> victims;
|
||||||
|
lock (_lock)
|
||||||
|
{
|
||||||
|
_refusing = true;
|
||||||
|
victims = [.. _sockets];
|
||||||
|
_sockets.Clear();
|
||||||
|
}
|
||||||
|
foreach (var s in victims) Reset(s);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Accept and forward new connections again (the server is back); the reset ones stay dead.</summary>
|
||||||
|
public void Reopen() => _refusing = false;
|
||||||
|
|
||||||
|
/// <summary>Connections refused while <see cref="HardClose"/> was in force — a reconnect ATTEMPT by the client.</summary>
|
||||||
|
public int RefusedConnections { get; private set; }
|
||||||
|
|
||||||
|
private static void Reset(TcpClient s)
|
||||||
|
{
|
||||||
|
try { s.Client.LingerState = new LingerOption(true, 0); } catch { }
|
||||||
|
try { s.Dispose(); } catch { }
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Releases everything held since <see cref="Pause"/>, in order.</summary>
|
/// <summary>Releases everything held since <see cref="Pause"/>, in order.</summary>
|
||||||
public void Resume()
|
public void Resume()
|
||||||
{
|
{
|
||||||
@@ -73,6 +102,12 @@ internal sealed class TcpStallProxy : IAsyncDisposable
|
|||||||
TcpClient client;
|
TcpClient client;
|
||||||
try { client = await _listener.AcceptTcpClientAsync(_cts.Token); }
|
try { client = await _listener.AcceptTcpClientAsync(_cts.Token); }
|
||||||
catch { return; }
|
catch { return; }
|
||||||
|
if (_refusing)
|
||||||
|
{
|
||||||
|
lock (_lock) RefusedConnections++;
|
||||||
|
Reset(client);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
var upstream = new TcpClient();
|
var upstream = new TcpClient();
|
||||||
try { await upstream.ConnectAsync(IPAddress.Loopback, _targetPort, _cts.Token); }
|
try { await upstream.ConnectAsync(IPAddress.Loopback, _targetPort, _cts.Token); }
|
||||||
catch
|
catch
|
||||||
|
|||||||
Reference in New Issue
Block a user