diff --git a/CLAUDE.md b/CLAUDE.md index 8a11da70..3c93a96b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -323,8 +323,9 @@ non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. T reading. - **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises `ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient: - keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted` or a subscription the - SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only + keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted`, a SUSTAINED closed + connection (2nd `BadSecureChannelClosed`/`BadConnectionClosed` in one outage — what a real process stop produces; the + SDK never reopens it) or a subscription the SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with the watchdog's conjunction. `docs/DriverLifecycle.md`. diff --git a/docs/DriverLifecycle.md b/docs/DriverLifecycle.md index ae56473f..ef029efb 100644 --- a/docs/DriverLifecycle.md +++ b/docs/DriverLifecycle.md @@ -458,11 +458,15 @@ alternatives: communication" — must not raise: a stalled link that resumes keeps its session, subscriptions and in-place recovery, and tearing it down would cost all three. Those stay with the watchdog, whose conjunction is the honest test for them. - **Today only `OpcUaClientDriver` implements it:** keep-alive `BadSessionIdInvalid` / `BadSessionClosed` / - `BadSessionNotActivated` / `BadServerHalted`, or a subscription the server REPORTED dropped (not the driver's + `BadSessionNotActivated` / `BadServerHalted`; a CLOSED connection (`BadSecureChannelClosed` / + `BadConnectionClosed`) once sustained — the second closed-channel keep-alive in one outage, which is what a real + process stop actually produces (live gate A1), and the SDK never reopens a closed connection by itself; or a + subscription the server REPORTED dropped (not the driver's publishing-stall heuristic, which is an inference from silence and stays with the watchdog; see [OpcUaClient.md](drivers/OpcUaClient.md) §"Upstream outage and recovery" for why `BadServerHalted` is on the list — - it was measured, the SDK never re-opens a halted channel). A gracefully restarted upstream therefore recovers in - about one reconnect interval instead of ~100 s; a killed one still waits for the watchdog. + it was measured, the SDK never re-opens a halted channel). A restarted, stopped or killed upstream therefore + recovers in about one keep-alive plus one reconnect interval instead of ~100 s; a stall/pause + (`BadRequestTimeout`/`BadNoCommunication`) still goes through the watchdog and keeps its in-place recovery. - ⚠️ **Known gap:** "only while `Connected`" plus "once per connection" means a raise the host receives while not yet Connected (e.g. the protocol proves the fresh connection dead before `InitializeSucceeded` is processed) is swallowed, and that connection falls back to the watchdog — the pre-#676 recovery time, in a narrow window. diff --git a/docs/drivers/OpcUaClient.md b/docs/drivers/OpcUaClient.md index 6c1f6da7..13cd6db8 100644 --- a/docs/drivers/OpcUaClient.md +++ b/docs/drivers/OpcUaClient.md @@ -137,11 +137,21 @@ The session keep-alive (`KeepAliveInterval`, default 5 s) is the liveness signal because it was **measured**: on the restart wire test a gracefully stopped server reports `BadServerHalted` (and the keep-alives alternate it with `BadNoCommunication`), and the SDK never re-opens a channel the server halted — 15 s after the server was back, the same session still failed every request with `BadServerHalted`; `BadSessionIdInvalid` - never arrived. Everything else — `BadNoCommunication`, `BadConnectionClosed`, `BadNotConnected`, - `BadRequestTimeout`/`BadTimeout` (what a stalled link produces) — is **ambiguous** (the link may resume and keep its - session, see the stall test) and stays with the watchdog. Consequence: a gracefully restarted upstream recovers in - roughly one reconnect interval (≈10 s) instead of ~100 s; an upstream that vanishes without a halt (killed, cable - pulled) still waits for the watchdog. ⚠️ **Known gap:** the host acts on the signal only while the driver is + never arrived. + **A CLOSED connection is definitive too, once sustained (live gate A1).** Against a real opc-plc on docker-dev, none of + `docker restart`, a graceful SIGINT stop (opc-plc itself logged `BadServerHalted`) or `docker kill` delivered a + session status to the client: the keep-alives reported `BadSecureChannelClosed` (0x80860000) once, then + `BadConnectionClosed` (0x80AE0000) interleaved with `BadNoCommunication` for the whole outage, so the first cut of the + fast path never fired and recovery took 91–127 s. Reproduced in-process with `TcpStallProxy.HardClose` (every + connection reset, new ones refused, the server AND its session still alive behind it): the same sequence, and the SDK + 1.5.378 session **never opens a new connection by itself** — 48 s after the proxy accepted connections again, not one + attempt. So a closed channel can never recover in place; on the **second** closed-channel keep-alive failure in one + outage (`IsClosedChannel`; a good keep-alive resets the count, so a single transient does not count) the driver raises + `ConnectionLost` — about one keep-alive interval after the connection dropped. Everything else — + `BadNoCommunication`, `BadNotConnected`, `BadRequestTimeout`/`BadTimeout` (what a stalled or paused link produces) — + is **ambiguous** (the link may resume and keep its session, see the stall test) and stays with the watchdog. + Consequence: a restarted, stopped or killed upstream is re-initialised within roughly one keep-alive plus one + reconnect interval instead of ~100 s; a pause (`docker pause`, a congested link) still recovers in place. ⚠️ **Known gap:** the host acts on the signal only while the driver is `Connected`, and the driver raises it once per session — so a definitive loss raised while the host is not Connected (e.g. `BadServerHalted` on the fresh session before the host has processed the successful initialise) is swallowed and that session falls back to the watchdog. Re-raising on later keep-alives would close it but was left out @@ -318,7 +328,10 @@ the reconnect handler, #670). They are not part of `DriverHealth` or the used to be live-gate-only: the driver connects through `TcpStallProxy`, an in-test TCP forwarder whose pause holds every byte without closing anything (which only works because of the endpoint rewrite above); it asserts one Bad, a frozen read stamp, recovery on the SAME session, the identical value republished Good, a change published right - after recovery still arriving, and no source-time regression once the SDK's late backlog flush has arrived. + after recovery still arriving, and no source-time regression once the SDK's late backlog flush has arrived. Its + second test uses `HardClose` (reset every connection, refuse new ones) — the shape of a real process stop — and + asserts the live-gate status sequence, that the SDK never reconnects on its own once the proxy accepts again, + `ConnectionLost` exactly once, and the re-initialise bringing the identical value back Good. `OpcUaClientDriverReconnectWireTests` covers the server restart (incl. the `BadServerHalted` → `ConnectionLost` signal) and the subscription-lost case. diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs index a5d82caa..b25ed5a9 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs @@ -298,6 +298,14 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit /// cleared when the session is released, so a re-initialised driver reports its next loss. private int _connectionLostRaised; + /// #676 live gate A1 — closed-channel keep-alive failures () in the current + /// outage; reset by a good keep-alive and when the session is released. Two make a closed channel "sustained". + private int _closedChannelFailures; + + /// Closed-channel keep-alive failures in one outage that make the loss definitive (a single one may be a + /// transient that the next good keep-alive contradicts). + private const int SustainedClosedChannelFailures = 2; + /// #676 — the trailing reconcile read scheduled by the last in-place recovery. Test seam. private Task _reconcileTask = Task.CompletedTask; @@ -1081,6 +1089,7 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit Interlocked.Exchange(ref _publishingStoppedSinceTicks, 0); // #676: re-arm the definitive-loss signal for the next session. Interlocked.Exchange(ref _connectionLostRaised, 0); + Interlocked.Exchange(ref _closedChannelFailures, 0); // Review of 3b88056f: the released session's post-recovery timers must not outlive it. CancelSessionWork(); } @@ -2869,6 +2878,7 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit if (!ServiceResult.IsBad(e.Status)) { + Interlocked.Exchange(ref _closedChannelFailures, 0); TransitionTo(HostState.Running); // M1: the session answers but a subscription is dead — no stamp, no recovery; the watchdog reinit is the fix. if (Volatile.Read(ref _subscriptionLost) == 1) @@ -2913,7 +2923,9 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit // #676: the first bad keep-alive is often ambiguous and a definitive one follows, so this is checked on EVERY // bad keep-alive — but only after the outage below has been published, so the Bad reaches subscribers first. var keepAliveCode = e.Status?.StatusCode ?? StatusCodes.Good; - var definitive = IsDefinitiveSessionLoss(keepAliveCode); + var definitive = IsDefinitiveSessionLoss(keepAliveCode) + || (IsClosedChannel(keepAliveCode) + && Interlocked.Increment(ref _closedChannelFailures) >= SustainedClosedChannelFailures); // A subscription loss already published Bad for this connection. if (Interlocked.Exchange(ref _sessionLost, 1) == 0 && Volatile.Read(ref _subscriptionLost) == 0) { @@ -2927,6 +2939,23 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit RaiseConnectionLost($"upstream session keep-alive reported {StatusCodes.GetBrowseName(keepAliveCode.Code)}"); } + /// + /// #676 live gate A1 — the connection under the session is CLOSED (not merely silent). A real upstream process stop + /// (docker restart, SIGINT, docker kill) never produced a session status on the docker-dev gate: the keep-alives + /// reported BadSecureChannelClosed once, then BadConnectionClosed interleaved with + /// BadNoCommunication for the whole outage. Reproduced in-process (a proxy that resets every connection + /// while the server and its session stay alive): the SDK 1.5.378 session never opens a new connection on its + /// own — over 48 s with the server reachable again, not one connection attempt — so a closed channel can never + /// recover in place. It is treated as definitive once SUSTAINED ( in + /// one outage), so a single transient contradicted by the next good keep-alive does not count. A stall (a paused + /// upstream, a congested link) produces BadRequestTimeout/BadNoCommunication instead and keeps + /// its in-place recovery. + /// + /// A keep-alive status. + /// True for a closed secure channel or connection. + internal static bool IsClosedChannel(StatusCode status) => + status == StatusCodes.BadSecureChannelClosed || status == StatusCodes.BadConnectionClosed; + /// /// #676 — keep-alive statuses that prove the session can NEVER recover in place, so waiting out the #527 /// watchdog would only delay the inevitable re-initialise. The server forgot the session @@ -2934,9 +2963,10 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit /// (BadSessionClosed) or never activated it (BadSessionNotActivated); or the server HALTED /// (BadServerHalted) — measured on the #676 restart wire test, the SDK never re-opens a channel the /// server halted: 15 s after the server was back the same session still failed every request with - /// BadServerHalted, so a halted session is dead for good. Everything else — BadNoCommunication, - /// BadConnectionClosed, BadNotConnected, request timeouts — is AMBIGUOUS (a stalled link that - /// resumes keeps its session, see the stall wire test) and stays with the watchdog. + /// BadServerHalted, so a halted session is dead for good. A closed connection is definitive too, but only + /// once sustained — see . Everything else — BadNoCommunication, + /// BadNotConnected, request timeouts — is AMBIGUOUS (a stalled link that resumes keeps its session, see the + /// stall wire test) and stays with the watchdog. /// /// A keep-alive status. /// True when the session is definitively gone. diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/OpcUaClientConnectionLossSignalTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/OpcUaClientConnectionLossSignalTests.cs index d2db998f..ae0ffe91 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/OpcUaClientConnectionLossSignalTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/OpcUaClientConnectionLossSignalTests.cs @@ -105,11 +105,9 @@ public sealed class OpcUaClientConnectionLossSignalTests public static TheoryData AmbiguousStatuses => [ StatusCodes.BadNoCommunication, - StatusCodes.BadConnectionClosed, StatusCodes.BadNotConnected, StatusCodes.BadRequestTimeout, StatusCodes.BadTimeout, - StatusCodes.BadSecureChannelClosed, ]; [Theory] @@ -153,6 +151,54 @@ public sealed class OpcUaClientConnectionLossSignalTests raised.ShouldBe(0, "an ambiguous loss may yet resume in place — it belongs to the #527 watchdog"); } + /// + /// #676 live gate A1: a real upstream process stop (docker restart / SIGINT / docker kill) never produced a + /// session status — the keep-alives carried BadSecureChannelClosed once, then BadConnectionClosed + /// (interleaved with BadNoCommunication) for the whole outage. Measured in-process (hard-closing proxy, the + /// server and its session still alive): the SDK 1.5.378 session NEVER opens a new connection on its own, so a + /// closed channel is dead for good. Sustained — a second closed-channel failure in the same outage — it raises. + /// + [Fact] + public void A_sustained_closed_channel_raises_ConnectionLost_once() + { + var (drv, _, _, _) = NewConnectedDriver(); + using var _ = drv; + var raised = new List(); + drv.ConnectionLost += (_, e) => raised.Add(e.Reason); + + drv.RaiseKeepAliveForTest(StatusCodes.BadSecureChannelClosed); // the live sequence … + raised.ShouldBeEmpty("one closed-channel failure may be a transient"); + drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication); + drv.RaiseKeepAliveForTest(StatusCodes.BadConnectionClosed); // … sustained + drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication); + drv.RaiseKeepAliveForTest(StatusCodes.BadConnectionClosed); + + raised.ShouldHaveSingleItem().ShouldContain("BadConnectionClosed"); + } + + [Theory] + [InlineData(0x80860000u)] // BadSecureChannelClosed + [InlineData(0x80AE0000u)] // BadConnectionClosed + public void A_single_closed_channel_blip_that_recovers_does_not_raise(uint closed) + { + var (drv, _, _, _) = NewConnectedDriver(); + using var _ = drv; + var raised = 0; + drv.ConnectionLost += (_, _) => raised++; + + drv.RaiseKeepAliveForTest(closed); + drv.RaiseKeepAliveForTest(StatusCodes.BadNoCommunication); + drv.RaiseKeepAliveForTest(StatusCodes.Good); // recovered — the count starts again + drv.RaiseKeepAliveForTest(closed); + drv.RaiseKeepAliveForTest(StatusCodes.Good); + + raised.ShouldBe(0); + // Positive control: the same driver DOES raise once a closed channel is sustained. + drv.RaiseKeepAliveForTest(closed); + drv.RaiseKeepAliveForTest(closed); + raised.ShouldBe(1); + } + [Fact] public void An_ambiguous_outage_that_turns_definitive_raises_once() { diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/OpcUaClientDriverStallWireTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/OpcUaClientDriverStallWireTests.cs index 82504d64..b844adc4 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/OpcUaClientDriverStallWireTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/OpcUaClientDriverStallWireTests.cs @@ -156,4 +156,76 @@ public sealed class OpcUaClientDriverStallWireTests Volatile.Read(ref connectionLost).ShouldBe(0, "a stall is AMBIGUOUS (the session survived it) — it must go through the #527 watchdog, never the fast path"); } + + /// + /// #676 live gate A1 — a real upstream process stop is a CLOSED connection, not a stall: the keep-alives report + /// BadSecureChannelClosed, then BadConnectionClosed (with BadNoCommunication) for the rest of + /// the outage, and the SDK never opens a new connection on its own — not even when the server (here, with the + /// session still alive behind the proxy) is reachable again. The driver must therefore raise + /// ConnectionLost once the closed channel is sustained, so the host re-initialises in seconds instead of + /// waiting ~90 s for the watchdog; the re-initialise brings the identical value back Good. + /// + [Fact] + public async Task A_hard_closed_connection_raises_ConnectionLost_once_and_the_reinit_recovers_in_seconds() + { + var ct = TestContext.Current.CancellationToken; + await using var rig = await OpcUaClientDriverWireTests.StartRigAsync("drv-hardclose", ct); + await using var proxy = TcpStallProxy.Start(new Uri(rig.EndpointUrl).Port); + OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 10.0f); + + await using var drv = OpcUaClientDriverWireTests.NewDriver( + rig, "wire-hardclose", endpointUrl: proxy.Through(rig.EndpointUrl), sessionTimeout: TimeSpan.FromSeconds(120)); + var lost = new List(); + drv.ConnectionLost += (_, e) => { lock (lost) lost.Add(e.Reason); }; + await drv.InitializeAsync("{}", ct); + var seen = new List(); + drv.OnDataChange += (_, e) => + { + if (e.FullReference == OpcUaClientDriverWireTests.SpeedRef) + lock (seen) seen.Add(new Seen(e.Snapshot.StatusCode, e.Snapshot.Value, e.Snapshot.SourceTimestampUtc)); + }; + await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct); + await WaitAsync(() => { lock (seen) return seen.Any(s => s is { Status: 0u, Value: 10.0f }); }, "initial value"); + var session = drv.Session.ShouldNotBeNull(); + var keepAlives = new List(); + session.KeepAlive += (_, e) => { lock (keepAlives) keepAlives.Add(e.Status?.StatusCode.Code ?? 0); }; + + // ── The upstream process goes away: every connection reset, new ones refused ────────────────────────────── + var killedAt = DateTime.UtcNow; + proxy.HardClose(); + await WaitAsync(() => { lock (lost) return lost.Count > 0; }, + "a sustained closed channel must raise ConnectionLost", TimeSpan.FromSeconds(15)); + (DateTime.UtcNow - killedAt).ShouldBeLessThan(TimeSpan.FromSeconds(15), "seconds, not the watchdog's ~90 s"); + lock (seen) seen.Count(s => s.Status == BadNoCommunication).ShouldBe(1, "the outage was published Bad first"); + lock (keepAlives) + { + keepAlives.ShouldContain(0x80860000u, "BadSecureChannelClosed — the live-gate sequence"); + keepAlives.ShouldContain(0x80AE0000u, "BadConnectionClosed — the live-gate sequence"); + } + + // The server is back — the session still exists on it — yet the SDK never reconnects by itself: no new + // connection reaches the proxy while further keep-alives fail. That is what makes the closed channel definitive. + proxy.Reopen(); + var accepted = proxy.AcceptedConnections; + int kaAtReopen; + lock (keepAlives) kaAtReopen = keepAlives.Count; + await WaitAsync(() => { lock (keepAlives) return keepAlives.Count >= kaAtReopen + 4; }, "keep-alives keep failing"); + proxy.AcceptedConnections.ShouldBe(accepted, "the SDK session never reopens a closed connection on its own"); + drv.GetHealth().State.ShouldNotBe(DriverState.Healthy); + lock (lost) lost.Count.ShouldBe(1, "once per session, however many closed-channel keep-alives follow"); + + // ── The host's response (DriverInstanceActor: Reconnecting → ShutdownAsync + InitializeAsync → re-subscribe) ── + await drv.ShutdownAsync(ct); + await drv.InitializeAsync("{}", ct); + int before; + lock (seen) before = seen.Count; + await drv.SubscribeAsync([OpcUaClientDriverWireTests.SpeedRef], TimeSpan.FromMilliseconds(100), ct); + await WaitAsync(() => { lock (seen) return seen.Skip(before).Any(s => s is { Status: 0u, Value: 10.0f }); }, + "the identical value must come back Good after the re-initialise"); + (DateTime.UtcNow - killedAt).ShouldBeLessThan(TimeSpan.FromSeconds(60), "well under the watchdog's 90 s"); + OpcUaClientDriverWireTests.Publish(rig, OpcUaClientDriverWireTests.SpeedId, 77.0f); + await WaitAsync(() => { lock (seen) return seen.Any(s => s is { Status: 0u, Value: 77.0f }); }, "a later change arrives"); + drv.GetHealth().State.ShouldBe(DriverState.Healthy); + lock (lost) lost.Count.ShouldBe(1, "the healthy re-initialised session reports no loss"); + } } diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/TcpStallProxy.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/TcpStallProxy.cs index a808c0cf..67af3567 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/TcpStallProxy.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/TcpStallProxy.cs @@ -19,6 +19,7 @@ internal sealed class TcpStallProxy : IAsyncDisposable private readonly List _sockets = []; private readonly Lock _lock = new(); private TaskCompletionSource _gate = Open(); + private volatile bool _refusing; private Task _acceptLoop = Task.CompletedTask; private TcpStallProxy(int targetPort) => _targetPort = targetPort; @@ -48,6 +49,34 @@ internal sealed class TcpStallProxy : IAsyncDisposable if (_gate.Task.IsCompleted) _gate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); } + /// + /// The shape of a killed / stopped server process (#676 live gate): every open connection is RESET (linger 0 ⇒ + /// RST, both sides) and new connections are refused (accepted and reset at once) until . + /// + public void HardClose() + { + List victims; + lock (_lock) + { + _refusing = true; + victims = [.. _sockets]; + _sockets.Clear(); + } + foreach (var s in victims) Reset(s); + } + + /// Accept and forward new connections again (the server is back); the reset ones stay dead. + public void Reopen() => _refusing = false; + + /// Connections refused while was in force — a reconnect ATTEMPT by the client. + public int RefusedConnections { get; private set; } + + private static void Reset(TcpClient s) + { + try { s.Client.LingerState = new LingerOption(true, 0); } catch { } + try { s.Dispose(); } catch { } + } + /// Releases everything held since , in order. public void Resume() { @@ -73,6 +102,12 @@ internal sealed class TcpStallProxy : IAsyncDisposable TcpClient client; try { client = await _listener.AcceptTcpClientAsync(_cts.Token); } catch { return; } + if (_refusing) + { + lock (_lock) RefusedConnections++; + Reset(client); + continue; + } var upstream = new TcpClient(); try { await upstream.ConnectAsync(IPAddress.Loopback, _targetPort, _cts.Token); } catch