fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)

QA Warm gate (lmxopcua#907, 2026-11-13): a graceful stop of the Warm Primary
reported ServiceLevel 0 for ~6 s - shorter than the 10 s degraded window - the
next unreadable sample cleared the streak, and the connection retried the stopped
node FailoverRetryCount times: Good on the peer after 66 s, peer Active in 0.2 s.

Two going-away triggers on top of #138, both still requiring a Healthy (>= 200),
higher level on the other endpoint and honouring the minimum dwell:
- ServiceLevel 0 (Part 4 Maintenance) probes the other endpoint at once instead
  of waiting out the window - once per degraded streak, so a non-Healthy peer is
  re-probed once per window, not every poll. Other Degraded levels (standby 90/50,
  booting 1) still wait the window.
- A lost link whose session's last KNOWN ServiceLevel was < 200 (unreadable
  samples don't overwrite it) probes the other endpoint before the first
  reconnect attempt, holding the reconnect timer. Healthy -> switch and reconnect
  at zero delay, the stopped node is never retried; otherwise the ordinary retry
  cycle runs (less the probe's time). After only Healthy levels (250/240, Unset),
  inside the dwell, or with the feature off: unchanged retry-then-failover.

Tests: DataConnectionGoingAwayFailoverTests (10 cases, each trigger and
non-trigger with positive controls, connect-order artifact assertions) and a real
in-process OPC UA server pair graceful-stop e2e on production defaults: 1.1 s
(lost-link trigger alone 12.1 s, neither 33.1 s -> fails the 10 s bound).
13 mutants, all killed. DCL tests 608/608 (3 full runs); solution 0 warnings.
Docs: Component-DataConnectionLayer, components/DataConnectionLayer.

Not deployed; QA live re-check pending.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 09:36:59 -05:00
parent b19836788b
commit c4ec51e89a
7 changed files with 562 additions and 17 deletions
@@ -42,6 +42,23 @@ internal sealed class ServiceLevelRig
public int ConnectCount(string endpoint) => Connects.Count(e => e == endpoint);
/// <summary>Every adapter that opened a session, so a test can kill the live one (ScadaBridge#141).</summary>
public ConcurrentQueue<FakeServiceLevelConnection> Sessions { get; } = new();
/// <summary>
/// The server at <paramref name="endpoint"/> goes away: it refuses new sessions and every live
/// session on it raises <c>Disconnected</c> — what a stopped OtOpcUa node looks like to the client.
/// </summary>
public void Stop(string endpoint)
{
SetReachable(endpoint, false);
foreach (var session in Sessions.Where(s => s.Endpoint == endpoint && s.IsConnected))
session.RaiseDisconnected();
}
/// <summary>The endpoints sessions were opened on after the first <paramref name="skip"/> connects.</summary>
public string[] ConnectsAfter(int skip) => Connects.Skip(skip).ToArray();
public int ReadCount(string endpoint) => Reads.GetValueOrDefault(endpoint);
}
@@ -72,9 +89,12 @@ internal sealed class FakeServiceLevelConnection : IDataConnection, IServiceLeve
if (!_rig.IsReachable(Endpoint))
return Task.FromException(new InvalidOperationException($"{Endpoint} refused the session"));
_connected = true;
_rig.Sessions.Enqueue(this);
return Task.CompletedTask;
}
public bool IsConnected => _connected;
public Task DisconnectAsync(CancellationToken cancellationToken = default)
{
_connected = false;