fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)
QA Warm gate (lmxopcua#907, 2026-11-13): a graceful stop of the Warm Primary reported ServiceLevel 0 for ~6 s - shorter than the 10 s degraded window - the next unreadable sample cleared the streak, and the connection retried the stopped node FailoverRetryCount times: Good on the peer after 66 s, peer Active in 0.2 s. Two going-away triggers on top of #138, both still requiring a Healthy (>= 200), higher level on the other endpoint and honouring the minimum dwell: - ServiceLevel 0 (Part 4 Maintenance) probes the other endpoint at once instead of waiting out the window - once per degraded streak, so a non-Healthy peer is re-probed once per window, not every poll. Other Degraded levels (standby 90/50, booting 1) still wait the window. - A lost link whose session's last KNOWN ServiceLevel was < 200 (unreadable samples don't overwrite it) probes the other endpoint before the first reconnect attempt, holding the reconnect timer. Healthy -> switch and reconnect at zero delay, the stopped node is never retried; otherwise the ordinary retry cycle runs (less the probe's time). After only Healthy levels (250/240, Unset), inside the dwell, or with the feature off: unchanged retry-then-failover. Tests: DataConnectionGoingAwayFailoverTests (10 cases, each trigger and non-trigger with positive controls, connect-order artifact assertions) and a real in-process OPC UA server pair graceful-stop e2e on production defaults: 1.1 s (lost-link trigger alone 12.1 s, neither 33.1 s -> fails the 10 s bound). 13 mutants, all killed. DCL tests 608/608 (3 full runs); solution 0 warnings. Docs: Component-DataConnectionLayer, components/DataConnectionLayer. Not deployed; QA live re-check pending. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
+20
@@ -42,6 +42,23 @@ internal sealed class ServiceLevelRig
|
||||
|
||||
public int ConnectCount(string endpoint) => Connects.Count(e => e == endpoint);
|
||||
|
||||
/// <summary>Every adapter that opened a session, so a test can kill the live one (ScadaBridge#141).</summary>
|
||||
public ConcurrentQueue<FakeServiceLevelConnection> Sessions { get; } = new();
|
||||
|
||||
/// <summary>
|
||||
/// The server at <paramref name="endpoint"/> goes away: it refuses new sessions and every live
|
||||
/// session on it raises <c>Disconnected</c> — what a stopped OtOpcUa node looks like to the client.
|
||||
/// </summary>
|
||||
public void Stop(string endpoint)
|
||||
{
|
||||
SetReachable(endpoint, false);
|
||||
foreach (var session in Sessions.Where(s => s.Endpoint == endpoint && s.IsConnected))
|
||||
session.RaiseDisconnected();
|
||||
}
|
||||
|
||||
/// <summary>The endpoints sessions were opened on after the first <paramref name="skip"/> connects.</summary>
|
||||
public string[] ConnectsAfter(int skip) => Connects.Skip(skip).ToArray();
|
||||
|
||||
public int ReadCount(string endpoint) => Reads.GetValueOrDefault(endpoint);
|
||||
}
|
||||
|
||||
@@ -72,9 +89,12 @@ internal sealed class FakeServiceLevelConnection : IDataConnection, IServiceLeve
|
||||
if (!_rig.IsReachable(Endpoint))
|
||||
return Task.FromException(new InvalidOperationException($"{Endpoint} refused the session"));
|
||||
_connected = true;
|
||||
_rig.Sessions.Enqueue(this);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public bool IsConnected => _connected;
|
||||
|
||||
public Task DisconnectAsync(CancellationToken cancellationToken = default)
|
||||
{
|
||||
_connected = false;
|
||||
|
||||
Reference in New Issue
Block a user