fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)
QA Warm gate (lmxopcua#907, 2026-11-13): a graceful stop of the Warm Primary reported ServiceLevel 0 for ~6 s - shorter than the 10 s degraded window - the next unreadable sample cleared the streak, and the connection retried the stopped node FailoverRetryCount times: Good on the peer after 66 s, peer Active in 0.2 s. Two going-away triggers on top of #138, both still requiring a Healthy (>= 200), higher level on the other endpoint and honouring the minimum dwell: - ServiceLevel 0 (Part 4 Maintenance) probes the other endpoint at once instead of waiting out the window - once per degraded streak, so a non-Healthy peer is re-probed once per window, not every poll. Other Degraded levels (standby 90/50, booting 1) still wait the window. - A lost link whose session's last KNOWN ServiceLevel was < 200 (unreadable samples don't overwrite it) probes the other endpoint before the first reconnect attempt, holding the reconnect timer. Healthy -> switch and reconnect at zero delay, the stopped node is never retried; otherwise the ordinary retry cycle runs (less the probe's time). After only Healthy levels (250/240, Unset), inside the dwell, or with the feature off: unchanged retry-then-failover. Tests: DataConnectionGoingAwayFailoverTests (10 cases, each trigger and non-trigger with positive controls, connect-order artifact assertions) and a real in-process OPC UA server pair graceful-stop e2e on production defaults: 1.1 s (lost-link trigger alone 12.1 s, neither 33.1 s -> fails the 10 s bound). 13 mutants, all killed. DCL tests 608/608 (3 full runs); solution 0 warnings. Docs: Component-DataConnectionLayer, components/DataConnectionLayer. Not deployed; QA live re-check pending. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
+50
@@ -208,4 +208,54 @@ public sealed class OpcUaServiceLevelTests : TestKit
|
||||
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionEndpoint)
|
||||
&& c.GetArguments() is ["today", "Connected to backup"]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#141 end to end, the QA Warm gate's step 4a: the connection is on the Warm Primary
|
||||
/// (A, 250) with the standby (B, 90) as backup, both on the DEFAULT policy (10 s window, 60 s
|
||||
/// dwell) and the default 5 s reconnect interval with three retries. A is stopped gracefully the
|
||||
/// way OtOpcUa does it — the peer promotes to 250, A reports ServiceLevel 0 for a moment, then
|
||||
/// the server goes away. The connection must be live on B within seconds; the connectivity path
|
||||
/// alone (keep-alive + three 5 s retries of the stopped A) needs well over 15 s.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task RealServerPair_GracefulStopOfTheWarmPrimary_FailsOverWithinSeconds()
|
||||
{
|
||||
var a = await SecureOpcUaTestServer.StartAsync();
|
||||
await using var b = await SecureOpcUaTestServer.StartAsync();
|
||||
try
|
||||
{
|
||||
var factory = new DataConnectionFactory(NullLoggerFactory.Instance, Options.Create(ClientOptions()));
|
||||
var options = new DataConnectionOptions(); // production defaults: 5 s reconnect, 2 s poll, 15 s probe
|
||||
a.SetServiceLevel(250);
|
||||
b.SetServiceLevel(90);
|
||||
var health = Substitute.For<ISiteHealthCollector>();
|
||||
var primary = Flat(a.Url); // null ServiceLevelFailover = defaults (enabled, 10 s, 60 s)
|
||||
|
||||
bool ConnectedTo(string endpoint) => health.ReceivedCalls().Any(c =>
|
||||
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionEndpoint)
|
||||
&& c.GetArguments() is ["stop", var label] && (string)label! == $"Connected to {endpoint}");
|
||||
|
||||
var actor = Sys.ActorOf(Props.Create(() => new DataConnectionActor(
|
||||
"stop", factory.Create("OpcUa", primary), options, health, factory, "OpcUa",
|
||||
primary, Flat(b.Url), 3, null)));
|
||||
AwaitCondition(() => ConnectedTo("primary"), TimeSpan.FromSeconds(30));
|
||||
await Task.Delay(TimeSpan.FromSeconds(3)); // a few Healthy polls: no connect-time effect left
|
||||
|
||||
var sw = System.Diagnostics.Stopwatch.StartNew();
|
||||
b.SetServiceLevel(250); // the peer promotes (0.2 s on QA)
|
||||
a.SetServiceLevel(0); // the stopping node reports Maintenance...
|
||||
await Task.Delay(TimeSpan.FromSeconds(1));
|
||||
await a.DisposeAsync(); // ...and is gone well inside the 10 s window
|
||||
|
||||
AwaitCondition(() => ConnectedTo("backup"), TimeSpan.FromSeconds(30));
|
||||
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(10),
|
||||
$"failover took {sw.Elapsed.TotalSeconds:F1}s (QA before #141: 66 s)");
|
||||
Assert.Equal("Backup", (await actor.Ask<DataConnectionHealthReport>(
|
||||
new DataConnectionActor.GetHealthReport(), TimeSpan.FromSeconds(5))).ActiveEndpoint);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await a.DisposeAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user