fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)

QA Warm gate (lmxopcua#907, 2026-11-13): a graceful stop of the Warm Primary
reported ServiceLevel 0 for ~6 s - shorter than the 10 s degraded window - the
next unreadable sample cleared the streak, and the connection retried the stopped
node FailoverRetryCount times: Good on the peer after 66 s, peer Active in 0.2 s.

Two going-away triggers on top of #138, both still requiring a Healthy (>= 200),
higher level on the other endpoint and honouring the minimum dwell:
- ServiceLevel 0 (Part 4 Maintenance) probes the other endpoint at once instead
  of waiting out the window - once per degraded streak, so a non-Healthy peer is
  re-probed once per window, not every poll. Other Degraded levels (standby 90/50,
  booting 1) still wait the window.
- A lost link whose session's last KNOWN ServiceLevel was < 200 (unreadable
  samples don't overwrite it) probes the other endpoint before the first
  reconnect attempt, holding the reconnect timer. Healthy -> switch and reconnect
  at zero delay, the stopped node is never retried; otherwise the ordinary retry
  cycle runs (less the probe's time). After only Healthy levels (250/240, Unset),
  inside the dwell, or with the feature off: unchanged retry-then-failover.

Tests: DataConnectionGoingAwayFailoverTests (10 cases, each trigger and
non-trigger with positive controls, connect-order artifact assertions) and a real
in-process OPC UA server pair graceful-stop e2e on production defaults: 1.1 s
(lost-link trigger alone 12.1 s, neither 33.1 s -> fails the 10 s bound).
13 mutants, all killed. DCL tests 608/608 (3 full runs); solution 0 warnings.
Docs: Component-DataConnectionLayer, components/DataConnectionLayer.

Not deployed; QA live re-check pending.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 09:36:59 -05:00
parent b19836788b
commit c4ec51e89a
7 changed files with 562 additions and 17 deletions
@@ -208,4 +208,54 @@ public sealed class OpcUaServiceLevelTests : TestKit
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionEndpoint)
&& c.GetArguments() is ["today", "Connected to backup"]);
}
/// <summary>
/// ScadaBridge#141 end to end, the QA Warm gate's step 4a: the connection is on the Warm Primary
/// (A, 250) with the standby (B, 90) as backup, both on the DEFAULT policy (10 s window, 60 s
/// dwell) and the default 5 s reconnect interval with three retries. A is stopped gracefully the
/// way OtOpcUa does it — the peer promotes to 250, A reports ServiceLevel 0 for a moment, then
/// the server goes away. The connection must be live on B within seconds; the connectivity path
/// alone (keep-alive + three 5 s retries of the stopped A) needs well over 15 s.
/// </summary>
[Fact]
public async Task RealServerPair_GracefulStopOfTheWarmPrimary_FailsOverWithinSeconds()
{
var a = await SecureOpcUaTestServer.StartAsync();
await using var b = await SecureOpcUaTestServer.StartAsync();
try
{
var factory = new DataConnectionFactory(NullLoggerFactory.Instance, Options.Create(ClientOptions()));
var options = new DataConnectionOptions(); // production defaults: 5 s reconnect, 2 s poll, 15 s probe
a.SetServiceLevel(250);
b.SetServiceLevel(90);
var health = Substitute.For<ISiteHealthCollector>();
var primary = Flat(a.Url); // null ServiceLevelFailover = defaults (enabled, 10 s, 60 s)
bool ConnectedTo(string endpoint) => health.ReceivedCalls().Any(c =>
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionEndpoint)
&& c.GetArguments() is ["stop", var label] && (string)label! == $"Connected to {endpoint}");
var actor = Sys.ActorOf(Props.Create(() => new DataConnectionActor(
"stop", factory.Create("OpcUa", primary), options, health, factory, "OpcUa",
primary, Flat(b.Url), 3, null)));
AwaitCondition(() => ConnectedTo("primary"), TimeSpan.FromSeconds(30));
await Task.Delay(TimeSpan.FromSeconds(3)); // a few Healthy polls: no connect-time effect left
var sw = System.Diagnostics.Stopwatch.StartNew();
b.SetServiceLevel(250); // the peer promotes (0.2 s on QA)
a.SetServiceLevel(0); // the stopping node reports Maintenance...
await Task.Delay(TimeSpan.FromSeconds(1));
await a.DisposeAsync(); // ...and is gone well inside the 10 s window
AwaitCondition(() => ConnectedTo("backup"), TimeSpan.FromSeconds(30));
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(10),
$"failover took {sw.Elapsed.TotalSeconds:F1}s (QA before #141: 66 s)");
Assert.Equal("Backup", (await actor.Ask<DataConnectionHealthReport>(
new DataConnectionActor.GetHealthReport(), TimeSpan.FromSeconds(5))).ActiveEndpoint);
}
finally
{
await a.DisposeAsync();
}
}
}