dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)

Browse Source 
QA Warm gate (lmxopcua#907, 2026-11-13): a graceful stop of the Warm Primary
reported ServiceLevel 0 for ~6 s - shorter than the 10 s degraded window - the
next unreadable sample cleared the streak, and the connection retried the stopped
node FailoverRetryCount times: Good on the peer after 66 s, peer Active in 0.2 s.

Two going-away triggers on top of #138, both still requiring a Healthy (>= 200),
higher level on the other endpoint and honouring the minimum dwell:
- ServiceLevel 0 (Part 4 Maintenance) probes the other endpoint at once instead
  of waiting out the window - once per degraded streak, so a non-Healthy peer is
  re-probed once per window, not every poll. Other Degraded levels (standby 90/50,
  booting 1) still wait the window.
- A lost link whose session's last KNOWN ServiceLevel was < 200 (unreadable
  samples don't overwrite it) probes the other endpoint before the first
  reconnect attempt, holding the reconnect timer. Healthy -> switch and reconnect
  at zero delay, the stopped node is never retried; otherwise the ordinary retry
  cycle runs (less the probe's time). After only Healthy levels (250/240, Unset),
  inside the dwell, or with the feature off: unchanged retry-then-failover.

Tests: DataConnectionGoingAwayFailoverTests (10 cases, each trigger and
non-trigger with positive controls, connect-order artifact assertions) and a real
in-process OPC UA server pair graceful-stop e2e on production defaults: 1.1 s
(lost-link trigger alone 12.1 s, neither 33.1 s -> fails the 10 s bound).
13 mutants, all killed. DCL tests 608/608 (3 full runs); solution 0 warnings.
Docs: Component-DataConnectionLayer, components/DataConnectionLayer.

Not deployed; QA live re-check pending.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 09:36:59 -05:00
parent b19836788b
commit c4ec51e89a
7 changed files with 562 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.ServiceLevel.cs
+145 -15
View File
@@ -37,12 +37,32 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
/// monitored at all. The switch itself reuses <see cref="SwitchToOtherEndpoint"/> and the reconnect
/// cycle, exactly like the connection-driven failovers.
/// </para>
///
/// <para>
/// <b>A server going away (ScadaBridge#141).</b> A graceful stop of a redundant OtOpcUa node reports
/// ServiceLevel 0 for only a few seconds before the session closes — shorter than the degraded window
/// — and the following unreadable sample cleared the streak, so the connection fell back to the
/// connectivity path and retried the stopped node <c>FailoverRetryCount</c> times (66 s on QA). Two
/// going-away triggers now skip that wait: (1) a sample of <see cref="MaintenanceServiceLevel"/> (0,
/// Part 4 "Maintenance": the server is not available for use) probes the other endpoint at once,
/// without the window; and (2) when the link is lost while the last KNOWN level of the session was
/// Degraded (below 200), the other endpoint is probed before the first reconnect attempt and, if it
/// answers Healthy, the connection switches straight to it instead of spending the retry count on the
/// endpoint that announced it was degraded. Both keep the "Healthy and higher" rule and the dwell; a
/// pair that only ever reports Healthy levels (250/240) never meets either condition.
/// </para>
/// </summary>
public partial class DataConnectionActor
{
/// <summary>The lowest ServiceLevel in the OPC UA Healthy band (Part 4 §6.6.2.4.2).</summary>
internal const byte HealthyServiceLevelMinimum = 200;
/// <summary>
/// Part 4 §6.6.2.4.2 "Maintenance": the server is not available for use — OtOpcUa reports it while
/// it is shutting down or its endpoint has failed. Skips the degraded window (ScadaBridge#141).
/// </summary>
internal const byte MaintenanceServiceLevel = 0;
private const string ServiceLevelPollTimerKey = "servicelevel-poll";
private ServiceLevelFailoverPolicy _serviceLevelPolicy;
@@ -59,6 +79,27 @@ public partial class DataConnectionActor
/// <summary>True until the first sample of a session is evaluated (connect-time preference).</summary>
private bool _serviceLevelConnectCheckPending;
/// <summary>
/// The last level this session actually read (unknown samples do not overwrite it); null when
/// none has been read. Drives the lost-link probe (ScadaBridge#141).
/// </summary>
private byte? _lastKnownServiceLevel;
/// <summary>
/// True while the lost-link probe of the other endpoint is outstanding; the first reconnect
/// attempt is held until it answers (ScadaBridge#141).
/// </summary>
private bool _disconnectProbePending;
/// <summary>
/// True once ServiceLevel 0 has started a probe in the current degraded streak: 0 skips the window
/// once, so a peer that is not Healthy is then re-probed once per window, not on every poll.
/// </summary>
private bool _goingAwayProbeStarted;
/// <summary>When the outstanding lost-link probe started (to shorten the reconnect wait after it).</summary>
private DateTimeOffset _disconnectProbeStartedAt;
/// <summary>Why <see cref="BecomeReconnecting"/> is re-establishing the link.</summary>
internal enum ReconnectCause
{
@@ -104,6 +145,8 @@ public partial class DataConnectionActor
_serviceLevelReadInFlight = false;
_serviceLevelProbeInFlight = false;
_serviceLevelConnectCheckPending = false;
_lastKnownServiceLevel = null;
_goingAwayProbeStarted = false;
}
private void HandleServiceLevelPollTick()
@@ -131,6 +174,8 @@ public partial class DataConnectionActor
_serviceLevelReadInFlight = false;
var connectCheck = _serviceLevelConnectCheckPending;
_serviceLevelConnectCheckPending = false;
if (sampled.Level is { } known)
_lastKnownServiceLevel = known;
if (sampled.Level is not { } level || level >= HealthyServiceLevelMinimum)
{
@@ -140,10 +185,14 @@ public partial class DataConnectionActor
_log.Info("[{0}] {1} endpoint ServiceLevel is {2} — degraded watch cleared",
_connectionName, _activeEndpoint, sampled.Level?.ToString(CultureInfo.InvariantCulture) ?? "unknown");
_degradedSince = null;
_goingAwayProbeStarted = false;
return;
}
var now = DateTimeOffset.UtcNow;
// ScadaBridge#141: 0 means the server is going away (graceful stop) — it will not be there
// for a full window, so the other endpoint is checked now. Still "Healthy and higher" + dwell.
var goingAway = level == MaintenanceServiceLevel && !_goingAwayProbeStarted;
if (_degradedSince is null)
{
_degradedSince = now;
@@ -151,10 +200,11 @@ public partial class DataConnectionActor
+ "will check the other endpoint {4}",
_connectionName, _activeEndpoint, level, HealthyServiceLevelMinimum,
connectCheck ? "now (connect-time preference)"
: goingAway ? "now (ServiceLevel 0: the server is going away)"
: $"if it persists for {_serviceLevelPolicy.DegradedWindow.TotalSeconds:F0}s");
}
if (!connectCheck && now - _degradedSince.Value < _serviceLevelPolicy.DegradedWindow)
if (!connectCheck && !goingAway && now - _degradedSince.Value < _serviceLevelPolicy.DegradedWindow)
return;
if (now - _lastServiceLevelSwitchAt < _serviceLevelPolicy.MinDwell)
@@ -165,6 +215,8 @@ public partial class DataConnectionActor
return;
}
if (goingAway)
_goingAwayProbeStarted = true;
StartServiceLevelProbe(level);
}
@@ -173,9 +225,10 @@ public partial class DataConnectionActor
/// configuration — connect, read, dispose — off the actor thread, bounded by
/// <see cref="DataConnectionOptions.ServiceLevelProbeTimeout"/>. The live adapter is untouched.
/// </summary>
private void StartServiceLevelProbe(byte currentLevel)
private void StartServiceLevelProbe(byte currentLevel, bool afterDisconnect = false)
{
_serviceLevelProbeInFlight = true;
if (!afterDisconnect)
_serviceLevelProbeInFlight = true;
var generation = _adapterGeneration;
var otherConfig = _activeEndpoint == ActiveEndpoint.Primary ? _backupConfig! : _primaryConfig;
var factory = _factory;
@@ -184,11 +237,12 @@ public partial class DataConnectionActor
Task.Run(() => ProbeServiceLevelAsync(factory, protocol, otherConfig).WaitAsync(timeout))
.ContinueWith(t => t.IsCompletedSuccessfully
? new ServiceLevelProbed(generation, currentLevel, t.Result.Level, t.Result.Error)
? new ServiceLevelProbed(generation, currentLevel, t.Result.Level, t.Result.Error, afterDisconnect)
: new ServiceLevelProbed(generation, currentLevel, null,
t.Exception?.GetBaseException() is TimeoutException
? $"no answer within {timeout.TotalSeconds:F0}s"
: t.Exception?.GetBaseException().Message ?? "probe cancelled"),
: t.Exception?.GetBaseException().Message ?? "probe cancelled",
afterDisconnect),
TaskScheduler.Default)
.PipeTo(_self);
}
@@ -223,7 +277,8 @@ public partial class DataConnectionActor
private void HandleServiceLevelProbed(ServiceLevelProbed probed)
{
if (probed.AdapterGeneration != _adapterGeneration)
// A lost-link probe belongs to the Reconnecting state (HandleDisconnectProbed).
if (probed.AfterDisconnect || probed.AdapterGeneration != _adapterGeneration)
return;
_serviceLevelProbeInFlight = false;
@@ -252,25 +307,95 @@ public partial class DataConnectionActor
}
var degradedFor = now - _degradedSince.Value;
SwitchForServiceLevel(probed.CurrentLevel, otherLevel,
$"ServiceLevel {probed.CurrentLevel} Degraded for {degradedFor.TotalSeconds:F0}s");
BecomeReconnecting(ReconnectCause.ServiceLevelFailover);
}
/// <summary>
/// The endpoint switch shared by the Connected-state and the lost-link ServiceLevel triggers:
/// flips the adapter, resets both connection-failure counters, starts the dwell and logs it.
/// The caller re-arms the reconnect.
/// </summary>
private void SwitchForServiceLevel(byte currentLevel, byte otherLevel, string why)
{
var previousEndpoint = SwitchToOtherEndpoint();
_consecutiveFailures = 0;
_consecutiveUnstableDisconnects = 0;
_lastServiceLevelSwitchAt = now;
_lastServiceLevelSwitchAt = DateTimeOffset.UtcNow;
_log.Warning("[{0}] Failing over from {1} to {2} (ServiceLevel {3} Degraded for {4:F0}s; {2} reports {5})",
_connectionName, previousEndpoint, _activeEndpoint, probed.CurrentLevel,
degradedFor.TotalSeconds, otherLevel);
_log.Warning("[{0}] Failing over from {1} to {2} ({3}; {2} reports {4})",
_connectionName, previousEndpoint, _activeEndpoint, why, otherLevel);
if (_siteEventLogger != null)
{
_ = _siteEventLogger.LogEventAsync(
"connection", "Warning", null, _connectionName,
$"Failover from {previousEndpoint} to {_activeEndpoint} (ServiceLevel)",
$"{previousEndpoint} reported ServiceLevel {probed.CurrentLevel} for {degradedFor.TotalSeconds:F0}s; "
+ $"{_activeEndpoint} reports {otherLevel}");
$"{previousEndpoint}: {why} (last ServiceLevel {currentLevel}); {_activeEndpoint} reports {otherLevel}");
}
}
/// <summary>
/// ScadaBridge#141 — called by <see cref="BecomeReconnecting"/> for a LOST link. When the session's
/// last known ServiceLevel was Degraded (below 200: the server had said it was not the one to be
/// on, typically 0 during a graceful stop), the other endpoint is probed before the first reconnect
/// attempt instead of retrying the endpoint that is going away. Not when the policy is not in
/// force, the session never read a Degraded level (250/240 pairs, unknown-only), or the dwell of
/// the previous ServiceLevel switch has not passed.
/// </summary>
/// <param name="lastKnownLevel">The ended session's last known level, captured before it was reset.</param>
/// <returns>True when the probe was started; the caller must then NOT arm the reconnect timer.</returns>
private bool TryStartDisconnectProbe(byte? lastKnownLevel)
{
if (lastKnownLevel is not { } level || level >= HealthyServiceLevelMinimum || !ServiceLevelFailoverInForce)
return false;
var sinceSwitch = DateTimeOffset.UtcNow - _lastServiceLevelSwitchAt;
if (sinceSwitch < _serviceLevelPolicy.MinDwell)
{
_log.Info("[{0}] Link lost after ServiceLevel {1}, but the last ServiceLevel switch was {2:F0}s ago "
+ "(dwell {3:F0}s) — normal reconnect cycle", _connectionName, level,
sinceSwitch.TotalSeconds, _serviceLevelPolicy.MinDwell.TotalSeconds);
return false;
}
BecomeReconnecting(ReconnectCause.ServiceLevelFailover);
_log.Warning("[{0}] Link to the {1} endpoint lost after it reported ServiceLevel {2} — checking the "
+ "other endpoint before retrying", _connectionName, _activeEndpoint, level);
_disconnectProbePending = true;
_disconnectProbeStartedAt = DateTimeOffset.UtcNow;
StartServiceLevelProbe(level, afterDisconnect: true);
return true;
}
/// <summary>
/// ScadaBridge#141 — the lost-link probe's verdict (Reconnecting state). A Healthy level on the
/// other endpoint switches to it and reconnects at once; anything else releases the held reconnect
/// into the ordinary retry cycle on the same endpoint, minus the time the probe already took.
/// </summary>
private void HandleDisconnectProbed(ServiceLevelProbed probed)
{
if (!probed.AfterDisconnect || !_disconnectProbePending || probed.AdapterGeneration != _adapterGeneration)
return;
_disconnectProbePending = false;
var other = _activeEndpoint == ActiveEndpoint.Primary ? ActiveEndpoint.Backup : ActiveEndpoint.Primary;
if (probed.OtherLevel is { } otherLevel
&& otherLevel >= HealthyServiceLevelMinimum && otherLevel > probed.CurrentLevel)
{
SwitchForServiceLevel(probed.CurrentLevel, otherLevel,
$"link lost after ServiceLevel {probed.CurrentLevel}");
Timers.StartSingleTimer("reconnect", new AttemptConnect(), TimeSpan.Zero);
return;
}
_log.Warning("[{0}] Link to the {1} endpoint lost after ServiceLevel {2}, but the {3} endpoint {4} — "
+ "normal reconnect cycle", _connectionName, _activeEndpoint, probed.CurrentLevel, other,
probed.OtherLevel is { } l ? $"reports {l} (not a Healthy, higher level)" : $"is not available ({probed.Error})");
var remaining = _options.ReconnectInterval - (DateTimeOffset.UtcNow - _disconnectProbeStartedAt);
Timers.StartSingleTimer("reconnect", new AttemptConnect(),
remaining > TimeSpan.Zero ? remaining : TimeSpan.Zero);
}
/// <summary>Poll tick for the connected endpoint's ServiceLevel.</summary>
@@ -283,8 +408,13 @@ public partial class DataConnectionActor
/// <summary>One ServiceLevel sample of the connected endpoint (null = unknown).</summary>
internal sealed record ServiceLevelSampled(int AdapterGeneration, byte? Level);
/// <summary>The other endpoint's ServiceLevel (null = unreachable / unreadable, see Error).</summary>
internal sealed record ServiceLevelProbed(int AdapterGeneration, byte CurrentLevel, byte? OtherLevel, string? Error);
/// <summary>
/// The other endpoint's ServiceLevel (null = unreachable / unreadable, see Error).
/// <paramref name="AfterDisconnect"/> marks the lost-link probe (ScadaBridge#141), which only the
/// Reconnecting state acts on.
/// </summary>
internal sealed record ServiceLevelProbed(
int AdapterGeneration, byte CurrentLevel, byte? OtherLevel, string? Error, bool AfterDisconnect = false);
}
/// <summary>