feat(dcl): ServiceLevel-aware OPC UA failover for primary/backup pairs (#138)
Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.
- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
the first sample after any (re)connect, triggers a probe of the other endpoint
(short-lived session: connect, read, dispose). Switch only when it reports a
Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
the previous ServiceLevel switch. Both degraded, other unreachable or level
unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
(enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
migration. Central UI: fields in the Backup card next to Failover Retry Count.
Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
+320
@@ -0,0 +1,320 @@
|
||||
using System.Globalization;
|
||||
using Akka.Actor;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Serialization;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.DataConnections;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#138 — ServiceLevel-aware failover between a primary and a backup endpoint.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Why.</b> Until now the connection failed over only when the link itself misbehaved
|
||||
/// (<c>FailoverRetryCount</c> consecutive connect failures or unstable disconnects). Under OtOpcUa
|
||||
/// Warm redundancy (lmxopcua#717) the standby keeps a perfectly healthy session but serves
|
||||
/// <c>BadOutOfService</c> data at a standby ServiceLevel (90 healthy / 50 degraded / 1 booting), so a
|
||||
/// connection that lands there would sit on Bad quality indefinitely.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>What.</b> While Connected, the connected endpoint's <c>Server_ServiceLevel</c> is polled every
|
||||
/// <see cref="DataConnectionOptions.ServiceLevelPollInterval"/>. A level below
|
||||
/// <see cref="HealthyServiceLevelMinimum"/> (200, the bottom of the Part 4 Healthy band) starts a
|
||||
/// degraded streak; any Healthy or unreadable sample ends it. Once the streak has lasted the
|
||||
/// connection's degraded window — or immediately on the first sample after a (re)connect, which is
|
||||
/// the "prefer the better endpoint on connect" rule — the other endpoint is probed with a
|
||||
/// short-lived session. The connection switches only when that probe reads a Healthy level that is
|
||||
/// also higher than the connected one, and never within the dwell time of the previous
|
||||
/// ServiceLevel switch. Otherwise it stays put and the window re-arms.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Unchanged behaviour.</b> A level of 200 or more never starts a streak, so today's OtOpcUa pair
|
||||
/// (250 Primary / 240 Secondary) never triggers anything; an adapter without
|
||||
/// <see cref="IServiceLevelReadable"/> (MxGateway) and a connection without a backup are never
|
||||
/// monitored at all. The switch itself reuses <see cref="SwitchToOtherEndpoint"/> and the reconnect
|
||||
/// cycle, exactly like the connection-driven failovers.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public partial class DataConnectionActor
|
||||
{
|
||||
/// <summary>The lowest ServiceLevel in the OPC UA Healthy band (Part 4 §6.6.2.4.2).</summary>
|
||||
internal const byte HealthyServiceLevelMinimum = 200;
|
||||
|
||||
private const string ServiceLevelPollTimerKey = "servicelevel-poll";
|
||||
|
||||
private ServiceLevelFailoverPolicy _serviceLevelPolicy;
|
||||
|
||||
/// <summary>When the current unbroken run of Degraded samples began; null when not degraded.</summary>
|
||||
private DateTimeOffset? _degradedSince;
|
||||
|
||||
/// <summary>When the last ServiceLevel-driven switch happened (dwell / anti-flap).</summary>
|
||||
private DateTimeOffset _lastServiceLevelSwitchAt = DateTimeOffset.MinValue;
|
||||
|
||||
private bool _serviceLevelReadInFlight;
|
||||
private bool _serviceLevelProbeInFlight;
|
||||
|
||||
/// <summary>True until the first sample of a session is evaluated (connect-time preference).</summary>
|
||||
private bool _serviceLevelConnectCheckPending;
|
||||
|
||||
/// <summary>Why <see cref="BecomeReconnecting"/> is re-establishing the link.</summary>
|
||||
internal enum ReconnectCause
|
||||
{
|
||||
/// <summary>The link was lost (disconnect, connection-level failure).</summary>
|
||||
ConnectionLost,
|
||||
|
||||
/// <summary>An in-place reconfiguration swapped the adapter.</summary>
|
||||
Reconfigured,
|
||||
|
||||
/// <summary>A sustained Degraded ServiceLevel swapped to the other endpoint (#138).</summary>
|
||||
ServiceLevelFailover,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when this connection's ServiceLevel is watched: the policy is enabled, there is a
|
||||
/// backup to go to, and the live adapter can read a ServiceLevel.
|
||||
/// </summary>
|
||||
private bool ServiceLevelFailoverInForce =>
|
||||
_serviceLevelPolicy.Enabled && _backupConfig != null && _adapter is IServiceLevelReadable;
|
||||
|
||||
/// <summary>Called on entering Connected. Arms the poll (first tick immediately).</summary>
|
||||
private void StartServiceLevelMonitoring()
|
||||
{
|
||||
ResetServiceLevelSession();
|
||||
if (!ServiceLevelFailoverInForce)
|
||||
return;
|
||||
|
||||
_serviceLevelConnectCheckPending = true;
|
||||
Timers.StartPeriodicTimer(ServiceLevelPollTimerKey, ServiceLevelPollTick.Instance,
|
||||
TimeSpan.Zero, _options.ServiceLevelPollInterval);
|
||||
}
|
||||
|
||||
/// <summary>Called on leaving Connected. Results still in flight are fenced by generation.</summary>
|
||||
private void StopServiceLevelMonitoring()
|
||||
{
|
||||
Timers.Cancel(ServiceLevelPollTimerKey);
|
||||
ResetServiceLevelSession();
|
||||
}
|
||||
|
||||
private void ResetServiceLevelSession()
|
||||
{
|
||||
_degradedSince = null;
|
||||
_serviceLevelReadInFlight = false;
|
||||
_serviceLevelProbeInFlight = false;
|
||||
_serviceLevelConnectCheckPending = false;
|
||||
}
|
||||
|
||||
private void HandleServiceLevelPollTick()
|
||||
{
|
||||
// One read at a time, and none while the other endpoint is being probed: the probe's
|
||||
// verdict is taken against the level it started from.
|
||||
if (!ServiceLevelFailoverInForce || _serviceLevelReadInFlight || _serviceLevelProbeInFlight)
|
||||
return;
|
||||
|
||||
_serviceLevelReadInFlight = true;
|
||||
var generation = _adapterGeneration;
|
||||
var reader = (IServiceLevelReadable)_adapter;
|
||||
var timeout = _options.ServiceLevelProbeTimeout;
|
||||
Task.Run(() => reader.ReadServiceLevelAsync().WaitAsync(timeout))
|
||||
.ContinueWith(t => new ServiceLevelSampled(generation,
|
||||
t.IsCompletedSuccessfully ? t.Result : null), TaskScheduler.Default)
|
||||
.PipeTo(_self);
|
||||
}
|
||||
|
||||
private void HandleServiceLevelSampled(ServiceLevelSampled sampled)
|
||||
{
|
||||
if (sampled.AdapterGeneration != _adapterGeneration)
|
||||
return;
|
||||
|
||||
_serviceLevelReadInFlight = false;
|
||||
var connectCheck = _serviceLevelConnectCheckPending;
|
||||
_serviceLevelConnectCheckPending = false;
|
||||
|
||||
if (sampled.Level is not { } level || level >= HealthyServiceLevelMinimum)
|
||||
{
|
||||
// Healthy, or unknown: either way the run of Degraded samples is broken. Unknown never
|
||||
// counts toward a switch — acting on missing information is how a pair flaps.
|
||||
if (_degradedSince != null)
|
||||
_log.Info("[{0}] {1} endpoint ServiceLevel is {2} — degraded watch cleared",
|
||||
_connectionName, _activeEndpoint, sampled.Level?.ToString(CultureInfo.InvariantCulture) ?? "unknown");
|
||||
_degradedSince = null;
|
||||
return;
|
||||
}
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
if (_degradedSince is null)
|
||||
{
|
||||
_degradedSince = now;
|
||||
_log.Warning("[{0}] {1} endpoint reports ServiceLevel {2} (Degraded band, < {3}) — "
|
||||
+ "will check the other endpoint {4}",
|
||||
_connectionName, _activeEndpoint, level, HealthyServiceLevelMinimum,
|
||||
connectCheck ? "now (connect-time preference)"
|
||||
: $"if it persists for {_serviceLevelPolicy.DegradedWindow.TotalSeconds:F0}s");
|
||||
}
|
||||
|
||||
if (!connectCheck && now - _degradedSince.Value < _serviceLevelPolicy.DegradedWindow)
|
||||
return;
|
||||
|
||||
if (now - _lastServiceLevelSwitchAt < _serviceLevelPolicy.MinDwell)
|
||||
{
|
||||
_log.Debug("[{0}] ServiceLevel {1} still Degraded but the last ServiceLevel switch was {2:F0}s ago "
|
||||
+ "(dwell {3:F0}s) — staying", _connectionName, level,
|
||||
(now - _lastServiceLevelSwitchAt).TotalSeconds, _serviceLevelPolicy.MinDwell.TotalSeconds);
|
||||
return;
|
||||
}
|
||||
|
||||
StartServiceLevelProbe(level);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads the OTHER endpoint's ServiceLevel through a short-lived adapter built from its
|
||||
/// configuration — connect, read, dispose — off the actor thread, bounded by
|
||||
/// <see cref="DataConnectionOptions.ServiceLevelProbeTimeout"/>. The live adapter is untouched.
|
||||
/// </summary>
|
||||
private void StartServiceLevelProbe(byte currentLevel)
|
||||
{
|
||||
_serviceLevelProbeInFlight = true;
|
||||
var generation = _adapterGeneration;
|
||||
var otherConfig = _activeEndpoint == ActiveEndpoint.Primary ? _backupConfig! : _primaryConfig;
|
||||
var factory = _factory;
|
||||
var protocol = _protocolType;
|
||||
var timeout = _options.ServiceLevelProbeTimeout;
|
||||
|
||||
Task.Run(() => ProbeServiceLevelAsync(factory, protocol, otherConfig).WaitAsync(timeout))
|
||||
.ContinueWith(t => t.IsCompletedSuccessfully
|
||||
? new ServiceLevelProbed(generation, currentLevel, t.Result.Level, t.Result.Error)
|
||||
: new ServiceLevelProbed(generation, currentLevel, null,
|
||||
t.Exception?.GetBaseException() is TimeoutException
|
||||
? $"no answer within {timeout.TotalSeconds:F0}s"
|
||||
: t.Exception?.GetBaseException().Message ?? "probe cancelled"),
|
||||
TaskScheduler.Default)
|
||||
.PipeTo(_self);
|
||||
}
|
||||
|
||||
private static async Task<(byte? Level, string? Error)> ProbeServiceLevelAsync(
|
||||
IDataConnectionFactory factory, string protocol, IDictionary<string, string> config)
|
||||
{
|
||||
IDataConnection? probe = null;
|
||||
try
|
||||
{
|
||||
probe = factory.Create(protocol, config);
|
||||
if (probe is not IServiceLevelReadable reader)
|
||||
return (null, "the adapter cannot read a ServiceLevel");
|
||||
|
||||
await probe.ConnectAsync(config).ConfigureAwait(false);
|
||||
var level = await reader.ReadServiceLevelAsync().ConfigureAwait(false);
|
||||
return (level, level is null ? "ServiceLevel unreadable" : null);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
return (null, ex.GetBaseException().Message);
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Fire-and-forget: closing a session against a black-holed server can block for the
|
||||
// operation timeout, and the probe's verdict must not wait for it.
|
||||
if (probe != null)
|
||||
_ = probe.DisposeAsync().AsTask().ContinueWith(
|
||||
t => _ = t.Exception, TaskContinuationOptions.OnlyOnFaulted);
|
||||
}
|
||||
}
|
||||
|
||||
private void HandleServiceLevelProbed(ServiceLevelProbed probed)
|
||||
{
|
||||
if (probed.AdapterGeneration != _adapterGeneration)
|
||||
return;
|
||||
|
||||
_serviceLevelProbeInFlight = false;
|
||||
if (!ServiceLevelFailoverInForce || _degradedSince is null)
|
||||
return;
|
||||
|
||||
var other = _activeEndpoint == ActiveEndpoint.Primary ? ActiveEndpoint.Backup : ActiveEndpoint.Primary;
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
|
||||
if (probed.OtherLevel is not { } otherLevel)
|
||||
{
|
||||
_log.Warning("[{0}] {1} endpoint ServiceLevel {2} is Degraded but the {3} endpoint is not "
|
||||
+ "available ({4}) — staying", _connectionName, _activeEndpoint, probed.CurrentLevel, other,
|
||||
probed.Error);
|
||||
_degradedSince = now; // re-arm: look again after another full window
|
||||
return;
|
||||
}
|
||||
|
||||
if (otherLevel < HealthyServiceLevelMinimum || otherLevel <= probed.CurrentLevel)
|
||||
{
|
||||
_log.Warning("[{0}] {1} endpoint ServiceLevel {2} is Degraded and the {3} endpoint reports {4} "
|
||||
+ "(not a Healthy, higher level) — staying", _connectionName, _activeEndpoint,
|
||||
probed.CurrentLevel, other, otherLevel);
|
||||
_degradedSince = now;
|
||||
return;
|
||||
}
|
||||
|
||||
var degradedFor = now - _degradedSince.Value;
|
||||
var previousEndpoint = SwitchToOtherEndpoint();
|
||||
_consecutiveFailures = 0;
|
||||
_consecutiveUnstableDisconnects = 0;
|
||||
_lastServiceLevelSwitchAt = now;
|
||||
|
||||
_log.Warning("[{0}] Failing over from {1} to {2} (ServiceLevel {3} Degraded for {4:F0}s; {2} reports {5})",
|
||||
_connectionName, previousEndpoint, _activeEndpoint, probed.CurrentLevel,
|
||||
degradedFor.TotalSeconds, otherLevel);
|
||||
|
||||
if (_siteEventLogger != null)
|
||||
{
|
||||
_ = _siteEventLogger.LogEventAsync(
|
||||
"connection", "Warning", null, _connectionName,
|
||||
$"Failover from {previousEndpoint} to {_activeEndpoint} (ServiceLevel)",
|
||||
$"{previousEndpoint} reported ServiceLevel {probed.CurrentLevel} for {degradedFor.TotalSeconds:F0}s; "
|
||||
+ $"{_activeEndpoint} reports {otherLevel}");
|
||||
}
|
||||
|
||||
BecomeReconnecting(ReconnectCause.ServiceLevelFailover);
|
||||
}
|
||||
|
||||
/// <summary>Poll tick for the connected endpoint's ServiceLevel.</summary>
|
||||
internal sealed record ServiceLevelPollTick
|
||||
{
|
||||
/// <summary>The single instance.</summary>
|
||||
public static readonly ServiceLevelPollTick Instance = new();
|
||||
}
|
||||
|
||||
/// <summary>One ServiceLevel sample of the connected endpoint (null = unknown).</summary>
|
||||
internal sealed record ServiceLevelSampled(int AdapterGeneration, byte? Level);
|
||||
|
||||
/// <summary>The other endpoint's ServiceLevel (null = unreachable / unreadable, see Error).</summary>
|
||||
internal sealed record ServiceLevelProbed(int AdapterGeneration, byte CurrentLevel, byte? OtherLevel, string? Error);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The connection-level ServiceLevel failover settings, read from the PRIMARY endpoint's flattened
|
||||
/// configuration (<see cref="OpcUaServiceLevelFailoverConfig"/>). Absent keys mean the defaults.
|
||||
/// </summary>
|
||||
internal readonly record struct ServiceLevelFailoverPolicy(bool Enabled, TimeSpan DegradedWindow, TimeSpan MinDwell)
|
||||
{
|
||||
/// <summary>Parses the policy; malformed or out-of-range values fall back to safe values.</summary>
|
||||
/// <param name="primaryConfig">The primary endpoint's flattened configuration.</param>
|
||||
/// <returns>The policy in force.</returns>
|
||||
public static ServiceLevelFailoverPolicy From(IDictionary<string, string>? primaryConfig)
|
||||
{
|
||||
var enabled = true;
|
||||
var window = OpcUaServiceLevelFailoverConfig.DefaultDegradedWindowSeconds;
|
||||
var dwell = OpcUaServiceLevelFailoverConfig.DefaultMinDwellSeconds;
|
||||
|
||||
if (primaryConfig != null)
|
||||
{
|
||||
if (primaryConfig.TryGetValue(OpcUaEndpointConfigSerializer.ServiceLevelFailoverEnabledKey, out var e)
|
||||
&& bool.TryParse(e, out var parsedEnabled))
|
||||
enabled = parsedEnabled;
|
||||
if (primaryConfig.TryGetValue(OpcUaEndpointConfigSerializer.ServiceLevelFailoverWindowKey, out var w)
|
||||
&& int.TryParse(w, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsedWindow))
|
||||
window = Math.Max(1, parsedWindow);
|
||||
if (primaryConfig.TryGetValue(OpcUaEndpointConfigSerializer.ServiceLevelFailoverDwellKey, out var d)
|
||||
&& int.TryParse(d, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsedDwell))
|
||||
dwell = Math.Max(0, parsedDwell);
|
||||
}
|
||||
|
||||
return new ServiceLevelFailoverPolicy(enabled, TimeSpan.FromSeconds(window), TimeSpan.FromSeconds(dwell));
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user