feat(dcl): ServiceLevel-aware OPC UA failover for primary/backup pairs (#138)

Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.

- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
  endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
  Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
  the first sample after any (re)connect, triggers a probe of the other endpoint
  (short-lived session: connect, read, dispose). Switch only when it reports a
  Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
  the previous ServiceLevel switch. Both degraded, other unreachable or level
  unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
  three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
  zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
  minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
  (enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
  migration. Central UI: fields in the Backup card next to Failover Retry Count.
  Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
  ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
  promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
  and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 03:06:45 -05:00
parent 16209831ca
commit 712c5117ee
23 changed files with 1760 additions and 36 deletions
@@ -165,6 +165,40 @@
min="1" max="20" @bind="_formFailoverRetryCount" />
<div class="form-text">Retries before failing over to backup endpoint.</div>
</div>
@if (_protocol != "MxGateway")
{
<div class="mb-2" data-test="servicelevel-failover">
<div class="form-check">
<input type="checkbox" class="form-check-input" id="sl-failover-enabled"
@bind="_formSlEnabled" />
<label class="form-check-label small" for="sl-failover-enabled">
ServiceLevel failover
</label>
</div>
<div class="form-text">
Fail over when the connected endpoint's ServiceLevel stays below 200 (Degraded)
for the window below and the other endpoint reports a Healthy (200+), higher level.
Needed for an OtOpcUa pair in Warm redundancy; a 250/240 pair never triggers it.
</div>
@if (_formSlEnabled)
{
<div class="row g-2 mt-1">
<div class="col-auto">
<label class="form-label small" for="sl-failover-window">Degraded window (s)</label>
<input type="number" id="sl-failover-window" class="form-control form-control-sm"
style="max-width: 120px;" min="1" max="3600" @bind="_formSlWindowSeconds" />
@RenderPrimaryError("ServiceLevelFailover.DegradedWindowSeconds")
</div>
<div class="col-auto">
<label class="form-label small" for="sl-failover-dwell">Minimum dwell (s)</label>
<input type="number" id="sl-failover-dwell" class="form-control form-control-sm"
style="max-width: 120px;" min="0" max="86400" @bind="_formSlDwellSeconds" />
@RenderPrimaryError("ServiceLevelFailover.MinDwellSeconds")
</div>
</div>
}
</div>
}
<div class="mb-0">
<button type="button" class="btn btn-outline-danger btn-sm"
@onclick="RemoveBackup">Remove Backup</button>
@@ -207,6 +241,10 @@
private bool _backupIsLegacy;
private bool _showBackup;
private int _formFailoverRetryCount = 3;
// ScadaBridge#138: connection-level ServiceLevel failover, stored on the PRIMARY OPC UA endpoint.
private bool _formSlEnabled = true;
private int _formSlWindowSeconds = OpcUaServiceLevelFailoverConfig.DefaultDegradedWindowSeconds;
private int _formSlDwellSeconds = OpcUaServiceLevelFailoverConfig.DefaultMinDwellSeconds;
private ValidationResult? _primaryErrors;
private ValidationResult? _backupErrors;
private string? _formError;
@@ -276,6 +314,10 @@
{
(_primaryConfig, _primaryIsLegacy) =
OpcUaEndpointConfigSerializer.Deserialize(conn.PrimaryConfiguration);
var sl = _primaryConfig.ServiceLevelFailover ?? new OpcUaServiceLevelFailoverConfig();
_formSlEnabled = sl.Enabled;
_formSlWindowSeconds = sl.DegradedWindowSeconds;
_formSlDwellSeconds = sl.MinDwellSeconds;
if (!string.IsNullOrWhiteSpace(conn.BackupConfiguration))
{
(_backupConfig, _backupIsLegacy) =
@@ -313,6 +355,15 @@
}
else
{
// Only meaningful with a backup; without one the setting is not stored at all.
_primaryConfig.ServiceLevelFailover = _showBackup
? new OpcUaServiceLevelFailoverConfig
{
Enabled = _formSlEnabled,
DegradedWindowSeconds = _formSlWindowSeconds,
MinDwellSeconds = _formSlDwellSeconds,
}
: null;
_primaryErrors = OpcUaEndpointConfigValidator.Validate(_primaryConfig, "Primary.");
_backupErrors = _showBackup
? OpcUaEndpointConfigValidator.Validate(_backupConfig, "Backup.")
@@ -350,6 +401,17 @@
_backupMx = new MxGatewayEndpointConfig();
_backupIsLegacy = false;
_formFailoverRetryCount = 3;
_formSlEnabled = true;
_formSlWindowSeconds = OpcUaServiceLevelFailoverConfig.DefaultDegradedWindowSeconds;
_formSlDwellSeconds = OpcUaServiceLevelFailoverConfig.DefaultMinDwellSeconds;
}
private RenderFragment? RenderPrimaryError(string field)
{
var match = _primaryErrors?.Errors.FirstOrDefault(e => e.EntityName == "Primary." + field);
return match is null
? null
: @<div class="text-danger small">@match.Message</div>;
}
private void GoBack() => NavigationManager.NavigateTo("/design/connections");