feat(dcl): ServiceLevel-aware OPC UA failover for primary/backup pairs (#138)

Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.

- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
  endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
  Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
  the first sample after any (re)connect, triggers a probe of the other endpoint
  (short-lived session: connect, read, dispose). Switch only when it reports a
  Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
  the previous ServiceLevel switch. Both degraded, other unreachable or level
  unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
  three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
  zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
  minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
  (enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
  migration. Central UI: fields in the Backup card next to Failover Retry Count.
  Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
  ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
  promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
  and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 03:06:45 -05:00
parent 16209831ca
commit 712c5117ee
23 changed files with 1760 additions and 36 deletions
@@ -167,4 +167,74 @@ public class DataConnectionFormTests : BunitContext
Assert.Equal(60000,
doc.RootElement.GetProperty("sessionTimeoutMs").GetInt32());
}
/// <summary>
/// ScadaBridge#138: with a backup, the ServiceLevel failover fields shown next to Failover
/// Retry Count are stored on the PRIMARY endpoint's JSON; the backup's copy stays null.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Save_WithBackup_StoresServiceLevelFailoverOnThePrimary()
{
DataConnection? captured = null;
await _siteRepo.AddDataConnectionAsync(Arg.Do<DataConnection>(d => captured = d));
var cut = RenderForCreateSite(1);
cut.FindAll("input[type='text']")
.First(i => !i.HasAttribute("readonly")
&& (i.GetAttribute("placeholder") is null
|| !i.GetAttribute("placeholder")!.StartsWith("opc.tcp")))
.Change("OtOpcUa-Pair");
cut.FindAll("input[type='text']")
.First(i => i.GetAttribute("placeholder")?.StartsWith("opc.tcp") == true)
.Change("opc.tcp://otopcua-a:4840");
await cut.FindAll("button").First(b => b.TextContent.Trim() == "Add Backup Endpoint").ClickAsync(new());
cut.FindAll("input[type='text']")
.Where(i => i.GetAttribute("placeholder")?.StartsWith("opc.tcp") == true)
.ElementAt(1)
.Change("opc.tcp://otopcua-b:4840");
// Defaults are on screen, then edited.
Assert.Equal("10", cut.Find("#sl-failover-window").GetAttribute("value"));
Assert.Equal("60", cut.Find("#sl-failover-dwell").GetAttribute("value"));
cut.Find("#sl-failover-window").Change("15");
cut.Find("#sl-failover-dwell").Change("90");
await cut.FindAll("button").First(b => b.TextContent.Trim() == "Save").ClickAsync(new());
Assert.NotNull(captured);
using var primary = JsonDocument.Parse(captured!.PrimaryConfiguration!);
var sl = primary.RootElement.GetProperty("serviceLevelFailover");
Assert.True(sl.GetProperty("enabled").GetBoolean());
Assert.Equal(15, sl.GetProperty("degradedWindowSeconds").GetInt32());
Assert.Equal(90, sl.GetProperty("minDwellSeconds").GetInt32());
using var backup = JsonDocument.Parse(captured.BackupConfiguration!);
Assert.Equal(JsonValueKind.Null, backup.RootElement.GetProperty("serviceLevelFailover").ValueKind);
}
/// <summary>ScadaBridge#138: without a backup the setting is neither shown nor stored.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Save_WithoutBackup_StoresNoServiceLevelFailover()
{
DataConnection? captured = null;
await _siteRepo.AddDataConnectionAsync(Arg.Do<DataConnection>(d => captured = d));
var cut = RenderForCreateSite(1);
Assert.Empty(cut.FindAll("#sl-failover-enabled"));
cut.FindAll("input[type='text']")
.First(i => !i.HasAttribute("readonly")
&& (i.GetAttribute("placeholder") is null
|| !i.GetAttribute("placeholder")!.StartsWith("opc.tcp")))
.Change("PLC-1");
cut.FindAll("input[type='text']")
.First(i => i.GetAttribute("placeholder")?.StartsWith("opc.tcp") == true)
.Change("opc.tcp://plant-a:4840");
await cut.FindAll("button").First(b => b.TextContent.Trim() == "Save").ClickAsync(new());
Assert.NotNull(captured);
using var primary = JsonDocument.Parse(captured!.PrimaryConfiguration!);
Assert.Equal(JsonValueKind.Null, primary.RootElement.GetProperty("serviceLevelFailover").ValueKind);
}
}