feat(dcl): ServiceLevel-aware OPC UA failover for primary/backup pairs (#138)

Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.

- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
  endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
  Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
  the first sample after any (re)connect, triggers a probe of the other endpoint
  (short-lived session: connect, read, dispose). Switch only when it reports a
  Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
  the previous ServiceLevel switch. Both degraded, other unreachable or level
  unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
  three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
  zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
  minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
  (enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
  migration. Central UI: fields in the Backup card next to Failover Retry Count.
  Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
  ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
  promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
  and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

This commit is contained in:

Joseph Doherty

2026-11-13 03:06:45 -05:00

parent 16209831ca

commit 712c5117ee

23 changed files with 1760 additions and 36 deletions

									
										docs/requirements/Component-CLI.md
									
		+2
		
												View File
												
				@@ -190,6 +190,8 @@ scadabridge data-connection delete --id <id>

				scadabridge data-connection move --id <id> --site-id <destination-site-id>

				```

				For an OPC UA connection with a backup, ServiceLevel-aware failover (Gitea #138) is set inside `--primary-config` as `"serviceLevelFailover": {"enabled": true, "degradedWindowSeconds": 10, "minDwellSeconds": 60}`; omitted means those defaults. See `Component-DataConnectionLayer.md` → *ServiceLevel-aware failover*.

				#### Site-routed OPC UA operations (browse / search / verify-endpoint / certs)

				CLI parity for the site-routed data-connection operations the ManagementActor dispatches