feat(dcl): ServiceLevel-aware OPC UA failover for primary/backup pairs (#138)

Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.

- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
  endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
  Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
  the first sample after any (re)connect, triggers a probe of the other endpoint
  (short-lived session: connect, read, dispose). Switch only when it reports a
  Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
  the previous ServiceLevel switch. Both degraded, other unreachable or level
  unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
  three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
  zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
  minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
  (enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
  migration. Central UI: fields in the Backup card next to Failover Retry Count.
  Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
  ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
  promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
  and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/components/DataConnectionLayer.md

@@ -119,6 +119,8 @@ if (result.Success)
 
 When `BackupConnectionDetails` is provided, the actor tracks consecutive failures and unstable disconnects. After `FailoverRetryCount` consecutive failures (or unstable connections shorter than `StableConnectionThreshold`), the actor disposes the current adapter, creates a fresh one with the other endpoint's config via `DataConnectionFactory.Create`, and re-arms the connect timer. Failover is round-robin: primary → backup → primary. There is no auto-failback; the connection stays on whichever endpoint is currently working.
 
+**ServiceLevel-aware failover (OPC UA, Gitea #138).** A third trigger, for redundant OPC UA pairs whose standby keeps a healthy session but serves Bad data (OtOpcUa Warm, lmxopcua#717). It lives in `Actors/DataConnectionActor.ServiceLevel.cs`. While Connected, and only when the adapter implements `IServiceLevelReadable`, the connection has a backup and the primary config's `ServiceLevelFailover.Enabled` is true, a periodic timer reads `Server_ServiceLevel` (`i=2267`). A level below 200 starts a degraded streak; a Healthy or unreadable sample ends it. After the degraded window (default 10 s) — or at once on the first sample after a (re)connect — the actor probes the other endpoint with a throwaway adapter from `DataConnectionFactory.Create` (connect, read, dispose; bounded by `ServiceLevelProbeTimeout`). It switches only if that endpoint reports ≥ 200 and higher, and not within the minimum dwell (default 60 s) of the previous ServiceLevel switch; otherwise it stays and re-arms the window. The switch is `SwitchToOtherEndpoint()` — the helper the two connectivity triggers now share — followed by `BecomeReconnecting(ReconnectCause.ServiceLevelFailover)` (zero delay, no unstable accounting). Poll and probe results carry the adapter generation, so a result from a session that has ended is dropped. Today's OtOpcUa (250/240) never starts a streak. Full rule set: `docs/requirements/Component-DataConnectionLayer.md` → *ServiceLevel-aware failover*.
+
 ### Write path
 
 Writes are fire-and-forget from the script's view only in the sense that there is no store-and-forward. The `DataConnectionActor` pipes `_adapter.WriteAsync` back to the original sender:
@@ -206,7 +208,7 @@ private void SwapAdapter(IDictionary<string, string> newConfig)
 }
 ```
 
-It then re-enters the reconnect cycle immediately (`BecomeReconnecting(afterReconfigure: true)` — zero delay, and the unstable-disconnect/failover accounting suppressed, since the link was dropped on purpose), and `ReSubscribeAll` / `ReSubscribeAllAlarms` re-issue everything from the untouched subscription registries. A reconfigure carrying the configuration already in force is a no-op.
+It then re-enters the reconnect cycle immediately (`BecomeReconnecting(ReconnectCause.Reconfigured)` — zero delay, and the unstable-disconnect/failover accounting suppressed, since the link was dropped on purpose), and `ReSubscribeAll` / `ReSubscribeAllAlarms` re-issue everything from the untouched subscription registries. A reconfigure carrying the configuration already in force is a no-op.
 
 ### Subscribing and receiving values
 
@@ -253,6 +255,8 @@ All settings under `DataConnectionLayer` in `appsettings.json` bind to `DataConn
 | `TagResolutionRetryInterval` | `00:00:10` | Retry interval for tag paths that could not be resolved on the device. |
 | `WriteTimeout` | `00:00:30` | Timeout applied to each `WriteAsync` call. A hung write times out and returns an error to the calling script. |
 | `StableConnectionThreshold` | `00:01:00` | A connection that drops before this duration is counted as an unstable disconnect toward `FailoverRetryCount`. |
+| `ServiceLevelPollInterval` | `00:00:02` | Read interval for the connected OPC UA endpoint's ServiceLevel when ServiceLevel-aware failover is in force (#138). |
+| `ServiceLevelProbeTimeout` | `00:00:15` | Budget for one ServiceLevel read and for probing the other endpoint (connect + read). |
 
 ### OPC UA global settings (`OpcUa` section)
 
@@ -286,6 +290,9 @@ Stored in the `DataConnection.PrimaryConfiguration` JSON dict, parsed by `OpcUaE
 | `QueueSize` | `10` | Per-item notification buffer size. |
 | `SecurityMode` | `None` | `None`, `Sign`, or `SignAndEncrypt`. |
 | `AutoAcceptUntrustedCerts` | `true` | Accept untrusted server certificates. |
+| `ServiceLevelFailover.Enabled` | `true` | ServiceLevel-aware failover (#138). Primary endpoint only; needs a backup. JSON `serviceLevelFailover` object; absent = defaults. |
+| `ServiceLevelFailover.DegradedWindowSeconds` | `10` | Seconds below ServiceLevel 200 before the other endpoint is probed. |
+| `ServiceLevelFailover.MinDwellSeconds` | `60` | Minimum seconds between ServiceLevel-driven switches. |
 
 ### Per-connection settings (MxGateway)
 
@@ -310,6 +317,7 @@ Carried on `CreateConnectionCommand` from the deployment artifact.
 |-------|---------|-------------|
 | `BackupConnectionDetails` | `null` | If absent, the connection retries its single endpoint indefinitely. |
 | `FailoverRetryCount` | `3` | Consecutive failures (or unstable disconnects) before switching endpoints. |
+| `ServiceLevelFailover` (OPC UA, inside the primary's configuration JSON) | enabled, 10 s, 60 s | ServiceLevel-aware failover (#138); see the OPC UA per-connection table. Lives in the protocol JSON rather than a column, so it needs no schema change and is encrypted at rest with the rest of the configuration. |
 
 ## Dependencies & Interactions
 

docs/requirements/Component-CLI.md

@@ -190,6 +190,8 @@ scadabridge data-connection delete --id <id>
 scadabridge data-connection move --id <id> --site-id <destination-site-id>
 ```
 
+For an OPC UA connection with a backup, ServiceLevel-aware failover (Gitea #138) is set inside `--primary-config` as `"serviceLevelFailover": {"enabled": true, "degradedWindowSeconds": 10, "minDwellSeconds": 60}`; omitted means those defaults. See `Component-DataConnectionLayer.md` → *ServiceLevel-aware failover*.
+
 #### Site-routed OPC UA operations (browse / search / verify-endpoint / certs)
 
 CLI parity for the site-routed data-connection operations the ManagementActor dispatches

docs/requirements/Component-DataConnectionLayer.md

@@ -137,7 +137,7 @@ flowchart TD
 ```
 
 - **Round-robin**: primary → backup → primary → backup. No preferred endpoint after first failover — the connection stays on whichever endpoint is working.
-- **No auto-failback**: The connection remains on the active endpoint until it fails.
+- **No auto-failback**: The connection remains on the active endpoint until it fails — or, for OPC UA, until its ServiceLevel says it is no longer the server to be on (see *ServiceLevel-aware failover* below).
 - **Single-endpoint connections** (no backup): Retry indefinitely on the same endpoint, preserving existing behavior.
 - **Adapter lifecycle on failover**: The actor disposes the current `IDataConnection` adapter and creates a fresh one via `DataConnectionFactory.Create()` with the other endpoint's configuration. Clean slate — no stale state.
 
@@ -152,6 +152,34 @@ flowchart TD
 
 See [`2026-03-22-primary-backup-data-connections-design.md`](../plans/2026-03-22-primary-backup-data-connections-design.md) for the full design.
 
+### ServiceLevel-aware failover (OPC UA, Gitea #138)
+
+The failover above is driven only by the link misbehaving. A redundant OPC UA server pair in **Warm** mode (OtOpcUa lmxopcua#717, design `OtOpcUa/docs/plans/2026-11-11-warm-redundancy-717-design.md` §4.3/§5) breaks that assumption: the standby keeps a perfectly healthy session but serves `BadOutOfService` data and advertises a standby ServiceLevel, so a connection that lands on it would sit on Bad quality forever. OPC UA connections with a backup therefore also watch `Server_ServiceLevel` (`i=2267`).
+
+| Band (OPC UA Part 4 §6.6.2.4.2) | Levels | OtOpcUa |
+|---|---|---|
+| Healthy | 200–255 | Active: 250 Primary / 240 Secondary (today and under Warm), 200 Active-stale |
+| Degraded | 100–199, and anything below 200 here | Active critically degraded 100; Warm standby 90 (healthy) / 50 (degraded) |
+| NoData / Maintenance | 1 / 0 | Warm booting 1; down / endpoint failed 0 |
+
+**Rule.** While Connected, the actor reads the connected endpoint's ServiceLevel every `ServiceLevelPollInterval` (2 s). A level **below 200** starts a *degraded streak*; any level of 200 or more, or an unreadable level (Bad status, no such node, read failure), ends it. When the streak has lasted the connection's **degraded window** (default 10 s), the actor probes the **other** endpoint — a short-lived session built from that endpoint's configuration: connect, read `i=2267`, close, bounded by `ServiceLevelProbeTimeout` (15 s). It fails over only when the other endpoint answers with a **Healthy level (≥ 200) higher than the connected one**. Otherwise (other endpoint unreachable, or also below 200) it **stays put** and the window re-arms, so the other endpoint is probed at most once per window.
+
+- **Connect-time preference.** The first sample after every (re)connect is evaluated at once, without the window: a connection that comes up on a Degraded endpoint while the other is Healthy moves straight over. Because the trigger needs the connected level below 200, a connection to a 240 Secondary is never moved to a 250 Primary.
+- **Anti-flap.** A ServiceLevel switch is not repeated within the connection's **minimum dwell** (default 60 s), connect-time preference included. Together with the "Healthy and higher" requirement and the unbroken-streak rule (one Healthy sample resets it) this stops a pair that flips roles from bouncing the connection.
+- **Same switch as the connectivity failover.** The switch flips the active endpoint and swaps the adapter through the same helper as the two connection-driven triggers, resets both failure counters, then re-enters the reconnect cycle immediately with the unstable-disconnect accounting suppressed (it is a deliberate swap, like an in-place reconfigure). `ReSubscribeAll` / `ReSubscribeAllAlarms` then re-issue every subscription on the new endpoint; tags go Bad for the swap exactly as on any failover. Site event log: `Failover from Primary to Backup (ServiceLevel)` (Warning) with the two levels.
+- **Scope.** Only adapters implementing `IServiceLevelReadable` (the OPC UA adapter) are watched, and only when the connection has a backup and the setting is enabled. MxGateway connections, single-endpoint connections and disabled connections never read a ServiceLevel. The ServiceLevel read never raises the adapter's `Disconnected` event: a dead session stays the keep-alive's business.
+- **Behaviour against today's servers.** A pair that only ever reports Healthy levels (OtOpcUa 250/240 without Warm, or a server that always reports 255) never starts a streak: no probe, no switch — the default-on setting changes nothing there. It does act on a non-Warm OtOpcUa node that has dropped to 100/110 or 0/10 while its peer is ≥ 200, which is the desired move.
+
+**Configuration** (per connection, on the **primary** endpoint's OPC UA JSON — the backup's copy is ignored; edited in the Central UI's *Backup endpoint* card next to *Failover Retry Count*; via the CLI it is part of `--primary-config`):
+
+```json
+"serviceLevelFailover": { "enabled": true, "degradedWindowSeconds": 10, "minDwellSeconds": 60 }
+```
+
+Absent or `null` means the defaults (enabled, 10 s, 60 s), so stored connections need no migration; the object travels inside the protocol configuration, which is encrypted at rest with the rest of it. Validation: window 1–3600 s, dwell 0–86400 s.
+
+**Alarms across a promotion.** On Standby → Active the promoted OtOpcUa node runs a ConditionRefresh on every live event item on its own (design §4.5). The existing items therefore receive an **unsolicited** RefreshStart, the retained conditions, RefreshEnd. `RealOpcUaClient` frames it like the refresh it triggers on subscribe — `SnapshotStart` / `Snapshot`… / `SnapshotComplete` under a fresh epoch, live derivation resuming after RefreshEnd — and the actor's epoch-scoped snapshot cache replaces the source's state with it, dropping conditions the promoted node no longer retains. Covered by `RealOpcUaPromotionRefreshTests` and `DataConnectionPromotionRefreshTests`.
+
 ### In-place reconfiguration (register row 97)
 
 A `ReconfigureConnectionCommand` replaces a **live** connection's configuration without stopping the connection actor. It is the same adapter swap the failover path above performs — the two share one `SwapAdapter` helper — so the diagram applies verbatim from *dispose adapter* onwards:
@@ -186,6 +214,9 @@ All settings are parsed from the data connection's configuration JSON dictionari
 | `MaxMonitoredItemsPerSubscription` | int | `5000` | Monitored-item budget per OPC UA subscription. Above it the adapter **shards** onto an additional subscription on the same session (37,500 tags → 8 shards). Per endpoint, because item-count ceilings are a property of the target server. |
 | `SecurityMode` | string | `None` | Preferred endpoint security: `None`, `Sign`, or `SignAndEncrypt` |
 | `AutoAcceptUntrustedCerts` | bool | `true` | Accept untrusted server certificates |
+| `ServiceLevelFailover.Enabled` | bool | `true` | ServiceLevel-aware failover (#138). Read from the **primary** endpoint only; needs a backup. JSON: `serviceLevelFailover.enabled` |
+| `ServiceLevelFailover.DegradedWindowSeconds` | int | `10` | How long the connected endpoint's ServiceLevel must stay below 200 before the other endpoint is probed (1–3600) |
+| `ServiceLevelFailover.MinDwellSeconds` | int | `60` | Minimum time between ServiceLevel-driven switches (0–86400) |
 
 ### OPC UA application identity (deployment-wide, `ScadaBridge:OpcUa`)
 
@@ -242,6 +273,8 @@ These are configured via `DataConnectionOptions` in `appsettings.json`, not per-
 | `SubscribeBatchDelay` | 50ms | Delay BETWEEN reconnect re-subscribe chunks. Not applied on the instance-driven subscribe path — the Deployment Manager already staggers instance startup there. |
 | `QualityFlushInterval` | 1s | Coalescing window for pushing tag-quality counters to the health collector |
 | `MxSupervisoryAdviseParallelism` | 16 | Max in-flight supervisory advise commands on the MxGateway bulk-subscribe path when the endpoint has no write-user context |
+| `ServiceLevelPollInterval` | 2s | How often the connected OPC UA endpoint's `Server_ServiceLevel` is read when ServiceLevel-aware failover is in force (#138) |
+| `ServiceLevelProbeTimeout` | 15s | Budget for one ServiceLevel read, and for probing the other endpoint (connect + read); no answer in time = unreachable, no switch |
 | `MxAlarmStreamReopenDebounce` | 200ms | Window over which MxGateway alarm subscribes are coalesced into ONE reopen of the shared alarm stream (0–5 s; `0` = reopen per subscribe, the pre-2026-10-21 behaviour). Every new source still receives a fresh standing-state snapshot. |
 
 ## Subscription Management
@@ -347,6 +380,7 @@ The registered `SourceReference` is a **NodeId** for OPC UA (the picker, CSV and
 - **State gating**: `SubscribeAlarmsRequest` is handled only in the **Connected** state; requests arriving while **Connecting**/**Reconnecting** are coalesced to the latest request per subscriber actor/source and processed on entering Connected. Unsubscribes and subscriber termination are handled in every state.
 - **Capability check**: if `_adapter is not IAlarmSubscribableConnection`, the actor replies `SubscribeAlarmsResponse(Success = false, ...)`.
 - **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state.
+- **Server-initiated refresh**: a RefreshStart/RefreshEnd pair the server sends on its own on an existing feed (OtOpcUa's promotion refresh under Warm redundancy, #138) takes the same snapshot path — forwarded in order under its own epoch, and it replaces the cached state for the source.
 - **Shared condition filter (last-subscriber-wins)**: because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a *different* filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter.
 - **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; snapshot boundaries bypass the condition filter, but OPC UA boundaries are sent only to the exact source binding. Connection-wide Mx boundaries broadcast to all subscribers while preserving their refresh epoch.
 - **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it for each new source, including one covered by the existing prefix, so every newly bound source receives standing state. Restarts are **coalesced** (review 2026-10-21): subscribes arriving within `MxAlarmStreamReopenDebounce` (200 ms) of the first share ONE restart on the union prefix, so a staggered deploy batch of N alarm-bound instances costs one close/reopen and one connection-wide snapshot replay instead of N (each of which was broadcast to every already-subscribed source — O(N·M) boundary messages). A reconnect, teardown or last unsubscribe invalidates a pending restart. Each transport refresh has connection-wide start/end markers and a distinct epoch. An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which may increase bandwidth. Correctness also depends on the epoch-scoped snapshot cache merging concurrent live updates and rejecting stale snapshot ends (DC-5); a broad filter alone provides no such guarantee. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.