feat(dcl): ServiceLevel-aware OPC UA failover for primary/backup pairs (#138)

Prerequisite for OtOpcUa Warm redundancy (lmxopcua#717): a Warm standby keeps a
healthy session but serves BadOutOfService at a standby ServiceLevel (90/50, boot 1),
so a connection that lands there sat on Bad quality forever.

- OPC UA connections with a backup poll Server_ServiceLevel (i=2267) on the connected
  endpoint (IServiceLevelReadable, implemented by OpcUaDataConnection; never raises
  Disconnected). A level < 200 sustained for the degraded window (default 10 s), or
  the first sample after any (re)connect, triggers a probe of the other endpoint
  (short-lived session: connect, read, dispose). Switch only when it reports a
  Healthy (>= 200) higher level, and not within the minimum dwell (default 60 s) of
  the previous ServiceLevel switch. Both degraded, other unreachable or level
  unknown: stay put. 250/240 (today's OtOpcUa) never starts a streak.
- The switch reuses the failover path: SwitchToOtherEndpoint() now shared by all
  three triggers, then BecomeReconnecting(ReconnectCause.ServiceLevelFailover) -
  zero delay, no unstable accounting, ReSubscribeAll on the new endpoint.
- Config: OpcUaEndpointConfig.ServiceLevelFailover {enabled, degradedWindowSeconds,
  minDwellSeconds}, read from the PRIMARY endpoint's JSON only; null = defaults
  (enabled, 10, 60). Inside the encrypted protocol JSON, so no schema change or
  migration. Central UI: fields in the Backup card next to Failover Retry Count.
  Global: ScadaBridge:DataConnection:ServiceLevelPollInterval (2 s) and
  ServiceLevelProbeTimeout (15 s).
- Tests: actor tests per trigger/non-trigger with positive controls, an unsolicited
  promotion RefreshStart/RefreshEnd test at client and actor level, adapter tests,
  and a real in-process OPC UA server pair end to end. All 15 mutants killed.
- Docs: Component-DataConnectionLayer, components/DataConnectionLayer, Component-CLI.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/DataConnectionForm.razor

@@ -165,6 +165,40 @@
                                min="1" max="20" @bind="_formFailoverRetryCount" />
                         <div class="form-text">Retries before failing over to backup endpoint.</div>
                     </div>
+                    @if (_protocol != "MxGateway")
+                    {
+                        <div class="mb-2" data-test="servicelevel-failover">
+                            <div class="form-check">
+                                <input type="checkbox" class="form-check-input" id="sl-failover-enabled"
+                                       @bind="_formSlEnabled" />
+                                <label class="form-check-label small" for="sl-failover-enabled">
+                                    ServiceLevel failover
+                                </label>
+                            </div>
+                            <div class="form-text">
+                                Fail over when the connected endpoint's ServiceLevel stays below 200 (Degraded)
+                                for the window below and the other endpoint reports a Healthy (200+), higher level.
+                                Needed for an OtOpcUa pair in Warm redundancy; a 250/240 pair never triggers it.
+                            </div>
+                            @if (_formSlEnabled)
+                            {
+                                <div class="row g-2 mt-1">
+                                    <div class="col-auto">
+                                        <label class="form-label small" for="sl-failover-window">Degraded window (s)</label>
+                                        <input type="number" id="sl-failover-window" class="form-control form-control-sm"
+                                               style="max-width: 120px;" min="1" max="3600" @bind="_formSlWindowSeconds" />
+                                        @RenderPrimaryError("ServiceLevelFailover.DegradedWindowSeconds")
+                                    </div>
+                                    <div class="col-auto">
+                                        <label class="form-label small" for="sl-failover-dwell">Minimum dwell (s)</label>
+                                        <input type="number" id="sl-failover-dwell" class="form-control form-control-sm"
+                                               style="max-width: 120px;" min="0" max="86400" @bind="_formSlDwellSeconds" />
+                                        @RenderPrimaryError("ServiceLevelFailover.MinDwellSeconds")
+                                    </div>
+                                </div>
+                            }
+                        </div>
+                    }
                     <div class="mb-0">
                         <button type="button" class="btn btn-outline-danger btn-sm"
                                 @onclick="RemoveBackup">Remove Backup</button>
@@ -207,6 +241,10 @@
     private bool _backupIsLegacy;
     private bool _showBackup;
     private int _formFailoverRetryCount = 3;
+    // ScadaBridge#138: connection-level ServiceLevel failover, stored on the PRIMARY OPC UA endpoint.
+    private bool _formSlEnabled = true;
+    private int _formSlWindowSeconds = OpcUaServiceLevelFailoverConfig.DefaultDegradedWindowSeconds;
+    private int _formSlDwellSeconds = OpcUaServiceLevelFailoverConfig.DefaultMinDwellSeconds;
     private ValidationResult? _primaryErrors;
     private ValidationResult? _backupErrors;
     private string? _formError;
@@ -276,6 +314,10 @@
         {
             (_primaryConfig, _primaryIsLegacy) =
                 OpcUaEndpointConfigSerializer.Deserialize(conn.PrimaryConfiguration);
+            var sl = _primaryConfig.ServiceLevelFailover ?? new OpcUaServiceLevelFailoverConfig();
+            _formSlEnabled = sl.Enabled;
+            _formSlWindowSeconds = sl.DegradedWindowSeconds;
+            _formSlDwellSeconds = sl.MinDwellSeconds;
             if (!string.IsNullOrWhiteSpace(conn.BackupConfiguration))
             {
                 (_backupConfig, _backupIsLegacy) =
@@ -313,6 +355,15 @@
         }
         else
         {
+            // Only meaningful with a backup; without one the setting is not stored at all.
+            _primaryConfig.ServiceLevelFailover = _showBackup
+                ? new OpcUaServiceLevelFailoverConfig
+                {
+                    Enabled = _formSlEnabled,
+                    DegradedWindowSeconds = _formSlWindowSeconds,
+                    MinDwellSeconds = _formSlDwellSeconds,
+                }
+                : null;
             _primaryErrors = OpcUaEndpointConfigValidator.Validate(_primaryConfig, "Primary.");
             _backupErrors = _showBackup
                 ? OpcUaEndpointConfigValidator.Validate(_backupConfig, "Backup.")
@@ -350,6 +401,17 @@
         _backupMx = new MxGatewayEndpointConfig();
         _backupIsLegacy = false;
         _formFailoverRetryCount = 3;
+        _formSlEnabled = true;
+        _formSlWindowSeconds = OpcUaServiceLevelFailoverConfig.DefaultDegradedWindowSeconds;
+        _formSlDwellSeconds = OpcUaServiceLevelFailoverConfig.DefaultMinDwellSeconds;
+    }
+
+    private RenderFragment? RenderPrimaryError(string field)
+    {
+        var match = _primaryErrors?.Errors.FirstOrDefault(e => e.EntityName == "Primary." + field);
+        return match is null
+            ? null
+            : @<div class="text-danger small">@match.Message</div>;
     }
 
     private void GoBack() => NavigationManager.NavigateTo("/design/connections");

src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Protocol/IServiceLevelReadable.cs

@@ -0,0 +1,22 @@
+namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
+
+/// <summary>
+/// Optional capability for an <see cref="IDataConnection"/> whose server publishes a redundancy
+/// ServiceLevel (OPC UA <c>Server_ServiceLevel</c>, <c>i=2267</c>). The DataConnectionActor polls it
+/// on the connected endpoint, and probes the other endpoint with it, to drive ServiceLevel-aware
+/// failover (ScadaBridge#138). An adapter without the capability is never switched on ServiceLevel.
+/// </summary>
+public interface IServiceLevelReadable
+{
+    /// <summary>
+    /// Reads the server's current ServiceLevel. Never raises the connection's
+    /// <see cref="IDataConnection.Disconnected"/> event: a lost session is the reconnect machine's
+    /// business, not this read's.
+    /// </summary>
+    /// <param name="cancellationToken">Bounds the read.</param>
+    /// <returns>
+    /// The ServiceLevel (0–255), or null when it is unknown: not connected, a Bad status, a value
+    /// that is not a byte, or a failed read.
+    /// </returns>
+    Task<byte?> ReadServiceLevelAsync(CancellationToken cancellationToken = default);
+}

src/ZB.MOM.WW.ScadaBridge.Commons/Serialization/OpcUaEndpointConfigSerializer.cs

@@ -83,6 +83,15 @@ public readonly record struct OpcUaConfigParseResult
 /// </summary>
 public static class OpcUaEndpointConfigSerializer
 {
+    /// <summary>Flat-dict key for <see cref="OpcUaServiceLevelFailoverConfig.Enabled"/>.</summary>
+    public const string ServiceLevelFailoverEnabledKey = "ServiceLevelFailover.Enabled";
+
+    /// <summary>Flat-dict key for <see cref="OpcUaServiceLevelFailoverConfig.DegradedWindowSeconds"/>.</summary>
+    public const string ServiceLevelFailoverWindowKey = "ServiceLevelFailover.DegradedWindowSeconds";
+
+    /// <summary>Flat-dict key for <see cref="OpcUaServiceLevelFailoverConfig.MinDwellSeconds"/>.</summary>
+    public const string ServiceLevelFailoverDwellKey = "ServiceLevelFailover.MinDwellSeconds";
+
     private static readonly JsonSerializerOptions JsonOpts = new()
     {
         PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
@@ -216,6 +225,12 @@ public static class OpcUaEndpointConfigSerializer
             dict["Deadband.Type"] = db.Type.ToString();
             dict["Deadband.Value"] = db.Value.ToString(System.Globalization.CultureInfo.InvariantCulture);
         }
+        if (config.ServiceLevelFailover is { } sl)
+        {
+            dict[ServiceLevelFailoverEnabledKey] = sl.Enabled.ToString();
+            dict[ServiceLevelFailoverWindowKey] = sl.DegradedWindowSeconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
+            dict[ServiceLevelFailoverDwellKey] = sl.MinDwellSeconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
+        }
         return dict;
     }
 
@@ -291,6 +306,18 @@ public static class OpcUaEndpointConfigSerializer
             c.Deadband = db;
         }
 
+        if (dict.ContainsKey(ServiceLevelFailoverEnabledKey)
+            || dict.ContainsKey(ServiceLevelFailoverWindowKey)
+            || dict.ContainsKey(ServiceLevelFailoverDwellKey))
+        {
+            var sl = new OpcUaServiceLevelFailoverConfig();
+            if (dict.TryGetValue(ServiceLevelFailoverEnabledKey, out var slEn) && bool.TryParse(slEn, out var slEnabled))
+                sl.Enabled = slEnabled;
+            TryAssignInt(dict, ServiceLevelFailoverWindowKey, v => sl.DegradedWindowSeconds = v);
+            TryAssignInt(dict, ServiceLevelFailoverDwellKey, v => sl.MinDwellSeconds = v);
+            c.ServiceLevelFailover = sl;
+        }
+
         return c;
     }
 

src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaEndpointConfig.cs

@@ -96,4 +96,12 @@ public sealed class OpcUaEndpointConfig
     /// Heartbeat configuration for connection monitoring.
     /// </summary>
     public OpcUaHeartbeatConfig? Heartbeat { get; set; }
+
+    // Redundancy (optional; null = defaults, see OpcUaServiceLevelFailoverConfig)
+    /// <summary>
+    /// ServiceLevel-aware failover settings (ScadaBridge#138). Connection-level: only the primary
+    /// endpoint's value is read, and only when the connection has a backup. Null means the
+    /// defaults (enabled, 10 s window, 60 s dwell).
+    /// </summary>
+    public OpcUaServiceLevelFailoverConfig? ServiceLevelFailover { get; set; }
 }

src/ZB.MOM.WW.ScadaBridge.Commons/Types/DataConnections/OpcUaServiceLevelFailoverConfig.cs

@@ -0,0 +1,46 @@
+namespace ZB.MOM.WW.ScadaBridge.Commons.Types.DataConnections;
+
+/// <summary>
+/// ServiceLevel-aware failover for an OPC UA connection with a primary AND a backup endpoint
+/// (ScadaBridge#138; OtOpcUa Warm redundancy, lmxopcua#717). A connection-level setting: only the
+/// <b>primary</b> endpoint's copy is read, and it has no effect on a connection without a backup.
+///
+/// <para>
+/// The connected endpoint's <c>Server_ServiceLevel</c> (<c>i=2267</c>) is polled. When it stays in
+/// the Degraded band (below 200, OPC UA Part 4 §6.6.2.4.2) for <see cref="DegradedWindowSeconds"/>
+/// and the other endpoint is reachable and reports a Healthy-band level (200 or more) that is also
+/// higher, the connection fails over to it. A switch is not repeated within
+/// <see cref="MinDwellSeconds"/>. Both endpoints degraded, or the other one unreachable, means the
+/// connection stays where it is. On connect, a Degraded endpoint is left immediately (no window) for
+/// a Healthy, higher other endpoint, subject to the same dwell.
+/// </para>
+///
+/// <para>
+/// A stored configuration without this object behaves as the defaults: enabled, 10 s window,
+/// 60 s dwell. With today's OtOpcUa pair (250 Primary / 240 Secondary, both Healthy band) it never
+/// triggers.
+/// </para>
+/// </summary>
+public sealed class OpcUaServiceLevelFailoverConfig
+{
+    /// <summary>Default for <see cref="DegradedWindowSeconds"/>.</summary>
+    public const int DefaultDegradedWindowSeconds = 10;
+
+    /// <summary>Default for <see cref="MinDwellSeconds"/>.</summary>
+    public const int DefaultMinDwellSeconds = 60;
+
+    /// <summary>Whether ServiceLevel can trigger a failover. Default true.</summary>
+    public bool Enabled { get; set; } = true;
+
+    /// <summary>
+    /// How long, in seconds, the connected endpoint must report a Degraded-band ServiceLevel
+    /// without a break before the other endpoint is checked. Default 10.
+    /// </summary>
+    public int DegradedWindowSeconds { get; set; } = DefaultDegradedWindowSeconds;
+
+    /// <summary>
+    /// Minimum time, in seconds, after a ServiceLevel-driven switch before another one may happen
+    /// (anti-flap). Default 60.
+    /// </summary>
+    public int MinDwellSeconds { get; set; } = DefaultMinDwellSeconds;
+}

src/ZB.MOM.WW.ScadaBridge.Commons/Validators/OpcUaEndpointConfigValidator.cs

@@ -74,6 +74,14 @@ public static class OpcUaEndpointConfigValidator
         if (config.Deadband is { } db && db.Value <= 0)
             errors.Add(Err("Deadband.Value", "Must be > 0."));
 
+        if (config.ServiceLevelFailover is { } sl)
+        {
+            if (sl.DegradedWindowSeconds < 1 || sl.DegradedWindowSeconds > 3600)
+                errors.Add(Err("ServiceLevelFailover.DegradedWindowSeconds", "Must be between 1 and 3600."));
+            if (sl.MinDwellSeconds < 0 || sl.MinDwellSeconds > 86400)
+                errors.Add(Err("ServiceLevelFailover.MinDwellSeconds", "Must be between 0 and 86400."));
+        }
+
         return errors.Count == 0
             ? ValidationResult.Success()
             : ValidationResult.FromErrors(errors.ToArray());

src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.ServiceLevel.cs

@@ -0,0 +1,320 @@
+using System.Globalization;
+using Akka.Actor;
+using Akka.Event;
+using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
+using ZB.MOM.WW.ScadaBridge.Commons.Serialization;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.DataConnections;
+
+namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
+
+/// <summary>
+/// ScadaBridge#138 — ServiceLevel-aware failover between a primary and a backup endpoint.
+///
+/// <para>
+/// <b>Why.</b> Until now the connection failed over only when the link itself misbehaved
+/// (<c>FailoverRetryCount</c> consecutive connect failures or unstable disconnects). Under OtOpcUa
+/// Warm redundancy (lmxopcua#717) the standby keeps a perfectly healthy session but serves
+/// <c>BadOutOfService</c> data at a standby ServiceLevel (90 healthy / 50 degraded / 1 booting), so a
+/// connection that lands there would sit on Bad quality indefinitely.
+/// </para>
+///
+/// <para>
+/// <b>What.</b> While Connected, the connected endpoint's <c>Server_ServiceLevel</c> is polled every
+/// <see cref="DataConnectionOptions.ServiceLevelPollInterval"/>. A level below
+/// <see cref="HealthyServiceLevelMinimum"/> (200, the bottom of the Part 4 Healthy band) starts a
+/// degraded streak; any Healthy or unreadable sample ends it. Once the streak has lasted the
+/// connection's degraded window — or immediately on the first sample after a (re)connect, which is
+/// the "prefer the better endpoint on connect" rule — the other endpoint is probed with a
+/// short-lived session. The connection switches only when that probe reads a Healthy level that is
+/// also higher than the connected one, and never within the dwell time of the previous
+/// ServiceLevel switch. Otherwise it stays put and the window re-arms.
+/// </para>
+///
+/// <para>
+/// <b>Unchanged behaviour.</b> A level of 200 or more never starts a streak, so today's OtOpcUa pair
+/// (250 Primary / 240 Secondary) never triggers anything; an adapter without
+/// <see cref="IServiceLevelReadable"/> (MxGateway) and a connection without a backup are never
+/// monitored at all. The switch itself reuses <see cref="SwitchToOtherEndpoint"/> and the reconnect
+/// cycle, exactly like the connection-driven failovers.
+/// </para>
+/// </summary>
+public partial class DataConnectionActor
+{
+    /// <summary>The lowest ServiceLevel in the OPC UA Healthy band (Part 4 §6.6.2.4.2).</summary>
+    internal const byte HealthyServiceLevelMinimum = 200;
+
+    private const string ServiceLevelPollTimerKey = "servicelevel-poll";
+
+    private ServiceLevelFailoverPolicy _serviceLevelPolicy;
+
+    /// <summary>When the current unbroken run of Degraded samples began; null when not degraded.</summary>
+    private DateTimeOffset? _degradedSince;
+
+    /// <summary>When the last ServiceLevel-driven switch happened (dwell / anti-flap).</summary>
+    private DateTimeOffset _lastServiceLevelSwitchAt = DateTimeOffset.MinValue;
+
+    private bool _serviceLevelReadInFlight;
+    private bool _serviceLevelProbeInFlight;
+
+    /// <summary>True until the first sample of a session is evaluated (connect-time preference).</summary>
+    private bool _serviceLevelConnectCheckPending;
+
+    /// <summary>Why <see cref="BecomeReconnecting"/> is re-establishing the link.</summary>
+    internal enum ReconnectCause
+    {
+        /// <summary>The link was lost (disconnect, connection-level failure).</summary>
+        ConnectionLost,
+
+        /// <summary>An in-place reconfiguration swapped the adapter.</summary>
+        Reconfigured,
+
+        /// <summary>A sustained Degraded ServiceLevel swapped to the other endpoint (#138).</summary>
+        ServiceLevelFailover,
+    }
+
+    /// <summary>
+    /// True when this connection's ServiceLevel is watched: the policy is enabled, there is a
+    /// backup to go to, and the live adapter can read a ServiceLevel.
+    /// </summary>
+    private bool ServiceLevelFailoverInForce =>
+        _serviceLevelPolicy.Enabled && _backupConfig != null && _adapter is IServiceLevelReadable;
+
+    /// <summary>Called on entering Connected. Arms the poll (first tick immediately).</summary>
+    private void StartServiceLevelMonitoring()
+    {
+        ResetServiceLevelSession();
+        if (!ServiceLevelFailoverInForce)
+            return;
+
+        _serviceLevelConnectCheckPending = true;
+        Timers.StartPeriodicTimer(ServiceLevelPollTimerKey, ServiceLevelPollTick.Instance,
+            TimeSpan.Zero, _options.ServiceLevelPollInterval);
+    }
+
+    /// <summary>Called on leaving Connected. Results still in flight are fenced by generation.</summary>
+    private void StopServiceLevelMonitoring()
+    {
+        Timers.Cancel(ServiceLevelPollTimerKey);
+        ResetServiceLevelSession();
+    }
+
+    private void ResetServiceLevelSession()
+    {
+        _degradedSince = null;
+        _serviceLevelReadInFlight = false;
+        _serviceLevelProbeInFlight = false;
+        _serviceLevelConnectCheckPending = false;
+    }
+
+    private void HandleServiceLevelPollTick()
+    {
+        // One read at a time, and none while the other endpoint is being probed: the probe's
+        // verdict is taken against the level it started from.
+        if (!ServiceLevelFailoverInForce || _serviceLevelReadInFlight || _serviceLevelProbeInFlight)
+            return;
+
+        _serviceLevelReadInFlight = true;
+        var generation = _adapterGeneration;
+        var reader = (IServiceLevelReadable)_adapter;
+        var timeout = _options.ServiceLevelProbeTimeout;
+        Task.Run(() => reader.ReadServiceLevelAsync().WaitAsync(timeout))
+            .ContinueWith(t => new ServiceLevelSampled(generation,
+                t.IsCompletedSuccessfully ? t.Result : null), TaskScheduler.Default)
+            .PipeTo(_self);
+    }
+
+    private void HandleServiceLevelSampled(ServiceLevelSampled sampled)
+    {
+        if (sampled.AdapterGeneration != _adapterGeneration)
+            return;
+
+        _serviceLevelReadInFlight = false;
+        var connectCheck = _serviceLevelConnectCheckPending;
+        _serviceLevelConnectCheckPending = false;
+
+        if (sampled.Level is not { } level || level >= HealthyServiceLevelMinimum)
+        {
+            // Healthy, or unknown: either way the run of Degraded samples is broken. Unknown never
+            // counts toward a switch — acting on missing information is how a pair flaps.
+            if (_degradedSince != null)
+                _log.Info("[{0}] {1} endpoint ServiceLevel is {2} — degraded watch cleared",
+                    _connectionName, _activeEndpoint, sampled.Level?.ToString(CultureInfo.InvariantCulture) ?? "unknown");
+            _degradedSince = null;
+            return;
+        }
+
+        var now = DateTimeOffset.UtcNow;
+        if (_degradedSince is null)
+        {
+            _degradedSince = now;
+            _log.Warning("[{0}] {1} endpoint reports ServiceLevel {2} (Degraded band, < {3}) — "
+                + "will check the other endpoint {4}",
+                _connectionName, _activeEndpoint, level, HealthyServiceLevelMinimum,
+                connectCheck ? "now (connect-time preference)"
+                    : $"if it persists for {_serviceLevelPolicy.DegradedWindow.TotalSeconds:F0}s");
+        }
+
+        if (!connectCheck && now - _degradedSince.Value < _serviceLevelPolicy.DegradedWindow)
+            return;
+
+        if (now - _lastServiceLevelSwitchAt < _serviceLevelPolicy.MinDwell)
+        {
+            _log.Debug("[{0}] ServiceLevel {1} still Degraded but the last ServiceLevel switch was {2:F0}s ago "
+                + "(dwell {3:F0}s) — staying", _connectionName, level,
+                (now - _lastServiceLevelSwitchAt).TotalSeconds, _serviceLevelPolicy.MinDwell.TotalSeconds);
+            return;
+        }
+
+        StartServiceLevelProbe(level);
+    }
+
+    /// <summary>
+    /// Reads the OTHER endpoint's ServiceLevel through a short-lived adapter built from its
+    /// configuration — connect, read, dispose — off the actor thread, bounded by
+    /// <see cref="DataConnectionOptions.ServiceLevelProbeTimeout"/>. The live adapter is untouched.
+    /// </summary>
+    private void StartServiceLevelProbe(byte currentLevel)
+    {
+        _serviceLevelProbeInFlight = true;
+        var generation = _adapterGeneration;
+        var otherConfig = _activeEndpoint == ActiveEndpoint.Primary ? _backupConfig! : _primaryConfig;
+        var factory = _factory;
+        var protocol = _protocolType;
+        var timeout = _options.ServiceLevelProbeTimeout;
+
+        Task.Run(() => ProbeServiceLevelAsync(factory, protocol, otherConfig).WaitAsync(timeout))
+            .ContinueWith(t => t.IsCompletedSuccessfully
+                    ? new ServiceLevelProbed(generation, currentLevel, t.Result.Level, t.Result.Error)
+                    : new ServiceLevelProbed(generation, currentLevel, null,
+                        t.Exception?.GetBaseException() is TimeoutException
+                            ? $"no answer within {timeout.TotalSeconds:F0}s"
+                            : t.Exception?.GetBaseException().Message ?? "probe cancelled"),
+                TaskScheduler.Default)
+            .PipeTo(_self);
+    }
+
+    private static async Task<(byte? Level, string? Error)> ProbeServiceLevelAsync(
+        IDataConnectionFactory factory, string protocol, IDictionary<string, string> config)
+    {
+        IDataConnection? probe = null;
+        try
+        {
+            probe = factory.Create(protocol, config);
+            if (probe is not IServiceLevelReadable reader)
+                return (null, "the adapter cannot read a ServiceLevel");
+
+            await probe.ConnectAsync(config).ConfigureAwait(false);
+            var level = await reader.ReadServiceLevelAsync().ConfigureAwait(false);
+            return (level, level is null ? "ServiceLevel unreadable" : null);
+        }
+        catch (Exception ex)
+        {
+            return (null, ex.GetBaseException().Message);
+        }
+        finally
+        {
+            // Fire-and-forget: closing a session against a black-holed server can block for the
+            // operation timeout, and the probe's verdict must not wait for it.
+            if (probe != null)
+                _ = probe.DisposeAsync().AsTask().ContinueWith(
+                    t => _ = t.Exception, TaskContinuationOptions.OnlyOnFaulted);
+        }
+    }
+
+    private void HandleServiceLevelProbed(ServiceLevelProbed probed)
+    {
+        if (probed.AdapterGeneration != _adapterGeneration)
+            return;
+
+        _serviceLevelProbeInFlight = false;
+        if (!ServiceLevelFailoverInForce || _degradedSince is null)
+            return;
+
+        var other = _activeEndpoint == ActiveEndpoint.Primary ? ActiveEndpoint.Backup : ActiveEndpoint.Primary;
+        var now = DateTimeOffset.UtcNow;
+
+        if (probed.OtherLevel is not { } otherLevel)
+        {
+            _log.Warning("[{0}] {1} endpoint ServiceLevel {2} is Degraded but the {3} endpoint is not "
+                + "available ({4}) — staying", _connectionName, _activeEndpoint, probed.CurrentLevel, other,
+                probed.Error);
+            _degradedSince = now; // re-arm: look again after another full window
+            return;
+        }
+
+        if (otherLevel < HealthyServiceLevelMinimum || otherLevel <= probed.CurrentLevel)
+        {
+            _log.Warning("[{0}] {1} endpoint ServiceLevel {2} is Degraded and the {3} endpoint reports {4} "
+                + "(not a Healthy, higher level) — staying", _connectionName, _activeEndpoint,
+                probed.CurrentLevel, other, otherLevel);
+            _degradedSince = now;
+            return;
+        }
+
+        var degradedFor = now - _degradedSince.Value;
+        var previousEndpoint = SwitchToOtherEndpoint();
+        _consecutiveFailures = 0;
+        _consecutiveUnstableDisconnects = 0;
+        _lastServiceLevelSwitchAt = now;
+
+        _log.Warning("[{0}] Failing over from {1} to {2} (ServiceLevel {3} Degraded for {4:F0}s; {2} reports {5})",
+            _connectionName, previousEndpoint, _activeEndpoint, probed.CurrentLevel,
+            degradedFor.TotalSeconds, otherLevel);
+
+        if (_siteEventLogger != null)
+        {
+            _ = _siteEventLogger.LogEventAsync(
+                "connection", "Warning", null, _connectionName,
+                $"Failover from {previousEndpoint} to {_activeEndpoint} (ServiceLevel)",
+                $"{previousEndpoint} reported ServiceLevel {probed.CurrentLevel} for {degradedFor.TotalSeconds:F0}s; "
+                + $"{_activeEndpoint} reports {otherLevel}");
+        }
+
+        BecomeReconnecting(ReconnectCause.ServiceLevelFailover);
+    }
+
+    /// <summary>Poll tick for the connected endpoint's ServiceLevel.</summary>
+    internal sealed record ServiceLevelPollTick
+    {
+        /// <summary>The single instance.</summary>
+        public static readonly ServiceLevelPollTick Instance = new();
+    }
+
+    /// <summary>One ServiceLevel sample of the connected endpoint (null = unknown).</summary>
+    internal sealed record ServiceLevelSampled(int AdapterGeneration, byte? Level);
+
+    /// <summary>The other endpoint's ServiceLevel (null = unreachable / unreadable, see Error).</summary>
+    internal sealed record ServiceLevelProbed(int AdapterGeneration, byte CurrentLevel, byte? OtherLevel, string? Error);
+}
+
+/// <summary>
+/// The connection-level ServiceLevel failover settings, read from the PRIMARY endpoint's flattened
+/// configuration (<see cref="OpcUaServiceLevelFailoverConfig"/>). Absent keys mean the defaults.
+/// </summary>
+internal readonly record struct ServiceLevelFailoverPolicy(bool Enabled, TimeSpan DegradedWindow, TimeSpan MinDwell)
+{
+    /// <summary>Parses the policy; malformed or out-of-range values fall back to safe values.</summary>
+    /// <param name="primaryConfig">The primary endpoint's flattened configuration.</param>
+    /// <returns>The policy in force.</returns>
+    public static ServiceLevelFailoverPolicy From(IDictionary<string, string>? primaryConfig)
+    {
+        var enabled = true;
+        var window = OpcUaServiceLevelFailoverConfig.DefaultDegradedWindowSeconds;
+        var dwell = OpcUaServiceLevelFailoverConfig.DefaultMinDwellSeconds;
+
+        if (primaryConfig != null)
+        {
+            if (primaryConfig.TryGetValue(OpcUaEndpointConfigSerializer.ServiceLevelFailoverEnabledKey, out var e)
+                && bool.TryParse(e, out var parsedEnabled))
+                enabled = parsedEnabled;
+            if (primaryConfig.TryGetValue(OpcUaEndpointConfigSerializer.ServiceLevelFailoverWindowKey, out var w)
+                && int.TryParse(w, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsedWindow))
+                window = Math.Max(1, parsedWindow);
+            if (primaryConfig.TryGetValue(OpcUaEndpointConfigSerializer.ServiceLevelFailoverDwellKey, out var d)
+                && int.TryParse(d, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsedDwell))
+                dwell = Math.Max(0, parsedDwell);
+        }
+
+        return new ServiceLevelFailoverPolicy(enabled, TimeSpan.FromSeconds(window), TimeSpan.FromSeconds(dwell));
+    }
+}

src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs

@@ -29,7 +29,7 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
 /// Health reporting (connection status + tag resolution counts).
 /// Subscription lifecycle (register on create, cleanup on stop).
 /// </summary>
-public class DataConnectionActor : UntypedActor, IWithTimers
+public partial class DataConnectionActor : UntypedActor, IWithTimers
 {
     public enum ActiveEndpoint { Primary, Backup }
 
@@ -328,6 +328,7 @@ public class DataConnectionActor : UntypedActor, IWithTimers
         _siteEventLogger = siteEventLogger;
         _connectionDetails = _primaryConfig;
         _tagResolutionInterval = _options.TagResolutionRetryInterval;
+        _serviceLevelPolicy = ServiceLevelFailoverPolicy.From(_primaryConfig);
     }
 
     /// <summary>
@@ -491,6 +492,9 @@ public class DataConnectionActor : UntypedActor, IWithTimers
                 // No session yet — a probe would fail for every tag. ReSubscribeAll's chunk
                 // completions re-arm the timer once the link is up.
                 break;
+            case ServiceLevelPollTick or ServiceLevelSampled or ServiceLevelProbed:
+                // ScadaBridge#138: belongs to a session that is over; BecomeConnected restarts it.
+                break;
             case QualityFlushTick:
                 FlushQualityCountersIfDirty();
                 break;
@@ -538,6 +542,8 @@ public class DataConnectionActor : UntypedActor, IWithTimers
         // Register row 54 — the only transition that raises the connected gauge.
         ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Connected);
         Become(Connected);
+        // ScadaBridge#138: watch this endpoint's ServiceLevel (no-op unless in force).
+        StartServiceLevelMonitoring();
         foreach (var pending in _pendingTagSubscriptions.Values)
             Self.Tell(pending.Request, pending.Owner);
         _pendingTagSubscriptions.Clear();
@@ -592,6 +598,15 @@ public class DataConnectionActor : UntypedActor, IWithTimers
             case QualityFlushTick:
                 FlushQualityCountersIfDirty();
                 break;
+            case ServiceLevelPollTick:
+                HandleServiceLevelPollTick();
+                break;
+            case ServiceLevelSampled sampled:
+                HandleServiceLevelSampled(sampled);
+                break;
+            case ServiceLevelProbed probed:
+                HandleServiceLevelProbed(probed);
+                break;
             case AdapterDisconnected disconnected:
                 // Fence the disconnect on its adapter generation. The outgoing adapter's
                 // event can fire concurrently with SwapAdapter's unhook, so a Disconnected from
@@ -629,28 +644,36 @@ public class DataConnectionActor : UntypedActor, IWithTimers
 
     // ── Reconnecting State ──
 
-    /// <param name="afterReconfigure">
-    /// True when this transition was caused by an in-place reconfiguration
-    /// (<see cref="HandleReconfigure"/>) rather than by losing the link. It suppresses the
-    /// unstable-disconnect accounting and the failover check — the adapter has just been
-    /// swapped ON PURPOSE against a brand-new configuration, so counting it as an unstable
-    /// cycle would let a reconfigure trip a failover onto the new backup before the new primary
-    /// has been tried even once — and it reconnects immediately instead of waiting out
+    /// <param name="cause">
+    /// Why the link is being re-established. <see cref="ReconnectCause.Reconfigured"/> (an
+    /// in-place reconfiguration, <see cref="HandleReconfigure"/>) and
+    /// <see cref="ReconnectCause.ServiceLevelFailover"/> (ScadaBridge#138) both mean the adapter
+    /// has just been swapped ON PURPOSE rather than the link lost. They suppress the
+    /// unstable-disconnect accounting and the failover check — counting a deliberate swap as an
+    /// unstable cycle would let it trip a second, connection-driven failover before the new
+    /// endpoint has been tried even once — and reconnect immediately instead of waiting out
     /// <see cref="DataConnectionOptions.ReconnectInterval"/>, since there is nothing to wait for.
     /// </param>
-    private void BecomeReconnecting(bool afterReconfigure = false)
+    private void BecomeReconnecting(ReconnectCause cause = ReconnectCause.ConnectionLost)
     {
+        var deliberateSwap = cause != ReconnectCause.ConnectionLost;
         _log.Warning("[{0}] Entering Reconnecting state{1}", _connectionName,
-            afterReconfigure ? " (reconfigured in place)" : string.Empty);
-        // The session is over: abandon its re-seed (02/L4).
+            cause switch
+            {
+                ReconnectCause.Reconfigured => " (reconfigured in place)",
+                ReconnectCause.ServiceLevelFailover => " (ServiceLevel failover)",
+                _ => string.Empty,
+            });
+        // The session is over: abandon its re-seed (02/L4) and its ServiceLevel watch (#138).
         CancelSessionWork();
+        StopServiceLevelMonitoring();
         _healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Disconnected);
         _healthCollector.UpdateConnectionEndpoint(_connectionName, "Disconnected");
         // Register row 54 — drops this connection out of the connected gauge while the
         // total holds, which is exactly the connected < total alert condition.
         ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Disconnected);
 
-        if (!afterReconfigure)
+        if (!deliberateSwap)
         {
             // Track unstable connections toward failover.
             // If we were connected for less than the stability threshold, this counts
@@ -671,15 +694,10 @@ public class DataConnectionActor : UntypedActor, IWithTimers
             // Failover if we keep connecting and going stale repeatedly
             if (_backupConfig != null && _consecutiveUnstableDisconnects >= _failoverRetryCount)
             {
-                var previousEndpoint = _activeEndpoint;
-                _activeEndpoint = _activeEndpoint == ActiveEndpoint.Primary
-                    ? ActiveEndpoint.Backup
-                    : ActiveEndpoint.Primary;
+                var previousEndpoint = SwitchToOtherEndpoint();
                 _consecutiveUnstableDisconnects = 0;
                 _consecutiveFailures = 0;
 
-                SwapAdapter(_activeEndpoint == ActiveEndpoint.Primary ? _primaryConfig : _backupConfig);
-
                 _log.Warning("[{0}] Failing over from {1} to {2} (unstable connection pattern)",
                     _connectionName, previousEndpoint, _activeEndpoint);
 
@@ -697,10 +715,13 @@ public class DataConnectionActor : UntypedActor, IWithTimers
         if (_siteEventLogger != null)
         {
             _ = _siteEventLogger.LogEventAsync(
-                "connection", afterReconfigure ? "Info" : "Warning", null, _connectionName,
-                afterReconfigure
-                    ? "Connection reconfigured — reconnecting on the new configuration"
-                    : "Connection lost — entering reconnect cycle",
+                "connection", cause == ReconnectCause.ConnectionLost ? "Warning" : "Info", null, _connectionName,
+                cause switch
+                {
+                    ReconnectCause.Reconfigured => "Connection reconfigured — reconnecting on the new configuration",
+                    ReconnectCause.ServiceLevelFailover => $"ServiceLevel failover — reconnecting on the {_activeEndpoint} endpoint",
+                    _ => "Connection lost — entering reconnect cycle",
+                },
                 null);
         }
 
@@ -716,7 +737,25 @@ public class DataConnectionActor : UntypedActor, IWithTimers
         // Schedule reconnect attempt. StartSingleTimer REPLACES any reconnect already armed,
         // so a reconfigure landing mid-backoff cannot leave two AttemptConnect messages racing.
         Timers.StartSingleTimer("reconnect", new AttemptConnect(),
-            afterReconfigure ? TimeSpan.Zero : _options.ReconnectInterval);
+            deliberateSwap ? TimeSpan.Zero : _options.ReconnectInterval);
+    }
+
+    /// <summary>
+    /// Flips <see cref="_activeEndpoint"/> to the other configured endpoint and swaps the adapter
+    /// onto it — the one endpoint switch shared by every failover trigger: consecutive connect
+    /// failures (<see cref="CountFailureAndMaybeFailover"/>), the unstable-connection pattern
+    /// (<see cref="BecomeReconnecting"/>) and a sustained Degraded ServiceLevel (ScadaBridge#138).
+    /// Callers reset whichever counters their trigger owns. Requires a backup configuration.
+    /// </summary>
+    /// <returns>The endpoint that was active before the switch.</returns>
+    private ActiveEndpoint SwitchToOtherEndpoint()
+    {
+        var previousEndpoint = _activeEndpoint;
+        _activeEndpoint = _activeEndpoint == ActiveEndpoint.Primary
+            ? ActiveEndpoint.Backup
+            : ActiveEndpoint.Primary;
+        SwapAdapter(_activeEndpoint == ActiveEndpoint.Primary ? _primaryConfig : _backupConfig!);
+        return previousEndpoint;
     }
 
     /// <summary>
@@ -845,6 +884,8 @@ public class DataConnectionActor : UntypedActor, IWithTimers
         _primaryConfig = command.PrimaryConnectionDetails;
         _backupConfig = command.BackupConnectionDetails;
         _failoverRetryCount = command.FailoverRetryCount;
+        _serviceLevelPolicy = ServiceLevelFailoverPolicy.From(_primaryConfig);
+        _lastServiceLevelSwitchAt = DateTimeOffset.MinValue;
 
         _activeEndpoint = ActiveEndpoint.Primary;
         _consecutiveFailures = 0;
@@ -852,7 +893,7 @@ public class DataConnectionActor : UntypedActor, IWithTimers
 
         SwapAdapter(_primaryConfig);
 
-        BecomeReconnecting(afterReconfigure: true);
+        BecomeReconnecting(ReconnectCause.Reconfigured);
     }
 
     /// <summary>
@@ -930,6 +971,9 @@ public class DataConnectionActor : UntypedActor, IWithTimers
                 // Ignore — the adapter has no live session; ReSubscribeAll's chunk
                 // completions re-arm the retry timer after reconnect.
                 break;
+            case ServiceLevelPollTick or ServiceLevelSampled or ServiceLevelProbed:
+                // ScadaBridge#138: belongs to a session that is over; BecomeConnected restarts it.
+                break;
             case QualityFlushTick:
                 FlushQualityCountersIfDirty();
                 break;
@@ -1219,14 +1263,9 @@ public class DataConnectionActor : UntypedActor, IWithTimers
         // Failover: switch endpoint after exhausting retry count (only if backup is configured)
         if (_backupConfig != null && _consecutiveFailures >= _failoverRetryCount)
         {
-            var previousEndpoint = _activeEndpoint;
-            _activeEndpoint = _activeEndpoint == ActiveEndpoint.Primary
-                ? ActiveEndpoint.Backup
-                : ActiveEndpoint.Primary;
+            var previousEndpoint = SwitchToOtherEndpoint();
             _consecutiveFailures = 0;
 
-            SwapAdapter(_activeEndpoint == ActiveEndpoint.Primary ? _primaryConfig : _backupConfig);
-
             _log.Warning("[{0}] Failing over from {1} to {2}",
                 _connectionName, previousEndpoint, _activeEndpoint);
 

src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs

@@ -21,8 +21,11 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters;
 /// </summary>
 public class OpcUaDataConnection
     : IDataConnection, IBrowsableDataConnection, IAlarmSubscribableConnection, IAddressSpaceSearchable,
-      IBatchSubscribableConnection
+      IBatchSubscribableConnection, IServiceLevelReadable
 {
+    /// <summary>The standard <c>Server_ServiceLevel</c> node (ns=0), OPC UA Part 5.</summary>
+    internal const string ServiceLevelNodeId = "i=2267";
+
     private readonly IOpcUaClientFactory _clientFactory;
     private readonly ILogger<OpcUaDataConnection> _logger;
     private IOpcUaClient? _client;
@@ -291,6 +294,39 @@ public class OpcUaDataConnection
         }
     }
 
+    /// <inheritdoc />
+    /// <remarks>
+    /// ScadaBridge#138. Unlike <see cref="ReadAsync"/>, a failure here never raises
+    /// <see cref="Disconnected"/>: the poll runs every few seconds alongside the keep-alive, and a
+    /// dead session is already detected (and reconnected) there. A failure only means "unknown".
+    /// </remarks>
+    public async Task<byte?> ReadServiceLevelAsync(CancellationToken cancellationToken = default)
+    {
+        var client = _client;
+        if (client == null || !client.IsConnected)
+            return null;
+
+        try
+        {
+            var (value, _, statusCode) = await client.ReadValueAsync(ServiceLevelNodeId, cancellationToken);
+            if (MapStatusCode(statusCode) == QualityCode.Bad)
+                return null;
+            return value switch
+            {
+                byte b => b,
+                sbyte or short or ushort or int or uint or long or ulong
+                    when Convert.ToInt64(value, System.Globalization.CultureInfo.InvariantCulture) is >= 0 and <= 255
+                    => (byte)Convert.ToInt64(value, System.Globalization.CultureInfo.InvariantCulture),
+                _ => null,
+            };
+        }
+        catch (Exception ex) when (ex is not OperationCanceledException)
+        {
+            _logger.LogDebug(ex, "OPC UA ServiceLevel read failed for {Endpoint}", _endpointUrl);
+            return null;
+        }
+    }
+
     /// <inheritdoc />
     public async Task<IReadOnlyDictionary<string, ReadResult>> ReadBatchAsync(IEnumerable<string> tagPaths, CancellationToken cancellationToken = default)
     {

src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptions.cs

@@ -119,4 +119,19 @@ public class DataConnectionOptions
     /// <c>(SeedReadMaxAttempts - 1) × SeedReadRetryDelay</c> regardless of tag count.
     /// </summary>
     public TimeSpan SeedReadRetryDelay { get; set; } = TimeSpan.FromMilliseconds(250);
+
+    /// <summary>
+    /// ScadaBridge#138: how often a connected endpoint's ServiceLevel (<c>i=2267</c>) is read when
+    /// ServiceLevel-aware failover is in force (OPC UA connection with a backup and the
+    /// connection's <c>ServiceLevelFailover.Enabled</c>). One single-node Read per poll. Must be
+    /// shorter than the smallest configured degraded window to make that window meaningful.
+    /// </summary>
+    public TimeSpan ServiceLevelPollInterval { get; set; } = TimeSpan.FromSeconds(2);
+
+    /// <summary>
+    /// ScadaBridge#138: budget for reading a ServiceLevel — the poll on the connected endpoint,
+    /// and the probe of the other endpoint (connect a short-lived session, read, close). A probe
+    /// that does not finish in time counts as "other endpoint unreachable", so no switch happens.
+    /// </summary>
+    public TimeSpan ServiceLevelProbeTimeout { get; set; } = TimeSpan.FromSeconds(15);
 }

src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/DataConnectionOptionsValidator.cs

@@ -64,5 +64,11 @@ public sealed class DataConnectionOptionsValidator : OptionsValidatorBase<DataCo
         builder.RequireThat(options.MxAlarmStreamReopenDebounce >= TimeSpan.Zero
                 && options.MxAlarmStreamReopenDebounce <= TimeSpan.FromSeconds(5),
             $"ScadaBridge:DataConnection:MxAlarmStreamReopenDebounce must be between 0 and 5 seconds (was {options.MxAlarmStreamReopenDebounce}).");
+
+        builder.RequireThat(options.ServiceLevelPollInterval > TimeSpan.Zero,
+            $"ScadaBridge:DataConnection:ServiceLevelPollInterval must be a positive duration (was {options.ServiceLevelPollInterval}).");
+
+        builder.RequireThat(options.ServiceLevelProbeTimeout > TimeSpan.Zero,
+            $"ScadaBridge:DataConnection:ServiceLevelProbeTimeout must be a positive duration (was {options.ServiceLevelProbeTimeout}).");
     }
 }