diff --git a/docs/components/DataConnectionLayer.md b/docs/components/DataConnectionLayer.md index 75982926..a2e982b1 100644 --- a/docs/components/DataConnectionLayer.md +++ b/docs/components/DataConnectionLayer.md @@ -121,6 +121,8 @@ When `BackupConnectionDetails` is provided, the actor tracks consecutive failure **ServiceLevel-aware failover (OPC UA, Gitea #138).** A third trigger, for redundant OPC UA pairs whose standby keeps a healthy session but serves Bad data (OtOpcUa Warm, lmxopcua#717). It lives in `Actors/DataConnectionActor.ServiceLevel.cs`. While Connected, and only when the adapter implements `IServiceLevelReadable`, the connection has a backup and the primary config's `ServiceLevelFailover.Enabled` is true, a periodic timer reads `Server_ServiceLevel` (`i=2267`). A level below 200 starts a degraded streak; a Healthy or unreadable sample ends it. After the degraded window (default 10 s) — or at once on the first sample after a (re)connect — the actor probes the other endpoint with a throwaway adapter from `DataConnectionFactory.Create` (connect, read, dispose; bounded by `ServiceLevelProbeTimeout`). It switches only if that endpoint reports ≥ 200 and higher, and not within the minimum dwell (default 60 s) of the previous ServiceLevel switch; otherwise it stays and re-arms the window. The switch is `SwitchToOtherEndpoint()` — the helper the two connectivity triggers now share — followed by `BecomeReconnecting(ReconnectCause.ServiceLevelFailover)` (zero delay, no unstable accounting). Poll and probe results carry the adapter generation, so a result from a session that has ended is dropped. Today's OtOpcUa (250/240) never starts a streak. Full rule set: `docs/requirements/Component-DataConnectionLayer.md` → *ServiceLevel-aware failover*. +**A server going away (Gitea #141).** Two shortcuts on top of the window, both in the same file and both still bound by "Healthy and higher" and the dwell. (1) A sample of `MaintenanceServiceLevel` (0) probes the other endpoint at once instead of waiting out the window — once per degraded streak (`_goingAwayProbeStarted`), so a peer that is not Healthy is re-probed once per window, not every poll. (2) `BecomeReconnecting` for a lost link captures the session's `_lastKnownServiceLevel` (the last level actually read; unknown samples don't overwrite it) before the watch resets. If that level was below 200, `TryStartDisconnectProbe` probes the other endpoint and **holds** the reconnect timer. `HandleDisconnectProbed` (Reconnecting state; the probe result carries `AfterDisconnect: true`) either switches (`SwitchForServiceLevel`, reconnect at zero delay) or releases the ordinary retry cycle on the same endpoint, less the time the probe took. A link lost after Healthy levels, inside the dwell, or with the feature off takes the ordinary retry cycle as before. Why: an OtOpcUa graceful stop reports 0 for only ~6 s and then the session closes, so on QA the connection spent 66 s retrying the stopped node. + ### Write path Writes are fire-and-forget from the script's view only in the sense that there is no store-and-forward. The `DataConnectionActor` pipes `_adapter.WriteAsync` back to the original sender: diff --git a/docs/requirements/Component-DataConnectionLayer.md b/docs/requirements/Component-DataConnectionLayer.md index 0c6b81a4..20d61d8d 100644 --- a/docs/requirements/Component-DataConnectionLayer.md +++ b/docs/requirements/Component-DataConnectionLayer.md @@ -160,14 +160,19 @@ The failover above is driven only by the link misbehaving. A redundant OPC UA se |---|---|---| | Healthy | 200–255 | Active: 250 Primary / 240 Secondary (today and under Warm), 200 Active-stale | | Degraded | 100–199, and anything below 200 here | Active critically degraded 100; Warm standby 90 (healthy) / 50 (degraded) | -| NoData / Maintenance | 1 / 0 | Warm booting 1; down / endpoint failed 0 | +| NoData / Maintenance | 1 / 0 | Warm booting 1; down / endpoint failed / shutting down 0 (skips the window, #141) | -**Rule.** While Connected, the actor reads the connected endpoint's ServiceLevel every `ServiceLevelPollInterval` (2 s). A level **below 200** starts a *degraded streak*; any level of 200 or more, or an unreadable level (Bad status, no such node, read failure), ends it. When the streak has lasted the connection's **degraded window** (default 10 s), the actor probes the **other** endpoint — a short-lived session built from that endpoint's configuration: connect, read `i=2267`, close, bounded by `ServiceLevelProbeTimeout` (15 s). It fails over only when the other endpoint answers with a **Healthy level (≥ 200) higher than the connected one**. Otherwise (other endpoint unreachable, or also below 200) it **stays put** and the window re-arms, so the other endpoint is probed at most once per window. +**Rule.** While Connected, the actor reads the connected endpoint's ServiceLevel every `ServiceLevelPollInterval` (2 s). A level **below 200** starts a *degraded streak*; any level of 200 or more, or an unreadable level (Bad status, no such node, read failure), ends it. When the streak has lasted the connection's **degraded window** (default 10 s) — or at once for a level of 0, see *A server going away* below — the actor probes the **other** endpoint — a short-lived session built from that endpoint's configuration: connect, read `i=2267`, close, bounded by `ServiceLevelProbeTimeout` (15 s). It fails over only when the other endpoint answers with a **Healthy level (≥ 200) higher than the connected one**. Otherwise (other endpoint unreachable, or also below 200) it **stays put** and the window re-arms, so the other endpoint is probed at most once per window. - **Connect-time preference.** The first sample after every (re)connect is evaluated at once, without the window: a connection that comes up on a Degraded endpoint while the other is Healthy moves straight over. Because the trigger needs the connected level below 200, a connection to a 240 Secondary is never moved to a 250 Primary. - **Anti-flap.** A ServiceLevel switch is not repeated within the connection's **minimum dwell** (default 60 s), connect-time preference included. Together with the "Healthy and higher" requirement and the unbroken-streak rule (one Healthy sample resets it) this stops a pair that flips roles from bouncing the connection. - **Same switch as the connectivity failover.** The switch flips the active endpoint and swaps the adapter through the same helper as the two connection-driven triggers, resets both failure counters, then re-enters the reconnect cycle immediately with the unstable-disconnect accounting suppressed (it is a deliberate swap, like an in-place reconfigure). `ReSubscribeAll` / `ReSubscribeAllAlarms` then re-issue every subscription on the new endpoint; tags go Bad for the swap exactly as on any failover. Site event log: `Failover from Primary to Backup (ServiceLevel)` (Warning) with the two levels. - **Scope.** Only adapters implementing `IServiceLevelReadable` (the OPC UA adapter) are watched, and only when the connection has a backup and the setting is enabled. MxGateway connections, single-endpoint connections and disabled connections never read a ServiceLevel. The ServiceLevel read never raises the adapter's `Disconnected` event: a dead session stays the keep-alive's business. +- **A server going away (Gitea #141).** A graceful stop of a redundant OtOpcUa node reports ServiceLevel **0** for a few seconds and then closes the session — shorter than the degraded window, and the unreadable sample that follows ends the streak. Without a shortcut the connection fell back to the connectivity path and retried the stopped node `FailoverRetryCount` times (QA Warm gate 2026-11-13: Good on the peer after 66 s, though the peer was Active in 0.2 s). Two rules close that gap, both still requiring a **Healthy, higher** level on the other endpoint and honouring the **dwell**: + - **ServiceLevel 0 skips the window.** A sample of 0 (Part 4 *Maintenance*: not available for use) probes the other endpoint at once. Only 0 does this — 1 (booting) and the standby levels still wait out the window. It happens once per degraded streak: if the other endpoint is not Healthy, it is re-probed once per window as usual, not on every poll. + - **A lost link after a Degraded level probes the other endpoint first.** When the link drops and the last ServiceLevel this session actually read (unreadable samples don't replace it) was below 200, the actor probes the other endpoint **before the first reconnect attempt**. If it is Healthy and higher, the connection switches straight to it and reconnects at once — the stopped node is not retried. Otherwise the ordinary retry cycle starts on the same endpoint (the probe's time counts against the first reconnect interval). Site event log: `Failover from Primary to Backup (ServiceLevel)` with *link lost after ServiceLevel N*. + + A link lost after only Healthy levels (250/240 pairs, `Mode=Unset` without a 0), inside the dwell of the previous ServiceLevel switch, or with the setting disabled keeps today's retry-then-failover behaviour exactly. Expected failover on a graceful stop: the next ServiceLevel poll plus one probe — a few seconds. - **Behaviour against today's servers.** A pair that only ever reports Healthy levels (OtOpcUa 250/240 without Warm, or a server that always reports 255) never starts a streak: no probe, no switch — the default-on setting changes nothing there. It does act on a non-Warm OtOpcUa node that has dropped to 100/110 or 0/10 while its peer is ≥ 200, which is the desired move. **Configuration** (per connection, on the **primary** endpoint's OPC UA JSON — the backup's copy is ignored; edited in the Central UI's *Backup endpoint* card next to *Failover Retry Count*; via the CLI it is part of `--primary-config`): diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.ServiceLevel.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.ServiceLevel.cs index bcff598c..ecfc784a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.ServiceLevel.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.ServiceLevel.cs @@ -37,12 +37,32 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors; /// monitored at all. The switch itself reuses and the reconnect /// cycle, exactly like the connection-driven failovers. /// +/// +/// +/// A server going away (ScadaBridge#141). A graceful stop of a redundant OtOpcUa node reports +/// ServiceLevel 0 for only a few seconds before the session closes — shorter than the degraded window +/// — and the following unreadable sample cleared the streak, so the connection fell back to the +/// connectivity path and retried the stopped node FailoverRetryCount times (66 s on QA). Two +/// going-away triggers now skip that wait: (1) a sample of (0, +/// Part 4 "Maintenance": the server is not available for use) probes the other endpoint at once, +/// without the window; and (2) when the link is lost while the last KNOWN level of the session was +/// Degraded (below 200), the other endpoint is probed before the first reconnect attempt and, if it +/// answers Healthy, the connection switches straight to it instead of spending the retry count on the +/// endpoint that announced it was degraded. Both keep the "Healthy and higher" rule and the dwell; a +/// pair that only ever reports Healthy levels (250/240) never meets either condition. +/// /// public partial class DataConnectionActor { /// The lowest ServiceLevel in the OPC UA Healthy band (Part 4 §6.6.2.4.2). internal const byte HealthyServiceLevelMinimum = 200; + /// + /// Part 4 §6.6.2.4.2 "Maintenance": the server is not available for use — OtOpcUa reports it while + /// it is shutting down or its endpoint has failed. Skips the degraded window (ScadaBridge#141). + /// + internal const byte MaintenanceServiceLevel = 0; + private const string ServiceLevelPollTimerKey = "servicelevel-poll"; private ServiceLevelFailoverPolicy _serviceLevelPolicy; @@ -59,6 +79,27 @@ public partial class DataConnectionActor /// True until the first sample of a session is evaluated (connect-time preference). private bool _serviceLevelConnectCheckPending; + /// + /// The last level this session actually read (unknown samples do not overwrite it); null when + /// none has been read. Drives the lost-link probe (ScadaBridge#141). + /// + private byte? _lastKnownServiceLevel; + + /// + /// True while the lost-link probe of the other endpoint is outstanding; the first reconnect + /// attempt is held until it answers (ScadaBridge#141). + /// + private bool _disconnectProbePending; + + /// + /// True once ServiceLevel 0 has started a probe in the current degraded streak: 0 skips the window + /// once, so a peer that is not Healthy is then re-probed once per window, not on every poll. + /// + private bool _goingAwayProbeStarted; + + /// When the outstanding lost-link probe started (to shorten the reconnect wait after it). + private DateTimeOffset _disconnectProbeStartedAt; + /// Why is re-establishing the link. internal enum ReconnectCause { @@ -104,6 +145,8 @@ public partial class DataConnectionActor _serviceLevelReadInFlight = false; _serviceLevelProbeInFlight = false; _serviceLevelConnectCheckPending = false; + _lastKnownServiceLevel = null; + _goingAwayProbeStarted = false; } private void HandleServiceLevelPollTick() @@ -131,6 +174,8 @@ public partial class DataConnectionActor _serviceLevelReadInFlight = false; var connectCheck = _serviceLevelConnectCheckPending; _serviceLevelConnectCheckPending = false; + if (sampled.Level is { } known) + _lastKnownServiceLevel = known; if (sampled.Level is not { } level || level >= HealthyServiceLevelMinimum) { @@ -140,10 +185,14 @@ public partial class DataConnectionActor _log.Info("[{0}] {1} endpoint ServiceLevel is {2} — degraded watch cleared", _connectionName, _activeEndpoint, sampled.Level?.ToString(CultureInfo.InvariantCulture) ?? "unknown"); _degradedSince = null; + _goingAwayProbeStarted = false; return; } var now = DateTimeOffset.UtcNow; + // ScadaBridge#141: 0 means the server is going away (graceful stop) — it will not be there + // for a full window, so the other endpoint is checked now. Still "Healthy and higher" + dwell. + var goingAway = level == MaintenanceServiceLevel && !_goingAwayProbeStarted; if (_degradedSince is null) { _degradedSince = now; @@ -151,10 +200,11 @@ public partial class DataConnectionActor + "will check the other endpoint {4}", _connectionName, _activeEndpoint, level, HealthyServiceLevelMinimum, connectCheck ? "now (connect-time preference)" + : goingAway ? "now (ServiceLevel 0: the server is going away)" : $"if it persists for {_serviceLevelPolicy.DegradedWindow.TotalSeconds:F0}s"); } - if (!connectCheck && now - _degradedSince.Value < _serviceLevelPolicy.DegradedWindow) + if (!connectCheck && !goingAway && now - _degradedSince.Value < _serviceLevelPolicy.DegradedWindow) return; if (now - _lastServiceLevelSwitchAt < _serviceLevelPolicy.MinDwell) @@ -165,6 +215,8 @@ public partial class DataConnectionActor return; } + if (goingAway) + _goingAwayProbeStarted = true; StartServiceLevelProbe(level); } @@ -173,9 +225,10 @@ public partial class DataConnectionActor /// configuration — connect, read, dispose — off the actor thread, bounded by /// . The live adapter is untouched. /// - private void StartServiceLevelProbe(byte currentLevel) + private void StartServiceLevelProbe(byte currentLevel, bool afterDisconnect = false) { - _serviceLevelProbeInFlight = true; + if (!afterDisconnect) + _serviceLevelProbeInFlight = true; var generation = _adapterGeneration; var otherConfig = _activeEndpoint == ActiveEndpoint.Primary ? _backupConfig! : _primaryConfig; var factory = _factory; @@ -184,11 +237,12 @@ public partial class DataConnectionActor Task.Run(() => ProbeServiceLevelAsync(factory, protocol, otherConfig).WaitAsync(timeout)) .ContinueWith(t => t.IsCompletedSuccessfully - ? new ServiceLevelProbed(generation, currentLevel, t.Result.Level, t.Result.Error) + ? new ServiceLevelProbed(generation, currentLevel, t.Result.Level, t.Result.Error, afterDisconnect) : new ServiceLevelProbed(generation, currentLevel, null, t.Exception?.GetBaseException() is TimeoutException ? $"no answer within {timeout.TotalSeconds:F0}s" - : t.Exception?.GetBaseException().Message ?? "probe cancelled"), + : t.Exception?.GetBaseException().Message ?? "probe cancelled", + afterDisconnect), TaskScheduler.Default) .PipeTo(_self); } @@ -223,7 +277,8 @@ public partial class DataConnectionActor private void HandleServiceLevelProbed(ServiceLevelProbed probed) { - if (probed.AdapterGeneration != _adapterGeneration) + // A lost-link probe belongs to the Reconnecting state (HandleDisconnectProbed). + if (probed.AfterDisconnect || probed.AdapterGeneration != _adapterGeneration) return; _serviceLevelProbeInFlight = false; @@ -252,25 +307,95 @@ public partial class DataConnectionActor } var degradedFor = now - _degradedSince.Value; + SwitchForServiceLevel(probed.CurrentLevel, otherLevel, + $"ServiceLevel {probed.CurrentLevel} Degraded for {degradedFor.TotalSeconds:F0}s"); + BecomeReconnecting(ReconnectCause.ServiceLevelFailover); + } + + /// + /// The endpoint switch shared by the Connected-state and the lost-link ServiceLevel triggers: + /// flips the adapter, resets both connection-failure counters, starts the dwell and logs it. + /// The caller re-arms the reconnect. + /// + private void SwitchForServiceLevel(byte currentLevel, byte otherLevel, string why) + { var previousEndpoint = SwitchToOtherEndpoint(); _consecutiveFailures = 0; _consecutiveUnstableDisconnects = 0; - _lastServiceLevelSwitchAt = now; + _lastServiceLevelSwitchAt = DateTimeOffset.UtcNow; - _log.Warning("[{0}] Failing over from {1} to {2} (ServiceLevel {3} Degraded for {4:F0}s; {2} reports {5})", - _connectionName, previousEndpoint, _activeEndpoint, probed.CurrentLevel, - degradedFor.TotalSeconds, otherLevel); + _log.Warning("[{0}] Failing over from {1} to {2} ({3}; {2} reports {4})", + _connectionName, previousEndpoint, _activeEndpoint, why, otherLevel); if (_siteEventLogger != null) { _ = _siteEventLogger.LogEventAsync( "connection", "Warning", null, _connectionName, $"Failover from {previousEndpoint} to {_activeEndpoint} (ServiceLevel)", - $"{previousEndpoint} reported ServiceLevel {probed.CurrentLevel} for {degradedFor.TotalSeconds:F0}s; " - + $"{_activeEndpoint} reports {otherLevel}"); + $"{previousEndpoint}: {why} (last ServiceLevel {currentLevel}); {_activeEndpoint} reports {otherLevel}"); + } + } + + /// + /// ScadaBridge#141 — called by for a LOST link. When the session's + /// last known ServiceLevel was Degraded (below 200: the server had said it was not the one to be + /// on, typically 0 during a graceful stop), the other endpoint is probed before the first reconnect + /// attempt instead of retrying the endpoint that is going away. Not when the policy is not in + /// force, the session never read a Degraded level (250/240 pairs, unknown-only), or the dwell of + /// the previous ServiceLevel switch has not passed. + /// + /// The ended session's last known level, captured before it was reset. + /// True when the probe was started; the caller must then NOT arm the reconnect timer. + private bool TryStartDisconnectProbe(byte? lastKnownLevel) + { + if (lastKnownLevel is not { } level || level >= HealthyServiceLevelMinimum || !ServiceLevelFailoverInForce) + return false; + + var sinceSwitch = DateTimeOffset.UtcNow - _lastServiceLevelSwitchAt; + if (sinceSwitch < _serviceLevelPolicy.MinDwell) + { + _log.Info("[{0}] Link lost after ServiceLevel {1}, but the last ServiceLevel switch was {2:F0}s ago " + + "(dwell {3:F0}s) — normal reconnect cycle", _connectionName, level, + sinceSwitch.TotalSeconds, _serviceLevelPolicy.MinDwell.TotalSeconds); + return false; } - BecomeReconnecting(ReconnectCause.ServiceLevelFailover); + _log.Warning("[{0}] Link to the {1} endpoint lost after it reported ServiceLevel {2} — checking the " + + "other endpoint before retrying", _connectionName, _activeEndpoint, level); + _disconnectProbePending = true; + _disconnectProbeStartedAt = DateTimeOffset.UtcNow; + StartServiceLevelProbe(level, afterDisconnect: true); + return true; + } + + /// + /// ScadaBridge#141 — the lost-link probe's verdict (Reconnecting state). A Healthy level on the + /// other endpoint switches to it and reconnects at once; anything else releases the held reconnect + /// into the ordinary retry cycle on the same endpoint, minus the time the probe already took. + /// + private void HandleDisconnectProbed(ServiceLevelProbed probed) + { + if (!probed.AfterDisconnect || !_disconnectProbePending || probed.AdapterGeneration != _adapterGeneration) + return; + + _disconnectProbePending = false; + var other = _activeEndpoint == ActiveEndpoint.Primary ? ActiveEndpoint.Backup : ActiveEndpoint.Primary; + + if (probed.OtherLevel is { } otherLevel + && otherLevel >= HealthyServiceLevelMinimum && otherLevel > probed.CurrentLevel) + { + SwitchForServiceLevel(probed.CurrentLevel, otherLevel, + $"link lost after ServiceLevel {probed.CurrentLevel}"); + Timers.StartSingleTimer("reconnect", new AttemptConnect(), TimeSpan.Zero); + return; + } + + _log.Warning("[{0}] Link to the {1} endpoint lost after ServiceLevel {2}, but the {3} endpoint {4} — " + + "normal reconnect cycle", _connectionName, _activeEndpoint, probed.CurrentLevel, other, + probed.OtherLevel is { } l ? $"reports {l} (not a Healthy, higher level)" : $"is not available ({probed.Error})"); + var remaining = _options.ReconnectInterval - (DateTimeOffset.UtcNow - _disconnectProbeStartedAt); + Timers.StartSingleTimer("reconnect", new AttemptConnect(), + remaining > TimeSpan.Zero ? remaining : TimeSpan.Zero); } /// Poll tick for the connected endpoint's ServiceLevel. @@ -283,8 +408,13 @@ public partial class DataConnectionActor /// One ServiceLevel sample of the connected endpoint (null = unknown). internal sealed record ServiceLevelSampled(int AdapterGeneration, byte? Level); - /// The other endpoint's ServiceLevel (null = unreachable / unreadable, see Error). - internal sealed record ServiceLevelProbed(int AdapterGeneration, byte CurrentLevel, byte? OtherLevel, string? Error); + /// + /// The other endpoint's ServiceLevel (null = unreachable / unreadable, see Error). + /// marks the lost-link probe (ScadaBridge#141), which only the + /// Reconnecting state acts on. + /// + internal sealed record ServiceLevelProbed( + int AdapterGeneration, byte CurrentLevel, byte? OtherLevel, string? Error, bool AfterDisconnect = false); } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs index 93819772..64ddc575 100644 --- a/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs @@ -665,6 +665,9 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers _ => string.Empty, }); // The session is over: abandon its re-seed (02/L4) and its ServiceLevel watch (#138). + // #141: what the session last said about itself survives the reset for the lost-link probe. + var lastKnownServiceLevel = _lastKnownServiceLevel; + _disconnectProbePending = false; CancelSessionWork(); StopServiceLevelMonitoring(); _healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Disconnected); @@ -673,6 +676,7 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers // total holds, which is exactly the connected < total alert condition. ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Disconnected); + var switchedForUnstable = false; if (!deliberateSwap) { // Track unstable connections toward failover. @@ -697,6 +701,7 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers var previousEndpoint = SwitchToOtherEndpoint(); _consecutiveUnstableDisconnects = 0; _consecutiveFailures = 0; + switchedForUnstable = true; _log.Warning("[{0}] Failing over from {1} to {2} (unstable connection pattern)", _connectionName, previousEndpoint, _activeEndpoint); @@ -734,6 +739,15 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers // (mark mirrored alarms uncertain; the reconnect snapshot reconciles them). PushAlarmSourceUnavailable(); + // ScadaBridge#141: a link lost right after the endpoint said it was Degraded (ServiceLevel 0 + // on a graceful stop) checks the other endpoint first; the reconnect is then armed by the + // probe's verdict (HandleDisconnectProbed) instead of here. Cancel any timer left armed. + if (!deliberateSwap && !switchedForUnstable && TryStartDisconnectProbe(lastKnownServiceLevel)) + { + Timers.Cancel("reconnect"); + return; + } + // Schedule reconnect attempt. StartSingleTimer REPLACES any reconnect already armed, // so a reconfigure landing mid-backoff cannot leave two AttemptConnect messages racing. Timers.StartSingleTimer("reconnect", new AttemptConnect(), @@ -971,6 +985,10 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers // Ignore — the adapter has no live session; ReSubscribeAll's chunk // completions re-arm the retry timer after reconnect. break; + case ServiceLevelProbed { AfterDisconnect: true } lostLinkProbe: + // ScadaBridge#141: the other endpoint's verdict for a link lost after a Degraded level. + HandleDisconnectProbed(lostLinkProbe); + break; case ServiceLevelPollTick or ServiceLevelSampled or ServiceLevelProbed: // ScadaBridge#138: belongs to a session that is over; BecomeConnected restarts it. break; diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionGoingAwayFailoverTests.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionGoingAwayFailoverTests.cs new file mode 100644 index 00000000..2ec74014 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionGoingAwayFailoverTests.cs @@ -0,0 +1,320 @@ +using System.Diagnostics; +using Akka.Actor; +using Akka.TestKit.Xunit2; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection; +using ZB.MOM.WW.ScadaBridge.Commons.Serialization; +using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors; +using ZB.MOM.WW.ScadaBridge.HealthMonitoring; + +namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors; + +/// +/// ScadaBridge#141 — a redundant OPC UA server that is going away. On the QA Warm gate a graceful +/// stop of the Primary reported ServiceLevel 0 for ~6 s (shorter than the 10 s degraded window), the +/// next sample was unreadable and cleared the streak, and the connection then retried the stopped +/// node FailoverRetryCount times before failing over: 66 s, while the peer was Active in 0.2 s. +/// +/// +/// Two triggers, each with its non-trigger: (1) ServiceLevel 0 probes the other endpoint without the +/// window (other Degraded levels still wait it out); (2) a link lost while the session's last KNOWN +/// level was Degraded probes the other endpoint before the first reconnect and switches if it is +/// Healthy (a link lost after Healthy levels — today's 250/240 pair — retries exactly as before). +/// Anti-flap is kept: Healthy-and-higher and the dwell. The "no shortcut" assertions are made on the +/// order of sessions opened (the artifact), each paired with a positive control. +/// +/// +public class DataConnectionGoingAwayFailoverTests : TestKit +{ + private const string Primary = "opc.tcp://otopcua-a:4840"; + private const string Backup = "opc.tcp://otopcua-b:4840"; + private const string Tag = "nsu=https://zb.com/otopcua/uns;s=Line1.Pump.Speed"; + private const int RetryCount = 3; + + private readonly ServiceLevelRig _rig = new(); + private readonly ISiteHealthCollector _health = Substitute.For(); + private readonly IDataConnectionFactory _factory = Substitute.For(); + + // ReconnectInterval 1 s: the old path to the backup after a lost link needs RetryCount failed + // reconnects of the stopped node, i.e. at least 3 s and three Primary sessions attempted. + private readonly DataConnectionOptions _options = new() + { + ReconnectInterval = TimeSpan.FromSeconds(1), + TagResolutionRetryInterval = TimeSpan.FromMilliseconds(200), + WriteTimeout = TimeSpan.FromSeconds(5), + SeedReadMaxAttempts = 1, + SeedReadRetryDelay = TimeSpan.FromMilliseconds(10), + ServiceLevelPollInterval = TimeSpan.FromMilliseconds(50), + ServiceLevelProbeTimeout = TimeSpan.FromSeconds(2), + }; + + /// Wires the factory to build a rig-bound fake for whichever endpoint it is asked for. + public DataConnectionGoingAwayFailoverTests() + : base(@"akka.loglevel = DEBUG") + { + _factory.Create(Arg.Any(), Arg.Any>()) + .Returns(ci => new FakeServiceLevelConnection(_rig, ((IDictionary)ci[1])["Endpoint"])); + } + + private static Dictionary Config( + string endpoint, bool? enabled = null, int? windowSeconds = null, int? dwellSeconds = null) + { + var config = new Dictionary { ["Endpoint"] = endpoint }; + if (enabled is { } e) config[OpcUaEndpointConfigSerializer.ServiceLevelFailoverEnabledKey] = e.ToString(); + if (windowSeconds is { } w) config[OpcUaEndpointConfigSerializer.ServiceLevelFailoverWindowKey] = w.ToString(); + if (dwellSeconds is { } d) config[OpcUaEndpointConfigSerializer.ServiceLevelFailoverDwellKey] = d.ToString(); + return config; + } + + private IActorRef Start(string name, IDictionary primaryConfig, IDictionary? backupConfig) + { + var actor = Sys.ActorOf(Props.Create(() => new DataConnectionActor( + name, new FakeServiceLevelConnection(_rig, Primary), _options, _health, _factory, "OpcUa", + primaryConfig, backupConfig, RetryCount, null)), name); + actor.Tell(new SubscribeTagsRequest("c1", "inst1", name, [Tag], DateTimeOffset.UtcNow)); + ExpectMsg(TimeSpan.FromSeconds(5)); + return actor; + } + + private string ActiveEndpoint(IActorRef actor) => + actor.Ask(new DataConnectionActor.GetHealthReport(), TimeSpan.FromSeconds(3)) + .GetAwaiter().GetResult().ActiveEndpoint; + + private bool LiveOn(string endpoint) => _rig.Subscribes.Contains($"{endpoint}|{Tag}"); + + /// Waits until the connected endpoint has been sampled more times. + private void AwaitSamples(string endpoint, int more) + { + var target = _rig.ReadCount(endpoint) + more; + // Generous: under a full parallel test run the 50 ms poll can be starved for seconds. + AwaitCondition(() => _rig.ReadCount(endpoint) >= target, TimeSpan.FromSeconds(15)); + } + + // ── Trigger 1: ServiceLevel 0 skips the degraded window ── + + /// + /// The graceful stop as seen by the poll: the Primary drops from 250 to 0 while the peer is + /// promoted to 250. With a 30 s window the connection still moves within a couple of seconds. + /// + [Fact] + public void ServiceLevel0_SkipsTheWindow_AndFailsOverToTheHealthyPeer() + { + _rig.SetLevel(Primary, 250); + _rig.SetLevel(Backup, 90); + var actor = Start("ga-zero", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup)); + AwaitSamples(Primary, 3); + + var sw = Stopwatch.StartNew(); + _rig.SetLevel(Primary, 0); + _rig.SetLevel(Backup, 250); + + AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(10)); + Assert.True(sw.Elapsed < TimeSpan.FromSeconds(5), $"took {sw.Elapsed.TotalSeconds:F1}s against a 30s window"); + Assert.Equal("Backup", ActiveEndpoint(actor)); + } + + /// + /// Non-trigger: any other Degraded level (Warm standby 90, booting 1) still waits out the window — + /// the shortcut is for 0 only. Positive control in the same connection: the level was sampled + /// across the wait, and the moment it reads 0 the connection does move. + /// + /// A Degraded level other than 0. + [Theory] + [InlineData(90)] + [InlineData(1)] + public void OtherDegradedLevels_StillWaitTheWindow_Until0IsReported(byte degraded) + { + _rig.SetLevel(Primary, 250); + _rig.SetLevel(Backup, 250); + var actor = Start($"ga-window-{degraded}", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup)); + AwaitSamples(Primary, 3); + + _rig.SetLevel(Primary, degraded); + AwaitSamples(Primary, 10); // well inside the 30 s window + Assert.Equal(0, _rig.ConnectCount(Backup)); // not even probed + Assert.Equal("Primary", ActiveEndpoint(actor)); + + _rig.SetLevel(Primary, 0); + AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(10)); + } + + /// + /// Anti-flap kept: ServiceLevel 0 with the peer Degraded too stays put (positive control: the + /// peer WAS probed). The shortcut is taken once per streak — the peer is then re-probed once per + /// window, not on every poll — and once the peer turns Healthy the next window moves the connection. + /// + [Fact] + public void ServiceLevel0_PeerNotHealthy_StaysPut_AndReprobesOncePerWindow() + { + _rig.SetLevel(Primary, 250); + _rig.SetLevel(Backup, 90); + var actor = Start("ga-zero-both", Config(Primary, windowSeconds: 3, dwellSeconds: 0), Config(Backup)); + AwaitSamples(Primary, 3); + + _rig.SetLevel(Primary, 0); + AwaitCondition(() => _rig.ReadCount(Backup) >= 1, TimeSpan.FromSeconds(5)); + AwaitSamples(Primary, 20); // ~1 s of polls at 50 ms, inside the 3 s window + Assert.Equal(1, _rig.ReadCount(Backup)); + Assert.Equal("Primary", ActiveEndpoint(actor)); + Assert.False(LiveOn(Backup)); + + _rig.SetLevel(Backup, 250); + AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(10)); + } + + /// + /// Anti-flap kept: ServiceLevel 0 inside the dwell of a previous ServiceLevel switch does not move + /// the connection back; after the dwell it does. + /// + [Fact] + public void ServiceLevel0_InsideTheDwell_Waits_ThenSwitches() + { + _rig.SetLevel(Primary, 90); + _rig.SetLevel(Backup, 250); + var actor = Start("ga-zero-dwell", Config(Primary, windowSeconds: 30, dwellSeconds: 6), Config(Backup)); + AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(5)); // connect-time switch starts the dwell + var switchedAt = Stopwatch.StartNew(); + + _rig.SetLevel(Backup, 0); + _rig.SetLevel(Primary, 250); + var primaryConnects = _rig.ConnectCount(Primary); + Thread.Sleep(2500); + Assert.Equal(primaryConnects, _rig.ConnectCount(Primary)); + Assert.Equal("Backup", ActiveEndpoint(actor)); + + AwaitCondition(() => ActiveEndpoint(actor) == "Primary", TimeSpan.FromSeconds(15)); + Assert.True(switchedAt.Elapsed >= TimeSpan.FromSeconds(4), + $"switched back after {switchedAt.Elapsed.TotalSeconds:F1}s, inside the 6s dwell"); + } + + // ── Trigger 2: a link lost after a Degraded level ── + + /// + /// The QA sequence: Degraded, then unreadable (the server is closing — the streak is cleared), + /// then the link drops. The last KNOWN level was Degraded, so the peer is probed first and the + /// stopped node is never retried. + /// + [Fact] + public void LinkLost_AfterDegradedThenUnknown_ProbesThePeer_AndNeverRetriesTheStoppedNode() + { + _rig.SetLevel(Primary, 250); + _rig.SetLevel(Backup, 240); + var actor = Start("ga-lost", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup)); + AwaitSamples(Primary, 3); + + _rig.SetLevel(Primary, 90); + AwaitSamples(Primary, 3); + _rig.SetLevel(Primary, (byte?)null); + AwaitSamples(Primary, 3); + Assert.Equal(0, _rig.ConnectCount(Backup)); + + var connectsBefore = _rig.Connects.Count; + var sw = Stopwatch.StartNew(); + _rig.Stop(Primary); + + AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(10)); + Assert.True(sw.Elapsed < TimeSpan.FromSeconds(RetryCount * _options.ReconnectInterval.TotalSeconds), + $"took {sw.Elapsed.TotalSeconds:F1}s — as long as the retry cycle"); + Assert.DoesNotContain(Primary, _rig.ConnectsAfter(connectsBefore)); + Assert.Equal("Backup", ActiveEndpoint(actor)); + } + + /// + /// Non-trigger: today's 250/240 pair. A link lost after Healthy levels retries the same node + /// FailoverRetryCount times before the connectivity failover — exactly as before #141. + /// Positive control: the connectivity failover itself still happens afterwards. + /// + [Fact] + public void LinkLost_AfterHealthyLevels_RetriesAsBefore_ThenFailsOver() + { + _rig.SetLevel(Primary, 250); + _rig.SetLevel(Backup, 240); + var actor = Start("ga-lost-healthy", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup)); + AwaitSamples(Primary, 3); + + var connectsBefore = _rig.Connects.Count; + _rig.Stop(Primary); + + AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(15)); + var after = _rig.ConnectsAfter(connectsBefore); + Assert.Equal(Enumerable.Repeat(Primary, RetryCount), after.Take(RetryCount)); + Assert.Equal(Backup, after[RetryCount]); + Assert.Equal("Backup", ActiveEndpoint(actor)); + } + + /// + /// Non-trigger (anti-flap): link lost after a Degraded level, but the peer is Degraded too. The + /// peer is probed (positive control), then the ordinary retry cycle runs on the same node. + /// + [Fact] + public void LinkLost_AfterDegraded_PeerNotHealthy_FallsBackToTheRetryCycle() + { + _rig.SetLevel(Primary, 250); + _rig.SetLevel(Backup, 50); + var actor = Start("ga-lost-both", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup)); + AwaitSamples(Primary, 3); + _rig.SetLevel(Primary, 90); + AwaitSamples(Primary, 3); + + var connectsBefore = _rig.Connects.Count; + _rig.Stop(Primary); + + AwaitCondition(() => _rig.ConnectsAfter(connectsBefore).Count(e => e == Primary) >= 2, TimeSpan.FromSeconds(10)); + var after = _rig.ConnectsAfter(connectsBefore); + Assert.Equal(Backup, after[0]); // the probe + Assert.Equal(Primary, after[1]); // then the normal retry + Assert.False(LiveOn(Backup)); + Assert.Equal("Primary", ActiveEndpoint(actor)); + } + + /// + /// Non-trigger (anti-flap): a link lost inside the dwell of the previous ServiceLevel switch + /// takes the normal retry cycle — no probe. Positive control: the identical sequence after the + /// dwell () + /// does probe; here the backup it sits on is retried first. + /// + [Fact] + public void LinkLost_InsideTheDwell_TakesTheRetryCycle() + { + _rig.SetLevel(Primary, 90); + _rig.SetLevel(Backup, 250); + var actor = Start("ga-lost-dwell", Config(Primary, windowSeconds: 30, dwellSeconds: 120), Config(Backup)); + AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(5)); // connect-time switch starts the dwell + AwaitSamples(Backup, 3); + _rig.SetLevel(Backup, 90); + _rig.SetLevel(Primary, 250); + AwaitSamples(Backup, 3); + + var connectsBefore = _rig.Connects.Count; + _rig.Stop(Backup); + + AwaitCondition(() => _rig.ConnectsAfter(connectsBefore).Length >= 2, TimeSpan.FromSeconds(10)); + var after = _rig.ConnectsAfter(connectsBefore); + Assert.Equal(Backup, after[0]); + Assert.Equal(Backup, after[1]); + Assert.Equal("Backup", ActiveEndpoint(actor)); + } + + /// + /// Non-trigger: ServiceLevel failover disabled — a lost link never probes, whatever the levels. + /// Positive control: the retry cycle runs (the stopped node is attempted again). + /// + [Fact] + public void LinkLost_FeatureDisabled_NeverProbes() + { + _rig.SetLevel(Primary, 0); + _rig.SetLevel(Backup, 250); + var actor = Start("ga-lost-disabled", Config(Primary, enabled: false), Config(Backup)); + AwaitCondition(() => LiveOn(Primary), TimeSpan.FromSeconds(5)); + + var connectsBefore = _rig.Connects.Count; + _rig.Stop(Primary); + + AwaitCondition(() => _rig.ConnectsAfter(connectsBefore).Length >= 2, TimeSpan.FromSeconds(10)); + Assert.All(_rig.ConnectsAfter(connectsBefore).Take(2), e => Assert.Equal(Primary, e)); + Assert.Equal(0, _rig.ReadCount(Backup)); + Assert.Equal("Primary", ActiveEndpoint(actor)); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/FakeServiceLevelConnection.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/FakeServiceLevelConnection.cs index 0e67a667..5002ea5e 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/FakeServiceLevelConnection.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/FakeServiceLevelConnection.cs @@ -42,6 +42,23 @@ internal sealed class ServiceLevelRig public int ConnectCount(string endpoint) => Connects.Count(e => e == endpoint); + /// Every adapter that opened a session, so a test can kill the live one (ScadaBridge#141). + public ConcurrentQueue Sessions { get; } = new(); + + /// + /// The server at goes away: it refuses new sessions and every live + /// session on it raises Disconnected — what a stopped OtOpcUa node looks like to the client. + /// + public void Stop(string endpoint) + { + SetReachable(endpoint, false); + foreach (var session in Sessions.Where(s => s.Endpoint == endpoint && s.IsConnected)) + session.RaiseDisconnected(); + } + + /// The endpoints sessions were opened on after the first connects. + public string[] ConnectsAfter(int skip) => Connects.Skip(skip).ToArray(); + public int ReadCount(string endpoint) => Reads.GetValueOrDefault(endpoint); } @@ -72,9 +89,12 @@ internal sealed class FakeServiceLevelConnection : IDataConnection, IServiceLeve if (!_rig.IsReachable(Endpoint)) return Task.FromException(new InvalidOperationException($"{Endpoint} refused the session")); _connected = true; + _rig.Sessions.Enqueue(this); return Task.CompletedTask; } + public bool IsConnected => _connected; + public Task DisconnectAsync(CancellationToken cancellationToken = default) { _connected = false; diff --git a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/OpcUaServiceLevelTests.cs b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/OpcUaServiceLevelTests.cs index 19ce98fb..eacd7519 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/OpcUaServiceLevelTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/OpcUaServiceLevelTests.cs @@ -208,4 +208,54 @@ public sealed class OpcUaServiceLevelTests : TestKit c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionEndpoint) && c.GetArguments() is ["today", "Connected to backup"]); } + + /// + /// ScadaBridge#141 end to end, the QA Warm gate's step 4a: the connection is on the Warm Primary + /// (A, 250) with the standby (B, 90) as backup, both on the DEFAULT policy (10 s window, 60 s + /// dwell) and the default 5 s reconnect interval with three retries. A is stopped gracefully the + /// way OtOpcUa does it — the peer promotes to 250, A reports ServiceLevel 0 for a moment, then + /// the server goes away. The connection must be live on B within seconds; the connectivity path + /// alone (keep-alive + three 5 s retries of the stopped A) needs well over 15 s. + /// + [Fact] + public async Task RealServerPair_GracefulStopOfTheWarmPrimary_FailsOverWithinSeconds() + { + var a = await SecureOpcUaTestServer.StartAsync(); + await using var b = await SecureOpcUaTestServer.StartAsync(); + try + { + var factory = new DataConnectionFactory(NullLoggerFactory.Instance, Options.Create(ClientOptions())); + var options = new DataConnectionOptions(); // production defaults: 5 s reconnect, 2 s poll, 15 s probe + a.SetServiceLevel(250); + b.SetServiceLevel(90); + var health = Substitute.For(); + var primary = Flat(a.Url); // null ServiceLevelFailover = defaults (enabled, 10 s, 60 s) + + bool ConnectedTo(string endpoint) => health.ReceivedCalls().Any(c => + c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionEndpoint) + && c.GetArguments() is ["stop", var label] && (string)label! == $"Connected to {endpoint}"); + + var actor = Sys.ActorOf(Props.Create(() => new DataConnectionActor( + "stop", factory.Create("OpcUa", primary), options, health, factory, "OpcUa", + primary, Flat(b.Url), 3, null))); + AwaitCondition(() => ConnectedTo("primary"), TimeSpan.FromSeconds(30)); + await Task.Delay(TimeSpan.FromSeconds(3)); // a few Healthy polls: no connect-time effect left + + var sw = System.Diagnostics.Stopwatch.StartNew(); + b.SetServiceLevel(250); // the peer promotes (0.2 s on QA) + a.SetServiceLevel(0); // the stopping node reports Maintenance... + await Task.Delay(TimeSpan.FromSeconds(1)); + await a.DisposeAsync(); // ...and is gone well inside the 10 s window + + AwaitCondition(() => ConnectedTo("backup"), TimeSpan.FromSeconds(30)); + Assert.True(sw.Elapsed < TimeSpan.FromSeconds(10), + $"failover took {sw.Elapsed.TotalSeconds:F1}s (QA before #141: 66 s)"); + Assert.Equal("Backup", (await actor.Ask( + new DataConnectionActor.GetHealthReport(), TimeSpan.FromSeconds(5))).ActiveEndpoint); + } + finally + { + await a.DisposeAsync(); + } + } }