fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)
QA Warm gate (lmxopcua#907, 2026-11-13): a graceful stop of the Warm Primary reported ServiceLevel 0 for ~6 s - shorter than the 10 s degraded window - the next unreadable sample cleared the streak, and the connection retried the stopped node FailoverRetryCount times: Good on the peer after 66 s, peer Active in 0.2 s. Two going-away triggers on top of #138, both still requiring a Healthy (>= 200), higher level on the other endpoint and honouring the minimum dwell: - ServiceLevel 0 (Part 4 Maintenance) probes the other endpoint at once instead of waiting out the window - once per degraded streak, so a non-Healthy peer is re-probed once per window, not every poll. Other Degraded levels (standby 90/50, booting 1) still wait the window. - A lost link whose session's last KNOWN ServiceLevel was < 200 (unreadable samples don't overwrite it) probes the other endpoint before the first reconnect attempt, holding the reconnect timer. Healthy -> switch and reconnect at zero delay, the stopped node is never retried; otherwise the ordinary retry cycle runs (less the probe's time). After only Healthy levels (250/240, Unset), inside the dwell, or with the feature off: unchanged retry-then-failover. Tests: DataConnectionGoingAwayFailoverTests (10 cases, each trigger and non-trigger with positive controls, connect-order artifact assertions) and a real in-process OPC UA server pair graceful-stop e2e on production defaults: 1.1 s (lost-link trigger alone 12.1 s, neither 33.1 s -> fails the 10 s bound). 13 mutants, all killed. DCL tests 608/608 (3 full runs); solution 0 warnings. Docs: Component-DataConnectionLayer, components/DataConnectionLayer. Not deployed; QA live re-check pending. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
@@ -121,6 +121,8 @@ When `BackupConnectionDetails` is provided, the actor tracks consecutive failure
|
||||
|
||||
**ServiceLevel-aware failover (OPC UA, Gitea #138).** A third trigger, for redundant OPC UA pairs whose standby keeps a healthy session but serves Bad data (OtOpcUa Warm, lmxopcua#717). It lives in `Actors/DataConnectionActor.ServiceLevel.cs`. While Connected, and only when the adapter implements `IServiceLevelReadable`, the connection has a backup and the primary config's `ServiceLevelFailover.Enabled` is true, a periodic timer reads `Server_ServiceLevel` (`i=2267`). A level below 200 starts a degraded streak; a Healthy or unreadable sample ends it. After the degraded window (default 10 s) — or at once on the first sample after a (re)connect — the actor probes the other endpoint with a throwaway adapter from `DataConnectionFactory.Create` (connect, read, dispose; bounded by `ServiceLevelProbeTimeout`). It switches only if that endpoint reports ≥ 200 and higher, and not within the minimum dwell (default 60 s) of the previous ServiceLevel switch; otherwise it stays and re-arms the window. The switch is `SwitchToOtherEndpoint()` — the helper the two connectivity triggers now share — followed by `BecomeReconnecting(ReconnectCause.ServiceLevelFailover)` (zero delay, no unstable accounting). Poll and probe results carry the adapter generation, so a result from a session that has ended is dropped. Today's OtOpcUa (250/240) never starts a streak. Full rule set: `docs/requirements/Component-DataConnectionLayer.md` → *ServiceLevel-aware failover*.
|
||||
|
||||
**A server going away (Gitea #141).** Two shortcuts on top of the window, both in the same file and both still bound by "Healthy and higher" and the dwell. (1) A sample of `MaintenanceServiceLevel` (0) probes the other endpoint at once instead of waiting out the window — once per degraded streak (`_goingAwayProbeStarted`), so a peer that is not Healthy is re-probed once per window, not every poll. (2) `BecomeReconnecting` for a lost link captures the session's `_lastKnownServiceLevel` (the last level actually read; unknown samples don't overwrite it) before the watch resets. If that level was below 200, `TryStartDisconnectProbe` probes the other endpoint and **holds** the reconnect timer. `HandleDisconnectProbed` (Reconnecting state; the probe result carries `AfterDisconnect: true`) either switches (`SwitchForServiceLevel`, reconnect at zero delay) or releases the ordinary retry cycle on the same endpoint, less the time the probe took. A link lost after Healthy levels, inside the dwell, or with the feature off takes the ordinary retry cycle as before. Why: an OtOpcUa graceful stop reports 0 for only ~6 s and then the session closes, so on QA the connection spent 66 s retrying the stopped node.
|
||||
|
||||
### Write path
|
||||
|
||||
Writes are fire-and-forget from the script's view only in the sense that there is no store-and-forward. The `DataConnectionActor` pipes `_adapter.WriteAsync` back to the original sender:
|
||||
|
||||
@@ -160,14 +160,19 @@ The failover above is driven only by the link misbehaving. A redundant OPC UA se
|
||||
|---|---|---|
|
||||
| Healthy | 200–255 | Active: 250 Primary / 240 Secondary (today and under Warm), 200 Active-stale |
|
||||
| Degraded | 100–199, and anything below 200 here | Active critically degraded 100; Warm standby 90 (healthy) / 50 (degraded) |
|
||||
| NoData / Maintenance | 1 / 0 | Warm booting 1; down / endpoint failed 0 |
|
||||
| NoData / Maintenance | 1 / 0 | Warm booting 1; down / endpoint failed / shutting down 0 (skips the window, #141) |
|
||||
|
||||
**Rule.** While Connected, the actor reads the connected endpoint's ServiceLevel every `ServiceLevelPollInterval` (2 s). A level **below 200** starts a *degraded streak*; any level of 200 or more, or an unreadable level (Bad status, no such node, read failure), ends it. When the streak has lasted the connection's **degraded window** (default 10 s), the actor probes the **other** endpoint — a short-lived session built from that endpoint's configuration: connect, read `i=2267`, close, bounded by `ServiceLevelProbeTimeout` (15 s). It fails over only when the other endpoint answers with a **Healthy level (≥ 200) higher than the connected one**. Otherwise (other endpoint unreachable, or also below 200) it **stays put** and the window re-arms, so the other endpoint is probed at most once per window.
|
||||
**Rule.** While Connected, the actor reads the connected endpoint's ServiceLevel every `ServiceLevelPollInterval` (2 s). A level **below 200** starts a *degraded streak*; any level of 200 or more, or an unreadable level (Bad status, no such node, read failure), ends it. When the streak has lasted the connection's **degraded window** (default 10 s) — or at once for a level of 0, see *A server going away* below — the actor probes the **other** endpoint — a short-lived session built from that endpoint's configuration: connect, read `i=2267`, close, bounded by `ServiceLevelProbeTimeout` (15 s). It fails over only when the other endpoint answers with a **Healthy level (≥ 200) higher than the connected one**. Otherwise (other endpoint unreachable, or also below 200) it **stays put** and the window re-arms, so the other endpoint is probed at most once per window.
|
||||
|
||||
- **Connect-time preference.** The first sample after every (re)connect is evaluated at once, without the window: a connection that comes up on a Degraded endpoint while the other is Healthy moves straight over. Because the trigger needs the connected level below 200, a connection to a 240 Secondary is never moved to a 250 Primary.
|
||||
- **Anti-flap.** A ServiceLevel switch is not repeated within the connection's **minimum dwell** (default 60 s), connect-time preference included. Together with the "Healthy and higher" requirement and the unbroken-streak rule (one Healthy sample resets it) this stops a pair that flips roles from bouncing the connection.
|
||||
- **Same switch as the connectivity failover.** The switch flips the active endpoint and swaps the adapter through the same helper as the two connection-driven triggers, resets both failure counters, then re-enters the reconnect cycle immediately with the unstable-disconnect accounting suppressed (it is a deliberate swap, like an in-place reconfigure). `ReSubscribeAll` / `ReSubscribeAllAlarms` then re-issue every subscription on the new endpoint; tags go Bad for the swap exactly as on any failover. Site event log: `Failover from Primary to Backup (ServiceLevel)` (Warning) with the two levels.
|
||||
- **Scope.** Only adapters implementing `IServiceLevelReadable` (the OPC UA adapter) are watched, and only when the connection has a backup and the setting is enabled. MxGateway connections, single-endpoint connections and disabled connections never read a ServiceLevel. The ServiceLevel read never raises the adapter's `Disconnected` event: a dead session stays the keep-alive's business.
|
||||
- **A server going away (Gitea #141).** A graceful stop of a redundant OtOpcUa node reports ServiceLevel **0** for a few seconds and then closes the session — shorter than the degraded window, and the unreadable sample that follows ends the streak. Without a shortcut the connection fell back to the connectivity path and retried the stopped node `FailoverRetryCount` times (QA Warm gate 2026-11-13: Good on the peer after 66 s, though the peer was Active in 0.2 s). Two rules close that gap, both still requiring a **Healthy, higher** level on the other endpoint and honouring the **dwell**:
|
||||
- **ServiceLevel 0 skips the window.** A sample of 0 (Part 4 *Maintenance*: not available for use) probes the other endpoint at once. Only 0 does this — 1 (booting) and the standby levels still wait out the window. It happens once per degraded streak: if the other endpoint is not Healthy, it is re-probed once per window as usual, not on every poll.
|
||||
- **A lost link after a Degraded level probes the other endpoint first.** When the link drops and the last ServiceLevel this session actually read (unreadable samples don't replace it) was below 200, the actor probes the other endpoint **before the first reconnect attempt**. If it is Healthy and higher, the connection switches straight to it and reconnects at once — the stopped node is not retried. Otherwise the ordinary retry cycle starts on the same endpoint (the probe's time counts against the first reconnect interval). Site event log: `Failover from Primary to Backup (ServiceLevel)` with *link lost after ServiceLevel N*.
|
||||
|
||||
A link lost after only Healthy levels (250/240 pairs, `Mode=Unset` without a 0), inside the dwell of the previous ServiceLevel switch, or with the setting disabled keeps today's retry-then-failover behaviour exactly. Expected failover on a graceful stop: the next ServiceLevel poll plus one probe — a few seconds.
|
||||
- **Behaviour against today's servers.** A pair that only ever reports Healthy levels (OtOpcUa 250/240 without Warm, or a server that always reports 255) never starts a streak: no probe, no switch — the default-on setting changes nothing there. It does act on a non-Warm OtOpcUa node that has dropped to 100/110 or 0/10 while its peer is ≥ 200, which is the desired move.
|
||||
|
||||
**Configuration** (per connection, on the **primary** endpoint's OPC UA JSON — the backup's copy is ignored; edited in the Central UI's *Backup endpoint* card next to *Failover Retry Count*; via the CLI it is part of `--primary-config`):
|
||||
|
||||
+145
-15
@@ -37,12 +37,32 @@ namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
|
||||
/// monitored at all. The switch itself reuses <see cref="SwitchToOtherEndpoint"/> and the reconnect
|
||||
/// cycle, exactly like the connection-driven failovers.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>A server going away (ScadaBridge#141).</b> A graceful stop of a redundant OtOpcUa node reports
|
||||
/// ServiceLevel 0 for only a few seconds before the session closes — shorter than the degraded window
|
||||
/// — and the following unreadable sample cleared the streak, so the connection fell back to the
|
||||
/// connectivity path and retried the stopped node <c>FailoverRetryCount</c> times (66 s on QA). Two
|
||||
/// going-away triggers now skip that wait: (1) a sample of <see cref="MaintenanceServiceLevel"/> (0,
|
||||
/// Part 4 "Maintenance": the server is not available for use) probes the other endpoint at once,
|
||||
/// without the window; and (2) when the link is lost while the last KNOWN level of the session was
|
||||
/// Degraded (below 200), the other endpoint is probed before the first reconnect attempt and, if it
|
||||
/// answers Healthy, the connection switches straight to it instead of spending the retry count on the
|
||||
/// endpoint that announced it was degraded. Both keep the "Healthy and higher" rule and the dwell; a
|
||||
/// pair that only ever reports Healthy levels (250/240) never meets either condition.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public partial class DataConnectionActor
|
||||
{
|
||||
/// <summary>The lowest ServiceLevel in the OPC UA Healthy band (Part 4 §6.6.2.4.2).</summary>
|
||||
internal const byte HealthyServiceLevelMinimum = 200;
|
||||
|
||||
/// <summary>
|
||||
/// Part 4 §6.6.2.4.2 "Maintenance": the server is not available for use — OtOpcUa reports it while
|
||||
/// it is shutting down or its endpoint has failed. Skips the degraded window (ScadaBridge#141).
|
||||
/// </summary>
|
||||
internal const byte MaintenanceServiceLevel = 0;
|
||||
|
||||
private const string ServiceLevelPollTimerKey = "servicelevel-poll";
|
||||
|
||||
private ServiceLevelFailoverPolicy _serviceLevelPolicy;
|
||||
@@ -59,6 +79,27 @@ public partial class DataConnectionActor
|
||||
/// <summary>True until the first sample of a session is evaluated (connect-time preference).</summary>
|
||||
private bool _serviceLevelConnectCheckPending;
|
||||
|
||||
/// <summary>
|
||||
/// The last level this session actually read (unknown samples do not overwrite it); null when
|
||||
/// none has been read. Drives the lost-link probe (ScadaBridge#141).
|
||||
/// </summary>
|
||||
private byte? _lastKnownServiceLevel;
|
||||
|
||||
/// <summary>
|
||||
/// True while the lost-link probe of the other endpoint is outstanding; the first reconnect
|
||||
/// attempt is held until it answers (ScadaBridge#141).
|
||||
/// </summary>
|
||||
private bool _disconnectProbePending;
|
||||
|
||||
/// <summary>
|
||||
/// True once ServiceLevel 0 has started a probe in the current degraded streak: 0 skips the window
|
||||
/// once, so a peer that is not Healthy is then re-probed once per window, not on every poll.
|
||||
/// </summary>
|
||||
private bool _goingAwayProbeStarted;
|
||||
|
||||
/// <summary>When the outstanding lost-link probe started (to shorten the reconnect wait after it).</summary>
|
||||
private DateTimeOffset _disconnectProbeStartedAt;
|
||||
|
||||
/// <summary>Why <see cref="BecomeReconnecting"/> is re-establishing the link.</summary>
|
||||
internal enum ReconnectCause
|
||||
{
|
||||
@@ -104,6 +145,8 @@ public partial class DataConnectionActor
|
||||
_serviceLevelReadInFlight = false;
|
||||
_serviceLevelProbeInFlight = false;
|
||||
_serviceLevelConnectCheckPending = false;
|
||||
_lastKnownServiceLevel = null;
|
||||
_goingAwayProbeStarted = false;
|
||||
}
|
||||
|
||||
private void HandleServiceLevelPollTick()
|
||||
@@ -131,6 +174,8 @@ public partial class DataConnectionActor
|
||||
_serviceLevelReadInFlight = false;
|
||||
var connectCheck = _serviceLevelConnectCheckPending;
|
||||
_serviceLevelConnectCheckPending = false;
|
||||
if (sampled.Level is { } known)
|
||||
_lastKnownServiceLevel = known;
|
||||
|
||||
if (sampled.Level is not { } level || level >= HealthyServiceLevelMinimum)
|
||||
{
|
||||
@@ -140,10 +185,14 @@ public partial class DataConnectionActor
|
||||
_log.Info("[{0}] {1} endpoint ServiceLevel is {2} — degraded watch cleared",
|
||||
_connectionName, _activeEndpoint, sampled.Level?.ToString(CultureInfo.InvariantCulture) ?? "unknown");
|
||||
_degradedSince = null;
|
||||
_goingAwayProbeStarted = false;
|
||||
return;
|
||||
}
|
||||
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
// ScadaBridge#141: 0 means the server is going away (graceful stop) — it will not be there
|
||||
// for a full window, so the other endpoint is checked now. Still "Healthy and higher" + dwell.
|
||||
var goingAway = level == MaintenanceServiceLevel && !_goingAwayProbeStarted;
|
||||
if (_degradedSince is null)
|
||||
{
|
||||
_degradedSince = now;
|
||||
@@ -151,10 +200,11 @@ public partial class DataConnectionActor
|
||||
+ "will check the other endpoint {4}",
|
||||
_connectionName, _activeEndpoint, level, HealthyServiceLevelMinimum,
|
||||
connectCheck ? "now (connect-time preference)"
|
||||
: goingAway ? "now (ServiceLevel 0: the server is going away)"
|
||||
: $"if it persists for {_serviceLevelPolicy.DegradedWindow.TotalSeconds:F0}s");
|
||||
}
|
||||
|
||||
if (!connectCheck && now - _degradedSince.Value < _serviceLevelPolicy.DegradedWindow)
|
||||
if (!connectCheck && !goingAway && now - _degradedSince.Value < _serviceLevelPolicy.DegradedWindow)
|
||||
return;
|
||||
|
||||
if (now - _lastServiceLevelSwitchAt < _serviceLevelPolicy.MinDwell)
|
||||
@@ -165,6 +215,8 @@ public partial class DataConnectionActor
|
||||
return;
|
||||
}
|
||||
|
||||
if (goingAway)
|
||||
_goingAwayProbeStarted = true;
|
||||
StartServiceLevelProbe(level);
|
||||
}
|
||||
|
||||
@@ -173,9 +225,10 @@ public partial class DataConnectionActor
|
||||
/// configuration — connect, read, dispose — off the actor thread, bounded by
|
||||
/// <see cref="DataConnectionOptions.ServiceLevelProbeTimeout"/>. The live adapter is untouched.
|
||||
/// </summary>
|
||||
private void StartServiceLevelProbe(byte currentLevel)
|
||||
private void StartServiceLevelProbe(byte currentLevel, bool afterDisconnect = false)
|
||||
{
|
||||
_serviceLevelProbeInFlight = true;
|
||||
if (!afterDisconnect)
|
||||
_serviceLevelProbeInFlight = true;
|
||||
var generation = _adapterGeneration;
|
||||
var otherConfig = _activeEndpoint == ActiveEndpoint.Primary ? _backupConfig! : _primaryConfig;
|
||||
var factory = _factory;
|
||||
@@ -184,11 +237,12 @@ public partial class DataConnectionActor
|
||||
|
||||
Task.Run(() => ProbeServiceLevelAsync(factory, protocol, otherConfig).WaitAsync(timeout))
|
||||
.ContinueWith(t => t.IsCompletedSuccessfully
|
||||
? new ServiceLevelProbed(generation, currentLevel, t.Result.Level, t.Result.Error)
|
||||
? new ServiceLevelProbed(generation, currentLevel, t.Result.Level, t.Result.Error, afterDisconnect)
|
||||
: new ServiceLevelProbed(generation, currentLevel, null,
|
||||
t.Exception?.GetBaseException() is TimeoutException
|
||||
? $"no answer within {timeout.TotalSeconds:F0}s"
|
||||
: t.Exception?.GetBaseException().Message ?? "probe cancelled"),
|
||||
: t.Exception?.GetBaseException().Message ?? "probe cancelled",
|
||||
afterDisconnect),
|
||||
TaskScheduler.Default)
|
||||
.PipeTo(_self);
|
||||
}
|
||||
@@ -223,7 +277,8 @@ public partial class DataConnectionActor
|
||||
|
||||
private void HandleServiceLevelProbed(ServiceLevelProbed probed)
|
||||
{
|
||||
if (probed.AdapterGeneration != _adapterGeneration)
|
||||
// A lost-link probe belongs to the Reconnecting state (HandleDisconnectProbed).
|
||||
if (probed.AfterDisconnect || probed.AdapterGeneration != _adapterGeneration)
|
||||
return;
|
||||
|
||||
_serviceLevelProbeInFlight = false;
|
||||
@@ -252,25 +307,95 @@ public partial class DataConnectionActor
|
||||
}
|
||||
|
||||
var degradedFor = now - _degradedSince.Value;
|
||||
SwitchForServiceLevel(probed.CurrentLevel, otherLevel,
|
||||
$"ServiceLevel {probed.CurrentLevel} Degraded for {degradedFor.TotalSeconds:F0}s");
|
||||
BecomeReconnecting(ReconnectCause.ServiceLevelFailover);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The endpoint switch shared by the Connected-state and the lost-link ServiceLevel triggers:
|
||||
/// flips the adapter, resets both connection-failure counters, starts the dwell and logs it.
|
||||
/// The caller re-arms the reconnect.
|
||||
/// </summary>
|
||||
private void SwitchForServiceLevel(byte currentLevel, byte otherLevel, string why)
|
||||
{
|
||||
var previousEndpoint = SwitchToOtherEndpoint();
|
||||
_consecutiveFailures = 0;
|
||||
_consecutiveUnstableDisconnects = 0;
|
||||
_lastServiceLevelSwitchAt = now;
|
||||
_lastServiceLevelSwitchAt = DateTimeOffset.UtcNow;
|
||||
|
||||
_log.Warning("[{0}] Failing over from {1} to {2} (ServiceLevel {3} Degraded for {4:F0}s; {2} reports {5})",
|
||||
_connectionName, previousEndpoint, _activeEndpoint, probed.CurrentLevel,
|
||||
degradedFor.TotalSeconds, otherLevel);
|
||||
_log.Warning("[{0}] Failing over from {1} to {2} ({3}; {2} reports {4})",
|
||||
_connectionName, previousEndpoint, _activeEndpoint, why, otherLevel);
|
||||
|
||||
if (_siteEventLogger != null)
|
||||
{
|
||||
_ = _siteEventLogger.LogEventAsync(
|
||||
"connection", "Warning", null, _connectionName,
|
||||
$"Failover from {previousEndpoint} to {_activeEndpoint} (ServiceLevel)",
|
||||
$"{previousEndpoint} reported ServiceLevel {probed.CurrentLevel} for {degradedFor.TotalSeconds:F0}s; "
|
||||
+ $"{_activeEndpoint} reports {otherLevel}");
|
||||
$"{previousEndpoint}: {why} (last ServiceLevel {currentLevel}); {_activeEndpoint} reports {otherLevel}");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#141 — called by <see cref="BecomeReconnecting"/> for a LOST link. When the session's
|
||||
/// last known ServiceLevel was Degraded (below 200: the server had said it was not the one to be
|
||||
/// on, typically 0 during a graceful stop), the other endpoint is probed before the first reconnect
|
||||
/// attempt instead of retrying the endpoint that is going away. Not when the policy is not in
|
||||
/// force, the session never read a Degraded level (250/240 pairs, unknown-only), or the dwell of
|
||||
/// the previous ServiceLevel switch has not passed.
|
||||
/// </summary>
|
||||
/// <param name="lastKnownLevel">The ended session's last known level, captured before it was reset.</param>
|
||||
/// <returns>True when the probe was started; the caller must then NOT arm the reconnect timer.</returns>
|
||||
private bool TryStartDisconnectProbe(byte? lastKnownLevel)
|
||||
{
|
||||
if (lastKnownLevel is not { } level || level >= HealthyServiceLevelMinimum || !ServiceLevelFailoverInForce)
|
||||
return false;
|
||||
|
||||
var sinceSwitch = DateTimeOffset.UtcNow - _lastServiceLevelSwitchAt;
|
||||
if (sinceSwitch < _serviceLevelPolicy.MinDwell)
|
||||
{
|
||||
_log.Info("[{0}] Link lost after ServiceLevel {1}, but the last ServiceLevel switch was {2:F0}s ago "
|
||||
+ "(dwell {3:F0}s) — normal reconnect cycle", _connectionName, level,
|
||||
sinceSwitch.TotalSeconds, _serviceLevelPolicy.MinDwell.TotalSeconds);
|
||||
return false;
|
||||
}
|
||||
|
||||
BecomeReconnecting(ReconnectCause.ServiceLevelFailover);
|
||||
_log.Warning("[{0}] Link to the {1} endpoint lost after it reported ServiceLevel {2} — checking the "
|
||||
+ "other endpoint before retrying", _connectionName, _activeEndpoint, level);
|
||||
_disconnectProbePending = true;
|
||||
_disconnectProbeStartedAt = DateTimeOffset.UtcNow;
|
||||
StartServiceLevelProbe(level, afterDisconnect: true);
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#141 — the lost-link probe's verdict (Reconnecting state). A Healthy level on the
|
||||
/// other endpoint switches to it and reconnects at once; anything else releases the held reconnect
|
||||
/// into the ordinary retry cycle on the same endpoint, minus the time the probe already took.
|
||||
/// </summary>
|
||||
private void HandleDisconnectProbed(ServiceLevelProbed probed)
|
||||
{
|
||||
if (!probed.AfterDisconnect || !_disconnectProbePending || probed.AdapterGeneration != _adapterGeneration)
|
||||
return;
|
||||
|
||||
_disconnectProbePending = false;
|
||||
var other = _activeEndpoint == ActiveEndpoint.Primary ? ActiveEndpoint.Backup : ActiveEndpoint.Primary;
|
||||
|
||||
if (probed.OtherLevel is { } otherLevel
|
||||
&& otherLevel >= HealthyServiceLevelMinimum && otherLevel > probed.CurrentLevel)
|
||||
{
|
||||
SwitchForServiceLevel(probed.CurrentLevel, otherLevel,
|
||||
$"link lost after ServiceLevel {probed.CurrentLevel}");
|
||||
Timers.StartSingleTimer("reconnect", new AttemptConnect(), TimeSpan.Zero);
|
||||
return;
|
||||
}
|
||||
|
||||
_log.Warning("[{0}] Link to the {1} endpoint lost after ServiceLevel {2}, but the {3} endpoint {4} — "
|
||||
+ "normal reconnect cycle", _connectionName, _activeEndpoint, probed.CurrentLevel, other,
|
||||
probed.OtherLevel is { } l ? $"reports {l} (not a Healthy, higher level)" : $"is not available ({probed.Error})");
|
||||
var remaining = _options.ReconnectInterval - (DateTimeOffset.UtcNow - _disconnectProbeStartedAt);
|
||||
Timers.StartSingleTimer("reconnect", new AttemptConnect(),
|
||||
remaining > TimeSpan.Zero ? remaining : TimeSpan.Zero);
|
||||
}
|
||||
|
||||
/// <summary>Poll tick for the connected endpoint's ServiceLevel.</summary>
|
||||
@@ -283,8 +408,13 @@ public partial class DataConnectionActor
|
||||
/// <summary>One ServiceLevel sample of the connected endpoint (null = unknown).</summary>
|
||||
internal sealed record ServiceLevelSampled(int AdapterGeneration, byte? Level);
|
||||
|
||||
/// <summary>The other endpoint's ServiceLevel (null = unreachable / unreadable, see Error).</summary>
|
||||
internal sealed record ServiceLevelProbed(int AdapterGeneration, byte CurrentLevel, byte? OtherLevel, string? Error);
|
||||
/// <summary>
|
||||
/// The other endpoint's ServiceLevel (null = unreachable / unreadable, see Error).
|
||||
/// <paramref name="AfterDisconnect"/> marks the lost-link probe (ScadaBridge#141), which only the
|
||||
/// Reconnecting state acts on.
|
||||
/// </summary>
|
||||
internal sealed record ServiceLevelProbed(
|
||||
int AdapterGeneration, byte CurrentLevel, byte? OtherLevel, string? Error, bool AfterDisconnect = false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -665,6 +665,9 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers
|
||||
_ => string.Empty,
|
||||
});
|
||||
// The session is over: abandon its re-seed (02/L4) and its ServiceLevel watch (#138).
|
||||
// #141: what the session last said about itself survives the reset for the lost-link probe.
|
||||
var lastKnownServiceLevel = _lastKnownServiceLevel;
|
||||
_disconnectProbePending = false;
|
||||
CancelSessionWork();
|
||||
StopServiceLevelMonitoring();
|
||||
_healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Disconnected);
|
||||
@@ -673,6 +676,7 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers
|
||||
// total holds, which is exactly the connected < total alert condition.
|
||||
ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Disconnected);
|
||||
|
||||
var switchedForUnstable = false;
|
||||
if (!deliberateSwap)
|
||||
{
|
||||
// Track unstable connections toward failover.
|
||||
@@ -697,6 +701,7 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers
|
||||
var previousEndpoint = SwitchToOtherEndpoint();
|
||||
_consecutiveUnstableDisconnects = 0;
|
||||
_consecutiveFailures = 0;
|
||||
switchedForUnstable = true;
|
||||
|
||||
_log.Warning("[{0}] Failing over from {1} to {2} (unstable connection pattern)",
|
||||
_connectionName, previousEndpoint, _activeEndpoint);
|
||||
@@ -734,6 +739,15 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers
|
||||
// (mark mirrored alarms uncertain; the reconnect snapshot reconciles them).
|
||||
PushAlarmSourceUnavailable();
|
||||
|
||||
// ScadaBridge#141: a link lost right after the endpoint said it was Degraded (ServiceLevel 0
|
||||
// on a graceful stop) checks the other endpoint first; the reconnect is then armed by the
|
||||
// probe's verdict (HandleDisconnectProbed) instead of here. Cancel any timer left armed.
|
||||
if (!deliberateSwap && !switchedForUnstable && TryStartDisconnectProbe(lastKnownServiceLevel))
|
||||
{
|
||||
Timers.Cancel("reconnect");
|
||||
return;
|
||||
}
|
||||
|
||||
// Schedule reconnect attempt. StartSingleTimer REPLACES any reconnect already armed,
|
||||
// so a reconfigure landing mid-backoff cannot leave two AttemptConnect messages racing.
|
||||
Timers.StartSingleTimer("reconnect", new AttemptConnect(),
|
||||
@@ -971,6 +985,10 @@ public partial class DataConnectionActor : UntypedActor, IWithTimers
|
||||
// Ignore — the adapter has no live session; ReSubscribeAll's chunk
|
||||
// completions re-arm the retry timer after reconnect.
|
||||
break;
|
||||
case ServiceLevelProbed { AfterDisconnect: true } lostLinkProbe:
|
||||
// ScadaBridge#141: the other endpoint's verdict for a link lost after a Degraded level.
|
||||
HandleDisconnectProbed(lostLinkProbe);
|
||||
break;
|
||||
case ServiceLevelPollTick or ServiceLevelSampled or ServiceLevelProbed:
|
||||
// ScadaBridge#138: belongs to a session that is over; BecomeConnected restarts it.
|
||||
break;
|
||||
|
||||
+320
@@ -0,0 +1,320 @@
|
||||
using System.Diagnostics;
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Serialization;
|
||||
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#141 — a redundant OPC UA server that is going away. On the QA Warm gate a graceful
|
||||
/// stop of the Primary reported ServiceLevel 0 for ~6 s (shorter than the 10 s degraded window), the
|
||||
/// next sample was unreadable and cleared the streak, and the connection then retried the stopped
|
||||
/// node <c>FailoverRetryCount</c> times before failing over: 66 s, while the peer was Active in 0.2 s.
|
||||
///
|
||||
/// <para>
|
||||
/// Two triggers, each with its non-trigger: (1) ServiceLevel 0 probes the other endpoint without the
|
||||
/// window (other Degraded levels still wait it out); (2) a link lost while the session's last KNOWN
|
||||
/// level was Degraded probes the other endpoint before the first reconnect and switches if it is
|
||||
/// Healthy (a link lost after Healthy levels — today's 250/240 pair — retries exactly as before).
|
||||
/// Anti-flap is kept: Healthy-and-higher and the dwell. The "no shortcut" assertions are made on the
|
||||
/// order of sessions opened (the artifact), each paired with a positive control.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class DataConnectionGoingAwayFailoverTests : TestKit
|
||||
{
|
||||
private const string Primary = "opc.tcp://otopcua-a:4840";
|
||||
private const string Backup = "opc.tcp://otopcua-b:4840";
|
||||
private const string Tag = "nsu=https://zb.com/otopcua/uns;s=Line1.Pump.Speed";
|
||||
private const int RetryCount = 3;
|
||||
|
||||
private readonly ServiceLevelRig _rig = new();
|
||||
private readonly ISiteHealthCollector _health = Substitute.For<ISiteHealthCollector>();
|
||||
private readonly IDataConnectionFactory _factory = Substitute.For<IDataConnectionFactory>();
|
||||
|
||||
// ReconnectInterval 1 s: the old path to the backup after a lost link needs RetryCount failed
|
||||
// reconnects of the stopped node, i.e. at least 3 s and three Primary sessions attempted.
|
||||
private readonly DataConnectionOptions _options = new()
|
||||
{
|
||||
ReconnectInterval = TimeSpan.FromSeconds(1),
|
||||
TagResolutionRetryInterval = TimeSpan.FromMilliseconds(200),
|
||||
WriteTimeout = TimeSpan.FromSeconds(5),
|
||||
SeedReadMaxAttempts = 1,
|
||||
SeedReadRetryDelay = TimeSpan.FromMilliseconds(10),
|
||||
ServiceLevelPollInterval = TimeSpan.FromMilliseconds(50),
|
||||
ServiceLevelProbeTimeout = TimeSpan.FromSeconds(2),
|
||||
};
|
||||
|
||||
/// <summary>Wires the factory to build a rig-bound fake for whichever endpoint it is asked for.</summary>
|
||||
public DataConnectionGoingAwayFailoverTests()
|
||||
: base(@"akka.loglevel = DEBUG")
|
||||
{
|
||||
_factory.Create(Arg.Any<string>(), Arg.Any<IDictionary<string, string>>())
|
||||
.Returns(ci => new FakeServiceLevelConnection(_rig, ((IDictionary<string, string>)ci[1])["Endpoint"]));
|
||||
}
|
||||
|
||||
private static Dictionary<string, string> Config(
|
||||
string endpoint, bool? enabled = null, int? windowSeconds = null, int? dwellSeconds = null)
|
||||
{
|
||||
var config = new Dictionary<string, string> { ["Endpoint"] = endpoint };
|
||||
if (enabled is { } e) config[OpcUaEndpointConfigSerializer.ServiceLevelFailoverEnabledKey] = e.ToString();
|
||||
if (windowSeconds is { } w) config[OpcUaEndpointConfigSerializer.ServiceLevelFailoverWindowKey] = w.ToString();
|
||||
if (dwellSeconds is { } d) config[OpcUaEndpointConfigSerializer.ServiceLevelFailoverDwellKey] = d.ToString();
|
||||
return config;
|
||||
}
|
||||
|
||||
private IActorRef Start(string name, IDictionary<string, string> primaryConfig, IDictionary<string, string>? backupConfig)
|
||||
{
|
||||
var actor = Sys.ActorOf(Props.Create(() => new DataConnectionActor(
|
||||
name, new FakeServiceLevelConnection(_rig, Primary), _options, _health, _factory, "OpcUa",
|
||||
primaryConfig, backupConfig, RetryCount, null)), name);
|
||||
actor.Tell(new SubscribeTagsRequest("c1", "inst1", name, [Tag], DateTimeOffset.UtcNow));
|
||||
ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(5));
|
||||
return actor;
|
||||
}
|
||||
|
||||
private string ActiveEndpoint(IActorRef actor) =>
|
||||
actor.Ask<DataConnectionHealthReport>(new DataConnectionActor.GetHealthReport(), TimeSpan.FromSeconds(3))
|
||||
.GetAwaiter().GetResult().ActiveEndpoint;
|
||||
|
||||
private bool LiveOn(string endpoint) => _rig.Subscribes.Contains($"{endpoint}|{Tag}");
|
||||
|
||||
/// <summary>Waits until the connected endpoint has been sampled <paramref name="more"/> more times.</summary>
|
||||
private void AwaitSamples(string endpoint, int more)
|
||||
{
|
||||
var target = _rig.ReadCount(endpoint) + more;
|
||||
// Generous: under a full parallel test run the 50 ms poll can be starved for seconds.
|
||||
AwaitCondition(() => _rig.ReadCount(endpoint) >= target, TimeSpan.FromSeconds(15));
|
||||
}
|
||||
|
||||
// ── Trigger 1: ServiceLevel 0 skips the degraded window ──
|
||||
|
||||
/// <summary>
|
||||
/// The graceful stop as seen by the poll: the Primary drops from 250 to 0 while the peer is
|
||||
/// promoted to 250. With a 30 s window the connection still moves within a couple of seconds.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ServiceLevel0_SkipsTheWindow_AndFailsOverToTheHealthyPeer()
|
||||
{
|
||||
_rig.SetLevel(Primary, 250);
|
||||
_rig.SetLevel(Backup, 90);
|
||||
var actor = Start("ga-zero", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup));
|
||||
AwaitSamples(Primary, 3);
|
||||
|
||||
var sw = Stopwatch.StartNew();
|
||||
_rig.SetLevel(Primary, 0);
|
||||
_rig.SetLevel(Backup, 250);
|
||||
|
||||
AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(10));
|
||||
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(5), $"took {sw.Elapsed.TotalSeconds:F1}s against a 30s window");
|
||||
Assert.Equal("Backup", ActiveEndpoint(actor));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Non-trigger: any other Degraded level (Warm standby 90, booting 1) still waits out the window —
|
||||
/// the shortcut is for 0 only. Positive control in the same connection: the level was sampled
|
||||
/// across the wait, and the moment it reads 0 the connection does move.
|
||||
/// </summary>
|
||||
/// <param name="degraded">A Degraded level other than 0.</param>
|
||||
[Theory]
|
||||
[InlineData(90)]
|
||||
[InlineData(1)]
|
||||
public void OtherDegradedLevels_StillWaitTheWindow_Until0IsReported(byte degraded)
|
||||
{
|
||||
_rig.SetLevel(Primary, 250);
|
||||
_rig.SetLevel(Backup, 250);
|
||||
var actor = Start($"ga-window-{degraded}", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup));
|
||||
AwaitSamples(Primary, 3);
|
||||
|
||||
_rig.SetLevel(Primary, degraded);
|
||||
AwaitSamples(Primary, 10); // well inside the 30 s window
|
||||
Assert.Equal(0, _rig.ConnectCount(Backup)); // not even probed
|
||||
Assert.Equal("Primary", ActiveEndpoint(actor));
|
||||
|
||||
_rig.SetLevel(Primary, 0);
|
||||
AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(10));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Anti-flap kept: ServiceLevel 0 with the peer Degraded too stays put (positive control: the
|
||||
/// peer WAS probed). The shortcut is taken once per streak — the peer is then re-probed once per
|
||||
/// window, not on every poll — and once the peer turns Healthy the next window moves the connection.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ServiceLevel0_PeerNotHealthy_StaysPut_AndReprobesOncePerWindow()
|
||||
{
|
||||
_rig.SetLevel(Primary, 250);
|
||||
_rig.SetLevel(Backup, 90);
|
||||
var actor = Start("ga-zero-both", Config(Primary, windowSeconds: 3, dwellSeconds: 0), Config(Backup));
|
||||
AwaitSamples(Primary, 3);
|
||||
|
||||
_rig.SetLevel(Primary, 0);
|
||||
AwaitCondition(() => _rig.ReadCount(Backup) >= 1, TimeSpan.FromSeconds(5));
|
||||
AwaitSamples(Primary, 20); // ~1 s of polls at 50 ms, inside the 3 s window
|
||||
Assert.Equal(1, _rig.ReadCount(Backup));
|
||||
Assert.Equal("Primary", ActiveEndpoint(actor));
|
||||
Assert.False(LiveOn(Backup));
|
||||
|
||||
_rig.SetLevel(Backup, 250);
|
||||
AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(10));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Anti-flap kept: ServiceLevel 0 inside the dwell of a previous ServiceLevel switch does not move
|
||||
/// the connection back; after the dwell it does.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ServiceLevel0_InsideTheDwell_Waits_ThenSwitches()
|
||||
{
|
||||
_rig.SetLevel(Primary, 90);
|
||||
_rig.SetLevel(Backup, 250);
|
||||
var actor = Start("ga-zero-dwell", Config(Primary, windowSeconds: 30, dwellSeconds: 6), Config(Backup));
|
||||
AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(5)); // connect-time switch starts the dwell
|
||||
var switchedAt = Stopwatch.StartNew();
|
||||
|
||||
_rig.SetLevel(Backup, 0);
|
||||
_rig.SetLevel(Primary, 250);
|
||||
var primaryConnects = _rig.ConnectCount(Primary);
|
||||
Thread.Sleep(2500);
|
||||
Assert.Equal(primaryConnects, _rig.ConnectCount(Primary));
|
||||
Assert.Equal("Backup", ActiveEndpoint(actor));
|
||||
|
||||
AwaitCondition(() => ActiveEndpoint(actor) == "Primary", TimeSpan.FromSeconds(15));
|
||||
Assert.True(switchedAt.Elapsed >= TimeSpan.FromSeconds(4),
|
||||
$"switched back after {switchedAt.Elapsed.TotalSeconds:F1}s, inside the 6s dwell");
|
||||
}
|
||||
|
||||
// ── Trigger 2: a link lost after a Degraded level ──
|
||||
|
||||
/// <summary>
|
||||
/// The QA sequence: Degraded, then unreadable (the server is closing — the streak is cleared),
|
||||
/// then the link drops. The last KNOWN level was Degraded, so the peer is probed first and the
|
||||
/// stopped node is never retried.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void LinkLost_AfterDegradedThenUnknown_ProbesThePeer_AndNeverRetriesTheStoppedNode()
|
||||
{
|
||||
_rig.SetLevel(Primary, 250);
|
||||
_rig.SetLevel(Backup, 240);
|
||||
var actor = Start("ga-lost", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup));
|
||||
AwaitSamples(Primary, 3);
|
||||
|
||||
_rig.SetLevel(Primary, 90);
|
||||
AwaitSamples(Primary, 3);
|
||||
_rig.SetLevel(Primary, (byte?)null);
|
||||
AwaitSamples(Primary, 3);
|
||||
Assert.Equal(0, _rig.ConnectCount(Backup));
|
||||
|
||||
var connectsBefore = _rig.Connects.Count;
|
||||
var sw = Stopwatch.StartNew();
|
||||
_rig.Stop(Primary);
|
||||
|
||||
AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(10));
|
||||
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(RetryCount * _options.ReconnectInterval.TotalSeconds),
|
||||
$"took {sw.Elapsed.TotalSeconds:F1}s — as long as the retry cycle");
|
||||
Assert.DoesNotContain(Primary, _rig.ConnectsAfter(connectsBefore));
|
||||
Assert.Equal("Backup", ActiveEndpoint(actor));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Non-trigger: today's 250/240 pair. A link lost after Healthy levels retries the same node
|
||||
/// <c>FailoverRetryCount</c> times before the connectivity failover — exactly as before #141.
|
||||
/// Positive control: the connectivity failover itself still happens afterwards.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void LinkLost_AfterHealthyLevels_RetriesAsBefore_ThenFailsOver()
|
||||
{
|
||||
_rig.SetLevel(Primary, 250);
|
||||
_rig.SetLevel(Backup, 240);
|
||||
var actor = Start("ga-lost-healthy", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup));
|
||||
AwaitSamples(Primary, 3);
|
||||
|
||||
var connectsBefore = _rig.Connects.Count;
|
||||
_rig.Stop(Primary);
|
||||
|
||||
AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(15));
|
||||
var after = _rig.ConnectsAfter(connectsBefore);
|
||||
Assert.Equal(Enumerable.Repeat(Primary, RetryCount), after.Take(RetryCount));
|
||||
Assert.Equal(Backup, after[RetryCount]);
|
||||
Assert.Equal("Backup", ActiveEndpoint(actor));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Non-trigger (anti-flap): link lost after a Degraded level, but the peer is Degraded too. The
|
||||
/// peer is probed (positive control), then the ordinary retry cycle runs on the same node.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void LinkLost_AfterDegraded_PeerNotHealthy_FallsBackToTheRetryCycle()
|
||||
{
|
||||
_rig.SetLevel(Primary, 250);
|
||||
_rig.SetLevel(Backup, 50);
|
||||
var actor = Start("ga-lost-both", Config(Primary, windowSeconds: 30, dwellSeconds: 0), Config(Backup));
|
||||
AwaitSamples(Primary, 3);
|
||||
_rig.SetLevel(Primary, 90);
|
||||
AwaitSamples(Primary, 3);
|
||||
|
||||
var connectsBefore = _rig.Connects.Count;
|
||||
_rig.Stop(Primary);
|
||||
|
||||
AwaitCondition(() => _rig.ConnectsAfter(connectsBefore).Count(e => e == Primary) >= 2, TimeSpan.FromSeconds(10));
|
||||
var after = _rig.ConnectsAfter(connectsBefore);
|
||||
Assert.Equal(Backup, after[0]); // the probe
|
||||
Assert.Equal(Primary, after[1]); // then the normal retry
|
||||
Assert.False(LiveOn(Backup));
|
||||
Assert.Equal("Primary", ActiveEndpoint(actor));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Non-trigger (anti-flap): a link lost inside the dwell of the previous ServiceLevel switch
|
||||
/// takes the normal retry cycle — no probe. Positive control: the identical sequence after the
|
||||
/// dwell (<see cref="LinkLost_AfterDegradedThenUnknown_ProbesThePeer_AndNeverRetriesTheStoppedNode"/>)
|
||||
/// does probe; here the backup it sits on is retried first.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void LinkLost_InsideTheDwell_TakesTheRetryCycle()
|
||||
{
|
||||
_rig.SetLevel(Primary, 90);
|
||||
_rig.SetLevel(Backup, 250);
|
||||
var actor = Start("ga-lost-dwell", Config(Primary, windowSeconds: 30, dwellSeconds: 120), Config(Backup));
|
||||
AwaitCondition(() => LiveOn(Backup), TimeSpan.FromSeconds(5)); // connect-time switch starts the dwell
|
||||
AwaitSamples(Backup, 3);
|
||||
_rig.SetLevel(Backup, 90);
|
||||
_rig.SetLevel(Primary, 250);
|
||||
AwaitSamples(Backup, 3);
|
||||
|
||||
var connectsBefore = _rig.Connects.Count;
|
||||
_rig.Stop(Backup);
|
||||
|
||||
AwaitCondition(() => _rig.ConnectsAfter(connectsBefore).Length >= 2, TimeSpan.FromSeconds(10));
|
||||
var after = _rig.ConnectsAfter(connectsBefore);
|
||||
Assert.Equal(Backup, after[0]);
|
||||
Assert.Equal(Backup, after[1]);
|
||||
Assert.Equal("Backup", ActiveEndpoint(actor));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Non-trigger: ServiceLevel failover disabled — a lost link never probes, whatever the levels.
|
||||
/// Positive control: the retry cycle runs (the stopped node is attempted again).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void LinkLost_FeatureDisabled_NeverProbes()
|
||||
{
|
||||
_rig.SetLevel(Primary, 0);
|
||||
_rig.SetLevel(Backup, 250);
|
||||
var actor = Start("ga-lost-disabled", Config(Primary, enabled: false), Config(Backup));
|
||||
AwaitCondition(() => LiveOn(Primary), TimeSpan.FromSeconds(5));
|
||||
|
||||
var connectsBefore = _rig.Connects.Count;
|
||||
_rig.Stop(Primary);
|
||||
|
||||
AwaitCondition(() => _rig.ConnectsAfter(connectsBefore).Length >= 2, TimeSpan.FromSeconds(10));
|
||||
Assert.All(_rig.ConnectsAfter(connectsBefore).Take(2), e => Assert.Equal(Primary, e));
|
||||
Assert.Equal(0, _rig.ReadCount(Backup));
|
||||
Assert.Equal("Primary", ActiveEndpoint(actor));
|
||||
}
|
||||
}
|
||||
+20
@@ -42,6 +42,23 @@ internal sealed class ServiceLevelRig
|
||||
|
||||
public int ConnectCount(string endpoint) => Connects.Count(e => e == endpoint);
|
||||
|
||||
/// <summary>Every adapter that opened a session, so a test can kill the live one (ScadaBridge#141).</summary>
|
||||
public ConcurrentQueue<FakeServiceLevelConnection> Sessions { get; } = new();
|
||||
|
||||
/// <summary>
|
||||
/// The server at <paramref name="endpoint"/> goes away: it refuses new sessions and every live
|
||||
/// session on it raises <c>Disconnected</c> — what a stopped OtOpcUa node looks like to the client.
|
||||
/// </summary>
|
||||
public void Stop(string endpoint)
|
||||
{
|
||||
SetReachable(endpoint, false);
|
||||
foreach (var session in Sessions.Where(s => s.Endpoint == endpoint && s.IsConnected))
|
||||
session.RaiseDisconnected();
|
||||
}
|
||||
|
||||
/// <summary>The endpoints sessions were opened on after the first <paramref name="skip"/> connects.</summary>
|
||||
public string[] ConnectsAfter(int skip) => Connects.Skip(skip).ToArray();
|
||||
|
||||
public int ReadCount(string endpoint) => Reads.GetValueOrDefault(endpoint);
|
||||
}
|
||||
|
||||
@@ -72,9 +89,12 @@ internal sealed class FakeServiceLevelConnection : IDataConnection, IServiceLeve
|
||||
if (!_rig.IsReachable(Endpoint))
|
||||
return Task.FromException(new InvalidOperationException($"{Endpoint} refused the session"));
|
||||
_connected = true;
|
||||
_rig.Sessions.Enqueue(this);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public bool IsConnected => _connected;
|
||||
|
||||
public Task DisconnectAsync(CancellationToken cancellationToken = default)
|
||||
{
|
||||
_connected = false;
|
||||
|
||||
+50
@@ -208,4 +208,54 @@ public sealed class OpcUaServiceLevelTests : TestKit
|
||||
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionEndpoint)
|
||||
&& c.GetArguments() is ["today", "Connected to backup"]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#141 end to end, the QA Warm gate's step 4a: the connection is on the Warm Primary
|
||||
/// (A, 250) with the standby (B, 90) as backup, both on the DEFAULT policy (10 s window, 60 s
|
||||
/// dwell) and the default 5 s reconnect interval with three retries. A is stopped gracefully the
|
||||
/// way OtOpcUa does it — the peer promotes to 250, A reports ServiceLevel 0 for a moment, then
|
||||
/// the server goes away. The connection must be live on B within seconds; the connectivity path
|
||||
/// alone (keep-alive + three 5 s retries of the stopped A) needs well over 15 s.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task RealServerPair_GracefulStopOfTheWarmPrimary_FailsOverWithinSeconds()
|
||||
{
|
||||
var a = await SecureOpcUaTestServer.StartAsync();
|
||||
await using var b = await SecureOpcUaTestServer.StartAsync();
|
||||
try
|
||||
{
|
||||
var factory = new DataConnectionFactory(NullLoggerFactory.Instance, Options.Create(ClientOptions()));
|
||||
var options = new DataConnectionOptions(); // production defaults: 5 s reconnect, 2 s poll, 15 s probe
|
||||
a.SetServiceLevel(250);
|
||||
b.SetServiceLevel(90);
|
||||
var health = Substitute.For<ISiteHealthCollector>();
|
||||
var primary = Flat(a.Url); // null ServiceLevelFailover = defaults (enabled, 10 s, 60 s)
|
||||
|
||||
bool ConnectedTo(string endpoint) => health.ReceivedCalls().Any(c =>
|
||||
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionEndpoint)
|
||||
&& c.GetArguments() is ["stop", var label] && (string)label! == $"Connected to {endpoint}");
|
||||
|
||||
var actor = Sys.ActorOf(Props.Create(() => new DataConnectionActor(
|
||||
"stop", factory.Create("OpcUa", primary), options, health, factory, "OpcUa",
|
||||
primary, Flat(b.Url), 3, null)));
|
||||
AwaitCondition(() => ConnectedTo("primary"), TimeSpan.FromSeconds(30));
|
||||
await Task.Delay(TimeSpan.FromSeconds(3)); // a few Healthy polls: no connect-time effect left
|
||||
|
||||
var sw = System.Diagnostics.Stopwatch.StartNew();
|
||||
b.SetServiceLevel(250); // the peer promotes (0.2 s on QA)
|
||||
a.SetServiceLevel(0); // the stopping node reports Maintenance...
|
||||
await Task.Delay(TimeSpan.FromSeconds(1));
|
||||
await a.DisposeAsync(); // ...and is gone well inside the 10 s window
|
||||
|
||||
AwaitCondition(() => ConnectedTo("backup"), TimeSpan.FromSeconds(30));
|
||||
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(10),
|
||||
$"failover took {sw.Elapsed.TotalSeconds:F1}s (QA before #141: 66 s)");
|
||||
Assert.Equal("Backup", (await actor.Ask<DataConnectionHealthReport>(
|
||||
new DataConnectionActor.GetHealthReport(), TimeSpan.FromSeconds(5))).ActiveEndpoint);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await a.DisposeAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user