fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)

QA Warm gate (lmxopcua#907, 2026-11-13): a graceful stop of the Warm Primary
reported ServiceLevel 0 for ~6 s - shorter than the 10 s degraded window - the
next unreadable sample cleared the streak, and the connection retried the stopped
node FailoverRetryCount times: Good on the peer after 66 s, peer Active in 0.2 s.

Two going-away triggers on top of #138, both still requiring a Healthy (>= 200),
higher level on the other endpoint and honouring the minimum dwell:
- ServiceLevel 0 (Part 4 Maintenance) probes the other endpoint at once instead
  of waiting out the window - once per degraded streak, so a non-Healthy peer is
  re-probed once per window, not every poll. Other Degraded levels (standby 90/50,
  booting 1) still wait the window.
- A lost link whose session's last KNOWN ServiceLevel was < 200 (unreadable
  samples don't overwrite it) probes the other endpoint before the first
  reconnect attempt, holding the reconnect timer. Healthy -> switch and reconnect
  at zero delay, the stopped node is never retried; otherwise the ordinary retry
  cycle runs (less the probe's time). After only Healthy levels (250/240, Unset),
  inside the dwell, or with the feature off: unchanged retry-then-failover.

Tests: DataConnectionGoingAwayFailoverTests (10 cases, each trigger and
non-trigger with positive controls, connect-order artifact assertions) and a real
in-process OPC UA server pair graceful-stop e2e on production defaults: 1.1 s
(lost-link trigger alone 12.1 s, neither 33.1 s -> fails the 10 s bound).
13 mutants, all killed. DCL tests 608/608 (3 full runs); solution 0 warnings.
Docs: Component-DataConnectionLayer, components/DataConnectionLayer.

Not deployed; QA live re-check pending.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 09:36:59 -05:00
parent b19836788b
commit c4ec51e89a
7 changed files with 562 additions and 17 deletions
@@ -160,14 +160,19 @@ The failover above is driven only by the link misbehaving. A redundant OPC UA se
|---|---|---|
| Healthy | 200–255 | Active: 250 Primary / 240 Secondary (today and under Warm), 200 Active-stale |
| Degraded | 100–199, and anything below 200 here | Active critically degraded 100; Warm standby 90 (healthy) / 50 (degraded) |
| NoData / Maintenance | 1 / 0 | Warm booting 1; down / endpoint failed 0 |
| NoData / Maintenance | 1 / 0 | Warm booting 1; down / endpoint failed / shutting down 0 (skips the window, #141) |
**Rule.** While Connected, the actor reads the connected endpoint's ServiceLevel every `ServiceLevelPollInterval` (2 s). A level **below 200** starts a *degraded streak*; any level of 200 or more, or an unreadable level (Bad status, no such node, read failure), ends it. When the streak has lasted the connection's **degraded window** (default 10 s), the actor probes the **other** endpoint — a short-lived session built from that endpoint's configuration: connect, read `i=2267`, close, bounded by `ServiceLevelProbeTimeout` (15 s). It fails over only when the other endpoint answers with a **Healthy level (≥ 200) higher than the connected one**. Otherwise (other endpoint unreachable, or also below 200) it **stays put** and the window re-arms, so the other endpoint is probed at most once per window.
**Rule.** While Connected, the actor reads the connected endpoint's ServiceLevel every `ServiceLevelPollInterval` (2 s). A level **below 200** starts a *degraded streak*; any level of 200 or more, or an unreadable level (Bad status, no such node, read failure), ends it. When the streak has lasted the connection's **degraded window** (default 10 s) — or at once for a level of 0, see *A server going away* below — the actor probes the **other** endpoint — a short-lived session built from that endpoint's configuration: connect, read `i=2267`, close, bounded by `ServiceLevelProbeTimeout` (15 s). It fails over only when the other endpoint answers with a **Healthy level (≥ 200) higher than the connected one**. Otherwise (other endpoint unreachable, or also below 200) it **stays put** and the window re-arms, so the other endpoint is probed at most once per window.
- **Connect-time preference.** The first sample after every (re)connect is evaluated at once, without the window: a connection that comes up on a Degraded endpoint while the other is Healthy moves straight over. Because the trigger needs the connected level below 200, a connection to a 240 Secondary is never moved to a 250 Primary.
- **Anti-flap.** A ServiceLevel switch is not repeated within the connection's **minimum dwell** (default 60 s), connect-time preference included. Together with the "Healthy and higher" requirement and the unbroken-streak rule (one Healthy sample resets it) this stops a pair that flips roles from bouncing the connection.
- **Same switch as the connectivity failover.** The switch flips the active endpoint and swaps the adapter through the same helper as the two connection-driven triggers, resets both failure counters, then re-enters the reconnect cycle immediately with the unstable-disconnect accounting suppressed (it is a deliberate swap, like an in-place reconfigure). `ReSubscribeAll` / `ReSubscribeAllAlarms` then re-issue every subscription on the new endpoint; tags go Bad for the swap exactly as on any failover. Site event log: `Failover from Primary to Backup (ServiceLevel)` (Warning) with the two levels.
- **Scope.** Only adapters implementing `IServiceLevelReadable` (the OPC UA adapter) are watched, and only when the connection has a backup and the setting is enabled. MxGateway connections, single-endpoint connections and disabled connections never read a ServiceLevel. The ServiceLevel read never raises the adapter's `Disconnected` event: a dead session stays the keep-alive's business.
- **A server going away (Gitea #141).** A graceful stop of a redundant OtOpcUa node reports ServiceLevel **0** for a few seconds and then closes the session — shorter than the degraded window, and the unreadable sample that follows ends the streak. Without a shortcut the connection fell back to the connectivity path and retried the stopped node `FailoverRetryCount` times (QA Warm gate 2026-11-13: Good on the peer after 66 s, though the peer was Active in 0.2 s). Two rules close that gap, both still requiring a **Healthy, higher** level on the other endpoint and honouring the **dwell**:
- **ServiceLevel 0 skips the window.** A sample of 0 (Part 4 *Maintenance*: not available for use) probes the other endpoint at once. Only 0 does this — 1 (booting) and the standby levels still wait out the window. It happens once per degraded streak: if the other endpoint is not Healthy, it is re-probed once per window as usual, not on every poll.
- **A lost link after a Degraded level probes the other endpoint first.** When the link drops and the last ServiceLevel this session actually read (unreadable samples don't replace it) was below 200, the actor probes the other endpoint **before the first reconnect attempt**. If it is Healthy and higher, the connection switches straight to it and reconnects at once — the stopped node is not retried. Otherwise the ordinary retry cycle starts on the same endpoint (the probe's time counts against the first reconnect interval). Site event log: `Failover from Primary to Backup (ServiceLevel)` with *link lost after ServiceLevel N*.
A link lost after only Healthy levels (250/240 pairs, `Mode=Unset` without a 0), inside the dwell of the previous ServiceLevel switch, or with the setting disabled keeps today's retry-then-failover behaviour exactly. Expected failover on a graceful stop: the next ServiceLevel poll plus one probe — a few seconds.
- **Behaviour against today's servers.** A pair that only ever reports Healthy levels (OtOpcUa 250/240 without Warm, or a server that always reports 255) never starts a streak: no probe, no switch — the default-on setting changes nothing there. It does act on a non-Warm OtOpcUa node that has dropped to 100/110 or 0/10 while its peer is ≥ 200, which is the desired move.
**Configuration** (per connection, on the **primary** endpoint's OPC UA JSON — the backup's copy is ignored; edited in the Central UI's *Backup endpoint* card next to *Failover Retry Count*; via the CLI it is part of `--primary-config`):