fix(dcl): fail over within seconds when the connected OPC UA server is going away (#141)
QA Warm gate (lmxopcua#907, 2026-11-13): a graceful stop of the Warm Primary reported ServiceLevel 0 for ~6 s - shorter than the 10 s degraded window - the next unreadable sample cleared the streak, and the connection retried the stopped node FailoverRetryCount times: Good on the peer after 66 s, peer Active in 0.2 s. Two going-away triggers on top of #138, both still requiring a Healthy (>= 200), higher level on the other endpoint and honouring the minimum dwell: - ServiceLevel 0 (Part 4 Maintenance) probes the other endpoint at once instead of waiting out the window - once per degraded streak, so a non-Healthy peer is re-probed once per window, not every poll. Other Degraded levels (standby 90/50, booting 1) still wait the window. - A lost link whose session's last KNOWN ServiceLevel was < 200 (unreadable samples don't overwrite it) probes the other endpoint before the first reconnect attempt, holding the reconnect timer. Healthy -> switch and reconnect at zero delay, the stopped node is never retried; otherwise the ordinary retry cycle runs (less the probe's time). After only Healthy levels (250/240, Unset), inside the dwell, or with the feature off: unchanged retry-then-failover. Tests: DataConnectionGoingAwayFailoverTests (10 cases, each trigger and non-trigger with positive controls, connect-order artifact assertions) and a real in-process OPC UA server pair graceful-stop e2e on production defaults: 1.1 s (lost-link trigger alone 12.1 s, neither 33.1 s -> fails the 10 s bound). 13 mutants, all killed. DCL tests 608/608 (3 full runs); solution 0 warnings. Docs: Component-DataConnectionLayer, components/DataConnectionLayer. Not deployed; QA live re-check pending. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -121,6 +121,8 @@ When `BackupConnectionDetails` is provided, the actor tracks consecutive failure
|
||||
|
||||
**ServiceLevel-aware failover (OPC UA, Gitea #138).** A third trigger, for redundant OPC UA pairs whose standby keeps a healthy session but serves Bad data (OtOpcUa Warm, lmxopcua#717). It lives in `Actors/DataConnectionActor.ServiceLevel.cs`. While Connected, and only when the adapter implements `IServiceLevelReadable`, the connection has a backup and the primary config's `ServiceLevelFailover.Enabled` is true, a periodic timer reads `Server_ServiceLevel` (`i=2267`). A level below 200 starts a degraded streak; a Healthy or unreadable sample ends it. After the degraded window (default 10 s) — or at once on the first sample after a (re)connect — the actor probes the other endpoint with a throwaway adapter from `DataConnectionFactory.Create` (connect, read, dispose; bounded by `ServiceLevelProbeTimeout`). It switches only if that endpoint reports ≥ 200 and higher, and not within the minimum dwell (default 60 s) of the previous ServiceLevel switch; otherwise it stays and re-arms the window. The switch is `SwitchToOtherEndpoint()` — the helper the two connectivity triggers now share — followed by `BecomeReconnecting(ReconnectCause.ServiceLevelFailover)` (zero delay, no unstable accounting). Poll and probe results carry the adapter generation, so a result from a session that has ended is dropped. Today's OtOpcUa (250/240) never starts a streak. Full rule set: `docs/requirements/Component-DataConnectionLayer.md` → *ServiceLevel-aware failover*.
|
||||
|
||||
**A server going away (Gitea #141).** Two shortcuts on top of the window, both in the same file and both still bound by "Healthy and higher" and the dwell. (1) A sample of `MaintenanceServiceLevel` (0) probes the other endpoint at once instead of waiting out the window — once per degraded streak (`_goingAwayProbeStarted`), so a peer that is not Healthy is re-probed once per window, not every poll. (2) `BecomeReconnecting` for a lost link captures the session's `_lastKnownServiceLevel` (the last level actually read; unknown samples don't overwrite it) before the watch resets. If that level was below 200, `TryStartDisconnectProbe` probes the other endpoint and **holds** the reconnect timer. `HandleDisconnectProbed` (Reconnecting state; the probe result carries `AfterDisconnect: true`) either switches (`SwitchForServiceLevel`, reconnect at zero delay) or releases the ordinary retry cycle on the same endpoint, less the time the probe took. A link lost after Healthy levels, inside the dwell, or with the feature off takes the ordinary retry cycle as before. Why: an OtOpcUa graceful stop reports 0 for only ~6 s and then the session closes, so on QA the connection spent 66 s retrying the stopped node.
|
||||
|
||||
### Write path
|
||||
|
||||
Writes are fire-and-forget from the script's view only in the sense that there is no store-and-forward. The `DataConnectionActor` pipes `_adapter.WriteAsync` back to the original sender:
|
||||
|
||||
Reference in New Issue
Block a user