fix(drivers): detect mid-run comms loss and recover from it (#527)
DriverInstanceActor.DisconnectObserved had no production sender. The only paths into Reconnecting were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed Connected indefinitely: publishing Bad for every tag, never re-initialising, and NOT recovering when the device returned. PublishHealthSnapshot read GetHealth() for reporting only and never acted on a driver reporting itself unhealthy. Both comms-loss quality paths -- #477's native-condition annotation and #481's scripted-alarm input fan-out -- hang off ConnectivityChanged, so both were correctly wired to a signal that never fired. The Connected health tick now evaluates a watchdog and self-sends DisconnectObserved when the driver reports a non-Healthy state AND its LastSuccessfulRead has been frozen for DefaultCommsLossTimeout (90s). The conjunction is the design, and the issue's own proposal -- "non-Healthy for N consecutive polls" -- is wrong. Degraded means *connected but some data is bad*; several drivers write health per tag inside the read loop (ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever against a perfectly reachable device, and a state-only watchdog would reconnect that driver every 90s indefinitely -- worse than the bug. LastSuccessfulRead is the honest evidence: preserved across Degraded/Faulted writes, advanced by any success. The state check remains the second conjunct so an idle Healthy driver with nothing subscribed is not declared dead for not reading. Verified across all 12 drivers that a sustained failure does reach a non-Healthy state -- nine poll drivers via Degraded, MQTT via live connection state, Galaxy via its reconnect supervisor. CalculationDriver reports Healthy while running and so can never trip, which is correct: it has no transport. Also: the Reconnecting retry now calls ShutdownAsync before InitializeAsync. ModbusDriver (_transport) and MqttDriver (_connection) overwrite an IAsyncDisposable field inside InitializeAsync without disposing the previous one, so re-initialising without a teardown leaks a socket per attempt -- latent while only a manual Force Reconnect reached this path, but the watchdog makes it automatic and unbounded (~360 leaks for an hour-long outage at the 10s retry). Tests: 8 new in DriverInstanceActorCommsLossWatchdogTests, each absence assertion carrying a positive control (a proven count of processed health-poll ticks). Falsified five ways -- dropping either conjunct reds exactly the test written for it, and removing the teardown, the null baseline, or the watchdog call each reds its own guard. Live gate PASSED on docker-dev (Modbus + otopcua-pymodbus-standard), run twice: - fixture stopped -> watchdog fires at ~108s, "declaring comms lost" - fan-out observed at Debug: "pushed BadNoCommunication for 2 mux ref(s)" (#481 Layer 4) and the #477 native-condition annotation on a driver owning conditions - fixture restarted -> reconnected in ~8s with NO node restart; both the raw (ns=2) and UNS (ns=3) NodeIds read Good with fresh timestamps Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
@@ -210,6 +210,38 @@ nothing ever writes a `DriverHostStatus` row (the table was re-created in the v3
|
||||
confirm intent before deleting). The separable `OnHostStatusChanged` event is the useful half. Tracked
|
||||
as Gitea **#521**.
|
||||
|
||||
### Comms-loss detection: `GetHealth()` is load-bearing (Gitea #527, closed)
|
||||
|
||||
Until #527, **`DriverInstanceActor.DisconnectObserved` had no production sender.** The only paths into
|
||||
`Reconnecting` were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected
|
||||
and then lost its device stayed `Connected` indefinitely — publishing Bad for every tag, never
|
||||
re-initialising, and **not recovering when the device returned** (live-confirmed: a Modbus driver against a
|
||||
stopped fixture needed a node restart). `PublishHealthSnapshot` read `GetHealth()` for *reporting only*.
|
||||
|
||||
The `Connected` health tick now runs a watchdog that self-sends `DisconnectObserved` when a driver reports a
|
||||
non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. Things to know before touching it:
|
||||
|
||||
- ⚠️ **The conjunction is the design — "non-Healthy for N polls" is wrong and was the issue's own proposal.**
|
||||
`Degraded` means *connected but some data is bad*, and several drivers write health **per tag inside the
|
||||
read loop** (`ModbusDriver.ReadAsync`), so one unreadable address latches `Degraded` forever against a
|
||||
perfectly reachable device. State-only tripping would reconnect that driver every 90 s indefinitely.
|
||||
`LastSuccessfulRead` is the honest evidence — preserved across `Degraded`/`Faulted` writes, advanced by any
|
||||
success. Each conjunct has a test that goes red when the *other* is deleted.
|
||||
- **The mechanism rests on every driver reaching a non-Healthy state on sustained failure.** Surveyed across
|
||||
all 12: nine poll drivers via `Degraded`, MQTT via live connection state, Galaxy via its reconnect
|
||||
supervisor. `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has
|
||||
no transport. Re-check this if a new driver lands.
|
||||
- ⚠️ **The reconnect retry MUST tear down before re-initialising.** `ModbusDriver` (`_transport`) and
|
||||
`MqttDriver` (`_connection`) overwrite an `IAsyncDisposable` field inside `InitializeAsync` without
|
||||
disposing the old one, so re-initialising without `ShutdownAsync` leaks a socket per attempt — ~360 for an
|
||||
hour-long outage at the 10 s retry interval. Latent while only a manual Force Reconnect reached this path.
|
||||
- The threshold is hardcoded (`DefaultCommsLossTimeout`), matching `HealthPollInterval` /
|
||||
`DefaultReconnectInterval`. Unlike the #523 drift detector this generates **no** traffic on a healthy
|
||||
system — it is a local `GetHealth()` read, and only reaches the network once a driver has already stopped
|
||||
reading.
|
||||
|
||||
See `docs/DriverLifecycle.md` §"The comms-loss watchdog".
|
||||
|
||||
## mxaccessgw
|
||||
|
||||
The gateway lives in a sibling repo at `~/Desktop/MxAccessGateway/` (`mxaccessgw` — see [Related Projects](#related-projects-scadaproj-umbrella)). See `docs/v2/Galaxy.ParityRig.md` for the gw setup recipe (build, API key provisioning via `apikey create-key`, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its `gateway.md`) is the canonical MxAccess API doc; the standalone `mxaccess_documentation.md` previously kept in this repo retired in PR 7.3.
|
||||
|
||||
Reference in New Issue
Block a user