fix(drivers): detect mid-run comms loss and recover from it (#527)
DriverInstanceActor.DisconnectObserved had no production sender. The only paths into Reconnecting were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed Connected indefinitely: publishing Bad for every tag, never re-initialising, and NOT recovering when the device returned. PublishHealthSnapshot read GetHealth() for reporting only and never acted on a driver reporting itself unhealthy. Both comms-loss quality paths -- #477's native-condition annotation and #481's scripted-alarm input fan-out -- hang off ConnectivityChanged, so both were correctly wired to a signal that never fired. The Connected health tick now evaluates a watchdog and self-sends DisconnectObserved when the driver reports a non-Healthy state AND its LastSuccessfulRead has been frozen for DefaultCommsLossTimeout (90s). The conjunction is the design, and the issue's own proposal -- "non-Healthy for N consecutive polls" -- is wrong. Degraded means *connected but some data is bad*; several drivers write health per tag inside the read loop (ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever against a perfectly reachable device, and a state-only watchdog would reconnect that driver every 90s indefinitely -- worse than the bug. LastSuccessfulRead is the honest evidence: preserved across Degraded/Faulted writes, advanced by any success. The state check remains the second conjunct so an idle Healthy driver with nothing subscribed is not declared dead for not reading. Verified across all 12 drivers that a sustained failure does reach a non-Healthy state -- nine poll drivers via Degraded, MQTT via live connection state, Galaxy via its reconnect supervisor. CalculationDriver reports Healthy while running and so can never trip, which is correct: it has no transport. Also: the Reconnecting retry now calls ShutdownAsync before InitializeAsync. ModbusDriver (_transport) and MqttDriver (_connection) overwrite an IAsyncDisposable field inside InitializeAsync without disposing the previous one, so re-initialising without a teardown leaks a socket per attempt -- latent while only a manual Force Reconnect reached this path, but the watchdog makes it automatic and unbounded (~360 leaks for an hour-long outage at the 10s retry). Tests: 8 new in DriverInstanceActorCommsLossWatchdogTests, each absence assertion carrying a positive control (a proven count of processed health-poll ticks). Falsified five ways -- dropping either conjunct reds exactly the test written for it, and removing the teardown, the null baseline, or the watchdog call each reds its own guard. Live gate PASSED on docker-dev (Modbus + otopcua-pymodbus-standard), run twice: - fixture stopped -> watchdog fires at ~108s, "declaring comms lost" - fan-out observed at Debug: "pushed BadNoCommunication for 2 mux ref(s)" (#481 Layer 4) and the #477 native-condition annotation on a driver owning conditions - fixture restarted -> reconnected in ~8s with NO node restart; both the raw (ns=2) and UNS (ns=3) NodeIds read Good with fresh timestamps Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
@@ -219,6 +219,57 @@ So a driver whose `InitializeAsync` disposes anything on its failure path must b
|
||||
`MTConnectDriver` does today, because it is the only one that tears down inside a `catch`. The
|
||||
shutdown paths deliberately keep the single trailing write.
|
||||
|
||||
### The comms-loss watchdog (Gitea #527) — `GetHealth()` is now load-bearing
|
||||
|
||||
`GetHealth()` used to be **reporting-only**: `DriverInstanceActor.PublishHealthSnapshot` read it every 30 s
|
||||
and forwarded it to the AdminUI, and nothing ever acted on a driver that reported itself unhealthy. That
|
||||
left `DisconnectObserved` with **no production sender at all** — the only paths into `Reconnecting` were a
|
||||
failed *initial* connect and an operator's Force Reconnect. A driver that connected and then lost its
|
||||
device stayed `Connected` forever: publishing Bad for every tag, never re-initialising, and **not
|
||||
recovering when the device came back** (live-confirmed against a stopped Modbus fixture, which needed a
|
||||
node restart). Both comms-loss quality paths — #477's native-condition annotation and #481's
|
||||
scripted-alarm input fan-out — hang off `ConnectivityChanged`, so both were wired to a signal that never
|
||||
fired.
|
||||
|
||||
The `Connected`-state health tick now also evaluates a watchdog, and self-sends `DisconnectObserved` when
|
||||
**both** of these hold:
|
||||
|
||||
1. the driver reports a state other than `Healthy`, **and**
|
||||
2. `DriverHealth.LastSuccessfulRead` has not advanced for `DefaultCommsLossTimeout` (90 s = three ticks),
|
||||
measured from the Connected entry when the driver has never reported a successful read.
|
||||
|
||||
**The conjunction is the whole design, and "non-Healthy for N polls" is the wrong rule.** `Degraded` means
|
||||
*connected but some data is bad* — the enum says so. Several drivers publish health **per tag inside their
|
||||
read loop** (`ModbusDriver.ReadAsync` writes `Healthy` or `Degraded` once per reference), so a single
|
||||
unreadable address leaves the driver latched `Degraded` whenever it happens to be read last, indefinitely,
|
||||
against a device that is perfectly reachable. A state-only watchdog would tear that driver down every 90 s
|
||||
forever — strictly worse than the bug it fixes. `LastSuccessfulRead` is the honest liveness evidence: every
|
||||
driver advances it on any successful read and *preserves* it across `Degraded`/`Faulted` writes, so one bad
|
||||
tag among good ones keeps it moving while a device that is genuinely gone freezes it. The state check is
|
||||
still needed as the second conjunct, because a `Healthy` driver with nothing subscribed legitimately never
|
||||
reads and must not be declared dead for being idle. Both halves are pinned by their own test in
|
||||
`DriverInstanceActorCommsLossWatchdogTests`, each of which goes red when the *other* conjunct is deleted.
|
||||
|
||||
Two consequences worth knowing:
|
||||
|
||||
- **`CalculationDriver` can never trip it, correctly.** It reports `Healthy` whenever it is running and has
|
||||
no transport to be degraded about. Every other driver reaches a non-Healthy state on sustained failure:
|
||||
the nine poll drivers via `Degraded`, MQTT via its live connection state, Galaxy via its reconnect
|
||||
supervisor. This was surveyed across all 12 before relying on it — it is the assumption the whole
|
||||
mechanism rests on.
|
||||
- **A device that accepts a connection but serves no data will cycle**, reconnecting once per threshold.
|
||||
That is the correct reading of a broken device, it is rate-limited, and it is visible in the health
|
||||
snapshot's error count — but do not mistake it for a bug when reading logs.
|
||||
|
||||
**The reconnect retry now tears down before re-initialising.** `Reconnecting`'s `RetryConnect` calls
|
||||
`ShutdownAsync` before `InitializeAsync` — the sequence every driver's own `ReinitializeAsync` already
|
||||
uses — because several drivers **overwrite** an `IAsyncDisposable` transport field inside `InitializeAsync`
|
||||
without disposing the previous one (`ModbusDriver._transport`, `MqttDriver._connection`). Re-initialising
|
||||
without a teardown leaks one socket per attempt. That was latent while only a manual Force Reconnect
|
||||
re-entered this path; the watchdog makes it automatic and unbounded, so a device down for an hour would
|
||||
have leaked ~360 transports at the 10 s retry interval. The shutdown is best-effort: a driver that throws
|
||||
on it still gets its initialise attempt.
|
||||
|
||||
---
|
||||
|
||||
## IDriverConfigEditor — custom AdminUI config editor (plug-point)
|
||||
|
||||
Reference in New Issue
Block a user