fix(drivers): detect mid-run comms loss and recover from it (#527)

DriverInstanceActor.DisconnectObserved had no production sender. The only paths
into Reconnecting were a failed *initial* connect and an operator's Force
Reconnect, so a driver that connected and then lost its device stayed Connected
indefinitely: publishing Bad for every tag, never re-initialising, and NOT
recovering when the device returned. PublishHealthSnapshot read GetHealth() for
reporting only and never acted on a driver reporting itself unhealthy.

Both comms-loss quality paths -- #477's native-condition annotation and #481's
scripted-alarm input fan-out -- hang off ConnectivityChanged, so both were
correctly wired to a signal that never fired.

The Connected health tick now evaluates a watchdog and self-sends
DisconnectObserved when the driver reports a non-Healthy state AND its
LastSuccessfulRead has been frozen for DefaultCommsLossTimeout (90s).

The conjunction is the design, and the issue's own proposal -- "non-Healthy for
N consecutive polls" -- is wrong. Degraded means *connected but some data is
bad*; several drivers write health per tag inside the read loop
(ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever
against a perfectly reachable device, and a state-only watchdog would reconnect
that driver every 90s indefinitely -- worse than the bug. LastSuccessfulRead is
the honest evidence: preserved across Degraded/Faulted writes, advanced by any
success. The state check remains the second conjunct so an idle Healthy driver
with nothing subscribed is not declared dead for not reading.

Verified across all 12 drivers that a sustained failure does reach a non-Healthy
state -- nine poll drivers via Degraded, MQTT via live connection state, Galaxy
via its reconnect supervisor. CalculationDriver reports Healthy while running
and so can never trip, which is correct: it has no transport.

Also: the Reconnecting retry now calls ShutdownAsync before InitializeAsync.
ModbusDriver (_transport) and MqttDriver (_connection) overwrite an
IAsyncDisposable field inside InitializeAsync without disposing the previous
one, so re-initialising without a teardown leaks a socket per attempt -- latent
while only a manual Force Reconnect reached this path, but the watchdog makes it
automatic and unbounded (~360 leaks for an hour-long outage at the 10s retry).

Tests: 8 new in DriverInstanceActorCommsLossWatchdogTests, each absence
assertion carrying a positive control (a proven count of processed health-poll
ticks). Falsified five ways -- dropping either conjunct reds exactly the test
written for it, and removing the teardown, the null baseline, or the watchdog
call each reds its own guard.

Live gate PASSED on docker-dev (Modbus + otopcua-pymodbus-standard), run twice:
  - fixture stopped -> watchdog fires at ~108s, "declaring comms lost"
  - fan-out observed at Debug: "pushed BadNoCommunication for 2 mux ref(s)"
    (#481 Layer 4) and the #477 native-condition annotation on a driver
    owning conditions
  - fixture restarted -> reconnected in ~8s with NO node restart; both the raw
    (ns=2) and UNS (ns=3) NodeIds read Good with fresh timestamps

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
2026-07-31 10:58:38 -04:00
parent 9fa701b14b
commit 5a7a08deca
6 changed files with 587 additions and 21 deletions
+16 -9
View File
@@ -183,15 +183,22 @@ Two things worth knowing before changing this:
and is the correct reading of "this input's source is unreachable", but it is a behaviour change beyond
scripted alarms.
⚠️ **The signal this hangs off does not currently fire for a mid-run device loss — see #527.** The #481 live
gate on docker-dev found that `DriverInstanceActor.DisconnectObserved` has **no production sender**: the only
paths into `Reconnecting` are a failed *initial* connect and an operator Force Reconnect. A Modbus driver
whose fixture was stopped stayed `Connected` for >10 minutes publishing per-tag Bad, and did not recover when
the fixture came back until the node was restarted. So the fan-out above — and the native `#477` Layer 2
annotation it mirrors — are correctly wired to a signal that a dying device never raises. What partially
covers the gap today is that Modbus publishes Bad *per tag* rather than going silent, so #478 Layer 3 does
carry Bad into scripted input quality for that driver; that is a per-driver accident, not a guarantee, and it
does nothing for native conditions or for recovery. Closing #527 is what makes this path reachable.
**The signal this hangs off now fires for a mid-run device loss (#527, closed).** The #481 live gate on
docker-dev found that `DriverInstanceActor.DisconnectObserved` had **no production sender**: the only paths
into `Reconnecting` were a failed *initial* connect and an operator Force Reconnect. A Modbus driver whose
fixture was stopped stayed `Connected` for >10 minutes publishing per-tag Bad, and did not recover when the
fixture came back until the node was restarted. So the fan-out above — and the native #477 Layer 2
annotation it mirrors — were correctly wired to a signal that a dying device never raised. #527 added the
comms-loss watchdog on the Connected health tick, which self-sends `DisconnectObserved` once a driver has
gone 90 s without a successful read *while* reporting itself non-Healthy. See
[`DriverLifecycle.md`](DriverLifecycle.md) §"The comms-loss watchdog" for why the trip condition is that
conjunction and not simply "non-Healthy for N polls" — the naive rule would reconnect a healthy device's
driver every 90 s forever whenever one authored address is unreadable.
Note the partial cover that existed before, because it explains why *some* of this appeared to work: Modbus
publishes Bad *per tag* rather than going silent, so #478 Layer 3 did carry Bad into scripted input quality
for that driver. That was a per-driver accident, not a guarantee, and it did nothing for native conditions
or for recovery.
**Cold start vs. a driver reporting Bad (#478 Finding 3, closed by #481).** `WorstInputStatus` used to skip
every input whose *value* was null as "no data yet". A comms-lost driver produces exactly that shape — Bad