Merge fix/670-followups: outage Bad stamped at detection; post-recovery replay guard (#670 live-gate F1/F2)

This commit is contained in:
Joseph Doherty
2026-10-11 09:16:03 -04:00
5 changed files with 277 additions and 5 deletions
+10 -2
View File
@@ -66,12 +66,19 @@ The session keep-alive (`KeepAliveInterval`, default 5 s) is the liveness signal
subscription-only driver never calls `ReadAsync`, so without this its read stamp froze at connect time.
- **Bad keep-alive** ⇒ health `Degraded` (`LastError = "keep-alive: <status>"`, read stamp preserved), host status
`Stopped`, and — **once per outage**, not per keep-alive — every subscribed ref is published `BadNoCommunication`
with its last value retained, plus a quality-only `BadNoCommunication` annotation (#626 pattern, no alarm-state
with its last value retained and **source + server timestamp = the detection time** (live-gate F1: stamped with the
last Good's source time it collided with that row's `(TagId, SourceTimeUtc)` history key and was never stored), plus a quality-only `BadNoCommunication` annotation (#626 pattern, no alarm-state
change, no `/alerts` row) on every subscribed alarm condition.
- **The session survives** (a network blip; the keep-alive turns Good again) ⇒ recovered in place: health `Healthy`,
the refs still Bad are re-read once and published Good (a server whose subscription survived does not resend an
unchanged value, and a `(value, status)` dedup downstream lets the identical value through), alarm quality goes
back Good and a `ConditionRefresh` re-reports anything missed. A notification that lands first wins over the read.
For a bounded **replay window** after such a recovery — max(30 s, 2 x the longest subscription keep-alive window) —
a notification whose source time is OLDER than the ref's last Good upstream source time (raised by the recovery
read) is dropped: the SDK was seen delivering a missed one 4–5 s late, which regressed the node (live-gate F2).
A ref disarms on its first in-order (equal or newer) notification. Outside that window every notification is
accepted whatever its timestamp, so an upstream clock that steps backwards (NTP, a PLC clock reset) never
freezes a node. The outage Bad never moves the comparison; a re-initialise resets it.
- **The session does not come back** (an upstream restart: the server forgot the session) ⇒ the driver stays
`Degraded` with its read stamp frozen, which is exactly the #527 comms-loss watchdog's trip condition. After 90 s
`DriverInstanceActor` runs `ShutdownAsync` + `InitializeAsync` (retried every 10 s while the server is down) and
@@ -228,7 +235,8 @@ every `InitializeAsync`, including comms-loss reconnects). The discovery channel
Session/publish events are published on the shared OTel meter as the counter
`otopcua.driver.opcuaclient.events`, tagged `driver_instance` (the driver instance id) and `event` =
`publish_response`, `notification` (items), `data_change`, `bad_status_value`, `publish_error`,
`keepalive_miss`, `keepalive_recovered` (a bad keep-alive followed by a good one on the same session). A climbing
`keepalive_miss`, `keepalive_recovered` (a bad keep-alive followed by a good one on the same session),
`stale_notification_dropped` (a late replay dropped inside the post-recovery window, see above). A climbing
`keepalive_miss` / `keepalive_recovered` rate is the signature of a flapping upstream (`reconnect_*` were removed with
the reconnect handler, #670). They are not part of `DriverHealth` or the
`/hosts` panel.