fix(opcuaclient): #670 live-gate follow-ups — historize the outage Bad, drop late replays

F1: the driver's outage BadNoCommunication carried the last Good sample's
source timestamp. The recorder historizes SourceTimestampUtc ??
ServerTimestampUtc, so the Bad landed on the Good row's (TagId,
SourceTimeUtc) key and the insert-if-absent historian dropped it - no
outage marker until the host's watchdog marker ~95 s in, none at all for
a short blip. Now stamped with the detection time (source + server), as
the host's comms-loss fan does.

F2: after a same-session recovery the SDK delivered a missed notification
seconds late with an older source time and the node regressed. A
notification older than the newest Good upstream source time for its ref
(notification or recovery read) is dropped and counted
(stale_notification_dropped); equal timestamps pass; the outage Bad never
moves the comparison; a re-initialise resets it.
This commit is contained in:
Joseph Doherty
2026-10-11 09:11:34 -04:00
parent db6446ce73
commit 99c13e4459
5 changed files with 147 additions and 5 deletions
+8 -2
View File
@@ -66,12 +66,17 @@ The session keep-alive (`KeepAliveInterval`, default 5 s) is the liveness signal
subscription-only driver never calls `ReadAsync`, so without this its read stamp froze at connect time.
- **Bad keep-alive** ⇒ health `Degraded` (`LastError = "keep-alive: <status>"`, read stamp preserved), host status
`Stopped`, and — **once per outage**, not per keep-alive — every subscribed ref is published `BadNoCommunication`
with its last value retained, plus a quality-only `BadNoCommunication` annotation (#626 pattern, no alarm-state
with its last value retained and **source + server timestamp = the detection time** (live-gate F1: stamped with the
last Good's source time it collided with that row's `(TagId, SourceTimeUtc)` history key and was never stored), plus a quality-only `BadNoCommunication` annotation (#626 pattern, no alarm-state
change, no `/alerts` row) on every subscribed alarm condition.
- **The session survives** (a network blip; the keep-alive turns Good again) ⇒ recovered in place: health `Healthy`,
the refs still Bad are re-read once and published Good (a server whose subscription survived does not resend an
unchanged value, and a `(value, status)` dedup downstream lets the identical value through), alarm quality goes
back Good and a `ConditionRefresh` re-reports anything missed. A notification that lands first wins over the read.
A notification whose source time is OLDER than the newest Good upstream source time already published for its ref
(notification or recovery read) is dropped — the SDK can deliver a missed one seconds late after a same-session
recovery, which used to regress the node (live-gate F2). Equal timestamps pass (status-only changes); the outage
Bad never moves the comparison; a re-initialise resets it.
- **The session does not come back** (an upstream restart: the server forgot the session) ⇒ the driver stays
`Degraded` with its read stamp frozen, which is exactly the #527 comms-loss watchdog's trip condition. After 90 s
`DriverInstanceActor` runs `ShutdownAsync` + `InitializeAsync` (retried every 10 s while the server is down) and
@@ -228,7 +233,8 @@ every `InitializeAsync`, including comms-loss reconnects). The discovery channel
Session/publish events are published on the shared OTel meter as the counter
`otopcua.driver.opcuaclient.events`, tagged `driver_instance` (the driver instance id) and `event` =
`publish_response`, `notification` (items), `data_change`, `bad_status_value`, `publish_error`,
`keepalive_miss`, `keepalive_recovered` (a bad keep-alive followed by a good one on the same session). A climbing
`keepalive_miss`, `keepalive_recovered` (a bad keep-alive followed by a good one on the same session),
`stale_notification_dropped` (an out-of-order late notification, see above). A climbing
`keepalive_miss` / `keepalive_recovered` rate is the signature of a flapping upstream (`reconnect_*` were removed with
the reconnect handler, #670). They are not part of `DriverHealth` or the
`/hosts` panel.