fix(opcua): ConditionRefresh on a Server-object item is not overtaken by older deferred condition events (#854)

While a folder subscriber's SDK channel is full, a condition's later events wait in
its #838 deferral queue and reach i=2253 items only when the drain gets to them, off
Lock. A ConditionRefresh in that window replayed the current state, then the older
deferred events followed RefreshEnd (stale state, or a condition the refresh had just
dropped re-added until the newest drained).

- Condition events are reported as ConditionEventSnapshot (an InstanceStateSnapshot
  carrying a report sequence, condition NodeId and Retain); events queued behind a
  deferred report are marked Deferred.
- Server-object refresh (ConditionRefresh and ConditionRefresh2, flag on AND off) now
  runs in RefreshServerObjectItems: watermark, "anything deferred?", the replay and its
  queueing in ONE Lock hold, so the replay is exactly the state at the cut. Per item it
  queues the replay and installs a RefreshCut under the item's gate.
- OnConditionReportEvent holds that gate across "superseded? -> QueueEvent" for
  deferred events only. Withheld from that item alone: events at/below the cut whose
  condition the replay re-sent, or Retain=true events of a condition it omitted; an
  older Retain=false event of an omitted condition is still delivered. Queues, other
  Server-object items and folder items are untouched; refresh never waits for a drain.
- Lock order Lock -> item gate -> SDK item lock, same as a synchronous report.
- Meter otopcua.opcua.condition_events.superseded.
- Folder items keep a documented residual (SDK channel backlog + memoised per-item
  verdict leave no per-event hook).

Tests: ConditionRefreshDeferralWireTests (deterministic overload by holding the folder
item's SDK lock; 6 variants fail before the fix; probe-driven gate test fails with the
gate removed; concurrent reports/refreshes/stall stress test fails before the fix).
This commit is contained in:
Joseph Doherty
2026-11-11 06:37:22 -05:00
parent d1b6cad299
commit 1622d07f2e
5 changed files with 680 additions and 43 deletions
+27 -4
View File
@@ -882,15 +882,38 @@ Behaviours worth knowing before enabling it:
changes which folders a later hop reaches — and the liveness check covers removal); `MonitoredNode2.Dispose` completes
its channel writer, which fails a pending `WriteAsync` with `ChannelClosedException` (swallowed by the SDK), so a
disposed node cannot stall a condition's queue; and the delivery task catches per-event exceptions, so it keeps draining.
- ⚠️ **Residual (documented, not fixed): `ConditionRefresh` does not flush deferrals.** While a notifier channel is
overloaded, a client that refreshes may receive older deferred events of a condition after `RefreshEndEvent`. They
are still in that condition's order and its newest event arrives last, so the state a client ends up holding is
correct; making refresh wait for the queues would block a service call on the very drain the overload is stalling.
- **`ConditionRefresh` is not overtaken by deferred events on Server-object items (Gitea #854).** A refresh of an
`i=2253` item replays each condition's current state between RefreshStart and RefreshEnd; the condition's deferred
events — all older — used to follow RefreshEnd, briefly showing a stale state or re-adding an alarm the refresh had
just dropped (ScadaBridge's snapshot swap drops absent conditions, then a stale active event re-added it). Now every
condition event carries its report sequence; the refresh takes its **cut** (the sequence, whether anything is
deferred, the replay, and its queueing) in **one hold of `Lock`**, so the replay is exactly the state at the cut, and
installs it per item under that item's gate, which a deferred delivery also holds across its check and
`QueueEvent`. A deferred event at or below the cut is withheld **from that item only** when the refresh re-sent its
condition, or when it is `Retain=true` for a condition the refresh omitted; an older `Retain=false` event of an
omitted condition still arrives (it agrees with the omission, and a client that does not reconcile on RefreshEnd
needs it). The deferral queues, every other Server-object item and every folder item are untouched; the refresh never
waits for a drain. Meter: `otopcua.opcua.condition_events.superseded` (per item and event). Both ConditionRefresh and
ConditionRefresh2, flag on and off.
- ⚠️ **Residual: folder (notifier) items.** A refresh of an item on an equipment/device folder can still be followed
by older events of a condition. Two sources, neither reachable per item: the SDK's own `MonitoredNode2` channel
(up to 4096 events already accepted before the refresh — this exists in the stock SDK too, and in normal operation
for the few events in transit), and our deferral queue, whose folder hop goes through that channel, where the SDK
memoises the per-item permission verdict by (item, event type, source) so there is no per-event, per-item hook. The
events stay in the condition's order and its newest arrives last, so the client converges. Discarding the queue
globally on refresh would drop transitions for every non-refreshing subscriber while leaving the channel backlog;
making refresh wait would block a service call on the stalled drain. ScadaBridge also ignores a transition older
than the one it holds for a condition.
Pinned by `FolderEventFloodWireTests`: 6000 events under one `Lock` hold with publishing disabled (flag on and off)
complete, drain to zero, coalesce, and the item delivers at most its revised `QueueSize` in increasing order ending
with the last event; and a condition **removed** in the same `Lock` hold has none of its deferred events delivered to
either the folder or the Server-object item (the flag-off variant fails without the liveness checks).
`ConditionRefreshDeferralWireTests` (#854) stalls a folder item's SDK lock so its channel fills without `Lock` held,
refreshes the Server-object item (ConditionRefresh and ConditionRefresh2, flag on/off, condition retained or cleared)
and pins that nothing older than the replay follows RefreshEnd while the non-refreshing folder item still receives the
whole tail; a probe-driven test pins that a refresh waits for a delivery already past its check (fails without the
item gate); and a stress test runs reports, refreshes and an intermittent stall concurrently.
**Model-change and audit events on the Server object are gated too** (Gitea #588). The node manager's
own server-wide events — `GeneralModelChangeEvent` (NodeAdded / NodeDeleted / DataTypeChanged) and the