feat(opcua): per-item delivery for folder event items; folder ConditionRefresh gets the #854 cut (#855)

Condition events reaching one of our notifiers no longer go through the SDK
MonitoredNode2 channel: the walk serves the attached items itself, per item and
per event (audit guard, uncached INodeManager3 verdict, item gate/cut,
QueueEvent), as the Server-object path does. No notifier of ours defers any
more; the #838 deferral queue and the #854 cut stay as the safety net and
folder items share the cut (RefreshDirectItems, flag on and off). An event
whose walk suspends is marked deferred before it suspends.

Tests produce deferral with a notifier-stall probe instead of holding a folder
item's SDK lock (which would now block the reporter). New
FolderConditionRefreshWireTests; design note in docs/plans.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 12:18:11 -05:00
parent e857135e8b
commit d9b23e0b95
10 changed files with 1302 additions and 213 deletions
+41 -22
View File
@@ -901,7 +901,7 @@ Behaviours worth knowing before enabling it:
older `Retain=false` event of such a condition still arrives (it agrees with the omission), and a condition that was
retained but omitted for the item's **grants** is never withheld (a grant added before the drain lets its events
through; the ACL decides them at delivery as ever).
- The deferral queues, every other Server-object item and every folder item are untouched. A refresh waits for the
- The deferral queues, every other Server-object item and every non-refreshing folder item are untouched. A refresh waits for the
drain at most for one in-progress `QueueEvent` on that item (the gate), never for the drain itself; a reporter
holding `Lock` likewise waits at most for one. Each item keeps one gate/cut state for its lifetime (keyed by the
item, so an unsubscribe + resubscribe keeps it), and cuts are cleared when the deferral count returns to zero.
@@ -912,29 +912,48 @@ Behaviours worth knowing before enabling it:
- **Measured** (3000 retained conditions, longest `Lock` wait another thread saw during one refresh): the first
#854 cut (`1622d07f`, everything under `Lock`) 95–106 ms idle / 102–127 ms deferring; now 0–3 ms idle /
23–32 ms deferring (the frozen snapshots).
- ⚠️ **Residual: folder (notifier) items.** A refresh of an item on an equipment/device folder can still be followed
by older events of a condition. Two sources, neither reachable per item: the SDK's own `MonitoredNode2` channel
(up to 4096 events already accepted before the refresh — this exists in the stock SDK too, and in normal operation
for the few events in transit), and our deferral queue, whose folder hop goes through that channel, where the SDK
memoises the per-item permission verdict by (item, event type, source) so there is no per-event, per-item hook. The
events stay in the condition's order and its newest arrives last, so the client converges. Discarding the queue
globally on refresh would drop transitions for every non-refreshing subscriber while leaving the channel backlog;
making refresh wait would block a service call on the stalled drain. ScadaBridge also ignores a transition older
than the one it holds for a condition.
- **Folder (notifier) items are delivered per item (Gitea #855).** Before #855 a folder item was fed by the SDK's own
`MonitoredNode2` channel (up to 4096 events accepted before a refresh), with the per-item verdict memoised by (item,
event type, source), so a refresh of a folder item could still be followed by older events. Now, at every notifier
the node manager owns, the walk serves the attached items itself (`DeliverToNotifierItems`), exactly as it serves
Server-object items: the SDK's audit guard, the verdict through `INodeManager3.ValidateEventRolePermissions` (SDK
type/source `ReceiveEvents` + the #826 decision under enforcement) **per event, uncached**, then the item's gate and
cut, then `QueueEvent`. The `MonitoredNode2` stays for item attachment and for what still uses its channel (a
condition's SDK method audit events, data-change items on the folder), so `InvalidateFolderEventVerdicts` stays.
- **No notifier of ours defers any more.** `QueueEvent` never waits for a consumer (the item queue discards oldest at
`QueueSize`) and the verdict runs on the reporter's thread, which already holds `Lock` — so a burst under one
`Lock` hold completes with nothing deferred, and the #838 deadlock shape is gone for condition events. Folder items
therefore see condition events in report order. The cost moves with it: the per-event verdict and filter for
folder items now run under `Lock` (as for Server-object items), where the SDK used to cache the verdict off it.
- **The deferral queue and the cut stay as the safety net**, for a hop that is not ours (none exists today:
conditions are never event notifiers and variables cannot host event items) — and folder items share the cut:
a `ConditionRefresh`/`ConditionRefresh2` of a folder item takes the same two paths as the Server object (plain walk
off `Lock` when nothing is deferred; otherwise verdicts off `Lock`, then one `Lock` hold for the watermark, the
Retain set and, per item, its notifier walk frozen as snapshots, with the cut installed under the item gate and the
replay queued off `Lock`). An event whose walk suspended at a hop is marked deferred **before** it suspends, so
every later hop judges it against the cut; the plain refresh path's closing check takes `Lock` once, so a report
that suspended under it has registered its deferral by then. Lock order is unchanged (`Lock` → item gate → item
lock); off `Lock` the verdict is computed before the gate.
Pinned by `FolderEventFloodWireTests`: 6000 events under one `Lock` hold with publishing disabled (flag on and off)
complete, drain to zero, coalesce, and the item delivers at most its revised `QueueSize` in increasing order ending
with the last event; and a condition **removed** in the same `Lock` hold has none of its deferred events delivered to
either the folder or the Server-object item (the flag-off variant fails without the liveness checks).
`ConditionRefreshDeferralWireTests` (#854) stalls a folder item's SDK lock so its channel fills without `Lock` held,
refreshes the Server-object item (ConditionRefresh and ConditionRefresh2, flag on/off, condition retained or cleared)
and pins that nothing older than the replay follows RefreshEnd while the non-refreshing folder item still receives the
whole tail (also across an unsubscribe + resubscribe of the item, and the cut is cleared after the drain); a
probe-driven test pins that a refresh waits for a delivery already past its check (fails without the item gate); an
ACL-on test pins that a condition omitted for want of a grant is not withheld once the grant lands; a 3000-condition
test measures the `Lock` hold and pins that neither path queues its replay under `Lock`; and a stress test runs
reports, refreshes and an intermittent stall concurrently (its stream never goes backwards outside a refresh, and
every refresh ends on the newest state delivered).
complete **with nothing deferred** (fails if the per-item delivery stops recognising the SDK fan-out), and the item
delivers at most its revised `QueueSize` in increasing order ending with the last event; and a condition **removed**
while its events wait (delivery stalled at the folder) has none of them delivered to either the folder or the
Server-object item. Tests that need a deferral stall delivery at a folder through a test probe
(`NotifierStall` → `OtOpcUaNodeManager.NotifierDeliveryStall`, null in production), which suspends the walk exactly
as a full channel did — holding a folder item's SDK lock, as the pre-#855 tests did, would now block the reporter.
`ConditionRefreshDeferralWireTests` (#854) refreshes the Server-object item while the folder is stalled
(ConditionRefresh and ConditionRefresh2, flag on/off, condition retained or cleared) and pins that nothing older than
the replay follows RefreshEnd while the non-refreshing folder item still receives the whole tail (also across an
unsubscribe + resubscribe of the item, and the cut is cleared after the drain); a probe-driven test pins that a
refresh waits for a delivery already past its check; an ACL-on test pins that a condition omitted for want of a grant
is not withheld once the grant lands; a 3000-condition test measures the `Lock` hold; and a stress test runs reports,
refreshes and an intermittent stall concurrently. `FolderConditionRefreshWireTests` (#855) is the folder-item twin:
the same matrix on a refreshed folder item (a second subscriber on the same folder keeps the whole tail; a
ModifyMonitoredItems re-subscribe keeps the cut), the gate race, a newer-than-cut event on the raw device folder, a
condition omitted by the ACL (reference dropped) not withheld once it returns, a revocation after the refresh, a
production-shape overload (bursts under `Lock` + repeated refreshes, no stall: nothing deferred, the stream never
goes backwards, no replay queued under `Lock`) and a stress test with an intermittent stall.
**Model-change and audit events on the Server object are gated too** (Gitea #588). The node manager's
own server-wide events — `GeneralModelChangeEvent` (NodeAdded / NodeDeleted / DataTypeChanged) and the