Files
lmxopcua/docs/plans/2026-11-11-folder-event-per-item-delivery.md
T
Joseph Doherty ca0e73fbca fix(opcua): #855 perf review 1 — groups evaluate their own filter clone; lazy fields; tighter equivalence tests
- A filter group owns a deep clone of its first filter, validated once and
  evaluated under the group's lock: no item-owned filter is mutated or raced
  (Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
  only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
  registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
  item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
  modified mid-run, a session with other locales, an ungranted anonymous
  Server-object item, per-rig counter; plus a small overflowing queue with
  DiscardOldest true and false.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 13:59:22 -05:00

137 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Folder (notifier) event items: per-item delivery (Gitea #855)
Status: design for #855 (owner decision 2026-11-11: build per-item delivery). Background: #838 (deferral instead of
blocking under `Lock`), #854 (sequence cut on Server-object refresh), #826 (per-folder event ACL).
## Problem
A condition event reaches an event item on an equipment / raw device folder through the SDK's `MonitoredNode2`
for that folder: `OnReportEventAsync` writes it to a bounded channel (4096), and a per-node consumer task decides
each item's verdict (memoised per `(item, event type, SourceNode)`) and calls `QueueEvent`. That gives the node
manager no per-event, per-item decision point, so a `ConditionRefresh` of a folder item cannot withhold an older
event that is still in the channel or in our deferral queue: the refresh is overtaken (the #854 guarantee covers only
Server-object items). The channel's consumer also needs `Lock` on a verdict miss, which is why a burst under `Lock`
fills the channel and defers (#838).
## Design
### Where delivery hooks in
`DeliverConditionEventAsync` already walks the condition's notifier chain hop by hop (our copy of the SDK's
`ReportEventAsync`, #838). At a hop whose `OnReportEventAsync` is the SDK `MonitoredNode2` of **a node this manager
owns** (`handler.Target is MonitoredNode2 m && m.Node == node`), the walk no longer calls the handler. It delivers the
event itself to that node's attached items (`m.EventMonitoredItems`), one by one: the SDK's audit guard, the verdict
(`INodeManager3.ValidateEventRolePermissions` — the SDK's type/source `ReceiveEvents` check plus, under enforcement,
the #826 live-notifier + grant check), then the item's gate and cut, then `QueueEvent`.
This is exactly how Server-object items are served (`OnConditionReportEvent`, #586/#854): same verdict per event, same
gate, same cut, same `QueueEvent`.
### Bypass / coexistence with `MonitoredNode2`
- **Condition events bypass the channel** for our notifiers. The `MonitoredNode2` stays: the SDK still owns item
attachment (subscribe / unsubscribe / transfer / tombstone / revalidation stand-down detach), and its channel still
carries everything else that reaches the node — a condition's SDK-raised method **audit** events (only with auditing
on) and data-change notifications for attribute items on the folder. Those keep the SDK's cached verdict, so
`InvalidateFolderEventVerdicts` stays.
- A stood-down item is not in `EventMonitoredItems`, so it receives nothing, as before.
- A hop whose handler is anything else (none exists today: conditions are never event notifiers and variables cannot
host event items) still goes through the handler — the fallback that keeps the #838 deferral reachable.
### Consequence for overload
Every hop the node manager owns now completes synchronously: `QueueEvent` never blocks (the item's own queue discards
oldest at `QueueSize`), and the verdict runs on the reporter's thread, which already holds `Lock`. So a burst under one
`Lock` hold no longer defers at all, and the #838 deadlock shape (a consumer needing `Lock` while the reporter waits on
its channel) is gone for condition events. Folder items stay in report order per item, exactly like Server-object items.
The #838 deferral queue and the #854 cut machinery remain as the safety net for a hop that does suspend (the fallback
above, or a future SDK change). Folder items are brought under the same cut:
- An event whose delivery continues off `Lock` — queued in the deferral queue, **or** the in-flight event whose walk
suspended at a hop (new: marked when the walk suspends, so the hops after the suspension take the gate) — takes the
item's gate and is withheld when the item's latest refresh cut supersedes it (`RefreshCut.Supersedes`, unchanged).
- `ConditionRefresh` / `ConditionRefresh2` of folder items has the same two paths as the Server object, flag on and off:
- **Nothing deferred:** the SDK base walk of the item's notifier (off `Lock`, live condition nodes), per-item verdict
as before (#826 filter under enforcement). Kept only if no deferral episode existed at any moment of it; otherwise
the cut path runs too and supersedes it inside the same RefreshStart..RefreshEnd. Its closing check takes `Lock`
once: a report whose walk suspended (marked under `Lock` before suspending) has registered its deferral by then —
without it a folder refresh could pass the check in the instant between the suspension and the count.
- **Deferring — the cut:** verdicts per (item, condition) first, off `Lock`, from a live walk. Then one `Lock` hold:
watermark, episode, the Retain set, and per item the same walk producing frozen snapshots, with the cut + "replay
pending" marker installed under the item's gate. The replay is queued off `Lock`, one event per gate hold.
- State per item (`EventItemState`, previously Server-object only) is keyed by the item object for its lifetime, so it
survives unsubscribe + resubscribe; stale cuts are cleared when the deferral episode ends, for folder and Server items.
### Locking
Unchanged order, now for folder items too: `Lock` → item gate → the SDK item's own lock. Synchronous delivery runs
under the reporter's `Lock`; the verdict takes `Lock` re-entrantly and the core node manager's / authorizer's locks
(as the Server-object path already does). Off-`Lock` (deferred) delivery computes the verdict **before** taking the item
gate, so nothing under a gate takes `Lock` or the event-report gate. A reporter waits at most for one in-progress
`QueueEvent` on an item; a refresh waits at most for one deferred delivery already inside the gate, never for a drain.
No reporter ever waits on a channel.
### Bounds
No new queue. Per item: the SDK item queue (`QueueSize`, DiscardOldest). Deferral (only on the fallback): unchanged,
1024 per condition / 65 536 total, coalescing. Cuts: one per refreshed item, sets sized by the conditions, cleared at the
end of the deferral episode.
### ACL (#826)
Each event is decided per item at delivery against the current generation — no memoised verdict for condition events
any more, so a deploy / revocation / reference removal takes effect on the next event without waiting for an
invalidation. Refresh replays only conditions the item may see (the same check); a condition omitted for want of a grant
is never withheld afterwards (the cut's existing rule).
### Cost
The per-event verdict for folder items moves from the SDK's consumer task (cached) to the reporter thread under `Lock`
(uncached), as Server-object items already pay. Folder subscribers are few (ScadaBridge subscribes at `i=2253`).
## Tests
- `ConditionRefreshDeferralWireTests` / `FolderEventFloodWireTests` / `EventQueueOverflowAclWireTests`: deferral is no
longer produced by flooding a folder item; tests that need it stall the folder's hop through a test probe
(`NotifierDeliveryStall`, null in production) instead of the folder item's SDK lock (which would now block the reporter,
as it would on the Server object). Flood tests now pin that folder delivery never defers.
- New `FolderConditionRefreshWireTests`: folder-item refresh during deferral (flag on/off, retained / cleared,
ConditionRefresh2, resubscribe), the gate race, newer-than-cut replay, ACL omission not withheld, ACL revocation during
deferral, the in-flight suspended event, and two stress tests (production overload with no stall; stall + refresh +
reports concurrently).
## Follow-up: cost under `Lock` (projected delivery)
Measured with `FolderEventDeliveryPerfTests` (opt-in, `OTOPCUA_EVENT_PERF=1`): the per-item work moved under `Lock` by
this change cost ~5–6 µs/item publishing and ~10–30 µs/item at a full queue (N = 50 stalled items: p50 0.4–1.6 ms).
Profiling split it into the SDK's role-permission check (~3–9 µs/item) and `MonitoredItem.QueueEvent(IFilterTarget)`,
whose duplicate scan over the last ≤ 1000 queued events grows with the queue (~3 µs at QueueSize 10, ~10–16 at 1000,
~16–31 at 10 000) on top of the per-item where-clause + select-clause projection.
`ProjectedEventDelivery` / `EventDeliveryCache` (per report, hung on the condition snapshot, dropped when the walk ends):
- the SDK role verdict once per (session, effective identity); the #826 grant check once per identity per hop;
- the where clause and the select-clause projection once per (value-equal `EventFilter`, preferred locales), in the
SDK's order (fields only for an item that will queue) — filters are grouped once per filter object (`Utils.IsEqual`,
≤ 4096 shared groups, then private groups, counted); a group evaluates its OWN deep clone, validated once, under its own
lock, never an item's filter object — and the field values shared read-only by every item queued from it (each item still gets its own `EventFieldList`: the SDK rewrites a queued list's
`ClientHandle` on modify);
- the SDK's `QueueEvent(instance, false)` body reproduced under the item's own lock (event item; Disabled ⇒ nothing;
duplicate ⇒ nothing — asked of the report's "already queued" set instead of the queue; the filter in use;
`SetQueueOverflowIfFull`; then the public `QueueEvent(EventFieldList)`), reading three private members (`m_lock`,
`m_eventQueueHandler`, `FilterToUse`) — pinned by a guard test; anything else (members missing, a subclassed item, a
non-`EventFilter` filter, a `SupportsFilteredRetain` condition, a non-snapshot event, refresh replays) uses the SDK's
own `QueueEvent(IFilterTarget)`.
`ProjectedEventDeliveryWireTests` runs one scenario through each path (`UseProjectedEventDelivery`, which also turns the
verdict sharing off) and requires
identical streams on folder and Server-object items (value-equal filters, a severity where clause, an OfType where
clause with an absent field, the same select list under three where clauses, a filter modified mid-run, a second session
with other locales, an ungranted anonymous Server-object item, a Disabled item) plus a small overflowing queue with
DiscardOldest true and false; it fails if the where clause, a field or the grouping is wrong.
Residual: at a full queue the SDK's own `EventMonitoredItemQueue` still does `List.RemoveAt(0)` per item (memmove of the
whole queue) — ~1–10 µs/item at QueueSize 1000–10 000; replacing the queue (`IMonitoredItemQueueFactory`) is a separate
change.