ca0e73fbca
- A filter group owns a deep clone of its first filter, validated once and evaluated under the group's lock: no item-owned filter is mutated or raced (Validate mutates a select clause). - The projection is computed in the SDK's order: where clause first, fields only for an item that will queue. - Unambiguous locale key; counter for filters given a private group past the registry cap; DeliveryCache getter returns its own allocation. - UseProjectedEventDelivery=false also asks the role and grant checks per item, so the equivalence test compares against the pre-optimisation path. - Equivalence test: same select list under three where clauses, a filter modified mid-run, a session with other locales, an ungranted anonymous Server-object item, per-rig counter; plus a small overflowing queue with DiscardOldest true and false. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
137 lines
10 KiB
Markdown
137 lines
10 KiB
Markdown
# Folder (notifier) event items: per-item delivery (Gitea #855)
|
||
|
||
Status: design for #855 (owner decision 2026-11-11: build per-item delivery). Background: #838 (deferral instead of
|
||
blocking under `Lock`), #854 (sequence cut on Server-object refresh), #826 (per-folder event ACL).
|
||
|
||
## Problem
|
||
|
||
A condition event reaches an event item on an equipment / raw device folder through the SDK's `MonitoredNode2`
|
||
for that folder: `OnReportEventAsync` writes it to a bounded channel (4096), and a per-node consumer task decides
|
||
each item's verdict (memoised per `(item, event type, SourceNode)`) and calls `QueueEvent`. That gives the node
|
||
manager no per-event, per-item decision point, so a `ConditionRefresh` of a folder item cannot withhold an older
|
||
event that is still in the channel or in our deferral queue: the refresh is overtaken (the #854 guarantee covers only
|
||
Server-object items). The channel's consumer also needs `Lock` on a verdict miss, which is why a burst under `Lock`
|
||
fills the channel and defers (#838).
|
||
|
||
## Design
|
||
|
||
### Where delivery hooks in
|
||
|
||
`DeliverConditionEventAsync` already walks the condition's notifier chain hop by hop (our copy of the SDK's
|
||
`ReportEventAsync`, #838). At a hop whose `OnReportEventAsync` is the SDK `MonitoredNode2` of **a node this manager
|
||
owns** (`handler.Target is MonitoredNode2 m && m.Node == node`), the walk no longer calls the handler. It delivers the
|
||
event itself to that node's attached items (`m.EventMonitoredItems`), one by one: the SDK's audit guard, the verdict
|
||
(`INodeManager3.ValidateEventRolePermissions` — the SDK's type/source `ReceiveEvents` check plus, under enforcement,
|
||
the #826 live-notifier + grant check), then the item's gate and cut, then `QueueEvent`.
|
||
|
||
This is exactly how Server-object items are served (`OnConditionReportEvent`, #586/#854): same verdict per event, same
|
||
gate, same cut, same `QueueEvent`.
|
||
|
||
### Bypass / coexistence with `MonitoredNode2`
|
||
|
||
- **Condition events bypass the channel** for our notifiers. The `MonitoredNode2` stays: the SDK still owns item
|
||
attachment (subscribe / unsubscribe / transfer / tombstone / revalidation stand-down detach), and its channel still
|
||
carries everything else that reaches the node — a condition's SDK-raised method **audit** events (only with auditing
|
||
on) and data-change notifications for attribute items on the folder. Those keep the SDK's cached verdict, so
|
||
`InvalidateFolderEventVerdicts` stays.
|
||
- A stood-down item is not in `EventMonitoredItems`, so it receives nothing, as before.
|
||
- A hop whose handler is anything else (none exists today: conditions are never event notifiers and variables cannot
|
||
host event items) still goes through the handler — the fallback that keeps the #838 deferral reachable.
|
||
|
||
### Consequence for overload
|
||
|
||
Every hop the node manager owns now completes synchronously: `QueueEvent` never blocks (the item's own queue discards
|
||
oldest at `QueueSize`), and the verdict runs on the reporter's thread, which already holds `Lock`. So a burst under one
|
||
`Lock` hold no longer defers at all, and the #838 deadlock shape (a consumer needing `Lock` while the reporter waits on
|
||
its channel) is gone for condition events. Folder items stay in report order per item, exactly like Server-object items.
|
||
|
||
The #838 deferral queue and the #854 cut machinery remain as the safety net for a hop that does suspend (the fallback
|
||
above, or a future SDK change). Folder items are brought under the same cut:
|
||
|
||
- An event whose delivery continues off `Lock` — queued in the deferral queue, **or** the in-flight event whose walk
|
||
suspended at a hop (new: marked when the walk suspends, so the hops after the suspension take the gate) — takes the
|
||
item's gate and is withheld when the item's latest refresh cut supersedes it (`RefreshCut.Supersedes`, unchanged).
|
||
- `ConditionRefresh` / `ConditionRefresh2` of folder items has the same two paths as the Server object, flag on and off:
|
||
- **Nothing deferred:** the SDK base walk of the item's notifier (off `Lock`, live condition nodes), per-item verdict
|
||
as before (#826 filter under enforcement). Kept only if no deferral episode existed at any moment of it; otherwise
|
||
the cut path runs too and supersedes it inside the same RefreshStart..RefreshEnd. Its closing check takes `Lock`
|
||
once: a report whose walk suspended (marked under `Lock` before suspending) has registered its deferral by then —
|
||
without it a folder refresh could pass the check in the instant between the suspension and the count.
|
||
- **Deferring — the cut:** verdicts per (item, condition) first, off `Lock`, from a live walk. Then one `Lock` hold:
|
||
watermark, episode, the Retain set, and per item the same walk producing frozen snapshots, with the cut + "replay
|
||
pending" marker installed under the item's gate. The replay is queued off `Lock`, one event per gate hold.
|
||
- State per item (`EventItemState`, previously Server-object only) is keyed by the item object for its lifetime, so it
|
||
survives unsubscribe + resubscribe; stale cuts are cleared when the deferral episode ends, for folder and Server items.
|
||
|
||
### Locking
|
||
|
||
Unchanged order, now for folder items too: `Lock` → item gate → the SDK item's own lock. Synchronous delivery runs
|
||
under the reporter's `Lock`; the verdict takes `Lock` re-entrantly and the core node manager's / authorizer's locks
|
||
(as the Server-object path already does). Off-`Lock` (deferred) delivery computes the verdict **before** taking the item
|
||
gate, so nothing under a gate takes `Lock` or the event-report gate. A reporter waits at most for one in-progress
|
||
`QueueEvent` on an item; a refresh waits at most for one deferred delivery already inside the gate, never for a drain.
|
||
No reporter ever waits on a channel.
|
||
|
||
### Bounds
|
||
|
||
No new queue. Per item: the SDK item queue (`QueueSize`, DiscardOldest). Deferral (only on the fallback): unchanged,
|
||
1024 per condition / 65 536 total, coalescing. Cuts: one per refreshed item, sets sized by the conditions, cleared at the
|
||
end of the deferral episode.
|
||
|
||
### ACL (#826)
|
||
|
||
Each event is decided per item at delivery against the current generation — no memoised verdict for condition events
|
||
any more, so a deploy / revocation / reference removal takes effect on the next event without waiting for an
|
||
invalidation. Refresh replays only conditions the item may see (the same check); a condition omitted for want of a grant
|
||
is never withheld afterwards (the cut's existing rule).
|
||
|
||
### Cost
|
||
|
||
The per-event verdict for folder items moves from the SDK's consumer task (cached) to the reporter thread under `Lock`
|
||
(uncached), as Server-object items already pay. Folder subscribers are few (ScadaBridge subscribes at `i=2253`).
|
||
|
||
## Tests
|
||
|
||
- `ConditionRefreshDeferralWireTests` / `FolderEventFloodWireTests` / `EventQueueOverflowAclWireTests`: deferral is no
|
||
longer produced by flooding a folder item; tests that need it stall the folder's hop through a test probe
|
||
(`NotifierDeliveryStall`, null in production) instead of the folder item's SDK lock (which would now block the reporter,
|
||
as it would on the Server object). Flood tests now pin that folder delivery never defers.
|
||
- New `FolderConditionRefreshWireTests`: folder-item refresh during deferral (flag on/off, retained / cleared,
|
||
ConditionRefresh2, resubscribe), the gate race, newer-than-cut replay, ACL omission not withheld, ACL revocation during
|
||
deferral, the in-flight suspended event, and two stress tests (production overload with no stall; stall + refresh +
|
||
reports concurrently).
|
||
|
||
## Follow-up: cost under `Lock` (projected delivery)
|
||
|
||
Measured with `FolderEventDeliveryPerfTests` (opt-in, `OTOPCUA_EVENT_PERF=1`): the per-item work moved under `Lock` by
|
||
this change cost ~5–6 µs/item publishing and ~10–30 µs/item at a full queue (N = 50 stalled items: p50 0.4–1.6 ms).
|
||
Profiling split it into the SDK's role-permission check (~3–9 µs/item) and `MonitoredItem.QueueEvent(IFilterTarget)`,
|
||
whose duplicate scan over the last ≤ 1000 queued events grows with the queue (~3 µs at QueueSize 10, ~10–16 at 1000,
|
||
~16–31 at 10 000) on top of the per-item where-clause + select-clause projection.
|
||
|
||
`ProjectedEventDelivery` / `EventDeliveryCache` (per report, hung on the condition snapshot, dropped when the walk ends):
|
||
|
||
- the SDK role verdict once per (session, effective identity); the #826 grant check once per identity per hop;
|
||
- the where clause and the select-clause projection once per (value-equal `EventFilter`, preferred locales), in the
|
||
SDK's order (fields only for an item that will queue) — filters are grouped once per filter object (`Utils.IsEqual`,
|
||
≤ 4096 shared groups, then private groups, counted); a group evaluates its OWN deep clone, validated once, under its own
|
||
lock, never an item's filter object — and the field values shared read-only by every item queued from it (each item still gets its own `EventFieldList`: the SDK rewrites a queued list's
|
||
`ClientHandle` on modify);
|
||
- the SDK's `QueueEvent(instance, false)` body reproduced under the item's own lock (event item; Disabled ⇒ nothing;
|
||
duplicate ⇒ nothing — asked of the report's "already queued" set instead of the queue; the filter in use;
|
||
`SetQueueOverflowIfFull`; then the public `QueueEvent(EventFieldList)`), reading three private members (`m_lock`,
|
||
`m_eventQueueHandler`, `FilterToUse`) — pinned by a guard test; anything else (members missing, a subclassed item, a
|
||
non-`EventFilter` filter, a `SupportsFilteredRetain` condition, a non-snapshot event, refresh replays) uses the SDK's
|
||
own `QueueEvent(IFilterTarget)`.
|
||
|
||
`ProjectedEventDeliveryWireTests` runs one scenario through each path (`UseProjectedEventDelivery`, which also turns the
|
||
verdict sharing off) and requires
|
||
identical streams on folder and Server-object items (value-equal filters, a severity where clause, an OfType where
|
||
clause with an absent field, the same select list under three where clauses, a filter modified mid-run, a second session
|
||
with other locales, an ungranted anonymous Server-object item, a Disabled item) plus a small overflowing queue with
|
||
DiscardOldest true and false; it fails if the where clause, a field or the grouping is wrong.
|
||
|
||
Residual: at a full queue the SDK's own `EventMonitoredItemQueue` still does `List.RemoveAt(0)` per item (memmove of the
|
||
whole queue) — ~1–10 µs/item at QueueSize 1000–10 000; replacing the queue (`IMonitoredItemQueueFactory`) is a separate
|
||
change.
|