fix(opcua): Server-object ConditionRefresh off Lock unless deferring; cut path O(conditions) with replay queued off Lock (#854 review)

Review of 1622d07f:
1. Perf: every i=2253 refresh ran the whole notifier walk + permission checks +
   QueueEvent under Lock. Now, with nothing deferred (the common case), the refresh is
   the SDK-shaped walk off Lock (Lock only to copy the root notifiers), kept only if no
   deferral episode existed at any moment of it (count 0 before and after, episode
   unchanged); otherwise the cut path also runs and its later replay supersedes. The
   cut path takes ACL/role verdicts off Lock, then one short Lock hold: watermark +
   frozen snapshots of Retain conditions from _alarmConditions (no tree walk) + per-item
   cut and "replay pending" marker; the replay is queued OFF Lock, one event per hold of
   the item gate. While pending, a condition event newer than the cut marks its
   condition so the older frozen entry is dropped rather than following it. A
   concurrent fast-path refresh cancels a pending cut replay of the same item.
   Measured, 3000 retained conditions, longest Lock wait seen by another thread during
   one refresh: 1622d07f 95-106 ms idle / 102-127 ms deferring; now 0-3 ms idle /
   23-32 ms deferring.
2. ACL: a retained condition omitted from the replay for want of a grant is never
   withheld (RefreshCut.RetainedAtCut); only Retain=true events of a condition omitted
   because it was NOT retained are.
3. One gate/cut state per item for its lifetime (ConditionalWeakTable keyed by the
   item): an unsubscribe + resubscribe between refresh and drain keeps the cut.
4. Cuts carry their deferral episode; ClearStaleRefreshCuts runs when the count
   returns to zero, and a stale-episode cut is inert.
5. Dispose guard for the custom refresh (ObjectDisposedException once disposing;
   Dispose waits for refreshes in flight).
6. Docs: a refresh waits at most for one in-progress QueueEvent on the item.

Tests: resubscribe variants, ACL-grant-lands test, pending-replay probe test,
dispose test, 3000-condition Lock-hold measurement with a probe pinning that neither
path queues its replay under Lock; stress invariant tightened (never backwards outside a
refresh; each refresh ends on the newest delivered state) and tolerant of the SDK's own
BadRefreshInProgress overlap refusal. Each new test fails against its targeted mutation.
This commit is contained in:
Joseph Doherty
2026-11-11 07:28:08 -05:00
parent 1622d07f2e
commit fe48458a41
4 changed files with 688 additions and 122 deletions
+33 -12
View File
@@ -885,16 +885,33 @@ Behaviours worth knowing before enabling it:
- **`ConditionRefresh` is not overtaken by deferred events on Server-object items (Gitea #854).** A refresh of an
`i=2253` item replays each condition's current state between RefreshStart and RefreshEnd; the condition's deferred
events — all older — used to follow RefreshEnd, briefly showing a stale state or re-adding an alarm the refresh had
just dropped (ScadaBridge's snapshot swap drops absent conditions, then a stale active event re-added it). Now every
condition event carries its report sequence; the refresh takes its **cut** (the sequence, whether anything is
deferred, the replay, and its queueing) in **one hold of `Lock`**, so the replay is exactly the state at the cut, and
installs it per item under that item's gate, which a deferred delivery also holds across its check and
`QueueEvent`. A deferred event at or below the cut is withheld **from that item only** when the refresh re-sent its
condition, or when it is `Retain=true` for a condition the refresh omitted; an older `Retain=false` event of an
omitted condition still arrives (it agrees with the omission, and a client that does not reconcile on RefreshEnd
needs it). The deferral queues, every other Server-object item and every folder item are untouched; the refresh never
waits for a drain. Meter: `otopcua.opcua.condition_events.superseded` (per item and event). Both ConditionRefresh and
ConditionRefresh2, flag on and off.
just dropped (ScadaBridge's snapshot swap drops absent conditions, then a stale active event re-added it). Every
condition event now carries its report sequence. Two paths, chosen per refresh:
- **Nothing deferred (the common case): unchanged and off `Lock`.** The SDK-shaped notifier walk, queued as live
condition nodes; `Lock` is taken only to copy the root-notifier list. It stands only if no deferral episode existed
at any moment of it (deferral count zero before and after, episode number unchanged); otherwise the cut path runs
too, and its replay — later in the same RefreshStart..RefreshEnd — supersedes anything that slipped in between.
- **Deferring: the cut.** ACL/role verdicts first, off `Lock`. Then ONE short `Lock` hold takes the sequence
watermark and a frozen snapshot of each Retain condition straight from the condition map (O(conditions), no tree
walk), and installs per item, under that item's gate, the **cut** plus a "replay pending" marker. The replay is
queued **off `Lock`**, one event per hold of the item gate. While the marker is set, every condition event to the
item takes the gate: one newer than the cut marks its condition so the older frozen entry is dropped instead of
following it; a deferred one at or below the cut is withheld **from that item only** when the refresh re-sent its
condition, or when it is `Retain=true` for a condition the refresh omitted **because it was not retained**. An
older `Retain=false` event of such a condition still arrives (it agrees with the omission), and a condition that was
retained but omitted for the item's **grants** is never withheld (a grant added before the drain lets its events
through; the ACL decides them at delivery as ever).
- The deferral queues, every other Server-object item and every folder item are untouched. A refresh waits for the
drain at most for one in-progress `QueueEvent` on that item (the gate), never for the drain itself; a reporter
holding `Lock` likewise waits at most for one. Each item keeps one gate/cut state for its lifetime (keyed by the
item, so an unsubscribe + resubscribe keeps it), and cuts are cleared when the deferral count returns to zero.
The custom refresh is refused (`ObjectDisposedException`) once the node manager is disposing, and disposal waits
for refreshes in flight, as the base's private operation guard does. Meter:
`otopcua.opcua.condition_events.superseded` (per item and event). ConditionRefresh and ConditionRefresh2, flag
on and off.
- **Measured** (3000 retained conditions, longest `Lock` wait another thread saw during one refresh): the first
#854 cut (`1622d07f`, everything under `Lock`) 95–106 ms idle / 102–127 ms deferring; now 0–3 ms idle /
23–32 ms deferring (the frozen snapshots).
- ⚠️ **Residual: folder (notifier) items.** A refresh of an item on an equipment/device folder can still be followed
by older events of a condition. Two sources, neither reachable per item: the SDK's own `MonitoredNode2` channel
(up to 4096 events already accepted before the refresh — this exists in the stock SDK too, and in normal operation
@@ -912,8 +929,12 @@ Behaviours worth knowing before enabling it:
`ConditionRefreshDeferralWireTests` (#854) stalls a folder item's SDK lock so its channel fills without `Lock` held,
refreshes the Server-object item (ConditionRefresh and ConditionRefresh2, flag on/off, condition retained or cleared)
and pins that nothing older than the replay follows RefreshEnd while the non-refreshing folder item still receives the
whole tail; a probe-driven test pins that a refresh waits for a delivery already past its check (fails without the
item gate); and a stress test runs reports, refreshes and an intermittent stall concurrently.
whole tail (also across an unsubscribe + resubscribe of the item, and the cut is cleared after the drain); a
probe-driven test pins that a refresh waits for a delivery already past its check (fails without the item gate); an
ACL-on test pins that a condition omitted for want of a grant is not withheld once the grant lands; a 3000-condition
test measures the `Lock` hold and pins that neither path queues its replay under `Lock`; and a stress test runs
reports, refreshes and an intermittent stall concurrently (its stream never goes backwards outside a refresh, and
every refresh ends on the newest state delivered).
**Model-change and audit events on the Server object are gated too** (Gitea #588). The node manager's
own server-wide events — `GeneralModelChangeEvent` (NodeAdded / NodeDeleted / DataTypeChanged) and the