Folder event delivery: ring-buffer event queue so a full queue does not shift under Lock (large N / large QueueSize residual from #855) #870

New Issue

2026-11-11T14:34:48-05:00

dohertj2 commented

2026-11-11 14:34:48 -05:00

Follow-up to #855. Since #855, condition events are delivered to folder event items under the node-manager Lock. The shared per-report projection brought N=50 stalled items at QueueSize 1000 back to p50 81–181 µs and p99 335–555 µs.

Two cases stay above the pre-#855 baseline:

N=200 stalled: p50 137–400 µs, p99 5.4–12 ms. Pre-#855 it was ~30 µs / ≤0.35 ms.
QueueSize 10000: p50 263–489 µs.

Cause: at a full queue, the SDK's EventMonitoredItemQueue shifts its whole list (List.RemoveAt(0)) on every enqueue. That costs ~1 µs per item at q=1000 and ~10 µs at q=10000. GC pauses inside the Lock window make up most of the N=200 p99 tail.

Fix: a ring-buffer event queue through the SDK's IMonitoredItemQueueFactory. Measure with FolderEventDeliveryPerfTests (OTOPCUA_EVENT_PERF=1); raw rows are in the #855 work. ScadaBridge (Server-object subscriber, few or no folder items) is not affected at today's scale.

Follow-up to #855. Since #855, condition events are delivered to folder event items under the node-manager `Lock`. The shared per-report projection brought N=50 stalled items at QueueSize 1000 back to p50 81–181 µs and p99 335–555 µs. Two cases stay above the pre-#855 baseline: - **N=200 stalled:** p50 137–400 µs, p99 5.4–12 ms. Pre-#855 it was ~30 µs / ≤0.35 ms. - **QueueSize 10000:** p50 263–489 µs. **Cause:** at a full queue, the SDK's `EventMonitoredItemQueue` shifts its whole list (`List.RemoveAt(0)`) on every enqueue. That costs ~1 µs per item at q=1000 and ~10 µs at q=10000. GC pauses inside the Lock window make up most of the N=200 p99 tail. **Fix:** a ring-buffer event queue through the SDK's `IMonitoredItemQueueFactory`. Measure with `FolderEventDeliveryPerfTests` (`OTOPCUA_EVENT_PERF=1`); raw rows are in the #855 work. ScadaBridge (Server-object subscriber, few or no folder items) is not affected at today's scale.

dohertj2 added the priority:p3 type:followup labels 2026-11-11 14:34:48 -05:00

dohertj2 referenced this issue

2026-11-11 14:53:53 -05:00

OPC UA ConditionRefresh on equipment/device folder subscriptions can still be followed by older deferred events #855

dohertj2 referenced this issue from a commit

2026-11-11 15:38:12 -05:00

perf(opcua): #870 ring-buffer event monitored-item queue

dohertj2 referenced this issue from a commit

2026-11-11 15:38:12 -05:00

test(opcua): #870 review — pin the 1000th-newest duplicate-window boundary; serialise the ring-queue switch users

dohertj2 referenced this issue from a commit

2026-11-11 15:38:12 -05:00

Merge #870 ring-buffer event monitored-item queue

dohertj2 referenced this issue from a commit

2026-11-11 15:38:12 -05:00

Merge #866 #870 #871 #872: MxStatus detail vocabulary + quality grading, ring event queue, CI races

dohertj2 commented

2026-11-11 15:38:32 -05:00

Fixed, pushed to master in 363f0d81 (bcde9305, review follow-up 36583ee6).

Change: RingEventMonitoredItemQueue is supplied through IMonitoredItemQueueFactory, via RingEventQueueFactory in OtOpcUaSdkServer.CreateMonitoredItemQueueFactory. Data-change and durable queues stay the SDK's own.

Parity with the SDK 2.0.1 EventMonitoredItemQueue:

size-0 throw;
drop-oldest at full;
shrink direction for both DiscardOldest values;
FIFO dequeue;
duplicate check over the newest 1000 entries, using the SDK's IsEventRaisedFor;
non-durable, and Dispose does nothing.

DiscardOldest and overflow-event logic stay in the SDK's EventQueueHandler, which is unchanged.

Tests: 89 parity cases, including random sequences against the SDK queue and against EventQueueHandler over both factories. Breaking the implementation on purpose made them fail. An independent review found no defects. It added a pin on the 1000th-newest boundary and serialised the tests that use the perf harness's process-wide switch.

Same-process A/B, Lock hold in µs (p50 / p99), measured on a loaded machine:

Case	SDK queue	Ring queue
N=50, QueueSize 1000	148 / 425	112 / 374
N=200, ACL off	257 / 6160	161 / 3673
N=200, ACL on	319 / 10492	175 / 4476
N=50, QueueSize 10000	399 / 1098	116 / 408

At QueueSize 10000 the per-item cost dropped from about 10 µs to 3–5 µs. The N=200 p99 tail of a few ms (pre-#855 it was ≤ 0.35 ms) is not attributed. Earlier profiling pointed at GC pauses inside the Lock window. Re-measure on a quiet machine if that scale becomes real; ScadaBridge does not reach it today.

Fixed, pushed to master in `363f0d81` (`bcde9305`, review follow-up `36583ee6`). **Change:** `RingEventMonitoredItemQueue` is supplied through `IMonitoredItemQueueFactory`, via `RingEventQueueFactory` in `OtOpcUaSdkServer.CreateMonitoredItemQueueFactory`. Data-change and durable queues stay the SDK's own. **Parity with the SDK 2.0.1 `EventMonitoredItemQueue`:** - size-0 throw; - drop-oldest at full; - shrink direction for both DiscardOldest values; - FIFO dequeue; - duplicate check over the newest 1000 entries, using the SDK's `IsEventRaisedFor`; - non-durable, and `Dispose` does nothing. DiscardOldest and overflow-event logic stay in the SDK's `EventQueueHandler`, which is unchanged. **Tests:** 89 parity cases, including random sequences against the SDK queue and against `EventQueueHandler` over both factories. Breaking the implementation on purpose made them fail. An independent review found no defects. It added a pin on the 1000th-newest boundary and serialised the tests that use the perf harness's process-wide switch. **Same-process A/B, Lock hold in µs (p50 / p99), measured on a loaded machine:** | Case | SDK queue | Ring queue | |---|---|---| | N=50, QueueSize 1000 | 148 / 425 | 112 / 374 | | N=200, ACL off | 257 / 6160 | 161 / 3673 | | N=200, ACL on | 319 / 10492 | 175 / 4476 | | N=50, QueueSize 10000 | 399 / 1098 | 116 / 408 | At QueueSize 10000 the per-item cost dropped from about 10 µs to 3–5 µs. The N=200 p99 tail of a few ms (pre-#855 it was ≤ 0.35 ms) is not attributed. Earlier profiling pointed at GC pauses inside the Lock window. Re-measure on a quiet machine if that scale becomes real; ScadaBridge does not reach it today.

dohertj2 closed this issue

2026-11-11 15:38:33 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#870