fix(opcua): deferred condition events never outlive their condition; per-condition bounded deferral; canonical lockout identity (#838, #844 review)

#838 review of 91e28e87:
1. Ghost events. The deferred report ran the SDK walk off Lock with no
   liveness check, so an event waiting behind a full channel when a deploy
   removed its condition still reached i=2253 subscribers after the NodeDeleted
   (a ghost Retain alarm for ScadaBridge) and folder items. Delivery now walks
   the SDK path itself (OnReportEvent, OnReportEventAsync, inverse notifiers,
   then the Parent as BaseInstanceState does) re-checking before EVERY hop that
   the condition is the live instance; OnConditionReportEvent checks the same,
   which also stops a non-live scripted condition being admitted through its
   old equipment folder (#808). A guard test pins that only NodeState and
   BaseInstanceState define the walk. New wire test removes the condition in
   the same Lock hold as a flood; the flag-off variant fails without the checks.
2. Back-pressure. Deferral is per condition (one condition's events stay
   ordered; others are not held up). Overflow policy: coalesce, newest wins --
   past 1024 waiting per condition or 65,536 total, the oldest not-yet-started
   event is discarded (a condition event carries full state). Warning on
   entering deferral (<=1/min), Information on draining, meters
   otopcua.opcua.condition_events.deferred / .discarded{reason}.
3. Verified by decompiling SDK 2.0.1: the notifier list is snapshotted under the
   node's own lock, MonitoredNode2.Dispose completes its writer (pending writes
   fail with ChannelClosedException, swallowed), and the consumer catches
   per-event exceptions -- no fix needed; documented.
4. ConditionRefresh does not flush deferrals; documented as a residual.

#844 review: DescribeCredential canonicalises the key id (trim, lowercase;
non-hex ids fold into one identity) and the username, since ConfigDbApiKeyStore
matches with SQL Server's case-insensitive, trailing-space-blind `=`.
This commit is contained in:
Joseph Doherty
2026-11-11 04:35:30 -05:00
parent fc3a39b1ca
commit 7482a88888
8 changed files with 428 additions and 64 deletions
@@ -154,6 +154,24 @@ public static class OtOpcUaTelemetry
Meter.CreateCounter<long>("otopcua.opcua.apikey_store.bounded", unit: "{lookup}",
description: "API-key store lookups the bound intervened on (outcome=timeout|cache_hit).");
/// <summary>
/// Part 9 condition events whose delivery is currently deferred because an event notifier's SDK delivery
/// channel was full (Gitea #838). Zero in steady state; non-zero means some folder's event subscribers are
/// not keeping up with the alarm event rate (or the node manager lock is held across a large burst).
/// </summary>
public static readonly UpDownCounter<long> ConditionEventReportsDeferred =
Meter.CreateUpDownCounter<long>("otopcua.opcua.condition_events.deferred", unit: "{event}",
description: "Condition events waiting for room in an event notifier's delivery channel.");
/// <summary>
/// Deferred condition events discarded before delivery (Gitea #838): <c>reason=coalesced</c> — the per-condition
/// deferral budget was exceeded and the condition's oldest not-yet-started event gave way to its newest;
/// <c>reason=not_live</c> — the condition was removed or replaced before its deferred event was delivered.
/// </summary>
public static readonly Counter<long> ConditionEventReportsDiscarded =
Meter.CreateCounter<long>("otopcua.opcua.condition_events.discarded", unit: "{event}",
description: "Deferred condition events discarded before delivery (reason=coalesced|not_live).");
// ---------------- Convenience helpers ----------------
/// <summary>