fix(opcua): deferred condition events never outlive their condition; per-condition bounded deferral; canonical lockout identity (#838, #844 review)

#838 review of 91e28e87:
1. Ghost events. The deferred report ran the SDK walk off Lock with no
   liveness check, so an event waiting behind a full channel when a deploy
   removed its condition still reached i=2253 subscribers after the NodeDeleted
   (a ghost Retain alarm for ScadaBridge) and folder items. Delivery now walks
   the SDK path itself (OnReportEvent, OnReportEventAsync, inverse notifiers,
   then the Parent as BaseInstanceState does) re-checking before EVERY hop that
   the condition is the live instance; OnConditionReportEvent checks the same,
   which also stops a non-live scripted condition being admitted through its
   old equipment folder (#808). A guard test pins that only NodeState and
   BaseInstanceState define the walk. New wire test removes the condition in
   the same Lock hold as a flood; the flag-off variant fails without the checks.
2. Back-pressure. Deferral is per condition (one condition's events stay
   ordered; others are not held up). Overflow policy: coalesce, newest wins --
   past 1024 waiting per condition or 65,536 total, the oldest not-yet-started
   event is discarded (a condition event carries full state). Warning on
   entering deferral (<=1/min), Information on draining, meters
   otopcua.opcua.condition_events.deferred / .discarded{reason}.
3. Verified by decompiling SDK 2.0.1: the notifier list is snapshotted under the
   node's own lock, MonitoredNode2.Dispose completes its writer (pending writes
   fail with ChannelClosedException, swallowed), and the consumer catches
   per-event exceptions -- no fix needed; documented.
4. ConditionRefresh does not flush deferrals; documented as a residual.

#844 review: DescribeCredential canonicalises the key id (trim, lowercase;
non-hex ids fold into one identity) and the username, since ConfigDbApiKeyStore
matches with SQL Server's case-insensitive, trailing-space-blind `=`.
This commit is contained in:
Joseph Doherty
2026-11-11 04:35:30 -05:00
parent fc3a39b1ca
commit 7482a88888
8 changed files with 428 additions and 64 deletions
+40 -8
View File
@@ -335,7 +335,11 @@ The host now sets `MaxFailedAuthenticationAttempts = 0` on every configuration i
- **Two counts per client (#844).** Each *definite* failure is charged to the (client, **credential identity**)
pair and to the client as a whole. The credential identity is `IOpcUaUserAuthenticator.DescribeCredential` —
for API keys `key:<keyId>` (the public lookup index, never the secret), for anything that is not a well-formed key
`user:<username>`.
`user:<username>`. It is **canonical**: the ConfigDb store matches the key id with SQL Server `=` (case-insensitive,
trailing spaces ignored), so the id is trimmed and lower-cased, and an id outside the minted lowercase-hex charset
(which no real key carries) folds into one shared `key:<not-a-minted-id>` identity; the username is trimmed and
lower-cased too. Otherwise `otua_abc…`, `otua_ABC…` and `otua_abc… ` would each get a fresh per-credential count for
one key.
- **Per credential:** at `MaxFailedAttempts` (5) failures of one identity, each within `FailureWindowSeconds` (60) of
the previous, that client is refused **that identity** `BadUserAccessDenied` for `LockoutSeconds` (300), without
the key store being consulted. A success with **that same identity** clears its count.
@@ -852,13 +856,41 @@ Behaviours worth knowing before enabling it:
own `GetNodeMetadata` (so **with the flag off too**, i.e. this predates #826) and in the #826 check. A burst of more
than 4096 events under one hold of `Lock`, starting at a miss (a new item, or right after a deploy invalidated the
verdicts), therefore deadlocked the node manager: `FolderEventFloodWireTests` stalled at exactly 4097 events, flag on
and off. Condition events now go through `ReportEventWithoutBlocking`: `ReportEventAsync` completes inline while the
channels have room (the normal case, unchanged); when one is full that report continues asynchronously, holding no
thread and no lock, and later reports queue behind it so events still leave **in report order**. The deferred work
holds only frozen snapshots — bounded by the burst — and drains once `Lock` is released; what a client receives is
still bounded by its item's queue (oldest discarded). Pinned by `FolderEventFloodWireTests` (6000 events under one
`Lock` hold with publishing disabled: completes, deferred reports drain to zero, the item delivers at most its
revised `QueueSize` in increasing order ending with the last event, and the next event is delivered).
and off. Condition events now go through `ReportEventWithoutBlocking`:
- **Delivery** walks the same path as the SDK's `ReportEventAsync` (the condition's sync `OnReportEvent` → Server-object
items, its own `MonitoredNode2`, then each inverse notifier recursively from a snapshot of the notifier list), and
completes **inline** while the channels have room — the normal case, unchanged. It re-checks **before every hop** that
the condition is still the live instance, and `OnConditionReportEvent` checks the same, so an event that waited past a
deploy that removed or replaced its condition is **dropped, never delivered as a ghost** (a ghost Retain alarm after
the NodeDeleted for ScadaBridge on `i=2253`; with the ACL on, a non-live scripted condition could otherwise be admitted
through its OLD equipment folder, #808).
- **Deferral is per condition.** When a channel is full that event continues asynchronously (no thread, no lock held)
and the condition gets a queue; its later events wait behind it, so **one condition's events stay in report order**,
while conditions whose channels have room are not held up. Events of different conditions may interleave differently
from report order — each carries its condition's full state, so nothing a client derives depends on that.
- **Overflow policy — coalesce, newest wins.** A condition holds at most 1024 waiting events; past that, or while more
than 65 536 are deferred in total, its **oldest not-yet-started** event is discarded. A condition event carries the
condition's complete state, so for every Retain-based consumer the newest supersedes the rest; what is lost is
intermediate transitions on the OPC UA event stream only (alarm history is recorded on the engine/driver path, not
from these events). Dropping the newest would leave subscribers on a stale state; blocking is the deadlock above. Each
condition always keeps its newest, so the ceiling is 65 536 + the number of conditions.
- **Visibility:** entering deferral logs a **Warning** (at most once a minute) naming the first condition; draining logs
an **Information** line with the episode's deferred/discarded counts. Meters: `otopcua.opcua.condition_events.deferred`
(live depth, up/down) and `otopcua.opcua.condition_events.discarded{reason=coalesced|not_live}`.
- **SDK behaviour verified (2.0.1, decompiled):** `NodeState.ReportEventAsync`/`GetNotifiers` copy the notifier list under
the node's own notifier lock, so a concurrent `RemoveNotifier` under `Lock` cannot break a deferred walk (it only
changes which folders a later hop reaches — and the liveness check covers removal); `MonitoredNode2.Dispose` completes
its channel writer, which fails a pending `WriteAsync` with `ChannelClosedException` (swallowed by the SDK), so a
disposed node cannot stall a condition's queue; and the delivery task catches per-event exceptions, so it keeps draining.
- ⚠️ **Residual (documented, not fixed): `ConditionRefresh` does not flush deferrals.** While a notifier channel is
overloaded, a client that refreshes may receive older deferred events of a condition after `RefreshEndEvent`. They
are still in that condition's order and its newest event arrives last, so the state a client ends up holding is
correct; making refresh wait for the queues would block a service call on the very drain the overload is stalling.
Pinned by `FolderEventFloodWireTests`: 6000 events under one `Lock` hold with publishing disabled (flag on and off)
complete, drain to zero, coalesce, and the item delivers at most its revised `QueueSize` in increasing order ending
with the last event; and a condition **removed** in the same `Lock` hold has none of its deferred events delivered to
either the folder or the Server-object item (the flag-off variant fails without the liveness checks).
**Model-change and audit events on the Server object are gated too** (Gitea #588). The node manager's
own server-wide events — `GeneralModelChangeEvent` (NodeAdded / NodeDeleted / DataTypeChanged) and the