fix(opcua): deferred condition events never outlive their condition; per-condition bounded deferral; canonical lockout identity (#838, #844 review)
#838 review of 91e28e87:
1. Ghost events. The deferred report ran the SDK walk off Lock with no
liveness check, so an event waiting behind a full channel when a deploy
removed its condition still reached i=2253 subscribers after the NodeDeleted
(a ghost Retain alarm for ScadaBridge) and folder items. Delivery now walks
the SDK path itself (OnReportEvent, OnReportEventAsync, inverse notifiers,
then the Parent as BaseInstanceState does) re-checking before EVERY hop that
the condition is the live instance; OnConditionReportEvent checks the same,
which also stops a non-live scripted condition being admitted through its
old equipment folder (#808). A guard test pins that only NodeState and
BaseInstanceState define the walk. New wire test removes the condition in
the same Lock hold as a flood; the flag-off variant fails without the checks.
2. Back-pressure. Deferral is per condition (one condition's events stay
ordered; others are not held up). Overflow policy: coalesce, newest wins --
past 1024 waiting per condition or 65,536 total, the oldest not-yet-started
event is discarded (a condition event carries full state). Warning on
entering deferral (<=1/min), Information on draining, meters
otopcua.opcua.condition_events.deferred / .discarded{reason}.
3. Verified by decompiling SDK 2.0.1: the notifier list is snapshotted under the
node's own lock, MonitoredNode2.Dispose completes its writer (pending writes
fail with ChannelClosedException, swallowed), and the consumer catches
per-event exceptions -- no fix needed; documented.
4. ConditionRefresh does not flush deferrals; documented as a residual.
#844 review: DescribeCredential canonicalises the key id (trim, lowercase;
non-hex ids fold into one identity) and the username, since ConfigDbApiKeyStore
matches with SQL Server's case-insensitive, trailing-space-blind `=`.
This commit is contained in:
+40
-8
@@ -335,7 +335,11 @@ The host now sets `MaxFailedAuthenticationAttempts = 0` on every configuration i
|
||||
- **Two counts per client (#844).** Each *definite* failure is charged to the (client, **credential identity**)
|
||||
pair and to the client as a whole. The credential identity is `IOpcUaUserAuthenticator.DescribeCredential` —
|
||||
for API keys `key:<keyId>` (the public lookup index, never the secret), for anything that is not a well-formed key
|
||||
`user:<username>`.
|
||||
`user:<username>`. It is **canonical**: the ConfigDb store matches the key id with SQL Server `=` (case-insensitive,
|
||||
trailing spaces ignored), so the id is trimmed and lower-cased, and an id outside the minted lowercase-hex charset
|
||||
(which no real key carries) folds into one shared `key:<not-a-minted-id>` identity; the username is trimmed and
|
||||
lower-cased too. Otherwise `otua_abc…`, `otua_ABC…` and `otua_abc… ` would each get a fresh per-credential count for
|
||||
one key.
|
||||
- **Per credential:** at `MaxFailedAttempts` (5) failures of one identity, each within `FailureWindowSeconds` (60) of
|
||||
the previous, that client is refused **that identity** `BadUserAccessDenied` for `LockoutSeconds` (300), without
|
||||
the key store being consulted. A success with **that same identity** clears its count.
|
||||
@@ -852,13 +856,41 @@ Behaviours worth knowing before enabling it:
|
||||
own `GetNodeMetadata` (so **with the flag off too**, i.e. this predates #826) and in the #826 check. A burst of more
|
||||
than 4096 events under one hold of `Lock`, starting at a miss (a new item, or right after a deploy invalidated the
|
||||
verdicts), therefore deadlocked the node manager: `FolderEventFloodWireTests` stalled at exactly 4097 events, flag on
|
||||
and off. Condition events now go through `ReportEventWithoutBlocking`: `ReportEventAsync` completes inline while the
|
||||
channels have room (the normal case, unchanged); when one is full that report continues asynchronously, holding no
|
||||
thread and no lock, and later reports queue behind it so events still leave **in report order**. The deferred work
|
||||
holds only frozen snapshots — bounded by the burst — and drains once `Lock` is released; what a client receives is
|
||||
still bounded by its item's queue (oldest discarded). Pinned by `FolderEventFloodWireTests` (6000 events under one
|
||||
`Lock` hold with publishing disabled: completes, deferred reports drain to zero, the item delivers at most its
|
||||
revised `QueueSize` in increasing order ending with the last event, and the next event is delivered).
|
||||
and off. Condition events now go through `ReportEventWithoutBlocking`:
|
||||
- **Delivery** walks the same path as the SDK's `ReportEventAsync` (the condition's sync `OnReportEvent` → Server-object
|
||||
items, its own `MonitoredNode2`, then each inverse notifier recursively from a snapshot of the notifier list), and
|
||||
completes **inline** while the channels have room — the normal case, unchanged. It re-checks **before every hop** that
|
||||
the condition is still the live instance, and `OnConditionReportEvent` checks the same, so an event that waited past a
|
||||
deploy that removed or replaced its condition is **dropped, never delivered as a ghost** (a ghost Retain alarm after
|
||||
the NodeDeleted for ScadaBridge on `i=2253`; with the ACL on, a non-live scripted condition could otherwise be admitted
|
||||
through its OLD equipment folder, #808).
|
||||
- **Deferral is per condition.** When a channel is full that event continues asynchronously (no thread, no lock held)
|
||||
and the condition gets a queue; its later events wait behind it, so **one condition's events stay in report order**,
|
||||
while conditions whose channels have room are not held up. Events of different conditions may interleave differently
|
||||
from report order — each carries its condition's full state, so nothing a client derives depends on that.
|
||||
- **Overflow policy — coalesce, newest wins.** A condition holds at most 1024 waiting events; past that, or while more
|
||||
than 65 536 are deferred in total, its **oldest not-yet-started** event is discarded. A condition event carries the
|
||||
condition's complete state, so for every Retain-based consumer the newest supersedes the rest; what is lost is
|
||||
intermediate transitions on the OPC UA event stream only (alarm history is recorded on the engine/driver path, not
|
||||
from these events). Dropping the newest would leave subscribers on a stale state; blocking is the deadlock above. Each
|
||||
condition always keeps its newest, so the ceiling is 65 536 + the number of conditions.
|
||||
- **Visibility:** entering deferral logs a **Warning** (at most once a minute) naming the first condition; draining logs
|
||||
an **Information** line with the episode's deferred/discarded counts. Meters: `otopcua.opcua.condition_events.deferred`
|
||||
(live depth, up/down) and `otopcua.opcua.condition_events.discarded{reason=coalesced|not_live}`.
|
||||
- **SDK behaviour verified (2.0.1, decompiled):** `NodeState.ReportEventAsync`/`GetNotifiers` copy the notifier list under
|
||||
the node's own notifier lock, so a concurrent `RemoveNotifier` under `Lock` cannot break a deferred walk (it only
|
||||
changes which folders a later hop reaches — and the liveness check covers removal); `MonitoredNode2.Dispose` completes
|
||||
its channel writer, which fails a pending `WriteAsync` with `ChannelClosedException` (swallowed by the SDK), so a
|
||||
disposed node cannot stall a condition's queue; and the delivery task catches per-event exceptions, so it keeps draining.
|
||||
- ⚠️ **Residual (documented, not fixed): `ConditionRefresh` does not flush deferrals.** While a notifier channel is
|
||||
overloaded, a client that refreshes may receive older deferred events of a condition after `RefreshEndEvent`. They
|
||||
are still in that condition's order and its newest event arrives last, so the state a client ends up holding is
|
||||
correct; making refresh wait for the queues would block a service call on the very drain the overload is stalling.
|
||||
|
||||
Pinned by `FolderEventFloodWireTests`: 6000 events under one `Lock` hold with publishing disabled (flag on and off)
|
||||
complete, drain to zero, coalesce, and the item delivers at most its revised `QueueSize` in increasing order ending
|
||||
with the last event; and a condition **removed** in the same `Lock` hold has none of its deferred events delivered to
|
||||
either the folder or the Server-object item (the flag-off variant fails without the liveness checks).
|
||||
|
||||
**Model-change and audit events on the Server object are gated too** (Gitea #588). The node manager's
|
||||
own server-wide events — `GeneralModelChangeEvent` (NodeAdded / NodeDeleted / DataTypeChanged) and the
|
||||
|
||||
Reference in New Issue
Block a user