M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
as the revision advanced, up to ~105 s before the address-space outcome.
The coordinator's recovery counts any non-Applying row as an ack, so a
coordinator restart in that window sealed CLEAN while the node was about to
report degraded (#601). The row now stays Applying until ReleasePendingAck
(or the no-publish-actor path) writes the final Applied, clean or with the
degraded reason. A node restart in the window replays the orphan Applying
row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
HandleDeadline persists that backlog first and defers its decision while
any remain, so a node that acked is never timed out as silent; PreRestart
re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
apply to a row still Dispatching/AwaitingApplyAcks.
Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
§12.1 gains the 2026-11-11 windev (VMware VM) perf table: idle p50 581 us
(MISS), kill latency after T+G 8-16 ms (PASS), kill both shards 2.12-2.14 s
(MISS by ~6 %), worker RSS 160 MB compiled / 36 MB warm-cache (PASS under the
revised 200 MB budget), host CPU +131 % vs InProcess (MISS, untuned).
It also records the first Windows run of Category=ScriptWorker: 107/0/2 from
C:\build, all 13 WorkerSoftClockTests cases (5 freezing a real worker with
NtSuspendProcess) green with no code change. From a user-profile directory
the restricted-identity isolation test fails because the restricted token
cannot read the worker dll there (profile ACL), which is environment, not a
defect.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
When the data starts inside an interval (no start bound) and its first value is
Uncertain, the frozen 1.5.378.182 AverageAggregateCalculator integrated the
uncovered span as a ramp from 0 and divided TimeAverage by the whole interval:
GetRegionsInValueSet (ignoreBadData, sloped) overwrites the BadNoData start
region's status with UncertainDataSubNormal in its one branch without a not-Bad
guard. Live AggregateFilter therefore disagreed with HistoryRead and Part 13
§5.4.3.6/§5.4.3.8 ("No Start Bound: Calculate Partial Interval"); QP: Total
188.96 vs 163.96, TimeAverage 9.448 vs 10.931. SDK 2.0.1 stock does the same.
Owner-approved 2026-11-11.
- New app-owned CoveredTimeAverageAggregateCalculator subclasses the baseline
and recomputes only TimeAverage/Total, restoring the Bad status of a region
that starts at a Bad point (only the start bound can). Only such an interval's
value changes; status, timestamps, type and bits are the baseline's. Stepped
and TreatUncertainAsBad=true never reached the defect. CompatibilityAggregate-
Factory routes exactly those two IDs; copied SDK files untouched.
- Golden test: 0 fixture vectors change (every data set has a usable value at
-10 s), asserted. New sweep over late-start variants of every data set (cut
+5 s / +25 s, first non-Bad sample Uncertain, all directions/stepping/configs/
incremental): 150 / 142 of 384 change, each pinned to a baseline-derived
oracle (untouched baseline's value with that sample Good, its status/bits on
the variant itself) - never our calculator. Plus a hand-computed case.
- Live-vs-HistoryRead parity (TimeWeightedAggregator) forward + reverse, no
masks, with discrimination against the pre-#821 baseline; QualityMix now
compares QP's TimeAverage/Total with the live oracle on both SQL backends;
booted-runtime facts test expects 163.96 / 10.93.
- README recorded decision, ServerAggregateConfiguration remarks, oracle
remarks, docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
TwinCATDriverProbe ran a first-phase TCP check against the AmsNetId's
first four octets on the AMS port (801/851). An AmsNetId is not an IP and
the AMS port is not a TCP port, so Test Connect timed out against every
real runtime (Z35474: dialled 5.118.56.93, an unrelated public host).
The probe now opens no socket of its own: it takes the driver's path,
AdsClient.Connect + ReadStateAsync through the local AMS router, bounded
by the probe budget (Task.Run + WaitAsync), and classifies failures as
no local AMS router / no route or target unreachable / timeout / other.
The degrade-to-green 'TCP reachability only' outcome is gone - with no
TCP check nothing was contacted, so a missing router is reported red.
Tests: the regression test listens on the AmsNetId-derived endpoint and
asserts no connection arrives (fails on the old probe); classification
and budget tests drive the handshake through an internal seam.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The owner revised the per-shard worker idle RSS budget from 80 MB to 200 MB
+ scripts on 2026-11-11: compiles run in the worker, so Roslyn's code and
metadata stay resident (measured 159-187 MB) and no GC setting moves it.
The perf harness now reports both RSS rows against the 200 MB constant with
a PASS/MISS mark; A-3 records what the measurement showed.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The frozen 1.5.378.182 StatusAggregateCalculator counts Uncertain regions as
Good under TreatUncertainAsBad=false, so live AggregateFilter PercentGood /
DurationGood disagreed with HistoryRead (Part 13 §5.4.3.31/§5.4.3.33, Mantis
0006166) whenever Uncertain data was present. Owner-approved 2026-11-11.
- New app-owned GoodOnlyStatusAggregateCalculator subclasses the baseline and
replaces only the PercentGood/DurationGood computation (Good regions only);
CompatibilityAggregateFactory routes exactly those two IDs to it. The copied
SDK files are untouched; the other 35 aggregates are byte-identical.
- Golden test: the 64 changed vectors (38 PercentGood + 26 DurationGood) are
exempted exactly — those whose baseline output differs from the same case's
baseline TreatUncertainAsBad=true twin — and pinned to that twin (baseline
evidence, not regenerated from the implementation); plus a full Good-only
pin over all 352 PercentGood/DurationGood vectors and a hand-computed case.
- Live-vs-history parity over Good/Uncertain/Bad mixes (no masks), with a
discrimination check against the pre-#809 baseline; the booted-runtime facts
test now expects 8000 ms / 40 %.
- SdkAggregateOracle: Run/RunLive = the registered factory; RunSdkAsIs = the
untouched baseline; its Part13GoodCalculator duplicate is removed.
- README (recorded decision), ServerAggregateConfiguration remarks,
docs/Historian.md and .claude/rules/historian.md updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Owner decision 2026-11-11: certificateValidation.validateServerHostName stays
opt-in (unset = off). The AdminUI toggle moves from the "Subscriptions, limits &
discovery" panel to the Security panel beside mode/policy, with help text on the
NAT/IP caveat, IPv6 zone ids and the loose first-label match. Binding unchanged
(tri-state, "Default (off)" leaves the key absent). OpcUaClient.md states the
decision; drivers rule updated; source-scan test pins placement + caveats.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Owner decision 2026-11-11 (Java parity): a genuinely empty observation on a
String/Reference or unauthored item is Good "" instead of BadNoCommunication.
Non-string items (numeric, bool, DateTime, arrays) keep BadNoCommunication -
the Agent supplied no value; BadTypeMismatch would wrongly blame the authored
type. The UNAVAILABLE sentinel is unchanged, and a structured observation with
no text stays BadNotSupported (structured check now precedes the empty check).
Parity matrix, quality-mapping section and drivers rule updated.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
With Reconnect.ReplayOnSessionLost=false a recreated gateway session carries none of
the old advises, so every pre-reconnect subscription went silently deaf: nodes kept
their last value as Good and health returned to Healthy (only a Warning log).
ReplayAsync now records the stranded subscriptions and publishes BadNoCommunication
(StatusCodeMap.BadNoCommunication, 0x80310000; value null — no driver value cache)
for every tag of each, straight to OnDataChange so LastSuccessfulRead is not stamped.
GetHealth reports Degraded naming the setting while any stranded subscription is
still registered; it clears on UnsubscribeAsync (the host's re-subscribe path) or
runtime teardown, and pruned ids never pin it. A subscribe after the reconnect is an
ordinary new-session subscription. With nothing else flowing, the comms-loss
watchdog re-initializes the driver after 90 s (a delayed replay).
Tests: Runtime/StrandedSubscriptionTests (6). Docs: drivers/Galaxy.md, which also
no longer claims the replay uses the gateway's ReplaySubscriptions RPC.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
B-1: the subscribe-once read's cleanup unsubscribes what the registry holds at
removal (the replay may have rebound it to new-session handles), not its
original bindings; nothing when the entry still holds dead old-session
handles. Found while testing it: reads were attached to the pump INSTANCE, so
a read across a reconnect's pump restart never saw the replayed value — reads
now listen on a driver-level ReadDataChange raised for whichever pump is live.
I-A: PerPlatformProbeWatcher records the registry session with each probe
handle. Publishing goes through the new fenced TryUpsert and binds only
current-session handles; removal/stop unsubscribe and purge only those; a
seed reply that straddled a reconnect records nothing and is re-issued (max 3).
I-B: PurgeParked takes the session captured BEFORE the unsubscribe RPC and is
a no-op under the park gate if the session changed.
I-C: ReopenAsync starts a new registry session before AND after the gateway
session swap; ReplayAsync does so only when no reopen ran (injected subscriber
/ tests) and replays only subscriptions still holding older-session handles,
so a subscribe issued after the swap is live and not double-advised.
P-1: ReplayAsync restarts the pump before any early return (no entries used to
leave the dead pump in place). P-2: ReplayOnSessionLost=false logs a Warning;
misleading comment about gateway-side replay fixed.
M-a: TryRebind returns BindOutcome; UnknownId releases the handles just
advised for a subscription removed mid-replay. M-b: the replay/re-snapshot
pass reads the session once and stops if it changes. M-c: parked entries drop
their MxEvent when they leave the parked set. M-d (retry a straddled read)
not done.
Tests (TestWait) for each, each checked red against a mutation of its fix.
Refs #859, #861.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
I-2: unclaimed parked events are trimmed by window sequence. Each window gets a
monotonically increasing sequence at Begin, each parked event is stamped with
the newest one, and on every End events stamped before the oldest open
window's sequence are dropped (amortised O(1) via a park-order queue). Under
concurrent reads the open count rarely hit 0, so trailing events used to pile
up to the bound and push genuine early values into overflow.
I-1 / #861: SubscriptionRegistry.StartNewSession() — called FIRST in
ReplayAsync on every reconnect — clears held events, invalidates the reverse
map (subscriptions kept) and bumps a session number. New-session events whose
number an old binding still held now park until the replay's per-subscription
Rebind promotes them, instead of resolving to the wrong tag. TryRegister /
TryRebind refuse a stale session atomically under the park gate: a
SubscribeAsync whose reply straddled a reconnect is re-issued on the new
session (max 3 attempts), a subscribe-once read returns BadCommunicationError,
ResubscribeAllAsync stops if the session changes again. Old-session handle
numbers are never unsubscribed (Remove reports whether the entry belongs to
the current session; the read's late-reply path checks too), and an old
session's pump can no longer resolve through the map (fast path checks the
session).
I-3 (interim): a completed UnsubscribeBulk purges events parked for its
handles (driver + probe watcher). Full fix needs a worker sequence on the
SubscribeBulk reply (mxaccessgw, optional) — documented.
M-1: a resolved handle with a stranded parked list promotes it and resolves
the live event (never parks it). M-2: ResolveOrPark returns the ready events
to replay first, one gate acquisition per event while anything is held.
M-3: SubscribeAsync/read dispose the window before the orphan-release RPC.
M-5: ResolveOrPark remarks document the single-dispatching-pump invariant.
Tests (TestWait): overlapping windows don't accumulate, purge, stranded list,
session fence + map invalidation, stale-session registration refused,
end-to-end replay with swapped handle numbers and event-before-reply,
subscribe and read straddling a reconnect, read deadline closes its window.
Each was checked red against a mutation of the fix it pins.
Refs #859, #861.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
1. ReadAsync (subscribe-once) now opens the early-event window around its
SubscribeBulk + registration; an initial value that overtook the reply was
dropped and the read returned BadTimeout.
2. Register/Rebind write the reverse map and promote parked events in ONE
critical section under the park gate, and ResolveOrPark resolves under that
gate whenever anything is held (lock-free fast path only when nothing is),
so a later live event can no longer overtake the parked initial value. A
resolved handle that still holds a parked list queues behind it.
3. Held events are fenced by a session epoch: each new EventPump calls
StartNewEpoch(), discarding parked AND promoted-but-unreplayed events; an
older pump can neither park nor replay. Item-handle reuse across a reconnect
can no longer put an old-session value on a new tag.
4. ResubscribeAllAsync opens one window per subscription, not across the whole
replay.
5. Promoted-but-unreplayed (ready) events count toward MaxParkedEvents.
6. Begin/End of a window run under the park gate, so N->0 discard and 0->1
open are atomic (no inherited leftovers).
Documented that ParkedEventsReady may run under _registryGate/_syncLock and
relies on the channel's AllowSynchronousContinuations=false.
Tests use TestWait (#858); each new test was checked red against a mutation
of the fix it pins.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
mxaccessgw gives no reply-before-event ordering: the worker completes the STA
command via a RunContinuationsAsynchronously TCS, so the reply is framed on a
thread-pool continuation while the STA already pumps the OnDataChange Advise
queued (drained/framed by a separate loop), and the reply (unary Invoke) and
the event (StreamEvents) are separate HTTP/2 streams. The driver then also
registers handles concurrently with the pump's dispatch loop. MXAccess's
initial value could therefore be dropped as "stale after unsubscribe",
leaving the node at its pre-subscribe value until the next change.
SubscriptionRegistry gains an early-event window: every SubscribeBulk call
site (SubscribeAsync, ResubscribeAllAsync, PerPlatformProbeWatcher) opens
BeginPendingSubscribe() before the RPC and closes it after registering.
Inside the window unknown-handle events are parked (bounded, 100k); the
Register/Rebind that binds a handle promotes them, and the EventPump replays
them on its single dispatch thread before any later event for that handle
(woken by a sentinel when no further event arrives). Closing the last window
discards unclaimed events, so the stale drop is unchanged outside it.
New counters galaxy.events.early_{parked,replayed,overflow}.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review of 1622d07f:
1. Perf: every i=2253 refresh ran the whole notifier walk + permission checks +
QueueEvent under Lock. Now, with nothing deferred (the common case), the refresh is
the SDK-shaped walk off Lock (Lock only to copy the root notifiers), kept only if no
deferral episode existed at any moment of it (count 0 before and after, episode
unchanged); otherwise the cut path also runs and its later replay supersedes. The
cut path takes ACL/role verdicts off Lock, then one short Lock hold: watermark +
frozen snapshots of Retain conditions from _alarmConditions (no tree walk) + per-item
cut and "replay pending" marker; the replay is queued OFF Lock, one event per hold of
the item gate. While pending, a condition event newer than the cut marks its
condition so the older frozen entry is dropped rather than following it. A
concurrent fast-path refresh cancels a pending cut replay of the same item.
Measured, 3000 retained conditions, longest Lock wait seen by another thread during
one refresh: 1622d07f 95-106 ms idle / 102-127 ms deferring; now 0-3 ms idle /
23-32 ms deferring.
2. ACL: a retained condition omitted from the replay for want of a grant is never
withheld (RefreshCut.RetainedAtCut); only Retain=true events of a condition omitted
because it was NOT retained are.
3. One gate/cut state per item for its lifetime (ConditionalWeakTable keyed by the
item): an unsubscribe + resubscribe between refresh and drain keeps the cut.
4. Cuts carry their deferral episode; ClearStaleRefreshCuts runs when the count
returns to zero, and a stale-episode cut is inert.
5. Dispose guard for the custom refresh (ObjectDisposedException once disposing;
Dispose waits for refreshes in flight).
6. Docs: a refresh waits at most for one in-progress QueueEvent on the item.
Tests: resubscribe variants, ACL-grant-lands test, pending-replay probe test,
dispose test, 3000-condition Lock-hold measurement with a probe pinning that neither
path queues its replay under Lock; stress invariant tightened (never backwards outside a
refresh; each refresh ends on the newest delivered state) and tolerant of the SDK's own
BadRefreshInProgress overlap refusal. Each new test fails against its targeted mutation.
While a folder subscriber's SDK channel is full, a condition's later events wait in
its #838 deferral queue and reach i=2253 items only when the drain gets to them, off
Lock. A ConditionRefresh in that window replayed the current state, then the older
deferred events followed RefreshEnd (stale state, or a condition the refresh had just
dropped re-added until the newest drained).
- Condition events are reported as ConditionEventSnapshot (an InstanceStateSnapshot
carrying a report sequence, condition NodeId and Retain); events queued behind a
deferred report are marked Deferred.
- Server-object refresh (ConditionRefresh and ConditionRefresh2, flag on AND off) now
runs in RefreshServerObjectItems: watermark, "anything deferred?", the replay and its
queueing in ONE Lock hold, so the replay is exactly the state at the cut. Per item it
queues the replay and installs a RefreshCut under the item's gate.
- OnConditionReportEvent holds that gate across "superseded? -> QueueEvent" for
deferred events only. Withheld from that item alone: events at/below the cut whose
condition the replay re-sent, or Retain=true events of a condition it omitted; an
older Retain=false event of an omitted condition is still delivered. Queues, other
Server-object items and folder items are untouched; refresh never waits for a drain.
- Lock order Lock -> item gate -> SDK item lock, same as a synchronous report.
- Meter otopcua.opcua.condition_events.superseded.
- Folder items keep a documented residual (SDK channel backlog + memoised per-item
verdict leave no per-event hook).
Tests: ConditionRefreshDeferralWireTests (deterministic overload by holding the folder
item's SDK lock; 6 variants fail before the fix; probe-driven gate test fails with the
gate removed; concurrent reports/refreshes/stall stress test fails before the fix).
- AdminUI MTConnect driver form: "Comms-loss confirm (ms)" in the Reconnect section, always written
(0 = first refusal, negative = off is never dropped), case-insensitive load, unknown keys preserved,
advisory at >= 90000 ms (the watchdog fires first). Form output byte-pinned and parsed through
MTConnectDriver.ParseOptions on the driver side.
- Driver: a confirmed refused-connection outage now annotates every told CONDITION quality-only
BadNoCommunication (state/ack kept, once per outage, UNAVAILABLE-Bad skipped); the first delivery
afterwards re-diffs all subscribed conditions so unchanged ones get a quality-only Good.
- docker-dev/docker-compose.mtconnect-gate.yml: throwaway Agent in its adapter's network namespace so a
stop is a genuine ConnectionRefused (a plain stopped container is a DNS failure = ambiguous).
- Docs: MTConnect.md comms-loss section + live-gate recipe, docker-dev README, drivers.md rule.
Refs #852
#838 review of 91e28e87:
1. Ghost events. The deferred report ran the SDK walk off Lock with no
liveness check, so an event waiting behind a full channel when a deploy
removed its condition still reached i=2253 subscribers after the NodeDeleted
(a ghost Retain alarm for ScadaBridge) and folder items. Delivery now walks
the SDK path itself (OnReportEvent, OnReportEventAsync, inverse notifiers,
then the Parent as BaseInstanceState does) re-checking before EVERY hop that
the condition is the live instance; OnConditionReportEvent checks the same,
which also stops a non-live scripted condition being admitted through its
old equipment folder (#808). A guard test pins that only NodeState and
BaseInstanceState define the walk. New wire test removes the condition in
the same Lock hold as a flood; the flag-off variant fails without the checks.
2. Back-pressure. Deferral is per condition (one condition's events stay
ordered; others are not held up). Overflow policy: coalesce, newest wins --
past 1024 waiting per condition or 65,536 total, the oldest not-yet-started
event is discarded (a condition event carries full state). Warning on
entering deferral (<=1/min), Information on draining, meters
otopcua.opcua.condition_events.deferred / .discarded{reason}.
3. Verified by decompiling SDK 2.0.1: the notifier list is snapshotted under the
node's own lock, MonitoredNode2.Dispose completes its writer (pending writes
fail with ChannelClosedException, swallowed), and the consumer catches
per-event exceptions -- no fix needed; documented.
4. ConditionRefresh does not flush deferrals; documented as a residual.
#844 review: DescribeCredential canonicalises the key id (trim, lowercase;
non-hex ids fold into one identity) and the username, since ConfigDbApiKeyStore
matches with SQL Server's case-insensitive, trailing-space-blind `=`.
#838(a): each notifier's SDK MonitoredNode2 drains a bounded channel (4096,
FullMode=Wait) and the synchronous NodeState.ReportEvent blocks when it is
full. Condition events are reported under Lock, and the delivery task takes
Lock on a verdict-cache miss (the SDK's own GetNodeMetadata, flag off too, and
#826's IsNotifierEventVisibleTo). A burst of >4096 events under one Lock hold
therefore deadlocked the node manager -- FolderEventFloodWireTests stalled at
exactly 4097 events with enforcement on AND off (so it predates #826).
ReportConditionEvent now goes through ReportEventWithoutBlocking:
ReportEventAsync completes inline while channels have room (unchanged normal
path); on overflow the report continues asynchronously holding no thread or
lock, and later reports queue behind it so events keep report order. The
deferred work holds only frozen snapshots and drains once Lock is released;
client delivery stays bounded by each item's queue.
#838(b): documented in docs/security.md that Part 3 ReceiveEvents applies to
the event's EventType and SourceNode (the SDK check, kept first), not to the
notifier; our decision is layered on top and non-condition events on a folder
item fail closed under enforcement. The reflection item stays open.
- A /sample or re-baseline failure whose chain holds SocketError.ConnectionRefused, unbroken for
CommsLossConfirmMs (new, default 2000; 0 = first refusal; negative = off), publishes every subscribed
value BadNoCommunication once per outage; the first delivery afterwards republishes all of them from
the index (unchanged values included). Timeouts/resets/DNS/pause keep the #527 watchdog contract.
- SampleCountCeiling clamps the adaptive raise to the /probe bufferSize (never below SampleCount).
- Docs: MTConnect.md comms-loss section + config key; drivers.md rule.
Refs #846
#844: a success cleared the whole per-client count, so a client holding a valid
key could make four guesses at another key id, sign in with its own, and repeat
forever. Failures are now charged to the (client, credential identity) pair --
IOpcUaUserAuthenticator.DescribeCredential, key:<keyId> for API keys -- and to
the client as a whole. A success clears only its own identity's count; the
per-identity threshold locks that identity; a new per-client cap
(OpcUa:CredentialLockout:MaxFailedAttemptsPerClient, default 2 x
MaxFailedAttempts) that a success never resets bounds spraying across key ids
and locks every credential. Slots are bounded on both counts; a waiter whose
identity has room is not held behind one whose identity has none.
#845: (1) store timeouts, miss-budget refusals, store/authenticator exceptions,
an unset pepper and cancellation are reported Transient
(OpcUaUserAuthResult.Unavailable) and released uncounted, so a ConfigDb stall
can no longer lock every keyed client out; the client still sees the same
opaque denial. (2) a full 4096-entry table defers its next prune until the
earliest entry could age out instead of rescanning under the lock per failure.
(3) ApplicationUri deliberately NOT added to the key: on a None channel it is a
free string, so it would let an attacker reset the count per guess --
documented. (4) wire tests: distinct client certificates on a Sign endpoint are
locked out separately; the ApplicationConfigPath branch keeps the SDK lockout
off (falsified by removing the loaded-branch disable).
A legitimate consumer opening several keyed sessions at once from one key
(e.g. a ScadaBridge site node behind one NAT IP on a None endpoint) must
never see BadUserAccessDenied for parallelism. BeginAsync now hands out at
most MaxFailedAttempts - recordedFailures verification slots per key;
further activations queue FIFO and are woken as slots free. A waiter is
refused only when the key locks out (never reaching the authenticator) or,
uncounted, after OpcUa:CredentialLockout:SlotWaitSeconds (default 10,
BadTimeout). Cancelled waiters are dequeued; pruning accounts for waiters.
The security property is unchanged: no more than MaxFailedAttempts
unverified guesses in progress-or-failed per window.
- Reserve an in-flight slot per client before authenticating (TryBegin /
Complete / Abandon under one lock); refuse once recorded failures +
in-flight attempts reach MaxFailedAttempts, so a parallel burst cannot
pass the check before its first failure is recorded.
- An expired lockout resets the count (LockoutSeconds < FailureWindowSeconds
relocked on every single failure after the first lockout).
- Validate OpcUa:CredentialLockout at host start: disabled, or both
durations > 0.
- Log the first refusal of a lockout episode at Information, the rest at
Debug.
- Document that on None endpoints the key is effectively IP-only (NAT
neighbours share a UserName count) and a null peer shares one key.
The proto comment, VirtualTags.md and CLAUDE.md said the soft deadline
measures the script's own execution and that starvation never counts. Only
the time before the run STARTS is excluded; from the start it is wall-clock
time, as in process, so a GC pause or a starved / suspended worker during
the run still counts. A mid-run freeze test is not added: it races the
script's own return against the watchdog's notice; documented as not covered.
Also documents F2 (late notice cancels and counts), F3 (progress ceiling,
killed_unresponsive) and F5 (answer = stalled | retrigger).
1. Orphaned subscriptions (session closed without deleting subs) have Session == null, so the public
MonitoredNode2.InvalidatePermissionCacheForSession never reached their items: a cached allow survived a
deploy, the moved alarm's events were queued and delivered on TransferSubscriptions. Each notifier's
cache is now cleared whole through the SDK's private MonitoredNode2.InvalidatePermissionCaches
(reflection, guard-tested; fallback = per-session drop + Remove/Add of session-less items).
2. The verdict is decided on the condition LIVE under the event's NodeId, not the event's own instance:
the SDK key has no instance identity, so a replaced instance's late event cached a deny its successor
inherited until the next invalidation.
3. Invalidation is skipped while enforcement is off (forced across an on/off authorizer transition) and is
one O(cache) clear per node instead of a whole-cache scan per item.
4. TagReferencingEquipment carries alarm-bearing raw tags only (TagConfigIntent.Alarm); the snapshot adopts
an OrdinalIgnoreCase set instead of copying it again.
Tests (each fails with its fix reverted): FolderEventVerdictInvalidationTests (orphan + deploy + transfer;
replaced instance's late event; ACL-off skip; SDK private-method guard), DeploymentArtifactScopePathTests
(alarm-bearing projection).
User decision: save, Test Connect, live browse and the deploy gate keep
refusing out-of-namespace credential references, but an ALREADY-DEPLOYED
driver must not fault at the upgrade restart or (Galaxy) at the next
gateway reconnect, since ReopenAsync re-resolves Gateway.ApiKeySecretRef.
- LegacyCredentialReferenceTracker (Core.Abstractions): the one, deprecated,
call-site-chosen compatibility path. Running Galaxy (InitializeAsync,
ReopenAsync, ack fallback) and OpcUaClient (session open) pass it and keep
the pre-#815 semantics (case-insensitive prefixes, Galaxy file:, OpcUaClient
literal pass-through) with a rate-limited Warning (driver/field/allowed
form, never the value) and a credential-compat:<field> diagnostics entry
(both drivers now implement IDriverDiagnostics). Browsers and probes pass
none and stay hard-fail; Heidenhain has none.
- Deploy-gate message names the migration.
- Upgrade preflight: CredentialReferencePreflightService (admin nodes, once
after ApplicationStarted, ConfigDb only, resolves nothing) logs a Warning
per Draft row / Deployed-artifact finding; the /raw banner shows them plus
Running drivers reporting warn-only use (driver-health diagnostics).
- docs/security.md: exact replacement table (incl. the old form default
env:MX_API_KEY, file:, un-namespaced secret:, casing changes), the
ReopenAsync reconnect behaviour, the warn-only window and its removal.
Co-Authored-By: Claude <noreply@anthropic.com>
SDK 2.0.1 checks the server host name in TWO places: Session.OpenAsync (after
the chain is trusted) and DefaultSessionFactory.CreateChannelAsync, against the
certificate in the GetEndpoints reply and before any security-mode test. opc-plc
sends a certificate on its None endpoint, so `connect -u
opc.tcp://10.100.0.35:50000 --validate-host-name` was refused on an
unauthenticated channel, where a host check proves nothing.
All three call sites now pass checkDomain only when the selected endpoint's
SecurityMode != None: OpcUaEndpointSelection.CheckDomainFor for the driver
(OpenSessionOnEndpointAsync, so the failover sweep too) and the AdminUI browser;
the same expression in Client.Shared ConnectToEndpointAsync (shared by
failover). New live tests: driver None + flag on vs :50000 reads; browser None +
flag on vs the NAT fixture browses; the secure + flag refusals still hold.
Docs: name both check sites; a zone-qualified IPv6 host never matches (FindDomain
compares the zone too), so link-local needs validateServerHostName=false; the
match is loose (first label alone, subject DC= parts count, no wildcards), which
caveats the CA recommendation.
Event items on an equipment / raw device folder were fed by the SDK's MonitoredNode2 fan-out with no ACL
decision, so after a deploy moved a scripted alarm (or dropped the reference projecting a native alarm into
an equipment) the OLD folder's subscribers kept receiving it - EventId included - until the rebuild landed,
indefinitely after a failed apply.
- OtOpcUaNodeManager re-implements INodeManager3 so ValidateEventRolePermissions (the SDK's per-event,
per-item hook; base not virtual) adds IsNotifierEventVisibleTo: live condition instance, notifier still a
legitimate notifier of it, and the #586/#808 Server-object rule (IsConditionVisibleTo over
ConditionVisibilityScopes).
- The SDK memoizes that verdict per (item, event type, SourceNode): InvalidateFolderEventVerdicts runs on the
authority/authorizer setters, RevalidateMonitoredItems (deploy + timer) and CompleteAddressSpaceApply.
- ConditionRefresh on direct items goes through RefreshDirectItems (base loop + the same decision).
- New generation field TagReferencingEquipment (RawPath -> EquipmentIds, from the artifact's
UnsTagReferences via AclScopePathResolver.ResolveTagRawPath): a native alarm's wired equipment folder only
counts (folder delivery AND Server-object scopes) while the current generation still references it.
- Secondary #808 edge: UNS variables wired UNS->Raw Organizes by AddReference are tracked and always placed
as reference variables (an Area named like its equipment's id no longer narrows them).
Tests: UnsFolderScopeWireTests (moved alarm on a folder item + ConditionRefresh + rebuild; area named like
equipment id), NativeAlarmFolderEventAclWireTests (dropped reference: folder + Server object + refresh +
restore), DeploymentArtifactScopePathTests (projection). Rig/event tests now supply tag references.
Live gate against the cppagent 2.7 fixture: count=16384 with bufferSize
16384 streams; 16385 is OUT_OF_RANGE ('count' must be less than 16385).
Warn only when SampleCount > bufferSize, pin both sides of the edge
against the live Agent, and correct the #804 wording that said 'at or
above'.
Joseph Doherty