Under heavy parallel load (14 concurrent test processes + 14 CPU burners, with
DOTNET_ThreadPool_ForceMinWorkerThreads=1 as the reproducer) the Galaxy tests
with 1-3 s fixed waits failed 16/70 runs. Every failure was a background loop
started with Task.Run (alarm feed, DeployWatcher, EventPump, ReconnectSupervisor)
not being SCHEDULED within the ceiling - e.g. DeployWatcher's stream had not even
been opened after 2 s. No production race: the code under test was checked for
each (alarm feed dispatch is synchronous in-order, DeployWatcher/ReconnectSupervisor
state is lock- or loop-local, subscribe registers before returning).
- New shared TestWait (UntilAsync / ForAsync / HealthyAsync): 30 s presence
ceiling, failure reports elapsed ms + an observed-state snapshot (#856 shape;
GalaxyDriverFixRound1Tests.WaitUntilAsync now delegates to it).
- Converted the short waits in GatewayGalaxyAlarmFeedTests, DeployWatcherTests,
GalaxyDriverSubscribeTests, GalaxyDriverRawPathKeyingTests (its WaitForAsync
never threw on timeout), EventPumpStreamFaultTests, EventPumpReplayGapTests,
ReconnectSupervisorTests; FixRound2 InitBudget 2 s -> 10 s (the parked fetch
never completes, so any finite bound keeps the verdict).
- Absence by short sleep replaced with positive controls: DeployWatcher bootstrap
and same-deploy-time (follow with a real change, assert exactly that one raise -
verified red when suppression is removed), subscribe failed-tag / unsubscribed
handle (follow with a live handle's event), faulted pump (wait for its
onStreamFault instead of 100 ms).
- Fixed sleeps that raced scheduling: subscribe-once read tests wait for the
read's registration (TrackedSubscriptionCountForTests) instead of 50 ms;
supervisor Dispose test waits for reopen entry instead of 50 ms; the
already-Healthy WaitForHealthy test asserts synchronous completion instead of a
wall-clock bound. FakeSubscriber.Map is now a ConcurrentDictionary.
Loaded run (14 parallel x 5, 14 burners, ForceMinWorkerThreads=1):
before 16/70 failed runs; after 0/350 (70 + 210 + 70). Without the thread-pool
override: 0/70 before and after on this host. Serial: 472 passed, 8 skipped.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Found while load-verifying #856: ShutdownAsync_cancels_the_event_stream_and_releases_the_pump
timed out 7/700 under parallel load. The fake's StreamCancelled was set by a
CancellationToken registration scoped by `using` inside the async iterator.
On Cancel() the newer ReadAllAsync registration runs first and resumes the
iterator on a pool thread, whose `using` can dispose our registration while
Cancel() is still walking the callback list - so it never runs. Signal from
the iterator's finally (when the token is cancelled) instead, which also
proves the stream actually ended. 0/700 after.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
PerPlatformProbeWatcher records a platform's handle and publishes the probe
binding into the driver registry as one step under _syncLock, but
WatchedPlatforms read the handle map lock-free - so it could report a
platform one step ahead of the registry Upsert. A ScanState event raised in
that window was dispatched by the EventPump, missed ResolveSubscribers and
was dropped as "stale after unsubscribe", leaving the host status empty.
That is the #856 flake: A_fetch_completing_after_init_populates_host_statuses
waited for WatchedPlatforms, emitted ScanState, and timed out at 10 s on the
final Running wait. Reproduced 11/700 (28 parallel runners x 25 + 14 CPU
burners), every failure at that step with the binding present by the time of
diagnosis and hostStatuses=[] - i.e. the event was dropped. 0/1400 after.
WatchedPlatforms now reads under _syncLock (test seam only; the dispatch path
already checks the map after the registry, the safe order). New
PerPlatformProbeWatcherVisibilityTests spins a reader on another thread and
asserts the registry entry exists the moment the platform is visible
(old code: torn in 1691/2000 iterations). WaitUntilAsync gains a describe
overload; the #856 test's waits now report elapsed time and the observed
seed/binding/pump/host-status chain on timeout.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The Controls panel primed its per-node view from the snapshot store with
ReceivedUtc = now, so a node that stopped publishing before the dialog
opened looked live for the first 30 s stale window.
InMemoryDriverStatusSnapshotStore now records each snapshot's LOCAL
receive time (TimeProvider-injectable) and exposes it via
GetAllWithReceiveTime(); the panel primes through the pure seam
DriverStatusByNode.Prime/PrimedReceiveTime, which keeps the store's
receive time (clamped to now; still skew-proof). A store that does not
track one (default interface impl -> null) falls back to the publish
time only when that is already older than the stale window, else now.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review of 1622d07f:
1. Perf: every i=2253 refresh ran the whole notifier walk + permission checks +
QueueEvent under Lock. Now, with nothing deferred (the common case), the refresh is
the SDK-shaped walk off Lock (Lock only to copy the root notifiers), kept only if no
deferral episode existed at any moment of it (count 0 before and after, episode
unchanged); otherwise the cut path also runs and its later replay supersedes. The
cut path takes ACL/role verdicts off Lock, then one short Lock hold: watermark +
frozen snapshots of Retain conditions from _alarmConditions (no tree walk) + per-item
cut and "replay pending" marker; the replay is queued OFF Lock, one event per hold of
the item gate. While pending, a condition event newer than the cut marks its
condition so the older frozen entry is dropped rather than following it. A
concurrent fast-path refresh cancels a pending cut replay of the same item.
Measured, 3000 retained conditions, longest Lock wait seen by another thread during
one refresh: 1622d07f 95-106 ms idle / 102-127 ms deferring; now 0-3 ms idle /
23-32 ms deferring.
2. ACL: a retained condition omitted from the replay for want of a grant is never
withheld (RefreshCut.RetainedAtCut); only Retain=true events of a condition omitted
because it was NOT retained are.
3. One gate/cut state per item for its lifetime (ConditionalWeakTable keyed by the
item): an unsubscribe + resubscribe between refresh and drain keeps the cut.
4. Cuts carry their deferral episode; ClearStaleRefreshCuts runs when the count
returns to zero, and a stale-episode cut is inert.
5. Dispose guard for the custom refresh (ObjectDisposedException once disposing;
Dispose waits for refreshes in flight).
6. Docs: a refresh waits at most for one in-progress QueueEvent on the item.
Tests: resubscribe variants, ACL-grant-lands test, pending-replay probe test,
dispose test, 3000-condition Lock-hold measurement with a probe pinning that neither
path queues its replay under Lock; stress invariant tightened (never backwards outside a
refresh; each refresh ends on the newest delivered state) and tolerant of the SDK's own
BadRefreshInProgress overlap refusal. Each new test fails against its targeted mutation.
While a folder subscriber's SDK channel is full, a condition's later events wait in
its #838 deferral queue and reach i=2253 items only when the drain gets to them, off
Lock. A ConditionRefresh in that window replayed the current state, then the older
deferred events followed RefreshEnd (stale state, or a condition the refresh had just
dropped re-added until the newest drained).
- Condition events are reported as ConditionEventSnapshot (an InstanceStateSnapshot
carrying a report sequence, condition NodeId and Retain); events queued behind a
deferred report are marked Deferred.
- Server-object refresh (ConditionRefresh and ConditionRefresh2, flag on AND off) now
runs in RefreshServerObjectItems: watermark, "anything deferred?", the replay and its
queueing in ONE Lock hold, so the replay is exactly the state at the cut. Per item it
queues the replay and installs a RefreshCut under the item's gate.
- OnConditionReportEvent holds that gate across "superseded? -> QueueEvent" for
deferred events only. Withheld from that item alone: events at/below the cut whose
condition the replay re-sent, or Retain=true events of a condition it omitted; an
older Retain=false event of an omitted condition is still delivered. Queues, other
Server-object items and folder items are untouched; refresh never waits for a drain.
- Lock order Lock -> item gate -> SDK item lock, same as a synchronous report.
- Meter otopcua.opcua.condition_events.superseded.
- Folder items keep a documented residual (SDK channel backlog + memoised per-item
verdict leave no per-event hook).
Tests: ConditionRefreshDeferralWireTests (deterministic overload by holding the folder
item's SDK lock; 6 variants fail before the fix; probe-driven gate test fails with the
gate removed; concurrent reports/refreshes/stall stress test fails before the fix).
A driver named 'drv' (env form DRV) matched every sibling's id-form variable
env:<PREFIX>DRV_<12HEX>_... as 'another instance', breaking the recommended id
convention fleet-wide - including for a name that pre-dates the rule.
ReferenceNames is split into ReferenceNamesId / ReferenceNamesName; a reference
whose instance part is id-shaped (secret: DRV-<12 hex>, env: DRV_<12 HEX>[_...])
is matched by id only. DescribeNameChangeProblemsAsync also refuses new names
whose env form is DRV or starts with an id form (IsReservedInstanceName) at
create, /raw rename and config-modal update.
Co-Authored-By: Claude <noreply@anthropic.com>
The deploy gate's merged-config path check is now one helper,
CredentialReferenceDeployChecks.MergedOnlyProblems (problems the runtime merge
reveals that the per-config checks do not), shared by the gate and
CredentialReferencePreflight.Scan - so an AuthType switch in one config and a
UserCertificatePath in the other is listed as a Draft finding '(merged with its
devices)' on the banner before a deploy refuses it.
Co-Authored-By: Claude <noreply@anthropic.com>
RawTreeService.CreateDriverAsync / RenameDriverAsync / UpdateDriverAsync run
DriverConfigSaveGate.ValidateNameAsync -> DriverInstanceIdentities.DescribeNameChangeProblemsAsync:
- claim: the new name may not be the instance part (secret: segment or env: part)
of a reference STORED on another same-type driver's DriverConfig or DeviceConfig -
naming a driver 'gw' used to fail every sharer's next deploy;
- pin: a driver may not be renamed away from a name its own references are named
after (unless they also name its id or the new name) - the rename would free that
name for every other driver's save / Test Connect / browse. Re-pointing to the id
in the same config-modal save is allowed; a case-only rename is never refused.
Messages name the other driver, id, device and field path, never a value. The
driver + device reads are pinned on SQL Server by a [SqlServerFact].
Co-Authored-By: Claude <noreply@anthropic.com>
env:<PREFIX><INSTANCE>_... is the per-instance convention for every prefix of a
driver's scope (OTOPCUA_GALAXY_, GALAXY_MXGW_, OTOPCUA_OPCUACLIENT_,
OTOPCUA_HEIDENHAIN_), INSTANCE = a driver id or name upper-cased with every other
character as '_'. A variable whose part after the prefix equals or starts with
(at a '_' boundary) ANOTHER same-type instance's id or name is refused with
CredentialReferenceOtherInstance at save, deploy, Test Connect and browse - the
same DriverCredentialPolicy.DescribeOtherInstanceProblems the secret: rule uses
(now over CollectInstanceReferences / ReferenceNames). Another-instance matches
win over the owner's, as for secrets. Shared variables that name no instance
(GALAXY_MXGW_API_KEY) keep working. HasInstanceSegments now also gates on env:,
so an env:-only browse needs the ConfigDb (fail-closed without it).
Co-Authored-By: Claude <noreply@anthropic.com>
- AdminUI MTConnect driver form: "Comms-loss confirm (ms)" in the Reconnect section, always written
(0 = first refusal, negative = off is never dropped), case-insensitive load, unknown keys preserved,
advisory at >= 90000 ms (the watchdog fires first). Form output byte-pinned and parsed through
MTConnectDriver.ParseOptions on the driver side.
- Driver: a confirmed refused-connection outage now annotates every told CONDITION quality-only
BadNoCommunication (state/ack kept, once per outage, UNAVAILABLE-Bad skipped); the first delivery
afterwards re-diffs all subscribed conditions so unchanged ones get a quality-only Good.
- docker-dev/docker-compose.mtconnect-gate.yml: throwaway Agent in its adapter's network namespace so a
stop is a genuine ConnectionRefused (a plain stopped container is a DNS failure = ambiguous).
- Docs: MTConnect.md comms-loss section + live-gate recipe, docker-dev README, drivers.md rule.
Refs #852
A frozen pre-#847 copy of DriverHealth (test-only telemetry_pre847.proto, generated via
Grpc.Tools in the test project) stands in for an old node / old central: new->new round-trips
the node id; old->new parses as absent (HasReportingNodeId false -> null, not "") with every
other field intact; new->old ignores field 15 without shifting known fields and preserves it as
an unknown field.
DriverStatusPanel (the /hosts Controls dialog) showed one snapshot per driver - the most
recently published across both nodes of the redundant pair - so a driver Faulted on one node
and Healthy on its peer flipped on every publish. It now primes from the store's GetAll() and
keeps one entry per ReportingNodeId (DriverStatusByNode): per-node state chip in the header,
per-node last-success / 5-min errors / last error, per-node stale dimming. A snapshot with no
node id (pre-#847 publisher) renders unlabelled, sorted last. A primed copy older than a racing
live publish for the same node is dropped.
The one failure on record (48 s vs 3 s, load 140-280) was a bare 'Condition
not met within 00:00:45'. It did not reproduce here (0/10 alone and 0/6 full
Host integration runs under 28 CPU burners at load ~300), so no timing was
changed; the timeout now carries the deployment row (status, artifact size,
failure reason), every per-node state (status, started/applied, failure) vs
the expected node ids, and both nodes' cluster views - enough to tell a slow
apply from a node that never got the dispatch from a failed apply.
Co-Authored-By: Claude <noreply@anthropic.com>
RoslynVirtualTagEvaluatorTests' race test failed 1 of 3 full Host
integration runs under 28 CPU burners (load ~300) with 24 of 4,800
evaluations refused 'previous evaluation still running' - impossible for one
sequential loop per id unless a finished run still held its slot after the
caller resumed. It did: ScriptRunGuard released the slot from a ContinueWith
on the run task, and a task is observably complete before its continuations
run, so the next run of the key (the next trigger, or the evaluator's
immediate retry after a raced ObjectDisposedException) could be refused - a
Bad publish for a script that had finished. Same class #789 fixed for the
out-of-process runner.
The release (and the late allocation-breach quarantine, now ahead of it) runs
in the run's own finally, before the task completes; the continuation only
disposes the CTS and observes the fault. New
ScriptRunGuardTests.A_caller_that_awaited_its_run_is_never_refused_by_it
widens the release with a test hook (SlotReleasingForTest) - mutation-checked:
releasing from a continuation again refuses 49 of 50.
The race test now prints elapsed / evaluations / clears / pool state / failure
histogram on every assertion and on its hang detector. After: 3/3 full Host
integration runs pass under the same load. Core.Scripting 298, ScriptedAlarms
109, Calculation 84 green.
Co-Authored-By: Claude <noreply@anthropic.com>
Reproduced 10/10 under 28 CPU burners (load 100-250): every run killed for
157-309 ms of silence with every compile on its dedicated thread - the OS
scheduler, not the pool. A 1 s limit still failed 1/10 at load ~300
(1,012 ms). The limit is now production's (3 x the default 1 s heartbeat):
10/10 at load 360-430. The failure message now carries the burst length,
the limit and the kill line.
Also corrects the doc: its mutation check (compile on the pool again =>
killed for silence) no longer holds - re-checked on a 14-core Mac, the
mutated worker passed at both 150 ms and 1 s, because the host's silence
clock counts any frame and since #789 each compile writes its own
LoadStarted/LoadResult frames inline.
Co-Authored-By: Claude <noreply@anthropic.com>
#838 review of 91e28e87:
1. Ghost events. The deferred report ran the SDK walk off Lock with no
liveness check, so an event waiting behind a full channel when a deploy
removed its condition still reached i=2253 subscribers after the NodeDeleted
(a ghost Retain alarm for ScadaBridge) and folder items. Delivery now walks
the SDK path itself (OnReportEvent, OnReportEventAsync, inverse notifiers,
then the Parent as BaseInstanceState does) re-checking before EVERY hop that
the condition is the live instance; OnConditionReportEvent checks the same,
which also stops a non-live scripted condition being admitted through its
old equipment folder (#808). A guard test pins that only NodeState and
BaseInstanceState define the walk. New wire test removes the condition in
the same Lock hold as a flood; the flag-off variant fails without the checks.
2. Back-pressure. Deferral is per condition (one condition's events stay
ordered; others are not held up). Overflow policy: coalesce, newest wins --
past 1024 waiting per condition or 65,536 total, the oldest not-yet-started
event is discarded (a condition event carries full state). Warning on
entering deferral (<=1/min), Information on draining, meters
otopcua.opcua.condition_events.deferred / .discarded{reason}.
3. Verified by decompiling SDK 2.0.1: the notifier list is snapshotted under the
node's own lock, MonitoredNode2.Dispose completes its writer (pending writes
fail with ChannelClosedException, swallowed), and the consumer catches
per-event exceptions -- no fix needed; documented.
4. ConditionRefresh does not flush deferrals; documented as a residual.
#844 review: DescribeCredential canonicalises the key id (trim, lowercase;
non-hex ids fold into one identity) and the username, since ConfigDbApiKeyStore
matches with SQL Server's case-insensitive, trailing-space-blind `=`.
I-1: Test Connect / browse configs may carry unsaved edits (the device modal merges
the stored DriverConfig with the typed device config). The server
(DriverInstanceIdentities.EffectiveOwnerAsync, used by AdminOperationsActor and
BrowserSessionService) honours the named owner only when every endpoint value
(new DriverCredentialRules.EndpointFields: Galaxy Endpoint, OpcUaClient
EndpointUrl/EndpointUrls/DiscoveryUrl, Heidenhain Host/Port) is one that driver
stores; otherwise the owner is null and its instance-named secrets are refused.
Every Test Connect / browse whose config carries an in-scope secret:/env: reference
is recorded (CredentialUseAudit: IAuditWriter event category CredentialUse + a
structured Information line: user, driver, endpoint hosts, outcome).
L-1: the UserCertificatePath rule is skipped only in an object that explicitly sets
AuthType to non-Certificate (string or ordinal); the deploy gate also checks the
runtime-merged driver+device config so the switch cannot be split from the path; the
form no longer persists the hidden field.
L-2: a ConfigDb fault in the browse check is a refusal; RawBrowseModal catches open
failures into its error message.
L-4: a relative UserCertificateDirectory resolves against AppContext.BaseDirectory.
Co-Authored-By: Claude <noreply@anthropic.com>
#838(a): each notifier's SDK MonitoredNode2 drains a bounded channel (4096,
FullMode=Wait) and the synchronous NodeState.ReportEvent blocks when it is
full. Condition events are reported under Lock, and the delivery task takes
Lock on a verdict-cache miss (the SDK's own GetNodeMetadata, flag off too, and
#826's IsNotifierEventVisibleTo). A burst of >4096 events under one Lock hold
therefore deadlocked the node manager -- FolderEventFloodWireTests stalled at
exactly 4097 events with enforcement on AND off (so it predates #826).
ReportConditionEvent now goes through ReportEventWithoutBlocking:
ReportEventAsync completes inline while channels have room (unchanged normal
path); on overflow the report continues asynchronously holding no thread or
lock, and later reports queue behind it so events keep report order. The
deferred work holds only frozen snapshots and drains once Lock is released;
client delivery stays bounded by each item's queue.
#838(b): documented in docs/security.md that Part 3 ReceiveEvents applies to
the event's EventType and SourceNode (the SDK check, kept first), not to the
notifier; our decision is layered on top and non-condition events on a folder
item fail closed under enforcement. The reflection item stays open.
#835: inside one driver namespace, a secret:<ns>/<X>/... reference whose first
segment names ANOTHER existing same-type driver instance (id or name,
case-insensitive; another-instance match beats an own-name match, so renaming
one's driver after a victim's unlocks nothing) is refused at save
(DriverConfigSaveGate.ValidateAsync), deploy (CredentialReferenceOtherInstance,
Heidenhain included), Test Connect (TestDriverConnect.DriverInstanceId, ConfigDb
read off the mailbox) and live browse (BrowserSessionService, fail-closed without
a ConfigDb). One rule: DriverCredentialPolicy.DescribeOtherInstanceProblems +
DriverInstanceIdentities. Shared names and the driver's own id stay valid; env:
is not instance-scoped. Running drivers are unchanged (warn-only window: the
deploy gate and the upgrade preflight / /raw banner list them).
#836: OpcUaClient UserCertificatePath is a relative path inside
Drivers:OpcUaClient:UserCertificateDirectory (default
<LocalAppData>/OtOpcUa/pki/user-certificates). Syntactic rule (CredentialFilePaths,
a DriverCredentialPolicy path field) at every #815 seam; at load time
ResolveSecretRefsAsync joins, canonicalises and requires lexical AND
link-resolved containment before the PFX password is resolved, and
LoadCertificate refuses an unresolved path. A running driver's pre-#836 absolute
path keeps loading with a rate-limited warning (LegacyCredentialReferenceTracker.ReportPath).
Co-Authored-By: Claude <noreply@anthropic.com>
#820 served DurationGood/DurationBad (ms) on SqlServer/QuestDB but only
QualityMixConformance asserted them. The shared Processed suite now covers
them like the percentages: hand values over tag A (20000/10000/20000/BadNoData
and 0/10000/0) with statuses + timestamps and Duration = Percent x interval;
the SDK-calculator parity theory (forward/reverse/truncated); outage markers;
the only-Bad interval; string tags; and the time-weighted row cap.
Live: SqlServer.IntegrationTests 211/211 (central SQL), QuestDb.IntegrationTests
249 passed / 1 skipped (shared /opt/otopcua-questdb fixture).
Co-Authored-By: Claude <noreply@anthropic.com>
Reproduced 4/10 alone under 28 CPU burners (load ~100-250): outcomes were
timeout, refused, refused, quarantined... - the guard right, but the test
labelled a refusal 'young' only if its OWN stopwatch (started before the
first trigger) read < 600 ms, and the first trigger's dispatch + sibling +
cadence ate that. A ScriptRunRefusedException IS the young-orphan answer
(behind a stuck orphan the guard quarantines instead), so: any refusal
before the quarantine proves the rule; stuck factor 50 (3 s) so the second
trigger lands young with room; loop until three quarantined answers; the one
remaining wall-clock check runs the safe way (quarantine time on the test's
stopwatch >= threshold - the orphan is never older than it); runaway is a
held spin so it cannot end on its own; failures print the timeline.
After: 10/10 under the same load.
Co-Authored-By: Claude <noreply@anthropic.com>
- A /sample or re-baseline failure whose chain holds SocketError.ConnectionRefused, unbroken for
CommsLossConfirmMs (new, default 2000; 0 = first refusal; negative = off), publishes every subscribed
value BadNoCommunication once per outage; the first delivery afterwards republishes all of them from
the index (unchanged values included). Timeouts/resets/DNS/pause keep the #527 watchdog contract.
- SampleCountCeiling clamps the adaptive raise to the /probe bufferSize (never below SampleCount).
- Docs: MTConnect.md comms-loss section + config key; drivers.md rule.
Refs #846
#844: a success cleared the whole per-client count, so a client holding a valid
key could make four guesses at another key id, sign in with its own, and repeat
forever. Failures are now charged to the (client, credential identity) pair --
IOpcUaUserAuthenticator.DescribeCredential, key:<keyId> for API keys -- and to
the client as a whole. A success clears only its own identity's count; the
per-identity threshold locks that identity; a new per-client cap
(OpcUa:CredentialLockout:MaxFailedAttemptsPerClient, default 2 x
MaxFailedAttempts) that a success never resets bounds spraying across key ids
and locks every credential. Slots are bounded on both counts; a waiter whose
identity has room is not held behind one whose identity has none.
#845: (1) store timeouts, miss-budget refusals, store/authenticator exceptions,
an unset pepper and cancellation are reported Transient
(OpcUaUserAuthResult.Unavailable) and released uncounted, so a ConfigDb stall
can no longer lock every keyed client out; the client still sees the same
opaque denial. (2) a full 4096-entry table defers its next prune until the
earliest entry could age out instead of rescanning under the lock per failure.
(3) ApplicationUri deliberately NOT added to the key: on a None channel it is a
free string, so it would let an attacker reset the count per guess --
documented. (4) wire tests: distinct client certificates on a Sign endpoint are
locked out separately; the ApplicationConfigPath branch keeps the SDK lockout
off (falsified by removing the loaded-branch disable).
A legitimate consumer opening several keyed sessions at once from one key
(e.g. a ScadaBridge site node behind one NAT IP on a None endpoint) must
never see BadUserAccessDenied for parallelism. BeginAsync now hands out at
most MaxFailedAttempts - recordedFailures verification slots per key;
further activations queue FIFO and are woken as slots free. A waiter is
refused only when the key locks out (never reaching the authenticator) or,
uncounted, after OpcUa:CredentialLockout:SlotWaitSeconds (default 10,
BadTimeout). Cancelled waiters are dequeued; pruning accounts for waiters.
The security property is unchanged: no more than MaxFailedAttempts
unverified guesses in progress-or-failed per window.
- Reserve an in-flight slot per client before authenticating (TryBegin /
Complete / Abandon under one lock); refuse once recorded failures +
in-flight attempts reach MaxFailedAttempts, so a parallel burst cannot
pass the check before its first failure is recorded.
- An expired lockout resets the count (LockoutSeconds < FailureWindowSeconds
relocked on every single failure after the first lockout).
- Validate OpcUa:CredentialLockout at host start: disabled, or both
durations > 0.
- Log the first refusal of a lockout episode at Information, the rest at
Debug.
- Document that on None endpoints the key is effectively IP-only (NAT
neighbours share a UserName count) and a null peer shares one key.
The proto comment, VirtualTags.md and CLAUDE.md said the soft deadline
measures the script's own execution and that starvation never counts. Only
the time before the run STARTS is excluded; from the start it is wall-clock
time, as in process, so a GC pause or a starved / suspended worker during
the run still counts. A mid-run freeze test is not added: it races the
script's own return against the watchdog's notice; documented as not covered.
Also documents F2 (late notice cancels and counts), F3 (progress ceiling,
killed_unresponsive) and F5 (answer = stalled | retrigger).
1. Orphaned subscriptions (session closed without deleting subs) have Session == null, so the public
MonitoredNode2.InvalidatePermissionCacheForSession never reached their items: a cached allow survived a
deploy, the moved alarm's events were queued and delivered on TransferSubscriptions. Each notifier's
cache is now cleared whole through the SDK's private MonitoredNode2.InvalidatePermissionCaches
(reflection, guard-tested; fallback = per-session drop + Remove/Add of session-less items).
2. The verdict is decided on the condition LIVE under the event's NodeId, not the event's own instance:
the SDK key has no instance identity, so a replaced instance's late event cached a deny its successor
inherited until the next invalidation.
3. Invalidation is skipped while enforcement is off (forced across an on/off authorizer transition) and is
one O(cache) clear per node instead of a whole-cache scan per item.
4. TagReferencingEquipment carries alarm-bearing raw tags only (TagConfigIntent.Alarm); the snapshot adopts
an OrdinalIgnoreCase set instead of copying it again.
Tests (each fails with its fix reverted): FolderEventVerdictInvalidationTests (orphan + deploy + transfer;
replaced instance's late event; ACL-off skip; SDK private-method guard), DeploymentArtifactScopePathTests
(alarm-bearing projection).
otopcua.script.ipc.timeouts gains the tag answer = stalled | retrigger: a
later trigger refused while the key's stalled run is still unanswered was
counted nowhere (not a script refusal, so not ScriptRunGuard's counter).
KeyState carries its Kind for the tag. Test: the stalled-worker test listens
for the instrument and sees one stalled and one retrigger (before: only
stalled).
With the worker soft clock the host's per-run timer is only the transport
deadline and the kill G is armed by the worker's soft notice, so a worker
whose watchdog never sends one (the design treats the worker as untrusted)
held the run, the key's slot and the tag's BadResourceUnavailable forever;
a deferred run had no host clock at all.
WorkerConnection now tracks evaluation progress (any Results frame, or the
submit that made the worker busy again). The heartbeat monitor kills the
worker when evaluations are outstanding and nothing progressed for
3 x the largest outstanding T + G plus the silence limit:
WorkerExitReason.KilledUnresponsive (metric reason killed_unresponsive), no
culprit, attributed like a heartbeat kill (suspects, no strike on a shard).
A legitimate run has answered or been killed by T + G.
Test seams: ProgressCeilingOverride, DropSoftNoticesForTest. Tests: a worker
whose notices are dropped (G = 60 s, so its own fail-fast cannot end it) is
killed KilledUnresponsive after the 1.5 s ceiling and the run is released
(with the check disabled the test times out); positive control: 20 queued
150 ms runs behind one thread under a 1 s ceiling are all answered.
F1: PendingEval.SoftExpired runs on the pipe reader and completed the
caller's outcome inline (production outcomes have no async continuations, by
design, because results resume on the ResultDispatcher). The caller's code
(the runner's Cancel write, the seam's publish) therefore ran on the reader,
which could hold up heartbeats (a false silence kill) and, with blocking I/O,
write to the pipe from the reader. The soft outcome is now posted to the
dispatcher like a result. Test: a 1.5 s continuation on a soft timeout with a
300 ms silence limit resumes on ScriptWorker[t].Results and the worker lives
(before: it resumed on the Reader).
F2: after a transport timeout ("worker unresponsive"), a late soft notice
armed G but nobody sent Cancel, so a polling script could not end itself and
the whole worker was killed at notice + G with a kill strike. PendingEval now
exposes LateSoftExpiry (asynchronous continuations, cancelled when the run
answers); the runner sends the Cancel and counts the timeout
(NoteTimeout) from it. A genuinely slow script whose notice is delayed past
T + G is thus still a timeout. Test: stall past T + G, thaw, a polling
script overruns T: the worker survives G and the key is quarantined for
timeouts, not a hard deadline (before: the worker was killed).
Joseph Doherty