The persistent NAT PKI (ae91e6b7) re-verified 2026-11-11: same thumbprint
1FBF8791... across --force-recreate on 10.100.0.35, matching the committed
opc-plc-nat-server.der; NAT class 7/7 against the shared stack and against a
CI-style throwaway container + docker-exec cert hand-off. MIN_EXECUTED stays
at 440: no hosted v2-integration run has reached the test step (#860).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Review of 1622d07f:
1. Perf: every i=2253 refresh ran the whole notifier walk + permission checks +
QueueEvent under Lock. Now, with nothing deferred (the common case), the refresh is
the SDK-shaped walk off Lock (Lock only to copy the root notifiers), kept only if no
deferral episode existed at any moment of it (count 0 before and after, episode
unchanged); otherwise the cut path also runs and its later replay supersedes. The
cut path takes ACL/role verdicts off Lock, then one short Lock hold: watermark +
frozen snapshots of Retain conditions from _alarmConditions (no tree walk) + per-item
cut and "replay pending" marker; the replay is queued OFF Lock, one event per hold of
the item gate. While pending, a condition event newer than the cut marks its
condition so the older frozen entry is dropped rather than following it. A
concurrent fast-path refresh cancels a pending cut replay of the same item.
Measured, 3000 retained conditions, longest Lock wait seen by another thread during
one refresh: 1622d07f 95-106 ms idle / 102-127 ms deferring; now 0-3 ms idle /
23-32 ms deferring.
2. ACL: a retained condition omitted from the replay for want of a grant is never
withheld (RefreshCut.RetainedAtCut); only Retain=true events of a condition omitted
because it was NOT retained are.
3. One gate/cut state per item for its lifetime (ConditionalWeakTable keyed by the
item): an unsubscribe + resubscribe between refresh and drain keeps the cut.
4. Cuts carry their deferral episode; ClearStaleRefreshCuts runs when the count
returns to zero, and a stale-episode cut is inert.
5. Dispose guard for the custom refresh (ObjectDisposedException once disposing;
Dispose waits for refreshes in flight).
6. Docs: a refresh waits at most for one in-progress QueueEvent on the item.
Tests: resubscribe variants, ACL-grant-lands test, pending-replay probe test,
dispose test, 3000-condition Lock-hold measurement with a probe pinning that neither
path queues its replay under Lock; stress invariant tightened (never backwards outside a
refresh; each refresh ends on the newest delivered state) and tolerant of the SDK's own
BadRefreshInProgress overlap refusal. Each new test fails against its targeted mutation.
While a folder subscriber's SDK channel is full, a condition's later events wait in
its #838 deferral queue and reach i=2253 items only when the drain gets to them, off
Lock. A ConditionRefresh in that window replayed the current state, then the older
deferred events followed RefreshEnd (stale state, or a condition the refresh had just
dropped re-added until the newest drained).
- Condition events are reported as ConditionEventSnapshot (an InstanceStateSnapshot
carrying a report sequence, condition NodeId and Retain); events queued behind a
deferred report are marked Deferred.
- Server-object refresh (ConditionRefresh and ConditionRefresh2, flag on AND off) now
runs in RefreshServerObjectItems: watermark, "anything deferred?", the replay and its
queueing in ONE Lock hold, so the replay is exactly the state at the cut. Per item it
queues the replay and installs a RefreshCut under the item's gate.
- OnConditionReportEvent holds that gate across "superseded? -> QueueEvent" for
deferred events only. Withheld from that item alone: events at/below the cut whose
condition the replay re-sent, or Retain=true events of a condition it omitted; an
older Retain=false event of an omitted condition is still delivered. Queues, other
Server-object items and folder items are untouched; refresh never waits for a drain.
- Lock order Lock -> item gate -> SDK item lock, same as a synchronous report.
- Meter otopcua.opcua.condition_events.superseded.
- Folder items keep a documented residual (SDK channel backlog + memoised per-item
verdict leave no per-event hook).
Tests: ConditionRefreshDeferralWireTests (deterministic overload by holding the folder
item's SDK lock; 6 variants fail before the fix; probe-driven gate test fails with the
gate removed; concurrent reports/refreshes/stall stress test fails before the fix).
A driver named 'drv' (env form DRV) matched every sibling's id-form variable
env:<PREFIX>DRV_<12HEX>_... as 'another instance', breaking the recommended id
convention fleet-wide - including for a name that pre-dates the rule.
ReferenceNames is split into ReferenceNamesId / ReferenceNamesName; a reference
whose instance part is id-shaped (secret: DRV-<12 hex>, env: DRV_<12 HEX>[_...])
is matched by id only. DescribeNameChangeProblemsAsync also refuses new names
whose env form is DRV or starts with an id form (IsReservedInstanceName) at
create, /raw rename and config-modal update.
Co-Authored-By: Claude <noreply@anthropic.com>
The deploy gate's merged-config path check is now one helper,
CredentialReferenceDeployChecks.MergedOnlyProblems (problems the runtime merge
reveals that the per-config checks do not), shared by the gate and
CredentialReferencePreflight.Scan - so an AuthType switch in one config and a
UserCertificatePath in the other is listed as a Draft finding '(merged with its
devices)' on the banner before a deploy refuses it.
Co-Authored-By: Claude <noreply@anthropic.com>
RawTreeService.CreateDriverAsync / RenameDriverAsync / UpdateDriverAsync run
DriverConfigSaveGate.ValidateNameAsync -> DriverInstanceIdentities.DescribeNameChangeProblemsAsync:
- claim: the new name may not be the instance part (secret: segment or env: part)
of a reference STORED on another same-type driver's DriverConfig or DeviceConfig -
naming a driver 'gw' used to fail every sharer's next deploy;
- pin: a driver may not be renamed away from a name its own references are named
after (unless they also name its id or the new name) - the rename would free that
name for every other driver's save / Test Connect / browse. Re-pointing to the id
in the same config-modal save is allowed; a case-only rename is never refused.
Messages name the other driver, id, device and field path, never a value. The
driver + device reads are pinned on SQL Server by a [SqlServerFact].
Co-Authored-By: Claude <noreply@anthropic.com>
env:<PREFIX><INSTANCE>_... is the per-instance convention for every prefix of a
driver's scope (OTOPCUA_GALAXY_, GALAXY_MXGW_, OTOPCUA_OPCUACLIENT_,
OTOPCUA_HEIDENHAIN_), INSTANCE = a driver id or name upper-cased with every other
character as '_'. A variable whose part after the prefix equals or starts with
(at a '_' boundary) ANOTHER same-type instance's id or name is refused with
CredentialReferenceOtherInstance at save, deploy, Test Connect and browse - the
same DriverCredentialPolicy.DescribeOtherInstanceProblems the secret: rule uses
(now over CollectInstanceReferences / ReferenceNames). Another-instance matches
win over the owner's, as for secrets. Shared variables that name no instance
(GALAXY_MXGW_API_KEY) keep working. HasInstanceSegments now also gates on env:,
so an env:-only browse needs the ConfigDb (fail-closed without it).
Co-Authored-By: Claude <noreply@anthropic.com>
- AdminUI MTConnect driver form: "Comms-loss confirm (ms)" in the Reconnect section, always written
(0 = first refusal, negative = off is never dropped), case-insensitive load, unknown keys preserved,
advisory at >= 90000 ms (the watchdog fires first). Form output byte-pinned and parsed through
MTConnectDriver.ParseOptions on the driver side.
- Driver: a confirmed refused-connection outage now annotates every told CONDITION quality-only
BadNoCommunication (state/ack kept, once per outage, UNAVAILABLE-Bad skipped); the first delivery
afterwards re-diffs all subscribed conditions so unchanged ones get a quality-only Good.
- docker-dev/docker-compose.mtconnect-gate.yml: throwaway Agent in its adapter's network namespace so a
stop is a genuine ConnectionRefused (a plain stopped container is a DNS failure = ambiguous).
- Docs: MTConnect.md comms-loss section + live-gate recipe, docker-dev README, drivers.md rule.
Refs #852
A frozen pre-#847 copy of DriverHealth (test-only telemetry_pre847.proto, generated via
Grpc.Tools in the test project) stands in for an old node / old central: new->new round-trips
the node id; old->new parses as absent (HasReportingNodeId false -> null, not "") with every
other field intact; new->old ignores field 15 without shifting known fields and preserves it as
an unknown field.
DriverStatusPanel (the /hosts Controls dialog) showed one snapshot per driver - the most
recently published across both nodes of the redundant pair - so a driver Faulted on one node
and Healthy on its peer flipped on every publish. It now primes from the store's GetAll() and
keeps one entry per ReportingNodeId (DriverStatusByNode): per-node state chip in the header,
per-node last-success / 5-min errors / last error, per-node stale dimming. A snapshot with no
node id (pre-#847 publisher) renders unlabelled, sorted last. A primed copy older than a racing
live publish for the same node is dropped.
The one failure on record (48 s vs 3 s, load 140-280) was a bare 'Condition
not met within 00:00:45'. It did not reproduce here (0/10 alone and 0/6 full
Host integration runs under 28 CPU burners at load ~300), so no timing was
changed; the timeout now carries the deployment row (status, artifact size,
failure reason), every per-node state (status, started/applied, failure) vs
the expected node ids, and both nodes' cluster views - enough to tell a slow
apply from a node that never got the dispatch from a failed apply.
Co-Authored-By: Claude <noreply@anthropic.com>
RoslynVirtualTagEvaluatorTests' race test failed 1 of 3 full Host
integration runs under 28 CPU burners (load ~300) with 24 of 4,800
evaluations refused 'previous evaluation still running' - impossible for one
sequential loop per id unless a finished run still held its slot after the
caller resumed. It did: ScriptRunGuard released the slot from a ContinueWith
on the run task, and a task is observably complete before its continuations
run, so the next run of the key (the next trigger, or the evaluator's
immediate retry after a raced ObjectDisposedException) could be refused - a
Bad publish for a script that had finished. Same class #789 fixed for the
out-of-process runner.
The release (and the late allocation-breach quarantine, now ahead of it) runs
in the run's own finally, before the task completes; the continuation only
disposes the CTS and observes the fault. New
ScriptRunGuardTests.A_caller_that_awaited_its_run_is_never_refused_by_it
widens the release with a test hook (SlotReleasingForTest) - mutation-checked:
releasing from a continuation again refuses 49 of 50.
The race test now prints elapsed / evaluations / clears / pool state / failure
histogram on every assertion and on its hang detector. After: 3/3 full Host
integration runs pass under the same load. Core.Scripting 298, ScriptedAlarms
109, Calculation 84 green.
Co-Authored-By: Claude <noreply@anthropic.com>
Reproduced 10/10 under 28 CPU burners (load 100-250): every run killed for
157-309 ms of silence with every compile on its dedicated thread - the OS
scheduler, not the pool. A 1 s limit still failed 1/10 at load ~300
(1,012 ms). The limit is now production's (3 x the default 1 s heartbeat):
10/10 at load 360-430. The failure message now carries the burst length,
the limit and the kill line.
Also corrects the doc: its mutation check (compile on the pool again =>
killed for silence) no longer holds - re-checked on a 14-core Mac, the
mutated worker passed at both 150 ms and 1 s, because the host's silence
clock counts any frame and since #789 each compile writes its own
LoadStarted/LoadResult frames inline.
Co-Authored-By: Claude <noreply@anthropic.com>
#838 review of 91e28e87:
1. Ghost events. The deferred report ran the SDK walk off Lock with no
liveness check, so an event waiting behind a full channel when a deploy
removed its condition still reached i=2253 subscribers after the NodeDeleted
(a ghost Retain alarm for ScadaBridge) and folder items. Delivery now walks
the SDK path itself (OnReportEvent, OnReportEventAsync, inverse notifiers,
then the Parent as BaseInstanceState does) re-checking before EVERY hop that
the condition is the live instance; OnConditionReportEvent checks the same,
which also stops a non-live scripted condition being admitted through its
old equipment folder (#808). A guard test pins that only NodeState and
BaseInstanceState define the walk. New wire test removes the condition in
the same Lock hold as a flood; the flag-off variant fails without the checks.
2. Back-pressure. Deferral is per condition (one condition's events stay
ordered; others are not held up). Overflow policy: coalesce, newest wins --
past 1024 waiting per condition or 65,536 total, the oldest not-yet-started
event is discarded (a condition event carries full state). Warning on
entering deferral (<=1/min), Information on draining, meters
otopcua.opcua.condition_events.deferred / .discarded{reason}.
3. Verified by decompiling SDK 2.0.1: the notifier list is snapshotted under the
node's own lock, MonitoredNode2.Dispose completes its writer (pending writes
fail with ChannelClosedException, swallowed), and the consumer catches
per-event exceptions -- no fix needed; documented.
4. ConditionRefresh does not flush deferrals; documented as a residual.
#844 review: DescribeCredential canonicalises the key id (trim, lowercase;
non-hex ids fold into one identity) and the username, since ConfigDbApiKeyStore
matches with SQL Server's case-insensitive, trailing-space-blind `=`.
I-1: Test Connect / browse configs may carry unsaved edits (the device modal merges
the stored DriverConfig with the typed device config). The server
(DriverInstanceIdentities.EffectiveOwnerAsync, used by AdminOperationsActor and
BrowserSessionService) honours the named owner only when every endpoint value
(new DriverCredentialRules.EndpointFields: Galaxy Endpoint, OpcUaClient
EndpointUrl/EndpointUrls/DiscoveryUrl, Heidenhain Host/Port) is one that driver
stores; otherwise the owner is null and its instance-named secrets are refused.
Every Test Connect / browse whose config carries an in-scope secret:/env: reference
is recorded (CredentialUseAudit: IAuditWriter event category CredentialUse + a
structured Information line: user, driver, endpoint hosts, outcome).
L-1: the UserCertificatePath rule is skipped only in an object that explicitly sets
AuthType to non-Certificate (string or ordinal); the deploy gate also checks the
runtime-merged driver+device config so the switch cannot be split from the path; the
form no longer persists the hidden field.
L-2: a ConfigDb fault in the browse check is a refusal; RawBrowseModal catches open
failures into its error message.
L-4: a relative UserCertificateDirectory resolves against AppContext.BaseDirectory.
Co-Authored-By: Claude <noreply@anthropic.com>
#838(a): each notifier's SDK MonitoredNode2 drains a bounded channel (4096,
FullMode=Wait) and the synchronous NodeState.ReportEvent blocks when it is
full. Condition events are reported under Lock, and the delivery task takes
Lock on a verdict-cache miss (the SDK's own GetNodeMetadata, flag off too, and
#826's IsNotifierEventVisibleTo). A burst of >4096 events under one Lock hold
therefore deadlocked the node manager -- FolderEventFloodWireTests stalled at
exactly 4097 events with enforcement on AND off (so it predates #826).
ReportConditionEvent now goes through ReportEventWithoutBlocking:
ReportEventAsync completes inline while channels have room (unchanged normal
path); on overflow the report continues asynchronously holding no thread or
lock, and later reports queue behind it so events keep report order. The
deferred work holds only frozen snapshots and drains once Lock is released;
client delivery stays bounded by each item's queue.
#838(b): documented in docs/security.md that Part 3 ReceiveEvents applies to
the event's EventType and SourceNode (the SDK check, kept first), not to the
notifier; our decision is layered on top and non-condition events on a folder
item fail closed under enforcement. The reflection item stays open.
#835: inside one driver namespace, a secret:<ns>/<X>/... reference whose first
segment names ANOTHER existing same-type driver instance (id or name,
case-insensitive; another-instance match beats an own-name match, so renaming
one's driver after a victim's unlocks nothing) is refused at save
(DriverConfigSaveGate.ValidateAsync), deploy (CredentialReferenceOtherInstance,
Heidenhain included), Test Connect (TestDriverConnect.DriverInstanceId, ConfigDb
read off the mailbox) and live browse (BrowserSessionService, fail-closed without
a ConfigDb). One rule: DriverCredentialPolicy.DescribeOtherInstanceProblems +
DriverInstanceIdentities. Shared names and the driver's own id stay valid; env:
is not instance-scoped. Running drivers are unchanged (warn-only window: the
deploy gate and the upgrade preflight / /raw banner list them).
#836: OpcUaClient UserCertificatePath is a relative path inside
Drivers:OpcUaClient:UserCertificateDirectory (default
<LocalAppData>/OtOpcUa/pki/user-certificates). Syntactic rule (CredentialFilePaths,
a DriverCredentialPolicy path field) at every #815 seam; at load time
ResolveSecretRefsAsync joins, canonicalises and requires lexical AND
link-resolved containment before the PFX password is resolved, and
LoadCertificate refuses an unresolved path. A running driver's pre-#836 absolute
path keeps loading with a rate-limited warning (LegacyCredentialReferenceTracker.ReportPath).
Co-Authored-By: Claude <noreply@anthropic.com>
Decision: CI stays hermetic (no dependency on the 10.100.0.35 box or its
state), but must run the same engine. integration-fixtures.sh now takes the
questdb/questdb:<tag> line from the #822 fixture compose file (10.0.1 today,
same as the docker-dev overlay) and fails 'up' if it is missing;
CI_QDB_IMAGE overrides for a trial. Documented in scripts/ci/README.md.
Co-Authored-By: Claude <noreply@anthropic.com>
#820 served DurationGood/DurationBad (ms) on SqlServer/QuestDB but only
QualityMixConformance asserted them. The shared Processed suite now covers
them like the percentages: hand values over tag A (20000/10000/20000/BadNoData
and 0/10000/0) with statuses + timestamps and Duration = Percent x interval;
the SDK-calculator parity theory (forward/reverse/truncated); outage markers;
the only-Bad interval; string tags; and the time-weighted row cap.
Live: SqlServer.IntegrationTests 211/211 (central SQL), QuestDb.IntegrationTests
249 passed / 1 skipped (shared /opt/otopcua-questdb fixture).
Co-Authored-By: Claude <noreply@anthropic.com>
Reproduced 4/10 alone under 28 CPU burners (load ~100-250): outcomes were
timeout, refused, refused, quarantined... - the guard right, but the test
labelled a refusal 'young' only if its OWN stopwatch (started before the
first trigger) read < 600 ms, and the first trigger's dispatch + sibling +
cadence ate that. A ScriptRunRefusedException IS the young-orphan answer
(behind a stuck orphan the guard quarantines instead), so: any refusal
before the quarantine proves the rule; stuck factor 50 (3 s) so the second
trigger lands young with room; loop until three quarantined answers; the one
remaining wall-clock check runs the safe way (quarantine time on the test's
stopwatch >= threshold - the orphan is never older than it); runaway is a
held spin so it cannot end on its own; failures print the timeline.
After: 10/10 under the same load.
Co-Authored-By: Claude <noreply@anthropic.com>
- A /sample or re-baseline failure whose chain holds SocketError.ConnectionRefused, unbroken for
CommsLossConfirmMs (new, default 2000; 0 = first refusal; negative = off), publishes every subscribed
value BadNoCommunication once per outage; the first delivery afterwards republishes all of them from
the index (unchanged values included). Timeouts/resets/DNS/pause keep the #527 watchdog contract.
- SampleCountCeiling clamps the adaptive raise to the /probe bufferSize (never below SampleCount).
- Docs: MTConnect.md comms-loss section + config key; drivers.md rule.
Refs #846
#844: a success cleared the whole per-client count, so a client holding a valid
key could make four guesses at another key id, sign in with its own, and repeat
forever. Failures are now charged to the (client, credential identity) pair --
IOpcUaUserAuthenticator.DescribeCredential, key:<keyId> for API keys -- and to
the client as a whole. A success clears only its own identity's count; the
per-identity threshold locks that identity; a new per-client cap
(OpcUa:CredentialLockout:MaxFailedAttemptsPerClient, default 2 x
MaxFailedAttempts) that a success never resets bounds spraying across key ids
and locks every credential. Slots are bounded on both counts; a waiter whose
identity has room is not held behind one whose identity has none.
#845: (1) store timeouts, miss-budget refusals, store/authenticator exceptions,
an unset pepper and cancellation are reported Transient
(OpcUaUserAuthResult.Unavailable) and released uncounted, so a ConfigDb stall
can no longer lock every keyed client out; the client still sees the same
opaque denial. (2) a full 4096-entry table defers its next prune until the
earliest entry could age out instead of rescanning under the lock per failure.
(3) ApplicationUri deliberately NOT added to the key: on a None channel it is a
free string, so it would let an attacker reset the count per guess --
documented. (4) wire tests: distinct client certificates on a Sign endpoint are
locked out separately; the ApplicationConfigPath branch keeps the SDK lockout
off (falsified by removing the loaded-branch disable).
A legitimate consumer opening several keyed sessions at once from one key
(e.g. a ScadaBridge site node behind one NAT IP on a None endpoint) must
never see BadUserAccessDenied for parallelism. BeginAsync now hands out at
most MaxFailedAttempts - recordedFailures verification slots per key;
further activations queue FIFO and are woken as slots free. A waiter is
refused only when the key locks out (never reaching the authenticator) or,
uncounted, after OpcUa:CredentialLockout:SlotWaitSeconds (default 10,
BadTimeout). Cancelled waiters are dequeued; pruning accounts for waiters.
The security property is unchanged: no more than MaxFailedAttempts
unverified guesses in progress-or-failed per window.
- Reserve an in-flight slot per client before authenticating (TryBegin /
Complete / Abandon under one lock); refuse once recorded failures +
in-flight attempts reach MaxFailedAttempts, so a parallel burst cannot
pass the check before its first failure is recorded.
- An expired lockout resets the count (LockoutSeconds < FailureWindowSeconds
relocked on every single failure after the first lockout).
- Validate OpcUa:CredentialLockout at host start: disabled, or both
durations > 0.
- Log the first refusal of a lockout episode at Information, the rest at
Debug.
- Document that on None endpoints the key is effectively IP-only (NAT
neighbours share a UserName count) and a null peer shares one key.
Joseph Doherty