Profiled the remaining lmxopcua#789 misses on windev, a VMware VM, using per-thread CPU and context-switch accounting
plus ETW CPU sampling.
- One blocking hand-off costs ~26 us of latency and ~18 us of CPU there, about 10x macOS.
- An idle evaluation makes four hand-offs in a row: host writer -> worker reader -> evaluation thread -> host reader
-> dispatcher.
- 48% of host CPU at 10k/s is in the kernel: the per-RunAsync NtWriteFile, thread-pool wakes, reads and waits.
Spin-before-block was measured at every hand-off and is not kept. It meets 150 us idle p50 (118 us) only by keeping
about two host cores and four worker cores busy for one stream. At 10k/s it reads +250 points.
Harness: new "paced 10,000/s, batched" row. It uses the same rate and keys, with each 1 ms tick submitted as one
RunBatch call, which is the production seams' shape. It runs after the original four phases, so their rows stay
comparable. On windev it halves host CPU, to +36..+45 points against +71..+84 unbatched. That is still above 25.
Design doc:
- 12.8: the record.
- 14 #15: the owner's decision on the Windows budgets. Recommended: measure on bare-metal Windows, then state the two
budgets per platform, with the batched row as the gate. The alternative is a shared-memory transport.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Measured on windev (3 alternated sessions) and macOS: a pool resume in place
of the dispatcher, and an inline-on-reader upper bound, with the harness as
committed and with its paced continuation made synchronous.
- pool: 10k/s host CPU unchanged, idle p50 and idle CPU worse (macOS idle CPU
x3 from pool spin); only the closed loop gains. Not kept.
- inline (unsafe for every seam; the ceiling for a VirtualTag-only opt-in):
~8 points of a core at 10k/s raw and ~50 us idle p50 on windev with a cheap
continuation, nothing with the harness as committed. Both budgets still
missed by a wide margin, so the opt-in was not built.
No code change. Records the remaining attribution (harness pool hop ~15
points raw; reader wake + syscalls; worker side untouched).
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
- A filter group owns a deep clone of its first filter, validated once and
evaluated under the group's lock: no item-owned filter is mutated or raced
(Validate mutates a select clause).
- The projection is computed in the SDK's order: where clause first, fields
only for an item that will queue.
- Unambiguous locale key; counter for filters given a private group past the
registry cap; DeliveryCache getter returns its own allocation.
- UseProjectedEventDelivery=false also asks the role and grant checks per
item, so the equivalence test compares against the pre-optimisation path.
- Equivalence test: same select list under three where clauses, a filter
modified mid-run, a session with other locales, an ungranted anonymous
Server-object item, per-rig counter; plus a small overflowing queue with
DiscardOldest true and false.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events delivered to many event items (folder and Server-object)
no longer pay the SDK's per-item QueueEvent(IFilterTarget) work under the
node-manager Lock: the role-permission verdict is shared per (session,
identity), the #826 grant check per identity per hop, and the where clause +
select-clause projection per (value-equal EventFilter, locales); each item
gets its own EventFieldList over the shared read-only fields. The SDK's
QueueEvent body is reproduced under the item's own lock (three private SDK
members, guard-tested); anything outside that shape falls back to the SDK.
The per-report duplicate check replaces the SDK's O(queue) scan.
ProjectedEventDeliveryWireTests compares both paths over the wire.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
§12.1 gains the 2026-11-11 windev (VMware VM) perf table: idle p50 581 us
(MISS), kill latency after T+G 8-16 ms (PASS), kill both shards 2.12-2.14 s
(MISS by ~6 %), worker RSS 160 MB compiled / 36 MB warm-cache (PASS under the
revised 200 MB budget), host CPU +131 % vs InProcess (MISS, untuned).
It also records the first Windows run of Category=ScriptWorker: 107/0/2 from
C:\build, all 13 WorkerSoftClockTests cases (5 freezing a real worker with
NtSuspendProcess) green with no code change. From a user-profile directory
the restricted-identity isolation test fails because the restricted token
cannot read the worker dll there (profile ACL), which is environment, not a
defect.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
Condition events reaching one of our notifiers no longer go through the SDK
MonitoredNode2 channel: the walk serves the attached items itself, per item and
per event (audit guard, uncached INodeManager3 verdict, item gate/cut,
QueueEvent), as the Server-object path does. No notifier of ours defers any
more; the #838 deferral queue and the #854 cut stay as the safety net and
folder items share the cut (RefreshDirectItems, flag on and off). An event
whose walk suspends is marked deferred before it suspends.
Tests produce deferral with a notifier-stall probe instead of holding a folder
item's SDK lock (which would now block the reporter). New
FolderConditionRefreshWireTests; design note in docs/plans.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
The owner revised the per-shard worker idle RSS budget from 80 MB to 200 MB
+ scripts on 2026-11-11: compiles run in the worker, so Roslyn's code and
metadata stay resident (measured 159-187 MB) and no GC setting moves it.
The perf harness now reports both RSS rows against the 200 MB constant with
a PASS/MISS mark; A-3 records what the measurement showed.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
#816 finding: the SDK checks that the server certificate names the host only
when a session is created with checkDomain:true. The driver, the AdminUI
browser and Client.Shared all called the 8-argument
DefaultSessionFactory.CreateAsync, which forwards checkDomain:false — so no
host-name check ever ran (why a trusted NAT cert for *.invalid browsed fine
over an IP in #813). When enabled the SDK (Session.OpenAsync ->
ValidateDomains -> FindDomain) compares ConfiguredEndpoint.EndpointUrl.IdnHost
— the CONFIGURED host, because OpcUaEndpointSelection/EndpointSelector
rewrote the advertised one — against the cert's DNS/IP SANs; secured
sessions only, and the auto-accept handler accepts a mismatch.
Policy: certificateValidation.validateServerHostName, tri-state. Unset =
OpcUaCertificateValidationOptions.DefaultValidateServerHostName = false, the
pre-#816 behaviour, so nothing changes on upgrade; true = validate the
configured host; false = explicit NAT/port-publish opt-out. Threaded into the
driver, the AdminUI browser (same JSON) and the AdminUI form (a select that
keeps "unset" unset), Client.Shared ConnectionSettings.ValidateServerHostName
and the CLI --validate-host-name. A driver refusal names the opt-out.
#818: the NAT fixture's /app/pki is on the opc-plc-nat-pki volume (verified:
same thumbprint across --force-recreate), its PUBLIC cert is committed as
Docker/nat/opc-plc-nat-server.der, and NatPublishedEndpointBrowseTests run
every secure browse on a throwaway PKI (internal browser seam): refusal with
nothing trusted, positive with the cert trusted (unset/false host check), and
a BadCertificateHostNameInvalid refusal with the check on. The trusted cert is
OPCUA_NAT_SERVER_CERT or the committed one (skip on thumbprint mismatch);
OPCUA_NAT_SERVER_CERT_TRUSTED is retired. CI exports its throwaway NAT
container's public cert. ServerHostNameValidationTests pin the driver against
the plain opc-plc. MIN_EXECUTED not raised (needs a hosted run): +6 in
v2-integration (440 -> 446), documented in scripts/ci/README.md.
docs/drivers/OpcUaClient.md, docs/plans/2026-05-28-driver-browsers-design.md and
docs/Client.CLI.md: one OpcUaEndpointUrlRewriter for driver, AdminUI browser and
Client CLI/UI; configured host+port win, advertised path kept for opc.tcp; policy/mode
selection and certificate validation unchanged; NAT test recipe against the #813 fixture.
Owner decisions §14 #5/#6 as built (design §12.5):
- Scripting:Worker:Identity (default unset, Windows only, fail-closed validation): each worker
starts under a restricted token derived from the host's. Every privilege but SeChangeNotify is
removed; the restricting SIDs are NT SERVICE\OtOpcUaScriptWorker's SID + Everyone + Users +
RESTRICTED; the token and default DACLs are host + SYSTEM + identity + RESTRICTED. The worker
is created suspended, put in its Job Object before it runs, inherits only its stdio pipes, gets
an ACL'd work dir as TEMP, and runs with DOTNET_EnableDiagnostics=0. The identity is granted on
the host's window station and desktop, because USER32 init made coreclr.dll fail with
ERROR_DLL_INIT_FAILED on windev. The logon SID is not used because the host's process DACL
gives it PROCESS_VM_READ.
- The worker pipe on Windows always has an explicit protected DACL: host account + identity
only. There is no owner ACE, so an elevated host's Administrators gets no access.
- The worker treats access denied on its parent-pid poll as alive; the Job Object covers that case.
- scripts/install/Install-ScriptWorkerIsolation.ps1 + ScriptWorkerIsolation.psm1 are idempotent
and support -WhatIf, -Uninstall and -PlanOnly. They create the disabled anchor service, the
PeCacheDir ACL (protected, no Users, key and markers host-only, pe\ Modify for the identity),
read+execute on script-worker\, a ReadData deny on the install root, the outbound firewall
block on the worker exe, and optionally the host-service env.
- Tests: the dry-run harness scripts/install/tests/Test-ScriptWorkerIsolation.ps1 (46
assertions, mutation-checked); ScriptWorkerIdentityTests; WorkerIsolationTests, which cover
the pipe ACL, token access and a real restricted worker running a script, all green on windev.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ForbiddenTypeAnalyzer now denies System.AppContext, System.Console, GCSettings, ProfileOptimization,
EncodingProvider/CodePagesEncodingProvider, TypeDescriptor and CryptoConfig as types, and the ambient
CultureInfo.{DefaultThreadCurrentCulture,DefaultThreadCurrentUICulture,CurrentCulture,CurrentUICulture}
and TimeZoneInfo.ClearCachedData members (member deny-list now covers properties and member-access reads).
Shadow-mode tests diverge via culture-sensitive string.Compare (worker is InvariantGlobalization) instead of
AppContext.BaseDirectory, with a premise test. Mutation-proved: 21 new cases fail with the analyzer reverted.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Shadow on central-1/2 with all three seams: zero mismatches on ordinary VTs / Calc / alarms, counted mismatches on
exactly the divergent (value) and decimal (unsupported_type) scripts, kill -9 of the shadow workers left every
published value Good. OutOfProcess with ScriptedAlarm listed: alarms raise and clear, a runaway predicate is killed
twice and quarantined (BadInternalError) with its siblings Active and Good, kill -9 under an in-flight predicate
freezes it Active + BadResourceUnavailable and it re-evaluates in isolation. Answers A-7 (Modbus UInt16 boxes as
ushort) and A-8 (a decimal VT publishes BadNotSupported in process). Rig restored to revision e221b179.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results.
The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a
repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm
engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged.
- Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults
deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run
queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill.
- Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of
process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes
batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults
freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider
carries the runner to DriverHostActor.
- Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial);
design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms;
batched p99 <= 53 ms at 1,000.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
#789 (partial, further tuning deferred): combining inline frame writer, buffered
frame reader, one timer per evaluation for T and T+G, slot release on the reader
before the caller resumes, a non-spinning per-connection result dispatcher, and
blocking pipe I/O on Unix (both ends) so the socket never enters the socket
engine; TerminateJobObject kill on Windows. macOS host CPU at 10k evals/s K=2:
228-240 % -> 79 % (InProcess 69 %, IPC share +10 vs budget 25); windev 153 % ->
123 % (+115, still a miss). Harness reports the "vs InProcess" share, warm-cache
worker RSS, and times kill latency from the soft-timeout answer.
#790: the worker sends LoadStarted; the compile timeout is armed then, so a load
queued behind other compiles is never killed or struck.
#788: one Error with stack per down period, rate-limited Warnings with attempt
count, one Information on recovery.
Design doc §8.2 notes + §12.2 record.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Joseph Doherty