F1: PendingEval.SoftExpired runs on the pipe reader and completed the
caller's outcome inline (production outcomes have no async continuations, by
design, because results resume on the ResultDispatcher). The caller's code
(the runner's Cancel write, the seam's publish) therefore ran on the reader,
which could hold up heartbeats (a false silence kill) and, with blocking I/O,
write to the pipe from the reader. The soft outcome is now posted to the
dispatcher like a result. Test: a 1.5 s continuation on a soft timeout with a
300 ms silence limit resumes on ScriptWorker[t].Results and the worker lives
(before: it resumed on the Reader).
F2: after a transport timeout ("worker unresponsive"), a late soft notice
armed G but nobody sent Cancel, so a polling script could not end itself and
the whole worker was killed at notice + G with a kill strike. PendingEval now
exposes LateSoftExpiry (asynchronous continuations, cancelled when the run
answers); the runner sends the Cancel and counts the timeout
(NoteTimeout) from it. A genuinely slow script whose notice is delayed past
T + G is thus still a timeout. Test: stall past T + G, thaw, a polling
script overruns T: the worker survives G and the key is quarantined for
timeouts, not a hard deadline (before: the worker was killed).
A_burst_of_cold_compiles_does_not_starve_the_heartbeat (50 ms heartbeat) and
A_caller_that_blocks_after_its_result_does_not_starve_the_reader (100 ms)
launched their worker under the tight silence kill, so its cold start (launch,
first compile, Roslyn JIT) had to stay within 150/300 ms of silence. Under 28
busy processes on 14 cores that failed 2 in 10 runs in the WARM-UP, killed
for ~165 ms of silence, and replacement workers were killed the same way.
New test seam: WorkerLaunchSpec.HeartbeatSilenceLimit (runner internal ctor
heartbeatSilenceLimit) and WorkerConnection.SilenceLimit. The tests launch
with a 30 s limit, warm, then tighten to three intervals for the phase they
are about; the assertions on that phase are unchanged.
The soft deadline T was armed on the host at submit, so the pipe round trip
and the time a starved worker waits to be scheduled counted against the
script: under load a trivial script reported 'script timed out', in Shadow
mode that was a script mismatch, and at T + G the host killed the worker and
struck the key.
- Protocol (additive, negotiated): Hello/Ready.worker_soft_clock and
EvalResults.soft_expired_eval_ids. protocol_version stays 1; a worker that
does not echo the flag keeps the pre-#827 host clock.
- Worker: the watchdog runs each run's soft clock from the moment an
evaluation thread starts it and reports the crossing (woken early for
budgets shorter than its 50 ms poll). It still never cancels by itself;
its fail-fast is anchored at the soft notice + G + margin.
- Host (PendingEval): the soft notice answers the caller and arms the kill G
later. The host's own timer becomes a transport deadline, T + G from submit
(restarted at a deferred run's start); passing it with no result and no
notice yields TransportTimedOut -> Unavailable 'script worker unresponsive'
(BadResourceUnavailable, recovery = the run's completion), no strike, no
kill, no Cancel. A re-trigger refused by that still-unanswered run is
Unavailable too, not 'previous run still running'. Counted as
otopcua.script.ipc.timeouts{kind}.
- Frame writer: coalescing now carries soft_expired_eval_ids (it silently
dropped them); the host processes soft notices before results in a frame.
- Shadow: unresponsive is shadow_unavailable (Unavailable), never a mismatch.
Tests: WorkerSoftClockTests suspends the real worker (SIGSTOP via libc kill /
NtSuspendProcess) - a stall past T still returns 7; a stall past T + G is a
runner fault with nothing struck; a late-starting runaway is still timed out
at T from its start and killed G after; Shadow stays match /
shadow_unavailable; both clocks enforce T and G; PendingEval state machine.
The test parks the removed child in PostStop (ShutdownGate) to hold the
retired-but-not-terminated window open, but PostStop abandons a shutdown after
DefaultShutdownTimeout (5 s). Under a loaded suite run more than 5 s can pass
between the stop and the host processing the stale burst; the child then
terminates, the host forgets it on Terminated, and the burst sent AS it is
forwarded and re-retained. Reproduced deterministically with a 6 s sleep
before the burst. Not a production path: a terminated actor cannot send.
DriverHostActor gains a childShutdownTimeout test seam (last parameter, passed
to DriverInstanceActor.Props). The test raises it to 5 min, asserts the child
was really stopped (ShutdownCount) before the burst, checks the premise as a
positive control (no Terminated by the time the fence has passed), and expects
Terminated once the gate is released. With an 8 s sleep before the burst the
fixed test passes.
The Test-Connect probe set is resolved inside the AdminOperations Akka starter; resolving
ISecretResolver there walks AddZbSecrets -> IAuditWriter -> ActorAuditWriter, whose ctor needs
AuditWriterActorKey (registered later in the same builder). Found by the docker-dev live gate.
Regression test: resolving the probe set never resolves ISecretResolver (fails before the fix).
Also: TagRowProblem doc comment re-attached; Heidenhain credential help text names the
secret:heidenhain/ and env:OTOPCUA_HEIDENHAIN_ namespaces.
- Review 1: RawTreeService Create/UpdateDriver and Create/UpdateDevice refuse a Heidenhain
config whose password fields are not blank / secret:heidenhain/... / env:OTOPCUA_HEIDENHAIN_...
(DriverConfigSaveGate over HeidenhainOptionsParser.DescribeCredentialProblems: top level,
Access, Devices[], case-insensitive keys; the value is never echoed), so a literal password
never reaches the draft DB. The driver and device modals show the reason and disable Save.
- Review 8: CreateTag/UpdateTag/ImportTags (tag modal, manual entry, CSV import, browse commit)
refuse a ReadWrite Heidenhain tag or a Tag.DataType that disagrees with what the driver
publishes (HeidenhainTagRules: catalog type, Float64 => Double, or the binding's dataType).
The tag modal and manual-entry grid fix the access level at Read and derive the data type from
the address (RawTagRowDefaults). New deploy gate HeidenhainTagDataTypeMismatch.
- docs/drivers/Heidenhain.md: credential namespace convention + why, the save-path and deploy
enforcement, optional-role credential behaviour, BadConfigurationError, backoff/ordering notes.
Tests: HeidenhainSavePathTests (service save path; red with the gates disabled) and new
DraftValidatorHeidenhainTests cases.
- Credential exfiltration (review 2): password references are scoped to
secret:heidenhain/<name> and env:OTOPCUA_HEIDENHAIN_<NAME> only. Enforced by
HeidenhainOptionsParser.TryParseCredentialReference (parser, probe, deploy gate) AND again at
resolve time in HeidenhainCredentials (fail closed without consulting the store or the
environment), so Test Connect can no longer send env:ConnectionStrings__ConfigDb or another
subsystem's secret to an arbitrary Host. The probe also refuses an unsendable password.
- Optional-role credentials (review 6): an unresolvable/NUL/non-Latin-1/>247-byte DNC or DATA
credential never sends a login and never fails the session; the reads needing that role are
BadConfigurationError (capability ConfigurationError) and INSPECT telemetry continues. A
policy refusal inside AcquireAsync is per-tag, not a session teardown.
- Start after DisposeAsync is a no-op (review 3); initial subscription push and Publish are
ordered under one lock (review 4); JSON-null sections are normalised and the acquisition loop
catches every fault (review 5); backoff resets only after a session published a poll batch,
with clamped delays (review 7).
Tests: HeidenhainReviewFixTests + namespace/normalisation tests in HeidenhainConfigTests;
each was confirmed red against the reverted fix.
Review of af5c9744:
1. UNS reference variables (s=Area/Line/Equip/Eff) of a moved or renamed
equipment stayed readable/writable under the OLD names' grants: they now
resolve as ReferenceVariable and are authorized at their split path only
while it equals their live equipment folder's placement (live chain ==
generation chain) + one name; otherwise denied (Read and Write).
2. TryPlaceScriptedCondition now also requires the owner folder's live chain
to match the generation (AuthorizeScriptedCondition gains ownerFolderChain),
so a moved equipment's alarm/children/commands follow its folder.
3. Identifiers containing NUL are denied before the decision cache, keeping
client ids disjoint from the NUL-prefixed resolved-scope keys (a client
could otherwise poison a real folder's cached verdict for its key shard).
4. An id no live node holds is Unknown: placed at [\0unknown] so only
Cluster / Namespace-uns grants decide it (a removed opaque id can no longer
split onto a same-named narrow grant; narrow grantees get no NodeDeleted for
removed UNS folders/vtags/conditions — documented); 4-segment
reference-shaped ids keep their split.
5. A wire test drives placements from DeploymentArtifact.ParseDataPlaneAuthority.
Baseline updates: #797 owner tests (unit + wire) and Sdk2RepeatedAlarmAuthority
now install folder scopes; Sdk2HistoryAuthority's UNS alias used the RAW path
as its UNS id (never minted in production) and now carries EqA/HistoryAlias.
Each new guard was falsified (reverting it fails its test).
Real Heidenhain driver (factory-built from a merged config) against the in-process LSV2
simulator inside DriverInstanceActor: subscribe publishes Good, an OPC write is refused by
the runtime (no IWritable) with nothing sent to the control, an outage stamps
BadNoCommunication, recovery re-publishes the identical value Good, and only allowlisted
telegrams ever reach the simulated control.
UNS Area/Line/Equipment folders are minted under their opaque entity ids
(s=UA-/UL-/EQ-...), virtual tags under s=EQ-.../Name and scripted conditions
(+ #629 children) under s=<ScriptedAlarmId>[/...], while every UNS grant is
planted at a NAME path. DataPlaneAuthorizer split those ids, so with
Security:Acl:Enabled a narrow (Area/Line/Equipment) grant could not Browse,
Read, subscribe or HistoryRead them, and Server-object delivery never reached
it — so a narrow key never held the EventId Ack/Confirm/AddComment need.
- The generation carries UnsFolderScopes (folderId -> path + id chain), built
by DeploymentArtifact from the same AclScopePathResolver as the grants; ids
two entity kinds share are dropped; plumbed through IDataPlaneAuthoritySink.
- OtOpcUaNodeManager.ResolveUnsAclSubject reports live facts (folder id
ancestry / scripted owner binding); attached by the Authorizer setter.
- Evaluate places the subject against the CURRENT generation before the
decision cache, which now caches the walk keyed on the resolved scope.
Folders require live chain == generation chain (a moved equipment is denied
under old and new line until the rebuild). Owned nodes (vtags, conditions,
children) get owner path + a reserved segment, so Tag grants and their
Browse transit never expose them. Conditions share TryPlaceScriptedCondition
with the #797 command gate.
- Server-object visibility of a scripted condition is decided on the
condition alone, so a moved alarm cannot reach its old owner there.
Tests: UnsFolderScopeWireTests (opaque ids, end-to-end browse -> EventId ->
Acknowledge on an Equipment grant; sibling denied; Tag-only grant; moved alarm
and moved equipment stale windows). Rig tests now install folder scopes
explicitly (ServerEventRig.FolderScopes); the #797 recreate test deliberately
asserts the moved alarm is invisible to the Cluster observer in the window.
- HeidenhainDeployChecks (wired into DraftValidator): HeidenhainTagWritable refuses a
ReadWrite access level on any Heidenhain tag, so no node is ever advertised writable;
HeidenhainCredentialPersisted refuses a literal password in DriverConfig/DeviceConfig
(only blank / secret: / env: references; the value is never echoed);
HeidenhainConfigInvalid for out-of-range settings or a tagged device without an endpoint.
- EquipmentTagConfigInspector maps Heidenhain to HeidenhainTagDefinitionFactory.Inspect.
- Driver validates only devices that carry tags (an unconfigured default Device1 no longer
faults the driver).
- Lift an Agent's MTConnectError body under a non-2xx status (cppagent 2.x serves
OUT_OF_RANGE as HTTP 400, NO_DEVICE as 404) into the refusal-marked
InvalidDataException, reading at most 64 KiB. The 400 used to be a transport
fault, so the OUT_OF_RANGE re-baseline never ran against a real 2.x Agent.
- Run the instanceId restart test on every re-baseline snapshot: on 2.x a restart
is first seen there (the old cursor gets a 400), and was adopted silently.
- Report the stream Degraded after 3 consecutive re-baselines with no /sample
chunk (it read Healthy for ever); LastSuccessfulRead keeps advancing.
- Prohibit DTDs on the /probe and write-answer parses; bound the streams walk to
64 levels (unbounded recursion = uncatchable StackOverflowException).
- Skip structural elements under <Components>/<Devices> instead of failing the probe.
- Never serve a non-finite float as Good; serve integer-valued decimals on
integral tags losslessly (a real fraction stays BadOutOfRange).
- Merge sibling browse containers that resolve to one folder name.
Unit 678 -> 714, integration 18 -> 20 (live fixture); every gap test failed first.
Native LSV2 TCP transport (default port 19000; no Java/Python/vendor SDK), ported from
the ignitionoee v0.20.3 reference driver:
- Lsv2ReadOnlyPolicy: payload-aware outbound allowlist (A_LG INSPECT/DNC/DATA, R_VR, R_PR,
R_RI 22-28/30-34/51, bounded R_MB, R_DP), applied in Lsv2Codec.Frame before any byte is
written; no public raw-command API.
- Lsv2Client: serialized transactions, one monotonic deadline per transaction, EOF/oversize/
unexpected/trickle close the client; Close never waits on the gate.
- Lsv2Data / HeidenhainBindingCodec: reference decoders, R_PR byte-capacity + element-index
rules, next-area cap, DATA Boolean 00/01/FF, strict non-truncating coercion.
- HeidenhainSession: per-batch wire cache, refusal suppression, provisional DNC denial retry,
alarm cursor (57 = end, overflow = BadOutOfRange), capability strings.
- HeidenhainDeviceRuntime: one loop per control, generation-guarded publication, all-node
BadNoCommunication on comms loss, bounded exponential reconnect.
- HeidenhainDriver: cached ReadAsync, push subscriptions deduped on (value,status), static
catalog discovery (no connect), host status, diagnostics, skipped-tag report. No IWritable.
- Probe (INSPECT login + version) and factory registered in DriverFactoryBootstrap (probe on
admin nodes too); DriverTypeNames.Heidenhain.
Tests: 159 unit tests against an in-process LSV2 simulator incl. the Java fixtures' captured
iTNC 530 replies as golden vectors; env-gated LiveIntegration category for real hardware.
QualityMixConformance (SqlServer + QuestDB): tag Q mixes Good/Uncertain/Bad
regions across eight 10 s intervals (two all-Uncertain - Mantis 0006166 0%/0% -
one through an Uncertain bound, plus Uncertain bounds from a sloped non-Good end
and from a hold into an outage); tag QP has data starting and ending inside
intervals. PercentGood/PercentBad are asserted against hand-derived Part 13
DurationGood/DurationBad, the oracle is checked against the same hand values,
and Count/Average/TimeAverage(2)/Total(2)/Start/End/Delta/Interpolative are
compared with the #796 oracle forward, reverse and truncated.
Teeth: with SdkAggregateOracle.Run answering PercentGood from the baseline
(Uncertain as Good), 8 tests fail per backend; all pass with the fixed oracle.
Records a new SDK divergence: with an Uncertain first sample the baseline
TimeAverage/Total integrate the pre-data span as a ramp from 0; ours covers
only the data (Part 13) - pinned by hand.
A wrong published value now reports its Success/Value/Reason/StatusCode, the
publish's wall time against the run budget, the run guard's in-flight /
refused / quarantined counters, thread-pool and GC state around the call,
the shadow stats, and the tails of the script and runner logs.
Reproduced under heavy load (32 `yes`, test at nice 20, load avg 300+):
3 of 8 runs failed, every one because the SIBLING's shadow run missed the
300 ms soft deadline in the starved worker ("shadow Failed [budget] script
timed out after 0.3s") and was rightly recorded as a sibling mismatch. No
published value was ever wrong. The out-of-process soft deadline is armed at
submit, so it covers the pipe round trip and worker scheduling too.
The run budget (which is also the worker's soft deadline) goes to the 2 s
production default; no assertion changes. The runaway still runs away and is
killed at T + HardGrace, so nothing the test proves depends on T being short.
docs/drivers/OpcUaClient.md, docs/plans/2026-05-28-driver-browsers-design.md and
docs/Client.CLI.md: one OpcUaEndpointUrlRewriter for driver, AdminUI browser and
Client CLI/UI; configured host+port win, advertised path kept for opc.tcp; policy/mode
selection and certificate validation unchanged; NAT test recipe against the #813 fixture.
Docker/nat/docker-compose.yml (stack /opt/otopcua-opcuaclient-nat on 10.100.0.35,
project: lmxopcua label): opc-plc 2.14.10 advertising --ph=otopcua-opc-plc-nat-private.invalid
on internal 50000, published on 50010, so every advertised URL is unreachable on host AND
port. Deployed; NatPublishedEndpointBrowseTests 3/3 against opc.tcp://10.100.0.35:50010
(and the secure case positive once, with the fixture cert temporarily trusted).
CI: integration-fixtures.sh starts a throwaway otopcua-ci-opcplc-nat (50110->50000) and
exports OPCUA_NAT_ENDPOINT; v2-e2e.yml points it at the shared stack. infra/README.md,
scripts/ci/README.md and the fixture README document it.
Client.Shared now project-references Driver.OpcUaClient.Contracts (Core.Abstractions
only, no NuGet/SDK; pinned by ContractsIsDependencyLightTests) and delegates the
host/port rewrite to OpcUaEndpointUrlRewriter, deleting its own copy.
The two copies differed; reconciled toward the client so its pinned behaviour is
unchanged (EndpointSelectorRewriteParityTests, written green against the old copy):
- no-op rewrite (same host case-insensitively + same port) now returns the advertised
string verbatim in the shared rewriter too (it re-serialised: added a trailing slash
and case-folded a case-only host difference);
- a real rewrite serialises via AbsoluteUri, not Uri.ToString(), which un-escaped the
advertised path (/x%20y became /x y) in the driver + browser;
- unusable URLs: the rewriter still falls back to the configured URL (driver/browser);
the client keeps its loud failure as call-site validation (null -> InvalidOperation,
unparseable -> UriFormatException).
Reuses the #806 rewrite cases plus the Client.Shared-specific edges (verbatim
no-op, case-only host difference, escaped path, loud failure on unusable
URLs). Green against the pre-#812 copy: 343/343.
Review of 3d1b86b4: OpcUaPublishActor installs a new authority generation
before applying its address space, and an apply that throws keeps the old one.
A scripted alarm moved from equipment A to B therefore stayed commandable with
A's grant under B's generation (briefly before the rebuild, or indefinitely
after a failed apply).
The generation now carries ScriptedAlarmOwners (ScriptedAlarmId -> EquipmentId,
decoded by ParseDataPlaneAuthority from the artifact's ScriptedAlarms rows, the
same rows the composition materialises from). AuthorizeScriptedCondition denies
unless the node manager's live binding names the same owner as the generation:
a moved alarm is denied under the old owner's, the new owner's and a Cluster
grant until the address space re-binds it; an alarm the generation no longer
deploys is denied. Install/apply order unchanged.
Tests: window (a) generation moved / binding stale => both owners and Cluster
denied, then rebuild under the new owner => new owner allowed, old denied;
window (b) apply failed after install => denied across clock and further
generations, then a successful rebuild restores the new owner only; undeployed
alarm denied; artifact decode of the owner map.
Scripted condition NodeIds are opaque ScriptedAlarmIds, so splitting them as a
topology path could never match an Equipment grant: a narrow equipment grant was
denied while a Cluster grant passed.
- OtOpcUaNodeManager binds each LIVE scripted condition instance to the
EquipmentId it was materialised under (only when it hangs under that folder);
dropped on remove / drop-recreate / equipment-subtree removal / rebuild; a
same-id re-materialise under another equipment re-parents and re-binds.
- DataPlaneAuthoritySnapshot carries EquipmentScopePaths (EquipmentId ->
[uns, Area, Line, Equipment]) decoded from the artifact by the same
AclScopePathResolver call that plants Equipment grants.
- DataPlaneAuthorizer.AuthorizeScriptedCondition re-reads key, grant and
generation per call (uncached) and evaluates at the owner path; unbound,
replaced or unplaceable owners deny under every grant. No id-split or
realm-root fallback.
- Native conditions (RawPath) and the ACL-off role gate are unchanged.
NodeIds and AlarmCommand.AlarmId are unchanged.
Sdk2OpaqueScriptedScopeBaselineTests deliberately flipped: it pinned the denial
and now pins owner-grant allow / sibling deny / cluster allow. Follow-up #808
(folder ids + Server-object event visibility for narrow UNS grants).
ShadowModeTests' runaway case asserted "slowest < 250 ms" around evaluations
whose timing included cold Roslyn compile/JIT of new script text, so it failed
on a healthy published path (540-880 ms measured, both SDK baselines).
The test now puts a test-owned barrier runner (GatedShadow) between the real
ShadowScriptRunner and the real worker pool. The runaway's first shadow run is
held before it reaches the pool, then (after it runs away in the worker, the
pool answers T and the worker is KilledDeadline with the runaway as culprit)
its outcome is held before comparison. Published evaluations made while a
barrier is closed must return the in-process values; a Mirror that waited for
its shadow hangs into a 30 s liveness failure (mutation-checked). Output
correctness, pool kill/quarantine, published-side never quarantined, the single
runaway mismatch (sibling is a victim, not compared) and key_busy drops are kept.
The wall-clock number moves to an opt-in Category=Perf case
(Shadow_published_path_latency_while_the_shadow_runs_away, OTOPCUA_SCRIPT_PERF=1)
on warmed evaluators, reporting hardware and load average.
Sibling: ContainmentTests' hard-deadline case timed the runaway's first
(cold-compile) run; it now compiles the runaway's source first with an input
that does not spin.
The 1.5.378.182 StatusAggregateCalculator (vendored as the application's
compatibility baseline) counts Uncertain regions as Good time under
TreatUncertainAsBad = false; 1.5.378.106 counted Good regions only, as Part 13
§5.4.3.31/§5.4.3.33 and Mantis 0006166 require. Our evaluator follows Part 13,
so the oracle moved, not the server: 10 SdkAggregateParityTests failed.
SdkAggregateOracle.Run now answers PercentGood through a subclass of the
baseline StatusAggregateCalculator that counts Good regions only (same bounds,
regions, status/Partial bits and configuration as PercentBad). RunSdkAsIs keeps
the untouched baseline. Rejected the proposed 100 - PercentBad under
TreatUncertainAsBad = true: that flag also changes the region machinery (the
end-of-data 1 ms rule), so it computes over a different total; pinned by a test.
New tests pin the divergence on hand-computed G/U/B/G (50 vs baseline 75) and
all-Uncertain (0/0 vs baseline 100) intervals, forward and reverse, plus a
data-starts-inside interval where oracle == baseline.
Sql.Common.Tests 265/275 -> 280/280.
The AdminUI OPC UA browse picker connected to the EndpointUrl the server
advertised in GetEndpoints verbatim, so behind NAT / a container port
publish it dialled the server's private hostname + internal port and
failed DNS where the runtime driver (which rewrites per #676) connected.
- Move the rewrite to the shared OpcUaEndpointUrlRewriter (.Contracts,
pure Uri logic, no SDK dependency) and the policy+mode selection to
OpcUaEndpointSelection (.Session leaf). Driver and browser both select
through it, so the browse connects exactly where the driver does.
- Policy/mode filter, server certificate and strict browse PKI
(no auto-accept) are unchanged.
- Unit tests: rewrite (host/port differ, same, IPv6 both sides, path
kept, port-less, scheme mismatch untouched), browser selection seam.
- Live NAT tests (Fixture=opc-plc-nat, OPCUA_NAT_ENDPOINT): browse via a
published port with a premise check that the advertised host:port
differs and does not resolve, nsu= round-trip, unreachable endpoint
fails cleanly, SignAndEncrypt reaches the server and is refused
BadCertificateUntrusted. Verified red without the rewrite.
Refs #806
Joseph Doherty
dohertj2