Merge branch 'master' into worktree-agent-a720278a57ae41792

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 13:25:58 -05:00
69 changed files with 6709 additions and 915 deletions
+20 -13
View File
@@ -1827,10 +1827,13 @@ is `Good` (plus Calculated / Partial). Value = `100 × region time ÷ interval d
**`DurationGood` / `DurationBad`** (#820) are the same Good / Bad region times as `PercentGood` / `PercentBad` — every rule above
(regions, simple bounds, time before the first sample = Bad, end of data, `BadNoData`, status, Partial) applies unchanged — reported as
an OPC UA `Duration`, i.e. a `double` in **milliseconds** (the SDK's unit): `DurationGood = PercentGood × interval ms ÷ 100`. Uncertain
time counts toward neither (Mantis 0006166). Before #820 both were `BadAggregateNotSupported` on the SQL backends. ⚠️ A **live**
`AggregateFilter` subscription computes DurationGood/PercentGood with the frozen 1.5.378.182 calculators, which count Uncertain time as
Good — they disagree with HistoryRead whenever Uncertain data is present (#809, owner decision pending). The Gateway backend still
refuses both (`BadAggregateNotSupported`, not advertised in i=11201).
time counts toward neither (Mantis 0006166). Before #820 both were `BadAggregateNotSupported` on the SQL backends. ⚠️ **#809
(client-visible, owner-approved 2026-11-11):** a **live** `AggregateFilter` subscription now computes DurationGood/PercentGood the same
way — Good time only. They used to come from the frozen 1.5.378.182 calculators, which count Uncertain time as Good; the server now
serves those two through the application's `GoodOnlyStatusAggregateCalculator` (every other live aggregate is still the frozen
baseline). A live PercentGood/DurationGood over Uncertain data therefore drops by the Uncertain share. Pinned live-vs-history by
`SdkAggregateParityTests.Live_AggregateFilter_PercentGood_and_DurationGood_equal_HistoryRead_over_Uncertain_samples`. The Gateway
backend still refuses both (`BadAggregateNotSupported`, not advertised in i=11201).
All carry the Calculated bit. A non-Bad result also carries the **Partial** bit (`0x4`, `AggregateBits.Partial`) — never on a Bad
result — for: the **truncated interval** (shorter than the processing interval — the latest one forward, the earliest one reverse;
@@ -1903,15 +1906,19 @@ timestamp, and the same treatment as the Min/Max-at-interval-start case above; m
And an interval holding a **Good String (or DateTime) sample** is `BadAggregateInvalidInputs` for Average/Minimum/Maximum/Range/
*ActualTime here (the documented #684 refusal rule — a Good non-numeric value is not a number to aggregate), while the SDK skips the
string and serves the numeric samples (e.g. Average 4 Good); an Uncertain non-numeric sample refuses nothing on either side.
One more, found by the #810 Uncertain conformance data (kept deliberately): over the interval the data **starts inside** when the
**first sample is Uncertain**, the SDK's `TimeAverage`/`Total` integrate the uncovered span before it as a ramp from 0 (QP: `Total`
188.96 for 163.96 over the covered 15 s, and `TimeAverage` 9.448 = 188.96 ÷ 20 s for 10.931 = 163.96 ÷ 15 s); ours integrates only
the covered part (Part 13, as with a Good first sample, where all agree). The same answer comes from the frozen calculators the server
registers for **live** `AggregateFilter` items and from SDK 2.0.1's own stock calculators, in both read directions — pinned by
`SdkAggregateParityTests.An_Uncertain_first_sample_in_a_late_start_interval_makes_the_SDK_integrate_the_time_before_the_data`
(ours / frozen / stock 2.0.1, with a Good-first control) and `OpcUaServer.IntegrationTests/LiveAggregateLateStartFactsTests` (through
the booted server's runtime `AggregateManager`), and on both SQL backends by `QualityMixConformance.Time_weighted_values_over_partial_data_cover_only_the_data`
(forward) and `…_in_a_reverse_read` (#821). Whether live subscriptions should follow is the open #809/#821 owner decision. The same
One more, found by the #810 Uncertain conformance data: over the interval the data **starts inside** when the **first sample is
Uncertain**, the SDK's `TimeAverage`/`Total` integrate the uncovered span before it as a ramp from 0 (QP: `Total` 188.96 for 163.96
over the covered 15 s, and `TimeAverage` 9.448 = 188.96 ÷ 20 s for 10.931 = 163.96 ÷ 15 s); ours integrates only the covered part
(Part 13 "No Start Bound: Calculate Partial Interval", as with a Good first sample, where all agree). SDK 2.0.1's stock calculators
answer the same as the frozen 1.5.378.182 ones, in both read directions. ⚠️ **Since #821 (owner-approved 2026-11-11, client-visible)
the LIVE `AggregateFilter` calculators follow Part 13 here too** — `CoveredTimeAverageAggregateCalculator` overrides only
TimeAverage/Total of the frozen copies (only the value of such an interval changes; status, timestamps and bits are the baseline's).
Pinned by `SdkAggregateParityTests.An_Uncertain_first_sample_in_a_late_start_interval_makes_the_SDK_integrate_the_time_before_the_data`
(ours / live / untouched baseline / stock 2.0.1, with a Good-first control), `…Live_AggregateFilter_TimeAverage_and_Total_equal_HistoryRead_over_an_Uncertain_first_late_start`
(forward + reverse, no masks), `OpcUaServer.IntegrationTests/LiveAggregateLateStartFactsTests` (through the booted server's runtime
`AggregateManager`), `Sdk2AggregateCompatibilityTests.Issue821_*`, and on both SQL backends by `QualityMixConformance` (QP's
TimeAverage/Total against the live oracle, and `Time_weighted_values_over_partial_data_cover_only_the_data` forward / `…_in_a_reverse_read`).
(#809, the PercentGood/DurationGood half, is likewise decided: live follows Part 13.) The same
suite feeds PercentGood/PercentBad, DurationGood/DurationBad (and Count, Average, TimeAverage(2), Total(2), Start/End/Delta, Interpolative) Good/Uncertain/Bad
mixes, all-Uncertain intervals (0 % Good **and** 0 % Bad, Mantis 0006166) and partial-data intervals, hand-computed and against the
#796 oracle — the baseline calculator's Uncertain-as-Good PercentGood fails it.
+15
View File
@@ -676,6 +676,21 @@ deployment green fleet-wide. Now:
no second rebuild), reporting clean only if that pass is clean.
- **A node with a ConfigDb also writes the reason on its own row** (`ApplyAck.DegradedReasonPrefix`), because
central only records it when it *accepts* the ack — a timed-out deployment's late ack is discarded.
**That row stays `Applying` until the outcome is known** (lmxopcua#868): it used to be written `Applied`/clean
the moment the revision advanced, and `ConfigPublishCoordinator`'s recovery counts any non-`Applying` row as
an ack, so a coordinator restart inside the ~105 s outcome window sealed the deployment clean while the node
was about to report degraded. `/fleet` therefore shows the node *Applying* until its outcome lands.
A restart of the node inside that window depends on how it boots:
- a `Direct` node that can read the ConfigDb finds the orphan `Applying` row and **replays** the apply, which
acks the real outcome;
- a node that boots **from its LocalDb cache** (every `FetchAndCache` driver-only node, or a `Direct` node
whose ConfigDb is unreachable) restores the cached revision — the cache is written before the outcome — and
sends **no ack**, so central times that deployment out even though the node runs it. The restored revision
is marked *unconfirmed*, so **re-deploying it re-applies** and acks the real outcome rather than
short-circuiting a clean `Applied`.
- **Upgrade note (lmxopcua#868): do not deploy while the central pair is mid rolling-upgrade across this
change.** An old-build fused central node still writes its own row `Applied` before the outcome, which a
new-build coordinator's recovery would count as a clean ack. Finish upgrading both central nodes, then deploy.
- **Mixed-version fleets.** An old node never sets the flag, so its acks are treated exactly as before
(clean). A new node acking an old central: the old coordinator ignores the unknown field and seals as it
always did — no worse than today.
+15 -3
View File
@@ -271,6 +271,18 @@ item. A `CONDITION` observation's value is taken from its **element name**
condition element normalises onto the same `UNAVAILABLE` sentinel as every
other category so one comparison covers all three.
**An empty value is not `UNAVAILABLE` (#824, owner decision 2026-11-11 — Java parity).** A
genuinely empty observation (`<Program …></Program>`, `<Program …/>`, or whitespace-only text,
which the parser trims to `""`) on a **string-typed** item (`String`/`Reference`, or an unauthored
item, which coerces as String) is served **Good `""`** — an Agent can legitimately report a blank
`Program`, `Block` or `Message`. On any **non-string** item (numeric, `Boolean`, `DateTime`, a
`TIME_SERIES` array) `""` cannot be a value, so it stays **`BadNoCommunication`** with a null value
— deliberately not `BadTypeMismatch`, which would tell the operator the Agent sent text of the wrong
type and point them at the tag's authored data type; the Agent sent no value at all. Only the literal
`UNAVAILABLE` sentinel is the "no value" statement on a string item. Before #824 every empty value
was `BadNoCommunication`. A `DATA_SET`/`TABLE`-shaped observation is unaffected (its entry-less
form is still Good `{}`; non-`<Entry>` children are still `BadNotSupported`).
**Integer-valued decimals are served on integral tags (#804).** Agents format integer-valued items as
`42.000000`; an `Int*`/`UInt*` tag accepts any decimal whose value *is* an integer in range (`42.000000` → `42`,
`1.5e3` → `1500`) losslessly. A real fraction stays `BadOutOfRange` — deliberately stricter than the Java
@@ -296,7 +308,7 @@ limitations 4; a probe that does declare one yields a fixed `ArrayDim`).
Parsing: the observation text is split on any whitespace, each token parsed
invariant-culture. The result is **all-or-nothing** — one non-numeric /
non-finite token is `BadTypeMismatch`, a literal `UNAVAILABLE` token or an
empty body is `BadNoCommunication`, and a count that disagrees with the
empty body is `BadNoCommunication` (an array is non-string, so #824's Good `""` never applies), and a count that disagrees with the
observation's own `sampleCount` attribute (or a fixed `ArrayDim`) is
`BadDecodingError`. A partial vector is never served as Good. Only a
**Float64** array tag is served; any other declared element type stays
@@ -677,7 +689,7 @@ deliberately different, kept.
| Type inference SAMPLE→Double, numeric EVENT→Int64, else String (`MTConnectTypes`) | covered, **.NET exceeds** (TIME_SERIES → `double[]`, separator-insensitive types) | `MTConnectDataTypeInference` · `MTConnectDataTypeInferenceTests` |
| UNAVAILABLE → `Bad_NoCommunication`, null value, source timestamp kept (`MTConnectValues`) | covered | `MTConnectObservationIndex.BuildSnapshot` · `Unavailable_observation_maps_to_BadNoCommunication_with_null_value`, `Unavailable_observation_keeps_the_agent_source_timestamp` |
| UNAVAILABLE matched case-insensitively | divergence (kept) | ordinal match on the parser-normalised token, so a free-text EVENT reading "Unavailable" stays a value · `Unavailable_sentinel_is_matched_on_the_exact_literal` |
| Blank text on a String node is a Good empty string | divergence (kept) | empty text is `BadNoCommunication` (an empty element is not an empty-string value) |
| Blank text on a String node is a Good empty string | **covered** (was a divergence until #824, owner decision 2026-11-11) | `BuildSnapshot` serves Good `""` on a string-typed or unauthored item; a non-string item stays `BadNoCommunication` (Java has no typed node to refuse `""` on); `UNAVAILABLE` unchanged · `Empty_observation_value_on_a_string_item_is_a_Good_empty_string`, `A_parsed_empty_event_on_a_string_item_is_a_Good_empty_string`, controls `Empty_observation_value_on_a_non_string_item_stays_BadNoCommunication`, `Unavailable_on_a_string_item_is_still_BadNoCommunication` |
| No fabricated numbers: narrow-integer range checks, float overflow ≠ Infinity, NaN/Infinity refused (`MTConnectValues.coerce`) | covered for integers; **gap → closed** for floats (`1e400`, `NaN`, `Infinity`, `1e39` on Float32 were served Good) | `Coerce`, `NonFiniteStatus` · `A_non_finite_float_is_never_served_as_good` |
| Integer-valued decimal (`42.000000`) on an integral node | **gap → closed** (was `BadOutOfRange`) — lossless only; a real fraction is still refused, never truncated (Java truncates) | `TryIntegral` · `An_integer_valued_decimal_converts_losslessly`, control `A_fraction_or_an_out_of_range_decimal_stays_out_of_range` |
| Source timestamps preserved; a non-Zulu offset normalised | covered | `ReadTimestampUtc` · `A_timestamp_with_a_non_zulu_offset_is_normalized_to_the_same_utc_instant` |
@@ -932,7 +944,7 @@ for anything beyond values-and-conditions.
- **Unit tests** — `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/`
(canned-XML fixtures, no network) — 714/714 at #804 (2026-11-10). `MTConnectJavaParityTests` holds the #804 gap tests and the ported Java regression cases (see [Java reference parity](#java-reference-parity-804)). Covers
discovery tree shape, `MTConnectDataTypeInference`, observation indexing,
`UNAVAILABLE` → `BadNoCommunication`, CONDITION state-word mapping,
`UNAVAILABLE` → `BadNoCommunication` (an empty string-typed value is Good `""`, #824), CONDITION state-word mapping,
multipart chunk framing, and ring-buffer re-baseline paging (both the
sequence-gap and the `OUT_OF_RANGE` legs). The write path adds
`MTConnectAgentClientWriteTests` (`9f17d7ef` — verb, URL, agent-error-in-200
+10 -2
View File
@@ -189,8 +189,16 @@ Consequences:
- Test Connect (`OpcUaClientDriverProbe`) only calls GetEndpoints on an unsecured discovery channel — no session, so no
certificate or host-name check.
**Policy: `certificateValidation.validateServerHostName`** (driver JSON; AdminUI driver form "Server host-name check";
the browser reads the same JSON):
**Decision (#837, owner, 2026-11-11): the check stays OPT-IN — unset means off, and it will not become a default.**
Turning it on by default would refuse every NAT / port-published / connect-by-IP endpoint whose certificate does not
list that exact host, and every link-local IPv6 endpoint, on upgrade; and because the match is loose (first label) it
would not deliver the identity guarantee a default implies. Operators enable it per driver where trust is CA-issued
and certificate names are fully-qualified and site-unique. The AdminUI shows the toggle in the driver form's
**Security** panel (beside mode/policy) as "Server certificate host-name check", with help text covering the NAT/IP
caveat, IPv6 zone ids and the loose first-label match.
**Policy: `certificateValidation.validateServerHostName`** (driver JSON; AdminUI driver form, Security panel,
"Server certificate host-name check"; the browser reads the same JSON):
| Value | Meaning |
|---|---|
+24 -17
View File
@@ -46,7 +46,7 @@ with a human-readable explanation rather than a false-green TCP-open tick.
| **S7** | `Plc.OpenAsync` (COTP CR/CC + S7 setup-communication), check `IsConnected`, then `Close`. Wrong rack/slot or a non-S7 server causes `OpenAsync` to throw → handshake fail. | `"S7 connected (CPU …)"` | `10.100.0.35:1102` (python-snap7 sim) |
| **AbCip** | `libplctag` Tag `InitializeAsync` (EIP session + CIP Forward Open). A CIP-level error such as tag-not-found still proves the controller answered CIP → `Ok`. A session/ForwardOpen/connect error → handshake fail. | `"CIP session OK"` | `10.100.0.35:44818` (CIP sim) |
| **AbLegacy** | Same `libplctag` `InitializeAsync` handshake as AbCip, PCCC protocol family. | `"CIP session OK"` (PCCC family) | Deferred — no PLC5/SLC sim |
| **TwinCAT** | `AdsClient.Connect` + `ReadStateAsync`. See [degrade semantics](#twincat-degrade) below. | `"ADS state: {state}"` | Deferred — no ADS target |
| **TwinCAT** | `AdsClient.Connect` + `ReadStateAsync` through the **local AMS router** — no TCP pre-check. See [TwinCAT: no TCP pre-check](#twincat-no-tcp-pre-check) below. | `"ADS state: {state}"` | Z35474 (production TC3 press, read-only, via an in-process TcpRouter) |
| **FOCAS** | `cnc_allclibhndl3` via a direct `DllImport("fwlib32")` in the probe. See [degrade semantics](#focas-degrade) below. | `"FOCAS handle OK"` | Deferred — no CNC + FWLIB |
| **Galaxy** | gRPC unary call to `GalaxyRepository.TestConnection` on the configured mxaccessgw endpoint. See [auth-rejection rule](#galaxy-auth-rejection) below. | `"gateway gRPC OK"` | `http://10.100.0.48:5120` (mxaccessgw) |
| **Heidenhain** | LSV2 `A_LG INSPECT` (password resolved from its `secret:`/`env:` reference) + `R_VR`, through the same read-only allowlist as the driver; a refusal or malformed version record → fail. No logout — the socket close ends the session. | `"LSV2 INSPECT session OK at host:port: <model> NC <version>"` | Deferred — no LSV2 control; simulator-proven |
@@ -61,29 +61,36 @@ backend is now the external HistorianGateway (a gRPC client package, not a probe
## Degrade semantics
Three drivers have environmental constraints that can prevent the handshake
Some drivers have environmental constraints that can prevent the handshake
from running on certain hosts. The **degradation principle** is: the probe
must never produce a result *worse* than today's TCP-only probe. A genuine
protocol rejection from a reachable device is a correct RED; an inability to
*run* the handshake at all (no FWLIB, no managed router) degrades to the
existing TCP-reachability message — still a green tick but annotated.
*run* the handshake at all (no FWLIB) degrades to the
existing TCP-reachability message — still a green tick but annotated. TwinCAT
no longer degrades (see below).
### TwinCAT degrade
### TwinCAT: no TCP pre-check
Where the handshake is available:
TwinCAT is the exception to the "TCP connect fails" row above: it opens **no socket of its own**
(#863). The driver has no IP field — an AMS Net ID is a Beckhoff identifier the AMS router maps to a
host through its route table, and the AMS port (801/851) is not a TCP port. The probe therefore takes
exactly the driver's path (`AdsTwinCATClient.ConnectAsync`): `AdsClient.Connect(netId, port)` through
the local AMS router (TwinCAT XAR, or `Beckhoff.TwinCAT.Ads.TcpRouter` hosted in-process), then one
read-only `ReadStateAsync`, all inside the probe budget. Until #863 a first-phase check dialled the Net
ID's first four octets as an IPv4 address on the AMS port — for Z35474 (`5.118.56.93.1.1`) an
unrelated public host — so Test Connect timed out against every real runtime.
- `AdsClient.Connect(netId, port)` + `ReadStateAsync` → `Ok=true`,
`"ADS state: {state}"` (Run / Config / Stop).
- An ADS **route-table rejection** from a reachable ADS router is a **true RED**:
`"Reachable at {host}:{port} but ADS handshake failed: {detail} — check the
target's ADS route table authorizes this host"`. This is the correct result:
the driver would also be unable to function without an authorized route.
| Outcome | `Ok` | Message |
|---------|------|---------|
| ReadState succeeds | `true` | `"ADS state: {state}"` (Run / Config / Stop) |
| No local AMS router (Beckhoff "Check for a running TwinCAT router instance!", `ClientPortNotOpen`, router not active, ADS client cannot load) | `false` | `"No local AMS router on this host — start TwinCAT XAR or host the Beckhoff TcpRouter; the target was not contacted (…)"` |
| Router answered but cannot reach the target (`TargetMachineNotFound`, `TargetPortNotFound`, port/access refused, host unreachable) | `false` | `"No route to {netId}:{port} or target unreachable: {code} — check the local AMS router has a route …"` |
| Sync/device timeout code, or the budget expired | `false` | `"Probe timed out after {n}s waiting for ADS state from {netId}:{port} …"` |
| Any other ADS error | `false` | `"ADS handshake with {netId}:{port} failed: {code}"` |
Where the handshake is unavailable (headless server, no TwinCAT runtime, the
managed AMS router cannot start):
- Probe degrades to TCP-reachability: `Ok=true`,
`"(ADS handshake unavailable on this host — TCP reachability only)"`.
There is **no degrade-to-green** any more: with no TCP check, a host that cannot run the handshake
has contacted nothing, so it reports the missing router rather than a green tick. A stock admin node
has no AMS router, so TwinCAT Test Connect is red there by design — the driver needs the same router.
### FOCAS degrade
@@ -536,7 +536,7 @@ size for 2× headroom (10k/s). The passthrough fast path is excluded because it
| Host CPU for IPC | ≤ 25% of one core at 10k evals/s | Serialisation + pipe I/O, measured by an opt-in perf test. Since lmxopcua#789 the harness reports it as the out-of-process runner's host CPU minus the in-process runner's in the same phase ("vs InProcess"): the raw figure includes the harness's own pacing loop and pool, which InProcess pays too (§12.2) |
| Bytes per eval on the wire | ≤ 64 B + 24 B per input | Positional slots, §4.4 |
| Shard restart → first result, 1,000 cached scripts | ≤ 2 s | §7 lazy PE load |
| Worker idle RSS | ≤ 80 MB + scripts **[A-3]** | Per shard. **Proposed revision (lmxopcua#789, §12.2):** ≤ 80 MB + scripts for a worker that has only loaded cached PEs (measured 36–58 MB); ≤ 200 MB + scripts once Roslyn has run in it (measured 159–187 MB), because compiling in the worker (§14 #7) keeps Roslyn's code and metadata resident and no GC setting moves it |
| Worker idle RSS | ≤ 200 MB + scripts | Per shard. **Revised by the owner 2026-11-11 (lmxopcua#789)** from the original ≤ 80 MB + scripts **[A-3]**. Compiling in the worker (§14 #7) keeps Roslyn's code and metadata resident once it has run (measured 159–187 MB), and no GC setting moves it (§12.2). A worker that has only loaded cached PEs sits far below that (measured 36–58 MB). The perf harness reports both against 200 MB |
| Kill latency after `T+G` | ≤ 100 ms | Host timer resolution plus `Process.Kill` |
**Comparison with today:** the in-process cost per compiled run is one `Task.Run` hop plus `WaitAsync`, which
@@ -884,6 +884,58 @@ Both misses → lmxopcua#789 (the §4.5 batching and per-eval allocation work).
The host load timeout still counts the time a load waits behind other compiles → lmxopcua#790.
**windev re-measure, 2026-11-11 (lmxopcua#789, after the §12.2–§12.3 changes; budget revision applied).**
- **windev is a VM:** VMware20,1 on an Intel Xeon E5-2697 v4 @ 2.30 GHz, 36 logical CPUs, 32 GB RAM, Windows,
.NET 10.0.8. The box was otherwise idle (about 4 % total CPU before the run). The issue's "re-measure on a non-VM
box" request is therefore still open; these are VM figures.
- Run from `C:\build\otopcua-842` (a `git archive` of the branch head) with `-c Release`, `OTOPCUA_SCRIPT_PERF=1`,
10 s phases, filtered to `Throughput_and_latency_against_the_accepted_gate` and `Restart_rss_and_kill_latency`.
The restart test ran twice; both figures are given.
- Framework-dependent worker built beside the tests (`dotnet exec`), as in the earlier windev rows.
| Measure (K = 2 unless stated) | windev 2026-11-11 | §8.2 budget | Verdict |
|---|---:|---|---|
| Added latency p50, idle sequential | 581 µs (K=1 490, K=4 537; InProcess 134) | ≤ 150 µs | MISS |
| p99 at paced 10k/s | 3,220 µs (K=1 5,380, K=4 1,358) | ≤ 2 ms | MISS at K=2 (was 1,676 µs in §12.2) |
| Host CPU at 10k/s, vs InProcess | +131 % (raw 141 %, InProcess 10 %) | ≤ 25 % | MISS (not tuned in this task) |
| Closed loop ×64 | 54,152 evals/s, 0 not ok | gate ≥ 5,000 | PASS |
| Kill latency after T+G (5 runs × 2) | 16 / 11 / 10 / 11 / 12 and 16 / 12 / 8 / 11 / 12 ms | ≤ 100 ms | PASS |
| Kill both shards → every script's next correct result (1,000 cached) | 2,135 ms and 2,117 ms | ≤ 2,000 ms | MISS by about 6 % |
| Warm restart, 1,000 cached, fresh runner | 1,847 ms and 1,829 ms | ≤ 2,000 ms | PASS |
| Worker RSS after first eval, compiled (max of 2 shards) | 160 MB (both runs) | ≤ 200 MB + scripts | PASS |
| Worker RSS after first eval, warm PE cache, no compile | 36 MB (both runs) | ≤ 200 MB + scripts | PASS |
| Worker RSS, 500 scripts compiled per shard | 236 / 262 MB | — (200 MB + scripts) | — |
| Worker RSS, 500 scripts loaded from the cache per shard | 93 / 96 MB | — | — |
| 1,000 scripts, empty cache, all first results | 12,380 / 12,800 ms | — | — |
Notes:
- Under the revised budget, RSS passes. Kill latency stays fixed since §12.2: the earlier 210 ms was compile time.
- Kill-both-shards is about 120 ms over the line. It includes the 250 ms first restart backoff and the warm-cache
load of 1,000 PEs. The warm-cache restart alone takes about 1.83 s on this VM, so the kill path adds only about
300 ms on top.
- Idle p50 and host CPU are the overlapped-I/O path that §12.2 left open; CPU tuning is deliberately not part of
this re-measure. The K=2 p99 at 10k/s reads worse than §12.2's 1,676 µs. K=4 is 1,358 µs in the same run, so this
is likely VM scheduling noise on a single run, but it was not re-run to confirm.
**Windows `ScriptWorker.Tests` run, 2026-11-11 (lmxopcua#842).** `--filter Category=ScriptWorker`, Debug, on the
same VM. This is the first Windows run of the #827 soft-clock tests, which freeze the worker with `NtSuspendProcess`
/ `NtResumeProcess`.
- From `C:\build\otopcua-842`: **107 passed, 0 failed, 2 skipped (109)**, 13 m 37 s, 0 build warnings.
- All 13 `WorkerSoftClockTests` cases passed, including the 5 that suspend a real worker process: stalled > T,
stalled past T+G, late-starting runaway, Shadow under a stall (two freezes), and a notice that arrives after the
transport deadline. No code change was needed.
- Skipped by design: `A_published_self_contained_worker_runs_as_its_own_process` (needs
`OTOPCUA_SCRIPTWORKER_TEST_PATH`, §12.1) and the off-Windows identity refusal.
- From `C:\Users\dohertj2\otopcua-842`: 106 / **1 failed** / 2. The failure is
`WorkerIsolationTests.A_worker_under_the_identity_runs_scripts_with_a_restricted_token_and_a_plain_one_does_not`:
"The application to execute does not exist: …\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.dll", exit
`0x80008081`.
- **Cause: the test environment, not the code.** The restricted worker token cannot read the worker dll inside a
user profile, whose ACL is SYSTEM, Administrators and the owner only. That is the access §6 intends.
- In production the installer grants the identity read+execute on `script-worker\` (§12.5).
- Run the Windows suite from a directory the `BUILTIN\Users` group can read, such as `C:\build`. The §12.5 green
run used that.
### 12.2 lmxopcua#789 / #790 / #788 record (2026-11-10)
**What cost the host CPU (#789).** With `DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0` a scratch probe showed the
@@ -1367,7 +1419,7 @@ Each has a recommendation, marked ★.
|---|---|---|
| A-1 | Unbounded recursion in a script throws `StackOverflowException` and kills the host today | Reproduce in a throwaway process (not the test host) |
| A-2 | A collectible ALC whose code is still executing on an orphan thread cannot be unloaded | .NET docs / a `WeakReference` test with a spinning orphan |
| A-3 | A minimal .NET 10 worker idles at tens of MB RSS | Measure the P1 worker |
| A-3 | A minimal .NET 10 worker idles at tens of MB RSS | Measured (§12.1/§12.2): true only for a worker that has never compiled (36–58 MB); a worker that has run Roslyn idles at 159–187 MB, so §8.2 was revised to 200 MB |
| A-4 | `P = max(2, cores/K)` evaluation threads per worker is enough | P1 perf harness |
| A-5 | Passthrough scripts are a large share of evaluations on real deployments | Count `PassthroughScript.TryMatch` hits against compiled runs on QA (add a counter in P0) |
| A-6 | A local pipe round trip costs tens of µs | P1 perf harness |
@@ -0,0 +1,102 @@
# Folder (notifier) event items: per-item delivery (Gitea #855)
Status: design for #855 (owner decision 2026-11-11: build per-item delivery). Background: #838 (deferral instead of
blocking under `Lock`), #854 (sequence cut on Server-object refresh), #826 (per-folder event ACL).
## Problem
A condition event reaches an event item on an equipment / raw device folder through the SDK's `MonitoredNode2`
for that folder: `OnReportEventAsync` writes it to a bounded channel (4096), and a per-node consumer task decides
each item's verdict (memoised per `(item, event type, SourceNode)`) and calls `QueueEvent`. That gives the node
manager no per-event, per-item decision point, so a `ConditionRefresh` of a folder item cannot withhold an older
event that is still in the channel or in our deferral queue: the refresh is overtaken (the #854 guarantee covers only
Server-object items). The channel's consumer also needs `Lock` on a verdict miss, which is why a burst under `Lock`
fills the channel and defers (#838).
## Design
### Where delivery hooks in
`DeliverConditionEventAsync` already walks the condition's notifier chain hop by hop (our copy of the SDK's
`ReportEventAsync`, #838). At a hop whose `OnReportEventAsync` is the SDK `MonitoredNode2` of **a node this manager
owns** (`handler.Target is MonitoredNode2 m && m.Node == node`), the walk no longer calls the handler. It delivers the
event itself to that node's attached items (`m.EventMonitoredItems`), one by one: the SDK's audit guard, the verdict
(`INodeManager3.ValidateEventRolePermissions` — the SDK's type/source `ReceiveEvents` check plus, under enforcement,
the #826 live-notifier + grant check), then the item's gate and cut, then `QueueEvent`.
This is exactly how Server-object items are served (`OnConditionReportEvent`, #586/#854): same verdict per event, same
gate, same cut, same `QueueEvent`.
### Bypass / coexistence with `MonitoredNode2`
- **Condition events bypass the channel** for our notifiers. The `MonitoredNode2` stays: the SDK still owns item
attachment (subscribe / unsubscribe / transfer / tombstone / revalidation stand-down detach), and its channel still
carries everything else that reaches the node — a condition's SDK-raised method **audit** events (only with auditing
on) and data-change notifications for attribute items on the folder. Those keep the SDK's cached verdict, so
`InvalidateFolderEventVerdicts` stays.
- A stood-down item is not in `EventMonitoredItems`, so it receives nothing, as before.
- A hop whose handler is anything else (none exists today: conditions are never event notifiers and variables cannot
host event items) still goes through the handler — the fallback that keeps the #838 deferral reachable.
### Consequence for overload
Every hop the node manager owns now completes synchronously: `QueueEvent` never blocks (the item's own queue discards
oldest at `QueueSize`), and the verdict runs on the reporter's thread, which already holds `Lock`. So a burst under one
`Lock` hold no longer defers at all, and the #838 deadlock shape (a consumer needing `Lock` while the reporter waits on
its channel) is gone for condition events. Folder items stay in report order per item, exactly like Server-object items.
The #838 deferral queue and the #854 cut machinery remain as the safety net for a hop that does suspend (the fallback
above, or a future SDK change). Folder items are brought under the same cut:
- An event whose delivery continues off `Lock` — queued in the deferral queue, **or** the in-flight event whose walk
suspended at a hop (new: marked when the walk suspends, so the hops after the suspension take the gate) — takes the
item's gate and is withheld when the item's latest refresh cut supersedes it (`RefreshCut.Supersedes`, unchanged).
- `ConditionRefresh` / `ConditionRefresh2` of folder items has the same two paths as the Server object, flag on and off:
- **Nothing deferred:** the SDK base walk of the item's notifier (off `Lock`, live condition nodes), per-item verdict
as before (#826 filter under enforcement). Kept only if no deferral episode existed at any moment of it; otherwise
the cut path runs too and supersedes it inside the same RefreshStart..RefreshEnd. Its closing check takes `Lock`
once: a report whose walk suspended (marked under `Lock` before suspending) has registered its deferral by then —
without it a folder refresh could pass the check in the instant between the suspension and the count.
- **Deferring — the cut:** verdicts per (item, condition) first, off `Lock`, from a live walk. Then one `Lock` hold:
watermark, episode, the Retain set, and per item the same walk producing frozen snapshots, with the cut + "replay
pending" marker installed under the item's gate. The replay is queued off `Lock`, one event per gate hold.
- State per item (`EventItemState`, previously Server-object only) is keyed by the item object for its lifetime, so it
survives unsubscribe + resubscribe; stale cuts are cleared when the deferral episode ends, for folder and Server items.
### Locking
Unchanged order, now for folder items too: `Lock` → item gate → the SDK item's own lock. Synchronous delivery runs
under the reporter's `Lock`; the verdict takes `Lock` re-entrantly and the core node manager's / authorizer's locks
(as the Server-object path already does). Off-`Lock` (deferred) delivery computes the verdict **before** taking the item
gate, so nothing under a gate takes `Lock` or the event-report gate. A reporter waits at most for one in-progress
`QueueEvent` on an item; a refresh waits at most for one deferred delivery already inside the gate, never for a drain.
No reporter ever waits on a channel.
### Bounds
No new queue. Per item: the SDK item queue (`QueueSize`, DiscardOldest). Deferral (only on the fallback): unchanged,
1024 per condition / 65 536 total, coalescing. Cuts: one per refreshed item, sets sized by the conditions, cleared at the
end of the deferral episode.
### ACL (#826)
Each event is decided per item at delivery against the current generation — no memoised verdict for condition events
any more, so a deploy / revocation / reference removal takes effect on the next event without waiting for an
invalidation. Refresh replays only conditions the item may see (the same check); a condition omitted for want of a grant
is never withheld afterwards (the cut's existing rule).
### Cost
The per-event verdict for folder items moves from the SDK's consumer task (cached) to the reporter thread under `Lock`
(uncached), as Server-object items already pay. Folder subscribers are few (ScadaBridge subscribes at `i=2253`).
## Tests
- `ConditionRefreshDeferralWireTests` / `FolderEventFloodWireTests` / `EventQueueOverflowAclWireTests`: deferral is no
longer produced by flooding a folder item; tests that need it stall the folder's hop through a test probe
(`NotifierDeliveryStall`, null in production) instead of the folder item's SDK lock (which would now block the reporter,
as it would on the Server object). Flood tests now pin that folder delivery never defers.
- New `FolderConditionRefreshWireTests`: folder-item refresh during deferral (flag on/off, retained / cleared,
ConditionRefresh2, resubscribe), the gate race, newer-than-cut replay, ACL omission not withheld, ACL revocation during
deferral, the in-flight suspended event, and two stress tests (production overload with no stall; stall + refresh +
reports concurrently).
+46 -22
View File
@@ -901,7 +901,7 @@ Behaviours worth knowing before enabling it:
older `Retain=false` event of such a condition still arrives (it agrees with the omission), and a condition that was
retained but omitted for the item's **grants** is never withheld (a grant added before the drain lets its events
through; the ACL decides them at delivery as ever).
- The deferral queues, every other Server-object item and every folder item are untouched. A refresh waits for the
- The deferral queues, every other Server-object item and every non-refreshing folder item are untouched. A refresh waits for the
drain at most for one in-progress `QueueEvent` on that item (the gate), never for the drain itself; a reporter
holding `Lock` likewise waits at most for one. Each item keeps one gate/cut state for its lifetime (keyed by the
item, so an unsubscribe + resubscribe keeps it), and cuts are cleared when the deferral count returns to zero.
@@ -912,29 +912,53 @@ Behaviours worth knowing before enabling it:
- **Measured** (3000 retained conditions, longest `Lock` wait another thread saw during one refresh): the first
#854 cut (`1622d07f`, everything under `Lock`) 95–106 ms idle / 102–127 ms deferring; now 0–3 ms idle /
23–32 ms deferring (the frozen snapshots).
- ⚠️ **Residual: folder (notifier) items.** A refresh of an item on an equipment/device folder can still be followed
by older events of a condition. Two sources, neither reachable per item: the SDK's own `MonitoredNode2` channel
(up to 4096 events already accepted before the refresh — this exists in the stock SDK too, and in normal operation
for the few events in transit), and our deferral queue, whose folder hop goes through that channel, where the SDK
memoises the per-item permission verdict by (item, event type, source) so there is no per-event, per-item hook. The
events stay in the condition's order and its newest arrives last, so the client converges. Discarding the queue
globally on refresh would drop transitions for every non-refreshing subscriber while leaving the channel backlog;
making refresh wait would block a service call on the stalled drain. ScadaBridge also ignores a transition older
than the one it holds for a condition.
- **Folder (notifier) items are delivered per item (Gitea #855).** Before #855 a folder item was fed by the SDK's own
`MonitoredNode2` channel (up to 4096 events accepted before a refresh), with the per-item verdict memoised by (item,
event type, source), so a refresh of a folder item could still be followed by older events. Now, at every notifier
the node manager owns, the walk serves the attached items itself (`DeliverToNotifierItems`), exactly as it serves
Server-object items: the SDK's audit guard, the verdict through `INodeManager3.ValidateEventRolePermissions` (SDK
type/source `ReceiveEvents` + the #826 decision under enforcement — its item-independent part, the live condition,
the notifier check and the visibility scopes, resolved once per hop) **per event, uncached**, then the item's gate and
cut, then `QueueEvent`. The `MonitoredNode2` stays for item attachment and for what still uses its channel (a
condition's SDK method audit events, data-change items on the folder), so `InvalidateFolderEventVerdicts` stays.
- **No notifier of ours defers any more.** `QueueEvent` never waits for a consumer (the item queue discards oldest at
`QueueSize`) and the verdict runs on the reporter's thread, which already holds `Lock` — so a burst under one
`Lock` hold completes with nothing deferred, and the #838 deadlock shape is gone for condition events. Folder items
therefore see condition events in report order. The cost moves with it: the per-event verdict and filter for
folder items now run under `Lock` (as for Server-object items), where the SDK used to cache the verdict off it —
including the SDK item queue's own per-event costs (its duplicate scan, and at a full queue a list shift), so many
folder items on one notifier whose clients stop publishing make each report proportionally slower under `Lock`.
With auditing on (off in this repo), a condition's SDK method **audit** event still takes the folder's channel, so
at a folder item it may now arrive after the condition event that method produced (they were one FIFO before).
- **The deferral queue and the cut stay as the safety net**, for a hop that is not ours (none exists today:
conditions are never event notifiers and variables cannot host event items) — and folder items share the cut:
a `ConditionRefresh`/`ConditionRefresh2` of a folder item takes the same two paths as the Server object (plain walk
off `Lock` when nothing is deferred; otherwise verdicts off `Lock`, then one `Lock` hold for the watermark, the
Retain set and, per item, its notifier walk frozen as snapshots, with the cut installed under the item gate and the
replay queued off `Lock`). An event whose walk suspended at a hop is marked deferred **before** it suspends, so
every later hop judges it against the cut; the plain refresh path's closing check takes `Lock` once, so a report
that suspended under it has registered its deferral by then. Lock order is unchanged (`Lock` → item gate → item
lock); off `Lock` the verdict is computed before the gate.
Pinned by `FolderEventFloodWireTests`: 6000 events under one `Lock` hold with publishing disabled (flag on and off)
complete, drain to zero, coalesce, and the item delivers at most its revised `QueueSize` in increasing order ending
with the last event; and a condition **removed** in the same `Lock` hold has none of its deferred events delivered to
either the folder or the Server-object item (the flag-off variant fails without the liveness checks).
`ConditionRefreshDeferralWireTests` (#854) stalls a folder item's SDK lock so its channel fills without `Lock` held,
refreshes the Server-object item (ConditionRefresh and ConditionRefresh2, flag on/off, condition retained or cleared)
and pins that nothing older than the replay follows RefreshEnd while the non-refreshing folder item still receives the
whole tail (also across an unsubscribe + resubscribe of the item, and the cut is cleared after the drain); a
probe-driven test pins that a refresh waits for a delivery already past its check (fails without the item gate); an
ACL-on test pins that a condition omitted for want of a grant is not withheld once the grant lands; a 3000-condition
test measures the `Lock` hold and pins that neither path queues its replay under `Lock`; and a stress test runs
reports, refreshes and an intermittent stall concurrently (its stream never goes backwards outside a refresh, and
every refresh ends on the newest state delivered).
complete **with nothing deferred** (fails if the per-item delivery stops recognising the SDK fan-out), and the item
delivers at most its revised `QueueSize` in increasing order ending with the last event; and a condition **removed**
while its events wait (delivery stalled at the folder) has none of them delivered to either the folder or the
Server-object item. Tests that need a deferral stall delivery at a folder through a test probe
(`NotifierStall` → `OtOpcUaNodeManager.NotifierDeliveryStall`, null in production), which suspends the walk exactly
as a full channel did — holding a folder item's SDK lock, as the pre-#855 tests did, would now block the reporter.
`ConditionRefreshDeferralWireTests` (#854) refreshes the Server-object item while the folder is stalled
(ConditionRefresh and ConditionRefresh2, flag on/off, condition retained or cleared) and pins that nothing older than
the replay follows RefreshEnd while the non-refreshing folder item still receives the whole tail (also across an
unsubscribe + resubscribe of the item, and the cut is cleared after the drain); a probe-driven test pins that a
refresh waits for a delivery already past its check; an ACL-on test pins that a condition omitted for want of a grant
is not withheld once the grant lands; a 3000-condition test measures the `Lock` hold; and a stress test runs reports,
refreshes and an intermittent stall concurrently. `FolderConditionRefreshWireTests` (#855) is the folder-item twin:
the same matrix on a refreshed folder item (a second subscriber on the same folder keeps the whole tail; a
ModifyMonitoredItems re-subscribe keeps the cut), the gate race, a newer-than-cut event on the raw device folder, a
condition omitted by the ACL (reference dropped) not withheld once it returns, a revocation after the refresh, a
production-shape overload (bursts under `Lock` + repeated refreshes, no stall: nothing deferred, the stream never
goes backwards, no replay queued under `Lock`) and a stress test with an intermittent stall.
**Model-change and audit events on the Server object are gated too** (Gitea #588). The node manager's
own server-wide events — `GeneralModelChangeEvent` (NodeAdded / NodeDeleted / DataTypeChanged) and the