Merge branch 'master' into worktree-agent-a720278a57ae41792
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
+20
-13
@@ -1827,10 +1827,13 @@ is `Good` (plus Calculated / Partial). Value = `100 × region time ÷ interval d
|
||||
**`DurationGood` / `DurationBad`** (#820) are the same Good / Bad region times as `PercentGood` / `PercentBad` — every rule above
|
||||
(regions, simple bounds, time before the first sample = Bad, end of data, `BadNoData`, status, Partial) applies unchanged — reported as
|
||||
an OPC UA `Duration`, i.e. a `double` in **milliseconds** (the SDK's unit): `DurationGood = PercentGood × interval ms ÷ 100`. Uncertain
|
||||
time counts toward neither (Mantis 0006166). Before #820 both were `BadAggregateNotSupported` on the SQL backends. ⚠️ A **live**
|
||||
`AggregateFilter` subscription computes DurationGood/PercentGood with the frozen 1.5.378.182 calculators, which count Uncertain time as
|
||||
Good — they disagree with HistoryRead whenever Uncertain data is present (#809, owner decision pending). The Gateway backend still
|
||||
refuses both (`BadAggregateNotSupported`, not advertised in i=11201).
|
||||
time counts toward neither (Mantis 0006166). Before #820 both were `BadAggregateNotSupported` on the SQL backends. ⚠️ **#809
|
||||
(client-visible, owner-approved 2026-11-11):** a **live** `AggregateFilter` subscription now computes DurationGood/PercentGood the same
|
||||
way — Good time only. They used to come from the frozen 1.5.378.182 calculators, which count Uncertain time as Good; the server now
|
||||
serves those two through the application's `GoodOnlyStatusAggregateCalculator` (every other live aggregate is still the frozen
|
||||
baseline). A live PercentGood/DurationGood over Uncertain data therefore drops by the Uncertain share. Pinned live-vs-history by
|
||||
`SdkAggregateParityTests.Live_AggregateFilter_PercentGood_and_DurationGood_equal_HistoryRead_over_Uncertain_samples`. The Gateway
|
||||
backend still refuses both (`BadAggregateNotSupported`, not advertised in i=11201).
|
||||
|
||||
All carry the Calculated bit. A non-Bad result also carries the **Partial** bit (`0x4`, `AggregateBits.Partial`) — never on a Bad
|
||||
result — for: the **truncated interval** (shorter than the processing interval — the latest one forward, the earliest one reverse;
|
||||
@@ -1903,15 +1906,19 @@ timestamp, and the same treatment as the Min/Max-at-interval-start case above; m
|
||||
And an interval holding a **Good String (or DateTime) sample** is `BadAggregateInvalidInputs` for Average/Minimum/Maximum/Range/
|
||||
*ActualTime here (the documented #684 refusal rule — a Good non-numeric value is not a number to aggregate), while the SDK skips the
|
||||
string and serves the numeric samples (e.g. Average 4 Good); an Uncertain non-numeric sample refuses nothing on either side.
|
||||
One more, found by the #810 Uncertain conformance data (kept deliberately): over the interval the data **starts inside** when the
|
||||
**first sample is Uncertain**, the SDK's `TimeAverage`/`Total` integrate the uncovered span before it as a ramp from 0 (QP: `Total`
|
||||
188.96 for 163.96 over the covered 15 s, and `TimeAverage` 9.448 = 188.96 ÷ 20 s for 10.931 = 163.96 ÷ 15 s); ours integrates only
|
||||
the covered part (Part 13, as with a Good first sample, where all agree). The same answer comes from the frozen calculators the server
|
||||
registers for **live** `AggregateFilter` items and from SDK 2.0.1's own stock calculators, in both read directions — pinned by
|
||||
`SdkAggregateParityTests.An_Uncertain_first_sample_in_a_late_start_interval_makes_the_SDK_integrate_the_time_before_the_data`
|
||||
(ours / frozen / stock 2.0.1, with a Good-first control) and `OpcUaServer.IntegrationTests/LiveAggregateLateStartFactsTests` (through
|
||||
the booted server's runtime `AggregateManager`), and on both SQL backends by `QualityMixConformance.Time_weighted_values_over_partial_data_cover_only_the_data`
|
||||
(forward) and `…_in_a_reverse_read` (#821). Whether live subscriptions should follow is the open #809/#821 owner decision. The same
|
||||
One more, found by the #810 Uncertain conformance data: over the interval the data **starts inside** when the **first sample is
|
||||
Uncertain**, the SDK's `TimeAverage`/`Total` integrate the uncovered span before it as a ramp from 0 (QP: `Total` 188.96 for 163.96
|
||||
over the covered 15 s, and `TimeAverage` 9.448 = 188.96 ÷ 20 s for 10.931 = 163.96 ÷ 15 s); ours integrates only the covered part
|
||||
(Part 13 "No Start Bound: Calculate Partial Interval", as with a Good first sample, where all agree). SDK 2.0.1's stock calculators
|
||||
answer the same as the frozen 1.5.378.182 ones, in both read directions. ⚠️ **Since #821 (owner-approved 2026-11-11, client-visible)
|
||||
the LIVE `AggregateFilter` calculators follow Part 13 here too** — `CoveredTimeAverageAggregateCalculator` overrides only
|
||||
TimeAverage/Total of the frozen copies (only the value of such an interval changes; status, timestamps and bits are the baseline's).
|
||||
Pinned by `SdkAggregateParityTests.An_Uncertain_first_sample_in_a_late_start_interval_makes_the_SDK_integrate_the_time_before_the_data`
|
||||
(ours / live / untouched baseline / stock 2.0.1, with a Good-first control), `…Live_AggregateFilter_TimeAverage_and_Total_equal_HistoryRead_over_an_Uncertain_first_late_start`
|
||||
(forward + reverse, no masks), `OpcUaServer.IntegrationTests/LiveAggregateLateStartFactsTests` (through the booted server's runtime
|
||||
`AggregateManager`), `Sdk2AggregateCompatibilityTests.Issue821_*`, and on both SQL backends by `QualityMixConformance` (QP's
|
||||
TimeAverage/Total against the live oracle, and `Time_weighted_values_over_partial_data_cover_only_the_data` forward / `…_in_a_reverse_read`).
|
||||
(#809, the PercentGood/DurationGood half, is likewise decided: live follows Part 13.) The same
|
||||
suite feeds PercentGood/PercentBad, DurationGood/DurationBad (and Count, Average, TimeAverage(2), Total(2), Start/End/Delta, Interpolative) Good/Uncertain/Bad
|
||||
mixes, all-Uncertain intervals (0 % Good **and** 0 % Bad, Mantis 0006166) and partial-data intervals, hand-computed and against the
|
||||
#796 oracle — the baseline calculator's Uncertain-as-Good PercentGood fails it.
|
||||
|
||||
@@ -676,6 +676,21 @@ deployment green fleet-wide. Now:
|
||||
no second rebuild), reporting clean only if that pass is clean.
|
||||
- **A node with a ConfigDb also writes the reason on its own row** (`ApplyAck.DegradedReasonPrefix`), because
|
||||
central only records it when it *accepts* the ack — a timed-out deployment's late ack is discarded.
|
||||
**That row stays `Applying` until the outcome is known** (lmxopcua#868): it used to be written `Applied`/clean
|
||||
the moment the revision advanced, and `ConfigPublishCoordinator`'s recovery counts any non-`Applying` row as
|
||||
an ack, so a coordinator restart inside the ~105 s outcome window sealed the deployment clean while the node
|
||||
was about to report degraded. `/fleet` therefore shows the node *Applying* until its outcome lands.
|
||||
A restart of the node inside that window depends on how it boots:
|
||||
- a `Direct` node that can read the ConfigDb finds the orphan `Applying` row and **replays** the apply, which
|
||||
acks the real outcome;
|
||||
- a node that boots **from its LocalDb cache** (every `FetchAndCache` driver-only node, or a `Direct` node
|
||||
whose ConfigDb is unreachable) restores the cached revision — the cache is written before the outcome — and
|
||||
sends **no ack**, so central times that deployment out even though the node runs it. The restored revision
|
||||
is marked *unconfirmed*, so **re-deploying it re-applies** and acks the real outcome rather than
|
||||
short-circuiting a clean `Applied`.
|
||||
- **Upgrade note (lmxopcua#868): do not deploy while the central pair is mid rolling-upgrade across this
|
||||
change.** An old-build fused central node still writes its own row `Applied` before the outcome, which a
|
||||
new-build coordinator's recovery would count as a clean ack. Finish upgrading both central nodes, then deploy.
|
||||
- **Mixed-version fleets.** An old node never sets the flag, so its acks are treated exactly as before
|
||||
(clean). A new node acking an old central: the old coordinator ignores the unknown field and seals as it
|
||||
always did — no worse than today.
|
||||
|
||||
@@ -271,6 +271,18 @@ item. A `CONDITION` observation's value is taken from its **element name**
|
||||
condition element normalises onto the same `UNAVAILABLE` sentinel as every
|
||||
other category so one comparison covers all three.
|
||||
|
||||
**An empty value is not `UNAVAILABLE` (#824, owner decision 2026-11-11 — Java parity).** A
|
||||
genuinely empty observation (`<Program …></Program>`, `<Program …/>`, or whitespace-only text,
|
||||
which the parser trims to `""`) on a **string-typed** item (`String`/`Reference`, or an unauthored
|
||||
item, which coerces as String) is served **Good `""`** — an Agent can legitimately report a blank
|
||||
`Program`, `Block` or `Message`. On any **non-string** item (numeric, `Boolean`, `DateTime`, a
|
||||
`TIME_SERIES` array) `""` cannot be a value, so it stays **`BadNoCommunication`** with a null value
|
||||
— deliberately not `BadTypeMismatch`, which would tell the operator the Agent sent text of the wrong
|
||||
type and point them at the tag's authored data type; the Agent sent no value at all. Only the literal
|
||||
`UNAVAILABLE` sentinel is the "no value" statement on a string item. Before #824 every empty value
|
||||
was `BadNoCommunication`. A `DATA_SET`/`TABLE`-shaped observation is unaffected (its entry-less
|
||||
form is still Good `{}`; non-`<Entry>` children are still `BadNotSupported`).
|
||||
|
||||
**Integer-valued decimals are served on integral tags (#804).** Agents format integer-valued items as
|
||||
`42.000000`; an `Int*`/`UInt*` tag accepts any decimal whose value *is* an integer in range (`42.000000` → `42`,
|
||||
`1.5e3` → `1500`) losslessly. A real fraction stays `BadOutOfRange` — deliberately stricter than the Java
|
||||
@@ -296,7 +308,7 @@ limitations 4; a probe that does declare one yields a fixed `ArrayDim`).
|
||||
Parsing: the observation text is split on any whitespace, each token parsed
|
||||
invariant-culture. The result is **all-or-nothing** — one non-numeric /
|
||||
non-finite token is `BadTypeMismatch`, a literal `UNAVAILABLE` token or an
|
||||
empty body is `BadNoCommunication`, and a count that disagrees with the
|
||||
empty body is `BadNoCommunication` (an array is non-string, so #824's Good `""` never applies), and a count that disagrees with the
|
||||
observation's own `sampleCount` attribute (or a fixed `ArrayDim`) is
|
||||
`BadDecodingError`. A partial vector is never served as Good. Only a
|
||||
**Float64** array tag is served; any other declared element type stays
|
||||
@@ -677,7 +689,7 @@ deliberately different, kept.
|
||||
| Type inference SAMPLE→Double, numeric EVENT→Int64, else String (`MTConnectTypes`) | covered, **.NET exceeds** (TIME_SERIES → `double[]`, separator-insensitive types) | `MTConnectDataTypeInference` · `MTConnectDataTypeInferenceTests` |
|
||||
| UNAVAILABLE → `Bad_NoCommunication`, null value, source timestamp kept (`MTConnectValues`) | covered | `MTConnectObservationIndex.BuildSnapshot` · `Unavailable_observation_maps_to_BadNoCommunication_with_null_value`, `Unavailable_observation_keeps_the_agent_source_timestamp` |
|
||||
| UNAVAILABLE matched case-insensitively | divergence (kept) | ordinal match on the parser-normalised token, so a free-text EVENT reading "Unavailable" stays a value · `Unavailable_sentinel_is_matched_on_the_exact_literal` |
|
||||
| Blank text on a String node is a Good empty string | divergence (kept) | empty text is `BadNoCommunication` (an empty element is not an empty-string value) |
|
||||
| Blank text on a String node is a Good empty string | **covered** (was a divergence until #824, owner decision 2026-11-11) | `BuildSnapshot` serves Good `""` on a string-typed or unauthored item; a non-string item stays `BadNoCommunication` (Java has no typed node to refuse `""` on); `UNAVAILABLE` unchanged · `Empty_observation_value_on_a_string_item_is_a_Good_empty_string`, `A_parsed_empty_event_on_a_string_item_is_a_Good_empty_string`, controls `Empty_observation_value_on_a_non_string_item_stays_BadNoCommunication`, `Unavailable_on_a_string_item_is_still_BadNoCommunication` |
|
||||
| No fabricated numbers: narrow-integer range checks, float overflow ≠ Infinity, NaN/Infinity refused (`MTConnectValues.coerce`) | covered for integers; **gap → closed** for floats (`1e400`, `NaN`, `Infinity`, `1e39` on Float32 were served Good) | `Coerce`, `NonFiniteStatus` · `A_non_finite_float_is_never_served_as_good` |
|
||||
| Integer-valued decimal (`42.000000`) on an integral node | **gap → closed** (was `BadOutOfRange`) — lossless only; a real fraction is still refused, never truncated (Java truncates) | `TryIntegral` · `An_integer_valued_decimal_converts_losslessly`, control `A_fraction_or_an_out_of_range_decimal_stays_out_of_range` |
|
||||
| Source timestamps preserved; a non-Zulu offset normalised | covered | `ReadTimestampUtc` · `A_timestamp_with_a_non_zulu_offset_is_normalized_to_the_same_utc_instant` |
|
||||
@@ -932,7 +944,7 @@ for anything beyond values-and-conditions.
|
||||
- **Unit tests** — `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/`
|
||||
(canned-XML fixtures, no network) — 714/714 at #804 (2026-11-10). `MTConnectJavaParityTests` holds the #804 gap tests and the ported Java regression cases (see [Java reference parity](#java-reference-parity-804)). Covers
|
||||
discovery tree shape, `MTConnectDataTypeInference`, observation indexing,
|
||||
`UNAVAILABLE` → `BadNoCommunication`, CONDITION state-word mapping,
|
||||
`UNAVAILABLE` → `BadNoCommunication` (an empty string-typed value is Good `""`, #824), CONDITION state-word mapping,
|
||||
multipart chunk framing, and ring-buffer re-baseline paging (both the
|
||||
sequence-gap and the `OUT_OF_RANGE` legs). The write path adds
|
||||
`MTConnectAgentClientWriteTests` (`9f17d7ef` — verb, URL, agent-error-in-200
|
||||
|
||||
@@ -189,8 +189,16 @@ Consequences:
|
||||
- Test Connect (`OpcUaClientDriverProbe`) only calls GetEndpoints on an unsecured discovery channel — no session, so no
|
||||
certificate or host-name check.
|
||||
|
||||
**Policy: `certificateValidation.validateServerHostName`** (driver JSON; AdminUI driver form "Server host-name check";
|
||||
the browser reads the same JSON):
|
||||
**Decision (#837, owner, 2026-11-11): the check stays OPT-IN — unset means off, and it will not become a default.**
|
||||
Turning it on by default would refuse every NAT / port-published / connect-by-IP endpoint whose certificate does not
|
||||
list that exact host, and every link-local IPv6 endpoint, on upgrade; and because the match is loose (first label) it
|
||||
would not deliver the identity guarantee a default implies. Operators enable it per driver where trust is CA-issued
|
||||
and certificate names are fully-qualified and site-unique. The AdminUI shows the toggle in the driver form's
|
||||
**Security** panel (beside mode/policy) as "Server certificate host-name check", with help text covering the NAT/IP
|
||||
caveat, IPv6 zone ids and the loose first-label match.
|
||||
|
||||
**Policy: `certificateValidation.validateServerHostName`** (driver JSON; AdminUI driver form, Security panel,
|
||||
"Server certificate host-name check"; the browser reads the same JSON):
|
||||
|
||||
| Value | Meaning |
|
||||
|---|---|
|
||||
|
||||
@@ -46,7 +46,7 @@ with a human-readable explanation rather than a false-green TCP-open tick.
|
||||
| **S7** | `Plc.OpenAsync` (COTP CR/CC + S7 setup-communication), check `IsConnected`, then `Close`. Wrong rack/slot or a non-S7 server causes `OpenAsync` to throw → handshake fail. | `"S7 connected (CPU …)"` | `10.100.0.35:1102` (python-snap7 sim) |
|
||||
| **AbCip** | `libplctag` Tag `InitializeAsync` (EIP session + CIP Forward Open). A CIP-level error such as tag-not-found still proves the controller answered CIP → `Ok`. A session/ForwardOpen/connect error → handshake fail. | `"CIP session OK"` | `10.100.0.35:44818` (CIP sim) |
|
||||
| **AbLegacy** | Same `libplctag` `InitializeAsync` handshake as AbCip, PCCC protocol family. | `"CIP session OK"` (PCCC family) | Deferred — no PLC5/SLC sim |
|
||||
| **TwinCAT** | `AdsClient.Connect` + `ReadStateAsync`. See [degrade semantics](#twincat-degrade) below. | `"ADS state: {state}"` | Deferred — no ADS target |
|
||||
| **TwinCAT** | `AdsClient.Connect` + `ReadStateAsync` through the **local AMS router** — no TCP pre-check. See [TwinCAT: no TCP pre-check](#twincat-no-tcp-pre-check) below. | `"ADS state: {state}"` | Z35474 (production TC3 press, read-only, via an in-process TcpRouter) |
|
||||
| **FOCAS** | `cnc_allclibhndl3` via a direct `DllImport("fwlib32")` in the probe. See [degrade semantics](#focas-degrade) below. | `"FOCAS handle OK"` | Deferred — no CNC + FWLIB |
|
||||
| **Galaxy** | gRPC unary call to `GalaxyRepository.TestConnection` on the configured mxaccessgw endpoint. See [auth-rejection rule](#galaxy-auth-rejection) below. | `"gateway gRPC OK"` | `http://10.100.0.48:5120` (mxaccessgw) |
|
||||
| **Heidenhain** | LSV2 `A_LG INSPECT` (password resolved from its `secret:`/`env:` reference) + `R_VR`, through the same read-only allowlist as the driver; a refusal or malformed version record → fail. No logout — the socket close ends the session. | `"LSV2 INSPECT session OK at host:port: <model> NC <version>"` | Deferred — no LSV2 control; simulator-proven |
|
||||
@@ -61,29 +61,36 @@ backend is now the external HistorianGateway (a gRPC client package, not a probe
|
||||
|
||||
## Degrade semantics
|
||||
|
||||
Three drivers have environmental constraints that can prevent the handshake
|
||||
Some drivers have environmental constraints that can prevent the handshake
|
||||
from running on certain hosts. The **degradation principle** is: the probe
|
||||
must never produce a result *worse* than today's TCP-only probe. A genuine
|
||||
protocol rejection from a reachable device is a correct RED; an inability to
|
||||
*run* the handshake at all (no FWLIB, no managed router) degrades to the
|
||||
existing TCP-reachability message — still a green tick but annotated.
|
||||
*run* the handshake at all (no FWLIB) degrades to the
|
||||
existing TCP-reachability message — still a green tick but annotated. TwinCAT
|
||||
no longer degrades (see below).
|
||||
|
||||
### TwinCAT degrade
|
||||
### TwinCAT: no TCP pre-check
|
||||
|
||||
Where the handshake is available:
|
||||
TwinCAT is the exception to the "TCP connect fails" row above: it opens **no socket of its own**
|
||||
(#863). The driver has no IP field — an AMS Net ID is a Beckhoff identifier the AMS router maps to a
|
||||
host through its route table, and the AMS port (801/851) is not a TCP port. The probe therefore takes
|
||||
exactly the driver's path (`AdsTwinCATClient.ConnectAsync`): `AdsClient.Connect(netId, port)` through
|
||||
the local AMS router (TwinCAT XAR, or `Beckhoff.TwinCAT.Ads.TcpRouter` hosted in-process), then one
|
||||
read-only `ReadStateAsync`, all inside the probe budget. Until #863 a first-phase check dialled the Net
|
||||
ID's first four octets as an IPv4 address on the AMS port — for Z35474 (`5.118.56.93.1.1`) an
|
||||
unrelated public host — so Test Connect timed out against every real runtime.
|
||||
|
||||
- `AdsClient.Connect(netId, port)` + `ReadStateAsync` → `Ok=true`,
|
||||
`"ADS state: {state}"` (Run / Config / Stop).
|
||||
- An ADS **route-table rejection** from a reachable ADS router is a **true RED**:
|
||||
`"Reachable at {host}:{port} but ADS handshake failed: {detail} — check the
|
||||
target's ADS route table authorizes this host"`. This is the correct result:
|
||||
the driver would also be unable to function without an authorized route.
|
||||
| Outcome | `Ok` | Message |
|
||||
|---------|------|---------|
|
||||
| ReadState succeeds | `true` | `"ADS state: {state}"` (Run / Config / Stop) |
|
||||
| No local AMS router (Beckhoff "Check for a running TwinCAT router instance!", `ClientPortNotOpen`, router not active, ADS client cannot load) | `false` | `"No local AMS router on this host — start TwinCAT XAR or host the Beckhoff TcpRouter; the target was not contacted (…)"` |
|
||||
| Router answered but cannot reach the target (`TargetMachineNotFound`, `TargetPortNotFound`, port/access refused, host unreachable) | `false` | `"No route to {netId}:{port} or target unreachable: {code} — check the local AMS router has a route …"` |
|
||||
| Sync/device timeout code, or the budget expired | `false` | `"Probe timed out after {n}s waiting for ADS state from {netId}:{port} …"` |
|
||||
| Any other ADS error | `false` | `"ADS handshake with {netId}:{port} failed: {code}"` |
|
||||
|
||||
Where the handshake is unavailable (headless server, no TwinCAT runtime, the
|
||||
managed AMS router cannot start):
|
||||
|
||||
- Probe degrades to TCP-reachability: `Ok=true`,
|
||||
`"(ADS handshake unavailable on this host — TCP reachability only)"`.
|
||||
There is **no degrade-to-green** any more: with no TCP check, a host that cannot run the handshake
|
||||
has contacted nothing, so it reports the missing router rather than a green tick. A stock admin node
|
||||
has no AMS router, so TwinCAT Test Connect is red there by design — the driver needs the same router.
|
||||
|
||||
### FOCAS degrade
|
||||
|
||||
|
||||
@@ -536,7 +536,7 @@ size for 2× headroom (10k/s). The passthrough fast path is excluded because it
|
||||
| Host CPU for IPC | ≤ 25% of one core at 10k evals/s | Serialisation + pipe I/O, measured by an opt-in perf test. Since lmxopcua#789 the harness reports it as the out-of-process runner's host CPU minus the in-process runner's in the same phase ("vs InProcess"): the raw figure includes the harness's own pacing loop and pool, which InProcess pays too (§12.2) |
|
||||
| Bytes per eval on the wire | ≤ 64 B + 24 B per input | Positional slots, §4.4 |
|
||||
| Shard restart → first result, 1,000 cached scripts | ≤ 2 s | §7 lazy PE load |
|
||||
| Worker idle RSS | ≤ 80 MB + scripts **[A-3]** | Per shard. **Proposed revision (lmxopcua#789, §12.2):** ≤ 80 MB + scripts for a worker that has only loaded cached PEs (measured 36–58 MB); ≤ 200 MB + scripts once Roslyn has run in it (measured 159–187 MB), because compiling in the worker (§14 #7) keeps Roslyn's code and metadata resident and no GC setting moves it |
|
||||
| Worker idle RSS | ≤ 200 MB + scripts | Per shard. **Revised by the owner 2026-11-11 (lmxopcua#789)** from the original ≤ 80 MB + scripts **[A-3]**. Compiling in the worker (§14 #7) keeps Roslyn's code and metadata resident once it has run (measured 159–187 MB), and no GC setting moves it (§12.2). A worker that has only loaded cached PEs sits far below that (measured 36–58 MB). The perf harness reports both against 200 MB |
|
||||
| Kill latency after `T+G` | ≤ 100 ms | Host timer resolution plus `Process.Kill` |
|
||||
|
||||
**Comparison with today:** the in-process cost per compiled run is one `Task.Run` hop plus `WaitAsync`, which
|
||||
@@ -884,6 +884,58 @@ Both misses → lmxopcua#789 (the §4.5 batching and per-eval allocation work).
|
||||
|
||||
The host load timeout still counts the time a load waits behind other compiles → lmxopcua#790.
|
||||
|
||||
**windev re-measure, 2026-11-11 (lmxopcua#789, after the §12.2–§12.3 changes; budget revision applied).**
|
||||
- **windev is a VM:** VMware20,1 on an Intel Xeon E5-2697 v4 @ 2.30 GHz, 36 logical CPUs, 32 GB RAM, Windows,
|
||||
.NET 10.0.8. The box was otherwise idle (about 4 % total CPU before the run). The issue's "re-measure on a non-VM
|
||||
box" request is therefore still open; these are VM figures.
|
||||
- Run from `C:\build\otopcua-842` (a `git archive` of the branch head) with `-c Release`, `OTOPCUA_SCRIPT_PERF=1`,
|
||||
10 s phases, filtered to `Throughput_and_latency_against_the_accepted_gate` and `Restart_rss_and_kill_latency`.
|
||||
The restart test ran twice; both figures are given.
|
||||
- Framework-dependent worker built beside the tests (`dotnet exec`), as in the earlier windev rows.
|
||||
|
||||
| Measure (K = 2 unless stated) | windev 2026-11-11 | §8.2 budget | Verdict |
|
||||
|---|---:|---|---|
|
||||
| Added latency p50, idle sequential | 581 µs (K=1 490, K=4 537; InProcess 134) | ≤ 150 µs | MISS |
|
||||
| p99 at paced 10k/s | 3,220 µs (K=1 5,380, K=4 1,358) | ≤ 2 ms | MISS at K=2 (was 1,676 µs in §12.2) |
|
||||
| Host CPU at 10k/s, vs InProcess | +131 % (raw 141 %, InProcess 10 %) | ≤ 25 % | MISS (not tuned in this task) |
|
||||
| Closed loop ×64 | 54,152 evals/s, 0 not ok | gate ≥ 5,000 | PASS |
|
||||
| Kill latency after T+G (5 runs × 2) | 16 / 11 / 10 / 11 / 12 and 16 / 12 / 8 / 11 / 12 ms | ≤ 100 ms | PASS |
|
||||
| Kill both shards → every script's next correct result (1,000 cached) | 2,135 ms and 2,117 ms | ≤ 2,000 ms | MISS by about 6 % |
|
||||
| Warm restart, 1,000 cached, fresh runner | 1,847 ms and 1,829 ms | ≤ 2,000 ms | PASS |
|
||||
| Worker RSS after first eval, compiled (max of 2 shards) | 160 MB (both runs) | ≤ 200 MB + scripts | PASS |
|
||||
| Worker RSS after first eval, warm PE cache, no compile | 36 MB (both runs) | ≤ 200 MB + scripts | PASS |
|
||||
| Worker RSS, 500 scripts compiled per shard | 236 / 262 MB | — (200 MB + scripts) | — |
|
||||
| Worker RSS, 500 scripts loaded from the cache per shard | 93 / 96 MB | — | — |
|
||||
| 1,000 scripts, empty cache, all first results | 12,380 / 12,800 ms | — | — |
|
||||
|
||||
Notes:
|
||||
- Under the revised budget, RSS passes. Kill latency stays fixed since §12.2: the earlier 210 ms was compile time.
|
||||
- Kill-both-shards is about 120 ms over the line. It includes the 250 ms first restart backoff and the warm-cache
|
||||
load of 1,000 PEs. The warm-cache restart alone takes about 1.83 s on this VM, so the kill path adds only about
|
||||
300 ms on top.
|
||||
- Idle p50 and host CPU are the overlapped-I/O path that §12.2 left open; CPU tuning is deliberately not part of
|
||||
this re-measure. The K=2 p99 at 10k/s reads worse than §12.2's 1,676 µs. K=4 is 1,358 µs in the same run, so this
|
||||
is likely VM scheduling noise on a single run, but it was not re-run to confirm.
|
||||
|
||||
**Windows `ScriptWorker.Tests` run, 2026-11-11 (lmxopcua#842).** `--filter Category=ScriptWorker`, Debug, on the
|
||||
same VM. This is the first Windows run of the #827 soft-clock tests, which freeze the worker with `NtSuspendProcess`
|
||||
/ `NtResumeProcess`.
|
||||
- From `C:\build\otopcua-842`: **107 passed, 0 failed, 2 skipped (109)**, 13 m 37 s, 0 build warnings.
|
||||
- All 13 `WorkerSoftClockTests` cases passed, including the 5 that suspend a real worker process: stalled > T,
|
||||
stalled past T+G, late-starting runaway, Shadow under a stall (two freezes), and a notice that arrives after the
|
||||
transport deadline. No code change was needed.
|
||||
- Skipped by design: `A_published_self_contained_worker_runs_as_its_own_process` (needs
|
||||
`OTOPCUA_SCRIPTWORKER_TEST_PATH`, §12.1) and the off-Windows identity refusal.
|
||||
- From `C:\Users\dohertj2\otopcua-842`: 106 / **1 failed** / 2. The failure is
|
||||
`WorkerIsolationTests.A_worker_under_the_identity_runs_scripts_with_a_restricted_token_and_a_plain_one_does_not`:
|
||||
"The application to execute does not exist: …\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.dll", exit
|
||||
`0x80008081`.
|
||||
- **Cause: the test environment, not the code.** The restricted worker token cannot read the worker dll inside a
|
||||
user profile, whose ACL is SYSTEM, Administrators and the owner only. That is the access §6 intends.
|
||||
- In production the installer grants the identity read+execute on `script-worker\` (§12.5).
|
||||
- Run the Windows suite from a directory the `BUILTIN\Users` group can read, such as `C:\build`. The §12.5 green
|
||||
run used that.
|
||||
|
||||
### 12.2 lmxopcua#789 / #790 / #788 record (2026-11-10)
|
||||
|
||||
**What cost the host CPU (#789).** With `DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0` a scratch probe showed the
|
||||
@@ -1367,7 +1419,7 @@ Each has a recommendation, marked ★.
|
||||
|---|---|---|
|
||||
| A-1 | Unbounded recursion in a script throws `StackOverflowException` and kills the host today | Reproduce in a throwaway process (not the test host) |
|
||||
| A-2 | A collectible ALC whose code is still executing on an orphan thread cannot be unloaded | .NET docs / a `WeakReference` test with a spinning orphan |
|
||||
| A-3 | A minimal .NET 10 worker idles at tens of MB RSS | Measure the P1 worker |
|
||||
| A-3 | A minimal .NET 10 worker idles at tens of MB RSS | Measured (§12.1/§12.2): true only for a worker that has never compiled (36–58 MB); a worker that has run Roslyn idles at 159–187 MB, so §8.2 was revised to 200 MB |
|
||||
| A-4 | `P = max(2, cores/K)` evaluation threads per worker is enough | P1 perf harness |
|
||||
| A-5 | Passthrough scripts are a large share of evaluations on real deployments | Count `PassthroughScript.TryMatch` hits against compiled runs on QA (add a counter in P0) |
|
||||
| A-6 | A local pipe round trip costs tens of µs | P1 perf harness |
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
# Folder (notifier) event items: per-item delivery (Gitea #855)
|
||||
|
||||
Status: design for #855 (owner decision 2026-11-11: build per-item delivery). Background: #838 (deferral instead of
|
||||
blocking under `Lock`), #854 (sequence cut on Server-object refresh), #826 (per-folder event ACL).
|
||||
|
||||
## Problem
|
||||
|
||||
A condition event reaches an event item on an equipment / raw device folder through the SDK's `MonitoredNode2`
|
||||
for that folder: `OnReportEventAsync` writes it to a bounded channel (4096), and a per-node consumer task decides
|
||||
each item's verdict (memoised per `(item, event type, SourceNode)`) and calls `QueueEvent`. That gives the node
|
||||
manager no per-event, per-item decision point, so a `ConditionRefresh` of a folder item cannot withhold an older
|
||||
event that is still in the channel or in our deferral queue: the refresh is overtaken (the #854 guarantee covers only
|
||||
Server-object items). The channel's consumer also needs `Lock` on a verdict miss, which is why a burst under `Lock`
|
||||
fills the channel and defers (#838).
|
||||
|
||||
## Design
|
||||
|
||||
### Where delivery hooks in
|
||||
|
||||
`DeliverConditionEventAsync` already walks the condition's notifier chain hop by hop (our copy of the SDK's
|
||||
`ReportEventAsync`, #838). At a hop whose `OnReportEventAsync` is the SDK `MonitoredNode2` of **a node this manager
|
||||
owns** (`handler.Target is MonitoredNode2 m && m.Node == node`), the walk no longer calls the handler. It delivers the
|
||||
event itself to that node's attached items (`m.EventMonitoredItems`), one by one: the SDK's audit guard, the verdict
|
||||
(`INodeManager3.ValidateEventRolePermissions` — the SDK's type/source `ReceiveEvents` check plus, under enforcement,
|
||||
the #826 live-notifier + grant check), then the item's gate and cut, then `QueueEvent`.
|
||||
|
||||
This is exactly how Server-object items are served (`OnConditionReportEvent`, #586/#854): same verdict per event, same
|
||||
gate, same cut, same `QueueEvent`.
|
||||
|
||||
### Bypass / coexistence with `MonitoredNode2`
|
||||
|
||||
- **Condition events bypass the channel** for our notifiers. The `MonitoredNode2` stays: the SDK still owns item
|
||||
attachment (subscribe / unsubscribe / transfer / tombstone / revalidation stand-down detach), and its channel still
|
||||
carries everything else that reaches the node — a condition's SDK-raised method **audit** events (only with auditing
|
||||
on) and data-change notifications for attribute items on the folder. Those keep the SDK's cached verdict, so
|
||||
`InvalidateFolderEventVerdicts` stays.
|
||||
- A stood-down item is not in `EventMonitoredItems`, so it receives nothing, as before.
|
||||
- A hop whose handler is anything else (none exists today: conditions are never event notifiers and variables cannot
|
||||
host event items) still goes through the handler — the fallback that keeps the #838 deferral reachable.
|
||||
|
||||
### Consequence for overload
|
||||
|
||||
Every hop the node manager owns now completes synchronously: `QueueEvent` never blocks (the item's own queue discards
|
||||
oldest at `QueueSize`), and the verdict runs on the reporter's thread, which already holds `Lock`. So a burst under one
|
||||
`Lock` hold no longer defers at all, and the #838 deadlock shape (a consumer needing `Lock` while the reporter waits on
|
||||
its channel) is gone for condition events. Folder items stay in report order per item, exactly like Server-object items.
|
||||
|
||||
The #838 deferral queue and the #854 cut machinery remain as the safety net for a hop that does suspend (the fallback
|
||||
above, or a future SDK change). Folder items are brought under the same cut:
|
||||
|
||||
- An event whose delivery continues off `Lock` — queued in the deferral queue, **or** the in-flight event whose walk
|
||||
suspended at a hop (new: marked when the walk suspends, so the hops after the suspension take the gate) — takes the
|
||||
item's gate and is withheld when the item's latest refresh cut supersedes it (`RefreshCut.Supersedes`, unchanged).
|
||||
- `ConditionRefresh` / `ConditionRefresh2` of folder items has the same two paths as the Server object, flag on and off:
|
||||
- **Nothing deferred:** the SDK base walk of the item's notifier (off `Lock`, live condition nodes), per-item verdict
|
||||
as before (#826 filter under enforcement). Kept only if no deferral episode existed at any moment of it; otherwise
|
||||
the cut path runs too and supersedes it inside the same RefreshStart..RefreshEnd. Its closing check takes `Lock`
|
||||
once: a report whose walk suspended (marked under `Lock` before suspending) has registered its deferral by then —
|
||||
without it a folder refresh could pass the check in the instant between the suspension and the count.
|
||||
- **Deferring — the cut:** verdicts per (item, condition) first, off `Lock`, from a live walk. Then one `Lock` hold:
|
||||
watermark, episode, the Retain set, and per item the same walk producing frozen snapshots, with the cut + "replay
|
||||
pending" marker installed under the item's gate. The replay is queued off `Lock`, one event per gate hold.
|
||||
- State per item (`EventItemState`, previously Server-object only) is keyed by the item object for its lifetime, so it
|
||||
survives unsubscribe + resubscribe; stale cuts are cleared when the deferral episode ends, for folder and Server items.
|
||||
|
||||
### Locking
|
||||
|
||||
Unchanged order, now for folder items too: `Lock` → item gate → the SDK item's own lock. Synchronous delivery runs
|
||||
under the reporter's `Lock`; the verdict takes `Lock` re-entrantly and the core node manager's / authorizer's locks
|
||||
(as the Server-object path already does). Off-`Lock` (deferred) delivery computes the verdict **before** taking the item
|
||||
gate, so nothing under a gate takes `Lock` or the event-report gate. A reporter waits at most for one in-progress
|
||||
`QueueEvent` on an item; a refresh waits at most for one deferred delivery already inside the gate, never for a drain.
|
||||
No reporter ever waits on a channel.
|
||||
|
||||
### Bounds
|
||||
|
||||
No new queue. Per item: the SDK item queue (`QueueSize`, DiscardOldest). Deferral (only on the fallback): unchanged,
|
||||
1024 per condition / 65 536 total, coalescing. Cuts: one per refreshed item, sets sized by the conditions, cleared at the
|
||||
end of the deferral episode.
|
||||
|
||||
### ACL (#826)
|
||||
|
||||
Each event is decided per item at delivery against the current generation — no memoised verdict for condition events
|
||||
any more, so a deploy / revocation / reference removal takes effect on the next event without waiting for an
|
||||
invalidation. Refresh replays only conditions the item may see (the same check); a condition omitted for want of a grant
|
||||
is never withheld afterwards (the cut's existing rule).
|
||||
|
||||
### Cost
|
||||
|
||||
The per-event verdict for folder items moves from the SDK's consumer task (cached) to the reporter thread under `Lock`
|
||||
(uncached), as Server-object items already pay. Folder subscribers are few (ScadaBridge subscribes at `i=2253`).
|
||||
|
||||
## Tests
|
||||
|
||||
- `ConditionRefreshDeferralWireTests` / `FolderEventFloodWireTests` / `EventQueueOverflowAclWireTests`: deferral is no
|
||||
longer produced by flooding a folder item; tests that need it stall the folder's hop through a test probe
|
||||
(`NotifierDeliveryStall`, null in production) instead of the folder item's SDK lock (which would now block the reporter,
|
||||
as it would on the Server object). Flood tests now pin that folder delivery never defers.
|
||||
- New `FolderConditionRefreshWireTests`: folder-item refresh during deferral (flag on/off, retained / cleared,
|
||||
ConditionRefresh2, resubscribe), the gate race, newer-than-cut replay, ACL omission not withheld, ACL revocation during
|
||||
deferral, the in-flight suspended event, and two stress tests (production overload with no stall; stall + refresh +
|
||||
reports concurrently).
|
||||
+46
-22
@@ -901,7 +901,7 @@ Behaviours worth knowing before enabling it:
|
||||
older `Retain=false` event of such a condition still arrives (it agrees with the omission), and a condition that was
|
||||
retained but omitted for the item's **grants** is never withheld (a grant added before the drain lets its events
|
||||
through; the ACL decides them at delivery as ever).
|
||||
- The deferral queues, every other Server-object item and every folder item are untouched. A refresh waits for the
|
||||
- The deferral queues, every other Server-object item and every non-refreshing folder item are untouched. A refresh waits for the
|
||||
drain at most for one in-progress `QueueEvent` on that item (the gate), never for the drain itself; a reporter
|
||||
holding `Lock` likewise waits at most for one. Each item keeps one gate/cut state for its lifetime (keyed by the
|
||||
item, so an unsubscribe + resubscribe keeps it), and cuts are cleared when the deferral count returns to zero.
|
||||
@@ -912,29 +912,53 @@ Behaviours worth knowing before enabling it:
|
||||
- **Measured** (3000 retained conditions, longest `Lock` wait another thread saw during one refresh): the first
|
||||
#854 cut (`1622d07f`, everything under `Lock`) 95–106 ms idle / 102–127 ms deferring; now 0–3 ms idle /
|
||||
23–32 ms deferring (the frozen snapshots).
|
||||
- ⚠️ **Residual: folder (notifier) items.** A refresh of an item on an equipment/device folder can still be followed
|
||||
by older events of a condition. Two sources, neither reachable per item: the SDK's own `MonitoredNode2` channel
|
||||
(up to 4096 events already accepted before the refresh — this exists in the stock SDK too, and in normal operation
|
||||
for the few events in transit), and our deferral queue, whose folder hop goes through that channel, where the SDK
|
||||
memoises the per-item permission verdict by (item, event type, source) so there is no per-event, per-item hook. The
|
||||
events stay in the condition's order and its newest arrives last, so the client converges. Discarding the queue
|
||||
globally on refresh would drop transitions for every non-refreshing subscriber while leaving the channel backlog;
|
||||
making refresh wait would block a service call on the stalled drain. ScadaBridge also ignores a transition older
|
||||
than the one it holds for a condition.
|
||||
- **Folder (notifier) items are delivered per item (Gitea #855).** Before #855 a folder item was fed by the SDK's own
|
||||
`MonitoredNode2` channel (up to 4096 events accepted before a refresh), with the per-item verdict memoised by (item,
|
||||
event type, source), so a refresh of a folder item could still be followed by older events. Now, at every notifier
|
||||
the node manager owns, the walk serves the attached items itself (`DeliverToNotifierItems`), exactly as it serves
|
||||
Server-object items: the SDK's audit guard, the verdict through `INodeManager3.ValidateEventRolePermissions` (SDK
|
||||
type/source `ReceiveEvents` + the #826 decision under enforcement — its item-independent part, the live condition,
|
||||
the notifier check and the visibility scopes, resolved once per hop) **per event, uncached**, then the item's gate and
|
||||
cut, then `QueueEvent`. The `MonitoredNode2` stays for item attachment and for what still uses its channel (a
|
||||
condition's SDK method audit events, data-change items on the folder), so `InvalidateFolderEventVerdicts` stays.
|
||||
- **No notifier of ours defers any more.** `QueueEvent` never waits for a consumer (the item queue discards oldest at
|
||||
`QueueSize`) and the verdict runs on the reporter's thread, which already holds `Lock` — so a burst under one
|
||||
`Lock` hold completes with nothing deferred, and the #838 deadlock shape is gone for condition events. Folder items
|
||||
therefore see condition events in report order. The cost moves with it: the per-event verdict and filter for
|
||||
folder items now run under `Lock` (as for Server-object items), where the SDK used to cache the verdict off it —
|
||||
including the SDK item queue's own per-event costs (its duplicate scan, and at a full queue a list shift), so many
|
||||
folder items on one notifier whose clients stop publishing make each report proportionally slower under `Lock`.
|
||||
With auditing on (off in this repo), a condition's SDK method **audit** event still takes the folder's channel, so
|
||||
at a folder item it may now arrive after the condition event that method produced (they were one FIFO before).
|
||||
- **The deferral queue and the cut stay as the safety net**, for a hop that is not ours (none exists today:
|
||||
conditions are never event notifiers and variables cannot host event items) — and folder items share the cut:
|
||||
a `ConditionRefresh`/`ConditionRefresh2` of a folder item takes the same two paths as the Server object (plain walk
|
||||
off `Lock` when nothing is deferred; otherwise verdicts off `Lock`, then one `Lock` hold for the watermark, the
|
||||
Retain set and, per item, its notifier walk frozen as snapshots, with the cut installed under the item gate and the
|
||||
replay queued off `Lock`). An event whose walk suspended at a hop is marked deferred **before** it suspends, so
|
||||
every later hop judges it against the cut; the plain refresh path's closing check takes `Lock` once, so a report
|
||||
that suspended under it has registered its deferral by then. Lock order is unchanged (`Lock` → item gate → item
|
||||
lock); off `Lock` the verdict is computed before the gate.
|
||||
|
||||
Pinned by `FolderEventFloodWireTests`: 6000 events under one `Lock` hold with publishing disabled (flag on and off)
|
||||
complete, drain to zero, coalesce, and the item delivers at most its revised `QueueSize` in increasing order ending
|
||||
with the last event; and a condition **removed** in the same `Lock` hold has none of its deferred events delivered to
|
||||
either the folder or the Server-object item (the flag-off variant fails without the liveness checks).
|
||||
`ConditionRefreshDeferralWireTests` (#854) stalls a folder item's SDK lock so its channel fills without `Lock` held,
|
||||
refreshes the Server-object item (ConditionRefresh and ConditionRefresh2, flag on/off, condition retained or cleared)
|
||||
and pins that nothing older than the replay follows RefreshEnd while the non-refreshing folder item still receives the
|
||||
whole tail (also across an unsubscribe + resubscribe of the item, and the cut is cleared after the drain); a
|
||||
probe-driven test pins that a refresh waits for a delivery already past its check (fails without the item gate); an
|
||||
ACL-on test pins that a condition omitted for want of a grant is not withheld once the grant lands; a 3000-condition
|
||||
test measures the `Lock` hold and pins that neither path queues its replay under `Lock`; and a stress test runs
|
||||
reports, refreshes and an intermittent stall concurrently (its stream never goes backwards outside a refresh, and
|
||||
every refresh ends on the newest state delivered).
|
||||
complete **with nothing deferred** (fails if the per-item delivery stops recognising the SDK fan-out), and the item
|
||||
delivers at most its revised `QueueSize` in increasing order ending with the last event; and a condition **removed**
|
||||
while its events wait (delivery stalled at the folder) has none of them delivered to either the folder or the
|
||||
Server-object item. Tests that need a deferral stall delivery at a folder through a test probe
|
||||
(`NotifierStall` → `OtOpcUaNodeManager.NotifierDeliveryStall`, null in production), which suspends the walk exactly
|
||||
as a full channel did — holding a folder item's SDK lock, as the pre-#855 tests did, would now block the reporter.
|
||||
`ConditionRefreshDeferralWireTests` (#854) refreshes the Server-object item while the folder is stalled
|
||||
(ConditionRefresh and ConditionRefresh2, flag on/off, condition retained or cleared) and pins that nothing older than
|
||||
the replay follows RefreshEnd while the non-refreshing folder item still receives the whole tail (also across an
|
||||
unsubscribe + resubscribe of the item, and the cut is cleared after the drain); a probe-driven test pins that a
|
||||
refresh waits for a delivery already past its check; an ACL-on test pins that a condition omitted for want of a grant
|
||||
is not withheld once the grant lands; a 3000-condition test measures the `Lock` hold; and a stress test runs reports,
|
||||
refreshes and an intermittent stall concurrently. `FolderConditionRefreshWireTests` (#855) is the folder-item twin:
|
||||
the same matrix on a refreshed folder item (a second subscriber on the same folder keeps the whole tail; a
|
||||
ModifyMonitoredItems re-subscribe keeps the cut), the gate race, a newer-than-cut event on the raw device folder, a
|
||||
condition omitted by the ACL (reference dropped) not withheld once it returns, a revocation after the refresh, a
|
||||
production-shape overload (bursts under `Lock` + repeated refreshes, no stall: nothing deferred, the stream never
|
||||
goes backwards, no replay queued under `Lock`) and a stress test with an intermittent stall.
|
||||
|
||||
**Model-change and audit events on the Server object are gated too** (Gitea #588). The node manager's
|
||||
own server-wide events — `GeneralModelChangeEvent` (NodeAdded / NodeDeleted / DataTypeChanged) and the
|
||||
|
||||
Reference in New Issue
Block a user