dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: dohertj2/lmxopcua:codex/opcua2-saturation
Branches Tags
dohertj2/lmxopcua:master dohertj2/lmxopcua:fix/ci-resource-bounds dohertj2/lmxopcua:fix/801-servicelevel-claim dohertj2/lmxopcua:codex/opcua2-late-slope-tools dohertj2/lmxopcua:codex/opcua-sdk2-disposal-tests dohertj2/lmxopcua:codex/opcua-baseline-canonical dohertj2/lmxopcua:codex/opcua2-saturation dohertj2/lmxopcua:codex/opcua2-test-convergence dohertj2/lmxopcua:codex/opcua2-server dohertj2/lmxopcua:codex/opcua2-client-lifecycle dohertj2/lmxopcua:codex/opcua2-cross-version dohertj2/lmxopcua:codex/opcua2-release-docs dohertj2/lmxopcua:codex/opcua2-driver dohertj2/lmxopcua:codex/opcua2-client dohertj2/lmxopcua:refactor/galaxy-mxgateway-client-rename dohertj2/lmxopcua:auto/driver-gaps-stash dohertj2/lmxopcua:auto/driver-gaps
..
pull from: dohertj2/lmxopcua:codex/opcua2-client-lifecycle
Branches Tags
dohertj2/lmxopcua:fix/ci-resource-bounds dohertj2/lmxopcua:master dohertj2/lmxopcua:fix/801-servicelevel-claim dohertj2/lmxopcua:codex/opcua2-late-slope-tools dohertj2/lmxopcua:codex/opcua-sdk2-disposal-tests dohertj2/lmxopcua:codex/opcua-baseline-canonical dohertj2/lmxopcua:codex/opcua2-saturation dohertj2/lmxopcua:codex/opcua2-test-convergence dohertj2/lmxopcua:codex/opcua2-server dohertj2/lmxopcua:codex/opcua2-client-lifecycle dohertj2/lmxopcua:codex/opcua2-cross-version dohertj2/lmxopcua:codex/opcua2-release-docs dohertj2/lmxopcua:codex/opcua2-driver dohertj2/lmxopcua:codex/opcua2-client dohertj2/lmxopcua:refactor/galaxy-mxgateway-client-rename dohertj2/lmxopcua:auto/driver-gaps-stash dohertj2/lmxopcua:auto/driver-gaps

3 Commits
codex/opcua2-saturation .. codex/opcua2-client-lifecycle

Author SHA1 Message Date
Joseph Doherty 38098ce142 test: add isolated Linux server lifecycle soak launcher 2026-11-10 13:08:44 -05:00
Joseph Doherty 74c938f37c Merge commit '642aefe3' into codex/opcua2-client-lifecycle 2026-11-10 13:01:14 -05:00
Joseph Doherty c25561c64d test: add production client lifecycle soak harness 2026-11-10 12:41:51 -05:00
68 changed files with 126 additions and 7043 deletions
Expand all files Collapse all files
Directory.Packages.props
+2 -2
View File
@@ -242,7 +242,7 @@
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Theme" Version="0.5.0" />
<PackageVersion Include="ZB.MOM.WW.HistorianGateway.Client" Version="1.0.0" />
<PackageVersion Include="ZB.MOM.WW.HistorianGateway.Contracts" Version="1.0.0" />
<PackageVersion Include="ZB.MOM.WW.HistorianGateway.Client" Version="0.4.0" />
<PackageVersion Include="ZB.MOM.WW.HistorianGateway.Contracts" Version="0.4.0" />
</ItemGroup>
</Project>
docs/Historian.md
+6 -9
View File
@@ -337,7 +337,7 @@ and is registered the same way for every backend.
> **Cross-repo follow-up — a multi-tag batch `WriteLiveValues` on the gateway contract.** The drain's
> unit of work is one gateway round-trip **per historian tag**, because
> `ZB.MOM.WW.HistorianGateway.Client` **1.0.0** exposes no RPC that writes values for more than one tag
> `ZB.MOM.WW.HistorianGateway.Client` **0.4.0** exposes no RPC that writes values for more than one tag
> in a call. Bounded parallelism (`DrainConcurrency`) is therefore the ceiling: it overlaps the
> round-trips but cannot remove them, so a wide tag set still pays `⌈tags / concurrency⌉` network
> hops per drain pass. Collapsing that to a single hop needs a **batch/multi-tag `WriteLiveValues`** on
@@ -345,11 +345,8 @@ and is registered the same way for every backend.
> lands, raising `DrainBatchSize` widens each hop's payload but not the hop count — which is why the
> concurrency window, not the batch width, is what governs drain throughput today.
The HistorianGateway **Client and Contracts packages are pinned together at 1.0.0**
([historiangw#131](https://gitea.dohertylan.com/dohertj2/historiangw/issues/131)). The adapter maps
`AllowUntrustedServerCertificate` directly and only for TLS: the default verifies the server, while an
explicit opt-out applies only when `UseTls=true`. Stale certificate options are ignored on plaintext h2c.
This follows the earlier [0.4.0 update](https://gitea.dohertylan.com/dohertj2/historiangw/issues/119). The removed
The HistorianGateway **Client and Contracts packages are pinned together at 0.4.0**
([historiangw#119](https://gitea.dohertylan.com/dohertj2/historiangw/issues/119)). The removed
`ReadBlocks` RPC and `ReadBlocksAsync` wrapper were unused here. A partially imported CA PEM bundle now
fails client creation instead of accepting a narrower certificate pin; repair the configured bundle if
startup reports `ArgumentException` for `CaCertificatePath`.
@@ -1851,9 +1848,9 @@ value at **10:00**, not 09:00, and no point ever falls before the window. The in
`(start − (k+1)·interval, start − k·interval]`, the earliest one truncated, each stamped with its later edge.
**SDK parity (#677).** `SdkAggregateParityTests` (unit, `Sql.Common.Tests`) and
`ProcessedConformance.Processed_reads_match_the_SDK_aggregate_calculators` (both SQL backends, through the real SQL) use the preserved
1.5.378.182 calculator source through `CompatibilityAggregateFactory`, not the new SDK 2 algorithms, over the same samples,
forward and reverse, and compare every
`ProcessedConformance.Processed_reads_match_the_SDK_aggregate_calculators` (both SQL backends, through the real SQL) run the pinned SDK's
own calculators (`AverageAggregateCalculator`, `MinMaxAggregateCalculator`, `CountAggregateCalculator`, the base `AggregateCalculator`
for Interpolative, `StatusAggregateCalculator` for PercentGood/Bad) over the same samples, forward and reverse, and compare every
interval's timestamp, value, status code and Partial bit. The per-sample comparison covers Average, Minimum, Maximum, Count, Range,
Start, End, Delta and Minimum/MaximumActualTime; the time-weighted one TimeAverage, TimeAverage2, Total, Total2, Interpolative and
PercentGood/Bad (#680), including datasets whose data ends inside the window and with an interval wholly inside an outage; reverse
docs/plans/2026-11-10-opcua-sdk-2-evidence.md
+5 -267
View File
@@ -4,7 +4,7 @@ Execution started 2026-11-10 for [#782](https://gitea.dohertylan.com/dohertj2/lm
## Baseline and authority
- Baseline `83ed880e4471344deaf008da9bd028c7e170bdc7` equaled `origin/master` when execution started; original checkout was clean. Master subsequently advanced to the independent HistorianGateway package update `16111a5e`, merged into the candidate without content conflicts. Dedicated managed worktree, branch `codex/opcua-sdk-2`.
- Baseline `83ed880e4471344deaf008da9bd028c7e170bdc7` equals fetched `origin/master`; original checkout clean. Dedicated managed worktree, branch `codex/opcua-sdk-2`.
- Target remains exact stable SDK `2.0.1`, upstream `07e62f72500cc0923a2bab940653c8c1116fa286`. Tagged source and restored assemblies control API decisions.
- Baseline packages are already `1.5.378.182`, not the plan preparation checkout's `.106`. #751 reflection helper exists. AggregateManager application configuration override already exists; preserve it.
- #782 previously parked the major upgrade; the user's current explicit execution instruction supersedes that scheduling decision. No deployment authorization overrides the Wonder-box remote-execution hold.
@@ -65,11 +65,11 @@ No optional ManagedSession/DI/AOT/cluster redesign, trust relaxation, silent war
## Gates
- G0 baseline/design: accepted for migration, with corrections and external/trust-matrix gaps recorded below.
- G1 restored 2.0 graph / break inventory: passed; compilation convergence subsequently passed G2.
- G2 compilation convergence: passed; whole-solution Release builds have zero warnings/errors and temporary migration assets are removed.
- G1 restored 2.0 graph / break inventory: passed for lane execution; compilation convergence pending.
- G2 compilation convergence: pending.
- G3 behavioral/security/interoperability: pending.
- G4 lifecycle/performance/publish: pending.
- Astra security/ownership source findings: closed for the reviewed identity, history, audit and browse changes; final release review remains pending.
- Astra security/ownership and release reviews: pending.
No upgraded artifact is approved for release.
@@ -129,266 +129,4 @@ Restore succeeds without downgrade suppression or source changes. The first comp
This is the exposed first wave, not the solution-wide final break count: server overrides, removed collections, immutable/null/Variant APIs, certificate manager, and async identity construction block downstream compilation. Log and exact diagnostics: `artifacts/opcua2/d9984028/sdk2-spike/{build.log,diagnostics.txt}`. Later diagnostics are inventoried by lane until G2 convergence; no compile failure is claimed as behavioral success.
Restored project graphs contain no mixed Foundation SDK versions: every Foundation package (including analyzer-only development transitive GDS assets) is 2.0.1. Host graph resolves Bcl.TimeProvider 10.0.12 and Http.Resilience 10.10.0; client assets resolve RateLimiting 10.0.12, while the Host shared framework prunes that separate package. G1 is satisfied for lane execution; full build/tests remain required at G2/G3. Frozen boundary contract is above; production and unit paths are split by plan ownership, integration/historian oracle edits coordinator-only.
## Integrated implementation and source reviews
The implementation is on `codex/opcua-sdk-2`, not deployed or merged to master.
Master's independently delivered HistorianGateway client/contracts 0.4.0 change
`16111a5e` is included, and master ancestry was merged at `88ee30e0` without a
content change. historiangw#119 is closed; historiangw#78 deployment is separate.
The server, client and driver lanes were integrated and validated in isolated
worktrees. Astra reviewed the design, aggregate compatibility architecture,
identity/peer provenance, certificate ownership, production history continuation
store, write conversions/revert ownership, alarm compatibility, and filtered event
and audit delivery. No source review is represented as release approval.
- Identity registry admission is installed before listeners and remains until
shutdown. Rejected/disabled credentials cannot fall through to a default identity.
SDK role augmentation preserves the application's principal and key ID.
- Activation uses public transport peer evidence, scoped across asynchronous work
and restored in `finally`. Unknown peers retain the existing process-wide budget;
this application has no CIDR admission or reverse-connect policy to invent.
- Immutable SDK values are normalized only at existing application boundaries.
Scalar numeric leniency, checked one-dimensional conversions, pass-through arrays,
byte/ByteString rank, null/empty and detached buffers remain pinned. Matrices remain
unsupported. Public SDK-facing client DTOs remain SDK-facing.
- Session history tokens are consumed/released through the real SDK store, with
exact node/realm, read kind, topology epoch and current authority checks. Existing
canonical-query semantics are retained; no undocumented filter fingerprint was
introduced.
- SDK2 alarm early AlreadyAcked/AlreadyConfirmed checks are adapted only for the
current nonempty trunk EventId, preserving application authorization and routing.
Write reverts bind the actual node instance and committed attempt, preventing a
late failure from undoing a newer write/publish or replacement node.
- Audit delivery uses a private, revocable operation context for one exact source
and type. Both existing permission callbacks can veto; no global role/node
permission array is widened. Ordinary SDK automatic audits remain denied. The
encrypted wire test proves exactly one application audit to the Read audience
and none to the denied audience.
A cross-version wire test found an additional SDK2 creation change: `Create`
reset an alarm's preassigned reference to the null NodeId, hiding it from its raw
parent's hierarchical browse. Restoring HasComponent **after** Create fixed all
three supported alarm types in both realms (`fa823378`, integrated `41e65f6a`).
All 12 actual-node/wire regressions failed first, then passed. Full server units
at that fix: **1115 passed, zero failures, one existing skip**. Astra approved
its source scope and the matching separate-process results.
## Preserved aggregate algorithms and independent oracles
The owner explicitly chose current application results over changed SDK2
aggregate semantics. A private source adapter contains the eight original MIT
algorithm/factory files from 1.5.378.182 with original source hashes. It is not an
old SDK runtime. A fresh SDK AggregateManager registers the same 37 factories
exactly once; calling the base registration first would duplicate diagnostic nodes.
The separately pinned original-SDK generator produced **6,144 vectors**: the
original 4,144 plus 2,000 numeric-width, nonfinite/conversion, tie/picosecond and
configuration edge cases. Fixture SHA-256:
`2ec2a1ac97f05d199c18da55157f8bbe0276853c3ba73ebf1b509c535c5a7dd8`.
Original SDK assembly SHA-256:
`d8d9706dea577858325730cd89896826ed82f7bfcf9503ccd0260311663a69f4`.
All cases pass both directly and through the booted production aggregate manager.
Values, CLR types, all status bits, source/server time and picoseconds, exceptions,
queue acceptance and incremental/partial results are compared. SDK2 FILETIME zero
is explicitly observed as the original CLR unset sentinel; no calculation or
expected output was regenerated from the candidate.
The SQL application's PercentGood policy already differs from SDK1.5.378.182.
The same ten of 275 parity tests fail on the original and candidate with identical
names/messages. Independent application results and regression expectations stay
unchanged; [#796](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/796)
tracks that baseline debt. It is not a green test run.
## Local compatibility and security evidence
- Full wire integration reached **187 passed, zero failures/skips** in the second
CI-equivalent run, excluding the separately gated E2E soak category. It includes
all original contracts plus production continuation, trusted-channel, audit,
hierarchy and current authority regressions.
- Focused final security probes: **17/17**, zero skips. Seven trusted cases cover
explicit DER trust, Sign and SignAndEncrypt, invalid credentials, an untrusted
client, same-session reactivation, and concurrent IPv4/IPv6 attempts correlated
with their own peer before/after an await barrier. Eighteen earlier continuation
cases cover raw/modified/events with alias, replacement, rebuild, grant/key
revocation and expiry, proving no backend dispatch or saved-data disclosure.
- Native current-trunk retry tests use a narrow raw-device grant with zero global
roles, then revoke/expire authority and require no route/state/EventId/event
change. Scripted retry tests explicitly use the existing cluster grant.
A separate real-session original/candidate probe preserves the opaque scripted
ID's equipment-scope denial and broad-grant positive. This **preexisting**
production feature gap remains [#797](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/797);
a slash-qualified synthetic ID is not a valid UI-authored workaround.
- Target certificate manager/endpoint identity tests: **7/7** (valid, expired,
revoked, missing issuer, issuer usage, hostname and URI). These are manager/API
checks, not seven network handshakes.
- Corrected a preexisting client creation-policy bug: 2048 meant months, not RSA
bits. New certificates use the SDK's 12-month default; existing longer-valid
certificates and identical private-key bytes are reused. A real-factory red
lifetime case became green; all **324 Shared tests passed**. RSA minimum remains
the SDK's separate 2048-bit default.
- Separate-process original→candidate, candidate→original, candidate→candidate,
and original→original Shared legs plus candidate driver/actual CLI legs:
**62/62 passed**. Strict alarm identities remain asserted. Shared/server/driver
use explicit mutual DER trust and encrypted API-key sessions; CLI's existing
autoaccept is disclosed and is not rejection-policy evidence. The same harness
first exposed the hierarchy defect at **60/62**. Commands, artifact hashes,
policies and outcomes are checked in under
`scripts/validation/opcua-sdk-compatibility/evidence/2026-11-10/`.
- The actual unchanged ScadaBridge adapter, SDK1.5.378.106, passed **9 None/API-key
cases against each server** (18 total). Strict encrypted connections fail on
both original/candidate because its application certificate is unconfigured:
[ScadaBridge#121](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/121).
That is a retained failure, not a secure interoperability pass. Reproducible
isolated consumer receipts are in `scripts/validation/scadabridge-sdk-compatibility/`.
- Real local SQL Server: **195/195** on each version. QuestDB10.0.1: **233 passed,
zero failed, one existing opt-in race-stress skip** on each version. Case maps
match exactly. Only task databases/prefixes and the task QuestDB container/volume
were removed. See [backend receipts](evidence/2026-11-10-opcua2/backend-matrix.md).
This is not real AVEVA evidence.
## Solution, performance and package evidence
Whole-solution Release builds pass with **zero warnings/errors**, including all
inventory consumers. All restored production Foundation packages are 2.0.1; no
MigrationAnalyzer reference/runtime asset remains. The only old SDK is isolated
in the explicitly original-SDK validation process outside the solution.
The first full CI-equivalent run executed **18,459** cases with execution guard
passed (floor12,000), but failed203: 192 from an incorrect test-oracle reflection
port (repaired and **554/554 Core.Abstractions tests passed**), the ten known
PercentGood baseline failures, and a ScriptWorker250ms timing assertion.
The latter also fails independently on untouched baseline83ed880e (880.13ms),
so no threshold/filter/skip was changed; it is
[#798](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/798).
The fresh second full solution run produced **63 TRX files, 19,121 total,
18,486 executed, 18,475 passed, 11 failed and 635 existing skipped cases**.
Its test exit is **1**, execution guard **0** (floor12,000). The exact eleven
failures are the same ten PercentGood cases (#796) and one timing case (#798);
there are no additional failure classes. This is baseline-matching evidence,
**not a green solution run**. The command/logs are retained under ignored
`artifacts/opcua2/88ee30e0/solution-tests2/`;
[versioned counters and exact failures](evidence/2026-11-10-opcua2/solution-tests2.json)
retain the nonzero status. Fixture-backed gates remain separate.
Three alternating warmed lock measurements per version show medians17→15ms for
10k-tag rebuild and81→80ms for50k-node removal, with equal visit counts.
See [bounded subset and host-load caveats](evidence/2026-11-10-opcua2/lock-benchmarks.md).
The separate 72,000-call warmed native-process experiment records median p95
changes +14.95% scalar read, +1.58% scalar write, -16.28% array read, -17.24%
array write, plus lower allocated bytes/workload operation. It retains the
scalar-read outlier, small-workload scope, host-load and JIT-mode caveats. Neither
these samples nor the short cold-tiered measurements certify full G4.
Four self-contained publishes (Linux ARM64 and Windows x64 Host/CLI) pass with
runtime10.0.14, unique SDK DLL basenames, native RID validation and file hashes.
Actual Linux Host passes two start/stop cycles with unchanged certificate identity,
healthy web checks, real HMAC-key positive/negative cases and strict peer negatives.
Console/rolling logs contain zero task-secret occurrences. See
[publish/runtime receipts](evidence/2026-11-10-opcua2/publish-smoke.md).
Windows service/runtime installation remains
[#800](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/800), not inferred from
cross-publishing. Approved live AVEVA/consumer/pair validation remains
[#801](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/801).
## Resource finding and long-run scope
The Mac SDK-server restart soak failed twice around17 minutes at the unchanged
initial-RSS+128MiB guard. One run completed250 cycles/62 restarts; managed memory,
FDs, cached stores and logical collections remained bounded, but process RSS did
not meet its limit. **No two-hour pass is claimed.**
[#799](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/799) retains the failed
TRX/JSONL and active investigation. Explicit SDK wrapper tracking found created=
disposed and no outstanding/unreachable-undisposed wrapper after cleanup;
controlled private PFX imports/SDK copies accumulate native Apple Security
objects, unlike shared-handle/public-only controls. These controls do not attribute the entire RSS increase or isolate the server
from the co-located SDK client. Astra found no safe narrow public copy-policy
setting in exact2.0.1. Switching channel/session factories would change lifecycle
semantics and requires separate design. No unproven production patch or resource-
budget waiver was applied.
The original Mac PKI-specific descriptor zero was invalid because `/var` and
`/private/var` were compared literally. Corrected canonical, directory-bounded
instrumentation proves0→1→0 for an open test file and reaps a timed-out probe.
The original logical/global-FD evidence remains usable; its PKI-zero claim does
not. A corrected90-second smoke is explicitly not two-hour evidence.
The same-cap self-contained **Linux ARM64 server/node-manager workload also
failed**, at **248 cycles / 61 restarts / 903.508 seconds**: RSS353,931,264
exceeded351,850,496 bytes (cycle3 baseline+128MiB). Managed memory~20.5MB,
descriptors189, PKI descriptors0, cached stores2 and logical collections remained
bounded. Runtime10.0.14, SDK2.0.1; test/container/launcher fail. This broadens
#799 beyond macOS; no complete/two-hour pass exists.
[Selected Linux failed-run receipts](evidence/2026-11-10-opcua2/server-linux-soak-failed.json)
retain hashes, unchanged limits, positive descriptor controls and failure.
The independent root-owned actual production Shared/driver/live-reinitialize/
browser lifecycle run on Linux is still in progress. Its final elapsed duration,
counters, resource limits and completion/exit guards must be recorded separately.
A short diagnostic, a running process, or a different workload is never substituted
for a required passing gate. G4 and release approval remain open.
## Final integrated source review and diagnostic instrumentation
Astra (`gpt-6-astra`, xhigh) reviewed integrated source `aa9af0dc` read-only and
reported **no new must-fix migration source/security finding**. It rechecked
identity registry/rejection, scoped public peer evidence, restricted identity
behavior, actual continuation ownership/node/topology binding, temporary audit
capability, exact alarm retry guards, write/node attempt ownership and client/
driver generation/disposal. Its independently inspected security receipt is
17/17 with zero skips. This is source/security review, **not release approval**:
Astra retains #799 and explicitly withholds G4/release approval on both platforms.
The reviewed production tree remains unchanged by later diagnostic commits.
Test-only instrumentation now writes the complete captured resource tuple before
resource assertions and includes GC committed/heap/fragmentation, pinned/finalizer,
JIT method/IL, thread and timer counters. Explicit bounded profile mode preserves
the first original budget rejection, marks subsequent samples rejected, and
rethrows it before completion; full/diagnostic modes still fail immediately.
The full7200-second minimum and every original cap are unchanged. Both descriptor
positive/timeout controls passed again, **2/2**, at integrated `b57595cb`.
Historical failed-run hashes and source revisions remain exact; they are not
relabeled as results from this newer instrumentation.
The pushed implementation is reviewable as
[draft PR#802](https://gitea.dohertylan.com/dohertj2/lmxopcua/pulls/802).
It is explicitly held from merge/release, and #782 remains open.
## Completed cross-platform RSS investigation
The [reproducers, report](../../scripts/validation/opcua-lifecycle-memory/README.md)
and [hashed evidence](../../scripts/validation/opcua-lifecycle-memory/evidence.json)
preserve four failed canonical workload/profile attempts and the unchanged caps.
A 30-minute Linux **diagnostic subset** completed on both versions: candidate
768 cycles/192 restarts, original736/184. Candidate retained managed memory was
19.25 MB, final RSS334.18 MB, with19,272 certificate wrappers created and disposed
and none outstanding. It has **no full-gate pass** and omits canonical history,
revocation, topology surgery and network-stall work. The original subset's growing
managed/timer footprint cannot provide a favorable steady-state performance
comparison denominator.
EventPipe attributed a large late step to6,690 distinct generated standard-node
factory methods receiving Tier1 compilation (49,141,484 cumulative native method
bytes). A later180-second interval had zero SDK/factory JIT starts and zero lost
events. These code-event sums are not RSS/PSS allocation measurements. This
supports a finite compilation burst in the subset; full-workload saturation has
not been established. macOS native Apple Security object retention continues
independently. No narrow application workaround, factory swap, runtime flag,
resource threshold change or acceptance-baseline reset was adopted.
Astra reviewed the packaged sources/hashes/raw-receipt consistency/sanitization
at integrated `6f793d25` and found no must-fix report/source issue. It approved
only a **prospective diagnostic saturation design**, recorded in the report:
384 fixed canonical cycles/96 restarts plus one settling cycle within1,800 seconds;
fixed traced windows321–352 and353–384; zero generated-factory compilations,
new SDK assembly loads and lost events; ownership checks active throughout.
That experiment has not run and cannot pass G4. Matched full acceptance requires
an original-SDK adapter for the identical canonical workload and at least7,200
seconds **after** a prospectively reviewed fixed warmup. Historical failures and
macOS native retention remain open in #799. No release approval was given.
Restored project graphs contain no mixed Foundation SDK versions: every Foundation package (including analyzer-only development transitive GDS assets) is 2.0.1. Host graph resolves Bcl.TimeProvider 10.0.12, RateLimiting 10.0.12, Http.Resilience 10.10.0. G1 is satisfied for lane execution; full build/tests remain required at G2/G3. Frozen boundary contract is above; production and unit paths are split by plan ownership, integration/historian oracle edits coordinator-only.
docs/plans/2026-11-10-opcua-sdk-2-rollout.md
+22 -65
View File
@@ -3,12 +3,10 @@
This operator runbook accompanies [#782](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/782),
the [implementation plan](2026-11-10-opcua-sdk-2-upgrade.md) and
[execution evidence](2026-11-10-opcua-sdk-2-evidence.md). Gitea is the single
status system. This runbook was reconciled with checkout `aa9af0dc` and the
commit-specific local receipts linked below. It records local validation, not
operator release approval or a fleet deployment. Receipt revisions vary; use each
report's full source/artifact identity rather than treating every result as a test
of this documentation commit. Open failures and operator gates still prevent a
release go decision.
status system. This draft was checked against source `a21104d2`; it does not
approve an artifact or record a deployment. The coordinator must attach final
commit-specific evidence in #782 before advancing a gate. All placeholders below
are **pending evidence**, not passing results.
The target is exact OPC Foundation SDK **2.0.1**; the known-good SDK baseline is
**1.5.378.182** (application baseline `83ed880e4471344deaf008da9bd028c7e170bdc7`).
@@ -21,7 +19,7 @@ Package downgrades alone cannot reverse the application/API migration.
No WinRM, psbridge or replacement remote-exec channel may be used until the user
clears a channel. UNC/SMB file copying and permitted SQL access do not authorize
service control, PowerShell execution or deployment on those hosts. Steps requiring
a command on a held host remain pending. No fleet deployment was performed.
a command on a held host remain pending. No deployment was performed for this draft.
Run the service commands below only from an approved local operator session on an
authorized Windows node. Record the node, account, channel and maintenance window
@@ -54,60 +52,23 @@ also touches gateway/retired sidecar paths outside this SDK rollout.
cluster rolling-upgrade capability is claimed. HistorianGateway Client/Contracts
**0.4.0** is already on gateway master `16111a5e` (historiangw#119 closed);
the real gateway gate historiangw#78 remains separate from package alignment.
- Certificate creation correction `bd3143b7` uses the SDK's **12-month** default
instead of treating 2048 as months. The separate RSA minimum stays 2048 bits;
existing certificates/private-key bytes and PKI identity are retained.
## Release record and preflight
The following local results have recorded evidence. They do not substitute for
unexecuted operator or duration gates. Keep secrets out of logs and release records.
Fill this record in #782 with links to immutable logs/manifests; do not paste keys,
passwords, connection strings, private certificates or unredacted environment dumps.
| Local validation | Recorded result and receipt |
| Required evidence | Record to supply before release |
|---|---|
| G2 compilation and graph | Whole-solution Release builds pass with **0 warnings/errors**; production Foundation packages are exact 2.0.1, with no runtime migration analyzer. [Build/package evidence](2026-11-10-opcua-sdk-2-evidence.md#solution-performance-and-package-evidence). |
| Security and application contracts | **17/17** final security probes; **187** wire cases; **324** Shared tests; **1115** server units with one existing skip. Seven certificate-manager/API checks are separate from network handshakes. [Case scope and provenance](2026-11-10-opcua-sdk-2-evidence.md#local-compatibility-and-security-evidence). |
| Original aggregate results | **6,144 vectors pass directly and through the production manager**. Fixture SHA-256 `2ec2a1ac97f05d199c18da55157f8bbe0276853c3ba73ebf1b509c535c5a7dd8`; [original-SDK provenance and comparison scope](2026-11-10-opcua-sdk-2-evidence.md#preserved-aggregate-algorithms-and-independent-oracles). |
| Separate-process compatibility | **62/62** baseline/candidate Shared, driver and actual CLI cases pass. Strict mutual DER trust/encrypted API-key coverage is separate from CLI autoaccept. [Matrix receipts](../../scripts/validation/opcua-sdk-compatibility/evidence/2026-11-10/). |
| Actual ScadaBridge adapter | **18 None/API-key cases pass**, nine against each server. Encrypted attempts fail against both due to the preexisting own-certificate configuration gap, [ScadaBridge#121](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/121); secure interoperability is blocked. [Consumer receipts](../../scripts/validation/scadabridge-sdk-compatibility/). |
| Local historian backends | SQL Server **195/195** on each version; QuestDB **233 passed, 0 failed, 1 existing opt-in race-stress skip** on each. Case outcomes match. [Backend matrix](evidence/2026-11-10-opcua2/backend-matrix.md). This is not live AVEVA evidence. |
| Bounded performance subset | Three warmed measurements per version: median rebuild **17→15 ms**, removal **81→80 ms**, equal visit counts. [Benchmark scope/load caveats](evidence/2026-11-10-opcua2/lock-benchmarks.md); these are not full wire-throughput, p95 or steady-RSS approval. |
| Publish/package and Linux smoke | **Four** self-contained Linux ARM64/Windows x64 Host/CLI publishes pass. Actual Linux full Host passes **two start/stop cycles**, strict security/health checks and unchanged PKI. [Per-artifact revisions, SHA-256 manifests and runtime receipts](evidence/2026-11-10-opcua2/publish-smoke.md). Windows execution is still pending. |
| Source review | Independent reviews covered identity, ownership, aggregate preservation, hierarchy and filtered audit paths. [Review scope](2026-11-10-opcua-sdk-2-evidence.md#integrated-implementation-and-source-reviews). Source review is not release approval. |
The second full CI-equivalent run completed with **63 TRX, 19,121 discovered,
18,486 executed, 18,475 passed, 11 failed and 635 existing skips**. Test exit was
**1**; execution guard exit was **0** at floor 12,000. It is not green: the exact
failures are ten unchanged PercentGood cases and one ShadowMode 250 ms timing
assertion (585.7377 ms), independently reproduced on baseline, with no new failure
classes. Summary: `/tmp/opcua2-ci2-summary.json`; final provenance is in the
[main evidence](2026-11-10-opcua-sdk-2-evidence.md#solution-performance-and-package-evidence).
The ten PercentGood failures are tracked in
[#796](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/796), and the separately
reproduced baseline ScriptWorker timing failure is
[#798](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/798).
The opaque scripted-alarm narrow-scope denial is a separate preexisting functionality
gap, [#797](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/797), not a new
SDK incompatibility or an approved synthetic-ID workaround.
**The server resource gate failed.** The Mac restart soak failed twice around
17 minutes at the unchanged RSS cap. The same-cap self-contained Linux server
workload also failed at **248 cycles / 903.5 seconds**: RSS **353,931,264 bytes**
exceeded **351,850,496 bytes**. Its retained result is
`/tmp/opcua2-server-linux-full-20261110/task/results/soak.trx`.
[#799](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/799) owns investigation;
no cap waiver or two-hour server pass is recorded. The distinct actual-production
Shared/driver/reinitialize/browser two-hour Linux lifecycle run is still active;
its partial duration is not a pass. See [long-run scope](2026-11-10-opcua-sdk-2-evidence.md#resource-finding-and-long-run-scope)
for instrumentation limitations and final-result updates.
| Operator release field | Still required before its release decision |
|---|---|
| Recovery set and candidate selection | Actual deployed known-good application/config archive hashes, protected PKI/permission backup, selected complete candidate artifact and matching source/manifest. Local artifact manifests above are available; a fleet recovery set is not inferred. |
| Windows runtime/service | Approved-node installation, service account/ACL checks, preserved-PKI two-start/stop and protocol checks: [#800](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/800). Cross-publishing does not satisfy this. |
| Live AVEVA/consumer/pair | Licensed gateway/backend, remaining secure consumer work and mixed-version Akka/config/failover validation: [#801](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/801), with historiangw#78 and ScadaBridge#121 linked as applicable. No rolling-safe claim. |
| Resource/performance acceptance | Resolve #799, record completed required-duration runs and remaining representative load/budget evidence; no shortened run or changed cap substitutes. |
| Rollout/rollback authorization and results | Named operator, cleared channel, environment/window, actual schema/serialization assessment, canary/rollback checks and final review/go decision in #782. The wonder-box hold remains binding. |
| Candidate source and graph | Pending: full commit, clean/dirty state, SDK/runtime/OS, resolved package graph showing exact 2.0.1 |
| Known-good recovery set | Pending: deployed application version, full binary/config archive SHA-256, protected PKI backup location and permission record |
| Candidate artifacts | Pending: Windows and Linux publish directories/archives, RID/self-contained choice, SHA-256 manifest, dependencies and script-worker contents |
| G2 compilation | Pending: restore/build logs and warning/error counts at the final SHA |
| G3 behavior/security | Pending: project TRX/counts, required case results, positive and negative authorization, audit privacy, CP lifecycle and 6,144-vector comparison |
| Interoperability | Pending: baseline/candidate client-server pairs, actual ScadaBridge adapter, upstream driver/browser products and mixed-version cluster gate |
| Historian | Pending: configured SqlServer/QuestDB/Gateway capability-specific results; historiangw#78 link/evidence for live gateway |
| G4 resources/performance | Pending: baseline comparison, at least three measured runs, two-hour soak, Windows/Linux start-stop-twice smoke with preserved PKI |
| Review/operations | Pending: independent security/release review, authorized channel/window, named operator, canary/rollback results and decision |
Before staging:
@@ -316,12 +277,8 @@ thresholds or count absent environments as passed.
## Results handoff
Local build, security/wire, aggregate, backend, compatibility, publish and Linux
smoke evidence is linked above with its actual scope and revisions. The server
RSS gate is failed (#799), full CI retains the 11 baseline failures (#796/#798),
the client lifecycle run is still active, secure ScadaBridge interoperability is
blocked (ScadaBridge#121), and
Windows/live AVEVA/pair/operator gates remain pending (#800/#801). No fleet rollout
or rollback has been executed. The coordinator records the final client-lifecycle result
and remaining decisions in #782 and the main evidence; none is inferred from a
passing subset or cross-published artifact.
Pending: coordinator to attach the final SHA, artifact hashes, TRX counts/skips,
interoperability/performance/runtime evidence, reviews and per-environment operator
records in #782. This document makes no claim of a passing build, test, ScadaBridge,
Windows/Linux smoke, live backend, soak or deployment. Required inaccessible steps
remain open gates unless formally separated into linked issues with accurate scope.
docs/plans/evidence/2026-11-10-historiangw-131.json
-113
View File
@@ -1,113 +0,0 @@
{
"issue": "https://gitea.dohertylan.com/dohertj2/historiangw/issues/131",
"consumerBase": "16111a5e",
"opcUaSdk": "1.5.378.182 (independent of SDK2 migration)",
"build": {
"command": "dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release -m:1",
"exit": 0,
"warnings": 0,
"errors": 0,
"seconds": 37.33,
"logSha256": "cb4f1ee96bf05254e5c707179c1e81d99d8267e28c5117a84afa0b0c853fa329"
},
"packages": [
{
"package": "ZB.MOM.WW.HistorianGateway.Client/1.0.0",
"type": "package",
"source": "https://gitea.dohertylan.com/api/packages/dohertj2/nuget/index.json",
"repositoryCommit": "ea9f21c850fb618767e12690924f9966f1e433ac",
"sha256": {
"lib/net10.0/ZB.MOM.WW.HistorianGateway.Client.dll": "b89a02e0d228ec549c4dfb755694f7f67b573a6077f06ff2781165e884fc5e12",
"zb.mom.ww.historiangateway.client.1.0.0.nupkg": "0a3f2836727671c07c0c393e705cc9737d2ed44258213a51fb4acf0d0ab7129e"
}
},
{
"package": "ZB.MOM.WW.HistorianGateway.Contracts/1.0.0",
"type": "package",
"source": "https://gitea.dohertylan.com/api/packages/dohertj2/nuget/index.json",
"repositoryCommit": "ea9f21c850fb618767e12690924f9966f1e433ac",
"sha256": {
"lib/net10.0/ZB.MOM.WW.HistorianGateway.Contracts.dll": "e60d3d4650be5d251982f7847404daf1b666f61d56e2b0ccb844af8677e97743",
"zb.mom.ww.historiangateway.contracts.1.0.0.nupkg": "83ffd12bcbbf919c973ba522560e10aa93a4cb8c686a3232030a2cfa5f00995e"
}
}
],
"sourceOverrides": false,
"tests": [
{
"project": "gateway",
"testExit": 0,
"executionGuardExit": 0,
"counters": {
"total": "529",
"executed": "523",
"passed": "523",
"failed": "0",
"error": "0",
"timeout": "0",
"aborted": "0",
"inconclusive": "0",
"passedButRunAborted": "0",
"notRunnable": "0",
"notExecuted": "0",
"disconnected": "0",
"warning": "0",
"completed": "0",
"inProgress": "0",
"pending": "0"
},
"outcomes": {
"Passed": 523,
"Failed": 0,
"NotExecuted": 6
},
"requiredTlsWireTests": [
{
"name": "ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.HistorianGatewayTlsPolicyWireTests.Explicit_tls_opt_out_reaches_the_same_self_signed_server_shape",
"outcome": "Passed"
},
{
"name": "ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.HistorianGatewayTlsPolicyWireTests.Default_tls_rejects_an_untrusted_server_before_the_rpc",
"outcome": "Passed"
},
{
"name": "ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.HistorianGatewayTlsPolicyWireTests.Plaintext_ignores_stale_tls_opt_out_and_ca_settings_and_reaches_the_rpc",
"outcome": "Passed"
}
],
"trxSha256": "68086e66da0942214c0be805366b47c443f633d90f217c91e07ec685e8aea9d9"
},
{
"project": "runtime",
"testExit": 0,
"executionGuardExit": 0,
"counters": {
"total": "1005",
"executed": "1002",
"passed": "1002",
"failed": "0",
"error": "0",
"timeout": "0",
"aborted": "0",
"inconclusive": "0",
"passedButRunAborted": "0",
"notRunnable": "0",
"notExecuted": "0",
"disconnected": "0",
"warning": "0",
"completed": "0",
"inProgress": "0",
"pending": "0"
},
"outcomes": {
"Passed": 1002,
"Failed": 0,
"NotExecuted": 3
},
"requiredTlsWireTests": [],
"trxSha256": "c12308fd650be0646beba044a4b3301dc2aefc43097b2da667d0eb5da408fc65"
}
],
"review": "Astra approved source/security mapping and corrected real-wire tests; actual published package verification complete.",
"deploymentPerformed": false
}
docs/plans/evidence/2026-11-10-opcua2/backend-matrix.md
-65
View File
@@ -1,65 +0,0 @@
# Local historian backend matrix — 2026-11-10
Candidate a8e1082d69a112ea0e16788f023067979698527a (SDK 2.0.1) compared with baseline 83ed880e4471344deaf008da9bd028c7e170bdc7 (SDK 1.5.378.182). This is the existing direct backend integration test matrix over real local SQL Server and QuestDB. It is not an OPC UA wire test, AVEVA validation, or stress/soak approval. No source edits, expected-value changes, new skips, suppressions, issue updates, pushes, or commits were made for this task.
| Version | Backend | Passed | Failed | Skipped | Build / test / guard exit |
|---|---|---:|---:|---:|---|
| candidate | SqlServer | 195 | 0 | 0 | 0 / 0 / 0 |
| candidate | QuestDb | 233 | 0 | 1 | 0 / 0 / 0 |
| baseline | SqlServer | 195 | 0 | 0 | 0 / 0 / 0 |
| baseline | QuestDb | 233 | 0 | 1 | 0 / 0 / 0 |
Every invocation was unfiltered, used a fresh SHA/project run directory and TRX, preserved the dotnet test exit status, and ran the repository MIN_EXECUTED=1 guard. Project builds were serial within each worktree; test invocations were serial against the same isolated fixture endpoints. Exact commands, UTC start/end, SDK dependency versions and test/SDK binary SHA-256 hashes are in each result.json.
Candidate positive coverage includes, on each backend, 13 raw-read, 14 AtTime, 11 modified-value, 7 event-read, 12 update, 64 processed-read, 6 bounding-value and 4 quality-contract conformance cases, plus backend-specific schema, write, provisioning, retention (SQL), tombstone/index/TTL (QuestDB), and alarm-writer tests.
SqlServer: baseline/candidate display names and outcomes identical = True across 195 discovered cases. Baseline failures: 0; candidate failures: 0.
QuestDb: baseline/candidate display names and outcomes identical = True across 234 discovered cases. Baseline failures: 0; candidate failures: 0.
No PercentGood failure appeared in these selected integration suites. Their empty failure sets match where both versions have completed; there is no failing message pair to compare. This does not resolve or replace the separate standalone SDK-oracle PercentGood issue #796.
The existing QuestDB LiveWriteReadRaceStressTests.Per_sample_reads_of_a_tag_being_written_never_fault_and_never_return_a_valueless_non_bad_result case skips because HISTORIAN_QUESTDB_STRESS_SECONDS is unset. It was not enabled, shortened, reclassified, or replaced by a normal passing case. No five-million-row workload or two-hour stress/soak was run. Those gates remain pending.
## Fixture ownership and exact images
SQL Server: existing otopcua-dev-sql-1, endpoint 127.0.0.1:14330. Its documented local dev credential was read from docs/Historian.md and supplied only via environment; the password and full secret-bearing connection string were not printed or included in the report. SqlHistorianFixture was verified before execution: it forces master only for CREATE/DROP, generates OtOpcUaHistorian_T_<GUID>, directs all test schemas into that new database, clears its pools, and drops only its generated name. The fixture source is unchanged between baseline and candidate. Five preexisting OtOpcUaHistorian_T_GUID databases were recorded and were never dropped by this task.
SQL container ID: `74ea75e5904c647f18e89b8b14c9ac385d7149c0484fbf13f6c16ec00cf3e24f`; image ID: `sha256:a2fbff321505724b8a271a214d4dec020a8e7a745c83eede0b9d762b51a9ab34`.
QuestDB: new task-owned otopcua-sdk2-questdb, loopback-only PG port 32770 and HTTP port 32771, new storage volume `otopcua-sdk2-questdb-data-20261110t170141z`. No existing QuestDB service or volume was used. Connection settings use ServerCompatibilityMode=NoTypeLoading and Maximum Pool Size=16; the fixture applies its existing one-second idle pruning. QuestDbFixture assigns unique t<12 hex>_ prefixes, validates them, locates owned tables with starts_with (not wildcard LIKE), and drops only those prefixes. The same unchanged fixture source and image were used by both versions.
QuestDB image: `questdb/questdb:10.0.1`; immutable image ID `sha256:41ceca901724ad10a0035da0cb0c1f99f88bf3692abdc9090baabdfff1d215f9`; digest `questdb/questdb@sha256:67eaed863ebb2383227919ea9a5499a4a39ceca3862c20b974c84a10e11cdf89`.
## Run evidence
| Version / backend | UTC start | UTC end | Directory |
|---|---|---|---|
| candidate SqlServer | 2026-11-10T17:03:02.101289+00:00 | 2026-11-10T17:03:52.950418+00:00 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/backend-matrix-20261110T170141Z/SqlServer` |
| candidate QuestDb | 2026-11-10T17:03:58.392642+00:00 | 2026-11-10T17:11:13.275488+00:00 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/backend-matrix-20261110T170141Z/QuestDb` |
| baseline SqlServer | 2026-11-10T17:11:19.005717+00:00 | 2026-11-10T17:12:11.817250+00:00 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/backend-matrix-20261110T170141Z/SqlServer` |
| baseline QuestDb | 2026-11-10T17:12:16.867323+00:00 | 2026-11-10T17:19:37.746023+00:00 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/backend-matrix-20261110T170141Z/QuestDb` |
Session index / fixture metadata / aggregate results / exact failure-set comparison:
/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/backend-matrix-20261110T170141Z
Each project directory contains build.log, test.log, a fresh TRX, guard.log and result.json. Candidate directories also contain class-counters.json. The session has fixture-metadata.json and readiness/WAL snapshots; cleanup.json and questdb-container.log are written after final cleanup.
## Cleanup
Completed at 2026-11-10T17:19:44.751608+00:00. SQL GUID-database inventory unchanged: True; SQL container ID unchanged: True. QuestDB tables remaining after all tests: 0. Task-owned container absent: True; task-owned volume absent: True. Only the captured task container ID and its verified task-labelled volume were removed. No other container, volume or database was removed.
## Reproduction and remaining gates
Completed scripts (kept in /tmp; they create new timestamped evidence directories and protect ownership):
```bash
python3 /tmp/opcua2-backend-setup.py
python3 /tmp/opcua2-backend-matrix.py
python3 /tmp/opcua2-backend-cleanup.py
python3 /tmp/opcua2-backend-report.py
```
For each version/backend the actual commands were dotnet build <owning-project.csproj> -c Release, then dotnet test <owning-project.csproj> -c Release --no-build --no-restore --logger trx --results-directory <fresh-directory>, with only the appropriate real connection environment variable set. Guard: MIN_EXECUTED=1 bash scripts/ci/assert-not-all-skipped.sh <fresh-directory>/*.trx. The orchestration preserves nonzero test and guard statuses instead of overwriting them. SQL connection acquisition is from the documented local dev recipe without echo; QuestDB endpoints are obtained from the newly created task container.
AVEVA/HistorianGateway live read/write/alarm/provisioning and licensed backend validation remain external and pending. The separate SDK-oracle issue #796, wire/security interoperability, five-million-row workload, two-hour stress/soak and release/performance gates are not completed by these local backend runs.
docs/plans/evidence/2026-11-10-opcua2/lock-benchmarks.md
-60
View File
@@ -1,60 +0,0 @@
# OPC UA SDK lock benchmark diagnostic — 2026-11-10
Compared baseline `83ed880e4471344deaf008da9bd028c7e170bdc7` (SDK 1.5.378.182) with candidate `a8e1082d69a112ea0e16788f023067979698527a` (SDK 2.0.1). Both Release project builds passed with 0 warnings/errors. No source edits or new commits were made for this evidence task.
Same Mac16,7, 14 logical/physical CPU cores, 48 GiB RAM, .NET SDK 10.0.403. OS details and exact assembly SHA-256 hashes are in metadata.json. All runs used OTOPCUA_BENCH=1 and the existing NodeManagerLockTimeBenchmarkTests class. Scenario sizes and assertions are identical; the only benchmark source differences are SDK-required OperationContext.RequestLifetime.None and IAsyncNodeManager interface changes. Each test boots a private server with its own temporary PKI and probed port. No containers or global fixture were used.
One warmup invocation per version was followed by three recorded invocations per version, alternating baseline/candidate with 25 seconds between processes. Each invocation is a fresh process; warmup primes host/file caches, not persistent in-process JIT state. Default xUnit test order varied. Warmups are excluded from the medians.
| Recorded metric | Baseline runs | SDK 2 runs | Baseline median | SDK 2 median | Change |
|---|---|---|---:|---:|---:|
| Rebuild 10k-tag folder | 18, 15, 17 | 15, 15, 14 | 17 ms | 15 ms | -11.76% |
| Remove 200 equipment / 50k nodes | 81, 50, 86 | 80, 68, 80 | 81 ms | 80 ms | -1.23% |
| Visits for 10 removals / 50 conditions | 10, 10, 10 | 10, 10, 10 | 10 visits | 10 visits | +0.00% |
| OS command maximum resident set size | 385.17, 136.20, 136.30 | 136.39, 136.81, 136.80 | 136.30 MiB | 136.80 MiB | +0.37% |
The median teardown timings do not exceed the provisional +15% latency-growth reference. Their inverse-time local teardown-rate proxies change by +13.33% (rebuild) and +1.25% (removal), so neither shows a >10% drop in this subset. These derived rates are not wire throughput, and three samples do not establish p95 latency. Command maximum-RSS median rose 0.37%, below the +15% reference, but it does not measure steady server RSS. No change to budgets or acceptance decision is implied.
All eight invocations passed all three cases: 24 passed, 0 failed, 0 skipped (18 recorded case executions plus 6 warmup). Every test exit and MIN_EXECUTED=3 guard exit was 0; aggregate orchestrator exit was 0. The original 5000 ms opt-in teardown assertions remained enforced; the largest recorded operation was 86 ms. The strict, hardware-independent visit counter was exactly 10 in all eight invocations.
Host-load caveat: other solution testing/build activity shared this host. Initial preparation saw load averages 19.78/30.02/23.48 on 14 cores; per-run before/after uptime snapshots are retained in result.json. Recorded one-minute load ranged 5.58–10.90; five-minute load ranged 15.26–20.32 and fifteen-minute load 18.45–20.56. BENCH uses integer-millisecond Stopwatch output. Timing spread, quantization and changing xUnit ordering preclude a stable performance claim. OS /usr/bin/time -l maximum RSS is command-invocation rusage, not isolated teardown allocation or steady server RSS; baseline run1 is an outlier and is included unchanged. GC/allocation counters are not exposed by the existing harness and were not measured. This subset does not complete wire throughput, p50/p95/p99 latency, allocation, steady RSS, history-page throughput, load, soak, publish, or resource-lifecycle gates.
## Recorded invocations
| Version / phase | UTC start | UTC end | Test / guard exit | Evidence directory |
|---|---|---|---|---|
| baseline warmup | 2026-11-10T16:54:37.980941+00:00 | 2026-11-10T16:54:42.012296+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/warmup` |
| candidate warmup | 2026-11-10T16:55:07.020267+00:00 | 2026-11-10T16:55:11.610491+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/warmup` |
| baseline run1 | 2026-11-10T16:55:36.615335+00:00 | 2026-11-10T16:55:40.023724+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/run1` |
| candidate run1 | 2026-11-10T16:56:05.032437+00:00 | 2026-11-10T16:56:09.352092+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/run1` |
| baseline run2 | 2026-11-10T16:56:34.361326+00:00 | 2026-11-10T16:56:37.712286+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/run2` |
| candidate run2 | 2026-11-10T16:57:02.720388+00:00 | 2026-11-10T16:57:06.894343+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/run2` |
| baseline run3 | 2026-11-10T16:57:31.903230+00:00 | 2026-11-10T16:57:35.306953+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/run3` |
| candidate run3 | 2026-11-10T16:58:00.315244+00:00 | 2026-11-10T16:58:04.720985+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/run3` |
Each directory contains a fresh TRX (BENCH stdout attached to individual test results), test.log, resource.log, guard.log, and result.json. Session metadata/results/summary:
/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z
Reproduction (in each version worktree; choose a new, nonexistent absolute result directory each invocation):
```bash
OPC_RUN_DIR=/absolute/fresh/result/directory
test ! -e "$OPC_RUN_DIR" || exit 2
mkdir -p "$OPC_RUN_DIR" || exit
opc_test_status=0
OTOPCUA_BENCH=1 /usr/bin/time -l dotnet test \
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj \
-c Release --no-build --no-restore \
--filter FullyQualifiedName~NodeManagerLockTimeBenchmarkTests \
--logger trx --logger "console;verbosity=detailed" \
--results-directory "$OPC_RUN_DIR" \
> "$OPC_RUN_DIR/test.log" 2> "$OPC_RUN_DIR/resource.log" || opc_test_status=$?
opc_guard_status=0
MIN_EXECUTED=3 bash scripts/ci/assert-not-all-skipped.sh "$OPC_RUN_DIR"/*.trx \
> "$OPC_RUN_DIR/guard.log" 2>&1 || opc_guard_status=$?
if [ "$opc_test_status" -ne 0 ]; then exit "$opc_test_status"; fi
exit "$opc_guard_status"
```
Exact completed orchestration script: /tmp/opcua2-lock-bench.py. It records environment, SDK dependencies, assembly hashes, start/end UTC, host load, metrics, counts and statuses, and alternates the eight invocations with 25-second gaps.
docs/plans/evidence/2026-11-10-opcua2/publish-smoke.md
-70
View File
@@ -1,70 +0,0 @@
# Actual Host / CLI publish and runtime smoke — 2026-11-10
Linux ARM64 actual full Host start/stop smoke PASSED. Windows x64 cross-publish and static inspection PASSED. Windows runtime/service installation remains a formal pending operator gate; this report does not complete Gate G4.
Reproducible probe commit: `3a11087b37728b4a97c2884d55f2d7d9949923b9`. Only four new files under scripts/validation/opcua-publish-smoke were added. No production, existing documentation, package policy, fleet, sister-repository or issue changes were made by this lane. Root commits 2d75cecb and bd3143b7 were merged into the isolated checkout before the applicable publishes.
## Published artifacts
The repository install recipe documents dotnet publish -c Release -r win-x64 --self-contained. The four commands used that setting, with PublishSingleFile=false and PublishTrimmed=false explicitly retained, one MSBuild node and new output folders. The Linux RID matches the cached ARM64 container. No Native AOT was attempted. All four actual publish exits were zero.
| RID / app | Source revision at publish | Files / bytes | SDK assemblies | Native inventory entries |
|---|---|---:|---:|---:|
| linux-arm64/host | `d31560cd360144132f1433a04fcb5acb25dfa3fd` | 1388 / 356618849 | 8 | 32 |
| linux-arm64/cli | `8a1028c4e2f308a29bcc4a504596df6d667554f8` | 236 / 111341495 | 7 | 15 |
| win-x64/host | `8a1028c4e2f308a29bcc4a504596df6d667554f8` | 1390 / 342188283 | 8 | 37 |
| win-x64/cli | `8a1028c4e2f308a29bcc4a504596df6d667554f8` | 236 / 103181689 | 7 | 16 |
Linux Host was published before the root client-factory-only correction; it has no Client.Shared dependency. Both CLI publishes and Windows Host include the merged bd3143b7 correction. Host production source is unchanged across these two publish revisions. The exact per-artifact source, command and SHA-256 manifest are retained in validation-receipt.json.
Every OPCFoundation dependency is 2.0.1. Each SDK assembly basename appears once per application publish tree; no old SDK copy or migration analyzer runtime artifact was found. All inspected Linux native entries are AArch64 ELF and Windows entries are x86-64 PE; apphosts and bundled runtimes match their RID. Both Host publishes contain the separately self-contained script-worker. Runtimeconfig includedFrameworks records Microsoft.NETCore.App 10.0.14, plus Microsoft.AspNetCore.App 10.0.14 on Host. This validates packaging and the in-process default only; the out-of-process script execution path was not exercised here.
| Artifact | Apphost SHA-256 | Full-file-manifest SHA-256 |
|---|---|---|
| linux-arm64/host | `d97f318f268246852603166a19d66c2da788a75a7f9124b2242e6c794985dcbb` | `32ec603b22ba7f7df62904bc738e9193e0c1275fcc7a9fb5c65b9aa5d65c0f5e` |
| linux-arm64/cli | `b4840db34c07ec7f0fa8cbc220c5552fac87527d3b222c4e47260c68c7eedb19` | `992bc65bcf74c0d57aa6d789085abdfcd2290b0496931a445a11e801de2312a6` |
| win-x64/host | `7896ec7a0db88dd00d893ea9a55bc22e40849e7c789af7eea79a4514ed587972` | `b42baf99acbbe0646c487fbaf193f9f4fea66adfd393a41b841b55ba95d53aa0` |
| win-x64/cli | `d880d4c83448f9ee5a2ea79c3573c1b2301b5fc17c277cf072153a589543bc5d` | `28a9872354539d600659c4af463e24c7ba40136c4ab67a5cb40c36b5d9575060` |
Complete paths, sizes and SHA-256 values are in the four *-sha256.json manifests. Managed SDK/package inventory is in publish-inventory.json; native paths, file-format/architecture and hashes are in native-inventory.json. Runtime-generated logs and private PKI/state are excluded from shipped-artifact manifests.
## Actual Linux runtime
Image: mcr.microsoft.com/dotnet/sdk:10.0.103-noble; cached immutable ID `sha256:b04611ee9e1b7d36797012ed921962c57189cb4dbb3dff12afb8e158110f293b`, architecture `arm64`, digest `mcr.microsoft.com/dotnet/sdk@sha256:e362a8dbcd691522456da26a5198b8f3ca1d7641c95624fadc5e3e82678bd08a`. The published self-contained OtOpcUa.Host executable was the container command; the SDK image supplies only the portable probe framework. No ProcessHost server substitute was used.
The real Host ran Cluster:Roles=driver, ConfigSource:Mode=FetchAndCache, a self-only otopcua Akka seed, and an unreachable loopback CentralFetchEndpoint. No ConfigDb was registered. A task-only LocalDb deployment cache was seeded through public production AddOtOpcUaLocalDb / IDeploymentArtifactCache APIs, using the public DeploymentId formatter. The artifact contains one raw folder and a real peppered HMAC API key with read/browse/subscribe grants. Production API-key authentication was used. PKI, LocalDb and encrypted secrets SQLite lived in one fresh GUID state directory. Random secrets were environment-only; none are included in this report or recorded arguments.
All preparation and runtime containers used --network none and task labels. HTTP, UA and Akka ports were randomly chosen within that private namespace; none were published to the host. The same task-owned container, application executable, LocalDb and GUID PKI were started and stopped twice.
| Executed check | Actual result |
|---|---|
| /healthz, /health/ready and /health/active on each boot | 6 requests passed; ready/active Healthy and Akka Up |
| Strict trusted Basic256Sha256 SignAndEncrypt + real API-key session / ServerStatus.State read | 2 passed, Good quality, peer autoaccept false |
| Wrong secret for the same real key | 2 denied with BadIdentityTokenRejected |
| Untrusted server certificate | Denied with BadCertificateUntrusted |
| Untrusted client certificate | Denied with BadSecurityChecksFailed |
| Actual published CLI --help | Exit 0 |
| Actual published CLI encrypted API-key ServerStatus.State reads | 2 passed, Good quality, exit 0 |
| Actual Host stop / restart / stop | Both stops exit 0; Running=false, OOMKilled=false |
| Cache restore | Production restored-served-state log appears on both boots |
| Required execution guard | Exactly 2 boots and 6 strict SDK probe cases executed; overall exit 0 |
The CLI retains its existing AutoAcceptCertificates=true behavior and has no strict-validation switch. Its successful reads are separate from the strict SDK peer-validation checks. Its matching application-name/URI certificate was prepared through the public SDK API, then trusted by the server before first boot. A closed-port CLI setup preflight intentionally returned nonzero; it is not counted as a passing connection case.
Application certificate SHA-256 on BOTH boots: `19000d33d71bb11c92f282724a3515b9119783e070df06667b05f49e41413640`. Preserved thumbprint on BOTH boots: `sha1 Fingerprint=DE:CB:33:B0:DE:32:9B:FB:40:FF:A8:5C:66:48:CE:44:01:2A:48:F8`. The public certificate bytes and thumbprint were unchanged; no rotation was performed.
Raw process stdout/stderr and this run’s appended rolling script/application/selflog bytes were scanned before sanitized copies were saved. Exact task API-key/secret, pepper, fetch key and master-key matches: zero. Successful-run appended Serilog selflog: 0 bytes. Negative authentication and certificate results remained diagnosable. The private environment file was mode 0600 and removed at the end.
Final successful runtime directory: `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/publish-smoke/20261110T172336Z/runtime-8e794a04a90d4b769fd94cd330aba544`. summary.json, commands.json, health JSON, individual protocol/CLI logs, full Host logs and appended rolling-file slices contain the actual results, UTC times, statuses and exits. The task-labelled container `f725df5c31d907db6626a70446294006e9e18d6dd7efb3bc9f1f385267faf7f2` was removed after its second clean stop. No task containers remain; no existing fleet container, volume, database or PKI was modified.
Earlier fresh setup attempts exited nonzero and remain in sibling runtime-* directories. They exposed fixture mistakes: closed-port CLI certificate preparation, an unnecessary split cluster role, the liveness route spelling (/healthz is correct), and a hyphenated deployment identifier. Those were corrected only in the new smoke harness. The application’s configuration validator failed closed, and valid configuration subsequently passed. No SDK regression was attributed to those setup errors.
## Reproduction and pending operator gate
The committed scripts/validation/opcua-publish-smoke/README.md contains exact four-publish, probe-build and runtime commands. The runtime script requires a fresh private state directory per invocation, preserves each command exit, enforces the two-boot/six-case count and raw-log checks, and removes only its captured task-labelled container. Build/publish commands and immutable hashes for this completed run are in validation-receipt.json.
Evidence root: `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/publish-smoke/20261110T172336Z`.
**PENDING — Windows runtime / installation:** cross-publishing PE artifacts on macOS does not execute Windows service registration, startup, endpoint negotiation or shutdown. An operator must perform scripts/install/Install-Services.ps1 and the two-start/stop, preserved-PKI health/UA checks on an approved Windows environment. The Wonder-box remote-execution hold remains binding: no WinRM, psbridge or replacement remote-execution channel was used or is authorized by this smoke. No deployment was performed.
Performance/load, full driver/native transport operation, Windows service ACL/account behavior, backend/live AVEVA, failover and the two-hour lifecycle soak remain separate gates.
docs/plans/evidence/2026-11-10-opcua2/server-linux-soak-failed.json
-361
View File
@@ -1,361 +0,0 @@
{
"issue": "#799",
"status": "failed",
"scope": "In-process SDK client and production server/node-manager lifecycle; not standalone server attribution",
"source": {
"launcherSourceRevision": "38098ce142ab8c382825aa754e0758071dd3a46b",
"publishedSourceRevision": "74c938f37cbb9cf08b6ca72e4effc511120dc274",
"productionEquivalentTo": "7b353d7bae5311a2f0a33c0a82c88d2c5b49d76f",
"productionDiffEmpty": true,
"canonicalDescriptorProbeCommit": "642aefe3",
"testSourceSha256": "e695783e05787d94bf5d167b25b1c48b446b800c28259f023c28ed6012d737d1",
"publishCommand": "reused self-contained artifact: /private/tmp/opcua2-server-linux-diagnostic-20261110a/app",
"appPath": "/private/tmp/opcua2-server-linux-diagnostic-20261110a/app",
"apphost": {
"format": "ELF64 AArch64",
"sha256": "97531b8005df9dc8b129d2b22ba2049cf572d5d970046f7e8644ec632e860032"
},
"runtimeConfig": {
"runtimeOptions": {
"tfm": "net10.0",
"includedFrameworks": [
{
"name": "Microsoft.NETCore.App",
"version": "10.0.14"
},
{
"name": "Microsoft.AspNetCore.App",
"version": "10.0.14"
}
],
"configProperties": {
"System.Reflection.Metadata.MetadataUpdater.IsSupported": false,
"System.Reflection.NullabilityInfoContext.IsSupported": true,
"System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization": false
}
}
},
"sdkDependencies": [
"OPCFoundation.NetStandard.Opc.Ua.Client/2.0.1",
"OPCFoundation.NetStandard.Opc.Ua.Configuration/2.0.1",
"OPCFoundation.NetStandard.Opc.Ua.Core.Schema/2.0.1",
"OPCFoundation.NetStandard.Opc.Ua.Core.Types/2.0.1",
"OPCFoundation.NetStandard.Opc.Ua.Core/2.0.1",
"OPCFoundation.NetStandard.Opc.Ua.Security.Certificates/2.0.1",
"OPCFoundation.NetStandard.Opc.Ua.Server/2.0.1",
"OPCFoundation.NetStandard.Opc.Ua.Types/2.0.1"
],
"image": {
"Id": "sha256:b04611ee9e1b7d36797012ed921962c57189cb4dbb3dff12afb8e158110f293b",
"RepoDigests": [
"mcr.microsoft.com/dotnet/sdk@sha256:e362a8dbcd691522456da26a5198b8f3ca1d7641c95624fadc5e3e82678bd08a"
],
"Architecture": "arm64",
"Os": "linux"
},
"hostDotnet": ".NET SDK:\n Version: 10.0.403\n Commit: 5a2162bea6\n Workload version: 10.0.400-manifests.a00bebb6\n MSBuild version: 18.9.15+5a2162bea\n\nRuntime Environment:\n OS Name: Mac OS X\n OS Version: 26.5\n OS Platform: Darwin\n RID: osx-arm64\n Base Path: /opt/homebrew/Cellar/dotnet/10.0.403/libexec/sdk/10.0.403/\n\n.NET workloads installed:\nThere are no installed workloads to display.\nConfigured to use workload sets when installing new manifests.\nNo workload sets are installed. Run \"dotnet workload restore\" to install a workload set.\n\nHost:\n Version: 10.0.14\n Architecture: arm64\n Commit: 0427b9abbe\n\n.NET SDKs installed:\n 10.0.403 [/opt/homebrew/Cellar/dotnet/10.0.403/libexec/sdk]\n\n.NET runtimes installed:\n Microsoft.AspNetCore.App 10.0.14 [/opt/homebrew/Cellar/dotnet/10.0.403/libexec/shared/Microsoft.AspNetCore.App]\n Microsoft.NETCore.App 10.0.14 [/opt/homebrew/Cellar/dotnet/10.0.403/libexec/shared/Microsoft.NETCore.App]\n\nOther architectures found:\n None\n\nEnvironment variables:\n DOTNET_ROOT [/opt/homebrew/Cellar/dotnet/10.0.403/libexec]\n\nglobal.json file:\n /Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/global.json\n\nLearn more:\n https://aka.ms/dotnet/info\n\nDownload .NET:\n https://aka.ms/dotnet/download\n",
"network": "none",
"mode": "full",
"durationSeconds": 7200,
"startedUtc": "2026-11-10T18:10:21Z"
},
"launcherResult": {
"status": "failed",
"containerExitCode": 1,
"containerElapsedSeconds": 907.5196822080761,
"validationError": "TRX expected exact passed/executed count 1: /private/tmp/opcua2-server-linux-full-20261110/task/results/soak.trx"
},
"start": {
"kind": "start",
"utc": "2026-11-10T18:10:24.882222+00:00",
"processId": 192,
"mode": "full",
"requestedSeconds": 7200,
"minimumFullSeconds": 7200,
"build": "38098ce142ab8c382825aa754e0758071dd3a46b",
"sdk": "2.0.1+07e62f72500cc0923a2bab940653c8c1116fa286",
"diagnosticInstrumentation": "SDK2.0.1 CP lists and certificate-validation cached stores; read-only",
"limits": {
"retainedManagedGrowthBytes": 67108864,
"rssGrowthBytes": 134217728,
"fdGrowth": 32
},
"coverage": "encrypted sessions; scalar/Int16[] Raw+UNS; subscription removal; browse abandonment; history release/cross-session/close; surgical changes/rebuild; revocation; paused TCP read/recovery; same-PKI restart"
},
"failed": {
"kind": "failed",
"cycles": 248,
"restarts": 61,
"seconds": 903.5081762,
"exceptionType": "Shouldly.ShouldAssertException"
},
"test": {
"executionId": "b3f13597-f945-44bf-9134-cb0282f6455c",
"testId": "3a3639bf-5159-da35-0b1a-0a6c89f03e2c",
"testName": "ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.Sdk2LifecycleSoakTests.Repeated_lifecycles_return_resources_to_a_steady_baseline",
"computerName": "b0dfc9462250",
"duration": "00:15:04.9982391",
"startTime": "2026-11-10T18:10:23.4089949+00:00",
"endTime": "2026-11-10T18:25:28.4141750+00:00",
"testType": "13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b",
"outcome": "Failed",
"testListId": "8c84fa94-04c1-424b-9868-57a2d4851a1d",
"relativeResultsDirectory": "b3f13597-f945-44bf-9134-cb0282f6455c"
},
"error": "Shouldly.ShouldAssertException : 353931264L\r\n should be less than or equal to\r\n351850496L\r\n but was not",
"counters": {
"total": "1",
"executed": "1",
"passed": "0",
"failed": "1",
"error": "0",
"timeout": "0",
"aborted": "0",
"inconclusive": "0",
"passedButRunAborted": "0",
"notRunnable": "0",
"notExecuted": "0",
"disconnected": "0",
"warning": "0",
"completed": "0",
"inProgress": "0",
"pending": "0"
},
"samples": [
{
"Cycle": 1,
"Seconds": 2.6707062,
"Sessions": 0,
"Subscriptions": 0,
"Items": 0,
"BrowsePoints": 0,
"HistoryPoints": 0,
"LiveDataItems": 0,
"LiveEventItems": 0,
"MonitoredNodes": 0,
"Variables": 19,
"Folders": 4,
"Conditions": 1,
"CachedCertificateStores": 2,
"Descriptors": 187,
"PkiDescriptors": 0,
"ManagedBytes": 19829880,
"RssBytes": 212717568,
"AllocatedBytes": 45694176,
"Gen0": 3,
"Gen1": 2,
"Gen2": 2
},
{
"Cycle": 3,
"Seconds": 7.5077403,
"Sessions": 0,
"Subscriptions": 0,
"Items": 0,
"BrowsePoints": 0,
"HistoryPoints": 0,
"LiveDataItems": 0,
"LiveEventItems": 0,
"MonitoredNodes": 0,
"Variables": 19,
"Folders": 4,
"Conditions": 1,
"CachedCertificateStores": 2,
"Descriptors": 187,
"PkiDescriptors": 0,
"ManagedBytes": 20362800,
"RssBytes": 217632768,
"AllocatedBytes": 52216848,
"Gen0": 7,
"Gen1": 6,
"Gen2": 6
},
{
"Cycle": 25,
"Seconds": 90.9064734,
"Sessions": 0,
"Subscriptions": 0,
"Items": 0,
"BrowsePoints": 0,
"HistoryPoints": 0,
"LiveDataItems": 0,
"LiveEventItems": 0,
"MonitoredNodes": 0,
"Variables": 19,
"Folders": 4,
"Conditions": 1,
"CachedCertificateStores": 2,
"Descriptors": 189,
"PkiDescriptors": 0,
"ManagedBytes": 20360784,
"RssBytes": 269185024,
"AllocatedBytes": 349530024,
"Gen0": 57,
"Gen1": 50,
"Gen2": 50
},
{
"Cycle": 50,
"Seconds": 181.3146113,
"Sessions": 0,
"Subscriptions": 0,
"Items": 0,
"BrowsePoints": 0,
"HistoryPoints": 0,
"LiveDataItems": 0,
"LiveEventItems": 0,
"MonitoredNodes": 0,
"Variables": 19,
"Folders": 4,
"Conditions": 1,
"CachedCertificateStores": 2,
"Descriptors": 189,
"PkiDescriptors": 0,
"ManagedBytes": 20444424,
"RssBytes": 230686720,
"AllocatedBytes": 652625024,
"Gen0": 113,
"Gen1": 100,
"Gen2": 100
},
{
"Cycle": 100,
"Seconds": 361.5237015,
"Sessions": 0,
"Subscriptions": 0,
"Items": 0,
"BrowsePoints": 0,
"HistoryPoints": 0,
"LiveDataItems": 0,
"LiveEventItems": 0,
"MonitoredNodes": 0,
"Variables": 19,
"Folders": 4,
"Conditions": 1,
"CachedCertificateStores": 2,
"Descriptors": 190,
"PkiDescriptors": 0,
"ManagedBytes": 20637848,
"RssBytes": 229531648,
"AllocatedBytes": 1261146560,
"Gen0": 225,
"Gen1": 200,
"Gen2": 200
},
{
"Cycle": 150,
"Seconds": 546.7687542,
"Sessions": 0,
"Subscriptions": 0,
"Items": 0,
"BrowsePoints": 0,
"HistoryPoints": 0,
"LiveDataItems": 0,
"LiveEventItems": 0,
"MonitoredNodes": 0,
"Variables": 19,
"Folders": 4,
"Conditions": 1,
"CachedCertificateStores": 2,
"Descriptors": 189,
"PkiDescriptors": 0,
"ManagedBytes": 20388720,
"RssBytes": 281825280,
"AllocatedBytes": 1907065880,
"Gen0": 338,
"Gen1": 300,
"Gen2": 300
},
{
"Cycle": 200,
"Seconds": 727.6899225,
"Sessions": 0,
"Subscriptions": 0,
"Items": 0,
"BrowsePoints": 0,
"HistoryPoints": 0,
"LiveDataItems": 0,
"LiveEventItems": 0,
"MonitoredNodes": 0,
"Variables": 19,
"Folders": 4,
"Conditions": 1,
"CachedCertificateStores": 2,
"Descriptors": 189,
"PkiDescriptors": 0,
"ManagedBytes": 20678608,
"RssBytes": 269180928,
"AllocatedBytes": 2516228704,
"Gen0": 450,
"Gen1": 400,
"Gen2": 400
},
{
"Cycle": 245,
"Seconds": 896.2277082,
"Sessions": 0,
"Subscriptions": 0,
"Items": 0,
"BrowsePoints": 0,
"HistoryPoints": 0,
"LiveDataItems": 0,
"LiveEventItems": 0,
"MonitoredNodes": 0,
"Variables": 19,
"Folders": 4,
"Conditions": 1,
"CachedCertificateStores": 2,
"Descriptors": 189,
"PkiDescriptors": 0,
"ManagedBytes": 20292328,
"RssBytes": 341528576,
"AllocatedBytes": 3109239976,
"Gen0": 552,
"Gen1": 490,
"Gen2": 490
},
{
"Cycle": 247,
"Seconds": 901.025049,
"Sessions": 0,
"Subscriptions": 0,
"Items": 0,
"BrowsePoints": 0,
"HistoryPoints": 0,
"LiveDataItems": 0,
"LiveEventItems": 0,
"MonitoredNodes": 0,
"Variables": 19,
"Folders": 4,
"Conditions": 1,
"CachedCertificateStores": 2,
"Descriptors": 189,
"PkiDescriptors": 0,
"ManagedBytes": 20547624,
"RssBytes": 349171712,
"AllocatedBytes": 3115447904,
"Gen0": 556,
"Gen1": 494,
"Gen2": 494
}
],
"maximumRecordedQuiescent": {
"ManagedBytes": 20890032,
"RssBytes": 349171712,
"Descriptors": 191,
"PkiDescriptors": 0
},
"evidenceSha256": {
"result.json": "cb3d4dc9c3dcd80392e66f21d6b6e36907d8438c550a1723dced57a0968e7e89",
"source.json": "5bd909d1f289d38039b996ed110b450cd3914a6a69d96c7f3aa83b016629497d",
"task/soak.jsonl": "1d24d32382b5be6ebca2615389929818dd6c9874b7787cb91482ceaa5d3b9cd3",
"task/results/soak.trx": "798c97c3158ec35d9dbae754e98f57a7bf403ae10b81473f98a52a89c2401265",
"task/results/descriptor-controls.trx": "99ca030ff7542e2dc43959d8cf9a2f129352db9c6a44d5d011c6e10fa9327f01"
},
"limitations": [
"No complete row; no two-hour pass.",
"Failed sample248 RSS is reported in TRX before JSONL quiescent emission.",
"No resource budget or production workaround changed.",
"Private PKI and full task output are deliberately excluded."
]
}
docs/plans/evidence/2026-11-10-opcua2/solution-tests2.json
-1403
View File
File diff suppressed because it is too large Load Diff
docs/security.md
+6 -7
View File
@@ -301,13 +301,12 @@ guards in front of the store, bound from the same `Security:ApiKeys` section
**Per-client accounting and its limits (#751).** `IOpcUaUserAuthenticator` has an endpoint-carrying
`AuthenticateUserNameAsync(username, password, remoteEndpoint, ct)` overload (default: ignore the endpoint
and delegate), which the server calls and `ApiKeyOpcUaUserAuthenticator` forwards to
`IApiKeyStore.FindAsync(..., remoteAddress, ...)`. SDK 2.0.1 supplies the public transport-observed
`SecureChannelContext.PeerAddress` on session activation. `OtOpcUaSdkServer` scopes it to the current
asynchronous activation and restores the prior context in `finally`; `OpcUaApplicationHost` adapts it to
the existing IP-budget seam with port zero (no peer port is exposed). The former private-channel-map
reflection helper is removed. An unknown address is charged to the process-wide bucket only, which
therefore stays as the backstop. `ApiKeyMissBudgetPerEndpointTests` (OpcUaServer.IntegrationTests)
pins per-client accounting over the wire; concurrent IPv4/IPv6 activation probes check peer isolation. At most `BoundedApiKeyStore.MaxTrackedEndpoints`
`IApiKeyStore.FindAsync(..., remoteAddress, ...)`. The OPC UA SDK (1.5.378) exposes no client address on any
public surface, so `SecureChannelRemoteEndpoint` reads it from the TCP listener's private channel map by
reflection, keyed by the session's secure channel id. It is fail-soft: an unresolvable address (an SDK upgrade
that renamed the members, or a client re-activating over a new channel) is charged to the process-wide bucket
only, which therefore stays as the backstop. `ApiKeyMissBudgetPerEndpointTests` (OpcUaServer.IntegrationTests)
pins the resolution over the wire so such an upgrade fails a test. At most `BoundedApiKeyStore.MaxTrackedEndpoints`
(4096) addresses are tracked; when full, idle (refilled) buckets are pruned and a new address beyond that is
charged process-wide only — never handed a fresh budget. Clients behind one NAT address share a bucket. Neither
guard applies to a driver-only node: its `ArtifactApiKeyStore` is two in-memory dictionary probes and is never
scripts/validation/historiangw-131/README.md
-56
View File
@@ -1,56 +0,0 @@
# HistorianGateway consumer TLS migration — issue 131
`consumer-1.0.0.patch` is a future consumer change, kept separate from the buildable
application while the paired Client/Contracts 1.0.0 packages are being released.
The application in this checkout remains pinned to published 0.4.0.
The patch changes both central package pins to 1.0.0 and maps the new option as:
```csharp
AllowUntrustedServerCertificate = options.UseTls && options.AllowUntrustedServerCertificate,
```
The old inverse-mapping and obsolete h2c-history comments are removed. The existing
h2c construction test now includes a stale opt-out as well as a stale CA path.
Three new bounded local wire tests verify behavior through the real adapter and
published client API, without reflection or replacing the transport handler:
- TLS with normal defaults rejects a self-signed server before the metadata RPC.
- Explicit opt-out permits that self-signed server shape and executes one RPC.
- Plaintext with stale opt-out and CA settings executes one RPC, proving TLS-only
settings were not forwarded to the client's validating options.
The local server uses an ephemeral loopback port and a fresh short-lived
self-signed certificate. RPC completion is bounded to 15 seconds; the consumer
call timeout is two seconds. No gateway, deployment, or global trust store is used.
Before applying the patch, query the live Gitea NuGet version indexes for BOTH
packages and require actual stable 1.0.0. Source `main` having the new API is not a
package release. At initial inspection, both feed indexes listed exactly
0.4.0, 0.3.0, 0.2.0, 0.1.0, and the new API was only in Unreleased at HG commit
`e4321915ebc5e0704a68ecad21b85c1cc1c80d6b`.
```bash
git apply --check scripts/validation/historiangw-131/consumer-1.0.0.patch
git apply scripts/validation/historiangw-131/consumer-1.0.0.patch
# From the consumer checkout, using a fresh run/results directory:
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests \
-c Release --logger trx --results-directory /absolute/fresh/results
# Preserve dotnet test's exit; run the repository execution-count guard separately.
MIN_EXECUTED=1 bash scripts/ci/assert-not-all-skipped.sh /absolute/fresh/results/*.trx
```
Published-package verification must use the real 1.0.0 packages, preserve all test
and guard exits, run the full owning project, and complete the consumer solution
build/review before issue 131 is closed. The patch is transport-only and can be
applied independently of the OPC UA SDK 2 rollout.
If release availability lags, validate only in an archived staging copy: materialize
exact HG commit e432191 with `git archive`, pack both packages with the local-only
version `1.0.0-validation.131`, set an isolated `NUGET_PACKAGES`, and map only those
two package IDs to a task-local feed. Apply the future patch to a separate consumer
archive, then replace ONLY its two package pins with the validation prerelease.
Never pack/build the user's HG checkout, publish these validation packages, place
them in the normal NuGet cache, or pass prerelease validation off as stable-release
verification. Local validation and release availability results are recorded in
`RESULTS.md` when the run completes.
scripts/validation/historiangw-131/RESULTS.md
-13
View File
@@ -1,13 +0,0 @@
# Validation status
Preparation only: future patch passes `git apply --check` in the SDK 2 candidate
checkout. Published Client/Contracts 1.0.0 package verification is pending.
Initial primary feed query returned versions 0.4.0, 0.3.0, 0.2.0, 0.1.0 for both
packages. HG e4321915ebc5e0704a68ecad21b85c1cc1c80d6b contains the new option under
Unreleased; both source package version declarations still read 0.4.0. The cached
published Client 0.4.0 exposes RequireCertificateValidation and does not expose the
new property. Current consumer files remain on 0.4.0 and unchanged.
No issue comment/close, package publication, or HG user-checkout modification has
been performed. Test results will be added after actual execution.
scripts/validation/historiangw-131/consumer-1.0.0.patch
-226
View File
@@ -1,226 +0,0 @@
diff --git a/Directory.Packages.props b/Directory.Packages.props
--- a/Directory.Packages.props
+++ b/Directory.Packages.props
@@ -242,7 +242,7 @@
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Theme" Version="0.5.0" />
- <PackageVersion Include="ZB.MOM.WW.HistorianGateway.Client" Version="0.4.0" />
- <PackageVersion Include="ZB.MOM.WW.HistorianGateway.Contracts" Version="0.4.0" />
+ <PackageVersion Include="ZB.MOM.WW.HistorianGateway.Client" Version="1.0.0" />
+ <PackageVersion Include="ZB.MOM.WW.HistorianGateway.Contracts" Version="1.0.0" />
</ItemGroup>
</Project>
\ No newline at end of file
diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs
--- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs
+++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs
@@ -57,23 +57,15 @@
$"ServerHistorian:Endpoint must be an absolute http(s) URI (e.g. https://host:5222); got '{options.Endpoint}'.");
}
- // TLS-only options must stay at their defaults on a plaintext h2c connection: the client rejects
- // each of them outright when UseTls=false ("<name> is a TLS-only option and requires
- // UseTls=true"). Forwarding them unconditionally made h2c UNREACHABLE — AllowUntrustedServer-
- // Certificate defaults to false, so RequireCertificateValidation was always sent as true and
- // every http:// deployment crashed at startup, even though the scheme is documented as the
- // supported way to select h2c. There is no certificate to have a posture about here.
+ // Certificate settings apply only to TLS. The client rejects an explicit opt-out or a
+ // CA path on h2c, where there is no certificate to validate.
var clientOptions = new HistorianGatewayClientOptions
{
Endpoint = endpointUri,
ApiKey = options.ApiKey,
UseTls = options.UseTls,
CaCertificatePath = options.UseTls ? options.CaCertificatePath : null,
- // INVERTED mapping (TLS only): ServerHistorianOptions.AllowUntrustedServerCertificate
- // (opt-in to accept a self-signed cert) is the negation of the client's
- // RequireCertificateValidation. Allowing an untrusted cert == not requiring validation; a
- // pinned CaCertificatePath always verifies.
- RequireCertificateValidation = options.UseTls && !options.AllowUntrustedServerCertificate,
+ AllowUntrustedServerCertificate = options.UseTls && options.AllowUntrustedServerCertificate,
DefaultCallTimeout = options.CallTimeout,
LoggerFactory = loggerFactory,
};
diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/HistorianGatewayClientAdapterTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/HistorianGatewayClientAdapterTests.cs
--- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/HistorianGatewayClientAdapterTests.cs
+++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/HistorianGatewayClientAdapterTests.cs
@@ -27,23 +27,6 @@
/// A plaintext <c>h2c</c> gateway (an <c>http://</c> endpoint with <c>UseTls=false</c>) must
/// construct, using nothing but documented defaults.
/// </summary>
- /// <remarks>
- /// <para>
- /// Found by the LocalDb Phase 2 live gate, on the third consecutive crash-loop of the rig.
- /// <c>docs/Historian.md</c> and CLAUDE.md both document <c>http://</c> as a supported
- /// transport ("scheme selects transport: https = TLS, http = h2c"), but it was in fact
- /// <b>unreachable</b>: this factory forwarded the TLS-only options unconditionally, and
- /// <see cref="ServerHistorianOptions.AllowUntrustedServerCertificate"/> defaults to
- /// <see langword="false"/>, so it always sent <c>RequireCertificateValidation=true</c> —
- /// which the client rejects outright when <c>UseTls=false</c>
- /// ("RequireCertificateValidation is a TLS-only option and requires UseTls=true").
- /// </para>
- /// <para>
- /// So <b>every</b> h2c deployment crashed at startup, and the only way to avoid it was to
- /// set <c>AllowUntrustedServerCertificate=true</c> — asserting a certificate posture for a
- /// connection that has no certificate at all.
- /// </para>
- /// </remarks>
[Fact]
public void Adapter_constructs_for_plaintext_h2c_endpoint()
{
@@ -71,6 +54,7 @@
ApiKey = "histgw_x_y",
UseTls = false,
CaCertificatePath = "/etc/ssl/certs/leftover-from-a-tls-config.pem",
+ AllowUntrustedServerCertificate = true,
};
using var adapter = HistorianGatewayClientAdapter.Create(opts, NullLoggerFactory.Instance);
diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/HistorianGatewayTlsPolicyWireTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/HistorianGatewayTlsPolicyWireTests.cs
new file mode 100644
--- /dev/null
+++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/HistorianGatewayTlsPolicyWireTests.cs
@@ -0,0 +1,142 @@
+using System.Net;
+using System.Security.Authentication;
+using System.Security.Cryptography;
+using System.Security.Cryptography.X509Certificates;
+using Grpc.Core;
+using Microsoft.AspNetCore.Builder;
+using Microsoft.AspNetCore.Hosting;
+using Microsoft.AspNetCore.Server.Kestrel.Core;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Logging.Abstractions;
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.HistorianGateway.Client;
+using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
+using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
+
+namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
+
+/// <summary>historiangw#131: verify the mapped TLS posture through an actual local gRPC connection.</summary>
+public sealed class HistorianGatewayTlsPolicyWireTests(ITestOutputHelper output)
+{
+ private static CancellationToken Ct => TestContext.Current.CancellationToken;
+
+ [Fact]
+ public async Task Default_tls_rejects_an_untrusted_server_before_the_rpc()
+ {
+ await using var server = await Server.StartAsync(tls: true);
+ using var adapter = HistorianGatewayClientAdapter.Create(server.Options(), NullLoggerFactory.Instance);
+
+ var error = await Should.ThrowAsync<HistorianGatewayUnavailableException>(
+ () => adapter.TagExistsAsync("T", Ct).WaitAsync(TimeSpan.FromSeconds(15), Ct));
+ output.WriteLine($"TLS rejection: {error}");
+ ContainsTlsAuthenticationFailure(error).ShouldBeTrue("the failure must be TLS authentication, not an arbitrary connection failure");
+ server.MetadataCalls.ShouldBe(0);
+ }
+
+ [Fact]
+ public async Task Explicit_tls_opt_out_reaches_the_same_self_signed_server_shape()
+ {
+ await using var server = await Server.StartAsync(tls: true);
+ var options = server.Options(allowUntrusted: true);
+ using var adapter = HistorianGatewayClientAdapter.Create(options, NullLoggerFactory.Instance);
+
+ (await adapter.TagExistsAsync("T", Ct).WaitAsync(TimeSpan.FromSeconds(15), Ct)).ShouldBeTrue();
+ server.MetadataCalls.ShouldBe(1);
+ }
+
+ [Fact]
+ public async Task Plaintext_ignores_stale_tls_opt_out_and_ca_settings_and_reaches_the_rpc()
+ {
+ await using var server = await Server.StartAsync(tls: false);
+ var options = server.Options(allowUntrusted: true, caPath: "/nonexistent/historiangw-131-stale-ca.pem");
+ using var adapter = HistorianGatewayClientAdapter.Create(options, NullLoggerFactory.Instance);
+
+ (await adapter.TagExistsAsync("T", Ct).WaitAsync(TimeSpan.FromSeconds(15), Ct)).ShouldBeTrue();
+ server.MetadataCalls.ShouldBe(1);
+ }
+
+ private static bool ContainsTlsAuthenticationFailure(Exception error)
+ {
+ var pending = new Stack<Exception>();
+ var seen = new HashSet<Exception>(ReferenceEqualityComparer.Instance);
+ pending.Push(error);
+ while (pending.TryPop(out var current))
+ {
+ if (!seen.Add(current)) continue;
+ if (current is AuthenticationException) return true;
+ if (current.InnerException is { } inner) pending.Push(inner);
+ if (current is RpcException { Status.DebugException: { } debug }) pending.Push(debug);
+ }
+ return false;
+ }
+
+ private sealed class TagsService : HistorianTags.HistorianTagsBase
+ {
+ public int Calls;
+ public override Task<GetTagMetadataReply> GetTagMetadata(GetTagMetadataRequest request, ServerCallContext context)
+ {
+ Interlocked.Increment(ref Calls);
+ return Task.FromResult(new GetTagMetadataReply
+ { Metadata = new HistorianTagMetadata { Name = request.Tag } });
+ }
+ }
+
+ private sealed class Server(WebApplication app, TagsService service, string endpoint,
+ RSA? key, X509Certificate2? certificate) : IAsyncDisposable
+ {
+ public int MetadataCalls => Volatile.Read(ref service.Calls);
+ public ServerHistorianOptions Options(bool allowUntrusted = false, string? caPath = null) => new()
+ {
+ Endpoint = endpoint, ApiKey = "histgw_131_test", UseTls = certificate is not null,
+ CallTimeout = TimeSpan.FromSeconds(2),
+ AllowUntrustedServerCertificate = allowUntrusted, CaCertificatePath = caPath,
+ };
+
+ public static async Task<Server> StartAsync(bool tls)
+ {
+ RSA? key = null;
+ X509Certificate2? certificate = null;
+ if (tls)
+ {
+ key = RSA.Create(2048);
+ var request = new CertificateRequest("CN=localhost", key, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
+ request.CertificateExtensions.Add(new X509BasicConstraintsExtension(false, false, 0, true));
+ request.CertificateExtensions.Add(new X509KeyUsageExtension(
+ X509KeyUsageFlags.DigitalSignature | X509KeyUsageFlags.KeyEncipherment, true));
+ var names = new SubjectAlternativeNameBuilder();
+ names.AddDnsName("localhost"); names.AddIpAddress(IPAddress.Loopback);
+ request.CertificateExtensions.Add(names.Build());
+ certificate = request.CreateSelfSigned(DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddDays(1));
+ }
+ var service = new TagsService();
+ var builder = WebApplication.CreateBuilder();
+ builder.Logging.ClearProviders();
+ builder.WebHost.ConfigureKestrel(options => options.Listen(IPAddress.Loopback, 0, listen =>
+ {
+ listen.Protocols = HttpProtocols.Http2;
+ if (certificate is not null) listen.UseHttps(certificate);
+ }));
+ builder.Services.AddGrpc(); builder.Services.AddSingleton(service);
+ var app = builder.Build();
+ app.MapGrpcService<TagsService>();
+ try
+ {
+ await app.StartAsync(Ct);
+ return new Server(app, service, app.Urls.Single(), key, certificate);
+ }
+ catch
+ {
+ await app.DisposeAsync(); certificate?.Dispose(); key?.Dispose();
+ throw;
+ }
+ }
+
+ public async ValueTask DisposeAsync()
+ {
+ try { await app.StopAsync(Ct); }
+ finally { await app.DisposeAsync(); certificate?.Dispose(); key?.Dispose(); }
+ }
+ }
+}
scripts/validation/opcua-lifecycle-memory/.gitignore
-7
View File
@@ -1,7 +0,0 @@
__pycache__/
bin/
obj/
private-fixture/
*.pfx
*.der
*.nettrace
scripts/validation/opcua-lifecycle-memory/ArtifactModuleInfo.cs
-26
View File
@@ -1,26 +0,0 @@
// Read-only managed metadata extraction; does not load or execute product assemblies.
using System.Reflection.Metadata;
using System.Reflection.PortableExecutable;
using System.Text.Json;
var root = Path.GetFullPath(args[0]);
var modules = new List<object>();
foreach (var directory in args.Skip(1))
{
foreach (var path in Directory.EnumerateFiles(directory, "*.dll", SearchOption.AllDirectories).Order())
{
try
{
using var file = File.OpenRead(path);
using var pe = new PEReader(file);
if (!pe.HasMetadata) continue;
var metadata = pe.GetMetadataReader();
if (!metadata.IsAssembly) continue;
var assembly = metadata.GetAssemblyDefinition();
modules.Add(new { file = Path.GetRelativePath(root, path).Replace(Path.DirectorySeparatorChar, '/'),
assembly = metadata.GetString(assembly.Name), version = assembly.Version.ToString(),
mvid = metadata.GetGuid(metadata.GetModuleDefinition().Mvid).ToString("D") });
}
catch (BadImageFormatException) { /* Native DLLs have no CLR module identity. */ }
}
}
Console.WriteLine(JsonSerializer.Serialize(modules));
scripts/validation/opcua-lifecycle-memory/CertificateControl.cs
-39
View File
@@ -1,39 +0,0 @@
using System.Diagnostics;
using System.Runtime;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using System.Text.Json;
using Opc.Ua.Security.Certificates;
var mode = args[0];
var directory = args[1];
Directory.CreateDirectory(directory);
using var key = RSA.Create(2048);
var request = new CertificateRequest("CN=NativeMemoryControl", key, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
var original = request.CreateSelfSigned(DateTimeOffset.UtcNow.AddDays(-1), DateTimeOffset.UtcNow.AddDays(1));
var pfx = original.Export(X509ContentType.Pfx);
// Certificate.From takes ownership of original; only wrapped disposes that native certificate.
using var wrapped = Certificate.From(original);
void Sample(int i, string kind)
{
var live = GC.GetTotalMemory(true);
GC.WaitForPendingFinalizers();
using var p = Process.GetCurrentProcess();
p.Refresh();
Console.WriteLine(JsonSerializer.Serialize(new { kind, i, pid = Environment.ProcessId, mode, live, rss = p.WorkingSet64, methods = JitInfo.GetCompiledMethodCount(), gc = GC.GetGCMemoryInfo().TotalCommittedBytes }));
Console.Out.Flush();
}
Sample(0, "ready");
while (!File.Exists(Path.Combine(directory, "go"))) Thread.Sleep(50);
for (int i = 1; i <= 1000; i++)
{
using X509Certificate2 copy = mode switch
{
"pfx" => System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadPkcs12(pfx, null, X509KeyStorageFlags.Exportable),
"sdk-copy" => wrapped.AsX509Certificate2(),
"shared-copy" => (X509Certificate2)typeof(X509Certificate2).GetConstructor(new[] { typeof(X509Certificate) })!.Invoke(new object[] { original }),
_ => System.Security.Cryptography.X509Certificates.X509CertificateLoader.LoadCertificate(original.RawData)
};
if (i % 100 == 0) Sample(i, "sample");
}
Sample(1000, "done");
while (!File.Exists(Path.Combine(directory, "stop"))) Thread.Sleep(50);
scripts/validation/opcua-lifecycle-memory/EventPipeCapture.cs
-15
View File
@@ -1,15 +0,0 @@
// Read-only temporary diagnostic attachment. Requires Microsoft.Diagnostics.NETCore.Client from dotnet-trace.
using System.Diagnostics.Tracing;
using Microsoft.Diagnostics.NETCore.Client;
var seconds = int.Parse(args[1]);
if (seconds < 1 || seconds > 300) throw new ArgumentOutOfRangeException(nameof(seconds), "Use a bounded 1..300 second diagnostic attachment");
var client = new DiagnosticsClient(int.Parse(args[0]));
var providers = new[] { new EventPipeProvider("Microsoft-Windows-DotNETRuntime", EventLevel.Verbose, 0x18) };
// Match the recorded experiment's API default (256 MiB capacity, not measured resident allocation).
using var session = client.StartEventPipeSession(providers, requestRundown: false, circularBufferMB: 256);
await using var file = File.Create(args[2]);
var copy = session.EventStream.CopyToAsync(file);
await Task.Delay(TimeSpan.FromSeconds(seconds));
session.Stop();
await copy;
scripts/validation/opcua-lifecycle-memory/LifecycleControl.cs
-208
View File
@@ -1,208 +0,0 @@
using System.Diagnostics;
using System.Collections;
using System.Reflection;
using System.Net;
using System.Net.Sockets;
using System.Runtime;
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.Logging.Abstractions;
using Opc.Ua;
using Opc.Ua.Client;
using Opc.Ua.Configuration;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
using ZB.MOM.WW.OtOpcUa.OpcUaServer.Security;
var root = Path.GetFullPath(args[0]);
Directory.CreateDirectory(root);
var seconds = int.Parse(args[1]);
using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(seconds + 120));
var ct = timeout.Token;
var clientPki = Path.Combine(root, "client-pki");
var telemetry = DefaultTelemetry.Create(_ => { });
var config = new ApplicationConfiguration(telemetry);
config.ApplicationName = "NativeLifecycleControl";
config.ApplicationUri = "urn:NativeLifecycleControl:" + Guid.NewGuid();
config.ApplicationType = ApplicationType.Client;
config.SecurityConfiguration = new SecurityConfiguration
{
ApplicationCertificate = new CertificateIdentifier { StoreType = CertificateStoreType.Directory, StorePath = Path.Combine(clientPki, "own"), SubjectName = "CN=NativeLifecycleControl" },
TrustedIssuerCertificates = new CertificateTrustList { StoreType = CertificateStoreType.Directory, StorePath = Path.Combine(clientPki, "issuer") },
TrustedPeerCertificates = new CertificateTrustList { StoreType = CertificateStoreType.Directory, StorePath = Path.Combine(clientPki, "trusted") },
RejectedCertificateStore = new CertificateTrustList { StoreType = CertificateStoreType.Directory, StorePath = Path.Combine(clientPki, "rejected") },
AutoAcceptUntrustedCertificates = true
};
config.ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 15000 };
var app = new ApplicationInstance(telemetry) { ApplicationConfiguration = config };
await config.ValidateAsync(ApplicationType.Client, ct);
await app.CheckApplicationInstanceCertificatesAsync(true, null, ct);
int port;
using (var listener = new TcpListener(IPAddress.Loopback, 0))
{
listener.Start();
port = ((IPEndPoint)listener.LocalEndpoint).Port;
}
var options = new OpcUaApplicationHostOptions { ApplicationName = "NativeLifecycleServer", ApplicationUri = "urn:NativeLifecycleServer:" + Guid.NewGuid(), OpcUaPort = port, PublicHostname = "127.0.0.1", PkiStoreRoot = Path.Combine(root, "server-pki"), AllowAnonymous = false, AutoAcceptUntrustedClientCertificates = true, EnabledSecurityProfiles = [OpcUaSecurityProfile.Basic256Sha256SignAndEncrypt] };
OpcUaApplicationHost host = null!;
OtOpcUaSdkServer server = null!;
async Task Boot()
{
server = new OtOpcUaSdkServer();
host = new OpcUaApplicationHost(options, NullLogger<OpcUaApplicationHost>.Instance, new Auth());
await host.StartAsync(server, ct);
var n = server.NodeManager!;
n.NodeWriteGateway = new Gateway();
n.EnsureFolder("Control", null, "Control", AddressSpaceRealm.Raw);
n.EnsureFolder("Equipment", null, "Equipment", AddressSpaceRealm.Uns);
n.EnsureVariable("Control/Scalar", "Control", "Scalar", "Int32", true, AddressSpaceRealm.Raw);
n.EnsureVariable("Control/Array", "Control", "Array", "Int16", true, AddressSpaceRealm.Raw, isArray: true, arrayLength: 4);
n.EnsureVariable("Equipment/Scalar", "Equipment", "Scalar", "Int32", true, AddressSpaceRealm.Uns);
for (var i = 0; i < 16; i++) n.EnsureVariable("Control/Browse" + i, "Control", "Browse" + i, "Int32", false, AddressSpaceRealm.Raw);
n.MaterialiseAlarmCondition("Control/Alarm", "Control", "Alarm", "OffNormalAlarm", 500, AddressSpaceRealm.Raw, isNative: true);
n.WriteValue("Control/Scalar", 0, 0, DateTime.UtcNow, AddressSpaceRealm.Raw);
n.WriteValue("Control/Array", new short[] { 0, 1, 2, 3 }, 0, DateTime.UtcNow, AddressSpaceRealm.Raw);
n.WriteValue("Equipment/Scalar", 0, 0, DateTime.UtcNow, AddressSpaceRealm.Uns);
}
async Task<ISession> Connect()
{
var t = DefaultTelemetry.Create(_ => { });
var endpoint = await CoreClientUtils.SelectEndpointAsync(config, $"opc.tcp://127.0.0.1:{port}/OtOpcUa", true, t, ct);
var factory = new DefaultSessionFactory(t);
var identity = new UserIdentity("control", Encoding.UTF8.GetBytes(FixtureCredential.Value));
return await factory.CreateAsync(config, new ConfiguredEndpoint(null, endpoint ?? throw new Exception("Missing endpoint"), EndpointConfiguration.Create(config)), false, false, "NativeLifecycleControl", 15000, identity, default, ct);
}
static DataValue Value(object v)
{
#if SDK2
if (!VariantHelper.TryCastFrom(v, out var variant)) throw new Exception("Unsupported control value");
return new DataValue(variant);
#else
return new DataValue(new Variant(v));
#endif
}
static object? Clr(DataValue v)
{
#if SDK2
return v.WrappedValue.AsBoxedObject(Variant.BoxingBehavior.Legacy);
#else
return v.Value;
#endif
}
object? CertificateStats()
{
#if SDK2
return new { created = Opc.Ua.Security.Certificates.Certificate.InstancesCreated, disposed = Opc.Ua.Security.Certificates.Certificate.InstancesDisposed, outstanding = Opc.Ua.Security.Certificates.Certificate.InstancesLeaked };
#else
return null;
#endif
}
void Sample(int cycle, string kind)
{
var live = GC.GetTotalMemory(true);
using var p = Process.GetCurrentProcess();
p.Refresh();
var m = GC.GetGCMemoryInfo();
Console.WriteLine(JsonSerializer.Serialize(new { kind, cycle, pid = Environment.ProcessId, live, rss = p.WorkingSet64, gcCommitted = m.TotalCommittedBytes, gcHeap = m.HeapSizeBytes, fragmented = m.FragmentedBytes, jit = JitInfo.GetCompiledMethodCount(), jitIl = JitInfo.GetCompiledILBytes(), sdk = typeof(NodeId).Assembly.GetName().Version?.ToString(), scope = "matched encrypted public lifecycle subset; no history/revocation/proxy", assemblies = AppDomain.CurrentDomain.GetAssemblies().Length, dynamicAssemblies = AppDomain.CurrentDomain.GetAssemblies().Count(a => a.IsDynamic), certificate = CertificateStats(), timers = Timer.ActiveCount, threads = p.Threads.Count, sessions = server.CurrentInstance.SessionManager.GetSessions().Count, subscriptions = server.CurrentInstance.SubscriptionManager.GetSubscriptions().Count }));
Console.Out.Flush();
}
int cycles = 0, restarts = 0;
await Boot();
var watch = Stopwatch.StartNew();
Sample(0, "start");
try
{
while (watch.Elapsed.TotalSeconds < seconds)
{
using (var owner = await Connect()) using (var other = await Connect())
{
var ns = (ushort)owner.NamespaceUris.GetIndex("https://zb.com/otopcua/raw");
var scalar = new NodeId("Control/Scalar", ns);
var array = new NodeId("Control/Array", ns);
using var subscription = new Subscription(owner.DefaultSubscription) { PublishingInterval = 100 };
owner.AddSubscription(subscription);
await subscription.CreateAsync(ct);
subscription.AddItem(new MonitoredItem(subscription.DefaultItem) { StartNodeId = scalar, AttributeId = Attributes.Value, SamplingInterval = 50 });
subscription.AddItem(new MonitoredItem(subscription.DefaultItem) { StartNodeId = array, AttributeId = Attributes.Value, SamplingInterval = 50 });
await subscription.ApplyChangesAsync(ct);
var response = await owner.WriteAsync(null, [new WriteValue { NodeId = scalar, AttributeId = Attributes.Value, Value = Value(++cycles) }, new WriteValue { NodeId = array, AttributeId = Attributes.Value, Value = Value(new short[] { 1, 2, 3, 4 }) }], ct);
if (response.Results.ToList().Any(s => StatusCode.IsBad(s))) throw new Exception("Control write denied");
if (!Equals(Clr(await owner.ReadValueAsync(scalar, ct)), cycles)) throw new Exception("Scalar changed");
if (!((short[])Clr(await owner.ReadValueAsync(array, ct))!).SequenceEqual(new short[] { 1, 2, 3, 4 })) throw new Exception("Array changed");
var browse = await owner.BrowseAsync(null, null, 1, [new BrowseDescription { NodeId = new NodeId("Control", ns), BrowseDirection = BrowseDirection.Forward, ReferenceTypeId = ReferenceTypeIds.HierarchicalReferences, IncludeSubtypes = true, ResultMask = (uint)BrowseResultMask.All }], ct);
await Task.Delay(1000, ct);
await subscription.DeleteAsync(true, ct);
await owner.CloseAsync(ct);
await other.CloseAsync(ct);
}
if (server.CurrentInstance.SessionManager.GetSessions().Count != 0 || server.CurrentInstance.SubscriptionManager.GetSubscriptions().Count != 0) throw new Exception("Control retained sessions/subscriptions");
Sample(cycles, "sample");
if (cycles % 4 == 0)
{
using var parked = await Connect();
using var sub = new Subscription(parked.DefaultSubscription) { PublishingInterval = 100 };
parked.AddSubscription(sub);
await sub.CreateAsync(ct);
sub.AddItem(new MonitoredItem(sub.DefaultItem) { StartNodeId = new NodeId("Control/Scalar", (ushort)parked.NamespaceUris.GetIndex("https://zb.com/otopcua/raw")), AttributeId = Attributes.Value, SamplingInterval = 50 });
await sub.ApplyChangesAsync(ct);
await host.DisposeAsync();
await Boot();
restarts++;
}
}
Console.WriteLine(JsonSerializer.Serialize(new { kind = "complete", cycles, restarts, seconds = watch.Elapsed.TotalSeconds, fullGate = false }));
}
finally
{
await host.DisposeAsync();
#if SDK2
await app.DisposeAsync();
#endif
}
#if SDK2
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
var certificateType = typeof(Opc.Ua.Security.Certificates.Certificate);
var report = new Dictionary<string, object>();
var trackingEnabled = Environment.GetEnvironmentVariable("OPCUA_CERTIFICATE_LEAK_TRACKING") == "1";
if (trackingEnabled)
{
foreach (var name in new[] { "EnumerateLiveCertificates", "EnumerateUnreachableUndisposedCertificates" })
{
var method = certificateType.GetMethod(name, BindingFlags.Static | BindingFlags.NonPublic)
?? throw new InvalidOperationException("SDK2 diagnostic method missing: " + name);
var values = method.Invoke(null, null) as IEnumerable
?? throw new InvalidOperationException("SDK2 diagnostic method return type changed: " + name);
var stacks = new List<string>();
foreach (var value in values)
{
var field = value!.GetType().GetField(name.Contains("Unreachable") ? "Item2" : "Item4")
?? throw new InvalidOperationException("SDK2 diagnostic tuple field missing: " + name);
if (field.FieldType != typeof(string))
throw new InvalidOperationException("SDK2 diagnostic stack field type changed");
// Deliberately omit tuple Item1 (certificate identity).
stacks.Add((string)field.GetValue(value)!);
}
report[name] = new
{
count = stacks.Count,
allocations = stacks.GroupBy(x => x).OrderByDescending(g => g.Count()).Take(15)
.Select(g => new { count = g.Count(), stack = g.Key }).ToArray()
};
}
}
Console.WriteLine(JsonSerializer.Serialize(new { kind = "certificate-final-diagnostics", trackingEnabled, stats = CertificateStats(), report }));
#endif
class Auth : IOpcUaUserAuthenticator
{
public Task<OpcUaUserAuthResult> AuthenticateUserNameAsync(string user, string password, CancellationToken ct) => Task.FromResult(user == "control" && password == FixtureCredential.Value ? OpcUaUserAuthResult.Allow("control", [OpcUaDataPlaneRoles.WriteOperate]) : OpcUaUserAuthResult.Deny("Control credential refused"));
}
class Gateway : IOpcUaNodeWriteGateway
{
public Task<NodeWriteOutcome> WriteAsync(string n, object? v, AddressSpaceRealm r, CancellationToken ct) => Task.FromResult(new NodeWriteOutcome(true, null));
}
static class FixtureCredential
{
internal static string Value { get; } = Environment.GetEnvironmentVariable("OPCUA_MEMORY_PASSWORD") ?? throw new InvalidOperationException("Set OPCUA_MEMORY_PASSWORD for the task-owned fixture.");
}
scripts/validation/opcua-lifecycle-memory/README.md
-286
View File
@@ -1,286 +0,0 @@
# SDK 2 lifecycle memory investigation
The required two-hour server lifecycle gate failed on macOS and Linux. The RSS
limit remains **cycle-3 baseline + 128 MiB**; retained managed memory remains
+64 MiB and descriptors +32. No threshold, production code, certificate provider,
or session factory was changed by this investigation. These controls do not
replace the full gate. Issue #799 remains open.
## Preserved full-workload failures
| Run | Cycles / restarts | Observation time | RSS / limit (bytes) | Outcome |
|---|---:|---:|---:|---|
| macOS `fa823378` | 250 / 62 | 1045.53 s | 365,215,744 / 363,331,584 | Failed |
| macOS `e135b7c9` profile reproduction | 245 / 61 | 1024.14 s | 368,656,384 / 363,053,056 | Failed |
| Linux full harness, recorded build `38098ce142ab8c382825aa754e0758071dd3a46b` | 248 / 61 | 903.508 s | 353,931,264 / 351,850,496 | Failed |
| macOS `6b2d4b81` bounded profile | 432 / 108 | 1807.479 s | First rejection: 364,199,936 / 363,266,048 | Failed; first assertion rethrown |
The last macOS row requested 1,800 seconds. It first rejected RSS at cycle 253 /
1055.482 seconds, continued recording explicitly rejected observations, then
threw the original assertion before any completion record. Its final observation
was RSS 380,059,648, live managed 41,364,096, GC committed 37,060,608, JIT
compilations 81,535, descriptors 200, and zero sessions, subscriptions, monitored
items, browse/history continuation points, and PKI descriptors. SDK cached stores
remained two during operation. The test runner returned exit 1; the nonzero-execution guard passed (exit 0),
with one executed test and zero skipped tests. There is no passing two-hour result here.
The first macOS run's reported PKI descriptor zero is invalid: `/var` and
`/private/var` aliases were not canonicalized. Corrected positive-handle and timeout
controls passed before subsequent profiles. Global descriptor counts and the RSS
failure itself were independent of that defect. A Shouldly source-expression
message incorrectly names `Descriptors` in the RSS failure; the actual numeric
comparison is RSS, as retained before the assertion.
## What the controls demonstrate
`LifecycleControl.cs` uses unchanged published product DLLs and public APIs for
both SDK generations. It opens two real encrypted **opc.tcp** sessions with
Basic256Sha256 / SignAndEncrypt, resolves namespace URIs, reads/writes Int32 and
Int16[] values, creates/deletes two data monitored items, abandons a browse
continuation, closes sessions, and restarts a server with an active subscription
every four cycles against the same task-owned PKI. Public session/subscription
counts must return to zero. It has no historian, revocation, surgical-rebuild,
paused-network, or private continuation-store coverage. It is a diagnostic subset,
not TLS and not the full soak workload.
The original DLLs are the immutable 1.5.378.182 baseline publication associated
with `83ed880e`; its assembly version alone is 1.5.378.0 and does not identify the
patch. The candidate uses SDK 2.0.1 and the server implementation from `fa823378`.
Neither baseline nor candidate product code was rebuilt for these controls.
The controls were compiled once for each reference set, then those exact control
artifacts were used on both operating systems. Selected DLL and input-artifact
hashes are in `evidence.json`. Neither reference set's principal SDK/server DLLs
has a managed native/ReadyToRun header.
| Ten-minute matched subset | Cycles / restarts | Final RSS | Change from cycle 3 | Final managed / change | Final JIT compilations |
|---|---:|---:|---:|---:|---:|
| macOS original | 212 / 53 | 330,301,440 | +121,683,968 | 271,168,424 / +169,990,456 | 28,066 |
| macOS SDK2 | 200 / 50 | 302,170,112 | +95,862,784 | 44,569,584 / +295,120 | 62,608 |
| Linux original | 249 / 62 | 384,950,272 | +176,836,608 | 147,139,568 / +99,356,752 | 28,321 |
| Linux SDK2 | 259 / 64 | 390,844,416 | +194,183,168 | 19,561,120 / +736 | 71,529 |
Both ten-minute controls exited zero because their lifecycle checks passed;
`fullGate=false` is explicit. They do **not** enforce the full harness memory
assertions. In particular, both Linux subset RSS deltas exceed 128 MiB and the
original retains managed memory/timers. These observations do not waive either
failed full gate. They are not proof of an SDK1 defect or an SDK2 improvement.
Both control branches explicitly delete subscriptions, close normal sessions,
and dispose sessions/subscriptions. Parked resources intentionally encounter
server shutdown and are then disposed. Each version calls its unchanged product
host's DisposeAsync. The baseline host only calls ApplicationInstance.StopAsync;
that original ApplicationInstance exposes no disposal interface. The SDK2 branch
also uses its new ApplicationInstance.DisposeAsync ownership API. Per-connection
DefaultTelemetry factories are left to the same ownership pattern on both
branches, separately probed above. Thus measured original growth belongs to this
published-product/fixture lifecycle combination; its allocation owner has not
been isolated. These are one matched run per platform, concurrent processes,
with different runtime versions; they are not performance distributions.
The thirty-minute Linux comparison uses the same immutable artifacts and image,
fresh independent fixtures, and SDK2 certificate allocation tracking enabled
before the first certificate (the same environment is passed to both versions).
Both finished with exit zero and `fullGate=false`:
| Thirty-minute Linux subset | Cycles / restarts | Final RSS | Final live managed | GC committed | JIT compilations | Active timers |
|---|---:|---:|---:|---:|---:|---:|
| Original | 736 / 184 | 540,057,600 | 342,626,200 | 408,399,872 | 29,177 | 926 |
| SDK2 | 768 / 192 | 334,180,352 | 19,248,984 | 42,983,424 | 72,087 | 19 |
Observed runtime was 1802.542 seconds original / 1804.523 seconds SDK2. The SDK2
RSS fell from 390,782,976 at the 632.7-second checkpoint to 334,180,352 at the last
observation; live managed stayed near 20 MB. After disposal and final collection,
SDK2 wrapper counters were 19,272 created = 19,272 disposed, with zero live or
unreachable-undisposed wrappers under enabled tracking. These finite results
support post-transition saturation of this subset, not a two-hour full-workload
pass or a guarantee of indefinite stability.
### Native and runtime observations
- During the macOS bounded profile, `Security::AppleDatabase` counts increased
from 278 at cycle 52 / 211 seconds to 1,953 at cycle 370 / 1546 seconds. These
native objects continued accumulating while managed memory stayed near 40 MiB.
- The single-owner certificate probe makes a self-signed RSA fixture in memory,
then disposes each of 1,000 copies/imports. On macOS, public-only and
shared-handle copies leave AppleDatabase at 2; PFX imports and SDK
`Certificate.AsX509Certificate2` private-key copies grow it from 2 to 1,002.
A separate PFX process retained that count through a 180-second idle hold.
`Certificate.From(original)` owns the original; the checked-in control has no
second owner. All four corrected single-owner modes were rerun successfully.
- The same isolated Linux probe ended at approximately 49 MB RSS for public or
shared copies, 61 MB for PFX imports, and 69 MB for the SDK copy. Those are
endpoint RSS observations including different JIT/GC costs, not evidence of a
Linux native-object leak or a complete attribution.
- SDK2 public certificate counters after the Linux ten-minute lifecycle run were
6,495 created = 6,495 disposed. A separate stack-tracked macOS lifecycle run
ended 591 = 591, zero live and zero unreachable-undisposed wrappers. Allocation
tracking is opt-in; empty internal reports without that opt-in are not proof.
- The Linux ten-minute native snapshots show SDK2 JIT `doublemapper` resident
mappings increasing from about 65.4 to 106.7 MiB, anonymous writable mappings
from 84.6 to 121.3 MiB, and `[heap]` around 19.2 MiB. Original JIT mappings were
about 33.3 to 37.7 MiB. Mapping totals are descriptive categories, not exclusive
object allocations: anonymous memory also contains GC regions, stacks, and
native allocator arenas. RSS includes file/shared pages and can fall with
reclamation. JIT counters are cumulative compilations, not unique live methods.
- 10,000 disposed versus undisposed `DefaultTelemetry` factories ended at similar
RSS (~63.4 MB) and live managed memory (~0.43 MB), with no timer growth. This
does not justify changing host telemetry ownership to fix this failure.
Exact SDK tag source shows private-key PFX copying in nonvirtual
`Certificate.AsX509Certificate2`. A 60-second macOS sampled-thread-time trace
instead found the dominant sampled import path through
`CertificateIdentifierResolver.LoadPrivateKeyCoreAsync` →
`DirectoryCertificateStore.LoadPrivateKeyAsync` → `X509CertificateLoader.LoadPkcs12`.
Inclusive sampled thread-time was about 4.79 seconds for store load and 2.99 seconds
for PKCS12, versus 8.5 milliseconds for AsX509. These are neither exclusive CPU
costs nor call counts. ApplicationInstance already normalizes its validation
input to public DER. The validation copy delegate is internal, the manager sealed,
and no safe narrow application copy-policy seam was identified. Changing session
factory, certificate exportability, or key storage would alter behavior and was
not used as a workaround.
A major JIT/RSS step appeared near 60–64 restarts in several workloads. This is a
hypothesis about runtime/code-cache warmup, not a finding that all RSS growth is
benign. A separate read-only EventPipe attachment observed the thirty-minute
Linux candidate around that transition using `Microsoft-Windows-DotNETRuntime`,
Verbose, keywords `0x18` (Loader + JIT), no rundown, 180 seconds. The transition was UTC 18:51:02.049–18:54:02.062. Factory JIT events
occurred UTC 18:52:48.249–18:53:21.434: 6,690 distinct method tokens from one
module, each once. The module is Opc.Ua.Core.Types.dll; metadata declares 8,442
static methods on OpcUaExtensions. The observed set is a subset, so this does
not prove every possible factory or the complete canonical workload is warmed.
Method-load events totalled 51,327,536 native code bytes, including 51,161,724
OptimizedTier1 bytes and 49,141,484 bytes in generated factories. Only 50
reflection InvokeStub JIT starts occurred; the transition trace reports zero lost events. MethodSize sums count loaded code
versions, not resident/PSS allocation bytes. An additional 180-second attachment
after the transition checks continuing factory compilation. The second attachment (UTC 19:01:44.116–19:04:44.127) recorded 85 JIT starts,
zero lost events, zero Opc.Ua starts, zero generated-factory starts, and zero generated-factory
loaded bytes. Only 18 ordinary Tier1 method loads (5,720 bytes) and 20
instrumented Tier1 loads (5,800 bytes) occurred. This supports a finite generated
factory burst in the observed subset; it does not prove complete canonical
workload saturation. The capture API's default circular-buffer capacity was 256 MiB; that is a
capacity setting, not measured resident overhead. Attachments can allocate event
buffers and perturb execution. Both prior full failures and the ten-minute
control were uninstrumented by these JIT attachments, and the thirty-minute final
interval is unattached. No captured native-code byte total is claimed to equal an
RSS delta. Raw traces/heaps are not committed.
## Reproduce the diagnostic subset
Requires Python 3 and .NET 10. `run.py` references existing DLL directories,
compiles only the control, records SHA-256 provenance, and creates a fresh local
artifact directory. First reference directory wins duplicate filenames. Use the
actual baseline publications, rather than restoring a different package version.
No production project or package file is changed.
```bash
python3 scripts/validation/opcua-lifecycle-memory/run.py lifecycle \
--references "$baselineHostDlls" --references "$baselineClientDlls" \
--output /tmp/opcua-memory-original --seconds 600
python3 scripts/validation/opcua-lifecycle-memory/run.py lifecycle --sdk2 \
--references "$candidateDlls" --output /tmp/opcua-memory-sdk2 --seconds 600 \
--certificate-tracking
```
Fixture credentials are generated in the runner environment and not logged.
PKI material stays under each local `private-fixture` directory. Do not publish
those directories or unredacted process/native logs. These controls grant a
fixture user WriteOperate and auto-accept task-owned certificates; they are
local test fixtures, not deployment configurations.
For a matched Linux pair, first build both with `--build-only` into fresh
`/tmp/opcua-memory-original` and `/tmp/opcua-memory-sdk2` output directories, then:
```bash
python3 scripts/validation/opcua-lifecycle-memory/linux_pair.py \
--baseline /tmp/opcua-memory-original/bin/Release/net10.0 \
--candidate /tmp/opcua-memory-sdk2/bin/Release/net10.0 \
--output /tmp/opcua-memory-linux-pair --seconds 1800 \
--image sha256:b04611ee9e1b7d36797012ed921962c57189cb4dbb3dff12afb8e158110f293b
```
The runner uses separate task-owned `--network none` containers, fresh PKI,
read-only executable mounts, bounded shutdown, and periodic read-only `/proc`
snapshots. It records child and container exit statuses independently. The image
is Ubuntu 24.04 arm64, .NET SDK 10.0.103 / runtime 10.0.3. The Mac runs used
macOS 26.5.2 arm64, SDK 10.0.403 / runtime 10.0.14. Runtime and OS differences are
explicit; do not attribute all cross-platform differences to SDK code.
On macOS, run each certificate mode with a fresh output directory:
```bash
python3 scripts/validation/opcua-lifecycle-memory/run.py certificate --sdk2 \
--references "$candidateDlls" --output /tmp/opcua-memory-pfx --mode pfx
```
Other modes are `sdk-copy`, `public`, and `shared-copy`. The last intentionally
uses the old copy constructor by reflection as a diagnostic control. It is not
proposed production code. The control exports no fixture key/certificate data.
`TelemetryControl.cs` is available through `run.py telemetry --sdk2` with modes
`dispose` or `undisposed`. `summarize.py` emits Linux mapping categories without
file paths from local `*-smaps.txt` files. All controls are opt-in and outside the
ordinary solution/test graph.
Optional trace reproduction uses the installed dotnet-trace 9.0.661903 managed
libraries, without adding a product dependency:
```bash
python3 scripts/validation/opcua-lifecycle-memory/build_trace_tools.py \
--libraries "$diagnosticToolLibraries" --output /tmp/opcua-memory-trace-tools
# Run capture in the same process namespace / user as the task-owned target.
dotnet /tmp/opcua-memory-trace-tools/capture/bin/Release/net10.0/capture.dll \
"$targetPid" 180 /tmp/opcua-memory-local.nettrace
dotnet /tmp/opcua-memory-trace-tools/summary/bin/Release/net10.0/summary.dll \
/tmp/opcua-memory-local.nettrace > /tmp/opcua-memory-method-summary.json
```
Capture is read-only and opt-in; raw traces remain local. The summary exports
static method names and aggregate counts rather than certificate identities,
arguments, or process mappings. The original transition parser also retained
method tokens/modules locally to cross-check name-key uniqueness.
## Prospective saturation experiment (not executed)
The observed finite factory burst supports a separately labeled readiness
experiment; it does not authorize moving the acceptance baseline. A reviewed
candidate proposal is exactly 384 canonical lifecycle cycles / 96 every-four-cycle
restarts, then one fixed settling cycle, all within 1,800 seconds. No early finish,
extension, or RSS-selected endpoint is permitted. Both fixed readiness windows
321–352 and 353–384 must contain zero generated-factory JIT/method-load events, zero SDK assembly
loads, and zero lost events. Other JIT/native-code totals remain review evidence,
not a claim that every runtime path is warmed. Trace capture is bounded and identical for both versions, then
detached/drained before the settling cycle. Ownership, FD, PKI, store, and logical
resource checks remain active throughout; original cold budget rejections remain
recorded. Any later full gate requires at least 7,200 additional seconds after a
prospectively approved baseline. No such warmed full gate has been run here.
The count is beyond the observed 60–64-restart transition and projects to about
1,606 seconds from the Mac canonical run, but completion within the fixed cutoff
is not guaranteed under load. Failure to complete or satisfy readiness is a failed
experiment, not permission to extend. The full canonical test currently depends
on SDK2 value APIs and read-only private SDK2 continuation/store instrumentation.
An original-SDK adapter proving identical history, revocation, surgical-edit,
and stalled-network operation counts is a prerequisite for a matched canonical
experiment; the public subset does not provide that proof. Continued Mac native
inventory growth remains an independent unresolved gate, even if JIT saturates.
## Evidence and limits
`evidence.json` contains sanitized numeric endpoints, native inventory counts,
and hashes of retained diagnostic inputs. Full artifacts remain in the originating
worktrees or `/tmp/opcua2-rss-investigation`; no PKI, certificate identifiers,
credentials, raw traces, unredacted heap/process dumps, or complete test logs are
checked in. Existing failures were preserved untouched.
Supported progress is a reproduced RSS failure, bounded logical/managed
inventories in SDK2, demonstrated macOS PFX native retention, and measured
cross-platform JIT/anonymous-memory growth. A complete allocation attribution and
safe production correction remain unresolved. The server two-hour gate remains
failed; a separate long-running client-only process is a different scope.
Package verification: baseline and candidate control builds, both optional trace
tools, and certificate/telemetry controls built with zero warnings/errors. Local
control smoke runs and the fresh Linux pair smoke exited zero. Python compilation
and whitespace checks passed. One reused-output smoke failed because a manual
incremental build used a different `/tmp` path alias and its runtimeconfig was
missing; fresh canonical outputs passed, and the Linux runner now rejects an
incomplete output directory before launch. These smoke checks validate tooling
only and do not substitute for either long run.
scripts/validation/opcua-lifecycle-memory/SATURATION.md
-120
View File
@@ -1,120 +0,0 @@
# Candidate saturation experiment — review before launch
This is a prospective diagnostic protocol, not an acceptance run or a replacement
for the failed cold server soak. Production code, the cycle-3 resource baseline,
and the +64 MiB managed / +128 MiB RSS / +32 descriptor caps remain unchanged.
Only `OTOPCUA_SDK2_SOAK_MODE=saturation` selects this behavior; the existing
`diagnostic`, `profile`, and `full` modes keep their existing stopping rules.
The fixed canonical phase performs exactly 384 lifecycle cycles, with the
existing active-resource restart after every fourth cycle: 96 completed restarts.
It then performs exactly one settling cycle, without another restart. The phase,
including observer handshakes, must finish within 1,800 seconds. Initial fixture
startup has a separate 120-second bound. There is no early completion, extension,
RSS-selected endpoint, budget reset, or two-hour/full-pass flag. Cleanup remains
bounded independently of the phase deadline.
All canonical operations remain in use: encrypted opc.tcp sessions, namespace URI
binding, scalar/typed-array read/write, subscriptions, browse abandonment, history
continuation churn/release/cross-session/close, surgical changes/rebuilds, key
revocation, paused network reads/recovery, and same-PKI server restart. All
ownership, continuation, PKI-descriptor, store and node inventory checks remain
active. Cold budget rejection is latched at its original observation and rethrown,
even if diagnostic readiness later succeeds. A rejected readiness receipt does
not become a passing test or a new resource baseline.
## Observer boundaries
A fresh task-owned control directory and UUID bind every request/receipt to this
run and target PID. Files are published atomically; existing, malformed, foreign,
or incomplete receipts fail closed. The observer is a separate diagnostic process
using public `DiagnosticsClient` / EventPipe APIs. Product dependencies are
unchanged. A separate short positive control first invokes a named non-SDK method
with NoInlining/NoOptimization after attaching and requires its JIT-start count in
the parsed receipt. This proves CLR provider delivery/classification independently
of custom marker delivery. That probe is never invoked by the canonical experiment.
One continuous trace covers both fixed windows, including their boundary gap:
| Marker | Before/after operation | Completed restarts |
|---|---|---:|
| window1-start | Before cycle 321 | 80 |
| window1-end | After cycle 352 and its restart | 88 |
| window2-start | Before cycle 353 | 88 |
| window2-end | After cycle 384 and its restart | 96 |
The target waits for the observer's active receipt and enabled marker provider
before its first window operation. The parser requires exactly these four
markers, the correct UUID and cycle/restart values, and strictly ordered
timestamps. A continuous veto also covers events outside the marker ranges
within the attached interval. Both windows and the entire attached interval
require zero `Opc.Ua.OpcUaExtensions` JIT starts and method-load events, zero new
`Opc.Ua.*` assembly-load events, and zero lost events. Other JIT counts and loaded
native method-byte sums are reported for review; they do not establish that all
runtime paths are warmed. Method-size sums do not equal resident memory.
Runtime capture is Verbose `Microsoft-Windows-DotNETRuntime`, keywords `0x18`
(Loader + JIT), plus Informational `OtOpcUa-Sdk2-Saturation` markers, no rundown,
and a prospectively fixed **16 MiB** circular-buffer capacity. This is a new
experiment and does not reinterpret the earlier 256 MiB-capacity traces.
After restart 96, the target requests stop and waits. The observer must stop its
session, drain the event stream, close it, parse the entire trace, and atomically
publish a PID/run-bound drained receipt. Its Stop/copy/dispose/parse stages have
10/10/5/10-second bounds. Target receipt waits are bounded at 45 seconds and remain
inside the 1,800-second phase deadline. Observer faults propagate immediately.
The target also verifies its marker provider is disabled before the settling
cycle. Missing markers, no events, event loss, or malformed stream cannot prove
readiness. A parser failure produces a failed observer receipt, not empty counts.
The `saturation-result` separates readiness, cold budget rejection, fixed operation
completion and verified shutdown ownership. It always records `acceptance=false`.
It is emitted before rethrowing the original cold assertion. The launcher retains
nonzero test/container exits and records diagnostic readiness separately.
## Prepare immutable Linux artifacts
Commit the reviewed source first. Preparation refuses a dirty checkout and
publishes fresh, separate app/observer artifacts for `linux-arm64`, self-contained
.NET **10.0.14**, without trimming or single-file publishing. It never starts the
experiment. The bundle records source revision, source hashes, exact publish
command, fixed protocol, image digest and SHA-256 of every app/observer/tool file.
It requires the landed HistorianGateway Client/Contracts 1.0.0 dependency graph
and records assembly versions/module MVIDs by reading PE metadata without loading
or executing product assemblies.
```bash
python3 scripts/validation/opcua-lifecycle-memory/prepare_saturation.py \
--output /tmp/opcua-saturation-reviewed \
--libraries "$diagnosticToolLibraries" \
--image sha256:b04611ee9e1b7d36797012ed921962c57189cb4dbb3dff12afb8e158110f293b
```
`diagnosticToolLibraries` is the installed dotnet-trace 9.0.661903
`tools/net8.0/any` directory. No product package reference is added. Artifact
preparation and review are prerequisites; root and Astra must review the actual
code, control tests and receipt before launch. The coordinator owns the long
execution.
```bash
python3 /tmp/opcua-saturation-reviewed/tools/launch_saturation.py \
--bundle /tmp/opcua-saturation-reviewed --receipt-sha256 "$reviewedReceiptHash" \
--output /tmp/opcua-saturation-run --verify-only
# After root/Astra review, root runs the same command without --verify-only.
```
The launcher rechecks every immutable file before and after the run. It creates a
fresh offline task-owned container, writable output/PKI directories, and read-only
app/observer/tool mounts. Descriptor controls run first in a separate process.
TRX guards require two descriptor tests and one canonical test actually executed,
with no skipped results. The launcher verifies the actual target and observer
mapped their own self-contained `libcoreclr.so`; the older SDK-image controller
runtime is not the target runtime. A 2,040-second outer timeout includes startup,
phase and bounded cleanup; it does not extend the 1,800-second phase.
No long saturation run has been launched during implementation. Existing cold
failures remain untouched. No original-SDK full-workload adapter, warmed
acceptance baseline, additional two-hour server run, or production correction is
part of this change. The separately demonstrated Mac native certificate-related
inventory growth remains unresolved even if generated-factory compilation
saturates.
scripts/validation/opcua-lifecycle-memory/SaturationObserver.cs
-133
View File
@@ -1,133 +0,0 @@
// Diagnostic-only observer. The target remains the unchanged canonical lifecycle test process.
using System.Diagnostics.Tracing;
using System.Text.Json;
using Microsoft.Diagnostics.NETCore.Client;
using Microsoft.Diagnostics.Tracing;
const string protocol = "sdk2-saturation-v1";
const string markerProvider = "OtOpcUa-Sdk2-Saturation";
var control = Path.GetFullPath(args[0]);
var runId = args[1];
if (!Guid.TryParseExact(runId, "D", out _)) throw new InvalidDataException("Malformed run identity");
using var lifetime = new CancellationTokenSource(TimeSpan.FromSeconds(1980));
var ct = lifetime.Token;
try
{
var request = await WaitJson("attach-request.json", ct);
using (request)
{
Identity(request.RootElement);
Require(request.RootElement.GetProperty("firstCycle").GetInt32() == 321
&& request.RootElement.GetProperty("completedRestarts").GetInt32() == 80, "Invalid attach boundary");
var targetPid = request.RootElement.GetProperty("targetPid").GetInt32();
Require(targetPid > 0 && targetPid != Environment.ProcessId, "Invalid target process");
var providers = new[]
{
new EventPipeProvider("Microsoft-Windows-DotNETRuntime", EventLevel.Verbose, 0x18),
new EventPipeProvider(markerProvider, EventLevel.Informational)
};
var client = new DiagnosticsClient(targetPid);
var session = await Task.Run(() => client.StartEventPipeSession(providers, requestRundown: false, circularBufferMB: 16), ct)
.WaitAsync(TimeSpan.FromSeconds(10), ct);
var trace = Path.Combine(control, "readiness.nettrace");
await using (var output = new FileStream(trace, FileMode.CreateNew, FileAccess.Write, FileShare.Read))
{
var copy = session.EventStream.CopyToAsync(output, ct);
await Publish("active.json", new { protocol, runId, targetPid, observerPid = Environment.ProcessId, active = true });
using var stop = await WaitJson("detach-request.json", ct);
Identity(stop.RootElement);
Require(stop.RootElement.GetProperty("targetPid").GetInt32() == targetPid
&& stop.RootElement.GetProperty("lastCycle").GetInt32() == 384
&& stop.RootElement.GetProperty("completedRestarts").GetInt32() == 96, "Invalid detach boundary");
await Task.Run(session.Stop, ct).WaitAsync(TimeSpan.FromSeconds(10), ct);
await copy.WaitAsync(TimeSpan.FromSeconds(10), ct);
await output.FlushAsync(ct);
}
await Task.Run(session.Dispose, ct).WaitAsync(TimeSpan.FromSeconds(5), ct);
var summary = await Task.Run(() => Parse(trace, runId), ct).WaitAsync(TimeSpan.FromSeconds(10), ct);
// Published only after Stop, stream drain, disposal and complete parsing. Never just a file-exists ACK.
await Publish("drained.json", new
{
protocol, runId, targetPid, observerPid = Environment.ProcessId, active = false, drained = true,
summary.markersValid, summary.eventsLost, summary.sdkFactoryJit, summary.sdkFactoryLoads,
summary.sdkAssemblyLoads, summary.windows, summary.otherJit, summary.loadedNativeBytes, summary.runtimeControlJit,
configuration = new { runtimeProvider = "Microsoft-Windows-DotNETRuntime", level = "Verbose", keywords = "0x18",
markerProvider, circularBufferMiB = 16, requestRundown = false },
acceptance = false
});
}
}
catch (Exception error)
{
await Publish("observer-failed.json", new { protocol, runId, exceptionType = error.GetType().FullName });
// An IPC/Stop worker can still be blocked. The parent also bounds and reaps this separate process.
Environment.Exit(1);
}
void Identity(JsonElement value) => Require(value.GetProperty("protocol").GetString() == protocol
&& value.GetProperty("runId").GetString() == runId, "Mismatched observer identity");
static void Require(bool condition, string message) { if (!condition) throw new InvalidDataException(message); }
async Task<JsonDocument> WaitJson(string name, CancellationToken token)
{
var path = Path.Combine(control, name);
while (!File.Exists(path)) await Task.Delay(25, token);
return JsonDocument.Parse(await File.ReadAllTextAsync(path, token));
}
async Task Publish(string name, object value)
{
var temporary = Path.Combine(control, name + ".tmp");
await using (var stream = new FileStream(temporary, FileMode.CreateNew, FileAccess.Write))
await JsonSerializer.SerializeAsync(stream, value);
File.Move(temporary, Path.Combine(control, name));
}
static TraceResult Parse(string path, string expectedRun)
{
var runtimeControlJit = 0;
var markers = new List<Marker>();
var events = new List<Observed>();
using var source = new EventPipeEventSource(path);
source.Clr.MethodJittingStarted += data =>
{
if (data.MethodNamespace == "ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.Sdk2LifecycleSoakTests"
&& data.MethodName == "SaturationProtocolRuntimeProbe") runtimeControlJit++;
events.Add(new(data.TimeStampRelativeMSec,
data.MethodNamespace == "Opc.Ua.OpcUaExtensions" ? "factory-jit" : "other-jit", 0));
};
source.Clr.MethodLoadVerbose += data => events.Add(new(data.TimeStampRelativeMSec,
data.MethodNamespace == "Opc.Ua.OpcUaExtensions" ? "factory-load" : "other-load", data.MethodSize));
source.Clr.LoaderAssemblyLoad += data =>
{
if (data.FullyQualifiedAssemblyName.StartsWith("Opc.Ua.", StringComparison.Ordinal))
events.Add(new(data.TimeStampRelativeMSec, "sdk-assembly", 0));
};
source.Dynamic.All += data =>
{
if (data.ProviderName == "OtOpcUa-Sdk2-Saturation" && data.EventName == "Boundary")
markers.Add(new(data.TimeStampRelativeMSec, (string)data.PayloadByName("runId"),
(string)data.PayloadByName("name"), (int)data.PayloadByName("cycle"), (int)data.PayloadByName("restarts")));
};
source.Process();
(string Name, int Cycle, int Restarts)[] expected =
[ ("window1-start", 321, 80), ("window1-end", 352, 88),
("window2-start", 353, 88), ("window2-end", 384, 96) ];
var valid = markers.Count == 4 && markers.Select((marker, index) =>
marker.Run == expectedRun && (marker.Name, marker.Cycle, marker.Restarts) == expected[index]).All(x => x)
&& markers.Zip(markers.Skip(1)).All(pair => pair.First.Ms < pair.Second.Ms);
var windows = Enumerable.Range(0, 2).Select(index =>
{
var selected = valid ? events.Where(x => x.Ms >= markers[index * 2].Ms && x.Ms <= markers[index * 2 + 1].Ms).ToArray() : [];
return new Window($"window{index + 1}", 321 + index * 32, 352 + index * 32, 80 + index * 8, 88 + index * 8,
selected.Count(x => x.Kind == "factory-jit"), selected.Count(x => x.Kind == "factory-load"),
selected.Count(x => x.Kind == "sdk-assembly"));
}).ToArray();
// Totals include the whole attached interval, including marker gaps, startup and stopping the collector.
return new TraceResult(valid, source.EventsLost, events.Count(x => x.Kind == "factory-jit"),
events.Count(x => x.Kind == "factory-load"), events.Count(x => x.Kind == "sdk-assembly"), windows,
events.Count(x => x.Kind == "other-jit"), events.Sum(x => x.Bytes), runtimeControlJit);
}
record Marker(double Ms, string Run, string Name, int Cycle, int Restarts);
record Observed(double Ms, string Kind, long Bytes);
record Window(string name, int firstCycle, int lastCycle, int startRestarts, int endRestarts,
int sdkFactoryJit, int sdkFactoryLoads, int sdkAssemblyLoads);
record TraceResult(bool markersValid, int eventsLost, int sdkFactoryJit, int sdkFactoryLoads,
int sdkAssemblyLoads, Window[] windows, int otherJit, long loadedNativeBytes, int runtimeControlJit);
scripts/validation/opcua-lifecycle-memory/TelemetryControl.cs
-22
View File
@@ -1,22 +0,0 @@
using System.Diagnostics;
using System.Runtime;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using Opc.Ua;
var dispose = args[0] == "dispose";
for (int i = 0; i <= 10000; i++)
{
if (i > 0)
{
var t = DefaultTelemetry.Create(_ => { });
t.LoggerFactory.CreateLogger("TelemetryControl").LogDebug("control");
if (dispose) ((IDisposable)t).Dispose();
}
if (i % 250 == 0)
{
var live = GC.GetTotalMemory(true);
using var p = Process.GetCurrentProcess();
p.Refresh();
Console.WriteLine(JsonSerializer.Serialize(new { i, dispose, live, rss = p.WorkingSet64, methods = JitInfo.GetCompiledMethodCount(), il = JitInfo.GetCompiledILBytes(), gc = GC.GetGCMemoryInfo().TotalCommittedBytes, threads = p.Threads.Count, timers = Timer.ActiveCount }));
}
}
scripts/validation/opcua-lifecycle-memory/TraceSummary.cs
-62
View File
@@ -1,62 +0,0 @@
// Redacted aggregate summary. Keep raw traces local.
using Microsoft.Diagnostics.Tracing;
using System.Text.Json;
var starts = new Dictionary<string, int>();
var tiers = new Dictionary<string, int>();
var nativeBytes = new Dictionary<string, long>();
var factoryNames = new Dictionary<string, int>();
var factoryTokens = new HashSet<int>();
var factoryModules = new HashSet<long>();
var firstFactoryMs = double.PositiveInfinity;
var lastFactoryMs = 0d;
var factoryNativeBytes = 0L;
var sdkAssemblyLoads = 0;
using var source = new EventPipeEventSource(args[0]);
source.Clr.MethodJittingStarted += data =>
{
var name = data.MethodNamespace + "." + data.MethodName;
starts[name] = starts.GetValueOrDefault(name) + 1;
if (!name.StartsWith("Opc.Ua.OpcUaExtensions.", StringComparison.Ordinal)) return;
factoryNames[name] = factoryNames.GetValueOrDefault(name) + 1;
factoryTokens.Add(data.MethodToken);
factoryModules.Add(data.ModuleID);
firstFactoryMs = Math.Min(firstFactoryMs, data.TimeStampRelativeMSec);
lastFactoryMs = Math.Max(lastFactoryMs, data.TimeStampRelativeMSec);
};
source.Clr.MethodLoadVerbose += data =>
{
var tier = data.OptimizationTier.ToString();
tiers[tier] = tiers.GetValueOrDefault(tier) + 1;
nativeBytes[tier] = nativeBytes.GetValueOrDefault(tier) + data.MethodSize;
if (data.MethodNamespace == "Opc.Ua.OpcUaExtensions") factoryNativeBytes += data.MethodSize;
};
source.Clr.LoaderAssemblyLoad += data =>
{
if (data.FullyQualifiedAssemblyName.StartsWith("Opc.Ua.", StringComparison.Ordinal)) sdkAssemblyLoads++;
};
source.Process();
Console.WriteLine(JsonSerializer.Serialize(new
{
traceStartUtc = source.SessionStartTime.ToUniversalTime(),
traceEndUtc = source.SessionEndTime.ToUniversalTime(),
eventsLost = source.EventsLost,
sdkAssemblyLoads,
jittingStarts = starts.Values.Sum(),
// Name keys omit generic signatures; token/module counts independently identify the factory set.
distinctNameKeys = starts.Count,
opcUaJittingStarts = starts.Where(x => x.Key.StartsWith("Opc.Ua.", StringComparison.Ordinal)).Sum(x => x.Value),
invokeStubStarts = starts.Where(x => x.Key.StartsWith("dynamicClass.InvokeStub", StringComparison.Ordinal)).Sum(x => x.Value),
tiers,
summedLoadedNativeMethodBytes = nativeBytes,
factory = new
{
starts = factoryNames.Values.Sum(), distinctNames = factoryNames.Count,
distinctTokens = factoryTokens.Count, moduleCount = factoryModules.Count,
everyNameOnce = factoryNames.Values.All(x => x == 1),
firstMs = double.IsFinite(firstFactoryMs) ? (double?)firstFactoryMs : null,
lastMs = factoryNames.Count == 0 ? (double?)null : lastFactoryMs,
summedLoadedNativeMethodBytes = factoryNativeBytes,
examples = factoryNames.Keys.Take(10).ToArray()
}
}, new JsonSerializerOptions { WriteIndented = true }));
scripts/validation/opcua-lifecycle-memory/build_saturation_observer.py
-46
View File
@@ -1,46 +0,0 @@
#!/usr/bin/env python3
"""Build the opt-in saturation observer without changing product dependencies."""
import argparse
import json
from pathlib import Path
import subprocess
import xml.etree.ElementTree as ET
def build(output, libraries, linux):
output = output.resolve()
output.mkdir(parents=True, exist_ok=False)
project = ET.Element("Project", Sdk="Microsoft.NET.Sdk")
properties = ET.SubElement(project, "PropertyGroup")
for key, value in {"OutputType": "Exe", "TargetFramework": "net10.0", "ImplicitUsings": "enable",
"TreatWarningsAsErrors": "true", "EnableDefaultCompileItems": "false"}.items():
ET.SubElement(properties, key).text = value
items = ET.SubElement(project, "ItemGroup")
ET.SubElement(items, "FrameworkReference", Include="Microsoft.AspNetCore.App")
ET.SubElement(items, "Compile", Include=str(Path(__file__).resolve().parent / "SaturationObserver.cs"))
for name in ["Microsoft.Diagnostics.NETCore.Client", "Microsoft.Diagnostics.Tracing.TraceEvent", "Microsoft.Diagnostics.FastSerialization"]:
path = (libraries / (name + ".dll")).resolve(strict=True)
reference = ET.SubElement(items, "Reference", Include=name)
ET.SubElement(reference, "HintPath").text = str(path)
project_path = output / "observer.csproj"
ET.indent(project)
ET.ElementTree(project).write(project_path, encoding="unicode")
command = ["dotnet", "publish" if linux else "build", str(project_path), "-c", "Release", "-m:2", "--nologo"]
if linux:
command += ["-r", "linux-arm64", "--self-contained", "true", "-p:RuntimeFrameworkVersion=10.0.14",
"-p:PublishSingleFile=false", "-p:PublishTrimmed=false", "-p:UseAppHost=true", "-o", str(output / "published")]
with (output / "build.log").open("w") as log:
result = subprocess.run(command, stdout=log, stderr=subprocess.STDOUT)
(output / "build-command.json").write_text(json.dumps(command, indent=2) + "\n")
if result.returncode:
raise subprocess.CalledProcessError(result.returncode, command)
return output / ("published" if linux else "bin/Release/net10.0")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--libraries", required=True, type=Path)
parser.add_argument("--linux-self-contained", action="store_true")
args = parser.parse_args()
print(build(args.output, args.libraries, args.linux_self_contained))
scripts/validation/opcua-lifecycle-memory/build_trace_tools.py
-38
View File
@@ -1,38 +0,0 @@
#!/usr/bin/env python3
"""Build optional local capture/summary tools using the user's installed dotnet-trace libraries."""
import argparse
from pathlib import Path
import subprocess
import xml.etree.ElementTree as ET
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--libraries", required=True, type=Path, help="dotnet-trace tools/net8.0/any directory")
parser.add_argument("--output", required=True, type=Path, help="Fresh local output directory")
args = parser.parse_args()
args.output = args.output.resolve()
args.output.mkdir(parents=True, exist_ok=False)
for name, source, references in [
("capture", "EventPipeCapture.cs", ["Microsoft.Diagnostics.NETCore.Client"]),
("summary", "TraceSummary.cs", ["Microsoft.Diagnostics.Tracing.TraceEvent", "Microsoft.Diagnostics.FastSerialization", "Microsoft.Diagnostics.NETCore.Client"])
]:
output = args.output / name
output.mkdir()
project = ET.Element("Project", Sdk="Microsoft.NET.Sdk")
properties = ET.SubElement(project, "PropertyGroup")
for key, value in {"OutputType": "Exe", "TargetFramework": "net10.0", "ImplicitUsings": "enable",
"EnableDefaultCompileItems": "false"}.items():
ET.SubElement(properties, key).text = value
items = ET.SubElement(project, "ItemGroup")
ET.SubElement(items, "FrameworkReference", Include="Microsoft.AspNetCore.App")
ET.SubElement(items, "Compile", Include=str(Path(__file__).resolve().parent / source))
for reference in references:
dll = (args.libraries / (reference + ".dll")).resolve(strict=True)
item = ET.SubElement(items, "Reference", Include=reference)
ET.SubElement(item, "HintPath").text = str(dll)
path = output / (name + ".csproj")
ET.indent(project)
ET.ElementTree(project).write(path, encoding="unicode")
with (output / "build.log").open("w") as log:
result = subprocess.run(["dotnet", "build", str(path), "-c", "Release", "-m:2"], stdout=log, stderr=subprocess.STDOUT)
if result.returncode:
raise SystemExit(result.returncode)
scripts/validation/opcua-lifecycle-memory/container_saturation.sh
-63
View File
@@ -1,63 +0,0 @@
#!/usr/bin/env bash
# Diagnostic only. The published xUnit apphost and observer are self-contained .NET 10.0.14.
set -euo pipefail
mkdir -p /task/tmp /task/data /task/results /task/control
export TMPDIR=/task/tmp XDG_DATA_HOME=/task/data
export OTOPCUA_SDK2_SOAK=1 OTOPCUA_SDK2_SOAK_MODE=saturation OTOPCUA_SDK2_SOAK_SECONDS=1800
export OTOPCUA_SDK2_SOAK_OUTPUT=/task/resources.jsonl
export OTOPCUA_SDK2_SATURATION_CONTROL=/task/control
export OPCUA_CERTIFICATE_LEAK_TRACKING=1
export DOTNET_CLI_TELEMETRY_OPTOUT=1 DOTNET_NOLOGO=1
[[ $(uname -s) == Linux && $(uname -m) == aarch64 && -d /proc/self/fd ]]
dotnet --info > /task/controller-runtime.txt
vstest_pid= observer_pid=
cleanup() {
for child in "$vstest_pid" "$observer_pid"; do
if [[ -n "$child" ]] && kill -0 "$child" 2>/dev/null; then kill "$child" 2>/dev/null || true; fi
done
}
trap cleanup EXIT
dotnet vstest /app/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.dll \
--TestCaseFilter:'FullyQualifiedName~Sdk2LifecycleSoakTests.Descriptor_probe_' \
--ResultsDirectory:/task/results '--Logger:trx;LogFileName=descriptor-controls.trx' > /task/descriptor-controls.log 2>&1
/observer/observer /task/control "$OTOPCUA_SDK2_SATURATION_RUN_ID" > /task/observer.log 2>&1 &
observer_pid=$!
dotnet vstest /app/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.dll \
--TestCaseFilter:'FullyQualifiedName=ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.Sdk2LifecycleSoakTests.Repeated_lifecycles_return_resources_to_a_steady_baseline' \
--ResultsDirectory:/task/results '--Logger:trx;LogFileName=saturation.trx' > /task/test.log 2>&1 &
vstest_pid=$!
observed=0
for attempt in $(seq 1 1200); do
if [[ -s /task/resources.jsonl ]]; then
target_pid=$(head -n 1 /task/resources.jsonl | sed -n 's/.*"processId":\([0-9]*\),.*/\1/p')
[[ -n "$target_pid" ]]
target_exe=$(readlink "/proc/$target_pid/exe")
[[ "$target_exe" == /app/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests ]]
printf 'targetPid=%s\nactualExecutable=%s\n' "$target_pid" "$target_exe" > /task/target-runtime.txt
awk '$6 ~ /\/(libcoreclr.so|libhostfxr.so|libhostpolicy.so|System.Private.CoreLib.dll)$/ {print $6}' "/proc/$target_pid/maps" | sort -u >> /task/target-runtime.txt
grep -Fx /app/libcoreclr.so /task/target-runtime.txt > /dev/null
[[ $(readlink "/proc/$observer_pid/exe") == /observer/observer ]]
awk '$6 ~ /\/libcoreclr.so$/ {print $6}' "/proc/$observer_pid/maps" | sort -u > /task/observer-runtime.txt
grep -Fx /observer/libcoreclr.so /task/observer-runtime.txt > /dev/null
observed=1
break
fi
kill -0 "$vstest_pid" 2>/dev/null || break
sleep 0.1
done
[[ "$observed" == 1 ]]
set +e
wait "$vstest_pid"
test_status=$?
printf '%s\n' "$test_status" > /task/test-exit-status.txt
for attempt in $(seq 1 400); do
kill -0 "$observer_pid" 2>/dev/null || break
sleep 0.1
done
if kill -0 "$observer_pid" 2>/dev/null; then kill "$observer_pid"; fi
wait "$observer_pid"
observer_status=$?
printf '%s\n' "$observer_status" > /task/observer-exit-status.txt
# Preserve a cold-cap failure even when diagnostic readiness succeeds.
if [[ "$test_status" != 0 ]]; then exit "$test_status"; fi
exit "$observer_status"
scripts/validation/opcua-lifecycle-memory/evidence.json
-1054
View File
File diff suppressed because it is too large Load Diff
scripts/validation/opcua-lifecycle-memory/launch_saturation.py
-114
View File
@@ -1,114 +0,0 @@
#!/usr/bin/env python3
"""Launch only a previously reviewed immutable candidate saturation bundle. Never an acceptance run."""
import argparse
import hashlib
import json
from pathlib import Path
import subprocess
import time
import uuid
import xml.etree.ElementTree as ET
def digest(path):
return hashlib.sha256(path.read_bytes()).hexdigest()
def require(condition, message):
if not condition:
raise ValueError(message)
def verify(root, expected):
receipt_path = root / "receipt.json"
if digest(receipt_path) != expected:
raise ValueError("Receipt hash differs from the reviewed artifact")
receipt = json.loads(receipt_path.read_text())
for name, value in receipt["files"].items():
if digest(root / name) != value:
raise ValueError("Immutable bundle file changed: " + name)
require(receipt["protocol"] == "sdk2-saturation-v1" and receipt["acceptance"] is False, "Invalid diagnostic protocol")
require((receipt["canonicalCycles"], receipt["completedRestarts"], receipt["settlingCycles"], receipt["maximumPhaseSeconds"]) == (384, 96, 1, 1800), "Changed prospective limits")
return receipt
def trx(path, minimum):
document = ET.parse(path)
counters = document.find(".//{*}Counters")
if counters is None:
raise ValueError("Missing TRX counters")
executed = int(counters.attrib.get("executed", "0"))
skipped = max(int(counters.attrib.get("notExecuted", "0")),
sum(result.get("outcome") == "NotExecuted" for result in document.findall("./{*}Results/{*}UnitTestResult")))
if executed < minimum or skipped:
raise ValueError("Required diagnostic tests were skipped or not executed")
return {"executed": executed, "skipped": skipped}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--bundle", type=Path, required=True)
parser.add_argument("--receipt-sha256", required=True, help="Hash reviewed by root and Astra before launch")
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--verify-only", action="store_true")
options = parser.parse_args()
root = options.bundle.resolve(strict=True)
receipt = verify(root, options.receipt_sha256)
if options.verify_only:
print("Immutable bundle verified; no experiment launched")
return 0
output = options.output.resolve()
output.mkdir(parents=True, exist_ok=False)
run_id = str(uuid.uuid4())
container = "opcua-saturation-" + uuid.uuid4().hex[:12]
command = ["docker", "run", "--rm", "--name", container, "--network", "none",
"-e", "OTOPCUA_SOAK_BUILD_SHA=" + receipt["sourceRevision"],
"-e", "OTOPCUA_SDK2_SATURATION_RUN_ID=" + run_id,
"-v", str(root / "app") + ":/app:ro", "-v", str(root / "observer") + ":/observer:ro",
"-v", str(root / "tools") + ":/tools:ro", "-v", str(output) + ":/task",
receipt["image"], "bash", "/tools/container_saturation.sh"]
(output / "launch.json").write_text(json.dumps({"command": command, "receiptSha256": options.receipt_sha256,
"sourceRevision": receipt["sourceRevision"], "acceptance": False}, indent=2) + "\n")
start = time.monotonic()
try:
with (output / "container.log").open("w") as log:
process = subprocess.run(command, stdout=log, stderr=subprocess.STDOUT, timeout=2040)
status = process.returncode
except subprocess.TimeoutExpired:
status = 124
finally:
subprocess.run(["docker", "stop", "--time", "10", container], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=20)
(output / "container-exit-status.txt").write_text(str(status) + "\n")
results = {"acceptance": False, "containerExitStatus": status, "wallSeconds": time.monotonic() - start,
"sourceRevision": receipt["sourceRevision"], "receiptSha256": options.receipt_sha256}
try:
results["descriptorTests"] = trx(output / "results/descriptor-controls.trx", 2)
results["experimentTests"] = trx(output / "results/saturation.trx", 1)
rows = [json.loads(line) for line in (output / "resources.jsonl").read_text().splitlines()]
header = rows[0]
require(header["mode"] == "saturation" and header["requestedSeconds"] == 1800 and header["build"] == receipt["sourceRevision"], "Wrong executed source or mode")
require(not any(row["kind"] == "complete" for row in rows), "Saturation must never emit an ordinary completion result")
outcome = [row for row in rows if row["kind"] == "saturation-result"]
require(len(outcome) == 1, "Missing unique diagnostic result")
outcome = outcome[0]
require((outcome["canonicalCycles"], outcome["settlingCycles"], outcome["restarts"]) == (384, 1, 96), "Incomplete fixed work")
require(outcome["phaseSeconds"] <= 1800 and outcome["shutdownOwnershipVerified"], "Cutoff or cleanup proof failed")
require(outcome["acceptance"] is False, "This is not an acceptance run")
results["diagnosticOutcome"] = outcome
observer_status = int((output / "observer-exit-status.txt").read_text())
test_status = int((output / "test-exit-status.txt").read_text())
require(observer_status == 0, "Observer failed")
require((test_status != 0) == (outcome["coldBudgetRejected"] or not outcome["readiness"]), "Cold-cap exit was not preserved")
results["protocolCompleted"] = True
except Exception as error:
results["protocolCompleted"] = False
results["validationErrorType"] = type(error).__name__
status = status or 1
verify(root, options.receipt_sha256)
(output / "run-receipt.json").write_text(json.dumps(results, indent=2) + "\n")
print(json.dumps(results, indent=2))
return status
if __name__ == "__main__":
raise SystemExit(main())
scripts/validation/opcua-lifecycle-memory/linux_pair.py
-113
View File
@@ -1,113 +0,0 @@
#!/usr/bin/env python3
"""Run two already-built public lifecycle controls in isolated, offline Linux containers."""
import argparse
from datetime import datetime, timezone
import hashlib
import json
import os
from pathlib import Path
import secrets
import subprocess
import time
import uuid
def latest_sample(path):
# A concurrently-written final line may be incomplete. All complete rows remain in the source log.
for row in reversed(path.read_text().splitlines()):
try:
value = json.loads(row)
except json.JSONDecodeError:
continue
if value.get("kind") == "sample":
return value
return None
def run(options):
options.output = options.output.resolve()
options.output.mkdir(parents=True, exist_ok=False)
processes = []
environment = os.environ.copy()
environment["OPCUA_MEMORY_PASSWORD"] = secrets.token_urlsafe(32)
# Both receive the same environment. SDK1 ignores the SDK2 diagnostic opt-in.
environment["OPCUA_CERTIFICATE_LEAK_TRACKING"] = "1"
started = time.monotonic()
try:
for label, directory in [("baseline", options.baseline), ("candidate", options.candidate)]:
directory = directory.resolve(strict=True)
for required in ["control.dll", "control.deps.json", "control.runtimeconfig.json"]:
if not (directory / required).is_file():
raise FileNotFoundError("Use a complete, canonically-built control output: missing " + required)
output = options.output / label
output.mkdir()
container = "opcua-memory-" + label + "-" + uuid.uuid4().hex[:10]
shell = ('dotnet /app/control.dll /task/private-fixture ' + str(options.seconds) +
' > /task/samples.jsonl 2>&1 & child=$!; echo "$child" > /task/dotnet.pid; '
'wait "$child"; status=$?; echo "$status" > /task/exit-status.txt; exit "$status"')
command = ["docker", "run", "--rm", "--name", container, "--network", "none",
"--env", "OPCUA_MEMORY_PASSWORD", "--env", "OPCUA_CERTIFICATE_LEAK_TRACKING",
"-v", str(directory) + ":/app:ro", "-v", str(output) + ":/task",
options.image, "bash", "-c", shell]
manifest = {"argv": command, "fullGate": False, "seconds": options.seconds,
"environmentNamesOnly": ["OPCUA_MEMORY_PASSWORD", "OPCUA_CERTIFICATE_LEAK_TRACKING"],
"startedUtc": datetime.now(timezone.utc).isoformat(),
"runtimeFiles": {p.name: hashlib.sha256(p.read_bytes()).hexdigest()
for p in sorted(directory.iterdir()) if p.is_file()}}
(output / "command.json").write_text(json.dumps(manifest, indent=2) + "\n")
stream = (output / "container.log").open("w")
process = subprocess.Popen(command, env=environment, stdout=stream, stderr=subprocess.STDOUT)
processes.append((label, container, output, stream, process))
next_sample = started + 60
while any(process.poll() is None for _, _, _, _, process in processes):
if time.monotonic() - started > options.seconds + 240:
raise TimeoutError("Diagnostic controls exceeded bounded shutdown allowance")
if time.monotonic() >= next_sample:
for label, container, output, _, process in processes:
pid_file = output / "dotnet.pid"
if process.poll() is not None or not pid_file.exists():
continue
pid = pid_file.read_text().strip()
if not pid.isdecimal():
raise RuntimeError("Unexpected task-owned process id")
stamp = str(int(time.time()))
latest = latest_sample(output / "samples.jsonl")
with (output / "observations.jsonl").open("a") as observed:
observed.write(json.dumps({"utc": datetime.now(timezone.utc).isoformat(),
"seconds": time.monotonic() - started,
"sample": latest}) + "\n")
for name in ["smaps", "smaps_rollup", "status"]:
with (output / (stamp + "-" + name + ".txt")).open("w") as destination:
result = subprocess.run(["docker", "exec", container, "cat", "/proc/" + pid + "/" + name],
stdout=destination, stderr=subprocess.STDOUT, timeout=15)
if result.returncode and process.poll() is None:
raise RuntimeError(label + " live process diagnostic read failed")
next_sample += 60
time.sleep(.5)
statuses = []
for label, _, output, _, process in processes:
status = process.wait()
(output / "docker-exit-status.txt").write_text(str(status) + "\n")
print(label + " exit " + str(status), flush=True)
statuses.append(status)
return 0 if all(status == 0 for status in statuses) else 1
finally:
for _, container, _, stream, process in processes:
if process.poll() is None:
subprocess.run(["docker", "stop", "--time", "10", container], stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, timeout=20)
process.wait(timeout=20)
stream.close()
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--baseline", type=Path, required=True, help="Already-built control output directory")
parser.add_argument("--candidate", type=Path, required=True, help="Already-built control output directory")
parser.add_argument("--output", type=Path, required=True, help="New local diagnostic directory")
parser.add_argument("--image", required=True, help="Immutable Linux .NET10 image digest")
parser.add_argument("--seconds", type=int, default=1800)
options = parser.parse_args()
if not 1 <= options.seconds <= 1800 or not options.image.startswith("sha256:"):
parser.error("Use an immutable image digest and duration 1..1800; never a full-soak gate")
raise SystemExit(run(options))
scripts/validation/opcua-lifecycle-memory/prepare_saturation.py
-83
View File
@@ -1,83 +0,0 @@
#!/usr/bin/env python3
"""Prepare a fresh immutable candidate bundle. This command never launches the experiment."""
import argparse
from datetime import datetime, timezone
import hashlib
import json
from pathlib import Path
import shutil
import subprocess
from build_saturation_observer import build
def sha(path):
return hashlib.sha256(path.read_bytes()).hexdigest()
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--libraries", type=Path, required=True)
parser.add_argument("--image", required=True)
options = parser.parse_args()
source = Path(__file__).resolve().parent
repo = source.parents[2]
revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=repo, text=True).strip()
if subprocess.check_output(["git", "status", "--porcelain"], cwd=repo, text=True).strip():
raise RuntimeError("Commit and review the exact source before preparing immutable artifacts")
if not options.image.startswith("sha256:"):
raise ValueError("Use an immutable Linux .NET SDK image digest")
root = options.output.resolve()
root.mkdir(parents=True, exist_ok=False)
project = repo / "tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj"
command = ["dotnet", "publish", str(project), "-c", "Release", "-r", "linux-arm64", "--self-contained", "true",
"-p:RuntimeFrameworkVersion=10.0.14", "-p:PublishSingleFile=false", "-p:PublishTrimmed=false", "-p:UseAppHost=true",
"--artifacts-path", str(root / "build"), "-o", str(root / "app"), "-m:2", "--nologo"]
with (root / "publish.log").open("w") as log:
subprocess.run(command, cwd=repo, stdout=log, stderr=subprocess.STDOUT, check=True)
observer = build(root / "observer-build", options.libraries, True)
shutil.copytree(observer, root / "observer")
tools = root / "tools"
tools.mkdir()
for name in ["launch_saturation.py", "container_saturation.sh", "SATURATION.md"]:
shutil.copy2(source / name, tools / name)
manifest = {str(path.relative_to(root)): sha(path) for directory in ["app", "observer", "tools"]
for path in sorted((root / directory).rglob("*")) if path.is_file()}
for config in ["app/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.runtimeconfig.json", "observer/observer.runtimeconfig.json"]:
runtime = json.loads((root / config).read_text())
included = runtime["runtimeOptions"]["includedFrameworks"]
if not included or any(value["version"] != "10.0.14" for value in included):
raise RuntimeError("Published participant is not the prospectively fixed self-contained runtime")
dependencies = json.loads((root / "app/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.deps.json").read_text())["libraries"]
gateway_libraries = [name for name in dependencies if name.startswith("ZB.MOM.WW.HistorianGateway.")]
if not gateway_libraries or any(name.rsplit("/", 1)[-1] != "1.0.0" for name in gateway_libraries):
raise RuntimeError("The reviewed candidate requires the landed HistorianGateway 1.0.0 graph")
metadata = root / "module-info-build"
metadata.mkdir()
(metadata / "module-info.csproj").write_text(
'<Project Sdk="Microsoft.NET.Sdk"><PropertyGroup><OutputType>Exe</OutputType>'
'<TargetFramework>net10.0</TargetFramework><ImplicitUsings>enable</ImplicitUsings>'
'<EnableDefaultCompileItems>false</EnableDefaultCompileItems></PropertyGroup>'
'<ItemGroup><Compile Include="' + str(source / "ArtifactModuleInfo.cs") + '" /></ItemGroup></Project>')
with (metadata / "build.log").open("w") as log:
subprocess.run(["dotnet", "build", str(metadata / "module-info.csproj"), "-c", "Release", "-m:2"],
stdout=log, stderr=subprocess.STDOUT, check=True)
modules = json.loads(subprocess.check_output(["dotnet", str(metadata / "bin/Release/net10.0/module-info.dll"),
str(root), str(root / "app"), str(root / "observer")], text=True))
receipt = {"protocol": "sdk2-saturation-v1", "sourceRevision": revision, "image": options.image,
"rid": "linux-arm64", "runtimeVersion": "10.0.14", "selfContained": True,
"canonicalCycles": 384, "completedRestarts": 96, "settlingCycles": 1, "maximumPhaseSeconds": 1800,
"windows": [[321, 352], [353, 384]], "observerBufferMiB": 16, "runtimeKeywords": "0x18", "rundown": False,
"acceptance": False, "preparedUtc": datetime.now(timezone.utc).isoformat(), "publishCommand": command,
"sdkLibraries": [name for name in dependencies if name.startswith("OPCFoundation.")],
"gatewayLibraries": gateway_libraries, "managedModules": modules,
"testSourceSha256": sha(project.parent / "Sdk2LifecycleSoakTests.cs"),
"observerSourceSha256": sha(source / "SaturationObserver.cs"), "files": manifest}
receipt_path = root / "receipt.json"
receipt_path.write_text(json.dumps(receipt, indent=2) + "\n")
print(json.dumps({"bundle": str(root), "receiptSha256": sha(receipt_path), "sourceRevision": revision,
"launched": False}, indent=2))
if __name__ == "__main__":
main()
scripts/validation/opcua-lifecycle-memory/run.py
-134
View File
@@ -1,134 +0,0 @@
#!/usr/bin/env python3
"""Build opt-in controls against existing published DLLs; never rebuild product code."""
import argparse
import hashlib
import json
import os
from pathlib import Path
import secrets
import subprocess
import sys
import time
import xml.etree.ElementTree as ET
def digest(path):
return hashlib.sha256(path.read_bytes()).hexdigest()
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("kind", choices=["lifecycle", "certificate", "telemetry"])
parser.add_argument("--references", type=Path, action="append", required=True,
help="Existing published DLL directory; first directory wins duplicate names")
parser.add_argument("--output", type=Path, required=True, help="New local artifact directory")
parser.add_argument("--sdk2", action="store_true")
parser.add_argument("--seconds", type=int, default=600)
parser.add_argument("--mode", help="Certificate: public/pfx/shared-copy/sdk-copy; telemetry: dispose/undisposed")
parser.add_argument("--build-only", action="store_true")
parser.add_argument("--certificate-tracking", action="store_true", help="SDK2 opt-in allocation-stack diagnostics")
options = parser.parse_args()
if options.kind != "lifecycle" and not options.sdk2:
parser.error("certificate and telemetry controls require --sdk2")
if not 1 <= options.seconds <= 1800:
parser.error("Diagnostic duration must be 1..1800 seconds; this is never a full soak")
options.output = options.output.resolve()
options.output.mkdir(parents=True, exist_ok=False)
names = {"lifecycle": "LifecycleControl.cs", "certificate": "CertificateControl.cs", "telemetry": "TelemetryControl.cs"}
source = Path(__file__).resolve().parent / names[options.kind]
references = {}
for directory in options.references:
directory = directory.resolve(strict=True)
for dll in sorted(directory.glob("*.dll")):
references.setdefault(dll.name, dll)
if "Opc.Ua.Core.dll" not in references:
parser.error("References must contain actual OPC UA managed assemblies")
project = ET.Element("Project", Sdk="Microsoft.NET.Sdk")
properties = ET.SubElement(project, "PropertyGroup")
for key, value in {"OutputType": "Exe", "TargetFramework": "net10.0", "ImplicitUsings": "enable",
"Nullable": "enable", "EnableDefaultCompileItems": "false",
"DefineConstants": "SDK2" if options.sdk2 else ""}.items():
ET.SubElement(properties, key).text = value
items = ET.SubElement(project, "ItemGroup")
ET.SubElement(items, "FrameworkReference", Include="Microsoft.AspNetCore.App")
ET.SubElement(items, "Compile", Include=str(source))
for name, path in references.items():
reference = ET.SubElement(items, "Reference", Include=path.stem)
ET.SubElement(reference, "HintPath").text = str(path)
project_path = options.output / "control.csproj"
ET.indent(project)
ET.ElementTree(project).write(project_path, encoding="unicode")
manifest = {"kind": options.kind, "sdk2": options.sdk2, "seconds": options.seconds,
"fullSoakGate": False, "source": {source.name: digest(source)},
"referencePrecedence": "first directory wins duplicate names",
"references": {name: digest(path) for name, path in references.items()}}
(options.output / "provenance.json").write_text(json.dumps(manifest, indent=2) + "\n")
with (options.output / "build.log").open("w") as log:
result = subprocess.run(["dotnet", "build", str(project_path), "-c", "Release", "-m:2"],
stdout=log, stderr=subprocess.STDOUT)
if result.returncode:
return result.returncode
if options.build_only:
return 0
executable = options.output / "bin/Release/net10.0/control.dll"
environment = os.environ.copy()
environment["OPCUA_MEMORY_PASSWORD"] = secrets.token_urlsafe(32)
if options.certificate_tracking:
environment["OPCUA_CERTIFICATE_LEAK_TRACKING"] = "1"
if options.kind == "lifecycle":
command = ["dotnet", str(executable), str(options.output / "private-fixture"), str(options.seconds)]
with (options.output / "samples.jsonl").open("w") as log:
result = subprocess.run(command, env=environment, stdout=log, stderr=subprocess.STDOUT,
timeout=options.seconds + 180)
status = result.returncode
elif options.kind == "telemetry":
if options.mode not in ["dispose", "undisposed"]:
parser.error("Telemetry mode must be dispose or undisposed")
with (options.output / "samples.jsonl").open("w") as log:
result = subprocess.run(["dotnet", str(executable), options.mode], env=environment,
stdout=log, stderr=subprocess.STDOUT, timeout=180)
status = result.returncode
else:
if sys.platform != "darwin":
parser.error("Native certificate control currently requires macOS heap/vmmap")
if options.mode not in ["public", "pfx", "shared-copy", "sdk-copy"]:
parser.error("Certificate mode must be public, pfx, shared-copy or sdk-copy")
status = certificate_control(executable, options.output, options.mode, environment)
(options.output / "exit-status.txt").write_text(str(status) + "\n")
return status
def certificate_control(executable, output, mode, environment):
# Native snapshots contain process metadata; retain locally, do not publish them unredacted.
with (output / "samples.jsonl").open("w") as log:
process = subprocess.Popen(["dotnet", str(executable), mode, str(output)], env=environment,
stdout=log, stderr=subprocess.STDOUT)
try:
deadline = time.monotonic() + 180
while not (output / "samples.jsonl").stat().st_size:
if process.poll() is not None or time.monotonic() > deadline:
raise RuntimeError("Certificate control failed before ready")
time.sleep(.05)
for label in ["before", "after"]:
if label == "after":
(output / "go").touch()
while True:
rows = (output / "samples.jsonl").read_text().splitlines()
if rows and json.loads(rows[-1]).get("kind") == "done":
break
if process.poll() is not None or time.monotonic() > deadline:
raise RuntimeError("Certificate control did not finish")
time.sleep(.1)
with (output / ("heap-" + label + ".txt")).open("w") as log_heap:
subprocess.run(["/usr/bin/heap", "--noContent", "-s", str(process.pid)],
stdout=log_heap, stderr=subprocess.STDOUT, timeout=20, check=True)
(output / "stop").touch()
return process.wait(timeout=10)
finally:
if process.poll() is None:
process.terminate()
process.wait(timeout=10)
if __name__ == "__main__":
raise SystemExit(main())
scripts/validation/opcua-lifecycle-memory/saturation_tool_controls.py
-37
View File
@@ -1,37 +0,0 @@
#!/usr/bin/env python3
"""Small negative controls for artifact tampering and VSTest's misleading skipped counter."""
import hashlib
import json
from pathlib import Path
import tempfile
from launch_saturation import trx, verify
with tempfile.TemporaryDirectory(prefix="opcua-saturation-tools-") as temporary:
root = Path(temporary)
artifact = root / "immutable"
artifact.write_text("control")
receipt = {"protocol": "sdk2-saturation-v1", "acceptance": False, "canonicalCycles": 384,
"completedRestarts": 96, "settlingCycles": 1, "maximumPhaseSeconds": 1800,
"files": {"immutable": hashlib.sha256(artifact.read_bytes()).hexdigest()}}
path = root / "receipt.json"
path.write_text(json.dumps(receipt))
digest = hashlib.sha256(path.read_bytes()).hexdigest()
verify(root, digest)
artifact.write_text("tampered")
try:
verify(root, digest)
except ValueError:
pass
else:
raise RuntimeError("Changed artifact was accepted")
xml = root / "skipped.trx"
xml.write_text('<TestRun xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"><Results>'
'<UnitTestResult outcome="NotExecuted" /></Results><ResultSummary>'
'<Counters executed="1" notExecuted="0" /></ResultSummary></TestRun>')
try:
trx(xml, 1)
except ValueError:
pass
else:
raise RuntimeError("Direct skipped result was hidden by the counter")
print("Artifact verification, tamper rejection and direct skipped-result controls passed (3/3)")
scripts/validation/opcua-lifecycle-memory/summarize.py
-45
View File
@@ -1,45 +0,0 @@
#!/usr/bin/env python3
"""Emit mapping-category totals without path names from local Linux /proc/PID/smaps."""
import argparse
from collections import defaultdict
import json
from pathlib import Path
import re
HEADER = re.compile(r"^[0-9a-f]+-[0-9a-f]+\s+(\S+)\s+\S+\s+\S+\s+\S+\s*(.*)$")
METRIC = re.compile(r"^(Size|Rss|Pss|Private_Dirty|Private_Clean|Anonymous):\s+(\d+) kB$")
def summarize(path):
groups = defaultdict(lambda: defaultdict(int))
group = None
for line in path.read_text().splitlines():
header = HEADER.match(line)
if header:
permissions, name = header.groups()
if "doublemapper" in name:
group = "jit-doublemapper"
elif name == "[heap]":
group = "native-heap"
elif name.startswith("[stack"):
group = "stack"
elif not name:
group = "anonymous-exec" if "x" in permissions else "anonymous-write" if "w" in permissions else "anonymous-read"
elif name.endswith(".dll"):
group = "managed-assembly-files"
elif ".so" in name:
group = "native-library-files"
elif name.startswith("["):
group = "other-kernel-mappings"
else:
group = "other-files"
elif group and (metric := METRIC.match(line)):
groups[group][metric[1]] += int(metric[2]) * 1024
return dict(groups)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("files", nargs="+", type=Path)
args = parser.parse_args()
print(json.dumps([{"sample": index, "groups": summarize(path)} for index, path in enumerate(args.files)], indent=2))
scripts/validation/opcua-publish-smoke/Program.cs
-93
View File
@@ -1,93 +0,0 @@
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Opc.Ua;
using Opc.Ua.Client;
using Opc.Ua.Configuration;
using ZB.MOM.WW.OtOpcUa.Core.Authorization;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
using var deadline = new CancellationTokenSource(TimeSpan.FromSeconds(45));
var ct = deadline.Token;
var mode = args[0];
var root = Path.GetFullPath(args[1]);
if (mode == "seed")
{
var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
{ ["LocalDb:Path"] = Path.Combine(root, "localdb.db") }).Build();
await using var provider = new ServiceCollection().AddLogging().AddOtOpcUaLocalDb(config).BuildServiceProvider();
var presented = Environment.GetEnvironmentVariable("SMOKE_API_KEY") ?? throw new InvalidOperationException("Missing task credential.");
if (!ApiKeySecret.TryParse(presented, out var id, out var secret)) throw new InvalidOperationException("Invalid task credential format.");
var pepper = Environment.GetEnvironmentVariable("Security__ApiKeys__Pepper") ?? throw new InvalidOperationException("Missing task pepper.");
var artifact = JsonSerializer.SerializeToUtf8Bytes(new
{
Clusters = new[] { new { ClusterId = "smoke", Name = "Smoke" } },
RawFolders = new[] { new { RawFolderId = "smoke", Name = "Smoke", ClusterId = "smoke" } },
ApiKeys = new[] { new { ApiKeyId = id, Name = "publish-smoke", Principal = "publish-smoke",
Enabled = true, SecretHash = ApiKeySecret.ComputeHash(secret, pepper), CreatedUtc = DateTime.UtcNow } },
NodeAcls = new[] { new { NodeAclId = "smoke-read", ClusterId = "smoke", Principal = "publish-smoke",
ScopeKind = 0, ScopeId = (string?)null, PermissionFlags = 7 } },
});
var revision = Convert.ToHexStringLower(SHA256.HashData(artifact));
await provider.GetRequiredService<IDeploymentArtifactCache>().StoreAsync("smoke", ZB.MOM.WW.OtOpcUa.Commons.Types.DeploymentId.NewId().ToString(), revision, artifact, ct);
Console.WriteLine(JsonSerializer.Serialize(new { mode, result = "seeded-through-public-cache", revision }));
return;
}
var telemetry = DefaultTelemetry.Create(_ => { });
var clientName = mode == "prepare-cli" ? "OtOpcUaClient" : "PublishSmokeClient";
var security = new SecurityConfiguration
{
ApplicationCertificate = new CertificateIdentifier { StoreType = CertificateStoreType.Directory,
StorePath = Path.Combine(root, "own"), SubjectName = "CN=" + clientName },
TrustedIssuerCertificates = new CertificateTrustList { StoreType = CertificateStoreType.Directory, StorePath = Path.Combine(root, "issuer") },
TrustedPeerCertificates = new CertificateTrustList { StoreType = CertificateStoreType.Directory, StorePath = Path.Combine(root, "trusted") },
RejectedCertificateStore = new CertificateTrustList { StoreType = CertificateStoreType.Directory, StorePath = Path.Combine(root, "rejected") },
AutoAcceptUntrustedCertificates = false,
};
var configuration = new ApplicationConfiguration(telemetry)
{
ApplicationName = clientName, ApplicationUri = "urn:localhost:" + clientName,
ApplicationType = ApplicationType.Client, SecurityConfiguration = security,
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
};
await using var application = new ApplicationInstance(telemetry)
{ ApplicationName = configuration.ApplicationName, ApplicationType = ApplicationType.Client, ApplicationConfiguration = configuration };
await configuration.ValidateAsync(ApplicationType.Client, ct);
await application.CheckApplicationInstanceCertificatesAsync(true, lifeTimeInMonths: 12, ct);
if (mode is "prepare" or "prepare-cli")
{
Console.WriteLine(JsonSerializer.Serialize(new { mode, result = "prepared", autoAccept = security.AutoAcceptUntrustedCertificates }));
return;
}
var endpoint = args[2];
var expected = args[3];
var credential = Environment.GetEnvironmentVariable("SMOKE_API_KEY") ?? throw new InvalidOperationException("Missing task credential.");
try
{
var selected = await CoreClientUtils.SelectEndpointAsync(configuration, endpoint, true, telemetry, ct)
?? throw new InvalidOperationException("No secure endpoint advertised.");
if (selected.SecurityMode != MessageSecurityMode.SignAndEncrypt || selected.SecurityPolicyUri != SecurityPolicies.Basic256Sha256)
throw new InvalidOperationException("Unexpected negotiated security policy.");
using var session = await new DefaultSessionFactory(telemetry).CreateAsync(configuration,
new ConfiguredEndpoint(null, selected, EndpointConfiguration.Create(configuration)), false, true,
"PublishSmoke", 60_000, new UserIdentity("apikey", Encoding.UTF8.GetBytes(credential)), default, ct);
if (expected != "allow") throw new InvalidOperationException("A denied test credential or peer was admitted.");
var response = await session.ReadAsync(null, 0, TimestampsToReturn.Both,
[new ReadValueId { NodeId = VariableIds.Server_ServerStatus_State, AttributeId = Attributes.Value }], ct);
if (response.Results.Count != 1 || StatusCode.IsBad(response.Results[0].StatusCode)) throw new InvalidOperationException("Server status read failed.");
Console.WriteLine(JsonSerializer.Serialize(new { mode, expected, result = "allowed", policy = selected.SecurityPolicyUri,
securityMode = selected.SecurityMode.ToString(), status = response.Results[0].StatusCode.ToString(), autoAccept = false }));
await session.CloseAsync(ct);
}
catch (ServiceResultException error) when (expected != "allow")
{
var accepted = expected == "key-deny"
? new[] { StatusCodes.BadUserAccessDenied, StatusCodes.BadIdentityTokenRejected, StatusCodes.BadIdentityTokenInvalid }
: new[] { StatusCodes.BadCertificateUntrusted, StatusCodes.BadSecurityChecksFailed, StatusCodes.BadSecureChannelClosed };
if (!accepted.Contains(error.StatusCode)) throw;
Console.WriteLine(JsonSerializer.Serialize(new { mode, expected, result = "denied", status = error.StatusCode.ToString(), autoAccept = false }));
}
scripts/validation/opcua-publish-smoke/PublishProbe.csproj
-11
View File
@@ -1,11 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="../../../src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj" />
</ItemGroup>
</Project>
scripts/validation/opcua-publish-smoke/README.md
-70
View File
@@ -1,70 +0,0 @@
# Actual Host publish smoke
From an isolated candidate checkout, publish the actual Host and CLI for Linux ARM64
and Windows x64 using the install recipe's self-contained setting. Do not use a
checkout that another process is building, and choose a new artifact directory.
```bash
task_output=/absolute/fresh/publish-smoke
test ! -e "$task_output" || exit 2
mkdir -p "$task_output"
for rid in linux-arm64 win-x64; do
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj \
-c Release -r "$rid" --self-contained true \
-p:PublishSingleFile=false -p:PublishTrimmed=false -m:1 \
-o "$task_output/$rid/host" || exit
dotnet publish src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/ZB.MOM.WW.OtOpcUa.Client.CLI.csproj \
-c Release -r "$rid" --self-contained true \
-p:PublishSingleFile=false -p:PublishTrimmed=false -m:1 \
-o "$task_output/$rid/cli" || exit
done
dotnet publish scripts/validation/opcua-publish-smoke/PublishProbe.csproj \
-c Release -m:1 -o "$task_output/probe" || exit
python3 scripts/validation/opcua-publish-smoke/runtime.py --artifacts "$task_output"
```
The cached `mcr.microsoft.com/dotnet/sdk:10.0.103-noble` image must be ARM64 for this
Linux artifact. The SDK image supplies the small probe's framework; the actual
Host and CLI are launched through their published self-contained apphosts. There
is no Native AOT, trimming, or single-file transformation.
The runtime script creates a fresh private state directory, ephemeral keys and
loopback ports. A temporary preparation container seeds the LocalDb cache through
public production registration/cache APIs. The artifact contains an empty driver
configuration, one raw folder, and a real HMAC API key with read/browse/subscribe
grants. No production authentication seam is replaced. The actual Host uses
`Cluster:Roles=driver`, `ConfigSource:Mode=FetchAndCache`, an unreachable loopback
central endpoint, and task-only LocalDb, secrets SQLite and GUID PKI paths. Its
single-node Akka seed is local. Every container has `--network none`; no fleet
endpoint, ConfigDb, existing container, volume, or PKI is used.
One task-owned container runs the actual Host as its entrypoint, stops cleanly,
then starts the same executable with the same state and PKI again. Each boot must
serve healthy readiness, liveness and active endpoints, admit a trusted
Basic256Sha256 SignAndEncrypt API-key client, reject a wrong key, and allow an
actual published CLI read of ServerStatus.State with Good quality. The first boot
also rejects an untrusted server and an untrusted client. The two certificate
thumbprints must match. Both stops must exit zero without an OOM kill. An explicit
execution guard requires two boots and all six strict SDK probe cases.
The probe retains its public SDK ApplicationInstance throughout its session and
disposes it afterward. Peer autoaccept is false. The CLI's existing configuration
sets autoaccept true and has no strict-validation flag, so its successful reads
are not counted as strict peer-validation coverage. Its certificate is prepared
with its normal application name and URI using public SDK APIs and trusted by the
server before startup. A closed-port CLI preflight is an expected nonzero setup
check, not a passing connection case.
`commands.json`, per-case logs, health JSON and `summary.json` retain actual exits,
results and certificate identity. Secrets are supplied through a mode-0600 task
environment file, never written into recorded arguments, and the file is removed
at the end. Raw process output and this run's appended rolling-file output are
checked for task secrets before redacted copies are saved. Only the captured,
task-labelled container is removed; private bind-mounted state remains local for
inspection and must not be redistributed with release artifacts.
Windows cross-publish and PE/native/dependency inspection can run on macOS.
Windows service installation and start/stop remain a separate operator gate on
an approved Windows environment. The Wonder-box remote-execution hold still
applies: no WinRM, psbridge, or replacement remote-execution channel. This smoke
does not complete backend, load, failover, or two-hour soak validation.
scripts/validation/opcua-publish-smoke/runtime.py
-152
View File
@@ -1,152 +0,0 @@
#!/usr/bin/env python3
"""Smoke actual self-contained Linux Host/CLI artifacts with task-only PKI and LocalDb."""
import argparse, base64, datetime, hashlib, json, os, re, secrets, shutil, subprocess, time, uuid
from pathlib import Path
p = argparse.ArgumentParser(description=__doc__)
p.add_argument('--artifacts', required=True, type=Path)
p.add_argument('--image', default='mcr.microsoft.com/dotnet/sdk:10.0.103-noble')
a = p.parse_args(); out = a.artifacts.resolve()
run = out / ('runtime-' + uuid.uuid4().hex); run.mkdir(exist_ok=False)
state = run / 'state'; state.mkdir(); os.chmod(state, 0o700)
identifier = uuid.uuid4().hex[:12]
web, ua, akka = [20000 + secrets.randbelow(10000) for _ in range(3)]
while len({web, ua, akka}) != 3: web, ua, akka = [20000 + secrets.randbelow(10000) for _ in range(3)]
key = 'otua_' + secrets.token_hex(8) + '_' + secrets.token_urlsafe(32)
pepper, fetch = secrets.token_urlsafe(32), secrets.token_urlsafe(32)
master = base64.b64encode(secrets.token_bytes(32)).decode()
env = {
'ASPNETCORE_URLS': f'http://127.0.0.1:{web}', 'ASPNETCORE_ENVIRONMENT': 'Production',
'Cluster__Roles__0': 'driver',
'Cluster__SystemName': 'otopcua', 'Cluster__Hostname': '127.0.0.1',
'Cluster__PublicHostname': '127.0.0.1', 'Cluster__Port': str(akka),
'Cluster__SeedNodes__0': f'akka.tcp://otopcua@127.0.0.1:{akka}',
'ConfigSource__Mode': 'FetchAndCache', 'ConfigSource__CentralFetchEndpoints__0': 'http://127.0.0.1:1',
'ConfigSource__ApiKey': fetch, 'ConfigSource__FetchTimeoutSeconds': '1',
'LocalDb__Path': '/state/localdb.db', 'Secrets__SqlitePath': '/state/secrets.db',
'ZB_SECRETS_MASTER_KEY': master, 'Security__ApiKeys__Pepper': pepper,
'Security__Ldap__Enabled': 'false', 'Security__Acl__Enabled': 'false',
'OpcUa__ApplicationName': 'PublishSmoke-' + identifier,
'OpcUa__ApplicationUri': 'urn:localhost:PublishSmoke:' + identifier,
'OpcUa__PublicHostname': 'localhost', 'OpcUa__OpcUaPort': str(ua),
'OpcUa__PkiStoreRoot': '/state/server-pki-' + identifier,
'OpcUa__AllowAnonymous': 'false', 'OpcUa__AutoAcceptUntrustedClientCertificates': 'false',
'OpcUa__EnabledSecurityProfiles__0': 'Basic256Sha256SignAndEncrypt',
'SMOKE_API_KEY': key, 'XDG_DATA_HOME': '/state/cli-data', 'HOME': '/state/home',
}
envfile = run / 'private.env'; envfile.write_text(''.join(f'{k}={v}\n' for k,v in env.items())); os.chmod(envfile, 0o600)
redactions = [key, key.split('_', 2)[2], pepper, fetch, master]
def clean(text):
for secret in redactions: text = text.replace(secret, '<ephemeral-secret>')
return text
records = []
raw_secret_findings = []
rolling_root = out / 'linux-arm64/host/logs'
rolling_offsets = {p: p.stat().st_size for p in rolling_root.glob('*') if p.is_file()}
def command(args, name, expected=0, timeout=90):
start = datetime.datetime.now(datetime.timezone.utc).isoformat()
result = subprocess.run(args, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout)
raw_secret_findings.append({'log': name, 'occurrences': sum(result.stdout.count(secret) for secret in redactions)})
(run / (name + '.log')).write_text(clean(result.stdout))
records.append({'name': name, 'utc': start, 'argv': [clean(str(x)) for x in args], 'exit': result.returncode})
(run / 'commands.json').write_text(json.dumps(records, indent=2))
if expected is not None and result.returncode != expected:
raise RuntimeError(f'{name}: exit {result.returncode}; see {run / (name + ".log")}')
return result
mounts = ['--mount', f'type=bind,src={out},dst=/work', '--mount', f'type=bind,src={state},dst=/state']
base = ['docker','run','--rm','--network','none','--label','codex.task=opcua-publish-smoke','--env-file',str(envfile),*mounts,'-w','/work/linux-arm64/host',a.image]
probe = ['dotnet','/work/probe/PublishProbe.dll']
cli = ['/work/linux-arm64/cli/ZB.MOM.WW.OtOpcUa.Client.CLI']
server_pki = state / ('server-pki-' + identifier)
container = None; summary = {'run': str(run), 'image': a.image, 'ports': {'http':web,'ua':ua,'akka':akka}, 'boots':[], 'cases':[]}
def copy_public(source, target):
target.mkdir(parents=True, exist_ok=True)
certs = list(source.glob('*.der'))
if not certs: raise RuntimeError(f'No public certificates at {source}')
for cert in certs: shutil.copy2(cert, target / cert.name)
def inspect(): return json.loads(subprocess.check_output(['docker','inspect',container], text=True))[0]
def save_logs(name):
r = subprocess.run(['docker','logs',container],text=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
raw_secret_findings.append({'log': name, 'occurrences': sum(r.stdout.count(secret) for secret in redactions)})
(run / (name + '.log')).write_text(clean(r.stdout))
def exec_probe(name, pki, expected, credential=None):
args = ['docker','exec']
if credential: args += ['-e','SMOKE_API_KEY=' + credential]
args += [container,*probe,'probe',pki,f'opc.tcp://localhost:{ua}/OtOpcUa',expected]
r = command(args,name)
payload = json.loads(r.stdout.strip().splitlines()[-1]); summary['cases'].append({'name':name,**payload})
try:
command([*base,*probe,'seed','/state'],'seed')
command([*base,*probe,'prepare','/state/client-pki'],'prepare-trusted-client')
command([*base,*probe,'prepare','/state/untrusted-client-pki'],'prepare-untrusted-client')
command([*base,*cli,'--help'],'cli-help')
command([*base,*probe,'prepare-cli','/state/cli-data/OtOpcUaClient/pki'],'prepare-cli-certificate')
failed = command([*base,*cli,'read','--url','opc.tcp://localhost:1/OtOpcUa','--security','encrypt','--username','apikey','--password',key,'--node','i=2259'],'cli-prepare-on-closed-port',expected=None)
if failed.returncode == 0: raise RuntimeError('Closed-port CLI fixture preparation unexpectedly succeeded.')
copy_public(state/'client-pki/own/certs',server_pki/'trusted/certs')
copy_public(state/'cli-data/OtOpcUaClient/pki/own/certs',server_pki/'trusted/certs')
name = 'otopcua-sdk2-publish-' + identifier
result = command(['docker','create','--name',name,'--network','none','--label','codex.task=opcua-publish-smoke','--env-file',str(envfile),*mounts,'-w','/work/linux-arm64/host',a.image,'./OtOpcUa.Host'],'container-create')
container = result.stdout.strip()
summary['container'] = container
for boot in [1,2]:
command(['docker','start',container],f'boot{boot}-start')
deadline = time.monotonic() + 90
health = None
while time.monotonic() < deadline:
if not inspect()['State']['Running']:
save_logs(f'boot{boot}-host'); raise RuntimeError('Full Host exited before health readiness.')
r = subprocess.run(['docker','exec',container,'curl','-fsS',f'http://127.0.0.1:{web}/health/ready'],text=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
if r.returncode == 0:
health = r.stdout; break
time.sleep(1)
if health is None:
save_logs(f'boot{boot}-host'); raise RuntimeError('Full Host readiness did not become healthy.')
if json.loads(health).get('status') != 'Healthy': raise RuntimeError('Readiness is degraded or unhealthy.')
(run/f'boot{boot}-ready.json').write_text(health)
command(['docker','exec',container,'curl','-fsS',f'http://127.0.0.1:{web}/healthz'],f'boot{boot}-live')
command(['docker','exec',container,'curl','-fsS',f'http://127.0.0.1:{web}/health/active'],f'boot{boot}-active')
certs = list((server_pki/'own/certs').glob('*.der'))
if len(certs) != 1: raise RuntimeError('Expected exactly one application certificate.')
cert_sha256 = hashlib.sha256(certs[0].read_bytes()).hexdigest()
thumbprint = subprocess.check_output(['openssl','x509','-inform','DER','-in',str(certs[0]),'-noout','-fingerprint','-sha1'],text=True).strip()
copy_public(server_pki/'own/certs',state/'client-pki/trusted/certs')
copy_public(server_pki/'own/certs',state/'untrusted-client-pki/trusted/certs')
exec_probe(f'boot{boot}-trusted-key','/state/client-pki','allow')
exec_probe(f'boot{boot}-wrong-key','/state/client-pki','key-deny',key+'-wrong')
if boot == 1:
hold = state/'hold-server-trust'; shutil.move(str(state/'client-pki/trusted/certs'),hold)
(state/'client-pki/trusted/certs').mkdir()
try: exec_probe('untrusted-server','/state/client-pki','peer-deny')
finally:
shutil.rmtree(state/'client-pki/trusted/certs'); shutil.move(str(hold),state/'client-pki/trusted/certs')
exec_probe('untrusted-client','/state/untrusted-client-pki','peer-deny')
r = command(['docker','exec',container,*cli,'read','--url',f'opc.tcp://localhost:{ua}/OtOpcUa','--security','encrypt','--username','apikey','--password',key,'--node','i=2259'],f'boot{boot}-actual-cli-read')
if not re.search(r'^Status:\s+Good(?:\s|$)', r.stdout, re.MULTILINE): raise RuntimeError('Actual CLI did not report a Good server status read.')
command(['docker','stop','--time','30',container],f'boot{boot}-stop',timeout=45)
stopped = inspect()['State']; save_logs(f'boot{boot}-host')
if stopped['Running'] or stopped['ExitCode'] != 0 or stopped['OOMKilled']: raise RuntimeError('Full Host did not shut down cleanly.')
summary['boots'].append({'boot':boot,'certificate_sha256':cert_sha256,'thumbprint':thumbprint,'exit':stopped['ExitCode'],'oomKilled':stopped['OOMKilled'],'health_ready':health})
if summary['boots'][0]['thumbprint'] != summary['boots'][1]['thumbprint']: raise RuntimeError('Application certificate changed between boots.')
summary['certificate_preserved'] = True
for path in rolling_root.glob('*'):
if not path.is_file(): continue
text = path.read_bytes()[rolling_offsets.get(path, 0):].decode(errors='replace')
raw_secret_findings.append({'log': 'rolling-' + path.name, 'occurrences': sum(text.count(secret) for secret in redactions)})
(run / ('rolling-' + path.name)).write_text(clean(text))
if len(summary['boots']) != 2 or len(summary['cases']) != 6: raise RuntimeError('Required smoke cases did not all execute.')
if any(x['occurrences'] for x in raw_secret_findings): raise RuntimeError('Task secret appeared in raw application output; see recorded occurrence counts.')
summary['result'] = 'passed'
finally:
if container:
save_logs('host-final')
owned = inspect()
if owned['Config']['Labels'].get('codex.task') != 'opcua-publish-smoke': raise RuntimeError('Task container ownership changed.')
command(['docker','rm','-f',container],'container-cleanup')
logs = '\n'.join(x.read_text(errors='replace') for x in run.glob('*.log'))
summary['task_secret_occurrences_in_saved_logs'] = sum(logs.count(secret) for secret in redactions)
summary['raw_log_secret_findings'] = raw_secret_findings
summary['completed_utc'] = datetime.datetime.now(datetime.timezone.utc).isoformat()
(run/'summary.json').write_text(json.dumps(summary,indent=2))
envfile.unlink(missing_ok=True)
print(json.dumps(summary,indent=2))
scripts/validation/opcua-scripted-alarm-baseline/README.md
-35
View File
@@ -1,35 +0,0 @@
# Independent scripted-alarm scope baseline
This SDK 1.5.378.182-only source probe records the existing opaque-ID behavior
tracked by [#797](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/797).
It is deliberately outside the candidate solution and is not a runtime adapter.
Use a clean, separate checkout at original application commit
`83ed880e4471344deaf008da9bd028c7e170bdc7`. Confirm its package pins before
running. Copy **only** `ScriptedScopeProbe.cs` temporarily into that checkout's
`tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/`, refusing to
overwrite an existing file. Run:
```sh
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests \
-c Release --filter FullyQualifiedName~BaselineScriptedScopeProbe \
--logger trx --results-directory /absolute/fresh-baseline-evidence
```
Preserve the test exit and run the baseline checkout's
`scripts/ci/assert-not-all-skipped.sh` with `MIN_EXECUTED=1` against the fresh TRX.
Remove only the temporary probe you copied after the run, keeping the source
and TRX evidence here/outside the checkout. Verify baseline tracked files remain
unchanged. Never load this assembly or old SDK into the candidate process.
The real session has no application-wide roles. The production-valid opaque
scripted ID is denied by an Equipment grant, routes nothing, and is then allowed
by an explicit Cluster grant. A passing probe pins **both** outcomes; it does not
claim narrow equipment authorization works. The independent candidate counterpart
is `Sdk2OpaqueScriptedScopeBaselineTests`.
Do not make the ID slash-qualified to get a positive result: the current UI
rejects `/`, and that would hide the production scope-resolution defect. A future
fix requires reviewed ownership/scope binding without changing opaque IDs or
widening permissions. This upgrade preserves the baseline denial and links the
separate defect.
scripts/validation/opcua-scripted-alarm-baseline/ScriptedScopeProbe.cs
-44
View File
@@ -1,44 +0,0 @@
// Deliberately compiled only in a temporary SDK1.5 baseline integration checkout; never in the candidate solution.
using Opc.Ua;
using Opc.Ua.Client;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using static ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.ServerEventRig;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
public sealed class BaselineScriptedScopeProbe
{
[Fact]
public async Task Opaque_scripted_ID_is_denied_by_its_equipment_grant_but_allowed_by_a_cluster_grant()
{
var ct = TestContext.Current.CancellationToken;
await using var rig = await StartAsync("BaselineScriptedScope", encrypted: false, ct);
rig.KeyRoles = [];
const NodePermissions permissions = NodePermissions.ReadOnly | NodePermissions.AlarmAcknowledge | NodePermissions.MethodCall;
rig.Enforce(true, [Key()], new NodeAcl
{
NodeAclId = "narrow", ClusterId = ServerEventRig.Cluster, Principal = KeyPrincipal,
ScopeKind = NodeAclScopeKind.Equipment, ScopeId = $"eq-{EqA}", PermissionFlags = permissions,
});
rig.MaterialiseScripted();
var routed = 0;
rig.Nm.AlarmCommandRouter = _ => routed++;
using var session = await rig.ConnectAsync(anonymous: false, ct);
rig.Project(false, new AlarmConditionSnapshot(true, true, true, true, AlarmShelvingKind.Unshelved, 500, "scope"));
var condition = rig.Nm.TryGetAlarmCondition(ScriptedAlarmId, AddressSpaceRealm.Uns).ShouldNotBeNull();
var eventId = condition.EventId.ShouldNotBeNull().Value;
var denied = await Should.ThrowAsync<ServiceResultException>(() => session.CallAsync(condition.NodeId,
MethodIds.AcknowledgeableConditionType_Acknowledge, ct, new object[] { eventId, new LocalizedText("denied") }));
denied.StatusCode.ShouldBe(StatusCodes.BadUserAccessDenied);
routed.ShouldBe(0);
rig.Enforce(true, [Key()], ClusterGrant(KeyPrincipal, "cluster-positive", permissions));
await session.CallAsync(condition.NodeId, MethodIds.AcknowledgeableConditionType_Acknowledge, ct,
new object[] { eventId, new LocalizedText("allowed") });
routed.ShouldBe(1);
await session.CloseAsync(ct);
}
}
scripts/validation/opcua-server-lifecycle-linux/README.md
+6 -38
View File
@@ -27,20 +27,15 @@ python3 scripts/validation/opcua-server-lifecycle-linux/run.py \
An already published artifact can be reused with
`--published-app /absolute/path/to/app`. Its parent must contain the launcher's
`publish-receipt.json`; the test source, apphost and test assembly hashes must
match. Reuse only a publish produced from the current canonical test source:
match. The recorded diagnostic artifact can be reused with:
```sh
python3 scripts/validation/opcua-server-lifecycle-linux/run.py \
python3 /Users/dohertj2/.codex/worktrees/opcua2-client-lifecycle/OtOpcUa/scripts/validation/opcua-server-lifecycle-linux/run.py \
--full --duration 7200 \
--published-app /absolute/path/to/current-diagnostic/app \
--output /tmp/otopcua-server-linux-full-another-unique
--published-app /tmp/opcua2-server-linux-diagnostic-20261110a/app \
--output /tmp/opcua2-server-linux-full-20261110
```
The historical 2026-11-10 diagnostic publish predates the later pre-assert/GC/JIT
instrumentation. Its recorded full-run command is retained in the failed-run
receipt; a current launcher correctly rejects that publish's test-source hash.
Republish to measure the current instrumented test.
The output directory must be fresh. It is mode 0700 and can contain task-owned
private PKI under `task/tmp` during a run or after a failure. Never commit the
entire output. The existing successful test removes its own stores. Only selected
@@ -51,11 +46,8 @@ verified diagnostic), RID `linux-arm64`, self-contained runtime 10.0.14, explici
`UseAppHost=true`, `PublishSingleFile=false`, `PublishTrimmed=false`, one MSBuild
node, and an external task-owned artifacts directory. All runtime packs were
already cached. No production or test source is patched during publish. The
launcher requires an empty production/package source diff against the reviewed
`cd60e469` production tree and
launcher requires an empty production/package source diff against `7b353d7b` and
records the actual published revision separately from the launcher revision.
The reference was advanced from `7b353d7b` after source review of comment-only
production edits; historical receipts retain their actual source/reference hashes.
That is source equivalence, not a claim that separately built Linux and macOS
assemblies have identical hashes. The full publish file hashes, exact test source
hash, runtimeconfig and SDK dependency versions are retained.
@@ -109,31 +101,7 @@ TRX result. Both descriptor tests passed: the symlink control observed
was reaped in 267 ms. All quiescent logical-resource counts and PKI FDs were zero.
Third-cycle to final deltas were **−145,080 managed bytes, +53,739,520 RSS bytes,
and +2 FDs**, within the unchanged diagnostic limits. These short observations do
not establish a two-hour resource trend. The attempted full Linux gate failed as recorded below.
not establish a two-hour resource trend. The full Linux gate remains pending.
CLI rejection of `--full --duration 90`, rejection of a short completion as a full
run, and rejection of the wrong TRX executed count were also verified. Selected
receipts are in `evidence/2026-11-10-diagnostic/`.
The actual full attempt on 2026-11-10 **failed** at 248 cycles, 61 restarts and
903.508 seconds: RSS353,931,264 exceeded351,850,496 bytes (third-cycle baseline
+128MiB). Managed memory (~20.5MB), global descriptors189, PKI descriptors0,
cached stores2 and logical collections remained bounded. This is a combined
in-process SDK client/server measurement, not standalone server attribution.
The test/container/launcher returned failure; no completion or two-hour pass
exists. Selected sanitized source/runtime/hash/TRX/resource receipts are in
[`server-linux-soak-failed.json`](../../../docs/plans/evidence/2026-11-10-opcua2/server-linux-soak-failed.json).
[#799](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/799) tracks the
cross-platform investigation; unchanged thresholds remain mandatory.
The canonical test additionally supports explicit `OTOPCUA_SDK2_SOAK_MODE=profile`
for 60–1800 seconds when invoked directly with the same opt-in/output/build
environment. This launcher deliberately exposes diagnostic/full modes only.
Profile mode logs each captured resource tuple before assertions, including
GC committed/fragmented bytes, JIT method/IL counts, timers and threads. After a
resource-budget failure it may continue bounded diagnostic collection, labels
subsequent rows `quiescent-budget-rejected`, and rethrows the first original
assertion before any completion record. It never changes thresholds, produces a
two-hour flag, or turns a rejected run into a pass. Full/diagnostic modes retain
immediate assertion failure. Native/private/file/anonymous RSS profiling and a
matched original-SDK control are needed before attributing the failed gate.
scripts/validation/opcua-server-lifecycle-linux/run.py
+2 -2
View File
@@ -17,7 +17,7 @@ ROOT = HERE.parents[2]
PROJECT = ROOT / 'tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj'
APPNAME = 'ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests'
TARGET = 'Repeated_lifecycles_return_resources_to_a_steady_baseline'
BASE = 'cd60e469a100412ee08f598accad07a6959778ba'
BASE = '7b353d7bae5311a2f0a33c0a82c88d2c5b49d76f'
def write(path, value):
@@ -97,7 +97,7 @@ def main():
revision = subprocess.check_output(['git', '-C', str(ROOT), 'rev-parse', 'HEAD'], text=True).strip()
production_diff = subprocess.check_output(['git', '-C', str(ROOT), 'diff', BASE, 'HEAD', '--', 'src', 'Directory.Packages.props', 'Directory.Build.props'], text=True)
if production_diff:
raise RuntimeError('Production/package source differs from reviewed cd60e469 source; review required')
raise RuntimeError('Production/package source differs from recorded 7b353d7b compatibility build; review required')
image = json.loads(subprocess.check_output(['docker', 'image', 'inspect', args.image, '--format', '{{json .}}'], text=True))
if image['Architecture'] != 'arm64' or image['Os'] != 'linux':
raise RuntimeError('Expected cached official Linux ARM64 image')
scripts/validation/scadabridge-sdk-compatibility/Directory.Build.props
-9
View File
@@ -1,9 +0,0 @@
<Project>
<!-- A separate consumer process: never inherit this repository's SDK2 package policy. -->
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable><ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<ManagePackageVersionsCentrally>false</ManagePackageVersionsCentrally>
</PropertyGroup>
</Project>
scripts/validation/scadabridge-sdk-compatibility/Directory.Packages.props
-1
View File
@@ -1 +0,0 @@
<Project><PropertyGroup><ManagePackageVersionsCentrally>false</ManagePackageVersionsCentrally></PropertyGroup></Project>
scripts/validation/scadabridge-sdk-compatibility/Probe.csproj
-9
View File
@@ -1,9 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup><OutputType>Exe</OutputType></PropertyGroup>
<ItemGroup>
<ProjectReference Include="$(ScadaBridgeSourceRoot)/src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.csproj" />
</ItemGroup>
<Target Name="RequireConsumerSource" BeforeTargets="CollectPackageReferences">
<Error Condition="'$(ScadaBridgeSourceRoot)' == ''" Text="Pass -p:ScadaBridgeSourceRoot=/absolute/ScadaBridge; this probe must use the actual consumer and its own pinned SDK." />
</Target>
</Project>
scripts/validation/scadabridge-sdk-compatibility/Program.cs
-129
View File
@@ -1,129 +0,0 @@
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Text.Json;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Serialization;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms;
using ZB.MOM.WW.ScadaBridge.Commons.Types.DataConnections;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer;
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters;
if (args.Length != 3) throw new ArgumentException("Usage: Probe <endpoint> <None|SignAndEncrypt> <new-result-directory>");
var endpoint = args[0];
var mode = Enum.Parse<OpcUaSecurityMode>(args[1]);
var root = Path.GetFullPath(args[2]);
if (Directory.Exists(root)) throw new IOException("Refusing to reuse an evidence/PKI directory.");
Directory.CreateDirectory(root);
using var deadline = new CancellationTokenSource(TimeSpan.FromMinutes(2));
var ct = deadline.Token;
var cases = new List<object>();
var failures = 0;
var options = new OpcUaGlobalOptions
{
ApplicationName = "ScadaBridge-SDK2-compatibility",
TrustedIssuerStorePath = Path.Combine(root, "pki", "issuers"),
TrustedPeerStorePath = Path.Combine(root, "pki", "trusted"),
RejectedCertificateStorePath = Path.Combine(root, "pki", "rejected"),
};
// Populate only this probe's trust list; never change the consumer's existing stores.
var trustSource = Environment.GetEnvironmentVariable("SCADABRIDGE_PROBE_SERVER_CERTS");
if (!string.IsNullOrEmpty(trustSource))
{
var destination = Path.Combine(options.TrustedPeerStorePath, "certs");
Directory.CreateDirectory(destination);
foreach (var cert in Directory.GetFiles(trustSource, "*.der")) File.Copy(cert, Path.Combine(destination, Path.GetFileName(cert)));
}
var factory = new RealOpcUaClientFactory(options, NullLoggerFactory.Instance);
await using var adapter = new OpcUaDataConnection(factory, NullLogger<OpcUaDataConnection>.Instance);
var config = new OpcUaEndpointConfig
{
EndpointUrl = endpoint, SecurityMode = mode, AutoAcceptUntrustedCerts = false,
PublishingIntervalMs = 100, SamplingIntervalMs = 100,
UserIdentity = new OpcUaUserIdentityConfig
{
TokenType = OpcUaUserTokenType.UsernamePassword, Username = "apikey",
Password = Environment.GetEnvironmentVariable("OPCUA_MATRIX_PASSWORD") ?? throw new InvalidOperationException("Missing task credential"),
},
};
static void Require(bool value, string reason) { if (!value) throw new InvalidOperationException(reason); }
async Task Check(string name, Func<Task> action)
{
try { await action(); cases.Add(new { name, status = "passed" }); }
catch (Exception e) { failures++; cases.Add(new { name, status = "failed", error = e.ToString() }); }
}
async Task Wait(Func<bool> predicate)
{
for (var i = 0; i < 200; i++) { if (predicate()) return; await Task.Delay(50, ct); }
Require(false, "No expected consumer notification within 10 seconds.");
}
const string raw = "nsu=https://zb.com/otopcua/raw;s=Matrix/dev/";
const string uns = "nsu=https://zb.com/otopcua/uns;s=Eq/";
await Check("actual-adapter-connect-api-key", () => adapter.ConnectAsync(OpcUaEndpointConfigSerializer.ToFlatDict(config), ct));
if (failures == 0)
{
foreach (var prefix in new[] { raw, uns })
{
await Check($"scalar-round-trip-{prefix}", async () =>
{
var write = await adapter.WriteAsync(prefix + "Scalar", 412, ct);
Require(write.Success, write.ErrorMessage ?? "Scalar write denied");
var read = await adapter.ReadAsync(prefix + "Scalar", ct);
Require(read.Success && read.Value?.Value is int i && i == 412, "Scalar value/type changed");
});
await Check($"array-round-trip-{prefix}", async () =>
{
var write = await adapter.WriteAsync(prefix + "Vector", new[] { 31, 32, 33 }, ct);
Require(write.Success, write.ErrorMessage ?? "Array write denied");
var read = await adapter.ReadAsync(prefix + "Vector", ct);
Require(read.Success && read.Value?.Value is int[] a && a.SequenceEqual(new[] { 31, 32, 33 }), "Array value/type changed");
});
}
await Check("actual-adapter-durable-browse", async () =>
{
var page = await adapter.BrowseChildrenAsync("nsu=https://zb.com/otopcua/raw;s=Matrix/dev", cancellationToken: ct);
Require(page.Children.Count(x => x.DisplayName is "Scalar" or "Vector") == 2, "Missing durable raw variable children");
Require(page.Children.Any(x => x.DisplayName == "Alarm"), "Missing native alarm under its raw parent");
var projected = await adapter.BrowseChildrenAsync("nsu=https://zb.com/otopcua/uns;s=Eq", cancellationToken: ct);
Require(projected.Children.Any(x => x.DisplayName == "Vector"), "Missing UNS variable child");
});
await Check("actual-adapter-monitor-quality-time", async () =>
{
var values = new ConcurrentQueue<TagValue>();
var handle = await adapter.SubscribeAsync(raw + "Scalar", (_, value) => values.Enqueue(value), ct);
Require((await adapter.WriteAsync(raw + "Scalar", 713, ct)).Success, "Monitored write denied");
await Wait(() => values.Any(x => x.Value is int i && i == 713));
var actual = values.Last(x => x.Value is int i && i == 713);
Require(actual.Timestamp > new DateTimeOffset(2026, 10, 1, 0, 0, 0, TimeSpan.Zero), "Missing source timestamp");
Require(actual.Quality.ToString() == "Good", "Quality was not Good");
await adapter.UnsubscribeAsync(handle, ct);
});
await Check("actual-adapter-alarm-filter-refresh", async () =>
{
var alarms = new ConcurrentQueue<NativeAlarmTransition>();
var handle = await adapter.SubscribeAlarmsAsync("i=2253", null, alarms.Enqueue, ct);
await Wait(() => alarms.Any(x => x.Kind == AlarmTransitionKind.SnapshotComplete)
&& alarms.Any(x => x.Message == "matrix-alarm" && x.SourceReference.Contains("Alarm", StringComparison.Ordinal)));
await adapter.UnsubscribeAlarmsAsync(handle, ct);
});
await Check("actual-adapter-disconnect-reconnect", async () =>
{
await adapter.DisconnectAsync(ct);
await adapter.ConnectAsync(OpcUaEndpointConfigSerializer.ToFlatDict(config), ct);
Require((await adapter.ReadAsync(raw + "Scalar", ct)).Success, "Consumer failed after reconnect");
await adapter.DisconnectAsync(ct);
});
}
var sdkAssembly = typeof(Opc.Ua.NodeId).Assembly;
var result = new
{
consumer = typeof(OpcUaDataConnection).Assembly.GetName().Version?.ToString(),
sdk = FileVersionInfo.GetVersionInfo(sdkAssembly.Location).ProductVersion,
process = Environment.ProcessId, mode = mode.ToString(), autoAccept = false,
endpoint, cases, failures,
notImplementedByConsumer = new[] { "HistoryRead/CP", "Part9 acknowledgement methods" },
};
File.WriteAllText(Path.Combine(root, "result.json"), JsonSerializer.Serialize(result, new JsonSerializerOptions { WriteIndented = true }));
Console.WriteLine($"Consumer probe: {cases.Count - failures} passed, {failures} failed; {Path.Combine(root, "result.json")}");
return failures == 0 ? 0 : 1;
scripts/validation/scadabridge-sdk-compatibility/README.md
-46
View File
@@ -1,46 +0,0 @@
# Actual ScadaBridge consumer compatibility
This isolated process references the real ScadaBridge `OpcUaDataConnection` and
`RealOpcUaClient`, with their own dependency graph. It never loads the candidate
SDK into that process. The verified consumer source is
`0e89966da670e289bd875166295bb0e85d8cf8b2`, pinned to SDK 1.5.378.106.
Build with an explicit clean consumer checkout:
```sh
dotnet build scripts/validation/scadabridge-sdk-compatibility/Probe.csproj \
-c Release -p:ScadaBridgeSourceRoot=/absolute/ScadaBridge
```
First build the separate application hosts with the parent
`scripts/e2e/test-opcua-sdk-compatibility.ps1` harness. Then choose a new output
directory and pass its original and candidate `ProcessHost.dll` paths:
```sh
python3 scripts/validation/scadabridge-sdk-compatibility/run.py \
--baseline-host /absolute/matrix/build-baseline/bin/Release/net10.0/ProcessHost.dll \
--candidate-host /absolute/matrix/build-candidate/bin/Release/net10.0/ProcessHost.dll \
--probe scripts/validation/scadabridge-sdk-compatibility/bin/Release/net10.0/Probe.dll \
--results /absolute/fresh-consumer-evidence
```
The helper runs four separate consumer/server pairs: each application version
with None and SignAndEncrypt. None is enabled only in its own test fixture;
the ordinary compatibility host remains strict SignAndEncrypt. The task key is
generated per process pair, issuer/trusted/rejected stores are task-local, and
server public certificates are explicitly copied. Autoaccept remains false.
Never commit an entire output directory: it contains the fixture's private PKI.
The consumed subset covers API-key activation, raw/UNS scalar and Int32[]
roundtrips, durable hierarchical browse including the native alarm, data quality
and timestamps, alarm filter/ConditionRefresh and disconnect/reconnect. History
and Part 9 acknowledgment are not exposed by the inspected consumer interfaces;
the parent Shared/CLI matrix covers those application services instead.
ScadaBridge#121 is a reproduced preexisting limitation: SignAndEncrypt fails
against **both** original and candidate servers because the consumer does not
configure an application certificate. A matching baseline failure is evidence,
not a secure-channel pass. The runner retains the nonzero result and each actual
failure; it does not loosen trust or change expected results to return success.
The current None subset and the missing secure identity must be reported
separately. No ScadaBridge production source or user PKI is modified.
scripts/validation/scadabridge-sdk-compatibility/evidence/2026-11-10/baseline-None.json
-51
View File
@@ -1,51 +0,0 @@
{
"consumer": "1.0.0.0",
"sdk": "1.5.378.106+99044fdbb638087d62f1e16580e9f73bfda364e2",
"process": 93934,
"mode": "None",
"autoAccept": false,
"endpoint": "opc.tcp://127.0.0.1:51314/OtOpcUa",
"cases": [
{
"name": "actual-adapter-connect-api-key",
"status": "passed"
},
{
"name": "scalar-round-trip-nsu=https://zb.com/otopcua/raw;s=Matrix/dev/",
"status": "passed"
},
{
"name": "array-round-trip-nsu=https://zb.com/otopcua/raw;s=Matrix/dev/",
"status": "passed"
},
{
"name": "scalar-round-trip-nsu=https://zb.com/otopcua/uns;s=Eq/",
"status": "passed"
},
{
"name": "array-round-trip-nsu=https://zb.com/otopcua/uns;s=Eq/",
"status": "passed"
},
{
"name": "actual-adapter-durable-browse",
"status": "passed"
},
{
"name": "actual-adapter-monitor-quality-time",
"status": "passed"
},
{
"name": "actual-adapter-alarm-filter-refresh",
"status": "passed"
},
{
"name": "actual-adapter-disconnect-reconnect",
"status": "passed"
}
],
"failures": 0,
"notImplementedByConsumer": [
"HistoryRead/CP",
"Part9 acknowledgement methods"
]
}
scripts/validation/scadabridge-sdk-compatibility/evidence/2026-11-10/baseline-SignAndEncrypt.json
-20
View File
@@ -1,20 +0,0 @@
{
"consumer": "1.0.0.0",
"sdk": "1.5.378.106+99044fdbb638087d62f1e16580e9f73bfda364e2",
"process": 93939,
"mode": "SignAndEncrypt",
"autoAccept": false,
"endpoint": "opc.tcp://127.0.0.1:51358/OtOpcUa",
"cases": [
{
"name": "actual-adapter-connect-api-key",
"status": "failed",
"error": "ApplicationCertificate for the security profile http://opcfoundation.org/UA/SecurityPolicy#Basic256Sha256 cannot be found.\n[80890000] 'ApplicationCertificate for the security profile http://opcfoundation.org/UA/SecurityPolicy#Basic256Sha256 cannot be found.'"
}
],
"failures": 1,
"notImplementedByConsumer": [
"HistoryRead/CP",
"Part9 acknowledgement methods"
]
}
scripts/validation/scadabridge-sdk-compatibility/evidence/2026-11-10/candidate-None.json
-51
View File
@@ -1,51 +0,0 @@
{
"consumer": "1.0.0.0",
"sdk": "1.5.378.106+99044fdbb638087d62f1e16580e9f73bfda364e2",
"process": 93941,
"mode": "None",
"autoAccept": false,
"endpoint": "opc.tcp://127.0.0.1:51387/OtOpcUa",
"cases": [
{
"name": "actual-adapter-connect-api-key",
"status": "passed"
},
{
"name": "scalar-round-trip-nsu=https://zb.com/otopcua/raw;s=Matrix/dev/",
"status": "passed"
},
{
"name": "array-round-trip-nsu=https://zb.com/otopcua/raw;s=Matrix/dev/",
"status": "passed"
},
{
"name": "scalar-round-trip-nsu=https://zb.com/otopcua/uns;s=Eq/",
"status": "passed"
},
{
"name": "array-round-trip-nsu=https://zb.com/otopcua/uns;s=Eq/",
"status": "passed"
},
{
"name": "actual-adapter-durable-browse",
"status": "passed"
},
{
"name": "actual-adapter-monitor-quality-time",
"status": "passed"
},
{
"name": "actual-adapter-alarm-filter-refresh",
"status": "passed"
},
{
"name": "actual-adapter-disconnect-reconnect",
"status": "passed"
}
],
"failures": 0,
"notImplementedByConsumer": [
"HistoryRead/CP",
"Part9 acknowledgement methods"
]
}
scripts/validation/scadabridge-sdk-compatibility/evidence/2026-11-10/candidate-SignAndEncrypt.json
-20
View File
@@ -1,20 +0,0 @@
{
"consumer": "1.0.0.0",
"sdk": "1.5.378.106+99044fdbb638087d62f1e16580e9f73bfda364e2",
"process": 93943,
"mode": "SignAndEncrypt",
"autoAccept": false,
"endpoint": "opc.tcp://127.0.0.1:51455/OtOpcUa",
"cases": [
{
"name": "actual-adapter-connect-api-key",
"status": "failed",
"error": "ApplicationCertificate for the security profile http://opcfoundation.org/UA/SecurityPolicy#Basic256Sha256 cannot be found.\n[80890000] 'ApplicationCertificate for the security profile http://opcfoundation.org/UA/SecurityPolicy#Basic256Sha256 cannot be found.'"
}
],
"failures": 1,
"notImplementedByConsumer": [
"HistoryRead/CP",
"Part9 acknowledgement methods"
]
}
scripts/validation/scadabridge-sdk-compatibility/evidence/2026-11-10/receipt.json
-98
View File
@@ -1,98 +0,0 @@
{
"artifacts": {
"baseline_host": {
"path": "/Users/dohertj2/.codex/worktrees/opcua2-cross-version/OtOpcUa/artifacts/cross-version/run-10/build-baseline/bin/Release/net10.0/ProcessHost.dll",
"sha256": "841f83aeb3a32d49c06518a98963743d720015bba41ae3138649126ce3a6af1c"
},
"candidate_host": {
"path": "/Users/dohertj2/.codex/worktrees/opcua2-cross-version/OtOpcUa/artifacts/cross-version/run-10/build-candidate/bin/Release/net10.0/ProcessHost.dll",
"sha256": "e025fc21ac3559c3796ec639f0ef7b9c3026ccbbdaa4783e14b8930eae3a2043"
},
"probe": {
"path": "/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/scripts/validation/scadabridge-sdk-compatibility/bin/Release/net10.0/Probe.dll",
"sha256": "913ef3e35ffac8a54567e5d4df6f934572043249eaa6a4e0f829efd07b9012d7"
}
},
"legs": [
{
"server": "baseline",
"mode": "None",
"serverCommand": [
"dotnet",
"/Users/dohertj2/.codex/worktrees/opcua2-cross-version/OtOpcUa/artifacts/cross-version/run-10/build-baseline/bin/Release/net10.0/ProcessHost.dll",
"server",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/baseline-None/server"
],
"command": [
"dotnet",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/scripts/validation/scadabridge-sdk-compatibility/bin/Release/net10.0/Probe.dll",
"opc.tcp://127.0.0.1:51314/OtOpcUa",
"None",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/baseline-None/consumer"
],
"exit": 0,
"result": "/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/baseline-None/consumer/result.json",
"serverExit": 0
},
{
"server": "baseline",
"mode": "SignAndEncrypt",
"serverCommand": [
"dotnet",
"/Users/dohertj2/.codex/worktrees/opcua2-cross-version/OtOpcUa/artifacts/cross-version/run-10/build-baseline/bin/Release/net10.0/ProcessHost.dll",
"server",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/baseline-SignAndEncrypt/server"
],
"command": [
"dotnet",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/scripts/validation/scadabridge-sdk-compatibility/bin/Release/net10.0/Probe.dll",
"opc.tcp://127.0.0.1:51358/OtOpcUa",
"SignAndEncrypt",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/baseline-SignAndEncrypt/consumer"
],
"exit": 1,
"result": "/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/baseline-SignAndEncrypt/consumer/result.json",
"serverExit": 0
},
{
"server": "candidate",
"mode": "None",
"serverCommand": [
"dotnet",
"/Users/dohertj2/.codex/worktrees/opcua2-cross-version/OtOpcUa/artifacts/cross-version/run-10/build-candidate/bin/Release/net10.0/ProcessHost.dll",
"server",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/candidate-None/server"
],
"command": [
"dotnet",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/scripts/validation/scadabridge-sdk-compatibility/bin/Release/net10.0/Probe.dll",
"opc.tcp://127.0.0.1:51387/OtOpcUa",
"None",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/candidate-None/consumer"
],
"exit": 0,
"result": "/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/candidate-None/consumer/result.json",
"serverExit": 0
},
{
"server": "candidate",
"mode": "SignAndEncrypt",
"serverCommand": [
"dotnet",
"/Users/dohertj2/.codex/worktrees/opcua2-cross-version/OtOpcUa/artifacts/cross-version/run-10/build-candidate/bin/Release/net10.0/ProcessHost.dll",
"server",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/candidate-SignAndEncrypt/server"
],
"command": [
"dotnet",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/scripts/validation/scadabridge-sdk-compatibility/bin/Release/net10.0/Probe.dll",
"opc.tcp://127.0.0.1:51455/OtOpcUa",
"SignAndEncrypt",
"/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/candidate-SignAndEncrypt/consumer"
],
"exit": 1,
"result": "/Users/dohertj2/.codex/worktrees/opcua-sdk-2-upgrade/OtOpcUa/artifacts/opcua2/88ee30e0/scadabridge-final-matrix/candidate-SignAndEncrypt/consumer/result.json",
"serverExit": 0
}
]
}
scripts/validation/scadabridge-sdk-compatibility/evidence/2026-11-10/summary.json
-11
View File
@@ -1,11 +0,0 @@
{
"consumerSource": "0e89966da670e289bd875166295bb0e85d8cf8b2",
"consumerSdk": "1.5.378.106",
"baselineServerSdk": "1.5.378.182",
"candidateServerSdk": "2.0.1",
"noneCasesPassed": 18,
"secureConnectionFailures": 2,
"samePreexistingFailure": "BadConfigurationError: ApplicationCertificate not configured",
"issue": "https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/121",
"releaseApproval": false
}
scripts/validation/scadabridge-sdk-compatibility/run.py
-75
View File
@@ -1,75 +0,0 @@
#!/usr/bin/env python3
"""Run the real ScadaBridge consumer in separate processes against both application versions."""
import argparse
import hashlib
import json
import os
from pathlib import Path
import secrets
import subprocess
import time
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--baseline-host", type=Path, required=True)
parser.add_argument("--candidate-host", type=Path, required=True)
parser.add_argument("--probe", type=Path, required=True)
parser.add_argument("--results", type=Path, required=True)
args = parser.parse_args()
root = args.results.resolve()
root.mkdir(parents=True, exist_ok=False)
artifacts = {name: path.resolve() for name, path in vars(args).items() if name != "results"}
for path in artifacts.values():
if not path.is_file():
raise FileNotFoundError(path)
receipt = {
"artifacts": {name: {"path": str(path), "sha256": hashlib.sha256(path.read_bytes()).hexdigest()}
for name, path in artifacts.items()},
"legs": [],
}
for version in ("baseline", "candidate"):
for mode in ("None", "SignAndEncrypt"):
leg = root / f"{version}-{mode}"
leg.mkdir()
server_root = leg / "server"
env = os.environ.copy()
env["OPCUA_MATRIX_PASSWORD"] = "otua_matrix_" + secrets.token_urlsafe(32)
if mode == "None":
env["OPCUA_MATRIX_SERVER_SECURITY"] = "none"
else:
env.pop("OPCUA_MATRIX_SERVER_SECURITY", None)
server_command = ["dotnet", str(artifacts[f"{version}_host"]), "server", str(server_root)]
result = {"server": version, "mode": mode, "serverCommand": server_command}
with (leg / "server.log").open("w") as server_log:
server = subprocess.Popen(server_command, env=env, stdout=server_log, stderr=subprocess.STDOUT)
try:
until = time.monotonic() + 40
while not (server_root / "endpoint.txt").is_file():
if server.poll() is not None or time.monotonic() >= until:
raise RuntimeError(f"Server failed to become ready; inspect {leg / 'server.log'}")
time.sleep(0.1)
endpoint = (server_root / "endpoint.txt").read_text().strip()
env["SCADABRIDGE_PROBE_SERVER_CERTS"] = str(server_root / "server-pki" / "own" / "certs")
command = ["dotnet", str(artifacts["probe"]), endpoint, mode, str(leg / "consumer")]
with (leg / "consumer.log").open("w") as log:
probe = subprocess.run(command, env=env, stdout=log, stderr=subprocess.STDOUT, timeout=150)
result.update(command=command, exit=probe.returncode,
result=str(leg / "consumer" / "result.json"))
finally:
server_root.mkdir(exist_ok=True)
(server_root / "stop").touch()
try:
server.wait(timeout=15)
except subprocess.TimeoutExpired:
server.terminate()
server.wait(timeout=10)
result["serverExit"] = server.returncode
receipt["legs"].append(result)
(root / "receipt.json").write_text(json.dumps(receipt, indent=2))
# A pre-existing consumer limitation is evidence to investigate, never converted into a passing leg.
return int(any(leg["exit"] != 0 or leg["serverExit"] != 0 for leg in receipt["legs"]))
if __name__ == "__main__":
raise SystemExit(main())
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs
+11 -3
View File
@@ -57,15 +57,23 @@ public sealed class HistorianGatewayClientAdapter : IHistorianGatewayClient, IDi
$"ServerHistorian:Endpoint must be an absolute http(s) URI (e.g. https://host:5222); got '{options.Endpoint}'.");
}
// Certificate settings apply only to TLS. The client rejects an explicit opt-out or a
// CA path on h2c, where there is no certificate to validate.
// TLS-only options must stay at their defaults on a plaintext h2c connection: the client rejects
// each of them outright when UseTls=false ("<name> is a TLS-only option and requires
// UseTls=true"). Forwarding them unconditionally made h2c UNREACHABLE — AllowUntrustedServer-
// Certificate defaults to false, so RequireCertificateValidation was always sent as true and
// every http:// deployment crashed at startup, even though the scheme is documented as the
// supported way to select h2c. There is no certificate to have a posture about here.
var clientOptions = new HistorianGatewayClientOptions
{
Endpoint = endpointUri,
ApiKey = options.ApiKey,
UseTls = options.UseTls,
CaCertificatePath = options.UseTls ? options.CaCertificatePath : null,
AllowUntrustedServerCertificate = options.UseTls && options.AllowUntrustedServerCertificate,
// INVERTED mapping (TLS only): ServerHistorianOptions.AllowUntrustedServerCertificate
// (opt-in to accept a self-signed cert) is the negation of the client's
// RequireCertificateValidation. Allowing an untrusted cert == not requiring validation; a
// pinned CaCertificatePath always verifies.
RequireCertificateValidation = options.UseTls && !options.AllowUntrustedServerCertificate,
DefaultCallTimeout = options.CallTimeout,
LoggerFactory = loggerFactory,
};
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs
+3 -2
View File
@@ -1086,8 +1086,9 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
{
// GetEndpoints returns everything the server advertises; policy + mode filter is
// applied client-side so the selection is explicit and fails loudly if the operator
// asks for a combination the server doesn't publish. Use the SDK's asynchronous
// discovery factory and pass the caller's cancellation token.
// asks for a combination the server doesn't publish. DiscoveryClient.CreateAsync
// is the non-obsolete path in SDK 1.5.378; the synchronous Create(..) variants are
// all deprecated.
using var client = await DiscoveryClient.CreateAsync(
appConfig, new Uri(endpointUrl), Opc.Ua.DiagnosticsMasks.None, ct).ConfigureAwait(false);
var all = await client.GetEndpointsAsync(default, ct).ConfigureAwait(false);
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Compatibility/Aggregates/README.md
+2 -2
View File
@@ -10,9 +10,9 @@ Representation adaptations:
- Nullable reference annotations describe the existing optional bounds/slices; no warning policy is disabled.
- Immutable StatusCode methods replace the old setters. NodeId extraction and QualifiedName construction use supported typed APIs.
- Removed TypeInfo.Cast/Variant(object, TypeInfo) operations use the SDK 2 typed conversion surface; baseline vectors verify value **types**, values, timestamps and all status bits.
- Interval, quality, sample selection, queue trimming, interpolation and partial-result algorithms remain the baseline algorithms. The server creates a fresh SDK AggregateManager and registers these 37 IDs exactly once through the public API. It must not call the base registration first: replacing an existing factory also adds a duplicate diagnostic node in SDK 2.0.1. No extra capability is advertised.
- Interval, quality, sample selection, queue trimming, interpolation and partial-result algorithms remain the baseline algorithms. Factory registration replaces the existing 37 IDs through the public SDK API; it must not add capabilities or duplicate advertised references.
The isolated `scripts/validation/opcua-aggregate-baseline` process pins only SDK 1.5.378.182 and generates the checked-in golden fixture. It is deliberately outside the solution and does not inherit the candidate package graph. The golden file records the baseline assembly hash, shared inputs and 6,144 outputs/cases across all 37 aggregate IDs: the original 4,144 cases plus 2,000 numeric-width, nonfinite/conversion, timestamp tie/picosecond and configuration-edge cases. Forward/reverse, stepped/sloped interpolation, uncertain-as-bad flags and incremental/partial processing are retained. The candidate tests exercise the same independent expectations both directly and through the booted runtime manager. Unset SDK 2 FILETIME timestamps map explicitly to the original CLR unset sentinel when observing results. No baseline binary is loaded by the candidate tests or production host.
The isolated `scripts/validation/opcua-aggregate-baseline` process pins only SDK 1.5.378.182 and generates the checked-in golden fixture. It is deliberately outside the solution and does not inherit the candidate package graph. The golden file records the baseline assembly hash, shared inputs and 4,144 outputs/cases across all 37 aggregate IDs, seven datasets, forward/reverse, stepped/sloped interpolation, uncertain-as-bad flags and incremental/partial processing. No baseline binary is loaded by the candidate tests or production host.
Run the generator with a **new** output path, then compare it with `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/Fixtures/opcua-1.5.378.182-aggregates.json`. It refuses to overwrite evidence. Never regenerate expectations from the compatibility implementation. Existing historian parity tests retain their application-specific documented divergences from the baseline SDK, while new compatibility tests check the complete old-calculator output.
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs
+11 -10
View File
@@ -161,8 +161,8 @@ public sealed class OpcUaQuotaOptions
/// Thin facade over the OPC Foundation .NET Standard SDK's application bootstrap.
/// Owns the <see cref="ApplicationInstance"/> + <see cref="ApplicationConfiguration"/> lifetime,
/// builds the <see cref="ApplicationConfiguration"/> programmatically (security profiles + user
/// token policies + PKI stores), installs an SDK identity-registry authenticator before
/// the <see cref="OtOpcUaSdkServer"/> opens its listeners via
/// token policies + PKI stores), starts a <see cref="StandardServer"/>, attaches the
/// <c>ImpersonateUser</c> hook for UserName-token authentication via
/// <see cref="IOpcUaUserAuthenticator"/>, and publishes the redundancy peer URIs through
/// <c>Server.ServerArray</c>. See <c>docs/OpcUaServer.md</c> and <c>docs/security.md</c>.
/// </summary>
@@ -205,8 +205,8 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
if (server is not OtOpcUaSdkServer sdkServer)
throw new ArgumentException("The server must support application authentication before its listeners open.", nameof(server));
_server = server;
// ApplicationInstance receives the host's ITelemetryContext. DefaultTelemetry.Create wires
// the SDK's internal logging; an empty builder keeps
// 1.5.378 requires an ITelemetryContext on the ApplicationInstance ctor (the parameterless ctor
// is obsolete). DefaultTelemetry.Create wires the SDK's internal logging; an empty builder keeps
// the SDK's trace off our ILogger (the host keeps its own _logger) — sufficient for the bootstrap.
_application = new ApplicationInstance(_telemetry)
{
@@ -363,7 +363,8 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
string password;
try
{
// The SDK UserName token handler exposes decrypted password bytes; decode the wire UTF-8.
// 1.5.378 exposes DecryptedPassword as raw bytes (was string); UserName token passwords
// are UTF-8 on the wire.
var decryptedBytes = (args.UserIdentityTokenHandler as UserNameIdentityTokenHandler)?.DecryptedPassword;
password = decryptedBytes is null ? string.Empty : Encoding.UTF8.GetString(decryptedBytes);
}
@@ -467,7 +468,7 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
private async Task EnsureApplicationCertificateAsync(CancellationToken cancellationToken)
{
// silent: false → SDK logs cert creation events through its own trace plumbing.
// A null lifetime keeps the SDK default (12 months). Key-size policy comes from the configuration.
// minimumKeySize/lifetimeInMonths: 0 → use SDK defaults (2048-bit, 12-month lifetime).
var ok = await _application!.CheckApplicationInstanceCertificatesAsync(
false, null, cancellationToken).ConfigureAwait(false);
if (!ok)
@@ -609,10 +610,10 @@ public sealed class OpcUaApplicationHost : IAsyncDisposable
/// <summary>
/// Anonymous + UserName token policies. UserName tokens are always SDK-encrypted with
/// the server certificate (see docs/security.md "UserName token encryption") so the
/// policy works on None endpoints too. The Host supplies API-key authentication through
/// <see cref="IOpcUaUserAuthenticator"/>; <see cref="AttachUserAuthenticator"/> installs it in the
/// SDK identity registry before listeners open. <see cref="HandleImpersonation"/> retains the
/// application decision seam, and <see cref="NullOpcUaUserAuthenticator"/> denies all
/// policy works on None endpoints too. The real LDAP-bound validator is wired through
/// <see cref="IOpcUaUserAuthenticator"/> on <c>StandardServer.SessionManager.ImpersonateUser</c>
/// (see <see cref="AttachUserAuthenticator"/> / <see cref="HandleImpersonation"/>); the Host
/// project supplies the LDAP adapter, and <see cref="NullOpcUaUserAuthenticator"/> denies all
/// UserName logins when none is registered.
/// </summary>
/// <param name="allowAnonymous">Whether the Anonymous policy is advertised.</param>
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/IOpcUaUserAuthenticator.cs
+1 -1
View File
@@ -32,7 +32,7 @@ public interface IOpcUaUserAuthenticator
/// endpoint and delegates, so an implementation that does not need it is unchanged.
/// </summary>
/// <remarks>
/// The endpoint uses the SDK's public transport peer address during activation and is null when unknown — an
/// The endpoint is best effort (<see cref="SecureChannelRemoteEndpoint"/>) and null when unknown — an
/// implementation must treat null as "no per-client information", never as a distinct client. Same
/// must-not-throw contract as the three-argument overload.
/// </remarks>
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/RoleCarryingUserIdentity.cs
+1 -1
View File
@@ -4,7 +4,7 @@ using Opc.Ua.Server;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Security;
/// <summary>
/// An SDK identity that additionally carries the application roles resolved for an API key
/// A <see cref="UserIdentity"/> that additionally carries the LDAP-resolved roles
/// (<see cref="OpcUaUserAuthResult.Roles"/>) for the authenticated session. The SDK may augment identities with mandatory and dynamically mapped roles. This subtype
/// preserves the application identity when those roles are added, so a method
/// handler can later recover the roles via
tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Helpers/VariantBoundaryTests.cs
+2 -2
View File
@@ -11,8 +11,8 @@ public class VariantBoundaryTests
{
foreach (var value in new object?[] { null, true, (sbyte)-1, (byte)1, (short)-2, (ushort)2,
-3, 3u, -4L, 4UL, 1.25f, 2.5, "", "text", Guid.Parse("de305d54-75b4-431b-adb2-eb6b9e546014"),
new DateTime(2026, 11, 10, 0, 0, 0, DateTimeKind.Utc), new byte[] { 1, 2 },
new[] { 1 }, new[] { 1, 2 }, new[] { "", "text" } })
new DateTime(2026, 11, 10, 0, 0, 0, DateTimeKind.Utc), Array.Empty<byte>(), new byte[] { 1, 2 },
Array.Empty<int>(), new[] { 1 }, new[] { 1, 2 }, Array.Empty<string>(), new[] { "", "text" } })
yield return [value];
foreach (var value in new object[] { true, (sbyte)-1, (byte)1, (short)-2, (ushort)2,
-3, 3u, -4L, 4UL, 1.25f, 2.5, "text", Guid.Empty,
tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/StatusCodeParityTests.cs
+3 -12
View File
@@ -47,18 +47,9 @@ public sealed class StatusCodeParityTests
/// <summary>Every <c>Opc.Ua.StatusCodes</c> member, by name — the oracle this test checks against.</summary>
private static readonly IReadOnlyDictionary<string, uint> SdkStatusCodes =
typeof(Opc.Ua.StatusCodes)
.GetFields(BindingFlags.Public | BindingFlags.Static)
.Where(f => f.FieldType == typeof(Opc.Ua.StatusCode))
.ToDictionary(f => f.Name, f => ((Opc.Ua.StatusCode)f.GetValue(null)!).Code, StringComparer.Ordinal);
[Fact]
public void SDK_oracle_contains_all_status_severities()
{
SdkStatusCodes.Count.ShouldBeGreaterThan(200);
SdkStatusCodes["Good"].ShouldBe(0u);
SdkStatusCodes["Uncertain"].ShouldBe(0x40000000u);
SdkStatusCodes["Bad"].ShouldBe(0x80000000u);
}
.GetProperties(BindingFlags.Public | BindingFlags.Static)
.Where(p => p.PropertyType == typeof(Opc.Ua.StatusCode))
.ToDictionary(p => p.Name, p => ((Opc.Ua.StatusCode)p.GetValue(null)!).Code, StringComparer.Ordinal);
/// <summary>
/// Every status-shaped <c>const uint</c> declared anywhere in the deployed driver assemblies, as
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/HistorianGatewayClientAdapterTests.cs
+17 -1
View File
@@ -27,6 +27,23 @@ public sealed class HistorianGatewayClientAdapterTests
/// A plaintext <c>h2c</c> gateway (an <c>http://</c> endpoint with <c>UseTls=false</c>) must
/// construct, using nothing but documented defaults.
/// </summary>
/// <remarks>
/// <para>
/// Found by the LocalDb Phase 2 live gate, on the third consecutive crash-loop of the rig.
/// <c>docs/Historian.md</c> and CLAUDE.md both document <c>http://</c> as a supported
/// transport ("scheme selects transport: https = TLS, http = h2c"), but it was in fact
/// <b>unreachable</b>: this factory forwarded the TLS-only options unconditionally, and
/// <see cref="ServerHistorianOptions.AllowUntrustedServerCertificate"/> defaults to
/// <see langword="false"/>, so it always sent <c>RequireCertificateValidation=true</c> —
/// which the client rejects outright when <c>UseTls=false</c>
/// ("RequireCertificateValidation is a TLS-only option and requires UseTls=true").
/// </para>
/// <para>
/// So <b>every</b> h2c deployment crashed at startup, and the only way to avoid it was to
/// set <c>AllowUntrustedServerCertificate=true</c> — asserting a certificate posture for a
/// connection that has no certificate at all.
/// </para>
/// </remarks>
[Fact]
public void Adapter_constructs_for_plaintext_h2c_endpoint()
{
@@ -54,7 +71,6 @@ public sealed class HistorianGatewayClientAdapterTests
ApiKey = "histgw_x_y",
UseTls = false,
CaCertificatePath = "/etc/ssl/certs/leftover-from-a-tls-config.pem",
AllowUntrustedServerCertificate = true,
};
using var adapter = HistorianGatewayClientAdapter.Create(opts, NullLoggerFactory.Instance);
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/HistorianGatewayTlsPolicyWireTests.cs
-142
View File
@@ -1,142 +0,0 @@
using System.Net;
using System.Security.Authentication;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using Grpc.Core;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.HistorianGateway.Client;
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
/// <summary>historiangw#131: verify the mapped TLS posture through an actual local gRPC connection.</summary>
public sealed class HistorianGatewayTlsPolicyWireTests(ITestOutputHelper output)
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
[Fact]
public async Task Default_tls_rejects_an_untrusted_server_before_the_rpc()
{
await using var server = await Server.StartAsync(tls: true);
using var adapter = HistorianGatewayClientAdapter.Create(server.Options(), NullLoggerFactory.Instance);
var error = await Should.ThrowAsync<HistorianGatewayException>(
() => adapter.TagExistsAsync("T", Ct).WaitAsync(TimeSpan.FromSeconds(15), Ct));
output.WriteLine($"TLS rejection: {error}");
ContainsTlsAuthenticationFailure(error).ShouldBeTrue("the failure must be TLS authentication, not an arbitrary connection failure");
server.MetadataCalls.ShouldBe(0);
}
[Fact]
public async Task Explicit_tls_opt_out_reaches_the_same_self_signed_server_shape()
{
await using var server = await Server.StartAsync(tls: true);
var options = server.Options(allowUntrusted: true);
using var adapter = HistorianGatewayClientAdapter.Create(options, NullLoggerFactory.Instance);
(await adapter.TagExistsAsync("T", Ct).WaitAsync(TimeSpan.FromSeconds(15), Ct)).ShouldBeTrue();
server.MetadataCalls.ShouldBe(1);
}
[Fact]
public async Task Plaintext_ignores_stale_tls_opt_out_and_ca_settings_and_reaches_the_rpc()
{
await using var server = await Server.StartAsync(tls: false);
var options = server.Options(allowUntrusted: true, caPath: "/nonexistent/historiangw-131-stale-ca.pem");
using var adapter = HistorianGatewayClientAdapter.Create(options, NullLoggerFactory.Instance);
(await adapter.TagExistsAsync("T", Ct).WaitAsync(TimeSpan.FromSeconds(15), Ct)).ShouldBeTrue();
server.MetadataCalls.ShouldBe(1);
}
private static bool ContainsTlsAuthenticationFailure(Exception error)
{
var pending = new Stack<Exception>();
var seen = new HashSet<Exception>(ReferenceEqualityComparer.Instance);
pending.Push(error);
while (pending.TryPop(out var current))
{
if (!seen.Add(current)) continue;
if (current is AuthenticationException) return true;
if (current.InnerException is { } inner) pending.Push(inner);
if (current is RpcException { Status.DebugException: { } debug }) pending.Push(debug);
}
return false;
}
private sealed class TagsService : HistorianTags.HistorianTagsBase
{
public int Calls;
public override Task<GetTagMetadataReply> GetTagMetadata(GetTagMetadataRequest request, ServerCallContext context)
{
Interlocked.Increment(ref Calls);
return Task.FromResult(new GetTagMetadataReply
{ Metadata = new HistorianTagMetadata { Name = request.Tag } });
}
}
private sealed class Server(WebApplication app, TagsService service, string endpoint,
RSA? key, X509Certificate2? certificate) : IAsyncDisposable
{
public int MetadataCalls => Volatile.Read(ref service.Calls);
public ServerHistorianOptions Options(bool allowUntrusted = false, string? caPath = null) => new()
{
Endpoint = endpoint, ApiKey = "histgw_131_test", UseTls = certificate is not null,
CallTimeout = TimeSpan.FromSeconds(2),
AllowUntrustedServerCertificate = allowUntrusted, CaCertificatePath = caPath,
};
public static async Task<Server> StartAsync(bool tls)
{
RSA? key = null;
X509Certificate2? certificate = null;
if (tls)
{
key = RSA.Create(2048);
var request = new CertificateRequest("CN=localhost", key, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
request.CertificateExtensions.Add(new X509BasicConstraintsExtension(false, false, 0, true));
request.CertificateExtensions.Add(new X509KeyUsageExtension(
X509KeyUsageFlags.DigitalSignature | X509KeyUsageFlags.KeyEncipherment, true));
var names = new SubjectAlternativeNameBuilder();
names.AddDnsName("localhost"); names.AddIpAddress(IPAddress.Loopback);
request.CertificateExtensions.Add(names.Build());
certificate = request.CreateSelfSigned(DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddDays(1));
}
var service = new TagsService();
var builder = WebApplication.CreateBuilder();
builder.Logging.ClearProviders();
builder.WebHost.ConfigureKestrel(options => options.Listen(IPAddress.Loopback, 0, listen =>
{
listen.Protocols = HttpProtocols.Http2;
if (certificate is not null) listen.UseHttps(certificate);
}));
builder.Services.AddGrpc(); builder.Services.AddSingleton(service);
var app = builder.Build();
app.MapGrpcService<TagsService>();
try
{
await app.StartAsync(Ct);
return new Server(app, service, app.Urls.Single(), key, certificate);
}
catch
{
await app.DisposeAsync(); certificate?.Dispose(); key?.Dispose();
throw;
}
}
public async ValueTask DisposeAsync()
{
try { await app.StopAsync(Ct); }
finally { await app.DisposeAsync(); certificate?.Dispose(); key?.Dispose(); }
}
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/Sdk2LifecycleSoakTests.cs
+17 -311
View File
@@ -1,12 +1,10 @@
using System.Collections;
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Diagnostics.Tracing;
using System.Globalization;
using System.Net;
using System.Net.Sockets;
using System.Reflection;
using System.Runtime.ExceptionServices;
using System.Text;
using System.Text.Json;
using Opc.Ua;
@@ -40,17 +38,14 @@ public sealed class Sdk2LifecycleSoakTests
private static readonly DateTime Epoch = new(2026, 10, 1, 0, 0, 0, DateTimeKind.Utc);
public static bool Enabled => Environment.GetEnvironmentVariable("OTOPCUA_SDK2_SOAK") == "1";
[Fact(SkipUnless = nameof(Enabled), Skip = "Opt in with OTOPCUA_SDK2_SOAK=1 and an explicit soak mode.")]
[Fact(SkipUnless = nameof(Enabled), Skip = "Opt in with OTOPCUA_SDK2_SOAK=1 and an explicit diagnostic/full mode.")]
public async Task Repeated_lifecycles_return_resources_to_a_steady_baseline()
{
var mode = Environment.GetEnvironmentVariable("OTOPCUA_SDK2_SOAK_MODE");
(mode is "diagnostic" or "profile" or "full" or "saturation").ShouldBeTrue("an explicit soak mode is required");
var saturation = mode == "saturation" ? SaturationControl.FromEnvironment() : null;
(mode is "diagnostic" or "full").ShouldBeTrue("explicit diagnostic or full mode is required");
var seconds = int.Parse(Environment.GetEnvironmentVariable("OTOPCUA_SDK2_SOAK_SECONDS")
?? (mode == "full" ? "7200" : saturation is not null ? "1800" : "90"), CultureInfo.InvariantCulture);
?? (mode == "full" ? "7200" : "90"), CultureInfo.InvariantCulture);
if (mode == "full") seconds.ShouldBeGreaterThanOrEqualTo(7200);
else if (saturation is not null) seconds.ShouldBe(1800, "the prospective saturation cutoff is fixed");
else if (mode == "profile") seconds.ShouldBeInRange(60, 1800); // Investigative only; never satisfies the full-duration flag.
else seconds.ShouldBeInRange(60, 120);
var output = Environment.GetEnvironmentVariable("OTOPCUA_SDK2_SOAK_OUTPUT")
?? throw new InvalidOperationException("An absolute, fresh OTOPCUA_SDK2_SOAK_OUTPUT JSONL path is required.");
@@ -58,7 +53,7 @@ public sealed class Sdk2LifecycleSoakTests
Directory.CreateDirectory(Path.GetDirectoryName(output)!);
await using var evidence = new StreamWriter(new FileStream(output, FileMode.CreateNew, FileAccess.Write, FileShare.Read));
using var run = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken);
run.CancelAfter(TimeSpan.FromSeconds(saturation is null ? seconds + 120 : 120));
run.CancelAfter(TimeSpan.FromSeconds(seconds + 120));
var ct = run.Token;
var watch = Stopwatch.StartNew();
var clientRoot = Path.Combine(Path.GetTempPath(), $"otua-soak-client-{Guid.NewGuid():N}");
@@ -68,7 +63,6 @@ public sealed class Sdk2LifecycleSoakTests
var cycles = 0;
var restarts = 0;
Sample? baseline = null;
ExceptionDispatchInfo? firstRejectedBudget = null;
var retained = new Queue<Sample>();
try
{
@@ -90,7 +84,6 @@ public sealed class Sdk2LifecycleSoakTests
Configure(rig);
// The required observation period excludes initial certificate creation and server startup.
watch.Restart();
if (saturation is not null) run.CancelAfter(TimeSpan.FromSeconds(1800));
await LogAsync(evidence, new
{
kind = "start", utc = DateTimeOffset.UtcNow, processId = Environment.ProcessId,
@@ -101,25 +94,12 @@ public sealed class Sdk2LifecycleSoakTests
limits = new { retainedManagedGrowthBytes = 64L << 20, rssGrowthBytes = 128L << 20, fdGrowth = 32 },
coverage = "encrypted sessions; scalar/Int16[] Raw+UNS; subscription removal; browse abandonment; history release/cross-session/close; surgical changes/rebuild; revocation; paused TCP read/recovery; same-PKI restart",
});
while (saturation is not null ? cycles < 385 : watch.Elapsed < TimeSpan.FromSeconds(seconds))
while (watch.Elapsed < TimeSpan.FromSeconds(seconds))
{
if (saturation is not null)
{
if (cycles == 320) await saturation.AttachAsync(ct);
if (cycles == 320) saturation.Boundary("window1-start", 321, restarts);
if (cycles == 352) saturation.Boundary("window2-start", 353, restarts);
if (cycles == 384)
{
await saturation.DetachAndDrainAsync(ct);
await LogAsync(evidence, new { kind = "saturation-settling-start", cycle = 385, restarts });
}
}
var cycle = Stopwatch.StartNew();
await ExerciseAsync(rig, config, ++cycles, evidence, ct);
await EventuallyAsync(() => PublicCounts(rig) == (0, 0, 0), ct);
var sample = await CaptureAsync(rig, config.CertificateManager, watch.Elapsed, cycles, clientRoot, serverRoot, ct);
// Retain the failing observation too; rejected diagnostic rows never count as passing quiescent samples.
await LogAsync(evidence, new { kind = "resource-observation", resources = sample });
sample.Sessions.ShouldBe(0); sample.Subscriptions.ShouldBe(0); sample.Items.ShouldBe(0);
sample.BrowsePoints.ShouldBe(0); sample.HistoryPoints.ShouldBe(0);
sample.LiveDataItems.ShouldBe(0); sample.LiveEventItems.ShouldBe(0); sample.MonitoredNodes.ShouldBe(0);
@@ -130,22 +110,9 @@ public sealed class Sdk2LifecycleSoakTests
sample.CachedCertificateStores.ShouldBe(baseline.CachedCertificateStores);
sample.Variables.ShouldBe(baseline.Variables); sample.Folders.ShouldBe(baseline.Folders);
sample.Conditions.ShouldBe(baseline.Conditions);
if (firstRejectedBudget is null)
{
try
{
sample.Descriptors.ShouldBeLessThanOrEqualTo(baseline.Descriptors + 32);
sample.ManagedBytes.ShouldBeLessThanOrEqualTo(baseline.ManagedBytes + (64L << 20));
sample.RssBytes.ShouldBeLessThanOrEqualTo(baseline.RssBytes + (128L << 20));
}
catch (ShouldAssertException error) when (mode == "profile" || saturation is not null)
{
// Investigative sampling may continue after rejection, but can never turn it into a pass.
firstRejectedBudget = ExceptionDispatchInfo.Capture(error);
await LogAsync(evidence, new { kind = "budget-rejected", cycle = cycles,
seconds = watch.Elapsed.TotalSeconds, resources = sample, baseline, failure = error.ToString() });
}
}
sample.Descriptors.ShouldBeLessThanOrEqualTo(baseline.Descriptors + 32);
sample.ManagedBytes.ShouldBeLessThanOrEqualTo(baseline.ManagedBytes + (64L << 20));
sample.RssBytes.ShouldBeLessThanOrEqualTo(baseline.RssBytes + (128L << 20));
retained.Enqueue(sample);
if (retained.Count > 24) retained.Dequeue();
if (retained.Count == 24)
@@ -156,8 +123,7 @@ public sealed class Sdk2LifecycleSoakTests
"retained managed memory must not grow monotonically across 24 quiescent cycles");
}
}
await LogAsync(evidence, new { kind = firstRejectedBudget is null ? "quiescent" : "quiescent-budget-rejected",
cycleSeconds = cycle.Elapsed.TotalSeconds, resources = sample });
await LogAsync(evidence, new { kind = "quiescent", cycleSeconds = cycle.Elapsed.TotalSeconds, resources = sample });
if (cycles % 4 == 0)
{
var restart = Stopwatch.StartNew();
@@ -166,47 +132,19 @@ public sealed class Sdk2LifecycleSoakTests
restarts++;
await LogAsync(evidence, new { kind = "restart", cycles, restarts, seconds = restart.Elapsed.TotalSeconds });
}
if (saturation is not null)
{
if (cycles == 352) saturation.Boundary("window1-end", cycles, restarts);
if (cycles == 384) saturation.Boundary("window2-end", cycles, restarts);
}
}
// Preserve the first assertion and its original stack even if later RSS samples decrease.
// This happens before any completion record; finally still releases the owned rig and client.
if (saturation is null) firstRejectedBudget?.Throw();
var phaseSeconds = watch.Elapsed.TotalSeconds;
if (saturation is not null)
{
cycles.ShouldBe(385); restarts.ShouldBe(96);
phaseSeconds.ShouldBeLessThanOrEqualTo(1800, "there is no saturation cutoff extension");
}
else
{
cycles.ShouldBeGreaterThanOrEqualTo(4, "diagnostic mode must exercise at least one actual restart");
restarts.ShouldBeGreaterThan(0);
watch.Elapsed.TotalSeconds.ShouldBeGreaterThanOrEqualTo(seconds);
}
var cleanupToken = saturation is null ? ct : TestContext.Current.CancellationToken;
cycles.ShouldBeGreaterThanOrEqualTo(4, "diagnostic mode must exercise at least one actual restart");
restarts.ShouldBeGreaterThan(0);
watch.Elapsed.TotalSeconds.ShouldBeGreaterThanOrEqualTo(seconds);
var finalClientCores = CertificateCores(config.CertificateManager);
var finalServerCores = CertificateCores(rig.Server.CertificateManager!);
await rig.DisposeAsync().AsTask().WaitAsync(TimeSpan.FromSeconds(20), cleanupToken);
await rig.DisposeAsync().AsTask().WaitAsync(TimeSpan.FromSeconds(20), ct);
rig = null;
await application.DisposeAsync().AsTask().WaitAsync(TimeSpan.FromSeconds(20), cleanupToken);
await application.DisposeAsync().AsTask().WaitAsync(TimeSpan.FromSeconds(20), ct);
application = null;
finalClientCores.Sum(CachedStores).ShouldBe(0);
finalServerCores.Sum(CachedStores).ShouldBe(0);
(await DescriptorCountsAsync(cleanupToken, clientRoot, serverRoot)).Pki.ShouldBe(0);
if (saturation is not null)
{
await LogAsync(evidence, new { kind = "saturation-result", utc = DateTimeOffset.UtcNow,
canonicalCycles = 384, settlingCycles = 1, restarts, phaseSeconds,
readiness = saturation.Readiness, coldBudgetRejected = firstRejectedBudget is not null,
shutdownOwnershipVerified = true, acceptance = false });
firstRejectedBudget?.Throw();
saturation.Readiness.ShouldBeTrue("diagnostic readiness requires both fixed observation windows");
return;
}
(await DescriptorCountsAsync(ct, clientRoot, serverRoot)).Pki.ShouldBe(0);
await LogAsync(evidence, new { kind = "complete", utc = DateTimeOffset.UtcNow, mode, cycles, restarts, seconds = watch.Elapsed.TotalSeconds,
satisfiesTwoHourDuration = mode == "full" && watch.Elapsed.TotalSeconds >= 7200 });
}
@@ -214,7 +152,6 @@ public sealed class Sdk2LifecycleSoakTests
{
await LogAsync(evidence, new { kind = "failed", cycles, restarts, seconds = watch.Elapsed.TotalSeconds,
exceptionType = error.GetType().FullName });
if (saturation is not null) firstRejectedBudget?.Throw();
throw;
}
finally
@@ -484,8 +421,7 @@ public sealed class Sdk2LifecycleSoakTests
int BrowsePoints, int HistoryPoints, int LiveDataItems, int LiveEventItems, int MonitoredNodes,
int Variables, int Folders, int Conditions, int CachedCertificateStores,
int Descriptors, int PkiDescriptors, long ManagedBytes, long RssBytes, long AllocatedBytes,
int Gen0, int Gen1, int Gen2, long GcCommittedBytes, long GcHeapBytes, long GcFragmentedBytes,
long PinnedObjects, long PendingFinalizers, long JitMethods, long JitIlBytes, long ActiveTimers, int Threads);
int Gen0, int Gen1, int Gen2);
private static async Task<Sample> CaptureAsync(ServerEventRig rig, object clientManager, TimeSpan elapsed, int cycle,
string clientRoot, string serverRoot, CancellationToken ct)
@@ -494,7 +430,6 @@ public sealed class Sdk2LifecycleSoakTests
var points = rig.Server.CurrentInstance.SessionManager.GetSessions().Select(ContinuationCounts).ToArray();
var descriptors = await DescriptorCountsAsync(ct, clientRoot, serverRoot);
var retained = GC.GetTotalMemory(forceFullCollection: true);
var memory = GC.GetGCMemoryInfo();
using var process = Process.GetCurrentProcess();
process.Refresh();
return new Sample(cycle, elapsed.TotalSeconds, counts.Sessions, counts.Subscriptions, counts.Items,
@@ -502,10 +437,7 @@ public sealed class Sdk2LifecycleSoakTests
rig.Nm.MonitoredNodeCount, rig.Nm.VariableCount, rig.Nm.FolderCount, rig.Nm.AlarmConditionCount,
CertificateCores(rig.Server.CertificateManager!).Sum(CachedStores) + CertificateCores(clientManager).Sum(CachedStores),
descriptors.Total, descriptors.Pki, retained, process.WorkingSet64, GC.GetTotalAllocatedBytes(),
GC.CollectionCount(0), GC.CollectionCount(1), GC.CollectionCount(2), memory.TotalCommittedBytes,
memory.HeapSizeBytes, memory.FragmentedBytes, memory.PinnedObjectsCount, memory.FinalizationPendingCount,
System.Runtime.JitInfo.GetCompiledMethodCount(), System.Runtime.JitInfo.GetCompiledILBytes(), Timer.ActiveCount,
process.Threads.Count);
GC.CollectionCount(0), GC.CollectionCount(1), GC.CollectionCount(2));
}
public static bool DescriptorProbeSupported => OperatingSystem.IsLinux() || OperatingSystem.IsMacOS();
@@ -672,232 +604,6 @@ public sealed class Sdk2LifecycleSoakTests
await output.WriteLineAsync(JsonSerializer.Serialize(value));
await output.FlushAsync();
}
// Test-only IPC. Receipts are fresh, run/PID-bound and published atomically by a separate observer.
private sealed class SaturationControl(string directory, string runId)
{
private const string Protocol = "sdk2-saturation-v1";
private int _observerPid;
private int _nextMarker;
internal bool Readiness { get; private set; }
internal static SaturationControl FromEnvironment()
{
var directory = Environment.GetEnvironmentVariable("OTOPCUA_SDK2_SATURATION_CONTROL")
?? throw new InvalidOperationException("A fresh saturation control directory is required.");
var runId = Environment.GetEnvironmentVariable("OTOPCUA_SDK2_SATURATION_RUN_ID")
?? throw new InvalidOperationException("An explicit saturation run id is required.");
if (!Path.IsPathFullyQualified(directory) || !Guid.TryParseExact(runId, "D", out _))
throw new InvalidOperationException("Saturation control identity is malformed.");
Directory.CreateDirectory(directory);
if (Directory.EnumerateFileSystemEntries(directory).Any())
throw new InvalidOperationException("Saturation control directory must be empty.");
_ = SaturationMarkers.Log;
return new SaturationControl(directory, runId);
}
internal async Task AttachAsync(CancellationToken ct)
{
await PublishAsync("attach-request.json", new { protocol = Protocol, runId,
targetPid = Environment.ProcessId, firstCycle = 321, completedRestarts = 80 }, ct);
using var receipt = await WaitAsync("active.json", ct);
var root = receipt.RootElement;
ValidateIdentity(root);
root.GetProperty("active").GetBoolean().ShouldBeTrue();
_observerPid = root.GetProperty("observerPid").GetInt32();
if (_observerPid <= 0 || _observerPid == Environment.ProcessId)
throw new InvalidDataException("A distinct observer process is required.");
using var enabled = CancellationTokenSource.CreateLinkedTokenSource(ct);
enabled.CancelAfter(TimeSpan.FromSeconds(5));
while (!SaturationMarkers.Log.IsEnabled()) await Task.Delay(10, enabled.Token);
}
internal void Boundary(string name, int cycle, int restarts)
{
(string Name, int Cycle, int Restarts)[] expected =
[ ("window1-start", 321, 80), ("window1-end", 352, 88),
("window2-start", 353, 88), ("window2-end", 384, 96) ];
if (_observerPid == 0 || _nextMarker >= expected.Length || expected[_nextMarker] != (name, cycle, restarts))
throw new InvalidOperationException("Saturation boundary is out of order.");
if (!SaturationMarkers.Log.IsEnabled()) throw new InvalidOperationException("Saturation observer is not active.");
SaturationMarkers.Log.Boundary(runId, name, cycle, restarts);
_nextMarker++;
}
internal async Task DetachAndDrainAsync(CancellationToken ct)
{
if (_nextMarker != 4) throw new InvalidOperationException("Both fixed observation windows are required.");
await PublishAsync("detach-request.json", new { protocol = Protocol, runId,
targetPid = Environment.ProcessId, lastCycle = 384, completedRestarts = 96 }, ct);
using var receipt = await WaitAsync("drained.json", ct);
Readiness = ValidateDrained(receipt.RootElement, runId, Environment.ProcessId, _observerPid);
using var disabled = CancellationTokenSource.CreateLinkedTokenSource(ct);
disabled.CancelAfter(TimeSpan.FromSeconds(5));
while (SaturationMarkers.Log.IsEnabled()) await Task.Delay(10, disabled.Token);
}
internal static bool ValidateDrained(JsonElement root, string expectedRun, int target, int observer)
{
if (root.GetProperty("protocol").GetString() != Protocol || root.GetProperty("runId").GetString() != expectedRun
|| root.GetProperty("targetPid").GetInt32() != target || root.GetProperty("observerPid").GetInt32() != observer
|| !root.GetProperty("drained").GetBoolean() || root.GetProperty("active").GetBoolean())
throw new InvalidDataException("Saturation drained receipt identity or ownership is invalid.");
var windows = root.GetProperty("windows").EnumerateArray().ToArray();
if (windows.Length != 2) throw new InvalidDataException("Exactly two fixed windows are required.");
var ready = root.GetProperty("markersValid").GetBoolean();
foreach (var name in new[] { "eventsLost", "sdkFactoryJit", "sdkFactoryLoads", "sdkAssemblyLoads" })
{
var value = root.GetProperty(name).GetInt64();
if (value < 0) throw new InvalidDataException("Negative diagnostic count.");
ready &= value == 0;
}
for (var index = 0; index < 2; index++)
{
var window = windows[index];
if (window.GetProperty("name").GetString() != $"window{index + 1}"
|| window.GetProperty("firstCycle").GetInt32() != 321 + index * 32
|| window.GetProperty("lastCycle").GetInt32() != 352 + index * 32
|| window.GetProperty("startRestarts").GetInt32() != 80 + index * 8
|| window.GetProperty("endRestarts").GetInt32() != 88 + index * 8)
throw new InvalidDataException("Saturation window boundaries are invalid.");
foreach (var name in new[] { "sdkFactoryJit", "sdkFactoryLoads", "sdkAssemblyLoads" })
{
var value = window.GetProperty(name).GetInt64();
if (value < 0) throw new InvalidDataException("Negative window diagnostic count.");
ready &= value == 0;
}
}
return ready;
}
private void ValidateIdentity(JsonElement root)
{
if (root.GetProperty("protocol").GetString() != Protocol || root.GetProperty("runId").GetString() != runId
|| root.GetProperty("targetPid").GetInt32() != Environment.ProcessId)
throw new InvalidDataException("Saturation observer receipt identity is invalid.");
}
private async Task PublishAsync(string name, object value, CancellationToken ct)
{
var temporary = Path.Combine(directory, name + ".tmp");
await using (var stream = new FileStream(temporary, FileMode.CreateNew, FileAccess.Write))
await JsonSerializer.SerializeAsync(stream, value, cancellationToken: ct);
File.Move(temporary, Path.Combine(directory, name));
}
private async Task<JsonDocument> WaitAsync(string name, CancellationToken ct)
{
using var bounded = CancellationTokenSource.CreateLinkedTokenSource(ct);
bounded.CancelAfter(TimeSpan.FromSeconds(45));
var path = Path.Combine(directory, name);
while (!File.Exists(path))
{
if (File.Exists(Path.Combine(directory, "observer-failed.json")))
throw new InvalidOperationException("The saturation observer failed before completing its receipt.");
await Task.Delay(25, bounded.Token);
}
return JsonDocument.Parse(await File.ReadAllTextAsync(path, bounded.Token));
}
}
[Fact]
public void Saturation_protocol_rejects_stale_identity_and_nonzero_readiness_counts()
{
var run = Guid.NewGuid().ToString("D");
JsonDocument Receipt(string identity, int pid = 100, int lost = 0, int jit = 0, int loads = 0, int assemblies = 0) =>
JsonDocument.Parse(JsonSerializer.Serialize(new { protocol = "sdk2-saturation-v1", runId = identity,
targetPid = pid, observerPid = 200, active = false, drained = true, markersValid = true,
eventsLost = lost, sdkFactoryJit = jit, sdkFactoryLoads = loads, sdkAssemblyLoads = assemblies,
windows = new[] {
new { name = "window1", firstCycle = 321, lastCycle = 352, startRestarts = 80, endRestarts = 88,
sdkFactoryJit = jit, sdkFactoryLoads = loads, sdkAssemblyLoads = assemblies },
new { name = "window2", firstCycle = 353, lastCycle = 384, startRestarts = 88, endRestarts = 96,
sdkFactoryJit = 0, sdkFactoryLoads = 0, sdkAssemblyLoads = 0 } } }));
using var good = Receipt(run);
SaturationControl.ValidateDrained(good.RootElement, run, 100, 200).ShouldBeTrue();
using var stale = Receipt(Guid.NewGuid().ToString("D"));
Should.Throw<InvalidDataException>(() => SaturationControl.ValidateDrained(stale.RootElement, run, 100, 200));
using var wrongPid = Receipt(run, pid: 101);
Should.Throw<InvalidDataException>(() => SaturationControl.ValidateDrained(wrongPid.RootElement, run, 100, 200));
foreach (var denied in new[] { Receipt(run, lost: 1), Receipt(run, jit: 1), Receipt(run, loads: 1), Receipt(run, assemblies: 1) })
{
using (denied) SaturationControl.ValidateDrained(denied.RootElement, run, 100, 200).ShouldBeFalse();
}
}
[Fact]
public async Task Saturation_protocol_missing_observer_is_bounded()
{
var directory = Path.Combine(Path.GetTempPath(), $"otua-saturation-control-{Guid.NewGuid():N}");
Directory.CreateDirectory(directory);
try
{
using var deadline = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken);
deadline.CancelAfter(TimeSpan.FromMilliseconds(100));
var control = new SaturationControl(directory, Guid.NewGuid().ToString("D"));
await Should.ThrowAsync<OperationCanceledException>(() => control.AttachAsync(deadline.Token));
File.Exists(Path.Combine(directory, "active.json")).ShouldBeFalse();
}
finally { Directory.Delete(directory, recursive: true); }
}
public static bool SaturationObserverAvailable => File.Exists(Environment.GetEnvironmentVariable("OTOPCUA_SDK2_SATURATION_OBSERVER"));
[Theory(SkipUnless = nameof(SaturationObserverAvailable), Skip = "Requires the separately built diagnostic observer.")]
[InlineData(false)]
[InlineData(true)]
public async Task Saturation_protocol_actual_observer_drains_and_rejects_extra_foreign_markers(bool foreignMarker)
{
var directory = Path.Combine(Path.GetTempPath(), $"otua-saturation-observer-{Guid.NewGuid():N}");
Directory.CreateDirectory(directory);
var run = Guid.NewGuid().ToString("D");
var control = new SaturationControl(directory, run);
var observerPath = Environment.GetEnvironmentVariable("OTOPCUA_SDK2_SATURATION_OBSERVER")!;
var frameworkDependent = observerPath.EndsWith(".dll", StringComparison.OrdinalIgnoreCase);
using var observer = new Process { StartInfo = new ProcessStartInfo(frameworkDependent ? "dotnet" : observerPath)
{ RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false } };
if (frameworkDependent) observer.StartInfo.ArgumentList.Add(observerPath);
observer.StartInfo.ArgumentList.Add(directory);
observer.StartInfo.ArgumentList.Add(run);
using var deadline = CancellationTokenSource.CreateLinkedTokenSource(TestContext.Current.CancellationToken);
deadline.CancelAfter(TimeSpan.FromSeconds(40));
try
{
observer.Start().ShouldBeTrue();
var stdout = observer.StandardOutput.ReadToEndAsync(deadline.Token);
var stderr = observer.StandardError.ReadToEndAsync(deadline.Token);
await control.AttachAsync(deadline.Token);
if (!foreignMarker) SaturationProtocolRuntimeProbe(1).ShouldBe(2);
if (foreignMarker) SaturationMarkers.Log.Boundary(Guid.NewGuid().ToString("D"), "window1-start", 321, 80);
control.Boundary("window1-start", 321, 80);
await Task.Delay(20, deadline.Token);
control.Boundary("window1-end", 352, 88);
control.Boundary("window2-start", 353, 88);
await Task.Delay(20, deadline.Token);
control.Boundary("window2-end", 384, 96);
await control.DetachAndDrainAsync(deadline.Token);
control.Readiness.ShouldBe(!foreignMarker);
if (!foreignMarker)
{
using var parsed = JsonDocument.Parse(await File.ReadAllTextAsync(Path.Combine(directory, "drained.json"), deadline.Token));
parsed.RootElement.GetProperty("runtimeControlJit").GetInt32().ShouldBeGreaterThan(0,
"the known non-SDK probe must prove runtime-provider delivery and method classification");
}
SaturationMarkers.Log.IsEnabled().ShouldBeFalse("Stop and stream drain precede return to the target");
await observer.WaitForExitAsync(deadline.Token);
observer.ExitCode.ShouldBe(0, await stderr);
await stdout;
}
finally
{
if (!observer.HasExited) { observer.Kill(entireProcessTree: true); await observer.WaitForExitAsync(TestContext.Current.CancellationToken); }
Directory.Delete(directory, recursive: true);
}
}
[System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.NoInlining
| System.Runtime.CompilerServices.MethodImplOptions.NoOptimization)]
private static int SaturationProtocolRuntimeProbe(int value) => value + 1;
[EventSource(Name = "OtOpcUa-Sdk2-Saturation")]
private sealed class SaturationMarkers : EventSource
{
internal static readonly SaturationMarkers Log = new();
[Event(1, Level = EventLevel.Informational)]
public void Boundary(string runId, string name, int cycle, int restarts) => WriteEvent(1, runId, name, cycle, restarts);
}
private sealed class AcceptWrites : IOpcUaNodeWriteGateway
{
public Task<NodeWriteOutcome> WriteAsync(string id, object? value, AddressSpaceRealm realm, CancellationToken ct) =>
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/Sdk2OpaqueScriptedScopeBaselineTests.cs
-44
View File
@@ -1,44 +0,0 @@
// The same application-authored opaque ID and two grant outcomes are independently exercised on SDK1.5.
using Opc.Ua;
using Opc.Ua.Client;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using static ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.ServerEventRig;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
public sealed class Sdk2OpaqueScriptedScopeBaselineTests
{
[Fact]
public async Task Opaque_scripted_ID_is_denied_by_its_equipment_grant_but_allowed_by_a_cluster_grant()
{
var ct = TestContext.Current.CancellationToken;
await using var rig = await StartAsync("BaselineScriptedScope", encrypted: false, ct);
rig.KeyRoles = [];
const NodePermissions permissions = NodePermissions.ReadOnly | NodePermissions.AlarmAcknowledge | NodePermissions.MethodCall;
rig.Enforce(true, [Key()], new NodeAcl
{
NodeAclId = "narrow", ClusterId = ServerEventRig.Cluster, Principal = KeyPrincipal,
ScopeKind = NodeAclScopeKind.Equipment, ScopeId = $"eq-{EqA}", PermissionFlags = permissions,
});
rig.MaterialiseScripted();
var routed = 0;
rig.Nm.AlarmCommandRouter = _ => routed++;
using var session = await rig.ConnectAsync(anonymous: false, ct);
rig.Project(false, new AlarmConditionSnapshot(true, true, true, true, AlarmShelvingKind.Unshelved, 500, "scope"));
var condition = rig.Nm.TryGetAlarmCondition(ScriptedAlarmId, AddressSpaceRealm.Uns).ShouldNotBeNull();
var eventId = condition.EventId.ShouldNotBeNull().Value;
var denied = await Should.ThrowAsync<ServiceResultException>(() => session.CallAsync(condition.NodeId,
MethodIds.AcknowledgeableConditionType_Acknowledge, ct, new Variant[] { Variant.From(eventId), Variant.From(new LocalizedText("denied")) }));
denied.StatusCode.ShouldBe(StatusCodes.BadUserAccessDenied);
routed.ShouldBe(0);
rig.Enforce(true, [Key()], ClusterGrant(KeyPrincipal, "cluster-positive", permissions));
await session.CallAsync(condition.NodeId, MethodIds.AcknowledgeableConditionType_Acknowledge, ct,
new Variant[] { Variant.From(eventId), Variant.From(new LocalizedText("allowed")) });
routed.ShouldBe(1);
await session.CloseAsync(ct);
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/Sdk2RepeatedAlarmAuthorityWireTests.cs
-126
View File
@@ -1,126 +0,0 @@
using Microsoft.Extensions.Logging.Abstractions;
using Opc.Ua;
using Opc.Ua.Client;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.Core.Authorization;
using ZB.MOM.WW.OtOpcUa.OpcUaServer.Security;
using static ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.ServerEventRig;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
/// <summary>The SDK2 already-acknowledged/confirmed compatibility path must still consult the current application authority.</summary>
public sealed class Sdk2RepeatedAlarmAuthorityWireTests
{
public static TheoryData<bool, string, string> Cases => new(
from operation in new[] { (Native: true, Method: "Acknowledge"), (Native: false, Method: "Acknowledge"), (Native: false, Method: "Confirm") }
from change in new[] { "grant", "key", "expiry" }
select (operation.Native, operation.Method, change));
[Theory]
[MemberData(nameof(Cases))]
public async Task A_current_trunk_retry_uses_its_application_grant_and_current_key(bool native, string method, string change)
{
var ct = TestContext.Current.CancellationToken;
await using var rig = await StartAsync("RepeatAlarmAuthority", encrypted: true, ct);
rig.KeyRoles = [];
var clock = new Clock();
var key = Key(); key.ExpiresUtc = clock.GetUtcNow().UtcDateTime.AddMinutes(10);
var permissions = NodePermissions.ReadOnly | NodePermissions.AlarmAcknowledge | NodePermissions.AlarmConfirm | NodePermissions.MethodCall;
var grant = new NodeAcl
{
NodeAclId = native ? "narrow-native-alarm" : "cluster-scripted-alarm", ClusterId = ServerEventRig.Cluster, Principal = KeyPrincipal,
ScopeKind = native ? NodeAclScopeKind.RawDevice : NodeAclScopeKind.Cluster,
ScopeId = native ? "alarm-owner" : null, PermissionFlags = permissions,
};
IReadOnlyDictionary<AclScopeRef, IReadOnlyList<NodeAclPath>> scopes =
new Dictionary<AclScopeRef, IReadOnlyList<NodeAclPath>>
{
[new(grant.ScopeKind, "alarm-owner")] = native
? [new([NodeAclRealms.Raw, "Plant", "Modbus", "dev1"])]
: [],
};
var state = rig.Enforce(true, [key], scopes, grant);
rig.Nm.Authorizer = new DataPlaneAuthorizer(state, new TriePermissionEvaluator(state.Tries), true, NullLogger.Instance, clock);
if (native) rig.MaterialiseNative(); else rig.MaterialiseScripted();
var realm = native ? AddressSpaceRealm.Raw : AddressSpaceRealm.Uns;
var path = native ? RawAlarmPath : ScriptedAlarmId;
var routed = 0;
rig.Nm.NativeAlarmAckRouter = _ => Interlocked.Increment(ref routed);
rig.Nm.AlarmCommandRouter = _ => Interlocked.Increment(ref routed);
using var session = await rig.ConnectAsync(anonymous: false, ct, encrypted: true);
var identity = rig.Server.CurrentInstance.SessionManager.GetSessions().ShouldHaveSingleItem().EffectiveIdentity
.ShouldBeOfType<RoleCarryingUserIdentity>();
identity.Roles.ShouldBeEmpty("the positive retry must be authorized by the application grant alone");
// Native: narrow raw-device scope. Scripted: preserve the existing cluster-scope contract;
// opaque scripted IDs are not under their equipment ACL path (a separately tracked baseline defect).
identity.ApiKeyId.ShouldBe(KeyId);
var events = new RecordedEvents();
var subscription = await RecordedEvents.SubscribeAsync(session, events, ct);
var item = RecordedEvents.AddEventItem(subscription, ObjectIds.Server);
await subscription.ApplyChangesAsync(ct);
item.Status.Error.ShouldBeNull();
rig.Project(native, new AlarmConditionSnapshot(true, true, true, true, AlarmShelvingKind.Unshelved, 500, "retry"));
await RecordedEvents.WaitUntilAsync(() => events.ConditionEvents(item, "retry").Count == 1);
var condition = rig.Nm.TryGetAlarmCondition(path, realm).ShouldNotBeNull();
var methodId = method == "Acknowledge" ? MethodIds.AcknowledgeableConditionType_Acknowledge : MethodIds.AcknowledgeableConditionType_Confirm;
var eventIdNode = condition.EventId.ShouldNotBeNull();
var commentNode = condition.Comment.ShouldNotBeNull();
var ackNode = condition.AckedState.ShouldNotBeNull().Id.ShouldNotBeNull();
var confirmNode = condition.ConfirmedState.ShouldNotBeNull().Id.ShouldNotBeNull();
var initialId = eventIdNode.Value;
await session.CallAsync(condition.NodeId, methodId, ct,
Variant.From(initialId), Variant.From(new LocalizedText("allowed retry")));
// A commented re-ack reports its comment. An already-confirmed retry has no state delta
// and remains silent under the existing Confirm reporting contract.
var expectedEvents = method == "Confirm" ? 1 : 2;
if (expectedEvents == 2)
await RecordedEvents.WaitUntilAsync(() => events.ConditionEvents(item, "retry").Count == expectedEvents);
routed.ShouldBe(1, "each authorized call reaches its owner once; the scripted owner decides whether a retry changes state");
var currentId = eventIdNode.Value;
currentId.IsEmpty.ShouldBeFalse();
var beforeEvents = events.Of(item).Count;
var beforeRoutes = Volatile.Read(ref routed);
var beforeComment = commentNode.Value;
var beforeAck = ackNode.Value;
var beforeConfirm = confirmNode.Value;
var keys = new[] { key };
var grants = new[] { grant };
if (change == "grant") grants = [];
if (change == "key") { var disabled = Key(); disabled.Enabled = false; keys = [disabled]; }
if (change == "expiry") clock.Advance(TimeSpan.FromMinutes(20));
else state.Apply(ServerEventRig.Cluster, "revoked", keys, grants, scopes);
var error = await Should.ThrowAsync<ServiceResultException>(() => session.CallAsync(condition.NodeId, methodId, ct,
Variant.From(currentId), Variant.From(new LocalizedText("must not be recorded"))));
error.StatusCode.ShouldBe(StatusCodes.BadUserAccessDenied);
Volatile.Read(ref routed).ShouldBe(beforeRoutes);
eventIdNode.Value.ShouldBe(currentId);
commentNode.Value.ShouldBe(beforeComment);
ackNode.Value.ShouldBe(beforeAck);
confirmNode.Value.ShouldBe(beforeConfirm);
// A subsequent authorized same-item sentinel drains any queued method event before the assertion.
clock.Reset();
state.Apply(ServerEventRig.Cluster, "restored", [key], [grant], scopes);
rig.Project(native, new AlarmConditionSnapshot(true, true, true, true, AlarmShelvingKind.Unshelved, 500, "sentinel"));
await RecordedEvents.WaitUntilAsync(() => events.Has(item, "sentinel"));
events.Of(item).Count.ShouldBe(beforeEvents + 1, "the denied retry emits nothing before the sentinel");
events.ConditionEvents(item, "retry").Count.ShouldBe(expectedEvents);
await subscription.DeleteAsync(true, ct);
await session.CloseAsync(ct);
}
private sealed class Clock : TimeProvider
{
private readonly DateTimeOffset _initial = DateTimeOffset.UtcNow;
private TimeSpan _offset;
public override DateTimeOffset GetUtcNow() => _initial + _offset;
public void Advance(TimeSpan time) => _offset += time;
public void Reset() => _offset = TimeSpan.Zero;
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/Sdk2TrustedChannelTests.cs
+9 -59
View File
@@ -52,44 +52,6 @@ public sealed class Sdk2TrustedChannelTests
rig.Writes.ShouldBe(1, "revocation must be checked before a cached narrow grant or dispatch");
}
[Theory]
[InlineData(OpcUaSecurityProfile.Basic256Sha256Sign)]
[InlineData(OpcUaSecurityProfile.Basic256Sha256SignAndEncrypt)]
public async Task Concurrent_IPv4_and_IPv6_activations_keep_their_own_transport_peer(OpcUaSecurityProfile profile)
{
var ct = TestContext.Current.CancellationToken;
await using var rig = await TrustedRig.StartAsync(profile, ct);
rig.HoldPeerActivations = true;
// Distinct test attempt markers share one key identity and trusted certificate. Both auth flows await the barrier.
var firstConnection = rig.ConnectAsync("contract-key-ipv4", ct, transportHost: "127.0.0.1");
var secondConnection = rig.ConnectAsync("contract-key-ipv6", ct, transportHost: "[::1]");
ISession[] connections;
try { connections = await Task.WhenAll(firstConnection, secondConnection); }
catch
{
if (firstConnection.IsCompletedSuccessfully) (await firstConnection).Dispose();
if (secondConnection.IsCompletedSuccessfully) (await secondConnection).Dispose();
throw;
}
using var ipv4 = connections[0];
using var ipv6 = connections[1];
rig.Peers.Count.ShouldBe(2);
var first4 = rig.PeerObservations.Single(x => x.Attempt == "IPv4");
var first6 = rig.PeerObservations.Single(x => x.Attempt == "IPv6");
first4.Peer.ShouldNotBeNull().Address.ShouldBe(IPAddress.Loopback);
first6.Peer.ShouldNotBeNull().Address.ShouldBe(IPAddress.IPv6Loopback);
first4.AfterAwaitPeer.ShouldBe(first4.Peer.Address);
first6.AfterAwaitPeer.ShouldBe(first6.Peer.Address);
foreach (var session in connections)
(await session.ReadValueAsync(rig.Node(session), ct)).StatusCode.ShouldBe(StatusCodes.Good);
await ipv4.ReconnectAsync(null, null, ct);
rig.Peers.Count.ShouldBe(3);
rig.Peers.Last().ShouldNotBeNull().Address.ShouldBe(IPAddress.Loopback,
"reactivation cannot inherit the concurrent IPv6 activation's peer");
await ipv6.CloseAsync(ct);
await ipv4.CloseAsync(ct);
}
[Theory]
[InlineData(OpcUaSecurityProfile.Basic256Sha256Sign)]
[InlineData(OpcUaSecurityProfile.Basic256Sha256SignAndEncrypt)]
@@ -123,19 +85,14 @@ public sealed class Sdk2TrustedChannelTests
private const string Cluster = "SDK2";
private const string PathId = "Plant/Device/Value";
private readonly string _root = Path.Combine(Path.GetTempPath(), $"otua-trusted-{Guid.NewGuid():N}");
private readonly ConcurrentDictionary<ApplicationConfiguration, ApplicationInstance> _applications = new();
private readonly Dictionary<ApplicationConfiguration, ApplicationInstance> _applications = [];
private readonly DataPlaneAuthorityState _authority = new();
private readonly OpcUaApplicationHostOptions _options;
private OpcUaApplicationHost _host = null!;
private int _port;
private int _writes;
private int _authCalls;
internal sealed record PeerObservation(string Attempt, IPEndPoint? Peer, IPAddress? AfterAwaitPeer);
internal ConcurrentQueue<PeerObservation> PeerObservations { get; } = new();
internal IReadOnlyList<IPEndPoint?> Peers => PeerObservations.Select(x => x.Peer).ToArray();
internal bool HoldPeerActivations { get; set; }
private readonly TaskCompletionSource _peerBarrier = new(TaskCreationOptions.RunContinuationsAsynchronously);
private int _peerBarrierArrivals;
internal ConcurrentQueue<IPEndPoint?> Peers { get; } = new();
internal OtOpcUaSdkServer Server { get; private set; } = null!;
internal int Writes => Volatile.Read(ref _writes);
internal int AuthCalls => Volatile.Read(ref _authCalls);
@@ -216,7 +173,7 @@ public sealed class Sdk2TrustedChannelTests
{
await config.ValidateAsync(ApplicationType.Client, ct);
await app.CheckApplicationInstanceCertificatesAsync(true, lifeTimeInMonths: 12, ct);
_applications[config] = app;
_applications.Add(config, app);
return config;
}
catch { await app.DisposeAsync(); throw; }
@@ -224,10 +181,10 @@ public sealed class Sdk2TrustedChannelTests
private async Task ReleaseConfigurationAsync(ApplicationConfiguration config)
{
if (_applications.TryRemove(config, out var app)) await app.DisposeAsync();
if (_applications.Remove(config, out var app)) await app.DisposeAsync();
}
internal async Task<ISession> ConnectAsync(string credential, CancellationToken ct, bool untrustedClient = false, string? transportHost = null)
internal async Task<ISession> ConnectAsync(string credential, CancellationToken ct, bool untrustedClient = false)
{
var pki = untrustedClient ? System.IO.Path.Combine(_root, $"untrusted-{Guid.NewGuid():N}") : ClientPki;
var config = await ConfigurationAsync(pki, ct);
@@ -241,7 +198,6 @@ public sealed class Sdk2TrustedChannelTests
var telemetry = DefaultTelemetry.Create(_ => { });
var endpoint = await CoreClientUtils.SelectEndpointAsync(config, $"opc.tcp://localhost:{_port}/OtOpcUa", true, telemetry, ct)
?? throw new InvalidOperationException("No secure endpoint.");
if (transportHost is not null) endpoint.EndpointUrl = $"opc.tcp://{transportHost}:{_port}/OtOpcUa";
return await new DefaultSessionFactory(telemetry).CreateAsync(config,
new ConfiguredEndpoint(null, endpoint, EndpointConfiguration.Create(config)), false, true,
"Sdk2TrustedContract", 60_000, new UserIdentity("apikey", Encoding.UTF8.GetBytes(credential)), default, ct);
@@ -261,22 +217,16 @@ public sealed class Sdk2TrustedChannelTests
new Dictionary<AclScopeRef, IReadOnlyList<NodeAclPath>>
{ [new(NodeAclScopeKind.RawDevice, "device")] = [new([NodeAclRealms.Raw, "Plant", "Device"])] });
public async Task<OpcUaUserAuthResult> AuthenticateUserNameAsync(string username, string password, IPEndPoint? peer, CancellationToken ct)
public Task<OpcUaUserAuthResult> AuthenticateUserNameAsync(string username, string password, IPEndPoint? peer, CancellationToken ct)
{
var attempt = password switch { "contract-key-ipv4" => "IPv4", "contract-key-ipv6" => "IPv6", _ => "Other" };
if (HoldPeerActivations && attempt != "Other")
{
if (Interlocked.Increment(ref _peerBarrierArrivals) == 2) _peerBarrier.TrySetResult();
await _peerBarrier.Task.WaitAsync(TimeSpan.FromSeconds(15), ct);
}
PeerObservations.Enqueue(new(attempt, peer is null ? null : new IPEndPoint(peer.Address, peer.Port), Server.ActivationPeerAddress));
return await AuthenticateUserNameAsync(username, password, ct);
Peers.Enqueue(peer is null ? null : new IPEndPoint(peer.Address, peer.Port));
return AuthenticateUserNameAsync(username, password, ct);
}
public Task<OpcUaUserAuthResult> AuthenticateUserNameAsync(string username, string password, CancellationToken ct)
{
Interlocked.Increment(ref _authCalls);
return Task.FromResult(username == "apikey" && (password is "contract-key" or "contract-key-ipv4" or "contract-key-ipv6")
return Task.FromResult(username == "apikey" && password == "contract-key"
? OpcUaUserAuthResult.AllowKey(Principal, [], KeyId) : OpcUaUserAuthResult.Deny("Invalid credential."));
}
public Task<NodeWriteOutcome> WriteAsync(string nodeId, object? value, AddressSpaceRealm realm, CancellationToken ct)