test: add isolated cross-version application compatibility harness

2026-11-10 12:18:15 -05:00
parent 41e65f6a22
commit 2d75cecbb9
6 changed files with 835 additions and 0 deletions
@@ -0,0 +1,96 @@
+# OPC UA SDK process compatibility
+
+Run from a dedicated candidate checkout with .NET 10, Python 3, PowerShell 7,
+and Docker. Supply the unchanged original application's existing `host/` and
+`client/` publish directories. The output directory must not already exist.
+
+```powershell
+./scripts/e2e/test-opcua-sdk-compatibility.ps1 `
+  -BaselineRoot /absolute/original-checkout `
+  -BaselineArtifacts /absolute/original-publishes `
+  -CandidateRoot /absolute/dedicated-candidate-checkout `
+  -OutputDirectory /absolute/fresh-evidence-directory
+```
+
+The baseline console references the exact published production assemblies;
+it does not rebuild or edit the original application. The candidate console
+references the production host, Client.Shared and OPC UA driver projects, and
+the candidate CLI is separately published. Different SDK versions always run
+in different processes. Source revisions, SDK assemblies, SHA-256 hashes,
+commands, per-case results, actual namespace URIs, negotiated Shared security,
+loaded client SDK assemblies, timestamps and backend request logs are retained.
+
+The required Shared legs are original→candidate, candidate→original and
+candidate→candidate. A fourth original→original leg provides a reference for
+both the application contract and the small performance sample. Every Shared
+leg checks the exact scalar, array and condition browse identities, both
+namespaces by URI, typed scalar/Int32[3] read/write, monitored Good quality and
+source/server timestamps, the production alarm event filter and native
+acknowledgment, and complete raw/event history across continuation points.
+Event-history results include timestamp ties, exact IDs/order and positive and
+negative `OfType` predicates. The historian is a deterministic fixture behind
+the production server's session-bound continuation store; these results do
+not certify any SQL Server, QuestDB or AVEVA backend.
+
+Server, Shared client and driver require trusted Basic256Sha256
+SignAndEncrypt peers, with certificate autoaccept disabled. Public certificates
+are exchanged before sessions open. The test authenticator accepts one exact,
+random, API-key-shaped credential from the process environment; production
+ACL, write and condition-method checks use the resulting key identity. This
+fixture does not replace cryptographic API-key-authenticator coverage.
+
+The driver and actual CLI run in a disposable `--network none` Linux container,
+alongside a separate baseline or candidate server process. PKI lives only in
+the mounted task directory; no host profile PKI or fleet endpoint is touched.
+The driver hardcodes `LocalApplicationData/OtOpcUa/pki`, so Linux's task-local
+`XDG_DATA_HOME` supplies isolation. The helper refuses a driver store outside
+its task root. CLI uses its own task-local canonical store. CLI currently
+forces `AutoAcceptCertificates=true` and exposes no strict-validation flag
+(`CommandBase.CreateConnectionSettings`); the exchanged peers and encrypted
+server remain in use, but CLI is not counted as strict client validation.
+Driver history is N/A because the application removed that capability (#655);
+Shared and CLI execute history reads instead.
+
+Three measured rounds follow ten warmup operations per kind in each round:
+50 scalar reads, scalar writes, array reads and array writes per kind. The
+sample records nearest-rank p50/p95/p99, sequential throughput, client CPU,
+allocated bytes, GC collections, working set, private bytes, threads and
+handles. This is a small local diagnostic on a shared development host. It
+has no production fan-out, reconnect, queue, backpressure, backend or soak
+performance claim and does not establish Gate G4.
+
+A separately published `server` helper can be used by other local consumer
+checks. Supply a fresh directory and `OPCUA_MATRIX_PASSWORD` in the process
+environment, then launch `dotnet ProcessHost.dll server <directory>`.
+`ready.json`/`endpoint.txt` identify the ephemeral loopback port; touching
+`<directory>/stop` shuts it down. Its default is strict SignAndEncrypt only.
+`OPCUA_MATRIX_SERVER_SECURITY=none` is an explicit fixture-only override for
+checking an existing unsecured consumer subset; the matrix runner clears that
+override. Anonymous is disabled in either mode. Secure clients must place
+their public certificates under `server-pki/trusted/certs` before server start
+and trust the resulting `server-pki/own/certs` certificate.
+
+Explicitly supplied timestamps are asserted. Unset source timestamps are
+recorded separately as observations, with their original application/SDK wire
+values. The harness does not change historian algorithms or normalize the
+unset-timestamp sentinel to manufacture parity.
+
+For the supplementary sustained warmup experiment, use the built consoles from
+one completed matrix run:
+
+```sh
+python3 scripts/validation/opcua-sdk-compatibility/performance.py \
+  --baseline-harness /absolute/run/build-baseline/bin/Release/net10.0/ProcessHost.dll \
+  --candidate-harness /absolute/run/build-candidate/bin/Release/net10.0/ProcessHost.dll \
+  --output /absolute/fresh-performance-directory
+```
+
+This runs three fresh baseline/candidate process pairs serially, with 1,000
+warmup and 2,000 measured operations per kind in each process. Both server and
+client receive `DOTNET_TieredCompilation=0` for this experiment only; the
+application's shipped configuration is unchanged. The recorded budget is a
+15% increase in the median of the three p95 samples for any operation kind.
+Exit code 2 means a diagnostic budget regression, while process failures
+remain errors. Even an experiment within that budget does not certify the
+complete production performance matrix. The allocation-per-operation field
+is the average across all four operation kinds in a measured workload.