Joseph Doherty
2ad7213046
The cross-version matrix (#782) used a 55-char 'otua_matrix_' key, which stayed under SDK 2's 64-byte RsaEncryptedSecret threshold and hid #880. It now uses the ApiKeySecret.Generate shape (65 chars, random per run, never recorded). Every Shared leg adds production-length-API-key-wrong-secret-reaches-key-check, which must get BadIdentityTokenRejected rather than a format failure. SDK 2 clients also add sdk2-default-RsaEncryptedSecret-control, the negative control: the SDK's default handler must be refused with BadIdentityTokenInvalid by a 1.x server and admitted by an SDK 2 server. The runner passes the server SDK version from ready.json. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
110 lines
6.5 KiB
Markdown
110 lines
6.5 KiB
Markdown
# OPC UA SDK process compatibility
|
|
|
|
Run from a dedicated candidate checkout with .NET 10, Python 3, PowerShell 7,
|
|
and Docker. Supply the unchanged original application's existing `host/` and
|
|
`client/` publish directories. The output directory must not already exist.
|
|
|
|
```powershell
|
|
./scripts/e2e/test-opcua-sdk-compatibility.ps1 `
|
|
-BaselineRoot /absolute/original-checkout `
|
|
-BaselineArtifacts /absolute/original-publishes `
|
|
-CandidateRoot /absolute/dedicated-candidate-checkout `
|
|
-OutputDirectory /absolute/fresh-evidence-directory
|
|
```
|
|
|
|
The baseline console references the exact published production assemblies;
|
|
it does not rebuild or edit the original application. The candidate console
|
|
references the production host, Client.Shared and OPC UA driver projects, and
|
|
the candidate CLI is separately published. Different SDK versions always run
|
|
in different processes. Source revisions, SDK assemblies, SHA-256 hashes,
|
|
commands, per-case results, actual namespace URIs, negotiated Shared security,
|
|
loaded client SDK assemblies, timestamps and backend request logs are retained.
|
|
|
|
The required Shared legs are original→candidate, candidate→original and
|
|
candidate→candidate. A fourth original→original leg provides a reference for
|
|
both the application contract and the small performance sample. Every Shared
|
|
leg checks the exact scalar, array and condition browse identities, both
|
|
namespaces by URI, typed scalar/Int32[3] read/write, monitored Good quality and
|
|
source/server timestamps, the production alarm event filter and native
|
|
acknowledgment, and complete raw/event history across continuation points.
|
|
Event-history results include timestamp ties, exact IDs/order and positive and
|
|
negative `OfType` predicates. The historian is a deterministic fixture behind
|
|
the production server's session-bound continuation store; these results do
|
|
not certify any SQL Server, QuestDB or AVEVA backend.
|
|
|
|
Server, Shared client and driver require trusted Basic256Sha256
|
|
SignAndEncrypt peers, with certificate autoaccept disabled. Public certificates
|
|
are exchanged before sessions open. The test authenticator accepts one exact,
|
|
random, API-key-shaped credential from the process environment; production
|
|
ACL, write and condition-method checks use the resulting key identity. This
|
|
fixture does not replace cryptographic API-key-authenticator coverage.
|
|
|
|
The credential has the production `ApiKeySecret.Generate` shape and length,
|
|
65 characters (lmxopcua#880). The 55-character key used before then stayed
|
|
under SDK 2's 64-byte threshold, so it hid a real failure. SDK 2.0.1's own
|
|
UserName handler sends a longer password as an `RsaEncryptedSecret`, and an
|
|
SDK 1.5 server rejects that with `BadIdentityTokenInvalid`. The production
|
|
clients (Shared/CLI and the driver) force the legacy RSA format. Every Shared
|
|
leg adds two cases. The first sends a wrong secret of the same length, which
|
|
must reach the key check (`BadIdentityTokenRejected`, not a format failure).
|
|
The second runs only on an SDK 2 client and is the negative control: the
|
|
SDK's default handler with the valid key must be refused with
|
|
`BadIdentityTokenInvalid` by a 1.x server and admitted by an SDK 2 server.
|
|
The Shared client passes the server's SDK version from `ready.json` for it.
|
|
|
|
The driver and actual CLI run in a disposable `--network none` Linux container,
|
|
alongside a separate baseline or candidate server process. PKI lives only in
|
|
the mounted task directory; no host profile PKI or fleet endpoint is touched.
|
|
The driver hardcodes `LocalApplicationData/OtOpcUa/pki`, so Linux's task-local
|
|
`XDG_DATA_HOME` supplies isolation. The helper refuses a driver store outside
|
|
its task root. CLI uses its own task-local canonical store. CLI currently
|
|
forces `AutoAcceptCertificates=true` and exposes no strict-validation flag
|
|
(`CommandBase.CreateConnectionSettings`); the exchanged peers and encrypted
|
|
server remain in use, but CLI is not counted as strict client validation.
|
|
Driver history is N/A because the application removed that capability (#655);
|
|
Shared and CLI execute history reads instead.
|
|
|
|
Three measured rounds follow ten warmup operations per kind in each round:
|
|
50 scalar reads, scalar writes, array reads and array writes per kind. The
|
|
sample records nearest-rank p50/p95/p99, sequential throughput, client CPU,
|
|
allocated bytes, GC collections, working set, private bytes, threads and
|
|
handles. This is a small local diagnostic on a shared development host. It
|
|
has no production fan-out, reconnect, queue, backpressure, backend or soak
|
|
performance claim and does not establish Gate G4.
|
|
|
|
A separately published `server` helper can be used by other local consumer
|
|
checks. Supply a fresh directory and `OPCUA_MATRIX_PASSWORD` in the process
|
|
environment, then launch `dotnet ProcessHost.dll server <directory>`.
|
|
`ready.json`/`endpoint.txt` identify the ephemeral loopback port; touching
|
|
`<directory>/stop` shuts it down. Its default is strict SignAndEncrypt only.
|
|
`OPCUA_MATRIX_SERVER_SECURITY=none` is an explicit fixture-only override for
|
|
checking an existing unsecured consumer subset; the matrix runner clears that
|
|
override. Anonymous is disabled in either mode. Secure clients must place
|
|
their public certificates under `server-pki/trusted/certs` before server start
|
|
and trust the resulting `server-pki/own/certs` certificate.
|
|
|
|
Explicitly supplied timestamps are asserted. Unset source timestamps are
|
|
recorded separately as observations, with their original application/SDK wire
|
|
values. The harness does not change historian algorithms or normalize the
|
|
unset-timestamp sentinel to manufacture parity.
|
|
|
|
For the supplementary sustained warmup experiment, use the built consoles from
|
|
one completed matrix run:
|
|
|
|
```sh
|
|
python3 scripts/validation/opcua-sdk-compatibility/performance.py \
|
|
--baseline-harness /absolute/run/build-baseline/bin/Release/net10.0/ProcessHost.dll \
|
|
--candidate-harness /absolute/run/build-candidate/bin/Release/net10.0/ProcessHost.dll \
|
|
--output /absolute/fresh-performance-directory
|
|
```
|
|
|
|
This runs three fresh baseline/candidate process pairs serially, with 1,000
|
|
warmup and 2,000 measured operations per kind in each process. Both server and
|
|
client receive `DOTNET_TieredCompilation=0` for this experiment only; the
|
|
application's shipped configuration is unchanged. The recorded budget is a
|
|
15% increase in the median of the three p95 samples for any operation kind.
|
|
Exit code 2 means a diagnostic budget regression, while process failures
|
|
remain errors. Even an experiment within that budget does not certify the
|
|
complete production performance matrix. The allocation-per-operation field
|
|
is the average across all four operation kinds in a measured workload.
|