lmxopcua/docs/plans/2026-11-10-opcua-sdk-2-evidence.md

# OPC UA SDK 2.0 execution evidence

Execution started 2026-11-10 for [#782](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/782), following the reviewed [plan](2026-11-10-opcua-sdk-2-upgrade.md). This records evidence and decisions; Gitea remains the status system.

## Baseline and authority

- Baseline `83ed880e4471344deaf008da9bd028c7e170bdc7` equaled `origin/master` when execution started; original checkout was clean. Master subsequently advanced to the independent HistorianGateway package update `16111a5e`, merged into the candidate without content conflicts. Dedicated managed worktree, branch `codex/opcua-sdk-2`.
- Target remains exact stable SDK `2.0.1`, upstream `07e62f72500cc0923a2bab940653c8c1116fa286`. Tagged source and restored assemblies control API decisions.
- Baseline packages are already `1.5.378.182`, not the plan preparation checkout's `.106`. #751 reflection helper exists. AggregateManager application configuration override already exists; preserve it.
- #782 previously parked the major upgrade; the user's current explicit execution instruction supersedes that scheduling decision. No deployment authorization overrides the Wonder-box remote-execution hold.
- Environment: macOS 26.5 arm64, .NET SDK 10.0.403, runtime 10.0.14; global.json 10.0.100 latestFeature accepts this SDK. Windows/Linux runtime, real ScadaBridge, AVEVA/backend and two-hour soak gates need actual execution evidence before release.
- Sources: nuget.org (`*` mapping), tracked `nuget-packages` MxGateway feed, Gitea family feed per NuGet.config. No source changes.
- Baseline restore succeeded; Release whole-solution build succeeded with **0 warnings, 0 errors** (18.44 s). Logs: ignored `artifacts/opcua2/83ed880e/baseline/`. Targeted baseline tests ran serially with `AKKA_TEST_TIMEFACTOR=3`, fresh per-project TRX and nonzero execution guards.

## SDK-consuming projects

Direct SDK references:

- `src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/ZB.MOM.WW.OtOpcUa.Client.Shared.csproj`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.csproj`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.csproj`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Session/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Session.csproj`
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ZB.MOM.WW.OtOpcUa.OpcUaServer.csproj`
- `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests.csproj`
- `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj`

Transitive SDK consumers (restored baseline graph):

- `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/ZB.MOM.WW.OtOpcUa.Client.CLI.csproj`
- `src/Client/ZB.MOM.WW.OtOpcUa.Client.UI/ZB.MOM.WW.OtOpcUa.Client.UI.csproj`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.csproj`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.csproj`
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.csproj`
- `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj`
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj`
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj`
- `tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests.csproj`
- `tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests.csproj`
- `tests/Client/ZB.MOM.WW.OtOpcUa.Client.UI.Tests/ZB.MOM.WW.OtOpcUa.Client.UI.Tests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.IntegrationTests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.Tests/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.Tests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests.csproj`
- `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests.csproj`
- `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ZB.MOM.WW.OtOpcUa.AdminUI.Tests.csproj`
- `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.csproj`
- `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.csproj`
- `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/ZB.MOM.WW.OtOpcUa.Host.Tests.csproj`
- `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj`
- `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ZB.MOM.WW.OtOpcUa.Runtime.Tests.csproj`

## Boundary decisions

Preserve existing CLR scalar widths, strings/nulls, Guid, UTC DateTime, byte[] and typed one-dimensional arrays at existing outward adapter/domain boundaries. Explicitly normalize ByteString/ArrayOf; no accidental SDK values in Core or driver Contracts. Copy mutable buffers when retained. Keep byte-array versus ByteString wire distinction at encoding sites. No new matrix capability. Existing Client.Shared SDK-facing DataValue/NodeId/Variant DTOs remain SDK-facing; this upgrade does not rewrite their public API. NodeIds bind namespace URI and identifier, including opaque identifiers.

Existing write conversions are part of the deployed contract: scalar Double→Float leniency and checked numeric-array conversions remain permitted where existing tests require them; Modbus/S7 refuse scalar-shaped array writes while pass-through drivers preserve their existing behavior. Reject unsupported rank/index range and unsafe conversions before driver I/O. Task 5's shorthand “wrong type rejected” means invalid **under this existing contract**, not blanket exact-type rejection. SDK stricter prevalidation must be reconciled with positive wire tests, without widening permissions.

No optional ManagedSession/DI/AOT/cluster redesign, trust relaxation, silent warning suppression, or SDK version substitution. All integration harness changes remain coordinator-owned; three implementation lanes start only after G0/G1 and frozen contract.

## Gates

- G0 baseline/design: accepted for migration, with corrections and external/trust-matrix gaps recorded below.
- G1 restored 2.0 graph / break inventory: passed; compilation convergence subsequently passed G2.
- G2 compilation convergence: passed; whole-solution Release builds have zero warnings/errors and temporary migration assets are removed.
- G3 behavioral/security/interoperability: pending.
- G4 lifecycle/performance/publish: failed/open for the canonical RSS gate (#799); scoped client lifecycle/publish results are recorded below.
- Astra security/ownership source findings: closed for the reviewed identity, history, audit and browse changes; final release review remains pending.

No upgraded artifact is approved for release.

## G0 design review corrections

Astra (`gpt-6-astra`, high) conditionally approved the design. Both actionable corrections are accepted:

1. Preserve scalar numeric leniency, checked numeric-array conversion (`int[]`→`short[]`) and pass-through scalar-shaped array dispatch. Positive `ArrayWriteWireTests` remain the compatibility oracle; never change expected statuses simply to accept SDK prevalidation regressions.
2. Peer evidence feeds `BoundedApiKeyStore` per-IP miss budgets, not a CIDR admission policy. Unknown peer keeps the process-wide budget fallback and valid-key admission. CIDR-only cases in the original plan are inapplicable to this baseline. Replace the reflection helper and preserve `ApiKeyMissBudgetPerEndpointTests`; `RemoteEndpointResolutionTests` does not exist.

Also preserve native alarm `ApiKeyId` dispatch/preflight, #763/#764 stale BadNoSubscription suppression and idle-session behavior (`OpcUaClientStaleNoSubscriptionTests`), typed event-history fields/statuses, xUnit v3 3.2.2 conventions and CI floor 12,000. Full G0 remains conditional on baseline test accounting.

### Frozen lane conversion recipe

Use exact target `VariantHelper.TryCastFrom(value, out Variant)` (no reflection fallback) only for already-supported built-ins; reject unsupported values with the existing owning operation status. Scalar and array wrappers must be detached from mutable caller buffers before retention. Use explicit typed Variant factories where the type is statically known. `AsBoxedObject(Variant.BoxingBehavior.Legacy)` normalizes ArrayOf to CLR arrays, but **does not normalize packed ByteString or DateTimeUtc scalars** in tagged 2.0.1: convert those explicitly to byte[] / UTC DateTime and map Uuid to Guid where existing domain payloads require it. Never claim default AsBoxedObject preserves CLR payloads. Preserve null versus empty wrapper semantics. Test mutation after publication/readback. Byte[] wire rank must match the existing node operation (Byte array versus scalar ByteString), not a global inference.

Existing Client.Shared ValueConverter text parsing policy is unchanged; do not expand supported text types. Domain snapshots have no picosecond fields; preserve SDK metadata on SDK-facing values but do not expand the domain snapshot schema. No new matrix flattening or materialization support. Preserve scalar-shaped pass-through upstream arrays already supported.

Baseline host and CLI artifacts published using existing Release outputs; 934 files hashed in `artifacts/opcua2/83ed880e/baseline/artifact-sha256.json`. They are local verification artifacts, not deployment packages or published releases.

## Baseline verification (G0)

| Project | Passed/executed | Skipped |
|---|---:|---:|
| ZB.MOM.WW.OtOpcUa.Client.CLI.Tests | 168/168 | 0 |
| ZB.MOM.WW.OtOpcUa.Client.Shared.Tests | 240/240 | 0 |
| ZB.MOM.WW.OtOpcUa.Client.UI.Tests | 130/130 | 0 |
| ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.IntegrationTests | 5/5 | 0 |
| ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.Tests | 19/19 | 0 |
| ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests | 5/5 | 0 |
| ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests | 298/298 | 0 |
| ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests | 133/133 | 0 |
| ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests | 1027/1027 | 1 |

**2025 baseline tests executed, zero failures.** All nine owning test projects passed nonzero execution guards. Server integration total is 133 (the earlier issue comment described 132). Skip-guard fixture tests 11/11 pass. The simulator is pinned `mcr.microsoft.com/iotedge/opc-plc:2.14.10`, task-owned container `otopcua-sdk2-baseline-plc` at reserved `opc.tcp://localhost:50000`; autoaccept/None is an interoperability baseline only, not trusted-channel proof. Existing server security tests cover profile configuration; a new explicit trusted secure-channel/certificate rejection matrix remains a G3 obligation. Windows/Linux runtime and actual ScadaBridge/backend evidence remain pending, not inferred from this macOS baseline.

G0 is accepted for migration with these explicitly accounted external/trust-matrix gaps and Astra's two design corrections adopted above. No baseline failure is hidden or attributed to SDK 2.0.

## Dependency uplift

Inspected all four exact 2.0.1 net10 nuspec dependency groups. They require Extensions Configuration/Abstractions, DI/Abstractions, Diagnostics, Http, Logging/Abstractions, Options/ConfigurationExtensions at 10.0.12; Client/Server require Hosting.Abstractions 10.0.12; Client also Http.Resilience 10.10.0; RateLimiting and Bcl.TimeProvider 10.0.12 are transitive. Raise existing central Extensions pins to 10.0.12 coherently (including Configuration.Json/Hosting/WindowsServices); no unrelated EF/ASP.NET uplift, no downgrade of Xml/Pkcs 10.0.13.

Standalone Extensions uplift with SDK still 1.5.378.182: whole-solution restore succeeds; Release build succeeds **0 warnings / 0 errors** (31.79s). Logs `artifacts/opcua2/56250a88/extensions-uplift/`. All-host runtime regressions (AdminUI, DataProtection/auth, ControlPlane, Host, Runtime) remain mandatory in G3 whole-solution testing; this compile check does not claim behavioral verification.

## G1 package/analyzer spike

Astra closed both G0 corrections and approved migration after independently verifying all nine TRX files. The standalone Extensions uplift built cleanly. All four SDK pins are now exactly 2.0.1. MigrationAnalyzer 2.0.1 is temporary in five direct production consumers, with `PrivateAssets=all` and **only** analyzers/build/buildtransitive assets: no runtime compatibility shim is referenced. Exact package ships Roslyn 4.14/5.0 assemblies; the selected analyzer loads on this SDK (UA diagnostics emitted, no CS9057). Tagged analyzer guidance is reviewed case by case; no blanket codefix pass. Generator collection shims must be removed with the package before G2.

Restore succeeds without downgrade suppression or source changes. The first compiler wave reports:

| Owning project | Distinct diagnostics |
|---|---:|
| ZB.MOM.WW.OtOpcUa.Client.Shared | 90 |
| ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests | 15 |
| ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Session | 8 |
| ZB.MOM.WW.OtOpcUa.OpcUaServer | 15 |

This is the exposed first wave, not the solution-wide final break count: server overrides, removed collections, immutable/null/Variant APIs, certificate manager, and async identity construction block downstream compilation. Log and exact diagnostics: `artifacts/opcua2/d9984028/sdk2-spike/{build.log,diagnostics.txt}`. Later diagnostics are inventoried by lane until G2 convergence; no compile failure is claimed as behavioral success.

Restored project graphs contain no mixed Foundation SDK versions: every Foundation package (including analyzer-only development transitive GDS assets) is 2.0.1. Host graph resolves Bcl.TimeProvider 10.0.12 and Http.Resilience 10.10.0; client assets resolve RateLimiting 10.0.12, while the Host shared framework prunes that separate package. G1 is satisfied for lane execution; full build/tests remain required at G2/G3. Frozen boundary contract is above; production and unit paths are split by plan ownership, integration/historian oracle edits coordinator-only.


## Integrated implementation and source reviews

The implementation is on `codex/opcua-sdk-2`, not deployed or merged to master.
Master's independently delivered HistorianGateway client/contracts 0.4.0 change
`16111a5e` is included, and master ancestry was merged at `88ee30e0` without a
content change. historiangw#119 is closed; historiangw#78 deployment is separate.

The server, client and driver lanes were integrated and validated in isolated
worktrees. Astra reviewed the design, aggregate compatibility architecture,
identity/peer provenance, certificate ownership, production history continuation
store, write conversions/revert ownership, alarm compatibility, and filtered event
and audit delivery. No source review is represented as release approval.

- Identity registry admission is installed before listeners and remains until
  shutdown. Rejected/disabled credentials cannot fall through to a default identity.
  SDK role augmentation preserves the application's principal and key ID.
- Activation uses public transport peer evidence, scoped across asynchronous work
  and restored in `finally`. Unknown peers retain the existing process-wide budget;
  this application has no CIDR admission or reverse-connect policy to invent.
- Immutable SDK values are normalized only at existing application boundaries.
  Scalar numeric leniency, checked one-dimensional conversions, pass-through arrays,
  byte/ByteString rank, null/empty and detached buffers remain pinned. Matrices remain
  unsupported. Public SDK-facing client DTOs remain SDK-facing.
- Session history tokens are consumed/released through the real SDK store, with
  exact node/realm, read kind, topology epoch and current authority checks. Existing
  canonical-query semantics are retained; no undocumented filter fingerprint was
  introduced.
- SDK2 alarm early AlreadyAcked/AlreadyConfirmed checks are adapted only for the
  current nonempty trunk EventId, preserving application authorization and routing.
  Write reverts bind the actual node instance and committed attempt, preventing a
  late failure from undoing a newer write/publish or replacement node.
- Audit delivery uses a private, revocable operation context for one exact source
  and type. Both existing permission callbacks can veto; no global role/node
  permission array is widened. Ordinary SDK automatic audits remain denied. The
  encrypted wire test proves exactly one application audit to the Read audience
  and none to the denied audience.

A cross-version wire test found an additional SDK2 creation change: `Create`
reset an alarm's preassigned reference to the null NodeId, hiding it from its raw
parent's hierarchical browse. Restoring HasComponent **after** Create fixed all
three supported alarm types in both realms (`fa823378`, integrated `41e65f6a`).
All 12 actual-node/wire regressions failed first, then passed. Full server units
at that fix: **1115 passed, zero failures, one existing skip**. Astra approved
its source scope and the matching separate-process results.

## Preserved aggregate algorithms and independent oracles

The owner explicitly chose current application results over changed SDK2
aggregate semantics. A private source adapter contains the eight original MIT
algorithm/factory files from 1.5.378.182 with original source hashes. It is not an
old SDK runtime. A fresh SDK AggregateManager registers the same 37 factories
exactly once; calling the base registration first would duplicate diagnostic nodes.

The separately pinned original-SDK generator produced **6,144 vectors**: the
original 4,144 plus 2,000 numeric-width, nonfinite/conversion, tie/picosecond and
configuration edge cases. Fixture SHA-256:
`2ec2a1ac97f05d199c18da55157f8bbe0276853c3ba73ebf1b509c535c5a7dd8`.
Original SDK assembly SHA-256:
`d8d9706dea577858325730cd89896826ed82f7bfcf9503ccd0260311663a69f4`.

All cases pass both directly and through the booted production aggregate manager.
Values, CLR types, all status bits, source/server time and picoseconds, exceptions,
queue acceptance and incremental/partial results are compared. SDK2 FILETIME zero
is explicitly observed as the original CLR unset sentinel; no calculation or
expected output was regenerated from the candidate.

The SQL application's PercentGood policy already differs from SDK1.5.378.182.
The same ten of 275 parity tests fail on the original and candidate with identical
names/messages. Independent application results and regression expectations stay
unchanged; [#796](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/796)
tracks that baseline debt. It is not a green test run.

## Local compatibility and security evidence

- Full wire integration reached **187 passed, zero failures/skips** in the second
  CI-equivalent run, excluding the separately gated E2E soak category. It includes
  all original contracts plus production continuation, trusted-channel, audit,
  hierarchy and current authority regressions.
- Focused final security probes: **17/17**, zero skips. Seven trusted cases cover
  explicit DER trust, Sign and SignAndEncrypt, invalid credentials, an untrusted
  client, same-session reactivation, and concurrent IPv4/IPv6 attempts correlated
  with their own peer before/after an await barrier. Eighteen earlier continuation
  cases cover raw/modified/events with alias, replacement, rebuild, grant/key
  revocation and expiry, proving no backend dispatch or saved-data disclosure.
- Native current-trunk retry tests use a narrow raw-device grant with zero global
  roles, then revoke/expire authority and require no route/state/EventId/event
  change. Scripted retry tests explicitly use the existing cluster grant.
  A separate real-session original/candidate probe preserves the opaque scripted
  ID's equipment-scope denial and broad-grant positive. This **preexisting**
  production feature gap remains [#797](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/797);
  a slash-qualified synthetic ID is not a valid UI-authored workaround.
- Target certificate manager/endpoint identity tests: **7/7** (valid, expired,
  revoked, missing issuer, issuer usage, hostname and URI). These are manager/API
  checks, not seven network handshakes.
- Corrected a preexisting client creation-policy bug: 2048 meant months, not RSA
  bits. New certificates use the SDK's 12-month default; existing longer-valid
  certificates and identical private-key bytes are reused. A real-factory red
  lifetime case became green; all **324 Shared tests passed**. RSA minimum remains
  the SDK's separate 2048-bit default.
- Separate-process original→candidate, candidate→original, candidate→candidate,
  and original→original Shared legs plus candidate driver/actual CLI legs:
  **62/62 passed**. Strict alarm identities remain asserted. Shared/server/driver
  use explicit mutual DER trust and encrypted API-key sessions; CLI's existing
  autoaccept is disclosed and is not rejection-policy evidence. The same harness
  first exposed the hierarchy defect at **60/62**. Commands, artifact hashes,
  policies and outcomes are checked in under
  `scripts/validation/opcua-sdk-compatibility/evidence/2026-11-10/`.
- The actual unchanged ScadaBridge adapter, SDK1.5.378.106, passed **9 None/API-key
  cases against each server** (18 total). Strict encrypted connections fail on
  both original/candidate because its application certificate is unconfigured:
  [ScadaBridge#121](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/121).
  That is a retained failure, not a secure interoperability pass. Reproducible
  isolated consumer receipts are in `scripts/validation/scadabridge-sdk-compatibility/`.
- Real local SQL Server: **195/195** on each version. QuestDB10.0.1: **233 passed,
  zero failed, one existing opt-in race-stress skip** on each version. Case maps
  match exactly. Only task databases/prefixes and the task QuestDB container/volume
  were removed. See [backend receipts](evidence/2026-11-10-opcua2/backend-matrix.md).
  This is not real AVEVA evidence.

## Solution, performance and package evidence

Whole-solution Release builds pass with **zero warnings/errors**, including all
inventory consumers. All restored production Foundation packages are 2.0.1; no
MigrationAnalyzer reference/runtime asset remains. The only old SDK is isolated
in the explicitly original-SDK validation process outside the solution.

The first full CI-equivalent run executed **18,459** cases with execution guard
passed (floor12,000), but failed203: 192 from an incorrect test-oracle reflection
port (repaired and **554/554 Core.Abstractions tests passed**), the ten known
PercentGood baseline failures, and a ScriptWorker250ms timing assertion.
The latter also fails independently on untouched baseline83ed880e (880.13ms),
so no threshold/filter/skip was changed; it is
[#798](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/798).
The fresh second full solution run produced **63 TRX files, 19,121 total,
18,486 executed, 18,475 passed, 11 failed and 635 existing skipped cases**.
Its test exit is **1**, execution guard **0** (floor12,000). The exact eleven
failures are the same ten PercentGood cases (#796) and one timing case (#798);
there are no additional failure classes. This is baseline-matching evidence,
**not a green solution run**. The command/logs are retained under ignored
`artifacts/opcua2/88ee30e0/solution-tests2/`;
[versioned counters and exact failures](evidence/2026-11-10-opcua2/solution-tests2.json)
retain the nonzero status. Fixture-backed gates remain separate.

Three alternating warmed lock measurements per version show medians17→15ms for
10k-tag rebuild and81→80ms for50k-node removal, with equal visit counts.
See [bounded subset and host-load caveats](evidence/2026-11-10-opcua2/lock-benchmarks.md).
The separate 72,000-call warmed native-process experiment records median p95
changes +14.95% scalar read, +1.58% scalar write, -16.28% array read, -17.24%
array write, plus lower allocated bytes/workload operation. It retains the
scalar-read outlier, small-workload scope, host-load and JIT-mode caveats. Neither
these samples nor the short cold-tiered measurements certify full G4.

Four self-contained publishes (Linux ARM64 and Windows x64 Host/CLI) pass with
runtime10.0.14, unique SDK DLL basenames, native RID validation and file hashes.
Actual Linux Host passes two start/stop cycles with unchanged certificate identity,
healthy web checks, real HMAC-key positive/negative cases and strict peer negatives.
Console/rolling logs contain zero task-secret occurrences. See
[publish/runtime receipts](evidence/2026-11-10-opcua2/publish-smoke.md).
Windows service/runtime installation remains
[#800](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/800), not inferred from
cross-publishing. Approved live AVEVA/consumer/pair validation remains
[#801](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/801).

## Resource finding and long-run scope

The Mac SDK-server restart soak failed twice around17 minutes at the unchanged
initial-RSS+128MiB guard. One run completed250 cycles/62 restarts; managed memory,
FDs, cached stores and logical collections remained bounded, but process RSS did
not meet its limit. **No two-hour pass is claimed.**
[#799](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/799) retains the failed
TRX/JSONL and active investigation. Explicit SDK wrapper tracking found created=
disposed and no outstanding/unreachable-undisposed wrapper after cleanup;
controlled private PFX imports/SDK copies accumulate native Apple Security
objects, unlike shared-handle/public-only controls. These controls do not attribute the entire RSS increase or isolate the server
from the co-located SDK client. Astra found no safe narrow public copy-policy
setting in exact2.0.1. Switching channel/session factories would change lifecycle
semantics and requires separate design. No unproven production patch or resource-
budget waiver was applied.

The original Mac PKI-specific descriptor zero was invalid because `/var` and
`/private/var` were compared literally. Corrected canonical, directory-bounded
instrumentation proves0→1→0 for an open test file and reaps a timed-out probe.
The original logical/global-FD evidence remains usable; its PKI-zero claim does
not. A corrected90-second smoke is explicitly not two-hour evidence.

The same-cap self-contained **Linux ARM64 server/node-manager workload also
failed**, at **248 cycles / 61 restarts / 903.508 seconds**: RSS353,931,264
exceeded351,850,496 bytes (cycle3 baseline+128MiB). Managed memory~20.5MB,
descriptors189, PKI descriptors0, cached stores2 and logical collections remained
bounded. Runtime10.0.14, SDK 2.0.1; test/container/launcher fail. This broadens
#799 beyond macOS; no complete/two-hour pass exists.
[Selected Linux failed-run receipts](evidence/2026-11-10-opcua2/server-linux-soak-failed.json)
retain hashes, unchanged limits, positive descriptor controls and failure.

The independent root-owned actual production Shared/driver/live-reinitialize/
browser lifecycle run on Linux **passed after 7,200.984 seconds**, with 5,420 cycles,
10,840 rounds per product, 10,840 additional live driver reinitializations and
43,360 verified certificate-manager disposals. All 543 samples had zero weak product
survivors and zero open PKI descriptors. Final managed memory was 3,988,520 bytes,
RSS 86,102,016 bytes and global FDs 148. Own certificates remained unchanged; the
manager disposal positive control passed. Container/test exit 0 and the real
monotonic duration guard passed. [Full counter/hash receipts](../../scripts/validation/opcua-client-lifecycle/evidence/2026-11-10-full/verification.json)
record source `c25561c6`, exact SDK 2.0.1 and Linux ARM64 runtime 10.0.3. The later
HG 1.0 consumer change is independently tested and does not relabel these artifacts.
This is a completed client lifecycle gate, **not an RSS performance-budget or full
G4 pass**. Its workload does not replace the failed canonical server soak.
G4 and release approval remain open.


## Final integrated source review and diagnostic instrumentation

Astra (`gpt-6-astra`, xhigh) reviewed integrated source `aa9af0dc` read-only and
reported **no new must-fix migration source/security finding**. It rechecked
identity registry/rejection, scoped public peer evidence, restricted identity
behavior, actual continuation ownership/node/topology binding, temporary audit
capability, exact alarm retry guards, write/node attempt ownership and client/
driver generation/disposal. Its independently inspected security receipt is
17/17 with zero skips. This is source/security review, **not release approval**:
Astra retains #799 and explicitly withholds G4/release approval on both platforms.
The reviewed production tree remains unchanged by later diagnostic commits.

Test-only instrumentation now writes the complete captured resource tuple before
resource assertions and includes GC committed/heap/fragmentation, pinned/finalizer,
JIT method/IL, thread and timer counters. Explicit bounded profile mode preserves
the first original budget rejection, marks subsequent samples rejected, and
rethrows it before completion; full/diagnostic modes still fail immediately.
The full7200-second minimum and every original cap are unchanged. Both descriptor
positive/timeout controls passed again, **2/2**, at integrated `b57595cb`.
Historical failed-run hashes and source revisions remain exact; they are not
relabeled as results from this newer instrumentation.

The pushed implementation is reviewable as
[draft PR#802](https://gitea.dohertylan.com/dohertj2/lmxopcua/pulls/802).
It is explicitly held from merge/release, and #782 remains open.


## Completed cross-platform RSS investigation

The [reproducers, report](../../scripts/validation/opcua-lifecycle-memory/README.md)
and [hashed evidence](../../scripts/validation/opcua-lifecycle-memory/evidence.json)
preserve four failed canonical workload/profile attempts and the unchanged caps.
A 30-minute Linux **diagnostic subset** completed on both versions: candidate
768 cycles/192 restarts, original736/184. Candidate retained managed memory was
19.25 MB, final RSS334.18 MB, with19,272 certificate wrappers created and disposed
and none outstanding. It has **no full-gate pass** and omits canonical history,
revocation, topology surgery and network-stall work. The original subset's growing
managed/timer footprint cannot provide a favorable steady-state performance
comparison denominator.

EventPipe attributed a large late step to6,690 distinct generated standard-node
factory methods receiving Tier1 compilation (49,141,484 cumulative native method
bytes). A later180-second interval had zero SDK/factory JIT starts and zero lost
events. These code-event sums are not RSS/PSS allocation measurements. This
supports a finite compilation burst in the subset; full-workload saturation has
not been established. macOS native Apple Security object retention continues
independently. No narrow application workaround, factory swap, runtime flag,
resource threshold change or acceptance-baseline reset was adopted.

Astra reviewed the packaged sources/hashes/raw-receipt consistency/sanitization
at integrated `6f793d25` and found no must-fix report/source issue. It approved
only a **prospective diagnostic saturation design**, recorded in the report:
384 fixed canonical cycles/96 restarts plus one settling cycle within1,800 seconds;
fixed traced windows321–352 and353–384; zero generated-factory compilations,
new SDK assembly loads and lost events; ownership checks active throughout.
That experiment had not run at that review; its later diagnostic result is recorded below and cannot pass G4. Matched full acceptance requires
an original-SDK adapter for the identical canonical workload and at least7,200
seconds **after** a prospectively reviewed fixed warmup. Historical failures and
macOS native retention remain open in #799. No release approval was given.


## Included HistorianGateway 1.0 consumer change

The user's added [historiangw#131](https://gitea.dohertylan.com/dohertj2/historiangw/issues/131)
was delivered independently on OtOpcUa master `230cef4e`, then merged into the
SDK2 branch at `67bc6123`. Both real published Client/Contracts packages are 1.0.0;
the adapter maps `AllowUntrustedServerCertificate = UseTls && configuredOptOut`
and retains strict TLS defaults and h2c certificate-option filtering. Astra
approved source/security and corrected real-wire tests. Full master Release
build: 0 warnings/errors; Gateway 523 passed/6 existing live skips, Runtime1,002
passed/3 existing retired-fixture skips, zero failures, execution guards passed.
The three new wire tests prove typed TLS rejection before RPC, explicit opt-out
success and h2c success. The issue was closed with a verification comment after
master push; [immutable package/test receipt](evidence/2026-11-10-historiangw-131.json).
Independent actual-package SDK2 Gateway verification also passed 523/0/6.

The integrated SDK2/HG 1.0 solution built with 0 warnings/errors in 46.91 seconds.
CI3 and the corrected follow-up run are recorded below; every command preserves
test/guard exits and the 12,000 execution floor. CI2 remains historical pre-HG 1.0 evidence.

All four self-contained Host/CLI publishes and actual Linux Host two-start/stop
checks were repeated on the HG 1.0 graph at `f2a8ce13`: all passed. Both Host
artifacts contain the exact published 1.0.0 Client/Contracts DLLs; the Foundation
SDK family remains 2.0.1. Six strict HMAC/peer cases, two Good published-CLI reads,
preserved certificate identity and zero task-secret matches passed. See the
[fresh publication/runtime report](evidence/2026-11-10-opcua2/publish-smoke-hg100.md).
Windows actual install/runtime #800 and live/paired-environment #801 remain pending.


## CI3 investigation and corrected final verification

The post-HG 1.0 full CI3 run at `f2a8ce13` produced **63 TRX, 19,124 total,
18,489 executed, 18,472 passed, 17 failed and 635 existing skipped** results.
Test exit 1 and execution guard 0 are retained in
[the complete summary](evidence/2026-11-10-opcua2/solution-tests3.json).
Eleven failures match the established baseline issues #796/#798. Five browser
failures came from a task simulator with differing advertised/internal and host
ports. Matching those ports restores browser 5/5 and driver 5/5 with no source
change. Actual SDK1.5.378.182 browser binaries also fail the different-port
advertisement with the same DNS error: [baseline NAT parity](evidence/2026-11-10-opcua2/browser-nat-parity.md).
The preexisting browser capability gap is [#806](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/806),
not a new SDK2 behavior or a security-policy workaround.

The final failure was a preexisting Runtime test-ordering race: an EF-presence
wait captured the first apply before its later NodeAdded announcement. A gated
controlled reproduction proves 1→2 with one EF and one announcement. The narrow
repair awaits correlated clean completion replies for both rebuilds, retaining
all idempotence assertions and existing budgets. Astra reviewed source causality
and RED/GREEN proof. SDK2 and independent SDK1.5 owning Runtime suites each pass
**1,002 tests, zero failures and 3 existing skips**. The test-only fix landed on
master `52d1373e`, merged at `75057d8c`; [#805](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/805)
closed with validation. [Exact repair evidence](evidence/2026-11-10-opcua2/rebuild-test-barrier.md).
No product code, expected result, timing budget or skip was changed.

The complete solution rebuilt at `75057d8c` with **zero warnings/errors**, 26.61 s.
Fresh full CI4 with the corrected simulator and completion-barrier repair completed
at `75057d8c`: **63 TRX, 19,124 total, 18,489 executed, 18,478 passed, 11 failed,
635 existing skips**. Test exit **1**, execution guard **0** at floor12,000.
All eleven remaining failures are the established ten PercentGood cases (#796)
and one ScriptWorker timing assertion (#798). The fixture and rebuild-ordering
cases pass in this same full run, with no additional failure classes or new skips.
[Versioned counters, exact failures and per-project hashes](evidence/2026-11-10-opcua2/solution-tests4.json)
retain the nonzero status. CI3 remains available; this baseline-matching result
is **not a green solution run**. Task-owned simulator was removed after completion.

## Canonical candidate generated-factory saturation result

The exact immutable bundle reviewed by Astra completed **384 canonical cycles,
96 restarts and one settling cycle in 1,425.242 seconds**, inside the fixed 1,800-second
cutoff. Both fixed trace windows and the continuous attached interval had zero
SDK generated-factory JIT/load events, zero new SDK assembly loads and zero lost
events. The observer detached/drained before settling; shutdown ownership checks
passed. This meets only the selected generated-factory readiness criterion.

The unchanged cold cap rejected at cycle 248: RSS 362,409,984 >358,838,272 bytes.
The original assertion was rethrown; test/container/launcher exit 1, no ordinary
completion or full-duration flag. Peak RSS 394,956,800 bytes; final managed memory
19,974,280 bytes. Allocation StackTrace tracking was enabled, so this is additional
instrumented diagnostic overhead and cannot serve as uninstrumented performance
or acceptance evidence. [Scoped report and hashed receipts](evidence/2026-11-10-opcua2/saturation-candidate.md).
#799 and G4 remain failed/open. A matched original-SDK canonical adapter is being
prepared before any further acceptance-protocol review; no warmed replacement
baseline, budget waiver, factory/runtime setting or production patch was adopted.


## Original canonical adapter limitation and final review boundary

The [original-SDK canonical adapter](../../scripts/validation/opcua-lifecycle-memory/baseline-canonical/README.md)
compiles against the unchanged `83ed880e` publications with zero warnings/errors;
its package/input/source hashes are preserved. A short normal-cycle control
completed encrypted sessions, typed writes/readbacks, all ten history calls,
revocation, surgery/rebuild, stalled-read recovery, subscription cleanup and both
client closes. The retained closed Session object still held one browse and six
history points while SessionManager had zero active sessions; the copied assertion
fails with exit1, including after a separate fixed250ms observation. Exact1.5
binary inspection shows Close removes diagnostics, while Dispose clears those
lists. Held-object state does not prove live server retention or an unbounded leak.

A separate active-session restart control cleared the observable session/
subscription/item/CP counts, then returned **exit2: unsupported ownership parity**.
Original ApplicationInstance/CertificateValidator have no SDK2 manager-disposal
observable; cached store references1→1 are not file-handle counts. Unsupported
metrics were not fabricated as zeros. This is not a passing four-cycle sequence,
a matched process-memory host, an accepted RSS denominator or a long-run result.
Astra independently verified208 reference hashes, all short-run projections and
the exact immutable close/dispose behavior. The diagnostic package is reviewable;
shared semantic/token/weak-root/resource-owner checks remain prerequisites to a
new acceptance procedure.

Astra's final integrated verdict is **source/security review closed, release HOLD**.
It independently verified all63 final CI4 TRX hashes/counters and exact baseline
failure attribution, the two-hour client receipts and428-file recursive manifest,
the candidate saturation result and original failure preservation, and the
original adapter's explicit limitations. No new must-fix migration source/security
finding remains. **CI remains red on11 baseline cases; G3 external evidence and
G4 resource/performance acceptance remain unresolved.** #799, #800, #801 and
ScadaBridge#121 stay open. Neither an SDK version substitution, production crypto/
session-factory workaround, resource-cap waiver nor warmed acceptance baseline
was approved. SDK2 remains on draft PR#802; it is not merged to master or deployed.

## Follow-on #799 close ownership and native attribution

The [candidate bounded close control](evidence/2026-11-10-opcua2/close-ownership-candidate.md)
at `564f1091` passed with one abandoned browse and six abandoned history points.
Old authentication returns exactly BadSessionIdInvalid; old continuation tokens
are rejected without a history-backend read while fresh-session positives work.
All 23 old-object weak targets collect with the same host still running; all 13
application/server/manager/core/store targets collect after normal shutdown.
SDK2's stronger `(0,0)` closed-list check, two manager-disposal checks, empty
caches and zero PKI descriptors remain enforced. Astra verified the actual TRX.
Six affected history tests also pass; their original launcher mistakenly expected
seven, and the retained false guard receipt has a separate independently reviewed
six-test source/TRX correction. No failed test was relabelled or rerun for that fix.

The [fixed macOS store-versus-registry control](../../scripts/validation/opcua-lifecycle-memory/native-retention/README.md)
completed its execution/ownership checks, **not native-retention acceptance**.
Two hundred configured-store acquisitions leave AppleDatabase inventory 3→203;
registry AddRefs remain 3→3. After normal disposal and five seconds idle, both
arms have zero SDK certificate owners, runtime-tracked temporary keychains,
temporary-keychain files, keychain descriptors and PKI descriptors. The installed
runtime calls CFRelease after native deletion; a missing managed release is not
indicated. Native ownership/delete status, complete RSS attribution and a supported
mitigation remain unresolved. Registry substitution would change deliberate
configured-store replacement semantics and was not adopted.

These results leave the original cold RSS failures and #799/G4 unchanged. The
[prospective resolution procedure](../../scripts/validation/opcua-lifecycle-memory/RESOLUTION.md)
is explicitly a design draft, not approval to launch a warmed acceptance run.