lmxopcua/scripts/validation/opcua-lifecycle-memory/native-retention/README.md

# macOS private-key reload native retention — #799

The bounded control identifies **configured-directory private-key loading** as a
trigger for continuing `Security::AppleDatabase` heap inventory on this machine,
while observed SDK certificate-core ownership and tracked .NET temporary-keychain
ownership return to zero. No narrow built-in application ownership correction was
found that preserves the current raw-session factory, Directory PKI, certificate
selection/update behavior and certificate properties. **#799 remains open.** This
is private reproduction/support material, not an approved production mitigation.

Only this new subfolder was changed. No product source, trust policy, SDK version,
certificate factory/crypto provider, exportability, key storage format, existing
PKI or resource budget was changed. No public upstream issue/message was posted.

## Reviewed, executed control

Astra reviewed the actual source and installed-runtime decompile before execution,
approved one fixed **200-acquisition** run per arm as attribution-only evidence,
and afterward independently matched all ten raw heap counts/bytes, checkpoint and
completion counts, source hashes, installed crypto hash and eight SDK DLL hashes.
The configured-store versus registry distinction below is supported by inspected
upstream regression source; that upstream regression test was **not executed**.

Both arms ran in separate fresh processes on macOS **26.5 arm64**, .NET SDK
**10.0.403**, runtime **10.0.14**. A normal SDK ApplicationInstance created one RSA
application PKI; its persisted tree was copied into separate private arm roots.
Both used the same application URI, public DER identity and private key. Normal
SDK configuration validation/certificate checks and teardown were identical. The
loop called these public APIs, disposing each returned owner before checkpoints:

| Arm | Public acquisition, repeated exactly 200 times |
|---|---|
| Store | `SecurityConfiguration.FindApplicationCertificateAsync(Basic256Sha256, privateKey: true, ...)` |
| Registry | `ICertificateRegistry.AcquireApplicationCertificateBySecurityPolicy(Basic256Sha256)` |

Each arm sampled ready, first acquisition, acquisition 200, application disposed,
and after a fixed five-second idle. Every checkpoint used the same full-GC /
finalizer / full-GC / 250 ms schedule, then a parent-controlled heap/lsof barrier.
Matched initial/first/last RSA sign-and-verify checks confirmed private-key use.
No iteration called `AsX509Certificate2`; no network, server or session was opened.
`AutoAcceptUntrustedCertificates=false` stayed configured; this key-acquisition
control makes no additional peer-validation wire claim.

Allocation-stack tracking was enabled in **both** arms, so its allocation/GC cost
is part of these observations. The 180-second child cancellation/checkpoint
limit is not a strict total wall deadline: a running heap probe (20 seconds),
lsof probe (five seconds), final wait and bounded terminate/kill reap can add
bounded overhead. Stdout is a file, with no unbounded `communicate` or pipe drain.
The recorded artifact-creation-to-result-write wall interval was **34.699 seconds**
(filesystem timestamps, not a monotonic acceptance gate). Ready-to-idle checkpoint
intervals were **21.504 seconds store** and **9.123 seconds registry**, including
probe barriers. Both process exits were zero and all children were reaped.

## Native inventory and ownership results

| Checkpoint | Store AppleDatabase | Registry AppleDatabase | Tracked runtime keychains, each arm | Outstanding SDK cores, each arm | Temporary keychain files, each arm |
|---|---:|---:|---:|---:|---:|
| Ready | 3 | 3 | 1 | 1 | 1 |
| First acquisition | 4 | 3 | 1 | 1 | 1 |
| Acquisition 200 | 203 | 3 | 1 | 1 | 1 |
| Application disposed | 203 | 3 | 0 | 0 | 0 |
| Five-second idle | 203 | 3 | 0 | 0 | 0 |

Configured PKI FDs and temporary-keychain FDs were **zero at every checkpoint**.
One other task-TMPDIR descriptor remained in each process; it is reported
separately and is not labelled a keychain descriptor. Total numeric FDs were
110 ready / 112 final in both arms. Manager disposal was observed through an
exact-SDK, read-only `m_disposed` boolean probe. Store certificate-core counters
finished **407 created / 407 disposed**; registry finished **7 / 7**. These public
counters concern underlying certificate cores, not every AddRef wrapper.

The exact installed runtime's internal temporary-keychain dictionary was read
under the same dictionary monitor, with a one-second bounded acquisition and
exact informational-version/type/field guards. Only its count was read; no native
pointer, handle, key or certificate material was enumerated or modified. Its
count returning zero establishes no remaining entries in that managed lookup.
The installed cleanup code removes the entry **before** calling native
`SecKeychainDelete`, suppresses delete errors, and then releases the CF handle.
The actual installed base `SafeKeychainHandle.ReleaseHandle` was separately
decompiled and does call `Interop.CoreFoundation.CFRelease(handle)`, then zeros
its handle. Apple's [SecKeychainDelete documentation](https://developer.apple.com/documentation/security/seckeychaindelete(_:).md)
distinguishes file/keychain deletion from releasing keychain-object memory; this
inspection finds the separate CFRelease in the installed runtime, not an obvious
missing base-release implementation. These are inspected calls, not native
return-status instrumentation. Therefore lookup zero does **not** prove that
native deletion succeeded.

| Memory observation | Store ready → final idle | Registry ready → final idle |
|---|---:|---:|
| Managed live bytes | 831,080 → 1,222,192 | 831,128 → 1,130,616 |
| RSS bytes | 87,621,632 → 115,130,368 | 87,654,400 → 97,927,168 |
| Compiled methods | 1,192 → 4,638 | 1,192 → 2,251 |

RSS includes different JIT, tracking, GC and allocator effects; this comparison
is not an exclusive native-byte attribution or a performance budget. The native
class's shallow heap bytes were 2,304 → 155,904 in store and 2,304 → 2,304 in
registry (768 bytes per classified object). Shallow class bytes exclude owned
allocations and do not explain the entire RSS difference. `PrivateMemorySize64`
returned zero on this runtime and is not used as a private-memory result.

`heap --noContent -s` reports allocations classified as the C++
`Security::AppleDatabase` class. Apple's published file-database source declares
that class as a `Database` subclass with database open/close/delete methods and
owned file/modifier state; it is not a file-descriptor counter. The published
source is conceptual evidence, **not** a claim that it is the exact source build
of this installed Security.framework. Retaining classified heap objects after
observed closure is different from proving an open handle, failed native delete,
or the identity of the retaining native owner. See the pinned
[Apple class definition](https://github.com/apple-oss-distributions/Security/blob/db15acbe6a7f257a859ad9a3bb86097bfe0679d9/OSX/libsecurity_filedb/lib/AppleDatabase.h#L485).

The leaf DER SHA-256 was identical before/after and across arms:
`fadcc11ec694732b82fe4dc47bb41bce6023d113339e53aca9fee5b079f200b2`.
After samples and process exits, every seed/arm PKI, TMPDIR and data fixture
was removed; cleanup receipt is true. Raw native snapshots stay local and private.

## Exact SDK path and public API limits

The inspected SDK checkout is the clean **2.0.1** tag at commit
`07e62f72500cc0923a2bab940653c8c1116fa286`; the control references immutable actual
published SDK 2.0.1 DLLs, with all eight hashes in `evidence/2026-11-10/provenance.json`.

| Source, pinned to the exact SDK commit | Relevant behavior |
|---|---|
| [DefaultSessionFactory.CreateChannelAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Client/Session/DefaultSessionFactory.cs#L406) | Loads a private certificate through the internal overload that defaults registry use to false. |
| [DefaultSessionFactory private CreateAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Client/Session/DefaultSessionFactory.cs#L579) | Constructs the raw Session with null certificate/chain. |
| [Session.LoadInstanceCertificateAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Client/Session/Session.cs#L6335) | Opening that Session independently loads its private certificate; registry preference depends on a channel manager, absent in the current raw factory. |
| [Session.LoadInstanceCertificateEntryAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Client/Session/Session.cs#L6432) | Documents configured-store selection for unmanaged callers and registry preference for managed callers. |
| [SecurityConfiguration.FindApplicationCertificateAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Core/Security/Certificates/SecurityConfiguration.cs#L182) | Private-key branch calls registry-less CertificateIdentifierResolver.LoadPrivateKeyAsync. |
| [CertificateIdentifierResolver.LoadPrivateKeyCoreAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Core/Security/Certificates/CertificateIdentifierResolver.cs#L249) | Opens the configured store and asks it to load the private key; returned reference is caller-owned. |
| [DirectoryCertificateStore.LoadPrivateKeyAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Core/Security/Certificates/DirectoryCertificateStore.cs#L758) | For PFX, macOS storage flags omit EphemeralKeySet and include Exportable, then import through X509CertificateLoader.LoadPkcs12. No public per-store private-key cache/flag setting is offered. |
| [ApplicationConfiguration.ValidateAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Core/Stack/Configuration/ApplicationConfiguration.cs#L735), [ApplicationInstance.CheckOrCreateCertificateAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Configuration/ApplicationInstance.cs#L425), [ServerBase.OnServerStartingAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Core/Stack/Server/ServerBase.cs#L1887) | Application validation, normal own-certificate checking and server bring-up also load configured private certificates. Checking disposes its temporary certificate; registry replacement disposes old entries. |
| [Certificate.AsX509Certificate2](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/src/Opc.Ua.Security.Certificates/X509Certificate/Certificate.cs#L258) | Nonvirtual export/reimport copy for exportable private keys. Existing isolated controls demonstrate its PFX retention trigger, but prior sampled real-workload stacks identified directory loading as the dominant sampled import path. |

`CertificateIdentifier` has **no Certificate preload/cache property** and only a
public default constructor. Actual-binary public API inventory is retained in
`seed-api-inventory.jsonl`. `RawData` carries public encoded certificate material;
its setter parses `Certificate.FromRawData` and derives metadata. It cannot carry
the reusable private Certificate owner needed by the raw private-key load path.
The [exact-tag manager documentation](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/docs/CertificateManager.md#materializing-a-certificate-from-a-certificateidentifier)
explicitly places materialized ownership in the manager registry.

The public registry acquisition API is reference-counted and supported. However,
substituting it in the raw session path changes certificate selection semantics:
[UnmanagedCertificateLoadHonorsAnExplicitConfiguredStoreReplacementAsync](https://github.com/OPCFoundation/UA-.NETStandard/blob/07e62f72500cc0923a2bab940653c8c1116fa286/tests/Opc.Ua.Client.Tests/Session/SessionCertificateRotationRegressionTests.cs#L198)
requires a configured replacement to win over an older registry certificate and
verifies the registry API is never called. This is **inspected upstream test
source, not a test run here**. A cache/factory rewrite must additionally preserve
update/removal visibility, cancellation/fault cleanup, concurrent owners,
reconnect and certificate/issuer-chain selection; the B arm proves none of those
replacement semantics. ChannelManagerSessionFactory changes architecture and was
not used. Overriding/reimplementing channel and session construction is not a
narrow application disposal fix.

Production source at `ed60970fc294ab171a80cda1b57942a7130b4c39` was inspected:
[OpcUaApplicationHost](../../../../src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs)
assigns configuration ownership before async validation and disposes its
ApplicationInstance; Shared transfers manager lifetime to the session adapter
and disposes failed configurations; driver/browser release their owned
ApplicationInstance after session cleanup. No missing owner was identified in
those paths. The store arm already keeps its one application/configuration and
registry alive across all 200 acquisitions, so simply lengthening that
application owner cannot remove these private-key imports. Existing full-client
lifecycle disposal checks are complementary
coordinator evidence, not additional executions in this control.

## Runtime dependency and upstream-ready boundary

Actual installed `System.Security.Cryptography.dll` informational version:
`10.0.14+0427b9abbead3fa64c6655eb4290c537397131fa`.
SHA-256: `f4957d7218a004a27bf8964e45147df6a88092c6e7344582082840d62c6c579a`.

The official .NET `v10.0.14` tag resolves to
`6aa72daa24f66c01c3d7b6a32519e9d21396deec`, different from the installed binary's
informational commit. It is not presented as exact installed source. Read-only
ILSpy inspection of the **actual binary** verified the temporary-keychain
creation, dictionary/monitor and release behavior used by diagnostics. Paths and
hashes of those retained local decompilations are in `verification.json`.

The public runtime source provides matching context for
[temporary keychain creation during non-persisted import and unsupported EphemeralKeySet](https://github.com/dotnet/runtime/blob/6aa72daa24f66c01c3d7b6a32519e9d21396deec/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509CertificateLoader.macOS.cs)
and [tracked-keychain release ordering](https://github.com/dotnet/runtime/blob/6aa72daa24f66c01c3d7b6a32519e9d21396deec/src/libraries/Common/src/Interop/OSX/System.Security.Cryptography.Native.Apple/Interop.Keychain.macOS.cs).
Microsoft documents that macOS PFX private-key loading requires a keychain and
cannot use EphemeralKeySet in its [cross-platform cryptography guide](https://learn.microsoft.com/en-us/dotnet/standard/security/cross-platform-cryptography#read-a-pkcs12pfx-file).
Changing those flags, exportability, storage format or native crypto was not an
application mitigation attempted here.

The remaining dependency is an upstream/platform investigation of why the
classified native file-database objects remain after observed SDK owner release,
tracked runtime owner release and backing-file cleanup, and whether there is a
supported correction at that boundary. Native delete status and the retaining
C++ owner are still unknown. This control does not identify a specific .NET or
Apple defect/fix. An upstream SDK improvement would also need to avoid redundant
imports **without losing the existing raw configured-store semantics**. No
supported application-only correction meeting the stated constraints was proven.

## Private reproduction and evidence

```sh
python3 scripts/validation/opcua-lifecycle-memory/native-retention/run.py \
  --references /tmp/opcua2-publish-hg100-20261110/linux-arm64/host \
  --output /tmp/opcua2-native-reload-FRESH
```

This builds only the standalone control against existing actual SDK DLLs, never
rebuilds product code, requires the exact installed runtime diagnostic layout,
and refuses an existing output directory. The fixed repetition count is not
configurable. Use a private fresh root and serialize with other ownership/native
controls. The checked-in selected receipts are in `evidence/2026-11-10/`;
raw artifacts are `/tmp/opcua2-native-reload-20261110a`. Raw heap/lsof output and
any failed-run private fixture material must not be redistributed wholesale.

No long run, deployment, trust weakening, SDK fork/version change, budget change,
public upstream post or #799 closure is authorized or claimed by these results.

A separately reviewed [plain native Apple empty-keychain control](PURE_APPLE.md)
then executed 200 successful create/delete/CFRelease operations and kept native
AppleDatabase at 1 throughout. That negative result rules out empty native
create/delete/release alone as sufficient here; it does not identify the
private-key/certificate import retaining owner.

The subsequent [installed native bridge import control](NATIVE_IMPORT.md)
reproduced AppleDatabase growth from 1 to 201 in a measured C process with no
managed caller. Native deletes returned zero in that control, all returned CF
owners balanced, and keychain files/FDs stayed zero. The installed bridge and
Security.framework remain in the path; their respective retaining ownership
is unresolved. The native delete statuses of this managed A/B remain unobserved.