test(opcua): package cross-platform lifecycle memory controls
This commit is contained in:
@@ -0,0 +1,286 @@
|
||||
# SDK 2 lifecycle memory investigation
|
||||
|
||||
The required two-hour server lifecycle gate failed on macOS and Linux. The RSS
|
||||
limit remains **cycle-3 baseline + 128 MiB**; retained managed memory remains
|
||||
+64 MiB and descriptors +32. No threshold, production code, certificate provider,
|
||||
or session factory was changed by this investigation. These controls do not
|
||||
replace the full gate. Issue #799 remains open.
|
||||
|
||||
## Preserved full-workload failures
|
||||
|
||||
| Run | Cycles / restarts | Observation time | RSS / limit (bytes) | Outcome |
|
||||
|---|---:|---:|---:|---|
|
||||
| macOS `fa823378` | 250 / 62 | 1045.53 s | 365,215,744 / 363,331,584 | Failed |
|
||||
| macOS `e135b7c9` profile reproduction | 245 / 61 | 1024.14 s | 368,656,384 / 363,053,056 | Failed |
|
||||
| Linux full harness, recorded build `38098ce142ab8c382825aa754e0758071dd3a46b` | 248 / 61 | 903.508 s | 353,931,264 / 351,850,496 | Failed |
|
||||
| macOS `6b2d4b81` bounded profile | 432 / 108 | 1807.479 s | First rejection: 364,199,936 / 363,266,048 | Failed; first assertion rethrown |
|
||||
|
||||
The last macOS row requested 1,800 seconds. It first rejected RSS at cycle 253 /
|
||||
1055.482 seconds, continued recording explicitly rejected observations, then
|
||||
threw the original assertion before any completion record. Its final observation
|
||||
was RSS 380,059,648, live managed 41,364,096, GC committed 37,060,608, JIT
|
||||
compilations 81,535, descriptors 200, and zero sessions, subscriptions, monitored
|
||||
items, browse/history continuation points, and PKI descriptors. SDK cached stores
|
||||
remained two during operation. The runner and skip guard returned exit 1 with one
|
||||
executed test and zero skipped tests. There is no passing two-hour result here.
|
||||
|
||||
The first macOS run's reported PKI descriptor zero is invalid: `/var` and
|
||||
`/private/var` aliases were not canonicalized. Corrected positive-handle and timeout
|
||||
controls passed before subsequent profiles. Global descriptor counts and the RSS
|
||||
failure itself were independent of that defect. A Shouldly source-expression
|
||||
message incorrectly names `Descriptors` in the RSS failure; the actual numeric
|
||||
comparison is RSS, as retained before the assertion.
|
||||
|
||||
## What the controls demonstrate
|
||||
|
||||
`LifecycleControl.cs` uses unchanged published product DLLs and public APIs for
|
||||
both SDK generations. It opens two real encrypted **opc.tcp** sessions with
|
||||
Basic256Sha256 / SignAndEncrypt, resolves namespace URIs, reads/writes Int32 and
|
||||
Int16[] values, creates/deletes two data monitored items, abandons a browse
|
||||
continuation, closes sessions, and restarts a server with an active subscription
|
||||
every four cycles against the same task-owned PKI. Public session/subscription
|
||||
counts must return to zero. It has no historian, revocation, surgical-rebuild,
|
||||
paused-network, or private continuation-store coverage. It is a diagnostic subset,
|
||||
not TLS and not the full soak workload.
|
||||
|
||||
The original DLLs are the immutable 1.5.378.182 baseline publication associated
|
||||
with `83ed880e`; its assembly version alone is 1.5.378.0 and does not identify the
|
||||
patch. The candidate uses SDK 2.0.1 and the server implementation from `fa823378`.
|
||||
Neither baseline nor candidate product code was rebuilt for these controls.
|
||||
The controls were compiled once for each reference set, then those exact control
|
||||
artifacts were used on both operating systems. Selected DLL and input-artifact
|
||||
hashes are in `evidence.json`. Neither reference set's principal SDK/server DLLs
|
||||
has a managed native/ReadyToRun header.
|
||||
|
||||
| Ten-minute matched subset | Cycles / restarts | Final RSS | Change from cycle 3 | Final managed / change | Final JIT compilations |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| macOS original | 212 / 53 | 330,301,440 | +121,683,968 | 271,168,424 / +169,990,456 | 28,066 |
|
||||
| macOS SDK2 | 200 / 50 | 302,170,112 | +95,862,784 | 44,569,584 / +295,120 | 62,608 |
|
||||
| Linux original | 249 / 62 | 384,950,272 | +176,836,608 | 147,139,568 / +99,356,752 | 28,321 |
|
||||
| Linux SDK2 | 259 / 64 | 390,844,416 | +194,183,168 | 19,561,120 / +736 | 71,529 |
|
||||
|
||||
Both ten-minute controls exited zero because their lifecycle checks passed;
|
||||
`fullGate=false` is explicit. They do **not** enforce the full harness memory
|
||||
assertions. In particular, both Linux subset RSS deltas exceed 128 MiB and the
|
||||
original retains managed memory/timers. These observations do not waive either
|
||||
failed full gate. They are not proof of an SDK1 defect or an SDK2 improvement.
|
||||
Both control branches explicitly delete subscriptions, close normal sessions,
|
||||
and dispose sessions/subscriptions. Parked resources intentionally encounter
|
||||
server shutdown and are then disposed. Each version calls its unchanged product
|
||||
host's DisposeAsync. The baseline host only calls ApplicationInstance.StopAsync;
|
||||
that original ApplicationInstance exposes no disposal interface. The SDK2 branch
|
||||
also uses its new ApplicationInstance.DisposeAsync ownership API. Per-connection
|
||||
DefaultTelemetry factories are left to the same ownership pattern on both
|
||||
branches, separately probed above. Thus measured original growth belongs to this
|
||||
published-product/fixture lifecycle combination; its allocation owner has not
|
||||
been isolated. These are one matched run per platform, concurrent processes,
|
||||
with different runtime versions; they are not performance distributions.
|
||||
|
||||
The thirty-minute Linux comparison uses the same immutable artifacts and image,
|
||||
fresh independent fixtures, and SDK2 certificate allocation tracking enabled
|
||||
before the first certificate (the same environment is passed to both versions).
|
||||
Both finished with exit zero and `fullGate=false`:
|
||||
|
||||
| Thirty-minute Linux subset | Cycles / restarts | Final RSS | Final live managed | GC committed | JIT compilations | Active timers |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| Original | 736 / 184 | 540,057,600 | 342,626,200 | 408,399,872 | 29,177 | 926 |
|
||||
| SDK2 | 768 / 192 | 334,180,352 | 19,248,984 | 42,983,424 | 72,087 | 19 |
|
||||
|
||||
Observed runtime was 1802.542 seconds original / 1804.523 seconds SDK2. The SDK2
|
||||
RSS fell from 390,782,976 at the 632.7-second checkpoint to 334,180,352 at the last
|
||||
observation; live managed stayed near 20 MB. After disposal and final collection,
|
||||
SDK2 wrapper counters were 19,272 created = 19,272 disposed, with zero live or
|
||||
unreachable-undisposed wrappers under enabled tracking. These finite results
|
||||
support post-transition saturation of this subset, not a two-hour full-workload
|
||||
pass or a guarantee of indefinite stability.
|
||||
|
||||
### Native and runtime observations
|
||||
|
||||
- During the macOS bounded profile, `Security::AppleDatabase` counts increased
|
||||
from 278 at cycle 52 / 211 seconds to 1,953 at cycle 370 / 1546 seconds. These
|
||||
native objects continued accumulating while managed memory stayed near 40 MiB.
|
||||
- The single-owner certificate probe makes a self-signed RSA fixture in memory,
|
||||
then disposes each of 1,000 copies/imports. On macOS, public-only and
|
||||
shared-handle copies leave AppleDatabase at 2; PFX imports and SDK
|
||||
`Certificate.AsX509Certificate2` private-key copies grow it from 2 to 1,002.
|
||||
A separate PFX process retained that count through a 180-second idle hold.
|
||||
`Certificate.From(original)` owns the original; the checked-in control has no
|
||||
second owner. All four corrected single-owner modes were rerun successfully.
|
||||
- The same isolated Linux probe ended at approximately 49 MB RSS for public or
|
||||
shared copies, 61 MB for PFX imports, and 69 MB for the SDK copy. Those are
|
||||
endpoint RSS observations including different JIT/GC costs, not evidence of a
|
||||
Linux native-object leak or a complete attribution.
|
||||
- SDK2 public certificate counters after the Linux ten-minute lifecycle run were
|
||||
6,495 created = 6,495 disposed. A separate stack-tracked macOS lifecycle run
|
||||
ended 591 = 591, zero live and zero unreachable-undisposed wrappers. Allocation
|
||||
tracking is opt-in; empty internal reports without that opt-in are not proof.
|
||||
- The Linux ten-minute native snapshots show SDK2 JIT `doublemapper` resident
|
||||
mappings increasing from about 65.4 to 106.7 MiB, anonymous writable mappings
|
||||
from 84.6 to 121.3 MiB, and `[heap]` around 19.2 MiB. Original JIT mappings were
|
||||
about 33.3 to 37.7 MiB. Mapping totals are descriptive categories, not exclusive
|
||||
object allocations: anonymous memory also contains GC regions, stacks, and
|
||||
native allocator arenas. RSS includes file/shared pages and can fall with
|
||||
reclamation. JIT counters are cumulative compilations, not unique live methods.
|
||||
- 10,000 disposed versus undisposed `DefaultTelemetry` factories ended at similar
|
||||
RSS (~63.4 MB) and live managed memory (~0.43 MB), with no timer growth. This
|
||||
does not justify changing host telemetry ownership to fix this failure.
|
||||
|
||||
Exact SDK tag source shows private-key PFX copying in nonvirtual
|
||||
`Certificate.AsX509Certificate2`. A 60-second macOS sampled-thread-time trace
|
||||
instead found the dominant sampled import path through
|
||||
`CertificateIdentifierResolver.LoadPrivateKeyCoreAsync` →
|
||||
`DirectoryCertificateStore.LoadPrivateKeyAsync` → `X509CertificateLoader.LoadPkcs12`.
|
||||
Inclusive sampled thread-time was about 4.79 seconds for store load and 2.99 seconds
|
||||
for PKCS12, versus 8.5 milliseconds for AsX509. These are neither exclusive CPU
|
||||
costs nor call counts. ApplicationInstance already normalizes its validation
|
||||
input to public DER. The validation copy delegate is internal, the manager sealed,
|
||||
and no safe narrow application copy-policy seam was identified. Changing session
|
||||
factory, certificate exportability, or key storage would alter behavior and was
|
||||
not used as a workaround.
|
||||
|
||||
A major JIT/RSS step appeared near 60–64 restarts in several workloads. This is a
|
||||
hypothesis about runtime/code-cache warmup, not a finding that all RSS growth is
|
||||
benign. A separate read-only EventPipe attachment observed the thirty-minute
|
||||
Linux candidate around that transition using `Microsoft-Windows-DotNETRuntime`,
|
||||
Verbose, keywords `0x18` (Loader + JIT), no rundown, 180 seconds. The transition was UTC 18:51:02.049–18:54:02.062. Factory JIT events
|
||||
occurred UTC 18:52:48.249–18:53:21.434: 6,690 distinct method tokens from one
|
||||
module, each once. The module is Opc.Ua.Core.Types.dll; metadata declares 8,442
|
||||
static methods on OpcUaExtensions. The observed set is a subset, so this does
|
||||
not prove every possible factory or the complete canonical workload is warmed.
|
||||
Method-load events totalled 51,327,536 native code bytes, including 51,161,724
|
||||
OptimizedTier1 bytes and 49,141,484 bytes in generated factories. Only 50
|
||||
reflection InvokeStub JIT starts occurred; the transition trace reports zero lost events. MethodSize sums count loaded code
|
||||
versions, not resident/PSS allocation bytes. An additional 180-second attachment
|
||||
after the transition checks continuing factory compilation. The second attachment (UTC 19:01:44.116–19:04:44.127) recorded 85 JIT starts,
|
||||
zero lost events, zero Opc.Ua starts, zero generated-factory starts, and zero generated-factory
|
||||
loaded bytes. Only 18 ordinary Tier1 method loads (5,720 bytes) and 20
|
||||
instrumented Tier1 loads (5,800 bytes) occurred. This supports a finite generated
|
||||
factory burst in the observed subset; it does not prove complete canonical
|
||||
workload saturation. The capture API's default circular-buffer capacity was 256 MiB; that is a
|
||||
capacity setting, not measured resident overhead. Attachments can allocate event
|
||||
buffers and perturb execution. Both prior full failures and the ten-minute
|
||||
control were uninstrumented by these JIT attachments, and the thirty-minute final
|
||||
interval is unattached. No captured native-code byte total is claimed to equal an
|
||||
RSS delta. Raw traces/heaps are not committed.
|
||||
|
||||
## Reproduce the diagnostic subset
|
||||
|
||||
Requires Python 3 and .NET 10. `run.py` references existing DLL directories,
|
||||
compiles only the control, records SHA-256 provenance, and creates a fresh local
|
||||
artifact directory. First reference directory wins duplicate filenames. Use the
|
||||
actual baseline publications, rather than restoring a different package version.
|
||||
No production project or package file is changed.
|
||||
|
||||
```bash
|
||||
python3 scripts/validation/opcua-lifecycle-memory/run.py lifecycle \
|
||||
--references "$baselineHostDlls" --references "$baselineClientDlls" \
|
||||
--output /tmp/opcua-memory-original --seconds 600
|
||||
python3 scripts/validation/opcua-lifecycle-memory/run.py lifecycle --sdk2 \
|
||||
--references "$candidateDlls" --output /tmp/opcua-memory-sdk2 --seconds 600 \
|
||||
--certificate-tracking
|
||||
```
|
||||
|
||||
Fixture credentials are generated in the runner environment and not logged.
|
||||
PKI material stays under each local `private-fixture` directory. Do not publish
|
||||
those directories or unredacted process/native logs. These controls grant a
|
||||
fixture user WriteOperate and auto-accept task-owned certificates; they are
|
||||
local test fixtures, not deployment configurations.
|
||||
|
||||
For a matched Linux pair, first build both with `--build-only` into fresh
|
||||
`/tmp/opcua-memory-original` and `/tmp/opcua-memory-sdk2` output directories, then:
|
||||
|
||||
```bash
|
||||
python3 scripts/validation/opcua-lifecycle-memory/linux_pair.py \
|
||||
--baseline /tmp/opcua-memory-original/bin/Release/net10.0 \
|
||||
--candidate /tmp/opcua-memory-sdk2/bin/Release/net10.0 \
|
||||
--output /tmp/opcua-memory-linux-pair --seconds 1800 \
|
||||
--image sha256:b04611ee9e1b7d36797012ed921962c57189cb4dbb3dff12afb8e158110f293b
|
||||
```
|
||||
|
||||
The runner uses separate task-owned `--network none` containers, fresh PKI,
|
||||
read-only executable mounts, bounded shutdown, and periodic read-only `/proc`
|
||||
snapshots. It records child and container exit statuses independently. The image
|
||||
is Ubuntu 24.04 arm64, .NET SDK 10.0.103 / runtime 10.0.3. The Mac runs used
|
||||
macOS 26.5.2 arm64, SDK 10.0.403 / runtime 10.0.14. Runtime and OS differences are
|
||||
explicit; do not attribute all cross-platform differences to SDK code.
|
||||
|
||||
On macOS, run each certificate mode with a fresh output directory:
|
||||
|
||||
```bash
|
||||
python3 scripts/validation/opcua-lifecycle-memory/run.py certificate --sdk2 \
|
||||
--references "$candidateDlls" --output /tmp/opcua-memory-pfx --mode pfx
|
||||
```
|
||||
|
||||
Other modes are `sdk-copy`, `public`, and `shared-copy`. The last intentionally
|
||||
uses the old copy constructor by reflection as a diagnostic control. It is not
|
||||
proposed production code. The control exports no fixture key/certificate data.
|
||||
`TelemetryControl.cs` is available through `run.py telemetry --sdk2` with modes
|
||||
`dispose` or `undisposed`. `summarize.py` emits Linux mapping categories without
|
||||
file paths from local `*-smaps.txt` files. All controls are opt-in and outside the
|
||||
ordinary solution/test graph.
|
||||
|
||||
Optional trace reproduction uses the installed dotnet-trace 9.0.661903 managed
|
||||
libraries, without adding a product dependency:
|
||||
|
||||
```bash
|
||||
python3 scripts/validation/opcua-lifecycle-memory/build_trace_tools.py \
|
||||
--libraries "$diagnosticToolLibraries" --output /tmp/opcua-memory-trace-tools
|
||||
# Run capture in the same process namespace / user as the task-owned target.
|
||||
dotnet /tmp/opcua-memory-trace-tools/capture/bin/Release/net10.0/capture.dll \
|
||||
"$targetPid" 180 /tmp/opcua-memory-local.nettrace
|
||||
dotnet /tmp/opcua-memory-trace-tools/summary/bin/Release/net10.0/summary.dll \
|
||||
/tmp/opcua-memory-local.nettrace > /tmp/opcua-memory-method-summary.json
|
||||
```
|
||||
|
||||
Capture is read-only and opt-in; raw traces remain local. The summary exports
|
||||
static method names and aggregate counts rather than certificate identities,
|
||||
arguments, or process mappings. The original transition parser also retained
|
||||
method tokens/modules locally to cross-check name-key uniqueness.
|
||||
|
||||
## Prospective saturation experiment (not executed)
|
||||
|
||||
The observed finite factory burst supports a separately labeled readiness
|
||||
experiment; it does not authorize moving the acceptance baseline. A reviewed
|
||||
candidate proposal is exactly 384 canonical lifecycle cycles / 96 every-four-cycle
|
||||
restarts, then one fixed settling cycle, all within 1,800 seconds. No early finish,
|
||||
extension, or RSS-selected endpoint is permitted. Both fixed readiness windows
|
||||
321–352 and 353–384 must contain zero generated-factory JIT/method-load events, zero SDK assembly
|
||||
loads, and zero lost events. Other JIT/native-code totals remain review evidence,
|
||||
not a claim that every runtime path is warmed. Trace capture is bounded and identical for both versions, then
|
||||
detached/drained before the settling cycle. Ownership, FD, PKI, store, and logical
|
||||
resource checks remain active throughout; original cold budget rejections remain
|
||||
recorded. Any later full gate requires at least 7,200 additional seconds after a
|
||||
prospectively approved baseline. No such warmed full gate has been run here.
|
||||
|
||||
The count is beyond the observed 60–64-restart transition and projects to about
|
||||
1,606 seconds from the Mac canonical run, but completion within the fixed cutoff
|
||||
is not guaranteed under load. Failure to complete or satisfy readiness is a failed
|
||||
experiment, not permission to extend. The full canonical test currently depends
|
||||
on SDK2 value APIs and read-only private SDK2 continuation/store instrumentation.
|
||||
An original-SDK adapter proving identical history, revocation, surgical-edit,
|
||||
and stalled-network operation counts is a prerequisite for a matched canonical
|
||||
experiment; the public subset does not provide that proof. Continued Mac native
|
||||
inventory growth remains an independent unresolved gate, even if JIT saturates.
|
||||
|
||||
## Evidence and limits
|
||||
|
||||
`evidence.json` contains sanitized numeric endpoints, native inventory counts,
|
||||
and hashes of retained diagnostic inputs. Full artifacts remain in the originating
|
||||
worktrees or `/tmp/opcua2-rss-investigation`; no PKI, certificate identifiers,
|
||||
credentials, raw traces, unredacted heap/process dumps, or complete test logs are
|
||||
checked in. Existing failures were preserved untouched.
|
||||
|
||||
Supported progress is a reproduced RSS failure, bounded logical/managed
|
||||
inventories in SDK2, demonstrated macOS PFX native retention, and measured
|
||||
cross-platform JIT/anonymous-memory growth. A complete allocation attribution and
|
||||
safe production correction remain unresolved. The server two-hour gate remains
|
||||
failed; a separate long-running client-only process is a different scope.
|
||||
|
||||
Package verification: baseline and candidate control builds, both optional trace
|
||||
tools, and certificate/telemetry controls built with zero warnings/errors. Local
|
||||
control smoke runs and the fresh Linux pair smoke exited zero. Python compilation
|
||||
and whitespace checks passed. One reused-output smoke failed because a manual
|
||||
incremental build used a different `/tmp` path alias and its runtimeconfig was
|
||||
missing; fresh canonical outputs passed, and the Linux runner now rejects an
|
||||
incomplete output directory before launch. These smoke checks validate tooling
|
||||
only and do not substitute for either long run.
|
||||
Reference in New Issue
Block a user