SDK2 validation: client/server lifecycle soak exceeds RSS guard on macOS and Linux #799
Assignees
dohertj2 (dohertj2)
Clear assignees
dohertj2 (dohertj2)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: dohertj2/lmxopcua#799
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Active finding from #782, not a release pass. Exact SDK2 server source fa823378 completed 250 lifecycle cycles and 62 in-process server restarts before the full-run RSS guard failed at 1,045.53 seconds (~17m27s). RSS 365,215,744 bytes exceeded the initial baseline +128MiB limit (363,331,584). The required two-hour run did not complete.
At the preceding quiescent sample, managed live memory was ~40MB, descriptors ~191, cached SDK certificate stores 2, and sessions/subscriptions/items/browse/history continuation points/live monitored nodes all returned to zero. This is a process-memory finding; the evidence does not yet establish an application-object leak or SDK regression. Original PKI-specific descriptor zeros are not valid because macOS canonicalized /var→/private/var; the measurement was repaired separately with open-file 0→1→0 and timeout controls, plus a successful 90s smoke. Global descriptors and SDK collection/disposal measurements remain meaningful.
Original failed logs/TRX/JSONL are retained under the opcua2-server worktree artifacts/opcua2/fa823378/soak-full-coordinator1. No budget was raised, no failed run was counted as passing, and no production change has been made on an unproven hypothesis.
Current investigation: unchanged-resource-budget 30-minute profile with GC committed/heap/fragmentation, finalizer/pinned counts, JIT methods/IL, threads/timers and native vmmap/heap checkpoints; matched original1.5.378.182 public-lifecycle control is being prepared. DefaultTelemetry undisposed vs disposed isolated control did not show a large memory differential, so that is not accepted as the RSS cause. Determine warmup/allocator retention versus continuing growth, correct proven ownership defects if any, and rerun real >=7200s guard before closing this finding. Keep #782 G4 and release approval open.
A separate root-owned two-hour Linux harness using actual Shared client, driver live reinitialization and AdminUI browser is running with strict mutual trust and actual ownership probes; its scope/result will be recorded independently, not substituted for this failing server-restart workload.
The unchanged Linux ARM64 full server/node-manager workload also failed the original RSS guard: cycle 248, 61 restarts, 903.508 seconds; RSS 353,931,264 bytes >351,850,496 (cycle-3 baseline +128MiB). Self-contained runtime 10.0.14, exact SDK2.0.1, network-none task container. Retained managed memory ~20.5MB, descriptors189, PKI descriptors0, cached stores2, and all logical collections remained bounded/zero. Test exit1 and launcher validation failed; no completion or two-hour pass exists. Evidence:
/tmp/opcua2-server-linux-full-20261110/{result.json,source.json}andtask/{soak.jsonl,results/soak.trx}(selected sanitized receipts will be checked in).This broadens the finding beyond macOS. Mac controls prove retained native Apple Security objects from repeated private-key PFX imports and SDK copies, but do not yet attribute the complete workload RSS growth; both client and server share the measured process. Astra found no safe narrow public copy-policy seam in exact SDK2.0.1 and advises against substituting channel/session factories merely to pass this gate. No production workaround or resource-budget change has been applied. Independent Linux production Shared/driver/browser two-hour run remains active and cannot replace this failed workload. #782 G4 remains open.
SDK2 validation: repeated-server soak exceeds RSS growth guard after 250 cyclesto SDK2 validation: client/server lifecycle soak exceeds RSS guard on macOS and LinuxThe fixed, prospectively Astra-reviewed canonical candidate saturation diagnostic is complete:384 lifecycle cycles/96 restarts plus one settling cycle in1,425.242s, within the1,800s cutoff. Both fixed windows321–352 and353–384 and the entire continuous attached interval had zero generated-factory JIT/load events, zero new SDK assembly loads and zero lost events. Shutdown ownership checks passed. Other JIT287; summed CLR MethodSize60,760bytes, not RSS/PSS.
The original cold failure remains: at cycle248 RSS362,409,984 >358,838,272bytes (cycle3+128MiB). The first assertion was rethrown; test/container/launcher exit1, one executed failed lifecycle test, no ordinary complete/two-hour result. Readiness=true is restricted to the selected generated-factory criterion; it is not G4 acceptance or a replacement baseline.
Exact source2738d7cc, SDK2.0.1/HG1.0.0, target and observer self-contained LinuxARM64 runtime10.0.14. Astra verified all1,275 immutable bundle files,1,064 module identities and6/6 Linux provider/protocol controls before launch. OPCUA_CERTIFICATE_LEAK_TRACKING=1 adds allocation StackTrace/tracker overhead; this is instrumented diagnostic evidence, not an uninstrumented performance denominator.
Versioned scoped result and hashed receipts. Prior cold failures and continued macOS native certificate-object retention are unchanged. No budget, factory/runtime setting, production crypto implementation or acceptance baseline was changed.
The independent actual production client/driver/browser lifecycle completed7,200.984s/5,420cycles/43,360manager disposals with zero weak survivors/PKI FDs; its client-only pass does not replace this failed workload. A diagnostic original-SDK canonical adapter is being prepared to examine matched coverage; original1.5 lacks SDK2 certificate-manager disposal observables, which will be reported explicitly rather than fabricated as zeros. #799 stays OPEN/failed; no release approval.
Follow-on #799 diagnostics are pushed on draft PR #802 at
2cf7c1d2; Astra reviewed the designs/code before execution and verified the raw results and selected evidence.Candidate close ownership: final source564f1091, one browse/six history points, 1 executed/pass/no skips. Exact old-auth rejection BadSessionIdInvalid; old browse/history tokens denied while fresh positives work and invalid history never reaches the backend. All23 old-object weak witnesses collect with the same host live; all13 fixture/application/manager/core/store witnesses collect after normal shutdown. Two managers disposed, caches empty, PKI FDs0. Six affected history tests pass. Their initial launcher incorrectly expected7; original false guard receipt is retained with a separately reviewed4+2 source/TRX correction. Report and hashed receipts.
Native attribution remains unresolved: fixed200 configured-store acquisitions retain AppleDatabase inventory3→203 versus registry AddRefs3→3. After disposal+5s idle, both arms have0 SDK certificate owners, runtime temporary-keychain owners/files, keychain FDs and PKI FDs. Installed runtime does call CFRelease after native deletion. This does not prove missing managed disposal, native delete failure, complete RSS attribution or an application-safe fix. Registry substitution changes deliberate configured-store replacement semantics and was not adopted. Exact source/runtime map and control evidence.
The original-SDK semantic control is being examined separately: closed session/point witnesses collect, but a stopped server graph remains during the bounded observation. A pure Apple create/delete/release control is being prepared to isolate the native boundary. Neither investigation is a usable RSS denominator yet.
#799 stays OPEN; original cold RSS failures remain failed. No resource cap, SDK/factory/crypto architecture or acceptance baseline changed. No new long acceptance run was approved, no release approval and no deployment. Prospective resolution design, explicitly not launch approval.
Ownership repairs and native attribution are pushed on draft PR #802 at
fbf73436. #799 remains open; these results do not pass the server RSS gate.efb9197a, #807 closed, five RED→GREEN cases,1032 owning passes/1 existing skip. Frozen83 publications/oracles unchanged; no accepted RSS denominator.The fixed7200s Linux late-slope procedure is approved for implementation only. It preserves the original cold baseline/caps, first failure and workload, and cannot emit acceptance. Actual code, controls and immutable artifacts require Astra review before launch. No new long acceptance run, baseline reset, SDK/factory substitution, merge or deployment. #782 G4 remains on HOLD.
The single Astra-approved fixed 7,200-second Linux late-slope diagnostic is running (source
b18f9723, SDK2.0.1/HG1.0.0, self-contained Linux ARM64 .NET10.0.14, bundle receiptf446a736…). Live mapped runtime and cycle-3 cold baseline are recorded. The same +64MiB managed/+128MiB RSS/+32 descriptor limits and first-rejection latch remain; all canonical operations and independent shutdown checks stay required. No warmed baseline or acceptance is being adopted.The read-only startup failure is retained. The private
/tmpcorrection preserves read-only root/app/tools; the corrected separate rehearsal passed 2/2 descriptor tests and 1/1 canonical test (16 cycles/4 restarts/60.053s), verified mapped CLR and cleanup. Protocol and linked receipts.The final paired native control retained the same AppleDatabase inventory with both deletion orders. Astra reviewed the private upstream handoff; no supported application-only mitigation was found and nothing was posted upstream. #799/G4 remain open. A completed Linux diagnostic cannot resolve the macOS component or approve release.