lmxopcua/scripts/validation/opcua-lifecycle-memory/SATURATION.md

Candidate saturation experiment — review before launch

This is a prospective diagnostic protocol, not an acceptance run or a replacement for the failed cold server soak. Production code, the cycle-3 resource baseline, and the +64 MiB managed / +128 MiB RSS / +32 descriptor caps remain unchanged. Only OTOPCUA_SDK2_SOAK_MODE=saturation selects this behavior; the existing diagnostic, profile, and full modes keep their existing stopping rules.

The fixed canonical phase performs exactly 384 lifecycle cycles, with the existing active-resource restart after every fourth cycle: 96 completed restarts. It then performs exactly one settling cycle, without another restart. The phase, including observer handshakes, must finish within 1,800 seconds. Initial fixture startup has a separate 120-second bound. There is no early completion, extension, RSS-selected endpoint, budget reset, or two-hour/full-pass flag. Cleanup remains bounded independently of the phase deadline.

All canonical operations remain in use: encrypted opc.tcp sessions, namespace URI binding, scalar/typed-array read/write, subscriptions, browse abandonment, history continuation churn/release/cross-session/close, surgical changes/rebuilds, key revocation, paused network reads/recovery, and same-PKI server restart. All ownership, continuation, PKI-descriptor, store and node inventory checks remain active. Cold budget rejection is latched at its original observation and rethrown, even if diagnostic readiness later succeeds. A rejected readiness receipt does not become a passing test or a new resource baseline.

Observer boundaries

A fresh task-owned control directory and UUID bind every request/receipt to this run and target PID. Files are published atomically; existing, malformed, foreign, or incomplete receipts fail closed. The observer is a separate diagnostic process using public DiagnosticsClient / EventPipe APIs. Product dependencies are unchanged. A separate short positive control first invokes a named non-SDK method with NoInlining/NoOptimization after attaching and requires its JIT-start count in the parsed receipt. This proves CLR provider delivery/classification independently of custom marker delivery. That probe is never invoked by the canonical experiment.

One continuous trace covers both fixed windows, including their boundary gap:

Marker	Before/after operation	Completed restarts
window1-start	Before cycle 321	80
window1-end	After cycle 352 and its restart	88
window2-start	Before cycle 353	88
window2-end	After cycle 384 and its restart	96

The target waits for the observer's active receipt and enabled marker provider before its first window operation. The parser requires exactly these four markers, the correct UUID and cycle/restart values, and strictly ordered timestamps. A continuous veto also covers events outside the marker ranges within the attached interval. Both windows and the entire attached interval require zero Opc.Ua.OpcUaExtensions JIT starts and method-load events, zero new Opc.Ua.* assembly-load events, and zero lost events. Other JIT counts and loaded native method-byte sums are reported for review; they do not establish that all runtime paths are warmed. Method-size sums do not equal resident memory.

Runtime capture is Verbose Microsoft-Windows-DotNETRuntime, keywords 0x18 (Loader + JIT), plus Informational OtOpcUa-Sdk2-Saturation markers, no rundown, and a prospectively fixed 16 MiB circular-buffer capacity. This is a new experiment and does not reinterpret the earlier 256 MiB-capacity traces.

After restart 96, the target requests stop and waits. The observer must stop its session, drain the event stream, close it, parse the entire trace, and atomically publish a PID/run-bound drained receipt. Its Stop/copy/dispose/parse stages have 10/10/5/10-second bounds. Target receipt waits are bounded at 45 seconds and remain inside the 1,800-second phase deadline. Observer faults propagate immediately. The target also verifies its marker provider is disabled before the settling cycle. Missing markers, no events, event loss, or malformed stream cannot prove readiness. A parser failure produces a failed observer receipt, not empty counts.

The saturation-result separates readiness, cold budget rejection, fixed operation completion and verified shutdown ownership. It always records acceptance=false. It is emitted before rethrowing the original cold assertion. The launcher retains nonzero test/container exits and records diagnostic readiness separately.

Prepare immutable Linux artifacts

Commit the reviewed source first. Preparation refuses a dirty checkout and publishes fresh, separate app/observer artifacts for linux-arm64, self-contained .NET 10.0.14, without trimming or single-file publishing. It never starts the experiment. The bundle records source revision, source hashes, exact publish command, fixed protocol, image digest and SHA-256 of every app/observer/tool file. It requires the landed HistorianGateway Client/Contracts 1.0.0 dependency graph and records assembly versions/module MVIDs by reading PE metadata without loading or executing product assemblies.

python3 scripts/validation/opcua-lifecycle-memory/prepare_saturation.py \
  --output /tmp/opcua-saturation-reviewed \
  --libraries "$diagnosticToolLibraries" \
  --image sha256:b04611ee9e1b7d36797012ed921962c57189cb4dbb3dff12afb8e158110f293b

diagnosticToolLibraries is the installed dotnet-trace 9.0.661903 tools/net8.0/any directory. No product package reference is added. Artifact preparation and review are prerequisites; root and Astra must review the actual code, control tests and receipt before launch. The coordinator owns the long execution.

python3 /tmp/opcua-saturation-reviewed/tools/launch_saturation.py \
  --bundle /tmp/opcua-saturation-reviewed --receipt-sha256 "$reviewedReceiptHash" \
  --output /tmp/opcua-saturation-run --verify-only
# After root/Astra review, root runs the same command without --verify-only.

The launcher rechecks every immutable file before and after the run. It creates a fresh offline task-owned container, writable output/PKI directories, and read-only app/observer/tool mounts. Descriptor controls run first in a separate process. TRX guards require two descriptor tests and one canonical test actually executed, with no skipped results. The launcher verifies the actual target and observer mapped their own self-contained libcoreclr.so; the older SDK-image controller runtime is not the target runtime. A 2,040-second outer timeout includes startup, phase and bounded cleanup; it does not extend the 1,800-second phase.

No long saturation run has been launched during implementation. Existing cold failures remain untouched. No original-SDK full-workload adapter, warmed acceptance baseline, additional two-hour server run, or production correction is part of this change. The separately demonstrated Mac native certificate-related inventory growth remains unresolved even if generated-factory compilation saturates.