lmxopcua/scripts/validation/opcua-lifecycle-memory/SATURATION.md

# Candidate saturation experiment — review before launch

This is a prospective diagnostic protocol, not an acceptance run or a replacement
for the failed cold server soak. Production code, the cycle-3 resource baseline,
and the +64 MiB managed / +128 MiB RSS / +32 descriptor caps remain unchanged.
Only `OTOPCUA_SDK2_SOAK_MODE=saturation` selects this behavior; the existing
`diagnostic`, `profile`, and `full` modes keep their existing stopping rules.

The fixed canonical phase performs exactly 384 lifecycle cycles, with the
existing active-resource restart after every fourth cycle: 96 completed restarts.
It then performs exactly one settling cycle, without another restart. The phase,
including observer handshakes, must finish within 1,800 seconds. Initial fixture
startup has a separate 120-second bound. There is no early completion, extension,
RSS-selected endpoint, budget reset, or two-hour/full-pass flag. Cleanup remains
bounded independently of the phase deadline.

All canonical operations remain in use: encrypted opc.tcp sessions, namespace URI
binding, scalar/typed-array read/write, subscriptions, browse abandonment, history
continuation churn/release/cross-session/close, surgical changes/rebuilds, key
revocation, paused network reads/recovery, and same-PKI server restart. All
ownership, continuation, PKI-descriptor, store and node inventory checks remain
active. Cold budget rejection is latched at its original observation and rethrown,
even if diagnostic readiness later succeeds. A rejected readiness receipt does
not become a passing test or a new resource baseline.

## Observer boundaries

A fresh task-owned control directory and UUID bind every request/receipt to this
run and target PID. Files are published atomically; existing, malformed, foreign,
or incomplete receipts fail closed. The observer is a separate diagnostic process
using public `DiagnosticsClient` / EventPipe APIs. Product dependencies are
unchanged. A separate short positive control first invokes a named non-SDK method
with NoInlining/NoOptimization after attaching and requires its JIT-start count in
the parsed receipt. This proves CLR provider delivery/classification independently
of custom marker delivery. That probe is never invoked by the canonical experiment.

One continuous trace covers both fixed windows, including their boundary gap:

| Marker | Before/after operation | Completed restarts |
|---|---|---:|
| window1-start | Before cycle 321 | 80 |
| window1-end | After cycle 352 and its restart | 88 |
| window2-start | Before cycle 353 | 88 |
| window2-end | After cycle 384 and its restart | 96 |

The target waits for the observer's active receipt and enabled marker provider
before its first window operation. The parser requires exactly these four
markers, the correct UUID and cycle/restart values, and strictly ordered
timestamps. A continuous veto also covers events outside the marker ranges
within the attached interval. Both windows and the entire attached interval
require zero `Opc.Ua.OpcUaExtensions` JIT starts and method-load events, zero new
`Opc.Ua.*` assembly-load events, and zero lost events. Other JIT counts and loaded
native method-byte sums are reported for review; they do not establish that all
runtime paths are warmed. Method-size sums do not equal resident memory.

Runtime capture is Verbose `Microsoft-Windows-DotNETRuntime`, keywords `0x18`
(Loader + JIT), plus Informational `OtOpcUa-Sdk2-Saturation` markers, no rundown,
and a prospectively fixed **16 MiB** circular-buffer capacity. This is a new
experiment and does not reinterpret the earlier 256 MiB-capacity traces.

After restart 96, the target requests stop and waits. The observer must stop its
session, drain the event stream, close it, parse the entire trace, and atomically
publish a PID/run-bound drained receipt. Its Stop/copy/dispose/parse stages have
10/10/5/10-second bounds. Target receipt waits are bounded at 45 seconds and remain
inside the 1,800-second phase deadline. Observer faults propagate immediately.
The target also verifies its marker provider is disabled before the settling
cycle. Missing markers, no events, event loss, or malformed stream cannot prove
readiness. A parser failure produces a failed observer receipt, not empty counts.

The `saturation-result` separates readiness, cold budget rejection, fixed operation
completion and verified shutdown ownership. It always records `acceptance=false`.
It is emitted before rethrowing the original cold assertion. The launcher retains
nonzero test/container exits and records diagnostic readiness separately.

## Prepare immutable Linux artifacts

Commit the reviewed source first. Preparation refuses a dirty checkout and
publishes fresh, separate app/observer artifacts for `linux-arm64`, self-contained
.NET **10.0.14**, without trimming or single-file publishing. It never starts the
experiment. The bundle records source revision, source hashes, exact publish
command, fixed protocol, image digest and SHA-256 of every app/observer/tool file.
It requires the landed HistorianGateway Client/Contracts 1.0.0 dependency graph
and records assembly versions/module MVIDs by reading PE metadata without loading
or executing product assemblies.

```bash
python3 scripts/validation/opcua-lifecycle-memory/prepare_saturation.py \
  --output /tmp/opcua-saturation-reviewed \
  --libraries "$diagnosticToolLibraries" \
  --image sha256:b04611ee9e1b7d36797012ed921962c57189cb4dbb3dff12afb8e158110f293b
```

`diagnosticToolLibraries` is the installed dotnet-trace 9.0.661903
`tools/net8.0/any` directory. No product package reference is added. Artifact
preparation and review are prerequisites; root and Astra must review the actual
code, control tests and receipt before launch. The coordinator owns the long
execution.

```bash
python3 /tmp/opcua-saturation-reviewed/tools/launch_saturation.py \
  --bundle /tmp/opcua-saturation-reviewed --receipt-sha256 "$reviewedReceiptHash" \
  --output /tmp/opcua-saturation-run --verify-only
# After root/Astra review, root runs the same command without --verify-only.
```

The launcher rechecks every immutable file before and after the run. It creates a
fresh offline task-owned container, writable output/PKI directories, and read-only
app/observer/tool mounts. Descriptor controls run first in a separate process.
TRX guards require two descriptor tests and one canonical test actually executed,
with no skipped results. The launcher verifies the actual target and observer
mapped their own self-contained `libcoreclr.so`; the older SDK-image controller
runtime is not the target runtime. A 2,040-second outer timeout includes startup,
phase and bounded cleanup; it does not extend the 1,800-second phase.

No long saturation run has been launched during implementation. Existing cold
failures remain untouched. No original-SDK full-workload adapter, warmed
acceptance baseline, additional two-hour server run, or production correction is
part of this change. The separately demonstrated Mac native certificate-related
inventory growth remains unresolved even if generated-factory compilation
saturates.