lmxopcua/scripts/validation/opcua-lifecycle-memory/LATE_SLOPE.md

# Proposed fixed-duration late-slope diagnostic

Design draft only. No new mode, artifact or long execution is approved by this
document. Resolve the current ownership/code gates and review native attribution
before deciding whether this experiment supplies a still-needed fact for #799.

The question is whether the canonical Linux process continues to grow well after
the observed generated-code step. This is separate from choosing a warmed
acceptance baseline, a cross-version performance denominator, or resolving macOS
private-key import retention. The existing `profile` allowance stays 60–1800s.

If approved, add a distinct diagnostic mode fixed at **7200 seconds** of canonical
cycling after startup, including its cold and qualification-era cycles. It must
not claim 7200 seconds *after warmup*. Keep every normal operation, the active
restart every fourth cycle, all logical/ownership checks and actual PKI descriptor
checks. Use fresh committed source and immutable self-contained Linux ARM64
.NET10.0.14 artifacts, exact SDK2.0.1 and the landed HG1.0.0 graph. Freeze image,
PKI identity, fixture and instrumentation before launch. Root owns the long run.

Keep the original cycle-3 baseline and +64MiB managed / +128MiB RSS / +32 FD caps.
Record the sample before asserting; catch only the three resource-cap assertions,
latch the first rejection permanently and continue **every canonical cycle and
restart**, with resource sampling. Managed-monotonic, logical/ownership, observer,
operation and shutdown failures remain fatal. After final ownership/shutdown
checks, emit a diagnostic receipt with `acceptance=false` and rethrow the original
cap rejection. Never emit ordinary `complete`/two-hour acceptance or convert an
earlier rejection to success. A duration completion is distinct from test success.

Prospectively report **all** quiescent observations, cold rejection, maximum
observed quiescent RSS and final values. Predeclare these elapsed-time windows:

| Window | Quiescent sample elapsed seconds |
|---|---:|
| Early late phase | 1800–2400 |
| Middle late phase | 3600–4200 |
| Final late phase | 6600–7200 |

Proposed fixed bounds are startup 120s, each complete cycle including its restart
30s, cycle deadline 7230s, shutdown ceiling 60s and container deadline 7440s.
The active restart helper has a separate 25s linked deadline; its elapsed time
is recorded before post-restart configuration. The final cycle may finish after 7200s, but cannot extend
its 30s bound. Validate these bounds with reviewed short controls before committing
the protocol; hitting one makes the diagnostic incomplete, not grounds to relax it.

The existing quiescent sample precedes the optional restart. Thus two adjacent
bounded cycles allow at most 60s between quiescent samples. Each 600s window must
have at least 10 samples, first sample no later than 60s after its start, last sample
no earlier than 60s before its end, observed span at least 480s, and no internal gap
above 60s. Apply the same 60s maximum gap over the entire 1800–7200s reporting range.
Missing coverage is incomplete evidence; it must not become a favorable slope.
Unless at least 385 cycles and 96 restarts complete by 1800s, report the elapsed
slopes descriptively but withhold attribution to the phase after the previously
observed restart milestone. Reaching that count still does not prove factory
saturation in this run; no EventPipe observer is attached.

For each window report sample count, first/last/min/max/median RSS and retained
managed bytes, FD range, GC committed range, JIT/timer/thread ranges and elapsed
coverage. Missing windows are incomplete evidence. Also report ordinary least
squares RSS and managed-byte slopes against actual sample elapsed seconds for
all observations in 1800–7200s, with units bytes/hour and the exact sample count.
Do not choose a favorable endpoint/window/statistic after observing results.
These descriptive slopes have **no new pass threshold**. Quiescent maxima are
not continuous process peak measurements, and native MethodSize is not RSS.

Retain certificate allocation StackTrace tracking at the same enabled setting as
the earlier canonical saturation experiment and disclose its overhead. Do not
attach EventPipe to this proposed run or claim generated-factory readiness from
absent counters; earlier traces remain separate source-scoped evidence. This
instrumented diagnostic cannot be relabelled as uninstrumented acceptance or as
one of the required representative-load comparison runs.

Freeze cycle-phase elapsed timestamps before cleanup and retain shutdown duration
separately; no cleanup sample feeds the windows or slopes. The new mode must
reach its explicit post-shutdown checks before rethrowing a latched resource cap;
the generic profile mode currently throws earlier and must keep its old behavior.
Cleanup must attempt both independent owners under their separate deadlines even
when one fails or the operation token is canceled. Preserve the first rejection
alongside any later cleanup failure; neither error may erase the other.
The shutdown ceiling covers 5s to capture manager references, 20s for each
owner, 5s to delete the private client directory and 10s to verify shutdown.
Any local deadline or overall elapsed miss is fatal and makes the diagnostic
incomplete; the container deadline is unchanged.

Before execution, implement and review the proposed setup/last-cycle/cleanup and
outer deadlines in the actual harness and launcher; no extension or retry-to-green.
Cycle cancellation is cooperative. If a call ignores its token and does not
return within the bound, the diagnostic is incomplete; the outer deadline reaps
the task-owned container. Never infer a completed bounded cycle from silence.
Require negative controls proving a latched first failure cannot disappear,
wrong duration/mode is rejected, no ordinary acceptance receipt can be emitted,
and malformed/missing sample windows fail the summarizer. Review actual source,
control results and every immutable artifact hash with Astra before launch.

The canonical mode is implemented. The [short control receipts](../../../docs/plans/evidence/2026-11-10-opcua2/late-slope-controls.json)
record three passing helper tests and actual rejection of a 90-second request
before setup. The original helper assertion-capture failure is retained; the
canonical source is byte-identical across that test-only correction. These
controls do not approve a bundle, start the workload or satisfy #799.

The first bundle verified but its separate rehearsal stopped in .NET CLI
startup: [retained startup failure](../../../docs/plans/evidence/2026-11-10-opcua2/late-slope-rehearsal-startup-failure.json).
The corrected launcher keeps the root read-only and aliases the same private
task temp directory at `/tmp`. The replacement bundle from `b18f9723` passed
manifest/source/MVID review and its separate [60-second rehearsal](../../../docs/plans/evidence/2026-11-10-opcua2/late-slope-rehearsal.json)
completed 16 cycles and four restarts with all three tests passed and cleanup
verified. Astra approved one fixed 7,200-second diagnostic on that exact bundle;
this approval does not adopt a new acceptance baseline or clear #799/G4.

The [fixed run result](../../../docs/plans/evidence/2026-11-10-opcua2/late-slope-linux.md)
completed all 7,201.014 seconds of cycling and verified shutdown, with 1,937
cycles, 484 restarts and no cold resource rejection. Peak quiescent RSS was
4.20 MiB below the original cap. The three fixed-window medians and whole late
phase fit describe one tracked Linux run. Earlier failures remain, the receipt
has `acceptance=false`, and #799/G4 stay open.