Files
lmxopcua/scripts/validation/opcua-lifecycle-memory/windows/launch.ps1
T
Joseph Doherty fea7de0aef
v2-ci / unit-tests (push) Waiting to run
docs(evidence): #799 acceptance runs on master aa60b3e3 - Windows x64 and Linux both fail the unchanged cold RSS cap
Owner scope 2026-11-12 (Windows x64 + Linux, unchanged cycle-3 caps, >=7200 s, SDK 1.5 comparator).

- Linux ARM64 (late-slope, 7200.12 s, 1964 cycles / 491 restarts, shutdown verified, analyzer-validated):
  first rejection RSS at cycle 247 / 900.73 s, 358,682,624 > 358,678,528 (+4 KiB); peak 382.98 MiB vs
  limit 342.06 MiB; late plateau medians 344.4/341.1/341.9 MiB, OLS -1.90 MiB/h. FAIL.
- Windows x64 on windev (same workload, working set as RSS, handle table as descriptors): first rejection RSS
  at cycle 285 / 1130.76 s, 303,505,408 > 302,174,208; peak 319.49 vs limit 288.18 MiB; plateau ~314-315 MiB,
  +0.93 MiB/h. FAIL. Also incomplete: the launcher's wall-clock outer deadline fired early after a +552 s
  w32time step and ended the run at test-phase 7014 s (rejection had already latched); launchers now use a
  Stopwatch.
- SDK 1.5.378.182 comparator (efb9197a publication, Windows, 7201 s): RSS peak +65.8 MiB (never over its cap),
  plateau ~+48 MiB, but the +32 handle cap tripped at cycle 17 (max 674 vs 642).
- Managed flat on both SDK2 runs; growth is the generated-factory JIT burst (+40k/+44k methods) - Linux JIT
  doublemapper ~180 MiB; Windows private bytes grew ~25 MiB of a ~132 MiB working-set rise.
- Retained: b61f3e05 cycle-1 fixture failure (#808 unplaced UNS folder), Windows/SDK1 smoke diagnostics.

No cap raised, no warm baseline, no failed run counted as passing.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 14:51:31 -05:00

160 lines
9.7 KiB
PowerShell

<#
.SYNOPSIS
Windows counterpart of ../late-slope/container.sh + launch.py for the lmxopcua#799 acceptance soak.
.DESCRIPTION
Runs the immutable self-contained win-x64 bundle through the installed `dotnet vstest` controller, exactly as the
Linux container runs it: the two descriptor controls must pass first, then the single canonical test
Repeated_lifecycles_return_resources_to_a_steady_baseline in late-slope mode (7,200 s, unchanged cycle-3 caps,
first-rejection latch, all canonical operations, independent shutdown checks).
Windows adaptations (documented in README.md):
- No container. Isolation is a fresh private task directory (inheritance removed, current user + SYSTEM only)
that holds TEMP/TMP, the CLI home, results and fixture PKI. The machine is otherwise idle.
- The target process is verified by PID: its image must be the bundle apphost and coreclr.dll must be mapped from
the bundle with the receipt's SHA-256 (the /proc/<pid>/exe + maps check on Linux).
- An external read-only sampler records the target's WorkingSet64 / PrivateMemorySize64 / HandleCount every 30 s
(descriptive only; the in-process quiescent samples remain the gated values), plus MxAccessGw readiness.
- The 7,440 s outer deadline kills only the task-owned process tree.
#>
param(
[Parameter(Mandatory)] [string] $Bundle,
[Parameter(Mandatory)] [string] $Output,
[Parameter(Mandatory)] [string] $SourceRevision,
[ValidateSet('late-slope', 'diagnostic')] [string] $Mode = 'late-slope',
[int] $Seconds = 7200
)
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version 3
$app = 'ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests'
$testClass = "$app.Sdk2LifecycleSoakTests"
$outerDeadline = 7440
if ($SourceRevision -notmatch '^[0-9a-f]{40}$') { throw 'Invalid source revision' }
if ($Mode -eq 'late-slope' -and $Seconds -ne 7200) { throw 'late-slope is fixed at 7200 s' }
$Bundle = (Resolve-Path $Bundle).Path
if (Test-Path $Output) { throw 'Use a fresh output directory' }
New-Item -ItemType Directory -Path $Output | Out-Null
$Output = (Resolve-Path $Output).Path
foreach ($d in 'tmp', 'cli', 'data', 'results') { New-Item -ItemType Directory -Path (Join-Path $Output $d) | Out-Null }
# Private task directory: no inherited ACEs, only the running user and SYSTEM.
& icacls $Output /inheritance:r /grant:r "$($env:USERNAME):(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' | Out-Null
$started = Get-Date
# Monotonic: the 2026-11-12 windev run lost its last 186 s when a +552 s host clock step (w32time) advanced a wall-clock deadline.
$clock = [System.Diagnostics.Stopwatch]::StartNew()
function Write-Json($path, $value) { $value | ConvertTo-Json -Depth 6 | Set-Content -Encoding UTF8 -Path $path }
# 1. Re-verify every bundle file against the prepared manifest before anything runs.
$bad = 0; $count = 0
foreach ($line in Get-Content (Join-Path $Bundle 'manifest.sha256')) {
$sha, $name = $line -split ' ', 2
$file = Join-Path $Bundle ($name -replace '/', '\')
$count++
if ((Get-FileHash -Algorithm SHA256 -LiteralPath $file).Hash.ToLowerInvariant() -ne $sha) { $bad++ }
}
if ($bad -ne 0 -or $count -lt 900) { throw "Bundle verification failed: $bad mismatches of $count" }
$appDir = Join-Path $Bundle 'app'
$coreClr = Join-Path $appDir 'coreclr.dll'
$coreClrSha = (Get-FileHash -Algorithm SHA256 $coreClr).Hash.ToLowerInvariant()
Write-Json (Join-Path $Output 'launch.json') ([ordered]@{
kind = 'launch'; protocol = 'sdk2-late-slope-v1'; platform = 'windows'; mode = $Mode; requestedSeconds = $Seconds
sourceRevision = $SourceRevision; bundlePath = $Bundle; outputPath = $Output; verifiedBundleFiles = $count
coreClrSha256 = $coreClrSha; outerDeadlineSeconds = $outerDeadline; startedUtc = $started.ToUniversalTime().ToString('o')
host = $env:COMPUTERNAME; os = [Environment]::OSVersion.VersionString
logicalProcessors = [Environment]::ProcessorCount
physicalMemoryBytes = (Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory })
# 2. Same environment as container.sh.
$env:TEMP = Join-Path $Output 'tmp'; $env:TMP = $env:TEMP
$env:DOTNET_CLI_HOME = Join-Path $Output 'cli'
$env:OTOPCUA_SDK2_SOAK = '1'; $env:OTOPCUA_SDK2_SOAK_MODE = $Mode; $env:OTOPCUA_SDK2_SOAK_SECONDS = "$Seconds"
$env:OTOPCUA_SDK2_SOAK_OUTPUT = Join-Path $Output 'resources.jsonl'; $env:OPCUA_CERTIFICATE_LEAK_TRACKING = '1'
$env:DOTNET_CLI_TELEMETRY_OPTOUT = '1'; $env:DOTNET_NOLOGO = '1'; $env:OTOPCUA_SOAK_BUILD_SHA = $SourceRevision
& dotnet --info > (Join-Path $Output 'controller-runtime.txt')
$dll = Join-Path $appDir "$app.dll"
$results = Join-Path $Output 'results'
# 3. Descriptor controls must actually execute and pass before the long test.
& dotnet vstest $dll "--TestCaseFilter:FullyQualifiedName~$testClass.Descriptor_probe_" "--ResultsDirectory:$results" `
'--Logger:trx;LogFileName=descriptor-controls.trx' > (Join-Path $Output 'descriptor-controls.log') 2>&1
[xml] $trx = Get-Content (Join-Path $results 'descriptor-controls.trx')
$c = $trx.TestRun.ResultSummary.Counters
if ($c.total -ne '2' -or $c.executed -ne '2' -or $c.passed -ne '2' -or $c.failed -ne '0') {
'controls-failed' | Set-Content (Join-Path $Output 'launcher-outcome.txt'); exit 2
}
# 4. The canonical test, one process, output captured.
$psi = New-Object System.Diagnostics.ProcessStartInfo 'dotnet'
$psi.Arguments = "vstest `"$dll`" `"--TestCaseFilter:FullyQualifiedName=$testClass.Repeated_lifecycles_return_resources_to_a_steady_baseline`" `"--ResultsDirectory:$results`" `"--Logger:trx;LogFileName=late-slope.trx`""
$psi.UseShellExecute = $false; $psi.RedirectStandardOutput = $true; $psi.RedirectStandardError = $true
$vstest = [System.Diagnostics.Process]::Start($psi)
$log = [System.IO.StreamWriter]::new((Join-Path $Output 'test.log'))
$outTask = $vstest.StandardOutput.ReadToEndAsync(); $errTask = $vstest.StandardError.ReadToEndAsync()
# 5. Verify the actual test process within 150 s: bundle apphost image + coreclr mapped from the bundle.
$resources = Join-Path $Output 'resources.jsonl'
$targetPid = $null; $deadline = (Get-Date).AddSeconds(150)
while ((Get-Date) -lt $deadline -and -not $vstest.HasExited) {
if ((Test-Path $resources) -and (Get-Item $resources).Length -gt 0) {
$fs = [System.IO.File]::Open($resources, 'Open', 'Read', 'ReadWrite')
try { $header = ([System.IO.StreamReader]::new($fs)).ReadLine() } finally { $fs.Dispose() }
if ($header -and $header.Contains('"kind":"start"')) {
$targetPid = [int](($header | ConvertFrom-Json).processId); break
}
}
Start-Sleep -Milliseconds 200
}
$verified = $false
if ($targetPid) {
$target = Get-Process -Id $targetPid
$image = $target.Path
$mapped = @($target.Modules | Where-Object { $_.ModuleName -ieq 'coreclr.dll' } | ForEach-Object FileName)
$mappedSha = if ($mapped.Count -eq 1) { (Get-FileHash -Algorithm SHA256 $mapped[0]).Hash.ToLowerInvariant() } else { '' }
$verified = ($image -ieq (Join-Path $appDir "$app.exe")) -and $mapped.Count -eq 1 -and ($mapped[0] -ieq $coreClr) -and ($mappedSha -eq $coreClrSha)
"targetPid=$targetPid`r`nactualExecutable=$image`r`nmappedCoreClr=$($mapped -join ';')`r`ncoreClrSha256=$mappedSha" |
Set-Content -Encoding ASCII (Join-Path $Output 'target-runtime.txt')
}
if (-not $verified) {
try { & taskkill /PID $vstest.Id /T /F | Out-Null } catch { }
}
# 6. External read-only sampler + outer deadline.
$sampler = Join-Path $Output 'external-samples.jsonl'
$nextHealth = Get-Date
while (-not $vstest.WaitForExit(30000)) {
$elapsed = $clock.Elapsed.TotalSeconds
$row = [ordered]@{ utc = (Get-Date).ToUniversalTime().ToString('o'); launcherElapsedSeconds = [math]::Round($elapsed, 1) }
try {
$p = Get-Process -Id $targetPid -ErrorAction Stop
$row.workingSet64 = $p.WorkingSet64; $row.privateMemorySize64 = $p.PrivateMemorySize64
$row.pagedMemorySize64 = $p.PagedMemorySize64; $row.peakWorkingSet64 = $p.PeakWorkingSet64
$row.handleCount = $p.HandleCount; $row.threads = $p.Threads.Count
} catch { $row.targetGone = $true }
if ((Get-Date) -ge $nextHealth) {
try { $row.mxAccessGwReady = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://localhost:5130/health/ready').StatusCode }
catch { $row.mxAccessGwReady = "error: $($_.Exception.Message)" }
$row.mxAccessGwService = (Get-Service MxAccessGw).Status.ToString()
$nextHealth = (Get-Date).AddMinutes(5)
}
($row | ConvertTo-Json -Compress) | Add-Content -Encoding UTF8 $sampler
if ($elapsed -gt $outerDeadline) {
& taskkill /PID $vstest.Id /T /F | Out-Null
'outer-deadline' | Set-Content (Join-Path $Output 'launcher-outcome.txt')
}
}
$log.Write($outTask.Result); $log.Write($errTask.Result); $log.Dispose()
$status = $vstest.ExitCode
"$status" | Set-Content -Encoding ASCII (Join-Path $Output 'test-exit-status.txt')
$wall = $clock.Elapsed.TotalSeconds
$evidence = [ordered]@{}
foreach ($f in Get-ChildItem -LiteralPath $Output -File -Recurse | Where-Object { $_.FullName -notlike "$Output\tmp\*" -and $_.FullName -notlike "$Output\cli\*" -and $_.FullName -notlike "$Output\data\*" }) {
$evidence[$f.FullName.Substring($Output.Length + 1).Replace('\', '/')] = (Get-FileHash -Algorithm SHA256 $f.FullName).Hash.ToLowerInvariant()
}
Write-Json (Join-Path $Output 'run-receipt.json') ([ordered]@{
kind = 'run'; platform = 'windows'; sourceRevision = $SourceRevision; mode = $Mode; testExitStatus = $status
targetVerified = $verified; wallSeconds = $wall; finishedUtc = (Get-Date).ToUniversalTime().ToString('o')
leftoverTaskProcesses = @(Get-CimInstance Win32_Process | Where-Object { $_.ExecutablePath -and $_.ExecutablePath.StartsWith($Bundle, 'OrdinalIgnoreCase') }).Count
evidenceSha256 = $evidence })
'done' | Add-Content (Join-Path $Output 'launcher-outcome.txt')
exit $status