fea7de0aef
v2-ci / unit-tests (push) Waiting to run
Owner scope 2026-11-12 (Windows x64 + Linux, unchanged cycle-3 caps, >=7200 s, SDK 1.5 comparator). - Linux ARM64 (late-slope, 7200.12 s, 1964 cycles / 491 restarts, shutdown verified, analyzer-validated): first rejection RSS at cycle 247 / 900.73 s, 358,682,624 > 358,678,528 (+4 KiB); peak 382.98 MiB vs limit 342.06 MiB; late plateau medians 344.4/341.1/341.9 MiB, OLS -1.90 MiB/h. FAIL. - Windows x64 on windev (same workload, working set as RSS, handle table as descriptors): first rejection RSS at cycle 285 / 1130.76 s, 303,505,408 > 302,174,208; peak 319.49 vs limit 288.18 MiB; plateau ~314-315 MiB, +0.93 MiB/h. FAIL. Also incomplete: the launcher's wall-clock outer deadline fired early after a +552 s w32time step and ended the run at test-phase 7014 s (rejection had already latched); launchers now use a Stopwatch. - SDK 1.5.378.182 comparator (efb9197apublication, Windows, 7201 s): RSS peak +65.8 MiB (never over its cap), plateau ~+48 MiB, but the +32 handle cap tripped at cycle 17 (max 674 vs 642). - Managed flat on both SDK2 runs; growth is the generated-factory JIT burst (+40k/+44k methods) - Linux JIT doublemapper ~180 MiB; Windows private bytes grew ~25 MiB of a ~132 MiB working-set rise. - Retained:b61f3e05cycle-1 fixture failure (#808 unplaced UNS folder), Windows/SDK1 smoke diagnostics. No cap raised, no warm baseline, no failed run counted as passing. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
160 lines
9.7 KiB
PowerShell
160 lines
9.7 KiB
PowerShell
<#
|
|
.SYNOPSIS
|
|
Windows counterpart of ../late-slope/container.sh + launch.py for the lmxopcua#799 acceptance soak.
|
|
|
|
.DESCRIPTION
|
|
Runs the immutable self-contained win-x64 bundle through the installed `dotnet vstest` controller, exactly as the
|
|
Linux container runs it: the two descriptor controls must pass first, then the single canonical test
|
|
Repeated_lifecycles_return_resources_to_a_steady_baseline in late-slope mode (7,200 s, unchanged cycle-3 caps,
|
|
first-rejection latch, all canonical operations, independent shutdown checks).
|
|
|
|
Windows adaptations (documented in README.md):
|
|
- No container. Isolation is a fresh private task directory (inheritance removed, current user + SYSTEM only)
|
|
that holds TEMP/TMP, the CLI home, results and fixture PKI. The machine is otherwise idle.
|
|
- The target process is verified by PID: its image must be the bundle apphost and coreclr.dll must be mapped from
|
|
the bundle with the receipt's SHA-256 (the /proc/<pid>/exe + maps check on Linux).
|
|
- An external read-only sampler records the target's WorkingSet64 / PrivateMemorySize64 / HandleCount every 30 s
|
|
(descriptive only; the in-process quiescent samples remain the gated values), plus MxAccessGw readiness.
|
|
- The 7,440 s outer deadline kills only the task-owned process tree.
|
|
#>
|
|
param(
|
|
[Parameter(Mandatory)] [string] $Bundle,
|
|
[Parameter(Mandatory)] [string] $Output,
|
|
[Parameter(Mandatory)] [string] $SourceRevision,
|
|
[ValidateSet('late-slope', 'diagnostic')] [string] $Mode = 'late-slope',
|
|
[int] $Seconds = 7200
|
|
)
|
|
$ErrorActionPreference = 'Stop'
|
|
Set-StrictMode -Version 3
|
|
$app = 'ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests'
|
|
$testClass = "$app.Sdk2LifecycleSoakTests"
|
|
$outerDeadline = 7440
|
|
if ($SourceRevision -notmatch '^[0-9a-f]{40}$') { throw 'Invalid source revision' }
|
|
if ($Mode -eq 'late-slope' -and $Seconds -ne 7200) { throw 'late-slope is fixed at 7200 s' }
|
|
$Bundle = (Resolve-Path $Bundle).Path
|
|
if (Test-Path $Output) { throw 'Use a fresh output directory' }
|
|
New-Item -ItemType Directory -Path $Output | Out-Null
|
|
$Output = (Resolve-Path $Output).Path
|
|
foreach ($d in 'tmp', 'cli', 'data', 'results') { New-Item -ItemType Directory -Path (Join-Path $Output $d) | Out-Null }
|
|
# Private task directory: no inherited ACEs, only the running user and SYSTEM.
|
|
& icacls $Output /inheritance:r /grant:r "$($env:USERNAME):(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' | Out-Null
|
|
$started = Get-Date
|
|
# Monotonic: the 2026-11-12 windev run lost its last 186 s when a +552 s host clock step (w32time) advanced a wall-clock deadline.
|
|
$clock = [System.Diagnostics.Stopwatch]::StartNew()
|
|
|
|
function Write-Json($path, $value) { $value | ConvertTo-Json -Depth 6 | Set-Content -Encoding UTF8 -Path $path }
|
|
|
|
# 1. Re-verify every bundle file against the prepared manifest before anything runs.
|
|
$bad = 0; $count = 0
|
|
foreach ($line in Get-Content (Join-Path $Bundle 'manifest.sha256')) {
|
|
$sha, $name = $line -split ' ', 2
|
|
$file = Join-Path $Bundle ($name -replace '/', '\')
|
|
$count++
|
|
if ((Get-FileHash -Algorithm SHA256 -LiteralPath $file).Hash.ToLowerInvariant() -ne $sha) { $bad++ }
|
|
}
|
|
if ($bad -ne 0 -or $count -lt 900) { throw "Bundle verification failed: $bad mismatches of $count" }
|
|
$appDir = Join-Path $Bundle 'app'
|
|
$coreClr = Join-Path $appDir 'coreclr.dll'
|
|
$coreClrSha = (Get-FileHash -Algorithm SHA256 $coreClr).Hash.ToLowerInvariant()
|
|
Write-Json (Join-Path $Output 'launch.json') ([ordered]@{
|
|
kind = 'launch'; protocol = 'sdk2-late-slope-v1'; platform = 'windows'; mode = $Mode; requestedSeconds = $Seconds
|
|
sourceRevision = $SourceRevision; bundlePath = $Bundle; outputPath = $Output; verifiedBundleFiles = $count
|
|
coreClrSha256 = $coreClrSha; outerDeadlineSeconds = $outerDeadline; startedUtc = $started.ToUniversalTime().ToString('o')
|
|
host = $env:COMPUTERNAME; os = [Environment]::OSVersion.VersionString
|
|
logicalProcessors = [Environment]::ProcessorCount
|
|
physicalMemoryBytes = (Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory })
|
|
|
|
# 2. Same environment as container.sh.
|
|
$env:TEMP = Join-Path $Output 'tmp'; $env:TMP = $env:TEMP
|
|
$env:DOTNET_CLI_HOME = Join-Path $Output 'cli'
|
|
$env:OTOPCUA_SDK2_SOAK = '1'; $env:OTOPCUA_SDK2_SOAK_MODE = $Mode; $env:OTOPCUA_SDK2_SOAK_SECONDS = "$Seconds"
|
|
$env:OTOPCUA_SDK2_SOAK_OUTPUT = Join-Path $Output 'resources.jsonl'; $env:OPCUA_CERTIFICATE_LEAK_TRACKING = '1'
|
|
$env:DOTNET_CLI_TELEMETRY_OPTOUT = '1'; $env:DOTNET_NOLOGO = '1'; $env:OTOPCUA_SOAK_BUILD_SHA = $SourceRevision
|
|
& dotnet --info > (Join-Path $Output 'controller-runtime.txt')
|
|
$dll = Join-Path $appDir "$app.dll"
|
|
$results = Join-Path $Output 'results'
|
|
|
|
# 3. Descriptor controls must actually execute and pass before the long test.
|
|
& dotnet vstest $dll "--TestCaseFilter:FullyQualifiedName~$testClass.Descriptor_probe_" "--ResultsDirectory:$results" `
|
|
'--Logger:trx;LogFileName=descriptor-controls.trx' > (Join-Path $Output 'descriptor-controls.log') 2>&1
|
|
[xml] $trx = Get-Content (Join-Path $results 'descriptor-controls.trx')
|
|
$c = $trx.TestRun.ResultSummary.Counters
|
|
if ($c.total -ne '2' -or $c.executed -ne '2' -or $c.passed -ne '2' -or $c.failed -ne '0') {
|
|
'controls-failed' | Set-Content (Join-Path $Output 'launcher-outcome.txt'); exit 2
|
|
}
|
|
|
|
# 4. The canonical test, one process, output captured.
|
|
$psi = New-Object System.Diagnostics.ProcessStartInfo 'dotnet'
|
|
$psi.Arguments = "vstest `"$dll`" `"--TestCaseFilter:FullyQualifiedName=$testClass.Repeated_lifecycles_return_resources_to_a_steady_baseline`" `"--ResultsDirectory:$results`" `"--Logger:trx;LogFileName=late-slope.trx`""
|
|
$psi.UseShellExecute = $false; $psi.RedirectStandardOutput = $true; $psi.RedirectStandardError = $true
|
|
$vstest = [System.Diagnostics.Process]::Start($psi)
|
|
$log = [System.IO.StreamWriter]::new((Join-Path $Output 'test.log'))
|
|
$outTask = $vstest.StandardOutput.ReadToEndAsync(); $errTask = $vstest.StandardError.ReadToEndAsync()
|
|
|
|
# 5. Verify the actual test process within 150 s: bundle apphost image + coreclr mapped from the bundle.
|
|
$resources = Join-Path $Output 'resources.jsonl'
|
|
$targetPid = $null; $deadline = (Get-Date).AddSeconds(150)
|
|
while ((Get-Date) -lt $deadline -and -not $vstest.HasExited) {
|
|
if ((Test-Path $resources) -and (Get-Item $resources).Length -gt 0) {
|
|
$fs = [System.IO.File]::Open($resources, 'Open', 'Read', 'ReadWrite')
|
|
try { $header = ([System.IO.StreamReader]::new($fs)).ReadLine() } finally { $fs.Dispose() }
|
|
if ($header -and $header.Contains('"kind":"start"')) {
|
|
$targetPid = [int](($header | ConvertFrom-Json).processId); break
|
|
}
|
|
}
|
|
Start-Sleep -Milliseconds 200
|
|
}
|
|
$verified = $false
|
|
if ($targetPid) {
|
|
$target = Get-Process -Id $targetPid
|
|
$image = $target.Path
|
|
$mapped = @($target.Modules | Where-Object { $_.ModuleName -ieq 'coreclr.dll' } | ForEach-Object FileName)
|
|
$mappedSha = if ($mapped.Count -eq 1) { (Get-FileHash -Algorithm SHA256 $mapped[0]).Hash.ToLowerInvariant() } else { '' }
|
|
$verified = ($image -ieq (Join-Path $appDir "$app.exe")) -and $mapped.Count -eq 1 -and ($mapped[0] -ieq $coreClr) -and ($mappedSha -eq $coreClrSha)
|
|
"targetPid=$targetPid`r`nactualExecutable=$image`r`nmappedCoreClr=$($mapped -join ';')`r`ncoreClrSha256=$mappedSha" |
|
|
Set-Content -Encoding ASCII (Join-Path $Output 'target-runtime.txt')
|
|
}
|
|
if (-not $verified) {
|
|
try { & taskkill /PID $vstest.Id /T /F | Out-Null } catch { }
|
|
}
|
|
|
|
# 6. External read-only sampler + outer deadline.
|
|
$sampler = Join-Path $Output 'external-samples.jsonl'
|
|
$nextHealth = Get-Date
|
|
while (-not $vstest.WaitForExit(30000)) {
|
|
$elapsed = $clock.Elapsed.TotalSeconds
|
|
$row = [ordered]@{ utc = (Get-Date).ToUniversalTime().ToString('o'); launcherElapsedSeconds = [math]::Round($elapsed, 1) }
|
|
try {
|
|
$p = Get-Process -Id $targetPid -ErrorAction Stop
|
|
$row.workingSet64 = $p.WorkingSet64; $row.privateMemorySize64 = $p.PrivateMemorySize64
|
|
$row.pagedMemorySize64 = $p.PagedMemorySize64; $row.peakWorkingSet64 = $p.PeakWorkingSet64
|
|
$row.handleCount = $p.HandleCount; $row.threads = $p.Threads.Count
|
|
} catch { $row.targetGone = $true }
|
|
if ((Get-Date) -ge $nextHealth) {
|
|
try { $row.mxAccessGwReady = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://localhost:5130/health/ready').StatusCode }
|
|
catch { $row.mxAccessGwReady = "error: $($_.Exception.Message)" }
|
|
$row.mxAccessGwService = (Get-Service MxAccessGw).Status.ToString()
|
|
$nextHealth = (Get-Date).AddMinutes(5)
|
|
}
|
|
($row | ConvertTo-Json -Compress) | Add-Content -Encoding UTF8 $sampler
|
|
if ($elapsed -gt $outerDeadline) {
|
|
& taskkill /PID $vstest.Id /T /F | Out-Null
|
|
'outer-deadline' | Set-Content (Join-Path $Output 'launcher-outcome.txt')
|
|
}
|
|
}
|
|
$log.Write($outTask.Result); $log.Write($errTask.Result); $log.Dispose()
|
|
$status = $vstest.ExitCode
|
|
"$status" | Set-Content -Encoding ASCII (Join-Path $Output 'test-exit-status.txt')
|
|
$wall = $clock.Elapsed.TotalSeconds
|
|
$evidence = [ordered]@{}
|
|
foreach ($f in Get-ChildItem -LiteralPath $Output -File -Recurse | Where-Object { $_.FullName -notlike "$Output\tmp\*" -and $_.FullName -notlike "$Output\cli\*" -and $_.FullName -notlike "$Output\data\*" }) {
|
|
$evidence[$f.FullName.Substring($Output.Length + 1).Replace('\', '/')] = (Get-FileHash -Algorithm SHA256 $f.FullName).Hash.ToLowerInvariant()
|
|
}
|
|
Write-Json (Join-Path $Output 'run-receipt.json') ([ordered]@{
|
|
kind = 'run'; platform = 'windows'; sourceRevision = $SourceRevision; mode = $Mode; testExitStatus = $status
|
|
targetVerified = $verified; wallSeconds = $wall; finishedUtc = (Get-Date).ToUniversalTime().ToString('o')
|
|
leftoverTaskProcesses = @(Get-CimInstance Win32_Process | Where-Object { $_.ExecutablePath -and $_.ExecutablePath.StartsWith($Bundle, 'OrdinalIgnoreCase') }).Count
|
|
evidenceSha256 = $evidence })
|
|
'done' | Add-Content (Join-Path $Output 'launcher-outcome.txt')
|
|
exit $status
|