test(soak): #799 place the soak's UNS folder after #808; Windows x64 port of the lifecycle soak harness
v2-ci / unit-tests (push) Has been cancelled

Acceptance runs for #799 (owner scope 2026-11-12: Windows x64 + Linux) on current master.

Fixture: since #808 an UNS folder the authority snapshot does not place is denied under enforcement (fail
closed). The canonical soak creates its own "SoakEquipment" folder but never placed it, so on master the very
first cycle's UNS monitored item came back bad (retained failed Linux launch on b61f3e05, cycle 1). The rig
gains AdditionalFolderScopes (merged into FolderScopes by Enforce) and the soak places its folder there, as a
deployed generation does - the same fixture gap #888 fixed for the SDK matrix. No server authorization code,
cap, baseline or canonical operation changed; the soak test bodies are otherwise byte-identical to b18f9723.

Windows port (test/harness only, opt-in):
- DescriptorCountsAsync gains a Windows branch: total = every handle in the process handle table, PKI = disk
  file/directory handles whose GetFinalPathNameByHandle path lies under a canonicalised PKI root; dedicated
  thread under the same 5 s bound as the macOS lsof probe.
- Descriptor controls enabled on Windows (symlink alias; timeout control via cmd.exe + ping).
- LateSlopeRun.ValidateRequest accepts Linux ARM64 or Windows x64; duration/runtime/leak tracking unchanged.
- late-slope/analyze.py: expected architecture is a module constant (Linux default Arm64 unchanged).
- windows/: prepare.py (win-x64 self-contained 10.0.14 bundle + manifest), launch.ps1 (container.sh
  equivalent: bundle re-verification, 2/2 controls first, PID/apphost/coreclr verification, external
  sampler, 7440 s outer deadline), analyze.py (Linux row/window logic, X64), README with every adaptation.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 12:42:16 -05:00
parent 8ba0e22b27
commit aa60b3e34e
9 changed files with 558 additions and 14 deletions
@@ -21,6 +21,8 @@ RESTART_SECONDS = 25
SHUTDOWN_SECONDS = 60
CONTAINER_SECONDS = 7440
RUNTIME = "10.0.14"
# The canonical start row's process architecture. Linux runs are ARM64; the Windows x64 analyzer (../windows) sets "X64".
ARCHITECTURE = "Arm64"
SDK = "2.0.1+07e62f72500cc0923a2bab940653c8c1116fa286"
APP = "ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests"
TEST_CLASS = APP + ".Sdk2LifecycleSoakTests."
@@ -168,8 +170,8 @@ def validate_rows(rows, revision):
and type(start["requestedSeconds"]) is int and start.get("minimumFullSeconds") == SECONDS
and start.get("acceptance") is False, "Wrong fixed-duration diagnostic mode")
require(start.get("build") == revision and re.fullmatch(r"[0-9a-f]{40}", revision), "Wrong source revision")
require(start.get("runtime") == RUNTIME and start.get("architecture") == "Arm64" and start.get("sdk") == SDK,
"Wrong Linux ARM64 runtime or SDK version")
require(start.get("runtime") == RUNTIME and start.get("architecture") == ARCHITECTURE and start.get("sdk") == SDK,
"Wrong runtime, architecture or SDK version")
count(start.get("processId"), "test process ID", 1)
seconds(start.get("startupSeconds"), "startup duration", maximum=SETTINGS["startupTimeoutSeconds"])
limits = start.get("limits")
@@ -0,0 +1,39 @@
# Windows x64 late-slope soak (lmxopcua#799 acceptance)
Owner scope 2026-11-12: #799 acceptance covers the deployment platforms, Windows x64 and Linux. This folder runs the
**same canonical test** as the Linux late-slope run (`Sdk2LifecycleSoakTests.Repeated_lifecycles_return_resources_to_a_steady_baseline`,
`OTOPCUA_SDK2_SOAK_MODE=late-slope`, 7,200 s) on Windows x64 with the protocol unchanged: cycle-3 cold baseline,
+64 MiB retained managed / +128 MiB RSS / +32 descriptors, first-rejection latch with every canonical cycle and
restart continuing, ownership/logical/operation/shutdown failures fatal, certificate leak tracking on, the same
startup/cycle/restart/shutdown bounds, and the same fixed windows and least-squares report (`analyze.py` imports
`../late-slope/analyze.py` and only switches its expected architecture to `X64`).
## Windows adaptations
| Linux element | Windows equivalent | Why |
|---|---|---|
| RSS = `Process.WorkingSet64` (= `VmRSS`) | **Working set**, the same `Process.WorkingSet64` call, unchanged test code | Resident physical pages of the process, the direct analog of Linux RSS. Private bytes (commit charge) counts committed-but-not-resident memory and pagefile-backed pages, so it is not comparable to RSS; the launcher records it externally every 30 s as a descriptive series only. |
| Descriptors = entries of `/proc/self/fd` | **Every handle in the process handle table** (`NtQuerySystemInformation(SystemExtendedHandleInformation)`, own PID) | Windows has no fd table; the handle table is the per-process kernel-resource table. It is broader than Linux fds (threads, events, keys, sections), so the unchanged +32 cap is if anything stricter. |
| PKI descriptors = fds whose link target lies under a PKI root | Disk-file/directory handles (`GetFileType = DISK`) whose `GetFinalPathNameByHandle` path lies under a root; roots canonicalised the same way, compared case-insensitively | Final paths resolve symlinks, junctions, 8.3 names and casing (the Windows analog of the macOS `/var`→`/private/var` lesson). |
| Probe bound: lsof child 5 s | Enumeration on a dedicated thread, 5 s bound; a blocked name query is a `TimeoutException`, never a favourable zero | Name queries can block on synchronous handles; only disk handles are name-queried. |
| Descriptor controls (symlink alias open/close; probe timeout reaps `/bin/sh`) | Same two tests, Windows enabled: `Directory.CreateSymbolicLink` alias (Developer Mode / elevated); the timeout control runs `cmd.exe` holding both pipes open behind a 30 s `ping` | Must execute 2/2 before the canonical test starts, as in `container.sh`. |
| Container: read-only root, `--network none`, private `/task` | Fresh private task directory (inheritance removed; user + SYSTEM) holding `TEMP`/`TMP`, CLI home, results and fixture PKI; loopback-only fixture; machine otherwise idle | No Windows container runtime on the deployment-class host. |
| `/proc/<pid>/exe` + mapped `libcoreclr.so` check | Target PID's image must be the bundle `…IntegrationTests.exe` and `coreclr.dll` must be mapped from the bundle with the receipt hash | Same proof that the self-contained 10.0.14 runtime is the one measured. |
| Container outer deadline 7,440 s | Launcher kills only the task-owned process tree at 7,440 s | Same bound. |
| — | External sampler: working set, private bytes, handle count, threads every 30 s; MxAccessGw readiness every 5 min | Descriptive diagnostics and proof the co-hosted service stayed healthy. Not gated. |
`LateSlopeRun.ValidateRequest` accepts Linux ARM64 or Windows x64 (owner scope); the 7,200 s duration, runtime
10.0.14 and leak-tracking requirements are unchanged.
## Run
```bash
python3 scripts/validation/opcua-lifecycle-memory/windows/prepare.py --source-revision "$rev" --output "$bundle"
# copy the bundle to the Windows host (COPYFILE_DISABLE=1 tar on macOS), then on the host:
powershell -NoProfile -ExecutionPolicy Bypass -File "$bundle\tools\launch.ps1" -Bundle "$bundle" -Output "$run" -SourceRevision "$rev"
# copy the run back, then:
python3 "$bundle/tools/analyze.py" --run "$run" --bundle-receipt "$bundle/receipt.json" --source-revision "$rev"
```
Launch the PowerShell launcher detached from any SSH session (e.g. `Win32_Process.Create`), because Win32-OpenSSH
terminates a session's job object on disconnect.
@@ -0,0 +1,72 @@
#!/usr/bin/env python3
"""Validate a recorded Windows x64 late-slope run (lmxopcua#799) with the Linux analyzer's own row/window logic.
The row ledger, cold cycle-3 caps, first-rejection check, fixed windows and least-squares slopes come unchanged from
../late-slope/analyze.py (ARCHITECTURE switched to X64). Only the run-envelope checks differ: there is no container, so
the launcher's PowerShell receipts (run-receipt.json, target-runtime.txt) take the place of the Docker receipts.
"""
import argparse
import json
from pathlib import Path
import re
import sys
sys.dont_write_bytecode = True
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "late-slope"))
import analyze as core # noqa: E402
core.ARCHITECTURE = "X64"
APP = core.APP
def read_text(path):
return path.read_text(encoding="utf-8-sig")
def analyze(run, bundle_receipt, source_revision):
bundle = json.loads(read_text(bundle_receipt))
core.require(bundle["sourceRevision"] == source_revision and bundle["rid"] == "win-x64"
and bundle["runtimeVersion"] == core.RUNTIME and bundle["limits"] == core.BUDGETS, "Wrong Windows bundle")
rows = [json.loads(line) for line in read_text(run / "resources.jsonl").splitlines() if line.strip()]
start, samples, completions, rejection, result = core.validate_rows(iter(rows), source_revision)
mapping = dict(line.split("=", 1) for line in read_text(run / "target-runtime.txt").splitlines() if "=" in line)
core.require(mapping.get("targetPid") == str(start["processId"])
and mapping.get("actualExecutable", "").lower().endswith(f"\\app\\{APP}.exe".lower())
and mapping.get("mappedCoreClr", "").lower().endswith("\\app\\coreclr.dll")
and mapping.get("coreClrSha256") == bundle["coreClrSha256"],
"Published Windows apphost or mapped CLR was not verified for the recorded PID")
test_exit = int(read_text(run / "test-exit-status.txt").strip())
core.require((test_exit != 0) == bool(rejection), "Original test exit was not preserved")
controls = core.check_trx(run / "results/descriptor-controls.trx", core.DESCRIPTOR_TESTS)
workload = core.check_trx(run / "results/late-slope.trx", {core.TEST_NAME}, rejection and rejection["assertionMessage"])
receipt = json.loads(read_text(run / "run-receipt.json"))
core.require(receipt["targetVerified"] is True and receipt["testExitStatus"] == test_exit
and receipt["leftoverTaskProcesses"] == 0, "Launcher did not verify the target or left processes behind")
wall = receipt["wallSeconds"]
core.require(start["startupSeconds"] + result["cycleSeconds"] + result["shutdownSeconds"] <= wall,
"Startup, cycle and shutdown phases exceed measured wall time")
return {"protocol": core.PROTOCOL, "platform": "windows-x64", "sourceRevision": source_revision,
"bundleCoreClrSha256": bundle["coreClrSha256"], "diagnosticOutcome": result, "firstColdRejection": rejection,
"descriptorTests": controls, "experimentTests": workload, "testExitStatus": test_exit, "wallSeconds": wall,
**core.describe(samples, completions)}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--run", type=Path, required=True)
parser.add_argument("--bundle-receipt", type=Path, required=True)
parser.add_argument("--source-revision", required=True)
options = parser.parse_args()
core.require(re.fullmatch(r"[0-9a-f]{40}", options.source_revision), "Invalid revision")
try:
print(json.dumps(analyze(options.run.resolve(strict=True), options.bundle_receipt, options.source_revision),
indent=2, allow_nan=False))
except (OSError, ValueError, KeyError, TypeError) as error:
print(f"Incomplete Windows late-slope evidence: {error}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,157 @@
<#
.SYNOPSIS
Windows counterpart of ../late-slope/container.sh + launch.py for the lmxopcua#799 acceptance soak.
.DESCRIPTION
Runs the immutable self-contained win-x64 bundle through the installed `dotnet vstest` controller, exactly as the
Linux container runs it: the two descriptor controls must pass first, then the single canonical test
Repeated_lifecycles_return_resources_to_a_steady_baseline in late-slope mode (7,200 s, unchanged cycle-3 caps,
first-rejection latch, all canonical operations, independent shutdown checks).
Windows adaptations (documented in README.md):
- No container. Isolation is a fresh private task directory (inheritance removed, current user + SYSTEM only)
that holds TEMP/TMP, the CLI home, results and fixture PKI. The machine is otherwise idle.
- The target process is verified by PID: its image must be the bundle apphost and coreclr.dll must be mapped from
the bundle with the receipt's SHA-256 (the /proc/<pid>/exe + maps check on Linux).
- An external read-only sampler records the target's WorkingSet64 / PrivateMemorySize64 / HandleCount every 30 s
(descriptive only; the in-process quiescent samples remain the gated values), plus MxAccessGw readiness.
- The 7,440 s outer deadline kills only the task-owned process tree.
#>
param(
[Parameter(Mandatory)] [string] $Bundle,
[Parameter(Mandatory)] [string] $Output,
[Parameter(Mandatory)] [string] $SourceRevision,
[ValidateSet('late-slope', 'diagnostic')] [string] $Mode = 'late-slope',
[int] $Seconds = 7200
)
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version 3
$app = 'ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests'
$testClass = "$app.Sdk2LifecycleSoakTests"
$outerDeadline = 7440
if ($SourceRevision -notmatch '^[0-9a-f]{40}$') { throw 'Invalid source revision' }
if ($Mode -eq 'late-slope' -and $Seconds -ne 7200) { throw 'late-slope is fixed at 7200 s' }
$Bundle = (Resolve-Path $Bundle).Path
if (Test-Path $Output) { throw 'Use a fresh output directory' }
New-Item -ItemType Directory -Path $Output | Out-Null
$Output = (Resolve-Path $Output).Path
foreach ($d in 'tmp', 'cli', 'data', 'results') { New-Item -ItemType Directory -Path (Join-Path $Output $d) | Out-Null }
# Private task directory: no inherited ACEs, only the running user and SYSTEM.
& icacls $Output /inheritance:r /grant:r "$($env:USERNAME):(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' | Out-Null
$started = Get-Date
function Write-Json($path, $value) { $value | ConvertTo-Json -Depth 6 | Set-Content -Encoding UTF8 -Path $path }
# 1. Re-verify every bundle file against the prepared manifest before anything runs.
$bad = 0; $count = 0
foreach ($line in Get-Content (Join-Path $Bundle 'manifest.sha256')) {
$sha, $name = $line -split ' ', 2
$file = Join-Path $Bundle ($name -replace '/', '\')
$count++
if ((Get-FileHash -Algorithm SHA256 -LiteralPath $file).Hash.ToLowerInvariant() -ne $sha) { $bad++ }
}
if ($bad -ne 0 -or $count -lt 900) { throw "Bundle verification failed: $bad mismatches of $count" }
$appDir = Join-Path $Bundle 'app'
$coreClr = Join-Path $appDir 'coreclr.dll'
$coreClrSha = (Get-FileHash -Algorithm SHA256 $coreClr).Hash.ToLowerInvariant()
Write-Json (Join-Path $Output 'launch.json') ([ordered]@{
kind = 'launch'; protocol = 'sdk2-late-slope-v1'; platform = 'windows'; mode = $Mode; requestedSeconds = $Seconds
sourceRevision = $SourceRevision; bundlePath = $Bundle; outputPath = $Output; verifiedBundleFiles = $count
coreClrSha256 = $coreClrSha; outerDeadlineSeconds = $outerDeadline; startedUtc = $started.ToUniversalTime().ToString('o')
host = $env:COMPUTERNAME; os = [Environment]::OSVersion.VersionString
logicalProcessors = [Environment]::ProcessorCount
physicalMemoryBytes = (Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory })
# 2. Same environment as container.sh.
$env:TEMP = Join-Path $Output 'tmp'; $env:TMP = $env:TEMP
$env:DOTNET_CLI_HOME = Join-Path $Output 'cli'
$env:OTOPCUA_SDK2_SOAK = '1'; $env:OTOPCUA_SDK2_SOAK_MODE = $Mode; $env:OTOPCUA_SDK2_SOAK_SECONDS = "$Seconds"
$env:OTOPCUA_SDK2_SOAK_OUTPUT = Join-Path $Output 'resources.jsonl'; $env:OPCUA_CERTIFICATE_LEAK_TRACKING = '1'
$env:DOTNET_CLI_TELEMETRY_OPTOUT = '1'; $env:DOTNET_NOLOGO = '1'; $env:OTOPCUA_SOAK_BUILD_SHA = $SourceRevision
& dotnet --info > (Join-Path $Output 'controller-runtime.txt')
$dll = Join-Path $appDir "$app.dll"
$results = Join-Path $Output 'results'
# 3. Descriptor controls must actually execute and pass before the long test.
& dotnet vstest $dll "--TestCaseFilter:FullyQualifiedName~$testClass.Descriptor_probe_" "--ResultsDirectory:$results" `
'--Logger:trx;LogFileName=descriptor-controls.trx' > (Join-Path $Output 'descriptor-controls.log') 2>&1
[xml] $trx = Get-Content (Join-Path $results 'descriptor-controls.trx')
$c = $trx.TestRun.ResultSummary.Counters
if ($c.total -ne '2' -or $c.executed -ne '2' -or $c.passed -ne '2' -or $c.failed -ne '0') {
'controls-failed' | Set-Content (Join-Path $Output 'launcher-outcome.txt'); exit 2
}
# 4. The canonical test, one process, output captured.
$psi = New-Object System.Diagnostics.ProcessStartInfo 'dotnet'
$psi.Arguments = "vstest `"$dll`" `"--TestCaseFilter:FullyQualifiedName=$testClass.Repeated_lifecycles_return_resources_to_a_steady_baseline`" `"--ResultsDirectory:$results`" `"--Logger:trx;LogFileName=late-slope.trx`""
$psi.UseShellExecute = $false; $psi.RedirectStandardOutput = $true; $psi.RedirectStandardError = $true
$vstest = [System.Diagnostics.Process]::Start($psi)
$log = [System.IO.StreamWriter]::new((Join-Path $Output 'test.log'))
$outTask = $vstest.StandardOutput.ReadToEndAsync(); $errTask = $vstest.StandardError.ReadToEndAsync()
# 5. Verify the actual test process within 150 s: bundle apphost image + coreclr mapped from the bundle.
$resources = Join-Path $Output 'resources.jsonl'
$targetPid = $null; $deadline = (Get-Date).AddSeconds(150)
while ((Get-Date) -lt $deadline -and -not $vstest.HasExited) {
if ((Test-Path $resources) -and (Get-Item $resources).Length -gt 0) {
$fs = [System.IO.File]::Open($resources, 'Open', 'Read', 'ReadWrite')
try { $header = ([System.IO.StreamReader]::new($fs)).ReadLine() } finally { $fs.Dispose() }
if ($header -and $header.Contains('"kind":"start"')) {
$targetPid = [int](($header | ConvertFrom-Json).processId); break
}
}
Start-Sleep -Milliseconds 200
}
$verified = $false
if ($targetPid) {
$target = Get-Process -Id $targetPid
$image = $target.Path
$mapped = @($target.Modules | Where-Object { $_.ModuleName -ieq 'coreclr.dll' } | ForEach-Object FileName)
$mappedSha = if ($mapped.Count -eq 1) { (Get-FileHash -Algorithm SHA256 $mapped[0]).Hash.ToLowerInvariant() } else { '' }
$verified = ($image -ieq (Join-Path $appDir "$app.exe")) -and $mapped.Count -eq 1 -and ($mapped[0] -ieq $coreClr) -and ($mappedSha -eq $coreClrSha)
"targetPid=$targetPid`r`nactualExecutable=$image`r`nmappedCoreClr=$($mapped -join ';')`r`ncoreClrSha256=$mappedSha" |
Set-Content -Encoding ASCII (Join-Path $Output 'target-runtime.txt')
}
if (-not $verified) {
try { & taskkill /PID $vstest.Id /T /F | Out-Null } catch { }
}
# 6. External read-only sampler + outer deadline.
$sampler = Join-Path $Output 'external-samples.jsonl'
$nextHealth = Get-Date
while (-not $vstest.WaitForExit(30000)) {
$elapsed = ((Get-Date) - $started).TotalSeconds
$row = [ordered]@{ utc = (Get-Date).ToUniversalTime().ToString('o'); launcherElapsedSeconds = [math]::Round($elapsed, 1) }
try {
$p = Get-Process -Id $targetPid -ErrorAction Stop
$row.workingSet64 = $p.WorkingSet64; $row.privateMemorySize64 = $p.PrivateMemorySize64
$row.pagedMemorySize64 = $p.PagedMemorySize64; $row.peakWorkingSet64 = $p.PeakWorkingSet64
$row.handleCount = $p.HandleCount; $row.threads = $p.Threads.Count
} catch { $row.targetGone = $true }
if ((Get-Date) -ge $nextHealth) {
try { $row.mxAccessGwReady = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://localhost:5130/health/ready').StatusCode }
catch { $row.mxAccessGwReady = "error: $($_.Exception.Message)" }
$row.mxAccessGwService = (Get-Service MxAccessGw).Status.ToString()
$nextHealth = (Get-Date).AddMinutes(5)
}
($row | ConvertTo-Json -Compress) | Add-Content -Encoding UTF8 $sampler
if ($elapsed -gt $outerDeadline) {
& taskkill /PID $vstest.Id /T /F | Out-Null
'outer-deadline' | Set-Content (Join-Path $Output 'launcher-outcome.txt')
}
}
$log.Write($outTask.Result); $log.Write($errTask.Result); $log.Dispose()
$status = $vstest.ExitCode
"$status" | Set-Content -Encoding ASCII (Join-Path $Output 'test-exit-status.txt')
$wall = ((Get-Date) - $started).TotalSeconds
$evidence = [ordered]@{}
foreach ($f in Get-ChildItem -LiteralPath $Output -File -Recurse | Where-Object { $_.FullName -notlike "$Output\tmp\*" -and $_.FullName -notlike "$Output\cli\*" -and $_.FullName -notlike "$Output\data\*" }) {
$evidence[$f.FullName.Substring($Output.Length + 1).Replace('\', '/')] = (Get-FileHash -Algorithm SHA256 $f.FullName).Hash.ToLowerInvariant()
}
Write-Json (Join-Path $Output 'run-receipt.json') ([ordered]@{
kind = 'run'; platform = 'windows'; sourceRevision = $SourceRevision; mode = $Mode; testExitStatus = $status
targetVerified = $verified; wallSeconds = $wall; finishedUtc = (Get-Date).ToUniversalTime().ToString('o')
leftoverTaskProcesses = @(Get-CimInstance Win32_Process | Where-Object { $_.ExecutablePath -and $_.ExecutablePath.StartsWith($Bundle, 'OrdinalIgnoreCase') }).Count
evidenceSha256 = $evidence })
'done' | Add-Content (Join-Path $Output 'launcher-outcome.txt')
exit $status
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
"""Publish and record a fresh immutable Windows x64 bundle of the canonical soak (lmxopcua#799) without running it.
The Windows counterpart of ../late-slope/prepare.py: same committed-source checks, same self-contained .NET 10.0.14
runtime, same exact SDK 2.0.1 / HistorianGateway 1.0.0 graph, same per-file SHA-256 manifest and managed MVIDs. Only
the RID (win-x64), the executable format check (PE32+ AMD64) and the absence of a Docker image differ.
"""
import argparse
from datetime import datetime, timezone
import json
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
sys.dont_write_bytecode = True
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parent / "late-slope"))
from analyze import APP, RUNTIME, digest, read_object, require # noqa: E402
from bundle import CANONICAL_SOURCES, GATEWAY_PACKAGES, SDK_PACKAGES, check_modules, manifest # noqa: E402
import prepare as linux_prepare # noqa: E402
REPO = HERE.parents[3]
PROJECT = REPO / "tests/Server" / APP / (APP + ".csproj")
RID = "win-x64"
TOOLS = ("launch.ps1", "analyze.py", "README.md")
WINDOWS_SOURCES = (f"tests/Server/{APP}/Sdk2LifecycleWindowsDescriptors.cs", f"tests/Server/{APP}/ServerEventRig.cs")
def check_pe_amd64(path):
data = path.read_bytes()[:4096]
require(data[:2] == b"MZ", f"{path.name} is not a PE image")
offset = int.from_bytes(data[0x3C:0x40], "little")
require(data[offset:offset + 4] == b"PE\0\0" and data[offset + 4:offset + 6] == b"\x64\x86", f"{path.name} is not AMD64")
def check_application(app):
check_pe_amd64(app / f"{APP}.exe")
check_pe_amd64(app / "coreclr.dll")
options = read_object(app / f"{APP}.runtimeconfig.json")["runtimeOptions"]
frameworks = options["includedFrameworks"]
require(options["tfm"] == "net10.0" and "framework" not in options and "frameworks" not in options
and any(value["name"] == "Microsoft.NETCore.App" for value in frameworks)
and all(value["version"] == RUNTIME for value in frameworks), "Wrong self-contained application runtime")
dependencies = read_object(app / f"{APP}.deps.json")
require(dependencies["runtimeTarget"]["name"] == f".NETCoreApp,Version=v10.0/{RID}", "Wrong application RID")
libraries = dependencies["libraries"]
sdk = {key for key in libraries if key.startswith("OPCFoundation.")}
gateway = {key for key in libraries if key.startswith("ZB.MOM.WW.HistorianGateway.")}
require(sdk == SDK_PACKAGES and gateway == GATEWAY_PACKAGES, "Expected exact SDK 2.0.1 and HistorianGateway 1.0.0 graph")
return sorted(sdk), sorted(gateway)
def committed_sources(revision):
sources = linux_prepare.check_source(revision)
for name in (*WINDOWS_SOURCES, *(f"scripts/validation/opcua-lifecycle-memory/windows/{tool}" for tool in TOOLS),
"scripts/validation/opcua-lifecycle-memory/windows/prepare.py"):
sources[name] = digest(REPO / name)
return sources
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--source-revision", required=True)
options = parser.parse_args()
sources = committed_sources(options.source_revision)
root = options.output.resolve()
root.mkdir(mode=0o700, parents=True, exist_ok=False)
with tempfile.TemporaryDirectory(prefix="opcua-windows-soak-publish-") as scratch:
build = Path(scratch)
command = ["dotnet", "publish", str(PROJECT), "-c", "Release", "-r", RID,
"--self-contained", "true", f"-p:RuntimeFrameworkVersion={RUNTIME}",
"-p:PublishSingleFile=false", "-p:PublishTrimmed=false", "-p:UseAppHost=true",
"--artifacts-path", str(build / "artifacts"), "-o", str(root / "app"), "-m:2", "--nologo"]
with (root / "publish.log").open("w", encoding="utf-8") as log:
subprocess.run(command, cwd=REPO, stdout=log, stderr=subprocess.STDOUT, check=True, timeout=900)
sdk, gateway = check_application(root / "app")
modules = linux_prepare.managed_modules(build, root)
tools = root / "tools"
tools.mkdir()
for name in TOOLS:
shutil.copy2(HERE / name, tools / name)
require(committed_sources(options.source_revision) == sources, "Committed inputs changed during publication")
files = manifest(root)
check_modules(modules, files)
receipt = {"kind": "windows-bundle", "protocol": "sdk2-late-slope-v1", "platform": "windows", "rid": RID,
"runtimeVersion": RUNTIME, "selfContained": True, "mode": "late-slope", "requestedSeconds": 7200,
"limits": {"fdGrowth": 32, "retainedManagedGrowthBytes": 64 << 20, "rssGrowthBytes": 128 << 20},
"certificateLeakTracking": True, "sourceRevision": options.source_revision, "sourceFiles": sources,
"preparedUtc": datetime.now(timezone.utc).isoformat(), "publishCommand": command,
"sdkLibraries": sdk, "gatewayLibraries": gateway, "managedModules": modules,
"coreClrSha256": files["app/coreclr.dll"], "files": files}
path = root / "receipt.json"
path.write_text(json.dumps(receipt, indent=2, allow_nan=False) + "\n", encoding="utf-8")
# A flat manifest the Windows launcher re-verifies on the target before launch.
(root / "manifest.sha256").write_text("".join(f"{sha} {name}\n" for name, sha in files.items()), encoding="utf-8")
print(json.dumps({"bundle": str(root), "sourceRevision": options.source_revision, "receiptSha256": digest(path),
"files": len(files), "workloadLaunched": False}, indent=2))
if __name__ == "__main__":
main()
@@ -32,8 +32,9 @@ public sealed partial class Sdk2LifecycleSoakTests
internal static void ValidateRequest(int seconds)
{
seconds.ShouldBe(Duration, "late-slope diagnostic duration is fixed at 7200 seconds");
OperatingSystem.IsLinux().ShouldBeTrue("this diagnostic is qualified only for Linux");
RuntimeInformation.ProcessArchitecture.ShouldBe(Architecture.Arm64);
// lmxopcua#799 owner scope 2026-11-12: the deployment platforms, Linux (qualified on ARM64) and Windows x64.
(OperatingSystem.IsLinux() || OperatingSystem.IsWindows()).ShouldBeTrue("this run is qualified only for Linux and Windows");
RuntimeInformation.ProcessArchitecture.ShouldBe(OperatingSystem.IsWindows() ? Architecture.X64 : Architecture.Arm64);
Environment.Version.ToString().ShouldBe("10.0.14");
Environment.GetEnvironmentVariable("OPCUA_CERTIFICATE_LEAK_TRACKING").ShouldBe("1");
string.IsNullOrEmpty(Environment.GetEnvironmentVariable("OTOPCUA_SDK2_SATURATION_OBSERVER")).ShouldBeTrue(
@@ -312,6 +312,12 @@ public sealed partial class Sdk2LifecycleSoakTests
.Select(i => new HistoricalEvent($"soak-{i}", "source", Epoch.AddSeconds(i), Epoch, "soak", 500)).ToList());
rig.Nm.NodeWriteGateway = new AcceptWrites();
MaterialiseValues(rig);
// lmxopcua#799: since #808 an unplaced UNS folder is denied under enforcement. Place the soak's own equipment
// folder (root folder: id chain is itself; ACL path by display name, as a deployed generation does).
rig.AdditionalFolderScopes ??= new Dictionary<string, UnsFolderScope>
{
["SoakEquipment"] = new(new NodeAclPath([NodeAclRealms.Uns, "Equipment"]), ["SoakEquipment"]),
};
rig.Enforce(true, [ServerEventRig.Key()], Grant());
}
@@ -587,9 +593,9 @@ public sealed partial class Sdk2LifecycleSoakTests
process.Threads.Count);
}
public static bool DescriptorProbeSupported => OperatingSystem.IsLinux() || OperatingSystem.IsMacOS();
public static bool DescriptorProbeSupported => OperatingSystem.IsLinux() || OperatingSystem.IsMacOS() || OperatingSystem.IsWindows();
[Fact(SkipUnless = nameof(DescriptorProbeSupported), Skip = "Descriptor probes require Linux procfs or macOS lsof.")]
[Fact(SkipUnless = nameof(DescriptorProbeSupported), Skip = "Descriptor probes require Linux procfs, macOS lsof or the Windows handle table.")]
public async Task Descriptor_probe_observes_an_open_PKI_file_through_a_symlink_alias_and_its_close()
{
var ct = TestContext.Current.CancellationToken;
@@ -627,17 +633,24 @@ public sealed partial class Sdk2LifecycleSoakTests
}
}
[Fact(SkipUnless = nameof(DescriptorProbeSupported), Skip = "The bounded probe control uses a task-owned Unix process.")]
[Fact(SkipUnless = nameof(DescriptorProbeSupported), Skip = "The bounded probe control uses a task-owned shell process.")]
public async Task Descriptor_probe_times_out_and_reaps_a_process_that_keeps_both_streams_open()
{
var ct = TestContext.Current.CancellationToken;
using var process = new Process
{
StartInfo = new ProcessStartInfo("/bin/sh")
{
RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false,
ArgumentList = { "-c", "printf control; printf control >&2; exec /bin/sleep 30" },
},
// Windows (lmxopcua#799): cmd keeps both redirected pipes open while its child ping runs for ~30 s.
StartInfo = OperatingSystem.IsWindows()
? new ProcessStartInfo("cmd.exe")
{
RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false,
ArgumentList = { "/d", "/c", "echo control& echo control 1>&2& ping -n 31 127.0.0.1 >nul" },
}
: new ProcessStartInfo("/bin/sh")
{
RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false,
ArgumentList = { "-c", "printf control; printf control >&2; exec /bin/sleep 30" },
},
};
var elapsed = Stopwatch.StartNew();
await Should.ThrowAsync<TimeoutException>(() => ReadDescriptorProbeAsync(process, TimeSpan.FromMilliseconds(250), ct));
@@ -649,6 +662,7 @@ public sealed partial class Sdk2LifecycleSoakTests
private static async Task<(int Total, int Pki)> DescriptorCountsAsync(CancellationToken ct, params string[] roots)
{
ct.ThrowIfCancellationRequested();
if (OperatingSystem.IsWindows()) return await WindowsDescriptorCountsAsync(ct, roots);
var canonicalRoots = roots.Select(root => CanonicalPath(root)).Distinct(StringComparer.Ordinal).ToArray();
if (OperatingSystem.IsLinux())
{
@@ -678,7 +692,7 @@ public sealed partial class Sdk2LifecycleSoakTests
}
return (total, pki);
}
throw new PlatformNotSupportedException("This soak requires Linux procfs or macOS lsof for actual descriptor evidence.");
throw new PlatformNotSupportedException("This soak requires Linux procfs, macOS lsof or the Windows handle table for actual descriptor evidence.");
}
private static async Task<string> ReadDescriptorProbeAsync(Process process, TimeSpan timeout, CancellationToken ct)
@@ -0,0 +1,143 @@
using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Runtime.Versioning;
using Microsoft.Win32.SafeHandles;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
/// <summary>
/// Windows port of the soak's descriptor observer (lmxopcua#799 acceptance). Windows has no fd table: the process
/// handle table is the equivalent kernel-resource table, so <c>Total</c> is every handle this process owns (the
/// whole-process figure, like Linux <c>/proc/self/fd</c>) and <c>Pki</c> is the open disk-file/directory handles whose
/// final path lies under a PKI root. Paths come from <c>GetFinalPathNameByHandle</c> (symlinks, junctions, 8.3 names
/// and casing resolved, the Windows analog of the macOS /var→/private/var canonicalisation), and the roots are
/// canonicalised the same way, compared case-insensitively as NTFS does. The probe is in-process (no child process
/// like macOS lsof) and runs on a dedicated thread under the same 5-second bound as the lsof probe: a name query that
/// blocks is a timeout, never a favourable zero.
/// </summary>
public sealed partial class Sdk2LifecycleSoakTests
{
[SupportedOSPlatform("windows")]
private static Task<(int Total, int Pki)> WindowsDescriptorCountsAsync(CancellationToken ct, IReadOnlyList<string> roots)
{
var result = new TaskCompletionSource<(int Total, int Pki)>(TaskCreationOptions.RunContinuationsAsynchronously);
var thread = new Thread(() =>
{
try { result.TrySetResult(WindowsHandleCounts(roots)); }
catch (Exception error) { result.TrySetException(error); }
}) { IsBackground = true, Name = "soak-descriptor-probe" };
thread.Start();
return BoundAsync();
async Task<(int Total, int Pki)> BoundAsync()
{
try { return await result.Task.WaitAsync(TimeSpan.FromSeconds(5), ct).ConfigureAwait(false); }
catch (TimeoutException) { throw new TimeoutException("Descriptor probe exceeded its handle-enumeration deadline."); }
}
}
[SupportedOSPlatform("windows")]
private static (int Total, int Pki) WindowsHandleCounts(IReadOnlyList<string> roots)
{
// Canonicalise the roots first and close those directory handles before the snapshot, so they are not counted.
var canonicalRoots = roots.Select(WindowsCanonicalRoot).Distinct(StringComparer.OrdinalIgnoreCase).ToArray();
var self = (nuint)Environment.ProcessId;
var length = 1 << 20;
while (true)
{
var buffer = Marshal.AllocHGlobal(length);
try
{
var status = NtQuerySystemInformation(SystemExtendedHandleInformation, buffer, length, out var needed);
if (status == StatusInfoLengthMismatch)
{
length = Math.Max(length * 2, needed + (1 << 16));
continue;
}
if (status != 0) throw new Win32Exception(RtlNtStatusToDosError(status), $"NtQuerySystemInformation failed (0x{status:X8}).");
var count = (long)Marshal.ReadIntPtr(buffer);
var entries = buffer + 2 * IntPtr.Size;
var entrySize = Marshal.SizeOf<HandleEntry>();
int total = 0, pki = 0;
var path = new char[32768];
for (long i = 0; i < count; i++)
{
var entry = Marshal.PtrToStructure<HandleEntry>(entries + (nint)(i * entrySize));
if (entry.UniqueProcessId != self) continue;
total++;
var handle = (nint)entry.HandleValue;
// GetFileType is answered by the I/O manager without the file-object lock; only disk files and
// directories go on to the name query (pipes, sockets and consoles cannot hold PKI material).
if (GetFileType(handle) != FileTypeDisk) continue;
var name = FinalPath(handle, path);
if (name is not null && WithinWindowsRoots(name, canonicalRoots)) pki++;
}
return (total, pki);
}
finally { Marshal.FreeHGlobal(buffer); }
}
}
[SupportedOSPlatform("windows")]
private static string WindowsCanonicalRoot(string root)
{
var full = Path.TrimEndingDirectorySeparator(Path.GetFullPath(root));
if (!Directory.Exists(full)) return CanonicalPath(full);
using var directory = CreateFileW(full, 0, FileShareAll, 0, OpenExisting, FileFlagBackupSemantics, 0);
if (directory.IsInvalid) throw new Win32Exception(Marshal.GetLastWin32Error(), "Cannot open a PKI root for canonicalisation.");
return FinalPath(directory.DangerousGetHandle(), new char[32768])
?? throw new IOException("A PKI root has no final path.");
}
private static string? FinalPath(nint handle, char[] buffer)
{
var written = GetFinalPathNameByHandleW(handle, buffer, buffer.Length, 0);
if (written == 0 || written >= buffer.Length) return null; // Closed/reused between snapshot and query, or not a named file.
var name = new string(buffer, 0, (int)written);
if (name.StartsWith(@"\\?\UNC\", StringComparison.Ordinal)) return @"\\" + name[8..];
return name.StartsWith(@"\\?\", StringComparison.Ordinal) ? name[4..] : name;
}
private static bool WithinWindowsRoots(string path, IReadOnlyList<string> roots)
{
path = Path.TrimEndingDirectorySeparator(path);
return roots.Any(root => path.Equals(root, StringComparison.OrdinalIgnoreCase)
|| path.StartsWith(root + Path.DirectorySeparatorChar, StringComparison.OrdinalIgnoreCase));
}
private const int SystemExtendedHandleInformation = 64;
private const int StatusInfoLengthMismatch = unchecked((int)0xC0000004);
private const uint FileTypeDisk = 1;
private const uint FileShareAll = 7;
private const uint OpenExisting = 3;
private const uint FileFlagBackupSemantics = 0x02000000;
[StructLayout(LayoutKind.Sequential)]
private struct HandleEntry
{
public nint Object;
public nuint UniqueProcessId;
public nuint HandleValue;
public uint GrantedAccess;
public ushort CreatorBackTraceIndex;
public ushort ObjectTypeIndex;
public uint HandleAttributes;
public uint Reserved;
}
[DllImport("ntdll.dll")]
private static extern int NtQuerySystemInformation(int informationClass, nint information, int length, out int returnLength);
[DllImport("ntdll.dll")]
private static extern int RtlNtStatusToDosError(int status);
[DllImport("kernel32.dll", SetLastError = true)]
private static extern uint GetFileType(nint handle);
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern uint GetFinalPathNameByHandleW(nint handle, [Out] char[] path, int length, uint flags);
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern SafeFileHandle CreateFileW(string name, uint access, uint share, nint security, uint disposition,
uint flags, nint template);
}
@@ -107,6 +107,14 @@ internal sealed class ServerEventRig : IAsyncDisposable
await BootAsync(ct);
}
/// <summary>
/// Extra UNS folder placements <see cref="Enforce(bool, ApiKey[], IReadOnlyDictionary{AclScopeRef, IReadOnlyList{NodeAclPath}}, IReadOnlyDictionary{string, NodeAclPath}, NodeAcl[])"/>
/// adds to <see cref="FolderScopes"/>, for a test that materialises its own UNS folder. Since #808 an UNS folder
/// the generation does not place is denied under enforcement (fail closed); a deployed generation places every
/// folder it creates, so a fixture folder must be placed too (lmxopcua#799, the same fixture gap as #888).
/// </summary>
public IReadOnlyDictionary<string, UnsFolderScope>? AdditionalFolderScopes { get; set; }
/// <summary>Wires the authorizer over the given keys and grants, enforcing or not.</summary>
public DataPlaneAuthorityState Enforce(bool enabled, ApiKey[] keys, params NodeAcl[] grants) =>
Enforce(enabled, keys, ScopePaths(), grants);
@@ -126,8 +134,10 @@ internal sealed class ServerEventRig : IAsyncDisposable
IReadOnlyDictionary<string, NodeAclPath> equipmentOwnerPaths, params NodeAcl[] grants)
{
var state = new DataPlaneAuthorityState();
var folders = AdditionalFolderScopes is null ? FolderScopes()
: FolderScopes().Concat(AdditionalFolderScopes).ToDictionary(pair => pair.Key, pair => pair.Value);
state.Apply(
Cluster, "rev-1", keys, grants, scopePaths, equipmentOwnerPaths, ScriptedAlarmOwners(), FolderScopes(),
Cluster, "rev-1", keys, grants, scopePaths, equipmentOwnerPaths, ScriptedAlarmOwners(), folders,
TagReferences());
Nm.DataPlaneAuthority = state;
Nm.Authorizer = new DataPlaneAuthorizer(