Files
lmxopcua/docs/plans/evidence/2026-11-12-799-acceptance
Joseph Doherty fea7de0aef
v2-ci / unit-tests (push) Waiting to run
docs(evidence): #799 acceptance runs on master aa60b3e3 - Windows x64 and Linux both fail the unchanged cold RSS cap
Owner scope 2026-11-12 (Windows x64 + Linux, unchanged cycle-3 caps, >=7200 s, SDK 1.5 comparator).

- Linux ARM64 (late-slope, 7200.12 s, 1964 cycles / 491 restarts, shutdown verified, analyzer-validated):
  first rejection RSS at cycle 247 / 900.73 s, 358,682,624 > 358,678,528 (+4 KiB); peak 382.98 MiB vs
  limit 342.06 MiB; late plateau medians 344.4/341.1/341.9 MiB, OLS -1.90 MiB/h. FAIL.
- Windows x64 on windev (same workload, working set as RSS, handle table as descriptors): first rejection RSS
  at cycle 285 / 1130.76 s, 303,505,408 > 302,174,208; peak 319.49 vs limit 288.18 MiB; plateau ~314-315 MiB,
  +0.93 MiB/h. FAIL. Also incomplete: the launcher's wall-clock outer deadline fired early after a +552 s
  w32time step and ended the run at test-phase 7014 s (rejection had already latched); launchers now use a
  Stopwatch.
- SDK 1.5.378.182 comparator (efb9197a publication, Windows, 7201 s): RSS peak +65.8 MiB (never over its cap),
  plateau ~+48 MiB, but the +32 handle cap tripped at cycle 17 (max 674 vs 642).
- Managed flat on both SDK2 runs; growth is the generated-factory JIT burst (+40k/+44k methods) - Linux JIT
  doublemapper ~180 MiB; Windows private bytes grew ~25 MiB of a ~132 MiB working-set rise.
- Retained: b61f3e05 cycle-1 fixture failure (#808 unplaced UNS folder), Windows/SDK1 smoke diagnostics.

No cap raised, no warm baseline, no failed run counted as passing.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 14:51:31 -05:00
..
…

#799 acceptance runs on current master: Windows x64 and Linux both FAIL the unchanged cold cap

Owner scope (2026-11-12, #799 comment 11926): acceptance needs one passing ≥7,200 s run on Windows x64 and one on Linux against current master, with the unchanged cold-baseline caps (+64 MiB retained managed / +128 MiB RSS / +32 descriptors against the cycle-3 sample), plus the matched SDK 1.5.378.182 comparator numbers. macOS is out of scope.

Neither acceptance run passed. Both tripped the RSS cap at the same workload point the earlier cold failures did (the restart ~60–70 generated-factory JIT step). Retained managed memory, descriptors, PKI descriptors and every logical inventory stayed flat on both. No cap was raised, no warm baseline was adopted and no failed run is counted as passing.

Results

Run Source Cycle-3 RSS RSS limit First rejection Peak RSS Late plateau (window medians, MiB) Late slope 1,800–7,200 s Managed / descriptors Verdict
Linux ARM64 acceptance aa60b3e3 214.06 MiB 342.06 MiB RSS, cycle 247, 900.73 s: 358,682,624 > 358,678,528 (+4,096 B) 382.98 MiB (cycle 254) 344.39 / 341.12 / 341.90 −1.90 MiB/h (1,472 samples) 18.93→peak 19.55 MiB; 191→peak 200 (limit 223) FAIL
Windows x64 acceptance aa60b3e3 160.18 MiB 288.18 MiB RSS, cycle 285, 1,130.76 s: 303,505,408 > 302,174,208 (+1.27 MiB) 319.49 MiB (cycle 321) 314.24 / 314.39 / 315.29 (last window partial) +0.93 MiB/h (1,319 samples, to 7,014 s) 19.73→peak 20.27 MiB; 626→peak 631 handles (limit 658) FAIL (also incomplete, see below)
SDK 1.5.378.182 comparator, Windows efb9197a publication 150.36 MiB 278.36 MiB descriptors (handles), cycle 17, 70.27 s: 651 > 642 216.14 MiB (cycle 245) 191.43 / 198.50 / 197.96 +0.17 MiB/h (1,325 samples) 49.21→peak 52.44 MiB; 610→peak 674 handles comparator only

RSS above the limit: Linux in 1,113 of 1,962 post-baseline samples; Windows in 1,489 of 1,771. The plateau sits right at the limit on Linux and about 26 MiB over it on Windows. Neither keeps growing late (Linux slope negative, Windows +0.93 MiB/h), but the protocol has no slope pass rule and the cold cap is binding.

Windows: SDK2 vs SDK1 under the same workload (each against its own cycle-3 sample): SDK2 peak growth +159.3 MiB and plateau ≈ +154 MiB; SDK1 peak growth +65.8 MiB and plateau ≈ +48 MiB. SDK1 never crossed its RSS cap. It did cross the +32 handle cap, in 519 of 1,765 samples, first at cycle 17. SDK2 never crossed the handle cap (max +5). SDK1 retained managed memory is about 50 MiB against SDK2's about 20 MiB, flat on both. The comparator is a standalone executable, not the VSTest testhost, and ran on windev concurrently with the SDK2 run (see below), so compare growth, not absolute RSS.

Diagnostics: what grows (managed vs native)

  • Managed is not the growth. Retained managed bytes were +0.6 MiB (Linux) and +0.5 MiB (Windows) at peak. GC committed was +5.9 MiB on Linux at the rejection.

  • JIT burst at the rejection. Linux, cycle 3 → 247: JIT-compiled methods 35,180 → 75,180 (+40,000), IL bytes +6.6 MB. Windows, cycle 3 → 285: 31,221 → 75,116 (+43,895). Threads and timers were unchanged or lower. This is the generated-factory burst recorded in ../2026-11-10-opcua2/saturation-candidate.md.

  • Linux mapping categories (linux-arm64-aa60b3e3/smaps-categories.json, read-only docker exec cat /proc/214/smaps at about cycle 260, cycle 990 and cycle 1480; path names stripped by summarize.py):

    • JIT doublemapper RSS: 180.7 / 179.1 / 179.1 MiB.
    • Anonymous writable: 144.0 / 143.6 / 169.8 MiB.
    • Managed assembly files: 41.3 / 7.1 / 6.0 MiB.
    • Native heap: 4.5 MiB.

    At its 10-minute checkpoint, the earlier SDK2 Linux subset run had about 65–107 MiB of JIT doublemapper.

  • Windows working-set split (windows-x64-aa60b3e3/external-samples.jsonl.gz, 30 s external sampler): the working set went from about 183 MiB to a 315 MiB plateau (+132 MiB). Private bytes (commit charge) only went from about 77–80 MiB to about 101–106 MiB (max 123 MiB). About 80% of the working-set growth is therefore not private commit. That fits shareable pages, for example .NET's W^X double-mapped JIT code (pagefile-backed sections) and image pages. This is an attribution hint, not a proof. No runtime setting (for example DOTNET_EnableWriteXorExecute) was tried or adopted, and no fix is claimed.

Provenance

  • Source: origin/master aa60b3e34eb9d792b0c4ce78f8dddd73240a59e1 at launch. Its parent is 8ba0e22b; the diff against b61f3e05 is test/harness only (git diff b61f3e05 aa60b3e3 -- src is empty). SDK 2.0.1+07e62f72500cc0923a2bab940653c8c1116fa286, HistorianGateway Client/Contracts 1.0.0, self-contained .NET 10.0.14 on both platforms, OPCUA_CERTIFICATE_LEAK_TRACKING=1, late-slope mode (7,200 s, 30 s cycle / 25 s restart / 120 s startup / 60 s shutdown bounds).
  • Linux: bundle receipt ee4ba62699cbe3229ecafaa93da7003509bb238e1057720b71a43bf4242b5204. Image sha256:b04611ee…f293b (mcr dotnet/sdk:10.0.103-noble arm64), OrbStack context, --network none, read-only root, the unchanged late-slope/{prepare,launch,analyze}.py.
    • Run: 7,200.12 s, 1,964 cycles, 491 restarts. Shutdown verified (0.02 s); container removed.
    • Descriptor controls 2/2 passed. The canonical test failed on the latched cap and the test, container and launcher all exited 1. diagnosticCompleted=true.
    • The official analyzer validated all raw evidence (analysis.json).
    • Raw resources.jsonl SHA-256 51c26871…cef5e5f, TRX 8a23d346…c7da8d0. Every evidence hash is in run-receipt.json.
  • Windows: windev (Windows 10 22H2 19045, 16 vCPU / 32 GB). Bundle receipt d64ff61f3e623caba55b9507d2fcc37f1f40857bc38378edec4e82afb3ab982b, archive 38bdd630…a7ed64e, built by windows/prepare.py, 947 files re-verified on the host.
    • The target PID ran the bundle apphost with the bundle coreclr.dll mapped (1bc65b8d45474b19adf3ca85ce881087aff599ffc72b2c1a323e878227757b65).
    • Descriptor controls 2/2 passed on Windows (symlink-alias positive control, cmd.exe timeout control).
    • MxAccessGw /health/ready returned 200 and the service stayed Running in all 23 five-minute checks. OtOpcUaWonderwareHistorian was not touched.
  • Windows run incomplete — launcher defect, disclosed: the launcher measured its 7,440 s outer deadline with the wall clock. windev's clock was about 9 min slow; at 19:05:32 host time w32time stepped it +552 s. The deadline therefore fired early and killed the run at test-phase 7,014.07 s (cycle 1,774, 443 restarts), before the 7,200 s phase ended.
    • There is no late-slope result row and no workload TRX. The 6,600–7,200 s window covers only 6,603–7,014 s (105 samples).
    • The in-test clocks are monotonic (Stopwatch), so the samples and timings are valid. The verdict does not depend on the missing 186 s: the RSS rejection had already latched at 1,130.76 s.
    • The launchers now use a Stopwatch.
  • Comparator: built by sdk1-comparator/build.py from the RID-specific (win-x64) framework-dependent publications of efb9197a Host and Client.CLI. Product code was not rebuilt.
    • SDK 1.5.378.182+6cca32dd…, self-contained .NET 10.0.14.
    • Comparator sources at 2e9cccbd, hashes in build-provenance.json. Exe 2bd97c5e…, dll e5bbee78….
    • Run: 7,201.35 s, 1,767 cycles, 441 restarts. Shutdown verified.
    • The held closed-session lists were (1 browse, 6 history) on every cycle. That is recorded, not asserted (SDK1 Close semantics, see ../../../../scripts/validation/opcua-lifecycle-memory/sdk1-comparator/README.md).
    • It started about 5 min after the SDK2 Windows run and overlapped it for the rest of the run on the 16-vCPU host. Its launcher was stopped at about 7,220 s wall time so the same clock step could not kill it; the comparator process finished on its own.
  • Linux fixture failure on b61f3e05 (retained, linux-b61f3e05-fixture-failure/): the first launch on the then-current master failed at cycle 1. A UNS monitored item was bad because, since #808, the soak's own unplaced SoakEquipment folder is denied under enforcement. This is the same fixture gap #888 fixed for the SDK matrix.
    • aa60b3e3 places the folder (ServerEventRig.AdditionalFolderScopes). No authorization code, operation, cap or baseline changed.
    • A 60 s macOS diagnostic and a 90 s Windows diagnostic (windows-rehearsal-diagnostic/) qualified the fix and the port. Neither is acceptance evidence.

verdict.json in each run folder is produced by make_verdicts.py from the raw samples (and the Linux analyzer report). A run passes only if the evidence validates, the tests pass, the 7,200 s phase completes, shutdown is verified and no cold-cap rejection latched. Slopes are descriptive only. Raw samples are the gzipped resources.jsonl files. No PKI, credentials or fixture keys are included.