Files
lmxopcua/docs/plans/evidence/2026-11-12-799-acceptance/README.md
T
Joseph Doherty fea7de0aef
v2-ci / unit-tests (push) Waiting to run
docs(evidence): #799 acceptance runs on master aa60b3e3 - Windows x64 and Linux both fail the unchanged cold RSS cap
Owner scope 2026-11-12 (Windows x64 + Linux, unchanged cycle-3 caps, >=7200 s, SDK 1.5 comparator).

- Linux ARM64 (late-slope, 7200.12 s, 1964 cycles / 491 restarts, shutdown verified, analyzer-validated):
  first rejection RSS at cycle 247 / 900.73 s, 358,682,624 > 358,678,528 (+4 KiB); peak 382.98 MiB vs
  limit 342.06 MiB; late plateau medians 344.4/341.1/341.9 MiB, OLS -1.90 MiB/h. FAIL.
- Windows x64 on windev (same workload, working set as RSS, handle table as descriptors): first rejection RSS
  at cycle 285 / 1130.76 s, 303,505,408 > 302,174,208; peak 319.49 vs limit 288.18 MiB; plateau ~314-315 MiB,
  +0.93 MiB/h. FAIL. Also incomplete: the launcher's wall-clock outer deadline fired early after a +552 s
  w32time step and ended the run at test-phase 7014 s (rejection had already latched); launchers now use a
  Stopwatch.
- SDK 1.5.378.182 comparator (efb9197a publication, Windows, 7201 s): RSS peak +65.8 MiB (never over its cap),
  plateau ~+48 MiB, but the +32 handle cap tripped at cycle 17 (max 674 vs 642).
- Managed flat on both SDK2 runs; growth is the generated-factory JIT burst (+40k/+44k methods) - Linux JIT
  doublemapper ~180 MiB; Windows private bytes grew ~25 MiB of a ~132 MiB working-set rise.
- Retained: b61f3e05 cycle-1 fixture failure (#808 unplaced UNS folder), Windows/SDK1 smoke diagnostics.

No cap raised, no warm baseline, no failed run counted as passing.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 14:51:31 -05:00

108 lines
9.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# #799 acceptance runs on current master: Windows x64 and Linux both FAIL the unchanged cold cap
Owner scope (2026-11-12, #799 comment 11926): acceptance needs one passing ≥7,200 s run on **Windows x64** and one
on **Linux** against current master, with the unchanged cold-baseline caps (+64 MiB retained managed / +128 MiB RSS /
+32 descriptors against the cycle-3 sample), plus the matched SDK 1.5.378.182 comparator numbers. macOS is out of scope.
**Neither acceptance run passed.** Both tripped the RSS cap at the same workload point the earlier cold failures
did (the restart ~60–70 generated-factory JIT step). Retained managed memory, descriptors, PKI descriptors and every
logical inventory stayed flat on both. No cap was raised, no warm baseline was adopted and no failed run is counted
as passing.
## Results
| Run | Source | Cycle-3 RSS | RSS limit | First rejection | Peak RSS | Late plateau (window medians, MiB) | Late slope 1,800–7,200 s | Managed / descriptors | Verdict |
|---|---|---:|---:|---|---:|---|---:|---|---|
| **Linux ARM64** acceptance | `aa60b3e3` | 214.06 MiB | 342.06 MiB | **RSS, cycle 247, 900.73 s**: 358,682,624 > 358,678,528 (+4,096 B) | 382.98 MiB (cycle 254) | 344.39 / 341.12 / 341.90 | **−1.90 MiB/h** (1,472 samples) | 18.93→peak 19.55 MiB; 191→peak 200 (limit 223) | **FAIL** |
| **Windows x64** acceptance | `aa60b3e3` | 160.18 MiB | 288.18 MiB | **RSS, cycle 285, 1,130.76 s**: 303,505,408 > 302,174,208 (+1.27 MiB) | 319.49 MiB (cycle 321) | 314.24 / 314.39 / 315.29 (last window partial) | **+0.93 MiB/h** (1,319 samples, to 7,014 s) | 19.73→peak 20.27 MiB; 626→peak 631 handles (limit 658) | **FAIL** (also incomplete, see below) |
| SDK 1.5.378.182 comparator, Windows | `efb9197a` publication | 150.36 MiB | 278.36 MiB | descriptors (handles), cycle 17, 70.27 s: 651 > 642 | 216.14 MiB (cycle 245) | 191.43 / 198.50 / 197.96 | +0.17 MiB/h (1,325 samples) | 49.21→peak 52.44 MiB; 610→peak 674 handles | comparator only |
RSS above the limit: Linux in 1,113 of 1,962 post-baseline samples; Windows in 1,489 of 1,771. The plateau sits
right at the limit on Linux and about 26 MiB over it on Windows. Neither keeps growing late (Linux slope negative,
Windows +0.93 MiB/h), but the protocol has no slope pass rule and the cold cap is binding.
**Windows: SDK2 vs SDK1 under the same workload** (each against its own cycle-3 sample): SDK2 peak growth
+159.3 MiB and plateau ≈ +154 MiB; SDK1 peak growth +65.8 MiB and plateau ≈ +48 MiB. SDK1 never crossed its RSS cap.
It did cross the +32 handle cap, in 519 of 1,765 samples, first at cycle 17. SDK2 never crossed the handle cap (max
+5). SDK1 retained managed memory is about 50 MiB against SDK2's about 20 MiB, flat on both. The comparator is a
standalone executable, not the VSTest testhost, and ran on windev concurrently with the SDK2 run (see below), so
compare growth, not absolute RSS.
## Diagnostics: what grows (managed vs native)
- **Managed is not the growth.** Retained managed bytes were +0.6 MiB (Linux) and +0.5 MiB (Windows) at peak. GC
committed was +5.9 MiB on Linux at the rejection.
- **JIT burst at the rejection.** Linux, cycle 3 → 247: JIT-compiled methods 35,180 → 75,180 (+40,000), IL bytes
+6.6 MB. Windows, cycle 3 → 285: 31,221 → 75,116 (+43,895). Threads and timers were unchanged or lower. This is
the generated-factory burst recorded in `../2026-11-10-opcua2/saturation-candidate.md`.
- **Linux mapping categories** (`linux-arm64-aa60b3e3/smaps-categories.json`, read-only `docker exec cat
/proc/214/smaps` at about cycle 260, cycle 990 and cycle 1480; path names stripped by `summarize.py`):
- JIT `doublemapper` RSS: 180.7 / 179.1 / 179.1 MiB.
- Anonymous writable: 144.0 / 143.6 / 169.8 MiB.
- Managed assembly files: 41.3 / 7.1 / 6.0 MiB.
- Native heap: 4.5 MiB.
At its 10-minute checkpoint, the earlier SDK2 Linux subset run had about 65–107 MiB of JIT doublemapper.
- **Windows working-set split** (`windows-x64-aa60b3e3/external-samples.jsonl.gz`, 30 s external sampler): the working
set went from about 183 MiB to a 315 MiB plateau (+132 MiB). Private bytes (commit charge) only went from about
77–80 MiB to about 101–106 MiB (max 123 MiB). About 80% of the working-set growth is therefore not private commit.
That fits shareable pages, for example .NET's W^X double-mapped JIT code (pagefile-backed sections) and image
pages. This is an attribution hint, not a proof. No runtime setting (for example `DOTNET_EnableWriteXorExecute`)
was tried or adopted, and no fix is claimed.
## Provenance
- **Source:** origin/master `aa60b3e34eb9d792b0c4ce78f8dddd73240a59e1` at launch. Its parent is `8ba0e22b`; the
diff against `b61f3e05` is test/harness only (`git diff b61f3e05 aa60b3e3 -- src` is empty). SDK
`2.0.1+07e62f72500cc0923a2bab940653c8c1116fa286`, HistorianGateway Client/Contracts 1.0.0, self-contained .NET
**10.0.14** on both platforms, `OPCUA_CERTIFICATE_LEAK_TRACKING=1`, late-slope mode (7,200 s, 30 s cycle / 25 s
restart / 120 s startup / 60 s shutdown bounds).
- **Linux:** bundle receipt `ee4ba62699cbe3229ecafaa93da7003509bb238e1057720b71a43bf4242b5204`. Image
`sha256:b04611ee…f293b` (mcr dotnet/sdk:10.0.103-noble arm64), OrbStack context, `--network none`, read-only root,
the unchanged `late-slope/{prepare,launch,analyze}.py`.
- Run: 7,200.12 s, 1,964 cycles, 491 restarts. Shutdown verified (0.02 s); container removed.
- Descriptor controls 2/2 passed. The canonical test failed on the latched cap and the test, container and
launcher all exited 1. `diagnosticCompleted=true`.
- The official analyzer validated all raw evidence (`analysis.json`).
- Raw `resources.jsonl` SHA-256 `51c26871…cef5e5f`, TRX `8a23d346…c7da8d0`. Every evidence hash is in
`run-receipt.json`.
- **Windows:** windev (Windows 10 22H2 19045, 16 vCPU / 32 GB). Bundle receipt
`d64ff61f3e623caba55b9507d2fcc37f1f40857bc38378edec4e82afb3ab982b`, archive `38bdd630…a7ed64e`, built by
`windows/prepare.py`, 947 files re-verified on the host.
- The target PID ran the bundle apphost with the bundle `coreclr.dll` mapped
(`1bc65b8d45474b19adf3ca85ce881087aff599ffc72b2c1a323e878227757b65`).
- Descriptor controls 2/2 passed on Windows (symlink-alias positive control, `cmd.exe` timeout control).
- MxAccessGw `/health/ready` returned 200 and the service stayed Running in all 23 five-minute checks.
OtOpcUaWonderwareHistorian was not touched.
- **Windows run incomplete — launcher defect, disclosed:** the launcher measured its 7,440 s outer deadline with the
wall clock. windev's clock was about 9 min slow; at 19:05:32 host time w32time stepped it +552 s. The deadline
therefore fired early and killed the run at **test-phase 7,014.07 s (cycle 1,774, 443 restarts)**, before the
7,200 s phase ended.
- There is no late-slope result row and no workload TRX. The 6,600–7,200 s window covers only 6,603–7,014 s
(105 samples).
- The in-test clocks are monotonic (Stopwatch), so the samples and timings are valid. The verdict does not depend
on the missing 186 s: the RSS rejection had already latched at 1,130.76 s.
- The launchers now use a Stopwatch.
- **Comparator:** built by `sdk1-comparator/build.py` from the RID-specific (`win-x64`) framework-dependent
publications of `efb9197a` Host and Client.CLI. Product code was not rebuilt.
- SDK `1.5.378.182+6cca32dd…`, self-contained .NET 10.0.14.
- Comparator sources at `2e9cccbd`, hashes in `build-provenance.json`. Exe `2bd97c5e…`, dll `e5bbee78…`.
- Run: 7,201.35 s, 1,767 cycles, 441 restarts. Shutdown verified.
- The held closed-session lists were (1 browse, 6 history) on every cycle. That is recorded, not asserted (SDK1
`Close` semantics, see `../../../../scripts/validation/opcua-lifecycle-memory/sdk1-comparator/README.md`).
- It started about 5 min after the SDK2 Windows run and overlapped it for the rest of the run on the 16-vCPU
host. Its launcher was stopped at about 7,220 s wall time so the same clock step could not kill it; the
comparator process finished on its own.
- **Linux fixture failure on `b61f3e05` (retained, `linux-b61f3e05-fixture-failure/`):** the first launch on the
then-current master failed at cycle 1. A UNS monitored item was bad because, since #808, the soak's own unplaced
`SoakEquipment` folder is denied under enforcement. This is the same fixture gap #888 fixed for the SDK matrix.
- `aa60b3e3` places the folder (`ServerEventRig.AdditionalFolderScopes`). No authorization code, operation, cap
or baseline changed.
- A 60 s macOS diagnostic and a 90 s Windows diagnostic (`windows-rehearsal-diagnostic/`) qualified the fix and the
port. Neither is acceptance evidence.
`verdict.json` in each run folder is produced by `make_verdicts.py` from the raw samples (and the Linux analyzer
report). A run passes only if the evidence validates, the tests pass, the 7,200 s phase completes, shutdown is
verified and no cold-cap rejection latched. Slopes are descriptive only. Raw samples are the gzipped
`resources.jsonl` files. No PKI, credentials or fixture keys are included.