fea7de0aef
v2-ci / unit-tests (push) Waiting to run
Owner scope 2026-11-12 (Windows x64 + Linux, unchanged cycle-3 caps, >=7200 s, SDK 1.5 comparator). - Linux ARM64 (late-slope, 7200.12 s, 1964 cycles / 491 restarts, shutdown verified, analyzer-validated): first rejection RSS at cycle 247 / 900.73 s, 358,682,624 > 358,678,528 (+4 KiB); peak 382.98 MiB vs limit 342.06 MiB; late plateau medians 344.4/341.1/341.9 MiB, OLS -1.90 MiB/h. FAIL. - Windows x64 on windev (same workload, working set as RSS, handle table as descriptors): first rejection RSS at cycle 285 / 1130.76 s, 303,505,408 > 302,174,208; peak 319.49 vs limit 288.18 MiB; plateau ~314-315 MiB, +0.93 MiB/h. FAIL. Also incomplete: the launcher's wall-clock outer deadline fired early after a +552 s w32time step and ended the run at test-phase 7014 s (rejection had already latched); launchers now use a Stopwatch. - SDK 1.5.378.182 comparator (efb9197apublication, Windows, 7201 s): RSS peak +65.8 MiB (never over its cap), plateau ~+48 MiB, but the +32 handle cap tripped at cycle 17 (max 674 vs 642). - Managed flat on both SDK2 runs; growth is the generated-factory JIT burst (+40k/+44k methods) - Linux JIT doublemapper ~180 MiB; Windows private bytes grew ~25 MiB of a ~132 MiB working-set rise. - Retained:b61f3e05cycle-1 fixture failure (#808 unplaced UNS folder), Windows/SDK1 smoke diagnostics. No cap raised, no warm baseline, no failed run counted as passing. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
108 lines
9.0 KiB
Markdown
108 lines
9.0 KiB
Markdown
# #799 acceptance runs on current master: Windows x64 and Linux both FAIL the unchanged cold cap
|
||
|
||
Owner scope (2026-11-12, #799 comment 11926): acceptance needs one passing ≥7,200 s run on **Windows x64** and one
|
||
on **Linux** against current master, with the unchanged cold-baseline caps (+64 MiB retained managed / +128 MiB RSS /
|
||
+32 descriptors against the cycle-3 sample), plus the matched SDK 1.5.378.182 comparator numbers. macOS is out of scope.
|
||
|
||
**Neither acceptance run passed.** Both tripped the RSS cap at the same workload point the earlier cold failures
|
||
did (the restart ~60–70 generated-factory JIT step). Retained managed memory, descriptors, PKI descriptors and every
|
||
logical inventory stayed flat on both. No cap was raised, no warm baseline was adopted and no failed run is counted
|
||
as passing.
|
||
|
||
## Results
|
||
|
||
| Run | Source | Cycle-3 RSS | RSS limit | First rejection | Peak RSS | Late plateau (window medians, MiB) | Late slope 1,800–7,200 s | Managed / descriptors | Verdict |
|
||
|---|---|---:|---:|---|---:|---|---:|---|---|
|
||
| **Linux ARM64** acceptance | `aa60b3e3` | 214.06 MiB | 342.06 MiB | **RSS, cycle 247, 900.73 s**: 358,682,624 > 358,678,528 (+4,096 B) | 382.98 MiB (cycle 254) | 344.39 / 341.12 / 341.90 | **−1.90 MiB/h** (1,472 samples) | 18.93→peak 19.55 MiB; 191→peak 200 (limit 223) | **FAIL** |
|
||
| **Windows x64** acceptance | `aa60b3e3` | 160.18 MiB | 288.18 MiB | **RSS, cycle 285, 1,130.76 s**: 303,505,408 > 302,174,208 (+1.27 MiB) | 319.49 MiB (cycle 321) | 314.24 / 314.39 / 315.29 (last window partial) | **+0.93 MiB/h** (1,319 samples, to 7,014 s) | 19.73→peak 20.27 MiB; 626→peak 631 handles (limit 658) | **FAIL** (also incomplete, see below) |
|
||
| SDK 1.5.378.182 comparator, Windows | `efb9197a` publication | 150.36 MiB | 278.36 MiB | descriptors (handles), cycle 17, 70.27 s: 651 > 642 | 216.14 MiB (cycle 245) | 191.43 / 198.50 / 197.96 | +0.17 MiB/h (1,325 samples) | 49.21→peak 52.44 MiB; 610→peak 674 handles | comparator only |
|
||
|
||
RSS above the limit: Linux in 1,113 of 1,962 post-baseline samples; Windows in 1,489 of 1,771. The plateau sits
|
||
right at the limit on Linux and about 26 MiB over it on Windows. Neither keeps growing late (Linux slope negative,
|
||
Windows +0.93 MiB/h), but the protocol has no slope pass rule and the cold cap is binding.
|
||
|
||
**Windows: SDK2 vs SDK1 under the same workload** (each against its own cycle-3 sample): SDK2 peak growth
|
||
+159.3 MiB and plateau ≈ +154 MiB; SDK1 peak growth +65.8 MiB and plateau ≈ +48 MiB. SDK1 never crossed its RSS cap.
|
||
It did cross the +32 handle cap, in 519 of 1,765 samples, first at cycle 17. SDK2 never crossed the handle cap (max
|
||
+5). SDK1 retained managed memory is about 50 MiB against SDK2's about 20 MiB, flat on both. The comparator is a
|
||
standalone executable, not the VSTest testhost, and ran on windev concurrently with the SDK2 run (see below), so
|
||
compare growth, not absolute RSS.
|
||
|
||
## Diagnostics: what grows (managed vs native)
|
||
|
||
- **Managed is not the growth.** Retained managed bytes were +0.6 MiB (Linux) and +0.5 MiB (Windows) at peak. GC
|
||
committed was +5.9 MiB on Linux at the rejection.
|
||
- **JIT burst at the rejection.** Linux, cycle 3 → 247: JIT-compiled methods 35,180 → 75,180 (+40,000), IL bytes
|
||
+6.6 MB. Windows, cycle 3 → 285: 31,221 → 75,116 (+43,895). Threads and timers were unchanged or lower. This is
|
||
the generated-factory burst recorded in `../2026-11-10-opcua2/saturation-candidate.md`.
|
||
- **Linux mapping categories** (`linux-arm64-aa60b3e3/smaps-categories.json`, read-only `docker exec cat
|
||
/proc/214/smaps` at about cycle 260, cycle 990 and cycle 1480; path names stripped by `summarize.py`):
|
||
- JIT `doublemapper` RSS: 180.7 / 179.1 / 179.1 MiB.
|
||
- Anonymous writable: 144.0 / 143.6 / 169.8 MiB.
|
||
- Managed assembly files: 41.3 / 7.1 / 6.0 MiB.
|
||
- Native heap: 4.5 MiB.
|
||
|
||
At its 10-minute checkpoint, the earlier SDK2 Linux subset run had about 65–107 MiB of JIT doublemapper.
|
||
- **Windows working-set split** (`windows-x64-aa60b3e3/external-samples.jsonl.gz`, 30 s external sampler): the working
|
||
set went from about 183 MiB to a 315 MiB plateau (+132 MiB). Private bytes (commit charge) only went from about
|
||
77–80 MiB to about 101–106 MiB (max 123 MiB). About 80% of the working-set growth is therefore not private commit.
|
||
That fits shareable pages, for example .NET's W^X double-mapped JIT code (pagefile-backed sections) and image
|
||
pages. This is an attribution hint, not a proof. No runtime setting (for example `DOTNET_EnableWriteXorExecute`)
|
||
was tried or adopted, and no fix is claimed.
|
||
|
||
## Provenance
|
||
|
||
- **Source:** origin/master `aa60b3e34eb9d792b0c4ce78f8dddd73240a59e1` at launch. Its parent is `8ba0e22b`; the
|
||
diff against `b61f3e05` is test/harness only (`git diff b61f3e05 aa60b3e3 -- src` is empty). SDK
|
||
`2.0.1+07e62f72500cc0923a2bab940653c8c1116fa286`, HistorianGateway Client/Contracts 1.0.0, self-contained .NET
|
||
**10.0.14** on both platforms, `OPCUA_CERTIFICATE_LEAK_TRACKING=1`, late-slope mode (7,200 s, 30 s cycle / 25 s
|
||
restart / 120 s startup / 60 s shutdown bounds).
|
||
- **Linux:** bundle receipt `ee4ba62699cbe3229ecafaa93da7003509bb238e1057720b71a43bf4242b5204`. Image
|
||
`sha256:b04611ee…f293b` (mcr dotnet/sdk:10.0.103-noble arm64), OrbStack context, `--network none`, read-only root,
|
||
the unchanged `late-slope/{prepare,launch,analyze}.py`.
|
||
- Run: 7,200.12 s, 1,964 cycles, 491 restarts. Shutdown verified (0.02 s); container removed.
|
||
- Descriptor controls 2/2 passed. The canonical test failed on the latched cap and the test, container and
|
||
launcher all exited 1. `diagnosticCompleted=true`.
|
||
- The official analyzer validated all raw evidence (`analysis.json`).
|
||
- Raw `resources.jsonl` SHA-256 `51c26871…cef5e5f`, TRX `8a23d346…c7da8d0`. Every evidence hash is in
|
||
`run-receipt.json`.
|
||
- **Windows:** windev (Windows 10 22H2 19045, 16 vCPU / 32 GB). Bundle receipt
|
||
`d64ff61f3e623caba55b9507d2fcc37f1f40857bc38378edec4e82afb3ab982b`, archive `38bdd630…a7ed64e`, built by
|
||
`windows/prepare.py`, 947 files re-verified on the host.
|
||
- The target PID ran the bundle apphost with the bundle `coreclr.dll` mapped
|
||
(`1bc65b8d45474b19adf3ca85ce881087aff599ffc72b2c1a323e878227757b65`).
|
||
- Descriptor controls 2/2 passed on Windows (symlink-alias positive control, `cmd.exe` timeout control).
|
||
- MxAccessGw `/health/ready` returned 200 and the service stayed Running in all 23 five-minute checks.
|
||
OtOpcUaWonderwareHistorian was not touched.
|
||
- **Windows run incomplete — launcher defect, disclosed:** the launcher measured its 7,440 s outer deadline with the
|
||
wall clock. windev's clock was about 9 min slow; at 19:05:32 host time w32time stepped it +552 s. The deadline
|
||
therefore fired early and killed the run at **test-phase 7,014.07 s (cycle 1,774, 443 restarts)**, before the
|
||
7,200 s phase ended.
|
||
- There is no late-slope result row and no workload TRX. The 6,600–7,200 s window covers only 6,603–7,014 s
|
||
(105 samples).
|
||
- The in-test clocks are monotonic (Stopwatch), so the samples and timings are valid. The verdict does not depend
|
||
on the missing 186 s: the RSS rejection had already latched at 1,130.76 s.
|
||
- The launchers now use a Stopwatch.
|
||
- **Comparator:** built by `sdk1-comparator/build.py` from the RID-specific (`win-x64`) framework-dependent
|
||
publications of `efb9197a` Host and Client.CLI. Product code was not rebuilt.
|
||
- SDK `1.5.378.182+6cca32dd…`, self-contained .NET 10.0.14.
|
||
- Comparator sources at `2e9cccbd`, hashes in `build-provenance.json`. Exe `2bd97c5e…`, dll `e5bbee78…`.
|
||
- Run: 7,201.35 s, 1,767 cycles, 441 restarts. Shutdown verified.
|
||
- The held closed-session lists were (1 browse, 6 history) on every cycle. That is recorded, not asserted (SDK1
|
||
`Close` semantics, see `../../../../scripts/validation/opcua-lifecycle-memory/sdk1-comparator/README.md`).
|
||
- It started about 5 min after the SDK2 Windows run and overlapped it for the rest of the run on the 16-vCPU
|
||
host. Its launcher was stopped at about 7,220 s wall time so the same clock step could not kill it; the
|
||
comparator process finished on its own.
|
||
- **Linux fixture failure on `b61f3e05` (retained, `linux-b61f3e05-fixture-failure/`):** the first launch on the
|
||
then-current master failed at cycle 1. A UNS monitored item was bad because, since #808, the soak's own unplaced
|
||
`SoakEquipment` folder is denied under enforcement. This is the same fixture gap #888 fixed for the SDK matrix.
|
||
- `aa60b3e3` places the folder (`ServerEventRig.AdditionalFolderScopes`). No authorization code, operation, cap
|
||
or baseline changed.
|
||
- A 60 s macOS diagnostic and a 90 s Windows diagnostic (`windows-rehearsal-diagnostic/`) qualified the fix and the
|
||
port. Neither is acceptance evidence.
|
||
|
||
`verdict.json` in each run folder is produced by `make_verdicts.py` from the raw samples (and the Linux analyzer
|
||
report). A run passes only if the evidence validates, the tests pass, the 7,200 s phase completes, shutdown is
|
||
verified and no cold-cap rejection latched. Slopes are descriptive only. Raw samples are the gzipped
|
||
`resources.jsonl` files. No PKI, credentials or fixture keys are included.
|