Joseph Doherty
61 lines
6.7 KiB
Markdown
61 lines
6.7 KiB
Markdown
# OPC UA SDK lock benchmark diagnostic — 2026-11-10
|
||
|
||
Compared baseline `83ed880e4471344deaf008da9bd028c7e170bdc7` (SDK 1.5.378.182) with candidate `a8e1082d69a112ea0e16788f023067979698527a` (SDK 2.0.1). Both Release project builds passed with 0 warnings/errors. No source edits or new commits were made for this evidence task.
|
||
|
||
Same Mac16,7, 14 logical/physical CPU cores, 48 GiB RAM, .NET SDK 10.0.403. OS details and exact assembly SHA-256 hashes are in metadata.json. All runs used OTOPCUA_BENCH=1 and the existing NodeManagerLockTimeBenchmarkTests class. Scenario sizes and assertions are identical; the only benchmark source differences are SDK-required OperationContext.RequestLifetime.None and IAsyncNodeManager interface changes. Each test boots a private server with its own temporary PKI and probed port. No containers or global fixture were used.
|
||
|
||
One warmup invocation per version was followed by three recorded invocations per version, alternating baseline/candidate with 25 seconds between processes. Each invocation is a fresh process; warmup primes host/file caches, not persistent in-process JIT state. Default xUnit test order varied. Warmups are excluded from the medians.
|
||
|
||
| Recorded metric | Baseline runs | SDK 2 runs | Baseline median | SDK 2 median | Change |
|
||
|---|---|---|---:|---:|---:|
|
||
| Rebuild 10k-tag folder | 18, 15, 17 | 15, 15, 14 | 17 ms | 15 ms | -11.76% |
|
||
| Remove 200 equipment / 50k nodes | 81, 50, 86 | 80, 68, 80 | 81 ms | 80 ms | -1.23% |
|
||
| Visits for 10 removals / 50 conditions | 10, 10, 10 | 10, 10, 10 | 10 visits | 10 visits | +0.00% |
|
||
| OS command maximum resident set size | 385.17, 136.20, 136.30 | 136.39, 136.81, 136.80 | 136.30 MiB | 136.80 MiB | +0.37% |
|
||
|
||
The median teardown timings do not exceed the provisional +15% latency-growth reference. Their inverse-time local teardown-rate proxies change by +13.33% (rebuild) and +1.25% (removal), so neither shows a >10% drop in this subset. These derived rates are not wire throughput, and three samples do not establish p95 latency. Command maximum-RSS median rose 0.37%, below the +15% reference, but it does not measure steady server RSS. No change to budgets or acceptance decision is implied.
|
||
|
||
All eight invocations passed all three cases: 24 passed, 0 failed, 0 skipped (18 recorded case executions plus 6 warmup). Every test exit and MIN_EXECUTED=3 guard exit was 0; aggregate orchestrator exit was 0. The original 5000 ms opt-in teardown assertions remained enforced; the largest recorded operation was 86 ms. The strict, hardware-independent visit counter was exactly 10 in all eight invocations.
|
||
|
||
Host-load caveat: other solution testing/build activity shared this host. Initial preparation saw load averages 19.78/30.02/23.48 on 14 cores; per-run before/after uptime snapshots are retained in result.json. Recorded one-minute load ranged 5.58–10.90; five-minute load ranged 15.26–20.32 and fifteen-minute load 18.45–20.56. BENCH uses integer-millisecond Stopwatch output. Timing spread, quantization and changing xUnit ordering preclude a stable performance claim. OS /usr/bin/time -l maximum RSS is command-invocation rusage, not isolated teardown allocation or steady server RSS; baseline run1 is an outlier and is included unchanged. GC/allocation counters are not exposed by the existing harness and were not measured. This subset does not complete wire throughput, p50/p95/p99 latency, allocation, steady RSS, history-page throughput, load, soak, publish, or resource-lifecycle gates.
|
||
|
||
## Recorded invocations
|
||
|
||
| Version / phase | UTC start | UTC end | Test / guard exit | Evidence directory |
|
||
|---|---|---|---|---|
|
||
| baseline warmup | 2026-11-10T16:54:37.980941+00:00 | 2026-11-10T16:54:42.012296+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/warmup` |
|
||
| candidate warmup | 2026-11-10T16:55:07.020267+00:00 | 2026-11-10T16:55:11.610491+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/warmup` |
|
||
| baseline run1 | 2026-11-10T16:55:36.615335+00:00 | 2026-11-10T16:55:40.023724+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/run1` |
|
||
| candidate run1 | 2026-11-10T16:56:05.032437+00:00 | 2026-11-10T16:56:09.352092+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/run1` |
|
||
| baseline run2 | 2026-11-10T16:56:34.361326+00:00 | 2026-11-10T16:56:37.712286+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/run2` |
|
||
| candidate run2 | 2026-11-10T16:57:02.720388+00:00 | 2026-11-10T16:57:06.894343+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/run2` |
|
||
| baseline run3 | 2026-11-10T16:57:31.903230+00:00 | 2026-11-10T16:57:35.306953+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/run3` |
|
||
| candidate run3 | 2026-11-10T16:58:00.315244+00:00 | 2026-11-10T16:58:04.720985+00:00 | 0 / 0 | `/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/run3` |
|
||
|
||
Each directory contains a fresh TRX (BENCH stdout attached to individual test results), test.log, resource.log, guard.log, and result.json. Session metadata/results/summary:
|
||
|
||
/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z
|
||
|
||
Reproduction (in each version worktree; choose a new, nonexistent absolute result directory each invocation):
|
||
|
||
```bash
|
||
OPC_RUN_DIR=/absolute/fresh/result/directory
|
||
test ! -e "$OPC_RUN_DIR" || exit 2
|
||
mkdir -p "$OPC_RUN_DIR" || exit
|
||
opc_test_status=0
|
||
OTOPCUA_BENCH=1 /usr/bin/time -l dotnet test \
|
||
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj \
|
||
-c Release --no-build --no-restore \
|
||
--filter FullyQualifiedName~NodeManagerLockTimeBenchmarkTests \
|
||
--logger trx --logger "console;verbosity=detailed" \
|
||
--results-directory "$OPC_RUN_DIR" \
|
||
> "$OPC_RUN_DIR/test.log" 2> "$OPC_RUN_DIR/resource.log" || opc_test_status=$?
|
||
opc_guard_status=0
|
||
MIN_EXECUTED=3 bash scripts/ci/assert-not-all-skipped.sh "$OPC_RUN_DIR"/*.trx \
|
||
> "$OPC_RUN_DIR/guard.log" 2>&1 || opc_guard_status=$?
|
||
if [ "$opc_test_status" -ne 0 ]; then exit "$opc_test_status"; fi
|
||
exit "$opc_guard_status"
|
||
```
|
||
|
||
Exact completed orchestration script: /tmp/opcua2-lock-bench.py. It records environment, SDK dependencies, assembly hashes, start/end UTC, host load, metrics, counts and statuses, and alternates the eight invocations with 25-second gaps.
|