Joseph Doherty
6.7 KiB
OPC UA SDK lock benchmark diagnostic — 2026-11-10
Compared baseline 83ed880e4471344deaf008da9bd028c7e170bdc7 (SDK 1.5.378.182) with candidate a8e1082d69a112ea0e16788f023067979698527a (SDK 2.0.1). Both Release project builds passed with 0 warnings/errors. No source edits or new commits were made for this evidence task.
Same Mac16,7, 14 logical/physical CPU cores, 48 GiB RAM, .NET SDK 10.0.403. OS details and exact assembly SHA-256 hashes are in metadata.json. All runs used OTOPCUA_BENCH=1 and the existing NodeManagerLockTimeBenchmarkTests class. Scenario sizes and assertions are identical; the only benchmark source differences are SDK-required OperationContext.RequestLifetime.None and IAsyncNodeManager interface changes. Each test boots a private server with its own temporary PKI and probed port. No containers or global fixture were used.
One warmup invocation per version was followed by three recorded invocations per version, alternating baseline/candidate with 25 seconds between processes. Each invocation is a fresh process; warmup primes host/file caches, not persistent in-process JIT state. Default xUnit test order varied. Warmups are excluded from the medians.
| Recorded metric | Baseline runs | SDK 2 runs | Baseline median | SDK 2 median | Change |
|---|---|---|---|---|---|
| Rebuild 10k-tag folder | 18, 15, 17 | 15, 15, 14 | 17 ms | 15 ms | -11.76% |
| Remove 200 equipment / 50k nodes | 81, 50, 86 | 80, 68, 80 | 81 ms | 80 ms | -1.23% |
| Visits for 10 removals / 50 conditions | 10, 10, 10 | 10, 10, 10 | 10 visits | 10 visits | +0.00% |
| OS command maximum resident set size | 385.17, 136.20, 136.30 | 136.39, 136.81, 136.80 | 136.30 MiB | 136.80 MiB | +0.37% |
The median teardown timings do not exceed the provisional +15% latency-growth reference. Their inverse-time local teardown-rate proxies change by +13.33% (rebuild) and +1.25% (removal), so neither shows a >10% drop in this subset. These derived rates are not wire throughput, and three samples do not establish p95 latency. Command maximum-RSS median rose 0.37%, below the +15% reference, but it does not measure steady server RSS. No change to budgets or acceptance decision is implied.
All eight invocations passed all three cases: 24 passed, 0 failed, 0 skipped (18 recorded case executions plus 6 warmup). Every test exit and MIN_EXECUTED=3 guard exit was 0; aggregate orchestrator exit was 0. The original 5000 ms opt-in teardown assertions remained enforced; the largest recorded operation was 86 ms. The strict, hardware-independent visit counter was exactly 10 in all eight invocations.
Host-load caveat: other solution testing/build activity shared this host. Initial preparation saw load averages 19.78/30.02/23.48 on 14 cores; per-run before/after uptime snapshots are retained in result.json. Recorded one-minute load ranged 5.58–10.90; five-minute load ranged 15.26–20.32 and fifteen-minute load 18.45–20.56. BENCH uses integer-millisecond Stopwatch output. Timing spread, quantization and changing xUnit ordering preclude a stable performance claim. OS /usr/bin/time -l maximum RSS is command-invocation rusage, not isolated teardown allocation or steady server RSS; baseline run1 is an outlier and is included unchanged. GC/allocation counters are not exposed by the existing harness and were not measured. This subset does not complete wire throughput, p50/p95/p99 latency, allocation, steady RSS, history-page throughput, load, soak, publish, or resource-lifecycle gates.
Recorded invocations
| Version / phase | UTC start | UTC end | Test / guard exit | Evidence directory |
|---|---|---|---|---|
| baseline warmup | 2026-11-10T16:54:37.980941+00:00 | 2026-11-10T16:54:42.012296+00:00 | 0 / 0 | /Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/warmup |
| candidate warmup | 2026-11-10T16:55:07.020267+00:00 | 2026-11-10T16:55:11.610491+00:00 | 0 / 0 | /Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/warmup |
| baseline run1 | 2026-11-10T16:55:36.615335+00:00 | 2026-11-10T16:55:40.023724+00:00 | 0 / 0 | /Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/run1 |
| candidate run1 | 2026-11-10T16:56:05.032437+00:00 | 2026-11-10T16:56:09.352092+00:00 | 0 / 0 | /Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/run1 |
| baseline run2 | 2026-11-10T16:56:34.361326+00:00 | 2026-11-10T16:56:37.712286+00:00 | 0 / 0 | /Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/run2 |
| candidate run2 | 2026-11-10T16:57:02.720388+00:00 | 2026-11-10T16:57:06.894343+00:00 | 0 / 0 | /Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/run2 |
| baseline run3 | 2026-11-10T16:57:31.903230+00:00 | 2026-11-10T16:57:35.306953+00:00 | 0 / 0 | /Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/83ed880e/lock-bench-20261110T165437Z/run3 |
| candidate run3 | 2026-11-10T16:58:00.315244+00:00 | 2026-11-10T16:58:04.720985+00:00 | 0 / 0 | /Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z/run3 |
Each directory contains a fresh TRX (BENCH stdout attached to individual test results), test.log, resource.log, guard.log, and result.json. Session metadata/results/summary:
/Users/dohertj2/.codex/worktrees/opcua2-test-convergence/OtOpcUa/artifacts/opcua2/a8e1082d/lock-bench-20261110T165437Z
Reproduction (in each version worktree; choose a new, nonexistent absolute result directory each invocation):
OPC_RUN_DIR=/absolute/fresh/result/directory
test ! -e "$OPC_RUN_DIR" || exit 2
mkdir -p "$OPC_RUN_DIR" || exit
opc_test_status=0
OTOPCUA_BENCH=1 /usr/bin/time -l dotnet test \
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj \
-c Release --no-build --no-restore \
--filter FullyQualifiedName~NodeManagerLockTimeBenchmarkTests \
--logger trx --logger "console;verbosity=detailed" \
--results-directory "$OPC_RUN_DIR" \
> "$OPC_RUN_DIR/test.log" 2> "$OPC_RUN_DIR/resource.log" || opc_test_status=$?
opc_guard_status=0
MIN_EXECUTED=3 bash scripts/ci/assert-not-all-skipped.sh "$OPC_RUN_DIR"/*.trx \
> "$OPC_RUN_DIR/guard.log" 2>&1 || opc_guard_status=$?
if [ "$opc_test_status" -ne 0 ]; then exit "$opc_test_status"; fi
exit "$opc_guard_status"
Exact completed orchestration script: /tmp/opcua2-lock-bench.py. It records environment, SDK dependencies, assembly hashes, start/end UTC, host load, metrics, counts and statuses, and alternates the eight invocations with 25-second gaps.