Rust''s debug profile costs the bench ~45% of solo throughput and ~3x of
p99 latency vs release (267 vs 184 solo calls/sec, p99 5.7 vs 16ms).
Debug disables inlining, runs overflow checks on every arithmetic op,
keeps Future state machines un-collapsed, and lets every Vec allocation
through unoptimized. Other compiled clients in the matrix don''t see
this gap: Go always builds optimized, Python is interpreted, and the
JIT-tiered runtimes (HotSpot for Java, CoreCLR Tier 1 for .NET) close
most of the gap during the warmup window.
The driver now requests `cargo run --release` for Rust and `dotnet run
-c Release --no-build` for .NET, so the two compiled-AOT clients race
under their production-equivalent profiles. Callers must `cargo build
--release -p mxgw-cli` and `dotnet build ... -c Release` once before
running the bench; `--no-build` then keeps each measurement window
free of compilation overhead.
Live re-run (5-way concurrent, 30s, bulkSize 6) after the switch:
rust: 145.35 calls/sec (was 123.26 in debug; 18% gain under contention)
go: 185.59 calls/sec
java: 171.80 calls/sec
dotnet:172.31 calls/sec
python:140.52 calls/sec
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a bench-read-bulk subcommand to every client CLI (.NET, Go, Rust,
Python, Java) and a PowerShell driver that runs all five concurrently
against the deployed gateway and prints a side-by-side comparison.
Each CLI''s bench:
- Opens its own session, registers, subscribes to bulk-size tags so the
worker''s MxAccessValueCache populates from real OnDataChange events.
- Runs a warmup-seconds-long pre-loop with identical calls so JIT /
connection-pool / first-call overhead is amortised before the
measurement window.
- Runs ReadBulk in a tight in-process loop for duration-seconds with
per-call high-resolution latency capture (Stopwatch in .NET,
time.Now in Go, std::time::Instant in Rust, time.perf_counter in
Python, System.nanoTime in Java).
- Unsubscribes + closes the session, then emits one JSON object with
the shared schema: { language, durationMs, totalCalls, successfulCalls,
failedCalls, totalReadResults, cachedReadResults, callsPerSecond,
latencyMs: { p50, p95, p99, max, mean } }.
The PS driver (scripts/bench-read-bulk.ps1) launches one detached process
per client, waits for all to finish, parses the trailing JSON object from
each stdout, prints a comparison table, and persists the combined report
under artifacts/bench/. Quoting around Java''s `gradle --args="..."` is
handled by writing a one-shot .bat that cmd.exe runs; the .NET CLI''s
per-call gRPC timeout is auto-scaled to (Duration + Warmup + 30s) so the
channel-wide timeout doesn''t cancel the bench mid-loop.
Live 30-second steady-state run against the deployed gateway, all five
clients hitting the same six TestMachine_001..006.TestChangingInt tags:
client calls/sec cached/total p50 ms p95 ms p99 ms max ms
dotnet 171.78 30924/30924 3.84 14.06 40.41 542.48
go 175.46 31590/31590 3.93 13.52 41.26 243.00
rust 123.26 22188/22188 5.52 15.78 48.11 544.41
python 145.79 26244/26244 4.86 14.85 41.65 645.84
java 181.12 32604/32604 3.80 10.59 33.37 344.27
143,550 ReadBulk results across all five clients during the 30s window;
100% were was_cached = true (the worker''s cache fast-path never fell
through to the snapshot lifecycle). Aggregate read throughput ~800
calls/sec against five concurrent sessions sharing the same cached tags.
A second variant with bulk-size 20 sustained the same per-client call
rate while delivering 3.3x more values per call (~37,000 cached reads/sec
aggregate across the five concurrent sessions), confirming the linear
per-tag cache lookup inside one call is not a bottleneck at this scale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Joseph Doherty