docs(plans): WP-4 target-scale load test RESULTS + close register #25 and row 50
Full-scale run executed on this machine: 10 sites x 500 instances x 75 tags =
375,000 live tag subscriptions, 37,518 updates/s achieved vs 37,500 nominal,
45,021,375 updates over a 20-minute steady-state window (M4 Pro, 14 cores, 48 GB,
with the 8-node docker rig still running so the figures are pessimistic).
11 clean passes, 1 pass with a caveat, 0 failures:
tag latency P50 0.88ms P99 4.57ms max 37.41ms (1.1M samples, end-to-end)
stream 900,675 delivered, 0 dropped at 100 live subscribers
health collect+ingest P99 0.31ms, 10/10 sites tracked
debug view P99 2.19ms, 264 completed, 0 timeouts
deploy 500 instances to a site in 2.6s
cpu 41% of ONE core = 2.9% of the box
memory working-set slope +8.83 MB/min
Three findings, reported rather than tuned away:
F1 (Low) 20 min with ZERO gen-2 collections cannot fully settle the leak
question; the heap demonstrably sawtooths but an uncompacted gen-2 makes a
positive slope ambiguous. The 1-hour run would settle it. Not tuned.
F2 (informational, by design) a deferred S&F backlog sits for one full
DefaultRetryInterval (28.9s measured) before anything drains --
EnqueueAsync(attemptImmediateDelivery:false) stamps LastAttemptAt. Easy to
misread as slow drainage, so drain is reported as two numbers: retry wait,
then 3,533 msg/s of actual capacity.
F3 (positive) slow-subscriber isolation is TOTAL: 4 healthy subscribers at
100.00% with zero drops while a peer lost 197,028/200,000 events entirely
within its own bounded channel. Mechanism recorded link by link.
Also records what the run does NOT prove (not clustered, not real-network, not
real OPC UA, not 1 hour) and the four WP-4 sub-criteria this harness does not
cover, so the evidence is not over-read.
Register rows 25 and 50 -> RESOLVED 2026-08-15; remediation execution-log
residual 7 -> resolved; phase-8-checklist WP-4 section replaced with the
measured numbers.
This commit is contained in:
@@ -110,9 +110,20 @@ Deliberately not fixed in this program — each has a stated reason, not an over
|
||||
library's wire-compatibility guarantee — not a blocker for this program.
|
||||
6. **Fragile `SandboxTests` timing pin.** Pre-existing, unrelated to this remediation's changes;
|
||||
noted so it isn't mistaken for a regression if it flakes later.
|
||||
7. **Target-scale load test (deferred-work register #25).** This program's exit criterion is the
|
||||
live probes above, not #25 — #25 remains the follow-on validation that the moved ceilings hold
|
||||
under real load; schedule separately.
|
||||
7. ~~**Target-scale load test (deferred-work register #25).**~~ **RESOLVED 2026-08-15.** The
|
||||
follow-on validation ran: 10 sites x 500 instances x 75 tags = **375,000 live tag
|
||||
subscriptions**, 37,518 updates/s achieved vs 37,500 nominal, **45,021,375 updates** over a
|
||||
20-minute steady-state window. The moved ceilings hold with room to spare — end-to-end tag
|
||||
latency **P99 4.57 ms**, zero dropped events at 100 live subscribers, health report
|
||||
collect+ingest **P99 0.31 ms**, debug view **P99 2.19 ms** with no timeouts, CPU **2.9% of the
|
||||
box**. 11 clean passes, 1 pass with a caveat (F1: 20 minutes with zero gen-2 collections cannot
|
||||
fully settle the leak question; the 1-hour run would), 0 failures. Register **row 50** (S&F
|
||||
drain rate + per-subscriber stream backpressure) was absorbed into the same run and closed with
|
||||
it — S&F drains at **3,533 msg/s**, and a stalled subscriber costs healthy peers **nothing**
|
||||
(100.00%, zero drops). Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`;
|
||||
design: `docs/plans/2026-08-15-target-scale-load-test-design.md`; harness:
|
||||
`tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`. Residual: WP-4's `[xc-3]`/`[xc-4]`/`[xc-5]`/`[xc-8]`
|
||||
are central-cluster/browser concerns outside this harness and remain open scope.
|
||||
8. **Playwright 14 pre-existing env failures.** Present on `main` too, rig-state related, not
|
||||
introduced by this branch.
|
||||
9. **`site_events` retention purge still oplog-visible.** WP3.2's sliced retention DELETE is a row
|
||||
|
||||
Reference in New Issue
Block a user