docs(plans): WP-4 target-scale load test RESULTS + close register #25 and row 50

Full-scale run executed on this machine: 10 sites x 500 instances x 75 tags =
375,000 live tag subscriptions, 37,518 updates/s achieved vs 37,500 nominal,
45,021,375 updates over a 20-minute steady-state window (M4 Pro, 14 cores, 48 GB,
with the 8-node docker rig still running so the figures are pessimistic).

11 clean passes, 1 pass with a caveat, 0 failures:
  tag latency  P50 0.88ms  P99 4.57ms  max 37.41ms  (1.1M samples, end-to-end)
  stream       900,675 delivered, 0 dropped at 100 live subscribers
  health       collect+ingest P99 0.31ms, 10/10 sites tracked
  debug view   P99 2.19ms, 264 completed, 0 timeouts
  deploy       500 instances to a site in 2.6s
  cpu          41% of ONE core = 2.9% of the box
  memory       working-set slope +8.83 MB/min

Three findings, reported rather than tuned away:
  F1 (Low) 20 min with ZERO gen-2 collections cannot fully settle the leak
     question; the heap demonstrably sawtooths but an uncompacted gen-2 makes a
     positive slope ambiguous. The 1-hour run would settle it. Not tuned.
  F2 (informational, by design) a deferred S&F backlog sits for one full
     DefaultRetryInterval (28.9s measured) before anything drains --
     EnqueueAsync(attemptImmediateDelivery:false) stamps LastAttemptAt. Easy to
     misread as slow drainage, so drain is reported as two numbers: retry wait,
     then 3,533 msg/s of actual capacity.
  F3 (positive) slow-subscriber isolation is TOTAL: 4 healthy subscribers at
     100.00% with zero drops while a peer lost 197,028/200,000 events entirely
     within its own bounded channel. Mechanism recorded link by link.

Also records what the run does NOT prove (not clustered, not real-network, not
real OPC UA, not 1 hour) and the four WP-4 sub-criteria this harness does not
cover, so the evidence is not over-read.

Register rows 25 and 50 -> RESOLVED 2026-08-15; remediation execution-log
residual 7 -> resolved; phase-8-checklist WP-4 section replaced with the
measured numbers.
This commit is contained in:
Joseph Doherty
2026-08-15 02:55:59 -04:00
parent 8abebdae33
commit da65605e41
5 changed files with 351 additions and 18 deletions
@@ -24,6 +24,12 @@ namespace ZB.MOM.WW.ScadaBridge.LoadHarness;
/// <param name="DriverSkippedNoCallback">Emissions skipped before subscriptions existed.</param>
/// <param name="SteadyStateResources">Resource behaviour over the measurement window.</param>
/// <param name="WholeRunResources">Resource behaviour over the whole run.</param>
/// <param name="ResourceSamples">
/// Every raw resource sample. Retained because the summary alone cannot distinguish a
/// slow leak from a heap that simply sawtooths — especially when the run records zero
/// gen-2 collections, where a positive least-squares slope may be nothing more than
/// gen-2 garbage that was never collected.
/// </param>
/// <param name="HealthReportLatency">Health report collect+ingest latency.</param>
/// <param name="HealthReportsDelivered">Health reports ingested by the central aggregator.</param>
/// <param name="SitesTrackedByAggregator">Sites the central aggregator ended up tracking.</param>
@@ -51,6 +57,7 @@ public sealed record HarnessRunResult(
long DriverSkippedNoCallback,
ResourceWindowSummary? SteadyStateResources,
ResourceWindowSummary? WholeRunResources,
IReadOnlyList<ResourceSample> ResourceSamples,
LatencySnapshot HealthReportLatency,
long HealthReportsDelivered,
int SitesTrackedByAggregator,
@@ -233,6 +240,7 @@ public static class HarnessRun
DriverSkippedNoCallback: driverSkipped,
SteadyStateResources: steadyResources,
WholeRunResources: sampler.Summarize(0, double.MaxValue),
ResourceSamples: sampler.Snapshot(),
HealthReportLatency: probes.HealthReportLatency.Snapshot(),
HealthReportsDelivered: probes.HealthReportsDelivered,
SitesTrackedByAggregator: aggregator.GetAllSiteStates().Count,