ScadaBridge/docs/plans/2026-08-15-target-scale-load-test-results.md

Target-Scale Load Test — Results (Phase-8 WP-4, deferred-work register #25 + row 50)

Date: 2026-08-15 Design memo: docs/plans/2026-08-15-target-scale-load-test-design.md Harness: tests/ZB.MOM.WW.ScadaBridge.LoadHarness/ Closes: deferred-work register #25 and row 50; residual 7 of docs/plans/2026-08-15-arch-review-remediation-execution-log.md.

Register row 25 recorded that WP-4 had been claimed complete on the strength of a 107-byte checklist stub with no per-work-package results and no linked run. This document is the run that stub never had.

---

1. Environment

Host	Josephs-MBP — Apple M4 Pro, 14 logical cores, 48 GB RAM
OS	macOS 26.5.2 (Darwin)
Runtime	.NET 10.0.5, server GC enabled, Release build
Branch / commit	target-scale-load-test @ 20f6b0b9 (harness), 8abebdae (design memo)
Concurrent load	The 8-node docker/ rig (OrbStack) remained running throughout — the user's live development cluster. Measured baseline: ~86% of one core (~6% of the box) and ~10-16 GB RSS. The harness figures below are therefore pessimistic, not idealised.
Scale executed	10 sites × 500 instances × 75 tags = 375,000 live tag subscriptions
Offered load	37,500 tag updates/second nominal (one update per tag per 10 s)
Measurement window	20 minutes steady state, after a 2-minute settle
Total wall time	1,510 s (~25 min)

One clean full-scale run is the dataset reported here. A second run was started to add raw per-sample resource capture (sharpening finding F1), but a verification build overlapped the start of its measurement window; it was discarded rather than reported, per the rule that a measurement contaminated by concurrent load is not evidence. The raw-sample capture it was meant to exercise is now in the harness (HarnessRunResult.ResourceSamples) and will be present in any future run, including the 1-hour run in §7 — the run reported below predates that field.

---

2. Verdict summary

#	Criterion	Threshold	Measured	Verdict
1	[2.5-1] 10 sites simultaneously operational	all 10 built, ramped, tracked centrally	10/10 sites tracked by CentralHealthAggregator	PASS
2	[2.5-2] 500 instances/site, active subscriptions	5,000 actors; SkippedNoCallback = 0	5,000 InstanceActors; skipped = 0	PASS
3	[2.5-3] 75 tags/instance (375,000 total)	375,000 subscribed via the real DCL	375,000 tag paths across 50 DataConnectionActors	PASS
4	Tag update latency	P99 < 250 ms, P50 < 50 ms	P50 0.88 ms, P95 2.83 ms, P99 4.57 ms, P99.9 16.04 ms, max 37.41 ms (1,100,675 samples)	PASS (55× margin at P99)
5	[xc-1] message patterns under load	≥ 95% of nominal, no unexpected loss	37,518/s achieved vs 37,500 nominal (100.0%); 45,021,375 updates offered; 0 events dropped at healthy subscribers	PASS
6	[xc-2] health reports within expected intervals	collect+ingest P99 < 1,000 ms; all sites tracked	P99 0.31 ms, max 4.60 ms over 440 reports; 10/10 sites	PASS (3,200× margin)
7	[xc-6] debug view without impacting site performance	snapshot P99 < 2,000 ms, 0 timeouts	P99 2.19 ms, max 2.72 ms, 264 completed, 0 timeouts	PASS
8	[xc-7] S&F concurrent buffering from many instances	20,000 messages, 25 origins, drains to 0	20,000 buffered at 15,368 msg/s; residual depth 0	PASS
9	Row 50 — S&F drain rate	> 500 msg/s	3,533 msg/s (20,000 messages in 5.66 s of active drain)	PASS (7× margin)
10	Row 50 — slow-subscriber backpressure	healthy = 100%; stalled loss confined to its own channel	healthy 100.00% (4/4, zero drops); stalled 1.18%, all 197,028 losses in its own bounded channel	PASS
11	Deploy 500 instances to a site	slowest site < 120 s	2.6 s	PASS (46× margin)
12	Memory within acceptable bounds	WS slope < 20 MB/min; no monotonic heap climb	WS +8.83 MB/min; heap sawtooths (peak 3,625 MB vs 2,313→2,523 MB endpoints) with a +19.52 MB/min drift — see finding F1	PASS with a caveat
13	CPU within acceptable bounds	mean < 50% of the box	41% of one core = 2.9% of the box (peak 55% of one core)	PASS (17× margin)

11 clean passes, 1 pass with a documented caveat, 0 failures. Four WP-4 sub-criteria were out of this harness's scope — see §6.

---

3. Measured detail

3.1 Deployment / ramp

site fixtures built     0.2s
all 5,000 instance actors   2.7s   (10 sites in parallel)
slowest single site     2.6s       (500 instances, production staggering)

The ramp used the production StartupBatchSize (20) / StartupBatchDelayMs (100) pacing, i.e. 25 batches × 100 ms = 2.5 s of deliberate delay per site. So the 2.6 s figure is almost entirely the configured stagger; actual construction of 500 InstanceActors (each deserializing a 75-attribute FlattenedConfiguration, loading static overrides from SQLite and issuing a DCL subscribe) costs ~0.1 s. Against a 120 s budget this criterion is not close to binding.

3.2 Tag update latency (DCL boundary → stream subscriber)

samples 1,100,675   mean 1.17ms
p50 0.88ms   p95 2.83ms   p99 4.57ms   p99.9 16.04ms   max 37.41ms

This is a true end-to-end path measurement, not a synthetic timer: the emit instant is stamped on the TagValue handed to the adapter callback and travels verbatim through DataConnectionActor → TagValueUpdate.Timestamp → AttributeValueChanged.Timestamp → the site stream → StreamRelayActor → the proto SiteStreamEvent, where the subscriber subtracts it. Every hop is production code.

Sub-millisecond median under 37,500 updates/s across 5,000 actors means the actor hierarchy is nowhere near its ceiling at target scale.

3.3 Stream delivery

events delivered  900,675
events dropped          0

Zero evictions at the 100 live subscribers across a 20-minute window at full rate. The site stream's DropHead buffering exists for pathological cases; at nominal target-scale load it never engaged.

3.4 Health reporting at scale

samples 440   mean 0.03ms   p50 0.01ms   p95 0.04ms   p99 0.31ms   max 4.60ms
reports ingested 440    sites tracked centrally 10

SiteHealthCollector.CollectReport for a site carrying 37,500 subscriptions plus CentralHealthAggregator.ProcessReport costs well under a millisecond. Against the 30-second report interval there is no plausible queueing risk.

3.5 Debug view under load

samples 264   mean 0.14ms   p50 0.08ms   p99 2.19ms   max 2.72ms
completed 264   timed out 0

Each sample is an Ask<DebugViewSnapshot> landing in the mailbox of an InstanceActor concurrently ingesting tag updates, so the figure includes real queueing behind production traffic. [xc-6]'s "without impacting site performance" also holds in the other direction: tag latency percentiles were measured with these snapshots running throughout, and show no degradation.

3.6 Resources

window          1200s over 121 samples
working set     2665 -> 3261 MB (peak 3261)   slope  +8.83 MB/min
managed heap    2313 -> 2523 MB (peak 3625)   slope +19.52 MB/min
cpu mean/peak   41% / 55% of one core  (2.9% of the box)
threads         58        gen2 GCs in window: 0

CPU is the headline: 375,000 subscriptions at 37,500 updates/s consumed under half of one core of fourteen. Thread count stayed flat at 58 for ten actor systems.

---

4. Findings

F1 — Memory: 20 minutes with zero gen-2 collections cannot fully settle the leak question (Low)

The working-set slope (+8.83 MB/min) is inside the threshold, and the managed heap clearly sawtooths — its peak (3,625 MB) sits well above both endpoints (2,313 MB → 2,523 MB), so the collector is demonstrably reclaiming. But GC.CollectionCount(2) recorded zero gen-2 collections across the entire 25-minute run, at 45 million events. Gen-2 was therefore never compacted, and a positive least-squares drift on an uncompacted heap cannot be distinguished from a genuine slow leak by this run alone.

This is a limitation of the shortened window (deviation D1), not an observed defect: no metric degraded, no drop appeared, and latency percentiles were flat from the first minute to the twentieth. Recorded honestly rather than reported as "no leak".

To settle it, run the full 1-hour protocol (§7) and check whether the heap slope flattens once gen-2 collections begin, or issue an explicit GC.Collect(2, Forced, blocking: true) at window start and end and compare. Deliberately not attempted here — the brief was to measure, not to tune.

F2 — Store-and-forward waits one full DefaultRetryInterval before draining a deferred backlog (informational, by design)

The drain measurement initially read as 33 msg/s, which would have been alarming. The progress series showed why: nothing drained for 29-30 seconds, then the entire backlog cleared in a fraction of a second.

Cause, confirmed in source: StoreAndForwardService.EnqueueAsync with attemptImmediateDelivery: false stamps message.LastAttemptAt = UtcNow (StoreAndForwardService.cs:754-757). GetMessagesForRetryAsync's due-predicate is last_attempt_at_ms IS NULL OR retry_interval_ms = 0 OR (now - last_attempt) >= retry_interval, so such a row is not due for one DefaultRetryInterval (30 s). Explicit TriggerSweep() calls find nothing due and correctly do nothing.

This is correct, intended behaviour — the caller declared delivery was not attempted, so the row waits one retry interval — but it is easy to misread as slow drainage. The harness therefore reports the two numbers separately:

retry wait before drain  28.9s  (DefaultRetryInterval)
drain throughput      3,533 msg/s  (active drain 5.66s)

Worth knowing operationally: a site that buffers a backlog this way will appear completely stalled for the first 30 seconds. Note the Notify.Send path (deferToSweep: true) deliberately leaves LastAttemptAt null so its rows are due immediately and skip this wait entirely.

F3 — Slow-subscriber isolation is complete, and the mechanism is worth recording (positive result)

Register row 50 asked what a slow/stalled gRPC subscriber does to per-subscriber buffering with several subscribers attached. With five subscribers on the same instance (identical offered sequence) and one reader stalled at 50 ms/event:

Subscriber	State	Received	Dropped	Delivery
slowprobe-00	STALLED	2,369	197,028	1.18%
slowprobe-01	healthy	200,000	0	100.00%
slowprobe-02	healthy	200,000	0	100.00%
slowprobe-03	healthy	200,000	0	100.00%
slowprobe-04	healthy	200,000	0	100.00%

Isolation is total — not merely "good". The stalled subscriber lost 98.8% of its own events and cost its peers exactly nothing. The reason is structural, and each link matters:

1. The per-subscriber graph is Where → Buffer(StreamBufferSize, DropHead) → Sink.ForEach(Tell). A DropHead buffer always accepts, so it never backpressures the BroadcastHub upstream — one subscriber cannot stall the hub for the others.
2. Sink.ForEach does an actor Tell, which never blocks.
3. StreamRelayActor.WriteToChannel uses TryWrite on a bounded DropOldest channel, which never blocks either — so the relay actor's mailbox drains regardless of reader speed, and loss is confined to that subscriber's own channel where it is counted.

The consequence for operators: a slow WAN link or wedged central client degrades only its own feed, and the loss is visible in the per-stream eviction counter and the RecordSiteStreamEventDropped telemetry rather than being silent.

One caveat established while building the probe: publishing an unpaced burst makes every subscriber lose events, because the publish source is a single Source.ActorRef(StreamBufferSize, DropHead) shared by all attribute subscribers, upstream of the hub. That shared stage — not per-subscriber buffering — is the real limit on burst absorption. The probe paces at 2,000 events/s to isolate the variable under test; the shared stage's own capacity is separately covered by SiteStreamThroughputTests.

F4 — Pre-existing test-isolation flake in QueueDepthGaugeTests (Low, not introduced here)

Noticed during this work package's verification pass, recorded so it is not later mistaken for a regression from the load-harness branch.

StoreAndForward.Tests.QueueDepthGaugeTests.Gauge_TracksBufferedDepth_AcrossEnqueueDrainAndPark fails in a full-suite run (Expected: 0, Actual: 2, QueueDepthGaugeTests.cs:116) but passes when run in isolation — the signature of a shared static gauge carrying state across tests, i.e. an ordering dependency rather than a product defect.

It cannot originate from this branch: the branch changes zero src/ files relative to its base (986e6e7a), and both the StoreAndForward source and its test project are byte-identical to that base. Left unfixed deliberately — out of scope for a measurement work package, and the brief was explicitly not to attempt unrelated fixes. Worth a separate triage alongside the pre-existing SandboxTests timing pin already recorded as residual 6 of the arch-review remediation execution log.

---

5. What this does not prove

Stated plainly so the evidence is not over-read:

• Not a clustered run. Sites are single, non-clustered ActorSystems (deviation D2). Failover, singleton handover and split-brain behaviour are covered elsewhere (FailoverTimingTests, SbrFailoverTests, docker/failover-drill.sh).
• Not a real-network run. The gRPC, health-transport and S&F-to-central hops are in-process. Serialization cost, TLS, WAN latency and socket backpressure are out of frame; the stream path's backpressure was measured deliberately by substituting a controllable reader for the socket (D4).
• Not a real OPC UA run. 375,000 monitored items were simulated at the adapter callback (D3). The DCL above that callback is real and fully exercised.
• Not 1 hour. 20 minutes (D1). See finding F1 for the one question this leaves genuinely open.

6. WP-4 criteria not covered by this harness

Recorded rather than silently dropped (deviation D7):

Criterion	Why not covered
[xc-3] site event logging within 30-day / 1 GB limits	Needs a long-horizon retention run, not a load run
[xc-4] audit logging does not degrade central	Needs a central MS SQL fixture; this harness builds site runtimes only
[xc-5] template flattening/validation for large templates	Template Engine is a central-cluster concern; unrelated to the site-runtime load path
[xc-8] UI workflows remain responsive	Needs a browser + a live central cluster (Playwright territory)

These remain open scope for WP-4 and should be tracked separately rather than being considered closed by this run.

7. Reproducing

Full protocol as executed (20-minute window):

dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
  --results loadharness-results.json

The full 1-hour version required by the WP-4 test protocol — identical in every other respect, and the run that would settle finding F1:

dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
  --sustain-minutes 60 --results loadharness-results-1h.json

CI-scale smoke (~80 s), which asserts the harness still produces coherent measurements end to end:

dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests \
  --filter "FullyQualifiedName~TargetScaleHarnessSmoke"

--help-style knobs: --sites, --instances-per-site, --tags-per-instance, --tag-update-period-seconds, --settle-minutes, --sustain-minutes, --sample-seconds, --health-interval-seconds, --debug-probe-interval-seconds, --subscribe-settle-seconds, --stream-probes-per-site, --sf-drain-messages, --slow-subscriber-events, --data-dir, --results. An unknown key is rejected rather than ignored, so a typo cannot silently change the measured scale.