Joseph Doherty
da65605e41
Full-scale run executed on this machine: 10 sites x 500 instances x 75 tags =
375,000 live tag subscriptions, 37,518 updates/s achieved vs 37,500 nominal,
45,021,375 updates over a 20-minute steady-state window (M4 Pro, 14 cores, 48 GB,
with the 8-node docker rig still running so the figures are pessimistic).
11 clean passes, 1 pass with a caveat, 0 failures:
tag latency P50 0.88ms P99 4.57ms max 37.41ms (1.1M samples, end-to-end)
stream 900,675 delivered, 0 dropped at 100 live subscribers
health collect+ingest P99 0.31ms, 10/10 sites tracked
debug view P99 2.19ms, 264 completed, 0 timeouts
deploy 500 instances to a site in 2.6s
cpu 41% of ONE core = 2.9% of the box
memory working-set slope +8.83 MB/min
Three findings, reported rather than tuned away:
F1 (Low) 20 min with ZERO gen-2 collections cannot fully settle the leak
question; the heap demonstrably sawtooths but an uncompacted gen-2 makes a
positive slope ambiguous. The 1-hour run would settle it. Not tuned.
F2 (informational, by design) a deferred S&F backlog sits for one full
DefaultRetryInterval (28.9s measured) before anything drains --
EnqueueAsync(attemptImmediateDelivery:false) stamps LastAttemptAt. Easy to
misread as slow drainage, so drain is reported as two numbers: retry wait,
then 3,533 msg/s of actual capacity.
F3 (positive) slow-subscriber isolation is TOTAL: 4 healthy subscribers at
100.00% with zero drops while a peer lost 197,028/200,000 events entirely
within its own bounded channel. Mechanism recorded link by link.
Also records what the run does NOT prove (not clustered, not real-network, not
real OPC UA, not 1 hour) and the four WP-4 sub-criteria this harness does not
cover, so the evidence is not over-read.
Register rows 25 and 50 -> RESOLVED 2026-08-15; remediation execution-log
residual 7 -> resolved; phase-8-checklist WP-4 section replaced with the
measured numbers.
290 lines
15 KiB
Markdown
290 lines
15 KiB
Markdown
# Target-Scale Load Test — Results (Phase-8 WP-4, deferred-work register #25 + row 50)
|
||
|
||
**Date:** 2026-08-15
|
||
**Design memo:** `docs/plans/2026-08-15-target-scale-load-test-design.md`
|
||
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`
|
||
**Closes:** deferred-work register **#25** and **row 50**; residual **7** of
|
||
`docs/plans/2026-08-15-arch-review-remediation-execution-log.md`.
|
||
|
||
> Register row 25 recorded that WP-4 had been *claimed complete* on the strength of a
|
||
> 107-byte checklist stub with no per-work-package results and no linked run. This
|
||
> document is the run that stub never had.
|
||
|
||
---
|
||
|
||
## 1. Environment
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host | `Josephs-MBP` — Apple M4 Pro, 14 logical cores, 48 GB RAM |
|
||
| OS | macOS 26.5.2 (Darwin) |
|
||
| Runtime | .NET 10.0.5, **server GC enabled**, Release build |
|
||
| Branch / commit | `target-scale-load-test` @ `20f6b0b9` (harness), `8abebdae` (design memo) |
|
||
| Concurrent load | The 8-node `docker/` rig (OrbStack) remained running throughout — the user's live development cluster. Measured baseline: **~86% of one core (~6% of the box)** and ~10-16 GB RSS. The harness figures below are therefore *pessimistic*, not idealised. |
|
||
| Scale executed | **10 sites × 500 instances × 75 tags = 375,000 live tag subscriptions** |
|
||
| Offered load | 37,500 tag updates/second nominal (one update per tag per 10 s) |
|
||
| Measurement window | 20 minutes steady state, after a 2-minute settle |
|
||
| Total wall time | 1,510 s (~25 min) |
|
||
|
||
Two independent full-scale runs were executed. **Run 1** is the primary dataset.
|
||
**Run 2** re-ran the identical protocol with raw per-sample resource capture added
|
||
(the only harness change between them) to resolve the memory question in §4.
|
||
|
||
---
|
||
|
||
## 2. Verdict summary
|
||
|
||
| # | Criterion | Threshold | Measured | Verdict |
|
||
|---|---|---|---|---|
|
||
| 1 | `[2.5-1]` 10 sites simultaneously operational | all 10 built, ramped, tracked centrally | 10/10 sites tracked by `CentralHealthAggregator` | **PASS** |
|
||
| 2 | `[2.5-2]` 500 instances/site, active subscriptions | 5,000 actors; `SkippedNoCallback` = 0 | 5,000 `InstanceActor`s; skipped = **0** | **PASS** |
|
||
| 3 | `[2.5-3]` 75 tags/instance (375,000 total) | 375,000 subscribed via the real DCL | 375,000 tag paths across 50 `DataConnectionActor`s | **PASS** |
|
||
| 4 | Tag update latency | P99 < 250 ms, P50 < 50 ms | **P50 0.88 ms, P95 2.83 ms, P99 4.57 ms, P99.9 16.04 ms, max 37.41 ms** (1,100,675 samples) | **PASS** (55× margin at P99) |
|
||
| 5 | `[xc-1]` message patterns under load | ≥ 95% of nominal, no unexpected loss | **37,518/s achieved vs 37,500 nominal (100.0%)**; 45,021,375 updates offered; **0 events dropped** at healthy subscribers | **PASS** |
|
||
| 6 | `[xc-2]` health reports within expected intervals | collect+ingest P99 < 1,000 ms; all sites tracked | **P99 0.31 ms, max 4.60 ms** over 440 reports; 10/10 sites | **PASS** (3,200× margin) |
|
||
| 7 | `[xc-6]` debug view without impacting site performance | snapshot P99 < 2,000 ms, 0 timeouts | **P99 2.19 ms, max 2.72 ms**, 264 completed, **0 timeouts** | **PASS** |
|
||
| 8 | `[xc-7]` S&F concurrent buffering from many instances | 20,000 messages, 25 origins, drains to 0 | 20,000 buffered at **15,368 msg/s**; residual depth **0** | **PASS** |
|
||
| 9 | Row 50 — S&F drain rate | > 500 msg/s | **3,533 msg/s** (20,000 messages in 5.66 s of active drain) | **PASS** (7× margin) |
|
||
| 10 | Row 50 — slow-subscriber backpressure | healthy = 100%; stalled loss confined to its own channel | **healthy 100.00% (4/4, zero drops)**; stalled 1.18%, all 197,028 losses in its own bounded channel | **PASS** |
|
||
| 11 | Deploy 500 instances to a site | slowest site < 120 s | **2.6 s** | **PASS** (46× margin) |
|
||
| 12 | Memory within acceptable bounds | WS slope < 20 MB/min; no monotonic heap climb | WS **+8.83 MB/min**; heap sawtooths (peak 3,625 MB vs 2,313→2,523 MB endpoints) with a **+19.52 MB/min** drift — see finding **F1** | **PASS with a caveat** |
|
||
| 13 | CPU within acceptable bounds | mean < 50% of the box | **41% of one core = 2.9% of the box** (peak 55% of one core) | **PASS** (17× margin) |
|
||
|
||
**11 clean passes, 1 pass with a documented caveat, 0 failures.** Four WP-4
|
||
sub-criteria were out of this harness's scope — see §6.
|
||
|
||
---
|
||
|
||
## 3. Measured detail
|
||
|
||
### 3.1 Deployment / ramp
|
||
|
||
```
|
||
site fixtures built 0.2s
|
||
all 5,000 instance actors 2.7s (10 sites in parallel)
|
||
slowest single site 2.6s (500 instances, production staggering)
|
||
```
|
||
|
||
The ramp used the production `StartupBatchSize` (20) / `StartupBatchDelayMs` (100)
|
||
pacing, i.e. 25 batches × 100 ms = 2.5 s of *deliberate* delay per site. So the 2.6 s
|
||
figure is almost entirely the configured stagger; actual construction of 500
|
||
`InstanceActor`s (each deserializing a 75-attribute `FlattenedConfiguration`, loading
|
||
static overrides from SQLite and issuing a DCL subscribe) costs ~0.1 s. Against a
|
||
120 s budget this criterion is not close to binding.
|
||
|
||
### 3.2 Tag update latency (DCL boundary → stream subscriber)
|
||
|
||
```
|
||
samples 1,100,675 mean 1.17ms
|
||
p50 0.88ms p95 2.83ms p99 4.57ms p99.9 16.04ms max 37.41ms
|
||
```
|
||
|
||
This is a true end-to-end path measurement, not a synthetic timer: the emit instant is
|
||
stamped on the `TagValue` handed to the adapter callback and travels verbatim through
|
||
`DataConnectionActor` → `TagValueUpdate.Timestamp` → `AttributeValueChanged.Timestamp`
|
||
→ the site stream → `StreamRelayActor` → the proto `SiteStreamEvent`, where the
|
||
subscriber subtracts it. Every hop is production code.
|
||
|
||
Sub-millisecond median under 37,500 updates/s across 5,000 actors means the actor
|
||
hierarchy is nowhere near its ceiling at target scale.
|
||
|
||
### 3.3 Stream delivery
|
||
|
||
```
|
||
events delivered 900,675
|
||
events dropped 0
|
||
```
|
||
|
||
Zero evictions at the 100 live subscribers across a 20-minute window at full rate.
|
||
The site stream's `DropHead` buffering exists for pathological cases; at nominal
|
||
target-scale load it never engaged.
|
||
|
||
### 3.4 Health reporting at scale
|
||
|
||
```
|
||
samples 440 mean 0.03ms p50 0.01ms p95 0.04ms p99 0.31ms max 4.60ms
|
||
reports ingested 440 sites tracked centrally 10
|
||
```
|
||
|
||
`SiteHealthCollector.CollectReport` for a site carrying 37,500 subscriptions plus
|
||
`CentralHealthAggregator.ProcessReport` costs well under a millisecond. Against the
|
||
30-second report interval there is no plausible queueing risk.
|
||
|
||
### 3.5 Debug view under load
|
||
|
||
```
|
||
samples 264 mean 0.14ms p50 0.08ms p99 2.19ms max 2.72ms
|
||
completed 264 timed out 0
|
||
```
|
||
|
||
Each sample is an `Ask<DebugViewSnapshot>` landing in the mailbox of an
|
||
`InstanceActor` concurrently ingesting tag updates, so the figure includes real
|
||
queueing behind production traffic. `[xc-6]`'s "without impacting site performance"
|
||
also holds in the other direction: tag latency percentiles were measured with these
|
||
snapshots running throughout, and show no degradation.
|
||
|
||
### 3.6 Resources
|
||
|
||
```
|
||
window 1200s over 121 samples
|
||
working set 2665 -> 3261 MB (peak 3261) slope +8.83 MB/min
|
||
managed heap 2313 -> 2523 MB (peak 3625) slope +19.52 MB/min
|
||
cpu mean/peak 41% / 55% of one core (2.9% of the box)
|
||
threads 58 gen2 GCs in window: 0
|
||
```
|
||
|
||
CPU is the headline: **375,000 subscriptions at 37,500 updates/s consumed under half
|
||
of one core of fourteen.** Thread count stayed flat at 58 for ten actor systems.
|
||
|
||
---
|
||
|
||
## 4. Findings
|
||
|
||
### F1 — Memory: 20 minutes with zero gen-2 collections cannot fully settle the leak question (Low)
|
||
|
||
The working-set slope (+8.83 MB/min) is inside the threshold, and the managed heap
|
||
clearly *sawtooths* — its peak (3,625 MB) sits well above both endpoints (2,313 MB →
|
||
2,523 MB), so the collector is demonstrably reclaiming. But **`GC.CollectionCount(2)`
|
||
recorded zero gen-2 collections across the entire 25-minute run**, at 45 million
|
||
events. Gen-2 was therefore never compacted, and a positive least-squares drift on an
|
||
uncompacted heap cannot be distinguished from a genuine slow leak by this run alone.
|
||
|
||
This is a limitation of the shortened window (deviation D1), not an observed defect:
|
||
no metric degraded, no drop appeared, and latency percentiles were flat from the first
|
||
minute to the twentieth. Recorded honestly rather than reported as "no leak".
|
||
|
||
**To settle it**, run the full 1-hour protocol (§7) and check whether the heap slope
|
||
flattens once gen-2 collections begin, or issue an explicit
|
||
`GC.Collect(2, Forced, blocking: true)` at window start and end and compare.
|
||
Deliberately *not* attempted here — the brief was to measure, not to tune.
|
||
|
||
### F2 — Store-and-forward waits one full `DefaultRetryInterval` before draining a deferred backlog (informational, by design)
|
||
|
||
The drain measurement initially read as **33 msg/s**, which would have been alarming.
|
||
The progress series showed why: **nothing drained for 29-30 seconds, then the entire
|
||
backlog cleared in a fraction of a second.**
|
||
|
||
Cause, confirmed in source: `StoreAndForwardService.EnqueueAsync` with
|
||
`attemptImmediateDelivery: false` stamps `message.LastAttemptAt = UtcNow`
|
||
(`StoreAndForwardService.cs:754-757`). `GetMessagesForRetryAsync`'s due-predicate is
|
||
`last_attempt_at_ms IS NULL OR retry_interval_ms = 0 OR (now - last_attempt) >= retry_interval`,
|
||
so such a row is not due for one `DefaultRetryInterval` (30 s). Explicit
|
||
`TriggerSweep()` calls find nothing due and correctly do nothing.
|
||
|
||
This is **correct, intended behaviour** — the caller declared delivery was not
|
||
attempted, so the row waits one retry interval — but it is easy to misread as slow
|
||
drainage. The harness therefore reports the two numbers separately:
|
||
|
||
```
|
||
retry wait before drain 28.9s (DefaultRetryInterval)
|
||
drain throughput 3,533 msg/s (active drain 5.66s)
|
||
```
|
||
|
||
Worth knowing operationally: a site that buffers a backlog this way will appear
|
||
completely stalled for the first 30 seconds. Note the `Notify.Send` path
|
||
(`deferToSweep: true`) deliberately leaves `LastAttemptAt` null so its rows are due
|
||
immediately and skip this wait entirely.
|
||
|
||
### F3 — Slow-subscriber isolation is complete, and the mechanism is worth recording (positive result)
|
||
|
||
Register row 50 asked what a slow/stalled gRPC subscriber does to per-subscriber
|
||
buffering with several subscribers attached. With five subscribers on the *same*
|
||
instance (identical offered sequence) and one reader stalled at 50 ms/event:
|
||
|
||
| Subscriber | State | Received | Dropped | Delivery |
|
||
|---|---|---|---|---|
|
||
| slowprobe-00 | **STALLED** | 2,369 | 197,028 | 1.18% |
|
||
| slowprobe-01 | healthy | 200,000 | 0 | **100.00%** |
|
||
| slowprobe-02 | healthy | 200,000 | 0 | **100.00%** |
|
||
| slowprobe-03 | healthy | 200,000 | 0 | **100.00%** |
|
||
| slowprobe-04 | healthy | 200,000 | 0 | **100.00%** |
|
||
|
||
**Isolation is total** — not merely "good". The stalled subscriber lost 98.8% of its
|
||
own events and cost its peers exactly nothing. The reason is structural, and each link
|
||
matters:
|
||
|
||
1. The per-subscriber graph is `Where → Buffer(StreamBufferSize, DropHead) → Sink.ForEach(Tell)`.
|
||
A `DropHead` buffer *always* accepts, so it never backpressures the `BroadcastHub`
|
||
upstream — one subscriber cannot stall the hub for the others.
|
||
2. `Sink.ForEach` does an actor `Tell`, which never blocks.
|
||
3. `StreamRelayActor.WriteToChannel` uses `TryWrite` on a bounded `DropOldest`
|
||
channel, which never blocks either — so the relay actor's mailbox drains regardless
|
||
of reader speed, and loss is confined to that subscriber's own channel where it is
|
||
counted.
|
||
|
||
The consequence for operators: a slow WAN link or wedged central client degrades
|
||
**only its own feed**, and the loss is visible in the per-stream eviction counter and
|
||
the `RecordSiteStreamEventDropped` telemetry rather than being silent.
|
||
|
||
One caveat established while building the probe: publishing an unpaced burst makes
|
||
*every* subscriber lose events, because the publish source is a single
|
||
`Source.ActorRef(StreamBufferSize, DropHead)` **shared** by all attribute subscribers,
|
||
upstream of the hub. That shared stage — not per-subscriber buffering — is the real
|
||
limit on burst absorption. The probe paces at 2,000 events/s to isolate the variable
|
||
under test; the shared stage's own capacity is separately covered by
|
||
`SiteStreamThroughputTests`.
|
||
|
||
---
|
||
|
||
## 5. What this does *not* prove
|
||
|
||
Stated plainly so the evidence is not over-read:
|
||
|
||
- **Not a clustered run.** Sites are single, non-clustered `ActorSystem`s (deviation
|
||
D2). Failover, singleton handover and split-brain behaviour are covered elsewhere
|
||
(`FailoverTimingTests`, `SbrFailoverTests`, `docker/failover-drill.sh`).
|
||
- **Not a real-network run.** The gRPC, health-transport and S&F-to-central hops are
|
||
in-process. Serialization cost, TLS, WAN latency and socket backpressure are out of
|
||
frame; the *stream* path's backpressure was measured deliberately by substituting a
|
||
controllable reader for the socket (D4).
|
||
- **Not a real OPC UA run.** 375,000 monitored items were simulated at the adapter
|
||
callback (D3). The DCL above that callback is real and fully exercised.
|
||
- **Not 1 hour.** 20 minutes (D1). See finding F1 for the one question this leaves
|
||
genuinely open.
|
||
|
||
## 6. WP-4 criteria not covered by this harness
|
||
|
||
Recorded rather than silently dropped (deviation D7):
|
||
|
||
| Criterion | Why not covered |
|
||
|---|---|
|
||
| `[xc-3]` site event logging within 30-day / 1 GB limits | Needs a long-horizon retention run, not a load run |
|
||
| `[xc-4]` audit logging does not degrade central | Needs a central MS SQL fixture; this harness builds site runtimes only |
|
||
| `[xc-5]` template flattening/validation for large templates | Template Engine is a central-cluster concern; unrelated to the site-runtime load path |
|
||
| `[xc-8]` UI workflows remain responsive | Needs a browser + a live central cluster (Playwright territory) |
|
||
|
||
These remain open scope for WP-4 and should be tracked separately rather than being
|
||
considered closed by this run.
|
||
|
||
## 7. Reproducing
|
||
|
||
Full protocol as executed (20-minute window):
|
||
|
||
```bash
|
||
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
|
||
--results loadharness-results.json
|
||
```
|
||
|
||
The full **1-hour** version required by the WP-4 test protocol — identical in every
|
||
other respect, and the run that would settle finding F1:
|
||
|
||
```bash
|
||
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
|
||
--sustain-minutes 60 --results loadharness-results-1h.json
|
||
```
|
||
|
||
CI-scale smoke (~80 s), which asserts the harness still produces coherent
|
||
measurements end to end:
|
||
|
||
```bash
|
||
dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests \
|
||
--filter "FullyQualifiedName~TargetScaleHarnessSmoke"
|
||
```
|
||
|
||
`--help`-style knobs: `--sites`, `--instances-per-site`, `--tags-per-instance`,
|
||
`--tag-update-period-seconds`, `--settle-minutes`, `--sustain-minutes`,
|
||
`--sample-seconds`, `--health-interval-seconds`, `--debug-probe-interval-seconds`,
|
||
`--subscribe-settle-seconds`, `--stream-probes-per-site`, `--sf-drain-messages`,
|
||
`--slow-subscriber-events`, `--data-dir`, `--results`. An unknown key is rejected
|
||
rather than ignored, so a typo cannot silently change the measured scale.
|