ScadaBridge/docs/plans/2026-08-15-target-scale-load-test-results.md

# Target-Scale Load Test — Results (Phase-8 WP-4, deferred-work register #25 + row 50)

**Date:** 2026-08-15
**Design memo:** `docs/plans/2026-08-15-target-scale-load-test-design.md`
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`
**Closes:** deferred-work register **#25** and **row 50**; residual **7** of
`docs/plans/2026-08-15-arch-review-remediation-execution-log.md`.

> Register row 25 recorded that WP-4 had been *claimed complete* on the strength of a
> 107-byte checklist stub with no per-work-package results and no linked run. This
> document is the run that stub never had.

---

## 1. Environment

| | |
|---|---|
| Host | `Josephs-MBP` — Apple M4 Pro, 14 logical cores, 48 GB RAM |
| OS | macOS 26.5.2 (Darwin) |
| Runtime | .NET 10.0.5, **server GC enabled**, Release build |
| Branch / commit | `target-scale-load-test` @ `20f6b0b9` (harness), `8abebdae` (design memo) |
| Concurrent load | The 8-node `docker/` rig (OrbStack) remained running throughout — the user's live development cluster. Measured baseline: **~86% of one core (~6% of the box)** and ~10-16 GB RSS. The harness figures below are therefore *pessimistic*, not idealised. |
| Scale executed | **10 sites × 500 instances × 75 tags = 375,000 live tag subscriptions** |
| Offered load | 37,500 tag updates/second nominal (one update per tag per 10 s) |
| Measurement window | 20 minutes steady state, after a 2-minute settle |
| Total wall time | 1,510 s (~25 min) |

**One clean full-scale run is the dataset reported here.** A second run was started to
add raw per-sample resource capture (sharpening finding F1), but a verification build
overlapped the start of its measurement window; it was **discarded rather than
reported**, per the rule that a measurement contaminated by concurrent load is not
evidence. The raw-sample capture it was meant to exercise is now in the harness
(`HarnessRunResult.ResourceSamples`) and will be present in any future run, including
the 1-hour run in §7 — the run reported below predates that field.

---

## 2. Verdict summary

| # | Criterion | Threshold | Measured | Verdict |
|---|---|---|---|---|
| 1 | `[2.5-1]` 10 sites simultaneously operational | all 10 built, ramped, tracked centrally | 10/10 sites tracked by `CentralHealthAggregator` | **PASS** |
| 2 | `[2.5-2]` 500 instances/site, active subscriptions | 5,000 actors; `SkippedNoCallback` = 0 | 5,000 `InstanceActor`s; skipped = **0** | **PASS** |
| 3 | `[2.5-3]` 75 tags/instance (375,000 total) | 375,000 subscribed via the real DCL | 375,000 tag paths across 50 `DataConnectionActor`s | **PASS** |
| 4 | Tag update latency | P99 < 250 ms, P50 < 50 ms | **P50 0.88 ms, P95 2.83 ms, P99 4.57 ms, P99.9 16.04 ms, max 37.41 ms** (1,100,675 samples) | **PASS** (55× margin at P99) |
| 5 | `[xc-1]` message patterns under load | ≥ 95% of nominal, no unexpected loss | **37,518/s achieved vs 37,500 nominal (100.0%)**; 45,021,375 updates offered; **0 events dropped** at healthy subscribers | **PASS** |
| 6 | `[xc-2]` health reports within expected intervals | collect+ingest P99 < 1,000 ms; all sites tracked | **P99 0.31 ms, max 4.60 ms** over 440 reports; 10/10 sites | **PASS** (3,200× margin) |
| 7 | `[xc-6]` debug view without impacting site performance | snapshot P99 < 2,000 ms, 0 timeouts | **P99 2.19 ms, max 2.72 ms**, 264 completed, **0 timeouts** | **PASS** |
| 8 | `[xc-7]` S&F concurrent buffering from many instances | 20,000 messages, 25 origins, drains to 0 | 20,000 buffered at **15,368 msg/s**; residual depth **0** | **PASS** |
| 9 | Row 50 — S&F drain rate | > 500 msg/s | **3,533 msg/s** (20,000 messages in 5.66 s of active drain) | **PASS** (7× margin) |
| 10 | Row 50 — slow-subscriber backpressure | healthy = 100%; stalled loss confined to its own channel | **healthy 100.00% (4/4, zero drops)**; stalled 1.18%, all 197,028 losses in its own bounded channel | **PASS** |
| 11 | Deploy 500 instances to a site | slowest site < 120 s | **2.6 s** | **PASS** (46× margin) |
| 12 | Memory within acceptable bounds | WS slope < 20 MB/min; no monotonic heap climb | WS **+8.83 MB/min**; heap sawtooths (peak 3,625 MB vs 2,313→2,523 MB endpoints) with a **+19.52 MB/min** drift — see finding **F1** | **PASS with a caveat** |
| 13 | CPU within acceptable bounds | mean < 50% of the box | **41% of one core = 2.9% of the box** (peak 55% of one core) | **PASS** (17× margin) |

**11 clean passes, 1 pass with a documented caveat, 0 failures.** Four WP-4
sub-criteria were out of this harness's scope — see §6.

---

## 3. Measured detail

### 3.1 Deployment / ramp

```
site fixtures built     0.2s
all 5,000 instance actors   2.7s   (10 sites in parallel)
slowest single site     2.6s       (500 instances, production staggering)
```

The ramp used the production `StartupBatchSize` (20) / `StartupBatchDelayMs` (100)
pacing, i.e. 25 batches × 100 ms = 2.5 s of *deliberate* delay per site. So the 2.6 s
figure is almost entirely the configured stagger; actual construction of 500
`InstanceActor`s (each deserializing a 75-attribute `FlattenedConfiguration`, loading
static overrides from SQLite and issuing a DCL subscribe) costs ~0.1 s. Against a
120 s budget this criterion is not close to binding.

### 3.2 Tag update latency (DCL boundary → stream subscriber)

```
samples 1,100,675   mean 1.17ms
p50 0.88ms   p95 2.83ms   p99 4.57ms   p99.9 16.04ms   max 37.41ms
```

This is a true end-to-end path measurement, not a synthetic timer: the emit instant is
stamped on the `TagValue` handed to the adapter callback and travels verbatim through
`DataConnectionActor` → `TagValueUpdate.Timestamp` → `AttributeValueChanged.Timestamp`
→ the site stream → `StreamRelayActor` → the proto `SiteStreamEvent`, where the
subscriber subtracts it. Every hop is production code.

Sub-millisecond median under 37,500 updates/s across 5,000 actors means the actor
hierarchy is nowhere near its ceiling at target scale.

### 3.3 Stream delivery

```
events delivered  900,675
events dropped          0
```

Zero evictions at the 100 live subscribers across a 20-minute window at full rate.
The site stream's `DropHead` buffering exists for pathological cases; at nominal
target-scale load it never engaged.

### 3.4 Health reporting at scale

```
samples 440   mean 0.03ms   p50 0.01ms   p95 0.04ms   p99 0.31ms   max 4.60ms
reports ingested 440    sites tracked centrally 10
```

`SiteHealthCollector.CollectReport` for a site carrying 37,500 subscriptions plus
`CentralHealthAggregator.ProcessReport` costs well under a millisecond. Against the
30-second report interval there is no plausible queueing risk.

### 3.5 Debug view under load

```
samples 264   mean 0.14ms   p50 0.08ms   p99 2.19ms   max 2.72ms
completed 264   timed out 0
```

Each sample is an `Ask<DebugViewSnapshot>` landing in the mailbox of an
`InstanceActor` concurrently ingesting tag updates, so the figure includes real
queueing behind production traffic. `[xc-6]`'s "without impacting site performance"
also holds in the other direction: tag latency percentiles were measured with these
snapshots running throughout, and show no degradation.

### 3.6 Resources

```
window          1200s over 121 samples
working set     2665 -> 3261 MB (peak 3261)   slope  +8.83 MB/min
managed heap    2313 -> 2523 MB (peak 3625)   slope +19.52 MB/min
cpu mean/peak   41% / 55% of one core  (2.9% of the box)
threads         58        gen2 GCs in window: 0
```

CPU is the headline: **375,000 subscriptions at 37,500 updates/s consumed under half
of one core of fourteen.** Thread count stayed flat at 58 for ten actor systems.

---

## 4. Findings

### F1 — Memory: 20 minutes with zero gen-2 collections cannot fully settle the leak question (Low)

The working-set slope (+8.83 MB/min) is inside the threshold, and the managed heap
clearly *sawtooths* — its peak (3,625 MB) sits well above both endpoints (2,313 MB →
2,523 MB), so the collector is demonstrably reclaiming. But **`GC.CollectionCount(2)`
recorded zero gen-2 collections across the entire 25-minute run**, at 45 million
events. Gen-2 was therefore never compacted, and a positive least-squares drift on an
uncompacted heap cannot be distinguished from a genuine slow leak by this run alone.

This is a limitation of the shortened window (deviation D1), not an observed defect:
no metric degraded, no drop appeared, and latency percentiles were flat from the first
minute to the twentieth. Recorded honestly rather than reported as "no leak".

**To settle it**, run the full 1-hour protocol (§7) and check whether the heap slope
flattens once gen-2 collections begin, or issue an explicit
`GC.Collect(2, Forced, blocking: true)` at window start and end and compare.
Deliberately *not* attempted here — the brief was to measure, not to tune.

### F2 — Store-and-forward waits one full `DefaultRetryInterval` before draining a deferred backlog (informational, by design)

The drain measurement initially read as **33 msg/s**, which would have been alarming.
The progress series showed why: **nothing drained for 29-30 seconds, then the entire
backlog cleared in a fraction of a second.**

Cause, confirmed in source: `StoreAndForwardService.EnqueueAsync` with
`attemptImmediateDelivery: false` stamps `message.LastAttemptAt = UtcNow`
(`StoreAndForwardService.cs:754-757`). `GetMessagesForRetryAsync`'s due-predicate is
`last_attempt_at_ms IS NULL OR retry_interval_ms = 0 OR (now - last_attempt) >= retry_interval`,
so such a row is not due for one `DefaultRetryInterval` (30 s). Explicit
`TriggerSweep()` calls find nothing due and correctly do nothing.

This is **correct, intended behaviour** — the caller declared delivery was not
attempted, so the row waits one retry interval — but it is easy to misread as slow
drainage. The harness therefore reports the two numbers separately:

```
retry wait before drain  28.9s  (DefaultRetryInterval)
drain throughput      3,533 msg/s  (active drain 5.66s)
```

Worth knowing operationally: a site that buffers a backlog this way will appear
completely stalled for the first 30 seconds. Note the `Notify.Send` path
(`deferToSweep: true`) deliberately leaves `LastAttemptAt` null so its rows are due
immediately and skip this wait entirely.

### F3 — Slow-subscriber isolation is complete, and the mechanism is worth recording (positive result)

Register row 50 asked what a slow/stalled gRPC subscriber does to per-subscriber
buffering with several subscribers attached. With five subscribers on the *same*
instance (identical offered sequence) and one reader stalled at 50 ms/event:

| Subscriber | State | Received | Dropped | Delivery |
|---|---|---|---|---|
| slowprobe-00 | **STALLED** | 2,369 | 197,028 | 1.18% |
| slowprobe-01 | healthy | 200,000 | 0 | **100.00%** |
| slowprobe-02 | healthy | 200,000 | 0 | **100.00%** |
| slowprobe-03 | healthy | 200,000 | 0 | **100.00%** |
| slowprobe-04 | healthy | 200,000 | 0 | **100.00%** |

**Isolation is total** — not merely "good". The stalled subscriber lost 98.8% of its
own events and cost its peers exactly nothing. The reason is structural, and each link
matters:

1. The per-subscriber graph is `Where → Buffer(StreamBufferSize, DropHead) → Sink.ForEach(Tell)`.
   A `DropHead` buffer *always* accepts, so it never backpressures the `BroadcastHub`
   upstream — one subscriber cannot stall the hub for the others.
2. `Sink.ForEach` does an actor `Tell`, which never blocks.
3. `StreamRelayActor.WriteToChannel` uses `TryWrite` on a bounded `DropOldest`
   channel, which never blocks either — so the relay actor's mailbox drains regardless
   of reader speed, and loss is confined to that subscriber's own channel where it is
   counted.

The consequence for operators: a slow WAN link or wedged central client degrades
**only its own feed**, and the loss is visible in the per-stream eviction counter and
the `RecordSiteStreamEventDropped` telemetry rather than being silent.

One caveat established while building the probe: publishing an unpaced burst makes
*every* subscriber lose events, because the publish source is a single
`Source.ActorRef(StreamBufferSize, DropHead)` **shared** by all attribute subscribers,
upstream of the hub. That shared stage — not per-subscriber buffering — is the real
limit on burst absorption. The probe paces at 2,000 events/s to isolate the variable
under test; the shared stage's own capacity is separately covered by
`SiteStreamThroughputTests`.

### F4 — Pre-existing test-isolation flake in `QueueDepthGaugeTests` (Low, not introduced here)

Noticed during this work package's verification pass, recorded so it is not later
mistaken for a regression from the load-harness branch.

`StoreAndForward.Tests.QueueDepthGaugeTests.Gauge_TracksBufferedDepth_AcrossEnqueueDrainAndPark`
**fails in a full-suite run** (`Expected: 0, Actual: 2`, `QueueDepthGaugeTests.cs:116`)
but **passes when run in isolation** — the signature of a shared static gauge carrying
state across tests, i.e. an ordering dependency rather than a product defect.

It cannot originate from this branch: the branch changes **zero `src/` files** relative
to its base (`986e6e7a`), and both the StoreAndForward source and its test project are
byte-identical to that base. Left unfixed deliberately — out of scope for a measurement
work package, and the brief was explicitly not to attempt unrelated fixes. Worth a
separate triage alongside the pre-existing `SandboxTests` timing pin already recorded as
residual 6 of the arch-review remediation execution log.

---

## 5. What this does *not* prove

Stated plainly so the evidence is not over-read:

- **Not a clustered run.** Sites are single, non-clustered `ActorSystem`s (deviation
  D2). Failover, singleton handover and split-brain behaviour are covered elsewhere
  (`FailoverTimingTests`, `SbrFailoverTests`, `docker/failover-drill.sh`).
- **Not a real-network run.** The gRPC, health-transport and S&F-to-central hops are
  in-process. Serialization cost, TLS, WAN latency and socket backpressure are out of
  frame; the *stream* path's backpressure was measured deliberately by substituting a
  controllable reader for the socket (D4).
- **Not a real OPC UA run.** 375,000 monitored items were simulated at the adapter
  callback (D3). The DCL above that callback is real and fully exercised.
- **Not 1 hour.** 20 minutes (D1). See finding F1 for the one question this leaves
  genuinely open.

## 6. WP-4 criteria not covered by this harness

Recorded rather than silently dropped (deviation D7):

| Criterion | Why not covered |
|---|---|
| `[xc-3]` site event logging within 30-day / 1 GB limits | Needs a long-horizon retention run, not a load run |
| `[xc-4]` audit logging does not degrade central | Needs a central MS SQL fixture; this harness builds site runtimes only |
| `[xc-5]` template flattening/validation for large templates | Template Engine is a central-cluster concern; unrelated to the site-runtime load path |
| `[xc-8]` UI workflows remain responsive | Needs a browser + a live central cluster (Playwright territory) |

These remain open scope for WP-4 and should be tracked separately rather than being
considered closed by this run.

## 7. Reproducing

Full protocol as executed (20-minute window):

```bash
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
  --results loadharness-results.json
```

The full **1-hour** version required by the WP-4 test protocol — identical in every
other respect, and the run that would settle finding F1:

```bash
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
  --sustain-minutes 60 --results loadharness-results-1h.json
```

CI-scale smoke (~80 s), which asserts the harness still produces coherent
measurements end to end:

```bash
dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests \
  --filter "FullyQualifiedName~TargetScaleHarnessSmoke"
```

`--help`-style knobs: `--sites`, `--instances-per-site`, `--tags-per-instance`,
`--tag-update-period-seconds`, `--settle-minutes`, `--sustain-minutes`,
`--sample-seconds`, `--health-interval-seconds`, `--debug-probe-interval-seconds`,
`--subscribe-settle-seconds`, `--stream-probes-per-site`, `--sf-drain-messages`,
`--slow-subscriber-events`, `--data-dir`, `--results`. An unknown key is rejected
rather than ignored, so a typo cannot silently change the measured scale.