# Target-Scale Load Test — Harness Design (Phase-8 WP-4, deferred-work register #25 + row 50)

**Date:** 2026-08-15
**Closes:** deferred-work register **#25** (target-scale load test) and **row 50**
(S&F drain rate + per-subscriber stream backpressure), and residual **7** of
`docs/plans/2026-08-15-arch-review-remediation-execution-log.md`.
**Spec:** `docs/plans/phase-8-production-readiness.md` WP-4 (`:152-170`) + test protocol (`:314-320`).
**Results:** `docs/plans/2026-08-15-target-scale-load-test-results.md`.
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (+ CI smoke in
`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`).

---

## 1. Why this exists

Register row 25 records that WP-4 was *claimed complete* on the strength of a
**107-byte checklist stub** ("Status: Complete / Tests: All passing / Build: 0 errors,
0 warnings") with no per-work-package results and no linked run. The nearest real
coverage was arithmetic:

- `PerformanceTests/StaggeredStartupTests.cs` — `TagCapacity_75TagsPer500Machines_37500Total`
  and the 10-site distribution test compute products of integers. No actor is created.
- `PerformanceTests/HealthAggregationTests.cs` — feeds 10 hand-built `SiteHealthReport`
  records to a real aggregator. Real, but not under load.
- `PerformanceTests/Streaming/SiteStreamThroughputTests.cs` — a genuine 100k-event
  throughput test, but **one subscriber** and no instances behind it.

So the job here is not to re-assert the claim. It is to produce the evidence the claim
never had, and to report what that evidence actually says — including where it is
uncomfortable.

## 2. Feasibility constraints that shaped the design

**The host is one macOS box** (Apple M4 Pro, 14 cores, 48 GB) running OrbStack with the
8-node `docker/` rig already up. The `docker/` topology has 3 sites and cannot host 10
real site pairs at 500 instances each; a full-docker WP-4 topology is out of scope.

**There is no simulated protocol adapter in the DCL.** Only `OpcUa` and `MxGateway`
ship. `DataConnectionFactory.RegisterAdapter` is the documented extension point, and
`tests/…DataConnectionLayer.Tests/Actors/FakeBatchDataConnection.cs` establishes the
fake-adapter shape.

**375,000 real OPC UA monitored items is not the system under test.** That would
measure the OPC UA SDK. The system under test is everything *above* the adapter
callback: the actor hierarchy, the site stream, store-and-forward, health, audit, and
the streaming relay.

The established repo pattern for scale/failover validation is in-process performance
tests (`PerformanceTests/Failover/FailoverTimingTests.cs` runs a real two-node cluster
in-process via `TwoNodeClusterFixture` with production `BuildHocon` and production
timings). This harness extends that pattern rather than inventing a new one.

## 3. Harness shape, and why

**A standalone console executable** (`tests/ZB.MOM.WW.ScadaBridge.LoadHarness`, an
`Exe`), **plus** a scaled-down `Category=Performance` `[Fact]` in `PerformanceTests`
that references it.

The reason it is not purely an xunit suite is specific and worth recording: in this
repo the `[Trait("Category", "Performance")]` attribute **enables a filter, it does not
exclude by default**. `dotnet test ZB.MOM.WW.ScadaBridge.slnx` runs the Performance
project's tests — `FailoverTimingTests` included, at 45-90s. A 20-minute full-scale
test dropped in there would be a 20-minute tax on every solution test run. Hence:

| | Full protocol | CI smoke |
|---|---|---|
| Where | `LoadHarness` executable | `TargetScaleHarnessSmokeTests` |
| Scale | 10 x 500 x 75 = 375,000 | 2 x 10 x 5 = 100 |
| Duration | ~28 min wall | ~80 s |
| Purpose | The WP-4 numbers | Harness does not bit-rot |

The smoke test asserts the *shape* of a healthy result (traffic flows end to end, all
sites tracked centrally, S&F drains to empty, a stalled subscriber costs healthy ones
nothing) so #25's evidence can be regenerated on demand rather than being a one-off.

### 3.1 Process topology

One process. Ten `SiteRuntimeFixture` instances, each owning:

- its own **`ActorSystem`** (non-clustered),
- its own **LocalDb SQLite file** (real `AddZbLocalDb` + real `SiteStorageService`),
- a real **`DataConnectionManagerActor`** with 5 `DataConnectionActor` children,
- 500 real **`InstanceActor`s**, each configured with 75 data-sourced attributes,
- a real **`SiteStreamManager`** at the production `StreamBufferSize` (1000),
- a real **`StoreAndForwardStorage` + `StoreAndForwardService`**,
- a real **`SiteHealthCollector`**.

Plus one shared real **`CentralHealthAggregator`** standing in for central.

### 3.2 What is real and what is faked

| Layer | Real / faked | Why the fake does not invalidate the measurement |
|---|---|---|
| OPC UA server + socket | **Faked** (`SimulatedDataConnection`) | Everything replaced is on the far side of the process boundary: socket I/O, the SDK's session/subscription machinery, the device. The system under test begins at the `SubscriptionCallback` invocation, which is exactly where the real adapter hands off. |
| `DataConnectionFactory` / `DataConnectionManagerActor` / `DataConnectionActor` | **Real** | The per-connection actor's `_instancesByTag` fan-out and its single-mailbox serialization point are a genuine scale surface — one of the things WP-4 must characterize. The sim adapter is registered through the documented `RegisterAdapter` seam. |
| `InstanceActor` | **Real** | The whole point. Real `TagValueUpdate` ingest, real type coercion, real `PublishAndNotifyChildren`. |
| `SiteStreamManager` | **Real**, production `StreamBufferSize` | Per-subscriber `Buffer(DropHead)` behaviour is under test. |
| `StreamRelayActor` + bounded `DropOldest` channel | **Real**, production capacity (1000) | This is `SiteStreamGrpcServer`'s per-subscription machinery, reused verbatim. |
| gRPC socket writer | **Faked** (a reader task) | Deliberate: it is *precisely* the hop whose slowness register row 50 asks about, and a controllable reader is the only way to hold it still. |
| `StoreAndForwardService` / `Storage` / SQLite | **Real** | Drain throughput is the measurement. |
| S&F delivery target (central) | **Faked** (counting stub returning `true`) | What is measured is the site-local buffer's capacity, not a remote endpoint's. |
| `SiteHealthCollector` / `CentralHealthAggregator` | **Real** | `CollectReport` at 37,500 subscriptions is the interesting term. |
| `IHealthReportTransport` (gRPC hop) | **Faked** (direct call) | The transport is a documented interface seam; the cost being measured is collect + ingest. |
| Akka cluster membership / failover | **Not exercised** | Already measured on a real two-node rig by `FailoverTimingTests` and `docker/failover-drill.sh`. WP-4 asks about the hierarchy *under* the singleton. |

### 3.3 The latency measurement is genuinely end to end

The driver stamps `DateTimeOffset.UtcNow` on the `TagValue` it hands the adapter
callback. That instant then travels **verbatim**, with no re-stamping:

```
driver → SubscriptionCallback → DataConnectionActor (self.Tell(TagValueReceived))
       → TagValueUpdate.Timestamp        (DataConnectionActor fan-out)
       → AttributeValueChanged.Timestamp (InstanceActor.HandleTagValueUpdate copies it)
       → SiteStreamManager hub → per-subscriber Buffer → StreamRelayActor
       → proto SiteStreamEvent.AttributeChanged.Timestamp
       → subscriber reader: now - Timestamp
```

Everything on that path is production code. This is the harness's single most
important property: the reported percentiles are not a synthetic stopwatch around a
method call, they are the real DCL-boundary-to-subscriber path.

## 4. Load model

WP-4 fixes the scale but **states no tag update rate**, so one has to be chosen and
justified.

- **Nominal rate: one update per tag per 10 seconds.** At 375,000 subscriptions that
  is **37,500 tag updates/second fleet-wide**, 3,750/s per site, 750/s per data
  connection actor.
- Rationale: a plant SCADA tag that genuinely changes every 10 s is a *busy* tag;
  assuming every one of 375,000 tags does so simultaneously is a deliberately
  pessimistic steady state, not a typical one. It also sits comfortably above the
  10k events/s floor the existing `SiteStreamThroughputTests` pins, so the two
  measurements bracket each other.
- **5 data connections per site.** A site with 37,500 tags behind a single OPC UA
  server is not the realistic shape, and it would make one actor mailbox the entire
  story. Five connections (7,500 tags each) is realistic; the per-connection rate is
  reported so the single-connection case is derivable.
- **10 live stream subscribers per site** (100 fleet-wide). Every subscriber's stream
  graph sees the full site event flow and filters it by instance name, so this is the
  fan-out multiplier on the hub — the pessimistic direction.

The driver reports `EmitLagSeconds` (cumulative slice overrun) and
`SkippedNoCallback`. If the harness itself cannot offer the nominal rate, those
numbers say so, and the achieved-vs-nominal ratio is published rather than the
nominal being quietly reported as if achieved.

## 5. Metric definitions

| Metric | Definition |
|---|---|
| **Tag update latency** | Subscriber receive instant minus the driver's emit stamp, in ms. Logarithmic histogram, 16 buckets/octave (bucket width ≤ 4.4%, so a reported percentile is within ~4.4% of truth). Mean/max are exact, not bucketed. **Steady-state window only** — the histogram is repointed at window start so ramp outliers cannot contaminate it. |
| **Instance ramp / deployment at scale** | Wall time to create every `InstanceActor`, at the production `StartupBatchSize` (20) / `StartupBatchDelayMs` (100) staggering. Both the fleet total (sites in parallel) and the **slowest single site** are reported; the latter is the "deploy 500 instances to a site" figure. |
| **Memory growth** | Working set and managed heap, sampled every 10 s. Reported as start→end delta, peak, **and** least-squares slope in MB/min over the steady-state window — because a run that sawtooths around a stable mean and one that climbs monotonically can share the same endpoint delta. |
| **CPU** | `Process.TotalProcessorTime` delta / wall delta, expressed as a percentage of **one core** (so 1400% = the whole 14-core box saturated) and also as a percentage of the box. |
| **Health report delivery timing** | `SiteHealthCollector.CollectReport(siteId)` + `CentralHealthAggregator.ProcessReport(report)`, timed together, once per site per 30 s tick under full load. |
| **Debug view latency** | `Ask<DebugViewSnapshot>(DebugSnapshotRequest)` round-trip to a randomly chosen live `InstanceActor` every 5 s under full load — so the measurement includes real queueing behind production traffic. |
| **S&F drain rate** | Reported as **two** numbers: time-to-first-delivery (the configured retry latency) and throughput measured from the **first** delivery to an empty buffer (the engine's actual capacity). A drain-progress series is captured so a steady rate can be told from a stall-then-burst. |
| **Slow-subscriber isolation** | Several subscribers on the *same* instance (identical offered event sequence); one reader stalled at 50 ms/event. Delivery ratio per subscriber, plus per-subscriber channel eviction counts. |

## 6. Pass/fail thresholds

Derived from the WP-4 acceptance criteria. Where the criterion is qualitative
("within acceptable time", "does not degrade"), the threshold is stated here so the
verdict is falsifiable rather than a judgement call made after seeing the number.

| # | Criterion (WP-4) | Threshold |
|---|---|---|
| 1 | `[2.5-1]` 10 sites simultaneously operational | All 10 sites built, ramped, and tracked by the central aggregator |
| 2 | `[2.5-2]` 500 instances/site with active subscriptions | 5,000 `InstanceActor`s alive; `SkippedNoCallback` = 0 in the steady window |
| 3 | `[2.5-3]` 75 live tags/instance (375,000 total) | 375,000 tag paths subscribed through the real DCL |
| 4 | Tag update latency | **P99 < 250 ms**, P50 < 50 ms. (A human-facing live value that lands within a quarter second is indistinguishable from instant; the site stream is explicitly best-effort/lossy under pressure, so this bounds the *delivered* path.) |
| 5 | `[xc-1]` message patterns function under load | Achieved offered load ≥ 95% of nominal, and no unexpected event loss at healthy subscribers |
| 6 | `[xc-2]` health reports arrive within expected intervals | Collect+ingest **P99 < 1,000 ms** (must be a small fraction of the 30 s report interval, else reports would queue) and all 10 sites tracked |
| 7 | `[xc-6]` debug view streams without impacting site performance | Snapshot round-trip **P99 < 2,000 ms**, zero ask timeouts, and no measurable tag-latency penalty |
| 8 | `[xc-7]` S&F handles concurrent buffering from multiple instances | 20,000 messages buffered concurrently from 25 origin instances with no error; buffer drains to depth 0 |
| 9 | Row 50 — S&F drain rate | Drain throughput **> 500 msg/s** (a 20,000-message backlog clears in well under a minute once due) |
| 10 | Row 50 — slow-subscriber backpressure | Healthy subscribers' delivery ratio **= 100%** while a peer is stalled; the stalled subscriber's loss is confined to its own bounded channel |
| 11 | Deployment of 500 instances to a site | Slowest single site **< 120 s** |
| 12 | Memory within acceptable bounds | Steady-state working-set slope **< 20 MB/min**, and no monotonic managed-heap climb across the window |
| 13 | CPU within acceptable bounds | Steady-state mean **< 50% of the box** at nominal load |

## 7. Deviations from the WP-4 protocol

Each is a deliberate, recorded trade — not an omission.

| # | Deviation | Reason |
|---|---|---|
| D1 | **Sustained window shortened from 1 hour to 20 minutes** at full scale | Practicality on a single shared workstation. Memory growth is reported as a *slope* precisely so a shorter window still answers the leak question: a leak shows as a positive slope in 20 min just as in 60. The 1-hour run is a single flag — `--sustain-minutes 60` — and the exact command is recorded in the results doc. |
| D2 | **In-process, non-clustered sites** rather than 10 real two-node Akka clusters | A single box cannot host 20 clustered nodes at this scale. Cluster membership/failover is already measured on a real two-node rig (`FailoverTimingTests`, `docker/failover-drill.sh`); WP-4's question is about the hierarchy under the singleton. |
| D3 | **Simulated data source** instead of real OPC UA | No simulated adapter exists in the DCL, and 375k real monitored items would measure the OPC UA SDK. Injected at the documented `RegisterAdapter` seam, at the exact hand-off point the real adapter uses. |
| D4 | **gRPC socket replaced by a reader task** on the stream path | Required by the row-50 measurement itself: a stalled subscriber must be *held* stalled. All of `SiteStreamGrpcServer`'s per-subscription machinery above the socket is real. |
| D5 | **Instance Actors are `/user` children**, not children of a `DeploymentManagerActor` | Constructing them directly is what lets the ramp be timed in isolation and keeps the harness independent of the deploy round-trip. Visible only as `InstanceActorInitialized` dead letters, which the harness suppresses. No measured path differs. |
| D6 | **Akka logging at WARNING** | At 37,500 updates/s, INFO output would itself become a measured load. |
| D7 | `[xc-3]` site event logging volume, `[xc-4]` audit-log degradation, `[xc-5]` template flattening, `[xc-8]` UI responsiveness **not covered** | Out of scope for this harness: the first three need central MS SQL and the Template Engine (a central-cluster fixture, not a site one), and `[xc-8]` needs a browser. Recorded as residual scope in the results doc rather than silently dropped. |
| D8 | **The `docker/` 8-node rig stays running** during the measurement | It is the user's live development cluster. Its baseline cost is measured and recorded alongside the result so the CPU figure is interpretable. |

## 8. Honest-reporting rule

If a criterion fails, that failure **is** the deliverable. Findings are recorded in the
results doc with the measured evidence; nothing is tuned to make a number go green,
and no risky fix is attempted under cover of this work package. Anything found gets
filed as a finding for separate triage.
