docs(plans): target-scale load test harness design memo (WP-4 / register #25)
Records harness architecture, the real-vs-faked table with a justification per fake, metric definitions, falsifiable pass/fail thresholds derived from the WP-4 acceptance criteria, and the eight deviations from the WP-4 protocol with reasons — including the 1-hour to 20-minute sustained-window shortening (memory reported as a slope so a shorter window still answers the leak question) and the four [xc-*] criteria this harness does not cover.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,205 @@
|
||||
# Target-Scale Load Test — Harness Design (Phase-8 WP-4, deferred-work register #25 + row 50)
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Closes:** deferred-work register **#25** (target-scale load test) and **row 50**
|
||||
(S&F drain rate + per-subscriber stream backpressure), and residual **7** of
|
||||
`docs/plans/2026-08-15-arch-review-remediation-execution-log.md`.
|
||||
**Spec:** `docs/plans/phase-8-production-readiness.md` WP-4 (`:152-170`) + test protocol (`:314-320`).
|
||||
**Results:** `docs/plans/2026-08-15-target-scale-load-test-results.md`.
|
||||
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (+ CI smoke in
|
||||
`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this exists
|
||||
|
||||
Register row 25 records that WP-4 was *claimed complete* on the strength of a
|
||||
**107-byte checklist stub** ("Status: Complete / Tests: All passing / Build: 0 errors,
|
||||
0 warnings") with no per-work-package results and no linked run. The nearest real
|
||||
coverage was arithmetic:
|
||||
|
||||
- `PerformanceTests/StaggeredStartupTests.cs` — `TagCapacity_75TagsPer500Machines_37500Total`
|
||||
and the 10-site distribution test compute products of integers. No actor is created.
|
||||
- `PerformanceTests/HealthAggregationTests.cs` — feeds 10 hand-built `SiteHealthReport`
|
||||
records to a real aggregator. Real, but not under load.
|
||||
- `PerformanceTests/Streaming/SiteStreamThroughputTests.cs` — a genuine 100k-event
|
||||
throughput test, but **one subscriber** and no instances behind it.
|
||||
|
||||
So the job here is not to re-assert the claim. It is to produce the evidence the claim
|
||||
never had, and to report what that evidence actually says — including where it is
|
||||
uncomfortable.
|
||||
|
||||
## 2. Feasibility constraints that shaped the design
|
||||
|
||||
**The host is one macOS box** (Apple M4 Pro, 14 cores, 48 GB) running OrbStack with the
|
||||
8-node `docker/` rig already up. The `docker/` topology has 3 sites and cannot host 10
|
||||
real site pairs at 500 instances each; a full-docker WP-4 topology is out of scope.
|
||||
|
||||
**There is no simulated protocol adapter in the DCL.** Only `OpcUa` and `MxGateway`
|
||||
ship. `DataConnectionFactory.RegisterAdapter` is the documented extension point, and
|
||||
`tests/…DataConnectionLayer.Tests/Actors/FakeBatchDataConnection.cs` establishes the
|
||||
fake-adapter shape.
|
||||
|
||||
**375,000 real OPC UA monitored items is not the system under test.** That would
|
||||
measure the OPC UA SDK. The system under test is everything *above* the adapter
|
||||
callback: the actor hierarchy, the site stream, store-and-forward, health, audit, and
|
||||
the streaming relay.
|
||||
|
||||
The established repo pattern for scale/failover validation is in-process performance
|
||||
tests (`PerformanceTests/Failover/FailoverTimingTests.cs` runs a real two-node cluster
|
||||
in-process via `TwoNodeClusterFixture` with production `BuildHocon` and production
|
||||
timings). This harness extends that pattern rather than inventing a new one.
|
||||
|
||||
## 3. Harness shape, and why
|
||||
|
||||
**A standalone console executable** (`tests/ZB.MOM.WW.ScadaBridge.LoadHarness`, an
|
||||
`Exe`), **plus** a scaled-down `Category=Performance` `[Fact]` in `PerformanceTests`
|
||||
that references it.
|
||||
|
||||
The reason it is not purely an xunit suite is specific and worth recording: in this
|
||||
repo the `[Trait("Category", "Performance")]` attribute **enables a filter, it does not
|
||||
exclude by default**. `dotnet test ZB.MOM.WW.ScadaBridge.slnx` runs the Performance
|
||||
project's tests — `FailoverTimingTests` included, at 45-90s. A 20-minute full-scale
|
||||
test dropped in there would be a 20-minute tax on every solution test run. Hence:
|
||||
|
||||
| | Full protocol | CI smoke |
|
||||
|---|---|---|
|
||||
| Where | `LoadHarness` executable | `TargetScaleHarnessSmokeTests` |
|
||||
| Scale | 10 x 500 x 75 = 375,000 | 2 x 10 x 5 = 100 |
|
||||
| Duration | ~28 min wall | ~80 s |
|
||||
| Purpose | The WP-4 numbers | Harness does not bit-rot |
|
||||
|
||||
The smoke test asserts the *shape* of a healthy result (traffic flows end to end, all
|
||||
sites tracked centrally, S&F drains to empty, a stalled subscriber costs healthy ones
|
||||
nothing) so #25's evidence can be regenerated on demand rather than being a one-off.
|
||||
|
||||
### 3.1 Process topology
|
||||
|
||||
One process. Ten `SiteRuntimeFixture` instances, each owning:
|
||||
|
||||
- its own **`ActorSystem`** (non-clustered),
|
||||
- its own **LocalDb SQLite file** (real `AddZbLocalDb` + real `SiteStorageService`),
|
||||
- a real **`DataConnectionManagerActor`** with 5 `DataConnectionActor` children,
|
||||
- 500 real **`InstanceActor`s**, each configured with 75 data-sourced attributes,
|
||||
- a real **`SiteStreamManager`** at the production `StreamBufferSize` (1000),
|
||||
- a real **`StoreAndForwardStorage` + `StoreAndForwardService`**,
|
||||
- a real **`SiteHealthCollector`**.
|
||||
|
||||
Plus one shared real **`CentralHealthAggregator`** standing in for central.
|
||||
|
||||
### 3.2 What is real and what is faked
|
||||
|
||||
| Layer | Real / faked | Why the fake does not invalidate the measurement |
|
||||
|---|---|---|
|
||||
| OPC UA server + socket | **Faked** (`SimulatedDataConnection`) | Everything replaced is on the far side of the process boundary: socket I/O, the SDK's session/subscription machinery, the device. The system under test begins at the `SubscriptionCallback` invocation, which is exactly where the real adapter hands off. |
|
||||
| `DataConnectionFactory` / `DataConnectionManagerActor` / `DataConnectionActor` | **Real** | The per-connection actor's `_instancesByTag` fan-out and its single-mailbox serialization point are a genuine scale surface — one of the things WP-4 must characterize. The sim adapter is registered through the documented `RegisterAdapter` seam. |
|
||||
| `InstanceActor` | **Real** | The whole point. Real `TagValueUpdate` ingest, real type coercion, real `PublishAndNotifyChildren`. |
|
||||
| `SiteStreamManager` | **Real**, production `StreamBufferSize` | Per-subscriber `Buffer(DropHead)` behaviour is under test. |
|
||||
| `StreamRelayActor` + bounded `DropOldest` channel | **Real**, production capacity (1000) | This is `SiteStreamGrpcServer`'s per-subscription machinery, reused verbatim. |
|
||||
| gRPC socket writer | **Faked** (a reader task) | Deliberate: it is *precisely* the hop whose slowness register row 50 asks about, and a controllable reader is the only way to hold it still. |
|
||||
| `StoreAndForwardService` / `Storage` / SQLite | **Real** | Drain throughput is the measurement. |
|
||||
| S&F delivery target (central) | **Faked** (counting stub returning `true`) | What is measured is the site-local buffer's capacity, not a remote endpoint's. |
|
||||
| `SiteHealthCollector` / `CentralHealthAggregator` | **Real** | `CollectReport` at 37,500 subscriptions is the interesting term. |
|
||||
| `IHealthReportTransport` (gRPC hop) | **Faked** (direct call) | The transport is a documented interface seam; the cost being measured is collect + ingest. |
|
||||
| Akka cluster membership / failover | **Not exercised** | Already measured on a real two-node rig by `FailoverTimingTests` and `docker/failover-drill.sh`. WP-4 asks about the hierarchy *under* the singleton. |
|
||||
|
||||
### 3.3 The latency measurement is genuinely end to end
|
||||
|
||||
The driver stamps `DateTimeOffset.UtcNow` on the `TagValue` it hands the adapter
|
||||
callback. That instant then travels **verbatim**, with no re-stamping:
|
||||
|
||||
```
|
||||
driver → SubscriptionCallback → DataConnectionActor (self.Tell(TagValueReceived))
|
||||
→ TagValueUpdate.Timestamp (DataConnectionActor fan-out)
|
||||
→ AttributeValueChanged.Timestamp (InstanceActor.HandleTagValueUpdate copies it)
|
||||
→ SiteStreamManager hub → per-subscriber Buffer → StreamRelayActor
|
||||
→ proto SiteStreamEvent.AttributeChanged.Timestamp
|
||||
→ subscriber reader: now - Timestamp
|
||||
```
|
||||
|
||||
Everything on that path is production code. This is the harness's single most
|
||||
important property: the reported percentiles are not a synthetic stopwatch around a
|
||||
method call, they are the real DCL-boundary-to-subscriber path.
|
||||
|
||||
## 4. Load model
|
||||
|
||||
WP-4 fixes the scale but **states no tag update rate**, so one has to be chosen and
|
||||
justified.
|
||||
|
||||
- **Nominal rate: one update per tag per 10 seconds.** At 375,000 subscriptions that
|
||||
is **37,500 tag updates/second fleet-wide**, 3,750/s per site, 750/s per data
|
||||
connection actor.
|
||||
- Rationale: a plant SCADA tag that genuinely changes every 10 s is a *busy* tag;
|
||||
assuming every one of 375,000 tags does so simultaneously is a deliberately
|
||||
pessimistic steady state, not a typical one. It also sits comfortably above the
|
||||
10k events/s floor the existing `SiteStreamThroughputTests` pins, so the two
|
||||
measurements bracket each other.
|
||||
- **5 data connections per site.** A site with 37,500 tags behind a single OPC UA
|
||||
server is not the realistic shape, and it would make one actor mailbox the entire
|
||||
story. Five connections (7,500 tags each) is realistic; the per-connection rate is
|
||||
reported so the single-connection case is derivable.
|
||||
- **10 live stream subscribers per site** (100 fleet-wide). Every subscriber's stream
|
||||
graph sees the full site event flow and filters it by instance name, so this is the
|
||||
fan-out multiplier on the hub — the pessimistic direction.
|
||||
|
||||
The driver reports `EmitLagSeconds` (cumulative slice overrun) and
|
||||
`SkippedNoCallback`. If the harness itself cannot offer the nominal rate, those
|
||||
numbers say so, and the achieved-vs-nominal ratio is published rather than the
|
||||
nominal being quietly reported as if achieved.
|
||||
|
||||
## 5. Metric definitions
|
||||
|
||||
| Metric | Definition |
|
||||
|---|---|
|
||||
| **Tag update latency** | Subscriber receive instant minus the driver's emit stamp, in ms. Logarithmic histogram, 16 buckets/octave (bucket width ≤ 4.4%, so a reported percentile is within ~4.4% of truth). Mean/max are exact, not bucketed. **Steady-state window only** — the histogram is repointed at window start so ramp outliers cannot contaminate it. |
|
||||
| **Instance ramp / deployment at scale** | Wall time to create every `InstanceActor`, at the production `StartupBatchSize` (20) / `StartupBatchDelayMs` (100) staggering. Both the fleet total (sites in parallel) and the **slowest single site** are reported; the latter is the "deploy 500 instances to a site" figure. |
|
||||
| **Memory growth** | Working set and managed heap, sampled every 10 s. Reported as start→end delta, peak, **and** least-squares slope in MB/min over the steady-state window — because a run that sawtooths around a stable mean and one that climbs monotonically can share the same endpoint delta. |
|
||||
| **CPU** | `Process.TotalProcessorTime` delta / wall delta, expressed as a percentage of **one core** (so 1400% = the whole 14-core box saturated) and also as a percentage of the box. |
|
||||
| **Health report delivery timing** | `SiteHealthCollector.CollectReport(siteId)` + `CentralHealthAggregator.ProcessReport(report)`, timed together, once per site per 30 s tick under full load. |
|
||||
| **Debug view latency** | `Ask<DebugViewSnapshot>(DebugSnapshotRequest)` round-trip to a randomly chosen live `InstanceActor` every 5 s under full load — so the measurement includes real queueing behind production traffic. |
|
||||
| **S&F drain rate** | Reported as **two** numbers: time-to-first-delivery (the configured retry latency) and throughput measured from the **first** delivery to an empty buffer (the engine's actual capacity). A drain-progress series is captured so a steady rate can be told from a stall-then-burst. |
|
||||
| **Slow-subscriber isolation** | Several subscribers on the *same* instance (identical offered event sequence); one reader stalled at 50 ms/event. Delivery ratio per subscriber, plus per-subscriber channel eviction counts. |
|
||||
|
||||
## 6. Pass/fail thresholds
|
||||
|
||||
Derived from the WP-4 acceptance criteria. Where the criterion is qualitative
|
||||
("within acceptable time", "does not degrade"), the threshold is stated here so the
|
||||
verdict is falsifiable rather than a judgement call made after seeing the number.
|
||||
|
||||
| # | Criterion (WP-4) | Threshold |
|
||||
|---|---|---|
|
||||
| 1 | `[2.5-1]` 10 sites simultaneously operational | All 10 sites built, ramped, and tracked by the central aggregator |
|
||||
| 2 | `[2.5-2]` 500 instances/site with active subscriptions | 5,000 `InstanceActor`s alive; `SkippedNoCallback` = 0 in the steady window |
|
||||
| 3 | `[2.5-3]` 75 live tags/instance (375,000 total) | 375,000 tag paths subscribed through the real DCL |
|
||||
| 4 | Tag update latency | **P99 < 250 ms**, P50 < 50 ms. (A human-facing live value that lands within a quarter second is indistinguishable from instant; the site stream is explicitly best-effort/lossy under pressure, so this bounds the *delivered* path.) |
|
||||
| 5 | `[xc-1]` message patterns function under load | Achieved offered load ≥ 95% of nominal, and no unexpected event loss at healthy subscribers |
|
||||
| 6 | `[xc-2]` health reports arrive within expected intervals | Collect+ingest **P99 < 1,000 ms** (must be a small fraction of the 30 s report interval, else reports would queue) and all 10 sites tracked |
|
||||
| 7 | `[xc-6]` debug view streams without impacting site performance | Snapshot round-trip **P99 < 2,000 ms**, zero ask timeouts, and no measurable tag-latency penalty |
|
||||
| 8 | `[xc-7]` S&F handles concurrent buffering from multiple instances | 20,000 messages buffered concurrently from 25 origin instances with no error; buffer drains to depth 0 |
|
||||
| 9 | Row 50 — S&F drain rate | Drain throughput **> 500 msg/s** (a 20,000-message backlog clears in well under a minute once due) |
|
||||
| 10 | Row 50 — slow-subscriber backpressure | Healthy subscribers' delivery ratio **= 100%** while a peer is stalled; the stalled subscriber's loss is confined to its own bounded channel |
|
||||
| 11 | Deployment of 500 instances to a site | Slowest single site **< 120 s** |
|
||||
| 12 | Memory within acceptable bounds | Steady-state working-set slope **< 20 MB/min**, and no monotonic managed-heap climb across the window |
|
||||
| 13 | CPU within acceptable bounds | Steady-state mean **< 50% of the box** at nominal load |
|
||||
|
||||
## 7. Deviations from the WP-4 protocol
|
||||
|
||||
Each is a deliberate, recorded trade — not an omission.
|
||||
|
||||
| # | Deviation | Reason |
|
||||
|---|---|---|
|
||||
| D1 | **Sustained window shortened from 1 hour to 20 minutes** at full scale | Practicality on a single shared workstation. Memory growth is reported as a *slope* precisely so a shorter window still answers the leak question: a leak shows as a positive slope in 20 min just as in 60. The 1-hour run is a single flag — `--sustain-minutes 60` — and the exact command is recorded in the results doc. |
|
||||
| D2 | **In-process, non-clustered sites** rather than 10 real two-node Akka clusters | A single box cannot host 20 clustered nodes at this scale. Cluster membership/failover is already measured on a real two-node rig (`FailoverTimingTests`, `docker/failover-drill.sh`); WP-4's question is about the hierarchy under the singleton. |
|
||||
| D3 | **Simulated data source** instead of real OPC UA | No simulated adapter exists in the DCL, and 375k real monitored items would measure the OPC UA SDK. Injected at the documented `RegisterAdapter` seam, at the exact hand-off point the real adapter uses. |
|
||||
| D4 | **gRPC socket replaced by a reader task** on the stream path | Required by the row-50 measurement itself: a stalled subscriber must be *held* stalled. All of `SiteStreamGrpcServer`'s per-subscription machinery above the socket is real. |
|
||||
| D5 | **Instance Actors are `/user` children**, not children of a `DeploymentManagerActor` | Constructing them directly is what lets the ramp be timed in isolation and keeps the harness independent of the deploy round-trip. Visible only as `InstanceActorInitialized` dead letters, which the harness suppresses. No measured path differs. |
|
||||
| D6 | **Akka logging at WARNING** | At 37,500 updates/s, INFO output would itself become a measured load. |
|
||||
| D7 | `[xc-3]` site event logging volume, `[xc-4]` audit-log degradation, `[xc-5]` template flattening, `[xc-8]` UI responsiveness **not covered** | Out of scope for this harness: the first three need central MS SQL and the Template Engine (a central-cluster fixture, not a site one), and `[xc-8]` needs a browser. Recorded as residual scope in the results doc rather than silently dropped. |
|
||||
| D8 | **The `docker/` 8-node rig stays running** during the measurement | It is the user's live development cluster. Its baseline cost is measured and recorded alongside the result so the CPU figure is interpretable. |
|
||||
|
||||
## 8. Honest-reporting rule
|
||||
|
||||
If a criterion fails, that failure **is** the deliverable. Findings are recorded in the
|
||||
results doc with the measured evidence; nothing is tuned to make a number go green,
|
||||
and no risky fix is attempted under cover of this work package. Anything found gets
|
||||
filed as a finding for separate triage.
|
||||
Reference in New Issue
Block a user