ScadaBridge/docs/plans/2026-08-15-target-scale-load-test-design.md

Target-Scale Load Test — Harness Design (Phase-8 WP-4, deferred-work register #25 + row 50)

Date: 2026-08-15 Closes: deferred-work register #25 (target-scale load test) and row 50 (S&F drain rate + per-subscriber stream backpressure), and residual 7 of docs/plans/2026-08-15-arch-review-remediation-execution-log.md. Spec: docs/plans/phase-8-production-readiness.md WP-4 (:152-170) + test protocol (:314-320). Results: docs/plans/2026-08-15-target-scale-load-test-results.md. Harness: tests/ZB.MOM.WW.ScadaBridge.LoadHarness/ (+ CI smoke in tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs).

---

1. Why this exists

Register row 25 records that WP-4 was claimed complete on the strength of a 107-byte checklist stub ("Status: Complete / Tests: All passing / Build: 0 errors, 0 warnings") with no per-work-package results and no linked run. The nearest real coverage was arithmetic:

• PerformanceTests/StaggeredStartupTests.cs — TagCapacity_75TagsPer500Machines_37500Total and the 10-site distribution test compute products of integers. No actor is created.
• PerformanceTests/HealthAggregationTests.cs — feeds 10 hand-built SiteHealthReport records to a real aggregator. Real, but not under load.
• PerformanceTests/Streaming/SiteStreamThroughputTests.cs — a genuine 100k-event throughput test, but one subscriber and no instances behind it.

So the job here is not to re-assert the claim. It is to produce the evidence the claim never had, and to report what that evidence actually says — including where it is uncomfortable.

2. Feasibility constraints that shaped the design

The host is one macOS box (Apple M4 Pro, 14 cores, 48 GB) running OrbStack with the 8-node docker/ rig already up. The docker/ topology has 3 sites and cannot host 10 real site pairs at 500 instances each; a full-docker WP-4 topology is out of scope.

There is no simulated protocol adapter in the DCL. Only OpcUa and MxGateway ship. DataConnectionFactory.RegisterAdapter is the documented extension point, and tests/…DataConnectionLayer.Tests/Actors/FakeBatchDataConnection.cs establishes the fake-adapter shape.

375,000 real OPC UA monitored items is not the system under test. That would measure the OPC UA SDK. The system under test is everything above the adapter callback: the actor hierarchy, the site stream, store-and-forward, health, audit, and the streaming relay.

The established repo pattern for scale/failover validation is in-process performance tests (PerformanceTests/Failover/FailoverTimingTests.cs runs a real two-node cluster in-process via TwoNodeClusterFixture with production BuildHocon and production timings). This harness extends that pattern rather than inventing a new one.

3. Harness shape, and why

A standalone console executable (tests/ZB.MOM.WW.ScadaBridge.LoadHarness, an Exe), plus a scaled-down Category=Performance [Fact] in PerformanceTests that references it.

The reason it is not purely an xunit suite is specific and worth recording: in this repo the [Trait("Category", "Performance")] attribute enables a filter, it does not exclude by default. dotnet test ZB.MOM.WW.ScadaBridge.slnx runs the Performance project's tests — FailoverTimingTests included, at 45-90s. A 20-minute full-scale test dropped in there would be a 20-minute tax on every solution test run. Hence:

	Full protocol	CI smoke
Where	LoadHarness executable	TargetScaleHarnessSmokeTests
Scale	10 x 500 x 75 = 375,000	2 x 10 x 5 = 100
Duration	~28 min wall	~80 s
Purpose	The WP-4 numbers	Harness does not bit-rot

The smoke test asserts the shape of a healthy result (traffic flows end to end, all sites tracked centrally, S&F drains to empty, a stalled subscriber costs healthy ones nothing) so #25's evidence can be regenerated on demand rather than being a one-off.

3.1 Process topology

One process. Ten SiteRuntimeFixture instances, each owning:

• its own ActorSystem (non-clustered),
• its own LocalDb SQLite file (real AddZbLocalDb + real SiteStorageService),
• a real DataConnectionManagerActor with 5 DataConnectionActor children,
• 500 real InstanceActors, each configured with 75 data-sourced attributes,
• a real SiteStreamManager at the production StreamBufferSize (1000),
• a real StoreAndForwardStorage + StoreAndForwardService,
• a real SiteHealthCollector.

Plus one shared real CentralHealthAggregator standing in for central.

3.2 What is real and what is faked

Layer	Real / faked	Why the fake does not invalidate the measurement
OPC UA server + socket	Faked (SimulatedDataConnection)	Everything replaced is on the far side of the process boundary: socket I/O, the SDK's session/subscription machinery, the device. The system under test begins at the SubscriptionCallback invocation, which is exactly where the real adapter hands off.
DataConnectionFactory / DataConnectionManagerActor / DataConnectionActor	Real	The per-connection actor's _instancesByTag fan-out and its single-mailbox serialization point are a genuine scale surface — one of the things WP-4 must characterize. The sim adapter is registered through the documented RegisterAdapter seam.
InstanceActor	Real	The whole point. Real TagValueUpdate ingest, real type coercion, real PublishAndNotifyChildren.
SiteStreamManager	Real, production StreamBufferSize	Per-subscriber Buffer(DropHead) behaviour is under test.
StreamRelayActor + bounded DropOldest channel	Real, production capacity (1000)	This is SiteStreamGrpcServer's per-subscription machinery, reused verbatim.
gRPC socket writer	Faked (a reader task)	Deliberate: it is precisely the hop whose slowness register row 50 asks about, and a controllable reader is the only way to hold it still.
StoreAndForwardService / Storage / SQLite	Real	Drain throughput is the measurement.
S&F delivery target (central)	Faked (counting stub returning true)	What is measured is the site-local buffer's capacity, not a remote endpoint's.
SiteHealthCollector / CentralHealthAggregator	Real	CollectReport at 37,500 subscriptions is the interesting term.
IHealthReportTransport (gRPC hop)	Faked (direct call)	The transport is a documented interface seam; the cost being measured is collect + ingest.
Akka cluster membership / failover	Not exercised	Already measured on a real two-node rig by FailoverTimingTests and docker/failover-drill.sh. WP-4 asks about the hierarchy under the singleton.

3.3 The latency measurement is genuinely end to end

The driver stamps DateTimeOffset.UtcNow on the TagValue it hands the adapter callback. That instant then travels verbatim, with no re-stamping:

driver → SubscriptionCallback → DataConnectionActor (self.Tell(TagValueReceived))
       → TagValueUpdate.Timestamp        (DataConnectionActor fan-out)
       → AttributeValueChanged.Timestamp (InstanceActor.HandleTagValueUpdate copies it)
       → SiteStreamManager hub → per-subscriber Buffer → StreamRelayActor
       → proto SiteStreamEvent.AttributeChanged.Timestamp
       → subscriber reader: now - Timestamp

Everything on that path is production code. This is the harness's single most important property: the reported percentiles are not a synthetic stopwatch around a method call, they are the real DCL-boundary-to-subscriber path.

4. Load model

WP-4 fixes the scale but states no tag update rate, so one has to be chosen and justified.

• Nominal rate: one update per tag per 10 seconds. At 375,000 subscriptions that is 37,500 tag updates/second fleet-wide, 3,750/s per site, 750/s per data connection actor.
• Rationale: a plant SCADA tag that genuinely changes every 10 s is a busy tag; assuming every one of 375,000 tags does so simultaneously is a deliberately pessimistic steady state, not a typical one. It also sits comfortably above the 10k events/s floor the existing SiteStreamThroughputTests pins, so the two measurements bracket each other.
• 5 data connections per site. A site with 37,500 tags behind a single OPC UA server is not the realistic shape, and it would make one actor mailbox the entire story. Five connections (7,500 tags each) is realistic; the per-connection rate is reported so the single-connection case is derivable.
• 10 live stream subscribers per site (100 fleet-wide). Every subscriber's stream graph sees the full site event flow and filters it by instance name, so this is the fan-out multiplier on the hub — the pessimistic direction.

The driver reports EmitLagSeconds (cumulative slice overrun) and SkippedNoCallback. If the harness itself cannot offer the nominal rate, those numbers say so, and the achieved-vs-nominal ratio is published rather than the nominal being quietly reported as if achieved.

5. Metric definitions

Metric	Definition
Tag update latency	Subscriber receive instant minus the driver's emit stamp, in ms. Logarithmic histogram, 16 buckets/octave (bucket width ≤ 4.4%, so a reported percentile is within ~4.4% of truth). Mean/max are exact, not bucketed. Steady-state window only — the histogram is repointed at window start so ramp outliers cannot contaminate it.
Instance ramp / deployment at scale	Wall time to create every InstanceActor, at the production StartupBatchSize (20) / StartupBatchDelayMs (100) staggering. Both the fleet total (sites in parallel) and the slowest single site are reported; the latter is the "deploy 500 instances to a site" figure.
Memory growth	Working set and managed heap, sampled every 10 s. Reported as start→end delta, peak, and least-squares slope in MB/min over the steady-state window — because a run that sawtooths around a stable mean and one that climbs monotonically can share the same endpoint delta.
CPU	Process.TotalProcessorTime delta / wall delta, expressed as a percentage of one core (so 1400% = the whole 14-core box saturated) and also as a percentage of the box.
Health report delivery timing	SiteHealthCollector.CollectReport(siteId) + CentralHealthAggregator.ProcessReport(report), timed together, once per site per 30 s tick under full load.
Debug view latency	Ask<DebugViewSnapshot>(DebugSnapshotRequest) round-trip to a randomly chosen live InstanceActor every 5 s under full load — so the measurement includes real queueing behind production traffic.
S&F drain rate	Reported as two numbers: time-to-first-delivery (the configured retry latency) and throughput measured from the first delivery to an empty buffer (the engine's actual capacity). A drain-progress series is captured so a steady rate can be told from a stall-then-burst.
Slow-subscriber isolation	Several subscribers on the same instance (identical offered event sequence); one reader stalled at 50 ms/event. Delivery ratio per subscriber, plus per-subscriber channel eviction counts.

6. Pass/fail thresholds

Derived from the WP-4 acceptance criteria. Where the criterion is qualitative ("within acceptable time", "does not degrade"), the threshold is stated here so the verdict is falsifiable rather than a judgement call made after seeing the number.

#	Criterion (WP-4)	Threshold
1	[2.5-1] 10 sites simultaneously operational	All 10 sites built, ramped, and tracked by the central aggregator
2	[2.5-2] 500 instances/site with active subscriptions	5,000 InstanceActors alive; SkippedNoCallback = 0 in the steady window
3	[2.5-3] 75 live tags/instance (375,000 total)	375,000 tag paths subscribed through the real DCL
4	Tag update latency	P99 < 250 ms, P50 < 50 ms. (A human-facing live value that lands within a quarter second is indistinguishable from instant; the site stream is explicitly best-effort/lossy under pressure, so this bounds the delivered path.)
5	[xc-1] message patterns function under load	Achieved offered load ≥ 95% of nominal, and no unexpected event loss at healthy subscribers
6	[xc-2] health reports arrive within expected intervals	Collect+ingest P99 < 1,000 ms (must be a small fraction of the 30 s report interval, else reports would queue) and all 10 sites tracked
7	[xc-6] debug view streams without impacting site performance	Snapshot round-trip P99 < 2,000 ms, zero ask timeouts, and no measurable tag-latency penalty
8	[xc-7] S&F handles concurrent buffering from multiple instances	20,000 messages buffered concurrently from 25 origin instances with no error; buffer drains to depth 0
9	Row 50 — S&F drain rate	Drain throughput > 500 msg/s (a 20,000-message backlog clears in well under a minute once due)
10	Row 50 — slow-subscriber backpressure	Healthy subscribers' delivery ratio = 100% while a peer is stalled; the stalled subscriber's loss is confined to its own bounded channel
11	Deployment of 500 instances to a site	Slowest single site < 120 s
12	Memory within acceptable bounds	Steady-state working-set slope < 20 MB/min, and no monotonic managed-heap climb across the window
13	CPU within acceptable bounds	Steady-state mean < 50% of the box at nominal load

7. Deviations from the WP-4 protocol

Each is a deliberate, recorded trade — not an omission.

#	Deviation	Reason
D1	Sustained window shortened from 1 hour to 20 minutes at full scale	Practicality on a single shared workstation. Memory growth is reported as a slope precisely so a shorter window still answers the leak question: a leak shows as a positive slope in 20 min just as in 60. The 1-hour run is a single flag — --sustain-minutes 60 — and the exact command is recorded in the results doc.
D2	In-process, non-clustered sites rather than 10 real two-node Akka clusters	A single box cannot host 20 clustered nodes at this scale. Cluster membership/failover is already measured on a real two-node rig (FailoverTimingTests, docker/failover-drill.sh); WP-4's question is about the hierarchy under the singleton.
D3	Simulated data source instead of real OPC UA	No simulated adapter exists in the DCL, and 375k real monitored items would measure the OPC UA SDK. Injected at the documented RegisterAdapter seam, at the exact hand-off point the real adapter uses.
D4	gRPC socket replaced by a reader task on the stream path	Required by the row-50 measurement itself: a stalled subscriber must be held stalled. All of SiteStreamGrpcServer's per-subscription machinery above the socket is real.
D5	Instance Actors are /user children, not children of a DeploymentManagerActor	Constructing them directly is what lets the ramp be timed in isolation and keeps the harness independent of the deploy round-trip. Visible only as InstanceActorInitialized dead letters, which the harness suppresses. No measured path differs.
D6	Akka logging at WARNING	At 37,500 updates/s, INFO output would itself become a measured load.
D7	[xc-3] site event logging volume, [xc-4] audit-log degradation, [xc-5] template flattening, [xc-8] UI responsiveness not covered	Out of scope for this harness: the first three need central MS SQL and the Template Engine (a central-cluster fixture, not a site one), and [xc-8] needs a browser. Recorded as residual scope in the results doc rather than silently dropped.
D8	The docker/ 8-node rig stays running during the measurement	It is the user's live development cluster. Its baseline cost is measured and recorded alongside the result so the CPU figure is interpretable.

8. Honest-reporting rule

If a criterion fails, that failure is the deliverable. Findings are recorded in the results doc with the measured evidence; nothing is tuned to make a number go green, and no risky fix is attempted under cover of this work package. Anything found gets filed as a finding for separate triage.