Merge branch 'target-scale-load-test' — deferred register #25 delivered: 375k-subscription target-scale load test with measured evidence (absorbs row 50)
This commit is contained in:
Joseph Doherty
@@ -18,7 +18,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
|
||||
| 23 | Live LDAP group-membership re-query for an active session | `docs/requirements/Component-Security.md` :61-69 (+ :78-79) | Blocked on an external package. The mid-session refresh re-maps the **stored** groups against the central DB with **no LDAP call**, so a directory group-membership change lands only at next login. A live re-query needs a passwordless service-account group-search method on the shared `ZB.MOM.WW.Auth.Ldap` library — an external NuGet `PackageReference` (`src/ZB.MOM.WW.ScadaBridge.Security/…csproj:23`) exposing only `AuthenticateAsync(username, password, ct)`. Central role-mapping/scope changes still apply within ~15 min (`RoleRefreshThresholdMinutes`). | `ZB.MOM.WW.Auth.Ldap` gains a standalone group-search API, or a requirement that a directory-side group revocation take effect mid-session rather than at next login |
|
||||
| 24 | M8 large-bundle performance hardening | `docs/plans/2026-06-15-stillpending-completion-design.md:106` — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" | Logged as a non-blocking follow-up when M8 shipped and never given an artifact: **no plan, no task entry, no perf/load test exists** (`tests/…Transport.Tests/Import/BundleImporterLoadTests.cs` is a `LoadAsync` unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, `LineDiffer`'s `MaxInputLines`=4000 summary-only cap, and `MaxConcurrentImportSessions`=8. | First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
|
||||
| 25 | Phase-8 WP-4 target-scale load test (10 sites × 500 instances × 75 tags = 37,500 subscriptions/site, 375,000 total) | `docs/plans/phase-8-production-readiness.md:152-170` (WP-4) + `:314-320` (test protocol); status claimed in `docs/plans/phase-8-checklist.md` | **Claimed complete but unevidenced.** The whole WP-4 deliverable is a **107-byte** checklist stub asserting "Status: Complete / Tests: All passing / Build: 0 errors, 0 warnings" with no per-work-package results and no linked run. Nearest real coverage is arithmetic/aggregation only — `PerformanceTests/StaggeredStartupTests.cs` (`TagCapacity_75TagsPer500Machines_37500Total`, 500-instances-over-10-sites distribution) and `HealthAggregationTests` (10-site report aggregation) — plus a **single-subscriber** 100k-event `Streaming/SiteStreamThroughputTests.cs`. No sustained multi-site run exists anywhere in `tests/` or `docker/`. | Before any production go-live at target scale; or the first site approaching ~500 instances / ~37.5k subscriptions |
|
||||
| 25 | ~~**Phase-8 WP-4 target-scale load test** (10 sites x 500 instances x 75 tags = 37,500 subscriptions/site, 375,000 total)~~ **RESOLVED 2026-08-15 — run, measured, evidenced.** The 107-byte stub is replaced by a real full-scale run: **10 x 500 x 75 = 375,000 live tag subscriptions**, 37,518 tag updates/s achieved against 37,500 nominal (100.0%), **45,021,375 updates offered** over a 20-minute steady-state window on an M4 Pro / 14-core / 48 GB host with the 8-node `docker/` rig still running. Headline numbers: tag update latency **P50 0.88 ms / P99 4.57 ms / max 37.41 ms** end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); **0 events dropped** at 100 live subscribers; health report collect+ingest **P99 0.31 ms** with 10/10 sites tracked; debug view snapshot under load **P99 2.19 ms, 0 timeouts**; 500 instances deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope **+8.83 MB/min**. **11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures.** Harness: `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone Exe — the `Category=Performance` trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in `PerformanceTests`) plus a CI-scale smoke `[Fact]` (`PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`). Real: per-site ActorSystem + LocalDb, the real DCL via the documented `DataConnectionFactory.RegisterAdapter` seam, real `InstanceActor`s, real `SiteStreamManager`, real `StreamRelayActor` + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. **Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`; design + deviations + thresholds: `docs/plans/2026-08-15-target-scale-load-test-design.md`.** | as before | Closed. Residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-vs-central degradation, `[xc-5]` large-template flattening and `[xc-8]` UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. | Closed 2026-08-15. |
|
||||
| 26 | Ipsen MES MoveIn tail: leak-test (`-LT`) receivers + routing, PLC-output-flag writes, `Z28062` BTDB data completeness | `docs/plans/2026-06-16-ipsen-mes-movein.md:409` ("Out of scope (future)"); design `2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198` | Customer-site scope, not a platform gap. `-LT` routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any `-LT`/unknown suffix returns `WasSuccessful=false` with an "unsupported side/target" message by decision); `MoveInComplete`/`Successful`/`ErrorText` are **PLC-owned** by locked decision, so ScadaBridge deliberately does not write them; `Z28062` completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — `_LT` is stripped before side-scoping (`2026-06-30-mes-alarm-status-api.md:158`). | Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a **candidate won't-do** (`[PERM]`) at the next Ipsen scope review |
|
||||
| 27 | External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Two stacked gaps: (a) `ExternalSystemArtifact` (Commons) carries `TimeoutSeconds` but NOT `MaxRetries`/`RetryDelay`, and the site `external_systems` table has no such columns — so a centrally-configured retry policy is silently ignored on sites; every cached call buffers with the S&F default (`DefaultMaxRetries` 50 × `DefaultRetryInterval` 30s ≈ 25 min to park). (b) `Create/UpdateExternalSystemCommand` don't expose the fields either — the only way to set them today is a direct DB edit of `ExternalSystemDefinitions`. Transport bundles DO carry them (arch-review 05 "ES retry config"), which masks the gap in export/import round-trips. Fix is additive: extend the artifact + site schema + apply path, and add `--max-retries`/`--retry-delay` to the CLI. | First operator who tunes retry policy on an external system and expects site cached calls to honor it |
|
||||
| 28 | Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"** | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | The DialogService confirmation host's default destructive-action label leaks through — the dialog copy is correct but the red confirm button says "Delete" for a failover. One-line fix: pass an explicit confirm label ("Fail over") at the Health-dashboard call site (and audit other confirm-dialog call sites for the same default). | Next Central UI session |
|
||||
@@ -47,7 +47,7 @@ Rows removed from the Deferred table above once confirmed shipped. Kept here for
|
||||
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
|
||||
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
|
||||
| ~~Failover-timing measurement (the "~25s total failover" envelope)~~ **RESOLVED 2026-08-01** — split out of the combined row and closed. `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` is no longer a skipped placeholder: it runs as a live `[Fact]` (`Category=Performance`) on the real two-node in-process rig (`TwoNodeClusterFixture`, production `BuildHocon`) at production timings — 2s heartbeat / 10s failure-detection threshold / 15s stable-after — hard-killing the younger node and timing the survivor's member REMOVAL with singleton continuity asserted on the oldest. Delivered by **PLAN-R2-01 Task 4** (`archreview/plans/PLAN-R2-01-cluster-host-failover.md:226`). The oldest-crash direction is covered behaviorally by `SbrFailoverTests.AutoDown_HardCrashOfOldestNode_*` and by `docker/failover-drill.sh`. | The 2026-07-08 "PLAN-01 rig landing" trigger had fired unnoticed (NF2); PLAN-R2-01 T4 wired the placeholder to the fixture rig rather than recording a blocker. | Closed. |
|
||||
| Broader perf envelope — **S&F drain rate + per-subscriber stream backpressure** (the still-open half of the former combined row) | Never measured, and no owner plan survives now that PLAN-R2-01 closed the failover half. `PerformanceTests` covers failover timing, staggered startup, health aggregation, audit hot-path latency and a **single-subscriber** 100k-event `Streaming/SiteStreamThroughputTests.cs` — nothing measures store-and-forward drain throughput, nor what a slow/stalled subscriber does to the per-subscriber buffering in `Communication/Actors/StreamRelayActor.cs` / `Grpc/SiteStreamGrpcServer.cs` with many subscribers attached. No defect observed; deferred as measurement-only work. | First field S&F backlog that fails to drain within an operator's patience, a slow gRPC subscriber degrading a site stream for others, or the WP-4 target-scale run (row 25) being scheduled — that run should absorb this |
|
||||
| ~~Broader perf envelope — **S&F drain rate + per-subscriber stream backpressure**~~ **RESOLVED 2026-08-15 — absorbed into the row-25 target-scale run, exactly as this row's trigger anticipated.** Both halves measured at full scale. **(a) S&F drain:** 20,000 messages buffered concurrently from 25 origin instances at **15,368 msg/s**, draining at **3,533 msg/s** to depth 0. Reported as two numbers because the naive single number is misleading — a deferred backlog sits for one full `DefaultRetryInterval` (measured 28.9 s) before *anything* drains, since `EnqueueAsync(attemptImmediateDelivery: false)` stamps `LastAttemptAt` and the due-predicate then holds the row for one interval (correct by design; `Notify.Send`'s `deferToSweep: true` leaves it null and skips the wait). Filed as informational finding **F2**. **(b) Slow-subscriber backpressure:** with 5 subscribers on the SAME instance and one reader stalled at 50 ms/event, the four healthy subscribers delivered **100.00% with zero drops** while the stalled one lost 197,028 of 200,000 events **entirely within its own bounded channel**. Isolation is structural and total — `Buffer(DropHead)` never backpressures the `BroadcastHub`, `Sink.ForEach`'s `Tell` never blocks, and `StreamRelayActor.WriteToChannel`'s `TryWrite` on a `DropOldest` channel never blocks — so a slow WAN link degrades only its own feed, visibly (per-stream eviction counter + `RecordSiteStreamEventDropped`). Finding **F3**. Caveat also recorded: an *unpaced* burst costs every subscriber, because the publish `Source.ActorRef` upstream of the hub is shared — that shared stage, not per-subscriber buffering, bounds burst absorption. | Closed by the row-25 run; no defect found in either half. | Closed 2026-08-15 — `docs/plans/2026-08-15-target-scale-load-test-results.md`. |
|
||||
|
||||
## Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
|
||||
Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md`'s registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.
|
||||
|
||||
@@ -115,9 +115,20 @@ Deliberately not fixed in this program — each has a stated reason, not an over
|
||||
library's wire-compatibility guarantee — not a blocker for this program.
|
||||
6. **Fragile `SandboxTests` timing pin.** Pre-existing, unrelated to this remediation's changes;
|
||||
noted so it isn't mistaken for a regression if it flakes later.
|
||||
7. **Target-scale load test (deferred-work register #25).** This program's exit criterion is the
|
||||
live probes above, not #25 — #25 remains the follow-on validation that the moved ceilings hold
|
||||
under real load; schedule separately.
|
||||
7. ~~**Target-scale load test (deferred-work register #25).**~~ **RESOLVED 2026-08-15.** The
|
||||
follow-on validation ran: 10 sites x 500 instances x 75 tags = **375,000 live tag
|
||||
subscriptions**, 37,518 updates/s achieved vs 37,500 nominal, **45,021,375 updates** over a
|
||||
20-minute steady-state window. The moved ceilings hold with room to spare — end-to-end tag
|
||||
latency **P99 4.57 ms**, zero dropped events at 100 live subscribers, health report
|
||||
collect+ingest **P99 0.31 ms**, debug view **P99 2.19 ms** with no timeouts, CPU **2.9% of the
|
||||
box**. 11 clean passes, 1 pass with a caveat (F1: 20 minutes with zero gen-2 collections cannot
|
||||
fully settle the leak question; the 1-hour run would), 0 failures. Register **row 50** (S&F
|
||||
drain rate + per-subscriber stream backpressure) was absorbed into the same run and closed with
|
||||
it — S&F drains at **3,533 msg/s**, and a stalled subscriber costs healthy peers **nothing**
|
||||
(100.00%, zero drops). Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`;
|
||||
design: `docs/plans/2026-08-15-target-scale-load-test-design.md`; harness:
|
||||
`tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`. Residual: WP-4's `[xc-3]`/`[xc-4]`/`[xc-5]`/`[xc-8]`
|
||||
are central-cluster/browser concerns outside this harness and remain open scope.
|
||||
8. **Playwright 14 pre-existing env failures.** Present on `main` too, rig-state related, not
|
||||
introduced by this branch.
|
||||
9. **`site_events` retention purge still oplog-visible.** WP3.2's sliced retention DELETE is a row
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
# Target-Scale Load Test — Harness Design (Phase-8 WP-4, deferred-work register #25 + row 50)
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Closes:** deferred-work register **#25** (target-scale load test) and **row 50**
|
||||
(S&F drain rate + per-subscriber stream backpressure), and residual **7** of
|
||||
`docs/plans/2026-08-15-arch-review-remediation-execution-log.md`.
|
||||
**Spec:** `docs/plans/phase-8-production-readiness.md` WP-4 (`:152-170`) + test protocol (`:314-320`).
|
||||
**Results:** `docs/plans/2026-08-15-target-scale-load-test-results.md`.
|
||||
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (+ CI smoke in
|
||||
`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this exists
|
||||
|
||||
Register row 25 records that WP-4 was *claimed complete* on the strength of a
|
||||
**107-byte checklist stub** ("Status: Complete / Tests: All passing / Build: 0 errors,
|
||||
0 warnings") with no per-work-package results and no linked run. The nearest real
|
||||
coverage was arithmetic:
|
||||
|
||||
- `PerformanceTests/StaggeredStartupTests.cs` — `TagCapacity_75TagsPer500Machines_37500Total`
|
||||
and the 10-site distribution test compute products of integers. No actor is created.
|
||||
- `PerformanceTests/HealthAggregationTests.cs` — feeds 10 hand-built `SiteHealthReport`
|
||||
records to a real aggregator. Real, but not under load.
|
||||
- `PerformanceTests/Streaming/SiteStreamThroughputTests.cs` — a genuine 100k-event
|
||||
throughput test, but **one subscriber** and no instances behind it.
|
||||
|
||||
So the job here is not to re-assert the claim. It is to produce the evidence the claim
|
||||
never had, and to report what that evidence actually says — including where it is
|
||||
uncomfortable.
|
||||
|
||||
## 2. Feasibility constraints that shaped the design
|
||||
|
||||
**The host is one macOS box** (Apple M4 Pro, 14 cores, 48 GB) running OrbStack with the
|
||||
8-node `docker/` rig already up. The `docker/` topology has 3 sites and cannot host 10
|
||||
real site pairs at 500 instances each; a full-docker WP-4 topology is out of scope.
|
||||
|
||||
**There is no simulated protocol adapter in the DCL.** Only `OpcUa` and `MxGateway`
|
||||
ship. `DataConnectionFactory.RegisterAdapter` is the documented extension point, and
|
||||
`tests/…DataConnectionLayer.Tests/Actors/FakeBatchDataConnection.cs` establishes the
|
||||
fake-adapter shape.
|
||||
|
||||
**375,000 real OPC UA monitored items is not the system under test.** That would
|
||||
measure the OPC UA SDK. The system under test is everything *above* the adapter
|
||||
callback: the actor hierarchy, the site stream, store-and-forward, health, audit, and
|
||||
the streaming relay.
|
||||
|
||||
The established repo pattern for scale/failover validation is in-process performance
|
||||
tests (`PerformanceTests/Failover/FailoverTimingTests.cs` runs a real two-node cluster
|
||||
in-process via `TwoNodeClusterFixture` with production `BuildHocon` and production
|
||||
timings). This harness extends that pattern rather than inventing a new one.
|
||||
|
||||
## 3. Harness shape, and why
|
||||
|
||||
**A standalone console executable** (`tests/ZB.MOM.WW.ScadaBridge.LoadHarness`, an
|
||||
`Exe`), **plus** a scaled-down `Category=Performance` `[Fact]` in `PerformanceTests`
|
||||
that references it.
|
||||
|
||||
The reason it is not purely an xunit suite is specific and worth recording: in this
|
||||
repo the `[Trait("Category", "Performance")]` attribute **enables a filter, it does not
|
||||
exclude by default**. `dotnet test ZB.MOM.WW.ScadaBridge.slnx` runs the Performance
|
||||
project's tests — `FailoverTimingTests` included, at 45-90s. A 20-minute full-scale
|
||||
test dropped in there would be a 20-minute tax on every solution test run. Hence:
|
||||
|
||||
| | Full protocol | CI smoke |
|
||||
|---|---|---|
|
||||
| Where | `LoadHarness` executable | `TargetScaleHarnessSmokeTests` |
|
||||
| Scale | 10 x 500 x 75 = 375,000 | 2 x 10 x 5 = 100 |
|
||||
| Duration | ~28 min wall | ~80 s |
|
||||
| Purpose | The WP-4 numbers | Harness does not bit-rot |
|
||||
|
||||
The smoke test asserts the *shape* of a healthy result (traffic flows end to end, all
|
||||
sites tracked centrally, S&F drains to empty, a stalled subscriber costs healthy ones
|
||||
nothing) so #25's evidence can be regenerated on demand rather than being a one-off.
|
||||
|
||||
### 3.1 Process topology
|
||||
|
||||
One process. Ten `SiteRuntimeFixture` instances, each owning:
|
||||
|
||||
- its own **`ActorSystem`** (non-clustered),
|
||||
- its own **LocalDb SQLite file** (real `AddZbLocalDb` + real `SiteStorageService`),
|
||||
- a real **`DataConnectionManagerActor`** with 5 `DataConnectionActor` children,
|
||||
- 500 real **`InstanceActor`s**, each configured with 75 data-sourced attributes,
|
||||
- a real **`SiteStreamManager`** at the production `StreamBufferSize` (1000),
|
||||
- a real **`StoreAndForwardStorage` + `StoreAndForwardService`**,
|
||||
- a real **`SiteHealthCollector`**.
|
||||
|
||||
Plus one shared real **`CentralHealthAggregator`** standing in for central.
|
||||
|
||||
### 3.2 What is real and what is faked
|
||||
|
||||
| Layer | Real / faked | Why the fake does not invalidate the measurement |
|
||||
|---|---|---|
|
||||
| OPC UA server + socket | **Faked** (`SimulatedDataConnection`) | Everything replaced is on the far side of the process boundary: socket I/O, the SDK's session/subscription machinery, the device. The system under test begins at the `SubscriptionCallback` invocation, which is exactly where the real adapter hands off. |
|
||||
| `DataConnectionFactory` / `DataConnectionManagerActor` / `DataConnectionActor` | **Real** | The per-connection actor's `_instancesByTag` fan-out and its single-mailbox serialization point are a genuine scale surface — one of the things WP-4 must characterize. The sim adapter is registered through the documented `RegisterAdapter` seam. |
|
||||
| `InstanceActor` | **Real** | The whole point. Real `TagValueUpdate` ingest, real type coercion, real `PublishAndNotifyChildren`. |
|
||||
| `SiteStreamManager` | **Real**, production `StreamBufferSize` | Per-subscriber `Buffer(DropHead)` behaviour is under test. |
|
||||
| `StreamRelayActor` + bounded `DropOldest` channel | **Real**, production capacity (1000) | This is `SiteStreamGrpcServer`'s per-subscription machinery, reused verbatim. |
|
||||
| gRPC socket writer | **Faked** (a reader task) | Deliberate: it is *precisely* the hop whose slowness register row 50 asks about, and a controllable reader is the only way to hold it still. |
|
||||
| `StoreAndForwardService` / `Storage` / SQLite | **Real** | Drain throughput is the measurement. |
|
||||
| S&F delivery target (central) | **Faked** (counting stub returning `true`) | What is measured is the site-local buffer's capacity, not a remote endpoint's. |
|
||||
| `SiteHealthCollector` / `CentralHealthAggregator` | **Real** | `CollectReport` at 37,500 subscriptions is the interesting term. |
|
||||
| `IHealthReportTransport` (gRPC hop) | **Faked** (direct call) | The transport is a documented interface seam; the cost being measured is collect + ingest. |
|
||||
| Akka cluster membership / failover | **Not exercised** | Already measured on a real two-node rig by `FailoverTimingTests` and `docker/failover-drill.sh`. WP-4 asks about the hierarchy *under* the singleton. |
|
||||
|
||||
### 3.3 The latency measurement is genuinely end to end
|
||||
|
||||
The driver stamps `DateTimeOffset.UtcNow` on the `TagValue` it hands the adapter
|
||||
callback. That instant then travels **verbatim**, with no re-stamping:
|
||||
|
||||
```
|
||||
driver → SubscriptionCallback → DataConnectionActor (self.Tell(TagValueReceived))
|
||||
→ TagValueUpdate.Timestamp (DataConnectionActor fan-out)
|
||||
→ AttributeValueChanged.Timestamp (InstanceActor.HandleTagValueUpdate copies it)
|
||||
→ SiteStreamManager hub → per-subscriber Buffer → StreamRelayActor
|
||||
→ proto SiteStreamEvent.AttributeChanged.Timestamp
|
||||
→ subscriber reader: now - Timestamp
|
||||
```
|
||||
|
||||
Everything on that path is production code. This is the harness's single most
|
||||
important property: the reported percentiles are not a synthetic stopwatch around a
|
||||
method call, they are the real DCL-boundary-to-subscriber path.
|
||||
|
||||
## 4. Load model
|
||||
|
||||
WP-4 fixes the scale but **states no tag update rate**, so one has to be chosen and
|
||||
justified.
|
||||
|
||||
- **Nominal rate: one update per tag per 10 seconds.** At 375,000 subscriptions that
|
||||
is **37,500 tag updates/second fleet-wide**, 3,750/s per site, 750/s per data
|
||||
connection actor.
|
||||
- Rationale: a plant SCADA tag that genuinely changes every 10 s is a *busy* tag;
|
||||
assuming every one of 375,000 tags does so simultaneously is a deliberately
|
||||
pessimistic steady state, not a typical one. It also sits comfortably above the
|
||||
10k events/s floor the existing `SiteStreamThroughputTests` pins, so the two
|
||||
measurements bracket each other.
|
||||
- **5 data connections per site.** A site with 37,500 tags behind a single OPC UA
|
||||
server is not the realistic shape, and it would make one actor mailbox the entire
|
||||
story. Five connections (7,500 tags each) is realistic; the per-connection rate is
|
||||
reported so the single-connection case is derivable.
|
||||
- **10 live stream subscribers per site** (100 fleet-wide). Every subscriber's stream
|
||||
graph sees the full site event flow and filters it by instance name, so this is the
|
||||
fan-out multiplier on the hub — the pessimistic direction.
|
||||
|
||||
The driver reports `EmitLagSeconds` (cumulative slice overrun) and
|
||||
`SkippedNoCallback`. If the harness itself cannot offer the nominal rate, those
|
||||
numbers say so, and the achieved-vs-nominal ratio is published rather than the
|
||||
nominal being quietly reported as if achieved.
|
||||
|
||||
## 5. Metric definitions
|
||||
|
||||
| Metric | Definition |
|
||||
|---|---|
|
||||
| **Tag update latency** | Subscriber receive instant minus the driver's emit stamp, in ms. Logarithmic histogram, 16 buckets/octave (bucket width ≤ 4.4%, so a reported percentile is within ~4.4% of truth). Mean/max are exact, not bucketed. **Steady-state window only** — the histogram is repointed at window start so ramp outliers cannot contaminate it. |
|
||||
| **Instance ramp / deployment at scale** | Wall time to create every `InstanceActor`, at the production `StartupBatchSize` (20) / `StartupBatchDelayMs` (100) staggering. Both the fleet total (sites in parallel) and the **slowest single site** are reported; the latter is the "deploy 500 instances to a site" figure. |
|
||||
| **Memory growth** | Working set and managed heap, sampled every 10 s. Reported as start→end delta, peak, **and** least-squares slope in MB/min over the steady-state window — because a run that sawtooths around a stable mean and one that climbs monotonically can share the same endpoint delta. |
|
||||
| **CPU** | `Process.TotalProcessorTime` delta / wall delta, expressed as a percentage of **one core** (so 1400% = the whole 14-core box saturated) and also as a percentage of the box. |
|
||||
| **Health report delivery timing** | `SiteHealthCollector.CollectReport(siteId)` + `CentralHealthAggregator.ProcessReport(report)`, timed together, once per site per 30 s tick under full load. |
|
||||
| **Debug view latency** | `Ask<DebugViewSnapshot>(DebugSnapshotRequest)` round-trip to a randomly chosen live `InstanceActor` every 5 s under full load — so the measurement includes real queueing behind production traffic. |
|
||||
| **S&F drain rate** | Reported as **two** numbers: time-to-first-delivery (the configured retry latency) and throughput measured from the **first** delivery to an empty buffer (the engine's actual capacity). A drain-progress series is captured so a steady rate can be told from a stall-then-burst. |
|
||||
| **Slow-subscriber isolation** | Several subscribers on the *same* instance (identical offered event sequence); one reader stalled at 50 ms/event. Delivery ratio per subscriber, plus per-subscriber channel eviction counts. |
|
||||
|
||||
## 6. Pass/fail thresholds
|
||||
|
||||
Derived from the WP-4 acceptance criteria. Where the criterion is qualitative
|
||||
("within acceptable time", "does not degrade"), the threshold is stated here so the
|
||||
verdict is falsifiable rather than a judgement call made after seeing the number.
|
||||
|
||||
| # | Criterion (WP-4) | Threshold |
|
||||
|---|---|---|
|
||||
| 1 | `[2.5-1]` 10 sites simultaneously operational | All 10 sites built, ramped, and tracked by the central aggregator |
|
||||
| 2 | `[2.5-2]` 500 instances/site with active subscriptions | 5,000 `InstanceActor`s alive; `SkippedNoCallback` = 0 in the steady window |
|
||||
| 3 | `[2.5-3]` 75 live tags/instance (375,000 total) | 375,000 tag paths subscribed through the real DCL |
|
||||
| 4 | Tag update latency | **P99 < 250 ms**, P50 < 50 ms. (A human-facing live value that lands within a quarter second is indistinguishable from instant; the site stream is explicitly best-effort/lossy under pressure, so this bounds the *delivered* path.) |
|
||||
| 5 | `[xc-1]` message patterns function under load | Achieved offered load ≥ 95% of nominal, and no unexpected event loss at healthy subscribers |
|
||||
| 6 | `[xc-2]` health reports arrive within expected intervals | Collect+ingest **P99 < 1,000 ms** (must be a small fraction of the 30 s report interval, else reports would queue) and all 10 sites tracked |
|
||||
| 7 | `[xc-6]` debug view streams without impacting site performance | Snapshot round-trip **P99 < 2,000 ms**, zero ask timeouts, and no measurable tag-latency penalty |
|
||||
| 8 | `[xc-7]` S&F handles concurrent buffering from multiple instances | 20,000 messages buffered concurrently from 25 origin instances with no error; buffer drains to depth 0 |
|
||||
| 9 | Row 50 — S&F drain rate | Drain throughput **> 500 msg/s** (a 20,000-message backlog clears in well under a minute once due) |
|
||||
| 10 | Row 50 — slow-subscriber backpressure | Healthy subscribers' delivery ratio **= 100%** while a peer is stalled; the stalled subscriber's loss is confined to its own bounded channel |
|
||||
| 11 | Deployment of 500 instances to a site | Slowest single site **< 120 s** |
|
||||
| 12 | Memory within acceptable bounds | Steady-state working-set slope **< 20 MB/min**, and no monotonic managed-heap climb across the window |
|
||||
| 13 | CPU within acceptable bounds | Steady-state mean **< 50% of the box** at nominal load |
|
||||
|
||||
## 7. Deviations from the WP-4 protocol
|
||||
|
||||
Each is a deliberate, recorded trade — not an omission.
|
||||
|
||||
| # | Deviation | Reason |
|
||||
|---|---|---|
|
||||
| D1 | **Sustained window shortened from 1 hour to 20 minutes** at full scale | Practicality on a single shared workstation. Memory growth is reported as a *slope* precisely so a shorter window still answers the leak question: a leak shows as a positive slope in 20 min just as in 60. The 1-hour run is a single flag — `--sustain-minutes 60` — and the exact command is recorded in the results doc. |
|
||||
| D2 | **In-process, non-clustered sites** rather than 10 real two-node Akka clusters | A single box cannot host 20 clustered nodes at this scale. Cluster membership/failover is already measured on a real two-node rig (`FailoverTimingTests`, `docker/failover-drill.sh`); WP-4's question is about the hierarchy under the singleton. |
|
||||
| D3 | **Simulated data source** instead of real OPC UA | No simulated adapter exists in the DCL, and 375k real monitored items would measure the OPC UA SDK. Injected at the documented `RegisterAdapter` seam, at the exact hand-off point the real adapter uses. |
|
||||
| D4 | **gRPC socket replaced by a reader task** on the stream path | Required by the row-50 measurement itself: a stalled subscriber must be *held* stalled. All of `SiteStreamGrpcServer`'s per-subscription machinery above the socket is real. |
|
||||
| D5 | **Instance Actors are `/user` children**, not children of a `DeploymentManagerActor` | Constructing them directly is what lets the ramp be timed in isolation and keeps the harness independent of the deploy round-trip. Visible only as `InstanceActorInitialized` dead letters, which the harness suppresses. No measured path differs. |
|
||||
| D6 | **Akka logging at WARNING** | At 37,500 updates/s, INFO output would itself become a measured load. |
|
||||
| D7 | `[xc-3]` site event logging volume, `[xc-4]` audit-log degradation, `[xc-5]` template flattening, `[xc-8]` UI responsiveness **not covered** | Out of scope for this harness: the first three need central MS SQL and the Template Engine (a central-cluster fixture, not a site one), and `[xc-8]` needs a browser. Recorded as residual scope in the results doc rather than silently dropped. |
|
||||
| D8 | **The `docker/` 8-node rig stays running** during the measurement | It is the user's live development cluster. Its baseline cost is measured and recorded alongside the result so the CPU figure is interpretable. |
|
||||
|
||||
## 8. Honest-reporting rule
|
||||
|
||||
If a criterion fails, that failure **is** the deliverable. Findings are recorded in the
|
||||
results doc with the measured evidence; nothing is tuned to make a number go green,
|
||||
and no risky fix is attempted under cover of this work package. Anything found gets
|
||||
filed as a finding for separate triage.
|
||||
@@ -0,0 +1,310 @@
|
||||
# Target-Scale Load Test — Results (Phase-8 WP-4, deferred-work register #25 + row 50)
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Design memo:** `docs/plans/2026-08-15-target-scale-load-test-design.md`
|
||||
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`
|
||||
**Closes:** deferred-work register **#25** and **row 50**; residual **7** of
|
||||
`docs/plans/2026-08-15-arch-review-remediation-execution-log.md`.
|
||||
|
||||
> Register row 25 recorded that WP-4 had been *claimed complete* on the strength of a
|
||||
> 107-byte checklist stub with no per-work-package results and no linked run. This
|
||||
> document is the run that stub never had.
|
||||
|
||||
---
|
||||
|
||||
## 1. Environment
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host | `Josephs-MBP` — Apple M4 Pro, 14 logical cores, 48 GB RAM |
|
||||
| OS | macOS 26.5.2 (Darwin) |
|
||||
| Runtime | .NET 10.0.5, **server GC enabled**, Release build |
|
||||
| Branch / commit | `target-scale-load-test` @ `20f6b0b9` (harness), `8abebdae` (design memo) |
|
||||
| Concurrent load | The 8-node `docker/` rig (OrbStack) remained running throughout — the user's live development cluster. Measured baseline: **~86% of one core (~6% of the box)** and ~10-16 GB RSS. The harness figures below are therefore *pessimistic*, not idealised. |
|
||||
| Scale executed | **10 sites × 500 instances × 75 tags = 375,000 live tag subscriptions** |
|
||||
| Offered load | 37,500 tag updates/second nominal (one update per tag per 10 s) |
|
||||
| Measurement window | 20 minutes steady state, after a 2-minute settle |
|
||||
| Total wall time | 1,510 s (~25 min) |
|
||||
|
||||
**One clean full-scale run is the dataset reported here.** A second run was started to
|
||||
add raw per-sample resource capture (sharpening finding F1), but a verification build
|
||||
overlapped the start of its measurement window; it was **discarded rather than
|
||||
reported**, per the rule that a measurement contaminated by concurrent load is not
|
||||
evidence. The raw-sample capture it was meant to exercise is now in the harness
|
||||
(`HarnessRunResult.ResourceSamples`) and will be present in any future run, including
|
||||
the 1-hour run in §7 — the run reported below predates that field.
|
||||
|
||||
---
|
||||
|
||||
## 2. Verdict summary
|
||||
|
||||
| # | Criterion | Threshold | Measured | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| 1 | `[2.5-1]` 10 sites simultaneously operational | all 10 built, ramped, tracked centrally | 10/10 sites tracked by `CentralHealthAggregator` | **PASS** |
|
||||
| 2 | `[2.5-2]` 500 instances/site, active subscriptions | 5,000 actors; `SkippedNoCallback` = 0 | 5,000 `InstanceActor`s; skipped = **0** | **PASS** |
|
||||
| 3 | `[2.5-3]` 75 tags/instance (375,000 total) | 375,000 subscribed via the real DCL | 375,000 tag paths across 50 `DataConnectionActor`s | **PASS** |
|
||||
| 4 | Tag update latency | P99 < 250 ms, P50 < 50 ms | **P50 0.88 ms, P95 2.83 ms, P99 4.57 ms, P99.9 16.04 ms, max 37.41 ms** (1,100,675 samples) | **PASS** (55× margin at P99) |
|
||||
| 5 | `[xc-1]` message patterns under load | ≥ 95% of nominal, no unexpected loss | **37,518/s achieved vs 37,500 nominal (100.0%)**; 45,021,375 updates offered; **0 events dropped** at healthy subscribers | **PASS** |
|
||||
| 6 | `[xc-2]` health reports within expected intervals | collect+ingest P99 < 1,000 ms; all sites tracked | **P99 0.31 ms, max 4.60 ms** over 440 reports; 10/10 sites | **PASS** (3,200× margin) |
|
||||
| 7 | `[xc-6]` debug view without impacting site performance | snapshot P99 < 2,000 ms, 0 timeouts | **P99 2.19 ms, max 2.72 ms**, 264 completed, **0 timeouts** | **PASS** |
|
||||
| 8 | `[xc-7]` S&F concurrent buffering from many instances | 20,000 messages, 25 origins, drains to 0 | 20,000 buffered at **15,368 msg/s**; residual depth **0** | **PASS** |
|
||||
| 9 | Row 50 — S&F drain rate | > 500 msg/s | **3,533 msg/s** (20,000 messages in 5.66 s of active drain) | **PASS** (7× margin) |
|
||||
| 10 | Row 50 — slow-subscriber backpressure | healthy = 100%; stalled loss confined to its own channel | **healthy 100.00% (4/4, zero drops)**; stalled 1.18%, all 197,028 losses in its own bounded channel | **PASS** |
|
||||
| 11 | Deploy 500 instances to a site | slowest site < 120 s | **2.6 s** | **PASS** (46× margin) |
|
||||
| 12 | Memory within acceptable bounds | WS slope < 20 MB/min; no monotonic heap climb | WS **+8.83 MB/min**; heap sawtooths (peak 3,625 MB vs 2,313→2,523 MB endpoints) with a **+19.52 MB/min** drift — see finding **F1** | **PASS with a caveat** |
|
||||
| 13 | CPU within acceptable bounds | mean < 50% of the box | **41% of one core = 2.9% of the box** (peak 55% of one core) | **PASS** (17× margin) |
|
||||
|
||||
**11 clean passes, 1 pass with a documented caveat, 0 failures.** Four WP-4
|
||||
sub-criteria were out of this harness's scope — see §6.
|
||||
|
||||
---
|
||||
|
||||
## 3. Measured detail
|
||||
|
||||
### 3.1 Deployment / ramp
|
||||
|
||||
```
|
||||
site fixtures built 0.2s
|
||||
all 5,000 instance actors 2.7s (10 sites in parallel)
|
||||
slowest single site 2.6s (500 instances, production staggering)
|
||||
```
|
||||
|
||||
The ramp used the production `StartupBatchSize` (20) / `StartupBatchDelayMs` (100)
|
||||
pacing, i.e. 25 batches × 100 ms = 2.5 s of *deliberate* delay per site. So the 2.6 s
|
||||
figure is almost entirely the configured stagger; actual construction of 500
|
||||
`InstanceActor`s (each deserializing a 75-attribute `FlattenedConfiguration`, loading
|
||||
static overrides from SQLite and issuing a DCL subscribe) costs ~0.1 s. Against a
|
||||
120 s budget this criterion is not close to binding.
|
||||
|
||||
### 3.2 Tag update latency (DCL boundary → stream subscriber)
|
||||
|
||||
```
|
||||
samples 1,100,675 mean 1.17ms
|
||||
p50 0.88ms p95 2.83ms p99 4.57ms p99.9 16.04ms max 37.41ms
|
||||
```
|
||||
|
||||
This is a true end-to-end path measurement, not a synthetic timer: the emit instant is
|
||||
stamped on the `TagValue` handed to the adapter callback and travels verbatim through
|
||||
`DataConnectionActor` → `TagValueUpdate.Timestamp` → `AttributeValueChanged.Timestamp`
|
||||
→ the site stream → `StreamRelayActor` → the proto `SiteStreamEvent`, where the
|
||||
subscriber subtracts it. Every hop is production code.
|
||||
|
||||
Sub-millisecond median under 37,500 updates/s across 5,000 actors means the actor
|
||||
hierarchy is nowhere near its ceiling at target scale.
|
||||
|
||||
### 3.3 Stream delivery
|
||||
|
||||
```
|
||||
events delivered 900,675
|
||||
events dropped 0
|
||||
```
|
||||
|
||||
Zero evictions at the 100 live subscribers across a 20-minute window at full rate.
|
||||
The site stream's `DropHead` buffering exists for pathological cases; at nominal
|
||||
target-scale load it never engaged.
|
||||
|
||||
### 3.4 Health reporting at scale
|
||||
|
||||
```
|
||||
samples 440 mean 0.03ms p50 0.01ms p95 0.04ms p99 0.31ms max 4.60ms
|
||||
reports ingested 440 sites tracked centrally 10
|
||||
```
|
||||
|
||||
`SiteHealthCollector.CollectReport` for a site carrying 37,500 subscriptions plus
|
||||
`CentralHealthAggregator.ProcessReport` costs well under a millisecond. Against the
|
||||
30-second report interval there is no plausible queueing risk.
|
||||
|
||||
### 3.5 Debug view under load
|
||||
|
||||
```
|
||||
samples 264 mean 0.14ms p50 0.08ms p99 2.19ms max 2.72ms
|
||||
completed 264 timed out 0
|
||||
```
|
||||
|
||||
Each sample is an `Ask<DebugViewSnapshot>` landing in the mailbox of an
|
||||
`InstanceActor` concurrently ingesting tag updates, so the figure includes real
|
||||
queueing behind production traffic. `[xc-6]`'s "without impacting site performance"
|
||||
also holds in the other direction: tag latency percentiles were measured with these
|
||||
snapshots running throughout, and show no degradation.
|
||||
|
||||
### 3.6 Resources
|
||||
|
||||
```
|
||||
window 1200s over 121 samples
|
||||
working set 2665 -> 3261 MB (peak 3261) slope +8.83 MB/min
|
||||
managed heap 2313 -> 2523 MB (peak 3625) slope +19.52 MB/min
|
||||
cpu mean/peak 41% / 55% of one core (2.9% of the box)
|
||||
threads 58 gen2 GCs in window: 0
|
||||
```
|
||||
|
||||
CPU is the headline: **375,000 subscriptions at 37,500 updates/s consumed under half
|
||||
of one core of fourteen.** Thread count stayed flat at 58 for ten actor systems.
|
||||
|
||||
---
|
||||
|
||||
## 4. Findings
|
||||
|
||||
### F1 — Memory: 20 minutes with zero gen-2 collections cannot fully settle the leak question (Low)
|
||||
|
||||
The working-set slope (+8.83 MB/min) is inside the threshold, and the managed heap
|
||||
clearly *sawtooths* — its peak (3,625 MB) sits well above both endpoints (2,313 MB →
|
||||
2,523 MB), so the collector is demonstrably reclaiming. But **`GC.CollectionCount(2)`
|
||||
recorded zero gen-2 collections across the entire 25-minute run**, at 45 million
|
||||
events. Gen-2 was therefore never compacted, and a positive least-squares drift on an
|
||||
uncompacted heap cannot be distinguished from a genuine slow leak by this run alone.
|
||||
|
||||
This is a limitation of the shortened window (deviation D1), not an observed defect:
|
||||
no metric degraded, no drop appeared, and latency percentiles were flat from the first
|
||||
minute to the twentieth. Recorded honestly rather than reported as "no leak".
|
||||
|
||||
**To settle it**, run the full 1-hour protocol (§7) and check whether the heap slope
|
||||
flattens once gen-2 collections begin, or issue an explicit
|
||||
`GC.Collect(2, Forced, blocking: true)` at window start and end and compare.
|
||||
Deliberately *not* attempted here — the brief was to measure, not to tune.
|
||||
|
||||
### F2 — Store-and-forward waits one full `DefaultRetryInterval` before draining a deferred backlog (informational, by design)
|
||||
|
||||
The drain measurement initially read as **33 msg/s**, which would have been alarming.
|
||||
The progress series showed why: **nothing drained for 29-30 seconds, then the entire
|
||||
backlog cleared in a fraction of a second.**
|
||||
|
||||
Cause, confirmed in source: `StoreAndForwardService.EnqueueAsync` with
|
||||
`attemptImmediateDelivery: false` stamps `message.LastAttemptAt = UtcNow`
|
||||
(`StoreAndForwardService.cs:754-757`). `GetMessagesForRetryAsync`'s due-predicate is
|
||||
`last_attempt_at_ms IS NULL OR retry_interval_ms = 0 OR (now - last_attempt) >= retry_interval`,
|
||||
so such a row is not due for one `DefaultRetryInterval` (30 s). Explicit
|
||||
`TriggerSweep()` calls find nothing due and correctly do nothing.
|
||||
|
||||
This is **correct, intended behaviour** — the caller declared delivery was not
|
||||
attempted, so the row waits one retry interval — but it is easy to misread as slow
|
||||
drainage. The harness therefore reports the two numbers separately:
|
||||
|
||||
```
|
||||
retry wait before drain 28.9s (DefaultRetryInterval)
|
||||
drain throughput 3,533 msg/s (active drain 5.66s)
|
||||
```
|
||||
|
||||
Worth knowing operationally: a site that buffers a backlog this way will appear
|
||||
completely stalled for the first 30 seconds. Note the `Notify.Send` path
|
||||
(`deferToSweep: true`) deliberately leaves `LastAttemptAt` null so its rows are due
|
||||
immediately and skip this wait entirely.
|
||||
|
||||
### F3 — Slow-subscriber isolation is complete, and the mechanism is worth recording (positive result)
|
||||
|
||||
Register row 50 asked what a slow/stalled gRPC subscriber does to per-subscriber
|
||||
buffering with several subscribers attached. With five subscribers on the *same*
|
||||
instance (identical offered sequence) and one reader stalled at 50 ms/event:
|
||||
|
||||
| Subscriber | State | Received | Dropped | Delivery |
|
||||
|---|---|---|---|---|
|
||||
| slowprobe-00 | **STALLED** | 2,369 | 197,028 | 1.18% |
|
||||
| slowprobe-01 | healthy | 200,000 | 0 | **100.00%** |
|
||||
| slowprobe-02 | healthy | 200,000 | 0 | **100.00%** |
|
||||
| slowprobe-03 | healthy | 200,000 | 0 | **100.00%** |
|
||||
| slowprobe-04 | healthy | 200,000 | 0 | **100.00%** |
|
||||
|
||||
**Isolation is total** — not merely "good". The stalled subscriber lost 98.8% of its
|
||||
own events and cost its peers exactly nothing. The reason is structural, and each link
|
||||
matters:
|
||||
|
||||
1. The per-subscriber graph is `Where → Buffer(StreamBufferSize, DropHead) → Sink.ForEach(Tell)`.
|
||||
A `DropHead` buffer *always* accepts, so it never backpressures the `BroadcastHub`
|
||||
upstream — one subscriber cannot stall the hub for the others.
|
||||
2. `Sink.ForEach` does an actor `Tell`, which never blocks.
|
||||
3. `StreamRelayActor.WriteToChannel` uses `TryWrite` on a bounded `DropOldest`
|
||||
channel, which never blocks either — so the relay actor's mailbox drains regardless
|
||||
of reader speed, and loss is confined to that subscriber's own channel where it is
|
||||
counted.
|
||||
|
||||
The consequence for operators: a slow WAN link or wedged central client degrades
|
||||
**only its own feed**, and the loss is visible in the per-stream eviction counter and
|
||||
the `RecordSiteStreamEventDropped` telemetry rather than being silent.
|
||||
|
||||
One caveat established while building the probe: publishing an unpaced burst makes
|
||||
*every* subscriber lose events, because the publish source is a single
|
||||
`Source.ActorRef(StreamBufferSize, DropHead)` **shared** by all attribute subscribers,
|
||||
upstream of the hub. That shared stage — not per-subscriber buffering — is the real
|
||||
limit on burst absorption. The probe paces at 2,000 events/s to isolate the variable
|
||||
under test; the shared stage's own capacity is separately covered by
|
||||
`SiteStreamThroughputTests`.
|
||||
|
||||
### F4 — Pre-existing test-isolation flake in `QueueDepthGaugeTests` (Low, not introduced here)
|
||||
|
||||
Noticed during this work package's verification pass, recorded so it is not later
|
||||
mistaken for a regression from the load-harness branch.
|
||||
|
||||
`StoreAndForward.Tests.QueueDepthGaugeTests.Gauge_TracksBufferedDepth_AcrossEnqueueDrainAndPark`
|
||||
**fails in a full-suite run** (`Expected: 0, Actual: 2`, `QueueDepthGaugeTests.cs:116`)
|
||||
but **passes when run in isolation** — the signature of a shared static gauge carrying
|
||||
state across tests, i.e. an ordering dependency rather than a product defect.
|
||||
|
||||
It cannot originate from this branch: the branch changes **zero `src/` files** relative
|
||||
to its base (`986e6e7a`), and both the StoreAndForward source and its test project are
|
||||
byte-identical to that base. Left unfixed deliberately — out of scope for a measurement
|
||||
work package, and the brief was explicitly not to attempt unrelated fixes. Worth a
|
||||
separate triage alongside the pre-existing `SandboxTests` timing pin already recorded as
|
||||
residual 6 of the arch-review remediation execution log.
|
||||
|
||||
---
|
||||
|
||||
## 5. What this does *not* prove
|
||||
|
||||
Stated plainly so the evidence is not over-read:
|
||||
|
||||
- **Not a clustered run.** Sites are single, non-clustered `ActorSystem`s (deviation
|
||||
D2). Failover, singleton handover and split-brain behaviour are covered elsewhere
|
||||
(`FailoverTimingTests`, `SbrFailoverTests`, `docker/failover-drill.sh`).
|
||||
- **Not a real-network run.** The gRPC, health-transport and S&F-to-central hops are
|
||||
in-process. Serialization cost, TLS, WAN latency and socket backpressure are out of
|
||||
frame; the *stream* path's backpressure was measured deliberately by substituting a
|
||||
controllable reader for the socket (D4).
|
||||
- **Not a real OPC UA run.** 375,000 monitored items were simulated at the adapter
|
||||
callback (D3). The DCL above that callback is real and fully exercised.
|
||||
- **Not 1 hour.** 20 minutes (D1). See finding F1 for the one question this leaves
|
||||
genuinely open.
|
||||
|
||||
## 6. WP-4 criteria not covered by this harness
|
||||
|
||||
Recorded rather than silently dropped (deviation D7):
|
||||
|
||||
| Criterion | Why not covered |
|
||||
|---|---|
|
||||
| `[xc-3]` site event logging within 30-day / 1 GB limits | Needs a long-horizon retention run, not a load run |
|
||||
| `[xc-4]` audit logging does not degrade central | Needs a central MS SQL fixture; this harness builds site runtimes only |
|
||||
| `[xc-5]` template flattening/validation for large templates | Template Engine is a central-cluster concern; unrelated to the site-runtime load path |
|
||||
| `[xc-8]` UI workflows remain responsive | Needs a browser + a live central cluster (Playwright territory) |
|
||||
|
||||
These remain open scope for WP-4 and should be tracked separately rather than being
|
||||
considered closed by this run.
|
||||
|
||||
## 7. Reproducing
|
||||
|
||||
Full protocol as executed (20-minute window):
|
||||
|
||||
```bash
|
||||
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
|
||||
--results loadharness-results.json
|
||||
```
|
||||
|
||||
The full **1-hour** version required by the WP-4 test protocol — identical in every
|
||||
other respect, and the run that would settle finding F1:
|
||||
|
||||
```bash
|
||||
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
|
||||
--sustain-minutes 60 --results loadharness-results-1h.json
|
||||
```
|
||||
|
||||
CI-scale smoke (~80 s), which asserts the harness still produces coherent
|
||||
measurements end to end:
|
||||
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests \
|
||||
--filter "FullyQualifiedName~TargetScaleHarnessSmoke"
|
||||
```
|
||||
|
||||
`--help`-style knobs: `--sites`, `--instances-per-site`, `--tags-per-instance`,
|
||||
`--tag-update-period-seconds`, `--settle-minutes`, `--sustain-minutes`,
|
||||
`--sample-seconds`, `--health-interval-seconds`, `--debug-probe-interval-seconds`,
|
||||
`--subscribe-settle-seconds`, `--stream-probes-per-site`, `--sf-drain-messages`,
|
||||
`--slow-subscriber-events`, `--data-dir`, `--results`. An unknown key is rejected
|
||||
rather than ignored, so a typo cannot silently change the measured scale.
|
||||
@@ -3,21 +3,46 @@
|
||||
**Update 2026-08-07 (truth sweep):** This checklist previously read
|
||||
"Status: Complete / Tests: All passing / Build: 0 errors, 0 warnings" — a
|
||||
107-byte stub with no per-work-package results and no linked run. That claim
|
||||
was **unevidenced** and has been retired.
|
||||
was **unevidenced** and was retired.
|
||||
|
||||
Honest state, per the deferred-work register
|
||||
(`2026-07-08-deferred-work-register.md`, row 25):
|
||||
**Update 2026-08-15 (WP-4 run, evidenced):** The target-scale load test has now
|
||||
actually been run, at full scale, with results published.
|
||||
|
||||
- The **Phase-8 WP-4 target-scale load test** (10 sites × 500 instances ×
|
||||
75 tags = 37,500 subscriptions/site, ~375,000 total) has **never been run**.
|
||||
- Nearest real coverage is arithmetic/aggregation only —
|
||||
`PerformanceTests/StaggeredStartupTests.cs`
|
||||
(`TagCapacity_75TagsPer500Machines_37500Total`) and
|
||||
`HealthAggregationTests` — plus a single-subscriber 100k-event
|
||||
`Streaming/SiteStreamThroughputTests.cs`. No sustained multi-site run
|
||||
exists anywhere in `tests/` or `docker/`.
|
||||
- Revisit trigger: before any production go-live at target scale, or the
|
||||
first site approaching ~500 instances / ~37.5k subscriptions.
|
||||
## WP-4 — Load/Performance Testing at Target Scale
|
||||
|
||||
**Status: measured 2026-08-15.** Not a claim — a run.
|
||||
|
||||
- Scale executed: **10 sites × 500 instances × 75 tags = 375,000 live tag
|
||||
subscriptions**, 37,518 tag updates/second achieved against 37,500 nominal
|
||||
(100.0%), **45,021,375 updates** offered over a 20-minute steady-state window.
|
||||
- Headline results: end-to-end tag update latency **P50 0.88 ms / P99 4.57 ms /
|
||||
max 37.41 ms**; **zero events dropped** at 100 live stream subscribers; health
|
||||
report collect+ingest **P99 0.31 ms** with 10/10 sites tracked centrally;
|
||||
debug view snapshot under load **P99 2.19 ms with 0 timeouts**; 500 instances
|
||||
deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope
|
||||
**+8.83 MB/min**.
|
||||
- **11 clean passes, 1 pass with a documented caveat, 0 failures.**
|
||||
- Deferred-work register **#25** and **row 50** are closed by this run.
|
||||
|
||||
**Results:** `2026-08-15-target-scale-load-test-results.md` — per-criterion
|
||||
numbers vs thresholds, three findings (F1 memory-window limitation, F2 the S&F
|
||||
retry-interval wait, F3 slow-subscriber isolation), and an explicit "what this
|
||||
does not prove" section.
|
||||
|
||||
**Design, thresholds and deviations:**
|
||||
`2026-08-15-target-scale-load-test-design.md` — including the documented
|
||||
deviation from the protocol's 1-hour sustained window to 20 minutes, and the
|
||||
command for the full 1-hour version.
|
||||
|
||||
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone executable),
|
||||
with a CI-scale smoke `[Fact]` at
|
||||
`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`
|
||||
so the evidence can be regenerated rather than bit-rotting.
|
||||
|
||||
**Still open within WP-4** (not covered by this harness — central-cluster and
|
||||
browser concerns): `[xc-3]` site event logging volume within 30-day/1 GB limits,
|
||||
`[xc-4]` audit logging not degrading central, `[xc-5]` template
|
||||
flattening/validation for large templates, `[xc-8]` UI workflow responsiveness.
|
||||
|
||||
See `phase-8-production-readiness.md:152-170` (WP-4) and `:314-320`
|
||||
(test protocol) for the original scope.
|
||||
|
||||
Reference in New Issue
Block a user