Merge branch 'target-scale-load-test' — deferred register #25 delivered: 375k-subscription target-scale load test with measured evidence (absorbs row 50)
This commit is contained in:
Joseph Doherty
@@ -53,6 +53,7 @@
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ZB.MOM.WW.ScadaBridge.CLI.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/ZB.MOM.WW.ScadaBridge.PerformanceTests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.LoadHarness/ZB.MOM.WW.ScadaBridge.LoadHarness.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.CentralUI.PlaywrightTests/ZB.MOM.WW.ScadaBridge.CentralUI.PlaywrightTests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.Transport.Tests/ZB.MOM.WW.ScadaBridge.Transport.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests/ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests.csproj" />
|
||||
|
||||
@@ -18,7 +18,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
|
||||
| 23 | Live LDAP group-membership re-query for an active session | `docs/requirements/Component-Security.md` :61-69 (+ :78-79) | Blocked on an external package. The mid-session refresh re-maps the **stored** groups against the central DB with **no LDAP call**, so a directory group-membership change lands only at next login. A live re-query needs a passwordless service-account group-search method on the shared `ZB.MOM.WW.Auth.Ldap` library — an external NuGet `PackageReference` (`src/ZB.MOM.WW.ScadaBridge.Security/…csproj:23`) exposing only `AuthenticateAsync(username, password, ct)`. Central role-mapping/scope changes still apply within ~15 min (`RoleRefreshThresholdMinutes`). | `ZB.MOM.WW.Auth.Ldap` gains a standalone group-search API, or a requirement that a directory-side group revocation take effect mid-session rather than at next login |
|
||||
| 24 | M8 large-bundle performance hardening | `docs/plans/2026-06-15-stillpending-completion-design.md:106` — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" | Logged as a non-blocking follow-up when M8 shipped and never given an artifact: **no plan, no task entry, no perf/load test exists** (`tests/…Transport.Tests/Import/BundleImporterLoadTests.cs` is a `LoadAsync` unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, `LineDiffer`'s `MaxInputLines`=4000 summary-only cap, and `MaxConcurrentImportSessions`=8. | First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
|
||||
| 25 | Phase-8 WP-4 target-scale load test (10 sites × 500 instances × 75 tags = 37,500 subscriptions/site, 375,000 total) | `docs/plans/phase-8-production-readiness.md:152-170` (WP-4) + `:314-320` (test protocol); status claimed in `docs/plans/phase-8-checklist.md` | **Claimed complete but unevidenced.** The whole WP-4 deliverable is a **107-byte** checklist stub asserting "Status: Complete / Tests: All passing / Build: 0 errors, 0 warnings" with no per-work-package results and no linked run. Nearest real coverage is arithmetic/aggregation only — `PerformanceTests/StaggeredStartupTests.cs` (`TagCapacity_75TagsPer500Machines_37500Total`, 500-instances-over-10-sites distribution) and `HealthAggregationTests` (10-site report aggregation) — plus a **single-subscriber** 100k-event `Streaming/SiteStreamThroughputTests.cs`. No sustained multi-site run exists anywhere in `tests/` or `docker/`. | Before any production go-live at target scale; or the first site approaching ~500 instances / ~37.5k subscriptions |
|
||||
| 25 | ~~**Phase-8 WP-4 target-scale load test** (10 sites x 500 instances x 75 tags = 37,500 subscriptions/site, 375,000 total)~~ **RESOLVED 2026-08-15 — run, measured, evidenced.** The 107-byte stub is replaced by a real full-scale run: **10 x 500 x 75 = 375,000 live tag subscriptions**, 37,518 tag updates/s achieved against 37,500 nominal (100.0%), **45,021,375 updates offered** over a 20-minute steady-state window on an M4 Pro / 14-core / 48 GB host with the 8-node `docker/` rig still running. Headline numbers: tag update latency **P50 0.88 ms / P99 4.57 ms / max 37.41 ms** end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); **0 events dropped** at 100 live subscribers; health report collect+ingest **P99 0.31 ms** with 10/10 sites tracked; debug view snapshot under load **P99 2.19 ms, 0 timeouts**; 500 instances deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope **+8.83 MB/min**. **11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures.** Harness: `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone Exe — the `Category=Performance` trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in `PerformanceTests`) plus a CI-scale smoke `[Fact]` (`PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`). Real: per-site ActorSystem + LocalDb, the real DCL via the documented `DataConnectionFactory.RegisterAdapter` seam, real `InstanceActor`s, real `SiteStreamManager`, real `StreamRelayActor` + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. **Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`; design + deviations + thresholds: `docs/plans/2026-08-15-target-scale-load-test-design.md`.** | as before | Closed. Residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-vs-central degradation, `[xc-5]` large-template flattening and `[xc-8]` UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. | Closed 2026-08-15. |
|
||||
| 26 | Ipsen MES MoveIn tail: leak-test (`-LT`) receivers + routing, PLC-output-flag writes, `Z28062` BTDB data completeness | `docs/plans/2026-06-16-ipsen-mes-movein.md:409` ("Out of scope (future)"); design `2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198` | Customer-site scope, not a platform gap. `-LT` routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any `-LT`/unknown suffix returns `WasSuccessful=false` with an "unsupported side/target" message by decision); `MoveInComplete`/`Successful`/`ErrorText` are **PLC-owned** by locked decision, so ScadaBridge deliberately does not write them; `Z28062` completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — `_LT` is stripped before side-scoping (`2026-06-30-mes-alarm-status-api.md:158`). | Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a **candidate won't-do** (`[PERM]`) at the next Ipsen scope review |
|
||||
| 27 | External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Two stacked gaps: (a) `ExternalSystemArtifact` (Commons) carries `TimeoutSeconds` but NOT `MaxRetries`/`RetryDelay`, and the site `external_systems` table has no such columns — so a centrally-configured retry policy is silently ignored on sites; every cached call buffers with the S&F default (`DefaultMaxRetries` 50 × `DefaultRetryInterval` 30s ≈ 25 min to park). (b) `Create/UpdateExternalSystemCommand` don't expose the fields either — the only way to set them today is a direct DB edit of `ExternalSystemDefinitions`. Transport bundles DO carry them (arch-review 05 "ES retry config"), which masks the gap in export/import round-trips. Fix is additive: extend the artifact + site schema + apply path, and add `--max-retries`/`--retry-delay` to the CLI. | First operator who tunes retry policy on an external system and expects site cached calls to honor it |
|
||||
| 28 | Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"** | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | The DialogService confirmation host's default destructive-action label leaks through — the dialog copy is correct but the red confirm button says "Delete" for a failover. One-line fix: pass an explicit confirm label ("Fail over") at the Health-dashboard call site (and audit other confirm-dialog call sites for the same default). | Next Central UI session |
|
||||
@@ -47,7 +47,7 @@ Rows removed from the Deferred table above once confirmed shipped. Kept here for
|
||||
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
|
||||
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
|
||||
| ~~Failover-timing measurement (the "~25s total failover" envelope)~~ **RESOLVED 2026-08-01** — split out of the combined row and closed. `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` is no longer a skipped placeholder: it runs as a live `[Fact]` (`Category=Performance`) on the real two-node in-process rig (`TwoNodeClusterFixture`, production `BuildHocon`) at production timings — 2s heartbeat / 10s failure-detection threshold / 15s stable-after — hard-killing the younger node and timing the survivor's member REMOVAL with singleton continuity asserted on the oldest. Delivered by **PLAN-R2-01 Task 4** (`archreview/plans/PLAN-R2-01-cluster-host-failover.md:226`). The oldest-crash direction is covered behaviorally by `SbrFailoverTests.AutoDown_HardCrashOfOldestNode_*` and by `docker/failover-drill.sh`. | The 2026-07-08 "PLAN-01 rig landing" trigger had fired unnoticed (NF2); PLAN-R2-01 T4 wired the placeholder to the fixture rig rather than recording a blocker. | Closed. |
|
||||
| Broader perf envelope — **S&F drain rate + per-subscriber stream backpressure** (the still-open half of the former combined row) | Never measured, and no owner plan survives now that PLAN-R2-01 closed the failover half. `PerformanceTests` covers failover timing, staggered startup, health aggregation, audit hot-path latency and a **single-subscriber** 100k-event `Streaming/SiteStreamThroughputTests.cs` — nothing measures store-and-forward drain throughput, nor what a slow/stalled subscriber does to the per-subscriber buffering in `Communication/Actors/StreamRelayActor.cs` / `Grpc/SiteStreamGrpcServer.cs` with many subscribers attached. No defect observed; deferred as measurement-only work. | First field S&F backlog that fails to drain within an operator's patience, a slow gRPC subscriber degrading a site stream for others, or the WP-4 target-scale run (row 25) being scheduled — that run should absorb this |
|
||||
| ~~Broader perf envelope — **S&F drain rate + per-subscriber stream backpressure**~~ **RESOLVED 2026-08-15 — absorbed into the row-25 target-scale run, exactly as this row's trigger anticipated.** Both halves measured at full scale. **(a) S&F drain:** 20,000 messages buffered concurrently from 25 origin instances at **15,368 msg/s**, draining at **3,533 msg/s** to depth 0. Reported as two numbers because the naive single number is misleading — a deferred backlog sits for one full `DefaultRetryInterval` (measured 28.9 s) before *anything* drains, since `EnqueueAsync(attemptImmediateDelivery: false)` stamps `LastAttemptAt` and the due-predicate then holds the row for one interval (correct by design; `Notify.Send`'s `deferToSweep: true` leaves it null and skips the wait). Filed as informational finding **F2**. **(b) Slow-subscriber backpressure:** with 5 subscribers on the SAME instance and one reader stalled at 50 ms/event, the four healthy subscribers delivered **100.00% with zero drops** while the stalled one lost 197,028 of 200,000 events **entirely within its own bounded channel**. Isolation is structural and total — `Buffer(DropHead)` never backpressures the `BroadcastHub`, `Sink.ForEach`'s `Tell` never blocks, and `StreamRelayActor.WriteToChannel`'s `TryWrite` on a `DropOldest` channel never blocks — so a slow WAN link degrades only its own feed, visibly (per-stream eviction counter + `RecordSiteStreamEventDropped`). Finding **F3**. Caveat also recorded: an *unpaced* burst costs every subscriber, because the publish `Source.ActorRef` upstream of the hub is shared — that shared stage, not per-subscriber buffering, bounds burst absorption. | Closed by the row-25 run; no defect found in either half. | Closed 2026-08-15 — `docs/plans/2026-08-15-target-scale-load-test-results.md`. |
|
||||
|
||||
## Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
|
||||
Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md`'s registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.
|
||||
|
||||
@@ -115,9 +115,20 @@ Deliberately not fixed in this program — each has a stated reason, not an over
|
||||
library's wire-compatibility guarantee — not a blocker for this program.
|
||||
6. **Fragile `SandboxTests` timing pin.** Pre-existing, unrelated to this remediation's changes;
|
||||
noted so it isn't mistaken for a regression if it flakes later.
|
||||
7. **Target-scale load test (deferred-work register #25).** This program's exit criterion is the
|
||||
live probes above, not #25 — #25 remains the follow-on validation that the moved ceilings hold
|
||||
under real load; schedule separately.
|
||||
7. ~~**Target-scale load test (deferred-work register #25).**~~ **RESOLVED 2026-08-15.** The
|
||||
follow-on validation ran: 10 sites x 500 instances x 75 tags = **375,000 live tag
|
||||
subscriptions**, 37,518 updates/s achieved vs 37,500 nominal, **45,021,375 updates** over a
|
||||
20-minute steady-state window. The moved ceilings hold with room to spare — end-to-end tag
|
||||
latency **P99 4.57 ms**, zero dropped events at 100 live subscribers, health report
|
||||
collect+ingest **P99 0.31 ms**, debug view **P99 2.19 ms** with no timeouts, CPU **2.9% of the
|
||||
box**. 11 clean passes, 1 pass with a caveat (F1: 20 minutes with zero gen-2 collections cannot
|
||||
fully settle the leak question; the 1-hour run would), 0 failures. Register **row 50** (S&F
|
||||
drain rate + per-subscriber stream backpressure) was absorbed into the same run and closed with
|
||||
it — S&F drains at **3,533 msg/s**, and a stalled subscriber costs healthy peers **nothing**
|
||||
(100.00%, zero drops). Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`;
|
||||
design: `docs/plans/2026-08-15-target-scale-load-test-design.md`; harness:
|
||||
`tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`. Residual: WP-4's `[xc-3]`/`[xc-4]`/`[xc-5]`/`[xc-8]`
|
||||
are central-cluster/browser concerns outside this harness and remain open scope.
|
||||
8. **Playwright 14 pre-existing env failures.** Present on `main` too, rig-state related, not
|
||||
introduced by this branch.
|
||||
9. **`site_events` retention purge still oplog-visible.** WP3.2's sliced retention DELETE is a row
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
# Target-Scale Load Test — Harness Design (Phase-8 WP-4, deferred-work register #25 + row 50)
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Closes:** deferred-work register **#25** (target-scale load test) and **row 50**
|
||||
(S&F drain rate + per-subscriber stream backpressure), and residual **7** of
|
||||
`docs/plans/2026-08-15-arch-review-remediation-execution-log.md`.
|
||||
**Spec:** `docs/plans/phase-8-production-readiness.md` WP-4 (`:152-170`) + test protocol (`:314-320`).
|
||||
**Results:** `docs/plans/2026-08-15-target-scale-load-test-results.md`.
|
||||
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (+ CI smoke in
|
||||
`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this exists
|
||||
|
||||
Register row 25 records that WP-4 was *claimed complete* on the strength of a
|
||||
**107-byte checklist stub** ("Status: Complete / Tests: All passing / Build: 0 errors,
|
||||
0 warnings") with no per-work-package results and no linked run. The nearest real
|
||||
coverage was arithmetic:
|
||||
|
||||
- `PerformanceTests/StaggeredStartupTests.cs` — `TagCapacity_75TagsPer500Machines_37500Total`
|
||||
and the 10-site distribution test compute products of integers. No actor is created.
|
||||
- `PerformanceTests/HealthAggregationTests.cs` — feeds 10 hand-built `SiteHealthReport`
|
||||
records to a real aggregator. Real, but not under load.
|
||||
- `PerformanceTests/Streaming/SiteStreamThroughputTests.cs` — a genuine 100k-event
|
||||
throughput test, but **one subscriber** and no instances behind it.
|
||||
|
||||
So the job here is not to re-assert the claim. It is to produce the evidence the claim
|
||||
never had, and to report what that evidence actually says — including where it is
|
||||
uncomfortable.
|
||||
|
||||
## 2. Feasibility constraints that shaped the design
|
||||
|
||||
**The host is one macOS box** (Apple M4 Pro, 14 cores, 48 GB) running OrbStack with the
|
||||
8-node `docker/` rig already up. The `docker/` topology has 3 sites and cannot host 10
|
||||
real site pairs at 500 instances each; a full-docker WP-4 topology is out of scope.
|
||||
|
||||
**There is no simulated protocol adapter in the DCL.** Only `OpcUa` and `MxGateway`
|
||||
ship. `DataConnectionFactory.RegisterAdapter` is the documented extension point, and
|
||||
`tests/…DataConnectionLayer.Tests/Actors/FakeBatchDataConnection.cs` establishes the
|
||||
fake-adapter shape.
|
||||
|
||||
**375,000 real OPC UA monitored items is not the system under test.** That would
|
||||
measure the OPC UA SDK. The system under test is everything *above* the adapter
|
||||
callback: the actor hierarchy, the site stream, store-and-forward, health, audit, and
|
||||
the streaming relay.
|
||||
|
||||
The established repo pattern for scale/failover validation is in-process performance
|
||||
tests (`PerformanceTests/Failover/FailoverTimingTests.cs` runs a real two-node cluster
|
||||
in-process via `TwoNodeClusterFixture` with production `BuildHocon` and production
|
||||
timings). This harness extends that pattern rather than inventing a new one.
|
||||
|
||||
## 3. Harness shape, and why
|
||||
|
||||
**A standalone console executable** (`tests/ZB.MOM.WW.ScadaBridge.LoadHarness`, an
|
||||
`Exe`), **plus** a scaled-down `Category=Performance` `[Fact]` in `PerformanceTests`
|
||||
that references it.
|
||||
|
||||
The reason it is not purely an xunit suite is specific and worth recording: in this
|
||||
repo the `[Trait("Category", "Performance")]` attribute **enables a filter, it does not
|
||||
exclude by default**. `dotnet test ZB.MOM.WW.ScadaBridge.slnx` runs the Performance
|
||||
project's tests — `FailoverTimingTests` included, at 45-90s. A 20-minute full-scale
|
||||
test dropped in there would be a 20-minute tax on every solution test run. Hence:
|
||||
|
||||
| | Full protocol | CI smoke |
|
||||
|---|---|---|
|
||||
| Where | `LoadHarness` executable | `TargetScaleHarnessSmokeTests` |
|
||||
| Scale | 10 x 500 x 75 = 375,000 | 2 x 10 x 5 = 100 |
|
||||
| Duration | ~28 min wall | ~80 s |
|
||||
| Purpose | The WP-4 numbers | Harness does not bit-rot |
|
||||
|
||||
The smoke test asserts the *shape* of a healthy result (traffic flows end to end, all
|
||||
sites tracked centrally, S&F drains to empty, a stalled subscriber costs healthy ones
|
||||
nothing) so #25's evidence can be regenerated on demand rather than being a one-off.
|
||||
|
||||
### 3.1 Process topology
|
||||
|
||||
One process. Ten `SiteRuntimeFixture` instances, each owning:
|
||||
|
||||
- its own **`ActorSystem`** (non-clustered),
|
||||
- its own **LocalDb SQLite file** (real `AddZbLocalDb` + real `SiteStorageService`),
|
||||
- a real **`DataConnectionManagerActor`** with 5 `DataConnectionActor` children,
|
||||
- 500 real **`InstanceActor`s**, each configured with 75 data-sourced attributes,
|
||||
- a real **`SiteStreamManager`** at the production `StreamBufferSize` (1000),
|
||||
- a real **`StoreAndForwardStorage` + `StoreAndForwardService`**,
|
||||
- a real **`SiteHealthCollector`**.
|
||||
|
||||
Plus one shared real **`CentralHealthAggregator`** standing in for central.
|
||||
|
||||
### 3.2 What is real and what is faked
|
||||
|
||||
| Layer | Real / faked | Why the fake does not invalidate the measurement |
|
||||
|---|---|---|
|
||||
| OPC UA server + socket | **Faked** (`SimulatedDataConnection`) | Everything replaced is on the far side of the process boundary: socket I/O, the SDK's session/subscription machinery, the device. The system under test begins at the `SubscriptionCallback` invocation, which is exactly where the real adapter hands off. |
|
||||
| `DataConnectionFactory` / `DataConnectionManagerActor` / `DataConnectionActor` | **Real** | The per-connection actor's `_instancesByTag` fan-out and its single-mailbox serialization point are a genuine scale surface — one of the things WP-4 must characterize. The sim adapter is registered through the documented `RegisterAdapter` seam. |
|
||||
| `InstanceActor` | **Real** | The whole point. Real `TagValueUpdate` ingest, real type coercion, real `PublishAndNotifyChildren`. |
|
||||
| `SiteStreamManager` | **Real**, production `StreamBufferSize` | Per-subscriber `Buffer(DropHead)` behaviour is under test. |
|
||||
| `StreamRelayActor` + bounded `DropOldest` channel | **Real**, production capacity (1000) | This is `SiteStreamGrpcServer`'s per-subscription machinery, reused verbatim. |
|
||||
| gRPC socket writer | **Faked** (a reader task) | Deliberate: it is *precisely* the hop whose slowness register row 50 asks about, and a controllable reader is the only way to hold it still. |
|
||||
| `StoreAndForwardService` / `Storage` / SQLite | **Real** | Drain throughput is the measurement. |
|
||||
| S&F delivery target (central) | **Faked** (counting stub returning `true`) | What is measured is the site-local buffer's capacity, not a remote endpoint's. |
|
||||
| `SiteHealthCollector` / `CentralHealthAggregator` | **Real** | `CollectReport` at 37,500 subscriptions is the interesting term. |
|
||||
| `IHealthReportTransport` (gRPC hop) | **Faked** (direct call) | The transport is a documented interface seam; the cost being measured is collect + ingest. |
|
||||
| Akka cluster membership / failover | **Not exercised** | Already measured on a real two-node rig by `FailoverTimingTests` and `docker/failover-drill.sh`. WP-4 asks about the hierarchy *under* the singleton. |
|
||||
|
||||
### 3.3 The latency measurement is genuinely end to end
|
||||
|
||||
The driver stamps `DateTimeOffset.UtcNow` on the `TagValue` it hands the adapter
|
||||
callback. That instant then travels **verbatim**, with no re-stamping:
|
||||
|
||||
```
|
||||
driver → SubscriptionCallback → DataConnectionActor (self.Tell(TagValueReceived))
|
||||
→ TagValueUpdate.Timestamp (DataConnectionActor fan-out)
|
||||
→ AttributeValueChanged.Timestamp (InstanceActor.HandleTagValueUpdate copies it)
|
||||
→ SiteStreamManager hub → per-subscriber Buffer → StreamRelayActor
|
||||
→ proto SiteStreamEvent.AttributeChanged.Timestamp
|
||||
→ subscriber reader: now - Timestamp
|
||||
```
|
||||
|
||||
Everything on that path is production code. This is the harness's single most
|
||||
important property: the reported percentiles are not a synthetic stopwatch around a
|
||||
method call, they are the real DCL-boundary-to-subscriber path.
|
||||
|
||||
## 4. Load model
|
||||
|
||||
WP-4 fixes the scale but **states no tag update rate**, so one has to be chosen and
|
||||
justified.
|
||||
|
||||
- **Nominal rate: one update per tag per 10 seconds.** At 375,000 subscriptions that
|
||||
is **37,500 tag updates/second fleet-wide**, 3,750/s per site, 750/s per data
|
||||
connection actor.
|
||||
- Rationale: a plant SCADA tag that genuinely changes every 10 s is a *busy* tag;
|
||||
assuming every one of 375,000 tags does so simultaneously is a deliberately
|
||||
pessimistic steady state, not a typical one. It also sits comfortably above the
|
||||
10k events/s floor the existing `SiteStreamThroughputTests` pins, so the two
|
||||
measurements bracket each other.
|
||||
- **5 data connections per site.** A site with 37,500 tags behind a single OPC UA
|
||||
server is not the realistic shape, and it would make one actor mailbox the entire
|
||||
story. Five connections (7,500 tags each) is realistic; the per-connection rate is
|
||||
reported so the single-connection case is derivable.
|
||||
- **10 live stream subscribers per site** (100 fleet-wide). Every subscriber's stream
|
||||
graph sees the full site event flow and filters it by instance name, so this is the
|
||||
fan-out multiplier on the hub — the pessimistic direction.
|
||||
|
||||
The driver reports `EmitLagSeconds` (cumulative slice overrun) and
|
||||
`SkippedNoCallback`. If the harness itself cannot offer the nominal rate, those
|
||||
numbers say so, and the achieved-vs-nominal ratio is published rather than the
|
||||
nominal being quietly reported as if achieved.
|
||||
|
||||
## 5. Metric definitions
|
||||
|
||||
| Metric | Definition |
|
||||
|---|---|
|
||||
| **Tag update latency** | Subscriber receive instant minus the driver's emit stamp, in ms. Logarithmic histogram, 16 buckets/octave (bucket width ≤ 4.4%, so a reported percentile is within ~4.4% of truth). Mean/max are exact, not bucketed. **Steady-state window only** — the histogram is repointed at window start so ramp outliers cannot contaminate it. |
|
||||
| **Instance ramp / deployment at scale** | Wall time to create every `InstanceActor`, at the production `StartupBatchSize` (20) / `StartupBatchDelayMs` (100) staggering. Both the fleet total (sites in parallel) and the **slowest single site** are reported; the latter is the "deploy 500 instances to a site" figure. |
|
||||
| **Memory growth** | Working set and managed heap, sampled every 10 s. Reported as start→end delta, peak, **and** least-squares slope in MB/min over the steady-state window — because a run that sawtooths around a stable mean and one that climbs monotonically can share the same endpoint delta. |
|
||||
| **CPU** | `Process.TotalProcessorTime` delta / wall delta, expressed as a percentage of **one core** (so 1400% = the whole 14-core box saturated) and also as a percentage of the box. |
|
||||
| **Health report delivery timing** | `SiteHealthCollector.CollectReport(siteId)` + `CentralHealthAggregator.ProcessReport(report)`, timed together, once per site per 30 s tick under full load. |
|
||||
| **Debug view latency** | `Ask<DebugViewSnapshot>(DebugSnapshotRequest)` round-trip to a randomly chosen live `InstanceActor` every 5 s under full load — so the measurement includes real queueing behind production traffic. |
|
||||
| **S&F drain rate** | Reported as **two** numbers: time-to-first-delivery (the configured retry latency) and throughput measured from the **first** delivery to an empty buffer (the engine's actual capacity). A drain-progress series is captured so a steady rate can be told from a stall-then-burst. |
|
||||
| **Slow-subscriber isolation** | Several subscribers on the *same* instance (identical offered event sequence); one reader stalled at 50 ms/event. Delivery ratio per subscriber, plus per-subscriber channel eviction counts. |
|
||||
|
||||
## 6. Pass/fail thresholds
|
||||
|
||||
Derived from the WP-4 acceptance criteria. Where the criterion is qualitative
|
||||
("within acceptable time", "does not degrade"), the threshold is stated here so the
|
||||
verdict is falsifiable rather than a judgement call made after seeing the number.
|
||||
|
||||
| # | Criterion (WP-4) | Threshold |
|
||||
|---|---|---|
|
||||
| 1 | `[2.5-1]` 10 sites simultaneously operational | All 10 sites built, ramped, and tracked by the central aggregator |
|
||||
| 2 | `[2.5-2]` 500 instances/site with active subscriptions | 5,000 `InstanceActor`s alive; `SkippedNoCallback` = 0 in the steady window |
|
||||
| 3 | `[2.5-3]` 75 live tags/instance (375,000 total) | 375,000 tag paths subscribed through the real DCL |
|
||||
| 4 | Tag update latency | **P99 < 250 ms**, P50 < 50 ms. (A human-facing live value that lands within a quarter second is indistinguishable from instant; the site stream is explicitly best-effort/lossy under pressure, so this bounds the *delivered* path.) |
|
||||
| 5 | `[xc-1]` message patterns function under load | Achieved offered load ≥ 95% of nominal, and no unexpected event loss at healthy subscribers |
|
||||
| 6 | `[xc-2]` health reports arrive within expected intervals | Collect+ingest **P99 < 1,000 ms** (must be a small fraction of the 30 s report interval, else reports would queue) and all 10 sites tracked |
|
||||
| 7 | `[xc-6]` debug view streams without impacting site performance | Snapshot round-trip **P99 < 2,000 ms**, zero ask timeouts, and no measurable tag-latency penalty |
|
||||
| 8 | `[xc-7]` S&F handles concurrent buffering from multiple instances | 20,000 messages buffered concurrently from 25 origin instances with no error; buffer drains to depth 0 |
|
||||
| 9 | Row 50 — S&F drain rate | Drain throughput **> 500 msg/s** (a 20,000-message backlog clears in well under a minute once due) |
|
||||
| 10 | Row 50 — slow-subscriber backpressure | Healthy subscribers' delivery ratio **= 100%** while a peer is stalled; the stalled subscriber's loss is confined to its own bounded channel |
|
||||
| 11 | Deployment of 500 instances to a site | Slowest single site **< 120 s** |
|
||||
| 12 | Memory within acceptable bounds | Steady-state working-set slope **< 20 MB/min**, and no monotonic managed-heap climb across the window |
|
||||
| 13 | CPU within acceptable bounds | Steady-state mean **< 50% of the box** at nominal load |
|
||||
|
||||
## 7. Deviations from the WP-4 protocol
|
||||
|
||||
Each is a deliberate, recorded trade — not an omission.
|
||||
|
||||
| # | Deviation | Reason |
|
||||
|---|---|---|
|
||||
| D1 | **Sustained window shortened from 1 hour to 20 minutes** at full scale | Practicality on a single shared workstation. Memory growth is reported as a *slope* precisely so a shorter window still answers the leak question: a leak shows as a positive slope in 20 min just as in 60. The 1-hour run is a single flag — `--sustain-minutes 60` — and the exact command is recorded in the results doc. |
|
||||
| D2 | **In-process, non-clustered sites** rather than 10 real two-node Akka clusters | A single box cannot host 20 clustered nodes at this scale. Cluster membership/failover is already measured on a real two-node rig (`FailoverTimingTests`, `docker/failover-drill.sh`); WP-4's question is about the hierarchy under the singleton. |
|
||||
| D3 | **Simulated data source** instead of real OPC UA | No simulated adapter exists in the DCL, and 375k real monitored items would measure the OPC UA SDK. Injected at the documented `RegisterAdapter` seam, at the exact hand-off point the real adapter uses. |
|
||||
| D4 | **gRPC socket replaced by a reader task** on the stream path | Required by the row-50 measurement itself: a stalled subscriber must be *held* stalled. All of `SiteStreamGrpcServer`'s per-subscription machinery above the socket is real. |
|
||||
| D5 | **Instance Actors are `/user` children**, not children of a `DeploymentManagerActor` | Constructing them directly is what lets the ramp be timed in isolation and keeps the harness independent of the deploy round-trip. Visible only as `InstanceActorInitialized` dead letters, which the harness suppresses. No measured path differs. |
|
||||
| D6 | **Akka logging at WARNING** | At 37,500 updates/s, INFO output would itself become a measured load. |
|
||||
| D7 | `[xc-3]` site event logging volume, `[xc-4]` audit-log degradation, `[xc-5]` template flattening, `[xc-8]` UI responsiveness **not covered** | Out of scope for this harness: the first three need central MS SQL and the Template Engine (a central-cluster fixture, not a site one), and `[xc-8]` needs a browser. Recorded as residual scope in the results doc rather than silently dropped. |
|
||||
| D8 | **The `docker/` 8-node rig stays running** during the measurement | It is the user's live development cluster. Its baseline cost is measured and recorded alongside the result so the CPU figure is interpretable. |
|
||||
|
||||
## 8. Honest-reporting rule
|
||||
|
||||
If a criterion fails, that failure **is** the deliverable. Findings are recorded in the
|
||||
results doc with the measured evidence; nothing is tuned to make a number go green,
|
||||
and no risky fix is attempted under cover of this work package. Anything found gets
|
||||
filed as a finding for separate triage.
|
||||
@@ -0,0 +1,310 @@
|
||||
# Target-Scale Load Test — Results (Phase-8 WP-4, deferred-work register #25 + row 50)
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Design memo:** `docs/plans/2026-08-15-target-scale-load-test-design.md`
|
||||
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`
|
||||
**Closes:** deferred-work register **#25** and **row 50**; residual **7** of
|
||||
`docs/plans/2026-08-15-arch-review-remediation-execution-log.md`.
|
||||
|
||||
> Register row 25 recorded that WP-4 had been *claimed complete* on the strength of a
|
||||
> 107-byte checklist stub with no per-work-package results and no linked run. This
|
||||
> document is the run that stub never had.
|
||||
|
||||
---
|
||||
|
||||
## 1. Environment
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host | `Josephs-MBP` — Apple M4 Pro, 14 logical cores, 48 GB RAM |
|
||||
| OS | macOS 26.5.2 (Darwin) |
|
||||
| Runtime | .NET 10.0.5, **server GC enabled**, Release build |
|
||||
| Branch / commit | `target-scale-load-test` @ `20f6b0b9` (harness), `8abebdae` (design memo) |
|
||||
| Concurrent load | The 8-node `docker/` rig (OrbStack) remained running throughout — the user's live development cluster. Measured baseline: **~86% of one core (~6% of the box)** and ~10-16 GB RSS. The harness figures below are therefore *pessimistic*, not idealised. |
|
||||
| Scale executed | **10 sites × 500 instances × 75 tags = 375,000 live tag subscriptions** |
|
||||
| Offered load | 37,500 tag updates/second nominal (one update per tag per 10 s) |
|
||||
| Measurement window | 20 minutes steady state, after a 2-minute settle |
|
||||
| Total wall time | 1,510 s (~25 min) |
|
||||
|
||||
**One clean full-scale run is the dataset reported here.** A second run was started to
|
||||
add raw per-sample resource capture (sharpening finding F1), but a verification build
|
||||
overlapped the start of its measurement window; it was **discarded rather than
|
||||
reported**, per the rule that a measurement contaminated by concurrent load is not
|
||||
evidence. The raw-sample capture it was meant to exercise is now in the harness
|
||||
(`HarnessRunResult.ResourceSamples`) and will be present in any future run, including
|
||||
the 1-hour run in §7 — the run reported below predates that field.
|
||||
|
||||
---
|
||||
|
||||
## 2. Verdict summary
|
||||
|
||||
| # | Criterion | Threshold | Measured | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| 1 | `[2.5-1]` 10 sites simultaneously operational | all 10 built, ramped, tracked centrally | 10/10 sites tracked by `CentralHealthAggregator` | **PASS** |
|
||||
| 2 | `[2.5-2]` 500 instances/site, active subscriptions | 5,000 actors; `SkippedNoCallback` = 0 | 5,000 `InstanceActor`s; skipped = **0** | **PASS** |
|
||||
| 3 | `[2.5-3]` 75 tags/instance (375,000 total) | 375,000 subscribed via the real DCL | 375,000 tag paths across 50 `DataConnectionActor`s | **PASS** |
|
||||
| 4 | Tag update latency | P99 < 250 ms, P50 < 50 ms | **P50 0.88 ms, P95 2.83 ms, P99 4.57 ms, P99.9 16.04 ms, max 37.41 ms** (1,100,675 samples) | **PASS** (55× margin at P99) |
|
||||
| 5 | `[xc-1]` message patterns under load | ≥ 95% of nominal, no unexpected loss | **37,518/s achieved vs 37,500 nominal (100.0%)**; 45,021,375 updates offered; **0 events dropped** at healthy subscribers | **PASS** |
|
||||
| 6 | `[xc-2]` health reports within expected intervals | collect+ingest P99 < 1,000 ms; all sites tracked | **P99 0.31 ms, max 4.60 ms** over 440 reports; 10/10 sites | **PASS** (3,200× margin) |
|
||||
| 7 | `[xc-6]` debug view without impacting site performance | snapshot P99 < 2,000 ms, 0 timeouts | **P99 2.19 ms, max 2.72 ms**, 264 completed, **0 timeouts** | **PASS** |
|
||||
| 8 | `[xc-7]` S&F concurrent buffering from many instances | 20,000 messages, 25 origins, drains to 0 | 20,000 buffered at **15,368 msg/s**; residual depth **0** | **PASS** |
|
||||
| 9 | Row 50 — S&F drain rate | > 500 msg/s | **3,533 msg/s** (20,000 messages in 5.66 s of active drain) | **PASS** (7× margin) |
|
||||
| 10 | Row 50 — slow-subscriber backpressure | healthy = 100%; stalled loss confined to its own channel | **healthy 100.00% (4/4, zero drops)**; stalled 1.18%, all 197,028 losses in its own bounded channel | **PASS** |
|
||||
| 11 | Deploy 500 instances to a site | slowest site < 120 s | **2.6 s** | **PASS** (46× margin) |
|
||||
| 12 | Memory within acceptable bounds | WS slope < 20 MB/min; no monotonic heap climb | WS **+8.83 MB/min**; heap sawtooths (peak 3,625 MB vs 2,313→2,523 MB endpoints) with a **+19.52 MB/min** drift — see finding **F1** | **PASS with a caveat** |
|
||||
| 13 | CPU within acceptable bounds | mean < 50% of the box | **41% of one core = 2.9% of the box** (peak 55% of one core) | **PASS** (17× margin) |
|
||||
|
||||
**11 clean passes, 1 pass with a documented caveat, 0 failures.** Four WP-4
|
||||
sub-criteria were out of this harness's scope — see §6.
|
||||
|
||||
---
|
||||
|
||||
## 3. Measured detail
|
||||
|
||||
### 3.1 Deployment / ramp
|
||||
|
||||
```
|
||||
site fixtures built 0.2s
|
||||
all 5,000 instance actors 2.7s (10 sites in parallel)
|
||||
slowest single site 2.6s (500 instances, production staggering)
|
||||
```
|
||||
|
||||
The ramp used the production `StartupBatchSize` (20) / `StartupBatchDelayMs` (100)
|
||||
pacing, i.e. 25 batches × 100 ms = 2.5 s of *deliberate* delay per site. So the 2.6 s
|
||||
figure is almost entirely the configured stagger; actual construction of 500
|
||||
`InstanceActor`s (each deserializing a 75-attribute `FlattenedConfiguration`, loading
|
||||
static overrides from SQLite and issuing a DCL subscribe) costs ~0.1 s. Against a
|
||||
120 s budget this criterion is not close to binding.
|
||||
|
||||
### 3.2 Tag update latency (DCL boundary → stream subscriber)
|
||||
|
||||
```
|
||||
samples 1,100,675 mean 1.17ms
|
||||
p50 0.88ms p95 2.83ms p99 4.57ms p99.9 16.04ms max 37.41ms
|
||||
```
|
||||
|
||||
This is a true end-to-end path measurement, not a synthetic timer: the emit instant is
|
||||
stamped on the `TagValue` handed to the adapter callback and travels verbatim through
|
||||
`DataConnectionActor` → `TagValueUpdate.Timestamp` → `AttributeValueChanged.Timestamp`
|
||||
→ the site stream → `StreamRelayActor` → the proto `SiteStreamEvent`, where the
|
||||
subscriber subtracts it. Every hop is production code.
|
||||
|
||||
Sub-millisecond median under 37,500 updates/s across 5,000 actors means the actor
|
||||
hierarchy is nowhere near its ceiling at target scale.
|
||||
|
||||
### 3.3 Stream delivery
|
||||
|
||||
```
|
||||
events delivered 900,675
|
||||
events dropped 0
|
||||
```
|
||||
|
||||
Zero evictions at the 100 live subscribers across a 20-minute window at full rate.
|
||||
The site stream's `DropHead` buffering exists for pathological cases; at nominal
|
||||
target-scale load it never engaged.
|
||||
|
||||
### 3.4 Health reporting at scale
|
||||
|
||||
```
|
||||
samples 440 mean 0.03ms p50 0.01ms p95 0.04ms p99 0.31ms max 4.60ms
|
||||
reports ingested 440 sites tracked centrally 10
|
||||
```
|
||||
|
||||
`SiteHealthCollector.CollectReport` for a site carrying 37,500 subscriptions plus
|
||||
`CentralHealthAggregator.ProcessReport` costs well under a millisecond. Against the
|
||||
30-second report interval there is no plausible queueing risk.
|
||||
|
||||
### 3.5 Debug view under load
|
||||
|
||||
```
|
||||
samples 264 mean 0.14ms p50 0.08ms p99 2.19ms max 2.72ms
|
||||
completed 264 timed out 0
|
||||
```
|
||||
|
||||
Each sample is an `Ask<DebugViewSnapshot>` landing in the mailbox of an
|
||||
`InstanceActor` concurrently ingesting tag updates, so the figure includes real
|
||||
queueing behind production traffic. `[xc-6]`'s "without impacting site performance"
|
||||
also holds in the other direction: tag latency percentiles were measured with these
|
||||
snapshots running throughout, and show no degradation.
|
||||
|
||||
### 3.6 Resources
|
||||
|
||||
```
|
||||
window 1200s over 121 samples
|
||||
working set 2665 -> 3261 MB (peak 3261) slope +8.83 MB/min
|
||||
managed heap 2313 -> 2523 MB (peak 3625) slope +19.52 MB/min
|
||||
cpu mean/peak 41% / 55% of one core (2.9% of the box)
|
||||
threads 58 gen2 GCs in window: 0
|
||||
```
|
||||
|
||||
CPU is the headline: **375,000 subscriptions at 37,500 updates/s consumed under half
|
||||
of one core of fourteen.** Thread count stayed flat at 58 for ten actor systems.
|
||||
|
||||
---
|
||||
|
||||
## 4. Findings
|
||||
|
||||
### F1 — Memory: 20 minutes with zero gen-2 collections cannot fully settle the leak question (Low)
|
||||
|
||||
The working-set slope (+8.83 MB/min) is inside the threshold, and the managed heap
|
||||
clearly *sawtooths* — its peak (3,625 MB) sits well above both endpoints (2,313 MB →
|
||||
2,523 MB), so the collector is demonstrably reclaiming. But **`GC.CollectionCount(2)`
|
||||
recorded zero gen-2 collections across the entire 25-minute run**, at 45 million
|
||||
events. Gen-2 was therefore never compacted, and a positive least-squares drift on an
|
||||
uncompacted heap cannot be distinguished from a genuine slow leak by this run alone.
|
||||
|
||||
This is a limitation of the shortened window (deviation D1), not an observed defect:
|
||||
no metric degraded, no drop appeared, and latency percentiles were flat from the first
|
||||
minute to the twentieth. Recorded honestly rather than reported as "no leak".
|
||||
|
||||
**To settle it**, run the full 1-hour protocol (§7) and check whether the heap slope
|
||||
flattens once gen-2 collections begin, or issue an explicit
|
||||
`GC.Collect(2, Forced, blocking: true)` at window start and end and compare.
|
||||
Deliberately *not* attempted here — the brief was to measure, not to tune.
|
||||
|
||||
### F2 — Store-and-forward waits one full `DefaultRetryInterval` before draining a deferred backlog (informational, by design)
|
||||
|
||||
The drain measurement initially read as **33 msg/s**, which would have been alarming.
|
||||
The progress series showed why: **nothing drained for 29-30 seconds, then the entire
|
||||
backlog cleared in a fraction of a second.**
|
||||
|
||||
Cause, confirmed in source: `StoreAndForwardService.EnqueueAsync` with
|
||||
`attemptImmediateDelivery: false` stamps `message.LastAttemptAt = UtcNow`
|
||||
(`StoreAndForwardService.cs:754-757`). `GetMessagesForRetryAsync`'s due-predicate is
|
||||
`last_attempt_at_ms IS NULL OR retry_interval_ms = 0 OR (now - last_attempt) >= retry_interval`,
|
||||
so such a row is not due for one `DefaultRetryInterval` (30 s). Explicit
|
||||
`TriggerSweep()` calls find nothing due and correctly do nothing.
|
||||
|
||||
This is **correct, intended behaviour** — the caller declared delivery was not
|
||||
attempted, so the row waits one retry interval — but it is easy to misread as slow
|
||||
drainage. The harness therefore reports the two numbers separately:
|
||||
|
||||
```
|
||||
retry wait before drain 28.9s (DefaultRetryInterval)
|
||||
drain throughput 3,533 msg/s (active drain 5.66s)
|
||||
```
|
||||
|
||||
Worth knowing operationally: a site that buffers a backlog this way will appear
|
||||
completely stalled for the first 30 seconds. Note the `Notify.Send` path
|
||||
(`deferToSweep: true`) deliberately leaves `LastAttemptAt` null so its rows are due
|
||||
immediately and skip this wait entirely.
|
||||
|
||||
### F3 — Slow-subscriber isolation is complete, and the mechanism is worth recording (positive result)
|
||||
|
||||
Register row 50 asked what a slow/stalled gRPC subscriber does to per-subscriber
|
||||
buffering with several subscribers attached. With five subscribers on the *same*
|
||||
instance (identical offered sequence) and one reader stalled at 50 ms/event:
|
||||
|
||||
| Subscriber | State | Received | Dropped | Delivery |
|
||||
|---|---|---|---|---|
|
||||
| slowprobe-00 | **STALLED** | 2,369 | 197,028 | 1.18% |
|
||||
| slowprobe-01 | healthy | 200,000 | 0 | **100.00%** |
|
||||
| slowprobe-02 | healthy | 200,000 | 0 | **100.00%** |
|
||||
| slowprobe-03 | healthy | 200,000 | 0 | **100.00%** |
|
||||
| slowprobe-04 | healthy | 200,000 | 0 | **100.00%** |
|
||||
|
||||
**Isolation is total** — not merely "good". The stalled subscriber lost 98.8% of its
|
||||
own events and cost its peers exactly nothing. The reason is structural, and each link
|
||||
matters:
|
||||
|
||||
1. The per-subscriber graph is `Where → Buffer(StreamBufferSize, DropHead) → Sink.ForEach(Tell)`.
|
||||
A `DropHead` buffer *always* accepts, so it never backpressures the `BroadcastHub`
|
||||
upstream — one subscriber cannot stall the hub for the others.
|
||||
2. `Sink.ForEach` does an actor `Tell`, which never blocks.
|
||||
3. `StreamRelayActor.WriteToChannel` uses `TryWrite` on a bounded `DropOldest`
|
||||
channel, which never blocks either — so the relay actor's mailbox drains regardless
|
||||
of reader speed, and loss is confined to that subscriber's own channel where it is
|
||||
counted.
|
||||
|
||||
The consequence for operators: a slow WAN link or wedged central client degrades
|
||||
**only its own feed**, and the loss is visible in the per-stream eviction counter and
|
||||
the `RecordSiteStreamEventDropped` telemetry rather than being silent.
|
||||
|
||||
One caveat established while building the probe: publishing an unpaced burst makes
|
||||
*every* subscriber lose events, because the publish source is a single
|
||||
`Source.ActorRef(StreamBufferSize, DropHead)` **shared** by all attribute subscribers,
|
||||
upstream of the hub. That shared stage — not per-subscriber buffering — is the real
|
||||
limit on burst absorption. The probe paces at 2,000 events/s to isolate the variable
|
||||
under test; the shared stage's own capacity is separately covered by
|
||||
`SiteStreamThroughputTests`.
|
||||
|
||||
### F4 — Pre-existing test-isolation flake in `QueueDepthGaugeTests` (Low, not introduced here)
|
||||
|
||||
Noticed during this work package's verification pass, recorded so it is not later
|
||||
mistaken for a regression from the load-harness branch.
|
||||
|
||||
`StoreAndForward.Tests.QueueDepthGaugeTests.Gauge_TracksBufferedDepth_AcrossEnqueueDrainAndPark`
|
||||
**fails in a full-suite run** (`Expected: 0, Actual: 2`, `QueueDepthGaugeTests.cs:116`)
|
||||
but **passes when run in isolation** — the signature of a shared static gauge carrying
|
||||
state across tests, i.e. an ordering dependency rather than a product defect.
|
||||
|
||||
It cannot originate from this branch: the branch changes **zero `src/` files** relative
|
||||
to its base (`986e6e7a`), and both the StoreAndForward source and its test project are
|
||||
byte-identical to that base. Left unfixed deliberately — out of scope for a measurement
|
||||
work package, and the brief was explicitly not to attempt unrelated fixes. Worth a
|
||||
separate triage alongside the pre-existing `SandboxTests` timing pin already recorded as
|
||||
residual 6 of the arch-review remediation execution log.
|
||||
|
||||
---
|
||||
|
||||
## 5. What this does *not* prove
|
||||
|
||||
Stated plainly so the evidence is not over-read:
|
||||
|
||||
- **Not a clustered run.** Sites are single, non-clustered `ActorSystem`s (deviation
|
||||
D2). Failover, singleton handover and split-brain behaviour are covered elsewhere
|
||||
(`FailoverTimingTests`, `SbrFailoverTests`, `docker/failover-drill.sh`).
|
||||
- **Not a real-network run.** The gRPC, health-transport and S&F-to-central hops are
|
||||
in-process. Serialization cost, TLS, WAN latency and socket backpressure are out of
|
||||
frame; the *stream* path's backpressure was measured deliberately by substituting a
|
||||
controllable reader for the socket (D4).
|
||||
- **Not a real OPC UA run.** 375,000 monitored items were simulated at the adapter
|
||||
callback (D3). The DCL above that callback is real and fully exercised.
|
||||
- **Not 1 hour.** 20 minutes (D1). See finding F1 for the one question this leaves
|
||||
genuinely open.
|
||||
|
||||
## 6. WP-4 criteria not covered by this harness
|
||||
|
||||
Recorded rather than silently dropped (deviation D7):
|
||||
|
||||
| Criterion | Why not covered |
|
||||
|---|---|
|
||||
| `[xc-3]` site event logging within 30-day / 1 GB limits | Needs a long-horizon retention run, not a load run |
|
||||
| `[xc-4]` audit logging does not degrade central | Needs a central MS SQL fixture; this harness builds site runtimes only |
|
||||
| `[xc-5]` template flattening/validation for large templates | Template Engine is a central-cluster concern; unrelated to the site-runtime load path |
|
||||
| `[xc-8]` UI workflows remain responsive | Needs a browser + a live central cluster (Playwright territory) |
|
||||
|
||||
These remain open scope for WP-4 and should be tracked separately rather than being
|
||||
considered closed by this run.
|
||||
|
||||
## 7. Reproducing
|
||||
|
||||
Full protocol as executed (20-minute window):
|
||||
|
||||
```bash
|
||||
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
|
||||
--results loadharness-results.json
|
||||
```
|
||||
|
||||
The full **1-hour** version required by the WP-4 test protocol — identical in every
|
||||
other respect, and the run that would settle finding F1:
|
||||
|
||||
```bash
|
||||
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
|
||||
--sustain-minutes 60 --results loadharness-results-1h.json
|
||||
```
|
||||
|
||||
CI-scale smoke (~80 s), which asserts the harness still produces coherent
|
||||
measurements end to end:
|
||||
|
||||
```bash
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests \
|
||||
--filter "FullyQualifiedName~TargetScaleHarnessSmoke"
|
||||
```
|
||||
|
||||
`--help`-style knobs: `--sites`, `--instances-per-site`, `--tags-per-instance`,
|
||||
`--tag-update-period-seconds`, `--settle-minutes`, `--sustain-minutes`,
|
||||
`--sample-seconds`, `--health-interval-seconds`, `--debug-probe-interval-seconds`,
|
||||
`--subscribe-settle-seconds`, `--stream-probes-per-site`, `--sf-drain-messages`,
|
||||
`--slow-subscriber-events`, `--data-dir`, `--results`. An unknown key is rejected
|
||||
rather than ignored, so a typo cannot silently change the measured scale.
|
||||
@@ -3,21 +3,46 @@
|
||||
**Update 2026-08-07 (truth sweep):** This checklist previously read
|
||||
"Status: Complete / Tests: All passing / Build: 0 errors, 0 warnings" — a
|
||||
107-byte stub with no per-work-package results and no linked run. That claim
|
||||
was **unevidenced** and has been retired.
|
||||
was **unevidenced** and was retired.
|
||||
|
||||
Honest state, per the deferred-work register
|
||||
(`2026-07-08-deferred-work-register.md`, row 25):
|
||||
**Update 2026-08-15 (WP-4 run, evidenced):** The target-scale load test has now
|
||||
actually been run, at full scale, with results published.
|
||||
|
||||
- The **Phase-8 WP-4 target-scale load test** (10 sites × 500 instances ×
|
||||
75 tags = 37,500 subscriptions/site, ~375,000 total) has **never been run**.
|
||||
- Nearest real coverage is arithmetic/aggregation only —
|
||||
`PerformanceTests/StaggeredStartupTests.cs`
|
||||
(`TagCapacity_75TagsPer500Machines_37500Total`) and
|
||||
`HealthAggregationTests` — plus a single-subscriber 100k-event
|
||||
`Streaming/SiteStreamThroughputTests.cs`. No sustained multi-site run
|
||||
exists anywhere in `tests/` or `docker/`.
|
||||
- Revisit trigger: before any production go-live at target scale, or the
|
||||
first site approaching ~500 instances / ~37.5k subscriptions.
|
||||
## WP-4 — Load/Performance Testing at Target Scale
|
||||
|
||||
**Status: measured 2026-08-15.** Not a claim — a run.
|
||||
|
||||
- Scale executed: **10 sites × 500 instances × 75 tags = 375,000 live tag
|
||||
subscriptions**, 37,518 tag updates/second achieved against 37,500 nominal
|
||||
(100.0%), **45,021,375 updates** offered over a 20-minute steady-state window.
|
||||
- Headline results: end-to-end tag update latency **P50 0.88 ms / P99 4.57 ms /
|
||||
max 37.41 ms**; **zero events dropped** at 100 live stream subscribers; health
|
||||
report collect+ingest **P99 0.31 ms** with 10/10 sites tracked centrally;
|
||||
debug view snapshot under load **P99 2.19 ms with 0 timeouts**; 500 instances
|
||||
deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope
|
||||
**+8.83 MB/min**.
|
||||
- **11 clean passes, 1 pass with a documented caveat, 0 failures.**
|
||||
- Deferred-work register **#25** and **row 50** are closed by this run.
|
||||
|
||||
**Results:** `2026-08-15-target-scale-load-test-results.md` — per-criterion
|
||||
numbers vs thresholds, three findings (F1 memory-window limitation, F2 the S&F
|
||||
retry-interval wait, F3 slow-subscriber isolation), and an explicit "what this
|
||||
does not prove" section.
|
||||
|
||||
**Design, thresholds and deviations:**
|
||||
`2026-08-15-target-scale-load-test-design.md` — including the documented
|
||||
deviation from the protocol's 1-hour sustained window to 20 minutes, and the
|
||||
command for the full 1-hour version.
|
||||
|
||||
**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone executable),
|
||||
with a CI-scale smoke `[Fact]` at
|
||||
`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`
|
||||
so the evidence can be regenerated rather than bit-rotting.
|
||||
|
||||
**Still open within WP-4** (not covered by this harness — central-cluster and
|
||||
browser concerns): `[xc-3]` site event logging volume within 30-day/1 GB limits,
|
||||
`[xc-4]` audit logging not degrading central, `[xc-5]` template
|
||||
flattening/validation for large templates, `[xc-8]` UI workflow responsiveness.
|
||||
|
||||
See `phase-8-production-readiness.md:152-170` (WP-4) and `:314-320`
|
||||
(test protocol) for the original scope.
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness;
|
||||
|
||||
/// <summary>
|
||||
/// Scale and duration knobs for a load-harness run. Defaults are the Phase-8 WP-4
|
||||
/// target scale (10 sites x 500 instances x 75 tags = 375,000 subscriptions).
|
||||
/// Every value is overridable from the command line so the same binary serves both
|
||||
/// the full-scale protocol run and the scaled-down CI smoke.
|
||||
/// </summary>
|
||||
public sealed record HarnessConfig
|
||||
{
|
||||
/// <summary>Number of simulated sites (WP-4 acceptance criterion <c>[2.5-1]</c>).</summary>
|
||||
public int Sites { get; init; } = 10;
|
||||
|
||||
/// <summary>Instance Actors per site (WP-4 <c>[2.5-2]</c>).</summary>
|
||||
public int InstancesPerSite { get; init; } = 500;
|
||||
|
||||
/// <summary>Data-sourced attributes ("live tags") per instance (WP-4 <c>[2.5-3]</c>).</summary>
|
||||
public int TagsPerInstance { get; init; } = 75;
|
||||
|
||||
/// <summary>
|
||||
/// Nominal per-tag update period. The driver emits each tag once per period, so
|
||||
/// the site-wide event rate is <c>InstancesPerSite * TagsPerInstance / period</c>.
|
||||
/// The 10 s default puts the target-scale fleet at 37,500 tag updates/second.
|
||||
/// </summary>
|
||||
public TimeSpan TagUpdatePeriod { get; init; } = TimeSpan.FromSeconds(10);
|
||||
|
||||
/// <summary>
|
||||
/// Length of the steady-state measurement window, measured AFTER the ramp
|
||||
/// completes and after the post-ramp settle. Memory-growth and CPU figures are
|
||||
/// computed over exactly this window.
|
||||
/// </summary>
|
||||
public TimeSpan SustainDuration { get; init; } = TimeSpan.FromMinutes(20);
|
||||
|
||||
/// <summary>
|
||||
/// Quiet period between the last instance starting and the start of the
|
||||
/// measurement window — lets startup allocations settle so the memory-growth
|
||||
/// slope reflects steady state, not the ramp.
|
||||
/// </summary>
|
||||
public TimeSpan SettleDuration { get; init; } = TimeSpan.FromMinutes(2);
|
||||
|
||||
/// <summary>Resource-sampling cadence (working set, GC heap, CPU, thread count).</summary>
|
||||
public TimeSpan SampleInterval { get; init; } = TimeSpan.FromSeconds(10);
|
||||
|
||||
/// <summary>
|
||||
/// Health report cadence. Defaults to the production
|
||||
/// <c>HealthMonitoringOptions.ReportInterval</c> (30 s) so the timing measured is
|
||||
/// the one that ships; only the CI smoke shortens it, because a 20-second smoke
|
||||
/// window would otherwise never see a single tick.
|
||||
/// </summary>
|
||||
public TimeSpan HealthReportInterval { get; init; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>How often to take a debug view snapshot of a random live instance.</summary>
|
||||
public TimeSpan DebugProbeInterval { get; init; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>
|
||||
/// Pause between the last Instance Actor starting and the tag driver starting, to
|
||||
/// let every instance complete its DCL subscribe round-trip. The adapter callback
|
||||
/// is captured at subscribe time, so an emit before that lands is silently
|
||||
/// discarded — this window is what keeps <c>SkippedNoCallback</c> at zero.
|
||||
/// </summary>
|
||||
public TimeSpan SubscribeSettleDuration { get; init; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// Instances per site that carry a live stream subscriber (a real
|
||||
/// <c>StreamRelayActor</c> + bounded DropOldest channel, i.e. the production
|
||||
/// Debug View / central shape). Every subscriber's stream graph sees the FULL
|
||||
/// site event flow and filters it, so this is the fan-out multiplier.
|
||||
/// </summary>
|
||||
public int StreamProbesPerSite { get; init; } = 10;
|
||||
|
||||
/// <summary>Store-and-forward messages enqueued for the drain-rate measurement (register row 50).</summary>
|
||||
public int StoreAndForwardDrainMessages { get; init; } = 20_000;
|
||||
|
||||
/// <summary>Events published at the slow-subscriber isolation probe (register row 50).</summary>
|
||||
public int SlowSubscriberEvents { get; init; } = 200_000;
|
||||
|
||||
/// <summary>Directory for the site SQLite files. A temp directory is used when null.</summary>
|
||||
public string? DataDirectory { get; init; }
|
||||
|
||||
/// <summary>Path the JSON metrics document is written to.</summary>
|
||||
public string ResultsPath { get; init; } = "loadharness-results.json";
|
||||
|
||||
/// <summary>Total live tag subscriptions across the fleet.</summary>
|
||||
public int TotalSubscriptions => Sites * InstancesPerSite * TagsPerInstance;
|
||||
|
||||
/// <summary>Nominal fleet-wide tag updates per second implied by the scale and update period.</summary>
|
||||
public double NominalUpdatesPerSecond => TotalSubscriptions / TagUpdatePeriod.TotalSeconds;
|
||||
|
||||
/// <summary>
|
||||
/// Parses <c>--key value</c> / <c>--key=value</c> arguments over the defaults.
|
||||
/// Unknown keys throw so a typo in a 20-minute run's command line fails fast
|
||||
/// rather than silently measuring the wrong scale.
|
||||
/// </summary>
|
||||
/// <param name="args">Raw command-line arguments.</param>
|
||||
/// <returns>The parsed configuration.</returns>
|
||||
public static HarnessConfig Parse(string[] args)
|
||||
{
|
||||
var config = new HarnessConfig();
|
||||
for (var i = 0; i < args.Length; i++)
|
||||
{
|
||||
var arg = args[i];
|
||||
if (!arg.StartsWith("--", StringComparison.Ordinal))
|
||||
throw new ArgumentException($"Unexpected argument '{arg}' (expected --key value).");
|
||||
|
||||
string key;
|
||||
string value;
|
||||
var eq = arg.IndexOf('=', StringComparison.Ordinal);
|
||||
if (eq >= 0)
|
||||
{
|
||||
key = arg[2..eq];
|
||||
value = arg[(eq + 1)..];
|
||||
}
|
||||
else
|
||||
{
|
||||
key = arg[2..];
|
||||
if (i + 1 >= args.Length)
|
||||
throw new ArgumentException($"Option '--{key}' requires a value.");
|
||||
value = args[++i];
|
||||
}
|
||||
|
||||
config = key switch
|
||||
{
|
||||
"sites" => config with { Sites = int.Parse(value) },
|
||||
"instances-per-site" => config with { InstancesPerSite = int.Parse(value) },
|
||||
"tags-per-instance" => config with { TagsPerInstance = int.Parse(value) },
|
||||
"tag-update-period-seconds" => config with { TagUpdatePeriod = TimeSpan.FromSeconds(double.Parse(value)) },
|
||||
"sustain-minutes" => config with { SustainDuration = TimeSpan.FromMinutes(double.Parse(value)) },
|
||||
"settle-minutes" => config with { SettleDuration = TimeSpan.FromMinutes(double.Parse(value)) },
|
||||
"subscribe-settle-seconds" => config with { SubscribeSettleDuration = TimeSpan.FromSeconds(double.Parse(value)) },
|
||||
"sample-seconds" => config with { SampleInterval = TimeSpan.FromSeconds(double.Parse(value)) },
|
||||
"health-interval-seconds" => config with { HealthReportInterval = TimeSpan.FromSeconds(double.Parse(value)) },
|
||||
"debug-probe-interval-seconds" => config with { DebugProbeInterval = TimeSpan.FromSeconds(double.Parse(value)) },
|
||||
"stream-probes-per-site" => config with { StreamProbesPerSite = int.Parse(value) },
|
||||
"sf-drain-messages" => config with { StoreAndForwardDrainMessages = int.Parse(value) },
|
||||
"slow-subscriber-events" => config with { SlowSubscriberEvents = int.Parse(value) },
|
||||
"data-dir" => config with { DataDirectory = value },
|
||||
"results" => config with { ResultsPath = value },
|
||||
_ => throw new ArgumentException($"Unknown option '--{key}'."),
|
||||
};
|
||||
}
|
||||
|
||||
return config;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,283 @@
|
||||
using System.Diagnostics;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness.Scenarios;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness;
|
||||
|
||||
/// <summary>Everything one harness run measured, ready for serialization.</summary>
|
||||
/// <param name="Config">The configuration the run executed under.</param>
|
||||
/// <param name="Environment">Host/runtime description.</param>
|
||||
/// <param name="StartedUtc">Run start.</param>
|
||||
/// <param name="TotalSeconds">Total wall time including ramp and teardown.</param>
|
||||
/// <param name="SiteRampSeconds">Wall time to build all site fixtures (before instances).</param>
|
||||
/// <param name="InstanceRampSeconds">Wall time to create every Instance Actor across all sites.</param>
|
||||
/// <param name="SlowestSiteInstanceRampSeconds">Slowest single site's instance ramp — the "deploy 500 instances to a site" figure.</param>
|
||||
/// <param name="TagUpdateLatency">End-to-end DCL-boundary to stream-subscriber latency.</param>
|
||||
/// <param name="EmittedTagUpdates">Tag updates offered during the whole run.</param>
|
||||
/// <param name="SteadyStateEmittedTagUpdates">Tag updates offered during the measurement window only.</param>
|
||||
/// <param name="AchievedUpdatesPerSecond">Offered load actually achieved in the measurement window.</param>
|
||||
/// <param name="NominalUpdatesPerSecond">Offered load the configuration called for.</param>
|
||||
/// <param name="DriverLagSeconds">Cumulative driver slice overrun (harness-bound load shortfall).</param>
|
||||
/// <param name="DriverSkippedNoCallback">Emissions skipped before subscriptions existed.</param>
|
||||
/// <param name="SteadyStateResources">Resource behaviour over the measurement window.</param>
|
||||
/// <param name="WholeRunResources">Resource behaviour over the whole run.</param>
|
||||
/// <param name="ResourceSamples">
|
||||
/// Every raw resource sample. Retained because the summary alone cannot distinguish a
|
||||
/// slow leak from a heap that simply sawtooths — especially when the run records zero
|
||||
/// gen-2 collections, where a positive least-squares slope may be nothing more than
|
||||
/// gen-2 garbage that was never collected.
|
||||
/// </param>
|
||||
/// <param name="HealthReportLatency">Health report collect+ingest latency.</param>
|
||||
/// <param name="HealthReportsDelivered">Health reports ingested by the central aggregator.</param>
|
||||
/// <param name="SitesTrackedByAggregator">Sites the central aggregator ended up tracking.</param>
|
||||
/// <param name="DebugSnapshotLatency">Debug view snapshot round-trip latency under load.</param>
|
||||
/// <param name="DebugSnapshotsCompleted">Debug snapshots that completed.</param>
|
||||
/// <param name="DebugSnapshotTimeouts">Debug snapshots that timed out.</param>
|
||||
/// <param name="StreamProbeReceived">Events delivered to live stream subscribers.</param>
|
||||
/// <param name="StreamProbeDropped">Events evicted by live subscribers' bounded channels.</param>
|
||||
/// <param name="StoreAndForwardDrain">Store-and-forward drain measurement (register row 50).</param>
|
||||
/// <param name="SlowSubscriber">Slow-subscriber isolation measurement (register row 50).</param>
|
||||
public sealed record HarnessRunResult(
|
||||
HarnessConfig Config,
|
||||
EnvironmentInfo Environment,
|
||||
DateTimeOffset StartedUtc,
|
||||
double TotalSeconds,
|
||||
double SiteRampSeconds,
|
||||
double InstanceRampSeconds,
|
||||
double SlowestSiteInstanceRampSeconds,
|
||||
LatencySnapshot TagUpdateLatency,
|
||||
long EmittedTagUpdates,
|
||||
long SteadyStateEmittedTagUpdates,
|
||||
double AchievedUpdatesPerSecond,
|
||||
double NominalUpdatesPerSecond,
|
||||
double DriverLagSeconds,
|
||||
long DriverSkippedNoCallback,
|
||||
ResourceWindowSummary? SteadyStateResources,
|
||||
ResourceWindowSummary? WholeRunResources,
|
||||
IReadOnlyList<ResourceSample> ResourceSamples,
|
||||
LatencySnapshot HealthReportLatency,
|
||||
long HealthReportsDelivered,
|
||||
int SitesTrackedByAggregator,
|
||||
LatencySnapshot DebugSnapshotLatency,
|
||||
long DebugSnapshotsCompleted,
|
||||
long DebugSnapshotTimeouts,
|
||||
long StreamProbeReceived,
|
||||
long StreamProbeDropped,
|
||||
StoreAndForwardDrainResult? StoreAndForwardDrain,
|
||||
SlowSubscriberResult? SlowSubscriber);
|
||||
|
||||
/// <summary>Host and runtime facts recorded alongside the numbers.</summary>
|
||||
/// <param name="MachineName">Host name.</param>
|
||||
/// <param name="OsDescription">Operating system description.</param>
|
||||
/// <param name="ProcessorCount">Logical processors visible to the process.</param>
|
||||
/// <param name="RuntimeVersion">.NET runtime version.</param>
|
||||
/// <param name="ServerGc">Whether server GC is active.</param>
|
||||
public sealed record EnvironmentInfo(
|
||||
string MachineName,
|
||||
string OsDescription,
|
||||
int ProcessorCount,
|
||||
string RuntimeVersion,
|
||||
bool ServerGc);
|
||||
|
||||
/// <summary>
|
||||
/// Orchestrates a full run: build sites, ramp instances, attach subscribers, drive
|
||||
/// tag updates for the sustained window while sampling resources and probing
|
||||
/// observability, then run the two register-row-50 scenarios on a dedicated site.
|
||||
/// </summary>
|
||||
public static class HarnessRun
|
||||
{
|
||||
/// <summary>Data connections each site spreads its tags across.</summary>
|
||||
public const int ConnectionsPerSite = 5;
|
||||
|
||||
/// <summary>Subscribers attached in the slow-subscriber isolation probe.</summary>
|
||||
public const int SlowSubscriberProbeCount = 5;
|
||||
|
||||
/// <summary>Concurrent enqueue tasks in the store-and-forward drain probe.</summary>
|
||||
public const int StoreAndForwardConcurrency = 25;
|
||||
|
||||
/// <summary>Executes a run end to end.</summary>
|
||||
/// <param name="config">Scale and duration configuration.</param>
|
||||
/// <param name="log">Progress sink (stdout in the console app).</param>
|
||||
/// <param name="cancellationToken">Cancels the run.</param>
|
||||
/// <returns>The measured result.</returns>
|
||||
public static async Task<HarnessRunResult> ExecuteAsync(
|
||||
HarnessConfig config, Action<string> log, CancellationToken cancellationToken)
|
||||
{
|
||||
var startedUtc = DateTimeOffset.UtcNow;
|
||||
var totalWatch = Stopwatch.StartNew();
|
||||
var dataRoot = config.DataDirectory
|
||||
?? Path.Combine(Path.GetTempPath(), $"scadabridge-loadharness-{Guid.NewGuid():N}");
|
||||
Directory.CreateDirectory(dataRoot);
|
||||
|
||||
var sampler = ResourceSampler.Start(config.SampleInterval);
|
||||
var sites = new List<SiteRuntimeFixture>(config.Sites);
|
||||
var latency = new LatencyHistogram();
|
||||
|
||||
TagUpdateDriver? driver = null;
|
||||
ObservabilityProbes? probes = null;
|
||||
StoreAndForwardDrainResult? drainResult = null;
|
||||
SlowSubscriberResult? slowResult = null;
|
||||
|
||||
var aggregator = new CentralHealthAggregator(
|
||||
Options.Create(new HealthMonitoringOptions
|
||||
{
|
||||
ReportInterval = config.HealthReportInterval,
|
||||
OfflineTimeout = config.HealthReportInterval * 2,
|
||||
}),
|
||||
NullLogger<CentralHealthAggregator>.Instance);
|
||||
|
||||
try
|
||||
{
|
||||
log($"Building {config.Sites} sites ({config.InstancesPerSite} instances x " +
|
||||
$"{config.TagsPerInstance} tags each = {config.TotalSubscriptions:N0} subscriptions)...");
|
||||
|
||||
var siteWatch = Stopwatch.StartNew();
|
||||
for (var s = 0; s < config.Sites; s++)
|
||||
sites.Add(await SiteRuntimeFixture.CreateAsync(s, config, dataRoot, ConnectionsPerSite));
|
||||
siteWatch.Stop();
|
||||
log($" sites built in {siteWatch.Elapsed.TotalSeconds:F1}s");
|
||||
|
||||
// Instance ramp — sites in parallel (they are independent actor systems,
|
||||
// exactly as 10 real sites would be), each site internally staggered at the
|
||||
// production StartupBatchSize/StartupBatchDelayMs pacing.
|
||||
var rampWatch = Stopwatch.StartNew();
|
||||
await Task.WhenAll(sites.Select(site => site.StartInstancesAsync(cancellationToken)));
|
||||
rampWatch.Stop();
|
||||
var slowestSiteRamp = sites.Max(s => s.InstanceRampDuration.TotalSeconds);
|
||||
log($" {config.Sites * config.InstancesPerSite:N0} instance actors created in " +
|
||||
$"{rampWatch.Elapsed.TotalSeconds:F1}s (slowest site {slowestSiteRamp:F1}s)");
|
||||
|
||||
foreach (var site in sites)
|
||||
site.AttachStreamProbes(latency);
|
||||
log($" {sites.Sum(s => s.Probes.Count)} live stream subscribers attached");
|
||||
|
||||
// Let Instance Actors complete their DCL subscribe round-trips before the
|
||||
// driver starts; an emit before SubscribeBatchAsync has captured the
|
||||
// callback would be silently discarded.
|
||||
await Task.Delay(config.SubscribeSettleDuration, cancellationToken);
|
||||
|
||||
driver = TagUpdateDriver.Start(sites, config);
|
||||
log($" tag driver started, nominal {config.NominalUpdatesPerSecond:N0} updates/s");
|
||||
|
||||
probes = ObservabilityProbes.Start(
|
||||
sites, aggregator, config.HealthReportInterval, config.DebugProbeInterval);
|
||||
|
||||
log($"Settling for {config.SettleDuration.TotalMinutes:F1} min...");
|
||||
await Task.Delay(config.SettleDuration, cancellationToken);
|
||||
|
||||
// Switch to a fresh latency histogram so ramp-window outliers do not
|
||||
// contaminate the steady-state percentiles. Everything reported as "steady
|
||||
// state" is measured strictly after this point; the subscriptions
|
||||
// themselves are left untouched.
|
||||
var steadyStateStartSeconds = sampler.Snapshot().LastOrDefault()?.ElapsedSeconds ?? 0;
|
||||
var emittedAtWindowStart = driver.EmittedCount;
|
||||
var receivedAtWindowStart = sites.Sum(s => s.Probes.Sum(p => p.ReceivedEvents));
|
||||
var droppedAtWindowStart = sites.Sum(s => s.Probes.Sum(p => p.DroppedEvents));
|
||||
|
||||
var steadyLatency = new LatencyHistogram();
|
||||
foreach (var site in sites)
|
||||
{
|
||||
foreach (var probe in site.Probes)
|
||||
probe.RetargetLatency(steadyLatency);
|
||||
}
|
||||
|
||||
var windowWatch = Stopwatch.StartNew();
|
||||
log($"Sustained measurement window: {config.SustainDuration.TotalMinutes:F1} min...");
|
||||
await Task.Delay(config.SustainDuration, cancellationToken);
|
||||
windowWatch.Stop();
|
||||
|
||||
var steadyStateEndSeconds = sampler.Snapshot().LastOrDefault()?.ElapsedSeconds ?? 0;
|
||||
var emittedInWindow = driver.EmittedCount - emittedAtWindowStart;
|
||||
|
||||
var steadyResources = sampler.Summarize(steadyStateStartSeconds, steadyStateEndSeconds);
|
||||
log($" window complete: {emittedInWindow:N0} updates offered, " +
|
||||
$"{emittedInWindow / windowWatch.Elapsed.TotalSeconds:N0}/s achieved");
|
||||
|
||||
var probeReceived = sites.Sum(s => s.Probes.Sum(p => p.ReceivedEvents)) - receivedAtWindowStart;
|
||||
var probeDropped = sites.Sum(s => s.Probes.Sum(p => p.DroppedEvents)) - droppedAtWindowStart;
|
||||
|
||||
// Register row 50 — measured after the sustained window so the numbers are
|
||||
// not competing with the full tag load for CPU, and reported separately for
|
||||
// the same reason.
|
||||
log("Stopping tag driver for the register row 50 scenarios...");
|
||||
var driverLagSeconds = driver.EmitLagSeconds;
|
||||
var driverSkipped = driver.SkippedNoCallback;
|
||||
await driver.DisposeAsync();
|
||||
driver = null;
|
||||
await probes.DisposeAsync();
|
||||
|
||||
log($"Store-and-forward drain: {config.StoreAndForwardDrainMessages:N0} messages...");
|
||||
drainResult = await StoreAndForwardDrainScenario.RunAsync(
|
||||
sites[0], config.StoreAndForwardDrainMessages, StoreAndForwardConcurrency, cancellationToken);
|
||||
log($" buffered {drainResult.EnqueuePerSecond:N0}/s, drained {drainResult.DrainPerSecond:N0}/s");
|
||||
|
||||
log($"Slow-subscriber isolation: {SlowSubscriberProbeCount} subscribers, " +
|
||||
$"{config.SlowSubscriberEvents:N0} events...");
|
||||
slowResult = await SlowSubscriberScenario.RunAsync(
|
||||
sites[1 % sites.Count], SlowSubscriberProbeCount, config.SlowSubscriberEvents, cancellationToken);
|
||||
log($" healthy min delivery {slowResult.HealthyMinDeliveryRatio:P2}, " +
|
||||
$"stalled {slowResult.SlowDeliveryRatio:P2}");
|
||||
|
||||
totalWatch.Stop();
|
||||
|
||||
return new HarnessRunResult(
|
||||
Config: config,
|
||||
Environment: CaptureEnvironment(),
|
||||
StartedUtc: startedUtc,
|
||||
TotalSeconds: totalWatch.Elapsed.TotalSeconds,
|
||||
SiteRampSeconds: siteWatch.Elapsed.TotalSeconds,
|
||||
InstanceRampSeconds: rampWatch.Elapsed.TotalSeconds,
|
||||
SlowestSiteInstanceRampSeconds: slowestSiteRamp,
|
||||
TagUpdateLatency: steadyLatency.Snapshot(),
|
||||
EmittedTagUpdates: emittedAtWindowStart + emittedInWindow,
|
||||
SteadyStateEmittedTagUpdates: emittedInWindow,
|
||||
AchievedUpdatesPerSecond: emittedInWindow / windowWatch.Elapsed.TotalSeconds,
|
||||
NominalUpdatesPerSecond: config.NominalUpdatesPerSecond,
|
||||
DriverLagSeconds: driverLagSeconds,
|
||||
DriverSkippedNoCallback: driverSkipped,
|
||||
SteadyStateResources: steadyResources,
|
||||
WholeRunResources: sampler.Summarize(0, double.MaxValue),
|
||||
ResourceSamples: sampler.Snapshot(),
|
||||
HealthReportLatency: probes.HealthReportLatency.Snapshot(),
|
||||
HealthReportsDelivered: probes.HealthReportsDelivered,
|
||||
SitesTrackedByAggregator: aggregator.GetAllSiteStates().Count,
|
||||
DebugSnapshotLatency: probes.DebugSnapshotLatency.Snapshot(),
|
||||
DebugSnapshotsCompleted: probes.DebugSnapshotsCompleted,
|
||||
DebugSnapshotTimeouts: probes.DebugSnapshotTimeouts,
|
||||
StreamProbeReceived: probeReceived,
|
||||
StreamProbeDropped: probeDropped,
|
||||
StoreAndForwardDrain: drainResult,
|
||||
SlowSubscriber: slowResult);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (driver != null) await driver.DisposeAsync();
|
||||
if (probes != null) await probes.DisposeAsync();
|
||||
await sampler.DisposeAsync();
|
||||
|
||||
log("Tearing down sites...");
|
||||
foreach (var site in sites)
|
||||
await site.DisposeAsync();
|
||||
|
||||
try
|
||||
{
|
||||
if (config.DataDirectory == null && Directory.Exists(dataRoot))
|
||||
Directory.Delete(dataRoot, recursive: true);
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
// Temp cleanup only.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static EnvironmentInfo CaptureEnvironment() => new(
|
||||
System.Environment.MachineName,
|
||||
System.Runtime.InteropServices.RuntimeInformation.OSDescription,
|
||||
System.Environment.ProcessorCount,
|
||||
System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription,
|
||||
System.Runtime.GCSettings.IsServerGC);
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
|
||||
|
||||
/// <summary>
|
||||
/// Lock-free logarithmic latency histogram sized for tens of thousands of samples
|
||||
/// per second across many threads.
|
||||
///
|
||||
/// <para>
|
||||
/// Buckets are 16-per-octave over microseconds, i.e. bucket <c>i</c> covers
|
||||
/// <c>[2^(i/16), 2^((i+1)/16))</c> µs. That bounds relative bucket width at
|
||||
/// <c>2^(1/16) - 1 ≈ 4.4%</c>, so a reported percentile is within ~4.4% of the true
|
||||
/// value — ample for the millisecond-scale thresholds this harness asserts, and far
|
||||
/// cheaper than retaining 45 million raw samples.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Recording is a <see cref="Math.Log2(double)"/> plus one
|
||||
/// <see cref="Interlocked.Increment(ref long)"/>; there is no allocation on the hot path.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class LatencyHistogram
|
||||
{
|
||||
private const int SubBucketsPerOctave = 16;
|
||||
private const int BucketCount = 64 * SubBucketsPerOctave;
|
||||
|
||||
private readonly long[] _buckets = new long[BucketCount];
|
||||
private long _count;
|
||||
private long _totalMicroseconds;
|
||||
private long _maxMicroseconds;
|
||||
|
||||
/// <summary>Number of samples recorded.</summary>
|
||||
public long Count => Interlocked.Read(ref _count);
|
||||
|
||||
/// <summary>Largest sample seen, in microseconds (exact — not bucketed).</summary>
|
||||
public double MaxMs => Interlocked.Read(ref _maxMicroseconds) / 1000.0;
|
||||
|
||||
/// <summary>Arithmetic mean in milliseconds (exact — accumulated, not bucketed).</summary>
|
||||
public double MeanMs
|
||||
{
|
||||
get
|
||||
{
|
||||
var count = Interlocked.Read(ref _count);
|
||||
return count == 0 ? 0 : Interlocked.Read(ref _totalMicroseconds) / 1000.0 / count;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records one sample. Negative durations (clock skew across the emit/receive
|
||||
/// boundary) are clamped to zero rather than discarded, so the sample count stays
|
||||
/// an honest denominator.
|
||||
/// </summary>
|
||||
/// <param name="elapsed">The measured latency.</param>
|
||||
public void Record(TimeSpan elapsed)
|
||||
{
|
||||
var micros = (long)(elapsed.TotalMilliseconds * 1000.0);
|
||||
if (micros < 0) micros = 0;
|
||||
|
||||
Interlocked.Increment(ref _count);
|
||||
Interlocked.Add(ref _totalMicroseconds, micros);
|
||||
|
||||
long observedMax;
|
||||
while (micros > (observedMax = Interlocked.Read(ref _maxMicroseconds)))
|
||||
{
|
||||
if (Interlocked.CompareExchange(ref _maxMicroseconds, micros, observedMax) == observedMax)
|
||||
break;
|
||||
}
|
||||
|
||||
Interlocked.Increment(ref _buckets[BucketIndex(micros)]);
|
||||
}
|
||||
|
||||
private static int BucketIndex(long micros)
|
||||
{
|
||||
if (micros <= 0) return 0;
|
||||
var index = (int)(Math.Log2(micros) * SubBucketsPerOctave);
|
||||
if (index < 0) return 0;
|
||||
return index >= BucketCount ? BucketCount - 1 : index;
|
||||
}
|
||||
|
||||
/// <summary>Bucket midpoint in milliseconds, used when reconstructing a percentile.</summary>
|
||||
private static double BucketMidpointMs(int index)
|
||||
{
|
||||
var low = Math.Pow(2, (double)index / SubBucketsPerOctave);
|
||||
var high = Math.Pow(2, (double)(index + 1) / SubBucketsPerOctave);
|
||||
return (low + high) / 2.0 / 1000.0;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns the requested percentile in milliseconds, or 0 when no samples were recorded.
|
||||
/// </summary>
|
||||
/// <param name="percentile">Percentile in the range 0..100 (e.g. 99 for P99).</param>
|
||||
/// <returns>The percentile value in milliseconds.</returns>
|
||||
public double PercentileMs(double percentile)
|
||||
{
|
||||
var total = Interlocked.Read(ref _count);
|
||||
if (total == 0) return 0;
|
||||
|
||||
var target = (long)Math.Ceiling(total * percentile / 100.0);
|
||||
if (target < 1) target = 1;
|
||||
|
||||
long cumulative = 0;
|
||||
for (var i = 0; i < BucketCount; i++)
|
||||
{
|
||||
cumulative += Interlocked.Read(ref _buckets[i]);
|
||||
if (cumulative >= target)
|
||||
return BucketMidpointMs(i);
|
||||
}
|
||||
|
||||
return MaxMs;
|
||||
}
|
||||
|
||||
/// <summary>Materializes the standard percentile set plus mean/max/count for reporting.</summary>
|
||||
/// <returns>A snapshot record of this histogram.</returns>
|
||||
public LatencySnapshot Snapshot() => new(
|
||||
Count,
|
||||
MeanMs,
|
||||
PercentileMs(50),
|
||||
PercentileMs(95),
|
||||
PercentileMs(99),
|
||||
PercentileMs(99.9),
|
||||
MaxMs);
|
||||
}
|
||||
|
||||
/// <summary>Point-in-time summary of a <see cref="LatencyHistogram"/>. All times in milliseconds.</summary>
|
||||
/// <param name="Count">Samples recorded.</param>
|
||||
/// <param name="MeanMs">Arithmetic mean.</param>
|
||||
/// <param name="P50Ms">Median.</param>
|
||||
/// <param name="P95Ms">95th percentile.</param>
|
||||
/// <param name="P99Ms">99th percentile.</param>
|
||||
/// <param name="P999Ms">99.9th percentile.</param>
|
||||
/// <param name="MaxMs">Largest observed sample.</param>
|
||||
public sealed record LatencySnapshot(
|
||||
long Count,
|
||||
double MeanMs,
|
||||
double P50Ms,
|
||||
double P95Ms,
|
||||
double P99Ms,
|
||||
double P999Ms,
|
||||
double MaxMs);
|
||||
@@ -0,0 +1,202 @@
|
||||
using System.Diagnostics;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
|
||||
|
||||
/// <summary>
|
||||
/// One periodic observation of process resource usage.
|
||||
/// </summary>
|
||||
/// <param name="ElapsedSeconds">Seconds since sampling started.</param>
|
||||
/// <param name="WorkingSetMb">Process working set (RSS).</param>
|
||||
/// <param name="ManagedHeapMb"><see cref="GC.GetTotalMemory(bool)"/> without forcing a collection.</param>
|
||||
/// <param name="CpuPercent">Mean CPU utilization since the previous sample, as a percentage of ONE core.</param>
|
||||
/// <param name="ThreadCount">OS threads in the process.</param>
|
||||
/// <param name="Gen2Collections">Cumulative gen-2 collections.</param>
|
||||
public sealed record ResourceSample(
|
||||
double ElapsedSeconds,
|
||||
double WorkingSetMb,
|
||||
double ManagedHeapMb,
|
||||
double CpuPercent,
|
||||
int ThreadCount,
|
||||
int Gen2Collections);
|
||||
|
||||
/// <summary>
|
||||
/// Samples working set, managed heap, CPU and thread count on a fixed cadence for the
|
||||
/// life of a run. CPU is differential (processor time delta / wall delta) so a sample
|
||||
/// reflects the interval it covers rather than the whole process lifetime.
|
||||
/// </summary>
|
||||
public sealed class ResourceSampler : IAsyncDisposable
|
||||
{
|
||||
private readonly List<ResourceSample> _samples = new();
|
||||
private readonly object _lock = new();
|
||||
private readonly CancellationTokenSource _cts = new();
|
||||
private readonly Task _loop;
|
||||
private readonly Stopwatch _wall = Stopwatch.StartNew();
|
||||
|
||||
private TimeSpan _lastCpu;
|
||||
private double _lastElapsedSeconds;
|
||||
|
||||
private ResourceSampler(TimeSpan interval)
|
||||
{
|
||||
_lastCpu = Process.GetCurrentProcess().TotalProcessorTime;
|
||||
_loop = Task.Run(() => SampleLoopAsync(interval, _cts.Token));
|
||||
}
|
||||
|
||||
/// <summary>Starts sampling at the given cadence.</summary>
|
||||
/// <param name="interval">Sampling interval.</param>
|
||||
/// <returns>The running sampler.</returns>
|
||||
public static ResourceSampler Start(TimeSpan interval) => new(interval);
|
||||
|
||||
/// <summary>All samples collected so far, oldest first.</summary>
|
||||
/// <returns>A snapshot copy of the sample list.</returns>
|
||||
public IReadOnlyList<ResourceSample> Snapshot()
|
||||
{
|
||||
lock (_lock) return _samples.ToList();
|
||||
}
|
||||
|
||||
private async Task SampleLoopAsync(TimeSpan interval, CancellationToken cancellationToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(interval);
|
||||
try
|
||||
{
|
||||
while (await timer.WaitForNextTickAsync(cancellationToken))
|
||||
Capture();
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Normal teardown.
|
||||
}
|
||||
}
|
||||
|
||||
private void Capture()
|
||||
{
|
||||
using var process = Process.GetCurrentProcess();
|
||||
process.Refresh();
|
||||
|
||||
var elapsedSeconds = _wall.Elapsed.TotalSeconds;
|
||||
var cpu = process.TotalProcessorTime;
|
||||
var wallDelta = elapsedSeconds - _lastElapsedSeconds;
|
||||
var cpuPercent = wallDelta > 0
|
||||
? (cpu - _lastCpu).TotalSeconds / wallDelta * 100.0
|
||||
: 0.0;
|
||||
|
||||
_lastCpu = cpu;
|
||||
_lastElapsedSeconds = elapsedSeconds;
|
||||
|
||||
var sample = new ResourceSample(
|
||||
elapsedSeconds,
|
||||
process.WorkingSet64 / 1024.0 / 1024.0,
|
||||
GC.GetTotalMemory(forceFullCollection: false) / 1024.0 / 1024.0,
|
||||
cpuPercent,
|
||||
process.Threads.Count,
|
||||
GC.CollectionCount(2));
|
||||
|
||||
lock (_lock) _samples.Add(sample);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Summarizes the samples falling inside a window, expressed as seconds since
|
||||
/// sampling started. Memory growth is reported both as an absolute delta and as a
|
||||
/// least-squares slope, because a run that sawtooths around a stable mean and a
|
||||
/// run that climbs monotonically can share the same endpoint delta.
|
||||
/// </summary>
|
||||
/// <param name="fromSeconds">Window start (inclusive), seconds since start.</param>
|
||||
/// <param name="toSeconds">Window end (inclusive), seconds since start.</param>
|
||||
/// <returns>The window summary, or null when fewer than two samples fall inside it.</returns>
|
||||
public ResourceWindowSummary? Summarize(double fromSeconds, double toSeconds)
|
||||
{
|
||||
var window = Snapshot()
|
||||
.Where(s => s.ElapsedSeconds >= fromSeconds && s.ElapsedSeconds <= toSeconds)
|
||||
.ToList();
|
||||
|
||||
if (window.Count < 2)
|
||||
return null;
|
||||
|
||||
var first = window[0];
|
||||
var last = window[^1];
|
||||
|
||||
return new ResourceWindowSummary(
|
||||
SampleCount: window.Count,
|
||||
DurationSeconds: last.ElapsedSeconds - first.ElapsedSeconds,
|
||||
WorkingSetStartMb: first.WorkingSetMb,
|
||||
WorkingSetEndMb: last.WorkingSetMb,
|
||||
WorkingSetPeakMb: window.Max(s => s.WorkingSetMb),
|
||||
WorkingSetSlopeMbPerMinute: Slope(window, s => s.WorkingSetMb) * 60.0,
|
||||
ManagedHeapStartMb: first.ManagedHeapMb,
|
||||
ManagedHeapEndMb: last.ManagedHeapMb,
|
||||
ManagedHeapPeakMb: window.Max(s => s.ManagedHeapMb),
|
||||
ManagedHeapSlopeMbPerMinute: Slope(window, s => s.ManagedHeapMb) * 60.0,
|
||||
MeanCpuPercentOfOneCore: window.Average(s => s.CpuPercent),
|
||||
PeakCpuPercentOfOneCore: window.Max(s => s.CpuPercent),
|
||||
MeanThreadCount: window.Average(s => s.ThreadCount),
|
||||
Gen2Collections: last.Gen2Collections - first.Gen2Collections);
|
||||
}
|
||||
|
||||
private static double Slope(IReadOnlyList<ResourceSample> samples, Func<ResourceSample, double> selector)
|
||||
{
|
||||
var n = samples.Count;
|
||||
var meanX = samples.Average(s => s.ElapsedSeconds);
|
||||
var meanY = samples.Average(selector);
|
||||
|
||||
double numerator = 0, denominator = 0;
|
||||
for (var i = 0; i < n; i++)
|
||||
{
|
||||
var dx = samples[i].ElapsedSeconds - meanX;
|
||||
numerator += dx * (selector(samples[i]) - meanY);
|
||||
denominator += dx * dx;
|
||||
}
|
||||
|
||||
return denominator == 0 ? 0 : numerator / denominator;
|
||||
}
|
||||
|
||||
private int _disposed;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _disposed, 1) != 0)
|
||||
return;
|
||||
|
||||
await _cts.CancelAsync();
|
||||
try
|
||||
{
|
||||
await _loop;
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Expected.
|
||||
}
|
||||
|
||||
_cts.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Aggregate resource behaviour over a measurement window.</summary>
|
||||
/// <param name="SampleCount">Samples in the window.</param>
|
||||
/// <param name="DurationSeconds">Window length.</param>
|
||||
/// <param name="WorkingSetStartMb">Working set at window start.</param>
|
||||
/// <param name="WorkingSetEndMb">Working set at window end.</param>
|
||||
/// <param name="WorkingSetPeakMb">Peak working set in the window.</param>
|
||||
/// <param name="WorkingSetSlopeMbPerMinute">Least-squares working-set growth rate.</param>
|
||||
/// <param name="ManagedHeapStartMb">Managed heap at window start.</param>
|
||||
/// <param name="ManagedHeapEndMb">Managed heap at window end.</param>
|
||||
/// <param name="ManagedHeapPeakMb">Peak managed heap in the window.</param>
|
||||
/// <param name="ManagedHeapSlopeMbPerMinute">Least-squares managed-heap growth rate.</param>
|
||||
/// <param name="MeanCpuPercentOfOneCore">Mean CPU as a percentage of one core (1400% = 14 cores saturated).</param>
|
||||
/// <param name="PeakCpuPercentOfOneCore">Peak single-sample CPU as a percentage of one core.</param>
|
||||
/// <param name="MeanThreadCount">Mean OS thread count.</param>
|
||||
/// <param name="Gen2Collections">Gen-2 collections during the window.</param>
|
||||
public sealed record ResourceWindowSummary(
|
||||
int SampleCount,
|
||||
double DurationSeconds,
|
||||
double WorkingSetStartMb,
|
||||
double WorkingSetEndMb,
|
||||
double WorkingSetPeakMb,
|
||||
double WorkingSetSlopeMbPerMinute,
|
||||
double ManagedHeapStartMb,
|
||||
double ManagedHeapEndMb,
|
||||
double ManagedHeapPeakMb,
|
||||
double ManagedHeapSlopeMbPerMinute,
|
||||
double MeanCpuPercentOfOneCore,
|
||||
double PeakCpuPercentOfOneCore,
|
||||
double MeanThreadCount,
|
||||
int Gen2Collections);
|
||||
@@ -0,0 +1,205 @@
|
||||
using System.Threading.Channels;
|
||||
using Akka.Actor;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Probes;
|
||||
|
||||
/// <summary>
|
||||
/// A live site-stream subscriber assembled from the SAME parts
|
||||
/// <c>SiteStreamGrpcServer.RunSubscriptionStreamAsync</c> uses:
|
||||
///
|
||||
/// <list type="number">
|
||||
/// <item><description><c>SiteStreamManager.Subscribe</c> — materializes the
|
||||
/// per-subscriber graph (<c>Where</c> instance filter → <c>Buffer(StreamBufferSize,
|
||||
/// DropHead)</c> → <c>Sink.ForEach(Tell)</c>).</description></item>
|
||||
/// <item><description>A real <see cref="StreamRelayActor"/>, which converts the Akka
|
||||
/// record to the protobuf <c>SiteStreamEvent</c> and <c>TryWrite</c>s it.</description></item>
|
||||
/// <item><description>A bounded <c>DropOldest</c> <see cref="Channel"/> of the
|
||||
/// production capacity (<c>GrpcInstanceStreamChannelCapacity</c> = 1000) with the
|
||||
/// eviction counter wired to <see cref="DroppedEvents"/>.</description></item>
|
||||
/// </list>
|
||||
///
|
||||
/// <para>
|
||||
/// The single substitution is the final hop: instead of
|
||||
/// <c>responseStream.WriteAsync</c> pushing onto a socket, a reader task drains the
|
||||
/// channel. That is deliberate — it is precisely the hop whose slowness register row
|
||||
/// 50 asks about, and a controllable reader is the only way to hold it still.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class StreamSubscriberProbe : IAsyncDisposable
|
||||
{
|
||||
/// <summary>Production per-instance stream channel capacity (<c>GrpcInstanceStreamChannelCapacity</c>).</summary>
|
||||
public const int ProductionChannelCapacity = 1000;
|
||||
|
||||
private readonly SiteStreamManager _manager;
|
||||
private readonly string _subscriptionId;
|
||||
private readonly IActorRef _relayActor;
|
||||
private readonly ActorSystem _system;
|
||||
private readonly Channel<SiteStreamEvent> _channel;
|
||||
private readonly CancellationTokenSource _cts = new();
|
||||
private readonly Task _readerTask;
|
||||
private volatile LatencyHistogram? _latency;
|
||||
|
||||
private readonly DropCounter _dropCounter;
|
||||
private long _received;
|
||||
private long _readerDelayMicroseconds;
|
||||
private int _disposed;
|
||||
|
||||
/// <summary>Human-readable probe name (also the relay actor's name suffix).</summary>
|
||||
public string Name { get; }
|
||||
|
||||
/// <summary>Events evicted by the bounded channel's DropOldest policy.</summary>
|
||||
public long DroppedEvents => _dropCounter.Value;
|
||||
|
||||
/// <summary>Events successfully drained by the reader (i.e. "sent to the client").</summary>
|
||||
public long ReceivedEvents => Interlocked.Read(ref _received);
|
||||
|
||||
/// <summary>
|
||||
/// Repoints the latency histogram this probe records into, without tearing the
|
||||
/// subscription down. Used to separate ramp-window samples from steady-state ones:
|
||||
/// re-attaching probes instead would open a zero-subscriber gap (during which
|
||||
/// <c>PublishAttributeValueChanged</c> short-circuits) and risk reusing an actor
|
||||
/// name whose previous incarnation has not finished terminating.
|
||||
/// </summary>
|
||||
/// <param name="latency">The histogram to record into from now on, or null to stop recording.</param>
|
||||
public void RetargetLatency(LatencyHistogram? latency) => _latency = latency;
|
||||
|
||||
/// <summary>
|
||||
/// Artificial per-event reader delay, in microseconds. Zero is a healthy
|
||||
/// subscriber; a large value models a stalled WAN link or a wedged client.
|
||||
/// </summary>
|
||||
public long ReaderDelayMicroseconds
|
||||
{
|
||||
get => Interlocked.Read(ref _readerDelayMicroseconds);
|
||||
set => Interlocked.Exchange(ref _readerDelayMicroseconds, value);
|
||||
}
|
||||
|
||||
private StreamSubscriberProbe(
|
||||
ActorSystem system,
|
||||
SiteStreamManager manager,
|
||||
string name,
|
||||
Channel<SiteStreamEvent> channel,
|
||||
IActorRef relayActor,
|
||||
string subscriptionId,
|
||||
LatencyHistogram? latency,
|
||||
DropCounter dropCounter)
|
||||
{
|
||||
_system = system;
|
||||
_manager = manager;
|
||||
Name = name;
|
||||
_channel = channel;
|
||||
_relayActor = relayActor;
|
||||
_subscriptionId = subscriptionId;
|
||||
_latency = latency;
|
||||
_dropCounter = dropCounter;
|
||||
_readerTask = Task.Run(() => ReadLoopAsync(_cts.Token));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds and attaches a probe subscribed to one instance's events.
|
||||
/// </summary>
|
||||
/// <param name="system">The site actor system.</param>
|
||||
/// <param name="manager">The site stream manager to subscribe against.</param>
|
||||
/// <param name="instanceUniqueName">Instance whose events this probe receives.</param>
|
||||
/// <param name="name">Probe name, used for the relay actor's path.</param>
|
||||
/// <param name="latency">Optional histogram fed with end-to-end event latency.</param>
|
||||
/// <returns>The attached probe.</returns>
|
||||
public static StreamSubscriberProbe Attach(
|
||||
ActorSystem system,
|
||||
SiteStreamManager manager,
|
||||
string instanceUniqueName,
|
||||
string name,
|
||||
LatencyHistogram? latency)
|
||||
{
|
||||
var dropCounter = new DropCounter();
|
||||
var channel = Channel.CreateBounded<SiteStreamEvent>(
|
||||
new BoundedChannelOptions(ProductionChannelCapacity)
|
||||
{
|
||||
FullMode = BoundedChannelFullMode.DropOldest,
|
||||
},
|
||||
_ => dropCounter.Increment());
|
||||
|
||||
var relayActor = system.ActorOf(
|
||||
Props.Create(typeof(StreamRelayActor), name, channel.Writer),
|
||||
$"stream-relay-{name}");
|
||||
|
||||
var subscriptionId = manager.Subscribe(instanceUniqueName, relayActor);
|
||||
|
||||
return new StreamSubscriberProbe(
|
||||
system, manager, name, channel, relayActor, subscriptionId, latency, dropCounter);
|
||||
}
|
||||
|
||||
private async Task ReadLoopAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
await foreach (var evt in _channel.Reader.ReadAllAsync(cancellationToken))
|
||||
{
|
||||
Interlocked.Increment(ref _received);
|
||||
|
||||
var latency = _latency;
|
||||
if (latency != null && evt.AttributeChanged != null)
|
||||
{
|
||||
// The emit instant travels verbatim: the driver stamps it on
|
||||
// TagValueUpdate.Timestamp, DataConnectionActor forwards it,
|
||||
// InstanceActor copies it onto AttributeValueChanged.Timestamp, and
|
||||
// StreamRelayActor maps it onto the proto Timestamp. So this is a
|
||||
// true end-to-end DCL-boundary → subscriber measurement.
|
||||
var emitted = evt.AttributeChanged.Timestamp.ToDateTimeOffset();
|
||||
latency.Record(DateTimeOffset.UtcNow - emitted);
|
||||
}
|
||||
|
||||
var delay = Interlocked.Read(ref _readerDelayMicroseconds);
|
||||
if (delay > 0)
|
||||
await Task.Delay(TimeSpan.FromMicroseconds(delay), cancellationToken);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Normal teardown.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Detaches the subscription and stops the relay actor and reader.</summary>
|
||||
/// <returns>A task that completes when the probe is torn down.</returns>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _disposed, 1) != 0)
|
||||
return;
|
||||
|
||||
_manager.Unsubscribe(_subscriptionId);
|
||||
_channel.Writer.TryComplete();
|
||||
await _cts.CancelAsync();
|
||||
|
||||
try
|
||||
{
|
||||
await _readerTask;
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Expected.
|
||||
}
|
||||
|
||||
_system.Stop(_relayActor);
|
||||
_cts.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Thread-safe counter for a bounded channel's <c>itemDropped</c> callback. A tiny
|
||||
/// class rather than a captured local so the probe and the channel share exactly one
|
||||
/// counter instance without a second closure.
|
||||
/// </summary>
|
||||
public sealed class DropCounter
|
||||
{
|
||||
private long _value;
|
||||
|
||||
/// <summary>Current count.</summary>
|
||||
public long Value => Interlocked.Read(ref _value);
|
||||
|
||||
/// <summary>Increments the counter.</summary>
|
||||
public void Increment() => Interlocked.Increment(ref _value);
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
using System.Text.Json;
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness;
|
||||
|
||||
// Target-scale load harness (Phase-8 WP-4 / deferred-work register #25 + row 50).
|
||||
//
|
||||
// Full-scale protocol run (the WP-4 numbers):
|
||||
// dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
|
||||
// --sustain-minutes 20 --results loadharness-results.json
|
||||
//
|
||||
// Full 1-hour version, unchanged in every other respect:
|
||||
// ... -- --sustain-minutes 60 --results loadharness-results-1h.json
|
||||
//
|
||||
// Scaled-down smoke (what the CI [Fact] runs):
|
||||
// ... -- --sites 2 --instances-per-site 10 --tags-per-instance 5 \
|
||||
// --settle-minutes 0.1 --sustain-minutes 0.2 --sample-seconds 2 \
|
||||
// --sf-drain-messages 200 --slow-subscriber-events 2000
|
||||
|
||||
var config = HarnessConfig.Parse(args);
|
||||
|
||||
Console.WriteLine("ScadaBridge target-scale load harness");
|
||||
Console.WriteLine($" sites {config.Sites}");
|
||||
Console.WriteLine($" instances/site {config.InstancesPerSite}");
|
||||
Console.WriteLine($" tags/instance {config.TagsPerInstance}");
|
||||
Console.WriteLine($" total subscriptions {config.TotalSubscriptions:N0}");
|
||||
Console.WriteLine($" nominal update rate {config.NominalUpdatesPerSecond:N0}/s");
|
||||
Console.WriteLine($" settle / sustain {config.SettleDuration.TotalMinutes:F1} / {config.SustainDuration.TotalMinutes:F1} min");
|
||||
Console.WriteLine();
|
||||
|
||||
using var cancellation = new CancellationTokenSource();
|
||||
Console.CancelKeyPress += (_, e) =>
|
||||
{
|
||||
e.Cancel = true;
|
||||
cancellation.Cancel();
|
||||
};
|
||||
|
||||
try
|
||||
{
|
||||
var result = await HarnessRun.ExecuteAsync(config, Console.WriteLine, cancellation.Token);
|
||||
|
||||
var json = JsonSerializer.Serialize(result, new JsonSerializerOptions { WriteIndented = true });
|
||||
await File.WriteAllTextAsync(config.ResultsPath, json, cancellation.Token);
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine(ResultsFormatter.Format(result));
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($"JSON metrics written to {Path.GetFullPath(config.ResultsPath)}");
|
||||
return 0;
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
Console.Error.WriteLine("Run cancelled.");
|
||||
return 130;
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
using System.Globalization;
|
||||
using System.Text;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness;
|
||||
|
||||
/// <summary>
|
||||
/// Renders a <see cref="HarnessRunResult"/> as the markdown table that goes into the
|
||||
/// results document, so the published numbers and the JSON come from one source.
|
||||
/// </summary>
|
||||
public static class ResultsFormatter
|
||||
{
|
||||
/// <summary>Formats a run result for console output and the results doc.</summary>
|
||||
/// <param name="result">The run to format.</param>
|
||||
/// <returns>A markdown-ish plain-text report.</returns>
|
||||
public static string Format(HarnessRunResult result)
|
||||
{
|
||||
var culture = CultureInfo.InvariantCulture;
|
||||
var builder = new StringBuilder();
|
||||
|
||||
builder.AppendLine("=== MEASURED ===");
|
||||
builder.AppendLine(culture, $"host {result.Environment.MachineName} / {result.Environment.OsDescription}");
|
||||
builder.AppendLine(culture, $"cpus / runtime / gc {result.Environment.ProcessorCount} / {result.Environment.RuntimeVersion} / serverGC={result.Environment.ServerGc}");
|
||||
builder.AppendLine(culture, $"scale {result.Config.Sites} sites x {result.Config.InstancesPerSite} instances x {result.Config.TagsPerInstance} tags = {result.Config.TotalSubscriptions:N0} subscriptions");
|
||||
builder.AppendLine(culture, $"total run {result.TotalSeconds:F1}s");
|
||||
builder.AppendLine();
|
||||
|
||||
builder.AppendLine("-- deployment / ramp --");
|
||||
builder.AppendLine(culture, $"site fixtures built {result.SiteRampSeconds:F1}s");
|
||||
builder.AppendLine(culture, $"all instance actors {result.InstanceRampSeconds:F1}s");
|
||||
builder.AppendLine(culture, $"slowest single site {result.SlowestSiteInstanceRampSeconds:F1}s ({result.Config.InstancesPerSite} instances)");
|
||||
builder.AppendLine();
|
||||
|
||||
builder.AppendLine("-- offered load (steady-state window) --");
|
||||
builder.AppendLine(culture, $"nominal {result.NominalUpdatesPerSecond:N0} updates/s");
|
||||
builder.AppendLine(culture, $"achieved {result.AchievedUpdatesPerSecond:N0} updates/s ({result.AchievedUpdatesPerSecond / result.NominalUpdatesPerSecond:P1} of nominal)");
|
||||
builder.AppendLine(culture, $"updates offered {result.SteadyStateEmittedTagUpdates:N0}");
|
||||
builder.AppendLine(culture, $"driver slice overrun {result.DriverLagSeconds:F1}s cumulative");
|
||||
builder.AppendLine(culture, $"skipped (no callback) {result.DriverSkippedNoCallback:N0}");
|
||||
builder.AppendLine();
|
||||
|
||||
builder.AppendLine("-- tag update latency (DCL boundary -> stream subscriber) --");
|
||||
AppendLatency(builder, culture, result.TagUpdateLatency);
|
||||
builder.AppendLine();
|
||||
|
||||
builder.AppendLine("-- live stream subscribers (steady-state window) --");
|
||||
builder.AppendLine(culture, $"events delivered {result.StreamProbeReceived:N0}");
|
||||
builder.AppendLine(culture, $"events dropped {result.StreamProbeDropped:N0}");
|
||||
builder.AppendLine();
|
||||
|
||||
builder.AppendLine("-- health report delivery --");
|
||||
AppendLatency(builder, culture, result.HealthReportLatency);
|
||||
builder.AppendLine(culture, $"reports ingested {result.HealthReportsDelivered:N0}");
|
||||
builder.AppendLine(culture, $"sites tracked centrally {result.SitesTrackedByAggregator}");
|
||||
builder.AppendLine();
|
||||
|
||||
builder.AppendLine("-- debug view snapshot (under load) --");
|
||||
AppendLatency(builder, culture, result.DebugSnapshotLatency);
|
||||
builder.AppendLine(culture, $"completed / timed out {result.DebugSnapshotsCompleted:N0} / {result.DebugSnapshotTimeouts:N0}");
|
||||
builder.AppendLine();
|
||||
|
||||
if (result.SteadyStateResources is { } steady)
|
||||
{
|
||||
builder.AppendLine("-- resources (steady-state window) --");
|
||||
builder.AppendLine(culture, $"window {steady.DurationSeconds:F0}s over {steady.SampleCount} samples");
|
||||
builder.AppendLine(culture, $"working set {steady.WorkingSetStartMb:F0} -> {steady.WorkingSetEndMb:F0} MB (peak {steady.WorkingSetPeakMb:F0} MB)");
|
||||
builder.AppendLine(culture, $"working set slope {steady.WorkingSetSlopeMbPerMinute:F2} MB/min");
|
||||
builder.AppendLine(culture, $"managed heap {steady.ManagedHeapStartMb:F0} -> {steady.ManagedHeapEndMb:F0} MB (peak {steady.ManagedHeapPeakMb:F0} MB)");
|
||||
builder.AppendLine(culture, $"managed heap slope {steady.ManagedHeapSlopeMbPerMinute:F2} MB/min");
|
||||
builder.AppendLine(culture, $"cpu mean / peak {steady.MeanCpuPercentOfOneCore:F0}% / {steady.PeakCpuPercentOfOneCore:F0}% of one core ({steady.MeanCpuPercentOfOneCore / result.Environment.ProcessorCount:F1}% of the box)");
|
||||
builder.AppendLine(culture, $"threads / gen2 GCs {steady.MeanThreadCount:F0} / {steady.Gen2Collections}");
|
||||
builder.AppendLine();
|
||||
}
|
||||
|
||||
if (result.StoreAndForwardDrain is { } drain)
|
||||
{
|
||||
builder.AppendLine("-- store-and-forward (register row 50) --");
|
||||
builder.AppendLine(culture, $"messages {drain.MessageCount:N0}");
|
||||
builder.AppendLine(culture, $"concurrent buffering {drain.EnqueuePerSecond:N0} msg/s ({drain.EnqueueSeconds:F1}s)");
|
||||
builder.AppendLine(culture, $"retry wait before drain {drain.TimeToFirstDeliverySeconds:F1}s (DefaultRetryInterval)");
|
||||
builder.AppendLine(culture, $"drain throughput {drain.DrainPerSecond:N0} msg/s (active drain {drain.DrainSeconds - drain.TimeToFirstDeliverySeconds:F2}s)");
|
||||
builder.AppendLine(culture, $"residual depth {drain.ResidualDepth}");
|
||||
builder.AppendLine();
|
||||
}
|
||||
|
||||
if (result.SlowSubscriber is { } slow)
|
||||
{
|
||||
builder.AppendLine("-- slow-subscriber isolation (register row 50) --");
|
||||
builder.AppendLine(culture, $"published {slow.PublishedEvents:N0} at {slow.PublishPerSecond:N0}/s");
|
||||
builder.AppendLine(culture, $"healthy min delivery {slow.HealthyMinDeliveryRatio:P2}");
|
||||
builder.AppendLine(culture, $"stalled delivery {slow.SlowDeliveryRatio:P2}");
|
||||
foreach (var outcome in slow.Outcomes)
|
||||
{
|
||||
builder.AppendLine(culture,
|
||||
$" {outcome.Name,-28} {(outcome.IsSlow ? "STALLED" : "healthy"),-8} " +
|
||||
$"recv {outcome.Received,8:N0} dropped {outcome.Dropped,8:N0} ratio {outcome.DeliveryRatio:P2}");
|
||||
}
|
||||
|
||||
builder.AppendLine();
|
||||
}
|
||||
|
||||
return builder.ToString();
|
||||
}
|
||||
|
||||
private static void AppendLatency(StringBuilder builder, CultureInfo culture, Metrics.LatencySnapshot snapshot)
|
||||
{
|
||||
builder.AppendLine(culture,
|
||||
$"samples {snapshot.Count:N0} mean {snapshot.MeanMs:F2}ms p50 {snapshot.P50Ms:F2}ms " +
|
||||
$"p95 {snapshot.P95Ms:F2}ms p99 {snapshot.P99Ms:F2}ms p99.9 {snapshot.P999Ms:F2}ms max {snapshot.MaxMs:F2}ms");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
using System.Diagnostics;
|
||||
using Akka.Actor;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
|
||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Scenarios;
|
||||
|
||||
/// <summary>
|
||||
/// The two "does observability still work at scale?" probes WP-4's test protocol
|
||||
/// names — health report delivery timing and debug view latency — run continuously
|
||||
/// alongside the sustained load rather than after it, so both are measured against a
|
||||
/// site that is actually busy.
|
||||
/// </summary>
|
||||
public sealed class ObservabilityProbes : IAsyncDisposable
|
||||
{
|
||||
private readonly CancellationTokenSource _cts = new();
|
||||
private readonly List<Task> _tasks = new();
|
||||
|
||||
/// <summary>
|
||||
/// End-to-end health report latency: <c>SiteHealthCollector.CollectReport</c> plus
|
||||
/// the transport hop plus <c>CentralHealthAggregator.ProcessReport</c>. The
|
||||
/// interesting term at scale is <c>CollectReport</c>, which materializes the
|
||||
/// per-connection dictionaries for a site carrying 37,500 subscriptions.
|
||||
/// </summary>
|
||||
public LatencyHistogram HealthReportLatency { get; } = new();
|
||||
|
||||
/// <summary>
|
||||
/// Debug view snapshot round-trip: an <c>Ask</c> of <c>DebugSnapshotRequest</c> to
|
||||
/// a live Instance Actor. This lands the request in the mailbox of an actor that
|
||||
/// is concurrently ingesting tag updates, so the measured time includes real
|
||||
/// queueing behind production traffic — which is the whole point of measuring it
|
||||
/// under load.
|
||||
/// </summary>
|
||||
public LatencyHistogram DebugSnapshotLatency { get; } = new();
|
||||
|
||||
/// <summary>Health reports successfully ingested by the central aggregator.</summary>
|
||||
public long HealthReportsDelivered => Interlocked.Read(ref _healthReports);
|
||||
|
||||
/// <summary>Debug snapshots that completed within the ask timeout.</summary>
|
||||
public long DebugSnapshotsCompleted => Interlocked.Read(ref _debugSnapshots);
|
||||
|
||||
/// <summary>Debug snapshot asks that timed out.</summary>
|
||||
public long DebugSnapshotTimeouts => Interlocked.Read(ref _debugTimeouts);
|
||||
|
||||
private long _healthReports;
|
||||
private long _debugSnapshots;
|
||||
private long _debugTimeouts;
|
||||
|
||||
/// <summary>
|
||||
/// Starts both probes.
|
||||
/// </summary>
|
||||
/// <param name="sites">Sites to probe.</param>
|
||||
/// <param name="aggregator">The real central aggregator receiving the reports.</param>
|
||||
/// <param name="reportInterval">Health report cadence (production default 30 s).</param>
|
||||
/// <param name="debugProbeInterval">How often to take a debug snapshot.</param>
|
||||
/// <returns>The running probes.</returns>
|
||||
public static ObservabilityProbes Start(
|
||||
IReadOnlyList<SiteRuntimeFixture> sites,
|
||||
CentralHealthAggregator aggregator,
|
||||
TimeSpan reportInterval,
|
||||
TimeSpan debugProbeInterval)
|
||||
{
|
||||
var probes = new ObservabilityProbes();
|
||||
probes._tasks.Add(Task.Run(() => probes.HealthLoopAsync(sites, aggregator, reportInterval, probes._cts.Token)));
|
||||
probes._tasks.Add(Task.Run(() => probes.DebugLoopAsync(sites, debugProbeInterval, probes._cts.Token)));
|
||||
return probes;
|
||||
}
|
||||
|
||||
private async Task HealthLoopAsync(
|
||||
IReadOnlyList<SiteRuntimeFixture> sites,
|
||||
CentralHealthAggregator aggregator,
|
||||
TimeSpan interval,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(interval);
|
||||
try
|
||||
{
|
||||
while (await timer.WaitForNextTickAsync(cancellationToken))
|
||||
{
|
||||
foreach (var site in sites)
|
||||
{
|
||||
var watch = Stopwatch.StartNew();
|
||||
var report = site.HealthCollector.CollectReport(site.SiteId);
|
||||
aggregator.ProcessReport(report);
|
||||
watch.Stop();
|
||||
|
||||
HealthReportLatency.Record(watch.Elapsed);
|
||||
Interlocked.Increment(ref _healthReports);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Normal teardown.
|
||||
}
|
||||
}
|
||||
|
||||
private async Task DebugLoopAsync(
|
||||
IReadOnlyList<SiteRuntimeFixture> sites,
|
||||
TimeSpan interval,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var random = new Random(20260815);
|
||||
using var timer = new PeriodicTimer(interval);
|
||||
try
|
||||
{
|
||||
while (await timer.WaitForNextTickAsync(cancellationToken))
|
||||
{
|
||||
var site = sites[random.Next(sites.Count)];
|
||||
if (site.InstanceActors.Count == 0)
|
||||
continue;
|
||||
|
||||
var index = random.Next(site.InstanceActors.Count);
|
||||
var actor = site.InstanceActors[index];
|
||||
var request = new DebugSnapshotRequest(site.InstanceName(index), Guid.NewGuid().ToString("N"));
|
||||
|
||||
var watch = Stopwatch.StartNew();
|
||||
try
|
||||
{
|
||||
await actor.Ask<DebugViewSnapshot>(request, TimeSpan.FromSeconds(10), cancellationToken);
|
||||
watch.Stop();
|
||||
DebugSnapshotLatency.Record(watch.Elapsed);
|
||||
Interlocked.Increment(ref _debugSnapshots);
|
||||
}
|
||||
catch (AskTimeoutException)
|
||||
{
|
||||
Interlocked.Increment(ref _debugTimeouts);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Normal teardown.
|
||||
}
|
||||
}
|
||||
|
||||
private int _disposed;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
// Idempotent: the orchestrator stops the probes early (so the register row 50
|
||||
// scenarios do not compete with them) and again in its finally block.
|
||||
if (Interlocked.Exchange(ref _disposed, 1) != 0)
|
||||
return;
|
||||
|
||||
await _cts.CancelAsync();
|
||||
try
|
||||
{
|
||||
await Task.WhenAll(_tasks);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Expected.
|
||||
}
|
||||
|
||||
_cts.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
using System.Diagnostics;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness.Probes;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Scenarios;
|
||||
|
||||
/// <summary>Per-subscriber outcome from the slow-subscriber isolation probe.</summary>
|
||||
/// <param name="Name">Probe name.</param>
|
||||
/// <param name="IsSlow">Whether this probe's reader was deliberately stalled.</param>
|
||||
/// <param name="Received">Events the reader drained.</param>
|
||||
/// <param name="Dropped">Events evicted by this probe's bounded DropOldest channel.</param>
|
||||
/// <param name="DeliveryRatio">Received / published, before accounting for the site-stream buffer.</param>
|
||||
public sealed record SubscriberOutcome(
|
||||
string Name,
|
||||
bool IsSlow,
|
||||
long Received,
|
||||
long Dropped,
|
||||
double DeliveryRatio);
|
||||
|
||||
/// <summary>Result of the slow-subscriber isolation measurement.</summary>
|
||||
/// <param name="PublishedEvents">Events published to the site stream during the probe.</param>
|
||||
/// <param name="PublishSeconds">Wall time the publisher took.</param>
|
||||
/// <param name="PublishPerSecond">Publish throughput observed by the producer.</param>
|
||||
/// <param name="Outcomes">Per-subscriber outcomes.</param>
|
||||
/// <param name="HealthyMinDeliveryRatio">Worst delivery ratio among the healthy subscribers.</param>
|
||||
/// <param name="SlowDeliveryRatio">Delivery ratio of the stalled subscriber.</param>
|
||||
public sealed record SlowSubscriberResult(
|
||||
int PublishedEvents,
|
||||
double PublishSeconds,
|
||||
double PublishPerSecond,
|
||||
IReadOnlyList<SubscriberOutcome> Outcomes,
|
||||
double HealthyMinDeliveryRatio,
|
||||
double SlowDeliveryRatio);
|
||||
|
||||
/// <summary>
|
||||
/// Register row 50, second half: what does a slow or stalled gRPC subscriber do to
|
||||
/// per-subscriber buffering when several subscribers are attached?
|
||||
///
|
||||
/// <para>
|
||||
/// Several probes are attached to the SAME instance so every one of them is offered
|
||||
/// exactly the same event sequence — otherwise a difference in delivery could be a
|
||||
/// difference in offered load rather than a backpressure effect. One probe's reader
|
||||
/// is then stalled (a large per-event delay, standing in for a wedged client or a
|
||||
/// dead WAN link) while the rest read as fast as they can. Events are published
|
||||
/// through the real <c>SiteStreamManager</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The question the numbers answer: does the stalled subscriber's backlog propagate
|
||||
/// upstream — evicting events for the healthy subscribers or slowing the publisher —
|
||||
/// or is it confined to its own <c>Buffer(DropHead)</c> stage and its own bounded
|
||||
/// <c>DropOldest</c> channel? The design intends the latter; this measures it.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class SlowSubscriberScenario
|
||||
{
|
||||
/// <summary>Per-event reader delay applied to the stalled subscriber.</summary>
|
||||
public const int SlowReaderDelayMicroseconds = 50_000;
|
||||
|
||||
/// <summary>
|
||||
/// Publish rate for the probe. Deliberately paced rather than a tight burst: the
|
||||
/// publish source is a single <c>Source.ActorRef(StreamBufferSize, DropHead)</c>
|
||||
/// SHARED by every attribute subscriber, so an unpaced burst saturates that shared
|
||||
/// stage and every subscriber loses events for a reason that has nothing to do
|
||||
/// with the slow one. Pacing below the shared stage's capacity is what isolates
|
||||
/// the variable under test.
|
||||
/// </summary>
|
||||
public const int PublishRatePerSecond = 2_000;
|
||||
|
||||
/// <summary>Runs the isolation probe on a dedicated site.</summary>
|
||||
/// <param name="site">Site whose stream manager is used.</param>
|
||||
/// <param name="subscriberCount">Total subscribers to attach (one of them is stalled).</param>
|
||||
/// <param name="eventCount">Events to publish.</param>
|
||||
/// <param name="cancellationToken">Cancels the measurement.</param>
|
||||
/// <returns>The measured result.</returns>
|
||||
public static async Task<SlowSubscriberResult> RunAsync(
|
||||
SiteRuntimeFixture site,
|
||||
int subscriberCount,
|
||||
int eventCount,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var instanceName = site.InstanceName(0);
|
||||
var probes = new List<StreamSubscriberProbe>(subscriberCount);
|
||||
|
||||
try
|
||||
{
|
||||
for (var i = 0; i < subscriberCount; i++)
|
||||
{
|
||||
var probe = StreamSubscriberProbe.Attach(
|
||||
site.System, site.StreamManager, instanceName,
|
||||
$"{site.SiteId}-slowprobe-{i:D2}", latency: null);
|
||||
|
||||
// Probe 0 is the pathological one.
|
||||
if (i == 0)
|
||||
probe.ReaderDelayMicroseconds = SlowReaderDelayMicroseconds;
|
||||
|
||||
probes.Add(probe);
|
||||
}
|
||||
|
||||
// Let every subscription's stream graph finish materializing before the burst.
|
||||
await Task.Delay(TimeSpan.FromSeconds(2), cancellationToken);
|
||||
|
||||
var watch = Stopwatch.StartNew();
|
||||
const int sliceMilliseconds = 50;
|
||||
var perSlice = Math.Max(1, PublishRatePerSecond * sliceMilliseconds / 1000);
|
||||
var published = 0;
|
||||
while (published < eventCount && !cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var sliceStart = Stopwatch.GetTimestamp();
|
||||
var end = Math.Min(published + perSlice, eventCount);
|
||||
for (var i = published; i < end; i++)
|
||||
{
|
||||
site.StreamManager.PublishAttributeValueChanged(new AttributeValueChanged(
|
||||
instanceName, "Tag000", "Tag000", i, "Good", DateTimeOffset.UtcNow));
|
||||
}
|
||||
|
||||
published = end;
|
||||
|
||||
var elapsedMs = (Stopwatch.GetTimestamp() - sliceStart) * 1000.0 / Stopwatch.Frequency;
|
||||
if (elapsedMs < sliceMilliseconds)
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(sliceMilliseconds - elapsedMs), cancellationToken);
|
||||
}
|
||||
|
||||
watch.Stop();
|
||||
|
||||
// Give the healthy readers time to finish; the stalled one will not.
|
||||
await Task.Delay(TimeSpan.FromSeconds(20), cancellationToken);
|
||||
|
||||
var outcomes = probes
|
||||
.Select((p, i) => new SubscriberOutcome(
|
||||
p.Name,
|
||||
IsSlow: i == 0,
|
||||
p.ReceivedEvents,
|
||||
p.DroppedEvents,
|
||||
p.ReceivedEvents / (double)eventCount))
|
||||
.ToList();
|
||||
|
||||
var healthy = outcomes.Where(o => !o.IsSlow).ToList();
|
||||
|
||||
return new SlowSubscriberResult(
|
||||
PublishedEvents: eventCount,
|
||||
PublishSeconds: watch.Elapsed.TotalSeconds,
|
||||
PublishPerSecond: eventCount / Math.Max(0.001, watch.Elapsed.TotalSeconds),
|
||||
Outcomes: outcomes,
|
||||
HealthyMinDeliveryRatio: healthy.Count == 0 ? 0 : healthy.Min(o => o.DeliveryRatio),
|
||||
SlowDeliveryRatio: outcomes[0].DeliveryRatio);
|
||||
}
|
||||
finally
|
||||
{
|
||||
foreach (var probe in probes)
|
||||
await probe.DisposeAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
using System.Diagnostics;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Scenarios;
|
||||
|
||||
/// <summary>Result of one store-and-forward drain measurement.</summary>
|
||||
/// <param name="SiteId">Site the buffer belonged to.</param>
|
||||
/// <param name="MessageCount">Messages buffered before the drain began.</param>
|
||||
/// <param name="EnqueueSeconds">Wall time to buffer them (concurrent, many origin instances).</param>
|
||||
/// <param name="EnqueuePerSecond">Buffering throughput.</param>
|
||||
/// <param name="TimeToFirstDeliverySeconds">
|
||||
/// Wall time from the first sweep to the first successful delivery. With
|
||||
/// <c>attemptImmediateDelivery: false</c> the engine stamps <c>LastAttemptAt</c>, so the
|
||||
/// row is not due until one <c>DefaultRetryInterval</c> (30 s) has passed — this is the
|
||||
/// configured retry latency, not drain slowness, and is reported separately for that reason.
|
||||
/// </param>
|
||||
/// <param name="DrainSeconds">Wall time from the first sweep to an empty buffer (includes the retry wait).</param>
|
||||
/// <param name="DrainPerSecond">
|
||||
/// Drain throughput measured from the FIRST delivery to an empty buffer — the engine's
|
||||
/// actual capacity, and the headline number for register row 50.
|
||||
/// </param>
|
||||
/// <param name="ResidualDepth">Buffer depth left when the measurement stopped (0 = fully drained).</param>
|
||||
/// <param name="Progress">Delivered-count samples during the drain, so a steady rate can be told from a stall-then-burst.</param>
|
||||
public sealed record StoreAndForwardDrainResult(
|
||||
string SiteId,
|
||||
int MessageCount,
|
||||
double EnqueueSeconds,
|
||||
double EnqueuePerSecond,
|
||||
double TimeToFirstDeliverySeconds,
|
||||
double DrainSeconds,
|
||||
double DrainPerSecond,
|
||||
int ResidualDepth,
|
||||
IReadOnlyList<DrainProgressSample> Progress);
|
||||
|
||||
/// <summary>One observation during the drain.</summary>
|
||||
/// <param name="ElapsedSeconds">Seconds since the drain began.</param>
|
||||
/// <param name="Delivered">Cumulative successful deliveries.</param>
|
||||
/// <param name="Depth">Remaining buffer depth.</param>
|
||||
public sealed record DrainProgressSample(double ElapsedSeconds, long Delivered, int Depth);
|
||||
|
||||
/// <summary>
|
||||
/// Measures store-and-forward buffering and drain throughput (deferred-work register
|
||||
/// row 50, first half) using the real <c>StoreAndForwardService</c>, the real
|
||||
/// <c>StoreAndForwardStorage</c> and the real SQLite file — only the delivery target
|
||||
/// is a counting stub, because what is being measured is the site-local buffer's
|
||||
/// throughput, not a remote endpoint's.
|
||||
///
|
||||
/// <para>
|
||||
/// Phase 1 buffers <paramref name="messageCount"/> messages with
|
||||
/// <c>attemptImmediateDelivery: false</c>, spread across many origin instance names
|
||||
/// and issued from many concurrent tasks — the "concurrent buffering from multiple
|
||||
/// instances" WP-4 asks about (<c>[xc-7]</c>). Phase 2 registers a delivery handler
|
||||
/// that always succeeds and drives sweeps to completion, timing the drain.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The sweep is driven explicitly rather than waiting on the 10 s
|
||||
/// <c>RetryTimerInterval</c> so the number reported is the engine's drain capacity,
|
||||
/// not its polling cadence. The per-sweep batch is <c>SweepBatchLimit</c> (500) with
|
||||
/// <c>SweepTargetParallelism</c> (4) lanes, both at their production defaults.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class StoreAndForwardDrainScenario
|
||||
{
|
||||
/// <summary>Runs the drain measurement against one site's real S&F engine.</summary>
|
||||
/// <param name="site">The site whose store-and-forward engine is exercised.</param>
|
||||
/// <param name="messageCount">Messages to buffer.</param>
|
||||
/// <param name="concurrency">Concurrent enqueue tasks (distinct origin instances).</param>
|
||||
/// <param name="cancellationToken">Cancels the measurement.</param>
|
||||
/// <returns>The measured result.</returns>
|
||||
public static async Task<StoreAndForwardDrainResult> RunAsync(
|
||||
SiteRuntimeFixture site,
|
||||
int messageCount,
|
||||
int concurrency,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var service = site.StoreAndForward;
|
||||
var payload = $"{{\"site\":\"{site.SiteId}\",\"body\":\"{new string('x', 256)}\"}}";
|
||||
|
||||
// Phase 1 — concurrent buffering from many instances, no delivery attempted.
|
||||
var enqueueWatch = Stopwatch.StartNew();
|
||||
var perTask = messageCount / concurrency;
|
||||
var enqueueTasks = new List<Task>(concurrency);
|
||||
for (var t = 0; t < concurrency; t++)
|
||||
{
|
||||
var taskIndex = t;
|
||||
enqueueTasks.Add(Task.Run(async () =>
|
||||
{
|
||||
for (var i = 0; i < perTask; i++)
|
||||
{
|
||||
await service.EnqueueAsync(
|
||||
StoreAndForwardCategory.ExternalSystem,
|
||||
target: $"load-target-{taskIndex % 4}",
|
||||
payloadJson: payload,
|
||||
originInstanceName: site.InstanceName(taskIndex),
|
||||
attemptImmediateDelivery: false);
|
||||
}
|
||||
}, cancellationToken));
|
||||
}
|
||||
|
||||
await Task.WhenAll(enqueueTasks);
|
||||
enqueueWatch.Stop();
|
||||
var buffered = perTask * concurrency;
|
||||
|
||||
// Phase 2 — a delivery target that always succeeds; time the drain to empty.
|
||||
var delivered = 0L;
|
||||
service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, _ =>
|
||||
{
|
||||
Interlocked.Increment(ref delivered);
|
||||
return Task.FromResult(true);
|
||||
});
|
||||
|
||||
// Sweeps are driven explicitly rather than waiting on the 10 s RetryTimerInterval:
|
||||
// the number wanted is the engine's drain CAPACITY, not its polling cadence. One
|
||||
// sweep moves at most SweepBatchLimit (500) messages, so a large backlog needs
|
||||
// many, and the progress series below is what distinguishes a genuinely slow
|
||||
// drain from an artefact of this polling loop.
|
||||
var drainWatch = Stopwatch.StartNew();
|
||||
var deadline = DateTimeOffset.UtcNow.AddMinutes(10);
|
||||
var progress = new List<DrainProgressSample>();
|
||||
int depth;
|
||||
while (true)
|
||||
{
|
||||
service.TriggerSweep();
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(100), cancellationToken);
|
||||
|
||||
var depths = await service.GetBufferDepthAsync();
|
||||
depth = depths.Values.Sum();
|
||||
progress.Add(new DrainProgressSample(
|
||||
drainWatch.Elapsed.TotalSeconds, Interlocked.Read(ref delivered), depth));
|
||||
|
||||
if (depth == 0 || DateTimeOffset.UtcNow > deadline)
|
||||
break;
|
||||
}
|
||||
|
||||
drainWatch.Stop();
|
||||
|
||||
// Split the retry wait from the drain: the first sample with a non-zero delivered
|
||||
// count marks the moment the backlog actually became due.
|
||||
var firstDelivery = progress.FirstOrDefault(s => s.Delivered > 0);
|
||||
var timeToFirstDelivery = firstDelivery?.ElapsedSeconds ?? drainWatch.Elapsed.TotalSeconds;
|
||||
var activeDrainSeconds = Math.Max(0.001, drainWatch.Elapsed.TotalSeconds - timeToFirstDelivery);
|
||||
|
||||
return new StoreAndForwardDrainResult(
|
||||
SiteId: site.SiteId,
|
||||
MessageCount: buffered,
|
||||
EnqueueSeconds: enqueueWatch.Elapsed.TotalSeconds,
|
||||
EnqueuePerSecond: buffered / Math.Max(0.001, enqueueWatch.Elapsed.TotalSeconds),
|
||||
TimeToFirstDeliverySeconds: timeToFirstDelivery,
|
||||
DrainSeconds: drainWatch.Elapsed.TotalSeconds,
|
||||
DrainPerSecond: Interlocked.Read(ref delivered) / activeDrainSeconds,
|
||||
ResidualDepth: depth,
|
||||
Progress: progress);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness;
|
||||
|
||||
/// <summary>
|
||||
/// In-process stand-in for an OPC UA / MxGateway server, registered on the REAL
|
||||
/// <c>DataConnectionFactory</c> under the <see cref="ProtocolName"/> protocol so the
|
||||
/// whole Data Connection Layer above it — <c>DataConnectionManagerActor</c>,
|
||||
/// <c>DataConnectionActor</c>, its <c>_instancesByTag</c> fan-out and the
|
||||
/// <c>TagValueUpdate</c> hand-off to Instance Actors — runs unmodified.
|
||||
///
|
||||
/// <para>
|
||||
/// It implements <see cref="IBatchSubscribableConnection"/> because the batch path is
|
||||
/// the one a real site takes at this scale; the per-tag fallback would make
|
||||
/// subscribe setup, not steady-state throughput, the thing being measured.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why faking here does not invalidate the measurement.</b> Everything this class
|
||||
/// replaces is on the far side of the process boundary: socket I/O, the OPC UA SDK's
|
||||
/// own session/subscription machinery, and the device. The system under test — the
|
||||
/// actor hierarchy, the site stream, store-and-forward, health, and the streaming
|
||||
/// relay — begins at the <see cref="SubscriptionCallback"/> invocation, which is
|
||||
/// exactly where the real adapter hands off. Driving 375,000 genuine monitored items
|
||||
/// would measure the OPC UA stack, not ScadaBridge.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class SimulatedDataConnection : IDataConnection, IBatchSubscribableConnection
|
||||
{
|
||||
/// <summary>Protocol discriminator this adapter registers under on the factory.</summary>
|
||||
public const string ProtocolName = "LoadSim";
|
||||
|
||||
private SubscriptionCallback? _callback;
|
||||
private int _subscriptionCounter;
|
||||
|
||||
/// <summary>
|
||||
/// The data connection name this adapter was created for, taken from the connection
|
||||
/// details. The DCL factory creates adapters as its manager actor processes the
|
||||
/// CreateConnectionCommands, so creation order is not connection order — the driver
|
||||
/// resolves an adapter by name rather than by index.
|
||||
/// </summary>
|
||||
public string ConnectionName { get; private set; } = string.Empty;
|
||||
|
||||
/// <summary>Key under which the connection name travels in the connection details.</summary>
|
||||
public const string ConnectionNameKey = "connectionName";
|
||||
|
||||
/// <inheritdoc />
|
||||
public ConnectionHealth Status { get; private set; } = ConnectionHealth.Disconnected;
|
||||
|
||||
/// <inheritdoc />
|
||||
public event Action? Disconnected;
|
||||
|
||||
/// <summary>
|
||||
/// The callback captured at subscribe time. The tag driver invokes this to inject
|
||||
/// a value change, mirroring what the OPC UA SDK's notification thread does.
|
||||
/// Null until the site's Instance Actors have subscribed.
|
||||
/// </summary>
|
||||
public SubscriptionCallback? ValueCallback => _callback;
|
||||
|
||||
/// <summary>Number of tag paths this connection has accepted subscriptions for.</summary>
|
||||
public int SubscribedTagCount => _subscriptionCounter;
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task ConnectAsync(IDictionary<string, string> connectionDetails, CancellationToken cancellationToken = default)
|
||||
{
|
||||
if (connectionDetails.TryGetValue(ConnectionNameKey, out var name))
|
||||
ConnectionName = name;
|
||||
|
||||
Status = ConnectionHealth.Connected;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task DisconnectAsync(CancellationToken cancellationToken = default)
|
||||
{
|
||||
Status = ConnectionHealth.Disconnected;
|
||||
Disconnected?.Invoke();
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<string> SubscribeAsync(string tagPath, SubscriptionCallback callback, CancellationToken cancellationToken = default)
|
||||
{
|
||||
_callback = callback;
|
||||
return Task.FromResult($"sub-{Interlocked.Increment(ref _subscriptionCounter)}");
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<TagSubscribeResult>> SubscribeBatchAsync(
|
||||
IReadOnlyList<string> tagPaths,
|
||||
SubscriptionCallback callback,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
_callback = callback;
|
||||
var results = new List<TagSubscribeResult>(tagPaths.Count);
|
||||
foreach (var path in tagPaths)
|
||||
{
|
||||
results.Add(new TagSubscribeResult(
|
||||
path, true, $"sub-{Interlocked.Increment(ref _subscriptionCounter)}", null));
|
||||
}
|
||||
|
||||
return Task.FromResult<IReadOnlyList<TagSubscribeResult>>(results);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task UnsubscribeAsync(string subscriptionId, CancellationToken cancellationToken = default)
|
||||
=> Task.CompletedTask;
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task UnsubscribeBatchAsync(IReadOnlyList<string> subscriptionIds, CancellationToken cancellationToken = default)
|
||||
=> Task.CompletedTask;
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<ReadResult> ReadAsync(string tagPath, CancellationToken cancellationToken = default)
|
||||
=> Task.FromResult(new ReadResult(true, new TagValue(0d, QualityCode.Good, DateTimeOffset.UtcNow), null));
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyDictionary<string, ReadResult>> ReadBatchAsync(
|
||||
IEnumerable<string> tagPaths, CancellationToken cancellationToken = default)
|
||||
{
|
||||
var now = DateTimeOffset.UtcNow;
|
||||
var results = new Dictionary<string, ReadResult>();
|
||||
foreach (var path in tagPaths)
|
||||
results[path] = new ReadResult(true, new TagValue(0d, QualityCode.Good, now), null);
|
||||
|
||||
return Task.FromResult<IReadOnlyDictionary<string, ReadResult>>(results);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<WriteResult> WriteAsync(string tagPath, object? value, CancellationToken cancellationToken = default)
|
||||
=> Task.FromResult(new WriteResult(true, null));
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyDictionary<string, WriteResult>> WriteBatchAsync(
|
||||
IDictionary<string, object?> values, CancellationToken cancellationToken = default)
|
||||
{
|
||||
var results = new Dictionary<string, WriteResult>();
|
||||
foreach (var key in values.Keys)
|
||||
results[key] = new WriteResult(true, null);
|
||||
|
||||
return Task.FromResult<IReadOnlyDictionary<string, WriteResult>>(results);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<bool> WriteBatchAndWaitAsync(
|
||||
IDictionary<string, object?> values, string flagPath, object? flagValue,
|
||||
string responsePath, object? responseValue, TimeSpan timeout,
|
||||
CancellationToken cancellationToken = default)
|
||||
=> Task.FromResult(true);
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
|
||||
}
|
||||
@@ -0,0 +1,371 @@
|
||||
using System.Diagnostics;
|
||||
using System.Text.Json;
|
||||
using Akka.Actor;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
|
||||
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer;
|
||||
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness.Probes;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness;
|
||||
|
||||
/// <summary>
|
||||
/// One simulated site: its own <see cref="ActorSystem"/>, its own LocalDb SQLite
|
||||
/// file, a real Data Connection Layer over <see cref="SimulatedDataConnection"/>
|
||||
/// adapters, real Instance Actors, a real <see cref="SiteStreamManager"/>, real
|
||||
/// store-and-forward, and a real <see cref="SiteHealthCollector"/>.
|
||||
///
|
||||
/// <para>
|
||||
/// The sites are separate, non-clustered ActorSystems rather than 10 real two-node
|
||||
/// Akka clusters. Cluster membership, singleton placement and failover timing are
|
||||
/// already measured on a real two-node rig by
|
||||
/// <c>PerformanceTests/Failover/FailoverTimingTests.cs</c>; what WP-4 asks about is
|
||||
/// the load-bearing hierarchy under each singleton, which is what this builds.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class SiteRuntimeFixture : IAsyncDisposable
|
||||
{
|
||||
private readonly HarnessConfig _config;
|
||||
private readonly List<SimulatedDataConnection> _connections;
|
||||
private readonly ServiceProvider _localDbProvider;
|
||||
private readonly ILocalDb _localDb;
|
||||
private readonly List<IActorRef> _instanceActors = new();
|
||||
private readonly List<StreamSubscriberProbe> _probes = new();
|
||||
private readonly string _dataDirectory;
|
||||
|
||||
/// <summary>Site identifier, e.g. <c>site-01</c>.</summary>
|
||||
public string SiteId { get; }
|
||||
|
||||
/// <summary>This site's actor system.</summary>
|
||||
public ActorSystem System { get; }
|
||||
|
||||
/// <summary>The real site-wide broadcast stream.</summary>
|
||||
public SiteStreamManager StreamManager { get; }
|
||||
|
||||
/// <summary>The real site health collector feeding the 30 s report.</summary>
|
||||
public SiteHealthCollector HealthCollector { get; }
|
||||
|
||||
/// <summary>The real store-and-forward engine for this site.</summary>
|
||||
public StoreAndForwardService StoreAndForward { get; }
|
||||
|
||||
/// <summary>The real DCL manager actor.</summary>
|
||||
public IActorRef DataConnectionManager { get; }
|
||||
|
||||
/// <summary>The simulated adapters, one per data connection, in creation order.</summary>
|
||||
public IReadOnlyList<SimulatedDataConnection> Connections
|
||||
{
|
||||
get { lock (_connections) return _connections.ToList(); }
|
||||
}
|
||||
|
||||
/// <summary>The live stream subscribers attached to this site.</summary>
|
||||
public IReadOnlyList<StreamSubscriberProbe> Probes => _probes;
|
||||
|
||||
/// <summary>Instance Actors created on this site.</summary>
|
||||
public IReadOnlyList<IActorRef> InstanceActors => _instanceActors;
|
||||
|
||||
/// <summary>Tag paths per connection index, in the order they were assigned.</summary>
|
||||
public IReadOnlyList<List<string>> TagPathsByConnection { get; }
|
||||
|
||||
/// <summary>Wall-clock time the instance ramp took, measured by <see cref="StartInstancesAsync"/>.</summary>
|
||||
public TimeSpan InstanceRampDuration { get; private set; }
|
||||
|
||||
private SiteRuntimeFixture(
|
||||
string siteId,
|
||||
HarnessConfig config,
|
||||
string dataDirectory,
|
||||
ServiceProvider localDbProvider,
|
||||
ILocalDb localDb,
|
||||
ActorSystem system,
|
||||
SiteStreamManager streamManager,
|
||||
SiteHealthCollector healthCollector,
|
||||
StoreAndForwardService storeAndForward,
|
||||
IActorRef dataConnectionManager,
|
||||
SiteStorageService storage,
|
||||
ScriptCompilationService compilationService,
|
||||
SharedScriptLibrary sharedScriptLibrary,
|
||||
SiteRuntimeOptions siteOptions,
|
||||
List<List<string>> tagPathsByConnection,
|
||||
List<SimulatedDataConnection> connections)
|
||||
{
|
||||
_connections = connections;
|
||||
SiteId = siteId;
|
||||
_config = config;
|
||||
_dataDirectory = dataDirectory;
|
||||
_localDbProvider = localDbProvider;
|
||||
_localDb = localDb;
|
||||
System = system;
|
||||
StreamManager = streamManager;
|
||||
HealthCollector = healthCollector;
|
||||
StoreAndForward = storeAndForward;
|
||||
DataConnectionManager = dataConnectionManager;
|
||||
Storage = storage;
|
||||
CompilationService = compilationService;
|
||||
SharedScriptLibrary = sharedScriptLibrary;
|
||||
SiteOptions = siteOptions;
|
||||
TagPathsByConnection = tagPathsByConnection;
|
||||
}
|
||||
|
||||
private SiteStorageService Storage { get; }
|
||||
private ScriptCompilationService CompilationService { get; }
|
||||
private SharedScriptLibrary SharedScriptLibrary { get; }
|
||||
private SiteRuntimeOptions SiteOptions { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Builds a site: LocalDb file, actor system, DCL with its simulated connections,
|
||||
/// stream manager, health collector and store-and-forward engine. Instance Actors
|
||||
/// are created separately by <see cref="StartInstancesAsync"/> so the deployment
|
||||
/// ramp can be timed on its own.
|
||||
/// </summary>
|
||||
/// <param name="siteIndex">Zero-based site index.</param>
|
||||
/// <param name="config">Harness configuration.</param>
|
||||
/// <param name="rootDataDirectory">Directory under which this site's SQLite files live.</param>
|
||||
/// <param name="connectionsPerSite">Number of data connections to spread the site's tags across.</param>
|
||||
/// <returns>The started fixture.</returns>
|
||||
public static async Task<SiteRuntimeFixture> CreateAsync(
|
||||
int siteIndex, HarnessConfig config, string rootDataDirectory, int connectionsPerSite)
|
||||
{
|
||||
var siteId = $"site-{siteIndex + 1:D2}";
|
||||
var dataDirectory = Path.Combine(rootDataDirectory, siteId);
|
||||
Directory.CreateDirectory(dataDirectory);
|
||||
|
||||
var configuration = new ConfigurationBuilder()
|
||||
.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["LocalDb:Path"] = Path.Combine(dataDirectory, "site-localdb.db"),
|
||||
})
|
||||
.Build();
|
||||
|
||||
var localDbProvider = new ServiceCollection()
|
||||
.AddZbLocalDb(configuration)
|
||||
.BuildServiceProvider();
|
||||
var localDb = localDbProvider.GetRequiredService<ILocalDb>();
|
||||
|
||||
var storage = new SiteStorageService(localDb, NullLogger<SiteStorageService>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
|
||||
var compilationService = new ScriptCompilationService(NullLogger<ScriptCompilationService>.Instance);
|
||||
var sharedScriptLibrary = new SharedScriptLibrary(compilationService, NullLogger<SharedScriptLibrary>.Instance);
|
||||
|
||||
// Production defaults throughout — the point of the run is to measure the
|
||||
// shipped configuration, so nothing here is tuned for the harness.
|
||||
var siteOptions = new SiteRuntimeOptions();
|
||||
|
||||
// WARNING-level logging. At 37,500 updates/s per site, Akka's INFO output would
|
||||
// itself become a measured load; and the InstanceActorInitialized dead letters are
|
||||
// a harness artifact (see StartInstancesAsync) rather than a real condition.
|
||||
var system = ActorSystem.Create($"loadharness-{siteId}", Akka.Configuration.ConfigurationFactory.ParseString(
|
||||
"akka.loglevel = WARNING\nakka.stdout-loglevel = WARNING\nakka.log-dead-letters = 0\nakka.log-dead-letters-during-shutdown = off"));
|
||||
|
||||
var streamManager = new SiteStreamManager(siteOptions, NullLogger<SiteStreamManager>.Instance);
|
||||
streamManager.Initialize(system);
|
||||
|
||||
var healthCollector = new SiteHealthCollector();
|
||||
healthCollector.SetActiveNode(true);
|
||||
healthCollector.SetNodeHostname($"{siteId}-node-a");
|
||||
|
||||
var sfStorage = new StoreAndForwardStorage(localDb, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
var storeAndForward = new StoreAndForwardService(
|
||||
sfStorage,
|
||||
new StoreAndForwardOptions(),
|
||||
NullLogger<StoreAndForwardService>.Instance,
|
||||
siteId: siteId);
|
||||
await storeAndForward.StartAsync();
|
||||
|
||||
// Real DCL, with the simulated adapter registered on the real factory via its
|
||||
// documented RegisterAdapter extension point.
|
||||
var loggerFactory = NullLoggerFactory.Instance;
|
||||
var factory = new DataConnectionFactory(loggerFactory);
|
||||
var connections = new List<SimulatedDataConnection>();
|
||||
factory.RegisterAdapter(SimulatedDataConnection.ProtocolName, _ =>
|
||||
{
|
||||
var connection = new SimulatedDataConnection();
|
||||
lock (connections) connections.Add(connection);
|
||||
return connection;
|
||||
});
|
||||
|
||||
var dclManager = system.ActorOf(
|
||||
Props.Create(() => new DataConnectionManagerActor(
|
||||
factory, new DataConnectionOptions(), healthCollector, null, null)),
|
||||
"data-connection-manager");
|
||||
|
||||
for (var c = 0; c < connectionsPerSite; c++)
|
||||
{
|
||||
dclManager.Tell(new CreateConnectionCommand(
|
||||
ConnectionName: ConnectionName(c),
|
||||
ProtocolType: SimulatedDataConnection.ProtocolName,
|
||||
PrimaryConnectionDetails: new Dictionary<string, string>
|
||||
{
|
||||
["endpoint"] = $"sim://{siteId}/{c}",
|
||||
[SimulatedDataConnection.ConnectionNameKey] = ConnectionName(c),
|
||||
}));
|
||||
}
|
||||
|
||||
var tagPathsByConnection = new List<List<string>>();
|
||||
for (var c = 0; c < connectionsPerSite; c++)
|
||||
tagPathsByConnection.Add(new List<string>());
|
||||
|
||||
var fixture = new SiteRuntimeFixture(
|
||||
siteId, config, dataDirectory, localDbProvider, localDb, system, streamManager,
|
||||
healthCollector, storeAndForward, dclManager, storage, compilationService,
|
||||
sharedScriptLibrary, siteOptions, tagPathsByConnection, connections);
|
||||
|
||||
return fixture;
|
||||
}
|
||||
|
||||
/// <summary>Deterministic connection name for a connection index.</summary>
|
||||
/// <param name="connectionIndex">Zero-based connection index.</param>
|
||||
/// <returns>The connection name used in configs and DCL commands.</returns>
|
||||
public static string ConnectionName(int connectionIndex) => $"sim-conn-{connectionIndex:D2}";
|
||||
|
||||
/// <summary>
|
||||
/// Creates this site's Instance Actors in production-shaped staggered batches
|
||||
/// (<see cref="SiteRuntimeOptions.StartupBatchSize"/> /
|
||||
/// <see cref="SiteRuntimeOptions.StartupBatchDelayMs"/>) and records how long the
|
||||
/// ramp took. This is the harness's stand-in for "deployment of 500 instances to
|
||||
/// a site" — it exercises the same per-instance construction, config
|
||||
/// deserialization, override load and DCL subscribe that a real deploy triggers.
|
||||
/// </summary>
|
||||
/// <param name="cancellationToken">Cancels the ramp.</param>
|
||||
/// <returns>A task that completes when every instance actor exists.</returns>
|
||||
public async Task StartInstancesAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var started = Stopwatch.StartNew();
|
||||
var connectionCount = TagPathsByConnection.Count;
|
||||
|
||||
for (var i = 0; i < _config.InstancesPerSite; i++)
|
||||
{
|
||||
var instanceName = InstanceName(i);
|
||||
var connectionIndex = i % connectionCount;
|
||||
var connectionName = ConnectionName(connectionIndex);
|
||||
|
||||
var attributes = new List<ResolvedAttribute>(_config.TagsPerInstance);
|
||||
for (var t = 0; t < _config.TagsPerInstance; t++)
|
||||
{
|
||||
var tagPath = $"{instanceName}.Tag{t:D3}";
|
||||
TagPathsByConnection[connectionIndex].Add(tagPath);
|
||||
attributes.Add(new ResolvedAttribute
|
||||
{
|
||||
CanonicalName = $"Tag{t:D3}",
|
||||
DataType = "Double",
|
||||
DataSourceReference = tagPath,
|
||||
BoundDataConnectionId = connectionIndex + 1,
|
||||
BoundDataConnectionName = connectionName,
|
||||
BoundDataConnectionProtocol = SimulatedDataConnection.ProtocolName,
|
||||
});
|
||||
}
|
||||
|
||||
var configuration = new FlattenedConfiguration
|
||||
{
|
||||
InstanceUniqueName = instanceName,
|
||||
TemplateId = 1,
|
||||
SiteId = 1,
|
||||
Attributes = attributes,
|
||||
Connections = new Dictionary<string, ConnectionConfig>
|
||||
{
|
||||
[connectionName] = new()
|
||||
{
|
||||
Protocol = SimulatedDataConnection.ProtocolName,
|
||||
ConfigurationJson = "{}",
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
var configJson = JsonSerializer.Serialize(configuration);
|
||||
var actor = System.ActorOf(
|
||||
Props.Create(() => new InstanceActor(
|
||||
instanceName, configJson, Storage, CompilationService, SharedScriptLibrary,
|
||||
StreamManager, SiteOptions, NullLogger<InstanceActor>.Instance,
|
||||
DataConnectionManager, HealthCollector, null, null)),
|
||||
instanceName);
|
||||
_instanceActors.Add(actor);
|
||||
|
||||
// Production staggered-startup pacing (SiteRuntimeOptions defaults):
|
||||
// batches of StartupBatchSize separated by StartupBatchDelayMs, which is
|
||||
// exactly what DeploymentManagerActor does on a real site start.
|
||||
if ((i + 1) % SiteOptions.StartupBatchSize == 0)
|
||||
await Task.Delay(SiteOptions.StartupBatchDelayMs, cancellationToken);
|
||||
}
|
||||
|
||||
started.Stop();
|
||||
InstanceRampDuration = started.Elapsed;
|
||||
|
||||
HealthCollector.SetInstanceCounts(
|
||||
_config.InstancesPerSite, _config.InstancesPerSite, 0);
|
||||
for (var c = 0; c < connectionCount; c++)
|
||||
{
|
||||
HealthCollector.UpdateTagResolution(
|
||||
ConnectionName(c), TagPathsByConnection[c].Count, TagPathsByConnection[c].Count);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Deterministic instance unique name for an instance index.</summary>
|
||||
/// <param name="instanceIndex">Zero-based instance index.</param>
|
||||
/// <returns>The instance unique name.</returns>
|
||||
public string InstanceName(int instanceIndex) => $"{SiteId}-inst-{instanceIndex:D4}";
|
||||
|
||||
/// <summary>
|
||||
/// Attaches <see cref="HarnessConfig.StreamProbesPerSite"/> live stream subscribers,
|
||||
/// each built from the production pieces the gRPC server uses: a real
|
||||
/// <see cref="StreamRelayActor"/> writing into a bounded <c>DropOldest</c> channel
|
||||
/// of the production capacity, subscribed through the real
|
||||
/// <see cref="SiteStreamManager.Subscribe"/>. Only the socket writer is replaced —
|
||||
/// by a reader task under the harness's control, which is what makes the
|
||||
/// slow-subscriber scenario possible at all.
|
||||
///
|
||||
/// <para>
|
||||
/// Attaching at least one subscriber is also load-bearing:
|
||||
/// <c>PublishAttributeValueChanged</c> short-circuits at zero subscribers, so an
|
||||
/// unsubscribed site would publish nothing and measure nothing.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="latency">Histogram that receives end-to-end tag update latencies.</param>
|
||||
public void AttachStreamProbes(LatencyHistogram latency)
|
||||
{
|
||||
var stride = Math.Max(1, _config.InstancesPerSite / Math.Max(1, _config.StreamProbesPerSite));
|
||||
for (var p = 0; p < _config.StreamProbesPerSite; p++)
|
||||
{
|
||||
var instanceIndex = Math.Min(p * stride, _config.InstancesPerSite - 1);
|
||||
var instanceName = InstanceName(instanceIndex);
|
||||
var probe = StreamSubscriberProbe.Attach(
|
||||
System, StreamManager, instanceName, $"{SiteId}-probe-{p:D2}", latency);
|
||||
_probes.Add(probe);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Total tag paths registered across this site's connections.</summary>
|
||||
public int TotalTagPaths => TagPathsByConnection.Sum(list => list.Count);
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
foreach (var probe in _probes)
|
||||
await probe.DisposeAsync();
|
||||
|
||||
await StoreAndForward.StopAsync();
|
||||
await System.Terminate();
|
||||
await _localDbProvider.DisposeAsync();
|
||||
|
||||
try
|
||||
{
|
||||
if (Directory.Exists(_dataDirectory))
|
||||
Directory.Delete(_dataDirectory, recursive: true);
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
// Best-effort cleanup of a temp directory; a lingering WAL handle is not
|
||||
// a harness failure and must not mask the measured result.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,175 @@
|
||||
using System.Diagnostics;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness;
|
||||
|
||||
/// <summary>
|
||||
/// Drives simulated tag value changes into a site's connections at a fixed nominal
|
||||
/// rate, standing in for an OPC UA server's notification thread.
|
||||
///
|
||||
/// <para>
|
||||
/// One driver task per data connection, because that mirrors production: each
|
||||
/// <c>DataConnectionActor</c> has exactly one adapter feeding it, and every update
|
||||
/// for that connection funnels through that actor's single mailbox. Sharding the
|
||||
/// emitters differently would hide the per-connection-actor serialization point,
|
||||
/// which is one of the things WP-4 needs to characterize.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The emitter walks the connection's tag list in slices sized so that one full pass
|
||||
/// takes <see cref="HarnessConfig.TagUpdatePeriod"/>, then sleeps out the remainder
|
||||
/// of each slice's budget. If a slice overruns its budget the driver does NOT try to
|
||||
/// catch up — it records the shortfall in <see cref="EmitLagSeconds"/> so the results
|
||||
/// can say honestly whether the offered load was actually delivered.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class TagUpdateDriver : IAsyncDisposable
|
||||
{
|
||||
private const int SlicesPerPeriod = 20;
|
||||
|
||||
private readonly List<Task> _tasks = new();
|
||||
private readonly CancellationTokenSource _cts = new();
|
||||
private long _emitted;
|
||||
private long _skippedNoCallback;
|
||||
private long _lagTicks;
|
||||
|
||||
/// <summary>Total tag value changes handed to adapter callbacks.</summary>
|
||||
public long EmittedCount => Interlocked.Read(ref _emitted);
|
||||
|
||||
/// <summary>
|
||||
/// Emissions skipped because no Instance Actor had subscribed to that connection
|
||||
/// yet (the adapter callback is captured at subscribe time). Non-zero only during
|
||||
/// the ramp; a non-zero value in the steady window would mean lost offered load.
|
||||
/// </summary>
|
||||
public long SkippedNoCallback => Interlocked.Read(ref _skippedNoCallback);
|
||||
|
||||
/// <summary>
|
||||
/// Cumulative seconds by which emit slices overran their time budget, summed
|
||||
/// across driver tasks. Large values mean the harness itself could not offer the
|
||||
/// nominal rate and the measured throughput is driver-bound, not system-bound.
|
||||
/// </summary>
|
||||
public double EmitLagSeconds => Interlocked.Read(ref _lagTicks) / (double)Stopwatch.Frequency;
|
||||
|
||||
/// <summary>
|
||||
/// Starts one emitter task per connection across every site.
|
||||
/// </summary>
|
||||
/// <param name="sites">The sites to drive.</param>
|
||||
/// <param name="config">Harness configuration supplying the update period.</param>
|
||||
/// <returns>The running driver.</returns>
|
||||
public static TagUpdateDriver Start(IReadOnlyList<SiteRuntimeFixture> sites, HarnessConfig config)
|
||||
{
|
||||
var driver = new TagUpdateDriver();
|
||||
foreach (var site in sites)
|
||||
{
|
||||
for (var c = 0; c < site.TagPathsByConnection.Count; c++)
|
||||
{
|
||||
var connectionIndex = c;
|
||||
var tagPaths = site.TagPathsByConnection[connectionIndex].ToArray();
|
||||
driver._tasks.Add(Task.Run(() => driver.EmitLoopAsync(
|
||||
site, connectionIndex, tagPaths, config.TagUpdatePeriod, driver._cts.Token)));
|
||||
}
|
||||
}
|
||||
|
||||
return driver;
|
||||
}
|
||||
|
||||
private async Task EmitLoopAsync(
|
||||
SiteRuntimeFixture site,
|
||||
int connectionIndex,
|
||||
string[] tagPaths,
|
||||
TimeSpan period,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (tagPaths.Length == 0)
|
||||
return;
|
||||
|
||||
var sliceBudget = period / SlicesPerPeriod;
|
||||
var sliceSize = Math.Max(1, (int)Math.Ceiling(tagPaths.Length / (double)SlicesPerPeriod));
|
||||
var cursor = 0;
|
||||
var sequence = 0d;
|
||||
|
||||
while (!cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
var sliceStart = Stopwatch.GetTimestamp();
|
||||
var callback = ResolveCallback(site, connectionIndex);
|
||||
|
||||
if (callback == null)
|
||||
{
|
||||
Interlocked.Add(ref _skippedNoCallback, sliceSize);
|
||||
}
|
||||
else
|
||||
{
|
||||
var end = Math.Min(cursor + sliceSize, tagPaths.Length);
|
||||
for (var i = cursor; i < end; i++)
|
||||
{
|
||||
// Stamped HERE: this instant rides TagValueUpdate.Timestamp all the
|
||||
// way to the subscriber, so the probe's subtraction is a genuine
|
||||
// end-to-end latency and not a re-stamped approximation.
|
||||
callback(tagPaths[i], new TagValue(sequence, QualityCode.Good, DateTimeOffset.UtcNow));
|
||||
}
|
||||
|
||||
Interlocked.Add(ref _emitted, end - cursor);
|
||||
cursor = end;
|
||||
}
|
||||
|
||||
if (cursor >= tagPaths.Length)
|
||||
{
|
||||
cursor = 0;
|
||||
sequence += 1d;
|
||||
}
|
||||
|
||||
var elapsed = Stopwatch.GetTimestamp() - sliceStart;
|
||||
var budgetTicks = (long)(sliceBudget.TotalSeconds * Stopwatch.Frequency);
|
||||
if (elapsed < budgetTicks)
|
||||
{
|
||||
var remaining = TimeSpan.FromSeconds((budgetTicks - elapsed) / (double)Stopwatch.Frequency);
|
||||
try
|
||||
{
|
||||
await Task.Delay(remaining, cancellationToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
Interlocked.Add(ref _lagTicks, elapsed - budgetTicks);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the live adapter callback for a connection, by NAME. The DCL factory
|
||||
/// appends adapters as its manager actor processes CreateConnectionCommands, so
|
||||
/// list position does not track connection index.
|
||||
/// </summary>
|
||||
private static SubscriptionCallback? ResolveCallback(SiteRuntimeFixture site, int connectionIndex)
|
||||
{
|
||||
var name = SiteRuntimeFixture.ConnectionName(connectionIndex);
|
||||
// Last match wins: a reconnect would create a fresh adapter for the same name,
|
||||
// and only the newest one holds the live subscription callback.
|
||||
return site.Connections.LastOrDefault(c => c.ConnectionName == name)?.ValueCallback;
|
||||
}
|
||||
|
||||
private int _disposed;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _disposed, 1) != 0)
|
||||
return;
|
||||
|
||||
await _cts.CancelAsync();
|
||||
try
|
||||
{
|
||||
await Task.WhenAll(_tasks);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Expected on shutdown.
|
||||
}
|
||||
|
||||
_cts.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<!--
|
||||
Target-scale load harness (deferred-work register #25 / Phase-8 WP-4, absorbing row 50).
|
||||
|
||||
Deliberately an Exe, not an xunit project: the full-scale protocol is a
|
||||
~20-minute sustained run at 10 sites x 500 instances x 75 tags, which must NOT
|
||||
execute as part of `dotnet test ZB.MOM.WW.ScadaBridge.slnx`. The PerformanceTests
|
||||
project carries a tiny scaled-down smoke [Fact] that references this assembly so
|
||||
the harness stays compiled and exercised by CI without the runtime cost.
|
||||
|
||||
Design memo: docs/plans/2026-08-15-target-scale-load-test-design.md
|
||||
Results: docs/plans/2026-08-15-target-scale-load-test-results.md
|
||||
-->
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<OutputType>Exe</OutputType>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<IsPackable>false</IsPackable>
|
||||
<RootNamespace>ZB.MOM.WW.ScadaBridge.LoadHarness</RootNamespace>
|
||||
<!-- Server GC: the harness holds ~5,000 actors and 375,000 attribute slots; the
|
||||
workstation collector's single heap is not representative of a site node,
|
||||
which runs under the ASP.NET Core host defaults (server GC on). -->
|
||||
<ServerGarbageCollection>true</ServerGarbageCollection>
|
||||
<ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Commons/ZB.MOM.WW.ScadaBridge.Commons.csproj" />
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ZB.MOM.WW.ScadaBridge.SiteRuntime.csproj" />
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" />
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ZB.MOM.WW.ScadaBridge.StoreAndForward.csproj" />
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.AuditLog/ZB.MOM.WW.ScadaBridge.AuditLog.csproj" />
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Communication/ZB.MOM.WW.ScadaBridge.Communication.csproj" />
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.TargetScale;
|
||||
|
||||
/// <summary>
|
||||
/// Keeps the target-scale load harness (deferred-work register #25 / Phase-8 WP-4)
|
||||
/// honest at CI scale.
|
||||
///
|
||||
/// <para>
|
||||
/// The full protocol — 10 sites x 500 instances x 75 tags sustained for 20 minutes —
|
||||
/// deliberately lives in the standalone <c>ZB.MOM.WW.ScadaBridge.LoadHarness</c>
|
||||
/// executable, NOT here: perf tests in this project run as part of an ordinary
|
||||
/// <c>dotnet test ZB.MOM.WW.ScadaBridge.slnx</c> (the <c>Category=Performance</c>
|
||||
/// trait enables a filter, it does not exclude by default), and a 20-minute test
|
||||
/// would be intolerable there. What this test protects is that the harness still
|
||||
/// compiles, wires up, and produces coherent measurements — so #25's evidence can be
|
||||
/// regenerated on demand rather than bit-rotting.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Results doc: <c>docs/plans/2026-08-15-target-scale-load-test-results.md</c>.
|
||||
/// Design memo: <c>docs/plans/2026-08-15-target-scale-load-test-design.md</c>.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class TargetScaleHarnessSmokeTests
|
||||
{
|
||||
/// <summary>
|
||||
/// Runs the harness at ~1/1000th of target scale for a few seconds and asserts the
|
||||
/// pipeline is intact end to end: tag updates reach live stream subscribers, the
|
||||
/// central health aggregator tracks every site, debug snapshots answer, the
|
||||
/// store-and-forward buffer drains to empty, and a stalled subscriber does not cost
|
||||
/// the healthy ones any events.
|
||||
/// </summary>
|
||||
/// <returns>A task representing the test run.</returns>
|
||||
[Trait("Category", "Performance")]
|
||||
[Fact]
|
||||
public async Task Harness_AtSmokeScale_ProducesCoherentMeasurements()
|
||||
{
|
||||
var config = new HarnessConfig
|
||||
{
|
||||
Sites = 2,
|
||||
InstancesPerSite = 10,
|
||||
TagsPerInstance = 5,
|
||||
TagUpdatePeriod = TimeSpan.FromSeconds(1),
|
||||
SettleDuration = TimeSpan.FromSeconds(5),
|
||||
SustainDuration = TimeSpan.FromSeconds(15),
|
||||
SampleInterval = TimeSpan.FromSeconds(2),
|
||||
// Shortened from the production 30 s only because the smoke window is 20 s.
|
||||
HealthReportInterval = TimeSpan.FromSeconds(2),
|
||||
DebugProbeInterval = TimeSpan.FromSeconds(2),
|
||||
SubscribeSettleDuration = TimeSpan.FromSeconds(5),
|
||||
StreamProbesPerSite = 3,
|
||||
StoreAndForwardDrainMessages = 200,
|
||||
SlowSubscriberEvents = 2_000,
|
||||
ResultsPath = Path.Combine(Path.GetTempPath(), $"loadharness-smoke-{Guid.NewGuid():N}.json"),
|
||||
};
|
||||
|
||||
using var cancellation = new CancellationTokenSource(TimeSpan.FromMinutes(10));
|
||||
var result = await HarnessRun.ExecuteAsync(config, _ => { }, cancellation.Token);
|
||||
|
||||
// Tag updates flowed all the way through DCL -> InstanceActor -> site stream ->
|
||||
// StreamRelayActor -> bounded channel -> subscriber.
|
||||
Assert.True(result.TagUpdateLatency.Count > 0,
|
||||
"No tag update latency samples — the DCL -> stream -> subscriber path did not carry traffic.");
|
||||
Assert.True(result.StreamProbeReceived > 0, "Live stream subscribers received nothing.");
|
||||
Assert.Equal(0, result.DriverSkippedNoCallback);
|
||||
|
||||
// Observability held up.
|
||||
Assert.Equal(config.Sites, result.SitesTrackedByAggregator);
|
||||
Assert.True(result.DebugSnapshotsCompleted > 0, "No debug snapshot completed.");
|
||||
Assert.Equal(0, result.DebugSnapshotTimeouts);
|
||||
|
||||
// Store-and-forward drained completely (register row 50, first half).
|
||||
Assert.NotNull(result.StoreAndForwardDrain);
|
||||
Assert.Equal(0, result.StoreAndForwardDrain!.ResidualDepth);
|
||||
Assert.True(result.StoreAndForwardDrain.DrainPerSecond > 0);
|
||||
|
||||
// A stalled subscriber costs the healthy ones nothing (register row 50, second
|
||||
// half). This is the design's isolation claim, asserted rather than assumed.
|
||||
Assert.NotNull(result.SlowSubscriber);
|
||||
Assert.Equal(1.0, result.SlowSubscriber!.HealthyMinDeliveryRatio, precision: 2);
|
||||
Assert.True(result.SlowSubscriber.SlowDeliveryRatio < 1.0,
|
||||
"The deliberately stalled subscriber kept up, so the probe proved nothing.");
|
||||
}
|
||||
}
|
||||
+3
@@ -31,6 +31,9 @@
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ZB.MOM.WW.ScadaBridge.SiteRuntime.csproj" />
|
||||
<!-- Two-node failover rig (TwoNodeClusterFixture) — brings Host + Akka transitively. -->
|
||||
<ProjectReference Include="../../tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj" />
|
||||
<!-- Target-scale load harness (register #25). Referenced so TargetScaleHarnessSmokeTests
|
||||
can run it at CI scale; the full-scale protocol run is the standalone executable. -->
|
||||
<ProjectReference Include="../../tests/ZB.MOM.WW.ScadaBridge.LoadHarness/ZB.MOM.WW.ScadaBridge.LoadHarness.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
Reference in New Issue
Block a user