test(loadharness)+docs: wire-path stream probes — R2 batching benchmark-proven at target scale
The WP-4 runs measured the stream path only up to the bounded send channel; the probe deliberately substituted the socket hop with a controllable reader, which is exactly where the R2 SiteStreamEventBatcher lives — so batching had never been in any benchmark's hot path. The harness now also attaches per-site wire probes: a real Kestrel h2c listener hosting the production SiteStreamGrpcServer, dialed by the production SiteStreamGrpcClient with batching negotiated, batch sizes read from the production batch_size histogram via a MeterListener (--wire-probes-per-site, default 10; 0 restores the old shape). Full-scale run (10x500x75, 20-min window, results doc §8 + raw JSON): 899,807 events in 12,462 frames (mean 72.2 ev/frame, p50=p99=75), wire P99 43.4 ms vs the 250 ms budget — every batch pays the full 25 ms linger since a 75-event burst never hits the 100 cap — zero drops, zero faults, offered load 100.0% of nominal with fan-out doubled. CI smoke Fact now asserts the wire path carries traffic and observes frames.
This commit is contained in:
Joseph Doherty
@@ -99,7 +99,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
|
||||
- Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role.
|
||||
- **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
|
||||
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror, then 24 for `AckTime`). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle, automated by `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` (it always restores the csproj; an active `<Protobuf>` item must never be committed — it breaks the Docker build). **Stream events are BATCHED as of R2 (2026-08-15)** — a site emitting ~37.5k events/s used to pay one gRPC message per event. Additive wire shape, no new RPC: `batching_supported` on the request (`InstanceStreamRequest` field 3 / `SiteStreamRequest` field 2) plus a `SiteStreamEvent.batch` oneof case (**field 4**) carrying a new `SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }`. **The proto3 default of that flag IS the negotiation, and it is load-bearing** — a batch frame reaches a pre-R2 central as `EventOneofCase.None`, whose `ConvertToDomainEvent` returns null, so the whole batch would vanish with no error anywhere; an old central cannot set the flag, so it never gets one, and an old site ignores the unknown request field and keeps sending per-event frames the new client's `ForEachEvent` unpack handles as the single-event case. Server side is `SiteStreamEventBatcher`, a per-subscriber pump **downstream of `StreamRelayActor`'s bounded DropOldest channel** — so it changes framing only and **does NOT move the burst ceiling** (deferred register row 31; that ceiling lives in the shared publish stage upstream of the BroadcastHub). It never delays a lone event: it drains the already-queued backlog for free and lingers only once a backlog is proven, then emits a single-event buffer as a plain frame. `GrpcStreamBatchMaxEvents` (100, `1` disables) and `GrpcStreamBatchWindow` (25 ms, validated **strictly under 250 ms** = the load test's end-to-end P99 threshold); measured worst case P99 25.4 ms. Histogram `scadabridge.site.stream.batch_size`, recorded only on negotiated streams (`ScadaBridgeTelemetry.MeterName` is already in the `ObservedMeters` allowlist). **The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit `34a3f4bb`)** — Kestrel's `MaxStreamLifetime`/`Grpc.AspNetCore.Server` max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and `IsLive` reflects the gap in between.
|
||||
- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror, then 24 for `AckTime`). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle, automated by `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` (it always restores the csproj; an active `<Protobuf>` item must never be committed — it breaks the Docker build). **Stream events are BATCHED as of R2 (2026-08-15)** — a site emitting ~37.5k events/s used to pay one gRPC message per event. Additive wire shape, no new RPC: `batching_supported` on the request (`InstanceStreamRequest` field 3 / `SiteStreamRequest` field 2) plus a `SiteStreamEvent.batch` oneof case (**field 4**) carrying a new `SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }`. **The proto3 default of that flag IS the negotiation, and it is load-bearing** — a batch frame reaches a pre-R2 central as `EventOneofCase.None`, whose `ConvertToDomainEvent` returns null, so the whole batch would vanish with no error anywhere; an old central cannot set the flag, so it never gets one, and an old site ignores the unknown request field and keeps sending per-event frames the new client's `ForEachEvent` unpack handles as the single-event case. Server side is `SiteStreamEventBatcher`, a per-subscriber pump **downstream of `StreamRelayActor`'s bounded DropOldest channel** — so it changes framing only and **does NOT move the burst ceiling** (deferred register row 31; that ceiling lives in the shared publish stage upstream of the BroadcastHub). It never delays a lone event: it drains the already-queued backlog for free and lingers only once a backlog is proven, then emits a single-event buffer as a plain frame. `GrpcStreamBatchMaxEvents` (100, `1` disables) and `GrpcStreamBatchWindow` (25 ms, validated **strictly under 250 ms** = the load test's end-to-end P99 threshold); measured worst case P99 25.4 ms. **Benchmark-proven at target scale 2026-08-18** (the LoadHarness wire path — real Kestrel h2c + real `SiteStreamGrpcClient`, `--wire-probes-per-site`): 899,807 events in 12,462 frames (mean 72.2 ev/frame), wire P99 43.4 ms (5.8× inside the 250 ms budget — a 75-event burst never hits the 100 cap, so every batch pays the full 25 ms window: this workload's worst-case latency posture), zero drops/faults; results doc §8. Histogram `scadabridge.site.stream.batch_size`, recorded only on negotiated streams (`ScadaBridgeTelemetry.MeterName` is already in the `ObservedMeters` allowlist). **The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit `34a3f4bb`)** — Kestrel's `MaxStreamLifetime`/`Grpc.AspNetCore.Server` max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and `IsLive` reflects the gap in between.
|
||||
- Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface.
|
||||
- **`AckTime` mirror enrichment + the `Alarms` script accessor (MES alarm-status API Phase 1, 2026-08-01).** `AlarmStateChanged` carries an additive `AckTime` (`DateTimeOffset?`), mirrored on the vendored `AlarmStateUpdate` proto as **field 24** and persisted inside `native_alarm_state`'s `metadata_json` — deliberately NOT a new column, because that table is `RegisterReplicated` and LocalDb builds its CDC triggers from the column list at registration time. Set only while a condition is active AND acknowledged (so it is null while unacked and cleared on re-raise); the DCL stamps the source's own ack instant for OPC UA (new SelectClause **index 18** = `AckedState/TransitionTime`) and its observation time of the ack transition for MxGateway, which supplies none. Site `Call` scripts read alarms via the new **`Alarms.CurrentAsync()`** accessor (`ScriptRuntimeContext` + `ScriptGlobals`, local Ask on `GetAlarmSnapshotRequest`, returns `Commons.Types.Scripts.ScriptAlarm`), mirrored on `ScriptCompileSurface` AND the Central UI `SandboxScriptHost` editor surface. The trust model needed no change — it is a deny-list over API roots, not an allow-list of context members. Plan: `docs/plans/2026-06-30-mes-alarm-status-api.md` (Phases 2–4 are deployed config, not repo).
|
||||
- OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover.
|
||||
|
||||
@@ -191,7 +191,7 @@ Each is a deliberate, recorded trade — not an omission.
|
||||
| D1 | **Sustained window shortened from 1 hour to 20 minutes** at full scale | Practicality on a single shared workstation. Memory growth is reported as a *slope* precisely so a shorter window still answers the leak question: a leak shows as a positive slope in 20 min just as in 60. The 1-hour run is a single flag — `--sustain-minutes 60` — and the exact command is recorded in the results doc. |
|
||||
| D2 | **In-process, non-clustered sites** rather than 10 real two-node Akka clusters | A single box cannot host 20 clustered nodes at this scale. Cluster membership/failover is already measured on a real two-node rig (`FailoverTimingTests`, `docker/failover-drill.sh`); WP-4's question is about the hierarchy under the singleton. |
|
||||
| D3 | **Simulated data source** instead of real OPC UA | No simulated adapter exists in the DCL, and 375k real monitored items would measure the OPC UA SDK. Injected at the documented `RegisterAdapter` seam, at the exact hand-off point the real adapter uses. |
|
||||
| D4 | **gRPC socket replaced by a reader task** on the stream path | Required by the row-50 measurement itself: a stalled subscriber must be *held* stalled. All of `SiteStreamGrpcServer`'s per-subscription machinery above the socket is real. |
|
||||
| D4 | **gRPC socket replaced by a reader task** on the stream path | Required by the row-50 measurement itself: a stalled subscriber must be *held* stalled. All of `SiteStreamGrpcServer`'s per-subscription machinery above the socket is real. **Partially retired 2026-08-18:** the harness now ALSO attaches per-site wire probes — real Kestrel h2c + real `SiteStreamGrpcClient` with R2 batching negotiated (`--wire-probes-per-site`, default 10) — so the batcher/serialization/HTTP/2 hop is measured too; the substituted-reader probes remain because the slow-subscriber scenario still needs them. Results doc §8. |
|
||||
| D5 | **Instance Actors are `/user` children**, not children of a `DeploymentManagerActor` | Constructing them directly is what lets the ramp be timed in isolation and keeps the harness independent of the deploy round-trip. Visible only as `InstanceActorInitialized` dead letters, which the harness suppresses. No measured path differs. |
|
||||
| D6 | **Akka logging at WARNING** | At 37,500 updates/s, INFO output would itself become a measured load. |
|
||||
| D7 | `[xc-3]` site event logging volume, `[xc-4]` audit-log degradation, `[xc-5]` template flattening, `[xc-8]` UI responsiveness **not covered** | Out of scope for this harness: the first three need central MS SQL and the Template Engine (a central-cluster fixture, not a site one), and `[xc-8]` needs a browser. Recorded as residual scope in the results doc rather than silently dropped. |
|
||||
|
||||
@@ -275,7 +275,10 @@ Stated plainly so the evidence is not over-read:
|
||||
- **Not a real-network run.** The gRPC, health-transport and S&F-to-central hops are
|
||||
in-process. Serialization cost, TLS, WAN latency and socket backpressure are out of
|
||||
frame; the *stream* path's backpressure was measured deliberately by substituting a
|
||||
controllable reader for the socket (D4).
|
||||
controllable reader for the socket (D4). **Partially superseded by §8 (2026-08-18):**
|
||||
the site→central stream hop now also runs over a real loopback HTTP/2 socket with
|
||||
the R2 batcher, protobuf serialization and Kestrel in the measured path. TLS and
|
||||
WAN latency remain out of frame.
|
||||
- **Not a real OPC UA run.** 375,000 monitored items were simulated at the adapter
|
||||
callback (D3). The DCL above that callback is real and fully exercised.
|
||||
- **Not 1 hour.** 20 minutes (D1). See finding F1 for the one question this leaves
|
||||
@@ -323,6 +326,111 @@ dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests \
|
||||
`--help`-style knobs: `--sites`, `--instances-per-site`, `--tags-per-instance`,
|
||||
`--tag-update-period-seconds`, `--settle-minutes`, `--sustain-minutes`,
|
||||
`--sample-seconds`, `--health-interval-seconds`, `--debug-probe-interval-seconds`,
|
||||
`--subscribe-settle-seconds`, `--stream-probes-per-site`, `--sf-drain-messages`,
|
||||
`--slow-subscriber-events`, `--data-dir`, `--results`. An unknown key is rejected
|
||||
rather than ignored, so a typo cannot silently change the measured scale.
|
||||
`--subscribe-settle-seconds`, `--stream-probes-per-site`, `--wire-probes-per-site`,
|
||||
`--sf-drain-messages`, `--slow-subscriber-events`, `--data-dir`, `--results`. An
|
||||
unknown key is rejected rather than ignored, so a typo cannot silently change the
|
||||
measured scale.
|
||||
|
||||
Since 2026-08-18 the default protocol also attaches 10 **wire** stream subscribers
|
||||
per site (real Kestrel + real `SiteStreamGrpcClient`, §8); `--wire-probes-per-site 0`
|
||||
reproduces the original in-process-only shape.
|
||||
|
||||
---
|
||||
|
||||
## 8. Wire-path run — R2 stream batching in the hot path (2026-08-18)
|
||||
|
||||
**Raw metrics:** `docs/plans/2026-08-18-loadharness-results-wire.json`
|
||||
|
||||
### 8.1 Why this run exists
|
||||
|
||||
Both prior runs (§2 and the §4 F1 1-hour sustain) measured the stream path only up
|
||||
to the bounded send channel: `StreamSubscriberProbe` deliberately substitutes the
|
||||
final socket hop with a controllable reader (deviation D4 — required by the row-50
|
||||
slow-subscriber measurement). The R2 gRPC event batcher (`SiteStreamEventBatcher`,
|
||||
merged `c75c79f8` on 2026-08-15, about an hour *after* the §2 run) lives exactly in
|
||||
that substituted hop — so no benchmark run had ever exercised it, only R2's own
|
||||
component-level measurement (worst-case P99 25.4 ms) and trickle-rate live traffic
|
||||
on the rig (register row 48).
|
||||
|
||||
This run adds a **wire path** to the harness without touching the existing probes:
|
||||
each site hosts the production `SiteStreamGrpcServer` on a real Kestrel HTTP/2 (h2c)
|
||||
loopback listener (production `CommunicationOptions` defaults — batch cap 100,
|
||||
window 25 ms, channel capacity 1000), and the production `SiteStreamGrpcClient`
|
||||
dials in with `BatchingSupported = true`, one channel per site with 10 streams
|
||||
multiplexed on it — the exact production shape and the exact production code on both
|
||||
ends. Latency is recorded in the client callback, i.e. after protobuf
|
||||
deserialization, batch unpacking and domain mapping. The batch-size figures are the
|
||||
production `scadabridge.site.stream.batch_size` histogram itself, captured by a
|
||||
`MeterListener`. Deliberately not in the path: `ControlPlaneAuthInterceptor`
|
||||
(authenticates once per stream lifetime, nothing per-event) and TLS (production is
|
||||
h2c too).
|
||||
|
||||
The 100 wire subscribers are attached **in addition to** the 100 in-process probes,
|
||||
watching the same instances (same stride), so one run yields both distributions and
|
||||
their difference isolates the wire hop's cost. Total hub fan-out is therefore
|
||||
double that of the §2 run.
|
||||
|
||||
### 8.2 Environment
|
||||
|
||||
Identical posture to §1: same host (M4 Pro, 14 cores, 48 GB), .NET 10.0.5, server
|
||||
GC, Release; the 8-node `docker/` rig stayed running throughout (D8); 20-minute
|
||||
window after a 2-minute settle; harness at the state committed with this section.
|
||||
|
||||
### 8.3 Measured
|
||||
|
||||
| | in-process probes (§2 path) | **wire probes (new)** |
|
||||
|---|---|---|
|
||||
| Subscribers | 100 | 100 |
|
||||
| Events delivered (window) | 899,807 | **899,807** |
|
||||
| Events dropped / faults | 0 / — | **0 / 0** |
|
||||
| Latency mean | 1.79 ms | **28.00 ms** |
|
||||
| Latency P50 | 1.36 ms | **26.97 ms** |
|
||||
| Latency P95 | 4.19 ms | **32.07 ms** |
|
||||
| Latency P99 | 6.74 ms | **43.44 ms** |
|
||||
| Latency P99.9 / max | 25.83 / 32.17 ms | **53.94 / 62.42 ms** |
|
||||
|
||||
Batching (production histogram, window deltas):
|
||||
|
||||
```
|
||||
events 899,807 frames 12,462 → mean 72.20 events/frame (72x frame reduction)
|
||||
batch size p50/p95/p99/max = 75/75/75/75 batched frames 11,998 single-event frames 464
|
||||
server channel drops 0 stream faults 0 graceful completions 0
|
||||
```
|
||||
|
||||
Offered load held at **37,497/s = 100.0% of nominal** with the doubled fan-out
|
||||
(driver overrun 0.0 s, skipped 0). Whole-run CPU mean **66% of one core (4.7% of
|
||||
the box)** vs 41% in §2 — the added ~25% of one core buys 10 Kestrel listeners, 100
|
||||
live HTTP/2 streams, protobuf serialization of ~900 k events and the client-side
|
||||
unpack. Working-set slope **−116 MB/min** (a large mid-window release; peak
|
||||
3,413 MB), heap sawtooth with 5 gen-2 collects and endpoints 1,876→1,943 MB —
|
||||
consistent with the §4 F1 resolution (row 29: not a leak).
|
||||
|
||||
### 8.4 Reading the numbers
|
||||
|
||||
- **The batcher engaged exactly as designed, at its natural batch size.** The
|
||||
driver emits each instance's 75 tags contiguously within a slice, so a subscriber
|
||||
receives one 75-event burst per 10 s period. 75 < the 100-event cap, so the
|
||||
size cap never closes a batch early — nearly every burst becomes exactly one
|
||||
75-event frame (p50 = p99 = max = 75). The 464 single-event frames are burst
|
||||
stragglers that arrived after their batch's window closed.
|
||||
- **The ~26 ms wire P50 is dominated by the deliberate linger, and that is the
|
||||
documented trade.** Because bursts never hit the cap, every batch pays the full
|
||||
25 ms `GrpcStreamBatchWindow` — this workload is the batcher's *worst-case latency
|
||||
posture and best-case framing win* simultaneously. Wire P50 26.97 ms ≈ in-process
|
||||
P50 1.36 ms + the 25 ms window + sub-millisecond socket/serialize cost. Against
|
||||
the 250 ms P99 threshold the wire path passes with a **5.8× margin** (43.44 ms),
|
||||
and the `CommunicationOptionsValidator` bound (window strictly < 250 ms) is
|
||||
confirmed to compose: end-to-end latency stayed threshold-compliant with the
|
||||
window fully paid on every frame.
|
||||
- **Framing efficiency at scale is now measured, not extrapolated:** 899,807 events
|
||||
crossed the wire in 12,462 frames. Pre-R2, the same window would have cost
|
||||
899,807 frames.
|
||||
- **Nothing degraded elsewhere.** The in-process path under doubled fan-out reads
|
||||
P99 6.74 ms vs 4.57 ms in §2 — same order, still a 37× margin — and every §2
|
||||
pass criterion that this run re-exercises (offered load, zero drops at healthy
|
||||
subscribers, health reports, debug snapshots, S&F drain, slow-subscriber
|
||||
isolation) repeated its pass.
|
||||
|
||||
**Verdict: PASS.** R2's negotiated batching is now benchmark-proven at target scale
|
||||
over a real socket: 72× fewer frames, zero loss, zero faults, and worst-case-posture
|
||||
latency 5.8× inside the budget.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -68,6 +68,19 @@ public sealed record HarnessConfig
|
||||
/// </summary>
|
||||
public int StreamProbesPerSite { get; init; } = 10;
|
||||
|
||||
/// <summary>
|
||||
/// Instances per site additionally watched over a REAL gRPC hop: a per-site
|
||||
/// Kestrel h2c listener hosts the production <c>SiteStreamGrpcServer</c> and the
|
||||
/// production <c>SiteStreamGrpcClient</c> dials in with batching negotiated, so
|
||||
/// the R2 <c>SiteStreamEventBatcher</c>, protobuf serialization and HTTP/2 flow
|
||||
/// control are inside the measured path (they are NOT for
|
||||
/// <see cref="StreamProbesPerSite"/>, whose final hop is deliberately an
|
||||
/// in-process reader). 0 disables the wire path entirely. Like the in-process
|
||||
/// probes, each wire subscriber's stream graph sees the full site event flow and
|
||||
/// filters it, so this adds to the fan-out multiplier.
|
||||
/// </summary>
|
||||
public int WireProbesPerSite { get; init; } = 10;
|
||||
|
||||
/// <summary>Store-and-forward messages enqueued for the drain-rate measurement (register row 50).</summary>
|
||||
public int StoreAndForwardDrainMessages { get; init; } = 20_000;
|
||||
|
||||
@@ -131,6 +144,7 @@ public sealed record HarnessConfig
|
||||
"health-interval-seconds" => config with { HealthReportInterval = TimeSpan.FromSeconds(double.Parse(value)) },
|
||||
"debug-probe-interval-seconds" => config with { DebugProbeInterval = TimeSpan.FromSeconds(double.Parse(value)) },
|
||||
"stream-probes-per-site" => config with { StreamProbesPerSite = int.Parse(value) },
|
||||
"wire-probes-per-site" => config with { WireProbesPerSite = int.Parse(value) },
|
||||
"sf-drain-messages" => config with { StoreAndForwardDrainMessages = int.Parse(value) },
|
||||
"slow-subscriber-events" => config with { SlowSubscriberEvents = int.Parse(value) },
|
||||
"data-dir" => config with { DataDirectory = value },
|
||||
|
||||
@@ -40,6 +40,10 @@ namespace ZB.MOM.WW.ScadaBridge.LoadHarness;
|
||||
/// <param name="StreamProbeDropped">Events evicted by live subscribers' bounded channels.</param>
|
||||
/// <param name="StoreAndForwardDrain">Store-and-forward drain measurement (register row 50).</param>
|
||||
/// <param name="SlowSubscriber">Slow-subscriber isolation measurement (register row 50).</param>
|
||||
/// <param name="WireStream">
|
||||
/// Wire-path stream measurement — real Kestrel + real <c>SiteStreamGrpcClient</c> with
|
||||
/// R2 batching negotiated — or null when <see cref="HarnessConfig.WireProbesPerSite"/> is 0.
|
||||
/// </param>
|
||||
public sealed record HarnessRunResult(
|
||||
HarnessConfig Config,
|
||||
EnvironmentInfo Environment,
|
||||
@@ -67,7 +71,30 @@ public sealed record HarnessRunResult(
|
||||
long StreamProbeReceived,
|
||||
long StreamProbeDropped,
|
||||
StoreAndForwardDrainResult? StoreAndForwardDrain,
|
||||
SlowSubscriberResult? SlowSubscriber);
|
||||
SlowSubscriberResult? SlowSubscriber,
|
||||
WireStreamResult? WireStream);
|
||||
|
||||
/// <summary>
|
||||
/// Steady-state-window measurement of the wire-path stream probes: the production
|
||||
/// gRPC server (Kestrel h2c) and client with the R2 batcher, protobuf serialization
|
||||
/// and HTTP/2 flow control INSIDE the measured path. All figures are deltas over the
|
||||
/// steady-state window only.
|
||||
/// </summary>
|
||||
/// <param name="ProbeCount">Wire subscribers attached across the fleet.</param>
|
||||
/// <param name="Latency">End-to-end DCL-boundary → post-unpack client callback latency.</param>
|
||||
/// <param name="EventsReceived">Domain events delivered by the client in the window.</param>
|
||||
/// <param name="ServerDroppedEvents">Events evicted by the wire servers' bounded send channels in the window.</param>
|
||||
/// <param name="StreamFaults">Stream faults reported by the client (expected 0).</param>
|
||||
/// <param name="GracefulCompletions">Graceful server-side stream completions (expected 0 inside the 4 h lifetime).</param>
|
||||
/// <param name="Batches">The frame/batch-size distribution the R2 batcher actually produced.</param>
|
||||
public sealed record WireStreamResult(
|
||||
int ProbeCount,
|
||||
LatencySnapshot Latency,
|
||||
long EventsReceived,
|
||||
long ServerDroppedEvents,
|
||||
long StreamFaults,
|
||||
long GracefulCompletions,
|
||||
BatchSizeWindow Batches);
|
||||
|
||||
/// <summary>Host and runtime facts recorded alongside the numbers.</summary>
|
||||
/// <param name="MachineName">Host name.</param>
|
||||
@@ -120,6 +147,7 @@ public static class HarnessRun
|
||||
ObservabilityProbes? probes = null;
|
||||
StoreAndForwardDrainResult? drainResult = null;
|
||||
SlowSubscriberResult? slowResult = null;
|
||||
BatchSizeAggregator? batchAggregator = null;
|
||||
|
||||
var aggregator = new CentralHealthAggregator(
|
||||
Options.Create(new HealthMonitoringOptions
|
||||
@@ -154,6 +182,18 @@ public static class HarnessRun
|
||||
site.AttachStreamProbes(latency);
|
||||
log($" {sites.Sum(s => s.Probes.Count)} live stream subscribers attached");
|
||||
|
||||
// Wire path (real gRPC): the aggregator must exist BEFORE the first stream
|
||||
// opens or the MeterListener misses early frames.
|
||||
var wireLatency = new LatencyHistogram();
|
||||
if (config.WireProbesPerSite > 0)
|
||||
{
|
||||
batchAggregator = new BatchSizeAggregator();
|
||||
foreach (var site in sites)
|
||||
await site.AttachWireStreamProbesAsync(config.WireProbesPerSite, wireLatency, log);
|
||||
log($" {sites.Sum(s => s.WireSubscribers.Count)} wire stream subscribers attached " +
|
||||
"(real Kestrel + SiteStreamGrpcClient, batching negotiated)");
|
||||
}
|
||||
|
||||
// Let Instance Actors complete their DCL subscribe round-trips before the
|
||||
// driver starts; an emit before SubscribeBatchAsync has captured the
|
||||
// callback would be silently discarded.
|
||||
@@ -184,6 +224,18 @@ public static class HarnessRun
|
||||
probe.RetargetLatency(steadyLatency);
|
||||
}
|
||||
|
||||
// Wire path: same ramp/steady histogram separation, plus window-start
|
||||
// baselines for the client/server counters and the batch distribution.
|
||||
var steadyWireLatency = new LatencyHistogram();
|
||||
var wireReceivedAtWindowStart = sites.Sum(s => s.WireSubscribers.Sum(p => p.ReceivedEvents));
|
||||
var wireDroppedAtWindowStart = sites.Sum(s => s.WireHost?.Server.DroppedStreamEventCount ?? 0);
|
||||
var batchesAtWindowStart = batchAggregator?.Snapshot();
|
||||
foreach (var site in sites)
|
||||
{
|
||||
foreach (var wire in site.WireSubscribers)
|
||||
wire.RetargetLatency(steadyWireLatency);
|
||||
}
|
||||
|
||||
var windowWatch = Stopwatch.StartNew();
|
||||
log($"Sustained measurement window: {config.SustainDuration.TotalMinutes:F1} min...");
|
||||
await Task.Delay(config.SustainDuration, cancellationToken);
|
||||
@@ -199,6 +251,24 @@ public static class HarnessRun
|
||||
var probeReceived = sites.Sum(s => s.Probes.Sum(p => p.ReceivedEvents)) - receivedAtWindowStart;
|
||||
var probeDropped = sites.Sum(s => s.Probes.Sum(p => p.DroppedEvents)) - droppedAtWindowStart;
|
||||
|
||||
// Wire-path window deltas, captured BEFORE the row-50 scenarios below so
|
||||
// their publishing cannot contaminate the wire figures.
|
||||
WireStreamResult? wireResult = null;
|
||||
if (batchAggregator is not null)
|
||||
{
|
||||
wireResult = new WireStreamResult(
|
||||
ProbeCount: sites.Sum(s => s.WireSubscribers.Count),
|
||||
Latency: steadyWireLatency.Snapshot(),
|
||||
EventsReceived: sites.Sum(s => s.WireSubscribers.Sum(p => p.ReceivedEvents)) - wireReceivedAtWindowStart,
|
||||
ServerDroppedEvents: sites.Sum(s => s.WireHost?.Server.DroppedStreamEventCount ?? 0) - wireDroppedAtWindowStart,
|
||||
StreamFaults: sites.Sum(s => s.WireSubscribers.Sum(p => p.Faults)),
|
||||
GracefulCompletions: sites.Sum(s => s.WireSubscribers.Sum(p => p.GracefulCompletions)),
|
||||
Batches: BatchSizeWindow.From(batchesAtWindowStart!, batchAggregator.Snapshot()));
|
||||
log($" wire path: {wireResult.EventsReceived:N0} events in {wireResult.Batches.Frames:N0} frames " +
|
||||
$"(mean {wireResult.Batches.MeanEventsPerFrame:F1} ev/frame, max {wireResult.Batches.Max}), " +
|
||||
$"P99 {wireResult.Latency.P99Ms:F2}ms");
|
||||
}
|
||||
|
||||
// Register row 50 — measured after the sustained window so the numbers are
|
||||
// not competing with the full tag load for CPU, and reported separately for
|
||||
// the same reason.
|
||||
@@ -250,12 +320,14 @@ public static class HarnessRun
|
||||
StreamProbeReceived: probeReceived,
|
||||
StreamProbeDropped: probeDropped,
|
||||
StoreAndForwardDrain: drainResult,
|
||||
SlowSubscriber: slowResult);
|
||||
SlowSubscriber: slowResult,
|
||||
WireStream: wireResult);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (driver != null) await driver.DisposeAsync();
|
||||
if (probes != null) await probes.DisposeAsync();
|
||||
batchAggregator?.Dispose();
|
||||
await sampler.DisposeAsync();
|
||||
|
||||
log("Tearing down sites...");
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
using System.Diagnostics.Metrics;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
|
||||
|
||||
/// <summary>
|
||||
/// Captures the production <c>scadabridge.site.stream.batch_size</c> histogram via a
|
||||
/// <see cref="MeterListener"/>, so a wire-path run can report the actual frame/batch
|
||||
/// distribution the R2 batcher produced — not a client-side reconstruction. The
|
||||
/// instrument is recorded once per emitted gRPC frame (negotiated streams only), with
|
||||
/// the frame's event count as the value, which makes <c>count = frames</c> and
|
||||
/// <c>sum = events</c> exactly.
|
||||
///
|
||||
/// <para>
|
||||
/// The harness hosts every site's <see cref="ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer"/>
|
||||
/// in-process, so this listener sees every wire frame the run emits and nothing else —
|
||||
/// no other component in the harness records this instrument.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class BatchSizeAggregator : IDisposable
|
||||
{
|
||||
/// <summary>
|
||||
/// Sizes above this are clamped into the last bucket. The production cap is
|
||||
/// <c>GrpcStreamBatchMaxEvents</c> = 100, so 512 leaves generous headroom for a
|
||||
/// configured-up run without an unbounded array.
|
||||
/// </summary>
|
||||
public const int MaxTrackedBatchSize = 512;
|
||||
|
||||
private readonly MeterListener _listener;
|
||||
private readonly long[] _framesBySize = new long[MaxTrackedBatchSize + 1];
|
||||
|
||||
/// <summary>Starts listening. Must be created BEFORE the first stream opens.</summary>
|
||||
public BatchSizeAggregator()
|
||||
{
|
||||
_listener = new MeterListener
|
||||
{
|
||||
InstrumentPublished = (instrument, listener) =>
|
||||
{
|
||||
if (instrument.Meter.Name == ScadaBridgeTelemetry.MeterName &&
|
||||
instrument.Name == "scadabridge.site.stream.batch_size")
|
||||
{
|
||||
listener.EnableMeasurementEvents(instrument);
|
||||
}
|
||||
},
|
||||
};
|
||||
_listener.SetMeasurementEventCallback<int>((_, value, _, _) =>
|
||||
{
|
||||
var size = Math.Clamp(value, 1, MaxTrackedBatchSize);
|
||||
Interlocked.Increment(ref _framesBySize[size]);
|
||||
});
|
||||
_listener.Start();
|
||||
}
|
||||
|
||||
/// <summary>Copies the cumulative per-size frame counts.</summary>
|
||||
/// <returns>Index = events-per-frame (clamped), value = frames emitted at that size.</returns>
|
||||
public long[] Snapshot()
|
||||
{
|
||||
var copy = new long[_framesBySize.Length];
|
||||
for (var i = 0; i < copy.Length; i++)
|
||||
copy[i] = Interlocked.Read(ref _framesBySize[i]);
|
||||
return copy;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose() => _listener.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Batch-size distribution over one measurement window, computed from two
|
||||
/// <see cref="BatchSizeAggregator.Snapshot"/>s. Percentiles are exact (integer sizes,
|
||||
/// full counts — nothing is bucketed away).
|
||||
/// </summary>
|
||||
/// <param name="Frames">gRPC frames written in the window.</param>
|
||||
/// <param name="Events">Events those frames carried.</param>
|
||||
/// <param name="BatchedFrames">Frames that carried more than one event.</param>
|
||||
/// <param name="MeanEventsPerFrame">Events / Frames.</param>
|
||||
/// <param name="P50">Median events-per-frame.</param>
|
||||
/// <param name="P95">95th percentile events-per-frame.</param>
|
||||
/// <param name="P99">99th percentile events-per-frame.</param>
|
||||
/// <param name="Max">Largest frame observed in the window.</param>
|
||||
public sealed record BatchSizeWindow(
|
||||
long Frames,
|
||||
long Events,
|
||||
long BatchedFrames,
|
||||
double MeanEventsPerFrame,
|
||||
int P50,
|
||||
int P95,
|
||||
int P99,
|
||||
int Max)
|
||||
{
|
||||
/// <summary>Computes the window distribution from cumulative start/end snapshots.</summary>
|
||||
/// <param name="start">Snapshot at window start.</param>
|
||||
/// <param name="end">Snapshot at window end.</param>
|
||||
/// <returns>The distribution of frames emitted between the two snapshots.</returns>
|
||||
public static BatchSizeWindow From(long[] start, long[] end)
|
||||
{
|
||||
long frames = 0, events = 0, batched = 0;
|
||||
var max = 0;
|
||||
var delta = new long[end.Length];
|
||||
for (var size = 1; size < end.Length; size++)
|
||||
{
|
||||
var n = end[size] - start[size];
|
||||
if (n <= 0) continue;
|
||||
delta[size] = n;
|
||||
frames += n;
|
||||
events += n * size;
|
||||
if (size > 1) batched += n;
|
||||
max = size;
|
||||
}
|
||||
|
||||
return new BatchSizeWindow(
|
||||
frames, events, batched,
|
||||
frames == 0 ? 0 : events / (double)frames,
|
||||
Percentile(delta, frames, 50),
|
||||
Percentile(delta, frames, 95),
|
||||
Percentile(delta, frames, 99),
|
||||
max);
|
||||
}
|
||||
|
||||
private static int Percentile(long[] delta, long frames, double percentile)
|
||||
{
|
||||
if (frames == 0) return 0;
|
||||
var target = (long)Math.Ceiling(frames * percentile / 100.0);
|
||||
if (target < 1) target = 1;
|
||||
|
||||
long cumulative = 0;
|
||||
for (var size = 1; size < delta.Length; size++)
|
||||
{
|
||||
cumulative += delta[size];
|
||||
if (cumulative >= target)
|
||||
return size;
|
||||
}
|
||||
|
||||
return delta.Length - 1;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Hosting;
|
||||
using Microsoft.AspNetCore.Hosting.Server;
|
||||
using Microsoft.AspNetCore.Hosting.Server.Features;
|
||||
using Microsoft.AspNetCore.Server.Kestrel.Core;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Probes;
|
||||
|
||||
/// <summary>
|
||||
/// The site half of the wire-path stream probe: a real Kestrel HTTP/2 (h2c) listener on
|
||||
/// a loopback ephemeral port, serving the PRODUCTION <see cref="SiteStreamGrpcServer"/>
|
||||
/// over the site's real <see cref="ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming.SiteStreamManager"/>.
|
||||
///
|
||||
/// <para>
|
||||
/// This is the hop <see cref="StreamSubscriberProbe"/> deliberately substitutes with a
|
||||
/// controllable in-process reader (which the slow-subscriber scenario needs), and
|
||||
/// therefore the hop the original WP-4 runs never exercised: the
|
||||
/// <c>SiteStreamEventBatcher</c> pump (R2), protobuf serialization,
|
||||
/// <c>IServerStreamWriter.WriteAsync</c>, and HTTP/2 flow control. The server is built
|
||||
/// through its production DI constructor with default <see cref="CommunicationOptions"/>,
|
||||
/// so the batch negotiation (100 events / 25 ms window), channel capacities and the
|
||||
/// concurrency cap are exactly what ships.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// Deliberately NOT in the path: <c>ControlPlaneAuthInterceptor</c> (it lives in the
|
||||
/// Host project and authenticates once per CALL — one header compare per stream
|
||||
/// lifetime, nothing on the per-event path) and TLS (production is h2c too).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class SiteWireStreamHost : IAsyncDisposable
|
||||
{
|
||||
private readonly WebApplication _app;
|
||||
|
||||
/// <summary>The hosted production gRPC server (drop counters, active stream count).</summary>
|
||||
public SiteStreamGrpcServer Server { get; }
|
||||
|
||||
/// <summary>The dialable endpoint, e.g. <c>http://127.0.0.1:54321</c>.</summary>
|
||||
public string Endpoint { get; }
|
||||
|
||||
private SiteWireStreamHost(WebApplication app, SiteStreamGrpcServer server, string endpoint)
|
||||
{
|
||||
_app = app;
|
||||
Server = server;
|
||||
Endpoint = endpoint;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds, starts and readies the listener for one site.
|
||||
/// </summary>
|
||||
/// <param name="site">The site whose stream manager and actor system back the server.</param>
|
||||
/// <returns>The started host.</returns>
|
||||
public static async Task<SiteWireStreamHost> StartAsync(SiteRuntimeFixture site)
|
||||
{
|
||||
var server = new SiteStreamGrpcServer(
|
||||
site.StreamManager,
|
||||
Microsoft.Extensions.Logging.Abstractions.NullLogger<SiteStreamGrpcServer>.Instance,
|
||||
Options.Create(new CommunicationOptions()));
|
||||
|
||||
var builder = WebApplication.CreateSlimBuilder();
|
||||
builder.Logging.ClearProviders();
|
||||
builder.WebHost.ConfigureKestrel(options =>
|
||||
options.Listen(System.Net.IPAddress.Loopback, 0, listenOptions =>
|
||||
listenOptions.Protocols = HttpProtocols.Http2));
|
||||
builder.Services.AddGrpc();
|
||||
builder.Services.AddSingleton(server);
|
||||
|
||||
var app = builder.Build();
|
||||
app.MapGrpcService<SiteStreamGrpcServer>();
|
||||
await app.StartAsync();
|
||||
|
||||
// Same post-start readiness handoff the production host performs.
|
||||
server.SetReady(site.System);
|
||||
|
||||
var address = app.Services.GetRequiredService<IServer>()
|
||||
.Features.Get<IServerAddressesFeature>()?.Addresses.FirstOrDefault()
|
||||
?? throw new InvalidOperationException("Kestrel bound no address.");
|
||||
|
||||
return new SiteWireStreamHost(app, server, address);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
// Mirrors the production shutdown ordering: cancel active streams first so
|
||||
// their pumps observe cancellation, then stop the listener.
|
||||
Server.CancelAllStreams();
|
||||
await _app.StopAsync();
|
||||
await _app.DisposeAsync();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
using ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Probes;
|
||||
|
||||
/// <summary>
|
||||
/// The central half of the wire-path stream probe: one instance subscription opened
|
||||
/// through the PRODUCTION <see cref="SiteStreamGrpcClient"/> — the exact
|
||||
/// <c>SubscribeAsync</c> path central's <c>DebugStreamBridgeActor</c> uses, including
|
||||
/// the R2 batch negotiation (<c>BatchingSupported = true</c>) and the client-side
|
||||
/// batch unpack + proto→domain conversion.
|
||||
///
|
||||
/// <para>
|
||||
/// Latency is recorded in <c>onEvent</c>, i.e. AFTER wire deserialization, batch
|
||||
/// unpacking and domain mapping — the same "DCL boundary → subscriber" measurement as
|
||||
/// <see cref="StreamSubscriberProbe"/>, extended across the real socket. The emit
|
||||
/// instant travels verbatim on the proto <c>Timestamp</c>, so the subtraction stays a
|
||||
/// genuine end-to-end figure (same process, same clock).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class WireStreamSubscriber
|
||||
{
|
||||
private Task _streamTask = Task.CompletedTask;
|
||||
private volatile LatencyHistogram? _latency;
|
||||
private long _received;
|
||||
private long _faults;
|
||||
private long _completions;
|
||||
|
||||
/// <summary>Probe name (also the stream's correlation id).</summary>
|
||||
public string Name { get; }
|
||||
|
||||
/// <summary>Domain events delivered by the client callback.</summary>
|
||||
public long ReceivedEvents => Interlocked.Read(ref _received);
|
||||
|
||||
/// <summary>Stream faults reported via <c>onError</c>. Expected 0 for a healthy run.</summary>
|
||||
public long Faults => Interlocked.Read(ref _faults);
|
||||
|
||||
/// <summary>Graceful server-side completions via <c>onCompleted</c>. Expected 0 inside a run shorter than the 4 h stream lifetime.</summary>
|
||||
public long GracefulCompletions => Interlocked.Read(ref _completions);
|
||||
|
||||
private WireStreamSubscriber(string name, LatencyHistogram? latency)
|
||||
{
|
||||
Name = name;
|
||||
_latency = latency;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Opens one instance stream on the shared per-site client. The subscription task
|
||||
/// runs until <paramref name="ct"/> cancels — the harness deliberately does NOT
|
||||
/// reconnect on fault (a fault inside a 25-minute loopback run is a finding, not
|
||||
/// an operational event to paper over), it counts it.
|
||||
/// </summary>
|
||||
/// <param name="client">The site's shared production gRPC client (one channel per site, streams multiplexed).</param>
|
||||
/// <param name="instanceUniqueName">Instance whose events this probe receives.</param>
|
||||
/// <param name="name">Probe/correlation name (must be a valid Akka path element).</param>
|
||||
/// <param name="latency">Histogram fed with end-to-end event latency.</param>
|
||||
/// <param name="log">Progress sink for fault reporting.</param>
|
||||
/// <param name="ct">Stops the subscription.</param>
|
||||
/// <returns>The attached subscriber.</returns>
|
||||
public static WireStreamSubscriber Attach(
|
||||
SiteStreamGrpcClient client,
|
||||
string instanceUniqueName,
|
||||
string name,
|
||||
LatencyHistogram? latency,
|
||||
Action<string> log,
|
||||
CancellationToken ct)
|
||||
{
|
||||
var self = new WireStreamSubscriber(name, latency);
|
||||
|
||||
self._streamTask = client.SubscribeAsync(
|
||||
correlationId: name,
|
||||
instanceUniqueName: instanceUniqueName,
|
||||
onEvent: evt =>
|
||||
{
|
||||
Interlocked.Increment(ref self._received);
|
||||
if (self._latency is { } histogram && evt is AttributeValueChanged changed)
|
||||
histogram.Record(DateTimeOffset.UtcNow - changed.Timestamp);
|
||||
},
|
||||
onError: ex =>
|
||||
{
|
||||
Interlocked.Increment(ref self._faults);
|
||||
log($" wire probe {name} stream fault: {ex.GetType().Name}: {ex.Message}");
|
||||
},
|
||||
onCompleted: () => Interlocked.Increment(ref self._completions),
|
||||
ct);
|
||||
|
||||
return self;
|
||||
}
|
||||
|
||||
/// <summary>Repoints the latency histogram (steady-state window separation) — mirrors <see cref="StreamSubscriberProbe.RetargetLatency"/>.</summary>
|
||||
/// <param name="latency">The histogram to record into from now on, or null to stop recording.</param>
|
||||
public void RetargetLatency(LatencyHistogram? latency) => _latency = latency;
|
||||
|
||||
/// <summary>Awaits the subscription task after its token has been cancelled.</summary>
|
||||
/// <returns>A task that completes when the stream loop has exited.</returns>
|
||||
public async Task WaitForShutdownAsync()
|
||||
{
|
||||
try
|
||||
{
|
||||
await _streamTask;
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Normal teardown.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,7 @@ Console.WriteLine($" sites {config.Sites}");
|
||||
Console.WriteLine($" instances/site {config.InstancesPerSite}");
|
||||
Console.WriteLine($" tags/instance {config.TagsPerInstance}");
|
||||
Console.WriteLine($" total subscriptions {config.TotalSubscriptions:N0}");
|
||||
Console.WriteLine($" stream probes/site {config.StreamProbesPerSite} in-process + {config.WireProbesPerSite} wire (real gRPC)");
|
||||
Console.WriteLine($" nominal update rate {config.NominalUpdatesPerSecond:N0}/s");
|
||||
Console.WriteLine($" settle / sustain {config.SettleDuration.TotalMinutes:F1} / {config.SustainDuration.TotalMinutes:F1} min");
|
||||
Console.WriteLine();
|
||||
|
||||
@@ -47,6 +47,18 @@ public static class ResultsFormatter
|
||||
builder.AppendLine(culture, $"events dropped {result.StreamProbeDropped:N0}");
|
||||
builder.AppendLine();
|
||||
|
||||
if (result.WireStream is { } wire)
|
||||
{
|
||||
builder.AppendLine("-- wire stream subscribers (real gRPC, R2 batching in path) --");
|
||||
builder.AppendLine(culture, $"probes {wire.ProbeCount} (Kestrel h2c + SiteStreamGrpcClient, batching negotiated)");
|
||||
AppendLatency(builder, culture, wire.Latency);
|
||||
builder.AppendLine(culture, $"events delivered {wire.EventsReceived:N0} in {wire.Batches.Frames:N0} frames");
|
||||
builder.AppendLine(culture, $"batching mean {wire.Batches.MeanEventsPerFrame:F2} ev/frame p50 {wire.Batches.P50} p95 {wire.Batches.P95} p99 {wire.Batches.P99} max {wire.Batches.Max} ({wire.Batches.BatchedFrames:N0} frames >1 event)");
|
||||
builder.AppendLine(culture, $"server channel drops {wire.ServerDroppedEvents:N0}");
|
||||
builder.AppendLine(culture, $"faults / completions {wire.StreamFaults:N0} / {wire.GracefulCompletions:N0}");
|
||||
builder.AppendLine();
|
||||
}
|
||||
|
||||
builder.AppendLine("-- health report delivery --");
|
||||
AppendLatency(builder, culture, result.HealthReportLatency);
|
||||
builder.AppendLine(culture, $"reports ingested {result.HealthReportsDelivered:N0}");
|
||||
|
||||
@@ -6,6 +6,7 @@ using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
|
||||
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer;
|
||||
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
|
||||
@@ -43,7 +44,11 @@ public sealed class SiteRuntimeFixture : IAsyncDisposable
|
||||
private readonly ILocalDb _localDb;
|
||||
private readonly List<IActorRef> _instanceActors = new();
|
||||
private readonly List<StreamSubscriberProbe> _probes = new();
|
||||
private readonly List<WireStreamSubscriber> _wireSubscribers = new();
|
||||
private readonly string _dataDirectory;
|
||||
private SiteWireStreamHost? _wireHost;
|
||||
private SiteStreamGrpcClient? _wireClient;
|
||||
private CancellationTokenSource? _wireCts;
|
||||
|
||||
/// <summary>Site identifier, e.g. <c>site-01</c>.</summary>
|
||||
public string SiteId { get; }
|
||||
@@ -72,6 +77,12 @@ public sealed class SiteRuntimeFixture : IAsyncDisposable
|
||||
/// <summary>The live stream subscribers attached to this site.</summary>
|
||||
public IReadOnlyList<StreamSubscriberProbe> Probes => _probes;
|
||||
|
||||
/// <summary>The wire-path (real gRPC) subscribers attached to this site, if any.</summary>
|
||||
public IReadOnlyList<WireStreamSubscriber> WireSubscribers => _wireSubscribers;
|
||||
|
||||
/// <summary>The site's real gRPC listener, when wire probes are attached.</summary>
|
||||
public SiteWireStreamHost? WireHost => _wireHost;
|
||||
|
||||
/// <summary>Instance Actors created on this site.</summary>
|
||||
public IReadOnlyList<IActorRef> InstanceActors => _instanceActors;
|
||||
|
||||
@@ -344,12 +355,63 @@ public sealed class SiteRuntimeFixture : IAsyncDisposable
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Starts the site's real Kestrel/gRPC listener and attaches
|
||||
/// <paramref name="count"/> subscribers through the production
|
||||
/// <see cref="SiteStreamGrpcClient"/> — one channel per site with the streams
|
||||
/// multiplexed on it, exactly the production shape (central keys one client per
|
||||
/// <c>(siteId, endpoint)</c>). Instance selection uses the same stride as
|
||||
/// <see cref="AttachStreamProbes"/>, so the wire probes watch the SAME instances
|
||||
/// as the in-process probes and the two latency distributions are directly
|
||||
/// comparable within one run.
|
||||
/// </summary>
|
||||
/// <param name="count">Wire subscribers to attach.</param>
|
||||
/// <param name="latency">Histogram that receives end-to-end (over-the-wire) latencies.</param>
|
||||
/// <param name="log">Progress sink for fault reporting.</param>
|
||||
/// <returns>
|
||||
/// A task that completes when the listener is up and every stream has been
|
||||
/// initiated. Stream ESTABLISHMENT finishes asynchronously — comfortably inside
|
||||
/// the subscribe-settle + settle windows that separate attach from measurement.
|
||||
/// </returns>
|
||||
public async Task AttachWireStreamProbesAsync(int count, LatencyHistogram latency, Action<string> log)
|
||||
{
|
||||
_wireHost = await SiteWireStreamHost.StartAsync(this);
|
||||
_wireClient = new SiteStreamGrpcClient(_wireHost.Endpoint, NullLogger.Instance);
|
||||
_wireCts = new CancellationTokenSource();
|
||||
|
||||
var stride = Math.Max(1, _config.InstancesPerSite / Math.Max(1, count));
|
||||
for (var p = 0; p < count; p++)
|
||||
{
|
||||
var instanceIndex = Math.Min(p * stride, _config.InstancesPerSite - 1);
|
||||
var subscriber = WireStreamSubscriber.Attach(
|
||||
_wireClient, InstanceName(instanceIndex), $"wire-{SiteId}-p{p:D2}",
|
||||
latency, log, _wireCts.Token);
|
||||
_wireSubscribers.Add(subscriber);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Total tag paths registered across this site's connections.</summary>
|
||||
public int TotalTagPaths => TagPathsByConnection.Sum(list => list.Count);
|
||||
|
||||
/// <inheritdoc />
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
// Wire path first: cancel the client streams, wait for their loops to exit,
|
||||
// then drop the channel and the listener — the same order production teardown
|
||||
// takes (client goes away, server cancels, actor system last).
|
||||
if (_wireCts is not null)
|
||||
{
|
||||
await _wireCts.CancelAsync();
|
||||
foreach (var subscriber in _wireSubscribers)
|
||||
await subscriber.WaitForShutdownAsync();
|
||||
_wireCts.Dispose();
|
||||
}
|
||||
|
||||
if (_wireClient is not null)
|
||||
await _wireClient.DisposeAsync();
|
||||
if (_wireHost is not null)
|
||||
await _wireHost.DisposeAsync();
|
||||
|
||||
foreach (var probe in _probes)
|
||||
await probe.DisposeAsync();
|
||||
|
||||
|
||||
@@ -27,6 +27,14 @@
|
||||
<ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<!-- Wire-path probes: each site hosts the production SiteStreamGrpcServer on a
|
||||
real Kestrel h2c listener so the R2 batcher + serialization + HTTP/2 flow
|
||||
control sit inside the measured path (see Probes/SiteWireStreamHost.cs). -->
|
||||
<FrameworkReference Include="Microsoft.AspNetCore.App" />
|
||||
<PackageReference Include="Grpc.AspNetCore" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Commons/ZB.MOM.WW.ScadaBridge.Commons.csproj" />
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ZB.MOM.WW.ScadaBridge.SiteRuntime.csproj" />
|
||||
|
||||
+13
@@ -49,6 +49,7 @@ public class TargetScaleHarnessSmokeTests
|
||||
DebugProbeInterval = TimeSpan.FromSeconds(2),
|
||||
SubscribeSettleDuration = TimeSpan.FromSeconds(5),
|
||||
StreamProbesPerSite = 3,
|
||||
WireProbesPerSite = 2,
|
||||
StoreAndForwardDrainMessages = 200,
|
||||
SlowSubscriberEvents = 2_000,
|
||||
ResultsPath = Path.Combine(Path.GetTempPath(), $"loadharness-smoke-{Guid.NewGuid():N}.json"),
|
||||
@@ -64,6 +65,18 @@ public class TargetScaleHarnessSmokeTests
|
||||
Assert.True(result.StreamProbeReceived > 0, "Live stream subscribers received nothing.");
|
||||
Assert.Equal(0, result.DriverSkippedNoCallback);
|
||||
|
||||
// The wire path (real Kestrel + real SiteStreamGrpcClient, R2 batching
|
||||
// negotiated) carried traffic end to end: events crossed a real HTTP/2
|
||||
// socket, the batch-size histogram observed real frames, and nothing faulted.
|
||||
Assert.NotNull(result.WireStream);
|
||||
Assert.True(result.WireStream!.EventsReceived > 0,
|
||||
"Wire subscribers received nothing — the Kestrel/gRPC hop did not carry traffic.");
|
||||
Assert.True(result.WireStream.Latency.Count > 0,
|
||||
"No wire latency samples — the over-the-wire measurement path is broken.");
|
||||
Assert.True(result.WireStream.Batches.Frames > 0,
|
||||
"The batch-size MeterListener observed no frames — the R2 histogram path is broken.");
|
||||
Assert.Equal(0, result.WireStream.StreamFaults);
|
||||
|
||||
// Observability held up.
|
||||
Assert.Equal(config.Sites, result.SitesTrackedByAggregator);
|
||||
Assert.True(result.DebugSnapshotsCompleted > 0, "No debug snapshot completed.");
|
||||
|
||||
Reference in New Issue
Block a user