test(loadharness)+docs: wire-path stream probes — R2 batching benchmark-proven at target scale

The WP-4 runs measured the stream path only up to the bounded send channel;
the probe deliberately substituted the socket hop with a controllable reader,
which is exactly where the R2 SiteStreamEventBatcher lives — so batching had
never been in any benchmark's hot path. The harness now also attaches per-site
wire probes: a real Kestrel h2c listener hosting the production
SiteStreamGrpcServer, dialed by the production SiteStreamGrpcClient with
batching negotiated, batch sizes read from the production batch_size histogram
via a MeterListener (--wire-probes-per-site, default 10; 0 restores the old
shape). Full-scale run (10x500x75, 20-min window, results doc §8 + raw JSON):
899,807 events in 12,462 frames (mean 72.2 ev/frame, p50=p99=75), wire P99
43.4 ms vs the 250 ms budget — every batch pays the full 25 ms linger since a
75-event burst never hits the 100 cap — zero drops, zero faults, offered load
100.0% of nominal with fan-out doubled. CI smoke Fact now asserts the wire
path carries traffic and observes frames.

CLAUDE.md

@@ -99,7 +99,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
 - **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here.
 - Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role.
 - **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-<siteId>}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`).
-- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror, then 24 for `AckTime`). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle, automated by `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` (it always restores the csproj; an active `<Protobuf>` item must never be committed — it breaks the Docker build). **Stream events are BATCHED as of R2 (2026-08-15)** — a site emitting ~37.5k events/s used to pay one gRPC message per event. Additive wire shape, no new RPC: `batching_supported` on the request (`InstanceStreamRequest` field 3 / `SiteStreamRequest` field 2) plus a `SiteStreamEvent.batch` oneof case (**field 4**) carrying a new `SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }`. **The proto3 default of that flag IS the negotiation, and it is load-bearing** — a batch frame reaches a pre-R2 central as `EventOneofCase.None`, whose `ConvertToDomainEvent` returns null, so the whole batch would vanish with no error anywhere; an old central cannot set the flag, so it never gets one, and an old site ignores the unknown request field and keeps sending per-event frames the new client's `ForEachEvent` unpack handles as the single-event case. Server side is `SiteStreamEventBatcher`, a per-subscriber pump **downstream of `StreamRelayActor`'s bounded DropOldest channel** — so it changes framing only and **does NOT move the burst ceiling** (deferred register row 31; that ceiling lives in the shared publish stage upstream of the BroadcastHub). It never delays a lone event: it drains the already-queued backlog for free and lingers only once a backlog is proven, then emits a single-event buffer as a plain frame. `GrpcStreamBatchMaxEvents` (100, `1` disables) and `GrpcStreamBatchWindow` (25 ms, validated **strictly under 250 ms** = the load test's end-to-end P99 threshold); measured worst case P99 25.4 ms. Histogram `scadabridge.site.stream.batch_size`, recorded only on negotiated streams (`ScadaBridgeTelemetry.MeterName` is already in the `ObservedMeters` allowlist). **The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit `34a3f4bb`)** — Kestrel's `MaxStreamLifetime`/`Grpc.AspNetCore.Server` max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and `IsLive` reflects the gap in between.
+- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror, then 24 for `AckTime`). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `<Protobuf>` include commented out — regeneration is a manual toggle-build-copy-untoggle, automated by `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` (it always restores the csproj; an active `<Protobuf>` item must never be committed — it breaks the Docker build). **Stream events are BATCHED as of R2 (2026-08-15)** — a site emitting ~37.5k events/s used to pay one gRPC message per event. Additive wire shape, no new RPC: `batching_supported` on the request (`InstanceStreamRequest` field 3 / `SiteStreamRequest` field 2) plus a `SiteStreamEvent.batch` oneof case (**field 4**) carrying a new `SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }`. **The proto3 default of that flag IS the negotiation, and it is load-bearing** — a batch frame reaches a pre-R2 central as `EventOneofCase.None`, whose `ConvertToDomainEvent` returns null, so the whole batch would vanish with no error anywhere; an old central cannot set the flag, so it never gets one, and an old site ignores the unknown request field and keeps sending per-event frames the new client's `ForEachEvent` unpack handles as the single-event case. Server side is `SiteStreamEventBatcher`, a per-subscriber pump **downstream of `StreamRelayActor`'s bounded DropOldest channel** — so it changes framing only and **does NOT move the burst ceiling** (deferred register row 31; that ceiling lives in the shared publish stage upstream of the BroadcastHub). It never delays a lone event: it drains the already-queued backlog for free and lingers only once a backlog is proven, then emits a single-event buffer as a plain frame. `GrpcStreamBatchMaxEvents` (100, `1` disables) and `GrpcStreamBatchWindow` (25 ms, validated **strictly under 250 ms** = the load test's end-to-end P99 threshold); measured worst case P99 25.4 ms. **Benchmark-proven at target scale 2026-08-18** (the LoadHarness wire path — real Kestrel h2c + real `SiteStreamGrpcClient`, `--wire-probes-per-site`): 899,807 events in 12,462 frames (mean 72.2 ev/frame), wire P99 43.4 ms (5.8× inside the 250 ms budget — a 75-event burst never hits the 100 cap, so every batch pays the full 25 ms window: this workload's worst-case latency posture), zero drops/faults; results doc §8. Histogram `scadabridge.site.stream.batch_size`, recorded only on negotiated streams (`ScadaBridgeTelemetry.MeterName` is already in the `ObservedMeters` allowlist). **The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit `34a3f4bb`)** — Kestrel's `MaxStreamLifetime`/`Grpc.AspNetCore.Server` max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and `IsLive` reflects the gap in between.
 - Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface.
   - **`AckTime` mirror enrichment + the `Alarms` script accessor (MES alarm-status API Phase 1, 2026-08-01).** `AlarmStateChanged` carries an additive `AckTime` (`DateTimeOffset?`), mirrored on the vendored `AlarmStateUpdate` proto as **field 24** and persisted inside `native_alarm_state`'s `metadata_json` — deliberately NOT a new column, because that table is `RegisterReplicated` and LocalDb builds its CDC triggers from the column list at registration time. Set only while a condition is active AND acknowledged (so it is null while unacked and cleared on re-raise); the DCL stamps the source's own ack instant for OPC UA (new SelectClause **index 18** = `AckedState/TransitionTime`) and its observation time of the ack transition for MxGateway, which supplies none. Site `Call` scripts read alarms via the new **`Alarms.CurrentAsync()`** accessor (`ScriptRuntimeContext` + `ScriptGlobals`, local Ask on `GetAlarmSnapshotRequest`, returns `Commons.Types.Scripts.ScriptAlarm`), mirrored on `ScriptCompileSurface` AND the Central UI `SandboxScriptHost` editor surface. The trust model needed no change — it is a deny-list over API roots, not an allow-list of context members. Plan: `docs/plans/2026-06-30-mes-alarm-status-api.md` (Phases 2–4 are deployed config, not repo).
 - OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover.

docs/plans/2026-08-15-target-scale-load-test-design.md

@@ -191,7 +191,7 @@ Each is a deliberate, recorded trade — not an omission.
 | D1 | **Sustained window shortened from 1 hour to 20 minutes** at full scale | Practicality on a single shared workstation. Memory growth is reported as a *slope* precisely so a shorter window still answers the leak question: a leak shows as a positive slope in 20 min just as in 60. The 1-hour run is a single flag — `--sustain-minutes 60` — and the exact command is recorded in the results doc. |
 | D2 | **In-process, non-clustered sites** rather than 10 real two-node Akka clusters | A single box cannot host 20 clustered nodes at this scale. Cluster membership/failover is already measured on a real two-node rig (`FailoverTimingTests`, `docker/failover-drill.sh`); WP-4's question is about the hierarchy under the singleton. |
 | D3 | **Simulated data source** instead of real OPC UA | No simulated adapter exists in the DCL, and 375k real monitored items would measure the OPC UA SDK. Injected at the documented `RegisterAdapter` seam, at the exact hand-off point the real adapter uses. |
-| D4 | **gRPC socket replaced by a reader task** on the stream path | Required by the row-50 measurement itself: a stalled subscriber must be *held* stalled. All of `SiteStreamGrpcServer`'s per-subscription machinery above the socket is real. |
+| D4 | **gRPC socket replaced by a reader task** on the stream path | Required by the row-50 measurement itself: a stalled subscriber must be *held* stalled. All of `SiteStreamGrpcServer`'s per-subscription machinery above the socket is real. **Partially retired 2026-08-18:** the harness now ALSO attaches per-site wire probes — real Kestrel h2c + real `SiteStreamGrpcClient` with R2 batching negotiated (`--wire-probes-per-site`, default 10) — so the batcher/serialization/HTTP/2 hop is measured too; the substituted-reader probes remain because the slow-subscriber scenario still needs them. Results doc §8. |
 | D5 | **Instance Actors are `/user` children**, not children of a `DeploymentManagerActor` | Constructing them directly is what lets the ramp be timed in isolation and keeps the harness independent of the deploy round-trip. Visible only as `InstanceActorInitialized` dead letters, which the harness suppresses. No measured path differs. |
 | D6 | **Akka logging at WARNING** | At 37,500 updates/s, INFO output would itself become a measured load. |
 | D7 | `[xc-3]` site event logging volume, `[xc-4]` audit-log degradation, `[xc-5]` template flattening, `[xc-8]` UI responsiveness **not covered** | Out of scope for this harness: the first three need central MS SQL and the Template Engine (a central-cluster fixture, not a site one), and `[xc-8]` needs a browser. Recorded as residual scope in the results doc rather than silently dropped. |

docs/plans/2026-08-15-target-scale-load-test-results.md

@@ -275,7 +275,10 @@ Stated plainly so the evidence is not over-read:
 - **Not a real-network run.** The gRPC, health-transport and S&F-to-central hops are
   in-process. Serialization cost, TLS, WAN latency and socket backpressure are out of
   frame; the *stream* path's backpressure was measured deliberately by substituting a
-  controllable reader for the socket (D4).
+  controllable reader for the socket (D4). **Partially superseded by §8 (2026-08-18):**
+  the site→central stream hop now also runs over a real loopback HTTP/2 socket with
+  the R2 batcher, protobuf serialization and Kestrel in the measured path. TLS and
+  WAN latency remain out of frame.
 - **Not a real OPC UA run.** 375,000 monitored items were simulated at the adapter
   callback (D3). The DCL above that callback is real and fully exercised.
 - **Not 1 hour.** 20 minutes (D1). See finding F1 for the one question this leaves
@@ -323,6 +326,111 @@ dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests \
 `--help`-style knobs: `--sites`, `--instances-per-site`, `--tags-per-instance`,
 `--tag-update-period-seconds`, `--settle-minutes`, `--sustain-minutes`,
 `--sample-seconds`, `--health-interval-seconds`, `--debug-probe-interval-seconds`,
-`--subscribe-settle-seconds`, `--stream-probes-per-site`, `--sf-drain-messages`,
+`--subscribe-settle-seconds`, `--stream-probes-per-site`, `--wire-probes-per-site`,
-`--slow-subscriber-events`, `--data-dir`, `--results`. An unknown key is rejected
+`--sf-drain-messages`, `--slow-subscriber-events`, `--data-dir`, `--results`. An
-rather than ignored, so a typo cannot silently change the measured scale.
+unknown key is rejected rather than ignored, so a typo cannot silently change the
+measured scale.
+
+Since 2026-08-18 the default protocol also attaches 10 **wire** stream subscribers
+per site (real Kestrel + real `SiteStreamGrpcClient`, §8); `--wire-probes-per-site 0`
+reproduces the original in-process-only shape.
+
+---
+
+## 8. Wire-path run — R2 stream batching in the hot path (2026-08-18)
+
+**Raw metrics:** `docs/plans/2026-08-18-loadharness-results-wire.json`
+
+### 8.1 Why this run exists
+
+Both prior runs (§2 and the §4 F1 1-hour sustain) measured the stream path only up
+to the bounded send channel: `StreamSubscriberProbe` deliberately substitutes the
+final socket hop with a controllable reader (deviation D4 — required by the row-50
+slow-subscriber measurement). The R2 gRPC event batcher (`SiteStreamEventBatcher`,
+merged `c75c79f8` on 2026-08-15, about an hour *after* the §2 run) lives exactly in
+that substituted hop — so no benchmark run had ever exercised it, only R2's own
+component-level measurement (worst-case P99 25.4 ms) and trickle-rate live traffic
+on the rig (register row 48).
+
+This run adds a **wire path** to the harness without touching the existing probes:
+each site hosts the production `SiteStreamGrpcServer` on a real Kestrel HTTP/2 (h2c)
+loopback listener (production `CommunicationOptions` defaults — batch cap 100,
+window 25 ms, channel capacity 1000), and the production `SiteStreamGrpcClient`
+dials in with `BatchingSupported = true`, one channel per site with 10 streams
+multiplexed on it — the exact production shape and the exact production code on both
+ends. Latency is recorded in the client callback, i.e. after protobuf
+deserialization, batch unpacking and domain mapping. The batch-size figures are the
+production `scadabridge.site.stream.batch_size` histogram itself, captured by a
+`MeterListener`. Deliberately not in the path: `ControlPlaneAuthInterceptor`
+(authenticates once per stream lifetime, nothing per-event) and TLS (production is
+h2c too).
+
+The 100 wire subscribers are attached **in addition to** the 100 in-process probes,
+watching the same instances (same stride), so one run yields both distributions and
+their difference isolates the wire hop's cost. Total hub fan-out is therefore
+double that of the §2 run.
+
+### 8.2 Environment
+
+Identical posture to §1: same host (M4 Pro, 14 cores, 48 GB), .NET 10.0.5, server
+GC, Release; the 8-node `docker/` rig stayed running throughout (D8); 20-minute
+window after a 2-minute settle; harness at the state committed with this section.
+
+### 8.3 Measured
+
+| | in-process probes (§2 path) | **wire probes (new)** |
+|---|---|---|
+| Subscribers | 100 | 100 |
+| Events delivered (window) | 899,807 | **899,807** |
+| Events dropped / faults | 0 / — | **0 / 0** |
+| Latency mean | 1.79 ms | **28.00 ms** |
+| Latency P50 | 1.36 ms | **26.97 ms** |
+| Latency P95 | 4.19 ms | **32.07 ms** |
+| Latency P99 | 6.74 ms | **43.44 ms** |
+| Latency P99.9 / max | 25.83 / 32.17 ms | **53.94 / 62.42 ms** |
+
+Batching (production histogram, window deltas):
+
+```
+events 899,807  frames 12,462          → mean 72.20 events/frame  (72x frame reduction)
+batch size p50/p95/p99/max = 75/75/75/75   batched frames 11,998   single-event frames 464
+server channel drops 0    stream faults 0    graceful completions 0
+```
+
+Offered load held at **37,497/s = 100.0% of nominal** with the doubled fan-out
+(driver overrun 0.0 s, skipped 0). Whole-run CPU mean **66% of one core (4.7% of
+the box)** vs 41% in §2 — the added ~25% of one core buys 10 Kestrel listeners, 100
+live HTTP/2 streams, protobuf serialization of ~900 k events and the client-side
+unpack. Working-set slope **−116 MB/min** (a large mid-window release; peak
+3,413 MB), heap sawtooth with 5 gen-2 collects and endpoints 1,876→1,943 MB —
+consistent with the §4 F1 resolution (row 29: not a leak).
+
+### 8.4 Reading the numbers
+
+- **The batcher engaged exactly as designed, at its natural batch size.** The
+  driver emits each instance's 75 tags contiguously within a slice, so a subscriber
+  receives one 75-event burst per 10 s period. 75 < the 100-event cap, so the
+  size cap never closes a batch early — nearly every burst becomes exactly one
+  75-event frame (p50 = p99 = max = 75). The 464 single-event frames are burst
+  stragglers that arrived after their batch's window closed.
+- **The ~26 ms wire P50 is dominated by the deliberate linger, and that is the
+  documented trade.** Because bursts never hit the cap, every batch pays the full
+  25 ms `GrpcStreamBatchWindow` — this workload is the batcher's *worst-case latency
+  posture and best-case framing win* simultaneously. Wire P50 26.97 ms ≈ in-process
+  P50 1.36 ms + the 25 ms window + sub-millisecond socket/serialize cost. Against
+  the 250 ms P99 threshold the wire path passes with a **5.8× margin** (43.44 ms),
+  and the `CommunicationOptionsValidator` bound (window strictly < 250 ms) is
+  confirmed to compose: end-to-end latency stayed threshold-compliant with the
+  window fully paid on every frame.
+- **Framing efficiency at scale is now measured, not extrapolated:** 899,807 events
+  crossed the wire in 12,462 frames. Pre-R2, the same window would have cost
+  899,807 frames.
+- **Nothing degraded elsewhere.** The in-process path under doubled fan-out reads
+  P99 6.74 ms vs 4.57 ms in §2 — same order, still a 37× margin — and every §2
+  pass criterion that this run re-exercises (offered load, zero drops at healthy
+  subscribers, health reports, debug snapshots, S&F drain, slow-subscriber
+  isolation) repeated its pass.
+
+**Verdict: PASS.** R2's negotiated batching is now benchmark-proven at target scale
+over a real socket: 72× fewer frames, zero loss, zero faults, and worst-case-posture
+latency 5.8× inside the budget.

tests/ZB.MOM.WW.ScadaBridge.LoadHarness/HarnessConfig.cs

@@ -68,6 +68,19 @@ public sealed record HarnessConfig
     /// </summary>
     public int StreamProbesPerSite { get; init; } = 10;
 
+    /// <summary>
+    /// Instances per site additionally watched over a REAL gRPC hop: a per-site
+    /// Kestrel h2c listener hosts the production <c>SiteStreamGrpcServer</c> and the
+    /// production <c>SiteStreamGrpcClient</c> dials in with batching negotiated, so
+    /// the R2 <c>SiteStreamEventBatcher</c>, protobuf serialization and HTTP/2 flow
+    /// control are inside the measured path (they are NOT for
+    /// <see cref="StreamProbesPerSite"/>, whose final hop is deliberately an
+    /// in-process reader). 0 disables the wire path entirely. Like the in-process
+    /// probes, each wire subscriber's stream graph sees the full site event flow and
+    /// filters it, so this adds to the fan-out multiplier.
+    /// </summary>
+    public int WireProbesPerSite { get; init; } = 10;
+
     /// <summary>Store-and-forward messages enqueued for the drain-rate measurement (register row 50).</summary>
     public int StoreAndForwardDrainMessages { get; init; } = 20_000;
 
@@ -131,6 +144,7 @@ public sealed record HarnessConfig
                 "health-interval-seconds" => config with { HealthReportInterval = TimeSpan.FromSeconds(double.Parse(value)) },
                 "debug-probe-interval-seconds" => config with { DebugProbeInterval = TimeSpan.FromSeconds(double.Parse(value)) },
                 "stream-probes-per-site" => config with { StreamProbesPerSite = int.Parse(value) },
+                "wire-probes-per-site" => config with { WireProbesPerSite = int.Parse(value) },
                 "sf-drain-messages" => config with { StoreAndForwardDrainMessages = int.Parse(value) },
                 "slow-subscriber-events" => config with { SlowSubscriberEvents = int.Parse(value) },
                 "data-dir" => config with { DataDirectory = value },

tests/ZB.MOM.WW.ScadaBridge.LoadHarness/HarnessRun.cs

@@ -40,6 +40,10 @@ namespace ZB.MOM.WW.ScadaBridge.LoadHarness;
 /// <param name="StreamProbeDropped">Events evicted by live subscribers' bounded channels.</param>
 /// <param name="StoreAndForwardDrain">Store-and-forward drain measurement (register row 50).</param>
 /// <param name="SlowSubscriber">Slow-subscriber isolation measurement (register row 50).</param>
+/// <param name="WireStream">
+/// Wire-path stream measurement — real Kestrel + real <c>SiteStreamGrpcClient</c> with
+/// R2 batching negotiated — or null when <see cref="HarnessConfig.WireProbesPerSite"/> is 0.
+/// </param>
 public sealed record HarnessRunResult(
     HarnessConfig Config,
     EnvironmentInfo Environment,
@@ -67,7 +71,30 @@ public sealed record HarnessRunResult(
     long StreamProbeReceived,
     long StreamProbeDropped,
     StoreAndForwardDrainResult? StoreAndForwardDrain,
-    SlowSubscriberResult? SlowSubscriber);
+    SlowSubscriberResult? SlowSubscriber,
+    WireStreamResult? WireStream);
+
+/// <summary>
+/// Steady-state-window measurement of the wire-path stream probes: the production
+/// gRPC server (Kestrel h2c) and client with the R2 batcher, protobuf serialization
+/// and HTTP/2 flow control INSIDE the measured path. All figures are deltas over the
+/// steady-state window only.
+/// </summary>
+/// <param name="ProbeCount">Wire subscribers attached across the fleet.</param>
+/// <param name="Latency">End-to-end DCL-boundary → post-unpack client callback latency.</param>
+/// <param name="EventsReceived">Domain events delivered by the client in the window.</param>
+/// <param name="ServerDroppedEvents">Events evicted by the wire servers' bounded send channels in the window.</param>
+/// <param name="StreamFaults">Stream faults reported by the client (expected 0).</param>
+/// <param name="GracefulCompletions">Graceful server-side stream completions (expected 0 inside the 4 h lifetime).</param>
+/// <param name="Batches">The frame/batch-size distribution the R2 batcher actually produced.</param>
+public sealed record WireStreamResult(
+    int ProbeCount,
+    LatencySnapshot Latency,
+    long EventsReceived,
+    long ServerDroppedEvents,
+    long StreamFaults,
+    long GracefulCompletions,
+    BatchSizeWindow Batches);
 
 /// <summary>Host and runtime facts recorded alongside the numbers.</summary>
 /// <param name="MachineName">Host name.</param>
@@ -120,6 +147,7 @@ public static class HarnessRun
         ObservabilityProbes? probes = null;
         StoreAndForwardDrainResult? drainResult = null;
         SlowSubscriberResult? slowResult = null;
+        BatchSizeAggregator? batchAggregator = null;
 
         var aggregator = new CentralHealthAggregator(
             Options.Create(new HealthMonitoringOptions
@@ -154,6 +182,18 @@ public static class HarnessRun
                 site.AttachStreamProbes(latency);
             log($"  {sites.Sum(s => s.Probes.Count)} live stream subscribers attached");
 
+            // Wire path (real gRPC): the aggregator must exist BEFORE the first stream
+            // opens or the MeterListener misses early frames.
+            var wireLatency = new LatencyHistogram();
+            if (config.WireProbesPerSite > 0)
+            {
+                batchAggregator = new BatchSizeAggregator();
+                foreach (var site in sites)
+                    await site.AttachWireStreamProbesAsync(config.WireProbesPerSite, wireLatency, log);
+                log($"  {sites.Sum(s => s.WireSubscribers.Count)} wire stream subscribers attached " +
+                    "(real Kestrel + SiteStreamGrpcClient, batching negotiated)");
+            }
+
             // Let Instance Actors complete their DCL subscribe round-trips before the
             // driver starts; an emit before SubscribeBatchAsync has captured the
             // callback would be silently discarded.
@@ -184,6 +224,18 @@ public static class HarnessRun
                     probe.RetargetLatency(steadyLatency);
             }
 
+            // Wire path: same ramp/steady histogram separation, plus window-start
+            // baselines for the client/server counters and the batch distribution.
+            var steadyWireLatency = new LatencyHistogram();
+            var wireReceivedAtWindowStart = sites.Sum(s => s.WireSubscribers.Sum(p => p.ReceivedEvents));
+            var wireDroppedAtWindowStart = sites.Sum(s => s.WireHost?.Server.DroppedStreamEventCount ?? 0);
+            var batchesAtWindowStart = batchAggregator?.Snapshot();
+            foreach (var site in sites)
+            {
+                foreach (var wire in site.WireSubscribers)
+                    wire.RetargetLatency(steadyWireLatency);
+            }
+
             var windowWatch = Stopwatch.StartNew();
             log($"Sustained measurement window: {config.SustainDuration.TotalMinutes:F1} min...");
             await Task.Delay(config.SustainDuration, cancellationToken);
@@ -199,6 +251,24 @@ public static class HarnessRun
             var probeReceived = sites.Sum(s => s.Probes.Sum(p => p.ReceivedEvents)) - receivedAtWindowStart;
             var probeDropped = sites.Sum(s => s.Probes.Sum(p => p.DroppedEvents)) - droppedAtWindowStart;
 
+            // Wire-path window deltas, captured BEFORE the row-50 scenarios below so
+            // their publishing cannot contaminate the wire figures.
+            WireStreamResult? wireResult = null;
+            if (batchAggregator is not null)
+            {
+                wireResult = new WireStreamResult(
+                    ProbeCount: sites.Sum(s => s.WireSubscribers.Count),
+                    Latency: steadyWireLatency.Snapshot(),
+                    EventsReceived: sites.Sum(s => s.WireSubscribers.Sum(p => p.ReceivedEvents)) - wireReceivedAtWindowStart,
+                    ServerDroppedEvents: sites.Sum(s => s.WireHost?.Server.DroppedStreamEventCount ?? 0) - wireDroppedAtWindowStart,
+                    StreamFaults: sites.Sum(s => s.WireSubscribers.Sum(p => p.Faults)),
+                    GracefulCompletions: sites.Sum(s => s.WireSubscribers.Sum(p => p.GracefulCompletions)),
+                    Batches: BatchSizeWindow.From(batchesAtWindowStart!, batchAggregator.Snapshot()));
+                log($"  wire path: {wireResult.EventsReceived:N0} events in {wireResult.Batches.Frames:N0} frames " +
+                    $"(mean {wireResult.Batches.MeanEventsPerFrame:F1} ev/frame, max {wireResult.Batches.Max}), " +
+                    $"P99 {wireResult.Latency.P99Ms:F2}ms");
+            }
+
             // Register row 50 — measured after the sustained window so the numbers are
             // not competing with the full tag load for CPU, and reported separately for
             // the same reason.
@@ -250,12 +320,14 @@ public static class HarnessRun
                 StreamProbeReceived: probeReceived,
                 StreamProbeDropped: probeDropped,
                 StoreAndForwardDrain: drainResult,
-                SlowSubscriber: slowResult);
+                SlowSubscriber: slowResult,
+                WireStream: wireResult);
         }
         finally
         {
             if (driver != null) await driver.DisposeAsync();
             if (probes != null) await probes.DisposeAsync();
+            batchAggregator?.Dispose();
             await sampler.DisposeAsync();
 
             log("Tearing down sites...");

tests/ZB.MOM.WW.ScadaBridge.LoadHarness/Metrics/BatchSizeAggregator.cs

@@ -0,0 +1,136 @@
+using System.Diagnostics.Metrics;
+using ZB.MOM.WW.ScadaBridge.Commons.Observability;
+
+namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
+
+/// <summary>
+/// Captures the production <c>scadabridge.site.stream.batch_size</c> histogram via a
+/// <see cref="MeterListener"/>, so a wire-path run can report the actual frame/batch
+/// distribution the R2 batcher produced — not a client-side reconstruction. The
+/// instrument is recorded once per emitted gRPC frame (negotiated streams only), with
+/// the frame's event count as the value, which makes <c>count = frames</c> and
+/// <c>sum = events</c> exactly.
+///
+/// <para>
+/// The harness hosts every site's <see cref="ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer"/>
+/// in-process, so this listener sees every wire frame the run emits and nothing else —
+/// no other component in the harness records this instrument.
+/// </para>
+/// </summary>
+public sealed class BatchSizeAggregator : IDisposable
+{
+    /// <summary>
+    /// Sizes above this are clamped into the last bucket. The production cap is
+    /// <c>GrpcStreamBatchMaxEvents</c> = 100, so 512 leaves generous headroom for a
+    /// configured-up run without an unbounded array.
+    /// </summary>
+    public const int MaxTrackedBatchSize = 512;
+
+    private readonly MeterListener _listener;
+    private readonly long[] _framesBySize = new long[MaxTrackedBatchSize + 1];
+
+    /// <summary>Starts listening. Must be created BEFORE the first stream opens.</summary>
+    public BatchSizeAggregator()
+    {
+        _listener = new MeterListener
+        {
+            InstrumentPublished = (instrument, listener) =>
+            {
+                if (instrument.Meter.Name == ScadaBridgeTelemetry.MeterName &&
+                    instrument.Name == "scadabridge.site.stream.batch_size")
+                {
+                    listener.EnableMeasurementEvents(instrument);
+                }
+            },
+        };
+        _listener.SetMeasurementEventCallback<int>((_, value, _, _) =>
+        {
+            var size = Math.Clamp(value, 1, MaxTrackedBatchSize);
+            Interlocked.Increment(ref _framesBySize[size]);
+        });
+        _listener.Start();
+    }
+
+    /// <summary>Copies the cumulative per-size frame counts.</summary>
+    /// <returns>Index = events-per-frame (clamped), value = frames emitted at that size.</returns>
+    public long[] Snapshot()
+    {
+        var copy = new long[_framesBySize.Length];
+        for (var i = 0; i < copy.Length; i++)
+            copy[i] = Interlocked.Read(ref _framesBySize[i]);
+        return copy;
+    }
+
+    /// <inheritdoc />
+    public void Dispose() => _listener.Dispose();
+}
+
+/// <summary>
+/// Batch-size distribution over one measurement window, computed from two
+/// <see cref="BatchSizeAggregator.Snapshot"/>s. Percentiles are exact (integer sizes,
+/// full counts — nothing is bucketed away).
+/// </summary>
+/// <param name="Frames">gRPC frames written in the window.</param>
+/// <param name="Events">Events those frames carried.</param>
+/// <param name="BatchedFrames">Frames that carried more than one event.</param>
+/// <param name="MeanEventsPerFrame">Events / Frames.</param>
+/// <param name="P50">Median events-per-frame.</param>
+/// <param name="P95">95th percentile events-per-frame.</param>
+/// <param name="P99">99th percentile events-per-frame.</param>
+/// <param name="Max">Largest frame observed in the window.</param>
+public sealed record BatchSizeWindow(
+    long Frames,
+    long Events,
+    long BatchedFrames,
+    double MeanEventsPerFrame,
+    int P50,
+    int P95,
+    int P99,
+    int Max)
+{
+    /// <summary>Computes the window distribution from cumulative start/end snapshots.</summary>
+    /// <param name="start">Snapshot at window start.</param>
+    /// <param name="end">Snapshot at window end.</param>
+    /// <returns>The distribution of frames emitted between the two snapshots.</returns>
+    public static BatchSizeWindow From(long[] start, long[] end)
+    {
+        long frames = 0, events = 0, batched = 0;
+        var max = 0;
+        var delta = new long[end.Length];
+        for (var size = 1; size < end.Length; size++)
+        {
+            var n = end[size] - start[size];
+            if (n <= 0) continue;
+            delta[size] = n;
+            frames += n;
+            events += n * size;
+            if (size > 1) batched += n;
+            max = size;
+        }
+
+        return new BatchSizeWindow(
+            frames, events, batched,
+            frames == 0 ? 0 : events / (double)frames,
+            Percentile(delta, frames, 50),
+            Percentile(delta, frames, 95),
+            Percentile(delta, frames, 99),
+            max);
+    }
+
+    private static int Percentile(long[] delta, long frames, double percentile)
+    {
+        if (frames == 0) return 0;
+        var target = (long)Math.Ceiling(frames * percentile / 100.0);
+        if (target < 1) target = 1;
+
+        long cumulative = 0;
+        for (var size = 1; size < delta.Length; size++)
+        {
+            cumulative += delta[size];
+            if (cumulative >= target)
+                return size;
+        }
+
+        return delta.Length - 1;
+    }
+}

tests/ZB.MOM.WW.ScadaBridge.LoadHarness/Probes/SiteWireStreamHost.cs

@@ -0,0 +1,96 @@
+using Microsoft.AspNetCore.Builder;
+using Microsoft.AspNetCore.Hosting;
+using Microsoft.AspNetCore.Hosting.Server;
+using Microsoft.AspNetCore.Hosting.Server.Features;
+using Microsoft.AspNetCore.Server.Kestrel.Core;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.ScadaBridge.Communication;
+using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
+
+namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Probes;
+
+/// <summary>
+/// The site half of the wire-path stream probe: a real Kestrel HTTP/2 (h2c) listener on
+/// a loopback ephemeral port, serving the PRODUCTION <see cref="SiteStreamGrpcServer"/>
+/// over the site's real <see cref="ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming.SiteStreamManager"/>.
+///
+/// <para>
+/// This is the hop <see cref="StreamSubscriberProbe"/> deliberately substitutes with a
+/// controllable in-process reader (which the slow-subscriber scenario needs), and
+/// therefore the hop the original WP-4 runs never exercised: the
+/// <c>SiteStreamEventBatcher</c> pump (R2), protobuf serialization,
+/// <c>IServerStreamWriter.WriteAsync</c>, and HTTP/2 flow control. The server is built
+/// through its production DI constructor with default <see cref="CommunicationOptions"/>,
+/// so the batch negotiation (100 events / 25 ms window), channel capacities and the
+/// concurrency cap are exactly what ships.
+/// </para>
+///
+/// <para>
+/// Deliberately NOT in the path: <c>ControlPlaneAuthInterceptor</c> (it lives in the
+/// Host project and authenticates once per CALL — one header compare per stream
+/// lifetime, nothing on the per-event path) and TLS (production is h2c too).
+/// </para>
+/// </summary>
+public sealed class SiteWireStreamHost : IAsyncDisposable
+{
+    private readonly WebApplication _app;
+
+    /// <summary>The hosted production gRPC server (drop counters, active stream count).</summary>
+    public SiteStreamGrpcServer Server { get; }
+
+    /// <summary>The dialable endpoint, e.g. <c>http://127.0.0.1:54321</c>.</summary>
+    public string Endpoint { get; }
+
+    private SiteWireStreamHost(WebApplication app, SiteStreamGrpcServer server, string endpoint)
+    {
+        _app = app;
+        Server = server;
+        Endpoint = endpoint;
+    }
+
+    /// <summary>
+    /// Builds, starts and readies the listener for one site.
+    /// </summary>
+    /// <param name="site">The site whose stream manager and actor system back the server.</param>
+    /// <returns>The started host.</returns>
+    public static async Task<SiteWireStreamHost> StartAsync(SiteRuntimeFixture site)
+    {
+        var server = new SiteStreamGrpcServer(
+            site.StreamManager,
+            Microsoft.Extensions.Logging.Abstractions.NullLogger<SiteStreamGrpcServer>.Instance,
+            Options.Create(new CommunicationOptions()));
+
+        var builder = WebApplication.CreateSlimBuilder();
+        builder.Logging.ClearProviders();
+        builder.WebHost.ConfigureKestrel(options =>
+            options.Listen(System.Net.IPAddress.Loopback, 0, listenOptions =>
+                listenOptions.Protocols = HttpProtocols.Http2));
+        builder.Services.AddGrpc();
+        builder.Services.AddSingleton(server);
+
+        var app = builder.Build();
+        app.MapGrpcService<SiteStreamGrpcServer>();
+        await app.StartAsync();
+
+        // Same post-start readiness handoff the production host performs.
+        server.SetReady(site.System);
+
+        var address = app.Services.GetRequiredService<IServer>()
+                          .Features.Get<IServerAddressesFeature>()?.Addresses.FirstOrDefault()
+                      ?? throw new InvalidOperationException("Kestrel bound no address.");
+
+        return new SiteWireStreamHost(app, server, address);
+    }
+
+    /// <inheritdoc />
+    public async ValueTask DisposeAsync()
+    {
+        // Mirrors the production shutdown ordering: cancel active streams first so
+        // their pumps observe cancellation, then stop the listener.
+        Server.CancelAllStreams();
+        await _app.StopAsync();
+        await _app.DisposeAsync();
+    }
+}

tests/ZB.MOM.WW.ScadaBridge.LoadHarness/Probes/WireStreamSubscriber.cs

@@ -0,0 +1,108 @@
+using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
+using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
+using ZB.MOM.WW.ScadaBridge.LoadHarness.Metrics;
+
+namespace ZB.MOM.WW.ScadaBridge.LoadHarness.Probes;
+
+/// <summary>
+/// The central half of the wire-path stream probe: one instance subscription opened
+/// through the PRODUCTION <see cref="SiteStreamGrpcClient"/> — the exact
+/// <c>SubscribeAsync</c> path central's <c>DebugStreamBridgeActor</c> uses, including
+/// the R2 batch negotiation (<c>BatchingSupported = true</c>) and the client-side
+/// batch unpack + proto→domain conversion.
+///
+/// <para>
+/// Latency is recorded in <c>onEvent</c>, i.e. AFTER wire deserialization, batch
+/// unpacking and domain mapping — the same "DCL boundary → subscriber" measurement as
+/// <see cref="StreamSubscriberProbe"/>, extended across the real socket. The emit
+/// instant travels verbatim on the proto <c>Timestamp</c>, so the subtraction stays a
+/// genuine end-to-end figure (same process, same clock).
+/// </para>
+/// </summary>
+public sealed class WireStreamSubscriber
+{
+    private Task _streamTask = Task.CompletedTask;
+    private volatile LatencyHistogram? _latency;
+    private long _received;
+    private long _faults;
+    private long _completions;
+
+    /// <summary>Probe name (also the stream's correlation id).</summary>
+    public string Name { get; }
+
+    /// <summary>Domain events delivered by the client callback.</summary>
+    public long ReceivedEvents => Interlocked.Read(ref _received);
+
+    /// <summary>Stream faults reported via <c>onError</c>. Expected 0 for a healthy run.</summary>
+    public long Faults => Interlocked.Read(ref _faults);
+
+    /// <summary>Graceful server-side completions via <c>onCompleted</c>. Expected 0 inside a run shorter than the 4 h stream lifetime.</summary>
+    public long GracefulCompletions => Interlocked.Read(ref _completions);
+
+    private WireStreamSubscriber(string name, LatencyHistogram? latency)
+    {
+        Name = name;
+        _latency = latency;
+    }
+
+    /// <summary>
+    /// Opens one instance stream on the shared per-site client. The subscription task
+    /// runs until <paramref name="ct"/> cancels — the harness deliberately does NOT
+    /// reconnect on fault (a fault inside a 25-minute loopback run is a finding, not
+    /// an operational event to paper over), it counts it.
+    /// </summary>
+    /// <param name="client">The site's shared production gRPC client (one channel per site, streams multiplexed).</param>
+    /// <param name="instanceUniqueName">Instance whose events this probe receives.</param>
+    /// <param name="name">Probe/correlation name (must be a valid Akka path element).</param>
+    /// <param name="latency">Histogram fed with end-to-end event latency.</param>
+    /// <param name="log">Progress sink for fault reporting.</param>
+    /// <param name="ct">Stops the subscription.</param>
+    /// <returns>The attached subscriber.</returns>
+    public static WireStreamSubscriber Attach(
+        SiteStreamGrpcClient client,
+        string instanceUniqueName,
+        string name,
+        LatencyHistogram? latency,
+        Action<string> log,
+        CancellationToken ct)
+    {
+        var self = new WireStreamSubscriber(name, latency);
+
+        self._streamTask = client.SubscribeAsync(
+            correlationId: name,
+            instanceUniqueName: instanceUniqueName,
+            onEvent: evt =>
+            {
+                Interlocked.Increment(ref self._received);
+                if (self._latency is { } histogram && evt is AttributeValueChanged changed)
+                    histogram.Record(DateTimeOffset.UtcNow - changed.Timestamp);
+            },
+            onError: ex =>
+            {
+                Interlocked.Increment(ref self._faults);
+                log($"  wire probe {name} stream fault: {ex.GetType().Name}: {ex.Message}");
+            },
+            onCompleted: () => Interlocked.Increment(ref self._completions),
+            ct);
+
+        return self;
+    }
+
+    /// <summary>Repoints the latency histogram (steady-state window separation) — mirrors <see cref="StreamSubscriberProbe.RetargetLatency"/>.</summary>
+    /// <param name="latency">The histogram to record into from now on, or null to stop recording.</param>
+    public void RetargetLatency(LatencyHistogram? latency) => _latency = latency;
+
+    /// <summary>Awaits the subscription task after its token has been cancelled.</summary>
+    /// <returns>A task that completes when the stream loop has exited.</returns>
+    public async Task WaitForShutdownAsync()
+    {
+        try
+        {
+            await _streamTask;
+        }
+        catch (OperationCanceledException)
+        {
+            // Normal teardown.
+        }
+    }
+}

tests/ZB.MOM.WW.ScadaBridge.LoadHarness/Program.cs

@@ -22,6 +22,7 @@ Console.WriteLine($"  sites                {config.Sites}");
 Console.WriteLine($"  instances/site       {config.InstancesPerSite}");
 Console.WriteLine($"  tags/instance        {config.TagsPerInstance}");
 Console.WriteLine($"  total subscriptions  {config.TotalSubscriptions:N0}");
+Console.WriteLine($"  stream probes/site   {config.StreamProbesPerSite} in-process + {config.WireProbesPerSite} wire (real gRPC)");
 Console.WriteLine($"  nominal update rate  {config.NominalUpdatesPerSecond:N0}/s");
 Console.WriteLine($"  settle / sustain     {config.SettleDuration.TotalMinutes:F1} / {config.SustainDuration.TotalMinutes:F1} min");
 Console.WriteLine();

tests/ZB.MOM.WW.ScadaBridge.LoadHarness/ResultsFormatter.cs

@@ -47,6 +47,18 @@ public static class ResultsFormatter
         builder.AppendLine(culture, $"events dropped          {result.StreamProbeDropped:N0}");
         builder.AppendLine();
 
+        if (result.WireStream is { } wire)
+        {
+            builder.AppendLine("-- wire stream subscribers (real gRPC, R2 batching in path) --");
+            builder.AppendLine(culture, $"probes                  {wire.ProbeCount} (Kestrel h2c + SiteStreamGrpcClient, batching negotiated)");
+            AppendLatency(builder, culture, wire.Latency);
+            builder.AppendLine(culture, $"events delivered        {wire.EventsReceived:N0} in {wire.Batches.Frames:N0} frames");
+            builder.AppendLine(culture, $"batching                mean {wire.Batches.MeanEventsPerFrame:F2} ev/frame  p50 {wire.Batches.P50}  p95 {wire.Batches.P95}  p99 {wire.Batches.P99}  max {wire.Batches.Max}  ({wire.Batches.BatchedFrames:N0} frames >1 event)");
+            builder.AppendLine(culture, $"server channel drops    {wire.ServerDroppedEvents:N0}");
+            builder.AppendLine(culture, $"faults / completions    {wire.StreamFaults:N0} / {wire.GracefulCompletions:N0}");
+            builder.AppendLine();
+        }
+
         builder.AppendLine("-- health report delivery --");
         AppendLatency(builder, culture, result.HealthReportLatency);
         builder.AppendLine(culture, $"reports ingested        {result.HealthReportsDelivered:N0}");

tests/ZB.MOM.WW.ScadaBridge.LoadHarness/SiteRuntimeFixture.cs

@@ -6,6 +6,7 @@ using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Logging.Abstractions;
 using ZB.MOM.WW.LocalDb;
 using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
+using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
 using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
 using ZB.MOM.WW.ScadaBridge.DataConnectionLayer;
 using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
@@ -43,7 +44,11 @@ public sealed class SiteRuntimeFixture : IAsyncDisposable
     private readonly ILocalDb _localDb;
     private readonly List<IActorRef> _instanceActors = new();
     private readonly List<StreamSubscriberProbe> _probes = new();
+    private readonly List<WireStreamSubscriber> _wireSubscribers = new();
     private readonly string _dataDirectory;
+    private SiteWireStreamHost? _wireHost;
+    private SiteStreamGrpcClient? _wireClient;
+    private CancellationTokenSource? _wireCts;
 
     /// <summary>Site identifier, e.g. <c>site-01</c>.</summary>
     public string SiteId { get; }
@@ -72,6 +77,12 @@ public sealed class SiteRuntimeFixture : IAsyncDisposable
     /// <summary>The live stream subscribers attached to this site.</summary>
     public IReadOnlyList<StreamSubscriberProbe> Probes => _probes;
 
+    /// <summary>The wire-path (real gRPC) subscribers attached to this site, if any.</summary>
+    public IReadOnlyList<WireStreamSubscriber> WireSubscribers => _wireSubscribers;
+
+    /// <summary>The site's real gRPC listener, when wire probes are attached.</summary>
+    public SiteWireStreamHost? WireHost => _wireHost;
+
     /// <summary>Instance Actors created on this site.</summary>
     public IReadOnlyList<IActorRef> InstanceActors => _instanceActors;
 
@@ -344,12 +355,63 @@ public sealed class SiteRuntimeFixture : IAsyncDisposable
         }
     }
 
+    /// <summary>
+    /// Starts the site's real Kestrel/gRPC listener and attaches
+    /// <paramref name="count"/> subscribers through the production
+    /// <see cref="SiteStreamGrpcClient"/> — one channel per site with the streams
+    /// multiplexed on it, exactly the production shape (central keys one client per
+    /// <c>(siteId, endpoint)</c>). Instance selection uses the same stride as
+    /// <see cref="AttachStreamProbes"/>, so the wire probes watch the SAME instances
+    /// as the in-process probes and the two latency distributions are directly
+    /// comparable within one run.
+    /// </summary>
+    /// <param name="count">Wire subscribers to attach.</param>
+    /// <param name="latency">Histogram that receives end-to-end (over-the-wire) latencies.</param>
+    /// <param name="log">Progress sink for fault reporting.</param>
+    /// <returns>
+    /// A task that completes when the listener is up and every stream has been
+    /// initiated. Stream ESTABLISHMENT finishes asynchronously — comfortably inside
+    /// the subscribe-settle + settle windows that separate attach from measurement.
+    /// </returns>
+    public async Task AttachWireStreamProbesAsync(int count, LatencyHistogram latency, Action<string> log)
+    {
+        _wireHost = await SiteWireStreamHost.StartAsync(this);
+        _wireClient = new SiteStreamGrpcClient(_wireHost.Endpoint, NullLogger.Instance);
+        _wireCts = new CancellationTokenSource();
+
+        var stride = Math.Max(1, _config.InstancesPerSite / Math.Max(1, count));
+        for (var p = 0; p < count; p++)
+        {
+            var instanceIndex = Math.Min(p * stride, _config.InstancesPerSite - 1);
+            var subscriber = WireStreamSubscriber.Attach(
+                _wireClient, InstanceName(instanceIndex), $"wire-{SiteId}-p{p:D2}",
+                latency, log, _wireCts.Token);
+            _wireSubscribers.Add(subscriber);
+        }
+    }
+
     /// <summary>Total tag paths registered across this site's connections.</summary>
     public int TotalTagPaths => TagPathsByConnection.Sum(list => list.Count);
 
     /// <inheritdoc />
     public async ValueTask DisposeAsync()
     {
+        // Wire path first: cancel the client streams, wait for their loops to exit,
+        // then drop the channel and the listener — the same order production teardown
+        // takes (client goes away, server cancels, actor system last).
+        if (_wireCts is not null)
+        {
+            await _wireCts.CancelAsync();
+            foreach (var subscriber in _wireSubscribers)
+                await subscriber.WaitForShutdownAsync();
+            _wireCts.Dispose();
+        }
+
+        if (_wireClient is not null)
+            await _wireClient.DisposeAsync();
+        if (_wireHost is not null)
+            await _wireHost.DisposeAsync();
+
         foreach (var probe in _probes)
             await probe.DisposeAsync();
 

tests/ZB.MOM.WW.ScadaBridge.LoadHarness/ZB.MOM.WW.ScadaBridge.LoadHarness.csproj

@@ -27,6 +27,14 @@
     <ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>
   </PropertyGroup>
 
+  <ItemGroup>
+    <!-- Wire-path probes: each site hosts the production SiteStreamGrpcServer on a
+         real Kestrel h2c listener so the R2 batcher + serialization + HTTP/2 flow
+         control sit inside the measured path (see Probes/SiteWireStreamHost.cs). -->
+    <FrameworkReference Include="Microsoft.AspNetCore.App" />
+    <PackageReference Include="Grpc.AspNetCore" />
+  </ItemGroup>
+
   <ItemGroup>
     <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Commons/ZB.MOM.WW.ScadaBridge.Commons.csproj" />
     <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ZB.MOM.WW.ScadaBridge.SiteRuntime.csproj" />

tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs

@@ -49,6 +49,7 @@ public class TargetScaleHarnessSmokeTests
             DebugProbeInterval = TimeSpan.FromSeconds(2),
             SubscribeSettleDuration = TimeSpan.FromSeconds(5),
             StreamProbesPerSite = 3,
+            WireProbesPerSite = 2,
             StoreAndForwardDrainMessages = 200,
             SlowSubscriberEvents = 2_000,
             ResultsPath = Path.Combine(Path.GetTempPath(), $"loadharness-smoke-{Guid.NewGuid():N}.json"),
@@ -64,6 +65,18 @@ public class TargetScaleHarnessSmokeTests
         Assert.True(result.StreamProbeReceived > 0, "Live stream subscribers received nothing.");
         Assert.Equal(0, result.DriverSkippedNoCallback);
 
+        // The wire path (real Kestrel + real SiteStreamGrpcClient, R2 batching
+        // negotiated) carried traffic end to end: events crossed a real HTTP/2
+        // socket, the batch-size histogram observed real frames, and nothing faulted.
+        Assert.NotNull(result.WireStream);
+        Assert.True(result.WireStream!.EventsReceived > 0,
+            "Wire subscribers received nothing — the Kestrel/gRPC hop did not carry traffic.");
+        Assert.True(result.WireStream.Latency.Count > 0,
+            "No wire latency samples — the over-the-wire measurement path is broken.");
+        Assert.True(result.WireStream.Batches.Frames > 0,
+            "The batch-size MeterListener observed no frames — the R2 histogram path is broken.");
+        Assert.Equal(0, result.WireStream.StreamFaults);
+
         // Observability held up.
         Assert.Equal(config.Sites, result.SitesTrackedByAggregator);
         Assert.True(result.DebugSnapshotsCompleted > 0, "No debug snapshot completed.");