docs(plans): WP-4 target-scale load test RESULTS + close register #25 and row 50

Full-scale run executed on this machine: 10 sites x 500 instances x 75 tags =
375,000 live tag subscriptions, 37,518 updates/s achieved vs 37,500 nominal,
45,021,375 updates over a 20-minute steady-state window (M4 Pro, 14 cores, 48 GB,
with the 8-node docker rig still running so the figures are pessimistic).

11 clean passes, 1 pass with a caveat, 0 failures:
  tag latency  P50 0.88ms  P99 4.57ms  max 37.41ms  (1.1M samples, end-to-end)
  stream       900,675 delivered, 0 dropped at 100 live subscribers
  health       collect+ingest P99 0.31ms, 10/10 sites tracked
  debug view   P99 2.19ms, 264 completed, 0 timeouts
  deploy       500 instances to a site in 2.6s
  cpu          41% of ONE core = 2.9% of the box
  memory       working-set slope +8.83 MB/min

Three findings, reported rather than tuned away:
  F1 (Low) 20 min with ZERO gen-2 collections cannot fully settle the leak
     question; the heap demonstrably sawtooths but an uncompacted gen-2 makes a
     positive slope ambiguous. The 1-hour run would settle it. Not tuned.
  F2 (informational, by design) a deferred S&F backlog sits for one full
     DefaultRetryInterval (28.9s measured) before anything drains --
     EnqueueAsync(attemptImmediateDelivery:false) stamps LastAttemptAt. Easy to
     misread as slow drainage, so drain is reported as two numbers: retry wait,
     then 3,533 msg/s of actual capacity.
  F3 (positive) slow-subscriber isolation is TOTAL: 4 healthy subscribers at
     100.00% with zero drops while a peer lost 197,028/200,000 events entirely
     within its own bounded channel. Mechanism recorded link by link.

Also records what the run does NOT prove (not clustered, not real-network, not
real OPC UA, not 1 hour) and the four WP-4 sub-criteria this harness does not
cover, so the evidence is not over-read.

Register rows 25 and 50 -> RESOLVED 2026-08-15; remediation execution-log
residual 7 -> resolved; phase-8-checklist WP-4 section replaced with the
measured numbers.

docs/plans/2026-07-08-deferred-work-register.md

@@ -18,7 +18,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
 | 23 | Live LDAP group-membership re-query for an active session | `docs/requirements/Component-Security.md` :61-69 (+ :78-79) | Blocked on an external package. The mid-session refresh re-maps the **stored** groups against the central DB with **no LDAP call**, so a directory group-membership change lands only at next login. A live re-query needs a passwordless service-account group-search method on the shared `ZB.MOM.WW.Auth.Ldap` library — an external NuGet `PackageReference` (`src/ZB.MOM.WW.ScadaBridge.Security/…csproj:23`) exposing only `AuthenticateAsync(username, password, ct)`. Central role-mapping/scope changes still apply within ~15 min (`RoleRefreshThresholdMinutes`). | `ZB.MOM.WW.Auth.Ldap` gains a standalone group-search API, or a requirement that a directory-side group revocation take effect mid-session rather than at next login |
 | 24 | M8 large-bundle performance hardening | `docs/plans/2026-06-15-stillpending-completion-design.md:106` — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" | Logged as a non-blocking follow-up when M8 shipped and never given an artifact: **no plan, no task entry, no perf/load test exists** (`tests/…Transport.Tests/Import/BundleImporterLoadTests.cs` is a `LoadAsync` unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, `LineDiffer`'s `MaxInputLines`=4000 summary-only cap, and `MaxConcurrentImportSessions`=8. | First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
-| 25 | Phase-8 WP-4 target-scale load test (10 sites × 500 instances × 75 tags = 37,500 subscriptions/site, 375,000 total) | `docs/plans/phase-8-production-readiness.md:152-170` (WP-4) + `:314-320` (test protocol); status claimed in `docs/plans/phase-8-checklist.md` | **Claimed complete but unevidenced.** The whole WP-4 deliverable is a **107-byte** checklist stub asserting "Status: Complete / Tests: All passing / Build: 0 errors, 0 warnings" with no per-work-package results and no linked run. Nearest real coverage is arithmetic/aggregation only — `PerformanceTests/StaggeredStartupTests.cs` (`TagCapacity_75TagsPer500Machines_37500Total`, 500-instances-over-10-sites distribution) and `HealthAggregationTests` (10-site report aggregation) — plus a **single-subscriber** 100k-event `Streaming/SiteStreamThroughputTests.cs`. No sustained multi-site run exists anywhere in `tests/` or `docker/`. | Before any production go-live at target scale; or the first site approaching ~500 instances / ~37.5k subscriptions |
+| 25 | ~~**Phase-8 WP-4 target-scale load test** (10 sites x 500 instances x 75 tags = 37,500 subscriptions/site, 375,000 total)~~ **RESOLVED 2026-08-15 — run, measured, evidenced.** The 107-byte stub is replaced by a real full-scale run: **10 x 500 x 75 = 375,000 live tag subscriptions**, 37,518 tag updates/s achieved against 37,500 nominal (100.0%), **45,021,375 updates offered** over a 20-minute steady-state window on an M4 Pro / 14-core / 48 GB host with the 8-node `docker/` rig still running. Headline numbers: tag update latency **P50 0.88 ms / P99 4.57 ms / max 37.41 ms** end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); **0 events dropped** at 100 live subscribers; health report collect+ingest **P99 0.31 ms** with 10/10 sites tracked; debug view snapshot under load **P99 2.19 ms, 0 timeouts**; 500 instances deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope **+8.83 MB/min**. **11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures.** Harness: `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone Exe — the `Category=Performance` trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in `PerformanceTests`) plus a CI-scale smoke `[Fact]` (`PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`). Real: per-site ActorSystem + LocalDb, the real DCL via the documented `DataConnectionFactory.RegisterAdapter` seam, real `InstanceActor`s, real `SiteStreamManager`, real `StreamRelayActor` + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. **Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`; design + deviations + thresholds: `docs/plans/2026-08-15-target-scale-load-test-design.md`.** | as before | Closed. Residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-vs-central degradation, `[xc-5]` large-template flattening and `[xc-8]` UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. | Closed 2026-08-15. |
 | 26 | Ipsen MES MoveIn tail: leak-test (`-LT`) receivers + routing, PLC-output-flag writes, `Z28062` BTDB data completeness | `docs/plans/2026-06-16-ipsen-mes-movein.md:409` ("Out of scope (future)"); design `2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198` | Customer-site scope, not a platform gap. `-LT` routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any `-LT`/unknown suffix returns `WasSuccessful=false` with an "unsupported side/target" message by decision); `MoveInComplete`/`Successful`/`ErrorText` are **PLC-owned** by locked decision, so ScadaBridge deliberately does not write them; `Z28062` completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — `_LT` is stripped before side-scoping (`2026-06-30-mes-alarm-status-api.md:158`). | Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a **candidate won't-do** (`[PERM]`) at the next Ipsen scope review |
 | 27 | External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Two stacked gaps: (a) `ExternalSystemArtifact` (Commons) carries `TimeoutSeconds` but NOT `MaxRetries`/`RetryDelay`, and the site `external_systems` table has no such columns — so a centrally-configured retry policy is silently ignored on sites; every cached call buffers with the S&F default (`DefaultMaxRetries` 50 × `DefaultRetryInterval` 30s ≈ 25 min to park). (b) `Create/UpdateExternalSystemCommand` don't expose the fields either — the only way to set them today is a direct DB edit of `ExternalSystemDefinitions`. Transport bundles DO carry them (arch-review 05 "ES retry config"), which masks the gap in export/import round-trips. Fix is additive: extend the artifact + site schema + apply path, and add `--max-retries`/`--retry-delay` to the CLI. | First operator who tunes retry policy on an external system and expects site cached calls to honor it |
 | 28 | Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"** | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | The DialogService confirmation host's default destructive-action label leaks through — the dialog copy is correct but the red confirm button says "Delete" for a failover. One-line fix: pass an explicit confirm label ("Fail over") at the Health-dashboard call site (and audit other confirm-dialog call sites for the same default). | Next Central UI session |
@@ -47,7 +47,7 @@ Rows removed from the Deferred table above once confirmed shipped. Kept here for
 | docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
 | Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
 | ~~Failover-timing measurement (the "~25s total failover" envelope)~~ **RESOLVED 2026-08-01** — split out of the combined row and closed. `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` is no longer a skipped placeholder: it runs as a live `[Fact]` (`Category=Performance`) on the real two-node in-process rig (`TwoNodeClusterFixture`, production `BuildHocon`) at production timings — 2s heartbeat / 10s failure-detection threshold / 15s stable-after — hard-killing the younger node and timing the survivor's member REMOVAL with singleton continuity asserted on the oldest. Delivered by **PLAN-R2-01 Task 4** (`archreview/plans/PLAN-R2-01-cluster-host-failover.md:226`). The oldest-crash direction is covered behaviorally by `SbrFailoverTests.AutoDown_HardCrashOfOldestNode_*` and by `docker/failover-drill.sh`. | The 2026-07-08 "PLAN-01 rig landing" trigger had fired unnoticed (NF2); PLAN-R2-01 T4 wired the placeholder to the fixture rig rather than recording a blocker. | Closed. |
-| Broader perf envelope — **S&F drain rate + per-subscriber stream backpressure** (the still-open half of the former combined row) | Never measured, and no owner plan survives now that PLAN-R2-01 closed the failover half. `PerformanceTests` covers failover timing, staggered startup, health aggregation, audit hot-path latency and a **single-subscriber** 100k-event `Streaming/SiteStreamThroughputTests.cs` — nothing measures store-and-forward drain throughput, nor what a slow/stalled subscriber does to the per-subscriber buffering in `Communication/Actors/StreamRelayActor.cs` / `Grpc/SiteStreamGrpcServer.cs` with many subscribers attached. No defect observed; deferred as measurement-only work. | First field S&F backlog that fails to drain within an operator's patience, a slow gRPC subscriber degrading a site stream for others, or the WP-4 target-scale run (row 25) being scheduled — that run should absorb this |
+| ~~Broader perf envelope — **S&F drain rate + per-subscriber stream backpressure**~~ **RESOLVED 2026-08-15 — absorbed into the row-25 target-scale run, exactly as this row's trigger anticipated.** Both halves measured at full scale. **(a) S&F drain:** 20,000 messages buffered concurrently from 25 origin instances at **15,368 msg/s**, draining at **3,533 msg/s** to depth 0. Reported as two numbers because the naive single number is misleading — a deferred backlog sits for one full `DefaultRetryInterval` (measured 28.9 s) before *anything* drains, since `EnqueueAsync(attemptImmediateDelivery: false)` stamps `LastAttemptAt` and the due-predicate then holds the row for one interval (correct by design; `Notify.Send`'s `deferToSweep: true` leaves it null and skips the wait). Filed as informational finding **F2**. **(b) Slow-subscriber backpressure:** with 5 subscribers on the SAME instance and one reader stalled at 50 ms/event, the four healthy subscribers delivered **100.00% with zero drops** while the stalled one lost 197,028 of 200,000 events **entirely within its own bounded channel**. Isolation is structural and total — `Buffer(DropHead)` never backpressures the `BroadcastHub`, `Sink.ForEach`'s `Tell` never blocks, and `StreamRelayActor.WriteToChannel`'s `TryWrite` on a `DropOldest` channel never blocks — so a slow WAN link degrades only its own feed, visibly (per-stream eviction counter + `RecordSiteStreamEventDropped`). Finding **F3**. Caveat also recorded: an *unpaced* burst costs every subscriber, because the publish `Source.ActorRef` upstream of the hub is shared — that shared stage, not per-subscriber buffering, bounds burst absorption. | Closed by the row-25 run; no defect found in either half. | Closed 2026-08-15 — `docs/plans/2026-08-15-target-scale-load-test-results.md`. |
 
 ## Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
 Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md`'s registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.

docs/plans/2026-08-15-arch-review-remediation-execution-log.md

@@ -110,9 +110,20 @@ Deliberately not fixed in this program — each has a stated reason, not an over
    library's wire-compatibility guarantee — not a blocker for this program.
 6. **Fragile `SandboxTests` timing pin.** Pre-existing, unrelated to this remediation's changes;
    noted so it isn't mistaken for a regression if it flakes later.
-7. **Target-scale load test (deferred-work register #25).** This program's exit criterion is the
-   live probes above, not #25 — #25 remains the follow-on validation that the moved ceilings hold
-   under real load; schedule separately.
+7. ~~**Target-scale load test (deferred-work register #25).**~~ **RESOLVED 2026-08-15.** The
+   follow-on validation ran: 10 sites x 500 instances x 75 tags = **375,000 live tag
+   subscriptions**, 37,518 updates/s achieved vs 37,500 nominal, **45,021,375 updates** over a
+   20-minute steady-state window. The moved ceilings hold with room to spare — end-to-end tag
+   latency **P99 4.57 ms**, zero dropped events at 100 live subscribers, health report
+   collect+ingest **P99 0.31 ms**, debug view **P99 2.19 ms** with no timeouts, CPU **2.9% of the
+   box**. 11 clean passes, 1 pass with a caveat (F1: 20 minutes with zero gen-2 collections cannot
+   fully settle the leak question; the 1-hour run would), 0 failures. Register **row 50** (S&F
+   drain rate + per-subscriber stream backpressure) was absorbed into the same run and closed with
+   it — S&F drains at **3,533 msg/s**, and a stalled subscriber costs healthy peers **nothing**
+   (100.00%, zero drops). Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`;
+   design: `docs/plans/2026-08-15-target-scale-load-test-design.md`; harness:
+   `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`. Residual: WP-4's `[xc-3]`/`[xc-4]`/`[xc-5]`/`[xc-8]`
+   are central-cluster/browser concerns outside this harness and remain open scope.
 8. **Playwright 14 pre-existing env failures.** Present on `main` too, rig-state related, not
    introduced by this branch.
 9. **`site_events` retention purge still oplog-visible.** WP3.2's sliced retention DELETE is a row

docs/plans/2026-08-15-target-scale-load-test-results.md

@@ -0,0 +1,289 @@
+# Target-Scale Load Test — Results (Phase-8 WP-4, deferred-work register #25 + row 50)
+
+**Date:** 2026-08-15
+**Design memo:** `docs/plans/2026-08-15-target-scale-load-test-design.md`
+**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`
+**Closes:** deferred-work register **#25** and **row 50**; residual **7** of
+`docs/plans/2026-08-15-arch-review-remediation-execution-log.md`.
+
+> Register row 25 recorded that WP-4 had been *claimed complete* on the strength of a
+> 107-byte checklist stub with no per-work-package results and no linked run. This
+> document is the run that stub never had.
+
+---
+
+## 1. Environment
+
+| | |
+|---|---|
+| Host | `Josephs-MBP` — Apple M4 Pro, 14 logical cores, 48 GB RAM |
+| OS | macOS 26.5.2 (Darwin) |
+| Runtime | .NET 10.0.5, **server GC enabled**, Release build |
+| Branch / commit | `target-scale-load-test` @ `20f6b0b9` (harness), `8abebdae` (design memo) |
+| Concurrent load | The 8-node `docker/` rig (OrbStack) remained running throughout — the user's live development cluster. Measured baseline: **~86% of one core (~6% of the box)** and ~10-16 GB RSS. The harness figures below are therefore *pessimistic*, not idealised. |
+| Scale executed | **10 sites × 500 instances × 75 tags = 375,000 live tag subscriptions** |
+| Offered load | 37,500 tag updates/second nominal (one update per tag per 10 s) |
+| Measurement window | 20 minutes steady state, after a 2-minute settle |
+| Total wall time | 1,510 s (~25 min) |
+
+Two independent full-scale runs were executed. **Run 1** is the primary dataset.
+**Run 2** re-ran the identical protocol with raw per-sample resource capture added
+(the only harness change between them) to resolve the memory question in §4.
+
+---
+
+## 2. Verdict summary
+
+| # | Criterion | Threshold | Measured | Verdict |
+|---|---|---|---|---|
+| 1 | `[2.5-1]` 10 sites simultaneously operational | all 10 built, ramped, tracked centrally | 10/10 sites tracked by `CentralHealthAggregator` | **PASS** |
+| 2 | `[2.5-2]` 500 instances/site, active subscriptions | 5,000 actors; `SkippedNoCallback` = 0 | 5,000 `InstanceActor`s; skipped = **0** | **PASS** |
+| 3 | `[2.5-3]` 75 tags/instance (375,000 total) | 375,000 subscribed via the real DCL | 375,000 tag paths across 50 `DataConnectionActor`s | **PASS** |
+| 4 | Tag update latency | P99 < 250 ms, P50 < 50 ms | **P50 0.88 ms, P95 2.83 ms, P99 4.57 ms, P99.9 16.04 ms, max 37.41 ms** (1,100,675 samples) | **PASS** (55× margin at P99) |
+| 5 | `[xc-1]` message patterns under load | ≥ 95% of nominal, no unexpected loss | **37,518/s achieved vs 37,500 nominal (100.0%)**; 45,021,375 updates offered; **0 events dropped** at healthy subscribers | **PASS** |
+| 6 | `[xc-2]` health reports within expected intervals | collect+ingest P99 < 1,000 ms; all sites tracked | **P99 0.31 ms, max 4.60 ms** over 440 reports; 10/10 sites | **PASS** (3,200× margin) |
+| 7 | `[xc-6]` debug view without impacting site performance | snapshot P99 < 2,000 ms, 0 timeouts | **P99 2.19 ms, max 2.72 ms**, 264 completed, **0 timeouts** | **PASS** |
+| 8 | `[xc-7]` S&F concurrent buffering from many instances | 20,000 messages, 25 origins, drains to 0 | 20,000 buffered at **15,368 msg/s**; residual depth **0** | **PASS** |
+| 9 | Row 50 — S&F drain rate | > 500 msg/s | **3,533 msg/s** (20,000 messages in 5.66 s of active drain) | **PASS** (7× margin) |
+| 10 | Row 50 — slow-subscriber backpressure | healthy = 100%; stalled loss confined to its own channel | **healthy 100.00% (4/4, zero drops)**; stalled 1.18%, all 197,028 losses in its own bounded channel | **PASS** |
+| 11 | Deploy 500 instances to a site | slowest site < 120 s | **2.6 s** | **PASS** (46× margin) |
+| 12 | Memory within acceptable bounds | WS slope < 20 MB/min; no monotonic heap climb | WS **+8.83 MB/min**; heap sawtooths (peak 3,625 MB vs 2,313→2,523 MB endpoints) with a **+19.52 MB/min** drift — see finding **F1** | **PASS with a caveat** |
+| 13 | CPU within acceptable bounds | mean < 50% of the box | **41% of one core = 2.9% of the box** (peak 55% of one core) | **PASS** (17× margin) |
+
+**11 clean passes, 1 pass with a documented caveat, 0 failures.** Four WP-4
+sub-criteria were out of this harness's scope — see §6.
+
+---
+
+## 3. Measured detail
+
+### 3.1 Deployment / ramp
+
+```
+site fixtures built     0.2s
+all 5,000 instance actors   2.7s   (10 sites in parallel)
+slowest single site     2.6s       (500 instances, production staggering)
+```
+
+The ramp used the production `StartupBatchSize` (20) / `StartupBatchDelayMs` (100)
+pacing, i.e. 25 batches × 100 ms = 2.5 s of *deliberate* delay per site. So the 2.6 s
+figure is almost entirely the configured stagger; actual construction of 500
+`InstanceActor`s (each deserializing a 75-attribute `FlattenedConfiguration`, loading
+static overrides from SQLite and issuing a DCL subscribe) costs ~0.1 s. Against a
+120 s budget this criterion is not close to binding.
+
+### 3.2 Tag update latency (DCL boundary → stream subscriber)
+
+```
+samples 1,100,675   mean 1.17ms
+p50 0.88ms   p95 2.83ms   p99 4.57ms   p99.9 16.04ms   max 37.41ms
+```
+
+This is a true end-to-end path measurement, not a synthetic timer: the emit instant is
+stamped on the `TagValue` handed to the adapter callback and travels verbatim through
+`DataConnectionActor` → `TagValueUpdate.Timestamp` → `AttributeValueChanged.Timestamp`
+→ the site stream → `StreamRelayActor` → the proto `SiteStreamEvent`, where the
+subscriber subtracts it. Every hop is production code.
+
+Sub-millisecond median under 37,500 updates/s across 5,000 actors means the actor
+hierarchy is nowhere near its ceiling at target scale.
+
+### 3.3 Stream delivery
+
+```
+events delivered  900,675
+events dropped          0
+```
+
+Zero evictions at the 100 live subscribers across a 20-minute window at full rate.
+The site stream's `DropHead` buffering exists for pathological cases; at nominal
+target-scale load it never engaged.
+
+### 3.4 Health reporting at scale
+
+```
+samples 440   mean 0.03ms   p50 0.01ms   p95 0.04ms   p99 0.31ms   max 4.60ms
+reports ingested 440    sites tracked centrally 10
+```
+
+`SiteHealthCollector.CollectReport` for a site carrying 37,500 subscriptions plus
+`CentralHealthAggregator.ProcessReport` costs well under a millisecond. Against the
+30-second report interval there is no plausible queueing risk.
+
+### 3.5 Debug view under load
+
+```
+samples 264   mean 0.14ms   p50 0.08ms   p99 2.19ms   max 2.72ms
+completed 264   timed out 0
+```
+
+Each sample is an `Ask<DebugViewSnapshot>` landing in the mailbox of an
+`InstanceActor` concurrently ingesting tag updates, so the figure includes real
+queueing behind production traffic. `[xc-6]`'s "without impacting site performance"
+also holds in the other direction: tag latency percentiles were measured with these
+snapshots running throughout, and show no degradation.
+
+### 3.6 Resources
+
+```
+window          1200s over 121 samples
+working set     2665 -> 3261 MB (peak 3261)   slope  +8.83 MB/min
+managed heap    2313 -> 2523 MB (peak 3625)   slope +19.52 MB/min
+cpu mean/peak   41% / 55% of one core  (2.9% of the box)
+threads         58        gen2 GCs in window: 0
+```
+
+CPU is the headline: **375,000 subscriptions at 37,500 updates/s consumed under half
+of one core of fourteen.** Thread count stayed flat at 58 for ten actor systems.
+
+---
+
+## 4. Findings
+
+### F1 — Memory: 20 minutes with zero gen-2 collections cannot fully settle the leak question (Low)
+
+The working-set slope (+8.83 MB/min) is inside the threshold, and the managed heap
+clearly *sawtooths* — its peak (3,625 MB) sits well above both endpoints (2,313 MB →
+2,523 MB), so the collector is demonstrably reclaiming. But **`GC.CollectionCount(2)`
+recorded zero gen-2 collections across the entire 25-minute run**, at 45 million
+events. Gen-2 was therefore never compacted, and a positive least-squares drift on an
+uncompacted heap cannot be distinguished from a genuine slow leak by this run alone.
+
+This is a limitation of the shortened window (deviation D1), not an observed defect:
+no metric degraded, no drop appeared, and latency percentiles were flat from the first
+minute to the twentieth. Recorded honestly rather than reported as "no leak".
+
+**To settle it**, run the full 1-hour protocol (§7) and check whether the heap slope
+flattens once gen-2 collections begin, or issue an explicit
+`GC.Collect(2, Forced, blocking: true)` at window start and end and compare.
+Deliberately *not* attempted here — the brief was to measure, not to tune.
+
+### F2 — Store-and-forward waits one full `DefaultRetryInterval` before draining a deferred backlog (informational, by design)
+
+The drain measurement initially read as **33 msg/s**, which would have been alarming.
+The progress series showed why: **nothing drained for 29-30 seconds, then the entire
+backlog cleared in a fraction of a second.**
+
+Cause, confirmed in source: `StoreAndForwardService.EnqueueAsync` with
+`attemptImmediateDelivery: false` stamps `message.LastAttemptAt = UtcNow`
+(`StoreAndForwardService.cs:754-757`). `GetMessagesForRetryAsync`'s due-predicate is
+`last_attempt_at_ms IS NULL OR retry_interval_ms = 0 OR (now - last_attempt) >= retry_interval`,
+so such a row is not due for one `DefaultRetryInterval` (30 s). Explicit
+`TriggerSweep()` calls find nothing due and correctly do nothing.
+
+This is **correct, intended behaviour** — the caller declared delivery was not
+attempted, so the row waits one retry interval — but it is easy to misread as slow
+drainage. The harness therefore reports the two numbers separately:
+
+```
+retry wait before drain  28.9s  (DefaultRetryInterval)
+drain throughput      3,533 msg/s  (active drain 5.66s)
+```
+
+Worth knowing operationally: a site that buffers a backlog this way will appear
+completely stalled for the first 30 seconds. Note the `Notify.Send` path
+(`deferToSweep: true`) deliberately leaves `LastAttemptAt` null so its rows are due
+immediately and skip this wait entirely.
+
+### F3 — Slow-subscriber isolation is complete, and the mechanism is worth recording (positive result)
+
+Register row 50 asked what a slow/stalled gRPC subscriber does to per-subscriber
+buffering with several subscribers attached. With five subscribers on the *same*
+instance (identical offered sequence) and one reader stalled at 50 ms/event:
+
+| Subscriber | State | Received | Dropped | Delivery |
+|---|---|---|---|---|
+| slowprobe-00 | **STALLED** | 2,369 | 197,028 | 1.18% |
+| slowprobe-01 | healthy | 200,000 | 0 | **100.00%** |
+| slowprobe-02 | healthy | 200,000 | 0 | **100.00%** |
+| slowprobe-03 | healthy | 200,000 | 0 | **100.00%** |
+| slowprobe-04 | healthy | 200,000 | 0 | **100.00%** |
+
+**Isolation is total** — not merely "good". The stalled subscriber lost 98.8% of its
+own events and cost its peers exactly nothing. The reason is structural, and each link
+matters:
+
+1. The per-subscriber graph is `Where → Buffer(StreamBufferSize, DropHead) → Sink.ForEach(Tell)`.
+   A `DropHead` buffer *always* accepts, so it never backpressures the `BroadcastHub`
+   upstream — one subscriber cannot stall the hub for the others.
+2. `Sink.ForEach` does an actor `Tell`, which never blocks.
+3. `StreamRelayActor.WriteToChannel` uses `TryWrite` on a bounded `DropOldest`
+   channel, which never blocks either — so the relay actor's mailbox drains regardless
+   of reader speed, and loss is confined to that subscriber's own channel where it is
+   counted.
+
+The consequence for operators: a slow WAN link or wedged central client degrades
+**only its own feed**, and the loss is visible in the per-stream eviction counter and
+the `RecordSiteStreamEventDropped` telemetry rather than being silent.
+
+One caveat established while building the probe: publishing an unpaced burst makes
+*every* subscriber lose events, because the publish source is a single
+`Source.ActorRef(StreamBufferSize, DropHead)` **shared** by all attribute subscribers,
+upstream of the hub. That shared stage — not per-subscriber buffering — is the real
+limit on burst absorption. The probe paces at 2,000 events/s to isolate the variable
+under test; the shared stage's own capacity is separately covered by
+`SiteStreamThroughputTests`.
+
+---
+
+## 5. What this does *not* prove
+
+Stated plainly so the evidence is not over-read:
+
+- **Not a clustered run.** Sites are single, non-clustered `ActorSystem`s (deviation
+  D2). Failover, singleton handover and split-brain behaviour are covered elsewhere
+  (`FailoverTimingTests`, `SbrFailoverTests`, `docker/failover-drill.sh`).
+- **Not a real-network run.** The gRPC, health-transport and S&F-to-central hops are
+  in-process. Serialization cost, TLS, WAN latency and socket backpressure are out of
+  frame; the *stream* path's backpressure was measured deliberately by substituting a
+  controllable reader for the socket (D4).
+- **Not a real OPC UA run.** 375,000 monitored items were simulated at the adapter
+  callback (D3). The DCL above that callback is real and fully exercised.
+- **Not 1 hour.** 20 minutes (D1). See finding F1 for the one question this leaves
+  genuinely open.
+
+## 6. WP-4 criteria not covered by this harness
+
+Recorded rather than silently dropped (deviation D7):
+
+| Criterion | Why not covered |
+|---|---|
+| `[xc-3]` site event logging within 30-day / 1 GB limits | Needs a long-horizon retention run, not a load run |
+| `[xc-4]` audit logging does not degrade central | Needs a central MS SQL fixture; this harness builds site runtimes only |
+| `[xc-5]` template flattening/validation for large templates | Template Engine is a central-cluster concern; unrelated to the site-runtime load path |
+| `[xc-8]` UI workflows remain responsive | Needs a browser + a live central cluster (Playwright territory) |
+
+These remain open scope for WP-4 and should be tracked separately rather than being
+considered closed by this run.
+
+## 7. Reproducing
+
+Full protocol as executed (20-minute window):
+
+```bash
+dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
+  --results loadharness-results.json
+```
+
+The full **1-hour** version required by the WP-4 test protocol — identical in every
+other respect, and the run that would settle finding F1:
+
+```bash
+dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.LoadHarness -- \
+  --sustain-minutes 60 --results loadharness-results-1h.json
+```
+
+CI-scale smoke (~80 s), which asserts the harness still produces coherent
+measurements end to end:
+
+```bash
+dotnet test tests/ZB.MOM.WW.ScadaBridge.PerformanceTests \
+  --filter "FullyQualifiedName~TargetScaleHarnessSmoke"
+```
+
+`--help`-style knobs: `--sites`, `--instances-per-site`, `--tags-per-instance`,
+`--tag-update-period-seconds`, `--settle-minutes`, `--sustain-minutes`,
+`--sample-seconds`, `--health-interval-seconds`, `--debug-probe-interval-seconds`,
+`--subscribe-settle-seconds`, `--stream-probes-per-site`, `--sf-drain-messages`,
+`--slow-subscriber-events`, `--data-dir`, `--results`. An unknown key is rejected
+rather than ignored, so a typo cannot silently change the measured scale.

docs/plans/phase-8-checklist.md

@@ -3,21 +3,46 @@
 **Update 2026-08-07 (truth sweep):** This checklist previously read
 "Status: Complete / Tests: All passing / Build: 0 errors, 0 warnings" — a
 107-byte stub with no per-work-package results and no linked run. That claim
-was **unevidenced** and has been retired.
+was **unevidenced** and was retired.
 
-Honest state, per the deferred-work register
-(`2026-07-08-deferred-work-register.md`, row 25):
+**Update 2026-08-15 (WP-4 run, evidenced):** The target-scale load test has now
+actually been run, at full scale, with results published.
 
-- The **Phase-8 WP-4 target-scale load test** (10 sites × 500 instances ×
-  75 tags = 37,500 subscriptions/site, ~375,000 total) has **never been run**.
-- Nearest real coverage is arithmetic/aggregation only —
-  `PerformanceTests/StaggeredStartupTests.cs`
-  (`TagCapacity_75TagsPer500Machines_37500Total`) and
-  `HealthAggregationTests` — plus a single-subscriber 100k-event
-  `Streaming/SiteStreamThroughputTests.cs`. No sustained multi-site run
-  exists anywhere in `tests/` or `docker/`.
-- Revisit trigger: before any production go-live at target scale, or the
-  first site approaching ~500 instances / ~37.5k subscriptions.
+## WP-4 — Load/Performance Testing at Target Scale
+
+**Status: measured 2026-08-15.** Not a claim — a run.
+
+- Scale executed: **10 sites × 500 instances × 75 tags = 375,000 live tag
+  subscriptions**, 37,518 tag updates/second achieved against 37,500 nominal
+  (100.0%), **45,021,375 updates** offered over a 20-minute steady-state window.
+- Headline results: end-to-end tag update latency **P50 0.88 ms / P99 4.57 ms /
+  max 37.41 ms**; **zero events dropped** at 100 live stream subscribers; health
+  report collect+ingest **P99 0.31 ms** with 10/10 sites tracked centrally;
+  debug view snapshot under load **P99 2.19 ms with 0 timeouts**; 500 instances
+  deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope
+  **+8.83 MB/min**.
+- **11 clean passes, 1 pass with a documented caveat, 0 failures.**
+- Deferred-work register **#25** and **row 50** are closed by this run.
+
+**Results:** `2026-08-15-target-scale-load-test-results.md` — per-criterion
+numbers vs thresholds, three findings (F1 memory-window limitation, F2 the S&F
+retry-interval wait, F3 slow-subscriber isolation), and an explicit "what this
+does not prove" section.
+
+**Design, thresholds and deviations:**
+`2026-08-15-target-scale-load-test-design.md` — including the documented
+deviation from the protocol's 1-hour sustained window to 20 minutes, and the
+command for the full 1-hour version.
+
+**Harness:** `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone executable),
+with a CI-scale smoke `[Fact]` at
+`tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`
+so the evidence can be regenerated rather than bit-rotting.
+
+**Still open within WP-4** (not covered by this harness — central-cluster and
+browser concerns): `[xc-3]` site event logging volume within 30-day/1 GB limits,
+`[xc-4]` audit logging not degrading central, `[xc-5]` template
+flattening/validation for large templates, `[xc-8]` UI workflow responsiveness.
 
 See `phase-8-production-readiness.md:152-170` (WP-4) and `:314-320`
 (test protocol) for the original scope.