New ResidualBench tool (eventlog, flatten, audit, audit-contention) and
docker/measure-ui-responsiveness.sh. All four residuals pass; results in
docs/plans/2026-11-10-wp4-residual-scope-results.md. Findings filed as
#122, #123 and #124 (fixed in the following commit).
@@ -30,7 +30,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 27 | ~~External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface~~**RESOLVED 2026-08-15 (`41a5ecf1`).** The gap was transport + persistence only — the consumption end already worked (`ExternalSystemClient.CachedCallAsync` already passed `system.MaxRetries`/`RetryDelay` to `EnqueueAsync` with 0-as-unset, and the Central UI form already collected both fields, inert until now with defaults 3×5s that will take effect on next deploy). Shipped: artifact fields, additive `site_command.proto` fields 7/8 (regen via `docker/regen-proto.sh`, csproj left clean), site `external_systems` columns `max_retries`/`retry_delay_ms` as REAL COLUMNS — not a JSON bag; verified safe against the AckTime precedent in the LocalDb source (capture triggers are DROP-then-CREATE from `PRAGMA table_info` each boot, and the fail-closed handshake digest turns a mixed-version pair into a loud sync refusal, no new constraint since both-nodes-together is already the standing pair rule; `timeout_seconds` set the additive-column precedent on this exact table) — apply path, and CLI `--max-retries`/`--retry-delay-seconds` (+`--retry-delay` alias). Bonus fix: `ArtifactDiff.CompareExternalSystem` never compared the retry fields, so a bundle import changing a retry budget previewed as "Identical". Fail-before verified on the site round-trip + apply tests; 12 suites green. | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Closed 2026-08-15. | — |
| 28 | ~~Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"**~~**RESOLVED 2026-08-15 (`0db253b5`).** Structural, not a missing argument: `IDialogService.ConfirmAsync` had NO label parameter — `DialogHost` derived the verb from `danger` alone (`danger ? "Delete" : "Confirm"`), so no call site could ever have passed one. New optional `confirmLabel`; `danger` now selects styling only. Full audit of all 34 `ConfirmAsync` call sites (the only confirmation path — no `window.confirm`, no per-page modals): NINE were wrongly labeled "Delete" — both failover buttons, secured-write **Approve** and **Reject**, instance Disable, and four Discard actions — all now name their action; 18 genuine deletes + 7 non-danger sites verified correct. The `danger→"Delete"` fallback deliberately kept: forcing an explicit label would churn ~20 correct sites, and a missed site would silently degrade to "Confirm", detectable only by the rig-dependent Playwright suite — contract pinned in the interface XML doc, Component-CentralUI.md, and a fallback test instead. Gotcha recorded in the tests: adding an optional parameter silently un-matches existing 3-arg NSubstitute stubs (returns null Task → NRE). Follow-ups: rows 75 (Approve's red styling), and the bUnit real-dialog deadlock hazard documented in `FailoverConfirmLabelTests`. | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | Closed 2026-08-15. | — |
| 29 | ~~**1-hour sustained run to settle the memory-slope ambiguity (load-test finding F1)**~~**RESOLVED 2026-08-15 — run, and F1 is settled: NOT a leak.** Full 1-hour protocol executed (same harness/scale, `--sustain-minutes 60`, 135,041,250 updates at 37,511/s = 100.0 % of nominal, concurrent with the live docker rig AND a central failover drill on the same host). Decisive: **working-set slope −28.89 MB/min** (2,722 → 1,757 MB over the 3,600 s window, peak 3,258 MB) with **one gen-2 collection observed** and threads flat at 59 — a working set that FALLS ~1 GB across a sustained hour is incompatible with a slow leak; the positive managed-heap least-squares slope was the sawtooth-segment artifact F1 hypothesised. All other metrics held or improved vs the 20-min run: P99 4.01 ms, 0 healthy-subscriber drops, 744/744 debug snapshots, S&F drain to depth 0. Full numbers appended to the results doc F1 section. | `docs/plans/2026-08-15-target-scale-load-test-results.md` F1 postscript | Closed 2026-08-15. | — |
| 30 | **→ #49** **WP-4 residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-load-vs-central degradation, `[xc-5]` large-template flattening time, `[xc-8]` UI responsiveness under load** | `docs/plans/2026-08-15-target-scale-load-test-results.md` §6 + row 25 closure note | The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
| 30 | **→ #49 — measured 2026-11-10, all four PASS (`docs/plans/2026-11-10-wp4-residual-scope-results.md`; follow-ups #122–#124)** **WP-4 residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-load-vs-central degradation, `[xc-5]` large-template flattening time, `[xc-8]` UI responsiveness under load** | `docs/plans/2026-08-15-target-scale-load-test-results.md` §6 + row 25 closure note | The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
| 31 | **Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F3 | Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded `DropOldest` channel; healthy subscribers delivered 100.00%), but the publish `Source.ActorRef`**upstream** of the `BroadcastHub` is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. | First field report of event loss on *healthy* subscribers during a burst, or any redesign of the site stream publish stage |
| 32 | ~~**`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)**~~**RESOLVED 2026-08-15 (`65b19516`).** Reproduced first (2 failures in 5 parallel runs — the gauge read a sibling class's provider). The register's suggested fix (per-test meter) was rejected on inspection: the slot is process-global by product design and 3 of the 6 facts assert the slot-ownership semantics themselves (compare-and-clear identity, stop/restart re-seed), so per-test instances would change what they prove. Correct isolation is mutual exclusion — the three `StartAsync`-calling classes now share xunit collection `QueueDepthGaugeCollection` (with a `[CallerFilePath]`-anchored source-scan drift guard that names an offending file), the other eight classes stay parallel, and `parallelizeTestCollections` was flipped back ON so the fix is exercised rather than masked. Negative control: dropping the attribute fails the guard AND resurfaces the original flake in the same run. 8/8 green parallel runs. | `tests/…StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs | Closed 2026-08-15. | — |
| 33 | ~~**`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment**~~**RESOLVED 2026-08-15 (`8da60f57`).** Both halves shipped: sampled into KPI history via `SiteHealthKpiSampleSource` (conditionally — 12 samples unreplicated / 14 replicated, so a series gap truthfully means "not replicated") plus a backlog trend chart on the site health panel, and a SUSTAINED threshold badge (`LocalDbBacklogThreshold` 1000 held for `LocalDbBacklogSustainedFor` 15 min, anchored by `LocalDbBacklogAboveSince` in the aggregator) — a bare instantaneous threshold would fire on every replicated site every day from the documented purge burst. Null backlog is a deliberate third branch: it leaves the anchor unchanged (neither evidence of a drain nor of a backlog), so unreplicated sites can never anchor and can never be badged. NOTE the row's premise was false — the `SiteAuditBacklog` "configurable threshold badge" it told us to mirror was described twice in `Component-HealthMonitoring.md` but NEVER BUILT; both stale doc claims corrected, and the new badge was modeled on the real Metrics-stale badge instead. | Found by R7 while documenting the purge burst | Closed 2026-08-15. | — |
| Commit | `48517b49` + the bench tool in this change |
| Rig data (for `[xc-8]`) | 14 instances, 36 templates, **10.9 M `AuditLog` rows, 4.35 M `Notifications`** (944,597 Parked) from months of soak traffic |
| SQL Server | `scadabridge-mssql` — SQL Server 2022 RTM-CU22 Developer, **amd64 image under emulation** on Apple silicon (OrbStack, 14 vCPU / 16 GB VM). Absolute SQL timings are pessimistic; compare ratios. |
| Concurrent load | The 8-node `docker/` rig stayed up throughout (it is the target of `[xc-8]`), as in the 2026-08-15 run. Benchmarks were run one at a time, never overlapping. |
## 2. Verdict summary
| Criterion | Threshold used | Measured | Verdict |
|---|---|---|---|
| `[xc-3]` site event logging handles volume within the 30-day / 1 GB limits | writer keeps up with any realistic event rate without loss; queries usable on a full 1 GB log; the cap is enforced | 1 GB = 2.36 M events = 30 days only at ≤ 0.91 events/s per site; writer 10,500 events/s with CDC (11,500× that); storms ≤ 10,000 absorbed with 0 drops; every query ≤ 0.5 s warm on a full 1 GB log; one purge tick restores the cap in 12.8 s; no writer impact from concurrent scans | **PASS**, with an operational caveat: above ~1 event/s the **1 GB cap**, not the 30 days, sets retention (§3.1) |
| `[xc-4]` audit logging does not degrade central | central read P95 stays inside the UI budget (< 1 s) while ingest runs at and above the expected fleet rate | 1 M rows / 3.26 GB seeded through the production ingest path; reads P95 ≤ 22 ms up to 500 rows/s ingest, ≤ 160 ms at the single-writer ceiling (~1,000–1,500 rows/s); non-audit reads unaffected; 0 deadlocks with one writer | **PASS**, with an observation: concurrent set-based inserts deadlock (1205, #124) |
| `[xc-5]` flattening/validation within acceptable time for large templates | deploy-gate validate of one large template < 5 s; 500-instance bulk CPU well inside the 120 s site-deploy budget | 4,400-attribute / 400-alarm / 220-script, 8-deep template: flatten 3.6 ms, deploy-gate validate 1.9 s cold / 2.3 ms warm, hash 5.7 ms; 500 instances flatten+validate+hash in 3.5 s | **PASS** (34× inside the 120 s budget) |
| `[xc-8]` UI workflows remain responsive | page usable P95 < 3 s under load, no page > 10 s | 14 pages, rig with 10.9 M AuditLog / 4.35 M Notification rows; under 6 concurrent operator sessions + 20 inbound API calls/s: P50 46 ms, P95 1.18 s, max 3.0 s (Audit Log). Idle: every page ≤ 0.7 s **except Health, up to 5.05 s** | **PASS** against the threshold, with one defect: the Health dashboard blocks up to 5 s on a cold KPI cache (#122) |
The requirements give no numeric thresholds for these four. The thresholds above were chosen
for this run and are stated so the verdicts can be re-judged.
---
## 3. `[xc-3]` Site event log — `ResidualBench eventlog`
Real `SiteEventLogger`, `EventLogQueryService` and `EventLogPurgeService` over a real
`ZB.MOM.WW.LocalDb` file. The event mix models the post-WP3.2 stream (per-run script events
- large: flatten+validate+hash of 500 instances sharing one template, serial: 1.97 s total; per instance n=500 P50 2.81 ms, P95 3.45 ms, P99 7.22 ms, max 477.72 ms
- extra-large: flatten+validate+hash of 500 instances sharing one template, serial: 3.49 s total; per instance n=500 P50 3.80 ms, P95 5.14 ms, P99 18.59 ms, max 1282.61 ms
- Flattening itself is milliseconds even for the 4,400-attribute shape.
- The cost is the first, cold compile of a template's scripts — about 1.5–2 s for 60–220
unique scripts — and it is paid once per distinct script body, thanks to
`ScriptCompileVerdictCache`.
- After that, every instance of the template is a cache hit. A 500-instance bulk deploy of
the extra-large template spends 3.5 s of central CPU in flatten+validate+hash, versus the
120 s budget for deploying 500 instances to a site.
---
## 5. `[xc-4]` Audit logging vs central — `ResidualBench audit`
A throwaway database on `scadabridge-mssql`, migrated with the production EF chain (the
partitioned `AuditLog`, clustered `(OccurredAtUtc, EventId)` from #111, all NC indexes),
dropped afterwards. Rows are site-shaped (~1.5 KB, 10 sites × 500 instances, 4% failures) and
inserted through the production `AuditLogRepository.InsertManyIfNotExistsAsync` in 256-row
packets — the same set-based anti-semi-join insert `AuditLogIngestActor` runs.
**One serial writer, as in production.** `AuditLogIngestActor` is a cluster singleton that
awaits each packet inside `ReceiveAsync`, so central never runs two site-ingest inserts at
once. A first attempt with four concurrent writers hit **SQL error 1205 (deadlock victim)**
within seconds. Production can still see concurrent writers on this table (the
reconciliation singleton, central direct writes), and `AuditIngestErrorClassifier` already
treats 1205 as transient, so the site re-sends. That is recorded here as an observation, not
a defect.
```
seeded 1,000,192 rows (45 days) in 934 s = 1,071 rows/s (1 serial writer, 256-row packets)
AuditLog size 3259 MB
```
### no ingest: achieved ingest 0 rows/s, deadlock retries so far 0
Real Chromium (host Playwright) against the rig through Traefik. A load is a full navigation,
timed until the page is usable: the Blazor circuit is up and no spinner or "Loading…" text
remains. 14 pages, 5 loads each, idle first and then under load.
**Under load**: 6 operator sessions (7.0 page loads/s), inbound API 20.0 calls/s offered, 652/652 HTTP 200, API latency P50 92 ms / P95 980 ms / max 1264 ms
| page | idle P50 ms | idle max ms | load P50 ms | load max ms |
|---|---|---|---|---|
| Dashboard | 29 | 32 | 52 | 86 |
| Health | 3362 | 5048 | 38 | 128 |
| Topology | 38 | 52 | 148 | 201 |
| Instances | 42 | 64 | 74 | 141 |
| Deployments | 54 | 57 | 39 | 247 |
| Templates | 61 | 74 | 144 | 455 |
| Audit Log | 425 | 696 | 1081 | 3046 |
| Config audit | 25 | 27 | 26 | 33 |
| Site Calls | 41 | 47 | 51 | 61 |
| Notifications | 358 | 365 | 962 | 1627 |
| Alarms | 25 | 28 | 26 | 27 |
| Event Logs | 27 | 29 | 27 | 35 |
| Sites | 28 | 33 | 38 | 59 |
| Flow Debugger | 33 | 89 | 31 | 38 |
| **all pages** | P50 40, P95 696 | max 5048 | P50 46, P95 1177 | max 3046 |
Reading the numbers:
- **Every page except Health is usable in ≤ 0.7 s idle**; most in 25–60 ms on loopback.
- **Health is the outlier: 3.4 s P50, up to 5.05 s idle, but 38 ms under load.** The
Notification Outbox KPI tile exceeds the page's 5 s `TileTimeout`. On this rig the outbox
holds 944,597 Parked rows, and counting them is a 374,043-read clustered scan (1.3 s per
query, warm). A timed-out snapshot is not cached, and the tiles load inside prerender, so
a lone viewer waits the full timeout on every load. Under load, another session's
successful refresh keeps the cache warm, which is why the number drops. Filed as **#122**
(p2).
- **Audit Log** (1.1 s P50 / 3.0 s max under load) and **Notifications** (1.0 s / 1.6 s)
are the slowest pages under load. Both read the rig's largest tables (10.9 M audit rows,
4.35 M notifications) and stay inside the 3 s budget.
- The inbound API kept up throughout: 652/652 HTTP 200 at 20 calls/s, P95 0.98 s,
including the `Route.To()` round trip to site-a.
---
## 7. Findings and follow-ups
| # | Finding | Severity | Issue |
|---|---|---|---|
| 1 | Health dashboard waits up to 5 s for a timed-out Notification Outbox KPI tile when the outbox holds ~1 M Parked rows (uncached timeout, prerender blocks) | p2 bug | **#122** |
| 2 | Event-log `severity` / `event_type` filters read and sort every matching row (2.3 s cold, ~0.1–0.15 s warm at 1 GB) | p3 | **#123** |
| 3 | Concurrent `AuditLog` set-based inserts deadlock (1205); transient-classified, but production has more than one writer | p3 | **#124** |
| 4 | 30-day site event retention holds only below ~0.91 events/s per site; above that the 1 GB cap governs | operational note | — (documented here and in §3.1; raise `MaxStorageMb` per site if needed) |
| 5 | A burst larger than the 10,000-event write queue, arriving faster than the writer drains, loses its oldest events (counted, surfaced on the health report) | by design | — |
Phase-8 WP-4 is now covered for all of `[xc-1]`–`[xc-8]`: `[xc-1]`, `[xc-2]`, `[xc-6]` and
`[xc-7]` by the 2026-08-15 run, and the four residuals by this one.
### 7.1 Follow-up fixes (same day)
The three issues were fixed after the measurement run; the verdicts above stand, and the numbers
in §3, §5 and §6 are the pre-fix measurements.
- **#122.** The real global outbox KPI cost was 0.7–2.6 s, not the 1.3 s quoted in §6: the earlier
probe ran under `sqlcmd` without `-I`, which disables filtered indexes. The OR'd pre-filter
forced a clustered scan; the snapshot now runs three status-equality legs (live, parked,
delivered) that each seek their own index, ~0.12 s on the rig.
- **#123.** The `event_type` and `severity` indexes are now composites with `timestamp`.
`ResidualBench eventlog` re-run: `severity=Error, 30 d` 2,287 → 15 ms cold and 115 → 1.6 ms
warm; `event_type=alarm, 24 h` 216 → 1.2 ms cold. The cost: 454 → 530 B/row on disk, so the
1 GB cap holds 2.03 M rows instead of 2.36 M and 30 days fit only below 0.78 events/s per site
(was 0.91); the CDC-on writer drains 8,760 events/s instead of 10,535.
- **#124.** The set-based ingest's `NOT EXISTS` probe is forced to seek. `ResidualBench
audit-contention`: 4 writers 775 rows/s with 76 deadlock victims in 30 s → 5,214 rows/s with
none; 8 writers 8,607 rows/s with none.
## 8. What this does not prove
- **The rig has few instances.**`[xc-8]` ran against the docker rig: 14 instances and 36
templates, though its audit and notification tables are large (10.9 M / 4.35 M rows). Page
cost that grows with fleet size (Topology, Instances, pending updates) is not exercised at
10 sites × 500 instances here. The audit and flattening benches cover the two biggest
central data-volume costs separately.
- **Emulated SQL Server.**`[xc-4]` absolute numbers come from an amd64 SQL Server under
emulation. Treat them as a pessimistic lower bound on throughput, not a production sizing.
- **No WAN.** As in the 2026-08-15 run, site↔central hops are loopback.
## 9. Reproducing
```bash
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- eventlog # ~4 min, ~1.1 GB temp disk
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- flatten # ~20 s
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- audit # needs MS SQL (default: local scadabridge-mssql sa)
bash docker/measure-ui-responsiveness.sh # needs the rig + seed-mes-alarm-demo.sh
See `phase-8-production-readiness.md:152-170` (WP-4) and `:314-320`
(test protocol) for the original scope.
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Joseph Doherty