# WP-4 residual scope — results (`[xc-3]`, `[xc-4]`, `[xc-5]`, `[xc-8]`, ScadaBridge#49) **Date:** 2026-11-10 **Closes:** the four WP-4 sub-criteria that the target-scale load test left open (`docs/plans/2026-08-15-target-scale-load-test-results.md` §6, deferred-work register row 30 → #49). **Tools (committed):** `tests/ZB.MOM.WW.ScadaBridge.ResidualBench/` (modes `eventlog`, `flatten`, `audit`) and `docker/measure-ui-responsiveness.sh`. ## 1. Environment | | | |---|---| | Host | Apple M4 Pro, 14 logical cores, 48 GB RAM, macOS 26.5.2 | | Runtime | .NET 10.0.14, Release, server GC | | Commit | `48517b49` + the bench tool in this change | | Rig data (for `[xc-8]`) | 14 instances, 36 templates, **10.9 M `AuditLog` rows, 4.35 M `Notifications`** (944,597 Parked) from months of soak traffic | | SQL Server | `scadabridge-mssql` — SQL Server 2022 RTM-CU22 Developer, **amd64 image under emulation** on Apple silicon (OrbStack, 14 vCPU / 16 GB VM). Absolute SQL timings are pessimistic; compare ratios. | | Concurrent load | The 8-node `docker/` rig stayed up throughout (it is the target of `[xc-8]`), as in the 2026-08-15 run. Benchmarks were run one at a time, never overlapping. | ## 2. Verdict summary | Criterion | Threshold used | Measured | Verdict | |---|---|---|---| | `[xc-3]` site event logging handles volume within the 30-day / 1 GB limits | writer keeps up with any realistic event rate without loss; queries usable on a full 1 GB log; the cap is enforced | 1 GB = 2.36 M events = 30 days only at ≤ 0.91 events/s per site; writer 10,500 events/s with CDC (11,500× that); storms ≤ 10,000 absorbed with 0 drops; every query ≤ 0.5 s warm on a full 1 GB log; one purge tick restores the cap in 12.8 s; no writer impact from concurrent scans | **PASS**, with an operational caveat: above ~1 event/s the **1 GB cap**, not the 30 days, sets retention (§3.1) | | `[xc-4]` audit logging does not degrade central | central read P95 stays inside the UI budget (< 1 s) while ingest runs at and above the expected fleet rate | 1 M rows / 3.26 GB seeded through the production ingest path; reads P95 ≤ 22 ms up to 500 rows/s ingest, ≤ 160 ms at the single-writer ceiling (~1,000–1,500 rows/s); non-audit reads unaffected; 0 deadlocks with one writer | **PASS**, with an observation: concurrent set-based inserts deadlock (1205, #124) | | `[xc-5]` flattening/validation within acceptable time for large templates | deploy-gate validate of one large template < 5 s; 500-instance bulk CPU well inside the 120 s site-deploy budget | 4,400-attribute / 400-alarm / 220-script, 8-deep template: flatten 3.6 ms, deploy-gate validate 1.9 s cold / 2.3 ms warm, hash 5.7 ms; 500 instances flatten+validate+hash in 3.5 s | **PASS** (34× inside the 120 s budget) | | `[xc-8]` UI workflows remain responsive | page usable P95 < 3 s under load, no page > 10 s | 14 pages, rig with 10.9 M AuditLog / 4.35 M Notification rows; under 6 concurrent operator sessions + 20 inbound API calls/s: P50 46 ms, P95 1.18 s, max 3.0 s (Audit Log). Idle: every page ≤ 0.7 s **except Health, up to 5.05 s** | **PASS** against the threshold, with one defect: the Health dashboard blocks up to 5 s on a cold KPI cache (#122) | The requirements give no numeric thresholds for these four. The thresholds above were chosen for this run and are stated so the verdicts can be re-judged. --- ## 3. `[xc-3]` Site event log — `ResidualBench eventlog` Real `SiteEventLogger`, `EventLogQueryService` and `EventLogPurgeService` over a real `ZB.MOM.WW.LocalDb` file. The event mix models the post-WP3.2 stream (per-run script events off): 45% alarm transitions, 25% script failures/warnings, 15% connection quality changes, 10% S&F parks, 5% lifecycle; 2-in-3 rows carry an instance id and 1-in-10 carry an ~800-byte stack trace. ### 3.1 What 1 GB holds | | | |---|---| | On-disk bytes per row (table + 4 indexes + PK) | **454** | | Rows at the 1 GB cap | **2,364,426** | | Sustained rate that keeps 30 days inside 1 GB | **≤ 0.91 events/s per site** (78,800/day; **158 per instance per day** at 500 instances) | | Retention actually achieved at 5 events/s | 5.5 days | | Retention actually achieved at 20 events/s | 1.4 days | **The binding limit is the 1 GB cap, not the 30-day window, for any site averaging more than about one event per second.** That is by design — `MaxStorageMb` wins over `RetentionDays` (the cap trim deletes oldest-first) — but it means "30-day event history" holds only for quiet sites. A 500-machine site whose alarms average more than ~150 transitions per machine per day will keep less than 30 days. Raise `ScadaBridge:SiteEventLog:MaxStorageMb` for such sites if 30 days matters; disk cost is ~0.45 KB per event. ### 3.2 Writer throughput | | CDC off (unreplicated site) | CDC on (`RegisterReplicated`, the replicated site-a shape) | |---|---|---| | Sustained drain (200,000 events, offered in 5,000-event windows) | **34,000 events/s**, 0 failed | **10,500 events/s**, 0 failed | | vs the 30-day-in-1-GB rate | 37,000× | 11,500× | | Storm of 5,000 / 10,000 events offered at once | settled in 0.23 / 0.42 s, **0 dropped** | 0.70 / 1.43 s, **0 dropped** | | Storm of 25,000 events offered at once | **14,723 dropped** | **14,733 dropped** | The bounded write queue (`WriteQueueCapacity` 10,000, DropOldest) absorbs any single burst up to its capacity. A burst much larger than that, arriving faster than the writer drains, loses the oldest queued events — counted in `FailedWriteCount` and surfaced on the health report as `SiteEventLogWriteFailures`, never silent. CDC capture triggers cost ~3.2× on write throughput; the replicated rate is still four orders of magnitude above the rate the storage cap can retain. ### 3.3 Queries on a full 1 GB, 30-day log (page size 500) | query (page size 500) | first ms | warm P50 ms | warm max ms | rows | has more | |---|---|---|---|---|---| | no filter, page 1 (oldest first) | 8.5 | 1.1 | 1.1 | 500 | True | | last 1 h | 1.3 | 1.1 | 1.1 | 500 | True | | last 24 h | 1.2 | 1.0 | 1.0 | 500 | True | | severity=Error, 30 d | 2286.8 | 114.5 | 485.6 | 500 | True | | severity=Error, last 24 h | 117.3 | 104.5 | 110.0 | 500 | True | | event_type=alarm, last 24 h | 215.6 | 148.5 | 168.1 | 500 | True | | instance=Machine-0042, 30 d | 4.3 | 3.7 | 4.3 | 500 | True | | keyword 'timeout' (clamped to 7 d) | 1.5 | 0.8 | 0.9 | 500 | True | | keyword with no match (full 7 d window scan) | 91.9 | 91.9 | 105.8 | 0 | False | | keyword 'Machine-0042', last 24 h | 14.2 | 13.3 | 13.5 | 105 | False | | 20 keyset pages, last 24 h | | 0.7 | 4.1 | 500/page | | Every shape is under 0.5 s warm. The slowest are equality filters on `severity` or `event_type`: SQLite (no `ANALYZE` statistics) picks the single-column index, then reads and sorts *every* matching row in the table before returning the first page (`USE TEMP B-TREE FOR ORDER BY`), so their cost grows with table size, not page size — 2.3 s on the first, cold run. Adding composite indexes would fix it, but would also change the row model that the storage-cap estimate depends on (register row 93). This is filed as a low-priority follow-up, not a failure. ### 3.4 Writer while queries run 200 events/s offered for 20 s (≈ 220× the 30-day rate), first with no queries, then while a back-to-back worst-case keyword scan (no match, a full 7-day-window scan, ~92 ms each) ran on another thread: - writer at 200 ev/s, idle: enqueue->durable n=3999 P50 0.39 ms, P95 1.46 ms, P99 17.27 ms, max 33.44 ms; failed 0 - writer at 200 ev/s, with a back-to-back no-match keyword scan loop (207 scans): enqueue->durable n=3999 P50 0.13 ms, P95 0.55 ms, P99 0.96 ms, max 43.38 ms; failed 0 No degradation: reads use their own WAL connection and never take the writer's lock (review 2026-10-21 02/P1), and the numbers confirm it. ### 3.5 Cap enforcement One real purge tick on a table 10% over the cap: **2,608,866 → 2,379,761 rows in 12.8 s** (229,105 deleted, under the 250,000 per-tick ceiling, so a single tick restored the cap to within the size model's error). Consistent with the 2026-08-15 measurement (~17 s for a full 250,000-row tick). --- ## 4. `[xc-5]` Template flattening and validation — `ResidualBench flatten` Production `FlatteningService` → `ValidationService.Validate` (the deploy gate: trust check plus a real Roslyn compile of every script, plus all semantic checks) → `RevisionHashService`, in-process. Each level of the inheritance chain adds attributes/alarms/scripts and overrides a fifth of its parent's attribute values. Feature modules are composed on the base template and each composes one sub-module. The instance overrides 10% of the attributes and binds every data-sourced attribute. Scripts are ~35 lines each, and every script's code is unique, so a cold compile really compiles every one. Not in frame: the configuration-database read `FlatteningPipeline` does before this (one `FlattenSession` per batch). | shape | chain depth | flattened attrs / alarms / scripts | flatten P50 ms | validate, cold compile ms | validate, warm cache P50 ms | validate, compile off P50 ms | hash P50 ms | errors | |---|---|---|---|---|---|---|---|---| | typical | 2 | 100 / 12 / 8 | 0.05 | 402 | 0.08 | 0.07 | 0.10 | 0 | | large | 5 | 1,100 / 100 / 60 | 0.70 | 1549 | 0.77 | 0.63 | 1.62 | 0 | | extra-large | 8 | 4,400 / 400 / 220 | 3.62 | 1876 | 2.27 | 1.89 | 5.73 | 0 | - large: flatten+validate+hash of 500 instances sharing one template, serial: 1.97 s total; per instance n=500 P50 2.81 ms, P95 3.45 ms, P99 7.22 ms, max 477.72 ms - extra-large: flatten+validate+hash of 500 instances sharing one template, serial: 3.49 s total; per instance n=500 P50 3.80 ms, P95 5.14 ms, P99 18.59 ms, max 1282.61 ms - Flattening itself is milliseconds even for the 4,400-attribute shape. - The cost is the first, cold compile of a template's scripts — about 1.5–2 s for 60–220 unique scripts — and it is paid once per distinct script body, thanks to `ScriptCompileVerdictCache`. - After that, every instance of the template is a cache hit. A 500-instance bulk deploy of the extra-large template spends 3.5 s of central CPU in flatten+validate+hash, versus the 120 s budget for deploying 500 instances to a site. --- ## 5. `[xc-4]` Audit logging vs central — `ResidualBench audit` A throwaway database on `scadabridge-mssql`, migrated with the production EF chain (the partitioned `AuditLog`, clustered `(OccurredAtUtc, EventId)` from #111, all NC indexes), dropped afterwards. Rows are site-shaped (~1.5 KB, 10 sites × 500 instances, 4% failures) and inserted through the production `AuditLogRepository.InsertManyIfNotExistsAsync` in 256-row packets — the same set-based anti-semi-join insert `AuditLogIngestActor` runs. **One serial writer, as in production.** `AuditLogIngestActor` is a cluster singleton that awaits each packet inside `ReceiveAsync`, so central never runs two site-ingest inserts at once. A first attempt with four concurrent writers hit **SQL error 1205 (deadlock victim)** within seconds. Production can still see concurrent writers on this table (the reconciliation singleton, central direct writes), and `AuditIngestErrorClassifier` already treats 1205 as transient, so the site re-sends. That is recorded here as an observation, not a defect. ``` seeded 1,000,192 rows (45 days) in 934 s = 1,071 rows/s (1 serial writer, 256-row packets) AuditLog size 3259 MB ``` ### no ingest: achieved ingest 0 rows/s, deadlock retries so far 0 | read | P50 ms | P95 ms | P99 ms | max ms | n | |---|---|---|---|---|---| | Audit Log page 1 (newest 50, no filter) | 1.8 | 10.4 | 18.9 | 149.5 | 3052 | | channel=ApiOutbound, last 24 h, 50 | 2.0 | 10.2 | 20.2 | 43.0 | 3052 | | site=site-07, last 7 d, 100 | 3.4 | 11.2 | 24.7 | 59.0 | 3052 | | status=Failed, last 24 h, 50 | 2.9 | 10.7 | 24.3 | 66.9 | 3052 | | KPI snapshot, 1 h window | 1.0 | 10.0 | 17.3 | 33.0 | 3052 | | non-audit: templates + sites list | 1.1 | 14.1 | 26.5 | 46.5 | 3052 | ### ingest 100 rows/s: achieved ingest 98 rows/s, deadlock retries so far 0; 256-row packet n=23 P50 230.05 ms, P95 495.30 ms, P99 731.88 ms, max 731.88 ms | read | P50 ms | P95 ms | P99 ms | max ms | n | |---|---|---|---|---|---| | Audit Log page 1 (newest 50, no filter) | 2.4 | 11.2 | 20.6 | 598.0 | 1635 | | channel=ApiOutbound, last 24 h, 50 | 2.7 | 11.2 | 12.7 | 78.0 | 1635 | | site=site-07, last 7 d, 100 | 4.4 | 13.1 | 30.3 | 166.5 | 1635 | | status=Failed, last 24 h, 50 | 4.0 | 12.3 | 24.6 | 151.0 | 1635 | | KPI snapshot, 1 h window | 2.9 | 10.7 | 17.0 | 214.7 | 1635 | | non-audit: templates + sites list | 1.9 | 21.0 | 23.3 | 56.9 | 1635 | ### ingest 500 rows/s: achieved ingest 499 rows/s, deadlock retries so far 0; 256-row packet n=117 P50 203.94 ms, P95 577.36 ms, P99 713.85 ms, max 1133.67 ms | read | P50 ms | P95 ms | P99 ms | max ms | n | |---|---|---|---|---|---| | Audit Log page 1 (newest 50, no filter) | 2.7 | 22.2 | 122.9 | 247.6 | 1273 | | channel=ApiOutbound, last 24 h, 50 | 2.8 | 11.2 | 104.4 | 1143.9 | 1273 | | site=site-07, last 7 d, 100 | 4.3 | 18.8 | 137.8 | 318.8 | 1273 | | status=Failed, last 24 h, 50 | 3.7 | 12.8 | 100.5 | 293.0 | 1273 | | KPI snapshot, 1 h window | 8.8 | 19.2 | 114.2 | 356.7 | 1273 | | non-audit: templates + sites list | 1.8 | 20.5 | 21.7 | 36.5 | 1273 | ### ingest 2,000 rows/s: achieved ingest 1,474 rows/s, deadlock retries so far 0; 256-row packet n=346 P50 125.09 ms, P95 397.36 ms, P99 732.35 ms, max 956.29 ms | read | P50 ms | P95 ms | P99 ms | max ms | n | |---|---|---|---|---|---| | Audit Log page 1 (newest 50, no filter) | 4.8 | 94.7 | 147.7 | 381.7 | 593 | | channel=ApiOutbound, last 24 h, 50 | 3.5 | 30.4 | 90.7 | 1380.4 | 593 | | site=site-07, last 7 d, 100 | 5.1 | 42.9 | 146.6 | 1750.2 | 593 | | status=Failed, last 24 h, 50 | 4.3 | 25.9 | 128.8 | 359.8 | 593 | | KPI snapshot, 1 h window | 30.6 | 135.4 | 188.5 | 704.0 | 593 | | non-audit: templates + sites list | 2.1 | 3.1 | 4.9 | 27.2 | 593 | ### ingest unthrottled (1 serial writer): achieved ingest 1,032 rows/s, deadlock retries so far 0; 256-row packet n=243 P50 190.53 ms, P95 557.43 ms, P99 900.67 ms, max 1223.08 ms | read | P50 ms | P95 ms | P99 ms | max ms | n | |---|---|---|---|---|---| | Audit Log page 1 (newest 50, no filter) | 12.3 | 116.8 | 199.9 | 284.5 | 382 | | channel=ApiOutbound, last 24 h, 50 | 3.7 | 13.3 | 107.7 | 159.1 | 382 | | site=site-07, last 7 d, 100 | 7.4 | 49.9 | 125.3 | 644.6 | 382 | | status=Failed, last 24 h, 50 | 4.7 | 76.7 | 176.2 | 1491.2 | 382 | | KPI snapshot, 1 h window | 69.3 | 160.0 | 345.6 | 681.9 | 382 | | non-audit: templates + sites list | 2.3 | 4.2 | 11.8 | 42.2 | 382 | Reading the numbers: - **Ingest ceiling, one serial writer: ~1,000–1,500 rows/s** on this emulated SQL Server. The seed sustained 1,071 rows/s while the table grew to 1 M rows / 3.26 GB. Each 256-row packet takes 125–230 ms P50. That is the central ingest capacity on this box, and it is far above any audit volume the rig produces. - **Reads barely move at realistic rates.** At 100 and 500 rows/s every read's P95 stays ≤ 22 ms, against 10–14 ms with no ingest. - **At the ceiling, P95 reaches 117–160 ms.** The worst are the Audit Log page 1 and the KPI snapshot, which compete with the insert for the newest partition's pages. There are rare outliers up to 1.75 s. Still well inside the 1 s P95 budget. - **Non-audit central reads are unaffected** (P95 ≤ 21 ms in every phase). - **Deadlocks: 0** with one writer. Four concurrent writers deadlocked within seconds (see above) — **#124** (p3). --- ## 6. `[xc-8]` UI responsiveness — `docker/measure-ui-responsiveness.sh` Real Chromium (host Playwright) against the rig through Traefik. A load is a full navigation, timed until the page is usable: the Blazor circuit is up and no spinner or "Loading…" text remains. 14 pages, 5 loads each, idle first and then under load. **Under load**: 6 operator sessions (7.0 page loads/s), inbound API 20.0 calls/s offered, 652/652 HTTP 200, API latency P50 92 ms / P95 980 ms / max 1264 ms | page | idle P50 ms | idle max ms | load P50 ms | load max ms | |---|---|---|---|---| | Dashboard | 29 | 32 | 52 | 86 | | Health | 3362 | 5048 | 38 | 128 | | Topology | 38 | 52 | 148 | 201 | | Instances | 42 | 64 | 74 | 141 | | Deployments | 54 | 57 | 39 | 247 | | Templates | 61 | 74 | 144 | 455 | | Audit Log | 425 | 696 | 1081 | 3046 | | Config audit | 25 | 27 | 26 | 33 | | Site Calls | 41 | 47 | 51 | 61 | | Notifications | 358 | 365 | 962 | 1627 | | Alarms | 25 | 28 | 26 | 27 | | Event Logs | 27 | 29 | 27 | 35 | | Sites | 28 | 33 | 38 | 59 | | Flow Debugger | 33 | 89 | 31 | 38 | | **all pages** | P50 40, P95 696 | max 5048 | P50 46, P95 1177 | max 3046 | Reading the numbers: - **Every page except Health is usable in ≤ 0.7 s idle**; most in 25–60 ms on loopback. - **Health is the outlier: 3.4 s P50, up to 5.05 s idle, but 38 ms under load.** The Notification Outbox KPI tile exceeds the page's 5 s `TileTimeout`. On this rig the outbox holds 944,597 Parked rows, and counting them is a 374,043-read clustered scan (1.3 s per query, warm). A timed-out snapshot is not cached, and the tiles load inside prerender, so a lone viewer waits the full timeout on every load. Under load, another session's successful refresh keeps the cache warm, which is why the number drops. Filed as **#122** (p2). - **Audit Log** (1.1 s P50 / 3.0 s max under load) and **Notifications** (1.0 s / 1.6 s) are the slowest pages under load. Both read the rig's largest tables (10.9 M audit rows, 4.35 M notifications) and stay inside the 3 s budget. - The inbound API kept up throughout: 652/652 HTTP 200 at 20 calls/s, P95 0.98 s, including the `Route.To()` round trip to site-a. --- ## 7. Findings and follow-ups | # | Finding | Severity | Issue | |---|---|---|---| | 1 | Health dashboard waits up to 5 s for a timed-out Notification Outbox KPI tile when the outbox holds ~1 M Parked rows (uncached timeout, prerender blocks) | p2 bug | **#122** | | 2 | Event-log `severity` / `event_type` filters read and sort every matching row (2.3 s cold, ~0.1–0.15 s warm at 1 GB) | p3 | **#123** | | 3 | Concurrent `AuditLog` set-based inserts deadlock (1205); transient-classified, but production has more than one writer | p3 | **#124** | | 4 | 30-day site event retention holds only below ~0.91 events/s per site; above that the 1 GB cap governs | operational note | — (documented here and in §3.1; raise `MaxStorageMb` per site if needed) | | 5 | A burst larger than the 10,000-event write queue, arriving faster than the writer drains, loses its oldest events (counted, surfaced on the health report) | by design | — | Phase-8 WP-4 is now covered for all of `[xc-1]`–`[xc-8]`: `[xc-1]`, `[xc-2]`, `[xc-6]` and `[xc-7]` by the 2026-08-15 run, and the four residuals by this one. ### 7.1 Follow-up fixes (same day) The three issues were fixed after the measurement run; the verdicts above stand, and the numbers in §3, §5 and §6 are the pre-fix measurements. - **#122.** The real global outbox KPI cost was 0.7–2.6 s, not the 1.3 s quoted in §6: the earlier probe ran under `sqlcmd` without `-I`, which disables filtered indexes. The OR'd pre-filter forced a clustered scan; the snapshot now runs three status-equality legs (live, parked, delivered) that each seek their own index, ~0.12 s on the rig. - **#123.** The `event_type` and `severity` indexes are now composites with `timestamp`. `ResidualBench eventlog` re-run: `severity=Error, 30 d` 2,287 → 15 ms cold and 115 → 1.6 ms warm; `event_type=alarm, 24 h` 216 → 1.2 ms cold. The cost: 454 → 530 B/row on disk, so the 1 GB cap holds 2.03 M rows instead of 2.36 M and 30 days fit only below 0.78 events/s per site (was 0.91); the CDC-on writer drains 8,760 events/s instead of 10,535. - **#124.** The set-based ingest's `NOT EXISTS` probe is forced to seek. `ResidualBench audit-contention`: 4 writers 775 rows/s with 76 deadlock victims in 30 s → 5,214 rows/s with none; 8 writers 8,607 rows/s with none. ## 8. What this does not prove - **The rig has few instances.** `[xc-8]` ran against the docker rig: 14 instances and 36 templates, though its audit and notification tables are large (10.9 M / 4.35 M rows). Page cost that grows with fleet size (Topology, Instances, pending updates) is not exercised at 10 sites × 500 instances here. The audit and flattening benches cover the two biggest central data-volume costs separately. - **Emulated SQL Server.** `[xc-4]` absolute numbers come from an amd64 SQL Server under emulation. Treat them as a pessimistic lower bound on throughput, not a production sizing. - **No WAN.** As in the 2026-08-15 run, site↔central hops are loopback. ## 9. Reproducing ```bash dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- eventlog # ~4 min, ~1.1 GB temp disk dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- flatten # ~20 s dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- audit # needs MS SQL (default: local scadabridge-mssql sa) bash docker/measure-ui-responsiveness.sh # needs the rig + seed-mes-alarm-demo.sh ``` Options: `eventlog --data-dir --calibration-rows --write-events --skip-full`; `flatten --instances --repeats`; `audit --connection-string --seed-rows --phase-seconds --keep`; `audit-contention --writers --seconds --seed-rows` (#124); UI `--operators --api-rps --repeats`. Unknown options are rejected.