New ResidualBench tool (eventlog, flatten, audit, audit-contention) and docker/measure-ui-responsiveness.sh. All four residuals pass; results in docs/plans/2026-11-10-wp4-residual-scope-results.md. Findings filed as #122, #123 and #124 (fixed in the following commit).
21 KiB
WP-4 residual scope — results ([xc-3], [xc-4], [xc-5], [xc-8], ScadaBridge#49)
Date: 2026-11-10
Closes: the four WP-4 sub-criteria that the target-scale load test left open
(docs/plans/2026-08-15-target-scale-load-test-results.md §6, deferred-work register row 30 → #49).
Tools (committed): tests/ZB.MOM.WW.ScadaBridge.ResidualBench/ (modes eventlog, flatten, audit)
and docker/measure-ui-responsiveness.sh.
1. Environment
| Host | Apple M4 Pro, 14 logical cores, 48 GB RAM, macOS 26.5.2 |
| Runtime | .NET 10.0.14, Release, server GC |
| Commit | 48517b49 + the bench tool in this change |
Rig data (for [xc-8]) |
14 instances, 36 templates, 10.9 M AuditLog rows, 4.35 M Notifications (944,597 Parked) from months of soak traffic |
| SQL Server | scadabridge-mssql — SQL Server 2022 RTM-CU22 Developer, amd64 image under emulation on Apple silicon (OrbStack, 14 vCPU / 16 GB VM). Absolute SQL timings are pessimistic; compare ratios. |
| Concurrent load | The 8-node docker/ rig stayed up throughout (it is the target of [xc-8]), as in the 2026-08-15 run. Benchmarks were run one at a time, never overlapping. |
2. Verdict summary
| Criterion | Threshold used | Measured | Verdict |
|---|---|---|---|
[xc-3] site event logging handles volume within the 30-day / 1 GB limits |
writer keeps up with any realistic event rate without loss; queries usable on a full 1 GB log; the cap is enforced | 1 GB = 2.36 M events = 30 days only at ≤ 0.91 events/s per site; writer 10,500 events/s with CDC (11,500× that); storms ≤ 10,000 absorbed with 0 drops; every query ≤ 0.5 s warm on a full 1 GB log; one purge tick restores the cap in 12.8 s; no writer impact from concurrent scans | PASS, with an operational caveat: above ~1 event/s the 1 GB cap, not the 30 days, sets retention (§3.1) |
[xc-4] audit logging does not degrade central |
central read P95 stays inside the UI budget (< 1 s) while ingest runs at and above the expected fleet rate | 1 M rows / 3.26 GB seeded through the production ingest path; reads P95 ≤ 22 ms up to 500 rows/s ingest, ≤ 160 ms at the single-writer ceiling (~1,000–1,500 rows/s); non-audit reads unaffected; 0 deadlocks with one writer | PASS, with an observation: concurrent set-based inserts deadlock (1205, #124) |
[xc-5] flattening/validation within acceptable time for large templates |
deploy-gate validate of one large template < 5 s; 500-instance bulk CPU well inside the 120 s site-deploy budget | 4,400-attribute / 400-alarm / 220-script, 8-deep template: flatten 3.6 ms, deploy-gate validate 1.9 s cold / 2.3 ms warm, hash 5.7 ms; 500 instances flatten+validate+hash in 3.5 s | PASS (34× inside the 120 s budget) |
[xc-8] UI workflows remain responsive |
page usable P95 < 3 s under load, no page > 10 s | 14 pages, rig with 10.9 M AuditLog / 4.35 M Notification rows; under 6 concurrent operator sessions + 20 inbound API calls/s: P50 46 ms, P95 1.18 s, max 3.0 s (Audit Log). Idle: every page ≤ 0.7 s except Health, up to 5.05 s | PASS against the threshold, with one defect: the Health dashboard blocks up to 5 s on a cold KPI cache (#122) |
The requirements give no numeric thresholds for these four. The thresholds above were chosen for this run and are stated so the verdicts can be re-judged.
3. [xc-3] Site event log — ResidualBench eventlog
Real SiteEventLogger, EventLogQueryService and EventLogPurgeService over a real
ZB.MOM.WW.LocalDb file. The event mix models the post-WP3.2 stream (per-run script events
off): 45% alarm transitions, 25% script failures/warnings, 15% connection quality changes,
10% S&F parks, 5% lifecycle; 2-in-3 rows carry an instance id and 1-in-10 carry an ~800-byte
stack trace.
3.1 What 1 GB holds
| On-disk bytes per row (table + 4 indexes + PK) | 454 |
| Rows at the 1 GB cap | 2,364,426 |
| Sustained rate that keeps 30 days inside 1 GB | ≤ 0.91 events/s per site (78,800/day; 158 per instance per day at 500 instances) |
| Retention actually achieved at 5 events/s | 5.5 days |
| Retention actually achieved at 20 events/s | 1.4 days |
The binding limit is the 1 GB cap, not the 30-day window, for any site averaging more than
about one event per second. That is by design — MaxStorageMb wins over RetentionDays
(the cap trim deletes oldest-first) — but it means "30-day event history" holds only for quiet
sites. A 500-machine site whose alarms average more than ~150 transitions per machine per
day will keep less than 30 days. Raise ScadaBridge:SiteEventLog:MaxStorageMb for such
sites if 30 days matters; disk cost is ~0.45 KB per event.
3.2 Writer throughput
| CDC off (unreplicated site) | CDC on (RegisterReplicated, the replicated site-a shape) |
|
|---|---|---|
| Sustained drain (200,000 events, offered in 5,000-event windows) | 34,000 events/s, 0 failed | 10,500 events/s, 0 failed |
| vs the 30-day-in-1-GB rate | 37,000× | 11,500× |
| Storm of 5,000 / 10,000 events offered at once | settled in 0.23 / 0.42 s, 0 dropped | 0.70 / 1.43 s, 0 dropped |
| Storm of 25,000 events offered at once | 14,723 dropped | 14,733 dropped |
The bounded write queue (WriteQueueCapacity 10,000, DropOldest) absorbs any single burst
up to its capacity. A burst much larger than that, arriving faster than the writer drains,
loses the oldest queued events — counted in FailedWriteCount and surfaced on the health
report as SiteEventLogWriteFailures, never silent. CDC capture triggers cost ~3.2× on
write throughput; the replicated rate is still four orders of magnitude above the rate the
storage cap can retain.
3.3 Queries on a full 1 GB, 30-day log (page size 500)
| query (page size 500) | first ms | warm P50 ms | warm max ms | rows | has more |
|---|---|---|---|---|---|
| no filter, page 1 (oldest first) | 8.5 | 1.1 | 1.1 | 500 | True |
| last 1 h | 1.3 | 1.1 | 1.1 | 500 | True |
| last 24 h | 1.2 | 1.0 | 1.0 | 500 | True |
| severity=Error, 30 d | 2286.8 | 114.5 | 485.6 | 500 | True |
| severity=Error, last 24 h | 117.3 | 104.5 | 110.0 | 500 | True |
| event_type=alarm, last 24 h | 215.6 | 148.5 | 168.1 | 500 | True |
| instance=Machine-0042, 30 d | 4.3 | 3.7 | 4.3 | 500 | True |
| keyword 'timeout' (clamped to 7 d) | 1.5 | 0.8 | 0.9 | 500 | True |
| keyword with no match (full 7 d window scan) | 91.9 | 91.9 | 105.8 | 0 | False |
| keyword 'Machine-0042', last 24 h | 14.2 | 13.3 | 13.5 | 105 | False |
| 20 keyset pages, last 24 h | 0.7 | 4.1 | 500/page |
Every shape is under 0.5 s warm. The slowest are equality filters on severity or
event_type: SQLite (no ANALYZE statistics) picks the single-column index, then reads and
sorts every matching row in the table before returning the first page (USE TEMP B-TREE FOR ORDER BY), so their cost grows with table size, not page size — 2.3 s on the first, cold
run. Adding composite indexes would fix it, but would also change the row model that the
storage-cap estimate depends on (register row 93). This is filed as a low-priority follow-up,
not a failure.
3.4 Writer while queries run
200 events/s offered for 20 s (≈ 220× the 30-day rate), first with no queries, then while a back-to-back worst-case keyword scan (no match, a full 7-day-window scan, ~92 ms each) ran on another thread:
- writer at 200 ev/s, idle: enqueue->durable n=3999 P50 0.39 ms, P95 1.46 ms, P99 17.27 ms, max 33.44 ms; failed 0
- writer at 200 ev/s, with a back-to-back no-match keyword scan loop (207 scans): enqueue->durable n=3999 P50 0.13 ms, P95 0.55 ms, P99 0.96 ms, max 43.38 ms; failed 0
No degradation: reads use their own WAL connection and never take the writer's lock (review 2026-10-21 02/P1), and the numbers confirm it.
3.5 Cap enforcement
One real purge tick on a table 10% over the cap: 2,608,866 → 2,379,761 rows in 12.8 s (229,105 deleted, under the 250,000 per-tick ceiling, so a single tick restored the cap to within the size model's error). Consistent with the 2026-08-15 measurement (~17 s for a full 250,000-row tick).
4. [xc-5] Template flattening and validation — ResidualBench flatten
Production FlatteningService → ValidationService.Validate (the deploy gate: trust check
plus a real Roslyn compile of every script, plus all semantic checks) → RevisionHashService,
in-process. Each level of the inheritance chain adds attributes/alarms/scripts and overrides
a fifth of its parent's attribute values. Feature modules are composed on the base template
and each composes one sub-module. The instance overrides 10% of the attributes and binds
every data-sourced attribute. Scripts are ~35 lines each, and every script's code is unique,
so a cold compile really compiles every one. Not in frame: the configuration-database read
FlatteningPipeline does before this (one FlattenSession per batch).
| shape | chain depth | flattened attrs / alarms / scripts | flatten P50 ms | validate, cold compile ms | validate, warm cache P50 ms | validate, compile off P50 ms | hash P50 ms | errors |
|---|---|---|---|---|---|---|---|---|
| typical | 2 | 100 / 12 / 8 | 0.05 | 402 | 0.08 | 0.07 | 0.10 | 0 |
| large | 5 | 1,100 / 100 / 60 | 0.70 | 1549 | 0.77 | 0.63 | 1.62 | 0 |
| extra-large | 8 | 4,400 / 400 / 220 | 3.62 | 1876 | 2.27 | 1.89 | 5.73 | 0 |
-
large: flatten+validate+hash of 500 instances sharing one template, serial: 1.97 s total; per instance n=500 P50 2.81 ms, P95 3.45 ms, P99 7.22 ms, max 477.72 ms
-
extra-large: flatten+validate+hash of 500 instances sharing one template, serial: 3.49 s total; per instance n=500 P50 3.80 ms, P95 5.14 ms, P99 18.59 ms, max 1282.61 ms
-
Flattening itself is milliseconds even for the 4,400-attribute shape.
-
The cost is the first, cold compile of a template's scripts — about 1.5–2 s for 60–220 unique scripts — and it is paid once per distinct script body, thanks to
ScriptCompileVerdictCache. -
After that, every instance of the template is a cache hit. A 500-instance bulk deploy of the extra-large template spends 3.5 s of central CPU in flatten+validate+hash, versus the 120 s budget for deploying 500 instances to a site.
5. [xc-4] Audit logging vs central — ResidualBench audit
A throwaway database on scadabridge-mssql, migrated with the production EF chain (the
partitioned AuditLog, clustered (OccurredAtUtc, EventId) from #111, all NC indexes),
dropped afterwards. Rows are site-shaped (~1.5 KB, 10 sites × 500 instances, 4% failures) and
inserted through the production AuditLogRepository.InsertManyIfNotExistsAsync in 256-row
packets — the same set-based anti-semi-join insert AuditLogIngestActor runs.
One serial writer, as in production. AuditLogIngestActor is a cluster singleton that
awaits each packet inside ReceiveAsync, so central never runs two site-ingest inserts at
once. A first attempt with four concurrent writers hit SQL error 1205 (deadlock victim)
within seconds. Production can still see concurrent writers on this table (the
reconciliation singleton, central direct writes), and AuditIngestErrorClassifier already
treats 1205 as transient, so the site re-sends. That is recorded here as an observation, not
a defect.
seeded 1,000,192 rows (45 days) in 934 s = 1,071 rows/s (1 serial writer, 256-row packets)
AuditLog size 3259 MB
no ingest: achieved ingest 0 rows/s, deadlock retries so far 0
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|---|---|---|---|---|---|
| Audit Log page 1 (newest 50, no filter) | 1.8 | 10.4 | 18.9 | 149.5 | 3052 |
| channel=ApiOutbound, last 24 h, 50 | 2.0 | 10.2 | 20.2 | 43.0 | 3052 |
| site=site-07, last 7 d, 100 | 3.4 | 11.2 | 24.7 | 59.0 | 3052 |
| status=Failed, last 24 h, 50 | 2.9 | 10.7 | 24.3 | 66.9 | 3052 |
| KPI snapshot, 1 h window | 1.0 | 10.0 | 17.3 | 33.0 | 3052 |
| non-audit: templates + sites list | 1.1 | 14.1 | 26.5 | 46.5 | 3052 |
ingest 100 rows/s: achieved ingest 98 rows/s, deadlock retries so far 0; 256-row packet n=23 P50 230.05 ms, P95 495.30 ms, P99 731.88 ms, max 731.88 ms
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|---|---|---|---|---|---|
| Audit Log page 1 (newest 50, no filter) | 2.4 | 11.2 | 20.6 | 598.0 | 1635 |
| channel=ApiOutbound, last 24 h, 50 | 2.7 | 11.2 | 12.7 | 78.0 | 1635 |
| site=site-07, last 7 d, 100 | 4.4 | 13.1 | 30.3 | 166.5 | 1635 |
| status=Failed, last 24 h, 50 | 4.0 | 12.3 | 24.6 | 151.0 | 1635 |
| KPI snapshot, 1 h window | 2.9 | 10.7 | 17.0 | 214.7 | 1635 |
| non-audit: templates + sites list | 1.9 | 21.0 | 23.3 | 56.9 | 1635 |
ingest 500 rows/s: achieved ingest 499 rows/s, deadlock retries so far 0; 256-row packet n=117 P50 203.94 ms, P95 577.36 ms, P99 713.85 ms, max 1133.67 ms
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|---|---|---|---|---|---|
| Audit Log page 1 (newest 50, no filter) | 2.7 | 22.2 | 122.9 | 247.6 | 1273 |
| channel=ApiOutbound, last 24 h, 50 | 2.8 | 11.2 | 104.4 | 1143.9 | 1273 |
| site=site-07, last 7 d, 100 | 4.3 | 18.8 | 137.8 | 318.8 | 1273 |
| status=Failed, last 24 h, 50 | 3.7 | 12.8 | 100.5 | 293.0 | 1273 |
| KPI snapshot, 1 h window | 8.8 | 19.2 | 114.2 | 356.7 | 1273 |
| non-audit: templates + sites list | 1.8 | 20.5 | 21.7 | 36.5 | 1273 |
ingest 2,000 rows/s: achieved ingest 1,474 rows/s, deadlock retries so far 0; 256-row packet n=346 P50 125.09 ms, P95 397.36 ms, P99 732.35 ms, max 956.29 ms
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|---|---|---|---|---|---|
| Audit Log page 1 (newest 50, no filter) | 4.8 | 94.7 | 147.7 | 381.7 | 593 |
| channel=ApiOutbound, last 24 h, 50 | 3.5 | 30.4 | 90.7 | 1380.4 | 593 |
| site=site-07, last 7 d, 100 | 5.1 | 42.9 | 146.6 | 1750.2 | 593 |
| status=Failed, last 24 h, 50 | 4.3 | 25.9 | 128.8 | 359.8 | 593 |
| KPI snapshot, 1 h window | 30.6 | 135.4 | 188.5 | 704.0 | 593 |
| non-audit: templates + sites list | 2.1 | 3.1 | 4.9 | 27.2 | 593 |
ingest unthrottled (1 serial writer): achieved ingest 1,032 rows/s, deadlock retries so far 0; 256-row packet n=243 P50 190.53 ms, P95 557.43 ms, P99 900.67 ms, max 1223.08 ms
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|---|---|---|---|---|---|
| Audit Log page 1 (newest 50, no filter) | 12.3 | 116.8 | 199.9 | 284.5 | 382 |
| channel=ApiOutbound, last 24 h, 50 | 3.7 | 13.3 | 107.7 | 159.1 | 382 |
| site=site-07, last 7 d, 100 | 7.4 | 49.9 | 125.3 | 644.6 | 382 |
| status=Failed, last 24 h, 50 | 4.7 | 76.7 | 176.2 | 1491.2 | 382 |
| KPI snapshot, 1 h window | 69.3 | 160.0 | 345.6 | 681.9 | 382 |
| non-audit: templates + sites list | 2.3 | 4.2 | 11.8 | 42.2 | 382 |
Reading the numbers:
- Ingest ceiling, one serial writer: ~1,000–1,500 rows/s on this emulated SQL Server. The seed sustained 1,071 rows/s while the table grew to 1 M rows / 3.26 GB. Each 256-row packet takes 125–230 ms P50. That is the central ingest capacity on this box, and it is far above any audit volume the rig produces.
- Reads barely move at realistic rates. At 100 and 500 rows/s every read's P95 stays ≤ 22 ms, against 10–14 ms with no ingest.
- At the ceiling, P95 reaches 117–160 ms. The worst are the Audit Log page 1 and the KPI snapshot, which compete with the insert for the newest partition's pages. There are rare outliers up to 1.75 s. Still well inside the 1 s P95 budget.
- Non-audit central reads are unaffected (P95 ≤ 21 ms in every phase).
- Deadlocks: 0 with one writer. Four concurrent writers deadlocked within seconds (see above) — #124 (p3).
6. [xc-8] UI responsiveness — docker/measure-ui-responsiveness.sh
Real Chromium (host Playwright) against the rig through Traefik. A load is a full navigation, timed until the page is usable: the Blazor circuit is up and no spinner or "Loading…" text remains. 14 pages, 5 loads each, idle first and then under load.
Under load: 6 operator sessions (7.0 page loads/s), inbound API 20.0 calls/s offered, 652/652 HTTP 200, API latency P50 92 ms / P95 980 ms / max 1264 ms
| page | idle P50 ms | idle max ms | load P50 ms | load max ms |
|---|---|---|---|---|
| Dashboard | 29 | 32 | 52 | 86 |
| Health | 3362 | 5048 | 38 | 128 |
| Topology | 38 | 52 | 148 | 201 |
| Instances | 42 | 64 | 74 | 141 |
| Deployments | 54 | 57 | 39 | 247 |
| Templates | 61 | 74 | 144 | 455 |
| Audit Log | 425 | 696 | 1081 | 3046 |
| Config audit | 25 | 27 | 26 | 33 |
| Site Calls | 41 | 47 | 51 | 61 |
| Notifications | 358 | 365 | 962 | 1627 |
| Alarms | 25 | 28 | 26 | 27 |
| Event Logs | 27 | 29 | 27 | 35 |
| Sites | 28 | 33 | 38 | 59 |
| Flow Debugger | 33 | 89 | 31 | 38 |
| all pages | P50 40, P95 696 | max 5048 | P50 46, P95 1177 | max 3046 |
Reading the numbers:
- Every page except Health is usable in ≤ 0.7 s idle; most in 25–60 ms on loopback.
- Health is the outlier: 3.4 s P50, up to 5.05 s idle, but 38 ms under load. The
Notification Outbox KPI tile exceeds the page's 5 s
TileTimeout. On this rig the outbox holds 944,597 Parked rows, and counting them is a 374,043-read clustered scan (1.3 s per query, warm). A timed-out snapshot is not cached, and the tiles load inside prerender, so a lone viewer waits the full timeout on every load. Under load, another session's successful refresh keeps the cache warm, which is why the number drops. Filed as #122 (p2). - Audit Log (1.1 s P50 / 3.0 s max under load) and Notifications (1.0 s / 1.6 s) are the slowest pages under load. Both read the rig's largest tables (10.9 M audit rows, 4.35 M notifications) and stay inside the 3 s budget.
- The inbound API kept up throughout: 652/652 HTTP 200 at 20 calls/s, P95 0.98 s,
including the
Route.To()round trip to site-a.
7. Findings and follow-ups
| # | Finding | Severity | Issue |
|---|---|---|---|
| 1 | Health dashboard waits up to 5 s for a timed-out Notification Outbox KPI tile when the outbox holds ~1 M Parked rows (uncached timeout, prerender blocks) | p2 bug | #122 |
| 2 | Event-log severity / event_type filters read and sort every matching row (2.3 s cold, ~0.1–0.15 s warm at 1 GB) |
p3 | #123 |
| 3 | Concurrent AuditLog set-based inserts deadlock (1205); transient-classified, but production has more than one writer |
p3 | #124 |
| 4 | 30-day site event retention holds only below ~0.91 events/s per site; above that the 1 GB cap governs | operational note | — (documented here and in §3.1; raise MaxStorageMb per site if needed) |
| 5 | A burst larger than the 10,000-event write queue, arriving faster than the writer drains, loses its oldest events (counted, surfaced on the health report) | by design | — |
Phase-8 WP-4 is now covered for all of [xc-1]–[xc-8]: [xc-1], [xc-2], [xc-6] and
[xc-7] by the 2026-08-15 run, and the four residuals by this one.
7.1 Follow-up fixes (same day)
The three issues were fixed after the measurement run; the verdicts above stand, and the numbers in §3, §5 and §6 are the pre-fix measurements.
- #122. The real global outbox KPI cost was 0.7–2.6 s, not the 1.3 s quoted in §6: the earlier
probe ran under
sqlcmdwithout-I, which disables filtered indexes. The OR'd pre-filter forced a clustered scan; the snapshot now runs three status-equality legs (live, parked, delivered) that each seek their own index, ~0.12 s on the rig. - #123. The
event_typeandseverityindexes are now composites withtimestamp.ResidualBench eventlogre-run:severity=Error, 30 d2,287 → 15 ms cold and 115 → 1.6 ms warm;event_type=alarm, 24 h216 → 1.2 ms cold. The cost: 454 → 530 B/row on disk, so the 1 GB cap holds 2.03 M rows instead of 2.36 M and 30 days fit only below 0.78 events/s per site (was 0.91); the CDC-on writer drains 8,760 events/s instead of 10,535. - #124. The set-based ingest's
NOT EXISTSprobe is forced to seek.ResidualBench audit-contention: 4 writers 775 rows/s with 76 deadlock victims in 30 s → 5,214 rows/s with none; 8 writers 8,607 rows/s with none.
8. What this does not prove
- The rig has few instances.
[xc-8]ran against the docker rig: 14 instances and 36 templates, though its audit and notification tables are large (10.9 M / 4.35 M rows). Page cost that grows with fleet size (Topology, Instances, pending updates) is not exercised at 10 sites × 500 instances here. The audit and flattening benches cover the two biggest central data-volume costs separately. - Emulated SQL Server.
[xc-4]absolute numbers come from an amd64 SQL Server under emulation. Treat them as a pessimistic lower bound on throughput, not a production sizing. - No WAN. As in the 2026-08-15 run, site↔central hops are loopback.
9. Reproducing
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- eventlog # ~4 min, ~1.1 GB temp disk
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- flatten # ~20 s
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- audit # needs MS SQL (default: local scadabridge-mssql sa)
bash docker/measure-ui-responsiveness.sh # needs the rig + seed-mes-alarm-demo.sh
Options: eventlog --data-dir --calibration-rows --write-events --skip-full;
flatten --instances --repeats; audit --connection-string --seed-rows --phase-seconds --keep;
audit-contention --writers --seconds --seed-rows (#124); UI --operators --api-rps --repeats.
Unknown options are rejected.