2b80a6defa
New ResidualBench tool (eventlog, flatten, audit, audit-contention) and docker/measure-ui-responsiveness.sh. All four residuals pass; results in docs/plans/2026-11-10-wp4-residual-scope-results.md. Findings filed as #122, #123 and #124 (fixed in the following commit).
340 lines
21 KiB
Markdown
340 lines
21 KiB
Markdown
# WP-4 residual scope — results (`[xc-3]`, `[xc-4]`, `[xc-5]`, `[xc-8]`, ScadaBridge#49)
|
||
|
||
**Date:** 2026-11-10
|
||
**Closes:** the four WP-4 sub-criteria that the target-scale load test left open
|
||
(`docs/plans/2026-08-15-target-scale-load-test-results.md` §6, deferred-work register row 30 → #49).
|
||
**Tools (committed):** `tests/ZB.MOM.WW.ScadaBridge.ResidualBench/` (modes `eventlog`, `flatten`, `audit`)
|
||
and `docker/measure-ui-responsiveness.sh`.
|
||
|
||
## 1. Environment
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host | Apple M4 Pro, 14 logical cores, 48 GB RAM, macOS 26.5.2 |
|
||
| Runtime | .NET 10.0.14, Release, server GC |
|
||
| Commit | `48517b49` + the bench tool in this change |
|
||
| Rig data (for `[xc-8]`) | 14 instances, 36 templates, **10.9 M `AuditLog` rows, 4.35 M `Notifications`** (944,597 Parked) from months of soak traffic |
|
||
| SQL Server | `scadabridge-mssql` — SQL Server 2022 RTM-CU22 Developer, **amd64 image under emulation** on Apple silicon (OrbStack, 14 vCPU / 16 GB VM). Absolute SQL timings are pessimistic; compare ratios. |
|
||
| Concurrent load | The 8-node `docker/` rig stayed up throughout (it is the target of `[xc-8]`), as in the 2026-08-15 run. Benchmarks were run one at a time, never overlapping. |
|
||
|
||
## 2. Verdict summary
|
||
|
||
| Criterion | Threshold used | Measured | Verdict |
|
||
|---|---|---|---|
|
||
| `[xc-3]` site event logging handles volume within the 30-day / 1 GB limits | writer keeps up with any realistic event rate without loss; queries usable on a full 1 GB log; the cap is enforced | 1 GB = 2.36 M events = 30 days only at ≤ 0.91 events/s per site; writer 10,500 events/s with CDC (11,500× that); storms ≤ 10,000 absorbed with 0 drops; every query ≤ 0.5 s warm on a full 1 GB log; one purge tick restores the cap in 12.8 s; no writer impact from concurrent scans | **PASS**, with an operational caveat: above ~1 event/s the **1 GB cap**, not the 30 days, sets retention (§3.1) |
|
||
| `[xc-4]` audit logging does not degrade central | central read P95 stays inside the UI budget (< 1 s) while ingest runs at and above the expected fleet rate | 1 M rows / 3.26 GB seeded through the production ingest path; reads P95 ≤ 22 ms up to 500 rows/s ingest, ≤ 160 ms at the single-writer ceiling (~1,000–1,500 rows/s); non-audit reads unaffected; 0 deadlocks with one writer | **PASS**, with an observation: concurrent set-based inserts deadlock (1205, #124) |
|
||
| `[xc-5]` flattening/validation within acceptable time for large templates | deploy-gate validate of one large template < 5 s; 500-instance bulk CPU well inside the 120 s site-deploy budget | 4,400-attribute / 400-alarm / 220-script, 8-deep template: flatten 3.6 ms, deploy-gate validate 1.9 s cold / 2.3 ms warm, hash 5.7 ms; 500 instances flatten+validate+hash in 3.5 s | **PASS** (34× inside the 120 s budget) |
|
||
| `[xc-8]` UI workflows remain responsive | page usable P95 < 3 s under load, no page > 10 s | 14 pages, rig with 10.9 M AuditLog / 4.35 M Notification rows; under 6 concurrent operator sessions + 20 inbound API calls/s: P50 46 ms, P95 1.18 s, max 3.0 s (Audit Log). Idle: every page ≤ 0.7 s **except Health, up to 5.05 s** | **PASS** against the threshold, with one defect: the Health dashboard blocks up to 5 s on a cold KPI cache (#122) |
|
||
|
||
The requirements give no numeric thresholds for these four. The thresholds above were chosen
|
||
for this run and are stated so the verdicts can be re-judged.
|
||
|
||
---
|
||
|
||
## 3. `[xc-3]` Site event log — `ResidualBench eventlog`
|
||
|
||
Real `SiteEventLogger`, `EventLogQueryService` and `EventLogPurgeService` over a real
|
||
`ZB.MOM.WW.LocalDb` file. The event mix models the post-WP3.2 stream (per-run script events
|
||
off): 45% alarm transitions, 25% script failures/warnings, 15% connection quality changes,
|
||
10% S&F parks, 5% lifecycle; 2-in-3 rows carry an instance id and 1-in-10 carry an ~800-byte
|
||
stack trace.
|
||
|
||
### 3.1 What 1 GB holds
|
||
|
||
| | |
|
||
|---|---|
|
||
| On-disk bytes per row (table + 4 indexes + PK) | **454** |
|
||
| Rows at the 1 GB cap | **2,364,426** |
|
||
| Sustained rate that keeps 30 days inside 1 GB | **≤ 0.91 events/s per site** (78,800/day; **158 per instance per day** at 500 instances) |
|
||
| Retention actually achieved at 5 events/s | 5.5 days |
|
||
| Retention actually achieved at 20 events/s | 1.4 days |
|
||
|
||
**The binding limit is the 1 GB cap, not the 30-day window, for any site averaging more than
|
||
about one event per second.** That is by design — `MaxStorageMb` wins over `RetentionDays`
|
||
(the cap trim deletes oldest-first) — but it means "30-day event history" holds only for quiet
|
||
sites. A 500-machine site whose alarms average more than ~150 transitions per machine per
|
||
day will keep less than 30 days. Raise `ScadaBridge:SiteEventLog:MaxStorageMb` for such
|
||
sites if 30 days matters; disk cost is ~0.45 KB per event.
|
||
|
||
### 3.2 Writer throughput
|
||
|
||
| | CDC off (unreplicated site) | CDC on (`RegisterReplicated`, the replicated site-a shape) |
|
||
|---|---|---|
|
||
| Sustained drain (200,000 events, offered in 5,000-event windows) | **34,000 events/s**, 0 failed | **10,500 events/s**, 0 failed |
|
||
| vs the 30-day-in-1-GB rate | 37,000× | 11,500× |
|
||
| Storm of 5,000 / 10,000 events offered at once | settled in 0.23 / 0.42 s, **0 dropped** | 0.70 / 1.43 s, **0 dropped** |
|
||
| Storm of 25,000 events offered at once | **14,723 dropped** | **14,733 dropped** |
|
||
|
||
The bounded write queue (`WriteQueueCapacity` 10,000, DropOldest) absorbs any single burst
|
||
up to its capacity. A burst much larger than that, arriving faster than the writer drains,
|
||
loses the oldest queued events — counted in `FailedWriteCount` and surfaced on the health
|
||
report as `SiteEventLogWriteFailures`, never silent. CDC capture triggers cost ~3.2× on
|
||
write throughput; the replicated rate is still four orders of magnitude above the rate the
|
||
storage cap can retain.
|
||
|
||
### 3.3 Queries on a full 1 GB, 30-day log (page size 500)
|
||
|
||
| query (page size 500) | first ms | warm P50 ms | warm max ms | rows | has more |
|
||
|---|---|---|---|---|---|
|
||
| no filter, page 1 (oldest first) | 8.5 | 1.1 | 1.1 | 500 | True |
|
||
| last 1 h | 1.3 | 1.1 | 1.1 | 500 | True |
|
||
| last 24 h | 1.2 | 1.0 | 1.0 | 500 | True |
|
||
| severity=Error, 30 d | 2286.8 | 114.5 | 485.6 | 500 | True |
|
||
| severity=Error, last 24 h | 117.3 | 104.5 | 110.0 | 500 | True |
|
||
| event_type=alarm, last 24 h | 215.6 | 148.5 | 168.1 | 500 | True |
|
||
| instance=Machine-0042, 30 d | 4.3 | 3.7 | 4.3 | 500 | True |
|
||
| keyword 'timeout' (clamped to 7 d) | 1.5 | 0.8 | 0.9 | 500 | True |
|
||
| keyword with no match (full 7 d window scan) | 91.9 | 91.9 | 105.8 | 0 | False |
|
||
| keyword 'Machine-0042', last 24 h | 14.2 | 13.3 | 13.5 | 105 | False |
|
||
| 20 keyset pages, last 24 h | | 0.7 | 4.1 | 500/page | |
|
||
|
||
Every shape is under 0.5 s warm. The slowest are equality filters on `severity` or
|
||
`event_type`: SQLite (no `ANALYZE` statistics) picks the single-column index, then reads and
|
||
sorts *every* matching row in the table before returning the first page (`USE TEMP B-TREE FOR
|
||
ORDER BY`), so their cost grows with table size, not page size — 2.3 s on the first, cold
|
||
run. Adding composite indexes would fix it, but would also change the row model that the
|
||
storage-cap estimate depends on (register row 93). This is filed as a low-priority follow-up,
|
||
not a failure.
|
||
|
||
### 3.4 Writer while queries run
|
||
|
||
200 events/s offered for 20 s (≈ 220× the 30-day rate), first with no queries, then while a
|
||
back-to-back worst-case keyword scan (no match, a full 7-day-window scan, ~92 ms each) ran on
|
||
another thread:
|
||
|
||
- writer at 200 ev/s, idle: enqueue->durable n=3999 P50 0.39 ms, P95 1.46 ms, P99 17.27 ms, max 33.44 ms; failed 0
|
||
- writer at 200 ev/s, with a back-to-back no-match keyword scan loop (207 scans): enqueue->durable n=3999 P50 0.13 ms, P95 0.55 ms, P99 0.96 ms, max 43.38 ms; failed 0
|
||
|
||
No degradation: reads use their own WAL connection and never take the writer's lock
|
||
(review 2026-10-21 02/P1), and the numbers confirm it.
|
||
|
||
### 3.5 Cap enforcement
|
||
|
||
One real purge tick on a table 10% over the cap: **2,608,866 → 2,379,761 rows in 12.8 s**
|
||
(229,105 deleted, under the 250,000 per-tick ceiling, so a single tick restored the cap to
|
||
within the size model's error). Consistent with the 2026-08-15 measurement (~17 s for a full
|
||
250,000-row tick).
|
||
|
||
---
|
||
|
||
## 4. `[xc-5]` Template flattening and validation — `ResidualBench flatten`
|
||
|
||
Production `FlatteningService` → `ValidationService.Validate` (the deploy gate: trust check
|
||
plus a real Roslyn compile of every script, plus all semantic checks) → `RevisionHashService`,
|
||
in-process. Each level of the inheritance chain adds attributes/alarms/scripts and overrides
|
||
a fifth of its parent's attribute values. Feature modules are composed on the base template
|
||
and each composes one sub-module. The instance overrides 10% of the attributes and binds
|
||
every data-sourced attribute. Scripts are ~35 lines each, and every script's code is unique,
|
||
so a cold compile really compiles every one. Not in frame: the configuration-database read
|
||
`FlatteningPipeline` does before this (one `FlattenSession` per batch).
|
||
|
||
| shape | chain depth | flattened attrs / alarms / scripts | flatten P50 ms | validate, cold compile ms | validate, warm cache P50 ms | validate, compile off P50 ms | hash P50 ms | errors |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| typical | 2 | 100 / 12 / 8 | 0.05 | 402 | 0.08 | 0.07 | 0.10 | 0 |
|
||
| large | 5 | 1,100 / 100 / 60 | 0.70 | 1549 | 0.77 | 0.63 | 1.62 | 0 |
|
||
| extra-large | 8 | 4,400 / 400 / 220 | 3.62 | 1876 | 2.27 | 1.89 | 5.73 | 0 |
|
||
|
||
- large: flatten+validate+hash of 500 instances sharing one template, serial: 1.97 s total; per instance n=500 P50 2.81 ms, P95 3.45 ms, P99 7.22 ms, max 477.72 ms
|
||
- extra-large: flatten+validate+hash of 500 instances sharing one template, serial: 3.49 s total; per instance n=500 P50 3.80 ms, P95 5.14 ms, P99 18.59 ms, max 1282.61 ms
|
||
|
||
- Flattening itself is milliseconds even for the 4,400-attribute shape.
|
||
- The cost is the first, cold compile of a template's scripts — about 1.5–2 s for 60–220
|
||
unique scripts — and it is paid once per distinct script body, thanks to
|
||
`ScriptCompileVerdictCache`.
|
||
- After that, every instance of the template is a cache hit. A 500-instance bulk deploy of
|
||
the extra-large template spends 3.5 s of central CPU in flatten+validate+hash, versus the
|
||
120 s budget for deploying 500 instances to a site.
|
||
|
||
---
|
||
|
||
## 5. `[xc-4]` Audit logging vs central — `ResidualBench audit`
|
||
|
||
A throwaway database on `scadabridge-mssql`, migrated with the production EF chain (the
|
||
partitioned `AuditLog`, clustered `(OccurredAtUtc, EventId)` from #111, all NC indexes),
|
||
dropped afterwards. Rows are site-shaped (~1.5 KB, 10 sites × 500 instances, 4% failures) and
|
||
inserted through the production `AuditLogRepository.InsertManyIfNotExistsAsync` in 256-row
|
||
packets — the same set-based anti-semi-join insert `AuditLogIngestActor` runs.
|
||
|
||
**One serial writer, as in production.** `AuditLogIngestActor` is a cluster singleton that
|
||
awaits each packet inside `ReceiveAsync`, so central never runs two site-ingest inserts at
|
||
once. A first attempt with four concurrent writers hit **SQL error 1205 (deadlock victim)**
|
||
within seconds. Production can still see concurrent writers on this table (the
|
||
reconciliation singleton, central direct writes), and `AuditIngestErrorClassifier` already
|
||
treats 1205 as transient, so the site re-sends. That is recorded here as an observation, not
|
||
a defect.
|
||
|
||
```
|
||
seeded 1,000,192 rows (45 days) in 934 s = 1,071 rows/s (1 serial writer, 256-row packets)
|
||
AuditLog size 3259 MB
|
||
```
|
||
|
||
### no ingest: achieved ingest 0 rows/s, deadlock retries so far 0
|
||
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|
||
|---|---|---|---|---|---|
|
||
| Audit Log page 1 (newest 50, no filter) | 1.8 | 10.4 | 18.9 | 149.5 | 3052 |
|
||
| channel=ApiOutbound, last 24 h, 50 | 2.0 | 10.2 | 20.2 | 43.0 | 3052 |
|
||
| site=site-07, last 7 d, 100 | 3.4 | 11.2 | 24.7 | 59.0 | 3052 |
|
||
| status=Failed, last 24 h, 50 | 2.9 | 10.7 | 24.3 | 66.9 | 3052 |
|
||
| KPI snapshot, 1 h window | 1.0 | 10.0 | 17.3 | 33.0 | 3052 |
|
||
| non-audit: templates + sites list | 1.1 | 14.1 | 26.5 | 46.5 | 3052 |
|
||
|
||
### ingest 100 rows/s: achieved ingest 98 rows/s, deadlock retries so far 0; 256-row packet n=23 P50 230.05 ms, P95 495.30 ms, P99 731.88 ms, max 731.88 ms
|
||
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|
||
|---|---|---|---|---|---|
|
||
| Audit Log page 1 (newest 50, no filter) | 2.4 | 11.2 | 20.6 | 598.0 | 1635 |
|
||
| channel=ApiOutbound, last 24 h, 50 | 2.7 | 11.2 | 12.7 | 78.0 | 1635 |
|
||
| site=site-07, last 7 d, 100 | 4.4 | 13.1 | 30.3 | 166.5 | 1635 |
|
||
| status=Failed, last 24 h, 50 | 4.0 | 12.3 | 24.6 | 151.0 | 1635 |
|
||
| KPI snapshot, 1 h window | 2.9 | 10.7 | 17.0 | 214.7 | 1635 |
|
||
| non-audit: templates + sites list | 1.9 | 21.0 | 23.3 | 56.9 | 1635 |
|
||
|
||
### ingest 500 rows/s: achieved ingest 499 rows/s, deadlock retries so far 0; 256-row packet n=117 P50 203.94 ms, P95 577.36 ms, P99 713.85 ms, max 1133.67 ms
|
||
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|
||
|---|---|---|---|---|---|
|
||
| Audit Log page 1 (newest 50, no filter) | 2.7 | 22.2 | 122.9 | 247.6 | 1273 |
|
||
| channel=ApiOutbound, last 24 h, 50 | 2.8 | 11.2 | 104.4 | 1143.9 | 1273 |
|
||
| site=site-07, last 7 d, 100 | 4.3 | 18.8 | 137.8 | 318.8 | 1273 |
|
||
| status=Failed, last 24 h, 50 | 3.7 | 12.8 | 100.5 | 293.0 | 1273 |
|
||
| KPI snapshot, 1 h window | 8.8 | 19.2 | 114.2 | 356.7 | 1273 |
|
||
| non-audit: templates + sites list | 1.8 | 20.5 | 21.7 | 36.5 | 1273 |
|
||
|
||
### ingest 2,000 rows/s: achieved ingest 1,474 rows/s, deadlock retries so far 0; 256-row packet n=346 P50 125.09 ms, P95 397.36 ms, P99 732.35 ms, max 956.29 ms
|
||
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|
||
|---|---|---|---|---|---|
|
||
| Audit Log page 1 (newest 50, no filter) | 4.8 | 94.7 | 147.7 | 381.7 | 593 |
|
||
| channel=ApiOutbound, last 24 h, 50 | 3.5 | 30.4 | 90.7 | 1380.4 | 593 |
|
||
| site=site-07, last 7 d, 100 | 5.1 | 42.9 | 146.6 | 1750.2 | 593 |
|
||
| status=Failed, last 24 h, 50 | 4.3 | 25.9 | 128.8 | 359.8 | 593 |
|
||
| KPI snapshot, 1 h window | 30.6 | 135.4 | 188.5 | 704.0 | 593 |
|
||
| non-audit: templates + sites list | 2.1 | 3.1 | 4.9 | 27.2 | 593 |
|
||
|
||
### ingest unthrottled (1 serial writer): achieved ingest 1,032 rows/s, deadlock retries so far 0; 256-row packet n=243 P50 190.53 ms, P95 557.43 ms, P99 900.67 ms, max 1223.08 ms
|
||
| read | P50 ms | P95 ms | P99 ms | max ms | n |
|
||
|---|---|---|---|---|---|
|
||
| Audit Log page 1 (newest 50, no filter) | 12.3 | 116.8 | 199.9 | 284.5 | 382 |
|
||
| channel=ApiOutbound, last 24 h, 50 | 3.7 | 13.3 | 107.7 | 159.1 | 382 |
|
||
| site=site-07, last 7 d, 100 | 7.4 | 49.9 | 125.3 | 644.6 | 382 |
|
||
| status=Failed, last 24 h, 50 | 4.7 | 76.7 | 176.2 | 1491.2 | 382 |
|
||
| KPI snapshot, 1 h window | 69.3 | 160.0 | 345.6 | 681.9 | 382 |
|
||
| non-audit: templates + sites list | 2.3 | 4.2 | 11.8 | 42.2 | 382 |
|
||
|
||
|
||
Reading the numbers:
|
||
|
||
- **Ingest ceiling, one serial writer: ~1,000–1,500 rows/s** on this emulated SQL Server.
|
||
The seed sustained 1,071 rows/s while the table grew to 1 M rows / 3.26 GB. Each
|
||
256-row packet takes 125–230 ms P50. That is the central ingest capacity on this box, and
|
||
it is far above any audit volume the rig produces.
|
||
- **Reads barely move at realistic rates.** At 100 and 500 rows/s every read's P95 stays
|
||
≤ 22 ms, against 10–14 ms with no ingest.
|
||
- **At the ceiling, P95 reaches 117–160 ms.** The worst are the Audit Log page 1 and the KPI
|
||
snapshot, which compete with the insert for the newest partition's pages. There are rare
|
||
outliers up to 1.75 s. Still well inside the 1 s P95 budget.
|
||
- **Non-audit central reads are unaffected** (P95 ≤ 21 ms in every phase).
|
||
- **Deadlocks: 0** with one writer. Four concurrent writers deadlocked within seconds (see
|
||
above) — **#124** (p3).
|
||
|
||
---
|
||
|
||
## 6. `[xc-8]` UI responsiveness — `docker/measure-ui-responsiveness.sh`
|
||
|
||
Real Chromium (host Playwright) against the rig through Traefik. A load is a full navigation,
|
||
timed until the page is usable: the Blazor circuit is up and no spinner or "Loading…" text
|
||
remains. 14 pages, 5 loads each, idle first and then under load.
|
||
|
||
**Under load**: 6 operator sessions (7.0 page loads/s), inbound API 20.0 calls/s offered, 652/652 HTTP 200, API latency P50 92 ms / P95 980 ms / max 1264 ms
|
||
|
||
| page | idle P50 ms | idle max ms | load P50 ms | load max ms |
|
||
|---|---|---|---|---|
|
||
| Dashboard | 29 | 32 | 52 | 86 |
|
||
| Health | 3362 | 5048 | 38 | 128 |
|
||
| Topology | 38 | 52 | 148 | 201 |
|
||
| Instances | 42 | 64 | 74 | 141 |
|
||
| Deployments | 54 | 57 | 39 | 247 |
|
||
| Templates | 61 | 74 | 144 | 455 |
|
||
| Audit Log | 425 | 696 | 1081 | 3046 |
|
||
| Config audit | 25 | 27 | 26 | 33 |
|
||
| Site Calls | 41 | 47 | 51 | 61 |
|
||
| Notifications | 358 | 365 | 962 | 1627 |
|
||
| Alarms | 25 | 28 | 26 | 27 |
|
||
| Event Logs | 27 | 29 | 27 | 35 |
|
||
| Sites | 28 | 33 | 38 | 59 |
|
||
| Flow Debugger | 33 | 89 | 31 | 38 |
|
||
| **all pages** | P50 40, P95 696 | max 5048 | P50 46, P95 1177 | max 3046 |
|
||
|
||
Reading the numbers:
|
||
|
||
- **Every page except Health is usable in ≤ 0.7 s idle**; most in 25–60 ms on loopback.
|
||
- **Health is the outlier: 3.4 s P50, up to 5.05 s idle, but 38 ms under load.** The
|
||
Notification Outbox KPI tile exceeds the page's 5 s `TileTimeout`. On this rig the outbox
|
||
holds 944,597 Parked rows, and counting them is a 374,043-read clustered scan (1.3 s per
|
||
query, warm). A timed-out snapshot is not cached, and the tiles load inside prerender, so
|
||
a lone viewer waits the full timeout on every load. Under load, another session's
|
||
successful refresh keeps the cache warm, which is why the number drops. Filed as **#122**
|
||
(p2).
|
||
- **Audit Log** (1.1 s P50 / 3.0 s max under load) and **Notifications** (1.0 s / 1.6 s)
|
||
are the slowest pages under load. Both read the rig's largest tables (10.9 M audit rows,
|
||
4.35 M notifications) and stay inside the 3 s budget.
|
||
- The inbound API kept up throughout: 652/652 HTTP 200 at 20 calls/s, P95 0.98 s,
|
||
including the `Route.To()` round trip to site-a.
|
||
|
||
|
||
---
|
||
|
||
## 7. Findings and follow-ups
|
||
|
||
| # | Finding | Severity | Issue |
|
||
|---|---|---|---|
|
||
| 1 | Health dashboard waits up to 5 s for a timed-out Notification Outbox KPI tile when the outbox holds ~1 M Parked rows (uncached timeout, prerender blocks) | p2 bug | **#122** |
|
||
| 2 | Event-log `severity` / `event_type` filters read and sort every matching row (2.3 s cold, ~0.1–0.15 s warm at 1 GB) | p3 | **#123** |
|
||
| 3 | Concurrent `AuditLog` set-based inserts deadlock (1205); transient-classified, but production has more than one writer | p3 | **#124** |
|
||
| 4 | 30-day site event retention holds only below ~0.91 events/s per site; above that the 1 GB cap governs | operational note | — (documented here and in §3.1; raise `MaxStorageMb` per site if needed) |
|
||
| 5 | A burst larger than the 10,000-event write queue, arriving faster than the writer drains, loses its oldest events (counted, surfaced on the health report) | by design | — |
|
||
|
||
Phase-8 WP-4 is now covered for all of `[xc-1]`–`[xc-8]`: `[xc-1]`, `[xc-2]`, `[xc-6]` and
|
||
`[xc-7]` by the 2026-08-15 run, and the four residuals by this one.
|
||
|
||
|
||
### 7.1 Follow-up fixes (same day)
|
||
|
||
The three issues were fixed after the measurement run; the verdicts above stand, and the numbers
|
||
in §3, §5 and §6 are the pre-fix measurements.
|
||
|
||
- **#122.** The real global outbox KPI cost was 0.7–2.6 s, not the 1.3 s quoted in §6: the earlier
|
||
probe ran under `sqlcmd` without `-I`, which disables filtered indexes. The OR'd pre-filter
|
||
forced a clustered scan; the snapshot now runs three status-equality legs (live, parked,
|
||
delivered) that each seek their own index, ~0.12 s on the rig.
|
||
- **#123.** The `event_type` and `severity` indexes are now composites with `timestamp`.
|
||
`ResidualBench eventlog` re-run: `severity=Error, 30 d` 2,287 → 15 ms cold and 115 → 1.6 ms
|
||
warm; `event_type=alarm, 24 h` 216 → 1.2 ms cold. The cost: 454 → 530 B/row on disk, so the
|
||
1 GB cap holds 2.03 M rows instead of 2.36 M and 30 days fit only below 0.78 events/s per site
|
||
(was 0.91); the CDC-on writer drains 8,760 events/s instead of 10,535.
|
||
- **#124.** The set-based ingest's `NOT EXISTS` probe is forced to seek. `ResidualBench
|
||
audit-contention`: 4 writers 775 rows/s with 76 deadlock victims in 30 s → 5,214 rows/s with
|
||
none; 8 writers 8,607 rows/s with none.
|
||
|
||
## 8. What this does not prove
|
||
|
||
- **The rig has few instances.** `[xc-8]` ran against the docker rig: 14 instances and 36
|
||
templates, though its audit and notification tables are large (10.9 M / 4.35 M rows). Page
|
||
cost that grows with fleet size (Topology, Instances, pending updates) is not exercised at
|
||
10 sites × 500 instances here. The audit and flattening benches cover the two biggest
|
||
central data-volume costs separately.
|
||
- **Emulated SQL Server.** `[xc-4]` absolute numbers come from an amd64 SQL Server under
|
||
emulation. Treat them as a pessimistic lower bound on throughput, not a production sizing.
|
||
- **No WAN.** As in the 2026-08-15 run, site↔central hops are loopback.
|
||
|
||
## 9. Reproducing
|
||
|
||
```bash
|
||
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- eventlog # ~4 min, ~1.1 GB temp disk
|
||
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- flatten # ~20 s
|
||
dotnet run -c Release --project tests/ZB.MOM.WW.ScadaBridge.ResidualBench -- audit # needs MS SQL (default: local scadabridge-mssql sa)
|
||
bash docker/measure-ui-responsiveness.sh # needs the rig + seed-mes-alarm-demo.sh
|
||
```
|
||
|
||
Options: `eventlog --data-dir --calibration-rows --write-events --skip-full`;
|
||
`flatten --instances --repeats`; `audit --connection-string --seed-rows --phase-seconds --keep`;
|
||
`audit-contention --writers --seconds --seed-rows` (#124); UI `--operators --api-rps --repeats`.
|
||
Unknown options are rejected.
|