diff --git a/docs/plans/2026-07-19-localdb-phase2-soak.md b/docs/plans/2026-07-19-localdb-phase2-soak.md new file mode 100644 index 00000000..99c69c86 --- /dev/null +++ b/docs/plans/2026-07-19-localdb-phase2-soak.md @@ -0,0 +1,277 @@ +# LocalDb Phase 2 — rig soak findings + +**Run date:** 2026-07-19 / 2026-07-20 (UTC) · **Rig:** local 8-node docker cluster, site-a pair +**Task:** Task 1 of [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md) + +> ## GATE VERDICT: **STOP — do not proceed to Task 3.** +> +> Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak +> instead surfaced a **pre-existing Phase 1 defect**: the consolidated LocalDb database +> (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every write on +> the **active** node under sustained concurrent load. Phase 2 moves eight more tables — +> including the two highest-volume ones — into exactly that database. See +> [Finding 1](#finding-1-blocker). + +--- + +## 1. Method as actually executed + +The plan's method needed four corrections before it would run. Recorded here so the next run +does not rediscover them. + +| Plan said | Reality | +|---|---| +| `docker exec … curl -s localhost:8084/metrics` | **No `curl` in the `aspnet:10.0` image.** Use a sidecar sharing the container netns: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` | +| Sample with host-side `sqlite3` against the bind mounts | **Unsafe under live writes** — SQLite's `-shm` shared-memory index is not reliable across the Docker VM boundary. Ultimately *not* the cause of Finding 1 (proven below), but it should not be done against a database the container is actively writing. Copy the files first, or read counters from `/metrics`. | +| Drive alarm churn on a deployed instance | **Not possible on this rig.** `infra/mssql/seed-config.sql` seeds zero `TemplateNativeAlarmSources`, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. `native_alarm_state` stayed at 0 rows throughout. Bounded analytically instead — see [Finding 3](#finding-3). | +| Drive S&F churn via template 4's `TestExternalSystem` script | Template 4 exists after a reseed but is **not deployable** (34 pre-deployment validation errors: 30 `ConnectionBinding` + 4 `ScriptCompilation`). A purpose-built `SoakGenerator` template was used instead — see below. | + +### The generator that worked + +`ExternalSystem.Call` does **not** buffer to store-and-forward in practice; `ExternalSystem.CachedCall` +is the buffering surface. This is the single most important operational detail for reproducing +S&F load. + +- Template `SoakGenerator` (id 2021), one `Interval` script at `{"intervalMs":5000}`: + ```csharp + var parms = new Dictionary { ["a"] = 2, ["b"] = 3 }; + await ExternalSystem.CachedCall("Test REST API", "Add", parms); + ``` +- No attributes, no compositions, no connection bindings — deliberately, so it deploys cleanly. +- `ExternalSystemDefinitions` id 1 repointed to `http://127.0.0.1:9` (discard port → connection + refused → classified transient → buffered). +- 4 instances (`soakgen-1..4`) deployed to site-a. + +Sustained rate observed: ~**2.9 HTTP attempts/sec** (688–864 connection-refused per 4 min). + +### Two rig-tooling bugs found and fixed en route + +1. **`docker/seed-sites.sh` seeded stale role names** — `Design`/`Deployment` instead of the + canonical `Designer`/`Deployer` (`src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs:46-47`). Every + Designer/Deployer-gated management command failed `UNAUTHORIZED` on a freshly reseeded rig, + including `seed-sites.sh`'s own trailing `deploy artifacts` and `reseed.sh` stage 6d. + **Fixed** (commit `cf46e596`). +2. **`infra/mssql/setup.sql` never executes.** It is mounted into + `/docker-entrypoint-initdb.d/`, a convention the official `mcr.microsoft.com/mssql/server` + image does not implement. After `reseed.sh` drops the volume (`docker compose down -v`), + nothing recreates `ScadaBridgeConfig` or the `scadabridge_app` login, so `reseed.sh` hangs + forever on its "Waiting for setup.sql to create ScadaBridgeConfig" poll. Worked around by + applying the three init scripts by hand. **NOT yet fixed in the repo.** + +--- + +## Finding 1 (BLOCKER) + +### The Phase 1 consolidated LocalDb fails under sustained write load on the active node + +`site-localdb.db` throws `SQLite Error 10: 'disk I/O error'` on essentially every write once the +active node is under concurrent load. Both Phase 1 tables and the audit telemetry paths are +affected. + +Representative stacks (`docker logs scadabridge-site-a-b`): + +``` +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync() + SiteEventLogger.cs:line 221/236 + +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...) + OperationTrackingStore.cs:line 137 + at ...CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) line 148 +``` + +User-visible symptom: `[ERR] Failed to record event: script from ScriptActor:SoakCall` — **site +event logging is silently dropping events on the floor under load.** + +#### It follows the load, not the node, not the observer + +The failure was isolated by moving the load between nodes: + +| Node | Role | Under load | `disk I/O error` in 4 min | +|---|---|---|---| +| site-a-a | active | yes | 2 175 | +| site-a-a | standby (after restart) | no | **0** | +| site-a-b | standby | no | **0** | +| site-a-b | active (after failover) | yes | **4 391** | + +#### It is LocalDb-specific, not the filesystem + +The decisive control. Under identical load, on the same node, in the same bind-mounted +directory, counting error-stack frames over 3 minutes: + +| Store | Backing file | Errors | +|---|---|---| +| `OperationTrackingStore` | `site-localdb.db` (LocalDb) | 13 044 | +| `SiteAuditTelemetryActor` | `site-localdb.db` (LocalDb) | 4 350 | +| `SiteEventLogger` | `site-localdb.db` (LocalDb) | 900 | +| `CachedCallTelemetryForwarder` | `site-localdb.db` (LocalDb) | 162 | +| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** | +| `SiteStorageService` | `scadabridge.db` (legacy) | **0** | + +Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed +database fails. + +#### Ruling out the observer + +Onset (04:56:37) was one second after a host-side `sqlite3` sample (04:56:36), which made +observer-induced `-shm` corruption the leading hypothesis. It is **excluded**: after node-a was +restarted (fresh open, recovered shm) and the load failed over to node-b, node-b began erroring +at a *higher* rate while no host process touched its files, and node-a — whose files *had* been +sampled — went to zero once it stopped carrying load. The variable that tracks the errors is +**load**, not host access. + +#### Secondary defect, same area + +``` +[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext, +this is most likely due to use of async operations from within this actor. +Cause: System.NotSupportedException +``` + +`SiteAuditTelemetryActor` is closing over `ActorContext` across an `await`. Likely a +contributing cause rather than a separate issue — it is in the same write path — but it is a +real bug on its own terms. + +#### Why this blocks Phase 2 + +Phase 2 registers **eight further tables** into this database, including `native_alarm_state` +(the highest-volume table in either DB) and `sf_messages`. It also **deletes** the bespoke +mechanisms (`SiteReplicationActor`, `ReplicationService`) that currently carry that data +independently of LocalDb. Cutting over onto a store that cannot absorb the *current* write +load — and doing so in the same commit that removes the fallback — would convert a logging +defect into site-wide config and buffer loss. + +This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3. + +--- + +## Finding 2 — D6 resolved: no stop condition, but `MaxBatchSize` must be lowered + +The plan asserts `deployed_configurations.config_json` is "documented to exceed 128 KB per row." +That misreads the source. `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md` +says the *escaped Akka envelope* exceeded the 128 KB frame, and that the default serializer +double-escapes the payload, so **"the raw flattened JSON only needs to be ~60-70 KB to blow the +128 KB frame."** + +So the largest known real production `config_json` is on the order of **60–70 KB**. + +Measured on the rig (4 deployed configs): **max 715 B, avg 714 B** — trivially small, as the plan +predicted, hence the production figure above is the one to size against. (A direct measurement +against wonder was attempted; `wonder-app-vd03.zmr.zimmer.com` resolves over the VPN but the +servecli SSH service on :2222 refuses connections, so the box was unreachable.) + +**Verdict: no stop condition.** Nothing approaches the 4 MB single-row ceiling. + +**But the batching risk is real.** Batching is row-count-only (`MaxBatchSize` default **500**), +and neither side configures gRPC message limits, so the 4 MB default receive cap applies: + +``` +500 rows × 70 KB ≈ 35 MB ≫ 4 MB → poison batch, stream wedged +``` + +Recommended for Task 19: + +``` +LocalDb:Replication:MaxBatchSize = 16 +``` + +`16 × 128 KB = 2 MB` — 2× headroom on row size over the known worst case, and 2× headroom +against the 4 MB cap. + +--- + +## Finding 3 — D4 corrected: alarm write rate is bounded, not unbounded + +The plan calls `native_alarm_state` "unbounded by design." It is not. +`NativeAlarmActor.MarkDirtyUpsert` (`NativeAlarmActor.cs:473-502`) coalesces into a dictionary +**keyed by `SourceReference`** and flushes on a timer (`_persistFlushInterval`, default +**100 ms**). One flush writes at most one row per distinct source reference, regardless of how +many transitions occurred in that window. + +``` +worst-case rows/sec = distinct_source_refs × 10 +``` + +An alarm storm on N sources costs N rows per 100 ms flush, not N × transition-rate. This makes +the table analytically sizeable without a generator — which matters, because this rig cannot +produce alarm load at all (see §1). + +**Not empirically validated.** `native_alarm_state` held 0 rows for the entire run. + +--- + +## Finding 4 — `sf_messages` has a hard structural ceiling of 50 rows/sec + +Confirmed by code rather than measurement, which is stronger here. The retry sweep takes at most +`SweepBatchLimit` messages every `RetryTimerInterval`, and each failed attempt is one `UPDATE` +incrementing `retry_count`: + +``` +SweepBatchLimit (500) ÷ RetryTimerInterval (10 s) = 50 row-writes/sec, hard ceiling +``` + +This confirms the plan's "~50 row-writes/sec worst case" — and it is a ceiling, not an estimate. +`DefaultMaxRetries = 50` at `DefaultRetryInterval = 30 s` bounds each message to 50 updates over +25 minutes. + +Observed during the run: ~2.9 attempts/sec, far below the ceiling. Row counts could not be +sampled reliably (see §1) and the run was cut short by Finding 1. + +--- + +## Finding 5 — exceeding the oplog caps is a graceful degradation, not a failure + +The plan's Task 1 step 7 treats "the shared oplog cannot absorb this write profile" as a hard +stop requiring the keyed-instances escape hatch. It is much weaker than that. + +`OplogStore.EnforceCapsAsync` (`OplogStore.cs:109-138`) prunes to the ceiling and sets +`needs_snapshot`; `MaintenanceBackgroundService.cs:57` logs *"Oplog backlog/age caps exceeded: +pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync."* +`SyncSession.ComputeSnapshotRequiredAsync` (`:413-415`) then forces a snapshot resync. + +So overrunning the caps costs a **full snapshot resync**, not data loss and not a wedged stream. +The caps therefore express *"how long may a peer be absent before it needs a full resync"*, and +should be sized to the longest tolerable peer outage rather than treated as a correctness +boundary. In healthy two-node operation the oplog drains continuously — `localdb_oplog_depth` +read **0** throughout. + +### Provisional cap recommendation (Task 19) + +Sized for a ~3-hour peer outage at a conservative 100 rows/sec aggregate, pending re-measurement +after Finding 1 is fixed: + +``` +LocalDb:Replication:MaxOplogRows = 1000000 # default; ≈2.8 h at 100 rows/s +LocalDb:Replication:MaxOplogAge = 3.00:00:00 +LocalDb:Replication:MaxBatchSize = 16 # Finding 2 — this one is NOT optional +``` + +Only `MaxBatchSize` is firmly evidence-backed. The other two rest on an assumed aggregate write +rate that this run could not measure. + +--- + +## 2. What still owes measurement + +Carry into the re-run once Finding 1 is fixed: + +1. Sustained `sf_messages` rows/sec under a saturating generator (needs many more instances or a + shorter interval to approach the 50/s ceiling). +2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an + A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does* + answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions + and a seeded `TemplateNativeAlarmSource`. (The test's own doc comment claiming the simulator + "does not reliably expose A&C" is stale.) +3. A production-representative `config_json` from wonder, to replace the ~60–70 KB inference. +4. The empirical oplog drain-under-churn check — Task 20 evidence item 10. + +## 3. Rig state left behind + +- Fully reseeded (central config volume dropped and replayed; site SQLite state wiped by + `reseed.sh` stage 2). +- `ExternalSystemDefinitions` id 1 is **repointed to `http://127.0.0.1:9`** — restore to + `http://scadabridge-restapi:5200` before using the rig for anything else. +- Template `SoakGenerator` (2021) and instances `soakgen-1..4` (ids 5–8) remain deployed on + site-a and are **still generating load**. Undeploy or delete them before the Task 20 live gate. +- `LdapGroupMappings` corrected in the live DB to the canonical role names.