ScadaBridge/docs/plans/2026-07-19-localdb-phase2-soak.md

LocalDb Phase 2 — rig soak findings

Run date: 2026-07-19 / 2026-07-20 (UTC) · Rig: local 8-node docker cluster, site-a pair Task: Task 1 of 2026-07-19-localdb-adoption-phase2.md

GATE VERDICT: STOP — do not proceed to Task 3.

Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak instead surfaced a pre-existing Phase 1 defect: the consolidated LocalDb database (site-localdb.db) throws SQLite Error 10: 'disk I/O error' on essentially every write on the active node under sustained concurrent load. Phase 2 moves eight more tables — including the two highest-volume ones — into exactly that database. See Finding 1.

---

1. Method as actually executed

The plan's method needed four corrections before it would run. Recorded here so the next run does not rediscover them.

Plan said	Reality
docker exec … curl -s localhost:8084/metrics	No curl in the aspnet:10.0 image. Use a sidecar sharing the container netns: docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics
Sample with host-side sqlite3 against the bind mounts	Unsafe under live writes — SQLite's -shm shared-memory index is not reliable across the Docker VM boundary. Ultimately not the cause of Finding 1 (proven below), but it should not be done against a database the container is actively writing. Copy the files first, or read counters from /metrics.
Drive alarm churn on a deployed instance	Not possible on this rig. infra/mssql/seed-config.sql seeds zero TemplateNativeAlarmSources, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. native_alarm_state stayed at 0 rows throughout. Bounded analytically instead — see Finding 3.
Drive S&F churn via template 4's TestExternalSystem script	Template 4 exists after a reseed but is not deployable (34 pre-deployment validation errors: 30 ConnectionBinding + 4 ScriptCompilation). A purpose-built SoakGenerator template was used instead — see below.

The generator that worked

ExternalSystem.Call does not buffer to store-and-forward in practice; ExternalSystem.CachedCall is the buffering surface. This is the single most important operational detail for reproducing S&F load.

• Template SoakGenerator (id 2021), one Interval script at {"intervalMs":5000}:

  var parms = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 };
  await ExternalSystem.CachedCall("Test REST API", "Add", parms);
  

• No attributes, no compositions, no connection bindings — deliberately, so it deploys cleanly.
• ExternalSystemDefinitions id 1 repointed to http://127.0.0.1:9 (discard port → connection refused → classified transient → buffered).
• 4 instances (soakgen-1..4) deployed to site-a.

Sustained rate observed: ~2.9 HTTP attempts/sec (688–864 connection-refused per 4 min).

Two rig-tooling bugs found and fixed en route

1. docker/seed-sites.sh seeded stale role names — Design/Deployment instead of the canonical Designer/Deployer (src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs:46-47). Every Designer/Deployer-gated management command failed UNAUTHORIZED on a freshly reseeded rig, including seed-sites.sh's own trailing deploy artifacts and reseed.sh stage 6d. Fixed (commit cf46e596).
2. infra/mssql/setup.sql never executes. It is mounted into /docker-entrypoint-initdb.d/, a convention the official mcr.microsoft.com/mssql/server image does not implement. After reseed.sh drops the volume (docker compose down -v), nothing recreates ScadaBridgeConfig or the scadabridge_app login, so reseed.sh hangs forever on its "Waiting for setup.sql to create ScadaBridgeConfig" poll. Worked around by applying the three init scripts by hand. NOT yet fixed in the repo.

---

Finding 1 (BLOCKER)

The Phase 1 consolidated LocalDb fails under sustained write load on the active node

site-localdb.db throws SQLite Error 10: 'disk I/O error' on essentially every write once the active node is under concurrent load. Both Phase 1 tables and the audit telemetry paths are affected.

Representative stacks (docker logs scadabridge-site-a-b):

Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
   at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync()
      SiteEventLogger.cs:line 221/236

Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
   at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...)
      OperationTrackingStore.cs:line 137
   at ...CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) line 148

User-visible symptom: [ERR] Failed to record event: script from ScriptActor:SoakCall — site event logging is silently dropping events on the floor under load.

It follows the load, not the node, not the observer

The failure was isolated by moving the load between nodes:

Node	Role	Under load	disk I/O error in 4 min
site-a-a	active	yes	2 175
site-a-a	standby (after restart)	no	0
site-a-b	standby	no	0
site-a-b	active (after failover)	yes	4 391

It is LocalDb-specific, not the filesystem

The decisive control. Under identical load, on the same node, in the same bind-mounted directory, counting error-stack frames over 3 minutes:

Store	Backing file	Errors
OperationTrackingStore	site-localdb.db (LocalDb)	13 044
SiteAuditTelemetryActor	site-localdb.db (LocalDb)	4 350
SiteEventLogger	site-localdb.db (LocalDb)	900
CachedCallTelemetryForwarder	site-localdb.db (LocalDb)	162
StoreAndForwardStorage	store-and-forward.db (legacy)	0
SiteStorageService	scadabridge.db (legacy)	0

Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed database fails.

Ruling out the observer

Onset (04:56:37) was one second after a host-side sqlite3 sample (04:56:36), which made observer-induced -shm corruption the leading hypothesis. It is excluded: after node-a was restarted (fresh open, recovered shm) and the load failed over to node-b, node-b began erroring at a higher rate while no host process touched its files, and node-a — whose files had been sampled — went to zero once it stopped carrying load. The variable that tracks the errors is load, not host access.

Secondary defect, same area

[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
this is most likely due to use of async operations from within this actor.
Cause: System.NotSupportedException

SiteAuditTelemetryActor is closing over ActorContext across an await. Likely a contributing cause rather than a separate issue — it is in the same write path — but it is a real bug on its own terms.

Why this blocks Phase 2

Phase 2 registers eight further tables into this database, including native_alarm_state (the highest-volume table in either DB) and sf_messages. It also deletes the bespoke mechanisms (SiteReplicationActor, ReplicationService) that currently carry that data independently of LocalDb. Cutting over onto a store that cannot absorb the current write load — and doing so in the same commit that removes the fallback — would convert a logging defect into site-wide config and buffer loss.

This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.

---

Finding 2 — D6 resolved: no stop condition, but MaxBatchSize must be lowered

The plan asserts deployed_configurations.config_json is "documented to exceed 128 KB per row." That misreads the source. docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md says the escaped Akka envelope exceeded the 128 KB frame, and that the default serializer double-escapes the payload, so "the raw flattened JSON only needs to be ~60-70 KB to blow the 128 KB frame."

So the largest known real production config_json is on the order of 60–70 KB.

Measured on the rig (4 deployed configs): max 715 B, avg 714 B — trivially small, as the plan predicted, hence the production figure above is the one to size against. (A direct measurement against wonder was attempted; wonder-app-vd03.zmr.zimmer.com resolves over the VPN but the servecli SSH service on :2222 refuses connections, so the box was unreachable.)

Verdict: no stop condition. Nothing approaches the 4 MB single-row ceiling.

But the batching risk is real. Batching is row-count-only (MaxBatchSize default 500), and neither side configures gRPC message limits, so the 4 MB default receive cap applies:

500 rows × 70 KB ≈ 35 MB  ≫  4 MB   →  poison batch, stream wedged

Recommended for Task 19:

LocalDb:Replication:MaxBatchSize = 16

16 × 128 KB = 2 MB — 2× headroom on row size over the known worst case, and 2× headroom against the 4 MB cap.

---

Finding 3 — D4 corrected: alarm write rate is bounded, not unbounded

The plan calls native_alarm_state "unbounded by design." It is not. NativeAlarmActor.MarkDirtyUpsert (NativeAlarmActor.cs:473-502) coalesces into a dictionary keyed by SourceReference and flushes on a timer (_persistFlushInterval, default 100 ms). One flush writes at most one row per distinct source reference, regardless of how many transitions occurred in that window.

worst-case rows/sec  =  distinct_source_refs × 10

An alarm storm on N sources costs N rows per 100 ms flush, not N × transition-rate. This makes the table analytically sizeable without a generator — which matters, because this rig cannot produce alarm load at all (see §1).

Not empirically validated. native_alarm_state held 0 rows for the entire run.

---

Finding 4 — sf_messages has a hard structural ceiling of 50 rows/sec

Confirmed by code rather than measurement, which is stronger here. The retry sweep takes at most SweepBatchLimit messages every RetryTimerInterval, and each failed attempt is one UPDATE incrementing retry_count:

SweepBatchLimit (500) ÷ RetryTimerInterval (10 s)  =  50 row-writes/sec, hard ceiling

This confirms the plan's "~50 row-writes/sec worst case" — and it is a ceiling, not an estimate. DefaultMaxRetries = 50 at DefaultRetryInterval = 30 s bounds each message to 50 updates over 25 minutes.

Observed during the run: ~2.9 attempts/sec, far below the ceiling. Row counts could not be sampled reliably (see §1) and the run was cut short by Finding 1.

---

Finding 5 — exceeding the oplog caps is a graceful degradation, not a failure

The plan's Task 1 step 7 treats "the shared oplog cannot absorb this write profile" as a hard stop requiring the keyed-instances escape hatch. It is much weaker than that.

OplogStore.EnforceCapsAsync (OplogStore.cs:109-138) prunes to the ceiling and sets needs_snapshot; MaintenanceBackgroundService.cs:57 logs "Oplog backlog/age caps exceeded: pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync." SyncSession.ComputeSnapshotRequiredAsync (:413-415) then forces a snapshot resync.

So overrunning the caps costs a full snapshot resync, not data loss and not a wedged stream. The caps therefore express "how long may a peer be absent before it needs a full resync", and should be sized to the longest tolerable peer outage rather than treated as a correctness boundary. In healthy two-node operation the oplog drains continuously — localdb_oplog_depth read 0 throughout.

Provisional cap recommendation (Task 19)

Sized for a ~3-hour peer outage at a conservative 100 rows/sec aggregate, pending re-measurement after Finding 1 is fixed:

LocalDb:Replication:MaxOplogRows = 1000000   # default; ≈2.8 h at 100 rows/s
LocalDb:Replication:MaxOplogAge  = 3.00:00:00
LocalDb:Replication:MaxBatchSize = 16        # Finding 2 — this one is NOT optional

Only MaxBatchSize is firmly evidence-backed. The other two rest on an assumed aggregate write rate that this run could not measure.

---

2. What still owes measurement

Carry into the re-run once Finding 1 is fixed:

1. Sustained sf_messages rows/sec under a saturating generator (needs many more instances or a shorter interval to approach the 50/s ceiling).
2. Any native_alarm_state measurement at all — requires building alarm-source seeding plus an A&C-capable server. The OpcUaAlarmLiveSmokeTests passed, so the rig's opc-plc does answer ConditionRefresh with a SnapshotComplete; the missing piece is ongoing transitions and a seeded TemplateNativeAlarmSource. (The test's own doc comment claiming the simulator "does not reliably expose A&C" is stale.)
3. A production-representative config_json from wonder, to replace the ~60–70 KB inference.
4. The empirical oplog drain-under-churn check — Task 20 evidence item 10.

3. Rig state left behind

• Fully reseeded (central config volume dropped and replayed; site SQLite state wiped by reseed.sh stage 2).
• ExternalSystemDefinitions id 1 is repointed to http://127.0.0.1:9 — restore to http://scadabridge-restapi:5200 before using the rig for anything else.
• Template SoakGenerator (2021) and instances soakgen-1..4 (ids 5–8) remain deployed on site-a and are still generating load. Undeploy or delete them before the Task 20 live gate.
• LdapGroupMappings corrected in the live DB to the canonical role names.