fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect. Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL databases. POSIX advisory locks do not propagate across the virtiofs boundary, so the host reader believes it is the only connection, checkpoints on close and resets the WAL to 0 bytes under the container. The container's still-mapped WAL index then references frames that no longer exist and every subsequent statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently until the process reopens the database. Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and produced the first error one second later) and in a minimal python:3.12-alpine repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened node sustained the full soak load 10+ minutes with zero errors. The original brief's isolation was confounded - both nodes had been poisoned by the same sampling pass, and a poisoned standby looks healthy only because it issues almost no statements. Corrections annotated in place. Hardening shipped alongside: - SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the LocalDb-adjacent catch sites (the missing extended code is what made the original diagnosis so slow). - SiteAuditTelemetryActor: stop touching ActorContext across an await (NotSupportedException), with regression coverage. - infra/reseed.sh: apply the MSSQL init scripts explicitly. The official mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh volume hung the reseed forever; compose mounts annotated as informational. Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends only against external interference, which is now prevented at the source, and same-kernel production readers see the locks correctly. Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
@@ -203,8 +203,23 @@ This is the gate item from `2026-07-19-localdb-phase2-gate.md` §5. It is measur
|
||||
> the oplog sizing arithmetically, and (b) that the Phase 1 oplog stays healthy alongside. The
|
||||
> *empirical* post-cutover drain check lives in Task 20 (evidence item 10).
|
||||
>
|
||||
> Also: the containers have **no `sqlite3` binary** (bare `aspnet:10.0` image), but the data dirs
|
||||
> are **host bind mounts** — run `sqlite3` on the host against `docker/site-a-node-*/data/`.
|
||||
> Also: the containers have **no `sqlite3` binary** (bare `aspnet:10.0` image), and the data dirs
|
||||
> are host bind mounts — but **NEVER run `sqlite3` on the macOS host against the live files**:
|
||||
> host↔container POSIX locks do not propagate across the virtiofs boundary, and a single host-side
|
||||
> read checkpoints + resets the WAL out from under the container, permanently poisoning its
|
||||
> connections (`SQLITE_IOERR_SHORT_READ` → `disk I/O error` storm until restart). This is the
|
||||
> **root cause of the 2026-07-20 "disk I/O error under load" incident** — see
|
||||
> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
|
||||
> Safe sampling: copy the file triplet first (`cp x.db x.db-wal x.db-shm /tmp/…` — plain `cp`
|
||||
> takes no SQLite locks and cannot harm the live DB), then query the copy. The helper used below:
|
||||
>
|
||||
> ```bash
|
||||
> snap() { # snap <node-dir> <db> "<sql>" — query a throwaway copy, never the live file
|
||||
> local d; d=$(mktemp -d)
|
||||
> cp "$1/$2" "$d/" && cp "$1/$2-wal" "$1/$2-shm" "$d/" 2>/dev/null
|
||||
> sqlite3 "$d/$2" "$3"; rm -rf "$d"
|
||||
> }
|
||||
> ```
|
||||
> Metrics are on port **8084** inside the container (not published to the host).
|
||||
|
||||
**Step 1: Bring up the rig with Phase 1 replication enabled**
|
||||
@@ -224,14 +239,14 @@ Expected: `localdb_*` series present (`localdb_oplog_depth`, `localdb_sync_*`
|
||||
`ZB.MOM.WW.LocalDb.Replication`). If absent, the meter allowlist regressed — see
|
||||
`SiteServiceRegistration.ObservedMeters`.
|
||||
|
||||
**Step 2: Capture baselines (host-side, against the bind mounts)**
|
||||
**Step 2: Capture baselines (from throwaway copies — never the live files, see the `snap` helper above)**
|
||||
|
||||
```bash
|
||||
cd ~/Desktop/ScadaBridge/docker
|
||||
sqlite3 site-a-node-a/data/site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;"
|
||||
sqlite3 site-a-node-a/data/store-and-forward.db \
|
||||
snap site-a-node-a/data site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;"
|
||||
snap site-a-node-a/data store-and-forward.db \
|
||||
"SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;"
|
||||
sqlite3 site-a-node-a/data/scadabridge.db "SELECT COUNT(*) FROM native_alarm_state;"
|
||||
snap site-a-node-a/data scadabridge.db "SELECT COUNT(*) FROM native_alarm_state;"
|
||||
```
|
||||
|
||||
Record all values and the wall-clock time. (`COUNT + SUM(retry_count)` is the S&F row-write
|
||||
@@ -253,11 +268,11 @@ Two generators, run concurrently:
|
||||
cd ~/Desktop/ScadaBridge/docker
|
||||
for i in $(seq 1 6); do
|
||||
echo "=== t+$((i*5))m ==="
|
||||
sqlite3 site-a-node-a/data/store-and-forward.db \
|
||||
snap site-a-node-a/data store-and-forward.db \
|
||||
"SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;"
|
||||
sqlite3 site-a-node-a/data/scadabridge.db \
|
||||
snap site-a-node-a/data scadabridge.db \
|
||||
"SELECT COUNT(*), COUNT(CASE WHEN last_transition_at > datetime('now','-5 minutes') THEN 1 END) FROM native_alarm_state;"
|
||||
sqlite3 site-a-node-a/data/site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;"
|
||||
snap site-a-node-a/data site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;"
|
||||
docker exec scadabridge-site-a-a curl -s localhost:8084/metrics \
|
||||
| grep -E '^localdb_(oplog_depth|sync)'
|
||||
sleep 300
|
||||
@@ -271,7 +286,7 @@ upserts — say so in the findings if alarm volume is near a threshold).
|
||||
**Step 5: Measure `config_json` sizes (D6)**
|
||||
|
||||
```bash
|
||||
sqlite3 site-a-node-a/data/scadabridge.db \
|
||||
snap site-a-node-a/data scadabridge.db \
|
||||
"SELECT COUNT(*), MAX(LENGTH(config_json)), AVG(LENGTH(config_json)) FROM deployed_configurations;"
|
||||
```
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
},
|
||||
"reviewPass": {
|
||||
"date": "2026-07-19",
|
||||
"note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - use host-side sqlite3 on the bind mounts; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)."
|
||||
"note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - sample via throwaway copies of the DB triplet, NEVER host-side sqlite3 against the live files [2026-07-20: that poisons the container's WAL state - see docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md]; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)."
|
||||
},
|
||||
"decisions": [
|
||||
{
|
||||
@@ -64,7 +64,7 @@
|
||||
"StoreAndForwardService: :39 is the _replication FIELD, :243 the ctor param; exactly 6 emission sites (:654,805,836,872,1120,1146). ServiceCollectionExtensions.cs:32 also resolves ReplicationService inside the StoreAndForwardService factory - must go in Task 14.",
|
||||
"ReplicationMessages.cs holds ONLY the 10 Replicate*/Apply* records; the four SfBuffer resync records live at the bottom of SiteReplicationActor.cs (:678-707) and die with the actor file.",
|
||||
"10 (not 9) appsettings.Site.json files set the legacy paths - deploy/wonder-app-vd03/appsettings.Site.json was missed (also sets ReplicationEnabled:false).",
|
||||
"Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts, run sqlite3 host-side; ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).",
|
||||
"Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts but NEVER run sqlite3 host-side against the live files (poisons the container's WAL state, root cause of the 2026-07-20 disk-I/O-error incident): cp the .db/-wal/-shm triplet and query the copy (see the snap() helper in the plan); ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).",
|
||||
"Phase 1's LocalDbSitePairConvergenceTests uses ONE shared API key (:47) - it never did a mismatched-key non-vacuity run; wrong-key denial is unit-covered in Host.Tests/LocalDbSyncAuthInterceptorTests.cs. Task 18's non-vacuity check must be done directly for the new scenarios.",
|
||||
"MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14."
|
||||
],
|
||||
@@ -88,7 +88,7 @@
|
||||
{"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "pending", "classification": "standard", "blockedBy": [16]},
|
||||
{"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17], "note": "Non-vacuity must be proven directly (Phase 1 never did the mismatched-key run the original plan claimed)."},
|
||||
{"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "pending", "classification": "small", "blockedBy": [17]},
|
||||
{"id": 20, "subject": "Task 20: Live gate on the docker rig (10 evidence items)", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19], "note": "Includes the post-cutover oplog drain-under-churn check (evidence 10) that Task 1 cannot measure pre-cutover. Host-side sqlite3 on bind mounts; metrics on :8084 in-container."},
|
||||
{"id": 20, "subject": "Task 20: Live gate on the docker rig (10 evidence items)", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19], "note": "Includes the post-cutover oplog drain-under-churn check (evidence 10) that Task 1 cannot measure pre-cutover. Sample DBs via throwaway copies (snap() helper), never host-side sqlite3 on live files; metrics on :8084 in-container."},
|
||||
{"id": 21, "subject": "Task 21: Documentation truth pass", "status": "pending", "classification": "standard", "blockedBy": [20]}
|
||||
],
|
||||
"knownFlakes": [
|
||||
|
||||
@@ -3,14 +3,22 @@
|
||||
**Run date:** 2026-07-19 / 2026-07-20 (UTC) · **Rig:** local 8-node docker cluster, site-a pair
|
||||
**Task:** Task 1 of [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md)
|
||||
|
||||
> ## GATE VERDICT: **STOP — do not proceed to Task 3.**
|
||||
> ## GATE VERDICT: **STOP — do not proceed to Task 3.** *(superseded — see update)*
|
||||
>
|
||||
> Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak
|
||||
> instead surfaced a **pre-existing Phase 1 defect**: the consolidated LocalDb database
|
||||
> (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every write on
|
||||
> the **active** node under sustained concurrent load. Phase 2 moves eight more tables —
|
||||
> including the two highest-volume ones — into exactly that database. See
|
||||
> instead surfaced what looked like a **pre-existing Phase 1 defect**: the consolidated LocalDb
|
||||
> database (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every
|
||||
> write on the **active** node under sustained concurrent load. See
|
||||
> [Finding 1](#finding-1-blocker).
|
||||
>
|
||||
> **Update 2026-07-20: Finding 1 is root-caused and is NOT a product defect.** The soak's own
|
||||
> host-side `sqlite3` sampling poisoned both nodes (WAL reset across the virtiofs boundary);
|
||||
> the unmodified code sustains the full soak load indefinitely with zero errors —
|
||||
> reproduced/refuted on demand, see
|
||||
> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
|
||||
> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the
|
||||
> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe
|
||||
> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read.
|
||||
|
||||
---
|
||||
|
||||
@@ -22,7 +30,7 @@ does not rediscover them.
|
||||
| Plan said | Reality |
|
||||
|---|---|
|
||||
| `docker exec … curl -s localhost:8084/metrics` | **No `curl` in the `aspnet:10.0` image.** Use a sidecar sharing the container netns: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` |
|
||||
| Sample with host-side `sqlite3` against the bind mounts | **Unsafe under live writes** — SQLite's `-shm` shared-memory index is not reliable across the Docker VM boundary. Ultimately *not* the cause of Finding 1 (proven below), but it should not be done against a database the container is actively writing. Copy the files first, or read counters from `/metrics`. |
|
||||
| Sample with host-side `sqlite3` against the bind mounts | **This IS the cause of Finding 1** (root-caused 2026-07-20 — the "ruled out" verdict below did not survive; see the known-issue doc §0). Host↔container POSIX locks don't propagate over virtiofs; a host-side read checkpoints + resets the WAL under the container and permanently poisons its connections. Copy the file triplet and query the copy, or read counters from `/metrics`. |
|
||||
| Drive alarm churn on a deployed instance | **Not possible on this rig.** `infra/mssql/seed-config.sql` seeds zero `TemplateNativeAlarmSources`, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. `native_alarm_state` stayed at 0 rows throughout. Bounded analytically instead — see [Finding 3](#finding-3). |
|
||||
| Drive S&F churn via template 4's `TestExternalSystem` script | Template 4 exists after a reseed but is **not deployable** (34 pre-deployment validation errors: 30 `ConnectionBinding` + 4 `ScriptCompilation`). A purpose-built `SoakGenerator` template was used instead — see below. |
|
||||
|
||||
@@ -60,7 +68,7 @@ Sustained rate observed: ~**2.9 HTTP attempts/sec** (688–864 connection-refuse
|
||||
|
||||
---
|
||||
|
||||
## Finding 1 (BLOCKER)
|
||||
## Finding 1 (BLOCKER) — *root-caused 2026-07-20: observer-induced, not a product defect; see the gate-verdict update and the known-issue doc §0*
|
||||
|
||||
### The Phase 1 consolidated LocalDb fails under sustained write load on the active node
|
||||
|
||||
@@ -95,7 +103,7 @@ The failure was isolated by moving the load between nodes:
|
||||
| site-a-b | standby | no | **0** |
|
||||
| site-a-b | active (after failover) | yes | **4 391** |
|
||||
|
||||
#### It is LocalDb-specific, not the filesystem
|
||||
#### It is LocalDb-specific, not the filesystem — *wrong: sampling-selection bias; only the LocalDb file was ever host-read*
|
||||
|
||||
The decisive control. Under identical load, on the same node, in the same bind-mounted
|
||||
directory, counting error-stack frames over 3 minutes:
|
||||
@@ -112,14 +120,20 @@ directory, counting error-stack frames over 3 minutes:
|
||||
Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed
|
||||
database fails.
|
||||
|
||||
#### Ruling out the observer
|
||||
#### Ruling out the observer — **RETRACTED 2026-07-20: the observer was the cause**
|
||||
|
||||
Onset (04:56:37) was one second after a host-side `sqlite3` sample (04:56:36), which made
|
||||
observer-induced `-shm` corruption the leading hypothesis. It is **excluded**: after node-a was
|
||||
restarted (fresh open, recovered shm) and the load failed over to node-b, node-b began erroring
|
||||
at a *higher* rate while no host process touched its files, and node-a — whose files *had* been
|
||||
sampled — went to zero once it stopped carrying load. The variable that tracks the errors is
|
||||
**load**, not host access.
|
||||
observer-induced `-shm` corruption the leading hypothesis. The original run "excluded" it:
|
||||
after node-a was restarted and the load failed over to node-b, node-b began erroring while no
|
||||
host process touched its files, and sampled node-a went to zero once idle.
|
||||
|
||||
**That exclusion was wrong.** The 04:56 sampling had poisoned *both* nodes' files (both carry
|
||||
the 04:56 main-DB mtime; node-b's WAL was left at 0 bytes) — node-b was silent only because a
|
||||
standby issues ~no LocalDb statements, and erupted on its first post-failover write. Verified
|
||||
2026-07-20: 10+ min of full soak load on a freshly-reopened node with zero errors, then a
|
||||
single host `sqlite3` read reset its 4.6 MiB WAL to 0 bytes and started the error storm one
|
||||
second later (`SQLITE_IOERR_SHORT_READ`, 522). Full mechanism + minimal repro:
|
||||
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
|
||||
|
||||
#### Secondary defect, same area
|
||||
|
||||
@@ -142,7 +156,9 @@ independently of LocalDb. Cutting over onto a store that cannot absorb the *curr
|
||||
load — and doing so in the same commit that removes the fallback — would convert a logging
|
||||
defect into site-wide config and buffer loss.
|
||||
|
||||
This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.
|
||||
~~This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.~~
|
||||
**Root-caused 2026-07-20: not a Phase 1 defect** — observer-induced WAL reset across the
|
||||
virtiofs bind-mount boundary; see the gate-verdict update at the top of this document.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user