dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups

Browse Source 
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect.
Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL
databases. POSIX advisory locks do not propagate across the virtiofs boundary,
so the host reader believes it is the only connection, checkpoints on close and
resets the WAL to 0 bytes under the container. The container's still-mapped
WAL index then references frames that no longer exist and every subsequent
statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently
until the process reopens the database.

Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and
produced the first error one second later) and in a minimal python:3.12-alpine
repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened
node sustained the full soak load 10+ minutes with zero errors.

The original brief's isolation was confounded - both nodes had been poisoned by
the same sampling pass, and a poisoned standby looks healthy only because it
issues almost no statements. Corrections annotated in place.

Hardening shipped alongside:
- SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the
  LocalDb-adjacent catch sites (the missing extended code is what made the
  original diagnosis so slow).
- SiteAuditTelemetryActor: stop touching ActorContext across an await
  (NotSupportedException), with regression coverage.
- infra/reseed.sh: apply the MSSQL init scripts explicitly. The official
  mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh
  volume hung the reseed forever; compose mounts annotated as informational.

Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends
only against external interference, which is now prevented at the source, and
same-kernel production readers see the locks correctly.

Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 02:01:00 -04:00
parent e9e11d635e
commit 8652eab98e
11 changed files with 378 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-07-19-localdb-phase2-soak.md
+31 -15
View File
@@ -3,14 +3,22 @@
**Run date:** 2026-07-19 / 2026-07-20 (UTC) · **Rig:** local 8-node docker cluster, site-a pair
**Task:** Task 1 of [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md)
> ## GATE VERDICT: **STOP — do not proceed to Task 3.**
> ## GATE VERDICT: **STOP — do not proceed to Task 3.** *(superseded — see update)*
>
> Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak
> instead surfaced a **pre-existing Phase 1 defect**: the consolidated LocalDb database
> (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every write on
> the **active** node under sustained concurrent load. Phase 2 moves eight more tables —
> including the two highest-volume ones — into exactly that database. See
> instead surfaced what looked like a **pre-existing Phase 1 defect**: the consolidated LocalDb
> database (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every
> write on the **active** node under sustained concurrent load. See
> [Finding 1](#finding-1-blocker).
>
> **Update 2026-07-20: Finding 1 is root-caused and is NOT a product defect.** The soak's own
> host-side `sqlite3` sampling poisoned both nodes (WAL reset across the virtiofs boundary);
> the unmodified code sustains the full soak load indefinitely with zero errors —
> reproduced/refuted on demand, see
> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the
> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe
> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read.
---
@@ -22,7 +30,7 @@ does not rediscover them.
| Plan said | Reality |
|---|---|
| `docker exec … curl -s localhost:8084/metrics` | **No `curl` in the `aspnet:10.0` image.** Use a sidecar sharing the container netns: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` |
| Sample with host-side `sqlite3` against the bind mounts | **Unsafe under live writes** — SQLite's `-shm` shared-memory index is not reliable across the Docker VM boundary. Ultimately *not* the cause of Finding 1 (proven below), but it should not be done against a database the container is actively writing. Copy the files first, or read counters from `/metrics`. |
| Sample with host-side `sqlite3` against the bind mounts | **This IS the cause of Finding 1** (root-caused 2026-07-20 — the "ruled out" verdict below did not survive; see the known-issue doc §0). Host↔container POSIX locks don't propagate over virtiofs; a host-side read checkpoints + resets the WAL under the container and permanently poisons its connections. Copy the file triplet and query the copy, or read counters from `/metrics`. |
| Drive alarm churn on a deployed instance | **Not possible on this rig.** `infra/mssql/seed-config.sql` seeds zero `TemplateNativeAlarmSources`, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. `native_alarm_state` stayed at 0 rows throughout. Bounded analytically instead — see [Finding 3](#finding-3). |
| Drive S&F churn via template 4's `TestExternalSystem` script | Template 4 exists after a reseed but is **not deployable** (34 pre-deployment validation errors: 30 `ConnectionBinding` + 4 `ScriptCompilation`). A purpose-built `SoakGenerator` template was used instead — see below. |
@@ -60,7 +68,7 @@ Sustained rate observed: ~**2.9 HTTP attempts/sec** (688–864 connection-refuse
---
## Finding 1 (BLOCKER)
## Finding 1 (BLOCKER) — *root-caused 2026-07-20: observer-induced, not a product defect; see the gate-verdict update and the known-issue doc §0*
### The Phase 1 consolidated LocalDb fails under sustained write load on the active node
@@ -95,7 +103,7 @@ The failure was isolated by moving the load between nodes:
| site-a-b | standby | no | **0** |
| site-a-b | active (after failover) | yes | **4 391** |
#### It is LocalDb-specific, not the filesystem
#### It is LocalDb-specific, not the filesystem — *wrong: sampling-selection bias; only the LocalDb file was ever host-read*
The decisive control. Under identical load, on the same node, in the same bind-mounted
directory, counting error-stack frames over 3 minutes:
@@ -112,14 +120,20 @@ directory, counting error-stack frames over 3 minutes:
Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed
database fails.
#### Ruling out the observer
#### Ruling out the observer — **RETRACTED 2026-07-20: the observer was the cause**
Onset (04:56:37) was one second after a host-side `sqlite3` sample (04:56:36), which made
observer-induced `-shm` corruption the leading hypothesis. It is **excluded**: after node-a was
restarted (fresh open, recovered shm) and the load failed over to node-b, node-b began erroring
at a *higher* rate while no host process touched its files, and node-a — whose files *had* been
sampled — went to zero once it stopped carrying load. The variable that tracks the errors is
**load**, not host access.
observer-induced `-shm` corruption the leading hypothesis. The original run "excluded" it:
after node-a was restarted and the load failed over to node-b, node-b began erroring while no
host process touched its files, and sampled node-a went to zero once idle.
**That exclusion was wrong.** The 04:56 sampling had poisoned *both* nodes' files (both carry
the 04:56 main-DB mtime; node-b's WAL was left at 0 bytes) — node-b was silent only because a
standby issues ~no LocalDb statements, and erupted on its first post-failover write. Verified
2026-07-20: 10+ min of full soak load on a freshly-reopened node with zero errors, then a
single host `sqlite3` read reset its 4.6 MiB WAL to 0 bytes and started the error storm one
second later (`SQLITE_IOERR_SHORT_READ`, 522). Full mechanism + minimal repro:
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
#### Secondary defect, same area
@@ -142,7 +156,9 @@ independently of LocalDb. Cutting over onto a store that cannot absorb the *curr
load — and doing so in the same commit that removes the fallback — would convert a logging
defect into site-wide config and buffer loss.
This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.
~~This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.~~
**Root-caused 2026-07-20: not a Phase 1 defect** — observer-induced WAL reset across the
virtiofs bind-mount boundary; see the gate-verdict update at the top of this document.
---