fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect. Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL databases. POSIX advisory locks do not propagate across the virtiofs boundary, so the host reader believes it is the only connection, checkpoints on close and resets the WAL to 0 bytes under the container. The container's still-mapped WAL index then references frames that no longer exist and every subsequent statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently until the process reopens the database. Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and produced the first error one second later) and in a minimal python:3.12-alpine repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened node sustained the full soak load 10+ minutes with zero errors. The original brief's isolation was confounded - both nodes had been poisoned by the same sampling pass, and a poisoned standby looks healthy only because it issues almost no statements. Corrections annotated in place. Hardening shipped alongside: - SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the LocalDb-adjacent catch sites (the missing extended code is what made the original diagnosis so slow). - SiteAuditTelemetryActor: stop touching ActorContext across an await (NotSupportedException), with regression coverage. - infra/reseed.sh: apply the MSSQL init scripts explicitly. The official mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh volume hung the reseed forever; compose mounts annotated as informational. Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends only against external interference, which is now prevented at the source, and same-kernel production readers see the locks correctly. Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
@@ -1,12 +1,106 @@
|
||||
# LocalDb throws `SQLite Error 10: 'disk I/O error'` on the active site node under sustained write load
|
||||
|
||||
**Date:** 2026-07-20 · **Status:** OPEN — not root-caused · **Severity:** High (silent event loss; blocks LocalDb Phase 2)
|
||||
**Date:** 2026-07-20 · **Status:** ROOT-CAUSED + FIX PASS COMPLETE 2026-07-20 (same day) — observer-induced, **not a LocalDb defect**; see §0 (cause) and §11a (fixes) · **Severity:** was High; resolved to an operational rule (now enforced in the tooling docs) + shipped hardening
|
||||
**Area:** `ZB.MOM.WW.LocalDb` (library, `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`) as consumed by ScadaBridge site nodes
|
||||
**Found by:** the Phase 2 rig soak — [`docs/plans/2026-07-19-localdb-phase2-soak.md`](../plans/2026-07-19-localdb-phase2-soak.md)
|
||||
**Branch:** `feat/localdb-phase2` (defect is in Phase 1 code, present on `feat/localdb-phase1`)
|
||||
**Branch:** `feat/localdb-phase2` (symptom observed on Phase 1 code)
|
||||
|
||||
> **This document is a root-cause brief.** The investigation below establishes *what* fails and
|
||||
> rules out several explanations. It does **not** identify the mechanism. Nothing has been fixed.
|
||||
> **Update 2026-07-20:** the mechanism has been identified and reproduced on demand, both in a
|
||||
> minimal SQLite-only repro and on the live rig, and the follow-up fixes have shipped.
|
||||
> Sections §0, §11a and §11 below are authoritative; the original brief (§1–§10) is preserved
|
||||
> as written, with corrections annotated where its conclusions did not survive
|
||||
> (§4.2, §4.3, §7, §9) and fix notes where they did (§5.1, §8).
|
||||
|
||||
---
|
||||
|
||||
## 0. ROOT CAUSE (verified)
|
||||
|
||||
**A host-side (macOS) `sqlite3` read of a live, bind-mounted WAL database checkpoints and
|
||||
resets the WAL out from under the container process, permanently poisoning that process's
|
||||
connections.** LocalDb, its connection handling, its UDF, and its triggers are not involved —
|
||||
the mechanism reproduces with plain Python `sqlite3` and no LocalDb code at all.
|
||||
|
||||
Mechanism, step by step:
|
||||
|
||||
1. POSIX advisory locks do **not** propagate across the Docker Desktop virtiofs bind-mount
|
||||
boundary. A host `sqlite3` opening the database cannot see the container's locks (and vice
|
||||
versa), so it believes it is the **only** connection.
|
||||
2. On close, the "last" connection in WAL mode runs a full checkpoint and **resets the WAL to
|
||||
0 bytes**. Because the read happened while the container was idle (a standby node, or a gap
|
||||
between write bursts), nothing blocks the checkpoint. This is exactly the file signature
|
||||
found on both rig nodes: main DB mtime + 0-byte `-wal` stamped at the sampling minute.
|
||||
3. The container process still holds the old WAL-index state (its per-inode `-shm` mapping,
|
||||
kept alive forever by the held-open `_master` connection and the Microsoft.Data.Sqlite
|
||||
connection pool). That index says the WAL contains N frames; the file now has none. Every
|
||||
subsequent statement — reads and writes both consult the WAL index — fails with
|
||||
**`SQLITE_IOERR_SHORT_READ` (extended code 522)**, surfaced as primary code 10
|
||||
`'disk I/O error'` (some paths surface `SQLITE_NOTADB` (26) instead). The poisoning is
|
||||
**permanent until the process reopens the database** (restart).
|
||||
|
||||
### Why the original brief's conclusions were wrong
|
||||
|
||||
- **"It tracks the load, not the node" (§4.1)** — confounded. *Both* nodes were poisoned by
|
||||
the 04:56 UTC host-side sampling (both nodes' `site-localdb.db` main files carry the 04:56
|
||||
mtime; node-b's WAL was left at 0 bytes). A poisoned **standby** shows zero errors only
|
||||
because a standby issues ~zero LocalDb statements; the errors "followed the load" because
|
||||
the load is what generates statements against an already-poisoned handle. Node-b's very
|
||||
first write attempt after failover (04:59:37, `OperationTrackingStore.RecordAttemptAsync`)
|
||||
failed — it had been poisoned for 3 minutes with nothing to say about it.
|
||||
- **"It is LocalDb-specific" (§4.2)** — sampling-selection bias. The legacy WAL databases in
|
||||
the same directory were healthy only because no host process ever read *them*. The minimal
|
||||
repro poisons an arbitrary WAL database the same way.
|
||||
- **"The observer has been ruled out" (§4.3)** — the exclusion assumed an error-free standby
|
||||
was an unpoisoned standby. It wasn't; it was a poisoned node with no traffic.
|
||||
|
||||
### Verification (2026-07-20, all on the live rig + minimal repro)
|
||||
|
||||
1. **Load alone is harmless:** restarted the poisoned active node (site-a-b); freshly-reopened
|
||||
site-a-a took the full soak load for **10+ minutes with zero errors** (the original model
|
||||
predicted onset within ~2 min), WAL growing/checkpointing normally, DB 188 KiB → 476 KiB.
|
||||
2. **One host read is sufficient and immediate:** a single
|
||||
`sqlite3 docker/site-a-node-a/data/site-localdb.db "SELECT count(*) FROM site_events;"`
|
||||
against the healthy loaded node reset its 4.6 MiB WAL to 0 bytes in place and produced the
|
||||
first `disk I/O error` **one second later** (05:32:48 → 05:32:49), 203 errors in the next
|
||||
40 s — the same one-second onset correlation as the original 04:56:36 → 04:56:37 incident.
|
||||
3. **Minimal repro (no LocalDb, no .NET):** a `python:3.12-alpine` container writing a
|
||||
WAL-mode SQLite DB on a bind mount (held master connection + fresh connection per op,
|
||||
`synchronous=NORMAL`, `busy_timeout=5000`). A host `sqlite3 "SELECT count(*)"`:
|
||||
- against the **actively-writing** DB → immediate `SQLITE_IOERR_SHORT_READ` (522) +
|
||||
`SQLITE_NOTADB` burst, then recovery (checkpoint could not fully reset a hot WAL);
|
||||
- during an **idle window** (connections held open, WAL populated) → WAL reset
|
||||
1.2 MiB → 0 bytes, then **every fresh-connection write failed for the rest of the run
|
||||
(200/200)** — the persistent variant, matching the rig.
|
||||
|
||||
### Consequences
|
||||
|
||||
1. **The operational rule in §5.3 ("do not query the databases with host-side `sqlite3`") is
|
||||
the root cause, not a hygiene note.** One violation silently destroys the node's local
|
||||
persistence until restart. This applies to *every* WAL SQLite file on the bind mount
|
||||
(`scadabridge.db`, `store-and-forward.db` included), not just LocalDb.
|
||||
2. **Safe inspection recipes:** copy the file triplet (`.db`, `-wal`, `-shm`) and open the
|
||||
copy; or read from inside the container boundary (same kernel ⇒ locks visible), e.g.
|
||||
`docker run --rm -v <dir>:/d alpine/sqlite3 sqlite3 /d/site-localdb.db "..."` — never the
|
||||
macOS host against live files.
|
||||
3. **LocalDb Phase 2 is unblocked** on this issue: the library sustained the full soak write
|
||||
load indefinitely once nothing external touched its file.
|
||||
4. Hardening follow-ups — **status as of the 2026-07-20 fix pass (see §11a):**
|
||||
- **DONE — extended-code logging:** the LocalDb-adjacent catch sites (`SiteAuditTelemetryActor`,
|
||||
`CachedCallTelemetryForwarder`, `SiteEventLogger`) now log
|
||||
`SqliteException` primary/extended codes (`sqlite 10/522`-style) via
|
||||
`SqliteErrorCodes.Describe` / `DescribeSqliteError`.
|
||||
- **DONE — §8 async-context bug** (see §8).
|
||||
- **DONE — §9.5 load regression test** (see §9).
|
||||
- **NOT DONE (deliberately):** a detect-and-reopen self-heal in `SqliteLocalDb` for
|
||||
persistent `SQLITE_IOERR`/`SQLITE_NOTADB`. This is a real library design change
|
||||
(pool clear + master reopen + in-flight coordination) protecting against *external
|
||||
interference only* — the trigger is operator/tooling action, now prevented at the
|
||||
source, and on same-kernel production deployments external readers see the locks and
|
||||
are safe. File as its own issue if production ever runs where a foreign-kernel reader
|
||||
can touch the files.
|
||||
|
||||
---
|
||||
|
||||
> **Original brief follows, preserved as written on 2026-07-20 before root-causing.**
|
||||
|
||||
---
|
||||
|
||||
@@ -95,14 +189,21 @@ becomes oldest-up and takes over):
|
||||
| site-a-b | standby | no | **0** |
|
||||
| site-a-b | active (after failover) | yes | **4 391** |
|
||||
|
||||
### 4.2 It is LocalDb-specific, not the filesystem or the bind mount
|
||||
### 4.2 It is LocalDb-specific, not the filesystem or the bind mount — **WRONG, see §0**
|
||||
|
||||
> **Correction 2026-07-20:** sampling-selection bias — only the LocalDb file was ever read
|
||||
> from the host. Any of these WAL databases is equally poisonable (minimal-repro-proven).
|
||||
|
||||
This is the strongest signal. `store-and-forward.db` and `scadabridge.db` live in the **same
|
||||
bind-mounted directory** (`/app/data`, host `docker/site-a-node-*/data/`), are opened by the
|
||||
**same process**, are also **WAL-mode**, and are being written **concurrently under the same
|
||||
load** — and they log zero errors. Only the LocalDb-managed file fails.
|
||||
|
||||
### 4.3 The observer has been ruled out
|
||||
### 4.3 The observer has been ruled out — **WRONG, see §0: the observer was the cause**
|
||||
|
||||
> **Correction 2026-07-20:** the exclusion below assumed an error-free standby was an
|
||||
> unpoisoned standby. Node-b's files carry the 04:56 sampling-time mtimes (WAL left at
|
||||
> 0 bytes); it was poisoned then and merely silent until failover gave it write traffic.
|
||||
|
||||
Onset (04:56:37) was **one second after** a host-side `sqlite3` read of the bind-mounted
|
||||
database (04:56:36), making observer-induced `-shm` corruption the leading hypothesis. It is
|
||||
@@ -131,11 +232,14 @@ Fully reproducible in ~10 minutes on the local docker rig.
|
||||
Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings:
|
||||
|
||||
- `docker/seed-sites.sh` role names — **already fixed** (commit `cf46e596`).
|
||||
- **`infra/mssql/setup.sql` never executes** — still broken. It is mounted into
|
||||
`/docker-entrypoint-initdb.d/`, which the official `mcr.microsoft.com/mssql/server` image does
|
||||
not implement. After `infra/reseed.sh` drops the volume, nothing creates `ScadaBridgeConfig` or
|
||||
the `scadabridge_app` login and `reseed.sh` hangs forever on its setup.sql poll. Work around
|
||||
by applying the init scripts by hand once MSSQL is accepting connections:
|
||||
- **`infra/mssql/setup.sql` never executes** — **FIXED 2026-07-20**: `infra/reseed.sh` now
|
||||
applies the three init scripts itself via `sqlcmd` once MSSQL accepts connections (the
|
||||
`/docker-entrypoint-initdb.d/` compose mounts are informational only — the official
|
||||
`mcr.microsoft.com/mssql/server` image does not implement that hook; noted in the compose
|
||||
file). The manual workaround below is retained for historical context / older checkouts.
|
||||
Original problem: after `infra/reseed.sh` dropped the volume, nothing created
|
||||
`ScadaBridgeConfig` or the `scadabridge_app` login and `reseed.sh` hung forever on its
|
||||
setup.sql poll. The by-hand equivalent:
|
||||
|
||||
```bash
|
||||
cd ~/Desktop/ScadaBridge/infra
|
||||
@@ -239,6 +343,11 @@ Facts relevant to the failure:
|
||||
|
||||
## 7. Hypotheses, ranked
|
||||
|
||||
> **Resolution 2026-07-20:** none of the four below is the cause. The mechanism is a variant
|
||||
> of #2's territory (bind-mount `-shm`/WAL fragility) but triggered *only* by a host-side
|
||||
> reader — LocalDb's concurrency, pooling, and UDF (hypotheses 1/3/4) are exonerated. The
|
||||
> extended code, since captured, is `SQLITE_IOERR_SHORT_READ` (522).
|
||||
|
||||
None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as
|
||||
healthy WAL databases".
|
||||
|
||||
@@ -279,7 +388,7 @@ likely settle this outright:
|
||||
Add the extended code to the exception logging (or attach a debugger / run the repro against a
|
||||
local non-container build) before pursuing any fix.
|
||||
|
||||
## 8. Secondary defect in the same path
|
||||
## 8. Secondary defect in the same path — **FIXED 2026-07-20**
|
||||
|
||||
```
|
||||
[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
|
||||
@@ -292,8 +401,32 @@ real bug independent of the I/O errors, though it sits in the same write path an
|
||||
contributing. Note the family-wide rule already recorded for Akka work: never read `Self`/`Context`
|
||||
after an `await` inside an actor.
|
||||
|
||||
> **Fixed 2026-07-20.** Root cause: both drain handlers await with `ConfigureAwait(false)`, so
|
||||
> their `finally`-block re-arm (`ScheduleNext`/`ScheduleNextCached`) runs on a pool thread with
|
||||
> no active ActorContext. Investigation found the failure is **bimodal**, and the second mode is
|
||||
> worse than the logged one: depending on what the pool thread's thread-static cell slot holds,
|
||||
> `Context`/`Self` either **throw** `NotSupportedException` (the logged variant — actor crashes
|
||||
> and restarts once per drain) or **silently resolve a STALE cell of whatever actor last ran on
|
||||
> that thread**, re-arming the tick at the *wrong actor* so the drain loop just stops (observed
|
||||
> under TestKit: the tick landed on the TestActor). Fix: capture `Context.System.Scheduler` and
|
||||
> `Self` into fields at construction (both are thread-safe immutable handles) and use only those
|
||||
> from the re-arm path. Regression test
|
||||
> `SiteAuditTelemetryActorTests.Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing`
|
||||
> forces the mock awaits to complete off the actor thread — which every pre-existing test
|
||||
> avoided by returning already-completed tasks — and catches **both** variants (EventFilter for
|
||||
> the throw, sustained-drain counts for the silent stall). AuditLog suite 355/355 green.
|
||||
|
||||
## 9. What a fix must satisfy
|
||||
|
||||
> **Resolution 2026-07-20:** criteria 1–4 are already satisfied by the unmodified code once no
|
||||
> host process touches the live files — verified 10+ min of soak load with zero errors, events
|
||||
> durably written, WAL checkpointing normally. Criterion 5 (a sustained concurrent-write load
|
||||
> test) would **not** have caught this — the trigger is an external reader, not load — but is
|
||||
> now in place anyway: `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (8 concurrent
|
||||
> writers × 250 inserts through fresh pooled connections against a registered/triggered table on
|
||||
> a real file, with concurrent readers; asserts zero failures + exact row/oplog counts; suite
|
||||
> 145/145). §8's `SiteAuditTelemetryActor` async-context bug is **fixed** — see §8.
|
||||
|
||||
1. The §5 repro runs for **30 minutes under sustained load with zero `disk I/O error`** on the
|
||||
active node.
|
||||
2. No `Failed to record event` errors — site events are durably written under load.
|
||||
@@ -313,3 +446,30 @@ after an `await` inside an actor.
|
||||
- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are **still deployed
|
||||
and still generating load** on site-a.
|
||||
- `LdapGroupMappings` corrected in the live DB to the canonical `Designer`/`Deployer` names.
|
||||
|
||||
## 11a. Fix pass (2026-07-20, same day — all verified)
|
||||
|
||||
Everything actionable that this incident identified is now fixed (uncommitted on each repo's
|
||||
current branch; ScadaBridge full solution builds clean, 0 warnings):
|
||||
|
||||
| # | Issue | Fix | Verification |
|
||||
|---|---|---|---|
|
||||
| 1 | §8 `SiteAuditTelemetryActor` async-context bug (bimodal: crash-per-drain OR silent tick misroute) | Capture `Context.System.Scheduler` + `Self` at construction; re-arm path never reads thread-static context | New red→green regression test forcing off-actor-thread continuations; AuditLog suite 355/355 |
|
||||
| 2 | Diagnostics gap: logs carried only the primary SQLite code | `SqliteErrorCodes.Describe` (AuditLog) + `DescribeSqliteError` (SiteEventLogging) — catch sites now log `sqlite <primary>/<extended>` | Builds clean; suites green (this gap cost the investigation a from-scratch repro to learn code 522) |
|
||||
| 3 | §5.1 `reseed.sh` hangs forever waiting on the initdb hook the mssql image doesn't have | `reseed.sh` applies `setup.sql`/`machinedata_seed.sql`/`setup-env2.sql` itself via `sqlcmd`; compose mounts annotated as informational | `bash -n` clean; scripts verified idempotent (`IF NOT EXISTS` guards) |
|
||||
| 4 | §9.5 missing concurrent-write load test | `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (scadaproj) — 8 writers × 250 pooled-connection inserts on a registered table + concurrent readers, exact row/oplog count asserts | LocalDb suite 145/145 |
|
||||
| 5 | Root cause itself (operator/tooling host reads) | Poisonous instructions removed from the Phase 2 plan + `.tasks.json` (safe `snap()` copy-based sampling); soak-doc verdict corrected; family-wide memory rule recorded | On-demand on/off reproduction, §0 |
|
||||
|
||||
Deliberately **not** done: the `SqliteLocalDb` detect-and-reopen self-heal (see §0
|
||||
consequence 4 for the rationale and the condition under which to file it).
|
||||
|
||||
## 11. Rig state after root-causing (2026-07-20 ~05:40 UTC)
|
||||
|
||||
- Both site-a nodes restarted during verification, curing both poisonings. End state:
|
||||
**site-a-b active** carrying the soak load, site-a-a standby, **zero `disk I/O error` on
|
||||
both** under sustained load.
|
||||
- The §10 items still stand: `ExternalSystemDefinitions` id 1 still points at
|
||||
`http://127.0.0.1:9`, and `SoakGenerator` + `soakgen-1..4` are still deployed and
|
||||
generating load — the Phase 2 soak can now proceed on a clean baseline.
|
||||
- Minimal-repro scripts (`writer.py` burst variant, `writer2.py` idle-window variant) lived in
|
||||
the session scratchpad; the recipe is fully described in §0 and takes ~2 minutes to rebuild.
|
||||
|
||||
Reference in New Issue
Block a user