fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect. Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL databases. POSIX advisory locks do not propagate across the virtiofs boundary, so the host reader believes it is the only connection, checkpoints on close and resets the WAL to 0 bytes under the container. The container's still-mapped WAL index then references frames that no longer exist and every subsequent statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently until the process reopens the database. Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and produced the first error one second later) and in a minimal python:3.12-alpine repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened node sustained the full soak load 10+ minutes with zero errors. The original brief's isolation was confounded - both nodes had been poisoned by the same sampling pass, and a poisoned standby looks healthy only because it issues almost no statements. Corrections annotated in place. Hardening shipped alongside: - SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the LocalDb-adjacent catch sites (the missing extended code is what made the original diagnosis so slow). - SiteAuditTelemetryActor: stop touching ActorContext across an await (NotSupportedException), with regression coverage. - infra/reseed.sh: apply the MSSQL init scripts explicitly. The official mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh volume hung the reseed forever; compose mounts annotated as informational. Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends only against external interference, which is now prevented at the source, and same-kernel production readers see the locks correctly. Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
@@ -203,8 +203,23 @@ This is the gate item from `2026-07-19-localdb-phase2-gate.md` §5. It is measur
|
||||
> the oplog sizing arithmetically, and (b) that the Phase 1 oplog stays healthy alongside. The
|
||||
> *empirical* post-cutover drain check lives in Task 20 (evidence item 10).
|
||||
>
|
||||
> Also: the containers have **no `sqlite3` binary** (bare `aspnet:10.0` image), but the data dirs
|
||||
> are **host bind mounts** — run `sqlite3` on the host against `docker/site-a-node-*/data/`.
|
||||
> Also: the containers have **no `sqlite3` binary** (bare `aspnet:10.0` image), and the data dirs
|
||||
> are host bind mounts — but **NEVER run `sqlite3` on the macOS host against the live files**:
|
||||
> host↔container POSIX locks do not propagate across the virtiofs boundary, and a single host-side
|
||||
> read checkpoints + resets the WAL out from under the container, permanently poisoning its
|
||||
> connections (`SQLITE_IOERR_SHORT_READ` → `disk I/O error` storm until restart). This is the
|
||||
> **root cause of the 2026-07-20 "disk I/O error under load" incident** — see
|
||||
> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
|
||||
> Safe sampling: copy the file triplet first (`cp x.db x.db-wal x.db-shm /tmp/…` — plain `cp`
|
||||
> takes no SQLite locks and cannot harm the live DB), then query the copy. The helper used below:
|
||||
>
|
||||
> ```bash
|
||||
> snap() { # snap <node-dir> <db> "<sql>" — query a throwaway copy, never the live file
|
||||
> local d; d=$(mktemp -d)
|
||||
> cp "$1/$2" "$d/" && cp "$1/$2-wal" "$1/$2-shm" "$d/" 2>/dev/null
|
||||
> sqlite3 "$d/$2" "$3"; rm -rf "$d"
|
||||
> }
|
||||
> ```
|
||||
> Metrics are on port **8084** inside the container (not published to the host).
|
||||
|
||||
**Step 1: Bring up the rig with Phase 1 replication enabled**
|
||||
@@ -224,14 +239,14 @@ Expected: `localdb_*` series present (`localdb_oplog_depth`, `localdb_sync_*`
|
||||
`ZB.MOM.WW.LocalDb.Replication`). If absent, the meter allowlist regressed — see
|
||||
`SiteServiceRegistration.ObservedMeters`.
|
||||
|
||||
**Step 2: Capture baselines (host-side, against the bind mounts)**
|
||||
**Step 2: Capture baselines (from throwaway copies — never the live files, see the `snap` helper above)**
|
||||
|
||||
```bash
|
||||
cd ~/Desktop/ScadaBridge/docker
|
||||
sqlite3 site-a-node-a/data/site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;"
|
||||
sqlite3 site-a-node-a/data/store-and-forward.db \
|
||||
snap site-a-node-a/data site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;"
|
||||
snap site-a-node-a/data store-and-forward.db \
|
||||
"SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;"
|
||||
sqlite3 site-a-node-a/data/scadabridge.db "SELECT COUNT(*) FROM native_alarm_state;"
|
||||
snap site-a-node-a/data scadabridge.db "SELECT COUNT(*) FROM native_alarm_state;"
|
||||
```
|
||||
|
||||
Record all values and the wall-clock time. (`COUNT + SUM(retry_count)` is the S&F row-write
|
||||
@@ -253,11 +268,11 @@ Two generators, run concurrently:
|
||||
cd ~/Desktop/ScadaBridge/docker
|
||||
for i in $(seq 1 6); do
|
||||
echo "=== t+$((i*5))m ==="
|
||||
sqlite3 site-a-node-a/data/store-and-forward.db \
|
||||
snap site-a-node-a/data store-and-forward.db \
|
||||
"SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;"
|
||||
sqlite3 site-a-node-a/data/scadabridge.db \
|
||||
snap site-a-node-a/data scadabridge.db \
|
||||
"SELECT COUNT(*), COUNT(CASE WHEN last_transition_at > datetime('now','-5 minutes') THEN 1 END) FROM native_alarm_state;"
|
||||
sqlite3 site-a-node-a/data/site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;"
|
||||
snap site-a-node-a/data site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;"
|
||||
docker exec scadabridge-site-a-a curl -s localhost:8084/metrics \
|
||||
| grep -E '^localdb_(oplog_depth|sync)'
|
||||
sleep 300
|
||||
@@ -271,7 +286,7 @@ upserts — say so in the findings if alarm volume is near a threshold).
|
||||
**Step 5: Measure `config_json` sizes (D6)**
|
||||
|
||||
```bash
|
||||
sqlite3 site-a-node-a/data/scadabridge.db \
|
||||
snap site-a-node-a/data scadabridge.db \
|
||||
"SELECT COUNT(*), MAX(LENGTH(config_json)), AVG(LENGTH(config_json)) FROM deployed_configurations;"
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user