# LocalDb throws `SQLite Error 10: 'disk I/O error'` on the active site node under sustained write load **Date:** 2026-07-20 · **Status:** ROOT-CAUSED + FIX PASS COMPLETE 2026-07-20 (same day) — observer-induced, **not a LocalDb defect**; see §0 (cause) and §11a (fixes) · **Severity:** was High; resolved to an operational rule (now enforced in the tooling docs) + shipped hardening **Area:** `ZB.MOM.WW.LocalDb` (library, `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`) as consumed by ScadaBridge site nodes **Found by:** the Phase 2 rig soak — [`docs/plans/2026-07-19-localdb-phase2-soak.md`](../plans/2026-07-19-localdb-phase2-soak.md) **Branch:** `feat/localdb-phase2` (symptom observed on Phase 1 code) > **Update 2026-07-20:** the mechanism has been identified and reproduced on demand, both in a > minimal SQLite-only repro and on the live rig, and the follow-up fixes have shipped. > Sections §0, §11a and §11 below are authoritative; the original brief (§1–§10) is preserved > as written, with corrections annotated where its conclusions did not survive > (§4.2, §4.3, §7, §9) and fix notes where they did (§5.1, §8). --- ## 0. ROOT CAUSE (verified) **A host-side (macOS) `sqlite3` read of a live, bind-mounted WAL database checkpoints and resets the WAL out from under the container process, permanently poisoning that process's connections.** LocalDb, its connection handling, its UDF, and its triggers are not involved — the mechanism reproduces with plain Python `sqlite3` and no LocalDb code at all. Mechanism, step by step: 1. POSIX advisory locks do **not** propagate across the Docker Desktop virtiofs bind-mount boundary. A host `sqlite3` opening the database cannot see the container's locks (and vice versa), so it believes it is the **only** connection. 2. On close, the "last" connection in WAL mode runs a full checkpoint and **resets the WAL to 0 bytes**. Because the read happened while the container was idle (a standby node, or a gap between write bursts), nothing blocks the checkpoint. This is exactly the file signature found on both rig nodes: main DB mtime + 0-byte `-wal` stamped at the sampling minute. 3. The container process still holds the old WAL-index state (its per-inode `-shm` mapping, kept alive forever by the held-open `_master` connection and the Microsoft.Data.Sqlite connection pool). That index says the WAL contains N frames; the file now has none. Every subsequent statement — reads and writes both consult the WAL index — fails with **`SQLITE_IOERR_SHORT_READ` (extended code 522)**, surfaced as primary code 10 `'disk I/O error'` (some paths surface `SQLITE_NOTADB` (26) instead). The poisoning is **permanent until the process reopens the database** (restart). ### Why the original brief's conclusions were wrong - **"It tracks the load, not the node" (§4.1)** — confounded. *Both* nodes were poisoned by the 04:56 UTC host-side sampling (both nodes' `site-localdb.db` main files carry the 04:56 mtime; node-b's WAL was left at 0 bytes). A poisoned **standby** shows zero errors only because a standby issues ~zero LocalDb statements; the errors "followed the load" because the load is what generates statements against an already-poisoned handle. Node-b's very first write attempt after failover (04:59:37, `OperationTrackingStore.RecordAttemptAsync`) failed — it had been poisoned for 3 minutes with nothing to say about it. - **"It is LocalDb-specific" (§4.2)** — sampling-selection bias. The legacy WAL databases in the same directory were healthy only because no host process ever read *them*. The minimal repro poisons an arbitrary WAL database the same way. - **"The observer has been ruled out" (§4.3)** — the exclusion assumed an error-free standby was an unpoisoned standby. It wasn't; it was a poisoned node with no traffic. ### Verification (2026-07-20, all on the live rig + minimal repro) 1. **Load alone is harmless:** restarted the poisoned active node (site-a-b); freshly-reopened site-a-a took the full soak load for **10+ minutes with zero errors** (the original model predicted onset within ~2 min), WAL growing/checkpointing normally, DB 188 KiB → 476 KiB. 2. **One host read is sufficient and immediate:** a single `sqlite3 docker/site-a-node-a/data/site-localdb.db "SELECT count(*) FROM site_events;"` against the healthy loaded node reset its 4.6 MiB WAL to 0 bytes in place and produced the first `disk I/O error` **one second later** (05:32:48 → 05:32:49), 203 errors in the next 40 s — the same one-second onset correlation as the original 04:56:36 → 04:56:37 incident. 3. **Minimal repro (no LocalDb, no .NET):** a `python:3.12-alpine` container writing a WAL-mode SQLite DB on a bind mount (held master connection + fresh connection per op, `synchronous=NORMAL`, `busy_timeout=5000`). A host `sqlite3 "SELECT count(*)"`: - against the **actively-writing** DB → immediate `SQLITE_IOERR_SHORT_READ` (522) + `SQLITE_NOTADB` burst, then recovery (checkpoint could not fully reset a hot WAL); - during an **idle window** (connections held open, WAL populated) → WAL reset 1.2 MiB → 0 bytes, then **every fresh-connection write failed for the rest of the run (200/200)** — the persistent variant, matching the rig. ### Consequences 1. **The operational rule in §5.3 ("do not query the databases with host-side `sqlite3`") is the root cause, not a hygiene note.** One violation silently destroys the node's local persistence until restart. This applies to *every* WAL SQLite file on the bind mount (`scadabridge.db`, `store-and-forward.db` included), not just LocalDb. 2. **Safe inspection recipes:** copy the file triplet (`.db`, `-wal`, `-shm`) and open the copy; or read from inside the container boundary (same kernel ⇒ locks visible), e.g. `docker run --rm -v :/d alpine/sqlite3 sqlite3 /d/site-localdb.db "..."` — never the macOS host against live files. 3. **LocalDb Phase 2 is unblocked** on this issue: the library sustained the full soak write load indefinitely once nothing external touched its file. 4. Hardening follow-ups — **status as of the 2026-07-20 fix pass (see §11a):** - **DONE — extended-code logging:** the LocalDb-adjacent catch sites (`SiteAuditTelemetryActor`, `CachedCallTelemetryForwarder`, `SiteEventLogger`) now log `SqliteException` primary/extended codes (`sqlite 10/522`-style) via `SqliteErrorCodes.Describe` / `DescribeSqliteError`. - **DONE — §8 async-context bug** (see §8). - **DONE — §9.5 load regression test** (see §9). - **NOT DONE (deliberately):** a detect-and-reopen self-heal in `SqliteLocalDb` for persistent `SQLITE_IOERR`/`SQLITE_NOTADB`. This is a real library design change (pool clear + master reopen + in-flight coordination) protecting against *external interference only* — the trigger is operator/tooling action, now prevented at the source, and on same-kernel production deployments external readers see the locks and are safe. File as its own issue if production ever runs where a foreign-kernel reader can touch the files. --- > **Original brief follows, preserved as written on 2026-07-20 before root-causing.** --- ## 1. Summary On a ScadaBridge site node, once the node is **active** and under sustained concurrent write load, effectively every write to the consolidated LocalDb database (`site-localdb.db`) fails with: ``` Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. ``` Observed rate: **~1 000–1 500 failures per minute**, sustained, not transient. The node stays up and reports healthy. Ordinary (non-LocalDb) SQLite databases in the same directory, in the same process, under the same load, are completely unaffected. ## 2. Why it matters 1. **Silent data loss today.** `SiteEventLogger` fails its inserts and logs `[ERR] Failed to record event: script from ScriptActor:…`. Site event logging is dropping events on the floor on the active node whenever the site is busy. `OperationTracking` writes fail too, which breaks cached-call status tracking (`Cached-telemetry drain: no tracking snapshot for …; skipping`). 2. **It blocks LocalDb Phase 2.** Phase 2 registers eight further tables into this same database — including `native_alarm_state` (highest-volume table on the node) and `sf_messages` — **and deletes the bespoke mechanisms that currently carry that data** (`SiteReplicationActor`, `StoreAndForward.ReplicationService`) in the same commit. Cutting over onto this store while removing the fallback would convert a logging defect into config and buffer loss. 3. Phase 1 was previously live-gated as PASS. That gate exercised correctness and convergence, **not sustained write load** — which is why this was not caught. ## 3. Exact symptom Two representative stacks, both from `docker logs scadabridge-site-a-b` while that node was active and under load: ``` Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. at Microsoft.Data.Sqlite.SqliteDataReader.NextResult() at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior) at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.<>c__DisplayClass15_0.b__0(SqliteConnection connection) in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 236 at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync() in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 221 ``` ``` Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. at Microsoft.Data.Sqlite.SqliteDataReader.NextResult() at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery() at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...) in /src/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:line 137 at ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) in /src/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:line 148 ``` Note both fail inside `SqliteDataReader.NextResult()` — i.e. at statement execution, not at `Open()`. Connections are being acquired successfully; the failure is on the write itself. ### Error-source distribution Error-stack frames counted over one 3-minute window on the loaded node: | Store | Backing file | Frames | |---|---|---| | `OperationTrackingStore` | `site-localdb.db` (**LocalDb**) | 13 044 | | `SiteAuditTelemetryActor` | `site-localdb.db` (**LocalDb**) | 4 350 | | `SiteEventLogger` | `site-localdb.db` (**LocalDb**) | 900 | | `CachedCallTelemetryForwarder` | `site-localdb.db` (**LocalDb**) | 162 | | `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** | | `SiteStorageService` | `scadabridge.db` (legacy) | **0** | ## 4. Evidence — what has been established ### 4.1 It tracks the load, not the node The load was moved between the two site-a nodes by restarting the active one (the surviving node becomes oldest-up and takes over): | Node | Role | Under load | `disk I/O error` / 4 min | |---|---|---|---| | site-a-a | active | yes | 2 175 | | site-a-a | standby (after restart) | no | **0** | | site-a-b | standby | no | **0** | | site-a-b | active (after failover) | yes | **4 391** | ### 4.2 It is LocalDb-specific, not the filesystem or the bind mount — **WRONG, see §0** > **Correction 2026-07-20:** sampling-selection bias — only the LocalDb file was ever read > from the host. Any of these WAL databases is equally poisonable (minimal-repro-proven). This is the strongest signal. `store-and-forward.db` and `scadabridge.db` live in the **same bind-mounted directory** (`/app/data`, host `docker/site-a-node-*/data/`), are opened by the **same process**, are also **WAL-mode**, and are being written **concurrently under the same load** — and they log zero errors. Only the LocalDb-managed file fails. ### 4.3 The observer has been ruled out — **WRONG, see §0: the observer was the cause** > **Correction 2026-07-20:** the exclusion below assumed an error-free standby was an > unpoisoned standby. Node-b's files carry the 04:56 sampling-time mtimes (WAL left at > 0 bytes); it was poisoned then and merely silent until failover gave it write traffic. Onset (04:56:37) was **one second after** a host-side `sqlite3` read of the bind-mounted database (04:56:36), making observer-induced `-shm` corruption the leading hypothesis. It is excluded: - After node-a was restarted (fresh open, `-shm` recovered) and load failed over to node-b, **node-b** — whose files no host process had touched since a single baseline read, and which had been error-free for the entire preceding period — began erroring immediately at a *higher* rate. - **node-a**, whose files *had* been sampled, dropped to zero once it stopped carrying load. The variable that tracks the errors is load. (Host-side `sqlite3` against a live WAL database over a bind mount is still unsafe and should be avoided — it is just not the cause here.) ### 4.4 Not disk pressure Host had 215 GiB free throughout (`df -h`: 76 % used on the data volume). Files are small: `site-localdb.db` 188 KiB, WAL peaked around 4.1 MiB then checkpointed to 0. ## 5. Reproduction Fully reproducible in ~10 minutes on the local docker rig. ### 5.1 Rig prerequisites Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings: - `docker/seed-sites.sh` role names — **already fixed** (commit `cf46e596`). - **`infra/mssql/setup.sql` never executes** — **FIXED 2026-07-20**: `infra/reseed.sh` now applies the three init scripts itself via `sqlcmd` once MSSQL accepts connections (the `/docker-entrypoint-initdb.d/` compose mounts are informational only — the official `mcr.microsoft.com/mssql/server` image does not implement that hook; noted in the compose file). The manual workaround below is retained for historical context / older checkouts. Original problem: after `infra/reseed.sh` dropped the volume, nothing created `ScadaBridgeConfig` or the `scadabridge_app` login and `reseed.sh` hung forever on its setup.sql poll. The by-hand equivalent: ```bash cd ~/Desktop/ScadaBridge/infra for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ -S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$f" done ``` Then restart the app containers so EF migrations run, and restart central again after `seed-sites.sh` writes `LdapGroupMappings` (they are cached at startup). ### 5.2 Build the load generator **The seeded `Motor Controller` template (id 4) cannot be used** — it fails pre-deployment validation with 34 errors (30 `ConnectionBinding`, 4 `ScriptCompilation`). Build a minimal one. **Critical:** `ExternalSystem.Call` does **not** buffer to store-and-forward in practice. `ExternalSystem.CachedCall` is the buffering surface. Using `Call` produces HTTP traffic and no S&F rows, and will not reproduce this. ```bash cd ~/Desktop/ScadaBridge SB=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge # dotnet build src/...CLI first AUTH="--url http://localhost:9000 --username multi-role --password password" # 1. Point the seeded external system at a refusing address (discard port). $SB $AUTH external-system update --id 1 --name "Test REST API" \ --endpoint-url "http://127.0.0.1:9" --auth-type ApiKey --auth-config "scadabridge-test-key-1" # 2. Minimal template: no attributes, no compositions, no connection bindings. $SB $AUTH --format json template create --name "SoakGenerator" # -> note the id $SB $AUTH --format json template script add --template-id --name "SoakCall" \ --trigger-type Interval --trigger-config '{"intervalMs":5000}' \ --code 'var parms = new Dictionary { ["a"] = 2, ["b"] = 3 }; await ExternalSystem.CachedCall("Test REST API", "Add", parms);' # 3. Four instances on site-a (site id 1), then deploy each. for i in 1 2 3 4; do $SB $AUTH --format json instance create --name "soakgen-$i" --template-id --site-id 1 done $SB $AUTH instance deploy --id ``` Note the CLI's `template script update` requires `--name` and `--trigger-type` even when only changing `--code`. In zsh, do not put the auth flags in an unquoted variable — zsh does not word-split, so pass them literally or use `${=AUTH}`. ### 5.3 Observe ```bash # Identify the ACTIVE node — it is the one running the ScriptActors. docker logs --since 4m scadabridge-site-a-a 2>&1 | grep -c "Connection refused" docker logs --since 4m scadabridge-site-a-b 2>&1 | grep -c "Connection refused" # Errors appear on that node within ~2 minutes of load starting. docker logs --since 4m scadabridge-site-a- 2>&1 | grep -c "disk I/O error" ``` Metrics (port 8084 is **not** published, and the `aspnet:10.0` image has **no `curl`**) — use a sidecar in the container's network namespace: ```bash docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest \ -s localhost:8084/metrics | grep '^localdb_' ``` Do **not** query the databases with host-side `sqlite3` while containers are writing them. ## 6. Code map ### Library — `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs` Facts relevant to the failure: - **A `_master` connection is held open for the object's entire lifetime** (`:31`), explicitly to "anchor the WAL journal". It is guarded by a `Lock _masterLock` because `SqliteConnection` is not thread-safe. - **`CreateConnection()` (`:83`) opens a brand-new `SqliteConnection` per call** — one per operation, from many concurrent actors. Every call then runs `PRAGMA synchronous=…; PRAGMA busy_timeout=…; PRAGMA foreign_keys=ON;` and registers a UDF: ```csharp conn.CreateFunction("zb_hlc_next", () => _clock.Next()); ``` - The connection string is **only** `DataSource=` (`:57`) — **connection pooling is left at the Microsoft.Data.Sqlite default (enabled)**, and no `Cache=` or `Mode=` is set. - Effective options on the rig are the defaults: `BusyTimeoutMs = 5000`, `Synchronous = NORMAL`. ScadaBridge's rig config (`docker/site-a-node-*/appsettings.Site.json`, `LocalDb` section) sets only `Path` and the replication block. - `zb_hlc_next()` is invoked **from inside the capture triggers**, i.e. on the SQLite thread during every INSERT/UPDATE/DELETE on a registered table, and it calls into the shared `HybridLogicalClock` from arbitrary threads. ### Failing call sites (ScadaBridge) - `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:221,236` — a channel-drained single-writer loop (`ProcessWriteQueueAsync`) using a `WithConnection(...)` helper. - `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:137` (`RecordEnqueueAsync`), `:260,266` (`GetStatusAsync`). - `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:148`. - `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs` — also see §8. ## 7. Hypotheses, ranked > **Resolution 2026-07-20:** none of the four below is the cause. The mechanism is a variant > of #2's territory (bind-mount `-shm`/WAL fragility) but triggered *only* by a host-side > reader — LocalDb's concurrency, pooling, and UDF (hypotheses 1/3/4) are exonerated. The > extended code, since captured, is `SQLITE_IOERR_SHORT_READ` (522). None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as healthy WAL databases". 1. **Connection churn × pooling × per-connection UDF registration.** LocalDb opens a fresh `SqliteConnection` per operation with pooling enabled, and calls `CreateFunction` on every acquisition. Under high concurrency this drives far more open/close and `-shm` mapping churn than the legacy stores (which reuse a small number of connections), and is the clearest structural difference between the failing and healthy databases. Suspect the interaction of the pool with the long-lived `_master` connection and WAL index growth. 2. **`-shm` / WAL-index growth over the bind mount, triggered only at LocalDb's concurrency.** Would explain why the same mount is fine for lower-concurrency databases. `mmap` of the shared WAL index across virtiofs is a known-fragile area. **Distinguishing test: run the same load with `LocalDb:Path` pointed at a container-local path (a `tmpfs` or a plain volume rather than the bind mount).** If the errors vanish, this is confirmed and the fix is environmental / deployment-shaped rather than a library bug. **Run this test first — it is cheap and it partitions the hypothesis space.** 3. **`zb_hlc_next` UDF failing inside a trigger.** An exception thrown out of the managed UDF callback during trigger execution can surface as a generic SQLite error at the statement level. Check `HybridLogicalClock.Next()` for thread-safety and for anything that can throw under contention (e.g. a spin/overflow path when many callers request stamps in the same millisecond). 4. **Busy-timeout exhaustion misreported.** `BusyTimeoutMs = 5000` with heavy multi-connection write contention on one file. This would normally surface as `SQLITE_BUSY` (5), not `SQLITE_IOERR` (10), so it is a weaker fit — but worth excluding. ### The single highest-value next step **Capture the extended result code.** The logs only show the primary code (`10` = `SQLITE_IOERR`), which is generic. `SqliteException.SqliteExtendedErrorCode` names the failing syscall and would likely settle this outright: | Extended code | Meaning | Points at | |---|---|---| | `SQLITE_IOERR_SHMMAP` (6154) / `SQLITE_IOERR_SHMSIZE` (4874) | WAL index mmap/resize failed | hypothesis 2 | | `SQLITE_IOERR_WRITE` (778) / `SQLITE_IOERR_FSYNC` (1034) | plain write/fsync failed | filesystem | | `SQLITE_IOERR_LOCK` (3850) | file locking failed | bind mount locking | Add the extended code to the exception logging (or attach a debugger / run the repro against a local non-container build) before pursuing any fix. ## 8. Secondary defect in the same path — **FIXED 2026-07-20** ``` [ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext, this is most likely due to use of async operations from within this actor. Cause: System.NotSupportedException ``` `SiteAuditTelemetryActor` is touching `Context` (or `Self`/`Sender`) after an `await`. This is a real bug independent of the I/O errors, though it sits in the same write path and may be contributing. Note the family-wide rule already recorded for Akka work: never read `Self`/`Context` after an `await` inside an actor. > **Fixed 2026-07-20.** Root cause: both drain handlers await with `ConfigureAwait(false)`, so > their `finally`-block re-arm (`ScheduleNext`/`ScheduleNextCached`) runs on a pool thread with > no active ActorContext. Investigation found the failure is **bimodal**, and the second mode is > worse than the logged one: depending on what the pool thread's thread-static cell slot holds, > `Context`/`Self` either **throw** `NotSupportedException` (the logged variant — actor crashes > and restarts once per drain) or **silently resolve a STALE cell of whatever actor last ran on > that thread**, re-arming the tick at the *wrong actor* so the drain loop just stops (observed > under TestKit: the tick landed on the TestActor). Fix: capture `Context.System.Scheduler` and > `Self` into fields at construction (both are thread-safe immutable handles) and use only those > from the re-arm path. Regression test > `SiteAuditTelemetryActorTests.Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing` > forces the mock awaits to complete off the actor thread — which every pre-existing test > avoided by returning already-completed tasks — and catches **both** variants (EventFilter for > the throw, sustained-drain counts for the silent stall). AuditLog suite 355/355 green. ## 9. What a fix must satisfy > **Resolution 2026-07-20:** criteria 1–4 are already satisfied by the unmodified code once no > host process touches the live files — verified 10+ min of soak load with zero errors, events > durably written, WAL checkpointing normally. Criterion 5 (a sustained concurrent-write load > test) would **not** have caught this — the trigger is an external reader, not load — but is > now in place anyway: `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (8 concurrent > writers × 250 inserts through fresh pooled connections against a registered/triggered table on > a real file, with concurrent readers; asserts zero failures + exact row/oplog counts; suite > 145/145). §8's `SiteAuditTelemetryActor` async-context bug is **fixed** — see §8. 1. The §5 repro runs for **30 minutes under sustained load with zero `disk I/O error`** on the active node. 2. No `Failed to record event` errors — site events are durably written under load. 3. `localdb_oplog_depth` rises under load and **drains** between bursts; zero dead letters. 4. Replication still converges across the site-a pair (Phase 1's existing convergence suite and live gate still pass). 5. A regression test that would have caught this — i.e. a **concurrent-write load test** against a real LocalDb file, not just the correctness/convergence tests Phase 1 shipped. Phase 1's gate passed precisely because no test applied sustained concurrent write pressure. ## 10. Rig state as left - Rig fully reseeded; central config volume dropped and replayed; site SQLite state wiped (`reseed.sh` stage 2 does `rm -rf docker/site-*/data/*`). - `ExternalSystemDefinitions` id 1 is **still repointed to `http://127.0.0.1:9`** — restore to `http://scadabridge-restapi:5200` when done. - Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are **still deployed and still generating load** on site-a. - `LdapGroupMappings` corrected in the live DB to the canonical `Designer`/`Deployer` names. ## 11a. Fix pass (2026-07-20, same day — all verified) Everything actionable that this incident identified is now fixed (uncommitted on each repo's current branch; ScadaBridge full solution builds clean, 0 warnings): | # | Issue | Fix | Verification | |---|---|---|---| | 1 | §8 `SiteAuditTelemetryActor` async-context bug (bimodal: crash-per-drain OR silent tick misroute) | Capture `Context.System.Scheduler` + `Self` at construction; re-arm path never reads thread-static context | New red→green regression test forcing off-actor-thread continuations; AuditLog suite 355/355 | | 2 | Diagnostics gap: logs carried only the primary SQLite code | `SqliteErrorCodes.Describe` (AuditLog) + `DescribeSqliteError` (SiteEventLogging) — catch sites now log `sqlite /` | Builds clean; suites green (this gap cost the investigation a from-scratch repro to learn code 522) | | 3 | §5.1 `reseed.sh` hangs forever waiting on the initdb hook the mssql image doesn't have | `reseed.sh` applies `setup.sql`/`machinedata_seed.sql`/`setup-env2.sql` itself via `sqlcmd`; compose mounts annotated as informational | `bash -n` clean; scripts verified idempotent (`IF NOT EXISTS` guards) | | 4 | §9.5 missing concurrent-write load test | `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (scadaproj) — 8 writers × 250 pooled-connection inserts on a registered table + concurrent readers, exact row/oplog count asserts | LocalDb suite 145/145 | | 5 | Root cause itself (operator/tooling host reads) | Poisonous instructions removed from the Phase 2 plan + `.tasks.json` (safe `snap()` copy-based sampling); soak-doc verdict corrected; family-wide memory rule recorded | On-demand on/off reproduction, §0 | Deliberately **not** done: the `SqliteLocalDb` detect-and-reopen self-heal (see §0 consequence 4 for the rationale and the condition under which to file it). ## 11. Rig state after root-causing (2026-07-20 ~05:40 UTC) - Both site-a nodes restarted during verification, curing both poisonings. End state: **site-a-b active** carrying the soak load, site-a-a standby, **zero `disk I/O error` on both** under sustained load. - The §10 items still stand: `ExternalSystemDefinitions` id 1 still points at `http://127.0.0.1:9`, and `SoakGenerator` + `soakgen-1..4` are still deployed and generating load — the Phase 2 soak can now proceed on a clean baseline. - Minimal-repro scripts (`writer.py` burst variant, `writer2.py` idle-window variant) lived in the session scratchpad; the recipe is fully described in §0 and takes ~2 minutes to rebuild.