ScadaBridge/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md

LocalDb throws SQLite Error 10: 'disk I/O error' on the active site node under sustained write load

Date: 2026-07-20 · Status: ROOT-CAUSED + FIX PASS COMPLETE 2026-07-20 (same day) — observer-induced, not a LocalDb defect; see §0 (cause) and §11a (fixes) · Severity: was High; resolved to an operational rule (now enforced in the tooling docs) + shipped hardening Area: ZB.MOM.WW.LocalDb (library, ~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/) as consumed by ScadaBridge site nodes Found by: the Phase 2 rig soak — docs/plans/2026-07-19-localdb-phase2-soak.md Branch: feat/localdb-phase2 (symptom observed on Phase 1 code)

Update 2026-07-20: the mechanism has been identified and reproduced on demand, both in a minimal SQLite-only repro and on the live rig, and the follow-up fixes have shipped. Sections §0, §11a and §11 below are authoritative; the original brief (§1–§10) is preserved as written, with corrections annotated where its conclusions did not survive (§4.2, §4.3, §7, §9) and fix notes where they did (§5.1, §8).

---

0. ROOT CAUSE (verified)

A host-side (macOS) sqlite3 read of a live, bind-mounted WAL database checkpoints and resets the WAL out from under the container process, permanently poisoning that process's connections. LocalDb, its connection handling, its UDF, and its triggers are not involved — the mechanism reproduces with plain Python sqlite3 and no LocalDb code at all.

Mechanism, step by step:

1. POSIX advisory locks do not propagate across the Docker Desktop virtiofs bind-mount boundary. A host sqlite3 opening the database cannot see the container's locks (and vice versa), so it believes it is the only connection.
2. On close, the "last" connection in WAL mode runs a full checkpoint and resets the WAL to 0 bytes. Because the read happened while the container was idle (a standby node, or a gap between write bursts), nothing blocks the checkpoint. This is exactly the file signature found on both rig nodes: main DB mtime + 0-byte -wal stamped at the sampling minute.
3. The container process still holds the old WAL-index state (its per-inode -shm mapping, kept alive forever by the held-open _master connection and the Microsoft.Data.Sqlite connection pool). That index says the WAL contains N frames; the file now has none. Every subsequent statement — reads and writes both consult the WAL index — fails with SQLITE_IOERR_SHORT_READ (extended code 522), surfaced as primary code 10 'disk I/O error' (some paths surface SQLITE_NOTADB (26) instead). The poisoning is permanent until the process reopens the database (restart).

Why the original brief's conclusions were wrong

• "It tracks the load, not the node" (§4.1) — confounded. Both nodes were poisoned by the 04:56 UTC host-side sampling (both nodes' site-localdb.db main files carry the 04:56 mtime; node-b's WAL was left at 0 bytes). A poisoned standby shows zero errors only because a standby issues ~zero LocalDb statements; the errors "followed the load" because the load is what generates statements against an already-poisoned handle. Node-b's very first write attempt after failover (04:59:37, OperationTrackingStore.RecordAttemptAsync) failed — it had been poisoned for 3 minutes with nothing to say about it.
• "It is LocalDb-specific" (§4.2) — sampling-selection bias. The legacy WAL databases in the same directory were healthy only because no host process ever read them. The minimal repro poisons an arbitrary WAL database the same way.
• "The observer has been ruled out" (§4.3) — the exclusion assumed an error-free standby was an unpoisoned standby. It wasn't; it was a poisoned node with no traffic.

Verification (2026-07-20, all on the live rig + minimal repro)

1. Load alone is harmless: restarted the poisoned active node (site-a-b); freshly-reopened site-a-a took the full soak load for 10+ minutes with zero errors (the original model predicted onset within ~2 min), WAL growing/checkpointing normally, DB 188 KiB → 476 KiB.
2. One host read is sufficient and immediate: a single sqlite3 docker/site-a-node-a/data/site-localdb.db "SELECT count(*) FROM site_events;" against the healthy loaded node reset its 4.6 MiB WAL to 0 bytes in place and produced the first disk I/O error one second later (05:32:48 → 05:32:49), 203 errors in the next 40 s — the same one-second onset correlation as the original 04:56:36 → 04:56:37 incident.
3. Minimal repro (no LocalDb, no .NET): a python:3.12-alpine container writing a WAL-mode SQLite DB on a bind mount (held master connection + fresh connection per op, synchronous=NORMAL, busy_timeout=5000). A host sqlite3 "SELECT count(*)":
   • against the actively-writing DB → immediate SQLITE_IOERR_SHORT_READ (522) + SQLITE_NOTADB burst, then recovery (checkpoint could not fully reset a hot WAL);
   • during an idle window (connections held open, WAL populated) → WAL reset 1.2 MiB → 0 bytes, then every fresh-connection write failed for the rest of the run (200/200) — the persistent variant, matching the rig.

Consequences

1. The operational rule in §5.3 ("do not query the databases with host-side sqlite3") is the root cause, not a hygiene note. One violation silently destroys the node's local persistence until restart. This applies to every WAL SQLite file on the bind mount (scadabridge.db, store-and-forward.db included), not just LocalDb.
2. Safe inspection recipes: copy the file triplet (.db, -wal, -shm) and open the copy; or read from inside the container boundary (same kernel ⇒ locks visible), e.g. docker run --rm -v <dir>:/d alpine/sqlite3 sqlite3 /d/site-localdb.db "..." — never the macOS host against live files.
3. LocalDb Phase 2 is unblocked on this issue: the library sustained the full soak write load indefinitely once nothing external touched its file.
4. Hardening follow-ups — status as of the 2026-07-20 fix pass (see §11a):
   • DONE — extended-code logging: the LocalDb-adjacent catch sites (SiteAuditTelemetryActor, CachedCallTelemetryForwarder, SiteEventLogger) now log SqliteException primary/extended codes (sqlite 10/522-style) via SqliteErrorCodes.Describe / DescribeSqliteError.
   • DONE — §8 async-context bug (see §8).
   • DONE — §9.5 load regression test (see §9).
   • NOT DONE (deliberately): a detect-and-reopen self-heal in SqliteLocalDb for persistent SQLITE_IOERR/SQLITE_NOTADB. This is a real library design change (pool clear + master reopen + in-flight coordination) protecting against external interference only — the trigger is operator/tooling action, now prevented at the source, and on same-kernel production deployments external readers see the locks and are safe. File as its own issue if production ever runs where a foreign-kernel reader can touch the files.

---

Original brief follows, preserved as written on 2026-07-20 before root-causing.

---

1. Summary

On a ScadaBridge site node, once the node is active and under sustained concurrent write load, effectively every write to the consolidated LocalDb database (site-localdb.db) fails with:

Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.

Observed rate: ~1 000–1 500 failures per minute, sustained, not transient. The node stays up and reports healthy. Ordinary (non-LocalDb) SQLite databases in the same directory, in the same process, under the same load, are completely unaffected.

2. Why it matters

1. Silent data loss today. SiteEventLogger fails its inserts and logs [ERR] Failed to record event: script from ScriptActor:…. Site event logging is dropping events on the floor on the active node whenever the site is busy. OperationTracking writes fail too, which breaks cached-call status tracking (Cached-telemetry drain: no tracking snapshot for …; skipping).
2. It blocks LocalDb Phase 2. Phase 2 registers eight further tables into this same database — including native_alarm_state (highest-volume table on the node) and sf_messages — and deletes the bespoke mechanisms that currently carry that data (SiteReplicationActor, StoreAndForward.ReplicationService) in the same commit. Cutting over onto this store while removing the fallback would convert a logging defect into config and buffer loss.
3. Phase 1 was previously live-gated as PASS. That gate exercised correctness and convergence, not sustained write load — which is why this was not caught.

3. Exact symptom

Two representative stacks, both from docker logs scadabridge-site-a-b while that node was active and under load:

Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
   at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.<>c__DisplayClass15_0.<ProcessWriteQueueAsync>b__0(SqliteConnection connection)
      in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 236
   at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync()
      in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 221

Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
   at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
   at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
   at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...)
      in /src/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:line 137
   at ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.CachedCallTelemetryForwarder.TryEmitTrackingAsync(...)
      in /src/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:line 148

Note both fail inside SqliteDataReader.NextResult() — i.e. at statement execution, not at Open(). Connections are being acquired successfully; the failure is on the write itself.

Error-source distribution

Error-stack frames counted over one 3-minute window on the loaded node:

Store	Backing file	Frames
OperationTrackingStore	site-localdb.db (LocalDb)	13 044
SiteAuditTelemetryActor	site-localdb.db (LocalDb)	4 350
SiteEventLogger	site-localdb.db (LocalDb)	900
CachedCallTelemetryForwarder	site-localdb.db (LocalDb)	162
StoreAndForwardStorage	store-and-forward.db (legacy)	0
SiteStorageService	scadabridge.db (legacy)	0

4. Evidence — what has been established

4.1 It tracks the load, not the node

The load was moved between the two site-a nodes by restarting the active one (the surviving node becomes oldest-up and takes over):

Node	Role	Under load	disk I/O error / 4 min
site-a-a	active	yes	2 175
site-a-a	standby (after restart)	no	0
site-a-b	standby	no	0
site-a-b	active (after failover)	yes	4 391

4.2 It is LocalDb-specific, not the filesystem or the bind mount — WRONG, see §0

Correction 2026-07-20: sampling-selection bias — only the LocalDb file was ever read from the host. Any of these WAL databases is equally poisonable (minimal-repro-proven).

This is the strongest signal. store-and-forward.db and scadabridge.db live in the same bind-mounted directory (/app/data, host docker/site-a-node-*/data/), are opened by the same process, are also WAL-mode, and are being written concurrently under the same load — and they log zero errors. Only the LocalDb-managed file fails.

4.3 The observer has been ruled out — WRONG, see §0: the observer was the cause

Correction 2026-07-20: the exclusion below assumed an error-free standby was an unpoisoned standby. Node-b's files carry the 04:56 sampling-time mtimes (WAL left at 0 bytes); it was poisoned then and merely silent until failover gave it write traffic.

Onset (04:56:37) was one second after a host-side sqlite3 read of the bind-mounted database (04:56:36), making observer-induced -shm corruption the leading hypothesis. It is excluded:

• After node-a was restarted (fresh open, -shm recovered) and load failed over to node-b, node-b — whose files no host process had touched since a single baseline read, and which had been error-free for the entire preceding period — began erroring immediately at a higher rate.
• node-a, whose files had been sampled, dropped to zero once it stopped carrying load.

The variable that tracks the errors is load. (Host-side sqlite3 against a live WAL database over a bind mount is still unsafe and should be avoided — it is just not the cause here.)

4.4 Not disk pressure

Host had 215 GiB free throughout (df -h: 76 % used on the data volume). Files are small: site-localdb.db 188 KiB, WAL peaked around 4.1 MiB then checkpointed to 0.

5. Reproduction

Fully reproducible in ~10 minutes on the local docker rig.

5.1 Rig prerequisites

Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings:

• docker/seed-sites.sh role names — already fixed (commit cf46e596).

• infra/mssql/setup.sql never executes — FIXED 2026-07-20: infra/reseed.sh now applies the three init scripts itself via sqlcmd once MSSQL accepts connections (the /docker-entrypoint-initdb.d/ compose mounts are informational only — the official mcr.microsoft.com/mssql/server image does not implement that hook; noted in the compose file). The manual workaround below is retained for historical context / older checkouts. Original problem: after infra/reseed.sh dropped the volume, nothing created ScadaBridgeConfig or the scadabridge_app login and reseed.sh hung forever on its setup.sql poll. The by-hand equivalent:

  cd ~/Desktop/ScadaBridge/infra
  for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do
    docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
      -S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$f"
  done
  

  Then restart the app containers so EF migrations run, and restart central again after seed-sites.sh writes LdapGroupMappings (they are cached at startup).

5.2 Build the load generator

The seeded Motor Controller template (id 4) cannot be used — it fails pre-deployment validation with 34 errors (30 ConnectionBinding, 4 ScriptCompilation). Build a minimal one.

Critical: ExternalSystem.Call does not buffer to store-and-forward in practice. ExternalSystem.CachedCall is the buffering surface. Using Call produces HTTP traffic and no S&F rows, and will not reproduce this.

cd ~/Desktop/ScadaBridge
SB=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge   # dotnet build src/...CLI first
AUTH="--url http://localhost:9000 --username multi-role --password password"

# 1. Point the seeded external system at a refusing address (discard port).
$SB $AUTH external-system update --id 1 --name "Test REST API" \
    --endpoint-url "http://127.0.0.1:9" --auth-type ApiKey --auth-config "scadabridge-test-key-1"

# 2. Minimal template: no attributes, no compositions, no connection bindings.
$SB $AUTH --format json template create --name "SoakGenerator"      # -> note the id

$SB $AUTH --format json template script add --template-id <TID> --name "SoakCall" \
    --trigger-type Interval --trigger-config '{"intervalMs":5000}' \
    --code 'var parms = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 }; await ExternalSystem.CachedCall("Test REST API", "Add", parms);'

# 3. Four instances on site-a (site id 1), then deploy each.
for i in 1 2 3 4; do
  $SB $AUTH --format json instance create --name "soakgen-$i" --template-id <TID> --site-id 1
done
$SB $AUTH instance deploy --id <each instance id>

Note the CLI's template script update requires --name and --trigger-type even when only changing --code. In zsh, do not put the auth flags in an unquoted variable — zsh does not word-split, so pass them literally or use ${=AUTH}.

5.3 Observe

# Identify the ACTIVE node — it is the one running the ScriptActors.
docker logs --since 4m scadabridge-site-a-a 2>&1 | grep -c "Connection refused"
docker logs --since 4m scadabridge-site-a-b 2>&1 | grep -c "Connection refused"

# Errors appear on that node within ~2 minutes of load starting.
docker logs --since 4m scadabridge-site-a-<active> 2>&1 | grep -c "disk I/O error"

Metrics (port 8084 is not published, and the aspnet:10.0 image has no curl) — use a sidecar in the container's network namespace:

docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest \
  -s localhost:8084/metrics | grep '^localdb_'

Do not query the databases with host-side sqlite3 while containers are writing them.

6. Code map

Library — ~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs

Facts relevant to the failure:

• A _master connection is held open for the object's entire lifetime (:31), explicitly to "anchor the WAL journal". It is guarded by a Lock _masterLock because SqliteConnection is not thread-safe.
• CreateConnection() (:83) opens a brand-new SqliteConnection per call — one per operation, from many concurrent actors. Every call then runs PRAGMA synchronous=…; PRAGMA busy_timeout=…; PRAGMA foreign_keys=ON; and registers a UDF:

  conn.CreateFunction("zb_hlc_next", () => _clock.Next());
  

• The connection string is only DataSource=<path> (:57) — connection pooling is left at the Microsoft.Data.Sqlite default (enabled), and no Cache= or Mode= is set.
• Effective options on the rig are the defaults: BusyTimeoutMs = 5000, Synchronous = NORMAL. ScadaBridge's rig config (docker/site-a-node-*/appsettings.Site.json, LocalDb section) sets only Path and the replication block.
• zb_hlc_next() is invoked from inside the capture triggers, i.e. on the SQLite thread during every INSERT/UPDATE/DELETE on a registered table, and it calls into the shared HybridLogicalClock from arbitrary threads.

Failing call sites (ScadaBridge)

• src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:221,236 — a channel-drained single-writer loop (ProcessWriteQueueAsync) using a WithConnection(...) helper.
• src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:137 (RecordEnqueueAsync), :260,266 (GetStatusAsync).
• src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:148.
• src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs — also see §8.

7. Hypotheses, ranked

Resolution 2026-07-20: none of the four below is the cause. The mechanism is a variant of #2's territory (bind-mount -shm/WAL fragility) but triggered only by a host-side reader — LocalDb's concurrency, pooling, and UDF (hypotheses 1/3/4) are exonerated. The extended code, since captured, is SQLITE_IOERR_SHORT_READ (522).

None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as healthy WAL databases".

1. Connection churn × pooling × per-connection UDF registration. LocalDb opens a fresh SqliteConnection per operation with pooling enabled, and calls CreateFunction on every acquisition. Under high concurrency this drives far more open/close and -shm mapping churn than the legacy stores (which reuse a small number of connections), and is the clearest structural difference between the failing and healthy databases. Suspect the interaction of the pool with the long-lived _master connection and WAL index growth.
2. -shm / WAL-index growth over the bind mount, triggered only at LocalDb's concurrency. Would explain why the same mount is fine for lower-concurrency databases. mmap of the shared WAL index across virtiofs is a known-fragile area. Distinguishing test: run the same load with LocalDb:Path pointed at a container-local path (a tmpfs or a plain volume rather than the bind mount). If the errors vanish, this is confirmed and the fix is environmental / deployment-shaped rather than a library bug. Run this test first — it is cheap and it partitions the hypothesis space.
3. zb_hlc_next UDF failing inside a trigger. An exception thrown out of the managed UDF callback during trigger execution can surface as a generic SQLite error at the statement level. Check HybridLogicalClock.Next() for thread-safety and for anything that can throw under contention (e.g. a spin/overflow path when many callers request stamps in the same millisecond).
4. Busy-timeout exhaustion misreported. BusyTimeoutMs = 5000 with heavy multi-connection write contention on one file. This would normally surface as SQLITE_BUSY (5), not SQLITE_IOERR (10), so it is a weaker fit — but worth excluding.

The single highest-value next step

Capture the extended result code. The logs only show the primary code (10 = SQLITE_IOERR), which is generic. SqliteException.SqliteExtendedErrorCode names the failing syscall and would likely settle this outright:

Extended code	Meaning	Points at
SQLITE_IOERR_SHMMAP (6154) / SQLITE_IOERR_SHMSIZE (4874)	WAL index mmap/resize failed	hypothesis 2
SQLITE_IOERR_WRITE (778) / SQLITE_IOERR_FSYNC (1034)	plain write/fsync failed	filesystem
SQLITE_IOERR_LOCK (3850)	file locking failed	bind mount locking

Add the extended code to the exception logging (or attach a debugger / run the repro against a local non-container build) before pursuing any fix.

8. Secondary defect in the same path — FIXED 2026-07-20

[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
this is most likely due to use of async operations from within this actor.
Cause: System.NotSupportedException

SiteAuditTelemetryActor is touching Context (or Self/Sender) after an await. This is a real bug independent of the I/O errors, though it sits in the same write path and may be contributing. Note the family-wide rule already recorded for Akka work: never read Self/Context after an await inside an actor.

Fixed 2026-07-20. Root cause: both drain handlers await with ConfigureAwait(false), so their finally-block re-arm (ScheduleNext/ScheduleNextCached) runs on a pool thread with no active ActorContext. Investigation found the failure is bimodal, and the second mode is worse than the logged one: depending on what the pool thread's thread-static cell slot holds, Context/Self either throw NotSupportedException (the logged variant — actor crashes and restarts once per drain) or silently resolve a STALE cell of whatever actor last ran on that thread, re-arming the tick at the wrong actor so the drain loop just stops (observed under TestKit: the tick landed on the TestActor). Fix: capture Context.System.Scheduler and Self into fields at construction (both are thread-safe immutable handles) and use only those from the re-arm path. Regression test SiteAuditTelemetryActorTests.Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing forces the mock awaits to complete off the actor thread — which every pre-existing test avoided by returning already-completed tasks — and catches both variants (EventFilter for the throw, sustained-drain counts for the silent stall). AuditLog suite 355/355 green.

9. What a fix must satisfy

Resolution 2026-07-20: criteria 1–4 are already satisfied by the unmodified code once no host process touches the live files — verified 10+ min of soak load with zero errors, events durably written, WAL checkpointing normally. Criterion 5 (a sustained concurrent-write load test) would not have caught this — the trigger is an external reader, not load — but is now in place anyway: ConcurrentWriteLoadTests in ZB.MOM.WW.LocalDb.Tests (8 concurrent writers × 250 inserts through fresh pooled connections against a registered/triggered table on a real file, with concurrent readers; asserts zero failures + exact row/oplog counts; suite 145/145). §8's SiteAuditTelemetryActor async-context bug is fixed — see §8.

1. The §5 repro runs for 30 minutes under sustained load with zero disk I/O error on the active node.
2. No Failed to record event errors — site events are durably written under load.
3. localdb_oplog_depth rises under load and drains between bursts; zero dead letters.
4. Replication still converges across the site-a pair (Phase 1's existing convergence suite and live gate still pass).
5. A regression test that would have caught this — i.e. a concurrent-write load test against a real LocalDb file, not just the correctness/convergence tests Phase 1 shipped. Phase 1's gate passed precisely because no test applied sustained concurrent write pressure.

10. Rig state as left

• Rig fully reseeded; central config volume dropped and replayed; site SQLite state wiped (reseed.sh stage 2 does rm -rf docker/site-*/data/*).
• ExternalSystemDefinitions id 1 is still repointed to http://127.0.0.1:9 — restore to http://scadabridge-restapi:5200 when done.
• Template SoakGenerator (id 2021) and instances soakgen-1..4 (ids 5–8) are still deployed and still generating load on site-a.
• LdapGroupMappings corrected in the live DB to the canonical Designer/Deployer names.

11a. Fix pass (2026-07-20, same day — all verified)

Everything actionable that this incident identified is now fixed (uncommitted on each repo's current branch; ScadaBridge full solution builds clean, 0 warnings):

#	Issue	Fix	Verification
1	§8 SiteAuditTelemetryActor async-context bug (bimodal: crash-per-drain OR silent tick misroute)	Capture Context.System.Scheduler + Self at construction; re-arm path never reads thread-static context	New red→green regression test forcing off-actor-thread continuations; AuditLog suite 355/355
2	Diagnostics gap: logs carried only the primary SQLite code	SqliteErrorCodes.Describe (AuditLog) + DescribeSqliteError (SiteEventLogging) — catch sites now log sqlite <primary>/<extended>	Builds clean; suites green (this gap cost the investigation a from-scratch repro to learn code 522)
3	§5.1 reseed.sh hangs forever waiting on the initdb hook the mssql image doesn't have	reseed.sh applies setup.sql/machinedata_seed.sql/setup-env2.sql itself via sqlcmd; compose mounts annotated as informational	bash -n clean; scripts verified idempotent (IF NOT EXISTS guards)
4	§9.5 missing concurrent-write load test	ConcurrentWriteLoadTests in ZB.MOM.WW.LocalDb.Tests (scadaproj) — 8 writers × 250 pooled-connection inserts on a registered table + concurrent readers, exact row/oplog count asserts	LocalDb suite 145/145
5	Root cause itself (operator/tooling host reads)	Poisonous instructions removed from the Phase 2 plan + .tasks.json (safe snap() copy-based sampling); soak-doc verdict corrected; family-wide memory rule recorded	On-demand on/off reproduction, §0

Deliberately not done: the SqliteLocalDb detect-and-reopen self-heal (see §0 consequence 4 for the rationale and the condition under which to file it).

11. Rig state after root-causing (2026-07-20 ~05:40 UTC)

• Both site-a nodes restarted during verification, curing both poisonings. End state: site-a-b active carrying the soak load, site-a-a standby, zero disk I/O error on both under sustained load.
• The §10 items still stand: ExternalSystemDefinitions id 1 still points at http://127.0.0.1:9, and SoakGenerator + soakgen-1..4 are still deployed and generating load — the Phase 2 soak can now proceed on a clean baseline.
• Minimal-repro scripts (writer.py burst variant, writer2.py idle-window variant) lived in the session scratchpad; the recipe is fully described in §0 and takes ~2 minutes to rebuild.